2024-09-18T07:46:08.617 INFO:root:teuthology version: 1.2.2.dev2+g3752d38 2024-09-18T07:46:08.926 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-09-18T07:46:09.011 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/gabrioux-2024-09-18_06:59:44-orch:cephadm-wip-guits-main-2024-09-17-1213-distro-default-smithi/7910340 branch: wip-guits-main-2024-09-17-1213 description: orch:cephadm/thrash/{0-distro/centos_9.stream 1-start 2-thrash 3-tasks/radosbench fixed-2 msgr/async-v2only root} email: gabrioux@ibm.com first_in_suite: false job_id: '7910340' kernel: kdb: 1 sha1: distro last_in_suite: false machine_type: smithi name: gabrioux-2024-09-18_06:59:44-orch:cephadm-wip-guits-main-2024-09-17-1213-distro-default-smithi no_nested_subset: false openstack: - volumes: count: 4 size: 10 os_type: centos os_version: 9.stream overrides: admin_socket: branch: wip-guits-main-2024-09-17-1213 ceph: conf: client.0: debug ms: 1 debug objecter: 20 debug rados: 20 global: ms bind msgr1: false ms bind msgr2: true ms type: async mgr: debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 mon min osdmap epochs: 50 mon osdmap full prune interval: 2 mon osdmap full prune min: 15 mon osdmap full prune txsize: 2 paxos service trim min: 10 osd: debug ms: 1 debug osd: 20 osd debug reject backfill probability: 0.3 osd delete sleep: 1 osd max backfills: 3 osd scrub max interval: 120 osd scrub min interval: 60 osd shutdown pgref assert: true osd snap trim sleep: 2 flavor: default log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - CEPHADM_STRAY_DAEMON - CEPHADM_FAILED_DAEMON - but it is still running - objects unfound and apparently lost - \(POOL_APP_NOT_ENABLED\) log-only-match: - CEPHADM_ sha1: e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e ceph-deploy: conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: {} cephadm: cephadm_mode: root install: ceph: flavor: default sha1: e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e selinux: allowlist: - scontext=system_u:system_r:logrotate_t:s0 - scontext=system_u:system_r:getty_t:s0 workunit: branch: wip-guits-main-2024-09-17-1213 sha1: e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e owner: scheduled_gabrioux@teuthology priority: 70 repo: https://git.ceph.com/ceph-ci.git roles: - - mon.a - mon.c - mgr.y - osd.0 - osd.1 - osd.2 - osd.3 - client.0 - ceph.rgw.foo.a - node-exporter.a - alertmanager.a - - mon.b - mgr.x - osd.4 - osd.5 - osd.6 - osd.7 - client.1 - prometheus.a - grafana.a - node-exporter.b - ceph.iscsi.iscsi.a seed: 8982 sha1: e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e sleep_before_teardown: 0 subset: 1/8 suite: orch:cephadm suite_branch: wip-guits-main-2024-09-17-1213 suite_path: /home/teuthworker/src/git.ceph.com_ceph-c_e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e/qa suite_relpath: qa suite_repo: https://git.ceph.com/ceph-ci.git suite_sha1: e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e targets: smithi007.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBPYkVuXWvxdNGFoUgrY3wEvIEMmW46TDgl+jyvRxWqmr12Unz0FidCtrdbIC+JvIs19BZXuMX0CLiiSlaGRjhIs= smithi050.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBEO6Sl2zyfi/P/W9j03+TnhJETbGDeQIS0cKJU7Js896Ud/LZ/kOg4QL77YAwn5WJXKYZjsgrU/+YMGvsUuDw+s= tasks: - pexec: all: - sudo dnf remove nvme-cli -y - sudo dnf install nvmetcli nvme-cli -y - install: null - cephadm: conf: mgr: debug mgr: 20 debug ms: 1 - thrashosds: chance_pgnum_grow: 1 chance_pgnum_shrink: 1 chance_pgpnum_fix: 1 timeout: 1200 - full_sequential: - radosbench: clients: - client.0 time: 90 - radosbench: clients: - client.0 time: 90 - radosbench: clients: - client.0 time: 90 - radosbench: clients: - client.0 time: 90 - radosbench: clients: - client.0 time: 90 teuthology: fragments_dropped: [] meta: {} postmerge: [] teuthology_branch: main teuthology_sha1: 3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a timestamp: 2024-09-18_06:59:44 tube: smithi user: gabrioux verbose: true worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.3401856 2024-09-18T07:46:09.012 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/git.ceph.com_ceph-c_e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e/qa; will attempt to use it 2024-09-18T07:46:09.012 INFO:teuthology.run:Found tasks at /home/teuthworker/src/git.ceph.com_ceph-c_e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e/qa/tasks 2024-09-18T07:46:09.013 INFO:teuthology.run_tasks:Running task internal.check_packages... 2024-09-18T07:46:09.014 INFO:teuthology.task.internal:Checking packages... 2024-09-18T07:46:09.039 INFO:teuthology.task.internal:Checking packages for os_type 'centos', flavor 'default' and ceph hash 'e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e' 2024-09-18T07:46:09.039 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-09-18T07:46:09.039 INFO:teuthology.packaging:ref: None 2024-09-18T07:46:09.039 INFO:teuthology.packaging:tag: None 2024-09-18T07:46:09.039 INFO:teuthology.packaging:branch: wip-guits-main-2024-09-17-1213 2024-09-18T07:46:09.039 INFO:teuthology.packaging:sha1: e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e 2024-09-18T07:46:09.040 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&ref=wip-guits-main-2024-09-17-1213 2024-09-18T07:46:09.187 INFO:teuthology.task.internal:Found packages for ceph version 19.3.0-4983.ge7fb7b56 2024-09-18T07:46:09.189 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2024-09-18T07:46:09.197 INFO:teuthology.task.internal:no buildpackages task found 2024-09-18T07:46:09.197 INFO:teuthology.run_tasks:Running task internal.save_config... 2024-09-18T07:46:09.237 INFO:teuthology.task.internal:Saving configuration 2024-09-18T07:46:09.250 INFO:teuthology.run_tasks:Running task internal.check_lock... 2024-09-18T07:46:09.256 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-09-18T07:46:09.280 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi007.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/gabrioux-2024-09-18_06:59:44-orch:cephadm-wip-guits-main-2024-09-17-1213-distro-default-smithi/7910340', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9.stream', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-09-18 07:41:28.622881', 'locked_by': 'scheduled_gabrioux@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBPYkVuXWvxdNGFoUgrY3wEvIEMmW46TDgl+jyvRxWqmr12Unz0FidCtrdbIC+JvIs19BZXuMX0CLiiSlaGRjhIs='} 2024-09-18T07:46:09.305 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi050.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/gabrioux-2024-09-18_06:59:44-orch:cephadm-wip-guits-main-2024-09-17-1213-distro-default-smithi/7910340', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9.stream', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-09-18 07:41:28.624383', 'locked_by': 'scheduled_gabrioux@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBEO6Sl2zyfi/P/W9j03+TnhJETbGDeQIS0cKJU7Js896Ud/LZ/kOg4QL77YAwn5WJXKYZjsgrU/+YMGvsUuDw+s='} 2024-09-18T07:46:09.306 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2024-09-18T07:46:09.312 INFO:teuthology.task.internal:roles: ubuntu@smithi007.front.sepia.ceph.com - ['mon.a', 'mon.c', 'mgr.y', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'ceph.rgw.foo.a', 'node-exporter.a', 'alertmanager.a'] 2024-09-18T07:46:09.312 INFO:teuthology.task.internal:roles: ubuntu@smithi050.front.sepia.ceph.com - ['mon.b', 'mgr.x', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'prometheus.a', 'grafana.a', 'node-exporter.b', 'ceph.iscsi.iscsi.a'] 2024-09-18T07:46:09.312 INFO:teuthology.run_tasks:Running task console_log... 2024-09-18T07:46:09.372 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7f1efb683be0>, signals=[15]) 2024-09-18T07:46:09.373 INFO:teuthology.run_tasks:Running task internal.connect... 2024-09-18T07:46:09.380 INFO:teuthology.task.internal:Opening connections... 2024-09-18T07:46:09.380 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi007.front.sepia.ceph.com 2024-09-18T07:46:09.382 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi007.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-18T07:46:09.462 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi050.front.sepia.ceph.com 2024-09-18T07:46:09.463 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi050.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-18T07:46:09.538 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2024-09-18T07:46:09.545 DEBUG:teuthology.orchestra.run.smithi007:> uname -m 2024-09-18T07:46:09.562 INFO:teuthology.orchestra.run.smithi007.stdout:x86_64 2024-09-18T07:46:09.563 DEBUG:teuthology.orchestra.run.smithi007:> cat /etc/os-release 2024-09-18T07:46:09.620 INFO:teuthology.orchestra.run.smithi007.stdout:NAME="CentOS Stream" 2024-09-18T07:46:09.620 INFO:teuthology.orchestra.run.smithi007.stdout:VERSION="9" 2024-09-18T07:46:09.620 INFO:teuthology.orchestra.run.smithi007.stdout:ID="centos" 2024-09-18T07:46:09.620 INFO:teuthology.orchestra.run.smithi007.stdout:ID_LIKE="rhel fedora" 2024-09-18T07:46:09.620 INFO:teuthology.orchestra.run.smithi007.stdout:VERSION_ID="9" 2024-09-18T07:46:09.620 INFO:teuthology.orchestra.run.smithi007.stdout:PLATFORM_ID="platform:el9" 2024-09-18T07:46:09.620 INFO:teuthology.orchestra.run.smithi007.stdout:PRETTY_NAME="CentOS Stream 9" 2024-09-18T07:46:09.621 INFO:teuthology.orchestra.run.smithi007.stdout:ANSI_COLOR="0;31" 2024-09-18T07:46:09.621 INFO:teuthology.orchestra.run.smithi007.stdout:LOGO="fedora-logo-icon" 2024-09-18T07:46:09.621 INFO:teuthology.orchestra.run.smithi007.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-09-18T07:46:09.621 INFO:teuthology.orchestra.run.smithi007.stdout:HOME_URL="https://centos.org/" 2024-09-18T07:46:09.621 INFO:teuthology.orchestra.run.smithi007.stdout:BUG_REPORT_URL="https://issues.redhat.com/" 2024-09-18T07:46:09.621 INFO:teuthology.orchestra.run.smithi007.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-09-18T07:46:09.621 INFO:teuthology.orchestra.run.smithi007.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-09-18T07:46:09.622 INFO:teuthology.lock.ops:Updating smithi007.front.sepia.ceph.com on lock server 2024-09-18T07:46:09.647 DEBUG:teuthology.orchestra.run.smithi050:> uname -m 2024-09-18T07:46:09.665 INFO:teuthology.orchestra.run.smithi050.stdout:x86_64 2024-09-18T07:46:09.665 DEBUG:teuthology.orchestra.run.smithi050:> cat /etc/os-release 2024-09-18T07:46:09.723 INFO:teuthology.orchestra.run.smithi050.stdout:NAME="CentOS Stream" 2024-09-18T07:46:09.723 INFO:teuthology.orchestra.run.smithi050.stdout:VERSION="9" 2024-09-18T07:46:09.723 INFO:teuthology.orchestra.run.smithi050.stdout:ID="centos" 2024-09-18T07:46:09.723 INFO:teuthology.orchestra.run.smithi050.stdout:ID_LIKE="rhel fedora" 2024-09-18T07:46:09.723 INFO:teuthology.orchestra.run.smithi050.stdout:VERSION_ID="9" 2024-09-18T07:46:09.723 INFO:teuthology.orchestra.run.smithi050.stdout:PLATFORM_ID="platform:el9" 2024-09-18T07:46:09.723 INFO:teuthology.orchestra.run.smithi050.stdout:PRETTY_NAME="CentOS Stream 9" 2024-09-18T07:46:09.724 INFO:teuthology.orchestra.run.smithi050.stdout:ANSI_COLOR="0;31" 2024-09-18T07:46:09.724 INFO:teuthology.orchestra.run.smithi050.stdout:LOGO="fedora-logo-icon" 2024-09-18T07:46:09.724 INFO:teuthology.orchestra.run.smithi050.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-09-18T07:46:09.724 INFO:teuthology.orchestra.run.smithi050.stdout:HOME_URL="https://centos.org/" 2024-09-18T07:46:09.724 INFO:teuthology.orchestra.run.smithi050.stdout:BUG_REPORT_URL="https://issues.redhat.com/" 2024-09-18T07:46:09.724 INFO:teuthology.orchestra.run.smithi050.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-09-18T07:46:09.724 INFO:teuthology.orchestra.run.smithi050.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-09-18T07:46:09.724 INFO:teuthology.lock.ops:Updating smithi050.front.sepia.ceph.com on lock server 2024-09-18T07:46:09.746 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2024-09-18T07:46:09.755 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2024-09-18T07:46:09.765 INFO:teuthology.task.internal:Checking for old test directory... 2024-09-18T07:46:09.765 DEBUG:teuthology.orchestra.run.smithi007:> test '!' -e /home/ubuntu/cephtest 2024-09-18T07:46:09.768 DEBUG:teuthology.orchestra.run.smithi050:> test '!' -e /home/ubuntu/cephtest 2024-09-18T07:46:09.784 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2024-09-18T07:46:09.790 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2024-09-18T07:46:09.790 DEBUG:teuthology.orchestra.run.smithi007:> test -z $(ls -A /var/lib/ceph) 2024-09-18T07:46:09.825 DEBUG:teuthology.orchestra.run.smithi050:> test -z $(ls -A /var/lib/ceph) 2024-09-18T07:46:09.852 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2024-09-18T07:46:09.904 INFO:teuthology.run_tasks:Running task kernel... 2024-09-18T07:46:09.919 INFO:teuthology.task.kernel:normalize config orig: {'kdb': 1, 'sha1': 'distro'} 2024-09-18T07:46:09.919 INFO:teuthology.task.kernel:config {'mon.a': {'kdb': 1, 'sha1': 'distro'}, 'mon.b': {'kdb': 1, 'sha1': 'distro'}}, timeout 300 2024-09-18T07:46:09.920 DEBUG:teuthology.orchestra.run.smithi007:> test -f /run/.containerenv -o -f /.dockerenv 2024-09-18T07:46:09.920 DEBUG:teuthology.orchestra.run.smithi050:> test -f /run/.containerenv -o -f /.dockerenv 2024-09-18T07:46:09.936 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:46:09.937 DEBUG:teuthology.orchestra.run.smithi007:> uname -r 2024-09-18T07:46:09.941 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:46:09.941 DEBUG:teuthology.orchestra.run.smithi050:> uname -r 2024-09-18T07:46:09.995 INFO:teuthology.orchestra.run.smithi007.stdout:5.14.0-503.el9.x86_64 2024-09-18T07:46:09.996 INFO:teuthology.task.kernel:Running kernel on smithi007: 5.14.0-503.el9.x86_64 2024-09-18T07:46:09.996 DEBUG:teuthology.orchestra.run.smithi007:> sudo yum install -y kernel 2024-09-18T07:46:09.999 INFO:teuthology.orchestra.run.smithi050.stdout:5.14.0-503.el9.x86_64 2024-09-18T07:46:09.999 INFO:teuthology.task.kernel:Running kernel on smithi050: 5.14.0-503.el9.x86_64 2024-09-18T07:46:09.999 DEBUG:teuthology.orchestra.run.smithi050:> sudo yum install -y kernel 2024-09-18T07:46:12.476 INFO:teuthology.orchestra.run.smithi007.stdout:CentOS Stream 9 - BaseOS 13 MB/s | 8.3 MB 00:00 2024-09-18T07:46:13.444 INFO:teuthology.orchestra.run.smithi050.stdout:CentOS Stream 9 - BaseOS 5.0 MB/s | 8.3 MB 00:01 2024-09-18T07:46:18.054 INFO:teuthology.orchestra.run.smithi050.stdout:CentOS Stream 9 - AppStream 5.9 MB/s | 20 MB 00:03 2024-09-18T07:46:23.013 INFO:teuthology.orchestra.run.smithi050.stdout:CentOS Stream 9 - CRB 7.0 MB/s | 6.5 MB 00:00 2024-09-18T07:46:24.605 INFO:teuthology.orchestra.run.smithi050.stdout:CentOS Stream 9 - Extras packages 57 kB/s | 19 kB 00:00 2024-09-18T07:46:25.893 INFO:teuthology.orchestra.run.smithi050.stdout:Extra Packages for Enterprise Linux 19 MB/s | 23 MB 00:01 2024-09-18T07:46:31.399 INFO:teuthology.orchestra.run.smithi050.stdout:lab-extras 29 kB/s | 1.7 kB 00:00 2024-09-18T07:46:33.332 INFO:teuthology.orchestra.run.smithi007.stdout:CentOS Stream 9 - AppStream 1.0 MB/s | 20 MB 00:19 2024-09-18T07:46:33.430 INFO:teuthology.orchestra.run.smithi050.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2024-09-18T07:46:33.431 INFO:teuthology.orchestra.run.smithi050.stdout:Package kernel-5.14.0-503.el9.x86_64 is already installed. 2024-09-18T07:46:33.531 INFO:teuthology.orchestra.run.smithi050.stdout:Dependencies resolved. 2024-09-18T07:46:33.549 INFO:teuthology.orchestra.run.smithi050.stdout:================================================================================ 2024-09-18T07:46:33.549 INFO:teuthology.orchestra.run.smithi050.stdout: Package Architecture Version Repository Size 2024-09-18T07:46:33.549 INFO:teuthology.orchestra.run.smithi050.stdout:================================================================================ 2024-09-18T07:46:33.549 INFO:teuthology.orchestra.run.smithi050.stdout:Installing: 2024-09-18T07:46:33.549 INFO:teuthology.orchestra.run.smithi050.stdout: kernel x86_64 5.14.0-508.el9 baseos 36 k 2024-09-18T07:46:33.549 INFO:teuthology.orchestra.run.smithi050.stdout:Installing dependencies: 2024-09-18T07:46:33.549 INFO:teuthology.orchestra.run.smithi050.stdout: kernel-core x86_64 5.14.0-508.el9 baseos 16 M 2024-09-18T07:46:33.549 INFO:teuthology.orchestra.run.smithi050.stdout: kernel-modules x86_64 5.14.0-508.el9 baseos 35 M 2024-09-18T07:46:33.549 INFO:teuthology.orchestra.run.smithi050.stdout: kernel-modules-core x86_64 5.14.0-508.el9 baseos 29 M 2024-09-18T07:46:33.549 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-09-18T07:46:33.550 INFO:teuthology.orchestra.run.smithi050.stdout:Transaction Summary 2024-09-18T07:46:33.550 INFO:teuthology.orchestra.run.smithi050.stdout:================================================================================ 2024-09-18T07:46:33.550 INFO:teuthology.orchestra.run.smithi050.stdout:Install 4 Packages 2024-09-18T07:46:33.550 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-09-18T07:46:33.550 INFO:teuthology.orchestra.run.smithi050.stdout:Total download size: 79 M 2024-09-18T07:46:33.550 INFO:teuthology.orchestra.run.smithi050.stdout:Installed size: 126 M 2024-09-18T07:46:33.551 INFO:teuthology.orchestra.run.smithi050.stdout:Downloading Packages: 2024-09-18T07:46:33.784 INFO:teuthology.orchestra.run.smithi050.stdout:(1/4): kernel-5.14.0-508.el9.x86_64.rpm 316 kB/s | 36 kB 00:00 2024-09-18T07:46:34.800 INFO:teuthology.orchestra.run.smithi050.stdout:(2/4): kernel-core-5.14.0-508.el9.x86_64.rpm 14 MB/s | 16 MB 00:01 2024-09-18T07:46:35.576 INFO:teuthology.orchestra.run.smithi050.stdout:(3/4): kernel-modules-5.14.0-508.el9.x86_64.rpm 18 MB/s | 35 MB 00:01 2024-09-18T07:46:36.251 INFO:teuthology.orchestra.run.smithi050.stdout:(4/4): kernel-modules-core-5.14.0-508.el9.x86_6 12 MB/s | 29 MB 00:02 2024-09-18T07:46:36.251 INFO:teuthology.orchestra.run.smithi050.stdout:-------------------------------------------------------------------------------- 2024-09-18T07:46:36.251 INFO:teuthology.orchestra.run.smithi050.stdout:Total 29 MB/s | 79 MB 00:02 2024-09-18T07:46:36.965 INFO:teuthology.orchestra.run.smithi050.stdout:Running transaction check 2024-09-18T07:46:37.345 INFO:teuthology.orchestra.run.smithi050.stdout:Transaction check succeeded. 2024-09-18T07:46:37.345 INFO:teuthology.orchestra.run.smithi050.stdout:Running transaction test 2024-09-18T07:46:37.858 INFO:teuthology.orchestra.run.smithi007.stdout:CentOS Stream 9 - CRB 11 MB/s | 6.5 MB 00:00 2024-09-18T07:46:38.351 INFO:teuthology.orchestra.run.smithi050.stdout:Transaction test succeeded. 2024-09-18T07:46:38.352 INFO:teuthology.orchestra.run.smithi050.stdout:Running transaction 2024-09-18T07:46:39.325 INFO:teuthology.orchestra.run.smithi007.stdout:CentOS Stream 9 - Extras packages 91 kB/s | 19 kB 00:00 2024-09-18T07:46:40.904 INFO:teuthology.orchestra.run.smithi007.stdout:Extra Packages for Enterprise Linux 16 MB/s | 23 MB 00:01 2024-09-18T07:46:40.941 INFO:teuthology.orchestra.run.smithi050.stdout: Preparing : 1/1 2024-09-18T07:46:41.625 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : kernel-modules-core-5.14.0-508.el9.x86_64 1/4 2024-09-18T07:46:41.713 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : kernel-core-5.14.0-508.el9.x86_64 2/4 2024-09-18T07:46:44.468 INFO:teuthology.orchestra.run.smithi050.stdout: Running scriptlet: kernel-core-5.14.0-508.el9.x86_64 2/4 2024-09-18T07:46:44.591 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : kernel-modules-5.14.0-508.el9.x86_64 3/4 2024-09-18T07:46:46.585 INFO:teuthology.orchestra.run.smithi007.stdout:lab-extras 32 kB/s | 1.7 kB 00:00 2024-09-18T07:46:48.558 INFO:teuthology.orchestra.run.smithi007.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2024-09-18T07:46:48.559 INFO:teuthology.orchestra.run.smithi007.stdout:Package kernel-5.14.0-503.el9.x86_64 is already installed. 2024-09-18T07:46:48.648 INFO:teuthology.orchestra.run.smithi007.stdout:Dependencies resolved. 2024-09-18T07:46:48.654 INFO:teuthology.orchestra.run.smithi007.stdout:================================================================================ 2024-09-18T07:46:48.654 INFO:teuthology.orchestra.run.smithi007.stdout: Package Architecture Version Repository Size 2024-09-18T07:46:48.654 INFO:teuthology.orchestra.run.smithi007.stdout:================================================================================ 2024-09-18T07:46:48.654 INFO:teuthology.orchestra.run.smithi007.stdout:Installing: 2024-09-18T07:46:48.655 INFO:teuthology.orchestra.run.smithi007.stdout: kernel x86_64 5.14.0-508.el9 baseos 36 k 2024-09-18T07:46:48.655 INFO:teuthology.orchestra.run.smithi007.stdout:Installing dependencies: 2024-09-18T07:46:48.655 INFO:teuthology.orchestra.run.smithi007.stdout: kernel-core x86_64 5.14.0-508.el9 baseos 16 M 2024-09-18T07:46:48.655 INFO:teuthology.orchestra.run.smithi007.stdout: kernel-modules x86_64 5.14.0-508.el9 baseos 35 M 2024-09-18T07:46:48.655 INFO:teuthology.orchestra.run.smithi007.stdout: kernel-modules-core x86_64 5.14.0-508.el9 baseos 29 M 2024-09-18T07:46:48.655 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-09-18T07:46:48.655 INFO:teuthology.orchestra.run.smithi007.stdout:Transaction Summary 2024-09-18T07:46:48.655 INFO:teuthology.orchestra.run.smithi007.stdout:================================================================================ 2024-09-18T07:46:48.655 INFO:teuthology.orchestra.run.smithi007.stdout:Install 4 Packages 2024-09-18T07:46:48.655 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-09-18T07:46:48.655 INFO:teuthology.orchestra.run.smithi007.stdout:Total download size: 79 M 2024-09-18T07:46:48.655 INFO:teuthology.orchestra.run.smithi007.stdout:Installed size: 126 M 2024-09-18T07:46:48.655 INFO:teuthology.orchestra.run.smithi007.stdout:Downloading Packages: 2024-09-18T07:46:49.236 INFO:teuthology.orchestra.run.smithi007.stdout:(1/4): kernel-5.14.0-508.el9.x86_64.rpm 64 kB/s | 36 kB 00:00 2024-09-18T07:46:49.718 INFO:teuthology.orchestra.run.smithi050.stdout: Running scriptlet: kernel-modules-5.14.0-508.el9.x86_64 3/4 2024-09-18T07:46:49.759 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : kernel-5.14.0-508.el9.x86_64 4/4 2024-09-18T07:46:50.072 INFO:teuthology.orchestra.run.smithi007.stdout:(2/4): kernel-core-5.14.0-508.el9.x86_64.rpm 11 MB/s | 16 MB 00:01 2024-09-18T07:46:50.889 INFO:teuthology.orchestra.run.smithi007.stdout:(3/4): kernel-modules-5.14.0-508.el9.x86_64.rpm 16 MB/s | 35 MB 00:02 2024-09-18T07:46:51.714 INFO:teuthology.orchestra.run.smithi007.stdout:(4/4): kernel-modules-core-5.14.0-508.el9.x86_6 12 MB/s | 29 MB 00:02 2024-09-18T07:46:51.714 INFO:teuthology.orchestra.run.smithi007.stdout:-------------------------------------------------------------------------------- 2024-09-18T07:46:51.715 INFO:teuthology.orchestra.run.smithi007.stdout:Total 26 MB/s | 79 MB 00:03 2024-09-18T07:46:52.430 INFO:teuthology.orchestra.run.smithi007.stdout:Running transaction check 2024-09-18T07:46:52.805 INFO:teuthology.orchestra.run.smithi007.stdout:Transaction check succeeded. 2024-09-18T07:46:52.805 INFO:teuthology.orchestra.run.smithi007.stdout:Running transaction test 2024-09-18T07:46:53.803 INFO:teuthology.orchestra.run.smithi007.stdout:Transaction test succeeded. 2024-09-18T07:46:53.803 INFO:teuthology.orchestra.run.smithi007.stdout:Running transaction 2024-09-18T07:46:54.899 INFO:teuthology.orchestra.run.smithi050.stdout: Running scriptlet: kernel-modules-core-5.14.0-508.el9.x86_64 4/4 2024-09-18T07:46:56.363 INFO:teuthology.orchestra.run.smithi007.stdout: Preparing : 1/1 2024-09-18T07:46:57.236 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : kernel-modules-core-5.14.0-508.el9.x86_64 1/4 2024-09-18T07:46:57.339 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : kernel-core-5.14.0-508.el9.x86_64 2/4 2024-09-18T07:47:00.105 INFO:teuthology.orchestra.run.smithi007.stdout: Running scriptlet: kernel-core-5.14.0-508.el9.x86_64 2/4 2024-09-18T07:47:00.240 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : kernel-modules-5.14.0-508.el9.x86_64 3/4 2024-09-18T07:47:05.374 INFO:teuthology.orchestra.run.smithi007.stdout: Running scriptlet: kernel-modules-5.14.0-508.el9.x86_64 3/4 2024-09-18T07:47:05.416 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : kernel-5.14.0-508.el9.x86_64 4/4 2024-09-18T07:47:10.553 INFO:teuthology.orchestra.run.smithi007.stdout: Running scriptlet: kernel-modules-core-5.14.0-508.el9.x86_64 4/4 2024-09-18T07:47:33.661 INFO:teuthology.orchestra.run.smithi050.stdout: Running scriptlet: kernel-core-5.14.0-508.el9.x86_64 4/4 2024-09-18T07:47:33.703 INFO:teuthology.orchestra.run.smithi050.stdout: Running scriptlet: kernel-modules-5.14.0-508.el9.x86_64 4/4 2024-09-18T07:47:34.032 INFO:teuthology.orchestra.run.smithi050.stdout: Running scriptlet: kernel-5.14.0-508.el9.x86_64 4/4 2024-09-18T07:47:34.032 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : kernel-5.14.0-508.el9.x86_64 1/4 2024-09-18T07:47:34.032 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : kernel-core-5.14.0-508.el9.x86_64 2/4 2024-09-18T07:47:34.032 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : kernel-modules-5.14.0-508.el9.x86_64 3/4 2024-09-18T07:47:34.889 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : kernel-modules-core-5.14.0-508.el9.x86_64 4/4 2024-09-18T07:47:34.889 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-09-18T07:47:34.889 INFO:teuthology.orchestra.run.smithi050.stdout:Installed: 2024-09-18T07:47:34.889 INFO:teuthology.orchestra.run.smithi050.stdout: kernel-5.14.0-508.el9.x86_64 2024-09-18T07:47:34.889 INFO:teuthology.orchestra.run.smithi050.stdout: kernel-core-5.14.0-508.el9.x86_64 2024-09-18T07:47:34.889 INFO:teuthology.orchestra.run.smithi050.stdout: kernel-modules-5.14.0-508.el9.x86_64 2024-09-18T07:47:34.889 INFO:teuthology.orchestra.run.smithi050.stdout: kernel-modules-core-5.14.0-508.el9.x86_64 2024-09-18T07:47:34.890 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-09-18T07:47:34.890 INFO:teuthology.orchestra.run.smithi050.stdout:Complete! 2024-09-18T07:47:35.215 DEBUG:teuthology.orchestra.run.smithi050:> rpm -q kernel | sort -rV | head -n 1 2024-09-18T07:47:35.294 INFO:teuthology.orchestra.run.smithi050.stdout:kernel-5.14.0-508.el9.x86_64 2024-09-18T07:47:35.294 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-508.el9.x86_64 2024-09-18T07:47:35.294 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.14.0-503.el9.x86_64 Expected: 5.14.0-508.el9.x86_64 2024-09-18T07:47:35.295 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-09-18T07:47:35.295 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-09-18T07:47:35.295 INFO:teuthology.task.kernel:Installing distro kernel on mon.b... 2024-09-18T07:47:35.295 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi050.front.sepia.ceph.com, path=None, version=distro) 2024-09-18T07:47:35.295 DEBUG:teuthology.orchestra.run.smithi050:> rpm -q kernel | sort -rV | head -n 1 2024-09-18T07:47:35.353 INFO:teuthology.orchestra.run.smithi050.stdout:kernel-5.14.0-508.el9.x86_64 2024-09-18T07:47:35.354 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-508.el9.x86_64 2024-09-18T07:47:35.354 DEBUG:teuthology.orchestra.run.smithi050:> sudo rpm -qi grub2-tools 2024-09-18T07:47:35.394 INFO:teuthology.orchestra.run.smithi050.stdout:Name : grub2-tools 2024-09-18T07:47:35.394 INFO:teuthology.orchestra.run.smithi050.stdout:Epoch : 1 2024-09-18T07:47:35.394 INFO:teuthology.orchestra.run.smithi050.stdout:Version : 2.06 2024-09-18T07:47:35.394 INFO:teuthology.orchestra.run.smithi050.stdout:Release : 82.el9 2024-09-18T07:47:35.394 INFO:teuthology.orchestra.run.smithi050.stdout:Architecture: x86_64 2024-09-18T07:47:35.394 INFO:teuthology.orchestra.run.smithi050.stdout:Install Date: Wed 28 Aug 2024 09:06:27 PM UTC 2024-09-18T07:47:35.395 INFO:teuthology.orchestra.run.smithi050.stdout:Group : Unspecified 2024-09-18T07:47:35.395 INFO:teuthology.orchestra.run.smithi050.stdout:Size : 8274670 2024-09-18T07:47:35.395 INFO:teuthology.orchestra.run.smithi050.stdout:License : GPLv3+ 2024-09-18T07:47:35.395 INFO:teuthology.orchestra.run.smithi050.stdout:Signature : RSA/SHA256, Mon 01 Jul 2024 07:46:42 AM UTC, Key ID 05b555b38483c65d 2024-09-18T07:47:35.395 INFO:teuthology.orchestra.run.smithi050.stdout:Source RPM : grub2-2.06-82.el9.src.rpm 2024-09-18T07:47:35.395 INFO:teuthology.orchestra.run.smithi050.stdout:Build Date : Fri 28 Jun 2024 09:24:44 AM UTC 2024-09-18T07:47:35.395 INFO:teuthology.orchestra.run.smithi050.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2024-09-18T07:47:35.395 INFO:teuthology.orchestra.run.smithi050.stdout:Packager : builder@centos.org 2024-09-18T07:47:35.395 INFO:teuthology.orchestra.run.smithi050.stdout:Vendor : CentOS 2024-09-18T07:47:35.395 INFO:teuthology.orchestra.run.smithi050.stdout:URL : http://www.gnu.org/software/grub/ 2024-09-18T07:47:35.395 INFO:teuthology.orchestra.run.smithi050.stdout:Summary : Support tools for GRUB. 2024-09-18T07:47:35.395 INFO:teuthology.orchestra.run.smithi050.stdout:Description : 2024-09-18T07:47:35.395 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-09-18T07:47:35.395 INFO:teuthology.orchestra.run.smithi050.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-09-18T07:47:35.396 INFO:teuthology.orchestra.run.smithi050.stdout:customizable bootloader with modular architecture. It supports a rich 2024-09-18T07:47:35.396 INFO:teuthology.orchestra.run.smithi050.stdout:variety of kernel formats, file systems, computer architectures and 2024-09-18T07:47:35.396 INFO:teuthology.orchestra.run.smithi050.stdout:hardware devices. 2024-09-18T07:47:35.396 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-09-18T07:47:35.396 INFO:teuthology.orchestra.run.smithi050.stdout:This subpackage provides tools for support of all platforms. 2024-09-18T07:47:35.397 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-09-18T07:47:35.397 INFO:teuthology.task.kernel:Updating grub on smithi050 to boot 5.14.0-508.el9.x86_64 2024-09-18T07:47:35.397 DEBUG:teuthology.orchestra.run.smithi050:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-09-18T07:47:35.966 INFO:teuthology.orchestra.run.smithi050.stderr:Generating grub configuration file ... 2024-09-18T07:47:37.327 INFO:teuthology.orchestra.run.smithi050.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2024-09-18T07:47:37.365 INFO:teuthology.orchestra.run.smithi050.stderr:done 2024-09-18T07:47:37.367 DEBUG:teuthology.orchestra.run.smithi050:> mktemp 2024-09-18T07:47:37.382 INFO:teuthology.orchestra.run.smithi050.stdout:/tmp/tmp.pjofFEhae2 2024-09-18T07:47:37.383 DEBUG:teuthology.orchestra.run.smithi050:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.pjofFEhae2 2024-09-18T07:47:37.448 DEBUG:teuthology.orchestra.run.smithi050:> sudo chmod 0666 /tmp/tmp.pjofFEhae2 2024-09-18T07:47:37.589 DEBUG:teuthology.orchestra.remote:smithi050:/tmp/tmp.pjofFEhae2 is 6KB 2024-09-18T07:47:37.609 DEBUG:teuthology.orchestra.run.smithi050:> rm -fr /tmp/tmp.pjofFEhae2 2024-09-18T07:47:37.624 DEBUG:teuthology.orchestra.run.smithi050:> sudo /bin/ls /boot/loader/entries || true 2024-09-18T07:47:37.686 INFO:teuthology.orchestra.run.smithi050.stdout:fbf6b6fa6cf04654a0e563f30ecbf43a-0-rescue.conf 2024-09-18T07:47:37.686 INFO:teuthology.orchestra.run.smithi050.stdout:fbf6b6fa6cf04654a0e563f30ecbf43a-5.14.0-239.el9.x86_64.conf 2024-09-18T07:47:37.686 INFO:teuthology.orchestra.run.smithi050.stdout:fbf6b6fa6cf04654a0e563f30ecbf43a-5.14.0-503.el9.x86_64.conf 2024-09-18T07:47:37.686 INFO:teuthology.orchestra.run.smithi050.stdout:fbf6b6fa6cf04654a0e563f30ecbf43a-5.14.0-508.el9.x86_64.conf 2024-09-18T07:47:37.687 DEBUG:teuthology.orchestra.run.smithi050:> sudo grub2-set-default fbf6b6fa6cf04654a0e563f30ecbf43a-5.14.0-508.el9.x86_64 2024-09-18T07:47:37.838 DEBUG:teuthology.orchestra.run.smithi050:> sudo shutdown -r now 2024-09-18T07:47:49.404 INFO:teuthology.orchestra.run.smithi007.stdout: Running scriptlet: kernel-core-5.14.0-508.el9.x86_64 4/4 2024-09-18T07:47:49.446 INFO:teuthology.orchestra.run.smithi007.stdout: Running scriptlet: kernel-modules-5.14.0-508.el9.x86_64 4/4 2024-09-18T07:47:49.843 INFO:teuthology.orchestra.run.smithi007.stdout: Running scriptlet: kernel-5.14.0-508.el9.x86_64 4/4 2024-09-18T07:47:49.843 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : kernel-5.14.0-508.el9.x86_64 1/4 2024-09-18T07:47:49.843 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : kernel-core-5.14.0-508.el9.x86_64 2/4 2024-09-18T07:47:49.843 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : kernel-modules-5.14.0-508.el9.x86_64 3/4 2024-09-18T07:47:50.684 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : kernel-modules-core-5.14.0-508.el9.x86_64 4/4 2024-09-18T07:47:50.684 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-09-18T07:47:50.684 INFO:teuthology.orchestra.run.smithi007.stdout:Installed: 2024-09-18T07:47:50.684 INFO:teuthology.orchestra.run.smithi007.stdout: kernel-5.14.0-508.el9.x86_64 2024-09-18T07:47:50.684 INFO:teuthology.orchestra.run.smithi007.stdout: kernel-core-5.14.0-508.el9.x86_64 2024-09-18T07:47:50.684 INFO:teuthology.orchestra.run.smithi007.stdout: kernel-modules-5.14.0-508.el9.x86_64 2024-09-18T07:47:50.684 INFO:teuthology.orchestra.run.smithi007.stdout: kernel-modules-core-5.14.0-508.el9.x86_64 2024-09-18T07:47:50.684 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-09-18T07:47:50.684 INFO:teuthology.orchestra.run.smithi007.stdout:Complete! 2024-09-18T07:47:51.035 DEBUG:teuthology.orchestra.run.smithi007:> rpm -q kernel | sort -rV | head -n 1 2024-09-18T07:47:51.127 INFO:teuthology.orchestra.run.smithi007.stdout:kernel-5.14.0-508.el9.x86_64 2024-09-18T07:47:51.127 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-508.el9.x86_64 2024-09-18T07:47:51.127 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.14.0-503.el9.x86_64 Expected: 5.14.0-508.el9.x86_64 2024-09-18T07:47:51.127 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-09-18T07:47:51.127 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-09-18T07:47:51.128 INFO:teuthology.task.kernel:Installing distro kernel on mon.a... 2024-09-18T07:47:51.128 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi007.front.sepia.ceph.com, path=None, version=distro) 2024-09-18T07:47:51.128 DEBUG:teuthology.orchestra.run.smithi007:> rpm -q kernel | sort -rV | head -n 1 2024-09-18T07:47:51.186 INFO:teuthology.orchestra.run.smithi007.stdout:kernel-5.14.0-508.el9.x86_64 2024-09-18T07:47:51.186 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-508.el9.x86_64 2024-09-18T07:47:51.187 DEBUG:teuthology.orchestra.run.smithi007:> sudo rpm -qi grub2-tools 2024-09-18T07:47:51.223 INFO:teuthology.orchestra.run.smithi007.stdout:Name : grub2-tools 2024-09-18T07:47:51.223 INFO:teuthology.orchestra.run.smithi007.stdout:Epoch : 1 2024-09-18T07:47:51.223 INFO:teuthology.orchestra.run.smithi007.stdout:Version : 2.06 2024-09-18T07:47:51.223 INFO:teuthology.orchestra.run.smithi007.stdout:Release : 82.el9 2024-09-18T07:47:51.223 INFO:teuthology.orchestra.run.smithi007.stdout:Architecture: x86_64 2024-09-18T07:47:51.224 INFO:teuthology.orchestra.run.smithi007.stdout:Install Date: Wed 28 Aug 2024 09:06:27 PM UTC 2024-09-18T07:47:51.224 INFO:teuthology.orchestra.run.smithi007.stdout:Group : Unspecified 2024-09-18T07:47:51.224 INFO:teuthology.orchestra.run.smithi007.stdout:Size : 8274670 2024-09-18T07:47:51.224 INFO:teuthology.orchestra.run.smithi007.stdout:License : GPLv3+ 2024-09-18T07:47:51.224 INFO:teuthology.orchestra.run.smithi007.stdout:Signature : RSA/SHA256, Mon 01 Jul 2024 07:46:42 AM UTC, Key ID 05b555b38483c65d 2024-09-18T07:47:51.224 INFO:teuthology.orchestra.run.smithi007.stdout:Source RPM : grub2-2.06-82.el9.src.rpm 2024-09-18T07:47:51.224 INFO:teuthology.orchestra.run.smithi007.stdout:Build Date : Fri 28 Jun 2024 09:24:44 AM UTC 2024-09-18T07:47:51.224 INFO:teuthology.orchestra.run.smithi007.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2024-09-18T07:47:51.224 INFO:teuthology.orchestra.run.smithi007.stdout:Packager : builder@centos.org 2024-09-18T07:47:51.224 INFO:teuthology.orchestra.run.smithi007.stdout:Vendor : CentOS 2024-09-18T07:47:51.224 INFO:teuthology.orchestra.run.smithi007.stdout:URL : http://www.gnu.org/software/grub/ 2024-09-18T07:47:51.224 INFO:teuthology.orchestra.run.smithi007.stdout:Summary : Support tools for GRUB. 2024-09-18T07:47:51.225 INFO:teuthology.orchestra.run.smithi007.stdout:Description : 2024-09-18T07:47:51.225 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-09-18T07:47:51.225 INFO:teuthology.orchestra.run.smithi007.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-09-18T07:47:51.225 INFO:teuthology.orchestra.run.smithi007.stdout:customizable bootloader with modular architecture. It supports a rich 2024-09-18T07:47:51.225 INFO:teuthology.orchestra.run.smithi007.stdout:variety of kernel formats, file systems, computer architectures and 2024-09-18T07:47:51.225 INFO:teuthology.orchestra.run.smithi007.stdout:hardware devices. 2024-09-18T07:47:51.225 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-09-18T07:47:51.225 INFO:teuthology.orchestra.run.smithi007.stdout:This subpackage provides tools for support of all platforms. 2024-09-18T07:47:51.226 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-09-18T07:47:51.227 INFO:teuthology.task.kernel:Updating grub on smithi007 to boot 5.14.0-508.el9.x86_64 2024-09-18T07:47:51.227 DEBUG:teuthology.orchestra.run.smithi007:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-09-18T07:47:51.920 INFO:teuthology.orchestra.run.smithi007.stderr:Generating grub configuration file ... 2024-09-18T07:47:53.231 INFO:teuthology.orchestra.run.smithi007.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2024-09-18T07:47:53.249 INFO:teuthology.orchestra.run.smithi007.stderr:done 2024-09-18T07:47:53.251 DEBUG:teuthology.orchestra.run.smithi007:> mktemp 2024-09-18T07:47:53.267 INFO:teuthology.orchestra.run.smithi007.stdout:/tmp/tmp.eMla3g3eIj 2024-09-18T07:47:53.267 DEBUG:teuthology.orchestra.run.smithi007:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.eMla3g3eIj 2024-09-18T07:47:53.333 DEBUG:teuthology.orchestra.run.smithi007:> sudo chmod 0666 /tmp/tmp.eMla3g3eIj 2024-09-18T07:47:53.469 DEBUG:teuthology.orchestra.remote:smithi007:/tmp/tmp.eMla3g3eIj is 6KB 2024-09-18T07:47:53.517 DEBUG:teuthology.orchestra.run.smithi007:> rm -fr /tmp/tmp.eMla3g3eIj 2024-09-18T07:47:53.532 DEBUG:teuthology.orchestra.run.smithi007:> sudo /bin/ls /boot/loader/entries || true 2024-09-18T07:47:53.594 INFO:teuthology.orchestra.run.smithi007.stdout:fbf6b6fa6cf04654a0e563f30ecbf43a-0-rescue.conf 2024-09-18T07:47:53.594 INFO:teuthology.orchestra.run.smithi007.stdout:fbf6b6fa6cf04654a0e563f30ecbf43a-5.14.0-239.el9.x86_64.conf 2024-09-18T07:47:53.594 INFO:teuthology.orchestra.run.smithi007.stdout:fbf6b6fa6cf04654a0e563f30ecbf43a-5.14.0-503.el9.x86_64.conf 2024-09-18T07:47:53.594 INFO:teuthology.orchestra.run.smithi007.stdout:fbf6b6fa6cf04654a0e563f30ecbf43a-5.14.0-508.el9.x86_64.conf 2024-09-18T07:47:53.596 DEBUG:teuthology.orchestra.run.smithi007:> sudo grub2-set-default fbf6b6fa6cf04654a0e563f30ecbf43a-5.14.0-508.el9.x86_64 2024-09-18T07:47:53.756 DEBUG:teuthology.orchestra.run.smithi007:> sudo shutdown -r now 2024-09-18T07:48:07.853 INFO:teuthology.task.kernel:Checking client mon.b for new kernel version... 2024-09-18T07:48:07.854 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi050.front.sepia.ceph.com' 2024-09-18T07:48:07.855 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi050.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-18T07:48:23.773 INFO:teuthology.task.kernel:Checking client mon.a for new kernel version... 2024-09-18T07:48:23.774 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi007.front.sepia.ceph.com' 2024-09-18T07:48:23.775 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi007.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-18T07:49:07.901 DEBUG:teuthology.orchestra.remote:timed out 2024-09-18T07:49:16.910 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi050.front.sepia.ceph.com' 2024-09-18T07:49:16.912 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi050.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-18T07:49:17.337 DEBUG:teuthology.orchestra.run.smithi050:> true 2024-09-18T07:49:17.723 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi050.front.sepia.ceph.com' 2024-09-18T07:49:17.723 INFO:teuthology.task.kernel:Checking kernel version of mon.b, want "5.14.0-508.el9.x86_64"... 2024-09-18T07:49:17.723 DEBUG:teuthology.orchestra.run.smithi050:> uname -r 2024-09-18T07:49:17.738 INFO:teuthology.orchestra.run.smithi050.stdout:5.14.0-508.el9.x86_64 2024-09-18T07:49:17.738 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-508.el9.x86_64 vs 5.14.0-508.el9.x86_64 2024-09-18T07:49:17.738 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-09-18T07:49:17.738 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-09-18T07:49:18.739 INFO:teuthology.task.kernel:Enabling kdb on mon.b... 2024-09-18T07:49:18.739 DEBUG:teuthology.orchestra.run.smithi050:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-09-18T07:49:18.859 INFO:teuthology.orchestra.run.smithi050.stdout:ttyS1 2024-09-18T07:49:18.890 DEBUG:teuthology.parallel:result is None 2024-09-18T07:49:23.777 DEBUG:teuthology.orchestra.remote:timed out 2024-09-18T07:49:32.780 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi007.front.sepia.ceph.com' 2024-09-18T07:49:32.780 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi007.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-18T07:49:33.168 DEBUG:teuthology.orchestra.run.smithi007:> true 2024-09-18T07:49:33.564 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi007.front.sepia.ceph.com' 2024-09-18T07:49:33.564 INFO:teuthology.task.kernel:Checking kernel version of mon.a, want "5.14.0-508.el9.x86_64"... 2024-09-18T07:49:33.564 DEBUG:teuthology.orchestra.run.smithi007:> uname -r 2024-09-18T07:49:33.580 INFO:teuthology.orchestra.run.smithi007.stdout:5.14.0-508.el9.x86_64 2024-09-18T07:49:33.580 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-508.el9.x86_64 vs 5.14.0-508.el9.x86_64 2024-09-18T07:49:33.580 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-09-18T07:49:33.580 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-09-18T07:49:34.581 INFO:teuthology.task.kernel:Enabling kdb on mon.a... 2024-09-18T07:49:34.582 DEBUG:teuthology.orchestra.run.smithi007:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-09-18T07:49:34.673 INFO:teuthology.orchestra.run.smithi007.stdout:ttyS1 2024-09-18T07:49:34.704 DEBUG:teuthology.parallel:result is None 2024-09-18T07:49:34.704 INFO:teuthology.run_tasks:Running task internal.base... 2024-09-18T07:49:34.712 INFO:teuthology.task.internal:Creating test directory... 2024-09-18T07:49:34.712 DEBUG:teuthology.orchestra.run.smithi007:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-09-18T07:49:34.715 DEBUG:teuthology.orchestra.run.smithi050:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-09-18T07:49:34.735 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2024-09-18T07:49:34.741 INFO:teuthology.run_tasks:Running task internal.archive... 2024-09-18T07:49:34.749 INFO:teuthology.task.internal:Creating archive directory... 2024-09-18T07:49:34.749 DEBUG:teuthology.orchestra.run.smithi007:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-09-18T07:49:34.770 DEBUG:teuthology.orchestra.run.smithi050:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-09-18T07:49:34.804 INFO:teuthology.run_tasks:Running task internal.coredump... 2024-09-18T07:49:34.810 INFO:teuthology.task.internal:Enabling coredump saving... 2024-09-18T07:49:34.810 DEBUG:teuthology.orchestra.run.smithi007:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-09-18T07:49:34.835 DEBUG:teuthology.orchestra.run.smithi050:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-09-18T07:49:34.858 INFO:teuthology.orchestra.run.smithi007.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-18T07:49:34.869 INFO:teuthology.orchestra.run.smithi007.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-18T07:49:34.876 INFO:teuthology.orchestra.run.smithi050.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-18T07:49:34.888 INFO:teuthology.orchestra.run.smithi050.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-18T07:49:34.890 INFO:teuthology.run_tasks:Running task internal.sudo... 2024-09-18T07:49:34.897 INFO:teuthology.task.internal:Configuring sudo... 2024-09-18T07:49:34.897 DEBUG:teuthology.orchestra.run.smithi007:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-09-18T07:49:34.913 DEBUG:teuthology.orchestra.run.smithi050:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-09-18T07:49:34.959 INFO:teuthology.run_tasks:Running task internal.syslog... 2024-09-18T07:49:34.968 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2024-09-18T07:49:34.968 DEBUG:teuthology.orchestra.run.smithi007:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-09-18T07:49:34.978 DEBUG:teuthology.orchestra.run.smithi050:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-09-18T07:49:35.017 DEBUG:teuthology.orchestra.run.smithi007:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-09-18T07:49:35.075 DEBUG:teuthology.orchestra.run.smithi007:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-09-18T07:49:35.123 DEBUG:teuthology.orchestra.run.smithi007:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-09-18T07:49:35.206 DEBUG:teuthology.orchestra.run.smithi007:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-09-18T07:49:35.245 DEBUG:teuthology.orchestra.run.smithi007:> set -ex 2024-09-18T07:49:35.246 DEBUG:teuthology.orchestra.run.smithi007:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-09-18T07:49:35.311 DEBUG:teuthology.orchestra.run.smithi050:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-09-18T07:49:35.357 DEBUG:teuthology.orchestra.run.smithi050:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-09-18T07:49:35.414 DEBUG:teuthology.orchestra.run.smithi050:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-09-18T07:49:35.501 DEBUG:teuthology.orchestra.run.smithi050:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-09-18T07:49:35.547 DEBUG:teuthology.orchestra.run.smithi050:> set -ex 2024-09-18T07:49:35.547 DEBUG:teuthology.orchestra.run.smithi050:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-09-18T07:49:35.618 DEBUG:teuthology.orchestra.run.smithi007:> sudo service rsyslog restart 2024-09-18T07:49:35.623 DEBUG:teuthology.orchestra.run.smithi050:> sudo service rsyslog restart 2024-09-18T07:49:35.718 INFO:teuthology.orchestra.run.smithi050.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-09-18T07:49:35.718 INFO:teuthology.orchestra.run.smithi007.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-09-18T07:49:36.220 INFO:teuthology.run_tasks:Running task internal.timer... 2024-09-18T07:49:36.227 INFO:teuthology.task.internal:Starting timer... 2024-09-18T07:49:36.227 INFO:teuthology.run_tasks:Running task pcp... 2024-09-18T07:49:36.258 INFO:teuthology.run_tasks:Running task selinux... 2024-09-18T07:49:36.268 DEBUG:teuthology.task:Applying overrides for task selinux: {'allowlist': ['scontext=system_u:system_r:logrotate_t:s0', 'scontext=system_u:system_r:getty_t:s0']} 2024-09-18T07:49:36.268 DEBUG:teuthology.orchestra.run.smithi007:> sudo service auditd rotate 2024-09-18T07:49:36.406 INFO:teuthology.orchestra.run.smithi007.stdout:Rotating logs: 2024-09-18T07:49:36.408 DEBUG:teuthology.orchestra.run.smithi050:> sudo service auditd rotate 2024-09-18T07:49:36.476 INFO:teuthology.orchestra.run.smithi050.stdout:Rotating logs: 2024-09-18T07:49:36.478 DEBUG:teuthology.task.selinux:Getting current SELinux state 2024-09-18T07:49:36.479 DEBUG:teuthology.orchestra.run.smithi007:> /usr/sbin/getenforce 2024-09-18T07:49:36.524 INFO:teuthology.orchestra.run.smithi007.stdout:Permissive 2024-09-18T07:49:36.524 DEBUG:teuthology.orchestra.run.smithi050:> /usr/sbin/getenforce 2024-09-18T07:49:36.557 INFO:teuthology.orchestra.run.smithi050.stdout:Permissive 2024-09-18T07:49:36.557 DEBUG:teuthology.task.selinux:Existing SELinux modes: {'ubuntu@smithi007.front.sepia.ceph.com': 'permissive', 'ubuntu@smithi050.front.sepia.ceph.com': 'permissive'} 2024-09-18T07:49:36.557 DEBUG:teuthology.orchestra.run.smithi007:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e context=system_u:system_r:getty_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-09-18T07:49:36.623 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:49:36.624 DEBUG:teuthology.orchestra.run.smithi050:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e context=system_u:system_r:getty_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-09-18T07:49:36.653 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:49:36.653 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2024-09-18T07:49:36.654 DEBUG:teuthology.orchestra.run.smithi007:> sudo /usr/sbin/setenforce permissive 2024-09-18T07:49:36.687 DEBUG:teuthology.orchestra.run.smithi050:> sudo /usr/sbin/setenforce permissive 2024-09-18T07:49:36.729 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2024-09-18T07:49:36.740 DEBUG:teuthology.repo_utils:Setting repo remote to https://git.ceph.com/ceph-cm-ansible.git 2024-09-18T07:49:36.743 INFO:teuthology.repo_utils:Fetching git.ceph.com_ceph-cm-ansible_main from origin 2024-09-18T07:49:36.848 DEBUG:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2024-09-18T07:49:36.858 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2024-09-18T07:49:36.859 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi007.front.sepia.ceph.com,smithi050.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2024-09-18T07:53:26.343 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi007.front.sepia.ceph.com'), Remote(name='ubuntu@smithi050.front.sepia.ceph.com')] 2024-09-18T07:53:26.345 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi007.front.sepia.ceph.com' 2024-09-18T07:53:26.346 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi007.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-18T07:53:26.424 DEBUG:teuthology.orchestra.run.smithi007:> true 2024-09-18T07:53:26.505 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi007.front.sepia.ceph.com' 2024-09-18T07:53:26.505 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi050.front.sepia.ceph.com' 2024-09-18T07:53:26.506 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi050.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-18T07:53:26.581 DEBUG:teuthology.orchestra.run.smithi050:> true 2024-09-18T07:53:26.657 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi050.front.sepia.ceph.com' 2024-09-18T07:53:26.658 INFO:teuthology.run_tasks:Running task clock... 2024-09-18T07:53:26.668 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2024-09-18T07:53:26.668 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-09-18T07:53:26.669 DEBUG:teuthology.orchestra.run.smithi007:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-09-18T07:53:26.672 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-09-18T07:53:26.672 DEBUG:teuthology.orchestra.run.smithi050:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-09-18T07:53:26.701 INFO:teuthology.orchestra.run.smithi007.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-09-18T07:53:26.716 INFO:teuthology.orchestra.run.smithi007.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-09-18T07:53:26.729 INFO:teuthology.orchestra.run.smithi050.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-09-18T07:53:26.742 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: ntpd: command not found 2024-09-18T07:53:26.744 INFO:teuthology.orchestra.run.smithi050.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-09-18T07:53:26.754 INFO:teuthology.orchestra.run.smithi007.stdout:506 Cannot talk to daemon 2024-09-18T07:53:26.768 INFO:teuthology.orchestra.run.smithi007.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-09-18T07:53:26.769 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: ntpd: command not found 2024-09-18T07:53:26.780 INFO:teuthology.orchestra.run.smithi050.stdout:506 Cannot talk to daemon 2024-09-18T07:53:26.783 INFO:teuthology.orchestra.run.smithi007.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-09-18T07:53:26.795 INFO:teuthology.orchestra.run.smithi050.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-09-18T07:53:26.810 INFO:teuthology.orchestra.run.smithi050.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-09-18T07:53:26.838 INFO:teuthology.orchestra.run.smithi007.stderr:bash: line 1: ntpq: command not found 2024-09-18T07:53:26.842 INFO:teuthology.orchestra.run.smithi007.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-09-18T07:53:26.842 INFO:teuthology.orchestra.run.smithi007.stdout:=============================================================================== 2024-09-18T07:53:26.842 INFO:teuthology.orchestra.run.smithi007.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-18T07:53:26.842 INFO:teuthology.orchestra.run.smithi007.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-18T07:53:26.842 INFO:teuthology.orchestra.run.smithi007.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-18T07:53:26.842 INFO:teuthology.orchestra.run.smithi007.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-18T07:53:26.873 INFO:teuthology.orchestra.run.smithi050.stderr:bash: line 1: ntpq: command not found 2024-09-18T07:53:26.878 INFO:teuthology.orchestra.run.smithi050.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-09-18T07:53:26.878 INFO:teuthology.orchestra.run.smithi050.stdout:=============================================================================== 2024-09-18T07:53:26.878 INFO:teuthology.orchestra.run.smithi050.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-18T07:53:26.878 INFO:teuthology.orchestra.run.smithi050.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-18T07:53:26.878 INFO:teuthology.orchestra.run.smithi050.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-18T07:53:26.878 INFO:teuthology.orchestra.run.smithi050.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-18T07:53:26.878 INFO:teuthology.run_tasks:Running task pexec... 2024-09-18T07:53:26.887 INFO:teuthology.task.pexec:Executing custom commands... 2024-09-18T07:53:26.887 INFO:teuthology.task.pexec:Running commands on host ubuntu@smithi007.front.sepia.ceph.com 2024-09-18T07:53:26.888 DEBUG:teuthology.orchestra.run.smithi007:> TESTDIR=/home/ubuntu/cephtest bash -s 2024-09-18T07:53:26.888 INFO:teuthology.task.pexec:Running commands on host ubuntu@smithi050.front.sepia.ceph.com 2024-09-18T07:53:26.888 DEBUG:teuthology.orchestra.run.smithi050:> TESTDIR=/home/ubuntu/cephtest bash -s 2024-09-18T07:53:27.254 INFO:teuthology.orchestra.run.smithi007.stdout:Dependencies resolved. 2024-09-18T07:53:27.256 INFO:teuthology.orchestra.run.smithi007.stdout:================================================================================ 2024-09-18T07:53:27.257 INFO:teuthology.orchestra.run.smithi007.stdout: Package Architecture Version Repository Size 2024-09-18T07:53:27.257 INFO:teuthology.orchestra.run.smithi007.stdout:================================================================================ 2024-09-18T07:53:27.257 INFO:teuthology.orchestra.run.smithi007.stdout:Removing: 2024-09-18T07:53:27.257 INFO:teuthology.orchestra.run.smithi007.stdout: nvme-cli x86_64 2.9.1-6.el9 @baseos 5.3 M 2024-09-18T07:53:27.257 INFO:teuthology.orchestra.run.smithi007.stdout:Removing unused dependencies: 2024-09-18T07:53:27.257 INFO:teuthology.orchestra.run.smithi007.stdout: libnvme x86_64 1.9-3.el9 @baseos 273 k 2024-09-18T07:53:27.257 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-09-18T07:53:27.257 INFO:teuthology.orchestra.run.smithi007.stdout:Transaction Summary 2024-09-18T07:53:27.257 INFO:teuthology.orchestra.run.smithi007.stdout:================================================================================ 2024-09-18T07:53:27.257 INFO:teuthology.orchestra.run.smithi007.stdout:Remove 2 Packages 2024-09-18T07:53:27.257 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-09-18T07:53:27.257 INFO:teuthology.orchestra.run.smithi007.stdout:Freed space: 5.5 M 2024-09-18T07:53:27.257 INFO:teuthology.orchestra.run.smithi007.stdout:Running transaction check 2024-09-18T07:53:27.261 INFO:teuthology.orchestra.run.smithi007.stdout:Transaction check succeeded. 2024-09-18T07:53:27.261 INFO:teuthology.orchestra.run.smithi007.stdout:Running transaction test 2024-09-18T07:53:27.285 INFO:teuthology.orchestra.run.smithi050.stdout:Dependencies resolved. 2024-09-18T07:53:27.293 INFO:teuthology.orchestra.run.smithi050.stdout:================================================================================ 2024-09-18T07:53:27.293 INFO:teuthology.orchestra.run.smithi050.stdout: Package Architecture Version Repository Size 2024-09-18T07:53:27.293 INFO:teuthology.orchestra.run.smithi050.stdout:================================================================================ 2024-09-18T07:53:27.294 INFO:teuthology.orchestra.run.smithi050.stdout:Removing: 2024-09-18T07:53:27.294 INFO:teuthology.orchestra.run.smithi050.stdout: nvme-cli x86_64 2.9.1-6.el9 @baseos 5.3 M 2024-09-18T07:53:27.294 INFO:teuthology.orchestra.run.smithi050.stdout:Removing unused dependencies: 2024-09-18T07:53:27.294 INFO:teuthology.orchestra.run.smithi050.stdout: libnvme x86_64 1.9-3.el9 @baseos 273 k 2024-09-18T07:53:27.294 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-09-18T07:53:27.294 INFO:teuthology.orchestra.run.smithi050.stdout:Transaction Summary 2024-09-18T07:53:27.294 INFO:teuthology.orchestra.run.smithi050.stdout:================================================================================ 2024-09-18T07:53:27.294 INFO:teuthology.orchestra.run.smithi050.stdout:Remove 2 Packages 2024-09-18T07:53:27.294 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-09-18T07:53:27.294 INFO:teuthology.orchestra.run.smithi050.stdout:Freed space: 5.5 M 2024-09-18T07:53:27.294 INFO:teuthology.orchestra.run.smithi050.stdout:Running transaction check 2024-09-18T07:53:27.298 INFO:teuthology.orchestra.run.smithi050.stdout:Transaction check succeeded. 2024-09-18T07:53:27.298 INFO:teuthology.orchestra.run.smithi050.stdout:Running transaction test 2024-09-18T07:53:27.338 INFO:teuthology.orchestra.run.smithi007.stdout:Transaction test succeeded. 2024-09-18T07:53:27.338 INFO:teuthology.orchestra.run.smithi007.stdout:Running transaction 2024-09-18T07:53:27.355 INFO:teuthology.orchestra.run.smithi050.stdout:Transaction test succeeded. 2024-09-18T07:53:27.356 INFO:teuthology.orchestra.run.smithi050.stdout:Running transaction 2024-09-18T07:53:27.493 INFO:teuthology.orchestra.run.smithi007.stdout: Preparing : 1/1 2024-09-18T07:53:27.498 INFO:teuthology.orchestra.run.smithi050.stdout: Preparing : 1/1 2024-09-18T07:53:27.573 INFO:teuthology.orchestra.run.smithi007.stdout: Erasing : nvme-cli-2.9.1-6.el9.x86_64 1/2 2024-09-18T07:53:27.599 INFO:teuthology.orchestra.run.smithi050.stdout: Erasing : nvme-cli-2.9.1-6.el9.x86_64 1/2 2024-09-18T07:53:27.637 INFO:teuthology.orchestra.run.smithi007.stdout: Erasing : libnvme-1.9-3.el9.x86_64 2/2 2024-09-18T07:53:27.660 INFO:teuthology.orchestra.run.smithi050.stdout: Erasing : libnvme-1.9-3.el9.x86_64 2/2 2024-09-18T07:53:28.100 INFO:teuthology.orchestra.run.smithi050.stdout: Running scriptlet: libnvme-1.9-3.el9.x86_64 2/2 2024-09-18T07:53:28.100 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : libnvme-1.9-3.el9.x86_64 1/2 2024-09-18T07:53:28.198 INFO:teuthology.orchestra.run.smithi007.stdout: Running scriptlet: libnvme-1.9-3.el9.x86_64 2/2 2024-09-18T07:53:28.198 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : libnvme-1.9-3.el9.x86_64 1/2 2024-09-18T07:53:28.325 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : nvme-cli-2.9.1-6.el9.x86_64 2/2 2024-09-18T07:53:28.325 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-09-18T07:53:28.326 INFO:teuthology.orchestra.run.smithi050.stdout:Removed: 2024-09-18T07:53:28.326 INFO:teuthology.orchestra.run.smithi050.stdout: libnvme-1.9-3.el9.x86_64 nvme-cli-2.9.1-6.el9.x86_64 2024-09-18T07:53:28.326 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-09-18T07:53:28.326 INFO:teuthology.orchestra.run.smithi050.stdout:Complete! 2024-09-18T07:53:28.455 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : nvme-cli-2.9.1-6.el9.x86_64 2/2 2024-09-18T07:53:28.455 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-09-18T07:53:28.455 INFO:teuthology.orchestra.run.smithi007.stdout:Removed: 2024-09-18T07:53:28.455 INFO:teuthology.orchestra.run.smithi007.stdout: libnvme-1.9-3.el9.x86_64 nvme-cli-2.9.1-6.el9.x86_64 2024-09-18T07:53:28.455 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-09-18T07:53:28.455 INFO:teuthology.orchestra.run.smithi007.stdout:Complete! 2024-09-18T07:53:29.177 INFO:teuthology.orchestra.run.smithi050.stdout:Last metadata expiration check: 0:00:46 ago on Wed 18 Sep 2024 07:52:42 AM UTC. 2024-09-18T07:53:29.274 INFO:teuthology.orchestra.run.smithi007.stdout:Last metadata expiration check: 0:00:46 ago on Wed 18 Sep 2024 07:52:42 AM UTC. 2024-09-18T07:53:29.323 INFO:teuthology.orchestra.run.smithi050.stdout:Dependencies resolved. 2024-09-18T07:53:29.324 INFO:teuthology.orchestra.run.smithi050.stdout:================================================================================ 2024-09-18T07:53:29.324 INFO:teuthology.orchestra.run.smithi050.stdout: Package Architecture Version Repository Size 2024-09-18T07:53:29.324 INFO:teuthology.orchestra.run.smithi050.stdout:================================================================================ 2024-09-18T07:53:29.324 INFO:teuthology.orchestra.run.smithi050.stdout:Installing: 2024-09-18T07:53:29.324 INFO:teuthology.orchestra.run.smithi050.stdout: nvme-cli x86_64 2.9.1-6.el9 baseos 906 k 2024-09-18T07:53:29.325 INFO:teuthology.orchestra.run.smithi050.stdout: nvmetcli noarch 0.7-3.el9 baseos 44 k 2024-09-18T07:53:29.325 INFO:teuthology.orchestra.run.smithi050.stdout:Installing dependencies: 2024-09-18T07:53:29.325 INFO:teuthology.orchestra.run.smithi050.stdout: libnvme x86_64 1.9-3.el9 baseos 106 k 2024-09-18T07:53:29.325 INFO:teuthology.orchestra.run.smithi050.stdout: python3-configshell noarch 1:1.1.30-1.el9 baseos 72 k 2024-09-18T07:53:29.325 INFO:teuthology.orchestra.run.smithi050.stdout: python3-kmod x86_64 0.9-32.el9 baseos 84 k 2024-09-18T07:53:29.325 INFO:teuthology.orchestra.run.smithi050.stdout: python3-urwid x86_64 2.1.2-4.el9 baseos 837 k 2024-09-18T07:53:29.325 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-09-18T07:53:29.325 INFO:teuthology.orchestra.run.smithi050.stdout:Transaction Summary 2024-09-18T07:53:29.325 INFO:teuthology.orchestra.run.smithi050.stdout:================================================================================ 2024-09-18T07:53:29.325 INFO:teuthology.orchestra.run.smithi050.stdout:Install 6 Packages 2024-09-18T07:53:29.325 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-09-18T07:53:29.331 INFO:teuthology.orchestra.run.smithi050.stdout:Total download size: 2.0 M 2024-09-18T07:53:29.331 INFO:teuthology.orchestra.run.smithi050.stdout:Installed size: 9.2 M 2024-09-18T07:53:29.331 INFO:teuthology.orchestra.run.smithi050.stdout:Downloading Packages: 2024-09-18T07:53:29.410 INFO:teuthology.orchestra.run.smithi007.stdout:Dependencies resolved. 2024-09-18T07:53:29.411 INFO:teuthology.orchestra.run.smithi007.stdout:================================================================================ 2024-09-18T07:53:29.411 INFO:teuthology.orchestra.run.smithi007.stdout: Package Architecture Version Repository Size 2024-09-18T07:53:29.411 INFO:teuthology.orchestra.run.smithi007.stdout:================================================================================ 2024-09-18T07:53:29.411 INFO:teuthology.orchestra.run.smithi007.stdout:Installing: 2024-09-18T07:53:29.411 INFO:teuthology.orchestra.run.smithi007.stdout: nvme-cli x86_64 2.9.1-6.el9 baseos 906 k 2024-09-18T07:53:29.411 INFO:teuthology.orchestra.run.smithi007.stdout: nvmetcli noarch 0.7-3.el9 baseos 44 k 2024-09-18T07:53:29.411 INFO:teuthology.orchestra.run.smithi007.stdout:Installing dependencies: 2024-09-18T07:53:29.411 INFO:teuthology.orchestra.run.smithi007.stdout: libnvme x86_64 1.9-3.el9 baseos 106 k 2024-09-18T07:53:29.411 INFO:teuthology.orchestra.run.smithi007.stdout: python3-configshell noarch 1:1.1.30-1.el9 baseos 72 k 2024-09-18T07:53:29.411 INFO:teuthology.orchestra.run.smithi007.stdout: python3-kmod x86_64 0.9-32.el9 baseos 84 k 2024-09-18T07:53:29.411 INFO:teuthology.orchestra.run.smithi007.stdout: python3-urwid x86_64 2.1.2-4.el9 baseos 837 k 2024-09-18T07:53:29.412 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-09-18T07:53:29.412 INFO:teuthology.orchestra.run.smithi007.stdout:Transaction Summary 2024-09-18T07:53:29.412 INFO:teuthology.orchestra.run.smithi007.stdout:================================================================================ 2024-09-18T07:53:29.412 INFO:teuthology.orchestra.run.smithi007.stdout:Install 6 Packages 2024-09-18T07:53:29.412 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-09-18T07:53:29.417 INFO:teuthology.orchestra.run.smithi007.stdout:Total download size: 2.0 M 2024-09-18T07:53:29.417 INFO:teuthology.orchestra.run.smithi007.stdout:Installed size: 9.2 M 2024-09-18T07:53:29.417 INFO:teuthology.orchestra.run.smithi007.stdout:Downloading Packages: 2024-09-18T07:53:29.608 INFO:teuthology.orchestra.run.smithi007.stdout:(1/6): nvmetcli-0.7-3.el9.noarch.rpm 307 kB/s | 44 kB 00:00 2024-09-18T07:53:29.633 INFO:teuthology.orchestra.run.smithi007.stdout:(2/6): libnvme-1.9-3.el9.x86_64.rpm 623 kB/s | 106 kB 00:00 2024-09-18T07:53:29.684 INFO:teuthology.orchestra.run.smithi007.stdout:(3/6): python3-kmod-0.9-32.el9.x86_64.rpm 1.6 MB/s | 84 kB 00:00 2024-09-18T07:53:29.717 INFO:teuthology.orchestra.run.smithi007.stdout:(4/6): python3-configshell-1.1.30-1.el9.noarch. 658 kB/s | 72 kB 00:00 2024-09-18T07:53:29.730 INFO:teuthology.orchestra.run.smithi050.stdout:(1/6): nvmetcli-0.7-3.el9.noarch.rpm 378 kB/s | 44 kB 00:00 2024-09-18T07:53:29.756 INFO:teuthology.orchestra.run.smithi050.stdout:(2/6): libnvme-1.9-3.el9.x86_64.rpm 741 kB/s | 106 kB 00:00 2024-09-18T07:53:29.809 INFO:teuthology.orchestra.run.smithi007.stdout:(5/6): nvme-cli-2.9.1-6.el9.x86_64.rpm 2.6 MB/s | 906 kB 00:00 2024-09-18T07:53:29.868 INFO:teuthology.orchestra.run.smithi007.stdout:(6/6): python3-urwid-2.1.2-4.el9.x86_64.rpm 4.4 MB/s | 837 kB 00:00 2024-09-18T07:53:29.868 INFO:teuthology.orchestra.run.smithi007.stdout:-------------------------------------------------------------------------------- 2024-09-18T07:53:29.868 INFO:teuthology.orchestra.run.smithi007.stdout:Total 4.4 MB/s | 2.0 MB 00:00 2024-09-18T07:53:29.929 INFO:teuthology.orchestra.run.smithi050.stdout:(3/6): nvme-cli-2.9.1-6.el9.x86_64.rpm 2.8 MB/s | 906 kB 00:00 2024-09-18T07:53:29.944 INFO:teuthology.orchestra.run.smithi007.stdout:Running transaction check 2024-09-18T07:53:29.959 INFO:teuthology.orchestra.run.smithi007.stdout:Transaction check succeeded. 2024-09-18T07:53:29.959 INFO:teuthology.orchestra.run.smithi007.stdout:Running transaction test 2024-09-18T07:53:29.964 INFO:teuthology.orchestra.run.smithi050.stdout:(4/6): python3-configshell-1.1.30-1.el9.noarch. 307 kB/s | 72 kB 00:00 2024-09-18T07:53:29.990 INFO:teuthology.orchestra.run.smithi050.stdout:(5/6): python3-kmod-0.9-32.el9.x86_64.rpm 360 kB/s | 84 kB 00:00 2024-09-18T07:53:30.049 INFO:teuthology.orchestra.run.smithi050.stdout:(6/6): python3-urwid-2.1.2-4.el9.x86_64.rpm 6.9 MB/s | 837 kB 00:00 2024-09-18T07:53:30.049 INFO:teuthology.orchestra.run.smithi050.stdout:-------------------------------------------------------------------------------- 2024-09-18T07:53:30.049 INFO:teuthology.orchestra.run.smithi050.stdout:Total 2.8 MB/s | 2.0 MB 00:00 2024-09-18T07:53:30.130 INFO:teuthology.orchestra.run.smithi050.stdout:Running transaction check 2024-09-18T07:53:30.146 INFO:teuthology.orchestra.run.smithi050.stdout:Transaction check succeeded. 2024-09-18T07:53:30.146 INFO:teuthology.orchestra.run.smithi050.stdout:Running transaction test 2024-09-18T07:53:30.277 INFO:teuthology.orchestra.run.smithi007.stdout:Transaction test succeeded. 2024-09-18T07:53:30.278 INFO:teuthology.orchestra.run.smithi007.stdout:Running transaction 2024-09-18T07:53:30.424 INFO:teuthology.orchestra.run.smithi050.stdout:Transaction test succeeded. 2024-09-18T07:53:30.425 INFO:teuthology.orchestra.run.smithi050.stdout:Running transaction 2024-09-18T07:53:30.748 INFO:teuthology.orchestra.run.smithi007.stdout: Preparing : 1/1 2024-09-18T07:53:30.846 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : python3-urwid-2.1.2-4.el9.x86_64 1/6 2024-09-18T07:53:30.870 INFO:teuthology.orchestra.run.smithi050.stdout: Preparing : 1/1 2024-09-18T07:53:30.880 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : python3-configshell-1:1.1.30-1.el9.noarch 2/6 2024-09-18T07:53:30.909 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : python3-kmod-0.9-32.el9.x86_64 3/6 2024-09-18T07:53:30.994 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : python3-urwid-2.1.2-4.el9.x86_64 1/6 2024-09-18T07:53:31.040 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : python3-configshell-1:1.1.30-1.el9.noarch 2/6 2024-09-18T07:53:31.073 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : python3-kmod-0.9-32.el9.x86_64 3/6 2024-09-18T07:53:31.130 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : libnvme-1.9-3.el9.x86_64 4/6 2024-09-18T07:53:31.165 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : nvme-cli-2.9.1-6.el9.x86_64 5/6 2024-09-18T07:53:31.298 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : libnvme-1.9-3.el9.x86_64 4/6 2024-09-18T07:53:31.337 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : nvme-cli-2.9.1-6.el9.x86_64 5/6 2024-09-18T07:53:31.648 INFO:teuthology.orchestra.run.smithi007.stdout: Running scriptlet: nvme-cli-2.9.1-6.el9.x86_64 5/6 2024-09-18T07:53:31.688 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : nvmetcli-0.7-3.el9.noarch 6/6 2024-09-18T07:53:31.828 INFO:teuthology.orchestra.run.smithi050.stdout: Running scriptlet: nvme-cli-2.9.1-6.el9.x86_64 5/6 2024-09-18T07:53:31.857 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : nvmetcli-0.7-3.el9.noarch 6/6 2024-09-18T07:53:32.825 INFO:teuthology.orchestra.run.smithi050.stdout: Running scriptlet: nvmetcli-0.7-3.el9.noarch 6/6 2024-09-18T07:53:32.825 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : libnvme-1.9-3.el9.x86_64 1/6 2024-09-18T07:53:32.825 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : nvme-cli-2.9.1-6.el9.x86_64 2/6 2024-09-18T07:53:32.826 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : nvmetcli-0.7-3.el9.noarch 3/6 2024-09-18T07:53:32.826 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : python3-configshell-1:1.1.30-1.el9.noarch 4/6 2024-09-18T07:53:32.826 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : python3-kmod-0.9-32.el9.x86_64 5/6 2024-09-18T07:53:32.887 INFO:teuthology.orchestra.run.smithi007.stdout: Running scriptlet: nvmetcli-0.7-3.el9.noarch 6/6 2024-09-18T07:53:32.888 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : libnvme-1.9-3.el9.x86_64 1/6 2024-09-18T07:53:32.888 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : nvme-cli-2.9.1-6.el9.x86_64 2/6 2024-09-18T07:53:32.888 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : nvmetcli-0.7-3.el9.noarch 3/6 2024-09-18T07:53:32.888 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : python3-configshell-1:1.1.30-1.el9.noarch 4/6 2024-09-18T07:53:32.888 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : python3-kmod-0.9-32.el9.x86_64 5/6 2024-09-18T07:53:33.172 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : python3-urwid-2.1.2-4.el9.x86_64 6/6 2024-09-18T07:53:33.172 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-09-18T07:53:33.172 INFO:teuthology.orchestra.run.smithi050.stdout:Installed: 2024-09-18T07:53:33.172 INFO:teuthology.orchestra.run.smithi050.stdout: libnvme-1.9-3.el9.x86_64 nvme-cli-2.9.1-6.el9.x86_64 2024-09-18T07:53:33.172 INFO:teuthology.orchestra.run.smithi050.stdout: nvmetcli-0.7-3.el9.noarch python3-configshell-1:1.1.30-1.el9.noarch 2024-09-18T07:53:33.172 INFO:teuthology.orchestra.run.smithi050.stdout: python3-kmod-0.9-32.el9.x86_64 python3-urwid-2.1.2-4.el9.x86_64 2024-09-18T07:53:33.172 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-09-18T07:53:33.172 INFO:teuthology.orchestra.run.smithi050.stdout:Complete! 2024-09-18T07:53:33.239 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : python3-urwid-2.1.2-4.el9.x86_64 6/6 2024-09-18T07:53:33.239 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-09-18T07:53:33.239 INFO:teuthology.orchestra.run.smithi007.stdout:Installed: 2024-09-18T07:53:33.239 INFO:teuthology.orchestra.run.smithi007.stdout: libnvme-1.9-3.el9.x86_64 nvme-cli-2.9.1-6.el9.x86_64 2024-09-18T07:53:33.239 INFO:teuthology.orchestra.run.smithi007.stdout: nvmetcli-0.7-3.el9.noarch python3-configshell-1:1.1.30-1.el9.noarch 2024-09-18T07:53:33.239 INFO:teuthology.orchestra.run.smithi007.stdout: python3-kmod-0.9-32.el9.x86_64 python3-urwid-2.1.2-4.el9.x86_64 2024-09-18T07:53:33.239 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-09-18T07:53:33.239 INFO:teuthology.orchestra.run.smithi007.stdout:Complete! 2024-09-18T07:53:33.529 DEBUG:teuthology.parallel:result is None 2024-09-18T07:53:33.592 DEBUG:teuthology.parallel:result is None 2024-09-18T07:53:33.592 INFO:teuthology.run_tasks:Running task install... 2024-09-18T07:53:33.600 DEBUG:teuthology.task.install:project ceph 2024-09-18T07:53:33.600 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'flavor': 'default', 'sha1': 'e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e'}} 2024-09-18T07:53:33.600 DEBUG:teuthology.task.install:config {'flavor': 'default', 'sha1': 'e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e'} 2024-09-18T07:53:33.601 INFO:teuthology.task.install:Using flavor: default 2024-09-18T07:53:33.606 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'ceph-volume', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'ceph-volume', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2024-09-18T07:53:33.606 INFO:teuthology.task.install:extra packages: [] 2024-09-18T07:53:33.606 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': [], 'extra_system_packages': [], 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': 'e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e', 'tag': None, 'wait_for_package': False} 2024-09-18T07:53:33.607 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e 2024-09-18T07:53:33.609 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': [], 'extra_system_packages': [], 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': 'e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e', 'tag': None, 'wait_for_package': False} 2024-09-18T07:53:33.609 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e 2024-09-18T07:53:33.749 INFO:teuthology.task.install.rpm:Pulling from https://4.chacra.ceph.com/r/ceph/wip-guits-main-2024-09-17-1213/e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e/centos/9/flavors/default/ 2024-09-18T07:53:33.749 INFO:teuthology.task.install.rpm:Package version is 19.3.0-4983.ge7fb7b56 2024-09-18T07:53:33.754 INFO:teuthology.task.install.rpm:Pulling from https://4.chacra.ceph.com/r/ceph/wip-guits-main-2024-09-17-1213/e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e/centos/9/flavors/default/ 2024-09-18T07:53:33.754 INFO:teuthology.task.install.rpm:Package version is 19.3.0-4983.ge7fb7b56 2024-09-18T07:53:33.859 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://4.chacra.ceph.com/r/ceph/wip-guits-main-2024-09-17-1213/e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://4.chacra.ceph.com/r/ceph/wip-guits-main-2024-09-17-1213/e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://4.chacra.ceph.com/r/ceph/wip-guits-main-2024-09-17-1213/e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-09-18T07:53:33.859 DEBUG:teuthology.orchestra.run.smithi007:> set -ex 2024-09-18T07:53:33.859 DEBUG:teuthology.orchestra.run.smithi007:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-09-18T07:53:33.864 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://4.chacra.ceph.com/r/ceph/wip-guits-main-2024-09-17-1213/e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://4.chacra.ceph.com/r/ceph/wip-guits-main-2024-09-17-1213/e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://4.chacra.ceph.com/r/ceph/wip-guits-main-2024-09-17-1213/e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-09-18T07:53:33.864 DEBUG:teuthology.orchestra.run.smithi050:> set -ex 2024-09-18T07:53:33.864 DEBUG:teuthology.orchestra.run.smithi050:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-09-18T07:53:33.890 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd on remote rpm x86_64 2024-09-18T07:53:33.890 DEBUG:teuthology.orchestra.run.smithi007:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-09-18T07:53:33.895 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd on remote rpm x86_64 2024-09-18T07:53:33.895 DEBUG:teuthology.orchestra.run.smithi050:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-09-18T07:53:33.959 DEBUG:teuthology.orchestra.run.smithi007:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-09-18T07:53:33.964 DEBUG:teuthology.orchestra.run.smithi050:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-09-18T07:53:34.036 DEBUG:teuthology.orchestra.run.smithi007:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-09-18T07:53:34.043 DEBUG:teuthology.orchestra.run.smithi050:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-09-18T07:53:34.102 INFO:teuthology.orchestra.run.smithi007.stdout:check_obsoletes = 1 2024-09-18T07:53:34.103 DEBUG:teuthology.orchestra.run.smithi007:> sudo yum clean all 2024-09-18T07:53:34.110 INFO:teuthology.orchestra.run.smithi050.stdout:check_obsoletes = 1 2024-09-18T07:53:34.112 DEBUG:teuthology.orchestra.run.smithi050:> sudo yum clean all 2024-09-18T07:53:34.355 INFO:teuthology.orchestra.run.smithi050.stdout:45 files removed 2024-09-18T07:53:34.374 INFO:teuthology.orchestra.run.smithi007.stdout:45 files removed 2024-09-18T07:53:34.382 DEBUG:teuthology.orchestra.run.smithi050:> sudo yum -y install ceph-radosgw 2024-09-18T07:53:34.402 DEBUG:teuthology.orchestra.run.smithi007:> sudo yum -y install ceph-radosgw 2024-09-18T07:53:34.963 INFO:teuthology.orchestra.run.smithi050.stdout:ceph packages for x86_64 300 kB/s | 84 kB 00:00 2024-09-18T07:53:34.973 INFO:teuthology.orchestra.run.smithi007.stdout:ceph packages for x86_64 310 kB/s | 84 kB 00:00 2024-09-18T07:53:35.255 INFO:teuthology.orchestra.run.smithi050.stdout:ceph noarch packages 99 kB/s | 24 kB 00:00 2024-09-18T07:53:35.266 INFO:teuthology.orchestra.run.smithi007.stdout:ceph noarch packages 98 kB/s | 24 kB 00:00 2024-09-18T07:53:35.531 INFO:teuthology.orchestra.run.smithi050.stdout:ceph source packages 8.2 kB/s | 1.9 kB 00:00 2024-09-18T07:53:35.551 INFO:teuthology.orchestra.run.smithi007.stdout:ceph source packages 7.9 kB/s | 1.9 kB 00:00 2024-09-18T07:53:36.015 INFO:teuthology.orchestra.run.smithi050.stdout:CentOS Stream 9 - BaseOS 18 MB/s | 8.3 MB 00:00 2024-09-18T07:53:37.957 INFO:teuthology.orchestra.run.smithi050.stdout:CentOS Stream 9 - AppStream 24 MB/s | 20 MB 00:00 2024-09-18T07:53:38.625 INFO:teuthology.orchestra.run.smithi007.stdout:CentOS Stream 9 - BaseOS 2.7 MB/s | 8.3 MB 00:03 2024-09-18T07:53:42.300 INFO:teuthology.orchestra.run.smithi050.stdout:CentOS Stream 9 - CRB 16 MB/s | 6.5 MB 00:00 2024-09-18T07:53:43.859 INFO:teuthology.orchestra.run.smithi050.stdout:CentOS Stream 9 - Extras packages 68 kB/s | 19 kB 00:00 2024-09-18T07:53:45.222 INFO:teuthology.orchestra.run.smithi050.stdout:Extra Packages for Enterprise Linux 18 MB/s | 23 MB 00:01 2024-09-18T07:53:50.911 INFO:teuthology.orchestra.run.smithi050.stdout:lab-extras 33 kB/s | 1.7 kB 00:00 2024-09-18T07:53:53.216 INFO:teuthology.orchestra.run.smithi050.stdout:Dependencies resolved. 2024-09-18T07:53:53.219 INFO:teuthology.orchestra.run.smithi050.stdout:================================================================================ 2024-09-18T07:53:53.219 INFO:teuthology.orchestra.run.smithi050.stdout: Package Arch Version Repository Size 2024-09-18T07:53:53.219 INFO:teuthology.orchestra.run.smithi050.stdout:================================================================================ 2024-09-18T07:53:53.219 INFO:teuthology.orchestra.run.smithi050.stdout:Installing: 2024-09-18T07:53:53.219 INFO:teuthology.orchestra.run.smithi050.stdout: ceph-radosgw x86_64 2:19.3.0-4983.ge7fb7b56.el9 ceph 11 M 2024-09-18T07:53:53.219 INFO:teuthology.orchestra.run.smithi050.stdout:Upgrading: 2024-09-18T07:53:53.219 INFO:teuthology.orchestra.run.smithi050.stdout: librados2 x86_64 2:19.3.0-4983.ge7fb7b56.el9 ceph 3.5 M 2024-09-18T07:53:53.219 INFO:teuthology.orchestra.run.smithi050.stdout: librbd1 x86_64 2:19.3.0-4983.ge7fb7b56.el9 ceph 3.2 M 2024-09-18T07:53:53.219 INFO:teuthology.orchestra.run.smithi050.stdout:Installing dependencies: 2024-09-18T07:53:53.219 INFO:teuthology.orchestra.run.smithi050.stdout: boost-program-options x86_64 1.75.0-8.el9 appstream 107 k 2024-09-18T07:53:53.219 INFO:teuthology.orchestra.run.smithi050.stdout: ceph-base x86_64 2:19.3.0-4983.ge7fb7b56.el9 ceph 5.5 M 2024-09-18T07:53:53.220 INFO:teuthology.orchestra.run.smithi050.stdout: ceph-common x86_64 2:19.3.0-4983.ge7fb7b56.el9 ceph 22 M 2024-09-18T07:53:53.220 INFO:teuthology.orchestra.run.smithi050.stdout: ceph-selinux x86_64 2:19.3.0-4983.ge7fb7b56.el9 ceph 25 k 2024-09-18T07:53:53.220 INFO:teuthology.orchestra.run.smithi050.stdout: gperftools-libs x86_64 2.9.1-3.el9 epel 308 k 2024-09-18T07:53:53.220 INFO:teuthology.orchestra.run.smithi050.stdout: libarrow x86_64 9.0.0-11.el9 epel 4.4 M 2024-09-18T07:53:53.220 INFO:teuthology.orchestra.run.smithi050.stdout: libarrow-doc noarch 9.0.0-11.el9 epel 26 k 2024-09-18T07:53:53.220 INFO:teuthology.orchestra.run.smithi050.stdout: libcephfs2 x86_64 2:19.3.0-4983.ge7fb7b56.el9 ceph 755 k 2024-09-18T07:53:53.220 INFO:teuthology.orchestra.run.smithi050.stdout: libnbd x86_64 1.20.2-2.el9 appstream 171 k 2024-09-18T07:53:53.220 INFO:teuthology.orchestra.run.smithi050.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-09-18T07:53:53.220 INFO:teuthology.orchestra.run.smithi050.stdout: libpmemobj x86_64 1.12.1-1.el9 appstream 160 k 2024-09-18T07:53:53.220 INFO:teuthology.orchestra.run.smithi050.stdout: librabbitmq x86_64 0.11.0-7.el9 appstream 45 k 2024-09-18T07:53:53.220 INFO:teuthology.orchestra.run.smithi050.stdout: libradosstriper1 x86_64 2:19.3.0-4983.ge7fb7b56.el9 ceph 509 k 2024-09-18T07:53:53.220 INFO:teuthology.orchestra.run.smithi050.stdout: librdkafka x86_64 1.6.1-102.el9 appstream 662 k 2024-09-18T07:53:53.221 INFO:teuthology.orchestra.run.smithi050.stdout: librgw2 x86_64 2:19.3.0-4983.ge7fb7b56.el9 ceph 5.5 M 2024-09-18T07:53:53.221 INFO:teuthology.orchestra.run.smithi050.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-09-18T07:53:53.221 INFO:teuthology.orchestra.run.smithi050.stdout: lttng-ust x86_64 2.12.0-6.el9 appstream 292 k 2024-09-18T07:53:53.221 INFO:teuthology.orchestra.run.smithi050.stdout: parquet-libs x86_64 9.0.0-11.el9 epel 839 k 2024-09-18T07:53:53.221 INFO:teuthology.orchestra.run.smithi050.stdout: python3-ceph-argparse x86_64 2:19.3.0-4983.ge7fb7b56.el9 ceph 45 k 2024-09-18T07:53:53.221 INFO:teuthology.orchestra.run.smithi050.stdout: python3-ceph-common x86_64 2:19.3.0-4983.ge7fb7b56.el9 ceph 143 k 2024-09-18T07:53:53.221 INFO:teuthology.orchestra.run.smithi050.stdout: python3-cephfs x86_64 2:19.3.0-4983.ge7fb7b56.el9 ceph 161 k 2024-09-18T07:53:53.221 INFO:teuthology.orchestra.run.smithi050.stdout: python3-prettytable noarch 0.7.2-27.el9 appstream 42 k 2024-09-18T07:53:53.221 INFO:teuthology.orchestra.run.smithi050.stdout: python3-rados x86_64 2:19.3.0-4983.ge7fb7b56.el9 ceph 320 k 2024-09-18T07:53:53.221 INFO:teuthology.orchestra.run.smithi050.stdout: python3-rbd x86_64 2:19.3.0-4983.ge7fb7b56.el9 ceph 306 k 2024-09-18T07:53:53.221 INFO:teuthology.orchestra.run.smithi050.stdout: python3-rgw x86_64 2:19.3.0-4983.ge7fb7b56.el9 ceph 99 k 2024-09-18T07:53:53.221 INFO:teuthology.orchestra.run.smithi050.stdout: qatlib x86_64 24.02.0-1.el9 appstream 221 k 2024-09-18T07:53:53.221 INFO:teuthology.orchestra.run.smithi050.stdout: qatzip-libs x86_64 1.2.0-1.el9 appstream 47 k 2024-09-18T07:53:53.222 INFO:teuthology.orchestra.run.smithi050.stdout: re2 x86_64 1:20211101-20.el9 epel 191 k 2024-09-18T07:53:53.222 INFO:teuthology.orchestra.run.smithi050.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-09-18T07:53:53.222 INFO:teuthology.orchestra.run.smithi050.stdout:Installing weak dependencies: 2024-09-18T07:53:53.222 INFO:teuthology.orchestra.run.smithi050.stdout: qatlib-service x86_64 24.02.0-1.el9 appstream 36 k 2024-09-18T07:53:53.222 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-09-18T07:53:53.222 INFO:teuthology.orchestra.run.smithi050.stdout:Transaction Summary 2024-09-18T07:53:53.222 INFO:teuthology.orchestra.run.smithi050.stdout:================================================================================ 2024-09-18T07:53:53.222 INFO:teuthology.orchestra.run.smithi050.stdout:Install 31 Packages 2024-09-18T07:53:53.222 INFO:teuthology.orchestra.run.smithi050.stdout:Upgrade 2 Packages 2024-09-18T07:53:53.222 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-09-18T07:53:53.223 INFO:teuthology.orchestra.run.smithi050.stdout:Total download size: 62 M 2024-09-18T07:53:53.223 INFO:teuthology.orchestra.run.smithi050.stdout:Downloading Packages: 2024-09-18T07:53:53.987 INFO:teuthology.orchestra.run.smithi050.stdout:(1/33): ceph-base-19.3.0-4983.ge7fb7b56.el9.x86 12 MB/s | 5.5 MB 00:00 2024-09-18T07:53:54.046 INFO:teuthology.orchestra.run.smithi050.stdout:(2/33): ceph-selinux-19.3.0-4983.ge7fb7b56.el9. 431 kB/s | 25 kB 00:00 2024-09-18T07:53:54.146 INFO:teuthology.orchestra.run.smithi050.stdout:(3/33): libcephfs2-19.3.0-4983.ge7fb7b56.el9.x8 7.4 MB/s | 755 kB 00:00 2024-09-18T07:53:54.239 INFO:teuthology.orchestra.run.smithi050.stdout:(4/33): libradosstriper1-19.3.0-4983.ge7fb7b56. 5.4 MB/s | 509 kB 00:00 2024-09-18T07:53:54.522 INFO:teuthology.orchestra.run.smithi050.stdout:(5/33): ceph-radosgw-19.3.0-4983.ge7fb7b56.el9. 11 MB/s | 11 MB 00:00 2024-09-18T07:53:54.739 INFO:teuthology.orchestra.run.smithi050.stdout:(6/33): ceph-common-19.3.0-4983.ge7fb7b56.el9.x 18 MB/s | 22 MB 00:01 2024-09-18T07:53:54.756 INFO:teuthology.orchestra.run.smithi050.stdout:(7/33): python3-ceph-argparse-19.3.0-4983.ge7fb 193 kB/s | 45 kB 00:00 2024-09-18T07:53:54.789 INFO:teuthology.orchestra.run.smithi050.stdout:(8/33): python3-ceph-common-19.3.0-4983.ge7fb7b 2.8 MB/s | 143 kB 00:00 2024-09-18T07:53:54.815 INFO:teuthology.orchestra.run.smithi050.stdout:(9/33): python3-cephfs-19.3.0-4983.ge7fb7b56.el 2.7 MB/s | 161 kB 00:00 2024-09-18T07:53:54.840 INFO:teuthology.orchestra.run.smithi050.stdout:(10/33): python3-rados-19.3.0-4983.ge7fb7b56.el 6.2 MB/s | 320 kB 00:00 2024-09-18T07:53:54.866 INFO:teuthology.orchestra.run.smithi050.stdout:(11/33): python3-rbd-19.3.0-4983.ge7fb7b56.el9. 6.0 MB/s | 306 kB 00:00 2024-09-18T07:53:54.891 INFO:teuthology.orchestra.run.smithi050.stdout:(12/33): python3-rgw-19.3.0-4983.ge7fb7b56.el9. 1.9 MB/s | 99 kB 00:00 2024-09-18T07:53:54.966 INFO:teuthology.orchestra.run.smithi050.stdout:(13/33): librgw2-19.3.0-4983.ge7fb7b56.el9.x86_ 7.6 MB/s | 5.5 MB 00:00 2024-09-18T07:53:55.050 INFO:teuthology.orchestra.run.smithi050.stdout:(14/33): boost-program-options-1.75.0-8.el9.x86 580 kB/s | 107 kB 00:00 2024-09-18T07:53:55.075 INFO:teuthology.orchestra.run.smithi050.stdout:(15/33): libnbd-1.20.2-2.el9.x86_64.rpm 927 kB/s | 171 kB 00:00 2024-09-18T07:53:55.100 INFO:teuthology.orchestra.run.smithi050.stdout:(16/33): libpmemobj-1.12.1-1.el9.x86_64.rpm 1.2 MB/s | 160 kB 00:00 2024-09-18T07:53:55.125 INFO:teuthology.orchestra.run.smithi050.stdout:(17/33): librabbitmq-0.11.0-7.el9.x86_64.rpm 601 kB/s | 45 kB 00:00 2024-09-18T07:53:55.159 INFO:teuthology.orchestra.run.smithi050.stdout:(18/33): python3-prettytable-0.7.2-27.el9.noarc 1.3 MB/s | 42 kB 00:00 2024-09-18T07:53:55.193 INFO:teuthology.orchestra.run.smithi050.stdout:(19/33): librdkafka-1.6.1-102.el9.x86_64.rpm 5.5 MB/s | 662 kB 00:00 2024-09-18T07:53:55.218 INFO:teuthology.orchestra.run.smithi050.stdout:(20/33): lttng-ust-2.12.0-6.el9.x86_64.rpm 3.1 MB/s | 292 kB 00:00 2024-09-18T07:53:55.235 INFO:teuthology.orchestra.run.smithi050.stdout:(21/33): qatlib-service-24.02.0-1.el9.x86_64.rp 860 kB/s | 36 kB 00:00 2024-09-18T07:53:55.269 INFO:teuthology.orchestra.run.smithi050.stdout:(22/33): qatlib-24.02.0-1.el9.x86_64.rpm 2.0 MB/s | 221 kB 00:00 2024-09-18T07:53:55.286 INFO:teuthology.orchestra.run.smithi050.stdout:(23/33): qatzip-libs-1.2.0-1.el9.x86_64.rpm 706 kB/s | 47 kB 00:00 2024-09-18T07:53:55.361 INFO:teuthology.orchestra.run.smithi050.stdout:(24/33): gperftools-libs-2.9.1-3.el9.x86_64.rpm 2.4 MB/s | 308 kB 00:00 2024-09-18T07:53:55.378 INFO:teuthology.orchestra.run.smithi050.stdout:(25/33): libarrow-doc-9.0.0-11.el9.noarch.rpm 342 kB/s | 26 kB 00:00 2024-09-18T07:53:55.395 INFO:teuthology.orchestra.run.smithi050.stdout:(26/33): liboath-2.6.7-2.el9.x86_64.rpm 1.4 MB/s | 49 kB 00:00 2024-09-18T07:53:55.420 INFO:teuthology.orchestra.run.smithi050.stdout:(27/33): libunwind-1.6.2-1.el9.x86_64.rpm 1.6 MB/s | 67 kB 00:00 2024-09-18T07:53:55.470 INFO:teuthology.orchestra.run.smithi050.stdout:(28/33): re2-20211101-20.el9.x86_64.rpm 3.8 MB/s | 191 kB 00:00 2024-09-18T07:53:55.512 INFO:teuthology.orchestra.run.smithi050.stdout:(29/33): parquet-libs-9.0.0-11.el9.x86_64.rpm 7.0 MB/s | 839 kB 00:00 2024-09-18T07:53:55.697 INFO:teuthology.orchestra.run.smithi050.stdout:(30/33): librados2-19.3.0-4983.ge7fb7b56.el9.x8 19 MB/s | 3.5 MB 00:00 2024-09-18T07:53:55.739 INFO:teuthology.orchestra.run.smithi050.stdout:(31/33): thrift-0.15.0-2.el9.x86_64.rpm 5.9 MB/s | 1.6 MB 00:00 2024-09-18T07:53:55.839 INFO:teuthology.orchestra.run.smithi050.stdout:(32/33): libarrow-9.0.0-11.el9.x86_64.rpm 7.7 MB/s | 4.4 MB 00:00 2024-09-18T07:53:55.914 INFO:teuthology.orchestra.run.smithi050.stdout:(33/33): librbd1-19.3.0-4983.ge7fb7b56.el9.x86_ 15 MB/s | 3.2 MB 00:00 2024-09-18T07:53:55.919 INFO:teuthology.orchestra.run.smithi050.stdout:-------------------------------------------------------------------------------- 2024-09-18T07:53:55.919 INFO:teuthology.orchestra.run.smithi050.stdout:Total 23 MB/s | 62 MB 00:02 2024-09-18T07:53:56.033 INFO:teuthology.orchestra.run.smithi050.stdout:Running transaction check 2024-09-18T07:53:56.075 INFO:teuthology.orchestra.run.smithi050.stdout:Transaction check succeeded. 2024-09-18T07:53:56.075 INFO:teuthology.orchestra.run.smithi050.stdout:Running transaction test 2024-09-18T07:53:56.600 INFO:teuthology.orchestra.run.smithi050.stdout:Transaction test succeeded. 2024-09-18T07:53:56.601 INFO:teuthology.orchestra.run.smithi050.stdout:Running transaction 2024-09-18T07:53:58.118 INFO:teuthology.orchestra.run.smithi050.stdout: Preparing : 1/1 2024-09-18T07:53:58.212 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 1/35 2024-09-18T07:53:58.245 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 2/35 2024-09-18T07:53:58.249 INFO:teuthology.orchestra.run.smithi007.stdout:CentOS Stream 9 - AppStream 1.1 MB/s | 20 MB 00:18 2024-09-18T07:53:58.278 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 3/35 2024-09-18T07:53:58.301 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : qatlib-service-24.02.0-1.el9.x86_64 4/35 2024-09-18T07:53:58.326 INFO:teuthology.orchestra.run.smithi050.stdout: Running scriptlet: qatlib-service-24.02.0-1.el9.x86_64 4/35 2024-09-18T07:53:58.521 INFO:teuthology.orchestra.run.smithi050.stdout: Running scriptlet: qatlib-24.02.0-1.el9.x86_64 5/35 2024-09-18T07:53:58.574 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : qatlib-24.02.0-1.el9.x86_64 5/35 2024-09-18T07:53:58.930 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : thrift-0.15.0-2.el9.x86_64 6/35 2024-09-18T07:53:59.047 INFO:teuthology.orchestra.run.smithi050.stdout: Upgrading : librados2-2:19.3.0-4983.ge7fb7b56.el9.x86_64 7/35 2024-09-18T07:53:59.140 INFO:teuthology.orchestra.run.smithi050.stdout: Running scriptlet: librados2-2:19.3.0-4983.ge7fb7b56.el9.x86_64 7/35 2024-09-18T07:53:59.255 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : python3-rados-2:19.3.0-4983.ge7fb7b56.el9.x86_64 8/35 2024-09-18T07:53:59.284 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : libcephfs2-2:19.3.0-4983.ge7fb7b56.el9.x86_64 9/35 2024-09-18T07:53:59.346 INFO:teuthology.orchestra.run.smithi050.stdout: Running scriptlet: libcephfs2-2:19.3.0-4983.ge7fb7b56.el9.x86_64 9/35 2024-09-18T07:53:59.382 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : liboath-2.6.7-2.el9.x86_64 10/35 2024-09-18T07:53:59.436 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : python3-ceph-argparse-2:19.3.0-4983.ge7fb7b56.el9. 11/35 2024-09-18T07:53:59.505 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : python3-cephfs-2:19.3.0-4983.ge7fb7b56.el9.x86_64 12/35 2024-09-18T07:53:59.534 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : libradosstriper1-2:19.3.0-4983.ge7fb7b56.el9.x86_6 13/35 2024-09-18T07:53:59.582 INFO:teuthology.orchestra.run.smithi050.stdout: Running scriptlet: libradosstriper1-2:19.3.0-4983.ge7fb7b56.el9.x86_6 13/35 2024-09-18T07:53:59.616 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : qatzip-libs-1.2.0-1.el9.x86_64 14/35 2024-09-18T07:53:59.744 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : re2-1:20211101-20.el9.x86_64 15/35 2024-09-18T07:53:59.784 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : libarrow-9.0.0-11.el9.x86_64 16/35 2024-09-18T07:53:59.829 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 17/35 2024-09-18T07:53:59.874 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : gperftools-libs-2.9.1-3.el9.x86_64 18/35 2024-09-18T07:53:59.903 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : libarrow-doc-9.0.0-11.el9.noarch 19/35 2024-09-18T07:53:59.945 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 20/35 2024-09-18T07:53:59.981 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 21/35 2024-09-18T07:54:00.318 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : libnbd-1.20.2-2.el9.x86_64 22/35 2024-09-18T07:54:00.358 INFO:teuthology.orchestra.run.smithi050.stdout: Upgrading : librbd1-2:19.3.0-4983.ge7fb7b56.el9.x86_64 23/35 2024-09-18T07:54:00.457 INFO:teuthology.orchestra.run.smithi050.stdout: Running scriptlet: librbd1-2:19.3.0-4983.ge7fb7b56.el9.x86_64 23/35 2024-09-18T07:54:00.488 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : python3-rbd-2:19.3.0-4983.ge7fb7b56.el9.x86_64 24/35 2024-09-18T07:54:00.525 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 25/35 2024-09-18T07:54:01.043 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : parquet-libs-9.0.0-11.el9.x86_64 26/35 2024-09-18T07:54:01.069 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : librgw2-2:19.3.0-4983.ge7fb7b56.el9.x86_64 27/35 2024-09-18T07:54:01.141 INFO:teuthology.orchestra.run.smithi050.stdout: Running scriptlet: librgw2-2:19.3.0-4983.ge7fb7b56.el9.x86_64 27/35 2024-09-18T07:54:01.221 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : python3-rgw-2:19.3.0-4983.ge7fb7b56.el9.x86_64 28/35 2024-09-18T07:54:01.246 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : python3-ceph-common-2:19.3.0-4983.ge7fb7b56.el9.x8 29/35 2024-09-18T07:54:03.729 INFO:teuthology.orchestra.run.smithi050.stdout: Running scriptlet: ceph-common-2:19.3.0-4983.ge7fb7b56.el9.x86_64 30/35 2024-09-18T07:54:03.768 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : ceph-common-2:19.3.0-4983.ge7fb7b56.el9.x86_64 30/35 2024-09-18T07:54:04.184 INFO:teuthology.orchestra.run.smithi007.stdout:CentOS Stream 9 - CRB 3.2 MB/s | 6.5 MB 00:02 2024-09-18T07:54:04.370 INFO:teuthology.orchestra.run.smithi050.stdout: Running scriptlet: ceph-common-2:19.3.0-4983.ge7fb7b56.el9.x86_64 30/35 2024-09-18T07:54:04.405 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : ceph-base-2:19.3.0-4983.ge7fb7b56.el9.x86_64 31/35 2024-09-18T07:54:04.504 INFO:teuthology.orchestra.run.smithi050.stdout: Running scriptlet: ceph-base-2:19.3.0-4983.ge7fb7b56.el9.x86_64 31/35 2024-09-18T07:54:04.504 INFO:teuthology.orchestra.run.smithi050.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-09-18T07:54:04.504 INFO:teuthology.orchestra.run.smithi050.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-09-18T07:54:04.504 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-09-18T07:54:04.539 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : ceph-selinux-2:19.3.0-4983.ge7fb7b56.el9.x86_64 32/35 2024-09-18T07:54:05.825 INFO:teuthology.orchestra.run.smithi007.stdout:CentOS Stream 9 - Extras packages 59 kB/s | 19 kB 00:00 2024-09-18T07:54:07.338 INFO:teuthology.orchestra.run.smithi007.stdout:Extra Packages for Enterprise Linux 16 MB/s | 23 MB 00:01 2024-09-18T07:54:12.960 INFO:teuthology.orchestra.run.smithi007.stdout:lab-extras 31 kB/s | 1.7 kB 00:00 2024-09-18T07:54:15.214 INFO:teuthology.orchestra.run.smithi050.stdout: Running scriptlet: ceph-selinux-2:19.3.0-4983.ge7fb7b56.el9.x86_64 32/35 2024-09-18T07:54:15.214 INFO:teuthology.orchestra.run.smithi050.stdout:skipping the directory /sys 2024-09-18T07:54:15.215 INFO:teuthology.orchestra.run.smithi050.stdout:skipping the directory /proc 2024-09-18T07:54:15.215 INFO:teuthology.orchestra.run.smithi050.stdout:skipping the directory /mnt 2024-09-18T07:54:15.215 INFO:teuthology.orchestra.run.smithi050.stdout:skipping the directory /var/tmp 2024-09-18T07:54:15.215 INFO:teuthology.orchestra.run.smithi050.stdout:skipping the directory /home 2024-09-18T07:54:15.215 INFO:teuthology.orchestra.run.smithi050.stdout:skipping the directory /root 2024-09-18T07:54:15.215 INFO:teuthology.orchestra.run.smithi050.stdout:skipping the directory /tmp 2024-09-18T07:54:15.215 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-09-18T07:54:15.280 INFO:teuthology.orchestra.run.smithi007.stdout:Dependencies resolved. 2024-09-18T07:54:15.283 INFO:teuthology.orchestra.run.smithi007.stdout:================================================================================ 2024-09-18T07:54:15.283 INFO:teuthology.orchestra.run.smithi007.stdout: Package Arch Version Repository Size 2024-09-18T07:54:15.283 INFO:teuthology.orchestra.run.smithi007.stdout:================================================================================ 2024-09-18T07:54:15.283 INFO:teuthology.orchestra.run.smithi007.stdout:Installing: 2024-09-18T07:54:15.283 INFO:teuthology.orchestra.run.smithi007.stdout: ceph-radosgw x86_64 2:19.3.0-4983.ge7fb7b56.el9 ceph 11 M 2024-09-18T07:54:15.283 INFO:teuthology.orchestra.run.smithi007.stdout:Upgrading: 2024-09-18T07:54:15.283 INFO:teuthology.orchestra.run.smithi007.stdout: librados2 x86_64 2:19.3.0-4983.ge7fb7b56.el9 ceph 3.5 M 2024-09-18T07:54:15.284 INFO:teuthology.orchestra.run.smithi007.stdout: librbd1 x86_64 2:19.3.0-4983.ge7fb7b56.el9 ceph 3.2 M 2024-09-18T07:54:15.284 INFO:teuthology.orchestra.run.smithi007.stdout:Installing dependencies: 2024-09-18T07:54:15.284 INFO:teuthology.orchestra.run.smithi007.stdout: boost-program-options x86_64 1.75.0-8.el9 appstream 107 k 2024-09-18T07:54:15.284 INFO:teuthology.orchestra.run.smithi007.stdout: ceph-base x86_64 2:19.3.0-4983.ge7fb7b56.el9 ceph 5.5 M 2024-09-18T07:54:15.284 INFO:teuthology.orchestra.run.smithi007.stdout: ceph-common x86_64 2:19.3.0-4983.ge7fb7b56.el9 ceph 22 M 2024-09-18T07:54:15.284 INFO:teuthology.orchestra.run.smithi007.stdout: ceph-selinux x86_64 2:19.3.0-4983.ge7fb7b56.el9 ceph 25 k 2024-09-18T07:54:15.284 INFO:teuthology.orchestra.run.smithi007.stdout: gperftools-libs x86_64 2.9.1-3.el9 epel 308 k 2024-09-18T07:54:15.284 INFO:teuthology.orchestra.run.smithi007.stdout: libarrow x86_64 9.0.0-11.el9 epel 4.4 M 2024-09-18T07:54:15.284 INFO:teuthology.orchestra.run.smithi007.stdout: libarrow-doc noarch 9.0.0-11.el9 epel 26 k 2024-09-18T07:54:15.284 INFO:teuthology.orchestra.run.smithi007.stdout: libcephfs2 x86_64 2:19.3.0-4983.ge7fb7b56.el9 ceph 755 k 2024-09-18T07:54:15.284 INFO:teuthology.orchestra.run.smithi007.stdout: libnbd x86_64 1.20.2-2.el9 appstream 171 k 2024-09-18T07:54:15.284 INFO:teuthology.orchestra.run.smithi007.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-09-18T07:54:15.284 INFO:teuthology.orchestra.run.smithi007.stdout: libpmemobj x86_64 1.12.1-1.el9 appstream 160 k 2024-09-18T07:54:15.285 INFO:teuthology.orchestra.run.smithi007.stdout: librabbitmq x86_64 0.11.0-7.el9 appstream 45 k 2024-09-18T07:54:15.285 INFO:teuthology.orchestra.run.smithi007.stdout: libradosstriper1 x86_64 2:19.3.0-4983.ge7fb7b56.el9 ceph 509 k 2024-09-18T07:54:15.285 INFO:teuthology.orchestra.run.smithi007.stdout: librdkafka x86_64 1.6.1-102.el9 appstream 662 k 2024-09-18T07:54:15.285 INFO:teuthology.orchestra.run.smithi007.stdout: librgw2 x86_64 2:19.3.0-4983.ge7fb7b56.el9 ceph 5.5 M 2024-09-18T07:54:15.285 INFO:teuthology.orchestra.run.smithi007.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-09-18T07:54:15.285 INFO:teuthology.orchestra.run.smithi007.stdout: lttng-ust x86_64 2.12.0-6.el9 appstream 292 k 2024-09-18T07:54:15.285 INFO:teuthology.orchestra.run.smithi007.stdout: parquet-libs x86_64 9.0.0-11.el9 epel 839 k 2024-09-18T07:54:15.285 INFO:teuthology.orchestra.run.smithi007.stdout: python3-ceph-argparse x86_64 2:19.3.0-4983.ge7fb7b56.el9 ceph 45 k 2024-09-18T07:54:15.285 INFO:teuthology.orchestra.run.smithi007.stdout: python3-ceph-common x86_64 2:19.3.0-4983.ge7fb7b56.el9 ceph 143 k 2024-09-18T07:54:15.285 INFO:teuthology.orchestra.run.smithi007.stdout: python3-cephfs x86_64 2:19.3.0-4983.ge7fb7b56.el9 ceph 161 k 2024-09-18T07:54:15.285 INFO:teuthology.orchestra.run.smithi007.stdout: python3-prettytable noarch 0.7.2-27.el9 appstream 42 k 2024-09-18T07:54:15.285 INFO:teuthology.orchestra.run.smithi007.stdout: python3-rados x86_64 2:19.3.0-4983.ge7fb7b56.el9 ceph 320 k 2024-09-18T07:54:15.285 INFO:teuthology.orchestra.run.smithi007.stdout: python3-rbd x86_64 2:19.3.0-4983.ge7fb7b56.el9 ceph 306 k 2024-09-18T07:54:15.286 INFO:teuthology.orchestra.run.smithi007.stdout: python3-rgw x86_64 2:19.3.0-4983.ge7fb7b56.el9 ceph 99 k 2024-09-18T07:54:15.286 INFO:teuthology.orchestra.run.smithi007.stdout: qatlib x86_64 24.02.0-1.el9 appstream 221 k 2024-09-18T07:54:15.286 INFO:teuthology.orchestra.run.smithi007.stdout: qatzip-libs x86_64 1.2.0-1.el9 appstream 47 k 2024-09-18T07:54:15.286 INFO:teuthology.orchestra.run.smithi007.stdout: re2 x86_64 1:20211101-20.el9 epel 191 k 2024-09-18T07:54:15.286 INFO:teuthology.orchestra.run.smithi007.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-09-18T07:54:15.286 INFO:teuthology.orchestra.run.smithi007.stdout:Installing weak dependencies: 2024-09-18T07:54:15.286 INFO:teuthology.orchestra.run.smithi007.stdout: qatlib-service x86_64 24.02.0-1.el9 appstream 36 k 2024-09-18T07:54:15.286 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-09-18T07:54:15.286 INFO:teuthology.orchestra.run.smithi007.stdout:Transaction Summary 2024-09-18T07:54:15.286 INFO:teuthology.orchestra.run.smithi007.stdout:================================================================================ 2024-09-18T07:54:15.286 INFO:teuthology.orchestra.run.smithi007.stdout:Install 31 Packages 2024-09-18T07:54:15.286 INFO:teuthology.orchestra.run.smithi007.stdout:Upgrade 2 Packages 2024-09-18T07:54:15.286 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-09-18T07:54:15.287 INFO:teuthology.orchestra.run.smithi007.stdout:Total download size: 62 M 2024-09-18T07:54:15.287 INFO:teuthology.orchestra.run.smithi007.stdout:Downloading Packages: 2024-09-18T07:54:15.895 INFO:teuthology.orchestra.run.smithi007.stdout:(1/33): ceph-base-19.3.0-4983.ge7fb7b56.el9.x86 11 MB/s | 5.5 MB 00:00 2024-09-18T07:54:15.962 INFO:teuthology.orchestra.run.smithi007.stdout:(2/33): ceph-selinux-19.3.0-4983.ge7fb7b56.el9. 377 kB/s | 25 kB 00:00 2024-09-18T07:54:16.070 INFO:teuthology.orchestra.run.smithi007.stdout:(3/33): libcephfs2-19.3.0-4983.ge7fb7b56.el9.x8 6.8 MB/s | 755 kB 00:00 2024-09-18T07:54:16.196 INFO:teuthology.orchestra.run.smithi007.stdout:(4/33): ceph-radosgw-19.3.0-4983.ge7fb7b56.el9. 14 MB/s | 11 MB 00:00 2024-09-18T07:54:16.229 INFO:teuthology.orchestra.run.smithi007.stdout:(5/33): libradosstriper1-19.3.0-4983.ge7fb7b56. 3.1 MB/s | 509 kB 00:00 2024-09-18T07:54:16.256 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : ceph-radosgw-2:19.3.0-4983.ge7fb7b56.el9.x86_64 33/35 2024-09-18T07:54:16.283 INFO:teuthology.orchestra.run.smithi050.stdout: Running scriptlet: ceph-radosgw-2:19.3.0-4983.ge7fb7b56.el9.x86_64 33/35 2024-09-18T07:54:16.283 INFO:teuthology.orchestra.run.smithi050.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-18T07:54:16.283 INFO:teuthology.orchestra.run.smithi050.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-09-18T07:54:16.283 INFO:teuthology.orchestra.run.smithi050.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-09-18T07:54:16.283 INFO:teuthology.orchestra.run.smithi050.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-09-18T07:54:16.283 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-09-18T07:54:16.285 INFO:teuthology.orchestra.run.smithi050.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 34/35 2024-09-18T07:54:16.288 INFO:teuthology.orchestra.run.smithi007.stdout:(6/33): python3-ceph-argparse-19.3.0-4983.ge7fb 772 kB/s | 45 kB 00:00 2024-09-18T07:54:16.387 INFO:teuthology.orchestra.run.smithi050.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 34/35 2024-09-18T07:54:16.387 INFO:teuthology.orchestra.run.smithi050.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 35/35 2024-09-18T07:54:16.388 INFO:teuthology.orchestra.run.smithi007.stdout:(7/33): python3-ceph-common-19.3.0-4983.ge7fb7b 1.4 MB/s | 143 kB 00:00 2024-09-18T07:54:16.464 INFO:teuthology.orchestra.run.smithi007.stdout:(8/33): librgw2-19.3.0-4983.ge7fb7b56.el9.x86_6 21 MB/s | 5.5 MB 00:00 2024-09-18T07:54:16.489 INFO:teuthology.orchestra.run.smithi007.stdout:(9/33): python3-cephfs-19.3.0-4983.ge7fb7b56.el 1.6 MB/s | 161 kB 00:00 2024-09-18T07:54:16.514 INFO:teuthology.orchestra.run.smithi007.stdout:(10/33): python3-rados-19.3.0-4983.ge7fb7b56.el 6.2 MB/s | 320 kB 00:00 2024-09-18T07:54:16.548 INFO:teuthology.orchestra.run.smithi007.stdout:(11/33): python3-rbd-19.3.0-4983.ge7fb7b56.el9. 5.1 MB/s | 306 kB 00:00 2024-09-18T07:54:16.573 INFO:teuthology.orchestra.run.smithi007.stdout:(12/33): python3-rgw-19.3.0-4983.ge7fb7b56.el9. 1.7 MB/s | 99 kB 00:00 2024-09-18T07:54:16.807 INFO:teuthology.orchestra.run.smithi007.stdout:(13/33): ceph-common-19.3.0-4983.ge7fb7b56.el9. 15 MB/s | 22 MB 00:01 2024-09-18T07:54:16.865 INFO:teuthology.orchestra.run.smithi007.stdout:(14/33): boost-program-options-1.75.0-8.el9.x86 337 kB/s | 107 kB 00:00 2024-09-18T07:54:16.907 INFO:teuthology.orchestra.run.smithi007.stdout:(15/33): librabbitmq-0.11.0-7.el9.x86_64.rpm 1.1 MB/s | 45 kB 00:00 2024-09-18T07:54:16.941 INFO:teuthology.orchestra.run.smithi007.stdout:(16/33): libnbd-1.20.2-2.el9.x86_64.rpm 464 kB/s | 171 kB 00:00 2024-09-18T07:54:16.966 INFO:teuthology.orchestra.run.smithi007.stdout:(17/33): libpmemobj-1.12.1-1.el9.x86_64.rpm 1.0 MB/s | 160 kB 00:00 2024-09-18T07:54:17.008 INFO:teuthology.orchestra.run.smithi007.stdout:(18/33): librdkafka-1.6.1-102.el9.x86_64.rpm 6.4 MB/s | 662 kB 00:00 2024-09-18T07:54:17.025 INFO:teuthology.orchestra.run.smithi007.stdout:(19/33): python3-prettytable-0.7.2-27.el9.noarc 724 kB/s | 42 kB 00:00 2024-09-18T07:54:17.050 INFO:teuthology.orchestra.run.smithi007.stdout:(20/33): lttng-ust-2.12.0-6.el9.x86_64.rpm 2.6 MB/s | 292 kB 00:00 2024-09-18T07:54:17.067 INFO:teuthology.orchestra.run.smithi007.stdout:(21/33): qatlib-service-24.02.0-1.el9.x86_64.rp 861 kB/s | 36 kB 00:00 2024-09-18T07:54:17.093 INFO:teuthology.orchestra.run.smithi007.stdout:(22/33): qatlib-24.02.0-1.el9.x86_64.rpm 2.6 MB/s | 221 kB 00:00 2024-09-18T07:54:17.118 INFO:teuthology.orchestra.run.smithi007.stdout:(23/33): qatzip-libs-1.2.0-1.el9.x86_64.rpm 702 kB/s | 47 kB 00:00 2024-09-18T07:54:17.185 INFO:teuthology.orchestra.run.smithi007.stdout:(24/33): libarrow-doc-9.0.0-11.el9.noarch.rpm 512 kB/s | 26 kB 00:00 2024-09-18T07:54:17.219 INFO:teuthology.orchestra.run.smithi007.stdout:(25/33): gperftools-libs-2.9.1-3.el9.x86_64.rpm 2.0 MB/s | 308 kB 00:00 2024-09-18T07:54:17.244 INFO:teuthology.orchestra.run.smithi007.stdout:(26/33): liboath-2.6.7-2.el9.x86_64.rpm 829 kB/s | 49 kB 00:00 2024-09-18T07:54:17.278 INFO:teuthology.orchestra.run.smithi007.stdout:(27/33): libunwind-1.6.2-1.el9.x86_64.rpm 1.1 MB/s | 67 kB 00:00 2024-09-18T07:54:17.328 INFO:teuthology.orchestra.run.smithi007.stdout:(28/33): re2-20211101-20.el9.x86_64.rpm 3.7 MB/s | 191 kB 00:00 2024-09-18T07:54:17.437 INFO:teuthology.orchestra.run.smithi007.stdout:(29/33): parquet-libs-9.0.0-11.el9.x86_64.rpm 4.3 MB/s | 839 kB 00:00 2024-09-18T07:54:17.553 INFO:teuthology.orchestra.run.smithi007.stdout:(30/33): thrift-0.15.0-2.el9.x86_64.rpm 7.0 MB/s | 1.6 MB 00:00 2024-09-18T07:54:17.637 INFO:teuthology.orchestra.run.smithi007.stdout:(31/33): librados2-19.3.0-4983.ge7fb7b56.el9.x8 17 MB/s | 3.5 MB 00:00 2024-09-18T07:54:17.641 INFO:teuthology.orchestra.run.smithi050.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 35/35 2024-09-18T07:54:17.641 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : ceph-base-2:19.3.0-4983.ge7fb7b56.el9.x86_64 1/35 2024-09-18T07:54:17.641 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : ceph-common-2:19.3.0-4983.ge7fb7b56.el9.x86_64 2/35 2024-09-18T07:54:17.641 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : ceph-radosgw-2:19.3.0-4983.ge7fb7b56.el9.x86_64 3/35 2024-09-18T07:54:17.641 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : ceph-selinux-2:19.3.0-4983.ge7fb7b56.el9.x86_64 4/35 2024-09-18T07:54:17.641 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : libcephfs2-2:19.3.0-4983.ge7fb7b56.el9.x86_64 5/35 2024-09-18T07:54:17.641 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : libradosstriper1-2:19.3.0-4983.ge7fb7b56.el9.x86_6 6/35 2024-09-18T07:54:17.641 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : librgw2-2:19.3.0-4983.ge7fb7b56.el9.x86_64 7/35 2024-09-18T07:54:17.642 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : python3-ceph-argparse-2:19.3.0-4983.ge7fb7b56.el9. 8/35 2024-09-18T07:54:17.642 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : python3-ceph-common-2:19.3.0-4983.ge7fb7b56.el9.x8 9/35 2024-09-18T07:54:17.642 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : python3-cephfs-2:19.3.0-4983.ge7fb7b56.el9.x86_64 10/35 2024-09-18T07:54:17.642 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : python3-rados-2:19.3.0-4983.ge7fb7b56.el9.x86_64 11/35 2024-09-18T07:54:17.642 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : python3-rbd-2:19.3.0-4983.ge7fb7b56.el9.x86_64 12/35 2024-09-18T07:54:17.642 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : python3-rgw-2:19.3.0-4983.ge7fb7b56.el9.x86_64 13/35 2024-09-18T07:54:17.642 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 14/35 2024-09-18T07:54:17.642 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : libnbd-1.20.2-2.el9.x86_64 15/35 2024-09-18T07:54:17.642 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 16/35 2024-09-18T07:54:17.642 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 17/35 2024-09-18T07:54:17.642 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 18/35 2024-09-18T07:54:17.642 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 19/35 2024-09-18T07:54:17.643 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 20/35 2024-09-18T07:54:17.643 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : qatlib-24.02.0-1.el9.x86_64 21/35 2024-09-18T07:54:17.643 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : qatlib-service-24.02.0-1.el9.x86_64 22/35 2024-09-18T07:54:17.643 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : qatzip-libs-1.2.0-1.el9.x86_64 23/35 2024-09-18T07:54:17.643 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : gperftools-libs-2.9.1-3.el9.x86_64 24/35 2024-09-18T07:54:17.643 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : libarrow-9.0.0-11.el9.x86_64 25/35 2024-09-18T07:54:17.643 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : libarrow-doc-9.0.0-11.el9.noarch 26/35 2024-09-18T07:54:17.643 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 27/35 2024-09-18T07:54:17.644 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 28/35 2024-09-18T07:54:17.644 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : parquet-libs-9.0.0-11.el9.x86_64 29/35 2024-09-18T07:54:17.644 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : re2-1:20211101-20.el9.x86_64 30/35 2024-09-18T07:54:17.644 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 31/35 2024-09-18T07:54:17.644 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : librados2-2:19.3.0-4983.ge7fb7b56.el9.x86_64 32/35 2024-09-18T07:54:17.644 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 33/35 2024-09-18T07:54:17.644 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : librbd1-2:19.3.0-4983.ge7fb7b56.el9.x86_64 34/35 2024-09-18T07:54:17.716 INFO:teuthology.orchestra.run.smithi007.stdout:(32/33): libarrow-9.0.0-11.el9.x86_64.rpm 7.1 MB/s | 4.4 MB 00:00 2024-09-18T07:54:17.799 INFO:teuthology.orchestra.run.smithi007.stdout:(33/33): librbd1-19.3.0-4983.ge7fb7b56.el9.x86_ 13 MB/s | 3.2 MB 00:00 2024-09-18T07:54:17.805 INFO:teuthology.orchestra.run.smithi007.stdout:-------------------------------------------------------------------------------- 2024-09-18T07:54:17.805 INFO:teuthology.orchestra.run.smithi007.stdout:Total 25 MB/s | 62 MB 00:02 2024-09-18T07:54:17.918 INFO:teuthology.orchestra.run.smithi007.stdout:Running transaction check 2024-09-18T07:54:17.961 INFO:teuthology.orchestra.run.smithi007.stdout:Transaction check succeeded. 2024-09-18T07:54:17.961 INFO:teuthology.orchestra.run.smithi007.stdout:Running transaction test 2024-09-18T07:54:18.490 INFO:teuthology.orchestra.run.smithi007.stdout:Transaction test succeeded. 2024-09-18T07:54:18.490 INFO:teuthology.orchestra.run.smithi007.stdout:Running transaction 2024-09-18T07:54:18.681 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 35/35 2024-09-18T07:54:18.681 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-09-18T07:54:18.681 INFO:teuthology.orchestra.run.smithi050.stdout:Upgraded: 2024-09-18T07:54:18.682 INFO:teuthology.orchestra.run.smithi050.stdout: librados2-2:19.3.0-4983.ge7fb7b56.el9.x86_64 2024-09-18T07:54:18.682 INFO:teuthology.orchestra.run.smithi050.stdout: librbd1-2:19.3.0-4983.ge7fb7b56.el9.x86_64 2024-09-18T07:54:18.682 INFO:teuthology.orchestra.run.smithi050.stdout:Installed: 2024-09-18T07:54:18.682 INFO:teuthology.orchestra.run.smithi050.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-09-18T07:54:18.682 INFO:teuthology.orchestra.run.smithi050.stdout: ceph-base-2:19.3.0-4983.ge7fb7b56.el9.x86_64 2024-09-18T07:54:18.682 INFO:teuthology.orchestra.run.smithi050.stdout: ceph-common-2:19.3.0-4983.ge7fb7b56.el9.x86_64 2024-09-18T07:54:18.682 INFO:teuthology.orchestra.run.smithi050.stdout: ceph-radosgw-2:19.3.0-4983.ge7fb7b56.el9.x86_64 2024-09-18T07:54:18.682 INFO:teuthology.orchestra.run.smithi050.stdout: ceph-selinux-2:19.3.0-4983.ge7fb7b56.el9.x86_64 2024-09-18T07:54:18.682 INFO:teuthology.orchestra.run.smithi050.stdout: gperftools-libs-2.9.1-3.el9.x86_64 2024-09-18T07:54:18.682 INFO:teuthology.orchestra.run.smithi050.stdout: libarrow-9.0.0-11.el9.x86_64 2024-09-18T07:54:18.682 INFO:teuthology.orchestra.run.smithi050.stdout: libarrow-doc-9.0.0-11.el9.noarch 2024-09-18T07:54:18.682 INFO:teuthology.orchestra.run.smithi050.stdout: libcephfs2-2:19.3.0-4983.ge7fb7b56.el9.x86_64 2024-09-18T07:54:18.683 INFO:teuthology.orchestra.run.smithi050.stdout: libnbd-1.20.2-2.el9.x86_64 2024-09-18T07:54:18.683 INFO:teuthology.orchestra.run.smithi050.stdout: liboath-2.6.7-2.el9.x86_64 2024-09-18T07:54:18.683 INFO:teuthology.orchestra.run.smithi050.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-09-18T07:54:18.683 INFO:teuthology.orchestra.run.smithi050.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-09-18T07:54:18.683 INFO:teuthology.orchestra.run.smithi050.stdout: libradosstriper1-2:19.3.0-4983.ge7fb7b56.el9.x86_64 2024-09-18T07:54:18.683 INFO:teuthology.orchestra.run.smithi050.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-09-18T07:54:18.683 INFO:teuthology.orchestra.run.smithi050.stdout: librgw2-2:19.3.0-4983.ge7fb7b56.el9.x86_64 2024-09-18T07:54:18.683 INFO:teuthology.orchestra.run.smithi050.stdout: libunwind-1.6.2-1.el9.x86_64 2024-09-18T07:54:18.683 INFO:teuthology.orchestra.run.smithi050.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-09-18T07:54:18.683 INFO:teuthology.orchestra.run.smithi050.stdout: parquet-libs-9.0.0-11.el9.x86_64 2024-09-18T07:54:18.683 INFO:teuthology.orchestra.run.smithi050.stdout: python3-ceph-argparse-2:19.3.0-4983.ge7fb7b56.el9.x86_64 2024-09-18T07:54:18.683 INFO:teuthology.orchestra.run.smithi050.stdout: python3-ceph-common-2:19.3.0-4983.ge7fb7b56.el9.x86_64 2024-09-18T07:54:18.683 INFO:teuthology.orchestra.run.smithi050.stdout: python3-cephfs-2:19.3.0-4983.ge7fb7b56.el9.x86_64 2024-09-18T07:54:18.684 INFO:teuthology.orchestra.run.smithi050.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-09-18T07:54:18.684 INFO:teuthology.orchestra.run.smithi050.stdout: python3-rados-2:19.3.0-4983.ge7fb7b56.el9.x86_64 2024-09-18T07:54:18.684 INFO:teuthology.orchestra.run.smithi050.stdout: python3-rbd-2:19.3.0-4983.ge7fb7b56.el9.x86_64 2024-09-18T07:54:18.684 INFO:teuthology.orchestra.run.smithi050.stdout: python3-rgw-2:19.3.0-4983.ge7fb7b56.el9.x86_64 2024-09-18T07:54:18.684 INFO:teuthology.orchestra.run.smithi050.stdout: qatlib-24.02.0-1.el9.x86_64 2024-09-18T07:54:18.684 INFO:teuthology.orchestra.run.smithi050.stdout: qatlib-service-24.02.0-1.el9.x86_64 2024-09-18T07:54:18.684 INFO:teuthology.orchestra.run.smithi050.stdout: qatzip-libs-1.2.0-1.el9.x86_64 2024-09-18T07:54:18.684 INFO:teuthology.orchestra.run.smithi050.stdout: re2-1:20211101-20.el9.x86_64 2024-09-18T07:54:18.684 INFO:teuthology.orchestra.run.smithi050.stdout: thrift-0.15.0-2.el9.x86_64 2024-09-18T07:54:18.684 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-09-18T07:54:18.684 INFO:teuthology.orchestra.run.smithi050.stdout:Complete! 2024-09-18T07:54:19.034 DEBUG:teuthology.orchestra.run.smithi050:> sudo yum -y install ceph-test 2024-09-18T07:54:19.640 INFO:teuthology.orchestra.run.smithi050.stdout:Last metadata expiration check: 0:00:29 ago on Wed 18 Sep 2024 07:53:50 AM UTC. 2024-09-18T07:54:19.779 INFO:teuthology.orchestra.run.smithi050.stdout:Dependencies resolved. 2024-09-18T07:54:19.779 INFO:teuthology.orchestra.run.smithi050.stdout:================================================================================ 2024-09-18T07:54:19.779 INFO:teuthology.orchestra.run.smithi050.stdout: Package Arch Version Repository Size 2024-09-18T07:54:19.779 INFO:teuthology.orchestra.run.smithi050.stdout:================================================================================ 2024-09-18T07:54:19.779 INFO:teuthology.orchestra.run.smithi050.stdout:Installing: 2024-09-18T07:54:19.779 INFO:teuthology.orchestra.run.smithi050.stdout: ceph-test x86_64 2:19.3.0-4983.ge7fb7b56.el9 ceph 46 M 2024-09-18T07:54:19.780 INFO:teuthology.orchestra.run.smithi050.stdout:Installing dependencies: 2024-09-18T07:54:19.780 INFO:teuthology.orchestra.run.smithi050.stdout: libcephsqlite x86_64 2:19.3.0-4983.ge7fb7b56.el9 ceph 161 k 2024-09-18T07:54:19.780 INFO:teuthology.orchestra.run.smithi050.stdout: socat x86_64 1.7.4.1-6.el9 appstream 304 k 2024-09-18T07:54:19.780 INFO:teuthology.orchestra.run.smithi050.stdout: xmlstarlet x86_64 1.6.1-20.el9 appstream 64 k 2024-09-18T07:54:19.780 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-09-18T07:54:19.780 INFO:teuthology.orchestra.run.smithi050.stdout:Transaction Summary 2024-09-18T07:54:19.780 INFO:teuthology.orchestra.run.smithi050.stdout:================================================================================ 2024-09-18T07:54:19.780 INFO:teuthology.orchestra.run.smithi050.stdout:Install 4 Packages 2024-09-18T07:54:19.780 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-09-18T07:54:19.781 INFO:teuthology.orchestra.run.smithi050.stdout:Total download size: 46 M 2024-09-18T07:54:19.781 INFO:teuthology.orchestra.run.smithi050.stdout:Installed size: 201 M 2024-09-18T07:54:19.781 INFO:teuthology.orchestra.run.smithi050.stdout:Downloading Packages: 2024-09-18T07:54:19.951 INFO:teuthology.orchestra.run.smithi050.stdout:(1/4): socat-1.7.4.1-6.el9.x86_64.rpm 2.0 MB/s | 304 kB 00:00 2024-09-18T07:54:19.986 INFO:teuthology.orchestra.run.smithi007.stdout: Preparing : 1/1 2024-09-18T07:54:19.992 INFO:teuthology.orchestra.run.smithi050.stdout:(2/4): xmlstarlet-1.6.1-20.el9.x86_64.rpm 1.5 MB/s | 64 kB 00:00 2024-09-18T07:54:20.034 INFO:teuthology.orchestra.run.smithi050.stdout:(3/4): libcephsqlite-19.3.0-4983.ge7fb7b56.el9. 698 kB/s | 161 kB 00:00 2024-09-18T07:54:20.064 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 1/35 2024-09-18T07:54:20.104 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 2/35 2024-09-18T07:54:20.139 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 3/35 2024-09-18T07:54:20.161 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : qatlib-service-24.02.0-1.el9.x86_64 4/35 2024-09-18T07:54:20.188 INFO:teuthology.orchestra.run.smithi007.stdout: Running scriptlet: qatlib-service-24.02.0-1.el9.x86_64 4/35 2024-09-18T07:54:20.381 INFO:teuthology.orchestra.run.smithi007.stdout: Running scriptlet: qatlib-24.02.0-1.el9.x86_64 5/35 2024-09-18T07:54:20.434 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : qatlib-24.02.0-1.el9.x86_64 5/35 2024-09-18T07:54:20.843 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : thrift-0.15.0-2.el9.x86_64 6/35 2024-09-18T07:54:20.898 INFO:teuthology.orchestra.run.smithi007.stdout: Upgrading : librados2-2:19.3.0-4983.ge7fb7b56.el9.x86_64 7/35 2024-09-18T07:54:20.984 INFO:teuthology.orchestra.run.smithi007.stdout: Running scriptlet: librados2-2:19.3.0-4983.ge7fb7b56.el9.x86_64 7/35 2024-09-18T07:54:21.082 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : python3-rados-2:19.3.0-4983.ge7fb7b56.el9.x86_64 8/35 2024-09-18T07:54:21.108 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : libcephfs2-2:19.3.0-4983.ge7fb7b56.el9.x86_64 9/35 2024-09-18T07:54:21.163 INFO:teuthology.orchestra.run.smithi007.stdout: Running scriptlet: libcephfs2-2:19.3.0-4983.ge7fb7b56.el9.x86_64 9/35 2024-09-18T07:54:21.192 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : liboath-2.6.7-2.el9.x86_64 10/35 2024-09-18T07:54:21.239 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : python3-ceph-argparse-2:19.3.0-4983.ge7fb7b56.el9. 11/35 2024-09-18T07:54:21.268 INFO:teuthology.orchestra.run.smithi050.stdout:(4/4): ceph-test-19.3.0-4983.ge7fb7b56.el9.x86_ 31 MB/s | 46 MB 00:01 2024-09-18T07:54:21.270 INFO:teuthology.orchestra.run.smithi050.stdout:-------------------------------------------------------------------------------- 2024-09-18T07:54:21.270 INFO:teuthology.orchestra.run.smithi050.stdout:Total 31 MB/s | 46 MB 00:01 2024-09-18T07:54:21.299 INFO:teuthology.orchestra.run.smithi050.stdout:Running transaction check 2024-09-18T07:54:21.322 INFO:teuthology.orchestra.run.smithi050.stdout:Transaction check succeeded. 2024-09-18T07:54:21.322 INFO:teuthology.orchestra.run.smithi050.stdout:Running transaction test 2024-09-18T07:54:21.324 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : python3-cephfs-2:19.3.0-4983.ge7fb7b56.el9.x86_64 12/35 2024-09-18T07:54:21.353 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : libradosstriper1-2:19.3.0-4983.ge7fb7b56.el9.x86_6 13/35 2024-09-18T07:54:21.409 INFO:teuthology.orchestra.run.smithi007.stdout: Running scriptlet: libradosstriper1-2:19.3.0-4983.ge7fb7b56.el9.x86_6 13/35 2024-09-18T07:54:21.443 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : qatzip-libs-1.2.0-1.el9.x86_64 14/35 2024-09-18T07:54:21.556 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : re2-1:20211101-20.el9.x86_64 15/35 2024-09-18T07:54:21.594 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : libarrow-9.0.0-11.el9.x86_64 16/35 2024-09-18T07:54:21.645 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 17/35 2024-09-18T07:54:21.676 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : gperftools-libs-2.9.1-3.el9.x86_64 18/35 2024-09-18T07:54:21.705 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : libarrow-doc-9.0.0-11.el9.noarch 19/35 2024-09-18T07:54:21.706 INFO:teuthology.orchestra.run.smithi050.stdout:Transaction test succeeded. 2024-09-18T07:54:21.707 INFO:teuthology.orchestra.run.smithi050.stdout:Running transaction 2024-09-18T07:54:21.738 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 20/35 2024-09-18T07:54:21.783 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 21/35 2024-09-18T07:54:22.127 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : libnbd-1.20.2-2.el9.x86_64 22/35 2024-09-18T07:54:22.236 INFO:teuthology.orchestra.run.smithi050.stdout: Preparing : 1/1 2024-09-18T07:54:22.259 INFO:teuthology.orchestra.run.smithi007.stdout: Upgrading : librbd1-2:19.3.0-4983.ge7fb7b56.el9.x86_64 23/35 2024-09-18T07:54:22.333 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 1/4 2024-09-18T07:54:22.357 INFO:teuthology.orchestra.run.smithi007.stdout: Running scriptlet: librbd1-2:19.3.0-4983.ge7fb7b56.el9.x86_64 23/35 2024-09-18T07:54:22.384 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : socat-1.7.4.1-6.el9.x86_64 2/4 2024-09-18T07:54:22.398 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : python3-rbd-2:19.3.0-4983.ge7fb7b56.el9.x86_64 24/35 2024-09-18T07:54:22.428 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : libcephsqlite-2:19.3.0-4983.ge7fb7b56.el9.x86_64 3/4 2024-09-18T07:54:22.435 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 25/35 2024-09-18T07:54:22.952 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : parquet-libs-9.0.0-11.el9.x86_64 26/35 2024-09-18T07:54:22.979 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : librgw2-2:19.3.0-4983.ge7fb7b56.el9.x86_64 27/35 2024-09-18T07:54:23.051 INFO:teuthology.orchestra.run.smithi007.stdout: Running scriptlet: librgw2-2:19.3.0-4983.ge7fb7b56.el9.x86_64 27/35 2024-09-18T07:54:23.157 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : python3-rgw-2:19.3.0-4983.ge7fb7b56.el9.x86_64 28/35 2024-09-18T07:54:23.198 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : python3-ceph-common-2:19.3.0-4983.ge7fb7b56.el9.x8 29/35 2024-09-18T07:54:25.658 INFO:teuthology.orchestra.run.smithi007.stdout: Running scriptlet: ceph-common-2:19.3.0-4983.ge7fb7b56.el9.x86_64 30/35 2024-09-18T07:54:25.695 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : ceph-common-2:19.3.0-4983.ge7fb7b56.el9.x86_64 30/35 2024-09-18T07:54:26.295 INFO:teuthology.orchestra.run.smithi007.stdout: Running scriptlet: ceph-common-2:19.3.0-4983.ge7fb7b56.el9.x86_64 30/35 2024-09-18T07:54:26.331 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : ceph-base-2:19.3.0-4983.ge7fb7b56.el9.x86_64 31/35 2024-09-18T07:54:26.419 INFO:teuthology.orchestra.run.smithi007.stdout: Running scriptlet: ceph-base-2:19.3.0-4983.ge7fb7b56.el9.x86_64 31/35 2024-09-18T07:54:26.419 INFO:teuthology.orchestra.run.smithi007.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-09-18T07:54:26.419 INFO:teuthology.orchestra.run.smithi007.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-09-18T07:54:26.419 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-09-18T07:54:26.466 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : ceph-selinux-2:19.3.0-4983.ge7fb7b56.el9.x86_64 32/35 2024-09-18T07:54:27.032 INFO:teuthology.orchestra.run.smithi050.stdout: Running scriptlet: libcephsqlite-2:19.3.0-4983.ge7fb7b56.el9.x86_64 3/4 2024-09-18T07:54:27.133 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : ceph-test-2:19.3.0-4983.ge7fb7b56.el9.x86_64 4/4 2024-09-18T07:54:29.535 INFO:teuthology.orchestra.run.smithi050.stdout: Running scriptlet: ceph-test-2:19.3.0-4983.ge7fb7b56.el9.x86_64 4/4 2024-09-18T07:54:29.535 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : ceph-test-2:19.3.0-4983.ge7fb7b56.el9.x86_64 1/4 2024-09-18T07:54:29.535 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : libcephsqlite-2:19.3.0-4983.ge7fb7b56.el9.x86_64 2/4 2024-09-18T07:54:29.535 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : socat-1.7.4.1-6.el9.x86_64 3/4 2024-09-18T07:54:29.934 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 4/4 2024-09-18T07:54:29.934 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-09-18T07:54:29.934 INFO:teuthology.orchestra.run.smithi050.stdout:Installed: 2024-09-18T07:54:29.934 INFO:teuthology.orchestra.run.smithi050.stdout: ceph-test-2:19.3.0-4983.ge7fb7b56.el9.x86_64 2024-09-18T07:54:29.934 INFO:teuthology.orchestra.run.smithi050.stdout: libcephsqlite-2:19.3.0-4983.ge7fb7b56.el9.x86_64 2024-09-18T07:54:29.935 INFO:teuthology.orchestra.run.smithi050.stdout: socat-1.7.4.1-6.el9.x86_64 2024-09-18T07:54:29.935 INFO:teuthology.orchestra.run.smithi050.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-09-18T07:54:29.935 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-09-18T07:54:29.935 INFO:teuthology.orchestra.run.smithi050.stdout:Complete! 2024-09-18T07:54:30.190 DEBUG:teuthology.orchestra.run.smithi050:> sudo yum -y install ceph 2024-09-18T07:54:30.805 INFO:teuthology.orchestra.run.smithi050.stdout:Last metadata expiration check: 0:00:40 ago on Wed 18 Sep 2024 07:53:50 AM UTC. 2024-09-18T07:54:30.949 INFO:teuthology.orchestra.run.smithi050.stdout:Dependencies resolved. 2024-09-18T07:54:30.951 INFO:teuthology.orchestra.run.smithi050.stdout:================================================================================ 2024-09-18T07:54:30.951 INFO:teuthology.orchestra.run.smithi050.stdout: Package Arch Version Repository Size 2024-09-18T07:54:30.951 INFO:teuthology.orchestra.run.smithi050.stdout:================================================================================ 2024-09-18T07:54:30.951 INFO:teuthology.orchestra.run.smithi050.stdout:Installing: 2024-09-18T07:54:30.952 INFO:teuthology.orchestra.run.smithi050.stdout: ceph x86_64 2:19.3.0-4983.ge7fb7b56.el9 ceph 6.6 k 2024-09-18T07:54:30.952 INFO:teuthology.orchestra.run.smithi050.stdout:Installing dependencies: 2024-09-18T07:54:30.952 INFO:teuthology.orchestra.run.smithi050.stdout: ceph-mds x86_64 2:19.3.0-4983.ge7fb7b56.el9 ceph 2.4 M 2024-09-18T07:54:30.952 INFO:teuthology.orchestra.run.smithi050.stdout: ceph-mgr x86_64 2:19.3.0-4983.ge7fb7b56.el9 ceph 1.5 M 2024-09-18T07:54:30.952 INFO:teuthology.orchestra.run.smithi050.stdout: ceph-mgr-modules-core noarch 2:19.3.0-4983.ge7fb7b56.el9 ceph-noarch 291 k 2024-09-18T07:54:30.952 INFO:teuthology.orchestra.run.smithi050.stdout: ceph-mon x86_64 2:19.3.0-4983.ge7fb7b56.el9 ceph 4.8 M 2024-09-18T07:54:30.952 INFO:teuthology.orchestra.run.smithi050.stdout: ceph-osd x86_64 2:19.3.0-4983.ge7fb7b56.el9 ceph 17 M 2024-09-18T07:54:30.952 INFO:teuthology.orchestra.run.smithi050.stdout: lua x86_64 5.4.4-4.el9 appstream 188 k 2024-09-18T07:54:30.952 INFO:teuthology.orchestra.run.smithi050.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2024-09-18T07:54:30.952 INFO:teuthology.orchestra.run.smithi050.stdout: luarocks noarch 3.9.2-5.el9 epel 151 k 2024-09-18T07:54:30.952 INFO:teuthology.orchestra.run.smithi050.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-09-18T07:54:30.952 INFO:teuthology.orchestra.run.smithi050.stdout: python3-cffi x86_64 1.14.5-5.el9 baseos 253 k 2024-09-18T07:54:30.952 INFO:teuthology.orchestra.run.smithi050.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-09-18T07:54:30.953 INFO:teuthology.orchestra.run.smithi050.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-09-18T07:54:30.953 INFO:teuthology.orchestra.run.smithi050.stdout: python3-cryptography x86_64 36.0.1-4.el9 baseos 1.2 M 2024-09-18T07:54:30.953 INFO:teuthology.orchestra.run.smithi050.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-09-18T07:54:30.953 INFO:teuthology.orchestra.run.smithi050.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-09-18T07:54:30.953 INFO:teuthology.orchestra.run.smithi050.stdout: python3-jaraco-collections 2024-09-18T07:54:30.953 INFO:teuthology.orchestra.run.smithi050.stdout: noarch 3.0.0-8.el9 epel 23 k 2024-09-18T07:54:30.953 INFO:teuthology.orchestra.run.smithi050.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-09-18T07:54:30.953 INFO:teuthology.orchestra.run.smithi050.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-09-18T07:54:30.953 INFO:teuthology.orchestra.run.smithi050.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-09-18T07:54:30.953 INFO:teuthology.orchestra.run.smithi050.stdout: python3-mako noarch 1.1.4-6.el9 appstream 172 k 2024-09-18T07:54:30.953 INFO:teuthology.orchestra.run.smithi050.stdout: python3-markupsafe x86_64 1.1.1-12.el9 appstream 35 k 2024-09-18T07:54:30.953 INFO:teuthology.orchestra.run.smithi050.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-09-18T07:54:30.953 INFO:teuthology.orchestra.run.smithi050.stdout: python3-packaging noarch 20.9-5.el9 appstream 77 k 2024-09-18T07:54:30.954 INFO:teuthology.orchestra.run.smithi050.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-09-18T07:54:30.954 INFO:teuthology.orchestra.run.smithi050.stdout: python3-ply noarch 3.11-14.el9 baseos 106 k 2024-09-18T07:54:30.954 INFO:teuthology.orchestra.run.smithi050.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-09-18T07:54:30.954 INFO:teuthology.orchestra.run.smithi050.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-09-18T07:54:30.954 INFO:teuthology.orchestra.run.smithi050.stdout: python3-pycparser noarch 2.20-6.el9 baseos 135 k 2024-09-18T07:54:30.954 INFO:teuthology.orchestra.run.smithi050.stdout: python3-pytz noarch 2021.1-5.el9 appstream 51 k 2024-09-18T07:54:30.954 INFO:teuthology.orchestra.run.smithi050.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-09-18T07:54:30.954 INFO:teuthology.orchestra.run.smithi050.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-09-18T07:54:30.954 INFO:teuthology.orchestra.run.smithi050.stdout: python3-toml noarch 0.10.2-6.el9 appstream 42 k 2024-09-18T07:54:30.954 INFO:teuthology.orchestra.run.smithi050.stdout: python3-urllib3 noarch 1.26.5-6.el9 baseos 215 k 2024-09-18T07:54:30.954 INFO:teuthology.orchestra.run.smithi050.stdout: python3-webob noarch 1.8.8-2.el9 epel 230 k 2024-09-18T07:54:30.955 INFO:teuthology.orchestra.run.smithi050.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-09-18T07:54:30.955 INFO:teuthology.orchestra.run.smithi050.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-09-18T07:54:30.955 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-09-18T07:54:30.955 INFO:teuthology.orchestra.run.smithi050.stdout:Transaction Summary 2024-09-18T07:54:30.955 INFO:teuthology.orchestra.run.smithi050.stdout:================================================================================ 2024-09-18T07:54:30.955 INFO:teuthology.orchestra.run.smithi050.stdout:Install 37 Packages 2024-09-18T07:54:30.955 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-09-18T07:54:30.956 INFO:teuthology.orchestra.run.smithi050.stdout:Total download size: 31 M 2024-09-18T07:54:30.956 INFO:teuthology.orchestra.run.smithi050.stdout:Installed size: 109 M 2024-09-18T07:54:30.956 INFO:teuthology.orchestra.run.smithi050.stdout:Downloading Packages: 2024-09-18T07:54:31.873 INFO:teuthology.orchestra.run.smithi050.stdout:(1/37): ceph-19.3.0-4983.ge7fb7b56.el9.x86_64.r 51 kB/s | 6.6 kB 00:00 2024-09-18T07:54:32.106 INFO:teuthology.orchestra.run.smithi050.stdout:(2/37): ceph-mds-19.3.0-4983.ge7fb7b56.el9.x86_ 6.6 MB/s | 2.4 MB 00:00 2024-09-18T07:54:32.148 INFO:teuthology.orchestra.run.smithi050.stdout:(3/37): ceph-mgr-19.3.0-4983.ge7fb7b56.el9.x86_ 3.8 MB/s | 1.5 MB 00:00 2024-09-18T07:54:32.240 INFO:teuthology.orchestra.run.smithi050.stdout:(4/37): ceph-mon-19.3.0-4983.ge7fb7b56.el9.x86_ 13 MB/s | 4.8 MB 00:00 2024-09-18T07:54:32.265 INFO:teuthology.orchestra.run.smithi050.stdout:(5/37): ceph-mgr-modules-core-19.3.0-4983.ge7fb 2.4 MB/s | 291 kB 00:00 2024-09-18T07:54:32.491 INFO:teuthology.orchestra.run.smithi050.stdout:(6/37): python3-cffi-1.14.5-5.el9.x86_64.rpm 1.0 MB/s | 253 kB 00:00 2024-09-18T07:54:32.674 INFO:teuthology.orchestra.run.smithi050.stdout:(7/37): ceph-osd-19.3.0-4983.ge7fb7b56.el9.x86_ 30 MB/s | 17 MB 00:00 2024-09-18T07:54:32.699 INFO:teuthology.orchestra.run.smithi050.stdout:(8/37): python3-ply-3.11-14.el9.noarch.rpm 511 kB/s | 106 kB 00:00 2024-09-18T07:54:32.733 INFO:teuthology.orchestra.run.smithi050.stdout:(9/37): python3-cryptography-36.0.1-4.el9.x86_6 2.6 MB/s | 1.2 MB 00:00 2024-09-18T07:54:32.766 INFO:teuthology.orchestra.run.smithi050.stdout:(10/37): python3-requests-2.25.1-8.el9.noarch.r 1.8 MB/s | 125 kB 00:00 2024-09-18T07:54:32.800 INFO:teuthology.orchestra.run.smithi050.stdout:(11/37): python3-urllib3-1.26.5-6.el9.noarch.rp 3.1 MB/s | 215 kB 00:00 2024-09-18T07:54:32.901 INFO:teuthology.orchestra.run.smithi050.stdout:(12/37): python3-pycparser-2.20-6.el9.noarch.rp 598 kB/s | 135 kB 00:00 2024-09-18T07:54:32.967 INFO:teuthology.orchestra.run.smithi050.stdout:(13/37): lua-5.4.4-4.el9.x86_64.rpm 939 kB/s | 188 kB 00:00 2024-09-18T07:54:33.001 INFO:teuthology.orchestra.run.smithi050.stdout:(14/37): python3-mako-1.1.4-6.el9.noarch.rpm 858 kB/s | 172 kB 00:00 2024-09-18T07:54:33.035 INFO:teuthology.orchestra.run.smithi050.stdout:(15/37): python3-markupsafe-1.1.1-12.el9.x86_64 260 kB/s | 35 kB 00:00 2024-09-18T07:54:33.060 INFO:teuthology.orchestra.run.smithi050.stdout:(16/37): python3-packaging-20.9-5.el9.noarch.rp 836 kB/s | 77 kB 00:00 2024-09-18T07:54:33.085 INFO:teuthology.orchestra.run.smithi050.stdout:(17/37): python3-pytz-2021.1-5.el9.noarch.rpm 609 kB/s | 51 kB 00:00 2024-09-18T07:54:33.128 INFO:teuthology.orchestra.run.smithi050.stdout:(18/37): python3-toml-0.10.2-6.el9.noarch.rpm 451 kB/s | 42 kB 00:00 2024-09-18T07:54:33.170 INFO:teuthology.orchestra.run.smithi050.stdout:(19/37): luarocks-3.9.2-5.el9.noarch.rpm 1.8 MB/s | 151 kB 00:00 2024-09-18T07:54:33.195 INFO:teuthology.orchestra.run.smithi050.stdout:(20/37): python3-bcrypt-3.2.2-1.el9.x86_64.rpm 671 kB/s | 43 kB 00:00 2024-09-18T07:54:33.220 INFO:teuthology.orchestra.run.smithi050.stdout:(21/37): lua-devel-5.4.4-4.el9.x86_64.rpm 166 kB/s | 22 kB 00:00 2024-09-18T07:54:33.254 INFO:teuthology.orchestra.run.smithi050.stdout:(22/37): python3-cheroot-8.6.0-4.el9.noarch.rpm 2.1 MB/s | 172 kB 00:00 2024-09-18T07:54:33.279 INFO:teuthology.orchestra.run.smithi050.stdout:(23/37): python3-jaraco-classes-3.2.1-5.el9.noa 774 kB/s | 18 kB 00:00 2024-09-18T07:54:33.304 INFO:teuthology.orchestra.run.smithi050.stdout:(24/37): python3-jaraco-8.2.1-3.el9.noarch.rpm 127 kB/s | 11 kB 00:00 2024-09-18T07:54:33.329 INFO:teuthology.orchestra.run.smithi050.stdout:(25/37): python3-jaraco-collections-3.0.0-8.el9 461 kB/s | 23 kB 00:00 2024-09-18T07:54:33.363 INFO:teuthology.orchestra.run.smithi050.stdout:(26/37): python3-cherrypy-18.6.1-2.el9.noarch.r 2.1 MB/s | 358 kB 00:00 2024-09-18T07:54:33.388 INFO:teuthology.orchestra.run.smithi050.stdout:(27/37): python3-jaraco-functools-3.5.0-2.el9.n 232 kB/s | 19 kB 00:00 2024-09-18T07:54:33.413 INFO:teuthology.orchestra.run.smithi050.stdout:(28/37): python3-jaraco-text-3.2.0-6.el9.noarch 235 kB/s | 20 kB 00:00 2024-09-18T07:54:33.439 INFO:teuthology.orchestra.run.smithi050.stdout:(29/37): python3-logutils-0.3.5-21.el9.noarch.r 925 kB/s | 46 kB 00:00 2024-09-18T07:54:33.464 INFO:teuthology.orchestra.run.smithi050.stdout:(30/37): python3-more-itertools-8.12.0-2.el9.no 1.0 MB/s | 79 kB 00:00 2024-09-18T07:54:33.489 INFO:teuthology.orchestra.run.smithi050.stdout:(31/37): python3-portend-3.1.0-2.el9.noarch.rpm 328 kB/s | 16 kB 00:00 2024-09-18T07:54:33.514 INFO:teuthology.orchestra.run.smithi050.stdout:(32/37): python3-tempora-5.0.0-2.el9.noarch.rpm 1.4 MB/s | 36 kB 00:00 2024-09-18T07:54:33.540 INFO:teuthology.orchestra.run.smithi050.stdout:(33/37): python3-pyOpenSSL-21.0.0-1.el9.noarch. 1.2 MB/s | 90 kB 00:00 2024-09-18T07:54:33.565 INFO:teuthology.orchestra.run.smithi050.stdout:(34/37): python3-pecan-1.4.2-3.el9.noarch.rpm 1.8 MB/s | 272 kB 00:00 2024-09-18T07:54:33.591 INFO:teuthology.orchestra.run.smithi050.stdout:(35/37): python3-webob-1.8.8-2.el9.noarch.rpm 3.0 MB/s | 230 kB 00:00 2024-09-18T07:54:33.616 INFO:teuthology.orchestra.run.smithi050.stdout:(36/37): python3-zc-lockfile-2.0-10.el9.noarch. 399 kB/s | 20 kB 00:00 2024-09-18T07:54:33.642 INFO:teuthology.orchestra.run.smithi050.stdout:(37/37): python3-werkzeug-2.0.3-3.el9.1.noarch. 4.1 MB/s | 427 kB 00:00 2024-09-18T07:54:33.646 INFO:teuthology.orchestra.run.smithi050.stdout:-------------------------------------------------------------------------------- 2024-09-18T07:54:33.646 INFO:teuthology.orchestra.run.smithi050.stdout:Total 11 MB/s | 31 MB 00:02 2024-09-18T07:54:33.799 INFO:teuthology.orchestra.run.smithi050.stdout:Running transaction check 2024-09-18T07:54:33.842 INFO:teuthology.orchestra.run.smithi050.stdout:Transaction check succeeded. 2024-09-18T07:54:33.842 INFO:teuthology.orchestra.run.smithi050.stdout:Running transaction test 2024-09-18T07:54:34.325 INFO:teuthology.orchestra.run.smithi050.stdout:Transaction test succeeded. 2024-09-18T07:54:34.325 INFO:teuthology.orchestra.run.smithi050.stdout:Running transaction 2024-09-18T07:54:35.649 INFO:teuthology.orchestra.run.smithi050.stdout: Preparing : 1/1 2024-09-18T07:54:35.744 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 1/37 2024-09-18T07:54:35.780 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 2/37 2024-09-18T07:54:35.813 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : lua-5.4.4-4.el9.x86_64 3/37 2024-09-18T07:54:35.889 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 4/37 2024-09-18T07:54:35.937 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : luarocks-3.9.2-5.el9.noarch 5/37 2024-09-18T07:54:35.970 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 6/37 2024-09-18T07:54:36.066 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 7/37 2024-09-18T07:54:36.151 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 8/37 2024-09-18T07:54:36.198 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : python3-webob-1.8.8-2.el9.noarch 9/37 2024-09-18T07:54:36.233 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 10/37 2024-09-18T07:54:36.262 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 11/37 2024-09-18T07:54:36.297 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 12/37 2024-09-18T07:54:36.330 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 13/37 2024-09-18T07:54:36.372 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 14/37 2024-09-18T07:54:36.423 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 15/37 2024-09-18T07:54:36.457 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 16/37 2024-09-18T07:54:36.506 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 17/37 2024-09-18T07:54:36.547 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : python3-packaging-20.9-5.el9.noarch 18/37 2024-09-18T07:54:36.623 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 19/37 2024-09-18T07:54:36.783 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 20/37 2024-09-18T07:54:36.865 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 21/37 2024-09-18T07:54:36.926 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : python3-urllib3-1.26.5-6.el9.noarch 22/37 2024-09-18T07:54:36.971 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 23/37 2024-09-18T07:54:37.029 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : python3-ply-3.11-14.el9.noarch 24/37 2024-09-18T07:54:37.094 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 25/37 2024-09-18T07:54:37.207 INFO:teuthology.orchestra.run.smithi007.stdout: Running scriptlet: ceph-selinux-2:19.3.0-4983.ge7fb7b56.el9.x86_64 32/35 2024-09-18T07:54:37.207 INFO:teuthology.orchestra.run.smithi007.stdout:skipping the directory /sys 2024-09-18T07:54:37.207 INFO:teuthology.orchestra.run.smithi007.stdout:skipping the directory /proc 2024-09-18T07:54:37.207 INFO:teuthology.orchestra.run.smithi007.stdout:skipping the directory /mnt 2024-09-18T07:54:37.207 INFO:teuthology.orchestra.run.smithi007.stdout:skipping the directory /var/tmp 2024-09-18T07:54:37.208 INFO:teuthology.orchestra.run.smithi007.stdout:skipping the directory /home 2024-09-18T07:54:37.208 INFO:teuthology.orchestra.run.smithi007.stdout:skipping the directory /root 2024-09-18T07:54:37.208 INFO:teuthology.orchestra.run.smithi007.stdout:skipping the directory /tmp 2024-09-18T07:54:37.208 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-09-18T07:54:37.289 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 26/37 2024-09-18T07:54:37.342 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 27/37 2024-09-18T07:54:37.423 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 28/37 2024-09-18T07:54:37.521 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 29/37 2024-09-18T07:54:37.566 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 30/37 2024-09-18T07:54:37.701 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 31/37 2024-09-18T07:54:37.871 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : ceph-mgr-modules-core-2:19.3.0-4983.ge7fb7b56.el9. 32/37 2024-09-18T07:54:37.909 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : ceph-mgr-2:19.3.0-4983.ge7fb7b56.el9.x86_64 33/37 2024-09-18T07:54:37.937 INFO:teuthology.orchestra.run.smithi050.stdout: Running scriptlet: ceph-mgr-2:19.3.0-4983.ge7fb7b56.el9.x86_64 33/37 2024-09-18T07:54:37.937 INFO:teuthology.orchestra.run.smithi050.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-18T07:54:37.937 INFO:teuthology.orchestra.run.smithi050.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-09-18T07:54:37.937 INFO:teuthology.orchestra.run.smithi050.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-09-18T07:54:37.937 INFO:teuthology.orchestra.run.smithi050.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-09-18T07:54:37.937 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-09-18T07:54:38.260 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : ceph-radosgw-2:19.3.0-4983.ge7fb7b56.el9.x86_64 33/35 2024-09-18T07:54:38.286 INFO:teuthology.orchestra.run.smithi007.stdout: Running scriptlet: ceph-radosgw-2:19.3.0-4983.ge7fb7b56.el9.x86_64 33/35 2024-09-18T07:54:38.286 INFO:teuthology.orchestra.run.smithi007.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-18T07:54:38.287 INFO:teuthology.orchestra.run.smithi007.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-09-18T07:54:38.287 INFO:teuthology.orchestra.run.smithi007.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-09-18T07:54:38.287 INFO:teuthology.orchestra.run.smithi007.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-09-18T07:54:38.287 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-09-18T07:54:38.289 INFO:teuthology.orchestra.run.smithi007.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 34/35 2024-09-18T07:54:38.379 INFO:teuthology.orchestra.run.smithi007.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 34/35 2024-09-18T07:54:38.379 INFO:teuthology.orchestra.run.smithi007.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 35/35 2024-09-18T07:54:39.531 INFO:teuthology.orchestra.run.smithi007.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 35/35 2024-09-18T07:54:39.531 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : ceph-base-2:19.3.0-4983.ge7fb7b56.el9.x86_64 1/35 2024-09-18T07:54:39.531 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : ceph-common-2:19.3.0-4983.ge7fb7b56.el9.x86_64 2/35 2024-09-18T07:54:39.531 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : ceph-radosgw-2:19.3.0-4983.ge7fb7b56.el9.x86_64 3/35 2024-09-18T07:54:39.531 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : ceph-selinux-2:19.3.0-4983.ge7fb7b56.el9.x86_64 4/35 2024-09-18T07:54:39.531 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : libcephfs2-2:19.3.0-4983.ge7fb7b56.el9.x86_64 5/35 2024-09-18T07:54:39.532 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : libradosstriper1-2:19.3.0-4983.ge7fb7b56.el9.x86_6 6/35 2024-09-18T07:54:39.532 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : librgw2-2:19.3.0-4983.ge7fb7b56.el9.x86_64 7/35 2024-09-18T07:54:39.532 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : python3-ceph-argparse-2:19.3.0-4983.ge7fb7b56.el9. 8/35 2024-09-18T07:54:39.532 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : python3-ceph-common-2:19.3.0-4983.ge7fb7b56.el9.x8 9/35 2024-09-18T07:54:39.532 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : python3-cephfs-2:19.3.0-4983.ge7fb7b56.el9.x86_64 10/35 2024-09-18T07:54:39.532 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : python3-rados-2:19.3.0-4983.ge7fb7b56.el9.x86_64 11/35 2024-09-18T07:54:39.532 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : python3-rbd-2:19.3.0-4983.ge7fb7b56.el9.x86_64 12/35 2024-09-18T07:54:39.532 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : python3-rgw-2:19.3.0-4983.ge7fb7b56.el9.x86_64 13/35 2024-09-18T07:54:39.532 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 14/35 2024-09-18T07:54:39.532 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : libnbd-1.20.2-2.el9.x86_64 15/35 2024-09-18T07:54:39.532 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 16/35 2024-09-18T07:54:39.532 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 17/35 2024-09-18T07:54:39.533 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 18/35 2024-09-18T07:54:39.533 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 19/35 2024-09-18T07:54:39.533 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 20/35 2024-09-18T07:54:39.533 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : qatlib-24.02.0-1.el9.x86_64 21/35 2024-09-18T07:54:39.533 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : qatlib-service-24.02.0-1.el9.x86_64 22/35 2024-09-18T07:54:39.533 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : qatzip-libs-1.2.0-1.el9.x86_64 23/35 2024-09-18T07:54:39.533 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : gperftools-libs-2.9.1-3.el9.x86_64 24/35 2024-09-18T07:54:39.533 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : libarrow-9.0.0-11.el9.x86_64 25/35 2024-09-18T07:54:39.533 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : libarrow-doc-9.0.0-11.el9.noarch 26/35 2024-09-18T07:54:39.533 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 27/35 2024-09-18T07:54:39.533 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 28/35 2024-09-18T07:54:39.533 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : parquet-libs-9.0.0-11.el9.x86_64 29/35 2024-09-18T07:54:39.534 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : re2-1:20211101-20.el9.x86_64 30/35 2024-09-18T07:54:39.534 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 31/35 2024-09-18T07:54:39.535 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : librados2-2:19.3.0-4983.ge7fb7b56.el9.x86_64 32/35 2024-09-18T07:54:39.535 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 33/35 2024-09-18T07:54:39.535 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : librbd1-2:19.3.0-4983.ge7fb7b56.el9.x86_64 34/35 2024-09-18T07:54:39.569 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : ceph-osd-2:19.3.0-4983.ge7fb7b56.el9.x86_64 34/37 2024-09-18T07:54:39.598 INFO:teuthology.orchestra.run.smithi050.stdout: Running scriptlet: ceph-osd-2:19.3.0-4983.ge7fb7b56.el9.x86_64 34/37 2024-09-18T07:54:39.598 INFO:teuthology.orchestra.run.smithi050.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-18T07:54:39.598 INFO:teuthology.orchestra.run.smithi050.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-09-18T07:54:39.598 INFO:teuthology.orchestra.run.smithi050.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-09-18T07:54:39.598 INFO:teuthology.orchestra.run.smithi050.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-09-18T07:54:39.598 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-09-18T07:54:40.095 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : ceph-mon-2:19.3.0-4983.ge7fb7b56.el9.x86_64 35/37 2024-09-18T07:54:40.120 INFO:teuthology.orchestra.run.smithi050.stdout: Running scriptlet: ceph-mon-2:19.3.0-4983.ge7fb7b56.el9.x86_64 35/37 2024-09-18T07:54:40.120 INFO:teuthology.orchestra.run.smithi050.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-18T07:54:40.120 INFO:teuthology.orchestra.run.smithi050.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-09-18T07:54:40.120 INFO:teuthology.orchestra.run.smithi050.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-09-18T07:54:40.120 INFO:teuthology.orchestra.run.smithi050.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-09-18T07:54:40.120 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-09-18T07:54:40.364 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : ceph-mds-2:19.3.0-4983.ge7fb7b56.el9.x86_64 36/37 2024-09-18T07:54:40.387 INFO:teuthology.orchestra.run.smithi050.stdout: Running scriptlet: ceph-mds-2:19.3.0-4983.ge7fb7b56.el9.x86_64 36/37 2024-09-18T07:54:40.387 INFO:teuthology.orchestra.run.smithi050.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-18T07:54:40.387 INFO:teuthology.orchestra.run.smithi050.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-09-18T07:54:40.387 INFO:teuthology.orchestra.run.smithi050.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-09-18T07:54:40.387 INFO:teuthology.orchestra.run.smithi050.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-09-18T07:54:40.387 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-09-18T07:54:40.453 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : ceph-2:19.3.0-4983.ge7fb7b56.el9.x86_64 37/37 2024-09-18T07:54:40.465 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 35/35 2024-09-18T07:54:40.465 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-09-18T07:54:40.465 INFO:teuthology.orchestra.run.smithi007.stdout:Upgraded: 2024-09-18T07:54:40.465 INFO:teuthology.orchestra.run.smithi007.stdout: librados2-2:19.3.0-4983.ge7fb7b56.el9.x86_64 2024-09-18T07:54:40.465 INFO:teuthology.orchestra.run.smithi007.stdout: librbd1-2:19.3.0-4983.ge7fb7b56.el9.x86_64 2024-09-18T07:54:40.465 INFO:teuthology.orchestra.run.smithi007.stdout:Installed: 2024-09-18T07:54:40.465 INFO:teuthology.orchestra.run.smithi007.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-09-18T07:54:40.465 INFO:teuthology.orchestra.run.smithi007.stdout: ceph-base-2:19.3.0-4983.ge7fb7b56.el9.x86_64 2024-09-18T07:54:40.465 INFO:teuthology.orchestra.run.smithi007.stdout: ceph-common-2:19.3.0-4983.ge7fb7b56.el9.x86_64 2024-09-18T07:54:40.465 INFO:teuthology.orchestra.run.smithi007.stdout: ceph-radosgw-2:19.3.0-4983.ge7fb7b56.el9.x86_64 2024-09-18T07:54:40.466 INFO:teuthology.orchestra.run.smithi007.stdout: ceph-selinux-2:19.3.0-4983.ge7fb7b56.el9.x86_64 2024-09-18T07:54:40.466 INFO:teuthology.orchestra.run.smithi007.stdout: gperftools-libs-2.9.1-3.el9.x86_64 2024-09-18T07:54:40.466 INFO:teuthology.orchestra.run.smithi007.stdout: libarrow-9.0.0-11.el9.x86_64 2024-09-18T07:54:40.466 INFO:teuthology.orchestra.run.smithi007.stdout: libarrow-doc-9.0.0-11.el9.noarch 2024-09-18T07:54:40.466 INFO:teuthology.orchestra.run.smithi007.stdout: libcephfs2-2:19.3.0-4983.ge7fb7b56.el9.x86_64 2024-09-18T07:54:40.466 INFO:teuthology.orchestra.run.smithi007.stdout: libnbd-1.20.2-2.el9.x86_64 2024-09-18T07:54:40.466 INFO:teuthology.orchestra.run.smithi007.stdout: liboath-2.6.7-2.el9.x86_64 2024-09-18T07:54:40.466 INFO:teuthology.orchestra.run.smithi007.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-09-18T07:54:40.466 INFO:teuthology.orchestra.run.smithi007.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-09-18T07:54:40.466 INFO:teuthology.orchestra.run.smithi007.stdout: libradosstriper1-2:19.3.0-4983.ge7fb7b56.el9.x86_64 2024-09-18T07:54:40.466 INFO:teuthology.orchestra.run.smithi007.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-09-18T07:54:40.466 INFO:teuthology.orchestra.run.smithi007.stdout: librgw2-2:19.3.0-4983.ge7fb7b56.el9.x86_64 2024-09-18T07:54:40.466 INFO:teuthology.orchestra.run.smithi007.stdout: libunwind-1.6.2-1.el9.x86_64 2024-09-18T07:54:40.466 INFO:teuthology.orchestra.run.smithi007.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-09-18T07:54:40.467 INFO:teuthology.orchestra.run.smithi007.stdout: parquet-libs-9.0.0-11.el9.x86_64 2024-09-18T07:54:40.467 INFO:teuthology.orchestra.run.smithi007.stdout: python3-ceph-argparse-2:19.3.0-4983.ge7fb7b56.el9.x86_64 2024-09-18T07:54:40.467 INFO:teuthology.orchestra.run.smithi007.stdout: python3-ceph-common-2:19.3.0-4983.ge7fb7b56.el9.x86_64 2024-09-18T07:54:40.467 INFO:teuthology.orchestra.run.smithi007.stdout: python3-cephfs-2:19.3.0-4983.ge7fb7b56.el9.x86_64 2024-09-18T07:54:40.467 INFO:teuthology.orchestra.run.smithi007.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-09-18T07:54:40.467 INFO:teuthology.orchestra.run.smithi007.stdout: python3-rados-2:19.3.0-4983.ge7fb7b56.el9.x86_64 2024-09-18T07:54:40.467 INFO:teuthology.orchestra.run.smithi007.stdout: python3-rbd-2:19.3.0-4983.ge7fb7b56.el9.x86_64 2024-09-18T07:54:40.467 INFO:teuthology.orchestra.run.smithi007.stdout: python3-rgw-2:19.3.0-4983.ge7fb7b56.el9.x86_64 2024-09-18T07:54:40.467 INFO:teuthology.orchestra.run.smithi007.stdout: qatlib-24.02.0-1.el9.x86_64 2024-09-18T07:54:40.467 INFO:teuthology.orchestra.run.smithi007.stdout: qatlib-service-24.02.0-1.el9.x86_64 2024-09-18T07:54:40.467 INFO:teuthology.orchestra.run.smithi007.stdout: qatzip-libs-1.2.0-1.el9.x86_64 2024-09-18T07:54:40.467 INFO:teuthology.orchestra.run.smithi007.stdout: re2-1:20211101-20.el9.x86_64 2024-09-18T07:54:40.467 INFO:teuthology.orchestra.run.smithi007.stdout: thrift-0.15.0-2.el9.x86_64 2024-09-18T07:54:40.467 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-09-18T07:54:40.468 INFO:teuthology.orchestra.run.smithi007.stdout:Complete! 2024-09-18T07:54:40.853 DEBUG:teuthology.orchestra.run.smithi007:> sudo yum -y install ceph-test 2024-09-18T07:54:41.463 INFO:teuthology.orchestra.run.smithi007.stdout:Last metadata expiration check: 0:00:29 ago on Wed 18 Sep 2024 07:54:12 AM UTC. 2024-09-18T07:54:41.606 INFO:teuthology.orchestra.run.smithi007.stdout:Dependencies resolved. 2024-09-18T07:54:41.607 INFO:teuthology.orchestra.run.smithi007.stdout:================================================================================ 2024-09-18T07:54:41.607 INFO:teuthology.orchestra.run.smithi007.stdout: Package Arch Version Repository Size 2024-09-18T07:54:41.607 INFO:teuthology.orchestra.run.smithi007.stdout:================================================================================ 2024-09-18T07:54:41.607 INFO:teuthology.orchestra.run.smithi007.stdout:Installing: 2024-09-18T07:54:41.607 INFO:teuthology.orchestra.run.smithi007.stdout: ceph-test x86_64 2:19.3.0-4983.ge7fb7b56.el9 ceph 46 M 2024-09-18T07:54:41.608 INFO:teuthology.orchestra.run.smithi007.stdout:Installing dependencies: 2024-09-18T07:54:41.608 INFO:teuthology.orchestra.run.smithi007.stdout: libcephsqlite x86_64 2:19.3.0-4983.ge7fb7b56.el9 ceph 161 k 2024-09-18T07:54:41.608 INFO:teuthology.orchestra.run.smithi007.stdout: socat x86_64 1.7.4.1-6.el9 appstream 304 k 2024-09-18T07:54:41.608 INFO:teuthology.orchestra.run.smithi007.stdout: xmlstarlet x86_64 1.6.1-20.el9 appstream 64 k 2024-09-18T07:54:41.608 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-09-18T07:54:41.608 INFO:teuthology.orchestra.run.smithi007.stdout:Transaction Summary 2024-09-18T07:54:41.608 INFO:teuthology.orchestra.run.smithi007.stdout:================================================================================ 2024-09-18T07:54:41.608 INFO:teuthology.orchestra.run.smithi007.stdout:Install 4 Packages 2024-09-18T07:54:41.608 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-09-18T07:54:41.609 INFO:teuthology.orchestra.run.smithi007.stdout:Total download size: 46 M 2024-09-18T07:54:41.609 INFO:teuthology.orchestra.run.smithi007.stdout:Installed size: 201 M 2024-09-18T07:54:41.609 INFO:teuthology.orchestra.run.smithi007.stdout:Downloading Packages: 2024-09-18T07:54:41.944 INFO:teuthology.orchestra.run.smithi007.stdout:(1/4): socat-1.7.4.1-6.el9.x86_64.rpm 1.4 MB/s | 304 kB 00:00 2024-09-18T07:54:41.970 INFO:teuthology.orchestra.run.smithi007.stdout:(2/4): libcephsqlite-19.3.0-4983.ge7fb7b56.el9. 694 kB/s | 161 kB 00:00 2024-09-18T07:54:41.987 INFO:teuthology.orchestra.run.smithi007.stdout:(3/4): xmlstarlet-1.6.1-20.el9.x86_64.rpm 1.5 MB/s | 64 kB 00:00 2024-09-18T07:54:42.324 INFO:teuthology.orchestra.run.smithi050.stdout: Running scriptlet: ceph-2:19.3.0-4983.ge7fb7b56.el9.x86_64 37/37 2024-09-18T07:54:42.324 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : ceph-2:19.3.0-4983.ge7fb7b56.el9.x86_64 1/37 2024-09-18T07:54:42.325 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : ceph-mds-2:19.3.0-4983.ge7fb7b56.el9.x86_64 2/37 2024-09-18T07:54:42.325 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : ceph-mgr-2:19.3.0-4983.ge7fb7b56.el9.x86_64 3/37 2024-09-18T07:54:42.325 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : ceph-mon-2:19.3.0-4983.ge7fb7b56.el9.x86_64 4/37 2024-09-18T07:54:42.325 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : ceph-osd-2:19.3.0-4983.ge7fb7b56.el9.x86_64 5/37 2024-09-18T07:54:42.325 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : ceph-mgr-modules-core-2:19.3.0-4983.ge7fb7b56.el9. 6/37 2024-09-18T07:54:42.325 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 7/37 2024-09-18T07:54:42.325 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 8/37 2024-09-18T07:54:42.325 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : python3-ply-3.11-14.el9.noarch 9/37 2024-09-18T07:54:42.325 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 10/37 2024-09-18T07:54:42.325 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 11/37 2024-09-18T07:54:42.325 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : python3-urllib3-1.26.5-6.el9.noarch 12/37 2024-09-18T07:54:42.325 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : lua-5.4.4-4.el9.x86_64 13/37 2024-09-18T07:54:42.325 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 14/37 2024-09-18T07:54:42.326 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 15/37 2024-09-18T07:54:42.326 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 16/37 2024-09-18T07:54:42.326 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 17/37 2024-09-18T07:54:42.326 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 18/37 2024-09-18T07:54:42.326 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 19/37 2024-09-18T07:54:42.326 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : luarocks-3.9.2-5.el9.noarch 20/37 2024-09-18T07:54:42.326 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 21/37 2024-09-18T07:54:42.326 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 22/37 2024-09-18T07:54:42.326 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 23/37 2024-09-18T07:54:42.326 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 24/37 2024-09-18T07:54:42.326 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 25/37 2024-09-18T07:54:42.326 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 26/37 2024-09-18T07:54:42.326 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 27/37 2024-09-18T07:54:42.327 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 28/37 2024-09-18T07:54:42.327 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 29/37 2024-09-18T07:54:42.327 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 30/37 2024-09-18T07:54:42.327 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 31/37 2024-09-18T07:54:42.327 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 32/37 2024-09-18T07:54:42.327 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 33/37 2024-09-18T07:54:42.327 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 34/37 2024-09-18T07:54:42.327 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : python3-webob-1.8.8-2.el9.noarch 35/37 2024-09-18T07:54:42.327 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 36/37 2024-09-18T07:54:43.141 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 37/37 2024-09-18T07:54:43.141 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-09-18T07:54:43.141 INFO:teuthology.orchestra.run.smithi050.stdout:Installed: 2024-09-18T07:54:43.141 INFO:teuthology.orchestra.run.smithi050.stdout: ceph-2:19.3.0-4983.ge7fb7b56.el9.x86_64 2024-09-18T07:54:43.141 INFO:teuthology.orchestra.run.smithi050.stdout: ceph-mds-2:19.3.0-4983.ge7fb7b56.el9.x86_64 2024-09-18T07:54:43.141 INFO:teuthology.orchestra.run.smithi050.stdout: ceph-mgr-2:19.3.0-4983.ge7fb7b56.el9.x86_64 2024-09-18T07:54:43.141 INFO:teuthology.orchestra.run.smithi050.stdout: ceph-mgr-modules-core-2:19.3.0-4983.ge7fb7b56.el9.noarch 2024-09-18T07:54:43.141 INFO:teuthology.orchestra.run.smithi050.stdout: ceph-mon-2:19.3.0-4983.ge7fb7b56.el9.x86_64 2024-09-18T07:54:43.141 INFO:teuthology.orchestra.run.smithi050.stdout: ceph-osd-2:19.3.0-4983.ge7fb7b56.el9.x86_64 2024-09-18T07:54:43.141 INFO:teuthology.orchestra.run.smithi050.stdout: lua-5.4.4-4.el9.x86_64 2024-09-18T07:54:43.142 INFO:teuthology.orchestra.run.smithi050.stdout: lua-devel-5.4.4-4.el9.x86_64 2024-09-18T07:54:43.142 INFO:teuthology.orchestra.run.smithi050.stdout: luarocks-3.9.2-5.el9.noarch 2024-09-18T07:54:43.142 INFO:teuthology.orchestra.run.smithi050.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-09-18T07:54:43.142 INFO:teuthology.orchestra.run.smithi050.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-09-18T07:54:43.142 INFO:teuthology.orchestra.run.smithi050.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-09-18T07:54:43.142 INFO:teuthology.orchestra.run.smithi050.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-09-18T07:54:43.142 INFO:teuthology.orchestra.run.smithi050.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-09-18T07:54:43.142 INFO:teuthology.orchestra.run.smithi050.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-09-18T07:54:43.142 INFO:teuthology.orchestra.run.smithi050.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-09-18T07:54:43.142 INFO:teuthology.orchestra.run.smithi050.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-09-18T07:54:43.142 INFO:teuthology.orchestra.run.smithi050.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-09-18T07:54:43.142 INFO:teuthology.orchestra.run.smithi050.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-09-18T07:54:43.142 INFO:teuthology.orchestra.run.smithi050.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-09-18T07:54:43.143 INFO:teuthology.orchestra.run.smithi050.stdout: python3-mako-1.1.4-6.el9.noarch 2024-09-18T07:54:43.143 INFO:teuthology.orchestra.run.smithi050.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-09-18T07:54:43.143 INFO:teuthology.orchestra.run.smithi050.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-09-18T07:54:43.143 INFO:teuthology.orchestra.run.smithi050.stdout: python3-packaging-20.9-5.el9.noarch 2024-09-18T07:54:43.143 INFO:teuthology.orchestra.run.smithi050.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-09-18T07:54:43.143 INFO:teuthology.orchestra.run.smithi050.stdout: python3-ply-3.11-14.el9.noarch 2024-09-18T07:54:43.143 INFO:teuthology.orchestra.run.smithi050.stdout: python3-portend-3.1.0-2.el9.noarch 2024-09-18T07:54:43.143 INFO:teuthology.orchestra.run.smithi050.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-09-18T07:54:43.143 INFO:teuthology.orchestra.run.smithi050.stdout: python3-pycparser-2.20-6.el9.noarch 2024-09-18T07:54:43.143 INFO:teuthology.orchestra.run.smithi050.stdout: python3-pytz-2021.1-5.el9.noarch 2024-09-18T07:54:43.143 INFO:teuthology.orchestra.run.smithi050.stdout: python3-requests-2.25.1-8.el9.noarch 2024-09-18T07:54:43.143 INFO:teuthology.orchestra.run.smithi050.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-09-18T07:54:43.143 INFO:teuthology.orchestra.run.smithi050.stdout: python3-toml-0.10.2-6.el9.noarch 2024-09-18T07:54:43.143 INFO:teuthology.orchestra.run.smithi050.stdout: python3-urllib3-1.26.5-6.el9.noarch 2024-09-18T07:54:43.143 INFO:teuthology.orchestra.run.smithi050.stdout: python3-webob-1.8.8-2.el9.noarch 2024-09-18T07:54:43.144 INFO:teuthology.orchestra.run.smithi050.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-09-18T07:54:43.144 INFO:teuthology.orchestra.run.smithi050.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-09-18T07:54:43.144 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-09-18T07:54:43.144 INFO:teuthology.orchestra.run.smithi050.stdout:Complete! 2024-09-18T07:54:43.212 INFO:teuthology.orchestra.run.smithi007.stdout:(4/4): ceph-test-19.3.0-4983.ge7fb7b56.el9.x86_ 31 MB/s | 46 MB 00:01 2024-09-18T07:54:43.214 INFO:teuthology.orchestra.run.smithi007.stdout:-------------------------------------------------------------------------------- 2024-09-18T07:54:43.214 INFO:teuthology.orchestra.run.smithi007.stdout:Total 29 MB/s | 46 MB 00:01 2024-09-18T07:54:43.243 INFO:teuthology.orchestra.run.smithi007.stdout:Running transaction check 2024-09-18T07:54:43.266 INFO:teuthology.orchestra.run.smithi007.stdout:Transaction check succeeded. 2024-09-18T07:54:43.266 INFO:teuthology.orchestra.run.smithi007.stdout:Running transaction test 2024-09-18T07:54:43.477 DEBUG:teuthology.orchestra.run.smithi050:> sudo yum -y install ceph-base 2024-09-18T07:54:43.641 INFO:teuthology.orchestra.run.smithi007.stdout:Transaction test succeeded. 2024-09-18T07:54:43.641 INFO:teuthology.orchestra.run.smithi007.stdout:Running transaction 2024-09-18T07:54:44.097 INFO:teuthology.orchestra.run.smithi050.stdout:Last metadata expiration check: 0:00:54 ago on Wed 18 Sep 2024 07:53:50 AM UTC. 2024-09-18T07:54:44.171 INFO:teuthology.orchestra.run.smithi007.stdout: Preparing : 1/1 2024-09-18T07:54:44.190 INFO:teuthology.orchestra.run.smithi050.stdout:Package ceph-base-2:19.3.0-4983.ge7fb7b56.el9.x86_64 is already installed. 2024-09-18T07:54:44.234 INFO:teuthology.orchestra.run.smithi050.stdout:Dependencies resolved. 2024-09-18T07:54:44.235 INFO:teuthology.orchestra.run.smithi050.stdout:Nothing to do. 2024-09-18T07:54:44.235 INFO:teuthology.orchestra.run.smithi050.stdout:Complete! 2024-09-18T07:54:44.266 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 1/4 2024-09-18T07:54:44.300 DEBUG:teuthology.orchestra.run.smithi050:> sudo yum -y install cephadm 2024-09-18T07:54:44.317 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : socat-1.7.4.1-6.el9.x86_64 2/4 2024-09-18T07:54:44.347 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : libcephsqlite-2:19.3.0-4983.ge7fb7b56.el9.x86_64 3/4 2024-09-18T07:54:44.899 INFO:teuthology.orchestra.run.smithi050.stdout:Last metadata expiration check: 0:00:54 ago on Wed 18 Sep 2024 07:53:50 AM UTC. 2024-09-18T07:54:45.039 INFO:teuthology.orchestra.run.smithi050.stdout:Dependencies resolved. 2024-09-18T07:54:45.040 INFO:teuthology.orchestra.run.smithi050.stdout:================================================================================ 2024-09-18T07:54:45.040 INFO:teuthology.orchestra.run.smithi050.stdout: Package Arch Version Repository Size 2024-09-18T07:54:45.040 INFO:teuthology.orchestra.run.smithi050.stdout:================================================================================ 2024-09-18T07:54:45.040 INFO:teuthology.orchestra.run.smithi050.stdout:Installing: 2024-09-18T07:54:45.040 INFO:teuthology.orchestra.run.smithi050.stdout: cephadm noarch 2:19.3.0-4983.ge7fb7b56.el9 ceph-noarch 784 k 2024-09-18T07:54:45.040 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-09-18T07:54:45.041 INFO:teuthology.orchestra.run.smithi050.stdout:Transaction Summary 2024-09-18T07:54:45.041 INFO:teuthology.orchestra.run.smithi050.stdout:================================================================================ 2024-09-18T07:54:45.041 INFO:teuthology.orchestra.run.smithi050.stdout:Install 1 Package 2024-09-18T07:54:45.041 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-09-18T07:54:45.041 INFO:teuthology.orchestra.run.smithi050.stdout:Total download size: 784 k 2024-09-18T07:54:45.041 INFO:teuthology.orchestra.run.smithi050.stdout:Installed size: 791 k 2024-09-18T07:54:45.041 INFO:teuthology.orchestra.run.smithi050.stdout:Downloading Packages: 2024-09-18T07:54:45.304 INFO:teuthology.orchestra.run.smithi050.stdout:cephadm-19.3.0-4983.ge7fb7b56.el9.noarch.rpm 2.9 MB/s | 784 kB 00:00 2024-09-18T07:54:45.304 INFO:teuthology.orchestra.run.smithi050.stdout:-------------------------------------------------------------------------------- 2024-09-18T07:54:45.305 INFO:teuthology.orchestra.run.smithi050.stdout:Total 2.9 MB/s | 784 kB 00:00 2024-09-18T07:54:45.305 INFO:teuthology.orchestra.run.smithi050.stdout:Running transaction check 2024-09-18T07:54:45.310 INFO:teuthology.orchestra.run.smithi050.stdout:Transaction check succeeded. 2024-09-18T07:54:45.310 INFO:teuthology.orchestra.run.smithi050.stdout:Running transaction test 2024-09-18T07:54:45.317 INFO:teuthology.orchestra.run.smithi050.stdout:Transaction test succeeded. 2024-09-18T07:54:45.317 INFO:teuthology.orchestra.run.smithi050.stdout:Running transaction 2024-09-18T07:54:45.394 INFO:teuthology.orchestra.run.smithi050.stdout: Preparing : 1/1 2024-09-18T07:54:45.883 INFO:teuthology.orchestra.run.smithi050.stdout: Running scriptlet: cephadm-2:19.3.0-4983.ge7fb7b56.el9.noarch 1/1 2024-09-18T07:54:45.969 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : cephadm-2:19.3.0-4983.ge7fb7b56.el9.noarch 1/1 2024-09-18T07:54:46.377 INFO:teuthology.orchestra.run.smithi050.stdout: Running scriptlet: cephadm-2:19.3.0-4983.ge7fb7b56.el9.noarch 1/1 2024-09-18T07:54:46.624 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : cephadm-2:19.3.0-4983.ge7fb7b56.el9.noarch 1/1 2024-09-18T07:54:46.624 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-09-18T07:54:46.624 INFO:teuthology.orchestra.run.smithi050.stdout:Installed: 2024-09-18T07:54:46.624 INFO:teuthology.orchestra.run.smithi050.stdout: cephadm-2:19.3.0-4983.ge7fb7b56.el9.noarch 2024-09-18T07:54:46.624 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-09-18T07:54:46.624 INFO:teuthology.orchestra.run.smithi050.stdout:Complete! 2024-09-18T07:54:46.852 DEBUG:teuthology.orchestra.run.smithi050:> sudo yum -y install ceph-immutable-object-cache 2024-09-18T07:54:47.460 INFO:teuthology.orchestra.run.smithi050.stdout:Last metadata expiration check: 0:00:57 ago on Wed 18 Sep 2024 07:53:50 AM UTC. 2024-09-18T07:54:47.602 INFO:teuthology.orchestra.run.smithi050.stdout:Dependencies resolved. 2024-09-18T07:54:47.603 INFO:teuthology.orchestra.run.smithi050.stdout:================================================================================ 2024-09-18T07:54:47.603 INFO:teuthology.orchestra.run.smithi050.stdout: Package Arch Version Repo Size 2024-09-18T07:54:47.603 INFO:teuthology.orchestra.run.smithi050.stdout:================================================================================ 2024-09-18T07:54:47.603 INFO:teuthology.orchestra.run.smithi050.stdout:Installing: 2024-09-18T07:54:47.603 INFO:teuthology.orchestra.run.smithi050.stdout: ceph-immutable-object-cache x86_64 2:19.3.0-4983.ge7fb7b56.el9 ceph 145 k 2024-09-18T07:54:47.603 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-09-18T07:54:47.603 INFO:teuthology.orchestra.run.smithi050.stdout:Transaction Summary 2024-09-18T07:54:47.603 INFO:teuthology.orchestra.run.smithi050.stdout:================================================================================ 2024-09-18T07:54:47.603 INFO:teuthology.orchestra.run.smithi050.stdout:Install 1 Package 2024-09-18T07:54:47.603 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-09-18T07:54:47.604 INFO:teuthology.orchestra.run.smithi050.stdout:Total download size: 145 k 2024-09-18T07:54:47.604 INFO:teuthology.orchestra.run.smithi050.stdout:Installed size: 432 k 2024-09-18T07:54:47.604 INFO:teuthology.orchestra.run.smithi050.stdout:Downloading Packages: 2024-09-18T07:54:47.810 INFO:teuthology.orchestra.run.smithi050.stdout:ceph-immutable-object-cache-19.3.0-4983.ge7fb7b 705 kB/s | 145 kB 00:00 2024-09-18T07:54:47.811 INFO:teuthology.orchestra.run.smithi050.stdout:-------------------------------------------------------------------------------- 2024-09-18T07:54:47.811 INFO:teuthology.orchestra.run.smithi050.stdout:Total 700 kB/s | 145 kB 00:00 2024-09-18T07:54:47.811 INFO:teuthology.orchestra.run.smithi050.stdout:Running transaction check 2024-09-18T07:54:47.819 INFO:teuthology.orchestra.run.smithi050.stdout:Transaction check succeeded. 2024-09-18T07:54:47.819 INFO:teuthology.orchestra.run.smithi050.stdout:Running transaction test 2024-09-18T07:54:47.880 INFO:teuthology.orchestra.run.smithi050.stdout:Transaction test succeeded. 2024-09-18T07:54:47.880 INFO:teuthology.orchestra.run.smithi050.stdout:Running transaction 2024-09-18T07:54:48.033 INFO:teuthology.orchestra.run.smithi050.stdout: Preparing : 1/1 2024-09-18T07:54:48.094 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : ceph-immutable-object-cache-2:19.3.0-4983.ge7fb7b56. 1/1 2024-09-18T07:54:48.121 INFO:teuthology.orchestra.run.smithi050.stdout: Running scriptlet: ceph-immutable-object-cache-2:19.3.0-4983.ge7fb7b56. 1/1 2024-09-18T07:54:48.121 INFO:teuthology.orchestra.run.smithi050.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-18T07:54:48.121 INFO:teuthology.orchestra.run.smithi050.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-09-18T07:54:48.121 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-09-18T07:54:48.958 INFO:teuthology.orchestra.run.smithi007.stdout: Running scriptlet: libcephsqlite-2:19.3.0-4983.ge7fb7b56.el9.x86_64 3/4 2024-09-18T07:54:49.045 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : ceph-test-2:19.3.0-4983.ge7fb7b56.el9.x86_64 4/4 2024-09-18T07:54:49.081 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : ceph-immutable-object-cache-2:19.3.0-4983.ge7fb7b56. 1/1 2024-09-18T07:54:49.081 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-09-18T07:54:49.081 INFO:teuthology.orchestra.run.smithi050.stdout:Installed: 2024-09-18T07:54:49.081 INFO:teuthology.orchestra.run.smithi050.stdout: ceph-immutable-object-cache-2:19.3.0-4983.ge7fb7b56.el9.x86_64 2024-09-18T07:54:49.081 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-09-18T07:54:49.081 INFO:teuthology.orchestra.run.smithi050.stdout:Complete! 2024-09-18T07:54:49.265 DEBUG:teuthology.orchestra.run.smithi050:> sudo yum -y install ceph-mgr 2024-09-18T07:54:49.873 INFO:teuthology.orchestra.run.smithi050.stdout:Last metadata expiration check: 0:00:59 ago on Wed 18 Sep 2024 07:53:50 AM UTC. 2024-09-18T07:54:49.966 INFO:teuthology.orchestra.run.smithi050.stdout:Package ceph-mgr-2:19.3.0-4983.ge7fb7b56.el9.x86_64 is already installed. 2024-09-18T07:54:50.011 INFO:teuthology.orchestra.run.smithi050.stdout:Dependencies resolved. 2024-09-18T07:54:50.012 INFO:teuthology.orchestra.run.smithi050.stdout:Nothing to do. 2024-09-18T07:54:50.012 INFO:teuthology.orchestra.run.smithi050.stdout:Complete! 2024-09-18T07:54:50.068 DEBUG:teuthology.orchestra.run.smithi050:> sudo yum -y install ceph-mgr-dashboard 2024-09-18T07:54:50.673 INFO:teuthology.orchestra.run.smithi050.stdout:Last metadata expiration check: 0:01:00 ago on Wed 18 Sep 2024 07:53:50 AM UTC. 2024-09-18T07:54:50.812 INFO:teuthology.orchestra.run.smithi050.stdout:Dependencies resolved. 2024-09-18T07:54:50.813 INFO:teuthology.orchestra.run.smithi050.stdout:================================================================================ 2024-09-18T07:54:50.813 INFO:teuthology.orchestra.run.smithi050.stdout: Package Arch Version Repository Size 2024-09-18T07:54:50.813 INFO:teuthology.orchestra.run.smithi050.stdout:================================================================================ 2024-09-18T07:54:50.813 INFO:teuthology.orchestra.run.smithi050.stdout:Installing: 2024-09-18T07:54:50.814 INFO:teuthology.orchestra.run.smithi050.stdout: ceph-mgr-dashboard noarch 2:19.3.0-4983.ge7fb7b56.el9 ceph-noarch 14 M 2024-09-18T07:54:50.814 INFO:teuthology.orchestra.run.smithi050.stdout:Installing dependencies: 2024-09-18T07:54:50.814 INFO:teuthology.orchestra.run.smithi050.stdout: abseil-cpp x86_64 20211102.0-3.el9 epel 548 k 2024-09-18T07:54:50.814 INFO:teuthology.orchestra.run.smithi050.stdout: ceph-grafana-dashboards noarch 2:19.3.0-4983.ge7fb7b56.el9 ceph-noarch 34 k 2024-09-18T07:54:50.814 INFO:teuthology.orchestra.run.smithi050.stdout: ceph-prometheus-alerts noarch 2:19.3.0-4983.ge7fb7b56.el9 ceph-noarch 16 k 2024-09-18T07:54:50.814 INFO:teuthology.orchestra.run.smithi050.stdout: grpc-data noarch 1.46.7-10.el9 epel 19 k 2024-09-18T07:54:50.814 INFO:teuthology.orchestra.run.smithi050.stdout: protobuf x86_64 3.14.0-13.el9 appstream 1.0 M 2024-09-18T07:54:50.814 INFO:teuthology.orchestra.run.smithi050.stdout: protobuf-compiler x86_64 3.14.0-13.el9 crb 863 k 2024-09-18T07:54:50.814 INFO:teuthology.orchestra.run.smithi050.stdout: python3-grpcio x86_64 1.46.7-10.el9 epel 2.0 M 2024-09-18T07:54:50.814 INFO:teuthology.orchestra.run.smithi050.stdout: python3-grpcio-tools x86_64 1.46.7-10.el9 epel 144 k 2024-09-18T07:54:50.814 INFO:teuthology.orchestra.run.smithi050.stdout: python3-protobuf noarch 3.14.0-13.el9 appstream 269 k 2024-09-18T07:54:50.814 INFO:teuthology.orchestra.run.smithi050.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-09-18T07:54:50.815 INFO:teuthology.orchestra.run.smithi050.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-09-18T07:54:50.815 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-09-18T07:54:50.815 INFO:teuthology.orchestra.run.smithi050.stdout:Transaction Summary 2024-09-18T07:54:50.815 INFO:teuthology.orchestra.run.smithi050.stdout:================================================================================ 2024-09-18T07:54:50.815 INFO:teuthology.orchestra.run.smithi050.stdout:Install 12 Packages 2024-09-18T07:54:50.815 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-09-18T07:54:50.815 INFO:teuthology.orchestra.run.smithi050.stdout:Total download size: 19 M 2024-09-18T07:54:50.815 INFO:teuthology.orchestra.run.smithi050.stdout:Installed size: 159 M 2024-09-18T07:54:50.816 INFO:teuthology.orchestra.run.smithi050.stdout:Downloading Packages: 2024-09-18T07:54:51.080 INFO:teuthology.orchestra.run.smithi050.stdout:(1/12): ceph-grafana-dashboards-19.3.0-4983.ge7 243 kB/s | 34 kB 00:00 2024-09-18T07:54:51.105 INFO:teuthology.orchestra.run.smithi050.stdout:(2/12): ceph-prometheus-alerts-19.3.0-4983.ge7f 99 kB/s | 16 kB 00:00 2024-09-18T07:54:51.314 INFO:teuthology.orchestra.run.smithi050.stdout:(3/12): python3-protobuf-3.14.0-13.el9.noarch.r 1.3 MB/s | 269 kB 00:00 2024-09-18T07:54:51.406 INFO:teuthology.orchestra.run.smithi050.stdout:(4/12): protobuf-3.14.0-13.el9.x86_64.rpm 3.1 MB/s | 1.0 MB 00:00 2024-09-18T07:54:51.590 INFO:teuthology.orchestra.run.smithi050.stdout:(5/12): ceph-mgr-dashboard-19.3.0-4983.ge7fb7b5 22 MB/s | 14 MB 00:00 2024-09-18T07:54:51.623 INFO:teuthology.orchestra.run.smithi050.stdout:(6/12): protobuf-compiler-3.14.0-13.el9.x86_64. 2.7 MB/s | 863 kB 00:00 2024-09-18T07:54:51.674 INFO:teuthology.orchestra.run.smithi050.stdout:(7/12): abseil-cpp-20211102.0-3.el9.x86_64.rpm 2.0 MB/s | 548 kB 00:00 2024-09-18T07:54:51.685 INFO:teuthology.orchestra.run.smithi007.stdout: Running scriptlet: ceph-test-2:19.3.0-4983.ge7fb7b56.el9.x86_64 4/4 2024-09-18T07:54:51.685 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : ceph-test-2:19.3.0-4983.ge7fb7b56.el9.x86_64 1/4 2024-09-18T07:54:51.685 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : libcephsqlite-2:19.3.0-4983.ge7fb7b56.el9.x86_64 2/4 2024-09-18T07:54:51.686 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : socat-1.7.4.1-6.el9.x86_64 3/4 2024-09-18T07:54:51.699 INFO:teuthology.orchestra.run.smithi050.stdout:(8/12): grpc-data-1.46.7-10.el9.noarch.rpm 179 kB/s | 19 kB 00:00 2024-09-18T07:54:51.724 INFO:teuthology.orchestra.run.smithi050.stdout:(9/12): python3-grpcio-tools-1.46.7-10.el9.x86_ 2.9 MB/s | 144 kB 00:00 2024-09-18T07:54:51.750 INFO:teuthology.orchestra.run.smithi050.stdout:(10/12): python3-repoze-lru-0.7-16.el9.noarch.r 614 kB/s | 31 kB 00:00 2024-09-18T07:54:51.775 INFO:teuthology.orchestra.run.smithi050.stdout:(11/12): python3-routes-2.5.1-5.el9.noarch.rpm 3.7 MB/s | 188 kB 00:00 2024-09-18T07:54:51.933 INFO:teuthology.orchestra.run.smithi050.stdout:(12/12): python3-grpcio-1.46.7-10.el9.x86_64.rp 6.8 MB/s | 2.0 MB 00:00 2024-09-18T07:54:51.937 INFO:teuthology.orchestra.run.smithi050.stdout:-------------------------------------------------------------------------------- 2024-09-18T07:54:51.937 INFO:teuthology.orchestra.run.smithi050.stdout:Total 17 MB/s | 19 MB 00:01 2024-09-18T07:54:51.987 INFO:teuthology.orchestra.run.smithi050.stdout:Running transaction check 2024-09-18T07:54:52.007 INFO:teuthology.orchestra.run.smithi050.stdout:Transaction check succeeded. 2024-09-18T07:54:52.007 INFO:teuthology.orchestra.run.smithi050.stdout:Running transaction test 2024-09-18T07:54:52.095 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 4/4 2024-09-18T07:54:52.095 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-09-18T07:54:52.095 INFO:teuthology.orchestra.run.smithi007.stdout:Installed: 2024-09-18T07:54:52.095 INFO:teuthology.orchestra.run.smithi007.stdout: ceph-test-2:19.3.0-4983.ge7fb7b56.el9.x86_64 2024-09-18T07:54:52.096 INFO:teuthology.orchestra.run.smithi007.stdout: libcephsqlite-2:19.3.0-4983.ge7fb7b56.el9.x86_64 2024-09-18T07:54:52.096 INFO:teuthology.orchestra.run.smithi007.stdout: socat-1.7.4.1-6.el9.x86_64 2024-09-18T07:54:52.096 INFO:teuthology.orchestra.run.smithi007.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-09-18T07:54:52.096 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-09-18T07:54:52.096 INFO:teuthology.orchestra.run.smithi007.stdout:Complete! 2024-09-18T07:54:52.329 INFO:teuthology.orchestra.run.smithi050.stdout:Transaction test succeeded. 2024-09-18T07:54:52.330 INFO:teuthology.orchestra.run.smithi050.stdout:Running transaction 2024-09-18T07:54:52.349 DEBUG:teuthology.orchestra.run.smithi007:> sudo yum -y install ceph 2024-09-18T07:54:52.940 INFO:teuthology.orchestra.run.smithi050.stdout: Preparing : 1/1 2024-09-18T07:54:52.960 INFO:teuthology.orchestra.run.smithi007.stdout:Last metadata expiration check: 0:00:40 ago on Wed 18 Sep 2024 07:54:12 AM UTC. 2024-09-18T07:54:53.026 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : protobuf-3.14.0-13.el9.x86_64 1/12 2024-09-18T07:54:53.061 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : protobuf-compiler-3.14.0-13.el9.x86_64 2/12 2024-09-18T07:54:53.103 INFO:teuthology.orchestra.run.smithi007.stdout:Dependencies resolved. 2024-09-18T07:54:53.106 INFO:teuthology.orchestra.run.smithi007.stdout:================================================================================ 2024-09-18T07:54:53.106 INFO:teuthology.orchestra.run.smithi007.stdout: Package Arch Version Repository Size 2024-09-18T07:54:53.106 INFO:teuthology.orchestra.run.smithi007.stdout:================================================================================ 2024-09-18T07:54:53.106 INFO:teuthology.orchestra.run.smithi007.stdout:Installing: 2024-09-18T07:54:53.106 INFO:teuthology.orchestra.run.smithi007.stdout: ceph x86_64 2:19.3.0-4983.ge7fb7b56.el9 ceph 6.6 k 2024-09-18T07:54:53.106 INFO:teuthology.orchestra.run.smithi007.stdout:Installing dependencies: 2024-09-18T07:54:53.106 INFO:teuthology.orchestra.run.smithi007.stdout: ceph-mds x86_64 2:19.3.0-4983.ge7fb7b56.el9 ceph 2.4 M 2024-09-18T07:54:53.106 INFO:teuthology.orchestra.run.smithi007.stdout: ceph-mgr x86_64 2:19.3.0-4983.ge7fb7b56.el9 ceph 1.5 M 2024-09-18T07:54:53.106 INFO:teuthology.orchestra.run.smithi007.stdout: ceph-mgr-modules-core noarch 2:19.3.0-4983.ge7fb7b56.el9 ceph-noarch 291 k 2024-09-18T07:54:53.106 INFO:teuthology.orchestra.run.smithi007.stdout: ceph-mon x86_64 2:19.3.0-4983.ge7fb7b56.el9 ceph 4.8 M 2024-09-18T07:54:53.106 INFO:teuthology.orchestra.run.smithi007.stdout: ceph-osd x86_64 2:19.3.0-4983.ge7fb7b56.el9 ceph 17 M 2024-09-18T07:54:53.107 INFO:teuthology.orchestra.run.smithi007.stdout: lua x86_64 5.4.4-4.el9 appstream 188 k 2024-09-18T07:54:53.107 INFO:teuthology.orchestra.run.smithi007.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2024-09-18T07:54:53.107 INFO:teuthology.orchestra.run.smithi007.stdout: luarocks noarch 3.9.2-5.el9 epel 151 k 2024-09-18T07:54:53.107 INFO:teuthology.orchestra.run.smithi007.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-09-18T07:54:53.107 INFO:teuthology.orchestra.run.smithi007.stdout: python3-cffi x86_64 1.14.5-5.el9 baseos 253 k 2024-09-18T07:54:53.107 INFO:teuthology.orchestra.run.smithi007.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-09-18T07:54:53.107 INFO:teuthology.orchestra.run.smithi007.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-09-18T07:54:53.107 INFO:teuthology.orchestra.run.smithi007.stdout: python3-cryptography x86_64 36.0.1-4.el9 baseos 1.2 M 2024-09-18T07:54:53.107 INFO:teuthology.orchestra.run.smithi007.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-09-18T07:54:53.107 INFO:teuthology.orchestra.run.smithi007.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-09-18T07:54:53.107 INFO:teuthology.orchestra.run.smithi007.stdout: python3-jaraco-collections 2024-09-18T07:54:53.107 INFO:teuthology.orchestra.run.smithi007.stdout: noarch 3.0.0-8.el9 epel 23 k 2024-09-18T07:54:53.108 INFO:teuthology.orchestra.run.smithi007.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-09-18T07:54:53.108 INFO:teuthology.orchestra.run.smithi007.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-09-18T07:54:53.108 INFO:teuthology.orchestra.run.smithi007.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-09-18T07:54:53.108 INFO:teuthology.orchestra.run.smithi007.stdout: python3-mako noarch 1.1.4-6.el9 appstream 172 k 2024-09-18T07:54:53.108 INFO:teuthology.orchestra.run.smithi007.stdout: python3-markupsafe x86_64 1.1.1-12.el9 appstream 35 k 2024-09-18T07:54:53.108 INFO:teuthology.orchestra.run.smithi007.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-09-18T07:54:53.108 INFO:teuthology.orchestra.run.smithi007.stdout: python3-packaging noarch 20.9-5.el9 appstream 77 k 2024-09-18T07:54:53.108 INFO:teuthology.orchestra.run.smithi007.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-09-18T07:54:53.108 INFO:teuthology.orchestra.run.smithi007.stdout: python3-ply noarch 3.11-14.el9 baseos 106 k 2024-09-18T07:54:53.108 INFO:teuthology.orchestra.run.smithi007.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-09-18T07:54:53.108 INFO:teuthology.orchestra.run.smithi007.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-09-18T07:54:53.108 INFO:teuthology.orchestra.run.smithi007.stdout: python3-pycparser noarch 2.20-6.el9 baseos 135 k 2024-09-18T07:54:53.109 INFO:teuthology.orchestra.run.smithi007.stdout: python3-pytz noarch 2021.1-5.el9 appstream 51 k 2024-09-18T07:54:53.109 INFO:teuthology.orchestra.run.smithi007.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-09-18T07:54:53.109 INFO:teuthology.orchestra.run.smithi007.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-09-18T07:54:53.109 INFO:teuthology.orchestra.run.smithi007.stdout: python3-toml noarch 0.10.2-6.el9 appstream 42 k 2024-09-18T07:54:53.109 INFO:teuthology.orchestra.run.smithi007.stdout: python3-urllib3 noarch 1.26.5-6.el9 baseos 215 k 2024-09-18T07:54:53.109 INFO:teuthology.orchestra.run.smithi007.stdout: python3-webob noarch 1.8.8-2.el9 epel 230 k 2024-09-18T07:54:53.109 INFO:teuthology.orchestra.run.smithi007.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-09-18T07:54:53.109 INFO:teuthology.orchestra.run.smithi007.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-09-18T07:54:53.109 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-09-18T07:54:53.109 INFO:teuthology.orchestra.run.smithi007.stdout:Transaction Summary 2024-09-18T07:54:53.109 INFO:teuthology.orchestra.run.smithi007.stdout:================================================================================ 2024-09-18T07:54:53.109 INFO:teuthology.orchestra.run.smithi007.stdout:Install 37 Packages 2024-09-18T07:54:53.109 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-09-18T07:54:53.110 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 3/12 2024-09-18T07:54:53.110 INFO:teuthology.orchestra.run.smithi007.stdout:Total download size: 31 M 2024-09-18T07:54:53.110 INFO:teuthology.orchestra.run.smithi007.stdout:Installed size: 109 M 2024-09-18T07:54:53.111 INFO:teuthology.orchestra.run.smithi007.stdout:Downloading Packages: 2024-09-18T07:54:53.155 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 4/12 2024-09-18T07:54:53.270 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : grpc-data-1.46.7-10.el9.noarch 5/12 2024-09-18T07:54:53.412 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : abseil-cpp-20211102.0-3.el9.x86_64 6/12 2024-09-18T07:54:53.506 INFO:teuthology.orchestra.run.smithi007.stdout:(1/37): ceph-19.3.0-4983.ge7fb7b56.el9.x86_64.r 46 kB/s | 6.6 kB 00:00 2024-09-18T07:54:53.511 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : python3-grpcio-1.46.7-10.el9.x86_64 7/12 2024-09-18T07:54:53.561 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : python3-protobuf-3.14.0-13.el9.noarch 8/12 2024-09-18T07:54:53.597 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : python3-grpcio-tools-1.46.7-10.el9.x86_64 9/12 2024-09-18T07:54:53.629 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : ceph-prometheus-alerts-2:19.3.0-4983.ge7fb7b56.el9 10/12 2024-09-18T07:54:53.679 INFO:teuthology.orchestra.run.smithi007.stdout:(2/37): ceph-mgr-19.3.0-4983.ge7fb7b56.el9.x86_ 4.8 MB/s | 1.5 MB 00:00 2024-09-18T07:54:53.729 INFO:teuthology.orchestra.run.smithi007.stdout:(3/37): ceph-mds-19.3.0-4983.ge7fb7b56.el9.x86_ 6.5 MB/s | 2.4 MB 00:00 2024-09-18T07:54:53.813 INFO:teuthology.orchestra.run.smithi007.stdout:(4/37): ceph-mgr-modules-core-19.3.0-4983.ge7fb 3.4 MB/s | 291 kB 00:00 2024-09-18T07:54:53.988 INFO:teuthology.orchestra.run.smithi007.stdout:(5/37): python3-cffi-1.14.5-5.el9.x86_64.rpm 1.4 MB/s | 253 kB 00:00 2024-09-18T07:54:54.088 INFO:teuthology.orchestra.run.smithi007.stdout:(6/37): ceph-mon-19.3.0-4983.ge7fb7b56.el9.x86_ 8.2 MB/s | 4.8 MB 00:00 2024-09-18T07:54:54.272 INFO:teuthology.orchestra.run.smithi007.stdout:(7/37): ceph-osd-19.3.0-4983.ge7fb7b56.el9.x86_ 29 MB/s | 17 MB 00:00 2024-09-18T07:54:54.314 INFO:teuthology.orchestra.run.smithi007.stdout:(8/37): python3-cryptography-36.0.1-4.el9.x86_6 3.8 MB/s | 1.2 MB 00:00 2024-09-18T07:54:54.339 INFO:teuthology.orchestra.run.smithi007.stdout:(9/37): python3-ply-3.11-14.el9.noarch.rpm 425 kB/s | 106 kB 00:00 2024-09-18T07:54:54.381 INFO:teuthology.orchestra.run.smithi007.stdout:(10/37): python3-requests-2.25.1-8.el9.noarch.r 1.8 MB/s | 125 kB 00:00 2024-09-18T07:54:54.440 INFO:teuthology.orchestra.run.smithi007.stdout:(11/37): python3-pycparser-2.20-6.el9.noarch.rp 807 kB/s | 135 kB 00:00 2024-09-18T07:54:54.474 INFO:teuthology.orchestra.run.smithi007.stdout:(12/37): python3-urllib3-1.26.5-6.el9.noarch.rp 1.6 MB/s | 215 kB 00:00 2024-09-18T07:54:54.582 INFO:teuthology.orchestra.run.smithi007.stdout:(13/37): lua-5.4.4-4.el9.x86_64.rpm 940 kB/s | 188 kB 00:00 2024-09-18T07:54:54.607 INFO:teuthology.orchestra.run.smithi007.stdout:(14/37): python3-markupsafe-1.1.1-12.el9.x86_64 260 kB/s | 35 kB 00:00 2024-09-18T07:54:54.633 INFO:teuthology.orchestra.run.smithi007.stdout:(15/37): python3-mako-1.1.4-6.el9.noarch.rpm 895 kB/s | 172 kB 00:00 2024-09-18T07:54:54.658 INFO:teuthology.orchestra.run.smithi007.stdout:(16/37): python3-packaging-20.9-5.el9.noarch.rp 1.0 MB/s | 77 kB 00:00 2024-09-18T07:54:54.683 INFO:teuthology.orchestra.run.smithi007.stdout:(17/37): python3-toml-0.10.2-6.el9.noarch.rpm 831 kB/s | 42 kB 00:00 2024-09-18T07:54:54.709 INFO:teuthology.orchestra.run.smithi007.stdout:(18/37): python3-pytz-2021.1-5.el9.noarch.rpm 505 kB/s | 51 kB 00:00 2024-09-18T07:54:54.752 INFO:teuthology.orchestra.run.smithi007.stdout:(19/37): lua-devel-5.4.4-4.el9.x86_64.rpm 242 kB/s | 22 kB 00:00 2024-09-18T07:54:54.793 INFO:teuthology.orchestra.run.smithi007.stdout:(20/37): python3-bcrypt-3.2.2-1.el9.x86_64.rpm 521 kB/s | 43 kB 00:00 2024-09-18T07:54:54.826 INFO:teuthology.orchestra.run.smithi007.stdout:(21/37): luarocks-3.9.2-5.el9.noarch.rpm 1.0 MB/s | 151 kB 00:00 2024-09-18T07:54:54.851 INFO:teuthology.orchestra.run.smithi007.stdout:(22/37): python3-jaraco-8.2.1-3.el9.noarch.rpm 432 kB/s | 11 kB 00:00 2024-09-18T07:54:54.877 INFO:teuthology.orchestra.run.smithi007.stdout:(23/37): python3-jaraco-classes-3.2.1-5.el9.noa 708 kB/s | 18 kB 00:00 2024-09-18T07:54:54.911 INFO:teuthology.orchestra.run.smithi007.stdout:(24/37): python3-cheroot-8.6.0-4.el9.noarch.rpm 1.1 MB/s | 172 kB 00:00 2024-09-18T07:54:54.936 INFO:teuthology.orchestra.run.smithi007.stdout:(25/37): python3-jaraco-collections-3.0.0-8.el9 395 kB/s | 23 kB 00:00 2024-09-18T07:54:54.961 INFO:teuthology.orchestra.run.smithi007.stdout:(26/37): python3-jaraco-functools-3.5.0-2.el9.n 386 kB/s | 19 kB 00:00 2024-09-18T07:54:54.986 INFO:teuthology.orchestra.run.smithi007.stdout:(27/37): python3-jaraco-text-3.2.0-6.el9.noarch 389 kB/s | 20 kB 00:00 2024-09-18T07:54:55.012 INFO:teuthology.orchestra.run.smithi007.stdout:(28/37): python3-logutils-0.3.5-21.el9.noarch.r 923 kB/s | 46 kB 00:00 2024-09-18T07:54:55.037 INFO:teuthology.orchestra.run.smithi007.stdout:(29/37): python3-cherrypy-18.6.1-2.el9.noarch.r 1.4 MB/s | 358 kB 00:00 2024-09-18T07:54:55.071 INFO:teuthology.orchestra.run.smithi007.stdout:(30/37): python3-more-itertools-8.12.0-2.el9.no 939 kB/s | 79 kB 00:00 2024-09-18T07:54:55.096 INFO:teuthology.orchestra.run.smithi007.stdout:(31/37): python3-portend-3.1.0-2.el9.noarch.rpm 280 kB/s | 16 kB 00:00 2024-09-18T07:54:55.121 INFO:teuthology.orchestra.run.smithi007.stdout:(32/37): python3-pyOpenSSL-21.0.0-1.el9.noarch. 1.8 MB/s | 90 kB 00:00 2024-09-18T07:54:55.147 INFO:teuthology.orchestra.run.smithi007.stdout:(33/37): python3-tempora-5.0.0-2.el9.noarch.rpm 713 kB/s | 36 kB 00:00 2024-09-18T07:54:55.181 INFO:teuthology.orchestra.run.smithi007.stdout:(34/37): python3-pecan-1.4.2-3.el9.noarch.rpm 1.6 MB/s | 272 kB 00:00 2024-09-18T07:54:55.206 INFO:teuthology.orchestra.run.smithi007.stdout:(35/37): python3-zc-lockfile-2.0-10.el9.noarch. 805 kB/s | 20 kB 00:00 2024-09-18T07:54:55.231 INFO:teuthology.orchestra.run.smithi007.stdout:(36/37): python3-webob-1.8.8-2.el9.noarch.rpm 2.1 MB/s | 230 kB 00:00 2024-09-18T07:54:55.256 INFO:teuthology.orchestra.run.smithi007.stdout:(37/37): python3-werkzeug-2.0.3-3.el9.1.noarch. 3.8 MB/s | 427 kB 00:00 2024-09-18T07:54:55.259 INFO:teuthology.orchestra.run.smithi007.stdout:-------------------------------------------------------------------------------- 2024-09-18T07:54:55.260 INFO:teuthology.orchestra.run.smithi007.stdout:Total 14 MB/s | 31 MB 00:02 2024-09-18T07:54:55.413 INFO:teuthology.orchestra.run.smithi007.stdout:Running transaction check 2024-09-18T07:54:55.457 INFO:teuthology.orchestra.run.smithi007.stdout:Transaction check succeeded. 2024-09-18T07:54:55.458 INFO:teuthology.orchestra.run.smithi007.stdout:Running transaction test 2024-09-18T07:54:55.938 INFO:teuthology.orchestra.run.smithi007.stdout:Transaction test succeeded. 2024-09-18T07:54:55.938 INFO:teuthology.orchestra.run.smithi007.stdout:Running transaction 2024-09-18T07:54:57.031 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : ceph-grafana-dashboards-2:19.3.0-4983.ge7fb7b56.el 11/12 2024-09-18T07:54:57.092 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : ceph-mgr-dashboard-2:19.3.0-4983.ge7fb7b56.el9.noa 12/12 2024-09-18T07:54:57.368 INFO:teuthology.orchestra.run.smithi007.stdout: Preparing : 1/1 2024-09-18T07:54:57.425 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 1/37 2024-09-18T07:54:57.470 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 2/37 2024-09-18T07:54:57.503 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : lua-5.4.4-4.el9.x86_64 3/37 2024-09-18T07:54:57.578 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 4/37 2024-09-18T07:54:57.636 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : luarocks-3.9.2-5.el9.noarch 5/37 2024-09-18T07:54:57.669 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 6/37 2024-09-18T07:54:57.766 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 7/37 2024-09-18T07:54:57.850 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 8/37 2024-09-18T07:54:57.914 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : python3-webob-1.8.8-2.el9.noarch 9/37 2024-09-18T07:54:57.948 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 10/37 2024-09-18T07:54:57.978 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 11/37 2024-09-18T07:54:58.012 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 12/37 2024-09-18T07:54:58.046 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 13/37 2024-09-18T07:54:58.087 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 14/37 2024-09-18T07:54:58.130 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 15/37 2024-09-18T07:54:58.165 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 16/37 2024-09-18T07:54:58.214 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 17/37 2024-09-18T07:54:58.254 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : python3-packaging-20.9-5.el9.noarch 18/37 2024-09-18T07:54:58.331 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 19/37 2024-09-18T07:54:58.500 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 20/37 2024-09-18T07:54:58.590 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 21/37 2024-09-18T07:54:58.649 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : python3-urllib3-1.26.5-6.el9.noarch 22/37 2024-09-18T07:54:58.687 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 23/37 2024-09-18T07:54:58.745 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : python3-ply-3.11-14.el9.noarch 24/37 2024-09-18T07:54:58.810 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 25/37 2024-09-18T07:54:59.007 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 26/37 2024-09-18T07:54:59.059 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 27/37 2024-09-18T07:54:59.062 INFO:teuthology.orchestra.run.smithi050.stdout: Running scriptlet: ceph-mgr-dashboard-2:19.3.0-4983.ge7fb7b56.el9.noa 12/12 2024-09-18T07:54:59.062 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : ceph-grafana-dashboards-2:19.3.0-4983.ge7fb7b56.el 1/12 2024-09-18T07:54:59.062 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : ceph-mgr-dashboard-2:19.3.0-4983.ge7fb7b56.el9.noa 2/12 2024-09-18T07:54:59.062 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : ceph-prometheus-alerts-2:19.3.0-4983.ge7fb7b56.el9 3/12 2024-09-18T07:54:59.062 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : protobuf-3.14.0-13.el9.x86_64 4/12 2024-09-18T07:54:59.062 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : python3-protobuf-3.14.0-13.el9.noarch 5/12 2024-09-18T07:54:59.062 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : protobuf-compiler-3.14.0-13.el9.x86_64 6/12 2024-09-18T07:54:59.062 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : abseil-cpp-20211102.0-3.el9.x86_64 7/12 2024-09-18T07:54:59.062 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : grpc-data-1.46.7-10.el9.noarch 8/12 2024-09-18T07:54:59.062 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : python3-grpcio-1.46.7-10.el9.x86_64 9/12 2024-09-18T07:54:59.062 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : python3-grpcio-tools-1.46.7-10.el9.x86_64 10/12 2024-09-18T07:54:59.062 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 11/12 2024-09-18T07:54:59.140 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 28/37 2024-09-18T07:54:59.238 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 29/37 2024-09-18T07:54:59.274 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 30/37 2024-09-18T07:54:59.406 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 12/12 2024-09-18T07:54:59.406 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-09-18T07:54:59.406 INFO:teuthology.orchestra.run.smithi050.stdout:Installed: 2024-09-18T07:54:59.406 INFO:teuthology.orchestra.run.smithi050.stdout: abseil-cpp-20211102.0-3.el9.x86_64 2024-09-18T07:54:59.406 INFO:teuthology.orchestra.run.smithi050.stdout: ceph-grafana-dashboards-2:19.3.0-4983.ge7fb7b56.el9.noarch 2024-09-18T07:54:59.406 INFO:teuthology.orchestra.run.smithi050.stdout: ceph-mgr-dashboard-2:19.3.0-4983.ge7fb7b56.el9.noarch 2024-09-18T07:54:59.406 INFO:teuthology.orchestra.run.smithi050.stdout: ceph-prometheus-alerts-2:19.3.0-4983.ge7fb7b56.el9.noarch 2024-09-18T07:54:59.406 INFO:teuthology.orchestra.run.smithi050.stdout: grpc-data-1.46.7-10.el9.noarch 2024-09-18T07:54:59.407 INFO:teuthology.orchestra.run.smithi050.stdout: protobuf-3.14.0-13.el9.x86_64 2024-09-18T07:54:59.407 INFO:teuthology.orchestra.run.smithi050.stdout: protobuf-compiler-3.14.0-13.el9.x86_64 2024-09-18T07:54:59.407 INFO:teuthology.orchestra.run.smithi050.stdout: python3-grpcio-1.46.7-10.el9.x86_64 2024-09-18T07:54:59.407 INFO:teuthology.orchestra.run.smithi050.stdout: python3-grpcio-tools-1.46.7-10.el9.x86_64 2024-09-18T07:54:59.407 INFO:teuthology.orchestra.run.smithi050.stdout: python3-protobuf-3.14.0-13.el9.noarch 2024-09-18T07:54:59.407 INFO:teuthology.orchestra.run.smithi050.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-09-18T07:54:59.407 INFO:teuthology.orchestra.run.smithi050.stdout: python3-routes-2.5.1-5.el9.noarch 2024-09-18T07:54:59.407 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-09-18T07:54:59.407 INFO:teuthology.orchestra.run.smithi050.stdout:Complete! 2024-09-18T07:54:59.412 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 31/37 2024-09-18T07:54:59.587 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : ceph-mgr-modules-core-2:19.3.0-4983.ge7fb7b56.el9. 32/37 2024-09-18T07:54:59.618 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : ceph-mgr-2:19.3.0-4983.ge7fb7b56.el9.x86_64 33/37 2024-09-18T07:54:59.645 INFO:teuthology.orchestra.run.smithi007.stdout: Running scriptlet: ceph-mgr-2:19.3.0-4983.ge7fb7b56.el9.x86_64 33/37 2024-09-18T07:54:59.646 INFO:teuthology.orchestra.run.smithi007.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-18T07:54:59.646 INFO:teuthology.orchestra.run.smithi007.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-09-18T07:54:59.646 INFO:teuthology.orchestra.run.smithi007.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-09-18T07:54:59.646 INFO:teuthology.orchestra.run.smithi007.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-09-18T07:54:59.646 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-09-18T07:54:59.667 DEBUG:teuthology.orchestra.run.smithi050:> sudo yum -y install ceph-mgr-diskprediction-local 2024-09-18T07:55:00.284 INFO:teuthology.orchestra.run.smithi050.stdout:Last metadata expiration check: 0:01:10 ago on Wed 18 Sep 2024 07:53:50 AM UTC. 2024-09-18T07:55:00.429 INFO:teuthology.orchestra.run.smithi050.stdout:Dependencies resolved. 2024-09-18T07:55:00.430 INFO:teuthology.orchestra.run.smithi050.stdout:======================================================================================= 2024-09-18T07:55:00.430 INFO:teuthology.orchestra.run.smithi050.stdout: Package Arch Version Repository Size 2024-09-18T07:55:00.430 INFO:teuthology.orchestra.run.smithi050.stdout:======================================================================================= 2024-09-18T07:55:00.430 INFO:teuthology.orchestra.run.smithi050.stdout:Installing: 2024-09-18T07:55:00.430 INFO:teuthology.orchestra.run.smithi050.stdout: ceph-mgr-diskprediction-local noarch 2:19.3.0-4983.ge7fb7b56.el9 ceph-noarch 7.4 M 2024-09-18T07:55:00.430 INFO:teuthology.orchestra.run.smithi050.stdout:Installing dependencies: 2024-09-18T07:55:00.430 INFO:teuthology.orchestra.run.smithi050.stdout: flexiblas x86_64 3.0.4-8.el9 appstream 33 k 2024-09-18T07:55:00.430 INFO:teuthology.orchestra.run.smithi050.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 appstream 3.0 M 2024-09-18T07:55:00.430 INFO:teuthology.orchestra.run.smithi050.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 appstream 18 k 2024-09-18T07:55:00.431 INFO:teuthology.orchestra.run.smithi050.stdout: libgfortran x86_64 11.5.0-2.el9 baseos 801 k 2024-09-18T07:55:00.431 INFO:teuthology.orchestra.run.smithi050.stdout: libquadmath x86_64 11.5.0-2.el9 baseos 191 k 2024-09-18T07:55:00.431 INFO:teuthology.orchestra.run.smithi050.stdout: openblas x86_64 0.3.26-2.el9 appstream 38 k 2024-09-18T07:55:00.431 INFO:teuthology.orchestra.run.smithi050.stdout: openblas-openmp x86_64 0.3.26-2.el9 appstream 4.9 M 2024-09-18T07:55:00.431 INFO:teuthology.orchestra.run.smithi050.stdout: python3-devel x86_64 3.9.19-8.el9 appstream 245 k 2024-09-18T07:55:00.431 INFO:teuthology.orchestra.run.smithi050.stdout: python3-numpy x86_64 1:1.20.1-5.el9 appstream 5.1 M 2024-09-18T07:55:00.431 INFO:teuthology.orchestra.run.smithi050.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 appstream 250 k 2024-09-18T07:55:00.431 INFO:teuthology.orchestra.run.smithi050.stdout: python3-scipy x86_64 1.6.2-8.el9 appstream 16 M 2024-09-18T07:55:00.431 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-09-18T07:55:00.431 INFO:teuthology.orchestra.run.smithi050.stdout:Transaction Summary 2024-09-18T07:55:00.431 INFO:teuthology.orchestra.run.smithi050.stdout:======================================================================================= 2024-09-18T07:55:00.431 INFO:teuthology.orchestra.run.smithi050.stdout:Install 12 Packages 2024-09-18T07:55:00.431 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-09-18T07:55:00.432 INFO:teuthology.orchestra.run.smithi050.stdout:Total download size: 38 M 2024-09-18T07:55:00.432 INFO:teuthology.orchestra.run.smithi050.stdout:Installed size: 212 M 2024-09-18T07:55:00.432 INFO:teuthology.orchestra.run.smithi050.stdout:Downloading Packages: 2024-09-18T07:55:01.008 INFO:teuthology.orchestra.run.smithi050.stdout:(1/12): libquadmath-11.5.0-2.el9.x86_64.rpm 1.0 MB/s | 191 kB 00:00 2024-09-18T07:55:01.041 INFO:teuthology.orchestra.run.smithi050.stdout:(2/12): libgfortran-11.5.0-2.el9.x86_64.rpm 3.7 MB/s | 801 kB 00:00 2024-09-18T07:55:01.125 INFO:teuthology.orchestra.run.smithi050.stdout:(3/12): flexiblas-3.0.4-8.el9.x86_64.rpm 280 kB/s | 33 kB 00:00 2024-09-18T07:55:01.175 INFO:teuthology.orchestra.run.smithi050.stdout:(4/12): flexiblas-openblas-openmp-3.0.4-8.el9.x 364 kB/s | 18 kB 00:00 2024-09-18T07:55:01.202 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : ceph-osd-2:19.3.0-4983.ge7fb7b56.el9.x86_64 34/37 2024-09-18T07:55:01.225 INFO:teuthology.orchestra.run.smithi050.stdout:(5/12): openblas-0.3.26-2.el9.x86_64.rpm 770 kB/s | 38 kB 00:00 2024-09-18T07:55:01.231 INFO:teuthology.orchestra.run.smithi007.stdout: Running scriptlet: ceph-osd-2:19.3.0-4983.ge7fb7b56.el9.x86_64 34/37 2024-09-18T07:55:01.231 INFO:teuthology.orchestra.run.smithi007.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-18T07:55:01.231 INFO:teuthology.orchestra.run.smithi007.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-09-18T07:55:01.231 INFO:teuthology.orchestra.run.smithi007.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-09-18T07:55:01.231 INFO:teuthology.orchestra.run.smithi007.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-09-18T07:55:01.231 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-09-18T07:55:01.326 INFO:teuthology.orchestra.run.smithi050.stdout:(6/12): ceph-mgr-diskprediction-local-19.3.0-49 15 MB/s | 7.4 MB 00:00 2024-09-18T07:55:01.535 INFO:teuthology.orchestra.run.smithi050.stdout:(7/12): python3-devel-3.9.19-8.el9.x86_64.rpm 1.1 MB/s | 245 kB 00:00 2024-09-18T07:55:01.719 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : ceph-mon-2:19.3.0-4983.ge7fb7b56.el9.x86_64 35/37 2024-09-18T07:55:01.744 INFO:teuthology.orchestra.run.smithi007.stdout: Running scriptlet: ceph-mon-2:19.3.0-4983.ge7fb7b56.el9.x86_64 35/37 2024-09-18T07:55:01.744 INFO:teuthology.orchestra.run.smithi007.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-18T07:55:01.745 INFO:teuthology.orchestra.run.smithi007.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-09-18T07:55:01.745 INFO:teuthology.orchestra.run.smithi007.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-09-18T07:55:01.745 INFO:teuthology.orchestra.run.smithi007.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-09-18T07:55:01.745 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-09-18T07:55:01.987 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : ceph-mds-2:19.3.0-4983.ge7fb7b56.el9.x86_64 36/37 2024-09-18T07:55:02.012 INFO:teuthology.orchestra.run.smithi007.stdout: Running scriptlet: ceph-mds-2:19.3.0-4983.ge7fb7b56.el9.x86_64 36/37 2024-09-18T07:55:02.012 INFO:teuthology.orchestra.run.smithi007.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-18T07:55:02.012 INFO:teuthology.orchestra.run.smithi007.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-09-18T07:55:02.012 INFO:teuthology.orchestra.run.smithi007.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-09-18T07:55:02.012 INFO:teuthology.orchestra.run.smithi007.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-09-18T07:55:02.012 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-09-18T07:55:02.079 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : ceph-2:19.3.0-4983.ge7fb7b56.el9.x86_64 37/37 2024-09-18T07:55:02.143 INFO:teuthology.orchestra.run.smithi050.stdout:(8/12): flexiblas-netlib-3.0.4-8.el9.x86_64.rpm 2.7 MB/s | 3.0 MB 00:01 2024-09-18T07:55:02.285 INFO:teuthology.orchestra.run.smithi050.stdout:(9/12): python3-numpy-f2py-1.20.1-5.el9.x86_64. 1.7 MB/s | 250 kB 00:00 2024-09-18T07:55:03.211 INFO:teuthology.orchestra.run.smithi050.stdout:(10/12): openblas-openmp-0.3.26-2.el9.x86_64.rp 2.5 MB/s | 4.9 MB 00:01 2024-09-18T07:55:03.461 INFO:teuthology.orchestra.run.smithi050.stdout:(11/12): python3-numpy-1.20.1-5.el9.x86_64.rpm 2.6 MB/s | 5.1 MB 00:01 2024-09-18T07:55:04.024 INFO:teuthology.orchestra.run.smithi007.stdout: Running scriptlet: ceph-2:19.3.0-4983.ge7fb7b56.el9.x86_64 37/37 2024-09-18T07:55:04.024 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : ceph-2:19.3.0-4983.ge7fb7b56.el9.x86_64 1/37 2024-09-18T07:55:04.024 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : ceph-mds-2:19.3.0-4983.ge7fb7b56.el9.x86_64 2/37 2024-09-18T07:55:04.024 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : ceph-mgr-2:19.3.0-4983.ge7fb7b56.el9.x86_64 3/37 2024-09-18T07:55:04.024 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : ceph-mon-2:19.3.0-4983.ge7fb7b56.el9.x86_64 4/37 2024-09-18T07:55:04.024 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : ceph-osd-2:19.3.0-4983.ge7fb7b56.el9.x86_64 5/37 2024-09-18T07:55:04.024 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : ceph-mgr-modules-core-2:19.3.0-4983.ge7fb7b56.el9. 6/37 2024-09-18T07:55:04.024 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 7/37 2024-09-18T07:55:04.025 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 8/37 2024-09-18T07:55:04.025 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : python3-ply-3.11-14.el9.noarch 9/37 2024-09-18T07:55:04.025 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 10/37 2024-09-18T07:55:04.025 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 11/37 2024-09-18T07:55:04.025 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : python3-urllib3-1.26.5-6.el9.noarch 12/37 2024-09-18T07:55:04.025 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : lua-5.4.4-4.el9.x86_64 13/37 2024-09-18T07:55:04.025 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 14/37 2024-09-18T07:55:04.025 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 15/37 2024-09-18T07:55:04.025 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 16/37 2024-09-18T07:55:04.025 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 17/37 2024-09-18T07:55:04.025 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 18/37 2024-09-18T07:55:04.025 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 19/37 2024-09-18T07:55:04.025 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : luarocks-3.9.2-5.el9.noarch 20/37 2024-09-18T07:55:04.026 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 21/37 2024-09-18T07:55:04.026 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 22/37 2024-09-18T07:55:04.026 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 23/37 2024-09-18T07:55:04.026 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 24/37 2024-09-18T07:55:04.026 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 25/37 2024-09-18T07:55:04.026 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 26/37 2024-09-18T07:55:04.026 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 27/37 2024-09-18T07:55:04.026 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 28/37 2024-09-18T07:55:04.026 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 29/37 2024-09-18T07:55:04.026 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 30/37 2024-09-18T07:55:04.026 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 31/37 2024-09-18T07:55:04.026 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 32/37 2024-09-18T07:55:04.026 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 33/37 2024-09-18T07:55:04.027 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 34/37 2024-09-18T07:55:04.027 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : python3-webob-1.8.8-2.el9.noarch 35/37 2024-09-18T07:55:04.027 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 36/37 2024-09-18T07:55:05.175 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 37/37 2024-09-18T07:55:05.176 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-09-18T07:55:05.176 INFO:teuthology.orchestra.run.smithi007.stdout:Installed: 2024-09-18T07:55:05.176 INFO:teuthology.orchestra.run.smithi007.stdout: ceph-2:19.3.0-4983.ge7fb7b56.el9.x86_64 2024-09-18T07:55:05.176 INFO:teuthology.orchestra.run.smithi007.stdout: ceph-mds-2:19.3.0-4983.ge7fb7b56.el9.x86_64 2024-09-18T07:55:05.176 INFO:teuthology.orchestra.run.smithi007.stdout: ceph-mgr-2:19.3.0-4983.ge7fb7b56.el9.x86_64 2024-09-18T07:55:05.176 INFO:teuthology.orchestra.run.smithi007.stdout: ceph-mgr-modules-core-2:19.3.0-4983.ge7fb7b56.el9.noarch 2024-09-18T07:55:05.176 INFO:teuthology.orchestra.run.smithi007.stdout: ceph-mon-2:19.3.0-4983.ge7fb7b56.el9.x86_64 2024-09-18T07:55:05.176 INFO:teuthology.orchestra.run.smithi007.stdout: ceph-osd-2:19.3.0-4983.ge7fb7b56.el9.x86_64 2024-09-18T07:55:05.176 INFO:teuthology.orchestra.run.smithi007.stdout: lua-5.4.4-4.el9.x86_64 2024-09-18T07:55:05.176 INFO:teuthology.orchestra.run.smithi007.stdout: lua-devel-5.4.4-4.el9.x86_64 2024-09-18T07:55:05.176 INFO:teuthology.orchestra.run.smithi007.stdout: luarocks-3.9.2-5.el9.noarch 2024-09-18T07:55:05.177 INFO:teuthology.orchestra.run.smithi007.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-09-18T07:55:05.177 INFO:teuthology.orchestra.run.smithi007.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-09-18T07:55:05.177 INFO:teuthology.orchestra.run.smithi007.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-09-18T07:55:05.177 INFO:teuthology.orchestra.run.smithi007.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-09-18T07:55:05.177 INFO:teuthology.orchestra.run.smithi007.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-09-18T07:55:05.177 INFO:teuthology.orchestra.run.smithi007.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-09-18T07:55:05.177 INFO:teuthology.orchestra.run.smithi007.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-09-18T07:55:05.177 INFO:teuthology.orchestra.run.smithi007.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-09-18T07:55:05.177 INFO:teuthology.orchestra.run.smithi007.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-09-18T07:55:05.177 INFO:teuthology.orchestra.run.smithi007.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-09-18T07:55:05.177 INFO:teuthology.orchestra.run.smithi007.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-09-18T07:55:05.177 INFO:teuthology.orchestra.run.smithi007.stdout: python3-mako-1.1.4-6.el9.noarch 2024-09-18T07:55:05.178 INFO:teuthology.orchestra.run.smithi007.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-09-18T07:55:05.178 INFO:teuthology.orchestra.run.smithi007.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-09-18T07:55:05.178 INFO:teuthology.orchestra.run.smithi007.stdout: python3-packaging-20.9-5.el9.noarch 2024-09-18T07:55:05.178 INFO:teuthology.orchestra.run.smithi007.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-09-18T07:55:05.178 INFO:teuthology.orchestra.run.smithi007.stdout: python3-ply-3.11-14.el9.noarch 2024-09-18T07:55:05.178 INFO:teuthology.orchestra.run.smithi007.stdout: python3-portend-3.1.0-2.el9.noarch 2024-09-18T07:55:05.178 INFO:teuthology.orchestra.run.smithi007.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-09-18T07:55:05.178 INFO:teuthology.orchestra.run.smithi007.stdout: python3-pycparser-2.20-6.el9.noarch 2024-09-18T07:55:05.178 INFO:teuthology.orchestra.run.smithi007.stdout: python3-pytz-2021.1-5.el9.noarch 2024-09-18T07:55:05.178 INFO:teuthology.orchestra.run.smithi007.stdout: python3-requests-2.25.1-8.el9.noarch 2024-09-18T07:55:05.178 INFO:teuthology.orchestra.run.smithi007.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-09-18T07:55:05.178 INFO:teuthology.orchestra.run.smithi007.stdout: python3-toml-0.10.2-6.el9.noarch 2024-09-18T07:55:05.179 INFO:teuthology.orchestra.run.smithi007.stdout: python3-urllib3-1.26.5-6.el9.noarch 2024-09-18T07:55:05.179 INFO:teuthology.orchestra.run.smithi007.stdout: python3-webob-1.8.8-2.el9.noarch 2024-09-18T07:55:05.179 INFO:teuthology.orchestra.run.smithi007.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-09-18T07:55:05.179 INFO:teuthology.orchestra.run.smithi007.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-09-18T07:55:05.179 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-09-18T07:55:05.179 INFO:teuthology.orchestra.run.smithi007.stdout:Complete! 2024-09-18T07:55:05.566 DEBUG:teuthology.orchestra.run.smithi007:> sudo yum -y install ceph-base 2024-09-18T07:55:06.185 INFO:teuthology.orchestra.run.smithi007.stdout:Last metadata expiration check: 0:00:54 ago on Wed 18 Sep 2024 07:54:12 AM UTC. 2024-09-18T07:55:06.278 INFO:teuthology.orchestra.run.smithi007.stdout:Package ceph-base-2:19.3.0-4983.ge7fb7b56.el9.x86_64 is already installed. 2024-09-18T07:55:06.322 INFO:teuthology.orchestra.run.smithi007.stdout:Dependencies resolved. 2024-09-18T07:55:06.323 INFO:teuthology.orchestra.run.smithi007.stdout:Nothing to do. 2024-09-18T07:55:06.323 INFO:teuthology.orchestra.run.smithi007.stdout:Complete! 2024-09-18T07:55:06.394 DEBUG:teuthology.orchestra.run.smithi007:> sudo yum -y install cephadm 2024-09-18T07:55:06.995 INFO:teuthology.orchestra.run.smithi007.stdout:Last metadata expiration check: 0:00:54 ago on Wed 18 Sep 2024 07:54:12 AM UTC. 2024-09-18T07:55:07.131 INFO:teuthology.orchestra.run.smithi007.stdout:Dependencies resolved. 2024-09-18T07:55:07.131 INFO:teuthology.orchestra.run.smithi007.stdout:================================================================================ 2024-09-18T07:55:07.132 INFO:teuthology.orchestra.run.smithi007.stdout: Package Arch Version Repository Size 2024-09-18T07:55:07.132 INFO:teuthology.orchestra.run.smithi007.stdout:================================================================================ 2024-09-18T07:55:07.132 INFO:teuthology.orchestra.run.smithi007.stdout:Installing: 2024-09-18T07:55:07.132 INFO:teuthology.orchestra.run.smithi007.stdout: cephadm noarch 2:19.3.0-4983.ge7fb7b56.el9 ceph-noarch 784 k 2024-09-18T07:55:07.132 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-09-18T07:55:07.132 INFO:teuthology.orchestra.run.smithi007.stdout:Transaction Summary 2024-09-18T07:55:07.132 INFO:teuthology.orchestra.run.smithi007.stdout:================================================================================ 2024-09-18T07:55:07.132 INFO:teuthology.orchestra.run.smithi007.stdout:Install 1 Package 2024-09-18T07:55:07.132 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-09-18T07:55:07.132 INFO:teuthology.orchestra.run.smithi007.stdout:Total download size: 784 k 2024-09-18T07:55:07.132 INFO:teuthology.orchestra.run.smithi007.stdout:Installed size: 791 k 2024-09-18T07:55:07.133 INFO:teuthology.orchestra.run.smithi007.stdout:Downloading Packages: 2024-09-18T07:55:07.420 INFO:teuthology.orchestra.run.smithi007.stdout:cephadm-19.3.0-4983.ge7fb7b56.el9.noarch.rpm 2.7 MB/s | 784 kB 00:00 2024-09-18T07:55:07.421 INFO:teuthology.orchestra.run.smithi007.stdout:-------------------------------------------------------------------------------- 2024-09-18T07:55:07.421 INFO:teuthology.orchestra.run.smithi007.stdout:Total 2.7 MB/s | 784 kB 00:00 2024-09-18T07:55:07.421 INFO:teuthology.orchestra.run.smithi007.stdout:Running transaction check 2024-09-18T07:55:07.427 INFO:teuthology.orchestra.run.smithi007.stdout:Transaction check succeeded. 2024-09-18T07:55:07.427 INFO:teuthology.orchestra.run.smithi007.stdout:Running transaction test 2024-09-18T07:55:07.433 INFO:teuthology.orchestra.run.smithi007.stdout:Transaction test succeeded. 2024-09-18T07:55:07.433 INFO:teuthology.orchestra.run.smithi007.stdout:Running transaction 2024-09-18T07:55:07.505 INFO:teuthology.orchestra.run.smithi007.stdout: Preparing : 1/1 2024-09-18T07:55:07.968 INFO:teuthology.orchestra.run.smithi007.stdout: Running scriptlet: cephadm-2:19.3.0-4983.ge7fb7b56.el9.noarch 1/1 2024-09-18T07:55:08.060 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : cephadm-2:19.3.0-4983.ge7fb7b56.el9.noarch 1/1 2024-09-18T07:55:08.467 INFO:teuthology.orchestra.run.smithi007.stdout: Running scriptlet: cephadm-2:19.3.0-4983.ge7fb7b56.el9.noarch 1/1 2024-09-18T07:55:08.767 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : cephadm-2:19.3.0-4983.ge7fb7b56.el9.noarch 1/1 2024-09-18T07:55:08.768 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-09-18T07:55:08.768 INFO:teuthology.orchestra.run.smithi007.stdout:Installed: 2024-09-18T07:55:08.768 INFO:teuthology.orchestra.run.smithi007.stdout: cephadm-2:19.3.0-4983.ge7fb7b56.el9.noarch 2024-09-18T07:55:08.768 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-09-18T07:55:08.768 INFO:teuthology.orchestra.run.smithi007.stdout:Complete! 2024-09-18T07:55:09.134 DEBUG:teuthology.orchestra.run.smithi007:> sudo yum -y install ceph-immutable-object-cache 2024-09-18T07:55:09.619 INFO:teuthology.orchestra.run.smithi050.stdout:(12/12): python3-scipy-1.6.2-8.el9.x86_64.rpm 2.2 MB/s | 16 MB 00:07 2024-09-18T07:55:09.620 INFO:teuthology.orchestra.run.smithi050.stdout:-------------------------------------------------------------------------------- 2024-09-18T07:55:09.620 INFO:teuthology.orchestra.run.smithi050.stdout:Total 4.2 MB/s | 38 MB 00:09 2024-09-18T07:55:09.740 INFO:teuthology.orchestra.run.smithi007.stdout:Last metadata expiration check: 0:00:57 ago on Wed 18 Sep 2024 07:54:12 AM UTC. 2024-09-18T07:55:09.881 INFO:teuthology.orchestra.run.smithi007.stdout:Dependencies resolved. 2024-09-18T07:55:09.881 INFO:teuthology.orchestra.run.smithi007.stdout:================================================================================ 2024-09-18T07:55:09.881 INFO:teuthology.orchestra.run.smithi007.stdout: Package Arch Version Repo Size 2024-09-18T07:55:09.882 INFO:teuthology.orchestra.run.smithi007.stdout:================================================================================ 2024-09-18T07:55:09.882 INFO:teuthology.orchestra.run.smithi007.stdout:Installing: 2024-09-18T07:55:09.882 INFO:teuthology.orchestra.run.smithi007.stdout: ceph-immutable-object-cache x86_64 2:19.3.0-4983.ge7fb7b56.el9 ceph 145 k 2024-09-18T07:55:09.882 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-09-18T07:55:09.882 INFO:teuthology.orchestra.run.smithi007.stdout:Transaction Summary 2024-09-18T07:55:09.882 INFO:teuthology.orchestra.run.smithi007.stdout:================================================================================ 2024-09-18T07:55:09.882 INFO:teuthology.orchestra.run.smithi007.stdout:Install 1 Package 2024-09-18T07:55:09.882 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-09-18T07:55:09.883 INFO:teuthology.orchestra.run.smithi007.stdout:Total download size: 145 k 2024-09-18T07:55:09.883 INFO:teuthology.orchestra.run.smithi007.stdout:Installed size: 432 k 2024-09-18T07:55:09.883 INFO:teuthology.orchestra.run.smithi007.stdout:Downloading Packages: 2024-09-18T07:55:09.993 INFO:teuthology.orchestra.run.smithi050.stdout:Running transaction check 2024-09-18T07:55:10.020 INFO:teuthology.orchestra.run.smithi050.stdout:Transaction check succeeded. 2024-09-18T07:55:10.020 INFO:teuthology.orchestra.run.smithi050.stdout:Running transaction test 2024-09-18T07:55:10.094 INFO:teuthology.orchestra.run.smithi007.stdout:ceph-immutable-object-cache-19.3.0-4983.ge7fb7b 690 kB/s | 145 kB 00:00 2024-09-18T07:55:10.094 INFO:teuthology.orchestra.run.smithi007.stdout:-------------------------------------------------------------------------------- 2024-09-18T07:55:10.094 INFO:teuthology.orchestra.run.smithi007.stdout:Total 684 kB/s | 145 kB 00:00 2024-09-18T07:55:10.094 INFO:teuthology.orchestra.run.smithi007.stdout:Running transaction check 2024-09-18T07:55:10.103 INFO:teuthology.orchestra.run.smithi007.stdout:Transaction check succeeded. 2024-09-18T07:55:10.103 INFO:teuthology.orchestra.run.smithi007.stdout:Running transaction test 2024-09-18T07:55:10.164 INFO:teuthology.orchestra.run.smithi007.stdout:Transaction test succeeded. 2024-09-18T07:55:10.164 INFO:teuthology.orchestra.run.smithi007.stdout:Running transaction 2024-09-18T07:55:10.312 INFO:teuthology.orchestra.run.smithi007.stdout: Preparing : 1/1 2024-09-18T07:55:10.386 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : ceph-immutable-object-cache-2:19.3.0-4983.ge7fb7b56. 1/1 2024-09-18T07:55:10.413 INFO:teuthology.orchestra.run.smithi007.stdout: Running scriptlet: ceph-immutable-object-cache-2:19.3.0-4983.ge7fb7b56. 1/1 2024-09-18T07:55:10.413 INFO:teuthology.orchestra.run.smithi007.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-18T07:55:10.413 INFO:teuthology.orchestra.run.smithi007.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-09-18T07:55:10.413 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-09-18T07:55:10.484 INFO:teuthology.orchestra.run.smithi050.stdout:Transaction test succeeded. 2024-09-18T07:55:10.485 INFO:teuthology.orchestra.run.smithi050.stdout:Running transaction 2024-09-18T07:55:11.272 INFO:teuthology.orchestra.run.smithi050.stdout: Preparing : 1/1 2024-09-18T07:55:11.347 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 1/12 2024-09-18T07:55:11.384 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : libquadmath-11.5.0-2.el9.x86_64 2/12 2024-09-18T07:55:11.410 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : ceph-immutable-object-cache-2:19.3.0-4983.ge7fb7b56. 1/1 2024-09-18T07:55:11.410 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-09-18T07:55:11.410 INFO:teuthology.orchestra.run.smithi007.stdout:Installed: 2024-09-18T07:55:11.410 INFO:teuthology.orchestra.run.smithi007.stdout: ceph-immutable-object-cache-2:19.3.0-4983.ge7fb7b56.el9.x86_64 2024-09-18T07:55:11.410 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-09-18T07:55:11.411 INFO:teuthology.orchestra.run.smithi007.stdout:Complete! 2024-09-18T07:55:11.505 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : libgfortran-11.5.0-2.el9.x86_64 3/12 2024-09-18T07:55:11.544 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : python3-devel-3.9.19-8.el9.x86_64 4/12 2024-09-18T07:55:11.641 DEBUG:teuthology.orchestra.run.smithi007:> sudo yum -y install ceph-mgr 2024-09-18T07:55:11.735 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : openblas-0.3.26-2.el9.x86_64 5/12 2024-09-18T07:55:11.775 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : openblas-openmp-0.3.26-2.el9.x86_64 6/12 2024-09-18T07:55:11.854 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 7/12 2024-09-18T07:55:12.255 INFO:teuthology.orchestra.run.smithi007.stdout:Last metadata expiration check: 0:01:00 ago on Wed 18 Sep 2024 07:54:12 AM UTC. 2024-09-18T07:55:12.348 INFO:teuthology.orchestra.run.smithi007.stdout:Package ceph-mgr-2:19.3.0-4983.ge7fb7b56.el9.x86_64 is already installed. 2024-09-18T07:55:12.393 INFO:teuthology.orchestra.run.smithi007.stdout:Dependencies resolved. 2024-09-18T07:55:12.394 INFO:teuthology.orchestra.run.smithi007.stdout:Nothing to do. 2024-09-18T07:55:12.394 INFO:teuthology.orchestra.run.smithi007.stdout:Complete! 2024-09-18T07:55:12.460 DEBUG:teuthology.orchestra.run.smithi007:> sudo yum -y install ceph-mgr-dashboard 2024-09-18T07:55:12.547 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 8/12 2024-09-18T07:55:12.668 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 9/12 2024-09-18T07:55:13.066 INFO:teuthology.orchestra.run.smithi007.stdout:Last metadata expiration check: 0:01:01 ago on Wed 18 Sep 2024 07:54:12 AM UTC. 2024-09-18T07:55:13.204 INFO:teuthology.orchestra.run.smithi007.stdout:Dependencies resolved. 2024-09-18T07:55:13.205 INFO:teuthology.orchestra.run.smithi007.stdout:================================================================================ 2024-09-18T07:55:13.205 INFO:teuthology.orchestra.run.smithi007.stdout: Package Arch Version Repository Size 2024-09-18T07:55:13.206 INFO:teuthology.orchestra.run.smithi007.stdout:================================================================================ 2024-09-18T07:55:13.206 INFO:teuthology.orchestra.run.smithi007.stdout:Installing: 2024-09-18T07:55:13.206 INFO:teuthology.orchestra.run.smithi007.stdout: ceph-mgr-dashboard noarch 2:19.3.0-4983.ge7fb7b56.el9 ceph-noarch 14 M 2024-09-18T07:55:13.206 INFO:teuthology.orchestra.run.smithi007.stdout:Installing dependencies: 2024-09-18T07:55:13.206 INFO:teuthology.orchestra.run.smithi007.stdout: abseil-cpp x86_64 20211102.0-3.el9 epel 548 k 2024-09-18T07:55:13.206 INFO:teuthology.orchestra.run.smithi007.stdout: ceph-grafana-dashboards noarch 2:19.3.0-4983.ge7fb7b56.el9 ceph-noarch 34 k 2024-09-18T07:55:13.206 INFO:teuthology.orchestra.run.smithi007.stdout: ceph-prometheus-alerts noarch 2:19.3.0-4983.ge7fb7b56.el9 ceph-noarch 16 k 2024-09-18T07:55:13.206 INFO:teuthology.orchestra.run.smithi007.stdout: grpc-data noarch 1.46.7-10.el9 epel 19 k 2024-09-18T07:55:13.206 INFO:teuthology.orchestra.run.smithi007.stdout: protobuf x86_64 3.14.0-13.el9 appstream 1.0 M 2024-09-18T07:55:13.206 INFO:teuthology.orchestra.run.smithi007.stdout: protobuf-compiler x86_64 3.14.0-13.el9 crb 863 k 2024-09-18T07:55:13.207 INFO:teuthology.orchestra.run.smithi007.stdout: python3-grpcio x86_64 1.46.7-10.el9 epel 2.0 M 2024-09-18T07:55:13.207 INFO:teuthology.orchestra.run.smithi007.stdout: python3-grpcio-tools x86_64 1.46.7-10.el9 epel 144 k 2024-09-18T07:55:13.207 INFO:teuthology.orchestra.run.smithi007.stdout: python3-protobuf noarch 3.14.0-13.el9 appstream 269 k 2024-09-18T07:55:13.207 INFO:teuthology.orchestra.run.smithi007.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-09-18T07:55:13.207 INFO:teuthology.orchestra.run.smithi007.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-09-18T07:55:13.207 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-09-18T07:55:13.207 INFO:teuthology.orchestra.run.smithi007.stdout:Transaction Summary 2024-09-18T07:55:13.207 INFO:teuthology.orchestra.run.smithi007.stdout:================================================================================ 2024-09-18T07:55:13.207 INFO:teuthology.orchestra.run.smithi007.stdout:Install 12 Packages 2024-09-18T07:55:13.207 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-09-18T07:55:13.208 INFO:teuthology.orchestra.run.smithi007.stdout:Total download size: 19 M 2024-09-18T07:55:13.208 INFO:teuthology.orchestra.run.smithi007.stdout:Installed size: 159 M 2024-09-18T07:55:13.208 INFO:teuthology.orchestra.run.smithi007.stdout:Downloading Packages: 2024-09-18T07:55:13.990 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 10/12 2024-09-18T07:55:14.131 INFO:teuthology.orchestra.run.smithi007.stdout:(1/12): ceph-grafana-dashboards-19.3.0-4983.ge7 228 kB/s | 34 kB 00:00 2024-09-18T07:55:14.156 INFO:teuthology.orchestra.run.smithi007.stdout:(2/12): ceph-prometheus-alerts-19.3.0-4983.ge7f 94 kB/s | 16 kB 00:00 2024-09-18T07:55:14.373 INFO:teuthology.orchestra.run.smithi007.stdout:(3/12): python3-protobuf-3.14.0-13.el9.noarch.r 1.2 MB/s | 269 kB 00:00 2024-09-18T07:55:14.440 INFO:teuthology.orchestra.run.smithi007.stdout:(4/12): protobuf-3.14.0-13.el9.x86_64.rpm 3.3 MB/s | 1.0 MB 00:00 2024-09-18T07:55:14.657 INFO:teuthology.orchestra.run.smithi007.stdout:(5/12): ceph-mgr-dashboard-19.3.0-4983.ge7fb7b5 21 MB/s | 14 MB 00:00 2024-09-18T07:55:14.691 INFO:teuthology.orchestra.run.smithi007.stdout:(6/12): abseil-cpp-20211102.0-3.el9.x86_64.rpm 2.1 MB/s | 548 kB 00:00 2024-09-18T07:55:14.725 INFO:teuthology.orchestra.run.smithi007.stdout:(7/12): protobuf-compiler-3.14.0-13.el9.x86_64. 2.4 MB/s | 863 kB 00:00 2024-09-18T07:55:14.750 INFO:teuthology.orchestra.run.smithi007.stdout:(8/12): grpc-data-1.46.7-10.el9.noarch.rpm 211 kB/s | 19 kB 00:00 2024-09-18T07:55:14.783 INFO:teuthology.orchestra.run.smithi007.stdout:(9/12): python3-repoze-lru-0.7-16.el9.noarch.rp 980 kB/s | 31 kB 00:00 2024-09-18T07:55:14.817 INFO:teuthology.orchestra.run.smithi007.stdout:(10/12): python3-grpcio-tools-1.46.7-10.el9.x86 1.6 MB/s | 144 kB 00:00 2024-09-18T07:55:14.851 INFO:teuthology.orchestra.run.smithi007.stdout:(11/12): python3-routes-2.5.1-5.el9.noarch.rpm 2.8 MB/s | 188 kB 00:00 2024-09-18T07:55:14.992 INFO:teuthology.orchestra.run.smithi007.stdout:(12/12): python3-grpcio-1.46.7-10.el9.x86_64.rp 6.9 MB/s | 2.0 MB 00:00 2024-09-18T07:55:14.997 INFO:teuthology.orchestra.run.smithi007.stdout:-------------------------------------------------------------------------------- 2024-09-18T07:55:14.997 INFO:teuthology.orchestra.run.smithi007.stdout:Total 11 MB/s | 19 MB 00:01 2024-09-18T07:55:15.049 INFO:teuthology.orchestra.run.smithi007.stdout:Running transaction check 2024-09-18T07:55:15.071 INFO:teuthology.orchestra.run.smithi007.stdout:Transaction check succeeded. 2024-09-18T07:55:15.072 INFO:teuthology.orchestra.run.smithi007.stdout:Running transaction test 2024-09-18T07:55:15.087 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 11/12 2024-09-18T07:55:15.116 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : ceph-mgr-diskprediction-local-2:19.3.0-4983.ge7fb7 12/12 2024-09-18T07:55:15.383 INFO:teuthology.orchestra.run.smithi007.stdout:Transaction test succeeded. 2024-09-18T07:55:15.383 INFO:teuthology.orchestra.run.smithi007.stdout:Running transaction 2024-09-18T07:55:15.999 INFO:teuthology.orchestra.run.smithi007.stdout: Preparing : 1/1 2024-09-18T07:55:16.085 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : protobuf-3.14.0-13.el9.x86_64 1/12 2024-09-18T07:55:16.129 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : protobuf-compiler-3.14.0-13.el9.x86_64 2/12 2024-09-18T07:55:16.176 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 3/12 2024-09-18T07:55:16.206 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 4/12 2024-09-18T07:55:16.310 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : grpc-data-1.46.7-10.el9.noarch 5/12 2024-09-18T07:55:16.447 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : abseil-cpp-20211102.0-3.el9.x86_64 6/12 2024-09-18T07:55:16.546 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : python3-grpcio-1.46.7-10.el9.x86_64 7/12 2024-09-18T07:55:16.595 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : python3-protobuf-3.14.0-13.el9.noarch 8/12 2024-09-18T07:55:16.631 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : python3-grpcio-tools-1.46.7-10.el9.x86_64 9/12 2024-09-18T07:55:16.663 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : ceph-prometheus-alerts-2:19.3.0-4983.ge7fb7b56.el9 10/12 2024-09-18T07:55:17.872 INFO:teuthology.orchestra.run.smithi050.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:19.3.0-4983.ge7fb7 12/12 2024-09-18T07:55:17.872 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : ceph-mgr-diskprediction-local-2:19.3.0-4983.ge7fb7 1/12 2024-09-18T07:55:17.872 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : libgfortran-11.5.0-2.el9.x86_64 2/12 2024-09-18T07:55:17.872 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : libquadmath-11.5.0-2.el9.x86_64 3/12 2024-09-18T07:55:17.872 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 4/12 2024-09-18T07:55:17.872 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 5/12 2024-09-18T07:55:17.872 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 6/12 2024-09-18T07:55:17.872 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : openblas-0.3.26-2.el9.x86_64 7/12 2024-09-18T07:55:17.873 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : openblas-openmp-0.3.26-2.el9.x86_64 8/12 2024-09-18T07:55:17.873 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : python3-devel-3.9.19-8.el9.x86_64 9/12 2024-09-18T07:55:17.873 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 10/12 2024-09-18T07:55:17.873 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 11/12 2024-09-18T07:55:18.331 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 12/12 2024-09-18T07:55:18.331 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-09-18T07:55:18.331 INFO:teuthology.orchestra.run.smithi050.stdout:Installed: 2024-09-18T07:55:18.331 INFO:teuthology.orchestra.run.smithi050.stdout: ceph-mgr-diskprediction-local-2:19.3.0-4983.ge7fb7b56.el9.noarch 2024-09-18T07:55:18.331 INFO:teuthology.orchestra.run.smithi050.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-09-18T07:55:18.331 INFO:teuthology.orchestra.run.smithi050.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-09-18T07:55:18.331 INFO:teuthology.orchestra.run.smithi050.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-09-18T07:55:18.332 INFO:teuthology.orchestra.run.smithi050.stdout: libgfortran-11.5.0-2.el9.x86_64 2024-09-18T07:55:18.332 INFO:teuthology.orchestra.run.smithi050.stdout: libquadmath-11.5.0-2.el9.x86_64 2024-09-18T07:55:18.332 INFO:teuthology.orchestra.run.smithi050.stdout: openblas-0.3.26-2.el9.x86_64 2024-09-18T07:55:18.332 INFO:teuthology.orchestra.run.smithi050.stdout: openblas-openmp-0.3.26-2.el9.x86_64 2024-09-18T07:55:18.332 INFO:teuthology.orchestra.run.smithi050.stdout: python3-devel-3.9.19-8.el9.x86_64 2024-09-18T07:55:18.332 INFO:teuthology.orchestra.run.smithi050.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-09-18T07:55:18.332 INFO:teuthology.orchestra.run.smithi050.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-09-18T07:55:18.332 INFO:teuthology.orchestra.run.smithi050.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-09-18T07:55:18.332 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-09-18T07:55:18.332 INFO:teuthology.orchestra.run.smithi050.stdout:Complete! 2024-09-18T07:55:18.722 DEBUG:teuthology.orchestra.run.smithi050:> sudo yum -y install ceph-mgr-rook 2024-09-18T07:55:19.326 INFO:teuthology.orchestra.run.smithi050.stdout:Last metadata expiration check: 0:01:29 ago on Wed 18 Sep 2024 07:53:50 AM UTC. 2024-09-18T07:55:19.465 INFO:teuthology.orchestra.run.smithi050.stdout:Dependencies resolved. 2024-09-18T07:55:19.466 INFO:teuthology.orchestra.run.smithi050.stdout:================================================================================ 2024-09-18T07:55:19.467 INFO:teuthology.orchestra.run.smithi050.stdout: Package Arch Version Repository Size 2024-09-18T07:55:19.467 INFO:teuthology.orchestra.run.smithi050.stdout:================================================================================ 2024-09-18T07:55:19.467 INFO:teuthology.orchestra.run.smithi050.stdout:Installing: 2024-09-18T07:55:19.467 INFO:teuthology.orchestra.run.smithi050.stdout: ceph-mgr-rook noarch 2:19.3.0-4983.ge7fb7b56.el9 ceph-noarch 49 k 2024-09-18T07:55:19.467 INFO:teuthology.orchestra.run.smithi050.stdout:Installing dependencies: 2024-09-18T07:55:19.467 INFO:teuthology.orchestra.run.smithi050.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-09-18T07:55:19.467 INFO:teuthology.orchestra.run.smithi050.stdout: python3-certifi noarch 2023.05.07-4.el9 epel 14 k 2024-09-18T07:55:19.467 INFO:teuthology.orchestra.run.smithi050.stdout: python3-google-auth noarch 1:2.34.0-1.el9 epel 220 k 2024-09-18T07:55:19.467 INFO:teuthology.orchestra.run.smithi050.stdout: python3-jsonpatch noarch 1.21-16.el9 appstream 26 k 2024-09-18T07:55:19.467 INFO:teuthology.orchestra.run.smithi050.stdout: python3-jsonpointer noarch 2.0-4.el9 appstream 19 k 2024-09-18T07:55:19.467 INFO:teuthology.orchestra.run.smithi050.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-09-18T07:55:19.468 INFO:teuthology.orchestra.run.smithi050.stdout: python3-oauthlib noarch 3.1.1-5.el9 appstream 222 k 2024-09-18T07:55:19.468 INFO:teuthology.orchestra.run.smithi050.stdout: python3-pyasn1 noarch 0.4.8-6.el9 appstream 159 k 2024-09-18T07:55:19.468 INFO:teuthology.orchestra.run.smithi050.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 appstream 279 k 2024-09-18T07:55:19.468 INFO:teuthology.orchestra.run.smithi050.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 appstream 54 k 2024-09-18T07:55:19.468 INFO:teuthology.orchestra.run.smithi050.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-09-18T07:55:19.468 INFO:teuthology.orchestra.run.smithi050.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-09-18T07:55:19.468 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-09-18T07:55:19.468 INFO:teuthology.orchestra.run.smithi050.stdout:Transaction Summary 2024-09-18T07:55:19.468 INFO:teuthology.orchestra.run.smithi050.stdout:================================================================================ 2024-09-18T07:55:19.469 INFO:teuthology.orchestra.run.smithi050.stdout:Install 13 Packages 2024-09-18T07:55:19.469 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-09-18T07:55:19.469 INFO:teuthology.orchestra.run.smithi050.stdout:Total download size: 2.2 M 2024-09-18T07:55:19.469 INFO:teuthology.orchestra.run.smithi050.stdout:Installed size: 26 M 2024-09-18T07:55:19.469 INFO:teuthology.orchestra.run.smithi050.stdout:Downloading Packages: 2024-09-18T07:55:19.708 INFO:teuthology.orchestra.run.smithi050.stdout:(1/13): python3-jsonpointer-2.0-4.el9.noarch.rp 238 kB/s | 19 kB 00:00 2024-09-18T07:55:19.725 INFO:teuthology.orchestra.run.smithi050.stdout:(2/13): python3-jsonpatch-1.21-16.el9.noarch.rp 263 kB/s | 26 kB 00:00 2024-09-18T07:55:19.792 INFO:teuthology.orchestra.run.smithi050.stdout:(3/13): ceph-mgr-rook-19.3.0-4983.ge7fb7b56.el9 299 kB/s | 49 kB 00:00 2024-09-18T07:55:19.818 INFO:teuthology.orchestra.run.smithi050.stdout:(4/13): python3-oauthlib-3.1.1-5.el9.noarch.rpm 2.0 MB/s | 222 kB 00:00 2024-09-18T07:55:19.843 INFO:teuthology.orchestra.run.smithi050.stdout:(5/13): python3-pyasn1-0.4.8-6.el9.noarch.rpm 1.3 MB/s | 159 kB 00:00 2024-09-18T07:55:19.868 INFO:teuthology.orchestra.run.smithi050.stdout:(6/13): python3-requests-oauthlib-1.3.0-12.el9. 1.0 MB/s | 54 kB 00:00 2024-09-18T07:55:19.919 INFO:teuthology.orchestra.run.smithi050.stdout:(7/13): python3-cachetools-4.2.4-1.el9.noarch.r 427 kB/s | 32 kB 00:00 2024-09-18T07:55:19.944 INFO:teuthology.orchestra.run.smithi050.stdout:(8/13): python3-pyasn1-modules-0.4.8-6.el9.noar 1.8 MB/s | 279 kB 00:00 2024-09-18T07:55:19.969 INFO:teuthology.orchestra.run.smithi050.stdout:(9/13): python3-certifi-2023.05.07-4.el9.noarch 140 kB/s | 14 kB 00:00 2024-09-18T07:55:20.011 INFO:teuthology.orchestra.run.smithi050.stdout:(10/13): python3-rsa-4.9-2.el9.noarch.rpm 1.7 MB/s | 59 kB 00:00 2024-09-18T07:55:20.036 INFO:teuthology.orchestra.run.smithi050.stdout:(11/13): python3-google-auth-2.34.0-1.el9.noarc 1.9 MB/s | 220 kB 00:00 2024-09-18T07:55:20.062 INFO:teuthology.orchestra.run.smithi050.stdout:(12/13): python3-websocket-client-1.2.3-2.el9.n 1.8 MB/s | 90 kB 00:00 2024-09-18T07:55:20.097 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : ceph-grafana-dashboards-2:19.3.0-4983.ge7fb7b56.el 11/12 2024-09-18T07:55:20.159 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : ceph-mgr-dashboard-2:19.3.0-4983.ge7fb7b56.el9.noa 12/12 2024-09-18T07:55:20.187 INFO:teuthology.orchestra.run.smithi050.stdout:(13/13): python3-kubernetes-26.1.0-2.el9.noarch 4.2 MB/s | 1.0 MB 00:00 2024-09-18T07:55:20.190 INFO:teuthology.orchestra.run.smithi050.stdout:-------------------------------------------------------------------------------- 2024-09-18T07:55:20.190 INFO:teuthology.orchestra.run.smithi050.stdout:Total 3.1 MB/s | 2.2 MB 00:00 2024-09-18T07:55:20.256 INFO:teuthology.orchestra.run.smithi050.stdout:Running transaction check 2024-09-18T07:55:20.272 INFO:teuthology.orchestra.run.smithi050.stdout:Transaction check succeeded. 2024-09-18T07:55:20.272 INFO:teuthology.orchestra.run.smithi050.stdout:Running transaction test 2024-09-18T07:55:20.370 INFO:teuthology.orchestra.run.smithi050.stdout:Transaction test succeeded. 2024-09-18T07:55:20.370 INFO:teuthology.orchestra.run.smithi050.stdout:Running transaction 2024-09-18T07:55:20.924 INFO:teuthology.orchestra.run.smithi050.stdout: Preparing : 1/1 2024-09-18T07:55:21.117 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 1/13 2024-09-18T07:55:21.181 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 2/13 2024-09-18T07:55:21.245 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : python3-rsa-4.9-2.el9.noarch 3/13 2024-09-18T07:55:21.281 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 4/13 2024-09-18T07:55:21.357 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : python3-certifi-2023.05.07-4.el9.noarch 5/13 2024-09-18T07:55:21.463 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 6/13 2024-09-18T07:55:21.581 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : python3-google-auth-1:2.34.0-1.el9.noarch 7/13 2024-09-18T07:55:21.642 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 8/13 2024-09-18T07:55:22.274 INFO:teuthology.orchestra.run.smithi007.stdout: Running scriptlet: ceph-mgr-dashboard-2:19.3.0-4983.ge7fb7b56.el9.noa 12/12 2024-09-18T07:55:22.274 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : ceph-grafana-dashboards-2:19.3.0-4983.ge7fb7b56.el 1/12 2024-09-18T07:55:22.274 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : ceph-mgr-dashboard-2:19.3.0-4983.ge7fb7b56.el9.noa 2/12 2024-09-18T07:55:22.274 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : ceph-prometheus-alerts-2:19.3.0-4983.ge7fb7b56.el9 3/12 2024-09-18T07:55:22.274 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : protobuf-3.14.0-13.el9.x86_64 4/12 2024-09-18T07:55:22.274 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : python3-protobuf-3.14.0-13.el9.noarch 5/12 2024-09-18T07:55:22.274 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : protobuf-compiler-3.14.0-13.el9.x86_64 6/12 2024-09-18T07:55:22.274 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : abseil-cpp-20211102.0-3.el9.x86_64 7/12 2024-09-18T07:55:22.275 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : grpc-data-1.46.7-10.el9.noarch 8/12 2024-09-18T07:55:22.275 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : python3-grpcio-1.46.7-10.el9.x86_64 9/12 2024-09-18T07:55:22.275 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : python3-grpcio-tools-1.46.7-10.el9.x86_64 10/12 2024-09-18T07:55:22.275 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 11/12 2024-09-18T07:55:22.411 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 9/13 2024-09-18T07:55:22.455 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 10/13 2024-09-18T07:55:22.489 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 11/13 2024-09-18T07:55:22.538 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 12/13 2024-09-18T07:55:22.566 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : ceph-mgr-rook-2:19.3.0-4983.ge7fb7b56.el9.noarch 13/13 2024-09-18T07:55:22.615 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 12/12 2024-09-18T07:55:22.616 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-09-18T07:55:22.616 INFO:teuthology.orchestra.run.smithi007.stdout:Installed: 2024-09-18T07:55:22.616 INFO:teuthology.orchestra.run.smithi007.stdout: abseil-cpp-20211102.0-3.el9.x86_64 2024-09-18T07:55:22.616 INFO:teuthology.orchestra.run.smithi007.stdout: ceph-grafana-dashboards-2:19.3.0-4983.ge7fb7b56.el9.noarch 2024-09-18T07:55:22.616 INFO:teuthology.orchestra.run.smithi007.stdout: ceph-mgr-dashboard-2:19.3.0-4983.ge7fb7b56.el9.noarch 2024-09-18T07:55:22.616 INFO:teuthology.orchestra.run.smithi007.stdout: ceph-prometheus-alerts-2:19.3.0-4983.ge7fb7b56.el9.noarch 2024-09-18T07:55:22.616 INFO:teuthology.orchestra.run.smithi007.stdout: grpc-data-1.46.7-10.el9.noarch 2024-09-18T07:55:22.616 INFO:teuthology.orchestra.run.smithi007.stdout: protobuf-3.14.0-13.el9.x86_64 2024-09-18T07:55:22.616 INFO:teuthology.orchestra.run.smithi007.stdout: protobuf-compiler-3.14.0-13.el9.x86_64 2024-09-18T07:55:22.616 INFO:teuthology.orchestra.run.smithi007.stdout: python3-grpcio-1.46.7-10.el9.x86_64 2024-09-18T07:55:22.616 INFO:teuthology.orchestra.run.smithi007.stdout: python3-grpcio-tools-1.46.7-10.el9.x86_64 2024-09-18T07:55:22.616 INFO:teuthology.orchestra.run.smithi007.stdout: python3-protobuf-3.14.0-13.el9.noarch 2024-09-18T07:55:22.617 INFO:teuthology.orchestra.run.smithi007.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-09-18T07:55:22.617 INFO:teuthology.orchestra.run.smithi007.stdout: python3-routes-2.5.1-5.el9.noarch 2024-09-18T07:55:22.617 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-09-18T07:55:22.617 INFO:teuthology.orchestra.run.smithi007.stdout:Complete! 2024-09-18T07:55:22.872 DEBUG:teuthology.orchestra.run.smithi007:> sudo yum -y install ceph-mgr-diskprediction-local 2024-09-18T07:55:23.305 INFO:teuthology.orchestra.run.smithi050.stdout: Running scriptlet: ceph-mgr-rook-2:19.3.0-4983.ge7fb7b56.el9.noarch 13/13 2024-09-18T07:55:23.305 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : ceph-mgr-rook-2:19.3.0-4983.ge7fb7b56.el9.noarch 1/13 2024-09-18T07:55:23.305 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 2/13 2024-09-18T07:55:23.305 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 3/13 2024-09-18T07:55:23.305 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 4/13 2024-09-18T07:55:23.306 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 5/13 2024-09-18T07:55:23.306 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 6/13 2024-09-18T07:55:23.306 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 7/13 2024-09-18T07:55:23.306 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 8/13 2024-09-18T07:55:23.306 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : python3-certifi-2023.05.07-4.el9.noarch 9/13 2024-09-18T07:55:23.306 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : python3-google-auth-1:2.34.0-1.el9.noarch 10/13 2024-09-18T07:55:23.306 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 11/13 2024-09-18T07:55:23.306 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 12/13 2024-09-18T07:55:23.479 INFO:teuthology.orchestra.run.smithi007.stdout:Last metadata expiration check: 0:01:11 ago on Wed 18 Sep 2024 07:54:12 AM UTC. 2024-09-18T07:55:23.619 INFO:teuthology.orchestra.run.smithi007.stdout:Dependencies resolved. 2024-09-18T07:55:23.620 INFO:teuthology.orchestra.run.smithi007.stdout:======================================================================================= 2024-09-18T07:55:23.620 INFO:teuthology.orchestra.run.smithi007.stdout: Package Arch Version Repository Size 2024-09-18T07:55:23.621 INFO:teuthology.orchestra.run.smithi007.stdout:======================================================================================= 2024-09-18T07:55:23.621 INFO:teuthology.orchestra.run.smithi007.stdout:Installing: 2024-09-18T07:55:23.621 INFO:teuthology.orchestra.run.smithi007.stdout: ceph-mgr-diskprediction-local noarch 2:19.3.0-4983.ge7fb7b56.el9 ceph-noarch 7.4 M 2024-09-18T07:55:23.621 INFO:teuthology.orchestra.run.smithi007.stdout:Installing dependencies: 2024-09-18T07:55:23.621 INFO:teuthology.orchestra.run.smithi007.stdout: flexiblas x86_64 3.0.4-8.el9 appstream 33 k 2024-09-18T07:55:23.621 INFO:teuthology.orchestra.run.smithi007.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 appstream 3.0 M 2024-09-18T07:55:23.621 INFO:teuthology.orchestra.run.smithi007.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 appstream 18 k 2024-09-18T07:55:23.621 INFO:teuthology.orchestra.run.smithi007.stdout: libgfortran x86_64 11.5.0-2.el9 baseos 801 k 2024-09-18T07:55:23.621 INFO:teuthology.orchestra.run.smithi007.stdout: libquadmath x86_64 11.5.0-2.el9 baseos 191 k 2024-09-18T07:55:23.621 INFO:teuthology.orchestra.run.smithi007.stdout: openblas x86_64 0.3.26-2.el9 appstream 38 k 2024-09-18T07:55:23.621 INFO:teuthology.orchestra.run.smithi007.stdout: openblas-openmp x86_64 0.3.26-2.el9 appstream 4.9 M 2024-09-18T07:55:23.621 INFO:teuthology.orchestra.run.smithi007.stdout: python3-devel x86_64 3.9.19-8.el9 appstream 245 k 2024-09-18T07:55:23.622 INFO:teuthology.orchestra.run.smithi007.stdout: python3-numpy x86_64 1:1.20.1-5.el9 appstream 5.1 M 2024-09-18T07:55:23.622 INFO:teuthology.orchestra.run.smithi007.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 appstream 250 k 2024-09-18T07:55:23.622 INFO:teuthology.orchestra.run.smithi007.stdout: python3-scipy x86_64 1.6.2-8.el9 appstream 16 M 2024-09-18T07:55:23.622 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-09-18T07:55:23.622 INFO:teuthology.orchestra.run.smithi007.stdout:Transaction Summary 2024-09-18T07:55:23.622 INFO:teuthology.orchestra.run.smithi007.stdout:======================================================================================= 2024-09-18T07:55:23.622 INFO:teuthology.orchestra.run.smithi007.stdout:Install 12 Packages 2024-09-18T07:55:23.622 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-09-18T07:55:23.623 INFO:teuthology.orchestra.run.smithi007.stdout:Total download size: 38 M 2024-09-18T07:55:23.623 INFO:teuthology.orchestra.run.smithi007.stdout:Installed size: 212 M 2024-09-18T07:55:23.623 INFO:teuthology.orchestra.run.smithi007.stdout:Downloading Packages: 2024-09-18T07:55:23.637 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 13/13 2024-09-18T07:55:23.637 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-09-18T07:55:23.638 INFO:teuthology.orchestra.run.smithi050.stdout:Installed: 2024-09-18T07:55:23.638 INFO:teuthology.orchestra.run.smithi050.stdout: ceph-mgr-rook-2:19.3.0-4983.ge7fb7b56.el9.noarch 2024-09-18T07:55:23.638 INFO:teuthology.orchestra.run.smithi050.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-09-18T07:55:23.638 INFO:teuthology.orchestra.run.smithi050.stdout: python3-certifi-2023.05.07-4.el9.noarch 2024-09-18T07:55:23.638 INFO:teuthology.orchestra.run.smithi050.stdout: python3-google-auth-1:2.34.0-1.el9.noarch 2024-09-18T07:55:23.638 INFO:teuthology.orchestra.run.smithi050.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-09-18T07:55:23.638 INFO:teuthology.orchestra.run.smithi050.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-09-18T07:55:23.638 INFO:teuthology.orchestra.run.smithi050.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-09-18T07:55:23.638 INFO:teuthology.orchestra.run.smithi050.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-09-18T07:55:23.638 INFO:teuthology.orchestra.run.smithi050.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-09-18T07:55:23.638 INFO:teuthology.orchestra.run.smithi050.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-09-18T07:55:23.638 INFO:teuthology.orchestra.run.smithi050.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-09-18T07:55:23.639 INFO:teuthology.orchestra.run.smithi050.stdout: python3-rsa-4.9-2.el9.noarch 2024-09-18T07:55:23.639 INFO:teuthology.orchestra.run.smithi050.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-09-18T07:55:23.639 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-09-18T07:55:23.639 INFO:teuthology.orchestra.run.smithi050.stdout:Complete! 2024-09-18T07:55:23.869 DEBUG:teuthology.orchestra.run.smithi050:> sudo yum -y install ceph-mgr-cephadm 2024-09-18T07:55:23.992 INFO:teuthology.orchestra.run.smithi007.stdout:(1/12): libquadmath-11.5.0-2.el9.x86_64.rpm 976 kB/s | 191 kB 00:00 2024-09-18T07:55:24.101 INFO:teuthology.orchestra.run.smithi007.stdout:(2/12): libgfortran-11.5.0-2.el9.x86_64.rpm 2.6 MB/s | 801 kB 00:00 2024-09-18T07:55:24.126 INFO:teuthology.orchestra.run.smithi007.stdout:(3/12): flexiblas-3.0.4-8.el9.x86_64.rpm 245 kB/s | 33 kB 00:00 2024-09-18T07:55:24.210 INFO:teuthology.orchestra.run.smithi007.stdout:(4/12): flexiblas-openblas-openmp-3.0.4-8.el9.x 218 kB/s | 18 kB 00:00 2024-09-18T07:55:24.302 INFO:teuthology.orchestra.run.smithi007.stdout:(5/12): ceph-mgr-diskprediction-local-19.3.0-49 15 MB/s | 7.4 MB 00:00 2024-09-18T07:55:24.327 INFO:teuthology.orchestra.run.smithi007.stdout:(6/12): openblas-0.3.26-2.el9.x86_64.rpm 329 kB/s | 38 kB 00:00 2024-09-18T07:55:24.436 INFO:teuthology.orchestra.run.smithi007.stdout:(7/12): python3-devel-3.9.19-8.el9.x86_64.rpm 2.2 MB/s | 245 kB 00:00 2024-09-18T07:55:24.479 INFO:teuthology.orchestra.run.smithi050.stdout:Last metadata expiration check: 0:01:34 ago on Wed 18 Sep 2024 07:53:50 AM UTC. 2024-09-18T07:55:24.612 INFO:teuthology.orchestra.run.smithi007.stdout:(8/12): flexiblas-netlib-3.0.4-8.el9.x86_64.rpm 5.9 MB/s | 3.0 MB 00:00 2024-09-18T07:55:24.619 INFO:teuthology.orchestra.run.smithi050.stdout:Dependencies resolved. 2024-09-18T07:55:24.620 INFO:teuthology.orchestra.run.smithi050.stdout:================================================================================ 2024-09-18T07:55:24.620 INFO:teuthology.orchestra.run.smithi050.stdout: Package Arch Version Repository Size 2024-09-18T07:55:24.620 INFO:teuthology.orchestra.run.smithi050.stdout:================================================================================ 2024-09-18T07:55:24.621 INFO:teuthology.orchestra.run.smithi050.stdout:Installing: 2024-09-18T07:55:24.621 INFO:teuthology.orchestra.run.smithi050.stdout: ceph-mgr-cephadm noarch 2:19.3.0-4983.ge7fb7b56.el9 ceph-noarch 154 k 2024-09-18T07:55:24.621 INFO:teuthology.orchestra.run.smithi050.stdout:Installing dependencies: 2024-09-18T07:55:24.621 INFO:teuthology.orchestra.run.smithi050.stdout: python3-asyncssh noarch 2.13.2-5.el9 epel 548 k 2024-09-18T07:55:24.621 INFO:teuthology.orchestra.run.smithi050.stdout: python3-babel noarch 2.9.1-2.el9 appstream 6.0 M 2024-09-18T07:55:24.621 INFO:teuthology.orchestra.run.smithi050.stdout: python3-jinja2 noarch 2.11.3-6.el9 appstream 248 k 2024-09-18T07:55:24.621 INFO:teuthology.orchestra.run.smithi050.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-09-18T07:55:24.621 INFO:teuthology.orchestra.run.smithi050.stdout: python3-typing-extensions noarch 4.12.2-2.el9 epel 74 k 2024-09-18T07:55:24.621 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-09-18T07:55:24.621 INFO:teuthology.orchestra.run.smithi050.stdout:Transaction Summary 2024-09-18T07:55:24.621 INFO:teuthology.orchestra.run.smithi050.stdout:================================================================================ 2024-09-18T07:55:24.621 INFO:teuthology.orchestra.run.smithi050.stdout:Install 6 Packages 2024-09-18T07:55:24.622 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-09-18T07:55:24.622 INFO:teuthology.orchestra.run.smithi050.stdout:Total download size: 7.0 M 2024-09-18T07:55:24.622 INFO:teuthology.orchestra.run.smithi050.stdout:Installed size: 33 M 2024-09-18T07:55:24.622 INFO:teuthology.orchestra.run.smithi050.stdout:Downloading Packages: 2024-09-18T07:55:24.670 INFO:teuthology.orchestra.run.smithi007.stdout:(9/12): python3-numpy-f2py-1.20.1-5.el9.x86_64. 4.2 MB/s | 250 kB 00:00 2024-09-18T07:55:25.215 INFO:teuthology.orchestra.run.smithi050.stdout:(1/6): python3-jinja2-2.11.3-6.el9.noarch.rpm 1.5 MB/s | 248 kB 00:00 2024-09-18T07:55:25.287 INFO:teuthology.orchestra.run.smithi007.stdout:(10/12): openblas-openmp-0.3.26-2.el9.x86_64.rp 5.0 MB/s | 4.9 MB 00:00 2024-09-18T07:55:25.289 INFO:teuthology.orchestra.run.smithi050.stdout:(2/6): ceph-mgr-cephadm-19.3.0-4983.ge7fb7b56.e 664 kB/s | 154 kB 00:00 2024-09-18T07:55:25.340 INFO:teuthology.orchestra.run.smithi050.stdout:(3/6): python3-natsort-7.1.1-5.el9.noarch.rpm 1.1 MB/s | 58 kB 00:00 2024-09-18T07:55:25.374 INFO:teuthology.orchestra.run.smithi050.stdout:(4/6): python3-asyncssh-2.13.2-5.el9.noarch.rpm 3.4 MB/s | 548 kB 00:00 2024-09-18T07:55:25.399 INFO:teuthology.orchestra.run.smithi050.stdout:(5/6): python3-typing-extensions-4.12.2-2.el9.n 1.2 MB/s | 74 kB 00:00 2024-09-18T07:55:25.554 INFO:teuthology.orchestra.run.smithi007.stdout:(11/12): python3-numpy-1.20.1-5.el9.x86_64.rpm 4.6 MB/s | 5.1 MB 00:01 2024-09-18T07:55:26.387 INFO:teuthology.orchestra.run.smithi007.stdout:(12/12): python3-scipy-1.6.2-8.el9.x86_64.rpm 9.5 MB/s | 16 MB 00:01 2024-09-18T07:55:26.388 INFO:teuthology.orchestra.run.smithi007.stdout:-------------------------------------------------------------------------------- 2024-09-18T07:55:26.389 INFO:teuthology.orchestra.run.smithi007.stdout:Total 14 MB/s | 38 MB 00:02 2024-09-18T07:55:26.532 INFO:teuthology.orchestra.run.smithi050.stdout:(6/6): python3-babel-2.9.1-2.el9.noarch.rpm 4.0 MB/s | 6.0 MB 00:01 2024-09-18T07:55:26.536 INFO:teuthology.orchestra.run.smithi050.stdout:-------------------------------------------------------------------------------- 2024-09-18T07:55:26.536 INFO:teuthology.orchestra.run.smithi050.stdout:Total 3.7 MB/s | 7.0 MB 00:01 2024-09-18T07:55:26.611 INFO:teuthology.orchestra.run.smithi050.stdout:Running transaction check 2024-09-18T07:55:26.624 INFO:teuthology.orchestra.run.smithi050.stdout:Transaction check succeeded. 2024-09-18T07:55:26.625 INFO:teuthology.orchestra.run.smithi050.stdout:Running transaction test 2024-09-18T07:55:26.734 INFO:teuthology.orchestra.run.smithi050.stdout:Transaction test succeeded. 2024-09-18T07:55:26.735 INFO:teuthology.orchestra.run.smithi050.stdout:Running transaction 2024-09-18T07:55:26.755 INFO:teuthology.orchestra.run.smithi007.stdout:Running transaction check 2024-09-18T07:55:26.782 INFO:teuthology.orchestra.run.smithi007.stdout:Transaction check succeeded. 2024-09-18T07:55:26.782 INFO:teuthology.orchestra.run.smithi007.stdout:Running transaction test 2024-09-18T07:55:27.049 INFO:teuthology.orchestra.run.smithi050.stdout: Preparing : 1/1 2024-09-18T07:55:27.230 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : python3-typing-extensions-4.12.2-2.el9.noarch 1/6 2024-09-18T07:55:27.242 INFO:teuthology.orchestra.run.smithi007.stdout:Transaction test succeeded. 2024-09-18T07:55:27.243 INFO:teuthology.orchestra.run.smithi007.stdout:Running transaction 2024-09-18T07:55:27.285 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : python3-asyncssh-2.13.2-5.el9.noarch 2/6 2024-09-18T07:55:27.837 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 3/6 2024-09-18T07:55:27.923 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 4/6 2024-09-18T07:55:28.000 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : python3-jinja2-2.11.3-6.el9.noarch 5/6 2024-09-18T07:55:28.031 INFO:teuthology.orchestra.run.smithi007.stdout: Preparing : 1/1 2024-09-18T07:55:28.038 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : ceph-mgr-cephadm-2:19.3.0-4983.ge7fb7b56.el9.noarch 6/6 2024-09-18T07:55:28.111 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 1/12 2024-09-18T07:55:28.153 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : libquadmath-11.5.0-2.el9.x86_64 2/12 2024-09-18T07:55:28.257 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : libgfortran-11.5.0-2.el9.x86_64 3/12 2024-09-18T07:55:28.295 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : python3-devel-3.9.19-8.el9.x86_64 4/12 2024-09-18T07:55:28.488 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : openblas-0.3.26-2.el9.x86_64 5/12 2024-09-18T07:55:28.520 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : openblas-openmp-0.3.26-2.el9.x86_64 6/12 2024-09-18T07:55:28.598 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 7/12 2024-09-18T07:55:28.830 INFO:teuthology.orchestra.run.smithi050.stdout: Running scriptlet: ceph-mgr-cephadm-2:19.3.0-4983.ge7fb7b56.el9.noarch 6/6 2024-09-18T07:55:28.830 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : ceph-mgr-cephadm-2:19.3.0-4983.ge7fb7b56.el9.noarch 1/6 2024-09-18T07:55:28.831 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 2/6 2024-09-18T07:55:28.831 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : python3-jinja2-2.11.3-6.el9.noarch 3/6 2024-09-18T07:55:28.831 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : python3-asyncssh-2.13.2-5.el9.noarch 4/6 2024-09-18T07:55:28.831 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 5/6 2024-09-18T07:55:29.119 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : python3-typing-extensions-4.12.2-2.el9.noarch 6/6 2024-09-18T07:55:29.119 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-09-18T07:55:29.119 INFO:teuthology.orchestra.run.smithi050.stdout:Installed: 2024-09-18T07:55:29.119 INFO:teuthology.orchestra.run.smithi050.stdout: ceph-mgr-cephadm-2:19.3.0-4983.ge7fb7b56.el9.noarch 2024-09-18T07:55:29.119 INFO:teuthology.orchestra.run.smithi050.stdout: python3-asyncssh-2.13.2-5.el9.noarch 2024-09-18T07:55:29.119 INFO:teuthology.orchestra.run.smithi050.stdout: python3-babel-2.9.1-2.el9.noarch 2024-09-18T07:55:29.119 INFO:teuthology.orchestra.run.smithi050.stdout: python3-jinja2-2.11.3-6.el9.noarch 2024-09-18T07:55:29.119 INFO:teuthology.orchestra.run.smithi050.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-09-18T07:55:29.119 INFO:teuthology.orchestra.run.smithi050.stdout: python3-typing-extensions-4.12.2-2.el9.noarch 2024-09-18T07:55:29.120 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-09-18T07:55:29.120 INFO:teuthology.orchestra.run.smithi050.stdout:Complete! 2024-09-18T07:55:29.277 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 8/12 2024-09-18T07:55:29.391 DEBUG:teuthology.orchestra.run.smithi050:> sudo yum -y install ceph-fuse 2024-09-18T07:55:29.412 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 9/12 2024-09-18T07:55:29.990 INFO:teuthology.orchestra.run.smithi050.stdout:Last metadata expiration check: 0:01:39 ago on Wed 18 Sep 2024 07:53:50 AM UTC. 2024-09-18T07:55:30.128 INFO:teuthology.orchestra.run.smithi050.stdout:Dependencies resolved. 2024-09-18T07:55:30.129 INFO:teuthology.orchestra.run.smithi050.stdout:================================================================================ 2024-09-18T07:55:30.129 INFO:teuthology.orchestra.run.smithi050.stdout: Package Arch Version Repository Size 2024-09-18T07:55:30.129 INFO:teuthology.orchestra.run.smithi050.stdout:================================================================================ 2024-09-18T07:55:30.129 INFO:teuthology.orchestra.run.smithi050.stdout:Installing: 2024-09-18T07:55:30.129 INFO:teuthology.orchestra.run.smithi050.stdout: ceph-fuse x86_64 2:19.3.0-4983.ge7fb7b56.el9 ceph 901 k 2024-09-18T07:55:30.129 INFO:teuthology.orchestra.run.smithi050.stdout:Installing dependencies: 2024-09-18T07:55:30.129 INFO:teuthology.orchestra.run.smithi050.stdout: fuse x86_64 2.9.9-16.el9 baseos 80 k 2024-09-18T07:55:30.130 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-09-18T07:55:30.130 INFO:teuthology.orchestra.run.smithi050.stdout:Transaction Summary 2024-09-18T07:55:30.130 INFO:teuthology.orchestra.run.smithi050.stdout:================================================================================ 2024-09-18T07:55:30.130 INFO:teuthology.orchestra.run.smithi050.stdout:Install 2 Packages 2024-09-18T07:55:30.130 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-09-18T07:55:30.130 INFO:teuthology.orchestra.run.smithi050.stdout:Total download size: 980 k 2024-09-18T07:55:30.130 INFO:teuthology.orchestra.run.smithi050.stdout:Installed size: 2.9 M 2024-09-18T07:55:30.131 INFO:teuthology.orchestra.run.smithi050.stdout:Downloading Packages: 2024-09-18T07:55:30.526 INFO:teuthology.orchestra.run.smithi050.stdout:(1/2): fuse-2.9.9-16.el9.x86_64.rpm 708 kB/s | 80 kB 00:00 2024-09-18T07:55:30.668 INFO:teuthology.orchestra.run.smithi050.stdout:(2/2): ceph-fuse-19.3.0-4983.ge7fb7b56.el9.x86_ 3.5 MB/s | 901 kB 00:00 2024-09-18T07:55:30.669 INFO:teuthology.orchestra.run.smithi050.stdout:-------------------------------------------------------------------------------- 2024-09-18T07:55:30.669 INFO:teuthology.orchestra.run.smithi050.stdout:Total 1.8 MB/s | 980 kB 00:00 2024-09-18T07:55:30.682 INFO:teuthology.orchestra.run.smithi050.stdout:Running transaction check 2024-09-18T07:55:30.707 INFO:teuthology.orchestra.run.smithi050.stdout:Transaction check succeeded. 2024-09-18T07:55:30.707 INFO:teuthology.orchestra.run.smithi050.stdout:Running transaction test 2024-09-18T07:55:30.737 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 10/12 2024-09-18T07:55:30.797 INFO:teuthology.orchestra.run.smithi050.stdout:Transaction test succeeded. 2024-09-18T07:55:30.797 INFO:teuthology.orchestra.run.smithi050.stdout:Running transaction 2024-09-18T07:55:30.998 INFO:teuthology.orchestra.run.smithi050.stdout: Preparing : 1/1 2024-09-18T07:55:31.144 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : fuse-2.9.9-16.el9.x86_64 1/2 2024-09-18T07:55:31.213 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : ceph-fuse-2:19.3.0-4983.ge7fb7b56.el9.x86_64 2/2 2024-09-18T07:55:31.815 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 11/12 2024-09-18T07:55:31.853 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : ceph-mgr-diskprediction-local-2:19.3.0-4983.ge7fb7 12/12 2024-09-18T07:55:31.974 INFO:teuthology.orchestra.run.smithi050.stdout: Running scriptlet: ceph-fuse-2:19.3.0-4983.ge7fb7b56.el9.x86_64 2/2 2024-09-18T07:55:31.974 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : ceph-fuse-2:19.3.0-4983.ge7fb7b56.el9.x86_64 1/2 2024-09-18T07:55:32.293 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : fuse-2.9.9-16.el9.x86_64 2/2 2024-09-18T07:55:32.294 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-09-18T07:55:32.294 INFO:teuthology.orchestra.run.smithi050.stdout:Installed: 2024-09-18T07:55:32.294 INFO:teuthology.orchestra.run.smithi050.stdout: ceph-fuse-2:19.3.0-4983.ge7fb7b56.el9.x86_64 fuse-2.9.9-16.el9.x86_64 2024-09-18T07:55:32.294 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-09-18T07:55:32.294 INFO:teuthology.orchestra.run.smithi050.stdout:Complete! 2024-09-18T07:55:32.551 DEBUG:teuthology.orchestra.run.smithi050:> sudo yum -y install ceph-volume 2024-09-18T07:55:33.161 INFO:teuthology.orchestra.run.smithi050.stdout:Last metadata expiration check: 0:01:43 ago on Wed 18 Sep 2024 07:53:50 AM UTC. 2024-09-18T07:55:33.300 INFO:teuthology.orchestra.run.smithi050.stdout:Dependencies resolved. 2024-09-18T07:55:33.301 INFO:teuthology.orchestra.run.smithi050.stdout:================================================================================ 2024-09-18T07:55:33.301 INFO:teuthology.orchestra.run.smithi050.stdout: Package Arch Version Repository Size 2024-09-18T07:55:33.301 INFO:teuthology.orchestra.run.smithi050.stdout:================================================================================ 2024-09-18T07:55:33.301 INFO:teuthology.orchestra.run.smithi050.stdout:Installing: 2024-09-18T07:55:33.301 INFO:teuthology.orchestra.run.smithi050.stdout: ceph-volume noarch 2:19.3.0-4983.ge7fb7b56.el9 ceph-noarch 285 k 2024-09-18T07:55:33.301 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-09-18T07:55:33.301 INFO:teuthology.orchestra.run.smithi050.stdout:Transaction Summary 2024-09-18T07:55:33.301 INFO:teuthology.orchestra.run.smithi050.stdout:================================================================================ 2024-09-18T07:55:33.301 INFO:teuthology.orchestra.run.smithi050.stdout:Install 1 Package 2024-09-18T07:55:33.301 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-09-18T07:55:33.301 INFO:teuthology.orchestra.run.smithi050.stdout:Total download size: 285 k 2024-09-18T07:55:33.301 INFO:teuthology.orchestra.run.smithi050.stdout:Installed size: 1.3 M 2024-09-18T07:55:33.301 INFO:teuthology.orchestra.run.smithi050.stdout:Downloading Packages: 2024-09-18T07:55:33.515 INFO:teuthology.orchestra.run.smithi050.stdout:ceph-volume-19.3.0-4983.ge7fb7b56.el9.noarch.rp 1.3 MB/s | 285 kB 00:00 2024-09-18T07:55:33.516 INFO:teuthology.orchestra.run.smithi050.stdout:-------------------------------------------------------------------------------- 2024-09-18T07:55:33.516 INFO:teuthology.orchestra.run.smithi050.stdout:Total 1.3 MB/s | 285 kB 00:00 2024-09-18T07:55:33.516 INFO:teuthology.orchestra.run.smithi050.stdout:Running transaction check 2024-09-18T07:55:33.525 INFO:teuthology.orchestra.run.smithi050.stdout:Transaction check succeeded. 2024-09-18T07:55:33.526 INFO:teuthology.orchestra.run.smithi050.stdout:Running transaction test 2024-09-18T07:55:33.554 INFO:teuthology.orchestra.run.smithi050.stdout:Transaction test succeeded. 2024-09-18T07:55:33.554 INFO:teuthology.orchestra.run.smithi050.stdout:Running transaction 2024-09-18T07:55:33.787 INFO:teuthology.orchestra.run.smithi050.stdout: Preparing : 1/1 2024-09-18T07:55:33.859 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : ceph-volume-2:19.3.0-4983.ge7fb7b56.el9.noarch 1/1 2024-09-18T07:55:33.877 INFO:teuthology.orchestra.run.smithi050.stdout: Running scriptlet: ceph-volume-2:19.3.0-4983.ge7fb7b56.el9.noarch 1/1 2024-09-18T07:55:33.877 INFO:teuthology.orchestra.run.smithi050.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-18T07:55:33.877 INFO:teuthology.orchestra.run.smithi050.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2024-09-18T07:55:33.877 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-09-18T07:55:34.524 INFO:teuthology.orchestra.run.smithi007.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:19.3.0-4983.ge7fb7 12/12 2024-09-18T07:55:34.525 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : ceph-mgr-diskprediction-local-2:19.3.0-4983.ge7fb7 1/12 2024-09-18T07:55:34.525 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : libgfortran-11.5.0-2.el9.x86_64 2/12 2024-09-18T07:55:34.525 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : libquadmath-11.5.0-2.el9.x86_64 3/12 2024-09-18T07:55:34.525 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 4/12 2024-09-18T07:55:34.525 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 5/12 2024-09-18T07:55:34.525 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 6/12 2024-09-18T07:55:34.525 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : openblas-0.3.26-2.el9.x86_64 7/12 2024-09-18T07:55:34.525 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : openblas-openmp-0.3.26-2.el9.x86_64 8/12 2024-09-18T07:55:34.525 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : python3-devel-3.9.19-8.el9.x86_64 9/12 2024-09-18T07:55:34.525 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 10/12 2024-09-18T07:55:34.525 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 11/12 2024-09-18T07:55:34.876 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : ceph-volume-2:19.3.0-4983.ge7fb7b56.el9.noarch 1/1 2024-09-18T07:55:34.876 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-09-18T07:55:34.876 INFO:teuthology.orchestra.run.smithi050.stdout:Installed: 2024-09-18T07:55:34.876 INFO:teuthology.orchestra.run.smithi050.stdout: ceph-volume-2:19.3.0-4983.ge7fb7b56.el9.noarch 2024-09-18T07:55:34.876 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-09-18T07:55:34.876 INFO:teuthology.orchestra.run.smithi050.stdout:Complete! 2024-09-18T07:55:35.058 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 12/12 2024-09-18T07:55:35.058 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-09-18T07:55:35.058 INFO:teuthology.orchestra.run.smithi007.stdout:Installed: 2024-09-18T07:55:35.058 INFO:teuthology.orchestra.run.smithi007.stdout: ceph-mgr-diskprediction-local-2:19.3.0-4983.ge7fb7b56.el9.noarch 2024-09-18T07:55:35.058 INFO:teuthology.orchestra.run.smithi007.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-09-18T07:55:35.058 INFO:teuthology.orchestra.run.smithi007.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-09-18T07:55:35.058 INFO:teuthology.orchestra.run.smithi007.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-09-18T07:55:35.059 INFO:teuthology.orchestra.run.smithi007.stdout: libgfortran-11.5.0-2.el9.x86_64 2024-09-18T07:55:35.059 INFO:teuthology.orchestra.run.smithi007.stdout: libquadmath-11.5.0-2.el9.x86_64 2024-09-18T07:55:35.059 INFO:teuthology.orchestra.run.smithi007.stdout: openblas-0.3.26-2.el9.x86_64 2024-09-18T07:55:35.059 INFO:teuthology.orchestra.run.smithi007.stdout: openblas-openmp-0.3.26-2.el9.x86_64 2024-09-18T07:55:35.059 INFO:teuthology.orchestra.run.smithi007.stdout: python3-devel-3.9.19-8.el9.x86_64 2024-09-18T07:55:35.059 INFO:teuthology.orchestra.run.smithi007.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-09-18T07:55:35.059 INFO:teuthology.orchestra.run.smithi007.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-09-18T07:55:35.059 INFO:teuthology.orchestra.run.smithi007.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-09-18T07:55:35.059 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-09-18T07:55:35.059 INFO:teuthology.orchestra.run.smithi007.stdout:Complete! 2024-09-18T07:55:35.195 DEBUG:teuthology.orchestra.run.smithi050:> sudo yum -y install librados-devel 2024-09-18T07:55:35.462 DEBUG:teuthology.orchestra.run.smithi007:> sudo yum -y install ceph-mgr-rook 2024-09-18T07:55:35.811 INFO:teuthology.orchestra.run.smithi050.stdout:Last metadata expiration check: 0:01:45 ago on Wed 18 Sep 2024 07:53:50 AM UTC. 2024-09-18T07:55:35.951 INFO:teuthology.orchestra.run.smithi050.stdout:Dependencies resolved. 2024-09-18T07:55:35.952 INFO:teuthology.orchestra.run.smithi050.stdout:================================================================================ 2024-09-18T07:55:35.952 INFO:teuthology.orchestra.run.smithi050.stdout: Package Arch Version Repo Size 2024-09-18T07:55:35.952 INFO:teuthology.orchestra.run.smithi050.stdout:================================================================================ 2024-09-18T07:55:35.952 INFO:teuthology.orchestra.run.smithi050.stdout:Installing: 2024-09-18T07:55:35.952 INFO:teuthology.orchestra.run.smithi050.stdout: librados-devel x86_64 2:19.3.0-4983.ge7fb7b56.el9 ceph 127 k 2024-09-18T07:55:35.952 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-09-18T07:55:35.952 INFO:teuthology.orchestra.run.smithi050.stdout:Transaction Summary 2024-09-18T07:55:35.952 INFO:teuthology.orchestra.run.smithi050.stdout:================================================================================ 2024-09-18T07:55:35.952 INFO:teuthology.orchestra.run.smithi050.stdout:Install 1 Package 2024-09-18T07:55:35.952 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-09-18T07:55:35.952 INFO:teuthology.orchestra.run.smithi050.stdout:Total download size: 127 k 2024-09-18T07:55:35.952 INFO:teuthology.orchestra.run.smithi050.stdout:Installed size: 456 k 2024-09-18T07:55:35.953 INFO:teuthology.orchestra.run.smithi050.stdout:Downloading Packages: 2024-09-18T07:55:36.074 INFO:teuthology.orchestra.run.smithi007.stdout:Last metadata expiration check: 0:01:24 ago on Wed 18 Sep 2024 07:54:12 AM UTC. 2024-09-18T07:55:36.179 INFO:teuthology.orchestra.run.smithi050.stdout:librados-devel-19.3.0-4983.ge7fb7b56.el9.x86_64 560 kB/s | 127 kB 00:00 2024-09-18T07:55:36.180 INFO:teuthology.orchestra.run.smithi050.stdout:-------------------------------------------------------------------------------- 2024-09-18T07:55:36.180 INFO:teuthology.orchestra.run.smithi050.stdout:Total 556 kB/s | 127 kB 00:00 2024-09-18T07:55:36.180 INFO:teuthology.orchestra.run.smithi050.stdout:Running transaction check 2024-09-18T07:55:36.186 INFO:teuthology.orchestra.run.smithi050.stdout:Transaction check succeeded. 2024-09-18T07:55:36.186 INFO:teuthology.orchestra.run.smithi050.stdout:Running transaction test 2024-09-18T07:55:36.212 INFO:teuthology.orchestra.run.smithi007.stdout:Dependencies resolved. 2024-09-18T07:55:36.213 INFO:teuthology.orchestra.run.smithi007.stdout:================================================================================ 2024-09-18T07:55:36.213 INFO:teuthology.orchestra.run.smithi007.stdout: Package Arch Version Repository Size 2024-09-18T07:55:36.214 INFO:teuthology.orchestra.run.smithi007.stdout:================================================================================ 2024-09-18T07:55:36.214 INFO:teuthology.orchestra.run.smithi007.stdout:Installing: 2024-09-18T07:55:36.214 INFO:teuthology.orchestra.run.smithi007.stdout: ceph-mgr-rook noarch 2:19.3.0-4983.ge7fb7b56.el9 ceph-noarch 49 k 2024-09-18T07:55:36.214 INFO:teuthology.orchestra.run.smithi007.stdout:Installing dependencies: 2024-09-18T07:55:36.214 INFO:teuthology.orchestra.run.smithi007.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-09-18T07:55:36.214 INFO:teuthology.orchestra.run.smithi007.stdout: python3-certifi noarch 2023.05.07-4.el9 epel 14 k 2024-09-18T07:55:36.214 INFO:teuthology.orchestra.run.smithi007.stdout: python3-google-auth noarch 1:2.34.0-1.el9 epel 220 k 2024-09-18T07:55:36.214 INFO:teuthology.orchestra.run.smithi007.stdout: python3-jsonpatch noarch 1.21-16.el9 appstream 26 k 2024-09-18T07:55:36.214 INFO:teuthology.orchestra.run.smithi007.stdout: python3-jsonpointer noarch 2.0-4.el9 appstream 19 k 2024-09-18T07:55:36.214 INFO:teuthology.orchestra.run.smithi007.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-09-18T07:55:36.214 INFO:teuthology.orchestra.run.smithi007.stdout: python3-oauthlib noarch 3.1.1-5.el9 appstream 222 k 2024-09-18T07:55:36.214 INFO:teuthology.orchestra.run.smithi007.stdout: python3-pyasn1 noarch 0.4.8-6.el9 appstream 159 k 2024-09-18T07:55:36.215 INFO:teuthology.orchestra.run.smithi007.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 appstream 279 k 2024-09-18T07:55:36.215 INFO:teuthology.orchestra.run.smithi007.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 appstream 54 k 2024-09-18T07:55:36.215 INFO:teuthology.orchestra.run.smithi007.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-09-18T07:55:36.215 INFO:teuthology.orchestra.run.smithi007.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-09-18T07:55:36.215 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-09-18T07:55:36.215 INFO:teuthology.orchestra.run.smithi007.stdout:Transaction Summary 2024-09-18T07:55:36.215 INFO:teuthology.orchestra.run.smithi007.stdout:================================================================================ 2024-09-18T07:55:36.215 INFO:teuthology.orchestra.run.smithi007.stdout:Install 13 Packages 2024-09-18T07:55:36.215 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-09-18T07:55:36.216 INFO:teuthology.orchestra.run.smithi007.stdout:Total download size: 2.2 M 2024-09-18T07:55:36.216 INFO:teuthology.orchestra.run.smithi007.stdout:Installed size: 26 M 2024-09-18T07:55:36.216 INFO:teuthology.orchestra.run.smithi007.stdout:Downloading Packages: 2024-09-18T07:55:36.257 INFO:teuthology.orchestra.run.smithi050.stdout:Transaction test succeeded. 2024-09-18T07:55:36.257 INFO:teuthology.orchestra.run.smithi050.stdout:Running transaction 2024-09-18T07:55:36.412 INFO:teuthology.orchestra.run.smithi050.stdout: Preparing : 1/1 2024-09-18T07:55:36.520 INFO:teuthology.orchestra.run.smithi007.stdout:(1/13): python3-jsonpointer-2.0-4.el9.noarch.rp 245 kB/s | 19 kB 00:00 2024-09-18T07:55:36.546 INFO:teuthology.orchestra.run.smithi007.stdout:(2/13): python3-jsonpatch-1.21-16.el9.noarch.rp 247 kB/s | 26 kB 00:00 2024-09-18T07:55:36.597 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : librados-devel-2:19.3.0-4983.ge7fb7b56.el9.x86_64 1/1 2024-09-18T07:55:36.604 INFO:teuthology.orchestra.run.smithi007.stdout:(3/13): ceph-mgr-rook-19.3.0-4983.ge7fb7b56.el9 303 kB/s | 49 kB 00:00 2024-09-18T07:55:36.638 INFO:teuthology.orchestra.run.smithi007.stdout:(4/13): python3-oauthlib-3.1.1-5.el9.noarch.rpm 1.8 MB/s | 222 kB 00:00 2024-09-18T07:55:36.672 INFO:teuthology.orchestra.run.smithi007.stdout:(5/13): python3-pyasn1-0.4.8-6.el9.noarch.rpm 1.2 MB/s | 159 kB 00:00 2024-09-18T07:55:36.697 INFO:teuthology.orchestra.run.smithi007.stdout:(6/13): python3-requests-oauthlib-1.3.0-12.el9. 917 kB/s | 54 kB 00:00 2024-09-18T07:55:36.739 INFO:teuthology.orchestra.run.smithi007.stdout:(7/13): python3-cachetools-4.2.4-1.el9.noarch.r 481 kB/s | 32 kB 00:00 2024-09-18T07:55:36.773 INFO:teuthology.orchestra.run.smithi007.stdout:(8/13): python3-certifi-2023.05.07-4.el9.noarch 188 kB/s | 14 kB 00:00 2024-09-18T07:55:36.823 INFO:teuthology.orchestra.run.smithi007.stdout:(9/13): python3-pyasn1-modules-0.4.8-6.el9.noar 1.2 MB/s | 279 kB 00:00 2024-09-18T07:55:36.868 INFO:teuthology.orchestra.run.smithi007.stdout:(10/13): python3-google-auth-2.34.0-1.el9.noarc 1.7 MB/s | 220 kB 00:00 2024-09-18T07:55:36.899 INFO:teuthology.orchestra.run.smithi007.stdout:(11/13): python3-rsa-4.9-2.el9.noarch.rpm 784 kB/s | 59 kB 00:00 2024-09-18T07:55:36.924 INFO:teuthology.orchestra.run.smithi007.stdout:(12/13): python3-websocket-client-1.2.3-2.el9.n 1.5 MB/s | 90 kB 00:00 2024-09-18T07:55:36.957 INFO:teuthology.orchestra.run.smithi007.stdout:(13/13): python3-kubernetes-26.1.0-2.el9.noarch 5.5 MB/s | 1.0 MB 00:00 2024-09-18T07:55:36.960 INFO:teuthology.orchestra.run.smithi007.stdout:-------------------------------------------------------------------------------- 2024-09-18T07:55:36.960 INFO:teuthology.orchestra.run.smithi007.stdout:Total 3.0 MB/s | 2.2 MB 00:00 2024-09-18T07:55:37.030 INFO:teuthology.orchestra.run.smithi007.stdout:Running transaction check 2024-09-18T07:55:37.034 INFO:teuthology.orchestra.run.smithi050.stdout: Running scriptlet: librados-devel-2:19.3.0-4983.ge7fb7b56.el9.x86_64 1/1 2024-09-18T07:55:37.046 INFO:teuthology.orchestra.run.smithi007.stdout:Transaction check succeeded. 2024-09-18T07:55:37.047 INFO:teuthology.orchestra.run.smithi007.stdout:Running transaction test 2024-09-18T07:55:37.145 INFO:teuthology.orchestra.run.smithi007.stdout:Transaction test succeeded. 2024-09-18T07:55:37.146 INFO:teuthology.orchestra.run.smithi007.stdout:Running transaction 2024-09-18T07:55:37.292 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : librados-devel-2:19.3.0-4983.ge7fb7b56.el9.x86_64 1/1 2024-09-18T07:55:37.292 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-09-18T07:55:37.293 INFO:teuthology.orchestra.run.smithi050.stdout:Installed: 2024-09-18T07:55:37.293 INFO:teuthology.orchestra.run.smithi050.stdout: librados-devel-2:19.3.0-4983.ge7fb7b56.el9.x86_64 2024-09-18T07:55:37.293 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-09-18T07:55:37.293 INFO:teuthology.orchestra.run.smithi050.stdout:Complete! 2024-09-18T07:55:37.591 DEBUG:teuthology.orchestra.run.smithi050:> sudo yum -y install libcephfs2 2024-09-18T07:55:37.781 INFO:teuthology.orchestra.run.smithi007.stdout: Preparing : 1/1 2024-09-18T07:55:37.945 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 1/13 2024-09-18T07:55:38.002 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 2/13 2024-09-18T07:55:38.058 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : python3-rsa-4.9-2.el9.noarch 3/13 2024-09-18T07:55:38.093 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 4/13 2024-09-18T07:55:38.135 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : python3-certifi-2023.05.07-4.el9.noarch 5/13 2024-09-18T07:55:38.203 INFO:teuthology.orchestra.run.smithi050.stdout:Last metadata expiration check: 0:01:48 ago on Wed 18 Sep 2024 07:53:50 AM UTC. 2024-09-18T07:55:38.242 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 6/13 2024-09-18T07:55:38.298 INFO:teuthology.orchestra.run.smithi050.stdout:Package libcephfs2-2:19.3.0-4983.ge7fb7b56.el9.x86_64 is already installed. 2024-09-18T07:55:38.344 INFO:teuthology.orchestra.run.smithi050.stdout:Dependencies resolved. 2024-09-18T07:55:38.345 INFO:teuthology.orchestra.run.smithi050.stdout:Nothing to do. 2024-09-18T07:55:38.346 INFO:teuthology.orchestra.run.smithi050.stdout:Complete! 2024-09-18T07:55:38.361 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : python3-google-auth-1:2.34.0-1.el9.noarch 7/13 2024-09-18T07:55:38.415 DEBUG:teuthology.orchestra.run.smithi050:> sudo yum -y install libcephfs-devel 2024-09-18T07:55:38.421 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 8/13 2024-09-18T07:55:39.026 INFO:teuthology.orchestra.run.smithi050.stdout:Last metadata expiration check: 0:01:49 ago on Wed 18 Sep 2024 07:53:50 AM UTC. 2024-09-18T07:55:39.166 INFO:teuthology.orchestra.run.smithi050.stdout:Dependencies resolved. 2024-09-18T07:55:39.167 INFO:teuthology.orchestra.run.smithi050.stdout:================================================================================ 2024-09-18T07:55:39.167 INFO:teuthology.orchestra.run.smithi050.stdout: Package Arch Version Repo Size 2024-09-18T07:55:39.167 INFO:teuthology.orchestra.run.smithi050.stdout:================================================================================ 2024-09-18T07:55:39.167 INFO:teuthology.orchestra.run.smithi050.stdout:Installing: 2024-09-18T07:55:39.167 INFO:teuthology.orchestra.run.smithi050.stdout: libcephfs-devel x86_64 2:19.3.0-4983.ge7fb7b56.el9 ceph 32 k 2024-09-18T07:55:39.167 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-09-18T07:55:39.167 INFO:teuthology.orchestra.run.smithi050.stdout:Transaction Summary 2024-09-18T07:55:39.167 INFO:teuthology.orchestra.run.smithi050.stdout:================================================================================ 2024-09-18T07:55:39.167 INFO:teuthology.orchestra.run.smithi050.stdout:Install 1 Package 2024-09-18T07:55:39.167 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-09-18T07:55:39.167 INFO:teuthology.orchestra.run.smithi050.stdout:Total download size: 32 k 2024-09-18T07:55:39.167 INFO:teuthology.orchestra.run.smithi050.stdout:Installed size: 142 k 2024-09-18T07:55:39.168 INFO:teuthology.orchestra.run.smithi050.stdout:Downloading Packages: 2024-09-18T07:55:39.183 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 9/13 2024-09-18T07:55:39.234 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 10/13 2024-09-18T07:55:39.268 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 11/13 2024-09-18T07:55:39.308 INFO:teuthology.orchestra.run.smithi050.stdout:libcephfs-devel-19.3.0-4983.ge7fb7b56.el9.x86_6 226 kB/s | 32 kB 00:00 2024-09-18T07:55:39.308 INFO:teuthology.orchestra.run.smithi050.stdout:-------------------------------------------------------------------------------- 2024-09-18T07:55:39.309 INFO:teuthology.orchestra.run.smithi050.stdout:Total 224 kB/s | 32 kB 00:00 2024-09-18T07:55:39.309 INFO:teuthology.orchestra.run.smithi050.stdout:Running transaction check 2024-09-18T07:55:39.313 INFO:teuthology.orchestra.run.smithi050.stdout:Transaction check succeeded. 2024-09-18T07:55:39.313 INFO:teuthology.orchestra.run.smithi050.stdout:Running transaction test 2024-09-18T07:55:39.317 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 12/13 2024-09-18T07:55:39.332 INFO:teuthology.orchestra.run.smithi050.stdout:Transaction test succeeded. 2024-09-18T07:55:39.333 INFO:teuthology.orchestra.run.smithi050.stdout:Running transaction 2024-09-18T07:55:39.346 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : ceph-mgr-rook-2:19.3.0-4983.ge7fb7b56.el9.noarch 13/13 2024-09-18T07:55:39.423 INFO:teuthology.orchestra.run.smithi050.stdout: Preparing : 1/1 2024-09-18T07:55:39.552 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : libcephfs-devel-2:19.3.0-4983.ge7fb7b56.el9.x86_64 1/1 2024-09-18T07:55:39.948 INFO:teuthology.orchestra.run.smithi050.stdout: Running scriptlet: libcephfs-devel-2:19.3.0-4983.ge7fb7b56.el9.x86_64 1/1 2024-09-18T07:55:40.111 INFO:teuthology.orchestra.run.smithi007.stdout: Running scriptlet: ceph-mgr-rook-2:19.3.0-4983.ge7fb7b56.el9.noarch 13/13 2024-09-18T07:55:40.111 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : ceph-mgr-rook-2:19.3.0-4983.ge7fb7b56.el9.noarch 1/13 2024-09-18T07:55:40.111 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 2/13 2024-09-18T07:55:40.111 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 3/13 2024-09-18T07:55:40.111 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 4/13 2024-09-18T07:55:40.111 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 5/13 2024-09-18T07:55:40.111 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 6/13 2024-09-18T07:55:40.111 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 7/13 2024-09-18T07:55:40.111 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 8/13 2024-09-18T07:55:40.111 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : python3-certifi-2023.05.07-4.el9.noarch 9/13 2024-09-18T07:55:40.112 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : python3-google-auth-1:2.34.0-1.el9.noarch 10/13 2024-09-18T07:55:40.112 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 11/13 2024-09-18T07:55:40.112 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 12/13 2024-09-18T07:55:40.184 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : libcephfs-devel-2:19.3.0-4983.ge7fb7b56.el9.x86_64 1/1 2024-09-18T07:55:40.184 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-09-18T07:55:40.184 INFO:teuthology.orchestra.run.smithi050.stdout:Installed: 2024-09-18T07:55:40.184 INFO:teuthology.orchestra.run.smithi050.stdout: libcephfs-devel-2:19.3.0-4983.ge7fb7b56.el9.x86_64 2024-09-18T07:55:40.184 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-09-18T07:55:40.184 INFO:teuthology.orchestra.run.smithi050.stdout:Complete! 2024-09-18T07:55:40.376 DEBUG:teuthology.orchestra.run.smithi050:> sudo yum -y install librados2 2024-09-18T07:55:40.464 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 13/13 2024-09-18T07:55:40.465 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-09-18T07:55:40.465 INFO:teuthology.orchestra.run.smithi007.stdout:Installed: 2024-09-18T07:55:40.465 INFO:teuthology.orchestra.run.smithi007.stdout: ceph-mgr-rook-2:19.3.0-4983.ge7fb7b56.el9.noarch 2024-09-18T07:55:40.465 INFO:teuthology.orchestra.run.smithi007.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-09-18T07:55:40.465 INFO:teuthology.orchestra.run.smithi007.stdout: python3-certifi-2023.05.07-4.el9.noarch 2024-09-18T07:55:40.465 INFO:teuthology.orchestra.run.smithi007.stdout: python3-google-auth-1:2.34.0-1.el9.noarch 2024-09-18T07:55:40.465 INFO:teuthology.orchestra.run.smithi007.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-09-18T07:55:40.465 INFO:teuthology.orchestra.run.smithi007.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-09-18T07:55:40.465 INFO:teuthology.orchestra.run.smithi007.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-09-18T07:55:40.465 INFO:teuthology.orchestra.run.smithi007.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-09-18T07:55:40.466 INFO:teuthology.orchestra.run.smithi007.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-09-18T07:55:40.466 INFO:teuthology.orchestra.run.smithi007.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-09-18T07:55:40.466 INFO:teuthology.orchestra.run.smithi007.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-09-18T07:55:40.466 INFO:teuthology.orchestra.run.smithi007.stdout: python3-rsa-4.9-2.el9.noarch 2024-09-18T07:55:40.466 INFO:teuthology.orchestra.run.smithi007.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-09-18T07:55:40.466 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-09-18T07:55:40.466 INFO:teuthology.orchestra.run.smithi007.stdout:Complete! 2024-09-18T07:55:40.737 DEBUG:teuthology.orchestra.run.smithi007:> sudo yum -y install ceph-mgr-cephadm 2024-09-18T07:55:40.990 INFO:teuthology.orchestra.run.smithi050.stdout:Last metadata expiration check: 0:01:50 ago on Wed 18 Sep 2024 07:53:50 AM UTC. 2024-09-18T07:55:41.084 INFO:teuthology.orchestra.run.smithi050.stdout:Package librados2-2:19.3.0-4983.ge7fb7b56.el9.x86_64 is already installed. 2024-09-18T07:55:41.131 INFO:teuthology.orchestra.run.smithi050.stdout:Dependencies resolved. 2024-09-18T07:55:41.132 INFO:teuthology.orchestra.run.smithi050.stdout:Nothing to do. 2024-09-18T07:55:41.132 INFO:teuthology.orchestra.run.smithi050.stdout:Complete! 2024-09-18T07:55:41.186 DEBUG:teuthology.orchestra.run.smithi050:> sudo yum -y install librbd1 2024-09-18T07:55:41.350 INFO:teuthology.orchestra.run.smithi007.stdout:Last metadata expiration check: 0:01:29 ago on Wed 18 Sep 2024 07:54:12 AM UTC. 2024-09-18T07:55:41.490 INFO:teuthology.orchestra.run.smithi007.stdout:Dependencies resolved. 2024-09-18T07:55:41.491 INFO:teuthology.orchestra.run.smithi007.stdout:================================================================================ 2024-09-18T07:55:41.491 INFO:teuthology.orchestra.run.smithi007.stdout: Package Arch Version Repository Size 2024-09-18T07:55:41.491 INFO:teuthology.orchestra.run.smithi007.stdout:================================================================================ 2024-09-18T07:55:41.491 INFO:teuthology.orchestra.run.smithi007.stdout:Installing: 2024-09-18T07:55:41.491 INFO:teuthology.orchestra.run.smithi007.stdout: ceph-mgr-cephadm noarch 2:19.3.0-4983.ge7fb7b56.el9 ceph-noarch 154 k 2024-09-18T07:55:41.491 INFO:teuthology.orchestra.run.smithi007.stdout:Installing dependencies: 2024-09-18T07:55:41.491 INFO:teuthology.orchestra.run.smithi007.stdout: python3-asyncssh noarch 2.13.2-5.el9 epel 548 k 2024-09-18T07:55:41.491 INFO:teuthology.orchestra.run.smithi007.stdout: python3-babel noarch 2.9.1-2.el9 appstream 6.0 M 2024-09-18T07:55:41.491 INFO:teuthology.orchestra.run.smithi007.stdout: python3-jinja2 noarch 2.11.3-6.el9 appstream 248 k 2024-09-18T07:55:41.491 INFO:teuthology.orchestra.run.smithi007.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-09-18T07:55:41.491 INFO:teuthology.orchestra.run.smithi007.stdout: python3-typing-extensions noarch 4.12.2-2.el9 epel 74 k 2024-09-18T07:55:41.491 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-09-18T07:55:41.491 INFO:teuthology.orchestra.run.smithi007.stdout:Transaction Summary 2024-09-18T07:55:41.492 INFO:teuthology.orchestra.run.smithi007.stdout:================================================================================ 2024-09-18T07:55:41.492 INFO:teuthology.orchestra.run.smithi007.stdout:Install 6 Packages 2024-09-18T07:55:41.492 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-09-18T07:55:41.492 INFO:teuthology.orchestra.run.smithi007.stdout:Total download size: 7.0 M 2024-09-18T07:55:41.492 INFO:teuthology.orchestra.run.smithi007.stdout:Installed size: 33 M 2024-09-18T07:55:41.492 INFO:teuthology.orchestra.run.smithi007.stdout:Downloading Packages: 2024-09-18T07:55:41.743 INFO:teuthology.orchestra.run.smithi007.stdout:(1/6): python3-jinja2-2.11.3-6.el9.noarch.rpm 1.4 MB/s | 248 kB 00:00 2024-09-18T07:55:41.786 INFO:teuthology.orchestra.run.smithi007.stdout:(2/6): ceph-mgr-cephadm-19.3.0-4983.ge7fb7b56.e 730 kB/s | 154 kB 00:00 2024-09-18T07:55:41.796 INFO:teuthology.orchestra.run.smithi050.stdout:Last metadata expiration check: 0:01:51 ago on Wed 18 Sep 2024 07:53:50 AM UTC. 2024-09-18T07:55:41.828 INFO:teuthology.orchestra.run.smithi007.stdout:(3/6): python3-natsort-7.1.1-5.el9.noarch.rpm 1.4 MB/s | 58 kB 00:00 2024-09-18T07:55:41.861 INFO:teuthology.orchestra.run.smithi007.stdout:(4/6): python3-asyncssh-2.13.2-5.el9.noarch.rpm 4.6 MB/s | 548 kB 00:00 2024-09-18T07:55:41.888 INFO:teuthology.orchestra.run.smithi050.stdout:Package librbd1-2:19.3.0-4983.ge7fb7b56.el9.x86_64 is already installed. 2024-09-18T07:55:41.935 INFO:teuthology.orchestra.run.smithi050.stdout:Dependencies resolved. 2024-09-18T07:55:41.937 INFO:teuthology.orchestra.run.smithi050.stdout:Nothing to do. 2024-09-18T07:55:41.937 INFO:teuthology.orchestra.run.smithi050.stdout:Complete! 2024-09-18T07:55:41.953 INFO:teuthology.orchestra.run.smithi007.stdout:(5/6): python3-babel-2.9.1-2.el9.noarch.rpm 16 MB/s | 6.0 MB 00:00 2024-09-18T07:55:41.986 INFO:teuthology.orchestra.run.smithi007.stdout:(6/6): python3-typing-extensions-4.12.2-2.el9.n 470 kB/s | 74 kB 00:00 2024-09-18T07:55:41.989 INFO:teuthology.orchestra.run.smithi007.stdout:-------------------------------------------------------------------------------- 2024-09-18T07:55:41.989 INFO:teuthology.orchestra.run.smithi007.stdout:Total 14 MB/s | 7.0 MB 00:00 2024-09-18T07:55:42.001 DEBUG:teuthology.orchestra.run.smithi050:> sudo yum -y install python3-rados 2024-09-18T07:55:42.063 INFO:teuthology.orchestra.run.smithi007.stdout:Running transaction check 2024-09-18T07:55:42.075 INFO:teuthology.orchestra.run.smithi007.stdout:Transaction check succeeded. 2024-09-18T07:55:42.075 INFO:teuthology.orchestra.run.smithi007.stdout:Running transaction test 2024-09-18T07:55:42.183 INFO:teuthology.orchestra.run.smithi007.stdout:Transaction test succeeded. 2024-09-18T07:55:42.188 INFO:teuthology.orchestra.run.smithi007.stdout:Running transaction 2024-09-18T07:55:42.503 INFO:teuthology.orchestra.run.smithi007.stdout: Preparing : 1/1 2024-09-18T07:55:42.621 INFO:teuthology.orchestra.run.smithi050.stdout:Last metadata expiration check: 0:01:52 ago on Wed 18 Sep 2024 07:53:50 AM UTC. 2024-09-18T07:55:42.679 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : python3-typing-extensions-4.12.2-2.el9.noarch 1/6 2024-09-18T07:55:42.714 INFO:teuthology.orchestra.run.smithi050.stdout:Package python3-rados-2:19.3.0-4983.ge7fb7b56.el9.x86_64 is already installed. 2024-09-18T07:55:42.732 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : python3-asyncssh-2.13.2-5.el9.noarch 2/6 2024-09-18T07:55:42.761 INFO:teuthology.orchestra.run.smithi050.stdout:Dependencies resolved. 2024-09-18T07:55:42.762 INFO:teuthology.orchestra.run.smithi050.stdout:Nothing to do. 2024-09-18T07:55:42.762 INFO:teuthology.orchestra.run.smithi050.stdout:Complete! 2024-09-18T07:55:42.835 DEBUG:teuthology.orchestra.run.smithi050:> sudo yum -y install python3-rgw 2024-09-18T07:55:43.294 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 3/6 2024-09-18T07:55:43.387 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 4/6 2024-09-18T07:55:43.448 INFO:teuthology.orchestra.run.smithi050.stdout:Last metadata expiration check: 0:01:53 ago on Wed 18 Sep 2024 07:53:50 AM UTC. 2024-09-18T07:55:43.463 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : python3-jinja2-2.11.3-6.el9.noarch 5/6 2024-09-18T07:55:43.493 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : ceph-mgr-cephadm-2:19.3.0-4983.ge7fb7b56.el9.noarch 6/6 2024-09-18T07:55:43.540 INFO:teuthology.orchestra.run.smithi050.stdout:Package python3-rgw-2:19.3.0-4983.ge7fb7b56.el9.x86_64 is already installed. 2024-09-18T07:55:43.586 INFO:teuthology.orchestra.run.smithi050.stdout:Dependencies resolved. 2024-09-18T07:55:43.587 INFO:teuthology.orchestra.run.smithi050.stdout:Nothing to do. 2024-09-18T07:55:43.587 INFO:teuthology.orchestra.run.smithi050.stdout:Complete! 2024-09-18T07:55:43.647 DEBUG:teuthology.orchestra.run.smithi050:> sudo yum -y install python3-cephfs 2024-09-18T07:55:44.259 INFO:teuthology.orchestra.run.smithi050.stdout:Last metadata expiration check: 0:01:54 ago on Wed 18 Sep 2024 07:53:50 AM UTC. 2024-09-18T07:55:44.354 INFO:teuthology.orchestra.run.smithi050.stdout:Package python3-cephfs-2:19.3.0-4983.ge7fb7b56.el9.x86_64 is already installed. 2024-09-18T07:55:44.400 INFO:teuthology.orchestra.run.smithi050.stdout:Dependencies resolved. 2024-09-18T07:55:44.401 INFO:teuthology.orchestra.run.smithi050.stdout:Nothing to do. 2024-09-18T07:55:44.401 INFO:teuthology.orchestra.run.smithi050.stdout:Complete! 2024-09-18T07:55:44.432 INFO:teuthology.orchestra.run.smithi007.stdout: Running scriptlet: ceph-mgr-cephadm-2:19.3.0-4983.ge7fb7b56.el9.noarch 6/6 2024-09-18T07:55:44.432 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : ceph-mgr-cephadm-2:19.3.0-4983.ge7fb7b56.el9.noarch 1/6 2024-09-18T07:55:44.432 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 2/6 2024-09-18T07:55:44.432 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : python3-jinja2-2.11.3-6.el9.noarch 3/6 2024-09-18T07:55:44.432 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : python3-asyncssh-2.13.2-5.el9.noarch 4/6 2024-09-18T07:55:44.432 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 5/6 2024-09-18T07:55:44.461 DEBUG:teuthology.orchestra.run.smithi050:> sudo yum -y install python3-rbd 2024-09-18T07:55:44.722 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : python3-typing-extensions-4.12.2-2.el9.noarch 6/6 2024-09-18T07:55:44.722 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-09-18T07:55:44.722 INFO:teuthology.orchestra.run.smithi007.stdout:Installed: 2024-09-18T07:55:44.722 INFO:teuthology.orchestra.run.smithi007.stdout: ceph-mgr-cephadm-2:19.3.0-4983.ge7fb7b56.el9.noarch 2024-09-18T07:55:44.722 INFO:teuthology.orchestra.run.smithi007.stdout: python3-asyncssh-2.13.2-5.el9.noarch 2024-09-18T07:55:44.722 INFO:teuthology.orchestra.run.smithi007.stdout: python3-babel-2.9.1-2.el9.noarch 2024-09-18T07:55:44.722 INFO:teuthology.orchestra.run.smithi007.stdout: python3-jinja2-2.11.3-6.el9.noarch 2024-09-18T07:55:44.722 INFO:teuthology.orchestra.run.smithi007.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-09-18T07:55:44.722 INFO:teuthology.orchestra.run.smithi007.stdout: python3-typing-extensions-4.12.2-2.el9.noarch 2024-09-18T07:55:44.723 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-09-18T07:55:44.723 INFO:teuthology.orchestra.run.smithi007.stdout:Complete! 2024-09-18T07:55:44.981 DEBUG:teuthology.orchestra.run.smithi007:> sudo yum -y install ceph-fuse 2024-09-18T07:55:45.071 INFO:teuthology.orchestra.run.smithi050.stdout:Last metadata expiration check: 0:01:55 ago on Wed 18 Sep 2024 07:53:50 AM UTC. 2024-09-18T07:55:45.164 INFO:teuthology.orchestra.run.smithi050.stdout:Package python3-rbd-2:19.3.0-4983.ge7fb7b56.el9.x86_64 is already installed. 2024-09-18T07:55:45.210 INFO:teuthology.orchestra.run.smithi050.stdout:Dependencies resolved. 2024-09-18T07:55:45.211 INFO:teuthology.orchestra.run.smithi050.stdout:Nothing to do. 2024-09-18T07:55:45.211 INFO:teuthology.orchestra.run.smithi050.stdout:Complete! 2024-09-18T07:55:45.272 DEBUG:teuthology.orchestra.run.smithi050:> sudo yum -y install rbd-fuse 2024-09-18T07:55:45.595 INFO:teuthology.orchestra.run.smithi007.stdout:Last metadata expiration check: 0:01:33 ago on Wed 18 Sep 2024 07:54:12 AM UTC. 2024-09-18T07:55:45.733 INFO:teuthology.orchestra.run.smithi007.stdout:Dependencies resolved. 2024-09-18T07:55:45.733 INFO:teuthology.orchestra.run.smithi007.stdout:================================================================================ 2024-09-18T07:55:45.734 INFO:teuthology.orchestra.run.smithi007.stdout: Package Arch Version Repository Size 2024-09-18T07:55:45.734 INFO:teuthology.orchestra.run.smithi007.stdout:================================================================================ 2024-09-18T07:55:45.734 INFO:teuthology.orchestra.run.smithi007.stdout:Installing: 2024-09-18T07:55:45.734 INFO:teuthology.orchestra.run.smithi007.stdout: ceph-fuse x86_64 2:19.3.0-4983.ge7fb7b56.el9 ceph 901 k 2024-09-18T07:55:45.734 INFO:teuthology.orchestra.run.smithi007.stdout:Installing dependencies: 2024-09-18T07:55:45.734 INFO:teuthology.orchestra.run.smithi007.stdout: fuse x86_64 2.9.9-16.el9 baseos 80 k 2024-09-18T07:55:45.734 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-09-18T07:55:45.734 INFO:teuthology.orchestra.run.smithi007.stdout:Transaction Summary 2024-09-18T07:55:45.734 INFO:teuthology.orchestra.run.smithi007.stdout:================================================================================ 2024-09-18T07:55:45.734 INFO:teuthology.orchestra.run.smithi007.stdout:Install 2 Packages 2024-09-18T07:55:45.734 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-09-18T07:55:45.735 INFO:teuthology.orchestra.run.smithi007.stdout:Total download size: 980 k 2024-09-18T07:55:45.735 INFO:teuthology.orchestra.run.smithi007.stdout:Installed size: 2.9 M 2024-09-18T07:55:45.735 INFO:teuthology.orchestra.run.smithi007.stdout:Downloading Packages: 2024-09-18T07:55:45.884 INFO:teuthology.orchestra.run.smithi050.stdout:Last metadata expiration check: 0:01:55 ago on Wed 18 Sep 2024 07:53:50 AM UTC. 2024-09-18T07:55:46.023 INFO:teuthology.orchestra.run.smithi050.stdout:Dependencies resolved. 2024-09-18T07:55:46.024 INFO:teuthology.orchestra.run.smithi050.stdout:================================================================================ 2024-09-18T07:55:46.024 INFO:teuthology.orchestra.run.smithi050.stdout: Package Architecture Version Repository Size 2024-09-18T07:55:46.024 INFO:teuthology.orchestra.run.smithi050.stdout:================================================================================ 2024-09-18T07:55:46.024 INFO:teuthology.orchestra.run.smithi050.stdout:Installing: 2024-09-18T07:55:46.024 INFO:teuthology.orchestra.run.smithi050.stdout: rbd-fuse x86_64 2:19.3.0-4983.ge7fb7b56.el9 ceph 85 k 2024-09-18T07:55:46.024 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-09-18T07:55:46.025 INFO:teuthology.orchestra.run.smithi050.stdout:Transaction Summary 2024-09-18T07:55:46.025 INFO:teuthology.orchestra.run.smithi050.stdout:================================================================================ 2024-09-18T07:55:46.025 INFO:teuthology.orchestra.run.smithi050.stdout:Install 1 Package 2024-09-18T07:55:46.025 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-09-18T07:55:46.025 INFO:teuthology.orchestra.run.smithi050.stdout:Total download size: 85 k 2024-09-18T07:55:46.025 INFO:teuthology.orchestra.run.smithi050.stdout:Installed size: 231 k 2024-09-18T07:55:46.025 INFO:teuthology.orchestra.run.smithi050.stdout:Downloading Packages: 2024-09-18T07:55:46.040 INFO:teuthology.orchestra.run.smithi007.stdout:(1/2): fuse-2.9.9-16.el9.x86_64.rpm 450 kB/s | 80 kB 00:00 2024-09-18T07:55:46.181 INFO:teuthology.orchestra.run.smithi007.stdout:(2/2): ceph-fuse-19.3.0-4983.ge7fb7b56.el9.x86_ 2.8 MB/s | 901 kB 00:00 2024-09-18T07:55:46.181 INFO:teuthology.orchestra.run.smithi007.stdout:-------------------------------------------------------------------------------- 2024-09-18T07:55:46.182 INFO:teuthology.orchestra.run.smithi007.stdout:Total 2.1 MB/s | 980 kB 00:00 2024-09-18T07:55:46.194 INFO:teuthology.orchestra.run.smithi007.stdout:Running transaction check 2024-09-18T07:55:46.218 INFO:teuthology.orchestra.run.smithi007.stdout:Transaction check succeeded. 2024-09-18T07:55:46.218 INFO:teuthology.orchestra.run.smithi007.stdout:Running transaction test 2024-09-18T07:55:46.222 INFO:teuthology.orchestra.run.smithi050.stdout:rbd-fuse-19.3.0-4983.ge7fb7b56.el9.x86_64.rpm 433 kB/s | 85 kB 00:00 2024-09-18T07:55:46.223 INFO:teuthology.orchestra.run.smithi050.stdout:-------------------------------------------------------------------------------- 2024-09-18T07:55:46.223 INFO:teuthology.orchestra.run.smithi050.stdout:Total 430 kB/s | 85 kB 00:00 2024-09-18T07:55:46.223 INFO:teuthology.orchestra.run.smithi050.stdout:Running transaction check 2024-09-18T07:55:46.231 INFO:teuthology.orchestra.run.smithi050.stdout:Transaction check succeeded. 2024-09-18T07:55:46.231 INFO:teuthology.orchestra.run.smithi050.stdout:Running transaction test 2024-09-18T07:55:46.305 INFO:teuthology.orchestra.run.smithi007.stdout:Transaction test succeeded. 2024-09-18T07:55:46.305 INFO:teuthology.orchestra.run.smithi007.stdout:Running transaction 2024-09-18T07:55:46.305 INFO:teuthology.orchestra.run.smithi050.stdout:Transaction test succeeded. 2024-09-18T07:55:46.305 INFO:teuthology.orchestra.run.smithi050.stdout:Running transaction 2024-09-18T07:55:46.470 INFO:teuthology.orchestra.run.smithi050.stdout: Preparing : 1/1 2024-09-18T07:55:46.504 INFO:teuthology.orchestra.run.smithi007.stdout: Preparing : 1/1 2024-09-18T07:55:46.573 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : rbd-fuse-2:19.3.0-4983.ge7fb7b56.el9.x86_64 1/1 2024-09-18T07:55:46.655 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : fuse-2.9.9-16.el9.x86_64 1/2 2024-09-18T07:55:46.750 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : ceph-fuse-2:19.3.0-4983.ge7fb7b56.el9.x86_64 2/2 2024-09-18T07:55:46.969 INFO:teuthology.orchestra.run.smithi050.stdout: Running scriptlet: rbd-fuse-2:19.3.0-4983.ge7fb7b56.el9.x86_64 1/1 2024-09-18T07:55:47.234 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : rbd-fuse-2:19.3.0-4983.ge7fb7b56.el9.x86_64 1/1 2024-09-18T07:55:47.234 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-09-18T07:55:47.234 INFO:teuthology.orchestra.run.smithi050.stdout:Installed: 2024-09-18T07:55:47.234 INFO:teuthology.orchestra.run.smithi050.stdout: rbd-fuse-2:19.3.0-4983.ge7fb7b56.el9.x86_64 2024-09-18T07:55:47.234 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-09-18T07:55:47.234 INFO:teuthology.orchestra.run.smithi050.stdout:Complete! 2024-09-18T07:55:47.475 DEBUG:teuthology.orchestra.run.smithi050:> sudo yum -y install rbd-mirror 2024-09-18T07:55:47.490 INFO:teuthology.orchestra.run.smithi007.stdout: Running scriptlet: ceph-fuse-2:19.3.0-4983.ge7fb7b56.el9.x86_64 2/2 2024-09-18T07:55:47.490 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : ceph-fuse-2:19.3.0-4983.ge7fb7b56.el9.x86_64 1/2 2024-09-18T07:55:47.797 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : fuse-2.9.9-16.el9.x86_64 2/2 2024-09-18T07:55:47.797 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-09-18T07:55:47.797 INFO:teuthology.orchestra.run.smithi007.stdout:Installed: 2024-09-18T07:55:47.797 INFO:teuthology.orchestra.run.smithi007.stdout: ceph-fuse-2:19.3.0-4983.ge7fb7b56.el9.x86_64 fuse-2.9.9-16.el9.x86_64 2024-09-18T07:55:47.798 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-09-18T07:55:47.798 INFO:teuthology.orchestra.run.smithi007.stdout:Complete! 2024-09-18T07:55:48.084 INFO:teuthology.orchestra.run.smithi050.stdout:Last metadata expiration check: 0:01:58 ago on Wed 18 Sep 2024 07:53:50 AM UTC. 2024-09-18T07:55:48.088 DEBUG:teuthology.orchestra.run.smithi007:> sudo yum -y install ceph-volume 2024-09-18T07:55:48.227 INFO:teuthology.orchestra.run.smithi050.stdout:Dependencies resolved. 2024-09-18T07:55:48.228 INFO:teuthology.orchestra.run.smithi050.stdout:================================================================================ 2024-09-18T07:55:48.228 INFO:teuthology.orchestra.run.smithi050.stdout: Package Arch Version Repo Size 2024-09-18T07:55:48.228 INFO:teuthology.orchestra.run.smithi050.stdout:================================================================================ 2024-09-18T07:55:48.228 INFO:teuthology.orchestra.run.smithi050.stdout:Installing: 2024-09-18T07:55:48.228 INFO:teuthology.orchestra.run.smithi050.stdout: rbd-mirror x86_64 2:19.3.0-4983.ge7fb7b56.el9 ceph 3.2 M 2024-09-18T07:55:48.228 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-09-18T07:55:48.229 INFO:teuthology.orchestra.run.smithi050.stdout:Transaction Summary 2024-09-18T07:55:48.229 INFO:teuthology.orchestra.run.smithi050.stdout:================================================================================ 2024-09-18T07:55:48.229 INFO:teuthology.orchestra.run.smithi050.stdout:Install 1 Package 2024-09-18T07:55:48.229 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-09-18T07:55:48.229 INFO:teuthology.orchestra.run.smithi050.stdout:Total download size: 3.2 M 2024-09-18T07:55:48.229 INFO:teuthology.orchestra.run.smithi050.stdout:Installed size: 14 M 2024-09-18T07:55:48.229 INFO:teuthology.orchestra.run.smithi050.stdout:Downloading Packages: 2024-09-18T07:55:48.576 INFO:teuthology.orchestra.run.smithi050.stdout:rbd-mirror-19.3.0-4983.ge7fb7b56.el9.x86_64.rpm 9.2 MB/s | 3.2 MB 00:00 2024-09-18T07:55:48.577 INFO:teuthology.orchestra.run.smithi050.stdout:-------------------------------------------------------------------------------- 2024-09-18T07:55:48.577 INFO:teuthology.orchestra.run.smithi050.stdout:Total 9.1 MB/s | 3.2 MB 00:00 2024-09-18T07:55:48.577 INFO:teuthology.orchestra.run.smithi050.stdout:Running transaction check 2024-09-18T07:55:48.588 INFO:teuthology.orchestra.run.smithi050.stdout:Transaction check succeeded. 2024-09-18T07:55:48.588 INFO:teuthology.orchestra.run.smithi050.stdout:Running transaction test 2024-09-18T07:55:48.676 INFO:teuthology.orchestra.run.smithi050.stdout:Transaction test succeeded. 2024-09-18T07:55:48.676 INFO:teuthology.orchestra.run.smithi050.stdout:Running transaction 2024-09-18T07:55:48.705 INFO:teuthology.orchestra.run.smithi007.stdout:Last metadata expiration check: 0:01:36 ago on Wed 18 Sep 2024 07:54:12 AM UTC. 2024-09-18T07:55:48.846 INFO:teuthology.orchestra.run.smithi007.stdout:Dependencies resolved. 2024-09-18T07:55:48.846 INFO:teuthology.orchestra.run.smithi007.stdout:================================================================================ 2024-09-18T07:55:48.846 INFO:teuthology.orchestra.run.smithi007.stdout: Package Arch Version Repository Size 2024-09-18T07:55:48.846 INFO:teuthology.orchestra.run.smithi007.stdout:================================================================================ 2024-09-18T07:55:48.846 INFO:teuthology.orchestra.run.smithi007.stdout:Installing: 2024-09-18T07:55:48.846 INFO:teuthology.orchestra.run.smithi007.stdout: ceph-volume noarch 2:19.3.0-4983.ge7fb7b56.el9 ceph-noarch 285 k 2024-09-18T07:55:48.846 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-09-18T07:55:48.846 INFO:teuthology.orchestra.run.smithi007.stdout:Transaction Summary 2024-09-18T07:55:48.846 INFO:teuthology.orchestra.run.smithi007.stdout:================================================================================ 2024-09-18T07:55:48.846 INFO:teuthology.orchestra.run.smithi007.stdout:Install 1 Package 2024-09-18T07:55:48.846 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-09-18T07:55:48.847 INFO:teuthology.orchestra.run.smithi007.stdout:Total download size: 285 k 2024-09-18T07:55:48.847 INFO:teuthology.orchestra.run.smithi007.stdout:Installed size: 1.3 M 2024-09-18T07:55:48.847 INFO:teuthology.orchestra.run.smithi007.stdout:Downloading Packages: 2024-09-18T07:55:49.053 INFO:teuthology.orchestra.run.smithi007.stdout:ceph-volume-19.3.0-4983.ge7fb7b56.el9.noarch.rp 1.4 MB/s | 285 kB 00:00 2024-09-18T07:55:49.053 INFO:teuthology.orchestra.run.smithi007.stdout:-------------------------------------------------------------------------------- 2024-09-18T07:55:49.053 INFO:teuthology.orchestra.run.smithi007.stdout:Total 1.3 MB/s | 285 kB 00:00 2024-09-18T07:55:49.054 INFO:teuthology.orchestra.run.smithi007.stdout:Running transaction check 2024-09-18T07:55:49.063 INFO:teuthology.orchestra.run.smithi007.stdout:Transaction check succeeded. 2024-09-18T07:55:49.063 INFO:teuthology.orchestra.run.smithi007.stdout:Running transaction test 2024-09-18T07:55:49.095 INFO:teuthology.orchestra.run.smithi007.stdout:Transaction test succeeded. 2024-09-18T07:55:49.096 INFO:teuthology.orchestra.run.smithi007.stdout:Running transaction 2024-09-18T07:55:49.173 INFO:teuthology.orchestra.run.smithi050.stdout: Preparing : 1/1 2024-09-18T07:55:49.227 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : rbd-mirror-2:19.3.0-4983.ge7fb7b56.el9.x86_64 1/1 2024-09-18T07:55:49.254 INFO:teuthology.orchestra.run.smithi050.stdout: Running scriptlet: rbd-mirror-2:19.3.0-4983.ge7fb7b56.el9.x86_64 1/1 2024-09-18T07:55:49.254 INFO:teuthology.orchestra.run.smithi050.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-18T07:55:49.254 INFO:teuthology.orchestra.run.smithi050.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-09-18T07:55:49.255 INFO:teuthology.orchestra.run.smithi050.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-09-18T07:55:49.255 INFO:teuthology.orchestra.run.smithi050.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-09-18T07:55:49.255 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-09-18T07:55:49.323 INFO:teuthology.orchestra.run.smithi007.stdout: Preparing : 1/1 2024-09-18T07:55:49.413 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : ceph-volume-2:19.3.0-4983.ge7fb7b56.el9.noarch 1/1 2024-09-18T07:55:49.430 INFO:teuthology.orchestra.run.smithi007.stdout: Running scriptlet: ceph-volume-2:19.3.0-4983.ge7fb7b56.el9.noarch 1/1 2024-09-18T07:55:49.430 INFO:teuthology.orchestra.run.smithi007.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-18T07:55:49.430 INFO:teuthology.orchestra.run.smithi007.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2024-09-18T07:55:49.430 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-09-18T07:55:50.382 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : rbd-mirror-2:19.3.0-4983.ge7fb7b56.el9.x86_64 1/1 2024-09-18T07:55:50.382 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-09-18T07:55:50.382 INFO:teuthology.orchestra.run.smithi050.stdout:Installed: 2024-09-18T07:55:50.382 INFO:teuthology.orchestra.run.smithi050.stdout: rbd-mirror-2:19.3.0-4983.ge7fb7b56.el9.x86_64 2024-09-18T07:55:50.382 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-09-18T07:55:50.382 INFO:teuthology.orchestra.run.smithi050.stdout:Complete! 2024-09-18T07:55:50.534 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : ceph-volume-2:19.3.0-4983.ge7fb7b56.el9.noarch 1/1 2024-09-18T07:55:50.534 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-09-18T07:55:50.535 INFO:teuthology.orchestra.run.smithi007.stdout:Installed: 2024-09-18T07:55:50.535 INFO:teuthology.orchestra.run.smithi007.stdout: ceph-volume-2:19.3.0-4983.ge7fb7b56.el9.noarch 2024-09-18T07:55:50.535 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-09-18T07:55:50.535 INFO:teuthology.orchestra.run.smithi007.stdout:Complete! 2024-09-18T07:55:50.589 DEBUG:teuthology.orchestra.run.smithi050:> sudo yum -y install rbd-nbd 2024-09-18T07:55:50.880 DEBUG:teuthology.orchestra.run.smithi007:> sudo yum -y install librados-devel 2024-09-18T07:55:51.193 INFO:teuthology.orchestra.run.smithi050.stdout:Last metadata expiration check: 0:02:01 ago on Wed 18 Sep 2024 07:53:50 AM UTC. 2024-09-18T07:55:51.330 INFO:teuthology.orchestra.run.smithi050.stdout:Dependencies resolved. 2024-09-18T07:55:51.330 INFO:teuthology.orchestra.run.smithi050.stdout:================================================================================ 2024-09-18T07:55:51.331 INFO:teuthology.orchestra.run.smithi050.stdout: Package Architecture Version Repository Size 2024-09-18T07:55:51.331 INFO:teuthology.orchestra.run.smithi050.stdout:================================================================================ 2024-09-18T07:55:51.331 INFO:teuthology.orchestra.run.smithi050.stdout:Installing: 2024-09-18T07:55:51.331 INFO:teuthology.orchestra.run.smithi050.stdout: rbd-nbd x86_64 2:19.3.0-4983.ge7fb7b56.el9 ceph 170 k 2024-09-18T07:55:51.331 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-09-18T07:55:51.331 INFO:teuthology.orchestra.run.smithi050.stdout:Transaction Summary 2024-09-18T07:55:51.331 INFO:teuthology.orchestra.run.smithi050.stdout:================================================================================ 2024-09-18T07:55:51.331 INFO:teuthology.orchestra.run.smithi050.stdout:Install 1 Package 2024-09-18T07:55:51.331 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-09-18T07:55:51.331 INFO:teuthology.orchestra.run.smithi050.stdout:Total download size: 170 k 2024-09-18T07:55:51.331 INFO:teuthology.orchestra.run.smithi050.stdout:Installed size: 490 k 2024-09-18T07:55:51.331 INFO:teuthology.orchestra.run.smithi050.stdout:Downloading Packages: 2024-09-18T07:55:51.493 INFO:teuthology.orchestra.run.smithi007.stdout:Last metadata expiration check: 0:01:39 ago on Wed 18 Sep 2024 07:54:12 AM UTC. 2024-09-18T07:55:51.564 INFO:teuthology.orchestra.run.smithi050.stdout:rbd-nbd-19.3.0-4983.ge7fb7b56.el9.x86_64.rpm 734 kB/s | 170 kB 00:00 2024-09-18T07:55:51.565 INFO:teuthology.orchestra.run.smithi050.stdout:-------------------------------------------------------------------------------- 2024-09-18T07:55:51.565 INFO:teuthology.orchestra.run.smithi050.stdout:Total 729 kB/s | 170 kB 00:00 2024-09-18T07:55:51.565 INFO:teuthology.orchestra.run.smithi050.stdout:Running transaction check 2024-09-18T07:55:51.573 INFO:teuthology.orchestra.run.smithi050.stdout:Transaction check succeeded. 2024-09-18T07:55:51.573 INFO:teuthology.orchestra.run.smithi050.stdout:Running transaction test 2024-09-18T07:55:51.632 INFO:teuthology.orchestra.run.smithi007.stdout:Dependencies resolved. 2024-09-18T07:55:51.633 INFO:teuthology.orchestra.run.smithi007.stdout:================================================================================ 2024-09-18T07:55:51.633 INFO:teuthology.orchestra.run.smithi007.stdout: Package Arch Version Repo Size 2024-09-18T07:55:51.633 INFO:teuthology.orchestra.run.smithi007.stdout:================================================================================ 2024-09-18T07:55:51.633 INFO:teuthology.orchestra.run.smithi007.stdout:Installing: 2024-09-18T07:55:51.633 INFO:teuthology.orchestra.run.smithi007.stdout: librados-devel x86_64 2:19.3.0-4983.ge7fb7b56.el9 ceph 127 k 2024-09-18T07:55:51.633 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-09-18T07:55:51.633 INFO:teuthology.orchestra.run.smithi007.stdout:Transaction Summary 2024-09-18T07:55:51.633 INFO:teuthology.orchestra.run.smithi007.stdout:================================================================================ 2024-09-18T07:55:51.634 INFO:teuthology.orchestra.run.smithi007.stdout:Install 1 Package 2024-09-18T07:55:51.634 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-09-18T07:55:51.634 INFO:teuthology.orchestra.run.smithi007.stdout:Total download size: 127 k 2024-09-18T07:55:51.634 INFO:teuthology.orchestra.run.smithi007.stdout:Installed size: 456 k 2024-09-18T07:55:51.634 INFO:teuthology.orchestra.run.smithi007.stdout:Downloading Packages: 2024-09-18T07:55:51.641 INFO:teuthology.orchestra.run.smithi050.stdout:Transaction test succeeded. 2024-09-18T07:55:51.642 INFO:teuthology.orchestra.run.smithi050.stdout:Running transaction 2024-09-18T07:55:51.829 INFO:teuthology.orchestra.run.smithi050.stdout: Preparing : 1/1 2024-09-18T07:55:51.834 INFO:teuthology.orchestra.run.smithi007.stdout:librados-devel-19.3.0-4983.ge7fb7b56.el9.x86_64 635 kB/s | 127 kB 00:00 2024-09-18T07:55:51.834 INFO:teuthology.orchestra.run.smithi007.stdout:-------------------------------------------------------------------------------- 2024-09-18T07:55:51.835 INFO:teuthology.orchestra.run.smithi007.stdout:Total 629 kB/s | 127 kB 00:00 2024-09-18T07:55:51.835 INFO:teuthology.orchestra.run.smithi007.stdout:Running transaction check 2024-09-18T07:55:51.841 INFO:teuthology.orchestra.run.smithi007.stdout:Transaction check succeeded. 2024-09-18T07:55:51.841 INFO:teuthology.orchestra.run.smithi007.stdout:Running transaction test 2024-09-18T07:55:51.910 INFO:teuthology.orchestra.run.smithi007.stdout:Transaction test succeeded. 2024-09-18T07:55:51.910 INFO:teuthology.orchestra.run.smithi007.stdout:Running transaction 2024-09-18T07:55:51.924 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : rbd-nbd-2:19.3.0-4983.ge7fb7b56.el9.x86_64 1/1 2024-09-18T07:55:52.066 INFO:teuthology.orchestra.run.smithi007.stdout: Preparing : 1/1 2024-09-18T07:55:52.168 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : librados-devel-2:19.3.0-4983.ge7fb7b56.el9.x86_64 1/1 2024-09-18T07:55:52.346 INFO:teuthology.orchestra.run.smithi050.stdout: Running scriptlet: rbd-nbd-2:19.3.0-4983.ge7fb7b56.el9.x86_64 1/1 2024-09-18T07:55:52.626 INFO:teuthology.orchestra.run.smithi007.stdout: Running scriptlet: librados-devel-2:19.3.0-4983.ge7fb7b56.el9.x86_64 1/1 2024-09-18T07:55:52.640 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : rbd-nbd-2:19.3.0-4983.ge7fb7b56.el9.x86_64 1/1 2024-09-18T07:55:52.640 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-09-18T07:55:52.640 INFO:teuthology.orchestra.run.smithi050.stdout:Installed: 2024-09-18T07:55:52.640 INFO:teuthology.orchestra.run.smithi050.stdout: rbd-nbd-2:19.3.0-4983.ge7fb7b56.el9.x86_64 2024-09-18T07:55:52.640 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-09-18T07:55:52.641 INFO:teuthology.orchestra.run.smithi050.stdout:Complete! 2024-09-18T07:55:52.877 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : librados-devel-2:19.3.0-4983.ge7fb7b56.el9.x86_64 1/1 2024-09-18T07:55:52.878 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-09-18T07:55:52.878 INFO:teuthology.orchestra.run.smithi007.stdout:Installed: 2024-09-18T07:55:52.878 INFO:teuthology.orchestra.run.smithi007.stdout: librados-devel-2:19.3.0-4983.ge7fb7b56.el9.x86_64 2024-09-18T07:55:52.878 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-09-18T07:55:52.878 INFO:teuthology.orchestra.run.smithi007.stdout:Complete! 2024-09-18T07:55:52.897 DEBUG:teuthology.parallel:result is None 2024-09-18T07:55:53.172 DEBUG:teuthology.orchestra.run.smithi007:> sudo yum -y install libcephfs2 2024-09-18T07:55:53.794 INFO:teuthology.orchestra.run.smithi007.stdout:Last metadata expiration check: 0:01:41 ago on Wed 18 Sep 2024 07:54:12 AM UTC. 2024-09-18T07:55:53.890 INFO:teuthology.orchestra.run.smithi007.stdout:Package libcephfs2-2:19.3.0-4983.ge7fb7b56.el9.x86_64 is already installed. 2024-09-18T07:55:53.938 INFO:teuthology.orchestra.run.smithi007.stdout:Dependencies resolved. 2024-09-18T07:55:53.939 INFO:teuthology.orchestra.run.smithi007.stdout:Nothing to do. 2024-09-18T07:55:53.939 INFO:teuthology.orchestra.run.smithi007.stdout:Complete! 2024-09-18T07:55:54.003 DEBUG:teuthology.orchestra.run.smithi007:> sudo yum -y install libcephfs-devel 2024-09-18T07:55:54.616 INFO:teuthology.orchestra.run.smithi007.stdout:Last metadata expiration check: 0:01:42 ago on Wed 18 Sep 2024 07:54:12 AM UTC. 2024-09-18T07:55:54.755 INFO:teuthology.orchestra.run.smithi007.stdout:Dependencies resolved. 2024-09-18T07:55:54.756 INFO:teuthology.orchestra.run.smithi007.stdout:================================================================================ 2024-09-18T07:55:54.756 INFO:teuthology.orchestra.run.smithi007.stdout: Package Arch Version Repo Size 2024-09-18T07:55:54.756 INFO:teuthology.orchestra.run.smithi007.stdout:================================================================================ 2024-09-18T07:55:54.756 INFO:teuthology.orchestra.run.smithi007.stdout:Installing: 2024-09-18T07:55:54.756 INFO:teuthology.orchestra.run.smithi007.stdout: libcephfs-devel x86_64 2:19.3.0-4983.ge7fb7b56.el9 ceph 32 k 2024-09-18T07:55:54.756 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-09-18T07:55:54.757 INFO:teuthology.orchestra.run.smithi007.stdout:Transaction Summary 2024-09-18T07:55:54.757 INFO:teuthology.orchestra.run.smithi007.stdout:================================================================================ 2024-09-18T07:55:54.757 INFO:teuthology.orchestra.run.smithi007.stdout:Install 1 Package 2024-09-18T07:55:54.757 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-09-18T07:55:54.757 INFO:teuthology.orchestra.run.smithi007.stdout:Total download size: 32 k 2024-09-18T07:55:54.757 INFO:teuthology.orchestra.run.smithi007.stdout:Installed size: 142 k 2024-09-18T07:55:54.758 INFO:teuthology.orchestra.run.smithi007.stdout:Downloading Packages: 2024-09-18T07:55:54.922 INFO:teuthology.orchestra.run.smithi007.stdout:libcephfs-devel-19.3.0-4983.ge7fb7b56.el9.x86_6 193 kB/s | 32 kB 00:00 2024-09-18T07:55:54.923 INFO:teuthology.orchestra.run.smithi007.stdout:-------------------------------------------------------------------------------- 2024-09-18T07:55:54.923 INFO:teuthology.orchestra.run.smithi007.stdout:Total 191 kB/s | 32 kB 00:00 2024-09-18T07:55:54.923 INFO:teuthology.orchestra.run.smithi007.stdout:Running transaction check 2024-09-18T07:55:54.927 INFO:teuthology.orchestra.run.smithi007.stdout:Transaction check succeeded. 2024-09-18T07:55:54.927 INFO:teuthology.orchestra.run.smithi007.stdout:Running transaction test 2024-09-18T07:55:54.948 INFO:teuthology.orchestra.run.smithi007.stdout:Transaction test succeeded. 2024-09-18T07:55:54.949 INFO:teuthology.orchestra.run.smithi007.stdout:Running transaction 2024-09-18T07:55:55.043 INFO:teuthology.orchestra.run.smithi007.stdout: Preparing : 1/1 2024-09-18T07:55:55.164 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : libcephfs-devel-2:19.3.0-4983.ge7fb7b56.el9.x86_64 1/1 2024-09-18T07:55:55.636 INFO:teuthology.orchestra.run.smithi007.stdout: Running scriptlet: libcephfs-devel-2:19.3.0-4983.ge7fb7b56.el9.x86_64 1/1 2024-09-18T07:55:55.877 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : libcephfs-devel-2:19.3.0-4983.ge7fb7b56.el9.x86_64 1/1 2024-09-18T07:55:55.877 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-09-18T07:55:55.877 INFO:teuthology.orchestra.run.smithi007.stdout:Installed: 2024-09-18T07:55:55.877 INFO:teuthology.orchestra.run.smithi007.stdout: libcephfs-devel-2:19.3.0-4983.ge7fb7b56.el9.x86_64 2024-09-18T07:55:55.878 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-09-18T07:55:55.878 INFO:teuthology.orchestra.run.smithi007.stdout:Complete! 2024-09-18T07:55:56.096 DEBUG:teuthology.orchestra.run.smithi007:> sudo yum -y install librados2 2024-09-18T07:55:56.697 INFO:teuthology.orchestra.run.smithi007.stdout:Last metadata expiration check: 0:01:44 ago on Wed 18 Sep 2024 07:54:12 AM UTC. 2024-09-18T07:55:56.790 INFO:teuthology.orchestra.run.smithi007.stdout:Package librados2-2:19.3.0-4983.ge7fb7b56.el9.x86_64 is already installed. 2024-09-18T07:55:56.836 INFO:teuthology.orchestra.run.smithi007.stdout:Dependencies resolved. 2024-09-18T07:55:56.838 INFO:teuthology.orchestra.run.smithi007.stdout:Nothing to do. 2024-09-18T07:55:56.838 INFO:teuthology.orchestra.run.smithi007.stdout:Complete! 2024-09-18T07:55:56.901 DEBUG:teuthology.orchestra.run.smithi007:> sudo yum -y install librbd1 2024-09-18T07:55:57.502 INFO:teuthology.orchestra.run.smithi007.stdout:Last metadata expiration check: 0:01:45 ago on Wed 18 Sep 2024 07:54:12 AM UTC. 2024-09-18T07:55:57.594 INFO:teuthology.orchestra.run.smithi007.stdout:Package librbd1-2:19.3.0-4983.ge7fb7b56.el9.x86_64 is already installed. 2024-09-18T07:55:57.641 INFO:teuthology.orchestra.run.smithi007.stdout:Dependencies resolved. 2024-09-18T07:55:57.642 INFO:teuthology.orchestra.run.smithi007.stdout:Nothing to do. 2024-09-18T07:55:57.642 INFO:teuthology.orchestra.run.smithi007.stdout:Complete! 2024-09-18T07:55:57.711 DEBUG:teuthology.orchestra.run.smithi007:> sudo yum -y install python3-rados 2024-09-18T07:55:58.314 INFO:teuthology.orchestra.run.smithi007.stdout:Last metadata expiration check: 0:01:46 ago on Wed 18 Sep 2024 07:54:12 AM UTC. 2024-09-18T07:55:58.407 INFO:teuthology.orchestra.run.smithi007.stdout:Package python3-rados-2:19.3.0-4983.ge7fb7b56.el9.x86_64 is already installed. 2024-09-18T07:55:58.455 INFO:teuthology.orchestra.run.smithi007.stdout:Dependencies resolved. 2024-09-18T07:55:58.456 INFO:teuthology.orchestra.run.smithi007.stdout:Nothing to do. 2024-09-18T07:55:58.456 INFO:teuthology.orchestra.run.smithi007.stdout:Complete! 2024-09-18T07:55:58.529 DEBUG:teuthology.orchestra.run.smithi007:> sudo yum -y install python3-rgw 2024-09-18T07:55:59.133 INFO:teuthology.orchestra.run.smithi007.stdout:Last metadata expiration check: 0:01:47 ago on Wed 18 Sep 2024 07:54:12 AM UTC. 2024-09-18T07:55:59.226 INFO:teuthology.orchestra.run.smithi007.stdout:Package python3-rgw-2:19.3.0-4983.ge7fb7b56.el9.x86_64 is already installed. 2024-09-18T07:55:59.272 INFO:teuthology.orchestra.run.smithi007.stdout:Dependencies resolved. 2024-09-18T07:55:59.273 INFO:teuthology.orchestra.run.smithi007.stdout:Nothing to do. 2024-09-18T07:55:59.273 INFO:teuthology.orchestra.run.smithi007.stdout:Complete! 2024-09-18T07:55:59.343 DEBUG:teuthology.orchestra.run.smithi007:> sudo yum -y install python3-cephfs 2024-09-18T07:55:59.948 INFO:teuthology.orchestra.run.smithi007.stdout:Last metadata expiration check: 0:01:47 ago on Wed 18 Sep 2024 07:54:12 AM UTC. 2024-09-18T07:56:00.042 INFO:teuthology.orchestra.run.smithi007.stdout:Package python3-cephfs-2:19.3.0-4983.ge7fb7b56.el9.x86_64 is already installed. 2024-09-18T07:56:00.088 INFO:teuthology.orchestra.run.smithi007.stdout:Dependencies resolved. 2024-09-18T07:56:00.089 INFO:teuthology.orchestra.run.smithi007.stdout:Nothing to do. 2024-09-18T07:56:00.090 INFO:teuthology.orchestra.run.smithi007.stdout:Complete! 2024-09-18T07:56:00.156 DEBUG:teuthology.orchestra.run.smithi007:> sudo yum -y install python3-rbd 2024-09-18T07:56:00.761 INFO:teuthology.orchestra.run.smithi007.stdout:Last metadata expiration check: 0:01:48 ago on Wed 18 Sep 2024 07:54:12 AM UTC. 2024-09-18T07:56:00.853 INFO:teuthology.orchestra.run.smithi007.stdout:Package python3-rbd-2:19.3.0-4983.ge7fb7b56.el9.x86_64 is already installed. 2024-09-18T07:56:00.899 INFO:teuthology.orchestra.run.smithi007.stdout:Dependencies resolved. 2024-09-18T07:56:00.901 INFO:teuthology.orchestra.run.smithi007.stdout:Nothing to do. 2024-09-18T07:56:00.901 INFO:teuthology.orchestra.run.smithi007.stdout:Complete! 2024-09-18T07:56:00.971 DEBUG:teuthology.orchestra.run.smithi007:> sudo yum -y install rbd-fuse 2024-09-18T07:56:01.583 INFO:teuthology.orchestra.run.smithi007.stdout:Last metadata expiration check: 0:01:49 ago on Wed 18 Sep 2024 07:54:12 AM UTC. 2024-09-18T07:56:01.723 INFO:teuthology.orchestra.run.smithi007.stdout:Dependencies resolved. 2024-09-18T07:56:01.724 INFO:teuthology.orchestra.run.smithi007.stdout:================================================================================ 2024-09-18T07:56:01.724 INFO:teuthology.orchestra.run.smithi007.stdout: Package Architecture Version Repository Size 2024-09-18T07:56:01.724 INFO:teuthology.orchestra.run.smithi007.stdout:================================================================================ 2024-09-18T07:56:01.724 INFO:teuthology.orchestra.run.smithi007.stdout:Installing: 2024-09-18T07:56:01.724 INFO:teuthology.orchestra.run.smithi007.stdout: rbd-fuse x86_64 2:19.3.0-4983.ge7fb7b56.el9 ceph 85 k 2024-09-18T07:56:01.724 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-09-18T07:56:01.724 INFO:teuthology.orchestra.run.smithi007.stdout:Transaction Summary 2024-09-18T07:56:01.724 INFO:teuthology.orchestra.run.smithi007.stdout:================================================================================ 2024-09-18T07:56:01.724 INFO:teuthology.orchestra.run.smithi007.stdout:Install 1 Package 2024-09-18T07:56:01.724 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-09-18T07:56:01.724 INFO:teuthology.orchestra.run.smithi007.stdout:Total download size: 85 k 2024-09-18T07:56:01.725 INFO:teuthology.orchestra.run.smithi007.stdout:Installed size: 231 k 2024-09-18T07:56:01.725 INFO:teuthology.orchestra.run.smithi007.stdout:Downloading Packages: 2024-09-18T07:56:01.926 INFO:teuthology.orchestra.run.smithi007.stdout:rbd-fuse-19.3.0-4983.ge7fb7b56.el9.x86_64.rpm 424 kB/s | 85 kB 00:00 2024-09-18T07:56:01.926 INFO:teuthology.orchestra.run.smithi007.stdout:-------------------------------------------------------------------------------- 2024-09-18T07:56:01.926 INFO:teuthology.orchestra.run.smithi007.stdout:Total 421 kB/s | 85 kB 00:00 2024-09-18T07:56:01.927 INFO:teuthology.orchestra.run.smithi007.stdout:Running transaction check 2024-09-18T07:56:01.935 INFO:teuthology.orchestra.run.smithi007.stdout:Transaction check succeeded. 2024-09-18T07:56:01.935 INFO:teuthology.orchestra.run.smithi007.stdout:Running transaction test 2024-09-18T07:56:02.007 INFO:teuthology.orchestra.run.smithi007.stdout:Transaction test succeeded. 2024-09-18T07:56:02.007 INFO:teuthology.orchestra.run.smithi007.stdout:Running transaction 2024-09-18T07:56:02.167 INFO:teuthology.orchestra.run.smithi007.stdout: Preparing : 1/1 2024-09-18T07:56:02.276 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : rbd-fuse-2:19.3.0-4983.ge7fb7b56.el9.x86_64 1/1 2024-09-18T07:56:02.738 INFO:teuthology.orchestra.run.smithi007.stdout: Running scriptlet: rbd-fuse-2:19.3.0-4983.ge7fb7b56.el9.x86_64 1/1 2024-09-18T07:56:03.010 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : rbd-fuse-2:19.3.0-4983.ge7fb7b56.el9.x86_64 1/1 2024-09-18T07:56:03.010 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-09-18T07:56:03.010 INFO:teuthology.orchestra.run.smithi007.stdout:Installed: 2024-09-18T07:56:03.010 INFO:teuthology.orchestra.run.smithi007.stdout: rbd-fuse-2:19.3.0-4983.ge7fb7b56.el9.x86_64 2024-09-18T07:56:03.010 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-09-18T07:56:03.010 INFO:teuthology.orchestra.run.smithi007.stdout:Complete! 2024-09-18T07:56:03.260 DEBUG:teuthology.orchestra.run.smithi007:> sudo yum -y install rbd-mirror 2024-09-18T07:56:03.871 INFO:teuthology.orchestra.run.smithi007.stdout:Last metadata expiration check: 0:01:51 ago on Wed 18 Sep 2024 07:54:12 AM UTC. 2024-09-18T07:56:04.010 INFO:teuthology.orchestra.run.smithi007.stdout:Dependencies resolved. 2024-09-18T07:56:04.011 INFO:teuthology.orchestra.run.smithi007.stdout:================================================================================ 2024-09-18T07:56:04.011 INFO:teuthology.orchestra.run.smithi007.stdout: Package Arch Version Repo Size 2024-09-18T07:56:04.011 INFO:teuthology.orchestra.run.smithi007.stdout:================================================================================ 2024-09-18T07:56:04.011 INFO:teuthology.orchestra.run.smithi007.stdout:Installing: 2024-09-18T07:56:04.011 INFO:teuthology.orchestra.run.smithi007.stdout: rbd-mirror x86_64 2:19.3.0-4983.ge7fb7b56.el9 ceph 3.2 M 2024-09-18T07:56:04.011 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-09-18T07:56:04.011 INFO:teuthology.orchestra.run.smithi007.stdout:Transaction Summary 2024-09-18T07:56:04.011 INFO:teuthology.orchestra.run.smithi007.stdout:================================================================================ 2024-09-18T07:56:04.012 INFO:teuthology.orchestra.run.smithi007.stdout:Install 1 Package 2024-09-18T07:56:04.012 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-09-18T07:56:04.012 INFO:teuthology.orchestra.run.smithi007.stdout:Total download size: 3.2 M 2024-09-18T07:56:04.012 INFO:teuthology.orchestra.run.smithi007.stdout:Installed size: 14 M 2024-09-18T07:56:04.012 INFO:teuthology.orchestra.run.smithi007.stdout:Downloading Packages: 2024-09-18T07:56:04.356 INFO:teuthology.orchestra.run.smithi007.stdout:rbd-mirror-19.3.0-4983.ge7fb7b56.el9.x86_64.rpm 9.3 MB/s | 3.2 MB 00:00 2024-09-18T07:56:04.356 INFO:teuthology.orchestra.run.smithi007.stdout:-------------------------------------------------------------------------------- 2024-09-18T07:56:04.357 INFO:teuthology.orchestra.run.smithi007.stdout:Total 9.2 MB/s | 3.2 MB 00:00 2024-09-18T07:56:04.357 INFO:teuthology.orchestra.run.smithi007.stdout:Running transaction check 2024-09-18T07:56:04.367 INFO:teuthology.orchestra.run.smithi007.stdout:Transaction check succeeded. 2024-09-18T07:56:04.367 INFO:teuthology.orchestra.run.smithi007.stdout:Running transaction test 2024-09-18T07:56:04.453 INFO:teuthology.orchestra.run.smithi007.stdout:Transaction test succeeded. 2024-09-18T07:56:04.453 INFO:teuthology.orchestra.run.smithi007.stdout:Running transaction 2024-09-18T07:56:04.974 INFO:teuthology.orchestra.run.smithi007.stdout: Preparing : 1/1 2024-09-18T07:56:05.040 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : rbd-mirror-2:19.3.0-4983.ge7fb7b56.el9.x86_64 1/1 2024-09-18T07:56:05.067 INFO:teuthology.orchestra.run.smithi007.stdout: Running scriptlet: rbd-mirror-2:19.3.0-4983.ge7fb7b56.el9.x86_64 1/1 2024-09-18T07:56:05.067 INFO:teuthology.orchestra.run.smithi007.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-18T07:56:05.067 INFO:teuthology.orchestra.run.smithi007.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-09-18T07:56:05.067 INFO:teuthology.orchestra.run.smithi007.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-09-18T07:56:05.067 INFO:teuthology.orchestra.run.smithi007.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-09-18T07:56:05.067 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-09-18T07:56:06.240 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : rbd-mirror-2:19.3.0-4983.ge7fb7b56.el9.x86_64 1/1 2024-09-18T07:56:06.240 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-09-18T07:56:06.240 INFO:teuthology.orchestra.run.smithi007.stdout:Installed: 2024-09-18T07:56:06.240 INFO:teuthology.orchestra.run.smithi007.stdout: rbd-mirror-2:19.3.0-4983.ge7fb7b56.el9.x86_64 2024-09-18T07:56:06.241 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-09-18T07:56:06.241 INFO:teuthology.orchestra.run.smithi007.stdout:Complete! 2024-09-18T07:56:06.458 DEBUG:teuthology.orchestra.run.smithi007:> sudo yum -y install rbd-nbd 2024-09-18T07:56:07.074 INFO:teuthology.orchestra.run.smithi007.stdout:Last metadata expiration check: 0:01:55 ago on Wed 18 Sep 2024 07:54:12 AM UTC. 2024-09-18T07:56:07.212 INFO:teuthology.orchestra.run.smithi007.stdout:Dependencies resolved. 2024-09-18T07:56:07.213 INFO:teuthology.orchestra.run.smithi007.stdout:================================================================================ 2024-09-18T07:56:07.213 INFO:teuthology.orchestra.run.smithi007.stdout: Package Architecture Version Repository Size 2024-09-18T07:56:07.213 INFO:teuthology.orchestra.run.smithi007.stdout:================================================================================ 2024-09-18T07:56:07.213 INFO:teuthology.orchestra.run.smithi007.stdout:Installing: 2024-09-18T07:56:07.213 INFO:teuthology.orchestra.run.smithi007.stdout: rbd-nbd x86_64 2:19.3.0-4983.ge7fb7b56.el9 ceph 170 k 2024-09-18T07:56:07.213 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-09-18T07:56:07.214 INFO:teuthology.orchestra.run.smithi007.stdout:Transaction Summary 2024-09-18T07:56:07.214 INFO:teuthology.orchestra.run.smithi007.stdout:================================================================================ 2024-09-18T07:56:07.214 INFO:teuthology.orchestra.run.smithi007.stdout:Install 1 Package 2024-09-18T07:56:07.214 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-09-18T07:56:07.214 INFO:teuthology.orchestra.run.smithi007.stdout:Total download size: 170 k 2024-09-18T07:56:07.214 INFO:teuthology.orchestra.run.smithi007.stdout:Installed size: 490 k 2024-09-18T07:56:07.214 INFO:teuthology.orchestra.run.smithi007.stdout:Downloading Packages: 2024-09-18T07:56:07.426 INFO:teuthology.orchestra.run.smithi007.stdout:rbd-nbd-19.3.0-4983.ge7fb7b56.el9.x86_64.rpm 804 kB/s | 170 kB 00:00 2024-09-18T07:56:07.426 INFO:teuthology.orchestra.run.smithi007.stdout:-------------------------------------------------------------------------------- 2024-09-18T07:56:07.427 INFO:teuthology.orchestra.run.smithi007.stdout:Total 798 kB/s | 170 kB 00:00 2024-09-18T07:56:07.427 INFO:teuthology.orchestra.run.smithi007.stdout:Running transaction check 2024-09-18T07:56:07.435 INFO:teuthology.orchestra.run.smithi007.stdout:Transaction check succeeded. 2024-09-18T07:56:07.435 INFO:teuthology.orchestra.run.smithi007.stdout:Running transaction test 2024-09-18T07:56:07.503 INFO:teuthology.orchestra.run.smithi007.stdout:Transaction test succeeded. 2024-09-18T07:56:07.503 INFO:teuthology.orchestra.run.smithi007.stdout:Running transaction 2024-09-18T07:56:07.681 INFO:teuthology.orchestra.run.smithi007.stdout: Preparing : 1/1 2024-09-18T07:56:07.786 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : rbd-nbd-2:19.3.0-4983.ge7fb7b56.el9.x86_64 1/1 2024-09-18T07:56:08.246 INFO:teuthology.orchestra.run.smithi007.stdout: Running scriptlet: rbd-nbd-2:19.3.0-4983.ge7fb7b56.el9.x86_64 1/1 2024-09-18T07:56:08.519 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : rbd-nbd-2:19.3.0-4983.ge7fb7b56.el9.x86_64 1/1 2024-09-18T07:56:08.519 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-09-18T07:56:08.519 INFO:teuthology.orchestra.run.smithi007.stdout:Installed: 2024-09-18T07:56:08.519 INFO:teuthology.orchestra.run.smithi007.stdout: rbd-nbd-2:19.3.0-4983.ge7fb7b56.el9.x86_64 2024-09-18T07:56:08.519 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-09-18T07:56:08.519 INFO:teuthology.orchestra.run.smithi007.stdout:Complete! 2024-09-18T07:56:08.789 DEBUG:teuthology.parallel:result is None 2024-09-18T07:56:08.790 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e 2024-09-18T07:56:08.935 DEBUG:teuthology.orchestra.run.smithi007:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-09-18T07:56:08.958 INFO:teuthology.orchestra.run.smithi007.stdout:19.3.0-4983.ge7fb7b56.el9 2024-09-18T07:56:08.959 INFO:teuthology.packaging:The installed version of ceph is 19.3.0-4983.ge7fb7b56.el9 2024-09-18T07:56:08.959 INFO:teuthology.task.install:The correct ceph version 19.3.0-4983.ge7fb7b56 is installed. 2024-09-18T07:56:08.961 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e 2024-09-18T07:56:09.101 DEBUG:teuthology.orchestra.run.smithi050:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-09-18T07:56:09.129 INFO:teuthology.orchestra.run.smithi050.stdout:19.3.0-4983.ge7fb7b56.el9 2024-09-18T07:56:09.129 INFO:teuthology.packaging:The installed version of ceph is 19.3.0-4983.ge7fb7b56.el9 2024-09-18T07:56:09.129 INFO:teuthology.task.install:The correct ceph version 19.3.0-4983.ge7fb7b56 is installed. 2024-09-18T07:56:09.132 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2024-09-18T07:56:09.132 DEBUG:teuthology.orchestra.run.smithi007:> set -ex 2024-09-18T07:56:09.132 DEBUG:teuthology.orchestra.run.smithi007:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-09-18T07:56:09.160 DEBUG:teuthology.orchestra.run.smithi050:> set -ex 2024-09-18T07:56:09.160 DEBUG:teuthology.orchestra.run.smithi050:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-09-18T07:56:09.201 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2024-09-18T07:56:09.201 DEBUG:teuthology.orchestra.run.smithi007:> set -ex 2024-09-18T07:56:09.201 DEBUG:teuthology.orchestra.run.smithi007:> sudo dd of=/usr/bin/daemon-helper 2024-09-18T07:56:09.227 DEBUG:teuthology.orchestra.run.smithi007:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-09-18T07:56:09.293 DEBUG:teuthology.orchestra.run.smithi050:> set -ex 2024-09-18T07:56:09.293 DEBUG:teuthology.orchestra.run.smithi050:> sudo dd of=/usr/bin/daemon-helper 2024-09-18T07:56:09.321 DEBUG:teuthology.orchestra.run.smithi050:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-09-18T07:56:09.387 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2024-09-18T07:56:09.388 DEBUG:teuthology.orchestra.run.smithi007:> set -ex 2024-09-18T07:56:09.388 DEBUG:teuthology.orchestra.run.smithi007:> sudo dd of=/usr/bin/adjust-ulimits 2024-09-18T07:56:09.413 DEBUG:teuthology.orchestra.run.smithi007:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-09-18T07:56:09.479 DEBUG:teuthology.orchestra.run.smithi050:> set -ex 2024-09-18T07:56:09.479 DEBUG:teuthology.orchestra.run.smithi050:> sudo dd of=/usr/bin/adjust-ulimits 2024-09-18T07:56:09.505 DEBUG:teuthology.orchestra.run.smithi050:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-09-18T07:56:09.572 INFO:teuthology.task.install.util:Shipping 'stdin-killer'... 2024-09-18T07:56:09.572 DEBUG:teuthology.orchestra.run.smithi007:> set -ex 2024-09-18T07:56:09.573 DEBUG:teuthology.orchestra.run.smithi007:> sudo dd of=/usr/bin/stdin-killer 2024-09-18T07:56:09.598 DEBUG:teuthology.orchestra.run.smithi007:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-09-18T07:56:09.664 DEBUG:teuthology.orchestra.run.smithi050:> set -ex 2024-09-18T07:56:09.664 DEBUG:teuthology.orchestra.run.smithi050:> sudo dd of=/usr/bin/stdin-killer 2024-09-18T07:56:09.690 DEBUG:teuthology.orchestra.run.smithi050:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-09-18T07:56:09.756 INFO:teuthology.run_tasks:Running task cephadm... 2024-09-18T07:56:09.855 INFO:tasks.cephadm:Config: {'conf': {'mgr': {'debug mgr': 20, 'debug ms': 1}, 'client.0': {'debug ms': 1, 'debug objecter': 20, 'debug rados': 20}, 'global': {'ms bind msgr1': False, 'ms bind msgr2': True, 'ms type': 'async'}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon min osdmap epochs': 50, 'mon osdmap full prune interval': 2, 'mon osdmap full prune min': 15, 'mon osdmap full prune txsize': 2, 'paxos service trim min': 10}, 'osd': {'debug ms': 1, 'debug osd': 20, 'osd debug reject backfill probability': 0.3, 'osd delete sleep': 1, 'osd max backfills': 3, 'osd scrub max interval': 120, 'osd scrub min interval': 60, 'osd shutdown pgref assert': True, 'osd snap trim sleep': 2}}, 'flavor': 'default', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', 'CEPHADM_STRAY_DAEMON', 'CEPHADM_FAILED_DAEMON', 'but it is still running', 'objects unfound and apparently lost', '\\(POOL_APP_NOT_ENABLED\\)'], 'log-only-match': ['CEPHADM_'], 'sha1': 'e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e', 'cephadm_mode': 'root'} 2024-09-18T07:56:09.855 INFO:tasks.cephadm:Cluster image is quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e 2024-09-18T07:56:09.855 INFO:tasks.cephadm:Cluster fsid is 779349ca-7593-11ef-bceb-c7b262605968 2024-09-18T07:56:09.856 INFO:tasks.cephadm:Choosing monitor IPs and ports... 2024-09-18T07:56:09.856 INFO:tasks.cephadm:Monitor IPs: {'mon.a': '172.21.15.7', 'mon.c': '[v2:172.21.15.7:3301,v1:172.21.15.7:6790]', 'mon.b': '172.21.15.50'} 2024-09-18T07:56:09.856 INFO:tasks.cephadm:First mon is mon.a on smithi007 2024-09-18T07:56:09.856 INFO:tasks.cephadm:First mgr is y 2024-09-18T07:56:09.856 INFO:tasks.cephadm:Normalizing hostnames... 2024-09-18T07:56:09.856 DEBUG:teuthology.orchestra.run.smithi007:> sudo hostname $(hostname -s) 2024-09-18T07:56:09.882 DEBUG:teuthology.orchestra.run.smithi050:> sudo hostname $(hostname -s) 2024-09-18T07:56:09.908 INFO:tasks.cephadm:Downloading "compiled" cephadm from cachra 2024-09-18T07:56:09.909 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e 2024-09-18T07:56:10.050 INFO:tasks.cephadm:builder_project result: [{'status': 'ready', 'sha1': 'e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e', 'extra': {'build_url': 'https://jenkins.ceph.com/job/ceph-dev-new-build/ARCH=x86_64,AVAILABLE_ARCH=x86_64,AVAILABLE_DIST=centos9,DIST=centos9,MACHINE_SIZE=gigantic/83115/', 'root_build_cause': 'SCMTRIGGER', 'version': '19.3.0-4983-ge7fb7b56', 'node_name': '172.21.2.6+braggi06', 'job_name': 'ceph-dev-new-build/ARCH=x86_64,AVAILABLE_ARCH=x86_64,AVAILABLE_DIST=centos9,DIST=centos9,MACHINE_SIZE=gigantic', 'package_manager_version': '19.3.0-4983.ge7fb7b56'}, 'url': 'https://4.chacra.ceph.com/r/ceph/wip-guits-main-2024-09-17-1213/e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e/centos/9/flavors/default/', 'distro_codename': None, 'modified': '2024-09-17 17:21:57.325397', 'distro_version': '9', 'project': 'ceph', 'flavor': 'default', 'ref': 'wip-guits-main-2024-09-17-1213', 'chacra_url': 'https://4.chacra.ceph.com/repos/ceph/wip-guits-main-2024-09-17-1213/e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e/centos/9/flavors/default/', 'archs': ['x86_64', 'source'], 'distro': 'centos'}] 2024-09-18T07:56:10.192 INFO:tasks.util.chacra:got chacra host 4.chacra.ceph.com, ref wip-guits-main-2024-09-17-1213, sha1 e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e from https://shaman.ceph.com/api/search/?project=ceph&distros=centos%2F9%2Fx86_64&flavor=default&sha1=e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e 2024-09-18T07:56:10.194 INFO:tasks.cephadm:Discovered cachra url: https://4.chacra.ceph.com/binaries/ceph/wip-guits-main-2024-09-17-1213/e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e/centos/9/x86_64/flavors/default/cephadm 2024-09-18T07:56:10.194 INFO:tasks.cephadm:Downloading cephadm from url: https://4.chacra.ceph.com/binaries/ceph/wip-guits-main-2024-09-17-1213/e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e/centos/9/x86_64/flavors/default/cephadm 2024-09-18T07:56:10.194 DEBUG:teuthology.orchestra.run.smithi007:> curl --silent -L https://4.chacra.ceph.com/binaries/ceph/wip-guits-main-2024-09-17-1213/e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e/centos/9/x86_64/flavors/default/cephadm > /home/ubuntu/cephtest/cephadm && ls -l /home/ubuntu/cephtest/cephadm 2024-09-18T07:56:10.714 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r--. 1 ubuntu ubuntu 804135 Sep 18 07:56 /home/ubuntu/cephtest/cephadm 2024-09-18T07:56:10.715 DEBUG:teuthology.orchestra.run.smithi050:> curl --silent -L https://4.chacra.ceph.com/binaries/ceph/wip-guits-main-2024-09-17-1213/e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e/centos/9/x86_64/flavors/default/cephadm > /home/ubuntu/cephtest/cephadm && ls -l /home/ubuntu/cephtest/cephadm 2024-09-18T07:56:11.230 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r--. 1 ubuntu ubuntu 804135 Sep 18 07:56 /home/ubuntu/cephtest/cephadm 2024-09-18T07:56:11.230 DEBUG:teuthology.orchestra.run.smithi007:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2024-09-18T07:56:11.251 DEBUG:teuthology.orchestra.run.smithi050:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2024-09-18T07:56:11.281 INFO:tasks.cephadm:Pulling image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e on all hosts... 2024-09-18T07:56:11.282 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e pull 2024-09-18T07:56:11.294 DEBUG:teuthology.orchestra.run.smithi050:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e pull 2024-09-18T07:56:12.868 INFO:teuthology.orchestra.run.smithi050.stderr:Pulling container image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e... 2024-09-18T07:56:13.065 INFO:teuthology.orchestra.run.smithi007.stderr:Pulling container image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e... 2024-09-18T07:56:43.069 INFO:teuthology.orchestra.run.smithi007.stdout:{ 2024-09-18T07:56:43.069 INFO:teuthology.orchestra.run.smithi007.stdout: "ceph_version": "ceph version 19.3.0-4983-ge7fb7b56 (e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e) squid (dev)", 2024-09-18T07:56:43.069 INFO:teuthology.orchestra.run.smithi007.stdout: "image_id": "b22e1e37ddf1ca6e22136cd6540a9ab61e7cafb8bfbc7a91a2f20f7f9941bd9a", 2024-09-18T07:56:43.069 INFO:teuthology.orchestra.run.smithi007.stdout: "repo_digests": [ 2024-09-18T07:56:43.069 INFO:teuthology.orchestra.run.smithi007.stdout: "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:1f5b5a3734f665ef333266f39143ac3ef1eb1951b21258f2307e10332e2d7ac9" 2024-09-18T07:56:43.070 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-09-18T07:56:43.070 INFO:teuthology.orchestra.run.smithi007.stdout:} 2024-09-18T07:56:45.753 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2024-09-18T07:56:45.753 INFO:teuthology.orchestra.run.smithi050.stdout: "ceph_version": "ceph version 19.3.0-4983-ge7fb7b56 (e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e) squid (dev)", 2024-09-18T07:56:45.753 INFO:teuthology.orchestra.run.smithi050.stdout: "image_id": "b22e1e37ddf1ca6e22136cd6540a9ab61e7cafb8bfbc7a91a2f20f7f9941bd9a", 2024-09-18T07:56:45.753 INFO:teuthology.orchestra.run.smithi050.stdout: "repo_digests": [ 2024-09-18T07:56:45.753 INFO:teuthology.orchestra.run.smithi050.stdout: "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:1f5b5a3734f665ef333266f39143ac3ef1eb1951b21258f2307e10332e2d7ac9" 2024-09-18T07:56:45.753 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-09-18T07:56:45.753 INFO:teuthology.orchestra.run.smithi050.stdout:} 2024-09-18T07:56:45.778 DEBUG:teuthology.orchestra.run.smithi007:> sudo mkdir -p /etc/ceph 2024-09-18T07:56:45.812 DEBUG:teuthology.orchestra.run.smithi050:> sudo mkdir -p /etc/ceph 2024-09-18T07:56:45.841 DEBUG:teuthology.orchestra.run.smithi007:> sudo chmod 777 /etc/ceph 2024-09-18T07:56:45.881 DEBUG:teuthology.orchestra.run.smithi050:> sudo chmod 777 /etc/ceph 2024-09-18T07:56:45.907 INFO:tasks.cephadm:Writing seed config... 2024-09-18T07:56:45.908 INFO:tasks.cephadm: override: [mgr] debug mgr = 20 2024-09-18T07:56:45.908 INFO:tasks.cephadm: override: [mgr] debug ms = 1 2024-09-18T07:56:45.909 INFO:tasks.cephadm: override: [client.0] debug ms = 1 2024-09-18T07:56:45.909 INFO:tasks.cephadm: override: [client.0] debug objecter = 20 2024-09-18T07:56:45.909 INFO:tasks.cephadm: override: [client.0] debug rados = 20 2024-09-18T07:56:45.909 INFO:tasks.cephadm: override: [global] ms bind msgr1 = False 2024-09-18T07:56:45.909 INFO:tasks.cephadm: override: [global] ms bind msgr2 = True 2024-09-18T07:56:45.909 INFO:tasks.cephadm: override: [global] ms type = async 2024-09-18T07:56:45.909 INFO:tasks.cephadm: override: [mon] debug mon = 20 2024-09-18T07:56:45.909 INFO:tasks.cephadm: override: [mon] debug ms = 1 2024-09-18T07:56:45.909 INFO:tasks.cephadm: override: [mon] debug paxos = 20 2024-09-18T07:56:45.909 INFO:tasks.cephadm: override: [mon] mon min osdmap epochs = 50 2024-09-18T07:56:45.909 INFO:tasks.cephadm: override: [mon] mon osdmap full prune interval = 2 2024-09-18T07:56:45.909 INFO:tasks.cephadm: override: [mon] mon osdmap full prune min = 15 2024-09-18T07:56:45.909 INFO:tasks.cephadm: override: [mon] mon osdmap full prune txsize = 2 2024-09-18T07:56:45.910 INFO:tasks.cephadm: override: [mon] paxos service trim min = 10 2024-09-18T07:56:45.910 INFO:tasks.cephadm: override: [osd] debug ms = 1 2024-09-18T07:56:45.910 INFO:tasks.cephadm: override: [osd] debug osd = 20 2024-09-18T07:56:45.910 INFO:tasks.cephadm: override: [osd] osd debug reject backfill probability = 0.3 2024-09-18T07:56:45.910 INFO:tasks.cephadm: override: [osd] osd delete sleep = 1 2024-09-18T07:56:45.910 INFO:tasks.cephadm: override: [osd] osd max backfills = 3 2024-09-18T07:56:45.910 INFO:tasks.cephadm: override: [osd] osd scrub max interval = 120 2024-09-18T07:56:45.910 INFO:tasks.cephadm: override: [osd] osd scrub min interval = 60 2024-09-18T07:56:45.910 INFO:tasks.cephadm: override: [osd] osd shutdown pgref assert = True 2024-09-18T07:56:45.910 INFO:tasks.cephadm: override: [osd] osd snap trim sleep = 2 2024-09-18T07:56:45.911 DEBUG:teuthology.orchestra.run.smithi007:> set -ex 2024-09-18T07:56:45.911 DEBUG:teuthology.orchestra.run.smithi007:> dd of=/home/ubuntu/cephtest/seed.ceph.conf 2024-09-18T07:56:45.940 DEBUG:tasks.cephadm:Final config: [global] # make logging friendly to teuthology log_to_file = true log_to_stderr = false log to journald = false mon cluster log to file = true mon cluster log file level = debug mon clock drift allowed = 1.000 # replicate across OSDs, not hosts osd crush chooseleaf type = 0 #osd pool default size = 2 osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd # enable some debugging auth debug = true ms die on old message = true ms die on bug = true debug asserts on shutdown = true # adjust warnings mon max pg per osd = 10000# >= luminous mon pg warn max object skew = 0 mon osd allow primary affinity = true mon osd allow pg remap = true mon warn on legacy crush tunables = false mon warn on crush straw calc version zero = false mon warn on no sortbitwise = false mon warn on osd down out interval zero = false mon warn on too few osds = false mon_warn_on_pool_pg_num_not_power_of_two = false # disable pg_autoscaler by default for new pools osd_pool_default_pg_autoscale_mode = off # tests delete pools mon allow pool delete = true fsid = 779349ca-7593-11ef-bceb-c7b262605968 ms bind msgr1 = False ms bind msgr2 = True ms type = async [osd] osd scrub load threshold = 5.0 osd scrub max interval = 120 osd mclock profile = high_recovery_ops osd recover clone overlap = true osd recovery max chunk = 1048576 osd deep scrub update digest min age = 30 osd map max advance = 10 osd memory target autotune = true # debugging osd debug shutdown = true osd debug op order = true osd debug verify stray on activate = true osd debug pg log writeout = true osd debug verify cached snaps = true osd debug verify missing on start = true osd debug misdirected ops = true osd op queue = debug_random osd op queue cut off = debug_random osd shutdown pgref assert = True bdev debug aio = true osd sloppy crc = true debug ms = 1 debug osd = 20 osd debug reject backfill probability = 0.3 osd delete sleep = 1 osd max backfills = 3 osd scrub min interval = 60 osd snap trim sleep = 2 [mgr] mon reweight min pgs per osd = 4 mon reweight min bytes per osd = 10 mgr/telemetry/nag = false debug mgr = 20 debug ms = 1 [mon] mon data avail warn = 5 mon mgr mkfs grace = 240 mon reweight min pgs per osd = 4 mon osd reporter subtree level = osd mon osd prime pg temp = true mon reweight min bytes per osd = 10 # rotate auth tickets quickly to exercise renewal paths auth mon ticket ttl = 660# 11m auth service ticket ttl = 240# 4m # don't complain about global id reclaim mon_warn_on_insecure_global_id_reclaim = false mon_warn_on_insecure_global_id_reclaim_allowed = false debug mon = 20 debug ms = 1 debug paxos = 20 mon min osdmap epochs = 50 mon osdmap full prune interval = 2 mon osdmap full prune min = 15 mon osdmap full prune txsize = 2 paxos service trim min = 10 [client.rgw] rgw cache enabled = true rgw enable ops log = true rgw enable usage log = true [client.0] debug ms = 1 debug objecter = 20 debug rados = 20 2024-09-18T07:56:45.941 DEBUG:teuthology.orchestra.run.smithi007:mon.a> sudo journalctl -f -n 0 -u ceph-779349ca-7593-11ef-bceb-c7b262605968@mon.a.service 2024-09-18T07:56:45.983 DEBUG:teuthology.orchestra.run.smithi007:mgr.y> sudo journalctl -f -n 0 -u ceph-779349ca-7593-11ef-bceb-c7b262605968@mgr.y.service 2024-09-18T07:56:46.026 INFO:tasks.cephadm:Bootstrapping... 2024-09-18T07:56:46.027 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e -v bootstrap --fsid 779349ca-7593-11ef-bceb-c7b262605968 --config /home/ubuntu/cephtest/seed.ceph.conf --output-config /etc/ceph/ceph.conf --output-keyring /etc/ceph/ceph.client.admin.keyring --output-pub-ssh-key /home/ubuntu/cephtest/ceph.pub --mon-id a --mgr-id y --orphan-initial-daemons --skip-monitoring-stack --mon-ip 172.21.15.7 --skip-admin-label && sudo chmod +r /etc/ceph/ceph.client.admin.keyring 2024-09-18T07:56:46.242 INFO:teuthology.orchestra.run.smithi007.stdout:-------------------------------------------------------------------------------- 2024-09-18T07:56:46.242 INFO:teuthology.orchestra.run.smithi007.stdout:cephadm ['--image', 'quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e', '-v', 'bootstrap', '--fsid', '779349ca-7593-11ef-bceb-c7b262605968', '--config', '/home/ubuntu/cephtest/seed.ceph.conf', '--output-config', '/etc/ceph/ceph.conf', '--output-keyring', '/etc/ceph/ceph.client.admin.keyring', '--output-pub-ssh-key', '/home/ubuntu/cephtest/ceph.pub', '--mon-id', 'a', '--mgr-id', 'y', '--orphan-initial-daemons', '--skip-monitoring-stack', '--mon-ip', '172.21.15.7', '--skip-admin-label'] 2024-09-18T07:56:46.242 INFO:teuthology.orchestra.run.smithi007.stderr:Specifying an fsid for your cluster offers no advantages and may increase the likelihood of fsid conflicts. 2024-09-18T07:56:46.242 INFO:teuthology.orchestra.run.smithi007.stdout:Verifying podman|docker is present... 2024-09-18T07:56:46.275 INFO:teuthology.orchestra.run.smithi007.stdout:/bin/podman: stdout 5.2.2 2024-09-18T07:56:46.275 INFO:teuthology.orchestra.run.smithi007.stdout:Verifying lvm2 is present... 2024-09-18T07:56:46.275 INFO:teuthology.orchestra.run.smithi007.stdout:Verifying time synchronization is in place... 2024-09-18T07:56:46.283 INFO:teuthology.orchestra.run.smithi007.stdout:Non-zero exit code 1 from systemctl is-enabled chrony.service 2024-09-18T07:56:46.283 INFO:teuthology.orchestra.run.smithi007.stdout:systemctl: stderr Failed to get unit file state for chrony.service: No such file or directory 2024-09-18T07:56:46.289 INFO:teuthology.orchestra.run.smithi007.stdout:Non-zero exit code 3 from systemctl is-active chrony.service 2024-09-18T07:56:46.290 INFO:teuthology.orchestra.run.smithi007.stdout:systemctl: stdout inactive 2024-09-18T07:56:46.296 INFO:teuthology.orchestra.run.smithi007.stdout:systemctl: stdout enabled 2024-09-18T07:56:46.303 INFO:teuthology.orchestra.run.smithi007.stdout:systemctl: stdout active 2024-09-18T07:56:46.303 INFO:teuthology.orchestra.run.smithi007.stdout:Unit chronyd.service is enabled and running 2024-09-18T07:56:46.303 INFO:teuthology.orchestra.run.smithi007.stdout:Repeating the final host check... 2024-09-18T07:56:46.345 INFO:teuthology.orchestra.run.smithi007.stdout:/bin/podman: stdout 5.2.2 2024-09-18T07:56:46.345 INFO:teuthology.orchestra.run.smithi007.stdout:podman (/bin/podman) version 5.2.2 is present 2024-09-18T07:56:46.345 INFO:teuthology.orchestra.run.smithi007.stdout:systemctl is present 2024-09-18T07:56:46.346 INFO:teuthology.orchestra.run.smithi007.stdout:lvcreate is present 2024-09-18T07:56:46.352 INFO:teuthology.orchestra.run.smithi007.stdout:Non-zero exit code 1 from systemctl is-enabled chrony.service 2024-09-18T07:56:46.352 INFO:teuthology.orchestra.run.smithi007.stdout:systemctl: stderr Failed to get unit file state for chrony.service: No such file or directory 2024-09-18T07:56:46.358 INFO:teuthology.orchestra.run.smithi007.stdout:Non-zero exit code 3 from systemctl is-active chrony.service 2024-09-18T07:56:46.358 INFO:teuthology.orchestra.run.smithi007.stdout:systemctl: stdout inactive 2024-09-18T07:56:46.365 INFO:teuthology.orchestra.run.smithi007.stdout:systemctl: stdout enabled 2024-09-18T07:56:46.372 INFO:teuthology.orchestra.run.smithi007.stdout:systemctl: stdout active 2024-09-18T07:56:46.372 INFO:teuthology.orchestra.run.smithi007.stdout:Unit chronyd.service is enabled and running 2024-09-18T07:56:46.372 INFO:teuthology.orchestra.run.smithi007.stdout:Host looks OK 2024-09-18T07:56:46.372 INFO:teuthology.orchestra.run.smithi007.stdout:Cluster fsid: 779349ca-7593-11ef-bceb-c7b262605968 2024-09-18T07:56:46.372 INFO:teuthology.orchestra.run.smithi007.stdout:Acquiring lock 140492381773440 on /run/cephadm/779349ca-7593-11ef-bceb-c7b262605968.lock 2024-09-18T07:56:46.372 INFO:teuthology.orchestra.run.smithi007.stdout:Lock 140492381773440 acquired on /run/cephadm/779349ca-7593-11ef-bceb-c7b262605968.lock 2024-09-18T07:56:46.373 INFO:teuthology.orchestra.run.smithi007.stdout:Verifying IP 172.21.15.7 port 3300 ... 2024-09-18T07:56:46.373 INFO:teuthology.orchestra.run.smithi007.stdout:Verifying IP 172.21.15.7 port 6789 ... 2024-09-18T07:56:46.373 INFO:teuthology.orchestra.run.smithi007.stdout:Base mon IP(s) is [172.21.15.7:3300, 172.21.15.7:6789], mon addrv is [v2:172.21.15.7:3300,v1:172.21.15.7:6789] 2024-09-18T07:56:46.376 INFO:teuthology.orchestra.run.smithi007.stdout:/sbin/ip: stdout default via 172.21.15.254 dev ens1f0 proto dhcp src 172.21.15.7 metric 100 2024-09-18T07:56:46.376 INFO:teuthology.orchestra.run.smithi007.stdout:/sbin/ip: stdout 172.21.0.0/20 dev ens1f0 proto kernel scope link src 172.21.15.7 metric 100 2024-09-18T07:56:46.379 INFO:teuthology.orchestra.run.smithi007.stdout:/sbin/ip: stdout ::1 dev lo proto kernel metric 256 pref medium 2024-09-18T07:56:46.379 INFO:teuthology.orchestra.run.smithi007.stdout:/sbin/ip: stdout fe80::/64 dev ens1f0 proto kernel metric 256 pref medium 2024-09-18T07:56:46.379 INFO:teuthology.orchestra.run.smithi007.stdout:/sbin/ip: stdout default via fe80::327c:5e00:6487:71e0 dev ens1f0 proto ra metric 1024 expires 1797sec hoplimit 64 pref medium 2024-09-18T07:56:46.381 INFO:teuthology.orchestra.run.smithi007.stdout:/sbin/ip: stdout 1: lo: mtu 65536 state UNKNOWN qlen 1000 2024-09-18T07:56:46.381 INFO:teuthology.orchestra.run.smithi007.stdout:/sbin/ip: stdout inet6 ::1/128 scope host 2024-09-18T07:56:46.381 INFO:teuthology.orchestra.run.smithi007.stdout:/sbin/ip: stdout valid_lft forever preferred_lft forever 2024-09-18T07:56:46.381 INFO:teuthology.orchestra.run.smithi007.stdout:/sbin/ip: stdout 4: ens1f0: mtu 1500 state UP qlen 1000 2024-09-18T07:56:46.382 INFO:teuthology.orchestra.run.smithi007.stdout:/sbin/ip: stdout inet6 fe80::ec4:7aff:febd:15ea/64 scope link 2024-09-18T07:56:46.382 INFO:teuthology.orchestra.run.smithi007.stdout:/sbin/ip: stdout valid_lft forever preferred_lft forever 2024-09-18T07:56:46.382 INFO:teuthology.orchestra.run.smithi007.stdout:Mon IP `172.21.15.7` is in CIDR network `172.21.0.0/20` 2024-09-18T07:56:46.382 INFO:teuthology.orchestra.run.smithi007.stdout:Mon IP `172.21.15.7` is in CIDR network `172.21.0.0/20` 2024-09-18T07:56:46.382 INFO:teuthology.orchestra.run.smithi007.stdout:Inferred mon public CIDR from local network configuration ['172.21.0.0/20', '172.21.0.0/20'] 2024-09-18T07:56:46.383 INFO:teuthology.orchestra.run.smithi007.stdout:Internal network (--cluster-network) has not been provided, OSD replication will default to the public_network 2024-09-18T07:56:46.383 INFO:teuthology.orchestra.run.smithi007.stdout:Pulling container image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e... 2024-09-18T07:56:46.811 INFO:teuthology.orchestra.run.smithi007.stdout:/bin/podman: stdout b22e1e37ddf1ca6e22136cd6540a9ab61e7cafb8bfbc7a91a2f20f7f9941bd9a 2024-09-18T07:56:46.811 INFO:teuthology.orchestra.run.smithi007.stdout:/bin/podman: stderr Trying to pull quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e... 2024-09-18T07:56:46.812 INFO:teuthology.orchestra.run.smithi007.stdout:/bin/podman: stderr Getting image source signatures 2024-09-18T07:56:46.812 INFO:teuthology.orchestra.run.smithi007.stdout:/bin/podman: stderr Copying blob sha256:dd5fdbcdb37f4ac53174664d5bf8ec5e6874a4a56dc34dfea3eb7684bf1294c4 2024-09-18T07:56:46.812 INFO:teuthology.orchestra.run.smithi007.stdout:/bin/podman: stderr Copying blob sha256:e8b54c863393b7a8216a71737771b73b16a8e43ec7acf927c7faa175c255e551 2024-09-18T07:56:46.812 INFO:teuthology.orchestra.run.smithi007.stdout:/bin/podman: stderr Copying config sha256:b22e1e37ddf1ca6e22136cd6540a9ab61e7cafb8bfbc7a91a2f20f7f9941bd9a 2024-09-18T07:56:46.812 INFO:teuthology.orchestra.run.smithi007.stdout:/bin/podman: stderr Writing manifest to image destination 2024-09-18T07:56:47.750 INFO:teuthology.orchestra.run.smithi007.stdout:ceph: stdout ceph version 19.3.0-4983-ge7fb7b56 (e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e) squid (dev) 2024-09-18T07:56:47.750 INFO:teuthology.orchestra.run.smithi007.stdout:Ceph version: ceph version 19.3.0-4983-ge7fb7b56 (e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e) squid (dev) 2024-09-18T07:56:47.750 INFO:teuthology.orchestra.run.smithi007.stdout:Extracting ceph user uid/gid from container image... 2024-09-18T07:56:48.731 INFO:teuthology.orchestra.run.smithi007.stdout:stat: stdout 167 167 2024-09-18T07:56:48.732 INFO:teuthology.orchestra.run.smithi007.stdout:Creating initial keys... 2024-09-18T07:56:55.889 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph-authtool: stdout AQDBh+pmctfNExAAS8jnjWcjziAV9yMTWCj9RA== 2024-09-18T07:57:03.974 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph-authtool: stdout AQDKh+pmP4CXKhAAJ93bliUHSx/z3ABqRmXcHw== 2024-09-18T07:57:04.979 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph-authtool: stdout AQDQh+pmwydUFBAAVx56hPA+635RjA+zIrIZSQ== 2024-09-18T07:57:04.980 INFO:teuthology.orchestra.run.smithi007.stdout:Creating initial monmap... 2024-09-18T07:57:05.912 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/monmaptool: stdout /usr/bin/monmaptool: monmap file /tmp/monmap 2024-09-18T07:57:05.912 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/monmaptool: stdout setting min_mon_release = quincy 2024-09-18T07:57:05.912 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/monmaptool: stdout /usr/bin/monmaptool: set fsid to 779349ca-7593-11ef-bceb-c7b262605968 2024-09-18T07:57:05.912 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/monmaptool: stdout /usr/bin/monmaptool: writing epoch 0 to /tmp/monmap (1 monitors) 2024-09-18T07:57:05.912 INFO:teuthology.orchestra.run.smithi007.stdout:monmaptool for a [v2:172.21.15.7:3300,v1:172.21.15.7:6789] on /usr/bin/monmaptool: monmap file /tmp/monmap 2024-09-18T07:57:05.912 INFO:teuthology.orchestra.run.smithi007.stdout:setting min_mon_release = quincy 2024-09-18T07:57:05.912 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/monmaptool: set fsid to 779349ca-7593-11ef-bceb-c7b262605968 2024-09-18T07:57:05.912 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/monmaptool: writing epoch 0 to /tmp/monmap (1 monitors) 2024-09-18T07:57:05.912 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-09-18T07:57:05.912 INFO:teuthology.orchestra.run.smithi007.stdout:Creating mon... 2024-09-18T07:57:07.059 INFO:teuthology.orchestra.run.smithi007.stdout:create mon.a on 2024-09-18T07:57:07.282 INFO:teuthology.orchestra.run.smithi007.stdout:systemctl: stderr Removed "/etc/systemd/system/multi-user.target.wants/ceph.target". 2024-09-18T07:57:07.441 INFO:teuthology.orchestra.run.smithi007.stdout:systemctl: stderr Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /etc/systemd/system/ceph.target. 2024-09-18T07:57:07.654 INFO:teuthology.orchestra.run.smithi007.stdout:systemctl: stderr Created symlink /etc/systemd/system/multi-user.target.wants/ceph-779349ca-7593-11ef-bceb-c7b262605968.target → /etc/systemd/system/ceph-779349ca-7593-11ef-bceb-c7b262605968.target. 2024-09-18T07:57:07.654 INFO:teuthology.orchestra.run.smithi007.stdout:systemctl: stderr Created symlink /etc/systemd/system/ceph.target.wants/ceph-779349ca-7593-11ef-bceb-c7b262605968.target → /etc/systemd/system/ceph-779349ca-7593-11ef-bceb-c7b262605968.target. 2024-09-18T07:57:07.910 INFO:teuthology.orchestra.run.smithi007.stdout:Non-zero exit code 1 from systemctl reset-failed ceph-779349ca-7593-11ef-bceb-c7b262605968@mon.a 2024-09-18T07:57:07.910 INFO:teuthology.orchestra.run.smithi007.stdout:systemctl: stderr Failed to reset failed state of unit ceph-779349ca-7593-11ef-bceb-c7b262605968@mon.a.service: Unit ceph-779349ca-7593-11ef-bceb-c7b262605968@mon.a.service not loaded. 2024-09-18T07:57:08.087 INFO:teuthology.orchestra.run.smithi007.stdout:systemctl: stderr Created symlink /etc/systemd/system/ceph-779349ca-7593-11ef-bceb-c7b262605968.target.wants/ceph-779349ca-7593-11ef-bceb-c7b262605968@mon.a.service → /etc/systemd/system/ceph-779349ca-7593-11ef-bceb-c7b262605968@.service. 2024-09-18T07:57:09.324 INFO:teuthology.orchestra.run.smithi007.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-09-18T07:57:09.324 INFO:teuthology.orchestra.run.smithi007.stdout:systemctl: stdout disabled 2024-09-18T07:57:09.331 INFO:teuthology.orchestra.run.smithi007.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-09-18T07:57:09.331 INFO:teuthology.orchestra.run.smithi007.stdout:systemctl: stdout inactive 2024-09-18T07:57:09.331 INFO:teuthology.orchestra.run.smithi007.stdout:firewalld.service is not enabled 2024-09-18T07:57:09.331 INFO:teuthology.orchestra.run.smithi007.stdout:Not possible to enable service . firewalld.service is not available 2024-09-18T07:57:09.332 INFO:teuthology.orchestra.run.smithi007.stdout:Waiting for mon to start... 2024-09-18T07:57:09.332 INFO:teuthology.orchestra.run.smithi007.stdout:Waiting for mon... 2024-09-18T07:57:10.416 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 07:57:10 smithi007 ceph-mon[29093]: from='client.? 172.21.15.7:0/1651989378' entity='client.admin' cmd={"prefix": "status"} : dispatch 2024-09-18T07:57:10.418 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout cluster: 2024-09-18T07:57:10.419 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout id: 779349ca-7593-11ef-bceb-c7b262605968 2024-09-18T07:57:10.420 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout health: HEALTH_OK 2024-09-18T07:57:10.420 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout 2024-09-18T07:57:10.420 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout services: 2024-09-18T07:57:10.420 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout mon: 1 daemons, quorum a (age 0.5452s) 2024-09-18T07:57:10.420 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout mgr: no daemons active 2024-09-18T07:57:10.420 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout osd: 0 osds: 0 up, 0 in 2024-09-18T07:57:10.420 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout 2024-09-18T07:57:10.420 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout data: 2024-09-18T07:57:10.420 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout pools: 0 pools, 0 pgs 2024-09-18T07:57:10.420 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout objects: 0 objects, 0 B 2024-09-18T07:57:10.420 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout usage: 0 B used, 0 B / 0 B avail 2024-09-18T07:57:10.420 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout pgs: 2024-09-18T07:57:10.421 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout 2024-09-18T07:57:10.421 INFO:teuthology.orchestra.run.smithi007.stdout:mon is available 2024-09-18T07:57:10.421 INFO:teuthology.orchestra.run.smithi007.stdout:Assimilating anything we can from ceph.conf... 2024-09-18T07:57:11.481 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 07:57:11 smithi007 ceph-mon[29093]: from='client.? 172.21.15.7:0/2998589199' entity='client.admin' cmd={"prefix": "config assimilate-conf"} : dispatch 2024-09-18T07:57:11.481 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 07:57:11 smithi007 ceph-mon[29093]: from='client.? 172.21.15.7:0/2998589199' entity='client.admin' cmd='[{"prefix": "config assimilate-conf"}]': finished 2024-09-18T07:57:11.493 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout 2024-09-18T07:57:11.493 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout [global] 2024-09-18T07:57:11.493 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout fsid = 779349ca-7593-11ef-bceb-c7b262605968 2024-09-18T07:57:11.493 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout mon_cluster_log_file_level = debug 2024-09-18T07:57:11.493 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout mon_host = [v2:172.21.15.7:3300,v1:172.21.15.7:6789] 2024-09-18T07:57:11.493 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout mon_osd_allow_pg_remap = true 2024-09-18T07:57:11.494 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout mon_osd_allow_primary_affinity = true 2024-09-18T07:57:11.494 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout mon_warn_on_no_sortbitwise = false 2024-09-18T07:57:11.494 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout osd_crush_chooseleaf_type = 0 2024-09-18T07:57:11.494 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout 2024-09-18T07:57:11.494 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout [mgr] 2024-09-18T07:57:11.494 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout mgr/telemetry/nag = false 2024-09-18T07:57:11.494 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout 2024-09-18T07:57:11.494 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout [osd] 2024-09-18T07:57:11.494 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout osd_map_max_advance = 10 2024-09-18T07:57:11.494 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout osd_sloppy_crc = true 2024-09-18T07:57:11.494 INFO:teuthology.orchestra.run.smithi007.stdout:Generating new minimal ceph.conf... 2024-09-18T07:57:12.529 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 07:57:12 smithi007 ceph-mon[29093]: from='client.? 172.21.15.7:0/2249110522' entity='client.admin' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-18T07:57:12.532 INFO:teuthology.orchestra.run.smithi007.stdout:Restarting the monitor... 2024-09-18T07:57:12.808 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 07:57:12 smithi007 systemd[1]: Stopping Ceph mon.a for 779349ca-7593-11ef-bceb-c7b262605968... 2024-09-18T07:57:13.132 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 07:57:12 smithi007 ceph-779349ca-7593-11ef-bceb-c7b262605968-mon-a[29089]: 2024-09-18T07:57:12.805+0000 7f0bef336640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.a -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-09-18T07:57:13.132 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 07:57:12 smithi007 ceph-779349ca-7593-11ef-bceb-c7b262605968-mon-a[29089]: 2024-09-18T07:57:12.805+0000 7f0bef336640 -1 mon.a@0(leader) e1 *** Got Signal Terminated *** 2024-09-18T07:57:13.132 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 07:57:12 smithi007 podman[29334]: 2024-09-18 07:57:12.815795058 +0000 UTC m=+0.116061517 container died 1b344eb8ff7d47fa162bf4ba5086d2f5ad3c329e07845af3b4df0d8ae75816d1 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e, name=ceph-779349ca-7593-11ef-bceb-c7b262605968-mon-a, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.37.2, RELEASE=wip-guits-main-2024-09-17-1213-e7fb7b5, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240731, GIT_CLEAN=True, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux ) 2024-09-18T07:57:13.132 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 07:57:13 smithi007 podman[29334]: 2024-09-18 07:57:13.129831422 +0000 UTC m=+0.430097927 container cleanup 1b344eb8ff7d47fa162bf4ba5086d2f5ad3c329e07845af3b4df0d8ae75816d1 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e, name=ceph-779349ca-7593-11ef-bceb-c7b262605968-mon-a, org.label-schema.vendor=CentOS, io.buildah.version=1.37.2, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.name=CentOS Stream 9 Base Image, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, org.label-schema.license=GPLv2, RELEASE=wip-guits-main-2024-09-17-1213-e7fb7b5, org.label-schema.build-date=20240731, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD) 2024-09-18T07:57:13.132 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 07:57:13 smithi007 bash[29334]: ceph-779349ca-7593-11ef-bceb-c7b262605968-mon-a 2024-09-18T07:57:13.416 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 07:57:13 smithi007 podman[29346]: 2024-09-18 07:57:13.322519857 +0000 UTC m=+0.503907593 container remove 1b344eb8ff7d47fa162bf4ba5086d2f5ad3c329e07845af3b4df0d8ae75816d1 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e, name=ceph-779349ca-7593-11ef-bceb-c7b262605968-mon-a, RELEASE=wip-guits-main-2024-09-17-1213-e7fb7b5, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.build-date=20240731, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.schema-version=1.0, io.buildah.version=1.37.2, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, GIT_BRANCH=HEAD) 2024-09-18T07:57:13.731 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 07:57:13 smithi007 systemd[1]: ceph-779349ca-7593-11ef-bceb-c7b262605968@mon.a.service: Deactivated successfully. 2024-09-18T07:57:13.731 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 07:57:13 smithi007 systemd[1]: Stopped Ceph mon.a for 779349ca-7593-11ef-bceb-c7b262605968. 2024-09-18T07:57:13.732 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 07:57:13 smithi007 systemd[1]: Starting Ceph mon.a for 779349ca-7593-11ef-bceb-c7b262605968... 2024-09-18T07:57:14.012 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 07:57:13 smithi007 podman[29440]: 2024-09-18 07:57:13.731731276 +0000 UTC m=+0.101108766 container create 60741299c8e8988ce730727399daef54ec09a04024a132d4d8b67d279d4f75d5 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e, name=ceph-779349ca-7593-11ef-bceb-c7b262605968-mon-a, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.37.2, org.label-schema.schema-version=1.0, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 9 Base Image, maintainer=Guillaume Abrioux , RELEASE=wip-guits-main-2024-09-17-1213-e7fb7b5, org.label-schema.build-date=20240731, ceph=True) 2024-09-18T07:57:14.012 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 07:57:13 smithi007 podman[29440]: 2024-09-18 07:57:13.646868552 +0000 UTC m=+0.016246058 image pull b22e1e37ddf1ca6e22136cd6540a9ab61e7cafb8bfbc7a91a2f20f7f9941bd9a quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e 2024-09-18T07:57:14.012 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 07:57:13 smithi007 podman[29440]: 2024-09-18 07:57:13.934477119 +0000 UTC m=+0.303854617 container init 60741299c8e8988ce730727399daef54ec09a04024a132d4d8b67d279d4f75d5 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e, name=ceph-779349ca-7593-11ef-bceb-c7b262605968-mon-a, CEPH_POINT_RELEASE=, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, ceph=True, io.buildah.version=1.37.2, org.label-schema.schema-version=1.0, RELEASE=wip-guits-main-2024-09-17-1213-e7fb7b5, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.build-date=20240731, org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-09-18T07:57:14.012 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 07:57:13 smithi007 podman[29440]: 2024-09-18 07:57:13.937585841 +0000 UTC m=+0.306963335 container start 60741299c8e8988ce730727399daef54ec09a04024a132d4d8b67d279d4f75d5 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e, name=ceph-779349ca-7593-11ef-bceb-c7b262605968-mon-a, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.37.2, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, ceph=True, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240731, RELEASE=wip-guits-main-2024-09-17-1213-e7fb7b5, CEPH_POINT_RELEASE=, GIT_CLEAN=True, GIT_BRANCH=HEAD) 2024-09-18T07:57:14.012 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 07:57:13 smithi007 ceph-mon[29459]: set uid:gid to 167:167 (ceph:ceph) 2024-09-18T07:57:14.012 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 07:57:13 smithi007 ceph-mon[29459]: ceph version 19.3.0-4983-ge7fb7b56 (e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e) squid (dev), process ceph-mon, pid 2 2024-09-18T07:57:14.012 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 07:57:13 smithi007 ceph-mon[29459]: pidfile_write: ignore empty --pid-file 2024-09-18T07:57:14.012 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 07:57:13 smithi007 ceph-mon[29459]: load: jerasure load: lrc 2024-09-18T07:57:14.013 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 07:57:13 smithi007 ceph-mon[29459]: rocksdb: RocksDB version: 7.9.2 2024-09-18T07:57:14.013 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 07:57:13 smithi007 ceph-mon[29459]: rocksdb: Git sha 0 2024-09-18T07:57:14.013 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 07:57:13 smithi007 ceph-mon[29459]: rocksdb: Compile date 2024-09-17 15:59:30 2024-09-18T07:57:14.013 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 07:57:13 smithi007 ceph-mon[29459]: rocksdb: DB SUMMARY 2024-09-18T07:57:14.013 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 07:57:13 smithi007 ceph-mon[29459]: rocksdb: DB Session ID: O8UL8A97QOX9TZTUBRS5 2024-09-18T07:57:14.013 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 07:57:13 smithi007 ceph-mon[29459]: rocksdb: CURRENT file: CURRENT 2024-09-18T07:57:14.013 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 07:57:13 smithi007 ceph-mon[29459]: rocksdb: IDENTITY file: IDENTITY 2024-09-18T07:57:14.013 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 07:57:13 smithi007 ceph-mon[29459]: rocksdb: MANIFEST file: MANIFEST-000010 size: 179 Bytes 2024-09-18T07:57:14.013 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 07:57:13 smithi007 ceph-mon[29459]: rocksdb: SST files in /var/lib/ceph/mon/ceph-a/store.db dir, Total Num: 1, files: 000008.sst 2024-09-18T07:57:14.013 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 07:57:13 smithi007 ceph-mon[29459]: rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-a/store.db: 000009.log size: 95864 ; 2024-09-18T07:57:14.013 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 07:57:13 smithi007 ceph-mon[29459]: rocksdb: Options.error_if_exists: 0 2024-09-18T07:57:14.014 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 07:57:13 smithi007 ceph-mon[29459]: rocksdb: Options.create_if_missing: 0 2024-09-18T07:57:14.014 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 07:57:13 smithi007 ceph-mon[29459]: rocksdb: Options.paranoid_checks: 1 2024-09-18T07:57:14.014 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 07:57:13 smithi007 ceph-mon[29459]: rocksdb: Options.flush_verify_memtable_count: 1 2024-09-18T07:57:14.014 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 07:57:13 smithi007 ceph-mon[29459]: rocksdb: Options.track_and_verify_wals_in_manifest: 0 2024-09-18T07:57:14.014 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 07:57:13 smithi007 ceph-mon[29459]: rocksdb: Options.verify_sst_unique_id_in_manifest: 1 2024-09-18T07:57:14.014 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 07:57:13 smithi007 ceph-mon[29459]: rocksdb: Options.env: 0x5567b8eae120 2024-09-18T07:57:14.014 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 07:57:13 smithi007 ceph-mon[29459]: rocksdb: Options.fs: PosixFileSystem 2024-09-18T07:57:14.014 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 07:57:13 smithi007 ceph-mon[29459]: rocksdb: Options.info_log: 0x5567bb46f700 2024-09-18T07:57:14.014 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 07:57:13 smithi007 ceph-mon[29459]: rocksdb: Options.max_file_opening_threads: 16 2024-09-18T07:57:14.014 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 07:57:13 smithi007 ceph-mon[29459]: rocksdb: Options.statistics: (nil) 2024-09-18T07:57:14.014 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 07:57:13 smithi007 ceph-mon[29459]: rocksdb: Options.use_fsync: 0 2024-09-18T07:57:14.015 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 07:57:13 smithi007 ceph-mon[29459]: rocksdb: Options.max_log_file_size: 0 2024-09-18T07:57:14.015 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 07:57:13 smithi007 ceph-mon[29459]: rocksdb: Options.max_manifest_file_size: 1073741824 2024-09-18T07:57:14.015 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 07:57:13 smithi007 ceph-mon[29459]: rocksdb: Options.log_file_time_to_roll: 0 2024-09-18T07:57:14.015 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 07:57:13 smithi007 ceph-mon[29459]: rocksdb: Options.keep_log_file_num: 1000 2024-09-18T07:57:14.015 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 07:57:13 smithi007 ceph-mon[29459]: rocksdb: Options.recycle_log_file_num: 0 2024-09-18T07:57:14.015 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 07:57:13 smithi007 ceph-mon[29459]: rocksdb: Options.allow_fallocate: 1 2024-09-18T07:57:14.015 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 07:57:13 smithi007 ceph-mon[29459]: rocksdb: Options.allow_mmap_reads: 0 2024-09-18T07:57:14.015 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 07:57:13 smithi007 ceph-mon[29459]: rocksdb: Options.allow_mmap_writes: 0 2024-09-18T07:57:14.015 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 07:57:13 smithi007 ceph-mon[29459]: rocksdb: Options.use_direct_reads: 0 2024-09-18T07:57:14.015 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 07:57:13 smithi007 ceph-mon[29459]: rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2024-09-18T07:57:14.015 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 07:57:13 smithi007 ceph-mon[29459]: rocksdb: Options.create_missing_column_families: 0 2024-09-18T07:57:14.015 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 07:57:13 smithi007 ceph-mon[29459]: rocksdb: Options.db_log_dir: 2024-09-18T07:57:14.015 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 07:57:13 smithi007 ceph-mon[29459]: rocksdb: Options.wal_dir: 2024-09-18T07:57:14.016 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 07:57:13 smithi007 ceph-mon[29459]: rocksdb: Options.table_cache_numshardbits: 6 2024-09-18T07:57:14.016 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 07:57:13 smithi007 ceph-mon[29459]: rocksdb: Options.WAL_ttl_seconds: 0 2024-09-18T07:57:14.016 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 07:57:13 smithi007 ceph-mon[29459]: rocksdb: Options.WAL_size_limit_MB: 0 2024-09-18T07:57:14.016 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 07:57:13 smithi007 ceph-mon[29459]: rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2024-09-18T07:57:14.016 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 07:57:13 smithi007 ceph-mon[29459]: rocksdb: Options.manifest_preallocation_size: 4194304 2024-09-18T07:57:14.016 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 07:57:13 smithi007 ceph-mon[29459]: rocksdb: Options.is_fd_close_on_exec: 1 2024-09-18T07:57:14.016 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 07:57:13 smithi007 ceph-mon[29459]: rocksdb: Options.advise_random_on_open: 1 2024-09-18T07:57:14.016 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 07:57:13 smithi007 ceph-mon[29459]: rocksdb: Options.db_write_buffer_size: 0 2024-09-18T07:57:14.016 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 07:57:13 smithi007 ceph-mon[29459]: rocksdb: Options.write_buffer_manager: 0x5567bb473b80 2024-09-18T07:57:14.016 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 07:57:13 smithi007 ceph-mon[29459]: rocksdb: Options.access_hint_on_compaction_start: 1 2024-09-18T07:57:14.016 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 07:57:13 smithi007 ceph-mon[29459]: rocksdb: Options.random_access_max_buffer_size: 1048576 2024-09-18T07:57:14.016 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 07:57:13 smithi007 ceph-mon[29459]: rocksdb: Options.use_adaptive_mutex: 0 2024-09-18T07:57:14.016 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 07:57:13 smithi007 ceph-mon[29459]: rocksdb: Options.rate_limiter: (nil) 2024-09-18T07:57:14.017 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 07:57:13 smithi007 ceph-mon[29459]: rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2024-09-18T07:57:14.018 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 07:57:13 smithi007 ceph-mon[29459]: rocksdb: Options.wal_recovery_mode: 2 2024-09-18T07:57:14.018 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 07:57:13 smithi007 ceph-mon[29459]: rocksdb: Options.enable_thread_tracking: 0 2024-09-18T07:57:14.018 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 07:57:13 smithi007 ceph-mon[29459]: rocksdb: Options.enable_pipelined_write: 0 2024-09-18T07:57:14.018 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 07:57:13 smithi007 ceph-mon[29459]: rocksdb: Options.unordered_write: 0 2024-09-18T07:57:14.018 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 07:57:13 smithi007 ceph-mon[29459]: rocksdb: Options.allow_concurrent_memtable_write: 1 2024-09-18T07:57:14.018 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 07:57:13 smithi007 ceph-mon[29459]: rocksdb: Options.enable_write_thread_adaptive_yield: 1 2024-09-18T07:57:14.018 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 07:57:13 smithi007 ceph-mon[29459]: rocksdb: Options.write_thread_max_yield_usec: 100 2024-09-18T07:57:14.018 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 07:57:13 smithi007 ceph-mon[29459]: rocksdb: Options.write_thread_slow_yield_usec: 3 2024-09-18T07:57:14.019 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 07:57:13 smithi007 ceph-mon[29459]: rocksdb: Options.row_cache: None 2024-09-18T07:57:14.019 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 07:57:13 smithi007 ceph-mon[29459]: rocksdb: Options.wal_filter: None 2024-09-18T07:57:14.019 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 07:57:13 smithi007 ceph-mon[29459]: rocksdb: Options.avoid_flush_during_recovery: 0 2024-09-18T07:57:14.019 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 07:57:13 smithi007 ceph-mon[29459]: rocksdb: Options.allow_ingest_behind: 0 2024-09-18T07:57:14.019 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 07:57:13 smithi007 ceph-mon[29459]: rocksdb: Options.two_write_queues: 0 2024-09-18T07:57:14.019 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 07:57:13 smithi007 ceph-mon[29459]: rocksdb: Options.manual_wal_flush: 0 2024-09-18T07:57:14.019 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 07:57:13 smithi007 ceph-mon[29459]: rocksdb: Options.wal_compression: 0 2024-09-18T07:57:14.019 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 07:57:13 smithi007 ceph-mon[29459]: rocksdb: Options.atomic_flush: 0 2024-09-18T07:57:14.019 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 07:57:13 smithi007 ceph-mon[29459]: rocksdb: Options.avoid_unnecessary_blocking_io: 0 2024-09-18T07:57:14.019 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 07:57:13 smithi007 ceph-mon[29459]: rocksdb: Options.persist_stats_to_disk: 0 2024-09-18T07:57:14.019 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 07:57:13 smithi007 ceph-mon[29459]: rocksdb: Options.write_dbid_to_manifest: 0 2024-09-18T07:57:14.019 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 07:57:13 smithi007 ceph-mon[29459]: rocksdb: Options.log_readahead_size: 0 2024-09-18T07:57:14.020 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 07:57:13 smithi007 ceph-mon[29459]: rocksdb: Options.file_checksum_gen_factory: Unknown 2024-09-18T07:57:14.020 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 07:57:13 smithi007 ceph-mon[29459]: rocksdb: Options.best_efforts_recovery: 0 2024-09-18T07:57:14.020 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 07:57:13 smithi007 ceph-mon[29459]: rocksdb: Options.max_bgerror_resume_count: 2147483647 2024-09-18T07:57:14.020 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 07:57:13 smithi007 ceph-mon[29459]: rocksdb: Options.bgerror_resume_retry_interval: 1000000 2024-09-18T07:57:14.020 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 07:57:13 smithi007 ceph-mon[29459]: rocksdb: Options.allow_data_in_errors: 0 2024-09-18T07:57:14.020 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 07:57:13 smithi007 ceph-mon[29459]: rocksdb: Options.db_host_id: __hostname__ 2024-09-18T07:57:14.020 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 07:57:13 smithi007 ceph-mon[29459]: rocksdb: Options.enforce_single_del_contracts: true 2024-09-18T07:57:14.020 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 07:57:13 smithi007 ceph-mon[29459]: rocksdb: Options.max_background_jobs: 2 2024-09-18T07:57:14.020 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 07:57:13 smithi007 ceph-mon[29459]: rocksdb: Options.max_background_compactions: -1 2024-09-18T07:57:14.020 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 07:57:13 smithi007 ceph-mon[29459]: rocksdb: Options.max_subcompactions: 1 2024-09-18T07:57:14.020 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 07:57:13 smithi007 ceph-mon[29459]: rocksdb: Options.avoid_flush_during_shutdown: 0 2024-09-18T07:57:14.020 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 07:57:13 smithi007 ceph-mon[29459]: rocksdb: Options.writable_file_max_buffer_size: 1048576 2024-09-18T07:57:14.020 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 07:57:13 smithi007 ceph-mon[29459]: rocksdb: Options.delayed_write_rate : 16777216 2024-09-18T07:57:14.021 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 07:57:13 smithi007 ceph-mon[29459]: rocksdb: Options.max_total_wal_size: 0 2024-09-18T07:57:14.021 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 07:57:13 smithi007 ceph-mon[29459]: rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2024-09-18T07:57:14.021 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 07:57:13 smithi007 ceph-mon[29459]: rocksdb: Options.stats_dump_period_sec: 600 2024-09-18T07:57:14.021 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 07:57:13 smithi007 ceph-mon[29459]: rocksdb: Options.stats_persist_period_sec: 600 2024-09-18T07:57:14.021 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 07:57:13 smithi007 ceph-mon[29459]: rocksdb: Options.stats_history_buffer_size: 1048576 2024-09-18T07:57:14.021 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 07:57:13 smithi007 ceph-mon[29459]: rocksdb: Options.max_open_files: -1 2024-09-18T07:57:14.021 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 07:57:13 smithi007 ceph-mon[29459]: rocksdb: Options.bytes_per_sync: 0 2024-09-18T07:57:14.021 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 07:57:13 smithi007 ceph-mon[29459]: rocksdb: Options.wal_bytes_per_sync: 0 2024-09-18T07:57:14.021 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 07:57:13 smithi007 ceph-mon[29459]: rocksdb: Options.strict_bytes_per_sync: 0 2024-09-18T07:57:14.021 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 07:57:13 smithi007 ceph-mon[29459]: rocksdb: Options.compaction_readahead_size: 0 2024-09-18T07:57:14.021 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 07:57:13 smithi007 ceph-mon[29459]: rocksdb: Options.max_background_flushes: -1 2024-09-18T07:57:14.021 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 07:57:13 smithi007 ceph-mon[29459]: rocksdb: Compression algorithms supported: 2024-09-18T07:57:14.021 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 07:57:13 smithi007 ceph-mon[29459]: rocksdb: kZSTD supported: 0 2024-09-18T07:57:14.022 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 07:57:13 smithi007 ceph-mon[29459]: rocksdb: kXpressCompression supported: 0 2024-09-18T07:57:14.022 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 07:57:13 smithi007 ceph-mon[29459]: rocksdb: kBZip2Compression supported: 0 2024-09-18T07:57:14.022 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 07:57:13 smithi007 ceph-mon[29459]: rocksdb: kZSTDNotFinalCompression supported: 0 2024-09-18T07:57:14.022 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 07:57:13 smithi007 ceph-mon[29459]: rocksdb: kLZ4Compression supported: 1 2024-09-18T07:57:14.022 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 07:57:13 smithi007 ceph-mon[29459]: rocksdb: kZlibCompression supported: 1 2024-09-18T07:57:14.022 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 07:57:13 smithi007 ceph-mon[29459]: rocksdb: kLZ4HCCompression supported: 1 2024-09-18T07:57:14.022 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 07:57:13 smithi007 ceph-mon[29459]: rocksdb: kSnappyCompression supported: 1 2024-09-18T07:57:14.022 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 07:57:13 smithi007 ceph-mon[29459]: rocksdb: Fast CRC32 supported: Supported on x86 2024-09-18T07:57:14.022 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 07:57:13 smithi007 ceph-mon[29459]: rocksdb: DMutex implementation: pthread_mutex_t 2024-09-18T07:57:14.022 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 07:57:13 smithi007 ceph-mon[29459]: rocksdb: [db/version_set.cc:5527] Recovering from manifest file: /var/lib/ceph/mon/ceph-a/store.db/MANIFEST-000010 2024-09-18T07:57:14.022 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 07:57:13 smithi007 ceph-mon[29459]: rocksdb: [db/column_family.cc:630] --------------- Options for column family [default]: 2024-09-18T07:57:14.022 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 07:57:13 smithi007 ceph-mon[29459]: rocksdb: Options.comparator: leveldb.BytewiseComparator 2024-09-18T07:57:14.022 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 07:57:13 smithi007 ceph-mon[29459]: rocksdb: Options.merge_operator: 2024-09-18T07:57:14.022 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 07:57:13 smithi007 ceph-mon[29459]: rocksdb: Options.compaction_filter: None 2024-09-18T07:57:14.023 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 07:57:13 smithi007 ceph-mon[29459]: rocksdb: Options.compaction_filter_factory: None 2024-09-18T07:57:14.023 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 07:57:13 smithi007 ceph-mon[29459]: rocksdb: Options.sst_partitioner_factory: None 2024-09-18T07:57:14.023 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 07:57:13 smithi007 ceph-mon[29459]: rocksdb: Options.memtable_factory: SkipListFactory 2024-09-18T07:57:14.023 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 07:57:13 smithi007 ceph-mon[29459]: rocksdb: Options.table_factory: BlockBasedTable 2024-09-18T07:57:14.023 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 07:57:13 smithi007 ceph-mon[29459]: rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x5567bb46f320) 2024-09-18T07:57:14.023 INFO:journalctl@ceph.mon.a.smithi007.stdout: cache_index_and_filter_blocks: 1 2024-09-18T07:57:14.023 INFO:journalctl@ceph.mon.a.smithi007.stdout: cache_index_and_filter_blocks_with_high_priority: 0 2024-09-18T07:57:14.023 INFO:journalctl@ceph.mon.a.smithi007.stdout: pin_l0_filter_and_index_blocks_in_cache: 0 2024-09-18T07:57:14.023 INFO:journalctl@ceph.mon.a.smithi007.stdout: pin_top_level_index_and_filter: 1 2024-09-18T07:57:14.023 INFO:journalctl@ceph.mon.a.smithi007.stdout: index_type: 0 2024-09-18T07:57:14.023 INFO:journalctl@ceph.mon.a.smithi007.stdout: data_block_index_type: 0 2024-09-18T07:57:14.023 INFO:journalctl@ceph.mon.a.smithi007.stdout: index_shortening: 1 2024-09-18T07:57:14.023 INFO:journalctl@ceph.mon.a.smithi007.stdout: data_block_hash_table_util_ratio: 0.750000 2024-09-18T07:57:14.024 INFO:journalctl@ceph.mon.a.smithi007.stdout: checksum: 4 2024-09-18T07:57:14.024 INFO:journalctl@ceph.mon.a.smithi007.stdout: no_block_cache: 0 2024-09-18T07:57:14.024 INFO:journalctl@ceph.mon.a.smithi007.stdout: block_cache: 0x5567bb463610 2024-09-18T07:57:14.024 INFO:journalctl@ceph.mon.a.smithi007.stdout: block_cache_name: BinnedLRUCache 2024-09-18T07:57:14.024 INFO:journalctl@ceph.mon.a.smithi007.stdout: block_cache_options: 2024-09-18T07:57:14.024 INFO:journalctl@ceph.mon.a.smithi007.stdout: capacity : 536870912 2024-09-18T07:57:14.024 INFO:journalctl@ceph.mon.a.smithi007.stdout: num_shard_bits : 4 2024-09-18T07:57:14.024 INFO:journalctl@ceph.mon.a.smithi007.stdout: strict_capacity_limit : 0 2024-09-18T07:57:14.024 INFO:journalctl@ceph.mon.a.smithi007.stdout: high_pri_pool_ratio: 0.000 2024-09-18T07:57:14.024 INFO:journalctl@ceph.mon.a.smithi007.stdout: block_cache_compressed: (nil) 2024-09-18T07:57:14.024 INFO:journalctl@ceph.mon.a.smithi007.stdout: persistent_cache: (nil) 2024-09-18T07:57:14.024 INFO:journalctl@ceph.mon.a.smithi007.stdout: block_size: 4096 2024-09-18T07:57:14.024 INFO:journalctl@ceph.mon.a.smithi007.stdout: block_size_deviation: 10 2024-09-18T07:57:14.024 INFO:journalctl@ceph.mon.a.smithi007.stdout: block_restart_interval: 16 2024-09-18T07:57:14.025 INFO:journalctl@ceph.mon.a.smithi007.stdout: index_block_restart_interval: 1 2024-09-18T07:57:14.025 INFO:journalctl@ceph.mon.a.smithi007.stdout: metadata_block_size: 4096 2024-09-18T07:57:14.025 INFO:journalctl@ceph.mon.a.smithi007.stdout: partition_filters: 0 2024-09-18T07:57:14.025 INFO:journalctl@ceph.mon.a.smithi007.stdout: use_delta_encoding: 1 2024-09-18T07:57:14.025 INFO:journalctl@ceph.mon.a.smithi007.stdout: filter_policy: bloomfilter 2024-09-18T07:57:14.025 INFO:journalctl@ceph.mon.a.smithi007.stdout: whole_key_filtering: 1 2024-09-18T07:57:14.025 INFO:journalctl@ceph.mon.a.smithi007.stdout: verify_compression: 0 2024-09-18T07:57:14.025 INFO:journalctl@ceph.mon.a.smithi007.stdout: read_amp_bytes_per_bit: 0 2024-09-18T07:57:14.025 INFO:journalctl@ceph.mon.a.smithi007.stdout: format_version: 5 2024-09-18T07:57:14.025 INFO:journalctl@ceph.mon.a.smithi007.stdout: enable_index_compression: 1 2024-09-18T07:57:14.025 INFO:journalctl@ceph.mon.a.smithi007.stdout: block_align: 0 2024-09-18T07:57:14.025 INFO:journalctl@ceph.mon.a.smithi007.stdout: max_auto_readahead_size: 262144 2024-09-18T07:57:14.025 INFO:journalctl@ceph.mon.a.smithi007.stdout: prepopulate_block_cache: 0 2024-09-18T07:57:14.025 INFO:journalctl@ceph.mon.a.smithi007.stdout: initial_auto_readahead_size: 8192 2024-09-18T07:57:14.025 INFO:journalctl@ceph.mon.a.smithi007.stdout: num_file_reads_for_auto_readahead: 2 2024-09-18T07:57:14.026 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 07:57:13 smithi007 ceph-mon[29459]: rocksdb: Options.write_buffer_size: 33554432 2024-09-18T07:57:14.026 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 07:57:13 smithi007 ceph-mon[29459]: rocksdb: Options.max_write_buffer_number: 2 2024-09-18T07:57:14.026 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 07:57:13 smithi007 ceph-mon[29459]: rocksdb: Options.compression: NoCompression 2024-09-18T07:57:14.026 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 07:57:13 smithi007 ceph-mon[29459]: rocksdb: Options.bottommost_compression: Disabled 2024-09-18T07:57:14.026 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 07:57:13 smithi007 ceph-mon[29459]: rocksdb: Options.prefix_extractor: nullptr 2024-09-18T07:57:14.026 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 07:57:13 smithi007 ceph-mon[29459]: rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2024-09-18T07:57:14.026 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 07:57:13 smithi007 ceph-mon[29459]: rocksdb: Options.num_levels: 7 2024-09-18T07:57:14.026 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 07:57:13 smithi007 ceph-mon[29459]: rocksdb: Options.min_write_buffer_number_to_merge: 1 2024-09-18T07:57:14.026 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 07:57:13 smithi007 ceph-mon[29459]: rocksdb: Options.max_write_buffer_number_to_maintain: 0 2024-09-18T07:57:14.026 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 07:57:13 smithi007 ceph-mon[29459]: rocksdb: Options.max_write_buffer_size_to_maintain: 0 2024-09-18T07:57:14.026 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 07:57:13 smithi007 ceph-mon[29459]: rocksdb: Options.bottommost_compression_opts.window_bits: -14 2024-09-18T07:57:14.026 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 07:57:13 smithi007 ceph-mon[29459]: rocksdb: Options.bottommost_compression_opts.level: 32767 2024-09-18T07:57:14.026 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 07:57:13 smithi007 ceph-mon[29459]: rocksdb: Options.bottommost_compression_opts.strategy: 0 2024-09-18T07:57:14.026 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 07:57:13 smithi007 ceph-mon[29459]: rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2024-09-18T07:57:14.027 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 07:57:13 smithi007 ceph-mon[29459]: rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2024-09-18T07:57:14.027 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 07:57:13 smithi007 ceph-mon[29459]: rocksdb: Options.bottommost_compression_opts.parallel_threads: 1 2024-09-18T07:57:14.027 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 07:57:13 smithi007 ceph-mon[29459]: rocksdb: Options.bottommost_compression_opts.enabled: false 2024-09-18T07:57:14.027 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 07:57:13 smithi007 ceph-mon[29459]: rocksdb: Options.bottommost_compression_opts.max_dict_buffer_bytes: 0 2024-09-18T07:57:14.027 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 07:57:13 smithi007 ceph-mon[29459]: rocksdb: Options.bottommost_compression_opts.use_zstd_dict_trainer: true 2024-09-18T07:57:14.027 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 07:57:13 smithi007 ceph-mon[29459]: rocksdb: Options.compression_opts.window_bits: -14 2024-09-18T07:57:14.027 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 07:57:13 smithi007 ceph-mon[29459]: rocksdb: Options.compression_opts.level: 32767 2024-09-18T07:57:14.027 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 07:57:13 smithi007 ceph-mon[29459]: rocksdb: Options.compression_opts.strategy: 0 2024-09-18T07:57:14.027 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 07:57:13 smithi007 ceph-mon[29459]: rocksdb: Options.compression_opts.max_dict_bytes: 0 2024-09-18T07:57:14.027 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 07:57:13 smithi007 ceph-mon[29459]: rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2024-09-18T07:57:14.027 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 07:57:13 smithi007 ceph-mon[29459]: rocksdb: Options.compression_opts.use_zstd_dict_trainer: true 2024-09-18T07:57:14.027 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 07:57:13 smithi007 ceph-mon[29459]: rocksdb: Options.compression_opts.parallel_threads: 1 2024-09-18T07:57:14.027 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 07:57:13 smithi007 ceph-mon[29459]: rocksdb: Options.compression_opts.enabled: false 2024-09-18T07:57:14.028 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 07:57:13 smithi007 ceph-mon[29459]: rocksdb: Options.compression_opts.max_dict_buffer_bytes: 0 2024-09-18T07:57:14.028 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 07:57:13 smithi007 ceph-mon[29459]: rocksdb: Options.level0_file_num_compaction_trigger: 4 2024-09-18T07:57:14.028 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 07:57:13 smithi007 ceph-mon[29459]: rocksdb: Options.level0_slowdown_writes_trigger: 20 2024-09-18T07:57:14.028 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 07:57:13 smithi007 ceph-mon[29459]: rocksdb: Options.level0_stop_writes_trigger: 36 2024-09-18T07:57:14.028 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 07:57:13 smithi007 ceph-mon[29459]: rocksdb: Options.target_file_size_base: 67108864 2024-09-18T07:57:14.028 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 07:57:13 smithi007 ceph-mon[29459]: rocksdb: Options.target_file_size_multiplier: 1 2024-09-18T07:57:14.028 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 07:57:13 smithi007 ceph-mon[29459]: rocksdb: Options.max_bytes_for_level_base: 268435456 2024-09-18T07:57:14.028 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 07:57:13 smithi007 ceph-mon[29459]: rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2024-09-18T07:57:14.028 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 07:57:13 smithi007 ceph-mon[29459]: rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2024-09-18T07:57:14.028 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 07:57:13 smithi007 ceph-mon[29459]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2024-09-18T07:57:14.028 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 07:57:13 smithi007 ceph-mon[29459]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2024-09-18T07:57:14.028 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 07:57:13 smithi007 ceph-mon[29459]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2024-09-18T07:57:14.028 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 07:57:13 smithi007 ceph-mon[29459]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2024-09-18T07:57:14.028 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 07:57:13 smithi007 ceph-mon[29459]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2024-09-18T07:57:14.029 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 07:57:13 smithi007 ceph-mon[29459]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2024-09-18T07:57:14.029 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 07:57:13 smithi007 ceph-mon[29459]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2024-09-18T07:57:14.029 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 07:57:13 smithi007 ceph-mon[29459]: rocksdb: Options.max_sequential_skip_in_iterations: 8 2024-09-18T07:57:14.029 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 07:57:13 smithi007 ceph-mon[29459]: rocksdb: Options.max_compaction_bytes: 1677721600 2024-09-18T07:57:14.029 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 07:57:13 smithi007 ceph-mon[29459]: rocksdb: Options.ignore_max_compaction_bytes_for_input: true 2024-09-18T07:57:14.029 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 07:57:13 smithi007 ceph-mon[29459]: rocksdb: Options.arena_block_size: 1048576 2024-09-18T07:57:14.029 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 07:57:13 smithi007 ceph-mon[29459]: rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2024-09-18T07:57:14.029 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 07:57:13 smithi007 ceph-mon[29459]: rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2024-09-18T07:57:14.029 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 07:57:13 smithi007 ceph-mon[29459]: rocksdb: Options.disable_auto_compactions: 0 2024-09-18T07:57:14.029 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 07:57:13 smithi007 ceph-mon[29459]: rocksdb: Options.compaction_style: kCompactionStyleLevel 2024-09-18T07:57:14.029 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 07:57:13 smithi007 ceph-mon[29459]: rocksdb: Options.compaction_pri: kMinOverlappingRatio 2024-09-18T07:57:14.029 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 07:57:13 smithi007 ceph-mon[29459]: rocksdb: Options.compaction_options_universal.size_ratio: 1 2024-09-18T07:57:14.029 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 07:57:13 smithi007 ceph-mon[29459]: rocksdb: Options.compaction_options_universal.min_merge_width: 2 2024-09-18T07:57:14.029 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 07:57:13 smithi007 ceph-mon[29459]: rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2024-09-18T07:57:14.030 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 07:57:13 smithi007 ceph-mon[29459]: rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2024-09-18T07:57:14.030 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 07:57:13 smithi007 ceph-mon[29459]: rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2024-09-18T07:57:14.030 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 07:57:13 smithi007 ceph-mon[29459]: rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2024-09-18T07:57:14.030 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 07:57:13 smithi007 ceph-mon[29459]: rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2024-09-18T07:57:14.030 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 07:57:13 smithi007 ceph-mon[29459]: rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2024-09-18T07:57:14.030 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 07:57:13 smithi007 ceph-mon[29459]: rocksdb: Options.table_properties_collectors: CompactOnDeletionCollector (Sliding window size = 32768 Deletion trigger = 16384 Deletion ratio = 0); 2024-09-18T07:57:14.030 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 07:57:13 smithi007 ceph-mon[29459]: rocksdb: Options.inplace_update_support: 0 2024-09-18T07:57:14.030 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 07:57:13 smithi007 ceph-mon[29459]: rocksdb: Options.inplace_update_num_locks: 10000 2024-09-18T07:57:14.030 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 07:57:13 smithi007 ceph-mon[29459]: rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2024-09-18T07:57:14.030 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 07:57:13 smithi007 ceph-mon[29459]: rocksdb: Options.memtable_whole_key_filtering: 0 2024-09-18T07:57:14.030 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 07:57:13 smithi007 ceph-mon[29459]: rocksdb: Options.memtable_huge_page_size: 0 2024-09-18T07:57:14.030 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 07:57:13 smithi007 ceph-mon[29459]: rocksdb: Options.bloom_locality: 0 2024-09-18T07:57:14.030 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 07:57:13 smithi007 ceph-mon[29459]: rocksdb: Options.max_successive_merges: 0 2024-09-18T07:57:14.030 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 07:57:13 smithi007 ceph-mon[29459]: rocksdb: Options.optimize_filters_for_hits: 0 2024-09-18T07:57:14.031 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 07:57:13 smithi007 ceph-mon[29459]: rocksdb: Options.paranoid_file_checks: 0 2024-09-18T07:57:14.031 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 07:57:13 smithi007 ceph-mon[29459]: rocksdb: Options.force_consistency_checks: 1 2024-09-18T07:57:14.031 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 07:57:13 smithi007 ceph-mon[29459]: rocksdb: Options.report_bg_io_stats: 0 2024-09-18T07:57:14.031 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 07:57:13 smithi007 ceph-mon[29459]: rocksdb: Options.ttl: 2592000 2024-09-18T07:57:14.031 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 07:57:13 smithi007 ceph-mon[29459]: rocksdb: Options.periodic_compaction_seconds: 0 2024-09-18T07:57:14.031 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 07:57:13 smithi007 ceph-mon[29459]: rocksdb: Options.preclude_last_level_data_seconds: 0 2024-09-18T07:57:14.031 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 07:57:13 smithi007 ceph-mon[29459]: rocksdb: Options.preserve_internal_time_seconds: 0 2024-09-18T07:57:14.031 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 07:57:13 smithi007 ceph-mon[29459]: rocksdb: Options.enable_blob_files: false 2024-09-18T07:57:14.031 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 07:57:13 smithi007 ceph-mon[29459]: rocksdb: Options.min_blob_size: 0 2024-09-18T07:57:14.031 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 07:57:13 smithi007 ceph-mon[29459]: rocksdb: Options.blob_file_size: 268435456 2024-09-18T07:57:14.031 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 07:57:13 smithi007 ceph-mon[29459]: rocksdb: Options.blob_compression_type: NoCompression 2024-09-18T07:57:14.031 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 07:57:13 smithi007 ceph-mon[29459]: rocksdb: Options.enable_blob_garbage_collection: false 2024-09-18T07:57:14.031 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 07:57:13 smithi007 ceph-mon[29459]: rocksdb: Options.blob_garbage_collection_age_cutoff: 0.250000 2024-09-18T07:57:14.032 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 07:57:13 smithi007 ceph-mon[29459]: rocksdb: Options.blob_garbage_collection_force_threshold: 1.000000 2024-09-18T07:57:14.032 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 07:57:13 smithi007 ceph-mon[29459]: rocksdb: Options.blob_compaction_readahead_size: 0 2024-09-18T07:57:14.032 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 07:57:13 smithi007 ceph-mon[29459]: rocksdb: Options.blob_file_starting_level: 0 2024-09-18T07:57:14.032 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 07:57:13 smithi007 ceph-mon[29459]: rocksdb: Options.experimental_mempurge_threshold: 0.000000 2024-09-18T07:57:14.032 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 07:57:13 smithi007 ceph-mon[29459]: rocksdb: [db/version_set.cc:5566] Recovered from manifest file:/var/lib/ceph/mon/ceph-a/store.db/MANIFEST-000010 succeeded,manifest_file_number is 10, next_file_number is 12, last_sequence is 5, log_number is 5,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 5 2024-09-18T07:57:14.032 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 07:57:13 smithi007 ceph-mon[29459]: rocksdb: [db/version_set.cc:5581] Column family [default] (ID 0), log number is 5 2024-09-18T07:57:14.032 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 07:57:13 smithi007 ceph-mon[29459]: rocksdb: [db/db_impl/db_impl_open.cc:539] DB ID: 99a46a61-48a7-47c0-961a-8fe4a7b2bcc9 2024-09-18T07:57:14.032 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 07:57:13 smithi007 ceph-mon[29459]: rocksdb: EVENT_LOG_v1 {"time_micros": 1726646233965784, "job": 1, "event": "recovery_started", "wal_files": [9]} 2024-09-18T07:57:14.032 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 07:57:13 smithi007 ceph-mon[29459]: rocksdb: [db/db_impl/db_impl_open.cc:1043] Recovering log #9 mode 2 2024-09-18T07:57:14.032 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 07:57:13 smithi007 ceph-mon[29459]: rocksdb: EVENT_LOG_v1 {"time_micros": 1726646233966750, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 13, "file_size": 91823, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 8, "largest_seqno": 289, "table_properties": {"data_size": 89871, "index_size": 272, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 1, "index_value_is_delta_encoded": 1, "filter_size": 645, "raw_key_size": 11920, "raw_average_key_size": 48, "raw_value_size": 83271, "raw_average_value_size": 335, "num_data_blocks": 12, "num_entries": 248, "num_filter_entries": 248, "num_deletions": 3, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "bloomfilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[CompactOnDeletionCollector]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; max_dict_buffer_bytes=0; use_zstd_dict_trainer=1; ", "creation_time": 1726646233, "oldest_key_time": 0, "file_creation_time": 0, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "99a46a61-48a7-47c0-961a-8fe4a7b2bcc9", "db_session_id": "O8UL8A97QOX9TZTUBRS5", "orig_file_number": 13, "seqno_to_time_mapping": "N/A"}} 2024-09-18T07:57:14.032 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 07:57:13 smithi007 ceph-mon[29459]: rocksdb: EVENT_LOG_v1 {"time_micros": 1726646233966854, "job": 1, "event": "recovery_finished"} 2024-09-18T07:57:14.032 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 07:57:13 smithi007 ceph-mon[29459]: rocksdb: [db/version_set.cc:5047] Creating manifest 15 2024-09-18T07:57:14.032 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 07:57:13 smithi007 ceph-mon[29459]: rocksdb: [file/delete_scheduler.cc:74] Deleted file /var/lib/ceph/mon/ceph-a/store.db/000009.log immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024-09-18T07:57:14.033 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 07:57:13 smithi007 ceph-mon[29459]: rocksdb: [db/db_impl/db_impl_open.cc:1987] SstFileManager instance 0x5567bb492e00 2024-09-18T07:57:14.033 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 07:57:13 smithi007 ceph-mon[29459]: rocksdb: DB pointer 0x5567bb594000 2024-09-18T07:57:14.033 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 07:57:13 smithi007 ceph-mon[29459]: rocksdb: [db/db_impl/db_impl.cc:1109] ------- DUMPING STATS ------- 2024-09-18T07:57:14.033 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 07:57:13 smithi007 ceph-mon[29459]: rocksdb: [db/db_impl/db_impl.cc:1111] 2024-09-18T07:57:14.033 INFO:journalctl@ceph.mon.a.smithi007.stdout: ** DB Stats ** 2024-09-18T07:57:14.033 INFO:journalctl@ceph.mon.a.smithi007.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-09-18T07:57:14.033 INFO:journalctl@ceph.mon.a.smithi007.stdout: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2024-09-18T07:57:14.033 INFO:journalctl@ceph.mon.a.smithi007.stdout: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-09-18T07:57:14.033 INFO:journalctl@ceph.mon.a.smithi007.stdout: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2024-09-18T07:57:14.033 INFO:journalctl@ceph.mon.a.smithi007.stdout: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2024-09-18T07:57:14.033 INFO:journalctl@ceph.mon.a.smithi007.stdout: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-09-18T07:57:14.033 INFO:journalctl@ceph.mon.a.smithi007.stdout: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2024-09-18T07:57:14.034 INFO:journalctl@ceph.mon.a.smithi007.stdout: 2024-09-18T07:57:14.034 INFO:journalctl@ceph.mon.a.smithi007.stdout: ** Compaction Stats [default] ** 2024-09-18T07:57:14.034 INFO:journalctl@ceph.mon.a.smithi007.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-09-18T07:57:14.034 INFO:journalctl@ceph.mon.a.smithi007.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ 2024-09-18T07:57:14.034 INFO:journalctl@ceph.mon.a.smithi007.stdout: L0 2/0 91.53 KB 0.5 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 147.4 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-09-18T07:57:14.034 INFO:journalctl@ceph.mon.a.smithi007.stdout: Sum 2/0 91.53 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 147.4 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-09-18T07:57:14.034 INFO:journalctl@ceph.mon.a.smithi007.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 147.4 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-09-18T07:57:14.034 INFO:journalctl@ceph.mon.a.smithi007.stdout: 2024-09-18T07:57:14.034 INFO:journalctl@ceph.mon.a.smithi007.stdout: ** Compaction Stats [default] ** 2024-09-18T07:57:14.034 INFO:journalctl@ceph.mon.a.smithi007.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-09-18T07:57:14.034 INFO:journalctl@ceph.mon.a.smithi007.stdout: --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-09-18T07:57:14.034 INFO:journalctl@ceph.mon.a.smithi007.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 147.4 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-09-18T07:57:14.034 INFO:journalctl@ceph.mon.a.smithi007.stdout: 2024-09-18T07:57:14.035 INFO:journalctl@ceph.mon.a.smithi007.stdout: Blob file count: 0, total size: 0.0 GB, garbage size: 0.0 GB, space amp: 0.0 2024-09-18T07:57:14.035 INFO:journalctl@ceph.mon.a.smithi007.stdout: 2024-09-18T07:57:14.035 INFO:journalctl@ceph.mon.a.smithi007.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-09-18T07:57:14.035 INFO:journalctl@ceph.mon.a.smithi007.stdout: Flush(GB): cumulative 0.000, interval 0.000 2024-09-18T07:57:14.035 INFO:journalctl@ceph.mon.a.smithi007.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2024-09-18T07:57:14.035 INFO:journalctl@ceph.mon.a.smithi007.stdout: AddFile(Total Files): cumulative 0, interval 0 2024-09-18T07:57:14.035 INFO:journalctl@ceph.mon.a.smithi007.stdout: AddFile(L0 Files): cumulative 0, interval 0 2024-09-18T07:57:14.035 INFO:journalctl@ceph.mon.a.smithi007.stdout: AddFile(Keys): cumulative 0, interval 0 2024-09-18T07:57:14.035 INFO:journalctl@ceph.mon.a.smithi007.stdout: Cumulative compaction: 0.00 GB write, 27.79 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-09-18T07:57:14.035 INFO:journalctl@ceph.mon.a.smithi007.stdout: Interval compaction: 0.00 GB write, 27.79 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-09-18T07:57:14.035 INFO:journalctl@ceph.mon.a.smithi007.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-09-18T07:57:14.035 INFO:journalctl@ceph.mon.a.smithi007.stdout: Block cache BinnedLRUCache@0x5567bb463610#2 capacity: 512.00 MB usage: 1.30 KB table_size: 0 occupancy: 18446744073709551615 collections: 1 last_copies: 0 last_secs: 9e-06 secs_since: 0 2024-09-18T07:57:14.035 INFO:journalctl@ceph.mon.a.smithi007.stdout: Block cache entry stats(count,size,portion): FilterBlock(2,0.83 KB,0.000157952%) IndexBlock(2,0.47 KB,8.9407e-05%) Misc(1,0.00 KB,0%) 2024-09-18T07:57:14.036 INFO:journalctl@ceph.mon.a.smithi007.stdout: 2024-09-18T07:57:14.036 INFO:journalctl@ceph.mon.a.smithi007.stdout: ** File Read Latency Histogram By Level [default] ** 2024-09-18T07:57:14.036 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 07:57:13 smithi007 ceph-mon[29459]: starting mon.a rank 0 at public addrs [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] at bind addrs [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon_data /var/lib/ceph/mon/ceph-a fsid 779349ca-7593-11ef-bceb-c7b262605968 2024-09-18T07:57:14.036 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 07:57:13 smithi007 ceph-mon[29459]: mon.a@-1(???) e1 preinit fsid 779349ca-7593-11ef-bceb-c7b262605968 2024-09-18T07:57:14.036 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 07:57:13 smithi007 ceph-mon[29459]: mon.a@-1(???).mds e1 new map 2024-09-18T07:57:14.036 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 07:57:13 smithi007 ceph-mon[29459]: mon.a@-1(???).mds e1 print_map 2024-09-18T07:57:14.036 INFO:journalctl@ceph.mon.a.smithi007.stdout: e1 2024-09-18T07:57:14.036 INFO:journalctl@ceph.mon.a.smithi007.stdout: btime 2024-09-18T07:57:09:295514+0000 2024-09-18T07:57:14.036 INFO:journalctl@ceph.mon.a.smithi007.stdout: enable_multiple, ever_enabled_multiple: 1,1 2024-09-18T07:57:14.036 INFO:journalctl@ceph.mon.a.smithi007.stdout: default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2,11=minor log segments,12=quiesce subvolumes} 2024-09-18T07:57:14.036 INFO:journalctl@ceph.mon.a.smithi007.stdout: legacy client fscid: -1 2024-09-18T07:57:14.036 INFO:journalctl@ceph.mon.a.smithi007.stdout: 2024-09-18T07:57:14.036 INFO:journalctl@ceph.mon.a.smithi007.stdout: No filesystems configured 2024-09-18T07:57:14.037 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 07:57:13 smithi007 ceph-mon[29459]: mon.a@-1(???).osd e1 crush map has features 3314932999778484224, adjusting msgr requires 2024-09-18T07:57:14.037 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 07:57:13 smithi007 ceph-mon[29459]: mon.a@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-09-18T07:57:14.037 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 07:57:13 smithi007 ceph-mon[29459]: mon.a@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-09-18T07:57:14.037 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 07:57:13 smithi007 ceph-mon[29459]: mon.a@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-09-18T07:57:14.037 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 07:57:13 smithi007 ceph-mon[29459]: mon.a@-1(???).paxosservice(auth 1..2) refresh upgraded, format 0 -> 3 2024-09-18T07:57:14.037 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 07:57:14 smithi007 bash[29440]: 60741299c8e8988ce730727399daef54ec09a04024a132d4d8b67d279d4f75d5 2024-09-18T07:57:14.037 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 07:57:14 smithi007 systemd[1]: Started Ceph mon.a for 779349ca-7593-11ef-bceb-c7b262605968. 2024-09-18T07:57:14.037 INFO:teuthology.orchestra.run.smithi007.stdout:Setting public_network to 172.21.0.0/20 in mon config section 2024-09-18T07:57:14.262 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 07:57:14 smithi007 ceph-mon[29459]: mon.a is new leader, mons a in quorum (ranks 0) 2024-09-18T07:57:14.262 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 07:57:14 smithi007 ceph-mon[29459]: monmap epoch 1 2024-09-18T07:57:14.262 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 07:57:14 smithi007 ceph-mon[29459]: fsid 779349ca-7593-11ef-bceb-c7b262605968 2024-09-18T07:57:14.262 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 07:57:14 smithi007 ceph-mon[29459]: last_changed 2024-09-18T07:57:05.327907+0000 2024-09-18T07:57:14.262 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 07:57:14 smithi007 ceph-mon[29459]: created 2024-09-18T07:57:05.327907+0000 2024-09-18T07:57:14.262 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 07:57:14 smithi007 ceph-mon[29459]: min_mon_release 19 (squid) 2024-09-18T07:57:14.262 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 07:57:14 smithi007 ceph-mon[29459]: election_strategy: 1 2024-09-18T07:57:14.262 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 07:57:14 smithi007 ceph-mon[29459]: 0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2024-09-18T07:57:14.263 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 07:57:14 smithi007 ceph-mon[29459]: fsmap 2024-09-18T07:57:14.263 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 07:57:14 smithi007 ceph-mon[29459]: osdmap e1: 0 total, 0 up, 0 in 2024-09-18T07:57:14.263 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 07:57:14 smithi007 ceph-mon[29459]: mgrmap e1: no daemons active 2024-09-18T07:57:15.481 INFO:teuthology.orchestra.run.smithi007.stdout:Wrote config to /etc/ceph/ceph.conf 2024-09-18T07:57:15.501 INFO:teuthology.orchestra.run.smithi007.stdout:Wrote keyring to /etc/ceph/ceph.client.admin.keyring 2024-09-18T07:57:15.502 INFO:teuthology.orchestra.run.smithi007.stdout:Creating mgr... 2024-09-18T07:57:15.502 INFO:teuthology.orchestra.run.smithi007.stdout:Verifying port 0.0.0.0:9283 ... 2024-09-18T07:57:15.503 INFO:teuthology.orchestra.run.smithi007.stdout:Verifying port 0.0.0.0:8765 ... 2024-09-18T07:57:15.578 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 07:57:15 smithi007 ceph-mon[29459]: from='client.? 172.21.15.7:0/3146701669' entity='client.admin' 2024-09-18T07:57:15.763 INFO:teuthology.orchestra.run.smithi007.stdout:Non-zero exit code 1 from systemctl reset-failed ceph-779349ca-7593-11ef-bceb-c7b262605968@mgr.y 2024-09-18T07:57:15.763 INFO:teuthology.orchestra.run.smithi007.stdout:systemctl: stderr Failed to reset failed state of unit ceph-779349ca-7593-11ef-bceb-c7b262605968@mgr.y.service: Unit ceph-779349ca-7593-11ef-bceb-c7b262605968@mgr.y.service not loaded. 2024-09-18T07:57:15.929 INFO:teuthology.orchestra.run.smithi007.stdout:systemctl: stderr Created symlink /etc/systemd/system/ceph-779349ca-7593-11ef-bceb-c7b262605968.target.wants/ceph-779349ca-7593-11ef-bceb-c7b262605968@mgr.y.service → /etc/systemd/system/ceph-779349ca-7593-11ef-bceb-c7b262605968@.service. 2024-09-18T07:57:16.231 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Sep 18 07:57:16 smithi007 podman[29673]: 2024-09-18 07:57:16.168935598 +0000 UTC m=+0.125467661 container create d00c3e2d5d59c0ed8a5bccb0b8816c17315a20b2c5081425b38d5dae022d6609 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e, name=ceph-779349ca-7593-11ef-bceb-c7b262605968-mgr-y, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, RELEASE=wip-guits-main-2024-09-17-1213-e7fb7b5, ceph=True, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, GIT_CLEAN=True, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, org.label-schema.license=GPLv2, org.label-schema.build-date=20240731, org.label-schema.vendor=CentOS, io.buildah.version=1.37.2, org.label-schema.name=CentOS Stream 9 Base Image) 2024-09-18T07:57:16.455 INFO:teuthology.orchestra.run.smithi007.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-09-18T07:57:16.455 INFO:teuthology.orchestra.run.smithi007.stdout:systemctl: stdout disabled 2024-09-18T07:57:16.462 INFO:teuthology.orchestra.run.smithi007.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-09-18T07:57:16.463 INFO:teuthology.orchestra.run.smithi007.stdout:systemctl: stdout inactive 2024-09-18T07:57:16.463 INFO:teuthology.orchestra.run.smithi007.stdout:firewalld.service is not enabled 2024-09-18T07:57:16.463 INFO:teuthology.orchestra.run.smithi007.stdout:Not possible to enable service . firewalld.service is not available 2024-09-18T07:57:16.471 INFO:teuthology.orchestra.run.smithi007.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-09-18T07:57:16.472 INFO:teuthology.orchestra.run.smithi007.stdout:systemctl: stdout disabled 2024-09-18T07:57:16.479 INFO:teuthology.orchestra.run.smithi007.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-09-18T07:57:16.480 INFO:teuthology.orchestra.run.smithi007.stdout:systemctl: stdout inactive 2024-09-18T07:57:16.480 INFO:teuthology.orchestra.run.smithi007.stdout:firewalld.service is not enabled 2024-09-18T07:57:16.480 INFO:teuthology.orchestra.run.smithi007.stdout:Not possible to open ports <[9283, 8765]>. firewalld.service is not available 2024-09-18T07:57:16.480 INFO:teuthology.orchestra.run.smithi007.stdout:Waiting for mgr to start... 2024-09-18T07:57:16.480 INFO:teuthology.orchestra.run.smithi007.stdout:Waiting for mgr... 2024-09-18T07:57:16.603 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Sep 18 07:57:16 smithi007 podman[29673]: 2024-09-18 07:57:16.379447646 +0000 UTC m=+0.335979711 container init d00c3e2d5d59c0ed8a5bccb0b8816c17315a20b2c5081425b38d5dae022d6609 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e, name=ceph-779349ca-7593-11ef-bceb-c7b262605968-mgr-y, org.label-schema.build-date=20240731, org.label-schema.name=CentOS Stream 9 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.37.2, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, ceph=True, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, RELEASE=wip-guits-main-2024-09-17-1213-e7fb7b5, org.label-schema.vendor=CentOS) 2024-09-18T07:57:16.603 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Sep 18 07:57:16 smithi007 podman[29673]: 2024-09-18 07:57:16.382181358 +0000 UTC m=+0.338713431 container start d00c3e2d5d59c0ed8a5bccb0b8816c17315a20b2c5081425b38d5dae022d6609 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e, name=ceph-779349ca-7593-11ef-bceb-c7b262605968-mgr-y, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, RELEASE=wip-guits-main-2024-09-17-1213-e7fb7b5, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 9 Base Image, ceph=True, io.buildah.version=1.37.2, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, org.label-schema.build-date=20240731, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_CLEAN=True) 2024-09-18T07:57:16.604 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Sep 18 07:57:16 smithi007 bash[29673]: d00c3e2d5d59c0ed8a5bccb0b8816c17315a20b2c5081425b38d5dae022d6609 2024-09-18T07:57:16.604 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Sep 18 07:57:16 smithi007 systemd[1]: Started Ceph mgr.y for 779349ca-7593-11ef-bceb-c7b262605968. 2024-09-18T07:57:17.231 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 07:57:17 smithi007 ceph-mon[29459]: from='client.? 172.21.15.7:0/2636678101' entity='client.admin' cmd={"prefix": "status", "format": "json-pretty"} : dispatch 2024-09-18T07:57:17.489 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout 2024-09-18T07:57:17.490 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout { 2024-09-18T07:57:17.490 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout "fsid": "779349ca-7593-11ef-bceb-c7b262605968", 2024-09-18T07:57:17.491 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout "health": { 2024-09-18T07:57:17.491 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2024-09-18T07:57:17.491 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout "checks": {}, 2024-09-18T07:57:17.491 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout "mutes": [] 2024-09-18T07:57:17.491 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout }, 2024-09-18T07:57:17.491 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2024-09-18T07:57:17.491 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout "quorum": [ 2024-09-18T07:57:17.491 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout 0 2024-09-18T07:57:17.491 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout ], 2024-09-18T07:57:17.492 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2024-09-18T07:57:17.492 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout "a" 2024-09-18T07:57:17.492 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout ], 2024-09-18T07:57:17.492 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout "quorum_age": 2, 2024-09-18T07:57:17.492 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout "monmap": { 2024-09-18T07:57:17.492 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-09-18T07:57:17.492 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "squid", 2024-09-18T07:57:17.492 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout "num_mons": 1 2024-09-18T07:57:17.492 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout }, 2024-09-18T07:57:17.492 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout "osdmap": { 2024-09-18T07:57:17.493 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-09-18T07:57:17.493 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2024-09-18T07:57:17.493 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2024-09-18T07:57:17.493 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2024-09-18T07:57:17.493 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2024-09-18T07:57:17.493 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2024-09-18T07:57:17.493 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2024-09-18T07:57:17.493 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout }, 2024-09-18T07:57:17.493 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout "pgmap": { 2024-09-18T07:57:17.493 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2024-09-18T07:57:17.493 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2024-09-18T07:57:17.493 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2024-09-18T07:57:17.493 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2024-09-18T07:57:17.493 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2024-09-18T07:57:17.494 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2024-09-18T07:57:17.494 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2024-09-18T07:57:17.494 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2024-09-18T07:57:17.494 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout }, 2024-09-18T07:57:17.494 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout "fsmap": { 2024-09-18T07:57:17.494 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-09-18T07:57:17.494 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout "btime": "2024-09-18T07:57:09:295514+0000", 2024-09-18T07:57:17.494 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout "by_rank": [], 2024-09-18T07:57:17.494 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout "up:standby": 0 2024-09-18T07:57:17.494 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout }, 2024-09-18T07:57:17.494 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout "mgrmap": { 2024-09-18T07:57:17.494 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout "available": false, 2024-09-18T07:57:17.495 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2024-09-18T07:57:17.495 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout "modules": [ 2024-09-18T07:57:17.495 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout "iostat", 2024-09-18T07:57:17.495 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout "nfs", 2024-09-18T07:57:17.495 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout "restful" 2024-09-18T07:57:17.495 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout ], 2024-09-18T07:57:17.495 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout "services": {} 2024-09-18T07:57:17.495 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout }, 2024-09-18T07:57:17.495 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout "servicemap": { 2024-09-18T07:57:17.495 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-09-18T07:57:17.495 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout "modified": "2024-09-18T07:57:09.296263+0000", 2024-09-18T07:57:17.495 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout "services": {} 2024-09-18T07:57:17.495 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout }, 2024-09-18T07:57:17.495 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout "progress_events": {} 2024-09-18T07:57:17.496 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout } 2024-09-18T07:57:17.496 INFO:teuthology.orchestra.run.smithi007.stdout:mgr not available, waiting (1/15)... 2024-09-18T07:57:20.231 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 07:57:19 smithi007 ceph-mon[29459]: from='client.? 172.21.15.7:0/2022303777' entity='client.admin' cmd={"prefix": "status", "format": "json-pretty"} : dispatch 2024-09-18T07:57:20.511 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout 2024-09-18T07:57:20.512 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout { 2024-09-18T07:57:20.512 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout "fsid": "779349ca-7593-11ef-bceb-c7b262605968", 2024-09-18T07:57:20.512 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout "health": { 2024-09-18T07:57:20.512 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2024-09-18T07:57:20.512 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout "checks": {}, 2024-09-18T07:57:20.512 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout "mutes": [] 2024-09-18T07:57:20.512 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout }, 2024-09-18T07:57:20.512 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2024-09-18T07:57:20.512 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout "quorum": [ 2024-09-18T07:57:20.512 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout 0 2024-09-18T07:57:20.512 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout ], 2024-09-18T07:57:20.512 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2024-09-18T07:57:20.513 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout "a" 2024-09-18T07:57:20.513 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout ], 2024-09-18T07:57:20.513 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout "quorum_age": 5, 2024-09-18T07:57:20.513 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout "monmap": { 2024-09-18T07:57:20.513 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-09-18T07:57:20.513 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "squid", 2024-09-18T07:57:20.513 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout "num_mons": 1 2024-09-18T07:57:20.513 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout }, 2024-09-18T07:57:20.513 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout "osdmap": { 2024-09-18T07:57:20.513 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-09-18T07:57:20.513 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2024-09-18T07:57:20.513 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2024-09-18T07:57:20.513 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2024-09-18T07:57:20.514 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2024-09-18T07:57:20.514 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2024-09-18T07:57:20.514 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2024-09-18T07:57:20.514 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout }, 2024-09-18T07:57:20.514 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout "pgmap": { 2024-09-18T07:57:20.514 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2024-09-18T07:57:20.514 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2024-09-18T07:57:20.514 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2024-09-18T07:57:20.514 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2024-09-18T07:57:20.514 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2024-09-18T07:57:20.514 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2024-09-18T07:57:20.514 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2024-09-18T07:57:20.514 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2024-09-18T07:57:20.514 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout }, 2024-09-18T07:57:20.515 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout "fsmap": { 2024-09-18T07:57:20.515 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-09-18T07:57:20.515 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout "btime": "2024-09-18T07:57:09:295514+0000", 2024-09-18T07:57:20.515 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout "by_rank": [], 2024-09-18T07:57:20.515 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout "up:standby": 0 2024-09-18T07:57:20.515 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout }, 2024-09-18T07:57:20.515 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout "mgrmap": { 2024-09-18T07:57:20.515 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout "available": false, 2024-09-18T07:57:20.515 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2024-09-18T07:57:20.515 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout "modules": [ 2024-09-18T07:57:20.515 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout "iostat", 2024-09-18T07:57:20.515 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout "nfs", 2024-09-18T07:57:20.515 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout "restful" 2024-09-18T07:57:20.516 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout ], 2024-09-18T07:57:20.516 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout "services": {} 2024-09-18T07:57:20.516 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout }, 2024-09-18T07:57:20.516 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout "servicemap": { 2024-09-18T07:57:20.516 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-09-18T07:57:20.516 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout "modified": "2024-09-18T07:57:09.296263+0000", 2024-09-18T07:57:20.516 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout "services": {} 2024-09-18T07:57:20.516 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout }, 2024-09-18T07:57:20.516 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout "progress_events": {} 2024-09-18T07:57:20.516 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout } 2024-09-18T07:57:20.516 INFO:teuthology.orchestra.run.smithi007.stdout:mgr not available, waiting (2/15)... 2024-09-18T07:57:21.981 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 07:57:21 smithi007 ceph-mon[29459]: Activating manager daemon y 2024-09-18T07:57:21.981 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 07:57:21 smithi007 ceph-mon[29459]: mgrmap e2: y(active, starting, since 0.00313368s) 2024-09-18T07:57:21.981 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 07:57:21 smithi007 ceph-mon[29459]: from='mgr.14100 172.21.15.7:0/2666024888' entity='mgr.y' cmd={"prefix": "mds metadata"} : dispatch 2024-09-18T07:57:21.981 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 07:57:21 smithi007 ceph-mon[29459]: from='mgr.14100 172.21.15.7:0/2666024888' entity='mgr.y' cmd={"prefix": "osd metadata"} : dispatch 2024-09-18T07:57:21.981 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 07:57:21 smithi007 ceph-mon[29459]: from='mgr.14100 172.21.15.7:0/2666024888' entity='mgr.y' cmd={"prefix": "mon metadata"} : dispatch 2024-09-18T07:57:21.981 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 07:57:21 smithi007 ceph-mon[29459]: from='mgr.14100 172.21.15.7:0/2666024888' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "a"} : dispatch 2024-09-18T07:57:21.982 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 07:57:21 smithi007 ceph-mon[29459]: from='mgr.14100 172.21.15.7:0/2666024888' entity='mgr.y' cmd={"prefix": "mgr metadata", "who": "y", "id": "y"} : dispatch 2024-09-18T07:57:21.982 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 07:57:21 smithi007 ceph-mon[29459]: Manager daemon y is now available 2024-09-18T07:57:21.982 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 07:57:21 smithi007 ceph-mon[29459]: from='mgr.14100 172.21.15.7:0/2666024888' entity='mgr.y' cmd={"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"} : dispatch 2024-09-18T07:57:21.982 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 07:57:21 smithi007 ceph-mon[29459]: from='mgr.14100 172.21.15.7:0/2666024888' entity='mgr.y' cmd={"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"} : dispatch 2024-09-18T07:57:21.982 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 07:57:21 smithi007 ceph-mon[29459]: from='mgr.14100 172.21.15.7:0/2666024888' entity='mgr.y' 2024-09-18T07:57:21.982 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 07:57:21 smithi007 ceph-mon[29459]: from='mgr.14100 172.21.15.7:0/2666024888' entity='mgr.y' 2024-09-18T07:57:21.982 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 07:57:21 smithi007 ceph-mon[29459]: from='mgr.14100 172.21.15.7:0/2666024888' entity='mgr.y' 2024-09-18T07:57:23.850 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout 2024-09-18T07:57:23.850 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout { 2024-09-18T07:57:23.850 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout "fsid": "779349ca-7593-11ef-bceb-c7b262605968", 2024-09-18T07:57:23.850 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout "health": { 2024-09-18T07:57:23.850 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2024-09-18T07:57:23.850 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout "checks": {}, 2024-09-18T07:57:23.850 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout "mutes": [] 2024-09-18T07:57:23.850 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout }, 2024-09-18T07:57:23.850 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2024-09-18T07:57:23.850 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout "quorum": [ 2024-09-18T07:57:23.850 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout 0 2024-09-18T07:57:23.851 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout ], 2024-09-18T07:57:23.851 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2024-09-18T07:57:23.851 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout "a" 2024-09-18T07:57:23.851 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout ], 2024-09-18T07:57:23.851 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout "quorum_age": 9, 2024-09-18T07:57:23.851 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout "monmap": { 2024-09-18T07:57:23.851 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-09-18T07:57:23.851 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "squid", 2024-09-18T07:57:23.851 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout "num_mons": 1 2024-09-18T07:57:23.851 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout }, 2024-09-18T07:57:23.851 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout "osdmap": { 2024-09-18T07:57:23.851 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-09-18T07:57:23.851 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2024-09-18T07:57:23.852 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2024-09-18T07:57:23.852 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2024-09-18T07:57:23.852 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2024-09-18T07:57:23.852 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2024-09-18T07:57:23.852 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2024-09-18T07:57:23.855 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout }, 2024-09-18T07:57:23.855 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout "pgmap": { 2024-09-18T07:57:23.855 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2024-09-18T07:57:23.855 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2024-09-18T07:57:23.855 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2024-09-18T07:57:23.855 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2024-09-18T07:57:23.856 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2024-09-18T07:57:23.856 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2024-09-18T07:57:23.856 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2024-09-18T07:57:23.856 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2024-09-18T07:57:23.856 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout }, 2024-09-18T07:57:23.856 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout "fsmap": { 2024-09-18T07:57:23.856 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-09-18T07:57:23.856 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout "btime": "2024-09-18T07:57:09:295514+0000", 2024-09-18T07:57:23.856 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout "by_rank": [], 2024-09-18T07:57:23.856 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout "up:standby": 0 2024-09-18T07:57:23.856 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout }, 2024-09-18T07:57:23.856 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout "mgrmap": { 2024-09-18T07:57:23.857 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout "available": true, 2024-09-18T07:57:23.857 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2024-09-18T07:57:23.857 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout "modules": [ 2024-09-18T07:57:23.857 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout "iostat", 2024-09-18T07:57:23.857 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout "nfs", 2024-09-18T07:57:23.857 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout "restful" 2024-09-18T07:57:23.857 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout ], 2024-09-18T07:57:23.857 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout "services": {} 2024-09-18T07:57:23.857 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout }, 2024-09-18T07:57:23.857 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout "servicemap": { 2024-09-18T07:57:23.857 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-09-18T07:57:23.857 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout "modified": "2024-09-18T07:57:09.296263+0000", 2024-09-18T07:57:23.857 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout "services": {} 2024-09-18T07:57:23.857 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout }, 2024-09-18T07:57:23.858 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout "progress_events": {} 2024-09-18T07:57:23.858 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout } 2024-09-18T07:57:23.858 INFO:teuthology.orchestra.run.smithi007.stdout:mgr is available 2024-09-18T07:57:23.969 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 07:57:23 smithi007 ceph-mon[29459]: mgrmap e3: y(active, since 1.00716s) 2024-09-18T07:57:23.970 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 07:57:23 smithi007 ceph-mon[29459]: from='client.? 172.21.15.7:0/3635809449' entity='client.admin' cmd={"prefix": "status", "format": "json-pretty"} : dispatch 2024-09-18T07:57:24.981 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 07:57:24 smithi007 ceph-mon[29459]: mgrmap e4: y(active, since 2s) 2024-09-18T07:57:24.981 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 07:57:24 smithi007 ceph-mon[29459]: from='client.? 172.21.15.7:0/1730335619' entity='client.admin' cmd={"prefix": "config assimilate-conf"} : dispatch 2024-09-18T07:57:25.040 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout 2024-09-18T07:57:25.040 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout [global] 2024-09-18T07:57:25.041 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout fsid = 779349ca-7593-11ef-bceb-c7b262605968 2024-09-18T07:57:25.041 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout mon_cluster_log_file_level = debug 2024-09-18T07:57:25.042 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout mon_host = [v2:172.21.15.7:3300,v1:172.21.15.7:6789] 2024-09-18T07:57:25.042 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout mon_osd_allow_pg_remap = true 2024-09-18T07:57:25.042 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout mon_osd_allow_primary_affinity = true 2024-09-18T07:57:25.043 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout mon_warn_on_no_sortbitwise = false 2024-09-18T07:57:25.043 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout osd_crush_chooseleaf_type = 0 2024-09-18T07:57:25.043 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout 2024-09-18T07:57:25.043 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout [mgr] 2024-09-18T07:57:25.044 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout mgr/telemetry/nag = false 2024-09-18T07:57:25.044 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout 2024-09-18T07:57:25.044 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout [osd] 2024-09-18T07:57:25.044 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout osd_map_max_advance = 10 2024-09-18T07:57:25.044 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout osd_sloppy_crc = true 2024-09-18T07:57:25.044 INFO:teuthology.orchestra.run.smithi007.stdout:Enabling cephadm module... 2024-09-18T07:57:26.231 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 07:57:25 smithi007 ceph-mon[29459]: from='client.? 172.21.15.7:0/816978973' entity='client.admin' cmd={"prefix": "mgr module enable", "module": "cephadm"} : dispatch 2024-09-18T07:57:26.231 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Sep 18 07:57:25 smithi007 ceph-779349ca-7593-11ef-bceb-c7b262605968-mgr-y[29688]: ignoring --setuser ceph since I am not root 2024-09-18T07:57:26.231 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Sep 18 07:57:25 smithi007 ceph-779349ca-7593-11ef-bceb-c7b262605968-mgr-y[29688]: ignoring --setgroup ceph since I am not root 2024-09-18T07:57:26.896 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 07:57:26 smithi007 ceph-mon[29459]: from='client.? 172.21.15.7:0/816978973' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "cephadm"}]': finished 2024-09-18T07:57:26.896 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 07:57:26 smithi007 ceph-mon[29459]: mgrmap e5: y(active, since 4s) 2024-09-18T07:57:27.592 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout { 2024-09-18T07:57:27.593 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout "epoch": 5, 2024-09-18T07:57:27.593 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout "available": true, 2024-09-18T07:57:27.593 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout "active_name": "y", 2024-09-18T07:57:27.593 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout "num_standby": 0 2024-09-18T07:57:27.593 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout } 2024-09-18T07:57:27.593 INFO:teuthology.orchestra.run.smithi007.stdout:Waiting for the mgr to restart... 2024-09-18T07:57:27.593 INFO:teuthology.orchestra.run.smithi007.stdout:Waiting for mgr epoch 5... 2024-09-18T07:57:27.901 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 07:57:27 smithi007 ceph-mon[29459]: from='client.? 172.21.15.7:0/2339504511' entity='client.admin' cmd={"prefix": "mgr stat"} : dispatch 2024-09-18T07:57:31.481 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 07:57:31 smithi007 ceph-mon[29459]: Active manager daemon y restarted 2024-09-18T07:57:31.481 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 07:57:31 smithi007 ceph-mon[29459]: Activating manager daemon y 2024-09-18T07:57:31.481 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 07:57:31 smithi007 ceph-mon[29459]: osdmap e2: 0 total, 0 up, 0 in 2024-09-18T07:57:31.481 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 07:57:31 smithi007 ceph-mon[29459]: mgrmap e6: y(active, starting, since 0.00338905s) 2024-09-18T07:57:31.481 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 07:57:31 smithi007 ceph-mon[29459]: from='mgr.14118 172.21.15.7:0/3020361701' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "a"} : dispatch 2024-09-18T07:57:31.481 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 07:57:31 smithi007 ceph-mon[29459]: from='mgr.14118 172.21.15.7:0/3020361701' entity='mgr.y' cmd={"prefix": "mgr metadata", "who": "y", "id": "y"} : dispatch 2024-09-18T07:57:31.482 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 07:57:31 smithi007 ceph-mon[29459]: from='mgr.14118 172.21.15.7:0/3020361701' entity='mgr.y' cmd={"prefix": "mds metadata"} : dispatch 2024-09-18T07:57:31.482 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 07:57:31 smithi007 ceph-mon[29459]: from='mgr.14118 172.21.15.7:0/3020361701' entity='mgr.y' cmd={"prefix": "osd metadata"} : dispatch 2024-09-18T07:57:31.482 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 07:57:31 smithi007 ceph-mon[29459]: from='mgr.14118 172.21.15.7:0/3020361701' entity='mgr.y' cmd={"prefix": "mon metadata"} : dispatch 2024-09-18T07:57:31.482 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 07:57:31 smithi007 ceph-mon[29459]: Manager daemon y is now available 2024-09-18T07:57:32.608 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout { 2024-09-18T07:57:32.608 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout "mgrmap_epoch": 7, 2024-09-18T07:57:32.608 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout "initialized": true 2024-09-18T07:57:32.608 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout } 2024-09-18T07:57:32.609 INFO:teuthology.orchestra.run.smithi007.stdout:mgr epoch 5 is available 2024-09-18T07:57:32.609 INFO:teuthology.orchestra.run.smithi007.stdout:Verifying orchestrator module is enabled... 2024-09-18T07:57:32.705 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 07:57:32 smithi007 ceph-mon[29459]: from='mgr.14118 172.21.15.7:0/3020361701' entity='mgr.y' 2024-09-18T07:57:32.705 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 07:57:32 smithi007 ceph-mon[29459]: from='mgr.14118 172.21.15.7:0/3020361701' entity='mgr.y' 2024-09-18T07:57:32.705 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 07:57:32 smithi007 ceph-mon[29459]: Found migration_current of "None". Setting to last migration. 2024-09-18T07:57:32.705 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 07:57:32 smithi007 ceph-mon[29459]: from='mgr.14118 172.21.15.7:0/3020361701' entity='mgr.y' 2024-09-18T07:57:32.705 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 07:57:32 smithi007 ceph-mon[29459]: from='mgr.14118 172.21.15.7:0/3020361701' entity='mgr.y' 2024-09-18T07:57:32.705 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 07:57:32 smithi007 ceph-mon[29459]: from='mgr.14118 172.21.15.7:0/3020361701' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-18T07:57:32.705 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 07:57:32 smithi007 ceph-mon[29459]: from='mgr.14118 172.21.15.7:0/3020361701' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-18T07:57:32.705 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 07:57:32 smithi007 ceph-mon[29459]: from='mgr.14118 172.21.15.7:0/3020361701' entity='mgr.y' cmd={"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"} : dispatch 2024-09-18T07:57:32.706 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 07:57:32 smithi007 ceph-mon[29459]: from='mgr.14118 172.21.15.7:0/3020361701' entity='mgr.y' cmd={"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"} : dispatch 2024-09-18T07:57:32.706 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 07:57:32 smithi007 ceph-mon[29459]: mgrmap e7: y(active, since 1.00518s) 2024-09-18T07:57:33.482 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Sep 18 07:57:33 smithi007 ceph-779349ca-7593-11ef-bceb-c7b262605968-mgr-y[29688]: 2024-09-18T07:57:33.150+0000 7f41603ba640 -1 log_channel(cephadm) log [ERR] : [18/Sep/2024:07:57:33] ENGINE Error in HTTPServer.serve 2024-09-18T07:57:33.482 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Sep 18 07:57:33 smithi007 ceph-779349ca-7593-11ef-bceb-c7b262605968-mgr-y[29688]: Traceback (most recent call last): 2024-09-18T07:57:33.482 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Sep 18 07:57:33 smithi007 ceph-779349ca-7593-11ef-bceb-c7b262605968-mgr-y[29688]: File "/lib/python3.9/site-packages/cheroot/server.py", line 1823, in serve 2024-09-18T07:57:33.483 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Sep 18 07:57:33 smithi007 ceph-779349ca-7593-11ef-bceb-c7b262605968-mgr-y[29688]: self._connections.run(self.expiration_interval) 2024-09-18T07:57:33.483 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Sep 18 07:57:33 smithi007 ceph-779349ca-7593-11ef-bceb-c7b262605968-mgr-y[29688]: File "/lib/python3.9/site-packages/cheroot/connections.py", line 203, in run 2024-09-18T07:57:33.483 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Sep 18 07:57:33 smithi007 ceph-779349ca-7593-11ef-bceb-c7b262605968-mgr-y[29688]: self._run(expiration_interval) 2024-09-18T07:57:33.483 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Sep 18 07:57:33 smithi007 ceph-779349ca-7593-11ef-bceb-c7b262605968-mgr-y[29688]: File "/lib/python3.9/site-packages/cheroot/connections.py", line 246, in _run 2024-09-18T07:57:33.483 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Sep 18 07:57:33 smithi007 ceph-779349ca-7593-11ef-bceb-c7b262605968-mgr-y[29688]: new_conn = self._from_server_socket(self.server.socket) 2024-09-18T07:57:33.483 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Sep 18 07:57:33 smithi007 ceph-779349ca-7593-11ef-bceb-c7b262605968-mgr-y[29688]: File "/lib/python3.9/site-packages/cheroot/connections.py", line 300, in _from_server_socket 2024-09-18T07:57:33.483 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Sep 18 07:57:33 smithi007 ceph-779349ca-7593-11ef-bceb-c7b262605968-mgr-y[29688]: s, ssl_env = self.server.ssl_adapter.wrap(s) 2024-09-18T07:57:33.484 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Sep 18 07:57:33 smithi007 ceph-779349ca-7593-11ef-bceb-c7b262605968-mgr-y[29688]: File "/lib/python3.9/site-packages/cheroot/ssl/builtin.py", line 277, in wrap 2024-09-18T07:57:33.484 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Sep 18 07:57:33 smithi007 ceph-779349ca-7593-11ef-bceb-c7b262605968-mgr-y[29688]: s = self.context.wrap_socket( 2024-09-18T07:57:33.484 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Sep 18 07:57:33 smithi007 ceph-779349ca-7593-11ef-bceb-c7b262605968-mgr-y[29688]: File "/lib64/python3.9/ssl.py", line 501, in wrap_socket 2024-09-18T07:57:33.484 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Sep 18 07:57:33 smithi007 ceph-779349ca-7593-11ef-bceb-c7b262605968-mgr-y[29688]: return self.sslsocket_class._create( 2024-09-18T07:57:33.484 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Sep 18 07:57:33 smithi007 ceph-779349ca-7593-11ef-bceb-c7b262605968-mgr-y[29688]: File "/lib64/python3.9/ssl.py", line 1074, in _create 2024-09-18T07:57:33.484 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Sep 18 07:57:33 smithi007 ceph-779349ca-7593-11ef-bceb-c7b262605968-mgr-y[29688]: self.do_handshake() 2024-09-18T07:57:33.484 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Sep 18 07:57:33 smithi007 ceph-779349ca-7593-11ef-bceb-c7b262605968-mgr-y[29688]: File "/lib64/python3.9/ssl.py", line 1343, in do_handshake 2024-09-18T07:57:33.484 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Sep 18 07:57:33 smithi007 ceph-779349ca-7593-11ef-bceb-c7b262605968-mgr-y[29688]: self._sslobj.do_handshake() 2024-09-18T07:57:33.484 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Sep 18 07:57:33 smithi007 ceph-779349ca-7593-11ef-bceb-c7b262605968-mgr-y[29688]: ssl.SSLZeroReturnError: TLS/SSL connection has been closed (EOF) (_ssl.c:1133) 2024-09-18T07:57:33.484 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Sep 18 07:57:33 smithi007 ceph-779349ca-7593-11ef-bceb-c7b262605968-mgr-y[29688]: 2024-09-18T07:57:33.911 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 07:57:33 smithi007 ceph-mon[29459]: from='client.14122 -' entity='client.admin' cmd=[{"prefix": "get_command_descriptions"}]: dispatch 2024-09-18T07:57:33.912 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 07:57:33 smithi007 ceph-mon[29459]: from='client.14122 -' entity='client.admin' cmd=[{"prefix": "mgr_status"}]: dispatch 2024-09-18T07:57:33.912 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 07:57:33 smithi007 ceph-mon[29459]: from='client.? 172.21.15.7:0/4185277897' entity='client.admin' cmd={"prefix": "mgr module enable", "module": "orchestrator"} : dispatch 2024-09-18T07:57:33.912 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 07:57:33 smithi007 ceph-mon[29459]: from='mgr.14118 172.21.15.7:0/3020361701' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-18T07:57:34.164 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr module 'orchestrator' is already enabled (always-on) 2024-09-18T07:57:34.164 INFO:teuthology.orchestra.run.smithi007.stdout:Setting orchestrator backend to cephadm... 2024-09-18T07:57:34.731 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 07:57:34 smithi007 ceph-mon[29459]: [18/Sep/2024:07:57:33] ENGINE Bus STARTING 2024-09-18T07:57:34.732 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 07:57:34 smithi007 ceph-mon[29459]: [18/Sep/2024:07:57:33] ENGINE Serving on https://172.21.15.7:7150 2024-09-18T07:57:34.732 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 07:57:34 smithi007 ceph-mon[29459]: [18/Sep/2024:07:57:33] ENGINE Error in HTTPServer.serve 2024-09-18T07:57:34.732 INFO:journalctl@ceph.mon.a.smithi007.stdout: Traceback (most recent call last): 2024-09-18T07:57:34.732 INFO:journalctl@ceph.mon.a.smithi007.stdout: File "/lib/python3.9/site-packages/cheroot/server.py", line 1823, in serve 2024-09-18T07:57:34.732 INFO:journalctl@ceph.mon.a.smithi007.stdout: self._connections.run(self.expiration_interval) 2024-09-18T07:57:34.732 INFO:journalctl@ceph.mon.a.smithi007.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 203, in run 2024-09-18T07:57:34.733 INFO:journalctl@ceph.mon.a.smithi007.stdout: self._run(expiration_interval) 2024-09-18T07:57:34.733 INFO:journalctl@ceph.mon.a.smithi007.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 246, in _run 2024-09-18T07:57:34.733 INFO:journalctl@ceph.mon.a.smithi007.stdout: new_conn = self._from_server_socket(self.server.socket) 2024-09-18T07:57:34.733 INFO:journalctl@ceph.mon.a.smithi007.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 300, in _from_server_socket 2024-09-18T07:57:34.733 INFO:journalctl@ceph.mon.a.smithi007.stdout: s, ssl_env = self.server.ssl_adapter.wrap(s) 2024-09-18T07:57:34.734 INFO:journalctl@ceph.mon.a.smithi007.stdout: File "/lib/python3.9/site-packages/cheroot/ssl/builtin.py", line 277, in wrap 2024-09-18T07:57:34.734 INFO:journalctl@ceph.mon.a.smithi007.stdout: s = self.context.wrap_socket( 2024-09-18T07:57:34.734 INFO:journalctl@ceph.mon.a.smithi007.stdout: File "/lib64/python3.9/ssl.py", line 501, in wrap_socket 2024-09-18T07:57:34.734 INFO:journalctl@ceph.mon.a.smithi007.stdout: return self.sslsocket_class._create( 2024-09-18T07:57:34.734 INFO:journalctl@ceph.mon.a.smithi007.stdout: File "/lib64/python3.9/ssl.py", line 1074, in _create 2024-09-18T07:57:34.735 INFO:journalctl@ceph.mon.a.smithi007.stdout: self.do_handshake() 2024-09-18T07:57:34.735 INFO:journalctl@ceph.mon.a.smithi007.stdout: File "/lib64/python3.9/ssl.py", line 1343, in do_handshake 2024-09-18T07:57:34.735 INFO:journalctl@ceph.mon.a.smithi007.stdout: self._sslobj.do_handshake() 2024-09-18T07:57:34.735 INFO:journalctl@ceph.mon.a.smithi007.stdout: ssl.SSLZeroReturnError: TLS/SSL connection has been closed (EOF) (_ssl.c:1133) 2024-09-18T07:57:34.735 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 07:57:34 smithi007 ceph-mon[29459]: [18/Sep/2024:07:57:33] ENGINE Serving on http://172.21.15.7:8765 2024-09-18T07:57:34.735 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 07:57:34 smithi007 ceph-mon[29459]: [18/Sep/2024:07:57:33] ENGINE Bus STARTED 2024-09-18T07:57:34.736 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 07:57:34 smithi007 ceph-mon[29459]: from='client.? 172.21.15.7:0/4185277897' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "orchestrator"}]': finished 2024-09-18T07:57:34.736 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 07:57:34 smithi007 ceph-mon[29459]: mgrmap e8: y(active, since 2s) 2024-09-18T07:57:35.947 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 07:57:35 smithi007 ceph-mon[29459]: from='client.14132 -' entity='client.admin' cmd=[{"prefix": "orch set backend", "module_name": "cephadm", "target": ["mon-mgr", ""]}]: dispatch 2024-09-18T07:57:35.947 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 07:57:35 smithi007 ceph-mon[29459]: from='mgr.14118 172.21.15.7:0/3020361701' entity='mgr.y' 2024-09-18T07:57:35.947 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 07:57:35 smithi007 ceph-mon[29459]: from='mgr.14118 172.21.15.7:0/3020361701' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-18T07:57:36.426 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout value unchanged 2024-09-18T07:57:36.426 INFO:teuthology.orchestra.run.smithi007.stdout:Generating ssh key... 2024-09-18T07:57:36.981 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 07:57:36 smithi007 ceph-mon[29459]: from='client.14134 -' entity='client.admin' cmd=[{"prefix": "cephadm set-user", "user": "root", "target": ["mon-mgr", ""]}]: dispatch 2024-09-18T07:57:37.422 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Sep 18 07:57:37 smithi007 ceph-779349ca-7593-11ef-bceb-c7b262605968-mgr-y[29688]: Generating public/private rsa key pair. 2024-09-18T07:57:37.422 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Sep 18 07:57:37 smithi007 ceph-779349ca-7593-11ef-bceb-c7b262605968-mgr-y[29688]: Your identification has been saved in /tmp/tmpyjlxh0h6/key 2024-09-18T07:57:37.422 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Sep 18 07:57:37 smithi007 ceph-779349ca-7593-11ef-bceb-c7b262605968-mgr-y[29688]: Your public key has been saved in /tmp/tmpyjlxh0h6/key.pub 2024-09-18T07:57:37.422 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Sep 18 07:57:37 smithi007 ceph-779349ca-7593-11ef-bceb-c7b262605968-mgr-y[29688]: The key fingerprint is: 2024-09-18T07:57:37.422 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Sep 18 07:57:37 smithi007 ceph-779349ca-7593-11ef-bceb-c7b262605968-mgr-y[29688]: SHA256:J9naT5+rSRGOLjU5zBFVYSD/q1+pfib0HepIZNoJjiw ceph-779349ca-7593-11ef-bceb-c7b262605968 2024-09-18T07:57:37.422 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Sep 18 07:57:37 smithi007 ceph-779349ca-7593-11ef-bceb-c7b262605968-mgr-y[29688]: The key's randomart image is: 2024-09-18T07:57:37.422 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Sep 18 07:57:37 smithi007 ceph-779349ca-7593-11ef-bceb-c7b262605968-mgr-y[29688]: +---[RSA 3072]----+ 2024-09-18T07:57:37.422 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Sep 18 07:57:37 smithi007 ceph-779349ca-7593-11ef-bceb-c7b262605968-mgr-y[29688]: | o.oo+. | 2024-09-18T07:57:37.422 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Sep 18 07:57:37 smithi007 ceph-779349ca-7593-11ef-bceb-c7b262605968-mgr-y[29688]: | + . | 2024-09-18T07:57:37.423 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Sep 18 07:57:37 smithi007 ceph-779349ca-7593-11ef-bceb-c7b262605968-mgr-y[29688]: | . o | 2024-09-18T07:57:37.423 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Sep 18 07:57:37 smithi007 ceph-779349ca-7593-11ef-bceb-c7b262605968-mgr-y[29688]: | = = o | 2024-09-18T07:57:37.423 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Sep 18 07:57:37 smithi007 ceph-779349ca-7593-11ef-bceb-c7b262605968-mgr-y[29688]: | S.@oo . | 2024-09-18T07:57:37.423 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Sep 18 07:57:37 smithi007 ceph-779349ca-7593-11ef-bceb-c7b262605968-mgr-y[29688]: | . oB*o.o o.| 2024-09-18T07:57:37.423 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Sep 18 07:57:37 smithi007 ceph-779349ca-7593-11ef-bceb-c7b262605968-mgr-y[29688]: | E oooo++ +.+| 2024-09-18T07:57:37.423 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Sep 18 07:57:37 smithi007 ceph-779349ca-7593-11ef-bceb-c7b262605968-mgr-y[29688]: | . ..+.*.*.| 2024-09-18T07:57:37.423 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Sep 18 07:57:37 smithi007 ceph-779349ca-7593-11ef-bceb-c7b262605968-mgr-y[29688]: | .B*O. | 2024-09-18T07:57:37.423 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Sep 18 07:57:37 smithi007 ceph-779349ca-7593-11ef-bceb-c7b262605968-mgr-y[29688]: +----[SHA256]-----+ 2024-09-18T07:57:38.279 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 07:57:38 smithi007 ceph-mon[29459]: from='client.14136 -' entity='client.admin' cmd=[{"prefix": "cephadm generate-key", "target": ["mon-mgr", ""]}]: dispatch 2024-09-18T07:57:38.279 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 07:57:38 smithi007 ceph-mon[29459]: Generating ssh key... 2024-09-18T07:57:38.279 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 07:57:38 smithi007 ceph-mon[29459]: from='mgr.14118 172.21.15.7:0/3020361701' entity='mgr.y' 2024-09-18T07:57:38.279 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 07:57:38 smithi007 ceph-mon[29459]: from='mgr.14118 172.21.15.7:0/3020361701' entity='mgr.y' 2024-09-18T07:57:38.807 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDI4Rnqr9CXMBJpOzU0EcHomxnXZT9RT47XsWC/kgIeZ5fIU2dkNdbwk6yCJM+tYiGEDWctU4AzpZPmZ/Uryw42lJ5n7W1s3KZcsRXiR/eCOAMkFgfxDxBwfQkwc9UktmvopTA3O5VySikjunkS8L5FJgHRIJp/fIProuIYE3L8+d7qaIHOPXeTAsyrx8PtPheE+oQimtWTcM/Cd7LC2Vu0vxR/qa5UJrBpP0BShJCCaWVEYgBIwkeZldgLPi82aUOIewnXwkZUMpTJZPkQ8fA6Y1n0hEHYk0/KxyASSA68eAAMb+Z5PdPwu5cwp5V0WA3hC1jvqUxaAX7G59B+li+qcL10mvC2dIYtO5w92E1/nvCPXOgCCrJxsnUtd90PCjnrkPEE5ed1UOB9gcAcflOIHAfcsn99Pi2KZy6tVcWmJkPnJeIWcySIRBQMjSsY+T2dO5GuEGPTMLZjGvw7oIf9mQXibAnyteGBM3TgWEB7/NzuEd3PXl2k66ri0aFt76U= ceph-779349ca-7593-11ef-bceb-c7b262605968 2024-09-18T07:57:38.807 INFO:teuthology.orchestra.run.smithi007.stdout:Wrote public SSH key to /home/ubuntu/cephtest/ceph.pub 2024-09-18T07:57:38.807 INFO:teuthology.orchestra.run.smithi007.stdout:Adding key to root@localhost authorized_keys... 2024-09-18T07:57:38.814 INFO:teuthology.orchestra.run.smithi007.stdout:Adding host smithi007... 2024-09-18T07:57:39.207 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 07:57:39 smithi007 ceph-mon[29459]: from='client.14138 -' entity='client.admin' cmd=[{"prefix": "cephadm get-pub-key", "target": ["mon-mgr", ""]}]: dispatch 2024-09-18T07:57:40.399 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 07:57:40 smithi007 ceph-mon[29459]: from='client.14140 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi007", "addr": "172.21.15.7", "target": ["mon-mgr", ""]}]: dispatch 2024-09-18T07:57:41.231 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 07:57:41 smithi007 ceph-mon[29459]: Deploying cephadm binary to smithi007 2024-09-18T07:57:43.833 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout Added host 'smithi007' with addr '172.21.15.7' 2024-09-18T07:57:43.833 INFO:teuthology.orchestra.run.smithi007.stdout:Deploying unmanaged mon service... 2024-09-18T07:57:44.410 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 07:57:44 smithi007 ceph-mon[29459]: from='mgr.14118 172.21.15.7:0/3020361701' entity='mgr.y' 2024-09-18T07:57:44.411 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 07:57:44 smithi007 ceph-mon[29459]: Added host smithi007 2024-09-18T07:57:44.411 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 07:57:44 smithi007 ceph-mon[29459]: from='mgr.14118 172.21.15.7:0/3020361701' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-18T07:57:45.921 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 07:57:45 smithi007 ceph-mon[29459]: from='client.14142 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "unmanaged": true, "target": ["mon-mgr", ""]}]: dispatch 2024-09-18T07:57:45.921 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 07:57:45 smithi007 ceph-mon[29459]: Saving service mon spec with placement count:5 2024-09-18T07:57:45.921 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 07:57:45 smithi007 ceph-mon[29459]: from='mgr.14118 172.21.15.7:0/3020361701' entity='mgr.y' 2024-09-18T07:57:45.921 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 07:57:45 smithi007 ceph-mon[29459]: from='mgr.14118 172.21.15.7:0/3020361701' entity='mgr.y' 2024-09-18T07:57:45.923 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout Scheduled mon update... 2024-09-18T07:57:45.923 INFO:teuthology.orchestra.run.smithi007.stdout:Deploying unmanaged mgr service... 2024-09-18T07:57:47.198 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout Scheduled mgr update... 2024-09-18T07:57:47.305 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 07:57:46 smithi007 ceph-mon[29459]: from='mgr.14118 172.21.15.7:0/3020361701' entity='mgr.y' 2024-09-18T07:57:47.305 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 07:57:46 smithi007 ceph-mon[29459]: from='client.14144 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "unmanaged": true, "target": ["mon-mgr", ""]}]: dispatch 2024-09-18T07:57:47.305 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 07:57:46 smithi007 ceph-mon[29459]: Saving service mgr spec with placement count:2 2024-09-18T07:57:47.305 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 07:57:46 smithi007 ceph-mon[29459]: from='mgr.14118 172.21.15.7:0/3020361701' entity='mgr.y' 2024-09-18T07:57:48.941 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 07:57:48 smithi007 ceph-mon[29459]: from='client.? 172.21.15.7:0/2552883730' entity='client.admin' 2024-09-18T07:57:48.942 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 07:57:48 smithi007 ceph-mon[29459]: from='mgr.14118 172.21.15.7:0/3020361701' entity='mgr.y' 2024-09-18T07:57:48.942 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 07:57:48 smithi007 ceph-mon[29459]: from='mgr.14118 172.21.15.7:0/3020361701' entity='mgr.y' 2024-09-18T07:57:50.731 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 07:57:50 smithi007 ceph-mon[29459]: from='client.? 172.21.15.7:0/402955381' entity='client.admin' 2024-09-18T07:57:50.814 INFO:teuthology.orchestra.run.smithi007.stdout:Enabling the dashboard module... 2024-09-18T07:57:52.299 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 07:57:52 smithi007 ceph-mon[29459]: pgmap v3: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-18T07:57:52.300 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 07:57:52 smithi007 ceph-mon[29459]: from='client.? 172.21.15.7:0/3347724396' entity='client.admin' cmd={"prefix": "mgr module enable", "module": "dashboard"} : dispatch 2024-09-18T07:57:52.300 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Sep 18 07:57:52 smithi007 ceph-779349ca-7593-11ef-bceb-c7b262605968-mgr-y[29688]: ignoring --setuser ceph since I am not root 2024-09-18T07:57:52.300 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Sep 18 07:57:52 smithi007 ceph-779349ca-7593-11ef-bceb-c7b262605968-mgr-y[29688]: ignoring --setgroup ceph since I am not root 2024-09-18T07:57:53.231 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 07:57:53 smithi007 ceph-mon[29459]: from='client.? 172.21.15.7:0/3347724396' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "dashboard"}]': finished 2024-09-18T07:57:53.231 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 07:57:53 smithi007 ceph-mon[29459]: mgrmap e9: y(active, since 21s) 2024-09-18T07:57:53.870 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout { 2024-09-18T07:57:53.870 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout "epoch": 9, 2024-09-18T07:57:53.870 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout "available": true, 2024-09-18T07:57:53.870 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout "active_name": "y", 2024-09-18T07:57:53.870 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout "num_standby": 0 2024-09-18T07:57:53.870 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout } 2024-09-18T07:57:53.870 INFO:teuthology.orchestra.run.smithi007.stdout:Waiting for the mgr to restart... 2024-09-18T07:57:53.870 INFO:teuthology.orchestra.run.smithi007.stdout:Waiting for mgr epoch 9... 2024-09-18T07:57:54.302 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 07:57:54 smithi007 ceph-mon[29459]: from='client.? 172.21.15.7:0/961122588' entity='client.admin' cmd={"prefix": "mgr stat"} : dispatch 2024-09-18T07:57:57.731 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 07:57:57 smithi007 ceph-mon[29459]: Active manager daemon y restarted 2024-09-18T07:57:57.731 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 07:57:57 smithi007 ceph-mon[29459]: Activating manager daemon y 2024-09-18T07:57:57.731 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 07:57:57 smithi007 ceph-mon[29459]: osdmap e3: 0 total, 0 up, 0 in 2024-09-18T07:57:57.731 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 07:57:57 smithi007 ceph-mon[29459]: mgrmap e10: y(active, starting, since 0.00320467s) 2024-09-18T07:57:57.731 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 07:57:57 smithi007 ceph-mon[29459]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "a"} : dispatch 2024-09-18T07:57:57.731 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 07:57:57 smithi007 ceph-mon[29459]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' cmd={"prefix": "mgr metadata", "who": "y", "id": "y"} : dispatch 2024-09-18T07:57:57.731 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 07:57:57 smithi007 ceph-mon[29459]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' cmd={"prefix": "mds metadata"} : dispatch 2024-09-18T07:57:57.731 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 07:57:57 smithi007 ceph-mon[29459]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' cmd={"prefix": "osd metadata"} : dispatch 2024-09-18T07:57:57.731 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 07:57:57 smithi007 ceph-mon[29459]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' cmd={"prefix": "mon metadata"} : dispatch 2024-09-18T07:57:57.732 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 07:57:57 smithi007 ceph-mon[29459]: Manager daemon y is now available 2024-09-18T07:57:58.481 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 07:57:58 smithi007 ceph-mon[29459]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-18T07:57:58.481 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 07:57:58 smithi007 ceph-mon[29459]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' cmd={"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"} : dispatch 2024-09-18T07:57:58.481 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 07:57:58 smithi007 ceph-mon[29459]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' cmd={"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"} : dispatch 2024-09-18T07:57:58.481 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 07:57:58 smithi007 ceph-mon[29459]: mgrmap e11: y(active, since 1.00702s) 2024-09-18T07:57:59.069 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout { 2024-09-18T07:57:59.069 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout "mgrmap_epoch": 11, 2024-09-18T07:57:59.069 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout "initialized": true 2024-09-18T07:57:59.069 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout } 2024-09-18T07:57:59.069 INFO:teuthology.orchestra.run.smithi007.stdout:mgr epoch 9 is available 2024-09-18T07:57:59.069 INFO:teuthology.orchestra.run.smithi007.stdout:Using certmgr to generate dashboard self-signed certificate... 2024-09-18T07:57:59.369 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Sep 18 07:57:59 smithi007 ceph-779349ca-7593-11ef-bceb-c7b262605968-mgr-y[29688]: 2024-09-18T07:57:59.167+0000 7f7c77f83640 -1 log_channel(cephadm) log [ERR] : [18/Sep/2024:07:57:59] ENGINE Error in HTTPServer.serve 2024-09-18T07:57:59.369 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Sep 18 07:57:59 smithi007 ceph-779349ca-7593-11ef-bceb-c7b262605968-mgr-y[29688]: Traceback (most recent call last): 2024-09-18T07:57:59.369 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Sep 18 07:57:59 smithi007 ceph-779349ca-7593-11ef-bceb-c7b262605968-mgr-y[29688]: File "/lib/python3.9/site-packages/cheroot/server.py", line 1823, in serve 2024-09-18T07:57:59.369 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Sep 18 07:57:59 smithi007 ceph-779349ca-7593-11ef-bceb-c7b262605968-mgr-y[29688]: self._connections.run(self.expiration_interval) 2024-09-18T07:57:59.369 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Sep 18 07:57:59 smithi007 ceph-779349ca-7593-11ef-bceb-c7b262605968-mgr-y[29688]: File "/lib/python3.9/site-packages/cheroot/connections.py", line 203, in run 2024-09-18T07:57:59.369 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Sep 18 07:57:59 smithi007 ceph-779349ca-7593-11ef-bceb-c7b262605968-mgr-y[29688]: self._run(expiration_interval) 2024-09-18T07:57:59.369 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Sep 18 07:57:59 smithi007 ceph-779349ca-7593-11ef-bceb-c7b262605968-mgr-y[29688]: File "/lib/python3.9/site-packages/cheroot/connections.py", line 246, in _run 2024-09-18T07:57:59.369 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Sep 18 07:57:59 smithi007 ceph-779349ca-7593-11ef-bceb-c7b262605968-mgr-y[29688]: new_conn = self._from_server_socket(self.server.socket) 2024-09-18T07:57:59.370 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Sep 18 07:57:59 smithi007 ceph-779349ca-7593-11ef-bceb-c7b262605968-mgr-y[29688]: File "/lib/python3.9/site-packages/cheroot/connections.py", line 300, in _from_server_socket 2024-09-18T07:57:59.370 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Sep 18 07:57:59 smithi007 ceph-779349ca-7593-11ef-bceb-c7b262605968-mgr-y[29688]: s, ssl_env = self.server.ssl_adapter.wrap(s) 2024-09-18T07:57:59.370 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Sep 18 07:57:59 smithi007 ceph-779349ca-7593-11ef-bceb-c7b262605968-mgr-y[29688]: File "/lib/python3.9/site-packages/cheroot/ssl/builtin.py", line 277, in wrap 2024-09-18T07:57:59.370 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Sep 18 07:57:59 smithi007 ceph-779349ca-7593-11ef-bceb-c7b262605968-mgr-y[29688]: s = self.context.wrap_socket( 2024-09-18T07:57:59.370 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Sep 18 07:57:59 smithi007 ceph-779349ca-7593-11ef-bceb-c7b262605968-mgr-y[29688]: File "/lib64/python3.9/ssl.py", line 501, in wrap_socket 2024-09-18T07:57:59.370 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Sep 18 07:57:59 smithi007 ceph-779349ca-7593-11ef-bceb-c7b262605968-mgr-y[29688]: return self.sslsocket_class._create( 2024-09-18T07:57:59.370 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Sep 18 07:57:59 smithi007 ceph-779349ca-7593-11ef-bceb-c7b262605968-mgr-y[29688]: File "/lib64/python3.9/ssl.py", line 1074, in _create 2024-09-18T07:57:59.370 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Sep 18 07:57:59 smithi007 ceph-779349ca-7593-11ef-bceb-c7b262605968-mgr-y[29688]: self.do_handshake() 2024-09-18T07:57:59.370 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Sep 18 07:57:59 smithi007 ceph-779349ca-7593-11ef-bceb-c7b262605968-mgr-y[29688]: File "/lib64/python3.9/ssl.py", line 1343, in do_handshake 2024-09-18T07:57:59.370 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Sep 18 07:57:59 smithi007 ceph-779349ca-7593-11ef-bceb-c7b262605968-mgr-y[29688]: self._sslobj.do_handshake() 2024-09-18T07:57:59.371 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Sep 18 07:57:59 smithi007 ceph-779349ca-7593-11ef-bceb-c7b262605968-mgr-y[29688]: ssl.SSLZeroReturnError: TLS/SSL connection has been closed (EOF) (_ssl.c:1133) 2024-09-18T07:57:59.371 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Sep 18 07:57:59 smithi007 ceph-779349ca-7593-11ef-bceb-c7b262605968-mgr-y[29688]: 2024-09-18T07:57:59.731 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 07:57:59 smithi007 ceph-mon[29459]: [18/Sep/2024:07:57:58] ENGINE Bus STARTING 2024-09-18T07:57:59.731 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 07:57:59 smithi007 ceph-mon[29459]: [18/Sep/2024:07:57:59] ENGINE Serving on http://172.21.15.7:8765 2024-09-18T07:58:00.731 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 07:58:00 smithi007 ceph-mon[29459]: [18/Sep/2024:07:57:59] ENGINE Serving on https://172.21.15.7:7150 2024-09-18T07:58:00.731 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 07:58:00 smithi007 ceph-mon[29459]: [18/Sep/2024:07:57:59] ENGINE Bus STARTED 2024-09-18T07:58:00.731 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 07:58:00 smithi007 ceph-mon[29459]: [18/Sep/2024:07:57:59] ENGINE Error in HTTPServer.serve 2024-09-18T07:58:00.731 INFO:journalctl@ceph.mon.a.smithi007.stdout: Traceback (most recent call last): 2024-09-18T07:58:00.731 INFO:journalctl@ceph.mon.a.smithi007.stdout: File "/lib/python3.9/site-packages/cheroot/server.py", line 1823, in serve 2024-09-18T07:58:00.731 INFO:journalctl@ceph.mon.a.smithi007.stdout: self._connections.run(self.expiration_interval) 2024-09-18T07:58:00.731 INFO:journalctl@ceph.mon.a.smithi007.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 203, in run 2024-09-18T07:58:00.732 INFO:journalctl@ceph.mon.a.smithi007.stdout: self._run(expiration_interval) 2024-09-18T07:58:00.732 INFO:journalctl@ceph.mon.a.smithi007.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 246, in _run 2024-09-18T07:58:00.732 INFO:journalctl@ceph.mon.a.smithi007.stdout: new_conn = self._from_server_socket(self.server.socket) 2024-09-18T07:58:00.732 INFO:journalctl@ceph.mon.a.smithi007.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 300, in _from_server_socket 2024-09-18T07:58:00.732 INFO:journalctl@ceph.mon.a.smithi007.stdout: s, ssl_env = self.server.ssl_adapter.wrap(s) 2024-09-18T07:58:00.732 INFO:journalctl@ceph.mon.a.smithi007.stdout: File "/lib/python3.9/site-packages/cheroot/ssl/builtin.py", line 277, in wrap 2024-09-18T07:58:00.732 INFO:journalctl@ceph.mon.a.smithi007.stdout: s = self.context.wrap_socket( 2024-09-18T07:58:00.732 INFO:journalctl@ceph.mon.a.smithi007.stdout: File "/lib64/python3.9/ssl.py", line 501, in wrap_socket 2024-09-18T07:58:00.732 INFO:journalctl@ceph.mon.a.smithi007.stdout: return self.sslsocket_class._create( 2024-09-18T07:58:00.732 INFO:journalctl@ceph.mon.a.smithi007.stdout: File "/lib64/python3.9/ssl.py", line 1074, in _create 2024-09-18T07:58:00.732 INFO:journalctl@ceph.mon.a.smithi007.stdout: self.do_handshake() 2024-09-18T07:58:00.732 INFO:journalctl@ceph.mon.a.smithi007.stdout: File "/lib64/python3.9/ssl.py", line 1343, in do_handshake 2024-09-18T07:58:00.733 INFO:journalctl@ceph.mon.a.smithi007.stdout: self._sslobj.do_handshake() 2024-09-18T07:58:00.733 INFO:journalctl@ceph.mon.a.smithi007.stdout: ssl.SSLZeroReturnError: TLS/SSL connection has been closed (EOF) (_ssl.c:1133) 2024-09-18T07:58:00.733 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 07:58:00 smithi007 ceph-mon[29459]: mgrmap e12: y(active, since 2s) 2024-09-18T07:58:00.733 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 07:58:00 smithi007 ceph-mon[29459]: from='client.14164 -' entity='client.admin' cmd=[{"prefix": "orch certmgr generate-certificates", "module_name": "dashboard", "target": ["mon-mgr", ""]}]: dispatch 2024-09-18T07:58:00.733 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 07:58:00 smithi007 ceph-mon[29459]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' 2024-09-18T07:58:01.170 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout {"cert": "-----BEGIN CERTIFICATE-----\nMIIE+jCCAuKgAwIBAgIUDjOfbexmAVMt/zlRdOTCAwq6kicwDQYJKoZIhvcNAQEL\nBQAwFzEVMBMGA1UEAwwMY2VwaGFkbS1yb290MB4XDTI0MDkxODA3NTgwMFoXDTM0\nMDkxOTA3NTgwMFowFjEUMBIGA1UEAwwLMTcyLjIxLjE1LjcwggIiMA0GCSqGSIb3\nDQEBAQUAA4ICDwAwggIKAoICAQC1d3e5sQt3UZpMTvK6JQBpoc3HhqwvXD8Ykvvs\nmgZvG0a7jUt3+Splt34s0I8vUVVH1GQbCI3f2YSn+TzKe4cdln9zNWFKNHBAHB3k\nT98SCopiUUogik6bnnhljAu/TF2Y47wzfB9TknLIZfE8lEU87HwhJjVYE2WmFtly\nL/ReDYJWSTveQv0QLZ2tjWCl/WVHoMNByP9eo1heS44QUTudEYSpjISFySE7WQpy\nP0rOhJve0dkEAmcB6+2quZHY0KJYYxj1nELI7uGouEPXVwv2Tcw2Rbellc+gT0Pp\nK4dujMDUdYnPIQcEJhaiHUrEl52K4W4TP15Gi/NY+EbHzem0qJyMZKLx/XOKqVc0\nOGs7to01xQWPsUQugqNdQbjZhRxKb/DK1dHBXLy7w6UCkQjjX/iwcsfAKBzqgqn4\nppgu0tFE6g9sIw6LGvKrw4yeaHIj2D2rtJS5Sd4l3zG/ep/S9ywXNait/T6kfXQE\nGeD4M2lMXvYfgHrR8Jd6CcdlwuGv2GgMTIslV5m73Dn3t5zWNAdwCh9WzDaai2o9\neMw3gFsfwEic2gL98P7QAAam4LbjCNrPdT/JQfx8AHjGhjyfdaCsN0eE9ZvUcu6T\n90btyXGmOf+4Gbm9uheelXDTVYZK7vYocdEgijo6QOu/dzlT+Sb8Tq7NqyqyJXyJ\n/D7BpQIDAQABoz8wPTAtBgNVHREEJjAkgglzbWl0aGkwMDeCEWRhc2hib2FyZF9z\nZXJ2ZXJzhwSsFQ8HMAwGA1UdEwEB/wQCMAAwDQYJKoZIhvcNAQELBQADggIBADGV\n/VsGreKzz+gbI/x19ihzNMEcOnFVQfCSDIvKNkmAiMgj+1L/56eTVaNYCIhY5om1\n80qb96b3hfUuf1V0BMoNCZuc9d4PupuFmpGk9BijiCvnWrK87K5/qG6W/FbXvlzL\nH8dSa/lpbIVtYWNLjjl5v4t8Fc+YZGLObizREtzF6mqTwe89len6cwHAF7xn0qZh\nDaFHOQJTRDs8TG9G85tGjjNuUQsjGxkyDWmD1pVGFV+dsgUhsD3zkgAAtxvT0QL9\nAsuq0BELzkjl+6ZgE2O5bG9j7Cxhsu25N9PHRZIrnh80/t1dli6cVLcoVD0vvFrU\nSS6sdHl2JnhbncvHQ4d1pJ8r1om4X3l4iQaIuNVm9485wACDeQ2FfeID34tLeEDa\nvXucIDzI9yYZlw42h86kURPlXXc+yRNk3nEmGiA21Lw+KGV5bOG6J9RBLR3nec7s\n15yEr0ffWTB/Z/lV1DM2CsxbnK3adPQFds3qQ5BqtHI1nkin3fwvRJOOU31JlzZy\npJJlXkEtTS+eW72Y6KUOa9lCovlAblVRFyZLt4mj5xJ+2CanHnN2lYQEhDZCpB6x\nkttcaa8vJ8CBpQSpl/xbAmHnlPtagRaUkJ2nzuEvE5oG+FJI0/ppHb1DUBs1OoFz\noy9QOkMwTY/bruNGU6+v0cLYHhP6cUsVHpEZ3589\n-----END CERTIFICATE-----\n", "key": "-----BEGIN RSA PRIVATE KEY-----\nMIIJKQIBAAKCAgEAtXd3ubELd1GaTE7yuiUAaaHNx4asL1w/GJL77JoGbxtGu41L\nd/kqZbd+LNCPL1FVR9RkGwiN39mEp/k8ynuHHZZ/czVhSjRwQBwd5E/fEgqKYlFK\nIIpOm554ZYwLv0xdmOO8M3wfU5JyyGXxPJRFPOx8ISY1WBNlphbZci/0Xg2CVkk7\n3kL9EC2drY1gpf1lR6DDQcj/XqNYXkuOEFE7nRGEqYyEhckhO1kKcj9KzoSb3tHZ\nBAJnAevtqrmR2NCiWGMY9ZxCyO7hqLhD11cL9k3MNkW3pZXPoE9D6SuHbozA1HWJ\nzyEHBCYWoh1KxJediuFuEz9eRovzWPhGx83ptKicjGSi8f1ziqlXNDhrO7aNNcUF\nj7FELoKjXUG42YUcSm/wytXRwVy8u8OlApEI41/4sHLHwCgc6oKp+KaYLtLRROoP\nbCMOixryq8OMnmhyI9g9q7SUuUneJd8xv3qf0vcsFzWorf0+pH10BBng+DNpTF72\nH4B60fCXegnHZcLhr9hoDEyLJVeZu9w597ec1jQHcAofVsw2motqPXjMN4BbH8BI\nnNoC/fD+0AAGpuC24wjaz3U/yUH8fAB4xoY8n3WgrDdHhPWb1HLuk/dG7clxpjn/\nuBm5vboXnpVw01WGSu72KHHRIIo6OkDrv3c5U/km/E6uzasqsiV8ifw+waUCAwEA\nAQKCAgAEJSN1+WZSpmmI3rDKsbetsgV6Tkg2mLvUOt9IS0hkV1IpTJPxvJ5MToGy\nUl1C7m+tQVu8Blj/SP+vupC561JtyZdFsW10fXgm7r18DpsA1X7Nuh+gWKhtA5Sq\ndSV7jCoDKQiOgIJSrWCz1ShKgvSF7Lz1IONNDLxgq9rxsci6MULN58RXUl5ndTNJ\n2JXhlMUweqqbIRU8Oyz0knM/Siu7bcADA3OvfqWkrmb/rxRtXTgWQeLcJZXmxFXc\ndXqwYiujKzhRu+0Gm/ZJnVPnYlSinqpQfw0v2hkxBz584AKq8sog0EKCmbWIUbpo\nn5Sqou/1USrrg6IKqH25hYpQXBK/hNW1oKcNOlrloua0+DIaip1r8RvkWVT6GYiN\nX80wYiZzha08Rc6e1SJkpA8/26H33Iiqa89NcHCoxgSiARfcPCo5hxOO9wbAnVXl\nZD5lt0yvdvG+xGdI+L/dT5HAwrDjp5zpkvTqkh5IxKFtlhlV1yzWeNIif23fcVDH\nTd0exmT5O4vXxiN4ca6Oy5JJi7L/UN++iy2vvd2jTRH1kcxmaNO1CsACFxNeg/4P\nC06GLpEOi+p5s+KW+zvj8Rt1Ok7qy6zSkx75fK+9PJ5Flx2/RSXWZ7ab69Qav0yV\nB3QWyCq6tgRVvggL6p0oXKLKnfulnIi2PFoNLbHIwbPFl2OYDQKCAQEA17wWh2TC\ngAY9F2Z6rCEeCEUB246Mo6qI82zrnuA2DlY0HzDuY+7FKU09MJzUE+2YsIkFwgZ5\nWlttQL2K0plobE2qfKk7ilPLgXCMzuWoBq/GUTbWdo1UhyvdmfSylHpQPepB1s72\nj9fdKjORZbaWq6EtrlA7xtg6rSdy0BlDMEu1IN+OkjRksuJ4gGiLlrwdwd8LTsxN\nMVR32dZQ0DSn9YhSkFPA0fuj5bXf1E/eJ1CXwZMF+rJ/L+jau6e5xLt/t70C4cen\nTtGgs+w4WGYRfKyyyEk4yInIQMKTqBnp4As+IrMZ62nR0dDaB06bIFqf9WvkVXJ1\n6xcHYER+sYZpVwKCAQEA11YJCnGt94F7xuTygrNUyzBWXW/sGYPa06z3/QriKog8\n8Qaq2FbR1SvQPTRQ4bB/cO2JbvbCfK5MvwhMswOB0leJGp3CtYHdK0qW+tg4RaUp\nxm9eXagT3hX4VdC+19d4uCJV/c3/aoTlRCtgfJnjlKmR1lZQ5zsnDK1QemuA4aKG\n2utHhXSUU3yGiwCZykEqs6TQ/lIGbrK80om5vUoO56lE0LIxhXh1V/g3P3QtmpPD\nVMlOSnLOSlxHUZLqmBhy6/S4OK0UQuHQf/20epEavGtrOiH0Co5tn8C41juuaspr\nPPLlFAjrbCELGRiRI7F0ihafZs2cBrW0xtawoTgDYwKCAQBgMWfCOzVWrHF1C6bL\nNYEUouBvK6KeH54tHHX7Q3IW9m0LM2A4gvC6t8v+7w7JS/n3TUGaABUAKyAN6/YI\nbYrYL9FDAJqh1Tc6ohrG7jP7t0iwUclS1zXHh9Gpoig2eEtrMBbnVyUtgk0FUjUU\nq1OHd7jMrs3YnfWb26kgDQOed0NuQwBpRJ9Z/A1LBFKphXunRHLgUcxhoI5IPX2P\nOMtjecOGwzGI77Fm7a++FVazJQZls87It+pqTGOrpN/6TrhmDwtH7SQ2ytXjJAMD\nt484jjCxiaSeFoGKDriUDenwklLEWiPGikqaSYvOEkitJnuIjPElAIJ/rqcFT8VG\nFu7FAoIBAQDR9sPt+RZmBU1JjCeipXWCgelURjNyVYzjZP6RLzGDwz+YzKV9OHp1\nXrk3VHGg/lXlKNge6GCO7e1LP655xmupS00WNMmL7cpX0e0RPg7wabEoGaNrst1c\naROxqU7Mav78nA444cmwdVCPfbQhCnRLmmPnGFbegGD9DWkxVO1gFPdlMpCUYi3W\nkc8mbjmGarBWZdiagiRff8x5hW0kUrfhzryYuYdr2x6f3hkQuBlXZcjKPVcnZHYd\nL4ki0xwHnZ9Az2dpa+obpM+GXTzDXgUxv5PYJQ8t0960LFiOG7dfzWtvDG+tb10O\nG9IUcE3k/bzigFiBBu6h5D8WunOTX2h9AoIBAQC8V7hXrg69c0VyVLclWWDlZltp\n/PmT88o9gkuuZvBPmEudy2N0yyBCyasnOaeagsVQcatOZH12PL4mSOq4WMAydXNX\n7rfU6TyFJ74AkdCsJXnviVKUDkqt0jro0GBr8qTOQbybdRUJcvY1FHAbrOSpN2yT\n5+08fJpnSP/jhfMe7YuaIWYGVFvdV7YWt+OnjLXoJFYwCajEEG+vXLqiOpssEK1p\noSbVPGZeQ1tisZQbcS+r54slEs1ORiU4nQTM5ZPR5bWwRl9HMop78WLQFZ6D9bpS\nF1+flnRljj3OEnyRjyJdnd74qlxfyW5Fi0KufdNeX6dyjG41gelEeAn902E0\n-----END RSA PRIVATE KEY-----\n"} 2024-09-18T07:58:02.138 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 07:58:01 smithi007 ceph-mon[29459]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' 2024-09-18T07:58:02.138 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 07:58:01 smithi007 ceph-mon[29459]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' 2024-09-18T07:58:03.089 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 07:58:02 smithi007 ceph-mon[29459]: from='client.14166 -' entity='client.admin' cmd=[{"prefix": "dashboard set-ssl-certificate", "target": ["mon-mgr", ""]}]: dispatch 2024-09-18T07:58:03.092 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout SSL certificate updated 2024-09-18T07:58:04.284 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout SSL certificate key updated 2024-09-18T07:58:04.284 INFO:teuthology.orchestra.run.smithi007.stdout:Creating initial admin user... 2024-09-18T07:58:04.981 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 07:58:04 smithi007 ceph-mon[29459]: from='client.14168 -' entity='client.admin' cmd=[{"prefix": "dashboard set-ssl-certificate-key", "target": ["mon-mgr", ""]}]: dispatch 2024-09-18T07:58:04.981 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 07:58:04 smithi007 ceph-mon[29459]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' 2024-09-18T07:58:05.716 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout {"username": "admin", "password": "$2b$12$hfbX6XgdZ/gRILB1wyGkfe3SnK1efusvIfmkR2UtWF5mUWgc3vNsO", "roles": ["administrator"], "name": null, "email": null, "lastUpdate": 1726646285, "enabled": true, "pwdExpirationDate": null, "pwdUpdateRequired": true} 2024-09-18T07:58:05.717 INFO:teuthology.orchestra.run.smithi007.stdout:Fetching dashboard port number... 2024-09-18T07:58:05.965 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 07:58:05 smithi007 ceph-mon[29459]: from='client.14170 -' entity='client.admin' cmd=[{"prefix": "dashboard ac-user-create", "username": "admin", "rolename": "administrator", "force_password": true, "pwd_update_required": true, "target": ["mon-mgr", ""]}]: dispatch 2024-09-18T07:58:05.965 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 07:58:05 smithi007 ceph-mon[29459]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' 2024-09-18T07:58:06.880 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout 8443 2024-09-18T07:58:06.890 INFO:teuthology.orchestra.run.smithi007.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-09-18T07:58:06.890 INFO:teuthology.orchestra.run.smithi007.stdout:systemctl: stdout disabled 2024-09-18T07:58:06.898 INFO:teuthology.orchestra.run.smithi007.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-09-18T07:58:06.898 INFO:teuthology.orchestra.run.smithi007.stdout:systemctl: stdout inactive 2024-09-18T07:58:06.898 INFO:teuthology.orchestra.run.smithi007.stdout:firewalld.service is not enabled 2024-09-18T07:58:06.898 INFO:teuthology.orchestra.run.smithi007.stdout:Not possible to open ports <[8443]>. firewalld.service is not available 2024-09-18T07:58:06.899 INFO:teuthology.orchestra.run.smithi007.stdout:Ceph Dashboard is now available at: 2024-09-18T07:58:06.900 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-09-18T07:58:06.900 INFO:teuthology.orchestra.run.smithi007.stdout: URL: https://smithi007.front.sepia.ceph.com:8443/ 2024-09-18T07:58:06.900 INFO:teuthology.orchestra.run.smithi007.stdout: User: admin 2024-09-18T07:58:06.900 INFO:teuthology.orchestra.run.smithi007.stdout: Password: zn9onk6u0b 2024-09-18T07:58:06.900 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-09-18T07:58:06.900 INFO:teuthology.orchestra.run.smithi007.stdout:Saving cluster configuration to /var/lib/ceph/779349ca-7593-11ef-bceb-c7b262605968/config directory 2024-09-18T07:58:06.981 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 07:58:06 smithi007 ceph-mon[29459]: from='client.? 172.21.15.7:0/1400350342' entity='client.admin' cmd={"prefix": "config get", "who": "mgr", "key": "mgr/dashboard/ssl_server_port"} : dispatch 2024-09-18T07:58:08.163 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr set mgr/dashboard/cluster/status 2024-09-18T07:58:08.163 INFO:teuthology.orchestra.run.smithi007.stdout:You can access the Ceph CLI as following in case of multi-cluster or non-default config: 2024-09-18T07:58:08.163 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-09-18T07:58:08.163 INFO:teuthology.orchestra.run.smithi007.stdout: sudo /home/ubuntu/cephtest/cephadm shell --fsid 779349ca-7593-11ef-bceb-c7b262605968 -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring 2024-09-18T07:58:08.163 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-09-18T07:58:08.163 INFO:teuthology.orchestra.run.smithi007.stdout:Or, if you are only running a single cluster on this host: 2024-09-18T07:58:08.163 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-09-18T07:58:08.163 INFO:teuthology.orchestra.run.smithi007.stdout: sudo /home/ubuntu/cephtest/cephadm shell 2024-09-18T07:58:08.164 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-09-18T07:58:08.164 INFO:teuthology.orchestra.run.smithi007.stdout:Please consider enabling telemetry to help improve Ceph: 2024-09-18T07:58:08.164 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-09-18T07:58:08.164 INFO:teuthology.orchestra.run.smithi007.stdout: ceph telemetry on 2024-09-18T07:58:08.164 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-09-18T07:58:08.164 INFO:teuthology.orchestra.run.smithi007.stdout:For more information see: 2024-09-18T07:58:08.164 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-09-18T07:58:08.164 INFO:teuthology.orchestra.run.smithi007.stdout: https://docs.ceph.com/en/latest/mgr/telemetry/ 2024-09-18T07:58:08.164 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-09-18T07:58:08.164 INFO:teuthology.orchestra.run.smithi007.stdout:Bootstrap complete. 2024-09-18T07:58:08.203 INFO:tasks.cephadm:Fetching config... 2024-09-18T07:58:08.203 DEBUG:teuthology.orchestra.run.smithi007:> set -ex 2024-09-18T07:58:08.203 DEBUG:teuthology.orchestra.run.smithi007:> dd if=/etc/ceph/ceph.conf of=/dev/stdout 2024-09-18T07:58:08.220 INFO:tasks.cephadm:Fetching client.admin keyring... 2024-09-18T07:58:08.221 DEBUG:teuthology.orchestra.run.smithi007:> set -ex 2024-09-18T07:58:08.221 DEBUG:teuthology.orchestra.run.smithi007:> dd if=/etc/ceph/ceph.client.admin.keyring of=/dev/stdout 2024-09-18T07:58:08.278 INFO:tasks.cephadm:Fetching mon keyring... 2024-09-18T07:58:08.278 DEBUG:teuthology.orchestra.run.smithi007:> set -ex 2024-09-18T07:58:08.278 DEBUG:teuthology.orchestra.run.smithi007:> sudo dd if=/var/lib/ceph/779349ca-7593-11ef-bceb-c7b262605968/mon.a/keyring of=/dev/stdout 2024-09-18T07:58:08.345 INFO:tasks.cephadm:Fetching pub ssh key... 2024-09-18T07:58:08.345 DEBUG:teuthology.orchestra.run.smithi007:> set -ex 2024-09-18T07:58:08.345 DEBUG:teuthology.orchestra.run.smithi007:> dd if=/home/ubuntu/cephtest/ceph.pub of=/dev/stdout 2024-09-18T07:58:08.402 INFO:tasks.cephadm:Installing pub ssh key for root users... 2024-09-18T07:58:08.403 DEBUG:teuthology.orchestra.run.smithi007:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDI4Rnqr9CXMBJpOzU0EcHomxnXZT9RT47XsWC/kgIeZ5fIU2dkNdbwk6yCJM+tYiGEDWctU4AzpZPmZ/Uryw42lJ5n7W1s3KZcsRXiR/eCOAMkFgfxDxBwfQkwc9UktmvopTA3O5VySikjunkS8L5FJgHRIJp/fIProuIYE3L8+d7qaIHOPXeTAsyrx8PtPheE+oQimtWTcM/Cd7LC2Vu0vxR/qa5UJrBpP0BShJCCaWVEYgBIwkeZldgLPi82aUOIewnXwkZUMpTJZPkQ8fA6Y1n0hEHYk0/KxyASSA68eAAMb+Z5PdPwu5cwp5V0WA3hC1jvqUxaAX7G59B+li+qcL10mvC2dIYtO5w92E1/nvCPXOgCCrJxsnUtd90PCjnrkPEE5ed1UOB9gcAcflOIHAfcsn99Pi2KZy6tVcWmJkPnJeIWcySIRBQMjSsY+T2dO5GuEGPTMLZjGvw7oIf9mQXibAnyteGBM3TgWEB7/NzuEd3PXl2k66ri0aFt76U= ceph-779349ca-7593-11ef-bceb-c7b262605968' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2024-09-18T07:58:08.484 INFO:teuthology.orchestra.run.smithi007.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDI4Rnqr9CXMBJpOzU0EcHomxnXZT9RT47XsWC/kgIeZ5fIU2dkNdbwk6yCJM+tYiGEDWctU4AzpZPmZ/Uryw42lJ5n7W1s3KZcsRXiR/eCOAMkFgfxDxBwfQkwc9UktmvopTA3O5VySikjunkS8L5FJgHRIJp/fIProuIYE3L8+d7qaIHOPXeTAsyrx8PtPheE+oQimtWTcM/Cd7LC2Vu0vxR/qa5UJrBpP0BShJCCaWVEYgBIwkeZldgLPi82aUOIewnXwkZUMpTJZPkQ8fA6Y1n0hEHYk0/KxyASSA68eAAMb+Z5PdPwu5cwp5V0WA3hC1jvqUxaAX7G59B+li+qcL10mvC2dIYtO5w92E1/nvCPXOgCCrJxsnUtd90PCjnrkPEE5ed1UOB9gcAcflOIHAfcsn99Pi2KZy6tVcWmJkPnJeIWcySIRBQMjSsY+T2dO5GuEGPTMLZjGvw7oIf9mQXibAnyteGBM3TgWEB7/NzuEd3PXl2k66ri0aFt76U= ceph-779349ca-7593-11ef-bceb-c7b262605968 2024-09-18T07:58:08.496 DEBUG:teuthology.orchestra.run.smithi050:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDI4Rnqr9CXMBJpOzU0EcHomxnXZT9RT47XsWC/kgIeZ5fIU2dkNdbwk6yCJM+tYiGEDWctU4AzpZPmZ/Uryw42lJ5n7W1s3KZcsRXiR/eCOAMkFgfxDxBwfQkwc9UktmvopTA3O5VySikjunkS8L5FJgHRIJp/fIProuIYE3L8+d7qaIHOPXeTAsyrx8PtPheE+oQimtWTcM/Cd7LC2Vu0vxR/qa5UJrBpP0BShJCCaWVEYgBIwkeZldgLPi82aUOIewnXwkZUMpTJZPkQ8fA6Y1n0hEHYk0/KxyASSA68eAAMb+Z5PdPwu5cwp5V0WA3hC1jvqUxaAX7G59B+li+qcL10mvC2dIYtO5w92E1/nvCPXOgCCrJxsnUtd90PCjnrkPEE5ed1UOB9gcAcflOIHAfcsn99Pi2KZy6tVcWmJkPnJeIWcySIRBQMjSsY+T2dO5GuEGPTMLZjGvw7oIf9mQXibAnyteGBM3TgWEB7/NzuEd3PXl2k66ri0aFt76U= ceph-779349ca-7593-11ef-bceb-c7b262605968' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2024-09-18T07:58:08.548 INFO:teuthology.orchestra.run.smithi050.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDI4Rnqr9CXMBJpOzU0EcHomxnXZT9RT47XsWC/kgIeZ5fIU2dkNdbwk6yCJM+tYiGEDWctU4AzpZPmZ/Uryw42lJ5n7W1s3KZcsRXiR/eCOAMkFgfxDxBwfQkwc9UktmvopTA3O5VySikjunkS8L5FJgHRIJp/fIProuIYE3L8+d7qaIHOPXeTAsyrx8PtPheE+oQimtWTcM/Cd7LC2Vu0vxR/qa5UJrBpP0BShJCCaWVEYgBIwkeZldgLPi82aUOIewnXwkZUMpTJZPkQ8fA6Y1n0hEHYk0/KxyASSA68eAAMb+Z5PdPwu5cwp5V0WA3hC1jvqUxaAX7G59B+li+qcL10mvC2dIYtO5w92E1/nvCPXOgCCrJxsnUtd90PCjnrkPEE5ed1UOB9gcAcflOIHAfcsn99Pi2KZy6tVcWmJkPnJeIWcySIRBQMjSsY+T2dO5GuEGPTMLZjGvw7oIf9mQXibAnyteGBM3TgWEB7/NzuEd3PXl2k66ri0aFt76U= ceph-779349ca-7593-11ef-bceb-c7b262605968 2024-09-18T07:58:08.562 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 779349ca-7593-11ef-bceb-c7b262605968 -- ceph config set mgr mgr/cephadm/allow_ptrace true 2024-09-18T07:58:08.731 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 07:58:08 smithi007 ceph-mon[29459]: from='client.? 172.21.15.7:0/1925844589' entity='client.admin' 2024-09-18T07:58:08.822 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/779349ca-7593-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T07:58:10.133 INFO:tasks.cephadm:Distributing conf and client.admin keyring to all hosts + 0755 2024-09-18T07:58:10.133 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 779349ca-7593-11ef-bceb-c7b262605968 -- ceph orch client-keyring set client.admin '*' --mode 0755 2024-09-18T07:58:10.573 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/779349ca-7593-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T07:58:10.689 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 07:58:10 smithi007 ceph-mon[29459]: from='client.? 172.21.15.7:0/429654664' entity='client.admin' 2024-09-18T07:58:10.689 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 07:58:10 smithi007 ceph-mon[29459]: mgrmap e13: y(active, since 12s) 2024-09-18T07:58:12.683 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 07:58:12 smithi007 ceph-mon[29459]: from='client.14178 -' entity='client.admin' cmd=[{"prefix": "orch client-keyring set", "entity": "client.admin", "placement": "*", "mode": "0755", "target": ["mon-mgr", ""]}]: dispatch 2024-09-18T07:58:12.683 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 07:58:12 smithi007 ceph-mon[29459]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' 2024-09-18T07:58:12.683 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 07:58:12 smithi007 ceph-mon[29459]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' 2024-09-18T07:58:12.683 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 07:58:12 smithi007 ceph-mon[29459]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' 2024-09-18T07:58:12.683 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 07:58:12 smithi007 ceph-mon[29459]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd/host:smithi007", "name": "osd_memory_target"} : dispatch 2024-09-18T07:58:12.683 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 07:58:12 smithi007 ceph-mon[29459]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-18T07:58:12.683 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 07:58:12 smithi007 ceph-mon[29459]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-18T07:58:12.714 INFO:tasks.cephadm:Writing (initial) conf and keyring to smithi050 2024-09-18T07:58:12.714 DEBUG:teuthology.orchestra.run.smithi050:> set -ex 2024-09-18T07:58:12.714 DEBUG:teuthology.orchestra.run.smithi050:> dd of=/etc/ceph/ceph.conf 2024-09-18T07:58:12.734 DEBUG:teuthology.orchestra.run.smithi050:> set -ex 2024-09-18T07:58:12.734 DEBUG:teuthology.orchestra.run.smithi050:> dd of=/etc/ceph/ceph.client.admin.keyring 2024-09-18T07:58:12.793 INFO:tasks.cephadm:Adding host smithi050 to orchestrator... 2024-09-18T07:58:12.794 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 779349ca-7593-11ef-bceb-c7b262605968 -- ceph orch host add smithi050 2024-09-18T07:58:13.049 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/779349ca-7593-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T07:58:14.231 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 07:58:13 smithi007 ceph-mon[29459]: Updating smithi007:/etc/ceph/ceph.conf 2024-09-18T07:58:14.231 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 07:58:13 smithi007 ceph-mon[29459]: Updating smithi007:/var/lib/ceph/779349ca-7593-11ef-bceb-c7b262605968/config/ceph.conf 2024-09-18T07:58:14.231 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 07:58:13 smithi007 ceph-mon[29459]: Updating smithi007:/etc/ceph/ceph.client.admin.keyring 2024-09-18T07:58:14.231 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 07:58:13 smithi007 ceph-mon[29459]: Updating smithi007:/var/lib/ceph/779349ca-7593-11ef-bceb-c7b262605968/config/ceph.client.admin.keyring 2024-09-18T07:58:14.231 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 07:58:13 smithi007 ceph-mon[29459]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' 2024-09-18T07:58:14.231 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 07:58:13 smithi007 ceph-mon[29459]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' 2024-09-18T07:58:14.231 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 07:58:13 smithi007 ceph-mon[29459]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' 2024-09-18T07:58:14.232 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 07:58:13 smithi007 ceph-mon[29459]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-18T07:58:14.232 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 07:58:13 smithi007 ceph-mon[29459]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-18T07:58:14.232 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 07:58:13 smithi007 ceph-mon[29459]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-18T07:58:14.232 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 07:58:13 smithi007 ceph-mon[29459]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' 2024-09-18T07:58:15.233 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 07:58:14 smithi007 ceph-mon[29459]: from='client.14180 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi050", "target": ["mon-mgr", ""]}]: dispatch 2024-09-18T07:58:16.231 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 07:58:15 smithi007 ceph-mon[29459]: Deploying cephadm binary to smithi050 2024-09-18T07:58:17.482 INFO:teuthology.orchestra.run.smithi007.stdout:Added host 'smithi050' with addr '172.21.15.50' 2024-09-18T07:58:18.087 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 779349ca-7593-11ef-bceb-c7b262605968 -- ceph orch host ls --format=json 2024-09-18T07:58:18.339 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/779349ca-7593-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T07:58:18.714 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 07:58:18 smithi007 ceph-mon[29459]: pgmap v3: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-18T07:58:18.715 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 07:58:18 smithi007 ceph-mon[29459]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' 2024-09-18T07:58:18.715 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 07:58:18 smithi007 ceph-mon[29459]: Added host smithi050 2024-09-18T07:58:18.715 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 07:58:18 smithi007 ceph-mon[29459]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-18T07:58:18.715 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 07:58:18 smithi007 ceph-mon[29459]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' 2024-09-18T07:58:18.715 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 07:58:18 smithi007 ceph-mon[29459]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' 2024-09-18T07:58:18.993 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-09-18T07:58:18.993 INFO:teuthology.orchestra.run.smithi007.stdout:[{"addr": "172.21.15.7", "hostname": "smithi007", "labels": [], "status": ""}, {"addr": "172.21.15.50", "hostname": "smithi050", "labels": [], "status": ""}] 2024-09-18T07:58:19.552 INFO:tasks.cephadm:Setting crush tunables to default 2024-09-18T07:58:19.552 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 779349ca-7593-11ef-bceb-c7b262605968 -- ceph osd crush tunables default 2024-09-18T07:58:19.802 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/779349ca-7593-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T07:58:20.125 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 07:58:19 smithi007 ceph-mon[29459]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' 2024-09-18T07:58:20.126 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 07:58:19 smithi007 ceph-mon[29459]: from='client.14182 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-18T07:58:20.814 INFO:teuthology.orchestra.run.smithi007.stderr:adjusted tunables profile to default 2024-09-18T07:58:21.120 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 07:58:20 smithi007 ceph-mon[29459]: pgmap v4: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-18T07:58:21.120 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 07:58:20 smithi007 ceph-mon[29459]: from='client.? 172.21.15.7:0/118748185' entity='client.admin' cmd={"prefix": "osd crush tunables", "profile": "default"} : dispatch 2024-09-18T07:58:21.422 INFO:tasks.cephadm:Adding mon.a on smithi007 2024-09-18T07:58:21.422 INFO:tasks.cephadm:Adding mon.c on smithi007 2024-09-18T07:58:21.422 INFO:tasks.cephadm:Adding mon.b on smithi050 2024-09-18T07:58:21.422 DEBUG:teuthology.orchestra.run.smithi050:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 779349ca-7593-11ef-bceb-c7b262605968 -- ceph orch apply mon '3;smithi007:172.21.15.7=a;smithi007:[v2:172.21.15.7:3301,v1:172.21.15.7:6790]=c;smithi050:172.21.15.50=b' 2024-09-18T07:58:21.674 INFO:teuthology.orchestra.run.smithi050.stderr:Inferring config /etc/ceph/ceph.conf 2024-09-18T07:58:21.721 INFO:teuthology.orchestra.run.smithi050.stderr:Inferring config /etc/ceph/ceph.conf 2024-09-18T07:58:22.231 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 07:58:21 smithi007 ceph-mon[29459]: from='client.? 172.21.15.7:0/118748185' entity='client.admin' cmd='[{"prefix": "osd crush tunables", "profile": "default"}]': finished 2024-09-18T07:58:22.231 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 07:58:21 smithi007 ceph-mon[29459]: osdmap e4: 0 total, 0 up, 0 in 2024-09-18T07:58:22.231 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 07:58:21 smithi007 ceph-mon[29459]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' 2024-09-18T07:58:22.231 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 07:58:21 smithi007 ceph-mon[29459]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' 2024-09-18T07:58:22.231 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 07:58:21 smithi007 ceph-mon[29459]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' 2024-09-18T07:58:22.231 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 07:58:21 smithi007 ceph-mon[29459]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' 2024-09-18T07:58:22.231 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 07:58:21 smithi007 ceph-mon[29459]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd/host:smithi050", "name": "osd_memory_target"} : dispatch 2024-09-18T07:58:22.231 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 07:58:21 smithi007 ceph-mon[29459]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-18T07:58:22.231 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 07:58:21 smithi007 ceph-mon[29459]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-18T07:58:22.420 INFO:teuthology.orchestra.run.smithi050.stdout:Scheduled mon update... 2024-09-18T07:58:23.003 DEBUG:teuthology.orchestra.run.smithi007:mon.c> sudo journalctl -f -n 0 -u ceph-779349ca-7593-11ef-bceb-c7b262605968@mon.c.service 2024-09-18T07:58:23.007 DEBUG:teuthology.orchestra.run.smithi050:mon.b> sudo journalctl -f -n 0 -u ceph-779349ca-7593-11ef-bceb-c7b262605968@mon.b.service 2024-09-18T07:58:23.010 INFO:tasks.cephadm:Waiting for 3 mons in monmap... 2024-09-18T07:58:23.010 DEBUG:teuthology.orchestra.run.smithi050:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 779349ca-7593-11ef-bceb-c7b262605968 -- ceph mon dump -f json 2024-09-18T07:58:23.231 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 07:58:22 smithi007 ceph-mon[29459]: pgmap v6: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-18T07:58:23.231 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 07:58:22 smithi007 ceph-mon[29459]: Updating smithi050:/etc/ceph/ceph.conf 2024-09-18T07:58:23.231 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 07:58:22 smithi007 ceph-mon[29459]: Updating smithi050:/var/lib/ceph/779349ca-7593-11ef-bceb-c7b262605968/config/ceph.conf 2024-09-18T07:58:23.231 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 07:58:22 smithi007 ceph-mon[29459]: Updating smithi050:/etc/ceph/ceph.client.admin.keyring 2024-09-18T07:58:23.231 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 07:58:22 smithi007 ceph-mon[29459]: Updating smithi050:/var/lib/ceph/779349ca-7593-11ef-bceb-c7b262605968/config/ceph.client.admin.keyring 2024-09-18T07:58:23.231 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 07:58:22 smithi007 ceph-mon[29459]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' 2024-09-18T07:58:23.231 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 07:58:22 smithi007 ceph-mon[29459]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' 2024-09-18T07:58:23.232 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 07:58:22 smithi007 ceph-mon[29459]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' 2024-09-18T07:58:23.232 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 07:58:22 smithi007 ceph-mon[29459]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' 2024-09-18T07:58:23.232 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 07:58:22 smithi007 ceph-mon[29459]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-18T07:58:23.232 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 07:58:22 smithi007 ceph-mon[29459]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-18T07:58:23.232 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 07:58:22 smithi007 ceph-mon[29459]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-18T07:58:23.232 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 07:58:22 smithi007 ceph-mon[29459]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' 2024-09-18T07:58:23.232 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 07:58:22 smithi007 ceph-mon[29459]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' cmd={"prefix": "auth get", "entity": "mon."} : dispatch 2024-09-18T07:58:23.232 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 07:58:22 smithi007 ceph-mon[29459]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-18T07:58:24.231 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 07:58:23 smithi007 ceph-mon[29459]: from='client.14186 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "placement": "3;smithi007:172.21.15.7=a;smithi007:[v2:172.21.15.7:3301,v1:172.21.15.7:6790]=c;smithi050:172.21.15.50=b", "target": ["mon-mgr", ""]}]: dispatch 2024-09-18T07:58:24.231 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 07:58:23 smithi007 ceph-mon[29459]: Saving service mon spec with placement smithi007:172.21.15.7=a;smithi007:[v2:172.21.15.7:3301,v1:172.21.15.7:6790]=c;smithi050:172.21.15.50=b;count:3 2024-09-18T07:58:24.231 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 07:58:23 smithi007 ceph-mon[29459]: Deploying daemon mon.b on smithi050 2024-09-18T07:58:24.747 INFO:teuthology.orchestra.run.smithi050.stderr:Inferring config /var/lib/ceph/779349ca-7593-11ef-bceb-c7b262605968/config/ceph.conf 2024-09-18T07:58:25.231 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 07:58:24 smithi007 ceph-mon[29459]: pgmap v7: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-18T07:58:25.851 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-09-18T07:58:25.851 INFO:teuthology.orchestra.run.smithi050.stdout:{"epoch":1,"fsid":"779349ca-7593-11ef-bceb-c7b262605968","modified":"2024-09-18T07:57:05.327907Z","created":"2024-09-18T07:57:05.327907Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:3300","nonce":0},{"type":"v1","addr":"172.21.15.7:6789","nonce":0}]},"addr":"172.21.15.7:6789/0","public_addr":"172.21.15.7:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-09-18T07:58:25.851 INFO:teuthology.orchestra.run.smithi050.stderr:dumped monmap epoch 1 2024-09-18T07:58:27.231 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 07:58:26 smithi007 ceph-mon[29459]: pgmap v8: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-18T07:58:27.231 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 07:58:26 smithi007 ceph-mon[29459]: from='client.? 172.21.15.50:0/1445195752' entity='client.admin' cmd={"prefix": "mon dump", "format": "json"} : dispatch 2024-09-18T07:58:27.406 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 07:58:27 smithi050 systemd[1]: Started Ceph mon.b for 779349ca-7593-11ef-bceb-c7b262605968. 2024-09-18T07:58:27.695 INFO:tasks.cephadm:Waiting for 3 mons in monmap... 2024-09-18T07:58:27.696 DEBUG:teuthology.orchestra.run.smithi050:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 779349ca-7593-11ef-bceb-c7b262605968 -- ceph mon dump -f json 2024-09-18T07:58:27.961 INFO:teuthology.orchestra.run.smithi050.stderr:Inferring config /var/lib/ceph/779349ca-7593-11ef-bceb-c7b262605968/mon.b/config 2024-09-18T07:58:32.430 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-09-18T07:58:32.430 INFO:teuthology.orchestra.run.smithi050.stdout:{"epoch":2,"fsid":"779349ca-7593-11ef-bceb-c7b262605968","modified":"2024-09-18T07:58:27.121545Z","created":"2024-09-18T07:57:05.327907Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:3300","nonce":0},{"type":"v1","addr":"172.21.15.7:6789","nonce":0}]},"addr":"172.21.15.7:6789/0","public_addr":"172.21.15.7:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:3300","nonce":0},{"type":"v1","addr":"172.21.15.50:6789","nonce":0}]},"addr":"172.21.15.50:6789/0","public_addr":"172.21.15.50:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0,1]} 2024-09-18T07:58:32.430 INFO:teuthology.orchestra.run.smithi050.stderr:dumped monmap epoch 2 2024-09-18T07:58:32.441 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 07:58:32 smithi050 ceph-mon[32371]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "a"} : dispatch 2024-09-18T07:58:32.441 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 07:58:32 smithi050 ceph-mon[32371]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-09-18T07:58:32.441 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 07:58:32 smithi050 ceph-mon[32371]: mon.a calling monitor election 2024-09-18T07:58:32.441 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 07:58:32 smithi050 ceph-mon[32371]: pgmap v9: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-18T07:58:32.442 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 07:58:32 smithi050 ceph-mon[32371]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-09-18T07:58:32.442 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 07:58:32 smithi050 ceph-mon[32371]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-09-18T07:58:32.442 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 07:58:32 smithi050 ceph-mon[32371]: mon.b calling monitor election 2024-09-18T07:58:32.442 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 07:58:32 smithi050 ceph-mon[32371]: pgmap v10: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-18T07:58:32.442 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 07:58:32 smithi050 ceph-mon[32371]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-09-18T07:58:32.442 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 07:58:32 smithi050 ceph-mon[32371]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-09-18T07:58:32.442 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 07:58:32 smithi050 ceph-mon[32371]: pgmap v11: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-18T07:58:32.442 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 07:58:32 smithi050 ceph-mon[32371]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-09-18T07:58:32.442 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 07:58:32 smithi050 ceph-mon[32371]: mon.a is new leader, mons a,b in quorum (ranks 0,1) 2024-09-18T07:58:32.443 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 07:58:32 smithi050 ceph-mon[32371]: monmap epoch 2 2024-09-18T07:58:32.443 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 07:58:32 smithi050 ceph-mon[32371]: fsid 779349ca-7593-11ef-bceb-c7b262605968 2024-09-18T07:58:32.443 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 07:58:32 smithi050 ceph-mon[32371]: last_changed 2024-09-18T07:58:27.121545+0000 2024-09-18T07:58:32.443 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 07:58:32 smithi050 ceph-mon[32371]: created 2024-09-18T07:57:05.327907+0000 2024-09-18T07:58:32.443 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 07:58:32 smithi050 ceph-mon[32371]: min_mon_release 19 (squid) 2024-09-18T07:58:32.443 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 07:58:32 smithi050 ceph-mon[32371]: election_strategy: 1 2024-09-18T07:58:32.443 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 07:58:32 smithi050 ceph-mon[32371]: 0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2024-09-18T07:58:32.443 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 07:58:32 smithi050 ceph-mon[32371]: 1: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.b 2024-09-18T07:58:32.443 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 07:58:32 smithi050 ceph-mon[32371]: fsmap 2024-09-18T07:58:32.443 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 07:58:32 smithi050 ceph-mon[32371]: osdmap e4: 0 total, 0 up, 0 in 2024-09-18T07:58:32.444 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 07:58:32 smithi050 ceph-mon[32371]: mgrmap e13: y(active, since 34s) 2024-09-18T07:58:32.444 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 07:58:32 smithi050 ceph-mon[32371]: overall HEALTH_OK 2024-09-18T07:58:32.444 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 07:58:32 smithi050 ceph-mon[32371]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' 2024-09-18T07:58:32.444 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 07:58:32 smithi050 ceph-mon[32371]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' 2024-09-18T07:58:32.444 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 07:58:32 smithi050 ceph-mon[32371]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' 2024-09-18T07:58:32.444 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 07:58:32 smithi050 ceph-mon[32371]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' cmd={"prefix": "auth get", "entity": "mon."} : dispatch 2024-09-18T07:58:32.445 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 07:58:32 smithi050 ceph-mon[32371]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-18T07:58:32.481 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 07:58:32 smithi007 ceph-mon[29459]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "a"} : dispatch 2024-09-18T07:58:32.481 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 07:58:32 smithi007 ceph-mon[29459]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-09-18T07:58:32.481 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 07:58:32 smithi007 ceph-mon[29459]: mon.a calling monitor election 2024-09-18T07:58:32.481 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 07:58:32 smithi007 ceph-mon[29459]: pgmap v9: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-18T07:58:32.481 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 07:58:32 smithi007 ceph-mon[29459]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-09-18T07:58:32.482 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 07:58:32 smithi007 ceph-mon[29459]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-09-18T07:58:32.482 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 07:58:32 smithi007 ceph-mon[29459]: mon.b calling monitor election 2024-09-18T07:58:32.482 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 07:58:32 smithi007 ceph-mon[29459]: pgmap v10: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-18T07:58:32.482 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 07:58:32 smithi007 ceph-mon[29459]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-09-18T07:58:32.482 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 07:58:32 smithi007 ceph-mon[29459]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-09-18T07:58:32.482 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 07:58:32 smithi007 ceph-mon[29459]: pgmap v11: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-18T07:58:32.482 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 07:58:32 smithi007 ceph-mon[29459]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-09-18T07:58:32.482 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 07:58:32 smithi007 ceph-mon[29459]: mon.a is new leader, mons a,b in quorum (ranks 0,1) 2024-09-18T07:58:32.482 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 07:58:32 smithi007 ceph-mon[29459]: monmap epoch 2 2024-09-18T07:58:32.482 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 07:58:32 smithi007 ceph-mon[29459]: fsid 779349ca-7593-11ef-bceb-c7b262605968 2024-09-18T07:58:32.482 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 07:58:32 smithi007 ceph-mon[29459]: last_changed 2024-09-18T07:58:27.121545+0000 2024-09-18T07:58:32.482 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 07:58:32 smithi007 ceph-mon[29459]: created 2024-09-18T07:57:05.327907+0000 2024-09-18T07:58:32.482 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 07:58:32 smithi007 ceph-mon[29459]: min_mon_release 19 (squid) 2024-09-18T07:58:32.483 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 07:58:32 smithi007 ceph-mon[29459]: election_strategy: 1 2024-09-18T07:58:32.483 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 07:58:32 smithi007 ceph-mon[29459]: 0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2024-09-18T07:58:32.483 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 07:58:32 smithi007 ceph-mon[29459]: 1: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.b 2024-09-18T07:58:32.483 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 07:58:32 smithi007 ceph-mon[29459]: fsmap 2024-09-18T07:58:32.483 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 07:58:32 smithi007 ceph-mon[29459]: osdmap e4: 0 total, 0 up, 0 in 2024-09-18T07:58:32.483 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 07:58:32 smithi007 ceph-mon[29459]: mgrmap e13: y(active, since 34s) 2024-09-18T07:58:32.483 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 07:58:32 smithi007 ceph-mon[29459]: overall HEALTH_OK 2024-09-18T07:58:32.483 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 07:58:32 smithi007 ceph-mon[29459]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' 2024-09-18T07:58:32.483 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 07:58:32 smithi007 ceph-mon[29459]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' 2024-09-18T07:58:32.483 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 07:58:32 smithi007 ceph-mon[29459]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' 2024-09-18T07:58:32.483 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 07:58:32 smithi007 ceph-mon[29459]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' cmd={"prefix": "auth get", "entity": "mon."} : dispatch 2024-09-18T07:58:32.483 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 07:58:32 smithi007 ceph-mon[29459]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-18T07:58:33.406 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 07:58:33 smithi050 ceph-mon[32371]: Deploying daemon mon.c on smithi007 2024-09-18T07:58:33.406 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 07:58:33 smithi050 ceph-mon[32371]: from='client.? 172.21.15.50:0/1788123659' entity='client.admin' cmd={"prefix": "mon dump", "format": "json"} : dispatch 2024-09-18T07:58:33.407 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 07:58:33 smithi050 ceph-mon[32371]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-09-18T07:58:33.481 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 07:58:33 smithi007 ceph-mon[29459]: Deploying daemon mon.c on smithi007 2024-09-18T07:58:33.481 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 07:58:33 smithi007 ceph-mon[29459]: from='client.? 172.21.15.50:0/1788123659' entity='client.admin' cmd={"prefix": "mon dump", "format": "json"} : dispatch 2024-09-18T07:58:33.481 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 07:58:33 smithi007 ceph-mon[29459]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-09-18T07:58:34.022 INFO:tasks.cephadm:Waiting for 3 mons in monmap... 2024-09-18T07:58:34.023 DEBUG:teuthology.orchestra.run.smithi050:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 779349ca-7593-11ef-bceb-c7b262605968 -- ceph mon dump -f json 2024-09-18T07:58:34.288 INFO:teuthology.orchestra.run.smithi050.stderr:Inferring config /var/lib/ceph/779349ca-7593-11ef-bceb-c7b262605968/mon.b/config 2024-09-18T07:58:34.406 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 07:58:34 smithi050 ceph-mon[32371]: pgmap v12: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-18T07:58:34.476 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 07:58:34 smithi007 ceph-mon[29459]: pgmap v12: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-18T07:58:34.993 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-09-18T07:58:34.993 INFO:teuthology.orchestra.run.smithi050.stdout:{"epoch":2,"fsid":"779349ca-7593-11ef-bceb-c7b262605968","modified":"2024-09-18T07:58:27.121545Z","created":"2024-09-18T07:57:05.327907Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:3300","nonce":0},{"type":"v1","addr":"172.21.15.7:6789","nonce":0}]},"addr":"172.21.15.7:6789/0","public_addr":"172.21.15.7:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:3300","nonce":0},{"type":"v1","addr":"172.21.15.50:6789","nonce":0}]},"addr":"172.21.15.50:6789/0","public_addr":"172.21.15.50:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0,1]} 2024-09-18T07:58:34.994 INFO:teuthology.orchestra.run.smithi050.stderr:dumped monmap epoch 2 2024-09-18T07:58:35.157 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 07:58:34 smithi007 systemd[1]: Starting Ceph mon.c for 779349ca-7593-11ef-bceb-c7b262605968... 2024-09-18T07:58:35.406 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 07:58:35 smithi050 ceph-mon[32371]: from='client.? 172.21.15.50:0/393846258' entity='client.admin' cmd={"prefix": "mon dump", "format": "json"} : dispatch 2024-09-18T07:58:35.421 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 07:58:35 smithi007 ceph-mon[29459]: from='client.? 172.21.15.50:0/393846258' entity='client.admin' cmd={"prefix": "mon dump", "format": "json"} : dispatch 2024-09-18T07:58:35.422 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 07:58:35 smithi007 podman[39230]: 2024-09-18 07:58:35.057806104 +0000 UTC m=+0.016169808 image pull b22e1e37ddf1ca6e22136cd6540a9ab61e7cafb8bfbc7a91a2f20f7f9941bd9a quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:1f5b5a3734f665ef333266f39143ac3ef1eb1951b21258f2307e10332e2d7ac9 2024-09-18T07:58:35.422 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 07:58:35 smithi007 podman[39230]: 2024-09-18 07:58:35.159199305 +0000 UTC m=+0.117562993 container create 3898467ba9eb184258f0533b36e37a04dbe605702e07917009b63fb31555588a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:1f5b5a3734f665ef333266f39143ac3ef1eb1951b21258f2307e10332e2d7ac9, name=ceph-779349ca-7593-11ef-bceb-c7b262605968-mon-c, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, RELEASE=wip-guits-main-2024-09-17-1213-e7fb7b5, org.label-schema.name=CentOS Stream 9 Base Image, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, GIT_BRANCH=HEAD, org.label-schema.build-date=20240731, org.label-schema.schema-version=1.0, ceph=True, org.label-schema.vendor=CentOS, io.buildah.version=1.37.2, maintainer=Guillaume Abrioux ) 2024-09-18T07:58:35.422 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 07:58:35 smithi007 podman[39230]: 2024-09-18 07:58:35.370130207 +0000 UTC m=+0.328493903 container init 3898467ba9eb184258f0533b36e37a04dbe605702e07917009b63fb31555588a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:1f5b5a3734f665ef333266f39143ac3ef1eb1951b21258f2307e10332e2d7ac9, name=ceph-779349ca-7593-11ef-bceb-c7b262605968-mon-c, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, io.buildah.version=1.37.2, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.build-date=20240731, org.label-schema.name=CentOS Stream 9 Base Image, RELEASE=wip-guits-main-2024-09-17-1213-e7fb7b5, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-09-18T07:58:35.422 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 07:58:35 smithi007 podman[39230]: 2024-09-18 07:58:35.373701646 +0000 UTC m=+0.332065333 container start 3898467ba9eb184258f0533b36e37a04dbe605702e07917009b63fb31555588a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:1f5b5a3734f665ef333266f39143ac3ef1eb1951b21258f2307e10332e2d7ac9, name=ceph-779349ca-7593-11ef-bceb-c7b262605968-mon-c, RELEASE=wip-guits-main-2024-09-17-1213-e7fb7b5, CEPH_POINT_RELEASE=, ceph=True, maintainer=Guillaume Abrioux , GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.build-date=20240731, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, io.buildah.version=1.37.2) 2024-09-18T07:58:35.422 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 07:58:35 smithi007 ceph-mon[39249]: set uid:gid to 167:167 (ceph:ceph) 2024-09-18T07:58:35.422 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 07:58:35 smithi007 ceph-mon[39249]: ceph version 19.3.0-4983-ge7fb7b56 (e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e) squid (dev), process ceph-mon, pid 2 2024-09-18T07:58:35.422 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 07:58:35 smithi007 ceph-mon[39249]: pidfile_write: ignore empty --pid-file 2024-09-18T07:58:35.422 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 07:58:35 smithi007 ceph-mon[39249]: load: jerasure load: lrc 2024-09-18T07:58:35.422 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 07:58:35 smithi007 ceph-mon[39249]: rocksdb: RocksDB version: 7.9.2 2024-09-18T07:58:35.422 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 07:58:35 smithi007 ceph-mon[39249]: rocksdb: Git sha 0 2024-09-18T07:58:35.423 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 07:58:35 smithi007 ceph-mon[39249]: rocksdb: Compile date 2024-09-17 15:59:30 2024-09-18T07:58:35.423 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 07:58:35 smithi007 ceph-mon[39249]: rocksdb: DB SUMMARY 2024-09-18T07:58:35.423 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 07:58:35 smithi007 ceph-mon[39249]: rocksdb: DB Session ID: Z6GZEIY8TUGDDETOGV5Z 2024-09-18T07:58:35.423 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 07:58:35 smithi007 ceph-mon[39249]: rocksdb: CURRENT file: CURRENT 2024-09-18T07:58:35.423 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 07:58:35 smithi007 ceph-mon[39249]: rocksdb: IDENTITY file: IDENTITY 2024-09-18T07:58:35.423 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 07:58:35 smithi007 ceph-mon[39249]: rocksdb: MANIFEST file: MANIFEST-000005 size: 59 Bytes 2024-09-18T07:58:35.423 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 07:58:35 smithi007 ceph-mon[39249]: rocksdb: SST files in /var/lib/ceph/mon/ceph-c/store.db dir, Total Num: 0, files: 2024-09-18T07:58:35.424 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 07:58:35 smithi007 ceph-mon[39249]: rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-c/store.db: 000004.log size: 636 ; 2024-09-18T07:58:35.424 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 07:58:35 smithi007 ceph-mon[39249]: rocksdb: Options.error_if_exists: 0 2024-09-18T07:58:35.424 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 07:58:35 smithi007 ceph-mon[39249]: rocksdb: Options.create_if_missing: 0 2024-09-18T07:58:35.424 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 07:58:35 smithi007 ceph-mon[39249]: rocksdb: Options.paranoid_checks: 1 2024-09-18T07:58:35.425 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 07:58:35 smithi007 ceph-mon[39249]: rocksdb: Options.flush_verify_memtable_count: 1 2024-09-18T07:58:35.425 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 07:58:35 smithi007 ceph-mon[39249]: rocksdb: Options.track_and_verify_wals_in_manifest: 0 2024-09-18T07:58:35.425 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 07:58:35 smithi007 ceph-mon[39249]: rocksdb: Options.verify_sst_unique_id_in_manifest: 1 2024-09-18T07:58:35.425 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 07:58:35 smithi007 ceph-mon[39249]: rocksdb: Options.env: 0x56530b72a120 2024-09-18T07:58:35.425 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 07:58:35 smithi007 ceph-mon[39249]: rocksdb: Options.fs: PosixFileSystem 2024-09-18T07:58:35.425 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 07:58:35 smithi007 ceph-mon[39249]: rocksdb: Options.info_log: 0x56530d0e7700 2024-09-18T07:58:35.426 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 07:58:35 smithi007 ceph-mon[39249]: rocksdb: Options.max_file_opening_threads: 16 2024-09-18T07:58:35.426 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 07:58:35 smithi007 ceph-mon[39249]: rocksdb: Options.statistics: (nil) 2024-09-18T07:58:35.426 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 07:58:35 smithi007 ceph-mon[39249]: rocksdb: Options.use_fsync: 0 2024-09-18T07:58:35.426 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 07:58:35 smithi007 ceph-mon[39249]: rocksdb: Options.max_log_file_size: 0 2024-09-18T07:58:35.426 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 07:58:35 smithi007 ceph-mon[39249]: rocksdb: Options.max_manifest_file_size: 1073741824 2024-09-18T07:58:35.426 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 07:58:35 smithi007 ceph-mon[39249]: rocksdb: Options.log_file_time_to_roll: 0 2024-09-18T07:58:35.427 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 07:58:35 smithi007 ceph-mon[39249]: rocksdb: Options.keep_log_file_num: 1000 2024-09-18T07:58:35.427 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 07:58:35 smithi007 ceph-mon[39249]: rocksdb: Options.recycle_log_file_num: 0 2024-09-18T07:58:35.427 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 07:58:35 smithi007 ceph-mon[39249]: rocksdb: Options.allow_fallocate: 1 2024-09-18T07:58:35.427 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 07:58:35 smithi007 ceph-mon[39249]: rocksdb: Options.allow_mmap_reads: 0 2024-09-18T07:58:35.427 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 07:58:35 smithi007 ceph-mon[39249]: rocksdb: Options.allow_mmap_writes: 0 2024-09-18T07:58:35.428 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 07:58:35 smithi007 ceph-mon[39249]: rocksdb: Options.use_direct_reads: 0 2024-09-18T07:58:35.428 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 07:58:35 smithi007 ceph-mon[39249]: rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2024-09-18T07:58:35.428 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 07:58:35 smithi007 ceph-mon[39249]: rocksdb: Options.create_missing_column_families: 0 2024-09-18T07:58:35.428 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 07:58:35 smithi007 ceph-mon[39249]: rocksdb: Options.db_log_dir: 2024-09-18T07:58:35.428 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 07:58:35 smithi007 ceph-mon[39249]: rocksdb: Options.wal_dir: 2024-09-18T07:58:35.429 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 07:58:35 smithi007 ceph-mon[39249]: rocksdb: Options.table_cache_numshardbits: 6 2024-09-18T07:58:35.429 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 07:58:35 smithi007 ceph-mon[39249]: rocksdb: Options.WAL_ttl_seconds: 0 2024-09-18T07:58:35.429 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 07:58:35 smithi007 ceph-mon[39249]: rocksdb: Options.WAL_size_limit_MB: 0 2024-09-18T07:58:35.429 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 07:58:35 smithi007 ceph-mon[39249]: rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2024-09-18T07:58:35.430 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 07:58:35 smithi007 ceph-mon[39249]: rocksdb: Options.manifest_preallocation_size: 4194304 2024-09-18T07:58:35.430 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 07:58:35 smithi007 ceph-mon[39249]: rocksdb: Options.is_fd_close_on_exec: 1 2024-09-18T07:58:35.430 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 07:58:35 smithi007 ceph-mon[39249]: rocksdb: Options.advise_random_on_open: 1 2024-09-18T07:58:35.430 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 07:58:35 smithi007 ceph-mon[39249]: rocksdb: Options.db_write_buffer_size: 0 2024-09-18T07:58:35.431 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 07:58:35 smithi007 ceph-mon[39249]: rocksdb: Options.write_buffer_manager: 0x56530d0ebb80 2024-09-18T07:58:35.431 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 07:58:35 smithi007 ceph-mon[39249]: rocksdb: Options.access_hint_on_compaction_start: 1 2024-09-18T07:58:35.431 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 07:58:35 smithi007 ceph-mon[39249]: rocksdb: Options.random_access_max_buffer_size: 1048576 2024-09-18T07:58:35.431 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 07:58:35 smithi007 ceph-mon[39249]: rocksdb: Options.use_adaptive_mutex: 0 2024-09-18T07:58:35.431 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 07:58:35 smithi007 ceph-mon[39249]: rocksdb: Options.rate_limiter: (nil) 2024-09-18T07:58:35.432 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 07:58:35 smithi007 ceph-mon[39249]: rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2024-09-18T07:58:35.432 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 07:58:35 smithi007 ceph-mon[39249]: rocksdb: Options.wal_recovery_mode: 2 2024-09-18T07:58:35.432 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 07:58:35 smithi007 ceph-mon[39249]: rocksdb: Options.enable_thread_tracking: 0 2024-09-18T07:58:35.432 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 07:58:35 smithi007 ceph-mon[39249]: rocksdb: Options.enable_pipelined_write: 0 2024-09-18T07:58:35.433 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 07:58:35 smithi007 ceph-mon[39249]: rocksdb: Options.unordered_write: 0 2024-09-18T07:58:35.433 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 07:58:35 smithi007 ceph-mon[39249]: rocksdb: Options.allow_concurrent_memtable_write: 1 2024-09-18T07:58:35.433 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 07:58:35 smithi007 ceph-mon[39249]: rocksdb: Options.enable_write_thread_adaptive_yield: 1 2024-09-18T07:58:35.433 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 07:58:35 smithi007 ceph-mon[39249]: rocksdb: Options.write_thread_max_yield_usec: 100 2024-09-18T07:58:35.433 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 07:58:35 smithi007 ceph-mon[39249]: rocksdb: Options.write_thread_slow_yield_usec: 3 2024-09-18T07:58:35.434 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 07:58:35 smithi007 ceph-mon[39249]: rocksdb: Options.row_cache: None 2024-09-18T07:58:35.434 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 07:58:35 smithi007 ceph-mon[39249]: rocksdb: Options.wal_filter: None 2024-09-18T07:58:35.434 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 07:58:35 smithi007 ceph-mon[39249]: rocksdb: Options.avoid_flush_during_recovery: 0 2024-09-18T07:58:35.434 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 07:58:35 smithi007 ceph-mon[39249]: rocksdb: Options.allow_ingest_behind: 0 2024-09-18T07:58:35.434 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 07:58:35 smithi007 ceph-mon[39249]: rocksdb: Options.two_write_queues: 0 2024-09-18T07:58:35.434 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 07:58:35 smithi007 ceph-mon[39249]: rocksdb: Options.manual_wal_flush: 0 2024-09-18T07:58:35.435 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 07:58:35 smithi007 ceph-mon[39249]: rocksdb: Options.wal_compression: 0 2024-09-18T07:58:35.435 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 07:58:35 smithi007 ceph-mon[39249]: rocksdb: Options.atomic_flush: 0 2024-09-18T07:58:35.435 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 07:58:35 smithi007 ceph-mon[39249]: rocksdb: Options.avoid_unnecessary_blocking_io: 0 2024-09-18T07:58:35.435 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 07:58:35 smithi007 ceph-mon[39249]: rocksdb: Options.persist_stats_to_disk: 0 2024-09-18T07:58:35.435 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 07:58:35 smithi007 ceph-mon[39249]: rocksdb: Options.write_dbid_to_manifest: 0 2024-09-18T07:58:35.436 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 07:58:35 smithi007 ceph-mon[39249]: rocksdb: Options.log_readahead_size: 0 2024-09-18T07:58:35.436 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 07:58:35 smithi007 ceph-mon[39249]: rocksdb: Options.file_checksum_gen_factory: Unknown 2024-09-18T07:58:35.436 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 07:58:35 smithi007 ceph-mon[39249]: rocksdb: Options.best_efforts_recovery: 0 2024-09-18T07:58:35.436 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 07:58:35 smithi007 ceph-mon[39249]: rocksdb: Options.max_bgerror_resume_count: 2147483647 2024-09-18T07:58:35.436 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 07:58:35 smithi007 ceph-mon[39249]: rocksdb: Options.bgerror_resume_retry_interval: 1000000 2024-09-18T07:58:35.436 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 07:58:35 smithi007 ceph-mon[39249]: rocksdb: Options.allow_data_in_errors: 0 2024-09-18T07:58:35.437 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 07:58:35 smithi007 ceph-mon[39249]: rocksdb: Options.db_host_id: __hostname__ 2024-09-18T07:58:35.437 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 07:58:35 smithi007 ceph-mon[39249]: rocksdb: Options.enforce_single_del_contracts: true 2024-09-18T07:58:35.437 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 07:58:35 smithi007 ceph-mon[39249]: rocksdb: Options.max_background_jobs: 2 2024-09-18T07:58:35.437 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 07:58:35 smithi007 ceph-mon[39249]: rocksdb: Options.max_background_compactions: -1 2024-09-18T07:58:35.437 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 07:58:35 smithi007 ceph-mon[39249]: rocksdb: Options.max_subcompactions: 1 2024-09-18T07:58:35.438 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 07:58:35 smithi007 ceph-mon[39249]: rocksdb: Options.avoid_flush_during_shutdown: 0 2024-09-18T07:58:35.438 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 07:58:35 smithi007 ceph-mon[39249]: rocksdb: Options.writable_file_max_buffer_size: 1048576 2024-09-18T07:58:35.438 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 07:58:35 smithi007 ceph-mon[39249]: rocksdb: Options.delayed_write_rate : 16777216 2024-09-18T07:58:35.438 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 07:58:35 smithi007 ceph-mon[39249]: rocksdb: Options.max_total_wal_size: 0 2024-09-18T07:58:35.438 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 07:58:35 smithi007 ceph-mon[39249]: rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2024-09-18T07:58:35.438 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 07:58:35 smithi007 ceph-mon[39249]: rocksdb: Options.stats_dump_period_sec: 600 2024-09-18T07:58:35.439 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 07:58:35 smithi007 ceph-mon[39249]: rocksdb: Options.stats_persist_period_sec: 600 2024-09-18T07:58:35.439 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 07:58:35 smithi007 ceph-mon[39249]: rocksdb: Options.stats_history_buffer_size: 1048576 2024-09-18T07:58:35.439 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 07:58:35 smithi007 ceph-mon[39249]: rocksdb: Options.max_open_files: -1 2024-09-18T07:58:35.439 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 07:58:35 smithi007 ceph-mon[39249]: rocksdb: Options.bytes_per_sync: 0 2024-09-18T07:58:35.439 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 07:58:35 smithi007 ceph-mon[39249]: rocksdb: Options.wal_bytes_per_sync: 0 2024-09-18T07:58:35.439 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 07:58:35 smithi007 ceph-mon[39249]: rocksdb: Options.strict_bytes_per_sync: 0 2024-09-18T07:58:35.440 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 07:58:35 smithi007 ceph-mon[39249]: rocksdb: Options.compaction_readahead_size: 0 2024-09-18T07:58:35.440 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 07:58:35 smithi007 ceph-mon[39249]: rocksdb: Options.max_background_flushes: -1 2024-09-18T07:58:35.440 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 07:58:35 smithi007 ceph-mon[39249]: rocksdb: Compression algorithms supported: 2024-09-18T07:58:35.440 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 07:58:35 smithi007 ceph-mon[39249]: rocksdb: kZSTD supported: 0 2024-09-18T07:58:35.440 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 07:58:35 smithi007 ceph-mon[39249]: rocksdb: kXpressCompression supported: 0 2024-09-18T07:58:35.441 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 07:58:35 smithi007 ceph-mon[39249]: rocksdb: kBZip2Compression supported: 0 2024-09-18T07:58:35.441 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 07:58:35 smithi007 ceph-mon[39249]: rocksdb: kZSTDNotFinalCompression supported: 0 2024-09-18T07:58:35.441 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 07:58:35 smithi007 ceph-mon[39249]: rocksdb: kLZ4Compression supported: 1 2024-09-18T07:58:35.441 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 07:58:35 smithi007 ceph-mon[39249]: rocksdb: kZlibCompression supported: 1 2024-09-18T07:58:35.441 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 07:58:35 smithi007 ceph-mon[39249]: rocksdb: kLZ4HCCompression supported: 1 2024-09-18T07:58:35.441 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 07:58:35 smithi007 ceph-mon[39249]: rocksdb: kSnappyCompression supported: 1 2024-09-18T07:58:35.442 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 07:58:35 smithi007 ceph-mon[39249]: rocksdb: Fast CRC32 supported: Supported on x86 2024-09-18T07:58:35.442 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 07:58:35 smithi007 ceph-mon[39249]: rocksdb: DMutex implementation: pthread_mutex_t 2024-09-18T07:58:35.442 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 07:58:35 smithi007 ceph-mon[39249]: rocksdb: [db/version_set.cc:5527] Recovering from manifest file: /var/lib/ceph/mon/ceph-c/store.db/MANIFEST-000005 2024-09-18T07:58:35.442 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 07:58:35 smithi007 ceph-mon[39249]: rocksdb: [db/column_family.cc:630] --------------- Options for column family [default]: 2024-09-18T07:58:35.442 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 07:58:35 smithi007 ceph-mon[39249]: rocksdb: Options.comparator: leveldb.BytewiseComparator 2024-09-18T07:58:35.442 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 07:58:35 smithi007 ceph-mon[39249]: rocksdb: Options.merge_operator: 2024-09-18T07:58:35.443 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 07:58:35 smithi007 ceph-mon[39249]: rocksdb: Options.compaction_filter: None 2024-09-18T07:58:35.443 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 07:58:35 smithi007 ceph-mon[39249]: rocksdb: Options.compaction_filter_factory: None 2024-09-18T07:58:35.443 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 07:58:35 smithi007 ceph-mon[39249]: rocksdb: Options.sst_partitioner_factory: None 2024-09-18T07:58:35.443 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 07:58:35 smithi007 ceph-mon[39249]: rocksdb: Options.memtable_factory: SkipListFactory 2024-09-18T07:58:35.443 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 07:58:35 smithi007 ceph-mon[39249]: rocksdb: Options.table_factory: BlockBasedTable 2024-09-18T07:58:35.443 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 07:58:35 smithi007 ceph-mon[39249]: rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x56530d0e7320) 2024-09-18T07:58:35.444 INFO:journalctl@ceph.mon.c.smithi007.stdout: cache_index_and_filter_blocks: 1 2024-09-18T07:58:35.444 INFO:journalctl@ceph.mon.c.smithi007.stdout: cache_index_and_filter_blocks_with_high_priority: 0 2024-09-18T07:58:35.444 INFO:journalctl@ceph.mon.c.smithi007.stdout: pin_l0_filter_and_index_blocks_in_cache: 0 2024-09-18T07:58:35.444 INFO:journalctl@ceph.mon.c.smithi007.stdout: pin_top_level_index_and_filter: 1 2024-09-18T07:58:35.444 INFO:journalctl@ceph.mon.c.smithi007.stdout: index_type: 0 2024-09-18T07:58:35.445 INFO:journalctl@ceph.mon.c.smithi007.stdout: data_block_index_type: 0 2024-09-18T07:58:35.445 INFO:journalctl@ceph.mon.c.smithi007.stdout: index_shortening: 1 2024-09-18T07:58:35.445 INFO:journalctl@ceph.mon.c.smithi007.stdout: data_block_hash_table_util_ratio: 0.750000 2024-09-18T07:58:35.445 INFO:journalctl@ceph.mon.c.smithi007.stdout: checksum: 4 2024-09-18T07:58:35.445 INFO:journalctl@ceph.mon.c.smithi007.stdout: no_block_cache: 0 2024-09-18T07:58:35.446 INFO:journalctl@ceph.mon.c.smithi007.stdout: block_cache: 0x56530d0db610 2024-09-18T07:58:35.446 INFO:journalctl@ceph.mon.c.smithi007.stdout: block_cache_name: BinnedLRUCache 2024-09-18T07:58:35.446 INFO:journalctl@ceph.mon.c.smithi007.stdout: block_cache_options: 2024-09-18T07:58:35.446 INFO:journalctl@ceph.mon.c.smithi007.stdout: capacity : 536870912 2024-09-18T07:58:35.446 INFO:journalctl@ceph.mon.c.smithi007.stdout: num_shard_bits : 4 2024-09-18T07:58:35.447 INFO:journalctl@ceph.mon.c.smithi007.stdout: strict_capacity_limit : 0 2024-09-18T07:58:35.447 INFO:journalctl@ceph.mon.c.smithi007.stdout: high_pri_pool_ratio: 0.000 2024-09-18T07:58:35.447 INFO:journalctl@ceph.mon.c.smithi007.stdout: block_cache_compressed: (nil) 2024-09-18T07:58:35.447 INFO:journalctl@ceph.mon.c.smithi007.stdout: persistent_cache: (nil) 2024-09-18T07:58:35.447 INFO:journalctl@ceph.mon.c.smithi007.stdout: block_size: 4096 2024-09-18T07:58:35.448 INFO:journalctl@ceph.mon.c.smithi007.stdout: block_size_deviation: 10 2024-09-18T07:58:35.448 INFO:journalctl@ceph.mon.c.smithi007.stdout: block_restart_interval: 16 2024-09-18T07:58:35.448 INFO:journalctl@ceph.mon.c.smithi007.stdout: index_block_restart_interval: 1 2024-09-18T07:58:35.448 INFO:journalctl@ceph.mon.c.smithi007.stdout: metadata_block_size: 4096 2024-09-18T07:58:35.448 INFO:journalctl@ceph.mon.c.smithi007.stdout: partition_filters: 0 2024-09-18T07:58:35.449 INFO:journalctl@ceph.mon.c.smithi007.stdout: use_delta_encoding: 1 2024-09-18T07:58:35.449 INFO:journalctl@ceph.mon.c.smithi007.stdout: filter_policy: bloomfilter 2024-09-18T07:58:35.449 INFO:journalctl@ceph.mon.c.smithi007.stdout: whole_key_filtering: 1 2024-09-18T07:58:35.449 INFO:journalctl@ceph.mon.c.smithi007.stdout: verify_compression: 0 2024-09-18T07:58:35.449 INFO:journalctl@ceph.mon.c.smithi007.stdout: read_amp_bytes_per_bit: 0 2024-09-18T07:58:35.450 INFO:journalctl@ceph.mon.c.smithi007.stdout: format_version: 5 2024-09-18T07:58:35.450 INFO:journalctl@ceph.mon.c.smithi007.stdout: enable_index_compression: 1 2024-09-18T07:58:35.450 INFO:journalctl@ceph.mon.c.smithi007.stdout: block_align: 0 2024-09-18T07:58:35.450 INFO:journalctl@ceph.mon.c.smithi007.stdout: max_auto_readahead_size: 262144 2024-09-18T07:58:35.450 INFO:journalctl@ceph.mon.c.smithi007.stdout: prepopulate_block_cache: 0 2024-09-18T07:58:35.450 INFO:journalctl@ceph.mon.c.smithi007.stdout: initial_auto_readahead_size: 8192 2024-09-18T07:58:35.450 INFO:journalctl@ceph.mon.c.smithi007.stdout: num_file_reads_for_auto_readahead: 2 2024-09-18T07:58:35.451 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 07:58:35 smithi007 ceph-mon[39249]: rocksdb: Options.write_buffer_size: 33554432 2024-09-18T07:58:35.451 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 07:58:35 smithi007 ceph-mon[39249]: rocksdb: Options.max_write_buffer_number: 2 2024-09-18T07:58:35.451 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 07:58:35 smithi007 ceph-mon[39249]: rocksdb: Options.compression: NoCompression 2024-09-18T07:58:35.451 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 07:58:35 smithi007 ceph-mon[39249]: rocksdb: Options.bottommost_compression: Disabled 2024-09-18T07:58:35.451 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 07:58:35 smithi007 ceph-mon[39249]: rocksdb: Options.prefix_extractor: nullptr 2024-09-18T07:58:35.451 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 07:58:35 smithi007 ceph-mon[39249]: rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2024-09-18T07:58:35.451 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 07:58:35 smithi007 ceph-mon[39249]: rocksdb: Options.num_levels: 7 2024-09-18T07:58:35.452 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 07:58:35 smithi007 ceph-mon[39249]: rocksdb: Options.min_write_buffer_number_to_merge: 1 2024-09-18T07:58:35.452 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 07:58:35 smithi007 ceph-mon[39249]: rocksdb: Options.max_write_buffer_number_to_maintain: 0 2024-09-18T07:58:35.452 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 07:58:35 smithi007 ceph-mon[39249]: rocksdb: Options.max_write_buffer_size_to_maintain: 0 2024-09-18T07:58:35.452 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 07:58:35 smithi007 ceph-mon[39249]: rocksdb: Options.bottommost_compression_opts.window_bits: -14 2024-09-18T07:58:35.452 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 07:58:35 smithi007 ceph-mon[39249]: rocksdb: Options.bottommost_compression_opts.level: 32767 2024-09-18T07:58:35.452 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 07:58:35 smithi007 ceph-mon[39249]: rocksdb: Options.bottommost_compression_opts.strategy: 0 2024-09-18T07:58:35.453 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 07:58:35 smithi007 ceph-mon[39249]: rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2024-09-18T07:58:35.453 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 07:58:35 smithi007 ceph-mon[39249]: rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2024-09-18T07:58:35.453 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 07:58:35 smithi007 ceph-mon[39249]: rocksdb: Options.bottommost_compression_opts.parallel_threads: 1 2024-09-18T07:58:35.453 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 07:58:35 smithi007 ceph-mon[39249]: rocksdb: Options.bottommost_compression_opts.enabled: false 2024-09-18T07:58:35.453 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 07:58:35 smithi007 ceph-mon[39249]: rocksdb: Options.bottommost_compression_opts.max_dict_buffer_bytes: 0 2024-09-18T07:58:35.453 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 07:58:35 smithi007 ceph-mon[39249]: rocksdb: Options.bottommost_compression_opts.use_zstd_dict_trainer: true 2024-09-18T07:58:35.453 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 07:58:35 smithi007 ceph-mon[39249]: rocksdb: Options.compression_opts.window_bits: -14 2024-09-18T07:58:35.453 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 07:58:35 smithi007 ceph-mon[39249]: rocksdb: Options.compression_opts.level: 32767 2024-09-18T07:58:35.454 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 07:58:35 smithi007 ceph-mon[39249]: rocksdb: Options.compression_opts.strategy: 0 2024-09-18T07:58:35.454 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 07:58:35 smithi007 ceph-mon[39249]: rocksdb: Options.compression_opts.max_dict_bytes: 0 2024-09-18T07:58:35.454 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 07:58:35 smithi007 ceph-mon[39249]: rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2024-09-18T07:58:35.454 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 07:58:35 smithi007 ceph-mon[39249]: rocksdb: Options.compression_opts.use_zstd_dict_trainer: true 2024-09-18T07:58:35.454 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 07:58:35 smithi007 ceph-mon[39249]: rocksdb: Options.compression_opts.parallel_threads: 1 2024-09-18T07:58:35.454 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 07:58:35 smithi007 ceph-mon[39249]: rocksdb: Options.compression_opts.enabled: false 2024-09-18T07:58:35.455 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 07:58:35 smithi007 ceph-mon[39249]: rocksdb: Options.compression_opts.max_dict_buffer_bytes: 0 2024-09-18T07:58:35.455 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 07:58:35 smithi007 ceph-mon[39249]: rocksdb: Options.level0_file_num_compaction_trigger: 4 2024-09-18T07:58:35.456 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 07:58:35 smithi007 ceph-mon[39249]: rocksdb: Options.level0_slowdown_writes_trigger: 20 2024-09-18T07:58:35.456 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 07:58:35 smithi007 ceph-mon[39249]: rocksdb: Options.level0_stop_writes_trigger: 36 2024-09-18T07:58:35.456 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 07:58:35 smithi007 ceph-mon[39249]: rocksdb: Options.target_file_size_base: 67108864 2024-09-18T07:58:35.456 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 07:58:35 smithi007 ceph-mon[39249]: rocksdb: Options.target_file_size_multiplier: 1 2024-09-18T07:58:35.456 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 07:58:35 smithi007 ceph-mon[39249]: rocksdb: Options.max_bytes_for_level_base: 268435456 2024-09-18T07:58:35.456 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 07:58:35 smithi007 ceph-mon[39249]: rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2024-09-18T07:58:35.456 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 07:58:35 smithi007 ceph-mon[39249]: rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2024-09-18T07:58:35.456 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 07:58:35 smithi007 ceph-mon[39249]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2024-09-18T07:58:35.456 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 07:58:35 smithi007 ceph-mon[39249]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2024-09-18T07:58:35.456 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 07:58:35 smithi007 ceph-mon[39249]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2024-09-18T07:58:35.456 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 07:58:35 smithi007 ceph-mon[39249]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2024-09-18T07:58:35.457 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 07:58:35 smithi007 ceph-mon[39249]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2024-09-18T07:58:35.457 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 07:58:35 smithi007 ceph-mon[39249]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2024-09-18T07:58:35.457 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 07:58:35 smithi007 ceph-mon[39249]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2024-09-18T07:58:35.457 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 07:58:35 smithi007 ceph-mon[39249]: rocksdb: Options.max_sequential_skip_in_iterations: 8 2024-09-18T07:58:35.457 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 07:58:35 smithi007 ceph-mon[39249]: rocksdb: Options.max_compaction_bytes: 1677721600 2024-09-18T07:58:35.457 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 07:58:35 smithi007 ceph-mon[39249]: rocksdb: Options.ignore_max_compaction_bytes_for_input: true 2024-09-18T07:58:35.457 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 07:58:35 smithi007 ceph-mon[39249]: rocksdb: Options.arena_block_size: 1048576 2024-09-18T07:58:35.457 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 07:58:35 smithi007 ceph-mon[39249]: rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2024-09-18T07:58:35.457 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 07:58:35 smithi007 ceph-mon[39249]: rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2024-09-18T07:58:35.457 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 07:58:35 smithi007 ceph-mon[39249]: rocksdb: Options.disable_auto_compactions: 0 2024-09-18T07:58:35.457 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 07:58:35 smithi007 ceph-mon[39249]: rocksdb: Options.compaction_style: kCompactionStyleLevel 2024-09-18T07:58:35.457 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 07:58:35 smithi007 ceph-mon[39249]: rocksdb: Options.compaction_pri: kMinOverlappingRatio 2024-09-18T07:58:35.458 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 07:58:35 smithi007 ceph-mon[39249]: rocksdb: Options.compaction_options_universal.size_ratio: 1 2024-09-18T07:58:35.458 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 07:58:35 smithi007 ceph-mon[39249]: rocksdb: Options.compaction_options_universal.min_merge_width: 2 2024-09-18T07:58:35.458 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 07:58:35 smithi007 ceph-mon[39249]: rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2024-09-18T07:58:35.458 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 07:58:35 smithi007 ceph-mon[39249]: rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2024-09-18T07:58:35.458 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 07:58:35 smithi007 ceph-mon[39249]: rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2024-09-18T07:58:35.458 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 07:58:35 smithi007 ceph-mon[39249]: rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2024-09-18T07:58:35.458 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 07:58:35 smithi007 ceph-mon[39249]: rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2024-09-18T07:58:35.458 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 07:58:35 smithi007 ceph-mon[39249]: rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2024-09-18T07:58:35.458 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 07:58:35 smithi007 ceph-mon[39249]: rocksdb: Options.table_properties_collectors: CompactOnDeletionCollector (Sliding window size = 32768 Deletion trigger = 16384 Deletion ratio = 0); 2024-09-18T07:58:35.458 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 07:58:35 smithi007 ceph-mon[39249]: rocksdb: Options.inplace_update_support: 0 2024-09-18T07:58:35.458 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 07:58:35 smithi007 ceph-mon[39249]: rocksdb: Options.inplace_update_num_locks: 10000 2024-09-18T07:58:35.458 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 07:58:35 smithi007 ceph-mon[39249]: rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2024-09-18T07:58:35.459 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 07:58:35 smithi007 ceph-mon[39249]: rocksdb: Options.memtable_whole_key_filtering: 0 2024-09-18T07:58:35.459 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 07:58:35 smithi007 ceph-mon[39249]: rocksdb: Options.memtable_huge_page_size: 0 2024-09-18T07:58:35.459 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 07:58:35 smithi007 ceph-mon[39249]: rocksdb: Options.bloom_locality: 0 2024-09-18T07:58:35.459 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 07:58:35 smithi007 ceph-mon[39249]: rocksdb: Options.max_successive_merges: 0 2024-09-18T07:58:35.459 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 07:58:35 smithi007 ceph-mon[39249]: rocksdb: Options.optimize_filters_for_hits: 0 2024-09-18T07:58:35.459 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 07:58:35 smithi007 ceph-mon[39249]: rocksdb: Options.paranoid_file_checks: 0 2024-09-18T07:58:35.459 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 07:58:35 smithi007 ceph-mon[39249]: rocksdb: Options.force_consistency_checks: 1 2024-09-18T07:58:35.459 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 07:58:35 smithi007 ceph-mon[39249]: rocksdb: Options.report_bg_io_stats: 0 2024-09-18T07:58:35.459 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 07:58:35 smithi007 ceph-mon[39249]: rocksdb: Options.ttl: 2592000 2024-09-18T07:58:35.459 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 07:58:35 smithi007 ceph-mon[39249]: rocksdb: Options.periodic_compaction_seconds: 0 2024-09-18T07:58:35.459 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 07:58:35 smithi007 ceph-mon[39249]: rocksdb: Options.preclude_last_level_data_seconds: 0 2024-09-18T07:58:35.459 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 07:58:35 smithi007 ceph-mon[39249]: rocksdb: Options.preserve_internal_time_seconds: 0 2024-09-18T07:58:35.459 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 07:58:35 smithi007 ceph-mon[39249]: rocksdb: Options.enable_blob_files: false 2024-09-18T07:58:35.460 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 07:58:35 smithi007 ceph-mon[39249]: rocksdb: Options.min_blob_size: 0 2024-09-18T07:58:35.460 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 07:58:35 smithi007 ceph-mon[39249]: rocksdb: Options.blob_file_size: 268435456 2024-09-18T07:58:35.460 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 07:58:35 smithi007 ceph-mon[39249]: rocksdb: Options.blob_compression_type: NoCompression 2024-09-18T07:58:35.460 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 07:58:35 smithi007 ceph-mon[39249]: rocksdb: Options.enable_blob_garbage_collection: false 2024-09-18T07:58:35.460 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 07:58:35 smithi007 ceph-mon[39249]: rocksdb: Options.blob_garbage_collection_age_cutoff: 0.250000 2024-09-18T07:58:35.460 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 07:58:35 smithi007 ceph-mon[39249]: rocksdb: Options.blob_garbage_collection_force_threshold: 1.000000 2024-09-18T07:58:35.460 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 07:58:35 smithi007 ceph-mon[39249]: rocksdb: Options.blob_compaction_readahead_size: 0 2024-09-18T07:58:35.460 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 07:58:35 smithi007 ceph-mon[39249]: rocksdb: Options.blob_file_starting_level: 0 2024-09-18T07:58:35.460 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 07:58:35 smithi007 ceph-mon[39249]: rocksdb: Options.experimental_mempurge_threshold: 0.000000 2024-09-18T07:58:35.461 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 07:58:35 smithi007 ceph-mon[39249]: rocksdb: [db/version_set.cc:5566] Recovered from manifest file:/var/lib/ceph/mon/ceph-c/store.db/MANIFEST-000005 succeeded,manifest_file_number is 5, next_file_number is 7, last_sequence is 0, log_number is 0,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 0 2024-09-18T07:58:35.461 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 07:58:35 smithi007 ceph-mon[39249]: rocksdb: [db/version_set.cc:5581] Column family [default] (ID 0), log number is 0 2024-09-18T07:58:35.461 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 07:58:35 smithi007 ceph-mon[39249]: rocksdb: [db/db_impl/db_impl_open.cc:539] DB ID: f8e1c18b-77bb-475c-9460-c2023689e39a 2024-09-18T07:58:35.461 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 07:58:35 smithi007 ceph-mon[39249]: rocksdb: EVENT_LOG_v1 {"time_micros": 1726646315401044, "job": 1, "event": "recovery_started", "wal_files": [4]} 2024-09-18T07:58:35.461 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 07:58:35 smithi007 ceph-mon[39249]: rocksdb: [db/db_impl/db_impl_open.cc:1043] Recovering log #4 mode 2 2024-09-18T07:58:35.461 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 07:58:35 smithi007 ceph-mon[39249]: rocksdb: EVENT_LOG_v1 {"time_micros": 1726646315401429, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 8, "file_size": 1773, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 1, "largest_seqno": 5, "table_properties": {"data_size": 648, "index_size": 31, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 1, "index_value_is_delta_encoded": 1, "filter_size": 69, "raw_key_size": 115, "raw_average_key_size": 23, "raw_value_size": 526, "raw_average_value_size": 105, "num_data_blocks": 1, "num_entries": 5, "num_filter_entries": 5, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "bloomfilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[CompactOnDeletionCollector]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; max_dict_buffer_bytes=0; use_zstd_dict_trainer=1; ", "creation_time": 1726646315, "oldest_key_time": 0, "file_creation_time": 0, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "f8e1c18b-77bb-475c-9460-c2023689e39a", "db_session_id": "Z6GZEIY8TUGDDETOGV5Z", "orig_file_number": 8, "seqno_to_time_mapping": "N/A"}} 2024-09-18T07:58:35.461 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 07:58:35 smithi007 ceph-mon[39249]: rocksdb: EVENT_LOG_v1 {"time_micros": 1726646315401527, "job": 1, "event": "recovery_finished"} 2024-09-18T07:58:35.461 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 07:58:35 smithi007 ceph-mon[39249]: rocksdb: [db/version_set.cc:5047] Creating manifest 10 2024-09-18T07:58:35.461 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 07:58:35 smithi007 ceph-mon[39249]: rocksdb: [file/delete_scheduler.cc:74] Deleted file /var/lib/ceph/mon/ceph-c/store.db/000004.log immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024-09-18T07:58:35.461 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 07:58:35 smithi007 ceph-mon[39249]: rocksdb: [db/db_impl/db_impl_open.cc:1987] SstFileManager instance 0x56530d10ae00 2024-09-18T07:58:35.461 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 07:58:35 smithi007 ceph-mon[39249]: rocksdb: DB pointer 0x56530d20c000 2024-09-18T07:58:35.462 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 07:58:35 smithi007 ceph-mon[39249]: rocksdb: [db/db_impl/db_impl.cc:1109] ------- DUMPING STATS ------- 2024-09-18T07:58:35.462 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 07:58:35 smithi007 ceph-mon[39249]: rocksdb: [db/db_impl/db_impl.cc:1111] 2024-09-18T07:58:35.462 INFO:journalctl@ceph.mon.c.smithi007.stdout: ** DB Stats ** 2024-09-18T07:58:35.462 INFO:journalctl@ceph.mon.c.smithi007.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-09-18T07:58:35.462 INFO:journalctl@ceph.mon.c.smithi007.stdout: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2024-09-18T07:58:35.462 INFO:journalctl@ceph.mon.c.smithi007.stdout: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-09-18T07:58:35.462 INFO:journalctl@ceph.mon.c.smithi007.stdout: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2024-09-18T07:58:35.462 INFO:journalctl@ceph.mon.c.smithi007.stdout: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2024-09-18T07:58:35.462 INFO:journalctl@ceph.mon.c.smithi007.stdout: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-09-18T07:58:35.462 INFO:journalctl@ceph.mon.c.smithi007.stdout: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2024-09-18T07:58:35.462 INFO:journalctl@ceph.mon.c.smithi007.stdout: 2024-09-18T07:58:35.462 INFO:journalctl@ceph.mon.c.smithi007.stdout: ** Compaction Stats [default] ** 2024-09-18T07:58:35.462 INFO:journalctl@ceph.mon.c.smithi007.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-09-18T07:58:35.463 INFO:journalctl@ceph.mon.c.smithi007.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ 2024-09-18T07:58:35.463 INFO:journalctl@ceph.mon.c.smithi007.stdout: L0 1/0 1.73 KB 0.2 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 4.5 0.00 0.00 1 0.000 0 0 0.0 0.0 2024-09-18T07:58:35.463 INFO:journalctl@ceph.mon.c.smithi007.stdout: Sum 1/0 1.73 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 4.5 0.00 0.00 1 0.000 0 0 0.0 0.0 2024-09-18T07:58:35.463 INFO:journalctl@ceph.mon.c.smithi007.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 4.5 0.00 0.00 1 0.000 0 0 0.0 0.0 2024-09-18T07:58:35.463 INFO:journalctl@ceph.mon.c.smithi007.stdout: 2024-09-18T07:58:35.463 INFO:journalctl@ceph.mon.c.smithi007.stdout: ** Compaction Stats [default] ** 2024-09-18T07:58:35.463 INFO:journalctl@ceph.mon.c.smithi007.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-09-18T07:58:35.463 INFO:journalctl@ceph.mon.c.smithi007.stdout: --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-09-18T07:58:35.463 INFO:journalctl@ceph.mon.c.smithi007.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 4.5 0.00 0.00 1 0.000 0 0 0.0 0.0 2024-09-18T07:58:35.463 INFO:journalctl@ceph.mon.c.smithi007.stdout: 2024-09-18T07:58:35.463 INFO:journalctl@ceph.mon.c.smithi007.stdout: Blob file count: 0, total size: 0.0 GB, garbage size: 0.0 GB, space amp: 0.0 2024-09-18T07:58:35.463 INFO:journalctl@ceph.mon.c.smithi007.stdout: 2024-09-18T07:58:35.464 INFO:journalctl@ceph.mon.c.smithi007.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-09-18T07:58:35.464 INFO:journalctl@ceph.mon.c.smithi007.stdout: Flush(GB): cumulative 0.000, interval 0.000 2024-09-18T07:58:35.464 INFO:journalctl@ceph.mon.c.smithi007.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2024-09-18T07:58:35.464 INFO:journalctl@ceph.mon.c.smithi007.stdout: AddFile(Total Files): cumulative 0, interval 0 2024-09-18T07:58:35.464 INFO:journalctl@ceph.mon.c.smithi007.stdout: AddFile(L0 Files): cumulative 0, interval 0 2024-09-18T07:58:35.464 INFO:journalctl@ceph.mon.c.smithi007.stdout: AddFile(Keys): cumulative 0, interval 0 2024-09-18T07:58:35.464 INFO:journalctl@ceph.mon.c.smithi007.stdout: Cumulative compaction: 0.00 GB write, 0.65 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-09-18T07:58:35.464 INFO:journalctl@ceph.mon.c.smithi007.stdout: Interval compaction: 0.00 GB write, 0.65 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-09-18T07:58:35.464 INFO:journalctl@ceph.mon.c.smithi007.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-09-18T07:58:35.464 INFO:journalctl@ceph.mon.c.smithi007.stdout: Block cache BinnedLRUCache@0x56530d0db610#2 capacity: 512.00 MB usage: 0.22 KB table_size: 0 occupancy: 18446744073709551615 collections: 1 last_copies: 0 last_secs: 1e-05 secs_since: 0 2024-09-18T07:58:35.464 INFO:journalctl@ceph.mon.c.smithi007.stdout: Block cache entry stats(count,size,portion): FilterBlock(1,0.11 KB,2.08616e-05%) IndexBlock(1,0.11 KB,2.08616e-05%) Misc(1,0.00 KB,0%) 2024-09-18T07:58:35.464 INFO:journalctl@ceph.mon.c.smithi007.stdout: 2024-09-18T07:58:35.464 INFO:journalctl@ceph.mon.c.smithi007.stdout: ** File Read Latency Histogram By Level [default] ** 2024-09-18T07:58:35.464 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 07:58:35 smithi007 ceph-mon[39249]: mon.c does not exist in monmap, will attempt to join an existing cluster 2024-09-18T07:58:35.465 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 07:58:35 smithi007 ceph-mon[39249]: using public_addrv [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] 2024-09-18T07:58:35.465 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 07:58:35 smithi007 ceph-mon[39249]: starting mon.c rank -1 at public addrs [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] at bind addrs [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon_data /var/lib/ceph/mon/ceph-c fsid 779349ca-7593-11ef-bceb-c7b262605968 2024-09-18T07:58:35.465 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 07:58:35 smithi007 ceph-mon[39249]: mon.c@-1(???) e0 preinit fsid 779349ca-7593-11ef-bceb-c7b262605968 2024-09-18T07:58:35.465 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 07:58:35 smithi007 ceph-mon[39249]: mon.c@-1(synchronizing).mds e1 new map 2024-09-18T07:58:35.465 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 07:58:35 smithi007 ceph-mon[39249]: mon.c@-1(synchronizing).mds e1 print_map 2024-09-18T07:58:35.465 INFO:journalctl@ceph.mon.c.smithi007.stdout: e1 2024-09-18T07:58:35.465 INFO:journalctl@ceph.mon.c.smithi007.stdout: btime 2024-09-18T07:57:09:295514+0000 2024-09-18T07:58:35.465 INFO:journalctl@ceph.mon.c.smithi007.stdout: enable_multiple, ever_enabled_multiple: 1,1 2024-09-18T07:58:35.465 INFO:journalctl@ceph.mon.c.smithi007.stdout: default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2,11=minor log segments,12=quiesce subvolumes} 2024-09-18T07:58:35.465 INFO:journalctl@ceph.mon.c.smithi007.stdout: legacy client fscid: -1 2024-09-18T07:58:35.465 INFO:journalctl@ceph.mon.c.smithi007.stdout: 2024-09-18T07:58:35.465 INFO:journalctl@ceph.mon.c.smithi007.stdout: No filesystems configured 2024-09-18T07:58:35.465 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 07:58:35 smithi007 ceph-mon[39249]: mon.c@-1(synchronizing).osd e0 _set_cache_ratios kv ratio 0.25 inc ratio 0.375 full ratio 0.375 2024-09-18T07:58:35.466 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 07:58:35 smithi007 ceph-mon[39249]: mon.c@-1(synchronizing).osd e0 register_cache_with_pcm pcm target: 2147483648 pcm max: 1020054732 pcm min: 134217728 inc_osd_cache size: 1 2024-09-18T07:58:35.466 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 07:58:35 smithi007 ceph-mon[39249]: mon.c@-1(synchronizing).osd e1 e1: 0 total, 0 up, 0 in 2024-09-18T07:58:35.466 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 07:58:35 smithi007 ceph-mon[39249]: mon.c@-1(synchronizing).osd e2 e2: 0 total, 0 up, 0 in 2024-09-18T07:58:35.466 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 07:58:35 smithi007 ceph-mon[39249]: mon.c@-1(synchronizing).osd e3 e3: 0 total, 0 up, 0 in 2024-09-18T07:58:35.466 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 07:58:35 smithi007 ceph-mon[39249]: mon.c@-1(synchronizing).osd e4 e4: 0 total, 0 up, 0 in 2024-09-18T07:58:35.466 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 07:58:35 smithi007 ceph-mon[39249]: mon.c@-1(synchronizing).osd e4 crush map has features 3314932999778484224, adjusting msgr requires 2024-09-18T07:58:35.466 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 07:58:35 smithi007 ceph-mon[39249]: mon.c@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-09-18T07:58:35.466 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 07:58:35 smithi007 ceph-mon[39249]: mon.c@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-09-18T07:58:35.466 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 07:58:35 smithi007 ceph-mon[39249]: mon.c@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-09-18T07:58:35.466 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 07:58:35 smithi007 ceph-mon[39249]: from='client.? 172.21.15.7:0/118748185' entity='client.admin' cmd='[{"prefix": "osd crush tunables", "profile": "default"}]': finished 2024-09-18T07:58:35.466 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 07:58:35 smithi007 ceph-mon[39249]: osdmap e4: 0 total, 0 up, 0 in 2024-09-18T07:58:35.466 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 07:58:35 smithi007 ceph-mon[39249]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' 2024-09-18T07:58:35.466 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 07:58:35 smithi007 ceph-mon[39249]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' 2024-09-18T07:58:35.466 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 07:58:35 smithi007 ceph-mon[39249]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' 2024-09-18T07:58:35.466 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 07:58:35 smithi007 ceph-mon[39249]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' 2024-09-18T07:58:35.467 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 07:58:35 smithi007 ceph-mon[39249]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd/host:smithi050", "name": "osd_memory_target"} : dispatch 2024-09-18T07:58:35.467 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 07:58:35 smithi007 ceph-mon[39249]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-18T07:58:35.467 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 07:58:35 smithi007 ceph-mon[39249]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-18T07:58:35.467 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 07:58:35 smithi007 ceph-mon[39249]: pgmap v6: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-18T07:58:35.467 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 07:58:35 smithi007 ceph-mon[39249]: Updating smithi050:/etc/ceph/ceph.conf 2024-09-18T07:58:35.467 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 07:58:35 smithi007 ceph-mon[39249]: Updating smithi050:/var/lib/ceph/779349ca-7593-11ef-bceb-c7b262605968/config/ceph.conf 2024-09-18T07:58:35.467 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 07:58:35 smithi007 ceph-mon[39249]: Updating smithi050:/etc/ceph/ceph.client.admin.keyring 2024-09-18T07:58:35.467 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 07:58:35 smithi007 ceph-mon[39249]: Updating smithi050:/var/lib/ceph/779349ca-7593-11ef-bceb-c7b262605968/config/ceph.client.admin.keyring 2024-09-18T07:58:35.467 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 07:58:35 smithi007 ceph-mon[39249]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' 2024-09-18T07:58:35.467 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 07:58:35 smithi007 ceph-mon[39249]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' 2024-09-18T07:58:35.467 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 07:58:35 smithi007 ceph-mon[39249]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' 2024-09-18T07:58:35.467 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 07:58:35 smithi007 ceph-mon[39249]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' 2024-09-18T07:58:35.467 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 07:58:35 smithi007 ceph-mon[39249]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-18T07:58:35.468 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 07:58:35 smithi007 ceph-mon[39249]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-18T07:58:35.468 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 07:58:35 smithi007 ceph-mon[39249]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-18T07:58:35.468 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 07:58:35 smithi007 ceph-mon[39249]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' 2024-09-18T07:58:35.468 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 07:58:35 smithi007 ceph-mon[39249]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' cmd={"prefix": "auth get", "entity": "mon."} : dispatch 2024-09-18T07:58:35.468 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 07:58:35 smithi007 ceph-mon[39249]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-18T07:58:35.468 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 07:58:35 smithi007 ceph-mon[39249]: from='client.14186 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "placement": "3;smithi007:172.21.15.7=a;smithi007:[v2:172.21.15.7:3301,v1:172.21.15.7:6790]=c;smithi050:172.21.15.50=b", "target": ["mon-mgr", ""]}]: dispatch 2024-09-18T07:58:35.468 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 07:58:35 smithi007 ceph-mon[39249]: Saving service mon spec with placement smithi007:172.21.15.7=a;smithi007:[v2:172.21.15.7:3301,v1:172.21.15.7:6790]=c;smithi050:172.21.15.50=b;count:3 2024-09-18T07:58:35.468 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 07:58:35 smithi007 ceph-mon[39249]: Deploying daemon mon.b on smithi050 2024-09-18T07:58:35.468 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 07:58:35 smithi007 ceph-mon[39249]: pgmap v7: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-18T07:58:35.468 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 07:58:35 smithi007 ceph-mon[39249]: pgmap v8: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-18T07:58:35.468 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 07:58:35 smithi007 ceph-mon[39249]: from='client.? 172.21.15.50:0/1445195752' entity='client.admin' cmd={"prefix": "mon dump", "format": "json"} : dispatch 2024-09-18T07:58:35.468 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 07:58:35 smithi007 ceph-mon[39249]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "a"} : dispatch 2024-09-18T07:58:35.468 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 07:58:35 smithi007 ceph-mon[39249]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-09-18T07:58:35.469 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 07:58:35 smithi007 ceph-mon[39249]: mon.a calling monitor election 2024-09-18T07:58:35.469 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 07:58:35 smithi007 ceph-mon[39249]: pgmap v9: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-18T07:58:35.469 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 07:58:35 smithi007 ceph-mon[39249]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-09-18T07:58:35.469 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 07:58:35 smithi007 ceph-mon[39249]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-09-18T07:58:35.469 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 07:58:35 smithi007 ceph-mon[39249]: mon.b calling monitor election 2024-09-18T07:58:35.469 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 07:58:35 smithi007 ceph-mon[39249]: pgmap v10: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-18T07:58:35.469 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 07:58:35 smithi007 ceph-mon[39249]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-09-18T07:58:35.469 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 07:58:35 smithi007 ceph-mon[39249]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-09-18T07:58:35.469 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 07:58:35 smithi007 ceph-mon[39249]: pgmap v11: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-18T07:58:35.469 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 07:58:35 smithi007 ceph-mon[39249]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-09-18T07:58:35.469 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 07:58:35 smithi007 ceph-mon[39249]: mon.a is new leader, mons a,b in quorum (ranks 0,1) 2024-09-18T07:58:35.469 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 07:58:35 smithi007 ceph-mon[39249]: monmap epoch 2 2024-09-18T07:58:35.469 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 07:58:35 smithi007 ceph-mon[39249]: fsid 779349ca-7593-11ef-bceb-c7b262605968 2024-09-18T07:58:35.470 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 07:58:35 smithi007 ceph-mon[39249]: last_changed 2024-09-18T07:58:27.121545+0000 2024-09-18T07:58:35.470 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 07:58:35 smithi007 ceph-mon[39249]: created 2024-09-18T07:57:05.327907+0000 2024-09-18T07:58:35.470 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 07:58:35 smithi007 ceph-mon[39249]: min_mon_release 19 (squid) 2024-09-18T07:58:35.470 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 07:58:35 smithi007 ceph-mon[39249]: election_strategy: 1 2024-09-18T07:58:35.470 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 07:58:35 smithi007 ceph-mon[39249]: 0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2024-09-18T07:58:35.470 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 07:58:35 smithi007 ceph-mon[39249]: 1: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.b 2024-09-18T07:58:35.470 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 07:58:35 smithi007 ceph-mon[39249]: fsmap 2024-09-18T07:58:35.470 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 07:58:35 smithi007 ceph-mon[39249]: osdmap e4: 0 total, 0 up, 0 in 2024-09-18T07:58:35.470 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 07:58:35 smithi007 ceph-mon[39249]: mgrmap e13: y(active, since 34s) 2024-09-18T07:58:35.470 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 07:58:35 smithi007 ceph-mon[39249]: overall HEALTH_OK 2024-09-18T07:58:35.470 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 07:58:35 smithi007 ceph-mon[39249]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' 2024-09-18T07:58:35.470 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 07:58:35 smithi007 ceph-mon[39249]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' 2024-09-18T07:58:35.470 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 07:58:35 smithi007 ceph-mon[39249]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' 2024-09-18T07:58:35.471 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 07:58:35 smithi007 ceph-mon[39249]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' cmd={"prefix": "auth get", "entity": "mon."} : dispatch 2024-09-18T07:58:35.471 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 07:58:35 smithi007 ceph-mon[39249]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-18T07:58:35.471 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 07:58:35 smithi007 ceph-mon[39249]: Deploying daemon mon.c on smithi007 2024-09-18T07:58:35.471 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 07:58:35 smithi007 ceph-mon[39249]: from='client.? 172.21.15.50:0/1788123659' entity='client.admin' cmd={"prefix": "mon dump", "format": "json"} : dispatch 2024-09-18T07:58:35.471 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 07:58:35 smithi007 ceph-mon[39249]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-09-18T07:58:35.471 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 07:58:35 smithi007 ceph-mon[39249]: pgmap v12: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-18T07:58:35.471 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 07:58:35 smithi007 ceph-mon[39249]: from='client.? 172.21.15.50:0/393846258' entity='client.admin' cmd={"prefix": "mon dump", "format": "json"} : dispatch 2024-09-18T07:58:35.471 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 07:58:35 smithi007 ceph-mon[39249]: mon.c@-1(synchronizing).paxosservice(auth 1..3) refresh upgraded, format 0 -> 3 2024-09-18T07:58:35.733 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 07:58:35 smithi007 bash[39230]: 3898467ba9eb184258f0533b36e37a04dbe605702e07917009b63fb31555588a 2024-09-18T07:58:35.733 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 07:58:35 smithi007 systemd[1]: Started Ceph mon.c for 779349ca-7593-11ef-bceb-c7b262605968. 2024-09-18T07:58:36.550 INFO:tasks.cephadm:Waiting for 3 mons in monmap... 2024-09-18T07:58:36.551 DEBUG:teuthology.orchestra.run.smithi050:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 779349ca-7593-11ef-bceb-c7b262605968 -- ceph mon dump -f json 2024-09-18T07:58:36.811 INFO:teuthology.orchestra.run.smithi050.stderr:Inferring config /var/lib/ceph/779349ca-7593-11ef-bceb-c7b262605968/mon.b/config 2024-09-18T07:58:40.797 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-09-18T07:58:40.797 INFO:teuthology.orchestra.run.smithi050.stdout:{"epoch":3,"fsid":"779349ca-7593-11ef-bceb-c7b262605968","modified":"2024-09-18T07:58:35.425064Z","created":"2024-09-18T07:57:05.327907Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:3300","nonce":0},{"type":"v1","addr":"172.21.15.7:6789","nonce":0}]},"addr":"172.21.15.7:6789/0","public_addr":"172.21.15.7:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:3300","nonce":0},{"type":"v1","addr":"172.21.15.50:6789","nonce":0}]},"addr":"172.21.15.50:6789/0","public_addr":"172.21.15.50:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:3301","nonce":0},{"type":"v1","addr":"172.21.15.7:6790","nonce":0}]},"addr":"172.21.15.7:6790/0","public_addr":"172.21.15.7:6790/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0,1,2]} 2024-09-18T07:58:40.798 INFO:teuthology.orchestra.run.smithi050.stderr:dumped monmap epoch 3 2024-09-18T07:58:40.798 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 07:58:40 smithi050 ceph-mon[32371]: pgmap v13: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-18T07:58:40.798 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 07:58:40 smithi050 ceph-mon[32371]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "a"} : dispatch 2024-09-18T07:58:40.798 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 07:58:40 smithi050 ceph-mon[32371]: mon.a calling monitor election 2024-09-18T07:58:40.798 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 07:58:40 smithi050 ceph-mon[32371]: mon.b calling monitor election 2024-09-18T07:58:40.798 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 07:58:40 smithi050 ceph-mon[32371]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-09-18T07:58:40.798 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 07:58:40 smithi050 ceph-mon[32371]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-09-18T07:58:40.798 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 07:58:40 smithi050 ceph-mon[32371]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-09-18T07:58:40.798 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 07:58:40 smithi050 ceph-mon[32371]: pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-18T07:58:40.799 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 07:58:40 smithi050 ceph-mon[32371]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-09-18T07:58:40.799 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 07:58:40 smithi050 ceph-mon[32371]: mon.c calling monitor election 2024-09-18T07:58:40.799 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 07:58:40 smithi050 ceph-mon[32371]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-09-18T07:58:40.799 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 07:58:40 smithi050 ceph-mon[32371]: pgmap v15: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-18T07:58:40.799 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 07:58:40 smithi050 ceph-mon[32371]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-09-18T07:58:40.799 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 07:58:40 smithi050 ceph-mon[32371]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-09-18T07:58:40.799 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 07:58:40 smithi050 ceph-mon[32371]: mon.a is new leader, mons a,b,c in quorum (ranks 0,1,2) 2024-09-18T07:58:40.799 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 07:58:40 smithi050 ceph-mon[32371]: monmap epoch 3 2024-09-18T07:58:40.799 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 07:58:40 smithi050 ceph-mon[32371]: fsid 779349ca-7593-11ef-bceb-c7b262605968 2024-09-18T07:58:40.799 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 07:58:40 smithi050 ceph-mon[32371]: last_changed 2024-09-18T07:58:35.425064+0000 2024-09-18T07:58:40.799 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 07:58:40 smithi050 ceph-mon[32371]: created 2024-09-18T07:57:05.327907+0000 2024-09-18T07:58:40.799 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 07:58:40 smithi050 ceph-mon[32371]: min_mon_release 19 (squid) 2024-09-18T07:58:40.800 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 07:58:40 smithi050 ceph-mon[32371]: election_strategy: 1 2024-09-18T07:58:40.800 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 07:58:40 smithi050 ceph-mon[32371]: 0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2024-09-18T07:58:40.800 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 07:58:40 smithi050 ceph-mon[32371]: 1: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.b 2024-09-18T07:58:40.800 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 07:58:40 smithi050 ceph-mon[32371]: 2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2024-09-18T07:58:40.800 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 07:58:40 smithi050 ceph-mon[32371]: fsmap 2024-09-18T07:58:40.800 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 07:58:40 smithi050 ceph-mon[32371]: osdmap e4: 0 total, 0 up, 0 in 2024-09-18T07:58:40.800 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 07:58:40 smithi050 ceph-mon[32371]: mgrmap e13: y(active, since 43s) 2024-09-18T07:58:40.800 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 07:58:40 smithi050 ceph-mon[32371]: overall HEALTH_OK 2024-09-18T07:58:40.800 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 07:58:40 smithi050 ceph-mon[32371]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' 2024-09-18T07:58:40.800 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 07:58:40 smithi050 ceph-mon[32371]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' 2024-09-18T07:58:40.800 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 07:58:40 smithi050 ceph-mon[32371]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' 2024-09-18T07:58:40.800 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 07:58:40 smithi050 ceph-mon[32371]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' 2024-09-18T07:58:40.982 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 07:58:40 smithi007 ceph-mon[29459]: pgmap v13: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-18T07:58:40.982 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 07:58:40 smithi007 ceph-mon[29459]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "a"} : dispatch 2024-09-18T07:58:40.982 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 07:58:40 smithi007 ceph-mon[29459]: mon.a calling monitor election 2024-09-18T07:58:40.982 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 07:58:40 smithi007 ceph-mon[29459]: mon.b calling monitor election 2024-09-18T07:58:40.982 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 07:58:40 smithi007 ceph-mon[29459]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-09-18T07:58:40.982 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 07:58:40 smithi007 ceph-mon[29459]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-09-18T07:58:40.982 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 07:58:40 smithi007 ceph-mon[29459]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-09-18T07:58:40.982 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 07:58:40 smithi007 ceph-mon[29459]: pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-18T07:58:40.983 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 07:58:40 smithi007 ceph-mon[29459]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-09-18T07:58:40.983 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 07:58:40 smithi007 ceph-mon[29459]: mon.c calling monitor election 2024-09-18T07:58:40.983 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 07:58:40 smithi007 ceph-mon[29459]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-09-18T07:58:40.983 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 07:58:40 smithi007 ceph-mon[29459]: pgmap v15: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-18T07:58:40.983 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 07:58:40 smithi007 ceph-mon[29459]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-09-18T07:58:40.983 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 07:58:40 smithi007 ceph-mon[29459]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-09-18T07:58:40.983 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 07:58:40 smithi007 ceph-mon[29459]: mon.a is new leader, mons a,b,c in quorum (ranks 0,1,2) 2024-09-18T07:58:40.983 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 07:58:40 smithi007 ceph-mon[29459]: monmap epoch 3 2024-09-18T07:58:40.983 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 07:58:40 smithi007 ceph-mon[29459]: fsid 779349ca-7593-11ef-bceb-c7b262605968 2024-09-18T07:58:40.983 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 07:58:40 smithi007 ceph-mon[29459]: last_changed 2024-09-18T07:58:35.425064+0000 2024-09-18T07:58:40.983 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 07:58:40 smithi007 ceph-mon[29459]: created 2024-09-18T07:57:05.327907+0000 2024-09-18T07:58:40.983 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 07:58:40 smithi007 ceph-mon[29459]: min_mon_release 19 (squid) 2024-09-18T07:58:40.983 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 07:58:40 smithi007 ceph-mon[29459]: election_strategy: 1 2024-09-18T07:58:40.984 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 07:58:40 smithi007 ceph-mon[29459]: 0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2024-09-18T07:58:40.984 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 07:58:40 smithi007 ceph-mon[29459]: 1: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.b 2024-09-18T07:58:40.984 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 07:58:40 smithi007 ceph-mon[29459]: 2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2024-09-18T07:58:40.984 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 07:58:40 smithi007 ceph-mon[29459]: fsmap 2024-09-18T07:58:40.984 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 07:58:40 smithi007 ceph-mon[29459]: osdmap e4: 0 total, 0 up, 0 in 2024-09-18T07:58:40.984 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 07:58:40 smithi007 ceph-mon[29459]: mgrmap e13: y(active, since 43s) 2024-09-18T07:58:40.984 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 07:58:40 smithi007 ceph-mon[29459]: overall HEALTH_OK 2024-09-18T07:58:40.984 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 07:58:40 smithi007 ceph-mon[29459]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' 2024-09-18T07:58:40.984 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 07:58:40 smithi007 ceph-mon[29459]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' 2024-09-18T07:58:40.984 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 07:58:40 smithi007 ceph-mon[29459]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' 2024-09-18T07:58:40.984 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 07:58:40 smithi007 ceph-mon[29459]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' 2024-09-18T07:58:40.985 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 07:58:40 smithi007 ceph-mon[39249]: pgmap v13: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-18T07:58:40.985 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 07:58:40 smithi007 ceph-mon[39249]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "a"} : dispatch 2024-09-18T07:58:40.985 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 07:58:40 smithi007 ceph-mon[39249]: mon.a calling monitor election 2024-09-18T07:58:40.985 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 07:58:40 smithi007 ceph-mon[39249]: mon.b calling monitor election 2024-09-18T07:58:40.985 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 07:58:40 smithi007 ceph-mon[39249]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-09-18T07:58:40.985 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 07:58:40 smithi007 ceph-mon[39249]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-09-18T07:58:40.985 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 07:58:40 smithi007 ceph-mon[39249]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-09-18T07:58:40.985 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 07:58:40 smithi007 ceph-mon[39249]: pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-18T07:58:40.985 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 07:58:40 smithi007 ceph-mon[39249]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-09-18T07:58:40.985 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 07:58:40 smithi007 ceph-mon[39249]: mon.c calling monitor election 2024-09-18T07:58:40.985 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 07:58:40 smithi007 ceph-mon[39249]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-09-18T07:58:40.985 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 07:58:40 smithi007 ceph-mon[39249]: pgmap v15: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-18T07:58:40.986 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 07:58:40 smithi007 ceph-mon[39249]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-09-18T07:58:40.986 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 07:58:40 smithi007 ceph-mon[39249]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-09-18T07:58:40.986 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 07:58:40 smithi007 ceph-mon[39249]: mon.a is new leader, mons a,b,c in quorum (ranks 0,1,2) 2024-09-18T07:58:40.986 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 07:58:40 smithi007 ceph-mon[39249]: monmap epoch 3 2024-09-18T07:58:40.986 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 07:58:40 smithi007 ceph-mon[39249]: fsid 779349ca-7593-11ef-bceb-c7b262605968 2024-09-18T07:58:40.986 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 07:58:40 smithi007 ceph-mon[39249]: last_changed 2024-09-18T07:58:35.425064+0000 2024-09-18T07:58:40.986 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 07:58:40 smithi007 ceph-mon[39249]: created 2024-09-18T07:57:05.327907+0000 2024-09-18T07:58:40.986 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 07:58:40 smithi007 ceph-mon[39249]: min_mon_release 19 (squid) 2024-09-18T07:58:40.986 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 07:58:40 smithi007 ceph-mon[39249]: election_strategy: 1 2024-09-18T07:58:40.986 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 07:58:40 smithi007 ceph-mon[39249]: 0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2024-09-18T07:58:40.986 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 07:58:40 smithi007 ceph-mon[39249]: 1: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.b 2024-09-18T07:58:40.986 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 07:58:40 smithi007 ceph-mon[39249]: 2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2024-09-18T07:58:40.986 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 07:58:40 smithi007 ceph-mon[39249]: fsmap 2024-09-18T07:58:40.987 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 07:58:40 smithi007 ceph-mon[39249]: osdmap e4: 0 total, 0 up, 0 in 2024-09-18T07:58:40.987 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 07:58:40 smithi007 ceph-mon[39249]: mgrmap e13: y(active, since 43s) 2024-09-18T07:58:40.987 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 07:58:40 smithi007 ceph-mon[39249]: overall HEALTH_OK 2024-09-18T07:58:40.987 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 07:58:40 smithi007 ceph-mon[39249]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' 2024-09-18T07:58:40.987 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 07:58:40 smithi007 ceph-mon[39249]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' 2024-09-18T07:58:40.987 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 07:58:40 smithi007 ceph-mon[39249]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' 2024-09-18T07:58:40.987 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 07:58:40 smithi007 ceph-mon[39249]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' 2024-09-18T07:58:41.360 INFO:tasks.cephadm:Generating final ceph.conf file... 2024-09-18T07:58:41.360 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 779349ca-7593-11ef-bceb-c7b262605968 -- ceph config generate-minimal-conf 2024-09-18T07:58:41.635 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/779349ca-7593-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T07:58:41.656 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 07:58:41 smithi050 ceph-mon[32371]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-18T07:58:41.656 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 07:58:41 smithi050 ceph-mon[32371]: from='client.? 172.21.15.50:0/2711056462' entity='client.admin' cmd={"prefix": "mon dump", "format": "json"} : dispatch 2024-09-18T07:58:41.656 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 07:58:41 smithi050 ceph-mon[32371]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-09-18T07:58:41.731 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 07:58:41 smithi007 ceph-mon[29459]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-18T07:58:41.731 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 07:58:41 smithi007 ceph-mon[29459]: from='client.? 172.21.15.50:0/2711056462' entity='client.admin' cmd={"prefix": "mon dump", "format": "json"} : dispatch 2024-09-18T07:58:41.731 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 07:58:41 smithi007 ceph-mon[29459]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-09-18T07:58:41.731 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 07:58:41 smithi007 ceph-mon[39249]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-18T07:58:41.731 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 07:58:41 smithi007 ceph-mon[39249]: from='client.? 172.21.15.50:0/2711056462' entity='client.admin' cmd={"prefix": "mon dump", "format": "json"} : dispatch 2024-09-18T07:58:41.731 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 07:58:41 smithi007 ceph-mon[39249]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-09-18T07:58:42.289 INFO:teuthology.orchestra.run.smithi007.stdout:# minimal ceph.conf for 779349ca-7593-11ef-bceb-c7b262605968 2024-09-18T07:58:42.290 INFO:teuthology.orchestra.run.smithi007.stdout:[global] 2024-09-18T07:58:42.290 INFO:teuthology.orchestra.run.smithi007.stdout: fsid = 779349ca-7593-11ef-bceb-c7b262605968 2024-09-18T07:58:42.290 INFO:teuthology.orchestra.run.smithi007.stdout: mon_host = [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] 2024-09-18T07:58:42.731 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 07:58:42 smithi007 ceph-mon[29459]: pgmap v16: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-18T07:58:42.732 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 07:58:42 smithi007 ceph-mon[29459]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' 2024-09-18T07:58:42.732 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 07:58:42 smithi007 ceph-mon[29459]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' 2024-09-18T07:58:42.732 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 07:58:42 smithi007 ceph-mon[29459]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' 2024-09-18T07:58:42.732 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 07:58:42 smithi007 ceph-mon[29459]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' 2024-09-18T07:58:42.732 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 07:58:42 smithi007 ceph-mon[29459]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-18T07:58:42.732 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 07:58:42 smithi007 ceph-mon[29459]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-18T07:58:42.732 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 07:58:42 smithi007 ceph-mon[29459]: Updating smithi007:/etc/ceph/ceph.conf 2024-09-18T07:58:42.732 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 07:58:42 smithi007 ceph-mon[29459]: Updating smithi050:/etc/ceph/ceph.conf 2024-09-18T07:58:42.732 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 07:58:42 smithi007 ceph-mon[29459]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' 2024-09-18T07:58:42.732 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 07:58:42 smithi007 ceph-mon[29459]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' 2024-09-18T07:58:42.732 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 07:58:42 smithi007 ceph-mon[29459]: from='client.? 172.21.15.7:0/140517025' entity='client.admin' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-18T07:58:42.733 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 07:58:42 smithi007 ceph-mon[29459]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' 2024-09-18T07:58:42.733 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 07:58:42 smithi007 ceph-mon[29459]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' 2024-09-18T07:58:42.733 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 07:58:42 smithi007 ceph-mon[29459]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' 2024-09-18T07:58:42.733 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 07:58:42 smithi007 ceph-mon[29459]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' 2024-09-18T07:58:42.733 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 07:58:42 smithi007 ceph-mon[29459]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' 2024-09-18T07:58:42.733 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 07:58:42 smithi007 ceph-mon[29459]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' 2024-09-18T07:58:42.733 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 07:58:42 smithi007 ceph-mon[29459]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' 2024-09-18T07:58:42.733 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 07:58:42 smithi007 ceph-mon[29459]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' cmd={"prefix": "auth get", "entity": "mon."} : dispatch 2024-09-18T07:58:42.733 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 07:58:42 smithi007 ceph-mon[29459]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' cmd={"prefix": "config get", "who": "mon", "key": "public_network"} : dispatch 2024-09-18T07:58:42.733 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 07:58:42 smithi007 ceph-mon[29459]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-18T07:58:42.733 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 07:58:42 smithi007 ceph-mon[39249]: pgmap v16: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-18T07:58:42.734 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 07:58:42 smithi007 ceph-mon[39249]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' 2024-09-18T07:58:42.734 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 07:58:42 smithi007 ceph-mon[39249]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' 2024-09-18T07:58:42.734 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 07:58:42 smithi007 ceph-mon[39249]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' 2024-09-18T07:58:42.734 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 07:58:42 smithi007 ceph-mon[39249]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' 2024-09-18T07:58:42.734 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 07:58:42 smithi007 ceph-mon[39249]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-18T07:58:42.734 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 07:58:42 smithi007 ceph-mon[39249]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-18T07:58:42.734 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 07:58:42 smithi007 ceph-mon[39249]: Updating smithi007:/etc/ceph/ceph.conf 2024-09-18T07:58:42.734 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 07:58:42 smithi007 ceph-mon[39249]: Updating smithi050:/etc/ceph/ceph.conf 2024-09-18T07:58:42.734 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 07:58:42 smithi007 ceph-mon[39249]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' 2024-09-18T07:58:42.734 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 07:58:42 smithi007 ceph-mon[39249]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' 2024-09-18T07:58:42.734 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 07:58:42 smithi007 ceph-mon[39249]: from='client.? 172.21.15.7:0/140517025' entity='client.admin' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-18T07:58:42.734 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 07:58:42 smithi007 ceph-mon[39249]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' 2024-09-18T07:58:42.735 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 07:58:42 smithi007 ceph-mon[39249]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' 2024-09-18T07:58:42.735 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 07:58:42 smithi007 ceph-mon[39249]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' 2024-09-18T07:58:42.735 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 07:58:42 smithi007 ceph-mon[39249]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' 2024-09-18T07:58:42.735 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 07:58:42 smithi007 ceph-mon[39249]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' 2024-09-18T07:58:42.735 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 07:58:42 smithi007 ceph-mon[39249]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' 2024-09-18T07:58:42.735 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 07:58:42 smithi007 ceph-mon[39249]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' 2024-09-18T07:58:42.736 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 07:58:42 smithi007 ceph-mon[39249]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' cmd={"prefix": "auth get", "entity": "mon."} : dispatch 2024-09-18T07:58:42.736 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 07:58:42 smithi007 ceph-mon[39249]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' cmd={"prefix": "config get", "who": "mon", "key": "public_network"} : dispatch 2024-09-18T07:58:42.736 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 07:58:42 smithi007 ceph-mon[39249]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-18T07:58:42.907 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 07:58:42 smithi050 ceph-mon[32371]: pgmap v16: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-18T07:58:42.907 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 07:58:42 smithi050 ceph-mon[32371]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' 2024-09-18T07:58:42.907 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 07:58:42 smithi050 ceph-mon[32371]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' 2024-09-18T07:58:42.907 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 07:58:42 smithi050 ceph-mon[32371]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' 2024-09-18T07:58:42.907 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 07:58:42 smithi050 ceph-mon[32371]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' 2024-09-18T07:58:42.907 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 07:58:42 smithi050 ceph-mon[32371]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-18T07:58:42.908 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 07:58:42 smithi050 ceph-mon[32371]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-18T07:58:42.908 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 07:58:42 smithi050 ceph-mon[32371]: Updating smithi007:/etc/ceph/ceph.conf 2024-09-18T07:58:42.908 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 07:58:42 smithi050 ceph-mon[32371]: Updating smithi050:/etc/ceph/ceph.conf 2024-09-18T07:58:42.908 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 07:58:42 smithi050 ceph-mon[32371]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' 2024-09-18T07:58:42.908 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 07:58:42 smithi050 ceph-mon[32371]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' 2024-09-18T07:58:42.908 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 07:58:42 smithi050 ceph-mon[32371]: from='client.? 172.21.15.7:0/140517025' entity='client.admin' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-18T07:58:42.908 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 07:58:42 smithi050 ceph-mon[32371]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' 2024-09-18T07:58:42.908 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 07:58:42 smithi050 ceph-mon[32371]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' 2024-09-18T07:58:42.908 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 07:58:42 smithi050 ceph-mon[32371]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' 2024-09-18T07:58:42.909 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 07:58:42 smithi050 ceph-mon[32371]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' 2024-09-18T07:58:42.909 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 07:58:42 smithi050 ceph-mon[32371]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' 2024-09-18T07:58:42.909 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 07:58:42 smithi050 ceph-mon[32371]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' 2024-09-18T07:58:42.909 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 07:58:42 smithi050 ceph-mon[32371]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' 2024-09-18T07:58:42.909 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 07:58:42 smithi050 ceph-mon[32371]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' cmd={"prefix": "auth get", "entity": "mon."} : dispatch 2024-09-18T07:58:42.909 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 07:58:42 smithi050 ceph-mon[32371]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' cmd={"prefix": "config get", "who": "mon", "key": "public_network"} : dispatch 2024-09-18T07:58:42.909 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 07:58:42 smithi050 ceph-mon[32371]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-18T07:58:42.930 INFO:tasks.cephadm:Distributing (final) config and client.admin keyring... 2024-09-18T07:58:42.931 DEBUG:teuthology.orchestra.run.smithi007:> set -ex 2024-09-18T07:58:42.931 DEBUG:teuthology.orchestra.run.smithi007:> sudo dd of=/etc/ceph/ceph.conf 2024-09-18T07:58:42.959 DEBUG:teuthology.orchestra.run.smithi007:> set -ex 2024-09-18T07:58:42.959 DEBUG:teuthology.orchestra.run.smithi007:> sudo dd of=/etc/ceph/ceph.client.admin.keyring 2024-09-18T07:58:43.026 DEBUG:teuthology.orchestra.run.smithi050:> set -ex 2024-09-18T07:58:43.026 DEBUG:teuthology.orchestra.run.smithi050:> sudo dd of=/etc/ceph/ceph.conf 2024-09-18T07:58:43.057 DEBUG:teuthology.orchestra.run.smithi050:> set -ex 2024-09-18T07:58:43.058 DEBUG:teuthology.orchestra.run.smithi050:> sudo dd of=/etc/ceph/ceph.client.admin.keyring 2024-09-18T07:58:43.127 INFO:tasks.cephadm:Adding mgr.y on smithi007 2024-09-18T07:58:43.127 INFO:tasks.cephadm:Adding mgr.x on smithi050 2024-09-18T07:58:43.127 DEBUG:teuthology.orchestra.run.smithi050:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 779349ca-7593-11ef-bceb-c7b262605968 -- ceph orch apply mgr '2;smithi007=y;smithi050=x' 2024-09-18T07:58:43.415 INFO:teuthology.orchestra.run.smithi050.stderr:Inferring config /var/lib/ceph/779349ca-7593-11ef-bceb-c7b262605968/mon.b/config 2024-09-18T07:58:43.801 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 07:58:43 smithi050 ceph-mon[32371]: Updating smithi050:/var/lib/ceph/779349ca-7593-11ef-bceb-c7b262605968/config/ceph.conf 2024-09-18T07:58:43.802 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 07:58:43 smithi050 ceph-mon[32371]: Updating smithi007:/var/lib/ceph/779349ca-7593-11ef-bceb-c7b262605968/config/ceph.conf 2024-09-18T07:58:43.802 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 07:58:43 smithi050 ceph-mon[32371]: Reconfiguring mon.a (unknown last config time)... 2024-09-18T07:58:43.802 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 07:58:43 smithi050 ceph-mon[32371]: Reconfiguring daemon mon.a on smithi007 2024-09-18T07:58:43.802 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 07:58:43 smithi050 ceph-mon[32371]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' 2024-09-18T07:58:43.974 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 07:58:43 smithi007 ceph-mon[29459]: Updating smithi050:/var/lib/ceph/779349ca-7593-11ef-bceb-c7b262605968/config/ceph.conf 2024-09-18T07:58:43.974 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 07:58:43 smithi007 ceph-mon[29459]: Updating smithi007:/var/lib/ceph/779349ca-7593-11ef-bceb-c7b262605968/config/ceph.conf 2024-09-18T07:58:43.975 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 07:58:43 smithi007 ceph-mon[29459]: Reconfiguring mon.a (unknown last config time)... 2024-09-18T07:58:43.975 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 07:58:43 smithi007 ceph-mon[29459]: Reconfiguring daemon mon.a on smithi007 2024-09-18T07:58:43.975 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 07:58:43 smithi007 ceph-mon[29459]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' 2024-09-18T07:58:43.975 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 07:58:43 smithi007 ceph-mon[39249]: Updating smithi050:/var/lib/ceph/779349ca-7593-11ef-bceb-c7b262605968/config/ceph.conf 2024-09-18T07:58:43.975 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 07:58:43 smithi007 ceph-mon[39249]: Updating smithi007:/var/lib/ceph/779349ca-7593-11ef-bceb-c7b262605968/config/ceph.conf 2024-09-18T07:58:43.975 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 07:58:43 smithi007 ceph-mon[39249]: Reconfiguring mon.a (unknown last config time)... 2024-09-18T07:58:43.975 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 07:58:43 smithi007 ceph-mon[39249]: Reconfiguring daemon mon.a on smithi007 2024-09-18T07:58:43.975 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 07:58:43 smithi007 ceph-mon[39249]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' 2024-09-18T07:58:44.214 INFO:teuthology.orchestra.run.smithi050.stdout:Scheduled mgr update... 2024-09-18T07:58:44.798 DEBUG:teuthology.orchestra.run.smithi050:mgr.x> sudo journalctl -f -n 0 -u ceph-779349ca-7593-11ef-bceb-c7b262605968@mgr.x.service 2024-09-18T07:58:44.801 DEBUG:tasks.cephadm:set 0 configs 2024-09-18T07:58:44.801 INFO:tasks.cephadm:Deploying OSDs... 2024-09-18T07:58:44.801 DEBUG:teuthology.orchestra.run.smithi007:> set -ex 2024-09-18T07:58:44.801 DEBUG:teuthology.orchestra.run.smithi007:> dd if=/scratch_devs of=/dev/stdout 2024-09-18T07:58:44.819 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-09-18T07:58:44.819 DEBUG:teuthology.orchestra.run.smithi007:> stat /dev/vg_nvme/lv_1 2024-09-18T07:58:44.875 INFO:teuthology.orchestra.run.smithi007.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-09-18T07:58:44.875 INFO:teuthology.orchestra.run.smithi007.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-18T07:58:44.875 INFO:teuthology.orchestra.run.smithi007.stdout:Device: 5h/5d Inode: 969 Links: 1 2024-09-18T07:58:44.875 INFO:teuthology.orchestra.run.smithi007.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-18T07:58:44.875 INFO:teuthology.orchestra.run.smithi007.stdout:Context: system_u:object_r:device_t:s0 2024-09-18T07:58:44.875 INFO:teuthology.orchestra.run.smithi007.stdout:Access: 2024-09-18 07:58:34.878477948 +0000 2024-09-18T07:58:44.875 INFO:teuthology.orchestra.run.smithi007.stdout:Modify: 2024-09-18 07:53:31.757431708 +0000 2024-09-18T07:58:44.875 INFO:teuthology.orchestra.run.smithi007.stdout:Change: 2024-09-18 07:53:31.757431708 +0000 2024-09-18T07:58:44.875 INFO:teuthology.orchestra.run.smithi007.stdout: Birth: 2024-09-18 07:53:31.757431708 +0000 2024-09-18T07:58:44.876 DEBUG:teuthology.orchestra.run.smithi007:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-09-18T07:58:44.942 INFO:teuthology.orchestra.run.smithi007.stderr:1+0 records in 2024-09-18T07:58:44.942 INFO:teuthology.orchestra.run.smithi007.stderr:1+0 records out 2024-09-18T07:58:44.942 INFO:teuthology.orchestra.run.smithi007.stderr:512 bytes copied, 0.000107263 s, 4.8 MB/s 2024-09-18T07:58:44.943 DEBUG:teuthology.orchestra.run.smithi007:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-09-18T07:58:45.001 DEBUG:teuthology.orchestra.run.smithi007:> stat /dev/vg_nvme/lv_2 2024-09-18T07:58:45.057 INFO:teuthology.orchestra.run.smithi007.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-09-18T07:58:45.058 INFO:teuthology.orchestra.run.smithi007.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-18T07:58:45.058 INFO:teuthology.orchestra.run.smithi007.stdout:Device: 5h/5d Inode: 970 Links: 1 2024-09-18T07:58:45.058 INFO:teuthology.orchestra.run.smithi007.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-18T07:58:45.058 INFO:teuthology.orchestra.run.smithi007.stdout:Context: system_u:object_r:device_t:s0 2024-09-18T07:58:45.058 INFO:teuthology.orchestra.run.smithi007.stdout:Access: 2024-09-18 07:58:34.878477948 +0000 2024-09-18T07:58:45.058 INFO:teuthology.orchestra.run.smithi007.stdout:Modify: 2024-09-18 07:53:31.757431708 +0000 2024-09-18T07:58:45.058 INFO:teuthology.orchestra.run.smithi007.stdout:Change: 2024-09-18 07:53:31.757431708 +0000 2024-09-18T07:58:45.058 INFO:teuthology.orchestra.run.smithi007.stdout: Birth: 2024-09-18 07:53:31.757431708 +0000 2024-09-18T07:58:45.058 DEBUG:teuthology.orchestra.run.smithi007:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-09-18T07:58:45.122 INFO:teuthology.orchestra.run.smithi007.stderr:1+0 records in 2024-09-18T07:58:45.122 INFO:teuthology.orchestra.run.smithi007.stderr:1+0 records out 2024-09-18T07:58:45.122 INFO:teuthology.orchestra.run.smithi007.stderr:512 bytes copied, 0.000109092 s, 4.7 MB/s 2024-09-18T07:58:45.124 DEBUG:teuthology.orchestra.run.smithi007:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-09-18T07:58:45.180 DEBUG:teuthology.orchestra.run.smithi007:> stat /dev/vg_nvme/lv_3 2024-09-18T07:58:45.231 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 07:58:45 smithi007 ceph-mon[39249]: pgmap v17: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-18T07:58:45.231 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 07:58:45 smithi007 ceph-mon[39249]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' 2024-09-18T07:58:45.231 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 07:58:45 smithi007 ceph-mon[39249]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' 2024-09-18T07:58:45.231 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 07:58:45 smithi007 ceph-mon[39249]: Reconfiguring mon.c (monmap changed)... 2024-09-18T07:58:45.231 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 07:58:45 smithi007 ceph-mon[39249]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' cmd={"prefix": "auth get", "entity": "mon."} : dispatch 2024-09-18T07:58:45.231 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 07:58:45 smithi007 ceph-mon[39249]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' cmd={"prefix": "config get", "who": "mon", "key": "public_network"} : dispatch 2024-09-18T07:58:45.231 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 07:58:45 smithi007 ceph-mon[39249]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-18T07:58:45.231 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 07:58:45 smithi007 ceph-mon[39249]: Reconfiguring daemon mon.c on smithi007 2024-09-18T07:58:45.231 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 07:58:45 smithi007 ceph-mon[39249]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' 2024-09-18T07:58:45.232 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 07:58:45 smithi007 ceph-mon[29459]: pgmap v17: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-18T07:58:45.232 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 07:58:45 smithi007 ceph-mon[29459]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' 2024-09-18T07:58:45.232 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 07:58:45 smithi007 ceph-mon[29459]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' 2024-09-18T07:58:45.232 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 07:58:45 smithi007 ceph-mon[29459]: Reconfiguring mon.c (monmap changed)... 2024-09-18T07:58:45.232 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 07:58:45 smithi007 ceph-mon[29459]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' cmd={"prefix": "auth get", "entity": "mon."} : dispatch 2024-09-18T07:58:45.232 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 07:58:45 smithi007 ceph-mon[29459]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' cmd={"prefix": "config get", "who": "mon", "key": "public_network"} : dispatch 2024-09-18T07:58:45.232 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 07:58:45 smithi007 ceph-mon[29459]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-18T07:58:45.232 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 07:58:45 smithi007 ceph-mon[29459]: Reconfiguring daemon mon.c on smithi007 2024-09-18T07:58:45.232 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 07:58:45 smithi007 ceph-mon[29459]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' 2024-09-18T07:58:45.237 INFO:teuthology.orchestra.run.smithi007.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-09-18T07:58:45.237 INFO:teuthology.orchestra.run.smithi007.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-18T07:58:45.237 INFO:teuthology.orchestra.run.smithi007.stdout:Device: 5h/5d Inode: 979 Links: 1 2024-09-18T07:58:45.237 INFO:teuthology.orchestra.run.smithi007.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-18T07:58:45.237 INFO:teuthology.orchestra.run.smithi007.stdout:Context: system_u:object_r:device_t:s0 2024-09-18T07:58:45.237 INFO:teuthology.orchestra.run.smithi007.stdout:Access: 2024-09-18 07:58:34.879477934 +0000 2024-09-18T07:58:45.237 INFO:teuthology.orchestra.run.smithi007.stdout:Modify: 2024-09-18 07:53:31.758431937 +0000 2024-09-18T07:58:45.237 INFO:teuthology.orchestra.run.smithi007.stdout:Change: 2024-09-18 07:53:31.758431937 +0000 2024-09-18T07:58:45.237 INFO:teuthology.orchestra.run.smithi007.stdout: Birth: 2024-09-18 07:53:31.758431937 +0000 2024-09-18T07:58:45.237 DEBUG:teuthology.orchestra.run.smithi007:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-09-18T07:58:45.302 INFO:teuthology.orchestra.run.smithi007.stderr:1+0 records in 2024-09-18T07:58:45.302 INFO:teuthology.orchestra.run.smithi007.stderr:1+0 records out 2024-09-18T07:58:45.302 INFO:teuthology.orchestra.run.smithi007.stderr:512 bytes copied, 0.000106858 s, 4.8 MB/s 2024-09-18T07:58:45.303 DEBUG:teuthology.orchestra.run.smithi007:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-09-18T07:58:45.361 DEBUG:teuthology.orchestra.run.smithi007:> stat /dev/vg_nvme/lv_4 2024-09-18T07:58:45.406 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 07:58:45 smithi050 ceph-mon[32371]: pgmap v17: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-18T07:58:45.406 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 07:58:45 smithi050 ceph-mon[32371]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' 2024-09-18T07:58:45.406 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 07:58:45 smithi050 ceph-mon[32371]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' 2024-09-18T07:58:45.406 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 07:58:45 smithi050 ceph-mon[32371]: Reconfiguring mon.c (monmap changed)... 2024-09-18T07:58:45.407 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 07:58:45 smithi050 ceph-mon[32371]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' cmd={"prefix": "auth get", "entity": "mon."} : dispatch 2024-09-18T07:58:45.407 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 07:58:45 smithi050 ceph-mon[32371]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' cmd={"prefix": "config get", "who": "mon", "key": "public_network"} : dispatch 2024-09-18T07:58:45.407 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 07:58:45 smithi050 ceph-mon[32371]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-18T07:58:45.407 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 07:58:45 smithi050 ceph-mon[32371]: Reconfiguring daemon mon.c on smithi007 2024-09-18T07:58:45.407 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 07:58:45 smithi050 ceph-mon[32371]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' 2024-09-18T07:58:45.417 INFO:teuthology.orchestra.run.smithi007.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-09-18T07:58:45.418 INFO:teuthology.orchestra.run.smithi007.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-18T07:58:45.418 INFO:teuthology.orchestra.run.smithi007.stdout:Device: 5h/5d Inode: 984 Links: 1 2024-09-18T07:58:45.418 INFO:teuthology.orchestra.run.smithi007.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-18T07:58:45.418 INFO:teuthology.orchestra.run.smithi007.stdout:Context: system_u:object_r:device_t:s0 2024-09-18T07:58:45.418 INFO:teuthology.orchestra.run.smithi007.stdout:Access: 2024-09-18 07:58:34.879477934 +0000 2024-09-18T07:58:45.418 INFO:teuthology.orchestra.run.smithi007.stdout:Modify: 2024-09-18 07:53:31.760432395 +0000 2024-09-18T07:58:45.418 INFO:teuthology.orchestra.run.smithi007.stdout:Change: 2024-09-18 07:53:31.760432395 +0000 2024-09-18T07:58:45.418 INFO:teuthology.orchestra.run.smithi007.stdout: Birth: 2024-09-18 07:53:31.760432395 +0000 2024-09-18T07:58:45.418 DEBUG:teuthology.orchestra.run.smithi007:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-09-18T07:58:45.482 INFO:teuthology.orchestra.run.smithi007.stderr:1+0 records in 2024-09-18T07:58:45.482 INFO:teuthology.orchestra.run.smithi007.stderr:1+0 records out 2024-09-18T07:58:45.482 INFO:teuthology.orchestra.run.smithi007.stderr:512 bytes copied, 9.5563e-05 s, 5.4 MB/s 2024-09-18T07:58:45.483 DEBUG:teuthology.orchestra.run.smithi007:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-09-18T07:58:45.540 DEBUG:teuthology.orchestra.run.smithi050:> set -ex 2024-09-18T07:58:45.540 DEBUG:teuthology.orchestra.run.smithi050:> dd if=/scratch_devs of=/dev/stdout 2024-09-18T07:58:45.557 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-09-18T07:58:45.557 DEBUG:teuthology.orchestra.run.smithi050:> stat /dev/vg_nvme/lv_1 2024-09-18T07:58:45.614 INFO:teuthology.orchestra.run.smithi050.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-09-18T07:58:45.614 INFO:teuthology.orchestra.run.smithi050.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-18T07:58:45.614 INFO:teuthology.orchestra.run.smithi050.stdout:Device: 5h/5d Inode: 973 Links: 1 2024-09-18T07:58:45.614 INFO:teuthology.orchestra.run.smithi050.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-18T07:58:45.615 INFO:teuthology.orchestra.run.smithi050.stdout:Context: system_u:object_r:device_t:s0 2024-09-18T07:58:45.615 INFO:teuthology.orchestra.run.smithi050.stdout:Access: 2024-09-18 07:58:26.422793154 +0000 2024-09-18T07:58:45.615 INFO:teuthology.orchestra.run.smithi050.stdout:Modify: 2024-09-18 07:53:31.942060107 +0000 2024-09-18T07:58:45.615 INFO:teuthology.orchestra.run.smithi050.stdout:Change: 2024-09-18 07:53:31.942060107 +0000 2024-09-18T07:58:45.615 INFO:teuthology.orchestra.run.smithi050.stdout: Birth: 2024-09-18 07:53:31.942060107 +0000 2024-09-18T07:58:45.615 DEBUG:teuthology.orchestra.run.smithi050:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-09-18T07:58:45.679 INFO:teuthology.orchestra.run.smithi050.stderr:1+0 records in 2024-09-18T07:58:45.679 INFO:teuthology.orchestra.run.smithi050.stderr:1+0 records out 2024-09-18T07:58:45.679 INFO:teuthology.orchestra.run.smithi050.stderr:512 bytes copied, 9.4739e-05 s, 5.4 MB/s 2024-09-18T07:58:45.680 DEBUG:teuthology.orchestra.run.smithi050:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-09-18T07:58:45.737 DEBUG:teuthology.orchestra.run.smithi050:> stat /dev/vg_nvme/lv_2 2024-09-18T07:58:45.794 INFO:teuthology.orchestra.run.smithi050.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-09-18T07:58:45.794 INFO:teuthology.orchestra.run.smithi050.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-18T07:58:45.794 INFO:teuthology.orchestra.run.smithi050.stdout:Device: 5h/5d Inode: 975 Links: 1 2024-09-18T07:58:45.794 INFO:teuthology.orchestra.run.smithi050.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-18T07:58:45.794 INFO:teuthology.orchestra.run.smithi050.stdout:Context: system_u:object_r:device_t:s0 2024-09-18T07:58:45.794 INFO:teuthology.orchestra.run.smithi050.stdout:Access: 2024-09-18 07:58:26.423793130 +0000 2024-09-18T07:58:45.794 INFO:teuthology.orchestra.run.smithi050.stdout:Modify: 2024-09-18 07:53:31.942060107 +0000 2024-09-18T07:58:45.794 INFO:teuthology.orchestra.run.smithi050.stdout:Change: 2024-09-18 07:53:31.942060107 +0000 2024-09-18T07:58:45.794 INFO:teuthology.orchestra.run.smithi050.stdout: Birth: 2024-09-18 07:53:31.942060107 +0000 2024-09-18T07:58:45.795 DEBUG:teuthology.orchestra.run.smithi050:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-09-18T07:58:45.858 INFO:teuthology.orchestra.run.smithi050.stderr:1+0 records in 2024-09-18T07:58:45.858 INFO:teuthology.orchestra.run.smithi050.stderr:1+0 records out 2024-09-18T07:58:45.858 INFO:teuthology.orchestra.run.smithi050.stderr:512 bytes copied, 0.000102637 s, 5.0 MB/s 2024-09-18T07:58:45.859 DEBUG:teuthology.orchestra.run.smithi050:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-09-18T07:58:45.916 DEBUG:teuthology.orchestra.run.smithi050:> stat /dev/vg_nvme/lv_3 2024-09-18T07:58:45.972 INFO:teuthology.orchestra.run.smithi050.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-09-18T07:58:45.972 INFO:teuthology.orchestra.run.smithi050.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-18T07:58:45.972 INFO:teuthology.orchestra.run.smithi050.stdout:Device: 5h/5d Inode: 984 Links: 1 2024-09-18T07:58:45.972 INFO:teuthology.orchestra.run.smithi050.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-18T07:58:45.972 INFO:teuthology.orchestra.run.smithi050.stdout:Context: system_u:object_r:device_t:s0 2024-09-18T07:58:45.973 INFO:teuthology.orchestra.run.smithi050.stdout:Access: 2024-09-18 07:58:26.423793130 +0000 2024-09-18T07:58:45.973 INFO:teuthology.orchestra.run.smithi050.stdout:Modify: 2024-09-18 07:53:31.944059714 +0000 2024-09-18T07:58:45.973 INFO:teuthology.orchestra.run.smithi050.stdout:Change: 2024-09-18 07:53:31.944059714 +0000 2024-09-18T07:58:45.973 INFO:teuthology.orchestra.run.smithi050.stdout: Birth: 2024-09-18 07:53:31.944059714 +0000 2024-09-18T07:58:45.973 DEBUG:teuthology.orchestra.run.smithi050:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-09-18T07:58:46.038 INFO:teuthology.orchestra.run.smithi050.stderr:1+0 records in 2024-09-18T07:58:46.038 INFO:teuthology.orchestra.run.smithi050.stderr:1+0 records out 2024-09-18T07:58:46.038 INFO:teuthology.orchestra.run.smithi050.stderr:512 bytes copied, 0.000102835 s, 5.0 MB/s 2024-09-18T07:58:46.039 DEBUG:teuthology.orchestra.run.smithi050:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-09-18T07:58:46.096 DEBUG:teuthology.orchestra.run.smithi050:> stat /dev/vg_nvme/lv_4 2024-09-18T07:58:46.152 INFO:teuthology.orchestra.run.smithi050.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-09-18T07:58:46.152 INFO:teuthology.orchestra.run.smithi050.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-18T07:58:46.153 INFO:teuthology.orchestra.run.smithi050.stdout:Device: 5h/5d Inode: 969 Links: 1 2024-09-18T07:58:46.153 INFO:teuthology.orchestra.run.smithi050.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-18T07:58:46.153 INFO:teuthology.orchestra.run.smithi050.stdout:Context: system_u:object_r:device_t:s0 2024-09-18T07:58:46.153 INFO:teuthology.orchestra.run.smithi050.stdout:Access: 2024-09-18 07:58:26.424793105 +0000 2024-09-18T07:58:46.153 INFO:teuthology.orchestra.run.smithi050.stdout:Modify: 2024-09-18 07:53:31.942060107 +0000 2024-09-18T07:58:46.153 INFO:teuthology.orchestra.run.smithi050.stdout:Change: 2024-09-18 07:53:31.942060107 +0000 2024-09-18T07:58:46.153 INFO:teuthology.orchestra.run.smithi050.stdout: Birth: 2024-09-18 07:53:31.942060107 +0000 2024-09-18T07:58:46.153 DEBUG:teuthology.orchestra.run.smithi050:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-09-18T07:58:46.217 INFO:teuthology.orchestra.run.smithi050.stderr:1+0 records in 2024-09-18T07:58:46.217 INFO:teuthology.orchestra.run.smithi050.stderr:1+0 records out 2024-09-18T07:58:46.217 INFO:teuthology.orchestra.run.smithi050.stderr:512 bytes copied, 0.000108567 s, 4.7 MB/s 2024-09-18T07:58:46.218 DEBUG:teuthology.orchestra.run.smithi050:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-09-18T07:58:46.275 INFO:tasks.cephadm:Deploying osd.0 on smithi007 with /dev/vg_nvme/lv_4... 2024-09-18T07:58:46.275 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e ceph-volume -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 779349ca-7593-11ef-bceb-c7b262605968 -- lvm zap /dev/vg_nvme/lv_4 2024-09-18T07:58:46.303 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 07:58:46 smithi007 ceph-mon[39249]: from='client.24100 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "placement": "2;smithi007=y;smithi050=x", "target": ["mon-mgr", ""]}]: dispatch 2024-09-18T07:58:46.303 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 07:58:46 smithi007 ceph-mon[39249]: Saving service mgr spec with placement smithi007=y;smithi050=x;count:2 2024-09-18T07:58:46.303 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 07:58:46 smithi007 ceph-mon[39249]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' 2024-09-18T07:58:46.303 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 07:58:46 smithi007 ceph-mon[39249]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' 2024-09-18T07:58:46.303 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 07:58:46 smithi007 ceph-mon[39249]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' cmd={"prefix": "auth get", "entity": "mon."} : dispatch 2024-09-18T07:58:46.303 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 07:58:46 smithi007 ceph-mon[39249]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' cmd={"prefix": "config get", "who": "mon", "key": "public_network"} : dispatch 2024-09-18T07:58:46.304 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 07:58:46 smithi007 ceph-mon[39249]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-18T07:58:46.304 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 07:58:46 smithi007 ceph-mon[29459]: from='client.24100 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "placement": "2;smithi007=y;smithi050=x", "target": ["mon-mgr", ""]}]: dispatch 2024-09-18T07:58:46.304 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 07:58:46 smithi007 ceph-mon[29459]: Saving service mgr spec with placement smithi007=y;smithi050=x;count:2 2024-09-18T07:58:46.304 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 07:58:46 smithi007 ceph-mon[29459]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' 2024-09-18T07:58:46.304 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 07:58:46 smithi007 ceph-mon[29459]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' 2024-09-18T07:58:46.304 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 07:58:46 smithi007 ceph-mon[29459]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' cmd={"prefix": "auth get", "entity": "mon."} : dispatch 2024-09-18T07:58:46.304 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 07:58:46 smithi007 ceph-mon[29459]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' cmd={"prefix": "config get", "who": "mon", "key": "public_network"} : dispatch 2024-09-18T07:58:46.304 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 07:58:46 smithi007 ceph-mon[29459]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-18T07:58:46.406 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 07:58:46 smithi050 ceph-mon[32371]: from='client.24100 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "placement": "2;smithi007=y;smithi050=x", "target": ["mon-mgr", ""]}]: dispatch 2024-09-18T07:58:46.406 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 07:58:46 smithi050 ceph-mon[32371]: Saving service mgr spec with placement smithi007=y;smithi050=x;count:2 2024-09-18T07:58:46.407 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 07:58:46 smithi050 ceph-mon[32371]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' 2024-09-18T07:58:46.407 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 07:58:46 smithi050 ceph-mon[32371]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' 2024-09-18T07:58:46.407 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 07:58:46 smithi050 ceph-mon[32371]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' cmd={"prefix": "auth get", "entity": "mon."} : dispatch 2024-09-18T07:58:46.407 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 07:58:46 smithi050 ceph-mon[32371]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' cmd={"prefix": "config get", "who": "mon", "key": "public_network"} : dispatch 2024-09-18T07:58:46.407 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 07:58:46 smithi050 ceph-mon[32371]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-18T07:58:46.538 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/779349ca-7593-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T07:58:47.231 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 07:58:47 smithi007 ceph-mon[29459]: pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-18T07:58:47.231 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 07:58:47 smithi007 ceph-mon[29459]: Reconfiguring mon.b (monmap changed)... 2024-09-18T07:58:47.231 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 07:58:47 smithi007 ceph-mon[29459]: Reconfiguring daemon mon.b on smithi050 2024-09-18T07:58:47.231 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 07:58:47 smithi007 ceph-mon[29459]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' 2024-09-18T07:58:47.231 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 07:58:47 smithi007 ceph-mon[29459]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' 2024-09-18T07:58:47.232 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 07:58:47 smithi007 ceph-mon[29459]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-18T07:58:47.232 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 07:58:47 smithi007 ceph-mon[29459]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-18T07:58:47.232 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 07:58:47 smithi007 ceph-mon[29459]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-18T07:58:47.232 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 07:58:47 smithi007 ceph-mon[29459]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' 2024-09-18T07:58:47.232 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 07:58:47 smithi007 ceph-mon[29459]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' cmd={"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]} : dispatch 2024-09-18T07:58:47.232 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 07:58:47 smithi007 ceph-mon[29459]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-09-18T07:58:47.232 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 07:58:47 smithi007 ceph-mon[29459]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' cmd={"prefix": "mgr services"} : dispatch 2024-09-18T07:58:47.232 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 07:58:47 smithi007 ceph-mon[29459]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-18T07:58:47.232 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 07:58:47 smithi007 ceph-mon[39249]: pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-18T07:58:47.233 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 07:58:47 smithi007 ceph-mon[39249]: Reconfiguring mon.b (monmap changed)... 2024-09-18T07:58:47.233 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 07:58:47 smithi007 ceph-mon[39249]: Reconfiguring daemon mon.b on smithi050 2024-09-18T07:58:47.233 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 07:58:47 smithi007 ceph-mon[39249]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' 2024-09-18T07:58:47.233 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 07:58:47 smithi007 ceph-mon[39249]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' 2024-09-18T07:58:47.233 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 07:58:47 smithi007 ceph-mon[39249]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-18T07:58:47.233 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 07:58:47 smithi007 ceph-mon[39249]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-18T07:58:47.233 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 07:58:47 smithi007 ceph-mon[39249]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-18T07:58:47.233 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 07:58:47 smithi007 ceph-mon[39249]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' 2024-09-18T07:58:47.233 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 07:58:47 smithi007 ceph-mon[39249]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' cmd={"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]} : dispatch 2024-09-18T07:58:47.233 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 07:58:47 smithi007 ceph-mon[39249]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-09-18T07:58:47.233 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 07:58:47 smithi007 ceph-mon[39249]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' cmd={"prefix": "mgr services"} : dispatch 2024-09-18T07:58:47.233 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 07:58:47 smithi007 ceph-mon[39249]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-18T07:58:47.341 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 07:58:47 smithi050 ceph-mon[32371]: pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-18T07:58:47.341 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 07:58:47 smithi050 ceph-mon[32371]: Reconfiguring mon.b (monmap changed)... 2024-09-18T07:58:47.341 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 07:58:47 smithi050 ceph-mon[32371]: Reconfiguring daemon mon.b on smithi050 2024-09-18T07:58:47.341 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 07:58:47 smithi050 ceph-mon[32371]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' 2024-09-18T07:58:47.341 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 07:58:47 smithi050 ceph-mon[32371]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' 2024-09-18T07:58:47.341 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 07:58:47 smithi050 ceph-mon[32371]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-18T07:58:47.341 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 07:58:47 smithi050 ceph-mon[32371]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-18T07:58:47.341 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 07:58:47 smithi050 ceph-mon[32371]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-18T07:58:47.341 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 07:58:47 smithi050 ceph-mon[32371]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' 2024-09-18T07:58:47.341 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 07:58:47 smithi050 ceph-mon[32371]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' cmd={"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]} : dispatch 2024-09-18T07:58:47.341 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 07:58:47 smithi050 ceph-mon[32371]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-09-18T07:58:47.342 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 07:58:47 smithi050 ceph-mon[32371]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' cmd={"prefix": "mgr services"} : dispatch 2024-09-18T07:58:47.342 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 07:58:47 smithi050 ceph-mon[32371]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-18T07:58:48.317 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 07:58:48 smithi050 ceph-mon[32371]: Deploying daemon mgr.x on smithi050 2024-09-18T07:58:48.445 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 07:58:48 smithi007 ceph-mon[29459]: Deploying daemon mgr.x on smithi050 2024-09-18T07:58:48.445 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 07:58:48 smithi007 ceph-mon[39249]: Deploying daemon mgr.x on smithi050 2024-09-18T07:58:48.656 INFO:journalctl@ceph.mgr.x.smithi050.stdout:Sep 18 07:58:48 smithi050 systemd[1]: Starting Ceph mgr.x for 779349ca-7593-11ef-bceb-c7b262605968... 2024-09-18T07:58:48.960 INFO:journalctl@ceph.mgr.x.smithi050.stdout:Sep 18 07:58:48 smithi050 podman[33853]: 2024-09-18 07:58:48.606616306 +0000 UTC m=+0.016729955 image pull b22e1e37ddf1ca6e22136cd6540a9ab61e7cafb8bfbc7a91a2f20f7f9941bd9a quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:1f5b5a3734f665ef333266f39143ac3ef1eb1951b21258f2307e10332e2d7ac9 2024-09-18T07:58:48.960 INFO:journalctl@ceph.mgr.x.smithi050.stdout:Sep 18 07:58:48 smithi050 podman[33853]: 2024-09-18 07:58:48.711168371 +0000 UTC m=+0.121282004 container create 1a9f4b29ff88c87571787a968ddfb81a0a4d4167ed27727568772c0d887d3f61 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:1f5b5a3734f665ef333266f39143ac3ef1eb1951b21258f2307e10332e2d7ac9, name=ceph-779349ca-7593-11ef-bceb-c7b262605968-mgr-x, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, RELEASE=wip-guits-main-2024-09-17-1213-e7fb7b5, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240731, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , GIT_CLEAN=True, ceph=True, org.label-schema.license=GPLv2, io.buildah.version=1.37.2, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965) 2024-09-18T07:58:48.960 INFO:journalctl@ceph.mgr.x.smithi050.stdout:Sep 18 07:58:48 smithi050 podman[33853]: 2024-09-18 07:58:48.930126519 +0000 UTC m=+0.340240154 container init 1a9f4b29ff88c87571787a968ddfb81a0a4d4167ed27727568772c0d887d3f61 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:1f5b5a3734f665ef333266f39143ac3ef1eb1951b21258f2307e10332e2d7ac9, name=ceph-779349ca-7593-11ef-bceb-c7b262605968-mgr-x, org.label-schema.build-date=20240731, ceph=True, org.label-schema.schema-version=1.0, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-guits-main-2024-09-17-1213-e7fb7b5, GIT_CLEAN=True, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.license=GPLv2, io.buildah.version=1.37.2) 2024-09-18T07:58:48.960 INFO:journalctl@ceph.mgr.x.smithi050.stdout:Sep 18 07:58:48 smithi050 podman[33853]: 2024-09-18 07:58:48.932914388 +0000 UTC m=+0.343028023 container start 1a9f4b29ff88c87571787a968ddfb81a0a4d4167ed27727568772c0d887d3f61 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:1f5b5a3734f665ef333266f39143ac3ef1eb1951b21258f2307e10332e2d7ac9, name=ceph-779349ca-7593-11ef-bceb-c7b262605968-mgr-x, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.build-date=20240731, ceph=True, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.license=GPLv2, io.buildah.version=1.37.2, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-guits-main-2024-09-17-1213-e7fb7b5, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=) 2024-09-18T07:58:49.023 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-09-18T07:58:49.053 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 779349ca-7593-11ef-bceb-c7b262605968 -- ceph orch daemon add osd smithi007:vg_nvme/lv_4 2024-09-18T07:58:49.351 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/779349ca-7593-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T07:58:49.406 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 07:58:49 smithi050 ceph-mon[32371]: pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-18T07:58:49.406 INFO:journalctl@ceph.mgr.x.smithi050.stdout:Sep 18 07:58:48 smithi050 ceph-mgr[33872]: -- 172.21.15.50:0/3477668003 <== mon.1 v2:172.21.15.50:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55f70d3350e0 con 0x55f70d312800 2024-09-18T07:58:49.407 INFO:journalctl@ceph.mgr.x.smithi050.stdout:Sep 18 07:58:49 smithi050 bash[33853]: 1a9f4b29ff88c87571787a968ddfb81a0a4d4167ed27727568772c0d887d3f61 2024-09-18T07:58:49.407 INFO:journalctl@ceph.mgr.x.smithi050.stdout:Sep 18 07:58:49 smithi050 systemd[1]: Started Ceph mgr.x for 779349ca-7593-11ef-bceb-c7b262605968. 2024-09-18T07:58:49.472 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 07:58:49 smithi007 ceph-mon[29459]: pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-18T07:58:49.473 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 07:58:49 smithi007 ceph-mon[39249]: pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-18T07:58:50.393 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 07:58:50 smithi007 ceph-mon[29459]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' 2024-09-18T07:58:50.393 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 07:58:50 smithi007 ceph-mon[29459]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' 2024-09-18T07:58:50.393 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 07:58:50 smithi007 ceph-mon[29459]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' 2024-09-18T07:58:50.393 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 07:58:50 smithi007 ceph-mon[29459]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' 2024-09-18T07:58:50.393 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 07:58:50 smithi007 ceph-mon[29459]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-18T07:58:50.393 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 07:58:50 smithi007 ceph-mon[29459]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' cmd={"prefix": "osd tree", "states": ["destroyed"], "format": "json"} : dispatch 2024-09-18T07:58:50.394 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 07:58:50 smithi007 ceph-mon[29459]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.bootstrap-osd"} : dispatch 2024-09-18T07:58:50.394 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 07:58:50 smithi007 ceph-mon[29459]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-18T07:58:50.394 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 07:58:50 smithi007 ceph-mon[39249]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' 2024-09-18T07:58:50.394 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 07:58:50 smithi007 ceph-mon[39249]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' 2024-09-18T07:58:50.394 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 07:58:50 smithi007 ceph-mon[39249]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' 2024-09-18T07:58:50.394 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 07:58:50 smithi007 ceph-mon[39249]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' 2024-09-18T07:58:50.394 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 07:58:50 smithi007 ceph-mon[39249]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-18T07:58:50.394 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 07:58:50 smithi007 ceph-mon[39249]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' cmd={"prefix": "osd tree", "states": ["destroyed"], "format": "json"} : dispatch 2024-09-18T07:58:50.395 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 07:58:50 smithi007 ceph-mon[39249]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.bootstrap-osd"} : dispatch 2024-09-18T07:58:50.395 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 07:58:50 smithi007 ceph-mon[39249]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-18T07:58:50.406 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 07:58:50 smithi050 ceph-mon[32371]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' 2024-09-18T07:58:50.406 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 07:58:50 smithi050 ceph-mon[32371]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' 2024-09-18T07:58:50.406 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 07:58:50 smithi050 ceph-mon[32371]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' 2024-09-18T07:58:50.406 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 07:58:50 smithi050 ceph-mon[32371]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' 2024-09-18T07:58:50.406 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 07:58:50 smithi050 ceph-mon[32371]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-18T07:58:50.406 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 07:58:50 smithi050 ceph-mon[32371]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' cmd={"prefix": "osd tree", "states": ["destroyed"], "format": "json"} : dispatch 2024-09-18T07:58:50.407 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 07:58:50 smithi050 ceph-mon[32371]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.bootstrap-osd"} : dispatch 2024-09-18T07:58:50.407 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 07:58:50 smithi050 ceph-mon[32371]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-18T07:58:51.356 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 07:58:51 smithi007 ceph-mon[29459]: pgmap v20: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-18T07:58:51.356 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 07:58:51 smithi007 ceph-mon[29459]: from='client.24107 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi007:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-09-18T07:58:51.356 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 07:58:51 smithi007 ceph-mon[29459]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' 2024-09-18T07:58:51.357 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 07:58:51 smithi007 ceph-mon[29459]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' 2024-09-18T07:58:51.357 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 07:58:51 smithi007 ceph-mon[29459]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-18T07:58:51.357 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 07:58:51 smithi007 ceph-mon[29459]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-18T07:58:51.357 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 07:58:51 smithi007 ceph-mon[29459]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' 2024-09-18T07:58:51.357 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 07:58:51 smithi007 ceph-mon[29459]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' cmd={"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]} : dispatch 2024-09-18T07:58:51.357 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 07:58:51 smithi007 ceph-mon[29459]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' cmd={"prefix": "mgr services"} : dispatch 2024-09-18T07:58:51.357 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 07:58:51 smithi007 ceph-mon[29459]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-18T07:58:51.357 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 07:58:51 smithi007 ceph-mon[39249]: pgmap v20: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-18T07:58:51.357 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 07:58:51 smithi007 ceph-mon[39249]: from='client.24107 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi007:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-09-18T07:58:51.357 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 07:58:51 smithi007 ceph-mon[39249]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' 2024-09-18T07:58:51.357 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 07:58:51 smithi007 ceph-mon[39249]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' 2024-09-18T07:58:51.358 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 07:58:51 smithi007 ceph-mon[39249]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-18T07:58:51.358 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 07:58:51 smithi007 ceph-mon[39249]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-18T07:58:51.358 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 07:58:51 smithi007 ceph-mon[39249]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' 2024-09-18T07:58:51.358 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 07:58:51 smithi007 ceph-mon[39249]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' cmd={"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]} : dispatch 2024-09-18T07:58:51.358 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 07:58:51 smithi007 ceph-mon[39249]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' cmd={"prefix": "mgr services"} : dispatch 2024-09-18T07:58:51.358 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 07:58:51 smithi007 ceph-mon[39249]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-18T07:58:51.406 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 07:58:51 smithi050 ceph-mon[32371]: pgmap v20: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-18T07:58:51.406 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 07:58:51 smithi050 ceph-mon[32371]: from='client.24107 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi007:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-09-18T07:58:51.406 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 07:58:51 smithi050 ceph-mon[32371]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' 2024-09-18T07:58:51.406 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 07:58:51 smithi050 ceph-mon[32371]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' 2024-09-18T07:58:51.407 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 07:58:51 smithi050 ceph-mon[32371]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-18T07:58:51.407 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 07:58:51 smithi050 ceph-mon[32371]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-18T07:58:51.407 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 07:58:51 smithi050 ceph-mon[32371]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' 2024-09-18T07:58:51.407 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 07:58:51 smithi050 ceph-mon[32371]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' cmd={"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]} : dispatch 2024-09-18T07:58:51.407 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 07:58:51 smithi050 ceph-mon[32371]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' cmd={"prefix": "mgr services"} : dispatch 2024-09-18T07:58:51.407 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 07:58:51 smithi050 ceph-mon[32371]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-18T07:58:52.369 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 07:58:52 smithi007 ceph-mon[29459]: Reconfiguring mgr.y (unknown last config time)... 2024-09-18T07:58:52.369 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 07:58:52 smithi007 ceph-mon[29459]: Reconfiguring daemon mgr.y on smithi007 2024-09-18T07:58:52.369 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 07:58:52 smithi007 ceph-mon[29459]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' 2024-09-18T07:58:52.369 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 07:58:52 smithi007 ceph-mon[29459]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' 2024-09-18T07:58:52.369 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 07:58:52 smithi007 ceph-mon[29459]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-18T07:58:52.369 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 07:58:52 smithi007 ceph-mon[29459]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-18T07:58:52.369 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 07:58:52 smithi007 ceph-mon[29459]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-18T07:58:52.369 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 07:58:52 smithi007 ceph-mon[29459]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' 2024-09-18T07:58:52.369 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 07:58:52 smithi007 ceph-mon[39249]: Reconfiguring mgr.y (unknown last config time)... 2024-09-18T07:58:52.370 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 07:58:52 smithi007 ceph-mon[39249]: Reconfiguring daemon mgr.y on smithi007 2024-09-18T07:58:52.370 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 07:58:52 smithi007 ceph-mon[39249]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' 2024-09-18T07:58:52.370 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 07:58:52 smithi007 ceph-mon[39249]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' 2024-09-18T07:58:52.370 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 07:58:52 smithi007 ceph-mon[39249]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-18T07:58:52.370 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 07:58:52 smithi007 ceph-mon[39249]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-18T07:58:52.370 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 07:58:52 smithi007 ceph-mon[39249]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-18T07:58:52.370 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 07:58:52 smithi007 ceph-mon[39249]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' 2024-09-18T07:58:52.406 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 07:58:52 smithi050 ceph-mon[32371]: Reconfiguring mgr.y (unknown last config time)... 2024-09-18T07:58:52.406 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 07:58:52 smithi050 ceph-mon[32371]: Reconfiguring daemon mgr.y on smithi007 2024-09-18T07:58:52.406 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 07:58:52 smithi050 ceph-mon[32371]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' 2024-09-18T07:58:52.406 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 07:58:52 smithi050 ceph-mon[32371]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' 2024-09-18T07:58:52.406 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 07:58:52 smithi050 ceph-mon[32371]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-18T07:58:52.406 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 07:58:52 smithi050 ceph-mon[32371]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-18T07:58:52.407 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 07:58:52 smithi050 ceph-mon[32371]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-18T07:58:52.407 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 07:58:52 smithi050 ceph-mon[32371]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' 2024-09-18T07:58:53.406 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 07:58:53 smithi050 ceph-mon[32371]: pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-18T07:58:53.406 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 07:58:53 smithi050 ceph-mon[32371]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' 2024-09-18T07:58:53.406 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 07:58:53 smithi050 ceph-mon[32371]: from='client.? 172.21.15.7:0/2365632697' entity='client.bootstrap-osd' cmd={"prefix": "osd new", "uuid": "ae8f60df-72ac-46fb-91a1-f132b851c876"} : dispatch 2024-09-18T07:58:53.407 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 07:58:53 smithi050 ceph-mon[32371]: from='client.? ' entity='client.bootstrap-osd' cmd={"prefix": "osd new", "uuid": "ae8f60df-72ac-46fb-91a1-f132b851c876"} : dispatch 2024-09-18T07:58:53.407 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 07:58:53 smithi050 ceph-mon[32371]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "ae8f60df-72ac-46fb-91a1-f132b851c876"}]': finished 2024-09-18T07:58:53.407 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 07:58:53 smithi050 ceph-mon[32371]: osdmap e5: 1 total, 0 up, 1 in 2024-09-18T07:58:53.407 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 07:58:53 smithi050 ceph-mon[32371]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 0} : dispatch 2024-09-18T07:58:53.481 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 07:58:53 smithi007 ceph-mon[39249]: pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-18T07:58:53.481 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 07:58:53 smithi007 ceph-mon[39249]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' 2024-09-18T07:58:53.481 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 07:58:53 smithi007 ceph-mon[39249]: from='client.? 172.21.15.7:0/2365632697' entity='client.bootstrap-osd' cmd={"prefix": "osd new", "uuid": "ae8f60df-72ac-46fb-91a1-f132b851c876"} : dispatch 2024-09-18T07:58:53.481 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 07:58:53 smithi007 ceph-mon[39249]: from='client.? ' entity='client.bootstrap-osd' cmd={"prefix": "osd new", "uuid": "ae8f60df-72ac-46fb-91a1-f132b851c876"} : dispatch 2024-09-18T07:58:53.481 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 07:58:53 smithi007 ceph-mon[39249]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "ae8f60df-72ac-46fb-91a1-f132b851c876"}]': finished 2024-09-18T07:58:53.482 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 07:58:53 smithi007 ceph-mon[39249]: osdmap e5: 1 total, 0 up, 1 in 2024-09-18T07:58:53.482 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 07:58:53 smithi007 ceph-mon[39249]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 0} : dispatch 2024-09-18T07:58:53.482 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 07:58:53 smithi007 ceph-mon[29459]: pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-18T07:58:53.482 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 07:58:53 smithi007 ceph-mon[29459]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' 2024-09-18T07:58:53.482 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 07:58:53 smithi007 ceph-mon[29459]: from='client.? 172.21.15.7:0/2365632697' entity='client.bootstrap-osd' cmd={"prefix": "osd new", "uuid": "ae8f60df-72ac-46fb-91a1-f132b851c876"} : dispatch 2024-09-18T07:58:53.482 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 07:58:53 smithi007 ceph-mon[29459]: from='client.? ' entity='client.bootstrap-osd' cmd={"prefix": "osd new", "uuid": "ae8f60df-72ac-46fb-91a1-f132b851c876"} : dispatch 2024-09-18T07:58:53.482 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 07:58:53 smithi007 ceph-mon[29459]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "ae8f60df-72ac-46fb-91a1-f132b851c876"}]': finished 2024-09-18T07:58:53.482 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 07:58:53 smithi007 ceph-mon[29459]: osdmap e5: 1 total, 0 up, 1 in 2024-09-18T07:58:53.482 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 07:58:53 smithi007 ceph-mon[29459]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 0} : dispatch 2024-09-18T07:58:54.406 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 07:58:54 smithi050 ceph-mon[32371]: from='client.? 172.21.15.7:0/3778505309' entity='client.bootstrap-osd' cmd={"prefix": "mon getmap"} : dispatch 2024-09-18T07:58:54.480 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 07:58:54 smithi007 ceph-mon[29459]: from='client.? 172.21.15.7:0/3778505309' entity='client.bootstrap-osd' cmd={"prefix": "mon getmap"} : dispatch 2024-09-18T07:58:54.481 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 07:58:54 smithi007 ceph-mon[39249]: from='client.? 172.21.15.7:0/3778505309' entity='client.bootstrap-osd' cmd={"prefix": "mon getmap"} : dispatch 2024-09-18T07:58:55.406 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 07:58:55 smithi050 ceph-mon[32371]: pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-18T07:58:55.407 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 07:58:55 smithi050 ceph-mon[32371]: Standby manager daemon x started 2024-09-18T07:58:55.407 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 07:58:55 smithi050 ceph-mon[32371]: from='mgr.? 172.21.15.50:0/4204768682' entity='mgr.x' cmd={"prefix": "config-key get", "key": "mgr/dashboard/x/crt"} : dispatch 2024-09-18T07:58:55.407 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 07:58:55 smithi050 ceph-mon[32371]: from='mgr.? 172.21.15.50:0/4204768682' entity='mgr.x' cmd={"prefix": "config-key get", "key": "mgr/dashboard/crt"} : dispatch 2024-09-18T07:58:55.407 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 07:58:55 smithi050 ceph-mon[32371]: from='mgr.? 172.21.15.50:0/4204768682' entity='mgr.x' cmd={"prefix": "config-key get", "key": "mgr/dashboard/x/key"} : dispatch 2024-09-18T07:58:55.407 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 07:58:55 smithi050 ceph-mon[32371]: from='mgr.? 172.21.15.50:0/4204768682' entity='mgr.x' cmd={"prefix": "config-key get", "key": "mgr/dashboard/key"} : dispatch 2024-09-18T07:58:55.481 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 07:58:55 smithi007 ceph-mon[29459]: pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-18T07:58:55.481 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 07:58:55 smithi007 ceph-mon[29459]: Standby manager daemon x started 2024-09-18T07:58:55.481 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 07:58:55 smithi007 ceph-mon[29459]: from='mgr.? 172.21.15.50:0/4204768682' entity='mgr.x' cmd={"prefix": "config-key get", "key": "mgr/dashboard/x/crt"} : dispatch 2024-09-18T07:58:55.481 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 07:58:55 smithi007 ceph-mon[29459]: from='mgr.? 172.21.15.50:0/4204768682' entity='mgr.x' cmd={"prefix": "config-key get", "key": "mgr/dashboard/crt"} : dispatch 2024-09-18T07:58:55.481 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 07:58:55 smithi007 ceph-mon[29459]: from='mgr.? 172.21.15.50:0/4204768682' entity='mgr.x' cmd={"prefix": "config-key get", "key": "mgr/dashboard/x/key"} : dispatch 2024-09-18T07:58:55.481 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 07:58:55 smithi007 ceph-mon[29459]: from='mgr.? 172.21.15.50:0/4204768682' entity='mgr.x' cmd={"prefix": "config-key get", "key": "mgr/dashboard/key"} : dispatch 2024-09-18T07:58:55.482 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 07:58:55 smithi007 ceph-mon[39249]: pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-18T07:58:55.482 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 07:58:55 smithi007 ceph-mon[39249]: Standby manager daemon x started 2024-09-18T07:58:55.482 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 07:58:55 smithi007 ceph-mon[39249]: from='mgr.? 172.21.15.50:0/4204768682' entity='mgr.x' cmd={"prefix": "config-key get", "key": "mgr/dashboard/x/crt"} : dispatch 2024-09-18T07:58:55.482 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 07:58:55 smithi007 ceph-mon[39249]: from='mgr.? 172.21.15.50:0/4204768682' entity='mgr.x' cmd={"prefix": "config-key get", "key": "mgr/dashboard/crt"} : dispatch 2024-09-18T07:58:55.482 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 07:58:55 smithi007 ceph-mon[39249]: from='mgr.? 172.21.15.50:0/4204768682' entity='mgr.x' cmd={"prefix": "config-key get", "key": "mgr/dashboard/x/key"} : dispatch 2024-09-18T07:58:55.482 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 07:58:55 smithi007 ceph-mon[39249]: from='mgr.? 172.21.15.50:0/4204768682' entity='mgr.x' cmd={"prefix": "config-key get", "key": "mgr/dashboard/key"} : dispatch 2024-09-18T07:58:56.406 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 07:58:56 smithi050 ceph-mon[32371]: mgrmap e14: y(active, since 57s), standbys: x 2024-09-18T07:58:56.406 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 07:58:56 smithi050 ceph-mon[32371]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' cmd={"prefix": "mgr metadata", "who": "x", "id": "x"} : dispatch 2024-09-18T07:58:56.406 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 07:58:56 smithi050 ceph-mon[32371]: pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-18T07:58:56.481 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 07:58:56 smithi007 ceph-mon[29459]: mgrmap e14: y(active, since 57s), standbys: x 2024-09-18T07:58:56.481 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 07:58:56 smithi007 ceph-mon[29459]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' cmd={"prefix": "mgr metadata", "who": "x", "id": "x"} : dispatch 2024-09-18T07:58:56.481 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 07:58:56 smithi007 ceph-mon[29459]: pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-18T07:58:56.481 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 07:58:56 smithi007 ceph-mon[39249]: mgrmap e14: y(active, since 57s), standbys: x 2024-09-18T07:58:56.481 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 07:58:56 smithi007 ceph-mon[39249]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' cmd={"prefix": "mgr metadata", "who": "x", "id": "x"} : dispatch 2024-09-18T07:58:56.481 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 07:58:56 smithi007 ceph-mon[39249]: pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-18T07:58:58.535 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 07:58:58 smithi007 ceph-mon[29459]: pgmap v25: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-18T07:58:58.535 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 07:58:58 smithi007 ceph-mon[39249]: pgmap v25: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-18T07:58:58.906 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 07:58:58 smithi050 ceph-mon[32371]: pgmap v25: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-18T07:59:00.685 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 07:59:00 smithi007 ceph-mon[29459]: pgmap v26: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-18T07:59:00.686 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 07:59:00 smithi007 ceph-mon[29459]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' cmd={"prefix": "auth get", "entity": "osd.0"} : dispatch 2024-09-18T07:59:00.686 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 07:59:00 smithi007 ceph-mon[29459]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-18T07:59:00.686 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 07:59:00 smithi007 ceph-mon[39249]: pgmap v26: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-18T07:59:00.686 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 07:59:00 smithi007 ceph-mon[39249]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' cmd={"prefix": "auth get", "entity": "osd.0"} : dispatch 2024-09-18T07:59:00.686 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 07:59:00 smithi007 ceph-mon[39249]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-18T07:59:00.906 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 07:59:00 smithi050 ceph-mon[32371]: pgmap v26: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-18T07:59:00.906 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 07:59:00 smithi050 ceph-mon[32371]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' cmd={"prefix": "auth get", "entity": "osd.0"} : dispatch 2024-09-18T07:59:00.906 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 07:59:00 smithi050 ceph-mon[32371]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-18T07:59:01.666 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 07:59:01 smithi007 ceph-mon[29459]: Deploying daemon osd.0 on smithi007 2024-09-18T07:59:01.666 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 07:59:01 smithi007 ceph-mon[39249]: Deploying daemon osd.0 on smithi007 2024-09-18T07:59:01.906 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 07:59:01 smithi050 ceph-mon[32371]: Deploying daemon osd.0 on smithi007 2024-09-18T07:59:02.731 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 07:59:02 smithi007 ceph-mon[29459]: pgmap v27: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-18T07:59:02.731 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 07:59:02 smithi007 ceph-mon[39249]: pgmap v27: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-18T07:59:02.906 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 07:59:02 smithi050 ceph-mon[32371]: pgmap v27: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-18T07:59:04.731 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 07:59:04 smithi007 ceph-mon[29459]: pgmap v28: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-18T07:59:04.731 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 07:59:04 smithi007 ceph-mon[39249]: pgmap v28: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-18T07:59:04.906 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 07:59:04 smithi050 ceph-mon[32371]: pgmap v28: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-18T07:59:06.427 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 07:59:06 smithi007 ceph-mon[29459]: pgmap v29: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-18T07:59:06.731 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 07:59:06 smithi007 ceph-mon[39249]: pgmap v29: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-18T07:59:06.906 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 07:59:06 smithi050 ceph-mon[32371]: pgmap v29: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-18T07:59:07.692 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 07:59:07 smithi007 ceph-mon[39249]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-18T07:59:07.692 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 07:59:07 smithi007 ceph-mon[39249]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' 2024-09-18T07:59:07.692 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 07:59:07 smithi007 ceph-mon[39249]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' 2024-09-18T07:59:07.692 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 07:59:07 smithi007 ceph-mon[29459]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-18T07:59:07.692 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 07:59:07 smithi007 ceph-mon[29459]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' 2024-09-18T07:59:07.692 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 07:59:07 smithi007 ceph-mon[29459]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' 2024-09-18T07:59:07.906 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 07:59:07 smithi050 ceph-mon[32371]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-18T07:59:07.906 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 07:59:07 smithi050 ceph-mon[32371]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' 2024-09-18T07:59:07.906 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 07:59:07 smithi050 ceph-mon[32371]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' 2024-09-18T07:59:08.731 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 07:59:08 smithi007 ceph-mon[29459]: pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-18T07:59:08.731 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 07:59:08 smithi007 ceph-mon[39249]: pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-18T07:59:08.906 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 07:59:08 smithi050 ceph-mon[32371]: pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-18T07:59:11.156 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 07:59:10 smithi050 ceph-mon[32371]: pgmap v31: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-18T07:59:11.156 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 07:59:10 smithi050 ceph-mon[32371]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' 2024-09-18T07:59:11.156 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 07:59:10 smithi050 ceph-mon[32371]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' 2024-09-18T07:59:11.157 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 07:59:10 smithi050 ceph-mon[32371]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-18T07:59:11.157 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 07:59:10 smithi050 ceph-mon[32371]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-18T07:59:11.157 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 07:59:10 smithi050 ceph-mon[32371]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' 2024-09-18T07:59:11.231 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 07:59:10 smithi007 ceph-mon[29459]: pgmap v31: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-18T07:59:11.231 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 07:59:10 smithi007 ceph-mon[29459]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' 2024-09-18T07:59:11.231 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 07:59:10 smithi007 ceph-mon[29459]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' 2024-09-18T07:59:11.231 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 07:59:10 smithi007 ceph-mon[29459]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-18T07:59:11.231 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 07:59:10 smithi007 ceph-mon[29459]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-18T07:59:11.231 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 07:59:10 smithi007 ceph-mon[29459]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' 2024-09-18T07:59:11.232 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 07:59:10 smithi007 ceph-mon[39249]: pgmap v31: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-18T07:59:11.232 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 07:59:10 smithi007 ceph-mon[39249]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' 2024-09-18T07:59:11.232 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 07:59:10 smithi007 ceph-mon[39249]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' 2024-09-18T07:59:11.232 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 07:59:10 smithi007 ceph-mon[39249]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-18T07:59:11.232 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 07:59:10 smithi007 ceph-mon[39249]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-18T07:59:11.232 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 07:59:10 smithi007 ceph-mon[39249]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' 2024-09-18T07:59:12.653 INFO:teuthology.orchestra.run.smithi007.stdout:Created osd(s) 0 on host 'smithi007' 2024-09-18T07:59:12.919 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 07:59:12 smithi007 ceph-mon[39249]: pgmap v32: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-18T07:59:12.919 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 07:59:12 smithi007 ceph-mon[39249]: from='osd.0 v2:172.21.15.7:6801/2896425794' entity='osd.0' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]} : dispatch 2024-09-18T07:59:12.920 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 07:59:12 smithi007 ceph-mon[39249]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-18T07:59:12.920 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 07:59:12 smithi007 ceph-mon[39249]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' 2024-09-18T07:59:12.920 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 07:59:12 smithi007 ceph-mon[39249]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' 2024-09-18T07:59:12.920 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 07:59:12 smithi007 ceph-mon[29459]: pgmap v32: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-18T07:59:12.921 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 07:59:12 smithi007 ceph-mon[29459]: from='osd.0 v2:172.21.15.7:6801/2896425794' entity='osd.0' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]} : dispatch 2024-09-18T07:59:12.921 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 07:59:12 smithi007 ceph-mon[29459]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-18T07:59:12.921 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 07:59:12 smithi007 ceph-mon[29459]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' 2024-09-18T07:59:12.921 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 07:59:12 smithi007 ceph-mon[29459]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' 2024-09-18T07:59:13.156 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 07:59:12 smithi050 ceph-mon[32371]: pgmap v32: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-18T07:59:13.156 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 07:59:12 smithi050 ceph-mon[32371]: from='osd.0 v2:172.21.15.7:6801/2896425794' entity='osd.0' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]} : dispatch 2024-09-18T07:59:13.157 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 07:59:12 smithi050 ceph-mon[32371]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-18T07:59:13.157 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 07:59:12 smithi050 ceph-mon[32371]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' 2024-09-18T07:59:13.157 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 07:59:12 smithi050 ceph-mon[32371]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' 2024-09-18T07:59:13.225 DEBUG:teuthology.orchestra.run.smithi007:osd.0> sudo journalctl -f -n 0 -u ceph-779349ca-7593-11ef-bceb-c7b262605968@osd.0.service 2024-09-18T07:59:13.227 INFO:tasks.cephadm:Deploying osd.1 on smithi007 with /dev/vg_nvme/lv_3... 2024-09-18T07:59:13.227 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e ceph-volume -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 779349ca-7593-11ef-bceb-c7b262605968 -- lvm zap /dev/vg_nvme/lv_3 2024-09-18T07:59:13.981 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 07:59:13 smithi007 ceph-mon[39249]: from='osd.0 v2:172.21.15.7:6801/2896425794' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-09-18T07:59:13.981 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 07:59:13 smithi007 ceph-mon[39249]: osdmap e6: 1 total, 0 up, 1 in 2024-09-18T07:59:13.981 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 07:59:13 smithi007 ceph-mon[39249]: from='osd.0 v2:172.21.15.7:6801/2896425794' entity='osd.0' cmd={"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi007", "root=default"]} : dispatch 2024-09-18T07:59:13.981 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 07:59:13 smithi007 ceph-mon[39249]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 0} : dispatch 2024-09-18T07:59:13.981 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 07:59:13 smithi007 ceph-mon[29459]: from='osd.0 v2:172.21.15.7:6801/2896425794' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-09-18T07:59:13.982 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 07:59:13 smithi007 ceph-mon[29459]: osdmap e6: 1 total, 0 up, 1 in 2024-09-18T07:59:13.982 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 07:59:13 smithi007 ceph-mon[29459]: from='osd.0 v2:172.21.15.7:6801/2896425794' entity='osd.0' cmd={"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi007", "root=default"]} : dispatch 2024-09-18T07:59:13.982 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 07:59:13 smithi007 ceph-mon[29459]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 0} : dispatch 2024-09-18T07:59:13.982 INFO:journalctl@ceph.osd.0.smithi007.stdout:Sep 18 07:59:13 smithi007 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd-0[42581]: 2024-09-18T07:59:13.751+0000 7f9019bfb640 -1 osd.0 0 waiting for initial osdmap 2024-09-18T07:59:13.982 INFO:journalctl@ceph.osd.0.smithi007.stdout:Sep 18 07:59:13 smithi007 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd-0[42581]: 2024-09-18T07:59:13.753+0000 7f9015a22640 -1 osd.0 7 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-18T07:59:14.156 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 07:59:13 smithi050 ceph-mon[32371]: from='osd.0 v2:172.21.15.7:6801/2896425794' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-09-18T07:59:14.157 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 07:59:13 smithi050 ceph-mon[32371]: osdmap e6: 1 total, 0 up, 1 in 2024-09-18T07:59:14.157 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 07:59:13 smithi050 ceph-mon[32371]: from='osd.0 v2:172.21.15.7:6801/2896425794' entity='osd.0' cmd={"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi007", "root=default"]} : dispatch 2024-09-18T07:59:14.157 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 07:59:13 smithi050 ceph-mon[32371]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 0} : dispatch 2024-09-18T07:59:14.802 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/779349ca-7593-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T07:59:14.924 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 07:59:14 smithi007 ceph-mon[29459]: pgmap v34: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-18T07:59:14.924 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 07:59:14 smithi007 ceph-mon[29459]: from='osd.0 v2:172.21.15.7:6801/2896425794' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi007", "root=default"]}]': finished 2024-09-18T07:59:14.924 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 07:59:14 smithi007 ceph-mon[29459]: osdmap e7: 1 total, 0 up, 1 in 2024-09-18T07:59:14.924 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 07:59:14 smithi007 ceph-mon[29459]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 0} : dispatch 2024-09-18T07:59:14.924 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 07:59:14 smithi007 ceph-mon[29459]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 0} : dispatch 2024-09-18T07:59:14.925 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 07:59:14 smithi007 ceph-mon[39249]: pgmap v34: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-18T07:59:14.925 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 07:59:14 smithi007 ceph-mon[39249]: from='osd.0 v2:172.21.15.7:6801/2896425794' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi007", "root=default"]}]': finished 2024-09-18T07:59:14.925 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 07:59:14 smithi007 ceph-mon[39249]: osdmap e7: 1 total, 0 up, 1 in 2024-09-18T07:59:14.925 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 07:59:14 smithi007 ceph-mon[39249]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 0} : dispatch 2024-09-18T07:59:14.925 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 07:59:14 smithi007 ceph-mon[39249]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 0} : dispatch 2024-09-18T07:59:15.156 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 07:59:14 smithi050 ceph-mon[32371]: pgmap v34: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-18T07:59:15.156 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 07:59:14 smithi050 ceph-mon[32371]: from='osd.0 v2:172.21.15.7:6801/2896425794' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi007", "root=default"]}]': finished 2024-09-18T07:59:15.156 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 07:59:14 smithi050 ceph-mon[32371]: osdmap e7: 1 total, 0 up, 1 in 2024-09-18T07:59:15.157 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 07:59:14 smithi050 ceph-mon[32371]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 0} : dispatch 2024-09-18T07:59:15.157 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 07:59:14 smithi050 ceph-mon[32371]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 0} : dispatch 2024-09-18T07:59:15.981 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 07:59:15 smithi007 ceph-mon[29459]: purged_snaps scrub starts 2024-09-18T07:59:15.981 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 07:59:15 smithi007 ceph-mon[29459]: purged_snaps scrub ok 2024-09-18T07:59:15.981 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 07:59:15 smithi007 ceph-mon[29459]: osd.0 v2:172.21.15.7:6801/2896425794 boot 2024-09-18T07:59:15.981 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 07:59:15 smithi007 ceph-mon[29459]: osdmap e8: 1 total, 1 up, 1 in 2024-09-18T07:59:15.981 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 07:59:15 smithi007 ceph-mon[29459]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 0} : dispatch 2024-09-18T07:59:15.981 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 07:59:15 smithi007 ceph-mon[39249]: purged_snaps scrub starts 2024-09-18T07:59:15.981 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 07:59:15 smithi007 ceph-mon[39249]: purged_snaps scrub ok 2024-09-18T07:59:15.982 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 07:59:15 smithi007 ceph-mon[39249]: osd.0 v2:172.21.15.7:6801/2896425794 boot 2024-09-18T07:59:15.982 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 07:59:15 smithi007 ceph-mon[39249]: osdmap e8: 1 total, 1 up, 1 in 2024-09-18T07:59:15.982 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 07:59:15 smithi007 ceph-mon[39249]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 0} : dispatch 2024-09-18T07:59:16.156 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 07:59:15 smithi050 ceph-mon[32371]: purged_snaps scrub starts 2024-09-18T07:59:16.156 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 07:59:15 smithi050 ceph-mon[32371]: purged_snaps scrub ok 2024-09-18T07:59:16.157 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 07:59:15 smithi050 ceph-mon[32371]: osd.0 v2:172.21.15.7:6801/2896425794 boot 2024-09-18T07:59:16.157 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 07:59:15 smithi050 ceph-mon[32371]: osdmap e8: 1 total, 1 up, 1 in 2024-09-18T07:59:16.157 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 07:59:15 smithi050 ceph-mon[32371]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 0} : dispatch 2024-09-18T07:59:16.981 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 07:59:16 smithi007 ceph-mon[29459]: pgmap v37: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-09-18T07:59:16.982 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 07:59:16 smithi007 ceph-mon[29459]: osdmap e9: 1 total, 1 up, 1 in 2024-09-18T07:59:16.982 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 07:59:16 smithi007 ceph-mon[29459]: Detected new or changed devices on smithi007 2024-09-18T07:59:16.982 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 07:59:16 smithi007 ceph-mon[29459]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' 2024-09-18T07:59:16.982 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 07:59:16 smithi007 ceph-mon[29459]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' 2024-09-18T07:59:16.982 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 07:59:16 smithi007 ceph-mon[29459]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"} : dispatch 2024-09-18T07:59:16.982 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 07:59:16 smithi007 ceph-mon[29459]: Adjusting osd_memory_target on smithi007 to 16068M 2024-09-18T07:59:16.982 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 07:59:16 smithi007 ceph-mon[29459]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' 2024-09-18T07:59:16.982 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 07:59:16 smithi007 ceph-mon[29459]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-18T07:59:16.982 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 07:59:16 smithi007 ceph-mon[29459]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-18T07:59:16.982 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 07:59:16 smithi007 ceph-mon[29459]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' 2024-09-18T07:59:16.983 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 07:59:16 smithi007 ceph-mon[39249]: pgmap v37: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-09-18T07:59:16.983 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 07:59:16 smithi007 ceph-mon[39249]: osdmap e9: 1 total, 1 up, 1 in 2024-09-18T07:59:16.983 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 07:59:16 smithi007 ceph-mon[39249]: Detected new or changed devices on smithi007 2024-09-18T07:59:16.983 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 07:59:16 smithi007 ceph-mon[39249]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' 2024-09-18T07:59:16.983 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 07:59:16 smithi007 ceph-mon[39249]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' 2024-09-18T07:59:16.983 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 07:59:16 smithi007 ceph-mon[39249]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"} : dispatch 2024-09-18T07:59:16.983 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 07:59:16 smithi007 ceph-mon[39249]: Adjusting osd_memory_target on smithi007 to 16068M 2024-09-18T07:59:16.983 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 07:59:16 smithi007 ceph-mon[39249]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' 2024-09-18T07:59:16.983 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 07:59:16 smithi007 ceph-mon[39249]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-18T07:59:16.983 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 07:59:16 smithi007 ceph-mon[39249]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-18T07:59:16.983 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 07:59:16 smithi007 ceph-mon[39249]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' 2024-09-18T07:59:17.156 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 07:59:16 smithi050 ceph-mon[32371]: pgmap v37: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-09-18T07:59:17.157 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 07:59:16 smithi050 ceph-mon[32371]: osdmap e9: 1 total, 1 up, 1 in 2024-09-18T07:59:17.157 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 07:59:16 smithi050 ceph-mon[32371]: Detected new or changed devices on smithi007 2024-09-18T07:59:17.157 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 07:59:16 smithi050 ceph-mon[32371]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' 2024-09-18T07:59:17.157 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 07:59:16 smithi050 ceph-mon[32371]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' 2024-09-18T07:59:17.157 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 07:59:16 smithi050 ceph-mon[32371]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"} : dispatch 2024-09-18T07:59:17.157 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 07:59:16 smithi050 ceph-mon[32371]: Adjusting osd_memory_target on smithi007 to 16068M 2024-09-18T07:59:17.157 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 07:59:16 smithi050 ceph-mon[32371]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' 2024-09-18T07:59:17.157 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 07:59:16 smithi050 ceph-mon[32371]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-18T07:59:17.157 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 07:59:16 smithi050 ceph-mon[32371]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-18T07:59:17.157 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 07:59:16 smithi050 ceph-mon[32371]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' 2024-09-18T07:59:17.937 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-09-18T07:59:17.964 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 779349ca-7593-11ef-bceb-c7b262605968 -- ceph orch daemon add osd smithi007:vg_nvme/lv_3 2024-09-18T07:59:18.220 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/779349ca-7593-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T07:59:18.981 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 07:59:18 smithi007 ceph-mon[29459]: pgmap v39: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-09-18T07:59:18.981 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 07:59:18 smithi007 ceph-mon[39249]: pgmap v39: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-09-18T07:59:19.156 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 07:59:18 smithi050 ceph-mon[32371]: pgmap v39: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-09-18T07:59:20.156 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 07:59:19 smithi050 ceph-mon[32371]: from='client.14250 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi007:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-09-18T07:59:20.156 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 07:59:19 smithi050 ceph-mon[32371]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' cmd={"prefix": "osd tree", "states": ["destroyed"], "format": "json"} : dispatch 2024-09-18T07:59:20.157 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 07:59:19 smithi050 ceph-mon[32371]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.bootstrap-osd"} : dispatch 2024-09-18T07:59:20.157 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 07:59:19 smithi050 ceph-mon[32371]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-18T07:59:20.169 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 07:59:19 smithi007 ceph-mon[29459]: from='client.14250 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi007:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-09-18T07:59:20.169 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 07:59:19 smithi007 ceph-mon[29459]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' cmd={"prefix": "osd tree", "states": ["destroyed"], "format": "json"} : dispatch 2024-09-18T07:59:20.169 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 07:59:19 smithi007 ceph-mon[29459]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.bootstrap-osd"} : dispatch 2024-09-18T07:59:20.169 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 07:59:19 smithi007 ceph-mon[29459]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-18T07:59:20.170 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 07:59:19 smithi007 ceph-mon[39249]: from='client.14250 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi007:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-09-18T07:59:20.170 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 07:59:19 smithi007 ceph-mon[39249]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' cmd={"prefix": "osd tree", "states": ["destroyed"], "format": "json"} : dispatch 2024-09-18T07:59:20.170 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 07:59:19 smithi007 ceph-mon[39249]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.bootstrap-osd"} : dispatch 2024-09-18T07:59:20.170 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 07:59:19 smithi007 ceph-mon[39249]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-18T07:59:21.406 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 07:59:20 smithi050 ceph-mon[32371]: pgmap v40: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-09-18T07:59:21.406 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 07:59:20 smithi050 ceph-mon[32371]: from='client.? 172.21.15.7:0/2954221938' entity='client.bootstrap-osd' cmd={"prefix": "osd new", "uuid": "be28b70c-1f73-494f-ad4b-cb7ec5311999"} : dispatch 2024-09-18T07:59:21.407 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 07:59:20 smithi050 ceph-mon[32371]: from='client.? 172.21.15.7:0/2954221938' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "be28b70c-1f73-494f-ad4b-cb7ec5311999"}]': finished 2024-09-18T07:59:21.407 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 07:59:20 smithi050 ceph-mon[32371]: osdmap e10: 2 total, 1 up, 2 in 2024-09-18T07:59:21.407 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 07:59:20 smithi050 ceph-mon[32371]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 1} : dispatch 2024-09-18T07:59:21.481 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 07:59:20 smithi007 ceph-mon[29459]: pgmap v40: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-09-18T07:59:21.481 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 07:59:20 smithi007 ceph-mon[29459]: from='client.? 172.21.15.7:0/2954221938' entity='client.bootstrap-osd' cmd={"prefix": "osd new", "uuid": "be28b70c-1f73-494f-ad4b-cb7ec5311999"} : dispatch 2024-09-18T07:59:21.481 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 07:59:20 smithi007 ceph-mon[29459]: from='client.? 172.21.15.7:0/2954221938' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "be28b70c-1f73-494f-ad4b-cb7ec5311999"}]': finished 2024-09-18T07:59:21.481 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 07:59:20 smithi007 ceph-mon[29459]: osdmap e10: 2 total, 1 up, 2 in 2024-09-18T07:59:21.481 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 07:59:20 smithi007 ceph-mon[29459]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 1} : dispatch 2024-09-18T07:59:21.481 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 07:59:20 smithi007 ceph-mon[39249]: pgmap v40: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-09-18T07:59:21.481 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 07:59:20 smithi007 ceph-mon[39249]: from='client.? 172.21.15.7:0/2954221938' entity='client.bootstrap-osd' cmd={"prefix": "osd new", "uuid": "be28b70c-1f73-494f-ad4b-cb7ec5311999"} : dispatch 2024-09-18T07:59:21.482 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 07:59:20 smithi007 ceph-mon[39249]: from='client.? 172.21.15.7:0/2954221938' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "be28b70c-1f73-494f-ad4b-cb7ec5311999"}]': finished 2024-09-18T07:59:21.482 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 07:59:20 smithi007 ceph-mon[39249]: osdmap e10: 2 total, 1 up, 2 in 2024-09-18T07:59:21.482 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 07:59:20 smithi007 ceph-mon[39249]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 1} : dispatch 2024-09-18T07:59:22.406 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 07:59:21 smithi050 ceph-mon[32371]: from='client.? 172.21.15.7:0/2088232975' entity='client.bootstrap-osd' cmd={"prefix": "mon getmap"} : dispatch 2024-09-18T07:59:22.481 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 07:59:21 smithi007 ceph-mon[29459]: from='client.? 172.21.15.7:0/2088232975' entity='client.bootstrap-osd' cmd={"prefix": "mon getmap"} : dispatch 2024-09-18T07:59:22.481 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 07:59:21 smithi007 ceph-mon[39249]: from='client.? 172.21.15.7:0/2088232975' entity='client.bootstrap-osd' cmd={"prefix": "mon getmap"} : dispatch 2024-09-18T07:59:23.406 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 07:59:22 smithi050 ceph-mon[32371]: pgmap v42: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-09-18T07:59:23.481 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 07:59:22 smithi007 ceph-mon[29459]: pgmap v42: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-09-18T07:59:23.481 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 07:59:22 smithi007 ceph-mon[39249]: pgmap v42: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-09-18T07:59:25.402 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 07:59:25 smithi007 ceph-mon[29459]: pgmap v43: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-09-18T07:59:25.402 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 07:59:25 smithi007 ceph-mon[39249]: pgmap v43: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-09-18T07:59:25.406 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 07:59:25 smithi050 ceph-mon[32371]: pgmap v43: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-09-18T07:59:27.231 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 07:59:27 smithi007 ceph-mon[29459]: pgmap v44: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-09-18T07:59:27.232 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 07:59:27 smithi007 ceph-mon[39249]: pgmap v44: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-09-18T07:59:27.406 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 07:59:27 smithi050 ceph-mon[32371]: pgmap v44: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-09-18T07:59:28.191 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 07:59:28 smithi007 ceph-mon[39249]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' cmd={"prefix": "auth get", "entity": "osd.1"} : dispatch 2024-09-18T07:59:28.191 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 07:59:28 smithi007 ceph-mon[39249]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-18T07:59:28.192 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 07:59:28 smithi007 ceph-mon[29459]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' cmd={"prefix": "auth get", "entity": "osd.1"} : dispatch 2024-09-18T07:59:28.192 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 07:59:28 smithi007 ceph-mon[29459]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-18T07:59:28.408 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 07:59:28 smithi050 ceph-mon[32371]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' cmd={"prefix": "auth get", "entity": "osd.1"} : dispatch 2024-09-18T07:59:28.409 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 07:59:28 smithi050 ceph-mon[32371]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-18T07:59:29.125 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 07:59:29 smithi007 ceph-mon[29459]: pgmap v45: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-09-18T07:59:29.125 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 07:59:29 smithi007 ceph-mon[29459]: Deploying daemon osd.1 on smithi007 2024-09-18T07:59:29.125 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 07:59:29 smithi007 ceph-mon[39249]: pgmap v45: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-09-18T07:59:29.125 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 07:59:29 smithi007 ceph-mon[39249]: Deploying daemon osd.1 on smithi007 2024-09-18T07:59:29.406 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 07:59:29 smithi050 ceph-mon[32371]: pgmap v45: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-09-18T07:59:29.407 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 07:59:29 smithi050 ceph-mon[32371]: Deploying daemon osd.1 on smithi007 2024-09-18T07:59:31.231 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 07:59:31 smithi007 ceph-mon[29459]: pgmap v46: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-09-18T07:59:31.232 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 07:59:31 smithi007 ceph-mon[39249]: pgmap v46: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-09-18T07:59:31.406 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 07:59:31 smithi050 ceph-mon[32371]: pgmap v46: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-09-18T07:59:33.374 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 07:59:33 smithi007 ceph-mon[29459]: pgmap v47: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-09-18T07:59:33.375 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 07:59:33 smithi007 ceph-mon[39249]: pgmap v47: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-09-18T07:59:33.406 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 07:59:33 smithi050 ceph-mon[32371]: pgmap v47: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-09-18T07:59:35.276 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 07:59:35 smithi007 ceph-mon[29459]: pgmap v48: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-09-18T07:59:35.276 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 07:59:35 smithi007 ceph-mon[29459]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-18T07:59:35.277 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 07:59:35 smithi007 ceph-mon[29459]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' 2024-09-18T07:59:35.277 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 07:59:35 smithi007 ceph-mon[29459]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' 2024-09-18T07:59:35.277 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 07:59:35 smithi007 ceph-mon[39249]: pgmap v48: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-09-18T07:59:35.277 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 07:59:35 smithi007 ceph-mon[39249]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-18T07:59:35.277 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 07:59:35 smithi007 ceph-mon[39249]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' 2024-09-18T07:59:35.277 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 07:59:35 smithi007 ceph-mon[39249]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' 2024-09-18T07:59:35.406 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 07:59:35 smithi050 ceph-mon[32371]: pgmap v48: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-09-18T07:59:35.406 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 07:59:35 smithi050 ceph-mon[32371]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-18T07:59:35.406 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 07:59:35 smithi050 ceph-mon[32371]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' 2024-09-18T07:59:35.406 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 07:59:35 smithi050 ceph-mon[32371]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' 2024-09-18T07:59:37.406 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 07:59:37 smithi050 ceph-mon[32371]: pgmap v49: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-09-18T07:59:37.481 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 07:59:37 smithi007 ceph-mon[29459]: pgmap v49: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-09-18T07:59:37.481 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 07:59:37 smithi007 ceph-mon[39249]: pgmap v49: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-09-18T07:59:38.656 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 07:59:38 smithi050 ceph-mon[32371]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' 2024-09-18T07:59:38.656 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 07:59:38 smithi050 ceph-mon[32371]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' 2024-09-18T07:59:38.657 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 07:59:38 smithi050 ceph-mon[32371]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-18T07:59:38.657 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 07:59:38 smithi050 ceph-mon[32371]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-18T07:59:38.657 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 07:59:38 smithi050 ceph-mon[32371]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' 2024-09-18T07:59:38.657 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 07:59:38 smithi050 ceph-mon[32371]: pgmap v50: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-09-18T07:59:38.731 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 07:59:38 smithi007 ceph-mon[29459]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' 2024-09-18T07:59:38.731 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 07:59:38 smithi007 ceph-mon[29459]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' 2024-09-18T07:59:38.731 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 07:59:38 smithi007 ceph-mon[29459]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-18T07:59:38.731 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 07:59:38 smithi007 ceph-mon[29459]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-18T07:59:38.731 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 07:59:38 smithi007 ceph-mon[29459]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' 2024-09-18T07:59:38.731 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 07:59:38 smithi007 ceph-mon[29459]: pgmap v50: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-09-18T07:59:38.731 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 07:59:38 smithi007 ceph-mon[39249]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' 2024-09-18T07:59:38.732 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 07:59:38 smithi007 ceph-mon[39249]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' 2024-09-18T07:59:38.732 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 07:59:38 smithi007 ceph-mon[39249]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-18T07:59:38.732 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 07:59:38 smithi007 ceph-mon[39249]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-18T07:59:38.732 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 07:59:38 smithi007 ceph-mon[39249]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' 2024-09-18T07:59:38.732 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 07:59:38 smithi007 ceph-mon[39249]: pgmap v50: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-09-18T07:59:39.569 INFO:teuthology.orchestra.run.smithi007.stdout:Created osd(s) 1 on host 'smithi007' 2024-09-18T07:59:40.125 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 07:59:39 smithi007 ceph-mon[39249]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-18T07:59:40.125 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 07:59:39 smithi007 ceph-mon[39249]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' 2024-09-18T07:59:40.125 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 07:59:39 smithi007 ceph-mon[39249]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' 2024-09-18T07:59:40.125 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 07:59:39 smithi007 ceph-mon[39249]: from='osd.1 v2:172.21.15.7:6805/1971397549' entity='osd.1' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]} : dispatch 2024-09-18T07:59:40.126 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 07:59:39 smithi007 ceph-mon[29459]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-18T07:59:40.126 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 07:59:39 smithi007 ceph-mon[29459]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' 2024-09-18T07:59:40.126 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 07:59:39 smithi007 ceph-mon[29459]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' 2024-09-18T07:59:40.126 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 07:59:39 smithi007 ceph-mon[29459]: from='osd.1 v2:172.21.15.7:6805/1971397549' entity='osd.1' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]} : dispatch 2024-09-18T07:59:40.127 DEBUG:teuthology.orchestra.run.smithi007:osd.1> sudo journalctl -f -n 0 -u ceph-779349ca-7593-11ef-bceb-c7b262605968@osd.1.service 2024-09-18T07:59:40.130 INFO:tasks.cephadm:Deploying osd.2 on smithi007 with /dev/vg_nvme/lv_2... 2024-09-18T07:59:40.130 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e ceph-volume -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 779349ca-7593-11ef-bceb-c7b262605968 -- lvm zap /dev/vg_nvme/lv_2 2024-09-18T07:59:40.406 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 07:59:39 smithi050 ceph-mon[32371]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-18T07:59:40.406 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 07:59:39 smithi050 ceph-mon[32371]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' 2024-09-18T07:59:40.406 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 07:59:39 smithi050 ceph-mon[32371]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' 2024-09-18T07:59:40.407 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 07:59:39 smithi050 ceph-mon[32371]: from='osd.1 v2:172.21.15.7:6805/1971397549' entity='osd.1' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]} : dispatch 2024-09-18T07:59:41.277 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 07:59:40 smithi007 ceph-mon[29459]: pgmap v51: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-09-18T07:59:41.277 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 07:59:40 smithi007 ceph-mon[29459]: from='osd.1 v2:172.21.15.7:6805/1971397549' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-09-18T07:59:41.277 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 07:59:40 smithi007 ceph-mon[29459]: osdmap e11: 2 total, 1 up, 2 in 2024-09-18T07:59:41.278 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 07:59:40 smithi007 ceph-mon[29459]: from='osd.1 v2:172.21.15.7:6805/1971397549' entity='osd.1' cmd={"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi007", "root=default"]} : dispatch 2024-09-18T07:59:41.278 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 07:59:40 smithi007 ceph-mon[29459]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 1} : dispatch 2024-09-18T07:59:41.278 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 07:59:40 smithi007 ceph-mon[39249]: pgmap v51: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-09-18T07:59:41.278 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 07:59:40 smithi007 ceph-mon[39249]: from='osd.1 v2:172.21.15.7:6805/1971397549' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-09-18T07:59:41.278 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 07:59:40 smithi007 ceph-mon[39249]: osdmap e11: 2 total, 1 up, 2 in 2024-09-18T07:59:41.278 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 07:59:40 smithi007 ceph-mon[39249]: from='osd.1 v2:172.21.15.7:6805/1971397549' entity='osd.1' cmd={"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi007", "root=default"]} : dispatch 2024-09-18T07:59:41.278 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 07:59:40 smithi007 ceph-mon[39249]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 1} : dispatch 2024-09-18T07:59:41.308 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 07:59:40 smithi050 ceph-mon[32371]: pgmap v51: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-09-18T07:59:41.308 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 07:59:40 smithi050 ceph-mon[32371]: from='osd.1 v2:172.21.15.7:6805/1971397549' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-09-18T07:59:41.308 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 07:59:40 smithi050 ceph-mon[32371]: osdmap e11: 2 total, 1 up, 2 in 2024-09-18T07:59:41.308 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 07:59:40 smithi050 ceph-mon[32371]: from='osd.1 v2:172.21.15.7:6805/1971397549' entity='osd.1' cmd={"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi007", "root=default"]} : dispatch 2024-09-18T07:59:41.308 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 07:59:40 smithi050 ceph-mon[32371]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 1} : dispatch 2024-09-18T07:59:41.577 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/779349ca-7593-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T07:59:41.824 INFO:journalctl@ceph.osd.1.smithi007.stdout:Sep 18 07:59:41 smithi007 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd-1[47960]: 2024-09-18T07:59:41.571+0000 7fcd01259640 -1 osd.1 0 waiting for initial osdmap 2024-09-18T07:59:41.824 INFO:journalctl@ceph.osd.1.smithi007.stdout:Sep 18 07:59:41 smithi007 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd-1[47960]: 2024-09-18T07:59:41.574+0000 7fccfd080640 -1 osd.1 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-18T07:59:42.731 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 07:59:42 smithi007 ceph-mon[29459]: pgmap v53: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-09-18T07:59:42.731 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 07:59:42 smithi007 ceph-mon[29459]: from='osd.1 v2:172.21.15.7:6805/1971397549' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi007", "root=default"]}]': finished 2024-09-18T07:59:42.731 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 07:59:42 smithi007 ceph-mon[29459]: osdmap e12: 2 total, 1 up, 2 in 2024-09-18T07:59:42.731 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 07:59:42 smithi007 ceph-mon[29459]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 1} : dispatch 2024-09-18T07:59:42.731 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 07:59:42 smithi007 ceph-mon[29459]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 1} : dispatch 2024-09-18T07:59:42.732 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 07:59:42 smithi007 ceph-mon[39249]: pgmap v53: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-09-18T07:59:42.732 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 07:59:42 smithi007 ceph-mon[39249]: from='osd.1 v2:172.21.15.7:6805/1971397549' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi007", "root=default"]}]': finished 2024-09-18T07:59:42.732 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 07:59:42 smithi007 ceph-mon[39249]: osdmap e12: 2 total, 1 up, 2 in 2024-09-18T07:59:42.732 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 07:59:42 smithi007 ceph-mon[39249]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 1} : dispatch 2024-09-18T07:59:42.732 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 07:59:42 smithi007 ceph-mon[39249]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 1} : dispatch 2024-09-18T07:59:42.906 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 07:59:42 smithi050 ceph-mon[32371]: pgmap v53: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-09-18T07:59:42.906 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 07:59:42 smithi050 ceph-mon[32371]: from='osd.1 v2:172.21.15.7:6805/1971397549' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi007", "root=default"]}]': finished 2024-09-18T07:59:42.906 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 07:59:42 smithi050 ceph-mon[32371]: osdmap e12: 2 total, 1 up, 2 in 2024-09-18T07:59:42.906 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 07:59:42 smithi050 ceph-mon[32371]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 1} : dispatch 2024-09-18T07:59:42.906 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 07:59:42 smithi050 ceph-mon[32371]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 1} : dispatch 2024-09-18T07:59:43.858 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 07:59:43 smithi007 ceph-mon[29459]: purged_snaps scrub starts 2024-09-18T07:59:43.858 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 07:59:43 smithi007 ceph-mon[29459]: purged_snaps scrub ok 2024-09-18T07:59:43.858 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 07:59:43 smithi007 ceph-mon[29459]: osd.1 v2:172.21.15.7:6805/1971397549 boot 2024-09-18T07:59:43.858 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 07:59:43 smithi007 ceph-mon[29459]: osdmap e13: 2 total, 2 up, 2 in 2024-09-18T07:59:43.858 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 07:59:43 smithi007 ceph-mon[29459]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 1} : dispatch 2024-09-18T07:59:43.858 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 07:59:43 smithi007 ceph-mon[29459]: Detected new or changed devices on smithi007 2024-09-18T07:59:43.858 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 07:59:43 smithi007 ceph-mon[29459]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' 2024-09-18T07:59:43.858 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 07:59:43 smithi007 ceph-mon[29459]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' 2024-09-18T07:59:43.858 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 07:59:43 smithi007 ceph-mon[29459]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"} : dispatch 2024-09-18T07:59:43.859 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 07:59:43 smithi007 ceph-mon[29459]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"} : dispatch 2024-09-18T07:59:43.859 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 07:59:43 smithi007 ceph-mon[29459]: Adjusting osd_memory_target on smithi007 to 8034M 2024-09-18T07:59:43.859 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 07:59:43 smithi007 ceph-mon[29459]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' 2024-09-18T07:59:43.859 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 07:59:43 smithi007 ceph-mon[29459]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-18T07:59:43.859 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 07:59:43 smithi007 ceph-mon[29459]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-18T07:59:43.859 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 07:59:43 smithi007 ceph-mon[29459]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' 2024-09-18T07:59:43.859 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 07:59:43 smithi007 ceph-mon[39249]: purged_snaps scrub starts 2024-09-18T07:59:43.859 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 07:59:43 smithi007 ceph-mon[39249]: purged_snaps scrub ok 2024-09-18T07:59:43.859 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 07:59:43 smithi007 ceph-mon[39249]: osd.1 v2:172.21.15.7:6805/1971397549 boot 2024-09-18T07:59:43.859 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 07:59:43 smithi007 ceph-mon[39249]: osdmap e13: 2 total, 2 up, 2 in 2024-09-18T07:59:43.859 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 07:59:43 smithi007 ceph-mon[39249]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 1} : dispatch 2024-09-18T07:59:43.860 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 07:59:43 smithi007 ceph-mon[39249]: Detected new or changed devices on smithi007 2024-09-18T07:59:43.860 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 07:59:43 smithi007 ceph-mon[39249]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' 2024-09-18T07:59:43.860 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 07:59:43 smithi007 ceph-mon[39249]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' 2024-09-18T07:59:43.860 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 07:59:43 smithi007 ceph-mon[39249]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"} : dispatch 2024-09-18T07:59:43.860 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 07:59:43 smithi007 ceph-mon[39249]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"} : dispatch 2024-09-18T07:59:43.860 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 07:59:43 smithi007 ceph-mon[39249]: Adjusting osd_memory_target on smithi007 to 8034M 2024-09-18T07:59:43.860 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 07:59:43 smithi007 ceph-mon[39249]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' 2024-09-18T07:59:43.860 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 07:59:43 smithi007 ceph-mon[39249]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-18T07:59:43.860 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 07:59:43 smithi007 ceph-mon[39249]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-18T07:59:43.860 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 07:59:43 smithi007 ceph-mon[39249]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' 2024-09-18T07:59:43.906 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 07:59:43 smithi050 ceph-mon[32371]: purged_snaps scrub starts 2024-09-18T07:59:43.907 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 07:59:43 smithi050 ceph-mon[32371]: purged_snaps scrub ok 2024-09-18T07:59:43.907 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 07:59:43 smithi050 ceph-mon[32371]: osd.1 v2:172.21.15.7:6805/1971397549 boot 2024-09-18T07:59:43.907 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 07:59:43 smithi050 ceph-mon[32371]: osdmap e13: 2 total, 2 up, 2 in 2024-09-18T07:59:43.907 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 07:59:43 smithi050 ceph-mon[32371]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 1} : dispatch 2024-09-18T07:59:43.907 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 07:59:43 smithi050 ceph-mon[32371]: Detected new or changed devices on smithi007 2024-09-18T07:59:43.907 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 07:59:43 smithi050 ceph-mon[32371]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' 2024-09-18T07:59:43.907 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 07:59:43 smithi050 ceph-mon[32371]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' 2024-09-18T07:59:43.907 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 07:59:43 smithi050 ceph-mon[32371]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"} : dispatch 2024-09-18T07:59:43.907 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 07:59:43 smithi050 ceph-mon[32371]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"} : dispatch 2024-09-18T07:59:43.907 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 07:59:43 smithi050 ceph-mon[32371]: Adjusting osd_memory_target on smithi007 to 8034M 2024-09-18T07:59:43.907 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 07:59:43 smithi050 ceph-mon[32371]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' 2024-09-18T07:59:43.908 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 07:59:43 smithi050 ceph-mon[32371]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-18T07:59:43.908 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 07:59:43 smithi050 ceph-mon[32371]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-18T07:59:43.908 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 07:59:43 smithi050 ceph-mon[32371]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' 2024-09-18T07:59:44.803 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-09-18T07:59:44.829 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 07:59:44 smithi007 ceph-mon[29459]: pgmap v56: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-09-18T07:59:44.829 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 07:59:44 smithi007 ceph-mon[29459]: osdmap e14: 2 total, 2 up, 2 in 2024-09-18T07:59:44.829 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 07:59:44 smithi007 ceph-mon[39249]: pgmap v56: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-09-18T07:59:44.829 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 07:59:44 smithi007 ceph-mon[39249]: osdmap e14: 2 total, 2 up, 2 in 2024-09-18T07:59:44.830 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 779349ca-7593-11ef-bceb-c7b262605968 -- ceph orch daemon add osd smithi007:vg_nvme/lv_2 2024-09-18T07:59:45.087 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/779349ca-7593-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T07:59:45.156 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 07:59:44 smithi050 ceph-mon[32371]: pgmap v56: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-09-18T07:59:45.156 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 07:59:44 smithi050 ceph-mon[32371]: osdmap e14: 2 total, 2 up, 2 in 2024-09-18T07:59:46.137 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 07:59:45 smithi007 ceph-mon[29459]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' cmd={"prefix": "osd tree", "states": ["destroyed"], "format": "json"} : dispatch 2024-09-18T07:59:46.137 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 07:59:45 smithi007 ceph-mon[29459]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.bootstrap-osd"} : dispatch 2024-09-18T07:59:46.137 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 07:59:45 smithi007 ceph-mon[29459]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-18T07:59:46.137 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 07:59:45 smithi007 ceph-mon[39249]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' cmd={"prefix": "osd tree", "states": ["destroyed"], "format": "json"} : dispatch 2024-09-18T07:59:46.137 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 07:59:45 smithi007 ceph-mon[39249]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.bootstrap-osd"} : dispatch 2024-09-18T07:59:46.137 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 07:59:45 smithi007 ceph-mon[39249]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-18T07:59:46.156 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 07:59:45 smithi050 ceph-mon[32371]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' cmd={"prefix": "osd tree", "states": ["destroyed"], "format": "json"} : dispatch 2024-09-18T07:59:46.156 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 07:59:45 smithi050 ceph-mon[32371]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.bootstrap-osd"} : dispatch 2024-09-18T07:59:46.156 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 07:59:45 smithi050 ceph-mon[32371]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-18T07:59:46.925 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 07:59:46 smithi007 ceph-mon[29459]: pgmap v58: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-09-18T07:59:46.926 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 07:59:46 smithi007 ceph-mon[29459]: from='client.14277 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi007:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-09-18T07:59:46.926 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 07:59:46 smithi007 ceph-mon[39249]: pgmap v58: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-09-18T07:59:46.926 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 07:59:46 smithi007 ceph-mon[39249]: from='client.14277 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi007:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-09-18T07:59:47.156 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 07:59:46 smithi050 ceph-mon[32371]: pgmap v58: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-09-18T07:59:47.156 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 07:59:46 smithi050 ceph-mon[32371]: from='client.14277 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi007:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-09-18T07:59:49.156 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 07:59:48 smithi050 ceph-mon[32371]: pgmap v59: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-09-18T07:59:49.156 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 07:59:48 smithi050 ceph-mon[32371]: from='client.? 172.21.15.7:0/1237376045' entity='client.bootstrap-osd' cmd={"prefix": "osd new", "uuid": "cfa4f083-c7fd-4743-be65-84a3040fe6dc"} : dispatch 2024-09-18T07:59:49.156 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 07:59:48 smithi050 ceph-mon[32371]: from='client.? ' entity='client.bootstrap-osd' cmd={"prefix": "osd new", "uuid": "cfa4f083-c7fd-4743-be65-84a3040fe6dc"} : dispatch 2024-09-18T07:59:49.157 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 07:59:48 smithi050 ceph-mon[32371]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "cfa4f083-c7fd-4743-be65-84a3040fe6dc"}]': finished 2024-09-18T07:59:49.157 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 07:59:48 smithi050 ceph-mon[32371]: osdmap e15: 3 total, 2 up, 3 in 2024-09-18T07:59:49.157 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 07:59:48 smithi050 ceph-mon[32371]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 2} : dispatch 2024-09-18T07:59:49.157 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 07:59:48 smithi050 ceph-mon[32371]: from='client.? 172.21.15.7:0/3001278020' entity='client.bootstrap-osd' cmd={"prefix": "mon getmap"} : dispatch 2024-09-18T07:59:49.231 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 07:59:48 smithi007 ceph-mon[29459]: pgmap v59: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-09-18T07:59:49.231 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 07:59:48 smithi007 ceph-mon[29459]: from='client.? 172.21.15.7:0/1237376045' entity='client.bootstrap-osd' cmd={"prefix": "osd new", "uuid": "cfa4f083-c7fd-4743-be65-84a3040fe6dc"} : dispatch 2024-09-18T07:59:49.231 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 07:59:48 smithi007 ceph-mon[29459]: from='client.? ' entity='client.bootstrap-osd' cmd={"prefix": "osd new", "uuid": "cfa4f083-c7fd-4743-be65-84a3040fe6dc"} : dispatch 2024-09-18T07:59:49.231 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 07:59:48 smithi007 ceph-mon[29459]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "cfa4f083-c7fd-4743-be65-84a3040fe6dc"}]': finished 2024-09-18T07:59:49.231 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 07:59:48 smithi007 ceph-mon[29459]: osdmap e15: 3 total, 2 up, 3 in 2024-09-18T07:59:49.232 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 07:59:48 smithi007 ceph-mon[29459]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 2} : dispatch 2024-09-18T07:59:49.232 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 07:59:48 smithi007 ceph-mon[29459]: from='client.? 172.21.15.7:0/3001278020' entity='client.bootstrap-osd' cmd={"prefix": "mon getmap"} : dispatch 2024-09-18T07:59:49.232 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 07:59:48 smithi007 ceph-mon[39249]: pgmap v59: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-09-18T07:59:49.232 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 07:59:48 smithi007 ceph-mon[39249]: from='client.? 172.21.15.7:0/1237376045' entity='client.bootstrap-osd' cmd={"prefix": "osd new", "uuid": "cfa4f083-c7fd-4743-be65-84a3040fe6dc"} : dispatch 2024-09-18T07:59:49.232 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 07:59:48 smithi007 ceph-mon[39249]: from='client.? ' entity='client.bootstrap-osd' cmd={"prefix": "osd new", "uuid": "cfa4f083-c7fd-4743-be65-84a3040fe6dc"} : dispatch 2024-09-18T07:59:49.232 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 07:59:48 smithi007 ceph-mon[39249]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "cfa4f083-c7fd-4743-be65-84a3040fe6dc"}]': finished 2024-09-18T07:59:49.232 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 07:59:48 smithi007 ceph-mon[39249]: osdmap e15: 3 total, 2 up, 3 in 2024-09-18T07:59:49.232 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 07:59:48 smithi007 ceph-mon[39249]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 2} : dispatch 2024-09-18T07:59:49.232 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 07:59:48 smithi007 ceph-mon[39249]: from='client.? 172.21.15.7:0/3001278020' entity='client.bootstrap-osd' cmd={"prefix": "mon getmap"} : dispatch 2024-09-18T07:59:51.156 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 07:59:50 smithi050 ceph-mon[32371]: pgmap v61: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-09-18T07:59:51.231 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 07:59:50 smithi007 ceph-mon[29459]: pgmap v61: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-09-18T07:59:51.231 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 07:59:50 smithi007 ceph-mon[39249]: pgmap v61: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-09-18T07:59:52.981 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 07:59:52 smithi007 ceph-mon[29459]: pgmap v62: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-09-18T07:59:52.981 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 07:59:52 smithi007 ceph-mon[39249]: pgmap v62: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-09-18T07:59:53.156 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 07:59:52 smithi050 ceph-mon[32371]: pgmap v62: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-09-18T07:59:55.156 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 07:59:54 smithi050 ceph-mon[32371]: pgmap v63: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-09-18T07:59:55.231 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 07:59:54 smithi007 ceph-mon[39249]: pgmap v63: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-09-18T07:59:55.231 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 07:59:54 smithi007 ceph-mon[29459]: pgmap v63: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-09-18T07:59:55.981 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 07:59:55 smithi007 ceph-mon[29459]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' cmd={"prefix": "auth get", "entity": "osd.2"} : dispatch 2024-09-18T07:59:55.981 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 07:59:55 smithi007 ceph-mon[29459]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-18T07:59:55.981 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 07:59:55 smithi007 ceph-mon[29459]: Deploying daemon osd.2 on smithi007 2024-09-18T07:59:55.981 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 07:59:55 smithi007 ceph-mon[39249]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' cmd={"prefix": "auth get", "entity": "osd.2"} : dispatch 2024-09-18T07:59:55.981 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 07:59:55 smithi007 ceph-mon[39249]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-18T07:59:55.981 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 07:59:55 smithi007 ceph-mon[39249]: Deploying daemon osd.2 on smithi007 2024-09-18T07:59:56.156 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 07:59:55 smithi050 ceph-mon[32371]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' cmd={"prefix": "auth get", "entity": "osd.2"} : dispatch 2024-09-18T07:59:56.156 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 07:59:55 smithi050 ceph-mon[32371]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-18T07:59:56.157 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 07:59:55 smithi050 ceph-mon[32371]: Deploying daemon osd.2 on smithi007 2024-09-18T07:59:56.971 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 07:59:56 smithi007 ceph-mon[29459]: pgmap v64: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-09-18T07:59:56.971 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 07:59:56 smithi007 ceph-mon[39249]: pgmap v64: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-09-18T07:59:57.156 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 07:59:56 smithi050 ceph-mon[32371]: pgmap v64: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-09-18T07:59:58.981 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 07:59:58 smithi007 ceph-mon[29459]: pgmap v65: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-09-18T07:59:58.981 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 07:59:58 smithi007 ceph-mon[39249]: pgmap v65: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-09-18T07:59:59.156 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 07:59:58 smithi050 ceph-mon[32371]: pgmap v65: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-09-18T08:00:01.384 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:00:01 smithi007 ceph-mon[29459]: pgmap v66: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-09-18T08:00:01.385 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:00:01 smithi007 ceph-mon[29459]: overall HEALTH_OK 2024-09-18T08:00:01.385 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:00:01 smithi007 ceph-mon[39249]: pgmap v66: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-09-18T08:00:01.385 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:00:01 smithi007 ceph-mon[39249]: overall HEALTH_OK 2024-09-18T08:00:01.406 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:00:01 smithi050 ceph-mon[32371]: pgmap v66: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-09-18T08:00:01.406 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:00:01 smithi050 ceph-mon[32371]: overall HEALTH_OK 2024-09-18T08:00:03.332 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:00:03 smithi007 ceph-mon[29459]: pgmap v67: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-09-18T08:00:03.333 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:00:03 smithi007 ceph-mon[29459]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-18T08:00:03.333 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:00:03 smithi007 ceph-mon[29459]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' 2024-09-18T08:00:03.333 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:00:03 smithi007 ceph-mon[29459]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' 2024-09-18T08:00:03.333 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:00:03 smithi007 ceph-mon[39249]: pgmap v67: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-09-18T08:00:03.333 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:00:03 smithi007 ceph-mon[39249]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-18T08:00:03.333 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:00:03 smithi007 ceph-mon[39249]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' 2024-09-18T08:00:03.333 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:00:03 smithi007 ceph-mon[39249]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' 2024-09-18T08:00:03.406 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:00:03 smithi050 ceph-mon[32371]: pgmap v67: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-09-18T08:00:03.406 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:00:03 smithi050 ceph-mon[32371]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-18T08:00:03.406 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:00:03 smithi050 ceph-mon[32371]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' 2024-09-18T08:00:03.406 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:00:03 smithi050 ceph-mon[32371]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' 2024-09-18T08:00:05.296 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:00:05 smithi007 ceph-mon[29459]: pgmap v68: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-09-18T08:00:05.296 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:00:05 smithi007 ceph-mon[39249]: pgmap v68: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-09-18T08:00:05.406 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:00:05 smithi050 ceph-mon[32371]: pgmap v68: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-09-18T08:00:06.906 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:00:06 smithi050 ceph-mon[32371]: pgmap v69: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-09-18T08:00:06.906 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:00:06 smithi050 ceph-mon[32371]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' 2024-09-18T08:00:06.906 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:00:06 smithi050 ceph-mon[32371]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' 2024-09-18T08:00:06.906 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:00:06 smithi050 ceph-mon[32371]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-18T08:00:06.907 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:00:06 smithi050 ceph-mon[32371]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-18T08:00:06.907 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:00:06 smithi050 ceph-mon[32371]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' 2024-09-18T08:00:06.981 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:00:06 smithi007 ceph-mon[29459]: pgmap v69: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-09-18T08:00:06.981 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:00:06 smithi007 ceph-mon[29459]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' 2024-09-18T08:00:06.981 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:00:06 smithi007 ceph-mon[29459]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' 2024-09-18T08:00:06.981 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:00:06 smithi007 ceph-mon[29459]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-18T08:00:06.981 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:00:06 smithi007 ceph-mon[29459]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-18T08:00:06.981 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:00:06 smithi007 ceph-mon[29459]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' 2024-09-18T08:00:06.981 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:00:06 smithi007 ceph-mon[39249]: pgmap v69: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-09-18T08:00:06.982 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:00:06 smithi007 ceph-mon[39249]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' 2024-09-18T08:00:06.982 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:00:06 smithi007 ceph-mon[39249]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' 2024-09-18T08:00:06.982 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:00:06 smithi007 ceph-mon[39249]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-18T08:00:06.982 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:00:06 smithi007 ceph-mon[39249]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-18T08:00:06.982 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:00:06 smithi007 ceph-mon[39249]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' 2024-09-18T08:00:08.378 INFO:teuthology.orchestra.run.smithi007.stdout:Created osd(s) 2 on host 'smithi007' 2024-09-18T08:00:08.874 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:00:08 smithi007 ceph-mon[29459]: pgmap v70: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-09-18T08:00:08.874 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:00:08 smithi007 ceph-mon[29459]: from='osd.2 v2:172.21.15.7:6809/556142559' entity='osd.2' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]} : dispatch 2024-09-18T08:00:08.874 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:00:08 smithi007 ceph-mon[29459]: from='osd.2 ' entity='osd.2' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]} : dispatch 2024-09-18T08:00:08.874 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:00:08 smithi007 ceph-mon[29459]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-18T08:00:08.874 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:00:08 smithi007 ceph-mon[29459]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' 2024-09-18T08:00:08.874 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:00:08 smithi007 ceph-mon[29459]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' 2024-09-18T08:00:08.874 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:00:08 smithi007 ceph-mon[39249]: pgmap v70: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-09-18T08:00:08.874 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:00:08 smithi007 ceph-mon[39249]: from='osd.2 v2:172.21.15.7:6809/556142559' entity='osd.2' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]} : dispatch 2024-09-18T08:00:08.875 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:00:08 smithi007 ceph-mon[39249]: from='osd.2 ' entity='osd.2' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]} : dispatch 2024-09-18T08:00:08.875 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:00:08 smithi007 ceph-mon[39249]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-18T08:00:08.875 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:00:08 smithi007 ceph-mon[39249]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' 2024-09-18T08:00:08.875 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:00:08 smithi007 ceph-mon[39249]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' 2024-09-18T08:00:08.906 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:00:08 smithi050 ceph-mon[32371]: pgmap v70: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-09-18T08:00:08.906 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:00:08 smithi050 ceph-mon[32371]: from='osd.2 v2:172.21.15.7:6809/556142559' entity='osd.2' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]} : dispatch 2024-09-18T08:00:08.906 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:00:08 smithi050 ceph-mon[32371]: from='osd.2 ' entity='osd.2' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]} : dispatch 2024-09-18T08:00:08.906 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:00:08 smithi050 ceph-mon[32371]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-18T08:00:08.906 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:00:08 smithi050 ceph-mon[32371]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' 2024-09-18T08:00:08.906 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:00:08 smithi050 ceph-mon[32371]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' 2024-09-18T08:00:09.035 DEBUG:teuthology.orchestra.run.smithi007:osd.2> sudo journalctl -f -n 0 -u ceph-779349ca-7593-11ef-bceb-c7b262605968@osd.2.service 2024-09-18T08:00:09.038 INFO:tasks.cephadm:Deploying osd.3 on smithi007 with /dev/vg_nvme/lv_1... 2024-09-18T08:00:09.038 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e ceph-volume -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 779349ca-7593-11ef-bceb-c7b262605968 -- lvm zap /dev/vg_nvme/lv_1 2024-09-18T08:00:09.714 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/779349ca-7593-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T08:00:09.731 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:00:09 smithi007 ceph-mon[29459]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-09-18T08:00:09.731 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:00:09 smithi007 ceph-mon[29459]: osdmap e16: 3 total, 2 up, 3 in 2024-09-18T08:00:09.731 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:00:09 smithi007 ceph-mon[29459]: from='osd.2 v2:172.21.15.7:6809/556142559' entity='osd.2' cmd={"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi007", "root=default"]} : dispatch 2024-09-18T08:00:09.731 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:00:09 smithi007 ceph-mon[29459]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 2} : dispatch 2024-09-18T08:00:09.731 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:00:09 smithi007 ceph-mon[29459]: from='osd.2 ' entity='osd.2' cmd={"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi007", "root=default"]} : dispatch 2024-09-18T08:00:09.732 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:00:09 smithi007 ceph-mon[39249]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-09-18T08:00:09.732 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:00:09 smithi007 ceph-mon[39249]: osdmap e16: 3 total, 2 up, 3 in 2024-09-18T08:00:09.732 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:00:09 smithi007 ceph-mon[39249]: from='osd.2 v2:172.21.15.7:6809/556142559' entity='osd.2' cmd={"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi007", "root=default"]} : dispatch 2024-09-18T08:00:09.732 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:00:09 smithi007 ceph-mon[39249]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 2} : dispatch 2024-09-18T08:00:09.732 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:00:09 smithi007 ceph-mon[39249]: from='osd.2 ' entity='osd.2' cmd={"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi007", "root=default"]} : dispatch 2024-09-18T08:00:09.906 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:00:09 smithi050 ceph-mon[32371]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-09-18T08:00:09.906 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:00:09 smithi050 ceph-mon[32371]: osdmap e16: 3 total, 2 up, 3 in 2024-09-18T08:00:09.906 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:00:09 smithi050 ceph-mon[32371]: from='osd.2 v2:172.21.15.7:6809/556142559' entity='osd.2' cmd={"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi007", "root=default"]} : dispatch 2024-09-18T08:00:09.906 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:00:09 smithi050 ceph-mon[32371]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 2} : dispatch 2024-09-18T08:00:09.906 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:00:09 smithi050 ceph-mon[32371]: from='osd.2 ' entity='osd.2' cmd={"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi007", "root=default"]} : dispatch 2024-09-18T08:00:10.098 INFO:journalctl@ceph.osd.2.smithi007.stdout:Sep 18 08:00:09 smithi007 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd-2[53338]: 2024-09-18T08:00:09.889+0000 7f7a5303f640 -1 osd.2 0 waiting for initial osdmap 2024-09-18T08:00:10.099 INFO:journalctl@ceph.osd.2.smithi007.stdout:Sep 18 08:00:09 smithi007 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd-2[53338]: 2024-09-18T08:00:09.892+0000 7f7a4ae5e640 -1 osd.2 17 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-18T08:00:10.887 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:00:10 smithi007 ceph-mon[29459]: pgmap v72: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-09-18T08:00:10.888 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:00:10 smithi007 ceph-mon[29459]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi007", "root=default"]}]': finished 2024-09-18T08:00:10.888 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:00:10 smithi007 ceph-mon[29459]: osdmap e17: 3 total, 2 up, 3 in 2024-09-18T08:00:10.888 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:00:10 smithi007 ceph-mon[29459]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 2} : dispatch 2024-09-18T08:00:10.888 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:00:10 smithi007 ceph-mon[29459]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 2} : dispatch 2024-09-18T08:00:10.888 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:00:10 smithi007 ceph-mon[29459]: from='osd.2 ' entity='osd.2' 2024-09-18T08:00:10.888 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:00:10 smithi007 ceph-mon[39249]: pgmap v72: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-09-18T08:00:10.888 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:00:10 smithi007 ceph-mon[39249]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi007", "root=default"]}]': finished 2024-09-18T08:00:10.888 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:00:10 smithi007 ceph-mon[39249]: osdmap e17: 3 total, 2 up, 3 in 2024-09-18T08:00:10.889 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:00:10 smithi007 ceph-mon[39249]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 2} : dispatch 2024-09-18T08:00:10.889 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:00:10 smithi007 ceph-mon[39249]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 2} : dispatch 2024-09-18T08:00:10.889 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:00:10 smithi007 ceph-mon[39249]: from='osd.2 ' entity='osd.2' 2024-09-18T08:00:10.906 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:00:10 smithi050 ceph-mon[32371]: pgmap v72: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-09-18T08:00:10.906 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:00:10 smithi050 ceph-mon[32371]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi007", "root=default"]}]': finished 2024-09-18T08:00:10.906 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:00:10 smithi050 ceph-mon[32371]: osdmap e17: 3 total, 2 up, 3 in 2024-09-18T08:00:10.906 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:00:10 smithi050 ceph-mon[32371]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 2} : dispatch 2024-09-18T08:00:10.906 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:00:10 smithi050 ceph-mon[32371]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 2} : dispatch 2024-09-18T08:00:10.907 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:00:10 smithi050 ceph-mon[32371]: from='osd.2 ' entity='osd.2' 2024-09-18T08:00:11.906 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:00:11 smithi050 ceph-mon[32371]: purged_snaps scrub starts 2024-09-18T08:00:11.906 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:00:11 smithi050 ceph-mon[32371]: purged_snaps scrub ok 2024-09-18T08:00:11.906 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:00:11 smithi050 ceph-mon[32371]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 2} : dispatch 2024-09-18T08:00:11.906 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:00:11 smithi050 ceph-mon[32371]: osd.2 v2:172.21.15.7:6809/556142559 boot 2024-09-18T08:00:11.906 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:00:11 smithi050 ceph-mon[32371]: osdmap e18: 3 total, 3 up, 3 in 2024-09-18T08:00:11.906 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:00:11 smithi050 ceph-mon[32371]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 2} : dispatch 2024-09-18T08:00:11.981 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:00:11 smithi007 ceph-mon[29459]: purged_snaps scrub starts 2024-09-18T08:00:11.981 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:00:11 smithi007 ceph-mon[29459]: purged_snaps scrub ok 2024-09-18T08:00:11.981 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:00:11 smithi007 ceph-mon[29459]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 2} : dispatch 2024-09-18T08:00:11.981 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:00:11 smithi007 ceph-mon[29459]: osd.2 v2:172.21.15.7:6809/556142559 boot 2024-09-18T08:00:11.981 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:00:11 smithi007 ceph-mon[29459]: osdmap e18: 3 total, 3 up, 3 in 2024-09-18T08:00:11.982 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:00:11 smithi007 ceph-mon[29459]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 2} : dispatch 2024-09-18T08:00:11.982 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:00:11 smithi007 ceph-mon[39249]: purged_snaps scrub starts 2024-09-18T08:00:11.982 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:00:11 smithi007 ceph-mon[39249]: purged_snaps scrub ok 2024-09-18T08:00:11.982 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:00:11 smithi007 ceph-mon[39249]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 2} : dispatch 2024-09-18T08:00:11.982 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:00:11 smithi007 ceph-mon[39249]: osd.2 v2:172.21.15.7:6809/556142559 boot 2024-09-18T08:00:11.982 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:00:11 smithi007 ceph-mon[39249]: osdmap e18: 3 total, 3 up, 3 in 2024-09-18T08:00:11.982 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:00:11 smithi007 ceph-mon[39249]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 2} : dispatch 2024-09-18T08:00:12.731 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:00:12 smithi007 ceph-mon[29459]: pgmap v75: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-09-18T08:00:12.731 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:00:12 smithi007 ceph-mon[29459]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' cmd={"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true} : dispatch 2024-09-18T08:00:12.731 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:00:12 smithi007 ceph-mon[29459]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' cmd='[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]': finished 2024-09-18T08:00:12.731 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:00:12 smithi007 ceph-mon[29459]: osdmap e19: 3 total, 3 up, 3 in 2024-09-18T08:00:12.731 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:00:12 smithi007 ceph-mon[29459]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' cmd={"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true} : dispatch 2024-09-18T08:00:12.732 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:00:12 smithi007 ceph-mon[39249]: pgmap v75: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-09-18T08:00:12.732 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:00:12 smithi007 ceph-mon[39249]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' cmd={"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true} : dispatch 2024-09-18T08:00:12.732 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:00:12 smithi007 ceph-mon[39249]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' cmd='[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]': finished 2024-09-18T08:00:12.732 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:00:12 smithi007 ceph-mon[39249]: osdmap e19: 3 total, 3 up, 3 in 2024-09-18T08:00:12.732 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:00:12 smithi007 ceph-mon[39249]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' cmd={"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true} : dispatch 2024-09-18T08:00:12.906 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:00:12 smithi050 ceph-mon[32371]: pgmap v75: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-09-18T08:00:12.906 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:00:12 smithi050 ceph-mon[32371]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' cmd={"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true} : dispatch 2024-09-18T08:00:12.906 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:00:12 smithi050 ceph-mon[32371]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' cmd='[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]': finished 2024-09-18T08:00:12.906 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:00:12 smithi050 ceph-mon[32371]: osdmap e19: 3 total, 3 up, 3 in 2024-09-18T08:00:12.906 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:00:12 smithi050 ceph-mon[32371]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' cmd={"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true} : dispatch 2024-09-18T08:00:13.025 INFO:journalctl@ceph.osd.1.smithi007.stdout:Sep 18 08:00:13 smithi007 sudo[56733]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-09-18T08:00:13.026 INFO:journalctl@ceph.osd.1.smithi007.stdout:Sep 18 08:00:13 smithi007 sudo[56733]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-09-18T08:00:13.026 INFO:journalctl@ceph.osd.1.smithi007.stdout:Sep 18 08:00:13 smithi007 sudo[56733]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=167) 2024-09-18T08:00:13.026 INFO:journalctl@ceph.osd.1.smithi007.stdout:Sep 18 08:00:13 smithi007 sudo[56733]: pam_unix(sudo:session): session closed for user root 2024-09-18T08:00:13.026 INFO:journalctl@ceph.osd.0.smithi007.stdout:Sep 18 08:00:12 smithi007 sudo[56727]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-09-18T08:00:13.026 INFO:journalctl@ceph.osd.0.smithi007.stdout:Sep 18 08:00:12 smithi007 sudo[56727]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-09-18T08:00:13.026 INFO:journalctl@ceph.osd.0.smithi007.stdout:Sep 18 08:00:12 smithi007 sudo[56727]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=167) 2024-09-18T08:00:13.026 INFO:journalctl@ceph.osd.0.smithi007.stdout:Sep 18 08:00:12 smithi007 sudo[56727]: pam_unix(sudo:session): session closed for user root 2024-09-18T08:00:13.026 INFO:journalctl@ceph.osd.0.smithi007.stdout:Sep 18 08:00:12 smithi007 sudo[56730]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-09-18T08:00:13.026 INFO:journalctl@ceph.osd.0.smithi007.stdout:Sep 18 08:00:12 smithi007 sudo[56730]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-09-18T08:00:13.026 INFO:journalctl@ceph.osd.0.smithi007.stdout:Sep 18 08:00:12 smithi007 sudo[56730]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=167) 2024-09-18T08:00:13.027 INFO:journalctl@ceph.osd.0.smithi007.stdout:Sep 18 08:00:12 smithi007 sudo[56730]: pam_unix(sudo:session): session closed for user root 2024-09-18T08:00:13.358 INFO:journalctl@ceph.osd.1.smithi007.stdout:Sep 18 08:00:13 smithi007 sudo[56736]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-09-18T08:00:13.359 INFO:journalctl@ceph.osd.1.smithi007.stdout:Sep 18 08:00:13 smithi007 sudo[56736]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-09-18T08:00:13.359 INFO:journalctl@ceph.osd.1.smithi007.stdout:Sep 18 08:00:13 smithi007 sudo[56736]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=167) 2024-09-18T08:00:13.359 INFO:journalctl@ceph.osd.1.smithi007.stdout:Sep 18 08:00:13 smithi007 sudo[56736]: pam_unix(sudo:session): session closed for user root 2024-09-18T08:00:13.359 INFO:journalctl@ceph.osd.2.smithi007.stdout:Sep 18 08:00:13 smithi007 sudo[56739]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-09-18T08:00:13.359 INFO:journalctl@ceph.osd.2.smithi007.stdout:Sep 18 08:00:13 smithi007 sudo[56739]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-09-18T08:00:13.359 INFO:journalctl@ceph.osd.2.smithi007.stdout:Sep 18 08:00:13 smithi007 sudo[56739]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=167) 2024-09-18T08:00:13.359 INFO:journalctl@ceph.osd.2.smithi007.stdout:Sep 18 08:00:13 smithi007 sudo[56739]: pam_unix(sudo:session): session closed for user root 2024-09-18T08:00:13.359 INFO:journalctl@ceph.osd.2.smithi007.stdout:Sep 18 08:00:13 smithi007 sudo[56742]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-09-18T08:00:13.360 INFO:journalctl@ceph.osd.2.smithi007.stdout:Sep 18 08:00:13 smithi007 sudo[56742]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-09-18T08:00:13.360 INFO:journalctl@ceph.osd.2.smithi007.stdout:Sep 18 08:00:13 smithi007 sudo[56742]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=167) 2024-09-18T08:00:13.360 INFO:journalctl@ceph.osd.2.smithi007.stdout:Sep 18 08:00:13 smithi007 sudo[56742]: pam_unix(sudo:session): session closed for user root 2024-09-18T08:00:13.360 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:00:13 smithi007 sudo[56745]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-09-18T08:00:13.360 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:00:13 smithi007 sudo[56745]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-09-18T08:00:13.360 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:00:13 smithi007 sudo[56745]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=167) 2024-09-18T08:00:13.360 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:00:13 smithi007 sudo[56745]: pam_unix(sudo:session): session closed for user root 2024-09-18T08:00:13.360 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:00:13 smithi007 sudo[56750]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-09-18T08:00:13.360 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:00:13 smithi007 sudo[56750]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-09-18T08:00:13.361 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:00:13 smithi007 sudo[56750]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=167) 2024-09-18T08:00:13.361 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:00:13 smithi007 sudo[56750]: pam_unix(sudo:session): session closed for user root 2024-09-18T08:00:13.361 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:00:13 smithi007 sudo[56756]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-09-18T08:00:13.361 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:00:13 smithi007 sudo[56756]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-09-18T08:00:13.361 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:00:13 smithi007 sudo[56756]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=167) 2024-09-18T08:00:13.361 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:00:13 smithi007 sudo[56756]: pam_unix(sudo:session): session closed for user root 2024-09-18T08:00:13.361 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:00:13 smithi007 sudo[56759]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-09-18T08:00:13.361 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:00:13 smithi007 sudo[56759]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-09-18T08:00:13.361 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:00:13 smithi007 sudo[56759]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=167) 2024-09-18T08:00:13.361 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:00:13 smithi007 sudo[56759]: pam_unix(sudo:session): session closed for user root 2024-09-18T08:00:13.406 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:00:13 smithi050 sudo[34317]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-09-18T08:00:13.406 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:00:13 smithi050 sudo[34317]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-09-18T08:00:13.406 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:00:13 smithi050 sudo[34317]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=167) 2024-09-18T08:00:13.406 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:00:13 smithi050 sudo[34317]: pam_unix(sudo:session): session closed for user root 2024-09-18T08:00:13.406 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:00:13 smithi050 sudo[34320]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-09-18T08:00:13.407 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:00:13 smithi050 sudo[34320]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-09-18T08:00:13.407 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:00:13 smithi050 sudo[34320]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=167) 2024-09-18T08:00:13.407 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:00:13 smithi050 sudo[34320]: pam_unix(sudo:session): session closed for user root 2024-09-18T08:00:14.156 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:00:13 smithi050 ceph-mon[32371]: Detected new or changed devices on smithi007 2024-09-18T08:00:14.157 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:00:13 smithi050 ceph-mon[32371]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' 2024-09-18T08:00:14.157 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:00:13 smithi050 ceph-mon[32371]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' 2024-09-18T08:00:14.157 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:00:13 smithi050 ceph-mon[32371]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"} : dispatch 2024-09-18T08:00:14.157 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:00:13 smithi050 ceph-mon[32371]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"} : dispatch 2024-09-18T08:00:14.157 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:00:13 smithi050 ceph-mon[32371]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"} : dispatch 2024-09-18T08:00:14.157 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:00:13 smithi050 ceph-mon[32371]: Adjusting osd_memory_target on smithi007 to 5356M 2024-09-18T08:00:14.157 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:00:13 smithi050 ceph-mon[32371]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' 2024-09-18T08:00:14.157 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:00:13 smithi050 ceph-mon[32371]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-18T08:00:14.157 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:00:13 smithi050 ceph-mon[32371]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-18T08:00:14.157 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:00:13 smithi050 ceph-mon[32371]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' 2024-09-18T08:00:14.158 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:00:13 smithi050 ceph-mon[32371]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2024-09-18T08:00:14.158 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:00:13 smithi050 ceph-mon[32371]: osdmap e20: 3 total, 3 up, 3 in 2024-09-18T08:00:14.158 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:00:13 smithi050 ceph-mon[32371]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' 2024-09-18T08:00:14.158 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:00:13 smithi050 ceph-mon[32371]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-09-18T08:00:14.158 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:00:13 smithi050 ceph-mon[32371]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-09-18T08:00:14.158 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:00:13 smithi050 ceph-mon[32371]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "a"} : dispatch 2024-09-18T08:00:14.158 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:00:13 smithi050 ceph-mon[32371]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-09-18T08:00:14.158 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:00:13 smithi050 ceph-mon[32371]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-09-18T08:00:14.158 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:00:13 smithi050 ceph-mon[32371]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-09-18T08:00:14.158 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:00:13 smithi050 ceph-mon[32371]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "a"} : dispatch 2024-09-18T08:00:14.158 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:00:13 smithi050 ceph-mon[32371]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-09-18T08:00:14.158 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:00:13 smithi050 ceph-mon[32371]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-09-18T08:00:14.158 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:00:13 smithi050 ceph-mon[32371]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-09-18T08:00:14.159 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:00:13 smithi050 ceph-mon[32371]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' 2024-09-18T08:00:14.159 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:00:13 smithi050 ceph-mon[32371]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-09-18T08:00:14.159 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:00:13 smithi050 ceph-mon[32371]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "a"} : dispatch 2024-09-18T08:00:14.159 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:00:13 smithi050 ceph-mon[32371]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-09-18T08:00:14.159 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:00:13 smithi050 ceph-mon[32371]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-09-18T08:00:14.159 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:00:13 smithi050 ceph-mon[32371]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-09-18T08:00:14.231 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:00:13 smithi007 ceph-mon[39249]: Detected new or changed devices on smithi007 2024-09-18T08:00:14.232 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:00:13 smithi007 ceph-mon[39249]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' 2024-09-18T08:00:14.232 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:00:13 smithi007 ceph-mon[39249]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' 2024-09-18T08:00:14.232 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:00:13 smithi007 ceph-mon[39249]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"} : dispatch 2024-09-18T08:00:14.232 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:00:13 smithi007 ceph-mon[39249]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"} : dispatch 2024-09-18T08:00:14.232 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:00:13 smithi007 ceph-mon[39249]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"} : dispatch 2024-09-18T08:00:14.232 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:00:13 smithi007 ceph-mon[39249]: Adjusting osd_memory_target on smithi007 to 5356M 2024-09-18T08:00:14.232 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:00:13 smithi007 ceph-mon[39249]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' 2024-09-18T08:00:14.232 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:00:13 smithi007 ceph-mon[39249]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-18T08:00:14.232 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:00:13 smithi007 ceph-mon[39249]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-18T08:00:14.232 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:00:13 smithi007 ceph-mon[39249]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' 2024-09-18T08:00:14.232 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:00:13 smithi007 ceph-mon[39249]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2024-09-18T08:00:14.232 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:00:13 smithi007 ceph-mon[39249]: osdmap e20: 3 total, 3 up, 3 in 2024-09-18T08:00:14.233 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:00:13 smithi007 ceph-mon[39249]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' 2024-09-18T08:00:14.233 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:00:13 smithi007 ceph-mon[39249]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-09-18T08:00:14.233 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:00:13 smithi007 ceph-mon[39249]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-09-18T08:00:14.233 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:00:13 smithi007 ceph-mon[39249]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "a"} : dispatch 2024-09-18T08:00:14.233 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:00:13 smithi007 ceph-mon[39249]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-09-18T08:00:14.233 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:00:13 smithi007 ceph-mon[39249]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-09-18T08:00:14.233 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:00:13 smithi007 ceph-mon[39249]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-09-18T08:00:14.233 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:00:13 smithi007 ceph-mon[39249]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "a"} : dispatch 2024-09-18T08:00:14.233 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:00:13 smithi007 ceph-mon[39249]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-09-18T08:00:14.233 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:00:13 smithi007 ceph-mon[39249]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-09-18T08:00:14.233 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:00:13 smithi007 ceph-mon[39249]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-09-18T08:00:14.233 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:00:13 smithi007 ceph-mon[39249]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' 2024-09-18T08:00:14.233 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:00:13 smithi007 ceph-mon[39249]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-09-18T08:00:14.233 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:00:13 smithi007 ceph-mon[39249]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "a"} : dispatch 2024-09-18T08:00:14.234 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:00:13 smithi007 ceph-mon[39249]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-09-18T08:00:14.234 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:00:13 smithi007 ceph-mon[39249]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-09-18T08:00:14.234 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:00:13 smithi007 ceph-mon[39249]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-09-18T08:00:14.234 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:00:13 smithi007 ceph-mon[29459]: Detected new or changed devices on smithi007 2024-09-18T08:00:14.234 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:00:13 smithi007 ceph-mon[29459]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' 2024-09-18T08:00:14.234 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:00:13 smithi007 ceph-mon[29459]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' 2024-09-18T08:00:14.234 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:00:13 smithi007 ceph-mon[29459]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"} : dispatch 2024-09-18T08:00:14.234 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:00:13 smithi007 ceph-mon[29459]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"} : dispatch 2024-09-18T08:00:14.234 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:00:13 smithi007 ceph-mon[29459]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"} : dispatch 2024-09-18T08:00:14.234 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:00:13 smithi007 ceph-mon[29459]: Adjusting osd_memory_target on smithi007 to 5356M 2024-09-18T08:00:14.234 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:00:13 smithi007 ceph-mon[29459]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' 2024-09-18T08:00:14.235 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:00:13 smithi007 ceph-mon[29459]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-18T08:00:14.235 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:00:13 smithi007 ceph-mon[29459]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-18T08:00:14.235 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:00:13 smithi007 ceph-mon[29459]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' 2024-09-18T08:00:14.235 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:00:13 smithi007 ceph-mon[29459]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2024-09-18T08:00:14.235 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:00:13 smithi007 ceph-mon[29459]: osdmap e20: 3 total, 3 up, 3 in 2024-09-18T08:00:14.235 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:00:13 smithi007 ceph-mon[29459]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' 2024-09-18T08:00:14.235 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:00:13 smithi007 ceph-mon[29459]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-09-18T08:00:14.235 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:00:13 smithi007 ceph-mon[29459]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-09-18T08:00:14.236 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:00:13 smithi007 ceph-mon[29459]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "a"} : dispatch 2024-09-18T08:00:14.236 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:00:13 smithi007 ceph-mon[29459]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-09-18T08:00:14.236 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:00:13 smithi007 ceph-mon[29459]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-09-18T08:00:14.236 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:00:13 smithi007 ceph-mon[29459]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-09-18T08:00:14.236 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:00:13 smithi007 ceph-mon[29459]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "a"} : dispatch 2024-09-18T08:00:14.236 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:00:13 smithi007 ceph-mon[29459]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-09-18T08:00:14.236 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:00:13 smithi007 ceph-mon[29459]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-09-18T08:00:14.236 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:00:13 smithi007 ceph-mon[29459]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-09-18T08:00:14.236 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:00:13 smithi007 ceph-mon[29459]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' 2024-09-18T08:00:14.236 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:00:13 smithi007 ceph-mon[29459]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-09-18T08:00:14.236 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:00:13 smithi007 ceph-mon[29459]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "a"} : dispatch 2024-09-18T08:00:14.236 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:00:13 smithi007 ceph-mon[29459]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-09-18T08:00:14.236 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:00:13 smithi007 ceph-mon[29459]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-09-18T08:00:14.237 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:00:13 smithi007 ceph-mon[29459]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-09-18T08:00:14.542 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-09-18T08:00:14.572 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 779349ca-7593-11ef-bceb-c7b262605968 -- ceph orch daemon add osd smithi007:vg_nvme/lv_1 2024-09-18T08:00:14.835 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/779349ca-7593-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T08:00:15.182 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:00:14 smithi007 ceph-mon[29459]: pgmap v78: 1 pgs: 1 creating+peering; 0 B data, 84 MiB used, 268 GiB / 268 GiB avail 2024-09-18T08:00:15.183 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:00:14 smithi007 ceph-mon[29459]: osdmap e21: 3 total, 3 up, 3 in 2024-09-18T08:00:15.183 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:00:14 smithi007 ceph-mon[29459]: mgrmap e15: y(active, since 2m), standbys: x 2024-09-18T08:00:15.183 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:00:14 smithi007 ceph-mon[39249]: pgmap v78: 1 pgs: 1 creating+peering; 0 B data, 84 MiB used, 268 GiB / 268 GiB avail 2024-09-18T08:00:15.183 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:00:14 smithi007 ceph-mon[39249]: osdmap e21: 3 total, 3 up, 3 in 2024-09-18T08:00:15.183 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:00:14 smithi007 ceph-mon[39249]: mgrmap e15: y(active, since 2m), standbys: x 2024-09-18T08:00:15.406 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:00:14 smithi050 ceph-mon[32371]: pgmap v78: 1 pgs: 1 creating+peering; 0 B data, 84 MiB used, 268 GiB / 268 GiB avail 2024-09-18T08:00:15.406 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:00:14 smithi050 ceph-mon[32371]: osdmap e21: 3 total, 3 up, 3 in 2024-09-18T08:00:15.406 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:00:14 smithi050 ceph-mon[32371]: mgrmap e15: y(active, since 2m), standbys: x 2024-09-18T08:00:16.086 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:00:15 smithi007 ceph-mon[29459]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' cmd={"prefix": "osd tree", "states": ["destroyed"], "format": "json"} : dispatch 2024-09-18T08:00:16.086 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:00:15 smithi007 ceph-mon[29459]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.bootstrap-osd"} : dispatch 2024-09-18T08:00:16.086 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:00:15 smithi007 ceph-mon[29459]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-18T08:00:16.086 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:00:15 smithi007 ceph-mon[39249]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' cmd={"prefix": "osd tree", "states": ["destroyed"], "format": "json"} : dispatch 2024-09-18T08:00:16.086 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:00:15 smithi007 ceph-mon[39249]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.bootstrap-osd"} : dispatch 2024-09-18T08:00:16.087 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:00:15 smithi007 ceph-mon[39249]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-18T08:00:16.406 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:00:15 smithi050 ceph-mon[32371]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' cmd={"prefix": "osd tree", "states": ["destroyed"], "format": "json"} : dispatch 2024-09-18T08:00:16.406 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:00:15 smithi050 ceph-mon[32371]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.bootstrap-osd"} : dispatch 2024-09-18T08:00:16.406 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:00:15 smithi050 ceph-mon[32371]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-18T08:00:16.976 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:00:16 smithi007 ceph-mon[29459]: pgmap v80: 1 pgs: 1 creating+peering; 0 B data, 85 MiB used, 268 GiB / 268 GiB avail 2024-09-18T08:00:16.976 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:00:16 smithi007 ceph-mon[29459]: from='client.14313 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi007:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-09-18T08:00:16.976 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:00:16 smithi007 ceph-mon[39249]: pgmap v80: 1 pgs: 1 creating+peering; 0 B data, 85 MiB used, 268 GiB / 268 GiB avail 2024-09-18T08:00:16.976 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:00:16 smithi007 ceph-mon[39249]: from='client.14313 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi007:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-09-18T08:00:17.406 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:00:16 smithi050 ceph-mon[32371]: pgmap v80: 1 pgs: 1 creating+peering; 0 B data, 85 MiB used, 268 GiB / 268 GiB avail 2024-09-18T08:00:17.406 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:00:16 smithi050 ceph-mon[32371]: from='client.14313 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi007:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-09-18T08:00:18.231 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:00:17 smithi007 ceph-mon[29459]: purged_snaps scrub starts 2024-09-18T08:00:18.231 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:00:17 smithi007 ceph-mon[29459]: purged_snaps scrub ok 2024-09-18T08:00:18.231 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:00:17 smithi007 ceph-mon[29459]: osdmap e22: 3 total, 3 up, 3 in 2024-09-18T08:00:18.231 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:00:17 smithi007 ceph-mon[29459]: from='client.? 172.21.15.7:0/3344510278' entity='client.bootstrap-osd' cmd={"prefix": "osd new", "uuid": "11ecd7e1-a751-4b75-95f5-00b32d2a5bc4"} : dispatch 2024-09-18T08:00:18.231 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:00:17 smithi007 ceph-mon[29459]: from='client.? 172.21.15.7:0/3344510278' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "11ecd7e1-a751-4b75-95f5-00b32d2a5bc4"}]': finished 2024-09-18T08:00:18.231 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:00:17 smithi007 ceph-mon[29459]: osdmap e23: 4 total, 3 up, 4 in 2024-09-18T08:00:18.231 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:00:17 smithi007 ceph-mon[29459]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 3} : dispatch 2024-09-18T08:00:18.232 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:00:17 smithi007 ceph-mon[39249]: purged_snaps scrub starts 2024-09-18T08:00:18.232 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:00:17 smithi007 ceph-mon[39249]: purged_snaps scrub ok 2024-09-18T08:00:18.232 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:00:17 smithi007 ceph-mon[39249]: osdmap e22: 3 total, 3 up, 3 in 2024-09-18T08:00:18.232 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:00:17 smithi007 ceph-mon[39249]: from='client.? 172.21.15.7:0/3344510278' entity='client.bootstrap-osd' cmd={"prefix": "osd new", "uuid": "11ecd7e1-a751-4b75-95f5-00b32d2a5bc4"} : dispatch 2024-09-18T08:00:18.232 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:00:17 smithi007 ceph-mon[39249]: from='client.? 172.21.15.7:0/3344510278' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "11ecd7e1-a751-4b75-95f5-00b32d2a5bc4"}]': finished 2024-09-18T08:00:18.232 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:00:17 smithi007 ceph-mon[39249]: osdmap e23: 4 total, 3 up, 4 in 2024-09-18T08:00:18.232 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:00:17 smithi007 ceph-mon[39249]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 3} : dispatch 2024-09-18T08:00:18.406 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:00:17 smithi050 ceph-mon[32371]: purged_snaps scrub starts 2024-09-18T08:00:18.406 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:00:17 smithi050 ceph-mon[32371]: purged_snaps scrub ok 2024-09-18T08:00:18.406 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:00:17 smithi050 ceph-mon[32371]: osdmap e22: 3 total, 3 up, 3 in 2024-09-18T08:00:18.406 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:00:17 smithi050 ceph-mon[32371]: from='client.? 172.21.15.7:0/3344510278' entity='client.bootstrap-osd' cmd={"prefix": "osd new", "uuid": "11ecd7e1-a751-4b75-95f5-00b32d2a5bc4"} : dispatch 2024-09-18T08:00:18.406 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:00:17 smithi050 ceph-mon[32371]: from='client.? 172.21.15.7:0/3344510278' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "11ecd7e1-a751-4b75-95f5-00b32d2a5bc4"}]': finished 2024-09-18T08:00:18.406 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:00:17 smithi050 ceph-mon[32371]: osdmap e23: 4 total, 3 up, 4 in 2024-09-18T08:00:18.407 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:00:17 smithi050 ceph-mon[32371]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 3} : dispatch 2024-09-18T08:00:19.231 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:00:18 smithi007 ceph-mon[29459]: pgmap v82: 1 pgs: 1 creating+peering; 0 B data, 85 MiB used, 268 GiB / 268 GiB avail 2024-09-18T08:00:19.231 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:00:18 smithi007 ceph-mon[29459]: from='client.? 172.21.15.7:0/404125552' entity='client.bootstrap-osd' cmd={"prefix": "mon getmap"} : dispatch 2024-09-18T08:00:19.231 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:00:18 smithi007 ceph-mon[39249]: pgmap v82: 1 pgs: 1 creating+peering; 0 B data, 85 MiB used, 268 GiB / 268 GiB avail 2024-09-18T08:00:19.231 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:00:18 smithi007 ceph-mon[39249]: from='client.? 172.21.15.7:0/404125552' entity='client.bootstrap-osd' cmd={"prefix": "mon getmap"} : dispatch 2024-09-18T08:00:19.406 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:00:18 smithi050 ceph-mon[32371]: pgmap v82: 1 pgs: 1 creating+peering; 0 B data, 85 MiB used, 268 GiB / 268 GiB avail 2024-09-18T08:00:19.406 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:00:18 smithi050 ceph-mon[32371]: from='client.? 172.21.15.7:0/404125552' entity='client.bootstrap-osd' cmd={"prefix": "mon getmap"} : dispatch 2024-09-18T08:00:21.406 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:00:21 smithi050 ceph-mon[32371]: pgmap v84: 1 pgs: 1 active+clean; 577 KiB data, 86 MiB used, 268 GiB / 268 GiB avail 2024-09-18T08:00:21.481 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:00:21 smithi007 ceph-mon[29459]: pgmap v84: 1 pgs: 1 active+clean; 577 KiB data, 86 MiB used, 268 GiB / 268 GiB avail 2024-09-18T08:00:21.481 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:00:21 smithi007 ceph-mon[39249]: pgmap v84: 1 pgs: 1 active+clean; 577 KiB data, 86 MiB used, 268 GiB / 268 GiB avail 2024-09-18T08:00:23.231 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:00:23 smithi007 ceph-mon[29459]: pgmap v85: 1 pgs: 1 active+clean; 577 KiB data, 86 MiB used, 268 GiB / 268 GiB avail 2024-09-18T08:00:23.231 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:00:23 smithi007 ceph-mon[39249]: pgmap v85: 1 pgs: 1 active+clean; 577 KiB data, 86 MiB used, 268 GiB / 268 GiB avail 2024-09-18T08:00:23.406 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:00:23 smithi050 ceph-mon[32371]: pgmap v85: 1 pgs: 1 active+clean; 577 KiB data, 86 MiB used, 268 GiB / 268 GiB avail 2024-09-18T08:00:25.406 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:00:25 smithi050 ceph-mon[32371]: pgmap v86: 1 pgs: 1 active+clean; 577 KiB data, 86 MiB used, 268 GiB / 268 GiB avail 2024-09-18T08:00:25.480 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:00:25 smithi007 ceph-mon[29459]: pgmap v86: 1 pgs: 1 active+clean; 577 KiB data, 86 MiB used, 268 GiB / 268 GiB avail 2024-09-18T08:00:25.481 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:00:25 smithi007 ceph-mon[39249]: pgmap v86: 1 pgs: 1 active+clean; 577 KiB data, 86 MiB used, 268 GiB / 268 GiB avail 2024-09-18T08:00:26.231 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:00:26 smithi007 ceph-mon[29459]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' cmd={"prefix": "auth get", "entity": "osd.3"} : dispatch 2024-09-18T08:00:26.231 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:00:26 smithi007 ceph-mon[29459]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-18T08:00:26.231 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:00:26 smithi007 ceph-mon[39249]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' cmd={"prefix": "auth get", "entity": "osd.3"} : dispatch 2024-09-18T08:00:26.231 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:00:26 smithi007 ceph-mon[39249]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-18T08:00:26.406 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:00:26 smithi050 ceph-mon[32371]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' cmd={"prefix": "auth get", "entity": "osd.3"} : dispatch 2024-09-18T08:00:26.406 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:00:26 smithi050 ceph-mon[32371]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-18T08:00:27.150 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:00:27 smithi007 ceph-mon[29459]: Deploying daemon osd.3 on smithi007 2024-09-18T08:00:27.150 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:00:27 smithi007 ceph-mon[29459]: pgmap v87: 1 pgs: 1 active+clean; 577 KiB data, 86 MiB used, 268 GiB / 268 GiB avail 2024-09-18T08:00:27.150 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:00:27 smithi007 ceph-mon[39249]: Deploying daemon osd.3 on smithi007 2024-09-18T08:00:27.151 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:00:27 smithi007 ceph-mon[39249]: pgmap v87: 1 pgs: 1 active+clean; 577 KiB data, 86 MiB used, 268 GiB / 268 GiB avail 2024-09-18T08:00:27.406 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:00:27 smithi050 ceph-mon[32371]: Deploying daemon osd.3 on smithi007 2024-09-18T08:00:27.406 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:00:27 smithi050 ceph-mon[32371]: pgmap v87: 1 pgs: 1 active+clean; 577 KiB data, 86 MiB used, 268 GiB / 268 GiB avail 2024-09-18T08:00:29.231 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:00:29 smithi007 ceph-mon[29459]: pgmap v88: 1 pgs: 1 active+clean; 577 KiB data, 86 MiB used, 268 GiB / 268 GiB avail 2024-09-18T08:00:29.231 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:00:29 smithi007 ceph-mon[39249]: pgmap v88: 1 pgs: 1 active+clean; 577 KiB data, 86 MiB used, 268 GiB / 268 GiB avail 2024-09-18T08:00:29.410 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:00:29 smithi050 ceph-mon[32371]: pgmap v88: 1 pgs: 1 active+clean; 577 KiB data, 86 MiB used, 268 GiB / 268 GiB avail 2024-09-18T08:00:31.334 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:00:31 smithi007 ceph-mon[29459]: pgmap v89: 1 pgs: 1 active+clean; 577 KiB data, 86 MiB used, 268 GiB / 268 GiB avail 2024-09-18T08:00:31.334 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:00:31 smithi007 ceph-mon[39249]: pgmap v89: 1 pgs: 1 active+clean; 577 KiB data, 86 MiB used, 268 GiB / 268 GiB avail 2024-09-18T08:00:31.406 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:00:31 smithi050 ceph-mon[32371]: pgmap v89: 1 pgs: 1 active+clean; 577 KiB data, 86 MiB used, 268 GiB / 268 GiB avail 2024-09-18T08:00:33.231 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:00:33 smithi007 ceph-mon[29459]: pgmap v90: 1 pgs: 1 active+clean; 577 KiB data, 86 MiB used, 268 GiB / 268 GiB avail 2024-09-18T08:00:33.231 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:00:33 smithi007 ceph-mon[29459]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-18T08:00:33.231 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:00:33 smithi007 ceph-mon[29459]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' 2024-09-18T08:00:33.231 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:00:33 smithi007 ceph-mon[29459]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' 2024-09-18T08:00:33.231 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:00:33 smithi007 ceph-mon[39249]: pgmap v90: 1 pgs: 1 active+clean; 577 KiB data, 86 MiB used, 268 GiB / 268 GiB avail 2024-09-18T08:00:33.231 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:00:33 smithi007 ceph-mon[39249]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-18T08:00:33.231 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:00:33 smithi007 ceph-mon[39249]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' 2024-09-18T08:00:33.231 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:00:33 smithi007 ceph-mon[39249]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' 2024-09-18T08:00:33.406 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:00:33 smithi050 ceph-mon[32371]: pgmap v90: 1 pgs: 1 active+clean; 577 KiB data, 86 MiB used, 268 GiB / 268 GiB avail 2024-09-18T08:00:33.406 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:00:33 smithi050 ceph-mon[32371]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-18T08:00:33.406 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:00:33 smithi050 ceph-mon[32371]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' 2024-09-18T08:00:33.406 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:00:33 smithi050 ceph-mon[32371]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' 2024-09-18T08:00:35.143 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:00:35 smithi007 ceph-mon[29459]: pgmap v91: 1 pgs: 1 active+clean; 577 KiB data, 86 MiB used, 268 GiB / 268 GiB avail 2024-09-18T08:00:35.143 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:00:35 smithi007 ceph-mon[39249]: pgmap v91: 1 pgs: 1 active+clean; 577 KiB data, 86 MiB used, 268 GiB / 268 GiB avail 2024-09-18T08:00:35.406 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:00:35 smithi050 ceph-mon[32371]: pgmap v91: 1 pgs: 1 active+clean; 577 KiB data, 86 MiB used, 268 GiB / 268 GiB avail 2024-09-18T08:00:36.656 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:00:36 smithi050 ceph-mon[32371]: pgmap v92: 1 pgs: 1 active+clean; 577 KiB data, 86 MiB used, 268 GiB / 268 GiB avail 2024-09-18T08:00:36.656 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:00:36 smithi050 ceph-mon[32371]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' 2024-09-18T08:00:36.656 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:00:36 smithi050 ceph-mon[32371]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' 2024-09-18T08:00:36.657 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:00:36 smithi050 ceph-mon[32371]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-18T08:00:36.657 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:00:36 smithi050 ceph-mon[32371]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-18T08:00:36.657 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:00:36 smithi050 ceph-mon[32371]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' 2024-09-18T08:00:36.731 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:00:36 smithi007 ceph-mon[29459]: pgmap v92: 1 pgs: 1 active+clean; 577 KiB data, 86 MiB used, 268 GiB / 268 GiB avail 2024-09-18T08:00:36.731 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:00:36 smithi007 ceph-mon[29459]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' 2024-09-18T08:00:36.731 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:00:36 smithi007 ceph-mon[29459]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' 2024-09-18T08:00:36.731 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:00:36 smithi007 ceph-mon[29459]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-18T08:00:36.731 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:00:36 smithi007 ceph-mon[29459]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-18T08:00:36.731 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:00:36 smithi007 ceph-mon[29459]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' 2024-09-18T08:00:36.731 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:00:36 smithi007 ceph-mon[39249]: pgmap v92: 1 pgs: 1 active+clean; 577 KiB data, 86 MiB used, 268 GiB / 268 GiB avail 2024-09-18T08:00:36.732 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:00:36 smithi007 ceph-mon[39249]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' 2024-09-18T08:00:36.732 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:00:36 smithi007 ceph-mon[39249]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' 2024-09-18T08:00:36.732 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:00:36 smithi007 ceph-mon[39249]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-18T08:00:36.732 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:00:36 smithi007 ceph-mon[39249]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-18T08:00:36.732 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:00:36 smithi007 ceph-mon[39249]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' 2024-09-18T08:00:38.257 INFO:teuthology.orchestra.run.smithi007.stdout:Created osd(s) 3 on host 'smithi007' 2024-09-18T08:00:38.481 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:00:38 smithi007 ceph-mon[29459]: pgmap v93: 1 pgs: 1 active+clean; 577 KiB data, 86 MiB used, 268 GiB / 268 GiB avail 2024-09-18T08:00:38.481 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:00:38 smithi007 ceph-mon[29459]: from='osd.3 v2:172.21.15.7:6813/163587772' entity='osd.3' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]} : dispatch 2024-09-18T08:00:38.481 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:00:38 smithi007 ceph-mon[29459]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-18T08:00:38.481 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:00:38 smithi007 ceph-mon[29459]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' 2024-09-18T08:00:38.481 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:00:38 smithi007 ceph-mon[29459]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' 2024-09-18T08:00:38.481 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:00:38 smithi007 ceph-mon[39249]: pgmap v93: 1 pgs: 1 active+clean; 577 KiB data, 86 MiB used, 268 GiB / 268 GiB avail 2024-09-18T08:00:38.482 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:00:38 smithi007 ceph-mon[39249]: from='osd.3 v2:172.21.15.7:6813/163587772' entity='osd.3' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]} : dispatch 2024-09-18T08:00:38.482 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:00:38 smithi007 ceph-mon[39249]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-18T08:00:38.482 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:00:38 smithi007 ceph-mon[39249]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' 2024-09-18T08:00:38.482 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:00:38 smithi007 ceph-mon[39249]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' 2024-09-18T08:00:38.656 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:00:38 smithi050 ceph-mon[32371]: pgmap v93: 1 pgs: 1 active+clean; 577 KiB data, 86 MiB used, 268 GiB / 268 GiB avail 2024-09-18T08:00:38.656 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:00:38 smithi050 ceph-mon[32371]: from='osd.3 v2:172.21.15.7:6813/163587772' entity='osd.3' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]} : dispatch 2024-09-18T08:00:38.656 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:00:38 smithi050 ceph-mon[32371]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-18T08:00:38.656 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:00:38 smithi050 ceph-mon[32371]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' 2024-09-18T08:00:38.656 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:00:38 smithi050 ceph-mon[32371]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' 2024-09-18T08:00:38.860 DEBUG:teuthology.orchestra.run.smithi007:osd.3> sudo journalctl -f -n 0 -u ceph-779349ca-7593-11ef-bceb-c7b262605968@osd.3.service 2024-09-18T08:00:38.862 INFO:tasks.cephadm:Deploying osd.4 on smithi050 with /dev/vg_nvme/lv_4... 2024-09-18T08:00:38.862 DEBUG:teuthology.orchestra.run.smithi050:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e ceph-volume -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 779349ca-7593-11ef-bceb-c7b262605968 -- lvm zap /dev/vg_nvme/lv_4 2024-09-18T08:00:39.116 INFO:teuthology.orchestra.run.smithi050.stderr:Inferring config /var/lib/ceph/779349ca-7593-11ef-bceb-c7b262605968/mon.b/config 2024-09-18T08:00:39.490 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:00:39 smithi050 ceph-mon[32371]: from='osd.3 v2:172.21.15.7:6813/163587772' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-09-18T08:00:39.490 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:00:39 smithi050 ceph-mon[32371]: osdmap e24: 4 total, 3 up, 4 in 2024-09-18T08:00:39.491 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:00:39 smithi050 ceph-mon[32371]: from='osd.3 v2:172.21.15.7:6813/163587772' entity='osd.3' cmd={"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi007", "root=default"]} : dispatch 2024-09-18T08:00:39.491 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:00:39 smithi050 ceph-mon[32371]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 3} : dispatch 2024-09-18T08:00:39.691 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:00:39 smithi007 ceph-mon[29459]: from='osd.3 v2:172.21.15.7:6813/163587772' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-09-18T08:00:39.692 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:00:39 smithi007 ceph-mon[29459]: osdmap e24: 4 total, 3 up, 4 in 2024-09-18T08:00:39.692 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:00:39 smithi007 ceph-mon[29459]: from='osd.3 v2:172.21.15.7:6813/163587772' entity='osd.3' cmd={"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi007", "root=default"]} : dispatch 2024-09-18T08:00:39.692 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:00:39 smithi007 ceph-mon[29459]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 3} : dispatch 2024-09-18T08:00:39.692 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:00:39 smithi007 ceph-mon[39249]: from='osd.3 v2:172.21.15.7:6813/163587772' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-09-18T08:00:39.692 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:00:39 smithi007 ceph-mon[39249]: osdmap e24: 4 total, 3 up, 4 in 2024-09-18T08:00:39.692 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:00:39 smithi007 ceph-mon[39249]: from='osd.3 v2:172.21.15.7:6813/163587772' entity='osd.3' cmd={"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi007", "root=default"]} : dispatch 2024-09-18T08:00:39.692 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:00:39 smithi007 ceph-mon[39249]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 3} : dispatch 2024-09-18T08:00:39.693 INFO:journalctl@ceph.osd.3.smithi007.stdout:Sep 18 08:00:39 smithi007 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd-3[58803]: 2024-09-18T08:00:39.393+0000 7f45b158f640 -1 osd.3 0 waiting for initial osdmap 2024-09-18T08:00:39.693 INFO:journalctl@ceph.osd.3.smithi007.stdout:Sep 18 08:00:39 smithi007 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd-3[58803]: 2024-09-18T08:00:39.396+0000 7f45ad3b6640 -1 osd.3 25 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-18T08:00:40.587 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:00:40 smithi050 ceph-mon[32371]: pgmap v95: 1 pgs: 1 active+clean; 577 KiB data, 86 MiB used, 268 GiB / 268 GiB avail 2024-09-18T08:00:40.588 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:00:40 smithi050 ceph-mon[32371]: from='osd.3 v2:172.21.15.7:6813/163587772' entity='osd.3' cmd='[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi007", "root=default"]}]': finished 2024-09-18T08:00:40.588 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:00:40 smithi050 ceph-mon[32371]: osdmap e25: 4 total, 3 up, 4 in 2024-09-18T08:00:40.588 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:00:40 smithi050 ceph-mon[32371]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 3} : dispatch 2024-09-18T08:00:40.588 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:00:40 smithi050 ceph-mon[32371]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 3} : dispatch 2024-09-18T08:00:40.731 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:00:40 smithi007 ceph-mon[29459]: pgmap v95: 1 pgs: 1 active+clean; 577 KiB data, 86 MiB used, 268 GiB / 268 GiB avail 2024-09-18T08:00:40.731 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:00:40 smithi007 ceph-mon[29459]: from='osd.3 v2:172.21.15.7:6813/163587772' entity='osd.3' cmd='[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi007", "root=default"]}]': finished 2024-09-18T08:00:40.731 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:00:40 smithi007 ceph-mon[29459]: osdmap e25: 4 total, 3 up, 4 in 2024-09-18T08:00:40.731 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:00:40 smithi007 ceph-mon[29459]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 3} : dispatch 2024-09-18T08:00:40.731 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:00:40 smithi007 ceph-mon[29459]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 3} : dispatch 2024-09-18T08:00:40.732 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:00:40 smithi007 ceph-mon[39249]: pgmap v95: 1 pgs: 1 active+clean; 577 KiB data, 86 MiB used, 268 GiB / 268 GiB avail 2024-09-18T08:00:40.732 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:00:40 smithi007 ceph-mon[39249]: from='osd.3 v2:172.21.15.7:6813/163587772' entity='osd.3' cmd='[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi007", "root=default"]}]': finished 2024-09-18T08:00:40.732 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:00:40 smithi007 ceph-mon[39249]: osdmap e25: 4 total, 3 up, 4 in 2024-09-18T08:00:40.732 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:00:40 smithi007 ceph-mon[39249]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 3} : dispatch 2024-09-18T08:00:40.732 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:00:40 smithi007 ceph-mon[39249]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 3} : dispatch 2024-09-18T08:00:41.616 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:00:41 smithi007 ceph-mon[29459]: purged_snaps scrub starts 2024-09-18T08:00:41.617 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:00:41 smithi007 ceph-mon[29459]: purged_snaps scrub ok 2024-09-18T08:00:41.617 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:00:41 smithi007 ceph-mon[29459]: osd.3 v2:172.21.15.7:6813/163587772 boot 2024-09-18T08:00:41.617 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:00:41 smithi007 ceph-mon[29459]: osdmap e26: 4 total, 4 up, 4 in 2024-09-18T08:00:41.617 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:00:41 smithi007 ceph-mon[29459]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 3} : dispatch 2024-09-18T08:00:41.617 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:00:41 smithi007 ceph-mon[39249]: purged_snaps scrub starts 2024-09-18T08:00:41.617 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:00:41 smithi007 ceph-mon[39249]: purged_snaps scrub ok 2024-09-18T08:00:41.617 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:00:41 smithi007 ceph-mon[39249]: osd.3 v2:172.21.15.7:6813/163587772 boot 2024-09-18T08:00:41.617 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:00:41 smithi007 ceph-mon[39249]: osdmap e26: 4 total, 4 up, 4 in 2024-09-18T08:00:41.617 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:00:41 smithi007 ceph-mon[39249]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 3} : dispatch 2024-09-18T08:00:41.648 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:00:41 smithi050 ceph-mon[32371]: purged_snaps scrub starts 2024-09-18T08:00:41.648 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:00:41 smithi050 ceph-mon[32371]: purged_snaps scrub ok 2024-09-18T08:00:41.648 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:00:41 smithi050 ceph-mon[32371]: osd.3 v2:172.21.15.7:6813/163587772 boot 2024-09-18T08:00:41.648 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:00:41 smithi050 ceph-mon[32371]: osdmap e26: 4 total, 4 up, 4 in 2024-09-18T08:00:41.648 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:00:41 smithi050 ceph-mon[32371]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 3} : dispatch 2024-09-18T08:00:42.054 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-09-18T08:00:42.080 DEBUG:teuthology.orchestra.run.smithi050:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 779349ca-7593-11ef-bceb-c7b262605968 -- ceph orch daemon add osd smithi050:vg_nvme/lv_4 2024-09-18T08:00:42.334 INFO:teuthology.orchestra.run.smithi050.stderr:Inferring config /var/lib/ceph/779349ca-7593-11ef-bceb-c7b262605968/mon.b/config 2024-09-18T08:00:42.656 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:00:42 smithi050 ceph-mon[32371]: pgmap v98: 1 pgs: 1 active+clean; 577 KiB data, 86 MiB used, 268 GiB / 268 GiB avail 2024-09-18T08:00:42.657 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:00:42 smithi050 ceph-mon[32371]: osdmap e27: 4 total, 4 up, 4 in 2024-09-18T08:00:42.657 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:00:42 smithi050 ceph-mon[32371]: Detected new or changed devices on smithi007 2024-09-18T08:00:42.657 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:00:42 smithi050 ceph-mon[32371]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' 2024-09-18T08:00:42.657 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:00:42 smithi050 ceph-mon[32371]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' 2024-09-18T08:00:42.657 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:00:42 smithi050 ceph-mon[32371]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"} : dispatch 2024-09-18T08:00:42.657 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:00:42 smithi050 ceph-mon[32371]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"} : dispatch 2024-09-18T08:00:42.657 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:00:42 smithi050 ceph-mon[32371]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"} : dispatch 2024-09-18T08:00:42.657 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:00:42 smithi050 ceph-mon[32371]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"} : dispatch 2024-09-18T08:00:42.657 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:00:42 smithi050 ceph-mon[32371]: Adjusting osd_memory_target on smithi007 to 4017M 2024-09-18T08:00:42.657 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:00:42 smithi050 ceph-mon[32371]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' 2024-09-18T08:00:42.657 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:00:42 smithi050 ceph-mon[32371]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-18T08:00:42.657 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:00:42 smithi050 ceph-mon[32371]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-18T08:00:42.657 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:00:42 smithi050 ceph-mon[32371]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' 2024-09-18T08:00:42.731 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:00:42 smithi007 ceph-mon[29459]: pgmap v98: 1 pgs: 1 active+clean; 577 KiB data, 86 MiB used, 268 GiB / 268 GiB avail 2024-09-18T08:00:42.731 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:00:42 smithi007 ceph-mon[29459]: osdmap e27: 4 total, 4 up, 4 in 2024-09-18T08:00:42.731 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:00:42 smithi007 ceph-mon[29459]: Detected new or changed devices on smithi007 2024-09-18T08:00:42.731 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:00:42 smithi007 ceph-mon[29459]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' 2024-09-18T08:00:42.731 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:00:42 smithi007 ceph-mon[29459]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' 2024-09-18T08:00:42.731 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:00:42 smithi007 ceph-mon[29459]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"} : dispatch 2024-09-18T08:00:42.731 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:00:42 smithi007 ceph-mon[29459]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"} : dispatch 2024-09-18T08:00:42.732 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:00:42 smithi007 ceph-mon[29459]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"} : dispatch 2024-09-18T08:00:42.732 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:00:42 smithi007 ceph-mon[29459]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"} : dispatch 2024-09-18T08:00:42.732 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:00:42 smithi007 ceph-mon[29459]: Adjusting osd_memory_target on smithi007 to 4017M 2024-09-18T08:00:42.732 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:00:42 smithi007 ceph-mon[29459]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' 2024-09-18T08:00:42.732 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:00:42 smithi007 ceph-mon[29459]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-18T08:00:42.732 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:00:42 smithi007 ceph-mon[29459]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-18T08:00:42.732 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:00:42 smithi007 ceph-mon[29459]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' 2024-09-18T08:00:42.732 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:00:42 smithi007 ceph-mon[39249]: pgmap v98: 1 pgs: 1 active+clean; 577 KiB data, 86 MiB used, 268 GiB / 268 GiB avail 2024-09-18T08:00:42.732 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:00:42 smithi007 ceph-mon[39249]: osdmap e27: 4 total, 4 up, 4 in 2024-09-18T08:00:42.732 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:00:42 smithi007 ceph-mon[39249]: Detected new or changed devices on smithi007 2024-09-18T08:00:42.732 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:00:42 smithi007 ceph-mon[39249]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' 2024-09-18T08:00:42.733 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:00:42 smithi007 ceph-mon[39249]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' 2024-09-18T08:00:42.733 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:00:42 smithi007 ceph-mon[39249]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"} : dispatch 2024-09-18T08:00:42.733 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:00:42 smithi007 ceph-mon[39249]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"} : dispatch 2024-09-18T08:00:42.733 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:00:42 smithi007 ceph-mon[39249]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"} : dispatch 2024-09-18T08:00:42.733 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:00:42 smithi007 ceph-mon[39249]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"} : dispatch 2024-09-18T08:00:42.733 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:00:42 smithi007 ceph-mon[39249]: Adjusting osd_memory_target on smithi007 to 4017M 2024-09-18T08:00:42.733 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:00:42 smithi007 ceph-mon[39249]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' 2024-09-18T08:00:42.733 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:00:42 smithi007 ceph-mon[39249]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-18T08:00:42.733 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:00:42 smithi007 ceph-mon[39249]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-18T08:00:42.733 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:00:42 smithi007 ceph-mon[39249]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' 2024-09-18T08:00:43.656 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:00:43 smithi050 ceph-mon[32371]: from='client.24196 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi050:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-09-18T08:00:43.656 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:00:43 smithi050 ceph-mon[32371]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' cmd={"prefix": "osd tree", "states": ["destroyed"], "format": "json"} : dispatch 2024-09-18T08:00:43.656 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:00:43 smithi050 ceph-mon[32371]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.bootstrap-osd"} : dispatch 2024-09-18T08:00:43.656 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:00:43 smithi050 ceph-mon[32371]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-18T08:00:43.731 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:00:43 smithi007 ceph-mon[39249]: from='client.24196 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi050:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-09-18T08:00:43.731 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:00:43 smithi007 ceph-mon[39249]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' cmd={"prefix": "osd tree", "states": ["destroyed"], "format": "json"} : dispatch 2024-09-18T08:00:43.731 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:00:43 smithi007 ceph-mon[39249]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.bootstrap-osd"} : dispatch 2024-09-18T08:00:43.731 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:00:43 smithi007 ceph-mon[39249]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-18T08:00:43.731 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:00:43 smithi007 ceph-mon[29459]: from='client.24196 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi050:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-09-18T08:00:43.731 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:00:43 smithi007 ceph-mon[29459]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' cmd={"prefix": "osd tree", "states": ["destroyed"], "format": "json"} : dispatch 2024-09-18T08:00:43.731 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:00:43 smithi007 ceph-mon[29459]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.bootstrap-osd"} : dispatch 2024-09-18T08:00:43.732 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:00:43 smithi007 ceph-mon[29459]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-18T08:00:45.156 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:00:45 smithi050 ceph-mon[32371]: pgmap v100: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail 2024-09-18T08:00:45.481 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:00:45 smithi007 ceph-mon[29459]: pgmap v100: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail 2024-09-18T08:00:45.481 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:00:45 smithi007 ceph-mon[39249]: pgmap v100: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail 2024-09-18T08:00:46.156 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:00:46 smithi050 ceph-mon[32371]: from='client.? 172.21.15.50:0/2687699540' entity='client.bootstrap-osd' cmd={"prefix": "osd new", "uuid": "98992b34-cfa1-45bf-97a1-7d557739252f"} : dispatch 2024-09-18T08:00:46.156 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:00:46 smithi050 ceph-mon[32371]: from='client.? ' entity='client.bootstrap-osd' cmd={"prefix": "osd new", "uuid": "98992b34-cfa1-45bf-97a1-7d557739252f"} : dispatch 2024-09-18T08:00:46.156 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:00:46 smithi050 ceph-mon[32371]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "98992b34-cfa1-45bf-97a1-7d557739252f"}]': finished 2024-09-18T08:00:46.156 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:00:46 smithi050 ceph-mon[32371]: osdmap e28: 5 total, 4 up, 5 in 2024-09-18T08:00:46.157 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:00:46 smithi050 ceph-mon[32371]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 4} : dispatch 2024-09-18T08:00:46.481 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:00:46 smithi007 ceph-mon[29459]: from='client.? 172.21.15.50:0/2687699540' entity='client.bootstrap-osd' cmd={"prefix": "osd new", "uuid": "98992b34-cfa1-45bf-97a1-7d557739252f"} : dispatch 2024-09-18T08:00:46.481 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:00:46 smithi007 ceph-mon[29459]: from='client.? ' entity='client.bootstrap-osd' cmd={"prefix": "osd new", "uuid": "98992b34-cfa1-45bf-97a1-7d557739252f"} : dispatch 2024-09-18T08:00:46.481 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:00:46 smithi007 ceph-mon[29459]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "98992b34-cfa1-45bf-97a1-7d557739252f"}]': finished 2024-09-18T08:00:46.481 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:00:46 smithi007 ceph-mon[29459]: osdmap e28: 5 total, 4 up, 5 in 2024-09-18T08:00:46.481 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:00:46 smithi007 ceph-mon[29459]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 4} : dispatch 2024-09-18T08:00:46.481 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:00:46 smithi007 ceph-mon[39249]: from='client.? 172.21.15.50:0/2687699540' entity='client.bootstrap-osd' cmd={"prefix": "osd new", "uuid": "98992b34-cfa1-45bf-97a1-7d557739252f"} : dispatch 2024-09-18T08:00:46.481 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:00:46 smithi007 ceph-mon[39249]: from='client.? ' entity='client.bootstrap-osd' cmd={"prefix": "osd new", "uuid": "98992b34-cfa1-45bf-97a1-7d557739252f"} : dispatch 2024-09-18T08:00:46.481 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:00:46 smithi007 ceph-mon[39249]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "98992b34-cfa1-45bf-97a1-7d557739252f"}]': finished 2024-09-18T08:00:46.481 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:00:46 smithi007 ceph-mon[39249]: osdmap e28: 5 total, 4 up, 5 in 2024-09-18T08:00:46.482 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:00:46 smithi007 ceph-mon[39249]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 4} : dispatch 2024-09-18T08:00:47.406 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:00:47 smithi050 ceph-mon[32371]: pgmap v101: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail 2024-09-18T08:00:47.406 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:00:47 smithi050 ceph-mon[32371]: from='client.? 172.21.15.50:0/40298090' entity='client.bootstrap-osd' cmd={"prefix": "mon getmap"} : dispatch 2024-09-18T08:00:47.481 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:00:47 smithi007 ceph-mon[29459]: pgmap v101: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail 2024-09-18T08:00:47.481 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:00:47 smithi007 ceph-mon[29459]: from='client.? 172.21.15.50:0/40298090' entity='client.bootstrap-osd' cmd={"prefix": "mon getmap"} : dispatch 2024-09-18T08:00:47.481 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:00:47 smithi007 ceph-mon[39249]: pgmap v101: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail 2024-09-18T08:00:47.481 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:00:47 smithi007 ceph-mon[39249]: from='client.? 172.21.15.50:0/40298090' entity='client.bootstrap-osd' cmd={"prefix": "mon getmap"} : dispatch 2024-09-18T08:00:49.360 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:00:49 smithi050 ceph-mon[32371]: pgmap v103: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail 2024-09-18T08:00:49.481 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:00:49 smithi007 ceph-mon[29459]: pgmap v103: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail 2024-09-18T08:00:49.481 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:00:49 smithi007 ceph-mon[39249]: pgmap v103: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail 2024-09-18T08:00:50.346 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:00:50 smithi050 ceph-mon[32371]: purged_snaps scrub starts 2024-09-18T08:00:50.346 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:00:50 smithi050 ceph-mon[32371]: purged_snaps scrub ok 2024-09-18T08:00:50.481 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:00:50 smithi007 ceph-mon[29459]: purged_snaps scrub starts 2024-09-18T08:00:50.481 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:00:50 smithi007 ceph-mon[29459]: purged_snaps scrub ok 2024-09-18T08:00:50.481 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:00:50 smithi007 ceph-mon[39249]: purged_snaps scrub starts 2024-09-18T08:00:50.481 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:00:50 smithi007 ceph-mon[39249]: purged_snaps scrub ok 2024-09-18T08:00:51.406 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:00:51 smithi050 ceph-mon[32371]: pgmap v104: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail 2024-09-18T08:00:51.406 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:00:51 smithi050 ceph-mon[32371]: osdmap e29: 5 total, 4 up, 5 in 2024-09-18T08:00:51.406 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:00:51 smithi050 ceph-mon[32371]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 4} : dispatch 2024-09-18T08:00:51.481 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:00:51 smithi007 ceph-mon[29459]: pgmap v104: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail 2024-09-18T08:00:51.481 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:00:51 smithi007 ceph-mon[29459]: osdmap e29: 5 total, 4 up, 5 in 2024-09-18T08:00:51.481 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:00:51 smithi007 ceph-mon[29459]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 4} : dispatch 2024-09-18T08:00:51.482 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:00:51 smithi007 ceph-mon[39249]: pgmap v104: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail 2024-09-18T08:00:51.482 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:00:51 smithi007 ceph-mon[39249]: osdmap e29: 5 total, 4 up, 5 in 2024-09-18T08:00:51.482 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:00:51 smithi007 ceph-mon[39249]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 4} : dispatch 2024-09-18T08:00:53.406 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:00:53 smithi050 ceph-mon[32371]: pgmap v106: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail 2024-09-18T08:00:53.406 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:00:53 smithi050 ceph-mon[32371]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' cmd={"prefix": "auth get", "entity": "osd.4"} : dispatch 2024-09-18T08:00:53.406 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:00:53 smithi050 ceph-mon[32371]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-18T08:00:53.481 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:00:53 smithi007 ceph-mon[29459]: pgmap v106: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail 2024-09-18T08:00:53.481 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:00:53 smithi007 ceph-mon[29459]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' cmd={"prefix": "auth get", "entity": "osd.4"} : dispatch 2024-09-18T08:00:53.481 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:00:53 smithi007 ceph-mon[29459]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-18T08:00:53.481 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:00:53 smithi007 ceph-mon[39249]: pgmap v106: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail 2024-09-18T08:00:53.481 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:00:53 smithi007 ceph-mon[39249]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' cmd={"prefix": "auth get", "entity": "osd.4"} : dispatch 2024-09-18T08:00:53.481 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:00:53 smithi007 ceph-mon[39249]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-18T08:00:54.406 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:00:54 smithi050 ceph-mon[32371]: Deploying daemon osd.4 on smithi050 2024-09-18T08:00:54.481 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:00:54 smithi007 ceph-mon[29459]: Deploying daemon osd.4 on smithi050 2024-09-18T08:00:54.481 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:00:54 smithi007 ceph-mon[39249]: Deploying daemon osd.4 on smithi050 2024-09-18T08:00:55.406 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:00:55 smithi050 ceph-mon[32371]: pgmap v107: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail 2024-09-18T08:00:55.481 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:00:55 smithi007 ceph-mon[29459]: pgmap v107: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail 2024-09-18T08:00:55.481 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:00:55 smithi007 ceph-mon[39249]: pgmap v107: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail 2024-09-18T08:00:57.157 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:00:57 smithi050 ceph-mon[32371]: pgmap v108: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail 2024-09-18T08:00:57.297 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:00:57 smithi007 ceph-mon[29459]: pgmap v108: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail 2024-09-18T08:00:57.298 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:00:57 smithi007 ceph-mon[39249]: pgmap v108: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail 2024-09-18T08:00:59.406 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:00:59 smithi050 ceph-mon[32371]: pgmap v109: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail 2024-09-18T08:00:59.481 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:00:59 smithi007 ceph-mon[39249]: pgmap v109: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail 2024-09-18T08:00:59.481 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:00:59 smithi007 ceph-mon[29459]: pgmap v109: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail 2024-09-18T08:01:01.298 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:01:01 smithi050 ceph-mon[32371]: pgmap v110: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail 2024-09-18T08:01:01.298 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:01:01 smithi050 ceph-mon[32371]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-18T08:01:01.298 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:01:01 smithi050 ceph-mon[32371]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' 2024-09-18T08:01:01.298 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:01:01 smithi050 ceph-mon[32371]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' 2024-09-18T08:01:01.481 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:01:01 smithi007 ceph-mon[29459]: pgmap v110: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail 2024-09-18T08:01:01.481 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:01:01 smithi007 ceph-mon[29459]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-18T08:01:01.481 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:01:01 smithi007 ceph-mon[29459]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' 2024-09-18T08:01:01.481 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:01:01 smithi007 ceph-mon[29459]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' 2024-09-18T08:01:01.482 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:01:01 smithi007 ceph-mon[39249]: pgmap v110: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail 2024-09-18T08:01:01.482 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:01:01 smithi007 ceph-mon[39249]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-18T08:01:01.482 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:01:01 smithi007 ceph-mon[39249]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' 2024-09-18T08:01:01.482 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:01:01 smithi007 ceph-mon[39249]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' 2024-09-18T08:01:03.213 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:01:03 smithi050 ceph-mon[32371]: pgmap v111: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail 2024-09-18T08:01:03.213 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:01:03 smithi050 ceph-mon[32371]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' 2024-09-18T08:01:03.213 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:01:03 smithi050 ceph-mon[32371]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' 2024-09-18T08:01:03.213 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:01:03 smithi050 ceph-mon[32371]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-18T08:01:03.213 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:01:03 smithi050 ceph-mon[32371]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-18T08:01:03.213 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:01:03 smithi050 ceph-mon[32371]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' 2024-09-18T08:01:03.481 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:01:03 smithi007 ceph-mon[29459]: pgmap v111: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail 2024-09-18T08:01:03.481 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:01:03 smithi007 ceph-mon[29459]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' 2024-09-18T08:01:03.481 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:01:03 smithi007 ceph-mon[29459]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' 2024-09-18T08:01:03.481 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:01:03 smithi007 ceph-mon[29459]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-18T08:01:03.481 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:01:03 smithi007 ceph-mon[29459]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-18T08:01:03.481 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:01:03 smithi007 ceph-mon[29459]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' 2024-09-18T08:01:03.481 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:01:03 smithi007 ceph-mon[39249]: pgmap v111: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail 2024-09-18T08:01:03.482 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:01:03 smithi007 ceph-mon[39249]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' 2024-09-18T08:01:03.482 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:01:03 smithi007 ceph-mon[39249]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' 2024-09-18T08:01:03.482 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:01:03 smithi007 ceph-mon[39249]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-18T08:01:03.482 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:01:03 smithi007 ceph-mon[39249]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-18T08:01:03.482 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:01:03 smithi007 ceph-mon[39249]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' 2024-09-18T08:01:05.406 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:01:05 smithi050 ceph-mon[32371]: pgmap v112: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail 2024-09-18T08:01:05.480 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:01:05 smithi007 ceph-mon[29459]: pgmap v112: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail 2024-09-18T08:01:05.481 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:01:05 smithi007 ceph-mon[39249]: pgmap v112: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail 2024-09-18T08:01:06.242 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:01:06 smithi050 ceph-mon[32371]: from='osd.4 v2:172.21.15.50:6800/3662119805' entity='osd.4' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]} : dispatch 2024-09-18T08:01:06.242 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:01:06 smithi050 ceph-mon[32371]: from='osd.4 ' entity='osd.4' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]} : dispatch 2024-09-18T08:01:06.290 INFO:teuthology.orchestra.run.smithi050.stdout:Created osd(s) 4 on host 'smithi050' 2024-09-18T08:01:06.481 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:01:06 smithi007 ceph-mon[39249]: from='osd.4 v2:172.21.15.50:6800/3662119805' entity='osd.4' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]} : dispatch 2024-09-18T08:01:06.481 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:01:06 smithi007 ceph-mon[39249]: from='osd.4 ' entity='osd.4' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]} : dispatch 2024-09-18T08:01:06.481 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:01:06 smithi007 ceph-mon[29459]: from='osd.4 v2:172.21.15.50:6800/3662119805' entity='osd.4' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]} : dispatch 2024-09-18T08:01:06.482 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:01:06 smithi007 ceph-mon[29459]: from='osd.4 ' entity='osd.4' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]} : dispatch 2024-09-18T08:01:06.959 DEBUG:teuthology.orchestra.run.smithi050:osd.4> sudo journalctl -f -n 0 -u ceph-779349ca-7593-11ef-bceb-c7b262605968@osd.4.service 2024-09-18T08:01:06.962 INFO:tasks.cephadm:Deploying osd.5 on smithi050 with /dev/vg_nvme/lv_3... 2024-09-18T08:01:06.963 DEBUG:teuthology.orchestra.run.smithi050:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e ceph-volume -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 779349ca-7593-11ef-bceb-c7b262605968 -- lvm zap /dev/vg_nvme/lv_3 2024-09-18T08:01:07.266 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:01:07 smithi050 ceph-mon[32371]: pgmap v113: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail 2024-09-18T08:01:07.267 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:01:07 smithi050 ceph-mon[32371]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-09-18T08:01:07.267 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:01:07 smithi050 ceph-mon[32371]: from='osd.4 v2:172.21.15.50:6800/3662119805' entity='osd.4' cmd={"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi050", "root=default"]} : dispatch 2024-09-18T08:01:07.267 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:01:07 smithi050 ceph-mon[32371]: osdmap e30: 5 total, 4 up, 5 in 2024-09-18T08:01:07.267 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:01:07 smithi050 ceph-mon[32371]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 4} : dispatch 2024-09-18T08:01:07.267 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:01:07 smithi050 ceph-mon[32371]: from='osd.4 ' entity='osd.4' cmd={"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi050", "root=default"]} : dispatch 2024-09-18T08:01:07.267 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:01:07 smithi050 ceph-mon[32371]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-18T08:01:07.268 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:01:07 smithi050 ceph-mon[32371]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' 2024-09-18T08:01:07.268 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:01:07 smithi050 ceph-mon[32371]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' 2024-09-18T08:01:07.268 INFO:journalctl@ceph.osd.4.smithi050.stdout:Sep 18 08:01:07 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd-4[36561]: 2024-09-18T08:01:07.056+0000 7f875d9b4640 -1 osd.4 0 waiting for initial osdmap 2024-09-18T08:01:07.268 INFO:journalctl@ceph.osd.4.smithi050.stdout:Sep 18 08:01:07 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd-4[36561]: 2024-09-18T08:01:07.061+0000 7f87597db640 -1 osd.4 31 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-18T08:01:07.481 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:01:07 smithi007 ceph-mon[29459]: pgmap v113: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail 2024-09-18T08:01:07.481 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:01:07 smithi007 ceph-mon[29459]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-09-18T08:01:07.481 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:01:07 smithi007 ceph-mon[29459]: from='osd.4 v2:172.21.15.50:6800/3662119805' entity='osd.4' cmd={"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi050", "root=default"]} : dispatch 2024-09-18T08:01:07.481 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:01:07 smithi007 ceph-mon[29459]: osdmap e30: 5 total, 4 up, 5 in 2024-09-18T08:01:07.481 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:01:07 smithi007 ceph-mon[29459]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 4} : dispatch 2024-09-18T08:01:07.481 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:01:07 smithi007 ceph-mon[29459]: from='osd.4 ' entity='osd.4' cmd={"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi050", "root=default"]} : dispatch 2024-09-18T08:01:07.481 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:01:07 smithi007 ceph-mon[29459]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-18T08:01:07.481 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:01:07 smithi007 ceph-mon[29459]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' 2024-09-18T08:01:07.481 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:01:07 smithi007 ceph-mon[29459]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' 2024-09-18T08:01:07.482 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:01:07 smithi007 ceph-mon[39249]: pgmap v113: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail 2024-09-18T08:01:07.482 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:01:07 smithi007 ceph-mon[39249]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-09-18T08:01:07.482 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:01:07 smithi007 ceph-mon[39249]: from='osd.4 v2:172.21.15.50:6800/3662119805' entity='osd.4' cmd={"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi050", "root=default"]} : dispatch 2024-09-18T08:01:07.482 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:01:07 smithi007 ceph-mon[39249]: osdmap e30: 5 total, 4 up, 5 in 2024-09-18T08:01:07.482 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:01:07 smithi007 ceph-mon[39249]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 4} : dispatch 2024-09-18T08:01:07.482 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:01:07 smithi007 ceph-mon[39249]: from='osd.4 ' entity='osd.4' cmd={"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi050", "root=default"]} : dispatch 2024-09-18T08:01:07.482 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:01:07 smithi007 ceph-mon[39249]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-18T08:01:07.482 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:01:07 smithi007 ceph-mon[39249]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' 2024-09-18T08:01:07.482 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:01:07 smithi007 ceph-mon[39249]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' 2024-09-18T08:01:08.015 INFO:teuthology.orchestra.run.smithi050.stderr:Inferring config /var/lib/ceph/779349ca-7593-11ef-bceb-c7b262605968/mon.b/config 2024-09-18T08:01:08.281 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:01:08 smithi050 ceph-mon[32371]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi050", "root=default"]}]': finished 2024-09-18T08:01:08.281 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:01:08 smithi050 ceph-mon[32371]: osdmap e31: 5 total, 4 up, 5 in 2024-09-18T08:01:08.282 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:01:08 smithi050 ceph-mon[32371]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 4} : dispatch 2024-09-18T08:01:08.282 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:01:08 smithi050 ceph-mon[32371]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 4} : dispatch 2024-09-18T08:01:08.481 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:01:08 smithi007 ceph-mon[29459]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi050", "root=default"]}]': finished 2024-09-18T08:01:08.481 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:01:08 smithi007 ceph-mon[29459]: osdmap e31: 5 total, 4 up, 5 in 2024-09-18T08:01:08.481 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:01:08 smithi007 ceph-mon[29459]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 4} : dispatch 2024-09-18T08:01:08.481 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:01:08 smithi007 ceph-mon[29459]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 4} : dispatch 2024-09-18T08:01:08.481 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:01:08 smithi007 ceph-mon[39249]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi050", "root=default"]}]': finished 2024-09-18T08:01:08.481 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:01:08 smithi007 ceph-mon[39249]: osdmap e31: 5 total, 4 up, 5 in 2024-09-18T08:01:08.482 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:01:08 smithi007 ceph-mon[39249]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 4} : dispatch 2024-09-18T08:01:08.482 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:01:08 smithi007 ceph-mon[39249]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 4} : dispatch 2024-09-18T08:01:09.406 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:01:09 smithi050 ceph-mon[32371]: purged_snaps scrub starts 2024-09-18T08:01:09.406 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:01:09 smithi050 ceph-mon[32371]: purged_snaps scrub ok 2024-09-18T08:01:09.406 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:01:09 smithi050 ceph-mon[32371]: pgmap v116: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail 2024-09-18T08:01:09.406 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:01:09 smithi050 ceph-mon[32371]: osd.4 v2:172.21.15.50:6800/3662119805 boot 2024-09-18T08:01:09.407 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:01:09 smithi050 ceph-mon[32371]: osdmap e32: 5 total, 5 up, 5 in 2024-09-18T08:01:09.407 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:01:09 smithi050 ceph-mon[32371]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 4} : dispatch 2024-09-18T08:01:09.407 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:01:09 smithi050 ceph-mon[32371]: osdmap e33: 5 total, 5 up, 5 in 2024-09-18T08:01:09.481 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:01:09 smithi007 ceph-mon[29459]: purged_snaps scrub starts 2024-09-18T08:01:09.481 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:01:09 smithi007 ceph-mon[29459]: purged_snaps scrub ok 2024-09-18T08:01:09.481 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:01:09 smithi007 ceph-mon[29459]: pgmap v116: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail 2024-09-18T08:01:09.481 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:01:09 smithi007 ceph-mon[29459]: osd.4 v2:172.21.15.50:6800/3662119805 boot 2024-09-18T08:01:09.481 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:01:09 smithi007 ceph-mon[29459]: osdmap e32: 5 total, 5 up, 5 in 2024-09-18T08:01:09.481 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:01:09 smithi007 ceph-mon[29459]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 4} : dispatch 2024-09-18T08:01:09.481 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:01:09 smithi007 ceph-mon[29459]: osdmap e33: 5 total, 5 up, 5 in 2024-09-18T08:01:09.482 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:01:09 smithi007 ceph-mon[39249]: purged_snaps scrub starts 2024-09-18T08:01:09.482 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:01:09 smithi007 ceph-mon[39249]: purged_snaps scrub ok 2024-09-18T08:01:09.482 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:01:09 smithi007 ceph-mon[39249]: pgmap v116: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail 2024-09-18T08:01:09.482 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:01:09 smithi007 ceph-mon[39249]: osd.4 v2:172.21.15.50:6800/3662119805 boot 2024-09-18T08:01:09.482 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:01:09 smithi007 ceph-mon[39249]: osdmap e32: 5 total, 5 up, 5 in 2024-09-18T08:01:09.482 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:01:09 smithi007 ceph-mon[39249]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 4} : dispatch 2024-09-18T08:01:09.482 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:01:09 smithi007 ceph-mon[39249]: osdmap e33: 5 total, 5 up, 5 in 2024-09-18T08:01:11.331 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:01:11 smithi050 ceph-mon[32371]: pgmap v119: 1 pgs: 1 peering; 577 KiB data, 139 MiB used, 447 GiB / 447 GiB avail 2024-09-18T08:01:11.331 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:01:11 smithi050 ceph-mon[32371]: purged_snaps scrub starts 2024-09-18T08:01:11.331 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:01:11 smithi050 ceph-mon[32371]: purged_snaps scrub ok 2024-09-18T08:01:11.331 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:01:11 smithi050 ceph-mon[32371]: osdmap e34: 5 total, 5 up, 5 in 2024-09-18T08:01:11.331 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:01:11 smithi050 ceph-mon[32371]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' 2024-09-18T08:01:11.331 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:01:11 smithi050 ceph-mon[32371]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' 2024-09-18T08:01:11.331 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:01:11 smithi050 ceph-mon[32371]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"} : dispatch 2024-09-18T08:01:11.331 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:01:11 smithi050 ceph-mon[32371]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' 2024-09-18T08:01:11.331 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:01:11 smithi050 ceph-mon[32371]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-18T08:01:11.331 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:01:11 smithi050 ceph-mon[32371]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-18T08:01:11.331 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:01:11 smithi050 ceph-mon[32371]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' 2024-09-18T08:01:11.481 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:01:11 smithi007 ceph-mon[39249]: pgmap v119: 1 pgs: 1 peering; 577 KiB data, 139 MiB used, 447 GiB / 447 GiB avail 2024-09-18T08:01:11.481 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:01:11 smithi007 ceph-mon[39249]: purged_snaps scrub starts 2024-09-18T08:01:11.481 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:01:11 smithi007 ceph-mon[39249]: purged_snaps scrub ok 2024-09-18T08:01:11.481 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:01:11 smithi007 ceph-mon[39249]: osdmap e34: 5 total, 5 up, 5 in 2024-09-18T08:01:11.481 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:01:11 smithi007 ceph-mon[39249]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' 2024-09-18T08:01:11.481 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:01:11 smithi007 ceph-mon[39249]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' 2024-09-18T08:01:11.481 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:01:11 smithi007 ceph-mon[39249]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"} : dispatch 2024-09-18T08:01:11.481 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:01:11 smithi007 ceph-mon[39249]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' 2024-09-18T08:01:11.481 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:01:11 smithi007 ceph-mon[39249]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-18T08:01:11.481 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:01:11 smithi007 ceph-mon[39249]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-18T08:01:11.482 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:01:11 smithi007 ceph-mon[39249]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' 2024-09-18T08:01:11.482 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:01:11 smithi007 ceph-mon[29459]: pgmap v119: 1 pgs: 1 peering; 577 KiB data, 139 MiB used, 447 GiB / 447 GiB avail 2024-09-18T08:01:11.482 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:01:11 smithi007 ceph-mon[29459]: purged_snaps scrub starts 2024-09-18T08:01:11.482 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:01:11 smithi007 ceph-mon[29459]: purged_snaps scrub ok 2024-09-18T08:01:11.482 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:01:11 smithi007 ceph-mon[29459]: osdmap e34: 5 total, 5 up, 5 in 2024-09-18T08:01:11.482 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:01:11 smithi007 ceph-mon[29459]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' 2024-09-18T08:01:11.482 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:01:11 smithi007 ceph-mon[29459]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' 2024-09-18T08:01:11.482 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:01:11 smithi007 ceph-mon[29459]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"} : dispatch 2024-09-18T08:01:11.482 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:01:11 smithi007 ceph-mon[29459]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' 2024-09-18T08:01:11.482 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:01:11 smithi007 ceph-mon[29459]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-18T08:01:11.483 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:01:11 smithi007 ceph-mon[29459]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-18T08:01:11.483 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:01:11 smithi007 ceph-mon[29459]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' 2024-09-18T08:01:11.943 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-09-18T08:01:11.970 DEBUG:teuthology.orchestra.run.smithi050:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 779349ca-7593-11ef-bceb-c7b262605968 -- ceph orch daemon add osd smithi050:vg_nvme/lv_3 2024-09-18T08:01:12.226 INFO:teuthology.orchestra.run.smithi050.stderr:Inferring config /var/lib/ceph/779349ca-7593-11ef-bceb-c7b262605968/mon.b/config 2024-09-18T08:01:12.346 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:01:12 smithi050 ceph-mon[32371]: Detected new or changed devices on smithi050 2024-09-18T08:01:12.347 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:01:12 smithi050 ceph-mon[32371]: Adjusting osd_memory_target on smithi050 to 17092M 2024-09-18T08:01:12.480 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:01:12 smithi007 ceph-mon[29459]: Detected new or changed devices on smithi050 2024-09-18T08:01:12.481 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:01:12 smithi007 ceph-mon[29459]: Adjusting osd_memory_target on smithi050 to 17092M 2024-09-18T08:01:12.481 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:01:12 smithi007 ceph-mon[39249]: Detected new or changed devices on smithi050 2024-09-18T08:01:12.481 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:01:12 smithi007 ceph-mon[39249]: Adjusting osd_memory_target on smithi050 to 17092M 2024-09-18T08:01:13.274 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:01:13 smithi050 ceph-mon[32371]: pgmap v121: 1 pgs: 1 peering; 577 KiB data, 139 MiB used, 447 GiB / 447 GiB avail 2024-09-18T08:01:13.274 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:01:13 smithi050 ceph-mon[32371]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' cmd={"prefix": "osd tree", "states": ["destroyed"], "format": "json"} : dispatch 2024-09-18T08:01:13.274 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:01:13 smithi050 ceph-mon[32371]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.bootstrap-osd"} : dispatch 2024-09-18T08:01:13.274 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:01:13 smithi050 ceph-mon[32371]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-18T08:01:13.481 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:01:13 smithi007 ceph-mon[29459]: pgmap v121: 1 pgs: 1 peering; 577 KiB data, 139 MiB used, 447 GiB / 447 GiB avail 2024-09-18T08:01:13.481 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:01:13 smithi007 ceph-mon[29459]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' cmd={"prefix": "osd tree", "states": ["destroyed"], "format": "json"} : dispatch 2024-09-18T08:01:13.481 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:01:13 smithi007 ceph-mon[29459]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.bootstrap-osd"} : dispatch 2024-09-18T08:01:13.481 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:01:13 smithi007 ceph-mon[29459]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-18T08:01:13.481 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:01:13 smithi007 ceph-mon[39249]: pgmap v121: 1 pgs: 1 peering; 577 KiB data, 139 MiB used, 447 GiB / 447 GiB avail 2024-09-18T08:01:13.481 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:01:13 smithi007 ceph-mon[39249]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' cmd={"prefix": "osd tree", "states": ["destroyed"], "format": "json"} : dispatch 2024-09-18T08:01:13.481 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:01:13 smithi007 ceph-mon[39249]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.bootstrap-osd"} : dispatch 2024-09-18T08:01:13.481 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:01:13 smithi007 ceph-mon[39249]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-18T08:01:14.156 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:01:14 smithi050 ceph-mon[32371]: from='client.24223 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi050:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-09-18T08:01:14.481 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:01:14 smithi007 ceph-mon[29459]: from='client.24223 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi050:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-09-18T08:01:14.481 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:01:14 smithi007 ceph-mon[39249]: from='client.24223 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi050:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-09-18T08:01:15.406 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:01:15 smithi050 ceph-mon[32371]: pgmap v122: 1 pgs: 1 active+clean; 577 KiB data, 139 MiB used, 447 GiB / 447 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-09-18T08:01:15.480 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:01:15 smithi007 ceph-mon[29459]: pgmap v122: 1 pgs: 1 active+clean; 577 KiB data, 139 MiB used, 447 GiB / 447 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-09-18T08:01:15.481 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:01:15 smithi007 ceph-mon[39249]: pgmap v122: 1 pgs: 1 active+clean; 577 KiB data, 139 MiB used, 447 GiB / 447 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-09-18T08:01:16.406 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:01:16 smithi050 ceph-mon[32371]: from='client.? 172.21.15.50:0/1776732104' entity='client.bootstrap-osd' cmd={"prefix": "osd new", "uuid": "7f290005-ebca-467b-87e4-02ead1063610"} : dispatch 2024-09-18T08:01:16.406 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:01:16 smithi050 ceph-mon[32371]: from='client.? ' entity='client.bootstrap-osd' cmd={"prefix": "osd new", "uuid": "7f290005-ebca-467b-87e4-02ead1063610"} : dispatch 2024-09-18T08:01:16.406 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:01:16 smithi050 ceph-mon[32371]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "7f290005-ebca-467b-87e4-02ead1063610"}]': finished 2024-09-18T08:01:16.406 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:01:16 smithi050 ceph-mon[32371]: osdmap e35: 6 total, 5 up, 6 in 2024-09-18T08:01:16.406 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:01:16 smithi050 ceph-mon[32371]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 5} : dispatch 2024-09-18T08:01:16.407 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:01:16 smithi050 ceph-mon[32371]: from='client.? 172.21.15.50:0/349229759' entity='client.bootstrap-osd' cmd={"prefix": "mon getmap"} : dispatch 2024-09-18T08:01:16.481 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:01:16 smithi007 ceph-mon[29459]: from='client.? 172.21.15.50:0/1776732104' entity='client.bootstrap-osd' cmd={"prefix": "osd new", "uuid": "7f290005-ebca-467b-87e4-02ead1063610"} : dispatch 2024-09-18T08:01:16.481 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:01:16 smithi007 ceph-mon[29459]: from='client.? ' entity='client.bootstrap-osd' cmd={"prefix": "osd new", "uuid": "7f290005-ebca-467b-87e4-02ead1063610"} : dispatch 2024-09-18T08:01:16.481 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:01:16 smithi007 ceph-mon[29459]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "7f290005-ebca-467b-87e4-02ead1063610"}]': finished 2024-09-18T08:01:16.481 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:01:16 smithi007 ceph-mon[29459]: osdmap e35: 6 total, 5 up, 6 in 2024-09-18T08:01:16.481 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:01:16 smithi007 ceph-mon[29459]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 5} : dispatch 2024-09-18T08:01:16.481 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:01:16 smithi007 ceph-mon[29459]: from='client.? 172.21.15.50:0/349229759' entity='client.bootstrap-osd' cmd={"prefix": "mon getmap"} : dispatch 2024-09-18T08:01:16.481 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:01:16 smithi007 ceph-mon[39249]: from='client.? 172.21.15.50:0/1776732104' entity='client.bootstrap-osd' cmd={"prefix": "osd new", "uuid": "7f290005-ebca-467b-87e4-02ead1063610"} : dispatch 2024-09-18T08:01:16.482 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:01:16 smithi007 ceph-mon[39249]: from='client.? ' entity='client.bootstrap-osd' cmd={"prefix": "osd new", "uuid": "7f290005-ebca-467b-87e4-02ead1063610"} : dispatch 2024-09-18T08:01:16.482 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:01:16 smithi007 ceph-mon[39249]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "7f290005-ebca-467b-87e4-02ead1063610"}]': finished 2024-09-18T08:01:16.482 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:01:16 smithi007 ceph-mon[39249]: osdmap e35: 6 total, 5 up, 6 in 2024-09-18T08:01:16.482 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:01:16 smithi007 ceph-mon[39249]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 5} : dispatch 2024-09-18T08:01:16.482 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:01:16 smithi007 ceph-mon[39249]: from='client.? 172.21.15.50:0/349229759' entity='client.bootstrap-osd' cmd={"prefix": "mon getmap"} : dispatch 2024-09-18T08:01:17.406 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:01:17 smithi050 ceph-mon[32371]: pgmap v124: 1 pgs: 1 active+clean; 577 KiB data, 139 MiB used, 447 GiB / 447 GiB avail; 91 KiB/s, 0 objects/s recovering 2024-09-18T08:01:17.480 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:01:17 smithi007 ceph-mon[29459]: pgmap v124: 1 pgs: 1 active+clean; 577 KiB data, 139 MiB used, 447 GiB / 447 GiB avail; 91 KiB/s, 0 objects/s recovering 2024-09-18T08:01:17.481 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:01:17 smithi007 ceph-mon[39249]: pgmap v124: 1 pgs: 1 active+clean; 577 KiB data, 139 MiB used, 447 GiB / 447 GiB avail; 91 KiB/s, 0 objects/s recovering 2024-09-18T08:01:19.328 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:01:19 smithi050 ceph-mon[32371]: pgmap v125: 1 pgs: 1 active+clean; 577 KiB data, 139 MiB used, 447 GiB / 447 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-09-18T08:01:19.328 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:01:19 smithi050 ceph-mon[32371]: purged_snaps scrub starts 2024-09-18T08:01:19.328 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:01:19 smithi050 ceph-mon[32371]: purged_snaps scrub ok 2024-09-18T08:01:19.481 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:01:19 smithi007 ceph-mon[29459]: pgmap v125: 1 pgs: 1 active+clean; 577 KiB data, 139 MiB used, 447 GiB / 447 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-09-18T08:01:19.481 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:01:19 smithi007 ceph-mon[29459]: purged_snaps scrub starts 2024-09-18T08:01:19.481 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:01:19 smithi007 ceph-mon[29459]: purged_snaps scrub ok 2024-09-18T08:01:19.481 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:01:19 smithi007 ceph-mon[39249]: pgmap v125: 1 pgs: 1 active+clean; 577 KiB data, 139 MiB used, 447 GiB / 447 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-09-18T08:01:19.482 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:01:19 smithi007 ceph-mon[39249]: purged_snaps scrub starts 2024-09-18T08:01:19.482 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:01:19 smithi007 ceph-mon[39249]: purged_snaps scrub ok 2024-09-18T08:01:20.378 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:01:20 smithi050 ceph-mon[32371]: osdmap e36: 6 total, 5 up, 6 in 2024-09-18T08:01:20.378 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:01:20 smithi050 ceph-mon[32371]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 5} : dispatch 2024-09-18T08:01:20.481 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:01:20 smithi007 ceph-mon[29459]: osdmap e36: 6 total, 5 up, 6 in 2024-09-18T08:01:20.481 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:01:20 smithi007 ceph-mon[29459]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 5} : dispatch 2024-09-18T08:01:20.481 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:01:20 smithi007 ceph-mon[39249]: osdmap e36: 6 total, 5 up, 6 in 2024-09-18T08:01:20.481 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:01:20 smithi007 ceph-mon[39249]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 5} : dispatch 2024-09-18T08:01:21.383 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:01:21 smithi050 ceph-mon[32371]: pgmap v127: 1 pgs: 1 active+clean; 577 KiB data, 139 MiB used, 447 GiB / 447 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-09-18T08:01:21.480 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:01:21 smithi007 ceph-mon[29459]: pgmap v127: 1 pgs: 1 active+clean; 577 KiB data, 139 MiB used, 447 GiB / 447 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-09-18T08:01:21.481 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:01:21 smithi007 ceph-mon[39249]: pgmap v127: 1 pgs: 1 active+clean; 577 KiB data, 139 MiB used, 447 GiB / 447 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-09-18T08:01:23.406 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:01:23 smithi050 ceph-mon[32371]: pgmap v128: 1 pgs: 1 active+clean; 577 KiB data, 139 MiB used, 447 GiB / 447 GiB avail 2024-09-18T08:01:23.406 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:01:23 smithi050 ceph-mon[32371]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' cmd={"prefix": "auth get", "entity": "osd.5"} : dispatch 2024-09-18T08:01:23.407 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:01:23 smithi050 ceph-mon[32371]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-18T08:01:23.481 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:01:23 smithi007 ceph-mon[29459]: pgmap v128: 1 pgs: 1 active+clean; 577 KiB data, 139 MiB used, 447 GiB / 447 GiB avail 2024-09-18T08:01:23.481 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:01:23 smithi007 ceph-mon[29459]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' cmd={"prefix": "auth get", "entity": "osd.5"} : dispatch 2024-09-18T08:01:23.481 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:01:23 smithi007 ceph-mon[29459]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-18T08:01:23.481 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:01:23 smithi007 ceph-mon[39249]: pgmap v128: 1 pgs: 1 active+clean; 577 KiB data, 139 MiB used, 447 GiB / 447 GiB avail 2024-09-18T08:01:23.481 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:01:23 smithi007 ceph-mon[39249]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' cmd={"prefix": "auth get", "entity": "osd.5"} : dispatch 2024-09-18T08:01:23.481 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:01:23 smithi007 ceph-mon[39249]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-18T08:01:24.354 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:01:24 smithi050 ceph-mon[32371]: Deploying daemon osd.5 on smithi050 2024-09-18T08:01:24.480 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:01:24 smithi007 ceph-mon[29459]: Deploying daemon osd.5 on smithi050 2024-09-18T08:01:24.481 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:01:24 smithi007 ceph-mon[39249]: Deploying daemon osd.5 on smithi050 2024-09-18T08:01:25.156 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:01:25 smithi050 ceph-mon[32371]: pgmap v129: 1 pgs: 1 active+clean; 577 KiB data, 139 MiB used, 447 GiB / 447 GiB avail 2024-09-18T08:01:25.481 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:01:25 smithi007 ceph-mon[39249]: pgmap v129: 1 pgs: 1 active+clean; 577 KiB data, 139 MiB used, 447 GiB / 447 GiB avail 2024-09-18T08:01:25.481 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:01:25 smithi007 ceph-mon[29459]: pgmap v129: 1 pgs: 1 active+clean; 577 KiB data, 139 MiB used, 447 GiB / 447 GiB avail 2024-09-18T08:01:27.406 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:01:27 smithi050 ceph-mon[32371]: pgmap v130: 1 pgs: 1 active+clean; 577 KiB data, 139 MiB used, 447 GiB / 447 GiB avail 2024-09-18T08:01:27.480 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:01:27 smithi007 ceph-mon[29459]: pgmap v130: 1 pgs: 1 active+clean; 577 KiB data, 139 MiB used, 447 GiB / 447 GiB avail 2024-09-18T08:01:27.481 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:01:27 smithi007 ceph-mon[39249]: pgmap v130: 1 pgs: 1 active+clean; 577 KiB data, 139 MiB used, 447 GiB / 447 GiB avail 2024-09-18T08:01:29.271 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:01:29 smithi050 ceph-mon[32371]: pgmap v131: 1 pgs: 1 active+clean; 577 KiB data, 139 MiB used, 447 GiB / 447 GiB avail 2024-09-18T08:01:29.480 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:01:29 smithi007 ceph-mon[29459]: pgmap v131: 1 pgs: 1 active+clean; 577 KiB data, 139 MiB used, 447 GiB / 447 GiB avail 2024-09-18T08:01:29.481 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:01:29 smithi007 ceph-mon[39249]: pgmap v131: 1 pgs: 1 active+clean; 577 KiB data, 139 MiB used, 447 GiB / 447 GiB avail 2024-09-18T08:01:30.350 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:01:30 smithi050 ceph-mon[32371]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-18T08:01:30.350 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:01:30 smithi050 ceph-mon[32371]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' 2024-09-18T08:01:30.350 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:01:30 smithi050 ceph-mon[32371]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' 2024-09-18T08:01:30.481 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:01:30 smithi007 ceph-mon[29459]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-18T08:01:30.481 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:01:30 smithi007 ceph-mon[29459]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' 2024-09-18T08:01:30.481 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:01:30 smithi007 ceph-mon[29459]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' 2024-09-18T08:01:30.481 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:01:30 smithi007 ceph-mon[39249]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-18T08:01:30.481 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:01:30 smithi007 ceph-mon[39249]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' 2024-09-18T08:01:30.481 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:01:30 smithi007 ceph-mon[39249]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' 2024-09-18T08:01:31.371 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:01:31 smithi050 ceph-mon[32371]: pgmap v132: 1 pgs: 1 active+clean; 577 KiB data, 139 MiB used, 447 GiB / 447 GiB avail 2024-09-18T08:01:31.372 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:01:31 smithi050 ceph-mon[32371]: 1.0 scrub starts 2024-09-18T08:01:31.372 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:01:31 smithi050 ceph-mon[32371]: 1.0 scrub ok 2024-09-18T08:01:31.481 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:01:31 smithi007 ceph-mon[29459]: pgmap v132: 1 pgs: 1 active+clean; 577 KiB data, 139 MiB used, 447 GiB / 447 GiB avail 2024-09-18T08:01:31.481 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:01:31 smithi007 ceph-mon[29459]: 1.0 scrub starts 2024-09-18T08:01:31.481 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:01:31 smithi007 ceph-mon[29459]: 1.0 scrub ok 2024-09-18T08:01:31.481 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:01:31 smithi007 ceph-mon[39249]: pgmap v132: 1 pgs: 1 active+clean; 577 KiB data, 139 MiB used, 447 GiB / 447 GiB avail 2024-09-18T08:01:31.481 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:01:31 smithi007 ceph-mon[39249]: 1.0 scrub starts 2024-09-18T08:01:31.481 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:01:31 smithi007 ceph-mon[39249]: 1.0 scrub ok 2024-09-18T08:01:33.231 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:01:32 smithi007 ceph-mon[39249]: pgmap v133: 1 pgs: 1 active+clean; 577 KiB data, 139 MiB used, 447 GiB / 447 GiB avail 2024-09-18T08:01:33.231 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:01:32 smithi007 ceph-mon[39249]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' 2024-09-18T08:01:33.231 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:01:32 smithi007 ceph-mon[39249]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' 2024-09-18T08:01:33.231 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:01:32 smithi007 ceph-mon[39249]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-18T08:01:33.231 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:01:32 smithi007 ceph-mon[39249]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-18T08:01:33.231 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:01:32 smithi007 ceph-mon[39249]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' 2024-09-18T08:01:33.231 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:01:32 smithi007 ceph-mon[29459]: pgmap v133: 1 pgs: 1 active+clean; 577 KiB data, 139 MiB used, 447 GiB / 447 GiB avail 2024-09-18T08:01:33.231 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:01:32 smithi007 ceph-mon[29459]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' 2024-09-18T08:01:33.231 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:01:32 smithi007 ceph-mon[29459]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' 2024-09-18T08:01:33.231 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:01:32 smithi007 ceph-mon[29459]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-18T08:01:33.232 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:01:32 smithi007 ceph-mon[29459]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-18T08:01:33.232 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:01:32 smithi007 ceph-mon[29459]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' 2024-09-18T08:01:33.406 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:01:32 smithi050 ceph-mon[32371]: pgmap v133: 1 pgs: 1 active+clean; 577 KiB data, 139 MiB used, 447 GiB / 447 GiB avail 2024-09-18T08:01:33.406 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:01:32 smithi050 ceph-mon[32371]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' 2024-09-18T08:01:33.406 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:01:32 smithi050 ceph-mon[32371]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' 2024-09-18T08:01:33.406 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:01:32 smithi050 ceph-mon[32371]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-18T08:01:33.406 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:01:32 smithi050 ceph-mon[32371]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-18T08:01:33.406 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:01:32 smithi050 ceph-mon[32371]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' 2024-09-18T08:01:35.357 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:01:34 smithi050 ceph-mon[32371]: pgmap v134: 1 pgs: 1 active+clean; 577 KiB data, 139 MiB used, 447 GiB / 447 GiB avail 2024-09-18T08:01:35.358 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:01:34 smithi050 ceph-mon[32371]: from='osd.5 v2:172.21.15.50:6804/1199797954' entity='osd.5' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]} : dispatch 2024-09-18T08:01:35.358 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:01:34 smithi050 ceph-mon[32371]: from='osd.5 ' entity='osd.5' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]} : dispatch 2024-09-18T08:01:35.407 INFO:teuthology.orchestra.run.smithi050.stdout:Created osd(s) 5 on host 'smithi050' 2024-09-18T08:01:35.480 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:01:34 smithi007 ceph-mon[29459]: pgmap v134: 1 pgs: 1 active+clean; 577 KiB data, 139 MiB used, 447 GiB / 447 GiB avail 2024-09-18T08:01:35.481 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:01:34 smithi007 ceph-mon[29459]: from='osd.5 v2:172.21.15.50:6804/1199797954' entity='osd.5' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]} : dispatch 2024-09-18T08:01:35.481 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:01:34 smithi007 ceph-mon[29459]: from='osd.5 ' entity='osd.5' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]} : dispatch 2024-09-18T08:01:35.481 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:01:34 smithi007 ceph-mon[39249]: pgmap v134: 1 pgs: 1 active+clean; 577 KiB data, 139 MiB used, 447 GiB / 447 GiB avail 2024-09-18T08:01:35.481 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:01:34 smithi007 ceph-mon[39249]: from='osd.5 v2:172.21.15.50:6804/1199797954' entity='osd.5' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]} : dispatch 2024-09-18T08:01:35.481 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:01:34 smithi007 ceph-mon[39249]: from='osd.5 ' entity='osd.5' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]} : dispatch 2024-09-18T08:01:36.034 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:01:35 smithi050 ceph-mon[32371]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-09-18T08:01:36.034 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:01:35 smithi050 ceph-mon[32371]: from='osd.5 v2:172.21.15.50:6804/1199797954' entity='osd.5' cmd={"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi050", "root=default"]} : dispatch 2024-09-18T08:01:36.035 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:01:35 smithi050 ceph-mon[32371]: osdmap e37: 6 total, 5 up, 6 in 2024-09-18T08:01:36.035 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:01:35 smithi050 ceph-mon[32371]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 5} : dispatch 2024-09-18T08:01:36.035 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:01:35 smithi050 ceph-mon[32371]: from='osd.5 ' entity='osd.5' cmd={"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi050", "root=default"]} : dispatch 2024-09-18T08:01:36.035 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:01:35 smithi050 ceph-mon[32371]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-18T08:01:36.035 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:01:35 smithi050 ceph-mon[32371]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' 2024-09-18T08:01:36.035 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:01:35 smithi050 ceph-mon[32371]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' 2024-09-18T08:01:36.035 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:01:35 smithi050 ceph-mon[32371]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi050", "root=default"]}]': finished 2024-09-18T08:01:36.037 DEBUG:teuthology.orchestra.run.smithi050:osd.5> sudo journalctl -f -n 0 -u ceph-779349ca-7593-11ef-bceb-c7b262605968@osd.5.service 2024-09-18T08:01:36.039 INFO:tasks.cephadm:Deploying osd.6 on smithi050 with /dev/vg_nvme/lv_2... 2024-09-18T08:01:36.039 DEBUG:teuthology.orchestra.run.smithi050:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e ceph-volume -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 779349ca-7593-11ef-bceb-c7b262605968 -- lvm zap /dev/vg_nvme/lv_2 2024-09-18T08:01:36.481 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:01:35 smithi007 ceph-mon[29459]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-09-18T08:01:36.481 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:01:35 smithi007 ceph-mon[29459]: from='osd.5 v2:172.21.15.50:6804/1199797954' entity='osd.5' cmd={"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi050", "root=default"]} : dispatch 2024-09-18T08:01:36.481 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:01:35 smithi007 ceph-mon[29459]: osdmap e37: 6 total, 5 up, 6 in 2024-09-18T08:01:36.481 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:01:35 smithi007 ceph-mon[29459]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 5} : dispatch 2024-09-18T08:01:36.481 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:01:35 smithi007 ceph-mon[29459]: from='osd.5 ' entity='osd.5' cmd={"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi050", "root=default"]} : dispatch 2024-09-18T08:01:36.481 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:01:35 smithi007 ceph-mon[29459]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-18T08:01:36.481 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:01:35 smithi007 ceph-mon[29459]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' 2024-09-18T08:01:36.481 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:01:35 smithi007 ceph-mon[29459]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' 2024-09-18T08:01:36.481 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:01:35 smithi007 ceph-mon[29459]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi050", "root=default"]}]': finished 2024-09-18T08:01:36.482 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:01:35 smithi007 ceph-mon[39249]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-09-18T08:01:36.482 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:01:35 smithi007 ceph-mon[39249]: from='osd.5 v2:172.21.15.50:6804/1199797954' entity='osd.5' cmd={"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi050", "root=default"]} : dispatch 2024-09-18T08:01:36.482 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:01:35 smithi007 ceph-mon[39249]: osdmap e37: 6 total, 5 up, 6 in 2024-09-18T08:01:36.482 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:01:35 smithi007 ceph-mon[39249]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 5} : dispatch 2024-09-18T08:01:36.482 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:01:35 smithi007 ceph-mon[39249]: from='osd.5 ' entity='osd.5' cmd={"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi050", "root=default"]} : dispatch 2024-09-18T08:01:36.482 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:01:35 smithi007 ceph-mon[39249]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-18T08:01:36.482 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:01:35 smithi007 ceph-mon[39249]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' 2024-09-18T08:01:36.482 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:01:35 smithi007 ceph-mon[39249]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' 2024-09-18T08:01:36.482 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:01:35 smithi007 ceph-mon[39249]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi050", "root=default"]}]': finished 2024-09-18T08:01:37.132 INFO:teuthology.orchestra.run.smithi050.stderr:Inferring config /var/lib/ceph/779349ca-7593-11ef-bceb-c7b262605968/mon.b/config 2024-09-18T08:01:37.256 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:01:36 smithi050 ceph-mon[32371]: pgmap v136: 1 pgs: 1 active+clean; 577 KiB data, 139 MiB used, 447 GiB / 447 GiB avail 2024-09-18T08:01:37.256 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:01:36 smithi050 ceph-mon[32371]: osdmap e38: 6 total, 5 up, 6 in 2024-09-18T08:01:37.256 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:01:36 smithi050 ceph-mon[32371]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 5} : dispatch 2024-09-18T08:01:37.256 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:01:36 smithi050 ceph-mon[32371]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 5} : dispatch 2024-09-18T08:01:37.256 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:01:36 smithi050 ceph-mon[32371]: osd.5 v2:172.21.15.50:6804/1199797954 boot 2024-09-18T08:01:37.256 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:01:36 smithi050 ceph-mon[32371]: osdmap e39: 6 total, 6 up, 6 in 2024-09-18T08:01:37.257 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:01:36 smithi050 ceph-mon[32371]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 5} : dispatch 2024-09-18T08:01:37.481 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:01:36 smithi007 ceph-mon[29459]: pgmap v136: 1 pgs: 1 active+clean; 577 KiB data, 139 MiB used, 447 GiB / 447 GiB avail 2024-09-18T08:01:37.481 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:01:36 smithi007 ceph-mon[29459]: osdmap e38: 6 total, 5 up, 6 in 2024-09-18T08:01:37.481 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:01:36 smithi007 ceph-mon[29459]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 5} : dispatch 2024-09-18T08:01:37.481 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:01:36 smithi007 ceph-mon[29459]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 5} : dispatch 2024-09-18T08:01:37.482 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:01:36 smithi007 ceph-mon[29459]: osd.5 v2:172.21.15.50:6804/1199797954 boot 2024-09-18T08:01:37.482 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:01:36 smithi007 ceph-mon[29459]: osdmap e39: 6 total, 6 up, 6 in 2024-09-18T08:01:37.482 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:01:36 smithi007 ceph-mon[29459]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 5} : dispatch 2024-09-18T08:01:37.482 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:01:36 smithi007 ceph-mon[39249]: pgmap v136: 1 pgs: 1 active+clean; 577 KiB data, 139 MiB used, 447 GiB / 447 GiB avail 2024-09-18T08:01:37.482 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:01:36 smithi007 ceph-mon[39249]: osdmap e38: 6 total, 5 up, 6 in 2024-09-18T08:01:37.482 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:01:36 smithi007 ceph-mon[39249]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 5} : dispatch 2024-09-18T08:01:37.482 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:01:36 smithi007 ceph-mon[39249]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 5} : dispatch 2024-09-18T08:01:37.482 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:01:36 smithi007 ceph-mon[39249]: osd.5 v2:172.21.15.50:6804/1199797954 boot 2024-09-18T08:01:37.482 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:01:36 smithi007 ceph-mon[39249]: osdmap e39: 6 total, 6 up, 6 in 2024-09-18T08:01:37.483 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:01:36 smithi007 ceph-mon[39249]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 5} : dispatch 2024-09-18T08:01:38.345 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:01:37 smithi050 ceph-mon[32371]: purged_snaps scrub starts 2024-09-18T08:01:38.345 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:01:37 smithi050 ceph-mon[32371]: purged_snaps scrub ok 2024-09-18T08:01:38.345 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:01:37 smithi050 ceph-mon[32371]: osdmap e40: 6 total, 6 up, 6 in 2024-09-18T08:01:38.481 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:01:37 smithi007 ceph-mon[29459]: purged_snaps scrub starts 2024-09-18T08:01:38.481 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:01:37 smithi007 ceph-mon[29459]: purged_snaps scrub ok 2024-09-18T08:01:38.481 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:01:37 smithi007 ceph-mon[29459]: osdmap e40: 6 total, 6 up, 6 in 2024-09-18T08:01:38.481 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:01:37 smithi007 ceph-mon[39249]: purged_snaps scrub starts 2024-09-18T08:01:38.481 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:01:37 smithi007 ceph-mon[39249]: purged_snaps scrub ok 2024-09-18T08:01:38.481 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:01:37 smithi007 ceph-mon[39249]: osdmap e40: 6 total, 6 up, 6 in 2024-09-18T08:01:39.406 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:01:38 smithi050 ceph-mon[32371]: pgmap v139: 1 pgs: 1 active+clean; 577 KiB data, 166 MiB used, 536 GiB / 536 GiB avail 2024-09-18T08:01:39.481 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:01:38 smithi007 ceph-mon[29459]: pgmap v139: 1 pgs: 1 active+clean; 577 KiB data, 166 MiB used, 536 GiB / 536 GiB avail 2024-09-18T08:01:39.481 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:01:38 smithi007 ceph-mon[39249]: pgmap v139: 1 pgs: 1 active+clean; 577 KiB data, 166 MiB used, 536 GiB / 536 GiB avail 2024-09-18T08:01:40.311 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:01:39 smithi050 ceph-mon[32371]: osdmap e41: 6 total, 6 up, 6 in 2024-09-18T08:01:40.311 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:01:39 smithi050 ceph-mon[32371]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' 2024-09-18T08:01:40.311 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:01:39 smithi050 ceph-mon[32371]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' 2024-09-18T08:01:40.311 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:01:39 smithi050 ceph-mon[32371]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"} : dispatch 2024-09-18T08:01:40.312 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:01:39 smithi050 ceph-mon[32371]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"} : dispatch 2024-09-18T08:01:40.312 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:01:39 smithi050 ceph-mon[32371]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' 2024-09-18T08:01:40.312 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:01:39 smithi050 ceph-mon[32371]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-18T08:01:40.312 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:01:39 smithi050 ceph-mon[32371]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-18T08:01:40.312 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:01:39 smithi050 ceph-mon[32371]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' 2024-09-18T08:01:40.481 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:01:39 smithi007 ceph-mon[29459]: osdmap e41: 6 total, 6 up, 6 in 2024-09-18T08:01:40.481 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:01:39 smithi007 ceph-mon[29459]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' 2024-09-18T08:01:40.481 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:01:39 smithi007 ceph-mon[29459]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' 2024-09-18T08:01:40.481 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:01:39 smithi007 ceph-mon[29459]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"} : dispatch 2024-09-18T08:01:40.481 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:01:39 smithi007 ceph-mon[29459]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"} : dispatch 2024-09-18T08:01:40.481 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:01:39 smithi007 ceph-mon[29459]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' 2024-09-18T08:01:40.481 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:01:39 smithi007 ceph-mon[29459]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-18T08:01:40.481 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:01:39 smithi007 ceph-mon[29459]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-18T08:01:40.481 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:01:39 smithi007 ceph-mon[29459]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' 2024-09-18T08:01:40.482 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:01:39 smithi007 ceph-mon[39249]: osdmap e41: 6 total, 6 up, 6 in 2024-09-18T08:01:40.482 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:01:39 smithi007 ceph-mon[39249]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' 2024-09-18T08:01:40.482 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:01:39 smithi007 ceph-mon[39249]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' 2024-09-18T08:01:40.482 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:01:39 smithi007 ceph-mon[39249]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"} : dispatch 2024-09-18T08:01:40.482 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:01:39 smithi007 ceph-mon[39249]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"} : dispatch 2024-09-18T08:01:40.482 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:01:39 smithi007 ceph-mon[39249]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' 2024-09-18T08:01:40.482 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:01:39 smithi007 ceph-mon[39249]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-18T08:01:40.482 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:01:39 smithi007 ceph-mon[39249]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-18T08:01:40.482 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:01:39 smithi007 ceph-mon[39249]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' 2024-09-18T08:01:41.373 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:01:41 smithi050 ceph-mon[32371]: pgmap v142: 1 pgs: 1 active+clean; 577 KiB data, 166 MiB used, 536 GiB / 536 GiB avail 2024-09-18T08:01:41.373 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:01:41 smithi050 ceph-mon[32371]: Detected new or changed devices on smithi050 2024-09-18T08:01:41.373 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:01:41 smithi050 ceph-mon[32371]: Adjusting osd_memory_target on smithi050 to 8546M 2024-09-18T08:01:41.373 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:01:41 smithi050 ceph-mon[32371]: purged_snaps scrub starts 2024-09-18T08:01:41.374 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:01:41 smithi050 ceph-mon[32371]: purged_snaps scrub ok 2024-09-18T08:01:41.376 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-09-18T08:01:41.405 DEBUG:teuthology.orchestra.run.smithi050:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 779349ca-7593-11ef-bceb-c7b262605968 -- ceph orch daemon add osd smithi050:vg_nvme/lv_2 2024-09-18T08:01:41.480 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:01:41 smithi007 ceph-mon[29459]: pgmap v142: 1 pgs: 1 active+clean; 577 KiB data, 166 MiB used, 536 GiB / 536 GiB avail 2024-09-18T08:01:41.481 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:01:41 smithi007 ceph-mon[29459]: Detected new or changed devices on smithi050 2024-09-18T08:01:41.481 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:01:41 smithi007 ceph-mon[29459]: Adjusting osd_memory_target on smithi050 to 8546M 2024-09-18T08:01:41.481 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:01:41 smithi007 ceph-mon[29459]: purged_snaps scrub starts 2024-09-18T08:01:41.481 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:01:41 smithi007 ceph-mon[29459]: purged_snaps scrub ok 2024-09-18T08:01:41.481 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:01:41 smithi007 ceph-mon[39249]: pgmap v142: 1 pgs: 1 active+clean; 577 KiB data, 166 MiB used, 536 GiB / 536 GiB avail 2024-09-18T08:01:41.481 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:01:41 smithi007 ceph-mon[39249]: Detected new or changed devices on smithi050 2024-09-18T08:01:41.481 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:01:41 smithi007 ceph-mon[39249]: Adjusting osd_memory_target on smithi050 to 8546M 2024-09-18T08:01:41.481 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:01:41 smithi007 ceph-mon[39249]: purged_snaps scrub starts 2024-09-18T08:01:41.481 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:01:41 smithi007 ceph-mon[39249]: purged_snaps scrub ok 2024-09-18T08:01:41.667 INFO:teuthology.orchestra.run.smithi050.stderr:Inferring config /var/lib/ceph/779349ca-7593-11ef-bceb-c7b262605968/mon.b/config 2024-09-18T08:01:42.406 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:01:42 smithi050 ceph-mon[32371]: osdmap e42: 6 total, 6 up, 6 in 2024-09-18T08:01:42.481 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:01:42 smithi007 ceph-mon[29459]: osdmap e42: 6 total, 6 up, 6 in 2024-09-18T08:01:42.481 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:01:42 smithi007 ceph-mon[39249]: osdmap e42: 6 total, 6 up, 6 in 2024-09-18T08:01:43.406 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:01:43 smithi050 ceph-mon[32371]: pgmap v144: 1 pgs: 1 active+clean; 577 KiB data, 166 MiB used, 536 GiB / 536 GiB avail 2024-09-18T08:01:43.406 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:01:43 smithi050 ceph-mon[32371]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' cmd={"prefix": "osd tree", "states": ["destroyed"], "format": "json"} : dispatch 2024-09-18T08:01:43.406 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:01:43 smithi050 ceph-mon[32371]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.bootstrap-osd"} : dispatch 2024-09-18T08:01:43.406 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:01:43 smithi050 ceph-mon[32371]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-18T08:01:43.480 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:01:43 smithi007 ceph-mon[29459]: pgmap v144: 1 pgs: 1 active+clean; 577 KiB data, 166 MiB used, 536 GiB / 536 GiB avail 2024-09-18T08:01:43.481 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:01:43 smithi007 ceph-mon[29459]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' cmd={"prefix": "osd tree", "states": ["destroyed"], "format": "json"} : dispatch 2024-09-18T08:01:43.481 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:01:43 smithi007 ceph-mon[29459]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.bootstrap-osd"} : dispatch 2024-09-18T08:01:43.481 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:01:43 smithi007 ceph-mon[29459]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-18T08:01:43.481 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:01:43 smithi007 ceph-mon[39249]: pgmap v144: 1 pgs: 1 active+clean; 577 KiB data, 166 MiB used, 536 GiB / 536 GiB avail 2024-09-18T08:01:43.481 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:01:43 smithi007 ceph-mon[39249]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' cmd={"prefix": "osd tree", "states": ["destroyed"], "format": "json"} : dispatch 2024-09-18T08:01:43.481 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:01:43 smithi007 ceph-mon[39249]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.bootstrap-osd"} : dispatch 2024-09-18T08:01:43.481 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:01:43 smithi007 ceph-mon[39249]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-18T08:01:44.062 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:01:44 smithi050 ceph-mon[32371]: from='client.24250 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi050:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-09-18T08:01:44.480 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:01:44 smithi007 ceph-mon[29459]: from='client.24250 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi050:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-09-18T08:01:44.481 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:01:44 smithi007 ceph-mon[39249]: from='client.24250 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi050:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-09-18T08:01:45.335 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:01:45 smithi050 ceph-mon[32371]: pgmap v145: 1 pgs: 1 active+clean; 577 KiB data, 166 MiB used, 536 GiB / 536 GiB avail 2024-09-18T08:01:45.335 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:01:45 smithi050 ceph-mon[32371]: from='client.? 172.21.15.50:0/2722500908' entity='client.bootstrap-osd' cmd={"prefix": "osd new", "uuid": "e07d846c-ae42-4fea-aa5c-3a25835d18a6"} : dispatch 2024-09-18T08:01:45.335 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:01:45 smithi050 ceph-mon[32371]: from='client.? ' entity='client.bootstrap-osd' cmd={"prefix": "osd new", "uuid": "e07d846c-ae42-4fea-aa5c-3a25835d18a6"} : dispatch 2024-09-18T08:01:45.335 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:01:45 smithi050 ceph-mon[32371]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "e07d846c-ae42-4fea-aa5c-3a25835d18a6"}]': finished 2024-09-18T08:01:45.335 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:01:45 smithi050 ceph-mon[32371]: osdmap e43: 7 total, 6 up, 7 in 2024-09-18T08:01:45.335 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:01:45 smithi050 ceph-mon[32371]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 6} : dispatch 2024-09-18T08:01:45.480 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:01:45 smithi007 ceph-mon[29459]: pgmap v145: 1 pgs: 1 active+clean; 577 KiB data, 166 MiB used, 536 GiB / 536 GiB avail 2024-09-18T08:01:45.481 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:01:45 smithi007 ceph-mon[29459]: from='client.? 172.21.15.50:0/2722500908' entity='client.bootstrap-osd' cmd={"prefix": "osd new", "uuid": "e07d846c-ae42-4fea-aa5c-3a25835d18a6"} : dispatch 2024-09-18T08:01:45.481 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:01:45 smithi007 ceph-mon[29459]: from='client.? ' entity='client.bootstrap-osd' cmd={"prefix": "osd new", "uuid": "e07d846c-ae42-4fea-aa5c-3a25835d18a6"} : dispatch 2024-09-18T08:01:45.481 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:01:45 smithi007 ceph-mon[29459]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "e07d846c-ae42-4fea-aa5c-3a25835d18a6"}]': finished 2024-09-18T08:01:45.481 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:01:45 smithi007 ceph-mon[29459]: osdmap e43: 7 total, 6 up, 7 in 2024-09-18T08:01:45.481 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:01:45 smithi007 ceph-mon[29459]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 6} : dispatch 2024-09-18T08:01:45.481 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:01:45 smithi007 ceph-mon[39249]: pgmap v145: 1 pgs: 1 active+clean; 577 KiB data, 166 MiB used, 536 GiB / 536 GiB avail 2024-09-18T08:01:45.481 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:01:45 smithi007 ceph-mon[39249]: from='client.? 172.21.15.50:0/2722500908' entity='client.bootstrap-osd' cmd={"prefix": "osd new", "uuid": "e07d846c-ae42-4fea-aa5c-3a25835d18a6"} : dispatch 2024-09-18T08:01:45.481 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:01:45 smithi007 ceph-mon[39249]: from='client.? ' entity='client.bootstrap-osd' cmd={"prefix": "osd new", "uuid": "e07d846c-ae42-4fea-aa5c-3a25835d18a6"} : dispatch 2024-09-18T08:01:45.481 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:01:45 smithi007 ceph-mon[39249]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "e07d846c-ae42-4fea-aa5c-3a25835d18a6"}]': finished 2024-09-18T08:01:45.482 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:01:45 smithi007 ceph-mon[39249]: osdmap e43: 7 total, 6 up, 7 in 2024-09-18T08:01:45.482 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:01:45 smithi007 ceph-mon[39249]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 6} : dispatch 2024-09-18T08:01:46.406 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:01:46 smithi050 ceph-mon[32371]: from='client.? 172.21.15.50:0/3943259447' entity='client.bootstrap-osd' cmd={"prefix": "mon getmap"} : dispatch 2024-09-18T08:01:46.481 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:01:46 smithi007 ceph-mon[29459]: from='client.? 172.21.15.50:0/3943259447' entity='client.bootstrap-osd' cmd={"prefix": "mon getmap"} : dispatch 2024-09-18T08:01:46.481 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:01:46 smithi007 ceph-mon[39249]: from='client.? 172.21.15.50:0/3943259447' entity='client.bootstrap-osd' cmd={"prefix": "mon getmap"} : dispatch 2024-09-18T08:01:47.406 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:01:47 smithi050 ceph-mon[32371]: pgmap v147: 1 pgs: 1 active+clean; 577 KiB data, 166 MiB used, 536 GiB / 536 GiB avail 2024-09-18T08:01:47.480 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:01:47 smithi007 ceph-mon[29459]: pgmap v147: 1 pgs: 1 active+clean; 577 KiB data, 166 MiB used, 536 GiB / 536 GiB avail 2024-09-18T08:01:47.481 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:01:47 smithi007 ceph-mon[39249]: pgmap v147: 1 pgs: 1 active+clean; 577 KiB data, 166 MiB used, 536 GiB / 536 GiB avail 2024-09-18T08:01:49.192 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:01:49 smithi050 ceph-mon[32371]: pgmap v148: 1 pgs: 1 active+clean; 577 KiB data, 166 MiB used, 536 GiB / 536 GiB avail 2024-09-18T08:01:49.481 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:01:49 smithi007 ceph-mon[29459]: pgmap v148: 1 pgs: 1 active+clean; 577 KiB data, 166 MiB used, 536 GiB / 536 GiB avail 2024-09-18T08:01:49.481 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:01:49 smithi007 ceph-mon[39249]: pgmap v148: 1 pgs: 1 active+clean; 577 KiB data, 166 MiB used, 536 GiB / 536 GiB avail 2024-09-18T08:01:51.222 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:01:51 smithi050 ceph-mon[32371]: pgmap v149: 1 pgs: 1 active+clean; 577 KiB data, 166 MiB used, 536 GiB / 536 GiB avail 2024-09-18T08:01:51.480 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:01:51 smithi007 ceph-mon[29459]: pgmap v149: 1 pgs: 1 active+clean; 577 KiB data, 166 MiB used, 536 GiB / 536 GiB avail 2024-09-18T08:01:51.481 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:01:51 smithi007 ceph-mon[39249]: pgmap v149: 1 pgs: 1 active+clean; 577 KiB data, 166 MiB used, 536 GiB / 536 GiB avail 2024-09-18T08:01:53.406 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:01:53 smithi050 ceph-mon[32371]: pgmap v150: 1 pgs: 1 active+clean; 577 KiB data, 166 MiB used, 536 GiB / 536 GiB avail 2024-09-18T08:01:53.406 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:01:53 smithi050 ceph-mon[32371]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' cmd={"prefix": "auth get", "entity": "osd.6"} : dispatch 2024-09-18T08:01:53.406 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:01:53 smithi050 ceph-mon[32371]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-18T08:01:53.406 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:01:53 smithi050 ceph-mon[32371]: Deploying daemon osd.6 on smithi050 2024-09-18T08:01:53.481 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:01:53 smithi007 ceph-mon[29459]: pgmap v150: 1 pgs: 1 active+clean; 577 KiB data, 166 MiB used, 536 GiB / 536 GiB avail 2024-09-18T08:01:53.481 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:01:53 smithi007 ceph-mon[29459]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' cmd={"prefix": "auth get", "entity": "osd.6"} : dispatch 2024-09-18T08:01:53.481 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:01:53 smithi007 ceph-mon[29459]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-18T08:01:53.481 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:01:53 smithi007 ceph-mon[29459]: Deploying daemon osd.6 on smithi050 2024-09-18T08:01:53.481 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:01:53 smithi007 ceph-mon[39249]: pgmap v150: 1 pgs: 1 active+clean; 577 KiB data, 166 MiB used, 536 GiB / 536 GiB avail 2024-09-18T08:01:53.481 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:01:53 smithi007 ceph-mon[39249]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' cmd={"prefix": "auth get", "entity": "osd.6"} : dispatch 2024-09-18T08:01:53.481 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:01:53 smithi007 ceph-mon[39249]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-18T08:01:53.482 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:01:53 smithi007 ceph-mon[39249]: Deploying daemon osd.6 on smithi050 2024-09-18T08:01:55.381 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:01:55 smithi050 ceph-mon[32371]: pgmap v151: 1 pgs: 1 active+clean; 577 KiB data, 166 MiB used, 536 GiB / 536 GiB avail 2024-09-18T08:01:55.480 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:01:55 smithi007 ceph-mon[29459]: pgmap v151: 1 pgs: 1 active+clean; 577 KiB data, 166 MiB used, 536 GiB / 536 GiB avail 2024-09-18T08:01:55.481 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:01:55 smithi007 ceph-mon[39249]: pgmap v151: 1 pgs: 1 active+clean; 577 KiB data, 166 MiB used, 536 GiB / 536 GiB avail 2024-09-18T08:01:57.406 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:01:57 smithi050 ceph-mon[32371]: pgmap v152: 1 pgs: 1 active+clean; 577 KiB data, 166 MiB used, 536 GiB / 536 GiB avail 2024-09-18T08:01:57.481 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:01:57 smithi007 ceph-mon[29459]: pgmap v152: 1 pgs: 1 active+clean; 577 KiB data, 166 MiB used, 536 GiB / 536 GiB avail 2024-09-18T08:01:57.481 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:01:57 smithi007 ceph-mon[39249]: pgmap v152: 1 pgs: 1 active+clean; 577 KiB data, 166 MiB used, 536 GiB / 536 GiB avail 2024-09-18T08:01:58.362 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:01:58 smithi050 ceph-mon[32371]: purged_snaps scrub starts 2024-09-18T08:01:58.362 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:01:58 smithi050 ceph-mon[32371]: purged_snaps scrub ok 2024-09-18T08:01:58.480 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:01:58 smithi007 ceph-mon[29459]: purged_snaps scrub starts 2024-09-18T08:01:58.481 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:01:58 smithi007 ceph-mon[29459]: purged_snaps scrub ok 2024-09-18T08:01:58.481 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:01:58 smithi007 ceph-mon[39249]: purged_snaps scrub starts 2024-09-18T08:01:58.481 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:01:58 smithi007 ceph-mon[39249]: purged_snaps scrub ok 2024-09-18T08:01:59.398 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:01:59 smithi050 ceph-mon[32371]: pgmap v153: 1 pgs: 1 active+clean; 577 KiB data, 166 MiB used, 536 GiB / 536 GiB avail 2024-09-18T08:01:59.398 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:01:59 smithi050 ceph-mon[32371]: osdmap e44: 7 total, 6 up, 7 in 2024-09-18T08:01:59.398 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:01:59 smithi050 ceph-mon[32371]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 6} : dispatch 2024-09-18T08:01:59.480 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:01:59 smithi007 ceph-mon[29459]: pgmap v153: 1 pgs: 1 active+clean; 577 KiB data, 166 MiB used, 536 GiB / 536 GiB avail 2024-09-18T08:01:59.481 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:01:59 smithi007 ceph-mon[29459]: osdmap e44: 7 total, 6 up, 7 in 2024-09-18T08:01:59.481 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:01:59 smithi007 ceph-mon[29459]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 6} : dispatch 2024-09-18T08:01:59.481 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:01:59 smithi007 ceph-mon[39249]: pgmap v153: 1 pgs: 1 active+clean; 577 KiB data, 166 MiB used, 536 GiB / 536 GiB avail 2024-09-18T08:01:59.481 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:01:59 smithi007 ceph-mon[39249]: osdmap e44: 7 total, 6 up, 7 in 2024-09-18T08:01:59.481 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:01:59 smithi007 ceph-mon[39249]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 6} : dispatch 2024-09-18T08:02:00.285 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:02:00 smithi050 ceph-mon[32371]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-18T08:02:00.285 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:02:00 smithi050 ceph-mon[32371]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' 2024-09-18T08:02:00.285 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:02:00 smithi050 ceph-mon[32371]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' 2024-09-18T08:02:00.481 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:02:00 smithi007 ceph-mon[29459]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-18T08:02:00.481 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:02:00 smithi007 ceph-mon[29459]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' 2024-09-18T08:02:00.481 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:02:00 smithi007 ceph-mon[29459]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' 2024-09-18T08:02:00.481 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:02:00 smithi007 ceph-mon[39249]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-18T08:02:00.481 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:02:00 smithi007 ceph-mon[39249]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' 2024-09-18T08:02:00.481 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:02:00 smithi007 ceph-mon[39249]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' 2024-09-18T08:02:01.156 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:02:01 smithi050 ceph-mon[32371]: pgmap v155: 1 pgs: 1 active+clean; 577 KiB data, 166 MiB used, 536 GiB / 536 GiB avail 2024-09-18T08:02:01.480 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:02:01 smithi007 ceph-mon[29459]: pgmap v155: 1 pgs: 1 active+clean; 577 KiB data, 166 MiB used, 536 GiB / 536 GiB avail 2024-09-18T08:02:01.481 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:02:01 smithi007 ceph-mon[39249]: pgmap v155: 1 pgs: 1 active+clean; 577 KiB data, 166 MiB used, 536 GiB / 536 GiB avail 2024-09-18T08:02:03.406 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:02:03 smithi050 ceph-mon[32371]: pgmap v156: 1 pgs: 1 active+clean; 577 KiB data, 166 MiB used, 536 GiB / 536 GiB avail 2024-09-18T08:02:03.406 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:02:03 smithi050 ceph-mon[32371]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' 2024-09-18T08:02:03.406 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:02:03 smithi050 ceph-mon[32371]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' 2024-09-18T08:02:03.407 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:02:03 smithi050 ceph-mon[32371]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-18T08:02:03.407 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:02:03 smithi050 ceph-mon[32371]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-18T08:02:03.407 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:02:03 smithi050 ceph-mon[32371]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' 2024-09-18T08:02:03.481 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:02:03 smithi007 ceph-mon[29459]: pgmap v156: 1 pgs: 1 active+clean; 577 KiB data, 166 MiB used, 536 GiB / 536 GiB avail 2024-09-18T08:02:03.481 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:02:03 smithi007 ceph-mon[29459]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' 2024-09-18T08:02:03.481 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:02:03 smithi007 ceph-mon[29459]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' 2024-09-18T08:02:03.481 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:02:03 smithi007 ceph-mon[29459]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-18T08:02:03.481 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:02:03 smithi007 ceph-mon[29459]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-18T08:02:03.481 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:02:03 smithi007 ceph-mon[29459]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' 2024-09-18T08:02:03.482 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:02:03 smithi007 ceph-mon[39249]: pgmap v156: 1 pgs: 1 active+clean; 577 KiB data, 166 MiB used, 536 GiB / 536 GiB avail 2024-09-18T08:02:03.482 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:02:03 smithi007 ceph-mon[39249]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' 2024-09-18T08:02:03.482 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:02:03 smithi007 ceph-mon[39249]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' 2024-09-18T08:02:03.482 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:02:03 smithi007 ceph-mon[39249]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-18T08:02:03.482 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:02:03 smithi007 ceph-mon[39249]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-18T08:02:03.482 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:02:03 smithi007 ceph-mon[39249]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' 2024-09-18T08:02:05.157 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:02:05 smithi050 ceph-mon[32371]: pgmap v157: 1 pgs: 1 active+clean; 577 KiB data, 166 MiB used, 536 GiB / 536 GiB avail 2024-09-18T08:02:05.157 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:02:05 smithi050 ceph-mon[32371]: from='osd.6 v2:172.21.15.50:6808/255960422' entity='osd.6' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]} : dispatch 2024-09-18T08:02:05.157 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:02:05 smithi050 ceph-mon[32371]: from='osd.6 ' entity='osd.6' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]} : dispatch 2024-09-18T08:02:05.248 INFO:teuthology.orchestra.run.smithi050.stdout:Created osd(s) 6 on host 'smithi050' 2024-09-18T08:02:05.481 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:02:05 smithi007 ceph-mon[29459]: pgmap v157: 1 pgs: 1 active+clean; 577 KiB data, 166 MiB used, 536 GiB / 536 GiB avail 2024-09-18T08:02:05.481 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:02:05 smithi007 ceph-mon[29459]: from='osd.6 v2:172.21.15.50:6808/255960422' entity='osd.6' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]} : dispatch 2024-09-18T08:02:05.481 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:02:05 smithi007 ceph-mon[29459]: from='osd.6 ' entity='osd.6' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]} : dispatch 2024-09-18T08:02:05.481 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:02:05 smithi007 ceph-mon[39249]: pgmap v157: 1 pgs: 1 active+clean; 577 KiB data, 166 MiB used, 536 GiB / 536 GiB avail 2024-09-18T08:02:05.481 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:02:05 smithi007 ceph-mon[39249]: from='osd.6 v2:172.21.15.50:6808/255960422' entity='osd.6' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]} : dispatch 2024-09-18T08:02:05.481 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:02:05 smithi007 ceph-mon[39249]: from='osd.6 ' entity='osd.6' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]} : dispatch 2024-09-18T08:02:05.906 DEBUG:teuthology.orchestra.run.smithi050:osd.6> sudo journalctl -f -n 0 -u ceph-779349ca-7593-11ef-bceb-c7b262605968@osd.6.service 2024-09-18T08:02:05.909 INFO:tasks.cephadm:Deploying osd.7 on smithi050 with /dev/vg_nvme/lv_1... 2024-09-18T08:02:05.909 DEBUG:teuthology.orchestra.run.smithi050:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e ceph-volume -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 779349ca-7593-11ef-bceb-c7b262605968 -- lvm zap /dev/vg_nvme/lv_1 2024-09-18T08:02:06.156 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:02:06 smithi050 ceph-mon[32371]: from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2024-09-18T08:02:06.156 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:02:06 smithi050 ceph-mon[32371]: from='osd.6 v2:172.21.15.50:6808/255960422' entity='osd.6' cmd={"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi050", "root=default"]} : dispatch 2024-09-18T08:02:06.156 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:02:06 smithi050 ceph-mon[32371]: osdmap e45: 7 total, 6 up, 7 in 2024-09-18T08:02:06.156 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:02:06 smithi050 ceph-mon[32371]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 6} : dispatch 2024-09-18T08:02:06.157 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:02:06 smithi050 ceph-mon[32371]: from='osd.6 ' entity='osd.6' cmd={"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi050", "root=default"]} : dispatch 2024-09-18T08:02:06.157 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:02:06 smithi050 ceph-mon[32371]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-18T08:02:06.157 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:02:06 smithi050 ceph-mon[32371]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' 2024-09-18T08:02:06.157 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:02:06 smithi050 ceph-mon[32371]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' 2024-09-18T08:02:06.157 INFO:journalctl@ceph.osd.6.smithi050.stdout:Sep 18 08:02:06 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd-6[47568]: 2024-09-18T08:02:06.089+0000 7f0a594f2640 -1 osd.6 0 waiting for initial osdmap 2024-09-18T08:02:06.157 INFO:journalctl@ceph.osd.6.smithi050.stdout:Sep 18 08:02:06 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd-6[47568]: 2024-09-18T08:02:06.096+0000 7f0a55319640 -1 osd.6 46 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-18T08:02:06.481 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:02:06 smithi007 ceph-mon[29459]: from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2024-09-18T08:02:06.481 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:02:06 smithi007 ceph-mon[29459]: from='osd.6 v2:172.21.15.50:6808/255960422' entity='osd.6' cmd={"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi050", "root=default"]} : dispatch 2024-09-18T08:02:06.481 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:02:06 smithi007 ceph-mon[29459]: osdmap e45: 7 total, 6 up, 7 in 2024-09-18T08:02:06.481 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:02:06 smithi007 ceph-mon[29459]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 6} : dispatch 2024-09-18T08:02:06.481 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:02:06 smithi007 ceph-mon[29459]: from='osd.6 ' entity='osd.6' cmd={"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi050", "root=default"]} : dispatch 2024-09-18T08:02:06.481 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:02:06 smithi007 ceph-mon[29459]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-18T08:02:06.481 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:02:06 smithi007 ceph-mon[29459]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' 2024-09-18T08:02:06.481 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:02:06 smithi007 ceph-mon[29459]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' 2024-09-18T08:02:06.481 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:02:06 smithi007 ceph-mon[39249]: from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2024-09-18T08:02:06.482 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:02:06 smithi007 ceph-mon[39249]: from='osd.6 v2:172.21.15.50:6808/255960422' entity='osd.6' cmd={"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi050", "root=default"]} : dispatch 2024-09-18T08:02:06.482 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:02:06 smithi007 ceph-mon[39249]: osdmap e45: 7 total, 6 up, 7 in 2024-09-18T08:02:06.482 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:02:06 smithi007 ceph-mon[39249]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 6} : dispatch 2024-09-18T08:02:06.482 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:02:06 smithi007 ceph-mon[39249]: from='osd.6 ' entity='osd.6' cmd={"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi050", "root=default"]} : dispatch 2024-09-18T08:02:06.482 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:02:06 smithi007 ceph-mon[39249]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-18T08:02:06.482 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:02:06 smithi007 ceph-mon[39249]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' 2024-09-18T08:02:06.482 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:02:06 smithi007 ceph-mon[39249]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' 2024-09-18T08:02:07.061 INFO:teuthology.orchestra.run.smithi050.stderr:Inferring config /var/lib/ceph/779349ca-7593-11ef-bceb-c7b262605968/mon.b/config 2024-09-18T08:02:07.343 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:02:07 smithi050 ceph-mon[32371]: pgmap v159: 1 pgs: 1 active+clean; 577 KiB data, 166 MiB used, 536 GiB / 536 GiB avail 2024-09-18T08:02:07.343 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:02:07 smithi050 ceph-mon[32371]: from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi050", "root=default"]}]': finished 2024-09-18T08:02:07.343 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:02:07 smithi050 ceph-mon[32371]: osdmap e46: 7 total, 6 up, 7 in 2024-09-18T08:02:07.344 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:02:07 smithi050 ceph-mon[32371]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 6} : dispatch 2024-09-18T08:02:07.344 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:02:07 smithi050 ceph-mon[32371]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 6} : dispatch 2024-09-18T08:02:07.481 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:02:07 smithi007 ceph-mon[29459]: pgmap v159: 1 pgs: 1 active+clean; 577 KiB data, 166 MiB used, 536 GiB / 536 GiB avail 2024-09-18T08:02:07.481 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:02:07 smithi007 ceph-mon[29459]: from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi050", "root=default"]}]': finished 2024-09-18T08:02:07.481 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:02:07 smithi007 ceph-mon[29459]: osdmap e46: 7 total, 6 up, 7 in 2024-09-18T08:02:07.481 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:02:07 smithi007 ceph-mon[29459]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 6} : dispatch 2024-09-18T08:02:07.481 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:02:07 smithi007 ceph-mon[29459]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 6} : dispatch 2024-09-18T08:02:07.481 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:02:07 smithi007 ceph-mon[39249]: pgmap v159: 1 pgs: 1 active+clean; 577 KiB data, 166 MiB used, 536 GiB / 536 GiB avail 2024-09-18T08:02:07.481 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:02:07 smithi007 ceph-mon[39249]: from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi050", "root=default"]}]': finished 2024-09-18T08:02:07.482 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:02:07 smithi007 ceph-mon[39249]: osdmap e46: 7 total, 6 up, 7 in 2024-09-18T08:02:07.482 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:02:07 smithi007 ceph-mon[39249]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 6} : dispatch 2024-09-18T08:02:07.482 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:02:07 smithi007 ceph-mon[39249]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 6} : dispatch 2024-09-18T08:02:08.397 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:02:08 smithi050 ceph-mon[32371]: purged_snaps scrub starts 2024-09-18T08:02:08.397 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:02:08 smithi050 ceph-mon[32371]: purged_snaps scrub ok 2024-09-18T08:02:08.397 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:02:08 smithi050 ceph-mon[32371]: osd.6 v2:172.21.15.50:6808/255960422 boot 2024-09-18T08:02:08.397 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:02:08 smithi050 ceph-mon[32371]: osdmap e47: 7 total, 7 up, 7 in 2024-09-18T08:02:08.397 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:02:08 smithi050 ceph-mon[32371]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 6} : dispatch 2024-09-18T08:02:08.481 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:02:08 smithi007 ceph-mon[29459]: purged_snaps scrub starts 2024-09-18T08:02:08.481 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:02:08 smithi007 ceph-mon[29459]: purged_snaps scrub ok 2024-09-18T08:02:08.481 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:02:08 smithi007 ceph-mon[29459]: osd.6 v2:172.21.15.50:6808/255960422 boot 2024-09-18T08:02:08.481 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:02:08 smithi007 ceph-mon[29459]: osdmap e47: 7 total, 7 up, 7 in 2024-09-18T08:02:08.481 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:02:08 smithi007 ceph-mon[29459]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 6} : dispatch 2024-09-18T08:02:08.481 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:02:08 smithi007 ceph-mon[39249]: purged_snaps scrub starts 2024-09-18T08:02:08.482 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:02:08 smithi007 ceph-mon[39249]: purged_snaps scrub ok 2024-09-18T08:02:08.482 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:02:08 smithi007 ceph-mon[39249]: osd.6 v2:172.21.15.50:6808/255960422 boot 2024-09-18T08:02:08.482 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:02:08 smithi007 ceph-mon[39249]: osdmap e47: 7 total, 7 up, 7 in 2024-09-18T08:02:08.482 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:02:08 smithi007 ceph-mon[39249]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 6} : dispatch 2024-09-18T08:02:09.406 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:02:09 smithi050 ceph-mon[32371]: pgmap v162: 1 pgs: 1 active+clean; 577 KiB data, 166 MiB used, 536 GiB / 536 GiB avail 2024-09-18T08:02:09.406 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:02:09 smithi050 ceph-mon[32371]: osdmap e48: 7 total, 7 up, 7 in 2024-09-18T08:02:09.480 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:02:09 smithi007 ceph-mon[29459]: pgmap v162: 1 pgs: 1 active+clean; 577 KiB data, 166 MiB used, 536 GiB / 536 GiB avail 2024-09-18T08:02:09.481 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:02:09 smithi007 ceph-mon[29459]: osdmap e48: 7 total, 7 up, 7 in 2024-09-18T08:02:09.481 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:02:09 smithi007 ceph-mon[39249]: pgmap v162: 1 pgs: 1 active+clean; 577 KiB data, 166 MiB used, 536 GiB / 536 GiB avail 2024-09-18T08:02:09.481 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:02:09 smithi007 ceph-mon[39249]: osdmap e48: 7 total, 7 up, 7 in 2024-09-18T08:02:10.382 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:02:10 smithi050 ceph-mon[32371]: osdmap e49: 7 total, 7 up, 7 in 2024-09-18T08:02:10.382 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:02:10 smithi050 ceph-mon[32371]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' 2024-09-18T08:02:10.382 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:02:10 smithi050 ceph-mon[32371]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' 2024-09-18T08:02:10.382 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:02:10 smithi050 ceph-mon[32371]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"} : dispatch 2024-09-18T08:02:10.382 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:02:10 smithi050 ceph-mon[32371]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"} : dispatch 2024-09-18T08:02:10.382 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:02:10 smithi050 ceph-mon[32371]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"} : dispatch 2024-09-18T08:02:10.383 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:02:10 smithi050 ceph-mon[32371]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' 2024-09-18T08:02:10.383 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:02:10 smithi050 ceph-mon[32371]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-18T08:02:10.383 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:02:10 smithi050 ceph-mon[32371]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-18T08:02:10.383 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:02:10 smithi050 ceph-mon[32371]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' 2024-09-18T08:02:10.481 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:02:10 smithi007 ceph-mon[29459]: osdmap e49: 7 total, 7 up, 7 in 2024-09-18T08:02:10.481 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:02:10 smithi007 ceph-mon[29459]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' 2024-09-18T08:02:10.481 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:02:10 smithi007 ceph-mon[29459]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' 2024-09-18T08:02:10.481 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:02:10 smithi007 ceph-mon[29459]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"} : dispatch 2024-09-18T08:02:10.481 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:02:10 smithi007 ceph-mon[29459]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"} : dispatch 2024-09-18T08:02:10.481 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:02:10 smithi007 ceph-mon[29459]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"} : dispatch 2024-09-18T08:02:10.481 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:02:10 smithi007 ceph-mon[29459]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' 2024-09-18T08:02:10.481 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:02:10 smithi007 ceph-mon[29459]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-18T08:02:10.481 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:02:10 smithi007 ceph-mon[29459]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-18T08:02:10.482 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:02:10 smithi007 ceph-mon[29459]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' 2024-09-18T08:02:10.482 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:02:10 smithi007 ceph-mon[39249]: osdmap e49: 7 total, 7 up, 7 in 2024-09-18T08:02:10.482 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:02:10 smithi007 ceph-mon[39249]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' 2024-09-18T08:02:10.482 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:02:10 smithi007 ceph-mon[39249]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' 2024-09-18T08:02:10.482 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:02:10 smithi007 ceph-mon[39249]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"} : dispatch 2024-09-18T08:02:10.482 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:02:10 smithi007 ceph-mon[39249]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"} : dispatch 2024-09-18T08:02:10.482 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:02:10 smithi007 ceph-mon[39249]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"} : dispatch 2024-09-18T08:02:10.482 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:02:10 smithi007 ceph-mon[39249]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' 2024-09-18T08:02:10.482 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:02:10 smithi007 ceph-mon[39249]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-18T08:02:10.482 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:02:10 smithi007 ceph-mon[39249]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-18T08:02:10.482 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:02:10 smithi007 ceph-mon[39249]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' 2024-09-18T08:02:11.140 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:02:11 smithi050 ceph-mon[32371]: pgmap v165: 1 pgs: 1 active+clean; 577 KiB data, 193 MiB used, 626 GiB / 626 GiB avail 2024-09-18T08:02:11.140 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:02:11 smithi050 ceph-mon[32371]: Detected new or changed devices on smithi050 2024-09-18T08:02:11.140 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:02:11 smithi050 ceph-mon[32371]: Adjusting osd_memory_target on smithi050 to 5697M 2024-09-18T08:02:11.143 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-09-18T08:02:11.170 DEBUG:teuthology.orchestra.run.smithi050:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 779349ca-7593-11ef-bceb-c7b262605968 -- ceph orch daemon add osd smithi050:vg_nvme/lv_1 2024-09-18T08:02:11.439 INFO:teuthology.orchestra.run.smithi050.stderr:Inferring config /var/lib/ceph/779349ca-7593-11ef-bceb-c7b262605968/mon.b/config 2024-09-18T08:02:11.481 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:02:11 smithi007 ceph-mon[29459]: pgmap v165: 1 pgs: 1 active+clean; 577 KiB data, 193 MiB used, 626 GiB / 626 GiB avail 2024-09-18T08:02:11.481 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:02:11 smithi007 ceph-mon[29459]: Detected new or changed devices on smithi050 2024-09-18T08:02:11.481 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:02:11 smithi007 ceph-mon[29459]: Adjusting osd_memory_target on smithi050 to 5697M 2024-09-18T08:02:11.481 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:02:11 smithi007 ceph-mon[39249]: pgmap v165: 1 pgs: 1 active+clean; 577 KiB data, 193 MiB used, 626 GiB / 626 GiB avail 2024-09-18T08:02:11.481 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:02:11 smithi007 ceph-mon[39249]: Detected new or changed devices on smithi050 2024-09-18T08:02:11.481 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:02:11 smithi007 ceph-mon[39249]: Adjusting osd_memory_target on smithi050 to 5697M 2024-09-18T08:02:12.406 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:02:12 smithi050 ceph-mon[32371]: purged_snaps scrub starts 2024-09-18T08:02:12.406 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:02:12 smithi050 ceph-mon[32371]: purged_snaps scrub ok 2024-09-18T08:02:12.406 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:02:12 smithi050 ceph-mon[32371]: osdmap e50: 7 total, 7 up, 7 in 2024-09-18T08:02:12.406 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:02:12 smithi050 ceph-mon[32371]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' cmd={"prefix": "osd tree", "states": ["destroyed"], "format": "json"} : dispatch 2024-09-18T08:02:12.407 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:02:12 smithi050 ceph-mon[32371]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.bootstrap-osd"} : dispatch 2024-09-18T08:02:12.407 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:02:12 smithi050 ceph-mon[32371]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-18T08:02:12.481 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:02:12 smithi007 ceph-mon[29459]: purged_snaps scrub starts 2024-09-18T08:02:12.481 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:02:12 smithi007 ceph-mon[29459]: purged_snaps scrub ok 2024-09-18T08:02:12.481 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:02:12 smithi007 ceph-mon[29459]: osdmap e50: 7 total, 7 up, 7 in 2024-09-18T08:02:12.481 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:02:12 smithi007 ceph-mon[29459]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' cmd={"prefix": "osd tree", "states": ["destroyed"], "format": "json"} : dispatch 2024-09-18T08:02:12.481 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:02:12 smithi007 ceph-mon[29459]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.bootstrap-osd"} : dispatch 2024-09-18T08:02:12.481 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:02:12 smithi007 ceph-mon[29459]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-18T08:02:12.481 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:02:12 smithi007 ceph-mon[39249]: purged_snaps scrub starts 2024-09-18T08:02:12.481 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:02:12 smithi007 ceph-mon[39249]: purged_snaps scrub ok 2024-09-18T08:02:12.482 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:02:12 smithi007 ceph-mon[39249]: osdmap e50: 7 total, 7 up, 7 in 2024-09-18T08:02:12.482 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:02:12 smithi007 ceph-mon[39249]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' cmd={"prefix": "osd tree", "states": ["destroyed"], "format": "json"} : dispatch 2024-09-18T08:02:12.482 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:02:12 smithi007 ceph-mon[39249]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.bootstrap-osd"} : dispatch 2024-09-18T08:02:12.482 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:02:12 smithi007 ceph-mon[39249]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-18T08:02:13.284 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:02:13 smithi050 ceph-mon[32371]: pgmap v167: 1 pgs: 1 active+clean; 577 KiB data, 193 MiB used, 626 GiB / 626 GiB avail 2024-09-18T08:02:13.284 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:02:13 smithi050 ceph-mon[32371]: from='client.24277 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi050:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-09-18T08:02:13.481 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:02:13 smithi007 ceph-mon[29459]: pgmap v167: 1 pgs: 1 active+clean; 577 KiB data, 193 MiB used, 626 GiB / 626 GiB avail 2024-09-18T08:02:13.481 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:02:13 smithi007 ceph-mon[29459]: from='client.24277 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi050:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-09-18T08:02:13.481 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:02:13 smithi007 ceph-mon[39249]: pgmap v167: 1 pgs: 1 active+clean; 577 KiB data, 193 MiB used, 626 GiB / 626 GiB avail 2024-09-18T08:02:13.481 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:02:13 smithi007 ceph-mon[39249]: from='client.24277 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi050:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-09-18T08:02:15.156 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:02:15 smithi050 ceph-mon[32371]: pgmap v168: 1 pgs: 1 active+clean; 577 KiB data, 193 MiB used, 626 GiB / 626 GiB avail 2024-09-18T08:02:15.157 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:02:15 smithi050 ceph-mon[32371]: from='client.? 172.21.15.50:0/76362755' entity='client.bootstrap-osd' cmd={"prefix": "osd new", "uuid": "c9dac3b7-d651-489d-b96e-5b10fa32d9af"} : dispatch 2024-09-18T08:02:15.157 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:02:15 smithi050 ceph-mon[32371]: from='client.? ' entity='client.bootstrap-osd' cmd={"prefix": "osd new", "uuid": "c9dac3b7-d651-489d-b96e-5b10fa32d9af"} : dispatch 2024-09-18T08:02:15.157 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:02:15 smithi050 ceph-mon[32371]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "c9dac3b7-d651-489d-b96e-5b10fa32d9af"}]': finished 2024-09-18T08:02:15.157 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:02:15 smithi050 ceph-mon[32371]: osdmap e51: 8 total, 7 up, 8 in 2024-09-18T08:02:15.157 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:02:15 smithi050 ceph-mon[32371]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 7} : dispatch 2024-09-18T08:02:15.157 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:02:15 smithi050 ceph-mon[32371]: from='client.? 172.21.15.50:0/3558619372' entity='client.bootstrap-osd' cmd={"prefix": "mon getmap"} : dispatch 2024-09-18T08:02:15.481 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:02:15 smithi007 ceph-mon[29459]: pgmap v168: 1 pgs: 1 active+clean; 577 KiB data, 193 MiB used, 626 GiB / 626 GiB avail 2024-09-18T08:02:15.481 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:02:15 smithi007 ceph-mon[29459]: from='client.? 172.21.15.50:0/76362755' entity='client.bootstrap-osd' cmd={"prefix": "osd new", "uuid": "c9dac3b7-d651-489d-b96e-5b10fa32d9af"} : dispatch 2024-09-18T08:02:15.481 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:02:15 smithi007 ceph-mon[29459]: from='client.? ' entity='client.bootstrap-osd' cmd={"prefix": "osd new", "uuid": "c9dac3b7-d651-489d-b96e-5b10fa32d9af"} : dispatch 2024-09-18T08:02:15.481 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:02:15 smithi007 ceph-mon[29459]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "c9dac3b7-d651-489d-b96e-5b10fa32d9af"}]': finished 2024-09-18T08:02:15.481 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:02:15 smithi007 ceph-mon[29459]: osdmap e51: 8 total, 7 up, 8 in 2024-09-18T08:02:15.481 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:02:15 smithi007 ceph-mon[29459]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 7} : dispatch 2024-09-18T08:02:15.481 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:02:15 smithi007 ceph-mon[29459]: from='client.? 172.21.15.50:0/3558619372' entity='client.bootstrap-osd' cmd={"prefix": "mon getmap"} : dispatch 2024-09-18T08:02:15.482 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:02:15 smithi007 ceph-mon[39249]: pgmap v168: 1 pgs: 1 active+clean; 577 KiB data, 193 MiB used, 626 GiB / 626 GiB avail 2024-09-18T08:02:15.482 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:02:15 smithi007 ceph-mon[39249]: from='client.? 172.21.15.50:0/76362755' entity='client.bootstrap-osd' cmd={"prefix": "osd new", "uuid": "c9dac3b7-d651-489d-b96e-5b10fa32d9af"} : dispatch 2024-09-18T08:02:15.482 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:02:15 smithi007 ceph-mon[39249]: from='client.? ' entity='client.bootstrap-osd' cmd={"prefix": "osd new", "uuid": "c9dac3b7-d651-489d-b96e-5b10fa32d9af"} : dispatch 2024-09-18T08:02:15.482 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:02:15 smithi007 ceph-mon[39249]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "c9dac3b7-d651-489d-b96e-5b10fa32d9af"}]': finished 2024-09-18T08:02:15.482 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:02:15 smithi007 ceph-mon[39249]: osdmap e51: 8 total, 7 up, 8 in 2024-09-18T08:02:15.482 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:02:15 smithi007 ceph-mon[39249]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 7} : dispatch 2024-09-18T08:02:15.482 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:02:15 smithi007 ceph-mon[39249]: from='client.? 172.21.15.50:0/3558619372' entity='client.bootstrap-osd' cmd={"prefix": "mon getmap"} : dispatch 2024-09-18T08:02:17.409 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:02:17 smithi050 ceph-mon[32371]: pgmap v170: 1 pgs: 1 active+clean; 577 KiB data, 193 MiB used, 626 GiB / 626 GiB avail 2024-09-18T08:02:17.480 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:02:17 smithi007 ceph-mon[29459]: pgmap v170: 1 pgs: 1 active+clean; 577 KiB data, 193 MiB used, 626 GiB / 626 GiB avail 2024-09-18T08:02:17.481 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:02:17 smithi007 ceph-mon[39249]: pgmap v170: 1 pgs: 1 active+clean; 577 KiB data, 193 MiB used, 626 GiB / 626 GiB avail 2024-09-18T08:02:19.406 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:02:19 smithi050 ceph-mon[32371]: pgmap v171: 1 pgs: 1 active+clean; 577 KiB data, 193 MiB used, 626 GiB / 626 GiB avail 2024-09-18T08:02:19.480 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:02:19 smithi007 ceph-mon[29459]: pgmap v171: 1 pgs: 1 active+clean; 577 KiB data, 193 MiB used, 626 GiB / 626 GiB avail 2024-09-18T08:02:19.481 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:02:19 smithi007 ceph-mon[39249]: pgmap v171: 1 pgs: 1 active+clean; 577 KiB data, 193 MiB used, 626 GiB / 626 GiB avail 2024-09-18T08:02:21.406 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:02:21 smithi050 ceph-mon[32371]: pgmap v172: 1 pgs: 1 active+clean; 577 KiB data, 193 MiB used, 626 GiB / 626 GiB avail 2024-09-18T08:02:21.480 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:02:21 smithi007 ceph-mon[29459]: pgmap v172: 1 pgs: 1 active+clean; 577 KiB data, 193 MiB used, 626 GiB / 626 GiB avail 2024-09-18T08:02:21.481 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:02:21 smithi007 ceph-mon[39249]: pgmap v172: 1 pgs: 1 active+clean; 577 KiB data, 193 MiB used, 626 GiB / 626 GiB avail 2024-09-18T08:02:22.406 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:02:22 smithi050 ceph-mon[32371]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' cmd={"prefix": "auth get", "entity": "osd.7"} : dispatch 2024-09-18T08:02:22.406 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:02:22 smithi050 ceph-mon[32371]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-18T08:02:22.480 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:02:22 smithi007 ceph-mon[39249]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' cmd={"prefix": "auth get", "entity": "osd.7"} : dispatch 2024-09-18T08:02:22.480 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:02:22 smithi007 ceph-mon[39249]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-18T08:02:22.481 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:02:22 smithi007 ceph-mon[29459]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' cmd={"prefix": "auth get", "entity": "osd.7"} : dispatch 2024-09-18T08:02:22.481 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:02:22 smithi007 ceph-mon[29459]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-18T08:02:23.379 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:02:23 smithi050 ceph-mon[32371]: pgmap v173: 1 pgs: 1 active+clean; 577 KiB data, 193 MiB used, 626 GiB / 626 GiB avail 2024-09-18T08:02:23.379 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:02:23 smithi050 ceph-mon[32371]: Deploying daemon osd.7 on smithi050 2024-09-18T08:02:23.480 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:02:23 smithi007 ceph-mon[29459]: pgmap v173: 1 pgs: 1 active+clean; 577 KiB data, 193 MiB used, 626 GiB / 626 GiB avail 2024-09-18T08:02:23.480 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:02:23 smithi007 ceph-mon[29459]: Deploying daemon osd.7 on smithi050 2024-09-18T08:02:23.481 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:02:23 smithi007 ceph-mon[39249]: pgmap v173: 1 pgs: 1 active+clean; 577 KiB data, 193 MiB used, 626 GiB / 626 GiB avail 2024-09-18T08:02:23.481 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:02:23 smithi007 ceph-mon[39249]: Deploying daemon osd.7 on smithi050 2024-09-18T08:02:24.406 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:02:24 smithi050 ceph-mon[32371]: pgmap v174: 1 pgs: 1 active+clean; 577 KiB data, 193 MiB used, 626 GiB / 626 GiB avail 2024-09-18T08:02:24.480 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:02:24 smithi007 ceph-mon[29459]: pgmap v174: 1 pgs: 1 active+clean; 577 KiB data, 193 MiB used, 626 GiB / 626 GiB avail 2024-09-18T08:02:24.481 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:02:24 smithi007 ceph-mon[39249]: pgmap v174: 1 pgs: 1 active+clean; 577 KiB data, 193 MiB used, 626 GiB / 626 GiB avail 2024-09-18T08:02:26.731 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:02:26 smithi007 ceph-mon[29459]: pgmap v175: 1 pgs: 1 active+clean; 577 KiB data, 193 MiB used, 626 GiB / 626 GiB avail 2024-09-18T08:02:26.731 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:02:26 smithi007 ceph-mon[39249]: pgmap v175: 1 pgs: 1 active+clean; 577 KiB data, 193 MiB used, 626 GiB / 626 GiB avail 2024-09-18T08:02:26.906 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:02:26 smithi050 ceph-mon[32371]: pgmap v175: 1 pgs: 1 active+clean; 577 KiB data, 193 MiB used, 626 GiB / 626 GiB avail 2024-09-18T08:02:27.730 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:02:27 smithi007 ceph-mon[29459]: purged_snaps scrub starts 2024-09-18T08:02:27.731 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:02:27 smithi007 ceph-mon[29459]: purged_snaps scrub ok 2024-09-18T08:02:27.731 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:02:27 smithi007 ceph-mon[39249]: purged_snaps scrub starts 2024-09-18T08:02:27.731 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:02:27 smithi007 ceph-mon[39249]: purged_snaps scrub ok 2024-09-18T08:02:27.898 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:02:27 smithi050 ceph-mon[32371]: purged_snaps scrub starts 2024-09-18T08:02:27.899 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:02:27 smithi050 ceph-mon[32371]: purged_snaps scrub ok 2024-09-18T08:02:28.730 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:02:28 smithi007 ceph-mon[29459]: pgmap v176: 1 pgs: 1 active+clean; 577 KiB data, 193 MiB used, 626 GiB / 626 GiB avail 2024-09-18T08:02:28.731 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:02:28 smithi007 ceph-mon[29459]: osdmap e52: 8 total, 7 up, 8 in 2024-09-18T08:02:28.731 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:02:28 smithi007 ceph-mon[29459]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 7} : dispatch 2024-09-18T08:02:28.731 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:02:28 smithi007 ceph-mon[39249]: pgmap v176: 1 pgs: 1 active+clean; 577 KiB data, 193 MiB used, 626 GiB / 626 GiB avail 2024-09-18T08:02:28.731 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:02:28 smithi007 ceph-mon[39249]: osdmap e52: 8 total, 7 up, 8 in 2024-09-18T08:02:28.731 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:02:28 smithi007 ceph-mon[39249]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 7} : dispatch 2024-09-18T08:02:28.838 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:02:28 smithi050 ceph-mon[32371]: pgmap v176: 1 pgs: 1 active+clean; 577 KiB data, 193 MiB used, 626 GiB / 626 GiB avail 2024-09-18T08:02:28.838 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:02:28 smithi050 ceph-mon[32371]: osdmap e52: 8 total, 7 up, 8 in 2024-09-18T08:02:28.838 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:02:28 smithi050 ceph-mon[32371]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 7} : dispatch 2024-09-18T08:02:29.730 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:02:29 smithi007 ceph-mon[29459]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-18T08:02:29.731 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:02:29 smithi007 ceph-mon[29459]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' 2024-09-18T08:02:29.731 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:02:29 smithi007 ceph-mon[29459]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' 2024-09-18T08:02:29.731 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:02:29 smithi007 ceph-mon[39249]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-18T08:02:29.731 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:02:29 smithi007 ceph-mon[39249]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' 2024-09-18T08:02:29.731 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:02:29 smithi007 ceph-mon[39249]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' 2024-09-18T08:02:29.856 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:02:29 smithi050 ceph-mon[32371]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-18T08:02:29.857 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:02:29 smithi050 ceph-mon[32371]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' 2024-09-18T08:02:29.857 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:02:29 smithi050 ceph-mon[32371]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' 2024-09-18T08:02:30.730 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:02:30 smithi007 ceph-mon[29459]: pgmap v178: 1 pgs: 1 active+clean; 577 KiB data, 193 MiB used, 626 GiB / 626 GiB avail 2024-09-18T08:02:30.731 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:02:30 smithi007 ceph-mon[39249]: pgmap v178: 1 pgs: 1 active+clean; 577 KiB data, 193 MiB used, 626 GiB / 626 GiB avail 2024-09-18T08:02:30.848 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:02:30 smithi050 ceph-mon[32371]: pgmap v178: 1 pgs: 1 active+clean; 577 KiB data, 193 MiB used, 626 GiB / 626 GiB avail 2024-09-18T08:02:31.731 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:02:31 smithi007 ceph-mon[29459]: purged_snaps scrub starts 2024-09-18T08:02:31.731 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:02:31 smithi007 ceph-mon[29459]: purged_snaps scrub ok 2024-09-18T08:02:31.731 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:02:31 smithi007 ceph-mon[29459]: osdmap e53: 8 total, 7 up, 8 in 2024-09-18T08:02:31.731 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:02:31 smithi007 ceph-mon[29459]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 7} : dispatch 2024-09-18T08:02:31.731 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:02:31 smithi007 ceph-mon[39249]: purged_snaps scrub starts 2024-09-18T08:02:31.731 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:02:31 smithi007 ceph-mon[39249]: purged_snaps scrub ok 2024-09-18T08:02:31.731 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:02:31 smithi007 ceph-mon[39249]: osdmap e53: 8 total, 7 up, 8 in 2024-09-18T08:02:31.731 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:02:31 smithi007 ceph-mon[39249]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 7} : dispatch 2024-09-18T08:02:31.788 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:02:31 smithi050 ceph-mon[32371]: purged_snaps scrub starts 2024-09-18T08:02:31.788 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:02:31 smithi050 ceph-mon[32371]: purged_snaps scrub ok 2024-09-18T08:02:31.788 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:02:31 smithi050 ceph-mon[32371]: osdmap e53: 8 total, 7 up, 8 in 2024-09-18T08:02:31.788 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:02:31 smithi050 ceph-mon[32371]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 7} : dispatch 2024-09-18T08:02:33.406 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:02:32 smithi050 ceph-mon[32371]: pgmap v180: 1 pgs: 1 active+clean; 577 KiB data, 193 MiB used, 626 GiB / 626 GiB avail 2024-09-18T08:02:33.406 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:02:32 smithi050 ceph-mon[32371]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' 2024-09-18T08:02:33.406 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:02:32 smithi050 ceph-mon[32371]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' 2024-09-18T08:02:33.406 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:02:32 smithi050 ceph-mon[32371]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-18T08:02:33.406 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:02:32 smithi050 ceph-mon[32371]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-18T08:02:33.406 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:02:32 smithi050 ceph-mon[32371]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' 2024-09-18T08:02:33.480 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:02:32 smithi007 ceph-mon[39249]: pgmap v180: 1 pgs: 1 active+clean; 577 KiB data, 193 MiB used, 626 GiB / 626 GiB avail 2024-09-18T08:02:33.481 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:02:32 smithi007 ceph-mon[39249]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' 2024-09-18T08:02:33.481 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:02:32 smithi007 ceph-mon[39249]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' 2024-09-18T08:02:33.481 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:02:32 smithi007 ceph-mon[39249]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-18T08:02:33.481 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:02:32 smithi007 ceph-mon[39249]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-18T08:02:33.481 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:02:32 smithi007 ceph-mon[39249]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' 2024-09-18T08:02:33.481 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:02:32 smithi007 ceph-mon[29459]: pgmap v180: 1 pgs: 1 active+clean; 577 KiB data, 193 MiB used, 626 GiB / 626 GiB avail 2024-09-18T08:02:33.481 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:02:32 smithi007 ceph-mon[29459]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' 2024-09-18T08:02:33.481 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:02:32 smithi007 ceph-mon[29459]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' 2024-09-18T08:02:33.481 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:02:32 smithi007 ceph-mon[29459]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-18T08:02:33.481 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:02:32 smithi007 ceph-mon[29459]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-18T08:02:33.482 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:02:32 smithi007 ceph-mon[29459]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' 2024-09-18T08:02:34.952 INFO:teuthology.orchestra.run.smithi050.stdout:Created osd(s) 7 on host 'smithi050' 2024-09-18T08:02:35.156 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:02:34 smithi050 ceph-mon[32371]: pgmap v181: 1 pgs: 1 active+clean; 577 KiB data, 193 MiB used, 626 GiB / 626 GiB avail 2024-09-18T08:02:35.156 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:02:34 smithi050 ceph-mon[32371]: from='osd.7 v2:172.21.15.50:6812/874923904' entity='osd.7' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]} : dispatch 2024-09-18T08:02:35.156 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:02:34 smithi050 ceph-mon[32371]: from='osd.7 ' entity='osd.7' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]} : dispatch 2024-09-18T08:02:35.156 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:02:34 smithi050 ceph-mon[32371]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-18T08:02:35.156 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:02:34 smithi050 ceph-mon[32371]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' 2024-09-18T08:02:35.156 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:02:34 smithi050 ceph-mon[32371]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' 2024-09-18T08:02:35.481 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:02:34 smithi007 ceph-mon[39249]: pgmap v181: 1 pgs: 1 active+clean; 577 KiB data, 193 MiB used, 626 GiB / 626 GiB avail 2024-09-18T08:02:35.481 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:02:34 smithi007 ceph-mon[39249]: from='osd.7 v2:172.21.15.50:6812/874923904' entity='osd.7' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]} : dispatch 2024-09-18T08:02:35.481 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:02:34 smithi007 ceph-mon[39249]: from='osd.7 ' entity='osd.7' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]} : dispatch 2024-09-18T08:02:35.481 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:02:34 smithi007 ceph-mon[39249]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-18T08:02:35.481 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:02:34 smithi007 ceph-mon[39249]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' 2024-09-18T08:02:35.481 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:02:34 smithi007 ceph-mon[39249]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' 2024-09-18T08:02:35.482 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:02:34 smithi007 ceph-mon[29459]: pgmap v181: 1 pgs: 1 active+clean; 577 KiB data, 193 MiB used, 626 GiB / 626 GiB avail 2024-09-18T08:02:35.482 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:02:34 smithi007 ceph-mon[29459]: from='osd.7 v2:172.21.15.50:6812/874923904' entity='osd.7' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]} : dispatch 2024-09-18T08:02:35.482 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:02:34 smithi007 ceph-mon[29459]: from='osd.7 ' entity='osd.7' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]} : dispatch 2024-09-18T08:02:35.482 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:02:34 smithi007 ceph-mon[29459]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-18T08:02:35.482 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:02:34 smithi007 ceph-mon[29459]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' 2024-09-18T08:02:35.482 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:02:34 smithi007 ceph-mon[29459]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' 2024-09-18T08:02:35.569 DEBUG:teuthology.orchestra.run.smithi050:osd.7> sudo journalctl -f -n 0 -u ceph-779349ca-7593-11ef-bceb-c7b262605968@osd.7.service 2024-09-18T08:02:35.571 INFO:tasks.cephadm:Waiting for 8 OSDs to come up... 2024-09-18T08:02:35.571 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 779349ca-7593-11ef-bceb-c7b262605968 -- ceph osd stat -f json 2024-09-18T08:02:35.839 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/779349ca-7593-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T08:02:36.156 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:02:35 smithi050 ceph-mon[32371]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2024-09-18T08:02:36.157 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:02:36 smithi050 ceph-mon[32371]: osdmap e54: 8 total, 7 up, 8 in 2024-09-18T08:02:36.157 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:02:36 smithi050 ceph-mon[32371]: from='osd.7 v2:172.21.15.50:6812/874923904' entity='osd.7' cmd={"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi050", "root=default"]} : dispatch 2024-09-18T08:02:36.157 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:02:36 smithi050 ceph-mon[32371]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 7} : dispatch 2024-09-18T08:02:36.157 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:02:36 smithi050 ceph-mon[32371]: from='osd.7 ' entity='osd.7' cmd={"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi050", "root=default"]} : dispatch 2024-09-18T08:02:36.157 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:02:36 smithi050 ceph-mon[32371]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi050", "root=default"]}]': finished 2024-09-18T08:02:36.157 INFO:journalctl@ceph.osd.7.smithi050.stdout:Sep 18 08:02:36 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd-7[53264]: 2024-09-18T08:02:35.994+0000 7f77c7895640 -1 osd.7 0 waiting for initial osdmap 2024-09-18T08:02:36.157 INFO:journalctl@ceph.osd.7.smithi050.stdout:Sep 18 08:02:36 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd-7[53264]: 2024-09-18T08:02:36.021+0000 7f77c36bc640 -1 osd.7 55 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-18T08:02:36.232 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:02:35 smithi007 ceph-mon[29459]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2024-09-18T08:02:36.232 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:02:35 smithi007 ceph-mon[29459]: osdmap e54: 8 total, 7 up, 8 in 2024-09-18T08:02:36.233 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:02:35 smithi007 ceph-mon[29459]: from='osd.7 v2:172.21.15.50:6812/874923904' entity='osd.7' cmd={"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi050", "root=default"]} : dispatch 2024-09-18T08:02:36.233 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:02:35 smithi007 ceph-mon[29459]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 7} : dispatch 2024-09-18T08:02:36.233 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:02:35 smithi007 ceph-mon[29459]: from='osd.7 ' entity='osd.7' cmd={"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi050", "root=default"]} : dispatch 2024-09-18T08:02:36.233 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:02:35 smithi007 ceph-mon[29459]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi050", "root=default"]}]': finished 2024-09-18T08:02:36.233 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:02:35 smithi007 ceph-mon[39249]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2024-09-18T08:02:36.233 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:02:35 smithi007 ceph-mon[39249]: osdmap e54: 8 total, 7 up, 8 in 2024-09-18T08:02:36.233 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:02:35 smithi007 ceph-mon[39249]: from='osd.7 v2:172.21.15.50:6812/874923904' entity='osd.7' cmd={"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi050", "root=default"]} : dispatch 2024-09-18T08:02:36.234 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:02:35 smithi007 ceph-mon[39249]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 7} : dispatch 2024-09-18T08:02:36.234 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:02:35 smithi007 ceph-mon[39249]: from='osd.7 ' entity='osd.7' cmd={"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi050", "root=default"]} : dispatch 2024-09-18T08:02:36.234 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:02:35 smithi007 ceph-mon[39249]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi050", "root=default"]}]': finished 2024-09-18T08:02:36.487 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-09-18T08:02:37.219 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:02:36 smithi007 ceph-mon[29459]: pgmap v183: 1 pgs: 1 active+clean; 577 KiB data, 193 MiB used, 626 GiB / 626 GiB avail 2024-09-18T08:02:37.219 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:02:36 smithi007 ceph-mon[29459]: osdmap e55: 8 total, 7 up, 8 in 2024-09-18T08:02:37.219 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:02:36 smithi007 ceph-mon[29459]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 7} : dispatch 2024-09-18T08:02:37.219 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:02:36 smithi007 ceph-mon[29459]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 7} : dispatch 2024-09-18T08:02:37.220 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:02:36 smithi007 ceph-mon[29459]: from='client.? 172.21.15.7:0/2667773626' entity='client.admin' cmd={"prefix": "osd stat", "format": "json"} : dispatch 2024-09-18T08:02:37.220 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:02:36 smithi007 ceph-mon[29459]: osd.7 v2:172.21.15.50:6812/874923904 boot 2024-09-18T08:02:37.220 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:02:36 smithi007 ceph-mon[29459]: osdmap e56: 8 total, 8 up, 8 in 2024-09-18T08:02:37.220 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:02:36 smithi007 ceph-mon[29459]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 7} : dispatch 2024-09-18T08:02:37.220 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:02:36 smithi007 ceph-mon[39249]: pgmap v183: 1 pgs: 1 active+clean; 577 KiB data, 193 MiB used, 626 GiB / 626 GiB avail 2024-09-18T08:02:37.220 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:02:36 smithi007 ceph-mon[39249]: osdmap e55: 8 total, 7 up, 8 in 2024-09-18T08:02:37.220 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:02:36 smithi007 ceph-mon[39249]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 7} : dispatch 2024-09-18T08:02:37.220 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:02:36 smithi007 ceph-mon[39249]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 7} : dispatch 2024-09-18T08:02:37.220 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:02:36 smithi007 ceph-mon[39249]: from='client.? 172.21.15.7:0/2667773626' entity='client.admin' cmd={"prefix": "osd stat", "format": "json"} : dispatch 2024-09-18T08:02:37.220 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:02:36 smithi007 ceph-mon[39249]: osd.7 v2:172.21.15.50:6812/874923904 boot 2024-09-18T08:02:37.221 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:02:36 smithi007 ceph-mon[39249]: osdmap e56: 8 total, 8 up, 8 in 2024-09-18T08:02:37.221 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:02:36 smithi007 ceph-mon[39249]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 7} : dispatch 2024-09-18T08:02:37.221 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":55,"num_osds":8,"num_up_osds":7,"osd_up_since":1726646527,"num_in_osds":8,"osd_in_since":1726646534,"num_remapped_pgs":0} 2024-09-18T08:02:37.406 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:02:36 smithi050 ceph-mon[32371]: pgmap v183: 1 pgs: 1 active+clean; 577 KiB data, 193 MiB used, 626 GiB / 626 GiB avail 2024-09-18T08:02:37.406 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:02:36 smithi050 ceph-mon[32371]: osdmap e55: 8 total, 7 up, 8 in 2024-09-18T08:02:37.406 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:02:36 smithi050 ceph-mon[32371]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 7} : dispatch 2024-09-18T08:02:37.406 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:02:36 smithi050 ceph-mon[32371]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 7} : dispatch 2024-09-18T08:02:37.406 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:02:36 smithi050 ceph-mon[32371]: from='client.? 172.21.15.7:0/2667773626' entity='client.admin' cmd={"prefix": "osd stat", "format": "json"} : dispatch 2024-09-18T08:02:37.406 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:02:36 smithi050 ceph-mon[32371]: osd.7 v2:172.21.15.50:6812/874923904 boot 2024-09-18T08:02:37.406 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:02:36 smithi050 ceph-mon[32371]: osdmap e56: 8 total, 8 up, 8 in 2024-09-18T08:02:37.407 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:02:36 smithi050 ceph-mon[32371]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 7} : dispatch 2024-09-18T08:02:38.223 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 779349ca-7593-11ef-bceb-c7b262605968 -- ceph osd stat -f json 2024-09-18T08:02:38.253 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:02:38 smithi007 ceph-mon[29459]: purged_snaps scrub starts 2024-09-18T08:02:38.254 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:02:38 smithi007 ceph-mon[29459]: purged_snaps scrub ok 2024-09-18T08:02:38.254 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:02:38 smithi007 ceph-mon[29459]: osdmap e57: 8 total, 8 up, 8 in 2024-09-18T08:02:38.254 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:02:38 smithi007 ceph-mon[39249]: purged_snaps scrub starts 2024-09-18T08:02:38.254 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:02:38 smithi007 ceph-mon[39249]: purged_snaps scrub ok 2024-09-18T08:02:38.254 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:02:38 smithi007 ceph-mon[39249]: osdmap e57: 8 total, 8 up, 8 in 2024-09-18T08:02:38.388 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:02:38 smithi050 ceph-mon[32371]: purged_snaps scrub starts 2024-09-18T08:02:38.388 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:02:38 smithi050 ceph-mon[32371]: purged_snaps scrub ok 2024-09-18T08:02:38.388 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:02:38 smithi050 ceph-mon[32371]: osdmap e57: 8 total, 8 up, 8 in 2024-09-18T08:02:38.492 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/779349ca-7593-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T08:02:39.148 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-09-18T08:02:39.149 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:02:39 smithi007 ceph-mon[39249]: pgmap v186: 1 pgs: 1 active+clean; 577 KiB data, 220 MiB used, 715 GiB / 715 GiB avail 2024-09-18T08:02:39.149 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:02:39 smithi007 ceph-mon[39249]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' 2024-09-18T08:02:39.149 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:02:39 smithi007 ceph-mon[39249]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' 2024-09-18T08:02:39.149 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:02:39 smithi007 ceph-mon[39249]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"} : dispatch 2024-09-18T08:02:39.149 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:02:39 smithi007 ceph-mon[39249]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"} : dispatch 2024-09-18T08:02:39.149 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:02:39 smithi007 ceph-mon[39249]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"} : dispatch 2024-09-18T08:02:39.149 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:02:39 smithi007 ceph-mon[39249]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"} : dispatch 2024-09-18T08:02:39.149 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:02:39 smithi007 ceph-mon[39249]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' 2024-09-18T08:02:39.149 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:02:39 smithi007 ceph-mon[39249]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-18T08:02:39.150 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:02:39 smithi007 ceph-mon[39249]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-18T08:02:39.150 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:02:39 smithi007 ceph-mon[39249]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' 2024-09-18T08:02:39.150 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:02:39 smithi007 ceph-mon[29459]: pgmap v186: 1 pgs: 1 active+clean; 577 KiB data, 220 MiB used, 715 GiB / 715 GiB avail 2024-09-18T08:02:39.150 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:02:39 smithi007 ceph-mon[29459]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' 2024-09-18T08:02:39.150 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:02:39 smithi007 ceph-mon[29459]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' 2024-09-18T08:02:39.150 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:02:39 smithi007 ceph-mon[29459]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"} : dispatch 2024-09-18T08:02:39.150 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:02:39 smithi007 ceph-mon[29459]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"} : dispatch 2024-09-18T08:02:39.150 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:02:39 smithi007 ceph-mon[29459]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"} : dispatch 2024-09-18T08:02:39.150 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:02:39 smithi007 ceph-mon[29459]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"} : dispatch 2024-09-18T08:02:39.150 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:02:39 smithi007 ceph-mon[29459]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' 2024-09-18T08:02:39.150 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:02:39 smithi007 ceph-mon[29459]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-18T08:02:39.151 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:02:39 smithi007 ceph-mon[29459]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-18T08:02:39.151 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:02:39 smithi007 ceph-mon[29459]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' 2024-09-18T08:02:39.406 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:02:39 smithi050 ceph-mon[32371]: pgmap v186: 1 pgs: 1 active+clean; 577 KiB data, 220 MiB used, 715 GiB / 715 GiB avail 2024-09-18T08:02:39.406 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:02:39 smithi050 ceph-mon[32371]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' 2024-09-18T08:02:39.406 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:02:39 smithi050 ceph-mon[32371]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' 2024-09-18T08:02:39.406 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:02:39 smithi050 ceph-mon[32371]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"} : dispatch 2024-09-18T08:02:39.406 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:02:39 smithi050 ceph-mon[32371]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"} : dispatch 2024-09-18T08:02:39.406 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:02:39 smithi050 ceph-mon[32371]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"} : dispatch 2024-09-18T08:02:39.407 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:02:39 smithi050 ceph-mon[32371]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"} : dispatch 2024-09-18T08:02:39.407 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:02:39 smithi050 ceph-mon[32371]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' 2024-09-18T08:02:39.407 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:02:39 smithi050 ceph-mon[32371]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-18T08:02:39.407 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:02:39 smithi050 ceph-mon[32371]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-18T08:02:39.407 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:02:39 smithi050 ceph-mon[32371]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' 2024-09-18T08:02:39.848 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":57,"num_osds":8,"num_up_osds":8,"osd_up_since":1726646556,"num_in_osds":8,"osd_in_since":1726646534,"num_remapped_pgs":0} 2024-09-18T08:02:39.848 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid 779349ca-7593-11ef-bceb-c7b262605968 -- ceph osd dump --format=json 2024-09-18T08:02:40.149 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/779349ca-7593-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T08:02:40.231 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:02:40 smithi007 ceph-mon[29459]: Detected new or changed devices on smithi050 2024-09-18T08:02:40.231 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:02:40 smithi007 ceph-mon[29459]: Adjusting osd_memory_target on smithi050 to 4273M 2024-09-18T08:02:40.231 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:02:40 smithi007 ceph-mon[29459]: from='client.? 172.21.15.7:0/2290457139' entity='client.admin' cmd={"prefix": "osd stat", "format": "json"} : dispatch 2024-09-18T08:02:40.231 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:02:40 smithi007 ceph-mon[29459]: osdmap e58: 8 total, 8 up, 8 in 2024-09-18T08:02:40.231 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:02:40 smithi007 ceph-mon[39249]: Detected new or changed devices on smithi050 2024-09-18T08:02:40.232 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:02:40 smithi007 ceph-mon[39249]: Adjusting osd_memory_target on smithi050 to 4273M 2024-09-18T08:02:40.232 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:02:40 smithi007 ceph-mon[39249]: from='client.? 172.21.15.7:0/2290457139' entity='client.admin' cmd={"prefix": "osd stat", "format": "json"} : dispatch 2024-09-18T08:02:40.232 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:02:40 smithi007 ceph-mon[39249]: osdmap e58: 8 total, 8 up, 8 in 2024-09-18T08:02:40.406 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:02:40 smithi050 ceph-mon[32371]: Detected new or changed devices on smithi050 2024-09-18T08:02:40.406 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:02:40 smithi050 ceph-mon[32371]: Adjusting osd_memory_target on smithi050 to 4273M 2024-09-18T08:02:40.406 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:02:40 smithi050 ceph-mon[32371]: from='client.? 172.21.15.7:0/2290457139' entity='client.admin' cmd={"prefix": "osd stat", "format": "json"} : dispatch 2024-09-18T08:02:40.406 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:02:40 smithi050 ceph-mon[32371]: osdmap e58: 8 total, 8 up, 8 in 2024-09-18T08:02:40.806 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-09-18T08:02:40.806 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":58,"fsid":"779349ca-7593-11ef-bceb-c7b262605968","created":"2024-09-18T07:57:09.295710+0000","modified":"2024-09-18T08:02:39.441632+0000","last_up_change":"2024-09-18T08:02:36.989876+0000","last_in_change":"2024-09-18T08:02:14.309111+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":18,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-18T08:00:11.802627+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":7.8899998664855957,"score_stable":7.8899998664855957,"optimal_score":0.37999999523162842,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"ae8f60df-72ac-46fb-91a1-f132b851c876","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":56,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6801","nonce":2896425794}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6802","nonce":2896425794}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6804","nonce":2896425794}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6803","nonce":2896425794}]},"public_addr":"172.21.15.7:6801/2896425794","cluster_addr":"172.21.15.7:6802/2896425794","heartbeat_back_addr":"172.21.15.7:6804/2896425794","heartbeat_front_addr":"172.21.15.7:6803/2896425794","state":["exists","up"]},{"osd":1,"uuid":"be28b70c-1f73-494f-ad4b-cb7ec5311999","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":33,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6805","nonce":1971397549}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6806","nonce":1971397549}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6808","nonce":1971397549}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6807","nonce":1971397549}]},"public_addr":"172.21.15.7:6805/1971397549","cluster_addr":"172.21.15.7:6806/1971397549","heartbeat_back_addr":"172.21.15.7:6808/1971397549","heartbeat_front_addr":"172.21.15.7:6807/1971397549","state":["exists","up"]},{"osd":2,"uuid":"cfa4f083-c7fd-4743-be65-84a3040fe6dc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6809","nonce":556142559}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6810","nonce":556142559}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6812","nonce":556142559}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6811","nonce":556142559}]},"public_addr":"172.21.15.7:6809/556142559","cluster_addr":"172.21.15.7:6810/556142559","heartbeat_back_addr":"172.21.15.7:6812/556142559","heartbeat_front_addr":"172.21.15.7:6811/556142559","state":["exists","up"]},{"osd":3,"uuid":"11ecd7e1-a751-4b75-95f5-00b32d2a5bc4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6813","nonce":163587772}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6814","nonce":163587772}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6816","nonce":163587772}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6815","nonce":163587772}]},"public_addr":"172.21.15.7:6813/163587772","cluster_addr":"172.21.15.7:6814/163587772","heartbeat_back_addr":"172.21.15.7:6816/163587772","heartbeat_front_addr":"172.21.15.7:6815/163587772","state":["exists","up"]},{"osd":4,"uuid":"98992b34-cfa1-45bf-97a1-7d557739252f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":32,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":3662119805}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6801","nonce":3662119805}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6803","nonce":3662119805}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":3662119805}]},"public_addr":"172.21.15.50:6800/3662119805","cluster_addr":"172.21.15.50:6801/3662119805","heartbeat_back_addr":"172.21.15.50:6803/3662119805","heartbeat_front_addr":"172.21.15.50:6802/3662119805","state":["exists","up"]},{"osd":5,"uuid":"7f290005-ebca-467b-87e4-02ead1063610","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":39,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":1199797954}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6805","nonce":1199797954}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6807","nonce":1199797954}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":1199797954}]},"public_addr":"172.21.15.50:6804/1199797954","cluster_addr":"172.21.15.50:6805/1199797954","heartbeat_back_addr":"172.21.15.50:6807/1199797954","heartbeat_front_addr":"172.21.15.50:6806/1199797954","state":["exists","up"]},{"osd":6,"uuid":"e07d846c-ae42-4fea-aa5c-3a25835d18a6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":47,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":255960422}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6809","nonce":255960422}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6811","nonce":255960422}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":255960422}]},"public_addr":"172.21.15.50:6808/255960422","cluster_addr":"172.21.15.50:6809/255960422","heartbeat_back_addr":"172.21.15.50:6811/255960422","heartbeat_front_addr":"172.21.15.50:6810/255960422","state":["exists","up"]},{"osd":7,"uuid":"c9dac3b7-d651-489d-b96e-5b10fa32d9af","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":56,"up_thru":57,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":874923904}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6813","nonce":874923904}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6815","nonce":874923904}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":874923904}]},"public_addr":"172.21.15.50:6812/874923904","cluster_addr":"172.21.15.50:6813/874923904","heartbeat_back_addr":"172.21.15.50:6815/874923904","heartbeat_front_addr":"172.21.15.50:6814/874923904","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-18T08:02:26.967281+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-18T08:01:57.541118+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-18T08:02:29.901882+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-18T08:01:40.839379+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-18T08:02:11.109010+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-18T08:01:35.651624+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-18T08:02:05.533581+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-18T08:02:35.425165+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.7:0/533727535":"2024-09-19T07:57:57.346149+0000","172.21.15.7:0/1277549846":"2024-09-19T07:57:57.346149+0000","172.21.15.7:6800/1013061071":"2024-09-19T07:57:57.346149+0000","172.21.15.7:0/4272027292":"2024-09-19T07:57:31.015269+0000","172.21.15.7:0/1668598084":"2024-09-19T07:57:57.346149+0000","172.21.15.7:0/3924911335":"2024-09-19T07:57:31.015269+0000","172.21.15.7:0/2514354773":"2024-09-19T07:57:31.015269+0000","172.21.15.7:6800/2918496291":"2024-09-19T07:57:31.015269+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-18T08:02:41.124 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:02:41 smithi007 ceph-mon[29459]: pgmap v188: 1 pgs: 1 active+clean; 577 KiB data, 220 MiB used, 715 GiB / 715 GiB avail 2024-09-18T08:02:41.124 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:02:41 smithi007 ceph-mon[29459]: from='client.? 172.21.15.7:0/1150956706' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2024-09-18T08:02:41.124 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:02:41 smithi007 ceph-mon[39249]: pgmap v188: 1 pgs: 1 active+clean; 577 KiB data, 220 MiB used, 715 GiB / 715 GiB avail 2024-09-18T08:02:41.125 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:02:41 smithi007 ceph-mon[39249]: from='client.? 172.21.15.7:0/1150956706' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2024-09-18T08:02:41.307 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:02:41 smithi050 ceph-mon[32371]: pgmap v188: 1 pgs: 1 active+clean; 577 KiB data, 220 MiB used, 715 GiB / 715 GiB avail 2024-09-18T08:02:41.307 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:02:41 smithi050 ceph-mon[32371]: from='client.? 172.21.15.7:0/1150956706' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2024-09-18T08:02:41.441 INFO:tasks.cephadm.ceph_manager.ceph:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-09-18T08:00:11.802627+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'is_stretch_pool': False, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 7.889999866485596, 'score_stable': 7.889999866485596, 'optimal_score': 0.3799999952316284, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-09-18T08:02:41.441 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid 779349ca-7593-11ef-bceb-c7b262605968 -- ceph osd pool get .mgr pg_num 2024-09-18T08:02:41.701 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/779349ca-7593-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T08:02:42.306 INFO:teuthology.orchestra.run.smithi007.stdout:pg_num: 1 2024-09-18T08:02:42.960 INFO:tasks.cephadm:Adding ceph.rgw.foo.a on smithi007 2024-09-18T08:02:42.961 DEBUG:teuthology.orchestra.run.smithi050:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 779349ca-7593-11ef-bceb-c7b262605968 -- ceph orch apply rgw foo.a --placement '1;smithi007=foo.a' 2024-09-18T08:02:43.223 INFO:teuthology.orchestra.run.smithi050.stderr:Inferring config /var/lib/ceph/779349ca-7593-11ef-bceb-c7b262605968/mon.b/config 2024-09-18T08:02:43.230 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:02:43 smithi007 ceph-mon[29459]: pgmap v190: 1 pgs: 1 active+clean; 577 KiB data, 220 MiB used, 715 GiB / 715 GiB avail 2024-09-18T08:02:43.230 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:02:43 smithi007 ceph-mon[29459]: from='client.? 172.21.15.7:0/415232580' entity='client.admin' cmd={"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} : dispatch 2024-09-18T08:02:43.231 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:02:43 smithi007 ceph-mon[39249]: pgmap v190: 1 pgs: 1 active+clean; 577 KiB data, 220 MiB used, 715 GiB / 715 GiB avail 2024-09-18T08:02:43.231 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:02:43 smithi007 ceph-mon[39249]: from='client.? 172.21.15.7:0/415232580' entity='client.admin' cmd={"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} : dispatch 2024-09-18T08:02:43.343 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:02:43 smithi050 ceph-mon[32371]: pgmap v190: 1 pgs: 1 active+clean; 577 KiB data, 220 MiB used, 715 GiB / 715 GiB avail 2024-09-18T08:02:43.343 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:02:43 smithi050 ceph-mon[32371]: from='client.? 172.21.15.7:0/415232580' entity='client.admin' cmd={"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} : dispatch 2024-09-18T08:02:43.859 INFO:teuthology.orchestra.run.smithi050.stdout:Scheduled rgw.foo.a update... 2024-09-18T08:02:44.458 DEBUG:teuthology.orchestra.run.smithi007:rgw.foo.a> sudo journalctl -f -n 0 -u ceph-779349ca-7593-11ef-bceb-c7b262605968@rgw.foo.a.service 2024-09-18T08:02:44.461 INFO:tasks.cephadm:Adding ceph.iscsi.iscsi.a on smithi050 2024-09-18T08:02:44.461 DEBUG:teuthology.orchestra.run.smithi050:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 779349ca-7593-11ef-bceb-c7b262605968 -- ceph osd pool create datapool 3 3 replicated 2024-09-18T08:02:44.717 INFO:teuthology.orchestra.run.smithi050.stderr:Inferring config /var/lib/ceph/779349ca-7593-11ef-bceb-c7b262605968/mon.b/config 2024-09-18T08:02:44.981 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:02:44 smithi007 ceph-mon[29459]: purged_snaps scrub starts 2024-09-18T08:02:44.981 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:02:44 smithi007 ceph-mon[29459]: purged_snaps scrub ok 2024-09-18T08:02:44.981 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:02:44 smithi007 ceph-mon[29459]: pgmap v191: 1 pgs: 1 active+clean; 577 KiB data, 221 MiB used, 715 GiB / 715 GiB avail 2024-09-18T08:02:44.981 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:02:44 smithi007 ceph-mon[29459]: from='client.24322 -' entity='client.admin' cmd=[{"prefix": "orch apply rgw", "svc_id": "foo.a", "placement": "1;smithi007=foo.a", "target": ["mon-mgr", ""]}]: dispatch 2024-09-18T08:02:44.981 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:02:44 smithi007 ceph-mon[29459]: Saving service rgw.foo.a spec with placement smithi007=foo.a;count:1 2024-09-18T08:02:44.981 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:02:44 smithi007 ceph-mon[29459]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' 2024-09-18T08:02:44.981 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:02:44 smithi007 ceph-mon[29459]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-18T08:02:44.982 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:02:44 smithi007 ceph-mon[29459]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-18T08:02:44.982 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:02:44 smithi007 ceph-mon[29459]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-18T08:02:44.982 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:02:44 smithi007 ceph-mon[29459]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' 2024-09-18T08:02:44.982 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:02:44 smithi007 ceph-mon[29459]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' cmd={"prefix": "auth get-or-create", "entity": "client.rgw.foo.a", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]} : dispatch 2024-09-18T08:02:44.982 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:02:44 smithi007 ceph-mon[29459]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.a", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]': finished 2024-09-18T08:02:44.982 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:02:44 smithi007 ceph-mon[29459]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' 2024-09-18T08:02:44.982 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:02:44 smithi007 ceph-mon[29459]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-18T08:02:44.982 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:02:44 smithi007 ceph-mon[29459]: Deploying daemon rgw.foo.a on smithi007 2024-09-18T08:02:44.982 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:02:44 smithi007 ceph-mon[29459]: osdmap e59: 8 total, 8 up, 8 in 2024-09-18T08:02:44.982 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:02:44 smithi007 ceph-mon[39249]: purged_snaps scrub starts 2024-09-18T08:02:44.982 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:02:44 smithi007 ceph-mon[39249]: purged_snaps scrub ok 2024-09-18T08:02:44.983 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:02:44 smithi007 ceph-mon[39249]: pgmap v191: 1 pgs: 1 active+clean; 577 KiB data, 221 MiB used, 715 GiB / 715 GiB avail 2024-09-18T08:02:44.983 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:02:44 smithi007 ceph-mon[39249]: from='client.24322 -' entity='client.admin' cmd=[{"prefix": "orch apply rgw", "svc_id": "foo.a", "placement": "1;smithi007=foo.a", "target": ["mon-mgr", ""]}]: dispatch 2024-09-18T08:02:44.983 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:02:44 smithi007 ceph-mon[39249]: Saving service rgw.foo.a spec with placement smithi007=foo.a;count:1 2024-09-18T08:02:44.983 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:02:44 smithi007 ceph-mon[39249]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' 2024-09-18T08:02:44.983 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:02:44 smithi007 ceph-mon[39249]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-18T08:02:44.983 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:02:44 smithi007 ceph-mon[39249]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-18T08:02:44.983 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:02:44 smithi007 ceph-mon[39249]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-18T08:02:44.983 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:02:44 smithi007 ceph-mon[39249]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' 2024-09-18T08:02:44.983 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:02:44 smithi007 ceph-mon[39249]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' cmd={"prefix": "auth get-or-create", "entity": "client.rgw.foo.a", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]} : dispatch 2024-09-18T08:02:44.983 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:02:44 smithi007 ceph-mon[39249]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.a", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]': finished 2024-09-18T08:02:44.984 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:02:44 smithi007 ceph-mon[39249]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' 2024-09-18T08:02:44.984 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:02:44 smithi007 ceph-mon[39249]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-18T08:02:44.984 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:02:44 smithi007 ceph-mon[39249]: Deploying daemon rgw.foo.a on smithi007 2024-09-18T08:02:44.984 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:02:44 smithi007 ceph-mon[39249]: osdmap e59: 8 total, 8 up, 8 in 2024-09-18T08:02:45.089 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:02:44 smithi050 ceph-mon[32371]: purged_snaps scrub starts 2024-09-18T08:02:45.090 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:02:44 smithi050 ceph-mon[32371]: purged_snaps scrub ok 2024-09-18T08:02:45.090 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:02:44 smithi050 ceph-mon[32371]: pgmap v191: 1 pgs: 1 active+clean; 577 KiB data, 221 MiB used, 715 GiB / 715 GiB avail 2024-09-18T08:02:45.090 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:02:44 smithi050 ceph-mon[32371]: from='client.24322 -' entity='client.admin' cmd=[{"prefix": "orch apply rgw", "svc_id": "foo.a", "placement": "1;smithi007=foo.a", "target": ["mon-mgr", ""]}]: dispatch 2024-09-18T08:02:45.090 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:02:44 smithi050 ceph-mon[32371]: Saving service rgw.foo.a spec with placement smithi007=foo.a;count:1 2024-09-18T08:02:45.090 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:02:44 smithi050 ceph-mon[32371]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' 2024-09-18T08:02:45.090 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:02:44 smithi050 ceph-mon[32371]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-18T08:02:45.090 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:02:44 smithi050 ceph-mon[32371]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-18T08:02:45.090 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:02:44 smithi050 ceph-mon[32371]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-18T08:02:45.090 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:02:44 smithi050 ceph-mon[32371]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' 2024-09-18T08:02:45.090 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:02:44 smithi050 ceph-mon[32371]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' cmd={"prefix": "auth get-or-create", "entity": "client.rgw.foo.a", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]} : dispatch 2024-09-18T08:02:45.091 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:02:44 smithi050 ceph-mon[32371]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.a", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]': finished 2024-09-18T08:02:45.091 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:02:44 smithi050 ceph-mon[32371]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' 2024-09-18T08:02:45.091 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:02:44 smithi050 ceph-mon[32371]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-18T08:02:45.091 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:02:44 smithi050 ceph-mon[32371]: Deploying daemon rgw.foo.a on smithi007 2024-09-18T08:02:45.091 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:02:44 smithi050 ceph-mon[32371]: osdmap e59: 8 total, 8 up, 8 in 2024-09-18T08:02:45.865 INFO:teuthology.orchestra.run.smithi050.stderr:pool 'datapool' created 2024-09-18T08:02:45.980 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:02:45 smithi007 ceph-mon[29459]: from='client.? 172.21.15.50:0/2754948336' entity='client.admin' cmd={"prefix": "osd pool create", "pool": "datapool", "pg_num": 3, "pgp_num": 3, "pool_type": "replicated"} : dispatch 2024-09-18T08:02:45.981 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:02:45 smithi007 ceph-mon[29459]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool create", "pool": "datapool", "pg_num": 3, "pgp_num": 3, "pool_type": "replicated"} : dispatch 2024-09-18T08:02:45.981 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:02:45 smithi007 ceph-mon[39249]: from='client.? 172.21.15.50:0/2754948336' entity='client.admin' cmd={"prefix": "osd pool create", "pool": "datapool", "pg_num": 3, "pgp_num": 3, "pool_type": "replicated"} : dispatch 2024-09-18T08:02:45.981 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:02:45 smithi007 ceph-mon[39249]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool create", "pool": "datapool", "pg_num": 3, "pgp_num": 3, "pool_type": "replicated"} : dispatch 2024-09-18T08:02:45.981 INFO:journalctl@ceph.rgw.foo.a.smithi007.stdout:Sep 18 08:02:45 smithi007 systemd[1]: Starting Ceph rgw.foo.a for 779349ca-7593-11ef-bceb-c7b262605968... 2024-09-18T08:02:46.156 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:02:45 smithi050 ceph-mon[32371]: from='client.? 172.21.15.50:0/2754948336' entity='client.admin' cmd={"prefix": "osd pool create", "pool": "datapool", "pg_num": 3, "pgp_num": 3, "pool_type": "replicated"} : dispatch 2024-09-18T08:02:46.156 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:02:45 smithi050 ceph-mon[32371]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool create", "pool": "datapool", "pg_num": 3, "pgp_num": 3, "pool_type": "replicated"} : dispatch 2024-09-18T08:02:46.353 INFO:journalctl@ceph.rgw.foo.a.smithi007.stdout:Sep 18 08:02:46 smithi007 podman[63256]: 2024-09-18 08:02:45.996020421 +0000 UTC m=+0.016475479 image pull b22e1e37ddf1ca6e22136cd6540a9ab61e7cafb8bfbc7a91a2f20f7f9941bd9a quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:1f5b5a3734f665ef333266f39143ac3ef1eb1951b21258f2307e10332e2d7ac9 2024-09-18T08:02:46.353 INFO:journalctl@ceph.rgw.foo.a.smithi007.stdout:Sep 18 08:02:46 smithi007 podman[63256]: 2024-09-18 08:02:46.098771544 +0000 UTC m=+0.119226587 container create 3ed88a6aa96c5c93fdf0e98eee7847b67e703815a163934a55be6e9ce5976047 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:1f5b5a3734f665ef333266f39143ac3ef1eb1951b21258f2307e10332e2d7ac9, name=ceph-779349ca-7593-11ef-bceb-c7b262605968-rgw-foo-a, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.license=GPLv2, RELEASE=wip-guits-main-2024-09-17-1213-e7fb7b5, io.buildah.version=1.37.2, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240731, org.label-schema.schema-version=1.0, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 9 Base Image) 2024-09-18T08:02:46.354 INFO:journalctl@ceph.rgw.foo.a.smithi007.stdout:Sep 18 08:02:46 smithi007 podman[63256]: 2024-09-18 08:02:46.276720074 +0000 UTC m=+0.297175134 container init 3ed88a6aa96c5c93fdf0e98eee7847b67e703815a163934a55be6e9ce5976047 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:1f5b5a3734f665ef333266f39143ac3ef1eb1951b21258f2307e10332e2d7ac9, name=ceph-779349ca-7593-11ef-bceb-c7b262605968-rgw-foo-a, org.label-schema.schema-version=1.0, RELEASE=wip-guits-main-2024-09-17-1213-e7fb7b5, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 9 Base Image, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, org.label-schema.license=GPLv2, org.label-schema.build-date=20240731, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, CEPH_POINT_RELEASE=, io.buildah.version=1.37.2) 2024-09-18T08:02:46.354 INFO:journalctl@ceph.rgw.foo.a.smithi007.stdout:Sep 18 08:02:46 smithi007 podman[63256]: 2024-09-18 08:02:46.27969501 +0000 UTC m=+0.300150061 container start 3ed88a6aa96c5c93fdf0e98eee7847b67e703815a163934a55be6e9ce5976047 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:1f5b5a3734f665ef333266f39143ac3ef1eb1951b21258f2307e10332e2d7ac9, name=ceph-779349ca-7593-11ef-bceb-c7b262605968-rgw-foo-a, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, io.buildah.version=1.37.2, maintainer=Guillaume Abrioux , RELEASE=wip-guits-main-2024-09-17-1213-e7fb7b5, CEPH_POINT_RELEASE=, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.build-date=20240731, GIT_CLEAN=True) 2024-09-18T08:02:46.354 INFO:journalctl@ceph.rgw.foo.a.smithi007.stdout:Sep 18 08:02:46 smithi007 bash[63256]: 3ed88a6aa96c5c93fdf0e98eee7847b67e703815a163934a55be6e9ce5976047 2024-09-18T08:02:46.500 DEBUG:teuthology.orchestra.run.smithi050:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 779349ca-7593-11ef-bceb-c7b262605968 -- rbd pool init datapool 2024-09-18T08:02:46.730 INFO:journalctl@ceph.rgw.foo.a.smithi007.stdout:Sep 18 08:02:46 smithi007 systemd[1]: Started Ceph rgw.foo.a for 779349ca-7593-11ef-bceb-c7b262605968. 2024-09-18T08:02:46.757 INFO:teuthology.orchestra.run.smithi050.stderr:Inferring config /var/lib/ceph/779349ca-7593-11ef-bceb-c7b262605968/mon.b/config 2024-09-18T08:02:47.139 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:02:46 smithi050 ceph-mon[32371]: pgmap v193: 1 pgs: 1 active+clean; 577 KiB data, 221 MiB used, 715 GiB / 715 GiB avail 2024-09-18T08:02:47.140 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:02:46 smithi050 ceph-mon[32371]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "datapool", "pg_num": 3, "pgp_num": 3, "pool_type": "replicated"}]': finished 2024-09-18T08:02:47.140 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:02:46 smithi050 ceph-mon[32371]: osdmap e60: 8 total, 8 up, 8 in 2024-09-18T08:02:47.140 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:02:46 smithi050 ceph-mon[32371]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' 2024-09-18T08:02:47.140 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:02:46 smithi050 ceph-mon[32371]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' 2024-09-18T08:02:47.140 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:02:46 smithi050 ceph-mon[32371]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' 2024-09-18T08:02:47.140 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:02:46 smithi050 ceph-mon[32371]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' 2024-09-18T08:02:47.140 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:02:46 smithi050 ceph-mon[32371]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' 2024-09-18T08:02:47.140 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:02:46 smithi050 ceph-mon[32371]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-18T08:02:47.173 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:02:46 smithi007 ceph-mon[39249]: pgmap v193: 1 pgs: 1 active+clean; 577 KiB data, 221 MiB used, 715 GiB / 715 GiB avail 2024-09-18T08:02:47.173 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:02:46 smithi007 ceph-mon[39249]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "datapool", "pg_num": 3, "pgp_num": 3, "pool_type": "replicated"}]': finished 2024-09-18T08:02:47.173 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:02:46 smithi007 ceph-mon[39249]: osdmap e60: 8 total, 8 up, 8 in 2024-09-18T08:02:47.173 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:02:46 smithi007 ceph-mon[39249]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' 2024-09-18T08:02:47.173 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:02:46 smithi007 ceph-mon[39249]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' 2024-09-18T08:02:47.173 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:02:46 smithi007 ceph-mon[39249]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' 2024-09-18T08:02:47.173 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:02:46 smithi007 ceph-mon[39249]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' 2024-09-18T08:02:47.173 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:02:46 smithi007 ceph-mon[39249]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' 2024-09-18T08:02:47.173 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:02:46 smithi007 ceph-mon[39249]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-18T08:02:47.174 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:02:46 smithi007 ceph-mon[29459]: pgmap v193: 1 pgs: 1 active+clean; 577 KiB data, 221 MiB used, 715 GiB / 715 GiB avail 2024-09-18T08:02:47.174 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:02:46 smithi007 ceph-mon[29459]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "datapool", "pg_num": 3, "pgp_num": 3, "pool_type": "replicated"}]': finished 2024-09-18T08:02:47.174 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:02:46 smithi007 ceph-mon[29459]: osdmap e60: 8 total, 8 up, 8 in 2024-09-18T08:02:47.174 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:02:46 smithi007 ceph-mon[29459]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' 2024-09-18T08:02:47.174 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:02:46 smithi007 ceph-mon[29459]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' 2024-09-18T08:02:47.174 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:02:46 smithi007 ceph-mon[29459]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' 2024-09-18T08:02:47.174 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:02:46 smithi007 ceph-mon[29459]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' 2024-09-18T08:02:47.174 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:02:46 smithi007 ceph-mon[29459]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' 2024-09-18T08:02:47.174 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:02:46 smithi007 ceph-mon[29459]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-18T08:02:48.156 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:02:47 smithi050 ceph-mon[32371]: Saving service rgw.foo.a spec with placement smithi007=foo.a;count:1 2024-09-18T08:02:48.156 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:02:47 smithi050 ceph-mon[32371]: osdmap e61: 8 total, 8 up, 8 in 2024-09-18T08:02:48.156 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:02:47 smithi050 ceph-mon[32371]: from='client.? 172.21.15.7:0/3761561647' entity='client.rgw.foo.a' cmd={"prefix": "osd pool application enable","pool": ".rgw.root","app": "rgw"} : dispatch 2024-09-18T08:02:48.156 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:02:47 smithi050 ceph-mon[32371]: from='client.? 172.21.15.50:0/2527459249' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "datapool","app": "rbd"} : dispatch 2024-09-18T08:02:48.156 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:02:47 smithi050 ceph-mon[32371]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "datapool","app": "rbd"} : dispatch 2024-09-18T08:02:48.157 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:02:47 smithi050 ceph-mon[32371]: purged_snaps scrub starts 2024-09-18T08:02:48.157 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:02:47 smithi050 ceph-mon[32371]: purged_snaps scrub ok 2024-09-18T08:02:48.157 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:02:47 smithi050 ceph-mon[32371]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' 2024-09-18T08:02:48.157 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:02:47 smithi050 ceph-mon[32371]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' 2024-09-18T08:02:48.157 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:02:47 smithi050 ceph-mon[32371]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-18T08:02:48.157 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:02:47 smithi050 ceph-mon[32371]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-18T08:02:48.157 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:02:47 smithi050 ceph-mon[32371]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' 2024-09-18T08:02:48.157 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:02:47 smithi050 ceph-mon[32371]: from='client.? 172.21.15.7:0/3761561647' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool application enable","pool": ".rgw.root","app": "rgw"}]': finished 2024-09-18T08:02:48.157 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:02:47 smithi050 ceph-mon[32371]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "datapool","app": "rbd"}]': finished 2024-09-18T08:02:48.157 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:02:47 smithi050 ceph-mon[32371]: osdmap e62: 8 total, 8 up, 8 in 2024-09-18T08:02:48.231 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:02:47 smithi007 ceph-mon[39249]: Saving service rgw.foo.a spec with placement smithi007=foo.a;count:1 2024-09-18T08:02:48.231 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:02:47 smithi007 ceph-mon[39249]: osdmap e61: 8 total, 8 up, 8 in 2024-09-18T08:02:48.231 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:02:47 smithi007 ceph-mon[39249]: from='client.? 172.21.15.7:0/3761561647' entity='client.rgw.foo.a' cmd={"prefix": "osd pool application enable","pool": ".rgw.root","app": "rgw"} : dispatch 2024-09-18T08:02:48.231 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:02:47 smithi007 ceph-mon[39249]: from='client.? 172.21.15.50:0/2527459249' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "datapool","app": "rbd"} : dispatch 2024-09-18T08:02:48.231 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:02:47 smithi007 ceph-mon[39249]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "datapool","app": "rbd"} : dispatch 2024-09-18T08:02:48.231 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:02:47 smithi007 ceph-mon[39249]: purged_snaps scrub starts 2024-09-18T08:02:48.231 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:02:47 smithi007 ceph-mon[39249]: purged_snaps scrub ok 2024-09-18T08:02:48.231 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:02:47 smithi007 ceph-mon[39249]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' 2024-09-18T08:02:48.231 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:02:47 smithi007 ceph-mon[39249]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' 2024-09-18T08:02:48.232 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:02:47 smithi007 ceph-mon[39249]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-18T08:02:48.232 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:02:47 smithi007 ceph-mon[39249]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-18T08:02:48.232 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:02:47 smithi007 ceph-mon[39249]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' 2024-09-18T08:02:48.232 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:02:47 smithi007 ceph-mon[39249]: from='client.? 172.21.15.7:0/3761561647' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool application enable","pool": ".rgw.root","app": "rgw"}]': finished 2024-09-18T08:02:48.232 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:02:47 smithi007 ceph-mon[39249]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "datapool","app": "rbd"}]': finished 2024-09-18T08:02:48.232 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:02:47 smithi007 ceph-mon[39249]: osdmap e62: 8 total, 8 up, 8 in 2024-09-18T08:02:48.232 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:02:47 smithi007 ceph-mon[29459]: Saving service rgw.foo.a spec with placement smithi007=foo.a;count:1 2024-09-18T08:02:48.232 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:02:47 smithi007 ceph-mon[29459]: osdmap e61: 8 total, 8 up, 8 in 2024-09-18T08:02:48.232 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:02:47 smithi007 ceph-mon[29459]: from='client.? 172.21.15.7:0/3761561647' entity='client.rgw.foo.a' cmd={"prefix": "osd pool application enable","pool": ".rgw.root","app": "rgw"} : dispatch 2024-09-18T08:02:48.232 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:02:47 smithi007 ceph-mon[29459]: from='client.? 172.21.15.50:0/2527459249' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "datapool","app": "rbd"} : dispatch 2024-09-18T08:02:48.232 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:02:47 smithi007 ceph-mon[29459]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "datapool","app": "rbd"} : dispatch 2024-09-18T08:02:48.233 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:02:47 smithi007 ceph-mon[29459]: purged_snaps scrub starts 2024-09-18T08:02:48.233 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:02:47 smithi007 ceph-mon[29459]: purged_snaps scrub ok 2024-09-18T08:02:48.233 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:02:47 smithi007 ceph-mon[29459]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' 2024-09-18T08:02:48.233 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:02:47 smithi007 ceph-mon[29459]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' 2024-09-18T08:02:48.233 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:02:47 smithi007 ceph-mon[29459]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-18T08:02:48.233 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:02:47 smithi007 ceph-mon[29459]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-18T08:02:48.233 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:02:47 smithi007 ceph-mon[29459]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' 2024-09-18T08:02:48.233 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:02:47 smithi007 ceph-mon[29459]: from='client.? 172.21.15.7:0/3761561647' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool application enable","pool": ".rgw.root","app": "rgw"}]': finished 2024-09-18T08:02:48.233 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:02:47 smithi007 ceph-mon[29459]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "datapool","app": "rbd"}]': finished 2024-09-18T08:02:48.233 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:02:47 smithi007 ceph-mon[29459]: osdmap e62: 8 total, 8 up, 8 in 2024-09-18T08:02:49.156 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:02:48 smithi050 ceph-mon[32371]: pgmap v196: 36 pgs: 3 active+clean, 12 creating+peering, 21 unknown; 577 KiB data, 221 MiB used, 715 GiB / 715 GiB avail 2024-09-18T08:02:49.156 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:02:48 smithi050 ceph-mon[32371]: Checking dashboard <-> RGW credentials 2024-09-18T08:02:49.156 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:02:48 smithi050 ceph-mon[32371]: Health check failed: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-18T08:02:49.230 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:02:48 smithi007 ceph-mon[29459]: pgmap v196: 36 pgs: 3 active+clean, 12 creating+peering, 21 unknown; 577 KiB data, 221 MiB used, 715 GiB / 715 GiB avail 2024-09-18T08:02:49.230 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:02:48 smithi007 ceph-mon[29459]: Checking dashboard <-> RGW credentials 2024-09-18T08:02:49.231 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:02:48 smithi007 ceph-mon[29459]: Health check failed: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-18T08:02:49.231 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:02:48 smithi007 ceph-mon[39249]: pgmap v196: 36 pgs: 3 active+clean, 12 creating+peering, 21 unknown; 577 KiB data, 221 MiB used, 715 GiB / 715 GiB avail 2024-09-18T08:02:49.231 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:02:48 smithi007 ceph-mon[39249]: Checking dashboard <-> RGW credentials 2024-09-18T08:02:49.231 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:02:48 smithi007 ceph-mon[39249]: Health check failed: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-18T08:02:50.156 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:02:49 smithi050 ceph-mon[32371]: osdmap e63: 8 total, 8 up, 8 in 2024-09-18T08:02:50.156 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:02:49 smithi050 ceph-mon[32371]: from='client.? 172.21.15.7:0/1364443164' entity='mgr.y' cmd={"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"} : dispatch 2024-09-18T08:02:50.156 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:02:49 smithi050 ceph-mon[32371]: from='client.? 172.21.15.7:0/219168083' entity='client.rgw.foo.a' cmd={"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"} : dispatch 2024-09-18T08:02:50.156 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:02:49 smithi050 ceph-mon[32371]: from='client.? ' entity='client.rgw.foo.a' cmd={"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"} : dispatch 2024-09-18T08:02:50.156 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:02:49 smithi050 ceph-mon[32371]: from='client.? 172.21.15.7:0/1364443164' entity='mgr.y' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]': finished 2024-09-18T08:02:50.231 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:02:49 smithi007 ceph-mon[29459]: osdmap e63: 8 total, 8 up, 8 in 2024-09-18T08:02:50.231 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:02:49 smithi007 ceph-mon[29459]: from='client.? 172.21.15.7:0/1364443164' entity='mgr.y' cmd={"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"} : dispatch 2024-09-18T08:02:50.231 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:02:49 smithi007 ceph-mon[29459]: from='client.? 172.21.15.7:0/219168083' entity='client.rgw.foo.a' cmd={"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"} : dispatch 2024-09-18T08:02:50.231 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:02:49 smithi007 ceph-mon[29459]: from='client.? ' entity='client.rgw.foo.a' cmd={"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"} : dispatch 2024-09-18T08:02:50.231 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:02:49 smithi007 ceph-mon[29459]: from='client.? 172.21.15.7:0/1364443164' entity='mgr.y' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]': finished 2024-09-18T08:02:50.231 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:02:49 smithi007 ceph-mon[39249]: osdmap e63: 8 total, 8 up, 8 in 2024-09-18T08:02:50.231 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:02:49 smithi007 ceph-mon[39249]: from='client.? 172.21.15.7:0/1364443164' entity='mgr.y' cmd={"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"} : dispatch 2024-09-18T08:02:50.232 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:02:49 smithi007 ceph-mon[39249]: from='client.? 172.21.15.7:0/219168083' entity='client.rgw.foo.a' cmd={"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"} : dispatch 2024-09-18T08:02:50.232 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:02:49 smithi007 ceph-mon[39249]: from='client.? ' entity='client.rgw.foo.a' cmd={"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"} : dispatch 2024-09-18T08:02:50.232 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:02:49 smithi007 ceph-mon[39249]: from='client.? 172.21.15.7:0/1364443164' entity='mgr.y' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]': finished 2024-09-18T08:02:50.550 DEBUG:teuthology.orchestra.run.smithi050:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 779349ca-7593-11ef-bceb-c7b262605968 -- ceph orch apply iscsi datapool admin admin --trusted_ip_list 172.21.15.50 --placement '1;smithi050=iscsi.a' 2024-09-18T08:02:50.809 INFO:teuthology.orchestra.run.smithi050.stderr:Inferring config /var/lib/ceph/779349ca-7593-11ef-bceb-c7b262605968/mon.b/config 2024-09-18T08:02:51.156 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:02:50 smithi050 ceph-mon[32371]: pgmap v199: 68 pgs: 9 active+clean, 15 creating+peering, 44 unknown; 577 KiB data, 221 MiB used, 715 GiB / 715 GiB avail 2024-09-18T08:02:51.156 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:02:50 smithi050 ceph-mon[32371]: from='client.? ' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]': finished 2024-09-18T08:02:51.156 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:02:50 smithi050 ceph-mon[32371]: osdmap e64: 8 total, 8 up, 8 in 2024-09-18T08:02:51.231 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:02:50 smithi007 ceph-mon[29459]: pgmap v199: 68 pgs: 9 active+clean, 15 creating+peering, 44 unknown; 577 KiB data, 221 MiB used, 715 GiB / 715 GiB avail 2024-09-18T08:02:51.231 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:02:50 smithi007 ceph-mon[29459]: from='client.? ' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]': finished 2024-09-18T08:02:51.231 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:02:50 smithi007 ceph-mon[29459]: osdmap e64: 8 total, 8 up, 8 in 2024-09-18T08:02:51.231 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:02:50 smithi007 ceph-mon[39249]: pgmap v199: 68 pgs: 9 active+clean, 15 creating+peering, 44 unknown; 577 KiB data, 221 MiB used, 715 GiB / 715 GiB avail 2024-09-18T08:02:51.231 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:02:50 smithi007 ceph-mon[39249]: from='client.? ' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]': finished 2024-09-18T08:02:51.231 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:02:50 smithi007 ceph-mon[39249]: osdmap e64: 8 total, 8 up, 8 in 2024-09-18T08:02:51.476 INFO:teuthology.orchestra.run.smithi050.stdout:Scheduled iscsi.datapool update... 2024-09-18T08:02:52.110 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:02:51 smithi050 ceph-mon[32371]: osdmap e65: 8 total, 8 up, 8 in 2024-09-18T08:02:52.110 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:02:51 smithi050 ceph-mon[32371]: from='client.? 172.21.15.7:0/219168083' entity='client.rgw.foo.a' cmd={"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"} : dispatch 2024-09-18T08:02:52.110 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:02:51 smithi050 ceph-mon[32371]: from='client.? ' entity='client.rgw.foo.a' cmd={"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"} : dispatch 2024-09-18T08:02:52.110 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:02:51 smithi050 ceph-mon[32371]: from='client.? 172.21.15.7:0/1364443164' entity='mgr.y' cmd={"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"} : dispatch 2024-09-18T08:02:52.110 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:02:51 smithi050 ceph-mon[32371]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' 2024-09-18T08:02:52.110 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:02:51 smithi050 ceph-mon[32371]: from='client.? ' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]': finished 2024-09-18T08:02:52.110 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:02:51 smithi050 ceph-mon[32371]: from='client.? 172.21.15.7:0/1364443164' entity='mgr.y' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]': finished 2024-09-18T08:02:52.110 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:02:51 smithi050 ceph-mon[32371]: osdmap e66: 8 total, 8 up, 8 in 2024-09-18T08:02:52.139 INFO:tasks.cephadm:Distributing iscsi-gateway.cfg... 2024-09-18T08:02:52.139 DEBUG:teuthology.orchestra.run.smithi007:> set -ex 2024-09-18T08:02:52.139 DEBUG:teuthology.orchestra.run.smithi007:> sudo dd of=/etc/ceph/iscsi-gateway.cfg 2024-09-18T08:02:52.167 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:02:51 smithi007 ceph-mon[39249]: osdmap e65: 8 total, 8 up, 8 in 2024-09-18T08:02:52.168 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:02:51 smithi007 ceph-mon[39249]: from='client.? 172.21.15.7:0/219168083' entity='client.rgw.foo.a' cmd={"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"} : dispatch 2024-09-18T08:02:52.168 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:02:51 smithi007 ceph-mon[39249]: from='client.? ' entity='client.rgw.foo.a' cmd={"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"} : dispatch 2024-09-18T08:02:52.168 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:02:51 smithi007 ceph-mon[39249]: from='client.? 172.21.15.7:0/1364443164' entity='mgr.y' cmd={"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"} : dispatch 2024-09-18T08:02:52.168 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:02:51 smithi007 ceph-mon[39249]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' 2024-09-18T08:02:52.168 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:02:51 smithi007 ceph-mon[39249]: from='client.? ' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]': finished 2024-09-18T08:02:52.168 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:02:51 smithi007 ceph-mon[39249]: from='client.? 172.21.15.7:0/1364443164' entity='mgr.y' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]': finished 2024-09-18T08:02:52.168 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:02:51 smithi007 ceph-mon[39249]: osdmap e66: 8 total, 8 up, 8 in 2024-09-18T08:02:52.168 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:02:51 smithi007 ceph-mon[29459]: osdmap e65: 8 total, 8 up, 8 in 2024-09-18T08:02:52.168 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:02:51 smithi007 ceph-mon[29459]: from='client.? 172.21.15.7:0/219168083' entity='client.rgw.foo.a' cmd={"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"} : dispatch 2024-09-18T08:02:52.169 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:02:51 smithi007 ceph-mon[29459]: from='client.? ' entity='client.rgw.foo.a' cmd={"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"} : dispatch 2024-09-18T08:02:52.169 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:02:51 smithi007 ceph-mon[29459]: from='client.? 172.21.15.7:0/1364443164' entity='mgr.y' cmd={"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"} : dispatch 2024-09-18T08:02:52.169 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:02:51 smithi007 ceph-mon[29459]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' 2024-09-18T08:02:52.169 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:02:51 smithi007 ceph-mon[29459]: from='client.? ' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]': finished 2024-09-18T08:02:52.169 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:02:51 smithi007 ceph-mon[29459]: from='client.? 172.21.15.7:0/1364443164' entity='mgr.y' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]': finished 2024-09-18T08:02:52.169 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:02:51 smithi007 ceph-mon[29459]: osdmap e66: 8 total, 8 up, 8 in 2024-09-18T08:02:52.172 DEBUG:teuthology.orchestra.run.smithi050:> set -ex 2024-09-18T08:02:52.173 DEBUG:teuthology.orchestra.run.smithi050:> sudo dd of=/etc/ceph/iscsi-gateway.cfg 2024-09-18T08:02:52.200 DEBUG:teuthology.orchestra.run.smithi050:iscsi.iscsi.a> sudo journalctl -f -n 0 -u ceph-779349ca-7593-11ef-bceb-c7b262605968@iscsi.iscsi.a.service 2024-09-18T08:02:52.243 INFO:tasks.cephadm:Adding prometheus.a on smithi050 2024-09-18T08:02:52.243 DEBUG:teuthology.orchestra.run.smithi050:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 779349ca-7593-11ef-bceb-c7b262605968 -- ceph orch apply prometheus '1;smithi050=a' 2024-09-18T08:02:52.533 INFO:teuthology.orchestra.run.smithi050.stderr:Inferring config /var/lib/ceph/779349ca-7593-11ef-bceb-c7b262605968/mon.b/config 2024-09-18T08:02:53.156 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:02:52 smithi050 ceph-mon[32371]: pgmap v202: 100 pgs: 15 active+clean, 20 creating+peering, 65 unknown; 578 KiB data, 221 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 511 B/s wr, 1 op/s 2024-09-18T08:02:53.156 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:02:52 smithi050 ceph-mon[32371]: from='client.24370 -' entity='client.admin' cmd=[{"prefix": "orch apply iscsi", "pool": "datapool", "api_user": "admin", "api_password": "admin", "trusted_ip_list": "172.21.15.50", "placement": "1;smithi050=iscsi.a", "target": ["mon-mgr", ""]}]: dispatch 2024-09-18T08:02:53.157 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:02:52 smithi050 ceph-mon[32371]: Saving service iscsi.datapool spec with placement smithi050=iscsi.a;count:1 2024-09-18T08:02:53.157 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:02:52 smithi050 ceph-mon[32371]: osdmap e67: 8 total, 8 up, 8 in 2024-09-18T08:02:53.157 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:02:52 smithi050 ceph-mon[32371]: from='client.? 172.21.15.7:0/219168083' entity='client.rgw.foo.a' cmd={"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"} : dispatch 2024-09-18T08:02:53.157 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:02:52 smithi050 ceph-mon[32371]: from='client.? 172.21.15.7:0/1364443164' entity='mgr.y' cmd={"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"} : dispatch 2024-09-18T08:02:53.157 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:02:52 smithi050 ceph-mon[32371]: from='client.? ' entity='client.rgw.foo.a' cmd={"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"} : dispatch 2024-09-18T08:02:53.200 INFO:teuthology.orchestra.run.smithi050.stdout:Scheduled prometheus update... 2024-09-18T08:02:53.231 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:02:52 smithi007 ceph-mon[29459]: pgmap v202: 100 pgs: 15 active+clean, 20 creating+peering, 65 unknown; 578 KiB data, 221 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 511 B/s wr, 1 op/s 2024-09-18T08:02:53.231 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:02:52 smithi007 ceph-mon[29459]: from='client.24370 -' entity='client.admin' cmd=[{"prefix": "orch apply iscsi", "pool": "datapool", "api_user": "admin", "api_password": "admin", "trusted_ip_list": "172.21.15.50", "placement": "1;smithi050=iscsi.a", "target": ["mon-mgr", ""]}]: dispatch 2024-09-18T08:02:53.231 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:02:52 smithi007 ceph-mon[29459]: Saving service iscsi.datapool spec with placement smithi050=iscsi.a;count:1 2024-09-18T08:02:53.231 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:02:52 smithi007 ceph-mon[29459]: osdmap e67: 8 total, 8 up, 8 in 2024-09-18T08:02:53.231 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:02:52 smithi007 ceph-mon[29459]: from='client.? 172.21.15.7:0/219168083' entity='client.rgw.foo.a' cmd={"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"} : dispatch 2024-09-18T08:02:53.231 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:02:52 smithi007 ceph-mon[29459]: from='client.? 172.21.15.7:0/1364443164' entity='mgr.y' cmd={"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"} : dispatch 2024-09-18T08:02:53.231 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:02:52 smithi007 ceph-mon[29459]: from='client.? ' entity='client.rgw.foo.a' cmd={"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"} : dispatch 2024-09-18T08:02:53.231 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:02:52 smithi007 ceph-mon[39249]: pgmap v202: 100 pgs: 15 active+clean, 20 creating+peering, 65 unknown; 578 KiB data, 221 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 511 B/s wr, 1 op/s 2024-09-18T08:02:53.232 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:02:52 smithi007 ceph-mon[39249]: from='client.24370 -' entity='client.admin' cmd=[{"prefix": "orch apply iscsi", "pool": "datapool", "api_user": "admin", "api_password": "admin", "trusted_ip_list": "172.21.15.50", "placement": "1;smithi050=iscsi.a", "target": ["mon-mgr", ""]}]: dispatch 2024-09-18T08:02:53.232 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:02:52 smithi007 ceph-mon[39249]: Saving service iscsi.datapool spec with placement smithi050=iscsi.a;count:1 2024-09-18T08:02:53.232 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:02:52 smithi007 ceph-mon[39249]: osdmap e67: 8 total, 8 up, 8 in 2024-09-18T08:02:53.232 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:02:52 smithi007 ceph-mon[39249]: from='client.? 172.21.15.7:0/219168083' entity='client.rgw.foo.a' cmd={"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"} : dispatch 2024-09-18T08:02:53.232 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:02:52 smithi007 ceph-mon[39249]: from='client.? 172.21.15.7:0/1364443164' entity='mgr.y' cmd={"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"} : dispatch 2024-09-18T08:02:53.232 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:02:52 smithi007 ceph-mon[39249]: from='client.? ' entity='client.rgw.foo.a' cmd={"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"} : dispatch 2024-09-18T08:02:53.864 DEBUG:teuthology.orchestra.run.smithi050:prometheus.a> sudo journalctl -f -n 0 -u ceph-779349ca-7593-11ef-bceb-c7b262605968@prometheus.a.service 2024-09-18T08:02:53.867 INFO:tasks.cephadm:Adding node-exporter.a on smithi007 2024-09-18T08:02:53.867 INFO:tasks.cephadm:Adding node-exporter.b on smithi050 2024-09-18T08:02:53.867 DEBUG:teuthology.orchestra.run.smithi050:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 779349ca-7593-11ef-bceb-c7b262605968 -- ceph orch apply node-exporter '2;smithi007=a;smithi050=b' 2024-09-18T08:02:54.186 INFO:teuthology.orchestra.run.smithi050.stderr:Inferring config /var/lib/ceph/779349ca-7593-11ef-bceb-c7b262605968/mon.b/config 2024-09-18T08:02:54.477 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:02:54 smithi050 ceph-mon[32371]: from='client.24376 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "prometheus", "placement": "1;smithi050=a", "target": ["mon-mgr", ""]}]: dispatch 2024-09-18T08:02:54.478 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:02:54 smithi050 ceph-mon[32371]: Saving service prometheus spec with placement smithi050=a;count:1 2024-09-18T08:02:54.478 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:02:54 smithi050 ceph-mon[32371]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' 2024-09-18T08:02:54.478 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:02:54 smithi050 ceph-mon[32371]: pgmap v205: 132 pgs: 83 active+clean, 19 creating+peering, 30 unknown; 578 KiB data, 221 MiB used, 715 GiB / 715 GiB avail; 5.0 KiB/s rd, 2.5 KiB/s wr, 9 op/s 2024-09-18T08:02:54.478 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:02:54 smithi050 ceph-mon[32371]: from='client.? 172.21.15.7:0/1364443164' entity='mgr.y' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]': finished 2024-09-18T08:02:54.478 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:02:54 smithi050 ceph-mon[32371]: from='client.? ' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]': finished 2024-09-18T08:02:54.478 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:02:54 smithi050 ceph-mon[32371]: osdmap e68: 8 total, 8 up, 8 in 2024-09-18T08:02:54.479 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:02:54 smithi050 ceph-mon[32371]: from='client.? 172.21.15.7:0/219168083' entity='client.rgw.foo.a' cmd={"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"} : dispatch 2024-09-18T08:02:54.479 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:02:54 smithi050 ceph-mon[32371]: from='client.? 172.21.15.7:0/1364443164' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"} : dispatch 2024-09-18T08:02:54.479 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:02:54 smithi050 ceph-mon[32371]: from='client.? ' entity='client.rgw.foo.a' cmd={"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"} : dispatch 2024-09-18T08:02:54.479 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:02:54 smithi050 ceph-mon[32371]: Health check update: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-18T08:02:54.479 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:02:54 smithi050 ceph-mon[32371]: from='client.? 172.21.15.7:0/1364443164' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]': finished 2024-09-18T08:02:54.479 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:02:54 smithi050 ceph-mon[32371]: from='client.? ' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]': finished 2024-09-18T08:02:54.480 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:02:54 smithi050 ceph-mon[32371]: osdmap e69: 8 total, 8 up, 8 in 2024-09-18T08:02:54.481 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:02:54 smithi007 ceph-mon[29459]: from='client.24376 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "prometheus", "placement": "1;smithi050=a", "target": ["mon-mgr", ""]}]: dispatch 2024-09-18T08:02:54.481 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:02:54 smithi007 ceph-mon[29459]: Saving service prometheus spec with placement smithi050=a;count:1 2024-09-18T08:02:54.481 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:02:54 smithi007 ceph-mon[29459]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' 2024-09-18T08:02:54.481 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:02:54 smithi007 ceph-mon[29459]: pgmap v205: 132 pgs: 83 active+clean, 19 creating+peering, 30 unknown; 578 KiB data, 221 MiB used, 715 GiB / 715 GiB avail; 5.0 KiB/s rd, 2.5 KiB/s wr, 9 op/s 2024-09-18T08:02:54.481 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:02:54 smithi007 ceph-mon[29459]: from='client.? 172.21.15.7:0/1364443164' entity='mgr.y' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]': finished 2024-09-18T08:02:54.482 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:02:54 smithi007 ceph-mon[29459]: from='client.? ' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]': finished 2024-09-18T08:02:54.482 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:02:54 smithi007 ceph-mon[29459]: osdmap e68: 8 total, 8 up, 8 in 2024-09-18T08:02:54.482 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:02:54 smithi007 ceph-mon[29459]: from='client.? 172.21.15.7:0/219168083' entity='client.rgw.foo.a' cmd={"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"} : dispatch 2024-09-18T08:02:54.482 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:02:54 smithi007 ceph-mon[29459]: from='client.? 172.21.15.7:0/1364443164' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"} : dispatch 2024-09-18T08:02:54.482 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:02:54 smithi007 ceph-mon[29459]: from='client.? ' entity='client.rgw.foo.a' cmd={"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"} : dispatch 2024-09-18T08:02:54.483 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:02:54 smithi007 ceph-mon[29459]: Health check update: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-18T08:02:54.483 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:02:54 smithi007 ceph-mon[29459]: from='client.? 172.21.15.7:0/1364443164' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]': finished 2024-09-18T08:02:54.483 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:02:54 smithi007 ceph-mon[29459]: from='client.? ' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]': finished 2024-09-18T08:02:54.483 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:02:54 smithi007 ceph-mon[29459]: osdmap e69: 8 total, 8 up, 8 in 2024-09-18T08:02:54.483 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:02:54 smithi007 ceph-mon[39249]: from='client.24376 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "prometheus", "placement": "1;smithi050=a", "target": ["mon-mgr", ""]}]: dispatch 2024-09-18T08:02:54.484 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:02:54 smithi007 ceph-mon[39249]: Saving service prometheus spec with placement smithi050=a;count:1 2024-09-18T08:02:54.484 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:02:54 smithi007 ceph-mon[39249]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' 2024-09-18T08:02:54.484 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:02:54 smithi007 ceph-mon[39249]: pgmap v205: 132 pgs: 83 active+clean, 19 creating+peering, 30 unknown; 578 KiB data, 221 MiB used, 715 GiB / 715 GiB avail; 5.0 KiB/s rd, 2.5 KiB/s wr, 9 op/s 2024-09-18T08:02:54.484 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:02:54 smithi007 ceph-mon[39249]: from='client.? 172.21.15.7:0/1364443164' entity='mgr.y' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]': finished 2024-09-18T08:02:54.484 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:02:54 smithi007 ceph-mon[39249]: from='client.? ' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]': finished 2024-09-18T08:02:54.485 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:02:54 smithi007 ceph-mon[39249]: osdmap e68: 8 total, 8 up, 8 in 2024-09-18T08:02:54.485 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:02:54 smithi007 ceph-mon[39249]: from='client.? 172.21.15.7:0/219168083' entity='client.rgw.foo.a' cmd={"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"} : dispatch 2024-09-18T08:02:54.485 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:02:54 smithi007 ceph-mon[39249]: from='client.? 172.21.15.7:0/1364443164' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"} : dispatch 2024-09-18T08:02:54.485 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:02:54 smithi007 ceph-mon[39249]: from='client.? ' entity='client.rgw.foo.a' cmd={"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"} : dispatch 2024-09-18T08:02:54.485 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:02:54 smithi007 ceph-mon[39249]: Health check update: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-18T08:02:54.485 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:02:54 smithi007 ceph-mon[39249]: from='client.? 172.21.15.7:0/1364443164' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]': finished 2024-09-18T08:02:54.486 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:02:54 smithi007 ceph-mon[39249]: from='client.? ' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]': finished 2024-09-18T08:02:54.486 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:02:54 smithi007 ceph-mon[39249]: osdmap e69: 8 total, 8 up, 8 in 2024-09-18T08:02:54.915 INFO:teuthology.orchestra.run.smithi050.stdout:Scheduled node-exporter update... 2024-09-18T08:02:55.406 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:02:55 smithi050 ceph-mon[32371]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' 2024-09-18T08:02:55.406 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:02:55 smithi050 ceph-mon[32371]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' 2024-09-18T08:02:55.406 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:02:55 smithi050 ceph-mon[32371]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' 2024-09-18T08:02:55.406 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:02:55 smithi050 ceph-mon[32371]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' cmd={"prefix": "auth get-or-create", "entity": "client.iscsi.iscsi.a", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]} : dispatch 2024-09-18T08:02:55.406 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:02:55 smithi050 ceph-mon[32371]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "client.iscsi.iscsi.a", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]': finished 2024-09-18T08:02:55.407 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:02:55 smithi050 ceph-mon[32371]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-18T08:02:55.407 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:02:55 smithi050 ceph-mon[32371]: Deploying daemon iscsi.iscsi.a on smithi050 2024-09-18T08:02:55.407 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:02:55 smithi050 ceph-mon[32371]: from='client.24394 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "node-exporter", "placement": "2;smithi007=a;smithi050=b", "target": ["mon-mgr", ""]}]: dispatch 2024-09-18T08:02:55.407 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:02:55 smithi050 ceph-mon[32371]: Saving service node-exporter spec with placement smithi007=a;smithi050=b;count:2 2024-09-18T08:02:55.407 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:02:55 smithi050 ceph-mon[32371]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' 2024-09-18T08:02:55.407 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:02:55 smithi050 ceph-mon[32371]: osdmap e70: 8 total, 8 up, 8 in 2024-09-18T08:02:55.731 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:02:55 smithi007 ceph-mon[29459]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' 2024-09-18T08:02:55.731 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:02:55 smithi007 ceph-mon[29459]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' 2024-09-18T08:02:55.731 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:02:55 smithi007 ceph-mon[29459]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' 2024-09-18T08:02:55.731 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:02:55 smithi007 ceph-mon[29459]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' cmd={"prefix": "auth get-or-create", "entity": "client.iscsi.iscsi.a", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]} : dispatch 2024-09-18T08:02:55.731 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:02:55 smithi007 ceph-mon[29459]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "client.iscsi.iscsi.a", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]': finished 2024-09-18T08:02:55.731 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:02:55 smithi007 ceph-mon[29459]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-18T08:02:55.731 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:02:55 smithi007 ceph-mon[29459]: Deploying daemon iscsi.iscsi.a on smithi050 2024-09-18T08:02:55.731 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:02:55 smithi007 ceph-mon[29459]: from='client.24394 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "node-exporter", "placement": "2;smithi007=a;smithi050=b", "target": ["mon-mgr", ""]}]: dispatch 2024-09-18T08:02:55.732 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:02:55 smithi007 ceph-mon[29459]: Saving service node-exporter spec with placement smithi007=a;smithi050=b;count:2 2024-09-18T08:02:55.732 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:02:55 smithi007 ceph-mon[29459]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' 2024-09-18T08:02:55.732 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:02:55 smithi007 ceph-mon[29459]: osdmap e70: 8 total, 8 up, 8 in 2024-09-18T08:02:55.732 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:02:55 smithi007 ceph-mon[39249]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' 2024-09-18T08:02:55.732 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:02:55 smithi007 ceph-mon[39249]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' 2024-09-18T08:02:55.732 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:02:55 smithi007 ceph-mon[39249]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' 2024-09-18T08:02:55.732 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:02:55 smithi007 ceph-mon[39249]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' cmd={"prefix": "auth get-or-create", "entity": "client.iscsi.iscsi.a", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]} : dispatch 2024-09-18T08:02:55.732 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:02:55 smithi007 ceph-mon[39249]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "client.iscsi.iscsi.a", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]': finished 2024-09-18T08:02:55.732 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:02:55 smithi007 ceph-mon[39249]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-18T08:02:55.732 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:02:55 smithi007 ceph-mon[39249]: Deploying daemon iscsi.iscsi.a on smithi050 2024-09-18T08:02:55.732 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:02:55 smithi007 ceph-mon[39249]: from='client.24394 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "node-exporter", "placement": "2;smithi007=a;smithi050=b", "target": ["mon-mgr", ""]}]: dispatch 2024-09-18T08:02:55.733 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:02:55 smithi007 ceph-mon[39249]: Saving service node-exporter spec with placement smithi007=a;smithi050=b;count:2 2024-09-18T08:02:55.733 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:02:55 smithi007 ceph-mon[39249]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' 2024-09-18T08:02:55.733 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:02:55 smithi007 ceph-mon[39249]: osdmap e70: 8 total, 8 up, 8 in 2024-09-18T08:02:55.813 DEBUG:teuthology.orchestra.run.smithi007:node-exporter.a> sudo journalctl -f -n 0 -u ceph-779349ca-7593-11ef-bceb-c7b262605968@node-exporter.a.service 2024-09-18T08:02:55.816 DEBUG:teuthology.orchestra.run.smithi050:node-exporter.b> sudo journalctl -f -n 0 -u ceph-779349ca-7593-11ef-bceb-c7b262605968@node-exporter.b.service 2024-09-18T08:02:55.818 INFO:tasks.cephadm:Adding alertmanager.a on smithi007 2024-09-18T08:02:55.819 DEBUG:teuthology.orchestra.run.smithi050:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 779349ca-7593-11ef-bceb-c7b262605968 -- ceph orch apply alertmanager '1;smithi007=a' 2024-09-18T08:02:56.226 INFO:teuthology.orchestra.run.smithi050.stderr:Inferring config /var/lib/ceph/779349ca-7593-11ef-bceb-c7b262605968/mon.b/config 2024-09-18T08:02:56.651 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:02:56 smithi050 ceph-mon[32371]: pgmap v209: 132 pgs: 96 active+clean, 14 creating+peering, 22 unknown; 579 KiB data, 221 MiB used, 715 GiB / 715 GiB avail; 25 KiB/s rd, 4.0 KiB/s wr, 58 op/s 2024-09-18T08:02:56.730 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:02:56 smithi007 ceph-mon[29459]: pgmap v209: 132 pgs: 96 active+clean, 14 creating+peering, 22 unknown; 579 KiB data, 221 MiB used, 715 GiB / 715 GiB avail; 25 KiB/s rd, 4.0 KiB/s wr, 58 op/s 2024-09-18T08:02:56.731 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:02:56 smithi007 ceph-mon[39249]: pgmap v209: 132 pgs: 96 active+clean, 14 creating+peering, 22 unknown; 579 KiB data, 221 MiB used, 715 GiB / 715 GiB avail; 25 KiB/s rd, 4.0 KiB/s wr, 58 op/s 2024-09-18T08:02:57.339 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:02:57 smithi050 ceph-mon[32371]: 1.0 scrub starts 2024-09-18T08:02:57.340 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:02:57 smithi050 ceph-mon[32371]: 1.0 scrub ok 2024-09-18T08:02:57.340 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:02:57 smithi050 ceph-mon[32371]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-09-18T08:02:57.340 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:02:57 smithi050 ceph-mon[32371]: Cluster is now healthy 2024-09-18T08:02:57.539 INFO:teuthology.orchestra.run.smithi050.stdout:Scheduled alertmanager update... 2024-09-18T08:02:57.632 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:02:57 smithi007 ceph-mon[29459]: 1.0 scrub starts 2024-09-18T08:02:57.633 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:02:57 smithi007 ceph-mon[29459]: 1.0 scrub ok 2024-09-18T08:02:57.633 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:02:57 smithi007 ceph-mon[29459]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-09-18T08:02:57.633 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:02:57 smithi007 ceph-mon[29459]: Cluster is now healthy 2024-09-18T08:02:57.633 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:02:57 smithi007 ceph-mon[39249]: 1.0 scrub starts 2024-09-18T08:02:57.633 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:02:57 smithi007 ceph-mon[39249]: 1.0 scrub ok 2024-09-18T08:02:57.633 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:02:57 smithi007 ceph-mon[39249]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-09-18T08:02:57.633 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:02:57 smithi007 ceph-mon[39249]: Cluster is now healthy 2024-09-18T08:02:57.655 INFO:journalctl@ceph.iscsi.iscsi.a.smithi050.stdout:Sep 18 08:02:57 smithi050 podman[57753]: 2024-09-18 08:02:57.564605981 +0000 UTC m=+0.121934458 container create 53112b8ba5efdba188bdb7f62f175a8b12cd5c24032ae7414cb046b8a68ce738 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:1f5b5a3734f665ef333266f39143ac3ef1eb1951b21258f2307e10332e2d7ac9, name=ceph-779349ca-7593-11ef-bceb-c7b262605968-iscsi-iscsi-a, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, GIT_CLEAN=True, io.buildah.version=1.37.2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240731, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 9 Base Image, ceph=True, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , RELEASE=wip-guits-main-2024-09-17-1213-e7fb7b5, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965) 2024-09-18T08:02:58.217 DEBUG:teuthology.orchestra.run.smithi007:alertmanager.a> sudo journalctl -f -n 0 -u ceph-779349ca-7593-11ef-bceb-c7b262605968@alertmanager.a.service 2024-09-18T08:02:58.220 INFO:tasks.cephadm:Adding grafana.a on smithi050 2024-09-18T08:02:58.220 DEBUG:teuthology.orchestra.run.smithi050:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 779349ca-7593-11ef-bceb-c7b262605968 -- ceph orch apply grafana '1;smithi050=a' 2024-09-18T08:02:58.537 INFO:journalctl@ceph.iscsi.iscsi.a.smithi050.stdout:Sep 18 08:02:58 smithi050 podman[57753]: 2024-09-18 08:02:58.312872503 +0000 UTC m=+0.870200987 container init 53112b8ba5efdba188bdb7f62f175a8b12cd5c24032ae7414cb046b8a68ce738 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:1f5b5a3734f665ef333266f39143ac3ef1eb1951b21258f2307e10332e2d7ac9, name=ceph-779349ca-7593-11ef-bceb-c7b262605968-iscsi-iscsi-a, ceph=True, GIT_BRANCH=HEAD, io.buildah.version=1.37.2, org.label-schema.build-date=20240731, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, RELEASE=wip-guits-main-2024-09-17-1213-e7fb7b5, GIT_CLEAN=True, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 9 Base Image, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=) 2024-09-18T08:02:58.537 INFO:journalctl@ceph.iscsi.iscsi.a.smithi050.stdout:Sep 18 08:02:58 smithi050 podman[57753]: 2024-09-18 08:02:58.316675831 +0000 UTC m=+0.874004309 container start 53112b8ba5efdba188bdb7f62f175a8b12cd5c24032ae7414cb046b8a68ce738 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:1f5b5a3734f665ef333266f39143ac3ef1eb1951b21258f2307e10332e2d7ac9, name=ceph-779349ca-7593-11ef-bceb-c7b262605968-iscsi-iscsi-a, ceph=True, io.buildah.version=1.37.2, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.build-date=20240731, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_CLEAN=True, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, GIT_BRANCH=HEAD, RELEASE=wip-guits-main-2024-09-17-1213-e7fb7b5, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-09-18T08:02:58.537 INFO:journalctl@ceph.iscsi.iscsi.a.smithi050.stdout:Sep 18 08:02:58 smithi050 bash[57753]: 53112b8ba5efdba188bdb7f62f175a8b12cd5c24032ae7414cb046b8a68ce738 2024-09-18T08:02:58.537 INFO:journalctl@ceph.iscsi.iscsi.a.smithi050.stdout:Sep 18 08:02:58 smithi050 systemd[1]: Started Ceph iscsi.iscsi.a for 779349ca-7593-11ef-bceb-c7b262605968. 2024-09-18T08:02:58.787 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:02:58 smithi050 ceph-mon[32371]: pgmap v210: 132 pgs: 124 active+clean, 5 creating+peering, 3 unknown; 580 KiB data, 222 MiB used, 715 GiB / 715 GiB avail; 47 KiB/s rd, 3.8 KiB/s wr, 114 op/s 2024-09-18T08:02:58.787 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:02:58 smithi050 ceph-mon[32371]: from='client.24400 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "alertmanager", "placement": "1;smithi007=a", "target": ["mon-mgr", ""]}]: dispatch 2024-09-18T08:02:58.787 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:02:58 smithi050 ceph-mon[32371]: Saving service alertmanager spec with placement smithi007=a;count:1 2024-09-18T08:02:58.788 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:02:58 smithi050 ceph-mon[32371]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' 2024-09-18T08:02:58.788 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:02:58 smithi050 ceph-mon[32371]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' 2024-09-18T08:02:58.788 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:02:58 smithi050 ceph-mon[32371]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' 2024-09-18T08:02:58.788 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:02:58 smithi050 ceph-mon[32371]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' 2024-09-18T08:02:58.788 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:02:58 smithi050 ceph-mon[32371]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' 2024-09-18T08:02:58.788 INFO:journalctl@ceph.iscsi.iscsi.a.smithi050.stdout:Sep 18 08:02:58 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-iscsi-iscsi-a[57782]: debug Started the configuration object watcher 2024-09-18T08:02:58.788 INFO:journalctl@ceph.iscsi.iscsi.a.smithi050.stdout:Sep 18 08:02:58 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-iscsi-iscsi-a[57782]: debug Checking for config object changes every 1s 2024-09-18T08:02:58.788 INFO:journalctl@ceph.iscsi.iscsi.a.smithi050.stdout:Sep 18 08:02:58 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-iscsi-iscsi-a[57782]: debug Processing osd blocklist entries for this node 2024-09-18T08:02:58.794 INFO:teuthology.orchestra.run.smithi050.stderr:Inferring config /var/lib/ceph/779349ca-7593-11ef-bceb-c7b262605968/mon.b/config 2024-09-18T08:02:58.937 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:02:58 smithi007 ceph-mon[39249]: pgmap v210: 132 pgs: 124 active+clean, 5 creating+peering, 3 unknown; 580 KiB data, 222 MiB used, 715 GiB / 715 GiB avail; 47 KiB/s rd, 3.8 KiB/s wr, 114 op/s 2024-09-18T08:02:58.937 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:02:58 smithi007 ceph-mon[39249]: from='client.24400 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "alertmanager", "placement": "1;smithi007=a", "target": ["mon-mgr", ""]}]: dispatch 2024-09-18T08:02:58.937 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:02:58 smithi007 ceph-mon[39249]: Saving service alertmanager spec with placement smithi007=a;count:1 2024-09-18T08:02:58.937 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:02:58 smithi007 ceph-mon[39249]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' 2024-09-18T08:02:58.937 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:02:58 smithi007 ceph-mon[39249]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' 2024-09-18T08:02:58.937 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:02:58 smithi007 ceph-mon[39249]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' 2024-09-18T08:02:58.937 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:02:58 smithi007 ceph-mon[39249]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' 2024-09-18T08:02:58.938 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:02:58 smithi007 ceph-mon[39249]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' 2024-09-18T08:02:58.938 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:02:58 smithi007 ceph-mon[29459]: pgmap v210: 132 pgs: 124 active+clean, 5 creating+peering, 3 unknown; 580 KiB data, 222 MiB used, 715 GiB / 715 GiB avail; 47 KiB/s rd, 3.8 KiB/s wr, 114 op/s 2024-09-18T08:02:58.938 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:02:58 smithi007 ceph-mon[29459]: from='client.24400 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "alertmanager", "placement": "1;smithi007=a", "target": ["mon-mgr", ""]}]: dispatch 2024-09-18T08:02:58.938 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:02:58 smithi007 ceph-mon[29459]: Saving service alertmanager spec with placement smithi007=a;count:1 2024-09-18T08:02:58.938 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:02:58 smithi007 ceph-mon[29459]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' 2024-09-18T08:02:58.938 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:02:58 smithi007 ceph-mon[29459]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' 2024-09-18T08:02:58.938 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:02:58 smithi007 ceph-mon[29459]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' 2024-09-18T08:02:58.938 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:02:58 smithi007 ceph-mon[29459]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' 2024-09-18T08:02:58.938 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:02:58 smithi007 ceph-mon[29459]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' 2024-09-18T08:02:59.156 INFO:journalctl@ceph.iscsi.iscsi.a.smithi050.stdout:Sep 18 08:02:58 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-iscsi-iscsi-a[57782]: debug Reading the configuration object to update local LIO configuration 2024-09-18T08:02:59.156 INFO:journalctl@ceph.iscsi.iscsi.a.smithi050.stdout:Sep 18 08:02:58 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-iscsi-iscsi-a[57782]: debug Configuration does not have an entry for this host(smithi050.front.sepia.ceph.com) - nothing to define to LIO 2024-09-18T08:02:59.156 INFO:journalctl@ceph.iscsi.iscsi.a.smithi050.stdout:Sep 18 08:02:58 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-iscsi-iscsi-a[57782]: * Serving Flask app 'rbd-target-api' (lazy loading) 2024-09-18T08:02:59.156 INFO:journalctl@ceph.iscsi.iscsi.a.smithi050.stdout:Sep 18 08:02:58 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-iscsi-iscsi-a[57782]: * Environment: production 2024-09-18T08:02:59.156 INFO:journalctl@ceph.iscsi.iscsi.a.smithi050.stdout:Sep 18 08:02:58 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-iscsi-iscsi-a[57782]: WARNING: This is a development server. Do not use it in a production deployment. 2024-09-18T08:02:59.157 INFO:journalctl@ceph.iscsi.iscsi.a.smithi050.stdout:Sep 18 08:02:58 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-iscsi-iscsi-a[57782]: Use a production WSGI server instead. 2024-09-18T08:02:59.157 INFO:journalctl@ceph.iscsi.iscsi.a.smithi050.stdout:Sep 18 08:02:58 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-iscsi-iscsi-a[57782]: * Debug mode: off 2024-09-18T08:02:59.157 INFO:journalctl@ceph.iscsi.iscsi.a.smithi050.stdout:Sep 18 08:02:58 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-iscsi-iscsi-a[57782]: debug * Running on all addresses. 2024-09-18T08:02:59.157 INFO:journalctl@ceph.iscsi.iscsi.a.smithi050.stdout:Sep 18 08:02:58 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-iscsi-iscsi-a[57782]: WARNING: This is a development server. Do not use it in a production deployment. 2024-09-18T08:02:59.157 INFO:journalctl@ceph.iscsi.iscsi.a.smithi050.stdout:Sep 18 08:02:58 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-iscsi-iscsi-a[57782]: * Running on all addresses. 2024-09-18T08:02:59.157 INFO:journalctl@ceph.iscsi.iscsi.a.smithi050.stdout:Sep 18 08:02:58 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-iscsi-iscsi-a[57782]: WARNING: This is a development server. Do not use it in a production deployment. 2024-09-18T08:02:59.157 INFO:journalctl@ceph.iscsi.iscsi.a.smithi050.stdout:Sep 18 08:02:58 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-iscsi-iscsi-a[57782]: debug * Running on http://[fe80::ec4:7aff:febd:1584]:5000/ (Press CTRL+C to quit) 2024-09-18T08:02:59.157 INFO:journalctl@ceph.iscsi.iscsi.a.smithi050.stdout:Sep 18 08:02:58 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-iscsi-iscsi-a[57782]: * Running on http://[fe80::ec4:7aff:febd:1584]:5000/ (Press CTRL+C to quit) 2024-09-18T08:02:59.709 INFO:teuthology.orchestra.run.smithi050.stdout:Scheduled grafana update... 2024-09-18T08:02:59.709 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:02:59 smithi050 ceph-mon[32371]: Checking pool "datapool" exists for service iscsi.datapool 2024-09-18T08:02:59.710 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:02:59 smithi050 ceph-mon[32371]: Deploying daemon prometheus.a on smithi050 2024-09-18T08:02:59.710 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:02:59 smithi050 ceph-mon[32371]: from='client.? 172.21.15.50:0/3986829138' entity='client.iscsi.iscsi.a' cmd={"prefix": "osd blocklist ls"} : dispatch 2024-09-18T08:02:59.710 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:02:59 smithi050 ceph-mon[32371]: osdmap e71: 8 total, 8 up, 8 in 2024-09-18T08:02:59.981 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:02:59 smithi007 ceph-mon[29459]: Checking pool "datapool" exists for service iscsi.datapool 2024-09-18T08:02:59.981 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:02:59 smithi007 ceph-mon[29459]: Deploying daemon prometheus.a on smithi050 2024-09-18T08:02:59.981 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:02:59 smithi007 ceph-mon[29459]: from='client.? 172.21.15.50:0/3986829138' entity='client.iscsi.iscsi.a' cmd={"prefix": "osd blocklist ls"} : dispatch 2024-09-18T08:02:59.981 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:02:59 smithi007 ceph-mon[29459]: osdmap e71: 8 total, 8 up, 8 in 2024-09-18T08:02:59.981 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:02:59 smithi007 ceph-mon[39249]: Checking pool "datapool" exists for service iscsi.datapool 2024-09-18T08:02:59.981 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:02:59 smithi007 ceph-mon[39249]: Deploying daemon prometheus.a on smithi050 2024-09-18T08:02:59.981 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:02:59 smithi007 ceph-mon[39249]: from='client.? 172.21.15.50:0/3986829138' entity='client.iscsi.iscsi.a' cmd={"prefix": "osd blocklist ls"} : dispatch 2024-09-18T08:02:59.981 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:02:59 smithi007 ceph-mon[39249]: osdmap e71: 8 total, 8 up, 8 in 2024-09-18T08:03:00.483 DEBUG:teuthology.orchestra.run.smithi050:grafana.a> sudo journalctl -f -n 0 -u ceph-779349ca-7593-11ef-bceb-c7b262605968@grafana.a.service 2024-09-18T08:03:00.485 INFO:tasks.cephadm:Setting up client nodes... 2024-09-18T08:03:00.486 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 779349ca-7593-11ef-bceb-c7b262605968 -- ceph auth get-or-create client.0 mon 'allow *' osd 'allow *' mds 'allow *' mgr 'allow *' 2024-09-18T08:03:00.756 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/779349ca-7593-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T08:03:00.878 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:03:00 smithi007 ceph-mon[29459]: pgmap v212: 132 pgs: 132 active+clean; 582 KiB data, 222 MiB used, 715 GiB / 715 GiB avail; 83 KiB/s rd, 6.2 KiB/s wr, 202 op/s 2024-09-18T08:03:00.878 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:03:00 smithi007 ceph-mon[29459]: from='client.24436 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "grafana", "placement": "1;smithi050=a", "target": ["mon-mgr", ""]}]: dispatch 2024-09-18T08:03:00.878 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:03:00 smithi007 ceph-mon[29459]: Saving service grafana spec with placement smithi050=a;count:1 2024-09-18T08:03:00.878 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:03:00 smithi007 ceph-mon[29459]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' 2024-09-18T08:03:00.878 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:03:00 smithi007 ceph-mon[29459]: mgrmap e16: y(active, since 5m), standbys: x 2024-09-18T08:03:00.879 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:03:00 smithi007 ceph-mon[39249]: pgmap v212: 132 pgs: 132 active+clean; 582 KiB data, 222 MiB used, 715 GiB / 715 GiB avail; 83 KiB/s rd, 6.2 KiB/s wr, 202 op/s 2024-09-18T08:03:00.879 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:03:00 smithi007 ceph-mon[39249]: from='client.24436 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "grafana", "placement": "1;smithi050=a", "target": ["mon-mgr", ""]}]: dispatch 2024-09-18T08:03:00.879 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:03:00 smithi007 ceph-mon[39249]: Saving service grafana spec with placement smithi050=a;count:1 2024-09-18T08:03:00.879 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:03:00 smithi007 ceph-mon[39249]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' 2024-09-18T08:03:00.879 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:03:00 smithi007 ceph-mon[39249]: mgrmap e16: y(active, since 5m), standbys: x 2024-09-18T08:03:01.156 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:03:00 smithi050 ceph-mon[32371]: pgmap v212: 132 pgs: 132 active+clean; 582 KiB data, 222 MiB used, 715 GiB / 715 GiB avail; 83 KiB/s rd, 6.2 KiB/s wr, 202 op/s 2024-09-18T08:03:01.156 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:03:00 smithi050 ceph-mon[32371]: from='client.24436 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "grafana", "placement": "1;smithi050=a", "target": ["mon-mgr", ""]}]: dispatch 2024-09-18T08:03:01.156 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:03:00 smithi050 ceph-mon[32371]: Saving service grafana spec with placement smithi050=a;count:1 2024-09-18T08:03:01.157 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:03:00 smithi050 ceph-mon[32371]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' 2024-09-18T08:03:01.157 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:03:00 smithi050 ceph-mon[32371]: mgrmap e16: y(active, since 5m), standbys: x 2024-09-18T08:03:01.499 INFO:teuthology.orchestra.run.smithi007.stdout:[client.0] 2024-09-18T08:03:01.500 INFO:teuthology.orchestra.run.smithi007.stdout: key = AQA1iepmtLOIHRAAXUw3ujx2Zfy9g1abAlLGPA== 2024-09-18T08:03:01.931 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:03:01 smithi007 ceph-mon[29459]: from='client.? 172.21.15.7:0/1569495603' entity='client.admin' cmd={"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]} : dispatch 2024-09-18T08:03:01.931 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:03:01 smithi007 ceph-mon[29459]: from='client.? 172.21.15.7:0/1569495603' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-09-18T08:03:01.931 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:03:01 smithi007 ceph-mon[39249]: from='client.? 172.21.15.7:0/1569495603' entity='client.admin' cmd={"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]} : dispatch 2024-09-18T08:03:01.931 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:03:01 smithi007 ceph-mon[39249]: from='client.? 172.21.15.7:0/1569495603' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-09-18T08:03:02.156 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:03:01 smithi050 ceph-mon[32371]: from='client.? 172.21.15.7:0/1569495603' entity='client.admin' cmd={"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]} : dispatch 2024-09-18T08:03:02.156 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:03:01 smithi050 ceph-mon[32371]: from='client.? 172.21.15.7:0/1569495603' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-09-18T08:03:02.388 DEBUG:teuthology.orchestra.run.smithi007:> set -ex 2024-09-18T08:03:02.388 DEBUG:teuthology.orchestra.run.smithi007:> sudo dd of=/etc/ceph/ceph.client.0.keyring 2024-09-18T08:03:02.388 DEBUG:teuthology.orchestra.run.smithi007:> sudo chmod 0644 /etc/ceph/ceph.client.0.keyring 2024-09-18T08:03:02.430 DEBUG:teuthology.orchestra.run.smithi050:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 779349ca-7593-11ef-bceb-c7b262605968 -- ceph auth get-or-create client.1 mon 'allow *' osd 'allow *' mds 'allow *' mgr 'allow *' 2024-09-18T08:03:02.689 INFO:teuthology.orchestra.run.smithi050.stderr:Inferring config /var/lib/ceph/779349ca-7593-11ef-bceb-c7b262605968/mon.b/config 2024-09-18T08:03:02.980 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:03:02 smithi007 ceph-mon[29459]: pgmap v213: 132 pgs: 132 active+clean; 582 KiB data, 222 MiB used, 715 GiB / 715 GiB avail; 62 KiB/s rd, 4.6 KiB/s wr, 151 op/s 2024-09-18T08:03:02.981 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:03:02 smithi007 ceph-mon[29459]: purged_snaps scrub starts 2024-09-18T08:03:02.981 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:03:02 smithi007 ceph-mon[29459]: purged_snaps scrub ok 2024-09-18T08:03:02.981 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:03:02 smithi007 ceph-mon[29459]: osdmap e72: 8 total, 8 up, 8 in 2024-09-18T08:03:02.981 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:03:02 smithi007 ceph-mon[39249]: pgmap v213: 132 pgs: 132 active+clean; 582 KiB data, 222 MiB used, 715 GiB / 715 GiB avail; 62 KiB/s rd, 4.6 KiB/s wr, 151 op/s 2024-09-18T08:03:02.981 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:03:02 smithi007 ceph-mon[39249]: purged_snaps scrub starts 2024-09-18T08:03:02.981 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:03:02 smithi007 ceph-mon[39249]: purged_snaps scrub ok 2024-09-18T08:03:02.981 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:03:02 smithi007 ceph-mon[39249]: osdmap e72: 8 total, 8 up, 8 in 2024-09-18T08:03:03.060 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:03:02 smithi050 ceph-mon[32371]: pgmap v213: 132 pgs: 132 active+clean; 582 KiB data, 222 MiB used, 715 GiB / 715 GiB avail; 62 KiB/s rd, 4.6 KiB/s wr, 151 op/s 2024-09-18T08:03:03.060 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:03:02 smithi050 ceph-mon[32371]: purged_snaps scrub starts 2024-09-18T08:03:03.060 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:03:02 smithi050 ceph-mon[32371]: purged_snaps scrub ok 2024-09-18T08:03:03.060 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:03:02 smithi050 ceph-mon[32371]: osdmap e72: 8 total, 8 up, 8 in 2024-09-18T08:03:03.737 INFO:teuthology.orchestra.run.smithi050.stdout:[client.1] 2024-09-18T08:03:03.737 INFO:teuthology.orchestra.run.smithi050.stdout: key = AQA3iepmLU6YKxAAX5zngla5vSMrOY+j+m2epQ== 2024-09-18T08:03:04.156 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:03:03 smithi050 ceph-mon[32371]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' 2024-09-18T08:03:04.156 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:03:03 smithi050 ceph-mon[32371]: from='client.? 172.21.15.50:0/2312847962' entity='client.admin' cmd={"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]} : dispatch 2024-09-18T08:03:04.156 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:03:03 smithi050 ceph-mon[32371]: from='client.? ' entity='client.admin' cmd={"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]} : dispatch 2024-09-18T08:03:04.156 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:03:03 smithi050 ceph-mon[32371]: from='client.? ' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-09-18T08:03:04.231 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:03:03 smithi007 ceph-mon[29459]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' 2024-09-18T08:03:04.231 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:03:03 smithi007 ceph-mon[29459]: from='client.? 172.21.15.50:0/2312847962' entity='client.admin' cmd={"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]} : dispatch 2024-09-18T08:03:04.231 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:03:03 smithi007 ceph-mon[29459]: from='client.? ' entity='client.admin' cmd={"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]} : dispatch 2024-09-18T08:03:04.231 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:03:03 smithi007 ceph-mon[29459]: from='client.? ' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-09-18T08:03:04.231 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:03:03 smithi007 ceph-mon[39249]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' 2024-09-18T08:03:04.231 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:03:03 smithi007 ceph-mon[39249]: from='client.? 172.21.15.50:0/2312847962' entity='client.admin' cmd={"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]} : dispatch 2024-09-18T08:03:04.231 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:03:03 smithi007 ceph-mon[39249]: from='client.? ' entity='client.admin' cmd={"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]} : dispatch 2024-09-18T08:03:04.231 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:03:03 smithi007 ceph-mon[39249]: from='client.? ' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-09-18T08:03:05.156 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:03:04 smithi050 ceph-mon[32371]: pgmap v215: 132 pgs: 132 active+clean; 583 KiB data, 222 MiB used, 715 GiB / 715 GiB avail; 49 KiB/s rd, 3.6 KiB/s wr, 119 op/s 2024-09-18T08:03:05.230 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:03:04 smithi007 ceph-mon[29459]: pgmap v215: 132 pgs: 132 active+clean; 583 KiB data, 222 MiB used, 715 GiB / 715 GiB avail; 49 KiB/s rd, 3.6 KiB/s wr, 119 op/s 2024-09-18T08:03:05.231 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:03:04 smithi007 ceph-mon[39249]: pgmap v215: 132 pgs: 132 active+clean; 583 KiB data, 222 MiB used, 715 GiB / 715 GiB avail; 49 KiB/s rd, 3.6 KiB/s wr, 119 op/s 2024-09-18T08:03:05.613 DEBUG:teuthology.orchestra.run.smithi050:> set -ex 2024-09-18T08:03:05.614 DEBUG:teuthology.orchestra.run.smithi050:> sudo dd of=/etc/ceph/ceph.client.1.keyring 2024-09-18T08:03:05.614 DEBUG:teuthology.orchestra.run.smithi050:> sudo chmod 0644 /etc/ceph/ceph.client.1.keyring 2024-09-18T08:03:05.656 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid 779349ca-7593-11ef-bceb-c7b262605968 -- ceph config log 1 --format=json 2024-09-18T08:03:05.928 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/779349ca-7593-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T08:03:06.582 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-09-18T08:03:06.980 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:03:06 smithi007 ceph-mon[29459]: pgmap v216: 132 pgs: 132 active+clean; 583 KiB data, 222 MiB used, 715 GiB / 715 GiB avail; 32 KiB/s rd, 2.4 KiB/s wr, 77 op/s 2024-09-18T08:03:06.981 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:03:06 smithi007 ceph-mon[29459]: from='client.? 172.21.15.7:0/2389225676' entity='client.admin' cmd={"prefix": "config log", "num": 1, "format": "json"} : dispatch 2024-09-18T08:03:06.981 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:03:06 smithi007 ceph-mon[39249]: pgmap v216: 132 pgs: 132 active+clean; 583 KiB data, 222 MiB used, 715 GiB / 715 GiB avail; 32 KiB/s rd, 2.4 KiB/s wr, 77 op/s 2024-09-18T08:03:06.981 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:03:06 smithi007 ceph-mon[39249]: from='client.? 172.21.15.7:0/2389225676' entity='client.admin' cmd={"prefix": "config log", "num": 1, "format": "json"} : dispatch 2024-09-18T08:03:07.089 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:03:06 smithi050 ceph-mon[32371]: pgmap v216: 132 pgs: 132 active+clean; 583 KiB data, 222 MiB used, 715 GiB / 715 GiB avail; 32 KiB/s rd, 2.4 KiB/s wr, 77 op/s 2024-09-18T08:03:07.090 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:03:06 smithi050 ceph-mon[32371]: from='client.? 172.21.15.7:0/2389225676' entity='client.admin' cmd={"prefix": "config log", "num": 1, "format": "json"} : dispatch 2024-09-18T08:03:07.309 INFO:teuthology.orchestra.run.smithi007.stdout:[{"version":21,"timestamp":"2024-09-18T08:02:54.277517+0000","name":"","changes":[{"name":"mgr/mgr/dashboard/RGW_API_SECRET_KEY","new_value":"7EBsuTb0gNUXRJuD9nAtrv3KB2JQBMPDNmXD81NG"}]}] 2024-09-18T08:03:07.309 INFO:tasks.ceph_manager:config epoch is 21 2024-09-18T08:03:07.310 INFO:tasks.ceph:Waiting until ceph daemons up and pgs clean... 2024-09-18T08:03:07.310 INFO:tasks.cephadm.ceph_manager.ceph:waiting for mgr available 2024-09-18T08:03:07.310 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid 779349ca-7593-11ef-bceb-c7b262605968 -- ceph mgr dump --format=json 2024-09-18T08:03:07.568 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/779349ca-7593-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T08:03:08.244 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-09-18T08:03:08.808 INFO:journalctl@ceph.iscsi.iscsi.a.smithi050.stdout:Sep 18 08:03:08 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-iscsi-iscsi-a[57782]: debug there is no tcmu-runner data available 2024-09-18T08:03:09.047 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":16,"flags":0,"active_gid":14152,"active_name":"y","active_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6800","nonce":1491061076}]},"active_addr":"172.21.15.7:6800/1491061076","active_change":"2024-09-18T07:57:57.346254+0000","active_mgr_features":4540719139924082687,"available":true,"standbys":[{"gid":24106,"name":"x","mgr_features":4540719139924082687,"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to, use commas to separate multiple","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","read","upmap","upmap-read"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.27.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/grafana:10.4.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:3.0.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_nginx":{"name":"container_image_nginx","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nginx:sclorg-nginx-126","min":"","max":"","enum_allowed":[],"desc":"Nginx container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.7.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:1.2.17","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_oauth2_proxy":{"name":"container_image_oauth2_proxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/oauth2-proxy/oauth2-proxy:v7.6.0","min":"","max":"","enum_allowed":[],"desc":"oauth2-proxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.51.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:3.0.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_samba":{"name":"container_image_samba","type":"str","level":"advanced","flags":0,"default_value":"quay.io/samba.org/samba-server:devbuilds-centos-amd64","min":"","max":"","enum_allowed":[],"desc":"Samba/SMB container image","long_desc":"","tags":[],"see_also":[]},"container_image_samba_metrics":{"name":"container_image_samba_metrics","type":"str","level":"advanced","flags":0,"default_value":"quay.io/samba.org/samba-metrics:latest","min":"","max":"","enum_allowed":[],"desc":"Samba/SMB metrics exporter container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"grafana_dashboards_path":{"name":"grafana_dashboards_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/grafana/dashboards/ceph-dashboard/","min":"","max":"","enum_allowed":[],"desc":"location of dashboards to include in grafana deployments","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"ssh_keepalive_count_max":{"name":"ssh_keepalive_count_max","type":"int","level":"advanced","flags":0,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"How many times ssh connections can fail liveness checks before the host is marked offline","long_desc":"","tags":[],"see_also":[]},"ssh_keepalive_interval":{"name":"ssh_keepalive_interval","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"How often ssh connections are checked for liveness","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"MANAGED_BY_CLUSTERS":{"name":"MANAGED_BY_CLUSTERS","type":"str","level":"advanced","flags":0,"default_value":"[]","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"MULTICLUSTER_CONFIG":{"name":"MULTICLUSTER_CONFIG","type":"str","level":"advanced","flags":0,"default_value":"{}","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sso_oauth2":{"name":"sso_oauth2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"k8sevents","can_run":true,"error_string":"","module_options":{"ceph_event_retention_days":{"name":"ceph_event_retention_days","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"Days to hold ceph event information within local cache","long_desc":"","tags":[],"see_also":[]},"config_check_secs":{"name":"config_check_secs","type":"int","level":"advanced","flags":0,"default_value":"10","min":"10","max":"","enum_allowed":[],"desc":"interval (secs) to check for cluster configuration changes","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_requests":{"name":"max_requests","type":"int","level":"advanced","flags":0,"default_value":"500","min":"","max":"","enum_allowed":[],"desc":"Maximum number of requests to keep in memory. When new request comes in, the oldest request will be removed if the number of requests exceeds the max request number.if un-finished request is removed, error message will be logged in the ceph-mgr log.","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"secondary_zone_period_retry_limit":{"name":"secondary_zone_period_retry_limit","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"RGW module period update retry limit for secondary site","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"smb","can_run":true,"error_string":"","module_options":{"internal_store_backend":{"name":"internal_store_backend","type":"str","level":"dev","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"set internal store backend. for develoment and testing only","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"update_orchestration":{"name":"update_orchestration","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically update orchestration when smb resources are changed","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_no_wait":{"name":"snapshot_clone_no_wait","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Reject subvolume clone request when cloner threads are busy","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}]}],"modules":["cephadm","dashboard","iostat","nfs","restful"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to, use commas to separate multiple","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","read","upmap","upmap-read"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.27.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/grafana:10.4.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:3.0.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_nginx":{"name":"container_image_nginx","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nginx:sclorg-nginx-126","min":"","max":"","enum_allowed":[],"desc":"Nginx container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.7.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:1.2.17","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_oauth2_proxy":{"name":"container_image_oauth2_proxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/oauth2-proxy/oauth2-proxy:v7.6.0","min":"","max":"","enum_allowed":[],"desc":"oauth2-proxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.51.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:3.0.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_samba":{"name":"container_image_samba","type":"str","level":"advanced","flags":0,"default_value":"quay.io/samba.org/samba-server:devbuilds-centos-amd64","min":"","max":"","enum_allowed":[],"desc":"Samba/SMB container image","long_desc":"","tags":[],"see_also":[]},"container_image_samba_metrics":{"name":"container_image_samba_metrics","type":"str","level":"advanced","flags":0,"default_value":"quay.io/samba.org/samba-metrics:latest","min":"","max":"","enum_allowed":[],"desc":"Samba/SMB metrics exporter container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"grafana_dashboards_path":{"name":"grafana_dashboards_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/grafana/dashboards/ceph-dashboard/","min":"","max":"","enum_allowed":[],"desc":"location of dashboards to include in grafana deployments","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"ssh_keepalive_count_max":{"name":"ssh_keepalive_count_max","type":"int","level":"advanced","flags":0,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"How many times ssh connections can fail liveness checks before the host is marked offline","long_desc":"","tags":[],"see_also":[]},"ssh_keepalive_interval":{"name":"ssh_keepalive_interval","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"How often ssh connections are checked for liveness","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"MANAGED_BY_CLUSTERS":{"name":"MANAGED_BY_CLUSTERS","type":"str","level":"advanced","flags":0,"default_value":"[]","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"MULTICLUSTER_CONFIG":{"name":"MULTICLUSTER_CONFIG","type":"str","level":"advanced","flags":0,"default_value":"{}","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sso_oauth2":{"name":"sso_oauth2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"k8sevents","can_run":true,"error_string":"","module_options":{"ceph_event_retention_days":{"name":"ceph_event_retention_days","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"Days to hold ceph event information within local cache","long_desc":"","tags":[],"see_also":[]},"config_check_secs":{"name":"config_check_secs","type":"int","level":"advanced","flags":0,"default_value":"10","min":"10","max":"","enum_allowed":[],"desc":"interval (secs) to check for cluster configuration changes","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_requests":{"name":"max_requests","type":"int","level":"advanced","flags":0,"default_value":"500","min":"","max":"","enum_allowed":[],"desc":"Maximum number of requests to keep in memory. When new request comes in, the oldest request will be removed if the number of requests exceeds the max request number.if un-finished request is removed, error message will be logged in the ceph-mgr log.","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"secondary_zone_period_retry_limit":{"name":"secondary_zone_period_retry_limit","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"RGW module period update retry limit for secondary site","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"smb","can_run":true,"error_string":"","module_options":{"internal_store_backend":{"name":"internal_store_backend","type":"str","level":"dev","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"set internal store backend. for develoment and testing only","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"update_orchestration":{"name":"update_orchestration","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically update orchestration when smb resources are changed","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_no_wait":{"name":"snapshot_clone_no_wait","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Reject subvolume clone request when cloner threads are busy","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{"dashboard":"https://172.21.15.7:8443/"},"always_on_modules":{"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"quincy":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"reef":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"squid":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"]},"last_failure_osd_epoch":3,"active_clients":[{"name":"cephadm","addrvec":[{"type":"v2","addr":"172.21.15.7:0","nonce":398367658}]},{"name":"devicehealth","addrvec":[{"type":"v2","addr":"172.21.15.7:0","nonce":1722968065}]},{"name":"libcephsqlite","addrvec":[{"type":"v2","addr":"172.21.15.7:0","nonce":1605217610}]},{"name":"rbd_support","addrvec":[{"type":"v2","addr":"172.21.15.7:0","nonce":232452890}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"172.21.15.7:0","nonce":3881670470}]}]} 2024-09-18T08:03:09.053 INFO:tasks.cephadm.ceph_manager.ceph:mgr available! 2024-09-18T08:03:09.053 INFO:tasks.cephadm.ceph_manager.ceph:waiting for all up 2024-09-18T08:03:09.053 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid 779349ca-7593-11ef-bceb-c7b262605968 -- ceph osd dump --format=json 2024-09-18T08:03:09.079 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:03:08 smithi007 ceph-mon[29459]: pgmap v217: 132 pgs: 132 active+clean; 583 KiB data, 222 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 122 B/s wr, 2 op/s 2024-09-18T08:03:09.079 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:03:08 smithi007 ceph-mon[29459]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' 2024-09-18T08:03:09.079 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:03:08 smithi007 ceph-mon[29459]: from='client.? 172.21.15.7:0/2596233040' entity='client.admin' cmd={"prefix": "mgr dump", "format": "json"} : dispatch 2024-09-18T08:03:09.080 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:03:08 smithi007 ceph-mon[39249]: pgmap v217: 132 pgs: 132 active+clean; 583 KiB data, 222 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 122 B/s wr, 2 op/s 2024-09-18T08:03:09.080 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:03:08 smithi007 ceph-mon[39249]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' 2024-09-18T08:03:09.080 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:03:08 smithi007 ceph-mon[39249]: from='client.? 172.21.15.7:0/2596233040' entity='client.admin' cmd={"prefix": "mgr dump", "format": "json"} : dispatch 2024-09-18T08:03:09.156 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:03:08 smithi050 ceph-mon[32371]: pgmap v217: 132 pgs: 132 active+clean; 583 KiB data, 222 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 122 B/s wr, 2 op/s 2024-09-18T08:03:09.156 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:03:08 smithi050 ceph-mon[32371]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' 2024-09-18T08:03:09.156 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:03:08 smithi050 ceph-mon[32371]: from='client.? 172.21.15.7:0/2596233040' entity='client.admin' cmd={"prefix": "mgr dump", "format": "json"} : dispatch 2024-09-18T08:03:09.313 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/779349ca-7593-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T08:03:09.906 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:03:09 smithi050 ceph-mon[32371]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-18T08:03:09.906 INFO:journalctl@ceph.prometheus.a.smithi050.stdout:Sep 18 08:03:09 smithi050 systemd[1]: Starting Ceph prometheus.a for 779349ca-7593-11ef-bceb-c7b262605968... 2024-09-18T08:03:09.931 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-09-18T08:03:09.931 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":72,"fsid":"779349ca-7593-11ef-bceb-c7b262605968","created":"2024-09-18T07:57:09.295710+0000","modified":"2024-09-18T08:03:02.497238+0000","last_up_change":"2024-09-18T08:02:36.989876+0000","last_in_change":"2024-09-18T08:02:14.309111+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":18,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":6,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-18T08:00:11.802627+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":7.8899998664855957,"score_stable":7.8899998664855957,"optimal_score":0.37999999523162842,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"datapool","create_time":"2024-09-18T08:02:45.374099+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":3,"pg_placement_num":3,"pg_placement_num_target":3,"pg_num_target":3,"pg_num_pending":3,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"64","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":2,"snap_epoch":64,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_type":"Fair distribution","score_acting":2.6500000953674316,"score_stable":2.6500000953674316,"optimal_score":0.87999999523162842,"raw_score_acting":2.3299999237060547,"raw_score_stable":2.3299999237060547,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":".rgw.root","create_time":"2024-09-18T08:02:46.319947+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"63","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rgw":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"default.rgw.log","create_time":"2024-09-18T08:02:47.948552+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"65","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rgw":{}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"default.rgw.control","create_time":"2024-09-18T08:02:49.904302+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"67","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rgw":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.25,"score_stable":1.25,"optimal_score":1,"raw_score_acting":1.25,"raw_score_stable":1.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":6,"pool_name":"default.rgw.meta","create_time":"2024-09-18T08:02:51.971465+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"70","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4},"application_metadata":{"rgw":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"ae8f60df-72ac-46fb-91a1-f132b851c876","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6801","nonce":2896425794}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6802","nonce":2896425794}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6804","nonce":2896425794}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6803","nonce":2896425794}]},"public_addr":"172.21.15.7:6801/2896425794","cluster_addr":"172.21.15.7:6802/2896425794","heartbeat_back_addr":"172.21.15.7:6804/2896425794","heartbeat_front_addr":"172.21.15.7:6803/2896425794","state":["exists","up"]},{"osd":1,"uuid":"be28b70c-1f73-494f-ad4b-cb7ec5311999","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6805","nonce":1971397549}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6806","nonce":1971397549}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6808","nonce":1971397549}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6807","nonce":1971397549}]},"public_addr":"172.21.15.7:6805/1971397549","cluster_addr":"172.21.15.7:6806/1971397549","heartbeat_back_addr":"172.21.15.7:6808/1971397549","heartbeat_front_addr":"172.21.15.7:6807/1971397549","state":["exists","up"]},{"osd":2,"uuid":"cfa4f083-c7fd-4743-be65-84a3040fe6dc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6809","nonce":556142559}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6810","nonce":556142559}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6812","nonce":556142559}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6811","nonce":556142559}]},"public_addr":"172.21.15.7:6809/556142559","cluster_addr":"172.21.15.7:6810/556142559","heartbeat_back_addr":"172.21.15.7:6812/556142559","heartbeat_front_addr":"172.21.15.7:6811/556142559","state":["exists","up"]},{"osd":3,"uuid":"11ecd7e1-a751-4b75-95f5-00b32d2a5bc4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6813","nonce":163587772}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6814","nonce":163587772}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6816","nonce":163587772}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6815","nonce":163587772}]},"public_addr":"172.21.15.7:6813/163587772","cluster_addr":"172.21.15.7:6814/163587772","heartbeat_back_addr":"172.21.15.7:6816/163587772","heartbeat_front_addr":"172.21.15.7:6815/163587772","state":["exists","up"]},{"osd":4,"uuid":"98992b34-cfa1-45bf-97a1-7d557739252f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":32,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":3662119805}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6801","nonce":3662119805}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6803","nonce":3662119805}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":3662119805}]},"public_addr":"172.21.15.50:6800/3662119805","cluster_addr":"172.21.15.50:6801/3662119805","heartbeat_back_addr":"172.21.15.50:6803/3662119805","heartbeat_front_addr":"172.21.15.50:6802/3662119805","state":["exists","up"]},{"osd":5,"uuid":"7f290005-ebca-467b-87e4-02ead1063610","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":39,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":1199797954}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6805","nonce":1199797954}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6807","nonce":1199797954}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":1199797954}]},"public_addr":"172.21.15.50:6804/1199797954","cluster_addr":"172.21.15.50:6805/1199797954","heartbeat_back_addr":"172.21.15.50:6807/1199797954","heartbeat_front_addr":"172.21.15.50:6806/1199797954","state":["exists","up"]},{"osd":6,"uuid":"e07d846c-ae42-4fea-aa5c-3a25835d18a6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":47,"up_thru":65,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":255960422}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6809","nonce":255960422}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6811","nonce":255960422}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":255960422}]},"public_addr":"172.21.15.50:6808/255960422","cluster_addr":"172.21.15.50:6809/255960422","heartbeat_back_addr":"172.21.15.50:6811/255960422","heartbeat_front_addr":"172.21.15.50:6810/255960422","state":["exists","up"]},{"osd":7,"uuid":"c9dac3b7-d651-489d-b96e-5b10fa32d9af","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":56,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":874923904}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6813","nonce":874923904}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6815","nonce":874923904}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":874923904}]},"public_addr":"172.21.15.50:6812/874923904","cluster_addr":"172.21.15.50:6813/874923904","heartbeat_back_addr":"172.21.15.50:6815/874923904","heartbeat_front_addr":"172.21.15.50:6814/874923904","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-18T08:02:26.967281+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-18T08:03:01.625260+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-18T08:02:29.901882+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-18T08:02:43.187317+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-18T08:02:11.109010+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-18T08:02:47.507971+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-18T08:02:05.533581+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-18T08:02:35.425165+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.7:0/533727535":"2024-09-19T07:57:57.346149+0000","172.21.15.7:0/1277549846":"2024-09-19T07:57:57.346149+0000","172.21.15.7:6800/1013061071":"2024-09-19T07:57:57.346149+0000","172.21.15.7:0/4272027292":"2024-09-19T07:57:31.015269+0000","172.21.15.7:0/1668598084":"2024-09-19T07:57:57.346149+0000","172.21.15.7:0/3924911335":"2024-09-19T07:57:31.015269+0000","172.21.15.7:0/2514354773":"2024-09-19T07:57:31.015269+0000","172.21.15.7:6800/2918496291":"2024-09-19T07:57:31.015269+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-18T08:03:09.980 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:03:09 smithi007 ceph-mon[29459]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-18T08:03:09.981 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:03:09 smithi007 ceph-mon[39249]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-18T08:03:10.292 INFO:journalctl@ceph.prometheus.a.smithi050.stdout:Sep 18 08:03:10 smithi050 podman[58680]: 2024-09-18 08:03:09.930979591 +0000 UTC m=+0.017689939 image pull 1d3b7f56885b6dd623f1785be963aa9c195f86bc256ea454e8d02a7980b79c53 quay.io/prometheus/prometheus:v2.51.0 2024-09-18T08:03:10.292 INFO:journalctl@ceph.prometheus.a.smithi050.stdout:Sep 18 08:03:10 smithi050 podman[58680]: 2024-09-18 08:03:10.033699999 +0000 UTC m=+0.120410380 container create 4f01531f2bb3ce7e6b7b5bb3adeb4fd754108415006149e95fe4eb669369892d (image=quay.io/prometheus/prometheus:v2.51.0, name=ceph-779349ca-7593-11ef-bceb-c7b262605968-prometheus-a, maintainer=The Prometheus Authors ) 2024-09-18T08:03:10.292 INFO:journalctl@ceph.prometheus.a.smithi050.stdout:Sep 18 08:03:10 smithi050 podman[58680]: 2024-09-18 08:03:10.244344327 +0000 UTC m=+0.331054670 container init 4f01531f2bb3ce7e6b7b5bb3adeb4fd754108415006149e95fe4eb669369892d (image=quay.io/prometheus/prometheus:v2.51.0, name=ceph-779349ca-7593-11ef-bceb-c7b262605968-prometheus-a, maintainer=The Prometheus Authors ) 2024-09-18T08:03:10.293 INFO:journalctl@ceph.prometheus.a.smithi050.stdout:Sep 18 08:03:10 smithi050 podman[58680]: 2024-09-18 08:03:10.247181095 +0000 UTC m=+0.333891436 container start 4f01531f2bb3ce7e6b7b5bb3adeb4fd754108415006149e95fe4eb669369892d (image=quay.io/prometheus/prometheus:v2.51.0, name=ceph-779349ca-7593-11ef-bceb-c7b262605968-prometheus-a, maintainer=The Prometheus Authors ) 2024-09-18T08:03:10.293 INFO:journalctl@ceph.prometheus.a.smithi050.stdout:Sep 18 08:03:10 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-prometheus-a[58694]: ts=2024-09-18T08:03:10.266Z caller=main.go:617 level=info msg="Starting Prometheus Server" mode=server version="(version=2.51.0, branch=HEAD, revision=c05c15512acb675e3f6cd662a6727854e93fc024)" 2024-09-18T08:03:10.293 INFO:journalctl@ceph.prometheus.a.smithi050.stdout:Sep 18 08:03:10 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-prometheus-a[58694]: ts=2024-09-18T08:03:10.266Z caller=main.go:622 level=info build_context="(go=go1.22.1, platform=linux/amd64, user=root@b5723e458358, date=20240319-10:54:45, tags=netgo,builtinassets,stringlabels)" 2024-09-18T08:03:10.293 INFO:journalctl@ceph.prometheus.a.smithi050.stdout:Sep 18 08:03:10 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-prometheus-a[58694]: ts=2024-09-18T08:03:10.266Z caller=main.go:623 level=info host_details="(Linux 5.14.0-508.el9.x86_64 #1 SMP PREEMPT_DYNAMIC Thu Sep 12 15:49:37 UTC 2024 x86_64 smithi050 (none))" 2024-09-18T08:03:10.293 INFO:journalctl@ceph.prometheus.a.smithi050.stdout:Sep 18 08:03:10 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-prometheus-a[58694]: ts=2024-09-18T08:03:10.266Z caller=main.go:624 level=info fd_limits="(soft=1048576, hard=1048576)" 2024-09-18T08:03:10.293 INFO:journalctl@ceph.prometheus.a.smithi050.stdout:Sep 18 08:03:10 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-prometheus-a[58694]: ts=2024-09-18T08:03:10.266Z caller=main.go:625 level=info vm_limits="(soft=unlimited, hard=unlimited)" 2024-09-18T08:03:10.293 INFO:journalctl@ceph.prometheus.a.smithi050.stdout:Sep 18 08:03:10 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-prometheus-a[58694]: ts=2024-09-18T08:03:10.268Z caller=web.go:568 level=info component=web msg="Start listening for connections" address=:9095 2024-09-18T08:03:10.293 INFO:journalctl@ceph.prometheus.a.smithi050.stdout:Sep 18 08:03:10 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-prometheus-a[58694]: ts=2024-09-18T08:03:10.268Z caller=main.go:1129 level=info msg="Starting TSDB ..." 2024-09-18T08:03:10.293 INFO:journalctl@ceph.prometheus.a.smithi050.stdout:Sep 18 08:03:10 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-prometheus-a[58694]: ts=2024-09-18T08:03:10.270Z caller=tls_config.go:313 level=info component=web msg="Listening on" address=[::]:9095 2024-09-18T08:03:10.293 INFO:journalctl@ceph.prometheus.a.smithi050.stdout:Sep 18 08:03:10 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-prometheus-a[58694]: ts=2024-09-18T08:03:10.271Z caller=tls_config.go:316 level=info component=web msg="TLS is disabled." http2=false address=[::]:9095 2024-09-18T08:03:10.293 INFO:journalctl@ceph.prometheus.a.smithi050.stdout:Sep 18 08:03:10 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-prometheus-a[58694]: ts=2024-09-18T08:03:10.272Z caller=head.go:616 level=info component=tsdb msg="Replaying on-disk memory mappable chunks if any" 2024-09-18T08:03:10.294 INFO:journalctl@ceph.prometheus.a.smithi050.stdout:Sep 18 08:03:10 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-prometheus-a[58694]: ts=2024-09-18T08:03:10.272Z caller=head.go:698 level=info component=tsdb msg="On-disk memory mappable chunks replay completed" duration=1.621µs 2024-09-18T08:03:10.294 INFO:journalctl@ceph.prometheus.a.smithi050.stdout:Sep 18 08:03:10 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-prometheus-a[58694]: ts=2024-09-18T08:03:10.272Z caller=head.go:706 level=info component=tsdb msg="Replaying WAL, this may take a while" 2024-09-18T08:03:10.294 INFO:journalctl@ceph.prometheus.a.smithi050.stdout:Sep 18 08:03:10 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-prometheus-a[58694]: ts=2024-09-18T08:03:10.272Z caller=head.go:778 level=info component=tsdb msg="WAL segment loaded" segment=0 maxSegment=0 2024-09-18T08:03:10.294 INFO:journalctl@ceph.prometheus.a.smithi050.stdout:Sep 18 08:03:10 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-prometheus-a[58694]: ts=2024-09-18T08:03:10.272Z caller=head.go:815 level=info component=tsdb msg="WAL replay completed" checkpoint_replay_duration=24.282µs wal_replay_duration=166.576µs wbl_replay_duration=91ns total_replay_duration=209.081µs 2024-09-18T08:03:10.294 INFO:journalctl@ceph.prometheus.a.smithi050.stdout:Sep 18 08:03:10 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-prometheus-a[58694]: ts=2024-09-18T08:03:10.273Z caller=main.go:1150 level=info fs_type=XFS_SUPER_MAGIC 2024-09-18T08:03:10.294 INFO:journalctl@ceph.prometheus.a.smithi050.stdout:Sep 18 08:03:10 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-prometheus-a[58694]: ts=2024-09-18T08:03:10.273Z caller=main.go:1153 level=info msg="TSDB started" 2024-09-18T08:03:10.294 INFO:journalctl@ceph.prometheus.a.smithi050.stdout:Sep 18 08:03:10 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-prometheus-a[58694]: ts=2024-09-18T08:03:10.273Z caller=main.go:1335 level=info msg="Loading configuration file" filename=/etc/prometheus/prometheus.yml 2024-09-18T08:03:10.294 INFO:journalctl@ceph.prometheus.a.smithi050.stdout:Sep 18 08:03:10 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-prometheus-a[58694]: ts=2024-09-18T08:03:10.289Z caller=main.go:1372 level=info msg="Completed loading of configuration file" filename=/etc/prometheus/prometheus.yml totalDuration=16.012844ms db_storage=820ns remote_storage=954ns web_handler=229ns query_engine=519ns scrape=412.06µs scrape_sd=89.306µs notify=928ns notify_sd=1.214µs rules=15.198931ms tracing=6.247µs 2024-09-18T08:03:10.294 INFO:journalctl@ceph.prometheus.a.smithi050.stdout:Sep 18 08:03:10 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-prometheus-a[58694]: ts=2024-09-18T08:03:10.290Z caller=main.go:1114 level=info msg="Server is ready to receive web requests." 2024-09-18T08:03:10.294 INFO:journalctl@ceph.prometheus.a.smithi050.stdout:Sep 18 08:03:10 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-prometheus-a[58694]: ts=2024-09-18T08:03:10.290Z caller=manager.go:163 level=info component="rule manager" msg="Starting rule manager..." 2024-09-18T08:03:10.656 INFO:journalctl@ceph.prometheus.a.smithi050.stdout:Sep 18 08:03:10 smithi050 bash[58680]: 4f01531f2bb3ce7e6b7b5bb3adeb4fd754108415006149e95fe4eb669369892d 2024-09-18T08:03:10.656 INFO:journalctl@ceph.prometheus.a.smithi050.stdout:Sep 18 08:03:10 smithi050 systemd[1]: Started Ceph prometheus.a for 779349ca-7593-11ef-bceb-c7b262605968. 2024-09-18T08:03:10.818 INFO:tasks.cephadm.ceph_manager.ceph:all up! 2024-09-18T08:03:10.818 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid 779349ca-7593-11ef-bceb-c7b262605968 -- ceph osd dump --format=json 2024-09-18T08:03:11.121 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/779349ca-7593-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T08:03:11.156 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:03:10 smithi050 ceph-mon[32371]: pgmap v218: 132 pgs: 132 active+clean; 583 KiB data, 222 MiB used, 715 GiB / 715 GiB avail; 921 B/s rd, 102 B/s wr, 1 op/s 2024-09-18T08:03:11.156 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:03:10 smithi050 ceph-mon[32371]: from='client.? 172.21.15.7:0/3302450984' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2024-09-18T08:03:11.156 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:03:10 smithi050 ceph-mon[32371]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' 2024-09-18T08:03:11.156 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:03:10 smithi050 ceph-mon[32371]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' 2024-09-18T08:03:11.157 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:03:10 smithi050 ceph-mon[32371]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' 2024-09-18T08:03:11.157 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:03:10 smithi050 ceph-mon[32371]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' cmd={"prefix": "mgr module enable", "module": "prometheus"} : dispatch 2024-09-18T08:03:11.230 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:03:10 smithi007 ceph-mon[29459]: pgmap v218: 132 pgs: 132 active+clean; 583 KiB data, 222 MiB used, 715 GiB / 715 GiB avail; 921 B/s rd, 102 B/s wr, 1 op/s 2024-09-18T08:03:11.231 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:03:10 smithi007 ceph-mon[29459]: from='client.? 172.21.15.7:0/3302450984' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2024-09-18T08:03:11.231 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:03:10 smithi007 ceph-mon[29459]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' 2024-09-18T08:03:11.231 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:03:10 smithi007 ceph-mon[29459]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' 2024-09-18T08:03:11.231 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:03:10 smithi007 ceph-mon[29459]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' 2024-09-18T08:03:11.231 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:03:10 smithi007 ceph-mon[29459]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' cmd={"prefix": "mgr module enable", "module": "prometheus"} : dispatch 2024-09-18T08:03:11.231 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:03:10 smithi007 ceph-mon[39249]: pgmap v218: 132 pgs: 132 active+clean; 583 KiB data, 222 MiB used, 715 GiB / 715 GiB avail; 921 B/s rd, 102 B/s wr, 1 op/s 2024-09-18T08:03:11.231 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:03:10 smithi007 ceph-mon[39249]: from='client.? 172.21.15.7:0/3302450984' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2024-09-18T08:03:11.231 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:03:10 smithi007 ceph-mon[39249]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' 2024-09-18T08:03:11.231 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:03:10 smithi007 ceph-mon[39249]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' 2024-09-18T08:03:11.231 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:03:10 smithi007 ceph-mon[39249]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' 2024-09-18T08:03:11.232 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:03:10 smithi007 ceph-mon[39249]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' cmd={"prefix": "mgr module enable", "module": "prometheus"} : dispatch 2024-09-18T08:03:11.677 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Sep 18 08:03:11 smithi007 ceph-779349ca-7593-11ef-bceb-c7b262605968-mgr-y[29688]: ignoring --setuser ceph since I am not root 2024-09-18T08:03:11.677 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Sep 18 08:03:11 smithi007 ceph-779349ca-7593-11ef-bceb-c7b262605968-mgr-y[29688]: ignoring --setgroup ceph since I am not root 2024-09-18T08:03:11.722 INFO:journalctl@ceph.mgr.x.smithi050.stdout:Sep 18 08:03:11 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-mgr-x[33868]: ignoring --setuser ceph since I am not root 2024-09-18T08:03:11.723 INFO:journalctl@ceph.mgr.x.smithi050.stdout:Sep 18 08:03:11 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-mgr-x[33868]: ignoring --setgroup ceph since I am not root 2024-09-18T08:03:11.723 INFO:journalctl@ceph.mgr.x.smithi050.stdout:Sep 18 08:03:11 smithi050 ceph-mgr[33872]: -- 172.21.15.50:0/4201833545 <== mon.1 v2:172.21.15.50:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x557c7a5bf0e0 con 0x557c7a59c800 2024-09-18T08:03:11.967 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-09-18T08:03:11.967 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":72,"fsid":"779349ca-7593-11ef-bceb-c7b262605968","created":"2024-09-18T07:57:09.295710+0000","modified":"2024-09-18T08:03:02.497238+0000","last_up_change":"2024-09-18T08:02:36.989876+0000","last_in_change":"2024-09-18T08:02:14.309111+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":18,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":6,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-18T08:00:11.802627+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":7.8899998664855957,"score_stable":7.8899998664855957,"optimal_score":0.37999999523162842,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"datapool","create_time":"2024-09-18T08:02:45.374099+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":3,"pg_placement_num":3,"pg_placement_num_target":3,"pg_num_target":3,"pg_num_pending":3,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"64","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":2,"snap_epoch":64,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_type":"Fair distribution","score_acting":2.6500000953674316,"score_stable":2.6500000953674316,"optimal_score":0.87999999523162842,"raw_score_acting":2.3299999237060547,"raw_score_stable":2.3299999237060547,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":".rgw.root","create_time":"2024-09-18T08:02:46.319947+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"63","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rgw":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"default.rgw.log","create_time":"2024-09-18T08:02:47.948552+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"65","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rgw":{}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"default.rgw.control","create_time":"2024-09-18T08:02:49.904302+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"67","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rgw":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.25,"score_stable":1.25,"optimal_score":1,"raw_score_acting":1.25,"raw_score_stable":1.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":6,"pool_name":"default.rgw.meta","create_time":"2024-09-18T08:02:51.971465+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"70","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4},"application_metadata":{"rgw":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"ae8f60df-72ac-46fb-91a1-f132b851c876","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6801","nonce":2896425794}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6802","nonce":2896425794}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6804","nonce":2896425794}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6803","nonce":2896425794}]},"public_addr":"172.21.15.7:6801/2896425794","cluster_addr":"172.21.15.7:6802/2896425794","heartbeat_back_addr":"172.21.15.7:6804/2896425794","heartbeat_front_addr":"172.21.15.7:6803/2896425794","state":["exists","up"]},{"osd":1,"uuid":"be28b70c-1f73-494f-ad4b-cb7ec5311999","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6805","nonce":1971397549}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6806","nonce":1971397549}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6808","nonce":1971397549}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6807","nonce":1971397549}]},"public_addr":"172.21.15.7:6805/1971397549","cluster_addr":"172.21.15.7:6806/1971397549","heartbeat_back_addr":"172.21.15.7:6808/1971397549","heartbeat_front_addr":"172.21.15.7:6807/1971397549","state":["exists","up"]},{"osd":2,"uuid":"cfa4f083-c7fd-4743-be65-84a3040fe6dc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6809","nonce":556142559}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6810","nonce":556142559}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6812","nonce":556142559}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6811","nonce":556142559}]},"public_addr":"172.21.15.7:6809/556142559","cluster_addr":"172.21.15.7:6810/556142559","heartbeat_back_addr":"172.21.15.7:6812/556142559","heartbeat_front_addr":"172.21.15.7:6811/556142559","state":["exists","up"]},{"osd":3,"uuid":"11ecd7e1-a751-4b75-95f5-00b32d2a5bc4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6813","nonce":163587772}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6814","nonce":163587772}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6816","nonce":163587772}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6815","nonce":163587772}]},"public_addr":"172.21.15.7:6813/163587772","cluster_addr":"172.21.15.7:6814/163587772","heartbeat_back_addr":"172.21.15.7:6816/163587772","heartbeat_front_addr":"172.21.15.7:6815/163587772","state":["exists","up"]},{"osd":4,"uuid":"98992b34-cfa1-45bf-97a1-7d557739252f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":32,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":3662119805}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6801","nonce":3662119805}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6803","nonce":3662119805}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":3662119805}]},"public_addr":"172.21.15.50:6800/3662119805","cluster_addr":"172.21.15.50:6801/3662119805","heartbeat_back_addr":"172.21.15.50:6803/3662119805","heartbeat_front_addr":"172.21.15.50:6802/3662119805","state":["exists","up"]},{"osd":5,"uuid":"7f290005-ebca-467b-87e4-02ead1063610","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":39,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":1199797954}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6805","nonce":1199797954}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6807","nonce":1199797954}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":1199797954}]},"public_addr":"172.21.15.50:6804/1199797954","cluster_addr":"172.21.15.50:6805/1199797954","heartbeat_back_addr":"172.21.15.50:6807/1199797954","heartbeat_front_addr":"172.21.15.50:6806/1199797954","state":["exists","up"]},{"osd":6,"uuid":"e07d846c-ae42-4fea-aa5c-3a25835d18a6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":47,"up_thru":65,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":255960422}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6809","nonce":255960422}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6811","nonce":255960422}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":255960422}]},"public_addr":"172.21.15.50:6808/255960422","cluster_addr":"172.21.15.50:6809/255960422","heartbeat_back_addr":"172.21.15.50:6811/255960422","heartbeat_front_addr":"172.21.15.50:6810/255960422","state":["exists","up"]},{"osd":7,"uuid":"c9dac3b7-d651-489d-b96e-5b10fa32d9af","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":56,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":874923904}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6813","nonce":874923904}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6815","nonce":874923904}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":874923904}]},"public_addr":"172.21.15.50:6812/874923904","cluster_addr":"172.21.15.50:6813/874923904","heartbeat_back_addr":"172.21.15.50:6815/874923904","heartbeat_front_addr":"172.21.15.50:6814/874923904","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-18T08:02:26.967281+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-18T08:03:01.625260+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-18T08:02:29.901882+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-18T08:02:43.187317+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-18T08:02:11.109010+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-18T08:02:47.507971+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-18T08:02:05.533581+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-18T08:02:35.425165+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.7:0/533727535":"2024-09-19T07:57:57.346149+0000","172.21.15.7:0/1277549846":"2024-09-19T07:57:57.346149+0000","172.21.15.7:6800/1013061071":"2024-09-19T07:57:57.346149+0000","172.21.15.7:0/4272027292":"2024-09-19T07:57:31.015269+0000","172.21.15.7:0/1668598084":"2024-09-19T07:57:57.346149+0000","172.21.15.7:0/3924911335":"2024-09-19T07:57:31.015269+0000","172.21.15.7:0/2514354773":"2024-09-19T07:57:31.015269+0000","172.21.15.7:6800/2918496291":"2024-09-19T07:57:31.015269+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-18T08:03:12.656 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:03:12 smithi050 ceph-mon[32371]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' cmd='[{"prefix": "mgr module enable", "module": "prometheus"}]': finished 2024-09-18T08:03:12.656 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:03:12 smithi050 ceph-mon[32371]: mgrmap e17: y(active, since 5m), standbys: x 2024-09-18T08:03:12.656 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:03:12 smithi050 ceph-mon[32371]: from='client.? 172.21.15.7:0/2253574058' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2024-09-18T08:03:12.664 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:03:12 smithi007 ceph-mon[29459]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' cmd='[{"prefix": "mgr module enable", "module": "prometheus"}]': finished 2024-09-18T08:03:12.665 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:03:12 smithi007 ceph-mon[29459]: mgrmap e17: y(active, since 5m), standbys: x 2024-09-18T08:03:12.665 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:03:12 smithi007 ceph-mon[29459]: from='client.? 172.21.15.7:0/2253574058' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2024-09-18T08:03:12.665 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:03:12 smithi007 ceph-mon[39249]: from='mgr.14152 172.21.15.7:0/2293643967' entity='mgr.y' cmd='[{"prefix": "mgr module enable", "module": "prometheus"}]': finished 2024-09-18T08:03:12.665 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:03:12 smithi007 ceph-mon[39249]: mgrmap e17: y(active, since 5m), standbys: x 2024-09-18T08:03:12.665 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:03:12 smithi007 ceph-mon[39249]: from='client.? 172.21.15.7:0/2253574058' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2024-09-18T08:03:12.695 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid 779349ca-7593-11ef-bceb-c7b262605968 -- ceph tell osd.0 flush_pg_stats 2024-09-18T08:03:12.696 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid 779349ca-7593-11ef-bceb-c7b262605968 -- ceph tell osd.1 flush_pg_stats 2024-09-18T08:03:12.696 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid 779349ca-7593-11ef-bceb-c7b262605968 -- ceph tell osd.2 flush_pg_stats 2024-09-18T08:03:12.696 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid 779349ca-7593-11ef-bceb-c7b262605968 -- ceph tell osd.3 flush_pg_stats 2024-09-18T08:03:12.697 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid 779349ca-7593-11ef-bceb-c7b262605968 -- ceph tell osd.4 flush_pg_stats 2024-09-18T08:03:12.697 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid 779349ca-7593-11ef-bceb-c7b262605968 -- ceph tell osd.5 flush_pg_stats 2024-09-18T08:03:12.698 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid 779349ca-7593-11ef-bceb-c7b262605968 -- ceph tell osd.6 flush_pg_stats 2024-09-18T08:03:12.698 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid 779349ca-7593-11ef-bceb-c7b262605968 -- ceph tell osd.7 flush_pg_stats 2024-09-18T08:03:13.216 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/779349ca-7593-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T08:03:13.223 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/779349ca-7593-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T08:03:13.223 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/779349ca-7593-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T08:03:13.262 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/779349ca-7593-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T08:03:13.477 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/779349ca-7593-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T08:03:13.477 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/779349ca-7593-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T08:03:13.478 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/779349ca-7593-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T08:03:15.058 INFO:teuthology.orchestra.run.smithi007.stdout:201863462927 2024-09-18T08:03:15.059 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid 779349ca-7593-11ef-bceb-c7b262605968 -- ceph osd last-stat-seq osd.6 2024-09-18T08:03:16.149 INFO:teuthology.orchestra.run.smithi007.stdout:34359738418 2024-09-18T08:03:16.150 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid 779349ca-7593-11ef-bceb-c7b262605968 -- ceph osd last-stat-seq osd.0 2024-09-18T08:03:16.906 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:03:16 smithi050 ceph-mon[32371]: Standby manager daemon x restarted 2024-09-18T08:03:16.906 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:03:16 smithi050 ceph-mon[32371]: Standby manager daemon x started 2024-09-18T08:03:16.906 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:03:16 smithi050 ceph-mon[32371]: from='mgr.? 172.21.15.50:0/527301329' entity='mgr.x' cmd={"prefix": "config-key get", "key": "mgr/dashboard/x/crt"} : dispatch 2024-09-18T08:03:16.906 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:03:16 smithi050 ceph-mon[32371]: from='mgr.? 172.21.15.50:0/527301329' entity='mgr.x' cmd={"prefix": "config-key get", "key": "mgr/dashboard/crt"} : dispatch 2024-09-18T08:03:16.906 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:03:16 smithi050 ceph-mon[32371]: from='mgr.? 172.21.15.50:0/527301329' entity='mgr.x' cmd={"prefix": "config-key get", "key": "mgr/dashboard/x/key"} : dispatch 2024-09-18T08:03:16.907 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:03:16 smithi050 ceph-mon[32371]: from='mgr.? 172.21.15.50:0/527301329' entity='mgr.x' cmd={"prefix": "config-key get", "key": "mgr/dashboard/key"} : dispatch 2024-09-18T08:03:16.907 INFO:journalctl@ceph.mgr.x.smithi050.stdout:Sep 18 08:03:16 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-mgr-x[33868]: [18/Sep/2024:08:03:16] ENGINE Bus STARTING 2024-09-18T08:03:16.907 INFO:journalctl@ceph.mgr.x.smithi050.stdout:Sep 18 08:03:16 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-mgr-x[33868]: CherryPy Checker: 2024-09-18T08:03:16.907 INFO:journalctl@ceph.mgr.x.smithi050.stdout:Sep 18 08:03:16 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-mgr-x[33868]: The Application mounted at '' has an empty config. 2024-09-18T08:03:16.907 INFO:journalctl@ceph.mgr.x.smithi050.stdout:Sep 18 08:03:16 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-mgr-x[33868]: 2024-09-18T08:03:16.907 INFO:journalctl@ceph.mgr.x.smithi050.stdout:Sep 18 08:03:16 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-mgr-x[33868]: [18/Sep/2024:08:03:16] ENGINE Serving on http://:::9283 2024-09-18T08:03:16.907 INFO:journalctl@ceph.mgr.x.smithi050.stdout:Sep 18 08:03:16 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-mgr-x[33868]: [18/Sep/2024:08:03:16] ENGINE Bus STARTED 2024-09-18T08:03:16.980 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:03:16 smithi007 ceph-mon[29459]: Standby manager daemon x restarted 2024-09-18T08:03:16.981 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:03:16 smithi007 ceph-mon[29459]: Standby manager daemon x started 2024-09-18T08:03:16.981 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:03:16 smithi007 ceph-mon[29459]: from='mgr.? 172.21.15.50:0/527301329' entity='mgr.x' cmd={"prefix": "config-key get", "key": "mgr/dashboard/x/crt"} : dispatch 2024-09-18T08:03:16.981 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:03:16 smithi007 ceph-mon[29459]: from='mgr.? 172.21.15.50:0/527301329' entity='mgr.x' cmd={"prefix": "config-key get", "key": "mgr/dashboard/crt"} : dispatch 2024-09-18T08:03:16.981 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:03:16 smithi007 ceph-mon[29459]: from='mgr.? 172.21.15.50:0/527301329' entity='mgr.x' cmd={"prefix": "config-key get", "key": "mgr/dashboard/x/key"} : dispatch 2024-09-18T08:03:16.981 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:03:16 smithi007 ceph-mon[29459]: from='mgr.? 172.21.15.50:0/527301329' entity='mgr.x' cmd={"prefix": "config-key get", "key": "mgr/dashboard/key"} : dispatch 2024-09-18T08:03:16.981 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:03:16 smithi007 ceph-mon[39249]: Standby manager daemon x restarted 2024-09-18T08:03:16.981 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:03:16 smithi007 ceph-mon[39249]: Standby manager daemon x started 2024-09-18T08:03:16.981 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:03:16 smithi007 ceph-mon[39249]: from='mgr.? 172.21.15.50:0/527301329' entity='mgr.x' cmd={"prefix": "config-key get", "key": "mgr/dashboard/x/crt"} : dispatch 2024-09-18T08:03:16.981 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:03:16 smithi007 ceph-mon[39249]: from='mgr.? 172.21.15.50:0/527301329' entity='mgr.x' cmd={"prefix": "config-key get", "key": "mgr/dashboard/crt"} : dispatch 2024-09-18T08:03:16.981 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:03:16 smithi007 ceph-mon[39249]: from='mgr.? 172.21.15.50:0/527301329' entity='mgr.x' cmd={"prefix": "config-key get", "key": "mgr/dashboard/x/key"} : dispatch 2024-09-18T08:03:16.982 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:03:16 smithi007 ceph-mon[39249]: from='mgr.? 172.21.15.50:0/527301329' entity='mgr.x' cmd={"prefix": "config-key get", "key": "mgr/dashboard/key"} : dispatch 2024-09-18T08:03:17.010 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/779349ca-7593-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T08:03:17.981 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:03:17 smithi007 ceph-mon[29459]: mgrmap e18: y(active, since 5m), standbys: x 2024-09-18T08:03:17.981 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:03:17 smithi007 ceph-mon[29459]: Active manager daemon y restarted 2024-09-18T08:03:17.982 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:03:17 smithi007 ceph-mon[29459]: Activating manager daemon y 2024-09-18T08:03:17.982 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:03:17 smithi007 ceph-mon[29459]: osdmap e73: 8 total, 8 up, 8 in 2024-09-18T08:03:17.982 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:03:17 smithi007 ceph-mon[29459]: mgrmap e19: y(active, starting, since 0.00901632s), standbys: x 2024-09-18T08:03:17.982 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:03:17 smithi007 ceph-mon[29459]: from='mgr.24470 172.21.15.7:0/2594390261' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "a"} : dispatch 2024-09-18T08:03:17.982 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:03:17 smithi007 ceph-mon[29459]: from='mgr.24470 172.21.15.7:0/2594390261' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-09-18T08:03:17.982 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:03:17 smithi007 ceph-mon[29459]: from='mgr.24470 172.21.15.7:0/2594390261' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-09-18T08:03:17.982 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:03:17 smithi007 ceph-mon[29459]: from='mgr.24470 172.21.15.7:0/2594390261' entity='mgr.y' cmd={"prefix": "mgr metadata", "who": "y", "id": "y"} : dispatch 2024-09-18T08:03:17.982 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:03:17 smithi007 ceph-mon[29459]: from='mgr.24470 172.21.15.7:0/2594390261' entity='mgr.y' cmd={"prefix": "mgr metadata", "who": "x", "id": "x"} : dispatch 2024-09-18T08:03:17.982 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:03:17 smithi007 ceph-mon[29459]: from='mgr.24470 172.21.15.7:0/2594390261' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 0} : dispatch 2024-09-18T08:03:17.982 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:03:17 smithi007 ceph-mon[29459]: from='mgr.24470 172.21.15.7:0/2594390261' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 1} : dispatch 2024-09-18T08:03:17.983 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:03:17 smithi007 ceph-mon[29459]: from='mgr.24470 172.21.15.7:0/2594390261' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 2} : dispatch 2024-09-18T08:03:17.983 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:03:17 smithi007 ceph-mon[29459]: from='mgr.24470 172.21.15.7:0/2594390261' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 3} : dispatch 2024-09-18T08:03:17.983 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:03:17 smithi007 ceph-mon[29459]: from='mgr.24470 172.21.15.7:0/2594390261' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 4} : dispatch 2024-09-18T08:03:17.983 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:03:17 smithi007 ceph-mon[29459]: from='mgr.24470 172.21.15.7:0/2594390261' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 5} : dispatch 2024-09-18T08:03:17.983 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:03:17 smithi007 ceph-mon[29459]: from='mgr.24470 172.21.15.7:0/2594390261' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 6} : dispatch 2024-09-18T08:03:17.983 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:03:17 smithi007 ceph-mon[29459]: from='mgr.24470 172.21.15.7:0/2594390261' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 7} : dispatch 2024-09-18T08:03:17.983 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:03:17 smithi007 ceph-mon[29459]: from='mgr.24470 172.21.15.7:0/2594390261' entity='mgr.y' cmd={"prefix": "mds metadata"} : dispatch 2024-09-18T08:03:17.983 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:03:17 smithi007 ceph-mon[29459]: from='mgr.24470 172.21.15.7:0/2594390261' entity='mgr.y' cmd={"prefix": "osd metadata"} : dispatch 2024-09-18T08:03:17.984 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:03:17 smithi007 ceph-mon[29459]: from='mgr.24470 172.21.15.7:0/2594390261' entity='mgr.y' cmd={"prefix": "mon metadata"} : dispatch 2024-09-18T08:03:17.984 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:03:17 smithi007 ceph-mon[29459]: Manager daemon y is now available 2024-09-18T08:03:17.984 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:03:17 smithi007 ceph-mon[29459]: from='mgr.24470 172.21.15.7:0/2594390261' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-18T08:03:17.984 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:03:17 smithi007 ceph-mon[29459]: from='mgr.24470 172.21.15.7:0/2594390261' entity='mgr.y' 2024-09-18T08:03:17.984 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:03:17 smithi007 ceph-mon[29459]: from='mgr.24470 172.21.15.7:0/2594390261' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-18T08:03:17.984 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:03:17 smithi007 ceph-mon[29459]: from='mgr.24470 172.21.15.7:0/2594390261' entity='mgr.y' cmd={"prefix": "orch prometheus get-credentials"} : dispatch 2024-09-18T08:03:17.984 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:03:17 smithi007 ceph-mon[29459]: from='mgr.24470 172.21.15.7:0/2594390261' entity='mgr.y' cmd={"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"} : dispatch 2024-09-18T08:03:17.984 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:03:17 smithi007 ceph-mon[29459]: from='mgr.24470 172.21.15.7:0/2594390261' entity='mgr.y' cmd={"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"} : dispatch 2024-09-18T08:03:17.985 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:03:17 smithi007 ceph-mon[39249]: mgrmap e18: y(active, since 5m), standbys: x 2024-09-18T08:03:17.985 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:03:17 smithi007 ceph-mon[39249]: Active manager daemon y restarted 2024-09-18T08:03:17.985 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:03:17 smithi007 ceph-mon[39249]: Activating manager daemon y 2024-09-18T08:03:17.985 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:03:17 smithi007 ceph-mon[39249]: osdmap e73: 8 total, 8 up, 8 in 2024-09-18T08:03:17.985 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:03:17 smithi007 ceph-mon[39249]: mgrmap e19: y(active, starting, since 0.00901632s), standbys: x 2024-09-18T08:03:17.985 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:03:17 smithi007 ceph-mon[39249]: from='mgr.24470 172.21.15.7:0/2594390261' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "a"} : dispatch 2024-09-18T08:03:17.985 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:03:17 smithi007 ceph-mon[39249]: from='mgr.24470 172.21.15.7:0/2594390261' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-09-18T08:03:17.986 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:03:17 smithi007 ceph-mon[39249]: from='mgr.24470 172.21.15.7:0/2594390261' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-09-18T08:03:17.986 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:03:17 smithi007 ceph-mon[39249]: from='mgr.24470 172.21.15.7:0/2594390261' entity='mgr.y' cmd={"prefix": "mgr metadata", "who": "y", "id": "y"} : dispatch 2024-09-18T08:03:17.986 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:03:17 smithi007 ceph-mon[39249]: from='mgr.24470 172.21.15.7:0/2594390261' entity='mgr.y' cmd={"prefix": "mgr metadata", "who": "x", "id": "x"} : dispatch 2024-09-18T08:03:17.986 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:03:17 smithi007 ceph-mon[39249]: from='mgr.24470 172.21.15.7:0/2594390261' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 0} : dispatch 2024-09-18T08:03:17.986 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:03:17 smithi007 ceph-mon[39249]: from='mgr.24470 172.21.15.7:0/2594390261' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 1} : dispatch 2024-09-18T08:03:17.986 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:03:17 smithi007 ceph-mon[39249]: from='mgr.24470 172.21.15.7:0/2594390261' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 2} : dispatch 2024-09-18T08:03:17.986 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:03:17 smithi007 ceph-mon[39249]: from='mgr.24470 172.21.15.7:0/2594390261' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 3} : dispatch 2024-09-18T08:03:17.986 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:03:17 smithi007 ceph-mon[39249]: from='mgr.24470 172.21.15.7:0/2594390261' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 4} : dispatch 2024-09-18T08:03:17.986 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:03:17 smithi007 ceph-mon[39249]: from='mgr.24470 172.21.15.7:0/2594390261' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 5} : dispatch 2024-09-18T08:03:17.987 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:03:17 smithi007 ceph-mon[39249]: from='mgr.24470 172.21.15.7:0/2594390261' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 6} : dispatch 2024-09-18T08:03:17.987 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:03:17 smithi007 ceph-mon[39249]: from='mgr.24470 172.21.15.7:0/2594390261' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 7} : dispatch 2024-09-18T08:03:17.987 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:03:17 smithi007 ceph-mon[39249]: from='mgr.24470 172.21.15.7:0/2594390261' entity='mgr.y' cmd={"prefix": "mds metadata"} : dispatch 2024-09-18T08:03:17.987 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:03:17 smithi007 ceph-mon[39249]: from='mgr.24470 172.21.15.7:0/2594390261' entity='mgr.y' cmd={"prefix": "osd metadata"} : dispatch 2024-09-18T08:03:17.987 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:03:17 smithi007 ceph-mon[39249]: from='mgr.24470 172.21.15.7:0/2594390261' entity='mgr.y' cmd={"prefix": "mon metadata"} : dispatch 2024-09-18T08:03:17.987 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:03:17 smithi007 ceph-mon[39249]: Manager daemon y is now available 2024-09-18T08:03:17.987 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:03:17 smithi007 ceph-mon[39249]: from='mgr.24470 172.21.15.7:0/2594390261' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-18T08:03:17.987 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:03:17 smithi007 ceph-mon[39249]: from='mgr.24470 172.21.15.7:0/2594390261' entity='mgr.y' 2024-09-18T08:03:17.988 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:03:17 smithi007 ceph-mon[39249]: from='mgr.24470 172.21.15.7:0/2594390261' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-18T08:03:17.988 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:03:17 smithi007 ceph-mon[39249]: from='mgr.24470 172.21.15.7:0/2594390261' entity='mgr.y' cmd={"prefix": "orch prometheus get-credentials"} : dispatch 2024-09-18T08:03:17.988 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:03:17 smithi007 ceph-mon[39249]: from='mgr.24470 172.21.15.7:0/2594390261' entity='mgr.y' cmd={"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"} : dispatch 2024-09-18T08:03:17.988 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:03:17 smithi007 ceph-mon[39249]: from='mgr.24470 172.21.15.7:0/2594390261' entity='mgr.y' cmd={"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"} : dispatch 2024-09-18T08:03:18.156 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:03:17 smithi050 ceph-mon[32371]: mgrmap e18: y(active, since 5m), standbys: x 2024-09-18T08:03:18.156 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:03:17 smithi050 ceph-mon[32371]: Active manager daemon y restarted 2024-09-18T08:03:18.156 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:03:17 smithi050 ceph-mon[32371]: Activating manager daemon y 2024-09-18T08:03:18.156 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:03:17 smithi050 ceph-mon[32371]: osdmap e73: 8 total, 8 up, 8 in 2024-09-18T08:03:18.157 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:03:17 smithi050 ceph-mon[32371]: mgrmap e19: y(active, starting, since 0.00901632s), standbys: x 2024-09-18T08:03:18.157 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:03:17 smithi050 ceph-mon[32371]: from='mgr.24470 172.21.15.7:0/2594390261' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "a"} : dispatch 2024-09-18T08:03:18.157 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:03:17 smithi050 ceph-mon[32371]: from='mgr.24470 172.21.15.7:0/2594390261' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-09-18T08:03:18.157 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:03:17 smithi050 ceph-mon[32371]: from='mgr.24470 172.21.15.7:0/2594390261' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-09-18T08:03:18.157 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:03:17 smithi050 ceph-mon[32371]: from='mgr.24470 172.21.15.7:0/2594390261' entity='mgr.y' cmd={"prefix": "mgr metadata", "who": "y", "id": "y"} : dispatch 2024-09-18T08:03:18.157 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:03:17 smithi050 ceph-mon[32371]: from='mgr.24470 172.21.15.7:0/2594390261' entity='mgr.y' cmd={"prefix": "mgr metadata", "who": "x", "id": "x"} : dispatch 2024-09-18T08:03:18.157 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:03:17 smithi050 ceph-mon[32371]: from='mgr.24470 172.21.15.7:0/2594390261' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 0} : dispatch 2024-09-18T08:03:18.157 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:03:17 smithi050 ceph-mon[32371]: from='mgr.24470 172.21.15.7:0/2594390261' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 1} : dispatch 2024-09-18T08:03:18.157 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:03:17 smithi050 ceph-mon[32371]: from='mgr.24470 172.21.15.7:0/2594390261' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 2} : dispatch 2024-09-18T08:03:18.157 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:03:17 smithi050 ceph-mon[32371]: from='mgr.24470 172.21.15.7:0/2594390261' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 3} : dispatch 2024-09-18T08:03:18.157 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:03:17 smithi050 ceph-mon[32371]: from='mgr.24470 172.21.15.7:0/2594390261' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 4} : dispatch 2024-09-18T08:03:18.157 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:03:17 smithi050 ceph-mon[32371]: from='mgr.24470 172.21.15.7:0/2594390261' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 5} : dispatch 2024-09-18T08:03:18.157 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:03:17 smithi050 ceph-mon[32371]: from='mgr.24470 172.21.15.7:0/2594390261' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 6} : dispatch 2024-09-18T08:03:18.157 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:03:17 smithi050 ceph-mon[32371]: from='mgr.24470 172.21.15.7:0/2594390261' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 7} : dispatch 2024-09-18T08:03:18.158 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:03:17 smithi050 ceph-mon[32371]: from='mgr.24470 172.21.15.7:0/2594390261' entity='mgr.y' cmd={"prefix": "mds metadata"} : dispatch 2024-09-18T08:03:18.158 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:03:17 smithi050 ceph-mon[32371]: from='mgr.24470 172.21.15.7:0/2594390261' entity='mgr.y' cmd={"prefix": "osd metadata"} : dispatch 2024-09-18T08:03:18.158 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:03:17 smithi050 ceph-mon[32371]: from='mgr.24470 172.21.15.7:0/2594390261' entity='mgr.y' cmd={"prefix": "mon metadata"} : dispatch 2024-09-18T08:03:18.158 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:03:17 smithi050 ceph-mon[32371]: Manager daemon y is now available 2024-09-18T08:03:18.158 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:03:17 smithi050 ceph-mon[32371]: from='mgr.24470 172.21.15.7:0/2594390261' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-18T08:03:18.158 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:03:17 smithi050 ceph-mon[32371]: from='mgr.24470 172.21.15.7:0/2594390261' entity='mgr.y' 2024-09-18T08:03:18.158 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:03:17 smithi050 ceph-mon[32371]: from='mgr.24470 172.21.15.7:0/2594390261' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-18T08:03:18.158 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:03:17 smithi050 ceph-mon[32371]: from='mgr.24470 172.21.15.7:0/2594390261' entity='mgr.y' cmd={"prefix": "orch prometheus get-credentials"} : dispatch 2024-09-18T08:03:18.158 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:03:17 smithi050 ceph-mon[32371]: from='mgr.24470 172.21.15.7:0/2594390261' entity='mgr.y' cmd={"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"} : dispatch 2024-09-18T08:03:18.158 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:03:17 smithi050 ceph-mon[32371]: from='mgr.24470 172.21.15.7:0/2594390261' entity='mgr.y' cmd={"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"} : dispatch 2024-09-18T08:03:18.286 INFO:teuthology.orchestra.run.smithi007.stdout:167503724565 2024-09-18T08:03:18.287 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid 779349ca-7593-11ef-bceb-c7b262605968 -- ceph osd last-stat-seq osd.5 2024-09-18T08:03:18.372 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Sep 18 08:03:18 smithi007 ceph-779349ca-7593-11ef-bceb-c7b262605968-mgr-y[29688]: [18/Sep/2024:08:03:18] ENGINE Bus STARTING 2024-09-18T08:03:18.372 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Sep 18 08:03:18 smithi007 ceph-779349ca-7593-11ef-bceb-c7b262605968-mgr-y[29688]: CherryPy Checker: 2024-09-18T08:03:18.372 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Sep 18 08:03:18 smithi007 ceph-779349ca-7593-11ef-bceb-c7b262605968-mgr-y[29688]: The Application mounted at '' has an empty config. 2024-09-18T08:03:18.372 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Sep 18 08:03:18 smithi007 ceph-779349ca-7593-11ef-bceb-c7b262605968-mgr-y[29688]: 2024-09-18T08:03:18.372 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Sep 18 08:03:18 smithi007 ceph-779349ca-7593-11ef-bceb-c7b262605968-mgr-y[29688]: [18/Sep/2024:08:03:18] ENGINE Serving on http://:::9283 2024-09-18T08:03:18.372 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Sep 18 08:03:18 smithi007 ceph-779349ca-7593-11ef-bceb-c7b262605968-mgr-y[29688]: [18/Sep/2024:08:03:18] ENGINE Bus STARTED 2024-09-18T08:03:19.065 INFO:journalctl@ceph.iscsi.iscsi.a.smithi050.stdout:Sep 18 08:03:18 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-iscsi-iscsi-a[57782]: debug there is no tcmu-runner data available 2024-09-18T08:03:19.121 INFO:teuthology.orchestra.run.smithi007.stdout:55834574891 2024-09-18T08:03:19.122 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid 779349ca-7593-11ef-bceb-c7b262605968 -- ceph osd last-stat-seq osd.1 2024-09-18T08:03:19.400 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:03:19 smithi050 ceph-mon[32371]: mgrmap e20: y(active, since 1.01207s), standbys: x 2024-09-18T08:03:19.401 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:03:19 smithi050 ceph-mon[32371]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "orch prometheus get-credentials"}]: dispatch 2024-09-18T08:03:19.401 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:03:19 smithi050 ceph-mon[32371]: pgmap v3: 132 pgs: 132 active+clean; 583 KiB data, 222 MiB used, 715 GiB / 715 GiB avail 2024-09-18T08:03:19.401 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:03:19 smithi050 ceph-mon[32371]: osdmap e74: 8 total, 8 up, 8 in 2024-09-18T08:03:19.422 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:03:19 smithi007 ceph-mon[29459]: mgrmap e20: y(active, since 1.01207s), standbys: x 2024-09-18T08:03:19.422 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:03:19 smithi007 ceph-mon[29459]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "orch prometheus get-credentials"}]: dispatch 2024-09-18T08:03:19.422 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:03:19 smithi007 ceph-mon[29459]: pgmap v3: 132 pgs: 132 active+clean; 583 KiB data, 222 MiB used, 715 GiB / 715 GiB avail 2024-09-18T08:03:19.423 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:03:19 smithi007 ceph-mon[29459]: osdmap e74: 8 total, 8 up, 8 in 2024-09-18T08:03:19.423 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:03:19 smithi007 ceph-mon[39249]: mgrmap e20: y(active, since 1.01207s), standbys: x 2024-09-18T08:03:19.423 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:03:19 smithi007 ceph-mon[39249]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "orch prometheus get-credentials"}]: dispatch 2024-09-18T08:03:19.424 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:03:19 smithi007 ceph-mon[39249]: pgmap v3: 132 pgs: 132 active+clean; 583 KiB data, 222 MiB used, 715 GiB / 715 GiB avail 2024-09-18T08:03:19.424 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:03:19 smithi007 ceph-mon[39249]: osdmap e74: 8 total, 8 up, 8 in 2024-09-18T08:03:19.424 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Sep 18 08:03:19 smithi007 ceph-779349ca-7593-11ef-bceb-c7b262605968-mgr-y[29688]: 2024-09-18T08:03:19.094+0000 7f9060a88640 -1 log_channel(cephadm) log [ERR] : [18/Sep/2024:08:03:19] ENGINE Error in HTTPServer.serve 2024-09-18T08:03:19.424 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Sep 18 08:03:19 smithi007 ceph-779349ca-7593-11ef-bceb-c7b262605968-mgr-y[29688]: Traceback (most recent call last): 2024-09-18T08:03:19.425 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Sep 18 08:03:19 smithi007 ceph-779349ca-7593-11ef-bceb-c7b262605968-mgr-y[29688]: File "/lib/python3.9/site-packages/cheroot/server.py", line 1823, in serve 2024-09-18T08:03:19.425 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Sep 18 08:03:19 smithi007 ceph-779349ca-7593-11ef-bceb-c7b262605968-mgr-y[29688]: self._connections.run(self.expiration_interval) 2024-09-18T08:03:19.425 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Sep 18 08:03:19 smithi007 ceph-779349ca-7593-11ef-bceb-c7b262605968-mgr-y[29688]: File "/lib/python3.9/site-packages/cheroot/connections.py", line 203, in run 2024-09-18T08:03:19.425 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Sep 18 08:03:19 smithi007 ceph-779349ca-7593-11ef-bceb-c7b262605968-mgr-y[29688]: self._run(expiration_interval) 2024-09-18T08:03:19.425 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Sep 18 08:03:19 smithi007 ceph-779349ca-7593-11ef-bceb-c7b262605968-mgr-y[29688]: File "/lib/python3.9/site-packages/cheroot/connections.py", line 246, in _run 2024-09-18T08:03:19.425 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Sep 18 08:03:19 smithi007 ceph-779349ca-7593-11ef-bceb-c7b262605968-mgr-y[29688]: new_conn = self._from_server_socket(self.server.socket) 2024-09-18T08:03:19.426 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Sep 18 08:03:19 smithi007 ceph-779349ca-7593-11ef-bceb-c7b262605968-mgr-y[29688]: File "/lib/python3.9/site-packages/cheroot/connections.py", line 300, in _from_server_socket 2024-09-18T08:03:19.426 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Sep 18 08:03:19 smithi007 ceph-779349ca-7593-11ef-bceb-c7b262605968-mgr-y[29688]: s, ssl_env = self.server.ssl_adapter.wrap(s) 2024-09-18T08:03:19.426 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Sep 18 08:03:19 smithi007 ceph-779349ca-7593-11ef-bceb-c7b262605968-mgr-y[29688]: File "/lib/python3.9/site-packages/cheroot/ssl/builtin.py", line 277, in wrap 2024-09-18T08:03:19.426 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Sep 18 08:03:19 smithi007 ceph-779349ca-7593-11ef-bceb-c7b262605968-mgr-y[29688]: s = self.context.wrap_socket( 2024-09-18T08:03:19.426 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Sep 18 08:03:19 smithi007 ceph-779349ca-7593-11ef-bceb-c7b262605968-mgr-y[29688]: File "/lib64/python3.9/ssl.py", line 501, in wrap_socket 2024-09-18T08:03:19.426 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Sep 18 08:03:19 smithi007 ceph-779349ca-7593-11ef-bceb-c7b262605968-mgr-y[29688]: return self.sslsocket_class._create( 2024-09-18T08:03:19.427 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Sep 18 08:03:19 smithi007 ceph-779349ca-7593-11ef-bceb-c7b262605968-mgr-y[29688]: File "/lib64/python3.9/ssl.py", line 1074, in _create 2024-09-18T08:03:19.427 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Sep 18 08:03:19 smithi007 ceph-779349ca-7593-11ef-bceb-c7b262605968-mgr-y[29688]: self.do_handshake() 2024-09-18T08:03:19.427 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Sep 18 08:03:19 smithi007 ceph-779349ca-7593-11ef-bceb-c7b262605968-mgr-y[29688]: File "/lib64/python3.9/ssl.py", line 1343, in do_handshake 2024-09-18T08:03:19.427 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Sep 18 08:03:19 smithi007 ceph-779349ca-7593-11ef-bceb-c7b262605968-mgr-y[29688]: self._sslobj.do_handshake() 2024-09-18T08:03:19.427 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Sep 18 08:03:19 smithi007 ceph-779349ca-7593-11ef-bceb-c7b262605968-mgr-y[29688]: ssl.SSLZeroReturnError: TLS/SSL connection has been closed (EOF) (_ssl.c:1133) 2024-09-18T08:03:19.427 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Sep 18 08:03:19 smithi007 ceph-779349ca-7593-11ef-bceb-c7b262605968-mgr-y[29688]: 2024-09-18T08:03:19.695 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/779349ca-7593-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T08:03:19.705 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/779349ca-7593-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T08:03:20.166 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:03:20 smithi007 ceph-mon[29459]: [18/Sep/2024:08:03:18] ENGINE Bus STARTING 2024-09-18T08:03:20.166 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:03:20 smithi007 ceph-mon[29459]: [18/Sep/2024:08:03:18] ENGINE Serving on http://172.21.15.7:8765 2024-09-18T08:03:20.166 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:03:20 smithi007 ceph-mon[29459]: pgmap v5: 132 pgs: 132 active+clean; 583 KiB data, 222 MiB used, 715 GiB / 715 GiB avail 2024-09-18T08:03:20.166 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:03:20 smithi007 ceph-mon[29459]: [18/Sep/2024:08:03:19] ENGINE Serving on https://172.21.15.7:7150 2024-09-18T08:03:20.166 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:03:20 smithi007 ceph-mon[29459]: [18/Sep/2024:08:03:19] ENGINE Bus STARTED 2024-09-18T08:03:20.166 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:03:20 smithi007 ceph-mon[29459]: [18/Sep/2024:08:03:19] ENGINE Error in HTTPServer.serve 2024-09-18T08:03:20.166 INFO:journalctl@ceph.mon.a.smithi007.stdout: Traceback (most recent call last): 2024-09-18T08:03:20.166 INFO:journalctl@ceph.mon.a.smithi007.stdout: File "/lib/python3.9/site-packages/cheroot/server.py", line 1823, in serve 2024-09-18T08:03:20.166 INFO:journalctl@ceph.mon.a.smithi007.stdout: self._connections.run(self.expiration_interval) 2024-09-18T08:03:20.166 INFO:journalctl@ceph.mon.a.smithi007.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 203, in run 2024-09-18T08:03:20.166 INFO:journalctl@ceph.mon.a.smithi007.stdout: self._run(expiration_interval) 2024-09-18T08:03:20.167 INFO:journalctl@ceph.mon.a.smithi007.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 246, in _run 2024-09-18T08:03:20.167 INFO:journalctl@ceph.mon.a.smithi007.stdout: new_conn = self._from_server_socket(self.server.socket) 2024-09-18T08:03:20.167 INFO:journalctl@ceph.mon.a.smithi007.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 300, in _from_server_socket 2024-09-18T08:03:20.167 INFO:journalctl@ceph.mon.a.smithi007.stdout: s, ssl_env = self.server.ssl_adapter.wrap(s) 2024-09-18T08:03:20.167 INFO:journalctl@ceph.mon.a.smithi007.stdout: File "/lib/python3.9/site-packages/cheroot/ssl/builtin.py", line 277, in wrap 2024-09-18T08:03:20.167 INFO:journalctl@ceph.mon.a.smithi007.stdout: s = self.context.wrap_socket( 2024-09-18T08:03:20.167 INFO:journalctl@ceph.mon.a.smithi007.stdout: File "/lib64/python3.9/ssl.py", line 501, in wrap_socket 2024-09-18T08:03:20.167 INFO:journalctl@ceph.mon.a.smithi007.stdout: return self.sslsocket_class._create( 2024-09-18T08:03:20.167 INFO:journalctl@ceph.mon.a.smithi007.stdout: File "/lib64/python3.9/ssl.py", line 1074, in _create 2024-09-18T08:03:20.167 INFO:journalctl@ceph.mon.a.smithi007.stdout: self.do_handshake() 2024-09-18T08:03:20.167 INFO:journalctl@ceph.mon.a.smithi007.stdout: File "/lib64/python3.9/ssl.py", line 1343, in do_handshake 2024-09-18T08:03:20.168 INFO:journalctl@ceph.mon.a.smithi007.stdout: self._sslobj.do_handshake() 2024-09-18T08:03:20.168 INFO:journalctl@ceph.mon.a.smithi007.stdout: ssl.SSLZeroReturnError: TLS/SSL connection has been closed (EOF) (_ssl.c:1133) 2024-09-18T08:03:20.168 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:03:20 smithi007 ceph-mon[29459]: mgrmap e21: y(active, since 2s), standbys: x 2024-09-18T08:03:20.168 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:03:20 smithi007 ceph-mon[39249]: [18/Sep/2024:08:03:18] ENGINE Bus STARTING 2024-09-18T08:03:20.168 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:03:20 smithi007 ceph-mon[39249]: [18/Sep/2024:08:03:18] ENGINE Serving on http://172.21.15.7:8765 2024-09-18T08:03:20.168 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:03:20 smithi007 ceph-mon[39249]: pgmap v5: 132 pgs: 132 active+clean; 583 KiB data, 222 MiB used, 715 GiB / 715 GiB avail 2024-09-18T08:03:20.168 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:03:20 smithi007 ceph-mon[39249]: [18/Sep/2024:08:03:19] ENGINE Serving on https://172.21.15.7:7150 2024-09-18T08:03:20.168 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:03:20 smithi007 ceph-mon[39249]: [18/Sep/2024:08:03:19] ENGINE Bus STARTED 2024-09-18T08:03:20.168 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:03:20 smithi007 ceph-mon[39249]: [18/Sep/2024:08:03:19] ENGINE Error in HTTPServer.serve 2024-09-18T08:03:20.169 INFO:journalctl@ceph.mon.c.smithi007.stdout: Traceback (most recent call last): 2024-09-18T08:03:20.169 INFO:journalctl@ceph.mon.c.smithi007.stdout: File "/lib/python3.9/site-packages/cheroot/server.py", line 1823, in serve 2024-09-18T08:03:20.169 INFO:journalctl@ceph.mon.c.smithi007.stdout: self._connections.run(self.expiration_interval) 2024-09-18T08:03:20.169 INFO:journalctl@ceph.mon.c.smithi007.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 203, in run 2024-09-18T08:03:20.169 INFO:journalctl@ceph.mon.c.smithi007.stdout: self._run(expiration_interval) 2024-09-18T08:03:20.169 INFO:journalctl@ceph.mon.c.smithi007.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 246, in _run 2024-09-18T08:03:20.169 INFO:journalctl@ceph.mon.c.smithi007.stdout: new_conn = self._from_server_socket(self.server.socket) 2024-09-18T08:03:20.169 INFO:journalctl@ceph.mon.c.smithi007.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 300, in _from_server_socket 2024-09-18T08:03:20.169 INFO:journalctl@ceph.mon.c.smithi007.stdout: s, ssl_env = self.server.ssl_adapter.wrap(s) 2024-09-18T08:03:20.169 INFO:journalctl@ceph.mon.c.smithi007.stdout: File "/lib/python3.9/site-packages/cheroot/ssl/builtin.py", line 277, in wrap 2024-09-18T08:03:20.169 INFO:journalctl@ceph.mon.c.smithi007.stdout: s = self.context.wrap_socket( 2024-09-18T08:03:20.169 INFO:journalctl@ceph.mon.c.smithi007.stdout: File "/lib64/python3.9/ssl.py", line 501, in wrap_socket 2024-09-18T08:03:20.170 INFO:journalctl@ceph.mon.c.smithi007.stdout: return self.sslsocket_class._create( 2024-09-18T08:03:20.170 INFO:journalctl@ceph.mon.c.smithi007.stdout: File "/lib64/python3.9/ssl.py", line 1074, in _create 2024-09-18T08:03:20.170 INFO:journalctl@ceph.mon.c.smithi007.stdout: self.do_handshake() 2024-09-18T08:03:20.170 INFO:journalctl@ceph.mon.c.smithi007.stdout: File "/lib64/python3.9/ssl.py", line 1343, in do_handshake 2024-09-18T08:03:20.170 INFO:journalctl@ceph.mon.c.smithi007.stdout: self._sslobj.do_handshake() 2024-09-18T08:03:20.170 INFO:journalctl@ceph.mon.c.smithi007.stdout: ssl.SSLZeroReturnError: TLS/SSL connection has been closed (EOF) (_ssl.c:1133) 2024-09-18T08:03:20.170 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:03:20 smithi007 ceph-mon[39249]: mgrmap e21: y(active, since 2s), standbys: x 2024-09-18T08:03:20.406 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:03:20 smithi050 ceph-mon[32371]: [18/Sep/2024:08:03:18] ENGINE Bus STARTING 2024-09-18T08:03:20.406 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:03:20 smithi050 ceph-mon[32371]: [18/Sep/2024:08:03:18] ENGINE Serving on http://172.21.15.7:8765 2024-09-18T08:03:20.406 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:03:20 smithi050 ceph-mon[32371]: pgmap v5: 132 pgs: 132 active+clean; 583 KiB data, 222 MiB used, 715 GiB / 715 GiB avail 2024-09-18T08:03:20.406 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:03:20 smithi050 ceph-mon[32371]: [18/Sep/2024:08:03:19] ENGINE Serving on https://172.21.15.7:7150 2024-09-18T08:03:20.406 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:03:20 smithi050 ceph-mon[32371]: [18/Sep/2024:08:03:19] ENGINE Bus STARTED 2024-09-18T08:03:20.406 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:03:20 smithi050 ceph-mon[32371]: [18/Sep/2024:08:03:19] ENGINE Error in HTTPServer.serve 2024-09-18T08:03:20.406 INFO:journalctl@ceph.mon.b.smithi050.stdout: Traceback (most recent call last): 2024-09-18T08:03:20.407 INFO:journalctl@ceph.mon.b.smithi050.stdout: File "/lib/python3.9/site-packages/cheroot/server.py", line 1823, in serve 2024-09-18T08:03:20.407 INFO:journalctl@ceph.mon.b.smithi050.stdout: self._connections.run(self.expiration_interval) 2024-09-18T08:03:20.407 INFO:journalctl@ceph.mon.b.smithi050.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 203, in run 2024-09-18T08:03:20.407 INFO:journalctl@ceph.mon.b.smithi050.stdout: self._run(expiration_interval) 2024-09-18T08:03:20.407 INFO:journalctl@ceph.mon.b.smithi050.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 246, in _run 2024-09-18T08:03:20.407 INFO:journalctl@ceph.mon.b.smithi050.stdout: new_conn = self._from_server_socket(self.server.socket) 2024-09-18T08:03:20.407 INFO:journalctl@ceph.mon.b.smithi050.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 300, in _from_server_socket 2024-09-18T08:03:20.407 INFO:journalctl@ceph.mon.b.smithi050.stdout: s, ssl_env = self.server.ssl_adapter.wrap(s) 2024-09-18T08:03:20.407 INFO:journalctl@ceph.mon.b.smithi050.stdout: File "/lib/python3.9/site-packages/cheroot/ssl/builtin.py", line 277, in wrap 2024-09-18T08:03:20.407 INFO:journalctl@ceph.mon.b.smithi050.stdout: s = self.context.wrap_socket( 2024-09-18T08:03:20.407 INFO:journalctl@ceph.mon.b.smithi050.stdout: File "/lib64/python3.9/ssl.py", line 501, in wrap_socket 2024-09-18T08:03:20.407 INFO:journalctl@ceph.mon.b.smithi050.stdout: return self.sslsocket_class._create( 2024-09-18T08:03:20.407 INFO:journalctl@ceph.mon.b.smithi050.stdout: File "/lib64/python3.9/ssl.py", line 1074, in _create 2024-09-18T08:03:20.408 INFO:journalctl@ceph.mon.b.smithi050.stdout: self.do_handshake() 2024-09-18T08:03:20.408 INFO:journalctl@ceph.mon.b.smithi050.stdout: File "/lib64/python3.9/ssl.py", line 1343, in do_handshake 2024-09-18T08:03:20.408 INFO:journalctl@ceph.mon.b.smithi050.stdout: self._sslobj.do_handshake() 2024-09-18T08:03:20.408 INFO:journalctl@ceph.mon.b.smithi050.stdout: ssl.SSLZeroReturnError: TLS/SSL connection has been closed (EOF) (_ssl.c:1133) 2024-09-18T08:03:20.408 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:03:20 smithi050 ceph-mon[32371]: mgrmap e21: y(active, since 2s), standbys: x 2024-09-18T08:03:20.598 INFO:teuthology.orchestra.run.smithi007.stdout:201863462928 2024-09-18T08:03:20.665 INFO:teuthology.orchestra.run.smithi007.stdout:34359738419 2024-09-18T08:03:21.231 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:03:21 smithi007 ceph-mon[29459]: purged_snaps scrub starts 2024-09-18T08:03:21.231 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:03:21 smithi007 ceph-mon[29459]: purged_snaps scrub ok 2024-09-18T08:03:21.231 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:03:21 smithi007 ceph-mon[29459]: osdmap e75: 8 total, 8 up, 8 in 2024-09-18T08:03:21.231 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:03:21 smithi007 ceph-mon[29459]: from='mgr.24470 172.21.15.7:0/2594390261' entity='mgr.y' 2024-09-18T08:03:21.231 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:03:21 smithi007 ceph-mon[29459]: from='mgr.24470 172.21.15.7:0/2594390261' entity='mgr.y' 2024-09-18T08:03:21.231 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:03:21 smithi007 ceph-mon[29459]: from='client.? 172.21.15.7:0/193126846' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 6} : dispatch 2024-09-18T08:03:21.231 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:03:21 smithi007 ceph-mon[29459]: from='client.? 172.21.15.7:0/2196837420' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 0} : dispatch 2024-09-18T08:03:21.231 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:03:21 smithi007 ceph-mon[29459]: from='mgr.24470 172.21.15.7:0/2594390261' entity='mgr.y' 2024-09-18T08:03:21.231 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:03:21 smithi007 ceph-mon[29459]: from='mgr.24470 172.21.15.7:0/2594390261' entity='mgr.y' 2024-09-18T08:03:21.231 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:03:21 smithi007 ceph-mon[29459]: from='mgr.24470 172.21.15.7:0/2594390261' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"} : dispatch 2024-09-18T08:03:21.232 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:03:21 smithi007 ceph-mon[29459]: from='mgr.24470 172.21.15.7:0/2594390261' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"} : dispatch 2024-09-18T08:03:21.232 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:03:21 smithi007 ceph-mon[29459]: from='mgr.24470 172.21.15.7:0/2594390261' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"} : dispatch 2024-09-18T08:03:21.232 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:03:21 smithi007 ceph-mon[29459]: from='mgr.24470 172.21.15.7:0/2594390261' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"} : dispatch 2024-09-18T08:03:21.232 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:03:21 smithi007 ceph-mon[29459]: from='mgr.24470 172.21.15.7:0/2594390261' entity='mgr.y' 2024-09-18T08:03:21.232 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:03:21 smithi007 ceph-mon[39249]: purged_snaps scrub starts 2024-09-18T08:03:21.232 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:03:21 smithi007 ceph-mon[39249]: purged_snaps scrub ok 2024-09-18T08:03:21.232 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:03:21 smithi007 ceph-mon[39249]: osdmap e75: 8 total, 8 up, 8 in 2024-09-18T08:03:21.232 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:03:21 smithi007 ceph-mon[39249]: from='mgr.24470 172.21.15.7:0/2594390261' entity='mgr.y' 2024-09-18T08:03:21.232 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:03:21 smithi007 ceph-mon[39249]: from='mgr.24470 172.21.15.7:0/2594390261' entity='mgr.y' 2024-09-18T08:03:21.232 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:03:21 smithi007 ceph-mon[39249]: from='client.? 172.21.15.7:0/193126846' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 6} : dispatch 2024-09-18T08:03:21.233 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:03:21 smithi007 ceph-mon[39249]: from='client.? 172.21.15.7:0/2196837420' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 0} : dispatch 2024-09-18T08:03:21.233 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:03:21 smithi007 ceph-mon[39249]: from='mgr.24470 172.21.15.7:0/2594390261' entity='mgr.y' 2024-09-18T08:03:21.233 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:03:21 smithi007 ceph-mon[39249]: from='mgr.24470 172.21.15.7:0/2594390261' entity='mgr.y' 2024-09-18T08:03:21.233 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:03:21 smithi007 ceph-mon[39249]: from='mgr.24470 172.21.15.7:0/2594390261' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"} : dispatch 2024-09-18T08:03:21.233 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:03:21 smithi007 ceph-mon[39249]: from='mgr.24470 172.21.15.7:0/2594390261' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"} : dispatch 2024-09-18T08:03:21.233 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:03:21 smithi007 ceph-mon[39249]: from='mgr.24470 172.21.15.7:0/2594390261' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"} : dispatch 2024-09-18T08:03:21.233 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:03:21 smithi007 ceph-mon[39249]: from='mgr.24470 172.21.15.7:0/2594390261' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"} : dispatch 2024-09-18T08:03:21.233 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:03:21 smithi007 ceph-mon[39249]: from='mgr.24470 172.21.15.7:0/2594390261' entity='mgr.y' 2024-09-18T08:03:21.308 INFO:teuthology.orchestra.run.smithi007.stdout:77309411367 2024-09-18T08:03:21.308 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid 779349ca-7593-11ef-bceb-c7b262605968 -- ceph osd last-stat-seq osd.2 2024-09-18T08:03:21.406 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:03:21 smithi050 ceph-mon[32371]: purged_snaps scrub starts 2024-09-18T08:03:21.406 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:03:21 smithi050 ceph-mon[32371]: purged_snaps scrub ok 2024-09-18T08:03:21.406 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:03:21 smithi050 ceph-mon[32371]: osdmap e75: 8 total, 8 up, 8 in 2024-09-18T08:03:21.406 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:03:21 smithi050 ceph-mon[32371]: from='mgr.24470 172.21.15.7:0/2594390261' entity='mgr.y' 2024-09-18T08:03:21.406 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:03:21 smithi050 ceph-mon[32371]: from='mgr.24470 172.21.15.7:0/2594390261' entity='mgr.y' 2024-09-18T08:03:21.406 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:03:21 smithi050 ceph-mon[32371]: from='client.? 172.21.15.7:0/193126846' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 6} : dispatch 2024-09-18T08:03:21.407 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:03:21 smithi050 ceph-mon[32371]: from='client.? 172.21.15.7:0/2196837420' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 0} : dispatch 2024-09-18T08:03:21.407 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:03:21 smithi050 ceph-mon[32371]: from='mgr.24470 172.21.15.7:0/2594390261' entity='mgr.y' 2024-09-18T08:03:21.407 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:03:21 smithi050 ceph-mon[32371]: from='mgr.24470 172.21.15.7:0/2594390261' entity='mgr.y' 2024-09-18T08:03:21.407 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:03:21 smithi050 ceph-mon[32371]: from='mgr.24470 172.21.15.7:0/2594390261' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"} : dispatch 2024-09-18T08:03:21.407 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:03:21 smithi050 ceph-mon[32371]: from='mgr.24470 172.21.15.7:0/2594390261' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"} : dispatch 2024-09-18T08:03:21.407 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:03:21 smithi050 ceph-mon[32371]: from='mgr.24470 172.21.15.7:0/2594390261' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"} : dispatch 2024-09-18T08:03:21.407 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:03:21 smithi050 ceph-mon[32371]: from='mgr.24470 172.21.15.7:0/2594390261' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"} : dispatch 2024-09-18T08:03:21.407 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:03:21 smithi050 ceph-mon[32371]: from='mgr.24470 172.21.15.7:0/2594390261' entity='mgr.y' 2024-09-18T08:03:21.944 INFO:tasks.cephadm.ceph_manager.ceph:need seq 201863462927 got 201863462928 for osd.6 2024-09-18T08:03:21.944 DEBUG:teuthology.parallel:result is None 2024-09-18T08:03:22.230 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:03:22 smithi007 ceph-mon[29459]: Adjusting osd_memory_target on smithi050 to 3761M 2024-09-18T08:03:22.231 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:03:22 smithi007 ceph-mon[29459]: pgmap v7: 132 pgs: 132 active+clean; 583 KiB data, 223 MiB used, 715 GiB / 715 GiB avail; 59 KiB/s rd, 0 B/s wr, 24 op/s 2024-09-18T08:03:22.231 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:03:22 smithi007 ceph-mon[29459]: mgrmap e22: y(active, since 4s), standbys: x 2024-09-18T08:03:22.231 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:03:22 smithi007 ceph-mon[39249]: Adjusting osd_memory_target on smithi050 to 3761M 2024-09-18T08:03:22.231 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:03:22 smithi007 ceph-mon[39249]: pgmap v7: 132 pgs: 132 active+clean; 583 KiB data, 223 MiB used, 715 GiB / 715 GiB avail; 59 KiB/s rd, 0 B/s wr, 24 op/s 2024-09-18T08:03:22.231 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:03:22 smithi007 ceph-mon[39249]: mgrmap e22: y(active, since 4s), standbys: x 2024-09-18T08:03:22.351 INFO:teuthology.orchestra.run.smithi007.stdout:137438953499 2024-09-18T08:03:22.351 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid 779349ca-7593-11ef-bceb-c7b262605968 -- ceph osd last-stat-seq osd.4 2024-09-18T08:03:22.406 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:03:22 smithi050 ceph-mon[32371]: Adjusting osd_memory_target on smithi050 to 3761M 2024-09-18T08:03:22.406 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:03:22 smithi050 ceph-mon[32371]: pgmap v7: 132 pgs: 132 active+clean; 583 KiB data, 223 MiB used, 715 GiB / 715 GiB avail; 59 KiB/s rd, 0 B/s wr, 24 op/s 2024-09-18T08:03:22.406 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:03:22 smithi050 ceph-mon[32371]: mgrmap e22: y(active, since 4s), standbys: x 2024-09-18T08:03:22.732 INFO:tasks.cephadm.ceph_manager.ceph:need seq 34359738418 got 34359738419 for osd.0 2024-09-18T08:03:22.732 DEBUG:teuthology.parallel:result is None 2024-09-18T08:03:23.193 INFO:teuthology.orchestra.run.smithi007.stdout:240518168586 2024-09-18T08:03:23.193 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid 779349ca-7593-11ef-bceb-c7b262605968 -- ceph osd last-stat-seq osd.7 2024-09-18T08:03:23.374 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/779349ca-7593-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T08:03:23.375 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/779349ca-7593-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T08:03:23.625 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/779349ca-7593-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T08:03:23.845 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/779349ca-7593-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T08:03:24.198 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:03:24 smithi007 ceph-mon[29459]: pgmap v8: 132 pgs: 132 active+clean; 583 KiB data, 223 MiB used, 715 GiB / 715 GiB avail; 35 KiB/s rd, 0 B/s wr, 14 op/s 2024-09-18T08:03:24.198 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:03:24 smithi007 ceph-mon[29459]: osdmap e76: 8 total, 8 up, 8 in 2024-09-18T08:03:24.198 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:03:24 smithi007 ceph-mon[39249]: pgmap v8: 132 pgs: 132 active+clean; 583 KiB data, 223 MiB used, 715 GiB / 715 GiB avail; 35 KiB/s rd, 0 B/s wr, 14 op/s 2024-09-18T08:03:24.199 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:03:24 smithi007 ceph-mon[39249]: osdmap e76: 8 total, 8 up, 8 in 2024-09-18T08:03:24.406 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:03:24 smithi050 ceph-mon[32371]: pgmap v8: 132 pgs: 132 active+clean; 583 KiB data, 223 MiB used, 715 GiB / 715 GiB avail; 35 KiB/s rd, 0 B/s wr, 14 op/s 2024-09-18T08:03:24.406 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:03:24 smithi050 ceph-mon[32371]: osdmap e76: 8 total, 8 up, 8 in 2024-09-18T08:03:24.716 INFO:teuthology.orchestra.run.smithi007.stdout:137438953499 2024-09-18T08:03:25.101 INFO:teuthology.orchestra.run.smithi007.stdout:55834574893 2024-09-18T08:03:25.406 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:03:25 smithi050 ceph-mon[32371]: from='client.? 172.21.15.7:0/2086183803' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 4} : dispatch 2024-09-18T08:03:25.481 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:03:25 smithi007 ceph-mon[29459]: from='client.? 172.21.15.7:0/2086183803' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 4} : dispatch 2024-09-18T08:03:25.481 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:03:25 smithi007 ceph-mon[39249]: from='client.? 172.21.15.7:0/2086183803' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 4} : dispatch 2024-09-18T08:03:25.690 INFO:tasks.cephadm.ceph_manager.ceph:need seq 55834574891 got 55834574893 for osd.1 2024-09-18T08:03:25.690 DEBUG:teuthology.parallel:result is None 2024-09-18T08:03:26.082 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:03:26 smithi007 ceph-mon[29459]: pgmap v10: 132 pgs: 132 active+clean; 583 KiB data, 223 MiB used, 715 GiB / 715 GiB avail; 30 KiB/s rd, 0 B/s wr, 12 op/s 2024-09-18T08:03:26.082 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:03:26 smithi007 ceph-mon[29459]: from='client.? 172.21.15.7:0/224971014' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 1} : dispatch 2024-09-18T08:03:26.105 INFO:teuthology.orchestra.run.smithi007.stdout:167503724567 2024-09-18T08:03:26.357 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:03:26 smithi007 ceph-mon[39249]: pgmap v10: 132 pgs: 132 active+clean; 583 KiB data, 223 MiB used, 715 GiB / 715 GiB avail; 30 KiB/s rd, 0 B/s wr, 12 op/s 2024-09-18T08:03:26.358 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:03:26 smithi007 ceph-mon[39249]: from='client.? 172.21.15.7:0/224971014' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 1} : dispatch 2024-09-18T08:03:26.383 INFO:teuthology.orchestra.run.smithi007.stdout:111669149731 2024-09-18T08:03:26.383 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid 779349ca-7593-11ef-bceb-c7b262605968 -- ceph osd last-stat-seq osd.3 2024-09-18T08:03:26.406 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:03:26 smithi050 ceph-mon[32371]: pgmap v10: 132 pgs: 132 active+clean; 583 KiB data, 223 MiB used, 715 GiB / 715 GiB avail; 30 KiB/s rd, 0 B/s wr, 12 op/s 2024-09-18T08:03:26.406 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:03:26 smithi050 ceph-mon[32371]: from='client.? 172.21.15.7:0/224971014' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 1} : dispatch 2024-09-18T08:03:26.923 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Sep 18 08:03:26 smithi007 ceph-779349ca-7593-11ef-bceb-c7b262605968-mgr-y[29688]: ::ffff:172.21.15.50 - - [18/Sep/2024:08:03:26] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-09-18T08:03:26.953 INFO:tasks.cephadm.ceph_manager.ceph:need seq 137438953499 got 137438953499 for osd.4 2024-09-18T08:03:26.953 DEBUG:teuthology.parallel:result is None 2024-09-18T08:03:27.230 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:03:27 smithi007 ceph-mon[29459]: from='client.? 172.21.15.7:0/940679289' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 5} : dispatch 2024-09-18T08:03:27.231 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:03:27 smithi007 ceph-mon[39249]: from='client.? 172.21.15.7:0/940679289' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 5} : dispatch 2024-09-18T08:03:27.406 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:03:27 smithi050 ceph-mon[32371]: from='client.? 172.21.15.7:0/940679289' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 5} : dispatch 2024-09-18T08:03:27.603 INFO:tasks.cephadm.ceph_manager.ceph:need seq 167503724565 got 167503724567 for osd.5 2024-09-18T08:03:27.603 DEBUG:teuthology.parallel:result is None 2024-09-18T08:03:28.205 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:03:28 smithi007 ceph-mon[29459]: pgmap v11: 132 pgs: 132 active+clean; 583 KiB data, 223 MiB used, 715 GiB / 715 GiB avail; 23 KiB/s rd, 0 B/s wr, 9 op/s 2024-09-18T08:03:28.206 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:03:28 smithi007 ceph-mon[39249]: pgmap v11: 132 pgs: 132 active+clean; 583 KiB data, 223 MiB used, 715 GiB / 715 GiB avail; 23 KiB/s rd, 0 B/s wr, 9 op/s 2024-09-18T08:03:28.294 INFO:teuthology.orchestra.run.smithi007.stdout:77309411369 2024-09-18T08:03:28.406 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:03:28 smithi050 ceph-mon[32371]: pgmap v11: 132 pgs: 132 active+clean; 583 KiB data, 223 MiB used, 715 GiB / 715 GiB avail; 23 KiB/s rd, 0 B/s wr, 9 op/s 2024-09-18T08:03:28.663 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/779349ca-7593-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T08:03:29.093 INFO:journalctl@ceph.iscsi.iscsi.a.smithi050.stdout:Sep 18 08:03:28 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-iscsi-iscsi-a[57782]: debug there is no tcmu-runner data available 2024-09-18T08:03:29.115 INFO:tasks.cephadm.ceph_manager.ceph:need seq 77309411367 got 77309411369 for osd.2 2024-09-18T08:03:29.115 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/779349ca-7593-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T08:03:29.116 DEBUG:teuthology.parallel:result is None 2024-09-18T08:03:29.337 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:03:29 smithi007 ceph-mon[29459]: from='client.? 172.21.15.7:0/2091569884' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 2} : dispatch 2024-09-18T08:03:29.337 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:03:29 smithi007 ceph-mon[39249]: from='client.? 172.21.15.7:0/2091569884' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 2} : dispatch 2024-09-18T08:03:29.406 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:03:29 smithi050 ceph-mon[32371]: from='client.? 172.21.15.7:0/2091569884' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 2} : dispatch 2024-09-18T08:03:29.974 INFO:teuthology.orchestra.run.smithi007.stdout:240518168588 2024-09-18T08:03:30.157 INFO:teuthology.orchestra.run.smithi007.stdout:111669149732 2024-09-18T08:03:30.230 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:03:30 smithi007 ceph-mon[29459]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-18T08:03:30.231 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:03:30 smithi007 ceph-mon[29459]: purged_snaps scrub starts 2024-09-18T08:03:30.231 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:03:30 smithi007 ceph-mon[29459]: purged_snaps scrub ok 2024-09-18T08:03:30.231 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:03:30 smithi007 ceph-mon[29459]: pgmap v12: 132 pgs: 132 active+clean; 583 KiB data, 223 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-18T08:03:30.231 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:03:30 smithi007 ceph-mon[29459]: osdmap e77: 8 total, 8 up, 8 in 2024-09-18T08:03:30.231 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:03:30 smithi007 ceph-mon[29459]: from='client.? 172.21.15.7:0/2788790266' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 7} : dispatch 2024-09-18T08:03:30.231 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:03:30 smithi007 ceph-mon[39249]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-18T08:03:30.231 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:03:30 smithi007 ceph-mon[39249]: purged_snaps scrub starts 2024-09-18T08:03:30.231 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:03:30 smithi007 ceph-mon[39249]: purged_snaps scrub ok 2024-09-18T08:03:30.232 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:03:30 smithi007 ceph-mon[39249]: pgmap v12: 132 pgs: 132 active+clean; 583 KiB data, 223 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-18T08:03:30.232 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:03:30 smithi007 ceph-mon[39249]: osdmap e77: 8 total, 8 up, 8 in 2024-09-18T08:03:30.232 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:03:30 smithi007 ceph-mon[39249]: from='client.? 172.21.15.7:0/2788790266' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 7} : dispatch 2024-09-18T08:03:30.406 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:03:30 smithi050 ceph-mon[32371]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-18T08:03:30.406 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:03:30 smithi050 ceph-mon[32371]: purged_snaps scrub starts 2024-09-18T08:03:30.406 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:03:30 smithi050 ceph-mon[32371]: purged_snaps scrub ok 2024-09-18T08:03:30.406 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:03:30 smithi050 ceph-mon[32371]: pgmap v12: 132 pgs: 132 active+clean; 583 KiB data, 223 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-18T08:03:30.406 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:03:30 smithi050 ceph-mon[32371]: osdmap e77: 8 total, 8 up, 8 in 2024-09-18T08:03:30.406 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:03:30 smithi050 ceph-mon[32371]: from='client.? 172.21.15.7:0/2788790266' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 7} : dispatch 2024-09-18T08:03:30.781 INFO:tasks.cephadm.ceph_manager.ceph:need seq 111669149731 got 111669149732 for osd.3 2024-09-18T08:03:30.781 DEBUG:teuthology.parallel:result is None 2024-09-18T08:03:31.254 INFO:tasks.cephadm.ceph_manager.ceph:need seq 240518168586 got 240518168588 for osd.7 2024-09-18T08:03:31.254 DEBUG:teuthology.parallel:result is None 2024-09-18T08:03:31.255 INFO:tasks.cephadm.ceph_manager.ceph:waiting for clean 2024-09-18T08:03:31.255 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid 779349ca-7593-11ef-bceb-c7b262605968 -- ceph pg dump --format=json 2024-09-18T08:03:31.406 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:03:31 smithi050 ceph-mon[32371]: from='client.? 172.21.15.7:0/637742096' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 3} : dispatch 2024-09-18T08:03:31.481 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:03:31 smithi007 ceph-mon[29459]: from='client.? 172.21.15.7:0/637742096' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 3} : dispatch 2024-09-18T08:03:31.481 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:03:31 smithi007 ceph-mon[39249]: from='client.? 172.21.15.7:0/637742096' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 3} : dispatch 2024-09-18T08:03:31.538 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/779349ca-7593-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T08:03:32.230 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:03:32 smithi007 ceph-mon[29459]: pgmap v14: 132 pgs: 132 active+clean; 583 KiB data, 223 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-09-18T08:03:32.230 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:03:32 smithi007 ceph-mon[29459]: from='mgr.24470 172.21.15.7:0/2594390261' entity='mgr.y' 2024-09-18T08:03:32.231 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:03:32 smithi007 ceph-mon[29459]: from='mgr.24470 172.21.15.7:0/2594390261' entity='mgr.y' 2024-09-18T08:03:32.231 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:03:32 smithi007 ceph-mon[39249]: pgmap v14: 132 pgs: 132 active+clean; 583 KiB data, 223 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-09-18T08:03:32.231 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:03:32 smithi007 ceph-mon[39249]: from='mgr.24470 172.21.15.7:0/2594390261' entity='mgr.y' 2024-09-18T08:03:32.231 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:03:32 smithi007 ceph-mon[39249]: from='mgr.24470 172.21.15.7:0/2594390261' entity='mgr.y' 2024-09-18T08:03:32.237 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-09-18T08:03:32.239 INFO:teuthology.orchestra.run.smithi007.stderr:dumped all 2024-09-18T08:03:32.406 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:03:32 smithi050 ceph-mon[32371]: pgmap v14: 132 pgs: 132 active+clean; 583 KiB data, 223 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-09-18T08:03:32.406 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:03:32 smithi050 ceph-mon[32371]: from='mgr.24470 172.21.15.7:0/2594390261' entity='mgr.y' 2024-09-18T08:03:32.406 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:03:32 smithi050 ceph-mon[32371]: from='mgr.24470 172.21.15.7:0/2594390261' entity='mgr.y' 2024-09-18T08:03:33.113 INFO:teuthology.orchestra.run.smithi007.stdout:{"pg_ready":true,"pg_map":{"version":14,"stamp":"2024-09-18T08:03:31.069837+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":596507,"num_objects":199,"num_object_clones":0,"num_object_copies":597,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":199,"num_whiteouts":0,"num_read":883,"num_read_kb":740,"num_write":549,"num_write_kb":1673,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":590368,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":552,"ondisk_log_size":552,"up":396,"acting":396,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":396,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":228124,"kb_used_data":7740,"kb_used_omap":12,"kb_used_meta":220211,"kb_avail":749733092,"statfs":{"total":767960285184,"available":767726686208,"internally_reserved":0,"allocated":7925760,"data_stored":4201825,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12750,"internal_metadata":225496626},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":14,"apply_latency_ms":14,"commit_latency_ns":14000000,"apply_latency_ns":14000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":9,"num_read_kb":9,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.001730"},"pg_stats":[{"pgid":"3.1f","version":"0'0","reported_seq":40,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-18T08:03:24.069241+0000","last_change":"2024-09-18T08:02:47.878020+0000","last_active":"2024-09-18T08:03:24.069241+0000","last_peered":"2024-09-18T08:03:24.069241+0000","last_clean":"2024-09-18T08:03:24.069241+0000","last_became_active":"2024-09-18T08:02:47.877698+0000","last_became_peered":"2024-09-18T08:02:47.877698+0000","last_unstale":"2024-09-18T08:03:24.069241+0000","last_undegraded":"2024-09-18T08:03:24.069241+0000","last_fullsized":"2024-09-18T08:03:24.069241+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:02:46.860092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:46.860092+0000","last_clean_scrub_stamp":"2024-09-18T08:02:46.860092+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:04:14.205967+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"69'9","reported_seq":52,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-18T08:03:24.044361+0000","last_change":"2024-09-18T08:02:49.895688+0000","last_active":"2024-09-18T08:03:24.044361+0000","last_peered":"2024-09-18T08:03:24.044361+0000","last_clean":"2024-09-18T08:03:24.044361+0000","last_became_active":"2024-09-18T08:02:49.894557+0000","last_became_peered":"2024-09-18T08:02:49.894557+0000","last_unstale":"2024-09-18T08:03:24.044361+0000","last_undegraded":"2024-09-18T08:03:24.044361+0000","last_fullsized":"2024-09-18T08:03:24.044361+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_clean_scrub_stamp":"2024-09-18T08:02:48.870718+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:03:59.206088+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":32,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-18T08:03:24.043667+0000","last_change":"2024-09-18T08:02:51.907566+0000","last_active":"2024-09-18T08:03:24.043667+0000","last_peered":"2024-09-18T08:03:24.043667+0000","last_clean":"2024-09-18T08:03:24.043667+0000","last_became_active":"2024-09-18T08:02:51.907140+0000","last_became_peered":"2024-09-18T08:02:51.907140+0000","last_unstale":"2024-09-18T08:03:24.043667+0000","last_undegraded":"2024-09-18T08:03:24.043667+0000","last_fullsized":"2024-09-18T08:03:24.043667+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:02:50.886830+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:50.886830+0000","last_clean_scrub_stamp":"2024-09-18T08:02:50.886830+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:04:07.003335+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":29,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-18T08:03:24.044523+0000","last_change":"2024-09-18T08:02:53.906914+0000","last_active":"2024-09-18T08:03:24.044523+0000","last_peered":"2024-09-18T08:03:24.044523+0000","last_clean":"2024-09-18T08:03:24.044523+0000","last_became_active":"2024-09-18T08:02:53.906622+0000","last_became_peered":"2024-09-18T08:02:53.906622+0000","last_unstale":"2024-09-18T08:03:24.044523+0000","last_undegraded":"2024-09-18T08:03:24.044523+0000","last_fullsized":"2024-09-18T08:03:24.044523+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_clean_scrub_stamp":"2024-09-18T08:02:52.890475+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:04:20.738381+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":28,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-18T08:03:24.069177+0000","last_change":"2024-09-18T08:02:54.047999+0000","last_active":"2024-09-18T08:03:24.069177+0000","last_peered":"2024-09-18T08:03:24.069177+0000","last_clean":"2024-09-18T08:03:24.069177+0000","last_became_active":"2024-09-18T08:02:54.047640+0000","last_became_peered":"2024-09-18T08:02:54.047640+0000","last_unstale":"2024-09-18T08:03:24.069177+0000","last_undegraded":"2024-09-18T08:03:24.069177+0000","last_fullsized":"2024-09-18T08:03:24.069177+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_clean_scrub_stamp":"2024-09-18T08:02:52.890475+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:04:16.013706+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":40,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-18T08:03:24.069606+0000","last_change":"2024-09-18T08:02:47.872585+0000","last_active":"2024-09-18T08:03:24.069606+0000","last_peered":"2024-09-18T08:03:24.069606+0000","last_clean":"2024-09-18T08:03:24.069606+0000","last_became_active":"2024-09-18T08:02:47.872316+0000","last_became_peered":"2024-09-18T08:02:47.872316+0000","last_unstale":"2024-09-18T08:03:24.069606+0000","last_undegraded":"2024-09-18T08:03:24.069606+0000","last_fullsized":"2024-09-18T08:03:24.069606+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:02:46.860092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:46.860092+0000","last_clean_scrub_stamp":"2024-09-18T08:02:46.860092+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:03:53.731838+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"69'15","reported_seq":61,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-18T08:03:24.068893+0000","last_change":"2024-09-18T08:02:49.887111+0000","last_active":"2024-09-18T08:03:24.068893+0000","last_peered":"2024-09-18T08:03:24.068893+0000","last_clean":"2024-09-18T08:03:24.068893+0000","last_became_active":"2024-09-18T08:02:49.886106+0000","last_became_peered":"2024-09-18T08:02:49.886106+0000","last_unstale":"2024-09-18T08:03:24.068893+0000","last_undegraded":"2024-09-18T08:03:24.068893+0000","last_fullsized":"2024-09-18T08:03:24.068893+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_clean_scrub_stamp":"2024-09-18T08:02:48.870718+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:04:15.737211+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":32,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-18T08:03:24.044350+0000","last_change":"2024-09-18T08:02:51.904853+0000","last_active":"2024-09-18T08:03:24.044350+0000","last_peered":"2024-09-18T08:03:24.044350+0000","last_clean":"2024-09-18T08:03:24.044350+0000","last_became_active":"2024-09-18T08:02:51.904184+0000","last_became_peered":"2024-09-18T08:02:51.904184+0000","last_unstale":"2024-09-18T08:03:24.044350+0000","last_undegraded":"2024-09-18T08:03:24.044350+0000","last_fullsized":"2024-09-18T08:03:24.044350+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:02:50.886830+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:50.886830+0000","last_clean_scrub_stamp":"2024-09-18T08:02:50.886830+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:04:07.696637+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":40,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-18T08:03:24.069382+0000","last_change":"2024-09-18T08:02:47.875508+0000","last_active":"2024-09-18T08:03:24.069382+0000","last_peered":"2024-09-18T08:03:24.069382+0000","last_clean":"2024-09-18T08:03:24.069382+0000","last_became_active":"2024-09-18T08:02:47.875124+0000","last_became_peered":"2024-09-18T08:02:47.875124+0000","last_unstale":"2024-09-18T08:03:24.069382+0000","last_undegraded":"2024-09-18T08:03:24.069382+0000","last_fullsized":"2024-09-18T08:03:24.069382+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:02:46.860092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:46.860092+0000","last_clean_scrub_stamp":"2024-09-18T08:02:46.860092+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:03:53.477209+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"69'9","reported_seq":52,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-18T08:03:24.044733+0000","last_change":"2024-09-18T08:02:49.887589+0000","last_active":"2024-09-18T08:03:24.044733+0000","last_peered":"2024-09-18T08:03:24.044733+0000","last_clean":"2024-09-18T08:03:24.044733+0000","last_became_active":"2024-09-18T08:02:49.887161+0000","last_became_peered":"2024-09-18T08:02:49.887161+0000","last_unstale":"2024-09-18T08:03:24.044733+0000","last_undegraded":"2024-09-18T08:03:24.044733+0000","last_fullsized":"2024-09-18T08:03:24.044733+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_clean_scrub_stamp":"2024-09-18T08:02:48.870718+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:03:51.892069+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":32,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-18T08:03:24.067767+0000","last_change":"2024-09-18T08:02:51.907597+0000","last_active":"2024-09-18T08:03:24.067767+0000","last_peered":"2024-09-18T08:03:24.067767+0000","last_clean":"2024-09-18T08:03:24.067767+0000","last_became_active":"2024-09-18T08:02:51.907296+0000","last_became_peered":"2024-09-18T08:02:51.907296+0000","last_unstale":"2024-09-18T08:03:24.067767+0000","last_undegraded":"2024-09-18T08:03:24.067767+0000","last_fullsized":"2024-09-18T08:03:24.067767+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:02:50.886830+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:50.886830+0000","last_clean_scrub_stamp":"2024-09-18T08:02:50.886830+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:04:06.057644+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":29,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-18T08:03:24.069577+0000","last_change":"2024-09-18T08:02:53.913803+0000","last_active":"2024-09-18T08:03:24.069577+0000","last_peered":"2024-09-18T08:03:24.069577+0000","last_clean":"2024-09-18T08:03:24.069577+0000","last_became_active":"2024-09-18T08:02:53.913519+0000","last_became_peered":"2024-09-18T08:02:53.913519+0000","last_unstale":"2024-09-18T08:03:24.069577+0000","last_undegraded":"2024-09-18T08:03:24.069577+0000","last_fullsized":"2024-09-18T08:03:24.069577+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_clean_scrub_stamp":"2024-09-18T08:02:52.890475+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:04:18.699540+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":40,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-18T08:03:24.067844+0000","last_change":"2024-09-18T08:02:47.877577+0000","last_active":"2024-09-18T08:03:24.067844+0000","last_peered":"2024-09-18T08:03:24.067844+0000","last_clean":"2024-09-18T08:03:24.067844+0000","last_became_active":"2024-09-18T08:02:47.875042+0000","last_became_peered":"2024-09-18T08:02:47.875042+0000","last_unstale":"2024-09-18T08:03:24.067844+0000","last_undegraded":"2024-09-18T08:03:24.067844+0000","last_fullsized":"2024-09-18T08:03:24.067844+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:02:46.860092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:46.860092+0000","last_clean_scrub_stamp":"2024-09-18T08:02:46.860092+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:04:14.469212+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1b","version":"69'5","reported_seq":46,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-18T08:03:24.044735+0000","last_change":"2024-09-18T08:02:49.895779+0000","last_active":"2024-09-18T08:03:24.044735+0000","last_peered":"2024-09-18T08:03:24.044735+0000","last_clean":"2024-09-18T08:03:24.044735+0000","last_became_active":"2024-09-18T08:02:49.895344+0000","last_became_peered":"2024-09-18T08:02:49.895344+0000","last_unstale":"2024-09-18T08:03:24.044735+0000","last_undegraded":"2024-09-18T08:03:24.044735+0000","last_fullsized":"2024-09-18T08:03:24.044735+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_clean_scrub_stamp":"2024-09-18T08:02:48.870718+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:04:08.365283+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":11,"num_read_kb":7,"num_write":6,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":32,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-18T08:03:24.068696+0000","last_change":"2024-09-18T08:02:51.906694+0000","last_active":"2024-09-18T08:03:24.068696+0000","last_peered":"2024-09-18T08:03:24.068696+0000","last_clean":"2024-09-18T08:03:24.068696+0000","last_became_active":"2024-09-18T08:02:51.905776+0000","last_became_peered":"2024-09-18T08:02:51.905776+0000","last_unstale":"2024-09-18T08:03:24.068696+0000","last_undegraded":"2024-09-18T08:03:24.068696+0000","last_fullsized":"2024-09-18T08:03:24.068696+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:02:50.886830+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:50.886830+0000","last_clean_scrub_stamp":"2024-09-18T08:02:50.886830+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:04:07.273083+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":29,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-18T08:03:24.067192+0000","last_change":"2024-09-18T08:02:53.911969+0000","last_active":"2024-09-18T08:03:24.067192+0000","last_peered":"2024-09-18T08:03:24.067192+0000","last_clean":"2024-09-18T08:03:24.067192+0000","last_became_active":"2024-09-18T08:02:53.911648+0000","last_became_peered":"2024-09-18T08:02:53.911648+0000","last_unstale":"2024-09-18T08:03:24.067192+0000","last_undegraded":"2024-09-18T08:03:24.067192+0000","last_fullsized":"2024-09-18T08:03:24.067192+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_clean_scrub_stamp":"2024-09-18T08:02:52.890475+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:04:21.185524+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":28,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-18T08:03:24.045022+0000","last_change":"2024-09-18T08:02:54.047071+0000","last_active":"2024-09-18T08:03:24.045022+0000","last_peered":"2024-09-18T08:03:24.045022+0000","last_clean":"2024-09-18T08:03:24.045022+0000","last_became_active":"2024-09-18T08:02:54.046776+0000","last_became_peered":"2024-09-18T08:02:54.046776+0000","last_unstale":"2024-09-18T08:03:24.045022+0000","last_undegraded":"2024-09-18T08:03:24.045022+0000","last_fullsized":"2024-09-18T08:03:24.045022+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_clean_scrub_stamp":"2024-09-18T08:02:52.890475+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:04:11.355797+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":40,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-18T08:03:24.069035+0000","last_change":"2024-09-18T08:02:47.876235+0000","last_active":"2024-09-18T08:03:24.069035+0000","last_peered":"2024-09-18T08:03:24.069035+0000","last_clean":"2024-09-18T08:03:24.069035+0000","last_became_active":"2024-09-18T08:02:47.875963+0000","last_became_peered":"2024-09-18T08:02:47.875963+0000","last_unstale":"2024-09-18T08:03:24.069035+0000","last_undegraded":"2024-09-18T08:03:24.069035+0000","last_fullsized":"2024-09-18T08:03:24.069035+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:02:46.860092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:46.860092+0000","last_clean_scrub_stamp":"2024-09-18T08:02:46.860092+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:04:11.609411+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1c","version":"69'15","reported_seq":61,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-18T08:03:24.044723+0000","last_change":"2024-09-18T08:02:49.885923+0000","last_active":"2024-09-18T08:03:24.044723+0000","last_peered":"2024-09-18T08:03:24.044723+0000","last_clean":"2024-09-18T08:03:24.044723+0000","last_became_active":"2024-09-18T08:02:49.885553+0000","last_became_peered":"2024-09-18T08:02:49.885553+0000","last_unstale":"2024-09-18T08:03:24.044723+0000","last_undegraded":"2024-09-18T08:03:24.044723+0000","last_fullsized":"2024-09-18T08:03:24.044723+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_clean_scrub_stamp":"2024-09-18T08:02:48.870718+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:04:11.220046+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":32,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-18T08:03:24.044618+0000","last_change":"2024-09-18T08:02:51.908750+0000","last_active":"2024-09-18T08:03:24.044618+0000","last_peered":"2024-09-18T08:03:24.044618+0000","last_clean":"2024-09-18T08:03:24.044618+0000","last_became_active":"2024-09-18T08:02:51.908430+0000","last_became_peered":"2024-09-18T08:02:51.908430+0000","last_unstale":"2024-09-18T08:03:24.044618+0000","last_undegraded":"2024-09-18T08:03:24.044618+0000","last_fullsized":"2024-09-18T08:03:24.044618+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:02:50.886830+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:50.886830+0000","last_clean_scrub_stamp":"2024-09-18T08:02:50.886830+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:03:58.654545+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":28,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-18T08:03:24.069371+0000","last_change":"2024-09-18T08:02:54.047573+0000","last_active":"2024-09-18T08:03:24.069371+0000","last_peered":"2024-09-18T08:03:24.069371+0000","last_clean":"2024-09-18T08:03:24.069371+0000","last_became_active":"2024-09-18T08:02:54.047162+0000","last_became_peered":"2024-09-18T08:02:54.047162+0000","last_unstale":"2024-09-18T08:03:24.069371+0000","last_undegraded":"2024-09-18T08:03:24.069371+0000","last_fullsized":"2024-09-18T08:03:24.069371+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_clean_scrub_stamp":"2024-09-18T08:02:52.890475+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:04:02.287865+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":40,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-18T08:03:24.044998+0000","last_change":"2024-09-18T08:02:47.871895+0000","last_active":"2024-09-18T08:03:24.044998+0000","last_peered":"2024-09-18T08:03:24.044998+0000","last_clean":"2024-09-18T08:03:24.044998+0000","last_became_active":"2024-09-18T08:02:47.871523+0000","last_became_peered":"2024-09-18T08:02:47.871523+0000","last_unstale":"2024-09-18T08:03:24.044998+0000","last_undegraded":"2024-09-18T08:03:24.044998+0000","last_fullsized":"2024-09-18T08:03:24.044998+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:02:46.860092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:46.860092+0000","last_clean_scrub_stamp":"2024-09-18T08:02:46.860092+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:04:03.333307+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"69'12","reported_seq":59,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-18T08:03:24.069115+0000","last_change":"2024-09-18T08:02:49.889182+0000","last_active":"2024-09-18T08:03:24.069115+0000","last_peered":"2024-09-18T08:03:24.069115+0000","last_clean":"2024-09-18T08:03:24.069115+0000","last_became_active":"2024-09-18T08:02:49.888831+0000","last_became_peered":"2024-09-18T08:02:49.888831+0000","last_unstale":"2024-09-18T08:03:24.069115+0000","last_undegraded":"2024-09-18T08:03:24.069115+0000","last_fullsized":"2024-09-18T08:03:24.069115+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_clean_scrub_stamp":"2024-09-18T08:02:48.870718+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:04:12.415919+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":25,"num_read_kb":16,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":32,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-18T08:03:24.044662+0000","last_change":"2024-09-18T08:02:51.910642+0000","last_active":"2024-09-18T08:03:24.044662+0000","last_peered":"2024-09-18T08:03:24.044662+0000","last_clean":"2024-09-18T08:03:24.044662+0000","last_became_active":"2024-09-18T08:02:51.910316+0000","last_became_peered":"2024-09-18T08:02:51.910316+0000","last_unstale":"2024-09-18T08:03:24.044662+0000","last_undegraded":"2024-09-18T08:03:24.044662+0000","last_fullsized":"2024-09-18T08:03:24.044662+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:02:50.886830+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:50.886830+0000","last_clean_scrub_stamp":"2024-09-18T08:02:50.886830+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:04:05.261448+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1c","version":"69'1","reported_seq":30,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-18T08:03:24.069248+0000","last_change":"2024-09-18T08:02:53.910861+0000","last_active":"2024-09-18T08:03:24.069248+0000","last_peered":"2024-09-18T08:03:24.069248+0000","last_clean":"2024-09-18T08:03:24.069248+0000","last_became_active":"2024-09-18T08:02:53.910589+0000","last_became_peered":"2024-09-18T08:02:53.910589+0000","last_unstale":"2024-09-18T08:03:24.069248+0000","last_undegraded":"2024-09-18T08:03:24.069248+0000","last_fullsized":"2024-09-18T08:03:24.069248+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_clean_scrub_stamp":"2024-09-18T08:02:52.890475+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:04:20.224154+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":403,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.19","version":"62'1","reported_seq":41,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-18T08:03:24.043807+0000","last_change":"2024-09-18T08:02:47.881314+0000","last_active":"2024-09-18T08:03:24.043807+0000","last_peered":"2024-09-18T08:03:24.043807+0000","last_clean":"2024-09-18T08:03:24.043807+0000","last_became_active":"2024-09-18T08:02:47.880998+0000","last_became_peered":"2024-09-18T08:02:47.880998+0000","last_unstale":"2024-09-18T08:03:24.043807+0000","last_undegraded":"2024-09-18T08:03:24.043807+0000","last_fullsized":"2024-09-18T08:03:24.043807+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:02:46.860092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:46.860092+0000","last_clean_scrub_stamp":"2024-09-18T08:02:46.860092+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:03:57.000100+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1e","version":"69'10","reported_seq":51,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-18T08:03:24.070020+0000","last_change":"2024-09-18T08:02:49.892755+0000","last_active":"2024-09-18T08:03:24.070020+0000","last_peered":"2024-09-18T08:03:24.070020+0000","last_clean":"2024-09-18T08:03:24.070020+0000","last_became_active":"2024-09-18T08:02:49.892024+0000","last_became_peered":"2024-09-18T08:02:49.892024+0000","last_unstale":"2024-09-18T08:03:24.070020+0000","last_undegraded":"2024-09-18T08:03:24.070020+0000","last_fullsized":"2024-09-18T08:03:24.070020+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_clean_scrub_stamp":"2024-09-18T08:02:48.870718+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:04:17.237822+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1f","version":"69'8","reported_seq":52,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-18T08:03:24.068593+0000","last_change":"2024-09-18T08:02:51.904933+0000","last_active":"2024-09-18T08:03:24.068593+0000","last_peered":"2024-09-18T08:03:24.068593+0000","last_clean":"2024-09-18T08:03:24.068593+0000","last_became_active":"2024-09-18T08:02:51.904588+0000","last_became_peered":"2024-09-18T08:02:51.904588+0000","last_unstale":"2024-09-18T08:03:24.068593+0000","last_undegraded":"2024-09-18T08:03:24.068593+0000","last_fullsized":"2024-09-18T08:03:24.068593+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:02:50.886830+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:50.886830+0000","last_clean_scrub_stamp":"2024-09-18T08:02:50.886830+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:03:59.243324+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.f","version":"69'15","reported_seq":61,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-18T08:03:24.045157+0000","last_change":"2024-09-18T08:02:49.899404+0000","last_active":"2024-09-18T08:03:24.045157+0000","last_peered":"2024-09-18T08:03:24.045157+0000","last_clean":"2024-09-18T08:03:24.045157+0000","last_became_active":"2024-09-18T08:02:49.899083+0000","last_became_peered":"2024-09-18T08:02:49.899083+0000","last_unstale":"2024-09-18T08:03:24.045157+0000","last_undegraded":"2024-09-18T08:03:24.045157+0000","last_fullsized":"2024-09-18T08:03:24.045157+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_clean_scrub_stamp":"2024-09-18T08:02:48.870718+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:04:06.513250+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":40,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-18T08:03:24.068402+0000","last_change":"2024-09-18T08:02:47.879723+0000","last_active":"2024-09-18T08:03:24.068402+0000","last_peered":"2024-09-18T08:03:24.068402+0000","last_clean":"2024-09-18T08:03:24.068402+0000","last_became_active":"2024-09-18T08:02:47.879328+0000","last_became_peered":"2024-09-18T08:02:47.879328+0000","last_unstale":"2024-09-18T08:03:24.068402+0000","last_undegraded":"2024-09-18T08:03:24.068402+0000","last_fullsized":"2024-09-18T08:03:24.068402+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:02:46.860092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:46.860092+0000","last_clean_scrub_stamp":"2024-09-18T08:02:46.860092+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:03:56.802301+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.e","version":"69'8","reported_seq":49,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-18T08:03:24.045230+0000","last_change":"2024-09-18T08:02:51.903673+0000","last_active":"2024-09-18T08:03:24.045230+0000","last_peered":"2024-09-18T08:03:24.045230+0000","last_clean":"2024-09-18T08:03:24.045230+0000","last_became_active":"2024-09-18T08:02:51.903369+0000","last_became_peered":"2024-09-18T08:02:51.903369+0000","last_unstale":"2024-09-18T08:03:24.045230+0000","last_undegraded":"2024-09-18T08:03:24.045230+0000","last_fullsized":"2024-09-18T08:03:24.045230+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:02:50.886830+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:50.886830+0000","last_clean_scrub_stamp":"2024-09-18T08:02:50.886830+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:03:51.149147+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":29,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-18T08:03:24.068993+0000","last_change":"2024-09-18T08:02:53.910161+0000","last_active":"2024-09-18T08:03:24.068993+0000","last_peered":"2024-09-18T08:03:24.068993+0000","last_clean":"2024-09-18T08:03:24.068993+0000","last_became_active":"2024-09-18T08:02:53.909834+0000","last_became_peered":"2024-09-18T08:02:53.909834+0000","last_unstale":"2024-09-18T08:03:24.068993+0000","last_undegraded":"2024-09-18T08:03:24.068993+0000","last_fullsized":"2024-09-18T08:03:24.068993+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_clean_scrub_stamp":"2024-09-18T08:02:52.890475+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:04:07.435248+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.0","version":"69'18","reported_seq":68,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-18T08:03:24.068289+0000","last_change":"2024-09-18T08:02:49.887135+0000","last_active":"2024-09-18T08:03:24.068289+0000","last_peered":"2024-09-18T08:03:24.068289+0000","last_clean":"2024-09-18T08:03:24.068289+0000","last_became_active":"2024-09-18T08:02:49.886818+0000","last_became_peered":"2024-09-18T08:02:49.886818+0000","last_unstale":"2024-09-18T08:03:24.068289+0000","last_undegraded":"2024-09-18T08:03:24.068289+0000","last_fullsized":"2024-09-18T08:03:24.068289+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_clean_scrub_stamp":"2024-09-18T08:02:48.870718+0000","objects_scrubbed":0,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:03:57.656879+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":34,"num_read_kb":22,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":40,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-18T08:03:24.069891+0000","last_change":"2024-09-18T08:02:47.880517+0000","last_active":"2024-09-18T08:03:24.069891+0000","last_peered":"2024-09-18T08:03:24.069891+0000","last_clean":"2024-09-18T08:03:24.069891+0000","last_became_active":"2024-09-18T08:02:47.880225+0000","last_became_peered":"2024-09-18T08:02:47.880225+0000","last_unstale":"2024-09-18T08:03:24.069891+0000","last_undegraded":"2024-09-18T08:03:24.069891+0000","last_fullsized":"2024-09-18T08:03:24.069891+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:02:46.860092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:46.860092+0000","last_clean_scrub_stamp":"2024-09-18T08:02:46.860092+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:03:51.223493+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":32,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-18T08:03:24.044360+0000","last_change":"2024-09-18T08:02:51.899911+0000","last_active":"2024-09-18T08:03:24.044360+0000","last_peered":"2024-09-18T08:03:24.044360+0000","last_clean":"2024-09-18T08:03:24.044360+0000","last_became_active":"2024-09-18T08:02:51.899594+0000","last_became_peered":"2024-09-18T08:02:51.899594+0000","last_unstale":"2024-09-18T08:03:24.044360+0000","last_undegraded":"2024-09-18T08:03:24.044360+0000","last_fullsized":"2024-09-18T08:03:24.044360+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:02:50.886830+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:50.886830+0000","last_clean_scrub_stamp":"2024-09-18T08:02:50.886830+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:04:02.261346+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":29,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-18T08:03:24.044645+0000","last_change":"2024-09-18T08:02:53.904262+0000","last_active":"2024-09-18T08:03:24.044645+0000","last_peered":"2024-09-18T08:03:24.044645+0000","last_clean":"2024-09-18T08:03:24.044645+0000","last_became_active":"2024-09-18T08:02:53.903957+0000","last_became_peered":"2024-09-18T08:02:53.903957+0000","last_unstale":"2024-09-18T08:03:24.044645+0000","last_undegraded":"2024-09-18T08:03:24.044645+0000","last_fullsized":"2024-09-18T08:03:24.044645+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_clean_scrub_stamp":"2024-09-18T08:02:52.890475+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:04:02.672046+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1","version":"69'14","reported_seq":57,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-18T08:03:24.044416+0000","last_change":"2024-09-18T08:02:49.894205+0000","last_active":"2024-09-18T08:03:24.044416+0000","last_peered":"2024-09-18T08:03:24.044416+0000","last_clean":"2024-09-18T08:03:24.044416+0000","last_became_active":"2024-09-18T08:02:49.893830+0000","last_became_peered":"2024-09-18T08:02:49.893830+0000","last_unstale":"2024-09-18T08:03:24.044416+0000","last_undegraded":"2024-09-18T08:03:24.044416+0000","last_fullsized":"2024-09-18T08:03:24.044416+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_clean_scrub_stamp":"2024-09-18T08:02:48.870718+0000","objects_scrubbed":0,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:03:51.325184+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":21,"num_read_kb":14,"num_write":14,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"62'2","reported_seq":47,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-18T08:03:24.070163+0000","last_change":"2024-09-18T08:02:47.873172+0000","last_active":"2024-09-18T08:03:24.070163+0000","last_peered":"2024-09-18T08:03:24.070163+0000","last_clean":"2024-09-18T08:03:24.070163+0000","last_became_active":"2024-09-18T08:02:47.872857+0000","last_became_peered":"2024-09-18T08:02:47.872857+0000","last_unstale":"2024-09-18T08:03:24.070163+0000","last_undegraded":"2024-09-18T08:03:24.070163+0000","last_fullsized":"2024-09-18T08:03:24.070163+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:02:46.860092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:46.860092+0000","last_clean_scrub_stamp":"2024-09-18T08:02:46.860092+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:04:03.733922+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1039,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.0","version":"69'8","reported_seq":49,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-18T08:03:24.068202+0000","last_change":"2024-09-18T08:02:51.910339+0000","last_active":"2024-09-18T08:03:24.068202+0000","last_peered":"2024-09-18T08:03:24.068202+0000","last_clean":"2024-09-18T08:03:24.068202+0000","last_became_active":"2024-09-18T08:02:51.909680+0000","last_became_peered":"2024-09-18T08:02:51.909680+0000","last_unstale":"2024-09-18T08:03:24.068202+0000","last_undegraded":"2024-09-18T08:03:24.068202+0000","last_fullsized":"2024-09-18T08:03:24.068202+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:02:50.886830+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:50.886830+0000","last_clean_scrub_stamp":"2024-09-18T08:02:50.886830+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:03:58.456713+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":29,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-18T08:03:24.067615+0000","last_change":"2024-09-18T08:02:54.047452+0000","last_active":"2024-09-18T08:03:24.067615+0000","last_peered":"2024-09-18T08:03:24.067615+0000","last_clean":"2024-09-18T08:03:24.067615+0000","last_became_active":"2024-09-18T08:02:54.047039+0000","last_became_peered":"2024-09-18T08:02:54.047039+0000","last_unstale":"2024-09-18T08:03:24.067615+0000","last_undegraded":"2024-09-18T08:03:24.067615+0000","last_fullsized":"2024-09-18T08:03:24.067615+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_clean_scrub_stamp":"2024-09-18T08:02:52.890475+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:03:58.508399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.2","version":"69'10","reported_seq":51,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-18T08:03:24.043910+0000","last_change":"2024-09-18T08:02:49.896889+0000","last_active":"2024-09-18T08:03:24.043910+0000","last_peered":"2024-09-18T08:03:24.043910+0000","last_clean":"2024-09-18T08:03:24.043910+0000","last_became_active":"2024-09-18T08:02:49.895265+0000","last_became_peered":"2024-09-18T08:02:49.895265+0000","last_unstale":"2024-09-18T08:03:24.043910+0000","last_undegraded":"2024-09-18T08:03:24.043910+0000","last_fullsized":"2024-09-18T08:03:24.043910+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_clean_scrub_stamp":"2024-09-18T08:02:48.870718+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:04:06.130656+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":40,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-18T08:03:24.069483+0000","last_change":"2024-09-18T08:02:47.877672+0000","last_active":"2024-09-18T08:03:24.069483+0000","last_peered":"2024-09-18T08:03:24.069483+0000","last_clean":"2024-09-18T08:03:24.069483+0000","last_became_active":"2024-09-18T08:02:47.874727+0000","last_became_peered":"2024-09-18T08:02:47.874727+0000","last_unstale":"2024-09-18T08:03:24.069483+0000","last_undegraded":"2024-09-18T08:03:24.069483+0000","last_fullsized":"2024-09-18T08:03:24.069483+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:02:46.860092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:46.860092+0000","last_clean_scrub_stamp":"2024-09-18T08:02:46.860092+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:03:47.082997+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.3","version":"69'8","reported_seq":49,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-18T08:03:24.068623+0000","last_change":"2024-09-18T08:02:51.906369+0000","last_active":"2024-09-18T08:03:24.068623+0000","last_peered":"2024-09-18T08:03:24.068623+0000","last_clean":"2024-09-18T08:03:24.068623+0000","last_became_active":"2024-09-18T08:02:51.906055+0000","last_became_peered":"2024-09-18T08:02:51.906055+0000","last_unstale":"2024-09-18T08:03:24.068623+0000","last_undegraded":"2024-09-18T08:03:24.068623+0000","last_fullsized":"2024-09-18T08:03:24.068623+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:02:50.886830+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:50.886830+0000","last_clean_scrub_stamp":"2024-09-18T08:02:50.886830+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:04:18.452195+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":29,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-18T08:03:24.069641+0000","last_change":"2024-09-18T08:02:53.910057+0000","last_active":"2024-09-18T08:03:24.069641+0000","last_peered":"2024-09-18T08:03:24.069641+0000","last_clean":"2024-09-18T08:03:24.069641+0000","last_became_active":"2024-09-18T08:02:53.909583+0000","last_became_peered":"2024-09-18T08:02:53.909583+0000","last_unstale":"2024-09-18T08:03:24.069641+0000","last_undegraded":"2024-09-18T08:03:24.069641+0000","last_fullsized":"2024-09-18T08:03:24.069641+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_clean_scrub_stamp":"2024-09-18T08:02:52.890475+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:04:07.545769+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.3","version":"69'19","reported_seq":72,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-18T08:03:24.068768+0000","last_change":"2024-09-18T08:02:49.890631+0000","last_active":"2024-09-18T08:03:24.068768+0000","last_peered":"2024-09-18T08:03:24.068768+0000","last_clean":"2024-09-18T08:03:24.068768+0000","last_became_active":"2024-09-18T08:02:49.890318+0000","last_became_peered":"2024-09-18T08:02:49.890318+0000","last_unstale":"2024-09-18T08:03:24.068768+0000","last_undegraded":"2024-09-18T08:03:24.068768+0000","last_fullsized":"2024-09-18T08:03:24.068768+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_clean_scrub_stamp":"2024-09-18T08:02:48.870718+0000","objects_scrubbed":0,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:03:57.953144+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":39,"num_read_kb":25,"num_write":22,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.4","version":"62'1","reported_seq":46,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-18T08:03:24.044885+0000","last_change":"2024-09-18T08:02:47.875046+0000","last_active":"2024-09-18T08:03:24.044885+0000","last_peered":"2024-09-18T08:03:24.044885+0000","last_clean":"2024-09-18T08:03:24.044885+0000","last_became_active":"2024-09-18T08:02:47.874764+0000","last_became_peered":"2024-09-18T08:02:47.874764+0000","last_unstale":"2024-09-18T08:03:24.044885+0000","last_undegraded":"2024-09-18T08:03:24.044885+0000","last_fullsized":"2024-09-18T08:03:24.044885+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:02:46.860092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:46.860092+0000","last_clean_scrub_stamp":"2024-09-18T08:02:46.860092+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:04:07.354571+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":436,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":32,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-18T08:03:24.067980+0000","last_change":"2024-09-18T08:02:51.908344+0000","last_active":"2024-09-18T08:03:24.067980+0000","last_peered":"2024-09-18T08:03:24.067980+0000","last_clean":"2024-09-18T08:03:24.067980+0000","last_became_active":"2024-09-18T08:02:51.908068+0000","last_became_peered":"2024-09-18T08:02:51.908068+0000","last_unstale":"2024-09-18T08:03:24.067980+0000","last_undegraded":"2024-09-18T08:03:24.067980+0000","last_fullsized":"2024-09-18T08:03:24.067980+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:02:50.886830+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:50.886830+0000","last_clean_scrub_stamp":"2024-09-18T08:02:50.886830+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:04:06.242055+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":28,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-18T08:03:24.043566+0000","last_change":"2024-09-18T08:02:54.046350+0000","last_active":"2024-09-18T08:03:24.043566+0000","last_peered":"2024-09-18T08:03:24.043566+0000","last_clean":"2024-09-18T08:03:24.043566+0000","last_became_active":"2024-09-18T08:02:54.045726+0000","last_became_peered":"2024-09-18T08:02:54.045726+0000","last_unstale":"2024-09-18T08:03:24.043566+0000","last_undegraded":"2024-09-18T08:03:24.043566+0000","last_fullsized":"2024-09-18T08:03:24.043566+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_clean_scrub_stamp":"2024-09-18T08:02:52.890475+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:03:55.585366+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.4","version":"69'28","reported_seq":93,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-18T08:03:24.044707+0000","last_change":"2024-09-18T08:02:49.897515+0000","last_active":"2024-09-18T08:03:24.044707+0000","last_peered":"2024-09-18T08:03:24.044707+0000","last_clean":"2024-09-18T08:03:24.044707+0000","last_became_active":"2024-09-18T08:02:49.897221+0000","last_became_peered":"2024-09-18T08:02:49.897221+0000","last_unstale":"2024-09-18T08:03:24.044707+0000","last_undegraded":"2024-09-18T08:03:24.044707+0000","last_fullsized":"2024-09-18T08:03:24.044707+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_clean_scrub_stamp":"2024-09-18T08:02:48.870718+0000","objects_scrubbed":0,"log_size":28,"log_dups_size":0,"ondisk_log_size":28,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:04:04.818428+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":48,"num_read_kb":33,"num_write":26,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":40,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-18T08:03:24.045161+0000","last_change":"2024-09-18T08:02:47.877445+0000","last_active":"2024-09-18T08:03:24.045161+0000","last_peered":"2024-09-18T08:03:24.045161+0000","last_clean":"2024-09-18T08:03:24.045161+0000","last_became_active":"2024-09-18T08:02:47.877085+0000","last_became_peered":"2024-09-18T08:02:47.877085+0000","last_unstale":"2024-09-18T08:03:24.045161+0000","last_undegraded":"2024-09-18T08:03:24.045161+0000","last_fullsized":"2024-09-18T08:03:24.045161+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:02:46.860092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:46.860092+0000","last_clean_scrub_stamp":"2024-09-18T08:02:46.860092+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:04:09.365906+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"64'2","reported_seq":50,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-18T08:03:24.067521+0000","last_change":"2024-09-18T08:02:49.883480+0000","last_active":"2024-09-18T08:03:24.067521+0000","last_peered":"2024-09-18T08:03:24.067521+0000","last_clean":"2024-09-18T08:03:24.067521+0000","last_became_active":"2024-09-18T08:02:46.867802+0000","last_became_peered":"2024-09-18T08:02:46.867802+0000","last_unstale":"2024-09-18T08:03:24.067521+0000","last_undegraded":"2024-09-18T08:03:24.067521+0000","last_fullsized":"2024-09-18T08:03:24.067521+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:02:45.855979+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:45.855979+0000","last_clean_scrub_stamp":"2024-09-18T08:02:45.855979+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:04:13.406442+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001348349,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":32,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-18T08:03:24.068924+0000","last_change":"2024-09-18T08:02:51.905694+0000","last_active":"2024-09-18T08:03:24.068924+0000","last_peered":"2024-09-18T08:03:24.068924+0000","last_clean":"2024-09-18T08:03:24.068924+0000","last_became_active":"2024-09-18T08:02:51.905357+0000","last_became_peered":"2024-09-18T08:02:51.905357+0000","last_unstale":"2024-09-18T08:03:24.068924+0000","last_undegraded":"2024-09-18T08:03:24.068924+0000","last_fullsized":"2024-09-18T08:03:24.068924+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:02:50.886830+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:50.886830+0000","last_clean_scrub_stamp":"2024-09-18T08:02:50.886830+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:04:15.739565+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":29,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-18T08:03:24.069370+0000","last_change":"2024-09-18T08:02:53.911634+0000","last_active":"2024-09-18T08:03:24.069370+0000","last_peered":"2024-09-18T08:03:24.069370+0000","last_clean":"2024-09-18T08:03:24.069370+0000","last_became_active":"2024-09-18T08:02:53.911036+0000","last_became_peered":"2024-09-18T08:02:53.911036+0000","last_unstale":"2024-09-18T08:03:24.069370+0000","last_undegraded":"2024-09-18T08:03:24.069370+0000","last_fullsized":"2024-09-18T08:03:24.069370+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_clean_scrub_stamp":"2024-09-18T08:02:52.890475+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:04:16.930353+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.7","version":"69'13","reported_seq":63,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-18T08:03:24.045212+0000","last_change":"2024-09-18T08:02:49.899404+0000","last_active":"2024-09-18T08:03:24.045212+0000","last_peered":"2024-09-18T08:03:24.045212+0000","last_clean":"2024-09-18T08:03:24.045212+0000","last_became_active":"2024-09-18T08:02:49.898987+0000","last_became_peered":"2024-09-18T08:02:49.898987+0000","last_unstale":"2024-09-18T08:03:24.045212+0000","last_undegraded":"2024-09-18T08:03:24.045212+0000","last_fullsized":"2024-09-18T08:03:24.045212+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_clean_scrub_stamp":"2024-09-18T08:02:48.870718+0000","objects_scrubbed":0,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:03:50.031411+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":19,"num_write":16,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":40,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-18T08:03:24.043728+0000","last_change":"2024-09-18T08:02:47.871732+0000","last_active":"2024-09-18T08:03:24.043728+0000","last_peered":"2024-09-18T08:03:24.043728+0000","last_clean":"2024-09-18T08:03:24.043728+0000","last_became_active":"2024-09-18T08:02:47.871416+0000","last_became_peered":"2024-09-18T08:02:47.871416+0000","last_unstale":"2024-09-18T08:03:24.043728+0000","last_undegraded":"2024-09-18T08:03:24.043728+0000","last_fullsized":"2024-09-18T08:03:24.043728+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:02:46.860092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:46.860092+0000","last_clean_scrub_stamp":"2024-09-18T08:02:46.860092+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:03:48.825242+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"62'1","reported_seq":49,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-18T08:03:24.044803+0000","last_change":"2024-09-18T08:02:49.877280+0000","last_active":"2024-09-18T08:03:24.044803+0000","last_peered":"2024-09-18T08:03:24.044803+0000","last_clean":"2024-09-18T08:03:24.044803+0000","last_became_active":"2024-09-18T08:02:46.868175+0000","last_became_peered":"2024-09-18T08:02:46.868175+0000","last_unstale":"2024-09-18T08:03:24.044803+0000","last_undegraded":"2024-09-18T08:03:24.044803+0000","last_fullsized":"2024-09-18T08:03:24.044803+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:02:45.855979+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:45.855979+0000","last_clean_scrub_stamp":"2024-09-18T08:02:45.855979+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:04:02.537405+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00066785499999999997,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":32,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-18T08:03:24.044662+0000","last_change":"2024-09-18T08:02:51.908417+0000","last_active":"2024-09-18T08:03:24.044662+0000","last_peered":"2024-09-18T08:03:24.044662+0000","last_clean":"2024-09-18T08:03:24.044662+0000","last_became_active":"2024-09-18T08:02:51.908156+0000","last_became_peered":"2024-09-18T08:02:51.908156+0000","last_unstale":"2024-09-18T08:03:24.044662+0000","last_undegraded":"2024-09-18T08:03:24.044662+0000","last_fullsized":"2024-09-18T08:03:24.044662+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:02:50.886830+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:50.886830+0000","last_clean_scrub_stamp":"2024-09-18T08:02:50.886830+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:04:04.453603+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":29,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-18T08:03:24.068185+0000","last_change":"2024-09-18T08:02:54.048417+0000","last_active":"2024-09-18T08:03:24.068185+0000","last_peered":"2024-09-18T08:03:24.068185+0000","last_clean":"2024-09-18T08:03:24.068185+0000","last_became_active":"2024-09-18T08:02:54.048004+0000","last_became_peered":"2024-09-18T08:02:54.048004+0000","last_unstale":"2024-09-18T08:03:24.068185+0000","last_undegraded":"2024-09-18T08:03:24.068185+0000","last_fullsized":"2024-09-18T08:03:24.068185+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_clean_scrub_stamp":"2024-09-18T08:02:52.890475+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:04:20.672566+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"69'12","reported_seq":54,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-18T08:03:24.070052+0000","last_change":"2024-09-18T08:02:49.898383+0000","last_active":"2024-09-18T08:03:24.070052+0000","last_peered":"2024-09-18T08:03:24.070052+0000","last_clean":"2024-09-18T08:03:24.070052+0000","last_became_active":"2024-09-18T08:02:49.898079+0000","last_became_peered":"2024-09-18T08:02:49.898079+0000","last_unstale":"2024-09-18T08:03:24.070052+0000","last_undegraded":"2024-09-18T08:03:24.070052+0000","last_fullsized":"2024-09-18T08:03:24.070052+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_clean_scrub_stamp":"2024-09-18T08:02:48.870718+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:04:14.339135+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":18,"num_read_kb":12,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":40,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-18T08:03:24.068516+0000","last_change":"2024-09-18T08:02:47.874335+0000","last_active":"2024-09-18T08:03:24.068516+0000","last_peered":"2024-09-18T08:03:24.068516+0000","last_clean":"2024-09-18T08:03:24.068516+0000","last_became_active":"2024-09-18T08:02:47.873809+0000","last_became_peered":"2024-09-18T08:02:47.873809+0000","last_unstale":"2024-09-18T08:03:24.068516+0000","last_undegraded":"2024-09-18T08:03:24.068516+0000","last_fullsized":"2024-09-18T08:03:24.068516+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:02:46.860092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:46.860092+0000","last_clean_scrub_stamp":"2024-09-18T08:02:46.860092+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:03:48.336434+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"70'5","reported_seq":86,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-18T08:03:26.696502+0000","last_change":"2024-09-18T08:02:49.877461+0000","last_active":"2024-09-18T08:03:26.696502+0000","last_peered":"2024-09-18T08:03:26.696502+0000","last_clean":"2024-09-18T08:03:26.696502+0000","last_became_active":"2024-09-18T08:02:46.872144+0000","last_became_peered":"2024-09-18T08:02:46.872144+0000","last_unstale":"2024-09-18T08:03:26.696502+0000","last_undegraded":"2024-09-18T08:03:26.696502+0000","last_fullsized":"2024-09-18T08:03:26.696502+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:02:45.855979+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:45.855979+0000","last_clean_scrub_stamp":"2024-09-18T08:02:45.855979+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:03:59.991143+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000504055,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":35,"num_read_kb":30,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":32,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-18T08:03:24.068735+0000","last_change":"2024-09-18T08:02:51.913423+0000","last_active":"2024-09-18T08:03:24.068735+0000","last_peered":"2024-09-18T08:03:24.068735+0000","last_clean":"2024-09-18T08:03:24.068735+0000","last_became_active":"2024-09-18T08:02:51.913112+0000","last_became_peered":"2024-09-18T08:02:51.913112+0000","last_unstale":"2024-09-18T08:03:24.068735+0000","last_undegraded":"2024-09-18T08:03:24.068735+0000","last_fullsized":"2024-09-18T08:03:24.068735+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:02:50.886830+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:50.886830+0000","last_clean_scrub_stamp":"2024-09-18T08:02:50.886830+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:04:04.493964+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":29,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-18T08:03:24.044499+0000","last_change":"2024-09-18T08:02:53.907689+0000","last_active":"2024-09-18T08:03:24.044499+0000","last_peered":"2024-09-18T08:03:24.044499+0000","last_clean":"2024-09-18T08:03:24.044499+0000","last_became_active":"2024-09-18T08:02:53.907404+0000","last_became_peered":"2024-09-18T08:02:53.907404+0000","last_unstale":"2024-09-18T08:03:24.044499+0000","last_undegraded":"2024-09-18T08:03:24.044499+0000","last_fullsized":"2024-09-18T08:03:24.044499+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_clean_scrub_stamp":"2024-09-18T08:02:52.890475+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:03:57.936771+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.5","version":"69'16","reported_seq":67,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-18T08:03:24.067407+0000","last_change":"2024-09-18T08:02:49.892366+0000","last_active":"2024-09-18T08:03:24.067407+0000","last_peered":"2024-09-18T08:03:24.067407+0000","last_clean":"2024-09-18T08:03:24.067407+0000","last_became_active":"2024-09-18T08:02:49.891985+0000","last_became_peered":"2024-09-18T08:02:49.891985+0000","last_unstale":"2024-09-18T08:03:24.067407+0000","last_undegraded":"2024-09-18T08:03:24.067407+0000","last_fullsized":"2024-09-18T08:03:24.067407+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_clean_scrub_stamp":"2024-09-18T08:02:48.870718+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:04:12.297325+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":25,"num_read_kb":15,"num_write":13,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":40,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-18T08:03:24.068917+0000","last_change":"2024-09-18T08:02:47.876225+0000","last_active":"2024-09-18T08:03:24.068917+0000","last_peered":"2024-09-18T08:03:24.068917+0000","last_clean":"2024-09-18T08:03:24.068917+0000","last_became_active":"2024-09-18T08:02:47.875950+0000","last_became_peered":"2024-09-18T08:02:47.875950+0000","last_unstale":"2024-09-18T08:03:24.068917+0000","last_undegraded":"2024-09-18T08:03:24.068917+0000","last_fullsized":"2024-09-18T08:03:24.068917+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:02:46.860092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:46.860092+0000","last_clean_scrub_stamp":"2024-09-18T08:02:46.860092+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:04:02.960274+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"74'79","reported_seq":86,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-18T08:03:24.069256+0000","last_change":"2024-09-18T08:02:55.472070+0000","last_active":"2024-09-18T08:03:24.069256+0000","last_peered":"2024-09-18T08:03:24.069256+0000","last_clean":"2024-09-18T08:03:24.069256+0000","last_became_active":"2024-09-18T08:02:39.446743+0000","last_became_peered":"2024-09-18T08:02:39.446743+0000","last_unstale":"2024-09-18T08:03:24.069256+0000","last_undegraded":"2024-09-18T08:03:24.069256+0000","last_fullsized":"2024-09-18T08:03:24.069256+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"20'72","last_scrub_stamp":"2024-09-18T08:02:55.471998+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:00:11.893742+0000","last_clean_scrub_stamp":"2024-09-18T08:02:55.471998+0000","objects_scrubbed":2,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:04:21.599062+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":231,"num_write":113,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":590368,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":32,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-18T08:03:24.069422+0000","last_change":"2024-09-18T08:02:51.905819+0000","last_active":"2024-09-18T08:03:24.069422+0000","last_peered":"2024-09-18T08:03:24.069422+0000","last_clean":"2024-09-18T08:03:24.069422+0000","last_became_active":"2024-09-18T08:02:51.905516+0000","last_became_peered":"2024-09-18T08:02:51.905516+0000","last_unstale":"2024-09-18T08:03:24.069422+0000","last_undegraded":"2024-09-18T08:03:24.069422+0000","last_fullsized":"2024-09-18T08:03:24.069422+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:02:50.886830+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:50.886830+0000","last_clean_scrub_stamp":"2024-09-18T08:02:50.886830+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:04:04.232646+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":29,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-18T08:03:24.068580+0000","last_change":"2024-09-18T08:02:53.909023+0000","last_active":"2024-09-18T08:03:24.068580+0000","last_peered":"2024-09-18T08:03:24.068580+0000","last_clean":"2024-09-18T08:03:24.068580+0000","last_became_active":"2024-09-18T08:02:53.908668+0000","last_became_peered":"2024-09-18T08:02:53.908668+0000","last_unstale":"2024-09-18T08:03:24.068580+0000","last_undegraded":"2024-09-18T08:03:24.068580+0000","last_fullsized":"2024-09-18T08:03:24.068580+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_clean_scrub_stamp":"2024-09-18T08:02:52.890475+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:03:57.881484+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.e","version":"69'11","reported_seq":55,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-18T08:03:24.045529+0000","last_change":"2024-09-18T08:02:49.895676+0000","last_active":"2024-09-18T08:03:24.045529+0000","last_peered":"2024-09-18T08:03:24.045529+0000","last_clean":"2024-09-18T08:03:24.045529+0000","last_became_active":"2024-09-18T08:02:49.895303+0000","last_became_peered":"2024-09-18T08:02:49.895303+0000","last_unstale":"2024-09-18T08:03:24.045529+0000","last_undegraded":"2024-09-18T08:03:24.045529+0000","last_fullsized":"2024-09-18T08:03:24.045529+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_clean_scrub_stamp":"2024-09-18T08:02:48.870718+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:04:11.479080+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":40,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-18T08:03:24.044781+0000","last_change":"2024-09-18T08:02:47.880265+0000","last_active":"2024-09-18T08:03:24.044781+0000","last_peered":"2024-09-18T08:03:24.044781+0000","last_clean":"2024-09-18T08:03:24.044781+0000","last_became_active":"2024-09-18T08:02:47.879764+0000","last_became_peered":"2024-09-18T08:02:47.879764+0000","last_unstale":"2024-09-18T08:03:24.044781+0000","last_undegraded":"2024-09-18T08:03:24.044781+0000","last_fullsized":"2024-09-18T08:03:24.044781+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:02:46.860092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:46.860092+0000","last_clean_scrub_stamp":"2024-09-18T08:02:46.860092+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:04:12.919106+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":32,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-18T08:03:24.068436+0000","last_change":"2024-09-18T08:02:51.911107+0000","last_active":"2024-09-18T08:03:24.068436+0000","last_peered":"2024-09-18T08:03:24.068436+0000","last_clean":"2024-09-18T08:03:24.068436+0000","last_became_active":"2024-09-18T08:02:51.910649+0000","last_became_peered":"2024-09-18T08:02:51.910649+0000","last_unstale":"2024-09-18T08:03:24.068436+0000","last_undegraded":"2024-09-18T08:03:24.068436+0000","last_fullsized":"2024-09-18T08:03:24.068436+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:02:50.886830+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:50.886830+0000","last_clean_scrub_stamp":"2024-09-18T08:02:50.886830+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:03:56.497732+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":28,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-18T08:03:24.069188+0000","last_change":"2024-09-18T08:02:54.045510+0000","last_active":"2024-09-18T08:03:24.069188+0000","last_peered":"2024-09-18T08:03:24.069188+0000","last_clean":"2024-09-18T08:03:24.069188+0000","last_became_active":"2024-09-18T08:02:54.045091+0000","last_became_peered":"2024-09-18T08:02:54.045091+0000","last_unstale":"2024-09-18T08:03:24.069188+0000","last_undegraded":"2024-09-18T08:03:24.069188+0000","last_fullsized":"2024-09-18T08:03:24.069188+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_clean_scrub_stamp":"2024-09-18T08:02:52.890475+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:04:22.368585+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.d","version":"69'17","reported_seq":64,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-18T08:03:24.045257+0000","last_change":"2024-09-18T08:02:49.895971+0000","last_active":"2024-09-18T08:03:24.045257+0000","last_peered":"2024-09-18T08:03:24.045257+0000","last_clean":"2024-09-18T08:03:24.045257+0000","last_became_active":"2024-09-18T08:02:49.895578+0000","last_became_peered":"2024-09-18T08:02:49.895578+0000","last_unstale":"2024-09-18T08:03:24.045257+0000","last_undegraded":"2024-09-18T08:03:24.045257+0000","last_fullsized":"2024-09-18T08:03:24.045257+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_clean_scrub_stamp":"2024-09-18T08:02:48.870718+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:04:17.784735+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":29,"num_read_kb":19,"num_write":18,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,1],"acting":[4,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":40,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-18T08:03:24.068291+0000","last_change":"2024-09-18T08:02:47.884431+0000","last_active":"2024-09-18T08:03:24.068291+0000","last_peered":"2024-09-18T08:03:24.068291+0000","last_clean":"2024-09-18T08:03:24.068291+0000","last_became_active":"2024-09-18T08:02:47.884132+0000","last_became_peered":"2024-09-18T08:02:47.884132+0000","last_unstale":"2024-09-18T08:03:24.068291+0000","last_undegraded":"2024-09-18T08:03:24.068291+0000","last_fullsized":"2024-09-18T08:03:24.068291+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:02:46.860092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:46.860092+0000","last_clean_scrub_stamp":"2024-09-18T08:02:46.860092+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:04:06.189609+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":32,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-18T08:03:24.044843+0000","last_change":"2024-09-18T08:02:51.907544+0000","last_active":"2024-09-18T08:03:24.044843+0000","last_peered":"2024-09-18T08:03:24.044843+0000","last_clean":"2024-09-18T08:03:24.044843+0000","last_became_active":"2024-09-18T08:02:51.907196+0000","last_became_peered":"2024-09-18T08:02:51.907196+0000","last_unstale":"2024-09-18T08:03:24.044843+0000","last_undegraded":"2024-09-18T08:03:24.044843+0000","last_fullsized":"2024-09-18T08:03:24.044843+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:02:50.886830+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:50.886830+0000","last_clean_scrub_stamp":"2024-09-18T08:02:50.886830+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:04:03.690724+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":29,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-18T08:03:24.044090+0000","last_change":"2024-09-18T08:02:53.904497+0000","last_active":"2024-09-18T08:03:24.044090+0000","last_peered":"2024-09-18T08:03:24.044090+0000","last_clean":"2024-09-18T08:03:24.044090+0000","last_became_active":"2024-09-18T08:02:53.904144+0000","last_became_peered":"2024-09-18T08:02:53.904144+0000","last_unstale":"2024-09-18T08:03:24.044090+0000","last_undegraded":"2024-09-18T08:03:24.044090+0000","last_fullsized":"2024-09-18T08:03:24.044090+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_clean_scrub_stamp":"2024-09-18T08:02:52.890475+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:03:58.164335+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.c","version":"69'10","reported_seq":51,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-18T08:03:24.044828+0000","last_change":"2024-09-18T08:02:49.892847+0000","last_active":"2024-09-18T08:03:24.044828+0000","last_peered":"2024-09-18T08:03:24.044828+0000","last_clean":"2024-09-18T08:03:24.044828+0000","last_became_active":"2024-09-18T08:02:49.892527+0000","last_became_peered":"2024-09-18T08:02:49.892527+0000","last_unstale":"2024-09-18T08:03:24.044828+0000","last_undegraded":"2024-09-18T08:03:24.044828+0000","last_fullsized":"2024-09-18T08:03:24.044828+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_clean_scrub_stamp":"2024-09-18T08:02:48.870718+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:04:11.391201+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":40,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-18T08:03:24.069611+0000","last_change":"2024-09-18T08:02:47.881814+0000","last_active":"2024-09-18T08:03:24.069611+0000","last_peered":"2024-09-18T08:03:24.069611+0000","last_clean":"2024-09-18T08:03:24.069611+0000","last_became_active":"2024-09-18T08:02:47.881512+0000","last_became_peered":"2024-09-18T08:02:47.881512+0000","last_unstale":"2024-09-18T08:03:24.069611+0000","last_undegraded":"2024-09-18T08:03:24.069611+0000","last_fullsized":"2024-09-18T08:03:24.069611+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:02:46.860092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:46.860092+0000","last_clean_scrub_stamp":"2024-09-18T08:02:46.860092+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:03:59.505935+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.d","version":"69'8","reported_seq":49,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-18T08:03:24.044104+0000","last_change":"2024-09-18T08:02:51.909290+0000","last_active":"2024-09-18T08:03:24.044104+0000","last_peered":"2024-09-18T08:03:24.044104+0000","last_clean":"2024-09-18T08:03:24.044104+0000","last_became_active":"2024-09-18T08:02:51.909011+0000","last_became_peered":"2024-09-18T08:02:51.909011+0000","last_unstale":"2024-09-18T08:03:24.044104+0000","last_undegraded":"2024-09-18T08:03:24.044104+0000","last_fullsized":"2024-09-18T08:03:24.044104+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:02:50.886830+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:50.886830+0000","last_clean_scrub_stamp":"2024-09-18T08:02:50.886830+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:04:10.431784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":29,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-18T08:03:24.044965+0000","last_change":"2024-09-18T08:02:53.902197+0000","last_active":"2024-09-18T08:03:24.044965+0000","last_peered":"2024-09-18T08:03:24.044965+0000","last_clean":"2024-09-18T08:03:24.044965+0000","last_became_active":"2024-09-18T08:02:53.901858+0000","last_became_peered":"2024-09-18T08:02:53.901858+0000","last_unstale":"2024-09-18T08:03:24.044965+0000","last_undegraded":"2024-09-18T08:03:24.044965+0000","last_fullsized":"2024-09-18T08:03:24.044965+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_clean_scrub_stamp":"2024-09-18T08:02:52.890475+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:04:08.426639+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.b","version":"69'9","reported_seq":52,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-18T08:03:24.068672+0000","last_change":"2024-09-18T08:02:49.898562+0000","last_active":"2024-09-18T08:03:24.068672+0000","last_peered":"2024-09-18T08:03:24.068672+0000","last_clean":"2024-09-18T08:03:24.068672+0000","last_became_active":"2024-09-18T08:02:49.897998+0000","last_became_peered":"2024-09-18T08:02:49.897998+0000","last_unstale":"2024-09-18T08:03:24.068672+0000","last_undegraded":"2024-09-18T08:03:24.068672+0000","last_fullsized":"2024-09-18T08:03:24.068672+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_clean_scrub_stamp":"2024-09-18T08:02:48.870718+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:03:56.995348+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":40,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-18T08:03:24.068017+0000","last_change":"2024-09-18T08:02:47.877928+0000","last_active":"2024-09-18T08:03:24.068017+0000","last_peered":"2024-09-18T08:03:24.068017+0000","last_clean":"2024-09-18T08:03:24.068017+0000","last_became_active":"2024-09-18T08:02:47.877161+0000","last_became_peered":"2024-09-18T08:02:47.877161+0000","last_unstale":"2024-09-18T08:03:24.068017+0000","last_undegraded":"2024-09-18T08:03:24.068017+0000","last_fullsized":"2024-09-18T08:03:24.068017+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:02:46.860092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:46.860092+0000","last_clean_scrub_stamp":"2024-09-18T08:02:46.860092+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:03:54.682345+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":32,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-18T08:03:24.043844+0000","last_change":"2024-09-18T08:02:51.901738+0000","last_active":"2024-09-18T08:03:24.043844+0000","last_peered":"2024-09-18T08:03:24.043844+0000","last_clean":"2024-09-18T08:03:24.043844+0000","last_became_active":"2024-09-18T08:02:51.901330+0000","last_became_peered":"2024-09-18T08:02:51.901330+0000","last_unstale":"2024-09-18T08:03:24.043844+0000","last_undegraded":"2024-09-18T08:03:24.043844+0000","last_fullsized":"2024-09-18T08:03:24.043844+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:02:50.886830+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:50.886830+0000","last_clean_scrub_stamp":"2024-09-18T08:02:50.886830+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:04:18.413020+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,3],"acting":[2,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":29,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-18T08:03:24.068416+0000","last_change":"2024-09-18T08:02:53.911400+0000","last_active":"2024-09-18T08:03:24.068416+0000","last_peered":"2024-09-18T08:03:24.068416+0000","last_clean":"2024-09-18T08:03:24.068416+0000","last_became_active":"2024-09-18T08:02:53.911068+0000","last_became_peered":"2024-09-18T08:02:53.911068+0000","last_unstale":"2024-09-18T08:03:24.068416+0000","last_undegraded":"2024-09-18T08:03:24.068416+0000","last_fullsized":"2024-09-18T08:03:24.068416+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_clean_scrub_stamp":"2024-09-18T08:02:52.890475+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:03:57.470235+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.a","version":"69'19","reported_seq":67,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-18T08:03:24.068087+0000","last_change":"2024-09-18T08:02:49.898366+0000","last_active":"2024-09-18T08:03:24.068087+0000","last_peered":"2024-09-18T08:03:24.068087+0000","last_clean":"2024-09-18T08:03:24.068087+0000","last_became_active":"2024-09-18T08:02:49.897960+0000","last_became_peered":"2024-09-18T08:02:49.897960+0000","last_unstale":"2024-09-18T08:03:24.068087+0000","last_undegraded":"2024-09-18T08:03:24.068087+0000","last_fullsized":"2024-09-18T08:03:24.068087+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_clean_scrub_stamp":"2024-09-18T08:02:48.870718+0000","objects_scrubbed":0,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:04:18.534695+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":32,"num_read_kb":21,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":40,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-18T08:03:24.068580+0000","last_change":"2024-09-18T08:02:47.881119+0000","last_active":"2024-09-18T08:03:24.068580+0000","last_peered":"2024-09-18T08:03:24.068580+0000","last_clean":"2024-09-18T08:03:24.068580+0000","last_became_active":"2024-09-18T08:02:47.880645+0000","last_became_peered":"2024-09-18T08:02:47.880645+0000","last_unstale":"2024-09-18T08:03:24.068580+0000","last_undegraded":"2024-09-18T08:03:24.068580+0000","last_fullsized":"2024-09-18T08:03:24.068580+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:02:46.860092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:46.860092+0000","last_clean_scrub_stamp":"2024-09-18T08:02:46.860092+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:04:02.022191+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":32,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-18T08:03:24.043973+0000","last_change":"2024-09-18T08:02:51.903944+0000","last_active":"2024-09-18T08:03:24.043973+0000","last_peered":"2024-09-18T08:03:24.043973+0000","last_clean":"2024-09-18T08:03:24.043973+0000","last_became_active":"2024-09-18T08:02:51.903439+0000","last_became_peered":"2024-09-18T08:02:51.903439+0000","last_unstale":"2024-09-18T08:03:24.043973+0000","last_undegraded":"2024-09-18T08:03:24.043973+0000","last_fullsized":"2024-09-18T08:03:24.043973+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:02:50.886830+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:50.886830+0000","last_clean_scrub_stamp":"2024-09-18T08:02:50.886830+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:03:52.993653+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":29,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-18T08:03:24.069042+0000","last_change":"2024-09-18T08:02:53.913096+0000","last_active":"2024-09-18T08:03:24.069042+0000","last_peered":"2024-09-18T08:03:24.069042+0000","last_clean":"2024-09-18T08:03:24.069042+0000","last_became_active":"2024-09-18T08:02:53.912762+0000","last_became_peered":"2024-09-18T08:02:53.912762+0000","last_unstale":"2024-09-18T08:03:24.069042+0000","last_undegraded":"2024-09-18T08:03:24.069042+0000","last_fullsized":"2024-09-18T08:03:24.069042+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_clean_scrub_stamp":"2024-09-18T08:02:52.890475+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:04:03.478804+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.9","version":"69'12","reported_seq":59,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-18T08:03:24.044605+0000","last_change":"2024-09-18T08:02:49.895136+0000","last_active":"2024-09-18T08:03:24.044605+0000","last_peered":"2024-09-18T08:03:24.044605+0000","last_clean":"2024-09-18T08:03:24.044605+0000","last_became_active":"2024-09-18T08:02:49.894762+0000","last_became_peered":"2024-09-18T08:02:49.894762+0000","last_unstale":"2024-09-18T08:03:24.044605+0000","last_undegraded":"2024-09-18T08:03:24.044605+0000","last_fullsized":"2024-09-18T08:03:24.044605+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_clean_scrub_stamp":"2024-09-18T08:02:48.870718+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:04:02.257011+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":25,"num_read_kb":16,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":40,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-18T08:03:24.069279+0000","last_change":"2024-09-18T08:02:47.884430+0000","last_active":"2024-09-18T08:03:24.069279+0000","last_peered":"2024-09-18T08:03:24.069279+0000","last_clean":"2024-09-18T08:03:24.069279+0000","last_became_active":"2024-09-18T08:02:47.884052+0000","last_became_peered":"2024-09-18T08:02:47.884052+0000","last_unstale":"2024-09-18T08:03:24.069279+0000","last_undegraded":"2024-09-18T08:03:24.069279+0000","last_fullsized":"2024-09-18T08:03:24.069279+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:02:46.860092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:46.860092+0000","last_clean_scrub_stamp":"2024-09-18T08:02:46.860092+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:04:04.711823+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":32,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-18T08:03:24.043796+0000","last_change":"2024-09-18T08:02:51.902541+0000","last_active":"2024-09-18T08:03:24.043796+0000","last_peered":"2024-09-18T08:03:24.043796+0000","last_clean":"2024-09-18T08:03:24.043796+0000","last_became_active":"2024-09-18T08:02:51.902163+0000","last_became_peered":"2024-09-18T08:02:51.902163+0000","last_unstale":"2024-09-18T08:03:24.043796+0000","last_undegraded":"2024-09-18T08:03:24.043796+0000","last_fullsized":"2024-09-18T08:03:24.043796+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:02:50.886830+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:50.886830+0000","last_clean_scrub_stamp":"2024-09-18T08:02:50.886830+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:03:54.180178+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":29,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-18T08:03:24.069270+0000","last_change":"2024-09-18T08:02:53.907752+0000","last_active":"2024-09-18T08:03:24.069270+0000","last_peered":"2024-09-18T08:03:24.069270+0000","last_clean":"2024-09-18T08:03:24.069270+0000","last_became_active":"2024-09-18T08:02:53.907412+0000","last_became_peered":"2024-09-18T08:02:53.907412+0000","last_unstale":"2024-09-18T08:03:24.069270+0000","last_undegraded":"2024-09-18T08:03:24.069270+0000","last_fullsized":"2024-09-18T08:03:24.069270+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_clean_scrub_stamp":"2024-09-18T08:02:52.890475+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:04:12.083897+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.8","version":"69'15","reported_seq":61,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-18T08:03:24.066953+0000","last_change":"2024-09-18T08:02:49.889133+0000","last_active":"2024-09-18T08:03:24.066953+0000","last_peered":"2024-09-18T08:03:24.066953+0000","last_clean":"2024-09-18T08:03:24.066953+0000","last_became_active":"2024-09-18T08:02:49.888799+0000","last_became_peered":"2024-09-18T08:02:49.888799+0000","last_unstale":"2024-09-18T08:03:24.066953+0000","last_undegraded":"2024-09-18T08:03:24.066953+0000","last_fullsized":"2024-09-18T08:03:24.066953+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_clean_scrub_stamp":"2024-09-18T08:02:48.870718+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:04:11.344161+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.f","version":"62'2","reported_seq":52,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-18T08:03:24.069407+0000","last_change":"2024-09-18T08:02:47.881441+0000","last_active":"2024-09-18T08:03:24.069407+0000","last_peered":"2024-09-18T08:03:24.069407+0000","last_clean":"2024-09-18T08:03:24.069407+0000","last_became_active":"2024-09-18T08:02:47.881165+0000","last_became_peered":"2024-09-18T08:02:47.881165+0000","last_unstale":"2024-09-18T08:03:24.069407+0000","last_undegraded":"2024-09-18T08:03:24.069407+0000","last_fullsized":"2024-09-18T08:03:24.069407+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:02:46.860092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:46.860092+0000","last_clean_scrub_stamp":"2024-09-18T08:02:46.860092+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:03:50.038477+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":4,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.9","version":"69'8","reported_seq":52,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-18T08:03:24.068924+0000","last_change":"2024-09-18T08:02:51.905340+0000","last_active":"2024-09-18T08:03:24.068924+0000","last_peered":"2024-09-18T08:03:24.068924+0000","last_clean":"2024-09-18T08:03:24.068924+0000","last_became_active":"2024-09-18T08:02:51.905059+0000","last_became_peered":"2024-09-18T08:02:51.905059+0000","last_unstale":"2024-09-18T08:03:24.068924+0000","last_undegraded":"2024-09-18T08:03:24.068924+0000","last_fullsized":"2024-09-18T08:03:24.068924+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:02:50.886830+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:50.886830+0000","last_clean_scrub_stamp":"2024-09-18T08:02:50.886830+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:03:52.023681+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":28,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-18T08:03:24.067162+0000","last_change":"2024-09-18T08:02:54.046138+0000","last_active":"2024-09-18T08:03:24.067162+0000","last_peered":"2024-09-18T08:03:24.067162+0000","last_clean":"2024-09-18T08:03:24.067162+0000","last_became_active":"2024-09-18T08:02:54.045845+0000","last_became_peered":"2024-09-18T08:02:54.045845+0000","last_unstale":"2024-09-18T08:03:24.067162+0000","last_undegraded":"2024-09-18T08:03:24.067162+0000","last_fullsized":"2024-09-18T08:03:24.067162+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_clean_scrub_stamp":"2024-09-18T08:02:52.890475+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:03:55.148975+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":40,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-18T08:03:24.067347+0000","last_change":"2024-09-18T08:02:47.882886+0000","last_active":"2024-09-18T08:03:24.067347+0000","last_peered":"2024-09-18T08:03:24.067347+0000","last_clean":"2024-09-18T08:03:24.067347+0000","last_became_active":"2024-09-18T08:02:47.882600+0000","last_became_peered":"2024-09-18T08:02:47.882600+0000","last_unstale":"2024-09-18T08:03:24.067347+0000","last_undegraded":"2024-09-18T08:03:24.067347+0000","last_fullsized":"2024-09-18T08:03:24.067347+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:02:46.860092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:46.860092+0000","last_clean_scrub_stamp":"2024-09-18T08:02:46.860092+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:04:15.079741+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"69'6","reported_seq":45,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-18T08:03:24.069830+0000","last_change":"2024-09-18T08:02:49.895477+0000","last_active":"2024-09-18T08:03:24.069830+0000","last_peered":"2024-09-18T08:03:24.069830+0000","last_clean":"2024-09-18T08:03:24.069830+0000","last_became_active":"2024-09-18T08:02:49.895067+0000","last_became_peered":"2024-09-18T08:02:49.895067+0000","last_unstale":"2024-09-18T08:03:24.069830+0000","last_undegraded":"2024-09-18T08:03:24.069830+0000","last_fullsized":"2024-09-18T08:03:24.069830+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_clean_scrub_stamp":"2024-09-18T08:02:48.870718+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:03:55.471468+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":9,"num_read_kb":6,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":32,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-18T08:03:24.068128+0000","last_change":"2024-09-18T08:02:51.912907+0000","last_active":"2024-09-18T08:03:24.068128+0000","last_peered":"2024-09-18T08:03:24.068128+0000","last_clean":"2024-09-18T08:03:24.068128+0000","last_became_active":"2024-09-18T08:02:51.912557+0000","last_became_peered":"2024-09-18T08:02:51.912557+0000","last_unstale":"2024-09-18T08:03:24.068128+0000","last_undegraded":"2024-09-18T08:03:24.068128+0000","last_fullsized":"2024-09-18T08:03:24.068128+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:02:50.886830+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:50.886830+0000","last_clean_scrub_stamp":"2024-09-18T08:02:50.886830+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:03:58.405243+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,1],"acting":[5,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":29,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-18T08:03:24.068062+0000","last_change":"2024-09-18T08:02:54.049521+0000","last_active":"2024-09-18T08:03:24.068062+0000","last_peered":"2024-09-18T08:03:24.068062+0000","last_clean":"2024-09-18T08:03:24.068062+0000","last_became_active":"2024-09-18T08:02:54.048959+0000","last_became_peered":"2024-09-18T08:02:54.048959+0000","last_unstale":"2024-09-18T08:03:24.068062+0000","last_undegraded":"2024-09-18T08:03:24.068062+0000","last_fullsized":"2024-09-18T08:03:24.068062+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_clean_scrub_stamp":"2024-09-18T08:02:52.890475+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:03:56.721996+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"69'9","reported_seq":52,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-18T08:03:24.070139+0000","last_change":"2024-09-18T08:02:49.892390+0000","last_active":"2024-09-18T08:03:24.070139+0000","last_peered":"2024-09-18T08:03:24.070139+0000","last_clean":"2024-09-18T08:03:24.070139+0000","last_became_active":"2024-09-18T08:02:49.891567+0000","last_became_peered":"2024-09-18T08:02:49.891567+0000","last_unstale":"2024-09-18T08:03:24.070139+0000","last_undegraded":"2024-09-18T08:03:24.070139+0000","last_fullsized":"2024-09-18T08:03:24.070139+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_clean_scrub_stamp":"2024-09-18T08:02:48.870718+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:03:51.625809+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":40,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-18T08:03:24.069122+0000","last_change":"2024-09-18T08:02:47.886233+0000","last_active":"2024-09-18T08:03:24.069122+0000","last_peered":"2024-09-18T08:03:24.069122+0000","last_clean":"2024-09-18T08:03:24.069122+0000","last_became_active":"2024-09-18T08:02:47.885790+0000","last_became_peered":"2024-09-18T08:02:47.885790+0000","last_unstale":"2024-09-18T08:03:24.069122+0000","last_undegraded":"2024-09-18T08:03:24.069122+0000","last_fullsized":"2024-09-18T08:03:24.069122+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:02:46.860092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:46.860092+0000","last_clean_scrub_stamp":"2024-09-18T08:02:46.860092+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:03:48.811474+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":32,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-18T08:03:24.069778+0000","last_change":"2024-09-18T08:02:51.908640+0000","last_active":"2024-09-18T08:03:24.069778+0000","last_peered":"2024-09-18T08:03:24.069778+0000","last_clean":"2024-09-18T08:03:24.069778+0000","last_became_active":"2024-09-18T08:02:51.907361+0000","last_became_peered":"2024-09-18T08:02:51.907361+0000","last_unstale":"2024-09-18T08:03:24.069778+0000","last_undegraded":"2024-09-18T08:03:24.069778+0000","last_fullsized":"2024-09-18T08:03:24.069778+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:02:50.886830+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:50.886830+0000","last_clean_scrub_stamp":"2024-09-18T08:02:50.886830+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:04:06.357747+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":29,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-18T08:03:24.044286+0000","last_change":"2024-09-18T08:02:53.909246+0000","last_active":"2024-09-18T08:03:24.044286+0000","last_peered":"2024-09-18T08:03:24.044286+0000","last_clean":"2024-09-18T08:03:24.044286+0000","last_became_active":"2024-09-18T08:02:53.908917+0000","last_became_peered":"2024-09-18T08:02:53.908917+0000","last_unstale":"2024-09-18T08:03:24.044286+0000","last_undegraded":"2024-09-18T08:03:24.044286+0000","last_fullsized":"2024-09-18T08:03:24.044286+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_clean_scrub_stamp":"2024-09-18T08:02:52.890475+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:03:57.367069+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.15","version":"69'9","reported_seq":52,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-18T08:03:24.069378+0000","last_change":"2024-09-18T08:02:49.890846+0000","last_active":"2024-09-18T08:03:24.069378+0000","last_peered":"2024-09-18T08:03:24.069378+0000","last_clean":"2024-09-18T08:03:24.069378+0000","last_became_active":"2024-09-18T08:02:49.890497+0000","last_became_peered":"2024-09-18T08:02:49.890497+0000","last_unstale":"2024-09-18T08:03:24.069378+0000","last_undegraded":"2024-09-18T08:03:24.069378+0000","last_fullsized":"2024-09-18T08:03:24.069378+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_clean_scrub_stamp":"2024-09-18T08:02:48.870718+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:04:02.479848+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":40,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-18T08:03:24.069242+0000","last_change":"2024-09-18T08:02:47.877632+0000","last_active":"2024-09-18T08:03:24.069242+0000","last_peered":"2024-09-18T08:03:24.069242+0000","last_clean":"2024-09-18T08:03:24.069242+0000","last_became_active":"2024-09-18T08:02:47.875983+0000","last_became_peered":"2024-09-18T08:02:47.875983+0000","last_unstale":"2024-09-18T08:03:24.069242+0000","last_undegraded":"2024-09-18T08:03:24.069242+0000","last_fullsized":"2024-09-18T08:03:24.069242+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:02:46.860092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:46.860092+0000","last_clean_scrub_stamp":"2024-09-18T08:02:46.860092+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:04:09.677911+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.14","version":"69'8","reported_seq":49,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-18T08:03:24.069336+0000","last_change":"2024-09-18T08:02:51.909896+0000","last_active":"2024-09-18T08:03:24.069336+0000","last_peered":"2024-09-18T08:03:24.069336+0000","last_clean":"2024-09-18T08:03:24.069336+0000","last_became_active":"2024-09-18T08:02:51.909513+0000","last_became_peered":"2024-09-18T08:02:51.909513+0000","last_unstale":"2024-09-18T08:03:24.069336+0000","last_undegraded":"2024-09-18T08:03:24.069336+0000","last_fullsized":"2024-09-18T08:03:24.069336+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:02:50.886830+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:50.886830+0000","last_clean_scrub_stamp":"2024-09-18T08:02:50.886830+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:03:54.167632+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":29,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-18T08:03:24.045248+0000","last_change":"2024-09-18T08:02:53.907729+0000","last_active":"2024-09-18T08:03:24.045248+0000","last_peered":"2024-09-18T08:03:24.045248+0000","last_clean":"2024-09-18T08:03:24.045248+0000","last_became_active":"2024-09-18T08:02:53.907428+0000","last_became_peered":"2024-09-18T08:02:53.907428+0000","last_unstale":"2024-09-18T08:03:24.045248+0000","last_undegraded":"2024-09-18T08:03:24.045248+0000","last_fullsized":"2024-09-18T08:03:24.045248+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_clean_scrub_stamp":"2024-09-18T08:02:52.890475+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:04:00.922586+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.14","version":"69'10","reported_seq":51,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-18T08:03:24.069603+0000","last_change":"2024-09-18T08:02:49.889483+0000","last_active":"2024-09-18T08:03:24.069603+0000","last_peered":"2024-09-18T08:03:24.069603+0000","last_clean":"2024-09-18T08:03:24.069603+0000","last_became_active":"2024-09-18T08:02:49.889191+0000","last_became_peered":"2024-09-18T08:02:49.889191+0000","last_unstale":"2024-09-18T08:03:24.069603+0000","last_undegraded":"2024-09-18T08:03:24.069603+0000","last_fullsized":"2024-09-18T08:03:24.069603+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_clean_scrub_stamp":"2024-09-18T08:02:48.870718+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:03:58.029274+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":40,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-18T08:03:24.067955+0000","last_change":"2024-09-18T08:02:47.882542+0000","last_active":"2024-09-18T08:03:24.067955+0000","last_peered":"2024-09-18T08:03:24.067955+0000","last_clean":"2024-09-18T08:03:24.067955+0000","last_became_active":"2024-09-18T08:02:47.882065+0000","last_became_peered":"2024-09-18T08:02:47.882065+0000","last_unstale":"2024-09-18T08:03:24.067955+0000","last_undegraded":"2024-09-18T08:03:24.067955+0000","last_fullsized":"2024-09-18T08:03:24.067955+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:02:46.860092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:46.860092+0000","last_clean_scrub_stamp":"2024-09-18T08:02:46.860092+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:03:59.012126+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.15","version":"69'8","reported_seq":49,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-18T08:03:24.069213+0000","last_change":"2024-09-18T08:02:51.914448+0000","last_active":"2024-09-18T08:03:24.069213+0000","last_peered":"2024-09-18T08:03:24.069213+0000","last_clean":"2024-09-18T08:03:24.069213+0000","last_became_active":"2024-09-18T08:02:51.914135+0000","last_became_peered":"2024-09-18T08:02:51.914135+0000","last_unstale":"2024-09-18T08:03:24.069213+0000","last_undegraded":"2024-09-18T08:03:24.069213+0000","last_fullsized":"2024-09-18T08:03:24.069213+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:02:50.886830+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:50.886830+0000","last_clean_scrub_stamp":"2024-09-18T08:02:50.886830+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:03:56.893696+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":29,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-18T08:03:24.069792+0000","last_change":"2024-09-18T08:02:53.912409+0000","last_active":"2024-09-18T08:03:24.069792+0000","last_peered":"2024-09-18T08:03:24.069792+0000","last_clean":"2024-09-18T08:03:24.069792+0000","last_became_active":"2024-09-18T08:02:53.912112+0000","last_became_peered":"2024-09-18T08:02:53.912112+0000","last_unstale":"2024-09-18T08:03:24.069792+0000","last_undegraded":"2024-09-18T08:03:24.069792+0000","last_fullsized":"2024-09-18T08:03:24.069792+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_clean_scrub_stamp":"2024-09-18T08:02:52.890475+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:03:54.310553+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.13","version":"69'11","reported_seq":55,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-18T08:03:24.044794+0000","last_change":"2024-09-18T08:02:49.897433+0000","last_active":"2024-09-18T08:03:24.044794+0000","last_peered":"2024-09-18T08:03:24.044794+0000","last_clean":"2024-09-18T08:03:24.044794+0000","last_became_active":"2024-09-18T08:02:49.896949+0000","last_became_peered":"2024-09-18T08:02:49.896949+0000","last_unstale":"2024-09-18T08:03:24.044794+0000","last_undegraded":"2024-09-18T08:03:24.044794+0000","last_fullsized":"2024-09-18T08:03:24.044794+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_clean_scrub_stamp":"2024-09-18T08:02:48.870718+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:04:08.564349+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":40,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-18T08:03:24.045330+0000","last_change":"2024-09-18T08:02:47.880202+0000","last_active":"2024-09-18T08:03:24.045330+0000","last_peered":"2024-09-18T08:03:24.045330+0000","last_clean":"2024-09-18T08:03:24.045330+0000","last_became_active":"2024-09-18T08:02:47.879771+0000","last_became_peered":"2024-09-18T08:02:47.879771+0000","last_unstale":"2024-09-18T08:03:24.045330+0000","last_undegraded":"2024-09-18T08:03:24.045330+0000","last_fullsized":"2024-09-18T08:03:24.045330+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:02:46.860092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:46.860092+0000","last_clean_scrub_stamp":"2024-09-18T08:02:46.860092+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:03:58.640926+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":32,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-18T08:03:24.043640+0000","last_change":"2024-09-18T08:02:51.912062+0000","last_active":"2024-09-18T08:03:24.043640+0000","last_peered":"2024-09-18T08:03:24.043640+0000","last_clean":"2024-09-18T08:03:24.043640+0000","last_became_active":"2024-09-18T08:02:51.910752+0000","last_became_peered":"2024-09-18T08:02:51.910752+0000","last_unstale":"2024-09-18T08:03:24.043640+0000","last_undegraded":"2024-09-18T08:03:24.043640+0000","last_fullsized":"2024-09-18T08:03:24.043640+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:02:50.886830+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:50.886830+0000","last_clean_scrub_stamp":"2024-09-18T08:02:50.886830+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:04:01.465747+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":29,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-18T08:03:24.068460+0000","last_change":"2024-09-18T08:02:53.911861+0000","last_active":"2024-09-18T08:03:24.068460+0000","last_peered":"2024-09-18T08:03:24.068460+0000","last_clean":"2024-09-18T08:03:24.068460+0000","last_became_active":"2024-09-18T08:02:53.911429+0000","last_became_peered":"2024-09-18T08:02:53.911429+0000","last_unstale":"2024-09-18T08:03:24.068460+0000","last_undegraded":"2024-09-18T08:03:24.068460+0000","last_fullsized":"2024-09-18T08:03:24.068460+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_clean_scrub_stamp":"2024-09-18T08:02:52.890475+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:03:58.350802+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.12","version":"69'9","reported_seq":52,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-18T08:03:24.044006+0000","last_change":"2024-09-18T08:02:49.893472+0000","last_active":"2024-09-18T08:03:24.044006+0000","last_peered":"2024-09-18T08:03:24.044006+0000","last_clean":"2024-09-18T08:03:24.044006+0000","last_became_active":"2024-09-18T08:02:49.885950+0000","last_became_peered":"2024-09-18T08:02:49.885950+0000","last_unstale":"2024-09-18T08:03:24.044006+0000","last_undegraded":"2024-09-18T08:03:24.044006+0000","last_fullsized":"2024-09-18T08:03:24.044006+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_clean_scrub_stamp":"2024-09-18T08:02:48.870718+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:03:57.261106+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":40,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-18T08:03:24.068424+0000","last_change":"2024-09-18T08:02:47.883002+0000","last_active":"2024-09-18T08:03:24.068424+0000","last_peered":"2024-09-18T08:03:24.068424+0000","last_clean":"2024-09-18T08:03:24.068424+0000","last_became_active":"2024-09-18T08:02:47.882096+0000","last_became_peered":"2024-09-18T08:02:47.882096+0000","last_unstale":"2024-09-18T08:03:24.068424+0000","last_undegraded":"2024-09-18T08:03:24.068424+0000","last_fullsized":"2024-09-18T08:03:24.068424+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:02:46.860092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:46.860092+0000","last_clean_scrub_stamp":"2024-09-18T08:02:46.860092+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:04:10.794338+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":32,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-18T08:03:24.069370+0000","last_change":"2024-09-18T08:02:51.906594+0000","last_active":"2024-09-18T08:03:24.069370+0000","last_peered":"2024-09-18T08:03:24.069370+0000","last_clean":"2024-09-18T08:03:24.069370+0000","last_became_active":"2024-09-18T08:02:51.906277+0000","last_became_peered":"2024-09-18T08:02:51.906277+0000","last_unstale":"2024-09-18T08:03:24.069370+0000","last_undegraded":"2024-09-18T08:03:24.069370+0000","last_fullsized":"2024-09-18T08:03:24.069370+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:02:50.886830+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:50.886830+0000","last_clean_scrub_stamp":"2024-09-18T08:02:50.886830+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:04:07.327398+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.10","version":"69'1","reported_seq":30,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-18T08:03:24.069487+0000","last_change":"2024-09-18T08:02:53.913013+0000","last_active":"2024-09-18T08:03:24.069487+0000","last_peered":"2024-09-18T08:03:24.069487+0000","last_clean":"2024-09-18T08:03:24.069487+0000","last_became_active":"2024-09-18T08:02:53.912699+0000","last_became_peered":"2024-09-18T08:02:53.912699+0000","last_unstale":"2024-09-18T08:03:24.069487+0000","last_undegraded":"2024-09-18T08:03:24.069487+0000","last_fullsized":"2024-09-18T08:03:24.069487+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_clean_scrub_stamp":"2024-09-18T08:02:52.890475+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:04:15.154915+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.11","version":"69'11","reported_seq":55,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-18T08:03:24.068602+0000","last_change":"2024-09-18T08:02:49.889895+0000","last_active":"2024-09-18T08:03:24.068602+0000","last_peered":"2024-09-18T08:03:24.068602+0000","last_clean":"2024-09-18T08:03:24.068602+0000","last_became_active":"2024-09-18T08:02:49.889611+0000","last_became_peered":"2024-09-18T08:02:49.889611+0000","last_unstale":"2024-09-18T08:03:24.068602+0000","last_undegraded":"2024-09-18T08:03:24.068602+0000","last_fullsized":"2024-09-18T08:03:24.068602+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_clean_scrub_stamp":"2024-09-18T08:02:48.870718+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:03:51.062461+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":40,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-18T08:03:24.068217+0000","last_change":"2024-09-18T08:02:47.879259+0000","last_active":"2024-09-18T08:03:24.068217+0000","last_peered":"2024-09-18T08:03:24.068217+0000","last_clean":"2024-09-18T08:03:24.068217+0000","last_became_active":"2024-09-18T08:02:47.878886+0000","last_became_peered":"2024-09-18T08:02:47.878886+0000","last_unstale":"2024-09-18T08:03:24.068217+0000","last_undegraded":"2024-09-18T08:03:24.068217+0000","last_fullsized":"2024-09-18T08:03:24.068217+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:02:46.860092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:46.860092+0000","last_clean_scrub_stamp":"2024-09-18T08:02:46.860092+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:03:47.500112+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":32,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-18T08:03:24.066855+0000","last_change":"2024-09-18T08:02:51.908058+0000","last_active":"2024-09-18T08:03:24.066855+0000","last_peered":"2024-09-18T08:03:24.066855+0000","last_clean":"2024-09-18T08:03:24.066855+0000","last_became_active":"2024-09-18T08:02:51.907762+0000","last_became_peered":"2024-09-18T08:02:51.907762+0000","last_unstale":"2024-09-18T08:03:24.066855+0000","last_undegraded":"2024-09-18T08:03:24.066855+0000","last_fullsized":"2024-09-18T08:03:24.066855+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:02:50.886830+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:50.886830+0000","last_clean_scrub_stamp":"2024-09-18T08:02:50.886830+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:04:01.256104+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":28,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-18T08:03:24.069521+0000","last_change":"2024-09-18T08:02:54.047687+0000","last_active":"2024-09-18T08:03:24.069521+0000","last_peered":"2024-09-18T08:03:24.069521+0000","last_clean":"2024-09-18T08:03:24.069521+0000","last_became_active":"2024-09-18T08:02:54.047181+0000","last_became_peered":"2024-09-18T08:02:54.047181+0000","last_unstale":"2024-09-18T08:03:24.069521+0000","last_undegraded":"2024-09-18T08:03:24.069521+0000","last_fullsized":"2024-09-18T08:03:24.069521+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_clean_scrub_stamp":"2024-09-18T08:02:52.890475+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:04:12.998775+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.10","version":"69'4","reported_seq":42,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-18T08:03:24.068202+0000","last_change":"2024-09-18T08:02:49.890435+0000","last_active":"2024-09-18T08:03:24.068202+0000","last_peered":"2024-09-18T08:03:24.068202+0000","last_clean":"2024-09-18T08:03:24.068202+0000","last_became_active":"2024-09-18T08:02:49.890126+0000","last_became_peered":"2024-09-18T08:02:49.890126+0000","last_unstale":"2024-09-18T08:03:24.068202+0000","last_undegraded":"2024-09-18T08:03:24.068202+0000","last_fullsized":"2024-09-18T08:03:24.068202+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_clean_scrub_stamp":"2024-09-18T08:02:48.870718+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:03:56.821965+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":6,"num_read_kb":4,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":40,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-18T08:03:24.069343+0000","last_change":"2024-09-18T08:02:47.877991+0000","last_active":"2024-09-18T08:03:24.069343+0000","last_peered":"2024-09-18T08:03:24.069343+0000","last_clean":"2024-09-18T08:03:24.069343+0000","last_became_active":"2024-09-18T08:02:47.877616+0000","last_became_peered":"2024-09-18T08:02:47.877616+0000","last_unstale":"2024-09-18T08:03:24.069343+0000","last_undegraded":"2024-09-18T08:03:24.069343+0000","last_fullsized":"2024-09-18T08:03:24.069343+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:02:46.860092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:46.860092+0000","last_clean_scrub_stamp":"2024-09-18T08:02:46.860092+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:04:04.436191+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":32,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-18T08:03:24.067644+0000","last_change":"2024-09-18T08:02:51.906818+0000","last_active":"2024-09-18T08:03:24.067644+0000","last_peered":"2024-09-18T08:03:24.067644+0000","last_clean":"2024-09-18T08:03:24.067644+0000","last_became_active":"2024-09-18T08:02:51.906372+0000","last_became_peered":"2024-09-18T08:02:51.906372+0000","last_unstale":"2024-09-18T08:03:24.067644+0000","last_undegraded":"2024-09-18T08:03:24.067644+0000","last_fullsized":"2024-09-18T08:03:24.067644+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:02:50.886830+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:50.886830+0000","last_clean_scrub_stamp":"2024-09-18T08:02:50.886830+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:03:51.034977+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":29,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-18T08:03:24.068585+0000","last_change":"2024-09-18T08:02:53.909445+0000","last_active":"2024-09-18T08:03:24.068585+0000","last_peered":"2024-09-18T08:03:24.068585+0000","last_clean":"2024-09-18T08:03:24.068585+0000","last_became_active":"2024-09-18T08:02:53.909146+0000","last_became_peered":"2024-09-18T08:02:53.909146+0000","last_unstale":"2024-09-18T08:03:24.068585+0000","last_undegraded":"2024-09-18T08:03:24.068585+0000","last_fullsized":"2024-09-18T08:03:24.068585+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_clean_scrub_stamp":"2024-09-18T08:02:52.890475+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:04:03.033337+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,4],"acting":[7,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":29,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-18T08:03:24.044393+0000","last_change":"2024-09-18T08:02:53.907055+0000","last_active":"2024-09-18T08:03:24.044393+0000","last_peered":"2024-09-18T08:03:24.044393+0000","last_clean":"2024-09-18T08:03:24.044393+0000","last_became_active":"2024-09-18T08:02:53.906726+0000","last_became_peered":"2024-09-18T08:02:53.906726+0000","last_unstale":"2024-09-18T08:03:24.044393+0000","last_undegraded":"2024-09-18T08:03:24.044393+0000","last_fullsized":"2024-09-18T08:03:24.044393+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_clean_scrub_stamp":"2024-09-18T08:02:52.890475+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:03:58.628509+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":40,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-18T08:03:24.069042+0000","last_change":"2024-09-18T08:02:47.879447+0000","last_active":"2024-09-18T08:03:24.069042+0000","last_peered":"2024-09-18T08:03:24.069042+0000","last_clean":"2024-09-18T08:03:24.069042+0000","last_became_active":"2024-09-18T08:02:47.878922+0000","last_became_peered":"2024-09-18T08:02:47.878922+0000","last_unstale":"2024-09-18T08:03:24.069042+0000","last_undegraded":"2024-09-18T08:03:24.069042+0000","last_fullsized":"2024-09-18T08:03:24.069042+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:02:46.860092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:46.860092+0000","last_clean_scrub_stamp":"2024-09-18T08:02:46.860092+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:03:54.771042+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"69'11","reported_seq":55,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-18T08:03:24.068564+0000","last_change":"2024-09-18T08:02:49.898630+0000","last_active":"2024-09-18T08:03:24.068564+0000","last_peered":"2024-09-18T08:03:24.068564+0000","last_clean":"2024-09-18T08:03:24.068564+0000","last_became_active":"2024-09-18T08:02:49.898048+0000","last_became_peered":"2024-09-18T08:02:49.898048+0000","last_unstale":"2024-09-18T08:03:24.068564+0000","last_undegraded":"2024-09-18T08:03:24.068564+0000","last_fullsized":"2024-09-18T08:03:24.068564+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_clean_scrub_stamp":"2024-09-18T08:02:48.870718+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:04:05.049957+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":32,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-18T08:03:24.069940+0000","last_change":"2024-09-18T08:02:51.900666+0000","last_active":"2024-09-18T08:03:24.069940+0000","last_peered":"2024-09-18T08:03:24.069940+0000","last_clean":"2024-09-18T08:03:24.069940+0000","last_became_active":"2024-09-18T08:02:51.900300+0000","last_became_peered":"2024-09-18T08:02:51.900300+0000","last_unstale":"2024-09-18T08:03:24.069940+0000","last_undegraded":"2024-09-18T08:03:24.069940+0000","last_fullsized":"2024-09-18T08:03:24.069940+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:02:50.886830+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:50.886830+0000","last_clean_scrub_stamp":"2024-09-18T08:02:50.886830+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:04:12.807594+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":416,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":64,"ondisk_log_size":64,"up":96,"acting":96,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":178,"num_object_clones":0,"num_object_copies":534,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":178,"num_whiteouts":0,"num_read":698,"num_read_kb":455,"num_write":417,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":417792,"data_stored":11106,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":393,"ondisk_log_size":393,"up":96,"acting":96,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":1613,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":24,"num_read_kb":24,"num_write":10,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":4839,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":6,"ondisk_log_size":6,"up":96,"acting":96,"num_store_stats":8},{"poolid":2,"num_pg":3,"stat_sum":{"num_bytes":408,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":35,"num_read_kb":30,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":8,"ondisk_log_size":8,"up":9,"acting":9,"num_store_stats":7},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":231,"num_write":113,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":590368,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":79,"ondisk_log_size":79,"up":3,"acting":3,"num_store_stats":3}],"osd_stats":[{"osd":7,"up_from":56,"seq":240518168588,"num_pgs":53,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28352,"kb_used_data":1324,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716800,"statfs":{"total":95995035648,"available":95966003200,"internally_reserved":0,"allocated":1355776,"data_stored":894044,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":6,"up_from":47,"seq":201863462930,"num_pgs":43,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28332,"kb_used_data":1304,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716820,"statfs":{"total":95995035648,"available":95966023680,"internally_reserved":0,"allocated":1335296,"data_stored":892959,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":5,"up_from":39,"seq":167503724568,"num_pgs":47,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27764,"kb_used_data":736,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717388,"statfs":{"total":95995035648,"available":95966605312,"internally_reserved":0,"allocated":753664,"data_stored":303399,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":4,"up_from":32,"seq":137438953501,"num_pgs":58,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27784,"kb_used_data":756,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717368,"statfs":{"total":95995035648,"available":95966584832,"internally_reserved":0,"allocated":774144,"data_stored":304189,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":26,"seq":111669149732,"num_pgs":56,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27788,"kb_used_data":760,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717364,"statfs":{"total":95995035648,"available":95966580736,"internally_reserved":0,"allocated":778240,"data_stored":303482,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":27654600},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411370,"num_pgs":36,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":31912,"kb_used_data":724,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713240,"statfs":{"total":95995035648,"available":95962357760,"internally_reserved":0,"allocated":741376,"data_stored":303175,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574894,"num_pgs":57,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27828,"kb_used_data":800,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717324,"statfs":{"total":95995035648,"available":95966539776,"internally_reserved":0,"allocated":819200,"data_stored":305818,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738421,"num_pgs":46,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28364,"kb_used_data":1336,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716788,"statfs":{"total":95995035648,"available":95965990912,"internally_reserved":0,"allocated":1368064,"data_stored":894759,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1131,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1521,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":436,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":46,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1177,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":436,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":90112,"data_stored":2338,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":798,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":1898,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1474,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":990,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1034,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1254,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":403,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":416,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":403,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T08:03:33.118 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid 779349ca-7593-11ef-bceb-c7b262605968 -- ceph pg dump --format=json 2024-09-18T08:03:33.399 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/779349ca-7593-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T08:03:33.481 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:03:33 smithi007 ceph-mon[39249]: from='mgr.24470 172.21.15.7:0/2594390261' entity='mgr.y' 2024-09-18T08:03:33.481 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:03:33 smithi007 ceph-mon[39249]: from='mgr.24470 172.21.15.7:0/2594390261' entity='mgr.y' 2024-09-18T08:03:33.481 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:03:33 smithi007 ceph-mon[39249]: from='mgr.24470 172.21.15.7:0/2594390261' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"} : dispatch 2024-09-18T08:03:33.481 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:03:33 smithi007 ceph-mon[39249]: from='mgr.24470 172.21.15.7:0/2594390261' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"} : dispatch 2024-09-18T08:03:33.481 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:03:33 smithi007 ceph-mon[39249]: from='mgr.24470 172.21.15.7:0/2594390261' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"} : dispatch 2024-09-18T08:03:33.481 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:03:33 smithi007 ceph-mon[39249]: from='mgr.24470 172.21.15.7:0/2594390261' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"} : dispatch 2024-09-18T08:03:33.481 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:03:33 smithi007 ceph-mon[39249]: Adjusting osd_memory_target on smithi007 to 3761M 2024-09-18T08:03:33.481 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:03:33 smithi007 ceph-mon[39249]: from='mgr.24470 172.21.15.7:0/2594390261' entity='mgr.y' 2024-09-18T08:03:33.482 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:03:33 smithi007 ceph-mon[39249]: from='mgr.24470 172.21.15.7:0/2594390261' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-18T08:03:33.482 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:03:33 smithi007 ceph-mon[39249]: from='mgr.24470 172.21.15.7:0/2594390261' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-18T08:03:33.482 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:03:33 smithi007 ceph-mon[39249]: Updating smithi007:/etc/ceph/ceph.conf 2024-09-18T08:03:33.482 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:03:33 smithi007 ceph-mon[39249]: Updating smithi050:/etc/ceph/ceph.conf 2024-09-18T08:03:33.482 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:03:33 smithi007 ceph-mon[39249]: from='client.14712 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-18T08:03:33.482 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:03:33 smithi007 ceph-mon[39249]: Updating smithi007:/var/lib/ceph/779349ca-7593-11ef-bceb-c7b262605968/config/ceph.conf 2024-09-18T08:03:33.482 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:03:33 smithi007 ceph-mon[39249]: Updating smithi050:/var/lib/ceph/779349ca-7593-11ef-bceb-c7b262605968/config/ceph.conf 2024-09-18T08:03:33.482 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:03:33 smithi007 ceph-mon[39249]: Updating smithi007:/etc/ceph/ceph.client.admin.keyring 2024-09-18T08:03:33.482 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:03:33 smithi007 ceph-mon[39249]: from='mgr.24470 172.21.15.7:0/2594390261' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-18T08:03:33.482 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:03:33 smithi007 ceph-mon[39249]: from='mgr.24470 172.21.15.7:0/2594390261' entity='mgr.y' 2024-09-18T08:03:33.482 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:03:33 smithi007 ceph-mon[39249]: from='mgr.24470 172.21.15.7:0/2594390261' entity='mgr.y' 2024-09-18T08:03:33.482 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:03:33 smithi007 ceph-mon[39249]: from='mgr.24470 172.21.15.7:0/2594390261' entity='mgr.y' 2024-09-18T08:03:33.482 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:03:33 smithi007 ceph-mon[39249]: from='mgr.24470 172.21.15.7:0/2594390261' entity='mgr.y' 2024-09-18T08:03:33.483 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:03:33 smithi007 ceph-mon[39249]: from='mgr.24470 172.21.15.7:0/2594390261' entity='mgr.y' 2024-09-18T08:03:33.483 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:03:33 smithi007 ceph-mon[29459]: from='mgr.24470 172.21.15.7:0/2594390261' entity='mgr.y' 2024-09-18T08:03:33.483 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:03:33 smithi007 ceph-mon[29459]: from='mgr.24470 172.21.15.7:0/2594390261' entity='mgr.y' 2024-09-18T08:03:33.483 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:03:33 smithi007 ceph-mon[29459]: from='mgr.24470 172.21.15.7:0/2594390261' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"} : dispatch 2024-09-18T08:03:33.483 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:03:33 smithi007 ceph-mon[29459]: from='mgr.24470 172.21.15.7:0/2594390261' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"} : dispatch 2024-09-18T08:03:33.484 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:03:33 smithi007 ceph-mon[29459]: from='mgr.24470 172.21.15.7:0/2594390261' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"} : dispatch 2024-09-18T08:03:33.484 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:03:33 smithi007 ceph-mon[29459]: from='mgr.24470 172.21.15.7:0/2594390261' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"} : dispatch 2024-09-18T08:03:33.484 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:03:33 smithi007 ceph-mon[29459]: Adjusting osd_memory_target on smithi007 to 3761M 2024-09-18T08:03:33.484 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:03:33 smithi007 ceph-mon[29459]: from='mgr.24470 172.21.15.7:0/2594390261' entity='mgr.y' 2024-09-18T08:03:33.484 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:03:33 smithi007 ceph-mon[29459]: from='mgr.24470 172.21.15.7:0/2594390261' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-18T08:03:33.484 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:03:33 smithi007 ceph-mon[29459]: from='mgr.24470 172.21.15.7:0/2594390261' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-18T08:03:33.484 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:03:33 smithi007 ceph-mon[29459]: Updating smithi007:/etc/ceph/ceph.conf 2024-09-18T08:03:33.484 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:03:33 smithi007 ceph-mon[29459]: Updating smithi050:/etc/ceph/ceph.conf 2024-09-18T08:03:33.484 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:03:33 smithi007 ceph-mon[29459]: from='client.14712 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-18T08:03:33.484 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:03:33 smithi007 ceph-mon[29459]: Updating smithi007:/var/lib/ceph/779349ca-7593-11ef-bceb-c7b262605968/config/ceph.conf 2024-09-18T08:03:33.484 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:03:33 smithi007 ceph-mon[29459]: Updating smithi050:/var/lib/ceph/779349ca-7593-11ef-bceb-c7b262605968/config/ceph.conf 2024-09-18T08:03:33.484 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:03:33 smithi007 ceph-mon[29459]: Updating smithi007:/etc/ceph/ceph.client.admin.keyring 2024-09-18T08:03:33.484 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:03:33 smithi007 ceph-mon[29459]: from='mgr.24470 172.21.15.7:0/2594390261' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-18T08:03:33.485 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:03:33 smithi007 ceph-mon[29459]: from='mgr.24470 172.21.15.7:0/2594390261' entity='mgr.y' 2024-09-18T08:03:33.485 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:03:33 smithi007 ceph-mon[29459]: from='mgr.24470 172.21.15.7:0/2594390261' entity='mgr.y' 2024-09-18T08:03:33.485 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:03:33 smithi007 ceph-mon[29459]: from='mgr.24470 172.21.15.7:0/2594390261' entity='mgr.y' 2024-09-18T08:03:33.485 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:03:33 smithi007 ceph-mon[29459]: from='mgr.24470 172.21.15.7:0/2594390261' entity='mgr.y' 2024-09-18T08:03:33.485 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:03:33 smithi007 ceph-mon[29459]: from='mgr.24470 172.21.15.7:0/2594390261' entity='mgr.y' 2024-09-18T08:03:33.656 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:03:33 smithi050 ceph-mon[32371]: from='mgr.24470 172.21.15.7:0/2594390261' entity='mgr.y' 2024-09-18T08:03:33.656 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:03:33 smithi050 ceph-mon[32371]: from='mgr.24470 172.21.15.7:0/2594390261' entity='mgr.y' 2024-09-18T08:03:33.656 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:03:33 smithi050 ceph-mon[32371]: from='mgr.24470 172.21.15.7:0/2594390261' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"} : dispatch 2024-09-18T08:03:33.656 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:03:33 smithi050 ceph-mon[32371]: from='mgr.24470 172.21.15.7:0/2594390261' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"} : dispatch 2024-09-18T08:03:33.656 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:03:33 smithi050 ceph-mon[32371]: from='mgr.24470 172.21.15.7:0/2594390261' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"} : dispatch 2024-09-18T08:03:33.657 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:03:33 smithi050 ceph-mon[32371]: from='mgr.24470 172.21.15.7:0/2594390261' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"} : dispatch 2024-09-18T08:03:33.657 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:03:33 smithi050 ceph-mon[32371]: Adjusting osd_memory_target on smithi007 to 3761M 2024-09-18T08:03:33.657 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:03:33 smithi050 ceph-mon[32371]: from='mgr.24470 172.21.15.7:0/2594390261' entity='mgr.y' 2024-09-18T08:03:33.657 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:03:33 smithi050 ceph-mon[32371]: from='mgr.24470 172.21.15.7:0/2594390261' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-18T08:03:33.657 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:03:33 smithi050 ceph-mon[32371]: from='mgr.24470 172.21.15.7:0/2594390261' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-18T08:03:33.657 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:03:33 smithi050 ceph-mon[32371]: Updating smithi007:/etc/ceph/ceph.conf 2024-09-18T08:03:33.657 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:03:33 smithi050 ceph-mon[32371]: Updating smithi050:/etc/ceph/ceph.conf 2024-09-18T08:03:33.657 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:03:33 smithi050 ceph-mon[32371]: from='client.14712 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-18T08:03:33.657 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:03:33 smithi050 ceph-mon[32371]: Updating smithi007:/var/lib/ceph/779349ca-7593-11ef-bceb-c7b262605968/config/ceph.conf 2024-09-18T08:03:33.657 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:03:33 smithi050 ceph-mon[32371]: Updating smithi050:/var/lib/ceph/779349ca-7593-11ef-bceb-c7b262605968/config/ceph.conf 2024-09-18T08:03:33.657 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:03:33 smithi050 ceph-mon[32371]: Updating smithi007:/etc/ceph/ceph.client.admin.keyring 2024-09-18T08:03:33.657 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:03:33 smithi050 ceph-mon[32371]: from='mgr.24470 172.21.15.7:0/2594390261' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-18T08:03:33.657 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:03:33 smithi050 ceph-mon[32371]: from='mgr.24470 172.21.15.7:0/2594390261' entity='mgr.y' 2024-09-18T08:03:33.658 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:03:33 smithi050 ceph-mon[32371]: from='mgr.24470 172.21.15.7:0/2594390261' entity='mgr.y' 2024-09-18T08:03:33.658 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:03:33 smithi050 ceph-mon[32371]: from='mgr.24470 172.21.15.7:0/2594390261' entity='mgr.y' 2024-09-18T08:03:33.658 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:03:33 smithi050 ceph-mon[32371]: from='mgr.24470 172.21.15.7:0/2594390261' entity='mgr.y' 2024-09-18T08:03:33.658 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:03:33 smithi050 ceph-mon[32371]: from='mgr.24470 172.21.15.7:0/2594390261' entity='mgr.y' 2024-09-18T08:03:34.158 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-09-18T08:03:34.160 INFO:teuthology.orchestra.run.smithi007.stderr:dumped all 2024-09-18T08:03:34.480 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:03:34 smithi007 ceph-mon[29459]: Updating smithi050:/etc/ceph/ceph.client.admin.keyring 2024-09-18T08:03:34.481 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:03:34 smithi007 ceph-mon[29459]: Updating smithi007:/var/lib/ceph/779349ca-7593-11ef-bceb-c7b262605968/config/ceph.client.admin.keyring 2024-09-18T08:03:34.481 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:03:34 smithi007 ceph-mon[29459]: Updating smithi050:/var/lib/ceph/779349ca-7593-11ef-bceb-c7b262605968/config/ceph.client.admin.keyring 2024-09-18T08:03:34.481 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:03:34 smithi007 ceph-mon[29459]: Deploying daemon alertmanager.a on smithi007 2024-09-18T08:03:34.481 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:03:34 smithi007 ceph-mon[29459]: pgmap v15: 132 pgs: 132 active+clean; 583 KiB data, 223 MiB used, 715 GiB / 715 GiB avail; 567 B/s rd, 0 op/s 2024-09-18T08:03:34.481 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:03:34 smithi007 ceph-mon[29459]: osdmap e78: 8 total, 8 up, 8 in 2024-09-18T08:03:34.481 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:03:34 smithi007 ceph-mon[39249]: Updating smithi050:/etc/ceph/ceph.client.admin.keyring 2024-09-18T08:03:34.481 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:03:34 smithi007 ceph-mon[39249]: Updating smithi007:/var/lib/ceph/779349ca-7593-11ef-bceb-c7b262605968/config/ceph.client.admin.keyring 2024-09-18T08:03:34.481 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:03:34 smithi007 ceph-mon[39249]: Updating smithi050:/var/lib/ceph/779349ca-7593-11ef-bceb-c7b262605968/config/ceph.client.admin.keyring 2024-09-18T08:03:34.482 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:03:34 smithi007 ceph-mon[39249]: Deploying daemon alertmanager.a on smithi007 2024-09-18T08:03:34.482 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:03:34 smithi007 ceph-mon[39249]: pgmap v15: 132 pgs: 132 active+clean; 583 KiB data, 223 MiB used, 715 GiB / 715 GiB avail; 567 B/s rd, 0 op/s 2024-09-18T08:03:34.482 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:03:34 smithi007 ceph-mon[39249]: osdmap e78: 8 total, 8 up, 8 in 2024-09-18T08:03:34.656 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:03:34 smithi050 ceph-mon[32371]: Updating smithi050:/etc/ceph/ceph.client.admin.keyring 2024-09-18T08:03:34.656 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:03:34 smithi050 ceph-mon[32371]: Updating smithi007:/var/lib/ceph/779349ca-7593-11ef-bceb-c7b262605968/config/ceph.client.admin.keyring 2024-09-18T08:03:34.656 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:03:34 smithi050 ceph-mon[32371]: Updating smithi050:/var/lib/ceph/779349ca-7593-11ef-bceb-c7b262605968/config/ceph.client.admin.keyring 2024-09-18T08:03:34.656 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:03:34 smithi050 ceph-mon[32371]: Deploying daemon alertmanager.a on smithi007 2024-09-18T08:03:34.656 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:03:34 smithi050 ceph-mon[32371]: pgmap v15: 132 pgs: 132 active+clean; 583 KiB data, 223 MiB used, 715 GiB / 715 GiB avail; 567 B/s rd, 0 op/s 2024-09-18T08:03:34.656 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:03:34 smithi050 ceph-mon[32371]: osdmap e78: 8 total, 8 up, 8 in 2024-09-18T08:03:35.480 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:03:35 smithi007 ceph-mon[29459]: from='client.24605 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-18T08:03:35.481 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:03:35 smithi007 ceph-mon[39249]: from='client.24605 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-18T08:03:35.600 INFO:teuthology.orchestra.run.smithi007.stdout:{"pg_ready":true,"pg_map":{"version":16,"stamp":"2024-09-18T08:03:34.045456+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":596507,"num_objects":199,"num_object_clones":0,"num_object_copies":597,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":199,"num_whiteouts":0,"num_read":888,"num_read_kb":745,"num_write":549,"num_write_kb":1673,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":590368,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":552,"ondisk_log_size":552,"up":396,"acting":396,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":396,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":228220,"kb_used_data":7836,"kb_used_omap":12,"kb_used_meta":220211,"kb_avail":749732996,"statfs":{"total":767960285184,"available":767726587904,"internally_reserved":0,"allocated":8024064,"data_stored":4253201,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12750,"internal_metadata":225496626},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.977026"},"pg_stats":[{"pgid":"3.1f","version":"0'0","reported_seq":42,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-09-18T08:03:29.423022+0000","last_change":"2024-09-18T08:02:47.878020+0000","last_active":"2024-09-18T08:03:29.423022+0000","last_peered":"2024-09-18T08:03:29.423022+0000","last_clean":"2024-09-18T08:03:29.423022+0000","last_became_active":"2024-09-18T08:02:47.877698+0000","last_became_peered":"2024-09-18T08:02:47.877698+0000","last_unstale":"2024-09-18T08:03:29.423022+0000","last_undegraded":"2024-09-18T08:03:29.423022+0000","last_fullsized":"2024-09-18T08:03:29.423022+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:02:46.860092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:46.860092+0000","last_clean_scrub_stamp":"2024-09-18T08:02:46.860092+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:04:14.205967+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"69'9","reported_seq":54,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-09-18T08:03:29.092963+0000","last_change":"2024-09-18T08:02:49.895688+0000","last_active":"2024-09-18T08:03:29.092963+0000","last_peered":"2024-09-18T08:03:29.092963+0000","last_clean":"2024-09-18T08:03:29.092963+0000","last_became_active":"2024-09-18T08:02:49.894557+0000","last_became_peered":"2024-09-18T08:02:49.894557+0000","last_unstale":"2024-09-18T08:03:29.092963+0000","last_undegraded":"2024-09-18T08:03:29.092963+0000","last_fullsized":"2024-09-18T08:03:29.092963+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_clean_scrub_stamp":"2024-09-18T08:02:48.870718+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:03:59.206088+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":34,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-09-18T08:03:29.092818+0000","last_change":"2024-09-18T08:02:51.907566+0000","last_active":"2024-09-18T08:03:29.092818+0000","last_peered":"2024-09-18T08:03:29.092818+0000","last_clean":"2024-09-18T08:03:29.092818+0000","last_became_active":"2024-09-18T08:02:51.907140+0000","last_became_peered":"2024-09-18T08:02:51.907140+0000","last_unstale":"2024-09-18T08:03:29.092818+0000","last_undegraded":"2024-09-18T08:03:29.092818+0000","last_fullsized":"2024-09-18T08:03:29.092818+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:02:50.886830+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:50.886830+0000","last_clean_scrub_stamp":"2024-09-18T08:02:50.886830+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:04:07.003335+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":31,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-09-18T08:03:29.093138+0000","last_change":"2024-09-18T08:02:53.906914+0000","last_active":"2024-09-18T08:03:29.093138+0000","last_peered":"2024-09-18T08:03:29.093138+0000","last_clean":"2024-09-18T08:03:29.093138+0000","last_became_active":"2024-09-18T08:02:53.906622+0000","last_became_peered":"2024-09-18T08:02:53.906622+0000","last_unstale":"2024-09-18T08:03:29.093138+0000","last_undegraded":"2024-09-18T08:03:29.093138+0000","last_fullsized":"2024-09-18T08:03:29.093138+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_clean_scrub_stamp":"2024-09-18T08:02:52.890475+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:04:20.738381+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":30,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-09-18T08:03:29.425118+0000","last_change":"2024-09-18T08:02:54.047999+0000","last_active":"2024-09-18T08:03:29.425118+0000","last_peered":"2024-09-18T08:03:29.425118+0000","last_clean":"2024-09-18T08:03:29.425118+0000","last_became_active":"2024-09-18T08:02:54.047640+0000","last_became_peered":"2024-09-18T08:02:54.047640+0000","last_unstale":"2024-09-18T08:03:29.425118+0000","last_undegraded":"2024-09-18T08:03:29.425118+0000","last_fullsized":"2024-09-18T08:03:29.425118+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_clean_scrub_stamp":"2024-09-18T08:02:52.890475+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:04:16.013706+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":42,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-09-18T08:03:29.425556+0000","last_change":"2024-09-18T08:02:47.872585+0000","last_active":"2024-09-18T08:03:29.425556+0000","last_peered":"2024-09-18T08:03:29.425556+0000","last_clean":"2024-09-18T08:03:29.425556+0000","last_became_active":"2024-09-18T08:02:47.872316+0000","last_became_peered":"2024-09-18T08:02:47.872316+0000","last_unstale":"2024-09-18T08:03:29.425556+0000","last_undegraded":"2024-09-18T08:03:29.425556+0000","last_fullsized":"2024-09-18T08:03:29.425556+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:02:46.860092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:46.860092+0000","last_clean_scrub_stamp":"2024-09-18T08:02:46.860092+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:03:53.731838+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"69'15","reported_seq":63,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-09-18T08:03:29.425064+0000","last_change":"2024-09-18T08:02:49.887111+0000","last_active":"2024-09-18T08:03:29.425064+0000","last_peered":"2024-09-18T08:03:29.425064+0000","last_clean":"2024-09-18T08:03:29.425064+0000","last_became_active":"2024-09-18T08:02:49.886106+0000","last_became_peered":"2024-09-18T08:02:49.886106+0000","last_unstale":"2024-09-18T08:03:29.425064+0000","last_undegraded":"2024-09-18T08:03:29.425064+0000","last_fullsized":"2024-09-18T08:03:29.425064+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_clean_scrub_stamp":"2024-09-18T08:02:48.870718+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:04:15.737211+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":34,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-09-18T08:03:29.092961+0000","last_change":"2024-09-18T08:02:51.904853+0000","last_active":"2024-09-18T08:03:29.092961+0000","last_peered":"2024-09-18T08:03:29.092961+0000","last_clean":"2024-09-18T08:03:29.092961+0000","last_became_active":"2024-09-18T08:02:51.904184+0000","last_became_peered":"2024-09-18T08:02:51.904184+0000","last_unstale":"2024-09-18T08:03:29.092961+0000","last_undegraded":"2024-09-18T08:03:29.092961+0000","last_fullsized":"2024-09-18T08:03:29.092961+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:02:50.886830+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:50.886830+0000","last_clean_scrub_stamp":"2024-09-18T08:02:50.886830+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:04:07.696637+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":42,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-09-18T08:03:29.427263+0000","last_change":"2024-09-18T08:02:47.875508+0000","last_active":"2024-09-18T08:03:29.427263+0000","last_peered":"2024-09-18T08:03:29.427263+0000","last_clean":"2024-09-18T08:03:29.427263+0000","last_became_active":"2024-09-18T08:02:47.875124+0000","last_became_peered":"2024-09-18T08:02:47.875124+0000","last_unstale":"2024-09-18T08:03:29.427263+0000","last_undegraded":"2024-09-18T08:03:29.427263+0000","last_fullsized":"2024-09-18T08:03:29.427263+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:02:46.860092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:46.860092+0000","last_clean_scrub_stamp":"2024-09-18T08:02:46.860092+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:03:53.477209+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"69'9","reported_seq":54,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-09-18T08:03:29.093330+0000","last_change":"2024-09-18T08:02:49.887589+0000","last_active":"2024-09-18T08:03:29.093330+0000","last_peered":"2024-09-18T08:03:29.093330+0000","last_clean":"2024-09-18T08:03:29.093330+0000","last_became_active":"2024-09-18T08:02:49.887161+0000","last_became_peered":"2024-09-18T08:02:49.887161+0000","last_unstale":"2024-09-18T08:03:29.093330+0000","last_undegraded":"2024-09-18T08:03:29.093330+0000","last_fullsized":"2024-09-18T08:03:29.093330+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_clean_scrub_stamp":"2024-09-18T08:02:48.870718+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:03:51.892069+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":34,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-09-18T08:03:29.426283+0000","last_change":"2024-09-18T08:02:51.907597+0000","last_active":"2024-09-18T08:03:29.426283+0000","last_peered":"2024-09-18T08:03:29.426283+0000","last_clean":"2024-09-18T08:03:29.426283+0000","last_became_active":"2024-09-18T08:02:51.907296+0000","last_became_peered":"2024-09-18T08:02:51.907296+0000","last_unstale":"2024-09-18T08:03:29.426283+0000","last_undegraded":"2024-09-18T08:03:29.426283+0000","last_fullsized":"2024-09-18T08:03:29.426283+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:02:50.886830+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:50.886830+0000","last_clean_scrub_stamp":"2024-09-18T08:02:50.886830+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:04:06.057644+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":31,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-09-18T08:03:29.422357+0000","last_change":"2024-09-18T08:02:53.913803+0000","last_active":"2024-09-18T08:03:29.422357+0000","last_peered":"2024-09-18T08:03:29.422357+0000","last_clean":"2024-09-18T08:03:29.422357+0000","last_became_active":"2024-09-18T08:02:53.913519+0000","last_became_peered":"2024-09-18T08:02:53.913519+0000","last_unstale":"2024-09-18T08:03:29.422357+0000","last_undegraded":"2024-09-18T08:03:29.422357+0000","last_fullsized":"2024-09-18T08:03:29.422357+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_clean_scrub_stamp":"2024-09-18T08:02:52.890475+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:04:18.699540+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":42,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-09-18T08:03:29.426784+0000","last_change":"2024-09-18T08:02:47.877577+0000","last_active":"2024-09-18T08:03:29.426784+0000","last_peered":"2024-09-18T08:03:29.426784+0000","last_clean":"2024-09-18T08:03:29.426784+0000","last_became_active":"2024-09-18T08:02:47.875042+0000","last_became_peered":"2024-09-18T08:02:47.875042+0000","last_unstale":"2024-09-18T08:03:29.426784+0000","last_undegraded":"2024-09-18T08:03:29.426784+0000","last_fullsized":"2024-09-18T08:03:29.426784+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:02:46.860092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:46.860092+0000","last_clean_scrub_stamp":"2024-09-18T08:02:46.860092+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:04:14.469212+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1b","version":"69'5","reported_seq":48,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-09-18T08:03:29.093314+0000","last_change":"2024-09-18T08:02:49.895779+0000","last_active":"2024-09-18T08:03:29.093314+0000","last_peered":"2024-09-18T08:03:29.093314+0000","last_clean":"2024-09-18T08:03:29.093314+0000","last_became_active":"2024-09-18T08:02:49.895344+0000","last_became_peered":"2024-09-18T08:02:49.895344+0000","last_unstale":"2024-09-18T08:03:29.093314+0000","last_undegraded":"2024-09-18T08:03:29.093314+0000","last_fullsized":"2024-09-18T08:03:29.093314+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_clean_scrub_stamp":"2024-09-18T08:02:48.870718+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:04:08.365283+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":11,"num_read_kb":7,"num_write":6,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":34,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-09-18T08:03:29.425938+0000","last_change":"2024-09-18T08:02:51.906694+0000","last_active":"2024-09-18T08:03:29.425938+0000","last_peered":"2024-09-18T08:03:29.425938+0000","last_clean":"2024-09-18T08:03:29.425938+0000","last_became_active":"2024-09-18T08:02:51.905776+0000","last_became_peered":"2024-09-18T08:02:51.905776+0000","last_unstale":"2024-09-18T08:03:29.425938+0000","last_undegraded":"2024-09-18T08:03:29.425938+0000","last_fullsized":"2024-09-18T08:03:29.425938+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:02:50.886830+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:50.886830+0000","last_clean_scrub_stamp":"2024-09-18T08:02:50.886830+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:04:07.273083+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":31,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-09-18T08:03:29.425272+0000","last_change":"2024-09-18T08:02:53.911969+0000","last_active":"2024-09-18T08:03:29.425272+0000","last_peered":"2024-09-18T08:03:29.425272+0000","last_clean":"2024-09-18T08:03:29.425272+0000","last_became_active":"2024-09-18T08:02:53.911648+0000","last_became_peered":"2024-09-18T08:02:53.911648+0000","last_unstale":"2024-09-18T08:03:29.425272+0000","last_undegraded":"2024-09-18T08:03:29.425272+0000","last_fullsized":"2024-09-18T08:03:29.425272+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_clean_scrub_stamp":"2024-09-18T08:02:52.890475+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:04:21.185524+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":30,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-09-18T08:03:29.093579+0000","last_change":"2024-09-18T08:02:54.047071+0000","last_active":"2024-09-18T08:03:29.093579+0000","last_peered":"2024-09-18T08:03:29.093579+0000","last_clean":"2024-09-18T08:03:29.093579+0000","last_became_active":"2024-09-18T08:02:54.046776+0000","last_became_peered":"2024-09-18T08:02:54.046776+0000","last_unstale":"2024-09-18T08:03:29.093579+0000","last_undegraded":"2024-09-18T08:03:29.093579+0000","last_fullsized":"2024-09-18T08:03:29.093579+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_clean_scrub_stamp":"2024-09-18T08:02:52.890475+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:04:11.355797+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":42,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-09-18T08:03:29.422928+0000","last_change":"2024-09-18T08:02:47.876235+0000","last_active":"2024-09-18T08:03:29.422928+0000","last_peered":"2024-09-18T08:03:29.422928+0000","last_clean":"2024-09-18T08:03:29.422928+0000","last_became_active":"2024-09-18T08:02:47.875963+0000","last_became_peered":"2024-09-18T08:02:47.875963+0000","last_unstale":"2024-09-18T08:03:29.422928+0000","last_undegraded":"2024-09-18T08:03:29.422928+0000","last_fullsized":"2024-09-18T08:03:29.422928+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:02:46.860092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:46.860092+0000","last_clean_scrub_stamp":"2024-09-18T08:02:46.860092+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:04:11.609411+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1c","version":"69'15","reported_seq":63,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-09-18T08:03:29.094484+0000","last_change":"2024-09-18T08:02:49.885923+0000","last_active":"2024-09-18T08:03:29.094484+0000","last_peered":"2024-09-18T08:03:29.094484+0000","last_clean":"2024-09-18T08:03:29.094484+0000","last_became_active":"2024-09-18T08:02:49.885553+0000","last_became_peered":"2024-09-18T08:02:49.885553+0000","last_unstale":"2024-09-18T08:03:29.094484+0000","last_undegraded":"2024-09-18T08:03:29.094484+0000","last_fullsized":"2024-09-18T08:03:29.094484+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_clean_scrub_stamp":"2024-09-18T08:02:48.870718+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:04:11.220046+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":34,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-09-18T08:03:29.093361+0000","last_change":"2024-09-18T08:02:51.908750+0000","last_active":"2024-09-18T08:03:29.093361+0000","last_peered":"2024-09-18T08:03:29.093361+0000","last_clean":"2024-09-18T08:03:29.093361+0000","last_became_active":"2024-09-18T08:02:51.908430+0000","last_became_peered":"2024-09-18T08:02:51.908430+0000","last_unstale":"2024-09-18T08:03:29.093361+0000","last_undegraded":"2024-09-18T08:03:29.093361+0000","last_fullsized":"2024-09-18T08:03:29.093361+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:02:50.886830+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:50.886830+0000","last_clean_scrub_stamp":"2024-09-18T08:02:50.886830+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:03:58.654545+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":30,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-09-18T08:03:29.425544+0000","last_change":"2024-09-18T08:02:54.047573+0000","last_active":"2024-09-18T08:03:29.425544+0000","last_peered":"2024-09-18T08:03:29.425544+0000","last_clean":"2024-09-18T08:03:29.425544+0000","last_became_active":"2024-09-18T08:02:54.047162+0000","last_became_peered":"2024-09-18T08:02:54.047162+0000","last_unstale":"2024-09-18T08:03:29.425544+0000","last_undegraded":"2024-09-18T08:03:29.425544+0000","last_fullsized":"2024-09-18T08:03:29.425544+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_clean_scrub_stamp":"2024-09-18T08:02:52.890475+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:04:02.287865+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":42,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-09-18T08:03:29.094003+0000","last_change":"2024-09-18T08:02:47.871895+0000","last_active":"2024-09-18T08:03:29.094003+0000","last_peered":"2024-09-18T08:03:29.094003+0000","last_clean":"2024-09-18T08:03:29.094003+0000","last_became_active":"2024-09-18T08:02:47.871523+0000","last_became_peered":"2024-09-18T08:02:47.871523+0000","last_unstale":"2024-09-18T08:03:29.094003+0000","last_undegraded":"2024-09-18T08:03:29.094003+0000","last_fullsized":"2024-09-18T08:03:29.094003+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:02:46.860092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:46.860092+0000","last_clean_scrub_stamp":"2024-09-18T08:02:46.860092+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:04:03.333307+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"69'12","reported_seq":61,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-09-18T08:03:29.425536+0000","last_change":"2024-09-18T08:02:49.889182+0000","last_active":"2024-09-18T08:03:29.425536+0000","last_peered":"2024-09-18T08:03:29.425536+0000","last_clean":"2024-09-18T08:03:29.425536+0000","last_became_active":"2024-09-18T08:02:49.888831+0000","last_became_peered":"2024-09-18T08:02:49.888831+0000","last_unstale":"2024-09-18T08:03:29.425536+0000","last_undegraded":"2024-09-18T08:03:29.425536+0000","last_fullsized":"2024-09-18T08:03:29.425536+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_clean_scrub_stamp":"2024-09-18T08:02:48.870718+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:04:12.415919+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":25,"num_read_kb":16,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":34,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-09-18T08:03:29.093479+0000","last_change":"2024-09-18T08:02:51.910642+0000","last_active":"2024-09-18T08:03:29.093479+0000","last_peered":"2024-09-18T08:03:29.093479+0000","last_clean":"2024-09-18T08:03:29.093479+0000","last_became_active":"2024-09-18T08:02:51.910316+0000","last_became_peered":"2024-09-18T08:02:51.910316+0000","last_unstale":"2024-09-18T08:03:29.093479+0000","last_undegraded":"2024-09-18T08:03:29.093479+0000","last_fullsized":"2024-09-18T08:03:29.093479+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:02:50.886830+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:50.886830+0000","last_clean_scrub_stamp":"2024-09-18T08:02:50.886830+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:04:05.261448+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1c","version":"69'1","reported_seq":32,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-09-18T08:03:29.427056+0000","last_change":"2024-09-18T08:02:53.910861+0000","last_active":"2024-09-18T08:03:29.427056+0000","last_peered":"2024-09-18T08:03:29.427056+0000","last_clean":"2024-09-18T08:03:29.427056+0000","last_became_active":"2024-09-18T08:02:53.910589+0000","last_became_peered":"2024-09-18T08:02:53.910589+0000","last_unstale":"2024-09-18T08:03:29.427056+0000","last_undegraded":"2024-09-18T08:03:29.427056+0000","last_fullsized":"2024-09-18T08:03:29.427056+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_clean_scrub_stamp":"2024-09-18T08:02:52.890475+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:04:20.224154+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":403,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.19","version":"62'1","reported_seq":43,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-09-18T08:03:29.092929+0000","last_change":"2024-09-18T08:02:47.881314+0000","last_active":"2024-09-18T08:03:29.092929+0000","last_peered":"2024-09-18T08:03:29.092929+0000","last_clean":"2024-09-18T08:03:29.092929+0000","last_became_active":"2024-09-18T08:02:47.880998+0000","last_became_peered":"2024-09-18T08:02:47.880998+0000","last_unstale":"2024-09-18T08:03:29.092929+0000","last_undegraded":"2024-09-18T08:03:29.092929+0000","last_fullsized":"2024-09-18T08:03:29.092929+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:02:46.860092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:46.860092+0000","last_clean_scrub_stamp":"2024-09-18T08:02:46.860092+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:03:57.000100+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1e","version":"69'10","reported_seq":53,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-09-18T08:03:29.422947+0000","last_change":"2024-09-18T08:02:49.892755+0000","last_active":"2024-09-18T08:03:29.422947+0000","last_peered":"2024-09-18T08:03:29.422947+0000","last_clean":"2024-09-18T08:03:29.422947+0000","last_became_active":"2024-09-18T08:02:49.892024+0000","last_became_peered":"2024-09-18T08:02:49.892024+0000","last_unstale":"2024-09-18T08:03:29.422947+0000","last_undegraded":"2024-09-18T08:03:29.422947+0000","last_fullsized":"2024-09-18T08:03:29.422947+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_clean_scrub_stamp":"2024-09-18T08:02:48.870718+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:04:17.237822+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1f","version":"69'8","reported_seq":55,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-09-18T08:03:29.427730+0000","last_change":"2024-09-18T08:02:51.904933+0000","last_active":"2024-09-18T08:03:29.427730+0000","last_peered":"2024-09-18T08:03:29.427730+0000","last_clean":"2024-09-18T08:03:29.427730+0000","last_became_active":"2024-09-18T08:02:51.904588+0000","last_became_peered":"2024-09-18T08:02:51.904588+0000","last_unstale":"2024-09-18T08:03:29.427730+0000","last_undegraded":"2024-09-18T08:03:29.427730+0000","last_fullsized":"2024-09-18T08:03:29.427730+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:02:50.886830+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:50.886830+0000","last_clean_scrub_stamp":"2024-09-18T08:02:50.886830+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:03:59.243324+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.f","version":"69'15","reported_seq":63,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-09-18T08:03:29.094148+0000","last_change":"2024-09-18T08:02:49.899404+0000","last_active":"2024-09-18T08:03:29.094148+0000","last_peered":"2024-09-18T08:03:29.094148+0000","last_clean":"2024-09-18T08:03:29.094148+0000","last_became_active":"2024-09-18T08:02:49.899083+0000","last_became_peered":"2024-09-18T08:02:49.899083+0000","last_unstale":"2024-09-18T08:03:29.094148+0000","last_undegraded":"2024-09-18T08:03:29.094148+0000","last_fullsized":"2024-09-18T08:03:29.094148+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_clean_scrub_stamp":"2024-09-18T08:02:48.870718+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:04:06.513250+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":42,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-09-18T08:03:29.424969+0000","last_change":"2024-09-18T08:02:47.879723+0000","last_active":"2024-09-18T08:03:29.424969+0000","last_peered":"2024-09-18T08:03:29.424969+0000","last_clean":"2024-09-18T08:03:29.424969+0000","last_became_active":"2024-09-18T08:02:47.879328+0000","last_became_peered":"2024-09-18T08:02:47.879328+0000","last_unstale":"2024-09-18T08:03:29.424969+0000","last_undegraded":"2024-09-18T08:03:29.424969+0000","last_fullsized":"2024-09-18T08:03:29.424969+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:02:46.860092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:46.860092+0000","last_clean_scrub_stamp":"2024-09-18T08:02:46.860092+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:03:56.802301+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.e","version":"69'8","reported_seq":52,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-09-18T08:03:29.093759+0000","last_change":"2024-09-18T08:02:51.903673+0000","last_active":"2024-09-18T08:03:29.093759+0000","last_peered":"2024-09-18T08:03:29.093759+0000","last_clean":"2024-09-18T08:03:29.093759+0000","last_became_active":"2024-09-18T08:02:51.903369+0000","last_became_peered":"2024-09-18T08:02:51.903369+0000","last_unstale":"2024-09-18T08:03:29.093759+0000","last_undegraded":"2024-09-18T08:03:29.093759+0000","last_fullsized":"2024-09-18T08:03:29.093759+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:02:50.886830+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:50.886830+0000","last_clean_scrub_stamp":"2024-09-18T08:02:50.886830+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:03:51.149147+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":31,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-09-18T08:03:29.426801+0000","last_change":"2024-09-18T08:02:53.910161+0000","last_active":"2024-09-18T08:03:29.426801+0000","last_peered":"2024-09-18T08:03:29.426801+0000","last_clean":"2024-09-18T08:03:29.426801+0000","last_became_active":"2024-09-18T08:02:53.909834+0000","last_became_peered":"2024-09-18T08:02:53.909834+0000","last_unstale":"2024-09-18T08:03:29.426801+0000","last_undegraded":"2024-09-18T08:03:29.426801+0000","last_fullsized":"2024-09-18T08:03:29.426801+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_clean_scrub_stamp":"2024-09-18T08:02:52.890475+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:04:07.435248+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.0","version":"69'18","reported_seq":70,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-09-18T08:03:29.424837+0000","last_change":"2024-09-18T08:02:49.887135+0000","last_active":"2024-09-18T08:03:29.424837+0000","last_peered":"2024-09-18T08:03:29.424837+0000","last_clean":"2024-09-18T08:03:29.424837+0000","last_became_active":"2024-09-18T08:02:49.886818+0000","last_became_peered":"2024-09-18T08:02:49.886818+0000","last_unstale":"2024-09-18T08:03:29.424837+0000","last_undegraded":"2024-09-18T08:03:29.424837+0000","last_fullsized":"2024-09-18T08:03:29.424837+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_clean_scrub_stamp":"2024-09-18T08:02:48.870718+0000","objects_scrubbed":0,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:03:57.656879+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":34,"num_read_kb":22,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":42,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-09-18T08:03:29.425737+0000","last_change":"2024-09-18T08:02:47.880517+0000","last_active":"2024-09-18T08:03:29.425737+0000","last_peered":"2024-09-18T08:03:29.425737+0000","last_clean":"2024-09-18T08:03:29.425737+0000","last_became_active":"2024-09-18T08:02:47.880225+0000","last_became_peered":"2024-09-18T08:02:47.880225+0000","last_unstale":"2024-09-18T08:03:29.425737+0000","last_undegraded":"2024-09-18T08:03:29.425737+0000","last_fullsized":"2024-09-18T08:03:29.425737+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:02:46.860092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:46.860092+0000","last_clean_scrub_stamp":"2024-09-18T08:02:46.860092+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:03:51.223493+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":34,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-09-18T08:03:29.092985+0000","last_change":"2024-09-18T08:02:51.899911+0000","last_active":"2024-09-18T08:03:29.092985+0000","last_peered":"2024-09-18T08:03:29.092985+0000","last_clean":"2024-09-18T08:03:29.092985+0000","last_became_active":"2024-09-18T08:02:51.899594+0000","last_became_peered":"2024-09-18T08:02:51.899594+0000","last_unstale":"2024-09-18T08:03:29.092985+0000","last_undegraded":"2024-09-18T08:03:29.092985+0000","last_fullsized":"2024-09-18T08:03:29.092985+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:02:50.886830+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:50.886830+0000","last_clean_scrub_stamp":"2024-09-18T08:02:50.886830+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:04:02.261346+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":31,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-09-18T08:03:29.093271+0000","last_change":"2024-09-18T08:02:53.904262+0000","last_active":"2024-09-18T08:03:29.093271+0000","last_peered":"2024-09-18T08:03:29.093271+0000","last_clean":"2024-09-18T08:03:29.093271+0000","last_became_active":"2024-09-18T08:02:53.903957+0000","last_became_peered":"2024-09-18T08:02:53.903957+0000","last_unstale":"2024-09-18T08:03:29.093271+0000","last_undegraded":"2024-09-18T08:03:29.093271+0000","last_fullsized":"2024-09-18T08:03:29.093271+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_clean_scrub_stamp":"2024-09-18T08:02:52.890475+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:04:02.672046+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1","version":"69'14","reported_seq":59,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-09-18T08:03:29.093046+0000","last_change":"2024-09-18T08:02:49.894205+0000","last_active":"2024-09-18T08:03:29.093046+0000","last_peered":"2024-09-18T08:03:29.093046+0000","last_clean":"2024-09-18T08:03:29.093046+0000","last_became_active":"2024-09-18T08:02:49.893830+0000","last_became_peered":"2024-09-18T08:02:49.893830+0000","last_unstale":"2024-09-18T08:03:29.093046+0000","last_undegraded":"2024-09-18T08:03:29.093046+0000","last_fullsized":"2024-09-18T08:03:29.093046+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_clean_scrub_stamp":"2024-09-18T08:02:48.870718+0000","objects_scrubbed":0,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:03:51.325184+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":21,"num_read_kb":14,"num_write":14,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"62'2","reported_seq":49,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-09-18T08:03:29.423126+0000","last_change":"2024-09-18T08:02:47.873172+0000","last_active":"2024-09-18T08:03:29.423126+0000","last_peered":"2024-09-18T08:03:29.423126+0000","last_clean":"2024-09-18T08:03:29.423126+0000","last_became_active":"2024-09-18T08:02:47.872857+0000","last_became_peered":"2024-09-18T08:02:47.872857+0000","last_unstale":"2024-09-18T08:03:29.423126+0000","last_undegraded":"2024-09-18T08:03:29.423126+0000","last_fullsized":"2024-09-18T08:03:29.423126+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:02:46.860092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:46.860092+0000","last_clean_scrub_stamp":"2024-09-18T08:02:46.860092+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:04:03.733922+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1039,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.0","version":"69'8","reported_seq":52,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-09-18T08:03:29.424781+0000","last_change":"2024-09-18T08:02:51.910339+0000","last_active":"2024-09-18T08:03:29.424781+0000","last_peered":"2024-09-18T08:03:29.424781+0000","last_clean":"2024-09-18T08:03:29.424781+0000","last_became_active":"2024-09-18T08:02:51.909680+0000","last_became_peered":"2024-09-18T08:02:51.909680+0000","last_unstale":"2024-09-18T08:03:29.424781+0000","last_undegraded":"2024-09-18T08:03:29.424781+0000","last_fullsized":"2024-09-18T08:03:29.424781+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:02:50.886830+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:50.886830+0000","last_clean_scrub_stamp":"2024-09-18T08:02:50.886830+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:03:58.456713+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":31,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-09-18T08:03:29.426246+0000","last_change":"2024-09-18T08:02:54.047452+0000","last_active":"2024-09-18T08:03:29.426246+0000","last_peered":"2024-09-18T08:03:29.426246+0000","last_clean":"2024-09-18T08:03:29.426246+0000","last_became_active":"2024-09-18T08:02:54.047039+0000","last_became_peered":"2024-09-18T08:02:54.047039+0000","last_unstale":"2024-09-18T08:03:29.426246+0000","last_undegraded":"2024-09-18T08:03:29.426246+0000","last_fullsized":"2024-09-18T08:03:29.426246+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_clean_scrub_stamp":"2024-09-18T08:02:52.890475+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:03:58.508399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.2","version":"69'10","reported_seq":53,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-09-18T08:03:29.093258+0000","last_change":"2024-09-18T08:02:49.896889+0000","last_active":"2024-09-18T08:03:29.093258+0000","last_peered":"2024-09-18T08:03:29.093258+0000","last_clean":"2024-09-18T08:03:29.093258+0000","last_became_active":"2024-09-18T08:02:49.895265+0000","last_became_peered":"2024-09-18T08:02:49.895265+0000","last_unstale":"2024-09-18T08:03:29.093258+0000","last_undegraded":"2024-09-18T08:03:29.093258+0000","last_fullsized":"2024-09-18T08:03:29.093258+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_clean_scrub_stamp":"2024-09-18T08:02:48.870718+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:04:06.130656+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":42,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-09-18T08:03:29.427336+0000","last_change":"2024-09-18T08:02:47.877672+0000","last_active":"2024-09-18T08:03:29.427336+0000","last_peered":"2024-09-18T08:03:29.427336+0000","last_clean":"2024-09-18T08:03:29.427336+0000","last_became_active":"2024-09-18T08:02:47.874727+0000","last_became_peered":"2024-09-18T08:02:47.874727+0000","last_unstale":"2024-09-18T08:03:29.427336+0000","last_undegraded":"2024-09-18T08:03:29.427336+0000","last_fullsized":"2024-09-18T08:03:29.427336+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:02:46.860092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:46.860092+0000","last_clean_scrub_stamp":"2024-09-18T08:02:46.860092+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:03:47.082997+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.3","version":"69'8","reported_seq":52,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-09-18T08:03:29.422690+0000","last_change":"2024-09-18T08:02:51.906369+0000","last_active":"2024-09-18T08:03:29.422690+0000","last_peered":"2024-09-18T08:03:29.422690+0000","last_clean":"2024-09-18T08:03:29.422690+0000","last_became_active":"2024-09-18T08:02:51.906055+0000","last_became_peered":"2024-09-18T08:02:51.906055+0000","last_unstale":"2024-09-18T08:03:29.422690+0000","last_undegraded":"2024-09-18T08:03:29.422690+0000","last_fullsized":"2024-09-18T08:03:29.422690+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:02:50.886830+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:50.886830+0000","last_clean_scrub_stamp":"2024-09-18T08:02:50.886830+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:04:18.452195+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":31,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-09-18T08:03:29.422472+0000","last_change":"2024-09-18T08:02:53.910057+0000","last_active":"2024-09-18T08:03:29.422472+0000","last_peered":"2024-09-18T08:03:29.422472+0000","last_clean":"2024-09-18T08:03:29.422472+0000","last_became_active":"2024-09-18T08:02:53.909583+0000","last_became_peered":"2024-09-18T08:02:53.909583+0000","last_unstale":"2024-09-18T08:03:29.422472+0000","last_undegraded":"2024-09-18T08:03:29.422472+0000","last_fullsized":"2024-09-18T08:03:29.422472+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_clean_scrub_stamp":"2024-09-18T08:02:52.890475+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:04:07.545769+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.3","version":"69'19","reported_seq":74,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-09-18T08:03:29.422747+0000","last_change":"2024-09-18T08:02:49.890631+0000","last_active":"2024-09-18T08:03:29.422747+0000","last_peered":"2024-09-18T08:03:29.422747+0000","last_clean":"2024-09-18T08:03:29.422747+0000","last_became_active":"2024-09-18T08:02:49.890318+0000","last_became_peered":"2024-09-18T08:02:49.890318+0000","last_unstale":"2024-09-18T08:03:29.422747+0000","last_undegraded":"2024-09-18T08:03:29.422747+0000","last_fullsized":"2024-09-18T08:03:29.422747+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_clean_scrub_stamp":"2024-09-18T08:02:48.870718+0000","objects_scrubbed":0,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:03:57.953144+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":39,"num_read_kb":25,"num_write":22,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.4","version":"62'1","reported_seq":48,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-09-18T08:03:29.093400+0000","last_change":"2024-09-18T08:02:47.875046+0000","last_active":"2024-09-18T08:03:29.093400+0000","last_peered":"2024-09-18T08:03:29.093400+0000","last_clean":"2024-09-18T08:03:29.093400+0000","last_became_active":"2024-09-18T08:02:47.874764+0000","last_became_peered":"2024-09-18T08:02:47.874764+0000","last_unstale":"2024-09-18T08:03:29.093400+0000","last_undegraded":"2024-09-18T08:03:29.093400+0000","last_fullsized":"2024-09-18T08:03:29.093400+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:02:46.860092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:46.860092+0000","last_clean_scrub_stamp":"2024-09-18T08:02:46.860092+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:04:07.354571+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":436,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":34,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-09-18T08:03:29.425973+0000","last_change":"2024-09-18T08:02:51.908344+0000","last_active":"2024-09-18T08:03:29.425973+0000","last_peered":"2024-09-18T08:03:29.425973+0000","last_clean":"2024-09-18T08:03:29.425973+0000","last_became_active":"2024-09-18T08:02:51.908068+0000","last_became_peered":"2024-09-18T08:02:51.908068+0000","last_unstale":"2024-09-18T08:03:29.425973+0000","last_undegraded":"2024-09-18T08:03:29.425973+0000","last_fullsized":"2024-09-18T08:03:29.425973+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:02:50.886830+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:50.886830+0000","last_clean_scrub_stamp":"2024-09-18T08:02:50.886830+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:04:06.242055+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":30,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-09-18T08:03:29.092743+0000","last_change":"2024-09-18T08:02:54.046350+0000","last_active":"2024-09-18T08:03:29.092743+0000","last_peered":"2024-09-18T08:03:29.092743+0000","last_clean":"2024-09-18T08:03:29.092743+0000","last_became_active":"2024-09-18T08:02:54.045726+0000","last_became_peered":"2024-09-18T08:02:54.045726+0000","last_unstale":"2024-09-18T08:03:29.092743+0000","last_undegraded":"2024-09-18T08:03:29.092743+0000","last_fullsized":"2024-09-18T08:03:29.092743+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_clean_scrub_stamp":"2024-09-18T08:02:52.890475+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:03:55.585366+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.4","version":"69'28","reported_seq":96,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-09-18T08:03:29.093241+0000","last_change":"2024-09-18T08:02:49.897515+0000","last_active":"2024-09-18T08:03:29.093241+0000","last_peered":"2024-09-18T08:03:29.093241+0000","last_clean":"2024-09-18T08:03:29.093241+0000","last_became_active":"2024-09-18T08:02:49.897221+0000","last_became_peered":"2024-09-18T08:02:49.897221+0000","last_unstale":"2024-09-18T08:03:29.093241+0000","last_undegraded":"2024-09-18T08:03:29.093241+0000","last_fullsized":"2024-09-18T08:03:29.093241+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_clean_scrub_stamp":"2024-09-18T08:02:48.870718+0000","objects_scrubbed":0,"log_size":28,"log_dups_size":0,"ondisk_log_size":28,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:04:04.818428+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":48,"num_read_kb":33,"num_write":26,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":42,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-09-18T08:03:29.093698+0000","last_change":"2024-09-18T08:02:47.877445+0000","last_active":"2024-09-18T08:03:29.093698+0000","last_peered":"2024-09-18T08:03:29.093698+0000","last_clean":"2024-09-18T08:03:29.093698+0000","last_became_active":"2024-09-18T08:02:47.877085+0000","last_became_peered":"2024-09-18T08:02:47.877085+0000","last_unstale":"2024-09-18T08:03:29.093698+0000","last_undegraded":"2024-09-18T08:03:29.093698+0000","last_fullsized":"2024-09-18T08:03:29.093698+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:02:46.860092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:46.860092+0000","last_clean_scrub_stamp":"2024-09-18T08:02:46.860092+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:04:09.365906+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"64'2","reported_seq":52,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-09-18T08:03:29.426013+0000","last_change":"2024-09-18T08:02:49.883480+0000","last_active":"2024-09-18T08:03:29.426013+0000","last_peered":"2024-09-18T08:03:29.426013+0000","last_clean":"2024-09-18T08:03:29.426013+0000","last_became_active":"2024-09-18T08:02:46.867802+0000","last_became_peered":"2024-09-18T08:02:46.867802+0000","last_unstale":"2024-09-18T08:03:29.426013+0000","last_undegraded":"2024-09-18T08:03:29.426013+0000","last_fullsized":"2024-09-18T08:03:29.426013+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:02:45.855979+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:45.855979+0000","last_clean_scrub_stamp":"2024-09-18T08:02:45.855979+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:04:13.406442+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001348349,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":34,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-09-18T08:03:29.422810+0000","last_change":"2024-09-18T08:02:51.905694+0000","last_active":"2024-09-18T08:03:29.422810+0000","last_peered":"2024-09-18T08:03:29.422810+0000","last_clean":"2024-09-18T08:03:29.422810+0000","last_became_active":"2024-09-18T08:02:51.905357+0000","last_became_peered":"2024-09-18T08:02:51.905357+0000","last_unstale":"2024-09-18T08:03:29.422810+0000","last_undegraded":"2024-09-18T08:03:29.422810+0000","last_fullsized":"2024-09-18T08:03:29.422810+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:02:50.886830+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:50.886830+0000","last_clean_scrub_stamp":"2024-09-18T08:02:50.886830+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:04:15.739565+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":31,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-09-18T08:03:29.425353+0000","last_change":"2024-09-18T08:02:53.911634+0000","last_active":"2024-09-18T08:03:29.425353+0000","last_peered":"2024-09-18T08:03:29.425353+0000","last_clean":"2024-09-18T08:03:29.425353+0000","last_became_active":"2024-09-18T08:02:53.911036+0000","last_became_peered":"2024-09-18T08:02:53.911036+0000","last_unstale":"2024-09-18T08:03:29.425353+0000","last_undegraded":"2024-09-18T08:03:29.425353+0000","last_fullsized":"2024-09-18T08:03:29.425353+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_clean_scrub_stamp":"2024-09-18T08:02:52.890475+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:04:16.930353+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.7","version":"69'13","reported_seq":65,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-09-18T08:03:29.094241+0000","last_change":"2024-09-18T08:02:49.899404+0000","last_active":"2024-09-18T08:03:29.094241+0000","last_peered":"2024-09-18T08:03:29.094241+0000","last_clean":"2024-09-18T08:03:29.094241+0000","last_became_active":"2024-09-18T08:02:49.898987+0000","last_became_peered":"2024-09-18T08:02:49.898987+0000","last_unstale":"2024-09-18T08:03:29.094241+0000","last_undegraded":"2024-09-18T08:03:29.094241+0000","last_fullsized":"2024-09-18T08:03:29.094241+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_clean_scrub_stamp":"2024-09-18T08:02:48.870718+0000","objects_scrubbed":0,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:03:50.031411+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":19,"num_write":16,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":42,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-09-18T08:03:29.092802+0000","last_change":"2024-09-18T08:02:47.871732+0000","last_active":"2024-09-18T08:03:29.092802+0000","last_peered":"2024-09-18T08:03:29.092802+0000","last_clean":"2024-09-18T08:03:29.092802+0000","last_became_active":"2024-09-18T08:02:47.871416+0000","last_became_peered":"2024-09-18T08:02:47.871416+0000","last_unstale":"2024-09-18T08:03:29.092802+0000","last_undegraded":"2024-09-18T08:03:29.092802+0000","last_fullsized":"2024-09-18T08:03:29.092802+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:02:46.860092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:46.860092+0000","last_clean_scrub_stamp":"2024-09-18T08:02:46.860092+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:03:48.825242+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"62'1","reported_seq":51,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-09-18T08:03:29.093627+0000","last_change":"2024-09-18T08:02:49.877280+0000","last_active":"2024-09-18T08:03:29.093627+0000","last_peered":"2024-09-18T08:03:29.093627+0000","last_clean":"2024-09-18T08:03:29.093627+0000","last_became_active":"2024-09-18T08:02:46.868175+0000","last_became_peered":"2024-09-18T08:02:46.868175+0000","last_unstale":"2024-09-18T08:03:29.093627+0000","last_undegraded":"2024-09-18T08:03:29.093627+0000","last_fullsized":"2024-09-18T08:03:29.093627+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:02:45.855979+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:45.855979+0000","last_clean_scrub_stamp":"2024-09-18T08:02:45.855979+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:04:02.537405+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00066785499999999997,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":34,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-09-18T08:03:29.094115+0000","last_change":"2024-09-18T08:02:51.908417+0000","last_active":"2024-09-18T08:03:29.094115+0000","last_peered":"2024-09-18T08:03:29.094115+0000","last_clean":"2024-09-18T08:03:29.094115+0000","last_became_active":"2024-09-18T08:02:51.908156+0000","last_became_peered":"2024-09-18T08:02:51.908156+0000","last_unstale":"2024-09-18T08:03:29.094115+0000","last_undegraded":"2024-09-18T08:03:29.094115+0000","last_fullsized":"2024-09-18T08:03:29.094115+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:02:50.886830+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:50.886830+0000","last_clean_scrub_stamp":"2024-09-18T08:02:50.886830+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:04:04.453603+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":31,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-09-18T08:03:29.427482+0000","last_change":"2024-09-18T08:02:54.048417+0000","last_active":"2024-09-18T08:03:29.427482+0000","last_peered":"2024-09-18T08:03:29.427482+0000","last_clean":"2024-09-18T08:03:29.427482+0000","last_became_active":"2024-09-18T08:02:54.048004+0000","last_became_peered":"2024-09-18T08:02:54.048004+0000","last_unstale":"2024-09-18T08:03:29.427482+0000","last_undegraded":"2024-09-18T08:03:29.427482+0000","last_fullsized":"2024-09-18T08:03:29.427482+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_clean_scrub_stamp":"2024-09-18T08:02:52.890475+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:04:20.672566+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"69'12","reported_seq":56,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-09-18T08:03:29.423025+0000","last_change":"2024-09-18T08:02:49.898383+0000","last_active":"2024-09-18T08:03:29.423025+0000","last_peered":"2024-09-18T08:03:29.423025+0000","last_clean":"2024-09-18T08:03:29.423025+0000","last_became_active":"2024-09-18T08:02:49.898079+0000","last_became_peered":"2024-09-18T08:02:49.898079+0000","last_unstale":"2024-09-18T08:03:29.423025+0000","last_undegraded":"2024-09-18T08:03:29.423025+0000","last_fullsized":"2024-09-18T08:03:29.423025+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_clean_scrub_stamp":"2024-09-18T08:02:48.870718+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:04:14.339135+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":18,"num_read_kb":12,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":42,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-09-18T08:03:29.422549+0000","last_change":"2024-09-18T08:02:47.874335+0000","last_active":"2024-09-18T08:03:29.422549+0000","last_peered":"2024-09-18T08:03:29.422549+0000","last_clean":"2024-09-18T08:03:29.422549+0000","last_became_active":"2024-09-18T08:02:47.873809+0000","last_became_peered":"2024-09-18T08:02:47.873809+0000","last_unstale":"2024-09-18T08:03:29.422549+0000","last_undegraded":"2024-09-18T08:03:29.422549+0000","last_fullsized":"2024-09-18T08:03:29.422549+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:02:46.860092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:46.860092+0000","last_clean_scrub_stamp":"2024-09-18T08:02:46.860092+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:03:48.336434+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"70'5","reported_seq":93,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-09-18T08:03:31.704938+0000","last_change":"2024-09-18T08:02:49.877461+0000","last_active":"2024-09-18T08:03:31.704938+0000","last_peered":"2024-09-18T08:03:31.704938+0000","last_clean":"2024-09-18T08:03:31.704938+0000","last_became_active":"2024-09-18T08:02:46.872144+0000","last_became_peered":"2024-09-18T08:02:46.872144+0000","last_unstale":"2024-09-18T08:03:31.704938+0000","last_undegraded":"2024-09-18T08:03:31.704938+0000","last_fullsized":"2024-09-18T08:03:31.704938+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:02:45.855979+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:45.855979+0000","last_clean_scrub_stamp":"2024-09-18T08:02:45.855979+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:03:59.991143+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000504055,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":40,"num_read_kb":35,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":34,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-09-18T08:03:29.426629+0000","last_change":"2024-09-18T08:02:51.913423+0000","last_active":"2024-09-18T08:03:29.426629+0000","last_peered":"2024-09-18T08:03:29.426629+0000","last_clean":"2024-09-18T08:03:29.426629+0000","last_became_active":"2024-09-18T08:02:51.913112+0000","last_became_peered":"2024-09-18T08:02:51.913112+0000","last_unstale":"2024-09-18T08:03:29.426629+0000","last_undegraded":"2024-09-18T08:03:29.426629+0000","last_fullsized":"2024-09-18T08:03:29.426629+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:02:50.886830+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:50.886830+0000","last_clean_scrub_stamp":"2024-09-18T08:02:50.886830+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:04:04.493964+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":31,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-09-18T08:03:29.094284+0000","last_change":"2024-09-18T08:02:53.907689+0000","last_active":"2024-09-18T08:03:29.094284+0000","last_peered":"2024-09-18T08:03:29.094284+0000","last_clean":"2024-09-18T08:03:29.094284+0000","last_became_active":"2024-09-18T08:02:53.907404+0000","last_became_peered":"2024-09-18T08:02:53.907404+0000","last_unstale":"2024-09-18T08:03:29.094284+0000","last_undegraded":"2024-09-18T08:03:29.094284+0000","last_fullsized":"2024-09-18T08:03:29.094284+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_clean_scrub_stamp":"2024-09-18T08:02:52.890475+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:03:57.936771+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.5","version":"69'16","reported_seq":70,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-09-18T08:03:29.426928+0000","last_change":"2024-09-18T08:02:49.892366+0000","last_active":"2024-09-18T08:03:29.426928+0000","last_peered":"2024-09-18T08:03:29.426928+0000","last_clean":"2024-09-18T08:03:29.426928+0000","last_became_active":"2024-09-18T08:02:49.891985+0000","last_became_peered":"2024-09-18T08:02:49.891985+0000","last_unstale":"2024-09-18T08:03:29.426928+0000","last_undegraded":"2024-09-18T08:03:29.426928+0000","last_fullsized":"2024-09-18T08:03:29.426928+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_clean_scrub_stamp":"2024-09-18T08:02:48.870718+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:04:12.297325+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":25,"num_read_kb":15,"num_write":13,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":42,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-09-18T08:03:29.425182+0000","last_change":"2024-09-18T08:02:47.876225+0000","last_active":"2024-09-18T08:03:29.425182+0000","last_peered":"2024-09-18T08:03:29.425182+0000","last_clean":"2024-09-18T08:03:29.425182+0000","last_became_active":"2024-09-18T08:02:47.875950+0000","last_became_peered":"2024-09-18T08:02:47.875950+0000","last_unstale":"2024-09-18T08:03:29.425182+0000","last_undegraded":"2024-09-18T08:03:29.425182+0000","last_fullsized":"2024-09-18T08:03:29.425182+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:02:46.860092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:46.860092+0000","last_clean_scrub_stamp":"2024-09-18T08:02:46.860092+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:04:02.960274+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"74'79","reported_seq":88,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-09-18T08:03:29.425597+0000","last_change":"2024-09-18T08:02:55.472070+0000","last_active":"2024-09-18T08:03:29.425597+0000","last_peered":"2024-09-18T08:03:29.425597+0000","last_clean":"2024-09-18T08:03:29.425597+0000","last_became_active":"2024-09-18T08:02:39.446743+0000","last_became_peered":"2024-09-18T08:02:39.446743+0000","last_unstale":"2024-09-18T08:03:29.425597+0000","last_undegraded":"2024-09-18T08:03:29.425597+0000","last_fullsized":"2024-09-18T08:03:29.425597+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"20'72","last_scrub_stamp":"2024-09-18T08:02:55.471998+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:00:11.893742+0000","last_clean_scrub_stamp":"2024-09-18T08:02:55.471998+0000","objects_scrubbed":2,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:04:21.599062+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":231,"num_write":113,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":590368,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":34,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-09-18T08:03:29.427254+0000","last_change":"2024-09-18T08:02:51.905819+0000","last_active":"2024-09-18T08:03:29.427254+0000","last_peered":"2024-09-18T08:03:29.427254+0000","last_clean":"2024-09-18T08:03:29.427254+0000","last_became_active":"2024-09-18T08:02:51.905516+0000","last_became_peered":"2024-09-18T08:02:51.905516+0000","last_unstale":"2024-09-18T08:03:29.427254+0000","last_undegraded":"2024-09-18T08:03:29.427254+0000","last_fullsized":"2024-09-18T08:03:29.427254+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:02:50.886830+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:50.886830+0000","last_clean_scrub_stamp":"2024-09-18T08:02:50.886830+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:04:04.232646+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":31,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-09-18T08:03:29.426445+0000","last_change":"2024-09-18T08:02:53.909023+0000","last_active":"2024-09-18T08:03:29.426445+0000","last_peered":"2024-09-18T08:03:29.426445+0000","last_clean":"2024-09-18T08:03:29.426445+0000","last_became_active":"2024-09-18T08:02:53.908668+0000","last_became_peered":"2024-09-18T08:02:53.908668+0000","last_unstale":"2024-09-18T08:03:29.426445+0000","last_undegraded":"2024-09-18T08:03:29.426445+0000","last_fullsized":"2024-09-18T08:03:29.426445+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_clean_scrub_stamp":"2024-09-18T08:02:52.890475+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:03:57.881484+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.e","version":"69'11","reported_seq":57,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-09-18T08:03:29.093751+0000","last_change":"2024-09-18T08:02:49.895676+0000","last_active":"2024-09-18T08:03:29.093751+0000","last_peered":"2024-09-18T08:03:29.093751+0000","last_clean":"2024-09-18T08:03:29.093751+0000","last_became_active":"2024-09-18T08:02:49.895303+0000","last_became_peered":"2024-09-18T08:02:49.895303+0000","last_unstale":"2024-09-18T08:03:29.093751+0000","last_undegraded":"2024-09-18T08:03:29.093751+0000","last_fullsized":"2024-09-18T08:03:29.093751+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_clean_scrub_stamp":"2024-09-18T08:02:48.870718+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:04:11.479080+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":42,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-09-18T08:03:29.093490+0000","last_change":"2024-09-18T08:02:47.880265+0000","last_active":"2024-09-18T08:03:29.093490+0000","last_peered":"2024-09-18T08:03:29.093490+0000","last_clean":"2024-09-18T08:03:29.093490+0000","last_became_active":"2024-09-18T08:02:47.879764+0000","last_became_peered":"2024-09-18T08:02:47.879764+0000","last_unstale":"2024-09-18T08:03:29.093490+0000","last_undegraded":"2024-09-18T08:03:29.093490+0000","last_fullsized":"2024-09-18T08:03:29.093490+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:02:46.860092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:46.860092+0000","last_clean_scrub_stamp":"2024-09-18T08:02:46.860092+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:04:12.919106+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":34,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-09-18T08:03:29.426274+0000","last_change":"2024-09-18T08:02:51.911107+0000","last_active":"2024-09-18T08:03:29.426274+0000","last_peered":"2024-09-18T08:03:29.426274+0000","last_clean":"2024-09-18T08:03:29.426274+0000","last_became_active":"2024-09-18T08:02:51.910649+0000","last_became_peered":"2024-09-18T08:02:51.910649+0000","last_unstale":"2024-09-18T08:03:29.426274+0000","last_undegraded":"2024-09-18T08:03:29.426274+0000","last_fullsized":"2024-09-18T08:03:29.426274+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:02:50.886830+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:50.886830+0000","last_clean_scrub_stamp":"2024-09-18T08:02:50.886830+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:03:56.497732+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":30,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-09-18T08:03:29.425190+0000","last_change":"2024-09-18T08:02:54.045510+0000","last_active":"2024-09-18T08:03:29.425190+0000","last_peered":"2024-09-18T08:03:29.425190+0000","last_clean":"2024-09-18T08:03:29.425190+0000","last_became_active":"2024-09-18T08:02:54.045091+0000","last_became_peered":"2024-09-18T08:02:54.045091+0000","last_unstale":"2024-09-18T08:03:29.425190+0000","last_undegraded":"2024-09-18T08:03:29.425190+0000","last_fullsized":"2024-09-18T08:03:29.425190+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_clean_scrub_stamp":"2024-09-18T08:02:52.890475+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:04:22.368585+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.d","version":"69'17","reported_seq":66,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-09-18T08:03:29.093836+0000","last_change":"2024-09-18T08:02:49.895971+0000","last_active":"2024-09-18T08:03:29.093836+0000","last_peered":"2024-09-18T08:03:29.093836+0000","last_clean":"2024-09-18T08:03:29.093836+0000","last_became_active":"2024-09-18T08:02:49.895578+0000","last_became_peered":"2024-09-18T08:02:49.895578+0000","last_unstale":"2024-09-18T08:03:29.093836+0000","last_undegraded":"2024-09-18T08:03:29.093836+0000","last_fullsized":"2024-09-18T08:03:29.093836+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_clean_scrub_stamp":"2024-09-18T08:02:48.870718+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:04:17.784735+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":29,"num_read_kb":19,"num_write":18,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,1],"acting":[4,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":42,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-09-18T08:03:29.426401+0000","last_change":"2024-09-18T08:02:47.884431+0000","last_active":"2024-09-18T08:03:29.426401+0000","last_peered":"2024-09-18T08:03:29.426401+0000","last_clean":"2024-09-18T08:03:29.426401+0000","last_became_active":"2024-09-18T08:02:47.884132+0000","last_became_peered":"2024-09-18T08:02:47.884132+0000","last_unstale":"2024-09-18T08:03:29.426401+0000","last_undegraded":"2024-09-18T08:03:29.426401+0000","last_fullsized":"2024-09-18T08:03:29.426401+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:02:46.860092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:46.860092+0000","last_clean_scrub_stamp":"2024-09-18T08:02:46.860092+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:04:06.189609+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":34,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-09-18T08:03:29.093056+0000","last_change":"2024-09-18T08:02:51.907544+0000","last_active":"2024-09-18T08:03:29.093056+0000","last_peered":"2024-09-18T08:03:29.093056+0000","last_clean":"2024-09-18T08:03:29.093056+0000","last_became_active":"2024-09-18T08:02:51.907196+0000","last_became_peered":"2024-09-18T08:02:51.907196+0000","last_unstale":"2024-09-18T08:03:29.093056+0000","last_undegraded":"2024-09-18T08:03:29.093056+0000","last_fullsized":"2024-09-18T08:03:29.093056+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:02:50.886830+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:50.886830+0000","last_clean_scrub_stamp":"2024-09-18T08:02:50.886830+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:04:03.690724+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":31,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-09-18T08:03:29.093524+0000","last_change":"2024-09-18T08:02:53.904497+0000","last_active":"2024-09-18T08:03:29.093524+0000","last_peered":"2024-09-18T08:03:29.093524+0000","last_clean":"2024-09-18T08:03:29.093524+0000","last_became_active":"2024-09-18T08:02:53.904144+0000","last_became_peered":"2024-09-18T08:02:53.904144+0000","last_unstale":"2024-09-18T08:03:29.093524+0000","last_undegraded":"2024-09-18T08:03:29.093524+0000","last_fullsized":"2024-09-18T08:03:29.093524+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_clean_scrub_stamp":"2024-09-18T08:02:52.890475+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:03:58.164335+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.c","version":"69'10","reported_seq":53,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-09-18T08:03:29.093589+0000","last_change":"2024-09-18T08:02:49.892847+0000","last_active":"2024-09-18T08:03:29.093589+0000","last_peered":"2024-09-18T08:03:29.093589+0000","last_clean":"2024-09-18T08:03:29.093589+0000","last_became_active":"2024-09-18T08:02:49.892527+0000","last_became_peered":"2024-09-18T08:02:49.892527+0000","last_unstale":"2024-09-18T08:03:29.093589+0000","last_undegraded":"2024-09-18T08:03:29.093589+0000","last_fullsized":"2024-09-18T08:03:29.093589+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_clean_scrub_stamp":"2024-09-18T08:02:48.870718+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:04:11.391201+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":42,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-09-18T08:03:29.425453+0000","last_change":"2024-09-18T08:02:47.881814+0000","last_active":"2024-09-18T08:03:29.425453+0000","last_peered":"2024-09-18T08:03:29.425453+0000","last_clean":"2024-09-18T08:03:29.425453+0000","last_became_active":"2024-09-18T08:02:47.881512+0000","last_became_peered":"2024-09-18T08:02:47.881512+0000","last_unstale":"2024-09-18T08:03:29.425453+0000","last_undegraded":"2024-09-18T08:03:29.425453+0000","last_fullsized":"2024-09-18T08:03:29.425453+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:02:46.860092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:46.860092+0000","last_clean_scrub_stamp":"2024-09-18T08:02:46.860092+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:03:59.505935+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.d","version":"69'8","reported_seq":52,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-09-18T08:03:29.093717+0000","last_change":"2024-09-18T08:02:51.909290+0000","last_active":"2024-09-18T08:03:29.093717+0000","last_peered":"2024-09-18T08:03:29.093717+0000","last_clean":"2024-09-18T08:03:29.093717+0000","last_became_active":"2024-09-18T08:02:51.909011+0000","last_became_peered":"2024-09-18T08:02:51.909011+0000","last_unstale":"2024-09-18T08:03:29.093717+0000","last_undegraded":"2024-09-18T08:03:29.093717+0000","last_fullsized":"2024-09-18T08:03:29.093717+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:02:50.886830+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:50.886830+0000","last_clean_scrub_stamp":"2024-09-18T08:02:50.886830+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:04:10.431784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":31,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-09-18T08:03:29.093431+0000","last_change":"2024-09-18T08:02:53.902197+0000","last_active":"2024-09-18T08:03:29.093431+0000","last_peered":"2024-09-18T08:03:29.093431+0000","last_clean":"2024-09-18T08:03:29.093431+0000","last_became_active":"2024-09-18T08:02:53.901858+0000","last_became_peered":"2024-09-18T08:02:53.901858+0000","last_unstale":"2024-09-18T08:03:29.093431+0000","last_undegraded":"2024-09-18T08:03:29.093431+0000","last_fullsized":"2024-09-18T08:03:29.093431+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_clean_scrub_stamp":"2024-09-18T08:02:52.890475+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:04:08.426639+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.b","version":"69'9","reported_seq":54,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-09-18T08:03:29.422737+0000","last_change":"2024-09-18T08:02:49.898562+0000","last_active":"2024-09-18T08:03:29.422737+0000","last_peered":"2024-09-18T08:03:29.422737+0000","last_clean":"2024-09-18T08:03:29.422737+0000","last_became_active":"2024-09-18T08:02:49.897998+0000","last_became_peered":"2024-09-18T08:02:49.897998+0000","last_unstale":"2024-09-18T08:03:29.422737+0000","last_undegraded":"2024-09-18T08:03:29.422737+0000","last_fullsized":"2024-09-18T08:03:29.422737+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_clean_scrub_stamp":"2024-09-18T08:02:48.870718+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:03:56.995348+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":42,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-09-18T08:03:29.426994+0000","last_change":"2024-09-18T08:02:47.877928+0000","last_active":"2024-09-18T08:03:29.426994+0000","last_peered":"2024-09-18T08:03:29.426994+0000","last_clean":"2024-09-18T08:03:29.426994+0000","last_became_active":"2024-09-18T08:02:47.877161+0000","last_became_peered":"2024-09-18T08:02:47.877161+0000","last_unstale":"2024-09-18T08:03:29.426994+0000","last_undegraded":"2024-09-18T08:03:29.426994+0000","last_fullsized":"2024-09-18T08:03:29.426994+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:02:46.860092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:46.860092+0000","last_clean_scrub_stamp":"2024-09-18T08:02:46.860092+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:03:54.682345+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":34,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-09-18T08:03:29.093840+0000","last_change":"2024-09-18T08:02:51.901738+0000","last_active":"2024-09-18T08:03:29.093840+0000","last_peered":"2024-09-18T08:03:29.093840+0000","last_clean":"2024-09-18T08:03:29.093840+0000","last_became_active":"2024-09-18T08:02:51.901330+0000","last_became_peered":"2024-09-18T08:02:51.901330+0000","last_unstale":"2024-09-18T08:03:29.093840+0000","last_undegraded":"2024-09-18T08:03:29.093840+0000","last_fullsized":"2024-09-18T08:03:29.093840+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:02:50.886830+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:50.886830+0000","last_clean_scrub_stamp":"2024-09-18T08:02:50.886830+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:04:18.413020+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,3],"acting":[2,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":31,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-09-18T08:03:29.422432+0000","last_change":"2024-09-18T08:02:53.911400+0000","last_active":"2024-09-18T08:03:29.422432+0000","last_peered":"2024-09-18T08:03:29.422432+0000","last_clean":"2024-09-18T08:03:29.422432+0000","last_became_active":"2024-09-18T08:02:53.911068+0000","last_became_peered":"2024-09-18T08:02:53.911068+0000","last_unstale":"2024-09-18T08:03:29.422432+0000","last_undegraded":"2024-09-18T08:03:29.422432+0000","last_fullsized":"2024-09-18T08:03:29.422432+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_clean_scrub_stamp":"2024-09-18T08:02:52.890475+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:03:57.470235+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.a","version":"69'19","reported_seq":69,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-09-18T08:03:29.426089+0000","last_change":"2024-09-18T08:02:49.898366+0000","last_active":"2024-09-18T08:03:29.426089+0000","last_peered":"2024-09-18T08:03:29.426089+0000","last_clean":"2024-09-18T08:03:29.426089+0000","last_became_active":"2024-09-18T08:02:49.897960+0000","last_became_peered":"2024-09-18T08:02:49.897960+0000","last_unstale":"2024-09-18T08:03:29.426089+0000","last_undegraded":"2024-09-18T08:03:29.426089+0000","last_fullsized":"2024-09-18T08:03:29.426089+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_clean_scrub_stamp":"2024-09-18T08:02:48.870718+0000","objects_scrubbed":0,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:04:18.534695+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":32,"num_read_kb":21,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":42,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-09-18T08:03:29.427761+0000","last_change":"2024-09-18T08:02:47.881119+0000","last_active":"2024-09-18T08:03:29.427761+0000","last_peered":"2024-09-18T08:03:29.427761+0000","last_clean":"2024-09-18T08:03:29.427761+0000","last_became_active":"2024-09-18T08:02:47.880645+0000","last_became_peered":"2024-09-18T08:02:47.880645+0000","last_unstale":"2024-09-18T08:03:29.427761+0000","last_undegraded":"2024-09-18T08:03:29.427761+0000","last_fullsized":"2024-09-18T08:03:29.427761+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:02:46.860092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:46.860092+0000","last_clean_scrub_stamp":"2024-09-18T08:02:46.860092+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:04:02.022191+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":34,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-09-18T08:03:29.093980+0000","last_change":"2024-09-18T08:02:51.903944+0000","last_active":"2024-09-18T08:03:29.093980+0000","last_peered":"2024-09-18T08:03:29.093980+0000","last_clean":"2024-09-18T08:03:29.093980+0000","last_became_active":"2024-09-18T08:02:51.903439+0000","last_became_peered":"2024-09-18T08:02:51.903439+0000","last_unstale":"2024-09-18T08:03:29.093980+0000","last_undegraded":"2024-09-18T08:03:29.093980+0000","last_fullsized":"2024-09-18T08:03:29.093980+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:02:50.886830+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:50.886830+0000","last_clean_scrub_stamp":"2024-09-18T08:02:50.886830+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:03:52.993653+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":31,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-09-18T08:03:29.425484+0000","last_change":"2024-09-18T08:02:53.913096+0000","last_active":"2024-09-18T08:03:29.425484+0000","last_peered":"2024-09-18T08:03:29.425484+0000","last_clean":"2024-09-18T08:03:29.425484+0000","last_became_active":"2024-09-18T08:02:53.912762+0000","last_became_peered":"2024-09-18T08:02:53.912762+0000","last_unstale":"2024-09-18T08:03:29.425484+0000","last_undegraded":"2024-09-18T08:03:29.425484+0000","last_fullsized":"2024-09-18T08:03:29.425484+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_clean_scrub_stamp":"2024-09-18T08:02:52.890475+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:04:03.478804+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.9","version":"69'12","reported_seq":61,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-09-18T08:03:29.093200+0000","last_change":"2024-09-18T08:02:49.895136+0000","last_active":"2024-09-18T08:03:29.093200+0000","last_peered":"2024-09-18T08:03:29.093200+0000","last_clean":"2024-09-18T08:03:29.093200+0000","last_became_active":"2024-09-18T08:02:49.894762+0000","last_became_peered":"2024-09-18T08:02:49.894762+0000","last_unstale":"2024-09-18T08:03:29.093200+0000","last_undegraded":"2024-09-18T08:03:29.093200+0000","last_fullsized":"2024-09-18T08:03:29.093200+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_clean_scrub_stamp":"2024-09-18T08:02:48.870718+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:04:02.257011+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":25,"num_read_kb":16,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":42,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-09-18T08:03:29.427910+0000","last_change":"2024-09-18T08:02:47.884430+0000","last_active":"2024-09-18T08:03:29.427910+0000","last_peered":"2024-09-18T08:03:29.427910+0000","last_clean":"2024-09-18T08:03:29.427910+0000","last_became_active":"2024-09-18T08:02:47.884052+0000","last_became_peered":"2024-09-18T08:02:47.884052+0000","last_unstale":"2024-09-18T08:03:29.427910+0000","last_undegraded":"2024-09-18T08:03:29.427910+0000","last_fullsized":"2024-09-18T08:03:29.427910+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:02:46.860092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:46.860092+0000","last_clean_scrub_stamp":"2024-09-18T08:02:46.860092+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:04:04.711823+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":34,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-09-18T08:03:29.093941+0000","last_change":"2024-09-18T08:02:51.902541+0000","last_active":"2024-09-18T08:03:29.093941+0000","last_peered":"2024-09-18T08:03:29.093941+0000","last_clean":"2024-09-18T08:03:29.093941+0000","last_became_active":"2024-09-18T08:02:51.902163+0000","last_became_peered":"2024-09-18T08:02:51.902163+0000","last_unstale":"2024-09-18T08:03:29.093941+0000","last_undegraded":"2024-09-18T08:03:29.093941+0000","last_fullsized":"2024-09-18T08:03:29.093941+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:02:50.886830+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:50.886830+0000","last_clean_scrub_stamp":"2024-09-18T08:02:50.886830+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:03:54.180178+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":31,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-09-18T08:03:29.425123+0000","last_change":"2024-09-18T08:02:53.907752+0000","last_active":"2024-09-18T08:03:29.425123+0000","last_peered":"2024-09-18T08:03:29.425123+0000","last_clean":"2024-09-18T08:03:29.425123+0000","last_became_active":"2024-09-18T08:02:53.907412+0000","last_became_peered":"2024-09-18T08:02:53.907412+0000","last_unstale":"2024-09-18T08:03:29.425123+0000","last_undegraded":"2024-09-18T08:03:29.425123+0000","last_fullsized":"2024-09-18T08:03:29.425123+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_clean_scrub_stamp":"2024-09-18T08:02:52.890475+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:04:12.083897+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.8","version":"69'15","reported_seq":63,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-09-18T08:03:29.425260+0000","last_change":"2024-09-18T08:02:49.889133+0000","last_active":"2024-09-18T08:03:29.425260+0000","last_peered":"2024-09-18T08:03:29.425260+0000","last_clean":"2024-09-18T08:03:29.425260+0000","last_became_active":"2024-09-18T08:02:49.888799+0000","last_became_peered":"2024-09-18T08:02:49.888799+0000","last_unstale":"2024-09-18T08:03:29.425260+0000","last_undegraded":"2024-09-18T08:03:29.425260+0000","last_fullsized":"2024-09-18T08:03:29.425260+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_clean_scrub_stamp":"2024-09-18T08:02:48.870718+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:04:11.344161+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.f","version":"62'2","reported_seq":54,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-09-18T08:03:29.427719+0000","last_change":"2024-09-18T08:02:47.881441+0000","last_active":"2024-09-18T08:03:29.427719+0000","last_peered":"2024-09-18T08:03:29.427719+0000","last_clean":"2024-09-18T08:03:29.427719+0000","last_became_active":"2024-09-18T08:02:47.881165+0000","last_became_peered":"2024-09-18T08:02:47.881165+0000","last_unstale":"2024-09-18T08:03:29.427719+0000","last_undegraded":"2024-09-18T08:03:29.427719+0000","last_fullsized":"2024-09-18T08:03:29.427719+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:02:46.860092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:46.860092+0000","last_clean_scrub_stamp":"2024-09-18T08:02:46.860092+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:03:50.038477+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":4,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.9","version":"69'8","reported_seq":55,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-09-18T08:03:29.425691+0000","last_change":"2024-09-18T08:02:51.905340+0000","last_active":"2024-09-18T08:03:29.425691+0000","last_peered":"2024-09-18T08:03:29.425691+0000","last_clean":"2024-09-18T08:03:29.425691+0000","last_became_active":"2024-09-18T08:02:51.905059+0000","last_became_peered":"2024-09-18T08:02:51.905059+0000","last_unstale":"2024-09-18T08:03:29.425691+0000","last_undegraded":"2024-09-18T08:03:29.425691+0000","last_fullsized":"2024-09-18T08:03:29.425691+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:02:50.886830+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:50.886830+0000","last_clean_scrub_stamp":"2024-09-18T08:02:50.886830+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:03:52.023681+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":30,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-09-18T08:03:29.425868+0000","last_change":"2024-09-18T08:02:54.046138+0000","last_active":"2024-09-18T08:03:29.425868+0000","last_peered":"2024-09-18T08:03:29.425868+0000","last_clean":"2024-09-18T08:03:29.425868+0000","last_became_active":"2024-09-18T08:02:54.045845+0000","last_became_peered":"2024-09-18T08:02:54.045845+0000","last_unstale":"2024-09-18T08:03:29.425868+0000","last_undegraded":"2024-09-18T08:03:29.425868+0000","last_fullsized":"2024-09-18T08:03:29.425868+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_clean_scrub_stamp":"2024-09-18T08:02:52.890475+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:03:55.148975+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":42,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-09-18T08:03:29.426352+0000","last_change":"2024-09-18T08:02:47.882886+0000","last_active":"2024-09-18T08:03:29.426352+0000","last_peered":"2024-09-18T08:03:29.426352+0000","last_clean":"2024-09-18T08:03:29.426352+0000","last_became_active":"2024-09-18T08:02:47.882600+0000","last_became_peered":"2024-09-18T08:02:47.882600+0000","last_unstale":"2024-09-18T08:03:29.426352+0000","last_undegraded":"2024-09-18T08:03:29.426352+0000","last_fullsized":"2024-09-18T08:03:29.426352+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:02:46.860092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:46.860092+0000","last_clean_scrub_stamp":"2024-09-18T08:02:46.860092+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:04:15.079741+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"69'6","reported_seq":47,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-09-18T08:03:29.425701+0000","last_change":"2024-09-18T08:02:49.895477+0000","last_active":"2024-09-18T08:03:29.425701+0000","last_peered":"2024-09-18T08:03:29.425701+0000","last_clean":"2024-09-18T08:03:29.425701+0000","last_became_active":"2024-09-18T08:02:49.895067+0000","last_became_peered":"2024-09-18T08:02:49.895067+0000","last_unstale":"2024-09-18T08:03:29.425701+0000","last_undegraded":"2024-09-18T08:03:29.425701+0000","last_fullsized":"2024-09-18T08:03:29.425701+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_clean_scrub_stamp":"2024-09-18T08:02:48.870718+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:03:55.471468+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":9,"num_read_kb":6,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":34,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-09-18T08:03:29.426707+0000","last_change":"2024-09-18T08:02:51.912907+0000","last_active":"2024-09-18T08:03:29.426707+0000","last_peered":"2024-09-18T08:03:29.426707+0000","last_clean":"2024-09-18T08:03:29.426707+0000","last_became_active":"2024-09-18T08:02:51.912557+0000","last_became_peered":"2024-09-18T08:02:51.912557+0000","last_unstale":"2024-09-18T08:03:29.426707+0000","last_undegraded":"2024-09-18T08:03:29.426707+0000","last_fullsized":"2024-09-18T08:03:29.426707+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:02:50.886830+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:50.886830+0000","last_clean_scrub_stamp":"2024-09-18T08:02:50.886830+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:03:58.405243+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,1],"acting":[5,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":31,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-09-18T08:03:29.427404+0000","last_change":"2024-09-18T08:02:54.049521+0000","last_active":"2024-09-18T08:03:29.427404+0000","last_peered":"2024-09-18T08:03:29.427404+0000","last_clean":"2024-09-18T08:03:29.427404+0000","last_became_active":"2024-09-18T08:02:54.048959+0000","last_became_peered":"2024-09-18T08:02:54.048959+0000","last_unstale":"2024-09-18T08:03:29.427404+0000","last_undegraded":"2024-09-18T08:03:29.427404+0000","last_fullsized":"2024-09-18T08:03:29.427404+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_clean_scrub_stamp":"2024-09-18T08:02:52.890475+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:03:56.721996+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"69'9","reported_seq":54,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-09-18T08:03:29.423054+0000","last_change":"2024-09-18T08:02:49.892390+0000","last_active":"2024-09-18T08:03:29.423054+0000","last_peered":"2024-09-18T08:03:29.423054+0000","last_clean":"2024-09-18T08:03:29.423054+0000","last_became_active":"2024-09-18T08:02:49.891567+0000","last_became_peered":"2024-09-18T08:02:49.891567+0000","last_unstale":"2024-09-18T08:03:29.423054+0000","last_undegraded":"2024-09-18T08:03:29.423054+0000","last_fullsized":"2024-09-18T08:03:29.423054+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_clean_scrub_stamp":"2024-09-18T08:02:48.870718+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:03:51.625809+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":42,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-09-18T08:03:29.425915+0000","last_change":"2024-09-18T08:02:47.886233+0000","last_active":"2024-09-18T08:03:29.425915+0000","last_peered":"2024-09-18T08:03:29.425915+0000","last_clean":"2024-09-18T08:03:29.425915+0000","last_became_active":"2024-09-18T08:02:47.885790+0000","last_became_peered":"2024-09-18T08:02:47.885790+0000","last_unstale":"2024-09-18T08:03:29.425915+0000","last_undegraded":"2024-09-18T08:03:29.425915+0000","last_fullsized":"2024-09-18T08:03:29.425915+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:02:46.860092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:46.860092+0000","last_clean_scrub_stamp":"2024-09-18T08:02:46.860092+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:03:48.811474+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":34,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-09-18T08:03:29.425652+0000","last_change":"2024-09-18T08:02:51.908640+0000","last_active":"2024-09-18T08:03:29.425652+0000","last_peered":"2024-09-18T08:03:29.425652+0000","last_clean":"2024-09-18T08:03:29.425652+0000","last_became_active":"2024-09-18T08:02:51.907361+0000","last_became_peered":"2024-09-18T08:02:51.907361+0000","last_unstale":"2024-09-18T08:03:29.425652+0000","last_undegraded":"2024-09-18T08:03:29.425652+0000","last_fullsized":"2024-09-18T08:03:29.425652+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:02:50.886830+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:50.886830+0000","last_clean_scrub_stamp":"2024-09-18T08:02:50.886830+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:04:06.357747+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":31,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-09-18T08:03:29.094163+0000","last_change":"2024-09-18T08:02:53.909246+0000","last_active":"2024-09-18T08:03:29.094163+0000","last_peered":"2024-09-18T08:03:29.094163+0000","last_clean":"2024-09-18T08:03:29.094163+0000","last_became_active":"2024-09-18T08:02:53.908917+0000","last_became_peered":"2024-09-18T08:02:53.908917+0000","last_unstale":"2024-09-18T08:03:29.094163+0000","last_undegraded":"2024-09-18T08:03:29.094163+0000","last_fullsized":"2024-09-18T08:03:29.094163+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_clean_scrub_stamp":"2024-09-18T08:02:52.890475+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:03:57.367069+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.15","version":"69'9","reported_seq":54,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-09-18T08:03:29.427201+0000","last_change":"2024-09-18T08:02:49.890846+0000","last_active":"2024-09-18T08:03:29.427201+0000","last_peered":"2024-09-18T08:03:29.427201+0000","last_clean":"2024-09-18T08:03:29.427201+0000","last_became_active":"2024-09-18T08:02:49.890497+0000","last_became_peered":"2024-09-18T08:02:49.890497+0000","last_unstale":"2024-09-18T08:03:29.427201+0000","last_undegraded":"2024-09-18T08:03:29.427201+0000","last_fullsized":"2024-09-18T08:03:29.427201+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_clean_scrub_stamp":"2024-09-18T08:02:48.870718+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:04:02.479848+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":42,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-09-18T08:03:29.422503+0000","last_change":"2024-09-18T08:02:47.877632+0000","last_active":"2024-09-18T08:03:29.422503+0000","last_peered":"2024-09-18T08:03:29.422503+0000","last_clean":"2024-09-18T08:03:29.422503+0000","last_became_active":"2024-09-18T08:02:47.875983+0000","last_became_peered":"2024-09-18T08:02:47.875983+0000","last_unstale":"2024-09-18T08:03:29.422503+0000","last_undegraded":"2024-09-18T08:03:29.422503+0000","last_fullsized":"2024-09-18T08:03:29.422503+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:02:46.860092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:46.860092+0000","last_clean_scrub_stamp":"2024-09-18T08:02:46.860092+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:04:09.677911+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.14","version":"69'8","reported_seq":52,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-09-18T08:03:29.425267+0000","last_change":"2024-09-18T08:02:51.909896+0000","last_active":"2024-09-18T08:03:29.425267+0000","last_peered":"2024-09-18T08:03:29.425267+0000","last_clean":"2024-09-18T08:03:29.425267+0000","last_became_active":"2024-09-18T08:02:51.909513+0000","last_became_peered":"2024-09-18T08:02:51.909513+0000","last_unstale":"2024-09-18T08:03:29.425267+0000","last_undegraded":"2024-09-18T08:03:29.425267+0000","last_fullsized":"2024-09-18T08:03:29.425267+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:02:50.886830+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:50.886830+0000","last_clean_scrub_stamp":"2024-09-18T08:02:50.886830+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:03:54.167632+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":31,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-09-18T08:03:29.093927+0000","last_change":"2024-09-18T08:02:53.907729+0000","last_active":"2024-09-18T08:03:29.093927+0000","last_peered":"2024-09-18T08:03:29.093927+0000","last_clean":"2024-09-18T08:03:29.093927+0000","last_became_active":"2024-09-18T08:02:53.907428+0000","last_became_peered":"2024-09-18T08:02:53.907428+0000","last_unstale":"2024-09-18T08:03:29.093927+0000","last_undegraded":"2024-09-18T08:03:29.093927+0000","last_fullsized":"2024-09-18T08:03:29.093927+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_clean_scrub_stamp":"2024-09-18T08:02:52.890475+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:04:00.922586+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.14","version":"69'10","reported_seq":53,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-09-18T08:03:29.425479+0000","last_change":"2024-09-18T08:02:49.889483+0000","last_active":"2024-09-18T08:03:29.425479+0000","last_peered":"2024-09-18T08:03:29.425479+0000","last_clean":"2024-09-18T08:03:29.425479+0000","last_became_active":"2024-09-18T08:02:49.889191+0000","last_became_peered":"2024-09-18T08:02:49.889191+0000","last_unstale":"2024-09-18T08:03:29.425479+0000","last_undegraded":"2024-09-18T08:03:29.425479+0000","last_fullsized":"2024-09-18T08:03:29.425479+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_clean_scrub_stamp":"2024-09-18T08:02:48.870718+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:03:58.029274+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":42,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-09-18T08:03:29.426830+0000","last_change":"2024-09-18T08:02:47.882542+0000","last_active":"2024-09-18T08:03:29.426830+0000","last_peered":"2024-09-18T08:03:29.426830+0000","last_clean":"2024-09-18T08:03:29.426830+0000","last_became_active":"2024-09-18T08:02:47.882065+0000","last_became_peered":"2024-09-18T08:02:47.882065+0000","last_unstale":"2024-09-18T08:03:29.426830+0000","last_undegraded":"2024-09-18T08:03:29.426830+0000","last_fullsized":"2024-09-18T08:03:29.426830+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:02:46.860092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:46.860092+0000","last_clean_scrub_stamp":"2024-09-18T08:02:46.860092+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:03:59.012126+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.15","version":"69'8","reported_seq":52,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-09-18T08:03:29.427052+0000","last_change":"2024-09-18T08:02:51.914448+0000","last_active":"2024-09-18T08:03:29.427052+0000","last_peered":"2024-09-18T08:03:29.427052+0000","last_clean":"2024-09-18T08:03:29.427052+0000","last_became_active":"2024-09-18T08:02:51.914135+0000","last_became_peered":"2024-09-18T08:02:51.914135+0000","last_unstale":"2024-09-18T08:03:29.427052+0000","last_undegraded":"2024-09-18T08:03:29.427052+0000","last_fullsized":"2024-09-18T08:03:29.427052+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:02:50.886830+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:50.886830+0000","last_clean_scrub_stamp":"2024-09-18T08:02:50.886830+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:03:56.893696+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":31,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-09-18T08:03:29.422823+0000","last_change":"2024-09-18T08:02:53.912409+0000","last_active":"2024-09-18T08:03:29.422823+0000","last_peered":"2024-09-18T08:03:29.422823+0000","last_clean":"2024-09-18T08:03:29.422823+0000","last_became_active":"2024-09-18T08:02:53.912112+0000","last_became_peered":"2024-09-18T08:02:53.912112+0000","last_unstale":"2024-09-18T08:03:29.422823+0000","last_undegraded":"2024-09-18T08:03:29.422823+0000","last_fullsized":"2024-09-18T08:03:29.422823+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_clean_scrub_stamp":"2024-09-18T08:02:52.890475+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:03:54.310553+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.13","version":"69'11","reported_seq":57,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-09-18T08:03:29.093367+0000","last_change":"2024-09-18T08:02:49.897433+0000","last_active":"2024-09-18T08:03:29.093367+0000","last_peered":"2024-09-18T08:03:29.093367+0000","last_clean":"2024-09-18T08:03:29.093367+0000","last_became_active":"2024-09-18T08:02:49.896949+0000","last_became_peered":"2024-09-18T08:02:49.896949+0000","last_unstale":"2024-09-18T08:03:29.093367+0000","last_undegraded":"2024-09-18T08:03:29.093367+0000","last_fullsized":"2024-09-18T08:03:29.093367+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_clean_scrub_stamp":"2024-09-18T08:02:48.870718+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:04:08.564349+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":42,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-09-18T08:03:29.094017+0000","last_change":"2024-09-18T08:02:47.880202+0000","last_active":"2024-09-18T08:03:29.094017+0000","last_peered":"2024-09-18T08:03:29.094017+0000","last_clean":"2024-09-18T08:03:29.094017+0000","last_became_active":"2024-09-18T08:02:47.879771+0000","last_became_peered":"2024-09-18T08:02:47.879771+0000","last_unstale":"2024-09-18T08:03:29.094017+0000","last_undegraded":"2024-09-18T08:03:29.094017+0000","last_fullsized":"2024-09-18T08:03:29.094017+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:02:46.860092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:46.860092+0000","last_clean_scrub_stamp":"2024-09-18T08:02:46.860092+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:03:58.640926+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":34,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-09-18T08:03:29.093105+0000","last_change":"2024-09-18T08:02:51.912062+0000","last_active":"2024-09-18T08:03:29.093105+0000","last_peered":"2024-09-18T08:03:29.093105+0000","last_clean":"2024-09-18T08:03:29.093105+0000","last_became_active":"2024-09-18T08:02:51.910752+0000","last_became_peered":"2024-09-18T08:02:51.910752+0000","last_unstale":"2024-09-18T08:03:29.093105+0000","last_undegraded":"2024-09-18T08:03:29.093105+0000","last_fullsized":"2024-09-18T08:03:29.093105+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:02:50.886830+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:50.886830+0000","last_clean_scrub_stamp":"2024-09-18T08:02:50.886830+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:04:01.465747+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":31,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-09-18T08:03:29.424784+0000","last_change":"2024-09-18T08:02:53.911861+0000","last_active":"2024-09-18T08:03:29.424784+0000","last_peered":"2024-09-18T08:03:29.424784+0000","last_clean":"2024-09-18T08:03:29.424784+0000","last_became_active":"2024-09-18T08:02:53.911429+0000","last_became_peered":"2024-09-18T08:02:53.911429+0000","last_unstale":"2024-09-18T08:03:29.424784+0000","last_undegraded":"2024-09-18T08:03:29.424784+0000","last_fullsized":"2024-09-18T08:03:29.424784+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_clean_scrub_stamp":"2024-09-18T08:02:52.890475+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:03:58.350802+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.12","version":"69'9","reported_seq":54,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-09-18T08:03:29.093364+0000","last_change":"2024-09-18T08:02:49.893472+0000","last_active":"2024-09-18T08:03:29.093364+0000","last_peered":"2024-09-18T08:03:29.093364+0000","last_clean":"2024-09-18T08:03:29.093364+0000","last_became_active":"2024-09-18T08:02:49.885950+0000","last_became_peered":"2024-09-18T08:02:49.885950+0000","last_unstale":"2024-09-18T08:03:29.093364+0000","last_undegraded":"2024-09-18T08:03:29.093364+0000","last_fullsized":"2024-09-18T08:03:29.093364+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_clean_scrub_stamp":"2024-09-18T08:02:48.870718+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:03:57.261106+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":42,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-09-18T08:03:29.427640+0000","last_change":"2024-09-18T08:02:47.883002+0000","last_active":"2024-09-18T08:03:29.427640+0000","last_peered":"2024-09-18T08:03:29.427640+0000","last_clean":"2024-09-18T08:03:29.427640+0000","last_became_active":"2024-09-18T08:02:47.882096+0000","last_became_peered":"2024-09-18T08:02:47.882096+0000","last_unstale":"2024-09-18T08:03:29.427640+0000","last_undegraded":"2024-09-18T08:03:29.427640+0000","last_fullsized":"2024-09-18T08:03:29.427640+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:02:46.860092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:46.860092+0000","last_clean_scrub_stamp":"2024-09-18T08:02:46.860092+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:04:10.794338+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":34,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-09-18T08:03:29.425247+0000","last_change":"2024-09-18T08:02:51.906594+0000","last_active":"2024-09-18T08:03:29.425247+0000","last_peered":"2024-09-18T08:03:29.425247+0000","last_clean":"2024-09-18T08:03:29.425247+0000","last_became_active":"2024-09-18T08:02:51.906277+0000","last_became_peered":"2024-09-18T08:02:51.906277+0000","last_unstale":"2024-09-18T08:03:29.425247+0000","last_undegraded":"2024-09-18T08:03:29.425247+0000","last_fullsized":"2024-09-18T08:03:29.425247+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:02:50.886830+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:50.886830+0000","last_clean_scrub_stamp":"2024-09-18T08:02:50.886830+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:04:07.327398+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.10","version":"69'1","reported_seq":32,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-09-18T08:03:29.422318+0000","last_change":"2024-09-18T08:02:53.913013+0000","last_active":"2024-09-18T08:03:29.422318+0000","last_peered":"2024-09-18T08:03:29.422318+0000","last_clean":"2024-09-18T08:03:29.422318+0000","last_became_active":"2024-09-18T08:02:53.912699+0000","last_became_peered":"2024-09-18T08:02:53.912699+0000","last_unstale":"2024-09-18T08:03:29.422318+0000","last_undegraded":"2024-09-18T08:03:29.422318+0000","last_fullsized":"2024-09-18T08:03:29.422318+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_clean_scrub_stamp":"2024-09-18T08:02:52.890475+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:04:15.154915+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.11","version":"69'11","reported_seq":57,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-09-18T08:03:29.424925+0000","last_change":"2024-09-18T08:02:49.889895+0000","last_active":"2024-09-18T08:03:29.424925+0000","last_peered":"2024-09-18T08:03:29.424925+0000","last_clean":"2024-09-18T08:03:29.424925+0000","last_became_active":"2024-09-18T08:02:49.889611+0000","last_became_peered":"2024-09-18T08:02:49.889611+0000","last_unstale":"2024-09-18T08:03:29.424925+0000","last_undegraded":"2024-09-18T08:03:29.424925+0000","last_fullsized":"2024-09-18T08:03:29.424925+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_clean_scrub_stamp":"2024-09-18T08:02:48.870718+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:03:51.062461+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":42,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-09-18T08:03:29.426808+0000","last_change":"2024-09-18T08:02:47.879259+0000","last_active":"2024-09-18T08:03:29.426808+0000","last_peered":"2024-09-18T08:03:29.426808+0000","last_clean":"2024-09-18T08:03:29.426808+0000","last_became_active":"2024-09-18T08:02:47.878886+0000","last_became_peered":"2024-09-18T08:02:47.878886+0000","last_unstale":"2024-09-18T08:03:29.426808+0000","last_undegraded":"2024-09-18T08:03:29.426808+0000","last_fullsized":"2024-09-18T08:03:29.426808+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:02:46.860092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:46.860092+0000","last_clean_scrub_stamp":"2024-09-18T08:02:46.860092+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:03:47.500112+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":34,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-09-18T08:03:29.425090+0000","last_change":"2024-09-18T08:02:51.908058+0000","last_active":"2024-09-18T08:03:29.425090+0000","last_peered":"2024-09-18T08:03:29.425090+0000","last_clean":"2024-09-18T08:03:29.425090+0000","last_became_active":"2024-09-18T08:02:51.907762+0000","last_became_peered":"2024-09-18T08:02:51.907762+0000","last_unstale":"2024-09-18T08:03:29.425090+0000","last_undegraded":"2024-09-18T08:03:29.425090+0000","last_fullsized":"2024-09-18T08:03:29.425090+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:02:50.886830+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:50.886830+0000","last_clean_scrub_stamp":"2024-09-18T08:02:50.886830+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:04:01.256104+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":30,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-09-18T08:03:29.425356+0000","last_change":"2024-09-18T08:02:54.047687+0000","last_active":"2024-09-18T08:03:29.425356+0000","last_peered":"2024-09-18T08:03:29.425356+0000","last_clean":"2024-09-18T08:03:29.425356+0000","last_became_active":"2024-09-18T08:02:54.047181+0000","last_became_peered":"2024-09-18T08:02:54.047181+0000","last_unstale":"2024-09-18T08:03:29.425356+0000","last_undegraded":"2024-09-18T08:03:29.425356+0000","last_fullsized":"2024-09-18T08:03:29.425356+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_clean_scrub_stamp":"2024-09-18T08:02:52.890475+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:04:12.998775+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.10","version":"69'4","reported_seq":44,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-09-18T08:03:29.424764+0000","last_change":"2024-09-18T08:02:49.890435+0000","last_active":"2024-09-18T08:03:29.424764+0000","last_peered":"2024-09-18T08:03:29.424764+0000","last_clean":"2024-09-18T08:03:29.424764+0000","last_became_active":"2024-09-18T08:02:49.890126+0000","last_became_peered":"2024-09-18T08:02:49.890126+0000","last_unstale":"2024-09-18T08:03:29.424764+0000","last_undegraded":"2024-09-18T08:03:29.424764+0000","last_fullsized":"2024-09-18T08:03:29.424764+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_clean_scrub_stamp":"2024-09-18T08:02:48.870718+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:03:56.821965+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":6,"num_read_kb":4,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":42,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-09-18T08:03:29.423051+0000","last_change":"2024-09-18T08:02:47.877991+0000","last_active":"2024-09-18T08:03:29.423051+0000","last_peered":"2024-09-18T08:03:29.423051+0000","last_clean":"2024-09-18T08:03:29.423051+0000","last_became_active":"2024-09-18T08:02:47.877616+0000","last_became_peered":"2024-09-18T08:02:47.877616+0000","last_unstale":"2024-09-18T08:03:29.423051+0000","last_undegraded":"2024-09-18T08:03:29.423051+0000","last_fullsized":"2024-09-18T08:03:29.423051+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:02:46.860092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:46.860092+0000","last_clean_scrub_stamp":"2024-09-18T08:02:46.860092+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:04:04.436191+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":34,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-09-18T08:03:29.425717+0000","last_change":"2024-09-18T08:02:51.906818+0000","last_active":"2024-09-18T08:03:29.425717+0000","last_peered":"2024-09-18T08:03:29.425717+0000","last_clean":"2024-09-18T08:03:29.425717+0000","last_became_active":"2024-09-18T08:02:51.906372+0000","last_became_peered":"2024-09-18T08:02:51.906372+0000","last_unstale":"2024-09-18T08:03:29.425717+0000","last_undegraded":"2024-09-18T08:03:29.425717+0000","last_fullsized":"2024-09-18T08:03:29.425717+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:02:50.886830+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:50.886830+0000","last_clean_scrub_stamp":"2024-09-18T08:02:50.886830+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:03:51.034977+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":31,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-09-18T08:03:29.425810+0000","last_change":"2024-09-18T08:02:53.909445+0000","last_active":"2024-09-18T08:03:29.425810+0000","last_peered":"2024-09-18T08:03:29.425810+0000","last_clean":"2024-09-18T08:03:29.425810+0000","last_became_active":"2024-09-18T08:02:53.909146+0000","last_became_peered":"2024-09-18T08:02:53.909146+0000","last_unstale":"2024-09-18T08:03:29.425810+0000","last_undegraded":"2024-09-18T08:03:29.425810+0000","last_fullsized":"2024-09-18T08:03:29.425810+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_clean_scrub_stamp":"2024-09-18T08:02:52.890475+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:04:03.033337+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,4],"acting":[7,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":31,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-09-18T08:03:29.093251+0000","last_change":"2024-09-18T08:02:53.907055+0000","last_active":"2024-09-18T08:03:29.093251+0000","last_peered":"2024-09-18T08:03:29.093251+0000","last_clean":"2024-09-18T08:03:29.093251+0000","last_became_active":"2024-09-18T08:02:53.906726+0000","last_became_peered":"2024-09-18T08:02:53.906726+0000","last_unstale":"2024-09-18T08:03:29.093251+0000","last_undegraded":"2024-09-18T08:03:29.093251+0000","last_fullsized":"2024-09-18T08:03:29.093251+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_clean_scrub_stamp":"2024-09-18T08:02:52.890475+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:03:58.628509+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":42,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-09-18T08:03:29.424966+0000","last_change":"2024-09-18T08:02:47.879447+0000","last_active":"2024-09-18T08:03:29.424966+0000","last_peered":"2024-09-18T08:03:29.424966+0000","last_clean":"2024-09-18T08:03:29.424966+0000","last_became_active":"2024-09-18T08:02:47.878922+0000","last_became_peered":"2024-09-18T08:02:47.878922+0000","last_unstale":"2024-09-18T08:03:29.424966+0000","last_undegraded":"2024-09-18T08:03:29.424966+0000","last_fullsized":"2024-09-18T08:03:29.424966+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:02:46.860092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:46.860092+0000","last_clean_scrub_stamp":"2024-09-18T08:02:46.860092+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:03:54.771042+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"69'11","reported_seq":57,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-09-18T08:03:29.427692+0000","last_change":"2024-09-18T08:02:49.898630+0000","last_active":"2024-09-18T08:03:29.427692+0000","last_peered":"2024-09-18T08:03:29.427692+0000","last_clean":"2024-09-18T08:03:29.427692+0000","last_became_active":"2024-09-18T08:02:49.898048+0000","last_became_peered":"2024-09-18T08:02:49.898048+0000","last_unstale":"2024-09-18T08:03:29.427692+0000","last_undegraded":"2024-09-18T08:03:29.427692+0000","last_fullsized":"2024-09-18T08:03:29.427692+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_clean_scrub_stamp":"2024-09-18T08:02:48.870718+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:04:05.049957+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":34,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-09-18T08:03:29.422915+0000","last_change":"2024-09-18T08:02:51.900666+0000","last_active":"2024-09-18T08:03:29.422915+0000","last_peered":"2024-09-18T08:03:29.422915+0000","last_clean":"2024-09-18T08:03:29.422915+0000","last_became_active":"2024-09-18T08:02:51.900300+0000","last_became_peered":"2024-09-18T08:02:51.900300+0000","last_unstale":"2024-09-18T08:03:29.422915+0000","last_undegraded":"2024-09-18T08:03:29.422915+0000","last_fullsized":"2024-09-18T08:03:29.422915+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:02:50.886830+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:50.886830+0000","last_clean_scrub_stamp":"2024-09-18T08:02:50.886830+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:04:12.807594+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":416,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":64,"ondisk_log_size":64,"up":96,"acting":96,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":178,"num_object_clones":0,"num_object_copies":534,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":178,"num_whiteouts":0,"num_read":698,"num_read_kb":455,"num_write":417,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":417792,"data_stored":11106,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":393,"ondisk_log_size":393,"up":96,"acting":96,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":1613,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":24,"num_read_kb":24,"num_write":10,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":4839,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":6,"ondisk_log_size":6,"up":96,"acting":96,"num_store_stats":8},{"poolid":2,"num_pg":3,"stat_sum":{"num_bytes":408,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":40,"num_read_kb":35,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":8,"ondisk_log_size":8,"up":9,"acting":9,"num_store_stats":7},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":231,"num_write":113,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":590368,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":79,"ondisk_log_size":79,"up":3,"acting":3,"num_store_stats":3}],"osd_stats":[{"osd":7,"up_from":56,"seq":240518168589,"num_pgs":53,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28364,"kb_used_data":1336,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716788,"statfs":{"total":95995035648,"available":95965990912,"internally_reserved":0,"allocated":1368064,"data_stored":900466,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":47,"seq":201863462931,"num_pgs":43,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28344,"kb_used_data":1316,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716808,"statfs":{"total":95995035648,"available":95966011392,"internally_reserved":0,"allocated":1347584,"data_stored":899381,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":39,"seq":167503724569,"num_pgs":47,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27776,"kb_used_data":748,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717376,"statfs":{"total":95995035648,"available":95966593024,"internally_reserved":0,"allocated":765952,"data_stored":309821,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":32,"seq":137438953502,"num_pgs":58,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27796,"kb_used_data":768,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717356,"statfs":{"total":95995035648,"available":95966572544,"internally_reserved":0,"allocated":786432,"data_stored":310611,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":26,"seq":111669149733,"num_pgs":56,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27800,"kb_used_data":772,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717352,"statfs":{"total":95995035648,"available":95966568448,"internally_reserved":0,"allocated":790528,"data_stored":309904,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":27654600},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411371,"num_pgs":36,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":31924,"kb_used_data":736,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713228,"statfs":{"total":95995035648,"available":95962345472,"internally_reserved":0,"allocated":753664,"data_stored":309597,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574895,"num_pgs":57,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27840,"kb_used_data":812,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717312,"statfs":{"total":95995035648,"available":95966527488,"internally_reserved":0,"allocated":831488,"data_stored":312240,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738422,"num_pgs":46,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28376,"kb_used_data":1348,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716776,"statfs":{"total":95995035648,"available":95965978624,"internally_reserved":0,"allocated":1380352,"data_stored":901181,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1131,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1521,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":436,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":46,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1177,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":436,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":90112,"data_stored":2338,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":798,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":1898,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1474,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":990,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1034,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1254,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":403,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":416,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":403,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T08:03:35.605 INFO:tasks.cephadm.ceph_manager.ceph:clean! 2024-09-18T08:03:35.605 INFO:tasks.ceph:Waiting until ceph cluster ceph is healthy... 2024-09-18T08:03:35.606 INFO:tasks.cephadm.ceph_manager.ceph:wait_until_healthy 2024-09-18T08:03:35.606 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid 779349ca-7593-11ef-bceb-c7b262605968 -- ceph health --format=json 2024-09-18T08:03:35.656 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:03:35 smithi050 ceph-mon[32371]: from='client.24605 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-18T08:03:35.867 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/779349ca-7593-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T08:03:36.411 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:03:36 smithi007 ceph-mon[29459]: pgmap v17: 132 pgs: 132 active+clean; 583 KiB data, 223 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-18T08:03:36.411 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:03:36 smithi007 ceph-mon[39249]: pgmap v17: 132 pgs: 132 active+clean; 583 KiB data, 223 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-18T08:03:36.656 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:03:36 smithi050 ceph-mon[32371]: pgmap v17: 132 pgs: 132 active+clean; 583 KiB data, 223 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-18T08:03:36.980 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Sep 18 08:03:36 smithi007 ceph-779349ca-7593-11ef-bceb-c7b262605968-mgr-y[29688]: ::ffff:172.21.15.50 - - [18/Sep/2024:08:03:36] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-09-18T08:03:37.081 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-09-18T08:03:37.081 INFO:teuthology.orchestra.run.smithi007.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2024-09-18T08:03:37.481 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:03:37 smithi007 ceph-mon[29459]: from='client.? 172.21.15.7:0/9810096' entity='client.admin' cmd={"prefix": "health", "format": "json"} : dispatch 2024-09-18T08:03:37.481 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:03:37 smithi007 ceph-mon[39249]: from='client.? 172.21.15.7:0/9810096' entity='client.admin' cmd={"prefix": "health", "format": "json"} : dispatch 2024-09-18T08:03:37.656 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:03:37 smithi050 ceph-mon[32371]: from='client.? 172.21.15.7:0/9810096' entity='client.admin' cmd={"prefix": "health", "format": "json"} : dispatch 2024-09-18T08:03:38.212 INFO:tasks.cephadm.ceph_manager.ceph:wait_until_healthy done 2024-09-18T08:03:38.212 INFO:tasks.cephadm:Setup complete, yielding 2024-09-18T08:03:38.212 INFO:teuthology.run_tasks:Running task thrashosds... 2024-09-18T08:03:38.228 INFO:tasks.thrashosds:config is {'chance_pgnum_grow': 1, 'chance_pgnum_shrink': 1, 'chance_pgpnum_fix': 1, 'timeout': 1200, 'sighup_delay': 0.1, 'optrack_toggle_delay': 2.0, 'dump_ops_enable': 'true', 'noscrub_toggle_delay': 2.0, 'random_eio': 0.0} 2024-09-18T08:03:38.228 INFO:tasks.thrashosds:overrides is {'admin_socket': {'branch': 'wip-guits-main-2024-09-17-1213'}, 'ceph': {'conf': {'client.0': {'debug ms': 1, 'debug objecter': 20, 'debug rados': 20}, 'global': {'ms bind msgr1': False, 'ms bind msgr2': True, 'ms type': 'async'}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon min osdmap epochs': 50, 'mon osdmap full prune interval': 2, 'mon osdmap full prune min': 15, 'mon osdmap full prune txsize': 2, 'paxos service trim min': 10}, 'osd': {'debug ms': 1, 'debug osd': 20, 'osd debug reject backfill probability': 0.3, 'osd delete sleep': 1, 'osd max backfills': 3, 'osd scrub max interval': 120, 'osd scrub min interval': 60, 'osd shutdown pgref assert': True, 'osd snap trim sleep': 2}}, 'flavor': 'default', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', 'CEPHADM_STRAY_DAEMON', 'CEPHADM_FAILED_DAEMON', 'but it is still running', 'objects unfound and apparently lost', '\\(POOL_APP_NOT_ENABLED\\)'], 'log-only-match': ['CEPHADM_'], 'sha1': 'e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e'}, 'ceph-deploy': {'conf': {'client': {'log file': '/var/log/ceph/ceph-$name.$pid.log'}, 'mon': {}}}, 'cephadm': {'cephadm_mode': 'root'}, 'install': {'ceph': {'flavor': 'default', 'sha1': 'e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e'}}, 'selinux': {'allowlist': ['scontext=system_u:system_r:logrotate_t:s0', 'scontext=system_u:system_r:getty_t:s0']}, 'workunit': {'branch': 'wip-guits-main-2024-09-17-1213', 'sha1': 'e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e'}} 2024-09-18T08:03:38.228 INFO:tasks.thrashosds:config is {'chance_pgnum_grow': 1, 'chance_pgnum_shrink': 1, 'chance_pgpnum_fix': 1, 'timeout': 1200, 'sighup_delay': 0.1, 'optrack_toggle_delay': 2.0, 'dump_ops_enable': 'true', 'noscrub_toggle_delay': 2.0, 'random_eio': 0.0} 2024-09-18T08:03:38.228 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid 779349ca-7593-11ef-bceb-c7b262605968 -- ceph config set mgr mgr_debug_aggressive_pg_num_changes true 2024-09-18T08:03:38.401 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:03:38 smithi007 ceph-mon[29459]: pgmap v18: 132 pgs: 132 active+clean; 583 KiB data, 223 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-09-18T08:03:38.402 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:03:38 smithi007 ceph-mon[39249]: pgmap v18: 132 pgs: 132 active+clean; 583 KiB data, 223 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-09-18T08:03:38.656 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:03:38 smithi050 ceph-mon[32371]: pgmap v18: 132 pgs: 132 active+clean; 583 KiB data, 223 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-09-18T08:03:39.156 INFO:journalctl@ceph.iscsi.iscsi.a.smithi050.stdout:Sep 18 08:03:38 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-iscsi-iscsi-a[57782]: debug there is no tcmu-runner data available 2024-09-18T08:03:40.424 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/779349ca-7593-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T08:03:40.480 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:03:40 smithi007 ceph-mon[29459]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-18T08:03:40.481 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:03:40 smithi007 ceph-mon[29459]: pgmap v19: 132 pgs: 132 active+clean; 583 KiB data, 223 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-09-18T08:03:40.481 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:03:40 smithi007 ceph-mon[39249]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-18T08:03:40.481 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:03:40 smithi007 ceph-mon[39249]: pgmap v19: 132 pgs: 132 active+clean; 583 KiB data, 223 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-09-18T08:03:40.656 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:03:40 smithi050 ceph-mon[32371]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-18T08:03:40.656 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:03:40 smithi050 ceph-mon[32371]: pgmap v19: 132 pgs: 132 active+clean; 583 KiB data, 223 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-09-18T08:03:41.393 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Sep 18 08:03:41 smithi007 ceph-779349ca-7593-11ef-bceb-c7b262605968-mgr-y[29688]: [18/Sep/2024:08:03:41] ENGINE Bus STOPPING 2024-09-18T08:03:41.393 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Sep 18 08:03:41 smithi007 ceph-779349ca-7593-11ef-bceb-c7b262605968-mgr-y[29688]: [18/Sep/2024:08:03:41] ENGINE HTTP Server cherrypy._cpwsgi_server.CPWSGIServer(('::', 9283)) shut down 2024-09-18T08:03:41.394 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Sep 18 08:03:41 smithi007 ceph-779349ca-7593-11ef-bceb-c7b262605968-mgr-y[29688]: [18/Sep/2024:08:03:41] ENGINE Bus STOPPED 2024-09-18T08:03:41.394 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Sep 18 08:03:41 smithi007 ceph-779349ca-7593-11ef-bceb-c7b262605968-mgr-y[29688]: [18/Sep/2024:08:03:41] ENGINE Bus STARTING 2024-09-18T08:03:41.394 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Sep 18 08:03:41 smithi007 ceph-779349ca-7593-11ef-bceb-c7b262605968-mgr-y[29688]: [18/Sep/2024:08:03:41] ENGINE Serving on http://:::9283 2024-09-18T08:03:41.394 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Sep 18 08:03:41 smithi007 ceph-779349ca-7593-11ef-bceb-c7b262605968-mgr-y[29688]: [18/Sep/2024:08:03:41] ENGINE Bus STARTED 2024-09-18T08:03:42.126 INFO:journalctl@ceph.alertmanager.a.smithi007.stdout:Sep 18 08:03:42 smithi007 systemd[1]: Starting Ceph alertmanager.a for 779349ca-7593-11ef-bceb-c7b262605968... 2024-09-18T08:03:42.353 INFO:tasks.thrashosds:Beginning thrashosds... 2024-09-18T08:03:42.353 INFO:tasks.cephadm.ceph_manager.ceph:waiting for clean 2024-09-18T08:03:42.353 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid 779349ca-7593-11ef-bceb-c7b262605968 -- ceph pg dump --format=json 2024-09-18T08:03:42.378 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:03:42 smithi007 ceph-mon[29459]: pgmap v20: 132 pgs: 132 active+clean; 583 KiB data, 223 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-18T08:03:42.378 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:03:42 smithi007 ceph-mon[29459]: from='client.? ' entity='client.admin' 2024-09-18T08:03:42.378 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:03:42 smithi007 ceph-mon[29459]: from='mgr.24470 172.21.15.7:0/2594390261' entity='mgr.y' cmd={"prefix": "orch prometheus get-credentials"} : dispatch 2024-09-18T08:03:42.378 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:03:42 smithi007 ceph-mon[29459]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "orch prometheus get-credentials"}]: dispatch 2024-09-18T08:03:42.378 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:03:42 smithi007 ceph-mon[39249]: pgmap v20: 132 pgs: 132 active+clean; 583 KiB data, 223 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-18T08:03:42.378 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:03:42 smithi007 ceph-mon[39249]: from='client.? ' entity='client.admin' 2024-09-18T08:03:42.378 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:03:42 smithi007 ceph-mon[39249]: from='mgr.24470 172.21.15.7:0/2594390261' entity='mgr.y' cmd={"prefix": "orch prometheus get-credentials"} : dispatch 2024-09-18T08:03:42.379 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:03:42 smithi007 ceph-mon[39249]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "orch prometheus get-credentials"}]: dispatch 2024-09-18T08:03:42.406 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:03:42 smithi050 ceph-mon[32371]: pgmap v20: 132 pgs: 132 active+clean; 583 KiB data, 223 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-18T08:03:42.406 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:03:42 smithi050 ceph-mon[32371]: from='client.? ' entity='client.admin' 2024-09-18T08:03:42.406 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:03:42 smithi050 ceph-mon[32371]: from='mgr.24470 172.21.15.7:0/2594390261' entity='mgr.y' cmd={"prefix": "orch prometheus get-credentials"} : dispatch 2024-09-18T08:03:42.406 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:03:42 smithi050 ceph-mon[32371]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "orch prometheus get-credentials"}]: dispatch 2024-09-18T08:03:42.927 INFO:journalctl@ceph.alertmanager.a.smithi007.stdout:Sep 18 08:03:42 smithi007 podman[70287]: 2024-09-18 08:03:42.529498596 +0000 UTC m=+0.023925357 image pull 11f11916f8cdfb17c1da74b20deb2075d778357b4d60474d82453303f8d461f5 quay.io/prometheus/alertmanager:v0.27.0 2024-09-18T08:03:42.927 INFO:journalctl@ceph.alertmanager.a.smithi007.stdout:Sep 18 08:03:42 smithi007 podman[70287]: 2024-09-18 08:03:42.63195854 +0000 UTC m=+0.126385301 volume create 1d799220f78d0a5081bb972ed6edb172e3eb23d5c0d107ce2b39e3b2af71480c 2024-09-18T08:03:42.928 INFO:journalctl@ceph.alertmanager.a.smithi007.stdout:Sep 18 08:03:42 smithi007 podman[70287]: 2024-09-18 08:03:42.749146258 +0000 UTC m=+0.243573021 container create 5fd76c77cedf61a73045fde95d31e55a84b78edda9fe5b68ede971e9ff30b7a3 (image=quay.io/prometheus/alertmanager:v0.27.0, name=ceph-779349ca-7593-11ef-bceb-c7b262605968-alertmanager-a, maintainer=The Prometheus Authors ) 2024-09-18T08:03:43.234 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/779349ca-7593-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T08:03:43.338 INFO:journalctl@ceph.alertmanager.a.smithi007.stdout:Sep 18 08:03:43 smithi007 podman[70287]: 2024-09-18 08:03:43.060196888 +0000 UTC m=+0.554623654 container init 5fd76c77cedf61a73045fde95d31e55a84b78edda9fe5b68ede971e9ff30b7a3 (image=quay.io/prometheus/alertmanager:v0.27.0, name=ceph-779349ca-7593-11ef-bceb-c7b262605968-alertmanager-a, maintainer=The Prometheus Authors ) 2024-09-18T08:03:43.338 INFO:journalctl@ceph.alertmanager.a.smithi007.stdout:Sep 18 08:03:43 smithi007 podman[70287]: 2024-09-18 08:03:43.063452788 +0000 UTC m=+0.557879550 container start 5fd76c77cedf61a73045fde95d31e55a84b78edda9fe5b68ede971e9ff30b7a3 (image=quay.io/prometheus/alertmanager:v0.27.0, name=ceph-779349ca-7593-11ef-bceb-c7b262605968-alertmanager-a, maintainer=The Prometheus Authors ) 2024-09-18T08:03:43.339 INFO:journalctl@ceph.alertmanager.a.smithi007.stdout:Sep 18 08:03:43 smithi007 ceph-779349ca-7593-11ef-bceb-c7b262605968-alertmanager-a[70326]: ts=2024-09-18T08:03:43.074Z caller=main.go:181 level=info msg="Starting Alertmanager" version="(version=0.27.0, branch=HEAD, revision=0aa3c2aad14cff039931923ab16b26b7481783b5)" 2024-09-18T08:03:43.339 INFO:journalctl@ceph.alertmanager.a.smithi007.stdout:Sep 18 08:03:43 smithi007 ceph-779349ca-7593-11ef-bceb-c7b262605968-alertmanager-a[70326]: ts=2024-09-18T08:03:43.074Z caller=main.go:182 level=info build_context="(go=go1.21.7, platform=linux/amd64, user=root@22cd11f671e9, date=20240228-11:51:20, tags=netgo)" 2024-09-18T08:03:43.339 INFO:journalctl@ceph.alertmanager.a.smithi007.stdout:Sep 18 08:03:43 smithi007 ceph-779349ca-7593-11ef-bceb-c7b262605968-alertmanager-a[70326]: ts=2024-09-18T08:03:43.075Z caller=cluster.go:186 level=info component=cluster msg="setting advertise address explicitly" addr=172.21.15.7 port=9094 2024-09-18T08:03:43.339 INFO:journalctl@ceph.alertmanager.a.smithi007.stdout:Sep 18 08:03:43 smithi007 ceph-779349ca-7593-11ef-bceb-c7b262605968-alertmanager-a[70326]: ts=2024-09-18T08:03:43.076Z caller=cluster.go:683 level=info component=cluster msg="Waiting for gossip to settle..." interval=2s 2024-09-18T08:03:43.339 INFO:journalctl@ceph.alertmanager.a.smithi007.stdout:Sep 18 08:03:43 smithi007 ceph-779349ca-7593-11ef-bceb-c7b262605968-alertmanager-a[70326]: ts=2024-09-18T08:03:43.098Z caller=coordinator.go:113 level=info component=configuration msg="Loading configuration file" file=/etc/alertmanager/alertmanager.yml 2024-09-18T08:03:43.339 INFO:journalctl@ceph.alertmanager.a.smithi007.stdout:Sep 18 08:03:43 smithi007 ceph-779349ca-7593-11ef-bceb-c7b262605968-alertmanager-a[70326]: ts=2024-09-18T08:03:43.098Z caller=coordinator.go:126 level=info component=configuration msg="Completed loading of configuration file" file=/etc/alertmanager/alertmanager.yml 2024-09-18T08:03:43.339 INFO:journalctl@ceph.alertmanager.a.smithi007.stdout:Sep 18 08:03:43 smithi007 ceph-779349ca-7593-11ef-bceb-c7b262605968-alertmanager-a[70326]: ts=2024-09-18T08:03:43.100Z caller=tls_config.go:313 level=info msg="Listening on" address=[::]:9093 2024-09-18T08:03:43.339 INFO:journalctl@ceph.alertmanager.a.smithi007.stdout:Sep 18 08:03:43 smithi007 ceph-779349ca-7593-11ef-bceb-c7b262605968-alertmanager-a[70326]: ts=2024-09-18T08:03:43.100Z caller=tls_config.go:316 level=info msg="TLS is disabled." http2=false address=[::]:9093 2024-09-18T08:03:43.339 INFO:journalctl@ceph.alertmanager.a.smithi007.stdout:Sep 18 08:03:43 smithi007 bash[70287]: 5fd76c77cedf61a73045fde95d31e55a84b78edda9fe5b68ede971e9ff30b7a3 2024-09-18T08:03:43.339 INFO:journalctl@ceph.alertmanager.a.smithi007.stdout:Sep 18 08:03:43 smithi007 systemd[1]: Started Ceph alertmanager.a for 779349ca-7593-11ef-bceb-c7b262605968. 2024-09-18T08:03:43.868 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-09-18T08:03:43.870 INFO:teuthology.orchestra.run.smithi007.stderr:dumped all 2024-09-18T08:03:44.480 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:03:44 smithi007 ceph-mon[29459]: pgmap v21: 132 pgs: 132 active+clean; 583 KiB data, 223 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-18T08:03:44.480 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:03:44 smithi007 ceph-mon[29459]: from='mgr.24470 172.21.15.7:0/2594390261' entity='mgr.y' 2024-09-18T08:03:44.481 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:03:44 smithi007 ceph-mon[29459]: from='mgr.24470 172.21.15.7:0/2594390261' entity='mgr.y' 2024-09-18T08:03:44.481 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:03:44 smithi007 ceph-mon[29459]: from='mgr.24470 172.21.15.7:0/2594390261' entity='mgr.y' 2024-09-18T08:03:44.481 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:03:44 smithi007 ceph-mon[29459]: from='mgr.24470 172.21.15.7:0/2594390261' entity='mgr.y' 2024-09-18T08:03:44.481 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:03:44 smithi007 ceph-mon[29459]: Regenerating cephadm self-signed grafana TLS certificates 2024-09-18T08:03:44.481 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:03:44 smithi007 ceph-mon[39249]: pgmap v21: 132 pgs: 132 active+clean; 583 KiB data, 223 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-18T08:03:44.481 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:03:44 smithi007 ceph-mon[39249]: from='mgr.24470 172.21.15.7:0/2594390261' entity='mgr.y' 2024-09-18T08:03:44.481 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:03:44 smithi007 ceph-mon[39249]: from='mgr.24470 172.21.15.7:0/2594390261' entity='mgr.y' 2024-09-18T08:03:44.481 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:03:44 smithi007 ceph-mon[39249]: from='mgr.24470 172.21.15.7:0/2594390261' entity='mgr.y' 2024-09-18T08:03:44.481 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:03:44 smithi007 ceph-mon[39249]: from='mgr.24470 172.21.15.7:0/2594390261' entity='mgr.y' 2024-09-18T08:03:44.481 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:03:44 smithi007 ceph-mon[39249]: Regenerating cephadm self-signed grafana TLS certificates 2024-09-18T08:03:44.656 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:03:44 smithi050 ceph-mon[32371]: pgmap v21: 132 pgs: 132 active+clean; 583 KiB data, 223 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-18T08:03:44.656 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:03:44 smithi050 ceph-mon[32371]: from='mgr.24470 172.21.15.7:0/2594390261' entity='mgr.y' 2024-09-18T08:03:44.656 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:03:44 smithi050 ceph-mon[32371]: from='mgr.24470 172.21.15.7:0/2594390261' entity='mgr.y' 2024-09-18T08:03:44.656 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:03:44 smithi050 ceph-mon[32371]: from='mgr.24470 172.21.15.7:0/2594390261' entity='mgr.y' 2024-09-18T08:03:44.656 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:03:44 smithi050 ceph-mon[32371]: from='mgr.24470 172.21.15.7:0/2594390261' entity='mgr.y' 2024-09-18T08:03:44.656 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:03:44 smithi050 ceph-mon[32371]: Regenerating cephadm self-signed grafana TLS certificates 2024-09-18T08:03:44.851 INFO:teuthology.orchestra.run.smithi007.stdout:{"pg_ready":true,"pg_map":{"version":21,"stamp":"2024-09-18T08:03:43.072106+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":596507,"num_objects":199,"num_object_clones":0,"num_object_copies":597,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":199,"num_whiteouts":0,"num_read":893,"num_read_kb":750,"num_write":549,"num_write_kb":1673,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":590368,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":552,"ondisk_log_size":552,"up":396,"acting":396,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":396,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":228316,"kb_used_data":7932,"kb_used_omap":12,"kb_used_meta":220211,"kb_avail":749732900,"statfs":{"total":767960285184,"available":767726489600,"internally_reserved":0,"allocated":8122368,"data_stored":4304177,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12750,"internal_metadata":225496626},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.001906"},"pg_stats":[{"pgid":"3.1f","version":"0'0","reported_seq":44,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-18T08:03:34.047520+0000","last_change":"2024-09-18T08:02:47.878020+0000","last_active":"2024-09-18T08:03:34.047520+0000","last_peered":"2024-09-18T08:03:34.047520+0000","last_clean":"2024-09-18T08:03:34.047520+0000","last_became_active":"2024-09-18T08:02:47.877698+0000","last_became_peered":"2024-09-18T08:02:47.877698+0000","last_unstale":"2024-09-18T08:03:34.047520+0000","last_undegraded":"2024-09-18T08:03:34.047520+0000","last_fullsized":"2024-09-18T08:03:34.047520+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:02:46.860092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:46.860092+0000","last_clean_scrub_stamp":"2024-09-18T08:02:46.860092+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:04:14.205967+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"69'9","reported_seq":56,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-18T08:03:34.047259+0000","last_change":"2024-09-18T08:02:49.895688+0000","last_active":"2024-09-18T08:03:34.047259+0000","last_peered":"2024-09-18T08:03:34.047259+0000","last_clean":"2024-09-18T08:03:34.047259+0000","last_became_active":"2024-09-18T08:02:49.894557+0000","last_became_peered":"2024-09-18T08:02:49.894557+0000","last_unstale":"2024-09-18T08:03:34.047259+0000","last_undegraded":"2024-09-18T08:03:34.047259+0000","last_fullsized":"2024-09-18T08:03:34.047259+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_clean_scrub_stamp":"2024-09-18T08:02:48.870718+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:03:59.206088+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":36,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-18T08:03:35.231793+0000","last_change":"2024-09-18T08:02:51.907566+0000","last_active":"2024-09-18T08:03:35.231793+0000","last_peered":"2024-09-18T08:03:35.231793+0000","last_clean":"2024-09-18T08:03:35.231793+0000","last_became_active":"2024-09-18T08:02:51.907140+0000","last_became_peered":"2024-09-18T08:02:51.907140+0000","last_unstale":"2024-09-18T08:03:35.231793+0000","last_undegraded":"2024-09-18T08:03:35.231793+0000","last_fullsized":"2024-09-18T08:03:35.231793+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:02:50.886830+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:50.886830+0000","last_clean_scrub_stamp":"2024-09-18T08:02:50.886830+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:04:07.003335+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":33,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-18T08:03:34.047497+0000","last_change":"2024-09-18T08:02:53.906914+0000","last_active":"2024-09-18T08:03:34.047497+0000","last_peered":"2024-09-18T08:03:34.047497+0000","last_clean":"2024-09-18T08:03:34.047497+0000","last_became_active":"2024-09-18T08:02:53.906622+0000","last_became_peered":"2024-09-18T08:02:53.906622+0000","last_unstale":"2024-09-18T08:03:34.047497+0000","last_undegraded":"2024-09-18T08:03:34.047497+0000","last_fullsized":"2024-09-18T08:03:34.047497+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_clean_scrub_stamp":"2024-09-18T08:02:52.890475+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:04:20.738381+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":32,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-18T08:03:34.872031+0000","last_change":"2024-09-18T08:02:54.047999+0000","last_active":"2024-09-18T08:03:34.872031+0000","last_peered":"2024-09-18T08:03:34.872031+0000","last_clean":"2024-09-18T08:03:34.872031+0000","last_became_active":"2024-09-18T08:02:54.047640+0000","last_became_peered":"2024-09-18T08:02:54.047640+0000","last_unstale":"2024-09-18T08:03:34.872031+0000","last_undegraded":"2024-09-18T08:03:34.872031+0000","last_fullsized":"2024-09-18T08:03:34.872031+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_clean_scrub_stamp":"2024-09-18T08:02:52.890475+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:04:16.013706+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":44,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-18T08:03:34.872645+0000","last_change":"2024-09-18T08:02:47.872585+0000","last_active":"2024-09-18T08:03:34.872645+0000","last_peered":"2024-09-18T08:03:34.872645+0000","last_clean":"2024-09-18T08:03:34.872645+0000","last_became_active":"2024-09-18T08:02:47.872316+0000","last_became_peered":"2024-09-18T08:02:47.872316+0000","last_unstale":"2024-09-18T08:03:34.872645+0000","last_undegraded":"2024-09-18T08:03:34.872645+0000","last_fullsized":"2024-09-18T08:03:34.872645+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:02:46.860092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:46.860092+0000","last_clean_scrub_stamp":"2024-09-18T08:02:46.860092+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:03:53.731838+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"69'15","reported_seq":65,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-18T08:03:34.872036+0000","last_change":"2024-09-18T08:02:49.887111+0000","last_active":"2024-09-18T08:03:34.872036+0000","last_peered":"2024-09-18T08:03:34.872036+0000","last_clean":"2024-09-18T08:03:34.872036+0000","last_became_active":"2024-09-18T08:02:49.886106+0000","last_became_peered":"2024-09-18T08:02:49.886106+0000","last_unstale":"2024-09-18T08:03:34.872036+0000","last_undegraded":"2024-09-18T08:03:34.872036+0000","last_fullsized":"2024-09-18T08:03:34.872036+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_clean_scrub_stamp":"2024-09-18T08:02:48.870718+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:04:15.737211+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":36,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-18T08:03:34.047246+0000","last_change":"2024-09-18T08:02:51.904853+0000","last_active":"2024-09-18T08:03:34.047246+0000","last_peered":"2024-09-18T08:03:34.047246+0000","last_clean":"2024-09-18T08:03:34.047246+0000","last_became_active":"2024-09-18T08:02:51.904184+0000","last_became_peered":"2024-09-18T08:02:51.904184+0000","last_unstale":"2024-09-18T08:03:34.047246+0000","last_undegraded":"2024-09-18T08:03:34.047246+0000","last_fullsized":"2024-09-18T08:03:34.047246+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:02:50.886830+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:50.886830+0000","last_clean_scrub_stamp":"2024-09-18T08:02:50.886830+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:04:07.696637+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":44,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-18T08:03:34.762777+0000","last_change":"2024-09-18T08:02:47.875508+0000","last_active":"2024-09-18T08:03:34.762777+0000","last_peered":"2024-09-18T08:03:34.762777+0000","last_clean":"2024-09-18T08:03:34.762777+0000","last_became_active":"2024-09-18T08:02:47.875124+0000","last_became_peered":"2024-09-18T08:02:47.875124+0000","last_unstale":"2024-09-18T08:03:34.762777+0000","last_undegraded":"2024-09-18T08:03:34.762777+0000","last_fullsized":"2024-09-18T08:03:34.762777+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:02:46.860092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:46.860092+0000","last_clean_scrub_stamp":"2024-09-18T08:02:46.860092+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:03:53.477209+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"69'9","reported_seq":56,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-18T08:03:34.047761+0000","last_change":"2024-09-18T08:02:49.887589+0000","last_active":"2024-09-18T08:03:34.047761+0000","last_peered":"2024-09-18T08:03:34.047761+0000","last_clean":"2024-09-18T08:03:34.047761+0000","last_became_active":"2024-09-18T08:02:49.887161+0000","last_became_peered":"2024-09-18T08:02:49.887161+0000","last_unstale":"2024-09-18T08:03:34.047761+0000","last_undegraded":"2024-09-18T08:03:34.047761+0000","last_fullsized":"2024-09-18T08:03:34.047761+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_clean_scrub_stamp":"2024-09-18T08:02:48.870718+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:03:51.892069+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":36,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-18T08:03:34.762220+0000","last_change":"2024-09-18T08:02:51.907597+0000","last_active":"2024-09-18T08:03:34.762220+0000","last_peered":"2024-09-18T08:03:34.762220+0000","last_clean":"2024-09-18T08:03:34.762220+0000","last_became_active":"2024-09-18T08:02:51.907296+0000","last_became_peered":"2024-09-18T08:02:51.907296+0000","last_unstale":"2024-09-18T08:03:34.762220+0000","last_undegraded":"2024-09-18T08:03:34.762220+0000","last_fullsized":"2024-09-18T08:03:34.762220+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:02:50.886830+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:50.886830+0000","last_clean_scrub_stamp":"2024-09-18T08:02:50.886830+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:04:06.057644+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":33,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-18T08:03:34.046546+0000","last_change":"2024-09-18T08:02:53.913803+0000","last_active":"2024-09-18T08:03:34.046546+0000","last_peered":"2024-09-18T08:03:34.046546+0000","last_clean":"2024-09-18T08:03:34.046546+0000","last_became_active":"2024-09-18T08:02:53.913519+0000","last_became_peered":"2024-09-18T08:02:53.913519+0000","last_unstale":"2024-09-18T08:03:34.046546+0000","last_undegraded":"2024-09-18T08:03:34.046546+0000","last_fullsized":"2024-09-18T08:03:34.046546+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_clean_scrub_stamp":"2024-09-18T08:02:52.890475+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:04:18.699540+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":44,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-18T08:03:34.762467+0000","last_change":"2024-09-18T08:02:47.877577+0000","last_active":"2024-09-18T08:03:34.762467+0000","last_peered":"2024-09-18T08:03:34.762467+0000","last_clean":"2024-09-18T08:03:34.762467+0000","last_became_active":"2024-09-18T08:02:47.875042+0000","last_became_peered":"2024-09-18T08:02:47.875042+0000","last_unstale":"2024-09-18T08:03:34.762467+0000","last_undegraded":"2024-09-18T08:03:34.762467+0000","last_fullsized":"2024-09-18T08:03:34.762467+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:02:46.860092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:46.860092+0000","last_clean_scrub_stamp":"2024-09-18T08:02:46.860092+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:04:14.469212+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1b","version":"69'5","reported_seq":50,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-18T08:03:34.047942+0000","last_change":"2024-09-18T08:02:49.895779+0000","last_active":"2024-09-18T08:03:34.047942+0000","last_peered":"2024-09-18T08:03:34.047942+0000","last_clean":"2024-09-18T08:03:34.047942+0000","last_became_active":"2024-09-18T08:02:49.895344+0000","last_became_peered":"2024-09-18T08:02:49.895344+0000","last_unstale":"2024-09-18T08:03:34.047942+0000","last_undegraded":"2024-09-18T08:03:34.047942+0000","last_fullsized":"2024-09-18T08:03:34.047942+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_clean_scrub_stamp":"2024-09-18T08:02:48.870718+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:04:08.365283+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":11,"num_read_kb":7,"num_write":6,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":36,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-18T08:03:34.572593+0000","last_change":"2024-09-18T08:02:51.906694+0000","last_active":"2024-09-18T08:03:34.572593+0000","last_peered":"2024-09-18T08:03:34.572593+0000","last_clean":"2024-09-18T08:03:34.572593+0000","last_became_active":"2024-09-18T08:02:51.905776+0000","last_became_peered":"2024-09-18T08:02:51.905776+0000","last_unstale":"2024-09-18T08:03:34.572593+0000","last_undegraded":"2024-09-18T08:03:34.572593+0000","last_fullsized":"2024-09-18T08:03:34.572593+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:02:50.886830+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:50.886830+0000","last_clean_scrub_stamp":"2024-09-18T08:02:50.886830+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:04:07.273083+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":33,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-18T08:03:34.762025+0000","last_change":"2024-09-18T08:02:53.911969+0000","last_active":"2024-09-18T08:03:34.762025+0000","last_peered":"2024-09-18T08:03:34.762025+0000","last_clean":"2024-09-18T08:03:34.762025+0000","last_became_active":"2024-09-18T08:02:53.911648+0000","last_became_peered":"2024-09-18T08:02:53.911648+0000","last_unstale":"2024-09-18T08:03:34.762025+0000","last_undegraded":"2024-09-18T08:03:34.762025+0000","last_fullsized":"2024-09-18T08:03:34.762025+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_clean_scrub_stamp":"2024-09-18T08:02:52.890475+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:04:21.185524+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":32,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-18T08:03:34.047503+0000","last_change":"2024-09-18T08:02:54.047071+0000","last_active":"2024-09-18T08:03:34.047503+0000","last_peered":"2024-09-18T08:03:34.047503+0000","last_clean":"2024-09-18T08:03:34.047503+0000","last_became_active":"2024-09-18T08:02:54.046776+0000","last_became_peered":"2024-09-18T08:02:54.046776+0000","last_unstale":"2024-09-18T08:03:34.047503+0000","last_undegraded":"2024-09-18T08:03:34.047503+0000","last_fullsized":"2024-09-18T08:03:34.047503+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_clean_scrub_stamp":"2024-09-18T08:02:52.890475+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:04:11.355797+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":44,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-18T08:03:34.047827+0000","last_change":"2024-09-18T08:02:47.876235+0000","last_active":"2024-09-18T08:03:34.047827+0000","last_peered":"2024-09-18T08:03:34.047827+0000","last_clean":"2024-09-18T08:03:34.047827+0000","last_became_active":"2024-09-18T08:02:47.875963+0000","last_became_peered":"2024-09-18T08:02:47.875963+0000","last_unstale":"2024-09-18T08:03:34.047827+0000","last_undegraded":"2024-09-18T08:03:34.047827+0000","last_fullsized":"2024-09-18T08:03:34.047827+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:02:46.860092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:46.860092+0000","last_clean_scrub_stamp":"2024-09-18T08:02:46.860092+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:04:11.609411+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1c","version":"69'15","reported_seq":65,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-18T08:03:34.049141+0000","last_change":"2024-09-18T08:02:49.885923+0000","last_active":"2024-09-18T08:03:34.049141+0000","last_peered":"2024-09-18T08:03:34.049141+0000","last_clean":"2024-09-18T08:03:34.049141+0000","last_became_active":"2024-09-18T08:02:49.885553+0000","last_became_peered":"2024-09-18T08:02:49.885553+0000","last_unstale":"2024-09-18T08:03:34.049141+0000","last_undegraded":"2024-09-18T08:03:34.049141+0000","last_fullsized":"2024-09-18T08:03:34.049141+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_clean_scrub_stamp":"2024-09-18T08:02:48.870718+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:04:11.220046+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":36,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-18T08:03:35.232338+0000","last_change":"2024-09-18T08:02:51.908750+0000","last_active":"2024-09-18T08:03:35.232338+0000","last_peered":"2024-09-18T08:03:35.232338+0000","last_clean":"2024-09-18T08:03:35.232338+0000","last_became_active":"2024-09-18T08:02:51.908430+0000","last_became_peered":"2024-09-18T08:02:51.908430+0000","last_unstale":"2024-09-18T08:03:35.232338+0000","last_undegraded":"2024-09-18T08:03:35.232338+0000","last_fullsized":"2024-09-18T08:03:35.232338+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:02:50.886830+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:50.886830+0000","last_clean_scrub_stamp":"2024-09-18T08:02:50.886830+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:03:58.654545+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":32,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-18T08:03:34.872572+0000","last_change":"2024-09-18T08:02:54.047573+0000","last_active":"2024-09-18T08:03:34.872572+0000","last_peered":"2024-09-18T08:03:34.872572+0000","last_clean":"2024-09-18T08:03:34.872572+0000","last_became_active":"2024-09-18T08:02:54.047162+0000","last_became_peered":"2024-09-18T08:02:54.047162+0000","last_unstale":"2024-09-18T08:03:34.872572+0000","last_undegraded":"2024-09-18T08:03:34.872572+0000","last_fullsized":"2024-09-18T08:03:34.872572+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_clean_scrub_stamp":"2024-09-18T08:02:52.890475+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:04:02.287865+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":44,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-18T08:03:34.047961+0000","last_change":"2024-09-18T08:02:47.871895+0000","last_active":"2024-09-18T08:03:34.047961+0000","last_peered":"2024-09-18T08:03:34.047961+0000","last_clean":"2024-09-18T08:03:34.047961+0000","last_became_active":"2024-09-18T08:02:47.871523+0000","last_became_peered":"2024-09-18T08:02:47.871523+0000","last_unstale":"2024-09-18T08:03:34.047961+0000","last_undegraded":"2024-09-18T08:03:34.047961+0000","last_fullsized":"2024-09-18T08:03:34.047961+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:02:46.860092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:46.860092+0000","last_clean_scrub_stamp":"2024-09-18T08:02:46.860092+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:04:03.333307+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"69'12","reported_seq":63,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-18T08:03:34.872420+0000","last_change":"2024-09-18T08:02:49.889182+0000","last_active":"2024-09-18T08:03:34.872420+0000","last_peered":"2024-09-18T08:03:34.872420+0000","last_clean":"2024-09-18T08:03:34.872420+0000","last_became_active":"2024-09-18T08:02:49.888831+0000","last_became_peered":"2024-09-18T08:02:49.888831+0000","last_unstale":"2024-09-18T08:03:34.872420+0000","last_undegraded":"2024-09-18T08:03:34.872420+0000","last_fullsized":"2024-09-18T08:03:34.872420+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_clean_scrub_stamp":"2024-09-18T08:02:48.870718+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:04:12.415919+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":25,"num_read_kb":16,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":36,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-18T08:03:34.048003+0000","last_change":"2024-09-18T08:02:51.910642+0000","last_active":"2024-09-18T08:03:34.048003+0000","last_peered":"2024-09-18T08:03:34.048003+0000","last_clean":"2024-09-18T08:03:34.048003+0000","last_became_active":"2024-09-18T08:02:51.910316+0000","last_became_peered":"2024-09-18T08:02:51.910316+0000","last_unstale":"2024-09-18T08:03:34.048003+0000","last_undegraded":"2024-09-18T08:03:34.048003+0000","last_fullsized":"2024-09-18T08:03:34.048003+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:02:50.886830+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:50.886830+0000","last_clean_scrub_stamp":"2024-09-18T08:02:50.886830+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:04:05.261448+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1c","version":"69'1","reported_seq":34,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-18T08:03:34.572852+0000","last_change":"2024-09-18T08:02:53.910861+0000","last_active":"2024-09-18T08:03:34.572852+0000","last_peered":"2024-09-18T08:03:34.572852+0000","last_clean":"2024-09-18T08:03:34.572852+0000","last_became_active":"2024-09-18T08:02:53.910589+0000","last_became_peered":"2024-09-18T08:02:53.910589+0000","last_unstale":"2024-09-18T08:03:34.572852+0000","last_undegraded":"2024-09-18T08:03:34.572852+0000","last_fullsized":"2024-09-18T08:03:34.572852+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_clean_scrub_stamp":"2024-09-18T08:02:52.890475+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:04:20.224154+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":403,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.19","version":"62'1","reported_seq":45,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-18T08:03:35.231922+0000","last_change":"2024-09-18T08:02:47.881314+0000","last_active":"2024-09-18T08:03:35.231922+0000","last_peered":"2024-09-18T08:03:35.231922+0000","last_clean":"2024-09-18T08:03:35.231922+0000","last_became_active":"2024-09-18T08:02:47.880998+0000","last_became_peered":"2024-09-18T08:02:47.880998+0000","last_unstale":"2024-09-18T08:03:35.231922+0000","last_undegraded":"2024-09-18T08:03:35.231922+0000","last_fullsized":"2024-09-18T08:03:35.231922+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:02:46.860092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:46.860092+0000","last_clean_scrub_stamp":"2024-09-18T08:02:46.860092+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:03:57.000100+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1e","version":"69'10","reported_seq":55,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-18T08:03:34.047651+0000","last_change":"2024-09-18T08:02:49.892755+0000","last_active":"2024-09-18T08:03:34.047651+0000","last_peered":"2024-09-18T08:03:34.047651+0000","last_clean":"2024-09-18T08:03:34.047651+0000","last_became_active":"2024-09-18T08:02:49.892024+0000","last_became_peered":"2024-09-18T08:02:49.892024+0000","last_unstale":"2024-09-18T08:03:34.047651+0000","last_undegraded":"2024-09-18T08:03:34.047651+0000","last_fullsized":"2024-09-18T08:03:34.047651+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_clean_scrub_stamp":"2024-09-18T08:02:48.870718+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:04:17.237822+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1f","version":"69'8","reported_seq":58,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-18T08:03:34.497813+0000","last_change":"2024-09-18T08:02:51.904933+0000","last_active":"2024-09-18T08:03:34.497813+0000","last_peered":"2024-09-18T08:03:34.497813+0000","last_clean":"2024-09-18T08:03:34.497813+0000","last_became_active":"2024-09-18T08:02:51.904588+0000","last_became_peered":"2024-09-18T08:02:51.904588+0000","last_unstale":"2024-09-18T08:03:34.497813+0000","last_undegraded":"2024-09-18T08:03:34.497813+0000","last_fullsized":"2024-09-18T08:03:34.497813+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:02:50.886830+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:50.886830+0000","last_clean_scrub_stamp":"2024-09-18T08:02:50.886830+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:03:59.243324+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.f","version":"69'15","reported_seq":65,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-18T08:03:35.232547+0000","last_change":"2024-09-18T08:02:49.899404+0000","last_active":"2024-09-18T08:03:35.232547+0000","last_peered":"2024-09-18T08:03:35.232547+0000","last_clean":"2024-09-18T08:03:35.232547+0000","last_became_active":"2024-09-18T08:02:49.899083+0000","last_became_peered":"2024-09-18T08:02:49.899083+0000","last_unstale":"2024-09-18T08:03:35.232547+0000","last_undegraded":"2024-09-18T08:03:35.232547+0000","last_fullsized":"2024-09-18T08:03:35.232547+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_clean_scrub_stamp":"2024-09-18T08:02:48.870718+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:04:06.513250+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":44,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-18T08:03:34.871892+0000","last_change":"2024-09-18T08:02:47.879723+0000","last_active":"2024-09-18T08:03:34.871892+0000","last_peered":"2024-09-18T08:03:34.871892+0000","last_clean":"2024-09-18T08:03:34.871892+0000","last_became_active":"2024-09-18T08:02:47.879328+0000","last_became_peered":"2024-09-18T08:02:47.879328+0000","last_unstale":"2024-09-18T08:03:34.871892+0000","last_undegraded":"2024-09-18T08:03:34.871892+0000","last_fullsized":"2024-09-18T08:03:34.871892+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:02:46.860092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:46.860092+0000","last_clean_scrub_stamp":"2024-09-18T08:02:46.860092+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:03:56.802301+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.e","version":"69'8","reported_seq":55,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-18T08:03:34.048746+0000","last_change":"2024-09-18T08:02:51.903673+0000","last_active":"2024-09-18T08:03:34.048746+0000","last_peered":"2024-09-18T08:03:34.048746+0000","last_clean":"2024-09-18T08:03:34.048746+0000","last_became_active":"2024-09-18T08:02:51.903369+0000","last_became_peered":"2024-09-18T08:02:51.903369+0000","last_unstale":"2024-09-18T08:03:34.048746+0000","last_undegraded":"2024-09-18T08:03:34.048746+0000","last_fullsized":"2024-09-18T08:03:34.048746+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:02:50.886830+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:50.886830+0000","last_clean_scrub_stamp":"2024-09-18T08:02:50.886830+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:03:51.149147+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":33,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-18T08:03:34.762381+0000","last_change":"2024-09-18T08:02:53.910161+0000","last_active":"2024-09-18T08:03:34.762381+0000","last_peered":"2024-09-18T08:03:34.762381+0000","last_clean":"2024-09-18T08:03:34.762381+0000","last_became_active":"2024-09-18T08:02:53.909834+0000","last_became_peered":"2024-09-18T08:02:53.909834+0000","last_unstale":"2024-09-18T08:03:34.762381+0000","last_undegraded":"2024-09-18T08:03:34.762381+0000","last_fullsized":"2024-09-18T08:03:34.762381+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_clean_scrub_stamp":"2024-09-18T08:02:52.890475+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:04:07.435248+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.0","version":"69'18","reported_seq":72,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-18T08:03:34.871648+0000","last_change":"2024-09-18T08:02:49.887135+0000","last_active":"2024-09-18T08:03:34.871648+0000","last_peered":"2024-09-18T08:03:34.871648+0000","last_clean":"2024-09-18T08:03:34.871648+0000","last_became_active":"2024-09-18T08:02:49.886818+0000","last_became_peered":"2024-09-18T08:02:49.886818+0000","last_unstale":"2024-09-18T08:03:34.871648+0000","last_undegraded":"2024-09-18T08:03:34.871648+0000","last_fullsized":"2024-09-18T08:03:34.871648+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_clean_scrub_stamp":"2024-09-18T08:02:48.870718+0000","objects_scrubbed":0,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:03:57.656879+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":34,"num_read_kb":22,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":44,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-18T08:03:34.872823+0000","last_change":"2024-09-18T08:02:47.880517+0000","last_active":"2024-09-18T08:03:34.872823+0000","last_peered":"2024-09-18T08:03:34.872823+0000","last_clean":"2024-09-18T08:03:34.872823+0000","last_became_active":"2024-09-18T08:02:47.880225+0000","last_became_peered":"2024-09-18T08:02:47.880225+0000","last_unstale":"2024-09-18T08:03:34.872823+0000","last_undegraded":"2024-09-18T08:03:34.872823+0000","last_fullsized":"2024-09-18T08:03:34.872823+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:02:46.860092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:46.860092+0000","last_clean_scrub_stamp":"2024-09-18T08:02:46.860092+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:03:51.223493+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":36,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-18T08:03:34.047272+0000","last_change":"2024-09-18T08:02:51.899911+0000","last_active":"2024-09-18T08:03:34.047272+0000","last_peered":"2024-09-18T08:03:34.047272+0000","last_clean":"2024-09-18T08:03:34.047272+0000","last_became_active":"2024-09-18T08:02:51.899594+0000","last_became_peered":"2024-09-18T08:02:51.899594+0000","last_unstale":"2024-09-18T08:03:34.047272+0000","last_undegraded":"2024-09-18T08:03:34.047272+0000","last_fullsized":"2024-09-18T08:03:34.047272+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:02:50.886830+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:50.886830+0000","last_clean_scrub_stamp":"2024-09-18T08:02:50.886830+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:04:02.261346+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":33,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-18T08:03:34.047646+0000","last_change":"2024-09-18T08:02:53.904262+0000","last_active":"2024-09-18T08:03:34.047646+0000","last_peered":"2024-09-18T08:03:34.047646+0000","last_clean":"2024-09-18T08:03:34.047646+0000","last_became_active":"2024-09-18T08:02:53.903957+0000","last_became_peered":"2024-09-18T08:02:53.903957+0000","last_unstale":"2024-09-18T08:03:34.047646+0000","last_undegraded":"2024-09-18T08:03:34.047646+0000","last_fullsized":"2024-09-18T08:03:34.047646+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_clean_scrub_stamp":"2024-09-18T08:02:52.890475+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:04:02.672046+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1","version":"69'14","reported_seq":61,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-18T08:03:34.047364+0000","last_change":"2024-09-18T08:02:49.894205+0000","last_active":"2024-09-18T08:03:34.047364+0000","last_peered":"2024-09-18T08:03:34.047364+0000","last_clean":"2024-09-18T08:03:34.047364+0000","last_became_active":"2024-09-18T08:02:49.893830+0000","last_became_peered":"2024-09-18T08:02:49.893830+0000","last_unstale":"2024-09-18T08:03:34.047364+0000","last_undegraded":"2024-09-18T08:03:34.047364+0000","last_fullsized":"2024-09-18T08:03:34.047364+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_clean_scrub_stamp":"2024-09-18T08:02:48.870718+0000","objects_scrubbed":0,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:03:51.325184+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":21,"num_read_kb":14,"num_write":14,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"62'2","reported_seq":51,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-18T08:03:34.047774+0000","last_change":"2024-09-18T08:02:47.873172+0000","last_active":"2024-09-18T08:03:34.047774+0000","last_peered":"2024-09-18T08:03:34.047774+0000","last_clean":"2024-09-18T08:03:34.047774+0000","last_became_active":"2024-09-18T08:02:47.872857+0000","last_became_peered":"2024-09-18T08:02:47.872857+0000","last_unstale":"2024-09-18T08:03:34.047774+0000","last_undegraded":"2024-09-18T08:03:34.047774+0000","last_fullsized":"2024-09-18T08:03:34.047774+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:02:46.860092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:46.860092+0000","last_clean_scrub_stamp":"2024-09-18T08:02:46.860092+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:04:03.733922+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1039,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.0","version":"69'8","reported_seq":55,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-18T08:03:34.871593+0000","last_change":"2024-09-18T08:02:51.910339+0000","last_active":"2024-09-18T08:03:34.871593+0000","last_peered":"2024-09-18T08:03:34.871593+0000","last_clean":"2024-09-18T08:03:34.871593+0000","last_became_active":"2024-09-18T08:02:51.909680+0000","last_became_peered":"2024-09-18T08:02:51.909680+0000","last_unstale":"2024-09-18T08:03:34.871593+0000","last_undegraded":"2024-09-18T08:03:34.871593+0000","last_fullsized":"2024-09-18T08:03:34.871593+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:02:50.886830+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:50.886830+0000","last_clean_scrub_stamp":"2024-09-18T08:02:50.886830+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:03:58.456713+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":33,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-18T08:03:34.572209+0000","last_change":"2024-09-18T08:02:54.047452+0000","last_active":"2024-09-18T08:03:34.572209+0000","last_peered":"2024-09-18T08:03:34.572209+0000","last_clean":"2024-09-18T08:03:34.572209+0000","last_became_active":"2024-09-18T08:02:54.047039+0000","last_became_peered":"2024-09-18T08:02:54.047039+0000","last_unstale":"2024-09-18T08:03:34.572209+0000","last_undegraded":"2024-09-18T08:03:34.572209+0000","last_fullsized":"2024-09-18T08:03:34.572209+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_clean_scrub_stamp":"2024-09-18T08:02:52.890475+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:03:58.508399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.2","version":"69'10","reported_seq":55,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-18T08:03:35.232029+0000","last_change":"2024-09-18T08:02:49.896889+0000","last_active":"2024-09-18T08:03:35.232029+0000","last_peered":"2024-09-18T08:03:35.232029+0000","last_clean":"2024-09-18T08:03:35.232029+0000","last_became_active":"2024-09-18T08:02:49.895265+0000","last_became_peered":"2024-09-18T08:02:49.895265+0000","last_unstale":"2024-09-18T08:03:35.232029+0000","last_undegraded":"2024-09-18T08:03:35.232029+0000","last_fullsized":"2024-09-18T08:03:35.232029+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_clean_scrub_stamp":"2024-09-18T08:02:48.870718+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:04:06.130656+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":44,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-18T08:03:34.762826+0000","last_change":"2024-09-18T08:02:47.877672+0000","last_active":"2024-09-18T08:03:34.762826+0000","last_peered":"2024-09-18T08:03:34.762826+0000","last_clean":"2024-09-18T08:03:34.762826+0000","last_became_active":"2024-09-18T08:02:47.874727+0000","last_became_peered":"2024-09-18T08:02:47.874727+0000","last_unstale":"2024-09-18T08:03:34.762826+0000","last_undegraded":"2024-09-18T08:03:34.762826+0000","last_fullsized":"2024-09-18T08:03:34.762826+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:02:46.860092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:46.860092+0000","last_clean_scrub_stamp":"2024-09-18T08:02:46.860092+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:03:47.082997+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.3","version":"69'8","reported_seq":56,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-18T08:03:37.938414+0000","last_change":"2024-09-18T08:02:51.906369+0000","last_active":"2024-09-18T08:03:37.938414+0000","last_peered":"2024-09-18T08:03:37.938414+0000","last_clean":"2024-09-18T08:03:37.938414+0000","last_became_active":"2024-09-18T08:02:51.906055+0000","last_became_peered":"2024-09-18T08:02:51.906055+0000","last_unstale":"2024-09-18T08:03:37.938414+0000","last_undegraded":"2024-09-18T08:03:37.938414+0000","last_fullsized":"2024-09-18T08:03:37.938414+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:02:50.886830+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:50.886830+0000","last_clean_scrub_stamp":"2024-09-18T08:02:50.886830+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:04:18.452195+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":33,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-18T08:03:34.046643+0000","last_change":"2024-09-18T08:02:53.910057+0000","last_active":"2024-09-18T08:03:34.046643+0000","last_peered":"2024-09-18T08:03:34.046643+0000","last_clean":"2024-09-18T08:03:34.046643+0000","last_became_active":"2024-09-18T08:02:53.909583+0000","last_became_peered":"2024-09-18T08:02:53.909583+0000","last_unstale":"2024-09-18T08:03:34.046643+0000","last_undegraded":"2024-09-18T08:03:34.046643+0000","last_fullsized":"2024-09-18T08:03:34.046643+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_clean_scrub_stamp":"2024-09-18T08:02:52.890475+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:04:07.545769+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.3","version":"69'19","reported_seq":76,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-18T08:03:34.047555+0000","last_change":"2024-09-18T08:02:49.890631+0000","last_active":"2024-09-18T08:03:34.047555+0000","last_peered":"2024-09-18T08:03:34.047555+0000","last_clean":"2024-09-18T08:03:34.047555+0000","last_became_active":"2024-09-18T08:02:49.890318+0000","last_became_peered":"2024-09-18T08:02:49.890318+0000","last_unstale":"2024-09-18T08:03:34.047555+0000","last_undegraded":"2024-09-18T08:03:34.047555+0000","last_fullsized":"2024-09-18T08:03:34.047555+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_clean_scrub_stamp":"2024-09-18T08:02:48.870718+0000","objects_scrubbed":0,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:03:57.953144+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":39,"num_read_kb":25,"num_write":22,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.4","version":"62'1","reported_seq":50,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-18T08:03:35.232342+0000","last_change":"2024-09-18T08:02:47.875046+0000","last_active":"2024-09-18T08:03:35.232342+0000","last_peered":"2024-09-18T08:03:35.232342+0000","last_clean":"2024-09-18T08:03:35.232342+0000","last_became_active":"2024-09-18T08:02:47.874764+0000","last_became_peered":"2024-09-18T08:02:47.874764+0000","last_unstale":"2024-09-18T08:03:35.232342+0000","last_undegraded":"2024-09-18T08:03:35.232342+0000","last_fullsized":"2024-09-18T08:03:35.232342+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:02:46.860092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:46.860092+0000","last_clean_scrub_stamp":"2024-09-18T08:02:46.860092+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:04:07.354571+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":436,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":36,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-18T08:03:34.497175+0000","last_change":"2024-09-18T08:02:51.908344+0000","last_active":"2024-09-18T08:03:34.497175+0000","last_peered":"2024-09-18T08:03:34.497175+0000","last_clean":"2024-09-18T08:03:34.497175+0000","last_became_active":"2024-09-18T08:02:51.908068+0000","last_became_peered":"2024-09-18T08:02:51.908068+0000","last_unstale":"2024-09-18T08:03:34.497175+0000","last_undegraded":"2024-09-18T08:03:34.497175+0000","last_fullsized":"2024-09-18T08:03:34.497175+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:02:50.886830+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:50.886830+0000","last_clean_scrub_stamp":"2024-09-18T08:02:50.886830+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:04:06.242055+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":32,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-18T08:03:35.231750+0000","last_change":"2024-09-18T08:02:54.046350+0000","last_active":"2024-09-18T08:03:35.231750+0000","last_peered":"2024-09-18T08:03:35.231750+0000","last_clean":"2024-09-18T08:03:35.231750+0000","last_became_active":"2024-09-18T08:02:54.045726+0000","last_became_peered":"2024-09-18T08:02:54.045726+0000","last_unstale":"2024-09-18T08:03:35.231750+0000","last_undegraded":"2024-09-18T08:03:35.231750+0000","last_fullsized":"2024-09-18T08:03:35.231750+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_clean_scrub_stamp":"2024-09-18T08:02:52.890475+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:03:55.585366+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.4","version":"69'28","reported_seq":99,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-18T08:03:35.232279+0000","last_change":"2024-09-18T08:02:49.897515+0000","last_active":"2024-09-18T08:03:35.232279+0000","last_peered":"2024-09-18T08:03:35.232279+0000","last_clean":"2024-09-18T08:03:35.232279+0000","last_became_active":"2024-09-18T08:02:49.897221+0000","last_became_peered":"2024-09-18T08:02:49.897221+0000","last_unstale":"2024-09-18T08:03:35.232279+0000","last_undegraded":"2024-09-18T08:03:35.232279+0000","last_fullsized":"2024-09-18T08:03:35.232279+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_clean_scrub_stamp":"2024-09-18T08:02:48.870718+0000","objects_scrubbed":0,"log_size":28,"log_dups_size":0,"ondisk_log_size":28,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:04:04.818428+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":48,"num_read_kb":33,"num_write":26,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":44,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-18T08:03:34.048351+0000","last_change":"2024-09-18T08:02:47.877445+0000","last_active":"2024-09-18T08:03:34.048351+0000","last_peered":"2024-09-18T08:03:34.048351+0000","last_clean":"2024-09-18T08:03:34.048351+0000","last_became_active":"2024-09-18T08:02:47.877085+0000","last_became_peered":"2024-09-18T08:02:47.877085+0000","last_unstale":"2024-09-18T08:03:34.048351+0000","last_undegraded":"2024-09-18T08:03:34.048351+0000","last_fullsized":"2024-09-18T08:03:34.048351+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:02:46.860092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:46.860092+0000","last_clean_scrub_stamp":"2024-09-18T08:02:46.860092+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:04:09.365906+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"64'2","reported_seq":54,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-18T08:03:34.762258+0000","last_change":"2024-09-18T08:02:49.883480+0000","last_active":"2024-09-18T08:03:34.762258+0000","last_peered":"2024-09-18T08:03:34.762258+0000","last_clean":"2024-09-18T08:03:34.762258+0000","last_became_active":"2024-09-18T08:02:46.867802+0000","last_became_peered":"2024-09-18T08:02:46.867802+0000","last_unstale":"2024-09-18T08:03:34.762258+0000","last_undegraded":"2024-09-18T08:03:34.762258+0000","last_fullsized":"2024-09-18T08:03:34.762258+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:02:45.855979+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:45.855979+0000","last_clean_scrub_stamp":"2024-09-18T08:02:45.855979+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:04:13.406442+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001348349,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":36,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-18T08:03:34.047360+0000","last_change":"2024-09-18T08:02:51.905694+0000","last_active":"2024-09-18T08:03:34.047360+0000","last_peered":"2024-09-18T08:03:34.047360+0000","last_clean":"2024-09-18T08:03:34.047360+0000","last_became_active":"2024-09-18T08:02:51.905357+0000","last_became_peered":"2024-09-18T08:02:51.905357+0000","last_unstale":"2024-09-18T08:03:34.047360+0000","last_undegraded":"2024-09-18T08:03:34.047360+0000","last_fullsized":"2024-09-18T08:03:34.047360+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:02:50.886830+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:50.886830+0000","last_clean_scrub_stamp":"2024-09-18T08:02:50.886830+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:04:15.739565+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":33,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-18T08:03:34.872458+0000","last_change":"2024-09-18T08:02:53.911634+0000","last_active":"2024-09-18T08:03:34.872458+0000","last_peered":"2024-09-18T08:03:34.872458+0000","last_clean":"2024-09-18T08:03:34.872458+0000","last_became_active":"2024-09-18T08:02:53.911036+0000","last_became_peered":"2024-09-18T08:02:53.911036+0000","last_unstale":"2024-09-18T08:03:34.872458+0000","last_undegraded":"2024-09-18T08:03:34.872458+0000","last_fullsized":"2024-09-18T08:03:34.872458+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_clean_scrub_stamp":"2024-09-18T08:02:52.890475+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:04:16.930353+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.7","version":"69'13","reported_seq":67,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-18T08:03:35.232554+0000","last_change":"2024-09-18T08:02:49.899404+0000","last_active":"2024-09-18T08:03:35.232554+0000","last_peered":"2024-09-18T08:03:35.232554+0000","last_clean":"2024-09-18T08:03:35.232554+0000","last_became_active":"2024-09-18T08:02:49.898987+0000","last_became_peered":"2024-09-18T08:02:49.898987+0000","last_unstale":"2024-09-18T08:03:35.232554+0000","last_undegraded":"2024-09-18T08:03:35.232554+0000","last_fullsized":"2024-09-18T08:03:35.232554+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_clean_scrub_stamp":"2024-09-18T08:02:48.870718+0000","objects_scrubbed":0,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:03:50.031411+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":19,"num_write":16,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":44,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-18T08:03:35.232038+0000","last_change":"2024-09-18T08:02:47.871732+0000","last_active":"2024-09-18T08:03:35.232038+0000","last_peered":"2024-09-18T08:03:35.232038+0000","last_clean":"2024-09-18T08:03:35.232038+0000","last_became_active":"2024-09-18T08:02:47.871416+0000","last_became_peered":"2024-09-18T08:02:47.871416+0000","last_unstale":"2024-09-18T08:03:35.232038+0000","last_undegraded":"2024-09-18T08:03:35.232038+0000","last_fullsized":"2024-09-18T08:03:35.232038+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:02:46.860092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:46.860092+0000","last_clean_scrub_stamp":"2024-09-18T08:02:46.860092+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:03:48.825242+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"62'1","reported_seq":53,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-18T08:03:34.048036+0000","last_change":"2024-09-18T08:02:49.877280+0000","last_active":"2024-09-18T08:03:34.048036+0000","last_peered":"2024-09-18T08:03:34.048036+0000","last_clean":"2024-09-18T08:03:34.048036+0000","last_became_active":"2024-09-18T08:02:46.868175+0000","last_became_peered":"2024-09-18T08:02:46.868175+0000","last_unstale":"2024-09-18T08:03:34.048036+0000","last_undegraded":"2024-09-18T08:03:34.048036+0000","last_fullsized":"2024-09-18T08:03:34.048036+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:02:45.855979+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:45.855979+0000","last_clean_scrub_stamp":"2024-09-18T08:02:45.855979+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:04:02.537405+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00066785499999999997,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":36,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-18T08:03:34.048609+0000","last_change":"2024-09-18T08:02:51.908417+0000","last_active":"2024-09-18T08:03:34.048609+0000","last_peered":"2024-09-18T08:03:34.048609+0000","last_clean":"2024-09-18T08:03:34.048609+0000","last_became_active":"2024-09-18T08:02:51.908156+0000","last_became_peered":"2024-09-18T08:02:51.908156+0000","last_unstale":"2024-09-18T08:03:34.048609+0000","last_undegraded":"2024-09-18T08:03:34.048609+0000","last_fullsized":"2024-09-18T08:03:34.048609+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:02:50.886830+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:50.886830+0000","last_clean_scrub_stamp":"2024-09-18T08:02:50.886830+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:04:04.453603+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":33,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-18T08:03:34.572969+0000","last_change":"2024-09-18T08:02:54.048417+0000","last_active":"2024-09-18T08:03:34.572969+0000","last_peered":"2024-09-18T08:03:34.572969+0000","last_clean":"2024-09-18T08:03:34.572969+0000","last_became_active":"2024-09-18T08:02:54.048004+0000","last_became_peered":"2024-09-18T08:02:54.048004+0000","last_unstale":"2024-09-18T08:03:34.572969+0000","last_undegraded":"2024-09-18T08:03:34.572969+0000","last_fullsized":"2024-09-18T08:03:34.572969+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_clean_scrub_stamp":"2024-09-18T08:02:52.890475+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:04:20.672566+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"69'12","reported_seq":58,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-18T08:03:34.047686+0000","last_change":"2024-09-18T08:02:49.898383+0000","last_active":"2024-09-18T08:03:34.047686+0000","last_peered":"2024-09-18T08:03:34.047686+0000","last_clean":"2024-09-18T08:03:34.047686+0000","last_became_active":"2024-09-18T08:02:49.898079+0000","last_became_peered":"2024-09-18T08:02:49.898079+0000","last_unstale":"2024-09-18T08:03:34.047686+0000","last_undegraded":"2024-09-18T08:03:34.047686+0000","last_fullsized":"2024-09-18T08:03:34.047686+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_clean_scrub_stamp":"2024-09-18T08:02:48.870718+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:04:14.339135+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":18,"num_read_kb":12,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":44,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-18T08:03:34.046872+0000","last_change":"2024-09-18T08:02:47.874335+0000","last_active":"2024-09-18T08:03:34.046872+0000","last_peered":"2024-09-18T08:03:34.046872+0000","last_clean":"2024-09-18T08:03:34.046872+0000","last_became_active":"2024-09-18T08:02:47.873809+0000","last_became_peered":"2024-09-18T08:02:47.873809+0000","last_unstale":"2024-09-18T08:03:34.046872+0000","last_undegraded":"2024-09-18T08:03:34.046872+0000","last_fullsized":"2024-09-18T08:03:34.046872+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:02:46.860092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:46.860092+0000","last_clean_scrub_stamp":"2024-09-18T08:02:46.860092+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:03:48.336434+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"70'5","reported_seq":100,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-18T08:03:36.712734+0000","last_change":"2024-09-18T08:02:49.877461+0000","last_active":"2024-09-18T08:03:36.712734+0000","last_peered":"2024-09-18T08:03:36.712734+0000","last_clean":"2024-09-18T08:03:36.712734+0000","last_became_active":"2024-09-18T08:02:46.872144+0000","last_became_peered":"2024-09-18T08:02:46.872144+0000","last_unstale":"2024-09-18T08:03:36.712734+0000","last_undegraded":"2024-09-18T08:03:36.712734+0000","last_fullsized":"2024-09-18T08:03:36.712734+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:02:45.855979+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:45.855979+0000","last_clean_scrub_stamp":"2024-09-18T08:02:45.855979+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:03:59.991143+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000504055,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":45,"num_read_kb":40,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":36,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-18T08:03:34.762850+0000","last_change":"2024-09-18T08:02:51.913423+0000","last_active":"2024-09-18T08:03:34.762850+0000","last_peered":"2024-09-18T08:03:34.762850+0000","last_clean":"2024-09-18T08:03:34.762850+0000","last_became_active":"2024-09-18T08:02:51.913112+0000","last_became_peered":"2024-09-18T08:02:51.913112+0000","last_unstale":"2024-09-18T08:03:34.762850+0000","last_undegraded":"2024-09-18T08:03:34.762850+0000","last_fullsized":"2024-09-18T08:03:34.762850+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:02:50.886830+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:50.886830+0000","last_clean_scrub_stamp":"2024-09-18T08:02:50.886830+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:04:04.493964+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":33,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-18T08:03:35.232003+0000","last_change":"2024-09-18T08:02:53.907689+0000","last_active":"2024-09-18T08:03:35.232003+0000","last_peered":"2024-09-18T08:03:35.232003+0000","last_clean":"2024-09-18T08:03:35.232003+0000","last_became_active":"2024-09-18T08:02:53.907404+0000","last_became_peered":"2024-09-18T08:02:53.907404+0000","last_unstale":"2024-09-18T08:03:35.232003+0000","last_undegraded":"2024-09-18T08:03:35.232003+0000","last_fullsized":"2024-09-18T08:03:35.232003+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_clean_scrub_stamp":"2024-09-18T08:02:52.890475+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:03:57.936771+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.5","version":"69'16","reported_seq":73,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-18T08:03:34.497664+0000","last_change":"2024-09-18T08:02:49.892366+0000","last_active":"2024-09-18T08:03:34.497664+0000","last_peered":"2024-09-18T08:03:34.497664+0000","last_clean":"2024-09-18T08:03:34.497664+0000","last_became_active":"2024-09-18T08:02:49.891985+0000","last_became_peered":"2024-09-18T08:02:49.891985+0000","last_unstale":"2024-09-18T08:03:34.497664+0000","last_undegraded":"2024-09-18T08:03:34.497664+0000","last_fullsized":"2024-09-18T08:03:34.497664+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_clean_scrub_stamp":"2024-09-18T08:02:48.870718+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:04:12.297325+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":25,"num_read_kb":15,"num_write":13,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":44,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-18T08:03:34.872048+0000","last_change":"2024-09-18T08:02:47.876225+0000","last_active":"2024-09-18T08:03:34.872048+0000","last_peered":"2024-09-18T08:03:34.872048+0000","last_clean":"2024-09-18T08:03:34.872048+0000","last_became_active":"2024-09-18T08:02:47.875950+0000","last_became_peered":"2024-09-18T08:02:47.875950+0000","last_unstale":"2024-09-18T08:03:34.872048+0000","last_undegraded":"2024-09-18T08:03:34.872048+0000","last_fullsized":"2024-09-18T08:03:34.872048+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:02:46.860092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:46.860092+0000","last_clean_scrub_stamp":"2024-09-18T08:02:46.860092+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:04:02.960274+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"74'79","reported_seq":90,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-18T08:03:34.572630+0000","last_change":"2024-09-18T08:02:55.472070+0000","last_active":"2024-09-18T08:03:34.572630+0000","last_peered":"2024-09-18T08:03:34.572630+0000","last_clean":"2024-09-18T08:03:34.572630+0000","last_became_active":"2024-09-18T08:02:39.446743+0000","last_became_peered":"2024-09-18T08:02:39.446743+0000","last_unstale":"2024-09-18T08:03:34.572630+0000","last_undegraded":"2024-09-18T08:03:34.572630+0000","last_fullsized":"2024-09-18T08:03:34.572630+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"20'72","last_scrub_stamp":"2024-09-18T08:02:55.471998+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:00:11.893742+0000","last_clean_scrub_stamp":"2024-09-18T08:02:55.471998+0000","objects_scrubbed":2,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:04:21.599062+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":231,"num_write":113,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":590368,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":36,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-18T08:03:34.573089+0000","last_change":"2024-09-18T08:02:51.905819+0000","last_active":"2024-09-18T08:03:34.573089+0000","last_peered":"2024-09-18T08:03:34.573089+0000","last_clean":"2024-09-18T08:03:34.573089+0000","last_became_active":"2024-09-18T08:02:51.905516+0000","last_became_peered":"2024-09-18T08:02:51.905516+0000","last_unstale":"2024-09-18T08:03:34.573089+0000","last_undegraded":"2024-09-18T08:03:34.573089+0000","last_fullsized":"2024-09-18T08:03:34.573089+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:02:50.886830+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:50.886830+0000","last_clean_scrub_stamp":"2024-09-18T08:02:50.886830+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:04:04.232646+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":33,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-18T08:03:34.762772+0000","last_change":"2024-09-18T08:02:53.909023+0000","last_active":"2024-09-18T08:03:34.762772+0000","last_peered":"2024-09-18T08:03:34.762772+0000","last_clean":"2024-09-18T08:03:34.762772+0000","last_became_active":"2024-09-18T08:02:53.908668+0000","last_became_peered":"2024-09-18T08:02:53.908668+0000","last_unstale":"2024-09-18T08:03:34.762772+0000","last_undegraded":"2024-09-18T08:03:34.762772+0000","last_fullsized":"2024-09-18T08:03:34.762772+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_clean_scrub_stamp":"2024-09-18T08:02:52.890475+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:03:57.881484+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.e","version":"69'11","reported_seq":59,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-18T08:03:34.048607+0000","last_change":"2024-09-18T08:02:49.895676+0000","last_active":"2024-09-18T08:03:34.048607+0000","last_peered":"2024-09-18T08:03:34.048607+0000","last_clean":"2024-09-18T08:03:34.048607+0000","last_became_active":"2024-09-18T08:02:49.895303+0000","last_became_peered":"2024-09-18T08:02:49.895303+0000","last_unstale":"2024-09-18T08:03:34.048607+0000","last_undegraded":"2024-09-18T08:03:34.048607+0000","last_fullsized":"2024-09-18T08:03:34.048607+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_clean_scrub_stamp":"2024-09-18T08:02:48.870718+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:04:11.479080+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":44,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-18T08:03:34.047619+0000","last_change":"2024-09-18T08:02:47.880265+0000","last_active":"2024-09-18T08:03:34.047619+0000","last_peered":"2024-09-18T08:03:34.047619+0000","last_clean":"2024-09-18T08:03:34.047619+0000","last_became_active":"2024-09-18T08:02:47.879764+0000","last_became_peered":"2024-09-18T08:02:47.879764+0000","last_unstale":"2024-09-18T08:03:34.047619+0000","last_undegraded":"2024-09-18T08:03:34.047619+0000","last_fullsized":"2024-09-18T08:03:34.047619+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:02:46.860092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:46.860092+0000","last_clean_scrub_stamp":"2024-09-18T08:02:46.860092+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:04:12.919106+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":36,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-18T08:03:34.762721+0000","last_change":"2024-09-18T08:02:51.911107+0000","last_active":"2024-09-18T08:03:34.762721+0000","last_peered":"2024-09-18T08:03:34.762721+0000","last_clean":"2024-09-18T08:03:34.762721+0000","last_became_active":"2024-09-18T08:02:51.910649+0000","last_became_peered":"2024-09-18T08:02:51.910649+0000","last_unstale":"2024-09-18T08:03:34.762721+0000","last_undegraded":"2024-09-18T08:03:34.762721+0000","last_fullsized":"2024-09-18T08:03:34.762721+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:02:50.886830+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:50.886830+0000","last_clean_scrub_stamp":"2024-09-18T08:02:50.886830+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:03:56.497732+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":32,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-18T08:03:34.872052+0000","last_change":"2024-09-18T08:02:54.045510+0000","last_active":"2024-09-18T08:03:34.872052+0000","last_peered":"2024-09-18T08:03:34.872052+0000","last_clean":"2024-09-18T08:03:34.872052+0000","last_became_active":"2024-09-18T08:02:54.045091+0000","last_became_peered":"2024-09-18T08:02:54.045091+0000","last_unstale":"2024-09-18T08:03:34.872052+0000","last_undegraded":"2024-09-18T08:03:34.872052+0000","last_fullsized":"2024-09-18T08:03:34.872052+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_clean_scrub_stamp":"2024-09-18T08:02:52.890475+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:04:22.368585+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.d","version":"69'17","reported_seq":68,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-18T08:03:34.048441+0000","last_change":"2024-09-18T08:02:49.895971+0000","last_active":"2024-09-18T08:03:34.048441+0000","last_peered":"2024-09-18T08:03:34.048441+0000","last_clean":"2024-09-18T08:03:34.048441+0000","last_became_active":"2024-09-18T08:02:49.895578+0000","last_became_peered":"2024-09-18T08:02:49.895578+0000","last_unstale":"2024-09-18T08:03:34.048441+0000","last_undegraded":"2024-09-18T08:03:34.048441+0000","last_fullsized":"2024-09-18T08:03:34.048441+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_clean_scrub_stamp":"2024-09-18T08:02:48.870718+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:04:17.784735+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":29,"num_read_kb":19,"num_write":18,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,1],"acting":[4,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":44,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-18T08:03:34.497394+0000","last_change":"2024-09-18T08:02:47.884431+0000","last_active":"2024-09-18T08:03:34.497394+0000","last_peered":"2024-09-18T08:03:34.497394+0000","last_clean":"2024-09-18T08:03:34.497394+0000","last_became_active":"2024-09-18T08:02:47.884132+0000","last_became_peered":"2024-09-18T08:02:47.884132+0000","last_unstale":"2024-09-18T08:03:34.497394+0000","last_undegraded":"2024-09-18T08:03:34.497394+0000","last_fullsized":"2024-09-18T08:03:34.497394+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:02:46.860092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:46.860092+0000","last_clean_scrub_stamp":"2024-09-18T08:02:46.860092+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:04:06.189609+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":36,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-18T08:03:35.232131+0000","last_change":"2024-09-18T08:02:51.907544+0000","last_active":"2024-09-18T08:03:35.232131+0000","last_peered":"2024-09-18T08:03:35.232131+0000","last_clean":"2024-09-18T08:03:35.232131+0000","last_became_active":"2024-09-18T08:02:51.907196+0000","last_became_peered":"2024-09-18T08:02:51.907196+0000","last_unstale":"2024-09-18T08:03:35.232131+0000","last_undegraded":"2024-09-18T08:03:35.232131+0000","last_fullsized":"2024-09-18T08:03:35.232131+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:02:50.886830+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:50.886830+0000","last_clean_scrub_stamp":"2024-09-18T08:02:50.886830+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:04:03.690724+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":33,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-18T08:03:34.048803+0000","last_change":"2024-09-18T08:02:53.904497+0000","last_active":"2024-09-18T08:03:34.048803+0000","last_peered":"2024-09-18T08:03:34.048803+0000","last_clean":"2024-09-18T08:03:34.048803+0000","last_became_active":"2024-09-18T08:02:53.904144+0000","last_became_peered":"2024-09-18T08:02:53.904144+0000","last_unstale":"2024-09-18T08:03:34.048803+0000","last_undegraded":"2024-09-18T08:03:34.048803+0000","last_fullsized":"2024-09-18T08:03:34.048803+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_clean_scrub_stamp":"2024-09-18T08:02:52.890475+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:03:58.164335+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.c","version":"69'10","reported_seq":55,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-18T08:03:34.048221+0000","last_change":"2024-09-18T08:02:49.892847+0000","last_active":"2024-09-18T08:03:34.048221+0000","last_peered":"2024-09-18T08:03:34.048221+0000","last_clean":"2024-09-18T08:03:34.048221+0000","last_became_active":"2024-09-18T08:02:49.892527+0000","last_became_peered":"2024-09-18T08:02:49.892527+0000","last_unstale":"2024-09-18T08:03:34.048221+0000","last_undegraded":"2024-09-18T08:03:34.048221+0000","last_fullsized":"2024-09-18T08:03:34.048221+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_clean_scrub_stamp":"2024-09-18T08:02:48.870718+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:04:11.391201+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":44,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-18T08:03:34.872284+0000","last_change":"2024-09-18T08:02:47.881814+0000","last_active":"2024-09-18T08:03:34.872284+0000","last_peered":"2024-09-18T08:03:34.872284+0000","last_clean":"2024-09-18T08:03:34.872284+0000","last_became_active":"2024-09-18T08:02:47.881512+0000","last_became_peered":"2024-09-18T08:02:47.881512+0000","last_unstale":"2024-09-18T08:03:34.872284+0000","last_undegraded":"2024-09-18T08:03:34.872284+0000","last_fullsized":"2024-09-18T08:03:34.872284+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:02:46.860092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:46.860092+0000","last_clean_scrub_stamp":"2024-09-18T08:02:46.860092+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:03:59.505935+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.d","version":"69'8","reported_seq":55,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-18T08:03:34.048230+0000","last_change":"2024-09-18T08:02:51.909290+0000","last_active":"2024-09-18T08:03:34.048230+0000","last_peered":"2024-09-18T08:03:34.048230+0000","last_clean":"2024-09-18T08:03:34.048230+0000","last_became_active":"2024-09-18T08:02:51.909011+0000","last_became_peered":"2024-09-18T08:02:51.909011+0000","last_unstale":"2024-09-18T08:03:34.048230+0000","last_undegraded":"2024-09-18T08:03:34.048230+0000","last_fullsized":"2024-09-18T08:03:34.048230+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:02:50.886830+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:50.886830+0000","last_clean_scrub_stamp":"2024-09-18T08:02:50.886830+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:04:10.431784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":33,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-18T08:03:34.048371+0000","last_change":"2024-09-18T08:02:53.902197+0000","last_active":"2024-09-18T08:03:34.048371+0000","last_peered":"2024-09-18T08:03:34.048371+0000","last_clean":"2024-09-18T08:03:34.048371+0000","last_became_active":"2024-09-18T08:02:53.901858+0000","last_became_peered":"2024-09-18T08:02:53.901858+0000","last_unstale":"2024-09-18T08:03:34.048371+0000","last_undegraded":"2024-09-18T08:03:34.048371+0000","last_fullsized":"2024-09-18T08:03:34.048371+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_clean_scrub_stamp":"2024-09-18T08:02:52.890475+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:04:08.426639+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.b","version":"69'9","reported_seq":56,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-18T08:03:34.047450+0000","last_change":"2024-09-18T08:02:49.898562+0000","last_active":"2024-09-18T08:03:34.047450+0000","last_peered":"2024-09-18T08:03:34.047450+0000","last_clean":"2024-09-18T08:03:34.047450+0000","last_became_active":"2024-09-18T08:02:49.897998+0000","last_became_peered":"2024-09-18T08:02:49.897998+0000","last_unstale":"2024-09-18T08:03:34.047450+0000","last_undegraded":"2024-09-18T08:03:34.047450+0000","last_fullsized":"2024-09-18T08:03:34.047450+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_clean_scrub_stamp":"2024-09-18T08:02:48.870718+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:03:56.995348+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":44,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-18T08:03:34.762561+0000","last_change":"2024-09-18T08:02:47.877928+0000","last_active":"2024-09-18T08:03:34.762561+0000","last_peered":"2024-09-18T08:03:34.762561+0000","last_clean":"2024-09-18T08:03:34.762561+0000","last_became_active":"2024-09-18T08:02:47.877161+0000","last_became_peered":"2024-09-18T08:02:47.877161+0000","last_unstale":"2024-09-18T08:03:34.762561+0000","last_undegraded":"2024-09-18T08:03:34.762561+0000","last_fullsized":"2024-09-18T08:03:34.762561+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:02:46.860092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:46.860092+0000","last_clean_scrub_stamp":"2024-09-18T08:02:46.860092+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:03:54.682345+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":36,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-18T08:03:34.047859+0000","last_change":"2024-09-18T08:02:51.901738+0000","last_active":"2024-09-18T08:03:34.047859+0000","last_peered":"2024-09-18T08:03:34.047859+0000","last_clean":"2024-09-18T08:03:34.047859+0000","last_became_active":"2024-09-18T08:02:51.901330+0000","last_became_peered":"2024-09-18T08:02:51.901330+0000","last_unstale":"2024-09-18T08:03:34.047859+0000","last_undegraded":"2024-09-18T08:03:34.047859+0000","last_fullsized":"2024-09-18T08:03:34.047859+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:02:50.886830+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:50.886830+0000","last_clean_scrub_stamp":"2024-09-18T08:02:50.886830+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:04:18.413020+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,3],"acting":[2,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":33,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-18T08:03:34.046755+0000","last_change":"2024-09-18T08:02:53.911400+0000","last_active":"2024-09-18T08:03:34.046755+0000","last_peered":"2024-09-18T08:03:34.046755+0000","last_clean":"2024-09-18T08:03:34.046755+0000","last_became_active":"2024-09-18T08:02:53.911068+0000","last_became_peered":"2024-09-18T08:02:53.911068+0000","last_unstale":"2024-09-18T08:03:34.046755+0000","last_undegraded":"2024-09-18T08:03:34.046755+0000","last_fullsized":"2024-09-18T08:03:34.046755+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_clean_scrub_stamp":"2024-09-18T08:02:52.890475+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:03:57.470235+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.a","version":"69'19","reported_seq":71,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-18T08:03:34.497158+0000","last_change":"2024-09-18T08:02:49.898366+0000","last_active":"2024-09-18T08:03:34.497158+0000","last_peered":"2024-09-18T08:03:34.497158+0000","last_clean":"2024-09-18T08:03:34.497158+0000","last_became_active":"2024-09-18T08:02:49.897960+0000","last_became_peered":"2024-09-18T08:02:49.897960+0000","last_unstale":"2024-09-18T08:03:34.497158+0000","last_undegraded":"2024-09-18T08:03:34.497158+0000","last_fullsized":"2024-09-18T08:03:34.497158+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_clean_scrub_stamp":"2024-09-18T08:02:48.870718+0000","objects_scrubbed":0,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:04:18.534695+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":32,"num_read_kb":21,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":44,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-18T08:03:34.573291+0000","last_change":"2024-09-18T08:02:47.881119+0000","last_active":"2024-09-18T08:03:34.573291+0000","last_peered":"2024-09-18T08:03:34.573291+0000","last_clean":"2024-09-18T08:03:34.573291+0000","last_became_active":"2024-09-18T08:02:47.880645+0000","last_became_peered":"2024-09-18T08:02:47.880645+0000","last_unstale":"2024-09-18T08:03:34.573291+0000","last_undegraded":"2024-09-18T08:03:34.573291+0000","last_fullsized":"2024-09-18T08:03:34.573291+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:02:46.860092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:46.860092+0000","last_clean_scrub_stamp":"2024-09-18T08:02:46.860092+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:04:02.022191+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":36,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-18T08:03:34.047799+0000","last_change":"2024-09-18T08:02:51.903944+0000","last_active":"2024-09-18T08:03:34.047799+0000","last_peered":"2024-09-18T08:03:34.047799+0000","last_clean":"2024-09-18T08:03:34.047799+0000","last_became_active":"2024-09-18T08:02:51.903439+0000","last_became_peered":"2024-09-18T08:02:51.903439+0000","last_unstale":"2024-09-18T08:03:34.047799+0000","last_undegraded":"2024-09-18T08:03:34.047799+0000","last_fullsized":"2024-09-18T08:03:34.047799+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:02:50.886830+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:50.886830+0000","last_clean_scrub_stamp":"2024-09-18T08:02:50.886830+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:03:52.993653+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":33,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-18T08:03:34.572548+0000","last_change":"2024-09-18T08:02:53.913096+0000","last_active":"2024-09-18T08:03:34.572548+0000","last_peered":"2024-09-18T08:03:34.572548+0000","last_clean":"2024-09-18T08:03:34.572548+0000","last_became_active":"2024-09-18T08:02:53.912762+0000","last_became_peered":"2024-09-18T08:02:53.912762+0000","last_unstale":"2024-09-18T08:03:34.572548+0000","last_undegraded":"2024-09-18T08:03:34.572548+0000","last_fullsized":"2024-09-18T08:03:34.572548+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_clean_scrub_stamp":"2024-09-18T08:02:52.890475+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:04:03.478804+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.9","version":"69'12","reported_seq":63,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-18T08:03:34.047514+0000","last_change":"2024-09-18T08:02:49.895136+0000","last_active":"2024-09-18T08:03:34.047514+0000","last_peered":"2024-09-18T08:03:34.047514+0000","last_clean":"2024-09-18T08:03:34.047514+0000","last_became_active":"2024-09-18T08:02:49.894762+0000","last_became_peered":"2024-09-18T08:02:49.894762+0000","last_unstale":"2024-09-18T08:03:34.047514+0000","last_undegraded":"2024-09-18T08:03:34.047514+0000","last_fullsized":"2024-09-18T08:03:34.047514+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_clean_scrub_stamp":"2024-09-18T08:02:48.870718+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:04:02.257011+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":25,"num_read_kb":16,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":44,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-18T08:03:34.573449+0000","last_change":"2024-09-18T08:02:47.884430+0000","last_active":"2024-09-18T08:03:34.573449+0000","last_peered":"2024-09-18T08:03:34.573449+0000","last_clean":"2024-09-18T08:03:34.573449+0000","last_became_active":"2024-09-18T08:02:47.884052+0000","last_became_peered":"2024-09-18T08:02:47.884052+0000","last_unstale":"2024-09-18T08:03:34.573449+0000","last_undegraded":"2024-09-18T08:03:34.573449+0000","last_fullsized":"2024-09-18T08:03:34.573449+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:02:46.860092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:46.860092+0000","last_clean_scrub_stamp":"2024-09-18T08:02:46.860092+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:04:04.711823+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":36,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-18T08:03:34.047203+0000","last_change":"2024-09-18T08:02:51.902541+0000","last_active":"2024-09-18T08:03:34.047203+0000","last_peered":"2024-09-18T08:03:34.047203+0000","last_clean":"2024-09-18T08:03:34.047203+0000","last_became_active":"2024-09-18T08:02:51.902163+0000","last_became_peered":"2024-09-18T08:02:51.902163+0000","last_unstale":"2024-09-18T08:03:34.047203+0000","last_undegraded":"2024-09-18T08:03:34.047203+0000","last_fullsized":"2024-09-18T08:03:34.047203+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:02:50.886830+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:50.886830+0000","last_clean_scrub_stamp":"2024-09-18T08:02:50.886830+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:03:54.180178+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":33,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-18T08:03:34.872076+0000","last_change":"2024-09-18T08:02:53.907752+0000","last_active":"2024-09-18T08:03:34.872076+0000","last_peered":"2024-09-18T08:03:34.872076+0000","last_clean":"2024-09-18T08:03:34.872076+0000","last_became_active":"2024-09-18T08:02:53.907412+0000","last_became_peered":"2024-09-18T08:02:53.907412+0000","last_unstale":"2024-09-18T08:03:34.872076+0000","last_undegraded":"2024-09-18T08:03:34.872076+0000","last_fullsized":"2024-09-18T08:03:34.872076+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_clean_scrub_stamp":"2024-09-18T08:02:52.890475+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:04:12.083897+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.8","version":"69'15","reported_seq":65,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-18T08:03:34.761818+0000","last_change":"2024-09-18T08:02:49.889133+0000","last_active":"2024-09-18T08:03:34.761818+0000","last_peered":"2024-09-18T08:03:34.761818+0000","last_clean":"2024-09-18T08:03:34.761818+0000","last_became_active":"2024-09-18T08:02:49.888799+0000","last_became_peered":"2024-09-18T08:02:49.888799+0000","last_unstale":"2024-09-18T08:03:34.761818+0000","last_undegraded":"2024-09-18T08:03:34.761818+0000","last_fullsized":"2024-09-18T08:03:34.761818+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_clean_scrub_stamp":"2024-09-18T08:02:48.870718+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:04:11.344161+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.f","version":"62'2","reported_seq":56,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-18T08:03:34.573437+0000","last_change":"2024-09-18T08:02:47.881441+0000","last_active":"2024-09-18T08:03:34.573437+0000","last_peered":"2024-09-18T08:03:34.573437+0000","last_clean":"2024-09-18T08:03:34.573437+0000","last_became_active":"2024-09-18T08:02:47.881165+0000","last_became_peered":"2024-09-18T08:02:47.881165+0000","last_unstale":"2024-09-18T08:03:34.573437+0000","last_undegraded":"2024-09-18T08:03:34.573437+0000","last_fullsized":"2024-09-18T08:03:34.573437+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:02:46.860092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:46.860092+0000","last_clean_scrub_stamp":"2024-09-18T08:02:46.860092+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:03:50.038477+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":4,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.9","version":"69'8","reported_seq":58,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-18T08:03:34.572710+0000","last_change":"2024-09-18T08:02:51.905340+0000","last_active":"2024-09-18T08:03:34.572710+0000","last_peered":"2024-09-18T08:03:34.572710+0000","last_clean":"2024-09-18T08:03:34.572710+0000","last_became_active":"2024-09-18T08:02:51.905059+0000","last_became_peered":"2024-09-18T08:02:51.905059+0000","last_unstale":"2024-09-18T08:03:34.572710+0000","last_undegraded":"2024-09-18T08:03:34.572710+0000","last_fullsized":"2024-09-18T08:03:34.572710+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:02:50.886830+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:50.886830+0000","last_clean_scrub_stamp":"2024-09-18T08:02:50.886830+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:03:52.023681+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":32,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-18T08:03:34.761721+0000","last_change":"2024-09-18T08:02:54.046138+0000","last_active":"2024-09-18T08:03:34.761721+0000","last_peered":"2024-09-18T08:03:34.761721+0000","last_clean":"2024-09-18T08:03:34.761721+0000","last_became_active":"2024-09-18T08:02:54.045845+0000","last_became_peered":"2024-09-18T08:02:54.045845+0000","last_unstale":"2024-09-18T08:03:34.761721+0000","last_undegraded":"2024-09-18T08:03:34.761721+0000","last_fullsized":"2024-09-18T08:03:34.761721+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_clean_scrub_stamp":"2024-09-18T08:02:52.890475+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:03:55.148975+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":44,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-18T08:03:34.497146+0000","last_change":"2024-09-18T08:02:47.882886+0000","last_active":"2024-09-18T08:03:34.497146+0000","last_peered":"2024-09-18T08:03:34.497146+0000","last_clean":"2024-09-18T08:03:34.497146+0000","last_became_active":"2024-09-18T08:02:47.882600+0000","last_became_peered":"2024-09-18T08:02:47.882600+0000","last_unstale":"2024-09-18T08:03:34.497146+0000","last_undegraded":"2024-09-18T08:03:34.497146+0000","last_fullsized":"2024-09-18T08:03:34.497146+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:02:46.860092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:46.860092+0000","last_clean_scrub_stamp":"2024-09-18T08:02:46.860092+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:04:15.079741+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"69'6","reported_seq":49,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-18T08:03:34.872748+0000","last_change":"2024-09-18T08:02:49.895477+0000","last_active":"2024-09-18T08:03:34.872748+0000","last_peered":"2024-09-18T08:03:34.872748+0000","last_clean":"2024-09-18T08:03:34.872748+0000","last_became_active":"2024-09-18T08:02:49.895067+0000","last_became_peered":"2024-09-18T08:02:49.895067+0000","last_unstale":"2024-09-18T08:03:34.872748+0000","last_undegraded":"2024-09-18T08:03:34.872748+0000","last_fullsized":"2024-09-18T08:03:34.872748+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_clean_scrub_stamp":"2024-09-18T08:02:48.870718+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:03:55.471468+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":9,"num_read_kb":6,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":36,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-18T08:03:34.762591+0000","last_change":"2024-09-18T08:02:51.912907+0000","last_active":"2024-09-18T08:03:34.762591+0000","last_peered":"2024-09-18T08:03:34.762591+0000","last_clean":"2024-09-18T08:03:34.762591+0000","last_became_active":"2024-09-18T08:02:51.912557+0000","last_became_peered":"2024-09-18T08:02:51.912557+0000","last_unstale":"2024-09-18T08:03:34.762591+0000","last_undegraded":"2024-09-18T08:03:34.762591+0000","last_fullsized":"2024-09-18T08:03:34.762591+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:02:50.886830+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:50.886830+0000","last_clean_scrub_stamp":"2024-09-18T08:02:50.886830+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:03:58.405243+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,1],"acting":[5,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":33,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-18T08:03:34.572238+0000","last_change":"2024-09-18T08:02:54.049521+0000","last_active":"2024-09-18T08:03:34.572238+0000","last_peered":"2024-09-18T08:03:34.572238+0000","last_clean":"2024-09-18T08:03:34.572238+0000","last_became_active":"2024-09-18T08:02:54.048959+0000","last_became_peered":"2024-09-18T08:02:54.048959+0000","last_unstale":"2024-09-18T08:03:34.572238+0000","last_undegraded":"2024-09-18T08:03:34.572238+0000","last_fullsized":"2024-09-18T08:03:34.572238+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_clean_scrub_stamp":"2024-09-18T08:02:52.890475+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:03:56.721996+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"69'9","reported_seq":56,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-18T08:03:34.047766+0000","last_change":"2024-09-18T08:02:49.892390+0000","last_active":"2024-09-18T08:03:34.047766+0000","last_peered":"2024-09-18T08:03:34.047766+0000","last_clean":"2024-09-18T08:03:34.047766+0000","last_became_active":"2024-09-18T08:02:49.891567+0000","last_became_peered":"2024-09-18T08:02:49.891567+0000","last_unstale":"2024-09-18T08:03:34.047766+0000","last_undegraded":"2024-09-18T08:03:34.047766+0000","last_fullsized":"2024-09-18T08:03:34.047766+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_clean_scrub_stamp":"2024-09-18T08:02:48.870718+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:03:51.625809+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":44,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-18T08:03:34.572817+0000","last_change":"2024-09-18T08:02:47.886233+0000","last_active":"2024-09-18T08:03:34.572817+0000","last_peered":"2024-09-18T08:03:34.572817+0000","last_clean":"2024-09-18T08:03:34.572817+0000","last_became_active":"2024-09-18T08:02:47.885790+0000","last_became_peered":"2024-09-18T08:02:47.885790+0000","last_unstale":"2024-09-18T08:03:34.572817+0000","last_undegraded":"2024-09-18T08:03:34.572817+0000","last_fullsized":"2024-09-18T08:03:34.572817+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:02:46.860092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:46.860092+0000","last_clean_scrub_stamp":"2024-09-18T08:02:46.860092+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:03:48.811474+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":36,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-18T08:03:34.872672+0000","last_change":"2024-09-18T08:02:51.908640+0000","last_active":"2024-09-18T08:03:34.872672+0000","last_peered":"2024-09-18T08:03:34.872672+0000","last_clean":"2024-09-18T08:03:34.872672+0000","last_became_active":"2024-09-18T08:02:51.907361+0000","last_became_peered":"2024-09-18T08:02:51.907361+0000","last_unstale":"2024-09-18T08:03:34.872672+0000","last_undegraded":"2024-09-18T08:03:34.872672+0000","last_fullsized":"2024-09-18T08:03:34.872672+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:02:50.886830+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:50.886830+0000","last_clean_scrub_stamp":"2024-09-18T08:02:50.886830+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:04:06.357747+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":33,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-18T08:03:34.048964+0000","last_change":"2024-09-18T08:02:53.909246+0000","last_active":"2024-09-18T08:03:34.048964+0000","last_peered":"2024-09-18T08:03:34.048964+0000","last_clean":"2024-09-18T08:03:34.048964+0000","last_became_active":"2024-09-18T08:02:53.908917+0000","last_became_peered":"2024-09-18T08:02:53.908917+0000","last_unstale":"2024-09-18T08:03:34.048964+0000","last_undegraded":"2024-09-18T08:03:34.048964+0000","last_fullsized":"2024-09-18T08:03:34.048964+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_clean_scrub_stamp":"2024-09-18T08:02:52.890475+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:03:57.367069+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.15","version":"69'9","reported_seq":56,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-18T08:03:34.762707+0000","last_change":"2024-09-18T08:02:49.890846+0000","last_active":"2024-09-18T08:03:34.762707+0000","last_peered":"2024-09-18T08:03:34.762707+0000","last_clean":"2024-09-18T08:03:34.762707+0000","last_became_active":"2024-09-18T08:02:49.890497+0000","last_became_peered":"2024-09-18T08:02:49.890497+0000","last_unstale":"2024-09-18T08:03:34.762707+0000","last_undegraded":"2024-09-18T08:03:34.762707+0000","last_fullsized":"2024-09-18T08:03:34.762707+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_clean_scrub_stamp":"2024-09-18T08:02:48.870718+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:04:02.479848+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":44,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-18T08:03:34.046975+0000","last_change":"2024-09-18T08:02:47.877632+0000","last_active":"2024-09-18T08:03:34.046975+0000","last_peered":"2024-09-18T08:03:34.046975+0000","last_clean":"2024-09-18T08:03:34.046975+0000","last_became_active":"2024-09-18T08:02:47.875983+0000","last_became_peered":"2024-09-18T08:02:47.875983+0000","last_unstale":"2024-09-18T08:03:34.046975+0000","last_undegraded":"2024-09-18T08:03:34.046975+0000","last_fullsized":"2024-09-18T08:03:34.046975+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:02:46.860092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:46.860092+0000","last_clean_scrub_stamp":"2024-09-18T08:02:46.860092+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:04:09.677911+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.14","version":"69'8","reported_seq":55,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-18T08:03:34.872139+0000","last_change":"2024-09-18T08:02:51.909896+0000","last_active":"2024-09-18T08:03:34.872139+0000","last_peered":"2024-09-18T08:03:34.872139+0000","last_clean":"2024-09-18T08:03:34.872139+0000","last_became_active":"2024-09-18T08:02:51.909513+0000","last_became_peered":"2024-09-18T08:02:51.909513+0000","last_unstale":"2024-09-18T08:03:34.872139+0000","last_undegraded":"2024-09-18T08:03:34.872139+0000","last_fullsized":"2024-09-18T08:03:34.872139+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:02:50.886830+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:50.886830+0000","last_clean_scrub_stamp":"2024-09-18T08:02:50.886830+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:03:54.167632+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":33,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-18T08:03:34.048706+0000","last_change":"2024-09-18T08:02:53.907729+0000","last_active":"2024-09-18T08:03:34.048706+0000","last_peered":"2024-09-18T08:03:34.048706+0000","last_clean":"2024-09-18T08:03:34.048706+0000","last_became_active":"2024-09-18T08:02:53.907428+0000","last_became_peered":"2024-09-18T08:02:53.907428+0000","last_unstale":"2024-09-18T08:03:34.048706+0000","last_undegraded":"2024-09-18T08:03:34.048706+0000","last_fullsized":"2024-09-18T08:03:34.048706+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_clean_scrub_stamp":"2024-09-18T08:02:52.890475+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:04:00.922586+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.14","version":"69'10","reported_seq":55,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-18T08:03:34.872329+0000","last_change":"2024-09-18T08:02:49.889483+0000","last_active":"2024-09-18T08:03:34.872329+0000","last_peered":"2024-09-18T08:03:34.872329+0000","last_clean":"2024-09-18T08:03:34.872329+0000","last_became_active":"2024-09-18T08:02:49.889191+0000","last_became_peered":"2024-09-18T08:02:49.889191+0000","last_unstale":"2024-09-18T08:03:34.872329+0000","last_undegraded":"2024-09-18T08:03:34.872329+0000","last_fullsized":"2024-09-18T08:03:34.872329+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_clean_scrub_stamp":"2024-09-18T08:02:48.870718+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:03:58.029274+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":44,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-18T08:03:34.573032+0000","last_change":"2024-09-18T08:02:47.882542+0000","last_active":"2024-09-18T08:03:34.573032+0000","last_peered":"2024-09-18T08:03:34.573032+0000","last_clean":"2024-09-18T08:03:34.573032+0000","last_became_active":"2024-09-18T08:02:47.882065+0000","last_became_peered":"2024-09-18T08:02:47.882065+0000","last_unstale":"2024-09-18T08:03:34.573032+0000","last_undegraded":"2024-09-18T08:03:34.573032+0000","last_fullsized":"2024-09-18T08:03:34.573032+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:02:46.860092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:46.860092+0000","last_clean_scrub_stamp":"2024-09-18T08:02:46.860092+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:03:59.012126+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.15","version":"69'8","reported_seq":55,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-18T08:03:34.762569+0000","last_change":"2024-09-18T08:02:51.914448+0000","last_active":"2024-09-18T08:03:34.762569+0000","last_peered":"2024-09-18T08:03:34.762569+0000","last_clean":"2024-09-18T08:03:34.762569+0000","last_became_active":"2024-09-18T08:02:51.914135+0000","last_became_peered":"2024-09-18T08:02:51.914135+0000","last_unstale":"2024-09-18T08:03:34.762569+0000","last_undegraded":"2024-09-18T08:03:34.762569+0000","last_fullsized":"2024-09-18T08:03:34.762569+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:02:50.886830+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:50.886830+0000","last_clean_scrub_stamp":"2024-09-18T08:02:50.886830+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:03:56.893696+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":33,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-18T08:03:34.047461+0000","last_change":"2024-09-18T08:02:53.912409+0000","last_active":"2024-09-18T08:03:34.047461+0000","last_peered":"2024-09-18T08:03:34.047461+0000","last_clean":"2024-09-18T08:03:34.047461+0000","last_became_active":"2024-09-18T08:02:53.912112+0000","last_became_peered":"2024-09-18T08:02:53.912112+0000","last_unstale":"2024-09-18T08:03:34.047461+0000","last_undegraded":"2024-09-18T08:03:34.047461+0000","last_fullsized":"2024-09-18T08:03:34.047461+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_clean_scrub_stamp":"2024-09-18T08:02:52.890475+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:03:54.310553+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.13","version":"69'11","reported_seq":59,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-18T08:03:34.047990+0000","last_change":"2024-09-18T08:02:49.897433+0000","last_active":"2024-09-18T08:03:34.047990+0000","last_peered":"2024-09-18T08:03:34.047990+0000","last_clean":"2024-09-18T08:03:34.047990+0000","last_became_active":"2024-09-18T08:02:49.896949+0000","last_became_peered":"2024-09-18T08:02:49.896949+0000","last_unstale":"2024-09-18T08:03:34.047990+0000","last_undegraded":"2024-09-18T08:03:34.047990+0000","last_fullsized":"2024-09-18T08:03:34.047990+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_clean_scrub_stamp":"2024-09-18T08:02:48.870718+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:04:08.564349+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":44,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-18T08:03:34.048395+0000","last_change":"2024-09-18T08:02:47.880202+0000","last_active":"2024-09-18T08:03:34.048395+0000","last_peered":"2024-09-18T08:03:34.048395+0000","last_clean":"2024-09-18T08:03:34.048395+0000","last_became_active":"2024-09-18T08:02:47.879771+0000","last_became_peered":"2024-09-18T08:02:47.879771+0000","last_unstale":"2024-09-18T08:03:34.048395+0000","last_undegraded":"2024-09-18T08:03:34.048395+0000","last_fullsized":"2024-09-18T08:03:34.048395+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:02:46.860092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:46.860092+0000","last_clean_scrub_stamp":"2024-09-18T08:02:46.860092+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:03:58.640926+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":36,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-18T08:03:35.231883+0000","last_change":"2024-09-18T08:02:51.912062+0000","last_active":"2024-09-18T08:03:35.231883+0000","last_peered":"2024-09-18T08:03:35.231883+0000","last_clean":"2024-09-18T08:03:35.231883+0000","last_became_active":"2024-09-18T08:02:51.910752+0000","last_became_peered":"2024-09-18T08:02:51.910752+0000","last_unstale":"2024-09-18T08:03:35.231883+0000","last_undegraded":"2024-09-18T08:03:35.231883+0000","last_fullsized":"2024-09-18T08:03:35.231883+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:02:50.886830+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:50.886830+0000","last_clean_scrub_stamp":"2024-09-18T08:02:50.886830+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:04:01.465747+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":33,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-18T08:03:34.871644+0000","last_change":"2024-09-18T08:02:53.911861+0000","last_active":"2024-09-18T08:03:34.871644+0000","last_peered":"2024-09-18T08:03:34.871644+0000","last_clean":"2024-09-18T08:03:34.871644+0000","last_became_active":"2024-09-18T08:02:53.911429+0000","last_became_peered":"2024-09-18T08:02:53.911429+0000","last_unstale":"2024-09-18T08:03:34.871644+0000","last_undegraded":"2024-09-18T08:03:34.871644+0000","last_fullsized":"2024-09-18T08:03:34.871644+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_clean_scrub_stamp":"2024-09-18T08:02:52.890475+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:03:58.350802+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.12","version":"69'9","reported_seq":56,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-18T08:03:35.232068+0000","last_change":"2024-09-18T08:02:49.893472+0000","last_active":"2024-09-18T08:03:35.232068+0000","last_peered":"2024-09-18T08:03:35.232068+0000","last_clean":"2024-09-18T08:03:35.232068+0000","last_became_active":"2024-09-18T08:02:49.885950+0000","last_became_peered":"2024-09-18T08:02:49.885950+0000","last_unstale":"2024-09-18T08:03:35.232068+0000","last_undegraded":"2024-09-18T08:03:35.232068+0000","last_fullsized":"2024-09-18T08:03:35.232068+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_clean_scrub_stamp":"2024-09-18T08:02:48.870718+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:03:57.261106+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":44,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-18T08:03:34.573125+0000","last_change":"2024-09-18T08:02:47.883002+0000","last_active":"2024-09-18T08:03:34.573125+0000","last_peered":"2024-09-18T08:03:34.573125+0000","last_clean":"2024-09-18T08:03:34.573125+0000","last_became_active":"2024-09-18T08:02:47.882096+0000","last_became_peered":"2024-09-18T08:02:47.882096+0000","last_unstale":"2024-09-18T08:03:34.573125+0000","last_undegraded":"2024-09-18T08:03:34.573125+0000","last_fullsized":"2024-09-18T08:03:34.573125+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:02:46.860092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:46.860092+0000","last_clean_scrub_stamp":"2024-09-18T08:02:46.860092+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:04:10.794338+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":36,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-18T08:03:34.872150+0000","last_change":"2024-09-18T08:02:51.906594+0000","last_active":"2024-09-18T08:03:34.872150+0000","last_peered":"2024-09-18T08:03:34.872150+0000","last_clean":"2024-09-18T08:03:34.872150+0000","last_became_active":"2024-09-18T08:02:51.906277+0000","last_became_peered":"2024-09-18T08:02:51.906277+0000","last_unstale":"2024-09-18T08:03:34.872150+0000","last_undegraded":"2024-09-18T08:03:34.872150+0000","last_fullsized":"2024-09-18T08:03:34.872150+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:02:50.886830+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:50.886830+0000","last_clean_scrub_stamp":"2024-09-18T08:02:50.886830+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:04:07.327398+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.10","version":"69'1","reported_seq":34,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-18T08:03:34.046464+0000","last_change":"2024-09-18T08:02:53.913013+0000","last_active":"2024-09-18T08:03:34.046464+0000","last_peered":"2024-09-18T08:03:34.046464+0000","last_clean":"2024-09-18T08:03:34.046464+0000","last_became_active":"2024-09-18T08:02:53.912699+0000","last_became_peered":"2024-09-18T08:02:53.912699+0000","last_unstale":"2024-09-18T08:03:34.046464+0000","last_undegraded":"2024-09-18T08:03:34.046464+0000","last_fullsized":"2024-09-18T08:03:34.046464+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_clean_scrub_stamp":"2024-09-18T08:02:52.890475+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:04:15.154915+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.11","version":"69'11","reported_seq":59,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-18T08:03:34.871764+0000","last_change":"2024-09-18T08:02:49.889895+0000","last_active":"2024-09-18T08:03:34.871764+0000","last_peered":"2024-09-18T08:03:34.871764+0000","last_clean":"2024-09-18T08:03:34.871764+0000","last_became_active":"2024-09-18T08:02:49.889611+0000","last_became_peered":"2024-09-18T08:02:49.889611+0000","last_unstale":"2024-09-18T08:03:34.871764+0000","last_undegraded":"2024-09-18T08:03:34.871764+0000","last_fullsized":"2024-09-18T08:03:34.871764+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_clean_scrub_stamp":"2024-09-18T08:02:48.870718+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:03:51.062461+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":44,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-18T08:03:34.762615+0000","last_change":"2024-09-18T08:02:47.879259+0000","last_active":"2024-09-18T08:03:34.762615+0000","last_peered":"2024-09-18T08:03:34.762615+0000","last_clean":"2024-09-18T08:03:34.762615+0000","last_became_active":"2024-09-18T08:02:47.878886+0000","last_became_peered":"2024-09-18T08:02:47.878886+0000","last_unstale":"2024-09-18T08:03:34.762615+0000","last_undegraded":"2024-09-18T08:03:34.762615+0000","last_fullsized":"2024-09-18T08:03:34.762615+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:02:46.860092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:46.860092+0000","last_clean_scrub_stamp":"2024-09-18T08:02:46.860092+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:03:47.500112+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":36,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-18T08:03:34.572309+0000","last_change":"2024-09-18T08:02:51.908058+0000","last_active":"2024-09-18T08:03:34.572309+0000","last_peered":"2024-09-18T08:03:34.572309+0000","last_clean":"2024-09-18T08:03:34.572309+0000","last_became_active":"2024-09-18T08:02:51.907762+0000","last_became_peered":"2024-09-18T08:02:51.907762+0000","last_unstale":"2024-09-18T08:03:34.572309+0000","last_undegraded":"2024-09-18T08:03:34.572309+0000","last_fullsized":"2024-09-18T08:03:34.572309+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:02:50.886830+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:50.886830+0000","last_clean_scrub_stamp":"2024-09-18T08:02:50.886830+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:04:01.256104+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":32,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-18T08:03:34.871461+0000","last_change":"2024-09-18T08:02:54.047687+0000","last_active":"2024-09-18T08:03:34.871461+0000","last_peered":"2024-09-18T08:03:34.871461+0000","last_clean":"2024-09-18T08:03:34.871461+0000","last_became_active":"2024-09-18T08:02:54.047181+0000","last_became_peered":"2024-09-18T08:02:54.047181+0000","last_unstale":"2024-09-18T08:03:34.871461+0000","last_undegraded":"2024-09-18T08:03:34.871461+0000","last_fullsized":"2024-09-18T08:03:34.871461+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_clean_scrub_stamp":"2024-09-18T08:02:52.890475+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:04:12.998775+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.10","version":"69'4","reported_seq":46,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-18T08:03:34.871632+0000","last_change":"2024-09-18T08:02:49.890435+0000","last_active":"2024-09-18T08:03:34.871632+0000","last_peered":"2024-09-18T08:03:34.871632+0000","last_clean":"2024-09-18T08:03:34.871632+0000","last_became_active":"2024-09-18T08:02:49.890126+0000","last_became_peered":"2024-09-18T08:02:49.890126+0000","last_unstale":"2024-09-18T08:03:34.871632+0000","last_undegraded":"2024-09-18T08:03:34.871632+0000","last_fullsized":"2024-09-18T08:03:34.871632+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_clean_scrub_stamp":"2024-09-18T08:02:48.870718+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:03:56.821965+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":6,"num_read_kb":4,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":44,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-18T08:03:34.047540+0000","last_change":"2024-09-18T08:02:47.877991+0000","last_active":"2024-09-18T08:03:34.047540+0000","last_peered":"2024-09-18T08:03:34.047540+0000","last_clean":"2024-09-18T08:03:34.047540+0000","last_became_active":"2024-09-18T08:02:47.877616+0000","last_became_peered":"2024-09-18T08:02:47.877616+0000","last_unstale":"2024-09-18T08:03:34.047540+0000","last_undegraded":"2024-09-18T08:03:34.047540+0000","last_fullsized":"2024-09-18T08:03:34.047540+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:02:46.860092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:46.860092+0000","last_clean_scrub_stamp":"2024-09-18T08:02:46.860092+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:04:04.436191+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":36,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-18T08:03:34.497035+0000","last_change":"2024-09-18T08:02:51.906818+0000","last_active":"2024-09-18T08:03:34.497035+0000","last_peered":"2024-09-18T08:03:34.497035+0000","last_clean":"2024-09-18T08:03:34.497035+0000","last_became_active":"2024-09-18T08:02:51.906372+0000","last_became_peered":"2024-09-18T08:02:51.906372+0000","last_unstale":"2024-09-18T08:03:34.497035+0000","last_undegraded":"2024-09-18T08:03:34.497035+0000","last_fullsized":"2024-09-18T08:03:34.497035+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:02:50.886830+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:50.886830+0000","last_clean_scrub_stamp":"2024-09-18T08:02:50.886830+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:03:51.034977+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":33,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-18T08:03:34.572563+0000","last_change":"2024-09-18T08:02:53.909445+0000","last_active":"2024-09-18T08:03:34.572563+0000","last_peered":"2024-09-18T08:03:34.572563+0000","last_clean":"2024-09-18T08:03:34.572563+0000","last_became_active":"2024-09-18T08:02:53.909146+0000","last_became_peered":"2024-09-18T08:02:53.909146+0000","last_unstale":"2024-09-18T08:03:34.572563+0000","last_undegraded":"2024-09-18T08:03:34.572563+0000","last_fullsized":"2024-09-18T08:03:34.572563+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_clean_scrub_stamp":"2024-09-18T08:02:52.890475+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:04:03.033337+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,4],"acting":[7,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":33,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-18T08:03:35.232234+0000","last_change":"2024-09-18T08:02:53.907055+0000","last_active":"2024-09-18T08:03:35.232234+0000","last_peered":"2024-09-18T08:03:35.232234+0000","last_clean":"2024-09-18T08:03:35.232234+0000","last_became_active":"2024-09-18T08:02:53.906726+0000","last_became_peered":"2024-09-18T08:02:53.906726+0000","last_unstale":"2024-09-18T08:03:35.232234+0000","last_undegraded":"2024-09-18T08:03:35.232234+0000","last_fullsized":"2024-09-18T08:03:35.232234+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_clean_scrub_stamp":"2024-09-18T08:02:52.890475+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:03:58.628509+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":44,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-18T08:03:34.871813+0000","last_change":"2024-09-18T08:02:47.879447+0000","last_active":"2024-09-18T08:03:34.871813+0000","last_peered":"2024-09-18T08:03:34.871813+0000","last_clean":"2024-09-18T08:03:34.871813+0000","last_became_active":"2024-09-18T08:02:47.878922+0000","last_became_peered":"2024-09-18T08:02:47.878922+0000","last_unstale":"2024-09-18T08:03:34.871813+0000","last_undegraded":"2024-09-18T08:03:34.871813+0000","last_fullsized":"2024-09-18T08:03:34.871813+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:02:46.860092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:46.860092+0000","last_clean_scrub_stamp":"2024-09-18T08:02:46.860092+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:03:54.771042+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"69'11","reported_seq":59,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-18T08:03:34.497817+0000","last_change":"2024-09-18T08:02:49.898630+0000","last_active":"2024-09-18T08:03:34.497817+0000","last_peered":"2024-09-18T08:03:34.497817+0000","last_clean":"2024-09-18T08:03:34.497817+0000","last_became_active":"2024-09-18T08:02:49.898048+0000","last_became_peered":"2024-09-18T08:02:49.898048+0000","last_unstale":"2024-09-18T08:03:34.497817+0000","last_undegraded":"2024-09-18T08:03:34.497817+0000","last_fullsized":"2024-09-18T08:03:34.497817+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_clean_scrub_stamp":"2024-09-18T08:02:48.870718+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:04:05.049957+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":36,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-18T08:03:34.047607+0000","last_change":"2024-09-18T08:02:51.900666+0000","last_active":"2024-09-18T08:03:34.047607+0000","last_peered":"2024-09-18T08:03:34.047607+0000","last_clean":"2024-09-18T08:03:34.047607+0000","last_became_active":"2024-09-18T08:02:51.900300+0000","last_became_peered":"2024-09-18T08:02:51.900300+0000","last_unstale":"2024-09-18T08:03:34.047607+0000","last_undegraded":"2024-09-18T08:03:34.047607+0000","last_fullsized":"2024-09-18T08:03:34.047607+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:02:50.886830+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:50.886830+0000","last_clean_scrub_stamp":"2024-09-18T08:02:50.886830+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:04:12.807594+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":416,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":64,"ondisk_log_size":64,"up":96,"acting":96,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":178,"num_object_clones":0,"num_object_copies":534,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":178,"num_whiteouts":0,"num_read":698,"num_read_kb":455,"num_write":417,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":417792,"data_stored":11106,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":393,"ondisk_log_size":393,"up":96,"acting":96,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":1613,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":24,"num_read_kb":24,"num_write":10,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":4839,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":6,"ondisk_log_size":6,"up":96,"acting":96,"num_store_stats":8},{"poolid":2,"num_pg":3,"stat_sum":{"num_bytes":408,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":45,"num_read_kb":40,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":8,"ondisk_log_size":8,"up":9,"acting":9,"num_store_stats":7},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":231,"num_write":113,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":590368,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":79,"ondisk_log_size":79,"up":3,"acting":3,"num_store_stats":3}],"osd_stats":[{"osd":7,"up_from":56,"seq":240518168590,"num_pgs":53,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28376,"kb_used_data":1348,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716776,"statfs":{"total":95995035648,"available":95965978624,"internally_reserved":0,"allocated":1380352,"data_stored":906838,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":47,"seq":201863462932,"num_pgs":43,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28356,"kb_used_data":1328,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716796,"statfs":{"total":95995035648,"available":95965999104,"internally_reserved":0,"allocated":1359872,"data_stored":905753,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":39,"seq":167503724570,"num_pgs":47,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27788,"kb_used_data":760,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717364,"statfs":{"total":95995035648,"available":95966580736,"internally_reserved":0,"allocated":778240,"data_stored":316193,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":32,"seq":137438953503,"num_pgs":58,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27808,"kb_used_data":780,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717344,"statfs":{"total":95995035648,"available":95966560256,"internally_reserved":0,"allocated":798720,"data_stored":316983,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":26,"seq":111669149734,"num_pgs":56,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27812,"kb_used_data":784,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717340,"statfs":{"total":95995035648,"available":95966556160,"internally_reserved":0,"allocated":802816,"data_stored":316276,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":27654600},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411372,"num_pgs":36,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":31936,"kb_used_data":748,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713216,"statfs":{"total":95995035648,"available":95962333184,"internally_reserved":0,"allocated":765952,"data_stored":315969,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574896,"num_pgs":57,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27852,"kb_used_data":824,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717300,"statfs":{"total":95995035648,"available":95966515200,"internally_reserved":0,"allocated":843776,"data_stored":318612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738424,"num_pgs":46,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28388,"kb_used_data":1360,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716764,"statfs":{"total":95995035648,"available":95965966336,"internally_reserved":0,"allocated":1392640,"data_stored":907553,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1131,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1521,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":436,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":46,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1177,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":436,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":90112,"data_stored":2338,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":798,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":1898,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1474,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":990,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1034,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1254,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":403,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":416,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":403,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T08:03:44.856 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid 779349ca-7593-11ef-bceb-c7b262605968 -- ceph pg dump --format=json 2024-09-18T08:03:45.078 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Sep 18 08:03:44 smithi007 ceph-779349ca-7593-11ef-bceb-c7b262605968-mgr-y[29688]: [18/Sep/2024:08:03:44] ENGINE Bus STOPPING 2024-09-18T08:03:45.128 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/779349ca-7593-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T08:03:45.337 INFO:journalctl@ceph.alertmanager.a.smithi007.stdout:Sep 18 08:03:45 smithi007 ceph-779349ca-7593-11ef-bceb-c7b262605968-alertmanager-a[70326]: ts=2024-09-18T08:03:45.076Z caller=cluster.go:708 level=info component=cluster msg="gossip not settled" polls=0 before=0 now=1 elapsed=2.000393982s 2024-09-18T08:03:45.338 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:03:45 smithi007 ceph-mon[29459]: from='client.24623 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-18T08:03:45.338 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:03:45 smithi007 ceph-mon[29459]: from='mgr.24470 172.21.15.7:0/2594390261' entity='mgr.y' 2024-09-18T08:03:45.338 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:03:45 smithi007 ceph-mon[29459]: from='mgr.24470 172.21.15.7:0/2594390261' entity='mgr.y' 2024-09-18T08:03:45.338 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:03:45 smithi007 ceph-mon[29459]: from='mgr.24470 172.21.15.7:0/2594390261' entity='mgr.y' cmd={"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"} : dispatch 2024-09-18T08:03:45.338 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:03:45 smithi007 ceph-mon[29459]: from='mgr.24470 172.21.15.7:0/2594390261' entity='mgr.y' 2024-09-18T08:03:45.338 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:03:45 smithi007 ceph-mon[39249]: from='client.24623 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-18T08:03:45.338 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:03:45 smithi007 ceph-mon[39249]: from='mgr.24470 172.21.15.7:0/2594390261' entity='mgr.y' 2024-09-18T08:03:45.338 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:03:45 smithi007 ceph-mon[39249]: from='mgr.24470 172.21.15.7:0/2594390261' entity='mgr.y' 2024-09-18T08:03:45.339 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:03:45 smithi007 ceph-mon[39249]: from='mgr.24470 172.21.15.7:0/2594390261' entity='mgr.y' cmd={"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"} : dispatch 2024-09-18T08:03:45.339 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:03:45 smithi007 ceph-mon[39249]: from='mgr.24470 172.21.15.7:0/2594390261' entity='mgr.y' 2024-09-18T08:03:45.339 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Sep 18 08:03:45 smithi007 ceph-779349ca-7593-11ef-bceb-c7b262605968-mgr-y[29688]: [18/Sep/2024:08:03:45] ENGINE HTTP Server cherrypy._cpwsgi_server.CPWSGIServer(('::', 9283)) shut down 2024-09-18T08:03:45.339 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Sep 18 08:03:45 smithi007 ceph-779349ca-7593-11ef-bceb-c7b262605968-mgr-y[29688]: [18/Sep/2024:08:03:45] ENGINE Bus STOPPED 2024-09-18T08:03:45.339 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Sep 18 08:03:45 smithi007 ceph-779349ca-7593-11ef-bceb-c7b262605968-mgr-y[29688]: [18/Sep/2024:08:03:45] ENGINE Bus STARTING 2024-09-18T08:03:45.652 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Sep 18 08:03:45 smithi007 ceph-779349ca-7593-11ef-bceb-c7b262605968-mgr-y[29688]: [18/Sep/2024:08:03:45] ENGINE Serving on http://:::9283 2024-09-18T08:03:45.653 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Sep 18 08:03:45 smithi007 ceph-779349ca-7593-11ef-bceb-c7b262605968-mgr-y[29688]: [18/Sep/2024:08:03:45] ENGINE Bus STARTED 2024-09-18T08:03:45.655 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:03:45 smithi050 ceph-mon[32371]: from='client.24623 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-18T08:03:45.656 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:03:45 smithi050 ceph-mon[32371]: from='mgr.24470 172.21.15.7:0/2594390261' entity='mgr.y' 2024-09-18T08:03:45.656 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:03:45 smithi050 ceph-mon[32371]: from='mgr.24470 172.21.15.7:0/2594390261' entity='mgr.y' 2024-09-18T08:03:45.656 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:03:45 smithi050 ceph-mon[32371]: from='mgr.24470 172.21.15.7:0/2594390261' entity='mgr.y' cmd={"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"} : dispatch 2024-09-18T08:03:45.656 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:03:45 smithi050 ceph-mon[32371]: from='mgr.24470 172.21.15.7:0/2594390261' entity='mgr.y' 2024-09-18T08:03:45.850 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-09-18T08:03:45.852 INFO:teuthology.orchestra.run.smithi007.stderr:dumped all 2024-09-18T08:03:46.293 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:03:46 smithi007 ceph-mon[29459]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-09-18T08:03:46.293 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:03:46 smithi007 ceph-mon[29459]: Deploying daemon grafana.a on smithi050 2024-09-18T08:03:46.293 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:03:46 smithi007 ceph-mon[29459]: pgmap v22: 132 pgs: 132 active+clean; 583 KiB data, 223 MiB used, 715 GiB / 715 GiB avail; 928 B/s rd, 0 op/s 2024-09-18T08:03:46.293 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:03:46 smithi007 ceph-mon[29459]: from='mgr.24470 172.21.15.7:0/2594390261' entity='mgr.y' cmd={"prefix": "orch prometheus get-credentials"} : dispatch 2024-09-18T08:03:46.293 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:03:46 smithi007 ceph-mon[29459]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "orch prometheus get-credentials"}]: dispatch 2024-09-18T08:03:46.294 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:03:46 smithi007 ceph-mon[39249]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-09-18T08:03:46.294 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:03:46 smithi007 ceph-mon[39249]: Deploying daemon grafana.a on smithi050 2024-09-18T08:03:46.294 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:03:46 smithi007 ceph-mon[39249]: pgmap v22: 132 pgs: 132 active+clean; 583 KiB data, 223 MiB used, 715 GiB / 715 GiB avail; 928 B/s rd, 0 op/s 2024-09-18T08:03:46.294 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:03:46 smithi007 ceph-mon[39249]: from='mgr.24470 172.21.15.7:0/2594390261' entity='mgr.y' cmd={"prefix": "orch prometheus get-credentials"} : dispatch 2024-09-18T08:03:46.294 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:03:46 smithi007 ceph-mon[39249]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "orch prometheus get-credentials"}]: dispatch 2024-09-18T08:03:46.653 INFO:teuthology.orchestra.run.smithi007.stdout:{"pg_ready":true,"pg_map":{"version":22,"stamp":"2024-09-18T08:03:45.072460+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":596507,"num_objects":199,"num_object_clones":0,"num_object_copies":597,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":199,"num_whiteouts":0,"num_read":898,"num_read_kb":755,"num_write":549,"num_write_kb":1673,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":590368,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":552,"ondisk_log_size":552,"up":396,"acting":396,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":396,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":228316,"kb_used_data":7932,"kb_used_omap":12,"kb_used_meta":220211,"kb_avail":749732900,"statfs":{"total":767960285184,"available":767726489600,"internally_reserved":0,"allocated":8122368,"data_stored":4304177,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12750,"internal_metadata":225496626},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"11.027003"},"pg_stats":[{"pgid":"3.1f","version":"0'0","reported_seq":44,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-18T08:03:34.047520+0000","last_change":"2024-09-18T08:02:47.878020+0000","last_active":"2024-09-18T08:03:34.047520+0000","last_peered":"2024-09-18T08:03:34.047520+0000","last_clean":"2024-09-18T08:03:34.047520+0000","last_became_active":"2024-09-18T08:02:47.877698+0000","last_became_peered":"2024-09-18T08:02:47.877698+0000","last_unstale":"2024-09-18T08:03:34.047520+0000","last_undegraded":"2024-09-18T08:03:34.047520+0000","last_fullsized":"2024-09-18T08:03:34.047520+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:02:46.860092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:46.860092+0000","last_clean_scrub_stamp":"2024-09-18T08:02:46.860092+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:04:14.205967+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"69'9","reported_seq":56,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-18T08:03:34.047259+0000","last_change":"2024-09-18T08:02:49.895688+0000","last_active":"2024-09-18T08:03:34.047259+0000","last_peered":"2024-09-18T08:03:34.047259+0000","last_clean":"2024-09-18T08:03:34.047259+0000","last_became_active":"2024-09-18T08:02:49.894557+0000","last_became_peered":"2024-09-18T08:02:49.894557+0000","last_unstale":"2024-09-18T08:03:34.047259+0000","last_undegraded":"2024-09-18T08:03:34.047259+0000","last_fullsized":"2024-09-18T08:03:34.047259+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_clean_scrub_stamp":"2024-09-18T08:02:48.870718+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:03:59.206088+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":36,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-18T08:03:35.231793+0000","last_change":"2024-09-18T08:02:51.907566+0000","last_active":"2024-09-18T08:03:35.231793+0000","last_peered":"2024-09-18T08:03:35.231793+0000","last_clean":"2024-09-18T08:03:35.231793+0000","last_became_active":"2024-09-18T08:02:51.907140+0000","last_became_peered":"2024-09-18T08:02:51.907140+0000","last_unstale":"2024-09-18T08:03:35.231793+0000","last_undegraded":"2024-09-18T08:03:35.231793+0000","last_fullsized":"2024-09-18T08:03:35.231793+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:02:50.886830+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:50.886830+0000","last_clean_scrub_stamp":"2024-09-18T08:02:50.886830+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:04:07.003335+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":33,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-18T08:03:34.047497+0000","last_change":"2024-09-18T08:02:53.906914+0000","last_active":"2024-09-18T08:03:34.047497+0000","last_peered":"2024-09-18T08:03:34.047497+0000","last_clean":"2024-09-18T08:03:34.047497+0000","last_became_active":"2024-09-18T08:02:53.906622+0000","last_became_peered":"2024-09-18T08:02:53.906622+0000","last_unstale":"2024-09-18T08:03:34.047497+0000","last_undegraded":"2024-09-18T08:03:34.047497+0000","last_fullsized":"2024-09-18T08:03:34.047497+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_clean_scrub_stamp":"2024-09-18T08:02:52.890475+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:04:20.738381+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":32,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-18T08:03:34.872031+0000","last_change":"2024-09-18T08:02:54.047999+0000","last_active":"2024-09-18T08:03:34.872031+0000","last_peered":"2024-09-18T08:03:34.872031+0000","last_clean":"2024-09-18T08:03:34.872031+0000","last_became_active":"2024-09-18T08:02:54.047640+0000","last_became_peered":"2024-09-18T08:02:54.047640+0000","last_unstale":"2024-09-18T08:03:34.872031+0000","last_undegraded":"2024-09-18T08:03:34.872031+0000","last_fullsized":"2024-09-18T08:03:34.872031+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_clean_scrub_stamp":"2024-09-18T08:02:52.890475+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:04:16.013706+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":44,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-18T08:03:34.872645+0000","last_change":"2024-09-18T08:02:47.872585+0000","last_active":"2024-09-18T08:03:34.872645+0000","last_peered":"2024-09-18T08:03:34.872645+0000","last_clean":"2024-09-18T08:03:34.872645+0000","last_became_active":"2024-09-18T08:02:47.872316+0000","last_became_peered":"2024-09-18T08:02:47.872316+0000","last_unstale":"2024-09-18T08:03:34.872645+0000","last_undegraded":"2024-09-18T08:03:34.872645+0000","last_fullsized":"2024-09-18T08:03:34.872645+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:02:46.860092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:46.860092+0000","last_clean_scrub_stamp":"2024-09-18T08:02:46.860092+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:03:53.731838+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"69'15","reported_seq":65,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-18T08:03:34.872036+0000","last_change":"2024-09-18T08:02:49.887111+0000","last_active":"2024-09-18T08:03:34.872036+0000","last_peered":"2024-09-18T08:03:34.872036+0000","last_clean":"2024-09-18T08:03:34.872036+0000","last_became_active":"2024-09-18T08:02:49.886106+0000","last_became_peered":"2024-09-18T08:02:49.886106+0000","last_unstale":"2024-09-18T08:03:34.872036+0000","last_undegraded":"2024-09-18T08:03:34.872036+0000","last_fullsized":"2024-09-18T08:03:34.872036+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_clean_scrub_stamp":"2024-09-18T08:02:48.870718+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:04:15.737211+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":36,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-18T08:03:34.047246+0000","last_change":"2024-09-18T08:02:51.904853+0000","last_active":"2024-09-18T08:03:34.047246+0000","last_peered":"2024-09-18T08:03:34.047246+0000","last_clean":"2024-09-18T08:03:34.047246+0000","last_became_active":"2024-09-18T08:02:51.904184+0000","last_became_peered":"2024-09-18T08:02:51.904184+0000","last_unstale":"2024-09-18T08:03:34.047246+0000","last_undegraded":"2024-09-18T08:03:34.047246+0000","last_fullsized":"2024-09-18T08:03:34.047246+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:02:50.886830+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:50.886830+0000","last_clean_scrub_stamp":"2024-09-18T08:02:50.886830+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:04:07.696637+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":44,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-18T08:03:34.762777+0000","last_change":"2024-09-18T08:02:47.875508+0000","last_active":"2024-09-18T08:03:34.762777+0000","last_peered":"2024-09-18T08:03:34.762777+0000","last_clean":"2024-09-18T08:03:34.762777+0000","last_became_active":"2024-09-18T08:02:47.875124+0000","last_became_peered":"2024-09-18T08:02:47.875124+0000","last_unstale":"2024-09-18T08:03:34.762777+0000","last_undegraded":"2024-09-18T08:03:34.762777+0000","last_fullsized":"2024-09-18T08:03:34.762777+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:02:46.860092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:46.860092+0000","last_clean_scrub_stamp":"2024-09-18T08:02:46.860092+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:03:53.477209+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"69'9","reported_seq":56,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-18T08:03:34.047761+0000","last_change":"2024-09-18T08:02:49.887589+0000","last_active":"2024-09-18T08:03:34.047761+0000","last_peered":"2024-09-18T08:03:34.047761+0000","last_clean":"2024-09-18T08:03:34.047761+0000","last_became_active":"2024-09-18T08:02:49.887161+0000","last_became_peered":"2024-09-18T08:02:49.887161+0000","last_unstale":"2024-09-18T08:03:34.047761+0000","last_undegraded":"2024-09-18T08:03:34.047761+0000","last_fullsized":"2024-09-18T08:03:34.047761+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_clean_scrub_stamp":"2024-09-18T08:02:48.870718+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:03:51.892069+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":36,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-18T08:03:34.762220+0000","last_change":"2024-09-18T08:02:51.907597+0000","last_active":"2024-09-18T08:03:34.762220+0000","last_peered":"2024-09-18T08:03:34.762220+0000","last_clean":"2024-09-18T08:03:34.762220+0000","last_became_active":"2024-09-18T08:02:51.907296+0000","last_became_peered":"2024-09-18T08:02:51.907296+0000","last_unstale":"2024-09-18T08:03:34.762220+0000","last_undegraded":"2024-09-18T08:03:34.762220+0000","last_fullsized":"2024-09-18T08:03:34.762220+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:02:50.886830+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:50.886830+0000","last_clean_scrub_stamp":"2024-09-18T08:02:50.886830+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:04:06.057644+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":33,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-18T08:03:34.046546+0000","last_change":"2024-09-18T08:02:53.913803+0000","last_active":"2024-09-18T08:03:34.046546+0000","last_peered":"2024-09-18T08:03:34.046546+0000","last_clean":"2024-09-18T08:03:34.046546+0000","last_became_active":"2024-09-18T08:02:53.913519+0000","last_became_peered":"2024-09-18T08:02:53.913519+0000","last_unstale":"2024-09-18T08:03:34.046546+0000","last_undegraded":"2024-09-18T08:03:34.046546+0000","last_fullsized":"2024-09-18T08:03:34.046546+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_clean_scrub_stamp":"2024-09-18T08:02:52.890475+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:04:18.699540+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":44,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-18T08:03:34.762467+0000","last_change":"2024-09-18T08:02:47.877577+0000","last_active":"2024-09-18T08:03:34.762467+0000","last_peered":"2024-09-18T08:03:34.762467+0000","last_clean":"2024-09-18T08:03:34.762467+0000","last_became_active":"2024-09-18T08:02:47.875042+0000","last_became_peered":"2024-09-18T08:02:47.875042+0000","last_unstale":"2024-09-18T08:03:34.762467+0000","last_undegraded":"2024-09-18T08:03:34.762467+0000","last_fullsized":"2024-09-18T08:03:34.762467+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:02:46.860092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:46.860092+0000","last_clean_scrub_stamp":"2024-09-18T08:02:46.860092+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:04:14.469212+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1b","version":"69'5","reported_seq":50,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-18T08:03:34.047942+0000","last_change":"2024-09-18T08:02:49.895779+0000","last_active":"2024-09-18T08:03:34.047942+0000","last_peered":"2024-09-18T08:03:34.047942+0000","last_clean":"2024-09-18T08:03:34.047942+0000","last_became_active":"2024-09-18T08:02:49.895344+0000","last_became_peered":"2024-09-18T08:02:49.895344+0000","last_unstale":"2024-09-18T08:03:34.047942+0000","last_undegraded":"2024-09-18T08:03:34.047942+0000","last_fullsized":"2024-09-18T08:03:34.047942+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_clean_scrub_stamp":"2024-09-18T08:02:48.870718+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:04:08.365283+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":11,"num_read_kb":7,"num_write":6,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":36,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-18T08:03:34.572593+0000","last_change":"2024-09-18T08:02:51.906694+0000","last_active":"2024-09-18T08:03:34.572593+0000","last_peered":"2024-09-18T08:03:34.572593+0000","last_clean":"2024-09-18T08:03:34.572593+0000","last_became_active":"2024-09-18T08:02:51.905776+0000","last_became_peered":"2024-09-18T08:02:51.905776+0000","last_unstale":"2024-09-18T08:03:34.572593+0000","last_undegraded":"2024-09-18T08:03:34.572593+0000","last_fullsized":"2024-09-18T08:03:34.572593+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:02:50.886830+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:50.886830+0000","last_clean_scrub_stamp":"2024-09-18T08:02:50.886830+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:04:07.273083+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":33,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-18T08:03:34.762025+0000","last_change":"2024-09-18T08:02:53.911969+0000","last_active":"2024-09-18T08:03:34.762025+0000","last_peered":"2024-09-18T08:03:34.762025+0000","last_clean":"2024-09-18T08:03:34.762025+0000","last_became_active":"2024-09-18T08:02:53.911648+0000","last_became_peered":"2024-09-18T08:02:53.911648+0000","last_unstale":"2024-09-18T08:03:34.762025+0000","last_undegraded":"2024-09-18T08:03:34.762025+0000","last_fullsized":"2024-09-18T08:03:34.762025+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_clean_scrub_stamp":"2024-09-18T08:02:52.890475+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:04:21.185524+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":32,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-18T08:03:34.047503+0000","last_change":"2024-09-18T08:02:54.047071+0000","last_active":"2024-09-18T08:03:34.047503+0000","last_peered":"2024-09-18T08:03:34.047503+0000","last_clean":"2024-09-18T08:03:34.047503+0000","last_became_active":"2024-09-18T08:02:54.046776+0000","last_became_peered":"2024-09-18T08:02:54.046776+0000","last_unstale":"2024-09-18T08:03:34.047503+0000","last_undegraded":"2024-09-18T08:03:34.047503+0000","last_fullsized":"2024-09-18T08:03:34.047503+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_clean_scrub_stamp":"2024-09-18T08:02:52.890475+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:04:11.355797+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":44,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-18T08:03:34.047827+0000","last_change":"2024-09-18T08:02:47.876235+0000","last_active":"2024-09-18T08:03:34.047827+0000","last_peered":"2024-09-18T08:03:34.047827+0000","last_clean":"2024-09-18T08:03:34.047827+0000","last_became_active":"2024-09-18T08:02:47.875963+0000","last_became_peered":"2024-09-18T08:02:47.875963+0000","last_unstale":"2024-09-18T08:03:34.047827+0000","last_undegraded":"2024-09-18T08:03:34.047827+0000","last_fullsized":"2024-09-18T08:03:34.047827+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:02:46.860092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:46.860092+0000","last_clean_scrub_stamp":"2024-09-18T08:02:46.860092+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:04:11.609411+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1c","version":"69'15","reported_seq":65,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-18T08:03:34.049141+0000","last_change":"2024-09-18T08:02:49.885923+0000","last_active":"2024-09-18T08:03:34.049141+0000","last_peered":"2024-09-18T08:03:34.049141+0000","last_clean":"2024-09-18T08:03:34.049141+0000","last_became_active":"2024-09-18T08:02:49.885553+0000","last_became_peered":"2024-09-18T08:02:49.885553+0000","last_unstale":"2024-09-18T08:03:34.049141+0000","last_undegraded":"2024-09-18T08:03:34.049141+0000","last_fullsized":"2024-09-18T08:03:34.049141+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_clean_scrub_stamp":"2024-09-18T08:02:48.870718+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:04:11.220046+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":36,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-18T08:03:35.232338+0000","last_change":"2024-09-18T08:02:51.908750+0000","last_active":"2024-09-18T08:03:35.232338+0000","last_peered":"2024-09-18T08:03:35.232338+0000","last_clean":"2024-09-18T08:03:35.232338+0000","last_became_active":"2024-09-18T08:02:51.908430+0000","last_became_peered":"2024-09-18T08:02:51.908430+0000","last_unstale":"2024-09-18T08:03:35.232338+0000","last_undegraded":"2024-09-18T08:03:35.232338+0000","last_fullsized":"2024-09-18T08:03:35.232338+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:02:50.886830+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:50.886830+0000","last_clean_scrub_stamp":"2024-09-18T08:02:50.886830+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:03:58.654545+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":32,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-18T08:03:34.872572+0000","last_change":"2024-09-18T08:02:54.047573+0000","last_active":"2024-09-18T08:03:34.872572+0000","last_peered":"2024-09-18T08:03:34.872572+0000","last_clean":"2024-09-18T08:03:34.872572+0000","last_became_active":"2024-09-18T08:02:54.047162+0000","last_became_peered":"2024-09-18T08:02:54.047162+0000","last_unstale":"2024-09-18T08:03:34.872572+0000","last_undegraded":"2024-09-18T08:03:34.872572+0000","last_fullsized":"2024-09-18T08:03:34.872572+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_clean_scrub_stamp":"2024-09-18T08:02:52.890475+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:04:02.287865+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":44,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-18T08:03:34.047961+0000","last_change":"2024-09-18T08:02:47.871895+0000","last_active":"2024-09-18T08:03:34.047961+0000","last_peered":"2024-09-18T08:03:34.047961+0000","last_clean":"2024-09-18T08:03:34.047961+0000","last_became_active":"2024-09-18T08:02:47.871523+0000","last_became_peered":"2024-09-18T08:02:47.871523+0000","last_unstale":"2024-09-18T08:03:34.047961+0000","last_undegraded":"2024-09-18T08:03:34.047961+0000","last_fullsized":"2024-09-18T08:03:34.047961+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:02:46.860092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:46.860092+0000","last_clean_scrub_stamp":"2024-09-18T08:02:46.860092+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:04:03.333307+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"69'12","reported_seq":63,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-18T08:03:34.872420+0000","last_change":"2024-09-18T08:02:49.889182+0000","last_active":"2024-09-18T08:03:34.872420+0000","last_peered":"2024-09-18T08:03:34.872420+0000","last_clean":"2024-09-18T08:03:34.872420+0000","last_became_active":"2024-09-18T08:02:49.888831+0000","last_became_peered":"2024-09-18T08:02:49.888831+0000","last_unstale":"2024-09-18T08:03:34.872420+0000","last_undegraded":"2024-09-18T08:03:34.872420+0000","last_fullsized":"2024-09-18T08:03:34.872420+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_clean_scrub_stamp":"2024-09-18T08:02:48.870718+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:04:12.415919+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":25,"num_read_kb":16,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":36,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-18T08:03:34.048003+0000","last_change":"2024-09-18T08:02:51.910642+0000","last_active":"2024-09-18T08:03:34.048003+0000","last_peered":"2024-09-18T08:03:34.048003+0000","last_clean":"2024-09-18T08:03:34.048003+0000","last_became_active":"2024-09-18T08:02:51.910316+0000","last_became_peered":"2024-09-18T08:02:51.910316+0000","last_unstale":"2024-09-18T08:03:34.048003+0000","last_undegraded":"2024-09-18T08:03:34.048003+0000","last_fullsized":"2024-09-18T08:03:34.048003+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:02:50.886830+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:50.886830+0000","last_clean_scrub_stamp":"2024-09-18T08:02:50.886830+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:04:05.261448+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1c","version":"69'1","reported_seq":34,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-18T08:03:34.572852+0000","last_change":"2024-09-18T08:02:53.910861+0000","last_active":"2024-09-18T08:03:34.572852+0000","last_peered":"2024-09-18T08:03:34.572852+0000","last_clean":"2024-09-18T08:03:34.572852+0000","last_became_active":"2024-09-18T08:02:53.910589+0000","last_became_peered":"2024-09-18T08:02:53.910589+0000","last_unstale":"2024-09-18T08:03:34.572852+0000","last_undegraded":"2024-09-18T08:03:34.572852+0000","last_fullsized":"2024-09-18T08:03:34.572852+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_clean_scrub_stamp":"2024-09-18T08:02:52.890475+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:04:20.224154+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":403,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.19","version":"62'1","reported_seq":45,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-18T08:03:35.231922+0000","last_change":"2024-09-18T08:02:47.881314+0000","last_active":"2024-09-18T08:03:35.231922+0000","last_peered":"2024-09-18T08:03:35.231922+0000","last_clean":"2024-09-18T08:03:35.231922+0000","last_became_active":"2024-09-18T08:02:47.880998+0000","last_became_peered":"2024-09-18T08:02:47.880998+0000","last_unstale":"2024-09-18T08:03:35.231922+0000","last_undegraded":"2024-09-18T08:03:35.231922+0000","last_fullsized":"2024-09-18T08:03:35.231922+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:02:46.860092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:46.860092+0000","last_clean_scrub_stamp":"2024-09-18T08:02:46.860092+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:03:57.000100+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1e","version":"69'10","reported_seq":55,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-18T08:03:34.047651+0000","last_change":"2024-09-18T08:02:49.892755+0000","last_active":"2024-09-18T08:03:34.047651+0000","last_peered":"2024-09-18T08:03:34.047651+0000","last_clean":"2024-09-18T08:03:34.047651+0000","last_became_active":"2024-09-18T08:02:49.892024+0000","last_became_peered":"2024-09-18T08:02:49.892024+0000","last_unstale":"2024-09-18T08:03:34.047651+0000","last_undegraded":"2024-09-18T08:03:34.047651+0000","last_fullsized":"2024-09-18T08:03:34.047651+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_clean_scrub_stamp":"2024-09-18T08:02:48.870718+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:04:17.237822+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1f","version":"69'8","reported_seq":59,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-18T08:03:37.938577+0000","last_change":"2024-09-18T08:02:51.904933+0000","last_active":"2024-09-18T08:03:37.938577+0000","last_peered":"2024-09-18T08:03:37.938577+0000","last_clean":"2024-09-18T08:03:37.938577+0000","last_became_active":"2024-09-18T08:02:51.904588+0000","last_became_peered":"2024-09-18T08:02:51.904588+0000","last_unstale":"2024-09-18T08:03:37.938577+0000","last_undegraded":"2024-09-18T08:03:37.938577+0000","last_fullsized":"2024-09-18T08:03:37.938577+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:02:50.886830+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:50.886830+0000","last_clean_scrub_stamp":"2024-09-18T08:02:50.886830+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:03:59.243324+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.f","version":"69'15","reported_seq":65,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-18T08:03:35.232547+0000","last_change":"2024-09-18T08:02:49.899404+0000","last_active":"2024-09-18T08:03:35.232547+0000","last_peered":"2024-09-18T08:03:35.232547+0000","last_clean":"2024-09-18T08:03:35.232547+0000","last_became_active":"2024-09-18T08:02:49.899083+0000","last_became_peered":"2024-09-18T08:02:49.899083+0000","last_unstale":"2024-09-18T08:03:35.232547+0000","last_undegraded":"2024-09-18T08:03:35.232547+0000","last_fullsized":"2024-09-18T08:03:35.232547+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_clean_scrub_stamp":"2024-09-18T08:02:48.870718+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:04:06.513250+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":44,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-18T08:03:34.871892+0000","last_change":"2024-09-18T08:02:47.879723+0000","last_active":"2024-09-18T08:03:34.871892+0000","last_peered":"2024-09-18T08:03:34.871892+0000","last_clean":"2024-09-18T08:03:34.871892+0000","last_became_active":"2024-09-18T08:02:47.879328+0000","last_became_peered":"2024-09-18T08:02:47.879328+0000","last_unstale":"2024-09-18T08:03:34.871892+0000","last_undegraded":"2024-09-18T08:03:34.871892+0000","last_fullsized":"2024-09-18T08:03:34.871892+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:02:46.860092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:46.860092+0000","last_clean_scrub_stamp":"2024-09-18T08:02:46.860092+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:03:56.802301+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.e","version":"69'8","reported_seq":56,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-18T08:03:37.938579+0000","last_change":"2024-09-18T08:02:51.903673+0000","last_active":"2024-09-18T08:03:37.938579+0000","last_peered":"2024-09-18T08:03:37.938579+0000","last_clean":"2024-09-18T08:03:37.938579+0000","last_became_active":"2024-09-18T08:02:51.903369+0000","last_became_peered":"2024-09-18T08:02:51.903369+0000","last_unstale":"2024-09-18T08:03:37.938579+0000","last_undegraded":"2024-09-18T08:03:37.938579+0000","last_fullsized":"2024-09-18T08:03:37.938579+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:02:50.886830+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:50.886830+0000","last_clean_scrub_stamp":"2024-09-18T08:02:50.886830+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:03:51.149147+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":33,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-18T08:03:34.762381+0000","last_change":"2024-09-18T08:02:53.910161+0000","last_active":"2024-09-18T08:03:34.762381+0000","last_peered":"2024-09-18T08:03:34.762381+0000","last_clean":"2024-09-18T08:03:34.762381+0000","last_became_active":"2024-09-18T08:02:53.909834+0000","last_became_peered":"2024-09-18T08:02:53.909834+0000","last_unstale":"2024-09-18T08:03:34.762381+0000","last_undegraded":"2024-09-18T08:03:34.762381+0000","last_fullsized":"2024-09-18T08:03:34.762381+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_clean_scrub_stamp":"2024-09-18T08:02:52.890475+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:04:07.435248+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.0","version":"69'18","reported_seq":72,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-18T08:03:34.871648+0000","last_change":"2024-09-18T08:02:49.887135+0000","last_active":"2024-09-18T08:03:34.871648+0000","last_peered":"2024-09-18T08:03:34.871648+0000","last_clean":"2024-09-18T08:03:34.871648+0000","last_became_active":"2024-09-18T08:02:49.886818+0000","last_became_peered":"2024-09-18T08:02:49.886818+0000","last_unstale":"2024-09-18T08:03:34.871648+0000","last_undegraded":"2024-09-18T08:03:34.871648+0000","last_fullsized":"2024-09-18T08:03:34.871648+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_clean_scrub_stamp":"2024-09-18T08:02:48.870718+0000","objects_scrubbed":0,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:03:57.656879+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":34,"num_read_kb":22,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":44,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-18T08:03:34.872823+0000","last_change":"2024-09-18T08:02:47.880517+0000","last_active":"2024-09-18T08:03:34.872823+0000","last_peered":"2024-09-18T08:03:34.872823+0000","last_clean":"2024-09-18T08:03:34.872823+0000","last_became_active":"2024-09-18T08:02:47.880225+0000","last_became_peered":"2024-09-18T08:02:47.880225+0000","last_unstale":"2024-09-18T08:03:34.872823+0000","last_undegraded":"2024-09-18T08:03:34.872823+0000","last_fullsized":"2024-09-18T08:03:34.872823+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:02:46.860092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:46.860092+0000","last_clean_scrub_stamp":"2024-09-18T08:02:46.860092+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:03:51.223493+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":36,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-18T08:03:34.047272+0000","last_change":"2024-09-18T08:02:51.899911+0000","last_active":"2024-09-18T08:03:34.047272+0000","last_peered":"2024-09-18T08:03:34.047272+0000","last_clean":"2024-09-18T08:03:34.047272+0000","last_became_active":"2024-09-18T08:02:51.899594+0000","last_became_peered":"2024-09-18T08:02:51.899594+0000","last_unstale":"2024-09-18T08:03:34.047272+0000","last_undegraded":"2024-09-18T08:03:34.047272+0000","last_fullsized":"2024-09-18T08:03:34.047272+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:02:50.886830+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:50.886830+0000","last_clean_scrub_stamp":"2024-09-18T08:02:50.886830+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:04:02.261346+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":33,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-18T08:03:34.047646+0000","last_change":"2024-09-18T08:02:53.904262+0000","last_active":"2024-09-18T08:03:34.047646+0000","last_peered":"2024-09-18T08:03:34.047646+0000","last_clean":"2024-09-18T08:03:34.047646+0000","last_became_active":"2024-09-18T08:02:53.903957+0000","last_became_peered":"2024-09-18T08:02:53.903957+0000","last_unstale":"2024-09-18T08:03:34.047646+0000","last_undegraded":"2024-09-18T08:03:34.047646+0000","last_fullsized":"2024-09-18T08:03:34.047646+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_clean_scrub_stamp":"2024-09-18T08:02:52.890475+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:04:02.672046+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1","version":"69'14","reported_seq":61,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-18T08:03:34.047364+0000","last_change":"2024-09-18T08:02:49.894205+0000","last_active":"2024-09-18T08:03:34.047364+0000","last_peered":"2024-09-18T08:03:34.047364+0000","last_clean":"2024-09-18T08:03:34.047364+0000","last_became_active":"2024-09-18T08:02:49.893830+0000","last_became_peered":"2024-09-18T08:02:49.893830+0000","last_unstale":"2024-09-18T08:03:34.047364+0000","last_undegraded":"2024-09-18T08:03:34.047364+0000","last_fullsized":"2024-09-18T08:03:34.047364+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_clean_scrub_stamp":"2024-09-18T08:02:48.870718+0000","objects_scrubbed":0,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:03:51.325184+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":21,"num_read_kb":14,"num_write":14,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"62'2","reported_seq":51,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-18T08:03:34.047774+0000","last_change":"2024-09-18T08:02:47.873172+0000","last_active":"2024-09-18T08:03:34.047774+0000","last_peered":"2024-09-18T08:03:34.047774+0000","last_clean":"2024-09-18T08:03:34.047774+0000","last_became_active":"2024-09-18T08:02:47.872857+0000","last_became_peered":"2024-09-18T08:02:47.872857+0000","last_unstale":"2024-09-18T08:03:34.047774+0000","last_undegraded":"2024-09-18T08:03:34.047774+0000","last_fullsized":"2024-09-18T08:03:34.047774+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:02:46.860092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:46.860092+0000","last_clean_scrub_stamp":"2024-09-18T08:02:46.860092+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:04:03.733922+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1039,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.0","version":"69'8","reported_seq":56,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-18T08:03:37.938449+0000","last_change":"2024-09-18T08:02:51.910339+0000","last_active":"2024-09-18T08:03:37.938449+0000","last_peered":"2024-09-18T08:03:37.938449+0000","last_clean":"2024-09-18T08:03:37.938449+0000","last_became_active":"2024-09-18T08:02:51.909680+0000","last_became_peered":"2024-09-18T08:02:51.909680+0000","last_unstale":"2024-09-18T08:03:37.938449+0000","last_undegraded":"2024-09-18T08:03:37.938449+0000","last_fullsized":"2024-09-18T08:03:37.938449+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:02:50.886830+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:50.886830+0000","last_clean_scrub_stamp":"2024-09-18T08:02:50.886830+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:03:58.456713+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":33,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-18T08:03:34.572209+0000","last_change":"2024-09-18T08:02:54.047452+0000","last_active":"2024-09-18T08:03:34.572209+0000","last_peered":"2024-09-18T08:03:34.572209+0000","last_clean":"2024-09-18T08:03:34.572209+0000","last_became_active":"2024-09-18T08:02:54.047039+0000","last_became_peered":"2024-09-18T08:02:54.047039+0000","last_unstale":"2024-09-18T08:03:34.572209+0000","last_undegraded":"2024-09-18T08:03:34.572209+0000","last_fullsized":"2024-09-18T08:03:34.572209+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_clean_scrub_stamp":"2024-09-18T08:02:52.890475+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:03:58.508399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.2","version":"69'10","reported_seq":55,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-18T08:03:35.232029+0000","last_change":"2024-09-18T08:02:49.896889+0000","last_active":"2024-09-18T08:03:35.232029+0000","last_peered":"2024-09-18T08:03:35.232029+0000","last_clean":"2024-09-18T08:03:35.232029+0000","last_became_active":"2024-09-18T08:02:49.895265+0000","last_became_peered":"2024-09-18T08:02:49.895265+0000","last_unstale":"2024-09-18T08:03:35.232029+0000","last_undegraded":"2024-09-18T08:03:35.232029+0000","last_fullsized":"2024-09-18T08:03:35.232029+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_clean_scrub_stamp":"2024-09-18T08:02:48.870718+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:04:06.130656+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":44,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-18T08:03:34.762826+0000","last_change":"2024-09-18T08:02:47.877672+0000","last_active":"2024-09-18T08:03:34.762826+0000","last_peered":"2024-09-18T08:03:34.762826+0000","last_clean":"2024-09-18T08:03:34.762826+0000","last_became_active":"2024-09-18T08:02:47.874727+0000","last_became_peered":"2024-09-18T08:02:47.874727+0000","last_unstale":"2024-09-18T08:03:34.762826+0000","last_undegraded":"2024-09-18T08:03:34.762826+0000","last_fullsized":"2024-09-18T08:03:34.762826+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:02:46.860092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:46.860092+0000","last_clean_scrub_stamp":"2024-09-18T08:02:46.860092+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:03:47.082997+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.3","version":"69'8","reported_seq":56,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-18T08:03:37.938414+0000","last_change":"2024-09-18T08:02:51.906369+0000","last_active":"2024-09-18T08:03:37.938414+0000","last_peered":"2024-09-18T08:03:37.938414+0000","last_clean":"2024-09-18T08:03:37.938414+0000","last_became_active":"2024-09-18T08:02:51.906055+0000","last_became_peered":"2024-09-18T08:02:51.906055+0000","last_unstale":"2024-09-18T08:03:37.938414+0000","last_undegraded":"2024-09-18T08:03:37.938414+0000","last_fullsized":"2024-09-18T08:03:37.938414+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:02:50.886830+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:50.886830+0000","last_clean_scrub_stamp":"2024-09-18T08:02:50.886830+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:04:18.452195+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":33,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-18T08:03:34.046643+0000","last_change":"2024-09-18T08:02:53.910057+0000","last_active":"2024-09-18T08:03:34.046643+0000","last_peered":"2024-09-18T08:03:34.046643+0000","last_clean":"2024-09-18T08:03:34.046643+0000","last_became_active":"2024-09-18T08:02:53.909583+0000","last_became_peered":"2024-09-18T08:02:53.909583+0000","last_unstale":"2024-09-18T08:03:34.046643+0000","last_undegraded":"2024-09-18T08:03:34.046643+0000","last_fullsized":"2024-09-18T08:03:34.046643+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_clean_scrub_stamp":"2024-09-18T08:02:52.890475+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:04:07.545769+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.3","version":"69'19","reported_seq":76,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-18T08:03:34.047555+0000","last_change":"2024-09-18T08:02:49.890631+0000","last_active":"2024-09-18T08:03:34.047555+0000","last_peered":"2024-09-18T08:03:34.047555+0000","last_clean":"2024-09-18T08:03:34.047555+0000","last_became_active":"2024-09-18T08:02:49.890318+0000","last_became_peered":"2024-09-18T08:02:49.890318+0000","last_unstale":"2024-09-18T08:03:34.047555+0000","last_undegraded":"2024-09-18T08:03:34.047555+0000","last_fullsized":"2024-09-18T08:03:34.047555+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_clean_scrub_stamp":"2024-09-18T08:02:48.870718+0000","objects_scrubbed":0,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:03:57.953144+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":39,"num_read_kb":25,"num_write":22,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.4","version":"62'1","reported_seq":50,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-18T08:03:35.232342+0000","last_change":"2024-09-18T08:02:47.875046+0000","last_active":"2024-09-18T08:03:35.232342+0000","last_peered":"2024-09-18T08:03:35.232342+0000","last_clean":"2024-09-18T08:03:35.232342+0000","last_became_active":"2024-09-18T08:02:47.874764+0000","last_became_peered":"2024-09-18T08:02:47.874764+0000","last_unstale":"2024-09-18T08:03:35.232342+0000","last_undegraded":"2024-09-18T08:03:35.232342+0000","last_fullsized":"2024-09-18T08:03:35.232342+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:02:46.860092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:46.860092+0000","last_clean_scrub_stamp":"2024-09-18T08:02:46.860092+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:04:07.354571+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":436,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":36,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-18T08:03:34.497175+0000","last_change":"2024-09-18T08:02:51.908344+0000","last_active":"2024-09-18T08:03:34.497175+0000","last_peered":"2024-09-18T08:03:34.497175+0000","last_clean":"2024-09-18T08:03:34.497175+0000","last_became_active":"2024-09-18T08:02:51.908068+0000","last_became_peered":"2024-09-18T08:02:51.908068+0000","last_unstale":"2024-09-18T08:03:34.497175+0000","last_undegraded":"2024-09-18T08:03:34.497175+0000","last_fullsized":"2024-09-18T08:03:34.497175+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:02:50.886830+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:50.886830+0000","last_clean_scrub_stamp":"2024-09-18T08:02:50.886830+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:04:06.242055+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":32,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-18T08:03:35.231750+0000","last_change":"2024-09-18T08:02:54.046350+0000","last_active":"2024-09-18T08:03:35.231750+0000","last_peered":"2024-09-18T08:03:35.231750+0000","last_clean":"2024-09-18T08:03:35.231750+0000","last_became_active":"2024-09-18T08:02:54.045726+0000","last_became_peered":"2024-09-18T08:02:54.045726+0000","last_unstale":"2024-09-18T08:03:35.231750+0000","last_undegraded":"2024-09-18T08:03:35.231750+0000","last_fullsized":"2024-09-18T08:03:35.231750+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_clean_scrub_stamp":"2024-09-18T08:02:52.890475+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:03:55.585366+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.4","version":"69'28","reported_seq":100,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-18T08:03:37.938420+0000","last_change":"2024-09-18T08:02:49.897515+0000","last_active":"2024-09-18T08:03:37.938420+0000","last_peered":"2024-09-18T08:03:37.938420+0000","last_clean":"2024-09-18T08:03:37.938420+0000","last_became_active":"2024-09-18T08:02:49.897221+0000","last_became_peered":"2024-09-18T08:02:49.897221+0000","last_unstale":"2024-09-18T08:03:37.938420+0000","last_undegraded":"2024-09-18T08:03:37.938420+0000","last_fullsized":"2024-09-18T08:03:37.938420+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_clean_scrub_stamp":"2024-09-18T08:02:48.870718+0000","objects_scrubbed":0,"log_size":28,"log_dups_size":0,"ondisk_log_size":28,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:04:04.818428+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":48,"num_read_kb":33,"num_write":26,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":44,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-18T08:03:34.048351+0000","last_change":"2024-09-18T08:02:47.877445+0000","last_active":"2024-09-18T08:03:34.048351+0000","last_peered":"2024-09-18T08:03:34.048351+0000","last_clean":"2024-09-18T08:03:34.048351+0000","last_became_active":"2024-09-18T08:02:47.877085+0000","last_became_peered":"2024-09-18T08:02:47.877085+0000","last_unstale":"2024-09-18T08:03:34.048351+0000","last_undegraded":"2024-09-18T08:03:34.048351+0000","last_fullsized":"2024-09-18T08:03:34.048351+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:02:46.860092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:46.860092+0000","last_clean_scrub_stamp":"2024-09-18T08:02:46.860092+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:04:09.365906+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"64'2","reported_seq":54,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-18T08:03:34.762258+0000","last_change":"2024-09-18T08:02:49.883480+0000","last_active":"2024-09-18T08:03:34.762258+0000","last_peered":"2024-09-18T08:03:34.762258+0000","last_clean":"2024-09-18T08:03:34.762258+0000","last_became_active":"2024-09-18T08:02:46.867802+0000","last_became_peered":"2024-09-18T08:02:46.867802+0000","last_unstale":"2024-09-18T08:03:34.762258+0000","last_undegraded":"2024-09-18T08:03:34.762258+0000","last_fullsized":"2024-09-18T08:03:34.762258+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:02:45.855979+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:45.855979+0000","last_clean_scrub_stamp":"2024-09-18T08:02:45.855979+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:04:13.406442+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001348349,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":36,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-18T08:03:34.047360+0000","last_change":"2024-09-18T08:02:51.905694+0000","last_active":"2024-09-18T08:03:34.047360+0000","last_peered":"2024-09-18T08:03:34.047360+0000","last_clean":"2024-09-18T08:03:34.047360+0000","last_became_active":"2024-09-18T08:02:51.905357+0000","last_became_peered":"2024-09-18T08:02:51.905357+0000","last_unstale":"2024-09-18T08:03:34.047360+0000","last_undegraded":"2024-09-18T08:03:34.047360+0000","last_fullsized":"2024-09-18T08:03:34.047360+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:02:50.886830+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:50.886830+0000","last_clean_scrub_stamp":"2024-09-18T08:02:50.886830+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:04:15.739565+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":33,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-18T08:03:34.872458+0000","last_change":"2024-09-18T08:02:53.911634+0000","last_active":"2024-09-18T08:03:34.872458+0000","last_peered":"2024-09-18T08:03:34.872458+0000","last_clean":"2024-09-18T08:03:34.872458+0000","last_became_active":"2024-09-18T08:02:53.911036+0000","last_became_peered":"2024-09-18T08:02:53.911036+0000","last_unstale":"2024-09-18T08:03:34.872458+0000","last_undegraded":"2024-09-18T08:03:34.872458+0000","last_fullsized":"2024-09-18T08:03:34.872458+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_clean_scrub_stamp":"2024-09-18T08:02:52.890475+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:04:16.930353+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.7","version":"69'13","reported_seq":67,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-18T08:03:35.232554+0000","last_change":"2024-09-18T08:02:49.899404+0000","last_active":"2024-09-18T08:03:35.232554+0000","last_peered":"2024-09-18T08:03:35.232554+0000","last_clean":"2024-09-18T08:03:35.232554+0000","last_became_active":"2024-09-18T08:02:49.898987+0000","last_became_peered":"2024-09-18T08:02:49.898987+0000","last_unstale":"2024-09-18T08:03:35.232554+0000","last_undegraded":"2024-09-18T08:03:35.232554+0000","last_fullsized":"2024-09-18T08:03:35.232554+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_clean_scrub_stamp":"2024-09-18T08:02:48.870718+0000","objects_scrubbed":0,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:03:50.031411+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":19,"num_write":16,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":44,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-18T08:03:35.232038+0000","last_change":"2024-09-18T08:02:47.871732+0000","last_active":"2024-09-18T08:03:35.232038+0000","last_peered":"2024-09-18T08:03:35.232038+0000","last_clean":"2024-09-18T08:03:35.232038+0000","last_became_active":"2024-09-18T08:02:47.871416+0000","last_became_peered":"2024-09-18T08:02:47.871416+0000","last_unstale":"2024-09-18T08:03:35.232038+0000","last_undegraded":"2024-09-18T08:03:35.232038+0000","last_fullsized":"2024-09-18T08:03:35.232038+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:02:46.860092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:46.860092+0000","last_clean_scrub_stamp":"2024-09-18T08:02:46.860092+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:03:48.825242+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"62'1","reported_seq":53,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-18T08:03:34.048036+0000","last_change":"2024-09-18T08:02:49.877280+0000","last_active":"2024-09-18T08:03:34.048036+0000","last_peered":"2024-09-18T08:03:34.048036+0000","last_clean":"2024-09-18T08:03:34.048036+0000","last_became_active":"2024-09-18T08:02:46.868175+0000","last_became_peered":"2024-09-18T08:02:46.868175+0000","last_unstale":"2024-09-18T08:03:34.048036+0000","last_undegraded":"2024-09-18T08:03:34.048036+0000","last_fullsized":"2024-09-18T08:03:34.048036+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:02:45.855979+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:45.855979+0000","last_clean_scrub_stamp":"2024-09-18T08:02:45.855979+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:04:02.537405+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00066785499999999997,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":36,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-18T08:03:34.048609+0000","last_change":"2024-09-18T08:02:51.908417+0000","last_active":"2024-09-18T08:03:34.048609+0000","last_peered":"2024-09-18T08:03:34.048609+0000","last_clean":"2024-09-18T08:03:34.048609+0000","last_became_active":"2024-09-18T08:02:51.908156+0000","last_became_peered":"2024-09-18T08:02:51.908156+0000","last_unstale":"2024-09-18T08:03:34.048609+0000","last_undegraded":"2024-09-18T08:03:34.048609+0000","last_fullsized":"2024-09-18T08:03:34.048609+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:02:50.886830+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:50.886830+0000","last_clean_scrub_stamp":"2024-09-18T08:02:50.886830+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:04:04.453603+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":33,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-18T08:03:34.572969+0000","last_change":"2024-09-18T08:02:54.048417+0000","last_active":"2024-09-18T08:03:34.572969+0000","last_peered":"2024-09-18T08:03:34.572969+0000","last_clean":"2024-09-18T08:03:34.572969+0000","last_became_active":"2024-09-18T08:02:54.048004+0000","last_became_peered":"2024-09-18T08:02:54.048004+0000","last_unstale":"2024-09-18T08:03:34.572969+0000","last_undegraded":"2024-09-18T08:03:34.572969+0000","last_fullsized":"2024-09-18T08:03:34.572969+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_clean_scrub_stamp":"2024-09-18T08:02:52.890475+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:04:20.672566+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"69'12","reported_seq":58,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-18T08:03:34.047686+0000","last_change":"2024-09-18T08:02:49.898383+0000","last_active":"2024-09-18T08:03:34.047686+0000","last_peered":"2024-09-18T08:03:34.047686+0000","last_clean":"2024-09-18T08:03:34.047686+0000","last_became_active":"2024-09-18T08:02:49.898079+0000","last_became_peered":"2024-09-18T08:02:49.898079+0000","last_unstale":"2024-09-18T08:03:34.047686+0000","last_undegraded":"2024-09-18T08:03:34.047686+0000","last_fullsized":"2024-09-18T08:03:34.047686+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_clean_scrub_stamp":"2024-09-18T08:02:48.870718+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:04:14.339135+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":18,"num_read_kb":12,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":44,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-18T08:03:34.046872+0000","last_change":"2024-09-18T08:02:47.874335+0000","last_active":"2024-09-18T08:03:34.046872+0000","last_peered":"2024-09-18T08:03:34.046872+0000","last_clean":"2024-09-18T08:03:34.046872+0000","last_became_active":"2024-09-18T08:02:47.873809+0000","last_became_peered":"2024-09-18T08:02:47.873809+0000","last_unstale":"2024-09-18T08:03:34.046872+0000","last_undegraded":"2024-09-18T08:03:34.046872+0000","last_fullsized":"2024-09-18T08:03:34.046872+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:02:46.860092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:46.860092+0000","last_clean_scrub_stamp":"2024-09-18T08:02:46.860092+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:03:48.336434+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"70'5","reported_seq":105,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-18T08:03:41.721534+0000","last_change":"2024-09-18T08:02:49.877461+0000","last_active":"2024-09-18T08:03:41.721534+0000","last_peered":"2024-09-18T08:03:41.721534+0000","last_clean":"2024-09-18T08:03:41.721534+0000","last_became_active":"2024-09-18T08:02:46.872144+0000","last_became_peered":"2024-09-18T08:02:46.872144+0000","last_unstale":"2024-09-18T08:03:41.721534+0000","last_undegraded":"2024-09-18T08:03:41.721534+0000","last_fullsized":"2024-09-18T08:03:41.721534+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:02:45.855979+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:45.855979+0000","last_clean_scrub_stamp":"2024-09-18T08:02:45.855979+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:03:59.991143+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000504055,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":50,"num_read_kb":45,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":36,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-18T08:03:34.762850+0000","last_change":"2024-09-18T08:02:51.913423+0000","last_active":"2024-09-18T08:03:34.762850+0000","last_peered":"2024-09-18T08:03:34.762850+0000","last_clean":"2024-09-18T08:03:34.762850+0000","last_became_active":"2024-09-18T08:02:51.913112+0000","last_became_peered":"2024-09-18T08:02:51.913112+0000","last_unstale":"2024-09-18T08:03:34.762850+0000","last_undegraded":"2024-09-18T08:03:34.762850+0000","last_fullsized":"2024-09-18T08:03:34.762850+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:02:50.886830+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:50.886830+0000","last_clean_scrub_stamp":"2024-09-18T08:02:50.886830+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:04:04.493964+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":33,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-18T08:03:35.232003+0000","last_change":"2024-09-18T08:02:53.907689+0000","last_active":"2024-09-18T08:03:35.232003+0000","last_peered":"2024-09-18T08:03:35.232003+0000","last_clean":"2024-09-18T08:03:35.232003+0000","last_became_active":"2024-09-18T08:02:53.907404+0000","last_became_peered":"2024-09-18T08:02:53.907404+0000","last_unstale":"2024-09-18T08:03:35.232003+0000","last_undegraded":"2024-09-18T08:03:35.232003+0000","last_fullsized":"2024-09-18T08:03:35.232003+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_clean_scrub_stamp":"2024-09-18T08:02:52.890475+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:03:57.936771+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.5","version":"69'16","reported_seq":74,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-18T08:03:37.938704+0000","last_change":"2024-09-18T08:02:49.892366+0000","last_active":"2024-09-18T08:03:37.938704+0000","last_peered":"2024-09-18T08:03:37.938704+0000","last_clean":"2024-09-18T08:03:37.938704+0000","last_became_active":"2024-09-18T08:02:49.891985+0000","last_became_peered":"2024-09-18T08:02:49.891985+0000","last_unstale":"2024-09-18T08:03:37.938704+0000","last_undegraded":"2024-09-18T08:03:37.938704+0000","last_fullsized":"2024-09-18T08:03:37.938704+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_clean_scrub_stamp":"2024-09-18T08:02:48.870718+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:04:12.297325+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":25,"num_read_kb":15,"num_write":13,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":44,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-18T08:03:34.872048+0000","last_change":"2024-09-18T08:02:47.876225+0000","last_active":"2024-09-18T08:03:34.872048+0000","last_peered":"2024-09-18T08:03:34.872048+0000","last_clean":"2024-09-18T08:03:34.872048+0000","last_became_active":"2024-09-18T08:02:47.875950+0000","last_became_peered":"2024-09-18T08:02:47.875950+0000","last_unstale":"2024-09-18T08:03:34.872048+0000","last_undegraded":"2024-09-18T08:03:34.872048+0000","last_fullsized":"2024-09-18T08:03:34.872048+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:02:46.860092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:46.860092+0000","last_clean_scrub_stamp":"2024-09-18T08:02:46.860092+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:04:02.960274+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"74'79","reported_seq":90,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-18T08:03:34.572630+0000","last_change":"2024-09-18T08:02:55.472070+0000","last_active":"2024-09-18T08:03:34.572630+0000","last_peered":"2024-09-18T08:03:34.572630+0000","last_clean":"2024-09-18T08:03:34.572630+0000","last_became_active":"2024-09-18T08:02:39.446743+0000","last_became_peered":"2024-09-18T08:02:39.446743+0000","last_unstale":"2024-09-18T08:03:34.572630+0000","last_undegraded":"2024-09-18T08:03:34.572630+0000","last_fullsized":"2024-09-18T08:03:34.572630+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"20'72","last_scrub_stamp":"2024-09-18T08:02:55.471998+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:00:11.893742+0000","last_clean_scrub_stamp":"2024-09-18T08:02:55.471998+0000","objects_scrubbed":2,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:04:21.599062+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":231,"num_write":113,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":590368,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":36,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-18T08:03:34.573089+0000","last_change":"2024-09-18T08:02:51.905819+0000","last_active":"2024-09-18T08:03:34.573089+0000","last_peered":"2024-09-18T08:03:34.573089+0000","last_clean":"2024-09-18T08:03:34.573089+0000","last_became_active":"2024-09-18T08:02:51.905516+0000","last_became_peered":"2024-09-18T08:02:51.905516+0000","last_unstale":"2024-09-18T08:03:34.573089+0000","last_undegraded":"2024-09-18T08:03:34.573089+0000","last_fullsized":"2024-09-18T08:03:34.573089+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:02:50.886830+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:50.886830+0000","last_clean_scrub_stamp":"2024-09-18T08:02:50.886830+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:04:04.232646+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":33,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-18T08:03:34.762772+0000","last_change":"2024-09-18T08:02:53.909023+0000","last_active":"2024-09-18T08:03:34.762772+0000","last_peered":"2024-09-18T08:03:34.762772+0000","last_clean":"2024-09-18T08:03:34.762772+0000","last_became_active":"2024-09-18T08:02:53.908668+0000","last_became_peered":"2024-09-18T08:02:53.908668+0000","last_unstale":"2024-09-18T08:03:34.762772+0000","last_undegraded":"2024-09-18T08:03:34.762772+0000","last_fullsized":"2024-09-18T08:03:34.762772+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_clean_scrub_stamp":"2024-09-18T08:02:52.890475+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:03:57.881484+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.e","version":"69'11","reported_seq":59,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-18T08:03:34.048607+0000","last_change":"2024-09-18T08:02:49.895676+0000","last_active":"2024-09-18T08:03:34.048607+0000","last_peered":"2024-09-18T08:03:34.048607+0000","last_clean":"2024-09-18T08:03:34.048607+0000","last_became_active":"2024-09-18T08:02:49.895303+0000","last_became_peered":"2024-09-18T08:02:49.895303+0000","last_unstale":"2024-09-18T08:03:34.048607+0000","last_undegraded":"2024-09-18T08:03:34.048607+0000","last_fullsized":"2024-09-18T08:03:34.048607+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_clean_scrub_stamp":"2024-09-18T08:02:48.870718+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:04:11.479080+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":44,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-18T08:03:34.047619+0000","last_change":"2024-09-18T08:02:47.880265+0000","last_active":"2024-09-18T08:03:34.047619+0000","last_peered":"2024-09-18T08:03:34.047619+0000","last_clean":"2024-09-18T08:03:34.047619+0000","last_became_active":"2024-09-18T08:02:47.879764+0000","last_became_peered":"2024-09-18T08:02:47.879764+0000","last_unstale":"2024-09-18T08:03:34.047619+0000","last_undegraded":"2024-09-18T08:03:34.047619+0000","last_fullsized":"2024-09-18T08:03:34.047619+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:02:46.860092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:46.860092+0000","last_clean_scrub_stamp":"2024-09-18T08:02:46.860092+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:04:12.919106+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":36,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-18T08:03:34.762721+0000","last_change":"2024-09-18T08:02:51.911107+0000","last_active":"2024-09-18T08:03:34.762721+0000","last_peered":"2024-09-18T08:03:34.762721+0000","last_clean":"2024-09-18T08:03:34.762721+0000","last_became_active":"2024-09-18T08:02:51.910649+0000","last_became_peered":"2024-09-18T08:02:51.910649+0000","last_unstale":"2024-09-18T08:03:34.762721+0000","last_undegraded":"2024-09-18T08:03:34.762721+0000","last_fullsized":"2024-09-18T08:03:34.762721+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:02:50.886830+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:50.886830+0000","last_clean_scrub_stamp":"2024-09-18T08:02:50.886830+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:03:56.497732+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":32,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-18T08:03:34.872052+0000","last_change":"2024-09-18T08:02:54.045510+0000","last_active":"2024-09-18T08:03:34.872052+0000","last_peered":"2024-09-18T08:03:34.872052+0000","last_clean":"2024-09-18T08:03:34.872052+0000","last_became_active":"2024-09-18T08:02:54.045091+0000","last_became_peered":"2024-09-18T08:02:54.045091+0000","last_unstale":"2024-09-18T08:03:34.872052+0000","last_undegraded":"2024-09-18T08:03:34.872052+0000","last_fullsized":"2024-09-18T08:03:34.872052+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_clean_scrub_stamp":"2024-09-18T08:02:52.890475+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:04:22.368585+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.d","version":"69'17","reported_seq":68,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-18T08:03:34.048441+0000","last_change":"2024-09-18T08:02:49.895971+0000","last_active":"2024-09-18T08:03:34.048441+0000","last_peered":"2024-09-18T08:03:34.048441+0000","last_clean":"2024-09-18T08:03:34.048441+0000","last_became_active":"2024-09-18T08:02:49.895578+0000","last_became_peered":"2024-09-18T08:02:49.895578+0000","last_unstale":"2024-09-18T08:03:34.048441+0000","last_undegraded":"2024-09-18T08:03:34.048441+0000","last_fullsized":"2024-09-18T08:03:34.048441+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_clean_scrub_stamp":"2024-09-18T08:02:48.870718+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:04:17.784735+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":29,"num_read_kb":19,"num_write":18,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,1],"acting":[4,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":44,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-18T08:03:34.497394+0000","last_change":"2024-09-18T08:02:47.884431+0000","last_active":"2024-09-18T08:03:34.497394+0000","last_peered":"2024-09-18T08:03:34.497394+0000","last_clean":"2024-09-18T08:03:34.497394+0000","last_became_active":"2024-09-18T08:02:47.884132+0000","last_became_peered":"2024-09-18T08:02:47.884132+0000","last_unstale":"2024-09-18T08:03:34.497394+0000","last_undegraded":"2024-09-18T08:03:34.497394+0000","last_fullsized":"2024-09-18T08:03:34.497394+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:02:46.860092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:46.860092+0000","last_clean_scrub_stamp":"2024-09-18T08:02:46.860092+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:04:06.189609+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":36,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-18T08:03:35.232131+0000","last_change":"2024-09-18T08:02:51.907544+0000","last_active":"2024-09-18T08:03:35.232131+0000","last_peered":"2024-09-18T08:03:35.232131+0000","last_clean":"2024-09-18T08:03:35.232131+0000","last_became_active":"2024-09-18T08:02:51.907196+0000","last_became_peered":"2024-09-18T08:02:51.907196+0000","last_unstale":"2024-09-18T08:03:35.232131+0000","last_undegraded":"2024-09-18T08:03:35.232131+0000","last_fullsized":"2024-09-18T08:03:35.232131+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:02:50.886830+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:50.886830+0000","last_clean_scrub_stamp":"2024-09-18T08:02:50.886830+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:04:03.690724+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":33,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-18T08:03:34.048803+0000","last_change":"2024-09-18T08:02:53.904497+0000","last_active":"2024-09-18T08:03:34.048803+0000","last_peered":"2024-09-18T08:03:34.048803+0000","last_clean":"2024-09-18T08:03:34.048803+0000","last_became_active":"2024-09-18T08:02:53.904144+0000","last_became_peered":"2024-09-18T08:02:53.904144+0000","last_unstale":"2024-09-18T08:03:34.048803+0000","last_undegraded":"2024-09-18T08:03:34.048803+0000","last_fullsized":"2024-09-18T08:03:34.048803+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_clean_scrub_stamp":"2024-09-18T08:02:52.890475+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:03:58.164335+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.c","version":"69'10","reported_seq":55,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-18T08:03:34.048221+0000","last_change":"2024-09-18T08:02:49.892847+0000","last_active":"2024-09-18T08:03:34.048221+0000","last_peered":"2024-09-18T08:03:34.048221+0000","last_clean":"2024-09-18T08:03:34.048221+0000","last_became_active":"2024-09-18T08:02:49.892527+0000","last_became_peered":"2024-09-18T08:02:49.892527+0000","last_unstale":"2024-09-18T08:03:34.048221+0000","last_undegraded":"2024-09-18T08:03:34.048221+0000","last_fullsized":"2024-09-18T08:03:34.048221+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_clean_scrub_stamp":"2024-09-18T08:02:48.870718+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:04:11.391201+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":44,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-18T08:03:34.872284+0000","last_change":"2024-09-18T08:02:47.881814+0000","last_active":"2024-09-18T08:03:34.872284+0000","last_peered":"2024-09-18T08:03:34.872284+0000","last_clean":"2024-09-18T08:03:34.872284+0000","last_became_active":"2024-09-18T08:02:47.881512+0000","last_became_peered":"2024-09-18T08:02:47.881512+0000","last_unstale":"2024-09-18T08:03:34.872284+0000","last_undegraded":"2024-09-18T08:03:34.872284+0000","last_fullsized":"2024-09-18T08:03:34.872284+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:02:46.860092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:46.860092+0000","last_clean_scrub_stamp":"2024-09-18T08:02:46.860092+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:03:59.505935+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.d","version":"69'8","reported_seq":56,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-18T08:03:37.938487+0000","last_change":"2024-09-18T08:02:51.909290+0000","last_active":"2024-09-18T08:03:37.938487+0000","last_peered":"2024-09-18T08:03:37.938487+0000","last_clean":"2024-09-18T08:03:37.938487+0000","last_became_active":"2024-09-18T08:02:51.909011+0000","last_became_peered":"2024-09-18T08:02:51.909011+0000","last_unstale":"2024-09-18T08:03:37.938487+0000","last_undegraded":"2024-09-18T08:03:37.938487+0000","last_fullsized":"2024-09-18T08:03:37.938487+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:02:50.886830+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:50.886830+0000","last_clean_scrub_stamp":"2024-09-18T08:02:50.886830+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:04:10.431784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":33,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-18T08:03:34.048371+0000","last_change":"2024-09-18T08:02:53.902197+0000","last_active":"2024-09-18T08:03:34.048371+0000","last_peered":"2024-09-18T08:03:34.048371+0000","last_clean":"2024-09-18T08:03:34.048371+0000","last_became_active":"2024-09-18T08:02:53.901858+0000","last_became_peered":"2024-09-18T08:02:53.901858+0000","last_unstale":"2024-09-18T08:03:34.048371+0000","last_undegraded":"2024-09-18T08:03:34.048371+0000","last_fullsized":"2024-09-18T08:03:34.048371+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_clean_scrub_stamp":"2024-09-18T08:02:52.890475+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:04:08.426639+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.b","version":"69'9","reported_seq":56,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-18T08:03:34.047450+0000","last_change":"2024-09-18T08:02:49.898562+0000","last_active":"2024-09-18T08:03:34.047450+0000","last_peered":"2024-09-18T08:03:34.047450+0000","last_clean":"2024-09-18T08:03:34.047450+0000","last_became_active":"2024-09-18T08:02:49.897998+0000","last_became_peered":"2024-09-18T08:02:49.897998+0000","last_unstale":"2024-09-18T08:03:34.047450+0000","last_undegraded":"2024-09-18T08:03:34.047450+0000","last_fullsized":"2024-09-18T08:03:34.047450+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_clean_scrub_stamp":"2024-09-18T08:02:48.870718+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:03:56.995348+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":44,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-18T08:03:34.762561+0000","last_change":"2024-09-18T08:02:47.877928+0000","last_active":"2024-09-18T08:03:34.762561+0000","last_peered":"2024-09-18T08:03:34.762561+0000","last_clean":"2024-09-18T08:03:34.762561+0000","last_became_active":"2024-09-18T08:02:47.877161+0000","last_became_peered":"2024-09-18T08:02:47.877161+0000","last_unstale":"2024-09-18T08:03:34.762561+0000","last_undegraded":"2024-09-18T08:03:34.762561+0000","last_fullsized":"2024-09-18T08:03:34.762561+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:02:46.860092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:46.860092+0000","last_clean_scrub_stamp":"2024-09-18T08:02:46.860092+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:03:54.682345+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":36,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-18T08:03:34.047859+0000","last_change":"2024-09-18T08:02:51.901738+0000","last_active":"2024-09-18T08:03:34.047859+0000","last_peered":"2024-09-18T08:03:34.047859+0000","last_clean":"2024-09-18T08:03:34.047859+0000","last_became_active":"2024-09-18T08:02:51.901330+0000","last_became_peered":"2024-09-18T08:02:51.901330+0000","last_unstale":"2024-09-18T08:03:34.047859+0000","last_undegraded":"2024-09-18T08:03:34.047859+0000","last_fullsized":"2024-09-18T08:03:34.047859+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:02:50.886830+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:50.886830+0000","last_clean_scrub_stamp":"2024-09-18T08:02:50.886830+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:04:18.413020+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,3],"acting":[2,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":33,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-18T08:03:34.046755+0000","last_change":"2024-09-18T08:02:53.911400+0000","last_active":"2024-09-18T08:03:34.046755+0000","last_peered":"2024-09-18T08:03:34.046755+0000","last_clean":"2024-09-18T08:03:34.046755+0000","last_became_active":"2024-09-18T08:02:53.911068+0000","last_became_peered":"2024-09-18T08:02:53.911068+0000","last_unstale":"2024-09-18T08:03:34.046755+0000","last_undegraded":"2024-09-18T08:03:34.046755+0000","last_fullsized":"2024-09-18T08:03:34.046755+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_clean_scrub_stamp":"2024-09-18T08:02:52.890475+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:03:57.470235+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.a","version":"69'19","reported_seq":71,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-18T08:03:34.497158+0000","last_change":"2024-09-18T08:02:49.898366+0000","last_active":"2024-09-18T08:03:34.497158+0000","last_peered":"2024-09-18T08:03:34.497158+0000","last_clean":"2024-09-18T08:03:34.497158+0000","last_became_active":"2024-09-18T08:02:49.897960+0000","last_became_peered":"2024-09-18T08:02:49.897960+0000","last_unstale":"2024-09-18T08:03:34.497158+0000","last_undegraded":"2024-09-18T08:03:34.497158+0000","last_fullsized":"2024-09-18T08:03:34.497158+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_clean_scrub_stamp":"2024-09-18T08:02:48.870718+0000","objects_scrubbed":0,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:04:18.534695+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":32,"num_read_kb":21,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":44,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-18T08:03:34.573291+0000","last_change":"2024-09-18T08:02:47.881119+0000","last_active":"2024-09-18T08:03:34.573291+0000","last_peered":"2024-09-18T08:03:34.573291+0000","last_clean":"2024-09-18T08:03:34.573291+0000","last_became_active":"2024-09-18T08:02:47.880645+0000","last_became_peered":"2024-09-18T08:02:47.880645+0000","last_unstale":"2024-09-18T08:03:34.573291+0000","last_undegraded":"2024-09-18T08:03:34.573291+0000","last_fullsized":"2024-09-18T08:03:34.573291+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:02:46.860092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:46.860092+0000","last_clean_scrub_stamp":"2024-09-18T08:02:46.860092+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:04:02.022191+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":36,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-18T08:03:34.047799+0000","last_change":"2024-09-18T08:02:51.903944+0000","last_active":"2024-09-18T08:03:34.047799+0000","last_peered":"2024-09-18T08:03:34.047799+0000","last_clean":"2024-09-18T08:03:34.047799+0000","last_became_active":"2024-09-18T08:02:51.903439+0000","last_became_peered":"2024-09-18T08:02:51.903439+0000","last_unstale":"2024-09-18T08:03:34.047799+0000","last_undegraded":"2024-09-18T08:03:34.047799+0000","last_fullsized":"2024-09-18T08:03:34.047799+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:02:50.886830+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:50.886830+0000","last_clean_scrub_stamp":"2024-09-18T08:02:50.886830+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:03:52.993653+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":33,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-18T08:03:34.572548+0000","last_change":"2024-09-18T08:02:53.913096+0000","last_active":"2024-09-18T08:03:34.572548+0000","last_peered":"2024-09-18T08:03:34.572548+0000","last_clean":"2024-09-18T08:03:34.572548+0000","last_became_active":"2024-09-18T08:02:53.912762+0000","last_became_peered":"2024-09-18T08:02:53.912762+0000","last_unstale":"2024-09-18T08:03:34.572548+0000","last_undegraded":"2024-09-18T08:03:34.572548+0000","last_fullsized":"2024-09-18T08:03:34.572548+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_clean_scrub_stamp":"2024-09-18T08:02:52.890475+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:04:03.478804+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.9","version":"69'12","reported_seq":63,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-18T08:03:34.047514+0000","last_change":"2024-09-18T08:02:49.895136+0000","last_active":"2024-09-18T08:03:34.047514+0000","last_peered":"2024-09-18T08:03:34.047514+0000","last_clean":"2024-09-18T08:03:34.047514+0000","last_became_active":"2024-09-18T08:02:49.894762+0000","last_became_peered":"2024-09-18T08:02:49.894762+0000","last_unstale":"2024-09-18T08:03:34.047514+0000","last_undegraded":"2024-09-18T08:03:34.047514+0000","last_fullsized":"2024-09-18T08:03:34.047514+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_clean_scrub_stamp":"2024-09-18T08:02:48.870718+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:04:02.257011+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":25,"num_read_kb":16,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":44,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-18T08:03:34.573449+0000","last_change":"2024-09-18T08:02:47.884430+0000","last_active":"2024-09-18T08:03:34.573449+0000","last_peered":"2024-09-18T08:03:34.573449+0000","last_clean":"2024-09-18T08:03:34.573449+0000","last_became_active":"2024-09-18T08:02:47.884052+0000","last_became_peered":"2024-09-18T08:02:47.884052+0000","last_unstale":"2024-09-18T08:03:34.573449+0000","last_undegraded":"2024-09-18T08:03:34.573449+0000","last_fullsized":"2024-09-18T08:03:34.573449+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:02:46.860092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:46.860092+0000","last_clean_scrub_stamp":"2024-09-18T08:02:46.860092+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:04:04.711823+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":36,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-18T08:03:34.047203+0000","last_change":"2024-09-18T08:02:51.902541+0000","last_active":"2024-09-18T08:03:34.047203+0000","last_peered":"2024-09-18T08:03:34.047203+0000","last_clean":"2024-09-18T08:03:34.047203+0000","last_became_active":"2024-09-18T08:02:51.902163+0000","last_became_peered":"2024-09-18T08:02:51.902163+0000","last_unstale":"2024-09-18T08:03:34.047203+0000","last_undegraded":"2024-09-18T08:03:34.047203+0000","last_fullsized":"2024-09-18T08:03:34.047203+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:02:50.886830+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:50.886830+0000","last_clean_scrub_stamp":"2024-09-18T08:02:50.886830+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:03:54.180178+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":33,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-18T08:03:34.872076+0000","last_change":"2024-09-18T08:02:53.907752+0000","last_active":"2024-09-18T08:03:34.872076+0000","last_peered":"2024-09-18T08:03:34.872076+0000","last_clean":"2024-09-18T08:03:34.872076+0000","last_became_active":"2024-09-18T08:02:53.907412+0000","last_became_peered":"2024-09-18T08:02:53.907412+0000","last_unstale":"2024-09-18T08:03:34.872076+0000","last_undegraded":"2024-09-18T08:03:34.872076+0000","last_fullsized":"2024-09-18T08:03:34.872076+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_clean_scrub_stamp":"2024-09-18T08:02:52.890475+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:04:12.083897+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.8","version":"69'15","reported_seq":65,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-18T08:03:34.761818+0000","last_change":"2024-09-18T08:02:49.889133+0000","last_active":"2024-09-18T08:03:34.761818+0000","last_peered":"2024-09-18T08:03:34.761818+0000","last_clean":"2024-09-18T08:03:34.761818+0000","last_became_active":"2024-09-18T08:02:49.888799+0000","last_became_peered":"2024-09-18T08:02:49.888799+0000","last_unstale":"2024-09-18T08:03:34.761818+0000","last_undegraded":"2024-09-18T08:03:34.761818+0000","last_fullsized":"2024-09-18T08:03:34.761818+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_clean_scrub_stamp":"2024-09-18T08:02:48.870718+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:04:11.344161+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.f","version":"62'2","reported_seq":56,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-18T08:03:34.573437+0000","last_change":"2024-09-18T08:02:47.881441+0000","last_active":"2024-09-18T08:03:34.573437+0000","last_peered":"2024-09-18T08:03:34.573437+0000","last_clean":"2024-09-18T08:03:34.573437+0000","last_became_active":"2024-09-18T08:02:47.881165+0000","last_became_peered":"2024-09-18T08:02:47.881165+0000","last_unstale":"2024-09-18T08:03:34.573437+0000","last_undegraded":"2024-09-18T08:03:34.573437+0000","last_fullsized":"2024-09-18T08:03:34.573437+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:02:46.860092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:46.860092+0000","last_clean_scrub_stamp":"2024-09-18T08:02:46.860092+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:03:50.038477+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":4,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.9","version":"69'8","reported_seq":59,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-18T08:03:37.938596+0000","last_change":"2024-09-18T08:02:51.905340+0000","last_active":"2024-09-18T08:03:37.938596+0000","last_peered":"2024-09-18T08:03:37.938596+0000","last_clean":"2024-09-18T08:03:37.938596+0000","last_became_active":"2024-09-18T08:02:51.905059+0000","last_became_peered":"2024-09-18T08:02:51.905059+0000","last_unstale":"2024-09-18T08:03:37.938596+0000","last_undegraded":"2024-09-18T08:03:37.938596+0000","last_fullsized":"2024-09-18T08:03:37.938596+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:02:50.886830+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:50.886830+0000","last_clean_scrub_stamp":"2024-09-18T08:02:50.886830+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:03:52.023681+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":32,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-18T08:03:34.761721+0000","last_change":"2024-09-18T08:02:54.046138+0000","last_active":"2024-09-18T08:03:34.761721+0000","last_peered":"2024-09-18T08:03:34.761721+0000","last_clean":"2024-09-18T08:03:34.761721+0000","last_became_active":"2024-09-18T08:02:54.045845+0000","last_became_peered":"2024-09-18T08:02:54.045845+0000","last_unstale":"2024-09-18T08:03:34.761721+0000","last_undegraded":"2024-09-18T08:03:34.761721+0000","last_fullsized":"2024-09-18T08:03:34.761721+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_clean_scrub_stamp":"2024-09-18T08:02:52.890475+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:03:55.148975+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":44,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-18T08:03:34.497146+0000","last_change":"2024-09-18T08:02:47.882886+0000","last_active":"2024-09-18T08:03:34.497146+0000","last_peered":"2024-09-18T08:03:34.497146+0000","last_clean":"2024-09-18T08:03:34.497146+0000","last_became_active":"2024-09-18T08:02:47.882600+0000","last_became_peered":"2024-09-18T08:02:47.882600+0000","last_unstale":"2024-09-18T08:03:34.497146+0000","last_undegraded":"2024-09-18T08:03:34.497146+0000","last_fullsized":"2024-09-18T08:03:34.497146+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:02:46.860092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:46.860092+0000","last_clean_scrub_stamp":"2024-09-18T08:02:46.860092+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:04:15.079741+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"69'6","reported_seq":49,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-18T08:03:34.872748+0000","last_change":"2024-09-18T08:02:49.895477+0000","last_active":"2024-09-18T08:03:34.872748+0000","last_peered":"2024-09-18T08:03:34.872748+0000","last_clean":"2024-09-18T08:03:34.872748+0000","last_became_active":"2024-09-18T08:02:49.895067+0000","last_became_peered":"2024-09-18T08:02:49.895067+0000","last_unstale":"2024-09-18T08:03:34.872748+0000","last_undegraded":"2024-09-18T08:03:34.872748+0000","last_fullsized":"2024-09-18T08:03:34.872748+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_clean_scrub_stamp":"2024-09-18T08:02:48.870718+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:03:55.471468+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":9,"num_read_kb":6,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":36,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-18T08:03:34.762591+0000","last_change":"2024-09-18T08:02:51.912907+0000","last_active":"2024-09-18T08:03:34.762591+0000","last_peered":"2024-09-18T08:03:34.762591+0000","last_clean":"2024-09-18T08:03:34.762591+0000","last_became_active":"2024-09-18T08:02:51.912557+0000","last_became_peered":"2024-09-18T08:02:51.912557+0000","last_unstale":"2024-09-18T08:03:34.762591+0000","last_undegraded":"2024-09-18T08:03:34.762591+0000","last_fullsized":"2024-09-18T08:03:34.762591+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:02:50.886830+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:50.886830+0000","last_clean_scrub_stamp":"2024-09-18T08:02:50.886830+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:03:58.405243+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,1],"acting":[5,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":33,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-18T08:03:34.572238+0000","last_change":"2024-09-18T08:02:54.049521+0000","last_active":"2024-09-18T08:03:34.572238+0000","last_peered":"2024-09-18T08:03:34.572238+0000","last_clean":"2024-09-18T08:03:34.572238+0000","last_became_active":"2024-09-18T08:02:54.048959+0000","last_became_peered":"2024-09-18T08:02:54.048959+0000","last_unstale":"2024-09-18T08:03:34.572238+0000","last_undegraded":"2024-09-18T08:03:34.572238+0000","last_fullsized":"2024-09-18T08:03:34.572238+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_clean_scrub_stamp":"2024-09-18T08:02:52.890475+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:03:56.721996+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"69'9","reported_seq":56,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-18T08:03:34.047766+0000","last_change":"2024-09-18T08:02:49.892390+0000","last_active":"2024-09-18T08:03:34.047766+0000","last_peered":"2024-09-18T08:03:34.047766+0000","last_clean":"2024-09-18T08:03:34.047766+0000","last_became_active":"2024-09-18T08:02:49.891567+0000","last_became_peered":"2024-09-18T08:02:49.891567+0000","last_unstale":"2024-09-18T08:03:34.047766+0000","last_undegraded":"2024-09-18T08:03:34.047766+0000","last_fullsized":"2024-09-18T08:03:34.047766+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_clean_scrub_stamp":"2024-09-18T08:02:48.870718+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:03:51.625809+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":44,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-18T08:03:34.572817+0000","last_change":"2024-09-18T08:02:47.886233+0000","last_active":"2024-09-18T08:03:34.572817+0000","last_peered":"2024-09-18T08:03:34.572817+0000","last_clean":"2024-09-18T08:03:34.572817+0000","last_became_active":"2024-09-18T08:02:47.885790+0000","last_became_peered":"2024-09-18T08:02:47.885790+0000","last_unstale":"2024-09-18T08:03:34.572817+0000","last_undegraded":"2024-09-18T08:03:34.572817+0000","last_fullsized":"2024-09-18T08:03:34.572817+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:02:46.860092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:46.860092+0000","last_clean_scrub_stamp":"2024-09-18T08:02:46.860092+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:03:48.811474+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":36,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-18T08:03:34.872672+0000","last_change":"2024-09-18T08:02:51.908640+0000","last_active":"2024-09-18T08:03:34.872672+0000","last_peered":"2024-09-18T08:03:34.872672+0000","last_clean":"2024-09-18T08:03:34.872672+0000","last_became_active":"2024-09-18T08:02:51.907361+0000","last_became_peered":"2024-09-18T08:02:51.907361+0000","last_unstale":"2024-09-18T08:03:34.872672+0000","last_undegraded":"2024-09-18T08:03:34.872672+0000","last_fullsized":"2024-09-18T08:03:34.872672+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:02:50.886830+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:50.886830+0000","last_clean_scrub_stamp":"2024-09-18T08:02:50.886830+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:04:06.357747+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":33,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-18T08:03:34.048964+0000","last_change":"2024-09-18T08:02:53.909246+0000","last_active":"2024-09-18T08:03:34.048964+0000","last_peered":"2024-09-18T08:03:34.048964+0000","last_clean":"2024-09-18T08:03:34.048964+0000","last_became_active":"2024-09-18T08:02:53.908917+0000","last_became_peered":"2024-09-18T08:02:53.908917+0000","last_unstale":"2024-09-18T08:03:34.048964+0000","last_undegraded":"2024-09-18T08:03:34.048964+0000","last_fullsized":"2024-09-18T08:03:34.048964+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_clean_scrub_stamp":"2024-09-18T08:02:52.890475+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:03:57.367069+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.15","version":"69'9","reported_seq":56,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-18T08:03:34.762707+0000","last_change":"2024-09-18T08:02:49.890846+0000","last_active":"2024-09-18T08:03:34.762707+0000","last_peered":"2024-09-18T08:03:34.762707+0000","last_clean":"2024-09-18T08:03:34.762707+0000","last_became_active":"2024-09-18T08:02:49.890497+0000","last_became_peered":"2024-09-18T08:02:49.890497+0000","last_unstale":"2024-09-18T08:03:34.762707+0000","last_undegraded":"2024-09-18T08:03:34.762707+0000","last_fullsized":"2024-09-18T08:03:34.762707+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_clean_scrub_stamp":"2024-09-18T08:02:48.870718+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:04:02.479848+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":44,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-18T08:03:34.046975+0000","last_change":"2024-09-18T08:02:47.877632+0000","last_active":"2024-09-18T08:03:34.046975+0000","last_peered":"2024-09-18T08:03:34.046975+0000","last_clean":"2024-09-18T08:03:34.046975+0000","last_became_active":"2024-09-18T08:02:47.875983+0000","last_became_peered":"2024-09-18T08:02:47.875983+0000","last_unstale":"2024-09-18T08:03:34.046975+0000","last_undegraded":"2024-09-18T08:03:34.046975+0000","last_fullsized":"2024-09-18T08:03:34.046975+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:02:46.860092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:46.860092+0000","last_clean_scrub_stamp":"2024-09-18T08:02:46.860092+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:04:09.677911+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.14","version":"69'8","reported_seq":56,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-18T08:03:37.938707+0000","last_change":"2024-09-18T08:02:51.909896+0000","last_active":"2024-09-18T08:03:37.938707+0000","last_peered":"2024-09-18T08:03:37.938707+0000","last_clean":"2024-09-18T08:03:37.938707+0000","last_became_active":"2024-09-18T08:02:51.909513+0000","last_became_peered":"2024-09-18T08:02:51.909513+0000","last_unstale":"2024-09-18T08:03:37.938707+0000","last_undegraded":"2024-09-18T08:03:37.938707+0000","last_fullsized":"2024-09-18T08:03:37.938707+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:02:50.886830+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:50.886830+0000","last_clean_scrub_stamp":"2024-09-18T08:02:50.886830+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:03:54.167632+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":33,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-18T08:03:34.048706+0000","last_change":"2024-09-18T08:02:53.907729+0000","last_active":"2024-09-18T08:03:34.048706+0000","last_peered":"2024-09-18T08:03:34.048706+0000","last_clean":"2024-09-18T08:03:34.048706+0000","last_became_active":"2024-09-18T08:02:53.907428+0000","last_became_peered":"2024-09-18T08:02:53.907428+0000","last_unstale":"2024-09-18T08:03:34.048706+0000","last_undegraded":"2024-09-18T08:03:34.048706+0000","last_fullsized":"2024-09-18T08:03:34.048706+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_clean_scrub_stamp":"2024-09-18T08:02:52.890475+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:04:00.922586+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.14","version":"69'10","reported_seq":55,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-18T08:03:34.872329+0000","last_change":"2024-09-18T08:02:49.889483+0000","last_active":"2024-09-18T08:03:34.872329+0000","last_peered":"2024-09-18T08:03:34.872329+0000","last_clean":"2024-09-18T08:03:34.872329+0000","last_became_active":"2024-09-18T08:02:49.889191+0000","last_became_peered":"2024-09-18T08:02:49.889191+0000","last_unstale":"2024-09-18T08:03:34.872329+0000","last_undegraded":"2024-09-18T08:03:34.872329+0000","last_fullsized":"2024-09-18T08:03:34.872329+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_clean_scrub_stamp":"2024-09-18T08:02:48.870718+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:03:58.029274+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":44,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-18T08:03:34.573032+0000","last_change":"2024-09-18T08:02:47.882542+0000","last_active":"2024-09-18T08:03:34.573032+0000","last_peered":"2024-09-18T08:03:34.573032+0000","last_clean":"2024-09-18T08:03:34.573032+0000","last_became_active":"2024-09-18T08:02:47.882065+0000","last_became_peered":"2024-09-18T08:02:47.882065+0000","last_unstale":"2024-09-18T08:03:34.573032+0000","last_undegraded":"2024-09-18T08:03:34.573032+0000","last_fullsized":"2024-09-18T08:03:34.573032+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:02:46.860092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:46.860092+0000","last_clean_scrub_stamp":"2024-09-18T08:02:46.860092+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:03:59.012126+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.15","version":"69'8","reported_seq":56,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-18T08:03:37.938581+0000","last_change":"2024-09-18T08:02:51.914448+0000","last_active":"2024-09-18T08:03:37.938581+0000","last_peered":"2024-09-18T08:03:37.938581+0000","last_clean":"2024-09-18T08:03:37.938581+0000","last_became_active":"2024-09-18T08:02:51.914135+0000","last_became_peered":"2024-09-18T08:02:51.914135+0000","last_unstale":"2024-09-18T08:03:37.938581+0000","last_undegraded":"2024-09-18T08:03:37.938581+0000","last_fullsized":"2024-09-18T08:03:37.938581+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:02:50.886830+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:50.886830+0000","last_clean_scrub_stamp":"2024-09-18T08:02:50.886830+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:03:56.893696+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":33,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-18T08:03:34.047461+0000","last_change":"2024-09-18T08:02:53.912409+0000","last_active":"2024-09-18T08:03:34.047461+0000","last_peered":"2024-09-18T08:03:34.047461+0000","last_clean":"2024-09-18T08:03:34.047461+0000","last_became_active":"2024-09-18T08:02:53.912112+0000","last_became_peered":"2024-09-18T08:02:53.912112+0000","last_unstale":"2024-09-18T08:03:34.047461+0000","last_undegraded":"2024-09-18T08:03:34.047461+0000","last_fullsized":"2024-09-18T08:03:34.047461+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_clean_scrub_stamp":"2024-09-18T08:02:52.890475+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:03:54.310553+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.13","version":"69'11","reported_seq":59,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-18T08:03:34.047990+0000","last_change":"2024-09-18T08:02:49.897433+0000","last_active":"2024-09-18T08:03:34.047990+0000","last_peered":"2024-09-18T08:03:34.047990+0000","last_clean":"2024-09-18T08:03:34.047990+0000","last_became_active":"2024-09-18T08:02:49.896949+0000","last_became_peered":"2024-09-18T08:02:49.896949+0000","last_unstale":"2024-09-18T08:03:34.047990+0000","last_undegraded":"2024-09-18T08:03:34.047990+0000","last_fullsized":"2024-09-18T08:03:34.047990+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_clean_scrub_stamp":"2024-09-18T08:02:48.870718+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:04:08.564349+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":44,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-18T08:03:34.048395+0000","last_change":"2024-09-18T08:02:47.880202+0000","last_active":"2024-09-18T08:03:34.048395+0000","last_peered":"2024-09-18T08:03:34.048395+0000","last_clean":"2024-09-18T08:03:34.048395+0000","last_became_active":"2024-09-18T08:02:47.879771+0000","last_became_peered":"2024-09-18T08:02:47.879771+0000","last_unstale":"2024-09-18T08:03:34.048395+0000","last_undegraded":"2024-09-18T08:03:34.048395+0000","last_fullsized":"2024-09-18T08:03:34.048395+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:02:46.860092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:46.860092+0000","last_clean_scrub_stamp":"2024-09-18T08:02:46.860092+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:03:58.640926+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":36,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-18T08:03:35.231883+0000","last_change":"2024-09-18T08:02:51.912062+0000","last_active":"2024-09-18T08:03:35.231883+0000","last_peered":"2024-09-18T08:03:35.231883+0000","last_clean":"2024-09-18T08:03:35.231883+0000","last_became_active":"2024-09-18T08:02:51.910752+0000","last_became_peered":"2024-09-18T08:02:51.910752+0000","last_unstale":"2024-09-18T08:03:35.231883+0000","last_undegraded":"2024-09-18T08:03:35.231883+0000","last_fullsized":"2024-09-18T08:03:35.231883+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:02:50.886830+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:50.886830+0000","last_clean_scrub_stamp":"2024-09-18T08:02:50.886830+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:04:01.465747+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":33,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-18T08:03:34.871644+0000","last_change":"2024-09-18T08:02:53.911861+0000","last_active":"2024-09-18T08:03:34.871644+0000","last_peered":"2024-09-18T08:03:34.871644+0000","last_clean":"2024-09-18T08:03:34.871644+0000","last_became_active":"2024-09-18T08:02:53.911429+0000","last_became_peered":"2024-09-18T08:02:53.911429+0000","last_unstale":"2024-09-18T08:03:34.871644+0000","last_undegraded":"2024-09-18T08:03:34.871644+0000","last_fullsized":"2024-09-18T08:03:34.871644+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_clean_scrub_stamp":"2024-09-18T08:02:52.890475+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:03:58.350802+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.12","version":"69'9","reported_seq":56,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-18T08:03:35.232068+0000","last_change":"2024-09-18T08:02:49.893472+0000","last_active":"2024-09-18T08:03:35.232068+0000","last_peered":"2024-09-18T08:03:35.232068+0000","last_clean":"2024-09-18T08:03:35.232068+0000","last_became_active":"2024-09-18T08:02:49.885950+0000","last_became_peered":"2024-09-18T08:02:49.885950+0000","last_unstale":"2024-09-18T08:03:35.232068+0000","last_undegraded":"2024-09-18T08:03:35.232068+0000","last_fullsized":"2024-09-18T08:03:35.232068+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_clean_scrub_stamp":"2024-09-18T08:02:48.870718+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:03:57.261106+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":44,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-18T08:03:34.573125+0000","last_change":"2024-09-18T08:02:47.883002+0000","last_active":"2024-09-18T08:03:34.573125+0000","last_peered":"2024-09-18T08:03:34.573125+0000","last_clean":"2024-09-18T08:03:34.573125+0000","last_became_active":"2024-09-18T08:02:47.882096+0000","last_became_peered":"2024-09-18T08:02:47.882096+0000","last_unstale":"2024-09-18T08:03:34.573125+0000","last_undegraded":"2024-09-18T08:03:34.573125+0000","last_fullsized":"2024-09-18T08:03:34.573125+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:02:46.860092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:46.860092+0000","last_clean_scrub_stamp":"2024-09-18T08:02:46.860092+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:04:10.794338+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":36,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-18T08:03:34.872150+0000","last_change":"2024-09-18T08:02:51.906594+0000","last_active":"2024-09-18T08:03:34.872150+0000","last_peered":"2024-09-18T08:03:34.872150+0000","last_clean":"2024-09-18T08:03:34.872150+0000","last_became_active":"2024-09-18T08:02:51.906277+0000","last_became_peered":"2024-09-18T08:02:51.906277+0000","last_unstale":"2024-09-18T08:03:34.872150+0000","last_undegraded":"2024-09-18T08:03:34.872150+0000","last_fullsized":"2024-09-18T08:03:34.872150+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:02:50.886830+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:50.886830+0000","last_clean_scrub_stamp":"2024-09-18T08:02:50.886830+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:04:07.327398+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.10","version":"69'1","reported_seq":34,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-18T08:03:34.046464+0000","last_change":"2024-09-18T08:02:53.913013+0000","last_active":"2024-09-18T08:03:34.046464+0000","last_peered":"2024-09-18T08:03:34.046464+0000","last_clean":"2024-09-18T08:03:34.046464+0000","last_became_active":"2024-09-18T08:02:53.912699+0000","last_became_peered":"2024-09-18T08:02:53.912699+0000","last_unstale":"2024-09-18T08:03:34.046464+0000","last_undegraded":"2024-09-18T08:03:34.046464+0000","last_fullsized":"2024-09-18T08:03:34.046464+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_clean_scrub_stamp":"2024-09-18T08:02:52.890475+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:04:15.154915+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.11","version":"69'11","reported_seq":59,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-18T08:03:34.871764+0000","last_change":"2024-09-18T08:02:49.889895+0000","last_active":"2024-09-18T08:03:34.871764+0000","last_peered":"2024-09-18T08:03:34.871764+0000","last_clean":"2024-09-18T08:03:34.871764+0000","last_became_active":"2024-09-18T08:02:49.889611+0000","last_became_peered":"2024-09-18T08:02:49.889611+0000","last_unstale":"2024-09-18T08:03:34.871764+0000","last_undegraded":"2024-09-18T08:03:34.871764+0000","last_fullsized":"2024-09-18T08:03:34.871764+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_clean_scrub_stamp":"2024-09-18T08:02:48.870718+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:03:51.062461+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":44,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-18T08:03:34.762615+0000","last_change":"2024-09-18T08:02:47.879259+0000","last_active":"2024-09-18T08:03:34.762615+0000","last_peered":"2024-09-18T08:03:34.762615+0000","last_clean":"2024-09-18T08:03:34.762615+0000","last_became_active":"2024-09-18T08:02:47.878886+0000","last_became_peered":"2024-09-18T08:02:47.878886+0000","last_unstale":"2024-09-18T08:03:34.762615+0000","last_undegraded":"2024-09-18T08:03:34.762615+0000","last_fullsized":"2024-09-18T08:03:34.762615+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:02:46.860092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:46.860092+0000","last_clean_scrub_stamp":"2024-09-18T08:02:46.860092+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:03:47.500112+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":36,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-18T08:03:34.572309+0000","last_change":"2024-09-18T08:02:51.908058+0000","last_active":"2024-09-18T08:03:34.572309+0000","last_peered":"2024-09-18T08:03:34.572309+0000","last_clean":"2024-09-18T08:03:34.572309+0000","last_became_active":"2024-09-18T08:02:51.907762+0000","last_became_peered":"2024-09-18T08:02:51.907762+0000","last_unstale":"2024-09-18T08:03:34.572309+0000","last_undegraded":"2024-09-18T08:03:34.572309+0000","last_fullsized":"2024-09-18T08:03:34.572309+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:02:50.886830+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:50.886830+0000","last_clean_scrub_stamp":"2024-09-18T08:02:50.886830+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:04:01.256104+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":32,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-18T08:03:34.871461+0000","last_change":"2024-09-18T08:02:54.047687+0000","last_active":"2024-09-18T08:03:34.871461+0000","last_peered":"2024-09-18T08:03:34.871461+0000","last_clean":"2024-09-18T08:03:34.871461+0000","last_became_active":"2024-09-18T08:02:54.047181+0000","last_became_peered":"2024-09-18T08:02:54.047181+0000","last_unstale":"2024-09-18T08:03:34.871461+0000","last_undegraded":"2024-09-18T08:03:34.871461+0000","last_fullsized":"2024-09-18T08:03:34.871461+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_clean_scrub_stamp":"2024-09-18T08:02:52.890475+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:04:12.998775+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.10","version":"69'4","reported_seq":46,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-18T08:03:34.871632+0000","last_change":"2024-09-18T08:02:49.890435+0000","last_active":"2024-09-18T08:03:34.871632+0000","last_peered":"2024-09-18T08:03:34.871632+0000","last_clean":"2024-09-18T08:03:34.871632+0000","last_became_active":"2024-09-18T08:02:49.890126+0000","last_became_peered":"2024-09-18T08:02:49.890126+0000","last_unstale":"2024-09-18T08:03:34.871632+0000","last_undegraded":"2024-09-18T08:03:34.871632+0000","last_fullsized":"2024-09-18T08:03:34.871632+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_clean_scrub_stamp":"2024-09-18T08:02:48.870718+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:03:56.821965+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":6,"num_read_kb":4,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":44,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-18T08:03:34.047540+0000","last_change":"2024-09-18T08:02:47.877991+0000","last_active":"2024-09-18T08:03:34.047540+0000","last_peered":"2024-09-18T08:03:34.047540+0000","last_clean":"2024-09-18T08:03:34.047540+0000","last_became_active":"2024-09-18T08:02:47.877616+0000","last_became_peered":"2024-09-18T08:02:47.877616+0000","last_unstale":"2024-09-18T08:03:34.047540+0000","last_undegraded":"2024-09-18T08:03:34.047540+0000","last_fullsized":"2024-09-18T08:03:34.047540+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:02:46.860092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:46.860092+0000","last_clean_scrub_stamp":"2024-09-18T08:02:46.860092+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:04:04.436191+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":36,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-18T08:03:34.497035+0000","last_change":"2024-09-18T08:02:51.906818+0000","last_active":"2024-09-18T08:03:34.497035+0000","last_peered":"2024-09-18T08:03:34.497035+0000","last_clean":"2024-09-18T08:03:34.497035+0000","last_became_active":"2024-09-18T08:02:51.906372+0000","last_became_peered":"2024-09-18T08:02:51.906372+0000","last_unstale":"2024-09-18T08:03:34.497035+0000","last_undegraded":"2024-09-18T08:03:34.497035+0000","last_fullsized":"2024-09-18T08:03:34.497035+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:02:50.886830+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:50.886830+0000","last_clean_scrub_stamp":"2024-09-18T08:02:50.886830+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:03:51.034977+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":33,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-18T08:03:34.572563+0000","last_change":"2024-09-18T08:02:53.909445+0000","last_active":"2024-09-18T08:03:34.572563+0000","last_peered":"2024-09-18T08:03:34.572563+0000","last_clean":"2024-09-18T08:03:34.572563+0000","last_became_active":"2024-09-18T08:02:53.909146+0000","last_became_peered":"2024-09-18T08:02:53.909146+0000","last_unstale":"2024-09-18T08:03:34.572563+0000","last_undegraded":"2024-09-18T08:03:34.572563+0000","last_fullsized":"2024-09-18T08:03:34.572563+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_clean_scrub_stamp":"2024-09-18T08:02:52.890475+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:04:03.033337+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,4],"acting":[7,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":33,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-18T08:03:35.232234+0000","last_change":"2024-09-18T08:02:53.907055+0000","last_active":"2024-09-18T08:03:35.232234+0000","last_peered":"2024-09-18T08:03:35.232234+0000","last_clean":"2024-09-18T08:03:35.232234+0000","last_became_active":"2024-09-18T08:02:53.906726+0000","last_became_peered":"2024-09-18T08:02:53.906726+0000","last_unstale":"2024-09-18T08:03:35.232234+0000","last_undegraded":"2024-09-18T08:03:35.232234+0000","last_fullsized":"2024-09-18T08:03:35.232234+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_clean_scrub_stamp":"2024-09-18T08:02:52.890475+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:03:58.628509+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":44,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-18T08:03:34.871813+0000","last_change":"2024-09-18T08:02:47.879447+0000","last_active":"2024-09-18T08:03:34.871813+0000","last_peered":"2024-09-18T08:03:34.871813+0000","last_clean":"2024-09-18T08:03:34.871813+0000","last_became_active":"2024-09-18T08:02:47.878922+0000","last_became_peered":"2024-09-18T08:02:47.878922+0000","last_unstale":"2024-09-18T08:03:34.871813+0000","last_undegraded":"2024-09-18T08:03:34.871813+0000","last_fullsized":"2024-09-18T08:03:34.871813+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:02:46.860092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:46.860092+0000","last_clean_scrub_stamp":"2024-09-18T08:02:46.860092+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:03:54.771042+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"69'11","reported_seq":59,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-18T08:03:34.497817+0000","last_change":"2024-09-18T08:02:49.898630+0000","last_active":"2024-09-18T08:03:34.497817+0000","last_peered":"2024-09-18T08:03:34.497817+0000","last_clean":"2024-09-18T08:03:34.497817+0000","last_became_active":"2024-09-18T08:02:49.898048+0000","last_became_peered":"2024-09-18T08:02:49.898048+0000","last_unstale":"2024-09-18T08:03:34.497817+0000","last_undegraded":"2024-09-18T08:03:34.497817+0000","last_fullsized":"2024-09-18T08:03:34.497817+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_clean_scrub_stamp":"2024-09-18T08:02:48.870718+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:04:05.049957+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":36,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-18T08:03:34.047607+0000","last_change":"2024-09-18T08:02:51.900666+0000","last_active":"2024-09-18T08:03:34.047607+0000","last_peered":"2024-09-18T08:03:34.047607+0000","last_clean":"2024-09-18T08:03:34.047607+0000","last_became_active":"2024-09-18T08:02:51.900300+0000","last_became_peered":"2024-09-18T08:02:51.900300+0000","last_unstale":"2024-09-18T08:03:34.047607+0000","last_undegraded":"2024-09-18T08:03:34.047607+0000","last_fullsized":"2024-09-18T08:03:34.047607+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:02:50.886830+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:50.886830+0000","last_clean_scrub_stamp":"2024-09-18T08:02:50.886830+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:04:12.807594+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":416,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":64,"ondisk_log_size":64,"up":96,"acting":96,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":178,"num_object_clones":0,"num_object_copies":534,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":178,"num_whiteouts":0,"num_read":698,"num_read_kb":455,"num_write":417,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":417792,"data_stored":11106,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":393,"ondisk_log_size":393,"up":96,"acting":96,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":1613,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":24,"num_read_kb":24,"num_write":10,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":4839,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":6,"ondisk_log_size":6,"up":96,"acting":96,"num_store_stats":8},{"poolid":2,"num_pg":3,"stat_sum":{"num_bytes":408,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":50,"num_read_kb":45,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":8,"ondisk_log_size":8,"up":9,"acting":9,"num_store_stats":7},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":231,"num_write":113,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":590368,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":79,"ondisk_log_size":79,"up":3,"acting":3,"num_store_stats":3}],"osd_stats":[{"osd":7,"up_from":56,"seq":240518168591,"num_pgs":53,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28376,"kb_used_data":1348,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716776,"statfs":{"total":95995035648,"available":95965978624,"internally_reserved":0,"allocated":1380352,"data_stored":906838,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":47,"seq":201863462933,"num_pgs":43,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28356,"kb_used_data":1328,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716796,"statfs":{"total":95995035648,"available":95965999104,"internally_reserved":0,"allocated":1359872,"data_stored":905753,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":39,"seq":167503724571,"num_pgs":47,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27788,"kb_used_data":760,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717364,"statfs":{"total":95995035648,"available":95966580736,"internally_reserved":0,"allocated":778240,"data_stored":316193,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":32,"seq":137438953504,"num_pgs":58,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27808,"kb_used_data":780,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717344,"statfs":{"total":95995035648,"available":95966560256,"internally_reserved":0,"allocated":798720,"data_stored":316983,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":26,"seq":111669149735,"num_pgs":56,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27812,"kb_used_data":784,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717340,"statfs":{"total":95995035648,"available":95966556160,"internally_reserved":0,"allocated":802816,"data_stored":316276,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":27654600},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411373,"num_pgs":36,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":31936,"kb_used_data":748,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713216,"statfs":{"total":95995035648,"available":95962333184,"internally_reserved":0,"allocated":765952,"data_stored":315969,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574897,"num_pgs":57,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27852,"kb_used_data":824,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717300,"statfs":{"total":95995035648,"available":95966515200,"internally_reserved":0,"allocated":843776,"data_stored":318612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738424,"num_pgs":46,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28388,"kb_used_data":1360,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716764,"statfs":{"total":95995035648,"available":95965966336,"internally_reserved":0,"allocated":1392640,"data_stored":907553,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1131,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1521,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":436,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":46,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1177,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":436,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":90112,"data_stored":2338,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":798,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":1898,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1474,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":990,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1034,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1254,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":403,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":416,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":403,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T08:03:46.659 INFO:tasks.cephadm.ceph_manager.ceph:clean! 2024-09-18T08:03:46.659 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid 779349ca-7593-11ef-bceb-c7b262605968 -- ceph osd dump 2024-09-18T08:03:46.660 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:03:46 smithi050 ceph-mon[32371]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-09-18T08:03:46.660 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:03:46 smithi050 ceph-mon[32371]: Deploying daemon grafana.a on smithi050 2024-09-18T08:03:46.660 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:03:46 smithi050 ceph-mon[32371]: pgmap v22: 132 pgs: 132 active+clean; 583 KiB data, 223 MiB used, 715 GiB / 715 GiB avail; 928 B/s rd, 0 op/s 2024-09-18T08:03:46.660 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:03:46 smithi050 ceph-mon[32371]: from='mgr.24470 172.21.15.7:0/2594390261' entity='mgr.y' cmd={"prefix": "orch prometheus get-credentials"} : dispatch 2024-09-18T08:03:46.660 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:03:46 smithi050 ceph-mon[32371]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "orch prometheus get-credentials"}]: dispatch 2024-09-18T08:03:46.942 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/779349ca-7593-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T08:03:46.980 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Sep 18 08:03:46 smithi007 ceph-779349ca-7593-11ef-bceb-c7b262605968-mgr-y[29688]: ::ffff:172.21.15.50 - - [18/Sep/2024:08:03:46] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-09-18T08:03:47.322 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:03:47 smithi007 ceph-mon[29459]: from='client.14742 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-18T08:03:47.322 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:03:47 smithi007 ceph-mon[39249]: from='client.14742 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-18T08:03:47.604 INFO:teuthology.orchestra.run.smithi007.stdout:epoch 79 2024-09-18T08:03:47.604 INFO:teuthology.orchestra.run.smithi007.stdout:fsid 779349ca-7593-11ef-bceb-c7b262605968 2024-09-18T08:03:47.604 INFO:teuthology.orchestra.run.smithi007.stdout:created 2024-09-18T07:57:09.295710+0000 2024-09-18T08:03:47.604 INFO:teuthology.orchestra.run.smithi007.stdout:modified 2024-09-18T08:03:47.207784+0000 2024-09-18T08:03:47.604 INFO:teuthology.orchestra.run.smithi007.stdout:flags sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit 2024-09-18T08:03:47.604 INFO:teuthology.orchestra.run.smithi007.stdout:crush_version 18 2024-09-18T08:03:47.604 INFO:teuthology.orchestra.run.smithi007.stdout:full_ratio 0.95 2024-09-18T08:03:47.604 INFO:teuthology.orchestra.run.smithi007.stdout:backfillfull_ratio 0.9 2024-09-18T08:03:47.604 INFO:teuthology.orchestra.run.smithi007.stdout:nearfull_ratio 0.85 2024-09-18T08:03:47.604 INFO:teuthology.orchestra.run.smithi007.stdout:require_min_compat_client luminous 2024-09-18T08:03:47.605 INFO:teuthology.orchestra.run.smithi007.stdout:min_compat_client jewel 2024-09-18T08:03:47.605 INFO:teuthology.orchestra.run.smithi007.stdout:require_osd_release squid 2024-09-18T08:03:47.605 INFO:teuthology.orchestra.run.smithi007.stdout:stretch_mode_enabled false 2024-09-18T08:03:47.605 INFO:teuthology.orchestra.run.smithi007.stdout:pool 1 '.mgr' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 1 pgp_num 1 autoscale_mode off last_change 21 flags hashpspool stripe_width 0 pg_num_max 32 pg_num_min 1 application mgr read_balance_score 7.89 2024-09-18T08:03:47.605 INFO:teuthology.orchestra.run.smithi007.stdout:pool 2 'datapool' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 3 pgp_num 3 autoscale_mode off last_change 64 flags hashpspool,selfmanaged_snaps stripe_width 0 application rbd read_balance_score 2.65 2024-09-18T08:03:47.605 INFO:teuthology.orchestra.run.smithi007.stdout:pool 3 '.rgw.root' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 32 pgp_num 32 autoscale_mode off last_change 63 flags hashpspool stripe_width 0 application rgw read_balance_score 1.50 2024-09-18T08:03:47.605 INFO:teuthology.orchestra.run.smithi007.stdout:pool 4 'default.rgw.log' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 32 pgp_num 32 autoscale_mode off last_change 65 flags hashpspool stripe_width 0 application rgw read_balance_score 2.25 2024-09-18T08:03:47.605 INFO:teuthology.orchestra.run.smithi007.stdout:pool 5 'default.rgw.control' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 32 pgp_num 32 autoscale_mode off last_change 67 flags hashpspool stripe_width 0 application rgw read_balance_score 1.25 2024-09-18T08:03:47.605 INFO:teuthology.orchestra.run.smithi007.stdout:pool 6 'default.rgw.meta' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 32 pgp_num 32 autoscale_mode off last_change 70 flags hashpspool stripe_width 0 pg_autoscale_bias 4 application rgw read_balance_score 1.75 2024-09-18T08:03:47.605 INFO:teuthology.orchestra.run.smithi007.stdout:max_osd 8 2024-09-18T08:03:47.605 INFO:teuthology.orchestra.run.smithi007.stdout:osd.0 up in weight 1 up_from 8 up_thru 67 down_at 0 last_clean_interval [0,0) v2:172.21.15.7:6801/2896425794 v2:172.21.15.7:6802/2896425794 exists,up ae8f60df-72ac-46fb-91a1-f132b851c876 2024-09-18T08:03:47.605 INFO:teuthology.orchestra.run.smithi007.stdout:osd.1 up in weight 1 up_from 13 up_thru 67 down_at 0 last_clean_interval [0,0) v2:172.21.15.7:6805/1971397549 v2:172.21.15.7:6806/1971397549 exists,up be28b70c-1f73-494f-ad4b-cb7ec5311999 2024-09-18T08:03:47.605 INFO:teuthology.orchestra.run.smithi007.stdout:osd.2 up in weight 1 up_from 18 up_thru 67 down_at 0 last_clean_interval [0,0) v2:172.21.15.7:6809/556142559 v2:172.21.15.7:6810/556142559 exists,up cfa4f083-c7fd-4743-be65-84a3040fe6dc 2024-09-18T08:03:47.606 INFO:teuthology.orchestra.run.smithi007.stdout:osd.3 up in weight 1 up_from 26 up_thru 67 down_at 0 last_clean_interval [0,0) v2:172.21.15.7:6813/163587772 v2:172.21.15.7:6814/163587772 exists,up 11ecd7e1-a751-4b75-95f5-00b32d2a5bc4 2024-09-18T08:03:47.606 INFO:teuthology.orchestra.run.smithi007.stdout:osd.4 up in weight 1 up_from 32 up_thru 67 down_at 0 last_clean_interval [0,0) v2:172.21.15.50:6800/3662119805 v2:172.21.15.50:6801/3662119805 exists,up 98992b34-cfa1-45bf-97a1-7d557739252f 2024-09-18T08:03:47.606 INFO:teuthology.orchestra.run.smithi007.stdout:osd.5 up in weight 1 up_from 39 up_thru 67 down_at 0 last_clean_interval [0,0) v2:172.21.15.50:6804/1199797954 v2:172.21.15.50:6805/1199797954 exists,up 7f290005-ebca-467b-87e4-02ead1063610 2024-09-18T08:03:47.606 INFO:teuthology.orchestra.run.smithi007.stdout:osd.6 up in weight 1 up_from 47 up_thru 65 down_at 0 last_clean_interval [0,0) v2:172.21.15.50:6808/255960422 v2:172.21.15.50:6809/255960422 exists,up e07d846c-ae42-4fea-aa5c-3a25835d18a6 2024-09-18T08:03:47.606 INFO:teuthology.orchestra.run.smithi007.stdout:osd.7 up in weight 1 up_from 56 up_thru 67 down_at 0 last_clean_interval [0,0) v2:172.21.15.50:6812/874923904 v2:172.21.15.50:6813/874923904 exists,up c9dac3b7-d651-489d-b96e-5b10fa32d9af 2024-09-18T08:03:47.606 INFO:teuthology.orchestra.run.smithi007.stdout:blocklist 172.21.15.7:0/398367658 expires 2024-09-19T08:03:17.052903+0000 2024-09-18T08:03:47.606 INFO:teuthology.orchestra.run.smithi007.stdout:blocklist 172.21.15.7:6800/2918496291 expires 2024-09-19T07:57:31.015269+0000 2024-09-18T08:03:47.606 INFO:teuthology.orchestra.run.smithi007.stdout:blocklist 172.21.15.7:0/2514354773 expires 2024-09-19T07:57:31.015269+0000 2024-09-18T08:03:47.606 INFO:teuthology.orchestra.run.smithi007.stdout:blocklist 172.21.15.7:0/3924911335 expires 2024-09-19T07:57:31.015269+0000 2024-09-18T08:03:47.606 INFO:teuthology.orchestra.run.smithi007.stdout:blocklist 172.21.15.7:0/1668598084 expires 2024-09-19T07:57:57.346149+0000 2024-09-18T08:03:47.606 INFO:teuthology.orchestra.run.smithi007.stdout:blocklist 172.21.15.7:0/4272027292 expires 2024-09-19T07:57:31.015269+0000 2024-09-18T08:03:47.606 INFO:teuthology.orchestra.run.smithi007.stdout:blocklist 172.21.15.7:6800/1013061071 expires 2024-09-19T07:57:57.346149+0000 2024-09-18T08:03:47.607 INFO:teuthology.orchestra.run.smithi007.stdout:blocklist 172.21.15.7:6800/1491061076 expires 2024-09-19T08:03:17.052903+0000 2024-09-18T08:03:47.607 INFO:teuthology.orchestra.run.smithi007.stdout:blocklist 172.21.15.7:0/533727535 expires 2024-09-19T07:57:57.346149+0000 2024-09-18T08:03:47.607 INFO:teuthology.orchestra.run.smithi007.stdout:blocklist 172.21.15.7:0/1605217610 expires 2024-09-19T08:03:17.052903+0000 2024-09-18T08:03:47.607 INFO:teuthology.orchestra.run.smithi007.stdout:blocklist 172.21.15.7:0/1722968065 expires 2024-09-19T08:03:17.052903+0000 2024-09-18T08:03:47.607 INFO:teuthology.orchestra.run.smithi007.stdout:blocklist 172.21.15.7:0/1277549846 expires 2024-09-19T07:57:57.346149+0000 2024-09-18T08:03:47.607 INFO:teuthology.orchestra.run.smithi007.stdout:blocklist 172.21.15.7:0/232452890 expires 2024-09-19T08:03:17.052903+0000 2024-09-18T08:03:47.607 INFO:teuthology.orchestra.run.smithi007.stdout:blocklist 172.21.15.7:0/3881670470 expires 2024-09-19T08:03:17.052903+0000 2024-09-18T08:03:47.656 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:03:47 smithi050 ceph-mon[32371]: from='client.14742 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-18T08:03:48.387 INFO:tasks.cephadm.ceph_manager.ceph:['osd.0 up in weight 1 up_from 8 up_thru 67 down_at 0 last_clean_interval [0,0) v2:172.21.15.7:6801/2896425794 v2:172.21.15.7:6802/2896425794 exists,up ae8f60df-72ac-46fb-91a1-f132b851c876', 'osd.1 up in weight 1 up_from 13 up_thru 67 down_at 0 last_clean_interval [0,0) v2:172.21.15.7:6805/1971397549 v2:172.21.15.7:6806/1971397549 exists,up be28b70c-1f73-494f-ad4b-cb7ec5311999', 'osd.2 up in weight 1 up_from 18 up_thru 67 down_at 0 last_clean_interval [0,0) v2:172.21.15.7:6809/556142559 v2:172.21.15.7:6810/556142559 exists,up cfa4f083-c7fd-4743-be65-84a3040fe6dc', 'osd.3 up in weight 1 up_from 26 up_thru 67 down_at 0 last_clean_interval [0,0) v2:172.21.15.7:6813/163587772 v2:172.21.15.7:6814/163587772 exists,up 11ecd7e1-a751-4b75-95f5-00b32d2a5bc4', 'osd.4 up in weight 1 up_from 32 up_thru 67 down_at 0 last_clean_interval [0,0) v2:172.21.15.50:6800/3662119805 v2:172.21.15.50:6801/3662119805 exists,up 98992b34-cfa1-45bf-97a1-7d557739252f', 'osd.5 up in weight 1 up_from 39 up_thru 67 down_at 0 last_clean_interval [0,0) v2:172.21.15.50:6804/1199797954 v2:172.21.15.50:6805/1199797954 exists,up 7f290005-ebca-467b-87e4-02ead1063610', 'osd.6 up in weight 1 up_from 47 up_thru 65 down_at 0 last_clean_interval [0,0) v2:172.21.15.50:6808/255960422 v2:172.21.15.50:6809/255960422 exists,up e07d846c-ae42-4fea-aa5c-3a25835d18a6', 'osd.7 up in weight 1 up_from 56 up_thru 67 down_at 0 last_clean_interval [0,0) v2:172.21.15.50:6812/874923904 v2:172.21.15.50:6813/874923904 exists,up c9dac3b7-d651-489d-b96e-5b10fa32d9af'] 2024-09-18T08:03:48.388 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid 779349ca-7593-11ef-bceb-c7b262605968 -- ceph daemon mon.a config show 2024-09-18T08:03:48.480 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:03:48 smithi007 ceph-mon[29459]: purged_snaps scrub starts 2024-09-18T08:03:48.481 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:03:48 smithi007 ceph-mon[29459]: purged_snaps scrub ok 2024-09-18T08:03:48.481 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:03:48 smithi007 ceph-mon[29459]: pgmap v23: 132 pgs: 132 active+clean; 583 KiB data, 223 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-18T08:03:48.481 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:03:48 smithi007 ceph-mon[29459]: osdmap e79: 8 total, 8 up, 8 in 2024-09-18T08:03:48.481 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:03:48 smithi007 ceph-mon[29459]: from='mgr.24470 172.21.15.7:0/2594390261' entity='mgr.y' 2024-09-18T08:03:48.481 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:03:48 smithi007 ceph-mon[29459]: from='mgr.24470 172.21.15.7:0/2594390261' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-18T08:03:48.481 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:03:48 smithi007 ceph-mon[29459]: from='client.? 172.21.15.7:0/87835021' entity='client.admin' cmd={"prefix": "osd dump"} : dispatch 2024-09-18T08:03:48.481 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:03:48 smithi007 ceph-mon[39249]: purged_snaps scrub starts 2024-09-18T08:03:48.481 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:03:48 smithi007 ceph-mon[39249]: purged_snaps scrub ok 2024-09-18T08:03:48.481 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:03:48 smithi007 ceph-mon[39249]: pgmap v23: 132 pgs: 132 active+clean; 583 KiB data, 223 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-18T08:03:48.481 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:03:48 smithi007 ceph-mon[39249]: osdmap e79: 8 total, 8 up, 8 in 2024-09-18T08:03:48.482 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:03:48 smithi007 ceph-mon[39249]: from='mgr.24470 172.21.15.7:0/2594390261' entity='mgr.y' 2024-09-18T08:03:48.482 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:03:48 smithi007 ceph-mon[39249]: from='mgr.24470 172.21.15.7:0/2594390261' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-18T08:03:48.482 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:03:48 smithi007 ceph-mon[39249]: from='client.? 172.21.15.7:0/87835021' entity='client.admin' cmd={"prefix": "osd dump"} : dispatch 2024-09-18T08:03:48.651 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/779349ca-7593-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T08:03:48.655 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:03:48 smithi050 ceph-mon[32371]: purged_snaps scrub starts 2024-09-18T08:03:48.656 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:03:48 smithi050 ceph-mon[32371]: purged_snaps scrub ok 2024-09-18T08:03:48.656 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:03:48 smithi050 ceph-mon[32371]: pgmap v23: 132 pgs: 132 active+clean; 583 KiB data, 223 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-18T08:03:48.656 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:03:48 smithi050 ceph-mon[32371]: osdmap e79: 8 total, 8 up, 8 in 2024-09-18T08:03:48.656 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:03:48 smithi050 ceph-mon[32371]: from='mgr.24470 172.21.15.7:0/2594390261' entity='mgr.y' 2024-09-18T08:03:48.656 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:03:48 smithi050 ceph-mon[32371]: from='mgr.24470 172.21.15.7:0/2594390261' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-18T08:03:48.656 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:03:48 smithi050 ceph-mon[32371]: from='client.? 172.21.15.7:0/87835021' entity='client.admin' cmd={"prefix": "osd dump"} : dispatch 2024-09-18T08:03:49.151 INFO:teuthology.orchestra.run.smithi007.stdout:{ 2024-09-18T08:03:49.151 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "mon.a", 2024-09-18T08:03:49.151 INFO:teuthology.orchestra.run.smithi007.stdout: "cluster": "ceph", 2024-09-18T08:03:49.151 INFO:teuthology.orchestra.run.smithi007.stdout: "admin_socket": "/var/run/ceph/ceph-mon.a.asok", 2024-09-18T08:03:49.151 INFO:teuthology.orchestra.run.smithi007.stdout: "admin_socket_mode": "", 2024-09-18T08:03:49.151 INFO:teuthology.orchestra.run.smithi007.stdout: "auth_allow_insecure_global_id_reclaim": "false", 2024-09-18T08:03:49.152 INFO:teuthology.orchestra.run.smithi007.stdout: "auth_client_required": "cephx, none", 2024-09-18T08:03:49.152 INFO:teuthology.orchestra.run.smithi007.stdout: "auth_cluster_required": "cephx", 2024-09-18T08:03:49.152 INFO:teuthology.orchestra.run.smithi007.stdout: "auth_debug": "true", 2024-09-18T08:03:49.152 INFO:teuthology.orchestra.run.smithi007.stdout: "auth_expose_insecure_global_id_reclaim": "true", 2024-09-18T08:03:49.152 INFO:teuthology.orchestra.run.smithi007.stdout: "auth_mon_ticket_ttl": "660.000000", 2024-09-18T08:03:49.152 INFO:teuthology.orchestra.run.smithi007.stdout: "auth_service_required": "cephx", 2024-09-18T08:03:49.152 INFO:teuthology.orchestra.run.smithi007.stdout: "auth_service_ticket_ttl": "240.000000", 2024-09-18T08:03:49.152 INFO:teuthology.orchestra.run.smithi007.stdout: "auth_supported": "", 2024-09-18T08:03:49.152 INFO:teuthology.orchestra.run.smithi007.stdout: "bdev_aio": "true", 2024-09-18T08:03:49.152 INFO:teuthology.orchestra.run.smithi007.stdout: "bdev_aio_max_queue_depth": "1024", 2024-09-18T08:03:49.152 INFO:teuthology.orchestra.run.smithi007.stdout: "bdev_aio_poll_ms": "250", 2024-09-18T08:03:49.153 INFO:teuthology.orchestra.run.smithi007.stdout: "bdev_aio_reap_max": "16", 2024-09-18T08:03:49.153 INFO:teuthology.orchestra.run.smithi007.stdout: "bdev_async_discard_threads": "0", 2024-09-18T08:03:49.153 INFO:teuthology.orchestra.run.smithi007.stdout: "bdev_block_size": "4096", 2024-09-18T08:03:49.153 INFO:teuthology.orchestra.run.smithi007.stdout: "bdev_debug_aio": "false", 2024-09-18T08:03:49.153 INFO:teuthology.orchestra.run.smithi007.stdout: "bdev_debug_aio_log_age": "5.000000", 2024-09-18T08:03:49.153 INFO:teuthology.orchestra.run.smithi007.stdout: "bdev_debug_aio_suicide_timeout": "60.000000", 2024-09-18T08:03:49.153 INFO:teuthology.orchestra.run.smithi007.stdout: "bdev_debug_inflight_ios": "false", 2024-09-18T08:03:49.153 INFO:teuthology.orchestra.run.smithi007.stdout: "bdev_enable_discard": "false", 2024-09-18T08:03:49.153 INFO:teuthology.orchestra.run.smithi007.stdout: "bdev_flock_retry": "3", 2024-09-18T08:03:49.153 INFO:teuthology.orchestra.run.smithi007.stdout: "bdev_flock_retry_interval": "0.100000", 2024-09-18T08:03:49.153 INFO:teuthology.orchestra.run.smithi007.stdout: "bdev_inject_crash": "0", 2024-09-18T08:03:49.153 INFO:teuthology.orchestra.run.smithi007.stdout: "bdev_inject_crash_flush_delay": "2", 2024-09-18T08:03:49.154 INFO:teuthology.orchestra.run.smithi007.stdout: "bdev_ioring": "false", 2024-09-18T08:03:49.154 INFO:teuthology.orchestra.run.smithi007.stdout: "bdev_ioring_hipri": "false", 2024-09-18T08:03:49.154 INFO:teuthology.orchestra.run.smithi007.stdout: "bdev_ioring_sqthread_poll": "false", 2024-09-18T08:03:49.154 INFO:teuthology.orchestra.run.smithi007.stdout: "bdev_nvme_unbind_from_kernel": "false", 2024-09-18T08:03:49.154 INFO:teuthology.orchestra.run.smithi007.stdout: "bdev_read_buffer_alignment": "4096", 2024-09-18T08:03:49.154 INFO:teuthology.orchestra.run.smithi007.stdout: "bdev_read_preallocated_huge_buffers": "", 2024-09-18T08:03:49.154 INFO:teuthology.orchestra.run.smithi007.stdout: "bdev_stalled_read_warn_lifetime": "86400", 2024-09-18T08:03:49.154 INFO:teuthology.orchestra.run.smithi007.stdout: "bdev_stalled_read_warn_threshold": "1", 2024-09-18T08:03:49.154 INFO:teuthology.orchestra.run.smithi007.stdout: "bdev_type": "", 2024-09-18T08:03:49.154 INFO:teuthology.orchestra.run.smithi007.stdout: "bluefs_alloc_size": "1048576", 2024-09-18T08:03:49.154 INFO:teuthology.orchestra.run.smithi007.stdout: "bluefs_allocator": "hybrid", 2024-09-18T08:03:49.154 INFO:teuthology.orchestra.run.smithi007.stdout: "bluefs_buffered_io": "true", 2024-09-18T08:03:49.154 INFO:teuthology.orchestra.run.smithi007.stdout: "bluefs_check_for_zeros": "false", 2024-09-18T08:03:49.155 INFO:teuthology.orchestra.run.smithi007.stdout: "bluefs_check_volume_selector_often": "false", 2024-09-18T08:03:49.155 INFO:teuthology.orchestra.run.smithi007.stdout: "bluefs_check_volume_selector_on_umount": "false", 2024-09-18T08:03:49.155 INFO:teuthology.orchestra.run.smithi007.stdout: "bluefs_compact_log_sync": "false", 2024-09-18T08:03:49.155 INFO:teuthology.orchestra.run.smithi007.stdout: "bluefs_failed_shared_alloc_cooldown": "600.000000", 2024-09-18T08:03:49.155 INFO:teuthology.orchestra.run.smithi007.stdout: "bluefs_log_compact_min_ratio": "5.000000", 2024-09-18T08:03:49.155 INFO:teuthology.orchestra.run.smithi007.stdout: "bluefs_log_compact_min_size": "16777216", 2024-09-18T08:03:49.155 INFO:teuthology.orchestra.run.smithi007.stdout: "bluefs_log_replay_check_allocations": "true", 2024-09-18T08:03:49.155 INFO:teuthology.orchestra.run.smithi007.stdout: "bluefs_max_log_runway": "4194304", 2024-09-18T08:03:49.155 INFO:teuthology.orchestra.run.smithi007.stdout: "bluefs_max_prefetch": "1048576", 2024-09-18T08:03:49.155 INFO:teuthology.orchestra.run.smithi007.stdout: "bluefs_min_flush_size": "524288", 2024-09-18T08:03:49.155 INFO:teuthology.orchestra.run.smithi007.stdout: "bluefs_min_log_runway": "1048576", 2024-09-18T08:03:49.155 INFO:teuthology.orchestra.run.smithi007.stdout: "bluefs_replay_recovery": "false", 2024-09-18T08:03:49.156 INFO:teuthology.orchestra.run.smithi007.stdout: "bluefs_replay_recovery_disable_compact": "false", 2024-09-18T08:03:49.156 INFO:teuthology.orchestra.run.smithi007.stdout: "bluefs_shared_alloc_size": "65536", 2024-09-18T08:03:49.156 INFO:teuthology.orchestra.run.smithi007.stdout: "bluefs_sync_write": "false", 2024-09-18T08:03:49.156 INFO:teuthology.orchestra.run.smithi007.stdout: "bluestore_2q_cache_kin_ratio": "0.500000", 2024-09-18T08:03:49.156 INFO:teuthology.orchestra.run.smithi007.stdout: "bluestore_2q_cache_kout_ratio": "0.500000", 2024-09-18T08:03:49.156 INFO:teuthology.orchestra.run.smithi007.stdout: "bluestore_alloc_stats_dump_interval": "86400.000000", 2024-09-18T08:03:49.156 INFO:teuthology.orchestra.run.smithi007.stdout: "bluestore_allocation_from_file": "true", 2024-09-18T08:03:49.156 INFO:teuthology.orchestra.run.smithi007.stdout: "bluestore_allocator": "hybrid", 2024-09-18T08:03:49.156 INFO:teuthology.orchestra.run.smithi007.stdout: "bluestore_async_db_compaction": "true", 2024-09-18T08:03:49.156 INFO:teuthology.orchestra.run.smithi007.stdout: "bluestore_avl_alloc_bf_free_pct": "4", 2024-09-18T08:03:49.156 INFO:teuthology.orchestra.run.smithi007.stdout: "bluestore_avl_alloc_bf_threshold": "131072", 2024-09-18T08:03:49.156 INFO:teuthology.orchestra.run.smithi007.stdout: "bluestore_avl_alloc_ff_max_search_bytes": "16777216", 2024-09-18T08:03:49.156 INFO:teuthology.orchestra.run.smithi007.stdout: "bluestore_avl_alloc_ff_max_search_count": "100", 2024-09-18T08:03:49.157 INFO:teuthology.orchestra.run.smithi007.stdout: "bluestore_bdev_label_multi": "true", 2024-09-18T08:03:49.157 INFO:teuthology.orchestra.run.smithi007.stdout: "bluestore_bdev_label_multi_upgrade": "false", 2024-09-18T08:03:49.157 INFO:teuthology.orchestra.run.smithi007.stdout: "bluestore_bdev_label_require_all": "true", 2024-09-18T08:03:49.157 INFO:teuthology.orchestra.run.smithi007.stdout: "bluestore_bitmapallocator_blocks_per_zone": "1024", 2024-09-18T08:03:49.157 INFO:teuthology.orchestra.run.smithi007.stdout: "bluestore_bitmapallocator_span_size": "1024", 2024-09-18T08:03:49.157 INFO:teuthology.orchestra.run.smithi007.stdout: "bluestore_blobid_prealloc": "10240", 2024-09-18T08:03:49.157 INFO:teuthology.orchestra.run.smithi007.stdout: "bluestore_block_create": "true", 2024-09-18T08:03:49.157 INFO:teuthology.orchestra.run.smithi007.stdout: "bluestore_block_db_create": "false", 2024-09-18T08:03:49.157 INFO:teuthology.orchestra.run.smithi007.stdout: "bluestore_block_db_path": "", 2024-09-18T08:03:49.157 INFO:teuthology.orchestra.run.smithi007.stdout: "bluestore_block_db_size": "0", 2024-09-18T08:03:49.157 INFO:teuthology.orchestra.run.smithi007.stdout: "bluestore_block_path": "", 2024-09-18T08:03:49.157 INFO:teuthology.orchestra.run.smithi007.stdout: "bluestore_block_preallocate_file": "false", 2024-09-18T08:03:49.158 INFO:teuthology.orchestra.run.smithi007.stdout: "bluestore_block_size": "107374182400", 2024-09-18T08:03:49.158 INFO:teuthology.orchestra.run.smithi007.stdout: "bluestore_block_wal_create": "false", 2024-09-18T08:03:49.158 INFO:teuthology.orchestra.run.smithi007.stdout: "bluestore_block_wal_path": "", 2024-09-18T08:03:49.158 INFO:teuthology.orchestra.run.smithi007.stdout: "bluestore_block_wal_size": "100663296", 2024-09-18T08:03:49.158 INFO:teuthology.orchestra.run.smithi007.stdout: "bluestore_bluefs": "true", 2024-09-18T08:03:49.158 INFO:teuthology.orchestra.run.smithi007.stdout: "bluestore_bluefs_alloc_failure_dump_interval": "0.000000", 2024-09-18T08:03:49.158 INFO:teuthology.orchestra.run.smithi007.stdout: "bluestore_bluefs_env_mirror": "false", 2024-09-18T08:03:49.158 INFO:teuthology.orchestra.run.smithi007.stdout: "bluestore_bluefs_max_free": "10737418240", 2024-09-18T08:03:49.158 INFO:teuthology.orchestra.run.smithi007.stdout: "bluestore_btree2_alloc_weight_factor": "2.000000", 2024-09-18T08:03:49.158 INFO:teuthology.orchestra.run.smithi007.stdout: "bluestore_cache_age_bin_interval": "1.000000", 2024-09-18T08:03:49.158 INFO:teuthology.orchestra.run.smithi007.stdout: "bluestore_cache_age_bins_data": "1 2 6 24 120 720 0 0 0 0", 2024-09-18T08:03:49.158 INFO:teuthology.orchestra.run.smithi007.stdout: "bluestore_cache_age_bins_kv": "1 2 6 24 120 720 0 0 0 0", 2024-09-18T08:03:49.158 INFO:teuthology.orchestra.run.smithi007.stdout: "bluestore_cache_age_bins_kv_onode": "0 0 0 0 0 0 0 0 0 720", 2024-09-18T08:03:49.158 INFO:teuthology.orchestra.run.smithi007.stdout: "bluestore_cache_age_bins_meta": "1 2 6 24 120 720 0 0 0 0", 2024-09-18T08:03:49.159 INFO:teuthology.orchestra.run.smithi007.stdout: "bluestore_cache_autotune": "true", 2024-09-18T08:03:49.159 INFO:teuthology.orchestra.run.smithi007.stdout: "bluestore_cache_autotune_interval": "5.000000", 2024-09-18T08:03:49.159 INFO:teuthology.orchestra.run.smithi007.stdout: "bluestore_cache_kv_onode_ratio": "0.040000", 2024-09-18T08:03:49.159 INFO:teuthology.orchestra.run.smithi007.stdout: "bluestore_cache_kv_ratio": "0.450000", 2024-09-18T08:03:49.159 INFO:teuthology.orchestra.run.smithi007.stdout: "bluestore_cache_meta_ratio": "0.450000", 2024-09-18T08:03:49.159 INFO:teuthology.orchestra.run.smithi007.stdout: "bluestore_cache_size": "0", 2024-09-18T08:03:49.159 INFO:teuthology.orchestra.run.smithi007.stdout: "bluestore_cache_size_hdd": "1073741824", 2024-09-18T08:03:49.159 INFO:teuthology.orchestra.run.smithi007.stdout: "bluestore_cache_size_ssd": "3221225472", 2024-09-18T08:03:49.159 INFO:teuthology.orchestra.run.smithi007.stdout: "bluestore_cache_trim_interval": "0.050000", 2024-09-18T08:03:49.159 INFO:teuthology.orchestra.run.smithi007.stdout: "bluestore_cache_trim_max_skip_pinned": "1000", 2024-09-18T08:03:49.159 INFO:teuthology.orchestra.run.smithi007.stdout: "bluestore_cache_type": "2q", 2024-09-18T08:03:49.159 INFO:teuthology.orchestra.run.smithi007.stdout: "bluestore_cleaner_sleep_interval": "5.000000", 2024-09-18T08:03:49.159 INFO:teuthology.orchestra.run.smithi007.stdout: "bluestore_clone_cow": "true", 2024-09-18T08:03:49.160 INFO:teuthology.orchestra.run.smithi007.stdout: "bluestore_compression_algorithm": "snappy", 2024-09-18T08:03:49.160 INFO:teuthology.orchestra.run.smithi007.stdout: "bluestore_compression_max_blob_size": "0", 2024-09-18T08:03:49.160 INFO:teuthology.orchestra.run.smithi007.stdout: "bluestore_compression_max_blob_size_hdd": "65536", 2024-09-18T08:03:49.160 INFO:teuthology.orchestra.run.smithi007.stdout: "bluestore_compression_max_blob_size_ssd": "65536", 2024-09-18T08:03:49.160 INFO:teuthology.orchestra.run.smithi007.stdout: "bluestore_compression_min_blob_size": "0", 2024-09-18T08:03:49.160 INFO:teuthology.orchestra.run.smithi007.stdout: "bluestore_compression_min_blob_size_hdd": "8192", 2024-09-18T08:03:49.160 INFO:teuthology.orchestra.run.smithi007.stdout: "bluestore_compression_min_blob_size_ssd": "65536", 2024-09-18T08:03:49.160 INFO:teuthology.orchestra.run.smithi007.stdout: "bluestore_compression_mode": "none", 2024-09-18T08:03:49.160 INFO:teuthology.orchestra.run.smithi007.stdout: "bluestore_compression_required_ratio": "0.875000", 2024-09-18T08:03:49.160 INFO:teuthology.orchestra.run.smithi007.stdout: "bluestore_csum_type": "crc32c", 2024-09-18T08:03:49.160 INFO:teuthology.orchestra.run.smithi007.stdout: "bluestore_debug_enforce_min_alloc_size": "0", 2024-09-18T08:03:49.160 INFO:teuthology.orchestra.run.smithi007.stdout: "bluestore_debug_enforce_settings": "default", 2024-09-18T08:03:49.160 INFO:teuthology.orchestra.run.smithi007.stdout: "bluestore_debug_freelist": "false", 2024-09-18T08:03:49.161 INFO:teuthology.orchestra.run.smithi007.stdout: "bluestore_debug_fsck_abort": "false", 2024-09-18T08:03:49.161 INFO:teuthology.orchestra.run.smithi007.stdout: "bluestore_debug_inject_allocation_from_file_failure": "0.000000", 2024-09-18T08:03:49.161 INFO:teuthology.orchestra.run.smithi007.stdout: "bluestore_debug_inject_csum_err_probability": "0.000000", 2024-09-18T08:03:49.161 INFO:teuthology.orchestra.run.smithi007.stdout: "bluestore_debug_inject_read_err": "false", 2024-09-18T08:03:49.161 INFO:teuthology.orchestra.run.smithi007.stdout: "bluestore_debug_legacy_omap": "false", 2024-09-18T08:03:49.161 INFO:teuthology.orchestra.run.smithi007.stdout: "bluestore_debug_no_reuse_blocks": "false", 2024-09-18T08:03:49.161 INFO:teuthology.orchestra.run.smithi007.stdout: "bluestore_debug_omit_block_device_write": "false", 2024-09-18T08:03:49.161 INFO:teuthology.orchestra.run.smithi007.stdout: "bluestore_debug_omit_kv_commit": "false", 2024-09-18T08:03:49.161 INFO:teuthology.orchestra.run.smithi007.stdout: "bluestore_debug_permit_any_bdev_label": "false", 2024-09-18T08:03:49.161 INFO:teuthology.orchestra.run.smithi007.stdout: "bluestore_debug_prefragment_max": "1048576", 2024-09-18T08:03:49.161 INFO:teuthology.orchestra.run.smithi007.stdout: "bluestore_debug_random_read_err": "0.000000", 2024-09-18T08:03:49.161 INFO:teuthology.orchestra.run.smithi007.stdout: "bluestore_debug_randomize_serial_transaction": "0", 2024-09-18T08:03:49.161 INFO:teuthology.orchestra.run.smithi007.stdout: "bluestore_debug_small_allocations": "0", 2024-09-18T08:03:49.162 INFO:teuthology.orchestra.run.smithi007.stdout: "bluestore_debug_too_many_blobs_threshold": "24576", 2024-09-18T08:03:49.162 INFO:teuthology.orchestra.run.smithi007.stdout: "bluestore_default_buffered_read": "true", 2024-09-18T08:03:49.162 INFO:teuthology.orchestra.run.smithi007.stdout: "bluestore_default_buffered_write": "false", 2024-09-18T08:03:49.162 INFO:teuthology.orchestra.run.smithi007.stdout: "bluestore_deferred_batch_ops": "0", 2024-09-18T08:03:49.162 INFO:teuthology.orchestra.run.smithi007.stdout: "bluestore_deferred_batch_ops_hdd": "64", 2024-09-18T08:03:49.162 INFO:teuthology.orchestra.run.smithi007.stdout: "bluestore_deferred_batch_ops_ssd": "16", 2024-09-18T08:03:49.162 INFO:teuthology.orchestra.run.smithi007.stdout: "bluestore_elastic_shared_blobs": "true", 2024-09-18T08:03:49.162 INFO:teuthology.orchestra.run.smithi007.stdout: "bluestore_extent_map_inline_shard_prealloc_size": "256", 2024-09-18T08:03:49.162 INFO:teuthology.orchestra.run.smithi007.stdout: "bluestore_extent_map_shard_max_size": "1200", 2024-09-18T08:03:49.162 INFO:teuthology.orchestra.run.smithi007.stdout: "bluestore_extent_map_shard_min_size": "150", 2024-09-18T08:03:49.162 INFO:teuthology.orchestra.run.smithi007.stdout: "bluestore_extent_map_shard_target_size": "500", 2024-09-18T08:03:49.162 INFO:teuthology.orchestra.run.smithi007.stdout: "bluestore_extent_map_shard_target_size_slop": "0.200000", 2024-09-18T08:03:49.162 INFO:teuthology.orchestra.run.smithi007.stdout: "bluestore_fail_eio": "false", 2024-09-18T08:03:49.162 INFO:teuthology.orchestra.run.smithi007.stdout: "bluestore_freelist_blocks_per_key": "128", 2024-09-18T08:03:49.163 INFO:teuthology.orchestra.run.smithi007.stdout: "bluestore_fsck_error_on_no_per_pg_omap": "false", 2024-09-18T08:03:49.163 INFO:teuthology.orchestra.run.smithi007.stdout: "bluestore_fsck_error_on_no_per_pool_omap": "false", 2024-09-18T08:03:49.163 INFO:teuthology.orchestra.run.smithi007.stdout: "bluestore_fsck_error_on_no_per_pool_stats": "false", 2024-09-18T08:03:49.163 INFO:teuthology.orchestra.run.smithi007.stdout: "bluestore_fsck_on_mkfs": "true", 2024-09-18T08:03:49.163 INFO:teuthology.orchestra.run.smithi007.stdout: "bluestore_fsck_on_mkfs_deep": "false", 2024-09-18T08:03:49.163 INFO:teuthology.orchestra.run.smithi007.stdout: "bluestore_fsck_on_mount": "false", 2024-09-18T08:03:49.163 INFO:teuthology.orchestra.run.smithi007.stdout: "bluestore_fsck_on_mount_deep": "false", 2024-09-18T08:03:49.163 INFO:teuthology.orchestra.run.smithi007.stdout: "bluestore_fsck_on_umount": "false", 2024-09-18T08:03:49.163 INFO:teuthology.orchestra.run.smithi007.stdout: "bluestore_fsck_on_umount_deep": "false", 2024-09-18T08:03:49.163 INFO:teuthology.orchestra.run.smithi007.stdout: "bluestore_fsck_quick_fix_on_mount": "false", 2024-09-18T08:03:49.163 INFO:teuthology.orchestra.run.smithi007.stdout: "bluestore_fsck_quick_fix_threads": "2", 2024-09-18T08:03:49.163 INFO:teuthology.orchestra.run.smithi007.stdout: "bluestore_fsck_read_bytes_cap": "67108864", 2024-09-18T08:03:49.163 INFO:teuthology.orchestra.run.smithi007.stdout: "bluestore_fsck_shared_blob_tracker_size": "0.031250", 2024-09-18T08:03:49.164 INFO:teuthology.orchestra.run.smithi007.stdout: "bluestore_gc_enable_blob_threshold": "0", 2024-09-18T08:03:49.164 INFO:teuthology.orchestra.run.smithi007.stdout: "bluestore_gc_enable_total_threshold": "0", 2024-09-18T08:03:49.164 INFO:teuthology.orchestra.run.smithi007.stdout: "bluestore_hybrid_alloc_mem_cap": "67108864", 2024-09-18T08:03:49.164 INFO:teuthology.orchestra.run.smithi007.stdout: "bluestore_ignore_data_csum": "false", 2024-09-18T08:03:49.164 INFO:teuthology.orchestra.run.smithi007.stdout: "bluestore_kv_sync_util_logging_s": "10.000000", 2024-09-18T08:03:49.164 INFO:teuthology.orchestra.run.smithi007.stdout: "bluestore_kvbackend": "rocksdb", 2024-09-18T08:03:49.164 INFO:teuthology.orchestra.run.smithi007.stdout: "bluestore_log_collection_list_age": "60.000000", 2024-09-18T08:03:49.164 INFO:teuthology.orchestra.run.smithi007.stdout: "bluestore_log_omap_iterator_age": "5.000000", 2024-09-18T08:03:49.164 INFO:teuthology.orchestra.run.smithi007.stdout: "bluestore_log_op_age": "5.000000", 2024-09-18T08:03:49.164 INFO:teuthology.orchestra.run.smithi007.stdout: "bluestore_max_alloc_size": "0", 2024-09-18T08:03:49.164 INFO:teuthology.orchestra.run.smithi007.stdout: "bluestore_max_blob_size": "0", 2024-09-18T08:03:49.164 INFO:teuthology.orchestra.run.smithi007.stdout: "bluestore_max_blob_size_hdd": "65536", 2024-09-18T08:03:49.164 INFO:teuthology.orchestra.run.smithi007.stdout: "bluestore_max_blob_size_ssd": "65536", 2024-09-18T08:03:49.165 INFO:teuthology.orchestra.run.smithi007.stdout: "bluestore_max_defer_interval": "3.000000", 2024-09-18T08:03:49.165 INFO:teuthology.orchestra.run.smithi007.stdout: "bluestore_max_deferred_txc": "32", 2024-09-18T08:03:49.165 INFO:teuthology.orchestra.run.smithi007.stdout: "bluestore_min_alloc_size": "0", 2024-09-18T08:03:49.165 INFO:teuthology.orchestra.run.smithi007.stdout: "bluestore_min_alloc_size_hdd": "4096", 2024-09-18T08:03:49.165 INFO:teuthology.orchestra.run.smithi007.stdout: "bluestore_min_alloc_size_ssd": "4096", 2024-09-18T08:03:49.165 INFO:teuthology.orchestra.run.smithi007.stdout: "bluestore_nid_prealloc": "1024", 2024-09-18T08:03:49.165 INFO:teuthology.orchestra.run.smithi007.stdout: "bluestore_prefer_deferred_size": "0", 2024-09-18T08:03:49.165 INFO:teuthology.orchestra.run.smithi007.stdout: "bluestore_prefer_deferred_size_hdd": "65536", 2024-09-18T08:03:49.165 INFO:teuthology.orchestra.run.smithi007.stdout: "bluestore_prefer_deferred_size_ssd": "0", 2024-09-18T08:03:49.165 INFO:teuthology.orchestra.run.smithi007.stdout: "bluestore_qfsck_on_mount": "true", 2024-09-18T08:03:49.165 INFO:teuthology.orchestra.run.smithi007.stdout: "bluestore_retry_disk_reads": "3", 2024-09-18T08:03:49.165 INFO:teuthology.orchestra.run.smithi007.stdout: "bluestore_rocksdb_cf": "true", 2024-09-18T08:03:49.165 INFO:teuthology.orchestra.run.smithi007.stdout: "bluestore_rocksdb_cfs": "m(3) p(3,0-12) O(3,0-13)=block_cache={type=binned_lru} L=min_write_buffer_number_to_merge=32 P=min_write_buffer_number_to_merge=32", 2024-09-18T08:03:49.166 INFO:teuthology.orchestra.run.smithi007.stdout: "bluestore_rocksdb_options": "compression=kLZ4Compression,max_write_buffer_number=64,min_write_buffer_number_to_merge=6,compaction_style=kCompactionStyleLevel,write_buffer_size=16777216,max_background_jobs=4,level0_file_num_compaction_trigger=8,max_bytes_for_level_base=1073741824,max_bytes_for_level_multiplier=8,compaction_readahead_size=2MB,max_total_wal_size=1073741824,writable_file_max_buffer_size=0", 2024-09-18T08:03:49.166 INFO:teuthology.orchestra.run.smithi007.stdout: "bluestore_rocksdb_options_annex": "", 2024-09-18T08:03:49.166 INFO:teuthology.orchestra.run.smithi007.stdout: "bluestore_slow_ops_warn_lifetime": "86400", 2024-09-18T08:03:49.166 INFO:teuthology.orchestra.run.smithi007.stdout: "bluestore_slow_ops_warn_threshold": "1", 2024-09-18T08:03:49.166 INFO:teuthology.orchestra.run.smithi007.stdout: "bluestore_spdk_coremask": "0x1", 2024-09-18T08:03:49.166 INFO:teuthology.orchestra.run.smithi007.stdout: "bluestore_spdk_io_sleep": "5", 2024-09-18T08:03:49.166 INFO:teuthology.orchestra.run.smithi007.stdout: "bluestore_spdk_max_io_completion": "0", 2024-09-18T08:03:49.166 INFO:teuthology.orchestra.run.smithi007.stdout: "bluestore_spdk_mem": "512", 2024-09-18T08:03:49.166 INFO:teuthology.orchestra.run.smithi007.stdout: "bluestore_sync_submit_transaction": "false", 2024-09-18T08:03:49.166 INFO:teuthology.orchestra.run.smithi007.stdout: "bluestore_throttle_bytes": "67108864", 2024-09-18T08:03:49.166 INFO:teuthology.orchestra.run.smithi007.stdout: "bluestore_throttle_cost_per_io": "0", 2024-09-18T08:03:49.166 INFO:teuthology.orchestra.run.smithi007.stdout: "bluestore_throttle_cost_per_io_hdd": "670000", 2024-09-18T08:03:49.166 INFO:teuthology.orchestra.run.smithi007.stdout: "bluestore_throttle_cost_per_io_ssd": "4000", 2024-09-18T08:03:49.167 INFO:teuthology.orchestra.run.smithi007.stdout: "bluestore_throttle_deferred_bytes": "134217728", 2024-09-18T08:03:49.167 INFO:teuthology.orchestra.run.smithi007.stdout: "bluestore_throttle_trace_rate": "0.000000", 2024-09-18T08:03:49.167 INFO:teuthology.orchestra.run.smithi007.stdout: "bluestore_tracing": "false", 2024-09-18T08:03:49.167 INFO:teuthology.orchestra.run.smithi007.stdout: "bluestore_use_optimal_io_size_for_min_alloc_size": "false", 2024-09-18T08:03:49.167 INFO:teuthology.orchestra.run.smithi007.stdout: "bluestore_volume_selection_policy": "use_some_extra", 2024-09-18T08:03:49.167 INFO:teuthology.orchestra.run.smithi007.stdout: "bluestore_volume_selection_reserved": "0", 2024-09-18T08:03:49.167 INFO:teuthology.orchestra.run.smithi007.stdout: "bluestore_volume_selection_reserved_factor": "2.000000", 2024-09-18T08:03:49.167 INFO:teuthology.orchestra.run.smithi007.stdout: "bluestore_warn_on_bluefs_spillover": "true", 2024-09-18T08:03:49.167 INFO:teuthology.orchestra.run.smithi007.stdout: "bluestore_warn_on_legacy_statfs": "true", 2024-09-18T08:03:49.167 INFO:teuthology.orchestra.run.smithi007.stdout: "bluestore_warn_on_no_per_pg_omap": "false", 2024-09-18T08:03:49.167 INFO:teuthology.orchestra.run.smithi007.stdout: "bluestore_warn_on_no_per_pool_omap": "true", 2024-09-18T08:03:49.167 INFO:teuthology.orchestra.run.smithi007.stdout: "bluestore_warn_on_spurious_read_errors": "true", 2024-09-18T08:03:49.167 INFO:teuthology.orchestra.run.smithi007.stdout: "bluestore_write_v2": "false", 2024-09-18T08:03:49.168 INFO:teuthology.orchestra.run.smithi007.stdout: "bluestore_write_v2_random": "false", 2024-09-18T08:03:49.168 INFO:teuthology.orchestra.run.smithi007.stdout: "bluestore_zero_block_detection": "false", 2024-09-18T08:03:49.168 INFO:teuthology.orchestra.run.smithi007.stdout: "cephadm_path": "/usr/sbin/cephadm", 2024-09-18T08:03:49.168 INFO:teuthology.orchestra.run.smithi007.stdout: "cephfs_mirror_action_update_interval": "2", 2024-09-18T08:03:49.168 INFO:teuthology.orchestra.run.smithi007.stdout: "cephfs_mirror_directory_scan_interval": "10", 2024-09-18T08:03:49.168 INFO:teuthology.orchestra.run.smithi007.stdout: "cephfs_mirror_max_concurrent_directory_syncs": "3", 2024-09-18T08:03:49.168 INFO:teuthology.orchestra.run.smithi007.stdout: "cephfs_mirror_max_consecutive_failures_per_directory": "10", 2024-09-18T08:03:49.168 INFO:teuthology.orchestra.run.smithi007.stdout: "cephfs_mirror_max_snapshot_sync_per_cycle": "3", 2024-09-18T08:03:49.168 INFO:teuthology.orchestra.run.smithi007.stdout: "cephfs_mirror_mount_timeout": "10", 2024-09-18T08:03:49.168 INFO:teuthology.orchestra.run.smithi007.stdout: "cephfs_mirror_perf_stats_prio": "5", 2024-09-18T08:03:49.168 INFO:teuthology.orchestra.run.smithi007.stdout: "cephfs_mirror_restart_mirror_on_blocklist_interval": "30", 2024-09-18T08:03:49.168 INFO:teuthology.orchestra.run.smithi007.stdout: "cephfs_mirror_restart_mirror_on_failure_interval": "20", 2024-09-18T08:03:49.168 INFO:teuthology.orchestra.run.smithi007.stdout: "cephfs_mirror_retry_failed_directories_interval": "60", 2024-09-18T08:03:49.169 INFO:teuthology.orchestra.run.smithi007.stdout: "cephsqlite_blocklist_dead_locker": "true", 2024-09-18T08:03:49.169 INFO:teuthology.orchestra.run.smithi007.stdout: "cephsqlite_lock_renewal_interval": "2000", 2024-09-18T08:03:49.169 INFO:teuthology.orchestra.run.smithi007.stdout: "cephsqlite_lock_renewal_timeout": "30000", 2024-09-18T08:03:49.169 INFO:teuthology.orchestra.run.smithi007.stdout: "cephx_cluster_require_signatures": "false", 2024-09-18T08:03:49.169 INFO:teuthology.orchestra.run.smithi007.stdout: "cephx_cluster_require_version": "2", 2024-09-18T08:03:49.169 INFO:teuthology.orchestra.run.smithi007.stdout: "cephx_require_signatures": "false", 2024-09-18T08:03:49.169 INFO:teuthology.orchestra.run.smithi007.stdout: "cephx_require_version": "2", 2024-09-18T08:03:49.169 INFO:teuthology.orchestra.run.smithi007.stdout: "cephx_service_require_signatures": "false", 2024-09-18T08:03:49.169 INFO:teuthology.orchestra.run.smithi007.stdout: "cephx_service_require_version": "2", 2024-09-18T08:03:49.169 INFO:teuthology.orchestra.run.smithi007.stdout: "cephx_sign_messages": "true", 2024-09-18T08:03:49.169 INFO:teuthology.orchestra.run.smithi007.stdout: "chdir": "", 2024-09-18T08:03:49.169 INFO:teuthology.orchestra.run.smithi007.stdout: "client_acl_type": "", 2024-09-18T08:03:49.169 INFO:teuthology.orchestra.run.smithi007.stdout: "client_asio_thread_count": "2", 2024-09-18T08:03:49.170 INFO:teuthology.orchestra.run.smithi007.stdout: "client_cache_mid": "0.750000", 2024-09-18T08:03:49.170 INFO:teuthology.orchestra.run.smithi007.stdout: "client_cache_size": "16384", 2024-09-18T08:03:49.170 INFO:teuthology.orchestra.run.smithi007.stdout: "client_caps_release_delay": "5", 2024-09-18T08:03:49.170 INFO:teuthology.orchestra.run.smithi007.stdout: "client_check_pool_perm": "true", 2024-09-18T08:03:49.170 INFO:teuthology.orchestra.run.smithi007.stdout: "client_collect_and_send_global_metrics": "false", 2024-09-18T08:03:49.170 INFO:teuthology.orchestra.run.smithi007.stdout: "client_debug_force_sync_read": "false", 2024-09-18T08:03:49.170 INFO:teuthology.orchestra.run.smithi007.stdout: "client_debug_getattr_caps": "false", 2024-09-18T08:03:49.170 INFO:teuthology.orchestra.run.smithi007.stdout: "client_debug_inject_features": "", 2024-09-18T08:03:49.170 INFO:teuthology.orchestra.run.smithi007.stdout: "client_debug_inject_tick_delay": "0", 2024-09-18T08:03:49.170 INFO:teuthology.orchestra.run.smithi007.stdout: "client_die_on_failed_dentry_invalidate": "true", 2024-09-18T08:03:49.170 INFO:teuthology.orchestra.run.smithi007.stdout: "client_die_on_failed_remount": "false", 2024-09-18T08:03:49.170 INFO:teuthology.orchestra.run.smithi007.stdout: "client_dirsize_rbytes": "true", 2024-09-18T08:03:49.170 INFO:teuthology.orchestra.run.smithi007.stdout: "client_force_lazyio": "false", 2024-09-18T08:03:49.171 INFO:teuthology.orchestra.run.smithi007.stdout: "client_fs": "", 2024-09-18T08:03:49.171 INFO:teuthology.orchestra.run.smithi007.stdout: "client_inject_fixed_oldest_tid": "false", 2024-09-18T08:03:49.171 INFO:teuthology.orchestra.run.smithi007.stdout: "client_inject_release_failure": "false", 2024-09-18T08:03:49.171 INFO:teuthology.orchestra.run.smithi007.stdout: "client_max_inline_size": "4096", 2024-09-18T08:03:49.171 INFO:teuthology.orchestra.run.smithi007.stdout: "client_max_retries_on_remount_failure": "5", 2024-09-18T08:03:49.171 INFO:teuthology.orchestra.run.smithi007.stdout: "client_mds_namespace": "", 2024-09-18T08:03:49.171 INFO:teuthology.orchestra.run.smithi007.stdout: "client_metadata": "", 2024-09-18T08:03:49.171 INFO:teuthology.orchestra.run.smithi007.stdout: "client_mount_gid": "-1", 2024-09-18T08:03:49.171 INFO:teuthology.orchestra.run.smithi007.stdout: "client_mount_timeout": "300", 2024-09-18T08:03:49.171 INFO:teuthology.orchestra.run.smithi007.stdout: "client_mount_uid": "-1", 2024-09-18T08:03:49.171 INFO:teuthology.orchestra.run.smithi007.stdout: "client_mountpoint": "/", 2024-09-18T08:03:49.171 INFO:teuthology.orchestra.run.smithi007.stdout: "client_notify_timeout": "10", 2024-09-18T08:03:49.171 INFO:teuthology.orchestra.run.smithi007.stdout: "client_oc": "true", 2024-09-18T08:03:49.171 INFO:teuthology.orchestra.run.smithi007.stdout: "client_oc_max_dirty": "104857600", 2024-09-18T08:03:49.172 INFO:teuthology.orchestra.run.smithi007.stdout: "client_oc_max_dirty_age": "5.000000", 2024-09-18T08:03:49.172 INFO:teuthology.orchestra.run.smithi007.stdout: "client_oc_max_objects": "1000", 2024-09-18T08:03:49.172 INFO:teuthology.orchestra.run.smithi007.stdout: "client_oc_size": "209715200", 2024-09-18T08:03:49.172 INFO:teuthology.orchestra.run.smithi007.stdout: "client_oc_target_dirty": "8388608", 2024-09-18T08:03:49.172 INFO:teuthology.orchestra.run.smithi007.stdout: "client_permissions": "true", 2024-09-18T08:03:49.172 INFO:teuthology.orchestra.run.smithi007.stdout: "client_quota": "true", 2024-09-18T08:03:49.172 INFO:teuthology.orchestra.run.smithi007.stdout: "client_quota_df": "true", 2024-09-18T08:03:49.172 INFO:teuthology.orchestra.run.smithi007.stdout: "client_readahead_max_bytes": "0", 2024-09-18T08:03:49.172 INFO:teuthology.orchestra.run.smithi007.stdout: "client_readahead_max_periods": "4", 2024-09-18T08:03:49.172 INFO:teuthology.orchestra.run.smithi007.stdout: "client_readahead_min": "131072", 2024-09-18T08:03:49.172 INFO:teuthology.orchestra.run.smithi007.stdout: "client_reconnect_stale": "false", 2024-09-18T08:03:49.172 INFO:teuthology.orchestra.run.smithi007.stdout: "client_shutdown_timeout": "30", 2024-09-18T08:03:49.172 INFO:teuthology.orchestra.run.smithi007.stdout: "client_snapdir": ".snap", 2024-09-18T08:03:49.173 INFO:teuthology.orchestra.run.smithi007.stdout: "client_tick_interval": "1", 2024-09-18T08:03:49.173 INFO:teuthology.orchestra.run.smithi007.stdout: "client_trace": "", 2024-09-18T08:03:49.173 INFO:teuthology.orchestra.run.smithi007.stdout: "client_try_dentry_invalidate": "false", 2024-09-18T08:03:49.173 INFO:teuthology.orchestra.run.smithi007.stdout: "client_use_faked_inos": "false", 2024-09-18T08:03:49.173 INFO:teuthology.orchestra.run.smithi007.stdout: "client_use_random_mds": "false", 2024-09-18T08:03:49.173 INFO:teuthology.orchestra.run.smithi007.stdout: "clog_to_graylog": "false", 2024-09-18T08:03:49.173 INFO:teuthology.orchestra.run.smithi007.stdout: "clog_to_graylog_host": "127.0.0.1", 2024-09-18T08:03:49.173 INFO:teuthology.orchestra.run.smithi007.stdout: "clog_to_graylog_port": "12201", 2024-09-18T08:03:49.173 INFO:teuthology.orchestra.run.smithi007.stdout: "clog_to_monitors": "default=true", 2024-09-18T08:03:49.173 INFO:teuthology.orchestra.run.smithi007.stdout: "clog_to_syslog": "false", 2024-09-18T08:03:49.173 INFO:teuthology.orchestra.run.smithi007.stdout: "clog_to_syslog_facility": "default=daemon audit=local0", 2024-09-18T08:03:49.173 INFO:teuthology.orchestra.run.smithi007.stdout: "clog_to_syslog_level": "info", 2024-09-18T08:03:49.173 INFO:teuthology.orchestra.run.smithi007.stdout: "cluster_addr": "-", 2024-09-18T08:03:49.174 INFO:teuthology.orchestra.run.smithi007.stdout: "cluster_network": "", 2024-09-18T08:03:49.174 INFO:teuthology.orchestra.run.smithi007.stdout: "cluster_network_interface": "", 2024-09-18T08:03:49.174 INFO:teuthology.orchestra.run.smithi007.stdout: "compressor_zlib_isal": "false", 2024-09-18T08:03:49.174 INFO:teuthology.orchestra.run.smithi007.stdout: "compressor_zlib_level": "5", 2024-09-18T08:03:49.174 INFO:teuthology.orchestra.run.smithi007.stdout: "compressor_zlib_winsize": "-15", 2024-09-18T08:03:49.174 INFO:teuthology.orchestra.run.smithi007.stdout: "compressor_zstd_level": "1", 2024-09-18T08:03:49.174 INFO:teuthology.orchestra.run.smithi007.stdout: "container_image": "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:1f5b5a3734f665ef333266f39143ac3ef1eb1951b21258f2307e10332e2d7ac9", 2024-09-18T08:03:49.174 INFO:teuthology.orchestra.run.smithi007.stdout: "crash_dir": "/var/lib/ceph/crash", 2024-09-18T08:03:49.174 INFO:teuthology.orchestra.run.smithi007.stdout: "crimson_alien_op_num_threads": "6", 2024-09-18T08:03:49.174 INFO:teuthology.orchestra.run.smithi007.stdout: "crimson_alien_thread_cpu_cores": "", 2024-09-18T08:03:49.175 INFO:teuthology.orchestra.run.smithi007.stdout: "crimson_osd_obc_lru_size": "512", 2024-09-18T08:03:49.175 INFO:teuthology.orchestra.run.smithi007.stdout: "crimson_osd_scheduler_concurrency": "0", 2024-09-18T08:03:49.175 INFO:teuthology.orchestra.run.smithi007.stdout: "crimson_osd_stat_interval": "0", 2024-09-18T08:03:49.175 INFO:teuthology.orchestra.run.smithi007.stdout: "crimson_seastar_cpu_cores": "", 2024-09-18T08:03:49.175 INFO:teuthology.orchestra.run.smithi007.stdout: "crimson_seastar_num_threads": "0", 2024-09-18T08:03:49.175 INFO:teuthology.orchestra.run.smithi007.stdout: "crush_location": "", 2024-09-18T08:03:49.175 INFO:teuthology.orchestra.run.smithi007.stdout: "crush_location_hook": "", 2024-09-18T08:03:49.175 INFO:teuthology.orchestra.run.smithi007.stdout: "crush_location_hook_timeout": "10", 2024-09-18T08:03:49.175 INFO:teuthology.orchestra.run.smithi007.stdout: "daemonize": "false", 2024-09-18T08:03:49.175 INFO:teuthology.orchestra.run.smithi007.stdout: "daos_pool": "tank", 2024-09-18T08:03:49.175 INFO:teuthology.orchestra.run.smithi007.stdout: "dbstore_config_uri": "file:/var/lib/ceph/radosgw/dbstore-config.db", 2024-09-18T08:03:49.175 INFO:teuthology.orchestra.run.smithi007.stdout: "dbstore_db_dir": "/var/lib/ceph/radosgw", 2024-09-18T08:03:49.175 INFO:teuthology.orchestra.run.smithi007.stdout: "dbstore_db_name_prefix": "dbstore", 2024-09-18T08:03:49.176 INFO:teuthology.orchestra.run.smithi007.stdout: "debug_alienstore": "0/5", 2024-09-18T08:03:49.176 INFO:teuthology.orchestra.run.smithi007.stdout: "debug_allow_any_pool_priority": "false", 2024-09-18T08:03:49.176 INFO:teuthology.orchestra.run.smithi007.stdout: "debug_asok": "1/5", 2024-09-18T08:03:49.176 INFO:teuthology.orchestra.run.smithi007.stdout: "debug_asok_assert_abort": "false", 2024-09-18T08:03:49.176 INFO:teuthology.orchestra.run.smithi007.stdout: "debug_asserts_on_shutdown": "true", 2024-09-18T08:03:49.176 INFO:teuthology.orchestra.run.smithi007.stdout: "debug_auth": "1/5", 2024-09-18T08:03:49.176 INFO:teuthology.orchestra.run.smithi007.stdout: "debug_bdev": "1/3", 2024-09-18T08:03:49.176 INFO:teuthology.orchestra.run.smithi007.stdout: "debug_bluefs": "1/5", 2024-09-18T08:03:49.176 INFO:teuthology.orchestra.run.smithi007.stdout: "debug_bluestore": "1/5", 2024-09-18T08:03:49.176 INFO:teuthology.orchestra.run.smithi007.stdout: "debug_buffer": "0/1", 2024-09-18T08:03:49.176 INFO:teuthology.orchestra.run.smithi007.stdout: "debug_ceph_dedup": "0/5", 2024-09-18T08:03:49.176 INFO:teuthology.orchestra.run.smithi007.stdout: "debug_ceph_exporter": "1/5", 2024-09-18T08:03:49.176 INFO:teuthology.orchestra.run.smithi007.stdout: "debug_cephfs_mirror": "0/5", 2024-09-18T08:03:49.176 INFO:teuthology.orchestra.run.smithi007.stdout: "debug_cephsqlite": "0/5", 2024-09-18T08:03:49.177 INFO:teuthology.orchestra.run.smithi007.stdout: "debug_client": "0/5", 2024-09-18T08:03:49.177 INFO:teuthology.orchestra.run.smithi007.stdout: "debug_compressor": "1/5", 2024-09-18T08:03:49.177 INFO:teuthology.orchestra.run.smithi007.stdout: "debug_context": "0/1", 2024-09-18T08:03:49.177 INFO:teuthology.orchestra.run.smithi007.stdout: "debug_crimson_interrupt": "0/5", 2024-09-18T08:03:49.177 INFO:teuthology.orchestra.run.smithi007.stdout: "debug_crush": "1/1", 2024-09-18T08:03:49.177 INFO:teuthology.orchestra.run.smithi007.stdout: "debug_crypto": "1/5", 2024-09-18T08:03:49.177 INFO:teuthology.orchestra.run.smithi007.stdout: "debug_cyanstore": "0/5", 2024-09-18T08:03:49.177 INFO:teuthology.orchestra.run.smithi007.stdout: "debug_deliberately_leak_memory": "false", 2024-09-18T08:03:49.177 INFO:teuthology.orchestra.run.smithi007.stdout: "debug_disable_randomized_ping": "false", 2024-09-18T08:03:49.177 INFO:teuthology.orchestra.run.smithi007.stdout: "debug_dpdk": "1/5", 2024-09-18T08:03:49.177 INFO:teuthology.orchestra.run.smithi007.stdout: "debug_eventtrace": "1/5", 2024-09-18T08:03:49.177 INFO:teuthology.orchestra.run.smithi007.stdout: "debug_filer": "0/1", 2024-09-18T08:03:49.177 INFO:teuthology.orchestra.run.smithi007.stdout: "debug_filestore": "1/3", 2024-09-18T08:03:49.178 INFO:teuthology.orchestra.run.smithi007.stdout: "debug_finisher": "1/1", 2024-09-18T08:03:49.178 INFO:teuthology.orchestra.run.smithi007.stdout: "debug_fuse": "1/5", 2024-09-18T08:03:49.178 INFO:teuthology.orchestra.run.smithi007.stdout: "debug_heartbeat_testing_span": "0", 2024-09-18T08:03:49.178 INFO:teuthology.orchestra.run.smithi007.stdout: "debug_heartbeatmap": "1/5", 2024-09-18T08:03:49.178 INFO:teuthology.orchestra.run.smithi007.stdout: "debug_immutable_obj_cache": "0/5", 2024-09-18T08:03:49.178 INFO:teuthology.orchestra.run.smithi007.stdout: "debug_javaclient": "1/5", 2024-09-18T08:03:49.178 INFO:teuthology.orchestra.run.smithi007.stdout: "debug_journal": "1/3", 2024-09-18T08:03:49.178 INFO:teuthology.orchestra.run.smithi007.stdout: "debug_journaler": "0/5", 2024-09-18T08:03:49.178 INFO:teuthology.orchestra.run.smithi007.stdout: "debug_kstore": "1/5", 2024-09-18T08:03:49.178 INFO:teuthology.orchestra.run.smithi007.stdout: "debug_lockdep": "0/1", 2024-09-18T08:03:49.179 INFO:teuthology.orchestra.run.smithi007.stdout: "debug_mclock": "1/5", 2024-09-18T08:03:49.179 INFO:teuthology.orchestra.run.smithi007.stdout: "debug_mds": "1/5", 2024-09-18T08:03:49.179 INFO:teuthology.orchestra.run.smithi007.stdout: "debug_mds_balancer": "1/5", 2024-09-18T08:03:49.179 INFO:teuthology.orchestra.run.smithi007.stdout: "debug_mds_locker": "1/5", 2024-09-18T08:03:49.179 INFO:teuthology.orchestra.run.smithi007.stdout: "debug_mds_log": "1/5", 2024-09-18T08:03:49.179 INFO:teuthology.orchestra.run.smithi007.stdout: "debug_mds_log_expire": "1/5", 2024-09-18T08:03:49.179 INFO:teuthology.orchestra.run.smithi007.stdout: "debug_mds_migrator": "1/5", 2024-09-18T08:03:49.179 INFO:teuthology.orchestra.run.smithi007.stdout: "debug_mds_quiesce": "3/5", 2024-09-18T08:03:49.179 INFO:teuthology.orchestra.run.smithi007.stdout: "debug_memstore": "1/5", 2024-09-18T08:03:49.179 INFO:teuthology.orchestra.run.smithi007.stdout: "debug_mgr": "2/5", 2024-09-18T08:03:49.179 INFO:teuthology.orchestra.run.smithi007.stdout: "debug_mgrc": "1/5", 2024-09-18T08:03:49.179 INFO:teuthology.orchestra.run.smithi007.stdout: "debug_mon": "20/20", 2024-09-18T08:03:49.179 INFO:teuthology.orchestra.run.smithi007.stdout: "debug_monc": "0/10", 2024-09-18T08:03:49.179 INFO:teuthology.orchestra.run.smithi007.stdout: "debug_ms": "1/1", 2024-09-18T08:03:49.180 INFO:teuthology.orchestra.run.smithi007.stdout: "debug_none": "0/5", 2024-09-18T08:03:49.180 INFO:teuthology.orchestra.run.smithi007.stdout: "debug_objclass": "0/5", 2024-09-18T08:03:49.180 INFO:teuthology.orchestra.run.smithi007.stdout: "debug_objectcacher": "0/5", 2024-09-18T08:03:49.180 INFO:teuthology.orchestra.run.smithi007.stdout: "debug_objecter": "0/1", 2024-09-18T08:03:49.180 INFO:teuthology.orchestra.run.smithi007.stdout: "debug_optracker": "0/5", 2024-09-18T08:03:49.180 INFO:teuthology.orchestra.run.smithi007.stdout: "debug_osd": "1/5", 2024-09-18T08:03:49.180 INFO:teuthology.orchestra.run.smithi007.stdout: "debug_paxos": "20/20", 2024-09-18T08:03:49.180 INFO:teuthology.orchestra.run.smithi007.stdout: "debug_perfcounter": "1/5", 2024-09-18T08:03:49.180 INFO:teuthology.orchestra.run.smithi007.stdout: "debug_prioritycache": "1/5", 2024-09-18T08:03:49.180 INFO:teuthology.orchestra.run.smithi007.stdout: "debug_rados": "0/5", 2024-09-18T08:03:49.180 INFO:teuthology.orchestra.run.smithi007.stdout: "debug_rbd": "0/5", 2024-09-18T08:03:49.180 INFO:teuthology.orchestra.run.smithi007.stdout: "debug_rbd_mirror": "0/5", 2024-09-18T08:03:49.181 INFO:teuthology.orchestra.run.smithi007.stdout: "debug_rbd_pwl": "0/5", 2024-09-18T08:03:49.181 INFO:teuthology.orchestra.run.smithi007.stdout: "debug_rbd_replay": "0/5", 2024-09-18T08:03:49.181 INFO:teuthology.orchestra.run.smithi007.stdout: "debug_refs": "0/0", 2024-09-18T08:03:49.181 INFO:teuthology.orchestra.run.smithi007.stdout: "debug_reserver": "1/1", 2024-09-18T08:03:49.181 INFO:teuthology.orchestra.run.smithi007.stdout: "debug_rgw": "1/5", 2024-09-18T08:03:49.181 INFO:teuthology.orchestra.run.smithi007.stdout: "debug_rgw_access": "1/5", 2024-09-18T08:03:49.181 INFO:teuthology.orchestra.run.smithi007.stdout: "debug_rgw_datacache": "1/5", 2024-09-18T08:03:49.181 INFO:teuthology.orchestra.run.smithi007.stdout: "debug_rgw_dbstore": "1/5", 2024-09-18T08:03:49.181 INFO:teuthology.orchestra.run.smithi007.stdout: "debug_rgw_flight": "1/5", 2024-09-18T08:03:49.182 INFO:journalctl@ceph.iscsi.iscsi.a.smithi050.stdout:Sep 18 08:03:48 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-iscsi-iscsi-a[57782]: debug there is no tcmu-runner data available 2024-09-18T08:03:49.183 INFO:teuthology.orchestra.run.smithi007.stdout: "debug_rgw_lifecycle": "1/5", 2024-09-18T08:03:49.183 INFO:teuthology.orchestra.run.smithi007.stdout: "debug_rgw_notification": "1/5", 2024-09-18T08:03:49.183 INFO:teuthology.orchestra.run.smithi007.stdout: "debug_rgw_sync": "1/5", 2024-09-18T08:03:49.183 INFO:teuthology.orchestra.run.smithi007.stdout: "debug_rocksdb": "4/5", 2024-09-18T08:03:49.183 INFO:teuthology.orchestra.run.smithi007.stdout: "debug_seastore": "0/5", 2024-09-18T08:03:49.183 INFO:teuthology.orchestra.run.smithi007.stdout: "debug_seastore_backref": "0/5", 2024-09-18T08:03:49.183 INFO:teuthology.orchestra.run.smithi007.stdout: "debug_seastore_cache": "0/5", 2024-09-18T08:03:49.183 INFO:teuthology.orchestra.run.smithi007.stdout: "debug_seastore_cleaner": "0/5", 2024-09-18T08:03:49.183 INFO:teuthology.orchestra.run.smithi007.stdout: "debug_seastore_device": "0/5", 2024-09-18T08:03:49.184 INFO:teuthology.orchestra.run.smithi007.stdout: "debug_seastore_epm": "0/5", 2024-09-18T08:03:49.184 INFO:teuthology.orchestra.run.smithi007.stdout: "debug_seastore_fixedkv_tree": "0/5", 2024-09-18T08:03:49.184 INFO:teuthology.orchestra.run.smithi007.stdout: "debug_seastore_journal": "0/5", 2024-09-18T08:03:49.184 INFO:teuthology.orchestra.run.smithi007.stdout: "debug_seastore_lba": "0/5", 2024-09-18T08:03:49.184 INFO:teuthology.orchestra.run.smithi007.stdout: "debug_seastore_odata": "0/5", 2024-09-18T08:03:49.184 INFO:teuthology.orchestra.run.smithi007.stdout: "debug_seastore_omap": "0/5", 2024-09-18T08:03:49.184 INFO:teuthology.orchestra.run.smithi007.stdout: "debug_seastore_onode": "0/5", 2024-09-18T08:03:49.184 INFO:teuthology.orchestra.run.smithi007.stdout: "debug_seastore_t": "0/5", 2024-09-18T08:03:49.184 INFO:teuthology.orchestra.run.smithi007.stdout: "debug_seastore_tm": "0/5", 2024-09-18T08:03:49.184 INFO:teuthology.orchestra.run.smithi007.stdout: "debug_striper": "0/1", 2024-09-18T08:03:49.184 INFO:teuthology.orchestra.run.smithi007.stdout: "debug_test": "0/5", 2024-09-18T08:03:49.184 INFO:teuthology.orchestra.run.smithi007.stdout: "debug_throttle": "1/1", 2024-09-18T08:03:49.185 INFO:teuthology.orchestra.run.smithi007.stdout: "debug_timer": "0/1", 2024-09-18T08:03:49.185 INFO:teuthology.orchestra.run.smithi007.stdout: "debug_tp": "0/5", 2024-09-18T08:03:49.185 INFO:teuthology.orchestra.run.smithi007.stdout: "debug_trace": "1/5", 2024-09-18T08:03:49.185 INFO:teuthology.orchestra.run.smithi007.stdout: "defer_client_eviction_on_laggy_osds": "false", 2024-09-18T08:03:49.185 INFO:teuthology.orchestra.run.smithi007.stdout: "device_failure_prediction_mode": "none", 2024-09-18T08:03:49.185 INFO:teuthology.orchestra.run.smithi007.stdout: "enable_experimental_unrecoverable_data_corrupting_features": "", 2024-09-18T08:03:49.185 INFO:teuthology.orchestra.run.smithi007.stdout: "erasure_code_dir": "/usr/lib64/ceph/erasure-code", 2024-09-18T08:03:49.185 INFO:teuthology.orchestra.run.smithi007.stdout: "err_to_graylog": "false", 2024-09-18T08:03:49.185 INFO:teuthology.orchestra.run.smithi007.stdout: "err_to_journald": "false", 2024-09-18T08:03:49.185 INFO:teuthology.orchestra.run.smithi007.stdout: "err_to_stderr": "true", 2024-09-18T08:03:49.185 INFO:teuthology.orchestra.run.smithi007.stdout: "err_to_syslog": "false", 2024-09-18T08:03:49.185 INFO:teuthology.orchestra.run.smithi007.stdout: "event_tracing": "false", 2024-09-18T08:03:49.185 INFO:teuthology.orchestra.run.smithi007.stdout: "exporter_addr": "0.0.0.0", 2024-09-18T08:03:49.186 INFO:teuthology.orchestra.run.smithi007.stdout: "exporter_cert_file": "", 2024-09-18T08:03:49.186 INFO:teuthology.orchestra.run.smithi007.stdout: "exporter_http_port": "9926", 2024-09-18T08:03:49.186 INFO:teuthology.orchestra.run.smithi007.stdout: "exporter_key_file": "", 2024-09-18T08:03:49.186 INFO:teuthology.orchestra.run.smithi007.stdout: "exporter_prio_limit": "5", 2024-09-18T08:03:49.186 INFO:teuthology.orchestra.run.smithi007.stdout: "exporter_sock_dir": "/var/run/ceph/", 2024-09-18T08:03:49.186 INFO:teuthology.orchestra.run.smithi007.stdout: "exporter_sort_metrics": "true", 2024-09-18T08:03:49.186 INFO:teuthology.orchestra.run.smithi007.stdout: "exporter_stats_period": "5", 2024-09-18T08:03:49.186 INFO:teuthology.orchestra.run.smithi007.stdout: "fake_statfs_for_testing": "0", 2024-09-18T08:03:49.186 INFO:teuthology.orchestra.run.smithi007.stdout: "fatal_signal_handlers": "true", 2024-09-18T08:03:49.186 INFO:teuthology.orchestra.run.smithi007.stdout: "filer_max_purge_ops": "10", 2024-09-18T08:03:49.186 INFO:teuthology.orchestra.run.smithi007.stdout: "filer_max_truncate_ops": "128", 2024-09-18T08:03:49.186 INFO:teuthology.orchestra.run.smithi007.stdout: "filestore_apply_finisher_threads": "1", 2024-09-18T08:03:49.186 INFO:teuthology.orchestra.run.smithi007.stdout: "filestore_blackhole": "false", 2024-09-18T08:03:49.187 INFO:teuthology.orchestra.run.smithi007.stdout: "filestore_btrfs_clone_range": "true", 2024-09-18T08:03:49.187 INFO:teuthology.orchestra.run.smithi007.stdout: "filestore_btrfs_snap": "true", 2024-09-18T08:03:49.187 INFO:teuthology.orchestra.run.smithi007.stdout: "filestore_caller_concurrency": "10", 2024-09-18T08:03:49.187 INFO:teuthology.orchestra.run.smithi007.stdout: "filestore_collect_device_partition_information": "true", 2024-09-18T08:03:49.187 INFO:teuthology.orchestra.run.smithi007.stdout: "filestore_commit_timeout": "600.000000", 2024-09-18T08:03:49.187 INFO:teuthology.orchestra.run.smithi007.stdout: "filestore_debug_inject_read_err": "false", 2024-09-18T08:03:49.187 INFO:teuthology.orchestra.run.smithi007.stdout: "filestore_debug_omap_check": "false", 2024-09-18T08:03:49.187 INFO:teuthology.orchestra.run.smithi007.stdout: "filestore_debug_verify_split": "false", 2024-09-18T08:03:49.187 INFO:teuthology.orchestra.run.smithi007.stdout: "filestore_dump_file": "", 2024-09-18T08:03:49.187 INFO:teuthology.orchestra.run.smithi007.stdout: "filestore_expected_throughput_bytes": "209715200.000000", 2024-09-18T08:03:49.187 INFO:teuthology.orchestra.run.smithi007.stdout: "filestore_expected_throughput_ops": "200.000000", 2024-09-18T08:03:49.187 INFO:teuthology.orchestra.run.smithi007.stdout: "filestore_fadvise": "true", 2024-09-18T08:03:49.187 INFO:teuthology.orchestra.run.smithi007.stdout: "filestore_fail_eio": "true", 2024-09-18T08:03:49.188 INFO:teuthology.orchestra.run.smithi007.stdout: "filestore_fd_cache_shards": "16", 2024-09-18T08:03:49.188 INFO:teuthology.orchestra.run.smithi007.stdout: "filestore_fd_cache_size": "128", 2024-09-18T08:03:49.188 INFO:teuthology.orchestra.run.smithi007.stdout: "filestore_fiemap": "false", 2024-09-18T08:03:49.188 INFO:teuthology.orchestra.run.smithi007.stdout: "filestore_fiemap_threshold": "4096", 2024-09-18T08:03:49.188 INFO:teuthology.orchestra.run.smithi007.stdout: "filestore_fsync_flushes_journal_data": "false", 2024-09-18T08:03:49.188 INFO:teuthology.orchestra.run.smithi007.stdout: "filestore_index_retry_probability": "0.000000", 2024-09-18T08:03:49.188 INFO:teuthology.orchestra.run.smithi007.stdout: "filestore_inject_stall": "0", 2024-09-18T08:03:49.188 INFO:teuthology.orchestra.run.smithi007.stdout: "filestore_journal_parallel": "false", 2024-09-18T08:03:49.188 INFO:teuthology.orchestra.run.smithi007.stdout: "filestore_journal_trailing": "false", 2024-09-18T08:03:49.188 INFO:teuthology.orchestra.run.smithi007.stdout: "filestore_journal_writeahead": "false", 2024-09-18T08:03:49.188 INFO:teuthology.orchestra.run.smithi007.stdout: "filestore_kill_at": "0", 2024-09-18T08:03:49.188 INFO:teuthology.orchestra.run.smithi007.stdout: "filestore_max_alloc_hint_size": "1048576", 2024-09-18T08:03:49.188 INFO:teuthology.orchestra.run.smithi007.stdout: "filestore_max_inline_xattr_size": "0", 2024-09-18T08:03:49.189 INFO:teuthology.orchestra.run.smithi007.stdout: "filestore_max_inline_xattr_size_btrfs": "2048", 2024-09-18T08:03:49.189 INFO:teuthology.orchestra.run.smithi007.stdout: "filestore_max_inline_xattr_size_other": "512", 2024-09-18T08:03:49.189 INFO:teuthology.orchestra.run.smithi007.stdout: "filestore_max_inline_xattr_size_xfs": "65536", 2024-09-18T08:03:49.189 INFO:teuthology.orchestra.run.smithi007.stdout: "filestore_max_inline_xattrs": "0", 2024-09-18T08:03:49.189 INFO:teuthology.orchestra.run.smithi007.stdout: "filestore_max_inline_xattrs_btrfs": "10", 2024-09-18T08:03:49.189 INFO:teuthology.orchestra.run.smithi007.stdout: "filestore_max_inline_xattrs_other": "2", 2024-09-18T08:03:49.189 INFO:teuthology.orchestra.run.smithi007.stdout: "filestore_max_inline_xattrs_xfs": "10", 2024-09-18T08:03:49.189 INFO:teuthology.orchestra.run.smithi007.stdout: "filestore_max_sync_interval": "5.000000", 2024-09-18T08:03:49.189 INFO:teuthology.orchestra.run.smithi007.stdout: "filestore_max_xattr_value_size": "0", 2024-09-18T08:03:49.189 INFO:teuthology.orchestra.run.smithi007.stdout: "filestore_max_xattr_value_size_btrfs": "65536", 2024-09-18T08:03:49.189 INFO:teuthology.orchestra.run.smithi007.stdout: "filestore_max_xattr_value_size_other": "1024", 2024-09-18T08:03:49.189 INFO:teuthology.orchestra.run.smithi007.stdout: "filestore_max_xattr_value_size_xfs": "65536", 2024-09-18T08:03:49.189 INFO:teuthology.orchestra.run.smithi007.stdout: "filestore_merge_threshold": "-10", 2024-09-18T08:03:49.189 INFO:teuthology.orchestra.run.smithi007.stdout: "filestore_min_sync_interval": "0.010000", 2024-09-18T08:03:49.190 INFO:teuthology.orchestra.run.smithi007.stdout: "filestore_odsync_write": "false", 2024-09-18T08:03:49.190 INFO:teuthology.orchestra.run.smithi007.stdout: "filestore_omap_backend": "rocksdb", 2024-09-18T08:03:49.190 INFO:teuthology.orchestra.run.smithi007.stdout: "filestore_omap_backend_path": "", 2024-09-18T08:03:49.190 INFO:teuthology.orchestra.run.smithi007.stdout: "filestore_omap_header_cache_size": "1024", 2024-09-18T08:03:49.190 INFO:teuthology.orchestra.run.smithi007.stdout: "filestore_ondisk_finisher_threads": "1", 2024-09-18T08:03:49.190 INFO:teuthology.orchestra.run.smithi007.stdout: "filestore_op_thread_suicide_timeout": "180", 2024-09-18T08:03:49.190 INFO:teuthology.orchestra.run.smithi007.stdout: "filestore_op_thread_timeout": "60", 2024-09-18T08:03:49.190 INFO:teuthology.orchestra.run.smithi007.stdout: "filestore_op_threads": "2", 2024-09-18T08:03:49.190 INFO:teuthology.orchestra.run.smithi007.stdout: "filestore_punch_hole": "false", 2024-09-18T08:03:49.190 INFO:teuthology.orchestra.run.smithi007.stdout: "filestore_queue_high_delay_multiple": "0.000000", 2024-09-18T08:03:49.190 INFO:teuthology.orchestra.run.smithi007.stdout: "filestore_queue_high_delay_multiple_bytes": "0.000000", 2024-09-18T08:03:49.190 INFO:teuthology.orchestra.run.smithi007.stdout: "filestore_queue_high_delay_multiple_ops": "0.000000", 2024-09-18T08:03:49.190 INFO:teuthology.orchestra.run.smithi007.stdout: "filestore_queue_high_threshhold": "0.900000", 2024-09-18T08:03:49.191 INFO:teuthology.orchestra.run.smithi007.stdout: "filestore_queue_low_threshhold": "0.300000", 2024-09-18T08:03:49.191 INFO:teuthology.orchestra.run.smithi007.stdout: "filestore_queue_max_bytes": "104857600", 2024-09-18T08:03:49.191 INFO:teuthology.orchestra.run.smithi007.stdout: "filestore_queue_max_delay_multiple": "0.000000", 2024-09-18T08:03:49.191 INFO:teuthology.orchestra.run.smithi007.stdout: "filestore_queue_max_delay_multiple_bytes": "0.000000", 2024-09-18T08:03:49.191 INFO:teuthology.orchestra.run.smithi007.stdout: "filestore_queue_max_delay_multiple_ops": "0.000000", 2024-09-18T08:03:49.191 INFO:teuthology.orchestra.run.smithi007.stdout: "filestore_queue_max_ops": "50", 2024-09-18T08:03:49.191 INFO:teuthology.orchestra.run.smithi007.stdout: "filestore_rocksdb_options": "max_background_jobs=10,compaction_readahead_size=2097152,compression=kNoCompression", 2024-09-18T08:03:49.191 INFO:teuthology.orchestra.run.smithi007.stdout: "filestore_seek_data_hole": "false", 2024-09-18T08:03:49.191 INFO:teuthology.orchestra.run.smithi007.stdout: "filestore_sloppy_crc": "false", 2024-09-18T08:03:49.191 INFO:teuthology.orchestra.run.smithi007.stdout: "filestore_sloppy_crc_block_size": "65536", 2024-09-18T08:03:49.191 INFO:teuthology.orchestra.run.smithi007.stdout: "filestore_splice": "false", 2024-09-18T08:03:49.191 INFO:teuthology.orchestra.run.smithi007.stdout: "filestore_split_multiple": "2", 2024-09-18T08:03:49.191 INFO:teuthology.orchestra.run.smithi007.stdout: "filestore_split_rand_factor": "20", 2024-09-18T08:03:49.192 INFO:teuthology.orchestra.run.smithi007.stdout: "filestore_update_to": "1000", 2024-09-18T08:03:49.192 INFO:teuthology.orchestra.run.smithi007.stdout: "filestore_wbthrottle_btrfs_bytes_hard_limit": "419430400", 2024-09-18T08:03:49.192 INFO:teuthology.orchestra.run.smithi007.stdout: "filestore_wbthrottle_btrfs_bytes_start_flusher": "41943040", 2024-09-18T08:03:49.192 INFO:teuthology.orchestra.run.smithi007.stdout: "filestore_wbthrottle_btrfs_inodes_hard_limit": "5000", 2024-09-18T08:03:49.192 INFO:teuthology.orchestra.run.smithi007.stdout: "filestore_wbthrottle_btrfs_inodes_start_flusher": "500", 2024-09-18T08:03:49.192 INFO:teuthology.orchestra.run.smithi007.stdout: "filestore_wbthrottle_btrfs_ios_hard_limit": "5000", 2024-09-18T08:03:49.192 INFO:teuthology.orchestra.run.smithi007.stdout: "filestore_wbthrottle_btrfs_ios_start_flusher": "500", 2024-09-18T08:03:49.192 INFO:teuthology.orchestra.run.smithi007.stdout: "filestore_wbthrottle_enable": "true", 2024-09-18T08:03:49.192 INFO:teuthology.orchestra.run.smithi007.stdout: "filestore_wbthrottle_xfs_bytes_hard_limit": "419430400", 2024-09-18T08:03:49.192 INFO:teuthology.orchestra.run.smithi007.stdout: "filestore_wbthrottle_xfs_bytes_start_flusher": "41943040", 2024-09-18T08:03:49.192 INFO:teuthology.orchestra.run.smithi007.stdout: "filestore_wbthrottle_xfs_inodes_hard_limit": "5000", 2024-09-18T08:03:49.192 INFO:teuthology.orchestra.run.smithi007.stdout: "filestore_wbthrottle_xfs_inodes_start_flusher": "500", 2024-09-18T08:03:49.193 INFO:teuthology.orchestra.run.smithi007.stdout: "filestore_wbthrottle_xfs_ios_hard_limit": "5000", 2024-09-18T08:03:49.193 INFO:teuthology.orchestra.run.smithi007.stdout: "filestore_wbthrottle_xfs_ios_start_flusher": "500", 2024-09-18T08:03:49.193 INFO:teuthology.orchestra.run.smithi007.stdout: "filestore_xfs_extsize": "false", 2024-09-18T08:03:49.193 INFO:teuthology.orchestra.run.smithi007.stdout: "filestore_zfs_snap": "false", 2024-09-18T08:03:49.193 INFO:teuthology.orchestra.run.smithi007.stdout: "fio_dir": "/tmp/fio", 2024-09-18T08:03:49.193 INFO:teuthology.orchestra.run.smithi007.stdout: "fsid": "779349ca-7593-11ef-bceb-c7b262605968", 2024-09-18T08:03:49.193 INFO:teuthology.orchestra.run.smithi007.stdout: "fuse_allow_other": "true", 2024-09-18T08:03:49.193 INFO:teuthology.orchestra.run.smithi007.stdout: "fuse_atomic_o_trunc": "true", 2024-09-18T08:03:49.193 INFO:teuthology.orchestra.run.smithi007.stdout: "fuse_big_writes": "true", 2024-09-18T08:03:49.193 INFO:teuthology.orchestra.run.smithi007.stdout: "fuse_debug": "false", 2024-09-18T08:03:49.193 INFO:teuthology.orchestra.run.smithi007.stdout: "fuse_default_permissions": "false", 2024-09-18T08:03:49.193 INFO:teuthology.orchestra.run.smithi007.stdout: "fuse_disable_pagecache": "false", 2024-09-18T08:03:49.193 INFO:teuthology.orchestra.run.smithi007.stdout: "fuse_max_write": "0", 2024-09-18T08:03:49.194 INFO:teuthology.orchestra.run.smithi007.stdout: "fuse_multithreaded": "true", 2024-09-18T08:03:49.194 INFO:teuthology.orchestra.run.smithi007.stdout: "fuse_require_active_mds": "true", 2024-09-18T08:03:49.194 INFO:teuthology.orchestra.run.smithi007.stdout: "fuse_set_user_groups": "true", 2024-09-18T08:03:49.194 INFO:teuthology.orchestra.run.smithi007.stdout: "fuse_splice_move": "true", 2024-09-18T08:03:49.194 INFO:teuthology.orchestra.run.smithi007.stdout: "fuse_splice_read": "true", 2024-09-18T08:03:49.194 INFO:teuthology.orchestra.run.smithi007.stdout: "fuse_splice_write": "true", 2024-09-18T08:03:49.194 INFO:teuthology.orchestra.run.smithi007.stdout: "fuse_syncfs_on_mksnap": "true", 2024-09-18T08:03:49.194 INFO:teuthology.orchestra.run.smithi007.stdout: "fuse_use_invalidate_cb": "true", 2024-09-18T08:03:49.194 INFO:teuthology.orchestra.run.smithi007.stdout: "gss_ktab_client_file": "/var/lib/ceph/mon.a/gss_client_mon.a.ktab", 2024-09-18T08:03:49.194 INFO:teuthology.orchestra.run.smithi007.stdout: "gss_target_name": "ceph", 2024-09-18T08:03:49.194 INFO:teuthology.orchestra.run.smithi007.stdout: "heartbeat_file": "", 2024-09-18T08:03:49.194 INFO:teuthology.orchestra.run.smithi007.stdout: "heartbeat_inject_failure": "0", 2024-09-18T08:03:49.194 INFO:teuthology.orchestra.run.smithi007.stdout: "heartbeat_interval": "5", 2024-09-18T08:03:49.195 INFO:teuthology.orchestra.run.smithi007.stdout: "host": "smithi007", 2024-09-18T08:03:49.195 INFO:teuthology.orchestra.run.smithi007.stdout: "immutable_object_cache_client_dedicated_thread_num": "2", 2024-09-18T08:03:49.195 INFO:teuthology.orchestra.run.smithi007.stdout: "immutable_object_cache_max_inflight_ops": "128", 2024-09-18T08:03:49.195 INFO:teuthology.orchestra.run.smithi007.stdout: "immutable_object_cache_max_size": "1073741824", 2024-09-18T08:03:49.195 INFO:teuthology.orchestra.run.smithi007.stdout: "immutable_object_cache_path": "/tmp/ceph_immutable_object_cache", 2024-09-18T08:03:49.195 INFO:teuthology.orchestra.run.smithi007.stdout: "immutable_object_cache_qos_bps_burst": "0", 2024-09-18T08:03:49.195 INFO:teuthology.orchestra.run.smithi007.stdout: "immutable_object_cache_qos_bps_burst_seconds": "1", 2024-09-18T08:03:49.195 INFO:teuthology.orchestra.run.smithi007.stdout: "immutable_object_cache_qos_bps_limit": "0", 2024-09-18T08:03:49.195 INFO:teuthology.orchestra.run.smithi007.stdout: "immutable_object_cache_qos_iops_burst": "0", 2024-09-18T08:03:49.195 INFO:teuthology.orchestra.run.smithi007.stdout: "immutable_object_cache_qos_iops_burst_seconds": "1", 2024-09-18T08:03:49.195 INFO:teuthology.orchestra.run.smithi007.stdout: "immutable_object_cache_qos_iops_limit": "0", 2024-09-18T08:03:49.195 INFO:teuthology.orchestra.run.smithi007.stdout: "immutable_object_cache_qos_schedule_tick_min": "50", 2024-09-18T08:03:49.195 INFO:teuthology.orchestra.run.smithi007.stdout: "immutable_object_cache_sock": "/var/run/ceph/immutable_object_cache_sock", 2024-09-18T08:03:49.195 INFO:teuthology.orchestra.run.smithi007.stdout: "immutable_object_cache_watermark": "0.900000", 2024-09-18T08:03:49.196 INFO:teuthology.orchestra.run.smithi007.stdout: "inject_early_sigterm": "false", 2024-09-18T08:03:49.196 INFO:teuthology.orchestra.run.smithi007.stdout: "jaeger_agent_port": "6799", 2024-09-18T08:03:49.196 INFO:teuthology.orchestra.run.smithi007.stdout: "jaeger_tracing_enable": "false", 2024-09-18T08:03:49.196 INFO:teuthology.orchestra.run.smithi007.stdout: "journal_aio": "true", 2024-09-18T08:03:49.196 INFO:teuthology.orchestra.run.smithi007.stdout: "journal_align_min_size": "65536", 2024-09-18T08:03:49.196 INFO:teuthology.orchestra.run.smithi007.stdout: "journal_block_align": "true", 2024-09-18T08:03:49.196 INFO:teuthology.orchestra.run.smithi007.stdout: "journal_block_size": "4096", 2024-09-18T08:03:49.196 INFO:teuthology.orchestra.run.smithi007.stdout: "journal_dio": "true", 2024-09-18T08:03:49.196 INFO:teuthology.orchestra.run.smithi007.stdout: "journal_discard": "false", 2024-09-18T08:03:49.196 INFO:teuthology.orchestra.run.smithi007.stdout: "journal_force_aio": "false", 2024-09-18T08:03:49.196 INFO:teuthology.orchestra.run.smithi007.stdout: "journal_ignore_corruption": "false", 2024-09-18T08:03:49.196 INFO:teuthology.orchestra.run.smithi007.stdout: "journal_max_write_bytes": "10485760", 2024-09-18T08:03:49.196 INFO:teuthology.orchestra.run.smithi007.stdout: "journal_max_write_entries": "100", 2024-09-18T08:03:49.197 INFO:teuthology.orchestra.run.smithi007.stdout: "journal_replay_from": "0", 2024-09-18T08:03:49.197 INFO:teuthology.orchestra.run.smithi007.stdout: "journal_throttle_high_multiple": "0.000000", 2024-09-18T08:03:49.197 INFO:teuthology.orchestra.run.smithi007.stdout: "journal_throttle_high_threshhold": "0.900000", 2024-09-18T08:03:49.197 INFO:teuthology.orchestra.run.smithi007.stdout: "journal_throttle_low_threshhold": "0.600000", 2024-09-18T08:03:49.197 INFO:teuthology.orchestra.run.smithi007.stdout: "journal_throttle_max_multiple": "0.000000", 2024-09-18T08:03:49.197 INFO:teuthology.orchestra.run.smithi007.stdout: "journal_write_header_frequency": "0", 2024-09-18T08:03:49.197 INFO:teuthology.orchestra.run.smithi007.stdout: "journal_zero_on_create": "false", 2024-09-18T08:03:49.197 INFO:teuthology.orchestra.run.smithi007.stdout: "journaler_prefetch_periods": "10", 2024-09-18T08:03:49.197 INFO:teuthology.orchestra.run.smithi007.stdout: "journaler_prezero_periods": "5", 2024-09-18T08:03:49.197 INFO:teuthology.orchestra.run.smithi007.stdout: "journaler_write_head_interval": "15", 2024-09-18T08:03:49.197 INFO:teuthology.orchestra.run.smithi007.stdout: "key": "", 2024-09-18T08:03:49.197 INFO:teuthology.orchestra.run.smithi007.stdout: "keyfile": "", 2024-09-18T08:03:49.197 INFO:teuthology.orchestra.run.smithi007.stdout: "keyring": "/var/lib/ceph/mon/ceph-a/keyring", 2024-09-18T08:03:49.198 INFO:teuthology.orchestra.run.smithi007.stdout: "kstore_backend": "rocksdb", 2024-09-18T08:03:49.198 INFO:teuthology.orchestra.run.smithi007.stdout: "kstore_default_stripe_size": "65536", 2024-09-18T08:03:49.198 INFO:teuthology.orchestra.run.smithi007.stdout: "kstore_fsck_on_mount": "false", 2024-09-18T08:03:49.198 INFO:teuthology.orchestra.run.smithi007.stdout: "kstore_fsck_on_mount_deep": "true", 2024-09-18T08:03:49.198 INFO:teuthology.orchestra.run.smithi007.stdout: "kstore_max_bytes": "67108864", 2024-09-18T08:03:49.198 INFO:teuthology.orchestra.run.smithi007.stdout: "kstore_max_ops": "512", 2024-09-18T08:03:49.198 INFO:teuthology.orchestra.run.smithi007.stdout: "kstore_nid_prealloc": "1024", 2024-09-18T08:03:49.198 INFO:teuthology.orchestra.run.smithi007.stdout: "kstore_onode_map_size": "1024", 2024-09-18T08:03:49.198 INFO:teuthology.orchestra.run.smithi007.stdout: "kstore_rocksdb_options": "compression=kNoCompression", 2024-09-18T08:03:49.198 INFO:teuthology.orchestra.run.smithi007.stdout: "kstore_sync_submit_transaction": "false", 2024-09-18T08:03:49.198 INFO:teuthology.orchestra.run.smithi007.stdout: "kstore_sync_transaction": "false", 2024-09-18T08:03:49.198 INFO:teuthology.orchestra.run.smithi007.stdout: "librados_thread_count": "2", 2024-09-18T08:03:49.199 INFO:teuthology.orchestra.run.smithi007.stdout: "lockdep": "false", 2024-09-18T08:03:49.199 INFO:teuthology.orchestra.run.smithi007.stdout: "lockdep_force_backtrace": "false", 2024-09-18T08:03:49.199 INFO:teuthology.orchestra.run.smithi007.stdout: "log_coarse_timestamps": "true", 2024-09-18T08:03:49.199 INFO:teuthology.orchestra.run.smithi007.stdout: "log_file": "/var/log/ceph/ceph-mon.a.log", 2024-09-18T08:03:49.199 INFO:teuthology.orchestra.run.smithi007.stdout: "log_flush_on_exit": "false", 2024-09-18T08:03:49.199 INFO:teuthology.orchestra.run.smithi007.stdout: "log_graylog_host": "127.0.0.1", 2024-09-18T08:03:49.199 INFO:teuthology.orchestra.run.smithi007.stdout: "log_graylog_port": "12201", 2024-09-18T08:03:49.199 INFO:teuthology.orchestra.run.smithi007.stdout: "log_max_new": "1000", 2024-09-18T08:03:49.199 INFO:teuthology.orchestra.run.smithi007.stdout: "log_max_recent": "10000", 2024-09-18T08:03:49.199 INFO:teuthology.orchestra.run.smithi007.stdout: "log_stderr_prefix": "", 2024-09-18T08:03:49.199 INFO:teuthology.orchestra.run.smithi007.stdout: "log_stop_at_utilization": "0.970000", 2024-09-18T08:03:49.199 INFO:teuthology.orchestra.run.smithi007.stdout: "log_to_file": "true", 2024-09-18T08:03:49.200 INFO:teuthology.orchestra.run.smithi007.stdout: "log_to_graylog": "false", 2024-09-18T08:03:49.200 INFO:teuthology.orchestra.run.smithi007.stdout: "log_to_journald": "false", 2024-09-18T08:03:49.200 INFO:teuthology.orchestra.run.smithi007.stdout: "log_to_stderr": "false", 2024-09-18T08:03:49.200 INFO:teuthology.orchestra.run.smithi007.stdout: "log_to_syslog": "false", 2024-09-18T08:03:49.200 INFO:teuthology.orchestra.run.smithi007.stdout: "max_rotating_auth_attempts": "10", 2024-09-18T08:03:49.200 INFO:teuthology.orchestra.run.smithi007.stdout: "mds_abort_on_newly_corrupt_dentry": "true", 2024-09-18T08:03:49.200 INFO:teuthology.orchestra.run.smithi007.stdout: "mds_action_on_write_error": "1", 2024-09-18T08:03:49.200 INFO:teuthology.orchestra.run.smithi007.stdout: "mds_alternate_name_max": "8192", 2024-09-18T08:03:49.200 INFO:teuthology.orchestra.run.smithi007.stdout: "mds_asio_thread_count": "2", 2024-09-18T08:03:49.200 INFO:teuthology.orchestra.run.smithi007.stdout: "mds_bal_export_pin": "true", 2024-09-18T08:03:49.200 INFO:teuthology.orchestra.run.smithi007.stdout: "mds_bal_fragment_dirs": "true", 2024-09-18T08:03:49.200 INFO:teuthology.orchestra.run.smithi007.stdout: "mds_bal_fragment_fast_factor": "1.500000", 2024-09-18T08:03:49.200 INFO:teuthology.orchestra.run.smithi007.stdout: "mds_bal_fragment_interval": "5", 2024-09-18T08:03:49.201 INFO:teuthology.orchestra.run.smithi007.stdout: "mds_bal_fragment_size_max": "100000", 2024-09-18T08:03:49.201 INFO:teuthology.orchestra.run.smithi007.stdout: "mds_bal_idle_threshold": "0.000000", 2024-09-18T08:03:49.201 INFO:teuthology.orchestra.run.smithi007.stdout: "mds_bal_interval": "10", 2024-09-18T08:03:49.201 INFO:teuthology.orchestra.run.smithi007.stdout: "mds_bal_max": "-1", 2024-09-18T08:03:49.201 INFO:teuthology.orchestra.run.smithi007.stdout: "mds_bal_max_until": "-1", 2024-09-18T08:03:49.201 INFO:teuthology.orchestra.run.smithi007.stdout: "mds_bal_merge_size": "50", 2024-09-18T08:03:49.201 INFO:teuthology.orchestra.run.smithi007.stdout: "mds_bal_midchunk": "0.300000", 2024-09-18T08:03:49.201 INFO:teuthology.orchestra.run.smithi007.stdout: "mds_bal_min_rebalance": "0.100000", 2024-09-18T08:03:49.201 INFO:teuthology.orchestra.run.smithi007.stdout: "mds_bal_min_start": "0.200000", 2024-09-18T08:03:49.202 INFO:teuthology.orchestra.run.smithi007.stdout: "mds_bal_minchunk": "0.001000", 2024-09-18T08:03:49.202 INFO:teuthology.orchestra.run.smithi007.stdout: "mds_bal_mode": "0", 2024-09-18T08:03:49.202 INFO:teuthology.orchestra.run.smithi007.stdout: "mds_bal_need_max": "1.200000", 2024-09-18T08:03:49.202 INFO:teuthology.orchestra.run.smithi007.stdout: "mds_bal_need_min": "0.800000", 2024-09-18T08:03:49.202 INFO:teuthology.orchestra.run.smithi007.stdout: "mds_bal_overload_epochs": "2", 2024-09-18T08:03:49.202 INFO:teuthology.orchestra.run.smithi007.stdout: "mds_bal_replicate_threshold": "8000.000000", 2024-09-18T08:03:49.202 INFO:teuthology.orchestra.run.smithi007.stdout: "mds_bal_sample_interval": "3.000000", 2024-09-18T08:03:49.202 INFO:teuthology.orchestra.run.smithi007.stdout: "mds_bal_split_bits": "3", 2024-09-18T08:03:49.202 INFO:teuthology.orchestra.run.smithi007.stdout: "mds_bal_split_rd": "25000.000000", 2024-09-18T08:03:49.202 INFO:teuthology.orchestra.run.smithi007.stdout: "mds_bal_split_size": "10000", 2024-09-18T08:03:49.202 INFO:teuthology.orchestra.run.smithi007.stdout: "mds_bal_split_wr": "10000.000000", 2024-09-18T08:03:49.202 INFO:teuthology.orchestra.run.smithi007.stdout: "mds_bal_target_decay": "10.000000", 2024-09-18T08:03:49.202 INFO:teuthology.orchestra.run.smithi007.stdout: "mds_bal_unreplicate_threshold": "0.000000", 2024-09-18T08:03:49.203 INFO:teuthology.orchestra.run.smithi007.stdout: "mds_beacon_grace": "15.000000", 2024-09-18T08:03:49.203 INFO:teuthology.orchestra.run.smithi007.stdout: "mds_beacon_interval": "4.000000", 2024-09-18T08:03:49.203 INFO:teuthology.orchestra.run.smithi007.stdout: "mds_beacon_mon_down_grace": "60", 2024-09-18T08:03:49.203 INFO:teuthology.orchestra.run.smithi007.stdout: "mds_cache_memory_limit": "4294967296", 2024-09-18T08:03:49.203 INFO:teuthology.orchestra.run.smithi007.stdout: "mds_cache_mid": "0.700000", 2024-09-18T08:03:49.203 INFO:teuthology.orchestra.run.smithi007.stdout: "mds_cache_quiesce_decay_rate": "1.000000", 2024-09-18T08:03:49.203 INFO:teuthology.orchestra.run.smithi007.stdout: "mds_cache_quiesce_delay": "0", 2024-09-18T08:03:49.203 INFO:teuthology.orchestra.run.smithi007.stdout: "mds_cache_quiesce_sleep": "200", 2024-09-18T08:03:49.203 INFO:teuthology.orchestra.run.smithi007.stdout: "mds_cache_quiesce_splitauth": "true", 2024-09-18T08:03:49.203 INFO:teuthology.orchestra.run.smithi007.stdout: "mds_cache_quiesce_threshold": "524288", 2024-09-18T08:03:49.204 INFO:teuthology.orchestra.run.smithi007.stdout: "mds_cache_release_free_interval": "10", 2024-09-18T08:03:49.204 INFO:teuthology.orchestra.run.smithi007.stdout: "mds_cache_reservation": "0.050000", 2024-09-18T08:03:49.204 INFO:teuthology.orchestra.run.smithi007.stdout: "mds_cache_trim_decay_rate": "1.000000", 2024-09-18T08:03:49.204 INFO:teuthology.orchestra.run.smithi007.stdout: "mds_cache_trim_interval": "1", 2024-09-18T08:03:49.204 INFO:teuthology.orchestra.run.smithi007.stdout: "mds_cache_trim_threshold": "262144", 2024-09-18T08:03:49.205 INFO:teuthology.orchestra.run.smithi007.stdout: "mds_cap_acquisition_throttle_retry_request_timeout": "0.500000", 2024-09-18T08:03:49.205 INFO:teuthology.orchestra.run.smithi007.stdout: "mds_cap_revoke_eviction_timeout": "0.000000", 2024-09-18T08:03:49.205 INFO:teuthology.orchestra.run.smithi007.stdout: "mds_client_delegate_inos_pct": "50", 2024-09-18T08:03:49.205 INFO:teuthology.orchestra.run.smithi007.stdout: "mds_client_prealloc_inos": "1000", 2024-09-18T08:03:49.205 INFO:teuthology.orchestra.run.smithi007.stdout: "mds_client_writeable_range_max_inc_objs": "1024", 2024-09-18T08:03:49.205 INFO:teuthology.orchestra.run.smithi007.stdout: "mds_connect_bootstrapping": "false", 2024-09-18T08:03:49.205 INFO:teuthology.orchestra.run.smithi007.stdout: "mds_damage_table_max_entries": "10000", 2024-09-18T08:03:49.205 INFO:teuthology.orchestra.run.smithi007.stdout: "mds_data": "/var/lib/ceph/mds/ceph-a", 2024-09-18T08:03:49.205 INFO:teuthology.orchestra.run.smithi007.stdout: "mds_debug_auth_pins": "false", 2024-09-18T08:03:49.206 INFO:teuthology.orchestra.run.smithi007.stdout: "mds_debug_frag": "false", 2024-09-18T08:03:49.206 INFO:teuthology.orchestra.run.smithi007.stdout: "mds_debug_scatterstat": "false", 2024-09-18T08:03:49.206 INFO:teuthology.orchestra.run.smithi007.stdout: "mds_debug_subtrees": "false", 2024-09-18T08:03:49.206 INFO:teuthology.orchestra.run.smithi007.stdout: "mds_decay_halflife": "5.000000", 2024-09-18T08:03:49.206 INFO:teuthology.orchestra.run.smithi007.stdout: "mds_default_dir_hash": "2", 2024-09-18T08:03:49.206 INFO:teuthology.orchestra.run.smithi007.stdout: "mds_defer_session_stale": "true", 2024-09-18T08:03:49.206 INFO:teuthology.orchestra.run.smithi007.stdout: "mds_deny_all_reconnect": "false", 2024-09-18T08:03:49.206 INFO:teuthology.orchestra.run.smithi007.stdout: "mds_dir_keys_per_op": "16384", 2024-09-18T08:03:49.206 INFO:teuthology.orchestra.run.smithi007.stdout: "mds_dir_max_commit_size": "10", 2024-09-18T08:03:49.206 INFO:teuthology.orchestra.run.smithi007.stdout: "mds_dir_max_entries": "0", 2024-09-18T08:03:49.206 INFO:teuthology.orchestra.run.smithi007.stdout: "mds_dir_prefetch": "true", 2024-09-18T08:03:49.206 INFO:teuthology.orchestra.run.smithi007.stdout: "mds_dirstat_min_interval": "1.000000", 2024-09-18T08:03:49.207 INFO:teuthology.orchestra.run.smithi007.stdout: "mds_dump_cache_after_rejoin": "false", 2024-09-18T08:03:49.207 INFO:teuthology.orchestra.run.smithi007.stdout: "mds_dump_cache_on_map": "false", 2024-09-18T08:03:49.207 INFO:teuthology.orchestra.run.smithi007.stdout: "mds_dump_cache_threshold_file": "0", 2024-09-18T08:03:49.207 INFO:teuthology.orchestra.run.smithi007.stdout: "mds_dump_cache_threshold_formatter": "1073741824", 2024-09-18T08:03:49.207 INFO:teuthology.orchestra.run.smithi007.stdout: "mds_early_reply": "true", 2024-09-18T08:03:49.207 INFO:teuthology.orchestra.run.smithi007.stdout: "mds_enable_op_tracker": "true", 2024-09-18T08:03:49.207 INFO:teuthology.orchestra.run.smithi007.stdout: "mds_enforce_unique_name": "true", 2024-09-18T08:03:49.207 INFO:teuthology.orchestra.run.smithi007.stdout: "mds_export_ephemeral_distributed": "true", 2024-09-18T08:03:49.207 INFO:teuthology.orchestra.run.smithi007.stdout: "mds_export_ephemeral_distributed_factor": "2.000000", 2024-09-18T08:03:49.207 INFO:teuthology.orchestra.run.smithi007.stdout: "mds_export_ephemeral_random": "true", 2024-09-18T08:03:49.207 INFO:teuthology.orchestra.run.smithi007.stdout: "mds_export_ephemeral_random_max": "0.010000", 2024-09-18T08:03:49.207 INFO:teuthology.orchestra.run.smithi007.stdout: "mds_extraordinary_events_dump_interval": "0", 2024-09-18T08:03:49.208 INFO:teuthology.orchestra.run.smithi007.stdout: "mds_forward_all_requests_to_auth": "false", 2024-09-18T08:03:49.208 INFO:teuthology.orchestra.run.smithi007.stdout: "mds_freeze_tree_timeout": "30.000000", 2024-09-18T08:03:49.208 INFO:teuthology.orchestra.run.smithi007.stdout: "mds_fscrypt_last_block_max_size": "4096", 2024-09-18T08:03:49.208 INFO:teuthology.orchestra.run.smithi007.stdout: "mds_go_bad_corrupt_dentry": "true", 2024-09-18T08:03:49.208 INFO:teuthology.orchestra.run.smithi007.stdout: "mds_hack_allow_loading_invalid_metadata": "false", 2024-09-18T08:03:49.208 INFO:teuthology.orchestra.run.smithi007.stdout: "mds_health_cache_threshold": "1.500000", 2024-09-18T08:03:49.208 INFO:teuthology.orchestra.run.smithi007.stdout: "mds_health_summarize_threshold": "10", 2024-09-18T08:03:49.208 INFO:teuthology.orchestra.run.smithi007.stdout: "mds_heartbeat_grace": "15.000000", 2024-09-18T08:03:49.208 INFO:teuthology.orchestra.run.smithi007.stdout: "mds_heartbeat_reset_grace": "1000", 2024-09-18T08:03:49.208 INFO:teuthology.orchestra.run.smithi007.stdout: "mds_inject_health_dummy": "false", 2024-09-18T08:03:49.208 INFO:teuthology.orchestra.run.smithi007.stdout: "mds_inject_journal_corrupt_dentry_first": "0.000000", 2024-09-18T08:03:49.208 INFO:teuthology.orchestra.run.smithi007.stdout: "mds_inject_migrator_session_race": "false", 2024-09-18T08:03:49.208 INFO:teuthology.orchestra.run.smithi007.stdout: "mds_inject_rename_corrupt_dentry_first": "0.000000", 2024-09-18T08:03:49.209 INFO:teuthology.orchestra.run.smithi007.stdout: "mds_inject_skip_replaying_inotable": "false", 2024-09-18T08:03:49.209 INFO:teuthology.orchestra.run.smithi007.stdout: "mds_inject_traceless_reply_probability": "0.000000", 2024-09-18T08:03:49.209 INFO:teuthology.orchestra.run.smithi007.stdout: "mds_join_fs": "", 2024-09-18T08:03:49.209 INFO:teuthology.orchestra.run.smithi007.stdout: "mds_journal_format": "1", 2024-09-18T08:03:49.209 INFO:teuthology.orchestra.run.smithi007.stdout: "mds_kill_after_journal_logs_flushed": "false", 2024-09-18T08:03:49.209 INFO:teuthology.orchestra.run.smithi007.stdout: "mds_kill_create_at": "0", 2024-09-18T08:03:49.209 INFO:teuthology.orchestra.run.smithi007.stdout: "mds_kill_dirfrag_at": "0", 2024-09-18T08:03:49.209 INFO:teuthology.orchestra.run.smithi007.stdout: "mds_kill_export_at": "0", 2024-09-18T08:03:49.209 INFO:teuthology.orchestra.run.smithi007.stdout: "mds_kill_import_at": "0", 2024-09-18T08:03:49.209 INFO:teuthology.orchestra.run.smithi007.stdout: "mds_kill_journal_at": "0", 2024-09-18T08:03:49.209 INFO:teuthology.orchestra.run.smithi007.stdout: "mds_kill_journal_expire_at": "0", 2024-09-18T08:03:49.209 INFO:teuthology.orchestra.run.smithi007.stdout: "mds_kill_journal_replay_at": "0", 2024-09-18T08:03:49.209 INFO:teuthology.orchestra.run.smithi007.stdout: "mds_kill_link_at": "0", 2024-09-18T08:03:49.210 INFO:teuthology.orchestra.run.smithi007.stdout: "mds_kill_mdstable_at": "0", 2024-09-18T08:03:49.210 INFO:teuthology.orchestra.run.smithi007.stdout: "mds_kill_openc_at": "0", 2024-09-18T08:03:49.210 INFO:teuthology.orchestra.run.smithi007.stdout: "mds_kill_rename_at": "0", 2024-09-18T08:03:49.210 INFO:teuthology.orchestra.run.smithi007.stdout: "mds_kill_shutdown_at": "0", 2024-09-18T08:03:49.210 INFO:teuthology.orchestra.run.smithi007.stdout: "mds_log_event_large_threshold": "524288", 2024-09-18T08:03:49.210 INFO:teuthology.orchestra.run.smithi007.stdout: "mds_log_events_per_segment": "1024", 2024-09-18T08:03:49.210 INFO:teuthology.orchestra.run.smithi007.stdout: "mds_log_major_segment_event_ratio": "12", 2024-09-18T08:03:49.210 INFO:teuthology.orchestra.run.smithi007.stdout: "mds_log_max_events": "-1", 2024-09-18T08:03:49.211 INFO:teuthology.orchestra.run.smithi007.stdout: "mds_log_max_segments": "128", 2024-09-18T08:03:49.211 INFO:teuthology.orchestra.run.smithi007.stdout: "mds_log_pause": "false", 2024-09-18T08:03:49.211 INFO:teuthology.orchestra.run.smithi007.stdout: "mds_log_segment_size": "0", 2024-09-18T08:03:49.211 INFO:teuthology.orchestra.run.smithi007.stdout: "mds_log_skip_corrupt_events": "false", 2024-09-18T08:03:49.211 INFO:teuthology.orchestra.run.smithi007.stdout: "mds_log_skip_unbounded_events": "false", 2024-09-18T08:03:49.211 INFO:teuthology.orchestra.run.smithi007.stdout: "mds_log_trim_decay_rate": "1.000000", 2024-09-18T08:03:49.211 INFO:teuthology.orchestra.run.smithi007.stdout: "mds_log_trim_threshold": "128", 2024-09-18T08:03:49.211 INFO:teuthology.orchestra.run.smithi007.stdout: "mds_log_trim_upkeep_interval": "1000", 2024-09-18T08:03:49.211 INFO:teuthology.orchestra.run.smithi007.stdout: "mds_log_warn_factor": "2.000000", 2024-09-18T08:03:49.211 INFO:teuthology.orchestra.run.smithi007.stdout: "mds_max_caps_per_client": "1048576", 2024-09-18T08:03:49.211 INFO:teuthology.orchestra.run.smithi007.stdout: "mds_max_completed_flushes": "100000", 2024-09-18T08:03:49.211 INFO:teuthology.orchestra.run.smithi007.stdout: "mds_max_completed_requests": "100000", 2024-09-18T08:03:49.211 INFO:teuthology.orchestra.run.smithi007.stdout: "mds_max_export_size": "20971520", 2024-09-18T08:03:49.211 INFO:teuthology.orchestra.run.smithi007.stdout: "mds_max_file_recover": "32", 2024-09-18T08:03:49.212 INFO:teuthology.orchestra.run.smithi007.stdout: "mds_max_purge_files": "64", 2024-09-18T08:03:49.212 INFO:teuthology.orchestra.run.smithi007.stdout: "mds_max_purge_ops": "8192", 2024-09-18T08:03:49.212 INFO:teuthology.orchestra.run.smithi007.stdout: "mds_max_purge_ops_per_pg": "0.500000", 2024-09-18T08:03:49.212 INFO:teuthology.orchestra.run.smithi007.stdout: "mds_max_scrub_ops_in_progress": "5", 2024-09-18T08:03:49.212 INFO:teuthology.orchestra.run.smithi007.stdout: "mds_max_snaps_per_dir": "100", 2024-09-18T08:03:49.212 INFO:teuthology.orchestra.run.smithi007.stdout: "mds_metrics_update_interval": "2", 2024-09-18T08:03:49.212 INFO:teuthology.orchestra.run.smithi007.stdout: "mds_min_caps_per_client": "100", 2024-09-18T08:03:49.212 INFO:teuthology.orchestra.run.smithi007.stdout: "mds_min_caps_working_set": "10000", 2024-09-18T08:03:49.212 INFO:teuthology.orchestra.run.smithi007.stdout: "mds_mon_shutdown_timeout": "5.000000", 2024-09-18T08:03:49.212 INFO:teuthology.orchestra.run.smithi007.stdout: "mds_numa_node": "-1", 2024-09-18T08:03:49.212 INFO:teuthology.orchestra.run.smithi007.stdout: "mds_oft_prefetch_dirfrags": "false", 2024-09-18T08:03:49.212 INFO:teuthology.orchestra.run.smithi007.stdout: "mds_op_complaint_time": "30.000000", 2024-09-18T08:03:49.212 INFO:teuthology.orchestra.run.smithi007.stdout: "mds_op_history_duration": "600", 2024-09-18T08:03:49.213 INFO:teuthology.orchestra.run.smithi007.stdout: "mds_op_history_size": "20", 2024-09-18T08:03:49.213 INFO:teuthology.orchestra.run.smithi007.stdout: "mds_op_history_slow_op_size": "20", 2024-09-18T08:03:49.213 INFO:teuthology.orchestra.run.smithi007.stdout: "mds_op_history_slow_op_threshold": "10", 2024-09-18T08:03:49.213 INFO:teuthology.orchestra.run.smithi007.stdout: "mds_op_log_threshold": "5", 2024-09-18T08:03:49.213 INFO:teuthology.orchestra.run.smithi007.stdout: "mds_ping_grace": "15", 2024-09-18T08:03:49.213 INFO:teuthology.orchestra.run.smithi007.stdout: "mds_ping_interval": "5", 2024-09-18T08:03:49.213 INFO:teuthology.orchestra.run.smithi007.stdout: "mds_purge_queue_busy_flush_period": "1.000000", 2024-09-18T08:03:49.213 INFO:teuthology.orchestra.run.smithi007.stdout: "mds_recall_global_max_decay_threshold": "131072", 2024-09-18T08:03:49.213 INFO:teuthology.orchestra.run.smithi007.stdout: "mds_recall_max_caps": "30000", 2024-09-18T08:03:49.213 INFO:teuthology.orchestra.run.smithi007.stdout: "mds_recall_max_decay_rate": "1.500000", 2024-09-18T08:03:49.213 INFO:teuthology.orchestra.run.smithi007.stdout: "mds_recall_max_decay_threshold": "131072", 2024-09-18T08:03:49.213 INFO:teuthology.orchestra.run.smithi007.stdout: "mds_recall_warning_decay_rate": "60.000000", 2024-09-18T08:03:49.213 INFO:teuthology.orchestra.run.smithi007.stdout: "mds_recall_warning_threshold": "262144", 2024-09-18T08:03:49.214 INFO:teuthology.orchestra.run.smithi007.stdout: "mds_reconnect_timeout": "45.000000", 2024-09-18T08:03:49.214 INFO:teuthology.orchestra.run.smithi007.stdout: "mds_replay_interval": "1.000000", 2024-09-18T08:03:49.214 INFO:teuthology.orchestra.run.smithi007.stdout: "mds_replay_unsafe_with_closed_session": "false", 2024-09-18T08:03:49.214 INFO:teuthology.orchestra.run.smithi007.stdout: "mds_request_load_average_decay_rate": "60.000000", 2024-09-18T08:03:49.214 INFO:teuthology.orchestra.run.smithi007.stdout: "mds_root_ino_gid": "0", 2024-09-18T08:03:49.214 INFO:teuthology.orchestra.run.smithi007.stdout: "mds_root_ino_uid": "0", 2024-09-18T08:03:49.214 INFO:teuthology.orchestra.run.smithi007.stdout: "mds_scatter_nudge_interval": "5.000000", 2024-09-18T08:03:49.214 INFO:teuthology.orchestra.run.smithi007.stdout: "mds_scrub_stats_review_period": "1", 2024-09-18T08:03:49.214 INFO:teuthology.orchestra.run.smithi007.stdout: "mds_server_dispatch_client_request_delay": "0", 2024-09-18T08:03:49.214 INFO:teuthology.orchestra.run.smithi007.stdout: "mds_server_dispatch_killpoint_random": "0.000000", 2024-09-18T08:03:49.214 INFO:teuthology.orchestra.run.smithi007.stdout: "mds_session_blocklist_on_evict": "true", 2024-09-18T08:03:49.214 INFO:teuthology.orchestra.run.smithi007.stdout: "mds_session_blocklist_on_timeout": "true", 2024-09-18T08:03:49.214 INFO:teuthology.orchestra.run.smithi007.stdout: "mds_session_cache_liveness_decay_rate": "300.000000", 2024-09-18T08:03:49.214 INFO:teuthology.orchestra.run.smithi007.stdout: "mds_session_cache_liveness_magnitude": "10", 2024-09-18T08:03:49.215 INFO:teuthology.orchestra.run.smithi007.stdout: "mds_session_cap_acquisition_decay_rate": "30.000000", 2024-09-18T08:03:49.215 INFO:teuthology.orchestra.run.smithi007.stdout: "mds_session_cap_acquisition_throttle": "100000", 2024-09-18T08:03:49.215 INFO:teuthology.orchestra.run.smithi007.stdout: "mds_session_max_caps_throttle_ratio": "1.100000", 2024-09-18T08:03:49.215 INFO:teuthology.orchestra.run.smithi007.stdout: "mds_session_metadata_threshold": "16777216", 2024-09-18T08:03:49.215 INFO:teuthology.orchestra.run.smithi007.stdout: "mds_sessionmap_keys_per_op": "1024", 2024-09-18T08:03:49.215 INFO:teuthology.orchestra.run.smithi007.stdout: "mds_shutdown_check": "0", 2024-09-18T08:03:49.215 INFO:teuthology.orchestra.run.smithi007.stdout: "mds_skip_ino": "0", 2024-09-18T08:03:49.215 INFO:teuthology.orchestra.run.smithi007.stdout: "mds_sleep_rank_change": "0.000000", 2024-09-18T08:03:49.215 INFO:teuthology.orchestra.run.smithi007.stdout: "mds_snap_max_uid": "4294967294", 2024-09-18T08:03:49.215 INFO:teuthology.orchestra.run.smithi007.stdout: "mds_snap_min_uid": "0", 2024-09-18T08:03:49.215 INFO:teuthology.orchestra.run.smithi007.stdout: "mds_snap_rstat": "false", 2024-09-18T08:03:49.215 INFO:teuthology.orchestra.run.smithi007.stdout: "mds_standby_replay_damaged": "false", 2024-09-18T08:03:49.216 INFO:teuthology.orchestra.run.smithi007.stdout: "mds_symlink_recovery": "true", 2024-09-18T08:03:49.216 INFO:teuthology.orchestra.run.smithi007.stdout: "mds_task_status_update_interval": "2.000000", 2024-09-18T08:03:49.216 INFO:teuthology.orchestra.run.smithi007.stdout: "mds_thrash_exports": "0", 2024-09-18T08:03:49.216 INFO:teuthology.orchestra.run.smithi007.stdout: "mds_thrash_fragments": "0", 2024-09-18T08:03:49.216 INFO:teuthology.orchestra.run.smithi007.stdout: "mds_tick_interval": "5.000000", 2024-09-18T08:03:49.216 INFO:teuthology.orchestra.run.smithi007.stdout: "mds_valgrind_exit": "false", 2024-09-18T08:03:49.216 INFO:teuthology.orchestra.run.smithi007.stdout: "mds_verify_backtrace": "1", 2024-09-18T08:03:49.217 INFO:teuthology.orchestra.run.smithi007.stdout: "mds_verify_scatter": "false", 2024-09-18T08:03:49.217 INFO:teuthology.orchestra.run.smithi007.stdout: "mds_wipe_ino_prealloc": "false", 2024-09-18T08:03:49.217 INFO:teuthology.orchestra.run.smithi007.stdout: "mds_wipe_sessions": "false", 2024-09-18T08:03:49.217 INFO:teuthology.orchestra.run.smithi007.stdout: "mempool_debug": "false", 2024-09-18T08:03:49.217 INFO:teuthology.orchestra.run.smithi007.stdout: "memstore_debug_omit_block_device_write": "false", 2024-09-18T08:03:49.217 INFO:teuthology.orchestra.run.smithi007.stdout: "memstore_device_bytes": "1073741824", 2024-09-18T08:03:49.217 INFO:teuthology.orchestra.run.smithi007.stdout: "memstore_page_set": "false", 2024-09-18T08:03:49.217 INFO:teuthology.orchestra.run.smithi007.stdout: "memstore_page_size": "65536", 2024-09-18T08:03:49.217 INFO:teuthology.orchestra.run.smithi007.stdout: "mgr_client_bytes": "134217728", 2024-09-18T08:03:49.217 INFO:teuthology.orchestra.run.smithi007.stdout: "mgr_client_messages": "512", 2024-09-18T08:03:49.218 INFO:teuthology.orchestra.run.smithi007.stdout: "mgr_client_service_daemon_unregister_timeout": "1.000000", 2024-09-18T08:03:49.218 INFO:teuthology.orchestra.run.smithi007.stdout: "mgr_connect_retry_interval": "1.000000", 2024-09-18T08:03:49.218 INFO:teuthology.orchestra.run.smithi007.stdout: "mgr_data": "/var/lib/ceph/mgr/ceph-a", 2024-09-18T08:03:49.218 INFO:teuthology.orchestra.run.smithi007.stdout: "mgr_debug_aggressive_pg_num_changes": "false", 2024-09-18T08:03:49.218 INFO:teuthology.orchestra.run.smithi007.stdout: "mgr_disabled_modules": "", 2024-09-18T08:03:49.218 INFO:teuthology.orchestra.run.smithi007.stdout: "mgr_enable_op_tracker": "true", 2024-09-18T08:03:49.218 INFO:teuthology.orchestra.run.smithi007.stdout: "mgr_initial_modules": "restful iostat nfs", 2024-09-18T08:03:49.218 INFO:teuthology.orchestra.run.smithi007.stdout: "mgr_max_pg_creating": "1024", 2024-09-18T08:03:49.219 INFO:teuthology.orchestra.run.smithi007.stdout: "mgr_max_pg_num_change": "128", 2024-09-18T08:03:49.219 INFO:teuthology.orchestra.run.smithi007.stdout: "mgr_mds_bytes": "134217728", 2024-09-18T08:03:49.219 INFO:teuthology.orchestra.run.smithi007.stdout: "mgr_mds_messages": "128", 2024-09-18T08:03:49.219 INFO:teuthology.orchestra.run.smithi007.stdout: "mgr_module_path": "/usr/share/ceph/mgr", 2024-09-18T08:03:49.219 INFO:teuthology.orchestra.run.smithi007.stdout: "mgr_mon_bytes": "134217728", 2024-09-18T08:03:49.219 INFO:teuthology.orchestra.run.smithi007.stdout: "mgr_mon_messages": "128", 2024-09-18T08:03:49.219 INFO:teuthology.orchestra.run.smithi007.stdout: "mgr_num_op_tracker_shard": "32", 2024-09-18T08:03:49.219 INFO:teuthology.orchestra.run.smithi007.stdout: "mgr_op_complaint_time": "30.000000", 2024-09-18T08:03:49.219 INFO:teuthology.orchestra.run.smithi007.stdout: "mgr_op_history_duration": "600", 2024-09-18T08:03:49.219 INFO:teuthology.orchestra.run.smithi007.stdout: "mgr_op_history_size": "20", 2024-09-18T08:03:49.219 INFO:teuthology.orchestra.run.smithi007.stdout: "mgr_op_history_slow_op_size": "20", 2024-09-18T08:03:49.219 INFO:teuthology.orchestra.run.smithi007.stdout: "mgr_op_history_slow_op_threshold": "10.000000", 2024-09-18T08:03:49.219 INFO:teuthology.orchestra.run.smithi007.stdout: "mgr_op_log_threshold": "5", 2024-09-18T08:03:49.220 INFO:teuthology.orchestra.run.smithi007.stdout: "mgr_osd_bytes": "536870912", 2024-09-18T08:03:49.220 INFO:teuthology.orchestra.run.smithi007.stdout: "mgr_osd_messages": "8192", 2024-09-18T08:03:49.220 INFO:teuthology.orchestra.run.smithi007.stdout: "mgr_pool": "true", 2024-09-18T08:03:49.220 INFO:teuthology.orchestra.run.smithi007.stdout: "mgr_service_beacon_grace": "60.000000", 2024-09-18T08:03:49.220 INFO:teuthology.orchestra.run.smithi007.stdout: "mgr_standby_modules": "true", 2024-09-18T08:03:49.220 INFO:teuthology.orchestra.run.smithi007.stdout: "mgr_stats_period": "5", 2024-09-18T08:03:49.220 INFO:teuthology.orchestra.run.smithi007.stdout: "mgr_stats_threshold": "5", 2024-09-18T08:03:49.220 INFO:teuthology.orchestra.run.smithi007.stdout: "mgr_tick_period": "2", 2024-09-18T08:03:49.220 INFO:teuthology.orchestra.run.smithi007.stdout: "mgr_ttl_cache_expire_seconds": "0", 2024-09-18T08:03:49.220 INFO:teuthology.orchestra.run.smithi007.stdout: "mon_accept_timeout_factor": "2.000000", 2024-09-18T08:03:49.220 INFO:teuthology.orchestra.run.smithi007.stdout: "mon_allow_pool_delete": "true", 2024-09-18T08:03:49.221 INFO:teuthology.orchestra.run.smithi007.stdout: "mon_allow_pool_size_one": "false", 2024-09-18T08:03:49.221 INFO:teuthology.orchestra.run.smithi007.stdout: "mon_auth_validate_all_caps": "true", 2024-09-18T08:03:49.221 INFO:teuthology.orchestra.run.smithi007.stdout: "mon_cache_target_full_warn_ratio": "0.660000", 2024-09-18T08:03:49.221 INFO:teuthology.orchestra.run.smithi007.stdout: "mon_clean_pg_upmaps_per_chunk": "256", 2024-09-18T08:03:49.221 INFO:teuthology.orchestra.run.smithi007.stdout: "mon_client_bytes": "104857600", 2024-09-18T08:03:49.221 INFO:teuthology.orchestra.run.smithi007.stdout: "mon_client_directed_command_retry": "2", 2024-09-18T08:03:49.221 INFO:teuthology.orchestra.run.smithi007.stdout: "mon_client_hunt_interval": "3.000000", 2024-09-18T08:03:49.221 INFO:teuthology.orchestra.run.smithi007.stdout: "mon_client_hunt_interval_backoff": "1.500000", 2024-09-18T08:03:49.221 INFO:teuthology.orchestra.run.smithi007.stdout: "mon_client_hunt_interval_max_multiple": "10.000000", 2024-09-18T08:03:49.221 INFO:teuthology.orchestra.run.smithi007.stdout: "mon_client_hunt_interval_min_multiple": "1.000000", 2024-09-18T08:03:49.221 INFO:teuthology.orchestra.run.smithi007.stdout: "mon_client_hunt_parallel": "3", 2024-09-18T08:03:49.221 INFO:teuthology.orchestra.run.smithi007.stdout: "mon_client_log_interval": "1.000000", 2024-09-18T08:03:49.221 INFO:teuthology.orchestra.run.smithi007.stdout: "mon_client_max_log_entries_per_message": "1000", 2024-09-18T08:03:49.222 INFO:teuthology.orchestra.run.smithi007.stdout: "mon_client_ping_interval": "10.000000", 2024-09-18T08:03:49.222 INFO:teuthology.orchestra.run.smithi007.stdout: "mon_client_ping_timeout": "30.000000", 2024-09-18T08:03:49.222 INFO:teuthology.orchestra.run.smithi007.stdout: "mon_client_target_rank": "-1", 2024-09-18T08:03:49.222 INFO:teuthology.orchestra.run.smithi007.stdout: "mon_clock_drift_allowed": "1.000000", 2024-09-18T08:03:49.222 INFO:teuthology.orchestra.run.smithi007.stdout: "mon_clock_drift_warn_backoff": "5.000000", 2024-09-18T08:03:49.222 INFO:teuthology.orchestra.run.smithi007.stdout: "mon_cluster_log_file": "default=/var/log/ceph/ceph.$channel.log cluster=/var/log/ceph/ceph.log", 2024-09-18T08:03:49.222 INFO:teuthology.orchestra.run.smithi007.stdout: "mon_cluster_log_level": "debug", 2024-09-18T08:03:49.222 INFO:teuthology.orchestra.run.smithi007.stdout: "mon_cluster_log_to_file": "true", 2024-09-18T08:03:49.222 INFO:teuthology.orchestra.run.smithi007.stdout: "mon_cluster_log_to_graylog": "false", 2024-09-18T08:03:49.222 INFO:teuthology.orchestra.run.smithi007.stdout: "mon_cluster_log_to_graylog_host": "127.0.0.1", 2024-09-18T08:03:49.222 INFO:teuthology.orchestra.run.smithi007.stdout: "mon_cluster_log_to_graylog_port": "12201", 2024-09-18T08:03:49.222 INFO:teuthology.orchestra.run.smithi007.stdout: "mon_cluster_log_to_journald": "true", 2024-09-18T08:03:49.222 INFO:teuthology.orchestra.run.smithi007.stdout: "mon_cluster_log_to_stderr": "false", 2024-09-18T08:03:49.223 INFO:teuthology.orchestra.run.smithi007.stdout: "mon_cluster_log_to_syslog": "default=false", 2024-09-18T08:03:49.223 INFO:teuthology.orchestra.run.smithi007.stdout: "mon_cluster_log_to_syslog_facility": "daemon", 2024-09-18T08:03:49.223 INFO:teuthology.orchestra.run.smithi007.stdout: "mon_compact_on_bootstrap": "false", 2024-09-18T08:03:49.223 INFO:teuthology.orchestra.run.smithi007.stdout: "mon_compact_on_start": "false", 2024-09-18T08:03:49.223 INFO:teuthology.orchestra.run.smithi007.stdout: "mon_compact_on_trim": "true", 2024-09-18T08:03:49.223 INFO:teuthology.orchestra.run.smithi007.stdout: "mon_con_tracker_persist_interval": "10", 2024-09-18T08:03:49.223 INFO:teuthology.orchestra.run.smithi007.stdout: "mon_con_tracker_score_halflife": "43200", 2024-09-18T08:03:49.223 INFO:teuthology.orchestra.run.smithi007.stdout: "mon_config_key_max_entry_size": "65536", 2024-09-18T08:03:49.223 INFO:teuthology.orchestra.run.smithi007.stdout: "mon_cpu_threads": "4", 2024-09-18T08:03:49.223 INFO:teuthology.orchestra.run.smithi007.stdout: "mon_crush_min_required_version": "hammer", 2024-09-18T08:03:49.223 INFO:teuthology.orchestra.run.smithi007.stdout: "mon_daemon_bytes": "419430400", 2024-09-18T08:03:49.223 INFO:teuthology.orchestra.run.smithi007.stdout: "mon_data": "/var/lib/ceph/mon/ceph-a", 2024-09-18T08:03:49.223 INFO:teuthology.orchestra.run.smithi007.stdout: "mon_data_avail_crit": "5", 2024-09-18T08:03:49.224 INFO:teuthology.orchestra.run.smithi007.stdout: "mon_data_avail_warn": "5", 2024-09-18T08:03:49.224 INFO:teuthology.orchestra.run.smithi007.stdout: "mon_data_size_warn": "16106127360", 2024-09-18T08:03:49.224 INFO:teuthology.orchestra.run.smithi007.stdout: "mon_debug_block_osdmap_trim": "false", 2024-09-18T08:03:49.224 INFO:teuthology.orchestra.run.smithi007.stdout: "mon_debug_deprecated_as_obsolete": "false", 2024-09-18T08:03:49.224 INFO:teuthology.orchestra.run.smithi007.stdout: "mon_debug_dump_json": "false", 2024-09-18T08:03:49.224 INFO:teuthology.orchestra.run.smithi007.stdout: "mon_debug_dump_location": "/var/log/ceph/ceph-mon.a.tdump", 2024-09-18T08:03:49.224 INFO:teuthology.orchestra.run.smithi007.stdout: "mon_debug_dump_transactions": "false", 2024-09-18T08:03:49.224 INFO:teuthology.orchestra.run.smithi007.stdout: "mon_debug_extra_checks": "false", 2024-09-18T08:03:49.224 INFO:teuthology.orchestra.run.smithi007.stdout: "mon_debug_no_initial_persistent_features": "false", 2024-09-18T08:03:49.224 INFO:teuthology.orchestra.run.smithi007.stdout: "mon_debug_no_require_bluestore_for_ec_overwrites": "false", 2024-09-18T08:03:49.224 INFO:teuthology.orchestra.run.smithi007.stdout: "mon_debug_no_require_reef": "false", 2024-09-18T08:03:49.225 INFO:teuthology.orchestra.run.smithi007.stdout: "mon_debug_no_require_squid": "false", 2024-09-18T08:03:49.225 INFO:teuthology.orchestra.run.smithi007.stdout: "mon_debug_unsafe_allow_tier_with_nonempty_snaps": "false", 2024-09-18T08:03:49.225 INFO:teuthology.orchestra.run.smithi007.stdout: "mon_delta_reset_interval": "10.000000", 2024-09-18T08:03:49.225 INFO:teuthology.orchestra.run.smithi007.stdout: "mon_dns_srv_name": "ceph-mon", 2024-09-18T08:03:49.225 INFO:teuthology.orchestra.run.smithi007.stdout: "mon_down_mkfs_grace": "60", 2024-09-18T08:03:49.225 INFO:teuthology.orchestra.run.smithi007.stdout: "mon_down_uptime_grace": "60", 2024-09-18T08:03:49.225 INFO:teuthology.orchestra.run.smithi007.stdout: "mon_election_default_strategy": "1", 2024-09-18T08:03:49.225 INFO:teuthology.orchestra.run.smithi007.stdout: "mon_election_timeout": "5.000000", 2024-09-18T08:03:49.225 INFO:teuthology.orchestra.run.smithi007.stdout: "mon_elector_ignore_propose_margin": "0.000500", 2024-09-18T08:03:49.225 INFO:teuthology.orchestra.run.smithi007.stdout: "mon_elector_ping_divisor": "2", 2024-09-18T08:03:49.225 INFO:teuthology.orchestra.run.smithi007.stdout: "mon_elector_ping_timeout": "2.000000", 2024-09-18T08:03:49.225 INFO:teuthology.orchestra.run.smithi007.stdout: "mon_enable_op_tracker": "true", 2024-09-18T08:03:49.226 INFO:teuthology.orchestra.run.smithi007.stdout: "mon_fake_pool_delete": "false", 2024-09-18T08:03:49.226 INFO:teuthology.orchestra.run.smithi007.stdout: "mon_force_quorum_join": "false", 2024-09-18T08:03:49.226 INFO:teuthology.orchestra.run.smithi007.stdout: "mon_fsmap_prune_threshold": "300", 2024-09-18T08:03:49.226 INFO:teuthology.orchestra.run.smithi007.stdout: "mon_globalid_prealloc": "10000", 2024-09-18T08:03:49.226 INFO:teuthology.orchestra.run.smithi007.stdout: "mon_health_detail_to_clog": "true", 2024-09-18T08:03:49.226 INFO:teuthology.orchestra.run.smithi007.stdout: "mon_health_log_update_period": "5", 2024-09-18T08:03:49.226 INFO:teuthology.orchestra.run.smithi007.stdout: "mon_health_max_detail": "50", 2024-09-18T08:03:49.226 INFO:teuthology.orchestra.run.smithi007.stdout: "mon_health_to_clog": "true", 2024-09-18T08:03:49.226 INFO:teuthology.orchestra.run.smithi007.stdout: "mon_health_to_clog_interval": "600", 2024-09-18T08:03:49.226 INFO:teuthology.orchestra.run.smithi007.stdout: "mon_health_to_clog_tick_interval": "60.000000", 2024-09-18T08:03:49.226 INFO:teuthology.orchestra.run.smithi007.stdout: "mon_host": "[v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0]", 2024-09-18T08:03:49.227 INFO:teuthology.orchestra.run.smithi007.stdout: "mon_host_override": "", 2024-09-18T08:03:49.227 INFO:teuthology.orchestra.run.smithi007.stdout: "mon_initial_members": "", 2024-09-18T08:03:49.227 INFO:teuthology.orchestra.run.smithi007.stdout: "mon_inject_pg_merge_bounce_probability": "0.000000", 2024-09-18T08:03:49.227 INFO:teuthology.orchestra.run.smithi007.stdout: "mon_inject_sync_get_chunk_delay": "0.000000", 2024-09-18T08:03:49.227 INFO:teuthology.orchestra.run.smithi007.stdout: "mon_inject_transaction_delay_max": "10.000000", 2024-09-18T08:03:49.227 INFO:teuthology.orchestra.run.smithi007.stdout: "mon_inject_transaction_delay_probability": "0.000000", 2024-09-18T08:03:49.227 INFO:teuthology.orchestra.run.smithi007.stdout: "mon_keyvaluedb": "rocksdb", 2024-09-18T08:03:49.227 INFO:teuthology.orchestra.run.smithi007.stdout: "mon_lease": "5.000000", 2024-09-18T08:03:49.227 INFO:teuthology.orchestra.run.smithi007.stdout: "mon_lease_ack_timeout_factor": "2.000000", 2024-09-18T08:03:49.227 INFO:teuthology.orchestra.run.smithi007.stdout: "mon_lease_renew_interval_factor": "0.600000", 2024-09-18T08:03:49.227 INFO:teuthology.orchestra.run.smithi007.stdout: "mon_log_full_interval": "50", 2024-09-18T08:03:49.227 INFO:teuthology.orchestra.run.smithi007.stdout: "mon_log_max": "10000", 2024-09-18T08:03:49.227 INFO:teuthology.orchestra.run.smithi007.stdout: "mon_log_max_summary": "50", 2024-09-18T08:03:49.227 INFO:teuthology.orchestra.run.smithi007.stdout: "mon_max_log_entries_per_event": "4096", 2024-09-18T08:03:49.228 INFO:teuthology.orchestra.run.smithi007.stdout: "mon_max_log_epochs": "500", 2024-09-18T08:03:49.228 INFO:teuthology.orchestra.run.smithi007.stdout: "mon_max_mdsmap_epochs": "500", 2024-09-18T08:03:49.228 INFO:teuthology.orchestra.run.smithi007.stdout: "mon_max_mgrmap_epochs": "500", 2024-09-18T08:03:49.228 INFO:teuthology.orchestra.run.smithi007.stdout: "mon_max_nvmeof_epochs": "500", 2024-09-18T08:03:49.228 INFO:teuthology.orchestra.run.smithi007.stdout: "mon_max_osd": "10000", 2024-09-18T08:03:49.228 INFO:teuthology.orchestra.run.smithi007.stdout: "mon_max_pg_per_osd": "10000", 2024-09-18T08:03:49.228 INFO:teuthology.orchestra.run.smithi007.stdout: "mon_max_pool_pg_num": "65536", 2024-09-18T08:03:49.228 INFO:teuthology.orchestra.run.smithi007.stdout: "mon_max_snap_prune_per_epoch": "100", 2024-09-18T08:03:49.228 INFO:teuthology.orchestra.run.smithi007.stdout: "mon_mds_blocklist_interval": "86400.000000", 2024-09-18T08:03:49.228 INFO:teuthology.orchestra.run.smithi007.stdout: "mon_mds_force_trim_to": "0", 2024-09-18T08:03:49.228 INFO:teuthology.orchestra.run.smithi007.stdout: "mon_mds_skip_sanity": "false", 2024-09-18T08:03:49.228 INFO:teuthology.orchestra.run.smithi007.stdout: "mon_memory_autotune": "true", 2024-09-18T08:03:49.229 INFO:teuthology.orchestra.run.smithi007.stdout: "mon_memory_target": "2147483648", 2024-09-18T08:03:49.229 INFO:teuthology.orchestra.run.smithi007.stdout: "mon_mgr_beacon_grace": "30", 2024-09-18T08:03:49.229 INFO:teuthology.orchestra.run.smithi007.stdout: "mon_mgr_blocklist_interval": "86400.000000", 2024-09-18T08:03:49.229 INFO:teuthology.orchestra.run.smithi007.stdout: "mon_mgr_digest_period": "5", 2024-09-18T08:03:49.229 INFO:teuthology.orchestra.run.smithi007.stdout: "mon_mgr_inactive_grace": "60", 2024-09-18T08:03:49.229 INFO:teuthology.orchestra.run.smithi007.stdout: "mon_mgr_mkfs_grace": "240", 2024-09-18T08:03:49.229 INFO:teuthology.orchestra.run.smithi007.stdout: "mon_mgr_proxy_client_bytes_ratio": "0.300000", 2024-09-18T08:03:49.229 INFO:teuthology.orchestra.run.smithi007.stdout: "mon_min_osdmap_epochs": "50", 2024-09-18T08:03:49.229 INFO:teuthology.orchestra.run.smithi007.stdout: "mon_nvmeofgw_beacon_grace": "10", 2024-09-18T08:03:49.229 INFO:teuthology.orchestra.run.smithi007.stdout: "mon_nvmeofgw_set_group_id_retry": "1000", 2024-09-18T08:03:49.229 INFO:teuthology.orchestra.run.smithi007.stdout: "mon_op_complaint_time": "30", 2024-09-18T08:03:49.229 INFO:teuthology.orchestra.run.smithi007.stdout: "mon_op_history_duration": "600", 2024-09-18T08:03:49.230 INFO:teuthology.orchestra.run.smithi007.stdout: "mon_op_history_size": "20", 2024-09-18T08:03:49.230 INFO:teuthology.orchestra.run.smithi007.stdout: "mon_op_history_slow_op_size": "20", 2024-09-18T08:03:49.230 INFO:teuthology.orchestra.run.smithi007.stdout: "mon_op_history_slow_op_threshold": "10", 2024-09-18T08:03:49.230 INFO:teuthology.orchestra.run.smithi007.stdout: "mon_op_log_threshold": "5", 2024-09-18T08:03:49.230 INFO:teuthology.orchestra.run.smithi007.stdout: "mon_osd_adjust_down_out_interval": "true", 2024-09-18T08:03:49.230 INFO:teuthology.orchestra.run.smithi007.stdout: "mon_osd_adjust_heartbeat_grace": "true", 2024-09-18T08:03:49.230 INFO:teuthology.orchestra.run.smithi007.stdout: "mon_osd_auto_mark_auto_out_in": "true", 2024-09-18T08:03:49.230 INFO:teuthology.orchestra.run.smithi007.stdout: "mon_osd_auto_mark_in": "false", 2024-09-18T08:03:49.230 INFO:teuthology.orchestra.run.smithi007.stdout: "mon_osd_auto_mark_new_in": "true", 2024-09-18T08:03:49.230 INFO:teuthology.orchestra.run.smithi007.stdout: "mon_osd_backfillfull_ratio": "0.900000", 2024-09-18T08:03:49.230 INFO:teuthology.orchestra.run.smithi007.stdout: "mon_osd_blocklist_default_expire": "3600.000000", 2024-09-18T08:03:49.230 INFO:teuthology.orchestra.run.smithi007.stdout: "mon_osd_cache_size": "500", 2024-09-18T08:03:49.231 INFO:teuthology.orchestra.run.smithi007.stdout: "mon_osd_cache_size_min": "134217728", 2024-09-18T08:03:49.231 INFO:teuthology.orchestra.run.smithi007.stdout: "mon_osd_crush_smoke_test": "true", 2024-09-18T08:03:49.231 INFO:teuthology.orchestra.run.smithi007.stdout: "mon_osd_destroyed_out_interval": "600", 2024-09-18T08:03:49.231 INFO:teuthology.orchestra.run.smithi007.stdout: "mon_osd_down_out_interval": "600", 2024-09-18T08:03:49.231 INFO:teuthology.orchestra.run.smithi007.stdout: "mon_osd_down_out_subtree_limit": "rack", 2024-09-18T08:03:49.231 INFO:teuthology.orchestra.run.smithi007.stdout: "mon_osd_err_op_age_ratio": "128.000000", 2024-09-18T08:03:49.231 INFO:teuthology.orchestra.run.smithi007.stdout: "mon_osd_force_trim_to": "0", 2024-09-18T08:03:49.231 INFO:teuthology.orchestra.run.smithi007.stdout: "mon_osd_full_ratio": "0.950000", 2024-09-18T08:03:49.231 INFO:teuthology.orchestra.run.smithi007.stdout: "mon_osd_initial_require_min_compat_client": "luminous", 2024-09-18T08:03:49.231 INFO:teuthology.orchestra.run.smithi007.stdout: "mon_osd_laggy_halflife": "3600", 2024-09-18T08:03:49.231 INFO:teuthology.orchestra.run.smithi007.stdout: "mon_osd_laggy_max_interval": "300", 2024-09-18T08:03:49.232 INFO:teuthology.orchestra.run.smithi007.stdout: "mon_osd_laggy_weight": "0.300000", 2024-09-18T08:03:49.232 INFO:teuthology.orchestra.run.smithi007.stdout: "mon_osd_mapping_pgs_per_chunk": "4096", 2024-09-18T08:03:49.232 INFO:teuthology.orchestra.run.smithi007.stdout: "mon_osd_max_initial_pgs": "1024", 2024-09-18T08:03:49.232 INFO:teuthology.orchestra.run.smithi007.stdout: "mon_osd_min_down_reporters": "2", 2024-09-18T08:03:49.232 INFO:teuthology.orchestra.run.smithi007.stdout: "mon_osd_min_in_ratio": "0.750000", 2024-09-18T08:03:49.232 INFO:teuthology.orchestra.run.smithi007.stdout: "mon_osd_min_up_ratio": "0.300000", 2024-09-18T08:03:49.232 INFO:teuthology.orchestra.run.smithi007.stdout: "mon_osd_nearfull_ratio": "0.850000", 2024-09-18T08:03:49.232 INFO:teuthology.orchestra.run.smithi007.stdout: "mon_osd_prime_pg_temp": "true", 2024-09-18T08:03:49.232 INFO:teuthology.orchestra.run.smithi007.stdout: "mon_osd_prime_pg_temp_max_estimate": "0.250000", 2024-09-18T08:03:49.232 INFO:teuthology.orchestra.run.smithi007.stdout: "mon_osd_prime_pg_temp_max_time": "0.500000", 2024-09-18T08:03:49.232 INFO:teuthology.orchestra.run.smithi007.stdout: "mon_osd_report_timeout": "900", 2024-09-18T08:03:49.232 INFO:teuthology.orchestra.run.smithi007.stdout: "mon_osd_reporter_subtree_level": "osd", 2024-09-18T08:03:49.232 INFO:teuthology.orchestra.run.smithi007.stdout: "mon_osd_snap_trim_queue_warn_on": "32768", 2024-09-18T08:03:49.233 INFO:teuthology.orchestra.run.smithi007.stdout: "mon_osd_warn_num_repaired": "10", 2024-09-18T08:03:49.233 INFO:teuthology.orchestra.run.smithi007.stdout: "mon_osd_warn_op_age": "32.000000", 2024-09-18T08:03:49.233 INFO:teuthology.orchestra.run.smithi007.stdout: "mon_osdmap_full_prune_enabled": "true", 2024-09-18T08:03:49.233 INFO:teuthology.orchestra.run.smithi007.stdout: "mon_osdmap_full_prune_interval": "2", 2024-09-18T08:03:49.233 INFO:teuthology.orchestra.run.smithi007.stdout: "mon_osdmap_full_prune_min": "15", 2024-09-18T08:03:49.233 INFO:teuthology.orchestra.run.smithi007.stdout: "mon_osdmap_full_prune_txsize": "2", 2024-09-18T08:03:49.233 INFO:teuthology.orchestra.run.smithi007.stdout: "mon_pg_check_down_all_threshold": "0.500000", 2024-09-18T08:03:49.233 INFO:teuthology.orchestra.run.smithi007.stdout: "mon_pg_stuck_threshold": "60", 2024-09-18T08:03:49.233 INFO:teuthology.orchestra.run.smithi007.stdout: "mon_pg_warn_max_object_skew": "0.000000", 2024-09-18T08:03:49.233 INFO:teuthology.orchestra.run.smithi007.stdout: "mon_pg_warn_min_objects": "10000", 2024-09-18T08:03:49.233 INFO:teuthology.orchestra.run.smithi007.stdout: "mon_pg_warn_min_per_osd": "0", 2024-09-18T08:03:49.233 INFO:teuthology.orchestra.run.smithi007.stdout: "mon_pg_warn_min_pool_objects": "1000", 2024-09-18T08:03:49.234 INFO:teuthology.orchestra.run.smithi007.stdout: "mon_pool_quota_crit_threshold": "0", 2024-09-18T08:03:49.234 INFO:teuthology.orchestra.run.smithi007.stdout: "mon_pool_quota_warn_threshold": "0", 2024-09-18T08:03:49.234 INFO:teuthology.orchestra.run.smithi007.stdout: "mon_probe_timeout": "2.000000", 2024-09-18T08:03:49.234 INFO:teuthology.orchestra.run.smithi007.stdout: "mon_reweight_max_change": "0.050000", 2024-09-18T08:03:49.234 INFO:teuthology.orchestra.run.smithi007.stdout: "mon_reweight_max_osds": "4", 2024-09-18T08:03:49.234 INFO:teuthology.orchestra.run.smithi007.stdout: "mon_reweight_min_bytes_per_osd": "10", 2024-09-18T08:03:49.234 INFO:teuthology.orchestra.run.smithi007.stdout: "mon_reweight_min_pgs_per_osd": "4", 2024-09-18T08:03:49.234 INFO:teuthology.orchestra.run.smithi007.stdout: "mon_rocksdb_options": "write_buffer_size=33554432,compression=kNoCompression,level_compaction_dynamic_level_bytes=true", 2024-09-18T08:03:49.234 INFO:teuthology.orchestra.run.smithi007.stdout: "mon_scrub_inject_crc_mismatch": "0.000000", 2024-09-18T08:03:49.234 INFO:teuthology.orchestra.run.smithi007.stdout: "mon_scrub_inject_missing_keys": "0.000000", 2024-09-18T08:03:49.234 INFO:teuthology.orchestra.run.smithi007.stdout: "mon_scrub_interval": "86400", 2024-09-18T08:03:49.234 INFO:teuthology.orchestra.run.smithi007.stdout: "mon_scrub_max_keys": "100", 2024-09-18T08:03:49.234 INFO:teuthology.orchestra.run.smithi007.stdout: "mon_scrub_timeout": "300", 2024-09-18T08:03:49.235 INFO:teuthology.orchestra.run.smithi007.stdout: "mon_session_timeout": "300", 2024-09-18T08:03:49.235 INFO:teuthology.orchestra.run.smithi007.stdout: "mon_smart_report_timeout": "5", 2024-09-18T08:03:49.235 INFO:teuthology.orchestra.run.smithi007.stdout: "mon_stat_smooth_intervals": "6", 2024-09-18T08:03:49.235 INFO:teuthology.orchestra.run.smithi007.stdout: "mon_stretch_cluster_recovery_ratio": "0.600000", 2024-09-18T08:03:49.235 INFO:teuthology.orchestra.run.smithi007.stdout: "mon_stretch_pool_min_size": "2", 2024-09-18T08:03:49.235 INFO:teuthology.orchestra.run.smithi007.stdout: "mon_stretch_pool_size": "4", 2024-09-18T08:03:49.235 INFO:teuthology.orchestra.run.smithi007.stdout: "mon_stretch_recovery_min_wait": "15.000000", 2024-09-18T08:03:49.235 INFO:teuthology.orchestra.run.smithi007.stdout: "mon_subscribe_interval": "86400.000000", 2024-09-18T08:03:49.235 INFO:teuthology.orchestra.run.smithi007.stdout: "mon_sync_debug": "false", 2024-09-18T08:03:49.236 INFO:teuthology.orchestra.run.smithi007.stdout: "mon_sync_max_payload_keys": "2000", 2024-09-18T08:03:49.236 INFO:teuthology.orchestra.run.smithi007.stdout: "mon_sync_max_payload_size": "1048576", 2024-09-18T08:03:49.236 INFO:teuthology.orchestra.run.smithi007.stdout: "mon_sync_provider_kill_at": "0", 2024-09-18T08:03:49.236 INFO:teuthology.orchestra.run.smithi007.stdout: "mon_sync_requester_kill_at": "0", 2024-09-18T08:03:49.236 INFO:teuthology.orchestra.run.smithi007.stdout: "mon_sync_timeout": "60.000000", 2024-09-18T08:03:49.236 INFO:teuthology.orchestra.run.smithi007.stdout: "mon_target_pg_per_osd": "100", 2024-09-18T08:03:49.236 INFO:teuthology.orchestra.run.smithi007.stdout: "mon_tick_interval": "5", 2024-09-18T08:03:49.236 INFO:teuthology.orchestra.run.smithi007.stdout: "mon_timecheck_interval": "300.000000", 2024-09-18T08:03:49.236 INFO:teuthology.orchestra.run.smithi007.stdout: "mon_timecheck_skew_interval": "30.000000", 2024-09-18T08:03:49.236 INFO:teuthology.orchestra.run.smithi007.stdout: "mon_use_min_delay_socket": "false", 2024-09-18T08:03:49.236 INFO:teuthology.orchestra.run.smithi007.stdout: "mon_warn_older_version_delay": "604800", 2024-09-18T08:03:49.237 INFO:teuthology.orchestra.run.smithi007.stdout: "mon_warn_on_cache_pools_without_hit_sets": "true", 2024-09-18T08:03:49.237 INFO:teuthology.orchestra.run.smithi007.stdout: "mon_warn_on_crush_straw_calc_version_zero": "false", 2024-09-18T08:03:49.237 INFO:teuthology.orchestra.run.smithi007.stdout: "mon_warn_on_degraded_stretch_mode": "true", 2024-09-18T08:03:49.237 INFO:teuthology.orchestra.run.smithi007.stdout: "mon_warn_on_filestore_osds": "true", 2024-09-18T08:03:49.237 INFO:teuthology.orchestra.run.smithi007.stdout: "mon_warn_on_insecure_global_id_reclaim": "false", 2024-09-18T08:03:49.237 INFO:teuthology.orchestra.run.smithi007.stdout: "mon_warn_on_insecure_global_id_reclaim_allowed": "false", 2024-09-18T08:03:49.237 INFO:teuthology.orchestra.run.smithi007.stdout: "mon_warn_on_legacy_crush_tunables": "false", 2024-09-18T08:03:49.237 INFO:teuthology.orchestra.run.smithi007.stdout: "mon_warn_on_misplaced": "false", 2024-09-18T08:03:49.237 INFO:teuthology.orchestra.run.smithi007.stdout: "mon_warn_on_msgr2_not_enabled": "true", 2024-09-18T08:03:49.237 INFO:teuthology.orchestra.run.smithi007.stdout: "mon_warn_on_older_version": "true", 2024-09-18T08:03:49.237 INFO:teuthology.orchestra.run.smithi007.stdout: "mon_warn_on_osd_down_out_interval_zero": "false", 2024-09-18T08:03:49.237 INFO:teuthology.orchestra.run.smithi007.stdout: "mon_warn_on_pool_no_app": "true", 2024-09-18T08:03:49.237 INFO:teuthology.orchestra.run.smithi007.stdout: "mon_warn_on_pool_no_redundancy": "true", 2024-09-18T08:03:49.238 INFO:teuthology.orchestra.run.smithi007.stdout: "mon_warn_on_pool_pg_num_not_power_of_two": "false", 2024-09-18T08:03:49.238 INFO:teuthology.orchestra.run.smithi007.stdout: "mon_warn_on_slow_ping_ratio": "0.050000", 2024-09-18T08:03:49.238 INFO:teuthology.orchestra.run.smithi007.stdout: "mon_warn_on_slow_ping_time": "0.000000", 2024-09-18T08:03:49.238 INFO:teuthology.orchestra.run.smithi007.stdout: "mon_warn_on_too_few_osds": "false", 2024-09-18T08:03:49.238 INFO:teuthology.orchestra.run.smithi007.stdout: "mon_warn_pg_not_deep_scrubbed_ratio": "0.750000", 2024-09-18T08:03:49.238 INFO:teuthology.orchestra.run.smithi007.stdout: "mon_warn_pg_not_scrubbed_ratio": "0.500000", 2024-09-18T08:03:49.238 INFO:teuthology.orchestra.run.smithi007.stdout: "monmap": "", 2024-09-18T08:03:49.238 INFO:teuthology.orchestra.run.smithi007.stdout: "motr_admin_endpoint": "192.168.180.182@tcp:12345:4:1", 2024-09-18T08:03:49.238 INFO:teuthology.orchestra.run.smithi007.stdout: "motr_admin_fid": "0x7200000000000001:0x0", 2024-09-18T08:03:49.238 INFO:teuthology.orchestra.run.smithi007.stdout: "motr_ha_endpoint": "192.168.180.182@tcp:12345:1:1", 2024-09-18T08:03:49.238 INFO:teuthology.orchestra.run.smithi007.stdout: "motr_my_endpoint": "192.168.180.182@tcp:12345:4:1", 2024-09-18T08:03:49.238 INFO:teuthology.orchestra.run.smithi007.stdout: "motr_my_fid": "0x7200000000000001:0x0", 2024-09-18T08:03:49.238 INFO:teuthology.orchestra.run.smithi007.stdout: "motr_profile_fid": "0x7000000000000001:0x0", 2024-09-18T08:03:49.239 INFO:teuthology.orchestra.run.smithi007.stdout: "motr_tracing_enabled": "false", 2024-09-18T08:03:49.239 INFO:teuthology.orchestra.run.smithi007.stdout: "ms_async_op_threads": "3", 2024-09-18T08:03:49.239 INFO:teuthology.orchestra.run.smithi007.stdout: "ms_async_rdma_buffer_size": "131072", 2024-09-18T08:03:49.239 INFO:teuthology.orchestra.run.smithi007.stdout: "ms_async_rdma_cm": "false", 2024-09-18T08:03:49.239 INFO:teuthology.orchestra.run.smithi007.stdout: "ms_async_rdma_device_name": "", 2024-09-18T08:03:49.239 INFO:teuthology.orchestra.run.smithi007.stdout: "ms_async_rdma_dscp": "96", 2024-09-18T08:03:49.239 INFO:teuthology.orchestra.run.smithi007.stdout: "ms_async_rdma_enable_hugepage": "false", 2024-09-18T08:03:49.239 INFO:teuthology.orchestra.run.smithi007.stdout: "ms_async_rdma_gid_idx": "0", 2024-09-18T08:03:49.239 INFO:teuthology.orchestra.run.smithi007.stdout: "ms_async_rdma_local_gid": "", 2024-09-18T08:03:49.239 INFO:teuthology.orchestra.run.smithi007.stdout: "ms_async_rdma_polling_us": "1000", 2024-09-18T08:03:49.240 INFO:teuthology.orchestra.run.smithi007.stdout: "ms_async_rdma_port_num": "1", 2024-09-18T08:03:49.240 INFO:teuthology.orchestra.run.smithi007.stdout: "ms_async_rdma_receive_buffers": "32768", 2024-09-18T08:03:49.240 INFO:teuthology.orchestra.run.smithi007.stdout: "ms_async_rdma_receive_queue_len": "4096", 2024-09-18T08:03:49.240 INFO:teuthology.orchestra.run.smithi007.stdout: "ms_async_rdma_roce_ver": "1", 2024-09-18T08:03:49.240 INFO:teuthology.orchestra.run.smithi007.stdout: "ms_async_rdma_send_buffers": "1024", 2024-09-18T08:03:49.240 INFO:teuthology.orchestra.run.smithi007.stdout: "ms_async_rdma_sl": "3", 2024-09-18T08:03:49.240 INFO:teuthology.orchestra.run.smithi007.stdout: "ms_async_rdma_support_srq": "true", 2024-09-18T08:03:49.240 INFO:teuthology.orchestra.run.smithi007.stdout: "ms_async_rdma_type": "ib", 2024-09-18T08:03:49.240 INFO:teuthology.orchestra.run.smithi007.stdout: "ms_async_reap_threshold": "5", 2024-09-18T08:03:49.240 INFO:teuthology.orchestra.run.smithi007.stdout: "ms_bind_before_connect": "false", 2024-09-18T08:03:49.240 INFO:teuthology.orchestra.run.smithi007.stdout: "ms_bind_ipv4": "true", 2024-09-18T08:03:49.240 INFO:teuthology.orchestra.run.smithi007.stdout: "ms_bind_ipv6": "false", 2024-09-18T08:03:49.241 INFO:teuthology.orchestra.run.smithi007.stdout: "ms_bind_msgr1": "false", 2024-09-18T08:03:49.241 INFO:teuthology.orchestra.run.smithi007.stdout: "ms_bind_msgr2": "true", 2024-09-18T08:03:49.241 INFO:teuthology.orchestra.run.smithi007.stdout: "ms_bind_port_max": "7568", 2024-09-18T08:03:49.241 INFO:teuthology.orchestra.run.smithi007.stdout: "ms_bind_port_min": "6800", 2024-09-18T08:03:49.241 INFO:teuthology.orchestra.run.smithi007.stdout: "ms_bind_prefer_ipv4": "false", 2024-09-18T08:03:49.241 INFO:teuthology.orchestra.run.smithi007.stdout: "ms_bind_retry_count": "3", 2024-09-18T08:03:49.241 INFO:teuthology.orchestra.run.smithi007.stdout: "ms_bind_retry_delay": "5", 2024-09-18T08:03:49.241 INFO:teuthology.orchestra.run.smithi007.stdout: "ms_blackhole_client": "false", 2024-09-18T08:03:49.241 INFO:teuthology.orchestra.run.smithi007.stdout: "ms_blackhole_mds": "false", 2024-09-18T08:03:49.241 INFO:teuthology.orchestra.run.smithi007.stdout: "ms_blackhole_mgr": "false", 2024-09-18T08:03:49.241 INFO:teuthology.orchestra.run.smithi007.stdout: "ms_blackhole_mon": "false", 2024-09-18T08:03:49.241 INFO:teuthology.orchestra.run.smithi007.stdout: "ms_blackhole_osd": "false", 2024-09-18T08:03:49.242 INFO:teuthology.orchestra.run.smithi007.stdout: "ms_client_mode": "crc secure", 2024-09-18T08:03:49.242 INFO:teuthology.orchestra.run.smithi007.stdout: "ms_client_throttle_retry_time_interval": "5000", 2024-09-18T08:03:49.242 INFO:teuthology.orchestra.run.smithi007.stdout: "ms_cluster_mode": "crc secure", 2024-09-18T08:03:49.242 INFO:teuthology.orchestra.run.smithi007.stdout: "ms_cluster_type": "", 2024-09-18T08:03:49.242 INFO:teuthology.orchestra.run.smithi007.stdout: "ms_compress_secure": "false", 2024-09-18T08:03:49.242 INFO:teuthology.orchestra.run.smithi007.stdout: "ms_connection_idle_timeout": "900", 2024-09-18T08:03:49.242 INFO:teuthology.orchestra.run.smithi007.stdout: "ms_connection_ready_timeout": "10", 2024-09-18T08:03:49.242 INFO:teuthology.orchestra.run.smithi007.stdout: "ms_crc_data": "true", 2024-09-18T08:03:49.242 INFO:teuthology.orchestra.run.smithi007.stdout: "ms_crc_header": "true", 2024-09-18T08:03:49.242 INFO:teuthology.orchestra.run.smithi007.stdout: "ms_die_on_bad_msg": "false", 2024-09-18T08:03:49.242 INFO:teuthology.orchestra.run.smithi007.stdout: "ms_die_on_bug": "true", 2024-09-18T08:03:49.242 INFO:teuthology.orchestra.run.smithi007.stdout: "ms_die_on_old_message": "true", 2024-09-18T08:03:49.242 INFO:teuthology.orchestra.run.smithi007.stdout: "ms_die_on_skipped_message": "false", 2024-09-18T08:03:49.243 INFO:teuthology.orchestra.run.smithi007.stdout: "ms_die_on_unhandled_msg": "false", 2024-09-18T08:03:49.243 INFO:teuthology.orchestra.run.smithi007.stdout: "ms_dispatch_throttle_bytes": "104857600", 2024-09-18T08:03:49.243 INFO:teuthology.orchestra.run.smithi007.stdout: "ms_dpdk_coremask": "0xF", 2024-09-18T08:03:49.243 INFO:teuthology.orchestra.run.smithi007.stdout: "ms_dpdk_debug_allow_loopback": "false", 2024-09-18T08:03:49.243 INFO:teuthology.orchestra.run.smithi007.stdout: "ms_dpdk_devs_allowlist": "", 2024-09-18T08:03:49.243 INFO:teuthology.orchestra.run.smithi007.stdout: "ms_dpdk_enable_tso": "true", 2024-09-18T08:03:49.243 INFO:teuthology.orchestra.run.smithi007.stdout: "ms_dpdk_gateway_ipv4_addr": "", 2024-09-18T08:03:49.243 INFO:teuthology.orchestra.run.smithi007.stdout: "ms_dpdk_host_ipv4_addr": "", 2024-09-18T08:03:49.243 INFO:teuthology.orchestra.run.smithi007.stdout: "ms_dpdk_hugepages": "", 2024-09-18T08:03:49.243 INFO:teuthology.orchestra.run.smithi007.stdout: "ms_dpdk_hw_flow_control": "true", 2024-09-18T08:03:49.243 INFO:teuthology.orchestra.run.smithi007.stdout: "ms_dpdk_hw_queue_weight": "1.000000", 2024-09-18T08:03:49.243 INFO:teuthology.orchestra.run.smithi007.stdout: "ms_dpdk_lro": "true", 2024-09-18T08:03:49.243 INFO:teuthology.orchestra.run.smithi007.stdout: "ms_dpdk_memory_channel": "4", 2024-09-18T08:03:49.244 INFO:teuthology.orchestra.run.smithi007.stdout: "ms_dpdk_netmask_ipv4_addr": "", 2024-09-18T08:03:49.244 INFO:teuthology.orchestra.run.smithi007.stdout: "ms_dpdk_pmd": "", 2024-09-18T08:03:49.244 INFO:teuthology.orchestra.run.smithi007.stdout: "ms_dpdk_port_id": "0", 2024-09-18T08:03:49.244 INFO:teuthology.orchestra.run.smithi007.stdout: "ms_dpdk_rx_buffer_count_per_core": "8192", 2024-09-18T08:03:49.244 INFO:teuthology.orchestra.run.smithi007.stdout: "ms_dump_corrupt_message_level": "1", 2024-09-18T08:03:49.244 INFO:teuthology.orchestra.run.smithi007.stdout: "ms_dump_on_send": "false", 2024-09-18T08:03:49.244 INFO:teuthology.orchestra.run.smithi007.stdout: "ms_initial_backoff": "0.200000", 2024-09-18T08:03:49.244 INFO:teuthology.orchestra.run.smithi007.stdout: "ms_inject_delay_max": "1.000000", 2024-09-18T08:03:49.244 INFO:teuthology.orchestra.run.smithi007.stdout: "ms_inject_delay_probability": "0.000000", 2024-09-18T08:03:49.244 INFO:teuthology.orchestra.run.smithi007.stdout: "ms_inject_delay_type": "", 2024-09-18T08:03:49.245 INFO:teuthology.orchestra.run.smithi007.stdout: "ms_inject_internal_delays": "0.000000", 2024-09-18T08:03:49.245 INFO:teuthology.orchestra.run.smithi007.stdout: "ms_inject_network_congestion": "0", 2024-09-18T08:03:49.245 INFO:teuthology.orchestra.run.smithi007.stdout: "ms_inject_socket_failures": "0", 2024-09-18T08:03:49.245 INFO:teuthology.orchestra.run.smithi007.stdout: "ms_learn_addr_from_peer": "true", 2024-09-18T08:03:49.245 INFO:teuthology.orchestra.run.smithi007.stdout: "ms_max_accept_failures": "4", 2024-09-18T08:03:49.245 INFO:teuthology.orchestra.run.smithi007.stdout: "ms_max_backoff": "15.000000", 2024-09-18T08:03:49.245 INFO:teuthology.orchestra.run.smithi007.stdout: "ms_mon_client_mode": "secure crc", 2024-09-18T08:03:49.245 INFO:teuthology.orchestra.run.smithi007.stdout: "ms_mon_cluster_mode": "secure crc", 2024-09-18T08:03:49.245 INFO:teuthology.orchestra.run.smithi007.stdout: "ms_mon_service_mode": "secure crc", 2024-09-18T08:03:49.245 INFO:teuthology.orchestra.run.smithi007.stdout: "ms_osd_compress_min_size": "1024", 2024-09-18T08:03:49.245 INFO:teuthology.orchestra.run.smithi007.stdout: "ms_osd_compress_mode": "none", 2024-09-18T08:03:49.245 INFO:teuthology.orchestra.run.smithi007.stdout: "ms_osd_compression_algorithm": "snappy", 2024-09-18T08:03:49.245 INFO:teuthology.orchestra.run.smithi007.stdout: "ms_pq_max_tokens_per_priority": "16777216", 2024-09-18T08:03:49.246 INFO:teuthology.orchestra.run.smithi007.stdout: "ms_pq_min_cost": "65536", 2024-09-18T08:03:49.246 INFO:teuthology.orchestra.run.smithi007.stdout: "ms_public_type": "", 2024-09-18T08:03:49.246 INFO:teuthology.orchestra.run.smithi007.stdout: "ms_service_mode": "crc secure", 2024-09-18T08:03:49.246 INFO:teuthology.orchestra.run.smithi007.stdout: "ms_tcp_listen_backlog": "512", 2024-09-18T08:03:49.246 INFO:teuthology.orchestra.run.smithi007.stdout: "ms_tcp_nodelay": "true", 2024-09-18T08:03:49.246 INFO:teuthology.orchestra.run.smithi007.stdout: "ms_tcp_prefetch_max_size": "4096", 2024-09-18T08:03:49.246 INFO:teuthology.orchestra.run.smithi007.stdout: "ms_tcp_rcvbuf": "0", 2024-09-18T08:03:49.246 INFO:teuthology.orchestra.run.smithi007.stdout: "ms_time_events_min_wait_interval": "1000", 2024-09-18T08:03:49.246 INFO:teuthology.orchestra.run.smithi007.stdout: "ms_type": "async+posix", 2024-09-18T08:03:49.246 INFO:teuthology.orchestra.run.smithi007.stdout: "no_config_file": "false", 2024-09-18T08:03:49.247 INFO:teuthology.orchestra.run.smithi007.stdout: "nvmeof_mon_client_disconnect_panic": "100", 2024-09-18T08:03:49.247 INFO:teuthology.orchestra.run.smithi007.stdout: "nvmeof_mon_client_tick_period": "2", 2024-09-18T08:03:49.247 INFO:teuthology.orchestra.run.smithi007.stdout: "objecter_completion_locks_per_session": "32", 2024-09-18T08:03:49.247 INFO:teuthology.orchestra.run.smithi007.stdout: "objecter_debug_inject_relock_delay": "false", 2024-09-18T08:03:49.247 INFO:teuthology.orchestra.run.smithi007.stdout: "objecter_inflight_op_bytes": "104857600", 2024-09-18T08:03:49.247 INFO:teuthology.orchestra.run.smithi007.stdout: "objecter_inflight_ops": "1024", 2024-09-18T08:03:49.247 INFO:teuthology.orchestra.run.smithi007.stdout: "objecter_inject_no_watch_ping": "false", 2024-09-18T08:03:49.247 INFO:teuthology.orchestra.run.smithi007.stdout: "objecter_retry_writes_after_first_reply": "false", 2024-09-18T08:03:49.247 INFO:teuthology.orchestra.run.smithi007.stdout: "objecter_tick_interval": "5.000000", 2024-09-18T08:03:49.247 INFO:teuthology.orchestra.run.smithi007.stdout: "objecter_timeout": "10.000000", 2024-09-18T08:03:49.247 INFO:teuthology.orchestra.run.smithi007.stdout: "objectstore_blackhole": "false", 2024-09-18T08:03:49.247 INFO:teuthology.orchestra.run.smithi007.stdout: "objectstore_debug_throw_on_failed_txc": "false", 2024-09-18T08:03:49.247 INFO:teuthology.orchestra.run.smithi007.stdout: "openssl_engine_opts": "", 2024-09-18T08:03:49.248 INFO:teuthology.orchestra.run.smithi007.stdout: "osd_agent_delay_time": "5.000000", 2024-09-18T08:03:49.248 INFO:teuthology.orchestra.run.smithi007.stdout: "osd_agent_hist_halflife": "1000", 2024-09-18T08:03:49.248 INFO:teuthology.orchestra.run.smithi007.stdout: "osd_agent_max_low_ops": "2", 2024-09-18T08:03:49.248 INFO:teuthology.orchestra.run.smithi007.stdout: "osd_agent_max_ops": "4", 2024-09-18T08:03:49.248 INFO:teuthology.orchestra.run.smithi007.stdout: "osd_agent_min_evict_effort": "0.100000", 2024-09-18T08:03:49.248 INFO:teuthology.orchestra.run.smithi007.stdout: "osd_agent_quantize_effort": "0.100000", 2024-09-18T08:03:49.248 INFO:teuthology.orchestra.run.smithi007.stdout: "osd_agent_slop": "0.020000", 2024-09-18T08:03:49.248 INFO:teuthology.orchestra.run.smithi007.stdout: "osd_aggregated_slow_ops_logging": "true", 2024-09-18T08:03:49.248 INFO:teuthology.orchestra.run.smithi007.stdout: "osd_allow_recovery_below_min_size": "true", 2024-09-18T08:03:49.248 INFO:teuthology.orchestra.run.smithi007.stdout: "osd_asio_thread_count": "2", 2024-09-18T08:03:49.248 INFO:teuthology.orchestra.run.smithi007.stdout: "osd_async_recovery_min_cost": "100", 2024-09-18T08:03:49.248 INFO:teuthology.orchestra.run.smithi007.stdout: "osd_auto_mark_unfound_lost": "false", 2024-09-18T08:03:49.248 INFO:teuthology.orchestra.run.smithi007.stdout: "osd_backfill_retry_interval": "30.000000", 2024-09-18T08:03:49.249 INFO:teuthology.orchestra.run.smithi007.stdout: "osd_backfill_scan_max": "512", 2024-09-18T08:03:49.249 INFO:teuthology.orchestra.run.smithi007.stdout: "osd_backfill_scan_min": "64", 2024-09-18T08:03:49.249 INFO:teuthology.orchestra.run.smithi007.stdout: "osd_backoff_on_degraded": "false", 2024-09-18T08:03:49.249 INFO:teuthology.orchestra.run.smithi007.stdout: "osd_backoff_on_peering": "false", 2024-09-18T08:03:49.249 INFO:teuthology.orchestra.run.smithi007.stdout: "osd_backoff_on_unfound": "true", 2024-09-18T08:03:49.249 INFO:teuthology.orchestra.run.smithi007.stdout: "osd_beacon_report_interval": "300", 2024-09-18T08:03:49.249 INFO:teuthology.orchestra.run.smithi007.stdout: "osd_bench_duration": "30", 2024-09-18T08:03:49.249 INFO:teuthology.orchestra.run.smithi007.stdout: "osd_bench_large_size_max_throughput": "104857600", 2024-09-18T08:03:49.249 INFO:teuthology.orchestra.run.smithi007.stdout: "osd_bench_max_block_size": "67108864", 2024-09-18T08:03:49.250 INFO:teuthology.orchestra.run.smithi007.stdout: "osd_bench_small_size_max_iops": "100", 2024-09-18T08:03:49.250 INFO:teuthology.orchestra.run.smithi007.stdout: "osd_blkin_trace_all": "false", 2024-09-18T08:03:49.250 INFO:teuthology.orchestra.run.smithi007.stdout: "osd_blocked_scrub_grace_period": "120", 2024-09-18T08:03:49.250 INFO:teuthology.orchestra.run.smithi007.stdout: "osd_calc_pg_upmaps_aggressively": "true", 2024-09-18T08:03:49.250 INFO:teuthology.orchestra.run.smithi007.stdout: "osd_calc_pg_upmaps_aggressively_fast": "true", 2024-09-18T08:03:49.250 INFO:teuthology.orchestra.run.smithi007.stdout: "osd_calc_pg_upmaps_local_fallback_retries": "100", 2024-09-18T08:03:49.251 INFO:teuthology.orchestra.run.smithi007.stdout: "osd_check_for_log_corruption": "false", 2024-09-18T08:03:49.251 INFO:teuthology.orchestra.run.smithi007.stdout: "osd_check_max_object_name_len_on_startup": "true", 2024-09-18T08:03:49.251 INFO:teuthology.orchestra.run.smithi007.stdout: "osd_class_default_list": "cephfs hello journal lock log numops otp rbd refcount rgw rgw_gc timeindex user version cas cmpomap queue 2pc_queue fifo", 2024-09-18T08:03:49.251 INFO:teuthology.orchestra.run.smithi007.stdout: "osd_class_dir": "/usr/lib64/rados-classes", 2024-09-18T08:03:49.251 INFO:teuthology.orchestra.run.smithi007.stdout: "osd_class_load_list": "cephfs hello journal lock log numops otp rbd refcount rgw rgw_gc timeindex user version cas cmpomap queue 2pc_queue fifo", 2024-09-18T08:03:49.251 INFO:teuthology.orchestra.run.smithi007.stdout: "osd_class_update_on_start": "true", 2024-09-18T08:03:49.252 INFO:teuthology.orchestra.run.smithi007.stdout: "osd_client_message_cap": "256", 2024-09-18T08:03:49.252 INFO:teuthology.orchestra.run.smithi007.stdout: "osd_client_message_size_cap": "524288000", 2024-09-18T08:03:49.252 INFO:teuthology.orchestra.run.smithi007.stdout: "osd_client_op_priority": "63", 2024-09-18T08:03:49.252 INFO:teuthology.orchestra.run.smithi007.stdout: "osd_client_watch_timeout": "30", 2024-09-18T08:03:49.252 INFO:teuthology.orchestra.run.smithi007.stdout: "osd_command_max_records": "256", 2024-09-18T08:03:49.252 INFO:teuthology.orchestra.run.smithi007.stdout: "osd_compact_on_start": "false", 2024-09-18T08:03:49.253 INFO:teuthology.orchestra.run.smithi007.stdout: "osd_copyfrom_max_chunk": "8388608", 2024-09-18T08:03:49.253 INFO:teuthology.orchestra.run.smithi007.stdout: "osd_crush_chooseleaf_type": "1", 2024-09-18T08:03:49.253 INFO:teuthology.orchestra.run.smithi007.stdout: "osd_crush_initial_weight": "-1.000000", 2024-09-18T08:03:49.253 INFO:teuthology.orchestra.run.smithi007.stdout: "osd_crush_update_on_start": "true", 2024-09-18T08:03:49.253 INFO:teuthology.orchestra.run.smithi007.stdout: "osd_crush_update_weight_set": "true", 2024-09-18T08:03:49.253 INFO:teuthology.orchestra.run.smithi007.stdout: "osd_data": "/var/lib/ceph/osd/ceph-a", 2024-09-18T08:03:49.254 INFO:teuthology.orchestra.run.smithi007.stdout: "osd_debug_crash_on_ignored_backoff": "false", 2024-09-18T08:03:49.254 INFO:teuthology.orchestra.run.smithi007.stdout: "osd_debug_deep_scrub_sleep": "0.000000", 2024-09-18T08:03:49.254 INFO:teuthology.orchestra.run.smithi007.stdout: "osd_debug_drop_ping_duration": "0", 2024-09-18T08:03:49.254 INFO:teuthology.orchestra.run.smithi007.stdout: "osd_debug_drop_ping_probability": "0.000000", 2024-09-18T08:03:49.254 INFO:teuthology.orchestra.run.smithi007.stdout: "osd_debug_feed_pullee": "-1", 2024-09-18T08:03:49.254 INFO:teuthology.orchestra.run.smithi007.stdout: "osd_debug_inject_copyfrom_error": "false", 2024-09-18T08:03:49.255 INFO:teuthology.orchestra.run.smithi007.stdout: "osd_debug_inject_dispatch_delay_duration": "0.100000", 2024-09-18T08:03:49.255 INFO:teuthology.orchestra.run.smithi007.stdout: "osd_debug_inject_dispatch_delay_probability": "0.000000", 2024-09-18T08:03:49.255 INFO:teuthology.orchestra.run.smithi007.stdout: "osd_debug_misdirected_ops": "false", 2024-09-18T08:03:49.255 INFO:teuthology.orchestra.run.smithi007.stdout: "osd_debug_no_acting_change": "false", 2024-09-18T08:03:49.255 INFO:teuthology.orchestra.run.smithi007.stdout: "osd_debug_no_purge_strays": "false", 2024-09-18T08:03:49.255 INFO:teuthology.orchestra.run.smithi007.stdout: "osd_debug_op_order": "false", 2024-09-18T08:03:49.256 INFO:teuthology.orchestra.run.smithi007.stdout: "osd_debug_pg_log_writeout": "false", 2024-09-18T08:03:49.256 INFO:teuthology.orchestra.run.smithi007.stdout: "osd_debug_pretend_recovery_active": "false", 2024-09-18T08:03:49.256 INFO:teuthology.orchestra.run.smithi007.stdout: "osd_debug_random_push_read_error": "0.000000", 2024-09-18T08:03:49.256 INFO:teuthology.orchestra.run.smithi007.stdout: "osd_debug_reject_backfill_probability": "0.000000", 2024-09-18T08:03:49.256 INFO:teuthology.orchestra.run.smithi007.stdout: "osd_debug_shutdown": "false", 2024-09-18T08:03:49.256 INFO:teuthology.orchestra.run.smithi007.stdout: "osd_debug_skip_full_check_in_backfill_reservation": "false", 2024-09-18T08:03:49.256 INFO:teuthology.orchestra.run.smithi007.stdout: "osd_debug_skip_full_check_in_recovery": "false", 2024-09-18T08:03:49.257 INFO:teuthology.orchestra.run.smithi007.stdout: "osd_debug_trim_objects": "false", 2024-09-18T08:03:49.257 INFO:teuthology.orchestra.run.smithi007.stdout: "osd_debug_verify_cached_snaps": "false", 2024-09-18T08:03:49.257 INFO:teuthology.orchestra.run.smithi007.stdout: "osd_debug_verify_missing_on_start": "false", 2024-09-18T08:03:49.257 INFO:teuthology.orchestra.run.smithi007.stdout: "osd_debug_verify_snaps": "false", 2024-09-18T08:03:49.257 INFO:teuthology.orchestra.run.smithi007.stdout: "osd_debug_verify_stray_on_activate": "false", 2024-09-18T08:03:49.257 INFO:teuthology.orchestra.run.smithi007.stdout: "osd_deep_scrub_interval": "604800.000000", 2024-09-18T08:03:49.257 INFO:teuthology.orchestra.run.smithi007.stdout: "osd_deep_scrub_keys": "1024", 2024-09-18T08:03:49.257 INFO:teuthology.orchestra.run.smithi007.stdout: "osd_deep_scrub_large_omap_object_key_threshold": "200000", 2024-09-18T08:03:49.257 INFO:teuthology.orchestra.run.smithi007.stdout: "osd_deep_scrub_large_omap_object_value_sum_threshold": "1073741824", 2024-09-18T08:03:49.257 INFO:teuthology.orchestra.run.smithi007.stdout: "osd_deep_scrub_randomize_ratio": "0.150000", 2024-09-18T08:03:49.257 INFO:teuthology.orchestra.run.smithi007.stdout: "osd_deep_scrub_stride": "524288", 2024-09-18T08:03:49.257 INFO:teuthology.orchestra.run.smithi007.stdout: "osd_deep_scrub_update_digest_min_age": "7200", 2024-09-18T08:03:49.258 INFO:teuthology.orchestra.run.smithi007.stdout: "osd_default_data_pool_replay_window": "45", 2024-09-18T08:03:49.258 INFO:teuthology.orchestra.run.smithi007.stdout: "osd_default_notify_timeout": "30", 2024-09-18T08:03:49.258 INFO:teuthology.orchestra.run.smithi007.stdout: "osd_delete_sleep": "0.000000", 2024-09-18T08:03:49.258 INFO:teuthology.orchestra.run.smithi007.stdout: "osd_delete_sleep_hdd": "5.000000", 2024-09-18T08:03:49.258 INFO:teuthology.orchestra.run.smithi007.stdout: "osd_delete_sleep_hybrid": "1.000000", 2024-09-18T08:03:49.258 INFO:teuthology.orchestra.run.smithi007.stdout: "osd_delete_sleep_ssd": "1.000000", 2024-09-18T08:03:49.258 INFO:teuthology.orchestra.run.smithi007.stdout: "osd_discard_disconnected_ops": "true", 2024-09-18T08:03:49.258 INFO:teuthology.orchestra.run.smithi007.stdout: "osd_ec_partial_reads": "true", 2024-09-18T08:03:49.258 INFO:teuthology.orchestra.run.smithi007.stdout: "osd_enable_op_tracker": "true", 2024-09-18T08:03:49.258 INFO:teuthology.orchestra.run.smithi007.stdout: "osd_erasure_code_plugins": "jerasure lrc", 2024-09-18T08:03:49.258 INFO:teuthology.orchestra.run.smithi007.stdout: "osd_extblkdev_plugins": "vdo", 2024-09-18T08:03:49.258 INFO:teuthology.orchestra.run.smithi007.stdout: "osd_failsafe_full_ratio": "0.970000", 2024-09-18T08:03:49.258 INFO:teuthology.orchestra.run.smithi007.stdout: "osd_fast_fail_on_connection_refused": "true", 2024-09-18T08:03:49.259 INFO:teuthology.orchestra.run.smithi007.stdout: "osd_fast_info": "true", 2024-09-18T08:03:49.259 INFO:teuthology.orchestra.run.smithi007.stdout: "osd_fast_shutdown": "true", 2024-09-18T08:03:49.259 INFO:teuthology.orchestra.run.smithi007.stdout: "osd_fast_shutdown_notify_mon": "true", 2024-09-18T08:03:49.259 INFO:teuthology.orchestra.run.smithi007.stdout: "osd_fast_shutdown_timeout": "15", 2024-09-18T08:03:49.259 INFO:teuthology.orchestra.run.smithi007.stdout: "osd_find_best_info_ignore_history_les": "false", 2024-09-18T08:03:49.259 INFO:teuthology.orchestra.run.smithi007.stdout: "osd_force_auth_primary_missing_objects": "100", 2024-09-18T08:03:49.259 INFO:teuthology.orchestra.run.smithi007.stdout: "osd_force_recovery_pg_log_entries_factor": "1.300000", 2024-09-18T08:03:49.259 INFO:teuthology.orchestra.run.smithi007.stdout: "osd_function_tracing": "false", 2024-09-18T08:03:49.259 INFO:teuthology.orchestra.run.smithi007.stdout: "osd_heartbeat_grace": "20", 2024-09-18T08:03:49.259 INFO:teuthology.orchestra.run.smithi007.stdout: "osd_heartbeat_interval": "6", 2024-09-18T08:03:49.259 INFO:teuthology.orchestra.run.smithi007.stdout: "osd_heartbeat_min_healthy_ratio": "0.330000", 2024-09-18T08:03:49.259 INFO:teuthology.orchestra.run.smithi007.stdout: "osd_heartbeat_min_peers": "10", 2024-09-18T08:03:49.259 INFO:teuthology.orchestra.run.smithi007.stdout: "osd_heartbeat_min_size": "2000", 2024-09-18T08:03:49.260 INFO:teuthology.orchestra.run.smithi007.stdout: "osd_heartbeat_stale": "600", 2024-09-18T08:03:49.260 INFO:teuthology.orchestra.run.smithi007.stdout: "osd_heartbeat_use_min_delay_socket": "false", 2024-09-18T08:03:49.260 INFO:teuthology.orchestra.run.smithi007.stdout: "osd_hit_set_max_size": "100000", 2024-09-18T08:03:49.260 INFO:teuthology.orchestra.run.smithi007.stdout: "osd_hit_set_min_size": "1000", 2024-09-18T08:03:49.260 INFO:teuthology.orchestra.run.smithi007.stdout: "osd_hit_set_namespace": ".ceph-internal", 2024-09-18T08:03:49.260 INFO:teuthology.orchestra.run.smithi007.stdout: "osd_ignore_stale_divergent_priors": "false", 2024-09-18T08:03:49.260 INFO:teuthology.orchestra.run.smithi007.stdout: "osd_inject_bad_map_crc_probability": "0.000000", 2024-09-18T08:03:49.260 INFO:teuthology.orchestra.run.smithi007.stdout: "osd_inject_failure_on_pg_removal": "false", 2024-09-18T08:03:49.260 INFO:teuthology.orchestra.run.smithi007.stdout: "osd_journal": "/var/lib/ceph/osd/ceph-a/journal", 2024-09-18T08:03:49.260 INFO:teuthology.orchestra.run.smithi007.stdout: "osd_journal_flush_on_shutdown": "true", 2024-09-18T08:03:49.260 INFO:teuthology.orchestra.run.smithi007.stdout: "osd_journal_size": "5120", 2024-09-18T08:03:49.260 INFO:teuthology.orchestra.run.smithi007.stdout: "osd_kill_backfill_at": "0", 2024-09-18T08:03:49.261 INFO:teuthology.orchestra.run.smithi007.stdout: "osd_loop_before_reset_tphandle": "64", 2024-09-18T08:03:49.261 INFO:teuthology.orchestra.run.smithi007.stdout: "osd_map_cache_size": "50", 2024-09-18T08:03:49.261 INFO:teuthology.orchestra.run.smithi007.stdout: "osd_map_dedup": "true", 2024-09-18T08:03:49.261 INFO:teuthology.orchestra.run.smithi007.stdout: "osd_map_message_max": "40", 2024-09-18T08:03:49.261 INFO:teuthology.orchestra.run.smithi007.stdout: "osd_map_message_max_bytes": "10485760", 2024-09-18T08:03:49.261 INFO:teuthology.orchestra.run.smithi007.stdout: "osd_map_share_max_epochs": "40", 2024-09-18T08:03:49.261 INFO:teuthology.orchestra.run.smithi007.stdout: "osd_max_attr_name_len": "100", 2024-09-18T08:03:49.261 INFO:teuthology.orchestra.run.smithi007.stdout: "osd_max_attr_size": "0", 2024-09-18T08:03:49.261 INFO:teuthology.orchestra.run.smithi007.stdout: "osd_max_backfills": "1", 2024-09-18T08:03:49.261 INFO:teuthology.orchestra.run.smithi007.stdout: "osd_max_markdown_count": "5", 2024-09-18T08:03:49.261 INFO:teuthology.orchestra.run.smithi007.stdout: "osd_max_markdown_period": "600", 2024-09-18T08:03:49.261 INFO:teuthology.orchestra.run.smithi007.stdout: "osd_max_object_name_len": "2048", 2024-09-18T08:03:49.261 INFO:teuthology.orchestra.run.smithi007.stdout: "osd_max_object_namespace_len": "256", 2024-09-18T08:03:49.262 INFO:teuthology.orchestra.run.smithi007.stdout: "osd_max_object_size": "134217728", 2024-09-18T08:03:49.262 INFO:teuthology.orchestra.run.smithi007.stdout: "osd_max_omap_bytes_per_request": "1073741824", 2024-09-18T08:03:49.262 INFO:teuthology.orchestra.run.smithi007.stdout: "osd_max_omap_entries_per_request": "1024", 2024-09-18T08:03:49.262 INFO:teuthology.orchestra.run.smithi007.stdout: "osd_max_pg_blocked_by": "16", 2024-09-18T08:03:49.262 INFO:teuthology.orchestra.run.smithi007.stdout: "osd_max_pg_log_entries": "10000", 2024-09-18T08:03:49.262 INFO:teuthology.orchestra.run.smithi007.stdout: "osd_max_pg_per_osd_hard_ratio": "3.000000", 2024-09-18T08:03:49.262 INFO:teuthology.orchestra.run.smithi007.stdout: "osd_max_pgls": "1024", 2024-09-18T08:03:49.262 INFO:teuthology.orchestra.run.smithi007.stdout: "osd_max_push_cost": "8388608", 2024-09-18T08:03:49.262 INFO:teuthology.orchestra.run.smithi007.stdout: "osd_max_push_objects": "10", 2024-09-18T08:03:49.262 INFO:teuthology.orchestra.run.smithi007.stdout: "osd_max_scrubs": "3", 2024-09-18T08:03:49.262 INFO:teuthology.orchestra.run.smithi007.stdout: "osd_max_snap_prune_intervals_per_epoch": "512", 2024-09-18T08:03:49.262 INFO:teuthology.orchestra.run.smithi007.stdout: "osd_max_trimming_pgs": "2", 2024-09-18T08:03:49.262 INFO:teuthology.orchestra.run.smithi007.stdout: "osd_max_write_op_reply_len": "64", 2024-09-18T08:03:49.263 INFO:teuthology.orchestra.run.smithi007.stdout: "osd_max_write_size": "90", 2024-09-18T08:03:49.263 INFO:teuthology.orchestra.run.smithi007.stdout: "osd_mclock_force_run_benchmark_on_init": "false", 2024-09-18T08:03:49.263 INFO:teuthology.orchestra.run.smithi007.stdout: "osd_mclock_iops_capacity_threshold_hdd": "500.000000", 2024-09-18T08:03:49.263 INFO:teuthology.orchestra.run.smithi007.stdout: "osd_mclock_iops_capacity_threshold_ssd": "80000.000000", 2024-09-18T08:03:49.263 INFO:teuthology.orchestra.run.smithi007.stdout: "osd_mclock_max_capacity_iops_hdd": "315.000000", 2024-09-18T08:03:49.263 INFO:teuthology.orchestra.run.smithi007.stdout: "osd_mclock_max_capacity_iops_ssd": "21500.000000", 2024-09-18T08:03:49.263 INFO:teuthology.orchestra.run.smithi007.stdout: "osd_mclock_max_sequential_bandwidth_hdd": "157286400", 2024-09-18T08:03:49.263 INFO:teuthology.orchestra.run.smithi007.stdout: "osd_mclock_max_sequential_bandwidth_ssd": "1258291200", 2024-09-18T08:03:49.263 INFO:teuthology.orchestra.run.smithi007.stdout: "osd_mclock_override_recovery_settings": "false", 2024-09-18T08:03:49.263 INFO:teuthology.orchestra.run.smithi007.stdout: "osd_mclock_profile": "balanced", 2024-09-18T08:03:49.263 INFO:teuthology.orchestra.run.smithi007.stdout: "osd_mclock_scheduler_anticipation_timeout": "0.000000", 2024-09-18T08:03:49.263 INFO:teuthology.orchestra.run.smithi007.stdout: "osd_mclock_scheduler_background_best_effort_lim": "0.000000", 2024-09-18T08:03:49.264 INFO:teuthology.orchestra.run.smithi007.stdout: "osd_mclock_scheduler_background_best_effort_res": "0.000000", 2024-09-18T08:03:49.264 INFO:teuthology.orchestra.run.smithi007.stdout: "osd_mclock_scheduler_background_best_effort_wgt": "1", 2024-09-18T08:03:49.264 INFO:teuthology.orchestra.run.smithi007.stdout: "osd_mclock_scheduler_background_recovery_lim": "0.000000", 2024-09-18T08:03:49.264 INFO:teuthology.orchestra.run.smithi007.stdout: "osd_mclock_scheduler_background_recovery_res": "0.000000", 2024-09-18T08:03:49.264 INFO:teuthology.orchestra.run.smithi007.stdout: "osd_mclock_scheduler_background_recovery_wgt": "1", 2024-09-18T08:03:49.264 INFO:teuthology.orchestra.run.smithi007.stdout: "osd_mclock_scheduler_client_lim": "0.000000", 2024-09-18T08:03:49.264 INFO:teuthology.orchestra.run.smithi007.stdout: "osd_mclock_scheduler_client_res": "0.000000", 2024-09-18T08:03:49.264 INFO:teuthology.orchestra.run.smithi007.stdout: "osd_mclock_scheduler_client_wgt": "1", 2024-09-18T08:03:49.265 INFO:teuthology.orchestra.run.smithi007.stdout: "osd_mclock_skip_benchmark": "false", 2024-09-18T08:03:49.265 INFO:teuthology.orchestra.run.smithi007.stdout: "osd_memory_base": "805306368", 2024-09-18T08:03:49.265 INFO:teuthology.orchestra.run.smithi007.stdout: "osd_memory_cache_min": "134217728", 2024-09-18T08:03:49.265 INFO:teuthology.orchestra.run.smithi007.stdout: "osd_memory_cache_resize_interval": "1.000000", 2024-09-18T08:03:49.265 INFO:teuthology.orchestra.run.smithi007.stdout: "osd_memory_expected_fragmentation": "0.150000", 2024-09-18T08:03:49.265 INFO:teuthology.orchestra.run.smithi007.stdout: "osd_memory_target": "4294967296", 2024-09-18T08:03:49.266 INFO:teuthology.orchestra.run.smithi007.stdout: "osd_memory_target_autotune": "false", 2024-09-18T08:03:49.266 INFO:teuthology.orchestra.run.smithi007.stdout: "osd_memory_target_cgroup_limit_ratio": "0.800000", 2024-09-18T08:03:49.266 INFO:teuthology.orchestra.run.smithi007.stdout: "osd_min_pg_log_entries": "250", 2024-09-18T08:03:49.266 INFO:teuthology.orchestra.run.smithi007.stdout: "osd_min_recovery_priority": "0", 2024-09-18T08:03:49.266 INFO:teuthology.orchestra.run.smithi007.stdout: "osd_mon_heartbeat_interval": "30", 2024-09-18T08:03:49.266 INFO:teuthology.orchestra.run.smithi007.stdout: "osd_mon_heartbeat_stat_stale": "3600", 2024-09-18T08:03:49.266 INFO:teuthology.orchestra.run.smithi007.stdout: "osd_mon_report_interval": "5", 2024-09-18T08:03:49.266 INFO:teuthology.orchestra.run.smithi007.stdout: "osd_mon_report_max_in_flight": "2", 2024-09-18T08:03:49.266 INFO:teuthology.orchestra.run.smithi007.stdout: "osd_mon_shutdown_timeout": "5.000000", 2024-09-18T08:03:49.266 INFO:teuthology.orchestra.run.smithi007.stdout: "osd_num_cache_shards": "32", 2024-09-18T08:03:49.267 INFO:teuthology.orchestra.run.smithi007.stdout: "osd_num_op_tracker_shard": "32", 2024-09-18T08:03:49.267 INFO:teuthology.orchestra.run.smithi007.stdout: "osd_numa_auto_affinity": "true", 2024-09-18T08:03:49.267 INFO:teuthology.orchestra.run.smithi007.stdout: "osd_numa_node": "-1", 2024-09-18T08:03:49.267 INFO:teuthology.orchestra.run.smithi007.stdout: "osd_numa_prefer_iface": "true", 2024-09-18T08:03:49.267 INFO:teuthology.orchestra.run.smithi007.stdout: "osd_object_clean_region_max_num_intervals": "10", 2024-09-18T08:03:49.267 INFO:teuthology.orchestra.run.smithi007.stdout: "osd_objecter_finishers": "1", 2024-09-18T08:03:49.267 INFO:teuthology.orchestra.run.smithi007.stdout: "osd_objectstore": "bluestore", 2024-09-18T08:03:49.267 INFO:teuthology.orchestra.run.smithi007.stdout: "osd_objectstore_fuse": "false", 2024-09-18T08:03:49.267 INFO:teuthology.orchestra.run.smithi007.stdout: "osd_objectstore_ideal_list_max": "64", 2024-09-18T08:03:49.268 INFO:teuthology.orchestra.run.smithi007.stdout: "osd_objectstore_tracing": "false", 2024-09-18T08:03:49.268 INFO:teuthology.orchestra.run.smithi007.stdout: "osd_op_complaint_time": "30.000000", 2024-09-18T08:03:49.268 INFO:teuthology.orchestra.run.smithi007.stdout: "osd_op_history_duration": "600", 2024-09-18T08:03:49.268 INFO:teuthology.orchestra.run.smithi007.stdout: "osd_op_history_size": "20", 2024-09-18T08:03:49.268 INFO:teuthology.orchestra.run.smithi007.stdout: "osd_op_history_slow_op_size": "20", 2024-09-18T08:03:49.268 INFO:teuthology.orchestra.run.smithi007.stdout: "osd_op_history_slow_op_threshold": "10.000000", 2024-09-18T08:03:49.268 INFO:teuthology.orchestra.run.smithi007.stdout: "osd_op_log_threshold": "5", 2024-09-18T08:03:49.268 INFO:teuthology.orchestra.run.smithi007.stdout: "osd_op_num_shards": "0", 2024-09-18T08:03:49.268 INFO:teuthology.orchestra.run.smithi007.stdout: "osd_op_num_shards_hdd": "5", 2024-09-18T08:03:49.268 INFO:teuthology.orchestra.run.smithi007.stdout: "osd_op_num_shards_ssd": "8", 2024-09-18T08:03:49.269 INFO:teuthology.orchestra.run.smithi007.stdout: "osd_op_num_threads_per_shard": "0", 2024-09-18T08:03:49.269 INFO:teuthology.orchestra.run.smithi007.stdout: "osd_op_num_threads_per_shard_hdd": "1", 2024-09-18T08:03:49.269 INFO:teuthology.orchestra.run.smithi007.stdout: "osd_op_num_threads_per_shard_ssd": "2", 2024-09-18T08:03:49.269 INFO:teuthology.orchestra.run.smithi007.stdout: "osd_op_pq_max_tokens_per_priority": "4194304", 2024-09-18T08:03:49.269 INFO:teuthology.orchestra.run.smithi007.stdout: "osd_op_pq_min_cost": "65536", 2024-09-18T08:03:49.269 INFO:teuthology.orchestra.run.smithi007.stdout: "osd_op_queue": "mclock_scheduler", 2024-09-18T08:03:49.269 INFO:teuthology.orchestra.run.smithi007.stdout: "osd_op_queue_cut_off": "high", 2024-09-18T08:03:49.269 INFO:teuthology.orchestra.run.smithi007.stdout: "osd_op_thread_suicide_timeout": "150", 2024-09-18T08:03:49.269 INFO:teuthology.orchestra.run.smithi007.stdout: "osd_op_thread_timeout": "15", 2024-09-18T08:03:49.269 INFO:teuthology.orchestra.run.smithi007.stdout: "osd_open_classes_on_start": "true", 2024-09-18T08:03:49.269 INFO:teuthology.orchestra.run.smithi007.stdout: "osd_os_flags": "0", 2024-09-18T08:03:49.270 INFO:teuthology.orchestra.run.smithi007.stdout: "osd_peering_op_priority": "255", 2024-09-18T08:03:49.270 INFO:teuthology.orchestra.run.smithi007.stdout: "osd_pg_delete_cost": "1048576", 2024-09-18T08:03:49.270 INFO:teuthology.orchestra.run.smithi007.stdout: "osd_pg_delete_priority": "5", 2024-09-18T08:03:49.270 INFO:teuthology.orchestra.run.smithi007.stdout: "osd_pg_epoch_max_lag_factor": "2.000000", 2024-09-18T08:03:49.270 INFO:teuthology.orchestra.run.smithi007.stdout: "osd_pg_epoch_persisted_max_stale": "40", 2024-09-18T08:03:49.270 INFO:teuthology.orchestra.run.smithi007.stdout: "osd_pg_log_dups_tracked": "3000", 2024-09-18T08:03:49.270 INFO:teuthology.orchestra.run.smithi007.stdout: "osd_pg_log_trim_max": "10000", 2024-09-18T08:03:49.270 INFO:teuthology.orchestra.run.smithi007.stdout: "osd_pg_log_trim_min": "100", 2024-09-18T08:03:49.270 INFO:teuthology.orchestra.run.smithi007.stdout: "osd_pg_max_concurrent_snap_trims": "2", 2024-09-18T08:03:49.271 INFO:teuthology.orchestra.run.smithi007.stdout: "osd_pg_object_context_cache_count": "64", 2024-09-18T08:03:49.271 INFO:teuthology.orchestra.run.smithi007.stdout: "osd_pg_stat_report_interval_max_epochs": "500", 2024-09-18T08:03:49.271 INFO:teuthology.orchestra.run.smithi007.stdout: "osd_pg_stat_report_interval_max_seconds": "5", 2024-09-18T08:03:49.271 INFO:teuthology.orchestra.run.smithi007.stdout: "osd_pool_default_cache_max_evict_check_size": "10", 2024-09-18T08:03:49.271 INFO:teuthology.orchestra.run.smithi007.stdout: "osd_pool_default_cache_min_evict_age": "0", 2024-09-18T08:03:49.271 INFO:teuthology.orchestra.run.smithi007.stdout: "osd_pool_default_cache_min_flush_age": "0", 2024-09-18T08:03:49.271 INFO:teuthology.orchestra.run.smithi007.stdout: "osd_pool_default_cache_target_dirty_high_ratio": "0.600000", 2024-09-18T08:03:49.271 INFO:teuthology.orchestra.run.smithi007.stdout: "osd_pool_default_cache_target_dirty_ratio": "0.400000", 2024-09-18T08:03:49.271 INFO:teuthology.orchestra.run.smithi007.stdout: "osd_pool_default_cache_target_full_ratio": "0.800000", 2024-09-18T08:03:49.271 INFO:teuthology.orchestra.run.smithi007.stdout: "osd_pool_default_crimson": "false", 2024-09-18T08:03:49.272 INFO:teuthology.orchestra.run.smithi007.stdout: "osd_pool_default_crush_rule": "-1", 2024-09-18T08:03:49.272 INFO:teuthology.orchestra.run.smithi007.stdout: "osd_pool_default_ec_fast_read": "false", 2024-09-18T08:03:49.272 INFO:teuthology.orchestra.run.smithi007.stdout: "osd_pool_default_erasure_code_profile": "plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd", 2024-09-18T08:03:49.272 INFO:teuthology.orchestra.run.smithi007.stdout: "osd_pool_default_flag_bulk": "false", 2024-09-18T08:03:49.272 INFO:teuthology.orchestra.run.smithi007.stdout: "osd_pool_default_flag_hashpspool": "true", 2024-09-18T08:03:49.272 INFO:teuthology.orchestra.run.smithi007.stdout: "osd_pool_default_flag_nodelete": "false", 2024-09-18T08:03:49.272 INFO:teuthology.orchestra.run.smithi007.stdout: "osd_pool_default_flag_nopgchange": "false", 2024-09-18T08:03:49.272 INFO:teuthology.orchestra.run.smithi007.stdout: "osd_pool_default_flag_nosizechange": "false", 2024-09-18T08:03:49.272 INFO:teuthology.orchestra.run.smithi007.stdout: "osd_pool_default_flags": "0", 2024-09-18T08:03:49.272 INFO:teuthology.orchestra.run.smithi007.stdout: "osd_pool_default_hit_set_bloom_fpp": "0.050000", 2024-09-18T08:03:49.273 INFO:teuthology.orchestra.run.smithi007.stdout: "osd_pool_default_min_size": "0", 2024-09-18T08:03:49.273 INFO:teuthology.orchestra.run.smithi007.stdout: "osd_pool_default_pg_autoscale_mode": "off", 2024-09-18T08:03:49.273 INFO:teuthology.orchestra.run.smithi007.stdout: "osd_pool_default_pg_num": "32", 2024-09-18T08:03:49.273 INFO:teuthology.orchestra.run.smithi007.stdout: "osd_pool_default_pgp_num": "0", 2024-09-18T08:03:49.273 INFO:teuthology.orchestra.run.smithi007.stdout: "osd_pool_default_read_lease_ratio": "0.800000", 2024-09-18T08:03:49.273 INFO:teuthology.orchestra.run.smithi007.stdout: "osd_pool_default_read_ratio": "70", 2024-09-18T08:03:49.273 INFO:teuthology.orchestra.run.smithi007.stdout: "osd_pool_default_size": "3", 2024-09-18T08:03:49.273 INFO:teuthology.orchestra.run.smithi007.stdout: "osd_pool_default_type": "replicated", 2024-09-18T08:03:49.273 INFO:teuthology.orchestra.run.smithi007.stdout: "osd_pool_erasure_code_stripe_unit": "4096", 2024-09-18T08:03:49.273 INFO:teuthology.orchestra.run.smithi007.stdout: "osd_pool_use_gmt_hitset": "true", 2024-09-18T08:03:49.273 INFO:teuthology.orchestra.run.smithi007.stdout: "osd_push_per_object_cost": "1000", 2024-09-18T08:03:49.273 INFO:teuthology.orchestra.run.smithi007.stdout: "osd_read_ec_check_for_errors": "false", 2024-09-18T08:03:49.274 INFO:teuthology.orchestra.run.smithi007.stdout: "osd_recover_clone_overlap": "true", 2024-09-18T08:03:49.274 INFO:teuthology.orchestra.run.smithi007.stdout: "osd_recover_clone_overlap_limit": "10", 2024-09-18T08:03:49.274 INFO:teuthology.orchestra.run.smithi007.stdout: "osd_recovery_cost": "20971520", 2024-09-18T08:03:49.274 INFO:teuthology.orchestra.run.smithi007.stdout: "osd_recovery_delay_start": "0.000000", 2024-09-18T08:03:49.274 INFO:teuthology.orchestra.run.smithi007.stdout: "osd_recovery_max_active": "0", 2024-09-18T08:03:49.274 INFO:teuthology.orchestra.run.smithi007.stdout: "osd_recovery_max_active_hdd": "3", 2024-09-18T08:03:49.274 INFO:teuthology.orchestra.run.smithi007.stdout: "osd_recovery_max_active_ssd": "10", 2024-09-18T08:03:49.274 INFO:teuthology.orchestra.run.smithi007.stdout: "osd_recovery_max_chunk": "8388608", 2024-09-18T08:03:49.274 INFO:teuthology.orchestra.run.smithi007.stdout: "osd_recovery_max_omap_entries_per_chunk": "8096", 2024-09-18T08:03:49.274 INFO:teuthology.orchestra.run.smithi007.stdout: "osd_recovery_max_single_start": "1", 2024-09-18T08:03:49.274 INFO:teuthology.orchestra.run.smithi007.stdout: "osd_recovery_op_priority": "3", 2024-09-18T08:03:49.274 INFO:teuthology.orchestra.run.smithi007.stdout: "osd_recovery_op_warn_multiple": "16", 2024-09-18T08:03:49.275 INFO:teuthology.orchestra.run.smithi007.stdout: "osd_recovery_priority": "5", 2024-09-18T08:03:49.275 INFO:teuthology.orchestra.run.smithi007.stdout: "osd_recovery_retry_interval": "30.000000", 2024-09-18T08:03:49.275 INFO:teuthology.orchestra.run.smithi007.stdout: "osd_recovery_sleep": "0.000000", 2024-09-18T08:03:49.275 INFO:teuthology.orchestra.run.smithi007.stdout: "osd_recovery_sleep_hdd": "0.100000", 2024-09-18T08:03:49.275 INFO:teuthology.orchestra.run.smithi007.stdout: "osd_recovery_sleep_hybrid": "0.025000", 2024-09-18T08:03:49.275 INFO:teuthology.orchestra.run.smithi007.stdout: "osd_recovery_sleep_ssd": "0.000000", 2024-09-18T08:03:49.275 INFO:teuthology.orchestra.run.smithi007.stdout: "osd_repair_during_recovery": "false", 2024-09-18T08:03:49.275 INFO:teuthology.orchestra.run.smithi007.stdout: "osd_requested_scrub_priority": "120", 2024-09-18T08:03:49.275 INFO:teuthology.orchestra.run.smithi007.stdout: "osd_rocksdb_iterator_bounds_enabled": "true", 2024-09-18T08:03:49.276 INFO:teuthology.orchestra.run.smithi007.stdout: "osd_rollback_to_cluster_snap": "", 2024-09-18T08:03:49.276 INFO:teuthology.orchestra.run.smithi007.stdout: "osd_scrub_auto_repair": "false", 2024-09-18T08:03:49.276 INFO:teuthology.orchestra.run.smithi007.stdout: "osd_scrub_auto_repair_num_errors": "5", 2024-09-18T08:03:49.276 INFO:teuthology.orchestra.run.smithi007.stdout: "osd_scrub_backoff_ratio": "0.660000", 2024-09-18T08:03:49.276 INFO:teuthology.orchestra.run.smithi007.stdout: "osd_scrub_begin_hour": "0", 2024-09-18T08:03:49.276 INFO:teuthology.orchestra.run.smithi007.stdout: "osd_scrub_begin_week_day": "0", 2024-09-18T08:03:49.276 INFO:teuthology.orchestra.run.smithi007.stdout: "osd_scrub_chunk_max": "15", 2024-09-18T08:03:49.276 INFO:teuthology.orchestra.run.smithi007.stdout: "osd_scrub_chunk_min": "5", 2024-09-18T08:03:49.276 INFO:teuthology.orchestra.run.smithi007.stdout: "osd_scrub_cost": "52428800", 2024-09-18T08:03:49.276 INFO:teuthology.orchestra.run.smithi007.stdout: "osd_scrub_disable_reservation_queuing": "false", 2024-09-18T08:03:49.276 INFO:teuthology.orchestra.run.smithi007.stdout: "osd_scrub_during_recovery": "false", 2024-09-18T08:03:49.276 INFO:teuthology.orchestra.run.smithi007.stdout: "osd_scrub_end_hour": "0", 2024-09-18T08:03:49.277 INFO:teuthology.orchestra.run.smithi007.stdout: "osd_scrub_end_week_day": "0", 2024-09-18T08:03:49.277 INFO:teuthology.orchestra.run.smithi007.stdout: "osd_scrub_event_cost": "4096", 2024-09-18T08:03:49.277 INFO:teuthology.orchestra.run.smithi007.stdout: "osd_scrub_extended_sleep": "0.000000", 2024-09-18T08:03:49.277 INFO:teuthology.orchestra.run.smithi007.stdout: "osd_scrub_interval_randomize_ratio": "0.500000", 2024-09-18T08:03:49.277 INFO:teuthology.orchestra.run.smithi007.stdout: "osd_scrub_invalid_stats": "true", 2024-09-18T08:03:49.277 INFO:teuthology.orchestra.run.smithi007.stdout: "osd_scrub_load_threshold": "0.500000", 2024-09-18T08:03:49.277 INFO:teuthology.orchestra.run.smithi007.stdout: "osd_scrub_max_interval": "604800.000000", 2024-09-18T08:03:49.277 INFO:teuthology.orchestra.run.smithi007.stdout: "osd_scrub_max_preemptions": "5", 2024-09-18T08:03:49.278 INFO:teuthology.orchestra.run.smithi007.stdout: "osd_scrub_min_interval": "86400.000000", 2024-09-18T08:03:49.278 INFO:teuthology.orchestra.run.smithi007.stdout: "osd_scrub_priority": "5", 2024-09-18T08:03:49.278 INFO:teuthology.orchestra.run.smithi007.stdout: "osd_scrub_retry_after_noscrub": "60", 2024-09-18T08:03:49.278 INFO:teuthology.orchestra.run.smithi007.stdout: "osd_scrub_retry_delay": "30", 2024-09-18T08:03:49.278 INFO:teuthology.orchestra.run.smithi007.stdout: "osd_scrub_retry_new_interval": "10", 2024-09-18T08:03:49.278 INFO:teuthology.orchestra.run.smithi007.stdout: "osd_scrub_retry_pg_state": "60", 2024-09-18T08:03:49.278 INFO:teuthology.orchestra.run.smithi007.stdout: "osd_scrub_retry_trimming": "10", 2024-09-18T08:03:49.278 INFO:teuthology.orchestra.run.smithi007.stdout: "osd_scrub_sleep": "0.000000", 2024-09-18T08:03:49.279 INFO:teuthology.orchestra.run.smithi007.stdout: "osd_shallow_scrub_chunk_max": "100", 2024-09-18T08:03:49.279 INFO:teuthology.orchestra.run.smithi007.stdout: "osd_shallow_scrub_chunk_min": "50", 2024-09-18T08:03:49.279 INFO:teuthology.orchestra.run.smithi007.stdout: "osd_shutdown_pgref_assert": "false", 2024-09-18T08:03:49.279 INFO:teuthology.orchestra.run.smithi007.stdout: "osd_skip_check_past_interval_bounds": "false", 2024-09-18T08:03:49.279 INFO:teuthology.orchestra.run.smithi007.stdout: "osd_skip_data_digest": "false", 2024-09-18T08:03:49.279 INFO:teuthology.orchestra.run.smithi007.stdout: "osd_smart_report_timeout": "5", 2024-09-18T08:03:49.279 INFO:teuthology.orchestra.run.smithi007.stdout: "osd_snap_trim_cost": "1048576", 2024-09-18T08:03:49.279 INFO:teuthology.orchestra.run.smithi007.stdout: "osd_snap_trim_priority": "5", 2024-09-18T08:03:49.279 INFO:teuthology.orchestra.run.smithi007.stdout: "osd_snap_trim_sleep": "0.000000", 2024-09-18T08:03:49.279 INFO:teuthology.orchestra.run.smithi007.stdout: "osd_snap_trim_sleep_hdd": "5.000000", 2024-09-18T08:03:49.279 INFO:teuthology.orchestra.run.smithi007.stdout: "osd_snap_trim_sleep_hybrid": "2.000000", 2024-09-18T08:03:49.279 INFO:teuthology.orchestra.run.smithi007.stdout: "osd_snap_trim_sleep_ssd": "0.000000", 2024-09-18T08:03:49.280 INFO:teuthology.orchestra.run.smithi007.stdout: "osd_stats_update_period_not_scrubbing": "120", 2024-09-18T08:03:49.280 INFO:teuthology.orchestra.run.smithi007.stdout: "osd_stats_update_period_scrubbing": "15", 2024-09-18T08:03:49.280 INFO:teuthology.orchestra.run.smithi007.stdout: "osd_target_pg_log_entries_per_osd": "300000", 2024-09-18T08:03:49.280 INFO:teuthology.orchestra.run.smithi007.stdout: "osd_target_transaction_size": "30", 2024-09-18T08:03:49.280 INFO:teuthology.orchestra.run.smithi007.stdout: "osd_tier_default_cache_hit_set_count": "4", 2024-09-18T08:03:49.280 INFO:teuthology.orchestra.run.smithi007.stdout: "osd_tier_default_cache_hit_set_grade_decay_rate": "20", 2024-09-18T08:03:49.280 INFO:teuthology.orchestra.run.smithi007.stdout: "osd_tier_default_cache_hit_set_period": "1200", 2024-09-18T08:03:49.280 INFO:teuthology.orchestra.run.smithi007.stdout: "osd_tier_default_cache_hit_set_search_last_n": "1", 2024-09-18T08:03:49.280 INFO:teuthology.orchestra.run.smithi007.stdout: "osd_tier_default_cache_hit_set_type": "bloom", 2024-09-18T08:03:49.280 INFO:teuthology.orchestra.run.smithi007.stdout: "osd_tier_default_cache_min_read_recency_for_promote": "1", 2024-09-18T08:03:49.280 INFO:teuthology.orchestra.run.smithi007.stdout: "osd_tier_default_cache_min_write_recency_for_promote": "1", 2024-09-18T08:03:49.280 INFO:teuthology.orchestra.run.smithi007.stdout: "osd_tier_default_cache_mode": "writeback", 2024-09-18T08:03:49.281 INFO:teuthology.orchestra.run.smithi007.stdout: "osd_tier_promote_max_bytes_sec": "5242880", 2024-09-18T08:03:49.281 INFO:teuthology.orchestra.run.smithi007.stdout: "osd_tier_promote_max_objects_sec": "25", 2024-09-18T08:03:49.281 INFO:teuthology.orchestra.run.smithi007.stdout: "osd_tracing": "false", 2024-09-18T08:03:49.281 INFO:teuthology.orchestra.run.smithi007.stdout: "osd_use_stale_snap": "false", 2024-09-18T08:03:49.281 INFO:teuthology.orchestra.run.smithi007.stdout: "osd_uuid": "00000000-0000-0000-0000-000000000000", 2024-09-18T08:03:49.281 INFO:teuthology.orchestra.run.smithi007.stdout: "osdc_blkin_trace_all": "false", 2024-09-18T08:03:49.281 INFO:teuthology.orchestra.run.smithi007.stdout: "paxos_kill_at": "0", 2024-09-18T08:03:49.281 INFO:teuthology.orchestra.run.smithi007.stdout: "paxos_max_join_drift": "10", 2024-09-18T08:03:49.281 INFO:teuthology.orchestra.run.smithi007.stdout: "paxos_min": "500", 2024-09-18T08:03:49.281 INFO:teuthology.orchestra.run.smithi007.stdout: "paxos_min_wait": "0.050000", 2024-09-18T08:03:49.281 INFO:teuthology.orchestra.run.smithi007.stdout: "paxos_propose_interval": "1.000000", 2024-09-18T08:03:49.281 INFO:teuthology.orchestra.run.smithi007.stdout: "paxos_service_trim_max": "500", 2024-09-18T08:03:49.281 INFO:teuthology.orchestra.run.smithi007.stdout: "paxos_service_trim_max_multiplier": "20", 2024-09-18T08:03:49.282 INFO:teuthology.orchestra.run.smithi007.stdout: "paxos_service_trim_min": "10", 2024-09-18T08:03:49.282 INFO:teuthology.orchestra.run.smithi007.stdout: "paxos_stash_full_interval": "25", 2024-09-18T08:03:49.282 INFO:teuthology.orchestra.run.smithi007.stdout: "paxos_trim_max": "500", 2024-09-18T08:03:49.282 INFO:teuthology.orchestra.run.smithi007.stdout: "paxos_trim_min": "250", 2024-09-18T08:03:49.282 INFO:teuthology.orchestra.run.smithi007.stdout: "perf": "true", 2024-09-18T08:03:49.282 INFO:teuthology.orchestra.run.smithi007.stdout: "pid_file": "", 2024-09-18T08:03:49.282 INFO:teuthology.orchestra.run.smithi007.stdout: "plugin_crypto_accelerator": "crypto_isal", 2024-09-18T08:03:49.282 INFO:teuthology.orchestra.run.smithi007.stdout: "plugin_dir": "/usr/lib64/ceph", 2024-09-18T08:03:49.282 INFO:teuthology.orchestra.run.smithi007.stdout: "public_addr": "-", 2024-09-18T08:03:49.282 INFO:teuthology.orchestra.run.smithi007.stdout: "public_addrv": "", 2024-09-18T08:03:49.282 INFO:teuthology.orchestra.run.smithi007.stdout: "public_bind_addr": "-", 2024-09-18T08:03:49.282 INFO:teuthology.orchestra.run.smithi007.stdout: "public_network": "", 2024-09-18T08:03:49.283 INFO:teuthology.orchestra.run.smithi007.stdout: "public_network_interface": "", 2024-09-18T08:03:49.283 INFO:teuthology.orchestra.run.smithi007.stdout: "qat_compressor_busy_polling": "false", 2024-09-18T08:03:49.283 INFO:teuthology.orchestra.run.smithi007.stdout: "qat_compressor_enabled": "false", 2024-09-18T08:03:49.283 INFO:teuthology.orchestra.run.smithi007.stdout: "qat_compressor_session_max_number": "256", 2024-09-18T08:03:49.283 INFO:teuthology.orchestra.run.smithi007.stdout: "rados_mon_op_timeout": "0", 2024-09-18T08:03:49.283 INFO:teuthology.orchestra.run.smithi007.stdout: "rados_osd_op_timeout": "0", 2024-09-18T08:03:49.283 INFO:teuthology.orchestra.run.smithi007.stdout: "rados_tracing": "false", 2024-09-18T08:03:49.283 INFO:teuthology.orchestra.run.smithi007.stdout: "rbd_atime_update_interval": "60", 2024-09-18T08:03:49.283 INFO:teuthology.orchestra.run.smithi007.stdout: "rbd_auto_exclusive_lock_until_manual_request": "true", 2024-09-18T08:03:49.283 INFO:teuthology.orchestra.run.smithi007.stdout: "rbd_balance_parent_reads": "false", 2024-09-18T08:03:49.283 INFO:teuthology.orchestra.run.smithi007.stdout: "rbd_balance_snap_reads": "false", 2024-09-18T08:03:49.283 INFO:teuthology.orchestra.run.smithi007.stdout: "rbd_blkin_trace_all": "false", 2024-09-18T08:03:49.283 INFO:teuthology.orchestra.run.smithi007.stdout: "rbd_blocklist_expire_seconds": "0", 2024-09-18T08:03:49.284 INFO:teuthology.orchestra.run.smithi007.stdout: "rbd_blocklist_on_break_lock": "true", 2024-09-18T08:03:49.284 INFO:teuthology.orchestra.run.smithi007.stdout: "rbd_cache": "true", 2024-09-18T08:03:49.284 INFO:teuthology.orchestra.run.smithi007.stdout: "rbd_cache_block_writes_upfront": "false", 2024-09-18T08:03:49.284 INFO:teuthology.orchestra.run.smithi007.stdout: "rbd_cache_max_dirty": "25165824", 2024-09-18T08:03:49.284 INFO:teuthology.orchestra.run.smithi007.stdout: "rbd_cache_max_dirty_age": "1.000000", 2024-09-18T08:03:49.284 INFO:teuthology.orchestra.run.smithi007.stdout: "rbd_cache_max_dirty_object": "0", 2024-09-18T08:03:49.284 INFO:teuthology.orchestra.run.smithi007.stdout: "rbd_cache_policy": "writearound", 2024-09-18T08:03:49.284 INFO:teuthology.orchestra.run.smithi007.stdout: "rbd_cache_size": "33554432", 2024-09-18T08:03:49.284 INFO:teuthology.orchestra.run.smithi007.stdout: "rbd_cache_target_dirty": "16777216", 2024-09-18T08:03:49.284 INFO:teuthology.orchestra.run.smithi007.stdout: "rbd_cache_writethrough_until_flush": "true", 2024-09-18T08:03:49.284 INFO:teuthology.orchestra.run.smithi007.stdout: "rbd_clone_copy_on_read": "false", 2024-09-18T08:03:49.284 INFO:teuthology.orchestra.run.smithi007.stdout: "rbd_compression_hint": "none", 2024-09-18T08:03:49.285 INFO:teuthology.orchestra.run.smithi007.stdout: "rbd_concurrent_management_ops": "10", 2024-09-18T08:03:49.285 INFO:teuthology.orchestra.run.smithi007.stdout: "rbd_config_pool_override_update_timestamp": "0", 2024-09-18T08:03:49.285 INFO:teuthology.orchestra.run.smithi007.stdout: "rbd_default_clone_format": "auto", 2024-09-18T08:03:49.285 INFO:teuthology.orchestra.run.smithi007.stdout: "rbd_default_data_pool": "", 2024-09-18T08:03:49.285 INFO:teuthology.orchestra.run.smithi007.stdout: "rbd_default_features": "61", 2024-09-18T08:03:49.285 INFO:teuthology.orchestra.run.smithi007.stdout: "rbd_default_format": "2", 2024-09-18T08:03:49.285 INFO:teuthology.orchestra.run.smithi007.stdout: "rbd_default_map_options": "", 2024-09-18T08:03:49.285 INFO:teuthology.orchestra.run.smithi007.stdout: "rbd_default_order": "22", 2024-09-18T08:03:49.285 INFO:teuthology.orchestra.run.smithi007.stdout: "rbd_default_pool": "rbd", 2024-09-18T08:03:49.285 INFO:teuthology.orchestra.run.smithi007.stdout: "rbd_default_snapshot_quiesce_mode": "required", 2024-09-18T08:03:49.285 INFO:teuthology.orchestra.run.smithi007.stdout: "rbd_default_stripe_count": "0", 2024-09-18T08:03:49.285 INFO:teuthology.orchestra.run.smithi007.stdout: "rbd_default_stripe_unit": "0", 2024-09-18T08:03:49.285 INFO:teuthology.orchestra.run.smithi007.stdout: "rbd_disable_zero_copy_writes": "true", 2024-09-18T08:03:49.286 INFO:teuthology.orchestra.run.smithi007.stdout: "rbd_discard_granularity_bytes": "65536", 2024-09-18T08:03:49.286 INFO:teuthology.orchestra.run.smithi007.stdout: "rbd_discard_on_zeroed_write_same": "true", 2024-09-18T08:03:49.286 INFO:teuthology.orchestra.run.smithi007.stdout: "rbd_enable_alloc_hint": "true", 2024-09-18T08:03:49.286 INFO:teuthology.orchestra.run.smithi007.stdout: "rbd_invalidate_object_map_on_timeout": "true", 2024-09-18T08:03:49.286 INFO:teuthology.orchestra.run.smithi007.stdout: "rbd_io_scheduler": "simple", 2024-09-18T08:03:49.286 INFO:teuthology.orchestra.run.smithi007.stdout: "rbd_io_scheduler_simple_max_delay": "0", 2024-09-18T08:03:49.286 INFO:teuthology.orchestra.run.smithi007.stdout: "rbd_journal_commit_age": "5.000000", 2024-09-18T08:03:49.286 INFO:teuthology.orchestra.run.smithi007.stdout: "rbd_journal_max_concurrent_object_sets": "0", 2024-09-18T08:03:49.286 INFO:teuthology.orchestra.run.smithi007.stdout: "rbd_journal_max_payload_bytes": "16384", 2024-09-18T08:03:49.286 INFO:teuthology.orchestra.run.smithi007.stdout: "rbd_journal_object_flush_age": "0.000000", 2024-09-18T08:03:49.286 INFO:teuthology.orchestra.run.smithi007.stdout: "rbd_journal_object_flush_bytes": "1048576", 2024-09-18T08:03:49.286 INFO:teuthology.orchestra.run.smithi007.stdout: "rbd_journal_object_flush_interval": "0", 2024-09-18T08:03:49.287 INFO:teuthology.orchestra.run.smithi007.stdout: "rbd_journal_object_max_in_flight_appends": "0", 2024-09-18T08:03:49.287 INFO:teuthology.orchestra.run.smithi007.stdout: "rbd_journal_object_writethrough_until_flush": "true", 2024-09-18T08:03:49.287 INFO:teuthology.orchestra.run.smithi007.stdout: "rbd_journal_order": "24", 2024-09-18T08:03:49.287 INFO:teuthology.orchestra.run.smithi007.stdout: "rbd_journal_pool": "", 2024-09-18T08:03:49.287 INFO:teuthology.orchestra.run.smithi007.stdout: "rbd_journal_splay_width": "4", 2024-09-18T08:03:49.287 INFO:teuthology.orchestra.run.smithi007.stdout: "rbd_localize_parent_reads": "false", 2024-09-18T08:03:49.287 INFO:teuthology.orchestra.run.smithi007.stdout: "rbd_localize_snap_reads": "false", 2024-09-18T08:03:49.288 INFO:teuthology.orchestra.run.smithi007.stdout: "rbd_mirror_concurrent_image_deletions": "1", 2024-09-18T08:03:49.289 INFO:teuthology.orchestra.run.smithi007.stdout: "rbd_mirror_concurrent_image_syncs": "5", 2024-09-18T08:03:49.289 INFO:teuthology.orchestra.run.smithi007.stdout: "rbd_mirror_delete_retry_interval": "30.000000", 2024-09-18T08:03:49.289 INFO:teuthology.orchestra.run.smithi007.stdout: "rbd_mirror_image_perf_stats_prio": "5", 2024-09-18T08:03:49.289 INFO:teuthology.orchestra.run.smithi007.stdout: "rbd_mirror_image_policy_migration_throttle": "300", 2024-09-18T08:03:49.289 INFO:teuthology.orchestra.run.smithi007.stdout: "rbd_mirror_image_policy_rebalance_timeout": "0.000000", 2024-09-18T08:03:49.289 INFO:teuthology.orchestra.run.smithi007.stdout: "rbd_mirror_image_policy_type": "simple", 2024-09-18T08:03:49.289 INFO:teuthology.orchestra.run.smithi007.stdout: "rbd_mirror_image_policy_update_throttle_interval": "1.000000", 2024-09-18T08:03:49.289 INFO:teuthology.orchestra.run.smithi007.stdout: "rbd_mirror_image_state_check_interval": "30", 2024-09-18T08:03:49.289 INFO:teuthology.orchestra.run.smithi007.stdout: "rbd_mirror_journal_commit_age": "5.000000", 2024-09-18T08:03:49.289 INFO:teuthology.orchestra.run.smithi007.stdout: "rbd_mirror_journal_poll_age": "5.000000", 2024-09-18T08:03:49.290 INFO:teuthology.orchestra.run.smithi007.stdout: "rbd_mirror_leader_heartbeat_interval": "5", 2024-09-18T08:03:49.290 INFO:teuthology.orchestra.run.smithi007.stdout: "rbd_mirror_leader_max_acquire_attempts_before_break": "3", 2024-09-18T08:03:49.290 INFO:teuthology.orchestra.run.smithi007.stdout: "rbd_mirror_leader_max_missed_heartbeats": "2", 2024-09-18T08:03:49.290 INFO:teuthology.orchestra.run.smithi007.stdout: "rbd_mirror_memory_autotune": "true", 2024-09-18T08:03:49.290 INFO:teuthology.orchestra.run.smithi007.stdout: "rbd_mirror_memory_base": "805306368", 2024-09-18T08:03:49.290 INFO:teuthology.orchestra.run.smithi007.stdout: "rbd_mirror_memory_cache_autotune_interval": "30.000000", 2024-09-18T08:03:49.290 INFO:teuthology.orchestra.run.smithi007.stdout: "rbd_mirror_memory_cache_min": "134217728", 2024-09-18T08:03:49.290 INFO:teuthology.orchestra.run.smithi007.stdout: "rbd_mirror_memory_cache_resize_interval": "5.000000", 2024-09-18T08:03:49.290 INFO:teuthology.orchestra.run.smithi007.stdout: "rbd_mirror_memory_expected_fragmentation": "0.150000", 2024-09-18T08:03:49.290 INFO:teuthology.orchestra.run.smithi007.stdout: "rbd_mirror_memory_target": "4294967296", 2024-09-18T08:03:49.290 INFO:teuthology.orchestra.run.smithi007.stdout: "rbd_mirror_perf_stats_prio": "5", 2024-09-18T08:03:49.290 INFO:teuthology.orchestra.run.smithi007.stdout: "rbd_mirror_pool_replayers_refresh_interval": "30", 2024-09-18T08:03:49.291 INFO:teuthology.orchestra.run.smithi007.stdout: "rbd_mirror_sync_point_update_age": "30.000000", 2024-09-18T08:03:49.291 INFO:teuthology.orchestra.run.smithi007.stdout: "rbd_mirroring_delete_delay": "0", 2024-09-18T08:03:49.291 INFO:teuthology.orchestra.run.smithi007.stdout: "rbd_mirroring_max_mirroring_snapshots": "5", 2024-09-18T08:03:49.291 INFO:teuthology.orchestra.run.smithi007.stdout: "rbd_mirroring_replay_delay": "0", 2024-09-18T08:03:49.291 INFO:teuthology.orchestra.run.smithi007.stdout: "rbd_mirroring_resync_after_disconnect": "false", 2024-09-18T08:03:49.291 INFO:teuthology.orchestra.run.smithi007.stdout: "rbd_move_parent_to_trash_on_remove": "false", 2024-09-18T08:03:49.291 INFO:teuthology.orchestra.run.smithi007.stdout: "rbd_move_to_trash_on_remove": "false", 2024-09-18T08:03:49.291 INFO:teuthology.orchestra.run.smithi007.stdout: "rbd_move_to_trash_on_remove_expire_seconds": "0", 2024-09-18T08:03:49.291 INFO:teuthology.orchestra.run.smithi007.stdout: "rbd_mtime_update_interval": "60", 2024-09-18T08:03:49.291 INFO:teuthology.orchestra.run.smithi007.stdout: "rbd_non_blocking_aio": "true", 2024-09-18T08:03:49.291 INFO:teuthology.orchestra.run.smithi007.stdout: "rbd_op_thread_timeout": "60", 2024-09-18T08:03:49.291 INFO:teuthology.orchestra.run.smithi007.stdout: "rbd_op_threads": "1", 2024-09-18T08:03:49.292 INFO:teuthology.orchestra.run.smithi007.stdout: "rbd_parent_cache_enabled": "false", 2024-09-18T08:03:49.292 INFO:teuthology.orchestra.run.smithi007.stdout: "rbd_persistent_cache_mode": "disabled", 2024-09-18T08:03:49.292 INFO:teuthology.orchestra.run.smithi007.stdout: "rbd_persistent_cache_path": "/tmp", 2024-09-18T08:03:49.292 INFO:teuthology.orchestra.run.smithi007.stdout: "rbd_persistent_cache_size": "1073741824", 2024-09-18T08:03:49.292 INFO:teuthology.orchestra.run.smithi007.stdout: "rbd_plugins": "", 2024-09-18T08:03:49.292 INFO:teuthology.orchestra.run.smithi007.stdout: "rbd_qos_bps_burst": "0", 2024-09-18T08:03:49.292 INFO:teuthology.orchestra.run.smithi007.stdout: "rbd_qos_bps_burst_seconds": "1", 2024-09-18T08:03:49.292 INFO:teuthology.orchestra.run.smithi007.stdout: "rbd_qos_bps_limit": "0", 2024-09-18T08:03:49.292 INFO:teuthology.orchestra.run.smithi007.stdout: "rbd_qos_exclude_ops": "0", 2024-09-18T08:03:49.292 INFO:teuthology.orchestra.run.smithi007.stdout: "rbd_qos_iops_burst": "0", 2024-09-18T08:03:49.292 INFO:teuthology.orchestra.run.smithi007.stdout: "rbd_qos_iops_burst_seconds": "1", 2024-09-18T08:03:49.292 INFO:teuthology.orchestra.run.smithi007.stdout: "rbd_qos_iops_limit": "0", 2024-09-18T08:03:49.293 INFO:teuthology.orchestra.run.smithi007.stdout: "rbd_qos_read_bps_burst": "0", 2024-09-18T08:03:49.293 INFO:teuthology.orchestra.run.smithi007.stdout: "rbd_qos_read_bps_burst_seconds": "1", 2024-09-18T08:03:49.293 INFO:teuthology.orchestra.run.smithi007.stdout: "rbd_qos_read_bps_limit": "0", 2024-09-18T08:03:49.293 INFO:teuthology.orchestra.run.smithi007.stdout: "rbd_qos_read_iops_burst": "0", 2024-09-18T08:03:49.293 INFO:teuthology.orchestra.run.smithi007.stdout: "rbd_qos_read_iops_burst_seconds": "1", 2024-09-18T08:03:49.293 INFO:teuthology.orchestra.run.smithi007.stdout: "rbd_qos_read_iops_limit": "0", 2024-09-18T08:03:49.293 INFO:teuthology.orchestra.run.smithi007.stdout: "rbd_qos_schedule_tick_min": "50", 2024-09-18T08:03:49.293 INFO:teuthology.orchestra.run.smithi007.stdout: "rbd_qos_write_bps_burst": "0", 2024-09-18T08:03:49.293 INFO:teuthology.orchestra.run.smithi007.stdout: "rbd_qos_write_bps_burst_seconds": "1", 2024-09-18T08:03:49.293 INFO:teuthology.orchestra.run.smithi007.stdout: "rbd_qos_write_bps_limit": "0", 2024-09-18T08:03:49.293 INFO:teuthology.orchestra.run.smithi007.stdout: "rbd_qos_write_iops_burst": "0", 2024-09-18T08:03:49.293 INFO:teuthology.orchestra.run.smithi007.stdout: "rbd_qos_write_iops_burst_seconds": "1", 2024-09-18T08:03:49.294 INFO:teuthology.orchestra.run.smithi007.stdout: "rbd_qos_write_iops_limit": "0", 2024-09-18T08:03:49.294 INFO:teuthology.orchestra.run.smithi007.stdout: "rbd_quiesce_notification_attempts": "10", 2024-09-18T08:03:49.294 INFO:teuthology.orchestra.run.smithi007.stdout: "rbd_read_from_replica_policy": "default", 2024-09-18T08:03:49.294 INFO:teuthology.orchestra.run.smithi007.stdout: "rbd_readahead_disable_after_bytes": "52428800", 2024-09-18T08:03:49.294 INFO:teuthology.orchestra.run.smithi007.stdout: "rbd_readahead_max_bytes": "524288", 2024-09-18T08:03:49.294 INFO:teuthology.orchestra.run.smithi007.stdout: "rbd_readahead_trigger_requests": "10", 2024-09-18T08:03:49.294 INFO:teuthology.orchestra.run.smithi007.stdout: "rbd_request_timed_out_seconds": "30", 2024-09-18T08:03:49.294 INFO:teuthology.orchestra.run.smithi007.stdout: "rbd_skip_partial_discard": "true", 2024-09-18T08:03:49.294 INFO:teuthology.orchestra.run.smithi007.stdout: "rbd_sparse_read_threshold_bytes": "65536", 2024-09-18T08:03:49.294 INFO:teuthology.orchestra.run.smithi007.stdout: "rbd_tracing": "false", 2024-09-18T08:03:49.294 INFO:teuthology.orchestra.run.smithi007.stdout: "rbd_validate_names": "true", 2024-09-18T08:03:49.294 INFO:teuthology.orchestra.run.smithi007.stdout: "rbd_validate_pool": "true", 2024-09-18T08:03:49.294 INFO:teuthology.orchestra.run.smithi007.stdout: "restapi_base_url": "", 2024-09-18T08:03:49.295 INFO:teuthology.orchestra.run.smithi007.stdout: "restapi_log_level": "", 2024-09-18T08:03:49.295 INFO:teuthology.orchestra.run.smithi007.stdout: "rgw_account_default_quota_max_objects": "-1", 2024-09-18T08:03:49.295 INFO:teuthology.orchestra.run.smithi007.stdout: "rgw_account_default_quota_max_size": "-1", 2024-09-18T08:03:49.295 INFO:teuthology.orchestra.run.smithi007.stdout: "rgw_acl_grants_max_num": "100", 2024-09-18T08:03:49.295 INFO:teuthology.orchestra.run.smithi007.stdout: "rgw_admin_entry": "admin", 2024-09-18T08:03:49.295 INFO:teuthology.orchestra.run.smithi007.stdout: "rgw_allow_notification_secrets_in_cleartext": "false", 2024-09-18T08:03:49.295 INFO:teuthology.orchestra.run.smithi007.stdout: "rgw_asio_assert_yielding": "false", 2024-09-18T08:03:49.295 INFO:teuthology.orchestra.run.smithi007.stdout: "rgw_backend_store": "rados", 2024-09-18T08:03:49.295 INFO:teuthology.orchestra.run.smithi007.stdout: "rgw_barbican_url": "", 2024-09-18T08:03:49.295 INFO:teuthology.orchestra.run.smithi007.stdout: "rgw_beast_enable_async": "true", 2024-09-18T08:03:49.295 INFO:teuthology.orchestra.run.smithi007.stdout: "rgw_bucket_counters_cache": "false", 2024-09-18T08:03:49.295 INFO:teuthology.orchestra.run.smithi007.stdout: "rgw_bucket_counters_cache_size": "10000", 2024-09-18T08:03:49.295 INFO:teuthology.orchestra.run.smithi007.stdout: "rgw_bucket_default_quota_max_objects": "-1", 2024-09-18T08:03:49.296 INFO:teuthology.orchestra.run.smithi007.stdout: "rgw_bucket_default_quota_max_size": "-1", 2024-09-18T08:03:49.296 INFO:teuthology.orchestra.run.smithi007.stdout: "rgw_bucket_index_max_aio": "128", 2024-09-18T08:03:49.296 INFO:teuthology.orchestra.run.smithi007.stdout: "rgw_bucket_index_transaction_instrumentation": "false", 2024-09-18T08:03:49.296 INFO:teuthology.orchestra.run.smithi007.stdout: "rgw_bucket_quota_cache_size": "10000", 2024-09-18T08:03:49.296 INFO:teuthology.orchestra.run.smithi007.stdout: "rgw_bucket_quota_ttl": "600", 2024-09-18T08:03:49.296 INFO:teuthology.orchestra.run.smithi007.stdout: "rgw_bucket_sync_spawn_window": "20", 2024-09-18T08:03:49.296 INFO:teuthology.orchestra.run.smithi007.stdout: "rgw_cache_enabled": "true", 2024-09-18T08:03:49.296 INFO:teuthology.orchestra.run.smithi007.stdout: "rgw_cache_expiry_interval": "900", 2024-09-18T08:03:49.296 INFO:teuthology.orchestra.run.smithi007.stdout: "rgw_cache_lru_size": "25000", 2024-09-18T08:03:49.296 INFO:teuthology.orchestra.run.smithi007.stdout: "rgw_config_store": "rados", 2024-09-18T08:03:49.296 INFO:teuthology.orchestra.run.smithi007.stdout: "rgw_content_length_compat": "false", 2024-09-18T08:03:49.296 INFO:teuthology.orchestra.run.smithi007.stdout: "rgw_copy_obj_progress": "true", 2024-09-18T08:03:49.296 INFO:teuthology.orchestra.run.smithi007.stdout: "rgw_copy_obj_progress_every_bytes": "1048576", 2024-09-18T08:03:49.297 INFO:teuthology.orchestra.run.smithi007.stdout: "rgw_cors_rules_max_num": "100", 2024-09-18T08:03:49.297 INFO:teuthology.orchestra.run.smithi007.stdout: "rgw_cross_domain_policy": "", 2024-09-18T08:03:49.297 INFO:teuthology.orchestra.run.smithi007.stdout: "rgw_crypt_default_encryption_key": "", 2024-09-18T08:03:49.297 INFO:teuthology.orchestra.run.smithi007.stdout: "rgw_crypt_kmip_addr": "", 2024-09-18T08:03:49.297 INFO:teuthology.orchestra.run.smithi007.stdout: "rgw_crypt_kmip_ca_path": "", 2024-09-18T08:03:49.297 INFO:teuthology.orchestra.run.smithi007.stdout: "rgw_crypt_kmip_client_cert": "", 2024-09-18T08:03:49.297 INFO:teuthology.orchestra.run.smithi007.stdout: "rgw_crypt_kmip_client_key": "", 2024-09-18T08:03:49.297 INFO:teuthology.orchestra.run.smithi007.stdout: "rgw_crypt_kmip_kms_key_template": "", 2024-09-18T08:03:49.297 INFO:teuthology.orchestra.run.smithi007.stdout: "rgw_crypt_kmip_password": "", 2024-09-18T08:03:49.297 INFO:teuthology.orchestra.run.smithi007.stdout: "rgw_crypt_kmip_s3_key_template": "$keyid", 2024-09-18T08:03:49.297 INFO:teuthology.orchestra.run.smithi007.stdout: "rgw_crypt_kmip_username": "", 2024-09-18T08:03:49.297 INFO:teuthology.orchestra.run.smithi007.stdout: "rgw_crypt_require_ssl": "true", 2024-09-18T08:03:49.298 INFO:teuthology.orchestra.run.smithi007.stdout: "rgw_crypt_s3_kms_backend": "barbican", 2024-09-18T08:03:49.298 INFO:teuthology.orchestra.run.smithi007.stdout: "rgw_crypt_s3_kms_encryption_keys": "", 2024-09-18T08:03:49.298 INFO:teuthology.orchestra.run.smithi007.stdout: "rgw_crypt_sse_s3_backend": "vault", 2024-09-18T08:03:49.298 INFO:teuthology.orchestra.run.smithi007.stdout: "rgw_crypt_sse_s3_key_template": "%bucket_id", 2024-09-18T08:03:49.298 INFO:teuthology.orchestra.run.smithi007.stdout: "rgw_crypt_sse_s3_vault_addr": "", 2024-09-18T08:03:49.298 INFO:teuthology.orchestra.run.smithi007.stdout: "rgw_crypt_sse_s3_vault_auth": "token", 2024-09-18T08:03:49.298 INFO:teuthology.orchestra.run.smithi007.stdout: "rgw_crypt_sse_s3_vault_namespace": "", 2024-09-18T08:03:49.298 INFO:teuthology.orchestra.run.smithi007.stdout: "rgw_crypt_sse_s3_vault_prefix": "", 2024-09-18T08:03:49.298 INFO:teuthology.orchestra.run.smithi007.stdout: "rgw_crypt_sse_s3_vault_secret_engine": "transit", 2024-09-18T08:03:49.298 INFO:teuthology.orchestra.run.smithi007.stdout: "rgw_crypt_sse_s3_vault_ssl_cacert": "", 2024-09-18T08:03:49.298 INFO:teuthology.orchestra.run.smithi007.stdout: "rgw_crypt_sse_s3_vault_ssl_clientcert": "", 2024-09-18T08:03:49.298 INFO:teuthology.orchestra.run.smithi007.stdout: "rgw_crypt_sse_s3_vault_ssl_clientkey": "", 2024-09-18T08:03:49.298 INFO:teuthology.orchestra.run.smithi007.stdout: "rgw_crypt_sse_s3_vault_token_file": "", 2024-09-18T08:03:49.299 INFO:teuthology.orchestra.run.smithi007.stdout: "rgw_crypt_sse_s3_vault_verify_ssl": "true", 2024-09-18T08:03:49.299 INFO:teuthology.orchestra.run.smithi007.stdout: "rgw_crypt_suppress_logs": "true", 2024-09-18T08:03:49.299 INFO:teuthology.orchestra.run.smithi007.stdout: "rgw_crypt_vault_addr": "", 2024-09-18T08:03:49.299 INFO:teuthology.orchestra.run.smithi007.stdout: "rgw_crypt_vault_auth": "token", 2024-09-18T08:03:49.299 INFO:teuthology.orchestra.run.smithi007.stdout: "rgw_crypt_vault_namespace": "", 2024-09-18T08:03:49.299 INFO:teuthology.orchestra.run.smithi007.stdout: "rgw_crypt_vault_prefix": "", 2024-09-18T08:03:49.299 INFO:teuthology.orchestra.run.smithi007.stdout: "rgw_crypt_vault_secret_engine": "transit", 2024-09-18T08:03:49.299 INFO:teuthology.orchestra.run.smithi007.stdout: "rgw_crypt_vault_ssl_cacert": "", 2024-09-18T08:03:49.299 INFO:teuthology.orchestra.run.smithi007.stdout: "rgw_crypt_vault_ssl_clientcert": "", 2024-09-18T08:03:49.299 INFO:teuthology.orchestra.run.smithi007.stdout: "rgw_crypt_vault_ssl_clientkey": "", 2024-09-18T08:03:49.300 INFO:teuthology.orchestra.run.smithi007.stdout: "rgw_crypt_vault_token_file": "", 2024-09-18T08:03:49.300 INFO:teuthology.orchestra.run.smithi007.stdout: "rgw_crypt_vault_verify_ssl": "true", 2024-09-18T08:03:49.300 INFO:teuthology.orchestra.run.smithi007.stdout: "rgw_curl_buffersize": "524288", 2024-09-18T08:03:49.300 INFO:teuthology.orchestra.run.smithi007.stdout: "rgw_curl_low_speed_limit": "1024", 2024-09-18T08:03:49.300 INFO:teuthology.orchestra.run.smithi007.stdout: "rgw_curl_low_speed_time": "300", 2024-09-18T08:03:49.300 INFO:teuthology.orchestra.run.smithi007.stdout: "rgw_curl_tcp_keepalive": "0", 2024-09-18T08:03:49.300 INFO:teuthology.orchestra.run.smithi007.stdout: "rgw_curl_wait_timeout_ms": "1000", 2024-09-18T08:03:49.300 INFO:teuthology.orchestra.run.smithi007.stdout: "rgw_d3n_l1_datacache_persistent_path": "/tmp/rgw_datacache/", 2024-09-18T08:03:49.300 INFO:teuthology.orchestra.run.smithi007.stdout: "rgw_d3n_l1_datacache_size": "1073741824", 2024-09-18T08:03:49.300 INFO:teuthology.orchestra.run.smithi007.stdout: "rgw_d3n_l1_evict_cache_on_start": "true", 2024-09-18T08:03:49.300 INFO:teuthology.orchestra.run.smithi007.stdout: "rgw_d3n_l1_eviction_policy": "lru", 2024-09-18T08:03:49.300 INFO:teuthology.orchestra.run.smithi007.stdout: "rgw_d3n_l1_fadvise": "4", 2024-09-18T08:03:49.301 INFO:teuthology.orchestra.run.smithi007.stdout: "rgw_d3n_l1_local_datacache_enabled": "false", 2024-09-18T08:03:49.301 INFO:teuthology.orchestra.run.smithi007.stdout: "rgw_d3n_libaio_aio_num": "64", 2024-09-18T08:03:49.301 INFO:teuthology.orchestra.run.smithi007.stdout: "rgw_d3n_libaio_aio_threads": "20", 2024-09-18T08:03:49.301 INFO:teuthology.orchestra.run.smithi007.stdout: "rgw_d4n_address": "127.0.0.1:6379", 2024-09-18T08:03:49.301 INFO:teuthology.orchestra.run.smithi007.stdout: "rgw_d4n_l1_datacache_address": "127.0.0.1:6379", 2024-09-18T08:03:49.301 INFO:teuthology.orchestra.run.smithi007.stdout: "rgw_d4n_l1_datacache_persistent_path": "/tmp/rgw_d4n_datacache/", 2024-09-18T08:03:49.301 INFO:teuthology.orchestra.run.smithi007.stdout: "rgw_d4n_l1_datacache_size": "1073741824", 2024-09-18T08:03:49.301 INFO:teuthology.orchestra.run.smithi007.stdout: "rgw_d4n_l1_evict_cache_on_start": "true", 2024-09-18T08:03:49.301 INFO:teuthology.orchestra.run.smithi007.stdout: "rgw_d4n_l1_fadvise": "4", 2024-09-18T08:03:49.301 INFO:teuthology.orchestra.run.smithi007.stdout: "rgw_d4n_libaio_aio_num": "64", 2024-09-18T08:03:49.301 INFO:teuthology.orchestra.run.smithi007.stdout: "rgw_d4n_libaio_aio_threads": "20", 2024-09-18T08:03:49.301 INFO:teuthology.orchestra.run.smithi007.stdout: "rgw_data": "/var/lib/ceph/radosgw/ceph-a", 2024-09-18T08:03:49.301 INFO:teuthology.orchestra.run.smithi007.stdout: "rgw_data_log_changes_size": "1000", 2024-09-18T08:03:49.302 INFO:teuthology.orchestra.run.smithi007.stdout: "rgw_data_log_num_shards": "128", 2024-09-18T08:03:49.302 INFO:teuthology.orchestra.run.smithi007.stdout: "rgw_data_log_obj_prefix": "data_log", 2024-09-18T08:03:49.302 INFO:teuthology.orchestra.run.smithi007.stdout: "rgw_data_log_window": "30", 2024-09-18T08:03:49.302 INFO:teuthology.orchestra.run.smithi007.stdout: "rgw_data_notify_interval_msec": "0", 2024-09-18T08:03:49.302 INFO:teuthology.orchestra.run.smithi007.stdout: "rgw_data_sync_poll_interval": "20", 2024-09-18T08:03:49.302 INFO:teuthology.orchestra.run.smithi007.stdout: "rgw_data_sync_spawn_window": "20", 2024-09-18T08:03:49.302 INFO:teuthology.orchestra.run.smithi007.stdout: "rgw_debug_inject_latency_bi_unlink": "0", 2024-09-18T08:03:49.302 INFO:teuthology.orchestra.run.smithi007.stdout: "rgw_debug_inject_olh_cancel_modification_err": "false", 2024-09-18T08:03:49.302 INFO:teuthology.orchestra.run.smithi007.stdout: "rgw_debug_inject_set_olh_err": "0", 2024-09-18T08:03:49.302 INFO:teuthology.orchestra.run.smithi007.stdout: "rgw_default_data_log_backing": "fifo", 2024-09-18T08:03:49.302 INFO:teuthology.orchestra.run.smithi007.stdout: "rgw_default_realm_info_oid": "default.realm", 2024-09-18T08:03:49.303 INFO:teuthology.orchestra.run.smithi007.stdout: "rgw_default_region_info_oid": "default.region", 2024-09-18T08:03:49.303 INFO:teuthology.orchestra.run.smithi007.stdout: "rgw_default_zone_info_oid": "default.zone", 2024-09-18T08:03:49.303 INFO:teuthology.orchestra.run.smithi007.stdout: "rgw_default_zonegroup_info_oid": "default.zonegroup", 2024-09-18T08:03:49.303 INFO:teuthology.orchestra.run.smithi007.stdout: "rgw_defer_to_bucket_acls": "", 2024-09-18T08:03:49.303 INFO:teuthology.orchestra.run.smithi007.stdout: "rgw_delete_multi_obj_max_num": "1000", 2024-09-18T08:03:49.303 INFO:teuthology.orchestra.run.smithi007.stdout: "rgw_disable_s3select": "false", 2024-09-18T08:03:49.303 INFO:teuthology.orchestra.run.smithi007.stdout: "rgw_dmclock_admin_lim": "0.000000", 2024-09-18T08:03:49.303 INFO:teuthology.orchestra.run.smithi007.stdout: "rgw_dmclock_admin_res": "100.000000", 2024-09-18T08:03:49.303 INFO:teuthology.orchestra.run.smithi007.stdout: "rgw_dmclock_admin_wgt": "100.000000", 2024-09-18T08:03:49.303 INFO:teuthology.orchestra.run.smithi007.stdout: "rgw_dmclock_auth_lim": "0.000000", 2024-09-18T08:03:49.303 INFO:teuthology.orchestra.run.smithi007.stdout: "rgw_dmclock_auth_res": "200.000000", 2024-09-18T08:03:49.303 INFO:teuthology.orchestra.run.smithi007.stdout: "rgw_dmclock_auth_wgt": "100.000000", 2024-09-18T08:03:49.303 INFO:teuthology.orchestra.run.smithi007.stdout: "rgw_dmclock_data_lim": "0.000000", 2024-09-18T08:03:49.304 INFO:teuthology.orchestra.run.smithi007.stdout: "rgw_dmclock_data_res": "500.000000", 2024-09-18T08:03:49.304 INFO:teuthology.orchestra.run.smithi007.stdout: "rgw_dmclock_data_wgt": "500.000000", 2024-09-18T08:03:49.304 INFO:teuthology.orchestra.run.smithi007.stdout: "rgw_dmclock_metadata_lim": "0.000000", 2024-09-18T08:03:49.304 INFO:teuthology.orchestra.run.smithi007.stdout: "rgw_dmclock_metadata_res": "500.000000", 2024-09-18T08:03:49.304 INFO:teuthology.orchestra.run.smithi007.stdout: "rgw_dmclock_metadata_wgt": "500.000000", 2024-09-18T08:03:49.304 INFO:teuthology.orchestra.run.smithi007.stdout: "rgw_dns_name": "", 2024-09-18T08:03:49.304 INFO:teuthology.orchestra.run.smithi007.stdout: "rgw_dns_s3website_name": "", 2024-09-18T08:03:49.304 INFO:teuthology.orchestra.run.smithi007.stdout: "rgw_dynamic_resharding": "true", 2024-09-18T08:03:49.304 INFO:teuthology.orchestra.run.smithi007.stdout: "rgw_dynamic_resharding_may_reduce": "true", 2024-09-18T08:03:49.304 INFO:teuthology.orchestra.run.smithi007.stdout: "rgw_dynamic_resharding_reduction_wait": "120", 2024-09-18T08:03:49.304 INFO:teuthology.orchestra.run.smithi007.stdout: "rgw_enable_apis": "s3, s3website, swift, swift_auth, admin, sts, iam, notifications", 2024-09-18T08:03:49.304 INFO:teuthology.orchestra.run.smithi007.stdout: "rgw_enable_gc_threads": "true", 2024-09-18T08:03:49.305 INFO:teuthology.orchestra.run.smithi007.stdout: "rgw_enable_lc_threads": "true", 2024-09-18T08:03:49.305 INFO:teuthology.orchestra.run.smithi007.stdout: "rgw_enable_mdsearch": "true", 2024-09-18T08:03:49.305 INFO:teuthology.orchestra.run.smithi007.stdout: "rgw_enable_ops_log": "false", 2024-09-18T08:03:49.305 INFO:teuthology.orchestra.run.smithi007.stdout: "rgw_enable_quota_threads": "true", 2024-09-18T08:03:49.305 INFO:teuthology.orchestra.run.smithi007.stdout: "rgw_enable_static_website": "false", 2024-09-18T08:03:49.305 INFO:teuthology.orchestra.run.smithi007.stdout: "rgw_enable_usage_log": "false", 2024-09-18T08:03:49.305 INFO:teuthology.orchestra.run.smithi007.stdout: "rgw_enforce_swift_acls": "true", 2024-09-18T08:03:49.305 INFO:teuthology.orchestra.run.smithi007.stdout: "rgw_exit_timeout_secs": "120", 2024-09-18T08:03:49.305 INFO:teuthology.orchestra.run.smithi007.stdout: "rgw_expose_bucket": "false", 2024-09-18T08:03:49.305 INFO:teuthology.orchestra.run.smithi007.stdout: "rgw_extended_http_attrs": "", 2024-09-18T08:03:49.305 INFO:teuthology.orchestra.run.smithi007.stdout: "rgw_filter": "none", 2024-09-18T08:03:49.305 INFO:teuthology.orchestra.run.smithi007.stdout: "rgw_frontend_defaults": "beast ssl_certificate=config://rgw/cert/$realm/$zone.crt ssl_private_key=config://rgw/cert/$realm/$zone.key", 2024-09-18T08:03:49.306 INFO:teuthology.orchestra.run.smithi007.stdout: "rgw_frontends": "beast port=7480", 2024-09-18T08:03:49.306 INFO:teuthology.orchestra.run.smithi007.stdout: "rgw_gc_max_concurrent_io": "10", 2024-09-18T08:03:49.306 INFO:teuthology.orchestra.run.smithi007.stdout: "rgw_gc_max_deferred": "50", 2024-09-18T08:03:49.306 INFO:teuthology.orchestra.run.smithi007.stdout: "rgw_gc_max_deferred_entries_size": "3072", 2024-09-18T08:03:49.306 INFO:teuthology.orchestra.run.smithi007.stdout: "rgw_gc_max_objs": "32", 2024-09-18T08:03:49.306 INFO:teuthology.orchestra.run.smithi007.stdout: "rgw_gc_max_queue_size": "134213632", 2024-09-18T08:03:49.306 INFO:teuthology.orchestra.run.smithi007.stdout: "rgw_gc_max_trim_chunk": "16", 2024-09-18T08:03:49.306 INFO:teuthology.orchestra.run.smithi007.stdout: "rgw_gc_obj_min_wait": "7200", 2024-09-18T08:03:49.306 INFO:teuthology.orchestra.run.smithi007.stdout: "rgw_gc_processor_max_time": "3600", 2024-09-18T08:03:49.306 INFO:teuthology.orchestra.run.smithi007.stdout: "rgw_gc_processor_period": "3600", 2024-09-18T08:03:49.306 INFO:teuthology.orchestra.run.smithi007.stdout: "rgw_get_obj_max_req_size": "4194304", 2024-09-18T08:03:49.306 INFO:teuthology.orchestra.run.smithi007.stdout: "rgw_get_obj_window_size": "16777216", 2024-09-18T08:03:49.306 INFO:teuthology.orchestra.run.smithi007.stdout: "rgw_healthcheck_disabling_path": "", 2024-09-18T08:03:49.307 INFO:teuthology.orchestra.run.smithi007.stdout: "rgw_ignore_get_invalid_range": "false", 2024-09-18T08:03:49.307 INFO:teuthology.orchestra.run.smithi007.stdout: "rgw_init_timeout": "300", 2024-09-18T08:03:49.307 INFO:teuthology.orchestra.run.smithi007.stdout: "rgw_inject_notify_timeout_probability": "0.000000", 2024-09-18T08:03:49.307 INFO:teuthology.orchestra.run.smithi007.stdout: "rgw_json_config": "/var/lib/ceph/radosgw/config.json", 2024-09-18T08:03:49.307 INFO:teuthology.orchestra.run.smithi007.stdout: "rgw_kafka_connection_idle": "300", 2024-09-18T08:03:49.307 INFO:teuthology.orchestra.run.smithi007.stdout: "rgw_kafka_message_timeout": "5000", 2024-09-18T08:03:49.307 INFO:teuthology.orchestra.run.smithi007.stdout: "rgw_kafka_sleep_timeout": "10", 2024-09-18T08:03:49.307 INFO:teuthology.orchestra.run.smithi007.stdout: "rgw_keystone_accepted_admin_roles": "", 2024-09-18T08:03:49.307 INFO:teuthology.orchestra.run.smithi007.stdout: "rgw_keystone_accepted_reader_roles": "", 2024-09-18T08:03:49.307 INFO:teuthology.orchestra.run.smithi007.stdout: "rgw_keystone_accepted_roles": "Member, admin", 2024-09-18T08:03:49.307 INFO:teuthology.orchestra.run.smithi007.stdout: "rgw_keystone_admin_domain": "", 2024-09-18T08:03:49.307 INFO:teuthology.orchestra.run.smithi007.stdout: "rgw_keystone_admin_password": "", 2024-09-18T08:03:49.307 INFO:teuthology.orchestra.run.smithi007.stdout: "rgw_keystone_admin_password_path": "", 2024-09-18T08:03:49.308 INFO:teuthology.orchestra.run.smithi007.stdout: "rgw_keystone_admin_project": "", 2024-09-18T08:03:49.308 INFO:teuthology.orchestra.run.smithi007.stdout: "rgw_keystone_admin_tenant": "", 2024-09-18T08:03:49.308 INFO:teuthology.orchestra.run.smithi007.stdout: "rgw_keystone_admin_token": "", 2024-09-18T08:03:49.308 INFO:teuthology.orchestra.run.smithi007.stdout: "rgw_keystone_admin_token_path": "", 2024-09-18T08:03:49.308 INFO:teuthology.orchestra.run.smithi007.stdout: "rgw_keystone_admin_user": "", 2024-09-18T08:03:49.308 INFO:teuthology.orchestra.run.smithi007.stdout: "rgw_keystone_barbican_domain": "", 2024-09-18T08:03:49.308 INFO:teuthology.orchestra.run.smithi007.stdout: "rgw_keystone_barbican_password": "", 2024-09-18T08:03:49.308 INFO:teuthology.orchestra.run.smithi007.stdout: "rgw_keystone_barbican_project": "", 2024-09-18T08:03:49.308 INFO:teuthology.orchestra.run.smithi007.stdout: "rgw_keystone_barbican_tenant": "", 2024-09-18T08:03:49.308 INFO:teuthology.orchestra.run.smithi007.stdout: "rgw_keystone_barbican_user": "", 2024-09-18T08:03:49.308 INFO:teuthology.orchestra.run.smithi007.stdout: "rgw_keystone_expired_token_cache_expiration": "3600", 2024-09-18T08:03:49.308 INFO:teuthology.orchestra.run.smithi007.stdout: "rgw_keystone_implicit_tenants": "false", 2024-09-18T08:03:49.308 INFO:teuthology.orchestra.run.smithi007.stdout: "rgw_keystone_service_token_accepted_roles": "admin", 2024-09-18T08:03:49.309 INFO:teuthology.orchestra.run.smithi007.stdout: "rgw_keystone_service_token_enabled": "false", 2024-09-18T08:03:49.309 INFO:teuthology.orchestra.run.smithi007.stdout: "rgw_keystone_token_cache_size": "10000", 2024-09-18T08:03:49.309 INFO:teuthology.orchestra.run.smithi007.stdout: "rgw_keystone_url": "", 2024-09-18T08:03:49.309 INFO:teuthology.orchestra.run.smithi007.stdout: "rgw_keystone_verify_ssl": "true", 2024-09-18T08:03:49.309 INFO:teuthology.orchestra.run.smithi007.stdout: "rgw_lc_debug_interval": "-1", 2024-09-18T08:03:49.309 INFO:teuthology.orchestra.run.smithi007.stdout: "rgw_lc_lock_max_time": "90", 2024-09-18T08:03:49.309 INFO:teuthology.orchestra.run.smithi007.stdout: "rgw_lc_max_objs": "32", 2024-09-18T08:03:49.309 INFO:teuthology.orchestra.run.smithi007.stdout: "rgw_lc_max_rules": "1000", 2024-09-18T08:03:49.309 INFO:teuthology.orchestra.run.smithi007.stdout: "rgw_lc_max_worker": "3", 2024-09-18T08:03:49.309 INFO:teuthology.orchestra.run.smithi007.stdout: "rgw_lc_max_wp_worker": "3", 2024-09-18T08:03:49.310 INFO:teuthology.orchestra.run.smithi007.stdout: "rgw_lc_thread_delay": "0", 2024-09-18T08:03:49.310 INFO:teuthology.orchestra.run.smithi007.stdout: "rgw_ldap_binddn": "uid=admin,cn=users,dc=example,dc=com", 2024-09-18T08:03:49.310 INFO:teuthology.orchestra.run.smithi007.stdout: "rgw_ldap_dnattr": "uid", 2024-09-18T08:03:49.310 INFO:teuthology.orchestra.run.smithi007.stdout: "rgw_ldap_searchdn": "cn=users,cn=accounts,dc=example,dc=com", 2024-09-18T08:03:49.310 INFO:teuthology.orchestra.run.smithi007.stdout: "rgw_ldap_searchfilter": "", 2024-09-18T08:03:49.310 INFO:teuthology.orchestra.run.smithi007.stdout: "rgw_ldap_secret": "/etc/openldap/secret", 2024-09-18T08:03:49.310 INFO:teuthology.orchestra.run.smithi007.stdout: "rgw_ldap_uri": "", 2024-09-18T08:03:49.310 INFO:teuthology.orchestra.run.smithi007.stdout: "rgw_lfuda_sync_frequency": "60", 2024-09-18T08:03:49.310 INFO:teuthology.orchestra.run.smithi007.stdout: "rgw_lifecycle_work_time": "00:00-06:00", 2024-09-18T08:03:49.310 INFO:teuthology.orchestra.run.smithi007.stdout: "rgw_list_bucket_min_readahead": "1000", 2024-09-18T08:03:49.310 INFO:teuthology.orchestra.run.smithi007.stdout: "rgw_list_buckets_max_chunk": "1000", 2024-09-18T08:03:49.311 INFO:teuthology.orchestra.run.smithi007.stdout: "rgw_log_http_headers": "", 2024-09-18T08:03:49.311 INFO:teuthology.orchestra.run.smithi007.stdout: "rgw_log_nonexistent_bucket": "false", 2024-09-18T08:03:49.311 INFO:teuthology.orchestra.run.smithi007.stdout: "rgw_log_object_name": "%Y-%m-%d-%H-%i-%n", 2024-09-18T08:03:49.311 INFO:teuthology.orchestra.run.smithi007.stdout: "rgw_log_object_name_utc": "false", 2024-09-18T08:03:49.311 INFO:teuthology.orchestra.run.smithi007.stdout: "rgw_lua_max_memory_per_state": "128000", 2024-09-18T08:03:49.311 INFO:teuthology.orchestra.run.smithi007.stdout: "rgw_luarocks_location": "/tmp/rgw_luarocks/mon.a", 2024-09-18T08:03:49.311 INFO:teuthology.orchestra.run.smithi007.stdout: "rgw_max_attr_name_len": "0", 2024-09-18T08:03:49.311 INFO:teuthology.orchestra.run.smithi007.stdout: "rgw_max_attr_size": "0", 2024-09-18T08:03:49.311 INFO:teuthology.orchestra.run.smithi007.stdout: "rgw_max_attrs_num_in_req": "0", 2024-09-18T08:03:49.311 INFO:teuthology.orchestra.run.smithi007.stdout: "rgw_max_chunk_size": "4194304", 2024-09-18T08:03:49.311 INFO:teuthology.orchestra.run.smithi007.stdout: "rgw_max_concurrent_requests": "1024", 2024-09-18T08:03:49.311 INFO:teuthology.orchestra.run.smithi007.stdout: "rgw_max_copy_obj_concurrent_io": "10", 2024-09-18T08:03:49.312 INFO:teuthology.orchestra.run.smithi007.stdout: "rgw_max_dynamic_shards": "1999", 2024-09-18T08:03:49.312 INFO:teuthology.orchestra.run.smithi007.stdout: "rgw_max_listing_results": "5000", 2024-09-18T08:03:49.312 INFO:teuthology.orchestra.run.smithi007.stdout: "rgw_max_notify_retries": "10", 2024-09-18T08:03:49.312 INFO:teuthology.orchestra.run.smithi007.stdout: "rgw_max_objs_per_shard": "100000", 2024-09-18T08:03:49.312 INFO:teuthology.orchestra.run.smithi007.stdout: "rgw_max_put_param_size": "1048576", 2024-09-18T08:03:49.312 INFO:teuthology.orchestra.run.smithi007.stdout: "rgw_max_put_size": "5368709120", 2024-09-18T08:03:49.312 INFO:teuthology.orchestra.run.smithi007.stdout: "rgw_max_slo_entries": "1000", 2024-09-18T08:03:49.312 INFO:teuthology.orchestra.run.smithi007.stdout: "rgw_md_log_max_shards": "64", 2024-09-18T08:03:49.312 INFO:teuthology.orchestra.run.smithi007.stdout: "rgw_md_notify_interval_msec": "200", 2024-09-18T08:03:49.312 INFO:teuthology.orchestra.run.smithi007.stdout: "rgw_meta_sync_poll_interval": "20", 2024-09-18T08:03:49.312 INFO:teuthology.orchestra.run.smithi007.stdout: "rgw_meta_sync_spawn_window": "20", 2024-09-18T08:03:49.312 INFO:teuthology.orchestra.run.smithi007.stdout: "rgw_mime_types_file": "/etc/mime.types", 2024-09-18T08:03:49.312 INFO:teuthology.orchestra.run.smithi007.stdout: "rgw_mp_lock_max_time": "600", 2024-09-18T08:03:49.313 INFO:teuthology.orchestra.run.smithi007.stdout: "rgw_multi_obj_del_max_aio": "16", 2024-09-18T08:03:49.313 INFO:teuthology.orchestra.run.smithi007.stdout: "rgw_multipart_min_part_size": "5242880", 2024-09-18T08:03:49.313 INFO:teuthology.orchestra.run.smithi007.stdout: "rgw_multipart_part_upload_limit": "10000", 2024-09-18T08:03:49.313 INFO:teuthology.orchestra.run.smithi007.stdout: "rgw_nfs_fhcache_partitions": "3", 2024-09-18T08:03:49.313 INFO:teuthology.orchestra.run.smithi007.stdout: "rgw_nfs_fhcache_size": "2017", 2024-09-18T08:03:49.313 INFO:teuthology.orchestra.run.smithi007.stdout: "rgw_nfs_frontends": "rgw-nfs", 2024-09-18T08:03:49.313 INFO:teuthology.orchestra.run.smithi007.stdout: "rgw_nfs_lru_lane_hiwat": "911", 2024-09-18T08:03:49.313 INFO:teuthology.orchestra.run.smithi007.stdout: "rgw_nfs_lru_lanes": "5", 2024-09-18T08:03:49.313 INFO:teuthology.orchestra.run.smithi007.stdout: "rgw_nfs_max_gc": "300", 2024-09-18T08:03:49.313 INFO:teuthology.orchestra.run.smithi007.stdout: "rgw_nfs_namespace_expire_secs": "300", 2024-09-18T08:03:49.313 INFO:teuthology.orchestra.run.smithi007.stdout: "rgw_nfs_run_gc_threads": "false", 2024-09-18T08:03:49.313 INFO:teuthology.orchestra.run.smithi007.stdout: "rgw_nfs_run_lc_threads": "false", 2024-09-18T08:03:49.314 INFO:teuthology.orchestra.run.smithi007.stdout: "rgw_nfs_run_quota_threads": "false", 2024-09-18T08:03:49.314 INFO:teuthology.orchestra.run.smithi007.stdout: "rgw_nfs_run_sync_thread": "false", 2024-09-18T08:03:49.314 INFO:teuthology.orchestra.run.smithi007.stdout: "rgw_nfs_s3_fast_attrs": "false", 2024-09-18T08:03:49.314 INFO:teuthology.orchestra.run.smithi007.stdout: "rgw_nfs_write_completion_interval_s": "10", 2024-09-18T08:03:49.314 INFO:teuthology.orchestra.run.smithi007.stdout: "rgw_num_async_rados_threads": "32", 2024-09-18T08:03:49.314 INFO:teuthology.orchestra.run.smithi007.stdout: "rgw_num_control_oids": "8", 2024-09-18T08:03:49.314 INFO:teuthology.orchestra.run.smithi007.stdout: "rgw_numa_node": "-1", 2024-09-18T08:03:49.314 INFO:teuthology.orchestra.run.smithi007.stdout: "rgw_obj_stripe_size": "4194304", 2024-09-18T08:03:49.314 INFO:teuthology.orchestra.run.smithi007.stdout: "rgw_obj_tombstone_cache_size": "1000", 2024-09-18T08:03:49.314 INFO:teuthology.orchestra.run.smithi007.stdout: "rgw_objexp_chunk_size": "100", 2024-09-18T08:03:49.314 INFO:teuthology.orchestra.run.smithi007.stdout: "rgw_objexp_gc_interval": "600", 2024-09-18T08:03:49.314 INFO:teuthology.orchestra.run.smithi007.stdout: "rgw_objexp_hints_num_shards": "127", 2024-09-18T08:03:49.314 INFO:teuthology.orchestra.run.smithi007.stdout: "rgw_olh_pending_timeout_sec": "3600", 2024-09-18T08:03:49.315 INFO:teuthology.orchestra.run.smithi007.stdout: "rgw_op_thread_suicide_timeout": "0", 2024-09-18T08:03:49.315 INFO:teuthology.orchestra.run.smithi007.stdout: "rgw_op_thread_timeout": "600", 2024-09-18T08:03:49.315 INFO:teuthology.orchestra.run.smithi007.stdout: "rgw_op_tracing": "false", 2024-09-18T08:03:49.315 INFO:teuthology.orchestra.run.smithi007.stdout: "rgw_opa_token": "", 2024-09-18T08:03:49.315 INFO:teuthology.orchestra.run.smithi007.stdout: "rgw_opa_url": "", 2024-09-18T08:03:49.315 INFO:teuthology.orchestra.run.smithi007.stdout: "rgw_opa_verify_ssl": "true", 2024-09-18T08:03:49.315 INFO:teuthology.orchestra.run.smithi007.stdout: "rgw_ops_log_data_backlog": "5242880", 2024-09-18T08:03:49.315 INFO:teuthology.orchestra.run.smithi007.stdout: "rgw_ops_log_file_path": "/var/log/ceph/ops-log-ceph-mon.a.log", 2024-09-18T08:03:49.315 INFO:teuthology.orchestra.run.smithi007.stdout: "rgw_ops_log_rados": "false", 2024-09-18T08:03:49.315 INFO:teuthology.orchestra.run.smithi007.stdout: "rgw_ops_log_socket_path": "", 2024-09-18T08:03:49.316 INFO:teuthology.orchestra.run.smithi007.stdout: "rgw_override_bucket_index_max_shards": "0", 2024-09-18T08:03:49.316 INFO:teuthology.orchestra.run.smithi007.stdout: "rgw_pending_bucket_index_op_expiration": "120", 2024-09-18T08:03:49.316 INFO:teuthology.orchestra.run.smithi007.stdout: "rgw_period_latest_epoch_info_oid": ".latest_epoch", 2024-09-18T08:03:49.316 INFO:teuthology.orchestra.run.smithi007.stdout: "rgw_period_push_interval": "2.000000", 2024-09-18T08:03:49.316 INFO:teuthology.orchestra.run.smithi007.stdout: "rgw_period_push_interval_max": "30.000000", 2024-09-18T08:03:49.316 INFO:teuthology.orchestra.run.smithi007.stdout: "rgw_period_root_pool": ".rgw.root", 2024-09-18T08:03:49.317 INFO:teuthology.orchestra.run.smithi007.stdout: "rgw_policy_reject_invalid_principals": "true", 2024-09-18T08:03:49.317 INFO:teuthology.orchestra.run.smithi007.stdout: "rgw_posix_base_path": "/tmp/rgw_posix_driver", 2024-09-18T08:03:49.317 INFO:teuthology.orchestra.run.smithi007.stdout: "rgw_posix_cache_lanes": "3", 2024-09-18T08:03:49.317 INFO:teuthology.orchestra.run.smithi007.stdout: "rgw_posix_cache_lmdb_count": "3", 2024-09-18T08:03:49.317 INFO:teuthology.orchestra.run.smithi007.stdout: "rgw_posix_cache_max_buckets": "100", 2024-09-18T08:03:49.317 INFO:teuthology.orchestra.run.smithi007.stdout: "rgw_posix_cache_partitions": "3", 2024-09-18T08:03:49.318 INFO:teuthology.orchestra.run.smithi007.stdout: "rgw_posix_database_root": "/var/lib/ceph/radosgw", 2024-09-18T08:03:49.318 INFO:teuthology.orchestra.run.smithi007.stdout: "rgw_print_continue": "true", 2024-09-18T08:03:49.318 INFO:teuthology.orchestra.run.smithi007.stdout: "rgw_print_prohibited_content_length": "false", 2024-09-18T08:03:49.318 INFO:teuthology.orchestra.run.smithi007.stdout: "rgw_put_obj_max_window_size": "67108864", 2024-09-18T08:03:49.318 INFO:teuthology.orchestra.run.smithi007.stdout: "rgw_put_obj_min_window_size": "16777216", 2024-09-18T08:03:49.318 INFO:teuthology.orchestra.run.smithi007.stdout: "rgw_rados_pool_autoscale_bias": "4.000000", 2024-09-18T08:03:49.319 INFO:teuthology.orchestra.run.smithi007.stdout: "rgw_rados_pool_recovery_priority": "5", 2024-09-18T08:03:49.319 INFO:teuthology.orchestra.run.smithi007.stdout: "rgw_rados_tracing": "false", 2024-09-18T08:03:49.319 INFO:teuthology.orchestra.run.smithi007.stdout: "rgw_realm": "", 2024-09-18T08:03:49.319 INFO:teuthology.orchestra.run.smithi007.stdout: "rgw_realm_id": "", 2024-09-18T08:03:49.319 INFO:teuthology.orchestra.run.smithi007.stdout: "rgw_realm_root_pool": ".rgw.root", 2024-09-18T08:03:49.320 INFO:teuthology.orchestra.run.smithi007.stdout: "rgw_region": "", 2024-09-18T08:03:49.320 INFO:teuthology.orchestra.run.smithi007.stdout: "rgw_region_root_pool": ".rgw.root", 2024-09-18T08:03:49.320 INFO:teuthology.orchestra.run.smithi007.stdout: "rgw_relaxed_region_enforcement": "false", 2024-09-18T08:03:49.320 INFO:teuthology.orchestra.run.smithi007.stdout: "rgw_relaxed_s3_bucket_names": "false", 2024-09-18T08:03:49.320 INFO:teuthology.orchestra.run.smithi007.stdout: "rgw_remote_addr_param": "REMOTE_ADDR", 2024-09-18T08:03:49.320 INFO:teuthology.orchestra.run.smithi007.stdout: "rgw_request_uri": "", 2024-09-18T08:03:49.321 INFO:teuthology.orchestra.run.smithi007.stdout: "rgw_reshard_batch_size": "64", 2024-09-18T08:03:49.321 INFO:teuthology.orchestra.run.smithi007.stdout: "rgw_reshard_bucket_lock_duration": "360", 2024-09-18T08:03:49.321 INFO:teuthology.orchestra.run.smithi007.stdout: "rgw_reshard_debug_interval": "-1", 2024-09-18T08:03:49.321 INFO:teuthology.orchestra.run.smithi007.stdout: "rgw_reshard_max_aio": "128", 2024-09-18T08:03:49.321 INFO:teuthology.orchestra.run.smithi007.stdout: "rgw_reshard_num_logs": "16", 2024-09-18T08:03:49.322 INFO:teuthology.orchestra.run.smithi007.stdout: "rgw_reshard_progress_judge_interval": "120", 2024-09-18T08:03:49.322 INFO:teuthology.orchestra.run.smithi007.stdout: "rgw_reshard_progress_judge_ratio": "0.500000", 2024-09-18T08:03:49.322 INFO:teuthology.orchestra.run.smithi007.stdout: "rgw_reshard_thread_interval": "600", 2024-09-18T08:03:49.322 INFO:teuthology.orchestra.run.smithi007.stdout: "rgw_reshardlog_threshold": "30000", 2024-09-18T08:03:49.322 INFO:teuthology.orchestra.run.smithi007.stdout: "rgw_resolve_cname": "false", 2024-09-18T08:03:49.322 INFO:teuthology.orchestra.run.smithi007.stdout: "rgw_rest_getusage_op_compat": "false", 2024-09-18T08:03:49.323 INFO:teuthology.orchestra.run.smithi007.stdout: "rgw_run_sync_thread": "true", 2024-09-18T08:03:49.323 INFO:teuthology.orchestra.run.smithi007.stdout: "rgw_s3_auth_disable_signature_url": "false", 2024-09-18T08:03:49.323 INFO:teuthology.orchestra.run.smithi007.stdout: "rgw_s3_auth_order": "sts, external, local", 2024-09-18T08:03:49.323 INFO:teuthology.orchestra.run.smithi007.stdout: "rgw_s3_auth_use_keystone": "false", 2024-09-18T08:03:49.323 INFO:teuthology.orchestra.run.smithi007.stdout: "rgw_s3_auth_use_ldap": "false", 2024-09-18T08:03:49.324 INFO:teuthology.orchestra.run.smithi007.stdout: "rgw_s3_auth_use_rados": "true", 2024-09-18T08:03:49.324 INFO:teuthology.orchestra.run.smithi007.stdout: "rgw_s3_auth_use_sts": "false", 2024-09-18T08:03:49.324 INFO:teuthology.orchestra.run.smithi007.stdout: "rgw_s3_client_max_sig_ver": "-1", 2024-09-18T08:03:49.324 INFO:teuthology.orchestra.run.smithi007.stdout: "rgw_s3_success_create_obj_status": "0", 2024-09-18T08:03:49.324 INFO:teuthology.orchestra.run.smithi007.stdout: "rgw_safe_max_objects_per_shard": "102400", 2024-09-18T08:03:49.324 INFO:teuthology.orchestra.run.smithi007.stdout: "rgw_scheduler_type": "throttler", 2024-09-18T08:03:49.325 INFO:teuthology.orchestra.run.smithi007.stdout: "rgw_script_uri": "", 2024-09-18T08:03:49.325 INFO:teuthology.orchestra.run.smithi007.stdout: "rgw_service_provider_name": "", 2024-09-18T08:03:49.325 INFO:teuthology.orchestra.run.smithi007.stdout: "rgw_shard_warning_threshold": "90.000000", 2024-09-18T08:03:49.325 INFO:teuthology.orchestra.run.smithi007.stdout: "rgw_sts_client_id": "", 2024-09-18T08:03:49.325 INFO:teuthology.orchestra.run.smithi007.stdout: "rgw_sts_client_secret": "", 2024-09-18T08:03:49.326 INFO:teuthology.orchestra.run.smithi007.stdout: "rgw_sts_entry": "sts", 2024-09-18T08:03:49.326 INFO:teuthology.orchestra.run.smithi007.stdout: "rgw_sts_key": "sts", 2024-09-18T08:03:49.326 INFO:teuthology.orchestra.run.smithi007.stdout: "rgw_sts_max_session_duration": "43200", 2024-09-18T08:03:49.326 INFO:teuthology.orchestra.run.smithi007.stdout: "rgw_sts_min_session_duration": "900", 2024-09-18T08:03:49.326 INFO:teuthology.orchestra.run.smithi007.stdout: "rgw_sts_token_introspection_url": "", 2024-09-18T08:03:49.327 INFO:teuthology.orchestra.run.smithi007.stdout: "rgw_swift_account_in_url": "false", 2024-09-18T08:03:49.327 INFO:teuthology.orchestra.run.smithi007.stdout: "rgw_swift_auth_entry": "auth", 2024-09-18T08:03:49.327 INFO:teuthology.orchestra.run.smithi007.stdout: "rgw_swift_auth_url": "", 2024-09-18T08:03:49.327 INFO:teuthology.orchestra.run.smithi007.stdout: "rgw_swift_custom_header": "", 2024-09-18T08:03:49.327 INFO:teuthology.orchestra.run.smithi007.stdout: "rgw_swift_enforce_content_length": "false", 2024-09-18T08:03:49.328 INFO:teuthology.orchestra.run.smithi007.stdout: "rgw_swift_need_stats": "true", 2024-09-18T08:03:49.328 INFO:teuthology.orchestra.run.smithi007.stdout: "rgw_swift_tenant_name": "", 2024-09-18T08:03:49.328 INFO:teuthology.orchestra.run.smithi007.stdout: "rgw_swift_token_expiration": "86400", 2024-09-18T08:03:49.328 INFO:teuthology.orchestra.run.smithi007.stdout: "rgw_swift_url": "", 2024-09-18T08:03:49.328 INFO:teuthology.orchestra.run.smithi007.stdout: "rgw_swift_url_prefix": "swift", 2024-09-18T08:03:49.329 INFO:teuthology.orchestra.run.smithi007.stdout: "rgw_swift_versioning_enabled": "false", 2024-09-18T08:03:49.329 INFO:teuthology.orchestra.run.smithi007.stdout: "rgw_sync_data_full_inject_err_probability": "0.000000", 2024-09-18T08:03:49.329 INFO:teuthology.orchestra.run.smithi007.stdout: "rgw_sync_data_inject_err_probability": "0.000000", 2024-09-18T08:03:49.329 INFO:teuthology.orchestra.run.smithi007.stdout: "rgw_sync_lease_period": "120", 2024-09-18T08:03:49.329 INFO:teuthology.orchestra.run.smithi007.stdout: "rgw_sync_log_trim_concurrent_buckets": "4", 2024-09-18T08:03:49.329 INFO:teuthology.orchestra.run.smithi007.stdout: "rgw_sync_log_trim_interval": "1200", 2024-09-18T08:03:49.330 INFO:teuthology.orchestra.run.smithi007.stdout: "rgw_sync_log_trim_max_buckets": "16", 2024-09-18T08:03:49.330 INFO:teuthology.orchestra.run.smithi007.stdout: "rgw_sync_log_trim_min_cold_buckets": "4", 2024-09-18T08:03:49.330 INFO:teuthology.orchestra.run.smithi007.stdout: "rgw_sync_meta_inject_err_probability": "0.000000", 2024-09-18T08:03:49.330 INFO:teuthology.orchestra.run.smithi007.stdout: "rgw_sync_obj_etag_verify": "false", 2024-09-18T08:03:49.330 INFO:teuthology.orchestra.run.smithi007.stdout: "rgw_sync_trace_history_size": "4096", 2024-09-18T08:03:49.330 INFO:teuthology.orchestra.run.smithi007.stdout: "rgw_sync_trace_per_node_log_size": "32", 2024-09-18T08:03:49.331 INFO:teuthology.orchestra.run.smithi007.stdout: "rgw_sync_trace_servicemap_update_interval": "10", 2024-09-18T08:03:49.331 INFO:teuthology.orchestra.run.smithi007.stdout: "rgw_thread_pool_size": "512", 2024-09-18T08:03:49.331 INFO:teuthology.orchestra.run.smithi007.stdout: "rgw_topic_persistency_max_retries": "0", 2024-09-18T08:03:49.331 INFO:teuthology.orchestra.run.smithi007.stdout: "rgw_topic_persistency_sleep_duration": "0", 2024-09-18T08:03:49.331 INFO:teuthology.orchestra.run.smithi007.stdout: "rgw_topic_persistency_time_to_live": "0", 2024-09-18T08:03:49.332 INFO:teuthology.orchestra.run.smithi007.stdout: "rgw_topic_require_publish_policy": "false", 2024-09-18T08:03:49.332 INFO:teuthology.orchestra.run.smithi007.stdout: "rgw_torrent_comment": "", 2024-09-18T08:03:49.332 INFO:teuthology.orchestra.run.smithi007.stdout: "rgw_torrent_createby": "", 2024-09-18T08:03:49.332 INFO:teuthology.orchestra.run.smithi007.stdout: "rgw_torrent_encoding": "", 2024-09-18T08:03:49.332 INFO:teuthology.orchestra.run.smithi007.stdout: "rgw_torrent_flag": "false", 2024-09-18T08:03:49.332 INFO:teuthology.orchestra.run.smithi007.stdout: "rgw_torrent_max_size": "5368709120", 2024-09-18T08:03:49.333 INFO:teuthology.orchestra.run.smithi007.stdout: "rgw_torrent_origin": "", 2024-09-18T08:03:49.333 INFO:teuthology.orchestra.run.smithi007.stdout: "rgw_torrent_sha_unit": "524288", 2024-09-18T08:03:49.333 INFO:teuthology.orchestra.run.smithi007.stdout: "rgw_torrent_tracker": "", 2024-09-18T08:03:49.333 INFO:teuthology.orchestra.run.smithi007.stdout: "rgw_trust_forwarded_https": "false", 2024-09-18T08:03:49.333 INFO:teuthology.orchestra.run.smithi007.stdout: "rgw_usage_log_flush_threshold": "1024", 2024-09-18T08:03:49.334 INFO:teuthology.orchestra.run.smithi007.stdout: "rgw_usage_log_tick_interval": "30", 2024-09-18T08:03:49.334 INFO:teuthology.orchestra.run.smithi007.stdout: "rgw_usage_max_shards": "32", 2024-09-18T08:03:49.334 INFO:teuthology.orchestra.run.smithi007.stdout: "rgw_usage_max_user_shards": "1", 2024-09-18T08:03:49.334 INFO:teuthology.orchestra.run.smithi007.stdout: "rgw_use_opa_authz": "false", 2024-09-18T08:03:49.334 INFO:teuthology.orchestra.run.smithi007.stdout: "rgw_user_counters_cache": "false", 2024-09-18T08:03:49.335 INFO:teuthology.orchestra.run.smithi007.stdout: "rgw_user_counters_cache_size": "10000", 2024-09-18T08:03:49.335 INFO:teuthology.orchestra.run.smithi007.stdout: "rgw_user_default_quota_max_objects": "-1", 2024-09-18T08:03:49.335 INFO:teuthology.orchestra.run.smithi007.stdout: "rgw_user_default_quota_max_size": "-1", 2024-09-18T08:03:49.335 INFO:teuthology.orchestra.run.smithi007.stdout: "rgw_user_max_buckets": "1000", 2024-09-18T08:03:49.335 INFO:teuthology.orchestra.run.smithi007.stdout: "rgw_user_policies_max_num": "100", 2024-09-18T08:03:49.335 INFO:teuthology.orchestra.run.smithi007.stdout: "rgw_user_quota_bucket_sync_interval": "180", 2024-09-18T08:03:49.336 INFO:teuthology.orchestra.run.smithi007.stdout: "rgw_user_quota_sync_idle_users": "false", 2024-09-18T08:03:49.336 INFO:teuthology.orchestra.run.smithi007.stdout: "rgw_user_quota_sync_interval": "86400", 2024-09-18T08:03:49.336 INFO:teuthology.orchestra.run.smithi007.stdout: "rgw_user_quota_sync_wait_time": "86400", 2024-09-18T08:03:49.336 INFO:teuthology.orchestra.run.smithi007.stdout: "rgw_user_unique_email": "true", 2024-09-18T08:03:49.336 INFO:teuthology.orchestra.run.smithi007.stdout: "rgw_verify_ssl": "true", 2024-09-18T08:03:49.337 INFO:teuthology.orchestra.run.smithi007.stdout: "rgw_website_routing_rules_max_num": "50", 2024-09-18T08:03:49.337 INFO:teuthology.orchestra.run.smithi007.stdout: "rgw_zone": "", 2024-09-18T08:03:49.337 INFO:teuthology.orchestra.run.smithi007.stdout: "rgw_zone_id": "", 2024-09-18T08:03:49.337 INFO:teuthology.orchestra.run.smithi007.stdout: "rgw_zone_root_pool": ".rgw.root", 2024-09-18T08:03:49.337 INFO:teuthology.orchestra.run.smithi007.stdout: "rgw_zonegroup": "", 2024-09-18T08:03:49.338 INFO:teuthology.orchestra.run.smithi007.stdout: "rgw_zonegroup_id": "", 2024-09-18T08:03:49.338 INFO:teuthology.orchestra.run.smithi007.stdout: "rgw_zonegroup_root_pool": ".rgw.root", 2024-09-18T08:03:49.338 INFO:teuthology.orchestra.run.smithi007.stdout: "rgwlc_auto_session_clear": "true", 2024-09-18T08:03:49.338 INFO:teuthology.orchestra.run.smithi007.stdout: "rgwlc_skip_bucket_step": "false", 2024-09-18T08:03:49.338 INFO:teuthology.orchestra.run.smithi007.stdout: "rocksdb_block_size": "4096", 2024-09-18T08:03:49.338 INFO:teuthology.orchestra.run.smithi007.stdout: "rocksdb_bloom_bits_per_key": "20", 2024-09-18T08:03:49.339 INFO:teuthology.orchestra.run.smithi007.stdout: "rocksdb_cache_index_and_filter_blocks": "true", 2024-09-18T08:03:49.339 INFO:teuthology.orchestra.run.smithi007.stdout: "rocksdb_cache_index_and_filter_blocks_with_high_priority": "false", 2024-09-18T08:03:49.339 INFO:teuthology.orchestra.run.smithi007.stdout: "rocksdb_cache_row_ratio": "0.000000", 2024-09-18T08:03:49.339 INFO:teuthology.orchestra.run.smithi007.stdout: "rocksdb_cache_shard_bits": "4", 2024-09-18T08:03:49.339 INFO:teuthology.orchestra.run.smithi007.stdout: "rocksdb_cache_size": "536870912", 2024-09-18T08:03:49.340 INFO:teuthology.orchestra.run.smithi007.stdout: "rocksdb_cache_type": "binned_lru", 2024-09-18T08:03:49.340 INFO:teuthology.orchestra.run.smithi007.stdout: "rocksdb_cf_compact_on_deletion": "true", 2024-09-18T08:03:49.340 INFO:teuthology.orchestra.run.smithi007.stdout: "rocksdb_cf_compact_on_deletion_sliding_window": "32768", 2024-09-18T08:03:49.340 INFO:teuthology.orchestra.run.smithi007.stdout: "rocksdb_cf_compact_on_deletion_trigger": "16384", 2024-09-18T08:03:49.340 INFO:teuthology.orchestra.run.smithi007.stdout: "rocksdb_collect_compaction_stats": "false", 2024-09-18T08:03:49.341 INFO:teuthology.orchestra.run.smithi007.stdout: "rocksdb_collect_extended_stats": "false", 2024-09-18T08:03:49.341 INFO:teuthology.orchestra.run.smithi007.stdout: "rocksdb_collect_memory_stats": "false", 2024-09-18T08:03:49.341 INFO:teuthology.orchestra.run.smithi007.stdout: "rocksdb_delete_range_threshold": "1048576", 2024-09-18T08:03:49.341 INFO:teuthology.orchestra.run.smithi007.stdout: "rocksdb_index_type": "binary_search", 2024-09-18T08:03:49.341 INFO:teuthology.orchestra.run.smithi007.stdout: "rocksdb_log_to_ceph_log": "true", 2024-09-18T08:03:49.341 INFO:teuthology.orchestra.run.smithi007.stdout: "rocksdb_metadata_block_size": "4096", 2024-09-18T08:03:49.342 INFO:teuthology.orchestra.run.smithi007.stdout: "rocksdb_partition_filters": "false", 2024-09-18T08:03:49.342 INFO:teuthology.orchestra.run.smithi007.stdout: "rocksdb_perf": "false", 2024-09-18T08:03:49.342 INFO:teuthology.orchestra.run.smithi007.stdout: "rocksdb_pin_l0_filter_and_index_blocks_in_cache": "false", 2024-09-18T08:03:49.342 INFO:teuthology.orchestra.run.smithi007.stdout: "rotating_keys_bootstrap_timeout": "30", 2024-09-18T08:03:49.342 INFO:teuthology.orchestra.run.smithi007.stdout: "rotating_keys_renewal_timeout": "10", 2024-09-18T08:03:49.342 INFO:teuthology.orchestra.run.smithi007.stdout: "run_dir": "/var/run/ceph", 2024-09-18T08:03:49.342 INFO:teuthology.orchestra.run.smithi007.stdout: "seastore_block_create": "true", 2024-09-18T08:03:49.343 INFO:teuthology.orchestra.run.smithi007.stdout: "seastore_cache_lru_size": "67108864", 2024-09-18T08:03:49.343 INFO:teuthology.orchestra.run.smithi007.stdout: "seastore_cbjournal_size": "5368709120", 2024-09-18T08:03:49.343 INFO:teuthology.orchestra.run.smithi007.stdout: "seastore_data_delta_based_overwrite": "0", 2024-09-18T08:03:49.343 INFO:teuthology.orchestra.run.smithi007.stdout: "seastore_default_max_object_size": "16777216", 2024-09-18T08:03:49.343 INFO:teuthology.orchestra.run.smithi007.stdout: "seastore_default_object_metadata_reservation": "16777216", 2024-09-18T08:03:49.343 INFO:teuthology.orchestra.run.smithi007.stdout: "seastore_device_size": "53687091200", 2024-09-18T08:03:49.343 INFO:teuthology.orchestra.run.smithi007.stdout: "seastore_disable_end_to_end_data_protection": "true", 2024-09-18T08:03:49.344 INFO:teuthology.orchestra.run.smithi007.stdout: "seastore_full_integrity_check": "false", 2024-09-18T08:03:49.344 INFO:teuthology.orchestra.run.smithi007.stdout: "seastore_journal_batch_capacity": "16", 2024-09-18T08:03:49.344 INFO:teuthology.orchestra.run.smithi007.stdout: "seastore_journal_batch_flush_size": "16777216", 2024-09-18T08:03:49.344 INFO:teuthology.orchestra.run.smithi007.stdout: "seastore_journal_batch_preferred_fullness": "0.950000", 2024-09-18T08:03:49.344 INFO:teuthology.orchestra.run.smithi007.stdout: "seastore_journal_iodepth_limit": "5", 2024-09-18T08:03:49.344 INFO:teuthology.orchestra.run.smithi007.stdout: "seastore_main_device_type": "SSD", 2024-09-18T08:03:49.345 INFO:teuthology.orchestra.run.smithi007.stdout: "seastore_max_concurrent_transactions": "8", 2024-09-18T08:03:49.345 INFO:teuthology.orchestra.run.smithi007.stdout: "seastore_max_data_allocation_size": "32768", 2024-09-18T08:03:49.345 INFO:teuthology.orchestra.run.smithi007.stdout: "seastore_multiple_tiers_default_evict_ratio": "0.600000", 2024-09-18T08:03:49.345 INFO:teuthology.orchestra.run.smithi007.stdout: "seastore_multiple_tiers_fast_evict_ratio": "0.700000", 2024-09-18T08:03:49.345 INFO:teuthology.orchestra.run.smithi007.stdout: "seastore_multiple_tiers_stop_evict_ratio": "0.500000", 2024-09-18T08:03:49.345 INFO:teuthology.orchestra.run.smithi007.stdout: "seastore_obj_data_write_amplification": "1.250000", 2024-09-18T08:03:49.345 INFO:teuthology.orchestra.run.smithi007.stdout: "seastore_segment_size": "67108864", 2024-09-18T08:03:49.345 INFO:teuthology.orchestra.run.smithi007.stdout: "set_keepcaps": "false", 2024-09-18T08:03:49.345 INFO:teuthology.orchestra.run.smithi007.stdout: "setgroup": "ceph", 2024-09-18T08:03:49.345 INFO:teuthology.orchestra.run.smithi007.stdout: "setuser": "ceph", 2024-09-18T08:03:49.345 INFO:teuthology.orchestra.run.smithi007.stdout: "setuser_match_path": "", 2024-09-18T08:03:49.346 INFO:teuthology.orchestra.run.smithi007.stdout: "target_max_misplaced_ratio": "0.050000", 2024-09-18T08:03:49.346 INFO:teuthology.orchestra.run.smithi007.stdout: "thp": "false", 2024-09-18T08:03:49.346 INFO:teuthology.orchestra.run.smithi007.stdout: "threadpool_default_timeout": "60", 2024-09-18T08:03:49.346 INFO:teuthology.orchestra.run.smithi007.stdout: "threadpool_empty_queue_max_wait": "2", 2024-09-18T08:03:49.346 INFO:teuthology.orchestra.run.smithi007.stdout: "throttler_perf_counter": "true", 2024-09-18T08:03:49.346 INFO:teuthology.orchestra.run.smithi007.stdout: "tmp_dir": "/tmp", 2024-09-18T08:03:49.346 INFO:teuthology.orchestra.run.smithi007.stdout: "tmp_file_template": "/tmp/ceph-mon.a.XXXXXX", 2024-09-18T08:03:49.346 INFO:teuthology.orchestra.run.smithi007.stdout: "uadk_compressor_enabled": "false", 2024-09-18T08:03:49.346 INFO:teuthology.orchestra.run.smithi007.stdout: "uadk_wd_sync_ctx_num": "2" 2024-09-18T08:03:49.346 INFO:teuthology.orchestra.run.smithi007.stdout:} 2024-09-18T08:03:49.480 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:03:49 smithi007 ceph-mon[29459]: 3.5 scrub starts 2024-09-18T08:03:49.480 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:03:49 smithi007 ceph-mon[29459]: 3.5 scrub ok 2024-09-18T08:03:49.481 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:03:49 smithi007 ceph-mon[29459]: osdmap e80: 8 total, 8 up, 8 in 2024-09-18T08:03:49.481 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:03:49 smithi007 ceph-mon[39249]: 3.5 scrub starts 2024-09-18T08:03:49.481 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:03:49 smithi007 ceph-mon[39249]: 3.5 scrub ok 2024-09-18T08:03:49.481 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:03:49 smithi007 ceph-mon[39249]: osdmap e80: 8 total, 8 up, 8 in 2024-09-18T08:03:49.656 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:03:49 smithi050 ceph-mon[32371]: 3.5 scrub starts 2024-09-18T08:03:49.656 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:03:49 smithi050 ceph-mon[32371]: 3.5 scrub ok 2024-09-18T08:03:49.656 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:03:49 smithi050 ceph-mon[32371]: osdmap e80: 8 total, 8 up, 8 in 2024-09-18T08:03:50.044 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid 779349ca-7593-11ef-bceb-c7b262605968 -- ceph -- tell 'mon.*' injectargs --mon_osd_down_out_interval=0 2024-09-18T08:03:50.306 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/779349ca-7593-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T08:03:50.400 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:03:50 smithi007 ceph-mon[29459]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-18T08:03:50.400 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:03:50 smithi007 ceph-mon[29459]: pgmap v26: 132 pgs: 132 active+clean; 583 KiB data, 223 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-18T08:03:50.401 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:03:50 smithi007 ceph-mon[39249]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-18T08:03:50.401 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:03:50 smithi007 ceph-mon[39249]: pgmap v26: 132 pgs: 132 active+clean; 583 KiB data, 223 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-18T08:03:50.656 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:03:50 smithi050 ceph-mon[32371]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-18T08:03:50.656 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:03:50 smithi050 ceph-mon[32371]: pgmap v26: 132 pgs: 132 active+clean; 583 KiB data, 223 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-18T08:03:50.736 INFO:teuthology.orchestra.run.smithi007.stdout:mon.a: {} 2024-09-18T08:03:50.736 INFO:teuthology.orchestra.run.smithi007.stderr:mon.a: mon_osd_down_out_interval = '' (not observed, change may require restart) 2024-09-18T08:03:50.738 INFO:teuthology.orchestra.run.smithi007.stdout:mon.b: {} 2024-09-18T08:03:50.739 INFO:teuthology.orchestra.run.smithi007.stderr:mon.b: mon_osd_down_out_interval = '' (not observed, change may require restart) 2024-09-18T08:03:50.747 INFO:teuthology.orchestra.run.smithi007.stdout:mon.c: {} 2024-09-18T08:03:50.747 INFO:teuthology.orchestra.run.smithi007.stderr:mon.c: mon_osd_down_out_interval = '' (not observed, change may require restart) 2024-09-18T08:03:51.450 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:03:51 smithi007 ceph-mon[29459]: 3.11 scrub starts 2024-09-18T08:03:51.450 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:03:51 smithi007 ceph-mon[29459]: 3.11 scrub ok 2024-09-18T08:03:51.450 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:03:51 smithi007 ceph-mon[29459]: 3.0 scrub starts 2024-09-18T08:03:51.450 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:03:51 smithi007 ceph-mon[29459]: 3.0 scrub ok 2024-09-18T08:03:51.450 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:03:51 smithi007 ceph-mon[29459]: 3.16 scrub starts 2024-09-18T08:03:51.450 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:03:51 smithi007 ceph-mon[29459]: 3.16 scrub ok 2024-09-18T08:03:51.450 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:03:51 smithi007 ceph-mon[29459]: from='client.? 172.21.15.7:0/2098271515' entity='client.admin' cmd={"prefix": "mon dump", "format": "json"} : dispatch 2024-09-18T08:03:51.450 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:03:51 smithi007 ceph-mon[39249]: 3.11 scrub starts 2024-09-18T08:03:51.451 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:03:51 smithi007 ceph-mon[39249]: 3.11 scrub ok 2024-09-18T08:03:51.451 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:03:51 smithi007 ceph-mon[39249]: 3.0 scrub starts 2024-09-18T08:03:51.451 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:03:51 smithi007 ceph-mon[39249]: 3.0 scrub ok 2024-09-18T08:03:51.451 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:03:51 smithi007 ceph-mon[39249]: 3.16 scrub starts 2024-09-18T08:03:51.451 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:03:51 smithi007 ceph-mon[39249]: 3.16 scrub ok 2024-09-18T08:03:51.451 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:03:51 smithi007 ceph-mon[39249]: from='client.? 172.21.15.7:0/2098271515' entity='client.admin' cmd={"prefix": "mon dump", "format": "json"} : dispatch 2024-09-18T08:03:51.486 INFO:teuthology.run_tasks:Running task full_sequential... 2024-09-18T08:03:51.496 INFO:teuthology.task.full_sequential:In full_sequential, running task radosbench... 2024-09-18T08:03:51.498 INFO:tasks.radosbench:Beginning radosbench... 2024-09-18T08:03:51.498 INFO:tasks.cephadm.ceph_manager.ceph:creating pool_name unique_pool_0 2024-09-18T08:03:51.498 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid 779349ca-7593-11ef-bceb-c7b262605968 -- ceph osd pool create unique_pool_0 16 2024-09-18T08:03:51.499 INFO:tasks.thrashosds.thrasher:starting do_thrash 2024-09-18T08:03:51.499 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-09-18T08:03:51.499 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-18T08:03:51.499 INFO:tasks.thrashosds.thrasher:starting do_sighup with a delay of 0.1 2024-09-18T08:03:51.499 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7 2024-09-18T08:03:51.500 INFO:tasks.thrashosds.thrasher:starting do_optrack_toggle with a delay of 2.0 2024-09-18T08:03:51.500 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid 779349ca-7593-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-18T08:03:51.500 INFO:tasks.thrashosds.thrasher:starting do_dump_ops 2024-09-18T08:03:51.501 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid 779349ca-7593-11ef-bceb-c7b262605968 -- ceph daemon osd.0 dump_ops_in_flight 2024-09-18T08:03:51.502 INFO:tasks.thrashosds.thrasher:starting do_noscrub_toggle with a delay of 2.0 2024-09-18T08:03:51.502 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid 779349ca-7593-11ef-bceb-c7b262605968 -- ceph osd set noscrub 2024-09-18T08:03:51.526 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:03:51.526 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:03:51.527 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7' 2024-09-18T08:03:51.628 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0 2024-09-18T08:03:51.656 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:03:51 smithi050 ceph-mon[32371]: 3.11 scrub starts 2024-09-18T08:03:51.656 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:03:51 smithi050 ceph-mon[32371]: 3.11 scrub ok 2024-09-18T08:03:51.656 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:03:51 smithi050 ceph-mon[32371]: 3.0 scrub starts 2024-09-18T08:03:51.656 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:03:51 smithi050 ceph-mon[32371]: 3.0 scrub ok 2024-09-18T08:03:51.656 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:03:51 smithi050 ceph-mon[32371]: 3.16 scrub starts 2024-09-18T08:03:51.656 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:03:51 smithi050 ceph-mon[32371]: 3.16 scrub ok 2024-09-18T08:03:51.656 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:03:51 smithi050 ceph-mon[32371]: from='client.? 172.21.15.7:0/2098271515' entity='client.admin' cmd={"prefix": "mon dump", "format": "json"} : dispatch 2024-09-18T08:03:51.661 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:03:51.662 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:03:51.662 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0' 2024-09-18T08:03:51.763 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3 2024-09-18T08:03:51.794 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:03:51.795 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:03:51.795 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3' 2024-09-18T08:03:51.829 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/779349ca-7593-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T08:03:51.831 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/779349ca-7593-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T08:03:51.837 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/779349ca-7593-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T08:03:51.838 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/779349ca-7593-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T08:03:51.896 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6 2024-09-18T08:03:51.918 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:03:51.919 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:03:51.921 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6' 2024-09-18T08:03:52.022 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1 2024-09-18T08:03:52.044 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:03:52.045 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:03:52.045 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1' 2024-09-18T08:03:52.146 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1 2024-09-18T08:03:52.169 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:03:52.170 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:03:52.170 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1' 2024-09-18T08:03:52.271 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0 2024-09-18T08:03:52.295 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:03:52.296 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:03:52.296 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0' 2024-09-18T08:03:52.397 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3 2024-09-18T08:03:52.406 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:03:52 smithi050 ceph-mon[32371]: 3.f scrub starts 2024-09-18T08:03:52.406 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:03:52 smithi050 ceph-mon[32371]: 3.f scrub ok 2024-09-18T08:03:52.406 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:03:52 smithi050 ceph-mon[32371]: 4.7 scrub starts 2024-09-18T08:03:52.406 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:03:52 smithi050 ceph-mon[32371]: 4.7 scrub ok 2024-09-18T08:03:52.406 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:03:52 smithi050 ceph-mon[32371]: purged_snaps scrub starts 2024-09-18T08:03:52.406 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:03:52 smithi050 ceph-mon[32371]: purged_snaps scrub ok 2024-09-18T08:03:52.406 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:03:52 smithi050 ceph-mon[32371]: pgmap v27: 132 pgs: 132 active+clean; 583 KiB data, 223 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-18T08:03:52.406 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:03:52 smithi050 ceph-mon[32371]: osdmap e81: 8 total, 8 up, 8 in 2024-09-18T08:03:52.407 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:03:52 smithi050 ceph-mon[32371]: 5.11 scrub starts 2024-09-18T08:03:52.407 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:03:52 smithi050 ceph-mon[32371]: 5.11 scrub ok 2024-09-18T08:03:52.407 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:03:52 smithi050 ceph-mon[32371]: 4.11 scrub starts 2024-09-18T08:03:52.407 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:03:52 smithi050 ceph-mon[32371]: 4.11 scrub ok 2024-09-18T08:03:52.421 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:03:52.422 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:03:52.423 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3' 2024-09-18T08:03:52.480 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:03:52 smithi007 ceph-mon[29459]: 3.f scrub starts 2024-09-18T08:03:52.481 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:03:52 smithi007 ceph-mon[29459]: 3.f scrub ok 2024-09-18T08:03:52.481 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:03:52 smithi007 ceph-mon[29459]: 4.7 scrub starts 2024-09-18T08:03:52.481 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:03:52 smithi007 ceph-mon[29459]: 4.7 scrub ok 2024-09-18T08:03:52.481 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:03:52 smithi007 ceph-mon[29459]: purged_snaps scrub starts 2024-09-18T08:03:52.481 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:03:52 smithi007 ceph-mon[29459]: purged_snaps scrub ok 2024-09-18T08:03:52.481 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:03:52 smithi007 ceph-mon[29459]: pgmap v27: 132 pgs: 132 active+clean; 583 KiB data, 223 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-18T08:03:52.481 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:03:52 smithi007 ceph-mon[29459]: osdmap e81: 8 total, 8 up, 8 in 2024-09-18T08:03:52.481 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:03:52 smithi007 ceph-mon[29459]: 5.11 scrub starts 2024-09-18T08:03:52.481 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:03:52 smithi007 ceph-mon[29459]: 5.11 scrub ok 2024-09-18T08:03:52.481 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:03:52 smithi007 ceph-mon[29459]: 4.11 scrub starts 2024-09-18T08:03:52.481 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:03:52 smithi007 ceph-mon[29459]: 4.11 scrub ok 2024-09-18T08:03:52.482 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:03:52 smithi007 ceph-mon[39249]: 3.f scrub starts 2024-09-18T08:03:52.482 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:03:52 smithi007 ceph-mon[39249]: 3.f scrub ok 2024-09-18T08:03:52.482 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:03:52 smithi007 ceph-mon[39249]: 4.7 scrub starts 2024-09-18T08:03:52.482 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:03:52 smithi007 ceph-mon[39249]: 4.7 scrub ok 2024-09-18T08:03:52.482 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:03:52 smithi007 ceph-mon[39249]: purged_snaps scrub starts 2024-09-18T08:03:52.482 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:03:52 smithi007 ceph-mon[39249]: purged_snaps scrub ok 2024-09-18T08:03:52.483 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:03:52 smithi007 ceph-mon[39249]: pgmap v27: 132 pgs: 132 active+clean; 583 KiB data, 223 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-18T08:03:52.483 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:03:52 smithi007 ceph-mon[39249]: osdmap e81: 8 total, 8 up, 8 in 2024-09-18T08:03:52.483 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:03:52 smithi007 ceph-mon[39249]: 5.11 scrub starts 2024-09-18T08:03:52.483 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:03:52 smithi007 ceph-mon[39249]: 5.11 scrub ok 2024-09-18T08:03:52.484 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:03:52 smithi007 ceph-mon[39249]: 4.11 scrub starts 2024-09-18T08:03:52.484 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:03:52 smithi007 ceph-mon[39249]: 4.11 scrub ok 2024-09-18T08:03:52.523 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6 2024-09-18T08:03:52.548 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:03:52.549 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:03:52.549 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6' 2024-09-18T08:03:52.607 INFO:teuthology.orchestra.run.smithi007.stdout:{ 2024-09-18T08:03:52.607 INFO:teuthology.orchestra.run.smithi007.stdout: "ops": [], 2024-09-18T08:03:52.607 INFO:teuthology.orchestra.run.smithi007.stdout: "num_ops": 0 2024-09-18T08:03:52.607 INFO:teuthology.orchestra.run.smithi007.stdout:} 2024-09-18T08:03:52.650 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5 2024-09-18T08:03:52.673 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:03:52.674 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:03:52.674 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5' 2024-09-18T08:03:52.775 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0 2024-09-18T08:03:52.798 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:03:52.799 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:03:52.799 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0' 2024-09-18T08:03:52.900 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2 2024-09-18T08:03:52.922 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:03:52.923 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:03:52.923 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2' 2024-09-18T08:03:53.025 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4 2024-09-18T08:03:53.049 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:03:53.049 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:03:53.050 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4' 2024-09-18T08:03:53.150 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1 2024-09-18T08:03:53.172 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:03:53.173 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:03:53.174 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1' 2024-09-18T08:03:53.251 INFO:teuthology.orchestra.run.smithi007.stdout:osd.0: {} 2024-09-18T08:03:53.251 INFO:teuthology.orchestra.run.smithi007.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-18T08:03:53.269 INFO:teuthology.orchestra.run.smithi007.stdout:osd.1: {} 2024-09-18T08:03:53.272 INFO:teuthology.orchestra.run.smithi007.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-18T08:03:53.275 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3 2024-09-18T08:03:53.285 INFO:teuthology.orchestra.run.smithi007.stdout:osd.2: {} 2024-09-18T08:03:53.285 INFO:teuthology.orchestra.run.smithi007.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-18T08:03:53.296 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:03:53.297 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:03:53.297 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3' 2024-09-18T08:03:53.302 INFO:teuthology.orchestra.run.smithi007.stdout:osd.3: {} 2024-09-18T08:03:53.302 INFO:teuthology.orchestra.run.smithi007.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-18T08:03:53.319 INFO:teuthology.orchestra.run.smithi007.stdout:osd.4: {} 2024-09-18T08:03:53.319 INFO:teuthology.orchestra.run.smithi007.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-18T08:03:53.336 INFO:teuthology.orchestra.run.smithi007.stdout:osd.5: {} 2024-09-18T08:03:53.336 INFO:teuthology.orchestra.run.smithi007.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-18T08:03:53.336 INFO:journalctl@ceph.alertmanager.a.smithi007.stdout:Sep 18 08:03:53 smithi007 ceph-779349ca-7593-11ef-bceb-c7b262605968-alertmanager-a[70326]: ts=2024-09-18T08:03:53.078Z caller=cluster.go:700 level=info component=cluster msg="gossip settled; proceeding" elapsed=10.002238538s 2024-09-18T08:03:53.337 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:03:53 smithi007 ceph-mon[29459]: 5.e scrub starts 2024-09-18T08:03:53.337 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:03:53 smithi007 ceph-mon[29459]: 5.e scrub ok 2024-09-18T08:03:53.337 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:03:53 smithi007 ceph-mon[29459]: 3.7 scrub starts 2024-09-18T08:03:53.337 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:03:53 smithi007 ceph-mon[29459]: 3.7 scrub ok 2024-09-18T08:03:53.337 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:03:53 smithi007 ceph-mon[39249]: 5.e scrub starts 2024-09-18T08:03:53.337 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:03:53 smithi007 ceph-mon[39249]: 5.e scrub ok 2024-09-18T08:03:53.337 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:03:53 smithi007 ceph-mon[39249]: 3.7 scrub starts 2024-09-18T08:03:53.337 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:03:53 smithi007 ceph-mon[39249]: 3.7 scrub ok 2024-09-18T08:03:53.353 INFO:teuthology.orchestra.run.smithi007.stdout:osd.6: {} 2024-09-18T08:03:53.353 INFO:teuthology.orchestra.run.smithi007.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-18T08:03:53.370 INFO:teuthology.orchestra.run.smithi007.stdout:osd.7: {} 2024-09-18T08:03:53.370 INFO:teuthology.orchestra.run.smithi007.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-18T08:03:53.399 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6 2024-09-18T08:03:53.424 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:03:53.425 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:03:53.425 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6' 2024-09-18T08:03:53.526 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0 2024-09-18T08:03:53.548 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:03:53.548 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:03:53.549 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0' 2024-09-18T08:03:53.553 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:03:53 smithi050 ceph-mon[32371]: 5.e scrub starts 2024-09-18T08:03:53.553 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:03:53 smithi050 ceph-mon[32371]: 5.e scrub ok 2024-09-18T08:03:53.553 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:03:53 smithi050 ceph-mon[32371]: 3.7 scrub starts 2024-09-18T08:03:53.553 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:03:53 smithi050 ceph-mon[32371]: 3.7 scrub ok 2024-09-18T08:03:53.650 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2 2024-09-18T08:03:53.672 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:03:53.673 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:03:53.673 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2' 2024-09-18T08:03:53.774 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2 2024-09-18T08:03:53.797 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:03:53.797 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:03:53.797 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2' 2024-09-18T08:03:53.898 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5 2024-09-18T08:03:53.925 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:03:53.926 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:03:53.926 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5' 2024-09-18T08:03:54.027 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0 2024-09-18T08:03:54.049 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:03:54.050 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:03:54.050 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0' 2024-09-18T08:03:54.055 INFO:teuthology.orchestra.run.smithi007.stderr:noscrub is set 2024-09-18T08:03:54.152 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0 2024-09-18T08:03:54.174 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:03:54.175 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:03:54.175 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0' 2024-09-18T08:03:54.276 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7 2024-09-18T08:03:54.305 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:03:54.305 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:03:54.305 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7' 2024-09-18T08:03:54.406 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4 2024-09-18T08:03:54.417 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:03:54 smithi007 ceph-mon[29459]: 5.9 scrub starts 2024-09-18T08:03:54.417 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:03:54 smithi007 ceph-mon[29459]: 5.9 scrub ok 2024-09-18T08:03:54.417 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:03:54 smithi007 ceph-mon[29459]: pgmap v29: 132 pgs: 132 active+clean; 583 KiB data, 223 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-18T08:03:54.417 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:03:54 smithi007 ceph-mon[29459]: from='client.? 172.21.15.7:0/765177557' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-18T08:03:54.417 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:03:54 smithi007 ceph-mon[29459]: from='client.? 172.21.15.7:0/174356376' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-18T08:03:54.417 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:03:54 smithi007 ceph-mon[29459]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-18T08:03:54.417 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:03:54 smithi007 ceph-mon[29459]: purged_snaps scrub starts 2024-09-18T08:03:54.417 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:03:54 smithi007 ceph-mon[29459]: purged_snaps scrub ok 2024-09-18T08:03:54.417 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:03:54 smithi007 ceph-mon[29459]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-18T08:03:54.417 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:03:54 smithi007 ceph-mon[29459]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-18T08:03:54.417 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:03:54 smithi007 ceph-mon[29459]: osdmap e82: 8 total, 8 up, 8 in 2024-09-18T08:03:54.418 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:03:54 smithi007 ceph-mon[39249]: 5.9 scrub starts 2024-09-18T08:03:54.418 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:03:54 smithi007 ceph-mon[39249]: 5.9 scrub ok 2024-09-18T08:03:54.418 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:03:54 smithi007 ceph-mon[39249]: pgmap v29: 132 pgs: 132 active+clean; 583 KiB data, 223 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-18T08:03:54.418 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:03:54 smithi007 ceph-mon[39249]: from='client.? 172.21.15.7:0/765177557' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-18T08:03:54.418 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:03:54 smithi007 ceph-mon[39249]: from='client.? 172.21.15.7:0/174356376' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-18T08:03:54.418 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:03:54 smithi007 ceph-mon[39249]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-18T08:03:54.418 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:03:54 smithi007 ceph-mon[39249]: purged_snaps scrub starts 2024-09-18T08:03:54.418 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:03:54 smithi007 ceph-mon[39249]: purged_snaps scrub ok 2024-09-18T08:03:54.418 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:03:54 smithi007 ceph-mon[39249]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-18T08:03:54.418 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:03:54 smithi007 ceph-mon[39249]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-18T08:03:54.418 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:03:54 smithi007 ceph-mon[39249]: osdmap e82: 8 total, 8 up, 8 in 2024-09-18T08:03:54.430 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:03:54.431 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:03:54.431 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4' 2024-09-18T08:03:54.532 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2 2024-09-18T08:03:54.555 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:03:54.556 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:03:54.556 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2' 2024-09-18T08:03:54.614 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:03:54 smithi050 ceph-mon[32371]: 5.9 scrub starts 2024-09-18T08:03:54.614 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:03:54 smithi050 ceph-mon[32371]: 5.9 scrub ok 2024-09-18T08:03:54.614 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:03:54 smithi050 ceph-mon[32371]: pgmap v29: 132 pgs: 132 active+clean; 583 KiB data, 223 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-18T08:03:54.615 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:03:54 smithi050 ceph-mon[32371]: from='client.? 172.21.15.7:0/765177557' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-18T08:03:54.615 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:03:54 smithi050 ceph-mon[32371]: from='client.? 172.21.15.7:0/174356376' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-18T08:03:54.615 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:03:54 smithi050 ceph-mon[32371]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-18T08:03:54.615 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:03:54 smithi050 ceph-mon[32371]: purged_snaps scrub starts 2024-09-18T08:03:54.615 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:03:54 smithi050 ceph-mon[32371]: purged_snaps scrub ok 2024-09-18T08:03:54.615 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:03:54 smithi050 ceph-mon[32371]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-18T08:03:54.615 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:03:54 smithi050 ceph-mon[32371]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-18T08:03:54.615 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:03:54 smithi050 ceph-mon[32371]: osdmap e82: 8 total, 8 up, 8 in 2024-09-18T08:03:54.657 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4 2024-09-18T08:03:54.680 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:03:54.681 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:03:54.681 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4' 2024-09-18T08:03:54.782 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0 2024-09-18T08:03:54.808 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:03:54.808 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:03:54.809 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0' 2024-09-18T08:03:54.906 INFO:journalctl@ceph.grafana.a.smithi050.stdout:Sep 18 08:03:54 smithi050 systemd[1]: Starting Ceph grafana.a for 779349ca-7593-11ef-bceb-c7b262605968... 2024-09-18T08:03:54.909 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7 2024-09-18T08:03:54.932 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:03:54.933 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:03:54.933 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7' 2024-09-18T08:03:55.035 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4 2024-09-18T08:03:55.057 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:03:55.058 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:03:55.058 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4' 2024-09-18T08:03:55.159 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6 2024-09-18T08:03:55.180 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:03:55.181 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:03:55.182 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6' 2024-09-18T08:03:55.279 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:03:55 smithi050 ceph-mon[32371]: 4.1 scrub starts 2024-09-18T08:03:55.279 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:03:55 smithi050 ceph-mon[32371]: 4.1 scrub ok 2024-09-18T08:03:55.279 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:03:55 smithi050 ceph-mon[32371]: 3.1d scrub starts 2024-09-18T08:03:55.279 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:03:55 smithi050 ceph-mon[32371]: 3.1d scrub ok 2024-09-18T08:03:55.279 INFO:journalctl@ceph.grafana.a.smithi050.stdout:Sep 18 08:03:55 smithi050 podman[60354]: 2024-09-18 08:03:54.917986733 +0000 UTC m=+0.017329645 image pull c8b91775d855b99270fc5d22f3c6737e8cca01ef4c25c8b0362295e0746fa39b quay.io/ceph/grafana:10.4.0 2024-09-18T08:03:55.280 INFO:journalctl@ceph.grafana.a.smithi050.stdout:Sep 18 08:03:55 smithi050 podman[60354]: 2024-09-18 08:03:55.025091339 +0000 UTC m=+0.124434242 container create 0b67d46a91e97bd3220d4aacdfb0ef42826594b693e572c2b690c12fe6d382f1 (image=quay.io/ceph/grafana:10.4.0, name=ceph-779349ca-7593-11ef-bceb-c7b262605968-grafana-a, maintainer=Grafana Labs ) 2024-09-18T08:03:55.280 INFO:journalctl@ceph.grafana.a.smithi050.stdout:Sep 18 08:03:55 smithi050 podman[60354]: 2024-09-18 08:03:55.277763061 +0000 UTC m=+0.377105973 container init 0b67d46a91e97bd3220d4aacdfb0ef42826594b693e572c2b690c12fe6d382f1 (image=quay.io/ceph/grafana:10.4.0, name=ceph-779349ca-7593-11ef-bceb-c7b262605968-grafana-a, maintainer=Grafana Labs ) 2024-09-18T08:03:55.282 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4 2024-09-18T08:03:55.306 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:03:55.307 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:03:55.307 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4' 2024-09-18T08:03:55.363 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:03:55 smithi007 ceph-mon[29459]: 4.1 scrub starts 2024-09-18T08:03:55.363 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:03:55 smithi007 ceph-mon[29459]: 4.1 scrub ok 2024-09-18T08:03:55.363 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:03:55 smithi007 ceph-mon[29459]: 3.1d scrub starts 2024-09-18T08:03:55.363 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:03:55 smithi007 ceph-mon[29459]: 3.1d scrub ok 2024-09-18T08:03:55.364 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:03:55 smithi007 ceph-mon[39249]: 4.1 scrub starts 2024-09-18T08:03:55.364 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:03:55 smithi007 ceph-mon[39249]: 4.1 scrub ok 2024-09-18T08:03:55.364 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:03:55 smithi007 ceph-mon[39249]: 3.1d scrub starts 2024-09-18T08:03:55.364 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:03:55 smithi007 ceph-mon[39249]: 3.1d scrub ok 2024-09-18T08:03:55.399 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid 779349ca-7593-11ef-bceb-c7b262605968 -- ceph daemon osd.0 dump_blocked_ops 2024-09-18T08:03:55.408 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6 2024-09-18T08:03:55.435 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:03:55.436 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:03:55.436 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6' 2024-09-18T08:03:55.534 INFO:journalctl@ceph.grafana.a.smithi050.stdout:Sep 18 08:03:55 smithi050 podman[60354]: 2024-09-18 08:03:55.281811536 +0000 UTC m=+0.381154431 container start 0b67d46a91e97bd3220d4aacdfb0ef42826594b693e572c2b690c12fe6d382f1 (image=quay.io/ceph/grafana:10.4.0, name=ceph-779349ca-7593-11ef-bceb-c7b262605968-grafana-a, maintainer=Grafana Labs ) 2024-09-18T08:03:55.534 INFO:journalctl@ceph.grafana.a.smithi050.stdout:Sep 18 08:03:55 smithi050 bash[60354]: 0b67d46a91e97bd3220d4aacdfb0ef42826594b693e572c2b690c12fe6d382f1 2024-09-18T08:03:55.534 INFO:journalctl@ceph.grafana.a.smithi050.stdout:Sep 18 08:03:55 smithi050 systemd[1]: Started Ceph grafana.a for 779349ca-7593-11ef-bceb-c7b262605968. 2024-09-18T08:03:55.534 INFO:journalctl@ceph.grafana.a.smithi050.stdout:Sep 18 08:03:55 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-grafana-a[60403]: logger=settings t=2024-09-18T08:03:55.412112518Z level=info msg="Starting Grafana" version=10.4.0 commit=03f502a94d17f7dc4e6c34acdf8428aedd986e4c branch=HEAD compiled=2024-09-18T08:03:55Z 2024-09-18T08:03:55.535 INFO:journalctl@ceph.grafana.a.smithi050.stdout:Sep 18 08:03:55 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-grafana-a[60403]: logger=settings t=2024-09-18T08:03:55.41233136Z level=info msg="Config loaded from" file=/usr/share/grafana/conf/defaults.ini 2024-09-18T08:03:55.535 INFO:journalctl@ceph.grafana.a.smithi050.stdout:Sep 18 08:03:55 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-grafana-a[60403]: logger=settings t=2024-09-18T08:03:55.412338982Z level=info msg="Config loaded from" file=/etc/grafana/grafana.ini 2024-09-18T08:03:55.535 INFO:journalctl@ceph.grafana.a.smithi050.stdout:Sep 18 08:03:55 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-grafana-a[60403]: logger=settings t=2024-09-18T08:03:55.412342314Z level=info msg="Config overridden from command line" arg="default.paths.data=/var/lib/grafana" 2024-09-18T08:03:55.535 INFO:journalctl@ceph.grafana.a.smithi050.stdout:Sep 18 08:03:55 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-grafana-a[60403]: logger=settings t=2024-09-18T08:03:55.412345265Z level=info msg="Config overridden from command line" arg="default.paths.logs=/var/log/grafana" 2024-09-18T08:03:55.535 INFO:journalctl@ceph.grafana.a.smithi050.stdout:Sep 18 08:03:55 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-grafana-a[60403]: logger=settings t=2024-09-18T08:03:55.412348056Z level=info msg="Config overridden from command line" arg="default.paths.plugins=/var/lib/grafana/plugins" 2024-09-18T08:03:55.535 INFO:journalctl@ceph.grafana.a.smithi050.stdout:Sep 18 08:03:55 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-grafana-a[60403]: logger=settings t=2024-09-18T08:03:55.412350742Z level=info msg="Config overridden from command line" arg="default.paths.provisioning=/etc/grafana/provisioning" 2024-09-18T08:03:55.535 INFO:journalctl@ceph.grafana.a.smithi050.stdout:Sep 18 08:03:55 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-grafana-a[60403]: logger=settings t=2024-09-18T08:03:55.412353546Z level=info msg="Config overridden from command line" arg="default.log.mode=console" 2024-09-18T08:03:55.535 INFO:journalctl@ceph.grafana.a.smithi050.stdout:Sep 18 08:03:55 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-grafana-a[60403]: logger=settings t=2024-09-18T08:03:55.412356749Z level=info msg="Config overridden from Environment variable" var="GF_PATHS_DATA=/var/lib/grafana" 2024-09-18T08:03:55.535 INFO:journalctl@ceph.grafana.a.smithi050.stdout:Sep 18 08:03:55 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-grafana-a[60403]: logger=settings t=2024-09-18T08:03:55.412359646Z level=info msg="Config overridden from Environment variable" var="GF_PATHS_LOGS=/var/log/grafana" 2024-09-18T08:03:55.535 INFO:journalctl@ceph.grafana.a.smithi050.stdout:Sep 18 08:03:55 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-grafana-a[60403]: logger=settings t=2024-09-18T08:03:55.412362215Z level=info msg="Config overridden from Environment variable" var="GF_PATHS_PLUGINS=/var/lib/grafana/plugins" 2024-09-18T08:03:55.535 INFO:journalctl@ceph.grafana.a.smithi050.stdout:Sep 18 08:03:55 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-grafana-a[60403]: logger=settings t=2024-09-18T08:03:55.412364889Z level=info msg="Config overridden from Environment variable" var="GF_PATHS_PROVISIONING=/etc/grafana/provisioning" 2024-09-18T08:03:55.536 INFO:journalctl@ceph.grafana.a.smithi050.stdout:Sep 18 08:03:55 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-grafana-a[60403]: logger=settings t=2024-09-18T08:03:55.412367733Z level=info msg=Target target=[all] 2024-09-18T08:03:55.536 INFO:journalctl@ceph.grafana.a.smithi050.stdout:Sep 18 08:03:55 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-grafana-a[60403]: logger=settings t=2024-09-18T08:03:55.412374122Z level=info msg="Path Home" path=/usr/share/grafana 2024-09-18T08:03:55.536 INFO:journalctl@ceph.grafana.a.smithi050.stdout:Sep 18 08:03:55 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-grafana-a[60403]: logger=settings t=2024-09-18T08:03:55.412376851Z level=info msg="Path Data" path=/var/lib/grafana 2024-09-18T08:03:55.536 INFO:journalctl@ceph.grafana.a.smithi050.stdout:Sep 18 08:03:55 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-grafana-a[60403]: logger=settings t=2024-09-18T08:03:55.412379458Z level=info msg="Path Logs" path=/var/log/grafana 2024-09-18T08:03:55.536 INFO:journalctl@ceph.grafana.a.smithi050.stdout:Sep 18 08:03:55 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-grafana-a[60403]: logger=settings t=2024-09-18T08:03:55.412382613Z level=info msg="Path Plugins" path=/var/lib/grafana/plugins 2024-09-18T08:03:55.536 INFO:journalctl@ceph.grafana.a.smithi050.stdout:Sep 18 08:03:55 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-grafana-a[60403]: logger=settings t=2024-09-18T08:03:55.412385285Z level=info msg="Path Provisioning" path=/etc/grafana/provisioning 2024-09-18T08:03:55.536 INFO:journalctl@ceph.grafana.a.smithi050.stdout:Sep 18 08:03:55 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-grafana-a[60403]: logger=settings t=2024-09-18T08:03:55.412387941Z level=info msg="App mode production" 2024-09-18T08:03:55.536 INFO:journalctl@ceph.grafana.a.smithi050.stdout:Sep 18 08:03:55 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-grafana-a[60403]: logger=sqlstore t=2024-09-18T08:03:55.412632322Z level=info msg="Connecting to DB" dbtype=sqlite3 2024-09-18T08:03:55.536 INFO:journalctl@ceph.grafana.a.smithi050.stdout:Sep 18 08:03:55 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-grafana-a[60403]: logger=sqlstore t=2024-09-18T08:03:55.412646253Z level=warn msg="SQLite database file has broader permissions than it should" path=/var/lib/grafana/grafana.db mode=-rw-r--r-- expected=-rw-r----- 2024-09-18T08:03:55.536 INFO:journalctl@ceph.grafana.a.smithi050.stdout:Sep 18 08:03:55 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-grafana-a[60403]: logger=migrator t=2024-09-18T08:03:55.413112423Z level=info msg="Starting DB migrations" 2024-09-18T08:03:55.536 INFO:journalctl@ceph.grafana.a.smithi050.stdout:Sep 18 08:03:55 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-grafana-a[60403]: logger=migrator t=2024-09-18T08:03:55.414198249Z level=info msg="Executing migration" id="create migration_log table" 2024-09-18T08:03:55.537 INFO:journalctl@ceph.grafana.a.smithi050.stdout:Sep 18 08:03:55 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-grafana-a[60403]: logger=migrator t=2024-09-18T08:03:55.415004166Z level=info msg="Migration successfully executed" id="create migration_log table" duration=805.714µs 2024-09-18T08:03:55.537 INFO:journalctl@ceph.grafana.a.smithi050.stdout:Sep 18 08:03:55 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-grafana-a[60403]: logger=migrator t=2024-09-18T08:03:55.415574711Z level=info msg="Executing migration" id="create user table" 2024-09-18T08:03:55.537 INFO:journalctl@ceph.grafana.a.smithi050.stdout:Sep 18 08:03:55 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-grafana-a[60403]: logger=migrator t=2024-09-18T08:03:55.416171303Z level=info msg="Migration successfully executed" id="create user table" duration=597.293µs 2024-09-18T08:03:55.537 INFO:journalctl@ceph.grafana.a.smithi050.stdout:Sep 18 08:03:55 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-grafana-a[60403]: logger=migrator t=2024-09-18T08:03:55.416613471Z level=info msg="Executing migration" id="add unique index user.login" 2024-09-18T08:03:55.537 INFO:journalctl@ceph.grafana.a.smithi050.stdout:Sep 18 08:03:55 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-grafana-a[60403]: logger=migrator t=2024-09-18T08:03:55.417178618Z level=info msg="Migration successfully executed" id="add unique index user.login" duration=563.117µs 2024-09-18T08:03:55.537 INFO:journalctl@ceph.grafana.a.smithi050.stdout:Sep 18 08:03:55 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-grafana-a[60403]: logger=migrator t=2024-09-18T08:03:55.417633923Z level=info msg="Executing migration" id="add unique index user.email" 2024-09-18T08:03:55.537 INFO:journalctl@ceph.grafana.a.smithi050.stdout:Sep 18 08:03:55 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-grafana-a[60403]: logger=migrator t=2024-09-18T08:03:55.418180749Z level=info msg="Migration successfully executed" id="add unique index user.email" duration=547.415µs 2024-09-18T08:03:55.537 INFO:journalctl@ceph.grafana.a.smithi050.stdout:Sep 18 08:03:55 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-grafana-a[60403]: logger=migrator t=2024-09-18T08:03:55.418571532Z level=info msg="Executing migration" id="drop index UQE_user_login - v1" 2024-09-18T08:03:55.537 INFO:journalctl@ceph.grafana.a.smithi050.stdout:Sep 18 08:03:55 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-grafana-a[60403]: logger=migrator t=2024-09-18T08:03:55.419076536Z level=info msg="Migration successfully executed" id="drop index UQE_user_login - v1" duration=505.081µs 2024-09-18T08:03:55.537 INFO:journalctl@ceph.grafana.a.smithi050.stdout:Sep 18 08:03:55 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-grafana-a[60403]: logger=migrator t=2024-09-18T08:03:55.419555459Z level=info msg="Executing migration" id="drop index UQE_user_email - v1" 2024-09-18T08:03:55.537 INFO:journalctl@ceph.grafana.a.smithi050.stdout:Sep 18 08:03:55 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-grafana-a[60403]: logger=migrator t=2024-09-18T08:03:55.420684588Z level=info msg="Migration successfully executed" id="drop index UQE_user_email - v1" duration=1.12522ms 2024-09-18T08:03:55.538 INFO:journalctl@ceph.grafana.a.smithi050.stdout:Sep 18 08:03:55 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-grafana-a[60403]: logger=migrator t=2024-09-18T08:03:55.421417623Z level=info msg="Executing migration" id="Rename table user to user_v1 - v1" 2024-09-18T08:03:55.538 INFO:journalctl@ceph.grafana.a.smithi050.stdout:Sep 18 08:03:55 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-grafana-a[60403]: logger=migrator t=2024-09-18T08:03:55.424168855Z level=info msg="Migration successfully executed" id="Rename table user to user_v1 - v1" duration=2.747457ms 2024-09-18T08:03:55.538 INFO:journalctl@ceph.grafana.a.smithi050.stdout:Sep 18 08:03:55 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-grafana-a[60403]: logger=migrator t=2024-09-18T08:03:55.424787083Z level=info msg="Executing migration" id="create user table v2" 2024-09-18T08:03:55.538 INFO:journalctl@ceph.grafana.a.smithi050.stdout:Sep 18 08:03:55 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-grafana-a[60403]: logger=migrator t=2024-09-18T08:03:55.425529199Z level=info msg="Migration successfully executed" id="create user table v2" duration=742.44µs 2024-09-18T08:03:55.538 INFO:journalctl@ceph.grafana.a.smithi050.stdout:Sep 18 08:03:55 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-grafana-a[60403]: logger=migrator t=2024-09-18T08:03:55.425859863Z level=info msg="Executing migration" id="create index UQE_user_login - v2" 2024-09-18T08:03:55.538 INFO:journalctl@ceph.grafana.a.smithi050.stdout:Sep 18 08:03:55 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-grafana-a[60403]: logger=migrator t=2024-09-18T08:03:55.426358949Z level=info msg="Migration successfully executed" id="create index UQE_user_login - v2" duration=498.709µs 2024-09-18T08:03:55.538 INFO:journalctl@ceph.grafana.a.smithi050.stdout:Sep 18 08:03:55 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-grafana-a[60403]: logger=migrator t=2024-09-18T08:03:55.426673235Z level=info msg="Executing migration" id="create index UQE_user_email - v2" 2024-09-18T08:03:55.538 INFO:journalctl@ceph.grafana.a.smithi050.stdout:Sep 18 08:03:55 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-grafana-a[60403]: logger=migrator t=2024-09-18T08:03:55.427142346Z level=info msg="Migration successfully executed" id="create index UQE_user_email - v2" duration=469.035µs 2024-09-18T08:03:55.538 INFO:journalctl@ceph.grafana.a.smithi050.stdout:Sep 18 08:03:55 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-grafana-a[60403]: logger=migrator t=2024-09-18T08:03:55.427563266Z level=info msg="Executing migration" id="copy data_source v1 to v2" 2024-09-18T08:03:55.538 INFO:journalctl@ceph.grafana.a.smithi050.stdout:Sep 18 08:03:55 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-grafana-a[60403]: logger=migrator t=2024-09-18T08:03:55.427974493Z level=info msg="Migration successfully executed" id="copy data_source v1 to v2" duration=411.418µs 2024-09-18T08:03:55.538 INFO:journalctl@ceph.grafana.a.smithi050.stdout:Sep 18 08:03:55 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-grafana-a[60403]: logger=migrator t=2024-09-18T08:03:55.428388821Z level=info msg="Executing migration" id="Drop old table user_v1" 2024-09-18T08:03:55.538 INFO:journalctl@ceph.grafana.a.smithi050.stdout:Sep 18 08:03:55 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-grafana-a[60403]: logger=migrator t=2024-09-18T08:03:55.428972107Z level=info msg="Migration successfully executed" id="Drop old table user_v1" duration=582.993µs 2024-09-18T08:03:55.539 INFO:journalctl@ceph.grafana.a.smithi050.stdout:Sep 18 08:03:55 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-grafana-a[60403]: logger=migrator t=2024-09-18T08:03:55.429402991Z level=info msg="Executing migration" id="Add column help_flags1 to user table" 2024-09-18T08:03:55.539 INFO:journalctl@ceph.grafana.a.smithi050.stdout:Sep 18 08:03:55 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-grafana-a[60403]: logger=migrator t=2024-09-18T08:03:55.43057519Z level=info msg="Migration successfully executed" id="Add column help_flags1 to user table" duration=1.171741ms 2024-09-18T08:03:55.539 INFO:journalctl@ceph.grafana.a.smithi050.stdout:Sep 18 08:03:55 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-grafana-a[60403]: logger=migrator t=2024-09-18T08:03:55.431024567Z level=info msg="Executing migration" id="Update user table charset" 2024-09-18T08:03:55.539 INFO:journalctl@ceph.grafana.a.smithi050.stdout:Sep 18 08:03:55 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-grafana-a[60403]: logger=migrator t=2024-09-18T08:03:55.431051801Z level=info msg="Migration successfully executed" id="Update user table charset" duration=28.199µs 2024-09-18T08:03:55.539 INFO:journalctl@ceph.grafana.a.smithi050.stdout:Sep 18 08:03:55 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-grafana-a[60403]: logger=migrator t=2024-09-18T08:03:55.431586747Z level=info msg="Executing migration" id="Add last_seen_at column to user" 2024-09-18T08:03:55.539 INFO:journalctl@ceph.grafana.a.smithi050.stdout:Sep 18 08:03:55 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-grafana-a[60403]: logger=migrator t=2024-09-18T08:03:55.432675839Z level=info msg="Migration successfully executed" id="Add last_seen_at column to user" duration=1.087062ms 2024-09-18T08:03:55.539 INFO:journalctl@ceph.grafana.a.smithi050.stdout:Sep 18 08:03:55 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-grafana-a[60403]: logger=migrator t=2024-09-18T08:03:55.433062947Z level=info msg="Executing migration" id="Add missing user data" 2024-09-18T08:03:55.539 INFO:journalctl@ceph.grafana.a.smithi050.stdout:Sep 18 08:03:55 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-grafana-a[60403]: logger=migrator t=2024-09-18T08:03:55.433214738Z level=info msg="Migration successfully executed" id="Add missing user data" duration=151.832µs 2024-09-18T08:03:55.539 INFO:journalctl@ceph.grafana.a.smithi050.stdout:Sep 18 08:03:55 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-grafana-a[60403]: logger=migrator t=2024-09-18T08:03:55.433587146Z level=info msg="Executing migration" id="Add is_disabled column to user" 2024-09-18T08:03:55.539 INFO:journalctl@ceph.grafana.a.smithi050.stdout:Sep 18 08:03:55 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-grafana-a[60403]: logger=migrator t=2024-09-18T08:03:55.434453739Z level=info msg="Migration successfully executed" id="Add is_disabled column to user" duration=865.729µs 2024-09-18T08:03:55.539 INFO:journalctl@ceph.grafana.a.smithi050.stdout:Sep 18 08:03:55 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-grafana-a[60403]: logger=migrator t=2024-09-18T08:03:55.434894415Z level=info msg="Executing migration" id="Add index user.login/user.email" 2024-09-18T08:03:55.539 INFO:journalctl@ceph.grafana.a.smithi050.stdout:Sep 18 08:03:55 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-grafana-a[60403]: logger=migrator t=2024-09-18T08:03:55.435472184Z level=info msg="Migration successfully executed" id="Add index user.login/user.email" duration=577.924µs 2024-09-18T08:03:55.539 INFO:journalctl@ceph.grafana.a.smithi050.stdout:Sep 18 08:03:55 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-grafana-a[60403]: logger=migrator t=2024-09-18T08:03:55.435808013Z level=info msg="Executing migration" id="Add is_service_account column to user" 2024-09-18T08:03:55.539 INFO:journalctl@ceph.grafana.a.smithi050.stdout:Sep 18 08:03:55 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-grafana-a[60403]: logger=migrator t=2024-09-18T08:03:55.436608287Z level=info msg="Migration successfully executed" id="Add is_service_account column to user" duration=801.296µs 2024-09-18T08:03:55.540 INFO:journalctl@ceph.grafana.a.smithi050.stdout:Sep 18 08:03:55 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-grafana-a[60403]: logger=migrator t=2024-09-18T08:03:55.436962475Z level=info msg="Executing migration" id="Update is_service_account column to nullable" 2024-09-18T08:03:55.540 INFO:journalctl@ceph.grafana.a.smithi050.stdout:Sep 18 08:03:55 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-grafana-a[60403]: logger=migrator t=2024-09-18T08:03:55.442575843Z level=info msg="Migration successfully executed" id="Update is_service_account column to nullable" duration=5.612833ms 2024-09-18T08:03:55.540 INFO:journalctl@ceph.grafana.a.smithi050.stdout:Sep 18 08:03:55 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-grafana-a[60403]: logger=migrator t=2024-09-18T08:03:55.442938281Z level=info msg="Executing migration" id="Add uid column to user" 2024-09-18T08:03:55.540 INFO:journalctl@ceph.grafana.a.smithi050.stdout:Sep 18 08:03:55 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-grafana-a[60403]: logger=migrator t=2024-09-18T08:03:55.443680881Z level=info msg="Migration successfully executed" id="Add uid column to user" duration=742.32µs 2024-09-18T08:03:55.540 INFO:journalctl@ceph.grafana.a.smithi050.stdout:Sep 18 08:03:55 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-grafana-a[60403]: logger=migrator t=2024-09-18T08:03:55.444005525Z level=info msg="Executing migration" id="Update uid column values for users" 2024-09-18T08:03:55.540 INFO:journalctl@ceph.grafana.a.smithi050.stdout:Sep 18 08:03:55 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-grafana-a[60403]: logger=migrator t=2024-09-18T08:03:55.444140718Z level=info msg="Migration successfully executed" id="Update uid column values for users" duration=135.333µs 2024-09-18T08:03:55.540 INFO:journalctl@ceph.grafana.a.smithi050.stdout:Sep 18 08:03:55 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-grafana-a[60403]: logger=migrator t=2024-09-18T08:03:55.444495586Z level=info msg="Executing migration" id="Add unique index user_uid" 2024-09-18T08:03:55.540 INFO:journalctl@ceph.grafana.a.smithi050.stdout:Sep 18 08:03:55 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-grafana-a[60403]: logger=migrator t=2024-09-18T08:03:55.444964192Z level=info msg="Migration successfully executed" id="Add unique index user_uid" duration=469.401µs 2024-09-18T08:03:55.540 INFO:journalctl@ceph.grafana.a.smithi050.stdout:Sep 18 08:03:55 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-grafana-a[60403]: logger=migrator t=2024-09-18T08:03:55.445349546Z level=info msg="Executing migration" id="create temp user table v1-7" 2024-09-18T08:03:55.540 INFO:journalctl@ceph.grafana.a.smithi050.stdout:Sep 18 08:03:55 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-grafana-a[60403]: logger=migrator t=2024-09-18T08:03:55.44594108Z level=info msg="Migration successfully executed" id="create temp user table v1-7" duration=591.316µs 2024-09-18T08:03:55.540 INFO:journalctl@ceph.grafana.a.smithi050.stdout:Sep 18 08:03:55 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-grafana-a[60403]: logger=migrator t=2024-09-18T08:03:55.446344306Z level=info msg="Executing migration" id="create index IDX_temp_user_email - v1-7" 2024-09-18T08:03:55.540 INFO:journalctl@ceph.grafana.a.smithi050.stdout:Sep 18 08:03:55 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-grafana-a[60403]: logger=migrator t=2024-09-18T08:03:55.446830139Z level=info msg="Migration successfully executed" id="create index IDX_temp_user_email - v1-7" duration=485.755µs 2024-09-18T08:03:55.541 INFO:journalctl@ceph.grafana.a.smithi050.stdout:Sep 18 08:03:55 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-grafana-a[60403]: logger=migrator t=2024-09-18T08:03:55.447206739Z level=info msg="Executing migration" id="create index IDX_temp_user_org_id - v1-7" 2024-09-18T08:03:55.541 INFO:journalctl@ceph.grafana.a.smithi050.stdout:Sep 18 08:03:55 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-grafana-a[60403]: logger=migrator t=2024-09-18T08:03:55.447732886Z level=info msg="Migration successfully executed" id="create index IDX_temp_user_org_id - v1-7" duration=525.763µs 2024-09-18T08:03:55.541 INFO:journalctl@ceph.grafana.a.smithi050.stdout:Sep 18 08:03:55 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-grafana-a[60403]: logger=migrator t=2024-09-18T08:03:55.448108144Z level=info msg="Executing migration" id="create index IDX_temp_user_code - v1-7" 2024-09-18T08:03:55.541 INFO:journalctl@ceph.grafana.a.smithi050.stdout:Sep 18 08:03:55 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-grafana-a[60403]: logger=migrator t=2024-09-18T08:03:55.448587846Z level=info msg="Migration successfully executed" id="create index IDX_temp_user_code - v1-7" duration=478.327µs 2024-09-18T08:03:55.541 INFO:journalctl@ceph.grafana.a.smithi050.stdout:Sep 18 08:03:55 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-grafana-a[60403]: logger=migrator t=2024-09-18T08:03:55.448953953Z level=info msg="Executing migration" id="create index IDX_temp_user_status - v1-7" 2024-09-18T08:03:55.541 INFO:journalctl@ceph.grafana.a.smithi050.stdout:Sep 18 08:03:55 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-grafana-a[60403]: logger=migrator t=2024-09-18T08:03:55.44943378Z level=info msg="Migration successfully executed" id="create index IDX_temp_user_status - v1-7" duration=479.704µs 2024-09-18T08:03:55.541 INFO:journalctl@ceph.grafana.a.smithi050.stdout:Sep 18 08:03:55 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-grafana-a[60403]: logger=migrator t=2024-09-18T08:03:55.449781266Z level=info msg="Executing migration" id="Update temp_user table charset" 2024-09-18T08:03:55.541 INFO:journalctl@ceph.grafana.a.smithi050.stdout:Sep 18 08:03:55 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-grafana-a[60403]: logger=migrator t=2024-09-18T08:03:55.449799494Z level=info msg="Migration successfully executed" id="Update temp_user table charset" duration=18.728µs 2024-09-18T08:03:55.541 INFO:journalctl@ceph.grafana.a.smithi050.stdout:Sep 18 08:03:55 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-grafana-a[60403]: logger=migrator t=2024-09-18T08:03:55.450187964Z level=info msg="Executing migration" id="drop index IDX_temp_user_email - v1" 2024-09-18T08:03:55.541 INFO:journalctl@ceph.grafana.a.smithi050.stdout:Sep 18 08:03:55 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-grafana-a[60403]: logger=migrator t=2024-09-18T08:03:55.450684945Z level=info msg="Migration successfully executed" id="drop index IDX_temp_user_email - v1" duration=497.427µs 2024-09-18T08:03:55.541 INFO:journalctl@ceph.grafana.a.smithi050.stdout:Sep 18 08:03:55 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-grafana-a[60403]: logger=migrator t=2024-09-18T08:03:55.451025262Z level=info msg="Executing migration" id="drop index IDX_temp_user_org_id - v1" 2024-09-18T08:03:55.541 INFO:journalctl@ceph.grafana.a.smithi050.stdout:Sep 18 08:03:55 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-grafana-a[60403]: logger=migrator t=2024-09-18T08:03:55.45150206Z level=info msg="Migration successfully executed" id="drop index IDX_temp_user_org_id - v1" duration=476.703µs 2024-09-18T08:03:55.541 INFO:journalctl@ceph.grafana.a.smithi050.stdout:Sep 18 08:03:55 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-grafana-a[60403]: logger=migrator t=2024-09-18T08:03:55.451847662Z level=info msg="Executing migration" id="drop index IDX_temp_user_code - v1" 2024-09-18T08:03:55.542 INFO:journalctl@ceph.grafana.a.smithi050.stdout:Sep 18 08:03:55 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-grafana-a[60403]: logger=migrator t=2024-09-18T08:03:55.45235157Z level=info msg="Migration successfully executed" id="drop index IDX_temp_user_code - v1" duration=504.487µs 2024-09-18T08:03:55.542 INFO:journalctl@ceph.grafana.a.smithi050.stdout:Sep 18 08:03:55 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-grafana-a[60403]: logger=migrator t=2024-09-18T08:03:55.452688951Z level=info msg="Executing migration" id="drop index IDX_temp_user_status - v1" 2024-09-18T08:03:55.542 INFO:journalctl@ceph.grafana.a.smithi050.stdout:Sep 18 08:03:55 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-grafana-a[60403]: logger=migrator t=2024-09-18T08:03:55.453174481Z level=info msg="Migration successfully executed" id="drop index IDX_temp_user_status - v1" duration=485.506µs 2024-09-18T08:03:55.542 INFO:journalctl@ceph.grafana.a.smithi050.stdout:Sep 18 08:03:55 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-grafana-a[60403]: logger=migrator t=2024-09-18T08:03:55.453510575Z level=info msg="Executing migration" id="Rename table temp_user to temp_user_tmp_qwerty - v1" 2024-09-18T08:03:55.542 INFO:journalctl@ceph.grafana.a.smithi050.stdout:Sep 18 08:03:55 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-grafana-a[60403]: logger=migrator t=2024-09-18T08:03:55.455678412Z level=info msg="Migration successfully executed" id="Rename table temp_user to temp_user_tmp_qwerty - v1" duration=2.16747ms 2024-09-18T08:03:55.542 INFO:journalctl@ceph.grafana.a.smithi050.stdout:Sep 18 08:03:55 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-grafana-a[60403]: logger=migrator t=2024-09-18T08:03:55.45603193Z level=info msg="Executing migration" id="create temp_user v2" 2024-09-18T08:03:55.542 INFO:journalctl@ceph.grafana.a.smithi050.stdout:Sep 18 08:03:55 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-grafana-a[60403]: logger=migrator t=2024-09-18T08:03:55.456601563Z level=info msg="Migration successfully executed" id="create temp_user v2" duration=569.272µs 2024-09-18T08:03:55.542 INFO:journalctl@ceph.grafana.a.smithi050.stdout:Sep 18 08:03:55 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-grafana-a[60403]: logger=migrator t=2024-09-18T08:03:55.456956753Z level=info msg="Executing migration" id="create index IDX_temp_user_email - v2" 2024-09-18T08:03:55.542 INFO:journalctl@ceph.grafana.a.smithi050.stdout:Sep 18 08:03:55 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-grafana-a[60403]: logger=migrator t=2024-09-18T08:03:55.457446031Z level=info msg="Migration successfully executed" id="create index IDX_temp_user_email - v2" duration=489.007µs 2024-09-18T08:03:55.542 INFO:journalctl@ceph.grafana.a.smithi050.stdout:Sep 18 08:03:55 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-grafana-a[60403]: logger=migrator t=2024-09-18T08:03:55.457781981Z level=info msg="Executing migration" id="create index IDX_temp_user_org_id - v2" 2024-09-18T08:03:55.542 INFO:journalctl@ceph.grafana.a.smithi050.stdout:Sep 18 08:03:55 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-grafana-a[60403]: logger=migrator t=2024-09-18T08:03:55.458280026Z level=info msg="Migration successfully executed" id="create index IDX_temp_user_org_id - v2" duration=499.193µs 2024-09-18T08:03:55.542 INFO:journalctl@ceph.grafana.a.smithi050.stdout:Sep 18 08:03:55 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-grafana-a[60403]: logger=migrator t=2024-09-18T08:03:55.458607588Z level=info msg="Executing migration" id="create index IDX_temp_user_code - v2" 2024-09-18T08:03:55.542 INFO:journalctl@ceph.grafana.a.smithi050.stdout:Sep 18 08:03:55 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-grafana-a[60403]: logger=migrator t=2024-09-18T08:03:55.459103696Z level=info msg="Migration successfully executed" id="create index IDX_temp_user_code - v2" duration=495.87µs 2024-09-18T08:03:55.543 INFO:journalctl@ceph.grafana.a.smithi050.stdout:Sep 18 08:03:55 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-grafana-a[60403]: logger=migrator t=2024-09-18T08:03:55.459430212Z level=info msg="Executing migration" id="create index IDX_temp_user_status - v2" 2024-09-18T08:03:55.543 INFO:journalctl@ceph.grafana.a.smithi050.stdout:Sep 18 08:03:55 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-grafana-a[60403]: logger=migrator t=2024-09-18T08:03:55.459927679Z level=info msg="Migration successfully executed" id="create index IDX_temp_user_status - v2" duration=497.114µs 2024-09-18T08:03:55.543 INFO:journalctl@ceph.grafana.a.smithi050.stdout:Sep 18 08:03:55 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-grafana-a[60403]: logger=migrator t=2024-09-18T08:03:55.460305639Z level=info msg="Executing migration" id="copy temp_user v1 to v2" 2024-09-18T08:03:55.543 INFO:journalctl@ceph.grafana.a.smithi050.stdout:Sep 18 08:03:55 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-grafana-a[60403]: logger=migrator t=2024-09-18T08:03:55.460602014Z level=info msg="Migration successfully executed" id="copy temp_user v1 to v2" duration=296.193µs 2024-09-18T08:03:55.543 INFO:journalctl@ceph.grafana.a.smithi050.stdout:Sep 18 08:03:55 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-grafana-a[60403]: logger=migrator t=2024-09-18T08:03:55.46095334Z level=info msg="Executing migration" id="drop temp_user_tmp_qwerty" 2024-09-18T08:03:55.543 INFO:journalctl@ceph.grafana.a.smithi050.stdout:Sep 18 08:03:55 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-grafana-a[60403]: logger=migrator t=2024-09-18T08:03:55.461349096Z level=info msg="Migration successfully executed" id="drop temp_user_tmp_qwerty" duration=395.668µs 2024-09-18T08:03:55.543 INFO:journalctl@ceph.grafana.a.smithi050.stdout:Sep 18 08:03:55 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-grafana-a[60403]: logger=migrator t=2024-09-18T08:03:55.461674539Z level=info msg="Executing migration" id="Set created for temp users that will otherwise prematurely expire" 2024-09-18T08:03:55.543 INFO:journalctl@ceph.grafana.a.smithi050.stdout:Sep 18 08:03:55 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-grafana-a[60403]: logger=migrator t=2024-09-18T08:03:55.461958031Z level=info msg="Migration successfully executed" id="Set created for temp users that will otherwise prematurely expire" duration=283.513µs 2024-09-18T08:03:55.543 INFO:journalctl@ceph.grafana.a.smithi050.stdout:Sep 18 08:03:55 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-grafana-a[60403]: logger=migrator t=2024-09-18T08:03:55.462343883Z level=info msg="Executing migration" id="create star table" 2024-09-18T08:03:55.543 INFO:journalctl@ceph.grafana.a.smithi050.stdout:Sep 18 08:03:55 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-grafana-a[60403]: logger=migrator t=2024-09-18T08:03:55.462787887Z level=info msg="Migration successfully executed" id="create star table" duration=443.77µs 2024-09-18T08:03:55.543 INFO:journalctl@ceph.grafana.a.smithi050.stdout:Sep 18 08:03:55 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-grafana-a[60403]: logger=migrator t=2024-09-18T08:03:55.463134427Z level=info msg="Executing migration" id="add unique index star.user_id_dashboard_id" 2024-09-18T08:03:55.543 INFO:journalctl@ceph.grafana.a.smithi050.stdout:Sep 18 08:03:55 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-grafana-a[60403]: logger=migrator t=2024-09-18T08:03:55.463633037Z level=info msg="Migration successfully executed" id="add unique index star.user_id_dashboard_id" duration=498.395µs 2024-09-18T08:03:55.544 INFO:journalctl@ceph.grafana.a.smithi050.stdout:Sep 18 08:03:55 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-grafana-a[60403]: logger=migrator t=2024-09-18T08:03:55.464005806Z level=info msg="Executing migration" id="create org table v1" 2024-09-18T08:03:55.544 INFO:journalctl@ceph.grafana.a.smithi050.stdout:Sep 18 08:03:55 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-grafana-a[60403]: logger=migrator t=2024-09-18T08:03:55.464519555Z level=info msg="Migration successfully executed" id="create org table v1" duration=514.408µs 2024-09-18T08:03:55.544 INFO:journalctl@ceph.grafana.a.smithi050.stdout:Sep 18 08:03:55 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-grafana-a[60403]: logger=migrator t=2024-09-18T08:03:55.464858347Z level=info msg="Executing migration" id="create index UQE_org_name - v1" 2024-09-18T08:03:55.544 INFO:journalctl@ceph.grafana.a.smithi050.stdout:Sep 18 08:03:55 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-grafana-a[60403]: logger=migrator t=2024-09-18T08:03:55.465358294Z level=info msg="Migration successfully executed" id="create index UQE_org_name - v1" duration=498.654µs 2024-09-18T08:03:55.544 INFO:journalctl@ceph.grafana.a.smithi050.stdout:Sep 18 08:03:55 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-grafana-a[60403]: logger=migrator t=2024-09-18T08:03:55.465716982Z level=info msg="Executing migration" id="create org_user table v1" 2024-09-18T08:03:55.544 INFO:journalctl@ceph.grafana.a.smithi050.stdout:Sep 18 08:03:55 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-grafana-a[60403]: logger=migrator t=2024-09-18T08:03:55.466198047Z level=info msg="Migration successfully executed" id="create org_user table v1" duration=480.789µs 2024-09-18T08:03:55.544 INFO:journalctl@ceph.grafana.a.smithi050.stdout:Sep 18 08:03:55 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-grafana-a[60403]: logger=migrator t=2024-09-18T08:03:55.466538536Z level=info msg="Executing migration" id="create index IDX_org_user_org_id - v1" 2024-09-18T08:03:55.544 INFO:journalctl@ceph.grafana.a.smithi050.stdout:Sep 18 08:03:55 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-grafana-a[60403]: logger=migrator t=2024-09-18T08:03:55.467064712Z level=info msg="Migration successfully executed" id="create index IDX_org_user_org_id - v1" duration=526.01µs 2024-09-18T08:03:55.544 INFO:journalctl@ceph.grafana.a.smithi050.stdout:Sep 18 08:03:55 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-grafana-a[60403]: logger=migrator t=2024-09-18T08:03:55.467409267Z level=info msg="Executing migration" id="create index UQE_org_user_org_id_user_id - v1" 2024-09-18T08:03:55.544 INFO:journalctl@ceph.grafana.a.smithi050.stdout:Sep 18 08:03:55 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-grafana-a[60403]: logger=migrator t=2024-09-18T08:03:55.467944071Z level=info msg="Migration successfully executed" id="create index UQE_org_user_org_id_user_id - v1" duration=533.465µs 2024-09-18T08:03:55.544 INFO:journalctl@ceph.grafana.a.smithi050.stdout:Sep 18 08:03:55 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-grafana-a[60403]: logger=migrator t=2024-09-18T08:03:55.468283614Z level=info msg="Executing migration" id="create index IDX_org_user_user_id - v1" 2024-09-18T08:03:55.544 INFO:journalctl@ceph.grafana.a.smithi050.stdout:Sep 18 08:03:55 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-grafana-a[60403]: logger=migrator t=2024-09-18T08:03:55.468793767Z level=info msg="Migration successfully executed" id="create index IDX_org_user_user_id - v1" duration=509.986µs 2024-09-18T08:03:55.545 INFO:journalctl@ceph.grafana.a.smithi050.stdout:Sep 18 08:03:55 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-grafana-a[60403]: logger=migrator t=2024-09-18T08:03:55.469186534Z level=info msg="Executing migration" id="Update org table charset" 2024-09-18T08:03:55.545 INFO:journalctl@ceph.grafana.a.smithi050.stdout:Sep 18 08:03:55 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-grafana-a[60403]: logger=migrator t=2024-09-18T08:03:55.46920359Z level=info msg="Migration successfully executed" id="Update org table charset" duration=17.907µs 2024-09-18T08:03:55.545 INFO:journalctl@ceph.grafana.a.smithi050.stdout:Sep 18 08:03:55 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-grafana-a[60403]: logger=migrator t=2024-09-18T08:03:55.469596423Z level=info msg="Executing migration" id="Update org_user table charset" 2024-09-18T08:03:55.545 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7 2024-09-18T08:03:55.547 INFO:journalctl@ceph.grafana.a.smithi050.stdout:Sep 18 08:03:55 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-grafana-a[60403]: logger=migrator t=2024-09-18T08:03:55.469612726Z level=info msg="Migration successfully executed" id="Update org_user table charset" duration=17.019µs 2024-09-18T08:03:55.547 INFO:journalctl@ceph.grafana.a.smithi050.stdout:Sep 18 08:03:55 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-grafana-a[60403]: logger=migrator t=2024-09-18T08:03:55.469950423Z level=info msg="Executing migration" id="Migrate all Read Only Viewers to Viewers" 2024-09-18T08:03:55.547 INFO:journalctl@ceph.grafana.a.smithi050.stdout:Sep 18 08:03:55 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-grafana-a[60403]: logger=migrator t=2024-09-18T08:03:55.470058714Z level=info msg="Migration successfully executed" id="Migrate all Read Only Viewers to Viewers" duration=108.319µs 2024-09-18T08:03:55.547 INFO:journalctl@ceph.grafana.a.smithi050.stdout:Sep 18 08:03:55 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-grafana-a[60403]: logger=migrator t=2024-09-18T08:03:55.470404171Z level=info msg="Executing migration" id="create dashboard table" 2024-09-18T08:03:55.547 INFO:journalctl@ceph.grafana.a.smithi050.stdout:Sep 18 08:03:55 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-grafana-a[60403]: logger=migrator t=2024-09-18T08:03:55.470921551Z level=info msg="Migration successfully executed" id="create dashboard table" duration=517.148µs 2024-09-18T08:03:55.547 INFO:journalctl@ceph.grafana.a.smithi050.stdout:Sep 18 08:03:55 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-grafana-a[60403]: logger=migrator t=2024-09-18T08:03:55.471244447Z level=info msg="Executing migration" id="add index dashboard.account_id" 2024-09-18T08:03:55.547 INFO:journalctl@ceph.grafana.a.smithi050.stdout:Sep 18 08:03:55 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-grafana-a[60403]: logger=migrator t=2024-09-18T08:03:55.471759685Z level=info msg="Migration successfully executed" id="add index dashboard.account_id" duration=514.895µs 2024-09-18T08:03:55.547 INFO:journalctl@ceph.grafana.a.smithi050.stdout:Sep 18 08:03:55 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-grafana-a[60403]: logger=migrator t=2024-09-18T08:03:55.472082645Z level=info msg="Executing migration" id="add unique index dashboard_account_id_slug" 2024-09-18T08:03:55.547 INFO:journalctl@ceph.grafana.a.smithi050.stdout:Sep 18 08:03:55 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-grafana-a[60403]: logger=migrator t=2024-09-18T08:03:55.472633154Z level=info msg="Migration successfully executed" id="add unique index dashboard_account_id_slug" duration=550.16µs 2024-09-18T08:03:55.547 INFO:journalctl@ceph.grafana.a.smithi050.stdout:Sep 18 08:03:55 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-grafana-a[60403]: logger=migrator t=2024-09-18T08:03:55.472966602Z level=info msg="Executing migration" id="create dashboard_tag table" 2024-09-18T08:03:55.547 INFO:journalctl@ceph.grafana.a.smithi050.stdout:Sep 18 08:03:55 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-grafana-a[60403]: logger=migrator t=2024-09-18T08:03:55.473397115Z level=info msg="Migration successfully executed" id="create dashboard_tag table" duration=430.285µs 2024-09-18T08:03:55.548 INFO:journalctl@ceph.grafana.a.smithi050.stdout:Sep 18 08:03:55 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-grafana-a[60403]: logger=migrator t=2024-09-18T08:03:55.473710943Z level=info msg="Executing migration" id="add unique index dashboard_tag.dasboard_id_term" 2024-09-18T08:03:55.548 INFO:journalctl@ceph.grafana.a.smithi050.stdout:Sep 18 08:03:55 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-grafana-a[60403]: logger=migrator t=2024-09-18T08:03:55.474214505Z level=info msg="Migration successfully executed" id="add unique index dashboard_tag.dasboard_id_term" duration=503.326µs 2024-09-18T08:03:55.548 INFO:journalctl@ceph.grafana.a.smithi050.stdout:Sep 18 08:03:55 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-grafana-a[60403]: logger=migrator t=2024-09-18T08:03:55.474523892Z level=info msg="Executing migration" id="drop index UQE_dashboard_tag_dashboard_id_term - v1" 2024-09-18T08:03:55.548 INFO:journalctl@ceph.grafana.a.smithi050.stdout:Sep 18 08:03:55 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-grafana-a[60403]: logger=migrator t=2024-09-18T08:03:55.474995685Z level=info msg="Migration successfully executed" id="drop index UQE_dashboard_tag_dashboard_id_term - v1" duration=471.807µs 2024-09-18T08:03:55.548 INFO:journalctl@ceph.grafana.a.smithi050.stdout:Sep 18 08:03:55 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-grafana-a[60403]: logger=migrator t=2024-09-18T08:03:55.475296634Z level=info msg="Executing migration" id="Rename table dashboard to dashboard_v1 - v1" 2024-09-18T08:03:55.548 INFO:journalctl@ceph.grafana.a.smithi050.stdout:Sep 18 08:03:55 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-grafana-a[60403]: logger=migrator t=2024-09-18T08:03:55.479068758Z level=info msg="Migration successfully executed" id="Rename table dashboard to dashboard_v1 - v1" duration=3.771872ms 2024-09-18T08:03:55.548 INFO:journalctl@ceph.grafana.a.smithi050.stdout:Sep 18 08:03:55 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-grafana-a[60403]: logger=migrator t=2024-09-18T08:03:55.47939764Z level=info msg="Executing migration" id="create dashboard v2" 2024-09-18T08:03:55.548 INFO:journalctl@ceph.grafana.a.smithi050.stdout:Sep 18 08:03:55 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-grafana-a[60403]: logger=migrator t=2024-09-18T08:03:55.479908775Z level=info msg="Migration successfully executed" id="create dashboard v2" duration=510.959µs 2024-09-18T08:03:55.548 INFO:journalctl@ceph.grafana.a.smithi050.stdout:Sep 18 08:03:55 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-grafana-a[60403]: logger=migrator t=2024-09-18T08:03:55.480223444Z level=info msg="Executing migration" id="create index IDX_dashboard_org_id - v2" 2024-09-18T08:03:55.548 INFO:journalctl@ceph.grafana.a.smithi050.stdout:Sep 18 08:03:55 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-grafana-a[60403]: logger=migrator t=2024-09-18T08:03:55.480683212Z level=info msg="Migration successfully executed" id="create index IDX_dashboard_org_id - v2" duration=458.486µs 2024-09-18T08:03:55.548 INFO:journalctl@ceph.grafana.a.smithi050.stdout:Sep 18 08:03:55 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-grafana-a[60403]: logger=migrator t=2024-09-18T08:03:55.481004415Z level=info msg="Executing migration" id="create index UQE_dashboard_org_id_slug - v2" 2024-09-18T08:03:55.548 INFO:journalctl@ceph.grafana.a.smithi050.stdout:Sep 18 08:03:55 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-grafana-a[60403]: logger=migrator t=2024-09-18T08:03:55.481490865Z level=info msg="Migration successfully executed" id="create index UQE_dashboard_org_id_slug - v2" duration=486.134µs 2024-09-18T08:03:55.549 INFO:journalctl@ceph.grafana.a.smithi050.stdout:Sep 18 08:03:55 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-grafana-a[60403]: logger=migrator t=2024-09-18T08:03:55.481812471Z level=info msg="Executing migration" id="copy dashboard v1 to v2" 2024-09-18T08:03:55.549 INFO:journalctl@ceph.grafana.a.smithi050.stdout:Sep 18 08:03:55 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-grafana-a[60403]: logger=migrator t=2024-09-18T08:03:55.48206313Z level=info msg="Migration successfully executed" id="copy dashboard v1 to v2" duration=250.633µs 2024-09-18T08:03:55.549 INFO:journalctl@ceph.grafana.a.smithi050.stdout:Sep 18 08:03:55 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-grafana-a[60403]: logger=migrator t=2024-09-18T08:03:55.482362453Z level=info msg="Executing migration" id="drop table dashboard_v1" 2024-09-18T08:03:55.549 INFO:journalctl@ceph.grafana.a.smithi050.stdout:Sep 18 08:03:55 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-grafana-a[60403]: logger=migrator t=2024-09-18T08:03:55.482983519Z level=info msg="Migration successfully executed" id="drop table dashboard_v1" duration=620.991µs 2024-09-18T08:03:55.549 INFO:journalctl@ceph.grafana.a.smithi050.stdout:Sep 18 08:03:55 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-grafana-a[60403]: logger=migrator t=2024-09-18T08:03:55.483302889Z level=info msg="Executing migration" id="alter dashboard.data to mediumtext v1" 2024-09-18T08:03:55.549 INFO:journalctl@ceph.grafana.a.smithi050.stdout:Sep 18 08:03:55 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-grafana-a[60403]: logger=migrator t=2024-09-18T08:03:55.48334286Z level=info msg="Migration successfully executed" id="alter dashboard.data to mediumtext v1" duration=40.365µs 2024-09-18T08:03:55.549 INFO:journalctl@ceph.grafana.a.smithi050.stdout:Sep 18 08:03:55 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-grafana-a[60403]: logger=migrator t=2024-09-18T08:03:55.483731378Z level=info msg="Executing migration" id="Add column updated_by in dashboard - v2" 2024-09-18T08:03:55.549 INFO:journalctl@ceph.grafana.a.smithi050.stdout:Sep 18 08:03:55 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-grafana-a[60403]: logger=migrator t=2024-09-18T08:03:55.484875932Z level=info msg="Migration successfully executed" id="Add column updated_by in dashboard - v2" duration=1.144945ms 2024-09-18T08:03:55.549 INFO:journalctl@ceph.grafana.a.smithi050.stdout:Sep 18 08:03:55 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-grafana-a[60403]: logger=migrator t=2024-09-18T08:03:55.485207624Z level=info msg="Executing migration" id="Add column created_by in dashboard - v2" 2024-09-18T08:03:55.549 INFO:journalctl@ceph.grafana.a.smithi050.stdout:Sep 18 08:03:55 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-grafana-a[60403]: logger=migrator t=2024-09-18T08:03:55.486343884Z level=info msg="Migration successfully executed" id="Add column created_by in dashboard - v2" duration=1.134817ms 2024-09-18T08:03:55.549 INFO:journalctl@ceph.grafana.a.smithi050.stdout:Sep 18 08:03:55 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-grafana-a[60403]: logger=migrator t=2024-09-18T08:03:55.486663058Z level=info msg="Executing migration" id="Add column gnetId in dashboard" 2024-09-18T08:03:55.549 INFO:journalctl@ceph.grafana.a.smithi050.stdout:Sep 18 08:03:55 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-grafana-a[60403]: logger=migrator t=2024-09-18T08:03:55.487810931Z level=info msg="Migration successfully executed" id="Add column gnetId in dashboard" duration=1.148634ms 2024-09-18T08:03:55.549 INFO:journalctl@ceph.grafana.a.smithi050.stdout:Sep 18 08:03:55 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-grafana-a[60403]: logger=migrator t=2024-09-18T08:03:55.488137453Z level=info msg="Executing migration" id="Add index for gnetId in dashboard" 2024-09-18T08:03:55.550 INFO:journalctl@ceph.grafana.a.smithi050.stdout:Sep 18 08:03:55 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-grafana-a[60403]: logger=migrator t=2024-09-18T08:03:55.488626316Z level=info msg="Migration successfully executed" id="Add index for gnetId in dashboard" duration=488.844µs 2024-09-18T08:03:55.550 INFO:journalctl@ceph.grafana.a.smithi050.stdout:Sep 18 08:03:55 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-grafana-a[60403]: logger=migrator t=2024-09-18T08:03:55.488949331Z level=info msg="Executing migration" id="Add column plugin_id in dashboard" 2024-09-18T08:03:55.550 INFO:journalctl@ceph.grafana.a.smithi050.stdout:Sep 18 08:03:55 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-grafana-a[60403]: logger=migrator t=2024-09-18T08:03:55.490098546Z level=info msg="Migration successfully executed" id="Add column plugin_id in dashboard" duration=1.150011ms 2024-09-18T08:03:55.550 INFO:journalctl@ceph.grafana.a.smithi050.stdout:Sep 18 08:03:55 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-grafana-a[60403]: logger=migrator t=2024-09-18T08:03:55.490417821Z level=info msg="Executing migration" id="Add index for plugin_id in dashboard" 2024-09-18T08:03:55.550 INFO:journalctl@ceph.grafana.a.smithi050.stdout:Sep 18 08:03:55 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-grafana-a[60403]: logger=migrator t=2024-09-18T08:03:55.490946094Z level=info msg="Migration successfully executed" id="Add index for plugin_id in dashboard" duration=527.802µs 2024-09-18T08:03:55.550 INFO:journalctl@ceph.grafana.a.smithi050.stdout:Sep 18 08:03:55 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-grafana-a[60403]: logger=migrator t=2024-09-18T08:03:55.491270352Z level=info msg="Executing migration" id="Add index for dashboard_id in dashboard_tag" 2024-09-18T08:03:55.550 INFO:journalctl@ceph.grafana.a.smithi050.stdout:Sep 18 08:03:55 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-grafana-a[60403]: logger=migrator t=2024-09-18T08:03:55.4917467Z level=info msg="Migration successfully executed" id="Add index for dashboard_id in dashboard_tag" duration=476.197µs 2024-09-18T08:03:55.550 INFO:journalctl@ceph.grafana.a.smithi050.stdout:Sep 18 08:03:55 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-grafana-a[60403]: logger=migrator t=2024-09-18T08:03:55.492078225Z level=info msg="Executing migration" id="Update dashboard table charset" 2024-09-18T08:03:55.550 INFO:journalctl@ceph.grafana.a.smithi050.stdout:Sep 18 08:03:55 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-grafana-a[60403]: logger=migrator t=2024-09-18T08:03:55.492095587Z level=info msg="Migration successfully executed" id="Update dashboard table charset" duration=18.314µs 2024-09-18T08:03:55.550 INFO:journalctl@ceph.grafana.a.smithi050.stdout:Sep 18 08:03:55 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-grafana-a[60403]: logger=migrator t=2024-09-18T08:03:55.492476447Z level=info msg="Executing migration" id="Update dashboard_tag table charset" 2024-09-18T08:03:55.550 INFO:journalctl@ceph.grafana.a.smithi050.stdout:Sep 18 08:03:55 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-grafana-a[60403]: logger=migrator t=2024-09-18T08:03:55.49249194Z level=info msg="Migration successfully executed" id="Update dashboard_tag table charset" duration=16.14µs 2024-09-18T08:03:55.550 INFO:journalctl@ceph.grafana.a.smithi050.stdout:Sep 18 08:03:55 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-grafana-a[60403]: logger=migrator t=2024-09-18T08:03:55.492832674Z level=info msg="Executing migration" id="Add column folder_id in dashboard" 2024-09-18T08:03:55.551 INFO:journalctl@ceph.grafana.a.smithi050.stdout:Sep 18 08:03:55 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-grafana-a[60403]: logger=migrator t=2024-09-18T08:03:55.49407309Z level=info msg="Migration successfully executed" id="Add column folder_id in dashboard" duration=1.240102ms 2024-09-18T08:03:55.551 INFO:journalctl@ceph.grafana.a.smithi050.stdout:Sep 18 08:03:55 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-grafana-a[60403]: logger=migrator t=2024-09-18T08:03:55.494392281Z level=info msg="Executing migration" id="Add column isFolder in dashboard" 2024-09-18T08:03:55.551 INFO:journalctl@ceph.grafana.a.smithi050.stdout:Sep 18 08:03:55 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-grafana-a[60403]: logger=migrator t=2024-09-18T08:03:55.495602925Z level=info msg="Migration successfully executed" id="Add column isFolder in dashboard" duration=1.210873ms 2024-09-18T08:03:55.551 INFO:journalctl@ceph.grafana.a.smithi050.stdout:Sep 18 08:03:55 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-grafana-a[60403]: logger=migrator t=2024-09-18T08:03:55.495927935Z level=info msg="Executing migration" id="Add column has_acl in dashboard" 2024-09-18T08:03:55.551 INFO:journalctl@ceph.grafana.a.smithi050.stdout:Sep 18 08:03:55 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-grafana-a[60403]: logger=migrator t=2024-09-18T08:03:55.497157002Z level=info msg="Migration successfully executed" id="Add column has_acl in dashboard" duration=1.227638ms 2024-09-18T08:03:55.551 INFO:journalctl@ceph.grafana.a.smithi050.stdout:Sep 18 08:03:55 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-grafana-a[60403]: logger=migrator t=2024-09-18T08:03:55.49748033Z level=info msg="Executing migration" id="Add column uid in dashboard" 2024-09-18T08:03:55.551 INFO:journalctl@ceph.grafana.a.smithi050.stdout:Sep 18 08:03:55 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-grafana-a[60403]: logger=migrator t=2024-09-18T08:03:55.498690987Z level=info msg="Migration successfully executed" id="Add column uid in dashboard" duration=1.21054ms 2024-09-18T08:03:55.551 INFO:journalctl@ceph.grafana.a.smithi050.stdout:Sep 18 08:03:55 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-grafana-a[60403]: logger=migrator t=2024-09-18T08:03:55.499036608Z level=info msg="Executing migration" id="Update uid column values in dashboard" 2024-09-18T08:03:55.551 INFO:journalctl@ceph.grafana.a.smithi050.stdout:Sep 18 08:03:55 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-grafana-a[60403]: logger=migrator t=2024-09-18T08:03:55.499183838Z level=info msg="Migration successfully executed" id="Update uid column values in dashboard" duration=147.281µs 2024-09-18T08:03:55.551 INFO:journalctl@ceph.grafana.a.smithi050.stdout:Sep 18 08:03:55 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-grafana-a[60403]: logger=migrator t=2024-09-18T08:03:55.499565714Z level=info msg="Executing migration" id="Add unique index dashboard_org_id_uid" 2024-09-18T08:03:55.551 INFO:journalctl@ceph.grafana.a.smithi050.stdout:Sep 18 08:03:55 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-grafana-a[60403]: logger=migrator t=2024-09-18T08:03:55.500055459Z level=info msg="Migration successfully executed" id="Add unique index dashboard_org_id_uid" duration=489.384µs 2024-09-18T08:03:55.551 INFO:journalctl@ceph.grafana.a.smithi050.stdout:Sep 18 08:03:55 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-grafana-a[60403]: logger=migrator t=2024-09-18T08:03:55.500493057Z level=info msg="Executing migration" id="Remove unique index org_id_slug" 2024-09-18T08:03:55.551 INFO:journalctl@ceph.grafana.a.smithi050.stdout:Sep 18 08:03:55 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-grafana-a[60403]: logger=migrator t=2024-09-18T08:03:55.500977985Z level=info msg="Migration successfully executed" id="Remove unique index org_id_slug" duration=483.829µs 2024-09-18T08:03:55.552 INFO:journalctl@ceph.grafana.a.smithi050.stdout:Sep 18 08:03:55 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-grafana-a[60403]: logger=migrator t=2024-09-18T08:03:55.501289556Z level=info msg="Executing migration" id="Update dashboard title length" 2024-09-18T08:03:55.552 INFO:journalctl@ceph.grafana.a.smithi050.stdout:Sep 18 08:03:55 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-grafana-a[60403]: logger=migrator t=2024-09-18T08:03:55.501306168Z level=info msg="Migration successfully executed" id="Update dashboard title length" duration=17.399µs 2024-09-18T08:03:55.552 INFO:journalctl@ceph.grafana.a.smithi050.stdout:Sep 18 08:03:55 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-grafana-a[60403]: logger=migrator t=2024-09-18T08:03:55.501688506Z level=info msg="Executing migration" id="Add unique index for dashboard_org_id_title_folder_id" 2024-09-18T08:03:55.552 INFO:journalctl@ceph.grafana.a.smithi050.stdout:Sep 18 08:03:55 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-grafana-a[60403]: logger=migrator t=2024-09-18T08:03:55.502185473Z level=info msg="Migration successfully executed" id="Add unique index for dashboard_org_id_title_folder_id" duration=496.528µs 2024-09-18T08:03:55.552 INFO:journalctl@ceph.grafana.a.smithi050.stdout:Sep 18 08:03:55 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-grafana-a[60403]: logger=migrator t=2024-09-18T08:03:55.502502297Z level=info msg="Executing migration" id="create dashboard_provisioning" 2024-09-18T08:03:55.552 INFO:journalctl@ceph.grafana.a.smithi050.stdout:Sep 18 08:03:55 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-grafana-a[60403]: logger=migrator t=2024-09-18T08:03:55.502970851Z level=info msg="Migration successfully executed" id="create dashboard_provisioning" duration=468.31µs 2024-09-18T08:03:55.552 INFO:journalctl@ceph.grafana.a.smithi050.stdout:Sep 18 08:03:55 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-grafana-a[60403]: logger=migrator t=2024-09-18T08:03:55.503297275Z level=info msg="Executing migration" id="Rename table dashboard_provisioning to dashboard_provisioning_tmp_qwerty - v1" 2024-09-18T08:03:55.552 INFO:journalctl@ceph.grafana.a.smithi050.stdout:Sep 18 08:03:55 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-grafana-a[60403]: logger=migrator t=2024-09-18T08:03:55.506559271Z level=info msg="Migration successfully executed" id="Rename table dashboard_provisioning to dashboard_provisioning_tmp_qwerty - v1" duration=3.261646ms 2024-09-18T08:03:55.552 INFO:journalctl@ceph.grafana.a.smithi050.stdout:Sep 18 08:03:55 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-grafana-a[60403]: logger=migrator t=2024-09-18T08:03:55.506888598Z level=info msg="Executing migration" id="create dashboard_provisioning v2" 2024-09-18T08:03:55.552 INFO:journalctl@ceph.grafana.a.smithi050.stdout:Sep 18 08:03:55 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-grafana-a[60403]: logger=migrator t=2024-09-18T08:03:55.507375368Z level=info msg="Migration successfully executed" id="create dashboard_provisioning v2" duration=485.628µs 2024-09-18T08:03:55.552 INFO:journalctl@ceph.grafana.a.smithi050.stdout:Sep 18 08:03:55 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-grafana-a[60403]: logger=migrator t=2024-09-18T08:03:55.507694999Z level=info msg="Executing migration" id="create index IDX_dashboard_provisioning_dashboard_id - v2" 2024-09-18T08:03:55.553 INFO:journalctl@ceph.grafana.a.smithi050.stdout:Sep 18 08:03:55 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-grafana-a[60403]: logger=migrator t=2024-09-18T08:03:55.508191297Z level=info msg="Migration successfully executed" id="create index IDX_dashboard_provisioning_dashboard_id - v2" duration=496.087µs 2024-09-18T08:03:55.553 INFO:journalctl@ceph.grafana.a.smithi050.stdout:Sep 18 08:03:55 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-grafana-a[60403]: logger=migrator t=2024-09-18T08:03:55.508502529Z level=info msg="Executing migration" id="create index IDX_dashboard_provisioning_dashboard_id_name - v2" 2024-09-18T08:03:55.553 INFO:journalctl@ceph.grafana.a.smithi050.stdout:Sep 18 08:03:55 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-grafana-a[60403]: logger=migrator t=2024-09-18T08:03:55.509007559Z level=info msg="Migration successfully executed" id="create index IDX_dashboard_provisioning_dashboard_id_name - v2" duration=503.847µs 2024-09-18T08:03:55.553 INFO:journalctl@ceph.grafana.a.smithi050.stdout:Sep 18 08:03:55 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-grafana-a[60403]: logger=migrator t=2024-09-18T08:03:55.509324377Z level=info msg="Executing migration" id="copy dashboard_provisioning v1 to v2" 2024-09-18T08:03:55.553 INFO:journalctl@ceph.grafana.a.smithi050.stdout:Sep 18 08:03:55 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-grafana-a[60403]: logger=migrator t=2024-09-18T08:03:55.509540572Z level=info msg="Migration successfully executed" id="copy dashboard_provisioning v1 to v2" duration=216.21µs 2024-09-18T08:03:55.553 INFO:journalctl@ceph.grafana.a.smithi050.stdout:Sep 18 08:03:55 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-grafana-a[60403]: logger=migrator t=2024-09-18T08:03:55.509841709Z level=info msg="Executing migration" id="drop dashboard_provisioning_tmp_qwerty" 2024-09-18T08:03:55.553 INFO:journalctl@ceph.grafana.a.smithi050.stdout:Sep 18 08:03:55 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-grafana-a[60403]: logger=migrator t=2024-09-18T08:03:55.510227868Z level=info msg="Migration successfully executed" id="drop dashboard_provisioning_tmp_qwerty" duration=386.022µs 2024-09-18T08:03:55.553 INFO:journalctl@ceph.grafana.a.smithi050.stdout:Sep 18 08:03:55 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-grafana-a[60403]: logger=migrator t=2024-09-18T08:03:55.510530441Z level=info msg="Executing migration" id="Add check_sum column" 2024-09-18T08:03:55.553 INFO:journalctl@ceph.grafana.a.smithi050.stdout:Sep 18 08:03:55 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-grafana-a[60403]: logger=migrator t=2024-09-18T08:03:55.511795061Z level=info msg="Migration successfully executed" id="Add check_sum column" duration=1.264422ms 2024-09-18T08:03:55.553 INFO:journalctl@ceph.grafana.a.smithi050.stdout:Sep 18 08:03:55 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-grafana-a[60403]: logger=migrator t=2024-09-18T08:03:55.512144775Z level=info msg="Executing migration" id="Add index for dashboard_title" 2024-09-18T08:03:55.553 INFO:journalctl@ceph.grafana.a.smithi050.stdout:Sep 18 08:03:55 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-grafana-a[60403]: logger=migrator t=2024-09-18T08:03:55.512613404Z level=info msg="Migration successfully executed" id="Add index for dashboard_title" duration=468.431µs 2024-09-18T08:03:55.553 INFO:journalctl@ceph.grafana.a.smithi050.stdout:Sep 18 08:03:55 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-grafana-a[60403]: logger=migrator t=2024-09-18T08:03:55.512922994Z level=info msg="Executing migration" id="delete tags for deleted dashboards" 2024-09-18T08:03:55.554 INFO:journalctl@ceph.grafana.a.smithi050.stdout:Sep 18 08:03:55 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-grafana-a[60403]: logger=migrator t=2024-09-18T08:03:55.513035728Z level=info msg="Migration successfully executed" id="delete tags for deleted dashboards" duration=111.728µs 2024-09-18T08:03:55.554 INFO:journalctl@ceph.grafana.a.smithi050.stdout:Sep 18 08:03:55 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-grafana-a[60403]: logger=migrator t=2024-09-18T08:03:55.513379924Z level=info msg="Executing migration" id="delete stars for deleted dashboards" 2024-09-18T08:03:55.554 INFO:journalctl@ceph.grafana.a.smithi050.stdout:Sep 18 08:03:55 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-grafana-a[60403]: logger=migrator t=2024-09-18T08:03:55.513483921Z level=info msg="Migration successfully executed" id="delete stars for deleted dashboards" duration=104.76µs 2024-09-18T08:03:55.554 INFO:journalctl@ceph.grafana.a.smithi050.stdout:Sep 18 08:03:55 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-grafana-a[60403]: logger=migrator t=2024-09-18T08:03:55.513797924Z level=info msg="Executing migration" id="Add index for dashboard_is_folder" 2024-09-18T08:03:55.554 INFO:journalctl@ceph.grafana.a.smithi050.stdout:Sep 18 08:03:55 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-grafana-a[60403]: logger=migrator t=2024-09-18T08:03:55.514282598Z level=info msg="Migration successfully executed" id="Add index for dashboard_is_folder" duration=484.464µs 2024-09-18T08:03:55.554 INFO:journalctl@ceph.grafana.a.smithi050.stdout:Sep 18 08:03:55 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-grafana-a[60403]: logger=migrator t=2024-09-18T08:03:55.514633264Z level=info msg="Executing migration" id="Add isPublic for dashboard" 2024-09-18T08:03:55.554 INFO:journalctl@ceph.grafana.a.smithi050.stdout:Sep 18 08:03:55 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-grafana-a[60403]: logger=migrator t=2024-09-18T08:03:55.515979671Z level=info msg="Migration successfully executed" id="Add isPublic for dashboard" duration=1.346159ms 2024-09-18T08:03:55.554 INFO:journalctl@ceph.grafana.a.smithi050.stdout:Sep 18 08:03:55 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-grafana-a[60403]: logger=migrator t=2024-09-18T08:03:55.516309134Z level=info msg="Executing migration" id="create data_source table" 2024-09-18T08:03:55.554 INFO:journalctl@ceph.grafana.a.smithi050.stdout:Sep 18 08:03:55 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-grafana-a[60403]: logger=migrator t=2024-09-18T08:03:55.516936102Z level=info msg="Migration successfully executed" id="create data_source table" duration=626.799µs 2024-09-18T08:03:55.554 INFO:journalctl@ceph.grafana.a.smithi050.stdout:Sep 18 08:03:55 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-grafana-a[60403]: logger=migrator t=2024-09-18T08:03:55.517279311Z level=info msg="Executing migration" id="add index data_source.account_id" 2024-09-18T08:03:55.554 INFO:journalctl@ceph.grafana.a.smithi050.stdout:Sep 18 08:03:55 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-grafana-a[60403]: logger=migrator t=2024-09-18T08:03:55.517768582Z level=info msg="Migration successfully executed" id="add index data_source.account_id" duration=489.069µs 2024-09-18T08:03:55.554 INFO:journalctl@ceph.grafana.a.smithi050.stdout:Sep 18 08:03:55 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-grafana-a[60403]: logger=migrator t=2024-09-18T08:03:55.518095237Z level=info msg="Executing migration" id="add unique index data_source.account_id_name" 2024-09-18T08:03:55.555 INFO:journalctl@ceph.grafana.a.smithi050.stdout:Sep 18 08:03:55 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-grafana-a[60403]: logger=migrator t=2024-09-18T08:03:55.518586262Z level=info msg="Migration successfully executed" id="add unique index data_source.account_id_name" duration=490.749µs 2024-09-18T08:03:55.555 INFO:journalctl@ceph.grafana.a.smithi050.stdout:Sep 18 08:03:55 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-grafana-a[60403]: logger=migrator t=2024-09-18T08:03:55.5189093Z level=info msg="Executing migration" id="drop index IDX_data_source_account_id - v1" 2024-09-18T08:03:55.555 INFO:journalctl@ceph.grafana.a.smithi050.stdout:Sep 18 08:03:55 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-grafana-a[60403]: logger=migrator t=2024-09-18T08:03:55.519379236Z level=info msg="Migration successfully executed" id="drop index IDX_data_source_account_id - v1" duration=470.564µs 2024-09-18T08:03:55.555 INFO:journalctl@ceph.grafana.a.smithi050.stdout:Sep 18 08:03:55 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-grafana-a[60403]: logger=migrator t=2024-09-18T08:03:55.519705232Z level=info msg="Executing migration" id="drop index UQE_data_source_account_id_name - v1" 2024-09-18T08:03:55.555 INFO:journalctl@ceph.grafana.a.smithi050.stdout:Sep 18 08:03:55 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-grafana-a[60403]: logger=migrator t=2024-09-18T08:03:55.520184559Z level=info msg="Migration successfully executed" id="drop index UQE_data_source_account_id_name - v1" duration=479.059µs 2024-09-18T08:03:55.555 INFO:journalctl@ceph.grafana.a.smithi050.stdout:Sep 18 08:03:55 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-grafana-a[60403]: logger=migrator t=2024-09-18T08:03:55.52048823Z level=info msg="Executing migration" id="Rename table data_source to data_source_v1 - v1" 2024-09-18T08:03:55.555 INFO:journalctl@ceph.grafana.a.smithi050.stdout:Sep 18 08:03:55 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-grafana-a[60403]: logger=migrator t=2024-09-18T08:03:55.524118249Z level=info msg="Migration successfully executed" id="Rename table data_source to data_source_v1 - v1" duration=3.629611ms 2024-09-18T08:03:55.555 INFO:journalctl@ceph.grafana.a.smithi050.stdout:Sep 18 08:03:55 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-grafana-a[60403]: logger=migrator t=2024-09-18T08:03:55.524470002Z level=info msg="Executing migration" id="create data_source table v2" 2024-09-18T08:03:55.555 INFO:journalctl@ceph.grafana.a.smithi050.stdout:Sep 18 08:03:55 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-grafana-a[60403]: logger=migrator t=2024-09-18T08:03:55.525083227Z level=info msg="Migration successfully executed" id="create data_source table v2" duration=613.176µs 2024-09-18T08:03:55.555 INFO:journalctl@ceph.grafana.a.smithi050.stdout:Sep 18 08:03:55 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-grafana-a[60403]: logger=migrator t=2024-09-18T08:03:55.52540612Z level=info msg="Executing migration" id="create index IDX_data_source_org_id - v2" 2024-09-18T08:03:55.555 INFO:journalctl@ceph.grafana.a.smithi050.stdout:Sep 18 08:03:55 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-grafana-a[60403]: logger=migrator t=2024-09-18T08:03:55.525933544Z level=info msg="Migration successfully executed" id="create index IDX_data_source_org_id - v2" duration=526.118µs 2024-09-18T08:03:55.556 INFO:journalctl@ceph.grafana.a.smithi050.stdout:Sep 18 08:03:55 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-grafana-a[60403]: logger=migrator t=2024-09-18T08:03:55.5262523Z level=info msg="Executing migration" id="create index UQE_data_source_org_id_name - v2" 2024-09-18T08:03:55.556 INFO:journalctl@ceph.grafana.a.smithi050.stdout:Sep 18 08:03:55 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-grafana-a[60403]: logger=migrator t=2024-09-18T08:03:55.526775865Z level=info msg="Migration successfully executed" id="create index UQE_data_source_org_id_name - v2" duration=522.314µs 2024-09-18T08:03:55.556 INFO:journalctl@ceph.grafana.a.smithi050.stdout:Sep 18 08:03:55 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-grafana-a[60403]: logger=migrator t=2024-09-18T08:03:55.527108872Z level=info msg="Executing migration" id="Drop old table data_source_v1 #2" 2024-09-18T08:03:55.556 INFO:journalctl@ceph.grafana.a.smithi050.stdout:Sep 18 08:03:55 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-grafana-a[60403]: logger=migrator t=2024-09-18T08:03:55.5274898Z level=info msg="Migration successfully executed" id="Drop old table data_source_v1 #2" duration=380.861µs 2024-09-18T08:03:55.556 INFO:journalctl@ceph.grafana.a.smithi050.stdout:Sep 18 08:03:55 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-grafana-a[60403]: logger=migrator t=2024-09-18T08:03:55.52781918Z level=info msg="Executing migration" id="Add column with_credentials" 2024-09-18T08:03:55.556 INFO:journalctl@ceph.grafana.a.smithi050.stdout:Sep 18 08:03:55 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-grafana-a[60403]: logger=migrator t=2024-09-18T08:03:55.529285473Z level=info msg="Migration successfully executed" id="Add column with_credentials" duration=1.466831ms 2024-09-18T08:03:55.556 INFO:journalctl@ceph.grafana.a.smithi050.stdout:Sep 18 08:03:55 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-grafana-a[60403]: logger=migrator t=2024-09-18T08:03:55.529630527Z level=info msg="Executing migration" id="Add secure json data column" 2024-09-18T08:03:55.556 INFO:journalctl@ceph.grafana.a.smithi050.stdout:Sep 18 08:03:55 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-grafana-a[60403]: logger=migrator t=2024-09-18T08:03:55.531138489Z level=info msg="Migration successfully executed" id="Add secure json data column" duration=1.507745ms 2024-09-18T08:03:55.556 INFO:journalctl@ceph.grafana.a.smithi050.stdout:Sep 18 08:03:55 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-grafana-a[60403]: logger=migrator t=2024-09-18T08:03:55.53146682Z level=info msg="Executing migration" id="Update data_source table charset" 2024-09-18T08:03:55.556 INFO:journalctl@ceph.grafana.a.smithi050.stdout:Sep 18 08:03:55 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-grafana-a[60403]: logger=migrator t=2024-09-18T08:03:55.531482854Z level=info msg="Migration successfully executed" id="Update data_source table charset" duration=16.768µs 2024-09-18T08:03:55.556 INFO:journalctl@ceph.grafana.a.smithi050.stdout:Sep 18 08:03:55 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-grafana-a[60403]: logger=migrator t=2024-09-18T08:03:55.531878167Z level=info msg="Executing migration" id="Update initial version to 1" 2024-09-18T08:03:55.556 INFO:journalctl@ceph.grafana.a.smithi050.stdout:Sep 18 08:03:55 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-grafana-a[60403]: logger=migrator t=2024-09-18T08:03:55.532022622Z level=info msg="Migration successfully executed" id="Update initial version to 1" duration=144.486µs 2024-09-18T08:03:55.557 INFO:journalctl@ceph.grafana.a.smithi050.stdout:Sep 18 08:03:55 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-grafana-a[60403]: logger=migrator t=2024-09-18T08:03:55.532395733Z level=info msg="Executing migration" id="Add read_only data column" 2024-09-18T08:03:55.557 INFO:journalctl@ceph.grafana.a.smithi050.stdout:Sep 18 08:03:55 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-grafana-a[60403]: logger=migrator t=2024-09-18T08:03:55.533938836Z level=info msg="Migration successfully executed" id="Add read_only data column" duration=1.542663ms 2024-09-18T08:03:55.557 INFO:journalctl@ceph.grafana.a.smithi050.stdout:Sep 18 08:03:55 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-grafana-a[60403]: logger=migrator t=2024-09-18T08:03:55.534265055Z level=info msg="Executing migration" id="Migrate logging ds to loki ds" 2024-09-18T08:03:55.557 INFO:journalctl@ceph.grafana.a.smithi050.stdout:Sep 18 08:03:55 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-grafana-a[60403]: logger=migrator t=2024-09-18T08:03:55.534400491Z level=info msg="Migration successfully executed" id="Migrate logging ds to loki ds" duration=135.584µs 2024-09-18T08:03:55.557 INFO:journalctl@ceph.grafana.a.smithi050.stdout:Sep 18 08:03:55 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-grafana-a[60403]: logger=migrator t=2024-09-18T08:03:55.534770542Z level=info msg="Executing migration" id="Update json_data with nulls" 2024-09-18T08:03:55.557 INFO:journalctl@ceph.grafana.a.smithi050.stdout:Sep 18 08:03:55 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-grafana-a[60403]: logger=migrator t=2024-09-18T08:03:55.534891038Z level=info msg="Migration successfully executed" id="Update json_data with nulls" duration=120.422µs 2024-09-18T08:03:55.557 INFO:journalctl@ceph.grafana.a.smithi050.stdout:Sep 18 08:03:55 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-grafana-a[60403]: logger=migrator t=2024-09-18T08:03:55.535280824Z level=info msg="Executing migration" id="Add uid column" 2024-09-18T08:03:55.557 INFO:journalctl@ceph.grafana.a.smithi050.stdout:Sep 18 08:03:55 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-grafana-a[60403]: logger=migrator t=2024-09-18T08:03:55.536779864Z level=info msg="Migration successfully executed" id="Add uid column" duration=1.498703ms 2024-09-18T08:03:55.557 INFO:journalctl@ceph.grafana.a.smithi050.stdout:Sep 18 08:03:55 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-grafana-a[60403]: logger=migrator t=2024-09-18T08:03:55.537145678Z level=info msg="Executing migration" id="Update uid value" 2024-09-18T08:03:55.579 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:03:55.579 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:03:55.580 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7' 2024-09-18T08:03:55.681 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7 2024-09-18T08:03:55.703 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:03:55.703 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:03:55.704 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7' 2024-09-18T08:03:55.705 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/779349ca-7593-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T08:03:55.781 INFO:journalctl@ceph.grafana.a.smithi050.stdout:Sep 18 08:03:55 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-grafana-a[60403]: logger=migrator t=2024-09-18T08:03:55.537280102Z level=info msg="Migration successfully executed" id="Update uid value" duration=134.367µs 2024-09-18T08:03:55.781 INFO:journalctl@ceph.grafana.a.smithi050.stdout:Sep 18 08:03:55 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-grafana-a[60403]: logger=migrator t=2024-09-18T08:03:55.537676084Z level=info msg="Executing migration" id="Add unique index datasource_org_id_uid" 2024-09-18T08:03:55.781 INFO:journalctl@ceph.grafana.a.smithi050.stdout:Sep 18 08:03:55 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-grafana-a[60403]: logger=migrator t=2024-09-18T08:03:55.538229829Z level=info msg="Migration successfully executed" id="Add unique index datasource_org_id_uid" duration=553.541µs 2024-09-18T08:03:55.782 INFO:journalctl@ceph.grafana.a.smithi050.stdout:Sep 18 08:03:55 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-grafana-a[60403]: logger=migrator t=2024-09-18T08:03:55.538605727Z level=info msg="Executing migration" id="add unique index datasource_org_id_is_default" 2024-09-18T08:03:55.782 INFO:journalctl@ceph.grafana.a.smithi050.stdout:Sep 18 08:03:55 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-grafana-a[60403]: logger=migrator t=2024-09-18T08:03:55.539137404Z level=info msg="Migration successfully executed" id="add unique index datasource_org_id_is_default" duration=529.969µs 2024-09-18T08:03:55.782 INFO:journalctl@ceph.grafana.a.smithi050.stdout:Sep 18 08:03:55 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-grafana-a[60403]: logger=migrator t=2024-09-18T08:03:55.539465456Z level=info msg="Executing migration" id="create api_key table" 2024-09-18T08:03:55.782 INFO:journalctl@ceph.grafana.a.smithi050.stdout:Sep 18 08:03:55 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-grafana-a[60403]: logger=migrator t=2024-09-18T08:03:55.540020597Z level=info msg="Migration successfully executed" id="create api_key table" duration=554.912µs 2024-09-18T08:03:55.782 INFO:journalctl@ceph.grafana.a.smithi050.stdout:Sep 18 08:03:55 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-grafana-a[60403]: logger=migrator t=2024-09-18T08:03:55.540379395Z level=info msg="Executing migration" id="add index api_key.account_id" 2024-09-18T08:03:55.782 INFO:journalctl@ceph.grafana.a.smithi050.stdout:Sep 18 08:03:55 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-grafana-a[60403]: logger=migrator t=2024-09-18T08:03:55.540930686Z level=info msg="Migration successfully executed" id="add index api_key.account_id" duration=552.301µs 2024-09-18T08:03:55.782 INFO:journalctl@ceph.grafana.a.smithi050.stdout:Sep 18 08:03:55 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-grafana-a[60403]: logger=migrator t=2024-09-18T08:03:55.541285565Z level=info msg="Executing migration" id="add index api_key.key" 2024-09-18T08:03:55.782 INFO:journalctl@ceph.grafana.a.smithi050.stdout:Sep 18 08:03:55 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-grafana-a[60403]: logger=migrator t=2024-09-18T08:03:55.541785037Z level=info msg="Migration successfully executed" id="add index api_key.key" duration=499.402µs 2024-09-18T08:03:55.782 INFO:journalctl@ceph.grafana.a.smithi050.stdout:Sep 18 08:03:55 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-grafana-a[60403]: logger=migrator t=2024-09-18T08:03:55.542146113Z level=info msg="Executing migration" id="add index api_key.account_id_name" 2024-09-18T08:03:55.782 INFO:journalctl@ceph.grafana.a.smithi050.stdout:Sep 18 08:03:55 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-grafana-a[60403]: logger=migrator t=2024-09-18T08:03:55.54268319Z level=info msg="Migration successfully executed" id="add index api_key.account_id_name" duration=538.386µs 2024-09-18T08:03:55.782 INFO:journalctl@ceph.grafana.a.smithi050.stdout:Sep 18 08:03:55 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-grafana-a[60403]: logger=migrator t=2024-09-18T08:03:55.543029749Z level=info msg="Executing migration" id="drop index IDX_api_key_account_id - v1" 2024-09-18T08:03:55.783 INFO:journalctl@ceph.grafana.a.smithi050.stdout:Sep 18 08:03:55 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-grafana-a[60403]: logger=migrator t=2024-09-18T08:03:55.543528608Z level=info msg="Migration successfully executed" id="drop index IDX_api_key_account_id - v1" duration=498.721µs 2024-09-18T08:03:55.783 INFO:journalctl@ceph.grafana.a.smithi050.stdout:Sep 18 08:03:55 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-grafana-a[60403]: logger=migrator t=2024-09-18T08:03:55.543833594Z level=info msg="Executing migration" id="drop index UQE_api_key_key - v1" 2024-09-18T08:03:55.783 INFO:journalctl@ceph.grafana.a.smithi050.stdout:Sep 18 08:03:55 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-grafana-a[60403]: logger=migrator t=2024-09-18T08:03:55.544331932Z level=info msg="Migration successfully executed" id="drop index UQE_api_key_key - v1" duration=498.397µs 2024-09-18T08:03:55.783 INFO:journalctl@ceph.grafana.a.smithi050.stdout:Sep 18 08:03:55 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-grafana-a[60403]: logger=migrator t=2024-09-18T08:03:55.544641105Z level=info msg="Executing migration" id="drop index UQE_api_key_account_id_name - v1" 2024-09-18T08:03:55.784 INFO:journalctl@ceph.grafana.a.smithi050.stdout:Sep 18 08:03:55 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-grafana-a[60403]: logger=migrator t=2024-09-18T08:03:55.545136698Z level=info msg="Migration successfully executed" id="drop index UQE_api_key_account_id_name - v1" duration=495.63µs 2024-09-18T08:03:55.784 INFO:journalctl@ceph.grafana.a.smithi050.stdout:Sep 18 08:03:55 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-grafana-a[60403]: logger=migrator t=2024-09-18T08:03:55.545447974Z level=info msg="Executing migration" id="Rename table api_key to api_key_v1 - v1" 2024-09-18T08:03:55.784 INFO:journalctl@ceph.grafana.a.smithi050.stdout:Sep 18 08:03:55 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-grafana-a[60403]: logger=migrator t=2024-09-18T08:03:55.549465863Z level=info msg="Migration successfully executed" id="Rename table api_key to api_key_v1 - v1" duration=4.017286ms 2024-09-18T08:03:55.784 INFO:journalctl@ceph.grafana.a.smithi050.stdout:Sep 18 08:03:55 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-grafana-a[60403]: logger=migrator t=2024-09-18T08:03:55.54979941Z level=info msg="Executing migration" id="create api_key table v2" 2024-09-18T08:03:55.784 INFO:journalctl@ceph.grafana.a.smithi050.stdout:Sep 18 08:03:55 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-grafana-a[60403]: logger=migrator t=2024-09-18T08:03:55.550315306Z level=info msg="Migration successfully executed" id="create api_key table v2" duration=515.835µs 2024-09-18T08:03:55.784 INFO:journalctl@ceph.grafana.a.smithi050.stdout:Sep 18 08:03:55 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-grafana-a[60403]: logger=migrator t=2024-09-18T08:03:55.55062725Z level=info msg="Executing migration" id="create index IDX_api_key_org_id - v2" 2024-09-18T08:03:55.784 INFO:journalctl@ceph.grafana.a.smithi050.stdout:Sep 18 08:03:55 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-grafana-a[60403]: logger=migrator t=2024-09-18T08:03:55.551151426Z level=info msg="Migration successfully executed" id="create index IDX_api_key_org_id - v2" duration=523.896µs 2024-09-18T08:03:55.784 INFO:journalctl@ceph.grafana.a.smithi050.stdout:Sep 18 08:03:55 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-grafana-a[60403]: logger=migrator t=2024-09-18T08:03:55.551468662Z level=info msg="Executing migration" id="create index UQE_api_key_key - v2" 2024-09-18T08:03:55.784 INFO:journalctl@ceph.grafana.a.smithi050.stdout:Sep 18 08:03:55 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-grafana-a[60403]: logger=migrator t=2024-09-18T08:03:55.552000213Z level=info msg="Migration successfully executed" id="create index UQE_api_key_key - v2" duration=531.327µs 2024-09-18T08:03:55.784 INFO:journalctl@ceph.grafana.a.smithi050.stdout:Sep 18 08:03:55 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-grafana-a[60403]: logger=migrator t=2024-09-18T08:03:55.552328766Z level=info msg="Executing migration" id="create index UQE_api_key_org_id_name - v2" 2024-09-18T08:03:55.785 INFO:journalctl@ceph.grafana.a.smithi050.stdout:Sep 18 08:03:55 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-grafana-a[60403]: logger=migrator t=2024-09-18T08:03:55.552981564Z level=info msg="Migration successfully executed" id="create index UQE_api_key_org_id_name - v2" duration=653.546µs 2024-09-18T08:03:55.785 INFO:journalctl@ceph.grafana.a.smithi050.stdout:Sep 18 08:03:55 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-grafana-a[60403]: logger=migrator t=2024-09-18T08:03:55.553399159Z level=info msg="Executing migration" id="copy api_key v1 to v2" 2024-09-18T08:03:55.785 INFO:journalctl@ceph.grafana.a.smithi050.stdout:Sep 18 08:03:55 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-grafana-a[60403]: logger=migrator t=2024-09-18T08:03:55.553685838Z level=info msg="Migration successfully executed" id="copy api_key v1 to v2" duration=285.897µs 2024-09-18T08:03:55.785 INFO:journalctl@ceph.grafana.a.smithi050.stdout:Sep 18 08:03:55 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-grafana-a[60403]: logger=migrator t=2024-09-18T08:03:55.554083316Z level=info msg="Executing migration" id="Drop old table api_key_v1" 2024-09-18T08:03:55.785 INFO:journalctl@ceph.grafana.a.smithi050.stdout:Sep 18 08:03:55 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-grafana-a[60403]: logger=migrator t=2024-09-18T08:03:55.554575425Z level=info msg="Migration successfully executed" id="Drop old table api_key_v1" duration=490.289µs 2024-09-18T08:03:55.785 INFO:journalctl@ceph.grafana.a.smithi050.stdout:Sep 18 08:03:55 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-grafana-a[60403]: logger=migrator t=2024-09-18T08:03:55.55498089Z level=info msg="Executing migration" id="Update api_key table charset" 2024-09-18T08:03:55.785 INFO:journalctl@ceph.grafana.a.smithi050.stdout:Sep 18 08:03:55 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-grafana-a[60403]: logger=migrator t=2024-09-18T08:03:55.555003189Z level=info msg="Migration successfully executed" id="Update api_key table charset" duration=22.428µs 2024-09-18T08:03:55.785 INFO:journalctl@ceph.grafana.a.smithi050.stdout:Sep 18 08:03:55 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-grafana-a[60403]: logger=migrator t=2024-09-18T08:03:55.555486148Z level=info msg="Executing migration" id="Add expires to api_key table" 2024-09-18T08:03:55.785 INFO:journalctl@ceph.grafana.a.smithi050.stdout:Sep 18 08:03:55 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-grafana-a[60403]: logger=migrator t=2024-09-18T08:03:55.557710529Z level=info msg="Migration successfully executed" id="Add expires to api_key table" duration=2.224627ms 2024-09-18T08:03:55.786 INFO:journalctl@ceph.grafana.a.smithi050.stdout:Sep 18 08:03:55 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-grafana-a[60403]: logger=migrator t=2024-09-18T08:03:55.558097686Z level=info msg="Executing migration" id="Add service account foreign key" 2024-09-18T08:03:55.786 INFO:journalctl@ceph.grafana.a.smithi050.stdout:Sep 18 08:03:55 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-grafana-a[60403]: logger=migrator t=2024-09-18T08:03:55.560189427Z level=info msg="Migration successfully executed" id="Add service account foreign key" duration=2.09163ms 2024-09-18T08:03:55.786 INFO:journalctl@ceph.grafana.a.smithi050.stdout:Sep 18 08:03:55 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-grafana-a[60403]: logger=migrator t=2024-09-18T08:03:55.560522032Z level=info msg="Executing migration" id="set service account foreign key to nil if 0" 2024-09-18T08:03:55.786 INFO:journalctl@ceph.grafana.a.smithi050.stdout:Sep 18 08:03:55 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-grafana-a[60403]: logger=migrator t=2024-09-18T08:03:55.560643109Z level=info msg="Migration successfully executed" id="set service account foreign key to nil if 0" duration=121.68µs 2024-09-18T08:03:55.786 INFO:journalctl@ceph.grafana.a.smithi050.stdout:Sep 18 08:03:55 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-grafana-a[60403]: logger=migrator t=2024-09-18T08:03:55.561019946Z level=info msg="Executing migration" id="Add last_used_at to api_key table" 2024-09-18T08:03:55.786 INFO:journalctl@ceph.grafana.a.smithi050.stdout:Sep 18 08:03:55 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-grafana-a[60403]: logger=migrator t=2024-09-18T08:03:55.562652269Z level=info msg="Migration successfully executed" id="Add last_used_at to api_key table" duration=1.632693ms 2024-09-18T08:03:55.786 INFO:journalctl@ceph.grafana.a.smithi050.stdout:Sep 18 08:03:55 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-grafana-a[60403]: logger=migrator t=2024-09-18T08:03:55.563003597Z level=info msg="Executing migration" id="Add is_revoked column to api_key table" 2024-09-18T08:03:55.786 INFO:journalctl@ceph.grafana.a.smithi050.stdout:Sep 18 08:03:55 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-grafana-a[60403]: logger=migrator t=2024-09-18T08:03:55.564603763Z level=info msg="Migration successfully executed" id="Add is_revoked column to api_key table" duration=1.600445ms 2024-09-18T08:03:55.787 INFO:journalctl@ceph.grafana.a.smithi050.stdout:Sep 18 08:03:55 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-grafana-a[60403]: logger=migrator t=2024-09-18T08:03:55.564940001Z level=info msg="Executing migration" id="create dashboard_snapshot table v4" 2024-09-18T08:03:55.787 INFO:journalctl@ceph.grafana.a.smithi050.stdout:Sep 18 08:03:55 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-grafana-a[60403]: logger=migrator t=2024-09-18T08:03:55.565475977Z level=info msg="Migration successfully executed" id="create dashboard_snapshot table v4" duration=534.869µs 2024-09-18T08:03:55.787 INFO:journalctl@ceph.grafana.a.smithi050.stdout:Sep 18 08:03:55 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-grafana-a[60403]: logger=migrator t=2024-09-18T08:03:55.565795971Z level=info msg="Executing migration" id="drop table dashboard_snapshot_v4 #1" 2024-09-18T08:03:55.787 INFO:journalctl@ceph.grafana.a.smithi050.stdout:Sep 18 08:03:55 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-grafana-a[60403]: logger=migrator t=2024-09-18T08:03:55.566216132Z level=info msg="Migration successfully executed" id="drop table dashboard_snapshot_v4 #1" duration=419.036µs 2024-09-18T08:03:55.787 INFO:journalctl@ceph.grafana.a.smithi050.stdout:Sep 18 08:03:55 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-grafana-a[60403]: logger=migrator t=2024-09-18T08:03:55.566540609Z level=info msg="Executing migration" id="create dashboard_snapshot table v5 #2" 2024-09-18T08:03:55.787 INFO:journalctl@ceph.grafana.a.smithi050.stdout:Sep 18 08:03:55 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-grafana-a[60403]: logger=migrator t=2024-09-18T08:03:55.567213094Z level=info msg="Migration successfully executed" id="create dashboard_snapshot table v5 #2" duration=673.925µs 2024-09-18T08:03:55.787 INFO:journalctl@ceph.grafana.a.smithi050.stdout:Sep 18 08:03:55 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-grafana-a[60403]: logger=migrator t=2024-09-18T08:03:55.567536248Z level=info msg="Executing migration" id="create index UQE_dashboard_snapshot_key - v5" 2024-09-18T08:03:55.787 INFO:journalctl@ceph.grafana.a.smithi050.stdout:Sep 18 08:03:55 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-grafana-a[60403]: logger=migrator t=2024-09-18T08:03:55.568117767Z level=info msg="Migration successfully executed" id="create index UQE_dashboard_snapshot_key - v5" duration=581.753µs 2024-09-18T08:03:55.787 INFO:journalctl@ceph.grafana.a.smithi050.stdout:Sep 18 08:03:55 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-grafana-a[60403]: logger=migrator t=2024-09-18T08:03:55.568447579Z level=info msg="Executing migration" id="create index UQE_dashboard_snapshot_delete_key - v5" 2024-09-18T08:03:55.787 INFO:journalctl@ceph.grafana.a.smithi050.stdout:Sep 18 08:03:55 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-grafana-a[60403]: logger=migrator t=2024-09-18T08:03:55.569035915Z level=info msg="Migration successfully executed" id="create index UQE_dashboard_snapshot_delete_key - v5" duration=588.003µs 2024-09-18T08:03:55.787 INFO:journalctl@ceph.grafana.a.smithi050.stdout:Sep 18 08:03:55 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-grafana-a[60403]: logger=migrator t=2024-09-18T08:03:55.569364077Z level=info msg="Executing migration" id="create index IDX_dashboard_snapshot_user_id - v5" 2024-09-18T08:03:55.787 INFO:journalctl@ceph.grafana.a.smithi050.stdout:Sep 18 08:03:55 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-grafana-a[60403]: logger=migrator t=2024-09-18T08:03:55.569925584Z level=info msg="Migration successfully executed" id="create index IDX_dashboard_snapshot_user_id - v5" duration=561.455µs 2024-09-18T08:03:55.788 INFO:journalctl@ceph.grafana.a.smithi050.stdout:Sep 18 08:03:55 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-grafana-a[60403]: logger=migrator t=2024-09-18T08:03:55.570254594Z level=info msg="Executing migration" id="alter dashboard_snapshot to mediumtext v2" 2024-09-18T08:03:55.788 INFO:journalctl@ceph.grafana.a.smithi050.stdout:Sep 18 08:03:55 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-grafana-a[60403]: logger=migrator t=2024-09-18T08:03:55.570294832Z level=info msg="Migration successfully executed" id="alter dashboard_snapshot to mediumtext v2" duration=40.53µs 2024-09-18T08:03:55.788 INFO:journalctl@ceph.grafana.a.smithi050.stdout:Sep 18 08:03:55 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-grafana-a[60403]: logger=migrator t=2024-09-18T08:03:55.570691041Z level=info msg="Executing migration" id="Update dashboard_snapshot table charset" 2024-09-18T08:03:55.788 INFO:journalctl@ceph.grafana.a.smithi050.stdout:Sep 18 08:03:55 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-grafana-a[60403]: logger=migrator t=2024-09-18T08:03:55.57071195Z level=info msg="Migration successfully executed" id="Update dashboard_snapshot table charset" duration=22.124µs 2024-09-18T08:03:55.788 INFO:journalctl@ceph.grafana.a.smithi050.stdout:Sep 18 08:03:55 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-grafana-a[60403]: logger=migrator t=2024-09-18T08:03:55.571114992Z level=info msg="Executing migration" id="Add column external_delete_url to dashboard_snapshots table" 2024-09-18T08:03:55.788 INFO:journalctl@ceph.grafana.a.smithi050.stdout:Sep 18 08:03:55 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-grafana-a[60403]: logger=migrator t=2024-09-18T08:03:55.572842068Z level=info msg="Migration successfully executed" id="Add column external_delete_url to dashboard_snapshots table" duration=1.726961ms 2024-09-18T08:03:55.788 INFO:journalctl@ceph.grafana.a.smithi050.stdout:Sep 18 08:03:55 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-grafana-a[60403]: logger=migrator t=2024-09-18T08:03:55.573180182Z level=info msg="Executing migration" id="Add encrypted dashboard json column" 2024-09-18T08:03:55.788 INFO:journalctl@ceph.grafana.a.smithi050.stdout:Sep 18 08:03:55 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-grafana-a[60403]: logger=migrator t=2024-09-18T08:03:55.57497809Z level=info msg="Migration successfully executed" id="Add encrypted dashboard json column" duration=1.795635ms 2024-09-18T08:03:55.788 INFO:journalctl@ceph.grafana.a.smithi050.stdout:Sep 18 08:03:55 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-grafana-a[60403]: logger=migrator t=2024-09-18T08:03:55.575337938Z level=info msg="Executing migration" id="Change dashboard_encrypted column to MEDIUMBLOB" 2024-09-18T08:03:55.788 INFO:journalctl@ceph.grafana.a.smithi050.stdout:Sep 18 08:03:55 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-grafana-a[60403]: logger=migrator t=2024-09-18T08:03:55.575378907Z level=info msg="Migration successfully executed" id="Change dashboard_encrypted column to MEDIUMBLOB" duration=41.349µs 2024-09-18T08:03:55.788 INFO:journalctl@ceph.grafana.a.smithi050.stdout:Sep 18 08:03:55 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-grafana-a[60403]: logger=migrator t=2024-09-18T08:03:55.575764792Z level=info msg="Executing migration" id="create quota table v1" 2024-09-18T08:03:55.788 INFO:journalctl@ceph.grafana.a.smithi050.stdout:Sep 18 08:03:55 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-grafana-a[60403]: logger=migrator t=2024-09-18T08:03:55.576289905Z level=info msg="Migration successfully executed" id="create quota table v1" duration=525.448µs 2024-09-18T08:03:55.788 INFO:journalctl@ceph.grafana.a.smithi050.stdout:Sep 18 08:03:55 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-grafana-a[60403]: logger=migrator t=2024-09-18T08:03:55.576613649Z level=info msg="Executing migration" id="create index UQE_quota_org_id_user_id_target - v1" 2024-09-18T08:03:55.789 INFO:journalctl@ceph.grafana.a.smithi050.stdout:Sep 18 08:03:55 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-grafana-a[60403]: logger=migrator t=2024-09-18T08:03:55.577149913Z level=info msg="Migration successfully executed" id="create index UQE_quota_org_id_user_id_target - v1" duration=535.894µs 2024-09-18T08:03:55.789 INFO:journalctl@ceph.grafana.a.smithi050.stdout:Sep 18 08:03:55 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-grafana-a[60403]: logger=migrator t=2024-09-18T08:03:55.577466097Z level=info msg="Executing migration" id="Update quota table charset" 2024-09-18T08:03:55.789 INFO:journalctl@ceph.grafana.a.smithi050.stdout:Sep 18 08:03:55 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-grafana-a[60403]: logger=migrator t=2024-09-18T08:03:55.577484388Z level=info msg="Migration successfully executed" id="Update quota table charset" duration=19.15µs 2024-09-18T08:03:55.789 INFO:journalctl@ceph.grafana.a.smithi050.stdout:Sep 18 08:03:55 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-grafana-a[60403]: logger=migrator t=2024-09-18T08:03:55.577909627Z level=info msg="Executing migration" id="create plugin_setting table" 2024-09-18T08:03:55.789 INFO:journalctl@ceph.grafana.a.smithi050.stdout:Sep 18 08:03:55 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-grafana-a[60403]: logger=migrator t=2024-09-18T08:03:55.578441957Z level=info msg="Migration successfully executed" id="create plugin_setting table" duration=532.234µs 2024-09-18T08:03:55.789 INFO:journalctl@ceph.grafana.a.smithi050.stdout:Sep 18 08:03:55 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-grafana-a[60403]: logger=migrator t=2024-09-18T08:03:55.578762219Z level=info msg="Executing migration" id="create index UQE_plugin_setting_org_id_plugin_id - v1" 2024-09-18T08:03:55.789 INFO:journalctl@ceph.grafana.a.smithi050.stdout:Sep 18 08:03:55 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-grafana-a[60403]: logger=migrator t=2024-09-18T08:03:55.579339572Z level=info msg="Migration successfully executed" id="create index UQE_plugin_setting_org_id_plugin_id - v1" duration=577.774µs 2024-09-18T08:03:55.789 INFO:journalctl@ceph.grafana.a.smithi050.stdout:Sep 18 08:03:55 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-grafana-a[60403]: logger=migrator t=2024-09-18T08:03:55.579662361Z level=info msg="Executing migration" id="Add column plugin_version to plugin_settings" 2024-09-18T08:03:55.789 INFO:journalctl@ceph.grafana.a.smithi050.stdout:Sep 18 08:03:55 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-grafana-a[60403]: logger=migrator t=2024-09-18T08:03:55.581466443Z level=info msg="Migration successfully executed" id="Add column plugin_version to plugin_settings" duration=1.804358ms 2024-09-18T08:03:55.789 INFO:journalctl@ceph.grafana.a.smithi050.stdout:Sep 18 08:03:55 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-grafana-a[60403]: logger=migrator t=2024-09-18T08:03:55.581782566Z level=info msg="Executing migration" id="Update plugin_setting table charset" 2024-09-18T08:03:55.789 INFO:journalctl@ceph.grafana.a.smithi050.stdout:Sep 18 08:03:55 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-grafana-a[60403]: logger=migrator t=2024-09-18T08:03:55.581803526Z level=info msg="Migration successfully executed" id="Update plugin_setting table charset" duration=22.38µs 2024-09-18T08:03:55.789 INFO:journalctl@ceph.grafana.a.smithi050.stdout:Sep 18 08:03:55 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-grafana-a[60403]: logger=migrator t=2024-09-18T08:03:55.5822164Z level=info msg="Executing migration" id="create session table" 2024-09-18T08:03:55.789 INFO:journalctl@ceph.grafana.a.smithi050.stdout:Sep 18 08:03:55 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-grafana-a[60403]: logger=migrator t=2024-09-18T08:03:55.582812071Z level=info msg="Migration successfully executed" id="create session table" duration=595.282µs 2024-09-18T08:03:55.790 INFO:journalctl@ceph.grafana.a.smithi050.stdout:Sep 18 08:03:55 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-grafana-a[60403]: logger=migrator t=2024-09-18T08:03:55.583183646Z level=info msg="Executing migration" id="Drop old table playlist table" 2024-09-18T08:03:55.790 INFO:journalctl@ceph.grafana.a.smithi050.stdout:Sep 18 08:03:55 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-grafana-a[60403]: logger=migrator t=2024-09-18T08:03:55.583248536Z level=info msg="Migration successfully executed" id="Drop old table playlist table" duration=64.452µs 2024-09-18T08:03:55.790 INFO:journalctl@ceph.grafana.a.smithi050.stdout:Sep 18 08:03:55 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-grafana-a[60403]: logger=migrator t=2024-09-18T08:03:55.583623971Z level=info msg="Executing migration" id="Drop old table playlist_item table" 2024-09-18T08:03:55.790 INFO:journalctl@ceph.grafana.a.smithi050.stdout:Sep 18 08:03:55 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-grafana-a[60403]: logger=migrator t=2024-09-18T08:03:55.583688397Z level=info msg="Migration successfully executed" id="Drop old table playlist_item table" duration=64.474µs 2024-09-18T08:03:55.790 INFO:journalctl@ceph.grafana.a.smithi050.stdout:Sep 18 08:03:55 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-grafana-a[60403]: logger=migrator t=2024-09-18T08:03:55.584079394Z level=info msg="Executing migration" id="create playlist table v2" 2024-09-18T08:03:55.790 INFO:journalctl@ceph.grafana.a.smithi050.stdout:Sep 18 08:03:55 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-grafana-a[60403]: logger=migrator t=2024-09-18T08:03:55.584581052Z level=info msg="Migration successfully executed" id="create playlist table v2" duration=502.332µs 2024-09-18T08:03:55.790 INFO:journalctl@ceph.grafana.a.smithi050.stdout:Sep 18 08:03:55 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-grafana-a[60403]: logger=migrator t=2024-09-18T08:03:55.584913656Z level=info msg="Executing migration" id="create playlist item table v2" 2024-09-18T08:03:55.790 INFO:journalctl@ceph.grafana.a.smithi050.stdout:Sep 18 08:03:55 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-grafana-a[60403]: logger=migrator t=2024-09-18T08:03:55.585416033Z level=info msg="Migration successfully executed" id="create playlist item table v2" duration=502.867µs 2024-09-18T08:03:55.790 INFO:journalctl@ceph.grafana.a.smithi050.stdout:Sep 18 08:03:55 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-grafana-a[60403]: logger=migrator t=2024-09-18T08:03:55.585774056Z level=info msg="Executing migration" id="Update playlist table charset" 2024-09-18T08:03:55.790 INFO:journalctl@ceph.grafana.a.smithi050.stdout:Sep 18 08:03:55 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-grafana-a[60403]: logger=migrator t=2024-09-18T08:03:55.585790059Z level=info msg="Migration successfully executed" id="Update playlist table charset" duration=16.649µs 2024-09-18T08:03:55.790 INFO:journalctl@ceph.grafana.a.smithi050.stdout:Sep 18 08:03:55 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-grafana-a[60403]: logger=migrator t=2024-09-18T08:03:55.586193969Z level=info msg="Executing migration" id="Update playlist_item table charset" 2024-09-18T08:03:55.790 INFO:journalctl@ceph.grafana.a.smithi050.stdout:Sep 18 08:03:55 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-grafana-a[60403]: logger=migrator t=2024-09-18T08:03:55.586216082Z level=info msg="Migration successfully executed" id="Update playlist_item table charset" duration=23.349µs 2024-09-18T08:03:55.791 INFO:journalctl@ceph.grafana.a.smithi050.stdout:Sep 18 08:03:55 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-grafana-a[60403]: logger=migrator t=2024-09-18T08:03:55.586611307Z level=info msg="Executing migration" id="Add playlist column created_at" 2024-09-18T08:03:55.791 INFO:journalctl@ceph.grafana.a.smithi050.stdout:Sep 18 08:03:55 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-grafana-a[60403]: logger=migrator t=2024-09-18T08:03:55.588562608Z level=info msg="Migration successfully executed" id="Add playlist column created_at" duration=1.952893ms 2024-09-18T08:03:55.791 INFO:journalctl@ceph.grafana.a.smithi050.stdout:Sep 18 08:03:55 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-grafana-a[60403]: logger=migrator t=2024-09-18T08:03:55.58888786Z level=info msg="Executing migration" id="Add playlist column updated_at" 2024-09-18T08:03:55.791 INFO:journalctl@ceph.grafana.a.smithi050.stdout:Sep 18 08:03:55 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-grafana-a[60403]: logger=migrator t=2024-09-18T08:03:55.590816687Z level=info msg="Migration successfully executed" id="Add playlist column updated_at" duration=1.928556ms 2024-09-18T08:03:55.791 INFO:journalctl@ceph.grafana.a.smithi050.stdout:Sep 18 08:03:55 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-grafana-a[60403]: logger=migrator t=2024-09-18T08:03:55.591150635Z level=info msg="Executing migration" id="drop preferences table v2" 2024-09-18T08:03:55.791 INFO:journalctl@ceph.grafana.a.smithi050.stdout:Sep 18 08:03:55 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-grafana-a[60403]: logger=migrator t=2024-09-18T08:03:55.591214645Z level=info msg="Migration successfully executed" id="drop preferences table v2" duration=63.606µs 2024-09-18T08:03:55.791 INFO:journalctl@ceph.grafana.a.smithi050.stdout:Sep 18 08:03:55 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-grafana-a[60403]: logger=migrator t=2024-09-18T08:03:55.591589715Z level=info msg="Executing migration" id="drop preferences table v3" 2024-09-18T08:03:55.791 INFO:journalctl@ceph.grafana.a.smithi050.stdout:Sep 18 08:03:55 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-grafana-a[60403]: logger=migrator t=2024-09-18T08:03:55.591653063Z level=info msg="Migration successfully executed" id="drop preferences table v3" duration=63.49µs 2024-09-18T08:03:55.791 INFO:journalctl@ceph.grafana.a.smithi050.stdout:Sep 18 08:03:55 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-grafana-a[60403]: logger=migrator t=2024-09-18T08:03:55.591990466Z level=info msg="Executing migration" id="create preferences table v3" 2024-09-18T08:03:55.791 INFO:journalctl@ceph.grafana.a.smithi050.stdout:Sep 18 08:03:55 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-grafana-a[60403]: logger=migrator t=2024-09-18T08:03:55.592510189Z level=info msg="Migration successfully executed" id="create preferences table v3" duration=519.979µs 2024-09-18T08:03:55.791 INFO:journalctl@ceph.grafana.a.smithi050.stdout:Sep 18 08:03:55 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-grafana-a[60403]: logger=migrator t=2024-09-18T08:03:55.592846651Z level=info msg="Executing migration" id="Update preferences table charset" 2024-09-18T08:03:55.791 INFO:journalctl@ceph.grafana.a.smithi050.stdout:Sep 18 08:03:55 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-grafana-a[60403]: logger=migrator t=2024-09-18T08:03:55.592862832Z level=info msg="Migration successfully executed" id="Update preferences table charset" duration=16.97µs 2024-09-18T08:03:55.794 INFO:journalctl@ceph.grafana.a.smithi050.stdout:Sep 18 08:03:55 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-grafana-a[60403]: logger=migrator t=2024-09-18T08:03:55.593269101Z level=info msg="Executing migration" id="Add column team_id in preferences" 2024-09-18T08:03:55.794 INFO:journalctl@ceph.grafana.a.smithi050.stdout:Sep 18 08:03:55 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-grafana-a[60403]: logger=migrator t=2024-09-18T08:03:55.595238265Z level=info msg="Migration successfully executed" id="Add column team_id in preferences" duration=1.968162ms 2024-09-18T08:03:55.794 INFO:journalctl@ceph.grafana.a.smithi050.stdout:Sep 18 08:03:55 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-grafana-a[60403]: logger=migrator t=2024-09-18T08:03:55.595578895Z level=info msg="Executing migration" id="Update team_id column values in preferences" 2024-09-18T08:03:55.794 INFO:journalctl@ceph.grafana.a.smithi050.stdout:Sep 18 08:03:55 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-grafana-a[60403]: logger=migrator t=2024-09-18T08:03:55.595689679Z level=info msg="Migration successfully executed" id="Update team_id column values in preferences" duration=111.596µs 2024-09-18T08:03:55.794 INFO:journalctl@ceph.grafana.a.smithi050.stdout:Sep 18 08:03:55 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-grafana-a[60403]: logger=migrator t=2024-09-18T08:03:55.596075863Z level=info msg="Executing migration" id="Add column week_start in preferences" 2024-09-18T08:03:55.794 INFO:journalctl@ceph.grafana.a.smithi050.stdout:Sep 18 08:03:55 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-grafana-a[60403]: logger=migrator t=2024-09-18T08:03:55.598064035Z level=info msg="Migration successfully executed" id="Add column week_start in preferences" duration=1.988414ms 2024-09-18T08:03:55.794 INFO:journalctl@ceph.grafana.a.smithi050.stdout:Sep 18 08:03:55 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-grafana-a[60403]: logger=migrator t=2024-09-18T08:03:55.598401893Z level=info msg="Executing migration" id="Add column preferences.json_data" 2024-09-18T08:03:55.794 INFO:journalctl@ceph.grafana.a.smithi050.stdout:Sep 18 08:03:55 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-grafana-a[60403]: logger=migrator t=2024-09-18T08:03:55.600372069Z level=info msg="Migration successfully executed" id="Add column preferences.json_data" duration=1.970437ms 2024-09-18T08:03:55.794 INFO:journalctl@ceph.grafana.a.smithi050.stdout:Sep 18 08:03:55 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-grafana-a[60403]: logger=migrator t=2024-09-18T08:03:55.600692168Z level=info msg="Executing migration" id="alter preferences.json_data to mediumtext v1" 2024-09-18T08:03:55.795 INFO:journalctl@ceph.grafana.a.smithi050.stdout:Sep 18 08:03:55 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-grafana-a[60403]: logger=migrator t=2024-09-18T08:03:55.600733373Z level=info msg="Migration successfully executed" id="alter preferences.json_data to mediumtext v1" duration=41.631µs 2024-09-18T08:03:55.795 INFO:journalctl@ceph.grafana.a.smithi050.stdout:Sep 18 08:03:55 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-grafana-a[60403]: logger=migrator t=2024-09-18T08:03:55.601155865Z level=info msg="Executing migration" id="Add preferences index org_id" 2024-09-18T08:03:55.795 INFO:journalctl@ceph.grafana.a.smithi050.stdout:Sep 18 08:03:55 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-grafana-a[60403]: logger=migrator t=2024-09-18T08:03:55.601759231Z level=info msg="Migration successfully executed" id="Add preferences index org_id" duration=603.616µs 2024-09-18T08:03:55.795 INFO:journalctl@ceph.grafana.a.smithi050.stdout:Sep 18 08:03:55 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-grafana-a[60403]: logger=migrator t=2024-09-18T08:03:55.602095448Z level=info msg="Executing migration" id="Add preferences index user_id" 2024-09-18T08:03:55.795 INFO:journalctl@ceph.grafana.a.smithi050.stdout:Sep 18 08:03:55 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-grafana-a[60403]: logger=migrator t=2024-09-18T08:03:55.602663437Z level=info msg="Migration successfully executed" id="Add preferences index user_id" duration=567.792µs 2024-09-18T08:03:55.795 INFO:journalctl@ceph.grafana.a.smithi050.stdout:Sep 18 08:03:55 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-grafana-a[60403]: logger=migrator t=2024-09-18T08:03:55.60301616Z level=info msg="Executing migration" id="create alert table v1" 2024-09-18T08:03:55.795 INFO:journalctl@ceph.grafana.a.smithi050.stdout:Sep 18 08:03:55 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-grafana-a[60403]: logger=migrator t=2024-09-18T08:03:55.603829542Z level=info msg="Migration successfully executed" id="create alert table v1" duration=813.524µs 2024-09-18T08:03:55.795 INFO:journalctl@ceph.grafana.a.smithi050.stdout:Sep 18 08:03:55 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-grafana-a[60403]: logger=migrator t=2024-09-18T08:03:55.604211188Z level=info msg="Executing migration" id="add index alert org_id & id " 2024-09-18T08:03:55.795 INFO:journalctl@ceph.grafana.a.smithi050.stdout:Sep 18 08:03:55 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-grafana-a[60403]: logger=migrator t=2024-09-18T08:03:55.604831696Z level=info msg="Migration successfully executed" id="add index alert org_id & id " duration=620.26µs 2024-09-18T08:03:55.796 INFO:journalctl@ceph.grafana.a.smithi050.stdout:Sep 18 08:03:55 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-grafana-a[60403]: logger=migrator t=2024-09-18T08:03:55.605194148Z level=info msg="Executing migration" id="add index alert state" 2024-09-18T08:03:55.796 INFO:journalctl@ceph.grafana.a.smithi050.stdout:Sep 18 08:03:55 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-grafana-a[60403]: logger=migrator t=2024-09-18T08:03:55.605729302Z level=info msg="Migration successfully executed" id="add index alert state" duration=535.017µs 2024-09-18T08:03:55.796 INFO:journalctl@ceph.grafana.a.smithi050.stdout:Sep 18 08:03:55 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-grafana-a[60403]: logger=migrator t=2024-09-18T08:03:55.606067363Z level=info msg="Executing migration" id="add index alert dashboard_id" 2024-09-18T08:03:55.796 INFO:journalctl@ceph.grafana.a.smithi050.stdout:Sep 18 08:03:55 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-grafana-a[60403]: logger=migrator t=2024-09-18T08:03:55.606617544Z level=info msg="Migration successfully executed" id="add index alert dashboard_id" duration=549.94µs 2024-09-18T08:03:55.796 INFO:journalctl@ceph.grafana.a.smithi050.stdout:Sep 18 08:03:55 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-grafana-a[60403]: logger=migrator t=2024-09-18T08:03:55.606956062Z level=info msg="Executing migration" id="Create alert_rule_tag table v1" 2024-09-18T08:03:55.796 INFO:journalctl@ceph.grafana.a.smithi050.stdout:Sep 18 08:03:55 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-grafana-a[60403]: logger=migrator t=2024-09-18T08:03:55.607412812Z level=info msg="Migration successfully executed" id="Create alert_rule_tag table v1" duration=455.564µs 2024-09-18T08:03:55.796 INFO:journalctl@ceph.grafana.a.smithi050.stdout:Sep 18 08:03:55 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-grafana-a[60403]: logger=migrator t=2024-09-18T08:03:55.607757876Z level=info msg="Executing migration" id="Add unique index alert_rule_tag.alert_id_tag_id" 2024-09-18T08:03:55.796 INFO:journalctl@ceph.grafana.a.smithi050.stdout:Sep 18 08:03:55 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-grafana-a[60403]: logger=migrator t=2024-09-18T08:03:55.608346642Z level=info msg="Migration successfully executed" id="Add unique index alert_rule_tag.alert_id_tag_id" duration=588.573µs 2024-09-18T08:03:55.796 INFO:journalctl@ceph.grafana.a.smithi050.stdout:Sep 18 08:03:55 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-grafana-a[60403]: logger=migrator t=2024-09-18T08:03:55.608692711Z level=info msg="Executing migration" id="drop index UQE_alert_rule_tag_alert_id_tag_id - v1" 2024-09-18T08:03:55.796 INFO:journalctl@ceph.grafana.a.smithi050.stdout:Sep 18 08:03:55 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-grafana-a[60403]: logger=migrator t=2024-09-18T08:03:55.609245937Z level=info msg="Migration successfully executed" id="drop index UQE_alert_rule_tag_alert_id_tag_id - v1" duration=552.486µs 2024-09-18T08:03:55.796 INFO:journalctl@ceph.grafana.a.smithi050.stdout:Sep 18 08:03:55 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-grafana-a[60403]: logger=migrator t=2024-09-18T08:03:55.609580766Z level=info msg="Executing migration" id="Rename table alert_rule_tag to alert_rule_tag_v1 - v1" 2024-09-18T08:03:55.797 INFO:journalctl@ceph.grafana.a.smithi050.stdout:Sep 18 08:03:55 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-grafana-a[60403]: logger=migrator t=2024-09-18T08:03:55.615858715Z level=info msg="Migration successfully executed" id="Rename table alert_rule_tag to alert_rule_tag_v1 - v1" duration=6.277567ms 2024-09-18T08:03:55.797 INFO:journalctl@ceph.grafana.a.smithi050.stdout:Sep 18 08:03:55 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-grafana-a[60403]: logger=migrator t=2024-09-18T08:03:55.616194289Z level=info msg="Executing migration" id="Create alert_rule_tag table v2" 2024-09-18T08:03:55.797 INFO:journalctl@ceph.grafana.a.smithi050.stdout:Sep 18 08:03:55 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-grafana-a[60403]: logger=migrator t=2024-09-18T08:03:55.616678806Z level=info msg="Migration successfully executed" id="Create alert_rule_tag table v2" duration=484.501µs 2024-09-18T08:03:55.797 INFO:journalctl@ceph.grafana.a.smithi050.stdout:Sep 18 08:03:55 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-grafana-a[60403]: logger=migrator t=2024-09-18T08:03:55.617014802Z level=info msg="Executing migration" id="create index UQE_alert_rule_tag_alert_id_tag_id - Add unique index alert_rule_tag.alert_id_tag_id V2" 2024-09-18T08:03:55.797 INFO:journalctl@ceph.grafana.a.smithi050.stdout:Sep 18 08:03:55 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-grafana-a[60403]: logger=migrator t=2024-09-18T08:03:55.617558046Z level=info msg="Migration successfully executed" id="create index UQE_alert_rule_tag_alert_id_tag_id - Add unique index alert_rule_tag.alert_id_tag_id V2" duration=542.985µs 2024-09-18T08:03:55.797 INFO:journalctl@ceph.grafana.a.smithi050.stdout:Sep 18 08:03:55 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-grafana-a[60403]: logger=migrator t=2024-09-18T08:03:55.617910216Z level=info msg="Executing migration" id="copy alert_rule_tag v1 to v2" 2024-09-18T08:03:55.797 INFO:journalctl@ceph.grafana.a.smithi050.stdout:Sep 18 08:03:55 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-grafana-a[60403]: logger=migrator t=2024-09-18T08:03:55.61812013Z level=info msg="Migration successfully executed" id="copy alert_rule_tag v1 to v2" duration=209.874µs 2024-09-18T08:03:55.797 INFO:journalctl@ceph.grafana.a.smithi050.stdout:Sep 18 08:03:55 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-grafana-a[60403]: logger=migrator t=2024-09-18T08:03:55.61844828Z level=info msg="Executing migration" id="drop table alert_rule_tag_v1" 2024-09-18T08:03:55.797 INFO:journalctl@ceph.grafana.a.smithi050.stdout:Sep 18 08:03:55 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-grafana-a[60403]: logger=migrator t=2024-09-18T08:03:55.618821908Z level=info msg="Migration successfully executed" id="drop table alert_rule_tag_v1" duration=373.514µs 2024-09-18T08:03:55.797 INFO:journalctl@ceph.grafana.a.smithi050.stdout:Sep 18 08:03:55 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-grafana-a[60403]: logger=migrator t=2024-09-18T08:03:55.619173236Z level=info msg="Executing migration" id="create alert_notification table v1" 2024-09-18T08:03:55.797 INFO:journalctl@ceph.grafana.a.smithi050.stdout:Sep 18 08:03:55 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-grafana-a[60403]: logger=migrator t=2024-09-18T08:03:55.619669133Z level=info msg="Migration successfully executed" id="create alert_notification table v1" duration=495.705µs 2024-09-18T08:03:55.797 INFO:journalctl@ceph.grafana.a.smithi050.stdout:Sep 18 08:03:55 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-grafana-a[60403]: logger=migrator t=2024-09-18T08:03:55.620029642Z level=info msg="Executing migration" id="Add column is_default" 2024-09-18T08:03:55.798 INFO:journalctl@ceph.grafana.a.smithi050.stdout:Sep 18 08:03:55 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-grafana-a[60403]: logger=migrator t=2024-09-18T08:03:55.622220965Z level=info msg="Migration successfully executed" id="Add column is_default" duration=2.191126ms 2024-09-18T08:03:55.798 INFO:journalctl@ceph.grafana.a.smithi050.stdout:Sep 18 08:03:55 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-grafana-a[60403]: logger=migrator t=2024-09-18T08:03:55.622564052Z level=info msg="Executing migration" id="Add column frequency" 2024-09-18T08:03:55.798 INFO:journalctl@ceph.grafana.a.smithi050.stdout:Sep 18 08:03:55 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-grafana-a[60403]: logger=migrator t=2024-09-18T08:03:55.624960859Z level=info msg="Migration successfully executed" id="Add column frequency" duration=2.396528ms 2024-09-18T08:03:55.798 INFO:journalctl@ceph.grafana.a.smithi050.stdout:Sep 18 08:03:55 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-grafana-a[60403]: logger=migrator t=2024-09-18T08:03:55.625306988Z level=info msg="Executing migration" id="Add column send_reminder" 2024-09-18T08:03:55.798 INFO:journalctl@ceph.grafana.a.smithi050.stdout:Sep 18 08:03:55 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-grafana-a[60403]: logger=migrator t=2024-09-18T08:03:55.627873383Z level=info msg="Migration successfully executed" id="Add column send_reminder" duration=2.560529ms 2024-09-18T08:03:55.798 INFO:journalctl@ceph.grafana.a.smithi050.stdout:Sep 18 08:03:55 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-grafana-a[60403]: logger=migrator t=2024-09-18T08:03:55.628224844Z level=info msg="Executing migration" id="Add column disable_resolve_message" 2024-09-18T08:03:55.798 INFO:journalctl@ceph.grafana.a.smithi050.stdout:Sep 18 08:03:55 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-grafana-a[60403]: logger=migrator t=2024-09-18T08:03:55.630446424Z level=info msg="Migration successfully executed" id="Add column disable_resolve_message" duration=2.221387ms 2024-09-18T08:03:55.798 INFO:journalctl@ceph.grafana.a.smithi050.stdout:Sep 18 08:03:55 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-grafana-a[60403]: logger=migrator t=2024-09-18T08:03:55.630773336Z level=info msg="Executing migration" id="add index alert_notification org_id & name" 2024-09-18T08:03:55.798 INFO:journalctl@ceph.grafana.a.smithi050.stdout:Sep 18 08:03:55 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-grafana-a[60403]: logger=migrator t=2024-09-18T08:03:55.631334663Z level=info msg="Migration successfully executed" id="add index alert_notification org_id & name" duration=561.074µs 2024-09-18T08:03:55.798 INFO:journalctl@ceph.grafana.a.smithi050.stdout:Sep 18 08:03:55 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-grafana-a[60403]: logger=migrator t=2024-09-18T08:03:55.631656453Z level=info msg="Executing migration" id="Update alert table charset" 2024-09-18T08:03:55.798 INFO:journalctl@ceph.grafana.a.smithi050.stdout:Sep 18 08:03:55 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-grafana-a[60403]: logger=migrator t=2024-09-18T08:03:55.631676924Z level=info msg="Migration successfully executed" id="Update alert table charset" duration=20.68µs 2024-09-18T08:03:55.798 INFO:journalctl@ceph.grafana.a.smithi050.stdout:Sep 18 08:03:55 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-grafana-a[60403]: logger=migrator t=2024-09-18T08:03:55.63206316Z level=info msg="Executing migration" id="Update alert_notification table charset" 2024-09-18T08:03:55.798 INFO:journalctl@ceph.grafana.a.smithi050.stdout:Sep 18 08:03:55 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-grafana-a[60403]: logger=migrator t=2024-09-18T08:03:55.632083251Z level=info msg="Migration successfully executed" id="Update alert_notification table charset" duration=20.231µs 2024-09-18T08:03:55.799 INFO:journalctl@ceph.grafana.a.smithi050.stdout:Sep 18 08:03:55 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-grafana-a[60403]: logger=migrator t=2024-09-18T08:03:55.632470161Z level=info msg="Executing migration" id="create notification_journal table v1" 2024-09-18T08:03:55.799 INFO:journalctl@ceph.grafana.a.smithi050.stdout:Sep 18 08:03:55 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-grafana-a[60403]: logger=migrator t=2024-09-18T08:03:55.632965852Z level=info msg="Migration successfully executed" id="create notification_journal table v1" duration=495.547µs 2024-09-18T08:03:55.799 INFO:journalctl@ceph.grafana.a.smithi050.stdout:Sep 18 08:03:55 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-grafana-a[60403]: logger=migrator t=2024-09-18T08:03:55.633315175Z level=info msg="Executing migration" id="add index notification_journal org_id & alert_id & notifier_id" 2024-09-18T08:03:55.799 INFO:journalctl@ceph.grafana.a.smithi050.stdout:Sep 18 08:03:55 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-grafana-a[60403]: logger=migrator t=2024-09-18T08:03:55.633859201Z level=info msg="Migration successfully executed" id="add index notification_journal org_id & alert_id & notifier_id" duration=543.701µs 2024-09-18T08:03:55.799 INFO:journalctl@ceph.grafana.a.smithi050.stdout:Sep 18 08:03:55 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-grafana-a[60403]: logger=migrator t=2024-09-18T08:03:55.634193195Z level=info msg="Executing migration" id="drop alert_notification_journal" 2024-09-18T08:03:55.799 INFO:journalctl@ceph.grafana.a.smithi050.stdout:Sep 18 08:03:55 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-grafana-a[60403]: logger=migrator t=2024-09-18T08:03:55.634734199Z level=info msg="Migration successfully executed" id="drop alert_notification_journal" duration=540.861µs 2024-09-18T08:03:55.799 INFO:journalctl@ceph.grafana.a.smithi050.stdout:Sep 18 08:03:55 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-grafana-a[60403]: logger=migrator t=2024-09-18T08:03:55.635086397Z level=info msg="Executing migration" id="create alert_notification_state table v1" 2024-09-18T08:03:55.799 INFO:journalctl@ceph.grafana.a.smithi050.stdout:Sep 18 08:03:55 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-grafana-a[60403]: logger=migrator t=2024-09-18T08:03:55.635622855Z level=info msg="Migration successfully executed" id="create alert_notification_state table v1" duration=536.305µs 2024-09-18T08:03:55.799 INFO:journalctl@ceph.grafana.a.smithi050.stdout:Sep 18 08:03:55 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-grafana-a[60403]: logger=migrator t=2024-09-18T08:03:55.635954642Z level=info msg="Executing migration" id="add index alert_notification_state org_id & alert_id & notifier_id" 2024-09-18T08:03:55.799 INFO:journalctl@ceph.grafana.a.smithi050.stdout:Sep 18 08:03:55 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-grafana-a[60403]: logger=migrator t=2024-09-18T08:03:55.636499612Z level=info msg="Migration successfully executed" id="add index alert_notification_state org_id & alert_id & notifier_id" duration=544.775µs 2024-09-18T08:03:55.799 INFO:journalctl@ceph.grafana.a.smithi050.stdout:Sep 18 08:03:55 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-grafana-a[60403]: logger=migrator t=2024-09-18T08:03:55.6368321Z level=info msg="Executing migration" id="Add for to alert table" 2024-09-18T08:03:55.799 INFO:journalctl@ceph.grafana.a.smithi050.stdout:Sep 18 08:03:55 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-grafana-a[60403]: logger=migrator t=2024-09-18T08:03:55.639137948Z level=info msg="Migration successfully executed" id="Add for to alert table" duration=2.305455ms 2024-09-18T08:03:55.799 INFO:journalctl@ceph.grafana.a.smithi050.stdout:Sep 18 08:03:55 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-grafana-a[60403]: logger=migrator t=2024-09-18T08:03:55.639460544Z level=info msg="Executing migration" id="Add column uid in alert_notification" 2024-09-18T08:03:55.800 INFO:journalctl@ceph.grafana.a.smithi050.stdout:Sep 18 08:03:55 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-grafana-a[60403]: logger=migrator t=2024-09-18T08:03:55.641922267Z level=info msg="Migration successfully executed" id="Add column uid in alert_notification" duration=2.461454ms 2024-09-18T08:03:55.800 INFO:journalctl@ceph.grafana.a.smithi050.stdout:Sep 18 08:03:55 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-grafana-a[60403]: logger=migrator t=2024-09-18T08:03:55.642242495Z level=info msg="Executing migration" id="Update uid column values in alert_notification" 2024-09-18T08:03:55.800 INFO:journalctl@ceph.grafana.a.smithi050.stdout:Sep 18 08:03:55 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-grafana-a[60403]: logger=migrator t=2024-09-18T08:03:55.642372807Z level=info msg="Migration successfully executed" id="Update uid column values in alert_notification" duration=130.453µs 2024-09-18T08:03:55.800 INFO:journalctl@ceph.grafana.a.smithi050.stdout:Sep 18 08:03:55 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-grafana-a[60403]: logger=migrator t=2024-09-18T08:03:55.642730345Z level=info msg="Executing migration" id="Add unique index alert_notification_org_id_uid" 2024-09-18T08:03:55.800 INFO:journalctl@ceph.grafana.a.smithi050.stdout:Sep 18 08:03:55 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-grafana-a[60403]: logger=migrator t=2024-09-18T08:03:55.643292493Z level=info msg="Migration successfully executed" id="Add unique index alert_notification_org_id_uid" duration=561.795µs 2024-09-18T08:03:55.800 INFO:journalctl@ceph.grafana.a.smithi050.stdout:Sep 18 08:03:55 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-grafana-a[60403]: logger=migrator t=2024-09-18T08:03:55.643620075Z level=info msg="Executing migration" id="Remove unique index org_id_name" 2024-09-18T08:03:55.800 INFO:journalctl@ceph.grafana.a.smithi050.stdout:Sep 18 08:03:55 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-grafana-a[60403]: logger=migrator t=2024-09-18T08:03:55.644172677Z level=info msg="Migration successfully executed" id="Remove unique index org_id_name" duration=552.512µs 2024-09-18T08:03:55.800 INFO:journalctl@ceph.grafana.a.smithi050.stdout:Sep 18 08:03:55 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-grafana-a[60403]: logger=migrator t=2024-09-18T08:03:55.644494111Z level=info msg="Executing migration" id="Add column secure_settings in alert_notification" 2024-09-18T08:03:55.800 INFO:journalctl@ceph.grafana.a.smithi050.stdout:Sep 18 08:03:55 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-grafana-a[60403]: logger=migrator t=2024-09-18T08:03:55.646785703Z level=info msg="Migration successfully executed" id="Add column secure_settings in alert_notification" duration=2.290241ms 2024-09-18T08:03:55.800 INFO:journalctl@ceph.grafana.a.smithi050.stdout:Sep 18 08:03:55 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-grafana-a[60403]: logger=migrator t=2024-09-18T08:03:55.647136425Z level=info msg="Executing migration" id="alter alert.settings to mediumtext" 2024-09-18T08:03:55.800 INFO:journalctl@ceph.grafana.a.smithi050.stdout:Sep 18 08:03:55 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-grafana-a[60403]: logger=migrator t=2024-09-18T08:03:55.647196458Z level=info msg="Migration successfully executed" id="alter alert.settings to mediumtext" duration=60.216µs 2024-09-18T08:03:55.800 INFO:journalctl@ceph.grafana.a.smithi050.stdout:Sep 18 08:03:55 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-grafana-a[60403]: logger=migrator t=2024-09-18T08:03:55.647707461Z level=info msg="Executing migration" id="Add non-unique index alert_notification_state_alert_id" 2024-09-18T08:03:55.800 INFO:journalctl@ceph.grafana.a.smithi050.stdout:Sep 18 08:03:55 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-grafana-a[60403]: logger=migrator t=2024-09-18T08:03:55.648243937Z level=info msg="Migration successfully executed" id="Add non-unique index alert_notification_state_alert_id" duration=536.488µs 2024-09-18T08:03:55.801 INFO:journalctl@ceph.grafana.a.smithi050.stdout:Sep 18 08:03:55 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-grafana-a[60403]: logger=migrator t=2024-09-18T08:03:55.648561462Z level=info msg="Executing migration" id="Add non-unique index alert_rule_tag_alert_id" 2024-09-18T08:03:55.801 INFO:journalctl@ceph.grafana.a.smithi050.stdout:Sep 18 08:03:55 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-grafana-a[60403]: logger=migrator t=2024-09-18T08:03:55.649175509Z level=info msg="Migration successfully executed" id="Add non-unique index alert_rule_tag_alert_id" duration=613.825µs 2024-09-18T08:03:55.801 INFO:journalctl@ceph.grafana.a.smithi050.stdout:Sep 18 08:03:55 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-grafana-a[60403]: logger=migrator t=2024-09-18T08:03:55.649520053Z level=info msg="Executing migration" id="Drop old annotation table v4" 2024-09-18T08:03:55.801 INFO:journalctl@ceph.grafana.a.smithi050.stdout:Sep 18 08:03:55 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-grafana-a[60403]: logger=migrator t=2024-09-18T08:03:55.649595771Z level=info msg="Migration successfully executed" id="Drop old annotation table v4" duration=75.867µs 2024-09-18T08:03:55.801 INFO:journalctl@ceph.grafana.a.smithi050.stdout:Sep 18 08:03:55 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-grafana-a[60403]: logger=migrator t=2024-09-18T08:03:55.649993585Z level=info msg="Executing migration" id="create annotation table v5" 2024-09-18T08:03:55.801 INFO:journalctl@ceph.grafana.a.smithi050.stdout:Sep 18 08:03:55 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-grafana-a[60403]: logger=migrator t=2024-09-18T08:03:55.650577616Z level=info msg="Migration successfully executed" id="create annotation table v5" duration=583.778µs 2024-09-18T08:03:55.801 INFO:journalctl@ceph.grafana.a.smithi050.stdout:Sep 18 08:03:55 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-grafana-a[60403]: logger=migrator t=2024-09-18T08:03:55.650922165Z level=info msg="Executing migration" id="add index annotation 0 v3" 2024-09-18T08:03:55.801 INFO:journalctl@ceph.grafana.a.smithi050.stdout:Sep 18 08:03:55 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-grafana-a[60403]: logger=migrator t=2024-09-18T08:03:55.651462383Z level=info msg="Migration successfully executed" id="add index annotation 0 v3" duration=539.917µs 2024-09-18T08:03:55.801 INFO:journalctl@ceph.grafana.a.smithi050.stdout:Sep 18 08:03:55 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-grafana-a[60403]: logger=migrator t=2024-09-18T08:03:55.651788538Z level=info msg="Executing migration" id="add index annotation 1 v3" 2024-09-18T08:03:55.801 INFO:journalctl@ceph.grafana.a.smithi050.stdout:Sep 18 08:03:55 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-grafana-a[60403]: logger=migrator t=2024-09-18T08:03:55.65232912Z level=info msg="Migration successfully executed" id="add index annotation 1 v3" duration=541.282µs 2024-09-18T08:03:55.801 INFO:journalctl@ceph.grafana.a.smithi050.stdout:Sep 18 08:03:55 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-grafana-a[60403]: logger=migrator t=2024-09-18T08:03:55.652655305Z level=info msg="Executing migration" id="add index annotation 2 v3" 2024-09-18T08:03:55.801 INFO:journalctl@ceph.grafana.a.smithi050.stdout:Sep 18 08:03:55 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-grafana-a[60403]: logger=migrator t=2024-09-18T08:03:55.653206603Z level=info msg="Migration successfully executed" id="add index annotation 2 v3" duration=551.09µs 2024-09-18T08:03:55.802 INFO:journalctl@ceph.grafana.a.smithi050.stdout:Sep 18 08:03:55 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-grafana-a[60403]: logger=migrator t=2024-09-18T08:03:55.65353126Z level=info msg="Executing migration" id="add index annotation 3 v3" 2024-09-18T08:03:55.802 INFO:journalctl@ceph.grafana.a.smithi050.stdout:Sep 18 08:03:55 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-grafana-a[60403]: logger=migrator t=2024-09-18T08:03:55.654140286Z level=info msg="Migration successfully executed" id="add index annotation 3 v3" duration=608.644µs 2024-09-18T08:03:55.802 INFO:journalctl@ceph.grafana.a.smithi050.stdout:Sep 18 08:03:55 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-grafana-a[60403]: logger=migrator t=2024-09-18T08:03:55.654473347Z level=info msg="Executing migration" id="add index annotation 4 v3" 2024-09-18T08:03:55.802 INFO:journalctl@ceph.grafana.a.smithi050.stdout:Sep 18 08:03:55 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-grafana-a[60403]: logger=migrator t=2024-09-18T08:03:55.655103376Z level=info msg="Migration successfully executed" id="add index annotation 4 v3" duration=629.763µs 2024-09-18T08:03:55.802 INFO:journalctl@ceph.grafana.a.smithi050.stdout:Sep 18 08:03:55 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-grafana-a[60403]: logger=migrator t=2024-09-18T08:03:55.655445889Z level=info msg="Executing migration" id="Update annotation table charset" 2024-09-18T08:03:55.802 INFO:journalctl@ceph.grafana.a.smithi050.stdout:Sep 18 08:03:55 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-grafana-a[60403]: logger=migrator t=2024-09-18T08:03:55.655466227Z level=info msg="Migration successfully executed" id="Update annotation table charset" duration=20.487µs 2024-09-18T08:03:55.802 INFO:journalctl@ceph.grafana.a.smithi050.stdout:Sep 18 08:03:55 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-grafana-a[60403]: logger=migrator t=2024-09-18T08:03:55.655846589Z level=info msg="Executing migration" id="Add column region_id to annotation table" 2024-09-18T08:03:55.802 INFO:journalctl@ceph.grafana.a.smithi050.stdout:Sep 18 08:03:55 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-grafana-a[60403]: logger=migrator t=2024-09-18T08:03:55.658490091Z level=info msg="Migration successfully executed" id="Add column region_id to annotation table" duration=2.643196ms 2024-09-18T08:03:55.802 INFO:journalctl@ceph.grafana.a.smithi050.stdout:Sep 18 08:03:55 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-grafana-a[60403]: logger=migrator t=2024-09-18T08:03:55.658835616Z level=info msg="Executing migration" id="Drop category_id index" 2024-09-18T08:03:55.802 INFO:journalctl@ceph.grafana.a.smithi050.stdout:Sep 18 08:03:55 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-grafana-a[60403]: logger=migrator t=2024-09-18T08:03:55.65943408Z level=info msg="Migration successfully executed" id="Drop category_id index" duration=598.289µs 2024-09-18T08:03:55.802 INFO:journalctl@ceph.grafana.a.smithi050.stdout:Sep 18 08:03:55 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-grafana-a[60403]: logger=migrator t=2024-09-18T08:03:55.659750431Z level=info msg="Executing migration" id="Add column tags to annotation table" 2024-09-18T08:03:55.802 INFO:journalctl@ceph.grafana.a.smithi050.stdout:Sep 18 08:03:55 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-grafana-a[60403]: logger=migrator t=2024-09-18T08:03:55.662201112Z level=info msg="Migration successfully executed" id="Add column tags to annotation table" duration=2.450269ms 2024-09-18T08:03:55.803 INFO:journalctl@ceph.grafana.a.smithi050.stdout:Sep 18 08:03:55 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-grafana-a[60403]: logger=migrator t=2024-09-18T08:03:55.662533443Z level=info msg="Executing migration" id="Create annotation_tag table v2" 2024-09-18T08:03:55.803 INFO:journalctl@ceph.grafana.a.smithi050.stdout:Sep 18 08:03:55 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-grafana-a[60403]: logger=migrator t=2024-09-18T08:03:55.663007832Z level=info msg="Migration successfully executed" id="Create annotation_tag table v2" duration=474.233µs 2024-09-18T08:03:55.803 INFO:journalctl@ceph.grafana.a.smithi050.stdout:Sep 18 08:03:55 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-grafana-a[60403]: logger=migrator t=2024-09-18T08:03:55.663328707Z level=info msg="Executing migration" id="Add unique index annotation_tag.annotation_id_tag_id" 2024-09-18T08:03:55.803 INFO:journalctl@ceph.grafana.a.smithi050.stdout:Sep 18 08:03:55 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-grafana-a[60403]: logger=migrator t=2024-09-18T08:03:55.663887165Z level=info msg="Migration successfully executed" id="Add unique index annotation_tag.annotation_id_tag_id" duration=558.204µs 2024-09-18T08:03:55.803 INFO:journalctl@ceph.grafana.a.smithi050.stdout:Sep 18 08:03:55 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-grafana-a[60403]: logger=migrator t=2024-09-18T08:03:55.664221276Z level=info msg="Executing migration" id="drop index UQE_annotation_tag_annotation_id_tag_id - v2" 2024-09-18T08:03:55.803 INFO:journalctl@ceph.grafana.a.smithi050.stdout:Sep 18 08:03:55 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-grafana-a[60403]: logger=migrator t=2024-09-18T08:03:55.664772582Z level=info msg="Migration successfully executed" id="drop index UQE_annotation_tag_annotation_id_tag_id - v2" duration=551.226µs 2024-09-18T08:03:55.803 INFO:journalctl@ceph.grafana.a.smithi050.stdout:Sep 18 08:03:55 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-grafana-a[60403]: logger=migrator t=2024-09-18T08:03:55.665117244Z level=info msg="Executing migration" id="Rename table annotation_tag to annotation_tag_v2 - v2" 2024-09-18T08:03:55.803 INFO:journalctl@ceph.grafana.a.smithi050.stdout:Sep 18 08:03:55 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-grafana-a[60403]: logger=migrator t=2024-09-18T08:03:55.672098705Z level=info msg="Migration successfully executed" id="Rename table annotation_tag to annotation_tag_v2 - v2" duration=6.981274ms 2024-09-18T08:03:55.803 INFO:journalctl@ceph.grafana.a.smithi050.stdout:Sep 18 08:03:55 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-grafana-a[60403]: logger=migrator t=2024-09-18T08:03:55.672425503Z level=info msg="Executing migration" id="Create annotation_tag table v3" 2024-09-18T08:03:55.803 INFO:journalctl@ceph.grafana.a.smithi050.stdout:Sep 18 08:03:55 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-grafana-a[60403]: logger=migrator t=2024-09-18T08:03:55.67289488Z level=info msg="Migration successfully executed" id="Create annotation_tag table v3" duration=469.106µs 2024-09-18T08:03:55.803 INFO:journalctl@ceph.grafana.a.smithi050.stdout:Sep 18 08:03:55 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-grafana-a[60403]: logger=migrator t=2024-09-18T08:03:55.673224634Z level=info msg="Executing migration" id="create index UQE_annotation_tag_annotation_id_tag_id - Add unique index annotation_tag.annotation_id_tag_id V3" 2024-09-18T08:03:55.803 INFO:journalctl@ceph.grafana.a.smithi050.stdout:Sep 18 08:03:55 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-grafana-a[60403]: logger=migrator t=2024-09-18T08:03:55.673777699Z level=info msg="Migration successfully executed" id="create index UQE_annotation_tag_annotation_id_tag_id - Add unique index annotation_tag.annotation_id_tag_id V3" duration=552.938µs 2024-09-18T08:03:55.803 INFO:journalctl@ceph.grafana.a.smithi050.stdout:Sep 18 08:03:55 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-grafana-a[60403]: logger=migrator t=2024-09-18T08:03:55.674117201Z level=info msg="Executing migration" id="copy annotation_tag v2 to v3" 2024-09-18T08:03:55.804 INFO:journalctl@ceph.grafana.a.smithi050.stdout:Sep 18 08:03:55 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-grafana-a[60403]: logger=migrator t=2024-09-18T08:03:55.674325999Z level=info msg="Migration successfully executed" id="copy annotation_tag v2 to v3" duration=208.755µs 2024-09-18T08:03:55.804 INFO:journalctl@ceph.grafana.a.smithi050.stdout:Sep 18 08:03:55 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-grafana-a[60403]: logger=migrator t=2024-09-18T08:03:55.674628439Z level=info msg="Executing migration" id="drop table annotation_tag_v2" 2024-09-18T08:03:55.804 INFO:journalctl@ceph.grafana.a.smithi050.stdout:Sep 18 08:03:55 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-grafana-a[60403]: logger=migrator t=2024-09-18T08:03:55.675027854Z level=info msg="Migration successfully executed" id="drop table annotation_tag_v2" duration=399.357µs 2024-09-18T08:03:55.804 INFO:journalctl@ceph.grafana.a.smithi050.stdout:Sep 18 08:03:55 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-grafana-a[60403]: logger=migrator t=2024-09-18T08:03:55.675342133Z level=info msg="Executing migration" id="Update alert annotations and set TEXT to empty" 2024-09-18T08:03:55.804 INFO:journalctl@ceph.grafana.a.smithi050.stdout:Sep 18 08:03:55 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-grafana-a[60403]: logger=migrator t=2024-09-18T08:03:55.675465054Z level=info msg="Migration successfully executed" id="Update alert annotations and set TEXT to empty" duration=123.909µs 2024-09-18T08:03:55.804 INFO:journalctl@ceph.grafana.a.smithi050.stdout:Sep 18 08:03:55 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-grafana-a[60403]: logger=migrator t=2024-09-18T08:03:55.675822201Z level=info msg="Executing migration" id="Add created time to annotation table" 2024-09-18T08:03:55.804 INFO:journalctl@ceph.grafana.a.smithi050.stdout:Sep 18 08:03:55 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-grafana-a[60403]: logger=migrator t=2024-09-18T08:03:55.678298094Z level=info msg="Migration successfully executed" id="Add created time to annotation table" duration=2.47557ms 2024-09-18T08:03:55.804 INFO:journalctl@ceph.grafana.a.smithi050.stdout:Sep 18 08:03:55 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-grafana-a[60403]: logger=migrator t=2024-09-18T08:03:55.678627312Z level=info msg="Executing migration" id="Add updated time to annotation table" 2024-09-18T08:03:55.804 INFO:journalctl@ceph.grafana.a.smithi050.stdout:Sep 18 08:03:55 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-grafana-a[60403]: logger=migrator t=2024-09-18T08:03:55.68111873Z level=info msg="Migration successfully executed" id="Add updated time to annotation table" duration=2.487718ms 2024-09-18T08:03:55.804 INFO:journalctl@ceph.grafana.a.smithi050.stdout:Sep 18 08:03:55 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-grafana-a[60403]: logger=migrator t=2024-09-18T08:03:55.681446746Z level=info msg="Executing migration" id="Add index for created in annotation table" 2024-09-18T08:03:55.804 INFO:journalctl@ceph.grafana.a.smithi050.stdout:Sep 18 08:03:55 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-grafana-a[60403]: logger=migrator t=2024-09-18T08:03:55.682184859Z level=info msg="Migration successfully executed" id="Add index for created in annotation table" duration=737.853µs 2024-09-18T08:03:55.804 INFO:journalctl@ceph.grafana.a.smithi050.stdout:Sep 18 08:03:55 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-grafana-a[60403]: logger=migrator t=2024-09-18T08:03:55.6825404Z level=info msg="Executing migration" id="Add index for updated in annotation table" 2024-09-18T08:03:55.804 INFO:journalctl@ceph.grafana.a.smithi050.stdout:Sep 18 08:03:55 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-grafana-a[60403]: logger=migrator t=2024-09-18T08:03:55.683129106Z level=info msg="Migration successfully executed" id="Add index for updated in annotation table" duration=588.458µs 2024-09-18T08:03:55.805 INFO:journalctl@ceph.grafana.a.smithi050.stdout:Sep 18 08:03:55 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-grafana-a[60403]: logger=migrator t=2024-09-18T08:03:55.68345937Z level=info msg="Executing migration" id="Convert existing annotations from seconds to milliseconds" 2024-09-18T08:03:55.805 INFO:journalctl@ceph.grafana.a.smithi050.stdout:Sep 18 08:03:55 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-grafana-a[60403]: logger=migrator t=2024-09-18T08:03:55.68362381Z level=info msg="Migration successfully executed" id="Convert existing annotations from seconds to milliseconds" duration=164.363µs 2024-09-18T08:03:55.805 INFO:journalctl@ceph.grafana.a.smithi050.stdout:Sep 18 08:03:55 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-grafana-a[60403]: logger=migrator t=2024-09-18T08:03:55.684435353Z level=info msg="Executing migration" id="Add epoch_end column" 2024-09-18T08:03:55.805 INFO:journalctl@ceph.grafana.a.smithi050.stdout:Sep 18 08:03:55 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-grafana-a[60403]: logger=migrator t=2024-09-18T08:03:55.687003131Z level=info msg="Migration successfully executed" id="Add epoch_end column" duration=2.568086ms 2024-09-18T08:03:55.805 INFO:journalctl@ceph.grafana.a.smithi050.stdout:Sep 18 08:03:55 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-grafana-a[60403]: logger=migrator t=2024-09-18T08:03:55.687333559Z level=info msg="Executing migration" id="Add index for epoch_end" 2024-09-18T08:03:55.805 INFO:journalctl@ceph.grafana.a.smithi050.stdout:Sep 18 08:03:55 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-grafana-a[60403]: logger=migrator t=2024-09-18T08:03:55.687909059Z level=info msg="Migration successfully executed" id="Add index for epoch_end" duration=574.368µs 2024-09-18T08:03:55.805 INFO:journalctl@ceph.grafana.a.smithi050.stdout:Sep 18 08:03:55 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-grafana-a[60403]: logger=migrator t=2024-09-18T08:03:55.688234913Z level=info msg="Executing migration" id="Make epoch_end the same as epoch" 2024-09-18T08:03:55.805 INFO:journalctl@ceph.grafana.a.smithi050.stdout:Sep 18 08:03:55 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-grafana-a[60403]: logger=migrator t=2024-09-18T08:03:55.688354945Z level=info msg="Migration successfully executed" id="Make epoch_end the same as epoch" duration=120.877µs 2024-09-18T08:03:55.805 INFO:journalctl@ceph.grafana.a.smithi050.stdout:Sep 18 08:03:55 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-grafana-a[60403]: logger=migrator t=2024-09-18T08:03:55.688724197Z level=info msg="Executing migration" id="Move region to single row" 2024-09-18T08:03:55.805 INFO:journalctl@ceph.grafana.a.smithi050.stdout:Sep 18 08:03:55 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-grafana-a[60403]: logger=migrator t=2024-09-18T08:03:55.68896698Z level=info msg="Migration successfully executed" id="Move region to single row" duration=242.745µs 2024-09-18T08:03:55.805 INFO:journalctl@ceph.grafana.a.smithi050.stdout:Sep 18 08:03:55 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-grafana-a[60403]: logger=migrator t=2024-09-18T08:03:55.689340286Z level=info msg="Executing migration" id="Remove index org_id_epoch from annotation table" 2024-09-18T08:03:55.805 INFO:journalctl@ceph.grafana.a.smithi050.stdout:Sep 18 08:03:55 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-grafana-a[60403]: logger=migrator t=2024-09-18T08:03:55.689950413Z level=info msg="Migration successfully executed" id="Remove index org_id_epoch from annotation table" duration=610.835µs 2024-09-18T08:03:55.806 INFO:journalctl@ceph.grafana.a.smithi050.stdout:Sep 18 08:03:55 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-grafana-a[60403]: logger=migrator t=2024-09-18T08:03:55.690287768Z level=info msg="Executing migration" id="Remove index org_id_dashboard_id_panel_id_epoch from annotation table" 2024-09-18T08:03:55.806 INFO:journalctl@ceph.grafana.a.smithi050.stdout:Sep 18 08:03:55 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-grafana-a[60403]: logger=migrator t=2024-09-18T08:03:55.69085455Z level=info msg="Migration successfully executed" id="Remove index org_id_dashboard_id_panel_id_epoch from annotation table" duration=568.84µs 2024-09-18T08:03:55.806 INFO:journalctl@ceph.grafana.a.smithi050.stdout:Sep 18 08:03:55 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-grafana-a[60403]: logger=migrator t=2024-09-18T08:03:55.691205841Z level=info msg="Executing migration" id="Add index for org_id_dashboard_id_epoch_end_epoch on annotation table" 2024-09-18T08:03:55.806 INFO:journalctl@ceph.grafana.a.smithi050.stdout:Sep 18 08:03:55 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-grafana-a[60403]: logger=migrator t=2024-09-18T08:03:55.691803773Z level=info msg="Migration successfully executed" id="Add index for org_id_dashboard_id_epoch_end_epoch on annotation table" duration=598.638µs 2024-09-18T08:03:55.806 INFO:journalctl@ceph.grafana.a.smithi050.stdout:Sep 18 08:03:55 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-grafana-a[60403]: logger=migrator t=2024-09-18T08:03:55.69225276Z level=info msg="Executing migration" id="Add index for org_id_epoch_end_epoch on annotation table" 2024-09-18T08:03:55.806 INFO:journalctl@ceph.grafana.a.smithi050.stdout:Sep 18 08:03:55 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-grafana-a[60403]: logger=migrator t=2024-09-18T08:03:55.692826649Z level=info msg="Migration successfully executed" id="Add index for org_id_epoch_end_epoch on annotation table" duration=573.408µs 2024-09-18T08:03:55.806 INFO:journalctl@ceph.grafana.a.smithi050.stdout:Sep 18 08:03:55 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-grafana-a[60403]: logger=migrator t=2024-09-18T08:03:55.693183908Z level=info msg="Executing migration" id="Remove index org_id_epoch_epoch_end from annotation table" 2024-09-18T08:03:55.806 INFO:journalctl@ceph.grafana.a.smithi050.stdout:Sep 18 08:03:55 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-grafana-a[60403]: logger=migrator t=2024-09-18T08:03:55.693752596Z level=info msg="Migration successfully executed" id="Remove index org_id_epoch_epoch_end from annotation table" duration=568.61µs 2024-09-18T08:03:55.806 INFO:journalctl@ceph.grafana.a.smithi050.stdout:Sep 18 08:03:55 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-grafana-a[60403]: logger=migrator t=2024-09-18T08:03:55.694087997Z level=info msg="Executing migration" id="Add index for alert_id on annotation table" 2024-09-18T08:03:55.806 INFO:journalctl@ceph.grafana.a.smithi050.stdout:Sep 18 08:03:55 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-grafana-a[60403]: logger=migrator t=2024-09-18T08:03:55.694645687Z level=info msg="Migration successfully executed" id="Add index for alert_id on annotation table" duration=557.502µs 2024-09-18T08:03:55.806 INFO:journalctl@ceph.grafana.a.smithi050.stdout:Sep 18 08:03:55 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-grafana-a[60403]: logger=migrator t=2024-09-18T08:03:55.69499054Z level=info msg="Executing migration" id="Increase tags column to length 4096" 2024-09-18T08:03:55.807 INFO:journalctl@ceph.grafana.a.smithi050.stdout:Sep 18 08:03:55 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-grafana-a[60403]: logger=migrator t=2024-09-18T08:03:55.695046717Z level=info msg="Migration successfully executed" id="Increase tags column to length 4096" duration=51.737µs 2024-09-18T08:03:55.807 INFO:journalctl@ceph.grafana.a.smithi050.stdout:Sep 18 08:03:55 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-grafana-a[60403]: logger=migrator t=2024-09-18T08:03:55.695468717Z level=info msg="Executing migration" id="create test_data table" 2024-09-18T08:03:55.807 INFO:journalctl@ceph.grafana.a.smithi050.stdout:Sep 18 08:03:55 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-grafana-a[60403]: logger=migrator t=2024-09-18T08:03:55.696027818Z level=info msg="Migration successfully executed" id="create test_data table" duration=558.87µs 2024-09-18T08:03:55.807 INFO:journalctl@ceph.grafana.a.smithi050.stdout:Sep 18 08:03:55 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-grafana-a[60403]: logger=migrator t=2024-09-18T08:03:55.696376116Z level=info msg="Executing migration" id="create dashboard_version table v1" 2024-09-18T08:03:55.807 INFO:journalctl@ceph.grafana.a.smithi050.stdout:Sep 18 08:03:55 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-grafana-a[60403]: logger=migrator t=2024-09-18T08:03:55.696931383Z level=info msg="Migration successfully executed" id="create dashboard_version table v1" duration=555.187µs 2024-09-18T08:03:55.807 INFO:journalctl@ceph.grafana.a.smithi050.stdout:Sep 18 08:03:55 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-grafana-a[60403]: logger=migrator t=2024-09-18T08:03:55.697268853Z level=info msg="Executing migration" id="add index dashboard_version.dashboard_id" 2024-09-18T08:03:55.807 INFO:journalctl@ceph.grafana.a.smithi050.stdout:Sep 18 08:03:55 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-grafana-a[60403]: logger=migrator t=2024-09-18T08:03:55.697813085Z level=info msg="Migration successfully executed" id="add index dashboard_version.dashboard_id" duration=544.13µs 2024-09-18T08:03:55.807 INFO:journalctl@ceph.grafana.a.smithi050.stdout:Sep 18 08:03:55 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-grafana-a[60403]: logger=migrator t=2024-09-18T08:03:55.69815394Z level=info msg="Executing migration" id="add unique index dashboard_version.dashboard_id and dashboard_version.version" 2024-09-18T08:03:55.807 INFO:journalctl@ceph.grafana.a.smithi050.stdout:Sep 18 08:03:55 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-grafana-a[60403]: logger=migrator t=2024-09-18T08:03:55.698739718Z level=info msg="Migration successfully executed" id="add unique index dashboard_version.dashboard_id and dashboard_version.version" duration=585.443µs 2024-09-18T08:03:55.807 INFO:journalctl@ceph.grafana.a.smithi050.stdout:Sep 18 08:03:55 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-grafana-a[60403]: logger=migrator t=2024-09-18T08:03:55.699094167Z level=info msg="Executing migration" id="Set dashboard version to 1 where 0" 2024-09-18T08:03:55.807 INFO:journalctl@ceph.grafana.a.smithi050.stdout:Sep 18 08:03:55 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-grafana-a[60403]: logger=migrator t=2024-09-18T08:03:55.699225184Z level=info msg="Migration successfully executed" id="Set dashboard version to 1 where 0" duration=131.12µs 2024-09-18T08:03:55.807 INFO:journalctl@ceph.grafana.a.smithi050.stdout:Sep 18 08:03:55 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-grafana-a[60403]: logger=migrator t=2024-09-18T08:03:55.699579974Z level=info msg="Executing migration" id="save existing dashboard data in dashboard_version table v1" 2024-09-18T08:03:55.808 INFO:journalctl@ceph.grafana.a.smithi050.stdout:Sep 18 08:03:55 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-grafana-a[60403]: logger=migrator t=2024-09-18T08:03:55.699829709Z level=info msg="Migration successfully executed" id="save existing dashboard data in dashboard_version table v1" duration=249.562µs 2024-09-18T08:03:55.808 INFO:journalctl@ceph.grafana.a.smithi050.stdout:Sep 18 08:03:55 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-grafana-a[60403]: logger=migrator t=2024-09-18T08:03:55.700146395Z level=info msg="Executing migration" id="alter dashboard_version.data to mediumtext v1" 2024-09-18T08:03:55.808 INFO:journalctl@ceph.grafana.a.smithi050.stdout:Sep 18 08:03:55 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-grafana-a[60403]: logger=migrator t=2024-09-18T08:03:55.700197992Z level=info msg="Migration successfully executed" id="alter dashboard_version.data to mediumtext v1" duration=51.678µs 2024-09-18T08:03:55.808 INFO:journalctl@ceph.grafana.a.smithi050.stdout:Sep 18 08:03:55 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-grafana-a[60403]: logger=migrator t=2024-09-18T08:03:55.700589518Z level=info msg="Executing migration" id="create team table" 2024-09-18T08:03:55.808 INFO:journalctl@ceph.grafana.a.smithi050.stdout:Sep 18 08:03:55 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-grafana-a[60403]: logger=migrator t=2024-09-18T08:03:55.701096352Z level=info msg="Migration successfully executed" id="create team table" duration=502.255µs 2024-09-18T08:03:55.808 INFO:journalctl@ceph.grafana.a.smithi050.stdout:Sep 18 08:03:55 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-grafana-a[60403]: logger=migrator t=2024-09-18T08:03:55.701425357Z level=info msg="Executing migration" id="add index team.org_id" 2024-09-18T08:03:55.808 INFO:journalctl@ceph.grafana.a.smithi050.stdout:Sep 18 08:03:55 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-grafana-a[60403]: logger=migrator t=2024-09-18T08:03:55.702126187Z level=info msg="Migration successfully executed" id="add index team.org_id" duration=701.1µs 2024-09-18T08:03:55.808 INFO:journalctl@ceph.grafana.a.smithi050.stdout:Sep 18 08:03:55 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-grafana-a[60403]: logger=migrator t=2024-09-18T08:03:55.702529066Z level=info msg="Executing migration" id="add unique index team_org_id_name" 2024-09-18T08:03:55.808 INFO:journalctl@ceph.grafana.a.smithi050.stdout:Sep 18 08:03:55 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-grafana-a[60403]: logger=migrator t=2024-09-18T08:03:55.703150393Z level=info msg="Migration successfully executed" id="add unique index team_org_id_name" duration=620.26µs 2024-09-18T08:03:55.808 INFO:journalctl@ceph.grafana.a.smithi050.stdout:Sep 18 08:03:55 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-grafana-a[60403]: logger=migrator t=2024-09-18T08:03:55.703496094Z level=info msg="Executing migration" id="Add column uid in team" 2024-09-18T08:03:55.809 INFO:journalctl@ceph.grafana.a.smithi050.stdout:Sep 18 08:03:55 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-grafana-a[60403]: logger=migrator t=2024-09-18T08:03:55.706224498Z level=info msg="Migration successfully executed" id="Add column uid in team" duration=2.728365ms 2024-09-18T08:03:55.809 INFO:journalctl@ceph.grafana.a.smithi050.stdout:Sep 18 08:03:55 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-grafana-a[60403]: logger=migrator t=2024-09-18T08:03:55.706553138Z level=info msg="Executing migration" id="Update uid column values in team" 2024-09-18T08:03:55.809 INFO:journalctl@ceph.grafana.a.smithi050.stdout:Sep 18 08:03:55 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-grafana-a[60403]: logger=migrator t=2024-09-18T08:03:55.706666215Z level=info msg="Migration successfully executed" id="Update uid column values in team" duration=113.297µs 2024-09-18T08:03:55.809 INFO:journalctl@ceph.grafana.a.smithi050.stdout:Sep 18 08:03:55 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-grafana-a[60403]: logger=migrator t=2024-09-18T08:03:55.707026944Z level=info msg="Executing migration" id="Add unique index team_org_id_uid" 2024-09-18T08:03:55.809 INFO:journalctl@ceph.grafana.a.smithi050.stdout:Sep 18 08:03:55 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-grafana-a[60403]: logger=migrator t=2024-09-18T08:03:55.707583498Z level=info msg="Migration successfully executed" id="Add unique index team_org_id_uid" duration=556.198µs 2024-09-18T08:03:55.809 INFO:journalctl@ceph.grafana.a.smithi050.stdout:Sep 18 08:03:55 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-grafana-a[60403]: logger=migrator t=2024-09-18T08:03:55.707926065Z level=info msg="Executing migration" id="create team member table" 2024-09-18T08:03:55.809 INFO:journalctl@ceph.grafana.a.smithi050.stdout:Sep 18 08:03:55 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-grafana-a[60403]: logger=migrator t=2024-09-18T08:03:55.708423885Z level=info msg="Migration successfully executed" id="create team member table" duration=497.67µs 2024-09-18T08:03:55.809 INFO:journalctl@ceph.grafana.a.smithi050.stdout:Sep 18 08:03:55 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-grafana-a[60403]: logger=migrator t=2024-09-18T08:03:55.708753932Z level=info msg="Executing migration" id="add index team_member.org_id" 2024-09-18T08:03:55.810 INFO:journalctl@ceph.grafana.a.smithi050.stdout:Sep 18 08:03:55 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-grafana-a[60403]: logger=migrator t=2024-09-18T08:03:55.709302209Z level=info msg="Migration successfully executed" id="add index team_member.org_id" duration=543.723µs 2024-09-18T08:03:55.810 INFO:journalctl@ceph.grafana.a.smithi050.stdout:Sep 18 08:03:55 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-grafana-a[60403]: logger=migrator t=2024-09-18T08:03:55.709632419Z level=info msg="Executing migration" id="add unique index team_member_org_id_team_id_user_id" 2024-09-18T08:03:55.810 INFO:journalctl@ceph.grafana.a.smithi050.stdout:Sep 18 08:03:55 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-grafana-a[60403]: logger=migrator t=2024-09-18T08:03:55.710223834Z level=info msg="Migration successfully executed" id="add unique index team_member_org_id_team_id_user_id" duration=591.033µs 2024-09-18T08:03:55.810 INFO:journalctl@ceph.grafana.a.smithi050.stdout:Sep 18 08:03:55 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-grafana-a[60403]: logger=migrator t=2024-09-18T08:03:55.710559043Z level=info msg="Executing migration" id="add index team_member.team_id" 2024-09-18T08:03:55.810 INFO:journalctl@ceph.grafana.a.smithi050.stdout:Sep 18 08:03:55 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-grafana-a[60403]: logger=migrator t=2024-09-18T08:03:55.711143525Z level=info msg="Migration successfully executed" id="add index team_member.team_id" duration=584.512µs 2024-09-18T08:03:55.810 INFO:journalctl@ceph.grafana.a.smithi050.stdout:Sep 18 08:03:55 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-grafana-a[60403]: logger=migrator t=2024-09-18T08:03:55.711492899Z level=info msg="Executing migration" id="Add column email to team table" 2024-09-18T08:03:55.810 INFO:journalctl@ceph.grafana.a.smithi050.stdout:Sep 18 08:03:55 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-grafana-a[60403]: logger=migrator t=2024-09-18T08:03:55.714474205Z level=info msg="Migration successfully executed" id="Add column email to team table" duration=2.980958ms 2024-09-18T08:03:55.810 INFO:journalctl@ceph.grafana.a.smithi050.stdout:Sep 18 08:03:55 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-grafana-a[60403]: logger=migrator t=2024-09-18T08:03:55.714804707Z level=info msg="Executing migration" id="Add column external to team_member table" 2024-09-18T08:03:55.810 INFO:journalctl@ceph.grafana.a.smithi050.stdout:Sep 18 08:03:55 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-grafana-a[60403]: logger=migrator t=2024-09-18T08:03:55.717928064Z level=info msg="Migration successfully executed" id="Add column external to team_member table" duration=3.123236ms 2024-09-18T08:03:55.810 INFO:journalctl@ceph.grafana.a.smithi050.stdout:Sep 18 08:03:55 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-grafana-a[60403]: logger=migrator t=2024-09-18T08:03:55.718241505Z level=info msg="Executing migration" id="Add column permission to team_member table" 2024-09-18T08:03:55.810 INFO:journalctl@ceph.grafana.a.smithi050.stdout:Sep 18 08:03:55 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-grafana-a[60403]: logger=migrator t=2024-09-18T08:03:55.721056654Z level=info msg="Migration successfully executed" id="Add column permission to team_member table" duration=2.81505ms 2024-09-18T08:03:55.810 INFO:journalctl@ceph.grafana.a.smithi050.stdout:Sep 18 08:03:55 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-grafana-a[60403]: logger=migrator t=2024-09-18T08:03:55.7214069Z level=info msg="Executing migration" id="create dashboard acl table" 2024-09-18T08:03:55.810 INFO:journalctl@ceph.grafana.a.smithi050.stdout:Sep 18 08:03:55 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-grafana-a[60403]: logger=migrator t=2024-09-18T08:03:55.722063963Z level=info msg="Migration successfully executed" id="create dashboard acl table" duration=656.929µs 2024-09-18T08:03:55.811 INFO:journalctl@ceph.grafana.a.smithi050.stdout:Sep 18 08:03:55 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-grafana-a[60403]: logger=migrator t=2024-09-18T08:03:55.722445399Z level=info msg="Executing migration" id="add index dashboard_acl_dashboard_id" 2024-09-18T08:03:55.811 INFO:journalctl@ceph.grafana.a.smithi050.stdout:Sep 18 08:03:55 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-grafana-a[60403]: logger=migrator t=2024-09-18T08:03:55.723062496Z level=info msg="Migration successfully executed" id="add index dashboard_acl_dashboard_id" duration=617.009µs 2024-09-18T08:03:55.811 INFO:journalctl@ceph.grafana.a.smithi050.stdout:Sep 18 08:03:55 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-grafana-a[60403]: logger=migrator t=2024-09-18T08:03:55.72341414Z level=info msg="Executing migration" id="add unique index dashboard_acl_dashboard_id_user_id" 2024-09-18T08:03:55.811 INFO:journalctl@ceph.grafana.a.smithi050.stdout:Sep 18 08:03:55 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-grafana-a[60403]: logger=migrator t=2024-09-18T08:03:55.724120053Z level=info msg="Migration successfully executed" id="add unique index dashboard_acl_dashboard_id_user_id" duration=705.371µs 2024-09-18T08:03:55.811 INFO:journalctl@ceph.grafana.a.smithi050.stdout:Sep 18 08:03:55 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-grafana-a[60403]: logger=migrator t=2024-09-18T08:03:55.724518788Z level=info msg="Executing migration" id="add unique index dashboard_acl_dashboard_id_team_id" 2024-09-18T08:03:55.811 INFO:journalctl@ceph.grafana.a.smithi050.stdout:Sep 18 08:03:55 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-grafana-a[60403]: logger=migrator t=2024-09-18T08:03:55.725144579Z level=info msg="Migration successfully executed" id="add unique index dashboard_acl_dashboard_id_team_id" duration=625.494µs 2024-09-18T08:03:55.811 INFO:journalctl@ceph.grafana.a.smithi050.stdout:Sep 18 08:03:55 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-grafana-a[60403]: logger=migrator t=2024-09-18T08:03:55.725493595Z level=info msg="Executing migration" id="add index dashboard_acl_user_id" 2024-09-18T08:03:55.811 INFO:journalctl@ceph.grafana.a.smithi050.stdout:Sep 18 08:03:55 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-grafana-a[60403]: logger=migrator t=2024-09-18T08:03:55.726084185Z level=info msg="Migration successfully executed" id="add index dashboard_acl_user_id" duration=590.389µs 2024-09-18T08:03:55.811 INFO:journalctl@ceph.grafana.a.smithi050.stdout:Sep 18 08:03:55 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-grafana-a[60403]: logger=migrator t=2024-09-18T08:03:55.726434857Z level=info msg="Executing migration" id="add index dashboard_acl_team_id" 2024-09-18T08:03:55.811 INFO:journalctl@ceph.grafana.a.smithi050.stdout:Sep 18 08:03:55 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-grafana-a[60403]: logger=migrator t=2024-09-18T08:03:55.727046643Z level=info msg="Migration successfully executed" id="add index dashboard_acl_team_id" duration=611.634µs 2024-09-18T08:03:55.811 INFO:journalctl@ceph.grafana.a.smithi050.stdout:Sep 18 08:03:55 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-grafana-a[60403]: logger=migrator t=2024-09-18T08:03:55.727396851Z level=info msg="Executing migration" id="add index dashboard_acl_org_id_role" 2024-09-18T08:03:55.811 INFO:journalctl@ceph.grafana.a.smithi050.stdout:Sep 18 08:03:55 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-grafana-a[60403]: logger=migrator t=2024-09-18T08:03:55.728017388Z level=info msg="Migration successfully executed" id="add index dashboard_acl_org_id_role" duration=620.347µs 2024-09-18T08:03:55.811 INFO:journalctl@ceph.grafana.a.smithi050.stdout:Sep 18 08:03:55 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-grafana-a[60403]: logger=migrator t=2024-09-18T08:03:55.728369415Z level=info msg="Executing migration" id="add index dashboard_permission" 2024-09-18T08:03:55.812 INFO:journalctl@ceph.grafana.a.smithi050.stdout:Sep 18 08:03:55 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-grafana-a[60403]: logger=migrator t=2024-09-18T08:03:55.729000213Z level=info msg="Migration successfully executed" id="add index dashboard_permission" duration=630.744µs 2024-09-18T08:03:55.812 INFO:journalctl@ceph.grafana.a.smithi050.stdout:Sep 18 08:03:55 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-grafana-a[60403]: logger=migrator t=2024-09-18T08:03:55.729350727Z level=info msg="Executing migration" id="save default acl rules in dashboard_acl table" 2024-09-18T08:03:55.812 INFO:journalctl@ceph.grafana.a.smithi050.stdout:Sep 18 08:03:55 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-grafana-a[60403]: logger=migrator t=2024-09-18T08:03:55.729707638Z level=info msg="Migration successfully executed" id="save default acl rules in dashboard_acl table" duration=356.701µs 2024-09-18T08:03:55.812 INFO:journalctl@ceph.grafana.a.smithi050.stdout:Sep 18 08:03:55 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-grafana-a[60403]: logger=migrator t=2024-09-18T08:03:55.730120608Z level=info msg="Executing migration" id="delete acl rules for deleted dashboards and folders" 2024-09-18T08:03:55.812 INFO:journalctl@ceph.grafana.a.smithi050.stdout:Sep 18 08:03:55 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-grafana-a[60403]: logger=migrator t=2024-09-18T08:03:55.730279404Z level=info msg="Migration successfully executed" id="delete acl rules for deleted dashboards and folders" duration=158.901µs 2024-09-18T08:03:55.812 INFO:journalctl@ceph.grafana.a.smithi050.stdout:Sep 18 08:03:55 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-grafana-a[60403]: logger=migrator t=2024-09-18T08:03:55.730667438Z level=info msg="Executing migration" id="create tag table" 2024-09-18T08:03:55.812 INFO:journalctl@ceph.grafana.a.smithi050.stdout:Sep 18 08:03:55 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-grafana-a[60403]: logger=migrator t=2024-09-18T08:03:55.731216351Z level=info msg="Migration successfully executed" id="create tag table" duration=548.917µs 2024-09-18T08:03:55.812 INFO:journalctl@ceph.grafana.a.smithi050.stdout:Sep 18 08:03:55 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-grafana-a[60403]: logger=migrator t=2024-09-18T08:03:55.731572721Z level=info msg="Executing migration" id="add index tag.key_value" 2024-09-18T08:03:55.813 INFO:journalctl@ceph.grafana.a.smithi050.stdout:Sep 18 08:03:55 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-grafana-a[60403]: logger=migrator t=2024-09-18T08:03:55.732171112Z level=info msg="Migration successfully executed" id="add index tag.key_value" duration=598.196µs 2024-09-18T08:03:55.813 INFO:journalctl@ceph.grafana.a.smithi050.stdout:Sep 18 08:03:55 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-grafana-a[60403]: logger=migrator t=2024-09-18T08:03:55.732523158Z level=info msg="Executing migration" id="create login attempt table" 2024-09-18T08:03:55.813 INFO:journalctl@ceph.grafana.a.smithi050.stdout:Sep 18 08:03:55 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-grafana-a[60403]: logger=migrator t=2024-09-18T08:03:55.733020057Z level=info msg="Migration successfully executed" id="create login attempt table" duration=497.774µs 2024-09-18T08:03:55.813 INFO:journalctl@ceph.grafana.a.smithi050.stdout:Sep 18 08:03:55 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-grafana-a[60403]: logger=migrator t=2024-09-18T08:03:55.73337021Z level=info msg="Executing migration" id="add index login_attempt.username" 2024-09-18T08:03:55.813 INFO:journalctl@ceph.grafana.a.smithi050.stdout:Sep 18 08:03:55 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-grafana-a[60403]: logger=migrator t=2024-09-18T08:03:55.733969813Z level=info msg="Migration successfully executed" id="add index login_attempt.username" duration=599.405µs 2024-09-18T08:03:55.813 INFO:journalctl@ceph.grafana.a.smithi050.stdout:Sep 18 08:03:55 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-grafana-a[60403]: logger=migrator t=2024-09-18T08:03:55.734320666Z level=info msg="Executing migration" id="drop index IDX_login_attempt_username - v1" 2024-09-18T08:03:55.813 INFO:journalctl@ceph.grafana.a.smithi050.stdout:Sep 18 08:03:55 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-grafana-a[60403]: logger=migrator t=2024-09-18T08:03:55.73492056Z level=info msg="Migration successfully executed" id="drop index IDX_login_attempt_username - v1" duration=600.348µs 2024-09-18T08:03:55.813 INFO:journalctl@ceph.grafana.a.smithi050.stdout:Sep 18 08:03:55 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-grafana-a[60403]: logger=migrator t=2024-09-18T08:03:55.73526517Z level=info msg="Executing migration" id="Rename table login_attempt to login_attempt_tmp_qwerty - v1" 2024-09-18T08:03:55.813 INFO:journalctl@ceph.grafana.a.smithi050.stdout:Sep 18 08:03:55 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-grafana-a[60403]: logger=migrator t=2024-09-18T08:03:55.744158814Z level=info msg="Migration successfully executed" id="Rename table login_attempt to login_attempt_tmp_qwerty - v1" duration=8.893688ms 2024-09-18T08:03:55.813 INFO:journalctl@ceph.grafana.a.smithi050.stdout:Sep 18 08:03:55 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-grafana-a[60403]: logger=migrator t=2024-09-18T08:03:55.744512343Z level=info msg="Executing migration" id="create login_attempt v2" 2024-09-18T08:03:55.813 INFO:journalctl@ceph.grafana.a.smithi050.stdout:Sep 18 08:03:55 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-grafana-a[60403]: logger=migrator t=2024-09-18T08:03:55.745001837Z level=info msg="Migration successfully executed" id="create login_attempt v2" duration=489.987µs 2024-09-18T08:03:55.813 INFO:journalctl@ceph.grafana.a.smithi050.stdout:Sep 18 08:03:55 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-grafana-a[60403]: logger=migrator t=2024-09-18T08:03:55.745336965Z level=info msg="Executing migration" id="create index IDX_login_attempt_username - v2" 2024-09-18T08:03:55.813 INFO:journalctl@ceph.grafana.a.smithi050.stdout:Sep 18 08:03:55 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-grafana-a[60403]: logger=migrator t=2024-09-18T08:03:55.745925422Z level=info msg="Migration successfully executed" id="create index IDX_login_attempt_username - v2" duration=588.361µs 2024-09-18T08:03:55.813 INFO:journalctl@ceph.grafana.a.smithi050.stdout:Sep 18 08:03:55 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-grafana-a[60403]: logger=migrator t=2024-09-18T08:03:55.746265331Z level=info msg="Executing migration" id="copy login_attempt v1 to v2" 2024-09-18T08:03:55.814 INFO:journalctl@ceph.grafana.a.smithi050.stdout:Sep 18 08:03:55 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-grafana-a[60403]: logger=migrator t=2024-09-18T08:03:55.746479766Z level=info msg="Migration successfully executed" id="copy login_attempt v1 to v2" duration=215.098µs 2024-09-18T08:03:55.814 INFO:journalctl@ceph.grafana.a.smithi050.stdout:Sep 18 08:03:55 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-grafana-a[60403]: logger=migrator t=2024-09-18T08:03:55.746797292Z level=info msg="Executing migration" id="drop login_attempt_tmp_qwerty" 2024-09-18T08:03:55.814 INFO:journalctl@ceph.grafana.a.smithi050.stdout:Sep 18 08:03:55 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-grafana-a[60403]: logger=migrator t=2024-09-18T08:03:55.747235877Z level=info msg="Migration successfully executed" id="drop login_attempt_tmp_qwerty" duration=440.29µs 2024-09-18T08:03:55.814 INFO:journalctl@ceph.grafana.a.smithi050.stdout:Sep 18 08:03:55 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-grafana-a[60403]: logger=migrator t=2024-09-18T08:03:55.747566639Z level=info msg="Executing migration" id="create user auth table" 2024-09-18T08:03:55.814 INFO:journalctl@ceph.grafana.a.smithi050.stdout:Sep 18 08:03:55 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-grafana-a[60403]: logger=migrator t=2024-09-18T08:03:55.748051077Z level=info msg="Migration successfully executed" id="create user auth table" duration=485.029µs 2024-09-18T08:03:55.814 INFO:journalctl@ceph.grafana.a.smithi050.stdout:Sep 18 08:03:55 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-grafana-a[60403]: logger=migrator t=2024-09-18T08:03:55.748385339Z level=info msg="Executing migration" id="create index IDX_user_auth_auth_module_auth_id - v1" 2024-09-18T08:03:55.814 INFO:journalctl@ceph.grafana.a.smithi050.stdout:Sep 18 08:03:55 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-grafana-a[60403]: logger=migrator t=2024-09-18T08:03:55.748989444Z level=info msg="Migration successfully executed" id="create index IDX_user_auth_auth_module_auth_id - v1" duration=604.604µs 2024-09-18T08:03:55.814 INFO:journalctl@ceph.grafana.a.smithi050.stdout:Sep 18 08:03:55 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-grafana-a[60403]: logger=migrator t=2024-09-18T08:03:55.749306019Z level=info msg="Executing migration" id="alter user_auth.auth_id to length 190" 2024-09-18T08:03:55.814 INFO:journalctl@ceph.grafana.a.smithi050.stdout:Sep 18 08:03:55 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-grafana-a[60403]: logger=migrator t=2024-09-18T08:03:55.749349711Z level=info msg="Migration successfully executed" id="alter user_auth.auth_id to length 190" duration=44.637µs 2024-09-18T08:03:55.814 INFO:journalctl@ceph.grafana.a.smithi050.stdout:Sep 18 08:03:55 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-grafana-a[60403]: logger=migrator t=2024-09-18T08:03:55.749754101Z level=info msg="Executing migration" id="Add OAuth access token to user_auth" 2024-09-18T08:03:55.814 INFO:journalctl@ceph.grafana.a.smithi050.stdout:Sep 18 08:03:55 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-grafana-a[60403]: logger=migrator t=2024-09-18T08:03:55.752918175Z level=info msg="Migration successfully executed" id="Add OAuth access token to user_auth" duration=3.164215ms 2024-09-18T08:03:55.814 INFO:journalctl@ceph.grafana.a.smithi050.stdout:Sep 18 08:03:55 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-grafana-a[60403]: logger=migrator t=2024-09-18T08:03:55.753232743Z level=info msg="Executing migration" id="Add OAuth refresh token to user_auth" 2024-09-18T08:03:55.814 INFO:journalctl@ceph.grafana.a.smithi050.stdout:Sep 18 08:03:55 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-grafana-a[60403]: logger=migrator t=2024-09-18T08:03:55.756322739Z level=info msg="Migration successfully executed" id="Add OAuth refresh token to user_auth" duration=3.088693ms 2024-09-18T08:03:55.814 INFO:journalctl@ceph.grafana.a.smithi050.stdout:Sep 18 08:03:55 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-grafana-a[60403]: logger=migrator t=2024-09-18T08:03:55.756637055Z level=info msg="Executing migration" id="Add OAuth token type to user_auth" 2024-09-18T08:03:55.815 INFO:journalctl@ceph.grafana.a.smithi050.stdout:Sep 18 08:03:55 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-grafana-a[60403]: logger=migrator t=2024-09-18T08:03:55.759756356Z level=info msg="Migration successfully executed" id="Add OAuth token type to user_auth" duration=3.118909ms 2024-09-18T08:03:55.815 INFO:journalctl@ceph.grafana.a.smithi050.stdout:Sep 18 08:03:55 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-grafana-a[60403]: logger=migrator t=2024-09-18T08:03:55.760084902Z level=info msg="Executing migration" id="Add OAuth expiry to user_auth" 2024-09-18T08:03:55.815 INFO:journalctl@ceph.grafana.a.smithi050.stdout:Sep 18 08:03:55 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-grafana-a[60403]: logger=migrator t=2024-09-18T08:03:55.763282982Z level=info msg="Migration successfully executed" id="Add OAuth expiry to user_auth" duration=3.197773ms 2024-09-18T08:03:55.815 INFO:journalctl@ceph.grafana.a.smithi050.stdout:Sep 18 08:03:55 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-grafana-a[60403]: logger=migrator t=2024-09-18T08:03:55.763598696Z level=info msg="Executing migration" id="Add index to user_id column in user_auth" 2024-09-18T08:03:55.815 INFO:journalctl@ceph.grafana.a.smithi050.stdout:Sep 18 08:03:55 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-grafana-a[60403]: logger=migrator t=2024-09-18T08:03:55.764180632Z level=info msg="Migration successfully executed" id="Add index to user_id column in user_auth" duration=581.621µs 2024-09-18T08:03:55.815 INFO:journalctl@ceph.grafana.a.smithi050.stdout:Sep 18 08:03:55 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-grafana-a[60403]: logger=migrator t=2024-09-18T08:03:55.76449536Z level=info msg="Executing migration" id="Add OAuth ID token to user_auth" 2024-09-18T08:03:55.815 INFO:journalctl@ceph.grafana.a.smithi050.stdout:Sep 18 08:03:55 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-grafana-a[60403]: logger=migrator t=2024-09-18T08:03:55.767672773Z level=info msg="Migration successfully executed" id="Add OAuth ID token to user_auth" duration=3.177036ms 2024-09-18T08:03:55.815 INFO:journalctl@ceph.grafana.a.smithi050.stdout:Sep 18 08:03:55 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-grafana-a[60403]: logger=migrator t=2024-09-18T08:03:55.767993863Z level=info msg="Executing migration" id="create server_lock table" 2024-09-18T08:03:55.815 INFO:journalctl@ceph.grafana.a.smithi050.stdout:Sep 18 08:03:55 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-grafana-a[60403]: logger=migrator t=2024-09-18T08:03:55.768518142Z level=info msg="Migration successfully executed" id="create server_lock table" duration=525.884µs 2024-09-18T08:03:55.815 INFO:journalctl@ceph.grafana.a.smithi050.stdout:Sep 18 08:03:55 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-grafana-a[60403]: logger=migrator t=2024-09-18T08:03:55.768845025Z level=info msg="Executing migration" id="add index server_lock.operation_uid" 2024-09-18T08:03:55.815 INFO:journalctl@ceph.grafana.a.smithi050.stdout:Sep 18 08:03:55 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-grafana-a[60403]: logger=migrator t=2024-09-18T08:03:55.769432032Z level=info msg="Migration successfully executed" id="add index server_lock.operation_uid" duration=586.87µs 2024-09-18T08:03:55.815 INFO:journalctl@ceph.grafana.a.smithi050.stdout:Sep 18 08:03:55 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-grafana-a[60403]: logger=migrator t=2024-09-18T08:03:55.76976207Z level=info msg="Executing migration" id="create user auth token table" 2024-09-18T08:03:55.815 INFO:journalctl@ceph.grafana.a.smithi050.stdout:Sep 18 08:03:55 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-grafana-a[60403]: logger=migrator t=2024-09-18T08:03:55.770342722Z level=info msg="Migration successfully executed" id="create user auth token table" duration=581.027µs 2024-09-18T08:03:55.815 INFO:journalctl@ceph.grafana.a.smithi050.stdout:Sep 18 08:03:55 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-grafana-a[60403]: logger=migrator t=2024-09-18T08:03:55.770664474Z level=info msg="Executing migration" id="add unique index user_auth_token.auth_token" 2024-09-18T08:03:55.816 INFO:journalctl@ceph.grafana.a.smithi050.stdout:Sep 18 08:03:55 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-grafana-a[60403]: logger=migrator t=2024-09-18T08:03:55.771257939Z level=info msg="Migration successfully executed" id="add unique index user_auth_token.auth_token" duration=593.223µs 2024-09-18T08:03:55.816 INFO:journalctl@ceph.grafana.a.smithi050.stdout:Sep 18 08:03:55 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-grafana-a[60403]: logger=migrator t=2024-09-18T08:03:55.771578346Z level=info msg="Executing migration" id="add unique index user_auth_token.prev_auth_token" 2024-09-18T08:03:55.816 INFO:journalctl@ceph.grafana.a.smithi050.stdout:Sep 18 08:03:55 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-grafana-a[60403]: logger=migrator t=2024-09-18T08:03:55.772187128Z level=info msg="Migration successfully executed" id="add unique index user_auth_token.prev_auth_token" duration=609.243µs 2024-09-18T08:03:55.816 INFO:journalctl@ceph.grafana.a.smithi050.stdout:Sep 18 08:03:55 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-grafana-a[60403]: logger=migrator t=2024-09-18T08:03:55.772506226Z level=info msg="Executing migration" id="add index user_auth_token.user_id" 2024-09-18T08:03:55.816 INFO:journalctl@ceph.grafana.a.smithi050.stdout:Sep 18 08:03:55 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-grafana-a[60403]: logger=migrator t=2024-09-18T08:03:55.773182894Z level=info msg="Migration successfully executed" id="add index user_auth_token.user_id" duration=676.43µs 2024-09-18T08:03:55.816 INFO:journalctl@ceph.grafana.a.smithi050.stdout:Sep 18 08:03:55 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-grafana-a[60403]: logger=migrator t=2024-09-18T08:03:55.773518872Z level=info msg="Executing migration" id="Add revoked_at to the user auth token" 2024-09-18T08:03:55.816 INFO:journalctl@ceph.grafana.a.smithi050.stdout:Sep 18 08:03:55 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-grafana-a[60403]: logger=migrator t=2024-09-18T08:03:55.776915777Z level=info msg="Migration successfully executed" id="Add revoked_at to the user auth token" duration=3.396648ms 2024-09-18T08:03:55.816 INFO:journalctl@ceph.grafana.a.smithi050.stdout:Sep 18 08:03:55 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-grafana-a[60403]: logger=migrator t=2024-09-18T08:03:55.777230213Z level=info msg="Executing migration" id="add index user_auth_token.revoked_at" 2024-09-18T08:03:55.816 INFO:journalctl@ceph.grafana.a.smithi050.stdout:Sep 18 08:03:55 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-grafana-a[60403]: logger=migrator t=2024-09-18T08:03:55.777830215Z level=info msg="Migration successfully executed" id="add index user_auth_token.revoked_at" duration=599.793µs 2024-09-18T08:03:55.816 INFO:journalctl@ceph.grafana.a.smithi050.stdout:Sep 18 08:03:55 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-grafana-a[60403]: logger=migrator t=2024-09-18T08:03:55.778160441Z level=info msg="Executing migration" id="create cache_data table" 2024-09-18T08:03:55.816 INFO:journalctl@ceph.grafana.a.smithi050.stdout:Sep 18 08:03:55 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-grafana-a[60403]: logger=migrator t=2024-09-18T08:03:55.778725379Z level=info msg="Migration successfully executed" id="create cache_data table" duration=564.763µs 2024-09-18T08:03:55.816 INFO:journalctl@ceph.grafana.a.smithi050.stdout:Sep 18 08:03:55 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-grafana-a[60403]: logger=migrator t=2024-09-18T08:03:55.779057095Z level=info msg="Executing migration" id="add unique index cache_data.cache_key" 2024-09-18T08:03:55.817 INFO:journalctl@ceph.grafana.a.smithi050.stdout:Sep 18 08:03:55 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-grafana-a[60403]: logger=migrator t=2024-09-18T08:03:55.779662432Z level=info msg="Migration successfully executed" id="add unique index cache_data.cache_key" duration=605.514µs 2024-09-18T08:03:55.817 INFO:journalctl@ceph.grafana.a.smithi050.stdout:Sep 18 08:03:55 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-grafana-a[60403]: logger=migrator t=2024-09-18T08:03:55.780030815Z level=info msg="Executing migration" id="create short_url table v1" 2024-09-18T08:03:55.817 INFO:journalctl@ceph.grafana.a.smithi050.stdout:Sep 18 08:03:55 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-grafana-a[60403]: logger=migrator t=2024-09-18T08:03:55.780646828Z level=info msg="Migration successfully executed" id="create short_url table v1" duration=615.708µs 2024-09-18T08:03:55.817 INFO:journalctl@ceph.grafana.a.smithi050.stdout:Sep 18 08:03:55 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-grafana-a[60403]: logger=migrator t=2024-09-18T08:03:55.781003969Z level=info msg="Executing migration" id="add index short_url.org_id-uid" 2024-09-18T08:03:55.817 INFO:journalctl@ceph.grafana.a.smithi050.stdout:Sep 18 08:03:55 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-grafana-a[60403]: logger=migrator t=2024-09-18T08:03:55.781672878Z level=info msg="Migration successfully executed" id="add index short_url.org_id-uid" duration=670.456µs 2024-09-18T08:03:55.817 INFO:journalctl@ceph.grafana.a.smithi050.stdout:Sep 18 08:03:55 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-grafana-a[60403]: logger=migrator t=2024-09-18T08:03:55.781997845Z level=info msg="Executing migration" id="alter table short_url alter column created_by type to bigint" 2024-09-18T08:03:55.817 INFO:journalctl@ceph.grafana.a.smithi050.stdout:Sep 18 08:03:55 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-grafana-a[60403]: logger=migrator t=2024-09-18T08:03:55.782038545Z level=info msg="Migration successfully executed" id="alter table short_url alter column created_by type to bigint" duration=41.075µs 2024-09-18T08:03:55.817 INFO:journalctl@ceph.grafana.a.smithi050.stdout:Sep 18 08:03:55 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-grafana-a[60403]: logger=migrator t=2024-09-18T08:03:55.782440899Z level=info msg="Executing migration" id="delete alert_definition table" 2024-09-18T08:03:55.817 INFO:journalctl@ceph.grafana.a.smithi050.stdout:Sep 18 08:03:55 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-grafana-a[60403]: logger=migrator t=2024-09-18T08:03:55.782498113Z level=info msg="Migration successfully executed" id="delete alert_definition table" duration=57.573µs 2024-09-18T08:03:55.817 INFO:journalctl@ceph.grafana.a.smithi050.stdout:Sep 18 08:03:55 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-grafana-a[60403]: logger=migrator t=2024-09-18T08:03:55.782931555Z level=info msg="Executing migration" id="recreate alert_definition table" 2024-09-18T08:03:55.817 INFO:journalctl@ceph.grafana.a.smithi050.stdout:Sep 18 08:03:55 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-grafana-a[60403]: logger=migrator t=2024-09-18T08:03:55.783517312Z level=info msg="Migration successfully executed" id="recreate alert_definition table" duration=585.4µs 2024-09-18T08:03:55.817 INFO:journalctl@ceph.grafana.a.smithi050.stdout:Sep 18 08:03:55 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-grafana-a[60403]: logger=migrator t=2024-09-18T08:03:55.783857108Z level=info msg="Executing migration" id="add index in alert_definition on org_id and title columns" 2024-09-18T08:03:55.818 INFO:journalctl@ceph.grafana.a.smithi050.stdout:Sep 18 08:03:55 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-grafana-a[60403]: logger=migrator t=2024-09-18T08:03:55.784518645Z level=info msg="Migration successfully executed" id="add index in alert_definition on org_id and title columns" duration=661.276µs 2024-09-18T08:03:55.818 INFO:journalctl@ceph.grafana.a.smithi050.stdout:Sep 18 08:03:55 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-grafana-a[60403]: logger=migrator t=2024-09-18T08:03:55.784875008Z level=info msg="Executing migration" id="add index in alert_definition on org_id and uid columns" 2024-09-18T08:03:55.818 INFO:journalctl@ceph.grafana.a.smithi050.stdout:Sep 18 08:03:55 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-grafana-a[60403]: logger=migrator t=2024-09-18T08:03:55.785550977Z level=info msg="Migration successfully executed" id="add index in alert_definition on org_id and uid columns" duration=676.268µs 2024-09-18T08:03:55.818 INFO:journalctl@ceph.grafana.a.smithi050.stdout:Sep 18 08:03:55 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-grafana-a[60403]: logger=migrator t=2024-09-18T08:03:55.785892489Z level=info msg="Executing migration" id="alter alert_definition table data column to mediumtext in mysql" 2024-09-18T08:03:55.818 INFO:journalctl@ceph.grafana.a.smithi050.stdout:Sep 18 08:03:55 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-grafana-a[60403]: logger=migrator t=2024-09-18T08:03:55.785945098Z level=info msg="Migration successfully executed" id="alter alert_definition table data column to mediumtext in mysql" duration=53.048µs 2024-09-18T08:03:55.818 INFO:journalctl@ceph.grafana.a.smithi050.stdout:Sep 18 08:03:55 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-grafana-a[60403]: logger=migrator t=2024-09-18T08:03:55.786342644Z level=info msg="Executing migration" id="drop index in alert_definition on org_id and title columns" 2024-09-18T08:03:55.818 INFO:journalctl@ceph.grafana.a.smithi050.stdout:Sep 18 08:03:55 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-grafana-a[60403]: logger=migrator t=2024-09-18T08:03:55.786974207Z level=info msg="Migration successfully executed" id="drop index in alert_definition on org_id and title columns" duration=632.105µs 2024-09-18T08:03:55.818 INFO:journalctl@ceph.grafana.a.smithi050.stdout:Sep 18 08:03:55 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-grafana-a[60403]: logger=migrator t=2024-09-18T08:03:55.787288146Z level=info msg="Executing migration" id="drop index in alert_definition on org_id and uid columns" 2024-09-18T08:03:55.818 INFO:journalctl@ceph.grafana.a.smithi050.stdout:Sep 18 08:03:55 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-grafana-a[60403]: logger=migrator t=2024-09-18T08:03:55.787929244Z level=info msg="Migration successfully executed" id="drop index in alert_definition on org_id and uid columns" duration=639.97µs 2024-09-18T08:03:55.818 INFO:journalctl@ceph.grafana.a.smithi050.stdout:Sep 18 08:03:55 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-grafana-a[60403]: logger=migrator t=2024-09-18T08:03:55.788261276Z level=info msg="Executing migration" id="add unique index in alert_definition on org_id and title columns" 2024-09-18T08:03:55.818 INFO:journalctl@ceph.grafana.a.smithi050.stdout:Sep 18 08:03:55 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-grafana-a[60403]: logger=migrator t=2024-09-18T08:03:55.78890732Z level=info msg="Migration successfully executed" id="add unique index in alert_definition on org_id and title columns" duration=645.696µs 2024-09-18T08:03:55.819 INFO:journalctl@ceph.grafana.a.smithi050.stdout:Sep 18 08:03:55 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-grafana-a[60403]: logger=migrator t=2024-09-18T08:03:55.789225272Z level=info msg="Executing migration" id="add unique index in alert_definition on org_id and uid columns" 2024-09-18T08:03:55.819 INFO:journalctl@ceph.grafana.a.smithi050.stdout:Sep 18 08:03:55 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-grafana-a[60403]: logger=migrator t=2024-09-18T08:03:55.789906356Z level=info msg="Migration successfully executed" id="add unique index in alert_definition on org_id and uid columns" duration=672.524µs 2024-09-18T08:03:55.819 INFO:journalctl@ceph.grafana.a.smithi050.stdout:Sep 18 08:03:55 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-grafana-a[60403]: logger=migrator t=2024-09-18T08:03:55.790218141Z level=info msg="Executing migration" id="Add column paused in alert_definition" 2024-09-18T08:03:55.819 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4 2024-09-18T08:03:55.841 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:03:55.842 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:03:55.842 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4' 2024-09-18T08:03:55.943 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5 2024-09-18T08:03:55.965 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:03:55.967 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:03:55.967 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5' 2024-09-18T08:03:56.042 INFO:journalctl@ceph.grafana.a.smithi050.stdout:Sep 18 08:03:55 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-grafana-a[60403]: logger=migrator t=2024-09-18T08:03:55.793939748Z level=info msg="Migration successfully executed" id="Add column paused in alert_definition" duration=3.721302ms 2024-09-18T08:03:56.042 INFO:journalctl@ceph.grafana.a.smithi050.stdout:Sep 18 08:03:55 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-grafana-a[60403]: logger=migrator t=2024-09-18T08:03:55.794248493Z level=info msg="Executing migration" id="drop alert_definition table" 2024-09-18T08:03:56.042 INFO:journalctl@ceph.grafana.a.smithi050.stdout:Sep 18 08:03:55 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-grafana-a[60403]: logger=migrator t=2024-09-18T08:03:55.794962366Z level=info msg="Migration successfully executed" id="drop alert_definition table" duration=713.633µs 2024-09-18T08:03:56.042 INFO:journalctl@ceph.grafana.a.smithi050.stdout:Sep 18 08:03:55 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-grafana-a[60403]: logger=migrator t=2024-09-18T08:03:55.795289476Z level=info msg="Executing migration" id="delete alert_definition_version table" 2024-09-18T08:03:56.042 INFO:journalctl@ceph.grafana.a.smithi050.stdout:Sep 18 08:03:55 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-grafana-a[60403]: logger=migrator t=2024-09-18T08:03:55.795366624Z level=info msg="Migration successfully executed" id="delete alert_definition_version table" duration=76.308µs 2024-09-18T08:03:56.042 INFO:journalctl@ceph.grafana.a.smithi050.stdout:Sep 18 08:03:55 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-grafana-a[60403]: logger=migrator t=2024-09-18T08:03:55.795743703Z level=info msg="Executing migration" id="recreate alert_definition_version table" 2024-09-18T08:03:56.043 INFO:journalctl@ceph.grafana.a.smithi050.stdout:Sep 18 08:03:55 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-grafana-a[60403]: logger=migrator t=2024-09-18T08:03:55.796341085Z level=info msg="Migration successfully executed" id="recreate alert_definition_version table" duration=597.329µs 2024-09-18T08:03:56.043 INFO:journalctl@ceph.grafana.a.smithi050.stdout:Sep 18 08:03:55 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-grafana-a[60403]: logger=migrator t=2024-09-18T08:03:55.796662265Z level=info msg="Executing migration" id="add index in alert_definition_version table on alert_definition_id and version columns" 2024-09-18T08:03:56.043 INFO:journalctl@ceph.grafana.a.smithi050.stdout:Sep 18 08:03:55 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-grafana-a[60403]: logger=migrator t=2024-09-18T08:03:55.797301606Z level=info msg="Migration successfully executed" id="add index in alert_definition_version table on alert_definition_id and version columns" duration=639.267µs 2024-09-18T08:03:56.043 INFO:journalctl@ceph.grafana.a.smithi050.stdout:Sep 18 08:03:55 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-grafana-a[60403]: logger=migrator t=2024-09-18T08:03:55.797626175Z level=info msg="Executing migration" id="add index in alert_definition_version table on alert_definition_uid and version columns" 2024-09-18T08:03:56.043 INFO:journalctl@ceph.grafana.a.smithi050.stdout:Sep 18 08:03:55 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-grafana-a[60403]: logger=migrator t=2024-09-18T08:03:55.798281475Z level=info msg="Migration successfully executed" id="add index in alert_definition_version table on alert_definition_uid and version columns" duration=655.093µs 2024-09-18T08:03:56.043 INFO:journalctl@ceph.grafana.a.smithi050.stdout:Sep 18 08:03:55 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-grafana-a[60403]: logger=migrator t=2024-09-18T08:03:55.798599114Z level=info msg="Executing migration" id="alter alert_definition_version table data column to mediumtext in mysql" 2024-09-18T08:03:56.043 INFO:journalctl@ceph.grafana.a.smithi050.stdout:Sep 18 08:03:55 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-grafana-a[60403]: logger=migrator t=2024-09-18T08:03:55.798651046Z level=info msg="Migration successfully executed" id="alter alert_definition_version table data column to mediumtext in mysql" duration=53.063µs 2024-09-18T08:03:56.043 INFO:journalctl@ceph.grafana.a.smithi050.stdout:Sep 18 08:03:55 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-grafana-a[60403]: logger=migrator t=2024-09-18T08:03:55.799046837Z level=info msg="Executing migration" id="drop alert_definition_version table" 2024-09-18T08:03:56.043 INFO:journalctl@ceph.grafana.a.smithi050.stdout:Sep 18 08:03:55 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-grafana-a[60403]: logger=migrator t=2024-09-18T08:03:55.799725109Z level=info msg="Migration successfully executed" id="drop alert_definition_version table" duration=678.965µs 2024-09-18T08:03:56.043 INFO:journalctl@ceph.grafana.a.smithi050.stdout:Sep 18 08:03:55 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-grafana-a[60403]: logger=migrator t=2024-09-18T08:03:55.800062758Z level=info msg="Executing migration" id="create alert_instance table" 2024-09-18T08:03:56.043 INFO:journalctl@ceph.grafana.a.smithi050.stdout:Sep 18 08:03:55 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-grafana-a[60403]: logger=migrator t=2024-09-18T08:03:55.800688072Z level=info msg="Migration successfully executed" id="create alert_instance table" duration=624.925µs 2024-09-18T08:03:56.044 INFO:journalctl@ceph.grafana.a.smithi050.stdout:Sep 18 08:03:55 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-grafana-a[60403]: logger=migrator t=2024-09-18T08:03:55.801025038Z level=info msg="Executing migration" id="add index in alert_instance table on def_org_id, def_uid and current_state columns" 2024-09-18T08:03:56.044 INFO:journalctl@ceph.grafana.a.smithi050.stdout:Sep 18 08:03:55 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-grafana-a[60403]: logger=migrator t=2024-09-18T08:03:55.801656683Z level=info msg="Migration successfully executed" id="add index in alert_instance table on def_org_id, def_uid and current_state columns" duration=631.237µs 2024-09-18T08:03:56.044 INFO:journalctl@ceph.grafana.a.smithi050.stdout:Sep 18 08:03:55 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-grafana-a[60403]: logger=migrator t=2024-09-18T08:03:55.80197618Z level=info msg="Executing migration" id="add index in alert_instance table on def_org_id, current_state columns" 2024-09-18T08:03:56.044 INFO:journalctl@ceph.grafana.a.smithi050.stdout:Sep 18 08:03:55 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-grafana-a[60403]: logger=migrator t=2024-09-18T08:03:55.802593096Z level=info msg="Migration successfully executed" id="add index in alert_instance table on def_org_id, current_state columns" duration=616.737µs 2024-09-18T08:03:56.044 INFO:journalctl@ceph.grafana.a.smithi050.stdout:Sep 18 08:03:55 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-grafana-a[60403]: logger=migrator t=2024-09-18T08:03:55.802934872Z level=info msg="Executing migration" id="add column current_state_end to alert_instance" 2024-09-18T08:03:56.044 INFO:journalctl@ceph.grafana.a.smithi050.stdout:Sep 18 08:03:55 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-grafana-a[60403]: logger=migrator t=2024-09-18T08:03:55.806605436Z level=info msg="Migration successfully executed" id="add column current_state_end to alert_instance" duration=3.670504ms 2024-09-18T08:03:56.044 INFO:journalctl@ceph.grafana.a.smithi050.stdout:Sep 18 08:03:55 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-grafana-a[60403]: logger=migrator t=2024-09-18T08:03:55.806947644Z level=info msg="Executing migration" id="remove index def_org_id, def_uid, current_state on alert_instance" 2024-09-18T08:03:56.044 INFO:journalctl@ceph.grafana.a.smithi050.stdout:Sep 18 08:03:55 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-grafana-a[60403]: logger=migrator t=2024-09-18T08:03:55.807541078Z level=info msg="Migration successfully executed" id="remove index def_org_id, def_uid, current_state on alert_instance" duration=593.481µs 2024-09-18T08:03:56.044 INFO:journalctl@ceph.grafana.a.smithi050.stdout:Sep 18 08:03:55 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-grafana-a[60403]: logger=migrator t=2024-09-18T08:03:55.807862199Z level=info msg="Executing migration" id="remove index def_org_id, current_state on alert_instance" 2024-09-18T08:03:56.044 INFO:journalctl@ceph.grafana.a.smithi050.stdout:Sep 18 08:03:55 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-grafana-a[60403]: logger=migrator t=2024-09-18T08:03:55.808461008Z level=info msg="Migration successfully executed" id="remove index def_org_id, current_state on alert_instance" duration=598.734µs 2024-09-18T08:03:56.044 INFO:journalctl@ceph.grafana.a.smithi050.stdout:Sep 18 08:03:55 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-grafana-a[60403]: logger=migrator t=2024-09-18T08:03:55.808777451Z level=info msg="Executing migration" id="rename def_org_id to rule_org_id in alert_instance" 2024-09-18T08:03:56.044 INFO:journalctl@ceph.grafana.a.smithi050.stdout:Sep 18 08:03:55 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-grafana-a[60403]: logger=migrator t=2024-09-18T08:03:55.829039049Z level=info msg="Migration successfully executed" id="rename def_org_id to rule_org_id in alert_instance" duration=20.258236ms 2024-09-18T08:03:56.045 INFO:journalctl@ceph.grafana.a.smithi050.stdout:Sep 18 08:03:55 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-grafana-a[60403]: logger=migrator t=2024-09-18T08:03:55.829499003Z level=info msg="Executing migration" id="rename def_uid to rule_uid in alert_instance" 2024-09-18T08:03:56.045 INFO:journalctl@ceph.grafana.a.smithi050.stdout:Sep 18 08:03:55 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-grafana-a[60403]: logger=migrator t=2024-09-18T08:03:55.848311657Z level=info msg="Migration successfully executed" id="rename def_uid to rule_uid in alert_instance" duration=18.809305ms 2024-09-18T08:03:56.045 INFO:journalctl@ceph.grafana.a.smithi050.stdout:Sep 18 08:03:55 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-grafana-a[60403]: logger=migrator t=2024-09-18T08:03:55.848761805Z level=info msg="Executing migration" id="add index rule_org_id, rule_uid, current_state on alert_instance" 2024-09-18T08:03:56.045 INFO:journalctl@ceph.grafana.a.smithi050.stdout:Sep 18 08:03:55 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-grafana-a[60403]: logger=migrator t=2024-09-18T08:03:55.849445062Z level=info msg="Migration successfully executed" id="add index rule_org_id, rule_uid, current_state on alert_instance" duration=682.945µs 2024-09-18T08:03:56.045 INFO:journalctl@ceph.grafana.a.smithi050.stdout:Sep 18 08:03:55 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-grafana-a[60403]: logger=migrator t=2024-09-18T08:03:55.84979601Z level=info msg="Executing migration" id="add index rule_org_id, current_state on alert_instance" 2024-09-18T08:03:56.045 INFO:journalctl@ceph.grafana.a.smithi050.stdout:Sep 18 08:03:55 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-grafana-a[60403]: logger=migrator t=2024-09-18T08:03:55.850414021Z level=info msg="Migration successfully executed" id="add index rule_org_id, current_state on alert_instance" duration=619.477µs 2024-09-18T08:03:56.045 INFO:journalctl@ceph.grafana.a.smithi050.stdout:Sep 18 08:03:55 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-grafana-a[60403]: logger=migrator t=2024-09-18T08:03:55.850761412Z level=info msg="Executing migration" id="add current_reason column related to current_state" 2024-09-18T08:03:56.045 INFO:journalctl@ceph.grafana.a.smithi050.stdout:Sep 18 08:03:55 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-grafana-a[60403]: logger=migrator t=2024-09-18T08:03:55.854287624Z level=info msg="Migration successfully executed" id="add current_reason column related to current_state" duration=3.52586ms 2024-09-18T08:03:56.045 INFO:journalctl@ceph.grafana.a.smithi050.stdout:Sep 18 08:03:55 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-grafana-a[60403]: logger=migrator t=2024-09-18T08:03:55.854637122Z level=info msg="Executing migration" id="add result_fingerprint column to alert_instance" 2024-09-18T08:03:56.045 INFO:journalctl@ceph.grafana.a.smithi050.stdout:Sep 18 08:03:55 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-grafana-a[60403]: logger=migrator t=2024-09-18T08:03:55.85818624Z level=info msg="Migration successfully executed" id="add result_fingerprint column to alert_instance" duration=3.548272ms 2024-09-18T08:03:56.045 INFO:journalctl@ceph.grafana.a.smithi050.stdout:Sep 18 08:03:55 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-grafana-a[60403]: logger=migrator t=2024-09-18T08:03:55.85856232Z level=info msg="Executing migration" id="create alert_rule table" 2024-09-18T08:03:56.045 INFO:journalctl@ceph.grafana.a.smithi050.stdout:Sep 18 08:03:55 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-grafana-a[60403]: logger=migrator t=2024-09-18T08:03:55.859236944Z level=info msg="Migration successfully executed" id="create alert_rule table" duration=674.523µs 2024-09-18T08:03:56.046 INFO:journalctl@ceph.grafana.a.smithi050.stdout:Sep 18 08:03:55 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-grafana-a[60403]: logger=migrator t=2024-09-18T08:03:55.859590546Z level=info msg="Executing migration" id="add index in alert_rule on org_id and title columns" 2024-09-18T08:03:56.046 INFO:journalctl@ceph.grafana.a.smithi050.stdout:Sep 18 08:03:55 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-grafana-a[60403]: logger=migrator t=2024-09-18T08:03:55.860275336Z level=info msg="Migration successfully executed" id="add index in alert_rule on org_id and title columns" duration=684.521µs 2024-09-18T08:03:56.046 INFO:journalctl@ceph.grafana.a.smithi050.stdout:Sep 18 08:03:55 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-grafana-a[60403]: logger=migrator t=2024-09-18T08:03:55.860620853Z level=info msg="Executing migration" id="add index in alert_rule on org_id and uid columns" 2024-09-18T08:03:56.046 INFO:journalctl@ceph.grafana.a.smithi050.stdout:Sep 18 08:03:55 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-grafana-a[60403]: logger=migrator t=2024-09-18T08:03:55.861275286Z level=info msg="Migration successfully executed" id="add index in alert_rule on org_id and uid columns" duration=654.304µs 2024-09-18T08:03:56.046 INFO:journalctl@ceph.grafana.a.smithi050.stdout:Sep 18 08:03:55 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-grafana-a[60403]: logger=migrator t=2024-09-18T08:03:55.861619705Z level=info msg="Executing migration" id="add index in alert_rule on org_id, namespace_uid, group_uid columns" 2024-09-18T08:03:56.046 INFO:journalctl@ceph.grafana.a.smithi050.stdout:Sep 18 08:03:55 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-grafana-a[60403]: logger=migrator t=2024-09-18T08:03:55.862339741Z level=info msg="Migration successfully executed" id="add index in alert_rule on org_id, namespace_uid, group_uid columns" duration=719.537µs 2024-09-18T08:03:56.046 INFO:journalctl@ceph.grafana.a.smithi050.stdout:Sep 18 08:03:55 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-grafana-a[60403]: logger=migrator t=2024-09-18T08:03:55.862784556Z level=info msg="Executing migration" id="alter alert_rule table data column to mediumtext in mysql" 2024-09-18T08:03:56.046 INFO:journalctl@ceph.grafana.a.smithi050.stdout:Sep 18 08:03:55 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-grafana-a[60403]: logger=migrator t=2024-09-18T08:03:55.862836634Z level=info msg="Migration successfully executed" id="alter alert_rule table data column to mediumtext in mysql" duration=52.276µs 2024-09-18T08:03:56.047 INFO:journalctl@ceph.grafana.a.smithi050.stdout:Sep 18 08:03:55 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-grafana-a[60403]: logger=migrator t=2024-09-18T08:03:55.863231186Z level=info msg="Executing migration" id="add column for to alert_rule" 2024-09-18T08:03:56.047 INFO:journalctl@ceph.grafana.a.smithi050.stdout:Sep 18 08:03:55 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-grafana-a[60403]: logger=migrator t=2024-09-18T08:03:55.866989102Z level=info msg="Migration successfully executed" id="add column for to alert_rule" duration=3.757689ms 2024-09-18T08:03:56.047 INFO:journalctl@ceph.grafana.a.smithi050.stdout:Sep 18 08:03:55 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-grafana-a[60403]: logger=migrator t=2024-09-18T08:03:55.867313138Z level=info msg="Executing migration" id="add column annotations to alert_rule" 2024-09-18T08:03:56.047 INFO:journalctl@ceph.grafana.a.smithi050.stdout:Sep 18 08:03:55 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-grafana-a[60403]: logger=migrator t=2024-09-18T08:03:55.870921058Z level=info msg="Migration successfully executed" id="add column annotations to alert_rule" duration=3.606258ms 2024-09-18T08:03:56.048 INFO:journalctl@ceph.grafana.a.smithi050.stdout:Sep 18 08:03:55 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-grafana-a[60403]: logger=migrator t=2024-09-18T08:03:55.871248382Z level=info msg="Executing migration" id="add column labels to alert_rule" 2024-09-18T08:03:56.048 INFO:journalctl@ceph.grafana.a.smithi050.stdout:Sep 18 08:03:55 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-grafana-a[60403]: logger=migrator t=2024-09-18T08:03:55.874871167Z level=info msg="Migration successfully executed" id="add column labels to alert_rule" duration=3.622597ms 2024-09-18T08:03:56.048 INFO:journalctl@ceph.grafana.a.smithi050.stdout:Sep 18 08:03:55 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-grafana-a[60403]: logger=migrator t=2024-09-18T08:03:55.875220769Z level=info msg="Executing migration" id="remove unique index from alert_rule on org_id, title columns" 2024-09-18T08:03:56.048 INFO:journalctl@ceph.grafana.a.smithi050.stdout:Sep 18 08:03:55 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-grafana-a[60403]: logger=migrator t=2024-09-18T08:03:55.875853349Z level=info msg="Migration successfully executed" id="remove unique index from alert_rule on org_id, title columns" duration=632.687µs 2024-09-18T08:03:56.048 INFO:journalctl@ceph.grafana.a.smithi050.stdout:Sep 18 08:03:55 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-grafana-a[60403]: logger=migrator t=2024-09-18T08:03:55.876221781Z level=info msg="Executing migration" id="add index in alert_rule on org_id, namespase_uid and title columns" 2024-09-18T08:03:56.048 INFO:journalctl@ceph.grafana.a.smithi050.stdout:Sep 18 08:03:55 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-grafana-a[60403]: logger=migrator t=2024-09-18T08:03:55.876858036Z level=info msg="Migration successfully executed" id="add index in alert_rule on org_id, namespase_uid and title columns" duration=635.968µs 2024-09-18T08:03:56.048 INFO:journalctl@ceph.grafana.a.smithi050.stdout:Sep 18 08:03:55 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-grafana-a[60403]: logger=migrator t=2024-09-18T08:03:55.87718561Z level=info msg="Executing migration" id="add dashboard_uid column to alert_rule" 2024-09-18T08:03:56.048 INFO:journalctl@ceph.grafana.a.smithi050.stdout:Sep 18 08:03:55 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-grafana-a[60403]: logger=migrator t=2024-09-18T08:03:55.880782773Z level=info msg="Migration successfully executed" id="add dashboard_uid column to alert_rule" duration=3.596864ms 2024-09-18T08:03:56.048 INFO:journalctl@ceph.grafana.a.smithi050.stdout:Sep 18 08:03:55 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-grafana-a[60403]: logger=migrator t=2024-09-18T08:03:55.881119183Z level=info msg="Executing migration" id="add panel_id column to alert_rule" 2024-09-18T08:03:56.048 INFO:journalctl@ceph.grafana.a.smithi050.stdout:Sep 18 08:03:55 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-grafana-a[60403]: logger=migrator t=2024-09-18T08:03:55.884757054Z level=info msg="Migration successfully executed" id="add panel_id column to alert_rule" duration=3.637391ms 2024-09-18T08:03:56.048 INFO:journalctl@ceph.grafana.a.smithi050.stdout:Sep 18 08:03:55 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-grafana-a[60403]: logger=migrator t=2024-09-18T08:03:55.885098556Z level=info msg="Executing migration" id="add index in alert_rule on org_id, dashboard_uid and panel_id columns" 2024-09-18T08:03:56.048 INFO:journalctl@ceph.grafana.a.smithi050.stdout:Sep 18 08:03:55 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-grafana-a[60403]: logger=migrator t=2024-09-18T08:03:55.885746973Z level=info msg="Migration successfully executed" id="add index in alert_rule on org_id, dashboard_uid and panel_id columns" duration=648.003µs 2024-09-18T08:03:56.048 INFO:journalctl@ceph.grafana.a.smithi050.stdout:Sep 18 08:03:55 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-grafana-a[60403]: logger=migrator t=2024-09-18T08:03:55.886087085Z level=info msg="Executing migration" id="add rule_group_idx column to alert_rule" 2024-09-18T08:03:56.049 INFO:journalctl@ceph.grafana.a.smithi050.stdout:Sep 18 08:03:55 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-grafana-a[60403]: logger=migrator t=2024-09-18T08:03:55.889706605Z level=info msg="Migration successfully executed" id="add rule_group_idx column to alert_rule" duration=3.619302ms 2024-09-18T08:03:56.049 INFO:journalctl@ceph.grafana.a.smithi050.stdout:Sep 18 08:03:55 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-grafana-a[60403]: logger=migrator t=2024-09-18T08:03:55.890043857Z level=info msg="Executing migration" id="add is_paused column to alert_rule table" 2024-09-18T08:03:56.049 INFO:journalctl@ceph.grafana.a.smithi050.stdout:Sep 18 08:03:55 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-grafana-a[60403]: logger=migrator t=2024-09-18T08:03:55.893672798Z level=info msg="Migration successfully executed" id="add is_paused column to alert_rule table" duration=3.628755ms 2024-09-18T08:03:56.049 INFO:journalctl@ceph.grafana.a.smithi050.stdout:Sep 18 08:03:55 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-grafana-a[60403]: logger=migrator t=2024-09-18T08:03:55.894015738Z level=info msg="Executing migration" id="fix is_paused column for alert_rule table" 2024-09-18T08:03:56.049 INFO:journalctl@ceph.grafana.a.smithi050.stdout:Sep 18 08:03:55 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-grafana-a[60403]: logger=migrator t=2024-09-18T08:03:55.894067897Z level=info msg="Migration successfully executed" id="fix is_paused column for alert_rule table" duration=52.389µs 2024-09-18T08:03:56.049 INFO:journalctl@ceph.grafana.a.smithi050.stdout:Sep 18 08:03:55 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-grafana-a[60403]: logger=migrator t=2024-09-18T08:03:55.894485639Z level=info msg="Executing migration" id="create alert_rule_version table" 2024-09-18T08:03:56.049 INFO:journalctl@ceph.grafana.a.smithi050.stdout:Sep 18 08:03:55 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-grafana-a[60403]: logger=migrator t=2024-09-18T08:03:55.895230861Z level=info msg="Migration successfully executed" id="create alert_rule_version table" duration=744.99µs 2024-09-18T08:03:56.049 INFO:journalctl@ceph.grafana.a.smithi050.stdout:Sep 18 08:03:55 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-grafana-a[60403]: logger=migrator t=2024-09-18T08:03:55.895621443Z level=info msg="Executing migration" id="add index in alert_rule_version table on rule_org_id, rule_uid and version columns" 2024-09-18T08:03:56.049 INFO:journalctl@ceph.grafana.a.smithi050.stdout:Sep 18 08:03:55 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-grafana-a[60403]: logger=migrator t=2024-09-18T08:03:55.896323715Z level=info msg="Migration successfully executed" id="add index in alert_rule_version table on rule_org_id, rule_uid and version columns" duration=702.133µs 2024-09-18T08:03:56.049 INFO:journalctl@ceph.grafana.a.smithi050.stdout:Sep 18 08:03:55 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-grafana-a[60403]: logger=migrator t=2024-09-18T08:03:55.896664376Z level=info msg="Executing migration" id="add index in alert_rule_version table on rule_org_id, rule_namespace_uid and rule_group columns" 2024-09-18T08:03:56.049 INFO:journalctl@ceph.grafana.a.smithi050.stdout:Sep 18 08:03:55 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-grafana-a[60403]: logger=migrator t=2024-09-18T08:03:55.897372796Z level=info msg="Migration successfully executed" id="add index in alert_rule_version table on rule_org_id, rule_namespace_uid and rule_group columns" duration=707.974µs 2024-09-18T08:03:56.049 INFO:journalctl@ceph.grafana.a.smithi050.stdout:Sep 18 08:03:55 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-grafana-a[60403]: logger=migrator t=2024-09-18T08:03:55.89771286Z level=info msg="Executing migration" id="alter alert_rule_version table data column to mediumtext in mysql" 2024-09-18T08:03:56.050 INFO:journalctl@ceph.grafana.a.smithi050.stdout:Sep 18 08:03:55 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-grafana-a[60403]: logger=migrator t=2024-09-18T08:03:55.897753837Z level=info msg="Migration successfully executed" id="alter alert_rule_version table data column to mediumtext in mysql" duration=41.324µs 2024-09-18T08:03:56.050 INFO:journalctl@ceph.grafana.a.smithi050.stdout:Sep 18 08:03:55 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-grafana-a[60403]: logger=migrator t=2024-09-18T08:03:55.898151602Z level=info msg="Executing migration" id="add column for to alert_rule_version" 2024-09-18T08:03:56.050 INFO:journalctl@ceph.grafana.a.smithi050.stdout:Sep 18 08:03:55 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-grafana-a[60403]: logger=migrator t=2024-09-18T08:03:55.902014682Z level=info msg="Migration successfully executed" id="add column for to alert_rule_version" duration=3.862142ms 2024-09-18T08:03:56.050 INFO:journalctl@ceph.grafana.a.smithi050.stdout:Sep 18 08:03:55 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-grafana-a[60403]: logger=migrator t=2024-09-18T08:03:55.902388468Z level=info msg="Executing migration" id="add column annotations to alert_rule_version" 2024-09-18T08:03:56.050 INFO:journalctl@ceph.grafana.a.smithi050.stdout:Sep 18 08:03:55 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-grafana-a[60403]: logger=migrator t=2024-09-18T08:03:55.90648426Z level=info msg="Migration successfully executed" id="add column annotations to alert_rule_version" duration=4.095001ms 2024-09-18T08:03:56.050 INFO:journalctl@ceph.grafana.a.smithi050.stdout:Sep 18 08:03:55 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-grafana-a[60403]: logger=migrator t=2024-09-18T08:03:55.906933266Z level=info msg="Executing migration" id="add column labels to alert_rule_version" 2024-09-18T08:03:56.050 INFO:journalctl@ceph.grafana.a.smithi050.stdout:Sep 18 08:03:55 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-grafana-a[60403]: logger=migrator t=2024-09-18T08:03:55.911643392Z level=info msg="Migration successfully executed" id="add column labels to alert_rule_version" duration=4.711774ms 2024-09-18T08:03:56.050 INFO:journalctl@ceph.grafana.a.smithi050.stdout:Sep 18 08:03:55 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-grafana-a[60403]: logger=migrator t=2024-09-18T08:03:55.912061999Z level=info msg="Executing migration" id="add rule_group_idx column to alert_rule_version" 2024-09-18T08:03:56.050 INFO:journalctl@ceph.grafana.a.smithi050.stdout:Sep 18 08:03:55 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-grafana-a[60403]: logger=migrator t=2024-09-18T08:03:55.916393703Z level=info msg="Migration successfully executed" id="add rule_group_idx column to alert_rule_version" duration=4.333975ms 2024-09-18T08:03:56.050 INFO:journalctl@ceph.grafana.a.smithi050.stdout:Sep 18 08:03:55 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-grafana-a[60403]: logger=migrator t=2024-09-18T08:03:55.916717658Z level=info msg="Executing migration" id="add is_paused column to alert_rule_versions table" 2024-09-18T08:03:56.050 INFO:journalctl@ceph.grafana.a.smithi050.stdout:Sep 18 08:03:55 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-grafana-a[60403]: logger=migrator t=2024-09-18T08:03:55.920482915Z level=info msg="Migration successfully executed" id="add is_paused column to alert_rule_versions table" duration=3.765068ms 2024-09-18T08:03:56.050 INFO:journalctl@ceph.grafana.a.smithi050.stdout:Sep 18 08:03:55 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-grafana-a[60403]: logger=migrator t=2024-09-18T08:03:55.920792941Z level=info msg="Executing migration" id="fix is_paused column for alert_rule_version table" 2024-09-18T08:03:56.050 INFO:journalctl@ceph.grafana.a.smithi050.stdout:Sep 18 08:03:55 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-grafana-a[60403]: logger=migrator t=2024-09-18T08:03:55.920833867Z level=info msg="Migration successfully executed" id="fix is_paused column for alert_rule_version table" duration=41.307µs 2024-09-18T08:03:56.050 INFO:journalctl@ceph.grafana.a.smithi050.stdout:Sep 18 08:03:55 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-grafana-a[60403]: logger=migrator t=2024-09-18T08:03:55.921228138Z level=info msg="Executing migration" id=create_alert_configuration_table 2024-09-18T08:03:56.051 INFO:journalctl@ceph.grafana.a.smithi050.stdout:Sep 18 08:03:55 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-grafana-a[60403]: logger=migrator t=2024-09-18T08:03:55.92172055Z level=info msg="Migration successfully executed" id=create_alert_configuration_table duration=492.263µs 2024-09-18T08:03:56.051 INFO:journalctl@ceph.grafana.a.smithi050.stdout:Sep 18 08:03:55 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-grafana-a[60403]: logger=migrator t=2024-09-18T08:03:55.922049316Z level=info msg="Executing migration" id="Add column default in alert_configuration" 2024-09-18T08:03:56.051 INFO:journalctl@ceph.grafana.a.smithi050.stdout:Sep 18 08:03:55 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-grafana-a[60403]: logger=migrator t=2024-09-18T08:03:55.925965662Z level=info msg="Migration successfully executed" id="Add column default in alert_configuration" duration=3.916072ms 2024-09-18T08:03:56.051 INFO:journalctl@ceph.grafana.a.smithi050.stdout:Sep 18 08:03:55 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-grafana-a[60403]: logger=migrator t=2024-09-18T08:03:55.926273836Z level=info msg="Executing migration" id="alert alert_configuration alertmanager_configuration column from TEXT to MEDIUMTEXT if mysql" 2024-09-18T08:03:56.051 INFO:journalctl@ceph.grafana.a.smithi050.stdout:Sep 18 08:03:55 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-grafana-a[60403]: logger=migrator t=2024-09-18T08:03:55.926313597Z level=info msg="Migration successfully executed" id="alert alert_configuration alertmanager_configuration column from TEXT to MEDIUMTEXT if mysql" duration=40.189µs 2024-09-18T08:03:56.051 INFO:journalctl@ceph.grafana.a.smithi050.stdout:Sep 18 08:03:55 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-grafana-a[60403]: logger=migrator t=2024-09-18T08:03:55.926682715Z level=info msg="Executing migration" id="add column org_id in alert_configuration" 2024-09-18T08:03:56.051 INFO:journalctl@ceph.grafana.a.smithi050.stdout:Sep 18 08:03:55 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-grafana-a[60403]: logger=migrator t=2024-09-18T08:03:55.930613094Z level=info msg="Migration successfully executed" id="add column org_id in alert_configuration" duration=3.931262ms 2024-09-18T08:03:56.051 INFO:journalctl@ceph.grafana.a.smithi050.stdout:Sep 18 08:03:55 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-grafana-a[60403]: logger=migrator t=2024-09-18T08:03:55.930940846Z level=info msg="Executing migration" id="add index in alert_configuration table on org_id column" 2024-09-18T08:03:56.051 INFO:journalctl@ceph.grafana.a.smithi050.stdout:Sep 18 08:03:55 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-grafana-a[60403]: logger=migrator t=2024-09-18T08:03:55.931554688Z level=info msg="Migration successfully executed" id="add index in alert_configuration table on org_id column" duration=613.646µs 2024-09-18T08:03:56.051 INFO:journalctl@ceph.grafana.a.smithi050.stdout:Sep 18 08:03:55 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-grafana-a[60403]: logger=migrator t=2024-09-18T08:03:55.931870011Z level=info msg="Executing migration" id="add configuration_hash column to alert_configuration" 2024-09-18T08:03:56.051 INFO:journalctl@ceph.grafana.a.smithi050.stdout:Sep 18 08:03:55 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-grafana-a[60403]: logger=migrator t=2024-09-18T08:03:55.935790087Z level=info msg="Migration successfully executed" id="add configuration_hash column to alert_configuration" duration=3.919846ms 2024-09-18T08:03:56.051 INFO:journalctl@ceph.grafana.a.smithi050.stdout:Sep 18 08:03:55 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-grafana-a[60403]: logger=migrator t=2024-09-18T08:03:55.936111276Z level=info msg="Executing migration" id=create_ngalert_configuration_table 2024-09-18T08:03:56.051 INFO:journalctl@ceph.grafana.a.smithi050.stdout:Sep 18 08:03:55 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-grafana-a[60403]: logger=migrator t=2024-09-18T08:03:55.936610787Z level=info msg="Migration successfully executed" id=create_ngalert_configuration_table duration=497.757µs 2024-09-18T08:03:56.052 INFO:journalctl@ceph.grafana.a.smithi050.stdout:Sep 18 08:03:55 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-grafana-a[60403]: logger=migrator t=2024-09-18T08:03:55.936931712Z level=info msg="Executing migration" id="add index in ngalert_configuration on org_id column" 2024-09-18T08:03:56.052 INFO:journalctl@ceph.grafana.a.smithi050.stdout:Sep 18 08:03:55 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-grafana-a[60403]: logger=migrator t=2024-09-18T08:03:55.937537657Z level=info msg="Migration successfully executed" id="add index in ngalert_configuration on org_id column" duration=605.77µs 2024-09-18T08:03:56.052 INFO:journalctl@ceph.grafana.a.smithi050.stdout:Sep 18 08:03:55 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-grafana-a[60403]: logger=migrator t=2024-09-18T08:03:55.937852236Z level=info msg="Executing migration" id="add column send_alerts_to in ngalert_configuration" 2024-09-18T08:03:56.052 INFO:journalctl@ceph.grafana.a.smithi050.stdout:Sep 18 08:03:55 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-grafana-a[60403]: logger=migrator t=2024-09-18T08:03:55.941795331Z level=info msg="Migration successfully executed" id="add column send_alerts_to in ngalert_configuration" duration=3.942906ms 2024-09-18T08:03:56.052 INFO:journalctl@ceph.grafana.a.smithi050.stdout:Sep 18 08:03:55 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-grafana-a[60403]: logger=migrator t=2024-09-18T08:03:55.942133494Z level=info msg="Executing migration" id="create provenance_type table" 2024-09-18T08:03:56.052 INFO:journalctl@ceph.grafana.a.smithi050.stdout:Sep 18 08:03:55 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-grafana-a[60403]: logger=migrator t=2024-09-18T08:03:55.94263086Z level=info msg="Migration successfully executed" id="create provenance_type table" duration=497.881µs 2024-09-18T08:03:56.052 INFO:journalctl@ceph.grafana.a.smithi050.stdout:Sep 18 08:03:55 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-grafana-a[60403]: logger=migrator t=2024-09-18T08:03:55.942970966Z level=info msg="Executing migration" id="add index to uniquify (record_key, record_type, org_id) columns" 2024-09-18T08:03:56.052 INFO:journalctl@ceph.grafana.a.smithi050.stdout:Sep 18 08:03:55 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-grafana-a[60403]: logger=migrator t=2024-09-18T08:03:55.943596109Z level=info msg="Migration successfully executed" id="add index to uniquify (record_key, record_type, org_id) columns" duration=624.788µs 2024-09-18T08:03:56.052 INFO:journalctl@ceph.grafana.a.smithi050.stdout:Sep 18 08:03:55 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-grafana-a[60403]: logger=migrator t=2024-09-18T08:03:55.943922438Z level=info msg="Executing migration" id="create alert_image table" 2024-09-18T08:03:56.052 INFO:journalctl@ceph.grafana.a.smithi050.stdout:Sep 18 08:03:55 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-grafana-a[60403]: logger=migrator t=2024-09-18T08:03:55.944437764Z level=info msg="Migration successfully executed" id="create alert_image table" duration=515.073µs 2024-09-18T08:03:56.052 INFO:journalctl@ceph.grafana.a.smithi050.stdout:Sep 18 08:03:55 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-grafana-a[60403]: logger=migrator t=2024-09-18T08:03:55.944751417Z level=info msg="Executing migration" id="add unique index on token to alert_image table" 2024-09-18T08:03:56.052 INFO:journalctl@ceph.grafana.a.smithi050.stdout:Sep 18 08:03:55 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-grafana-a[60403]: logger=migrator t=2024-09-18T08:03:55.945360956Z level=info msg="Migration successfully executed" id="add unique index on token to alert_image table" duration=608.264µs 2024-09-18T08:03:56.052 INFO:journalctl@ceph.grafana.a.smithi050.stdout:Sep 18 08:03:55 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-grafana-a[60403]: logger=migrator t=2024-09-18T08:03:55.945670758Z level=info msg="Executing migration" id="support longer URLs in alert_image table" 2024-09-18T08:03:56.053 INFO:journalctl@ceph.grafana.a.smithi050.stdout:Sep 18 08:03:55 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-grafana-a[60403]: logger=migrator t=2024-09-18T08:03:55.945710428Z level=info msg="Migration successfully executed" id="support longer URLs in alert_image table" duration=39.918µs 2024-09-18T08:03:56.053 INFO:journalctl@ceph.grafana.a.smithi050.stdout:Sep 18 08:03:55 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-grafana-a[60403]: logger=migrator t=2024-09-18T08:03:55.946105415Z level=info msg="Executing migration" id=create_alert_configuration_history_table 2024-09-18T08:03:56.053 INFO:journalctl@ceph.grafana.a.smithi050.stdout:Sep 18 08:03:55 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-grafana-a[60403]: logger=migrator t=2024-09-18T08:03:55.946726003Z level=info msg="Migration successfully executed" id=create_alert_configuration_history_table duration=620.274µs 2024-09-18T08:03:56.053 INFO:journalctl@ceph.grafana.a.smithi050.stdout:Sep 18 08:03:55 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-grafana-a[60403]: logger=migrator t=2024-09-18T08:03:55.947073762Z level=info msg="Executing migration" id="drop non-unique orgID index on alert_configuration" 2024-09-18T08:03:56.053 INFO:journalctl@ceph.grafana.a.smithi050.stdout:Sep 18 08:03:55 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-grafana-a[60403]: logger=migrator t=2024-09-18T08:03:55.947685961Z level=info msg="Migration successfully executed" id="drop non-unique orgID index on alert_configuration" duration=612.259µs 2024-09-18T08:03:56.053 INFO:journalctl@ceph.grafana.a.smithi050.stdout:Sep 18 08:03:55 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-grafana-a[60403]: logger=migrator t=2024-09-18T08:03:55.948006943Z level=info msg="Executing migration" id="drop unique orgID index on alert_configuration if exists" 2024-09-18T08:03:56.053 INFO:journalctl@ceph.grafana.a.smithi050.stdout:Sep 18 08:03:55 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-grafana-a[60403]: logger=migrator t=2024-09-18T08:03:55.948238524Z level=warn msg="Skipping migration: Already executed, but not recorded in migration log" id="drop unique orgID index on alert_configuration if exists" 2024-09-18T08:03:56.053 INFO:journalctl@ceph.grafana.a.smithi050.stdout:Sep 18 08:03:55 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-grafana-a[60403]: logger=migrator t=2024-09-18T08:03:55.948589273Z level=info msg="Executing migration" id="extract alertmanager configuration history to separate table" 2024-09-18T08:03:56.053 INFO:journalctl@ceph.grafana.a.smithi050.stdout:Sep 18 08:03:55 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-grafana-a[60403]: logger=migrator t=2024-09-18T08:03:55.948864323Z level=info msg="Migration successfully executed" id="extract alertmanager configuration history to separate table" duration=274.95µs 2024-09-18T08:03:56.053 INFO:journalctl@ceph.grafana.a.smithi050.stdout:Sep 18 08:03:55 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-grafana-a[60403]: logger=migrator t=2024-09-18T08:03:55.949185621Z level=info msg="Executing migration" id="add unique index on orgID to alert_configuration" 2024-09-18T08:03:56.053 INFO:journalctl@ceph.grafana.a.smithi050.stdout:Sep 18 08:03:55 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-grafana-a[60403]: logger=migrator t=2024-09-18T08:03:55.949796565Z level=info msg="Migration successfully executed" id="add unique index on orgID to alert_configuration" duration=610.682µs 2024-09-18T08:03:56.053 INFO:journalctl@ceph.grafana.a.smithi050.stdout:Sep 18 08:03:55 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-grafana-a[60403]: logger=migrator t=2024-09-18T08:03:55.950123242Z level=info msg="Executing migration" id="add last_applied column to alert_configuration_history" 2024-09-18T08:03:56.053 INFO:journalctl@ceph.grafana.a.smithi050.stdout:Sep 18 08:03:55 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-grafana-a[60403]: logger=migrator t=2024-09-18T08:03:55.954233101Z level=info msg="Migration successfully executed" id="add last_applied column to alert_configuration_history" duration=4.109372ms 2024-09-18T08:03:56.054 INFO:journalctl@ceph.grafana.a.smithi050.stdout:Sep 18 08:03:55 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-grafana-a[60403]: logger=migrator t=2024-09-18T08:03:55.954596041Z level=info msg="Executing migration" id="create library_element table v1" 2024-09-18T08:03:56.054 INFO:journalctl@ceph.grafana.a.smithi050.stdout:Sep 18 08:03:55 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-grafana-a[60403]: logger=migrator t=2024-09-18T08:03:55.955353329Z level=info msg="Migration successfully executed" id="create library_element table v1" duration=757.07µs 2024-09-18T08:03:56.054 INFO:journalctl@ceph.grafana.a.smithi050.stdout:Sep 18 08:03:55 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-grafana-a[60403]: logger=migrator t=2024-09-18T08:03:55.955706996Z level=info msg="Executing migration" id="add index library_element org_id-folder_id-name-kind" 2024-09-18T08:03:56.054 INFO:journalctl@ceph.grafana.a.smithi050.stdout:Sep 18 08:03:55 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-grafana-a[60403]: logger=migrator t=2024-09-18T08:03:55.956389104Z level=info msg="Migration successfully executed" id="add index library_element org_id-folder_id-name-kind" duration=681.518µs 2024-09-18T08:03:56.054 INFO:journalctl@ceph.grafana.a.smithi050.stdout:Sep 18 08:03:55 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-grafana-a[60403]: logger=migrator t=2024-09-18T08:03:55.956707357Z level=info msg="Executing migration" id="create library_element_connection table v1" 2024-09-18T08:03:56.054 INFO:journalctl@ceph.grafana.a.smithi050.stdout:Sep 18 08:03:55 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-grafana-a[60403]: logger=migrator t=2024-09-18T08:03:55.957271454Z level=info msg="Migration successfully executed" id="create library_element_connection table v1" duration=563.81µs 2024-09-18T08:03:56.054 INFO:journalctl@ceph.grafana.a.smithi050.stdout:Sep 18 08:03:55 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-grafana-a[60403]: logger=migrator t=2024-09-18T08:03:55.957601191Z level=info msg="Executing migration" id="add index library_element_connection element_id-kind-connection_id" 2024-09-18T08:03:56.054 INFO:journalctl@ceph.grafana.a.smithi050.stdout:Sep 18 08:03:55 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-grafana-a[60403]: logger=migrator t=2024-09-18T08:03:55.958312408Z level=info msg="Migration successfully executed" id="add index library_element_connection element_id-kind-connection_id" duration=710.822µs 2024-09-18T08:03:56.054 INFO:journalctl@ceph.grafana.a.smithi050.stdout:Sep 18 08:03:55 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-grafana-a[60403]: logger=migrator t=2024-09-18T08:03:55.958649683Z level=info msg="Executing migration" id="add unique index library_element org_id_uid" 2024-09-18T08:03:56.054 INFO:journalctl@ceph.grafana.a.smithi050.stdout:Sep 18 08:03:55 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-grafana-a[60403]: logger=migrator t=2024-09-18T08:03:55.959325456Z level=info msg="Migration successfully executed" id="add unique index library_element org_id_uid" duration=675.39µs 2024-09-18T08:03:56.054 INFO:journalctl@ceph.grafana.a.smithi050.stdout:Sep 18 08:03:55 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-grafana-a[60403]: logger=migrator t=2024-09-18T08:03:55.959643301Z level=info msg="Executing migration" id="increase max description length to 2048" 2024-09-18T08:03:56.054 INFO:journalctl@ceph.grafana.a.smithi050.stdout:Sep 18 08:03:55 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-grafana-a[60403]: logger=migrator t=2024-09-18T08:03:55.959659814Z level=info msg="Migration successfully executed" id="increase max description length to 2048" duration=17.233µs 2024-09-18T08:03:56.054 INFO:journalctl@ceph.grafana.a.smithi050.stdout:Sep 18 08:03:55 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-grafana-a[60403]: logger=migrator t=2024-09-18T08:03:55.960039037Z level=info msg="Executing migration" id="alter library_element model to mediumtext" 2024-09-18T08:03:56.055 INFO:journalctl@ceph.grafana.a.smithi050.stdout:Sep 18 08:03:55 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-grafana-a[60403]: logger=migrator t=2024-09-18T08:03:55.960079359Z level=info msg="Migration successfully executed" id="alter library_element model to mediumtext" duration=40.605µs 2024-09-18T08:03:56.055 INFO:journalctl@ceph.grafana.a.smithi050.stdout:Sep 18 08:03:55 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-grafana-a[60403]: logger=migrator t=2024-09-18T08:03:55.960439919Z level=info msg="Executing migration" id="clone move dashboard alerts to unified alerting" 2024-09-18T08:03:56.055 INFO:journalctl@ceph.grafana.a.smithi050.stdout:Sep 18 08:03:55 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-grafana-a[60403]: logger=migrator t=2024-09-18T08:03:55.960614112Z level=info msg="Migration successfully executed" id="clone move dashboard alerts to unified alerting" duration=174.156µs 2024-09-18T08:03:56.055 INFO:journalctl@ceph.grafana.a.smithi050.stdout:Sep 18 08:03:55 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-grafana-a[60403]: logger=migrator t=2024-09-18T08:03:55.960954784Z level=info msg="Executing migration" id="create data_keys table" 2024-09-18T08:03:56.055 INFO:journalctl@ceph.grafana.a.smithi050.stdout:Sep 18 08:03:55 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-grafana-a[60403]: logger=migrator t=2024-09-18T08:03:55.9615967Z level=info msg="Migration successfully executed" id="create data_keys table" duration=641.869µs 2024-09-18T08:03:56.055 INFO:journalctl@ceph.grafana.a.smithi050.stdout:Sep 18 08:03:55 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-grafana-a[60403]: logger=migrator t=2024-09-18T08:03:55.961946162Z level=info msg="Executing migration" id="create secrets table" 2024-09-18T08:03:56.055 INFO:journalctl@ceph.grafana.a.smithi050.stdout:Sep 18 08:03:55 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-grafana-a[60403]: logger=migrator t=2024-09-18T08:03:55.962486688Z level=info msg="Migration successfully executed" id="create secrets table" duration=540.415µs 2024-09-18T08:03:56.055 INFO:journalctl@ceph.grafana.a.smithi050.stdout:Sep 18 08:03:55 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-grafana-a[60403]: logger=migrator t=2024-09-18T08:03:55.962799472Z level=info msg="Executing migration" id="rename data_keys name column to id" 2024-09-18T08:03:56.055 INFO:journalctl@ceph.grafana.a.smithi050.stdout:Sep 18 08:03:55 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-grafana-a[60403]: logger=migrator t=2024-09-18T08:03:55.986282124Z level=info msg="Migration successfully executed" id="rename data_keys name column to id" duration=23.481419ms 2024-09-18T08:03:56.055 INFO:journalctl@ceph.grafana.a.smithi050.stdout:Sep 18 08:03:55 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-grafana-a[60403]: logger=migrator t=2024-09-18T08:03:55.986654309Z level=info msg="Executing migration" id="add name column into data_keys" 2024-09-18T08:03:56.055 INFO:journalctl@ceph.grafana.a.smithi050.stdout:Sep 18 08:03:55 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-grafana-a[60403]: logger=migrator t=2024-09-18T08:03:55.991482508Z level=info msg="Migration successfully executed" id="add name column into data_keys" duration=4.82789ms 2024-09-18T08:03:56.055 INFO:journalctl@ceph.grafana.a.smithi050.stdout:Sep 18 08:03:55 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-grafana-a[60403]: logger=migrator t=2024-09-18T08:03:55.99179425Z level=info msg="Executing migration" id="copy data_keys id column values into name" 2024-09-18T08:03:56.055 INFO:journalctl@ceph.grafana.a.smithi050.stdout:Sep 18 08:03:55 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-grafana-a[60403]: logger=migrator t=2024-09-18T08:03:55.991892326Z level=info msg="Migration successfully executed" id="copy data_keys id column values into name" duration=98.258µs 2024-09-18T08:03:56.056 INFO:journalctl@ceph.grafana.a.smithi050.stdout:Sep 18 08:03:55 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-grafana-a[60403]: logger=migrator t=2024-09-18T08:03:55.992256729Z level=info msg="Executing migration" id="rename data_keys name column to label" 2024-09-18T08:03:56.056 INFO:journalctl@ceph.grafana.a.smithi050.stdout:Sep 18 08:03:56 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-grafana-a[60403]: logger=migrator t=2024-09-18T08:03:56.015831626Z level=info msg="Migration successfully executed" id="rename data_keys name column to label" duration=23.574649ms 2024-09-18T08:03:56.056 INFO:journalctl@ceph.grafana.a.smithi050.stdout:Sep 18 08:03:56 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-grafana-a[60403]: logger=migrator t=2024-09-18T08:03:56.016174017Z level=info msg="Executing migration" id="rename data_keys id column back to name" 2024-09-18T08:03:56.056 INFO:journalctl@ceph.grafana.a.smithi050.stdout:Sep 18 08:03:56 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-grafana-a[60403]: logger=migrator t=2024-09-18T08:03:56.039597301Z level=info msg="Migration successfully executed" id="rename data_keys id column back to name" duration=23.424624ms 2024-09-18T08:03:56.056 INFO:journalctl@ceph.grafana.a.smithi050.stdout:Sep 18 08:03:56 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-grafana-a[60403]: logger=migrator t=2024-09-18T08:03:56.039988047Z level=info msg="Executing migration" id="create kv_store table v1" 2024-09-18T08:03:56.056 INFO:journalctl@ceph.grafana.a.smithi050.stdout:Sep 18 08:03:56 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-grafana-a[60403]: logger=migrator t=2024-09-18T08:03:56.040606952Z level=info msg="Migration successfully executed" id="create kv_store table v1" duration=618.72µs 2024-09-18T08:03:56.056 INFO:journalctl@ceph.grafana.a.smithi050.stdout:Sep 18 08:03:56 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-grafana-a[60403]: logger=migrator t=2024-09-18T08:03:56.040956477Z level=info msg="Executing migration" id="add index kv_store.org_id-namespace-key" 2024-09-18T08:03:56.056 INFO:journalctl@ceph.grafana.a.smithi050.stdout:Sep 18 08:03:56 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-grafana-a[60403]: logger=migrator t=2024-09-18T08:03:56.041652497Z level=info msg="Migration successfully executed" id="add index kv_store.org_id-namespace-key" duration=695.486µs 2024-09-18T08:03:56.056 INFO:journalctl@ceph.grafana.a.smithi050.stdout:Sep 18 08:03:56 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-grafana-a[60403]: logger=migrator t=2024-09-18T08:03:56.042005913Z level=info msg="Executing migration" id="update dashboard_uid and panel_id from existing annotations" 2024-09-18T08:03:56.056 INFO:journalctl@ceph.grafana.a.smithi050.stdout:Sep 18 08:03:56 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-grafana-a[60403]: logger=migrator t=2024-09-18T08:03:56.042176113Z level=info msg="Migration successfully executed" id="update dashboard_uid and panel_id from existing annotations" duration=168.427µs 2024-09-18T08:03:56.056 INFO:journalctl@ceph.grafana.a.smithi050.stdout:Sep 18 08:03:56 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-grafana-a[60403]: logger=migrator t=2024-09-18T08:03:56.042621934Z level=info msg="Executing migration" id="create permission table" 2024-09-18T08:03:56.056 INFO:journalctl@ceph.grafana.a.smithi050.stdout:Sep 18 08:03:56 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-grafana-a[60403]: logger=migrator t=2024-09-18T08:03:56.043272816Z level=info msg="Migration successfully executed" id="create permission table" duration=651.194µs 2024-09-18T08:03:56.056 INFO:journalctl@ceph.grafana.a.smithi050.stdout:Sep 18 08:03:56 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-grafana-a[60403]: logger=migrator t=2024-09-18T08:03:56.043761438Z level=info msg="Executing migration" id="add unique index permission.role_id" 2024-09-18T08:03:56.057 INFO:journalctl@ceph.grafana.a.smithi050.stdout:Sep 18 08:03:56 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-grafana-a[60403]: logger=migrator t=2024-09-18T08:03:56.044426302Z level=info msg="Migration successfully executed" id="add unique index permission.role_id" duration=665.209µs 2024-09-18T08:03:56.068 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5 2024-09-18T08:03:56.090 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:03:56.091 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:03:56.091 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5' 2024-09-18T08:03:56.108 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid 779349ca-7593-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-18T08:03:56.192 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0 2024-09-18T08:03:56.216 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:03:56.217 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:03:56.217 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0' 2024-09-18T08:03:56.304 INFO:journalctl@ceph.grafana.a.smithi050.stdout:Sep 18 08:03:56 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-grafana-a[60403]: logger=migrator t=2024-09-18T08:03:56.044753083Z level=info msg="Executing migration" id="add unique index role_id_action_scope" 2024-09-18T08:03:56.305 INFO:journalctl@ceph.grafana.a.smithi050.stdout:Sep 18 08:03:56 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-grafana-a[60403]: logger=migrator t=2024-09-18T08:03:56.045456606Z level=info msg="Migration successfully executed" id="add unique index role_id_action_scope" duration=703.312µs 2024-09-18T08:03:56.305 INFO:journalctl@ceph.grafana.a.smithi050.stdout:Sep 18 08:03:56 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-grafana-a[60403]: logger=migrator t=2024-09-18T08:03:56.045800216Z level=info msg="Executing migration" id="create role table" 2024-09-18T08:03:56.305 INFO:journalctl@ceph.grafana.a.smithi050.stdout:Sep 18 08:03:56 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-grafana-a[60403]: logger=migrator t=2024-09-18T08:03:56.046414415Z level=info msg="Migration successfully executed" id="create role table" duration=613.738µs 2024-09-18T08:03:56.305 INFO:journalctl@ceph.grafana.a.smithi050.stdout:Sep 18 08:03:56 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-grafana-a[60403]: logger=migrator t=2024-09-18T08:03:56.046751774Z level=info msg="Executing migration" id="add column display_name" 2024-09-18T08:03:56.305 INFO:journalctl@ceph.grafana.a.smithi050.stdout:Sep 18 08:03:56 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-grafana-a[60403]: logger=migrator t=2024-09-18T08:03:56.051560048Z level=info msg="Migration successfully executed" id="add column display_name" duration=4.808116ms 2024-09-18T08:03:56.305 INFO:journalctl@ceph.grafana.a.smithi050.stdout:Sep 18 08:03:56 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-grafana-a[60403]: logger=migrator t=2024-09-18T08:03:56.051912822Z level=info msg="Executing migration" id="add column group_name" 2024-09-18T08:03:56.305 INFO:journalctl@ceph.grafana.a.smithi050.stdout:Sep 18 08:03:56 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-grafana-a[60403]: logger=migrator t=2024-09-18T08:03:56.056421668Z level=info msg="Migration successfully executed" id="add column group_name" duration=4.508631ms 2024-09-18T08:03:56.305 INFO:journalctl@ceph.grafana.a.smithi050.stdout:Sep 18 08:03:56 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-grafana-a[60403]: logger=migrator t=2024-09-18T08:03:56.056771803Z level=info msg="Executing migration" id="add index role.org_id" 2024-09-18T08:03:56.305 INFO:journalctl@ceph.grafana.a.smithi050.stdout:Sep 18 08:03:56 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-grafana-a[60403]: logger=migrator t=2024-09-18T08:03:56.057442039Z level=info msg="Migration successfully executed" id="add index role.org_id" duration=670.068µs 2024-09-18T08:03:56.306 INFO:journalctl@ceph.grafana.a.smithi050.stdout:Sep 18 08:03:56 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-grafana-a[60403]: logger=migrator t=2024-09-18T08:03:56.057791925Z level=info msg="Executing migration" id="add unique index role_org_id_name" 2024-09-18T08:03:56.306 INFO:journalctl@ceph.grafana.a.smithi050.stdout:Sep 18 08:03:56 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-grafana-a[60403]: logger=migrator t=2024-09-18T08:03:56.058504018Z level=info msg="Migration successfully executed" id="add unique index role_org_id_name" duration=711.69µs 2024-09-18T08:03:56.306 INFO:journalctl@ceph.grafana.a.smithi050.stdout:Sep 18 08:03:56 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-grafana-a[60403]: logger=migrator t=2024-09-18T08:03:56.058856682Z level=info msg="Executing migration" id="add index role_org_id_uid" 2024-09-18T08:03:56.306 INFO:journalctl@ceph.grafana.a.smithi050.stdout:Sep 18 08:03:56 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-grafana-a[60403]: logger=migrator t=2024-09-18T08:03:56.059563832Z level=info msg="Migration successfully executed" id="add index role_org_id_uid" duration=703.988µs 2024-09-18T08:03:56.306 INFO:journalctl@ceph.grafana.a.smithi050.stdout:Sep 18 08:03:56 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-grafana-a[60403]: logger=migrator t=2024-09-18T08:03:56.059926027Z level=info msg="Executing migration" id="create team role table" 2024-09-18T08:03:56.306 INFO:journalctl@ceph.grafana.a.smithi050.stdout:Sep 18 08:03:56 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-grafana-a[60403]: logger=migrator t=2024-09-18T08:03:56.06050222Z level=info msg="Migration successfully executed" id="create team role table" duration=576.014µs 2024-09-18T08:03:56.306 INFO:journalctl@ceph.grafana.a.smithi050.stdout:Sep 18 08:03:56 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-grafana-a[60403]: logger=migrator t=2024-09-18T08:03:56.060857353Z level=info msg="Executing migration" id="add index team_role.org_id" 2024-09-18T08:03:56.306 INFO:journalctl@ceph.grafana.a.smithi050.stdout:Sep 18 08:03:56 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-grafana-a[60403]: logger=migrator t=2024-09-18T08:03:56.061555046Z level=info msg="Migration successfully executed" id="add index team_role.org_id" duration=697.419µs 2024-09-18T08:03:56.307 INFO:journalctl@ceph.grafana.a.smithi050.stdout:Sep 18 08:03:56 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-grafana-a[60403]: logger=migrator t=2024-09-18T08:03:56.06189458Z level=info msg="Executing migration" id="add unique index team_role_org_id_team_id_role_id" 2024-09-18T08:03:56.307 INFO:journalctl@ceph.grafana.a.smithi050.stdout:Sep 18 08:03:56 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-grafana-a[60403]: logger=migrator t=2024-09-18T08:03:56.06262307Z level=info msg="Migration successfully executed" id="add unique index team_role_org_id_team_id_role_id" duration=728.136µs 2024-09-18T08:03:56.307 INFO:journalctl@ceph.grafana.a.smithi050.stdout:Sep 18 08:03:56 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-grafana-a[60403]: logger=migrator t=2024-09-18T08:03:56.062980039Z level=info msg="Executing migration" id="add index team_role.team_id" 2024-09-18T08:03:56.307 INFO:journalctl@ceph.grafana.a.smithi050.stdout:Sep 18 08:03:56 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-grafana-a[60403]: logger=migrator t=2024-09-18T08:03:56.063622406Z level=info msg="Migration successfully executed" id="add index team_role.team_id" duration=642.197µs 2024-09-18T08:03:56.308 INFO:journalctl@ceph.grafana.a.smithi050.stdout:Sep 18 08:03:56 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-grafana-a[60403]: logger=migrator t=2024-09-18T08:03:56.064008763Z level=info msg="Executing migration" id="create user role table" 2024-09-18T08:03:56.308 INFO:journalctl@ceph.grafana.a.smithi050.stdout:Sep 18 08:03:56 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-grafana-a[60403]: logger=migrator t=2024-09-18T08:03:56.06457127Z level=info msg="Migration successfully executed" id="create user role table" duration=562.279µs 2024-09-18T08:03:56.308 INFO:journalctl@ceph.grafana.a.smithi050.stdout:Sep 18 08:03:56 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-grafana-a[60403]: logger=migrator t=2024-09-18T08:03:56.064925738Z level=info msg="Executing migration" id="add index user_role.org_id" 2024-09-18T08:03:56.308 INFO:journalctl@ceph.grafana.a.smithi050.stdout:Sep 18 08:03:56 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-grafana-a[60403]: logger=migrator t=2024-09-18T08:03:56.065606558Z level=info msg="Migration successfully executed" id="add index user_role.org_id" duration=680.634µs 2024-09-18T08:03:56.308 INFO:journalctl@ceph.grafana.a.smithi050.stdout:Sep 18 08:03:56 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-grafana-a[60403]: logger=migrator t=2024-09-18T08:03:56.065963847Z level=info msg="Executing migration" id="add unique index user_role_org_id_user_id_role_id" 2024-09-18T08:03:56.308 INFO:journalctl@ceph.grafana.a.smithi050.stdout:Sep 18 08:03:56 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-grafana-a[60403]: logger=migrator t=2024-09-18T08:03:56.066642892Z level=info msg="Migration successfully executed" id="add unique index user_role_org_id_user_id_role_id" duration=678.702µs 2024-09-18T08:03:56.308 INFO:journalctl@ceph.grafana.a.smithi050.stdout:Sep 18 08:03:56 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-grafana-a[60403]: logger=migrator t=2024-09-18T08:03:56.067000391Z level=info msg="Executing migration" id="add index user_role.user_id" 2024-09-18T08:03:56.308 INFO:journalctl@ceph.grafana.a.smithi050.stdout:Sep 18 08:03:56 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-grafana-a[60403]: logger=migrator t=2024-09-18T08:03:56.06766689Z level=info msg="Migration successfully executed" id="add index user_role.user_id" duration=666.342µs 2024-09-18T08:03:56.308 INFO:journalctl@ceph.grafana.a.smithi050.stdout:Sep 18 08:03:56 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-grafana-a[60403]: logger=migrator t=2024-09-18T08:03:56.068017042Z level=info msg="Executing migration" id="create builtin role table" 2024-09-18T08:03:56.308 INFO:journalctl@ceph.grafana.a.smithi050.stdout:Sep 18 08:03:56 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-grafana-a[60403]: logger=migrator t=2024-09-18T08:03:56.068580055Z level=info msg="Migration successfully executed" id="create builtin role table" duration=562.675µs 2024-09-18T08:03:56.309 INFO:journalctl@ceph.grafana.a.smithi050.stdout:Sep 18 08:03:56 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-grafana-a[60403]: logger=migrator t=2024-09-18T08:03:56.068937119Z level=info msg="Executing migration" id="add index builtin_role.role_id" 2024-09-18T08:03:56.309 INFO:journalctl@ceph.grafana.a.smithi050.stdout:Sep 18 08:03:56 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-grafana-a[60403]: logger=migrator t=2024-09-18T08:03:56.069797704Z level=info msg="Migration successfully executed" id="add index builtin_role.role_id" duration=859.541µs 2024-09-18T08:03:56.309 INFO:journalctl@ceph.grafana.a.smithi050.stdout:Sep 18 08:03:56 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-grafana-a[60403]: logger=migrator t=2024-09-18T08:03:56.070260106Z level=info msg="Executing migration" id="add index builtin_role.name" 2024-09-18T08:03:56.309 INFO:journalctl@ceph.grafana.a.smithi050.stdout:Sep 18 08:03:56 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-grafana-a[60403]: logger=migrator t=2024-09-18T08:03:56.071269772Z level=info msg="Migration successfully executed" id="add index builtin_role.name" duration=1.009296ms 2024-09-18T08:03:56.309 INFO:journalctl@ceph.grafana.a.smithi050.stdout:Sep 18 08:03:56 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-grafana-a[60403]: logger=migrator t=2024-09-18T08:03:56.071708652Z level=info msg="Executing migration" id="Add column org_id to builtin_role table" 2024-09-18T08:03:56.309 INFO:journalctl@ceph.grafana.a.smithi050.stdout:Sep 18 08:03:56 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-grafana-a[60403]: logger=migrator t=2024-09-18T08:03:56.07735308Z level=info msg="Migration successfully executed" id="Add column org_id to builtin_role table" duration=5.643954ms 2024-09-18T08:03:56.309 INFO:journalctl@ceph.grafana.a.smithi050.stdout:Sep 18 08:03:56 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-grafana-a[60403]: logger=migrator t=2024-09-18T08:03:56.077712435Z level=info msg="Executing migration" id="add index builtin_role.org_id" 2024-09-18T08:03:56.309 INFO:journalctl@ceph.grafana.a.smithi050.stdout:Sep 18 08:03:56 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-grafana-a[60403]: logger=migrator t=2024-09-18T08:03:56.07840737Z level=info msg="Migration successfully executed" id="add index builtin_role.org_id" duration=694.714µs 2024-09-18T08:03:56.309 INFO:journalctl@ceph.grafana.a.smithi050.stdout:Sep 18 08:03:56 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-grafana-a[60403]: logger=migrator t=2024-09-18T08:03:56.078746001Z level=info msg="Executing migration" id="add unique index builtin_role_org_id_role_id_role" 2024-09-18T08:03:56.309 INFO:journalctl@ceph.grafana.a.smithi050.stdout:Sep 18 08:03:56 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-grafana-a[60403]: logger=migrator t=2024-09-18T08:03:56.079439871Z level=info msg="Migration successfully executed" id="add unique index builtin_role_org_id_role_id_role" duration=694.069µs 2024-09-18T08:03:56.309 INFO:journalctl@ceph.grafana.a.smithi050.stdout:Sep 18 08:03:56 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-grafana-a[60403]: logger=migrator t=2024-09-18T08:03:56.079756115Z level=info msg="Executing migration" id="Remove unique index role_org_id_uid" 2024-09-18T08:03:56.309 INFO:journalctl@ceph.grafana.a.smithi050.stdout:Sep 18 08:03:56 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-grafana-a[60403]: logger=migrator t=2024-09-18T08:03:56.080441761Z level=info msg="Migration successfully executed" id="Remove unique index role_org_id_uid" duration=685.461µs 2024-09-18T08:03:56.310 INFO:journalctl@ceph.grafana.a.smithi050.stdout:Sep 18 08:03:56 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-grafana-a[60403]: logger=migrator t=2024-09-18T08:03:56.08076729Z level=info msg="Executing migration" id="add unique index role.uid" 2024-09-18T08:03:56.310 INFO:journalctl@ceph.grafana.a.smithi050.stdout:Sep 18 08:03:56 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-grafana-a[60403]: logger=migrator t=2024-09-18T08:03:56.081489691Z level=info msg="Migration successfully executed" id="add unique index role.uid" duration=722.255µs 2024-09-18T08:03:56.310 INFO:journalctl@ceph.grafana.a.smithi050.stdout:Sep 18 08:03:56 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-grafana-a[60403]: logger=migrator t=2024-09-18T08:03:56.081818377Z level=info msg="Executing migration" id="create seed assignment table" 2024-09-18T08:03:56.310 INFO:journalctl@ceph.grafana.a.smithi050.stdout:Sep 18 08:03:56 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-grafana-a[60403]: logger=migrator t=2024-09-18T08:03:56.082347573Z level=info msg="Migration successfully executed" id="create seed assignment table" duration=529.479µs 2024-09-18T08:03:56.310 INFO:journalctl@ceph.grafana.a.smithi050.stdout:Sep 18 08:03:56 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-grafana-a[60403]: logger=migrator t=2024-09-18T08:03:56.082717395Z level=info msg="Executing migration" id="add unique index builtin_role_role_name" 2024-09-18T08:03:56.310 INFO:journalctl@ceph.grafana.a.smithi050.stdout:Sep 18 08:03:56 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-grafana-a[60403]: logger=migrator t=2024-09-18T08:03:56.083382298Z level=info msg="Migration successfully executed" id="add unique index builtin_role_role_name" duration=664.649µs 2024-09-18T08:03:56.310 INFO:journalctl@ceph.grafana.a.smithi050.stdout:Sep 18 08:03:56 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-grafana-a[60403]: logger=migrator t=2024-09-18T08:03:56.083704908Z level=info msg="Executing migration" id="add column hidden to role table" 2024-09-18T08:03:56.310 INFO:journalctl@ceph.grafana.a.smithi050.stdout:Sep 18 08:03:56 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-grafana-a[60403]: logger=migrator t=2024-09-18T08:03:56.088997677Z level=info msg="Migration successfully executed" id="add column hidden to role table" duration=5.290993ms 2024-09-18T08:03:56.310 INFO:journalctl@ceph.grafana.a.smithi050.stdout:Sep 18 08:03:56 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-grafana-a[60403]: logger=migrator t=2024-09-18T08:03:56.089474589Z level=info msg="Executing migration" id="permission kind migration" 2024-09-18T08:03:56.310 INFO:journalctl@ceph.grafana.a.smithi050.stdout:Sep 18 08:03:56 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-grafana-a[60403]: logger=migrator t=2024-09-18T08:03:56.094784649Z level=info msg="Migration successfully executed" id="permission kind migration" duration=5.312943ms 2024-09-18T08:03:56.311 INFO:journalctl@ceph.grafana.a.smithi050.stdout:Sep 18 08:03:56 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-grafana-a[60403]: logger=migrator t=2024-09-18T08:03:56.095152744Z level=info msg="Executing migration" id="permission attribute migration" 2024-09-18T08:03:56.311 INFO:journalctl@ceph.grafana.a.smithi050.stdout:Sep 18 08:03:56 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-grafana-a[60403]: logger=migrator t=2024-09-18T08:03:56.100060229Z level=info msg="Migration successfully executed" id="permission attribute migration" duration=4.90683ms 2024-09-18T08:03:56.311 INFO:journalctl@ceph.grafana.a.smithi050.stdout:Sep 18 08:03:56 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-grafana-a[60403]: logger=migrator t=2024-09-18T08:03:56.10040688Z level=info msg="Executing migration" id="permission identifier migration" 2024-09-18T08:03:56.311 INFO:journalctl@ceph.grafana.a.smithi050.stdout:Sep 18 08:03:56 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-grafana-a[60403]: logger=migrator t=2024-09-18T08:03:56.105328711Z level=info msg="Migration successfully executed" id="permission identifier migration" duration=4.921469ms 2024-09-18T08:03:56.311 INFO:journalctl@ceph.grafana.a.smithi050.stdout:Sep 18 08:03:56 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-grafana-a[60403]: logger=migrator t=2024-09-18T08:03:56.105661538Z level=info msg="Executing migration" id="add permission identifier index" 2024-09-18T08:03:56.311 INFO:journalctl@ceph.grafana.a.smithi050.stdout:Sep 18 08:03:56 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-grafana-a[60403]: logger=migrator t=2024-09-18T08:03:56.106366819Z level=info msg="Migration successfully executed" id="add permission identifier index" duration=705.014µs 2024-09-18T08:03:56.311 INFO:journalctl@ceph.grafana.a.smithi050.stdout:Sep 18 08:03:56 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-grafana-a[60403]: logger=migrator t=2024-09-18T08:03:56.106808665Z level=info msg="Executing migration" id="add permission action scope role_id index" 2024-09-18T08:03:56.311 INFO:journalctl@ceph.grafana.a.smithi050.stdout:Sep 18 08:03:56 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-grafana-a[60403]: logger=migrator t=2024-09-18T08:03:56.107561133Z level=info msg="Migration successfully executed" id="add permission action scope role_id index" duration=752.138µs 2024-09-18T08:03:56.311 INFO:journalctl@ceph.grafana.a.smithi050.stdout:Sep 18 08:03:56 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-grafana-a[60403]: logger=migrator t=2024-09-18T08:03:56.107926292Z level=info msg="Executing migration" id="remove permission role_id action scope index" 2024-09-18T08:03:56.312 INFO:journalctl@ceph.grafana.a.smithi050.stdout:Sep 18 08:03:56 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-grafana-a[60403]: logger=migrator t=2024-09-18T08:03:56.108666037Z level=info msg="Migration successfully executed" id="remove permission role_id action scope index" duration=738.503µs 2024-09-18T08:03:56.312 INFO:journalctl@ceph.grafana.a.smithi050.stdout:Sep 18 08:03:56 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-grafana-a[60403]: logger=migrator t=2024-09-18T08:03:56.109023771Z level=info msg="Executing migration" id="create query_history table v1" 2024-09-18T08:03:56.312 INFO:journalctl@ceph.grafana.a.smithi050.stdout:Sep 18 08:03:56 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-grafana-a[60403]: logger=migrator t=2024-09-18T08:03:56.109673072Z level=info msg="Migration successfully executed" id="create query_history table v1" duration=649.046µs 2024-09-18T08:03:56.312 INFO:journalctl@ceph.grafana.a.smithi050.stdout:Sep 18 08:03:56 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-grafana-a[60403]: logger=migrator t=2024-09-18T08:03:56.110019378Z level=info msg="Executing migration" id="add index query_history.org_id-created_by-datasource_uid" 2024-09-18T08:03:56.312 INFO:journalctl@ceph.grafana.a.smithi050.stdout:Sep 18 08:03:56 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-grafana-a[60403]: logger=migrator t=2024-09-18T08:03:56.11072583Z level=info msg="Migration successfully executed" id="add index query_history.org_id-created_by-datasource_uid" duration=706.017µs 2024-09-18T08:03:56.312 INFO:journalctl@ceph.grafana.a.smithi050.stdout:Sep 18 08:03:56 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-grafana-a[60403]: logger=migrator t=2024-09-18T08:03:56.111077654Z level=info msg="Executing migration" id="alter table query_history alter column created_by type to bigint" 2024-09-18T08:03:56.312 INFO:journalctl@ceph.grafana.a.smithi050.stdout:Sep 18 08:03:56 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-grafana-a[60403]: logger=migrator t=2024-09-18T08:03:56.111130743Z level=info msg="Migration successfully executed" id="alter table query_history alter column created_by type to bigint" duration=53.373µs 2024-09-18T08:03:56.312 INFO:journalctl@ceph.grafana.a.smithi050.stdout:Sep 18 08:03:56 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-grafana-a[60403]: logger=migrator t=2024-09-18T08:03:56.111518133Z level=info msg="Executing migration" id="rbac disabled migrator" 2024-09-18T08:03:56.312 INFO:journalctl@ceph.grafana.a.smithi050.stdout:Sep 18 08:03:56 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-grafana-a[60403]: logger=migrator t=2024-09-18T08:03:56.111549982Z level=info msg="Migration successfully executed" id="rbac disabled migrator" duration=33.182µs 2024-09-18T08:03:56.312 INFO:journalctl@ceph.grafana.a.smithi050.stdout:Sep 18 08:03:56 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-grafana-a[60403]: logger=migrator t=2024-09-18T08:03:56.111891943Z level=info msg="Executing migration" id="teams permissions migration" 2024-09-18T08:03:56.312 INFO:journalctl@ceph.grafana.a.smithi050.stdout:Sep 18 08:03:56 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-grafana-a[60403]: logger=migrator t=2024-09-18T08:03:56.112193114Z level=info msg="Migration successfully executed" id="teams permissions migration" duration=301.582µs 2024-09-18T08:03:56.312 INFO:journalctl@ceph.grafana.a.smithi050.stdout:Sep 18 08:03:56 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-grafana-a[60403]: logger=migrator t=2024-09-18T08:03:56.11256215Z level=info msg="Executing migration" id="dashboard permissions" 2024-09-18T08:03:56.313 INFO:journalctl@ceph.grafana.a.smithi050.stdout:Sep 18 08:03:56 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-grafana-a[60403]: logger=migrator t=2024-09-18T08:03:56.11293611Z level=info msg="Migration successfully executed" id="dashboard permissions" duration=375.036µs 2024-09-18T08:03:56.313 INFO:journalctl@ceph.grafana.a.smithi050.stdout:Sep 18 08:03:56 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-grafana-a[60403]: logger=migrator t=2024-09-18T08:03:56.113307089Z level=info msg="Executing migration" id="dashboard permissions uid scopes" 2024-09-18T08:03:56.313 INFO:journalctl@ceph.grafana.a.smithi050.stdout:Sep 18 08:03:56 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-grafana-a[60403]: logger=migrator t=2024-09-18T08:03:56.11373751Z level=info msg="Migration successfully executed" id="dashboard permissions uid scopes" duration=428.787µs 2024-09-18T08:03:56.313 INFO:journalctl@ceph.grafana.a.smithi050.stdout:Sep 18 08:03:56 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-grafana-a[60403]: logger=migrator t=2024-09-18T08:03:56.11415955Z level=info msg="Executing migration" id="drop managed folder create actions" 2024-09-18T08:03:56.313 INFO:journalctl@ceph.grafana.a.smithi050.stdout:Sep 18 08:03:56 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-grafana-a[60403]: logger=migrator t=2024-09-18T08:03:56.114304009Z level=info msg="Migration successfully executed" id="drop managed folder create actions" duration=144.331µs 2024-09-18T08:03:56.313 INFO:journalctl@ceph.grafana.a.smithi050.stdout:Sep 18 08:03:56 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-grafana-a[60403]: logger=migrator t=2024-09-18T08:03:56.11466155Z level=info msg="Executing migration" id="alerting notification permissions" 2024-09-18T08:03:56.313 INFO:journalctl@ceph.grafana.a.smithi050.stdout:Sep 18 08:03:56 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-grafana-a[60403]: logger=migrator t=2024-09-18T08:03:56.114979349Z level=info msg="Migration successfully executed" id="alerting notification permissions" duration=317.933µs 2024-09-18T08:03:56.313 INFO:journalctl@ceph.grafana.a.smithi050.stdout:Sep 18 08:03:56 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-grafana-a[60403]: logger=migrator t=2024-09-18T08:03:56.11534006Z level=info msg="Executing migration" id="create query_history_star table v1" 2024-09-18T08:03:56.313 INFO:journalctl@ceph.grafana.a.smithi050.stdout:Sep 18 08:03:56 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-grafana-a[60403]: logger=migrator t=2024-09-18T08:03:56.115860898Z level=info msg="Migration successfully executed" id="create query_history_star table v1" duration=520.577µs 2024-09-18T08:03:56.313 INFO:journalctl@ceph.grafana.a.smithi050.stdout:Sep 18 08:03:56 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-grafana-a[60403]: logger=migrator t=2024-09-18T08:03:56.11619977Z level=info msg="Executing migration" id="add index query_history.user_id-query_uid" 2024-09-18T08:03:56.313 INFO:journalctl@ceph.grafana.a.smithi050.stdout:Sep 18 08:03:56 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-grafana-a[60403]: logger=migrator t=2024-09-18T08:03:56.116863573Z level=info msg="Migration successfully executed" id="add index query_history.user_id-query_uid" duration=663.595µs 2024-09-18T08:03:56.313 INFO:journalctl@ceph.grafana.a.smithi050.stdout:Sep 18 08:03:56 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-grafana-a[60403]: logger=migrator t=2024-09-18T08:03:56.117193128Z level=info msg="Executing migration" id="add column org_id in query_history_star" 2024-09-18T08:03:56.314 INFO:journalctl@ceph.grafana.a.smithi050.stdout:Sep 18 08:03:56 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-grafana-a[60403]: logger=migrator t=2024-09-18T08:03:56.122238956Z level=info msg="Migration successfully executed" id="add column org_id in query_history_star" duration=5.044696ms 2024-09-18T08:03:56.314 INFO:journalctl@ceph.grafana.a.smithi050.stdout:Sep 18 08:03:56 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-grafana-a[60403]: logger=migrator t=2024-09-18T08:03:56.122569605Z level=info msg="Executing migration" id="alter table query_history_star_mig column user_id type to bigint" 2024-09-18T08:03:56.314 INFO:journalctl@ceph.grafana.a.smithi050.stdout:Sep 18 08:03:56 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-grafana-a[60403]: logger=migrator t=2024-09-18T08:03:56.122610515Z level=info msg="Migration successfully executed" id="alter table query_history_star_mig column user_id type to bigint" duration=41.327µs 2024-09-18T08:03:56.314 INFO:journalctl@ceph.grafana.a.smithi050.stdout:Sep 18 08:03:56 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-grafana-a[60403]: logger=migrator t=2024-09-18T08:03:56.123021078Z level=info msg="Executing migration" id="create correlation table v1" 2024-09-18T08:03:56.314 INFO:journalctl@ceph.grafana.a.smithi050.stdout:Sep 18 08:03:56 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-grafana-a[60403]: logger=migrator t=2024-09-18T08:03:56.12372864Z level=info msg="Migration successfully executed" id="create correlation table v1" duration=707.251µs 2024-09-18T08:03:56.314 INFO:journalctl@ceph.grafana.a.smithi050.stdout:Sep 18 08:03:56 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-grafana-a[60403]: logger=migrator t=2024-09-18T08:03:56.12407899Z level=info msg="Executing migration" id="add index correlations.uid" 2024-09-18T08:03:56.314 INFO:journalctl@ceph.grafana.a.smithi050.stdout:Sep 18 08:03:56 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-grafana-a[60403]: logger=migrator t=2024-09-18T08:03:56.124734836Z level=info msg="Migration successfully executed" id="add index correlations.uid" duration=657.524µs 2024-09-18T08:03:56.314 INFO:journalctl@ceph.grafana.a.smithi050.stdout:Sep 18 08:03:56 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-grafana-a[60403]: logger=migrator t=2024-09-18T08:03:56.125062549Z level=info msg="Executing migration" id="add index correlations.source_uid" 2024-09-18T08:03:56.314 INFO:journalctl@ceph.grafana.a.smithi050.stdout:Sep 18 08:03:56 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-grafana-a[60403]: logger=migrator t=2024-09-18T08:03:56.125719223Z level=info msg="Migration successfully executed" id="add index correlations.source_uid" duration=656.433µs 2024-09-18T08:03:56.314 INFO:journalctl@ceph.grafana.a.smithi050.stdout:Sep 18 08:03:56 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-grafana-a[60403]: logger=migrator t=2024-09-18T08:03:56.126063589Z level=info msg="Executing migration" id="add correlation config column" 2024-09-18T08:03:56.314 INFO:journalctl@ceph.grafana.a.smithi050.stdout:Sep 18 08:03:56 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-grafana-a[60403]: logger=migrator t=2024-09-18T08:03:56.13132814Z level=info msg="Migration successfully executed" id="add correlation config column" duration=5.263951ms 2024-09-18T08:03:56.314 INFO:journalctl@ceph.grafana.a.smithi050.stdout:Sep 18 08:03:56 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-grafana-a[60403]: logger=migrator t=2024-09-18T08:03:56.131644463Z level=info msg="Executing migration" id="drop index IDX_correlation_uid - v1" 2024-09-18T08:03:56.314 INFO:journalctl@ceph.grafana.a.smithi050.stdout:Sep 18 08:03:56 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-grafana-a[60403]: logger=migrator t=2024-09-18T08:03:56.132313012Z level=info msg="Migration successfully executed" id="drop index IDX_correlation_uid - v1" duration=668.585µs 2024-09-18T08:03:56.315 INFO:journalctl@ceph.grafana.a.smithi050.stdout:Sep 18 08:03:56 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-grafana-a[60403]: logger=migrator t=2024-09-18T08:03:56.132617581Z level=info msg="Executing migration" id="drop index IDX_correlation_source_uid - v1" 2024-09-18T08:03:56.315 INFO:journalctl@ceph.grafana.a.smithi050.stdout:Sep 18 08:03:56 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-grafana-a[60403]: logger=migrator t=2024-09-18T08:03:56.133295843Z level=info msg="Migration successfully executed" id="drop index IDX_correlation_source_uid - v1" duration=678.201µs 2024-09-18T08:03:56.315 INFO:journalctl@ceph.grafana.a.smithi050.stdout:Sep 18 08:03:56 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-grafana-a[60403]: logger=migrator t=2024-09-18T08:03:56.133615055Z level=info msg="Executing migration" id="Rename table correlation to correlation_tmp_qwerty - v1" 2024-09-18T08:03:56.315 INFO:journalctl@ceph.grafana.a.smithi050.stdout:Sep 18 08:03:56 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-grafana-a[60403]: logger=migrator t=2024-09-18T08:03:56.148295814Z level=info msg="Migration successfully executed" id="Rename table correlation to correlation_tmp_qwerty - v1" duration=14.67659ms 2024-09-18T08:03:56.315 INFO:journalctl@ceph.grafana.a.smithi050.stdout:Sep 18 08:03:56 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-grafana-a[60403]: logger=migrator t=2024-09-18T08:03:56.148698113Z level=info msg="Executing migration" id="create correlation v2" 2024-09-18T08:03:56.315 INFO:journalctl@ceph.grafana.a.smithi050.stdout:Sep 18 08:03:56 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-grafana-a[60403]: logger=migrator t=2024-09-18T08:03:56.149491286Z level=info msg="Migration successfully executed" id="create correlation v2" duration=792.947µs 2024-09-18T08:03:56.315 INFO:journalctl@ceph.grafana.a.smithi050.stdout:Sep 18 08:03:56 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-grafana-a[60403]: logger=migrator t=2024-09-18T08:03:56.149827575Z level=info msg="Executing migration" id="create index IDX_correlation_uid - v2" 2024-09-18T08:03:56.315 INFO:journalctl@ceph.grafana.a.smithi050.stdout:Sep 18 08:03:56 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-grafana-a[60403]: logger=migrator t=2024-09-18T08:03:56.150493086Z level=info msg="Migration successfully executed" id="create index IDX_correlation_uid - v2" duration=665.385µs 2024-09-18T08:03:56.316 INFO:journalctl@ceph.grafana.a.smithi050.stdout:Sep 18 08:03:56 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-grafana-a[60403]: logger=migrator t=2024-09-18T08:03:56.150812749Z level=info msg="Executing migration" id="create index IDX_correlation_source_uid - v2" 2024-09-18T08:03:56.316 INFO:journalctl@ceph.grafana.a.smithi050.stdout:Sep 18 08:03:56 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-grafana-a[60403]: logger=migrator t=2024-09-18T08:03:56.15153987Z level=info msg="Migration successfully executed" id="create index IDX_correlation_source_uid - v2" duration=726.812µs 2024-09-18T08:03:56.316 INFO:journalctl@ceph.grafana.a.smithi050.stdout:Sep 18 08:03:56 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-grafana-a[60403]: logger=migrator t=2024-09-18T08:03:56.151871663Z level=info msg="Executing migration" id="create index IDX_correlation_org_id - v2" 2024-09-18T08:03:56.316 INFO:journalctl@ceph.grafana.a.smithi050.stdout:Sep 18 08:03:56 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-grafana-a[60403]: logger=migrator t=2024-09-18T08:03:56.152539847Z level=info msg="Migration successfully executed" id="create index IDX_correlation_org_id - v2" duration=668.162µs 2024-09-18T08:03:56.316 INFO:journalctl@ceph.grafana.a.smithi050.stdout:Sep 18 08:03:56 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-grafana-a[60403]: logger=migrator t=2024-09-18T08:03:56.152854743Z level=info msg="Executing migration" id="copy correlation v1 to v2" 2024-09-18T08:03:56.316 INFO:journalctl@ceph.grafana.a.smithi050.stdout:Sep 18 08:03:56 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-grafana-a[60403]: logger=migrator t=2024-09-18T08:03:56.153033316Z level=info msg="Migration successfully executed" id="copy correlation v1 to v2" duration=178.664µs 2024-09-18T08:03:56.316 INFO:journalctl@ceph.grafana.a.smithi050.stdout:Sep 18 08:03:56 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-grafana-a[60403]: logger=migrator t=2024-09-18T08:03:56.15338438Z level=info msg="Executing migration" id="drop correlation_tmp_qwerty" 2024-09-18T08:03:56.317 INFO:journalctl@ceph.grafana.a.smithi050.stdout:Sep 18 08:03:56 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-grafana-a[60403]: logger=migrator t=2024-09-18T08:03:56.153954718Z level=info msg="Migration successfully executed" id="drop correlation_tmp_qwerty" duration=570.286µs 2024-09-18T08:03:56.317 INFO:journalctl@ceph.grafana.a.smithi050.stdout:Sep 18 08:03:56 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-grafana-a[60403]: logger=migrator t=2024-09-18T08:03:56.15426847Z level=info msg="Executing migration" id="add provisioning column" 2024-09-18T08:03:56.317 INFO:journalctl@ceph.grafana.a.smithi050.stdout:Sep 18 08:03:56 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-grafana-a[60403]: logger=migrator t=2024-09-18T08:03:56.159457012Z level=info msg="Migration successfully executed" id="add provisioning column" duration=5.186376ms 2024-09-18T08:03:56.317 INFO:journalctl@ceph.grafana.a.smithi050.stdout:Sep 18 08:03:56 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-grafana-a[60403]: logger=migrator t=2024-09-18T08:03:56.159779446Z level=info msg="Executing migration" id="create entity_events table" 2024-09-18T08:03:56.317 INFO:journalctl@ceph.grafana.a.smithi050.stdout:Sep 18 08:03:56 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-grafana-a[60403]: logger=migrator t=2024-09-18T08:03:56.16032349Z level=info msg="Migration successfully executed" id="create entity_events table" duration=543.973µs 2024-09-18T08:03:56.317 INFO:journalctl@ceph.grafana.a.smithi050.stdout:Sep 18 08:03:56 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-grafana-a[60403]: logger=migrator t=2024-09-18T08:03:56.160639052Z level=info msg="Executing migration" id="create dashboard public config v1" 2024-09-18T08:03:56.317 INFO:journalctl@ceph.grafana.a.smithi050.stdout:Sep 18 08:03:56 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-grafana-a[60403]: logger=migrator t=2024-09-18T08:03:56.161340138Z level=info msg="Migration successfully executed" id="create dashboard public config v1" duration=701.163µs 2024-09-18T08:03:56.317 INFO:journalctl@ceph.grafana.a.smithi050.stdout:Sep 18 08:03:56 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-grafana-a[60403]: logger=migrator t=2024-09-18T08:03:56.161670514Z level=info msg="Executing migration" id="drop index UQE_dashboard_public_config_uid - v1" 2024-09-18T08:03:56.317 INFO:journalctl@ceph.grafana.a.smithi050.stdout:Sep 18 08:03:56 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-grafana-a[60403]: logger=migrator t=2024-09-18T08:03:56.16194946Z level=warn msg="Skipping migration: Already executed, but not recorded in migration log" id="drop index UQE_dashboard_public_config_uid - v1" 2024-09-18T08:03:56.317 INFO:journalctl@ceph.grafana.a.smithi050.stdout:Sep 18 08:03:56 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-grafana-a[60403]: logger=migrator t=2024-09-18T08:03:56.1623068Z level=info msg="Executing migration" id="drop index IDX_dashboard_public_config_org_id_dashboard_uid - v1" 2024-09-18T08:03:56.317 INFO:journalctl@ceph.grafana.a.smithi050.stdout:Sep 18 08:03:56 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-grafana-a[60403]: logger=migrator t=2024-09-18T08:03:56.162567273Z level=warn msg="Skipping migration: Already executed, but not recorded in migration log" id="drop index IDX_dashboard_public_config_org_id_dashboard_uid - v1" 2024-09-18T08:03:56.317 INFO:journalctl@ceph.grafana.a.smithi050.stdout:Sep 18 08:03:56 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-grafana-a[60403]: logger=migrator t=2024-09-18T08:03:56.162927608Z level=info msg="Executing migration" id="Drop old dashboard public config table" 2024-09-18T08:03:56.317 INFO:journalctl@ceph.grafana.a.smithi050.stdout:Sep 18 08:03:56 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-grafana-a[60403]: logger=migrator t=2024-09-18T08:03:56.163506944Z level=info msg="Migration successfully executed" id="Drop old dashboard public config table" duration=579.277µs 2024-09-18T08:03:56.318 INFO:journalctl@ceph.grafana.a.smithi050.stdout:Sep 18 08:03:56 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-grafana-a[60403]: logger=migrator t=2024-09-18T08:03:56.163835194Z level=info msg="Executing migration" id="recreate dashboard public config v1" 2024-09-18T08:03:56.318 INFO:journalctl@ceph.grafana.a.smithi050.stdout:Sep 18 08:03:56 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-grafana-a[60403]: logger=migrator t=2024-09-18T08:03:56.16451355Z level=info msg="Migration successfully executed" id="recreate dashboard public config v1" duration=678.082µs 2024-09-18T08:03:56.318 INFO:journalctl@ceph.grafana.a.smithi050.stdout:Sep 18 08:03:56 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-grafana-a[60403]: logger=migrator t=2024-09-18T08:03:56.164833754Z level=info msg="Executing migration" id="create index UQE_dashboard_public_config_uid - v1" 2024-09-18T08:03:56.318 INFO:journalctl@ceph.grafana.a.smithi050.stdout:Sep 18 08:03:56 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-grafana-a[60403]: logger=migrator t=2024-09-18T08:03:56.165521924Z level=info msg="Migration successfully executed" id="create index UQE_dashboard_public_config_uid - v1" duration=688.02µs 2024-09-18T08:03:56.318 INFO:journalctl@ceph.grafana.a.smithi050.stdout:Sep 18 08:03:56 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-grafana-a[60403]: logger=migrator t=2024-09-18T08:03:56.165850378Z level=info msg="Executing migration" id="create index IDX_dashboard_public_config_org_id_dashboard_uid - v1" 2024-09-18T08:03:56.318 INFO:journalctl@ceph.grafana.a.smithi050.stdout:Sep 18 08:03:56 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-grafana-a[60403]: logger=migrator t=2024-09-18T08:03:56.166554889Z level=info msg="Migration successfully executed" id="create index IDX_dashboard_public_config_org_id_dashboard_uid - v1" duration=704.375µs 2024-09-18T08:03:56.318 INFO:journalctl@ceph.grafana.a.smithi050.stdout:Sep 18 08:03:56 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-grafana-a[60403]: logger=migrator t=2024-09-18T08:03:56.166866165Z level=info msg="Executing migration" id="drop index UQE_dashboard_public_config_uid - v2" 2024-09-18T08:03:56.318 INFO:journalctl@ceph.grafana.a.smithi050.stdout:Sep 18 08:03:56 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-grafana-a[60403]: logger=migrator t=2024-09-18T08:03:56.167535297Z level=info msg="Migration successfully executed" id="drop index UQE_dashboard_public_config_uid - v2" duration=668.94µs 2024-09-18T08:03:56.318 INFO:journalctl@ceph.grafana.a.smithi050.stdout:Sep 18 08:03:56 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-grafana-a[60403]: logger=migrator t=2024-09-18T08:03:56.167845131Z level=info msg="Executing migration" id="drop index IDX_dashboard_public_config_org_id_dashboard_uid - v2" 2024-09-18T08:03:56.318 INFO:journalctl@ceph.grafana.a.smithi050.stdout:Sep 18 08:03:56 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-grafana-a[60403]: logger=migrator t=2024-09-18T08:03:56.168513372Z level=info msg="Migration successfully executed" id="drop index IDX_dashboard_public_config_org_id_dashboard_uid - v2" duration=668.178µs 2024-09-18T08:03:56.318 INFO:journalctl@ceph.grafana.a.smithi050.stdout:Sep 18 08:03:56 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-grafana-a[60403]: logger=migrator t=2024-09-18T08:03:56.168825187Z level=info msg="Executing migration" id="Drop public config table" 2024-09-18T08:03:56.318 INFO:journalctl@ceph.grafana.a.smithi050.stdout:Sep 18 08:03:56 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-grafana-a[60403]: logger=migrator t=2024-09-18T08:03:56.169406008Z level=info msg="Migration successfully executed" id="Drop public config table" duration=580.575µs 2024-09-18T08:03:56.319 INFO:journalctl@ceph.grafana.a.smithi050.stdout:Sep 18 08:03:56 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-grafana-a[60403]: logger=migrator t=2024-09-18T08:03:56.169796622Z level=info msg="Executing migration" id="Recreate dashboard public config v2" 2024-09-18T08:03:56.319 INFO:journalctl@ceph.grafana.a.smithi050.stdout:Sep 18 08:03:56 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-grafana-a[60403]: logger=migrator t=2024-09-18T08:03:56.170517163Z level=info msg="Migration successfully executed" id="Recreate dashboard public config v2" duration=720.333µs 2024-09-18T08:03:56.319 INFO:journalctl@ceph.grafana.a.smithi050.stdout:Sep 18 08:03:56 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-grafana-a[60403]: logger=migrator t=2024-09-18T08:03:56.170841597Z level=info msg="Executing migration" id="create index UQE_dashboard_public_config_uid - v2" 2024-09-18T08:03:56.319 INFO:journalctl@ceph.grafana.a.smithi050.stdout:Sep 18 08:03:56 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-grafana-a[60403]: logger=migrator t=2024-09-18T08:03:56.171544405Z level=info msg="Migration successfully executed" id="create index UQE_dashboard_public_config_uid - v2" duration=702.684µs 2024-09-18T08:03:56.319 INFO:journalctl@ceph.grafana.a.smithi050.stdout:Sep 18 08:03:56 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-grafana-a[60403]: logger=migrator t=2024-09-18T08:03:56.171852902Z level=info msg="Executing migration" id="create index IDX_dashboard_public_config_org_id_dashboard_uid - v2" 2024-09-18T08:03:56.319 INFO:journalctl@ceph.grafana.a.smithi050.stdout:Sep 18 08:03:56 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-grafana-a[60403]: logger=migrator t=2024-09-18T08:03:56.172563773Z level=info msg="Migration successfully executed" id="create index IDX_dashboard_public_config_org_id_dashboard_uid - v2" duration=710.81µs 2024-09-18T08:03:56.319 INFO:journalctl@ceph.grafana.a.smithi050.stdout:Sep 18 08:03:56 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-grafana-a[60403]: logger=migrator t=2024-09-18T08:03:56.172868428Z level=info msg="Executing migration" id="create index UQE_dashboard_public_config_access_token - v2" 2024-09-18T08:03:56.319 INFO:journalctl@ceph.grafana.a.smithi050.stdout:Sep 18 08:03:56 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-grafana-a[60403]: logger=migrator t=2024-09-18T08:03:56.173562323Z level=info msg="Migration successfully executed" id="create index UQE_dashboard_public_config_access_token - v2" duration=693.743µs 2024-09-18T08:03:56.319 INFO:journalctl@ceph.grafana.a.smithi050.stdout:Sep 18 08:03:56 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-grafana-a[60403]: logger=migrator t=2024-09-18T08:03:56.173882502Z level=info msg="Executing migration" id="Rename table dashboard_public_config to dashboard_public - v2" 2024-09-18T08:03:56.319 INFO:journalctl@ceph.grafana.a.smithi050.stdout:Sep 18 08:03:56 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-grafana-a[60403]: logger=migrator t=2024-09-18T08:03:56.192183216Z level=info msg="Migration successfully executed" id="Rename table dashboard_public_config to dashboard_public - v2" duration=18.306646ms 2024-09-18T08:03:56.319 INFO:journalctl@ceph.grafana.a.smithi050.stdout:Sep 18 08:03:56 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-grafana-a[60403]: logger=migrator t=2024-09-18T08:03:56.19256493Z level=info msg="Executing migration" id="add annotations_enabled column" 2024-09-18T08:03:56.319 INFO:journalctl@ceph.grafana.a.smithi050.stdout:Sep 18 08:03:56 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-grafana-a[60403]: logger=migrator t=2024-09-18T08:03:56.198209172Z level=info msg="Migration successfully executed" id="add annotations_enabled column" duration=5.643195ms 2024-09-18T08:03:56.319 INFO:journalctl@ceph.grafana.a.smithi050.stdout:Sep 18 08:03:56 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-grafana-a[60403]: logger=migrator t=2024-09-18T08:03:56.198542695Z level=info msg="Executing migration" id="add time_selection_enabled column" 2024-09-18T08:03:56.320 INFO:journalctl@ceph.grafana.a.smithi050.stdout:Sep 18 08:03:56 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-grafana-a[60403]: logger=migrator t=2024-09-18T08:03:56.203737868Z level=info msg="Migration successfully executed" id="add time_selection_enabled column" duration=5.195092ms 2024-09-18T08:03:56.320 INFO:journalctl@ceph.grafana.a.smithi050.stdout:Sep 18 08:03:56 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-grafana-a[60403]: logger=migrator t=2024-09-18T08:03:56.204066861Z level=info msg="Executing migration" id="delete orphaned public dashboards" 2024-09-18T08:03:56.320 INFO:journalctl@ceph.grafana.a.smithi050.stdout:Sep 18 08:03:56 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-grafana-a[60403]: logger=migrator t=2024-09-18T08:03:56.204216526Z level=info msg="Migration successfully executed" id="delete orphaned public dashboards" duration=149.802µs 2024-09-18T08:03:56.320 INFO:journalctl@ceph.grafana.a.smithi050.stdout:Sep 18 08:03:56 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-grafana-a[60403]: logger=migrator t=2024-09-18T08:03:56.204564867Z level=info msg="Executing migration" id="add share column" 2024-09-18T08:03:56.320 INFO:journalctl@ceph.grafana.a.smithi050.stdout:Sep 18 08:03:56 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-grafana-a[60403]: logger=migrator t=2024-09-18T08:03:56.209710166Z level=info msg="Migration successfully executed" id="add share column" duration=5.145004ms 2024-09-18T08:03:56.320 INFO:journalctl@ceph.grafana.a.smithi050.stdout:Sep 18 08:03:56 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-grafana-a[60403]: logger=migrator t=2024-09-18T08:03:56.210024366Z level=info msg="Executing migration" id="backfill empty share column fields with default of public" 2024-09-18T08:03:56.320 INFO:journalctl@ceph.grafana.a.smithi050.stdout:Sep 18 08:03:56 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-grafana-a[60403]: logger=migrator t=2024-09-18T08:03:56.21016136Z level=info msg="Migration successfully executed" id="backfill empty share column fields with default of public" duration=137.105µs 2024-09-18T08:03:56.320 INFO:journalctl@ceph.grafana.a.smithi050.stdout:Sep 18 08:03:56 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-grafana-a[60403]: logger=migrator t=2024-09-18T08:03:56.21051704Z level=info msg="Executing migration" id="create file table" 2024-09-18T08:03:56.320 INFO:journalctl@ceph.grafana.a.smithi050.stdout:Sep 18 08:03:56 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-grafana-a[60403]: logger=migrator t=2024-09-18T08:03:56.211137167Z level=info msg="Migration successfully executed" id="create file table" duration=620.046µs 2024-09-18T08:03:56.320 INFO:journalctl@ceph.grafana.a.smithi050.stdout:Sep 18 08:03:56 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-grafana-a[60403]: logger=migrator t=2024-09-18T08:03:56.211478726Z level=info msg="Executing migration" id="file table idx: path natural pk" 2024-09-18T08:03:56.320 INFO:journalctl@ceph.grafana.a.smithi050.stdout:Sep 18 08:03:56 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-grafana-a[60403]: logger=migrator t=2024-09-18T08:03:56.212156687Z level=info msg="Migration successfully executed" id="file table idx: path natural pk" duration=679.285µs 2024-09-18T08:03:56.320 INFO:journalctl@ceph.grafana.a.smithi050.stdout:Sep 18 08:03:56 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-grafana-a[60403]: logger=migrator t=2024-09-18T08:03:56.212471945Z level=info msg="Executing migration" id="file table idx: parent_folder_path_hash fast folder retrieval" 2024-09-18T08:03:56.321 INFO:journalctl@ceph.grafana.a.smithi050.stdout:Sep 18 08:03:56 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-grafana-a[60403]: logger=migrator t=2024-09-18T08:03:56.213143298Z level=info msg="Migration successfully executed" id="file table idx: parent_folder_path_hash fast folder retrieval" duration=671.122µs 2024-09-18T08:03:56.321 INFO:journalctl@ceph.grafana.a.smithi050.stdout:Sep 18 08:03:56 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-grafana-a[60403]: logger=migrator t=2024-09-18T08:03:56.21345799Z level=info msg="Executing migration" id="create file_meta table" 2024-09-18T08:03:56.321 INFO:journalctl@ceph.grafana.a.smithi050.stdout:Sep 18 08:03:56 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-grafana-a[60403]: logger=migrator t=2024-09-18T08:03:56.213977679Z level=info msg="Migration successfully executed" id="create file_meta table" duration=519.356µs 2024-09-18T08:03:56.321 INFO:journalctl@ceph.grafana.a.smithi050.stdout:Sep 18 08:03:56 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-grafana-a[60403]: logger=migrator t=2024-09-18T08:03:56.214290911Z level=info msg="Executing migration" id="file table idx: path key" 2024-09-18T08:03:56.321 INFO:journalctl@ceph.grafana.a.smithi050.stdout:Sep 18 08:03:56 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-grafana-a[60403]: logger=migrator t=2024-09-18T08:03:56.214987021Z level=info msg="Migration successfully executed" id="file table idx: path key" duration=695.824µs 2024-09-18T08:03:56.321 INFO:journalctl@ceph.grafana.a.smithi050.stdout:Sep 18 08:03:56 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-grafana-a[60403]: logger=migrator t=2024-09-18T08:03:56.215322168Z level=info msg="Executing migration" id="set path collation in file table" 2024-09-18T08:03:56.321 INFO:journalctl@ceph.grafana.a.smithi050.stdout:Sep 18 08:03:56 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-grafana-a[60403]: logger=migrator t=2024-09-18T08:03:56.215362384Z level=info msg="Migration successfully executed" id="set path collation in file table" duration=40.535µs 2024-09-18T08:03:56.321 INFO:journalctl@ceph.grafana.a.smithi050.stdout:Sep 18 08:03:56 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-grafana-a[60403]: logger=migrator t=2024-09-18T08:03:56.215725337Z level=info msg="Executing migration" id="migrate contents column to mediumblob for MySQL" 2024-09-18T08:03:56.321 INFO:journalctl@ceph.grafana.a.smithi050.stdout:Sep 18 08:03:56 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-grafana-a[60403]: logger=migrator t=2024-09-18T08:03:56.215764572Z level=info msg="Migration successfully executed" id="migrate contents column to mediumblob for MySQL" duration=39.551µs 2024-09-18T08:03:56.321 INFO:journalctl@ceph.grafana.a.smithi050.stdout:Sep 18 08:03:56 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-grafana-a[60403]: logger=migrator t=2024-09-18T08:03:56.216131879Z level=info msg="Executing migration" id="managed permissions migration" 2024-09-18T08:03:56.322 INFO:journalctl@ceph.grafana.a.smithi050.stdout:Sep 18 08:03:56 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-grafana-a[60403]: logger=migrator t=2024-09-18T08:03:56.216447128Z level=info msg="Migration successfully executed" id="managed permissions migration" duration=315.129µs 2024-09-18T08:03:56.322 INFO:journalctl@ceph.grafana.a.smithi050.stdout:Sep 18 08:03:56 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-grafana-a[60403]: logger=migrator t=2024-09-18T08:03:56.216794913Z level=info msg="Executing migration" id="managed folder permissions alert actions migration" 2024-09-18T08:03:56.322 INFO:journalctl@ceph.grafana.a.smithi050.stdout:Sep 18 08:03:56 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-grafana-a[60403]: logger=migrator t=2024-09-18T08:03:56.216933655Z level=info msg="Migration successfully executed" id="managed folder permissions alert actions migration" duration=138.734µs 2024-09-18T08:03:56.322 INFO:journalctl@ceph.grafana.a.smithi050.stdout:Sep 18 08:03:56 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-grafana-a[60403]: logger=migrator t=2024-09-18T08:03:56.217310248Z level=info msg="Executing migration" id="RBAC action name migrator" 2024-09-18T08:03:56.322 INFO:journalctl@ceph.grafana.a.smithi050.stdout:Sep 18 08:03:56 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-grafana-a[60403]: logger=migrator t=2024-09-18T08:03:56.218179484Z level=info msg="Migration successfully executed" id="RBAC action name migrator" duration=868.704µs 2024-09-18T08:03:56.322 INFO:journalctl@ceph.grafana.a.smithi050.stdout:Sep 18 08:03:56 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-grafana-a[60403]: logger=migrator t=2024-09-18T08:03:56.218539126Z level=info msg="Executing migration" id="Add UID column to playlist" 2024-09-18T08:03:56.322 INFO:journalctl@ceph.grafana.a.smithi050.stdout:Sep 18 08:03:56 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-grafana-a[60403]: logger=migrator t=2024-09-18T08:03:56.223812812Z level=info msg="Migration successfully executed" id="Add UID column to playlist" duration=5.273411ms 2024-09-18T08:03:56.322 INFO:journalctl@ceph.grafana.a.smithi050.stdout:Sep 18 08:03:56 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-grafana-a[60403]: logger=migrator t=2024-09-18T08:03:56.22414139Z level=info msg="Executing migration" id="Update uid column values in playlist" 2024-09-18T08:03:56.322 INFO:journalctl@ceph.grafana.a.smithi050.stdout:Sep 18 08:03:56 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-grafana-a[60403]: logger=migrator t=2024-09-18T08:03:56.22425365Z level=info msg="Migration successfully executed" id="Update uid column values in playlist" duration=112.373µs 2024-09-18T08:03:56.322 INFO:journalctl@ceph.grafana.a.smithi050.stdout:Sep 18 08:03:56 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-grafana-a[60403]: logger=migrator t=2024-09-18T08:03:56.224629161Z level=info msg="Executing migration" id="Add index for uid in playlist" 2024-09-18T08:03:56.322 INFO:journalctl@ceph.grafana.a.smithi050.stdout:Sep 18 08:03:56 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-grafana-a[60403]: logger=migrator t=2024-09-18T08:03:56.225408379Z level=info msg="Migration successfully executed" id="Add index for uid in playlist" duration=778.778µs 2024-09-18T08:03:56.322 INFO:journalctl@ceph.grafana.a.smithi050.stdout:Sep 18 08:03:56 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-grafana-a[60403]: logger=migrator t=2024-09-18T08:03:56.225740834Z level=info msg="Executing migration" id="update group index for alert rules" 2024-09-18T08:03:56.322 INFO:journalctl@ceph.grafana.a.smithi050.stdout:Sep 18 08:03:56 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-grafana-a[60403]: logger=migrator t=2024-09-18T08:03:56.226000544Z level=info msg="Migration successfully executed" id="update group index for alert rules" duration=260.344µs 2024-09-18T08:03:56.323 INFO:journalctl@ceph.grafana.a.smithi050.stdout:Sep 18 08:03:56 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-grafana-a[60403]: logger=migrator t=2024-09-18T08:03:56.22636622Z level=info msg="Executing migration" id="managed folder permissions alert actions repeated migration" 2024-09-18T08:03:56.323 INFO:journalctl@ceph.grafana.a.smithi050.stdout:Sep 18 08:03:56 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-grafana-a[60403]: logger=migrator t=2024-09-18T08:03:56.226509647Z level=info msg="Migration successfully executed" id="managed folder permissions alert actions repeated migration" duration=142.768µs 2024-09-18T08:03:56.323 INFO:journalctl@ceph.grafana.a.smithi050.stdout:Sep 18 08:03:56 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-grafana-a[60403]: logger=migrator t=2024-09-18T08:03:56.226842325Z level=info msg="Executing migration" id="admin only folder/dashboard permission" 2024-09-18T08:03:56.323 INFO:journalctl@ceph.grafana.a.smithi050.stdout:Sep 18 08:03:56 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-grafana-a[60403]: logger=migrator t=2024-09-18T08:03:56.227149921Z level=info msg="Migration successfully executed" id="admin only folder/dashboard permission" duration=307.404µs 2024-09-18T08:03:56.323 INFO:journalctl@ceph.grafana.a.smithi050.stdout:Sep 18 08:03:56 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-grafana-a[60403]: logger=migrator t=2024-09-18T08:03:56.227520651Z level=info msg="Executing migration" id="add action column to seed_assignment" 2024-09-18T08:03:56.323 INFO:journalctl@ceph.grafana.a.smithi050.stdout:Sep 18 08:03:56 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-grafana-a[60403]: logger=migrator t=2024-09-18T08:03:56.232830913Z level=info msg="Migration successfully executed" id="add action column to seed_assignment" duration=5.310681ms 2024-09-18T08:03:56.323 INFO:journalctl@ceph.grafana.a.smithi050.stdout:Sep 18 08:03:56 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-grafana-a[60403]: logger=migrator t=2024-09-18T08:03:56.233151323Z level=info msg="Executing migration" id="add scope column to seed_assignment" 2024-09-18T08:03:56.323 INFO:journalctl@ceph.grafana.a.smithi050.stdout:Sep 18 08:03:56 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-grafana-a[60403]: logger=migrator t=2024-09-18T08:03:56.238506157Z level=info msg="Migration successfully executed" id="add scope column to seed_assignment" duration=5.353254ms 2024-09-18T08:03:56.323 INFO:journalctl@ceph.grafana.a.smithi050.stdout:Sep 18 08:03:56 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-grafana-a[60403]: logger=migrator t=2024-09-18T08:03:56.238823308Z level=info msg="Executing migration" id="remove unique index builtin_role_role_name before nullable update" 2024-09-18T08:03:56.323 INFO:journalctl@ceph.grafana.a.smithi050.stdout:Sep 18 08:03:56 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-grafana-a[60403]: logger=migrator t=2024-09-18T08:03:56.239540403Z level=info msg="Migration successfully executed" id="remove unique index builtin_role_role_name before nullable update" duration=717.645µs 2024-09-18T08:03:56.323 INFO:journalctl@ceph.grafana.a.smithi050.stdout:Sep 18 08:03:56 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-grafana-a[60403]: logger=migrator t=2024-09-18T08:03:56.23984863Z level=info msg="Executing migration" id="update seed_assignment role_name column to nullable" 2024-09-18T08:03:56.323 INFO:journalctl@ceph.grafana.a.smithi050.stdout:Sep 18 08:03:56 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-grafana-a[60403]: logger=migrator t=2024-09-18T08:03:56.302698826Z level=info msg="Migration successfully executed" id="update seed_assignment role_name column to nullable" duration=62.846746ms 2024-09-18T08:03:56.323 INFO:journalctl@ceph.grafana.a.smithi050.stdout:Sep 18 08:03:56 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-grafana-a[60403]: logger=migrator t=2024-09-18T08:03:56.303258714Z level=info msg="Executing migration" id="add unique index builtin_role_name back" 2024-09-18T08:03:56.324 INFO:journalctl@ceph.grafana.a.smithi050.stdout:Sep 18 08:03:56 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-grafana-a[60403]: logger=migrator t=2024-09-18T08:03:56.304123333Z level=info msg="Migration successfully executed" id="add unique index builtin_role_name back" duration=865.738µs 2024-09-18T08:03:56.324 INFO:journalctl@ceph.grafana.a.smithi050.stdout:Sep 18 08:03:56 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-grafana-a[60403]: logger=migrator t=2024-09-18T08:03:56.304473926Z level=info msg="Executing migration" id="add unique index builtin_role_action_scope" 2024-09-18T08:03:56.324 INFO:journalctl@ceph.grafana.a.smithi050.stdout:Sep 18 08:03:56 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-grafana-a[60403]: logger=migrator t=2024-09-18T08:03:56.305252196Z level=info msg="Migration successfully executed" id="add unique index builtin_role_action_scope" duration=777.956µs 2024-09-18T08:03:56.324 INFO:journalctl@ceph.grafana.a.smithi050.stdout:Sep 18 08:03:56 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-grafana-a[60403]: logger=migrator t=2024-09-18T08:03:56.305590667Z level=info msg="Executing migration" id="add primary key to seed_assigment" 2024-09-18T08:03:56.324 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5 2024-09-18T08:03:56.349 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:03:56.350 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:03:56.350 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5' 2024-09-18T08:03:56.397 INFO:teuthology.orchestra.run.smithi007.stdout:ERROR: (22) Invalid argument 2024-09-18T08:03:56.397 INFO:teuthology.orchestra.run.smithi007.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-18T08:03:56.451 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5 2024-09-18T08:03:56.456 INFO:teuthology.orchestra.run.smithi007.stderr:pool 'unique_pool_0' created 2024-09-18T08:03:56.478 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:03:56.479 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:03:56.479 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5' 2024-09-18T08:03:56.557 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:03:56 smithi050 ceph-mon[32371]: pgmap v31: 132 pgs: 132 active+clean; 583 KiB data, 223 MiB used, 715 GiB / 715 GiB avail; 849 B/s rd, 0 op/s 2024-09-18T08:03:56.557 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:03:56 smithi050 ceph-mon[32371]: from='mgr.24470 172.21.15.7:0/2594390261' entity='mgr.y' 2024-09-18T08:03:56.557 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:03:56 smithi050 ceph-mon[32371]: from='mgr.24470 172.21.15.7:0/2594390261' entity='mgr.y' 2024-09-18T08:03:56.557 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:03:56 smithi050 ceph-mon[32371]: from='mgr.24470 172.21.15.7:0/2594390261' entity='mgr.y' 2024-09-18T08:03:56.557 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:03:56 smithi050 ceph-mon[32371]: from='mgr.24470 172.21.15.7:0/2594390261' entity='mgr.y' 2024-09-18T08:03:56.557 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:03:56 smithi050 ceph-mon[32371]: Deploying daemon node-exporter.a on smithi007 2024-09-18T08:03:56.557 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:03:56 smithi050 ceph-mon[32371]: from='client.? 172.21.15.7:0/1814546716' entity='client.admin' cmd={"prefix": "osd pool create", "pool": "unique_pool_0", "pg_num": 16} : dispatch 2024-09-18T08:03:56.558 INFO:journalctl@ceph.grafana.a.smithi050.stdout:Sep 18 08:03:56 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-grafana-a[60403]: logger=migrator t=2024-09-18T08:03:56.324220824Z level=info msg="Migration successfully executed" id="add primary key to seed_assigment" duration=18.630536ms 2024-09-18T08:03:56.558 INFO:journalctl@ceph.grafana.a.smithi050.stdout:Sep 18 08:03:56 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-grafana-a[60403]: logger=migrator t=2024-09-18T08:03:56.324678227Z level=info msg="Executing migration" id="add origin column to seed_assignment" 2024-09-18T08:03:56.558 INFO:journalctl@ceph.grafana.a.smithi050.stdout:Sep 18 08:03:56 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-grafana-a[60403]: logger=migrator t=2024-09-18T08:03:56.330350439Z level=info msg="Migration successfully executed" id="add origin column to seed_assignment" duration=5.669441ms 2024-09-18T08:03:56.558 INFO:journalctl@ceph.grafana.a.smithi050.stdout:Sep 18 08:03:56 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-grafana-a[60403]: logger=migrator t=2024-09-18T08:03:56.330814847Z level=info msg="Executing migration" id="add origin to plugin seed_assignment" 2024-09-18T08:03:56.558 INFO:journalctl@ceph.grafana.a.smithi050.stdout:Sep 18 08:03:56 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-grafana-a[60403]: logger=migrator t=2024-09-18T08:03:56.331061392Z level=info msg="Migration successfully executed" id="add origin to plugin seed_assignment" duration=246.303µs 2024-09-18T08:03:56.558 INFO:journalctl@ceph.grafana.a.smithi050.stdout:Sep 18 08:03:56 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-grafana-a[60403]: logger=migrator t=2024-09-18T08:03:56.331455321Z level=info msg="Executing migration" id="prevent seeding OnCall access" 2024-09-18T08:03:56.559 INFO:journalctl@ceph.grafana.a.smithi050.stdout:Sep 18 08:03:56 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-grafana-a[60403]: logger=migrator t=2024-09-18T08:03:56.331571884Z level=info msg="Migration successfully executed" id="prevent seeding OnCall access" duration=116.737µs 2024-09-18T08:03:56.559 INFO:journalctl@ceph.grafana.a.smithi050.stdout:Sep 18 08:03:56 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-grafana-a[60403]: logger=migrator t=2024-09-18T08:03:56.331914605Z level=info msg="Executing migration" id="managed folder permissions alert actions repeated fixed migration" 2024-09-18T08:03:56.559 INFO:journalctl@ceph.grafana.a.smithi050.stdout:Sep 18 08:03:56 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-grafana-a[60403]: logger=migrator t=2024-09-18T08:03:56.332060144Z level=info msg="Migration successfully executed" id="managed folder permissions alert actions repeated fixed migration" duration=145.531µs 2024-09-18T08:03:56.559 INFO:journalctl@ceph.grafana.a.smithi050.stdout:Sep 18 08:03:56 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-grafana-a[60403]: logger=migrator t=2024-09-18T08:03:56.332425714Z level=info msg="Executing migration" id="managed folder permissions library panel actions migration" 2024-09-18T08:03:56.559 INFO:journalctl@ceph.grafana.a.smithi050.stdout:Sep 18 08:03:56 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-grafana-a[60403]: logger=migrator t=2024-09-18T08:03:56.332558156Z level=info msg="Migration successfully executed" id="managed folder permissions library panel actions migration" duration=132.029µs 2024-09-18T08:03:56.559 INFO:journalctl@ceph.grafana.a.smithi050.stdout:Sep 18 08:03:56 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-grafana-a[60403]: logger=migrator t=2024-09-18T08:03:56.332909414Z level=info msg="Executing migration" id="migrate external alertmanagers to datsourcse" 2024-09-18T08:03:56.559 INFO:journalctl@ceph.grafana.a.smithi050.stdout:Sep 18 08:03:56 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-grafana-a[60403]: logger=migrator t=2024-09-18T08:03:56.333060234Z level=info msg="Migration successfully executed" id="migrate external alertmanagers to datsourcse" duration=150.655µs 2024-09-18T08:03:56.559 INFO:journalctl@ceph.grafana.a.smithi050.stdout:Sep 18 08:03:56 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-grafana-a[60403]: logger=migrator t=2024-09-18T08:03:56.333452963Z level=info msg="Executing migration" id="create folder table" 2024-09-18T08:03:56.559 INFO:journalctl@ceph.grafana.a.smithi050.stdout:Sep 18 08:03:56 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-grafana-a[60403]: logger=migrator t=2024-09-18T08:03:56.334125266Z level=info msg="Migration successfully executed" id="create folder table" duration=672.052µs 2024-09-18T08:03:56.559 INFO:journalctl@ceph.grafana.a.smithi050.stdout:Sep 18 08:03:56 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-grafana-a[60403]: logger=migrator t=2024-09-18T08:03:56.33447278Z level=info msg="Executing migration" id="Add index for parent_uid" 2024-09-18T08:03:56.559 INFO:journalctl@ceph.grafana.a.smithi050.stdout:Sep 18 08:03:56 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-grafana-a[60403]: logger=migrator t=2024-09-18T08:03:56.335338066Z level=info msg="Migration successfully executed" id="Add index for parent_uid" duration=865.515µs 2024-09-18T08:03:56.559 INFO:journalctl@ceph.grafana.a.smithi050.stdout:Sep 18 08:03:56 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-grafana-a[60403]: logger=migrator t=2024-09-18T08:03:56.335685577Z level=info msg="Executing migration" id="Add unique index for folder.uid and folder.org_id" 2024-09-18T08:03:56.560 INFO:journalctl@ceph.grafana.a.smithi050.stdout:Sep 18 08:03:56 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-grafana-a[60403]: logger=migrator t=2024-09-18T08:03:56.336442792Z level=info msg="Migration successfully executed" id="Add unique index for folder.uid and folder.org_id" duration=757.086µs 2024-09-18T08:03:56.560 INFO:journalctl@ceph.grafana.a.smithi050.stdout:Sep 18 08:03:56 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-grafana-a[60403]: logger=migrator t=2024-09-18T08:03:56.336770833Z level=info msg="Executing migration" id="Update folder title length" 2024-09-18T08:03:56.560 INFO:journalctl@ceph.grafana.a.smithi050.stdout:Sep 18 08:03:56 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-grafana-a[60403]: logger=migrator t=2024-09-18T08:03:56.336792511Z level=info msg="Migration successfully executed" id="Update folder title length" duration=22.884µs 2024-09-18T08:03:56.560 INFO:journalctl@ceph.grafana.a.smithi050.stdout:Sep 18 08:03:56 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-grafana-a[60403]: logger=migrator t=2024-09-18T08:03:56.337195661Z level=info msg="Executing migration" id="Add unique index for folder.title and folder.parent_uid" 2024-09-18T08:03:56.560 INFO:journalctl@ceph.grafana.a.smithi050.stdout:Sep 18 08:03:56 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-grafana-a[60403]: logger=migrator t=2024-09-18T08:03:56.33796759Z level=info msg="Migration successfully executed" id="Add unique index for folder.title and folder.parent_uid" duration=772.037µs 2024-09-18T08:03:56.560 INFO:journalctl@ceph.grafana.a.smithi050.stdout:Sep 18 08:03:56 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-grafana-a[60403]: logger=migrator t=2024-09-18T08:03:56.338297773Z level=info msg="Executing migration" id="Remove unique index for folder.title and folder.parent_uid" 2024-09-18T08:03:56.560 INFO:journalctl@ceph.grafana.a.smithi050.stdout:Sep 18 08:03:56 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-grafana-a[60403]: logger=migrator t=2024-09-18T08:03:56.339022204Z level=info msg="Migration successfully executed" id="Remove unique index for folder.title and folder.parent_uid" duration=724.197µs 2024-09-18T08:03:56.560 INFO:journalctl@ceph.grafana.a.smithi050.stdout:Sep 18 08:03:56 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-grafana-a[60403]: logger=migrator t=2024-09-18T08:03:56.339341785Z level=info msg="Executing migration" id="Add unique index for title, parent_uid, and org_id" 2024-09-18T08:03:56.560 INFO:journalctl@ceph.grafana.a.smithi050.stdout:Sep 18 08:03:56 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-grafana-a[60403]: logger=migrator t=2024-09-18T08:03:56.340125667Z level=info msg="Migration successfully executed" id="Add unique index for title, parent_uid, and org_id" duration=784.039µs 2024-09-18T08:03:56.560 INFO:journalctl@ceph.grafana.a.smithi050.stdout:Sep 18 08:03:56 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-grafana-a[60403]: logger=migrator t=2024-09-18T08:03:56.340469693Z level=info msg="Executing migration" id="Sync dashboard and folder table" 2024-09-18T08:03:56.560 INFO:journalctl@ceph.grafana.a.smithi050.stdout:Sep 18 08:03:56 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-grafana-a[60403]: logger=migrator t=2024-09-18T08:03:56.340788526Z level=info msg="Migration successfully executed" id="Sync dashboard and folder table" duration=317.022µs 2024-09-18T08:03:56.561 INFO:journalctl@ceph.grafana.a.smithi050.stdout:Sep 18 08:03:56 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-grafana-a[60403]: logger=migrator t=2024-09-18T08:03:56.341137898Z level=info msg="Executing migration" id="Remove ghost folders from the folder table" 2024-09-18T08:03:56.561 INFO:journalctl@ceph.grafana.a.smithi050.stdout:Sep 18 08:03:56 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-grafana-a[60403]: logger=migrator t=2024-09-18T08:03:56.341357366Z level=info msg="Migration successfully executed" id="Remove ghost folders from the folder table" duration=219.105µs 2024-09-18T08:03:56.561 INFO:journalctl@ceph.grafana.a.smithi050.stdout:Sep 18 08:03:56 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-grafana-a[60403]: logger=migrator t=2024-09-18T08:03:56.341761829Z level=info msg="Executing migration" id="Remove unique index UQE_folder_uid_org_id" 2024-09-18T08:03:56.561 INFO:journalctl@ceph.grafana.a.smithi050.stdout:Sep 18 08:03:56 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-grafana-a[60403]: logger=migrator t=2024-09-18T08:03:56.342609437Z level=info msg="Migration successfully executed" id="Remove unique index UQE_folder_uid_org_id" duration=847.537µs 2024-09-18T08:03:56.561 INFO:journalctl@ceph.grafana.a.smithi050.stdout:Sep 18 08:03:56 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-grafana-a[60403]: logger=migrator t=2024-09-18T08:03:56.342951819Z level=info msg="Executing migration" id="Add unique index UQE_folder_org_id_uid" 2024-09-18T08:03:56.561 INFO:journalctl@ceph.grafana.a.smithi050.stdout:Sep 18 08:03:56 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-grafana-a[60403]: logger=migrator t=2024-09-18T08:03:56.343701103Z level=info msg="Migration successfully executed" id="Add unique index UQE_folder_org_id_uid" duration=749.112µs 2024-09-18T08:03:56.561 INFO:journalctl@ceph.grafana.a.smithi050.stdout:Sep 18 08:03:56 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-grafana-a[60403]: logger=migrator t=2024-09-18T08:03:56.344032298Z level=info msg="Executing migration" id="Remove unique index UQE_folder_title_parent_uid_org_id" 2024-09-18T08:03:56.561 INFO:journalctl@ceph.grafana.a.smithi050.stdout:Sep 18 08:03:56 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-grafana-a[60403]: logger=migrator t=2024-09-18T08:03:56.344746858Z level=info msg="Migration successfully executed" id="Remove unique index UQE_folder_title_parent_uid_org_id" duration=714.909µs 2024-09-18T08:03:56.561 INFO:journalctl@ceph.grafana.a.smithi050.stdout:Sep 18 08:03:56 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-grafana-a[60403]: logger=migrator t=2024-09-18T08:03:56.345100701Z level=info msg="Executing migration" id="Add unique index UQE_folder_org_id_parent_uid_title" 2024-09-18T08:03:56.561 INFO:journalctl@ceph.grafana.a.smithi050.stdout:Sep 18 08:03:56 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-grafana-a[60403]: logger=migrator t=2024-09-18T08:03:56.345869703Z level=info msg="Migration successfully executed" id="Add unique index UQE_folder_org_id_parent_uid_title" duration=767.257µs 2024-09-18T08:03:56.561 INFO:journalctl@ceph.grafana.a.smithi050.stdout:Sep 18 08:03:56 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-grafana-a[60403]: logger=migrator t=2024-09-18T08:03:56.346199685Z level=info msg="Executing migration" id="Remove index IDX_folder_parent_uid_org_id" 2024-09-18T08:03:56.561 INFO:journalctl@ceph.grafana.a.smithi050.stdout:Sep 18 08:03:56 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-grafana-a[60403]: logger=migrator t=2024-09-18T08:03:56.346910966Z level=info msg="Migration successfully executed" id="Remove index IDX_folder_parent_uid_org_id" duration=711.668µs 2024-09-18T08:03:56.561 INFO:journalctl@ceph.grafana.a.smithi050.stdout:Sep 18 08:03:56 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-grafana-a[60403]: logger=migrator t=2024-09-18T08:03:56.347235199Z level=info msg="Executing migration" id="create anon_device table" 2024-09-18T08:03:56.562 INFO:journalctl@ceph.grafana.a.smithi050.stdout:Sep 18 08:03:56 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-grafana-a[60403]: logger=migrator t=2024-09-18T08:03:56.347822994Z level=info msg="Migration successfully executed" id="create anon_device table" duration=587.588µs 2024-09-18T08:03:56.562 INFO:journalctl@ceph.grafana.a.smithi050.stdout:Sep 18 08:03:56 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-grafana-a[60403]: logger=migrator t=2024-09-18T08:03:56.348181887Z level=info msg="Executing migration" id="add unique index anon_device.device_id" 2024-09-18T08:03:56.562 INFO:journalctl@ceph.grafana.a.smithi050.stdout:Sep 18 08:03:56 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-grafana-a[60403]: logger=migrator t=2024-09-18T08:03:56.349030022Z level=info msg="Migration successfully executed" id="add unique index anon_device.device_id" duration=848.02µs 2024-09-18T08:03:56.562 INFO:journalctl@ceph.grafana.a.smithi050.stdout:Sep 18 08:03:56 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-grafana-a[60403]: logger=migrator t=2024-09-18T08:03:56.349395397Z level=info msg="Executing migration" id="add index anon_device.updated_at" 2024-09-18T08:03:56.562 INFO:journalctl@ceph.grafana.a.smithi050.stdout:Sep 18 08:03:56 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-grafana-a[60403]: logger=migrator t=2024-09-18T08:03:56.350153238Z level=info msg="Migration successfully executed" id="add index anon_device.updated_at" duration=757.581µs 2024-09-18T08:03:56.562 INFO:journalctl@ceph.grafana.a.smithi050.stdout:Sep 18 08:03:56 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-grafana-a[60403]: logger=migrator t=2024-09-18T08:03:56.350499656Z level=info msg="Executing migration" id="create signing_key table" 2024-09-18T08:03:56.562 INFO:journalctl@ceph.grafana.a.smithi050.stdout:Sep 18 08:03:56 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-grafana-a[60403]: logger=migrator t=2024-09-18T08:03:56.35118692Z level=info msg="Migration successfully executed" id="create signing_key table" duration=687.012µs 2024-09-18T08:03:56.562 INFO:journalctl@ceph.grafana.a.smithi050.stdout:Sep 18 08:03:56 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-grafana-a[60403]: logger=migrator t=2024-09-18T08:03:56.351645532Z level=info msg="Executing migration" id="add unique index signing_key.key_id" 2024-09-18T08:03:56.562 INFO:journalctl@ceph.grafana.a.smithi050.stdout:Sep 18 08:03:56 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-grafana-a[60403]: logger=migrator t=2024-09-18T08:03:56.352471973Z level=info msg="Migration successfully executed" id="add unique index signing_key.key_id" duration=826.047µs 2024-09-18T08:03:56.562 INFO:journalctl@ceph.grafana.a.smithi050.stdout:Sep 18 08:03:56 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-grafana-a[60403]: logger=migrator t=2024-09-18T08:03:56.352813181Z level=info msg="Executing migration" id="set legacy alert migration status in kvstore" 2024-09-18T08:03:56.562 INFO:journalctl@ceph.grafana.a.smithi050.stdout:Sep 18 08:03:56 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-grafana-a[60403]: logger=migrator t=2024-09-18T08:03:56.353532986Z level=info msg="Migration successfully executed" id="set legacy alert migration status in kvstore" duration=719.723µs 2024-09-18T08:03:56.562 INFO:journalctl@ceph.grafana.a.smithi050.stdout:Sep 18 08:03:56 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-grafana-a[60403]: logger=migrator t=2024-09-18T08:03:56.353836444Z level=info msg="Executing migration" id="migrate record of created folders during legacy migration to kvstore" 2024-09-18T08:03:56.563 INFO:journalctl@ceph.grafana.a.smithi050.stdout:Sep 18 08:03:56 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-grafana-a[60403]: logger=migrator t=2024-09-18T08:03:56.354026263Z level=info msg="Migration successfully executed" id="migrate record of created folders during legacy migration to kvstore" duration=190.347µs 2024-09-18T08:03:56.563 INFO:journalctl@ceph.grafana.a.smithi050.stdout:Sep 18 08:03:56 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-grafana-a[60403]: logger=migrator t=2024-09-18T08:03:56.354383294Z level=info msg="Executing migration" id="Add folder_uid for dashboard" 2024-09-18T08:03:56.563 INFO:journalctl@ceph.grafana.a.smithi050.stdout:Sep 18 08:03:56 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-grafana-a[60403]: logger=migrator t=2024-09-18T08:03:56.360812959Z level=info msg="Migration successfully executed" id="Add folder_uid for dashboard" duration=6.427579ms 2024-09-18T08:03:56.563 INFO:journalctl@ceph.grafana.a.smithi050.stdout:Sep 18 08:03:56 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-grafana-a[60403]: logger=migrator t=2024-09-18T08:03:56.361255367Z level=info msg="Executing migration" id="Populate dashboard folder_uid column" 2024-09-18T08:03:56.563 INFO:journalctl@ceph.grafana.a.smithi050.stdout:Sep 18 08:03:56 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-grafana-a[60403]: logger=migrator t=2024-09-18T08:03:56.361782973Z level=info msg="Migration successfully executed" id="Populate dashboard folder_uid column" duration=528.351µs 2024-09-18T08:03:56.563 INFO:journalctl@ceph.grafana.a.smithi050.stdout:Sep 18 08:03:56 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-grafana-a[60403]: logger=migrator t=2024-09-18T08:03:56.362183085Z level=info msg="Executing migration" id="Add unique index for dashboard_org_id_folder_uid_title" 2024-09-18T08:03:56.563 INFO:journalctl@ceph.grafana.a.smithi050.stdout:Sep 18 08:03:56 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-grafana-a[60403]: logger=migrator t=2024-09-18T08:03:56.362940913Z level=info msg="Migration successfully executed" id="Add unique index for dashboard_org_id_folder_uid_title" duration=757.524µs 2024-09-18T08:03:56.563 INFO:journalctl@ceph.grafana.a.smithi050.stdout:Sep 18 08:03:56 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-grafana-a[60403]: logger=migrator t=2024-09-18T08:03:56.363279769Z level=info msg="Executing migration" id="Delete unique index for dashboard_org_id_folder_id_title" 2024-09-18T08:03:56.564 INFO:journalctl@ceph.grafana.a.smithi050.stdout:Sep 18 08:03:56 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-grafana-a[60403]: logger=migrator t=2024-09-18T08:03:56.364020111Z level=info msg="Migration successfully executed" id="Delete unique index for dashboard_org_id_folder_id_title" duration=744.177µs 2024-09-18T08:03:56.564 INFO:journalctl@ceph.grafana.a.smithi050.stdout:Sep 18 08:03:56 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-grafana-a[60403]: logger=migrator t=2024-09-18T08:03:56.364345714Z level=info msg="Executing migration" id="Delete unique index for dashboard_org_id_folder_uid_title" 2024-09-18T08:03:56.564 INFO:journalctl@ceph.grafana.a.smithi050.stdout:Sep 18 08:03:56 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-grafana-a[60403]: logger=migrator t=2024-09-18T08:03:56.365075308Z level=info msg="Migration successfully executed" id="Delete unique index for dashboard_org_id_folder_uid_title" duration=729.463µs 2024-09-18T08:03:56.564 INFO:journalctl@ceph.grafana.a.smithi050.stdout:Sep 18 08:03:56 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-grafana-a[60403]: logger=migrator t=2024-09-18T08:03:56.365386065Z level=info msg="Executing migration" id="Add unique index for dashboard_org_id_folder_uid_title_is_folder" 2024-09-18T08:03:56.564 INFO:journalctl@ceph.grafana.a.smithi050.stdout:Sep 18 08:03:56 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-grafana-a[60403]: logger=migrator t=2024-09-18T08:03:56.366134486Z level=info msg="Migration successfully executed" id="Add unique index for dashboard_org_id_folder_uid_title_is_folder" duration=748.036µs 2024-09-18T08:03:56.564 INFO:journalctl@ceph.grafana.a.smithi050.stdout:Sep 18 08:03:56 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-grafana-a[60403]: logger=migrator t=2024-09-18T08:03:56.366454187Z level=info msg="Executing migration" id="Restore index for dashboard_org_id_folder_id_title" 2024-09-18T08:03:56.564 INFO:journalctl@ceph.grafana.a.smithi050.stdout:Sep 18 08:03:56 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-grafana-a[60403]: logger=migrator t=2024-09-18T08:03:56.367164614Z level=info msg="Migration successfully executed" id="Restore index for dashboard_org_id_folder_id_title" duration=710.356µs 2024-09-18T08:03:56.565 INFO:journalctl@ceph.grafana.a.smithi050.stdout:Sep 18 08:03:56 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-grafana-a[60403]: logger=migrator t=2024-09-18T08:03:56.367479445Z level=info msg="Executing migration" id="create sso_setting table" 2024-09-18T08:03:56.565 INFO:journalctl@ceph.grafana.a.smithi050.stdout:Sep 18 08:03:56 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-grafana-a[60403]: logger=migrator t=2024-09-18T08:03:56.36817619Z level=info msg="Migration successfully executed" id="create sso_setting table" duration=698.272µs 2024-09-18T08:03:56.565 INFO:journalctl@ceph.grafana.a.smithi050.stdout:Sep 18 08:03:56 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-grafana-a[60403]: logger=migrator t=2024-09-18T08:03:56.368529542Z level=info msg="Executing migration" id="copy kvstore migration status to each org" 2024-09-18T08:03:56.565 INFO:journalctl@ceph.grafana.a.smithi050.stdout:Sep 18 08:03:56 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-grafana-a[60403]: logger=migrator t=2024-09-18T08:03:56.369044181Z level=info msg="Migration successfully executed" id="copy kvstore migration status to each org" duration=515.313µs 2024-09-18T08:03:56.565 INFO:journalctl@ceph.grafana.a.smithi050.stdout:Sep 18 08:03:56 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-grafana-a[60403]: logger=migrator t=2024-09-18T08:03:56.369361688Z level=info msg="Executing migration" id="add back entry for orgid=0 migrated status" 2024-09-18T08:03:56.565 INFO:journalctl@ceph.grafana.a.smithi050.stdout:Sep 18 08:03:56 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-grafana-a[60403]: logger=migrator t=2024-09-18T08:03:56.369527337Z level=info msg="Migration successfully executed" id="add back entry for orgid=0 migrated status" duration=166.239µs 2024-09-18T08:03:56.565 INFO:journalctl@ceph.grafana.a.smithi050.stdout:Sep 18 08:03:56 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-grafana-a[60403]: logger=migrator t=2024-09-18T08:03:56.369912506Z level=info msg="Executing migration" id="alter kv_store.value to longtext" 2024-09-18T08:03:56.565 INFO:journalctl@ceph.grafana.a.smithi050.stdout:Sep 18 08:03:56 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-grafana-a[60403]: logger=migrator t=2024-09-18T08:03:56.369955733Z level=info msg="Migration successfully executed" id="alter kv_store.value to longtext" duration=41.886µs 2024-09-18T08:03:56.566 INFO:journalctl@ceph.grafana.a.smithi050.stdout:Sep 18 08:03:56 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-grafana-a[60403]: logger=migrator t=2024-09-18T08:03:56.370347343Z level=info msg="Executing migration" id="add notification_settings column to alert_rule table" 2024-09-18T08:03:56.566 INFO:journalctl@ceph.grafana.a.smithi050.stdout:Sep 18 08:03:56 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-grafana-a[60403]: logger=migrator t=2024-09-18T08:03:56.375975462Z level=info msg="Migration successfully executed" id="add notification_settings column to alert_rule table" duration=5.626971ms 2024-09-18T08:03:56.566 INFO:journalctl@ceph.grafana.a.smithi050.stdout:Sep 18 08:03:56 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-grafana-a[60403]: logger=migrator t=2024-09-18T08:03:56.376308115Z level=info msg="Executing migration" id="add notification_settings column to alert_rule_version table" 2024-09-18T08:03:56.566 INFO:journalctl@ceph.grafana.a.smithi050.stdout:Sep 18 08:03:56 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-grafana-a[60403]: logger=migrator t=2024-09-18T08:03:56.382096699Z level=info msg="Migration successfully executed" id="add notification_settings column to alert_rule_version table" duration=5.788175ms 2024-09-18T08:03:56.566 INFO:journalctl@ceph.grafana.a.smithi050.stdout:Sep 18 08:03:56 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-grafana-a[60403]: logger=migrator t=2024-09-18T08:03:56.382411403Z level=info msg="Executing migration" id="removing scope from alert.instances:read action migration" 2024-09-18T08:03:56.566 INFO:journalctl@ceph.grafana.a.smithi050.stdout:Sep 18 08:03:56 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-grafana-a[60403]: logger=migrator t=2024-09-18T08:03:56.382634562Z level=info msg="Migration successfully executed" id="removing scope from alert.instances:read action migration" duration=223.272µs 2024-09-18T08:03:56.566 INFO:journalctl@ceph.grafana.a.smithi050.stdout:Sep 18 08:03:56 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-grafana-a[60403]: logger=migrator t=2024-09-18T08:03:56.382970082Z level=info msg="migrations completed" performed=547 skipped=0 duration=968.805941ms 2024-09-18T08:03:56.566 INFO:journalctl@ceph.grafana.a.smithi050.stdout:Sep 18 08:03:56 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-grafana-a[60403]: logger=sqlstore t=2024-09-18T08:03:56.383837537Z level=info msg="Created default organization" 2024-09-18T08:03:56.566 INFO:journalctl@ceph.grafana.a.smithi050.stdout:Sep 18 08:03:56 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-grafana-a[60403]: logger=secrets t=2024-09-18T08:03:56.384352578Z level=info msg="Envelope encryption state" enabled=true currentprovider=secretKey.v1 2024-09-18T08:03:56.566 INFO:journalctl@ceph.grafana.a.smithi050.stdout:Sep 18 08:03:56 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-grafana-a[60403]: logger=plugin.store t=2024-09-18T08:03:56.399649129Z level=info msg="Loading plugins..." 2024-09-18T08:03:56.566 INFO:journalctl@ceph.grafana.a.smithi050.stdout:Sep 18 08:03:56 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-grafana-a[60403]: logger=local.finder t=2024-09-18T08:03:56.445523453Z level=warn msg="Skipping finding plugins as directory does not exist" path=/usr/share/grafana/plugins-bundled 2024-09-18T08:03:56.566 INFO:journalctl@ceph.grafana.a.smithi050.stdout:Sep 18 08:03:56 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-grafana-a[60403]: logger=plugin.store t=2024-09-18T08:03:56.445542063Z level=info msg="Plugins loaded" count=55 duration=45.893632ms 2024-09-18T08:03:56.566 INFO:journalctl@ceph.grafana.a.smithi050.stdout:Sep 18 08:03:56 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-grafana-a[60403]: logger=query_data t=2024-09-18T08:03:56.450255897Z level=info msg="Query Service initialization" 2024-09-18T08:03:56.567 INFO:journalctl@ceph.grafana.a.smithi050.stdout:Sep 18 08:03:56 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-grafana-a[60403]: logger=live.push_http t=2024-09-18T08:03:56.453223719Z level=info msg="Live Push Gateway initialization" 2024-09-18T08:03:56.567 INFO:journalctl@ceph.grafana.a.smithi050.stdout:Sep 18 08:03:56 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-grafana-a[60403]: logger=ngalert.migration t=2024-09-18T08:03:56.455097861Z level=info msg=Starting 2024-09-18T08:03:56.567 INFO:journalctl@ceph.grafana.a.smithi050.stdout:Sep 18 08:03:56 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-grafana-a[60403]: logger=ngalert.migration t=2024-09-18T08:03:56.455648217Z level=info msg="Applying transition" currentType=Legacy desiredType=UnifiedAlerting cleanOnDowngrade=false cleanOnUpgrade=false 2024-09-18T08:03:56.567 INFO:journalctl@ceph.grafana.a.smithi050.stdout:Sep 18 08:03:56 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-grafana-a[60403]: logger=ngalert.migration orgID=1 t=2024-09-18T08:03:56.456124369Z level=info msg="Migrating alerts for organisation" 2024-09-18T08:03:56.567 INFO:journalctl@ceph.grafana.a.smithi050.stdout:Sep 18 08:03:56 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-grafana-a[60403]: logger=ngalert.migration orgID=1 t=2024-09-18T08:03:56.456728035Z level=info msg="Alerts found to migrate" alerts=0 2024-09-18T08:03:56.567 INFO:journalctl@ceph.grafana.a.smithi050.stdout:Sep 18 08:03:56 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-grafana-a[60403]: logger=ngalert.migration t=2024-09-18T08:03:56.459688576Z level=info msg="Completed alerting migration" 2024-09-18T08:03:56.567 INFO:journalctl@ceph.grafana.a.smithi050.stdout:Sep 18 08:03:56 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-grafana-a[60403]: logger=ngalert.state.manager t=2024-09-18T08:03:56.473542114Z level=info msg="Running in alternative execution of Error/NoData mode" 2024-09-18T08:03:56.567 INFO:journalctl@ceph.grafana.a.smithi050.stdout:Sep 18 08:03:56 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-grafana-a[60403]: logger=infra.usagestats.collector t=2024-09-18T08:03:56.474971521Z level=info msg="registering usage stat providers" usageStatsProvidersLen=2 2024-09-18T08:03:56.567 INFO:journalctl@ceph.grafana.a.smithi050.stdout:Sep 18 08:03:56 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-grafana-a[60403]: logger=provisioning.datasources t=2024-09-18T08:03:56.476752646Z level=info msg="inserting datasource from configuration" name=Dashboard1 uid=P43CA22E17D0F9596 2024-09-18T08:03:56.567 INFO:journalctl@ceph.grafana.a.smithi050.stdout:Sep 18 08:03:56 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-grafana-a[60403]: logger=provisioning.datasources t=2024-09-18T08:03:56.489710048Z level=info msg="inserting datasource from configuration" name=Loki uid=P8E80F9AEF21F6940 2024-09-18T08:03:56.567 INFO:journalctl@ceph.grafana.a.smithi050.stdout:Sep 18 08:03:56 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-grafana-a[60403]: logger=provisioning.alerting t=2024-09-18T08:03:56.503359954Z level=info msg="starting to provision alerting" 2024-09-18T08:03:56.567 INFO:journalctl@ceph.grafana.a.smithi050.stdout:Sep 18 08:03:56 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-grafana-a[60403]: logger=provisioning.alerting t=2024-09-18T08:03:56.503377752Z level=info msg="finished to provision alerting" 2024-09-18T08:03:56.567 INFO:journalctl@ceph.grafana.a.smithi050.stdout:Sep 18 08:03:56 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-grafana-a[60403]: logger=grafanaStorageLogger t=2024-09-18T08:03:56.50351629Z level=info msg="Storage starting" 2024-09-18T08:03:56.568 INFO:journalctl@ceph.grafana.a.smithi050.stdout:Sep 18 08:03:56 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-grafana-a[60403]: logger=http.server t=2024-09-18T08:03:56.508096106Z level=info msg="HTTP Server TLS settings" MinTLSVersion=TLS1.2 configuredciphers=TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA,TLS_ECDHE_ECDSA_WITH_AES_256_CBC_SHA,TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA,TLS_RSA_WITH_AES_128_GCM_SHA256,TLS_RSA_WITH_AES_256_GCM_SHA384,TLS_RSA_WITH_AES_128_CBC_SHA,TLS_RSA_WITH_AES_256_CBC_SHA 2024-09-18T08:03:56.568 INFO:journalctl@ceph.grafana.a.smithi050.stdout:Sep 18 08:03:56 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-grafana-a[60403]: logger=http.server t=2024-09-18T08:03:56.508441296Z level=info msg="HTTP Server Listen" address=[::]:3000 protocol=https subUrl= socket= 2024-09-18T08:03:56.568 INFO:journalctl@ceph.grafana.a.smithi050.stdout:Sep 18 08:03:56 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-grafana-a[60403]: logger=ngalert.state.manager t=2024-09-18T08:03:56.508521382Z level=info msg="Warming state cache for startup" 2024-09-18T08:03:56.568 INFO:journalctl@ceph.grafana.a.smithi050.stdout:Sep 18 08:03:56 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-grafana-a[60403]: logger=ngalert.multiorg.alertmanager t=2024-09-18T08:03:56.515701077Z level=info msg="Starting MultiOrg Alertmanager" 2024-09-18T08:03:56.568 INFO:journalctl@ceph.grafana.a.smithi050.stdout:Sep 18 08:03:56 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-grafana-a[60403]: logger=provisioning.dashboard t=2024-09-18T08:03:56.539100553Z level=info msg="starting to provision dashboards" 2024-09-18T08:03:56.568 INFO:journalctl@ceph.grafana.a.smithi050.stdout:Sep 18 08:03:56 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-grafana-a[60403]: logger=sqlstore.transactions t=2024-09-18T08:03:56.543080776Z level=info msg="Database locked, sleeping then retrying" error="database is locked" retry=0 code="database is locked" 2024-09-18T08:03:56.568 INFO:journalctl@ceph.grafana.a.smithi050.stdout:Sep 18 08:03:56 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-grafana-a[60403]: logger=ngalert.state.manager t=2024-09-18T08:03:56.551535209Z level=info msg="State cache has been initialized" states=0 duration=43.01289ms 2024-09-18T08:03:56.568 INFO:journalctl@ceph.grafana.a.smithi050.stdout:Sep 18 08:03:56 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-grafana-a[60403]: logger=ngalert.scheduler t=2024-09-18T08:03:56.551557851Z level=info msg="Starting scheduler" tickInterval=10s maxAttempts=1 2024-09-18T08:03:56.568 INFO:journalctl@ceph.grafana.a.smithi050.stdout:Sep 18 08:03:56 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-grafana-a[60403]: logger=ticker t=2024-09-18T08:03:56.551598658Z level=info msg=starting first_tick=2024-09-18T08:04:00Z 2024-09-18T08:03:56.568 INFO:journalctl@ceph.grafana.a.smithi050.stdout:Sep 18 08:03:56 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-grafana-a[60403]: logger=sqlstore.transactions t=2024-09-18T08:03:56.554242603Z level=info msg="Database locked, sleeping then retrying" error="database is locked" retry=1 code="database is locked" 2024-09-18T08:03:56.580 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4 2024-09-18T08:03:56.602 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:03:56.603 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:03:56.603 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4' 2024-09-18T08:03:56.697 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:03:56 smithi007 ceph-mon[29459]: pgmap v31: 132 pgs: 132 active+clean; 583 KiB data, 223 MiB used, 715 GiB / 715 GiB avail; 849 B/s rd, 0 op/s 2024-09-18T08:03:56.697 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:03:56 smithi007 ceph-mon[29459]: from='mgr.24470 172.21.15.7:0/2594390261' entity='mgr.y' 2024-09-18T08:03:56.697 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:03:56 smithi007 ceph-mon[29459]: from='mgr.24470 172.21.15.7:0/2594390261' entity='mgr.y' 2024-09-18T08:03:56.697 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:03:56 smithi007 ceph-mon[29459]: from='mgr.24470 172.21.15.7:0/2594390261' entity='mgr.y' 2024-09-18T08:03:56.698 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:03:56 smithi007 ceph-mon[29459]: from='mgr.24470 172.21.15.7:0/2594390261' entity='mgr.y' 2024-09-18T08:03:56.698 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:03:56 smithi007 ceph-mon[29459]: Deploying daemon node-exporter.a on smithi007 2024-09-18T08:03:56.698 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:03:56 smithi007 ceph-mon[29459]: from='client.? 172.21.15.7:0/1814546716' entity='client.admin' cmd={"prefix": "osd pool create", "pool": "unique_pool_0", "pg_num": 16} : dispatch 2024-09-18T08:03:56.698 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:03:56 smithi007 ceph-mon[39249]: pgmap v31: 132 pgs: 132 active+clean; 583 KiB data, 223 MiB used, 715 GiB / 715 GiB avail; 849 B/s rd, 0 op/s 2024-09-18T08:03:56.698 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:03:56 smithi007 ceph-mon[39249]: from='mgr.24470 172.21.15.7:0/2594390261' entity='mgr.y' 2024-09-18T08:03:56.698 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:03:56 smithi007 ceph-mon[39249]: from='mgr.24470 172.21.15.7:0/2594390261' entity='mgr.y' 2024-09-18T08:03:56.698 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:03:56 smithi007 ceph-mon[39249]: from='mgr.24470 172.21.15.7:0/2594390261' entity='mgr.y' 2024-09-18T08:03:56.698 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:03:56 smithi007 ceph-mon[39249]: from='mgr.24470 172.21.15.7:0/2594390261' entity='mgr.y' 2024-09-18T08:03:56.698 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:03:56 smithi007 ceph-mon[39249]: Deploying daemon node-exporter.a on smithi007 2024-09-18T08:03:56.698 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:03:56 smithi007 ceph-mon[39249]: from='client.? 172.21.15.7:0/1814546716' entity='client.admin' cmd={"prefix": "osd pool create", "pool": "unique_pool_0", "pg_num": 16} : dispatch 2024-09-18T08:03:56.704 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5 2024-09-18T08:03:56.727 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:03:56.728 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:03:56.728 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5' 2024-09-18T08:03:56.823 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid 779349ca-7593-11ef-bceb-c7b262605968 -- ceph osd set nodeep-scrub 2024-09-18T08:03:56.829 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0 2024-09-18T08:03:56.890 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:03:56.891 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:03:56.891 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0' 2024-09-18T08:03:56.906 INFO:journalctl@ceph.grafana.a.smithi050.stdout:Sep 18 08:03:56 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-grafana-a[60403]: logger=sqlstore.transactions t=2024-09-18T08:03:56.56520799Z level=info msg="Database locked, sleeping then retrying" error="database is locked" retry=2 code="database is locked" 2024-09-18T08:03:56.906 INFO:journalctl@ceph.grafana.a.smithi050.stdout:Sep 18 08:03:56 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-grafana-a[60403]: logger=sqlstore.transactions t=2024-09-18T08:03:56.576250249Z level=info msg="Database locked, sleeping then retrying" error="database is locked" retry=3 code="database is locked" 2024-09-18T08:03:56.906 INFO:journalctl@ceph.grafana.a.smithi050.stdout:Sep 18 08:03:56 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-grafana-a[60403]: logger=plugins.update.checker t=2024-09-18T08:03:56.592629383Z level=info msg="Update check succeeded" duration=89.134192ms 2024-09-18T08:03:56.906 INFO:journalctl@ceph.grafana.a.smithi050.stdout:Sep 18 08:03:56 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-grafana-a[60403]: logger=grafana.update.checker t=2024-09-18T08:03:56.598182398Z level=info msg="Update check succeeded" duration=94.678433ms 2024-09-18T08:03:56.906 INFO:journalctl@ceph.grafana.a.smithi050.stdout:Sep 18 08:03:56 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-grafana-a[60403]: logger=provisioning.dashboard t=2024-09-18T08:03:56.72148861Z level=info msg="finished to provision dashboards" 2024-09-18T08:03:56.980 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Sep 18 08:03:56 smithi007 ceph-779349ca-7593-11ef-bceb-c7b262605968-mgr-y[29688]: ::ffff:172.21.15.50 - - [18/Sep/2024:08:03:56] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-09-18T08:03:56.992 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5 2024-09-18T08:03:57.014 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:03:57.014 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:03:57.015 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5' 2024-09-18T08:03:57.062 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid 779349ca-7593-11ef-bceb-c7b262605968 -- ceph daemon osd.0 dump_historic_ops 2024-09-18T08:03:57.116 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1 2024-09-18T08:03:57.138 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:03:57.138 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:03:57.139 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1' 2024-09-18T08:03:57.239 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3 2024-09-18T08:03:57.262 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:03:57.262 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:03:57.263 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3' 2024-09-18T08:03:57.364 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5 2024-09-18T08:03:57.381 INFO:journalctl@ceph.grafana.a.smithi050.stdout:Sep 18 08:03:56 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-grafana-a[60403]: logger=grafana-apiserver t=2024-09-18T08:03:56.961873019Z level=info msg="Adding GroupVersion playlist.grafana.app v0alpha1 to ResourceManager" 2024-09-18T08:03:57.381 INFO:journalctl@ceph.grafana.a.smithi050.stdout:Sep 18 08:03:56 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-grafana-a[60403]: logger=grafana-apiserver t=2024-09-18T08:03:56.962335984Z level=info msg="Adding GroupVersion featuretoggle.grafana.app v0alpha1 to ResourceManager" 2024-09-18T08:03:57.386 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:03:57.387 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:03:57.387 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5' 2024-09-18T08:03:57.488 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4 2024-09-18T08:03:57.510 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:03:57.511 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:03:57.512 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4' 2024-09-18T08:03:57.612 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6 2024-09-18T08:03:57.634 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:03:57.635 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:03:57.636 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6' 2024-09-18T08:03:57.656 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:03:57 smithi050 ceph-mon[32371]: from='client.? 172.21.15.7:0/1814546716' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "unique_pool_0", "pg_num": 16}]': finished 2024-09-18T08:03:57.656 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:03:57 smithi050 ceph-mon[32371]: osdmap e83: 8 total, 8 up, 8 in 2024-09-18T08:03:57.731 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:03:57 smithi007 ceph-mon[29459]: from='client.? 172.21.15.7:0/1814546716' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "unique_pool_0", "pg_num": 16}]': finished 2024-09-18T08:03:57.731 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:03:57 smithi007 ceph-mon[29459]: osdmap e83: 8 total, 8 up, 8 in 2024-09-18T08:03:57.731 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:03:57 smithi007 ceph-mon[39249]: from='client.? 172.21.15.7:0/1814546716' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "unique_pool_0", "pg_num": 16}]': finished 2024-09-18T08:03:57.731 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:03:57 smithi007 ceph-mon[39249]: osdmap e83: 8 total, 8 up, 8 in 2024-09-18T08:03:57.737 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0 2024-09-18T08:03:57.760 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:03:57.761 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:03:57.761 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0' 2024-09-18T08:03:57.840 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid 779349ca-7593-11ef-bceb-c7b262605968 -- ceph osd pool application enable unique_pool_0 rados --yes-i-really-mean-it || true 2024-09-18T08:03:57.862 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3 2024-09-18T08:03:57.877 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/779349ca-7593-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T08:03:57.877 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/779349ca-7593-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T08:03:57.916 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:03:57.917 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:03:57.917 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3' 2024-09-18T08:03:57.935 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/779349ca-7593-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T08:03:58.018 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0 2024-09-18T08:03:58.041 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:03:58.041 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:03:58.042 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0' 2024-09-18T08:03:58.142 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1 2024-09-18T08:03:58.171 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:03:58.172 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:03:58.172 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1' 2024-09-18T08:03:58.273 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5 2024-09-18T08:03:58.298 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:03:58.299 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:03:58.299 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5' 2024-09-18T08:03:58.400 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3 2024-09-18T08:03:58.423 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:03:58.424 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:03:58.424 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3' 2024-09-18T08:03:58.525 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0 2024-09-18T08:03:58.548 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:03:58.549 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:03:58.549 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0' 2024-09-18T08:03:58.650 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5 2024-09-18T08:03:58.679 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:03:58.680 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:03:58.680 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5' 2024-09-18T08:03:58.715 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:03:58 smithi007 ceph-mon[29459]: pgmap v33: 148 pgs: 16 unknown, 132 active+clean; 583 KiB data, 223 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-18T08:03:58.715 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:03:58 smithi007 ceph-mon[29459]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-18T08:03:58.715 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:03:58 smithi007 ceph-mon[29459]: osdmap e84: 8 total, 8 up, 8 in 2024-09-18T08:03:58.715 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:03:58 smithi007 ceph-mon[29459]: from='mgr.24470 172.21.15.7:0/2594390261' entity='mgr.y' 2024-09-18T08:03:58.716 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:03:58 smithi007 ceph-mon[39249]: pgmap v33: 148 pgs: 16 unknown, 132 active+clean; 583 KiB data, 223 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-18T08:03:58.716 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:03:58 smithi007 ceph-mon[39249]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-18T08:03:58.716 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:03:58 smithi007 ceph-mon[39249]: osdmap e84: 8 total, 8 up, 8 in 2024-09-18T08:03:58.716 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:03:58 smithi007 ceph-mon[39249]: from='mgr.24470 172.21.15.7:0/2594390261' entity='mgr.y' 2024-09-18T08:03:58.781 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7 2024-09-18T08:03:58.808 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:03:58.809 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:03:58.810 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7' 2024-09-18T08:03:58.906 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:03:58 smithi050 ceph-mon[32371]: pgmap v33: 148 pgs: 16 unknown, 132 active+clean; 583 KiB data, 223 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-18T08:03:58.906 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:03:58 smithi050 ceph-mon[32371]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-18T08:03:58.906 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:03:58 smithi050 ceph-mon[32371]: osdmap e84: 8 total, 8 up, 8 in 2024-09-18T08:03:58.906 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:03:58 smithi050 ceph-mon[32371]: from='mgr.24470 172.21.15.7:0/2594390261' entity='mgr.y' 2024-09-18T08:03:58.907 INFO:journalctl@ceph.iscsi.iscsi.a.smithi050.stdout:Sep 18 08:03:58 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-iscsi-iscsi-a[57782]: debug there is no tcmu-runner data available 2024-09-18T08:03:58.911 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5 2024-09-18T08:03:58.938 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:03:58.938 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:03:58.939 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5' 2024-09-18T08:03:59.040 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0 2024-09-18T08:03:59.054 INFO:teuthology.orchestra.run.smithi007.stderr:nodeep-scrub is set 2024-09-18T08:03:59.069 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:03:59.069 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:03:59.070 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0' 2024-09-18T08:03:59.149 INFO:teuthology.orchestra.run.smithi007.stdout:ERROR: (22) Invalid argument 2024-09-18T08:03:59.149 INFO:teuthology.orchestra.run.smithi007.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-18T08:03:59.171 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1 2024-09-18T08:03:59.192 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:03:59.193 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:03:59.194 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1' 2024-09-18T08:03:59.295 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6 2024-09-18T08:03:59.322 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:03:59.322 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:03:59.323 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6' 2024-09-18T08:03:59.424 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7 2024-09-18T08:03:59.450 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:03:59.451 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:03:59.452 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7' 2024-09-18T08:03:59.552 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5 2024-09-18T08:03:59.580 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:03:59.580 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:03:59.581 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5' 2024-09-18T08:03:59.672 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:03:59 smithi007 ceph-mon[29459]: osdmap e85: 8 total, 8 up, 8 in 2024-09-18T08:03:59.672 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:03:59 smithi007 ceph-mon[29459]: from='client.? 172.21.15.7:0/3639046434' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-18T08:03:59.672 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:03:59 smithi007 ceph-mon[29459]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-18T08:03:59.672 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:03:59 smithi007 ceph-mon[29459]: from='client.? 172.21.15.7:0/3639046434' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-18T08:03:59.672 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:03:59 smithi007 ceph-mon[29459]: osdmap e86: 8 total, 8 up, 8 in 2024-09-18T08:03:59.673 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:03:59 smithi007 ceph-mon[39249]: osdmap e85: 8 total, 8 up, 8 in 2024-09-18T08:03:59.673 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:03:59 smithi007 ceph-mon[39249]: from='client.? 172.21.15.7:0/3639046434' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-18T08:03:59.673 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:03:59 smithi007 ceph-mon[39249]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-18T08:03:59.673 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:03:59 smithi007 ceph-mon[39249]: from='client.? 172.21.15.7:0/3639046434' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-18T08:03:59.673 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:03:59 smithi007 ceph-mon[39249]: osdmap e86: 8 total, 8 up, 8 in 2024-09-18T08:03:59.681 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6 2024-09-18T08:03:59.707 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:03:59.708 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:03:59.708 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6' 2024-09-18T08:03:59.809 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7 2024-09-18T08:03:59.836 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:03:59.836 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:03:59.837 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7' 2024-09-18T08:03:59.847 INFO:teuthology.orchestra.run.smithi007.stdout:osd.0: {} 2024-09-18T08:03:59.847 INFO:teuthology.orchestra.run.smithi007.stdout:osd.1: {} 2024-09-18T08:03:59.848 INFO:teuthology.orchestra.run.smithi007.stdout:osd.2: {} 2024-09-18T08:03:59.848 INFO:teuthology.orchestra.run.smithi007.stdout:osd.3: {} 2024-09-18T08:03:59.848 INFO:teuthology.orchestra.run.smithi007.stdout:osd.4: {} 2024-09-18T08:03:59.848 INFO:teuthology.orchestra.run.smithi007.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-18T08:03:59.848 INFO:teuthology.orchestra.run.smithi007.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-18T08:03:59.848 INFO:teuthology.orchestra.run.smithi007.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-18T08:03:59.848 INFO:teuthology.orchestra.run.smithi007.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-18T08:03:59.848 INFO:teuthology.orchestra.run.smithi007.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-18T08:03:59.864 INFO:teuthology.orchestra.run.smithi007.stdout:osd.5: {} 2024-09-18T08:03:59.865 INFO:teuthology.orchestra.run.smithi007.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-18T08:03:59.888 INFO:teuthology.orchestra.run.smithi007.stdout:osd.6: {} 2024-09-18T08:03:59.888 INFO:teuthology.orchestra.run.smithi007.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-18T08:03:59.906 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:03:59 smithi050 ceph-mon[32371]: osdmap e85: 8 total, 8 up, 8 in 2024-09-18T08:03:59.906 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:03:59 smithi050 ceph-mon[32371]: from='client.? 172.21.15.7:0/3639046434' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-18T08:03:59.906 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:03:59 smithi050 ceph-mon[32371]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-18T08:03:59.906 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:03:59 smithi050 ceph-mon[32371]: from='client.? 172.21.15.7:0/3639046434' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-18T08:03:59.907 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:03:59 smithi050 ceph-mon[32371]: osdmap e86: 8 total, 8 up, 8 in 2024-09-18T08:03:59.908 INFO:teuthology.orchestra.run.smithi007.stdout:osd.7: {} 2024-09-18T08:03:59.908 INFO:teuthology.orchestra.run.smithi007.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-18T08:03:59.938 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0 2024-09-18T08:03:59.960 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:03:59.961 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:03:59.961 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0' 2024-09-18T08:04:00.062 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0 2024-09-18T08:04:00.084 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:04:00.085 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:04:00.085 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0' 2024-09-18T08:04:00.186 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0 2024-09-18T08:04:00.208 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:04:00.209 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:04:00.209 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0' 2024-09-18T08:04:00.310 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2 2024-09-18T08:04:00.334 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:04:00.335 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:04:00.335 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2' 2024-09-18T08:04:00.436 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5 2024-09-18T08:04:00.463 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:04:00.464 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:04:00.464 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5' 2024-09-18T08:04:00.565 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7 2024-09-18T08:04:00.592 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:04:00.593 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:04:00.593 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7' 2024-09-18T08:04:00.694 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6 2024-09-18T08:04:00.720 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:04:00.720 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:04:00.721 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6' 2024-09-18T08:04:00.730 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:04:00 smithi007 ceph-mon[29459]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-18T08:04:00.730 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:04:00 smithi007 ceph-mon[29459]: pgmap v37: 148 pgs: 148 active+clean; 583 KiB data, 224 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-18T08:04:00.730 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:04:00 smithi007 ceph-mon[29459]: from='client.? 172.21.15.7:0/2627160778' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-18T08:04:00.730 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:04:00 smithi007 ceph-mon[39249]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-18T08:04:00.731 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:04:00 smithi007 ceph-mon[39249]: pgmap v37: 148 pgs: 148 active+clean; 583 KiB data, 224 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-18T08:04:00.731 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:04:00 smithi007 ceph-mon[39249]: from='client.? 172.21.15.7:0/2627160778' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-18T08:04:00.821 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7 2024-09-18T08:04:00.848 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:04:00.849 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:04:00.849 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7' 2024-09-18T08:04:00.906 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:04:00 smithi050 ceph-mon[32371]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-18T08:04:00.906 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:04:00 smithi050 ceph-mon[32371]: pgmap v37: 148 pgs: 148 active+clean; 583 KiB data, 224 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-18T08:04:00.906 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:04:00 smithi050 ceph-mon[32371]: from='client.? 172.21.15.7:0/2627160778' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-18T08:04:00.950 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6 2024-09-18T08:04:00.978 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:04:00.978 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:04:00.979 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6' 2024-09-18T08:04:01.080 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4 2024-09-18T08:04:01.108 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:04:01.109 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:04:01.109 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4' 2024-09-18T08:04:01.177 INFO:journalctl@ceph.node-exporter.a.smithi007.stdout:Sep 18 08:04:00 smithi007 bash[72200]: Copying config sha256:72c9c208898624938c9e4183d6686ea4a5fd3f912bc29bc3f00147924c521a3e 2024-09-18T08:04:01.177 INFO:journalctl@ceph.node-exporter.a.smithi007.stdout:Sep 18 08:04:01 smithi007 bash[72200]: Writing manifest to image destination 2024-09-18T08:04:01.210 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6 2024-09-18T08:04:01.236 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:04:01.237 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:04:01.237 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6' 2024-09-18T08:04:01.338 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6 2024-09-18T08:04:01.366 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:04:01.367 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:04:01.367 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6' 2024-09-18T08:04:01.468 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0 2024-09-18T08:04:01.496 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:04:01.496 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:04:01.496 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0' 2024-09-18T08:04:01.597 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6 2024-09-18T08:04:01.625 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:04:01.625 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:04:01.626 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6' 2024-09-18T08:04:01.724 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/779349ca-7593-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T08:04:01.726 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2 2024-09-18T08:04:01.730 INFO:journalctl@ceph.node-exporter.a.smithi007.stdout:Sep 18 08:04:01 smithi007 podman[72200]: 2024-09-18 08:04:01.659057848 +0000 UTC m=+3.721169873 image pull 72c9c208898624938c9e4183d6686ea4a5fd3f912bc29bc3f00147924c521a3e quay.io/prometheus/node-exporter:v1.7.0 2024-09-18T08:04:01.731 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:04:01 smithi007 ceph-mon[29459]: pgmap v38: 148 pgs: 148 active+clean; 583 KiB data, 224 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-09-18T08:04:01.731 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:04:01 smithi007 ceph-mon[39249]: pgmap v38: 148 pgs: 148 active+clean; 583 KiB data, 224 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-09-18T08:04:01.756 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:04:01.756 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:04:01.757 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2' 2024-09-18T08:04:01.759 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid 779349ca-7593-11ef-bceb-c7b262605968 -- ceph daemon osd.1 dump_ops_in_flight 2024-09-18T08:04:01.858 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7 2024-09-18T08:04:01.886 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:04:01.887 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:04:01.887 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7' 2024-09-18T08:04:01.906 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:04:01 smithi050 ceph-mon[32371]: pgmap v38: 148 pgs: 148 active+clean; 583 KiB data, 224 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-09-18T08:04:01.988 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4 2024-09-18T08:04:02.015 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:04:02.016 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:04:02.016 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4' 2024-09-18T08:04:02.078 INFO:journalctl@ceph.node-exporter.a.smithi007.stdout:Sep 18 08:04:01 smithi007 podman[72200]: 2024-09-18 08:04:01.857810869 +0000 UTC m=+3.919922897 container create de1be61914dbee9177222191bc24b1c6739c47c6a53ad7b0dd1b57b00fad33af (image=quay.io/prometheus/node-exporter:v1.7.0, name=ceph-779349ca-7593-11ef-bceb-c7b262605968-node-exporter-a, maintainer=The Prometheus Authors ) 2024-09-18T08:04:02.118 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2 2024-09-18T08:04:02.142 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:04:02.142 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:04:02.142 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2' 2024-09-18T08:04:02.243 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5 2024-09-18T08:04:02.270 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:04:02.271 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:04:02.271 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5' 2024-09-18T08:04:02.365 INFO:journalctl@ceph.node-exporter.a.smithi007.stdout:Sep 18 08:04:02 smithi007 podman[72200]: 2024-09-18 08:04:02.279158338 +0000 UTC m=+4.341270363 container init de1be61914dbee9177222191bc24b1c6739c47c6a53ad7b0dd1b57b00fad33af (image=quay.io/prometheus/node-exporter:v1.7.0, name=ceph-779349ca-7593-11ef-bceb-c7b262605968-node-exporter-a, maintainer=The Prometheus Authors ) 2024-09-18T08:04:02.365 INFO:journalctl@ceph.node-exporter.a.smithi007.stdout:Sep 18 08:04:02 smithi007 podman[72200]: 2024-09-18 08:04:02.282658809 +0000 UTC m=+4.344770836 container start de1be61914dbee9177222191bc24b1c6739c47c6a53ad7b0dd1b57b00fad33af (image=quay.io/prometheus/node-exporter:v1.7.0, name=ceph-779349ca-7593-11ef-bceb-c7b262605968-node-exporter-a, maintainer=The Prometheus Authors ) 2024-09-18T08:04:02.365 INFO:journalctl@ceph.node-exporter.a.smithi007.stdout:Sep 18 08:04:02 smithi007 ceph-779349ca-7593-11ef-bceb-c7b262605968-node-exporter-a[72697]: ts=2024-09-18T08:04:02.285Z caller=node_exporter.go:192 level=info msg="Starting node_exporter" version="(version=1.7.0, branch=HEAD, revision=7333465abf9efba81876303bb57e6fadb946041b)" 2024-09-18T08:04:02.366 INFO:journalctl@ceph.node-exporter.a.smithi007.stdout:Sep 18 08:04:02 smithi007 ceph-779349ca-7593-11ef-bceb-c7b262605968-node-exporter-a[72697]: ts=2024-09-18T08:04:02.286Z caller=node_exporter.go:193 level=info msg="Build context" build_context="(go=go1.21.4, platform=linux/amd64, user=root@35918982f6d8, date=20231112-23:53:35, tags=netgo osusergo static_build)" 2024-09-18T08:04:02.366 INFO:journalctl@ceph.node-exporter.a.smithi007.stdout:Sep 18 08:04:02 smithi007 ceph-779349ca-7593-11ef-bceb-c7b262605968-node-exporter-a[72697]: ts=2024-09-18T08:04:02.286Z caller=diskstats_common.go:111 level=info collector=diskstats msg="Parsed flag --collector.diskstats.device-exclude" flag=^(ram|loop|fd|(h|s|v|xv)d[a-z]|nvme\d+n\d+p)\d+$ 2024-09-18T08:04:02.366 INFO:journalctl@ceph.node-exporter.a.smithi007.stdout:Sep 18 08:04:02 smithi007 ceph-779349ca-7593-11ef-bceb-c7b262605968-node-exporter-a[72697]: ts=2024-09-18T08:04:02.286Z caller=diskstats_linux.go:265 level=error collector=diskstats msg="Failed to open directory, disabling udev device properties" path=/run/udev/data 2024-09-18T08:04:02.366 INFO:journalctl@ceph.node-exporter.a.smithi007.stdout:Sep 18 08:04:02 smithi007 ceph-779349ca-7593-11ef-bceb-c7b262605968-node-exporter-a[72697]: ts=2024-09-18T08:04:02.286Z caller=filesystem_common.go:111 level=info collector=filesystem msg="Parsed flag --collector.filesystem.mount-points-exclude" flag=^/(dev|proc|run/credentials/.+|sys|var/lib/docker/.+|var/lib/containers/storage/.+)($|/) 2024-09-18T08:04:02.366 INFO:journalctl@ceph.node-exporter.a.smithi007.stdout:Sep 18 08:04:02 smithi007 ceph-779349ca-7593-11ef-bceb-c7b262605968-node-exporter-a[72697]: ts=2024-09-18T08:04:02.286Z caller=filesystem_common.go:113 level=info collector=filesystem msg="Parsed flag --collector.filesystem.fs-types-exclude" flag=^(autofs|binfmt_misc|bpf|cgroup2?|configfs|debugfs|devpts|devtmpfs|fusectl|hugetlbfs|iso9660|mqueue|nsfs|overlay|proc|procfs|pstore|rpc_pipefs|securityfs|selinuxfs|squashfs|sysfs|tracefs)$ 2024-09-18T08:04:02.366 INFO:journalctl@ceph.node-exporter.a.smithi007.stdout:Sep 18 08:04:02 smithi007 ceph-779349ca-7593-11ef-bceb-c7b262605968-node-exporter-a[72697]: ts=2024-09-18T08:04:02.287Z caller=node_exporter.go:110 level=info msg="Enabled collectors" 2024-09-18T08:04:02.366 INFO:journalctl@ceph.node-exporter.a.smithi007.stdout:Sep 18 08:04:02 smithi007 ceph-779349ca-7593-11ef-bceb-c7b262605968-node-exporter-a[72697]: ts=2024-09-18T08:04:02.287Z caller=node_exporter.go:117 level=info collector=arp 2024-09-18T08:04:02.366 INFO:journalctl@ceph.node-exporter.a.smithi007.stdout:Sep 18 08:04:02 smithi007 ceph-779349ca-7593-11ef-bceb-c7b262605968-node-exporter-a[72697]: ts=2024-09-18T08:04:02.287Z caller=node_exporter.go:117 level=info collector=bcache 2024-09-18T08:04:02.366 INFO:journalctl@ceph.node-exporter.a.smithi007.stdout:Sep 18 08:04:02 smithi007 ceph-779349ca-7593-11ef-bceb-c7b262605968-node-exporter-a[72697]: ts=2024-09-18T08:04:02.287Z caller=node_exporter.go:117 level=info collector=bonding 2024-09-18T08:04:02.366 INFO:journalctl@ceph.node-exporter.a.smithi007.stdout:Sep 18 08:04:02 smithi007 ceph-779349ca-7593-11ef-bceb-c7b262605968-node-exporter-a[72697]: ts=2024-09-18T08:04:02.287Z caller=node_exporter.go:117 level=info collector=btrfs 2024-09-18T08:04:02.366 INFO:journalctl@ceph.node-exporter.a.smithi007.stdout:Sep 18 08:04:02 smithi007 ceph-779349ca-7593-11ef-bceb-c7b262605968-node-exporter-a[72697]: ts=2024-09-18T08:04:02.287Z caller=node_exporter.go:117 level=info collector=conntrack 2024-09-18T08:04:02.366 INFO:journalctl@ceph.node-exporter.a.smithi007.stdout:Sep 18 08:04:02 smithi007 ceph-779349ca-7593-11ef-bceb-c7b262605968-node-exporter-a[72697]: ts=2024-09-18T08:04:02.287Z caller=node_exporter.go:117 level=info collector=cpu 2024-09-18T08:04:02.366 INFO:journalctl@ceph.node-exporter.a.smithi007.stdout:Sep 18 08:04:02 smithi007 ceph-779349ca-7593-11ef-bceb-c7b262605968-node-exporter-a[72697]: ts=2024-09-18T08:04:02.287Z caller=node_exporter.go:117 level=info collector=cpufreq 2024-09-18T08:04:02.367 INFO:journalctl@ceph.node-exporter.a.smithi007.stdout:Sep 18 08:04:02 smithi007 ceph-779349ca-7593-11ef-bceb-c7b262605968-node-exporter-a[72697]: ts=2024-09-18T08:04:02.287Z caller=node_exporter.go:117 level=info collector=diskstats 2024-09-18T08:04:02.367 INFO:journalctl@ceph.node-exporter.a.smithi007.stdout:Sep 18 08:04:02 smithi007 ceph-779349ca-7593-11ef-bceb-c7b262605968-node-exporter-a[72697]: ts=2024-09-18T08:04:02.287Z caller=node_exporter.go:117 level=info collector=dmi 2024-09-18T08:04:02.367 INFO:journalctl@ceph.node-exporter.a.smithi007.stdout:Sep 18 08:04:02 smithi007 ceph-779349ca-7593-11ef-bceb-c7b262605968-node-exporter-a[72697]: ts=2024-09-18T08:04:02.287Z caller=node_exporter.go:117 level=info collector=edac 2024-09-18T08:04:02.367 INFO:journalctl@ceph.node-exporter.a.smithi007.stdout:Sep 18 08:04:02 smithi007 ceph-779349ca-7593-11ef-bceb-c7b262605968-node-exporter-a[72697]: ts=2024-09-18T08:04:02.287Z caller=node_exporter.go:117 level=info collector=entropy 2024-09-18T08:04:02.367 INFO:journalctl@ceph.node-exporter.a.smithi007.stdout:Sep 18 08:04:02 smithi007 ceph-779349ca-7593-11ef-bceb-c7b262605968-node-exporter-a[72697]: ts=2024-09-18T08:04:02.287Z caller=node_exporter.go:117 level=info collector=fibrechannel 2024-09-18T08:04:02.367 INFO:journalctl@ceph.node-exporter.a.smithi007.stdout:Sep 18 08:04:02 smithi007 ceph-779349ca-7593-11ef-bceb-c7b262605968-node-exporter-a[72697]: ts=2024-09-18T08:04:02.287Z caller=node_exporter.go:117 level=info collector=filefd 2024-09-18T08:04:02.367 INFO:journalctl@ceph.node-exporter.a.smithi007.stdout:Sep 18 08:04:02 smithi007 ceph-779349ca-7593-11ef-bceb-c7b262605968-node-exporter-a[72697]: ts=2024-09-18T08:04:02.287Z caller=node_exporter.go:117 level=info collector=filesystem 2024-09-18T08:04:02.368 INFO:journalctl@ceph.node-exporter.a.smithi007.stdout:Sep 18 08:04:02 smithi007 ceph-779349ca-7593-11ef-bceb-c7b262605968-node-exporter-a[72697]: ts=2024-09-18T08:04:02.287Z caller=node_exporter.go:117 level=info collector=hwmon 2024-09-18T08:04:02.368 INFO:journalctl@ceph.node-exporter.a.smithi007.stdout:Sep 18 08:04:02 smithi007 ceph-779349ca-7593-11ef-bceb-c7b262605968-node-exporter-a[72697]: ts=2024-09-18T08:04:02.287Z caller=node_exporter.go:117 level=info collector=infiniband 2024-09-18T08:04:02.368 INFO:journalctl@ceph.node-exporter.a.smithi007.stdout:Sep 18 08:04:02 smithi007 ceph-779349ca-7593-11ef-bceb-c7b262605968-node-exporter-a[72697]: ts=2024-09-18T08:04:02.287Z caller=node_exporter.go:117 level=info collector=ipvs 2024-09-18T08:04:02.368 INFO:journalctl@ceph.node-exporter.a.smithi007.stdout:Sep 18 08:04:02 smithi007 ceph-779349ca-7593-11ef-bceb-c7b262605968-node-exporter-a[72697]: ts=2024-09-18T08:04:02.287Z caller=node_exporter.go:117 level=info collector=loadavg 2024-09-18T08:04:02.368 INFO:journalctl@ceph.node-exporter.a.smithi007.stdout:Sep 18 08:04:02 smithi007 ceph-779349ca-7593-11ef-bceb-c7b262605968-node-exporter-a[72697]: ts=2024-09-18T08:04:02.287Z caller=node_exporter.go:117 level=info collector=mdadm 2024-09-18T08:04:02.368 INFO:journalctl@ceph.node-exporter.a.smithi007.stdout:Sep 18 08:04:02 smithi007 ceph-779349ca-7593-11ef-bceb-c7b262605968-node-exporter-a[72697]: ts=2024-09-18T08:04:02.287Z caller=node_exporter.go:117 level=info collector=meminfo 2024-09-18T08:04:02.368 INFO:journalctl@ceph.node-exporter.a.smithi007.stdout:Sep 18 08:04:02 smithi007 ceph-779349ca-7593-11ef-bceb-c7b262605968-node-exporter-a[72697]: ts=2024-09-18T08:04:02.287Z caller=node_exporter.go:117 level=info collector=netclass 2024-09-18T08:04:02.368 INFO:journalctl@ceph.node-exporter.a.smithi007.stdout:Sep 18 08:04:02 smithi007 ceph-779349ca-7593-11ef-bceb-c7b262605968-node-exporter-a[72697]: ts=2024-09-18T08:04:02.287Z caller=node_exporter.go:117 level=info collector=netdev 2024-09-18T08:04:02.368 INFO:journalctl@ceph.node-exporter.a.smithi007.stdout:Sep 18 08:04:02 smithi007 ceph-779349ca-7593-11ef-bceb-c7b262605968-node-exporter-a[72697]: ts=2024-09-18T08:04:02.287Z caller=node_exporter.go:117 level=info collector=netstat 2024-09-18T08:04:02.368 INFO:journalctl@ceph.node-exporter.a.smithi007.stdout:Sep 18 08:04:02 smithi007 ceph-779349ca-7593-11ef-bceb-c7b262605968-node-exporter-a[72697]: ts=2024-09-18T08:04:02.287Z caller=node_exporter.go:117 level=info collector=nfs 2024-09-18T08:04:02.368 INFO:journalctl@ceph.node-exporter.a.smithi007.stdout:Sep 18 08:04:02 smithi007 ceph-779349ca-7593-11ef-bceb-c7b262605968-node-exporter-a[72697]: ts=2024-09-18T08:04:02.287Z caller=node_exporter.go:117 level=info collector=nfsd 2024-09-18T08:04:02.369 INFO:journalctl@ceph.node-exporter.a.smithi007.stdout:Sep 18 08:04:02 smithi007 ceph-779349ca-7593-11ef-bceb-c7b262605968-node-exporter-a[72697]: ts=2024-09-18T08:04:02.287Z caller=node_exporter.go:117 level=info collector=nvme 2024-09-18T08:04:02.369 INFO:journalctl@ceph.node-exporter.a.smithi007.stdout:Sep 18 08:04:02 smithi007 ceph-779349ca-7593-11ef-bceb-c7b262605968-node-exporter-a[72697]: ts=2024-09-18T08:04:02.287Z caller=node_exporter.go:117 level=info collector=os 2024-09-18T08:04:02.369 INFO:journalctl@ceph.node-exporter.a.smithi007.stdout:Sep 18 08:04:02 smithi007 ceph-779349ca-7593-11ef-bceb-c7b262605968-node-exporter-a[72697]: ts=2024-09-18T08:04:02.287Z caller=node_exporter.go:117 level=info collector=powersupplyclass 2024-09-18T08:04:02.369 INFO:journalctl@ceph.node-exporter.a.smithi007.stdout:Sep 18 08:04:02 smithi007 ceph-779349ca-7593-11ef-bceb-c7b262605968-node-exporter-a[72697]: ts=2024-09-18T08:04:02.287Z caller=node_exporter.go:117 level=info collector=pressure 2024-09-18T08:04:02.369 INFO:journalctl@ceph.node-exporter.a.smithi007.stdout:Sep 18 08:04:02 smithi007 ceph-779349ca-7593-11ef-bceb-c7b262605968-node-exporter-a[72697]: ts=2024-09-18T08:04:02.287Z caller=node_exporter.go:117 level=info collector=rapl 2024-09-18T08:04:02.369 INFO:journalctl@ceph.node-exporter.a.smithi007.stdout:Sep 18 08:04:02 smithi007 ceph-779349ca-7593-11ef-bceb-c7b262605968-node-exporter-a[72697]: ts=2024-09-18T08:04:02.287Z caller=node_exporter.go:117 level=info collector=schedstat 2024-09-18T08:04:02.369 INFO:journalctl@ceph.node-exporter.a.smithi007.stdout:Sep 18 08:04:02 smithi007 ceph-779349ca-7593-11ef-bceb-c7b262605968-node-exporter-a[72697]: ts=2024-09-18T08:04:02.287Z caller=node_exporter.go:117 level=info collector=selinux 2024-09-18T08:04:02.369 INFO:journalctl@ceph.node-exporter.a.smithi007.stdout:Sep 18 08:04:02 smithi007 ceph-779349ca-7593-11ef-bceb-c7b262605968-node-exporter-a[72697]: ts=2024-09-18T08:04:02.287Z caller=node_exporter.go:117 level=info collector=sockstat 2024-09-18T08:04:02.369 INFO:journalctl@ceph.node-exporter.a.smithi007.stdout:Sep 18 08:04:02 smithi007 ceph-779349ca-7593-11ef-bceb-c7b262605968-node-exporter-a[72697]: ts=2024-09-18T08:04:02.287Z caller=node_exporter.go:117 level=info collector=softnet 2024-09-18T08:04:02.369 INFO:journalctl@ceph.node-exporter.a.smithi007.stdout:Sep 18 08:04:02 smithi007 ceph-779349ca-7593-11ef-bceb-c7b262605968-node-exporter-a[72697]: ts=2024-09-18T08:04:02.287Z caller=node_exporter.go:117 level=info collector=stat 2024-09-18T08:04:02.369 INFO:journalctl@ceph.node-exporter.a.smithi007.stdout:Sep 18 08:04:02 smithi007 ceph-779349ca-7593-11ef-bceb-c7b262605968-node-exporter-a[72697]: ts=2024-09-18T08:04:02.287Z caller=node_exporter.go:117 level=info collector=tapestats 2024-09-18T08:04:02.369 INFO:journalctl@ceph.node-exporter.a.smithi007.stdout:Sep 18 08:04:02 smithi007 ceph-779349ca-7593-11ef-bceb-c7b262605968-node-exporter-a[72697]: ts=2024-09-18T08:04:02.287Z caller=node_exporter.go:117 level=info collector=textfile 2024-09-18T08:04:02.370 INFO:journalctl@ceph.node-exporter.a.smithi007.stdout:Sep 18 08:04:02 smithi007 ceph-779349ca-7593-11ef-bceb-c7b262605968-node-exporter-a[72697]: ts=2024-09-18T08:04:02.287Z caller=node_exporter.go:117 level=info collector=thermal_zone 2024-09-18T08:04:02.370 INFO:journalctl@ceph.node-exporter.a.smithi007.stdout:Sep 18 08:04:02 smithi007 ceph-779349ca-7593-11ef-bceb-c7b262605968-node-exporter-a[72697]: ts=2024-09-18T08:04:02.287Z caller=node_exporter.go:117 level=info collector=time 2024-09-18T08:04:02.370 INFO:journalctl@ceph.node-exporter.a.smithi007.stdout:Sep 18 08:04:02 smithi007 ceph-779349ca-7593-11ef-bceb-c7b262605968-node-exporter-a[72697]: ts=2024-09-18T08:04:02.287Z caller=node_exporter.go:117 level=info collector=udp_queues 2024-09-18T08:04:02.370 INFO:journalctl@ceph.node-exporter.a.smithi007.stdout:Sep 18 08:04:02 smithi007 ceph-779349ca-7593-11ef-bceb-c7b262605968-node-exporter-a[72697]: ts=2024-09-18T08:04:02.287Z caller=node_exporter.go:117 level=info collector=uname 2024-09-18T08:04:02.370 INFO:journalctl@ceph.node-exporter.a.smithi007.stdout:Sep 18 08:04:02 smithi007 ceph-779349ca-7593-11ef-bceb-c7b262605968-node-exporter-a[72697]: ts=2024-09-18T08:04:02.287Z caller=node_exporter.go:117 level=info collector=vmstat 2024-09-18T08:04:02.370 INFO:journalctl@ceph.node-exporter.a.smithi007.stdout:Sep 18 08:04:02 smithi007 ceph-779349ca-7593-11ef-bceb-c7b262605968-node-exporter-a[72697]: ts=2024-09-18T08:04:02.287Z caller=node_exporter.go:117 level=info collector=xfs 2024-09-18T08:04:02.370 INFO:journalctl@ceph.node-exporter.a.smithi007.stdout:Sep 18 08:04:02 smithi007 ceph-779349ca-7593-11ef-bceb-c7b262605968-node-exporter-a[72697]: ts=2024-09-18T08:04:02.287Z caller=node_exporter.go:117 level=info collector=zfs 2024-09-18T08:04:02.370 INFO:journalctl@ceph.node-exporter.a.smithi007.stdout:Sep 18 08:04:02 smithi007 ceph-779349ca-7593-11ef-bceb-c7b262605968-node-exporter-a[72697]: ts=2024-09-18T08:04:02.287Z caller=tls_config.go:274 level=info msg="Listening on" address=[::]:9100 2024-09-18T08:04:02.370 INFO:journalctl@ceph.node-exporter.a.smithi007.stdout:Sep 18 08:04:02 smithi007 ceph-779349ca-7593-11ef-bceb-c7b262605968-node-exporter-a[72697]: ts=2024-09-18T08:04:02.287Z caller=tls_config.go:277 level=info msg="TLS is disabled." http2=false address=[::]:9100 2024-09-18T08:04:02.372 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7 2024-09-18T08:04:02.398 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:04:02.399 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:04:02.399 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7' 2024-09-18T08:04:02.500 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7 2024-09-18T08:04:02.527 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:04:02.528 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:04:02.528 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7' 2024-09-18T08:04:02.629 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6 2024-09-18T08:04:02.656 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:04:02.657 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:04:02.657 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6' 2024-09-18T08:04:02.730 INFO:journalctl@ceph.node-exporter.a.smithi007.stdout:Sep 18 08:04:02 smithi007 bash[72200]: de1be61914dbee9177222191bc24b1c6739c47c6a53ad7b0dd1b57b00fad33af 2024-09-18T08:04:02.731 INFO:journalctl@ceph.node-exporter.a.smithi007.stdout:Sep 18 08:04:02 smithi007 systemd[1]: Started Ceph node-exporter.a for 779349ca-7593-11ef-bceb-c7b262605968. 2024-09-18T08:04:02.758 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4 2024-09-18T08:04:02.781 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:04:02.782 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:04:02.782 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4' 2024-09-18T08:04:02.884 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1 2024-09-18T08:04:02.906 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:04:02.907 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:04:02.907 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1' 2024-09-18T08:04:03.008 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6 2024-09-18T08:04:03.031 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:04:03.031 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:04:03.032 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6' 2024-09-18T08:04:03.132 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6 2024-09-18T08:04:03.155 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:04:03.156 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:04:03.156 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6' 2024-09-18T08:04:03.257 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0 2024-09-18T08:04:03.280 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:04:03.280 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:04:03.281 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0' 2024-09-18T08:04:03.382 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3 2024-09-18T08:04:03.405 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:04:03.406 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:04:03.406 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3' 2024-09-18T08:04:03.443 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/779349ca-7593-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T08:04:03.507 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4 2024-09-18T08:04:03.534 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:04:03.534 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:04:03.535 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4' 2024-09-18T08:04:03.637 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1 2024-09-18T08:04:03.642 INFO:teuthology.orchestra.run.smithi007.stderr:enabled application 'rados' on pool 'unique_pool_0' 2024-09-18T08:04:03.656 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:04:03 smithi007 ceph-mon[29459]: purged_snaps scrub starts 2024-09-18T08:04:03.656 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:04:03 smithi007 ceph-mon[29459]: purged_snaps scrub ok 2024-09-18T08:04:03.657 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:04:03 smithi007 ceph-mon[29459]: osdmap e87: 8 total, 8 up, 8 in 2024-09-18T08:04:03.657 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:04:03 smithi007 ceph-mon[29459]: from='mgr.24470 172.21.15.7:0/2594390261' entity='mgr.y' 2024-09-18T08:04:03.657 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:04:03 smithi007 ceph-mon[29459]: from='mgr.24470 172.21.15.7:0/2594390261' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-18T08:04:03.657 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:04:03 smithi007 ceph-mon[29459]: from='mgr.24470 172.21.15.7:0/2594390261' entity='mgr.y' 2024-09-18T08:04:03.657 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:04:03 smithi007 ceph-mon[29459]: from='mgr.24470 172.21.15.7:0/2594390261' entity='mgr.y' 2024-09-18T08:04:03.657 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:04:03 smithi007 ceph-mon[29459]: from='mgr.24470 172.21.15.7:0/2594390261' entity='mgr.y' 2024-09-18T08:04:03.657 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:04:03 smithi007 ceph-mon[29459]: Deploying daemon node-exporter.b on smithi050 2024-09-18T08:04:03.657 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:04:03 smithi007 ceph-mon[29459]: from='client.? 172.21.15.7:0/1443128529' entity='client.admin' cmd={"prefix": "osd pool application enable", "pool": "unique_pool_0", "app": "rados", "yes_i_really_mean_it": true} : dispatch 2024-09-18T08:04:03.657 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:04:03 smithi007 ceph-mon[29459]: pgmap v40: 148 pgs: 148 active+clean; 583 KiB data, 224 MiB used, 715 GiB / 715 GiB avail; 910 B/s rd, 0 op/s 2024-09-18T08:04:03.657 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:04:03 smithi007 ceph-mon[39249]: purged_snaps scrub starts 2024-09-18T08:04:03.657 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:04:03 smithi007 ceph-mon[39249]: purged_snaps scrub ok 2024-09-18T08:04:03.657 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:04:03 smithi007 ceph-mon[39249]: osdmap e87: 8 total, 8 up, 8 in 2024-09-18T08:04:03.658 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:04:03 smithi007 ceph-mon[39249]: from='mgr.24470 172.21.15.7:0/2594390261' entity='mgr.y' 2024-09-18T08:04:03.658 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:04:03 smithi007 ceph-mon[39249]: from='mgr.24470 172.21.15.7:0/2594390261' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-18T08:04:03.658 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:04:03 smithi007 ceph-mon[39249]: from='mgr.24470 172.21.15.7:0/2594390261' entity='mgr.y' 2024-09-18T08:04:03.658 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:04:03 smithi007 ceph-mon[39249]: from='mgr.24470 172.21.15.7:0/2594390261' entity='mgr.y' 2024-09-18T08:04:03.658 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:04:03 smithi007 ceph-mon[39249]: from='mgr.24470 172.21.15.7:0/2594390261' entity='mgr.y' 2024-09-18T08:04:03.658 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:04:03 smithi007 ceph-mon[39249]: Deploying daemon node-exporter.b on smithi050 2024-09-18T08:04:03.658 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:04:03 smithi007 ceph-mon[39249]: from='client.? 172.21.15.7:0/1443128529' entity='client.admin' cmd={"prefix": "osd pool application enable", "pool": "unique_pool_0", "app": "rados", "yes_i_really_mean_it": true} : dispatch 2024-09-18T08:04:03.658 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:04:03 smithi007 ceph-mon[39249]: pgmap v40: 148 pgs: 148 active+clean; 583 KiB data, 224 MiB used, 715 GiB / 715 GiB avail; 910 B/s rd, 0 op/s 2024-09-18T08:04:03.663 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:04:03.663 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:04:03.663 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1' 2024-09-18T08:04:03.764 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6 2024-09-18T08:04:03.791 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:04:03.792 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:04:03.793 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6' 2024-09-18T08:04:03.894 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2 2024-09-18T08:04:03.906 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:04:03 smithi050 ceph-mon[32371]: purged_snaps scrub starts 2024-09-18T08:04:03.906 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:04:03 smithi050 ceph-mon[32371]: purged_snaps scrub ok 2024-09-18T08:04:03.907 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:04:03 smithi050 ceph-mon[32371]: osdmap e87: 8 total, 8 up, 8 in 2024-09-18T08:04:03.907 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:04:03 smithi050 ceph-mon[32371]: from='mgr.24470 172.21.15.7:0/2594390261' entity='mgr.y' 2024-09-18T08:04:03.907 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:04:03 smithi050 ceph-mon[32371]: from='mgr.24470 172.21.15.7:0/2594390261' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-18T08:04:03.907 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:04:03 smithi050 ceph-mon[32371]: from='mgr.24470 172.21.15.7:0/2594390261' entity='mgr.y' 2024-09-18T08:04:03.907 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:04:03 smithi050 ceph-mon[32371]: from='mgr.24470 172.21.15.7:0/2594390261' entity='mgr.y' 2024-09-18T08:04:03.907 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:04:03 smithi050 ceph-mon[32371]: from='mgr.24470 172.21.15.7:0/2594390261' entity='mgr.y' 2024-09-18T08:04:03.907 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:04:03 smithi050 ceph-mon[32371]: Deploying daemon node-exporter.b on smithi050 2024-09-18T08:04:03.907 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:04:03 smithi050 ceph-mon[32371]: from='client.? 172.21.15.7:0/1443128529' entity='client.admin' cmd={"prefix": "osd pool application enable", "pool": "unique_pool_0", "app": "rados", "yes_i_really_mean_it": true} : dispatch 2024-09-18T08:04:03.907 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:04:03 smithi050 ceph-mon[32371]: pgmap v40: 148 pgs: 148 active+clean; 583 KiB data, 224 MiB used, 715 GiB / 715 GiB avail; 910 B/s rd, 0 op/s 2024-09-18T08:04:03.919 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:04:03.920 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:04:03.920 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2' 2024-09-18T08:04:04.021 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7 2024-09-18T08:04:04.049 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:04:04.050 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:04:04.050 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7' 2024-09-18T08:04:04.151 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4 2024-09-18T08:04:04.174 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:04:04.175 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:04:04.175 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4' 2024-09-18T08:04:04.276 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0 2024-09-18T08:04:04.299 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:04:04.299 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:04:04.300 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0' 2024-09-18T08:04:04.401 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5 2024-09-18T08:04:04.424 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:04:04.424 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:04:04.425 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5' 2024-09-18T08:04:04.525 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0 2024-09-18T08:04:04.548 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:04:04.549 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:04:04.549 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0' 2024-09-18T08:04:04.650 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1 2024-09-18T08:04:04.657 INFO:teuthology.orchestra.run.smithi007.stdout:{ 2024-09-18T08:04:04.657 INFO:teuthology.orchestra.run.smithi007.stdout: "ops": [], 2024-09-18T08:04:04.657 INFO:teuthology.orchestra.run.smithi007.stdout: "num_ops": 0 2024-09-18T08:04:04.657 INFO:teuthology.orchestra.run.smithi007.stdout:} 2024-09-18T08:04:04.673 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:04:04.673 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:04:04.674 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1' 2024-09-18T08:04:04.730 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:04:04 smithi007 ceph-mon[29459]: from='client.? 172.21.15.7:0/1443128529' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "unique_pool_0", "app": "rados", "yes_i_really_mean_it": true}]': finished 2024-09-18T08:04:04.730 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:04:04 smithi007 ceph-mon[29459]: osdmap e88: 8 total, 8 up, 8 in 2024-09-18T08:04:04.731 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:04:04 smithi007 ceph-mon[39249]: from='client.? 172.21.15.7:0/1443128529' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "unique_pool_0", "app": "rados", "yes_i_really_mean_it": true}]': finished 2024-09-18T08:04:04.731 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:04:04 smithi007 ceph-mon[39249]: osdmap e88: 8 total, 8 up, 8 in 2024-09-18T08:04:04.774 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0 2024-09-18T08:04:04.796 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:04:04.797 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:04:04.797 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0' 2024-09-18T08:04:04.898 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3 2024-09-18T08:04:04.906 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:04:04 smithi050 ceph-mon[32371]: from='client.? 172.21.15.7:0/1443128529' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "unique_pool_0", "app": "rados", "yes_i_really_mean_it": true}]': finished 2024-09-18T08:04:04.906 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:04:04 smithi050 ceph-mon[32371]: osdmap e88: 8 total, 8 up, 8 in 2024-09-18T08:04:04.920 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:04:04.921 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:04:04.922 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3' 2024-09-18T08:04:05.022 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3 2024-09-18T08:04:05.045 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:04:05.046 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:04:05.046 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3' 2024-09-18T08:04:05.069 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid 779349ca-7593-11ef-bceb-c7b262605968 -- ceph osd unset noscrub 2024-09-18T08:04:05.147 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1 2024-09-18T08:04:05.170 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:04:05.171 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:04:05.171 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1' 2024-09-18T08:04:05.272 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4 2024-09-18T08:04:05.297 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:04:05.298 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:04:05.298 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4' 2024-09-18T08:04:05.309 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid 779349ca-7593-11ef-bceb-c7b262605968 -- ceph daemon osd.1 dump_blocked_ops 2024-09-18T08:04:05.350 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/779349ca-7593-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T08:04:05.399 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4 2024-09-18T08:04:05.405 INFO:journalctl@ceph.node-exporter.b.smithi050.stdout:Sep 18 08:04:04 smithi050 bash[61451]: Copying blob sha256:2abcce694348cd2c949c0e98a7400ebdfd8341021bcf6b541bc72033ce982510 2024-09-18T08:04:05.422 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:04:05.423 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:04:05.423 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4' 2024-09-18T08:04:05.439 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid 779349ca-7593-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-18T08:04:05.457 DEBUG:teuthology.orchestra.run.smithi007:> /bin/sh -c 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados --no-log-to-stderr --name client.0 -b 65536 --object-size 65536 -p unique_pool_0 bench 90 write' 2024-09-18T08:04:05.458 INFO:tasks.thrashosds.thrasher:fixing pg num pool .mgr 2024-09-18T08:04:05.458 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid 779349ca-7593-11ef-bceb-c7b262605968 -- ceph pg dump --format=json 2024-09-18T08:04:05.484 INFO:tasks.radosbench:joining radosbench (timing out after 3000s) 2024-09-18T08:04:05.484 INFO:teuthology.orchestra.run:waiting for 3000 2024-09-18T08:04:05.524 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7 2024-09-18T08:04:05.545 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:04:05.546 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:04:05.547 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7' 2024-09-18T08:04:05.566 INFO:tasks.radosbench.radosbench.0.smithi007.stdout:hints = 1 2024-09-18T08:04:05.566 INFO:tasks.radosbench.radosbench.0.smithi007.stdout:Maintaining 16 concurrent writes of 65536 bytes to objects of size 65536 for up to 90 seconds or 0 objects 2024-09-18T08:04:05.567 INFO:tasks.radosbench.radosbench.0.smithi007.stdout:Object prefix: benchmark_data_smithi007_73155 2024-09-18T08:04:05.567 INFO:tasks.radosbench.radosbench.0.smithi007.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-09-18T08:04:05.567 INFO:tasks.radosbench.radosbench.0.smithi007.stdout: 0 0 0 0 0 0 - 0 2024-09-18T08:04:05.648 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1 2024-09-18T08:04:05.716 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:04:05.717 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:04:05.717 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1' 2024-09-18T08:04:05.818 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6 2024-09-18T08:04:05.852 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:04:05.853 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:04:05.853 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6' 2024-09-18T08:04:05.938 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:04:05 smithi007 ceph-mon[39249]: pgmap v42: 148 pgs: 148 active+clean; 583 KiB data, 224 MiB used, 715 GiB / 715 GiB avail; 850 B/s rd, 0 op/s 2024-09-18T08:04:05.938 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:04:05 smithi007 ceph-mon[29459]: pgmap v42: 148 pgs: 148 active+clean; 583 KiB data, 224 MiB used, 715 GiB / 715 GiB avail; 850 B/s rd, 0 op/s 2024-09-18T08:04:05.954 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3 2024-09-18T08:04:06.007 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:04:06.008 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:04:06.009 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3' 2024-09-18T08:04:06.024 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/779349ca-7593-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T08:04:06.043 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/779349ca-7593-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T08:04:06.056 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/779349ca-7593-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T08:04:06.110 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7 2024-09-18T08:04:06.143 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:04:06.144 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:04:06.144 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7' 2024-09-18T08:04:06.155 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:04:05 smithi050 ceph-mon[32371]: pgmap v42: 148 pgs: 148 active+clean; 583 KiB data, 224 MiB used, 715 GiB / 715 GiB avail; 850 B/s rd, 0 op/s 2024-09-18T08:04:06.245 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1 2024-09-18T08:04:06.293 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:04:06.293 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:04:06.294 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1' 2024-09-18T08:04:06.395 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5 2024-09-18T08:04:06.420 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:04:06.420 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:04:06.421 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5' 2024-09-18T08:04:06.521 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3 2024-09-18T08:04:06.564 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:04:06.565 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:04:06.565 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3' 2024-09-18T08:04:06.567 INFO:tasks.radosbench.radosbench.0.smithi007.stdout: 1 16 2598 2582 161.366 161.375 0.00431778 0.00617058 2024-09-18T08:04:06.656 INFO:journalctl@ceph.node-exporter.b.smithi050.stdout:Sep 18 08:04:06 smithi050 bash[61451]: Copying config sha256:72c9c208898624938c9e4183d6686ea4a5fd3f912bc29bc3f00147924c521a3e 2024-09-18T08:04:06.656 INFO:journalctl@ceph.node-exporter.b.smithi050.stdout:Sep 18 08:04:06 smithi050 bash[61451]: Writing manifest to image destination 2024-09-18T08:04:06.666 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2 2024-09-18T08:04:06.669 INFO:teuthology.orchestra.run.smithi007.stderr:noscrub is unset 2024-09-18T08:04:06.714 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:04:06.715 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:04:06.715 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2' 2024-09-18T08:04:06.731 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:04:06 smithi007 ceph-mon[29459]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-09-18T08:04:06.731 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:04:06 smithi007 ceph-mon[29459]: from='client.? 172.21.15.7:0/46990774' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-18T08:04:06.731 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:04:06 smithi007 ceph-mon[29459]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-18T08:04:06.731 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:04:06 smithi007 ceph-mon[29459]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-18T08:04:06.731 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Sep 18 08:04:06 smithi007 ceph-779349ca-7593-11ef-bceb-c7b262605968-mgr-y[29688]: ::ffff:172.21.15.50 - - [18/Sep/2024:08:04:06] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-09-18T08:04:06.732 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:04:06 smithi007 ceph-mon[39249]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-09-18T08:04:06.732 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:04:06 smithi007 ceph-mon[39249]: from='client.? 172.21.15.7:0/46990774' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-18T08:04:06.732 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:04:06 smithi007 ceph-mon[39249]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-18T08:04:06.732 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:04:06 smithi007 ceph-mon[39249]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-18T08:04:06.816 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4 2024-09-18T08:04:06.849 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:04:06.849 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:04:06.850 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4' 2024-09-18T08:04:06.951 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5 2024-09-18T08:04:06.965 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:04:06 smithi050 ceph-mon[32371]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-09-18T08:04:06.965 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:04:06 smithi050 ceph-mon[32371]: from='client.? 172.21.15.7:0/46990774' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-18T08:04:06.965 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:04:06 smithi050 ceph-mon[32371]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-18T08:04:06.965 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:04:06 smithi050 ceph-mon[32371]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-18T08:04:06.979 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:04:06.979 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:04:06.980 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5' 2024-09-18T08:04:07.081 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7 2024-09-18T08:04:07.111 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:04:07.112 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:04:07.112 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7' 2024-09-18T08:04:07.121 INFO:teuthology.orchestra.run.smithi007.stdout:ERROR: (22) Invalid argument 2024-09-18T08:04:07.121 INFO:teuthology.orchestra.run.smithi007.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-18T08:04:07.160 INFO:teuthology.orchestra.run.smithi007.stdout:osd.0: {} 2024-09-18T08:04:07.160 INFO:teuthology.orchestra.run.smithi007.stdout:osd.1: {} 2024-09-18T08:04:07.160 INFO:teuthology.orchestra.run.smithi007.stdout:osd.2: {} 2024-09-18T08:04:07.160 INFO:teuthology.orchestra.run.smithi007.stdout:osd.3: {} 2024-09-18T08:04:07.160 INFO:teuthology.orchestra.run.smithi007.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-18T08:04:07.160 INFO:teuthology.orchestra.run.smithi007.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-18T08:04:07.160 INFO:teuthology.orchestra.run.smithi007.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-18T08:04:07.161 INFO:teuthology.orchestra.run.smithi007.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-18T08:04:07.165 INFO:teuthology.orchestra.run.smithi007.stdout:osd.4: {} 2024-09-18T08:04:07.165 INFO:teuthology.orchestra.run.smithi007.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-18T08:04:07.214 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1 2024-09-18T08:04:07.215 INFO:teuthology.orchestra.run.smithi007.stdout:osd.5: {} 2024-09-18T08:04:07.216 INFO:teuthology.orchestra.run.smithi007.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-18T08:04:07.248 INFO:teuthology.orchestra.run.smithi007.stdout:osd.6: {} 2024-09-18T08:04:07.248 INFO:teuthology.orchestra.run.smithi007.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-18T08:04:07.257 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:04:07.258 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:04:07.258 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1' 2024-09-18T08:04:07.272 INFO:journalctl@ceph.node-exporter.b.smithi050.stdout:Sep 18 08:04:06 smithi050 podman[61451]: 2024-09-18 08:04:06.911300219 +0000 UTC m=+2.368143700 image pull 72c9c208898624938c9e4183d6686ea4a5fd3f912bc29bc3f00147924c521a3e quay.io/prometheus/node-exporter:v1.7.0 2024-09-18T08:04:07.272 INFO:journalctl@ceph.node-exporter.b.smithi050.stdout:Sep 18 08:04:07 smithi050 podman[61451]: 2024-09-18 08:04:07.0342943 +0000 UTC m=+2.491137753 container create baba185ed32a51ea670d7d4b4dc68a79a0cb4fab8e8dfaa06ce93162294ccd25 (image=quay.io/prometheus/node-exporter:v1.7.0, name=ceph-779349ca-7593-11ef-bceb-c7b262605968-node-exporter-b, maintainer=The Prometheus Authors ) 2024-09-18T08:04:07.272 INFO:journalctl@ceph.node-exporter.b.smithi050.stdout:Sep 18 08:04:07 smithi050 podman[61451]: 2024-09-18 08:04:07.270680644 +0000 UTC m=+2.727524121 container init baba185ed32a51ea670d7d4b4dc68a79a0cb4fab8e8dfaa06ce93162294ccd25 (image=quay.io/prometheus/node-exporter:v1.7.0, name=ceph-779349ca-7593-11ef-bceb-c7b262605968-node-exporter-b, maintainer=The Prometheus Authors ) 2024-09-18T08:04:07.285 INFO:teuthology.orchestra.run.smithi007.stdout:osd.7: {} 2024-09-18T08:04:07.286 INFO:teuthology.orchestra.run.smithi007.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-18T08:04:07.359 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1 2024-09-18T08:04:07.400 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:04:07.400 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:04:07.401 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1' 2024-09-18T08:04:07.501 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7 2024-09-18T08:04:07.532 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:04:07.532 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:04:07.533 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7' 2024-09-18T08:04:07.567 INFO:tasks.radosbench.radosbench.0.smithi007.stdout: 2 16 5422 5406 168.926 176.5 0.00354558 0.00590702 2024-09-18T08:04:07.633 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3 2024-09-18T08:04:07.656 INFO:journalctl@ceph.node-exporter.b.smithi050.stdout:Sep 18 08:04:07 smithi050 podman[61451]: 2024-09-18 08:04:07.275713892 +0000 UTC m=+2.732557354 container start baba185ed32a51ea670d7d4b4dc68a79a0cb4fab8e8dfaa06ce93162294ccd25 (image=quay.io/prometheus/node-exporter:v1.7.0, name=ceph-779349ca-7593-11ef-bceb-c7b262605968-node-exporter-b, maintainer=The Prometheus Authors ) 2024-09-18T08:04:07.656 INFO:journalctl@ceph.node-exporter.b.smithi050.stdout:Sep 18 08:04:07 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-node-exporter-b[61677]: ts=2024-09-18T08:04:07.280Z caller=node_exporter.go:192 level=info msg="Starting node_exporter" version="(version=1.7.0, branch=HEAD, revision=7333465abf9efba81876303bb57e6fadb946041b)" 2024-09-18T08:04:07.656 INFO:journalctl@ceph.node-exporter.b.smithi050.stdout:Sep 18 08:04:07 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-node-exporter-b[61677]: ts=2024-09-18T08:04:07.280Z caller=node_exporter.go:193 level=info msg="Build context" build_context="(go=go1.21.4, platform=linux/amd64, user=root@35918982f6d8, date=20231112-23:53:35, tags=netgo osusergo static_build)" 2024-09-18T08:04:07.657 INFO:journalctl@ceph.node-exporter.b.smithi050.stdout:Sep 18 08:04:07 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-node-exporter-b[61677]: ts=2024-09-18T08:04:07.281Z caller=filesystem_common.go:111 level=info collector=filesystem msg="Parsed flag --collector.filesystem.mount-points-exclude" flag=^/(dev|proc|run/credentials/.+|sys|var/lib/docker/.+|var/lib/containers/storage/.+)($|/) 2024-09-18T08:04:07.657 INFO:journalctl@ceph.node-exporter.b.smithi050.stdout:Sep 18 08:04:07 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-node-exporter-b[61677]: ts=2024-09-18T08:04:07.281Z caller=filesystem_common.go:113 level=info collector=filesystem msg="Parsed flag --collector.filesystem.fs-types-exclude" flag=^(autofs|binfmt_misc|bpf|cgroup2?|configfs|debugfs|devpts|devtmpfs|fusectl|hugetlbfs|iso9660|mqueue|nsfs|overlay|proc|procfs|pstore|rpc_pipefs|securityfs|selinuxfs|squashfs|sysfs|tracefs)$ 2024-09-18T08:04:07.657 INFO:journalctl@ceph.node-exporter.b.smithi050.stdout:Sep 18 08:04:07 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-node-exporter-b[61677]: ts=2024-09-18T08:04:07.283Z caller=diskstats_common.go:111 level=info collector=diskstats msg="Parsed flag --collector.diskstats.device-exclude" flag=^(ram|loop|fd|(h|s|v|xv)d[a-z]|nvme\d+n\d+p)\d+$ 2024-09-18T08:04:07.657 INFO:journalctl@ceph.node-exporter.b.smithi050.stdout:Sep 18 08:04:07 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-node-exporter-b[61677]: ts=2024-09-18T08:04:07.283Z caller=diskstats_linux.go:265 level=error collector=diskstats msg="Failed to open directory, disabling udev device properties" path=/run/udev/data 2024-09-18T08:04:07.657 INFO:journalctl@ceph.node-exporter.b.smithi050.stdout:Sep 18 08:04:07 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-node-exporter-b[61677]: ts=2024-09-18T08:04:07.283Z caller=node_exporter.go:110 level=info msg="Enabled collectors" 2024-09-18T08:04:07.657 INFO:journalctl@ceph.node-exporter.b.smithi050.stdout:Sep 18 08:04:07 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-node-exporter-b[61677]: ts=2024-09-18T08:04:07.283Z caller=node_exporter.go:117 level=info collector=arp 2024-09-18T08:04:07.657 INFO:journalctl@ceph.node-exporter.b.smithi050.stdout:Sep 18 08:04:07 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-node-exporter-b[61677]: ts=2024-09-18T08:04:07.283Z caller=node_exporter.go:117 level=info collector=bcache 2024-09-18T08:04:07.657 INFO:journalctl@ceph.node-exporter.b.smithi050.stdout:Sep 18 08:04:07 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-node-exporter-b[61677]: ts=2024-09-18T08:04:07.283Z caller=node_exporter.go:117 level=info collector=bonding 2024-09-18T08:04:07.657 INFO:journalctl@ceph.node-exporter.b.smithi050.stdout:Sep 18 08:04:07 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-node-exporter-b[61677]: ts=2024-09-18T08:04:07.283Z caller=node_exporter.go:117 level=info collector=btrfs 2024-09-18T08:04:07.657 INFO:journalctl@ceph.node-exporter.b.smithi050.stdout:Sep 18 08:04:07 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-node-exporter-b[61677]: ts=2024-09-18T08:04:07.283Z caller=node_exporter.go:117 level=info collector=conntrack 2024-09-18T08:04:07.657 INFO:journalctl@ceph.node-exporter.b.smithi050.stdout:Sep 18 08:04:07 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-node-exporter-b[61677]: ts=2024-09-18T08:04:07.283Z caller=node_exporter.go:117 level=info collector=cpu 2024-09-18T08:04:07.657 INFO:journalctl@ceph.node-exporter.b.smithi050.stdout:Sep 18 08:04:07 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-node-exporter-b[61677]: ts=2024-09-18T08:04:07.283Z caller=node_exporter.go:117 level=info collector=cpufreq 2024-09-18T08:04:07.657 INFO:journalctl@ceph.node-exporter.b.smithi050.stdout:Sep 18 08:04:07 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-node-exporter-b[61677]: ts=2024-09-18T08:04:07.283Z caller=node_exporter.go:117 level=info collector=diskstats 2024-09-18T08:04:07.658 INFO:journalctl@ceph.node-exporter.b.smithi050.stdout:Sep 18 08:04:07 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-node-exporter-b[61677]: ts=2024-09-18T08:04:07.283Z caller=node_exporter.go:117 level=info collector=dmi 2024-09-18T08:04:07.658 INFO:journalctl@ceph.node-exporter.b.smithi050.stdout:Sep 18 08:04:07 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-node-exporter-b[61677]: ts=2024-09-18T08:04:07.283Z caller=node_exporter.go:117 level=info collector=edac 2024-09-18T08:04:07.658 INFO:journalctl@ceph.node-exporter.b.smithi050.stdout:Sep 18 08:04:07 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-node-exporter-b[61677]: ts=2024-09-18T08:04:07.283Z caller=node_exporter.go:117 level=info collector=entropy 2024-09-18T08:04:07.658 INFO:journalctl@ceph.node-exporter.b.smithi050.stdout:Sep 18 08:04:07 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-node-exporter-b[61677]: ts=2024-09-18T08:04:07.283Z caller=node_exporter.go:117 level=info collector=fibrechannel 2024-09-18T08:04:07.658 INFO:journalctl@ceph.node-exporter.b.smithi050.stdout:Sep 18 08:04:07 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-node-exporter-b[61677]: ts=2024-09-18T08:04:07.283Z caller=node_exporter.go:117 level=info collector=filefd 2024-09-18T08:04:07.658 INFO:journalctl@ceph.node-exporter.b.smithi050.stdout:Sep 18 08:04:07 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-node-exporter-b[61677]: ts=2024-09-18T08:04:07.283Z caller=node_exporter.go:117 level=info collector=filesystem 2024-09-18T08:04:07.658 INFO:journalctl@ceph.node-exporter.b.smithi050.stdout:Sep 18 08:04:07 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-node-exporter-b[61677]: ts=2024-09-18T08:04:07.283Z caller=node_exporter.go:117 level=info collector=hwmon 2024-09-18T08:04:07.659 INFO:journalctl@ceph.node-exporter.b.smithi050.stdout:Sep 18 08:04:07 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-node-exporter-b[61677]: ts=2024-09-18T08:04:07.283Z caller=node_exporter.go:117 level=info collector=infiniband 2024-09-18T08:04:07.659 INFO:journalctl@ceph.node-exporter.b.smithi050.stdout:Sep 18 08:04:07 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-node-exporter-b[61677]: ts=2024-09-18T08:04:07.283Z caller=node_exporter.go:117 level=info collector=ipvs 2024-09-18T08:04:07.659 INFO:journalctl@ceph.node-exporter.b.smithi050.stdout:Sep 18 08:04:07 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-node-exporter-b[61677]: ts=2024-09-18T08:04:07.283Z caller=node_exporter.go:117 level=info collector=loadavg 2024-09-18T08:04:07.659 INFO:journalctl@ceph.node-exporter.b.smithi050.stdout:Sep 18 08:04:07 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-node-exporter-b[61677]: ts=2024-09-18T08:04:07.283Z caller=node_exporter.go:117 level=info collector=mdadm 2024-09-18T08:04:07.659 INFO:journalctl@ceph.node-exporter.b.smithi050.stdout:Sep 18 08:04:07 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-node-exporter-b[61677]: ts=2024-09-18T08:04:07.283Z caller=node_exporter.go:117 level=info collector=meminfo 2024-09-18T08:04:07.659 INFO:journalctl@ceph.node-exporter.b.smithi050.stdout:Sep 18 08:04:07 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-node-exporter-b[61677]: ts=2024-09-18T08:04:07.283Z caller=node_exporter.go:117 level=info collector=netclass 2024-09-18T08:04:07.659 INFO:journalctl@ceph.node-exporter.b.smithi050.stdout:Sep 18 08:04:07 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-node-exporter-b[61677]: ts=2024-09-18T08:04:07.283Z caller=node_exporter.go:117 level=info collector=netdev 2024-09-18T08:04:07.659 INFO:journalctl@ceph.node-exporter.b.smithi050.stdout:Sep 18 08:04:07 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-node-exporter-b[61677]: ts=2024-09-18T08:04:07.283Z caller=node_exporter.go:117 level=info collector=netstat 2024-09-18T08:04:07.659 INFO:journalctl@ceph.node-exporter.b.smithi050.stdout:Sep 18 08:04:07 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-node-exporter-b[61677]: ts=2024-09-18T08:04:07.283Z caller=node_exporter.go:117 level=info collector=nfs 2024-09-18T08:04:07.659 INFO:journalctl@ceph.node-exporter.b.smithi050.stdout:Sep 18 08:04:07 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-node-exporter-b[61677]: ts=2024-09-18T08:04:07.283Z caller=node_exporter.go:117 level=info collector=nfsd 2024-09-18T08:04:07.659 INFO:journalctl@ceph.node-exporter.b.smithi050.stdout:Sep 18 08:04:07 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-node-exporter-b[61677]: ts=2024-09-18T08:04:07.283Z caller=node_exporter.go:117 level=info collector=nvme 2024-09-18T08:04:07.659 INFO:journalctl@ceph.node-exporter.b.smithi050.stdout:Sep 18 08:04:07 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-node-exporter-b[61677]: ts=2024-09-18T08:04:07.283Z caller=node_exporter.go:117 level=info collector=os 2024-09-18T08:04:07.659 INFO:journalctl@ceph.node-exporter.b.smithi050.stdout:Sep 18 08:04:07 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-node-exporter-b[61677]: ts=2024-09-18T08:04:07.283Z caller=node_exporter.go:117 level=info collector=powersupplyclass 2024-09-18T08:04:07.660 INFO:journalctl@ceph.node-exporter.b.smithi050.stdout:Sep 18 08:04:07 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-node-exporter-b[61677]: ts=2024-09-18T08:04:07.283Z caller=node_exporter.go:117 level=info collector=pressure 2024-09-18T08:04:07.660 INFO:journalctl@ceph.node-exporter.b.smithi050.stdout:Sep 18 08:04:07 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-node-exporter-b[61677]: ts=2024-09-18T08:04:07.283Z caller=node_exporter.go:117 level=info collector=rapl 2024-09-18T08:04:07.660 INFO:journalctl@ceph.node-exporter.b.smithi050.stdout:Sep 18 08:04:07 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-node-exporter-b[61677]: ts=2024-09-18T08:04:07.283Z caller=node_exporter.go:117 level=info collector=schedstat 2024-09-18T08:04:07.660 INFO:journalctl@ceph.node-exporter.b.smithi050.stdout:Sep 18 08:04:07 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-node-exporter-b[61677]: ts=2024-09-18T08:04:07.283Z caller=node_exporter.go:117 level=info collector=selinux 2024-09-18T08:04:07.660 INFO:journalctl@ceph.node-exporter.b.smithi050.stdout:Sep 18 08:04:07 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-node-exporter-b[61677]: ts=2024-09-18T08:04:07.283Z caller=node_exporter.go:117 level=info collector=sockstat 2024-09-18T08:04:07.660 INFO:journalctl@ceph.node-exporter.b.smithi050.stdout:Sep 18 08:04:07 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-node-exporter-b[61677]: ts=2024-09-18T08:04:07.283Z caller=node_exporter.go:117 level=info collector=softnet 2024-09-18T08:04:07.660 INFO:journalctl@ceph.node-exporter.b.smithi050.stdout:Sep 18 08:04:07 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-node-exporter-b[61677]: ts=2024-09-18T08:04:07.283Z caller=node_exporter.go:117 level=info collector=stat 2024-09-18T08:04:07.660 INFO:journalctl@ceph.node-exporter.b.smithi050.stdout:Sep 18 08:04:07 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-node-exporter-b[61677]: ts=2024-09-18T08:04:07.283Z caller=node_exporter.go:117 level=info collector=tapestats 2024-09-18T08:04:07.660 INFO:journalctl@ceph.node-exporter.b.smithi050.stdout:Sep 18 08:04:07 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-node-exporter-b[61677]: ts=2024-09-18T08:04:07.283Z caller=node_exporter.go:117 level=info collector=textfile 2024-09-18T08:04:07.660 INFO:journalctl@ceph.node-exporter.b.smithi050.stdout:Sep 18 08:04:07 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-node-exporter-b[61677]: ts=2024-09-18T08:04:07.283Z caller=node_exporter.go:117 level=info collector=thermal_zone 2024-09-18T08:04:07.660 INFO:journalctl@ceph.node-exporter.b.smithi050.stdout:Sep 18 08:04:07 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-node-exporter-b[61677]: ts=2024-09-18T08:04:07.283Z caller=node_exporter.go:117 level=info collector=time 2024-09-18T08:04:07.660 INFO:journalctl@ceph.node-exporter.b.smithi050.stdout:Sep 18 08:04:07 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-node-exporter-b[61677]: ts=2024-09-18T08:04:07.283Z caller=node_exporter.go:117 level=info collector=udp_queues 2024-09-18T08:04:07.660 INFO:journalctl@ceph.node-exporter.b.smithi050.stdout:Sep 18 08:04:07 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-node-exporter-b[61677]: ts=2024-09-18T08:04:07.283Z caller=node_exporter.go:117 level=info collector=uname 2024-09-18T08:04:07.660 INFO:journalctl@ceph.node-exporter.b.smithi050.stdout:Sep 18 08:04:07 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-node-exporter-b[61677]: ts=2024-09-18T08:04:07.283Z caller=node_exporter.go:117 level=info collector=vmstat 2024-09-18T08:04:07.661 INFO:journalctl@ceph.node-exporter.b.smithi050.stdout:Sep 18 08:04:07 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-node-exporter-b[61677]: ts=2024-09-18T08:04:07.283Z caller=node_exporter.go:117 level=info collector=xfs 2024-09-18T08:04:07.661 INFO:journalctl@ceph.node-exporter.b.smithi050.stdout:Sep 18 08:04:07 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-node-exporter-b[61677]: ts=2024-09-18T08:04:07.283Z caller=node_exporter.go:117 level=info collector=zfs 2024-09-18T08:04:07.661 INFO:journalctl@ceph.node-exporter.b.smithi050.stdout:Sep 18 08:04:07 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-node-exporter-b[61677]: ts=2024-09-18T08:04:07.284Z caller=tls_config.go:274 level=info msg="Listening on" address=[::]:9100 2024-09-18T08:04:07.661 INFO:journalctl@ceph.node-exporter.b.smithi050.stdout:Sep 18 08:04:07 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-node-exporter-b[61677]: ts=2024-09-18T08:04:07.284Z caller=tls_config.go:277 level=info msg="TLS is disabled." http2=false address=[::]:9100 2024-09-18T08:04:07.661 INFO:journalctl@ceph.node-exporter.b.smithi050.stdout:Sep 18 08:04:07 smithi050 bash[61451]: baba185ed32a51ea670d7d4b4dc68a79a0cb4fab8e8dfaa06ce93162294ccd25 2024-09-18T08:04:07.661 INFO:journalctl@ceph.node-exporter.b.smithi050.stdout:Sep 18 08:04:07 smithi050 systemd[1]: Started Ceph node-exporter.b for 779349ca-7593-11ef-bceb-c7b262605968. 2024-09-18T08:04:07.677 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:04:07.678 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:04:07.679 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3' 2024-09-18T08:04:07.779 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4 2024-09-18T08:04:07.812 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:04:07.813 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:04:07.813 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4' 2024-09-18T08:04:07.914 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2 2024-09-18T08:04:07.954 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:04:07.955 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:04:07.956 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2' 2024-09-18T08:04:07.981 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:04:07 smithi007 ceph-mon[39249]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-18T08:04:07.981 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:04:07 smithi007 ceph-mon[39249]: osdmap e89: 8 total, 8 up, 8 in 2024-09-18T08:04:07.981 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:04:07 smithi007 ceph-mon[39249]: 6.f scrub starts 2024-09-18T08:04:07.981 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:04:07 smithi007 ceph-mon[39249]: 6.f scrub ok 2024-09-18T08:04:07.981 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:04:07 smithi007 ceph-mon[39249]: 2.0 scrub starts 2024-09-18T08:04:07.981 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:04:07 smithi007 ceph-mon[39249]: 2.0 scrub ok 2024-09-18T08:04:07.981 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:04:07 smithi007 ceph-mon[39249]: from='client.? 172.21.15.7:0/1938461574' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-18T08:04:07.981 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:04:07 smithi007 ceph-mon[39249]: pgmap v44: 148 pgs: 148 active+clean; 583 KiB data, 224 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-18T08:04:07.982 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:04:07 smithi007 ceph-mon[39249]: from='mgr.24470 172.21.15.7:0/2594390261' entity='mgr.y' 2024-09-18T08:04:07.982 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:04:07 smithi007 ceph-mon[39249]: from='mgr.24470 172.21.15.7:0/2594390261' entity='mgr.y' 2024-09-18T08:04:07.982 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:04:07 smithi007 ceph-mon[39249]: from='mgr.24470 172.21.15.7:0/2594390261' entity='mgr.y' 2024-09-18T08:04:07.982 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:04:07 smithi007 ceph-mon[39249]: purged_snaps scrub starts 2024-09-18T08:04:07.982 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:04:07 smithi007 ceph-mon[39249]: purged_snaps scrub ok 2024-09-18T08:04:07.982 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:04:07 smithi007 ceph-mon[39249]: from='mgr.24470 172.21.15.7:0/2594390261' entity='mgr.y' 2024-09-18T08:04:07.982 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:04:07 smithi007 ceph-mon[39249]: from='mgr.24470 172.21.15.7:0/2594390261' entity='mgr.y' 2024-09-18T08:04:07.982 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:04:07 smithi007 ceph-mon[39249]: from='mgr.24470 172.21.15.7:0/2594390261' entity='mgr.y' 2024-09-18T08:04:07.982 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:04:07 smithi007 ceph-mon[39249]: from='mgr.24470 172.21.15.7:0/2594390261' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-18T08:04:07.982 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:04:07 smithi007 ceph-mon[29459]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-18T08:04:07.982 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:04:07 smithi007 ceph-mon[29459]: osdmap e89: 8 total, 8 up, 8 in 2024-09-18T08:04:07.982 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:04:07 smithi007 ceph-mon[29459]: 6.f scrub starts 2024-09-18T08:04:07.983 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:04:07 smithi007 ceph-mon[29459]: 6.f scrub ok 2024-09-18T08:04:07.983 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:04:07 smithi007 ceph-mon[29459]: 2.0 scrub starts 2024-09-18T08:04:07.983 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:04:07 smithi007 ceph-mon[29459]: 2.0 scrub ok 2024-09-18T08:04:07.983 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:04:07 smithi007 ceph-mon[29459]: from='client.? 172.21.15.7:0/1938461574' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-18T08:04:07.983 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:04:07 smithi007 ceph-mon[29459]: pgmap v44: 148 pgs: 148 active+clean; 583 KiB data, 224 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-18T08:04:07.983 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:04:07 smithi007 ceph-mon[29459]: from='mgr.24470 172.21.15.7:0/2594390261' entity='mgr.y' 2024-09-18T08:04:07.983 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:04:07 smithi007 ceph-mon[29459]: from='mgr.24470 172.21.15.7:0/2594390261' entity='mgr.y' 2024-09-18T08:04:07.983 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:04:07 smithi007 ceph-mon[29459]: from='mgr.24470 172.21.15.7:0/2594390261' entity='mgr.y' 2024-09-18T08:04:07.983 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:04:07 smithi007 ceph-mon[29459]: purged_snaps scrub starts 2024-09-18T08:04:07.983 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:04:07 smithi007 ceph-mon[29459]: purged_snaps scrub ok 2024-09-18T08:04:07.983 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:04:07 smithi007 ceph-mon[29459]: from='mgr.24470 172.21.15.7:0/2594390261' entity='mgr.y' 2024-09-18T08:04:07.983 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:04:07 smithi007 ceph-mon[29459]: from='mgr.24470 172.21.15.7:0/2594390261' entity='mgr.y' 2024-09-18T08:04:07.983 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:04:07 smithi007 ceph-mon[29459]: from='mgr.24470 172.21.15.7:0/2594390261' entity='mgr.y' 2024-09-18T08:04:07.984 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:04:07 smithi007 ceph-mon[29459]: from='mgr.24470 172.21.15.7:0/2594390261' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-18T08:04:08.056 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3 2024-09-18T08:04:08.098 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:04:08.099 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:04:08.099 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3' 2024-09-18T08:04:08.156 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:04:07 smithi050 ceph-mon[32371]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-18T08:04:08.156 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:04:07 smithi050 ceph-mon[32371]: osdmap e89: 8 total, 8 up, 8 in 2024-09-18T08:04:08.156 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:04:07 smithi050 ceph-mon[32371]: 6.f scrub starts 2024-09-18T08:04:08.156 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:04:07 smithi050 ceph-mon[32371]: 6.f scrub ok 2024-09-18T08:04:08.156 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:04:07 smithi050 ceph-mon[32371]: 2.0 scrub starts 2024-09-18T08:04:08.157 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:04:07 smithi050 ceph-mon[32371]: 2.0 scrub ok 2024-09-18T08:04:08.157 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:04:07 smithi050 ceph-mon[32371]: from='client.? 172.21.15.7:0/1938461574' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-18T08:04:08.157 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:04:07 smithi050 ceph-mon[32371]: pgmap v44: 148 pgs: 148 active+clean; 583 KiB data, 224 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-18T08:04:08.157 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:04:07 smithi050 ceph-mon[32371]: from='mgr.24470 172.21.15.7:0/2594390261' entity='mgr.y' 2024-09-18T08:04:08.157 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:04:07 smithi050 ceph-mon[32371]: from='mgr.24470 172.21.15.7:0/2594390261' entity='mgr.y' 2024-09-18T08:04:08.157 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:04:07 smithi050 ceph-mon[32371]: from='mgr.24470 172.21.15.7:0/2594390261' entity='mgr.y' 2024-09-18T08:04:08.157 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:04:07 smithi050 ceph-mon[32371]: purged_snaps scrub starts 2024-09-18T08:04:08.157 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:04:07 smithi050 ceph-mon[32371]: purged_snaps scrub ok 2024-09-18T08:04:08.157 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:04:07 smithi050 ceph-mon[32371]: from='mgr.24470 172.21.15.7:0/2594390261' entity='mgr.y' 2024-09-18T08:04:08.158 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:04:07 smithi050 ceph-mon[32371]: from='mgr.24470 172.21.15.7:0/2594390261' entity='mgr.y' 2024-09-18T08:04:08.158 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:04:07 smithi050 ceph-mon[32371]: from='mgr.24470 172.21.15.7:0/2594390261' entity='mgr.y' 2024-09-18T08:04:08.158 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:04:07 smithi050 ceph-mon[32371]: from='mgr.24470 172.21.15.7:0/2594390261' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-18T08:04:08.174 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid 779349ca-7593-11ef-bceb-c7b262605968 -- ceph daemon osd.1 dump_historic_ops 2024-09-18T08:04:08.200 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6 2024-09-18T08:04:08.231 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:04:08.231 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:04:08.231 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6' 2024-09-18T08:04:08.332 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4 2024-09-18T08:04:08.362 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:04:08.363 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:04:08.364 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4' 2024-09-18T08:04:08.464 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4 2024-09-18T08:04:08.498 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:04:08.499 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:04:08.499 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4' 2024-09-18T08:04:08.567 INFO:tasks.radosbench.radosbench.0.smithi007.stdout: 3 15 8204 8189 170.582 173.938 0.00629797 0.00585231 2024-09-18T08:04:08.600 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5 2024-09-18T08:04:08.631 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:04:08.632 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:04:08.632 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5' 2024-09-18T08:04:08.733 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0 2024-09-18T08:04:08.774 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:04:08.775 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:04:08.775 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0' 2024-09-18T08:04:08.876 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0 2024-09-18T08:04:08.915 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:04:08.916 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:04:08.917 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0' 2024-09-18T08:04:08.981 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:04:08 smithi007 ceph-mon[29459]: 4.3 scrub starts 2024-09-18T08:04:08.981 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:04:08 smithi007 ceph-mon[29459]: 4.3 scrub ok 2024-09-18T08:04:08.981 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:04:08 smithi007 ceph-mon[29459]: 5.1d scrub starts 2024-09-18T08:04:08.981 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:04:08 smithi007 ceph-mon[29459]: 5.1d scrub ok 2024-09-18T08:04:08.981 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:04:08 smithi007 ceph-mon[29459]: osdmap e90: 8 total, 8 up, 8 in 2024-09-18T08:04:08.982 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:04:08 smithi007 ceph-mon[39249]: 4.3 scrub starts 2024-09-18T08:04:08.982 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:04:08 smithi007 ceph-mon[39249]: 4.3 scrub ok 2024-09-18T08:04:08.982 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:04:08 smithi007 ceph-mon[39249]: 5.1d scrub starts 2024-09-18T08:04:08.982 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:04:08 smithi007 ceph-mon[39249]: 5.1d scrub ok 2024-09-18T08:04:08.982 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:04:08 smithi007 ceph-mon[39249]: osdmap e90: 8 total, 8 up, 8 in 2024-09-18T08:04:09.018 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3 2024-09-18T08:04:09.056 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:04:09.057 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:04:09.057 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3' 2024-09-18T08:04:09.119 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-09-18T08:04:09.121 INFO:teuthology.orchestra.run.smithi007.stderr:dumped all 2024-09-18T08:04:09.156 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:04:08 smithi050 ceph-mon[32371]: 4.3 scrub starts 2024-09-18T08:04:09.156 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:04:08 smithi050 ceph-mon[32371]: 4.3 scrub ok 2024-09-18T08:04:09.156 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:04:08 smithi050 ceph-mon[32371]: 5.1d scrub starts 2024-09-18T08:04:09.156 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:04:08 smithi050 ceph-mon[32371]: 5.1d scrub ok 2024-09-18T08:04:09.156 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:04:08 smithi050 ceph-mon[32371]: osdmap e90: 8 total, 8 up, 8 in 2024-09-18T08:04:09.156 INFO:journalctl@ceph.iscsi.iscsi.a.smithi050.stdout:Sep 18 08:04:08 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-iscsi-iscsi-a[57782]: debug there is no tcmu-runner data available 2024-09-18T08:04:09.158 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1 2024-09-18T08:04:09.198 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:04:09.199 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:04:09.200 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1' 2024-09-18T08:04:09.301 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3 2024-09-18T08:04:09.342 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:04:09.343 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:04:09.344 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3' 2024-09-18T08:04:09.445 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0 2024-09-18T08:04:09.485 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:04:09.486 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:04:09.486 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0' 2024-09-18T08:04:09.567 INFO:tasks.radosbench.radosbench.0.smithi007.stdout: 4 15 10947 10932 170.792 171.438 0.00601094 0.00584654 2024-09-18T08:04:09.587 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2 2024-09-18T08:04:09.627 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:04:09.628 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:04:09.628 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2' 2024-09-18T08:04:09.715 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:04:09 smithi050 ceph-mon[32371]: 4.17 scrub starts 2024-09-18T08:04:09.716 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:04:09 smithi050 ceph-mon[32371]: 4.17 scrub ok 2024-09-18T08:04:09.716 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:04:09 smithi050 ceph-mon[32371]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-18T08:04:09.716 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:04:09 smithi050 ceph-mon[32371]: pgmap v46: 148 pgs: 148 active+clean; 431 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 1.7 KiB/s rd, 72 MiB/s wr, 1.15k op/s 2024-09-18T08:04:09.716 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:04:09 smithi050 ceph-mon[32371]: from='client.24676 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-18T08:04:09.716 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:04:09 smithi050 ceph-mon[32371]: 3.a scrub starts 2024-09-18T08:04:09.716 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:04:09 smithi050 ceph-mon[32371]: 3.a scrub ok 2024-09-18T08:04:09.729 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1 2024-09-18T08:04:09.762 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:04:09 smithi007 ceph-mon[39249]: 4.17 scrub starts 2024-09-18T08:04:09.762 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:04:09 smithi007 ceph-mon[39249]: 4.17 scrub ok 2024-09-18T08:04:09.763 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:04:09 smithi007 ceph-mon[39249]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-18T08:04:09.763 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:04:09 smithi007 ceph-mon[39249]: pgmap v46: 148 pgs: 148 active+clean; 431 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 1.7 KiB/s rd, 72 MiB/s wr, 1.15k op/s 2024-09-18T08:04:09.763 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:04:09 smithi007 ceph-mon[39249]: from='client.24676 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-18T08:04:09.763 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:04:09 smithi007 ceph-mon[39249]: 3.a scrub starts 2024-09-18T08:04:09.763 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:04:09 smithi007 ceph-mon[39249]: 3.a scrub ok 2024-09-18T08:04:09.763 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:04:09 smithi007 ceph-mon[29459]: 4.17 scrub starts 2024-09-18T08:04:09.763 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:04:09 smithi007 ceph-mon[29459]: 4.17 scrub ok 2024-09-18T08:04:09.763 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:04:09 smithi007 ceph-mon[29459]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-18T08:04:09.763 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:04:09 smithi007 ceph-mon[29459]: pgmap v46: 148 pgs: 148 active+clean; 431 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 1.7 KiB/s rd, 72 MiB/s wr, 1.15k op/s 2024-09-18T08:04:09.763 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:04:09 smithi007 ceph-mon[29459]: from='client.24676 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-18T08:04:09.763 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:04:09 smithi007 ceph-mon[29459]: 3.a scrub starts 2024-09-18T08:04:09.763 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:04:09 smithi007 ceph-mon[29459]: 3.a scrub ok 2024-09-18T08:04:09.768 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:04:09.769 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:04:09.769 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1' 2024-09-18T08:04:09.870 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2 2024-09-18T08:04:09.910 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:04:09.911 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:04:09.911 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2' 2024-09-18T08:04:10.012 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3 2024-09-18T08:04:10.050 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:04:10.050 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:04:10.051 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3' 2024-09-18T08:04:10.152 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5 2024-09-18T08:04:10.184 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:04:10.185 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:04:10.186 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5' 2024-09-18T08:04:10.286 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2 2024-09-18T08:04:10.329 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:04:10.329 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:04:10.330 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2' 2024-09-18T08:04:10.430 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0 2024-09-18T08:04:10.466 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:04:10.467 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:04:10.467 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0' 2024-09-18T08:04:10.568 INFO:tasks.radosbench.radosbench.0.smithi007.stdout: 5 16 13737 13721 171.493 174.312 0.00487706 0.00582462 2024-09-18T08:04:10.568 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4 2024-09-18T08:04:10.601 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:04:10.602 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:04:10.602 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4' 2024-09-18T08:04:10.703 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1 2024-09-18T08:04:10.745 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:04:10.745 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:04:10.746 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1' 2024-09-18T08:04:10.846 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6 2024-09-18T08:04:10.869 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:04:10.870 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:04:10.870 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6' 2024-09-18T08:04:10.906 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:04:10 smithi050 ceph-mon[32371]: 3.8 scrub starts 2024-09-18T08:04:10.906 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:04:10 smithi050 ceph-mon[32371]: 3.8 scrub ok 2024-09-18T08:04:10.906 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:04:10 smithi050 ceph-mon[32371]: 3.6 scrub starts 2024-09-18T08:04:10.906 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:04:10 smithi050 ceph-mon[32371]: 3.6 scrub ok 2024-09-18T08:04:10.906 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:04:10 smithi050 ceph-mon[32371]: 2.1 scrub starts 2024-09-18T08:04:10.906 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:04:10 smithi050 ceph-mon[32371]: 2.1 scrub ok 2024-09-18T08:04:10.971 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6 2024-09-18T08:04:10.980 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:04:10 smithi007 ceph-mon[29459]: 3.8 scrub starts 2024-09-18T08:04:10.981 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:04:10 smithi007 ceph-mon[29459]: 3.8 scrub ok 2024-09-18T08:04:10.981 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:04:10 smithi007 ceph-mon[29459]: 3.6 scrub starts 2024-09-18T08:04:10.981 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:04:10 smithi007 ceph-mon[29459]: 3.6 scrub ok 2024-09-18T08:04:10.981 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:04:10 smithi007 ceph-mon[29459]: 2.1 scrub starts 2024-09-18T08:04:10.981 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:04:10 smithi007 ceph-mon[29459]: 2.1 scrub ok 2024-09-18T08:04:10.982 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:04:10 smithi007 ceph-mon[39249]: 3.8 scrub starts 2024-09-18T08:04:10.982 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:04:10 smithi007 ceph-mon[39249]: 3.8 scrub ok 2024-09-18T08:04:10.983 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:04:10 smithi007 ceph-mon[39249]: 3.6 scrub starts 2024-09-18T08:04:10.983 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:04:10 smithi007 ceph-mon[39249]: 3.6 scrub ok 2024-09-18T08:04:10.983 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:04:10 smithi007 ceph-mon[39249]: 2.1 scrub starts 2024-09-18T08:04:10.983 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:04:10 smithi007 ceph-mon[39249]: 2.1 scrub ok 2024-09-18T08:04:11.007 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:04:11.008 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:04:11.008 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6' 2024-09-18T08:04:11.109 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3 2024-09-18T08:04:11.157 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:04:11.159 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:04:11.159 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3' 2024-09-18T08:04:11.260 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7 2024-09-18T08:04:11.291 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:04:11.292 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:04:11.293 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7' 2024-09-18T08:04:11.394 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5 2024-09-18T08:04:11.429 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:04:11.429 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:04:11.430 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5' 2024-09-18T08:04:11.492 INFO:teuthology.orchestra.run.smithi007.stdout:{"pg_ready":true,"pg_map":{"version":46,"stamp":"2024-09-18T08:04:09.077653+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":451746331,"num_objects":7083,"num_object_clones":0,"num_object_copies":21249,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7083,"num_whiteouts":0,"num_read":923,"num_read_kb":780,"num_write":7433,"num_write_kb":442249,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":590368,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7436,"ondisk_log_size":7436,"up":444,"acting":444,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":444,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":1364984,"kb_used_data":1082140,"kb_used_omap":12,"kb_used_meta":282611,"kb_avail":748596232,"statfs":{"total":767960285184,"available":766562541568,"internally_reserved":0,"allocated":1108111360,"data_stored":1103802905,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12750,"internal_metadata":289394226},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,1,1,5,1],"upper_bound":32},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":451149824,"num_objects":6884,"num_object_clones":0,"num_object_copies":20652,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6884,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":6884,"num_write_kb":440576,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.001617"},"pg_stats":[{"pgid":"3.1f","version":"0'0","reported_seq":66,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-09-18T08:04:06.665569+0000","last_change":"2024-09-18T08:02:47.878020+0000","last_active":"2024-09-18T08:04:06.665569+0000","last_peered":"2024-09-18T08:04:06.665569+0000","last_clean":"2024-09-18T08:04:06.665569+0000","last_became_active":"2024-09-18T08:02:47.877698+0000","last_became_peered":"2024-09-18T08:02:47.877698+0000","last_unstale":"2024-09-18T08:04:06.665569+0000","last_undegraded":"2024-09-18T08:04:06.665569+0000","last_fullsized":"2024-09-18T08:04:06.665569+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:02:46.860092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:46.860092+0000","last_clean_scrub_stamp":"2024-09-18T08:02:46.860092+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:04:14.205967+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"69'9","reported_seq":78,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-09-18T08:04:06.660369+0000","last_change":"2024-09-18T08:02:49.895688+0000","last_active":"2024-09-18T08:04:06.660369+0000","last_peered":"2024-09-18T08:04:06.660369+0000","last_clean":"2024-09-18T08:04:06.660369+0000","last_became_active":"2024-09-18T08:02:49.894557+0000","last_became_peered":"2024-09-18T08:02:49.894557+0000","last_unstale":"2024-09-18T08:04:06.660369+0000","last_undegraded":"2024-09-18T08:04:06.660369+0000","last_fullsized":"2024-09-18T08:04:06.660369+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_clean_scrub_stamp":"2024-09-18T08:02:48.870718+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:05:05.275706+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":58,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-09-18T08:04:06.668169+0000","last_change":"2024-09-18T08:02:51.907566+0000","last_active":"2024-09-18T08:04:06.668169+0000","last_peered":"2024-09-18T08:04:06.668169+0000","last_clean":"2024-09-18T08:04:06.668169+0000","last_became_active":"2024-09-18T08:02:51.907140+0000","last_became_peered":"2024-09-18T08:02:51.907140+0000","last_unstale":"2024-09-18T08:04:06.668169+0000","last_undegraded":"2024-09-18T08:04:06.668169+0000","last_fullsized":"2024-09-18T08:04:06.668169+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:02:50.886830+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:50.886830+0000","last_clean_scrub_stamp":"2024-09-18T08:02:50.886830+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:04:07.003335+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":55,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-09-18T08:04:06.660611+0000","last_change":"2024-09-18T08:02:53.906914+0000","last_active":"2024-09-18T08:04:06.660611+0000","last_peered":"2024-09-18T08:04:06.660611+0000","last_clean":"2024-09-18T08:04:06.660611+0000","last_became_active":"2024-09-18T08:02:53.906622+0000","last_became_peered":"2024-09-18T08:02:53.906622+0000","last_unstale":"2024-09-18T08:04:06.660611+0000","last_undegraded":"2024-09-18T08:04:06.660611+0000","last_fullsized":"2024-09-18T08:04:06.660611+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_clean_scrub_stamp":"2024-09-18T08:02:52.890475+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:04:20.738381+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":54,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-09-18T08:04:06.672229+0000","last_change":"2024-09-18T08:02:54.047999+0000","last_active":"2024-09-18T08:04:06.672229+0000","last_peered":"2024-09-18T08:04:06.672229+0000","last_clean":"2024-09-18T08:04:06.672229+0000","last_became_active":"2024-09-18T08:02:54.047640+0000","last_became_peered":"2024-09-18T08:02:54.047640+0000","last_unstale":"2024-09-18T08:04:06.672229+0000","last_undegraded":"2024-09-18T08:04:06.672229+0000","last_fullsized":"2024-09-18T08:04:06.672229+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_clean_scrub_stamp":"2024-09-18T08:02:52.890475+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:04:16.013706+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":66,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-09-18T08:04:06.672230+0000","last_change":"2024-09-18T08:02:47.872585+0000","last_active":"2024-09-18T08:04:06.672230+0000","last_peered":"2024-09-18T08:04:06.672230+0000","last_clean":"2024-09-18T08:04:06.672230+0000","last_became_active":"2024-09-18T08:02:47.872316+0000","last_became_peered":"2024-09-18T08:02:47.872316+0000","last_unstale":"2024-09-18T08:04:06.672230+0000","last_undegraded":"2024-09-18T08:04:06.672230+0000","last_fullsized":"2024-09-18T08:04:06.672230+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:02:46.860092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:46.860092+0000","last_clean_scrub_stamp":"2024-09-18T08:02:46.860092+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:04:54.830078+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"69'15","reported_seq":87,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-09-18T08:04:06.671284+0000","last_change":"2024-09-18T08:02:49.887111+0000","last_active":"2024-09-18T08:04:06.671284+0000","last_peered":"2024-09-18T08:04:06.671284+0000","last_clean":"2024-09-18T08:04:06.671284+0000","last_became_active":"2024-09-18T08:02:49.886106+0000","last_became_peered":"2024-09-18T08:02:49.886106+0000","last_unstale":"2024-09-18T08:04:06.671284+0000","last_undegraded":"2024-09-18T08:04:06.671284+0000","last_fullsized":"2024-09-18T08:04:06.671284+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_clean_scrub_stamp":"2024-09-18T08:02:48.870718+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:04:15.737211+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":58,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-09-18T08:04:06.660754+0000","last_change":"2024-09-18T08:02:51.904853+0000","last_active":"2024-09-18T08:04:06.660754+0000","last_peered":"2024-09-18T08:04:06.660754+0000","last_clean":"2024-09-18T08:04:06.660754+0000","last_became_active":"2024-09-18T08:02:51.904184+0000","last_became_peered":"2024-09-18T08:02:51.904184+0000","last_unstale":"2024-09-18T08:04:06.660754+0000","last_undegraded":"2024-09-18T08:04:06.660754+0000","last_fullsized":"2024-09-18T08:04:06.660754+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:02:50.886830+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:50.886830+0000","last_clean_scrub_stamp":"2024-09-18T08:02:50.886830+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:04:07.696637+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":74,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-09-18T08:04:06.666251+0000","last_change":"2024-09-18T08:03:53.821133+0000","last_active":"2024-09-18T08:04:06.666251+0000","last_peered":"2024-09-18T08:04:06.666251+0000","last_clean":"2024-09-18T08:04:06.666251+0000","last_became_active":"2024-09-18T08:02:47.875124+0000","last_became_peered":"2024-09-18T08:02:47.875124+0000","last_unstale":"2024-09-18T08:04:06.666251+0000","last_undegraded":"2024-09-18T08:04:06.666251+0000","last_fullsized":"2024-09-18T08:04:06.666251+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:03:53.821070+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:46.860092+0000","last_clean_scrub_stamp":"2024-09-18T08:03:53.821070+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:05:19.305220+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"69'9","reported_seq":78,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-09-18T08:04:06.660886+0000","last_change":"2024-09-18T08:02:49.887589+0000","last_active":"2024-09-18T08:04:06.660886+0000","last_peered":"2024-09-18T08:04:06.660886+0000","last_clean":"2024-09-18T08:04:06.660886+0000","last_became_active":"2024-09-18T08:02:49.887161+0000","last_became_peered":"2024-09-18T08:02:49.887161+0000","last_unstale":"2024-09-18T08:04:06.660886+0000","last_undegraded":"2024-09-18T08:04:06.660886+0000","last_fullsized":"2024-09-18T08:04:06.660886+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_clean_scrub_stamp":"2024-09-18T08:02:48.870718+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:04:54.327914+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":58,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-09-18T08:04:06.665554+0000","last_change":"2024-09-18T08:02:51.907597+0000","last_active":"2024-09-18T08:04:06.665554+0000","last_peered":"2024-09-18T08:04:06.665554+0000","last_clean":"2024-09-18T08:04:06.665554+0000","last_became_active":"2024-09-18T08:02:51.907296+0000","last_became_peered":"2024-09-18T08:02:51.907296+0000","last_unstale":"2024-09-18T08:04:06.665554+0000","last_undegraded":"2024-09-18T08:04:06.665554+0000","last_fullsized":"2024-09-18T08:04:06.665554+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:02:50.886830+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:50.886830+0000","last_clean_scrub_stamp":"2024-09-18T08:02:50.886830+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":55,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-09-18T08:04:06.665348+0000","last_change":"2024-09-18T08:02:53.913803+0000","last_active":"2024-09-18T08:04:06.665348+0000","last_peered":"2024-09-18T08:04:06.665348+0000","last_clean":"2024-09-18T08:04:06.665348+0000","last_became_active":"2024-09-18T08:02:53.913519+0000","last_became_peered":"2024-09-18T08:02:53.913519+0000","last_unstale":"2024-09-18T08:04:06.665348+0000","last_undegraded":"2024-09-18T08:04:06.665348+0000","last_fullsized":"2024-09-18T08:04:06.665348+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_clean_scrub_stamp":"2024-09-18T08:02:52.890475+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:04:18.699540+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":66,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-09-18T08:04:06.665761+0000","last_change":"2024-09-18T08:02:47.877577+0000","last_active":"2024-09-18T08:04:06.665761+0000","last_peered":"2024-09-18T08:04:06.665761+0000","last_clean":"2024-09-18T08:04:06.665761+0000","last_became_active":"2024-09-18T08:02:47.875042+0000","last_became_peered":"2024-09-18T08:02:47.875042+0000","last_unstale":"2024-09-18T08:04:06.665761+0000","last_undegraded":"2024-09-18T08:04:06.665761+0000","last_fullsized":"2024-09-18T08:04:06.665761+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:02:46.860092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:46.860092+0000","last_clean_scrub_stamp":"2024-09-18T08:02:46.860092+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:04:14.469212+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1b","version":"69'5","reported_seq":72,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-09-18T08:04:06.661317+0000","last_change":"2024-09-18T08:02:49.895779+0000","last_active":"2024-09-18T08:04:06.661317+0000","last_peered":"2024-09-18T08:04:06.661317+0000","last_clean":"2024-09-18T08:04:06.661317+0000","last_became_active":"2024-09-18T08:02:49.895344+0000","last_became_peered":"2024-09-18T08:02:49.895344+0000","last_unstale":"2024-09-18T08:04:06.661317+0000","last_undegraded":"2024-09-18T08:04:06.661317+0000","last_fullsized":"2024-09-18T08:04:06.661317+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_clean_scrub_stamp":"2024-09-18T08:02:48.870718+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:04:08.365283+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":11,"num_read_kb":7,"num_write":6,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":58,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-09-18T08:04:06.665864+0000","last_change":"2024-09-18T08:02:51.906694+0000","last_active":"2024-09-18T08:04:06.665864+0000","last_peered":"2024-09-18T08:04:06.665864+0000","last_clean":"2024-09-18T08:04:06.665864+0000","last_became_active":"2024-09-18T08:02:51.905776+0000","last_became_peered":"2024-09-18T08:02:51.905776+0000","last_unstale":"2024-09-18T08:04:06.665864+0000","last_undegraded":"2024-09-18T08:04:06.665864+0000","last_fullsized":"2024-09-18T08:04:06.665864+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:02:50.886830+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:50.886830+0000","last_clean_scrub_stamp":"2024-09-18T08:02:50.886830+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:04:07.273083+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":55,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-09-18T08:04:06.665292+0000","last_change":"2024-09-18T08:02:53.911969+0000","last_active":"2024-09-18T08:04:06.665292+0000","last_peered":"2024-09-18T08:04:06.665292+0000","last_clean":"2024-09-18T08:04:06.665292+0000","last_became_active":"2024-09-18T08:02:53.911648+0000","last_became_peered":"2024-09-18T08:02:53.911648+0000","last_unstale":"2024-09-18T08:04:06.665292+0000","last_undegraded":"2024-09-18T08:04:06.665292+0000","last_fullsized":"2024-09-18T08:04:06.665292+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_clean_scrub_stamp":"2024-09-18T08:02:52.890475+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:04:21.185524+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":54,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-09-18T08:04:06.661590+0000","last_change":"2024-09-18T08:02:54.047071+0000","last_active":"2024-09-18T08:04:06.661590+0000","last_peered":"2024-09-18T08:04:06.661590+0000","last_clean":"2024-09-18T08:04:06.661590+0000","last_became_active":"2024-09-18T08:02:54.046776+0000","last_became_peered":"2024-09-18T08:02:54.046776+0000","last_unstale":"2024-09-18T08:04:06.661590+0000","last_undegraded":"2024-09-18T08:04:06.661590+0000","last_fullsized":"2024-09-18T08:04:06.661590+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_clean_scrub_stamp":"2024-09-18T08:02:52.890475+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:04:11.355797+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":66,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-09-18T08:04:06.666161+0000","last_change":"2024-09-18T08:02:47.876235+0000","last_active":"2024-09-18T08:04:06.666161+0000","last_peered":"2024-09-18T08:04:06.666161+0000","last_clean":"2024-09-18T08:04:06.666161+0000","last_became_active":"2024-09-18T08:02:47.875963+0000","last_became_peered":"2024-09-18T08:02:47.875963+0000","last_unstale":"2024-09-18T08:04:06.666161+0000","last_undegraded":"2024-09-18T08:04:06.666161+0000","last_fullsized":"2024-09-18T08:04:06.666161+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:02:46.860092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:46.860092+0000","last_clean_scrub_stamp":"2024-09-18T08:02:46.860092+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:04:11.609411+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1c","version":"69'15","reported_seq":87,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-09-18T08:04:06.664353+0000","last_change":"2024-09-18T08:02:49.885923+0000","last_active":"2024-09-18T08:04:06.664353+0000","last_peered":"2024-09-18T08:04:06.664353+0000","last_clean":"2024-09-18T08:04:06.664353+0000","last_became_active":"2024-09-18T08:02:49.885553+0000","last_became_peered":"2024-09-18T08:02:49.885553+0000","last_unstale":"2024-09-18T08:04:06.664353+0000","last_undegraded":"2024-09-18T08:04:06.664353+0000","last_fullsized":"2024-09-18T08:04:06.664353+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_clean_scrub_stamp":"2024-09-18T08:02:48.870718+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:04:11.220046+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":66,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-09-18T08:04:07.769959+0000","last_change":"2024-09-18T08:04:07.769959+0000","last_active":"2024-09-18T08:04:07.769959+0000","last_peered":"2024-09-18T08:04:07.769959+0000","last_clean":"2024-09-18T08:04:07.769959+0000","last_became_active":"2024-09-18T08:02:51.908430+0000","last_became_peered":"2024-09-18T08:02:51.908430+0000","last_unstale":"2024-09-18T08:04:07.769959+0000","last_undegraded":"2024-09-18T08:04:07.769959+0000","last_fullsized":"2024-09-18T08:04:07.769959+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:04:07.769902+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:50.886830+0000","last_clean_scrub_stamp":"2024-09-18T08:04:07.769902+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:05:23.120117+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":54,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-09-18T08:04:06.671728+0000","last_change":"2024-09-18T08:02:54.047573+0000","last_active":"2024-09-18T08:04:06.671728+0000","last_peered":"2024-09-18T08:04:06.671728+0000","last_clean":"2024-09-18T08:04:06.671728+0000","last_became_active":"2024-09-18T08:02:54.047162+0000","last_became_peered":"2024-09-18T08:02:54.047162+0000","last_unstale":"2024-09-18T08:04:06.671728+0000","last_undegraded":"2024-09-18T08:04:06.671728+0000","last_fullsized":"2024-09-18T08:04:06.671728+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_clean_scrub_stamp":"2024-09-18T08:02:52.890475+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":66,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-09-18T08:04:06.661027+0000","last_change":"2024-09-18T08:02:47.871895+0000","last_active":"2024-09-18T08:04:06.661027+0000","last_peered":"2024-09-18T08:04:06.661027+0000","last_clean":"2024-09-18T08:04:06.661027+0000","last_became_active":"2024-09-18T08:02:47.871523+0000","last_became_peered":"2024-09-18T08:02:47.871523+0000","last_unstale":"2024-09-18T08:04:06.661027+0000","last_undegraded":"2024-09-18T08:04:06.661027+0000","last_fullsized":"2024-09-18T08:04:06.661027+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:02:46.860092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:46.860092+0000","last_clean_scrub_stamp":"2024-09-18T08:02:46.860092+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"69'12","reported_seq":85,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-09-18T08:04:06.672546+0000","last_change":"2024-09-18T08:02:49.889182+0000","last_active":"2024-09-18T08:04:06.672546+0000","last_peered":"2024-09-18T08:04:06.672546+0000","last_clean":"2024-09-18T08:04:06.672546+0000","last_became_active":"2024-09-18T08:02:49.888831+0000","last_became_peered":"2024-09-18T08:02:49.888831+0000","last_unstale":"2024-09-18T08:04:06.672546+0000","last_undegraded":"2024-09-18T08:04:06.672546+0000","last_fullsized":"2024-09-18T08:04:06.672546+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_clean_scrub_stamp":"2024-09-18T08:02:48.870718+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:04:12.415919+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":25,"num_read_kb":16,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":58,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-09-18T08:04:06.661369+0000","last_change":"2024-09-18T08:02:51.910642+0000","last_active":"2024-09-18T08:04:06.661369+0000","last_peered":"2024-09-18T08:04:06.661369+0000","last_clean":"2024-09-18T08:04:06.661369+0000","last_became_active":"2024-09-18T08:02:51.910316+0000","last_became_peered":"2024-09-18T08:02:51.910316+0000","last_unstale":"2024-09-18T08:04:06.661369+0000","last_undegraded":"2024-09-18T08:04:06.661369+0000","last_fullsized":"2024-09-18T08:04:06.661369+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:02:50.886830+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:50.886830+0000","last_clean_scrub_stamp":"2024-09-18T08:02:50.886830+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1c","version":"69'1","reported_seq":56,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-09-18T08:04:06.666159+0000","last_change":"2024-09-18T08:02:53.910861+0000","last_active":"2024-09-18T08:04:06.666159+0000","last_peered":"2024-09-18T08:04:06.666159+0000","last_clean":"2024-09-18T08:04:06.666159+0000","last_became_active":"2024-09-18T08:02:53.910589+0000","last_became_peered":"2024-09-18T08:02:53.910589+0000","last_unstale":"2024-09-18T08:04:06.666159+0000","last_undegraded":"2024-09-18T08:04:06.666159+0000","last_fullsized":"2024-09-18T08:04:06.666159+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_clean_scrub_stamp":"2024-09-18T08:02:52.890475+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:04:20.224154+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":403,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.19","version":"62'1","reported_seq":67,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-09-18T08:04:06.668300+0000","last_change":"2024-09-18T08:02:47.881314+0000","last_active":"2024-09-18T08:04:06.668300+0000","last_peered":"2024-09-18T08:04:06.668300+0000","last_clean":"2024-09-18T08:04:06.668300+0000","last_became_active":"2024-09-18T08:02:47.880998+0000","last_became_peered":"2024-09-18T08:02:47.880998+0000","last_unstale":"2024-09-18T08:04:06.668300+0000","last_undegraded":"2024-09-18T08:04:06.668300+0000","last_fullsized":"2024-09-18T08:04:06.668300+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:02:46.860092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:46.860092+0000","last_clean_scrub_stamp":"2024-09-18T08:02:46.860092+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:04:57.727045+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1e","version":"69'10","reported_seq":77,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-09-18T08:04:06.669972+0000","last_change":"2024-09-18T08:02:49.892755+0000","last_active":"2024-09-18T08:04:06.669972+0000","last_peered":"2024-09-18T08:04:06.669972+0000","last_clean":"2024-09-18T08:04:06.669972+0000","last_became_active":"2024-09-18T08:02:49.892024+0000","last_became_peered":"2024-09-18T08:02:49.892024+0000","last_unstale":"2024-09-18T08:04:06.669972+0000","last_undegraded":"2024-09-18T08:04:06.669972+0000","last_fullsized":"2024-09-18T08:04:06.669972+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_clean_scrub_stamp":"2024-09-18T08:02:48.870718+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:04:17.237822+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1f","version":"69'8","reported_seq":86,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-09-18T08:04:06.664416+0000","last_change":"2024-09-18T08:02:51.904933+0000","last_active":"2024-09-18T08:04:06.664416+0000","last_peered":"2024-09-18T08:04:06.664416+0000","last_clean":"2024-09-18T08:04:06.664416+0000","last_became_active":"2024-09-18T08:02:51.904588+0000","last_became_peered":"2024-09-18T08:02:51.904588+0000","last_unstale":"2024-09-18T08:04:06.664416+0000","last_undegraded":"2024-09-18T08:04:06.664416+0000","last_fullsized":"2024-09-18T08:04:06.664416+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:02:50.886830+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:50.886830+0000","last_clean_scrub_stamp":"2024-09-18T08:02:50.886830+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:05:01.435066+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.c","version":"89'457","reported_seq":480,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-09-18T08:04:08.064702+0000","last_change":"2024-09-18T08:03:57.465564+0000","last_active":"2024-09-18T08:04:08.064702+0000","last_peered":"2024-09-18T08:04:08.064702+0000","last_clean":"2024-09-18T08:04:08.064702+0000","last_became_active":"2024-09-18T08:03:57.465197+0000","last_became_peered":"2024-09-18T08:03:57.465197+0000","last_unstale":"2024-09-18T08:04:08.064702+0000","last_undegraded":"2024-09-18T08:04:08.064702+0000","last_fullsized":"2024-09-18T08:04:08.064702+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:03:56.449957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:03:56.449957+0000","last_clean_scrub_stamp":"2024-09-18T08:03:56.449957+0000","objects_scrubbed":0,"log_size":457,"log_dups_size":0,"ondisk_log_size":457,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:05:01.552772+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29949952,"num_objects":457,"num_object_clones":0,"num_object_copies":1371,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":457,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":457,"num_write_kb":29248,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,7],"acting":[6,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.f","version":"69'15","reported_seq":87,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-09-18T08:04:06.671657+0000","last_change":"2024-09-18T08:02:49.899404+0000","last_active":"2024-09-18T08:04:06.671657+0000","last_peered":"2024-09-18T08:04:06.671657+0000","last_clean":"2024-09-18T08:04:06.671657+0000","last_became_active":"2024-09-18T08:02:49.899083+0000","last_became_peered":"2024-09-18T08:02:49.899083+0000","last_unstale":"2024-09-18T08:04:06.671657+0000","last_undegraded":"2024-09-18T08:04:06.671657+0000","last_fullsized":"2024-09-18T08:04:06.671657+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_clean_scrub_stamp":"2024-09-18T08:02:48.870718+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":66,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-09-18T08:04:06.671808+0000","last_change":"2024-09-18T08:02:47.879723+0000","last_active":"2024-09-18T08:04:06.671808+0000","last_peered":"2024-09-18T08:04:06.671808+0000","last_clean":"2024-09-18T08:04:06.671808+0000","last_became_active":"2024-09-18T08:02:47.879328+0000","last_became_peered":"2024-09-18T08:02:47.879328+0000","last_unstale":"2024-09-18T08:04:06.671808+0000","last_undegraded":"2024-09-18T08:04:06.671808+0000","last_fullsized":"2024-09-18T08:04:06.671808+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:02:46.860092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:46.860092+0000","last_clean_scrub_stamp":"2024-09-18T08:02:46.860092+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.e","version":"69'8","reported_seq":91,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-09-18T08:04:06.661709+0000","last_change":"2024-09-18T08:03:51.342453+0000","last_active":"2024-09-18T08:04:06.661709+0000","last_peered":"2024-09-18T08:04:06.661709+0000","last_clean":"2024-09-18T08:04:06.661709+0000","last_became_active":"2024-09-18T08:02:51.903369+0000","last_became_peered":"2024-09-18T08:02:51.903369+0000","last_unstale":"2024-09-18T08:04:06.661709+0000","last_undegraded":"2024-09-18T08:04:06.661709+0000","last_fullsized":"2024-09-18T08:04:06.661709+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'8","last_scrub_stamp":"2024-09-18T08:03:51.342402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:50.886830+0000","last_clean_scrub_stamp":"2024-09-18T08:03:51.342402+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:05:14.556564+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":55,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-09-18T08:04:06.665805+0000","last_change":"2024-09-18T08:02:53.910161+0000","last_active":"2024-09-18T08:04:06.665805+0000","last_peered":"2024-09-18T08:04:06.665805+0000","last_clean":"2024-09-18T08:04:06.665805+0000","last_became_active":"2024-09-18T08:02:53.909834+0000","last_became_peered":"2024-09-18T08:02:53.909834+0000","last_unstale":"2024-09-18T08:04:06.665805+0000","last_undegraded":"2024-09-18T08:04:06.665805+0000","last_fullsized":"2024-09-18T08:04:06.665805+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_clean_scrub_stamp":"2024-09-18T08:02:52.890475+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:04:07.435248+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.3","version":"89'413","reported_seq":436,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-09-18T08:04:08.078866+0000","last_change":"2024-09-18T08:03:57.462489+0000","last_active":"2024-09-18T08:04:08.078866+0000","last_peered":"2024-09-18T08:04:08.078866+0000","last_clean":"2024-09-18T08:04:08.078866+0000","last_became_active":"2024-09-18T08:03:57.462210+0000","last_became_peered":"2024-09-18T08:03:57.462210+0000","last_unstale":"2024-09-18T08:04:08.078866+0000","last_undegraded":"2024-09-18T08:04:08.078866+0000","last_fullsized":"2024-09-18T08:04:08.078866+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:03:56.449957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:03:56.449957+0000","last_clean_scrub_stamp":"2024-09-18T08:03:56.449957+0000","objects_scrubbed":0,"log_size":413,"log_dups_size":0,"ondisk_log_size":413,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:05:06.406380+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27066368,"num_objects":413,"num_object_clones":0,"num_object_copies":1239,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":413,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":413,"num_write_kb":26432,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,4],"acting":[3,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.0","version":"69'18","reported_seq":94,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-09-18T08:04:06.670912+0000","last_change":"2024-09-18T08:02:49.887135+0000","last_active":"2024-09-18T08:04:06.670912+0000","last_peered":"2024-09-18T08:04:06.670912+0000","last_clean":"2024-09-18T08:04:06.670912+0000","last_became_active":"2024-09-18T08:02:49.886818+0000","last_became_peered":"2024-09-18T08:02:49.886818+0000","last_unstale":"2024-09-18T08:04:06.670912+0000","last_undegraded":"2024-09-18T08:04:06.670912+0000","last_fullsized":"2024-09-18T08:04:06.670912+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_clean_scrub_stamp":"2024-09-18T08:02:48.870718+0000","objects_scrubbed":0,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":34,"num_read_kb":22,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":74,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-09-18T08:04:06.673050+0000","last_change":"2024-09-18T08:03:52.877506+0000","last_active":"2024-09-18T08:04:06.673050+0000","last_peered":"2024-09-18T08:04:06.673050+0000","last_clean":"2024-09-18T08:04:06.673050+0000","last_became_active":"2024-09-18T08:02:47.880225+0000","last_became_peered":"2024-09-18T08:02:47.880225+0000","last_unstale":"2024-09-18T08:04:06.673050+0000","last_undegraded":"2024-09-18T08:04:06.673050+0000","last_fullsized":"2024-09-18T08:04:06.673050+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:03:52.877449+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:46.860092+0000","last_clean_scrub_stamp":"2024-09-18T08:03:52.877449+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:05:16.125116+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":58,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-09-18T08:04:06.660831+0000","last_change":"2024-09-18T08:02:51.899911+0000","last_active":"2024-09-18T08:04:06.660831+0000","last_peered":"2024-09-18T08:04:06.660831+0000","last_clean":"2024-09-18T08:04:06.660831+0000","last_became_active":"2024-09-18T08:02:51.899594+0000","last_became_peered":"2024-09-18T08:02:51.899594+0000","last_unstale":"2024-09-18T08:04:06.660831+0000","last_undegraded":"2024-09-18T08:04:06.660831+0000","last_fullsized":"2024-09-18T08:04:06.660831+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:02:50.886830+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:50.886830+0000","last_clean_scrub_stamp":"2024-09-18T08:02:50.886830+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":55,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-09-18T08:04:06.660750+0000","last_change":"2024-09-18T08:02:53.904262+0000","last_active":"2024-09-18T08:04:06.660750+0000","last_peered":"2024-09-18T08:04:06.660750+0000","last_clean":"2024-09-18T08:04:06.660750+0000","last_became_active":"2024-09-18T08:02:53.903957+0000","last_became_peered":"2024-09-18T08:02:53.903957+0000","last_unstale":"2024-09-18T08:04:06.660750+0000","last_undegraded":"2024-09-18T08:04:06.660750+0000","last_fullsized":"2024-09-18T08:04:06.660750+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_clean_scrub_stamp":"2024-09-18T08:02:52.890475+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.2","version":"89'444","reported_seq":467,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-09-18T08:04:08.049394+0000","last_change":"2024-09-18T08:03:57.463750+0000","last_active":"2024-09-18T08:04:08.049394+0000","last_peered":"2024-09-18T08:04:08.049394+0000","last_clean":"2024-09-18T08:04:08.049394+0000","last_became_active":"2024-09-18T08:03:57.463459+0000","last_became_peered":"2024-09-18T08:03:57.463459+0000","last_unstale":"2024-09-18T08:04:08.049394+0000","last_undegraded":"2024-09-18T08:04:08.049394+0000","last_fullsized":"2024-09-18T08:04:08.049394+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:03:56.449957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:03:56.449957+0000","last_clean_scrub_stamp":"2024-09-18T08:03:56.449957+0000","objects_scrubbed":0,"log_size":444,"log_dups_size":0,"ondisk_log_size":444,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:05:23.542667+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29097984,"num_objects":444,"num_object_clones":0,"num_object_copies":1332,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":444,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":444,"num_write_kb":28416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"69'14","reported_seq":91,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-09-18T08:04:06.660459+0000","last_change":"2024-09-18T08:03:53.318484+0000","last_active":"2024-09-18T08:04:06.660459+0000","last_peered":"2024-09-18T08:04:06.660459+0000","last_clean":"2024-09-18T08:04:06.660459+0000","last_became_active":"2024-09-18T08:02:49.893830+0000","last_became_peered":"2024-09-18T08:02:49.893830+0000","last_unstale":"2024-09-18T08:04:06.660459+0000","last_undegraded":"2024-09-18T08:04:06.660459+0000","last_fullsized":"2024-09-18T08:04:06.660459+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'14","last_scrub_stamp":"2024-09-18T08:03:53.318377+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_clean_scrub_stamp":"2024-09-18T08:03:53.318377+0000","objects_scrubbed":7,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:04:56.314108+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":21,"num_read_kb":14,"num_write":14,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"62'2","reported_seq":73,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-09-18T08:04:06.670362+0000","last_change":"2024-09-18T08:02:47.873172+0000","last_active":"2024-09-18T08:04:06.670362+0000","last_peered":"2024-09-18T08:04:06.670362+0000","last_clean":"2024-09-18T08:04:06.670362+0000","last_became_active":"2024-09-18T08:02:47.872857+0000","last_became_peered":"2024-09-18T08:02:47.872857+0000","last_unstale":"2024-09-18T08:04:06.670362+0000","last_undegraded":"2024-09-18T08:04:06.670362+0000","last_fullsized":"2024-09-18T08:04:06.670362+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:02:46.860092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:46.860092+0000","last_clean_scrub_stamp":"2024-09-18T08:02:46.860092+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1039,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.0","version":"69'8","reported_seq":83,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-09-18T08:04:06.670788+0000","last_change":"2024-09-18T08:02:51.910339+0000","last_active":"2024-09-18T08:04:06.670788+0000","last_peered":"2024-09-18T08:04:06.670788+0000","last_clean":"2024-09-18T08:04:06.670788+0000","last_became_active":"2024-09-18T08:02:51.909680+0000","last_became_peered":"2024-09-18T08:02:51.909680+0000","last_unstale":"2024-09-18T08:04:06.670788+0000","last_undegraded":"2024-09-18T08:04:06.670788+0000","last_fullsized":"2024-09-18T08:04:06.670788+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:02:50.886830+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:50.886830+0000","last_clean_scrub_stamp":"2024-09-18T08:02:50.886830+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":55,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-09-18T08:04:06.665994+0000","last_change":"2024-09-18T08:02:54.047452+0000","last_active":"2024-09-18T08:04:06.665994+0000","last_peered":"2024-09-18T08:04:06.665994+0000","last_clean":"2024-09-18T08:04:06.665994+0000","last_became_active":"2024-09-18T08:02:54.047039+0000","last_became_peered":"2024-09-18T08:02:54.047039+0000","last_unstale":"2024-09-18T08:04:06.665994+0000","last_undegraded":"2024-09-18T08:04:06.665994+0000","last_fullsized":"2024-09-18T08:04:06.665994+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_clean_scrub_stamp":"2024-09-18T08:02:52.890475+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:05:02.687660+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.1","version":"89'405","reported_seq":427,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-09-18T08:04:08.071172+0000","last_change":"2024-09-18T08:03:57.465538+0000","last_active":"2024-09-18T08:04:08.071172+0000","last_peered":"2024-09-18T08:04:08.071172+0000","last_clean":"2024-09-18T08:04:08.071172+0000","last_became_active":"2024-09-18T08:03:57.465253+0000","last_became_peered":"2024-09-18T08:03:57.465253+0000","last_unstale":"2024-09-18T08:04:08.071172+0000","last_undegraded":"2024-09-18T08:04:08.071172+0000","last_fullsized":"2024-09-18T08:04:08.071172+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:03:56.449957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:03:56.449957+0000","last_clean_scrub_stamp":"2024-09-18T08:03:56.449957+0000","objects_scrubbed":0,"log_size":405,"log_dups_size":0,"ondisk_log_size":405,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:05:09.913473+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26542080,"num_objects":405,"num_object_clones":0,"num_object_copies":1215,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":405,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":405,"num_write_kb":25920,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.2","version":"69'10","reported_seq":77,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-09-18T08:04:06.672373+0000","last_change":"2024-09-18T08:02:49.896889+0000","last_active":"2024-09-18T08:04:06.672373+0000","last_peered":"2024-09-18T08:04:06.672373+0000","last_clean":"2024-09-18T08:04:06.672373+0000","last_became_active":"2024-09-18T08:02:49.895265+0000","last_became_peered":"2024-09-18T08:02:49.895265+0000","last_unstale":"2024-09-18T08:04:06.672373+0000","last_undegraded":"2024-09-18T08:04:06.672373+0000","last_fullsized":"2024-09-18T08:04:06.672373+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_clean_scrub_stamp":"2024-09-18T08:02:48.870718+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":74,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-09-18T08:04:06.666246+0000","last_change":"2024-09-18T08:03:48.009689+0000","last_active":"2024-09-18T08:04:06.666246+0000","last_peered":"2024-09-18T08:04:06.666246+0000","last_clean":"2024-09-18T08:04:06.666246+0000","last_became_active":"2024-09-18T08:02:47.874727+0000","last_became_peered":"2024-09-18T08:02:47.874727+0000","last_unstale":"2024-09-18T08:04:06.666246+0000","last_undegraded":"2024-09-18T08:04:06.666246+0000","last_fullsized":"2024-09-18T08:04:06.666246+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:03:48.009611+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:46.860092+0000","last_clean_scrub_stamp":"2024-09-18T08:03:48.009611+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:05:16.091115+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.3","version":"69'8","reported_seq":83,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-09-18T08:04:06.665938+0000","last_change":"2024-09-18T08:02:51.906369+0000","last_active":"2024-09-18T08:04:06.665938+0000","last_peered":"2024-09-18T08:04:06.665938+0000","last_clean":"2024-09-18T08:04:06.665938+0000","last_became_active":"2024-09-18T08:02:51.906055+0000","last_became_peered":"2024-09-18T08:02:51.906055+0000","last_unstale":"2024-09-18T08:04:06.665938+0000","last_undegraded":"2024-09-18T08:04:06.665938+0000","last_fullsized":"2024-09-18T08:04:06.665938+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:02:50.886830+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:50.886830+0000","last_clean_scrub_stamp":"2024-09-18T08:02:50.886830+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:04:18.452195+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":55,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-09-18T08:04:06.665512+0000","last_change":"2024-09-18T08:02:53.910057+0000","last_active":"2024-09-18T08:04:06.665512+0000","last_peered":"2024-09-18T08:04:06.665512+0000","last_clean":"2024-09-18T08:04:06.665512+0000","last_became_active":"2024-09-18T08:02:53.909583+0000","last_became_peered":"2024-09-18T08:02:53.909583+0000","last_unstale":"2024-09-18T08:04:06.665512+0000","last_undegraded":"2024-09-18T08:04:06.665512+0000","last_fullsized":"2024-09-18T08:04:06.665512+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_clean_scrub_stamp":"2024-09-18T08:02:52.890475+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:04:07.545769+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.0","version":"89'418","reported_seq":441,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-09-18T08:04:08.069824+0000","last_change":"2024-09-18T08:03:57.469331+0000","last_active":"2024-09-18T08:04:08.069824+0000","last_peered":"2024-09-18T08:04:08.069824+0000","last_clean":"2024-09-18T08:04:08.069824+0000","last_became_active":"2024-09-18T08:03:57.468985+0000","last_became_peered":"2024-09-18T08:03:57.468985+0000","last_unstale":"2024-09-18T08:04:08.069824+0000","last_undegraded":"2024-09-18T08:04:08.069824+0000","last_fullsized":"2024-09-18T08:04:08.069824+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:03:56.449957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:03:56.449957+0000","last_clean_scrub_stamp":"2024-09-18T08:03:56.449957+0000","objects_scrubbed":0,"log_size":418,"log_dups_size":0,"ondisk_log_size":418,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:04:58.317755+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27394048,"num_objects":418,"num_object_clones":0,"num_object_copies":1254,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":418,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":418,"num_write_kb":26752,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.3","version":"69'19","reported_seq":106,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-09-18T08:04:07.231406+0000","last_change":"2024-09-18T08:04:07.231406+0000","last_active":"2024-09-18T08:04:07.231406+0000","last_peered":"2024-09-18T08:04:07.231406+0000","last_clean":"2024-09-18T08:04:07.231406+0000","last_became_active":"2024-09-18T08:02:49.890318+0000","last_became_peered":"2024-09-18T08:02:49.890318+0000","last_unstale":"2024-09-18T08:04:07.231406+0000","last_undegraded":"2024-09-18T08:04:07.231406+0000","last_fullsized":"2024-09-18T08:04:07.231406+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'19","last_scrub_stamp":"2024-09-18T08:04:07.229846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_clean_scrub_stamp":"2024-09-18T08:04:07.229846+0000","objects_scrubbed":8,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:05:14.330580+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":39,"num_read_kb":25,"num_write":22,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.4","version":"62'1","reported_seq":72,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-09-18T08:04:06.669508+0000","last_change":"2024-09-18T08:02:47.875046+0000","last_active":"2024-09-18T08:04:06.669508+0000","last_peered":"2024-09-18T08:04:06.669508+0000","last_clean":"2024-09-18T08:04:06.669508+0000","last_became_active":"2024-09-18T08:02:47.874764+0000","last_became_peered":"2024-09-18T08:02:47.874764+0000","last_unstale":"2024-09-18T08:04:06.669508+0000","last_undegraded":"2024-09-18T08:04:06.669508+0000","last_fullsized":"2024-09-18T08:04:06.669508+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:02:46.860092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:46.860092+0000","last_clean_scrub_stamp":"2024-09-18T08:02:46.860092+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:04:07.354571+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":436,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":58,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-09-18T08:04:06.663465+0000","last_change":"2024-09-18T08:02:51.908344+0000","last_active":"2024-09-18T08:04:06.663465+0000","last_peered":"2024-09-18T08:04:06.663465+0000","last_clean":"2024-09-18T08:04:06.663465+0000","last_became_active":"2024-09-18T08:02:51.908068+0000","last_became_peered":"2024-09-18T08:02:51.908068+0000","last_unstale":"2024-09-18T08:04:06.663465+0000","last_undegraded":"2024-09-18T08:04:06.663465+0000","last_fullsized":"2024-09-18T08:04:06.663465+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:02:50.886830+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:50.886830+0000","last_clean_scrub_stamp":"2024-09-18T08:02:50.886830+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":54,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-09-18T08:04:06.668082+0000","last_change":"2024-09-18T08:02:54.046350+0000","last_active":"2024-09-18T08:04:06.668082+0000","last_peered":"2024-09-18T08:04:06.668082+0000","last_clean":"2024-09-18T08:04:06.668082+0000","last_became_active":"2024-09-18T08:02:54.045726+0000","last_became_peered":"2024-09-18T08:02:54.045726+0000","last_unstale":"2024-09-18T08:04:06.668082+0000","last_undegraded":"2024-09-18T08:04:06.668082+0000","last_fullsized":"2024-09-18T08:04:06.668082+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_clean_scrub_stamp":"2024-09-18T08:02:52.890475+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:04:55.706372+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.7","version":"89'416","reported_seq":439,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-09-18T08:04:08.070295+0000","last_change":"2024-09-18T08:03:57.469246+0000","last_active":"2024-09-18T08:04:08.070295+0000","last_peered":"2024-09-18T08:04:08.070295+0000","last_clean":"2024-09-18T08:04:08.070295+0000","last_became_active":"2024-09-18T08:03:57.468813+0000","last_became_peered":"2024-09-18T08:03:57.468813+0000","last_unstale":"2024-09-18T08:04:08.070295+0000","last_undegraded":"2024-09-18T08:04:08.070295+0000","last_fullsized":"2024-09-18T08:04:08.070295+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:03:56.449957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:03:56.449957+0000","last_clean_scrub_stamp":"2024-09-18T08:03:56.449957+0000","objects_scrubbed":0,"log_size":416,"log_dups_size":0,"ondisk_log_size":416,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:05:14.084455+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27262976,"num_objects":416,"num_object_clones":0,"num_object_copies":1248,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":416,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":416,"num_write_kb":26624,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,3],"acting":[6,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.4","version":"69'28","reported_seq":127,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-09-18T08:04:06.669351+0000","last_change":"2024-09-18T08:02:49.897515+0000","last_active":"2024-09-18T08:04:06.669351+0000","last_peered":"2024-09-18T08:04:06.669351+0000","last_clean":"2024-09-18T08:04:06.669351+0000","last_became_active":"2024-09-18T08:02:49.897221+0000","last_became_peered":"2024-09-18T08:02:49.897221+0000","last_unstale":"2024-09-18T08:04:06.669351+0000","last_undegraded":"2024-09-18T08:04:06.669351+0000","last_fullsized":"2024-09-18T08:04:06.669351+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_clean_scrub_stamp":"2024-09-18T08:02:48.870718+0000","objects_scrubbed":0,"log_size":28,"log_dups_size":0,"ondisk_log_size":28,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":48,"num_read_kb":33,"num_write":26,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":66,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-09-18T08:04:06.661637+0000","last_change":"2024-09-18T08:02:47.877445+0000","last_active":"2024-09-18T08:04:06.661637+0000","last_peered":"2024-09-18T08:04:06.661637+0000","last_clean":"2024-09-18T08:04:06.661637+0000","last_became_active":"2024-09-18T08:02:47.877085+0000","last_became_peered":"2024-09-18T08:02:47.877085+0000","last_unstale":"2024-09-18T08:04:06.661637+0000","last_undegraded":"2024-09-18T08:04:06.661637+0000","last_fullsized":"2024-09-18T08:04:06.661637+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:02:46.860092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:46.860092+0000","last_clean_scrub_stamp":"2024-09-18T08:02:46.860092+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:04:09.365906+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"64'2","reported_seq":76,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-09-18T08:04:06.665752+0000","last_change":"2024-09-18T08:02:49.883480+0000","last_active":"2024-09-18T08:04:06.665752+0000","last_peered":"2024-09-18T08:04:06.665752+0000","last_clean":"2024-09-18T08:04:06.665752+0000","last_became_active":"2024-09-18T08:02:46.867802+0000","last_became_peered":"2024-09-18T08:02:46.867802+0000","last_unstale":"2024-09-18T08:04:06.665752+0000","last_undegraded":"2024-09-18T08:04:06.665752+0000","last_fullsized":"2024-09-18T08:04:06.665752+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:02:45.855979+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:45.855979+0000","last_clean_scrub_stamp":"2024-09-18T08:02:45.855979+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:04:13.406442+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001348349,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":58,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-09-18T08:04:06.666090+0000","last_change":"2024-09-18T08:02:51.905694+0000","last_active":"2024-09-18T08:04:06.666090+0000","last_peered":"2024-09-18T08:04:06.666090+0000","last_clean":"2024-09-18T08:04:06.666090+0000","last_became_active":"2024-09-18T08:02:51.905357+0000","last_became_peered":"2024-09-18T08:02:51.905357+0000","last_unstale":"2024-09-18T08:04:06.666090+0000","last_undegraded":"2024-09-18T08:04:06.666090+0000","last_fullsized":"2024-09-18T08:04:06.666090+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:02:50.886830+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:50.886830+0000","last_clean_scrub_stamp":"2024-09-18T08:02:50.886830+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:04:15.739565+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":55,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-09-18T08:04:06.671912+0000","last_change":"2024-09-18T08:02:53.911634+0000","last_active":"2024-09-18T08:04:06.671912+0000","last_peered":"2024-09-18T08:04:06.671912+0000","last_clean":"2024-09-18T08:04:06.671912+0000","last_became_active":"2024-09-18T08:02:53.911036+0000","last_became_peered":"2024-09-18T08:02:53.911036+0000","last_unstale":"2024-09-18T08:04:06.671912+0000","last_undegraded":"2024-09-18T08:04:06.671912+0000","last_fullsized":"2024-09-18T08:04:06.671912+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_clean_scrub_stamp":"2024-09-18T08:02:52.890475+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:04:16.930353+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.4","version":"89'446","reported_seq":469,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-09-18T08:04:08.069615+0000","last_change":"2024-09-18T08:03:57.466551+0000","last_active":"2024-09-18T08:04:08.069615+0000","last_peered":"2024-09-18T08:04:08.069615+0000","last_clean":"2024-09-18T08:04:08.069615+0000","last_became_active":"2024-09-18T08:03:57.466232+0000","last_became_peered":"2024-09-18T08:03:57.466232+0000","last_unstale":"2024-09-18T08:04:08.069615+0000","last_undegraded":"2024-09-18T08:04:08.069615+0000","last_fullsized":"2024-09-18T08:04:08.069615+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:03:56.449957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:03:56.449957+0000","last_clean_scrub_stamp":"2024-09-18T08:03:56.449957+0000","objects_scrubbed":0,"log_size":446,"log_dups_size":0,"ondisk_log_size":446,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:05:22.600013+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29229056,"num_objects":446,"num_object_clones":0,"num_object_copies":1338,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":446,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":446,"num_write_kb":28544,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,1],"acting":[0,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.7","version":"69'13","reported_seq":97,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-09-18T08:04:06.671749+0000","last_change":"2024-09-18T08:03:50.700884+0000","last_active":"2024-09-18T08:04:06.671749+0000","last_peered":"2024-09-18T08:04:06.671749+0000","last_clean":"2024-09-18T08:04:06.671749+0000","last_became_active":"2024-09-18T08:02:49.898987+0000","last_became_peered":"2024-09-18T08:02:49.898987+0000","last_unstale":"2024-09-18T08:04:06.671749+0000","last_undegraded":"2024-09-18T08:04:06.671749+0000","last_fullsized":"2024-09-18T08:04:06.671749+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'13","last_scrub_stamp":"2024-09-18T08:03:50.700818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_clean_scrub_stamp":"2024-09-18T08:03:50.700818+0000","objects_scrubbed":5,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:05:02.612542+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":19,"num_write":16,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":74,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-09-18T08:04:06.668910+0000","last_change":"2024-09-18T08:03:49.704071+0000","last_active":"2024-09-18T08:04:06.668910+0000","last_peered":"2024-09-18T08:04:06.668910+0000","last_clean":"2024-09-18T08:04:06.668910+0000","last_became_active":"2024-09-18T08:02:47.871416+0000","last_became_peered":"2024-09-18T08:02:47.871416+0000","last_unstale":"2024-09-18T08:04:06.668910+0000","last_undegraded":"2024-09-18T08:04:06.668910+0000","last_fullsized":"2024-09-18T08:04:06.668910+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:03:49.704004+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:46.860092+0000","last_clean_scrub_stamp":"2024-09-18T08:03:49.704004+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:04:56.429029+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"62'1","reported_seq":75,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-09-18T08:04:06.663432+0000","last_change":"2024-09-18T08:02:49.877280+0000","last_active":"2024-09-18T08:04:06.663432+0000","last_peered":"2024-09-18T08:04:06.663432+0000","last_clean":"2024-09-18T08:04:06.663432+0000","last_became_active":"2024-09-18T08:02:46.868175+0000","last_became_peered":"2024-09-18T08:02:46.868175+0000","last_unstale":"2024-09-18T08:04:06.663432+0000","last_undegraded":"2024-09-18T08:04:06.663432+0000","last_fullsized":"2024-09-18T08:04:06.663432+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:02:45.855979+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:45.855979+0000","last_clean_scrub_stamp":"2024-09-18T08:02:45.855979+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00066785499999999997,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":58,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-09-18T08:04:06.664021+0000","last_change":"2024-09-18T08:02:51.908417+0000","last_active":"2024-09-18T08:04:06.664021+0000","last_peered":"2024-09-18T08:04:06.664021+0000","last_clean":"2024-09-18T08:04:06.664021+0000","last_became_active":"2024-09-18T08:02:51.908156+0000","last_became_peered":"2024-09-18T08:02:51.908156+0000","last_unstale":"2024-09-18T08:04:06.664021+0000","last_undegraded":"2024-09-18T08:04:06.664021+0000","last_fullsized":"2024-09-18T08:04:06.664021+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:02:50.886830+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:50.886830+0000","last_clean_scrub_stamp":"2024-09-18T08:02:50.886830+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":55,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-09-18T08:04:06.665866+0000","last_change":"2024-09-18T08:02:54.048417+0000","last_active":"2024-09-18T08:04:06.665866+0000","last_peered":"2024-09-18T08:04:06.665866+0000","last_clean":"2024-09-18T08:04:06.665866+0000","last_became_active":"2024-09-18T08:02:54.048004+0000","last_became_peered":"2024-09-18T08:02:54.048004+0000","last_unstale":"2024-09-18T08:04:06.665866+0000","last_undegraded":"2024-09-18T08:04:06.665866+0000","last_fullsized":"2024-09-18T08:04:06.665866+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_clean_scrub_stamp":"2024-09-18T08:02:52.890475+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:04:20.672566+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.5","version":"89'435","reported_seq":457,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-09-18T08:04:08.068407+0000","last_change":"2024-09-18T08:03:57.469151+0000","last_active":"2024-09-18T08:04:08.068407+0000","last_peered":"2024-09-18T08:04:08.068407+0000","last_clean":"2024-09-18T08:04:08.068407+0000","last_became_active":"2024-09-18T08:03:57.468853+0000","last_became_peered":"2024-09-18T08:03:57.468853+0000","last_unstale":"2024-09-18T08:04:08.068407+0000","last_undegraded":"2024-09-18T08:04:08.068407+0000","last_fullsized":"2024-09-18T08:04:08.068407+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:03:56.449957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:03:56.449957+0000","last_clean_scrub_stamp":"2024-09-18T08:03:56.449957+0000","objects_scrubbed":0,"log_size":435,"log_dups_size":0,"ondisk_log_size":435,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:05:16.916592+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28508160,"num_objects":435,"num_object_clones":0,"num_object_copies":1305,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":435,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":435,"num_write_kb":27840,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,0],"acting":[7,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"69'12","reported_seq":80,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-09-18T08:04:06.670071+0000","last_change":"2024-09-18T08:02:49.898383+0000","last_active":"2024-09-18T08:04:06.670071+0000","last_peered":"2024-09-18T08:04:06.670071+0000","last_clean":"2024-09-18T08:04:06.670071+0000","last_became_active":"2024-09-18T08:02:49.898079+0000","last_became_peered":"2024-09-18T08:02:49.898079+0000","last_unstale":"2024-09-18T08:04:06.670071+0000","last_undegraded":"2024-09-18T08:04:06.670071+0000","last_fullsized":"2024-09-18T08:04:06.670071+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_clean_scrub_stamp":"2024-09-18T08:02:48.870718+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:04:14.339135+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":18,"num_read_kb":12,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":66,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-09-18T08:04:06.667920+0000","last_change":"2024-09-18T08:02:47.874335+0000","last_active":"2024-09-18T08:04:06.667920+0000","last_peered":"2024-09-18T08:04:06.667920+0000","last_clean":"2024-09-18T08:04:06.667920+0000","last_became_active":"2024-09-18T08:02:47.873809+0000","last_became_peered":"2024-09-18T08:02:47.873809+0000","last_unstale":"2024-09-18T08:04:06.667920+0000","last_undegraded":"2024-09-18T08:04:06.667920+0000","last_fullsized":"2024-09-18T08:04:06.667920+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:02:46.860092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:46.860092+0000","last_clean_scrub_stamp":"2024-09-18T08:02:46.860092+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:04:57.127057+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"70'5","reported_seq":160,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-09-18T08:04:06.814704+0000","last_change":"2024-09-18T08:04:06.814615+0000","last_active":"2024-09-18T08:04:06.814704+0000","last_peered":"2024-09-18T08:04:06.814704+0000","last_clean":"2024-09-18T08:04:06.814704+0000","last_became_active":"2024-09-18T08:02:46.872144+0000","last_became_peered":"2024-09-18T08:02:46.872144+0000","last_unstale":"2024-09-18T08:04:06.814704+0000","last_undegraded":"2024-09-18T08:04:06.814704+0000","last_fullsized":"2024-09-18T08:04:06.814704+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"70'5","last_scrub_stamp":"2024-09-18T08:04:06.814540+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:45.855979+0000","last_clean_scrub_stamp":"2024-09-18T08:04:06.814540+0000","objects_scrubbed":1,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:05:23.913754+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.000504055,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":75,"num_read_kb":70,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":58,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-09-18T08:04:06.666470+0000","last_change":"2024-09-18T08:02:51.913423+0000","last_active":"2024-09-18T08:04:06.666470+0000","last_peered":"2024-09-18T08:04:06.666470+0000","last_clean":"2024-09-18T08:04:06.666470+0000","last_became_active":"2024-09-18T08:02:51.913112+0000","last_became_peered":"2024-09-18T08:02:51.913112+0000","last_unstale":"2024-09-18T08:04:06.666470+0000","last_undegraded":"2024-09-18T08:04:06.666470+0000","last_fullsized":"2024-09-18T08:04:06.666470+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:02:50.886830+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:50.886830+0000","last_clean_scrub_stamp":"2024-09-18T08:02:50.886830+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":55,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-09-18T08:04:06.669015+0000","last_change":"2024-09-18T08:02:53.907689+0000","last_active":"2024-09-18T08:04:06.669015+0000","last_peered":"2024-09-18T08:04:06.669015+0000","last_clean":"2024-09-18T08:04:06.669015+0000","last_became_active":"2024-09-18T08:02:53.907404+0000","last_became_peered":"2024-09-18T08:02:53.907404+0000","last_unstale":"2024-09-18T08:04:06.669015+0000","last_undegraded":"2024-09-18T08:04:06.669015+0000","last_fullsized":"2024-09-18T08:04:06.669015+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_clean_scrub_stamp":"2024-09-18T08:02:52.890475+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:05:00.697155+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.6","version":"89'385","reported_seq":408,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-09-18T08:04:08.057666+0000","last_change":"2024-09-18T08:03:57.466759+0000","last_active":"2024-09-18T08:04:08.057666+0000","last_peered":"2024-09-18T08:04:08.057666+0000","last_clean":"2024-09-18T08:04:08.057666+0000","last_became_active":"2024-09-18T08:03:57.466391+0000","last_became_peered":"2024-09-18T08:03:57.466391+0000","last_unstale":"2024-09-18T08:04:08.057666+0000","last_undegraded":"2024-09-18T08:04:08.057666+0000","last_fullsized":"2024-09-18T08:04:08.057666+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:03:56.449957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:03:56.449957+0000","last_clean_scrub_stamp":"2024-09-18T08:03:56.449957+0000","objects_scrubbed":0,"log_size":385,"log_dups_size":0,"ondisk_log_size":385,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:05:03.397982+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25231360,"num_objects":385,"num_object_clones":0,"num_object_copies":1155,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":385,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":385,"num_write_kb":24640,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.5","version":"69'16","reported_seq":101,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-09-18T08:04:06.664163+0000","last_change":"2024-09-18T08:02:49.892366+0000","last_active":"2024-09-18T08:04:06.664163+0000","last_peered":"2024-09-18T08:04:06.664163+0000","last_clean":"2024-09-18T08:04:06.664163+0000","last_became_active":"2024-09-18T08:02:49.891985+0000","last_became_peered":"2024-09-18T08:02:49.891985+0000","last_unstale":"2024-09-18T08:04:06.664163+0000","last_undegraded":"2024-09-18T08:04:06.664163+0000","last_fullsized":"2024-09-18T08:04:06.664163+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_clean_scrub_stamp":"2024-09-18T08:02:48.870718+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:04:12.297325+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":25,"num_read_kb":15,"num_write":13,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":66,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-09-18T08:04:06.674804+0000","last_change":"2024-09-18T08:02:47.876225+0000","last_active":"2024-09-18T08:04:06.674804+0000","last_peered":"2024-09-18T08:04:06.674804+0000","last_clean":"2024-09-18T08:04:06.674804+0000","last_became_active":"2024-09-18T08:02:47.875950+0000","last_became_peered":"2024-09-18T08:02:47.875950+0000","last_unstale":"2024-09-18T08:04:06.674804+0000","last_undegraded":"2024-09-18T08:04:06.674804+0000","last_fullsized":"2024-09-18T08:04:06.674804+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:02:46.860092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:46.860092+0000","last_clean_scrub_stamp":"2024-09-18T08:02:46.860092+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"74'79","reported_seq":112,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-09-18T08:04:06.665752+0000","last_change":"2024-09-18T08:02:55.472070+0000","last_active":"2024-09-18T08:04:06.665752+0000","last_peered":"2024-09-18T08:04:06.665752+0000","last_clean":"2024-09-18T08:04:06.665752+0000","last_became_active":"2024-09-18T08:02:39.446743+0000","last_became_peered":"2024-09-18T08:02:39.446743+0000","last_unstale":"2024-09-18T08:04:06.665752+0000","last_undegraded":"2024-09-18T08:04:06.665752+0000","last_fullsized":"2024-09-18T08:04:06.665752+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"20'72","last_scrub_stamp":"2024-09-18T08:02:55.471998+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:00:11.893742+0000","last_clean_scrub_stamp":"2024-09-18T08:02:55.471998+0000","objects_scrubbed":2,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:04:21.599062+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":231,"num_write":113,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":590368,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":58,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-09-18T08:04:06.666418+0000","last_change":"2024-09-18T08:02:51.905819+0000","last_active":"2024-09-18T08:04:06.666418+0000","last_peered":"2024-09-18T08:04:06.666418+0000","last_clean":"2024-09-18T08:04:06.666418+0000","last_became_active":"2024-09-18T08:02:51.905516+0000","last_became_peered":"2024-09-18T08:02:51.905516+0000","last_unstale":"2024-09-18T08:04:06.666418+0000","last_undegraded":"2024-09-18T08:04:06.666418+0000","last_fullsized":"2024-09-18T08:04:06.666418+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:02:50.886830+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:50.886830+0000","last_clean_scrub_stamp":"2024-09-18T08:02:50.886830+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":55,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-09-18T08:04:06.666332+0000","last_change":"2024-09-18T08:02:53.909023+0000","last_active":"2024-09-18T08:04:06.666332+0000","last_peered":"2024-09-18T08:04:06.666332+0000","last_clean":"2024-09-18T08:04:06.666332+0000","last_became_active":"2024-09-18T08:02:53.908668+0000","last_became_peered":"2024-09-18T08:02:53.908668+0000","last_unstale":"2024-09-18T08:04:06.666332+0000","last_undegraded":"2024-09-18T08:04:06.666332+0000","last_fullsized":"2024-09-18T08:04:06.666332+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_clean_scrub_stamp":"2024-09-18T08:02:52.890475+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.d","version":"89'445","reported_seq":468,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-09-18T08:04:08.050315+0000","last_change":"2024-09-18T08:03:57.466705+0000","last_active":"2024-09-18T08:04:08.050315+0000","last_peered":"2024-09-18T08:04:08.050315+0000","last_clean":"2024-09-18T08:04:08.050315+0000","last_became_active":"2024-09-18T08:03:57.466389+0000","last_became_peered":"2024-09-18T08:03:57.466389+0000","last_unstale":"2024-09-18T08:04:08.050315+0000","last_undegraded":"2024-09-18T08:04:08.050315+0000","last_fullsized":"2024-09-18T08:04:08.050315+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:03:56.449957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:03:56.449957+0000","last_clean_scrub_stamp":"2024-09-18T08:03:56.449957+0000","objects_scrubbed":0,"log_size":445,"log_dups_size":0,"ondisk_log_size":445,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:05:23.131094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29163520,"num_objects":445,"num_object_clones":0,"num_object_copies":1335,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":445,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":445,"num_write_kb":28480,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,0],"acting":[6,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.e","version":"69'11","reported_seq":81,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-09-18T08:04:06.661767+0000","last_change":"2024-09-18T08:02:49.895676+0000","last_active":"2024-09-18T08:04:06.661767+0000","last_peered":"2024-09-18T08:04:06.661767+0000","last_clean":"2024-09-18T08:04:06.661767+0000","last_became_active":"2024-09-18T08:02:49.895303+0000","last_became_peered":"2024-09-18T08:02:49.895303+0000","last_unstale":"2024-09-18T08:04:06.661767+0000","last_undegraded":"2024-09-18T08:04:06.661767+0000","last_fullsized":"2024-09-18T08:04:06.661767+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_clean_scrub_stamp":"2024-09-18T08:02:48.870718+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:04:11.479080+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":66,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-09-18T08:04:06.660935+0000","last_change":"2024-09-18T08:02:47.880265+0000","last_active":"2024-09-18T08:04:06.660935+0000","last_peered":"2024-09-18T08:04:06.660935+0000","last_clean":"2024-09-18T08:04:06.660935+0000","last_became_active":"2024-09-18T08:02:47.879764+0000","last_became_peered":"2024-09-18T08:02:47.879764+0000","last_unstale":"2024-09-18T08:04:06.660935+0000","last_undegraded":"2024-09-18T08:04:06.660935+0000","last_fullsized":"2024-09-18T08:04:06.660935+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:02:46.860092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:46.860092+0000","last_clean_scrub_stamp":"2024-09-18T08:02:46.860092+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:04:12.919106+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":58,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-09-18T08:04:06.666187+0000","last_change":"2024-09-18T08:02:51.911107+0000","last_active":"2024-09-18T08:04:06.666187+0000","last_peered":"2024-09-18T08:04:06.666187+0000","last_clean":"2024-09-18T08:04:06.666187+0000","last_became_active":"2024-09-18T08:02:51.910649+0000","last_became_peered":"2024-09-18T08:02:51.910649+0000","last_unstale":"2024-09-18T08:04:06.666187+0000","last_undegraded":"2024-09-18T08:04:06.666187+0000","last_fullsized":"2024-09-18T08:04:06.666187+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:02:50.886830+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:50.886830+0000","last_clean_scrub_stamp":"2024-09-18T08:02:50.886830+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:05:04.882890+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":54,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-09-18T08:04:06.671069+0000","last_change":"2024-09-18T08:02:54.045510+0000","last_active":"2024-09-18T08:04:06.671069+0000","last_peered":"2024-09-18T08:04:06.671069+0000","last_clean":"2024-09-18T08:04:06.671069+0000","last_became_active":"2024-09-18T08:02:54.045091+0000","last_became_peered":"2024-09-18T08:02:54.045091+0000","last_unstale":"2024-09-18T08:04:06.671069+0000","last_undegraded":"2024-09-18T08:04:06.671069+0000","last_fullsized":"2024-09-18T08:04:06.671069+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_clean_scrub_stamp":"2024-09-18T08:02:52.890475+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:04:22.368585+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.e","version":"89'440","reported_seq":463,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-09-18T08:04:08.073491+0000","last_change":"2024-09-18T08:03:57.464667+0000","last_active":"2024-09-18T08:04:08.073491+0000","last_peered":"2024-09-18T08:04:08.073491+0000","last_clean":"2024-09-18T08:04:08.073491+0000","last_became_active":"2024-09-18T08:03:57.464311+0000","last_became_peered":"2024-09-18T08:03:57.464311+0000","last_unstale":"2024-09-18T08:04:08.073491+0000","last_undegraded":"2024-09-18T08:04:08.073491+0000","last_fullsized":"2024-09-18T08:04:08.073491+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:03:56.449957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:03:56.449957+0000","last_clean_scrub_stamp":"2024-09-18T08:03:56.449957+0000","objects_scrubbed":0,"log_size":440,"log_dups_size":0,"ondisk_log_size":440,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:05:01.893934+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28835840,"num_objects":440,"num_object_clones":0,"num_object_copies":1320,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":440,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":440,"num_write_kb":28160,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,0],"acting":[2,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.d","version":"69'17","reported_seq":90,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-09-18T08:04:06.661630+0000","last_change":"2024-09-18T08:02:49.895971+0000","last_active":"2024-09-18T08:04:06.661630+0000","last_peered":"2024-09-18T08:04:06.661630+0000","last_clean":"2024-09-18T08:04:06.661630+0000","last_became_active":"2024-09-18T08:02:49.895578+0000","last_became_peered":"2024-09-18T08:02:49.895578+0000","last_unstale":"2024-09-18T08:04:06.661630+0000","last_undegraded":"2024-09-18T08:04:06.661630+0000","last_fullsized":"2024-09-18T08:04:06.661630+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_clean_scrub_stamp":"2024-09-18T08:02:48.870718+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:04:17.784735+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":29,"num_read_kb":19,"num_write":18,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,1],"acting":[4,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":66,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-09-18T08:04:06.663959+0000","last_change":"2024-09-18T08:02:47.884431+0000","last_active":"2024-09-18T08:04:06.663959+0000","last_peered":"2024-09-18T08:04:06.663959+0000","last_clean":"2024-09-18T08:04:06.663959+0000","last_became_active":"2024-09-18T08:02:47.884132+0000","last_became_peered":"2024-09-18T08:02:47.884132+0000","last_unstale":"2024-09-18T08:04:06.663959+0000","last_undegraded":"2024-09-18T08:04:06.663959+0000","last_fullsized":"2024-09-18T08:04:06.663959+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:02:46.860092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:46.860092+0000","last_clean_scrub_stamp":"2024-09-18T08:02:46.860092+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":58,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-09-18T08:04:06.669109+0000","last_change":"2024-09-18T08:02:51.907544+0000","last_active":"2024-09-18T08:04:06.669109+0000","last_peered":"2024-09-18T08:04:06.669109+0000","last_clean":"2024-09-18T08:04:06.669109+0000","last_became_active":"2024-09-18T08:02:51.907196+0000","last_became_peered":"2024-09-18T08:02:51.907196+0000","last_unstale":"2024-09-18T08:04:06.669109+0000","last_undegraded":"2024-09-18T08:04:06.669109+0000","last_fullsized":"2024-09-18T08:04:06.669109+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:02:50.886830+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:50.886830+0000","last_clean_scrub_stamp":"2024-09-18T08:02:50.886830+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":63,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-09-18T08:04:06.709703+0000","last_change":"2024-09-18T08:04:06.709703+0000","last_active":"2024-09-18T08:04:06.709703+0000","last_peered":"2024-09-18T08:04:06.709703+0000","last_clean":"2024-09-18T08:04:06.709703+0000","last_became_active":"2024-09-18T08:02:53.904144+0000","last_became_peered":"2024-09-18T08:02:53.904144+0000","last_unstale":"2024-09-18T08:04:06.709703+0000","last_undegraded":"2024-09-18T08:04:06.709703+0000","last_fullsized":"2024-09-18T08:04:06.709703+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:04:06.709631+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_clean_scrub_stamp":"2024-09-18T08:04:06.709631+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:05:06.970875+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.f","version":"89'451","reported_seq":472,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-09-18T08:04:08.073125+0000","last_change":"2024-09-18T08:03:57.468272+0000","last_active":"2024-09-18T08:04:08.073125+0000","last_peered":"2024-09-18T08:04:08.073125+0000","last_clean":"2024-09-18T08:04:08.073125+0000","last_became_active":"2024-09-18T08:03:57.467980+0000","last_became_peered":"2024-09-18T08:03:57.467980+0000","last_unstale":"2024-09-18T08:04:08.073125+0000","last_undegraded":"2024-09-18T08:04:08.073125+0000","last_fullsized":"2024-09-18T08:04:08.073125+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:03:56.449957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:03:56.449957+0000","last_clean_scrub_stamp":"2024-09-18T08:03:56.449957+0000","objects_scrubbed":0,"log_size":451,"log_dups_size":0,"ondisk_log_size":451,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:04:58.696232+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29556736,"num_objects":451,"num_object_clones":0,"num_object_copies":1353,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":451,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":451,"num_write_kb":28864,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.c","version":"69'10","reported_seq":77,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-09-18T08:04:06.661373+0000","last_change":"2024-09-18T08:02:49.892847+0000","last_active":"2024-09-18T08:04:06.661373+0000","last_peered":"2024-09-18T08:04:06.661373+0000","last_clean":"2024-09-18T08:04:06.661373+0000","last_became_active":"2024-09-18T08:02:49.892527+0000","last_became_peered":"2024-09-18T08:02:49.892527+0000","last_unstale":"2024-09-18T08:04:06.661373+0000","last_undegraded":"2024-09-18T08:04:06.661373+0000","last_fullsized":"2024-09-18T08:04:06.661373+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_clean_scrub_stamp":"2024-09-18T08:02:48.870718+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:04:11.391201+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":66,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-09-18T08:04:06.673292+0000","last_change":"2024-09-18T08:02:47.881814+0000","last_active":"2024-09-18T08:04:06.673292+0000","last_peered":"2024-09-18T08:04:06.673292+0000","last_clean":"2024-09-18T08:04:06.673292+0000","last_became_active":"2024-09-18T08:02:47.881512+0000","last_became_peered":"2024-09-18T08:02:47.881512+0000","last_unstale":"2024-09-18T08:04:06.673292+0000","last_undegraded":"2024-09-18T08:04:06.673292+0000","last_fullsized":"2024-09-18T08:04:06.673292+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:02:46.860092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:46.860092+0000","last_clean_scrub_stamp":"2024-09-18T08:02:46.860092+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.d","version":"69'8","reported_seq":83,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-09-18T08:04:06.664250+0000","last_change":"2024-09-18T08:02:51.909290+0000","last_active":"2024-09-18T08:04:06.664250+0000","last_peered":"2024-09-18T08:04:06.664250+0000","last_clean":"2024-09-18T08:04:06.664250+0000","last_became_active":"2024-09-18T08:02:51.909011+0000","last_became_peered":"2024-09-18T08:02:51.909011+0000","last_unstale":"2024-09-18T08:04:06.664250+0000","last_undegraded":"2024-09-18T08:04:06.664250+0000","last_fullsized":"2024-09-18T08:04:06.664250+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:02:50.886830+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:50.886830+0000","last_clean_scrub_stamp":"2024-09-18T08:02:50.886830+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:04:10.431784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":55,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-09-18T08:04:06.661495+0000","last_change":"2024-09-18T08:02:53.902197+0000","last_active":"2024-09-18T08:04:06.661495+0000","last_peered":"2024-09-18T08:04:06.661495+0000","last_clean":"2024-09-18T08:04:06.661495+0000","last_became_active":"2024-09-18T08:02:53.901858+0000","last_became_peered":"2024-09-18T08:02:53.901858+0000","last_unstale":"2024-09-18T08:04:06.661495+0000","last_undegraded":"2024-09-18T08:04:06.661495+0000","last_fullsized":"2024-09-18T08:04:06.661495+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_clean_scrub_stamp":"2024-09-18T08:02:52.890475+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:04:08.426639+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.8","version":"89'415","reported_seq":438,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-09-18T08:04:08.068146+0000","last_change":"2024-09-18T08:03:57.465051+0000","last_active":"2024-09-18T08:04:08.068146+0000","last_peered":"2024-09-18T08:04:08.068146+0000","last_clean":"2024-09-18T08:04:08.068146+0000","last_became_active":"2024-09-18T08:03:57.464738+0000","last_became_peered":"2024-09-18T08:03:57.464738+0000","last_unstale":"2024-09-18T08:04:08.068146+0000","last_undegraded":"2024-09-18T08:04:08.068146+0000","last_fullsized":"2024-09-18T08:04:08.068146+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:03:56.449957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:03:56.449957+0000","last_clean_scrub_stamp":"2024-09-18T08:03:56.449957+0000","objects_scrubbed":0,"log_size":415,"log_dups_size":0,"ondisk_log_size":415,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:05:20.392706+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27197440,"num_objects":415,"num_object_clones":0,"num_object_copies":1245,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":415,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":415,"num_write_kb":26560,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,4],"acting":[2,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.b","version":"69'9","reported_seq":78,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-09-18T08:04:06.665944+0000","last_change":"2024-09-18T08:02:49.898562+0000","last_active":"2024-09-18T08:04:06.665944+0000","last_peered":"2024-09-18T08:04:06.665944+0000","last_clean":"2024-09-18T08:04:06.665944+0000","last_became_active":"2024-09-18T08:02:49.897998+0000","last_became_peered":"2024-09-18T08:02:49.897998+0000","last_unstale":"2024-09-18T08:04:06.665944+0000","last_undegraded":"2024-09-18T08:04:06.665944+0000","last_fullsized":"2024-09-18T08:04:06.665944+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_clean_scrub_stamp":"2024-09-18T08:02:48.870718+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:05:04.151697+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":66,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-09-18T08:04:06.665866+0000","last_change":"2024-09-18T08:02:47.877928+0000","last_active":"2024-09-18T08:04:06.665866+0000","last_peered":"2024-09-18T08:04:06.665866+0000","last_clean":"2024-09-18T08:04:06.665866+0000","last_became_active":"2024-09-18T08:02:47.877161+0000","last_became_peered":"2024-09-18T08:02:47.877161+0000","last_unstale":"2024-09-18T08:04:06.665866+0000","last_undegraded":"2024-09-18T08:04:06.665866+0000","last_fullsized":"2024-09-18T08:04:06.665866+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:02:46.860092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:46.860092+0000","last_clean_scrub_stamp":"2024-09-18T08:02:46.860092+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:04:57.881575+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":58,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-09-18T08:04:06.664593+0000","last_change":"2024-09-18T08:02:51.901738+0000","last_active":"2024-09-18T08:04:06.664593+0000","last_peered":"2024-09-18T08:04:06.664593+0000","last_clean":"2024-09-18T08:04:06.664593+0000","last_became_active":"2024-09-18T08:02:51.901330+0000","last_became_peered":"2024-09-18T08:02:51.901330+0000","last_unstale":"2024-09-18T08:04:06.664593+0000","last_undegraded":"2024-09-18T08:04:06.664593+0000","last_fullsized":"2024-09-18T08:04:06.664593+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:02:50.886830+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:50.886830+0000","last_clean_scrub_stamp":"2024-09-18T08:02:50.886830+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:04:18.413020+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,3],"acting":[2,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":55,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-09-18T08:04:06.667882+0000","last_change":"2024-09-18T08:02:53.911400+0000","last_active":"2024-09-18T08:04:06.667882+0000","last_peered":"2024-09-18T08:04:06.667882+0000","last_clean":"2024-09-18T08:04:06.667882+0000","last_became_active":"2024-09-18T08:02:53.911068+0000","last_became_peered":"2024-09-18T08:02:53.911068+0000","last_unstale":"2024-09-18T08:04:06.667882+0000","last_undegraded":"2024-09-18T08:04:06.667882+0000","last_fullsized":"2024-09-18T08:04:06.667882+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_clean_scrub_stamp":"2024-09-18T08:02:52.890475+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:05:05.147042+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.9","version":"89'446","reported_seq":469,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-09-18T08:04:08.053142+0000","last_change":"2024-09-18T08:03:57.467481+0000","last_active":"2024-09-18T08:04:08.053142+0000","last_peered":"2024-09-18T08:04:08.053142+0000","last_clean":"2024-09-18T08:04:08.053142+0000","last_became_active":"2024-09-18T08:03:57.467065+0000","last_became_peered":"2024-09-18T08:03:57.467065+0000","last_unstale":"2024-09-18T08:04:08.053142+0000","last_undegraded":"2024-09-18T08:04:08.053142+0000","last_fullsized":"2024-09-18T08:04:08.053142+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:03:56.449957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:03:56.449957+0000","last_clean_scrub_stamp":"2024-09-18T08:03:56.449957+0000","objects_scrubbed":0,"log_size":446,"log_dups_size":0,"ondisk_log_size":446,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:05:20.601065+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29229056,"num_objects":446,"num_object_clones":0,"num_object_copies":1338,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":446,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":446,"num_write_kb":28544,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.a","version":"69'19","reported_seq":93,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-09-18T08:04:06.663563+0000","last_change":"2024-09-18T08:02:49.898366+0000","last_active":"2024-09-18T08:04:06.663563+0000","last_peered":"2024-09-18T08:04:06.663563+0000","last_clean":"2024-09-18T08:04:06.663563+0000","last_became_active":"2024-09-18T08:02:49.897960+0000","last_became_peered":"2024-09-18T08:02:49.897960+0000","last_unstale":"2024-09-18T08:04:06.663563+0000","last_undegraded":"2024-09-18T08:04:06.663563+0000","last_fullsized":"2024-09-18T08:04:06.663563+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_clean_scrub_stamp":"2024-09-18T08:02:48.870718+0000","objects_scrubbed":0,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:04:18.534695+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":32,"num_read_kb":21,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":66,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-09-18T08:04:06.666124+0000","last_change":"2024-09-18T08:02:47.881119+0000","last_active":"2024-09-18T08:04:06.666124+0000","last_peered":"2024-09-18T08:04:06.666124+0000","last_clean":"2024-09-18T08:04:06.666124+0000","last_became_active":"2024-09-18T08:02:47.880645+0000","last_became_peered":"2024-09-18T08:02:47.880645+0000","last_unstale":"2024-09-18T08:04:06.666124+0000","last_undegraded":"2024-09-18T08:04:06.666124+0000","last_fullsized":"2024-09-18T08:04:06.666124+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:02:46.860092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:46.860092+0000","last_clean_scrub_stamp":"2024-09-18T08:02:46.860092+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":58,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-09-18T08:04:06.663988+0000","last_change":"2024-09-18T08:02:51.903944+0000","last_active":"2024-09-18T08:04:06.663988+0000","last_peered":"2024-09-18T08:04:06.663988+0000","last_clean":"2024-09-18T08:04:06.663988+0000","last_became_active":"2024-09-18T08:02:51.903439+0000","last_became_peered":"2024-09-18T08:02:51.903439+0000","last_unstale":"2024-09-18T08:04:06.663988+0000","last_undegraded":"2024-09-18T08:04:06.663988+0000","last_fullsized":"2024-09-18T08:04:06.663988+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:02:50.886830+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:50.886830+0000","last_clean_scrub_stamp":"2024-09-18T08:02:50.886830+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:04:55.734472+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":55,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-09-18T08:04:06.665687+0000","last_change":"2024-09-18T08:02:53.913096+0000","last_active":"2024-09-18T08:04:06.665687+0000","last_peered":"2024-09-18T08:04:06.665687+0000","last_clean":"2024-09-18T08:04:06.665687+0000","last_became_active":"2024-09-18T08:02:53.912762+0000","last_became_peered":"2024-09-18T08:02:53.912762+0000","last_unstale":"2024-09-18T08:04:06.665687+0000","last_undegraded":"2024-09-18T08:04:06.665687+0000","last_fullsized":"2024-09-18T08:04:06.665687+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_clean_scrub_stamp":"2024-09-18T08:02:52.890475+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.a","version":"89'425","reported_seq":447,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-09-18T08:04:08.067703+0000","last_change":"2024-09-18T08:03:57.466483+0000","last_active":"2024-09-18T08:04:08.067703+0000","last_peered":"2024-09-18T08:04:08.067703+0000","last_clean":"2024-09-18T08:04:08.067703+0000","last_became_active":"2024-09-18T08:03:57.466168+0000","last_became_peered":"2024-09-18T08:03:57.466168+0000","last_unstale":"2024-09-18T08:04:08.067703+0000","last_undegraded":"2024-09-18T08:04:08.067703+0000","last_fullsized":"2024-09-18T08:04:08.067703+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:03:56.449957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:03:56.449957+0000","last_clean_scrub_stamp":"2024-09-18T08:03:56.449957+0000","objects_scrubbed":0,"log_size":425,"log_dups_size":0,"ondisk_log_size":425,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:05:10.765872+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27852800,"num_objects":425,"num_object_clones":0,"num_object_copies":1275,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":425,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":425,"num_write_kb":27200,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.9","version":"69'12","reported_seq":85,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-09-18T08:04:06.660610+0000","last_change":"2024-09-18T08:02:49.895136+0000","last_active":"2024-09-18T08:04:06.660610+0000","last_peered":"2024-09-18T08:04:06.660610+0000","last_clean":"2024-09-18T08:04:06.660610+0000","last_became_active":"2024-09-18T08:02:49.894762+0000","last_became_peered":"2024-09-18T08:02:49.894762+0000","last_unstale":"2024-09-18T08:04:06.660610+0000","last_undegraded":"2024-09-18T08:04:06.660610+0000","last_fullsized":"2024-09-18T08:04:06.660610+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_clean_scrub_stamp":"2024-09-18T08:02:48.870718+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":25,"num_read_kb":16,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":66,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-09-18T08:04:06.666783+0000","last_change":"2024-09-18T08:02:47.884430+0000","last_active":"2024-09-18T08:04:06.666783+0000","last_peered":"2024-09-18T08:04:06.666783+0000","last_clean":"2024-09-18T08:04:06.666783+0000","last_became_active":"2024-09-18T08:02:47.884052+0000","last_became_peered":"2024-09-18T08:02:47.884052+0000","last_unstale":"2024-09-18T08:04:06.666783+0000","last_undegraded":"2024-09-18T08:04:06.666783+0000","last_fullsized":"2024-09-18T08:04:06.666783+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:02:46.860092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:46.860092+0000","last_clean_scrub_stamp":"2024-09-18T08:02:46.860092+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":58,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-09-18T08:04:06.664771+0000","last_change":"2024-09-18T08:02:51.902541+0000","last_active":"2024-09-18T08:04:06.664771+0000","last_peered":"2024-09-18T08:04:06.664771+0000","last_clean":"2024-09-18T08:04:06.664771+0000","last_became_active":"2024-09-18T08:02:51.902163+0000","last_became_peered":"2024-09-18T08:02:51.902163+0000","last_unstale":"2024-09-18T08:04:06.664771+0000","last_undegraded":"2024-09-18T08:04:06.664771+0000","last_fullsized":"2024-09-18T08:04:06.664771+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:02:50.886830+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:50.886830+0000","last_clean_scrub_stamp":"2024-09-18T08:02:50.886830+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:05:03.695963+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":55,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-09-18T08:04:06.672992+0000","last_change":"2024-09-18T08:02:53.907752+0000","last_active":"2024-09-18T08:04:06.672992+0000","last_peered":"2024-09-18T08:04:06.672992+0000","last_clean":"2024-09-18T08:04:06.672992+0000","last_became_active":"2024-09-18T08:02:53.907412+0000","last_became_peered":"2024-09-18T08:02:53.907412+0000","last_unstale":"2024-09-18T08:04:06.672992+0000","last_undegraded":"2024-09-18T08:04:06.672992+0000","last_fullsized":"2024-09-18T08:04:06.672992+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_clean_scrub_stamp":"2024-09-18T08:02:52.890475+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:04:12.083897+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.b","version":"89'443","reported_seq":466,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-09-18T08:04:08.073217+0000","last_change":"2024-09-18T08:03:57.466525+0000","last_active":"2024-09-18T08:04:08.073217+0000","last_peered":"2024-09-18T08:04:08.073217+0000","last_clean":"2024-09-18T08:04:08.073217+0000","last_became_active":"2024-09-18T08:03:57.466106+0000","last_became_peered":"2024-09-18T08:03:57.466106+0000","last_unstale":"2024-09-18T08:04:08.073217+0000","last_undegraded":"2024-09-18T08:04:08.073217+0000","last_fullsized":"2024-09-18T08:04:08.073217+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:03:56.449957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:03:56.449957+0000","last_clean_scrub_stamp":"2024-09-18T08:03:56.449957+0000","objects_scrubbed":0,"log_size":443,"log_dups_size":0,"ondisk_log_size":443,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:05:01.891230+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29032448,"num_objects":443,"num_object_clones":0,"num_object_copies":1329,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":443,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":443,"num_write_kb":28352,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6],"acting":[1,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.8","version":"69'15","reported_seq":87,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-09-18T08:04:06.665207+0000","last_change":"2024-09-18T08:02:49.889133+0000","last_active":"2024-09-18T08:04:06.665207+0000","last_peered":"2024-09-18T08:04:06.665207+0000","last_clean":"2024-09-18T08:04:06.665207+0000","last_became_active":"2024-09-18T08:02:49.888799+0000","last_became_peered":"2024-09-18T08:02:49.888799+0000","last_unstale":"2024-09-18T08:04:06.665207+0000","last_undegraded":"2024-09-18T08:04:06.665207+0000","last_fullsized":"2024-09-18T08:04:06.665207+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_clean_scrub_stamp":"2024-09-18T08:02:48.870718+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:04:11.344161+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.f","version":"62'2","reported_seq":86,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-09-18T08:04:06.666209+0000","last_change":"2024-09-18T08:03:50.578695+0000","last_active":"2024-09-18T08:04:06.666209+0000","last_peered":"2024-09-18T08:04:06.666209+0000","last_clean":"2024-09-18T08:04:06.666209+0000","last_became_active":"2024-09-18T08:02:47.881165+0000","last_became_peered":"2024-09-18T08:02:47.881165+0000","last_unstale":"2024-09-18T08:04:06.666209+0000","last_undegraded":"2024-09-18T08:04:06.666209+0000","last_fullsized":"2024-09-18T08:04:06.666209+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"62'2","last_scrub_stamp":"2024-09-18T08:03:50.578625+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:46.860092+0000","last_clean_scrub_stamp":"2024-09-18T08:03:50.578625+0000","objects_scrubbed":2,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:05:11.413212+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":4,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.9","version":"69'8","reported_seq":94,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-09-18T08:04:06.665597+0000","last_change":"2024-09-18T08:03:52.572824+0000","last_active":"2024-09-18T08:04:06.665597+0000","last_peered":"2024-09-18T08:04:06.665597+0000","last_clean":"2024-09-18T08:04:06.665597+0000","last_became_active":"2024-09-18T08:02:51.905059+0000","last_became_peered":"2024-09-18T08:02:51.905059+0000","last_unstale":"2024-09-18T08:04:06.665597+0000","last_undegraded":"2024-09-18T08:04:06.665597+0000","last_fullsized":"2024-09-18T08:04:06.665597+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'8","last_scrub_stamp":"2024-09-18T08:03:52.572757+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:50.886830+0000","last_clean_scrub_stamp":"2024-09-18T08:03:52.572757+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:05:11.975416+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":54,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-09-18T08:04:06.665357+0000","last_change":"2024-09-18T08:02:54.046138+0000","last_active":"2024-09-18T08:04:06.665357+0000","last_peered":"2024-09-18T08:04:06.665357+0000","last_clean":"2024-09-18T08:04:06.665357+0000","last_became_active":"2024-09-18T08:02:54.045845+0000","last_became_peered":"2024-09-18T08:02:54.045845+0000","last_unstale":"2024-09-18T08:04:06.665357+0000","last_undegraded":"2024-09-18T08:04:06.665357+0000","last_fullsized":"2024-09-18T08:04:06.665357+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_clean_scrub_stamp":"2024-09-18T08:02:52.890475+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:05:00.877573+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":66,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-09-18T08:04:06.664051+0000","last_change":"2024-09-18T08:02:47.882886+0000","last_active":"2024-09-18T08:04:06.664051+0000","last_peered":"2024-09-18T08:04:06.664051+0000","last_clean":"2024-09-18T08:04:06.664051+0000","last_became_active":"2024-09-18T08:02:47.882600+0000","last_became_peered":"2024-09-18T08:02:47.882600+0000","last_unstale":"2024-09-18T08:04:06.664051+0000","last_undegraded":"2024-09-18T08:04:06.664051+0000","last_fullsized":"2024-09-18T08:04:06.664051+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:02:46.860092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:46.860092+0000","last_clean_scrub_stamp":"2024-09-18T08:02:46.860092+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:04:15.079741+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"69'6","reported_seq":79,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-09-18T08:04:07.990894+0000","last_change":"2024-09-18T08:04:07.990894+0000","last_active":"2024-09-18T08:04:07.990894+0000","last_peered":"2024-09-18T08:04:07.990894+0000","last_clean":"2024-09-18T08:04:07.990894+0000","last_became_active":"2024-09-18T08:02:49.895067+0000","last_became_peered":"2024-09-18T08:02:49.895067+0000","last_unstale":"2024-09-18T08:04:07.990894+0000","last_undegraded":"2024-09-18T08:04:07.990894+0000","last_fullsized":"2024-09-18T08:04:07.990894+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'6","last_scrub_stamp":"2024-09-18T08:04:07.990691+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_clean_scrub_stamp":"2024-09-18T08:04:07.990691+0000","objects_scrubbed":3,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:05:14.375549+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":9,"num_read_kb":6,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":58,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-09-18T08:04:06.665759+0000","last_change":"2024-09-18T08:02:51.912907+0000","last_active":"2024-09-18T08:04:06.665759+0000","last_peered":"2024-09-18T08:04:06.665759+0000","last_clean":"2024-09-18T08:04:06.665759+0000","last_became_active":"2024-09-18T08:02:51.912557+0000","last_became_peered":"2024-09-18T08:02:51.912557+0000","last_unstale":"2024-09-18T08:04:06.665759+0000","last_undegraded":"2024-09-18T08:04:06.665759+0000","last_fullsized":"2024-09-18T08:04:06.665759+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:02:50.886830+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:50.886830+0000","last_clean_scrub_stamp":"2024-09-18T08:02:50.886830+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,1],"acting":[5,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":55,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-09-18T08:04:06.665842+0000","last_change":"2024-09-18T08:02:54.049521+0000","last_active":"2024-09-18T08:04:06.665842+0000","last_peered":"2024-09-18T08:04:06.665842+0000","last_clean":"2024-09-18T08:04:06.665842+0000","last_became_active":"2024-09-18T08:02:54.048959+0000","last_became_peered":"2024-09-18T08:02:54.048959+0000","last_unstale":"2024-09-18T08:04:06.665842+0000","last_undegraded":"2024-09-18T08:04:06.665842+0000","last_fullsized":"2024-09-18T08:04:06.665842+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_clean_scrub_stamp":"2024-09-18T08:02:52.890475+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:05:00.656477+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"69'9","reported_seq":78,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-09-18T08:04:06.670250+0000","last_change":"2024-09-18T08:02:49.892390+0000","last_active":"2024-09-18T08:04:06.670250+0000","last_peered":"2024-09-18T08:04:06.670250+0000","last_clean":"2024-09-18T08:04:06.670250+0000","last_became_active":"2024-09-18T08:02:49.891567+0000","last_became_peered":"2024-09-18T08:02:49.891567+0000","last_unstale":"2024-09-18T08:04:06.670250+0000","last_undegraded":"2024-09-18T08:04:06.670250+0000","last_fullsized":"2024-09-18T08:04:06.670250+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_clean_scrub_stamp":"2024-09-18T08:02:48.870718+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:05:00.103111+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":74,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-09-18T08:04:06.665694+0000","last_change":"2024-09-18T08:03:49.591148+0000","last_active":"2024-09-18T08:04:06.665694+0000","last_peered":"2024-09-18T08:04:06.665694+0000","last_clean":"2024-09-18T08:04:06.665694+0000","last_became_active":"2024-09-18T08:02:47.885790+0000","last_became_peered":"2024-09-18T08:02:47.885790+0000","last_unstale":"2024-09-18T08:04:06.665694+0000","last_undegraded":"2024-09-18T08:04:06.665694+0000","last_fullsized":"2024-09-18T08:04:06.665694+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:03:49.591068+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:46.860092+0000","last_clean_scrub_stamp":"2024-09-18T08:03:49.591068+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:05:16.320042+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":58,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-09-18T08:04:06.672758+0000","last_change":"2024-09-18T08:02:51.908640+0000","last_active":"2024-09-18T08:04:06.672758+0000","last_peered":"2024-09-18T08:04:06.672758+0000","last_clean":"2024-09-18T08:04:06.672758+0000","last_became_active":"2024-09-18T08:02:51.907361+0000","last_became_peered":"2024-09-18T08:02:51.907361+0000","last_unstale":"2024-09-18T08:04:06.672758+0000","last_undegraded":"2024-09-18T08:04:06.672758+0000","last_fullsized":"2024-09-18T08:04:06.672758+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:02:50.886830+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:50.886830+0000","last_clean_scrub_stamp":"2024-09-18T08:02:50.886830+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":55,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-09-18T08:04:06.663773+0000","last_change":"2024-09-18T08:02:53.909246+0000","last_active":"2024-09-18T08:04:06.663773+0000","last_peered":"2024-09-18T08:04:06.663773+0000","last_clean":"2024-09-18T08:04:06.663773+0000","last_became_active":"2024-09-18T08:02:53.908917+0000","last_became_peered":"2024-09-18T08:02:53.908917+0000","last_unstale":"2024-09-18T08:04:06.663773+0000","last_undegraded":"2024-09-18T08:04:06.663773+0000","last_fullsized":"2024-09-18T08:04:06.663773+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_clean_scrub_stamp":"2024-09-18T08:02:52.890475+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:05:05.735422+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.15","version":"69'9","reported_seq":78,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-09-18T08:04:06.666137+0000","last_change":"2024-09-18T08:02:49.890846+0000","last_active":"2024-09-18T08:04:06.666137+0000","last_peered":"2024-09-18T08:04:06.666137+0000","last_clean":"2024-09-18T08:04:06.666137+0000","last_became_active":"2024-09-18T08:02:49.890497+0000","last_became_peered":"2024-09-18T08:02:49.890497+0000","last_unstale":"2024-09-18T08:04:06.666137+0000","last_undegraded":"2024-09-18T08:04:06.666137+0000","last_fullsized":"2024-09-18T08:04:06.666137+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_clean_scrub_stamp":"2024-09-18T08:02:48.870718+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":66,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-09-18T08:04:06.665651+0000","last_change":"2024-09-18T08:02:47.877632+0000","last_active":"2024-09-18T08:04:06.665651+0000","last_peered":"2024-09-18T08:04:06.665651+0000","last_clean":"2024-09-18T08:04:06.665651+0000","last_became_active":"2024-09-18T08:02:47.875983+0000","last_became_peered":"2024-09-18T08:02:47.875983+0000","last_unstale":"2024-09-18T08:04:06.665651+0000","last_undegraded":"2024-09-18T08:04:06.665651+0000","last_fullsized":"2024-09-18T08:04:06.665651+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:02:46.860092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:46.860092+0000","last_clean_scrub_stamp":"2024-09-18T08:02:46.860092+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:04:09.677911+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.14","version":"69'8","reported_seq":83,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-09-18T08:04:06.671196+0000","last_change":"2024-09-18T08:02:51.909896+0000","last_active":"2024-09-18T08:04:06.671196+0000","last_peered":"2024-09-18T08:04:06.671196+0000","last_clean":"2024-09-18T08:04:06.671196+0000","last_became_active":"2024-09-18T08:02:51.909513+0000","last_became_peered":"2024-09-18T08:02:51.909513+0000","last_unstale":"2024-09-18T08:04:06.671196+0000","last_undegraded":"2024-09-18T08:04:06.671196+0000","last_fullsized":"2024-09-18T08:04:06.671196+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:02:50.886830+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:50.886830+0000","last_clean_scrub_stamp":"2024-09-18T08:02:50.886830+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:04:56.810715+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":55,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-09-18T08:04:06.661856+0000","last_change":"2024-09-18T08:02:53.907729+0000","last_active":"2024-09-18T08:04:06.661856+0000","last_peered":"2024-09-18T08:04:06.661856+0000","last_clean":"2024-09-18T08:04:06.661856+0000","last_became_active":"2024-09-18T08:02:53.907428+0000","last_became_peered":"2024-09-18T08:02:53.907428+0000","last_unstale":"2024-09-18T08:04:06.661856+0000","last_undegraded":"2024-09-18T08:04:06.661856+0000","last_fullsized":"2024-09-18T08:04:06.661856+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_clean_scrub_stamp":"2024-09-18T08:02:52.890475+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:05:06.324449+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.14","version":"69'10","reported_seq":77,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-09-18T08:04:06.671521+0000","last_change":"2024-09-18T08:02:49.889483+0000","last_active":"2024-09-18T08:04:06.671521+0000","last_peered":"2024-09-18T08:04:06.671521+0000","last_clean":"2024-09-18T08:04:06.671521+0000","last_became_active":"2024-09-18T08:02:49.889191+0000","last_became_peered":"2024-09-18T08:02:49.889191+0000","last_unstale":"2024-09-18T08:04:06.671521+0000","last_undegraded":"2024-09-18T08:04:06.671521+0000","last_fullsized":"2024-09-18T08:04:06.671521+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_clean_scrub_stamp":"2024-09-18T08:02:48.870718+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":66,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-09-18T08:04:06.666141+0000","last_change":"2024-09-18T08:02:47.882542+0000","last_active":"2024-09-18T08:04:06.666141+0000","last_peered":"2024-09-18T08:04:06.666141+0000","last_clean":"2024-09-18T08:04:06.666141+0000","last_became_active":"2024-09-18T08:02:47.882065+0000","last_became_peered":"2024-09-18T08:02:47.882065+0000","last_unstale":"2024-09-18T08:04:06.666141+0000","last_undegraded":"2024-09-18T08:04:06.666141+0000","last_fullsized":"2024-09-18T08:04:06.666141+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:02:46.860092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:46.860092+0000","last_clean_scrub_stamp":"2024-09-18T08:02:46.860092+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:05:03.725663+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.15","version":"69'8","reported_seq":83,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-09-18T08:04:06.666026+0000","last_change":"2024-09-18T08:02:51.914448+0000","last_active":"2024-09-18T08:04:06.666026+0000","last_peered":"2024-09-18T08:04:06.666026+0000","last_clean":"2024-09-18T08:04:06.666026+0000","last_became_active":"2024-09-18T08:02:51.914135+0000","last_became_peered":"2024-09-18T08:02:51.914135+0000","last_unstale":"2024-09-18T08:04:06.666026+0000","last_undegraded":"2024-09-18T08:04:06.666026+0000","last_fullsized":"2024-09-18T08:04:06.666026+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:02:50.886830+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:50.886830+0000","last_clean_scrub_stamp":"2024-09-18T08:02:50.886830+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":55,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-09-18T08:04:06.669318+0000","last_change":"2024-09-18T08:02:53.912409+0000","last_active":"2024-09-18T08:04:06.669318+0000","last_peered":"2024-09-18T08:04:06.669318+0000","last_clean":"2024-09-18T08:04:06.669318+0000","last_became_active":"2024-09-18T08:02:53.912112+0000","last_became_peered":"2024-09-18T08:02:53.912112+0000","last_unstale":"2024-09-18T08:04:06.669318+0000","last_undegraded":"2024-09-18T08:04:06.669318+0000","last_fullsized":"2024-09-18T08:04:06.669318+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_clean_scrub_stamp":"2024-09-18T08:02:52.890475+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:05:03.142084+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.13","version":"69'11","reported_seq":81,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-09-18T08:04:06.661429+0000","last_change":"2024-09-18T08:02:49.897433+0000","last_active":"2024-09-18T08:04:06.661429+0000","last_peered":"2024-09-18T08:04:06.661429+0000","last_clean":"2024-09-18T08:04:06.661429+0000","last_became_active":"2024-09-18T08:02:49.896949+0000","last_became_peered":"2024-09-18T08:02:49.896949+0000","last_unstale":"2024-09-18T08:04:06.661429+0000","last_undegraded":"2024-09-18T08:04:06.661429+0000","last_fullsized":"2024-09-18T08:04:06.661429+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_clean_scrub_stamp":"2024-09-18T08:02:48.870718+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:04:08.564349+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":66,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-09-18T08:04:06.661593+0000","last_change":"2024-09-18T08:02:47.880202+0000","last_active":"2024-09-18T08:04:06.661593+0000","last_peered":"2024-09-18T08:04:06.661593+0000","last_clean":"2024-09-18T08:04:06.661593+0000","last_became_active":"2024-09-18T08:02:47.879771+0000","last_became_peered":"2024-09-18T08:02:47.879771+0000","last_unstale":"2024-09-18T08:04:06.661593+0000","last_undegraded":"2024-09-18T08:04:06.661593+0000","last_fullsized":"2024-09-18T08:04:06.661593+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:02:46.860092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:46.860092+0000","last_clean_scrub_stamp":"2024-09-18T08:02:46.860092+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:05:00.322786+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":58,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-09-18T08:04:06.672224+0000","last_change":"2024-09-18T08:02:51.912062+0000","last_active":"2024-09-18T08:04:06.672224+0000","last_peered":"2024-09-18T08:04:06.672224+0000","last_clean":"2024-09-18T08:04:06.672224+0000","last_became_active":"2024-09-18T08:02:51.910752+0000","last_became_peered":"2024-09-18T08:02:51.910752+0000","last_unstale":"2024-09-18T08:04:06.672224+0000","last_undegraded":"2024-09-18T08:04:06.672224+0000","last_fullsized":"2024-09-18T08:04:06.672224+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:02:50.886830+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:50.886830+0000","last_clean_scrub_stamp":"2024-09-18T08:02:50.886830+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":55,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-09-18T08:04:06.670887+0000","last_change":"2024-09-18T08:02:53.911861+0000","last_active":"2024-09-18T08:04:06.670887+0000","last_peered":"2024-09-18T08:04:06.670887+0000","last_clean":"2024-09-18T08:04:06.670887+0000","last_became_active":"2024-09-18T08:02:53.911429+0000","last_became_peered":"2024-09-18T08:02:53.911429+0000","last_unstale":"2024-09-18T08:04:06.670887+0000","last_undegraded":"2024-09-18T08:04:06.670887+0000","last_fullsized":"2024-09-18T08:04:06.670887+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_clean_scrub_stamp":"2024-09-18T08:02:52.890475+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.12","version":"69'9","reported_seq":78,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-09-18T08:04:06.672463+0000","last_change":"2024-09-18T08:02:49.893472+0000","last_active":"2024-09-18T08:04:06.672463+0000","last_peered":"2024-09-18T08:04:06.672463+0000","last_clean":"2024-09-18T08:04:06.672463+0000","last_became_active":"2024-09-18T08:02:49.885950+0000","last_became_peered":"2024-09-18T08:02:49.885950+0000","last_unstale":"2024-09-18T08:04:06.672463+0000","last_undegraded":"2024-09-18T08:04:06.672463+0000","last_fullsized":"2024-09-18T08:04:06.672463+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_clean_scrub_stamp":"2024-09-18T08:02:48.870718+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:04:59.719937+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":66,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-09-18T08:04:06.666025+0000","last_change":"2024-09-18T08:02:47.883002+0000","last_active":"2024-09-18T08:04:06.666025+0000","last_peered":"2024-09-18T08:04:06.666025+0000","last_clean":"2024-09-18T08:04:06.666025+0000","last_became_active":"2024-09-18T08:02:47.882096+0000","last_became_peered":"2024-09-18T08:02:47.882096+0000","last_unstale":"2024-09-18T08:04:06.666025+0000","last_undegraded":"2024-09-18T08:04:06.666025+0000","last_fullsized":"2024-09-18T08:04:06.666025+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:02:46.860092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:46.860092+0000","last_clean_scrub_stamp":"2024-09-18T08:02:46.860092+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:04:10.794338+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":58,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-09-18T08:04:06.673075+0000","last_change":"2024-09-18T08:02:51.906594+0000","last_active":"2024-09-18T08:04:06.673075+0000","last_peered":"2024-09-18T08:04:06.673075+0000","last_clean":"2024-09-18T08:04:06.673075+0000","last_became_active":"2024-09-18T08:02:51.906277+0000","last_became_peered":"2024-09-18T08:02:51.906277+0000","last_unstale":"2024-09-18T08:04:06.673075+0000","last_undegraded":"2024-09-18T08:04:06.673075+0000","last_fullsized":"2024-09-18T08:04:06.673075+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:02:50.886830+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:50.886830+0000","last_clean_scrub_stamp":"2024-09-18T08:02:50.886830+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:04:07.327398+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.10","version":"69'1","reported_seq":56,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-09-18T08:04:06.665228+0000","last_change":"2024-09-18T08:02:53.913013+0000","last_active":"2024-09-18T08:04:06.665228+0000","last_peered":"2024-09-18T08:04:06.665228+0000","last_clean":"2024-09-18T08:04:06.665228+0000","last_became_active":"2024-09-18T08:02:53.912699+0000","last_became_peered":"2024-09-18T08:02:53.912699+0000","last_unstale":"2024-09-18T08:04:06.665228+0000","last_undegraded":"2024-09-18T08:04:06.665228+0000","last_fullsized":"2024-09-18T08:04:06.665228+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_clean_scrub_stamp":"2024-09-18T08:02:52.890475+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:04:15.154915+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.11","version":"69'11","reported_seq":89,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-09-18T08:04:06.671027+0000","last_change":"2024-09-18T08:03:51.831091+0000","last_active":"2024-09-18T08:04:06.671027+0000","last_peered":"2024-09-18T08:04:06.671027+0000","last_clean":"2024-09-18T08:04:06.671027+0000","last_became_active":"2024-09-18T08:02:49.889611+0000","last_became_peered":"2024-09-18T08:02:49.889611+0000","last_unstale":"2024-09-18T08:04:06.671027+0000","last_undegraded":"2024-09-18T08:04:06.671027+0000","last_fullsized":"2024-09-18T08:04:06.671027+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'11","last_scrub_stamp":"2024-09-18T08:03:51.831006+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_clean_scrub_stamp":"2024-09-18T08:03:51.831006+0000","objects_scrubbed":5,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:05:05.641442+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":74,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-09-18T08:04:06.665873+0000","last_change":"2024-09-18T08:03:49.960690+0000","last_active":"2024-09-18T08:04:06.665873+0000","last_peered":"2024-09-18T08:04:06.665873+0000","last_clean":"2024-09-18T08:04:06.665873+0000","last_became_active":"2024-09-18T08:02:47.878886+0000","last_became_peered":"2024-09-18T08:02:47.878886+0000","last_unstale":"2024-09-18T08:04:06.665873+0000","last_undegraded":"2024-09-18T08:04:06.665873+0000","last_fullsized":"2024-09-18T08:04:06.665873+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:03:49.960617+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:46.860092+0000","last_clean_scrub_stamp":"2024-09-18T08:03:49.960617+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:05:10.790197+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":58,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-09-18T08:04:06.665340+0000","last_change":"2024-09-18T08:02:51.908058+0000","last_active":"2024-09-18T08:04:06.665340+0000","last_peered":"2024-09-18T08:04:06.665340+0000","last_clean":"2024-09-18T08:04:06.665340+0000","last_became_active":"2024-09-18T08:02:51.907762+0000","last_became_peered":"2024-09-18T08:02:51.907762+0000","last_unstale":"2024-09-18T08:04:06.665340+0000","last_undegraded":"2024-09-18T08:04:06.665340+0000","last_fullsized":"2024-09-18T08:04:06.665340+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:02:50.886830+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:50.886830+0000","last_clean_scrub_stamp":"2024-09-18T08:02:50.886830+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":54,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-09-18T08:04:06.673212+0000","last_change":"2024-09-18T08:02:54.047687+0000","last_active":"2024-09-18T08:04:06.673212+0000","last_peered":"2024-09-18T08:04:06.673212+0000","last_clean":"2024-09-18T08:04:06.673212+0000","last_became_active":"2024-09-18T08:02:54.047181+0000","last_became_peered":"2024-09-18T08:02:54.047181+0000","last_unstale":"2024-09-18T08:04:06.673212+0000","last_undegraded":"2024-09-18T08:04:06.673212+0000","last_fullsized":"2024-09-18T08:04:06.673212+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_clean_scrub_stamp":"2024-09-18T08:02:52.890475+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:04:12.998775+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.10","version":"69'4","reported_seq":68,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-09-18T08:04:06.670897+0000","last_change":"2024-09-18T08:02:49.890435+0000","last_active":"2024-09-18T08:04:06.670897+0000","last_peered":"2024-09-18T08:04:06.670897+0000","last_clean":"2024-09-18T08:04:06.670897+0000","last_became_active":"2024-09-18T08:02:49.890126+0000","last_became_peered":"2024-09-18T08:02:49.890126+0000","last_unstale":"2024-09-18T08:04:06.670897+0000","last_undegraded":"2024-09-18T08:04:06.670897+0000","last_fullsized":"2024-09-18T08:04:06.670897+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_clean_scrub_stamp":"2024-09-18T08:02:48.870718+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":6,"num_read_kb":4,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":66,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-09-18T08:04:06.665636+0000","last_change":"2024-09-18T08:02:47.877991+0000","last_active":"2024-09-18T08:04:06.665636+0000","last_peered":"2024-09-18T08:04:06.665636+0000","last_clean":"2024-09-18T08:04:06.665636+0000","last_became_active":"2024-09-18T08:02:47.877616+0000","last_became_peered":"2024-09-18T08:02:47.877616+0000","last_unstale":"2024-09-18T08:04:06.665636+0000","last_undegraded":"2024-09-18T08:04:06.665636+0000","last_fullsized":"2024-09-18T08:04:06.665636+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:02:46.860092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:46.860092+0000","last_clean_scrub_stamp":"2024-09-18T08:02:46.860092+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":66,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-09-18T08:04:06.663937+0000","last_change":"2024-09-18T08:03:51.444674+0000","last_active":"2024-09-18T08:04:06.663937+0000","last_peered":"2024-09-18T08:04:06.663937+0000","last_clean":"2024-09-18T08:04:06.663937+0000","last_became_active":"2024-09-18T08:02:51.906372+0000","last_became_peered":"2024-09-18T08:02:51.906372+0000","last_unstale":"2024-09-18T08:04:06.663937+0000","last_undegraded":"2024-09-18T08:04:06.663937+0000","last_fullsized":"2024-09-18T08:04:06.663937+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:03:51.444624+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:50.886830+0000","last_clean_scrub_stamp":"2024-09-18T08:03:51.444624+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:04:55.853846+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":55,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-09-18T08:04:06.665387+0000","last_change":"2024-09-18T08:02:53.909445+0000","last_active":"2024-09-18T08:04:06.665387+0000","last_peered":"2024-09-18T08:04:06.665387+0000","last_clean":"2024-09-18T08:04:06.665387+0000","last_became_active":"2024-09-18T08:02:53.909146+0000","last_became_peered":"2024-09-18T08:02:53.909146+0000","last_unstale":"2024-09-18T08:04:06.665387+0000","last_undegraded":"2024-09-18T08:04:06.665387+0000","last_fullsized":"2024-09-18T08:04:06.665387+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_clean_scrub_stamp":"2024-09-18T08:02:52.890475+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,4],"acting":[7,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":55,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-09-18T08:04:06.668058+0000","last_change":"2024-09-18T08:02:53.907055+0000","last_active":"2024-09-18T08:04:06.668058+0000","last_peered":"2024-09-18T08:04:06.668058+0000","last_clean":"2024-09-18T08:04:06.668058+0000","last_became_active":"2024-09-18T08:02:53.906726+0000","last_became_peered":"2024-09-18T08:02:53.906726+0000","last_unstale":"2024-09-18T08:04:06.668058+0000","last_undegraded":"2024-09-18T08:04:06.668058+0000","last_fullsized":"2024-09-18T08:04:06.668058+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_clean_scrub_stamp":"2024-09-18T08:02:52.890475+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:05:04.730046+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":66,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-09-18T08:04:06.671027+0000","last_change":"2024-09-18T08:02:47.879447+0000","last_active":"2024-09-18T08:04:06.671027+0000","last_peered":"2024-09-18T08:04:06.671027+0000","last_clean":"2024-09-18T08:04:06.671027+0000","last_became_active":"2024-09-18T08:02:47.878922+0000","last_became_peered":"2024-09-18T08:02:47.878922+0000","last_unstale":"2024-09-18T08:04:06.671027+0000","last_undegraded":"2024-09-18T08:04:06.671027+0000","last_fullsized":"2024-09-18T08:04:06.671027+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:02:46.860092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:46.860092+0000","last_clean_scrub_stamp":"2024-09-18T08:02:46.860092+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:04:58.828015+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"69'11","reported_seq":81,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-09-18T08:04:06.664393+0000","last_change":"2024-09-18T08:02:49.898630+0000","last_active":"2024-09-18T08:04:06.664393+0000","last_peered":"2024-09-18T08:04:06.664393+0000","last_clean":"2024-09-18T08:04:06.664393+0000","last_became_active":"2024-09-18T08:02:49.898048+0000","last_became_peered":"2024-09-18T08:02:49.898048+0000","last_unstale":"2024-09-18T08:04:06.664393+0000","last_undegraded":"2024-09-18T08:04:06.664393+0000","last_fullsized":"2024-09-18T08:04:06.664393+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_clean_scrub_stamp":"2024-09-18T08:02:48.870718+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:05:06.370782+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":58,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-09-18T08:04:06.669889+0000","last_change":"2024-09-18T08:02:51.900666+0000","last_active":"2024-09-18T08:04:06.669889+0000","last_peered":"2024-09-18T08:04:06.669889+0000","last_clean":"2024-09-18T08:04:06.669889+0000","last_became_active":"2024-09-18T08:02:51.900300+0000","last_became_peered":"2024-09-18T08:02:51.900300+0000","last_unstale":"2024-09-18T08:04:06.669889+0000","last_undegraded":"2024-09-18T08:04:06.669889+0000","last_fullsized":"2024-09-18T08:04:06.669889+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:02:50.886830+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:50.886830+0000","last_clean_scrub_stamp":"2024-09-18T08:02:50.886830+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:04:12.807594+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":7,"num_pg":16,"stat_sum":{"num_bytes":451149824,"num_objects":6884,"num_object_clones":0,"num_object_copies":20652,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6884,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6884,"num_write_kb":440576,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1354104832,"data_stored":1354104832,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":6884,"ondisk_log_size":6884,"up":48,"acting":48,"num_store_stats":8},{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":416,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":64,"ondisk_log_size":64,"up":96,"acting":96,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":178,"num_object_clones":0,"num_object_copies":534,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":178,"num_whiteouts":0,"num_read":698,"num_read_kb":455,"num_write":417,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":417792,"data_stored":11106,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":393,"ondisk_log_size":393,"up":96,"acting":96,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":1613,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":24,"num_read_kb":24,"num_write":10,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":4839,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":6,"ondisk_log_size":6,"up":96,"acting":96,"num_store_stats":8},{"poolid":2,"num_pg":3,"stat_sum":{"num_bytes":408,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":75,"num_read_kb":70,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":8,"ondisk_log_size":8,"up":9,"acting":9,"num_store_stats":7},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":231,"num_write":113,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":590368,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":79,"ondisk_log_size":79,"up":3,"acting":3,"num_store_stats":3}],"osd_stats":[{"osd":7,"up_from":56,"seq":240518168596,"num_pgs":58,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":159320,"kb_used_data":123976,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93585832,"statfs":{"total":95995035648,"available":95831891968,"internally_reserved":0,"allocated":126951424,"data_stored":126416627,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":36174282},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":47,"seq":201863462938,"num_pgs":48,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":128008,"kb_used_data":96820,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93617144,"statfs":{"total":95995035648,"available":95863955456,"internally_reserved":0,"allocated":99143680,"data_stored":98628278,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1],"upper_bound":16},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":39,"seq":167503724576,"num_pgs":51,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":136400,"kb_used_data":100988,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93608752,"statfs":{"total":95995035648,"available":95855362048,"internally_reserved":0,"allocated":103411712,"data_stored":102888382,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":36174278},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":32,"seq":137438953509,"num_pgs":63,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":129956,"kb_used_data":94608,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93615196,"statfs":{"total":95995035648,"available":95861960704,"internally_reserved":0,"allocated":96878592,"data_stored":96335572,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":26,"seq":111669149740,"num_pgs":65,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":270696,"kb_used_data":235348,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93474456,"statfs":{"total":95995035648,"available":95717842944,"internally_reserved":0,"allocated":240996352,"data_stored":240448529,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":36174280},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,1,1,4,1],"upper_bound":32},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411378,"num_pgs":42,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":178628,"kb_used_data":139120,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":93566524,"statfs":{"total":95995035648,"available":95812120576,"internally_reserved":0,"allocated":142458880,"data_stored":141947614,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":40434117},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574902,"num_pgs":63,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":183824,"kb_used_data":148476,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93561328,"statfs":{"total":95995035648,"available":95806799872,"internally_reserved":0,"allocated":152039424,"data_stored":151452977,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738429,"num_pgs":54,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":178152,"kb_used_data":142804,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93567000,"statfs":{"total":95995035648,"available":95812608000,"internally_reserved":0,"allocated":146231296,"data_stored":145684926,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1131,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1521,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":436,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":46,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1177,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":436,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":90112,"data_stored":2338,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":798,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":1898,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1474,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":990,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1034,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1254,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":403,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":416,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":403,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":227016704,"data_stored":227016704,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":173277184,"data_stored":173277184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":168558592,"data_stored":168558592,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":250347520,"data_stored":250347520,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":139264000,"data_stored":139264000,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":113049600,"data_stored":113049600,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":141950976,"data_stored":141950976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":140640256,"data_stored":140640256,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T08:04:11.499 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid 779349ca-7593-11ef-bceb-c7b262605968 -- ceph osd pool set .mgr pgp_num 1 2024-09-18T08:04:11.530 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1 2024-09-18T08:04:11.567 INFO:tasks.radosbench.radosbench.0.smithi007.stdout: 6 16 16275 16259 169.344 158.625 0.00577247 0.0058967 2024-09-18T08:04:11.595 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:04:11.596 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:04:11.596 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1' 2024-09-18T08:04:11.697 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5 2024-09-18T08:04:11.726 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:04:11.727 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:04:11.727 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5' 2024-09-18T08:04:11.828 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3 2024-09-18T08:04:11.873 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:04:11.874 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:04:11.874 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3' 2024-09-18T08:04:11.976 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7 2024-09-18T08:04:11.980 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:04:11 smithi007 ceph-mon[29459]: 3.17 scrub starts 2024-09-18T08:04:11.980 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:04:11 smithi007 ceph-mon[29459]: 3.17 scrub ok 2024-09-18T08:04:11.980 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:04:11 smithi007 ceph-mon[29459]: 5.12 scrub starts 2024-09-18T08:04:11.980 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:04:11 smithi007 ceph-mon[29459]: 5.12 scrub ok 2024-09-18T08:04:11.981 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:04:11 smithi007 ceph-mon[29459]: 5.15 scrub starts 2024-09-18T08:04:11.981 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:04:11 smithi007 ceph-mon[29459]: 5.15 scrub ok 2024-09-18T08:04:11.981 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:04:11 smithi007 ceph-mon[29459]: pgmap v47: 148 pgs: 148 active+clean; 431 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 688 B/s rd, 58 MiB/s wr, 926 op/s 2024-09-18T08:04:11.981 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:04:11 smithi007 ceph-mon[39249]: 3.17 scrub starts 2024-09-18T08:04:11.981 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:04:11 smithi007 ceph-mon[39249]: 3.17 scrub ok 2024-09-18T08:04:11.981 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:04:11 smithi007 ceph-mon[39249]: 5.12 scrub starts 2024-09-18T08:04:11.981 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:04:11 smithi007 ceph-mon[39249]: 5.12 scrub ok 2024-09-18T08:04:11.981 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:04:11 smithi007 ceph-mon[39249]: 5.15 scrub starts 2024-09-18T08:04:11.981 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:04:11 smithi007 ceph-mon[39249]: 5.15 scrub ok 2024-09-18T08:04:11.981 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:04:11 smithi007 ceph-mon[39249]: pgmap v47: 148 pgs: 148 active+clean; 431 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 688 B/s rd, 58 MiB/s wr, 926 op/s 2024-09-18T08:04:12.003 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:04:12.004 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:04:12.004 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7' 2024-09-18T08:04:12.029 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/779349ca-7593-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T08:04:12.055 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/779349ca-7593-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T08:04:12.105 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5 2024-09-18T08:04:12.138 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:04:12.141 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:04:11 smithi050 ceph-mon[32371]: 3.17 scrub starts 2024-09-18T08:04:12.141 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:04:11 smithi050 ceph-mon[32371]: 3.17 scrub ok 2024-09-18T08:04:12.141 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:04:11 smithi050 ceph-mon[32371]: 5.12 scrub starts 2024-09-18T08:04:12.141 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:04:11 smithi050 ceph-mon[32371]: 5.12 scrub ok 2024-09-18T08:04:12.141 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:04:11 smithi050 ceph-mon[32371]: 5.15 scrub starts 2024-09-18T08:04:12.141 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:04:11 smithi050 ceph-mon[32371]: 5.15 scrub ok 2024-09-18T08:04:12.141 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:04:11 smithi050 ceph-mon[32371]: pgmap v47: 148 pgs: 148 active+clean; 431 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 688 B/s rd, 58 MiB/s wr, 926 op/s 2024-09-18T08:04:12.142 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:04:12.142 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5' 2024-09-18T08:04:12.187 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid 779349ca-7593-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-18T08:04:12.243 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5 2024-09-18T08:04:12.274 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:04:12.274 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:04:12.275 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5' 2024-09-18T08:04:12.376 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6 2024-09-18T08:04:12.406 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:04:12.407 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:04:12.407 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6' 2024-09-18T08:04:12.508 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3 2024-09-18T08:04:12.555 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:04:12.556 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:04:12.557 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3' 2024-09-18T08:04:12.568 INFO:tasks.radosbench.radosbench.0.smithi007.stdout: 7 16 18441 18425 164.489 135.375 0.00704439 0.00607281 2024-09-18T08:04:12.658 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2 2024-09-18T08:04:12.719 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:04:12.720 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:04:12.720 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2' 2024-09-18T08:04:12.819 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:04:12 smithi050 ceph-mon[32371]: 5.c scrub starts 2024-09-18T08:04:12.819 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:04:12 smithi050 ceph-mon[32371]: 5.c scrub ok 2024-09-18T08:04:12.820 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:04:12 smithi050 ceph-mon[32371]: 6.7 scrub starts 2024-09-18T08:04:12.820 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:04:12 smithi050 ceph-mon[32371]: 6.7 scrub ok 2024-09-18T08:04:12.821 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2 2024-09-18T08:04:12.863 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:04:12.864 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:04:12.864 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2' 2024-09-18T08:04:12.930 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:04:12 smithi007 ceph-mon[29459]: 5.c scrub starts 2024-09-18T08:04:12.930 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:04:12 smithi007 ceph-mon[29459]: 5.c scrub ok 2024-09-18T08:04:12.930 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:04:12 smithi007 ceph-mon[29459]: 6.7 scrub starts 2024-09-18T08:04:12.930 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:04:12 smithi007 ceph-mon[29459]: 6.7 scrub ok 2024-09-18T08:04:12.931 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:04:12 smithi007 ceph-mon[39249]: 5.c scrub starts 2024-09-18T08:04:12.931 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:04:12 smithi007 ceph-mon[39249]: 5.c scrub ok 2024-09-18T08:04:12.931 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:04:12 smithi007 ceph-mon[39249]: 6.7 scrub starts 2024-09-18T08:04:12.931 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:04:12 smithi007 ceph-mon[39249]: 6.7 scrub ok 2024-09-18T08:04:12.965 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4 2024-09-18T08:04:12.998 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:04:12.998 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:04:12.999 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4' 2024-09-18T08:04:13.100 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2 2024-09-18T08:04:13.142 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:04:13.142 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:04:13.143 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2' 2024-09-18T08:04:13.243 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3 2024-09-18T08:04:13.285 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:04:13.286 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:04:13.286 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3' 2024-09-18T08:04:13.387 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7 2024-09-18T08:04:13.419 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:04:13.420 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:04:13.420 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7' 2024-09-18T08:04:13.521 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6 2024-09-18T08:04:13.549 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:04:13.550 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:04:13.550 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6' 2024-09-18T08:04:13.569 INFO:tasks.radosbench.radosbench.0.smithi007.stdout: 8 16 20658 20642 161.22 138.562 0.00737537 0.00619622 2024-09-18T08:04:13.651 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1 2024-09-18T08:04:13.698 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:04:13.699 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:04:13.699 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1' 2024-09-18T08:04:13.800 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2 2024-09-18T08:04:13.843 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:04:13.844 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:04:13.844 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2' 2024-09-18T08:04:13.870 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/779349ca-7593-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T08:04:13.910 INFO:teuthology.orchestra.run.smithi007.stdout:ERROR: (22) Invalid argument 2024-09-18T08:04:13.910 INFO:teuthology.orchestra.run.smithi007.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-18T08:04:13.931 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid 779349ca-7593-11ef-bceb-c7b262605968 -- ceph osd unset nodeep-scrub 2024-09-18T08:04:13.945 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3 2024-09-18T08:04:14.023 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:04:14.024 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:04:14.024 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3' 2024-09-18T08:04:14.114 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:04:14 smithi007 ceph-mon[29459]: 4.9 scrub starts 2024-09-18T08:04:14.114 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:04:14 smithi007 ceph-mon[29459]: 4.9 scrub ok 2024-09-18T08:04:14.115 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:04:14 smithi007 ceph-mon[29459]: 4.4 scrub starts 2024-09-18T08:04:14.115 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:04:14 smithi007 ceph-mon[29459]: 4.4 scrub ok 2024-09-18T08:04:14.115 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:04:14 smithi007 ceph-mon[29459]: 5.10 scrub starts 2024-09-18T08:04:14.115 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:04:14 smithi007 ceph-mon[29459]: 5.10 scrub ok 2024-09-18T08:04:14.115 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:04:14 smithi007 ceph-mon[29459]: 4.10 scrub starts 2024-09-18T08:04:14.115 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:04:14 smithi007 ceph-mon[29459]: 4.10 scrub ok 2024-09-18T08:04:14.115 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:04:14 smithi007 ceph-mon[29459]: pgmap v48: 148 pgs: 148 active+clean; 924 MiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 639 B/s rd, 115 MiB/s wr, 1.84k op/s 2024-09-18T08:04:14.115 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:04:14 smithi007 ceph-mon[29459]: from='mgr.24470 172.21.15.7:0/2594390261' entity='mgr.y' 2024-09-18T08:04:14.115 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:04:14 smithi007 ceph-mon[29459]: from='mgr.24470 172.21.15.7:0/2594390261' entity='mgr.y' 2024-09-18T08:04:14.116 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:04:14 smithi007 ceph-mon[29459]: from='client.? 172.21.15.7:0/4087203273' entity='client.admin' cmd={"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num", "val": "1"} : dispatch 2024-09-18T08:04:14.116 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:04:14 smithi007 ceph-mon[29459]: 5.2 scrub starts 2024-09-18T08:04:14.116 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:04:14 smithi007 ceph-mon[29459]: 5.2 scrub ok 2024-09-18T08:04:14.116 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:04:14 smithi007 ceph-mon[39249]: 4.9 scrub starts 2024-09-18T08:04:14.116 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:04:14 smithi007 ceph-mon[39249]: 4.9 scrub ok 2024-09-18T08:04:14.116 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:04:14 smithi007 ceph-mon[39249]: 4.4 scrub starts 2024-09-18T08:04:14.116 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:04:14 smithi007 ceph-mon[39249]: 4.4 scrub ok 2024-09-18T08:04:14.116 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:04:14 smithi007 ceph-mon[39249]: 5.10 scrub starts 2024-09-18T08:04:14.117 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:04:14 smithi007 ceph-mon[39249]: 5.10 scrub ok 2024-09-18T08:04:14.117 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:04:14 smithi007 ceph-mon[39249]: 4.10 scrub starts 2024-09-18T08:04:14.117 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:04:14 smithi007 ceph-mon[39249]: 4.10 scrub ok 2024-09-18T08:04:14.117 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:04:14 smithi007 ceph-mon[39249]: pgmap v48: 148 pgs: 148 active+clean; 924 MiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 639 B/s rd, 115 MiB/s wr, 1.84k op/s 2024-09-18T08:04:14.117 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:04:14 smithi007 ceph-mon[39249]: from='mgr.24470 172.21.15.7:0/2594390261' entity='mgr.y' 2024-09-18T08:04:14.117 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:04:14 smithi007 ceph-mon[39249]: from='mgr.24470 172.21.15.7:0/2594390261' entity='mgr.y' 2024-09-18T08:04:14.117 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:04:14 smithi007 ceph-mon[39249]: from='client.? 172.21.15.7:0/4087203273' entity='client.admin' cmd={"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num", "val": "1"} : dispatch 2024-09-18T08:04:14.117 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:04:14 smithi007 ceph-mon[39249]: 5.2 scrub starts 2024-09-18T08:04:14.117 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:04:14 smithi007 ceph-mon[39249]: 5.2 scrub ok 2024-09-18T08:04:14.118 INFO:teuthology.orchestra.run.smithi007.stderr:set pool 1 pgp_num to 1 2024-09-18T08:04:14.125 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3 2024-09-18T08:04:14.177 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:04:14.178 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:04:14.179 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3' 2024-09-18T08:04:14.279 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3 2024-09-18T08:04:14.325 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:04:14.325 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:04:14.326 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3' 2024-09-18T08:04:14.406 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:04:14 smithi050 ceph-mon[32371]: 4.9 scrub starts 2024-09-18T08:04:14.406 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:04:14 smithi050 ceph-mon[32371]: 4.9 scrub ok 2024-09-18T08:04:14.406 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:04:14 smithi050 ceph-mon[32371]: 4.4 scrub starts 2024-09-18T08:04:14.406 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:04:14 smithi050 ceph-mon[32371]: 4.4 scrub ok 2024-09-18T08:04:14.406 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:04:14 smithi050 ceph-mon[32371]: 5.10 scrub starts 2024-09-18T08:04:14.406 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:04:14 smithi050 ceph-mon[32371]: 5.10 scrub ok 2024-09-18T08:04:14.406 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:04:14 smithi050 ceph-mon[32371]: 4.10 scrub starts 2024-09-18T08:04:14.406 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:04:14 smithi050 ceph-mon[32371]: 4.10 scrub ok 2024-09-18T08:04:14.407 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:04:14 smithi050 ceph-mon[32371]: pgmap v48: 148 pgs: 148 active+clean; 924 MiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 639 B/s rd, 115 MiB/s wr, 1.84k op/s 2024-09-18T08:04:14.407 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:04:14 smithi050 ceph-mon[32371]: from='mgr.24470 172.21.15.7:0/2594390261' entity='mgr.y' 2024-09-18T08:04:14.407 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:04:14 smithi050 ceph-mon[32371]: from='mgr.24470 172.21.15.7:0/2594390261' entity='mgr.y' 2024-09-18T08:04:14.407 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:04:14 smithi050 ceph-mon[32371]: from='client.? 172.21.15.7:0/4087203273' entity='client.admin' cmd={"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num", "val": "1"} : dispatch 2024-09-18T08:04:14.407 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:04:14 smithi050 ceph-mon[32371]: 5.2 scrub starts 2024-09-18T08:04:14.407 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:04:14 smithi050 ceph-mon[32371]: 5.2 scrub ok 2024-09-18T08:04:14.426 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6 2024-09-18T08:04:14.455 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:04:14.456 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:04:14.456 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6' 2024-09-18T08:04:14.557 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4 2024-09-18T08:04:14.569 INFO:tasks.radosbench.radosbench.0.smithi007.stdout: 9 15 22743 22728 157.79 130.375 0.00508238 0.00633185 2024-09-18T08:04:14.587 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:04:14.587 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:04:14.588 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4' 2024-09-18T08:04:14.689 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2 2024-09-18T08:04:14.738 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:04:14.739 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:04:14.739 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2' 2024-09-18T08:04:14.840 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4 2024-09-18T08:04:14.872 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:04:14.873 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:04:14.874 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4' 2024-09-18T08:04:14.974 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4 2024-09-18T08:04:15.005 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:04:15.006 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:04:15.007 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4' 2024-09-18T08:04:15.107 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6 2024-09-18T08:04:15.138 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:04:15.138 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:04:15.139 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6' 2024-09-18T08:04:15.239 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1 2024-09-18T08:04:15.270 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:04:15.271 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:04:15.271 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1' 2024-09-18T08:04:15.372 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4 2024-09-18T08:04:15.403 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:04:15.404 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:04:15.405 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4' 2024-09-18T08:04:15.405 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:04:15 smithi050 ceph-mon[32371]: 5.16 scrub starts 2024-09-18T08:04:15.406 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:04:15 smithi050 ceph-mon[32371]: 5.16 scrub ok 2024-09-18T08:04:15.406 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:04:15 smithi050 ceph-mon[32371]: from='client.? 172.21.15.7:0/4087203273' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num", "val": "1"}]': finished 2024-09-18T08:04:15.406 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:04:15 smithi050 ceph-mon[32371]: osdmap e91: 8 total, 8 up, 8 in 2024-09-18T08:04:15.406 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:04:15 smithi050 ceph-mon[32371]: 4.5 scrub starts 2024-09-18T08:04:15.406 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:04:15 smithi050 ceph-mon[32371]: 4.5 scrub ok 2024-09-18T08:04:15.480 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:04:15 smithi007 ceph-mon[29459]: 5.16 scrub starts 2024-09-18T08:04:15.481 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:04:15 smithi007 ceph-mon[29459]: 5.16 scrub ok 2024-09-18T08:04:15.481 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:04:15 smithi007 ceph-mon[29459]: from='client.? 172.21.15.7:0/4087203273' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num", "val": "1"}]': finished 2024-09-18T08:04:15.481 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:04:15 smithi007 ceph-mon[29459]: osdmap e91: 8 total, 8 up, 8 in 2024-09-18T08:04:15.481 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:04:15 smithi007 ceph-mon[29459]: 4.5 scrub starts 2024-09-18T08:04:15.481 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:04:15 smithi007 ceph-mon[29459]: 4.5 scrub ok 2024-09-18T08:04:15.481 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:04:15 smithi007 ceph-mon[39249]: 5.16 scrub starts 2024-09-18T08:04:15.481 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:04:15 smithi007 ceph-mon[39249]: 5.16 scrub ok 2024-09-18T08:04:15.482 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:04:15 smithi007 ceph-mon[39249]: from='client.? 172.21.15.7:0/4087203273' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num", "val": "1"}]': finished 2024-09-18T08:04:15.482 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:04:15 smithi007 ceph-mon[39249]: osdmap e91: 8 total, 8 up, 8 in 2024-09-18T08:04:15.482 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:04:15 smithi007 ceph-mon[39249]: 4.5 scrub starts 2024-09-18T08:04:15.482 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:04:15 smithi007 ceph-mon[39249]: 4.5 scrub ok 2024-09-18T08:04:15.505 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0 2024-09-18T08:04:15.544 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:04:15.546 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:04:15.546 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0' 2024-09-18T08:04:15.569 INFO:tasks.radosbench.radosbench.0.smithi007.stdout: 10 16 25012 24996 156.185 141.75 0.00179429 0.00639367 2024-09-18T08:04:15.647 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1 2024-09-18T08:04:15.687 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:04:15.688 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:04:15.688 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1' 2024-09-18T08:04:15.789 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0 2024-09-18T08:04:15.830 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:04:15.831 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:04:15.831 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0' 2024-09-18T08:04:15.932 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0 2024-09-18T08:04:15.975 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:04:15.975 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:04:15.976 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0' 2024-09-18T08:04:16.077 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0 2024-09-18T08:04:16.099 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:04:16.100 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:04:16.100 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0' 2024-09-18T08:04:16.201 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3 2024-09-18T08:04:16.242 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:04:16.243 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:04:16.243 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3' 2024-09-18T08:04:16.344 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2 2024-09-18T08:04:16.366 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:04:16.367 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:04:16.367 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2' 2024-09-18T08:04:16.406 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:04:16 smithi050 ceph-mon[32371]: 5.1 scrub starts 2024-09-18T08:04:16.406 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:04:16 smithi050 ceph-mon[32371]: 5.1 scrub ok 2024-09-18T08:04:16.406 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:04:16 smithi050 ceph-mon[32371]: 5.6 scrub starts 2024-09-18T08:04:16.406 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:04:16 smithi050 ceph-mon[32371]: 5.6 scrub ok 2024-09-18T08:04:16.407 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:04:16 smithi050 ceph-mon[32371]: 4.0 scrub starts 2024-09-18T08:04:16.407 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:04:16 smithi050 ceph-mon[32371]: 4.0 scrub ok 2024-09-18T08:04:16.407 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:04:16 smithi050 ceph-mon[32371]: pgmap v50: 148 pgs: 148 active+clean; 1.2 GiB data, 3.7 GiB used, 712 GiB / 715 GiB avail; 1.2 KiB/s rd, 152 MiB/s wr, 2.44k op/s 2024-09-18T08:04:16.467 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2 2024-09-18T08:04:16.481 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:04:16 smithi007 ceph-mon[29459]: 5.1 scrub starts 2024-09-18T08:04:16.481 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:04:16 smithi007 ceph-mon[29459]: 5.1 scrub ok 2024-09-18T08:04:16.481 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:04:16 smithi007 ceph-mon[29459]: 5.6 scrub starts 2024-09-18T08:04:16.481 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:04:16 smithi007 ceph-mon[29459]: 5.6 scrub ok 2024-09-18T08:04:16.481 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:04:16 smithi007 ceph-mon[29459]: 4.0 scrub starts 2024-09-18T08:04:16.481 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:04:16 smithi007 ceph-mon[29459]: 4.0 scrub ok 2024-09-18T08:04:16.481 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:04:16 smithi007 ceph-mon[29459]: pgmap v50: 148 pgs: 148 active+clean; 1.2 GiB data, 3.7 GiB used, 712 GiB / 715 GiB avail; 1.2 KiB/s rd, 152 MiB/s wr, 2.44k op/s 2024-09-18T08:04:16.481 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:04:16 smithi007 ceph-mon[39249]: 5.1 scrub starts 2024-09-18T08:04:16.482 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:04:16 smithi007 ceph-mon[39249]: 5.1 scrub ok 2024-09-18T08:04:16.482 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:04:16 smithi007 ceph-mon[39249]: 5.6 scrub starts 2024-09-18T08:04:16.482 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:04:16 smithi007 ceph-mon[39249]: 5.6 scrub ok 2024-09-18T08:04:16.482 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:04:16 smithi007 ceph-mon[39249]: 4.0 scrub starts 2024-09-18T08:04:16.482 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:04:16 smithi007 ceph-mon[39249]: 4.0 scrub ok 2024-09-18T08:04:16.482 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:04:16 smithi007 ceph-mon[39249]: pgmap v50: 148 pgs: 148 active+clean; 1.2 GiB data, 3.7 GiB used, 712 GiB / 715 GiB avail; 1.2 KiB/s rd, 152 MiB/s wr, 2.44k op/s 2024-09-18T08:04:16.507 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:04:16.508 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:04:16.509 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2' 2024-09-18T08:04:16.569 INFO:tasks.radosbench.radosbench.0.smithi007.stdout: 11 16 26815 26799 152.23 112.688 0.00628266 0.00656334 2024-09-18T08:04:16.609 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7 2024-09-18T08:04:16.635 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:04:16.636 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:04:16.636 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7' 2024-09-18T08:04:16.737 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5 2024-09-18T08:04:16.765 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:04:16.766 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:04:16.766 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5' 2024-09-18T08:04:16.867 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2 2024-09-18T08:04:16.909 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:04:16.909 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:04:16.909 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2' 2024-09-18T08:04:16.980 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Sep 18 08:04:16 smithi007 ceph-779349ca-7593-11ef-bceb-c7b262605968-mgr-y[29688]: ::ffff:172.21.15.50 - - [18/Sep/2024:08:04:16] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-09-18T08:04:17.011 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4 2024-09-18T08:04:17.036 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:04:17.037 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:04:17.037 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4' 2024-09-18T08:04:17.138 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2 2024-09-18T08:04:17.181 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:04:17.182 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:04:17.182 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2' 2024-09-18T08:04:17.283 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4 2024-09-18T08:04:17.314 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:04:17.315 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:04:17.315 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4' 2024-09-18T08:04:17.406 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:04:17 smithi050 ceph-mon[32371]: 3.d scrub starts 2024-09-18T08:04:17.406 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:04:17 smithi050 ceph-mon[32371]: 3.d scrub ok 2024-09-18T08:04:17.406 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:04:17 smithi050 ceph-mon[32371]: 3.10 scrub starts 2024-09-18T08:04:17.406 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:04:17 smithi050 ceph-mon[32371]: 3.10 scrub ok 2024-09-18T08:04:17.406 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:04:17 smithi050 ceph-mon[32371]: from='mgr.24470 172.21.15.7:0/2594390261' entity='mgr.y' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.7", "id": [1, 2]} : dispatch 2024-09-18T08:04:17.406 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:04:17 smithi050 ceph-mon[32371]: from='mgr.24470 172.21.15.7:0/2594390261' entity='mgr.y' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.14", "id": [1, 2]} : dispatch 2024-09-18T08:04:17.406 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:04:17 smithi050 ceph-mon[32371]: from='mgr.24470 172.21.15.7:0/2594390261' entity='mgr.y' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1c", "id": [1, 5]} : dispatch 2024-09-18T08:04:17.417 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7 2024-09-18T08:04:17.440 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:04:17.440 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:04:17.440 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7' 2024-09-18T08:04:17.480 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:04:17 smithi007 ceph-mon[29459]: 3.d scrub starts 2024-09-18T08:04:17.481 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:04:17 smithi007 ceph-mon[29459]: 3.d scrub ok 2024-09-18T08:04:17.481 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:04:17 smithi007 ceph-mon[29459]: 3.10 scrub starts 2024-09-18T08:04:17.481 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:04:17 smithi007 ceph-mon[29459]: 3.10 scrub ok 2024-09-18T08:04:17.481 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:04:17 smithi007 ceph-mon[29459]: from='mgr.24470 172.21.15.7:0/2594390261' entity='mgr.y' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.7", "id": [1, 2]} : dispatch 2024-09-18T08:04:17.481 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:04:17 smithi007 ceph-mon[29459]: from='mgr.24470 172.21.15.7:0/2594390261' entity='mgr.y' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.14", "id": [1, 2]} : dispatch 2024-09-18T08:04:17.481 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:04:17 smithi007 ceph-mon[29459]: from='mgr.24470 172.21.15.7:0/2594390261' entity='mgr.y' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1c", "id": [1, 5]} : dispatch 2024-09-18T08:04:17.481 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:04:17 smithi007 ceph-mon[39249]: 3.d scrub starts 2024-09-18T08:04:17.481 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:04:17 smithi007 ceph-mon[39249]: 3.d scrub ok 2024-09-18T08:04:17.481 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:04:17 smithi007 ceph-mon[39249]: 3.10 scrub starts 2024-09-18T08:04:17.482 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:04:17 smithi007 ceph-mon[39249]: 3.10 scrub ok 2024-09-18T08:04:17.482 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:04:17 smithi007 ceph-mon[39249]: from='mgr.24470 172.21.15.7:0/2594390261' entity='mgr.y' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.7", "id": [1, 2]} : dispatch 2024-09-18T08:04:17.482 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:04:17 smithi007 ceph-mon[39249]: from='mgr.24470 172.21.15.7:0/2594390261' entity='mgr.y' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.14", "id": [1, 2]} : dispatch 2024-09-18T08:04:17.482 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:04:17 smithi007 ceph-mon[39249]: from='mgr.24470 172.21.15.7:0/2594390261' entity='mgr.y' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1c", "id": [1, 5]} : dispatch 2024-09-18T08:04:17.541 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0 2024-09-18T08:04:17.570 INFO:tasks.radosbench.radosbench.0.smithi007.stdout: 12 15 28226 28211 146.899 88.25 0.00653807 0.00680005 2024-09-18T08:04:17.573 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:04:17.574 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:04:17.574 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0' 2024-09-18T08:04:17.675 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5 2024-09-18T08:04:17.709 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:04:17.710 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:04:17.710 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5' 2024-09-18T08:04:17.811 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7 2024-09-18T08:04:17.841 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:04:17.842 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:04:17.843 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7' 2024-09-18T08:04:17.944 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2 2024-09-18T08:04:17.966 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:04:17.967 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:04:17.967 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2' 2024-09-18T08:04:18.068 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0 2024-09-18T08:04:18.108 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:04:18.109 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:04:18.110 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0' 2024-09-18T08:04:18.211 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0 2024-09-18T08:04:18.249 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:04:18.250 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:04:18.250 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0' 2024-09-18T08:04:18.351 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5 2024-09-18T08:04:18.381 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:04:18.382 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:04:18.383 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5' 2024-09-18T08:04:18.406 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:04:18 smithi050 ceph-mon[32371]: 6.2 scrub starts 2024-09-18T08:04:18.406 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:04:18 smithi050 ceph-mon[32371]: 6.2 scrub ok 2024-09-18T08:04:18.406 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:04:18 smithi050 ceph-mon[32371]: 6.12 scrub starts 2024-09-18T08:04:18.406 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:04:18 smithi050 ceph-mon[32371]: 6.12 scrub ok 2024-09-18T08:04:18.406 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:04:18 smithi050 ceph-mon[32371]: 4.14 scrub starts 2024-09-18T08:04:18.406 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:04:18 smithi050 ceph-mon[32371]: 4.14 scrub ok 2024-09-18T08:04:18.406 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:04:18 smithi050 ceph-mon[32371]: pgmap v51: 148 pgs: 148 active+clean; 1.2 GiB data, 3.7 GiB used, 712 GiB / 715 GiB avail; 597 B/s rd, 92 MiB/s wr, 1.47k op/s 2024-09-18T08:04:18.406 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:04:18 smithi050 ceph-mon[32371]: from='mgr.24470 172.21.15.7:0/2594390261' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.7", "id": [1, 2]}]': finished 2024-09-18T08:04:18.406 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:04:18 smithi050 ceph-mon[32371]: from='mgr.24470 172.21.15.7:0/2594390261' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.14", "id": [1, 2]}]': finished 2024-09-18T08:04:18.407 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:04:18 smithi050 ceph-mon[32371]: from='mgr.24470 172.21.15.7:0/2594390261' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1c", "id": [1, 5]}]': finished 2024-09-18T08:04:18.407 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:04:18 smithi050 ceph-mon[32371]: osdmap e92: 8 total, 8 up, 8 in 2024-09-18T08:04:18.407 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:04:18 smithi050 ceph-mon[32371]: from='mgr.24470 172.21.15.7:0/2594390261' entity='mgr.y' 2024-09-18T08:04:18.407 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:04:18 smithi050 ceph-mon[32371]: from='mgr.24470 172.21.15.7:0/2594390261' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-18T08:04:18.481 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:04:18 smithi007 ceph-mon[29459]: 6.2 scrub starts 2024-09-18T08:04:18.481 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:04:18 smithi007 ceph-mon[29459]: 6.2 scrub ok 2024-09-18T08:04:18.481 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:04:18 smithi007 ceph-mon[29459]: 6.12 scrub starts 2024-09-18T08:04:18.481 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:04:18 smithi007 ceph-mon[29459]: 6.12 scrub ok 2024-09-18T08:04:18.481 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:04:18 smithi007 ceph-mon[29459]: 4.14 scrub starts 2024-09-18T08:04:18.481 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:04:18 smithi007 ceph-mon[29459]: 4.14 scrub ok 2024-09-18T08:04:18.481 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:04:18 smithi007 ceph-mon[29459]: pgmap v51: 148 pgs: 148 active+clean; 1.2 GiB data, 3.7 GiB used, 712 GiB / 715 GiB avail; 597 B/s rd, 92 MiB/s wr, 1.47k op/s 2024-09-18T08:04:18.481 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:04:18 smithi007 ceph-mon[29459]: from='mgr.24470 172.21.15.7:0/2594390261' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.7", "id": [1, 2]}]': finished 2024-09-18T08:04:18.481 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:04:18 smithi007 ceph-mon[29459]: from='mgr.24470 172.21.15.7:0/2594390261' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.14", "id": [1, 2]}]': finished 2024-09-18T08:04:18.481 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:04:18 smithi007 ceph-mon[29459]: from='mgr.24470 172.21.15.7:0/2594390261' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1c", "id": [1, 5]}]': finished 2024-09-18T08:04:18.481 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:04:18 smithi007 ceph-mon[29459]: osdmap e92: 8 total, 8 up, 8 in 2024-09-18T08:04:18.481 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:04:18 smithi007 ceph-mon[29459]: from='mgr.24470 172.21.15.7:0/2594390261' entity='mgr.y' 2024-09-18T08:04:18.482 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:04:18 smithi007 ceph-mon[29459]: from='mgr.24470 172.21.15.7:0/2594390261' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-18T08:04:18.482 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:04:18 smithi007 ceph-mon[39249]: 6.2 scrub starts 2024-09-18T08:04:18.482 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:04:18 smithi007 ceph-mon[39249]: 6.2 scrub ok 2024-09-18T08:04:18.482 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:04:18 smithi007 ceph-mon[39249]: 6.12 scrub starts 2024-09-18T08:04:18.482 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:04:18 smithi007 ceph-mon[39249]: 6.12 scrub ok 2024-09-18T08:04:18.482 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:04:18 smithi007 ceph-mon[39249]: 4.14 scrub starts 2024-09-18T08:04:18.482 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:04:18 smithi007 ceph-mon[39249]: 4.14 scrub ok 2024-09-18T08:04:18.482 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:04:18 smithi007 ceph-mon[39249]: pgmap v51: 148 pgs: 148 active+clean; 1.2 GiB data, 3.7 GiB used, 712 GiB / 715 GiB avail; 597 B/s rd, 92 MiB/s wr, 1.47k op/s 2024-09-18T08:04:18.482 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:04:18 smithi007 ceph-mon[39249]: from='mgr.24470 172.21.15.7:0/2594390261' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.7", "id": [1, 2]}]': finished 2024-09-18T08:04:18.482 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:04:18 smithi007 ceph-mon[39249]: from='mgr.24470 172.21.15.7:0/2594390261' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.14", "id": [1, 2]}]': finished 2024-09-18T08:04:18.482 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:04:18 smithi007 ceph-mon[39249]: from='mgr.24470 172.21.15.7:0/2594390261' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1c", "id": [1, 5]}]': finished 2024-09-18T08:04:18.482 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:04:18 smithi007 ceph-mon[39249]: osdmap e92: 8 total, 8 up, 8 in 2024-09-18T08:04:18.483 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:04:18 smithi007 ceph-mon[39249]: from='mgr.24470 172.21.15.7:0/2594390261' entity='mgr.y' 2024-09-18T08:04:18.483 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:04:18 smithi007 ceph-mon[39249]: from='mgr.24470 172.21.15.7:0/2594390261' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-18T08:04:18.484 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0 2024-09-18T08:04:18.522 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:04:18.523 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:04:18.524 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0' 2024-09-18T08:04:18.570 INFO:tasks.radosbench.radosbench.0.smithi007.stdout: 13 16 30161 30145 144.896 120.875 0.0112903 0.00689551 2024-09-18T08:04:18.624 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3 2024-09-18T08:04:18.664 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:04:18.665 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:04:18.665 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3' 2024-09-18T08:04:18.766 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5 2024-09-18T08:04:18.798 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:04:18.799 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:04:18.800 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5' 2024-09-18T08:04:18.901 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1 2024-09-18T08:04:18.929 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:04:18.929 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:04:18.930 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1' 2024-09-18T08:04:19.030 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3 2024-09-18T08:04:19.073 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:04:19.074 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:04:19.074 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3' 2024-09-18T08:04:19.138 INFO:journalctl@ceph.iscsi.iscsi.a.smithi050.stdout:Sep 18 08:04:18 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-iscsi-iscsi-a[57782]: debug there is no tcmu-runner data available 2024-09-18T08:04:19.174 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2 2024-09-18T08:04:19.208 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:04:19.208 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:04:19.209 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2' 2024-09-18T08:04:19.310 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7 2024-09-18T08:04:19.340 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:04:19.341 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:04:19.342 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7' 2024-09-18T08:04:19.406 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:04:19 smithi050 ceph-mon[32371]: 6.8 scrub starts 2024-09-18T08:04:19.406 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:04:19 smithi050 ceph-mon[32371]: 6.8 scrub ok 2024-09-18T08:04:19.406 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:04:19 smithi050 ceph-mon[32371]: osdmap e93: 8 total, 8 up, 8 in 2024-09-18T08:04:19.406 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:04:19 smithi050 ceph-mon[32371]: osdmap e94: 8 total, 8 up, 8 in 2024-09-18T08:04:19.443 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0 2024-09-18T08:04:19.480 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:04:19 smithi007 ceph-mon[29459]: 6.8 scrub starts 2024-09-18T08:04:19.480 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:04:19 smithi007 ceph-mon[29459]: 6.8 scrub ok 2024-09-18T08:04:19.480 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:04:19 smithi007 ceph-mon[29459]: osdmap e93: 8 total, 8 up, 8 in 2024-09-18T08:04:19.480 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:04:19 smithi007 ceph-mon[29459]: osdmap e94: 8 total, 8 up, 8 in 2024-09-18T08:04:19.481 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:04:19 smithi007 ceph-mon[39249]: 6.8 scrub starts 2024-09-18T08:04:19.481 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:04:19 smithi007 ceph-mon[39249]: 6.8 scrub ok 2024-09-18T08:04:19.481 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:04:19 smithi007 ceph-mon[39249]: osdmap e93: 8 total, 8 up, 8 in 2024-09-18T08:04:19.481 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:04:19 smithi007 ceph-mon[39249]: osdmap e94: 8 total, 8 up, 8 in 2024-09-18T08:04:19.483 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:04:19.484 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:04:19.484 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0' 2024-09-18T08:04:19.570 INFO:tasks.radosbench.radosbench.0.smithi007.stdout: 14 15 31852 31837 142.1 105.75 0.00743517 0.00703093 2024-09-18T08:04:19.585 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3 2024-09-18T08:04:19.628 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:04:19.629 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:04:19.629 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3' 2024-09-18T08:04:19.731 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4 2024-09-18T08:04:19.763 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:04:19.764 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:04:19.764 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4' 2024-09-18T08:04:19.865 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7 2024-09-18T08:04:19.896 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:04:19.897 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:04:19.897 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7' 2024-09-18T08:04:19.998 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7 2024-09-18T08:04:20.020 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:04:20.021 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:04:20.021 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7' 2024-09-18T08:04:20.122 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1 2024-09-18T08:04:20.161 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:04:20.162 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:04:20.162 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1' 2024-09-18T08:04:20.263 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7 2024-09-18T08:04:20.294 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:04:20.294 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:04:20.295 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7' 2024-09-18T08:04:20.395 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3 2024-09-18T08:04:20.406 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:04:20 smithi050 ceph-mon[32371]: 6.0 scrub starts 2024-09-18T08:04:20.406 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:04:20 smithi050 ceph-mon[32371]: 6.0 scrub ok 2024-09-18T08:04:20.406 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:04:20 smithi050 ceph-mon[32371]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-18T08:04:20.406 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:04:20 smithi050 ceph-mon[32371]: 5.4 scrub starts 2024-09-18T08:04:20.406 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:04:20 smithi050 ceph-mon[32371]: 5.4 scrub ok 2024-09-18T08:04:20.406 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:04:20 smithi050 ceph-mon[32371]: 6.11 scrub starts 2024-09-18T08:04:20.406 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:04:20 smithi050 ceph-mon[32371]: 6.11 scrub ok 2024-09-18T08:04:20.406 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:04:20 smithi050 ceph-mon[32371]: pgmap v55: 148 pgs: 3 peering, 145 active+clean; 1.8 GiB data, 5.5 GiB used, 710 GiB / 715 GiB avail; 824 B/s rd, 122 MiB/s wr, 1.95k op/s 2024-09-18T08:04:20.406 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:04:20 smithi050 ceph-mon[32371]: Health check failed: Reduced data availability: 1 pg peering (PG_AVAILABILITY) 2024-09-18T08:04:20.438 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:04:20.439 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:04:20.439 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3' 2024-09-18T08:04:20.481 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:04:20 smithi007 ceph-mon[29459]: 6.0 scrub starts 2024-09-18T08:04:20.481 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:04:20 smithi007 ceph-mon[29459]: 6.0 scrub ok 2024-09-18T08:04:20.481 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:04:20 smithi007 ceph-mon[29459]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-18T08:04:20.481 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:04:20 smithi007 ceph-mon[29459]: 5.4 scrub starts 2024-09-18T08:04:20.481 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:04:20 smithi007 ceph-mon[29459]: 5.4 scrub ok 2024-09-18T08:04:20.481 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:04:20 smithi007 ceph-mon[29459]: 6.11 scrub starts 2024-09-18T08:04:20.481 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:04:20 smithi007 ceph-mon[29459]: 6.11 scrub ok 2024-09-18T08:04:20.481 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:04:20 smithi007 ceph-mon[29459]: pgmap v55: 148 pgs: 3 peering, 145 active+clean; 1.8 GiB data, 5.5 GiB used, 710 GiB / 715 GiB avail; 824 B/s rd, 122 MiB/s wr, 1.95k op/s 2024-09-18T08:04:20.481 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:04:20 smithi007 ceph-mon[29459]: Health check failed: Reduced data availability: 1 pg peering (PG_AVAILABILITY) 2024-09-18T08:04:20.482 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:04:20 smithi007 ceph-mon[39249]: 6.0 scrub starts 2024-09-18T08:04:20.482 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:04:20 smithi007 ceph-mon[39249]: 6.0 scrub ok 2024-09-18T08:04:20.482 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:04:20 smithi007 ceph-mon[39249]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-18T08:04:20.482 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:04:20 smithi007 ceph-mon[39249]: 5.4 scrub starts 2024-09-18T08:04:20.482 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:04:20 smithi007 ceph-mon[39249]: 5.4 scrub ok 2024-09-18T08:04:20.482 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:04:20 smithi007 ceph-mon[39249]: 6.11 scrub starts 2024-09-18T08:04:20.482 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:04:20 smithi007 ceph-mon[39249]: 6.11 scrub ok 2024-09-18T08:04:20.482 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:04:20 smithi007 ceph-mon[39249]: pgmap v55: 148 pgs: 3 peering, 145 active+clean; 1.8 GiB data, 5.5 GiB used, 710 GiB / 715 GiB avail; 824 B/s rd, 122 MiB/s wr, 1.95k op/s 2024-09-18T08:04:20.482 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:04:20 smithi007 ceph-mon[39249]: Health check failed: Reduced data availability: 1 pg peering (PG_AVAILABILITY) 2024-09-18T08:04:20.541 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3 2024-09-18T08:04:20.570 INFO:tasks.radosbench.radosbench.0.smithi007.stdout: 15 16 33776 33760 140.638 120.188 0.00459782 0.00710363 2024-09-18T08:04:20.584 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:04:20.584 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:04:20.585 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3' 2024-09-18T08:04:20.686 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3 2024-09-18T08:04:20.730 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:04:20.730 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:04:20.731 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3' 2024-09-18T08:04:20.832 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5 2024-09-18T08:04:20.860 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:04:20.861 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:04:20.862 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5' 2024-09-18T08:04:20.963 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3 2024-09-18T08:04:21.013 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:04:21.014 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:04:21.015 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3' 2024-09-18T08:04:21.015 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid 779349ca-7593-11ef-bceb-c7b262605968 -- ceph daemon osd.2 dump_ops_in_flight 2024-09-18T08:04:21.115 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2 2024-09-18T08:04:21.160 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:04:21 smithi007 ceph-mon[29459]: 4.1c scrub starts 2024-09-18T08:04:21.160 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:04:21 smithi007 ceph-mon[29459]: 4.1c scrub ok 2024-09-18T08:04:21.170 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:04:21.171 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:04:21.171 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2' 2024-09-18T08:04:21.248 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/779349ca-7593-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T08:04:21.272 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3 2024-09-18T08:04:21.287 INFO:teuthology.orchestra.run.smithi007.stdout:osd.0: {} 2024-09-18T08:04:21.288 INFO:teuthology.orchestra.run.smithi007.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-18T08:04:21.318 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:04:21.319 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:04:21.319 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3' 2024-09-18T08:04:21.325 INFO:teuthology.orchestra.run.smithi007.stdout:osd.1: {} 2024-09-18T08:04:21.326 INFO:teuthology.orchestra.run.smithi007.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-18T08:04:21.364 INFO:teuthology.orchestra.run.smithi007.stdout:osd.2: {} 2024-09-18T08:04:21.364 INFO:teuthology.orchestra.run.smithi007.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-18T08:04:21.403 INFO:teuthology.orchestra.run.smithi007.stdout:osd.3: {} 2024-09-18T08:04:21.403 INFO:teuthology.orchestra.run.smithi007.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-18T08:04:21.420 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0 2024-09-18T08:04:21.431 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:04:21 smithi007 ceph-mon[39249]: 4.1c scrub starts 2024-09-18T08:04:21.431 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:04:21 smithi007 ceph-mon[39249]: 4.1c scrub ok 2024-09-18T08:04:21.431 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:04:21 smithi007 ceph-mon[39249]: 4.15 scrub starts 2024-09-18T08:04:21.432 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:04:21 smithi007 ceph-mon[39249]: 4.15 scrub ok 2024-09-18T08:04:21.432 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:04:21 smithi007 ceph-mon[39249]: 4.a scrub starts 2024-09-18T08:04:21.432 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:04:21 smithi007 ceph-mon[39249]: 4.a scrub ok 2024-09-18T08:04:21.432 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:04:21 smithi007 ceph-mon[29459]: 4.15 scrub starts 2024-09-18T08:04:21.432 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:04:21 smithi007 ceph-mon[29459]: 4.15 scrub ok 2024-09-18T08:04:21.432 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:04:21 smithi007 ceph-mon[29459]: 4.a scrub starts 2024-09-18T08:04:21.432 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:04:21 smithi007 ceph-mon[29459]: 4.a scrub ok 2024-09-18T08:04:21.436 INFO:teuthology.orchestra.run.smithi007.stdout:osd.4: {} 2024-09-18T08:04:21.437 INFO:teuthology.orchestra.run.smithi007.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-18T08:04:21.462 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:04:21.463 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:04:21.463 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0' 2024-09-18T08:04:21.477 INFO:teuthology.orchestra.run.smithi007.stdout:osd.5: {} 2024-09-18T08:04:21.477 INFO:teuthology.orchestra.run.smithi007.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-18T08:04:21.512 INFO:teuthology.orchestra.run.smithi007.stdout:osd.6: {} 2024-09-18T08:04:21.512 INFO:teuthology.orchestra.run.smithi007.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-18T08:04:21.552 INFO:teuthology.orchestra.run.smithi007.stdout:osd.7: {} 2024-09-18T08:04:21.552 INFO:teuthology.orchestra.run.smithi007.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-18T08:04:21.564 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4 2024-09-18T08:04:21.570 INFO:tasks.radosbench.radosbench.0.smithi007.stdout: 16 15 35435 35420 138.332 103.75 0.0123566 0.00722122 2024-09-18T08:04:21.592 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:04:21.592 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:04:21.593 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4' 2024-09-18T08:04:21.656 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:04:21 smithi050 ceph-mon[32371]: 4.1c scrub starts 2024-09-18T08:04:21.656 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:04:21 smithi050 ceph-mon[32371]: 4.1c scrub ok 2024-09-18T08:04:21.656 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:04:21 smithi050 ceph-mon[32371]: 4.15 scrub starts 2024-09-18T08:04:21.656 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:04:21 smithi050 ceph-mon[32371]: 4.15 scrub ok 2024-09-18T08:04:21.656 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:04:21 smithi050 ceph-mon[32371]: 4.a scrub starts 2024-09-18T08:04:21.656 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:04:21 smithi050 ceph-mon[32371]: 4.a scrub ok 2024-09-18T08:04:21.694 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2 2024-09-18T08:04:21.740 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:04:21.741 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:04:21.741 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2' 2024-09-18T08:04:21.842 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6 2024-09-18T08:04:21.873 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:04:21.873 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:04:21.874 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6' 2024-09-18T08:04:21.975 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5 2024-09-18T08:04:22.007 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:04:22.008 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:04:22.008 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5' 2024-09-18T08:04:22.109 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4 2024-09-18T08:04:22.140 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:04:22.141 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:04:22.141 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4' 2024-09-18T08:04:22.230 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:04:22 smithi007 ceph-mon[29459]: 3.e scrub starts 2024-09-18T08:04:22.231 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:04:22 smithi007 ceph-mon[29459]: 3.e scrub ok 2024-09-18T08:04:22.231 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:04:22 smithi007 ceph-mon[29459]: 5.7 scrub starts 2024-09-18T08:04:22.231 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:04:22 smithi007 ceph-mon[29459]: 5.7 scrub ok 2024-09-18T08:04:22.231 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:04:22 smithi007 ceph-mon[29459]: pgmap v56: 148 pgs: 3 peering, 145 active+clean; 1.8 GiB data, 5.5 GiB used, 710 GiB / 715 GiB avail; 682 B/s rd, 101 MiB/s wr, 1.62k op/s 2024-09-18T08:04:22.231 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:04:22 smithi007 ceph-mon[29459]: from='client.? 172.21.15.7:0/3644604890' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-18T08:04:22.231 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:04:22 smithi007 ceph-mon[39249]: 3.e scrub starts 2024-09-18T08:04:22.231 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:04:22 smithi007 ceph-mon[39249]: 3.e scrub ok 2024-09-18T08:04:22.231 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:04:22 smithi007 ceph-mon[39249]: 5.7 scrub starts 2024-09-18T08:04:22.232 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:04:22 smithi007 ceph-mon[39249]: 5.7 scrub ok 2024-09-18T08:04:22.232 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:04:22 smithi007 ceph-mon[39249]: pgmap v56: 148 pgs: 3 peering, 145 active+clean; 1.8 GiB data, 5.5 GiB used, 710 GiB / 715 GiB avail; 682 B/s rd, 101 MiB/s wr, 1.62k op/s 2024-09-18T08:04:22.232 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:04:22 smithi007 ceph-mon[39249]: from='client.? 172.21.15.7:0/3644604890' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-18T08:04:22.242 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7 2024-09-18T08:04:22.273 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:04:22.274 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:04:22.275 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7' 2024-09-18T08:04:22.375 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3 2024-09-18T08:04:22.426 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:04:22.426 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:04:22.427 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3' 2024-09-18T08:04:22.527 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4 2024-09-18T08:04:22.557 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:04:22.557 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:04:22.557 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4' 2024-09-18T08:04:22.570 INFO:tasks.radosbench.radosbench.0.smithi007.stdout: 17 16 37319 37303 137.118 117.688 0.0101348 0.00728627 2024-09-18T08:04:22.648 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/779349ca-7593-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T08:04:22.656 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:04:22 smithi050 ceph-mon[32371]: 3.e scrub starts 2024-09-18T08:04:22.656 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:04:22 smithi050 ceph-mon[32371]: 3.e scrub ok 2024-09-18T08:04:22.656 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:04:22 smithi050 ceph-mon[32371]: 5.7 scrub starts 2024-09-18T08:04:22.656 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:04:22 smithi050 ceph-mon[32371]: 5.7 scrub ok 2024-09-18T08:04:22.656 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:04:22 smithi050 ceph-mon[32371]: pgmap v56: 148 pgs: 3 peering, 145 active+clean; 1.8 GiB data, 5.5 GiB used, 710 GiB / 715 GiB avail; 682 B/s rd, 101 MiB/s wr, 1.62k op/s 2024-09-18T08:04:22.656 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:04:22 smithi050 ceph-mon[32371]: from='client.? 172.21.15.7:0/3644604890' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-18T08:04:22.659 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1 2024-09-18T08:04:22.721 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:04:22.721 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:04:22.722 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1' 2024-09-18T08:04:22.823 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4 2024-09-18T08:04:22.849 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:04:22.850 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:04:22.850 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4' 2024-09-18T08:04:22.951 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3 2024-09-18T08:04:23.005 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:04:23.005 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:04:23.006 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3' 2024-09-18T08:04:23.107 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2 2024-09-18T08:04:23.156 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:04:23.156 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:04:23.157 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2' 2024-09-18T08:04:23.179 INFO:teuthology.orchestra.run.smithi007.stderr:nodeep-scrub is unset 2024-09-18T08:04:23.258 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5 2024-09-18T08:04:23.288 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:04:23.288 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:04:23.289 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5' 2024-09-18T08:04:23.369 INFO:teuthology.orchestra.run.smithi007.stdout:{ 2024-09-18T08:04:23.369 INFO:teuthology.orchestra.run.smithi007.stdout: "ops": [ 2024-09-18T08:04:23.369 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:04:23.369 INFO:teuthology.orchestra.run.smithi007.stdout: "description": "osd_repop(client.14805.0:38416 7.3 e95/83)", 2024-09-18T08:04:23.369 INFO:teuthology.orchestra.run.smithi007.stdout: "initiated_at": "2024-09-18T08:04:23.315167+0000", 2024-09-18T08:04:23.370 INFO:teuthology.orchestra.run.smithi007.stdout: "age": 0.00104257, 2024-09-18T08:04:23.370 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.001067283, 2024-09-18T08:04:23.370 INFO:teuthology.orchestra.run.smithi007.stdout: "continuous": false, 2024-09-18T08:04:23.370 INFO:teuthology.orchestra.run.smithi007.stdout: "type_data": { 2024-09-18T08:04:23.370 INFO:teuthology.orchestra.run.smithi007.stdout: "flag_point": "queued for pg", 2024-09-18T08:04:23.370 INFO:teuthology.orchestra.run.smithi007.stdout: "events": [ 2024-09-18T08:04:23.370 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:04:23.370 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "initiated", 2024-09-18T08:04:23.370 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:04:23.315167+0000", 2024-09-18T08:04:23.370 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-09-18T08:04:23.370 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:04:23.370 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:04:23.370 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "header_read", 2024-09-18T08:04:23.370 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:04:23.315167+0000", 2024-09-18T08:04:23.371 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-09-18T08:04:23.371 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:04:23.371 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:04:23.371 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "throttled", 2024-09-18T08:04:23.371 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:04:23.315168+0000", 2024-09-18T08:04:23.371 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 3.6399999999999998e-07 2024-09-18T08:04:23.371 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:04:23.371 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:04:23.371 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "all_read", 2024-09-18T08:04:23.371 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:04:23.315189+0000", 2024-09-18T08:04:23.371 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 2.1858999999999999e-05 2024-09-18T08:04:23.371 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:04:23.371 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:04:23.371 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "dispatched", 2024-09-18T08:04:23.371 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:04:23.315197+0000", 2024-09-18T08:04:23.372 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 7.7060000000000004e-06 2024-09-18T08:04:23.372 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:04:23.372 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:04:23.372 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "queued_for_pg", 2024-09-18T08:04:23.372 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:04:23.315203+0000", 2024-09-18T08:04:23.372 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 5.8989999999999998e-06 2024-09-18T08:04:23.372 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-09-18T08:04:23.372 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-09-18T08:04:23.372 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-09-18T08:04:23.372 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:04:23.372 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:04:23.372 INFO:teuthology.orchestra.run.smithi007.stdout: "description": "osd_op(client.14805.0:38414 7.e 7:7bce2cd9:::benchmark_data_smithi007_73155_object38413:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536 in=65536b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e95)", 2024-09-18T08:04:23.372 INFO:teuthology.orchestra.run.smithi007.stdout: "initiated_at": "2024-09-18T08:04:23.312173+0000", 2024-09-18T08:04:23.372 INFO:teuthology.orchestra.run.smithi007.stdout: "age": 0.0040364040000000004, 2024-09-18T08:04:23.373 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.004117547, 2024-09-18T08:04:23.373 INFO:teuthology.orchestra.run.smithi007.stdout: "continuous": false, 2024-09-18T08:04:23.373 INFO:teuthology.orchestra.run.smithi007.stdout: "type_data": { 2024-09-18T08:04:23.373 INFO:teuthology.orchestra.run.smithi007.stdout: "flag_point": "waiting for sub ops", 2024-09-18T08:04:23.373 INFO:teuthology.orchestra.run.smithi007.stdout: "client_info": { 2024-09-18T08:04:23.373 INFO:teuthology.orchestra.run.smithi007.stdout: "client": "client.14805", 2024-09-18T08:04:23.373 INFO:teuthology.orchestra.run.smithi007.stdout: "client_addr": "172.21.15.7:0/640320436", 2024-09-18T08:04:23.373 INFO:teuthology.orchestra.run.smithi007.stdout: "tid": 38414 2024-09-18T08:04:23.373 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:04:23.373 INFO:teuthology.orchestra.run.smithi007.stdout: "events": [ 2024-09-18T08:04:23.373 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:04:23.373 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "initiated", 2024-09-18T08:04:23.373 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:04:23.312173+0000", 2024-09-18T08:04:23.374 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-09-18T08:04:23.374 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:04:23.374 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:04:23.374 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "header_read", 2024-09-18T08:04:23.374 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:04:23.312173+0000", 2024-09-18T08:04:23.374 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-09-18T08:04:23.374 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:04:23.374 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:04:23.374 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "throttled", 2024-09-18T08:04:23.374 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:04:23.312175+0000", 2024-09-18T08:04:23.374 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.3540000000000001e-06 2024-09-18T08:04:23.374 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:04:23.375 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:04:23.375 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "all_read", 2024-09-18T08:04:23.375 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:04:23.312197+0000", 2024-09-18T08:04:23.375 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 2.2238e-05 2024-09-18T08:04:23.375 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:04:23.375 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:04:23.375 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "dispatched", 2024-09-18T08:04:23.375 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:04:23.312208+0000", 2024-09-18T08:04:23.375 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.0905e-05 2024-09-18T08:04:23.375 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:04:23.375 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:04:23.375 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "queued_for_pg", 2024-09-18T08:04:23.375 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:04:23.312215+0000", 2024-09-18T08:04:23.376 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 6.9979999999999997e-06 2024-09-18T08:04:23.376 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:04:23.376 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:04:23.376 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-09-18T08:04:23.376 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:04:23.312272+0000", 2024-09-18T08:04:23.376 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 5.6786999999999997e-05 2024-09-18T08:04:23.376 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:04:23.376 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:04:23.376 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "started", 2024-09-18T08:04:23.376 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:04:23.312386+0000", 2024-09-18T08:04:23.376 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.000114641 2024-09-18T08:04:23.376 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:04:23.376 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:04:23.376 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "waiting for subops from 0,5", 2024-09-18T08:04:23.377 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:04:23.312504+0000", 2024-09-18T08:04:23.377 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.00011760900000000001 2024-09-18T08:04:23.377 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:04:23.377 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:04:23.377 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "op_commit", 2024-09-18T08:04:23.377 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:04:23.314201+0000", 2024-09-18T08:04:23.377 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.0016971880000000001 2024-09-18T08:04:23.377 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:04:23.377 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:04:23.377 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:04:23.377 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:04:23.315418+0000", 2024-09-18T08:04:23.377 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.001217159 2024-09-18T08:04:23.377 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-09-18T08:04:23.378 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-09-18T08:04:23.378 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-09-18T08:04:23.378 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:04:23.378 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:04:23.378 INFO:teuthology.orchestra.run.smithi007.stdout: "description": "osd_repop(client.14805.0:38413 7.3 e95/83 7:c496efd6:::benchmark_data_smithi007_73155_object38412:head v 95'2397, mlcod=95'2395)", 2024-09-18T08:04:23.378 INFO:teuthology.orchestra.run.smithi007.stdout: "initiated_at": "2024-09-18T08:04:23.315100+0000", 2024-09-18T08:04:23.379 INFO:teuthology.orchestra.run.smithi007.stdout: "age": 0.0011100649999999999, 2024-09-18T08:04:23.380 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.0012692490000000001, 2024-09-18T08:04:23.380 INFO:teuthology.orchestra.run.smithi007.stdout: "continuous": false, 2024-09-18T08:04:23.380 INFO:teuthology.orchestra.run.smithi007.stdout: "type_data": { 2024-09-18T08:04:23.380 INFO:teuthology.orchestra.run.smithi007.stdout: "flag_point": "started", 2024-09-18T08:04:23.380 INFO:teuthology.orchestra.run.smithi007.stdout: "events": [ 2024-09-18T08:04:23.380 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:04:23.380 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "initiated", 2024-09-18T08:04:23.380 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:04:23.315100+0000", 2024-09-18T08:04:23.380 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-09-18T08:04:23.380 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:04:23.380 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:04:23.380 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "header_read", 2024-09-18T08:04:23.380 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:04:23.315100+0000", 2024-09-18T08:04:23.380 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-09-18T08:04:23.381 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:04:23.381 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:04:23.381 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "throttled", 2024-09-18T08:04:23.381 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:04:23.315101+0000", 2024-09-18T08:04:23.381 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.0750000000000001e-06 2024-09-18T08:04:23.381 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:04:23.381 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:04:23.381 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "all_read", 2024-09-18T08:04:23.381 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:04:23.315123+0000", 2024-09-18T08:04:23.381 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 2.2024000000000001e-05 2024-09-18T08:04:23.381 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:04:23.381 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:04:23.381 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "dispatched", 2024-09-18T08:04:23.382 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:04:23.315137+0000", 2024-09-18T08:04:23.382 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.4415e-05 2024-09-18T08:04:23.382 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:04:23.382 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:04:23.382 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "queued_for_pg", 2024-09-18T08:04:23.382 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:04:23.315145+0000", 2024-09-18T08:04:23.382 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 7.3819999999999998e-06 2024-09-18T08:04:23.382 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:04:23.382 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:04:23.382 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-09-18T08:04:23.382 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:04:23.315553+0000", 2024-09-18T08:04:23.382 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.00040860700000000001 2024-09-18T08:04:23.382 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:04:23.382 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:04:23.383 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "started", 2024-09-18T08:04:23.383 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:04:23.315573+0000", 2024-09-18T08:04:23.383 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 2.0010999999999999e-05 2024-09-18T08:04:23.383 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-09-18T08:04:23.383 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-09-18T08:04:23.383 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-09-18T08:04:23.383 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-09-18T08:04:23.383 INFO:teuthology.orchestra.run.smithi007.stdout: ], 2024-09-18T08:04:23.383 INFO:teuthology.orchestra.run.smithi007.stdout: "num_ops": 3 2024-09-18T08:04:23.383 INFO:teuthology.orchestra.run.smithi007.stdout:} 2024-09-18T08:04:23.390 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0 2024-09-18T08:04:23.434 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:04:23.434 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:04:23.435 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0' 2024-09-18T08:04:23.480 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:04:23 smithi007 ceph-mon[29459]: 4.2 scrub starts 2024-09-18T08:04:23.481 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:04:23 smithi007 ceph-mon[29459]: 4.2 scrub ok 2024-09-18T08:04:23.481 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:04:23 smithi007 ceph-mon[29459]: 5.1a scrub starts 2024-09-18T08:04:23.481 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:04:23 smithi007 ceph-mon[29459]: 5.1a scrub ok 2024-09-18T08:04:23.481 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:04:23 smithi007 ceph-mon[29459]: from='client.? 172.21.15.7:0/2383214211' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-18T08:04:23.481 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:04:23 smithi007 ceph-mon[29459]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-18T08:04:23.481 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:04:23 smithi007 ceph-mon[39249]: 4.2 scrub starts 2024-09-18T08:04:23.481 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:04:23 smithi007 ceph-mon[39249]: 4.2 scrub ok 2024-09-18T08:04:23.482 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:04:23 smithi007 ceph-mon[39249]: 5.1a scrub starts 2024-09-18T08:04:23.482 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:04:23 smithi007 ceph-mon[39249]: 5.1a scrub ok 2024-09-18T08:04:23.482 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:04:23 smithi007 ceph-mon[39249]: from='client.? 172.21.15.7:0/2383214211' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-18T08:04:23.482 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:04:23 smithi007 ceph-mon[39249]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-18T08:04:23.535 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7 2024-09-18T08:04:23.566 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:04:23.567 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:04:23.567 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7' 2024-09-18T08:04:23.570 INFO:tasks.radosbench.radosbench.0.smithi007.stdout: 18 16 38889 38873 134.951 98.125 0.00544363 0.00740479 2024-09-18T08:04:23.656 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:04:23 smithi050 ceph-mon[32371]: 4.2 scrub starts 2024-09-18T08:04:23.656 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:04:23 smithi050 ceph-mon[32371]: 4.2 scrub ok 2024-09-18T08:04:23.656 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:04:23 smithi050 ceph-mon[32371]: 5.1a scrub starts 2024-09-18T08:04:23.656 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:04:23 smithi050 ceph-mon[32371]: 5.1a scrub ok 2024-09-18T08:04:23.656 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:04:23 smithi050 ceph-mon[32371]: from='client.? 172.21.15.7:0/2383214211' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-18T08:04:23.656 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:04:23 smithi050 ceph-mon[32371]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-18T08:04:23.668 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1 2024-09-18T08:04:23.708 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:04:23.708 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:04:23.709 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1' 2024-09-18T08:04:23.810 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2 2024-09-18T08:04:23.852 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:04:23.853 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:04:23.853 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2' 2024-09-18T08:04:23.954 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4 2024-09-18T08:04:23.984 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:04:23.985 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:04:23.985 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4' 2024-09-18T08:04:24.086 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4 2024-09-18T08:04:24.116 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:04:24.117 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:04:24.117 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4' 2024-09-18T08:04:24.176 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:04:24 smithi007 ceph-mon[29459]: 3.1a scrub starts 2024-09-18T08:04:24.177 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:04:24 smithi007 ceph-mon[29459]: 3.1a scrub ok 2024-09-18T08:04:24.177 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:04:24 smithi007 ceph-mon[29459]: 4.f scrub starts 2024-09-18T08:04:24.177 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:04:24 smithi007 ceph-mon[29459]: 4.f scrub ok 2024-09-18T08:04:24.177 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:04:24 smithi007 ceph-mon[29459]: pgmap v57: 148 pgs: 3 peering, 145 active+clean; 2.2 GiB data, 6.6 GiB used, 709 GiB / 715 GiB avail; 1.5 KiB/s rd, 164 MiB/s wr, 2.62k op/s 2024-09-18T08:04:24.177 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:04:24 smithi007 ceph-mon[29459]: 3.12 scrub starts 2024-09-18T08:04:24.177 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:04:24 smithi007 ceph-mon[29459]: 3.12 scrub ok 2024-09-18T08:04:24.177 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:04:24 smithi007 ceph-mon[29459]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-18T08:04:24.177 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:04:24 smithi007 ceph-mon[29459]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-18T08:04:24.178 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:04:24 smithi007 ceph-mon[39249]: 3.1a scrub starts 2024-09-18T08:04:24.178 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:04:24 smithi007 ceph-mon[39249]: 3.1a scrub ok 2024-09-18T08:04:24.178 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:04:24 smithi007 ceph-mon[39249]: 4.f scrub starts 2024-09-18T08:04:24.178 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:04:24 smithi007 ceph-mon[39249]: 4.f scrub ok 2024-09-18T08:04:24.178 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:04:24 smithi007 ceph-mon[39249]: pgmap v57: 148 pgs: 3 peering, 145 active+clean; 2.2 GiB data, 6.6 GiB used, 709 GiB / 715 GiB avail; 1.5 KiB/s rd, 164 MiB/s wr, 2.62k op/s 2024-09-18T08:04:24.178 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:04:24 smithi007 ceph-mon[39249]: 3.12 scrub starts 2024-09-18T08:04:24.178 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:04:24 smithi007 ceph-mon[39249]: 3.12 scrub ok 2024-09-18T08:04:24.178 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:04:24 smithi007 ceph-mon[39249]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-18T08:04:24.178 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:04:24 smithi007 ceph-mon[39249]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-18T08:04:24.178 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:04:24 smithi007 ceph-mon[39249]: osdmap e95: 8 total, 8 up, 8 in 2024-09-18T08:04:24.178 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:04:24 smithi007 ceph-mon[39249]: 5.0 scrub starts 2024-09-18T08:04:24.178 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:04:24 smithi007 ceph-mon[39249]: 5.0 scrub ok 2024-09-18T08:04:24.179 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:04:24 smithi007 ceph-mon[39249]: osdmap e96: 8 total, 8 up, 8 in 2024-09-18T08:04:24.218 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2 2024-09-18T08:04:24.260 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:04:24.261 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:04:24.261 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2' 2024-09-18T08:04:24.362 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7 2024-09-18T08:04:24.373 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid 779349ca-7593-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-18T08:04:24.393 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:04:24.394 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:04:24.394 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7' 2024-09-18T08:04:24.406 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:04:24 smithi050 ceph-mon[32371]: 3.1a scrub starts 2024-09-18T08:04:24.406 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:04:24 smithi050 ceph-mon[32371]: 3.1a scrub ok 2024-09-18T08:04:24.406 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:04:24 smithi050 ceph-mon[32371]: 4.f scrub starts 2024-09-18T08:04:24.406 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:04:24 smithi050 ceph-mon[32371]: 4.f scrub ok 2024-09-18T08:04:24.406 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:04:24 smithi050 ceph-mon[32371]: pgmap v57: 148 pgs: 3 peering, 145 active+clean; 2.2 GiB data, 6.6 GiB used, 709 GiB / 715 GiB avail; 1.5 KiB/s rd, 164 MiB/s wr, 2.62k op/s 2024-09-18T08:04:24.406 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:04:24 smithi050 ceph-mon[32371]: 3.12 scrub starts 2024-09-18T08:04:24.406 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:04:24 smithi050 ceph-mon[32371]: 3.12 scrub ok 2024-09-18T08:04:24.406 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:04:24 smithi050 ceph-mon[32371]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-18T08:04:24.406 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:04:24 smithi050 ceph-mon[32371]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-18T08:04:24.406 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:04:24 smithi050 ceph-mon[32371]: osdmap e95: 8 total, 8 up, 8 in 2024-09-18T08:04:24.407 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:04:24 smithi050 ceph-mon[32371]: 5.0 scrub starts 2024-09-18T08:04:24.407 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:04:24 smithi050 ceph-mon[32371]: 5.0 scrub ok 2024-09-18T08:04:24.407 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:04:24 smithi050 ceph-mon[32371]: osdmap e96: 8 total, 8 up, 8 in 2024-09-18T08:04:24.480 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:04:24 smithi007 ceph-mon[29459]: osdmap e95: 8 total, 8 up, 8 in 2024-09-18T08:04:24.480 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:04:24 smithi007 ceph-mon[29459]: 5.0 scrub starts 2024-09-18T08:04:24.481 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:04:24 smithi007 ceph-mon[29459]: 5.0 scrub ok 2024-09-18T08:04:24.481 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:04:24 smithi007 ceph-mon[29459]: osdmap e96: 8 total, 8 up, 8 in 2024-09-18T08:04:24.495 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4 2024-09-18T08:04:24.524 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:04:24.525 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:04:24.527 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4' 2024-09-18T08:04:24.571 INFO:tasks.radosbench.radosbench.0.smithi007.stdout: 19 16 40655 40639 133.654 110.375 0.0158251 0.00747564 2024-09-18T08:04:24.617 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid 779349ca-7593-11ef-bceb-c7b262605968 -- ceph daemon osd.2 dump_blocked_ops 2024-09-18T08:04:24.628 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5 2024-09-18T08:04:24.656 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:04:24.657 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:04:24.657 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5' 2024-09-18T08:04:24.758 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5 2024-09-18T08:04:24.789 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:04:24.790 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:04:24.791 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5' 2024-09-18T08:04:24.891 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6 2024-09-18T08:04:24.920 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:04:24.921 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:04:24.921 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6' 2024-09-18T08:04:25.022 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3 2024-09-18T08:04:25.066 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:04:25.067 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:04:25.067 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3' 2024-09-18T08:04:25.168 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0 2024-09-18T08:04:25.217 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:04:25.217 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:04:25.217 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0' 2024-09-18T08:04:25.318 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2 2024-09-18T08:04:25.356 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:04:25.356 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:04:25.357 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2' 2024-09-18T08:04:25.458 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3 2024-09-18T08:04:25.480 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:04:25 smithi007 ceph-mon[29459]: 5.1b scrub starts 2024-09-18T08:04:25.480 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:04:25 smithi007 ceph-mon[29459]: 5.1b scrub ok 2024-09-18T08:04:25.480 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:04:25 smithi007 ceph-mon[29459]: 5.d scrub starts 2024-09-18T08:04:25.481 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:04:25 smithi007 ceph-mon[29459]: 5.d scrub ok 2024-09-18T08:04:25.481 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:04:25 smithi007 ceph-mon[29459]: 3.b scrub starts 2024-09-18T08:04:25.481 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:04:25 smithi007 ceph-mon[39249]: 5.1b scrub starts 2024-09-18T08:04:25.481 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:04:25 smithi007 ceph-mon[39249]: 5.1b scrub ok 2024-09-18T08:04:25.481 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:04:25 smithi007 ceph-mon[39249]: 5.d scrub starts 2024-09-18T08:04:25.481 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:04:25 smithi007 ceph-mon[39249]: 5.d scrub ok 2024-09-18T08:04:25.481 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:04:25 smithi007 ceph-mon[39249]: 3.b scrub starts 2024-09-18T08:04:25.494 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:04:25.495 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:04:25.496 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3' 2024-09-18T08:04:25.571 INFO:tasks.radosbench.radosbench.0.smithi007.stdout:2024-09-18T08:04:25.570046+0000 min lat: 0.00117713 max lat: 0.156524 avg lat: 0.00755221 2024-09-18T08:04:25.571 INFO:tasks.radosbench.radosbench.0.smithi007.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-09-18T08:04:25.571 INFO:tasks.radosbench.radosbench.0.smithi007.stdout: 20 16 42363 42347 132.308 106.75 0.00918578 0.00755221 2024-09-18T08:04:25.597 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5 2024-09-18T08:04:25.606 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-09-18T08:04:25.607 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-18T08:04:25.607 INFO:tasks.thrashosds.thrasher:inject_pause on osd.6 2024-09-18T08:04:25.607 INFO:tasks.thrashosds.thrasher:Testing filestore_inject_stall pause injection for duration 3 2024-09-18T08:04:25.607 INFO:tasks.thrashosds.thrasher:Checking after 0, should_be_down=False 2024-09-18T08:04:25.607 DEBUG:teuthology.orchestra.run.smithi050:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid 779349ca-7593-11ef-bceb-c7b262605968 -- ceph daemon osd.6 config set filestore_inject_stall 3 2024-09-18T08:04:25.629 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:04:25.630 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:04:25.630 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5' 2024-09-18T08:04:25.656 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:04:25 smithi050 ceph-mon[32371]: 5.1b scrub starts 2024-09-18T08:04:25.656 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:04:25 smithi050 ceph-mon[32371]: 5.1b scrub ok 2024-09-18T08:04:25.656 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:04:25 smithi050 ceph-mon[32371]: 5.d scrub starts 2024-09-18T08:04:25.656 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:04:25 smithi050 ceph-mon[32371]: 5.d scrub ok 2024-09-18T08:04:25.656 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:04:25 smithi050 ceph-mon[32371]: 3.b scrub starts 2024-09-18T08:04:25.731 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1 2024-09-18T08:04:25.788 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/779349ca-7593-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T08:04:25.801 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:04:25.802 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:04:25.802 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1' 2024-09-18T08:04:25.903 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1 2024-09-18T08:04:25.949 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:04:25.950 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:04:25.950 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1' 2024-09-18T08:04:25.987 INFO:teuthology.orchestra.run.smithi050.stderr:Inferring config /var/lib/ceph/779349ca-7593-11ef-bceb-c7b262605968/mon.b/config 2024-09-18T08:04:25.993 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/779349ca-7593-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T08:04:26.051 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2 2024-09-18T08:04:26.108 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:04:26.109 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:04:26.110 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2' 2024-09-18T08:04:26.211 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2 2024-09-18T08:04:26.250 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:04:26.251 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:04:26.252 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2' 2024-09-18T08:04:26.352 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5 2024-09-18T08:04:26.374 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:04:26 smithi007 ceph-mon[29459]: 3.15 scrub starts 2024-09-18T08:04:26.374 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:04:26 smithi007 ceph-mon[29459]: 3.15 scrub ok 2024-09-18T08:04:26.374 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:04:26 smithi007 ceph-mon[29459]: 6.d scrub starts 2024-09-18T08:04:26.374 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:04:26 smithi007 ceph-mon[29459]: 6.d scrub ok 2024-09-18T08:04:26.374 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:04:26 smithi007 ceph-mon[29459]: 3.b scrub ok 2024-09-18T08:04:26.374 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:04:26 smithi007 ceph-mon[29459]: pgmap v60: 148 pgs: 148 active+clean; 2.3 GiB data, 7.2 GiB used, 708 GiB / 715 GiB avail; 850 B/s rd, 91 MiB/s wr, 1.46k op/s; 73 B/s, 2 objects/s recovering 2024-09-18T08:04:26.375 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:04:26 smithi007 ceph-mon[29459]: 3.1b scrub starts 2024-09-18T08:04:26.375 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:04:26 smithi007 ceph-mon[29459]: 3.1b scrub ok 2024-09-18T08:04:26.375 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:04:26 smithi007 ceph-mon[29459]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg peering) 2024-09-18T08:04:26.375 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:04:26 smithi007 ceph-mon[29459]: Cluster is now healthy 2024-09-18T08:04:26.375 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:04:26 smithi007 ceph-mon[29459]: 5.1e scrub starts 2024-09-18T08:04:26.375 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:04:26 smithi007 ceph-mon[29459]: 5.1e scrub ok 2024-09-18T08:04:26.376 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:04:26 smithi007 ceph-mon[39249]: 3.15 scrub starts 2024-09-18T08:04:26.376 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:04:26 smithi007 ceph-mon[39249]: 3.15 scrub ok 2024-09-18T08:04:26.376 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:04:26 smithi007 ceph-mon[39249]: 6.d scrub starts 2024-09-18T08:04:26.376 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:04:26 smithi007 ceph-mon[39249]: 6.d scrub ok 2024-09-18T08:04:26.376 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:04:26 smithi007 ceph-mon[39249]: 3.b scrub ok 2024-09-18T08:04:26.376 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:04:26 smithi007 ceph-mon[39249]: pgmap v60: 148 pgs: 148 active+clean; 2.3 GiB data, 7.2 GiB used, 708 GiB / 715 GiB avail; 850 B/s rd, 91 MiB/s wr, 1.46k op/s; 73 B/s, 2 objects/s recovering 2024-09-18T08:04:26.376 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:04:26 smithi007 ceph-mon[39249]: 3.1b scrub starts 2024-09-18T08:04:26.376 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:04:26 smithi007 ceph-mon[39249]: 3.1b scrub ok 2024-09-18T08:04:26.376 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:04:26 smithi007 ceph-mon[39249]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg peering) 2024-09-18T08:04:26.377 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:04:26 smithi007 ceph-mon[39249]: Cluster is now healthy 2024-09-18T08:04:26.377 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:04:26 smithi007 ceph-mon[39249]: 5.1e scrub starts 2024-09-18T08:04:26.377 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:04:26 smithi007 ceph-mon[39249]: 5.1e scrub ok 2024-09-18T08:04:26.384 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:04:26.385 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:04:26.385 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5' 2024-09-18T08:04:26.398 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:04:26 smithi050 ceph-mon[32371]: 3.15 scrub starts 2024-09-18T08:04:26.398 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:04:26 smithi050 ceph-mon[32371]: 3.15 scrub ok 2024-09-18T08:04:26.398 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:04:26 smithi050 ceph-mon[32371]: 6.d scrub starts 2024-09-18T08:04:26.398 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:04:26 smithi050 ceph-mon[32371]: 6.d scrub ok 2024-09-18T08:04:26.399 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:04:26 smithi050 ceph-mon[32371]: 3.b scrub ok 2024-09-18T08:04:26.399 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:04:26 smithi050 ceph-mon[32371]: pgmap v60: 148 pgs: 148 active+clean; 2.3 GiB data, 7.2 GiB used, 708 GiB / 715 GiB avail; 850 B/s rd, 91 MiB/s wr, 1.46k op/s; 73 B/s, 2 objects/s recovering 2024-09-18T08:04:26.399 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:04:26 smithi050 ceph-mon[32371]: 3.1b scrub starts 2024-09-18T08:04:26.399 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:04:26 smithi050 ceph-mon[32371]: 3.1b scrub ok 2024-09-18T08:04:26.399 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:04:26 smithi050 ceph-mon[32371]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg peering) 2024-09-18T08:04:26.399 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:04:26 smithi050 ceph-mon[32371]: Cluster is now healthy 2024-09-18T08:04:26.399 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:04:26 smithi050 ceph-mon[32371]: 5.1e scrub starts 2024-09-18T08:04:26.399 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:04:26 smithi050 ceph-mon[32371]: 5.1e scrub ok 2024-09-18T08:04:26.486 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6 2024-09-18T08:04:26.516 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:04:26.516 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:04:26.517 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6' 2024-09-18T08:04:26.571 INFO:tasks.radosbench.radosbench.0.smithi007.stdout: 21 16 43919 43903 130.638 97.25 0.00911387 0.00764993 2024-09-18T08:04:26.618 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5 2024-09-18T08:04:26.654 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:04:26.654 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:04:26.654 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5' 2024-09-18T08:04:26.666 INFO:teuthology.orchestra.run.smithi007.stdout:osd.0: {} 2024-09-18T08:04:26.666 INFO:teuthology.orchestra.run.smithi007.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-18T08:04:26.676 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2024-09-18T08:04:26.676 INFO:teuthology.orchestra.run.smithi050.stdout: "success": "filestore_inject_stall = '' (not observed, change may require restart) " 2024-09-18T08:04:26.676 INFO:teuthology.orchestra.run.smithi050.stdout:} 2024-09-18T08:04:26.711 INFO:teuthology.orchestra.run.smithi007.stdout:osd.1: {} 2024-09-18T08:04:26.711 INFO:teuthology.orchestra.run.smithi007.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-18T08:04:26.747 INFO:teuthology.orchestra.run.smithi007.stdout:osd.2: {} 2024-09-18T08:04:26.748 INFO:teuthology.orchestra.run.smithi007.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-18T08:04:26.755 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1 2024-09-18T08:04:26.781 INFO:teuthology.orchestra.run.smithi007.stdout:osd.3: {} 2024-09-18T08:04:26.782 INFO:teuthology.orchestra.run.smithi007.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-18T08:04:26.797 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:04:26.798 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:04:26.798 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1' 2024-09-18T08:04:26.819 INFO:teuthology.orchestra.run.smithi007.stdout:osd.4: {} 2024-09-18T08:04:26.819 INFO:teuthology.orchestra.run.smithi007.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-18T08:04:26.856 INFO:teuthology.orchestra.run.smithi007.stdout:osd.5: {} 2024-09-18T08:04:26.857 INFO:teuthology.orchestra.run.smithi007.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-18T08:04:26.889 INFO:teuthology.orchestra.run.smithi007.stdout:osd.6: {} 2024-09-18T08:04:26.889 INFO:teuthology.orchestra.run.smithi007.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-18T08:04:26.899 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6 2024-09-18T08:04:26.924 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Sep 18 08:04:26 smithi007 ceph-779349ca-7593-11ef-bceb-c7b262605968-mgr-y[29688]: ::ffff:172.21.15.50 - - [18/Sep/2024:08:04:26] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-09-18T08:04:26.924 INFO:teuthology.orchestra.run.smithi007.stdout:osd.7: {} 2024-09-18T08:04:26.924 INFO:teuthology.orchestra.run.smithi007.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-18T08:04:26.933 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:04:26.934 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:04:26.934 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6' 2024-09-18T08:04:27.035 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1 2024-09-18T08:04:27.071 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:04:27.072 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:04:27.072 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1' 2024-09-18T08:04:27.173 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7 2024-09-18T08:04:27.202 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:04:27.202 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:04:27.203 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7' 2024-09-18T08:04:27.304 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3 2024-09-18T08:04:27.340 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:04:27 smithi050 ceph-mon[32371]: purged_snaps scrub starts 2024-09-18T08:04:27.340 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:04:27 smithi050 ceph-mon[32371]: purged_snaps scrub ok 2024-09-18T08:04:27.340 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:04:27 smithi050 ceph-mon[32371]: 5.a scrub starts 2024-09-18T08:04:27.340 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:04:27 smithi050 ceph-mon[32371]: 5.a scrub ok 2024-09-18T08:04:27.341 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:04:27 smithi050 ceph-mon[32371]: osdmap e97: 8 total, 8 up, 8 in 2024-09-18T08:04:27.341 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:04:27 smithi050 ceph-mon[32371]: from='client.? 172.21.15.7:0/2075758540' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-18T08:04:27.341 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:04:27 smithi050 ceph-mon[32371]: from='mgr.24470 172.21.15.7:0/2594390261' entity='mgr.y' 2024-09-18T08:04:27.341 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:04:27 smithi050 ceph-mon[32371]: from='mgr.24470 172.21.15.7:0/2594390261' entity='mgr.y' 2024-09-18T08:04:27.341 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:04:27 smithi050 ceph-mon[32371]: from='mgr.24470 172.21.15.7:0/2594390261' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-18T08:04:27.341 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:04:27 smithi050 ceph-mon[32371]: from='mgr.24470 172.21.15.7:0/2594390261' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-18T08:04:27.341 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:04:27 smithi050 ceph-mon[32371]: from='mgr.24470 172.21.15.7:0/2594390261' entity='mgr.y' 2024-09-18T08:04:27.344 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:04:27.346 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:04:27.346 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3' 2024-09-18T08:04:27.447 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4 2024-09-18T08:04:27.471 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:04:27 smithi007 ceph-mon[29459]: purged_snaps scrub starts 2024-09-18T08:04:27.471 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:04:27 smithi007 ceph-mon[29459]: purged_snaps scrub ok 2024-09-18T08:04:27.471 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:04:27 smithi007 ceph-mon[29459]: 5.a scrub starts 2024-09-18T08:04:27.471 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:04:27 smithi007 ceph-mon[29459]: 5.a scrub ok 2024-09-18T08:04:27.471 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:04:27 smithi007 ceph-mon[29459]: osdmap e97: 8 total, 8 up, 8 in 2024-09-18T08:04:27.471 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:04:27 smithi007 ceph-mon[29459]: from='client.? 172.21.15.7:0/2075758540' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-18T08:04:27.471 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:04:27 smithi007 ceph-mon[29459]: from='mgr.24470 172.21.15.7:0/2594390261' entity='mgr.y' 2024-09-18T08:04:27.471 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:04:27 smithi007 ceph-mon[29459]: from='mgr.24470 172.21.15.7:0/2594390261' entity='mgr.y' 2024-09-18T08:04:27.471 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:04:27 smithi007 ceph-mon[29459]: from='mgr.24470 172.21.15.7:0/2594390261' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-18T08:04:27.471 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:04:27 smithi007 ceph-mon[29459]: from='mgr.24470 172.21.15.7:0/2594390261' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-18T08:04:27.472 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:04:27 smithi007 ceph-mon[29459]: from='mgr.24470 172.21.15.7:0/2594390261' entity='mgr.y' 2024-09-18T08:04:27.472 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:04:27 smithi007 ceph-mon[39249]: purged_snaps scrub starts 2024-09-18T08:04:27.472 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:04:27 smithi007 ceph-mon[39249]: purged_snaps scrub ok 2024-09-18T08:04:27.472 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:04:27 smithi007 ceph-mon[39249]: 5.a scrub starts 2024-09-18T08:04:27.472 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:04:27 smithi007 ceph-mon[39249]: 5.a scrub ok 2024-09-18T08:04:27.472 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:04:27 smithi007 ceph-mon[39249]: osdmap e97: 8 total, 8 up, 8 in 2024-09-18T08:04:27.472 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:04:27 smithi007 ceph-mon[39249]: from='client.? 172.21.15.7:0/2075758540' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-18T08:04:27.472 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:04:27 smithi007 ceph-mon[39249]: from='mgr.24470 172.21.15.7:0/2594390261' entity='mgr.y' 2024-09-18T08:04:27.472 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:04:27 smithi007 ceph-mon[39249]: from='mgr.24470 172.21.15.7:0/2594390261' entity='mgr.y' 2024-09-18T08:04:27.473 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:04:27 smithi007 ceph-mon[39249]: from='mgr.24470 172.21.15.7:0/2594390261' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-18T08:04:27.473 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:04:27 smithi007 ceph-mon[39249]: from='mgr.24470 172.21.15.7:0/2594390261' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-18T08:04:27.473 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:04:27 smithi007 ceph-mon[39249]: from='mgr.24470 172.21.15.7:0/2594390261' entity='mgr.y' 2024-09-18T08:04:27.475 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:04:27.475 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:04:27.476 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4' 2024-09-18T08:04:27.571 INFO:tasks.radosbench.radosbench.0.smithi007.stdout: 22 16 45509 45493 129.217 99.375 0.0130279 0.00773349 2024-09-18T08:04:27.577 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6 2024-09-18T08:04:27.604 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:04:27.605 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:04:27.605 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6' 2024-09-18T08:04:27.706 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6 2024-09-18T08:04:27.741 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:04:27.742 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:04:27.742 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6' 2024-09-18T08:04:27.746 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid 779349ca-7593-11ef-bceb-c7b262605968 -- ceph osd set noscrub 2024-09-18T08:04:27.842 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5 2024-09-18T08:04:27.874 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:04:27.875 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:04:27.875 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5' 2024-09-18T08:04:27.976 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7 2024-09-18T08:04:28.006 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:04:28.007 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:04:28.007 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7' 2024-09-18T08:04:28.108 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3 2024-09-18T08:04:28.158 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:04:28.158 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:04:28.158 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3' 2024-09-18T08:04:28.259 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2 2024-09-18T08:04:28.321 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:04:28.322 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:04:28.322 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2' 2024-09-18T08:04:28.387 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:04:28 smithi007 ceph-mon[29459]: 4.8 scrub starts 2024-09-18T08:04:28.387 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:04:28 smithi007 ceph-mon[29459]: 4.8 scrub ok 2024-09-18T08:04:28.387 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:04:28 smithi007 ceph-mon[29459]: pgmap v62: 148 pgs: 148 active+clean; 2.3 GiB data, 7.2 GiB used, 708 GiB / 715 GiB avail; 853 B/s rd, 92 MiB/s wr, 1.47k op/s; 73 B/s, 2 objects/s recovering 2024-09-18T08:04:28.388 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:04:28 smithi007 ceph-mon[29459]: Reconfiguring alertmanager.a (dependencies changed)... 2024-09-18T08:04:28.388 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:04:28 smithi007 ceph-mon[29459]: Reconfiguring daemon alertmanager.a on smithi007 2024-09-18T08:04:28.388 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:04:28 smithi007 ceph-mon[39249]: 4.8 scrub starts 2024-09-18T08:04:28.388 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:04:28 smithi007 ceph-mon[39249]: 4.8 scrub ok 2024-09-18T08:04:28.389 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:04:28 smithi007 ceph-mon[39249]: pgmap v62: 148 pgs: 148 active+clean; 2.3 GiB data, 7.2 GiB used, 708 GiB / 715 GiB avail; 853 B/s rd, 92 MiB/s wr, 1.47k op/s; 73 B/s, 2 objects/s recovering 2024-09-18T08:04:28.389 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:04:28 smithi007 ceph-mon[39249]: Reconfiguring alertmanager.a (dependencies changed)... 2024-09-18T08:04:28.389 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:04:28 smithi007 ceph-mon[39249]: Reconfiguring daemon alertmanager.a on smithi007 2024-09-18T08:04:28.423 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0 2024-09-18T08:04:28.469 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:04:28.470 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:04:28.470 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0' 2024-09-18T08:04:28.481 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:04:28 smithi050 ceph-mon[32371]: 4.8 scrub starts 2024-09-18T08:04:28.481 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:04:28 smithi050 ceph-mon[32371]: 4.8 scrub ok 2024-09-18T08:04:28.481 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:04:28 smithi050 ceph-mon[32371]: pgmap v62: 148 pgs: 148 active+clean; 2.3 GiB data, 7.2 GiB used, 708 GiB / 715 GiB avail; 853 B/s rd, 92 MiB/s wr, 1.47k op/s; 73 B/s, 2 objects/s recovering 2024-09-18T08:04:28.481 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:04:28 smithi050 ceph-mon[32371]: Reconfiguring alertmanager.a (dependencies changed)... 2024-09-18T08:04:28.481 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:04:28 smithi050 ceph-mon[32371]: Reconfiguring daemon alertmanager.a on smithi007 2024-09-18T08:04:28.571 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5 2024-09-18T08:04:28.572 INFO:tasks.radosbench.radosbench.0.smithi007.stdout: 23 16 46928 46912 127.455 88.6875 0.00500119 0.00783952 2024-09-18T08:04:28.576 INFO:teuthology.orchestra.run.smithi007.stdout:ERROR: (22) Invalid argument 2024-09-18T08:04:28.576 INFO:teuthology.orchestra.run.smithi007.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-18T08:04:28.601 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:04:28.602 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:04:28.602 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5' 2024-09-18T08:04:28.703 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7 2024-09-18T08:04:28.730 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:04:28.731 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:04:28.731 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7' 2024-09-18T08:04:28.832 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5 2024-09-18T08:04:28.862 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:04:28.916 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:04:28.917 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5' 2024-09-18T08:04:29.017 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3 2024-09-18T08:04:29.065 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:04:29.066 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:04:29.066 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3' 2024-09-18T08:04:29.156 INFO:journalctl@ceph.iscsi.iscsi.a.smithi050.stdout:Sep 18 08:04:28 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-iscsi-iscsi-a[57782]: debug there is no tcmu-runner data available 2024-09-18T08:04:29.167 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3 2024-09-18T08:04:29.210 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:04:29.210 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:04:29.211 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3' 2024-09-18T08:04:29.312 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0 2024-09-18T08:04:29.354 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:04:29.355 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:04:29.355 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0' 2024-09-18T08:04:29.393 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:04:29 smithi007 ceph-mon[29459]: 5.1c scrub starts 2024-09-18T08:04:29.394 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:04:29 smithi007 ceph-mon[29459]: 5.1c scrub ok 2024-09-18T08:04:29.394 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:04:29 smithi007 ceph-mon[29459]: 5.19 scrub starts 2024-09-18T08:04:29.394 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:04:29 smithi007 ceph-mon[29459]: 5.19 scrub ok 2024-09-18T08:04:29.394 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:04:29 smithi007 ceph-mon[29459]: osdmap e98: 8 total, 8 up, 8 in 2024-09-18T08:04:29.394 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:04:29 smithi007 ceph-mon[39249]: 5.1c scrub starts 2024-09-18T08:04:29.394 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:04:29 smithi007 ceph-mon[39249]: 5.1c scrub ok 2024-09-18T08:04:29.394 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:04:29 smithi007 ceph-mon[39249]: 5.19 scrub starts 2024-09-18T08:04:29.394 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:04:29 smithi007 ceph-mon[39249]: 5.19 scrub ok 2024-09-18T08:04:29.394 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:04:29 smithi007 ceph-mon[39249]: osdmap e98: 8 total, 8 up, 8 in 2024-09-18T08:04:29.456 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5 2024-09-18T08:04:29.486 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:04:29.487 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:04:29.487 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5' 2024-09-18T08:04:29.571 INFO:tasks.radosbench.radosbench.0.smithi007.stdout: 24 15 48374 48359 125.912 90.4375 0.0204696 0.00793629 2024-09-18T08:04:29.588 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0 2024-09-18T08:04:29.630 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:04:29.631 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:04:29.631 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0' 2024-09-18T08:04:29.656 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:04:29 smithi050 ceph-mon[32371]: 5.1c scrub starts 2024-09-18T08:04:29.656 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:04:29 smithi050 ceph-mon[32371]: 5.1c scrub ok 2024-09-18T08:04:29.656 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:04:29 smithi050 ceph-mon[32371]: 5.19 scrub starts 2024-09-18T08:04:29.656 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:04:29 smithi050 ceph-mon[32371]: 5.19 scrub ok 2024-09-18T08:04:29.656 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:04:29 smithi050 ceph-mon[32371]: osdmap e98: 8 total, 8 up, 8 in 2024-09-18T08:04:29.732 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7 2024-09-18T08:04:29.760 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:04:29.760 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:04:29.761 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7' 2024-09-18T08:04:29.862 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7 2024-09-18T08:04:29.889 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:04:29.890 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:04:29.891 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7' 2024-09-18T08:04:29.992 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0 2024-09-18T08:04:30.013 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid 779349ca-7593-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-18T08:04:30.034 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:04:30.035 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:04:30.035 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0' 2024-09-18T08:04:30.137 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3 2024-09-18T08:04:30.175 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:04:30.176 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:04:30.176 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3' 2024-09-18T08:04:30.277 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7 2024-09-18T08:04:30.307 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:04:30.308 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:04:30.308 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7' 2024-09-18T08:04:30.409 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0 2024-09-18T08:04:30.455 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:04:30.456 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:04:30.456 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0' 2024-09-18T08:04:30.481 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:04:30 smithi007 ceph-mon[29459]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-18T08:04:30.481 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:04:30 smithi007 ceph-mon[29459]: 3.4 scrub starts 2024-09-18T08:04:30.481 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:04:30 smithi007 ceph-mon[29459]: 3.4 scrub ok 2024-09-18T08:04:30.481 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:04:30 smithi007 ceph-mon[29459]: 2.2 scrub starts 2024-09-18T08:04:30.481 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:04:30 smithi007 ceph-mon[29459]: 2.2 scrub ok 2024-09-18T08:04:30.481 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:04:30 smithi007 ceph-mon[29459]: pgmap v64: 148 pgs: 148 active+clean; 2.8 GiB data, 8.7 GiB used, 706 GiB / 715 GiB avail; 866 B/s rd, 88 MiB/s wr, 1.40k op/s 2024-09-18T08:04:30.481 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:04:30 smithi007 ceph-mon[39249]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-18T08:04:30.482 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:04:30 smithi007 ceph-mon[39249]: 3.4 scrub starts 2024-09-18T08:04:30.482 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:04:30 smithi007 ceph-mon[39249]: 3.4 scrub ok 2024-09-18T08:04:30.482 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:04:30 smithi007 ceph-mon[39249]: 2.2 scrub starts 2024-09-18T08:04:30.482 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:04:30 smithi007 ceph-mon[39249]: 2.2 scrub ok 2024-09-18T08:04:30.482 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:04:30 smithi007 ceph-mon[39249]: pgmap v64: 148 pgs: 148 active+clean; 2.8 GiB data, 8.7 GiB used, 706 GiB / 715 GiB avail; 866 B/s rd, 88 MiB/s wr, 1.40k op/s 2024-09-18T08:04:30.558 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0 2024-09-18T08:04:30.571 INFO:tasks.radosbench.radosbench.0.smithi007.stdout: 25 16 49822 49806 124.491 90.4375 0.00810566 0.0080275 2024-09-18T08:04:30.598 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:04:30.599 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:04:30.600 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0' 2024-09-18T08:04:30.655 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:04:30 smithi050 ceph-mon[32371]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-18T08:04:30.656 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:04:30 smithi050 ceph-mon[32371]: 3.4 scrub starts 2024-09-18T08:04:30.656 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:04:30 smithi050 ceph-mon[32371]: 3.4 scrub ok 2024-09-18T08:04:30.656 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:04:30 smithi050 ceph-mon[32371]: 2.2 scrub starts 2024-09-18T08:04:30.656 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:04:30 smithi050 ceph-mon[32371]: 2.2 scrub ok 2024-09-18T08:04:30.656 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:04:30 smithi050 ceph-mon[32371]: pgmap v64: 148 pgs: 148 active+clean; 2.8 GiB data, 8.7 GiB used, 706 GiB / 715 GiB avail; 866 B/s rd, 88 MiB/s wr, 1.40k op/s 2024-09-18T08:04:30.700 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1 2024-09-18T08:04:30.737 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:04:30.738 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:04:30.738 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1' 2024-09-18T08:04:30.839 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6 2024-09-18T08:04:30.866 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:04:30.867 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:04:30.867 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6' 2024-09-18T08:04:30.968 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7 2024-09-18T08:04:30.999 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:04:30.999 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:04:31.000 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7' 2024-09-18T08:04:31.101 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3 2024-09-18T08:04:31.142 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:04:31.143 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:04:31.143 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3' 2024-09-18T08:04:31.244 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0 2024-09-18T08:04:31.285 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:04:31.286 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:04:31.287 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0' 2024-09-18T08:04:31.388 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0 2024-09-18T08:04:31.428 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:04:31.428 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:04:31.429 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0' 2024-09-18T08:04:31.480 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:04:31 smithi007 ceph-mon[29459]: 5.18 scrub starts 2024-09-18T08:04:31.480 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:04:31 smithi007 ceph-mon[29459]: 5.18 scrub ok 2024-09-18T08:04:31.480 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:04:31 smithi007 ceph-mon[29459]: 6.1f scrub starts 2024-09-18T08:04:31.481 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:04:31 smithi007 ceph-mon[29459]: 6.1f scrub ok 2024-09-18T08:04:31.481 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:04:31 smithi007 ceph-mon[39249]: 5.18 scrub starts 2024-09-18T08:04:31.481 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:04:31 smithi007 ceph-mon[39249]: 5.18 scrub ok 2024-09-18T08:04:31.481 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:04:31 smithi007 ceph-mon[39249]: 6.1f scrub starts 2024-09-18T08:04:31.481 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:04:31 smithi007 ceph-mon[39249]: 6.1f scrub ok 2024-09-18T08:04:31.530 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5 2024-09-18T08:04:31.560 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:04:31.561 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:04:31.562 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5' 2024-09-18T08:04:31.572 INFO:tasks.radosbench.radosbench.0.smithi007.stdout: 26 16 51484 51468 123.698 103.875 0.00649641 0.00807997 2024-09-18T08:04:31.656 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:04:31 smithi050 ceph-mon[32371]: 5.18 scrub starts 2024-09-18T08:04:31.656 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:04:31 smithi050 ceph-mon[32371]: 5.18 scrub ok 2024-09-18T08:04:31.656 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:04:31 smithi050 ceph-mon[32371]: 6.1f scrub starts 2024-09-18T08:04:31.656 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:04:31 smithi050 ceph-mon[32371]: 6.1f scrub ok 2024-09-18T08:04:31.662 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5 2024-09-18T08:04:31.693 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:04:31.694 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:04:31.694 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5' 2024-09-18T08:04:31.795 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2 2024-09-18T08:04:31.841 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:04:31.842 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:04:31.842 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2' 2024-09-18T08:04:31.885 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid 779349ca-7593-11ef-bceb-c7b262605968 -- ceph daemon osd.2 dump_historic_ops 2024-09-18T08:04:31.943 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5 2024-09-18T08:04:31.972 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:04:31.973 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:04:31.973 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5' 2024-09-18T08:04:32.023 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/779349ca-7593-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T08:04:32.074 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7 2024-09-18T08:04:32.100 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:04:32.101 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:04:32.101 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7' 2024-09-18T08:04:32.202 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2 2024-09-18T08:04:32.253 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:04:32.254 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:04:32.254 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2' 2024-09-18T08:04:32.355 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4 2024-09-18T08:04:32.357 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:04:32 smithi007 ceph-mon[29459]: pgmap v65: 148 pgs: 148 active+clean; 2.8 GiB data, 8.7 GiB used, 706 GiB / 715 GiB avail; 729 B/s rd, 74 MiB/s wr, 1.18k op/s 2024-09-18T08:04:32.357 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:04:32 smithi007 ceph-mon[39249]: pgmap v65: 148 pgs: 148 active+clean; 2.8 GiB data, 8.7 GiB used, 706 GiB / 715 GiB avail; 729 B/s rd, 74 MiB/s wr, 1.18k op/s 2024-09-18T08:04:32.384 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:04:32.385 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:04:32.385 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4' 2024-09-18T08:04:32.390 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-09-18T08:04:32.390 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-18T08:04:32.390 INFO:tasks.thrashosds.thrasher:primary_affinity 2024-09-18T08:04:32.390 INFO:tasks.thrashosds.thrasher:Setting osd 6 primary_affinity to 0.000000 2024-09-18T08:04:32.390 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid 779349ca-7593-11ef-bceb-c7b262605968 -- ceph osd primary-affinity 6 0 2024-09-18T08:04:32.486 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0 2024-09-18T08:04:32.535 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:04:32.536 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:04:32.536 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0' 2024-09-18T08:04:32.572 INFO:tasks.radosbench.radosbench.0.smithi007.stdout: 27 15 52966 52951 122.549 92.6875 0.0150473 0.00815422 2024-09-18T08:04:32.637 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3 2024-09-18T08:04:32.656 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:04:32 smithi050 ceph-mon[32371]: pgmap v65: 148 pgs: 148 active+clean; 2.8 GiB data, 8.7 GiB used, 706 GiB / 715 GiB avail; 729 B/s rd, 74 MiB/s wr, 1.18k op/s 2024-09-18T08:04:32.680 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:04:32.682 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:04:32.683 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3' 2024-09-18T08:04:32.783 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4 2024-09-18T08:04:32.810 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:04:32.811 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:04:32.811 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4' 2024-09-18T08:04:32.912 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6 2024-09-18T08:04:32.940 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:04:32.941 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:04:32.941 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6' 2024-09-18T08:04:33.042 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7 2024-09-18T08:04:33.073 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:04:33.074 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:04:33.074 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7' 2024-09-18T08:04:33.175 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0 2024-09-18T08:04:33.217 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:04:33.217 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:04:33.218 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0' 2024-09-18T08:04:33.318 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1 2024-09-18T08:04:33.366 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:04:33.367 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:04:33.367 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1' 2024-09-18T08:04:33.468 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1 2024-09-18T08:04:33.508 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:04:33.508 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:04:33.508 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1' 2024-09-18T08:04:33.572 INFO:tasks.radosbench.radosbench.0.smithi007.stdout: 28 15 54476 54461 121.543 94.375 0.0119929 0.00822232 2024-09-18T08:04:33.609 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1 2024-09-18T08:04:33.651 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:04:33.652 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:04:33.652 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1' 2024-09-18T08:04:33.655 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:04:33 smithi050 ceph-mon[32371]: 4.1b scrub starts 2024-09-18T08:04:33.656 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:04:33 smithi050 ceph-mon[32371]: 4.1b scrub ok 2024-09-18T08:04:33.656 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:04:33 smithi050 ceph-mon[32371]: from='mgr.24470 172.21.15.7:0/2594390261' entity='mgr.y' 2024-09-18T08:04:33.656 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:04:33 smithi050 ceph-mon[32371]: from='mgr.24470 172.21.15.7:0/2594390261' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-18T08:04:33.730 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:04:33 smithi007 ceph-mon[29459]: 4.1b scrub starts 2024-09-18T08:04:33.730 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:04:33 smithi007 ceph-mon[29459]: 4.1b scrub ok 2024-09-18T08:04:33.730 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:04:33 smithi007 ceph-mon[29459]: from='mgr.24470 172.21.15.7:0/2594390261' entity='mgr.y' 2024-09-18T08:04:33.731 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:04:33 smithi007 ceph-mon[29459]: from='mgr.24470 172.21.15.7:0/2594390261' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-18T08:04:33.731 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:04:33 smithi007 ceph-mon[39249]: 4.1b scrub starts 2024-09-18T08:04:33.731 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:04:33 smithi007 ceph-mon[39249]: 4.1b scrub ok 2024-09-18T08:04:33.731 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:04:33 smithi007 ceph-mon[39249]: from='mgr.24470 172.21.15.7:0/2594390261' entity='mgr.y' 2024-09-18T08:04:33.731 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:04:33 smithi007 ceph-mon[39249]: from='mgr.24470 172.21.15.7:0/2594390261' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-18T08:04:33.753 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1 2024-09-18T08:04:33.793 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:04:33.794 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:04:33.794 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1' 2024-09-18T08:04:33.895 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2 2024-09-18T08:04:33.933 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:04:33.934 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:04:33.934 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2' 2024-09-18T08:04:34.035 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1 2024-09-18T08:04:34.074 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:04:34.074 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:04:34.075 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1' 2024-09-18T08:04:34.176 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5 2024-09-18T08:04:34.206 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:04:34.207 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:04:34.207 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5' 2024-09-18T08:04:34.308 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1 2024-09-18T08:04:34.345 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:04:34.346 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:04:34.346 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1' 2024-09-18T08:04:34.447 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0 2024-09-18T08:04:34.486 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:04:34.487 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:04:34.488 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0' 2024-09-18T08:04:34.499 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:04:34 smithi007 ceph-mon[29459]: 6.e scrub starts 2024-09-18T08:04:34.500 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:04:34 smithi007 ceph-mon[29459]: 6.e scrub ok 2024-09-18T08:04:34.500 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:04:34 smithi007 ceph-mon[29459]: 3.1c scrub starts 2024-09-18T08:04:34.500 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:04:34 smithi007 ceph-mon[29459]: 3.1c scrub ok 2024-09-18T08:04:34.500 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:04:34 smithi007 ceph-mon[29459]: pgmap v66: 148 pgs: 148 active+clean; 3.0 GiB data, 9.2 GiB used, 706 GiB / 715 GiB avail; 1.2 KiB/s rd, 86 MiB/s wr, 1.38k op/s 2024-09-18T08:04:34.500 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:04:34 smithi007 ceph-mon[39249]: 6.e scrub starts 2024-09-18T08:04:34.500 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:04:34 smithi007 ceph-mon[39249]: 6.e scrub ok 2024-09-18T08:04:34.500 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:04:34 smithi007 ceph-mon[39249]: 3.1c scrub starts 2024-09-18T08:04:34.500 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:04:34 smithi007 ceph-mon[39249]: 3.1c scrub ok 2024-09-18T08:04:34.501 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:04:34 smithi007 ceph-mon[39249]: pgmap v66: 148 pgs: 148 active+clean; 3.0 GiB data, 9.2 GiB used, 706 GiB / 715 GiB avail; 1.2 KiB/s rd, 86 MiB/s wr, 1.38k op/s 2024-09-18T08:04:34.572 INFO:tasks.radosbench.radosbench.0.smithi007.stdout: 29 16 55990 55974 120.612 94.5625 0.0102058 0.00828465 2024-09-18T08:04:34.588 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7 2024-09-18T08:04:34.615 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:04:34.616 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:04:34.616 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7' 2024-09-18T08:04:34.656 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:04:34 smithi050 ceph-mon[32371]: 6.e scrub starts 2024-09-18T08:04:34.656 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:04:34 smithi050 ceph-mon[32371]: 6.e scrub ok 2024-09-18T08:04:34.656 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:04:34 smithi050 ceph-mon[32371]: 3.1c scrub starts 2024-09-18T08:04:34.656 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:04:34 smithi050 ceph-mon[32371]: 3.1c scrub ok 2024-09-18T08:04:34.656 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:04:34 smithi050 ceph-mon[32371]: pgmap v66: 148 pgs: 148 active+clean; 3.0 GiB data, 9.2 GiB used, 706 GiB / 715 GiB avail; 1.2 KiB/s rd, 86 MiB/s wr, 1.38k op/s 2024-09-18T08:04:34.717 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5 2024-09-18T08:04:34.746 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:04:34.747 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:04:34.747 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5' 2024-09-18T08:04:34.848 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3 2024-09-18T08:04:34.871 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:04:34.871 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:04:34.872 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3' 2024-09-18T08:04:34.972 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3 2024-09-18T08:04:35.016 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:04:35.017 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:04:35.017 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3' 2024-09-18T08:04:35.118 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3 2024-09-18T08:04:35.144 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:04:35.145 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:04:35.145 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3' 2024-09-18T08:04:35.246 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1 2024-09-18T08:04:35.291 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:04:35.292 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:04:35.292 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1' 2024-09-18T08:04:35.393 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1 2024-09-18T08:04:35.416 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:04:35.416 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:04:35.416 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1' 2024-09-18T08:04:35.517 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1 2024-09-18T08:04:35.560 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:04:35.561 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:04:35.561 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1' 2024-09-18T08:04:35.572 INFO:tasks.radosbench.radosbench.0.smithi007.stdout: 30 16 57207 57191 119.127 76.0625 0.00817477 0.00838861 2024-09-18T08:04:35.626 INFO:teuthology.orchestra.run.smithi007.stderr:noscrub is set 2024-09-18T08:04:35.627 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:04:35 smithi007 ceph-mon[29459]: 1.0 scrub starts 2024-09-18T08:04:35.627 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:04:35 smithi007 ceph-mon[29459]: 1.0 scrub ok 2024-09-18T08:04:35.627 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:04:35 smithi007 ceph-mon[29459]: 3.1f scrub starts 2024-09-18T08:04:35.627 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:04:35 smithi007 ceph-mon[29459]: 3.1f scrub ok 2024-09-18T08:04:35.627 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:04:35 smithi007 ceph-mon[29459]: from='client.? 172.21.15.7:0/3601037845' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-18T08:04:35.627 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:04:35 smithi007 ceph-mon[39249]: 1.0 scrub starts 2024-09-18T08:04:35.627 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:04:35 smithi007 ceph-mon[39249]: 1.0 scrub ok 2024-09-18T08:04:35.628 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:04:35 smithi007 ceph-mon[39249]: 3.1f scrub starts 2024-09-18T08:04:35.628 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:04:35 smithi007 ceph-mon[39249]: 3.1f scrub ok 2024-09-18T08:04:35.628 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:04:35 smithi007 ceph-mon[39249]: from='client.? 172.21.15.7:0/3601037845' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-18T08:04:35.655 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:04:35 smithi050 ceph-mon[32371]: 1.0 scrub starts 2024-09-18T08:04:35.656 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:04:35 smithi050 ceph-mon[32371]: 1.0 scrub ok 2024-09-18T08:04:35.656 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:04:35 smithi050 ceph-mon[32371]: 3.1f scrub starts 2024-09-18T08:04:35.656 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:04:35 smithi050 ceph-mon[32371]: 3.1f scrub ok 2024-09-18T08:04:35.656 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:04:35 smithi050 ceph-mon[32371]: from='client.? 172.21.15.7:0/3601037845' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-18T08:04:35.662 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2 2024-09-18T08:04:35.702 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:04:35.702 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:04:35.703 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2' 2024-09-18T08:04:35.804 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3 2024-09-18T08:04:35.848 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:04:35.848 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:04:35.849 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3' 2024-09-18T08:04:35.950 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5 2024-09-18T08:04:35.983 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:04:35.984 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:04:35.984 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5' 2024-09-18T08:04:36.085 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2 2024-09-18T08:04:36.113 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:04:36.114 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:04:36.114 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2' 2024-09-18T08:04:36.215 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6 2024-09-18T08:04:36.246 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:04:36.246 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:04:36.247 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6' 2024-09-18T08:04:36.348 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4 2024-09-18T08:04:36.376 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:04:36.377 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:04:36.378 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4' 2024-09-18T08:04:36.478 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6 2024-09-18T08:04:36.501 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:04:36.501 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:04:36.501 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6' 2024-09-18T08:04:36.572 INFO:tasks.radosbench.radosbench.0.smithi007.stdout: 31 16 58330 58314 117.548 70.1875 0.00528392 0.00850181 2024-09-18T08:04:36.602 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0 2024-09-18T08:04:36.646 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:04:36.646 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:04:36.646 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0' 2024-09-18T08:04:36.656 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:04:36 smithi050 ceph-mon[32371]: 4.13 scrub starts 2024-09-18T08:04:36.656 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:04:36 smithi050 ceph-mon[32371]: 4.13 scrub ok 2024-09-18T08:04:36.656 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:04:36 smithi050 ceph-mon[32371]: purged_snaps scrub starts 2024-09-18T08:04:36.656 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:04:36 smithi050 ceph-mon[32371]: purged_snaps scrub ok 2024-09-18T08:04:36.656 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:04:36 smithi050 ceph-mon[32371]: pgmap v67: 148 pgs: 148 active+clean; 3.3 GiB data, 10 GiB used, 705 GiB / 715 GiB avail; 1.1 KiB/s rd, 110 MiB/s wr, 1.76k op/s 2024-09-18T08:04:36.656 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:04:36 smithi050 ceph-mon[32371]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-18T08:04:36.656 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:04:36 smithi050 ceph-mon[32371]: from='client.? 172.21.15.7:0/3601037845' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-18T08:04:36.656 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:04:36 smithi050 ceph-mon[32371]: osdmap e99: 8 total, 8 up, 8 in 2024-09-18T08:04:36.701 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:04:36 smithi007 ceph-mon[29459]: 4.13 scrub starts 2024-09-18T08:04:36.701 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:04:36 smithi007 ceph-mon[29459]: 4.13 scrub ok 2024-09-18T08:04:36.701 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:04:36 smithi007 ceph-mon[29459]: purged_snaps scrub starts 2024-09-18T08:04:36.701 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:04:36 smithi007 ceph-mon[29459]: purged_snaps scrub ok 2024-09-18T08:04:36.701 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:04:36 smithi007 ceph-mon[29459]: pgmap v67: 148 pgs: 148 active+clean; 3.3 GiB data, 10 GiB used, 705 GiB / 715 GiB avail; 1.1 KiB/s rd, 110 MiB/s wr, 1.76k op/s 2024-09-18T08:04:36.701 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:04:36 smithi007 ceph-mon[29459]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-18T08:04:36.701 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:04:36 smithi007 ceph-mon[29459]: from='client.? 172.21.15.7:0/3601037845' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-18T08:04:36.701 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:04:36 smithi007 ceph-mon[29459]: osdmap e99: 8 total, 8 up, 8 in 2024-09-18T08:04:36.702 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:04:36 smithi007 ceph-mon[39249]: 4.13 scrub starts 2024-09-18T08:04:36.702 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:04:36 smithi007 ceph-mon[39249]: 4.13 scrub ok 2024-09-18T08:04:36.702 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:04:36 smithi007 ceph-mon[39249]: purged_snaps scrub starts 2024-09-18T08:04:36.702 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:04:36 smithi007 ceph-mon[39249]: purged_snaps scrub ok 2024-09-18T08:04:36.702 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:04:36 smithi007 ceph-mon[39249]: pgmap v67: 148 pgs: 148 active+clean; 3.3 GiB data, 10 GiB used, 705 GiB / 715 GiB avail; 1.1 KiB/s rd, 110 MiB/s wr, 1.76k op/s 2024-09-18T08:04:36.702 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:04:36 smithi007 ceph-mon[39249]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-18T08:04:36.702 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:04:36 smithi007 ceph-mon[39249]: from='client.? 172.21.15.7:0/3601037845' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-18T08:04:36.702 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:04:36 smithi007 ceph-mon[39249]: osdmap e99: 8 total, 8 up, 8 in 2024-09-18T08:04:36.748 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1 2024-09-18T08:04:36.770 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:04:36.770 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:04:36.771 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1' 2024-09-18T08:04:36.871 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0 2024-09-18T08:04:36.911 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:04:36.912 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:04:36.912 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0' 2024-09-18T08:04:36.981 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Sep 18 08:04:36 smithi007 ceph-779349ca-7593-11ef-bceb-c7b262605968-mgr-y[29688]: ::ffff:172.21.15.50 - - [18/Sep/2024:08:04:36] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-09-18T08:04:37.013 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5 2024-09-18T08:04:37.041 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:04:37.042 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:04:37.042 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5' 2024-09-18T08:04:37.143 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7 2024-09-18T08:04:37.173 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:04:37.174 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:04:37.174 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7' 2024-09-18T08:04:37.275 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0 2024-09-18T08:04:37.312 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:04:37.312 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:04:37.313 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0' 2024-09-18T08:04:37.417 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7 2024-09-18T08:04:37.439 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:04:37.440 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:04:37.440 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7' 2024-09-18T08:04:37.541 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0 2024-09-18T08:04:37.572 INFO:tasks.radosbench.radosbench.0.smithi007.stdout: 32 16 59499 59483 116.158 73.0625 0.00567159 0.00860276 2024-09-18T08:04:37.580 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:04:37.581 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:04:37.581 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0' 2024-09-18T08:04:37.682 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3 2024-09-18T08:04:37.725 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:04:37.726 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:04:37.726 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3' 2024-09-18T08:04:37.827 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5 2024-09-18T08:04:37.854 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:04:37.855 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:04:37.855 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5' 2024-09-18T08:04:37.956 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4 2024-09-18T08:04:37.984 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:04:37.985 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:04:37.985 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4' 2024-09-18T08:04:38.086 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1 2024-09-18T08:04:38.119 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:04:38.119 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:04:38.120 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1' 2024-09-18T08:04:38.221 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4 2024-09-18T08:04:38.251 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:04:38.252 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:04:38.252 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4' 2024-09-18T08:04:38.353 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6 2024-09-18T08:04:38.382 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:04:38.383 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:04:38.383 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6' 2024-09-18T08:04:38.484 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2 2024-09-18T08:04:38.522 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:04:38.523 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:04:38.524 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2' 2024-09-18T08:04:38.572 INFO:tasks.radosbench.radosbench.0.smithi007.stdout: 33 16 60610 60594 114.742 69.4375 0.00503007 0.00871043 2024-09-18T08:04:38.577 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:04:38 smithi050 ceph-mon[32371]: purged_snaps scrub starts 2024-09-18T08:04:38.577 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:04:38 smithi050 ceph-mon[32371]: purged_snaps scrub ok 2024-09-18T08:04:38.577 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:04:38 smithi050 ceph-mon[32371]: pgmap v69: 148 pgs: 148 active+clean; 3.3 GiB data, 10 GiB used, 705 GiB / 715 GiB avail; 639 B/s rd, 58 MiB/s wr, 922 op/s 2024-09-18T08:04:38.577 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:04:38 smithi050 ceph-mon[32371]: osdmap e100: 8 total, 8 up, 8 in 2024-09-18T08:04:38.625 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6 2024-09-18T08:04:38.655 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:04:38.655 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:04:38.656 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6' 2024-09-18T08:04:38.730 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:04:38 smithi007 ceph-mon[29459]: purged_snaps scrub starts 2024-09-18T08:04:38.731 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:04:38 smithi007 ceph-mon[29459]: purged_snaps scrub ok 2024-09-18T08:04:38.731 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:04:38 smithi007 ceph-mon[29459]: pgmap v69: 148 pgs: 148 active+clean; 3.3 GiB data, 10 GiB used, 705 GiB / 715 GiB avail; 639 B/s rd, 58 MiB/s wr, 922 op/s 2024-09-18T08:04:38.731 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:04:38 smithi007 ceph-mon[29459]: osdmap e100: 8 total, 8 up, 8 in 2024-09-18T08:04:38.731 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:04:38 smithi007 ceph-mon[39249]: purged_snaps scrub starts 2024-09-18T08:04:38.731 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:04:38 smithi007 ceph-mon[39249]: purged_snaps scrub ok 2024-09-18T08:04:38.732 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:04:38 smithi007 ceph-mon[39249]: pgmap v69: 148 pgs: 148 active+clean; 3.3 GiB data, 10 GiB used, 705 GiB / 715 GiB avail; 639 B/s rd, 58 MiB/s wr, 922 op/s 2024-09-18T08:04:38.732 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:04:38 smithi007 ceph-mon[39249]: osdmap e100: 8 total, 8 up, 8 in 2024-09-18T08:04:38.756 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3 2024-09-18T08:04:38.797 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:04:38.798 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:04:38.798 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3' 2024-09-18T08:04:38.899 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4 2024-09-18T08:04:38.906 INFO:journalctl@ceph.iscsi.iscsi.a.smithi050.stdout:Sep 18 08:04:38 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-iscsi-iscsi-a[57782]: debug there is no tcmu-runner data available 2024-09-18T08:04:38.928 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:04:38.928 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:04:38.929 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4' 2024-09-18T08:04:39.029 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6 2024-09-18T08:04:39.059 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:04:39.060 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:04:39.061 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6' 2024-09-18T08:04:39.162 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5 2024-09-18T08:04:39.191 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:04:39.191 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:04:39.192 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5' 2024-09-18T08:04:39.293 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7 2024-09-18T08:04:39.319 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:04:39.319 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:04:39.320 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7' 2024-09-18T08:04:39.420 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0 2024-09-18T08:04:39.464 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:04:39.464 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:04:39.465 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0' 2024-09-18T08:04:39.565 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0 2024-09-18T08:04:39.572 INFO:tasks.radosbench.radosbench.0.smithi007.stdout: 34 16 62096 62080 114.099 92.875 0.00754442 0.00875882 2024-09-18T08:04:39.607 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:04:39.607 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:04:39.608 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0' 2024-09-18T08:04:39.709 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4 2024-09-18T08:04:39.734 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:04:39.735 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:04:39.735 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4' 2024-09-18T08:04:39.809 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/779349ca-7593-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T08:04:39.811 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/779349ca-7593-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T08:04:39.818 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/779349ca-7593-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T08:04:39.836 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2 2024-09-18T08:04:39.904 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:04:39.905 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:04:39.905 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2' 2024-09-18T08:04:39.956 INFO:journalctl@ceph.alertmanager.a.smithi007.stdout:Sep 18 08:04:39 smithi007 systemd[1]: Stopping Ceph alertmanager.a for 779349ca-7593-11ef-bceb-c7b262605968... 2024-09-18T08:04:40.006 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3 2024-09-18T08:04:40.047 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:04:40.047 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:04:40.048 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3' 2024-09-18T08:04:40.149 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6 2024-09-18T08:04:40.178 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:04:40.178 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:04:40.179 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6' 2024-09-18T08:04:40.280 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1 2024-09-18T08:04:40.318 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:04:40.319 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:04:40.319 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1' 2024-09-18T08:04:40.420 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3 2024-09-18T08:04:40.464 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:04:40.465 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:04:40.465 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3' 2024-09-18T08:04:40.524 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:04:40 smithi007 ceph-mon[29459]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-18T08:04:40.525 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:04:40 smithi007 ceph-mon[29459]: pgmap v71: 148 pgs: 148 active+clean; 3.7 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 1.2 KiB/s rd, 107 MiB/s wr, 1.71k op/s 2024-09-18T08:04:40.525 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:04:40 smithi007 ceph-mon[39249]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-18T08:04:40.525 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:04:40 smithi007 ceph-mon[39249]: pgmap v71: 148 pgs: 148 active+clean; 3.7 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 1.2 KiB/s rd, 107 MiB/s wr, 1.71k op/s 2024-09-18T08:04:40.566 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4 2024-09-18T08:04:40.581 INFO:tasks.radosbench.radosbench.0.smithi007.stdout: 35 16 63363 63347 113.074 79.1875 0.00827987 0.00883858 2024-09-18T08:04:40.597 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:04:40.597 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:04:40.598 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4' 2024-09-18T08:04:40.656 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:04:40 smithi050 ceph-mon[32371]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-18T08:04:40.656 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:04:40 smithi050 ceph-mon[32371]: pgmap v71: 148 pgs: 148 active+clean; 3.7 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 1.2 KiB/s rd, 107 MiB/s wr, 1.71k op/s 2024-09-18T08:04:40.698 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1 2024-09-18T08:04:40.742 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:04:40.742 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:04:40.743 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1' 2024-09-18T08:04:40.844 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6 2024-09-18T08:04:40.871 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:04:40.872 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:04:40.872 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6' 2024-09-18T08:04:40.919 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid 779349ca-7593-11ef-bceb-c7b262605968 -- ceph osd set nodeep-scrub 2024-09-18T08:04:40.973 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5 2024-09-18T08:04:40.997 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:04:40.997 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:04:40.998 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5' 2024-09-18T08:04:41.073 INFO:teuthology.orchestra.run.smithi007.stdout:osd.0: {} 2024-09-18T08:04:41.073 INFO:teuthology.orchestra.run.smithi007.stdout:osd.1: {} 2024-09-18T08:04:41.074 INFO:teuthology.orchestra.run.smithi007.stdout:osd.2: {} 2024-09-18T08:04:41.074 INFO:teuthology.orchestra.run.smithi007.stdout:osd.3: {} 2024-09-18T08:04:41.074 INFO:teuthology.orchestra.run.smithi007.stdout:osd.4: {} 2024-09-18T08:04:41.074 INFO:teuthology.orchestra.run.smithi007.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-18T08:04:41.074 INFO:teuthology.orchestra.run.smithi007.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-18T08:04:41.074 INFO:teuthology.orchestra.run.smithi007.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-18T08:04:41.074 INFO:teuthology.orchestra.run.smithi007.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-18T08:04:41.074 INFO:teuthology.orchestra.run.smithi007.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-18T08:04:41.088 INFO:teuthology.orchestra.run.smithi007.stdout:osd.5: {} 2024-09-18T08:04:41.089 INFO:teuthology.orchestra.run.smithi007.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-18T08:04:41.099 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5 2024-09-18T08:04:41.125 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:04:41.126 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:04:41.126 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5' 2024-09-18T08:04:41.127 INFO:teuthology.orchestra.run.smithi007.stdout:osd.6: {} 2024-09-18T08:04:41.127 INFO:teuthology.orchestra.run.smithi007.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-18T08:04:41.168 INFO:teuthology.orchestra.run.smithi007.stdout:osd.7: {} 2024-09-18T08:04:41.168 INFO:teuthology.orchestra.run.smithi007.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-18T08:04:41.227 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3 2024-09-18T08:04:41.290 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:04:41.290 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:04:41.291 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3' 2024-09-18T08:04:41.392 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3 2024-09-18T08:04:41.435 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:04:41.436 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:04:41.436 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3' 2024-09-18T08:04:41.444 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:04:41 smithi007 ceph-mon[29459]: from='client.? 172.21.15.7:0/1995580119' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-18T08:04:41.444 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:04:41 smithi007 ceph-mon[39249]: from='client.? 172.21.15.7:0/1995580119' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-18T08:04:41.537 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6 2024-09-18T08:04:41.563 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:04:41.564 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:04:41.564 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6' 2024-09-18T08:04:41.581 INFO:tasks.radosbench.radosbench.0.smithi007.stdout: 36 16 64256 64240 111.484 55.8125 0.00434531 0.00896429 2024-09-18T08:04:41.656 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:04:41 smithi050 ceph-mon[32371]: from='client.? 172.21.15.7:0/1995580119' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-18T08:04:41.656 INFO:journalctl@ceph.grafana.a.smithi050.stdout:Sep 18 08:04:41 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-grafana-a[60403]: logger=infra.usagestats t=2024-09-18T08:04:41.510783139Z level=info msg="Usage stats are ready to report" 2024-09-18T08:04:41.665 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7 2024-09-18T08:04:41.692 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:04:41.693 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:04:41.693 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7' 2024-09-18T08:04:41.714 INFO:teuthology.orchestra.run.smithi007.stdout:{ 2024-09-18T08:04:41.714 INFO:teuthology.orchestra.run.smithi007.stdout: "size": 20, 2024-09-18T08:04:41.714 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 600, 2024-09-18T08:04:41.714 INFO:teuthology.orchestra.run.smithi007.stdout: "ops": [ 2024-09-18T08:04:41.714 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:04:41.714 INFO:teuthology.orchestra.run.smithi007.stdout: "description": "osd_op(client.14805.0:266 7.8 7:12a84d33:::benchmark_data_smithi007_73155_object265:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e88)", 2024-09-18T08:04:41.714 INFO:teuthology.orchestra.run.smithi007.stdout: "initiated_at": "2024-09-18T08:04:05.682588+0000", 2024-09-18T08:04:41.715 INFO:teuthology.orchestra.run.smithi007.stdout: "age": 35.757532585999996, 2024-09-18T08:04:41.715 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.031631871999999998, 2024-09-18T08:04:41.715 INFO:teuthology.orchestra.run.smithi007.stdout: "continuous": false, 2024-09-18T08:04:41.715 INFO:teuthology.orchestra.run.smithi007.stdout: "type_data": { 2024-09-18T08:04:41.715 INFO:teuthology.orchestra.run.smithi007.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T08:04:41.715 INFO:teuthology.orchestra.run.smithi007.stdout: "client_info": { 2024-09-18T08:04:41.715 INFO:teuthology.orchestra.run.smithi007.stdout: "client": "client.14805", 2024-09-18T08:04:41.715 INFO:teuthology.orchestra.run.smithi007.stdout: "client_addr": "172.21.15.7:0/640320436", 2024-09-18T08:04:41.715 INFO:teuthology.orchestra.run.smithi007.stdout: "tid": 266 2024-09-18T08:04:41.715 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:04:41.715 INFO:teuthology.orchestra.run.smithi007.stdout: "events": [ 2024-09-18T08:04:41.715 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:04:41.715 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "initiated", 2024-09-18T08:04:41.716 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:04:05.682588+0000", 2024-09-18T08:04:41.716 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-09-18T08:04:41.716 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:04:41.716 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:04:41.716 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "header_read", 2024-09-18T08:04:41.716 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:04:05.682588+0000", 2024-09-18T08:04:41.716 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-09-18T08:04:41.716 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:04:41.716 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:04:41.716 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "throttled", 2024-09-18T08:04:41.716 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:04:05.682588+0000", 2024-09-18T08:04:41.716 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 5.1900000000000003e-07 2024-09-18T08:04:41.716 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:04:41.716 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:04:41.717 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "all_read", 2024-09-18T08:04:41.717 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:04:05.682636+0000", 2024-09-18T08:04:41.717 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 4.7342999999999997e-05 2024-09-18T08:04:41.717 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:04:41.717 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:04:41.717 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "dispatched", 2024-09-18T08:04:41.717 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:04:05.682642+0000", 2024-09-18T08:04:41.717 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 6.6560000000000003e-06 2024-09-18T08:04:41.717 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:04:41.717 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:04:41.717 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "queued_for_pg", 2024-09-18T08:04:41.717 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:04:05.682650+0000", 2024-09-18T08:04:41.718 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 7.8620000000000003e-06 2024-09-18T08:04:41.718 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:04:41.718 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:04:41.718 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-09-18T08:04:41.718 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:04:05.682981+0000", 2024-09-18T08:04:41.718 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.00033140300000000003 2024-09-18T08:04:41.718 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:04:41.718 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:04:41.718 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "started", 2024-09-18T08:04:41.718 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:04:05.683119+0000", 2024-09-18T08:04:41.718 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.00013752799999999999 2024-09-18T08:04:41.718 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:04:41.719 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:04:41.719 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "waiting for subops from 4,7", 2024-09-18T08:04:41.719 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:04:05.683263+0000", 2024-09-18T08:04:41.719 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.000144153 2024-09-18T08:04:41.719 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:04:41.719 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:04:41.719 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "op_commit", 2024-09-18T08:04:41.719 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:04:05.683659+0000", 2024-09-18T08:04:41.719 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.00039573800000000003 2024-09-18T08:04:41.719 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:04:41.719 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:04:41.719 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:04:41.719 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:04:05.686894+0000", 2024-09-18T08:04:41.720 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.0032354699999999998 2024-09-18T08:04:41.720 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:04:41.720 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:04:41.720 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:04:41.720 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:04:05.714104+0000", 2024-09-18T08:04:41.720 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.027209977 2024-09-18T08:04:41.720 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:04:41.720 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:04:41.720 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "commit_sent", 2024-09-18T08:04:41.720 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:04:05.714169+0000", 2024-09-18T08:04:41.720 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 6.4222999999999998e-05 2024-09-18T08:04:41.720 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:04:41.720 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:04:41.721 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "done", 2024-09-18T08:04:41.721 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:04:05.714220+0000", 2024-09-18T08:04:41.721 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 5.1e-05 2024-09-18T08:04:41.721 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-09-18T08:04:41.721 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-09-18T08:04:41.721 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-09-18T08:04:41.721 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:04:41.721 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:04:41.722 INFO:teuthology.orchestra.run.smithi007.stdout: "description": "osd_op(client.14805.0:2082 7.8 7:1f5c9847:::benchmark_data_smithi007_73155_object2081:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e88)", 2024-09-18T08:04:41.722 INFO:teuthology.orchestra.run.smithi007.stdout: "initiated_at": "2024-09-18T08:04:06.386302+0000", 2024-09-18T08:04:41.722 INFO:teuthology.orchestra.run.smithi007.stdout: "age": 35.053818221, 2024-09-18T08:04:41.722 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.034101237, 2024-09-18T08:04:41.722 INFO:teuthology.orchestra.run.smithi007.stdout: "continuous": false, 2024-09-18T08:04:41.722 INFO:teuthology.orchestra.run.smithi007.stdout: "type_data": { 2024-09-18T08:04:41.722 INFO:teuthology.orchestra.run.smithi007.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T08:04:41.722 INFO:teuthology.orchestra.run.smithi007.stdout: "client_info": { 2024-09-18T08:04:41.722 INFO:teuthology.orchestra.run.smithi007.stdout: "client": "client.14805", 2024-09-18T08:04:41.722 INFO:teuthology.orchestra.run.smithi007.stdout: "client_addr": "172.21.15.7:0/640320436", 2024-09-18T08:04:41.722 INFO:teuthology.orchestra.run.smithi007.stdout: "tid": 2082 2024-09-18T08:04:41.722 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:04:41.722 INFO:teuthology.orchestra.run.smithi007.stdout: "events": [ 2024-09-18T08:04:41.722 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:04:41.723 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "initiated", 2024-09-18T08:04:41.723 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:04:06.386302+0000", 2024-09-18T08:04:41.723 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-09-18T08:04:41.723 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:04:41.723 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:04:41.723 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "header_read", 2024-09-18T08:04:41.723 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:04:06.386302+0000", 2024-09-18T08:04:41.723 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-09-18T08:04:41.723 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:04:41.723 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:04:41.723 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "throttled", 2024-09-18T08:04:41.723 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:04:06.386303+0000", 2024-09-18T08:04:41.723 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.393e-06 2024-09-18T08:04:41.723 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:04:41.724 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:04:41.724 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "all_read", 2024-09-18T08:04:41.724 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:04:06.386325+0000", 2024-09-18T08:04:41.724 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 2.1107999999999998e-05 2024-09-18T08:04:41.724 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:04:41.724 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:04:41.724 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "dispatched", 2024-09-18T08:04:41.724 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:04:06.386339+0000", 2024-09-18T08:04:41.724 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.4919999999999999e-05 2024-09-18T08:04:41.724 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:04:41.724 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:04:41.724 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "queued_for_pg", 2024-09-18T08:04:41.724 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:04:06.386346+0000", 2024-09-18T08:04:41.724 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 7.0260000000000002e-06 2024-09-18T08:04:41.725 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:04:41.725 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:04:41.725 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-09-18T08:04:41.725 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:04:06.386410+0000", 2024-09-18T08:04:41.725 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 6.3493999999999999e-05 2024-09-18T08:04:41.725 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:04:41.725 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:04:41.725 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "started", 2024-09-18T08:04:41.725 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:04:06.386546+0000", 2024-09-18T08:04:41.725 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.000136196 2024-09-18T08:04:41.725 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:04:41.725 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:04:41.725 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "waiting for subops from 4,7", 2024-09-18T08:04:41.725 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:04:06.386682+0000", 2024-09-18T08:04:41.725 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.000135885 2024-09-18T08:04:41.726 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:04:41.726 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:04:41.726 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "op_commit", 2024-09-18T08:04:41.726 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:04:06.389009+0000", 2024-09-18T08:04:41.726 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.002327254 2024-09-18T08:04:41.726 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:04:41.726 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:04:41.726 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:04:41.726 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:04:06.399398+0000", 2024-09-18T08:04:41.726 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.010388243 2024-09-18T08:04:41.726 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:04:41.726 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:04:41.726 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:04:41.726 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:04:06.420309+0000", 2024-09-18T08:04:41.727 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.020910946999999999 2024-09-18T08:04:41.727 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:04:41.727 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:04:41.727 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "commit_sent", 2024-09-18T08:04:41.727 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:04:06.420364+0000", 2024-09-18T08:04:41.727 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 5.5220999999999999e-05 2024-09-18T08:04:41.727 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:04:41.727 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:04:41.727 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "done", 2024-09-18T08:04:41.727 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:04:06.420403+0000", 2024-09-18T08:04:41.727 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 3.9549999999999999e-05 2024-09-18T08:04:41.727 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-09-18T08:04:41.727 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-09-18T08:04:41.728 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-09-18T08:04:41.728 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:04:41.728 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:04:41.728 INFO:teuthology.orchestra.run.smithi007.stdout: "description": "osd_op(client.14805.0:2102 7.8 7:1adcbdc6:::benchmark_data_smithi007_73155_object2101:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e88)", 2024-09-18T08:04:41.728 INFO:teuthology.orchestra.run.smithi007.stdout: "initiated_at": "2024-09-18T08:04:06.394337+0000", 2024-09-18T08:04:41.728 INFO:teuthology.orchestra.run.smithi007.stdout: "age": 35.045783047, 2024-09-18T08:04:41.728 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.026200850000000001, 2024-09-18T08:04:41.728 INFO:teuthology.orchestra.run.smithi007.stdout: "continuous": false, 2024-09-18T08:04:41.728 INFO:teuthology.orchestra.run.smithi007.stdout: "type_data": { 2024-09-18T08:04:41.728 INFO:teuthology.orchestra.run.smithi007.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T08:04:41.728 INFO:teuthology.orchestra.run.smithi007.stdout: "client_info": { 2024-09-18T08:04:41.728 INFO:teuthology.orchestra.run.smithi007.stdout: "client": "client.14805", 2024-09-18T08:04:41.728 INFO:teuthology.orchestra.run.smithi007.stdout: "client_addr": "172.21.15.7:0/640320436", 2024-09-18T08:04:41.729 INFO:teuthology.orchestra.run.smithi007.stdout: "tid": 2102 2024-09-18T08:04:41.729 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:04:41.729 INFO:teuthology.orchestra.run.smithi007.stdout: "events": [ 2024-09-18T08:04:41.729 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:04:41.729 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "initiated", 2024-09-18T08:04:41.729 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:04:06.394337+0000", 2024-09-18T08:04:41.729 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-09-18T08:04:41.729 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:04:41.729 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:04:41.730 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "header_read", 2024-09-18T08:04:41.730 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:04:06.394337+0000", 2024-09-18T08:04:41.730 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-09-18T08:04:41.730 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:04:41.730 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:04:41.730 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "throttled", 2024-09-18T08:04:41.730 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:04:06.394338+0000", 2024-09-18T08:04:41.730 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.0699999999999999e-06 2024-09-18T08:04:41.730 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:04:41.730 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:04:41.730 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "all_read", 2024-09-18T08:04:41.730 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:04:06.394360+0000", 2024-09-18T08:04:41.730 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 2.1767e-05 2024-09-18T08:04:41.731 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:04:41.731 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:04:41.731 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "dispatched", 2024-09-18T08:04:41.731 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:04:06.394370+0000", 2024-09-18T08:04:41.731 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 9.7960000000000007e-06 2024-09-18T08:04:41.731 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:04:41.731 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:04:41.731 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "queued_for_pg", 2024-09-18T08:04:41.731 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:04:06.394377+0000", 2024-09-18T08:04:41.731 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 6.6819999999999997e-06 2024-09-18T08:04:41.731 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:04:41.731 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:04:41.731 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-09-18T08:04:41.732 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:04:06.394446+0000", 2024-09-18T08:04:41.732 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 6.9820000000000006e-05 2024-09-18T08:04:41.732 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:04:41.732 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:04:41.732 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "started", 2024-09-18T08:04:41.732 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:04:06.394539+0000", 2024-09-18T08:04:41.732 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 9.2239999999999998e-05 2024-09-18T08:04:41.732 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:04:41.732 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:04:41.732 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "waiting for subops from 4,7", 2024-09-18T08:04:41.732 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:04:06.394647+0000", 2024-09-18T08:04:41.732 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.00010856 2024-09-18T08:04:41.732 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:04:41.733 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:04:41.733 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "op_commit", 2024-09-18T08:04:41.733 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:04:06.395837+0000", 2024-09-18T08:04:41.733 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.001189811 2024-09-18T08:04:41.733 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:04:41.733 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:04:41.733 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:04:41.733 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:04:06.399434+0000", 2024-09-18T08:04:41.733 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.0035973300000000001 2024-09-18T08:04:41.733 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:04:41.733 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:04:41.733 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:04:41.733 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:04:06.420484+0000", 2024-09-18T08:04:41.733 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.021049223999999998 2024-09-18T08:04:41.734 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:04:41.734 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:04:41.734 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "commit_sent", 2024-09-18T08:04:41.734 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:04:06.420516+0000", 2024-09-18T08:04:41.734 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 3.2199000000000001e-05 2024-09-18T08:04:41.734 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:04:41.734 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:04:41.734 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "done", 2024-09-18T08:04:41.734 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:04:06.420538+0000", 2024-09-18T08:04:41.734 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 2.2351e-05 2024-09-18T08:04:41.734 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-09-18T08:04:41.734 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-09-18T08:04:41.734 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-09-18T08:04:41.734 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:04:41.735 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:04:41.735 INFO:teuthology.orchestra.run.smithi007.stdout: "description": "osd_op(client.14805.0:39813 7.8 7:1829805f:::benchmark_data_smithi007_73155_object39812:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e96)", 2024-09-18T08:04:41.735 INFO:teuthology.orchestra.run.smithi007.stdout: "initiated_at": "2024-09-18T08:04:24.073612+0000", 2024-09-18T08:04:41.735 INFO:teuthology.orchestra.run.smithi007.stdout: "age": 17.366508233000001, 2024-09-18T08:04:41.735 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.042014173000000002, 2024-09-18T08:04:41.735 INFO:teuthology.orchestra.run.smithi007.stdout: "continuous": false, 2024-09-18T08:04:41.735 INFO:teuthology.orchestra.run.smithi007.stdout: "type_data": { 2024-09-18T08:04:41.735 INFO:teuthology.orchestra.run.smithi007.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T08:04:41.735 INFO:teuthology.orchestra.run.smithi007.stdout: "client_info": { 2024-09-18T08:04:41.735 INFO:teuthology.orchestra.run.smithi007.stdout: "client": "client.14805", 2024-09-18T08:04:41.735 INFO:teuthology.orchestra.run.smithi007.stdout: "client_addr": "172.21.15.7:0/640320436", 2024-09-18T08:04:41.735 INFO:teuthology.orchestra.run.smithi007.stdout: "tid": 39813 2024-09-18T08:04:41.735 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:04:41.735 INFO:teuthology.orchestra.run.smithi007.stdout: "events": [ 2024-09-18T08:04:41.736 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:04:41.736 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "initiated", 2024-09-18T08:04:41.736 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:04:24.073612+0000", 2024-09-18T08:04:41.736 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-09-18T08:04:41.736 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:04:41.736 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:04:41.736 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "header_read", 2024-09-18T08:04:41.736 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:04:24.073612+0000", 2024-09-18T08:04:41.736 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-09-18T08:04:41.736 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:04:41.736 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:04:41.736 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "throttled", 2024-09-18T08:04:41.736 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:04:24.073614+0000", 2024-09-18T08:04:41.737 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.61e-06 2024-09-18T08:04:41.737 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:04:41.737 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:04:41.737 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "all_read", 2024-09-18T08:04:41.737 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:04:24.073637+0000", 2024-09-18T08:04:41.737 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 2.3536e-05 2024-09-18T08:04:41.737 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:04:41.737 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:04:41.737 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "dispatched", 2024-09-18T08:04:41.737 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:04:24.073647+0000", 2024-09-18T08:04:41.737 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.0183999999999999e-05 2024-09-18T08:04:41.737 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:04:41.737 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:04:41.737 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "queued_for_pg", 2024-09-18T08:04:41.738 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:04:24.073655+0000", 2024-09-18T08:04:41.738 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 7.2300000000000002e-06 2024-09-18T08:04:41.738 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:04:41.738 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:04:41.738 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-09-18T08:04:41.738 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:04:24.073715+0000", 2024-09-18T08:04:41.738 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 6.0807999999999998e-05 2024-09-18T08:04:41.738 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:04:41.738 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:04:41.738 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "started", 2024-09-18T08:04:41.738 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:04:24.073839+0000", 2024-09-18T08:04:41.738 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.00012399399999999999 2024-09-18T08:04:41.738 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:04:41.739 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:04:41.739 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "waiting for subops from 4,7", 2024-09-18T08:04:41.739 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:04:24.074007+0000", 2024-09-18T08:04:41.739 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.00016740599999999999 2024-09-18T08:04:41.739 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:04:41.739 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:04:41.739 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "op_commit", 2024-09-18T08:04:41.739 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:04:24.076155+0000", 2024-09-18T08:04:41.739 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.0021486729999999998 2024-09-18T08:04:41.739 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:04:41.739 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:04:41.739 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:04:41.739 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:04:24.101198+0000", 2024-09-18T08:04:41.739 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.025042754 2024-09-18T08:04:41.740 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:04:41.740 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:04:41.740 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:04:41.740 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:04:24.115559+0000", 2024-09-18T08:04:41.740 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.014360576 2024-09-18T08:04:41.740 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:04:41.740 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:04:41.740 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "commit_sent", 2024-09-18T08:04:41.740 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:04:24.115596+0000", 2024-09-18T08:04:41.740 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 3.7588e-05 2024-09-18T08:04:41.740 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:04:41.740 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:04:41.740 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "done", 2024-09-18T08:04:41.740 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:04:24.115626+0000", 2024-09-18T08:04:41.741 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 2.9813999999999999e-05 2024-09-18T08:04:41.741 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-09-18T08:04:41.741 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-09-18T08:04:41.741 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-09-18T08:04:41.741 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:04:41.741 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:04:41.741 INFO:teuthology.orchestra.run.smithi007.stdout: "description": "osd_op(client.14805.0:42753 7.8 7:192db7d7:::benchmark_data_smithi007_73155_object42752:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e96)", 2024-09-18T08:04:41.741 INFO:teuthology.orchestra.run.smithi007.stdout: "initiated_at": "2024-09-18T08:04:25.828423+0000", 2024-09-18T08:04:41.741 INFO:teuthology.orchestra.run.smithi007.stdout: "age": 15.61169683, 2024-09-18T08:04:41.741 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.029969326000000001, 2024-09-18T08:04:41.741 INFO:teuthology.orchestra.run.smithi007.stdout: "continuous": false, 2024-09-18T08:04:41.741 INFO:teuthology.orchestra.run.smithi007.stdout: "type_data": { 2024-09-18T08:04:41.741 INFO:teuthology.orchestra.run.smithi007.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T08:04:41.741 INFO:teuthology.orchestra.run.smithi007.stdout: "client_info": { 2024-09-18T08:04:41.742 INFO:teuthology.orchestra.run.smithi007.stdout: "client": "client.14805", 2024-09-18T08:04:41.742 INFO:teuthology.orchestra.run.smithi007.stdout: "client_addr": "172.21.15.7:0/640320436", 2024-09-18T08:04:41.742 INFO:teuthology.orchestra.run.smithi007.stdout: "tid": 42753 2024-09-18T08:04:41.742 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:04:41.742 INFO:teuthology.orchestra.run.smithi007.stdout: "events": [ 2024-09-18T08:04:41.742 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:04:41.742 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "initiated", 2024-09-18T08:04:41.742 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:04:25.828423+0000", 2024-09-18T08:04:41.742 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-09-18T08:04:41.742 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:04:41.742 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:04:41.742 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "header_read", 2024-09-18T08:04:41.742 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:04:25.828423+0000", 2024-09-18T08:04:41.742 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-09-18T08:04:41.742 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:04:41.743 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:04:41.743 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "throttled", 2024-09-18T08:04:41.743 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:04:25.828426+0000", 2024-09-18T08:04:41.743 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 2.131e-06 2024-09-18T08:04:41.743 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:04:41.743 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:04:41.743 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "all_read", 2024-09-18T08:04:41.743 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:04:25.828452+0000", 2024-09-18T08:04:41.743 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 2.6338000000000001e-05 2024-09-18T08:04:41.743 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:04:41.743 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:04:41.743 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "dispatched", 2024-09-18T08:04:41.743 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:04:25.828472+0000", 2024-09-18T08:04:41.744 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.963e-05 2024-09-18T08:04:41.744 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:04:41.744 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:04:41.744 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "queued_for_pg", 2024-09-18T08:04:41.744 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:04:25.828480+0000", 2024-09-18T08:04:41.744 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 8.8689999999999993e-06 2024-09-18T08:04:41.744 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:04:41.744 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:04:41.744 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-09-18T08:04:41.744 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:04:25.828576+0000", 2024-09-18T08:04:41.744 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 9.5564999999999997e-05 2024-09-18T08:04:41.744 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:04:41.744 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:04:41.744 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "started", 2024-09-18T08:04:41.745 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:04:25.828719+0000", 2024-09-18T08:04:41.745 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.000143072 2024-09-18T08:04:41.745 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:04:41.745 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:04:41.745 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "waiting for subops from 4,7", 2024-09-18T08:04:41.745 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:04:25.828873+0000", 2024-09-18T08:04:41.745 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.000153862 2024-09-18T08:04:41.745 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:04:41.745 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:04:41.745 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:04:41.745 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:04:25.830634+0000", 2024-09-18T08:04:41.745 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.001760852 2024-09-18T08:04:41.745 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:04:41.745 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:04:41.746 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "op_commit", 2024-09-18T08:04:41.746 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:04:25.835082+0000", 2024-09-18T08:04:41.746 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.00444853 2024-09-18T08:04:41.746 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:04:41.746 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:04:41.746 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:04:41.746 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:04:25.858269+0000", 2024-09-18T08:04:41.746 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.023187149000000001 2024-09-18T08:04:41.746 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:04:41.746 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:04:41.746 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "commit_sent", 2024-09-18T08:04:41.746 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:04:25.858339+0000", 2024-09-18T08:04:41.746 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 6.9889999999999997e-05 2024-09-18T08:04:41.746 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:04:41.746 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:04:41.747 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "done", 2024-09-18T08:04:41.747 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:04:25.858393+0000", 2024-09-18T08:04:41.747 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 5.3437999999999999e-05 2024-09-18T08:04:41.747 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-09-18T08:04:41.747 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-09-18T08:04:41.747 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-09-18T08:04:41.747 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:04:41.747 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:04:41.747 INFO:teuthology.orchestra.run.smithi007.stdout: "description": "osd_op(client.14805.0:42756 7.8 7:10cde384:::benchmark_data_smithi007_73155_object42755:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e96)", 2024-09-18T08:04:41.747 INFO:teuthology.orchestra.run.smithi007.stdout: "initiated_at": "2024-09-18T08:04:25.830889+0000", 2024-09-18T08:04:41.747 INFO:teuthology.orchestra.run.smithi007.stdout: "age": 15.609230993000001, 2024-09-18T08:04:41.747 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.030863092000000002, 2024-09-18T08:04:41.747 INFO:teuthology.orchestra.run.smithi007.stdout: "continuous": false, 2024-09-18T08:04:41.747 INFO:teuthology.orchestra.run.smithi007.stdout: "type_data": { 2024-09-18T08:04:41.748 INFO:teuthology.orchestra.run.smithi007.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T08:04:41.748 INFO:teuthology.orchestra.run.smithi007.stdout: "client_info": { 2024-09-18T08:04:41.748 INFO:teuthology.orchestra.run.smithi007.stdout: "client": "client.14805", 2024-09-18T08:04:41.748 INFO:teuthology.orchestra.run.smithi007.stdout: "client_addr": "172.21.15.7:0/640320436", 2024-09-18T08:04:41.748 INFO:teuthology.orchestra.run.smithi007.stdout: "tid": 42756 2024-09-18T08:04:41.748 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:04:41.748 INFO:teuthology.orchestra.run.smithi007.stdout: "events": [ 2024-09-18T08:04:41.748 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:04:41.748 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "initiated", 2024-09-18T08:04:41.748 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:04:25.830889+0000", 2024-09-18T08:04:41.748 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-09-18T08:04:41.748 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:04:41.749 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:04:41.749 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "header_read", 2024-09-18T08:04:41.749 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:04:25.830889+0000", 2024-09-18T08:04:41.749 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-09-18T08:04:41.749 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:04:41.749 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:04:41.749 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "throttled", 2024-09-18T08:04:41.749 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:04:25.830891+0000", 2024-09-18T08:04:41.749 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 2.2010000000000002e-06 2024-09-18T08:04:41.749 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:04:41.749 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:04:41.749 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "all_read", 2024-09-18T08:04:41.749 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:04:25.830924+0000", 2024-09-18T08:04:41.750 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 3.2920000000000003e-05 2024-09-18T08:04:41.750 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:04:41.750 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:04:41.750 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "dispatched", 2024-09-18T08:04:41.750 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:04:25.830937+0000", 2024-09-18T08:04:41.750 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.2449999999999999e-05 2024-09-18T08:04:41.750 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:04:41.750 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:04:41.750 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "queued_for_pg", 2024-09-18T08:04:41.750 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:04:25.830945+0000", 2024-09-18T08:04:41.750 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 7.7470000000000005e-06 2024-09-18T08:04:41.750 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:04:41.750 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:04:41.751 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-09-18T08:04:41.751 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:04:25.831002+0000", 2024-09-18T08:04:41.751 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 5.7673000000000002e-05 2024-09-18T08:04:41.751 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:04:41.751 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:04:41.751 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "started", 2024-09-18T08:04:41.751 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:04:25.831151+0000", 2024-09-18T08:04:41.751 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.000148524 2024-09-18T08:04:41.751 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:04:41.751 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:04:41.751 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "waiting for subops from 4,7", 2024-09-18T08:04:41.751 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:04:25.831295+0000", 2024-09-18T08:04:41.751 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.00014429199999999999 2024-09-18T08:04:41.751 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:04:41.752 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:04:41.752 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:04:41.752 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:04:25.833389+0000", 2024-09-18T08:04:41.752 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.0020941179999999998 2024-09-18T08:04:41.752 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:04:41.752 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:04:41.752 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "op_commit", 2024-09-18T08:04:41.752 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:04:25.835740+0000", 2024-09-18T08:04:41.752 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.0023510029999999999 2024-09-18T08:04:41.752 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:04:41.752 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:04:41.752 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:04:41.752 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:04:25.861597+0000", 2024-09-18T08:04:41.753 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.025856752 2024-09-18T08:04:41.753 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:04:41.753 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:04:41.753 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "commit_sent", 2024-09-18T08:04:41.753 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:04:25.861698+0000", 2024-09-18T08:04:41.753 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.00010074200000000001 2024-09-18T08:04:41.753 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:04:41.753 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:04:41.753 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "done", 2024-09-18T08:04:41.753 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:04:25.861752+0000", 2024-09-18T08:04:41.753 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 5.4669999999999997e-05 2024-09-18T08:04:41.753 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-09-18T08:04:41.753 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-09-18T08:04:41.753 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-09-18T08:04:41.754 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:04:41.754 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:04:41.754 INFO:teuthology.orchestra.run.smithi007.stdout: "description": "osd_op(client.14805.0:63810 7.8 7:1962f114:::benchmark_data_smithi007_73155_object63809:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e100)", 2024-09-18T08:04:41.754 INFO:teuthology.orchestra.run.smithi007.stdout: "initiated_at": "2024-09-18T08:04:41.058121+0000", 2024-09-18T08:04:41.754 INFO:teuthology.orchestra.run.smithi007.stdout: "age": 0.38199965000000002, 2024-09-18T08:04:41.754 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.033297535000000003, 2024-09-18T08:04:41.754 INFO:teuthology.orchestra.run.smithi007.stdout: "continuous": false, 2024-09-18T08:04:41.754 INFO:teuthology.orchestra.run.smithi007.stdout: "type_data": { 2024-09-18T08:04:41.754 INFO:teuthology.orchestra.run.smithi007.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T08:04:41.754 INFO:teuthology.orchestra.run.smithi007.stdout: "client_info": { 2024-09-18T08:04:41.754 INFO:teuthology.orchestra.run.smithi007.stdout: "client": "client.14805", 2024-09-18T08:04:41.754 INFO:teuthology.orchestra.run.smithi007.stdout: "client_addr": "172.21.15.7:0/640320436", 2024-09-18T08:04:41.754 INFO:teuthology.orchestra.run.smithi007.stdout: "tid": 63810 2024-09-18T08:04:41.754 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:04:41.755 INFO:teuthology.orchestra.run.smithi007.stdout: "events": [ 2024-09-18T08:04:41.755 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:04:41.755 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "initiated", 2024-09-18T08:04:41.755 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:04:41.058121+0000", 2024-09-18T08:04:41.755 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-09-18T08:04:41.755 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:04:41.755 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:04:41.755 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "header_read", 2024-09-18T08:04:41.755 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:04:41.058121+0000", 2024-09-18T08:04:41.755 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-09-18T08:04:41.755 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:04:41.755 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:04:41.755 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "throttled", 2024-09-18T08:04:41.755 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:04:41.058123+0000", 2024-09-18T08:04:41.756 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 2.2220000000000001e-06 2024-09-18T08:04:41.756 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:04:41.756 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:04:41.756 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "all_read", 2024-09-18T08:04:41.756 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:04:41.058146+0000", 2024-09-18T08:04:41.756 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 2.2702999999999999e-05 2024-09-18T08:04:41.756 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:04:41.756 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:04:41.756 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "dispatched", 2024-09-18T08:04:41.756 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:04:41.058167+0000", 2024-09-18T08:04:41.756 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 2.1800000000000001e-05 2024-09-18T08:04:41.756 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:04:41.756 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:04:41.756 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "queued_for_pg", 2024-09-18T08:04:41.756 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:04:41.058177+0000", 2024-09-18T08:04:41.757 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.0006e-05 2024-09-18T08:04:41.757 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:04:41.757 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:04:41.757 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-09-18T08:04:41.757 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:04:41.068076+0000", 2024-09-18T08:04:41.757 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.0098986149999999995 2024-09-18T08:04:41.757 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:04:41.757 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:04:41.757 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "started", 2024-09-18T08:04:41.757 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:04:41.068227+0000", 2024-09-18T08:04:41.757 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.000151268 2024-09-18T08:04:41.757 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:04:41.757 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:04:41.758 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "waiting for subops from 4,7", 2024-09-18T08:04:41.758 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:04:41.068402+0000", 2024-09-18T08:04:41.758 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.00017499799999999999 2024-09-18T08:04:41.758 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:04:41.758 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:04:41.758 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:04:41.758 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:04:41.078233+0000", 2024-09-18T08:04:41.758 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.0098310250000000002 2024-09-18T08:04:41.758 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:04:41.758 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:04:41.758 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:04:41.758 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:04:41.078464+0000", 2024-09-18T08:04:41.758 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.000231232 2024-09-18T08:04:41.758 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:04:41.759 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:04:41.759 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "op_commit", 2024-09-18T08:04:41.759 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:04:41.091255+0000", 2024-09-18T08:04:41.759 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.012790401999999999 2024-09-18T08:04:41.759 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:04:41.759 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:04:41.759 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "commit_sent", 2024-09-18T08:04:41.759 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:04:41.091363+0000", 2024-09-18T08:04:41.759 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.000108009 2024-09-18T08:04:41.759 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:04:41.759 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:04:41.759 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "done", 2024-09-18T08:04:41.759 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:04:41.091418+0000", 2024-09-18T08:04:41.759 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 5.5254999999999999e-05 2024-09-18T08:04:41.760 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-09-18T08:04:41.760 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-09-18T08:04:41.760 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-09-18T08:04:41.760 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:04:41.760 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:04:41.760 INFO:teuthology.orchestra.run.smithi007.stdout: "description": "osd_op(client.14805.0:63812 7.e 7:75bbc3f3:::benchmark_data_smithi007_73155_object63811:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e100)", 2024-09-18T08:04:41.760 INFO:teuthology.orchestra.run.smithi007.stdout: "initiated_at": "2024-09-18T08:04:41.058253+0000", 2024-09-18T08:04:41.760 INFO:teuthology.orchestra.run.smithi007.stdout: "age": 0.38186688800000002, 2024-09-18T08:04:41.760 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.024870944999999998, 2024-09-18T08:04:41.760 INFO:teuthology.orchestra.run.smithi007.stdout: "continuous": false, 2024-09-18T08:04:41.760 INFO:teuthology.orchestra.run.smithi007.stdout: "type_data": { 2024-09-18T08:04:41.760 INFO:teuthology.orchestra.run.smithi007.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T08:04:41.760 INFO:teuthology.orchestra.run.smithi007.stdout: "client_info": { 2024-09-18T08:04:41.761 INFO:teuthology.orchestra.run.smithi007.stdout: "client": "client.14805", 2024-09-18T08:04:41.761 INFO:teuthology.orchestra.run.smithi007.stdout: "client_addr": "172.21.15.7:0/640320436", 2024-09-18T08:04:41.761 INFO:teuthology.orchestra.run.smithi007.stdout: "tid": 63812 2024-09-18T08:04:41.761 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:04:41.761 INFO:teuthology.orchestra.run.smithi007.stdout: "events": [ 2024-09-18T08:04:41.761 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:04:41.761 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "initiated", 2024-09-18T08:04:41.761 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:04:41.058253+0000", 2024-09-18T08:04:41.761 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-09-18T08:04:41.761 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:04:41.761 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:04:41.761 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "header_read", 2024-09-18T08:04:41.761 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:04:41.058253+0000", 2024-09-18T08:04:41.761 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-09-18T08:04:41.762 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:04:41.762 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:04:41.762 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "throttled", 2024-09-18T08:04:41.762 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:04:41.058254+0000", 2024-09-18T08:04:41.762 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 4.8599999999999998e-07 2024-09-18T08:04:41.762 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:04:41.762 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:04:41.762 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "all_read", 2024-09-18T08:04:41.762 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:04:41.058273+0000", 2024-09-18T08:04:41.762 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.9624e-05 2024-09-18T08:04:41.762 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:04:41.762 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:04:41.762 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "dispatched", 2024-09-18T08:04:41.762 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:04:41.058280+0000", 2024-09-18T08:04:41.763 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 6.5189999999999998e-06 2024-09-18T08:04:41.763 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:04:41.763 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:04:41.763 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "queued_for_pg", 2024-09-18T08:04:41.763 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:04:41.058286+0000", 2024-09-18T08:04:41.763 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 5.8150000000000002e-06 2024-09-18T08:04:41.763 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:04:41.763 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:04:41.763 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-09-18T08:04:41.763 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:04:41.061211+0000", 2024-09-18T08:04:41.763 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.002925647 2024-09-18T08:04:41.763 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:04:41.763 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:04:41.764 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "started", 2024-09-18T08:04:41.764 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:04:41.061334+0000", 2024-09-18T08:04:41.764 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.00012227700000000001 2024-09-18T08:04:41.764 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:04:41.764 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:04:41.764 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "waiting for subops from 0,5", 2024-09-18T08:04:41.764 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:04:41.061567+0000", 2024-09-18T08:04:41.764 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.00023335399999999999 2024-09-18T08:04:41.764 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:04:41.764 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:04:41.764 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:04:41.764 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:04:41.065238+0000", 2024-09-18T08:04:41.764 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.0036711460000000001 2024-09-18T08:04:41.764 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:04:41.765 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:04:41.765 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "op_commit", 2024-09-18T08:04:41.765 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:04:41.072541+0000", 2024-09-18T08:04:41.765 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.0073032139999999997 2024-09-18T08:04:41.765 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:04:41.765 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:04:41.765 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:04:41.765 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:04:41.082949+0000", 2024-09-18T08:04:41.765 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.010407434 2024-09-18T08:04:41.765 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:04:41.765 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:04:41.765 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "commit_sent", 2024-09-18T08:04:41.765 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:04:41.083084+0000", 2024-09-18T08:04:41.765 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.00013471199999999999 2024-09-18T08:04:41.766 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:04:41.766 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:04:41.766 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "done", 2024-09-18T08:04:41.766 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:04:41.083124+0000", 2024-09-18T08:04:41.766 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 4.0717000000000003e-05 2024-09-18T08:04:41.766 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-09-18T08:04:41.766 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-09-18T08:04:41.766 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-09-18T08:04:41.766 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:04:41.766 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:04:41.766 INFO:teuthology.orchestra.run.smithi007.stdout: "description": "osd_op(client.14805.0:63843 7.e 7:7023132e:::benchmark_data_smithi007_73155_object63842:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e100)", 2024-09-18T08:04:41.766 INFO:teuthology.orchestra.run.smithi007.stdout: "initiated_at": "2024-09-18T08:04:41.105445+0000", 2024-09-18T08:04:41.766 INFO:teuthology.orchestra.run.smithi007.stdout: "age": 0.33467486400000002, 2024-09-18T08:04:41.766 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.029243248999999999, 2024-09-18T08:04:41.767 INFO:teuthology.orchestra.run.smithi007.stdout: "continuous": false, 2024-09-18T08:04:41.767 INFO:teuthology.orchestra.run.smithi007.stdout: "type_data": { 2024-09-18T08:04:41.767 INFO:teuthology.orchestra.run.smithi007.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T08:04:41.767 INFO:teuthology.orchestra.run.smithi007.stdout: "client_info": { 2024-09-18T08:04:41.767 INFO:teuthology.orchestra.run.smithi007.stdout: "client": "client.14805", 2024-09-18T08:04:41.767 INFO:teuthology.orchestra.run.smithi007.stdout: "client_addr": "172.21.15.7:0/640320436", 2024-09-18T08:04:41.767 INFO:teuthology.orchestra.run.smithi007.stdout: "tid": 63843 2024-09-18T08:04:41.767 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:04:41.767 INFO:teuthology.orchestra.run.smithi007.stdout: "events": [ 2024-09-18T08:04:41.767 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:04:41.768 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "initiated", 2024-09-18T08:04:41.768 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:04:41.105445+0000", 2024-09-18T08:04:41.768 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-09-18T08:04:41.768 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:04:41.769 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:04:41.769 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "header_read", 2024-09-18T08:04:41.769 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:04:41.105445+0000", 2024-09-18T08:04:41.769 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-09-18T08:04:41.769 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:04:41.769 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:04:41.769 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "throttled", 2024-09-18T08:04:41.769 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:04:41.105447+0000", 2024-09-18T08:04:41.769 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.792e-06 2024-09-18T08:04:41.769 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:04:41.769 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:04:41.770 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "all_read", 2024-09-18T08:04:41.770 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:04:41.105477+0000", 2024-09-18T08:04:41.770 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 2.9461000000000001e-05 2024-09-18T08:04:41.770 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:04:41.770 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:04:41.770 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "dispatched", 2024-09-18T08:04:41.770 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:04:41.105499+0000", 2024-09-18T08:04:41.770 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 2.2738000000000002e-05 2024-09-18T08:04:41.770 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:04:41.770 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:04:41.770 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "queued_for_pg", 2024-09-18T08:04:41.770 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:04:41.105509+0000", 2024-09-18T08:04:41.770 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 9.5820000000000008e-06 2024-09-18T08:04:41.770 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:04:41.771 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:04:41.771 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-09-18T08:04:41.771 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:04:41.105627+0000", 2024-09-18T08:04:41.771 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.000117806 2024-09-18T08:04:41.771 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:04:41.771 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:04:41.771 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "started", 2024-09-18T08:04:41.771 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:04:41.105751+0000", 2024-09-18T08:04:41.771 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.000124067 2024-09-18T08:04:41.771 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:04:41.771 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:04:41.771 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "waiting for subops from 0,5", 2024-09-18T08:04:41.771 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:04:41.105899+0000", 2024-09-18T08:04:41.771 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.00014772799999999999 2024-09-18T08:04:41.772 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:04:41.772 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:04:41.772 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "op_commit", 2024-09-18T08:04:41.772 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:04:41.112006+0000", 2024-09-18T08:04:41.772 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.006107538 2024-09-18T08:04:41.772 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:04:41.772 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:04:41.772 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:04:41.772 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:04:41.113291+0000", 2024-09-18T08:04:41.772 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.001285313 2024-09-18T08:04:41.772 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:04:41.772 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:04:41.772 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:04:41.772 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:04:41.134323+0000", 2024-09-18T08:04:41.773 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.021031109999999999 2024-09-18T08:04:41.773 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:04:41.773 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:04:41.773 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "commit_sent", 2024-09-18T08:04:41.773 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:04:41.134445+0000", 2024-09-18T08:04:41.773 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.00012262199999999999 2024-09-18T08:04:41.773 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:04:41.773 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:04:41.773 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "done", 2024-09-18T08:04:41.773 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:04:41.134689+0000", 2024-09-18T08:04:41.773 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.00024349199999999999 2024-09-18T08:04:41.773 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-09-18T08:04:41.773 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-09-18T08:04:41.774 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-09-18T08:04:41.774 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:04:41.774 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:04:41.774 INFO:teuthology.orchestra.run.smithi007.stdout: "description": "osd_op(client.14805.0:63849 7.e 7:73ddad45:::benchmark_data_smithi007_73155_object63848:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e100)", 2024-09-18T08:04:41.774 INFO:teuthology.orchestra.run.smithi007.stdout: "initiated_at": "2024-09-18T08:04:41.109061+0000", 2024-09-18T08:04:41.774 INFO:teuthology.orchestra.run.smithi007.stdout: "age": 0.33105932199999999, 2024-09-18T08:04:41.774 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.02576173, 2024-09-18T08:04:41.774 INFO:teuthology.orchestra.run.smithi007.stdout: "continuous": false, 2024-09-18T08:04:41.774 INFO:teuthology.orchestra.run.smithi007.stdout: "type_data": { 2024-09-18T08:04:41.774 INFO:teuthology.orchestra.run.smithi007.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T08:04:41.774 INFO:teuthology.orchestra.run.smithi007.stdout: "client_info": { 2024-09-18T08:04:41.774 INFO:teuthology.orchestra.run.smithi007.stdout: "client": "client.14805", 2024-09-18T08:04:41.774 INFO:teuthology.orchestra.run.smithi007.stdout: "client_addr": "172.21.15.7:0/640320436", 2024-09-18T08:04:41.774 INFO:teuthology.orchestra.run.smithi007.stdout: "tid": 63849 2024-09-18T08:04:41.775 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:04:41.775 INFO:teuthology.orchestra.run.smithi007.stdout: "events": [ 2024-09-18T08:04:41.775 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:04:41.775 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "initiated", 2024-09-18T08:04:41.775 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:04:41.109061+0000", 2024-09-18T08:04:41.775 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-09-18T08:04:41.775 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:04:41.775 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:04:41.775 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "header_read", 2024-09-18T08:04:41.775 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:04:41.109061+0000", 2024-09-18T08:04:41.775 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-09-18T08:04:41.775 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:04:41.775 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:04:41.775 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "throttled", 2024-09-18T08:04:41.776 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:04:41.109062+0000", 2024-09-18T08:04:41.776 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.283e-06 2024-09-18T08:04:41.776 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:04:41.776 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:04:41.776 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "all_read", 2024-09-18T08:04:41.776 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:04:41.109085+0000", 2024-09-18T08:04:41.776 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 2.2727000000000001e-05 2024-09-18T08:04:41.776 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:04:41.776 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:04:41.776 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "dispatched", 2024-09-18T08:04:41.776 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:04:41.109097+0000", 2024-09-18T08:04:41.776 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.1762e-05 2024-09-18T08:04:41.776 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:04:41.776 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:04:41.777 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "queued_for_pg", 2024-09-18T08:04:41.777 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:04:41.109104+0000", 2024-09-18T08:04:41.777 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 7.469e-06 2024-09-18T08:04:41.777 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:04:41.777 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:04:41.777 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-09-18T08:04:41.777 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:04:41.109188+0000", 2024-09-18T08:04:41.777 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 8.3508000000000001e-05 2024-09-18T08:04:41.777 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:04:41.777 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:04:41.777 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "started", 2024-09-18T08:04:41.777 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:04:41.109323+0000", 2024-09-18T08:04:41.777 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.00013495299999999999 2024-09-18T08:04:41.777 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:04:41.778 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:04:41.778 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "waiting for subops from 0,5", 2024-09-18T08:04:41.778 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:04:41.109656+0000", 2024-09-18T08:04:41.778 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.00033326800000000001 2024-09-18T08:04:41.778 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:04:41.778 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:04:41.778 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "op_commit", 2024-09-18T08:04:41.778 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:04:41.113368+0000", 2024-09-18T08:04:41.778 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.0037116620000000001 2024-09-18T08:04:41.778 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:04:41.778 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:04:41.778 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:04:41.778 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:04:41.117441+0000", 2024-09-18T08:04:41.778 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.0040733829999999999 2024-09-18T08:04:41.779 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:04:41.779 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:04:41.779 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:04:41.779 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:04:41.134736+0000", 2024-09-18T08:04:41.779 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.017294753 2024-09-18T08:04:41.779 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:04:41.779 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:04:41.779 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "commit_sent", 2024-09-18T08:04:41.779 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:04:41.134788+0000", 2024-09-18T08:04:41.779 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 5.1879000000000003e-05 2024-09-18T08:04:41.779 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:04:41.779 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:04:41.779 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "done", 2024-09-18T08:04:41.779 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:04:41.134823+0000", 2024-09-18T08:04:41.780 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 3.5083000000000003e-05 2024-09-18T08:04:41.780 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-09-18T08:04:41.780 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-09-18T08:04:41.780 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-09-18T08:04:41.780 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:04:41.780 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:04:41.780 INFO:teuthology.orchestra.run.smithi007.stdout: "description": "osd_op(client.14805.0:63912 7.e 7:75995abf:::benchmark_data_smithi007_73155_object63911:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e100)", 2024-09-18T08:04:41.780 INFO:teuthology.orchestra.run.smithi007.stdout: "initiated_at": "2024-09-18T08:04:41.188183+0000", 2024-09-18T08:04:41.780 INFO:teuthology.orchestra.run.smithi007.stdout: "age": 0.25193703099999998, 2024-09-18T08:04:41.780 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.042576744, 2024-09-18T08:04:41.780 INFO:teuthology.orchestra.run.smithi007.stdout: "continuous": false, 2024-09-18T08:04:41.780 INFO:teuthology.orchestra.run.smithi007.stdout: "type_data": { 2024-09-18T08:04:41.780 INFO:teuthology.orchestra.run.smithi007.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T08:04:41.780 INFO:teuthology.orchestra.run.smithi007.stdout: "client_info": { 2024-09-18T08:04:41.781 INFO:teuthology.orchestra.run.smithi007.stdout: "client": "client.14805", 2024-09-18T08:04:41.781 INFO:teuthology.orchestra.run.smithi007.stdout: "client_addr": "172.21.15.7:0/640320436", 2024-09-18T08:04:41.781 INFO:teuthology.orchestra.run.smithi007.stdout: "tid": 63912 2024-09-18T08:04:41.781 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:04:41.781 INFO:teuthology.orchestra.run.smithi007.stdout: "events": [ 2024-09-18T08:04:41.781 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:04:41.781 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "initiated", 2024-09-18T08:04:41.781 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:04:41.188183+0000", 2024-09-18T08:04:41.781 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-09-18T08:04:41.781 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:04:41.781 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:04:41.781 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "header_read", 2024-09-18T08:04:41.781 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:04:41.188183+0000", 2024-09-18T08:04:41.781 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-09-18T08:04:41.782 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:04:41.782 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:04:41.782 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "throttled", 2024-09-18T08:04:41.782 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:04:41.188197+0000", 2024-09-18T08:04:41.782 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.3854e-05 2024-09-18T08:04:41.782 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:04:41.782 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:04:41.782 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "all_read", 2024-09-18T08:04:41.782 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:04:41.188223+0000", 2024-09-18T08:04:41.782 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 2.6152e-05 2024-09-18T08:04:41.782 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:04:41.782 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:04:41.782 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "dispatched", 2024-09-18T08:04:41.782 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:04:41.188244+0000", 2024-09-18T08:04:41.782 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 2.0888000000000001e-05 2024-09-18T08:04:41.783 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:04:41.783 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:04:41.783 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "queued_for_pg", 2024-09-18T08:04:41.783 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:04:41.188256+0000", 2024-09-18T08:04:41.783 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.1895e-05 2024-09-18T08:04:41.783 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:04:41.783 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:04:41.783 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-09-18T08:04:41.783 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:04:41.188328+0000", 2024-09-18T08:04:41.783 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 7.2058999999999996e-05 2024-09-18T08:04:41.783 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:04:41.783 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:04:41.783 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "started", 2024-09-18T08:04:41.783 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:04:41.188470+0000", 2024-09-18T08:04:41.784 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.00014191299999999999 2024-09-18T08:04:41.784 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:04:41.784 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:04:41.784 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "waiting for subops from 0,5", 2024-09-18T08:04:41.784 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:04:41.188622+0000", 2024-09-18T08:04:41.784 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.000151889 2024-09-18T08:04:41.784 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:04:41.784 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:04:41.784 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "op_commit", 2024-09-18T08:04:41.784 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:04:41.194459+0000", 2024-09-18T08:04:41.784 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.0058372440000000001 2024-09-18T08:04:41.784 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:04:41.784 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:04:41.784 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:04:41.784 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:04:41.194502+0000", 2024-09-18T08:04:41.785 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 4.2503999999999998e-05 2024-09-18T08:04:41.785 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:04:41.785 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:04:41.785 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:04:41.785 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:04:41.230645+0000", 2024-09-18T08:04:41.785 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.036143382000000002 2024-09-18T08:04:41.785 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:04:41.785 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:04:41.785 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "commit_sent", 2024-09-18T08:04:41.785 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:04:41.230709+0000", 2024-09-18T08:04:41.785 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 6.3534000000000005e-05 2024-09-18T08:04:41.785 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:04:41.785 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:04:41.786 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "done", 2024-09-18T08:04:41.786 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:04:41.230760+0000", 2024-09-18T08:04:41.786 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 5.1430000000000001e-05 2024-09-18T08:04:41.786 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-09-18T08:04:41.786 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-09-18T08:04:41.786 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-09-18T08:04:41.786 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:04:41.786 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:04:41.786 INFO:teuthology.orchestra.run.smithi007.stdout: "description": "osd_repop(client.14805.0:63925 7.a e100/83 7:552ed094:::benchmark_data_smithi007_73155_object63924:head v 100'4035, mlcod=100'4033)", 2024-09-18T08:04:41.786 INFO:teuthology.orchestra.run.smithi007.stdout: "initiated_at": "2024-09-18T08:04:41.210423+0000", 2024-09-18T08:04:41.786 INFO:teuthology.orchestra.run.smithi007.stdout: "age": 0.229696814, 2024-09-18T08:04:41.786 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.025253533000000002, 2024-09-18T08:04:41.786 INFO:teuthology.orchestra.run.smithi007.stdout: "continuous": false, 2024-09-18T08:04:41.786 INFO:teuthology.orchestra.run.smithi007.stdout: "type_data": { 2024-09-18T08:04:41.787 INFO:teuthology.orchestra.run.smithi007.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T08:04:41.787 INFO:teuthology.orchestra.run.smithi007.stdout: "events": [ 2024-09-18T08:04:41.787 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:04:41.787 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "initiated", 2024-09-18T08:04:41.787 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:04:41.210423+0000", 2024-09-18T08:04:41.787 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-09-18T08:04:41.787 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:04:41.787 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:04:41.787 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "header_read", 2024-09-18T08:04:41.787 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:04:41.210423+0000", 2024-09-18T08:04:41.787 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-09-18T08:04:41.787 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:04:41.787 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:04:41.787 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "throttled", 2024-09-18T08:04:41.788 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:04:41.210430+0000", 2024-09-18T08:04:41.788 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 6.7460000000000003e-06 2024-09-18T08:04:41.788 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:04:41.788 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:04:41.788 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "all_read", 2024-09-18T08:04:41.788 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:04:41.210465+0000", 2024-09-18T08:04:41.788 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 3.4434000000000003e-05 2024-09-18T08:04:41.788 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:04:41.788 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:04:41.788 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "dispatched", 2024-09-18T08:04:41.788 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:04:41.210478+0000", 2024-09-18T08:04:41.788 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.3498e-05 2024-09-18T08:04:41.788 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:04:41.788 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:04:41.789 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "queued_for_pg", 2024-09-18T08:04:41.789 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:04:41.210487+0000", 2024-09-18T08:04:41.789 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 9.3530000000000001e-06 2024-09-18T08:04:41.789 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:04:41.789 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:04:41.789 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-09-18T08:04:41.789 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:04:41.216627+0000", 2024-09-18T08:04:41.789 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.0061392399999999998 2024-09-18T08:04:41.789 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:04:41.789 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:04:41.789 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "started", 2024-09-18T08:04:41.789 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:04:41.216684+0000", 2024-09-18T08:04:41.789 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 5.7401000000000002e-05 2024-09-18T08:04:41.789 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:04:41.790 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:04:41.790 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "commit_sent", 2024-09-18T08:04:41.790 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:04:41.235128+0000", 2024-09-18T08:04:41.790 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.018444354 2024-09-18T08:04:41.790 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:04:41.790 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:04:41.790 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "done", 2024-09-18T08:04:41.790 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:04:41.235677+0000", 2024-09-18T08:04:41.790 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.00054850700000000005 2024-09-18T08:04:41.790 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-09-18T08:04:41.790 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-09-18T08:04:41.790 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-09-18T08:04:41.790 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:04:41.790 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:04:41.791 INFO:teuthology.orchestra.run.smithi007.stdout: "description": "osd_op(client.14805.0:63931 7.e 7:7f27614f:::benchmark_data_smithi007_73155_object63930:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e100)", 2024-09-18T08:04:41.791 INFO:teuthology.orchestra.run.smithi007.stdout: "initiated_at": "2024-09-18T08:04:41.212395+0000", 2024-09-18T08:04:41.791 INFO:teuthology.orchestra.run.smithi007.stdout: "age": 0.227725123, 2024-09-18T08:04:41.791 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.030577607999999999, 2024-09-18T08:04:41.791 INFO:teuthology.orchestra.run.smithi007.stdout: "continuous": false, 2024-09-18T08:04:41.791 INFO:teuthology.orchestra.run.smithi007.stdout: "type_data": { 2024-09-18T08:04:41.791 INFO:teuthology.orchestra.run.smithi007.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T08:04:41.791 INFO:teuthology.orchestra.run.smithi007.stdout: "client_info": { 2024-09-18T08:04:41.791 INFO:teuthology.orchestra.run.smithi007.stdout: "client": "client.14805", 2024-09-18T08:04:41.791 INFO:teuthology.orchestra.run.smithi007.stdout: "client_addr": "172.21.15.7:0/640320436", 2024-09-18T08:04:41.791 INFO:teuthology.orchestra.run.smithi007.stdout: "tid": 63931 2024-09-18T08:04:41.791 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:04:41.791 INFO:teuthology.orchestra.run.smithi007.stdout: "events": [ 2024-09-18T08:04:41.792 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:04:41.792 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "initiated", 2024-09-18T08:04:41.792 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:04:41.212395+0000", 2024-09-18T08:04:41.792 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-09-18T08:04:41.792 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:04:41.792 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:04:41.792 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "header_read", 2024-09-18T08:04:41.792 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:04:41.212395+0000", 2024-09-18T08:04:41.792 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-09-18T08:04:41.792 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:04:41.792 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:04:41.792 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "throttled", 2024-09-18T08:04:41.792 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:04:41.212398+0000", 2024-09-18T08:04:41.792 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 2.362e-06 2024-09-18T08:04:41.793 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:04:41.793 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:04:41.793 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "all_read", 2024-09-18T08:04:41.793 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:04:41.212449+0000", 2024-09-18T08:04:41.793 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 5.1196000000000002e-05 2024-09-18T08:04:41.793 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:04:41.793 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:04:41.793 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "dispatched", 2024-09-18T08:04:41.793 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:04:41.212516+0000", 2024-09-18T08:04:41.793 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 6.7379000000000007e-05 2024-09-18T08:04:41.794 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:04:41.794 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:04:41.794 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "queued_for_pg", 2024-09-18T08:04:41.794 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:04:41.212556+0000", 2024-09-18T08:04:41.794 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 4.0379000000000001e-05 2024-09-18T08:04:41.794 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:04:41.794 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:04:41.794 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-09-18T08:04:41.794 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:04:41.212632+0000", 2024-09-18T08:04:41.794 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 7.5113999999999999e-05 2024-09-18T08:04:41.794 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:04:41.794 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:04:41.794 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "started", 2024-09-18T08:04:41.794 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:04:41.212824+0000", 2024-09-18T08:04:41.795 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.00019264799999999999 2024-09-18T08:04:41.795 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:04:41.795 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:04:41.795 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "waiting for subops from 0,5", 2024-09-18T08:04:41.795 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:04:41.212981+0000", 2024-09-18T08:04:41.795 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.000157121 2024-09-18T08:04:41.795 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:04:41.795 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:04:41.795 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "op_commit", 2024-09-18T08:04:41.795 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:04:41.215491+0000", 2024-09-18T08:04:41.795 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.0025096519999999998 2024-09-18T08:04:41.795 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:04:41.795 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:04:41.796 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:04:41.796 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:04:41.217622+0000", 2024-09-18T08:04:41.796 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.0021306049999999998 2024-09-18T08:04:41.796 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:04:41.796 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:04:41.796 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:04:41.796 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:04:41.242864+0000", 2024-09-18T08:04:41.796 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.025241950999999999 2024-09-18T08:04:41.796 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:04:41.796 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:04:41.796 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "commit_sent", 2024-09-18T08:04:41.796 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:04:41.242923+0000", 2024-09-18T08:04:41.796 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 5.9432000000000003e-05 2024-09-18T08:04:41.796 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:04:41.797 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:04:41.797 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "done", 2024-09-18T08:04:41.797 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:04:41.242973+0000", 2024-09-18T08:04:41.797 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 4.9768999999999997e-05 2024-09-18T08:04:41.797 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-09-18T08:04:41.797 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-09-18T08:04:41.797 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-09-18T08:04:41.797 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:04:41.797 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:04:41.797 INFO:teuthology.orchestra.run.smithi007.stdout: "description": "osd_op(client.14805.0:63946 7.e 7:75c27164:::benchmark_data_smithi007_73155_object63945:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e100)", 2024-09-18T08:04:41.797 INFO:teuthology.orchestra.run.smithi007.stdout: "initiated_at": "2024-09-18T08:04:41.239395+0000", 2024-09-18T08:04:41.797 INFO:teuthology.orchestra.run.smithi007.stdout: "age": 0.20072547600000001, 2024-09-18T08:04:41.797 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.026048753000000001, 2024-09-18T08:04:41.798 INFO:teuthology.orchestra.run.smithi007.stdout: "continuous": false, 2024-09-18T08:04:41.798 INFO:teuthology.orchestra.run.smithi007.stdout: "type_data": { 2024-09-18T08:04:41.798 INFO:teuthology.orchestra.run.smithi007.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T08:04:41.798 INFO:teuthology.orchestra.run.smithi007.stdout: "client_info": { 2024-09-18T08:04:41.798 INFO:teuthology.orchestra.run.smithi007.stdout: "client": "client.14805", 2024-09-18T08:04:41.798 INFO:teuthology.orchestra.run.smithi007.stdout: "client_addr": "172.21.15.7:0/640320436", 2024-09-18T08:04:41.798 INFO:teuthology.orchestra.run.smithi007.stdout: "tid": 63946 2024-09-18T08:04:41.798 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:04:41.798 INFO:teuthology.orchestra.run.smithi007.stdout: "events": [ 2024-09-18T08:04:41.798 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:04:41.798 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "initiated", 2024-09-18T08:04:41.798 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:04:41.239395+0000", 2024-09-18T08:04:41.798 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-09-18T08:04:41.799 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:04:41.799 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:04:41.799 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "header_read", 2024-09-18T08:04:41.799 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:04:41.239395+0000", 2024-09-18T08:04:41.799 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-09-18T08:04:41.799 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:04:41.799 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:04:41.799 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "throttled", 2024-09-18T08:04:41.799 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:04:41.239395+0000", 2024-09-18T08:04:41.799 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 5.6300000000000005e-07 2024-09-18T08:04:41.799 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:04:41.799 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:04:41.799 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "all_read", 2024-09-18T08:04:41.799 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:04:41.239416+0000", 2024-09-18T08:04:41.800 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 2.0679e-05 2024-09-18T08:04:41.800 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:04:41.800 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:04:41.800 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "dispatched", 2024-09-18T08:04:41.800 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:04:41.239423+0000", 2024-09-18T08:04:41.800 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 7.4209999999999998e-06 2024-09-18T08:04:41.800 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:04:41.800 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:04:41.800 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "queued_for_pg", 2024-09-18T08:04:41.800 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:04:41.239429+0000", 2024-09-18T08:04:41.800 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 5.9440000000000002e-06 2024-09-18T08:04:41.800 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:04:41.800 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:04:41.801 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-09-18T08:04:41.801 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:04:41.239568+0000", 2024-09-18T08:04:41.801 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.00013855200000000001 2024-09-18T08:04:41.801 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:04:41.801 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:04:41.801 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "started", 2024-09-18T08:04:41.801 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:04:41.239701+0000", 2024-09-18T08:04:41.801 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.00013312099999999999 2024-09-18T08:04:41.801 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:04:41.801 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:04:41.801 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "waiting for subops from 0,5", 2024-09-18T08:04:41.801 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:04:41.239844+0000", 2024-09-18T08:04:41.801 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.000142812 2024-09-18T08:04:41.802 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:04:41.802 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:04:41.802 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:04:41.802 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:04:41.242556+0000", 2024-09-18T08:04:41.802 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.0027124440000000001 2024-09-18T08:04:41.802 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:04:41.802 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:04:41.802 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "op_commit", 2024-09-18T08:04:41.802 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:04:41.242572+0000", 2024-09-18T08:04:41.802 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.5466999999999999e-05 2024-09-18T08:04:41.802 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:04:41.802 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:04:41.802 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:04:41.803 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:04:41.265331+0000", 2024-09-18T08:04:41.803 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.022758848000000002 2024-09-18T08:04:41.803 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:04:41.803 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:04:41.803 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "commit_sent", 2024-09-18T08:04:41.803 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:04:41.265394+0000", 2024-09-18T08:04:41.803 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 6.3361999999999994e-05 2024-09-18T08:04:41.803 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:04:41.803 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:04:41.803 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "done", 2024-09-18T08:04:41.803 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:04:41.265444+0000", 2024-09-18T08:04:41.803 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 4.9539999999999997e-05 2024-09-18T08:04:41.803 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-09-18T08:04:41.803 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-09-18T08:04:41.804 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-09-18T08:04:41.804 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:04:41.804 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:04:41.804 INFO:teuthology.orchestra.run.smithi007.stdout: "description": "osd_repop(client.14805.0:63948 7.a e100/83 7:5ccd60b2:::benchmark_data_smithi007_73155_object63947:head v 100'4037, mlcod=100'4034)", 2024-09-18T08:04:41.804 INFO:teuthology.orchestra.run.smithi007.stdout: "initiated_at": "2024-09-18T08:04:41.246424+0000", 2024-09-18T08:04:41.804 INFO:teuthology.orchestra.run.smithi007.stdout: "age": 0.19369651299999999, 2024-09-18T08:04:41.804 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.028289315999999998, 2024-09-18T08:04:41.804 INFO:teuthology.orchestra.run.smithi007.stdout: "continuous": false, 2024-09-18T08:04:41.804 INFO:teuthology.orchestra.run.smithi007.stdout: "type_data": { 2024-09-18T08:04:41.804 INFO:teuthology.orchestra.run.smithi007.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T08:04:41.804 INFO:teuthology.orchestra.run.smithi007.stdout: "events": [ 2024-09-18T08:04:41.804 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:04:41.804 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "initiated", 2024-09-18T08:04:41.805 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:04:41.246424+0000", 2024-09-18T08:04:41.805 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-09-18T08:04:41.805 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:04:41.805 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:04:41.805 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "header_read", 2024-09-18T08:04:41.805 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:04:41.246424+0000", 2024-09-18T08:04:41.805 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-09-18T08:04:41.805 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:04:41.805 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:04:41.805 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "throttled", 2024-09-18T08:04:41.805 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:04:41.246424+0000", 2024-09-18T08:04:41.805 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 7.2099999999999996e-07 2024-09-18T08:04:41.805 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:04:41.805 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:04:41.806 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "all_read", 2024-09-18T08:04:41.806 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:04:41.246451+0000", 2024-09-18T08:04:41.806 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 2.6061e-05 2024-09-18T08:04:41.806 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:04:41.806 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:04:41.806 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "dispatched", 2024-09-18T08:04:41.806 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:04:41.246461+0000", 2024-09-18T08:04:41.806 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.0267e-05 2024-09-18T08:04:41.806 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:04:41.806 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:04:41.806 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "queued_for_pg", 2024-09-18T08:04:41.806 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:04:41.246468+0000", 2024-09-18T08:04:41.806 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 7.1930000000000004e-06 2024-09-18T08:04:41.807 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:04:41.807 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:04:41.807 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-09-18T08:04:41.807 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:04:41.250261+0000", 2024-09-18T08:04:41.807 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.0037925630000000001 2024-09-18T08:04:41.807 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:04:41.807 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:04:41.807 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "started", 2024-09-18T08:04:41.807 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:04:41.250283+0000", 2024-09-18T08:04:41.807 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 2.2597999999999999e-05 2024-09-18T08:04:41.807 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:04:41.807 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:04:41.807 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "commit_sent", 2024-09-18T08:04:41.807 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:04:41.274664+0000", 2024-09-18T08:04:41.807 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.024381007999999999 2024-09-18T08:04:41.808 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:04:41.808 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:04:41.808 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "done", 2024-09-18T08:04:41.808 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:04:41.274713+0000", 2024-09-18T08:04:41.808 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 4.8905e-05 2024-09-18T08:04:41.808 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-09-18T08:04:41.808 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-09-18T08:04:41.808 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-09-18T08:04:41.808 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:04:41.808 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:04:41.808 INFO:teuthology.orchestra.run.smithi007.stdout: "description": "osd_op(client.14805.0:63955 7.8 7:1a9f7093:::benchmark_data_smithi007_73155_object63954:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e100)", 2024-09-18T08:04:41.808 INFO:teuthology.orchestra.run.smithi007.stdout: "initiated_at": "2024-09-18T08:04:41.246748+0000", 2024-09-18T08:04:41.808 INFO:teuthology.orchestra.run.smithi007.stdout: "age": 0.19337212100000001, 2024-09-18T08:04:41.808 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.041142450999999997, 2024-09-18T08:04:41.809 INFO:teuthology.orchestra.run.smithi007.stdout: "continuous": false, 2024-09-18T08:04:41.809 INFO:teuthology.orchestra.run.smithi007.stdout: "type_data": { 2024-09-18T08:04:41.809 INFO:teuthology.orchestra.run.smithi007.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T08:04:41.809 INFO:teuthology.orchestra.run.smithi007.stdout: "client_info": { 2024-09-18T08:04:41.809 INFO:teuthology.orchestra.run.smithi007.stdout: "client": "client.14805", 2024-09-18T08:04:41.809 INFO:teuthology.orchestra.run.smithi007.stdout: "client_addr": "172.21.15.7:0/640320436", 2024-09-18T08:04:41.809 INFO:teuthology.orchestra.run.smithi007.stdout: "tid": 63955 2024-09-18T08:04:41.809 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:04:41.809 INFO:teuthology.orchestra.run.smithi007.stdout: "events": [ 2024-09-18T08:04:41.809 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:04:41.809 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "initiated", 2024-09-18T08:04:41.809 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:04:41.246748+0000", 2024-09-18T08:04:41.809 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-09-18T08:04:41.809 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:04:41.810 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:04:41.810 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "header_read", 2024-09-18T08:04:41.810 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:04:41.246748+0000", 2024-09-18T08:04:41.810 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-09-18T08:04:41.810 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:04:41.810 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:04:41.810 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "throttled", 2024-09-18T08:04:41.810 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:04:41.246750+0000", 2024-09-18T08:04:41.810 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.6819999999999999e-06 2024-09-18T08:04:41.810 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:04:41.810 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:04:41.810 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "all_read", 2024-09-18T08:04:41.810 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:04:41.246775+0000", 2024-09-18T08:04:41.810 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 2.4970999999999999e-05 2024-09-18T08:04:41.811 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:04:41.811 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:04:41.811 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "dispatched", 2024-09-18T08:04:41.811 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:04:41.246788+0000", 2024-09-18T08:04:41.811 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.305e-05 2024-09-18T08:04:41.811 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:04:41.811 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:04:41.811 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "queued_for_pg", 2024-09-18T08:04:41.811 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:04:41.246796+0000", 2024-09-18T08:04:41.811 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 7.9170000000000006e-06 2024-09-18T08:04:41.811 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:04:41.811 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:04:41.811 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-09-18T08:04:41.812 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:04:41.251387+0000", 2024-09-18T08:04:41.812 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.0045909389999999996 2024-09-18T08:04:41.812 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:04:41.812 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:04:41.812 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "started", 2024-09-18T08:04:41.812 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:04:41.251756+0000", 2024-09-18T08:04:41.812 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.000369453 2024-09-18T08:04:41.812 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:04:41.812 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:04:41.812 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "waiting for subops from 4,7", 2024-09-18T08:04:41.812 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:04:41.255892+0000", 2024-09-18T08:04:41.812 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.0041355480000000002 2024-09-18T08:04:41.812 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:04:41.812 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:04:41.813 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:04:41.813 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:04:41.265954+0000", 2024-09-18T08:04:41.813 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.010062684000000001 2024-09-18T08:04:41.813 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:04:41.813 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:04:41.813 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:04:41.813 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:04:41.265996+0000", 2024-09-18T08:04:41.813 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 4.2011999999999997e-05 2024-09-18T08:04:41.813 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:04:41.813 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:04:41.813 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "op_commit", 2024-09-18T08:04:41.813 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:04:41.287754+0000", 2024-09-18T08:04:41.813 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.021757899000000001 2024-09-18T08:04:41.813 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:04:41.814 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:04:41.814 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "commit_sent", 2024-09-18T08:04:41.814 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:04:41.287821+0000", 2024-09-18T08:04:41.814 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 6.7118000000000004e-05 2024-09-18T08:04:41.814 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:04:41.814 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:04:41.814 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "done", 2024-09-18T08:04:41.814 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:04:41.287891+0000", 2024-09-18T08:04:41.814 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 6.9177999999999994e-05 2024-09-18T08:04:41.814 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-09-18T08:04:41.814 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-09-18T08:04:41.814 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-09-18T08:04:41.814 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:04:41.814 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:04:41.815 INFO:teuthology.orchestra.run.smithi007.stdout: "description": "osd_op(client.14805.0:63959 7.8 7:1bfeb5df:::benchmark_data_smithi007_73155_object63958:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e100)", 2024-09-18T08:04:41.815 INFO:teuthology.orchestra.run.smithi007.stdout: "initiated_at": "2024-09-18T08:04:41.261463+0000", 2024-09-18T08:04:41.815 INFO:teuthology.orchestra.run.smithi007.stdout: "age": 0.17865693199999999, 2024-09-18T08:04:41.815 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.026556781000000002, 2024-09-18T08:04:41.815 INFO:teuthology.orchestra.run.smithi007.stdout: "continuous": false, 2024-09-18T08:04:41.815 INFO:teuthology.orchestra.run.smithi007.stdout: "type_data": { 2024-09-18T08:04:41.815 INFO:teuthology.orchestra.run.smithi007.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T08:04:41.815 INFO:teuthology.orchestra.run.smithi007.stdout: "client_info": { 2024-09-18T08:04:41.815 INFO:teuthology.orchestra.run.smithi007.stdout: "client": "client.14805", 2024-09-18T08:04:41.815 INFO:teuthology.orchestra.run.smithi007.stdout: "client_addr": "172.21.15.7:0/640320436", 2024-09-18T08:04:41.815 INFO:teuthology.orchestra.run.smithi007.stdout: "tid": 63959 2024-09-18T08:04:41.815 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:04:41.815 INFO:teuthology.orchestra.run.smithi007.stdout: "events": [ 2024-09-18T08:04:41.815 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:04:41.816 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "initiated", 2024-09-18T08:04:41.816 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:04:41.261463+0000", 2024-09-18T08:04:41.816 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-09-18T08:04:41.816 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:04:41.816 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:04:41.816 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "header_read", 2024-09-18T08:04:41.816 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:04:41.261463+0000", 2024-09-18T08:04:41.816 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-09-18T08:04:41.816 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:04:41.816 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:04:41.816 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "throttled", 2024-09-18T08:04:41.816 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:04:41.261465+0000", 2024-09-18T08:04:41.816 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.598e-06 2024-09-18T08:04:41.817 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:04:41.817 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:04:41.817 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "all_read", 2024-09-18T08:04:41.817 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:04:41.261500+0000", 2024-09-18T08:04:41.817 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 3.5324999999999998e-05 2024-09-18T08:04:41.817 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:04:41.817 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:04:41.817 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "dispatched", 2024-09-18T08:04:41.817 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:04:41.261513+0000", 2024-09-18T08:04:41.817 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.2514000000000001e-05 2024-09-18T08:04:41.817 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:04:41.817 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:04:41.817 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "queued_for_pg", 2024-09-18T08:04:41.817 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:04:41.261521+0000", 2024-09-18T08:04:41.818 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 7.9400000000000002e-06 2024-09-18T08:04:41.818 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:04:41.818 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:04:41.818 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-09-18T08:04:41.818 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:04:41.266030+0000", 2024-09-18T08:04:41.818 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.0045088639999999996 2024-09-18T08:04:41.818 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:04:41.818 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:04:41.818 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "started", 2024-09-18T08:04:41.818 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:04:41.266163+0000", 2024-09-18T08:04:41.818 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.000133282 2024-09-18T08:04:41.818 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:04:41.818 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:04:41.818 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "waiting for subops from 4,7", 2024-09-18T08:04:41.819 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:04:41.266300+0000", 2024-09-18T08:04:41.819 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.00013689099999999999 2024-09-18T08:04:41.819 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:04:41.819 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:04:41.819 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:04:41.819 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:04:41.287741+0000", 2024-09-18T08:04:41.819 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.021440944 2024-09-18T08:04:41.819 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:04:41.819 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:04:41.819 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "op_commit", 2024-09-18T08:04:41.819 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:04:41.287900+0000", 2024-09-18T08:04:41.819 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.000159414 2024-09-18T08:04:41.819 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:04:41.820 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:04:41.820 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:04:41.820 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:04:41.287938+0000", 2024-09-18T08:04:41.820 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 3.7916000000000001e-05 2024-09-18T08:04:41.820 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:04:41.820 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:04:41.820 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "commit_sent", 2024-09-18T08:04:41.820 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:04:41.287988+0000", 2024-09-18T08:04:41.820 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 4.9673999999999999e-05 2024-09-18T08:04:41.820 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:04:41.820 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:04:41.820 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "done", 2024-09-18T08:04:41.820 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:04:41.288020+0000", 2024-09-18T08:04:41.820 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 3.2419000000000003e-05 2024-09-18T08:04:41.821 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-09-18T08:04:41.821 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-09-18T08:04:41.821 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-09-18T08:04:41.821 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:04:41.821 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:04:41.821 INFO:teuthology.orchestra.run.smithi007.stdout: "description": "osd_repop(client.14805.0:63950 7.a e100/83 7:5df2c2fd:::benchmark_data_smithi007_73155_object63949:head v 100'4038, mlcod=100'4034)", 2024-09-18T08:04:41.821 INFO:teuthology.orchestra.run.smithi007.stdout: "initiated_at": "2024-09-18T08:04:41.261561+0000", 2024-09-18T08:04:41.821 INFO:teuthology.orchestra.run.smithi007.stdout: "age": 0.17855951, 2024-09-18T08:04:41.821 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.032972860999999999, 2024-09-18T08:04:41.821 INFO:teuthology.orchestra.run.smithi007.stdout: "continuous": false, 2024-09-18T08:04:41.821 INFO:teuthology.orchestra.run.smithi007.stdout: "type_data": { 2024-09-18T08:04:41.821 INFO:teuthology.orchestra.run.smithi007.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T08:04:41.821 INFO:teuthology.orchestra.run.smithi007.stdout: "events": [ 2024-09-18T08:04:41.822 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:04:41.822 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "initiated", 2024-09-18T08:04:41.822 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:04:41.261561+0000", 2024-09-18T08:04:41.822 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-09-18T08:04:41.822 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:04:41.822 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:04:41.822 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "header_read", 2024-09-18T08:04:41.822 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:04:41.261561+0000", 2024-09-18T08:04:41.822 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-09-18T08:04:41.822 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:04:41.822 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:04:41.822 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "throttled", 2024-09-18T08:04:41.822 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:04:41.261561+0000", 2024-09-18T08:04:41.822 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 5.3600000000000004e-07 2024-09-18T08:04:41.823 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:04:41.823 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:04:41.823 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "all_read", 2024-09-18T08:04:41.823 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:04:41.261585+0000", 2024-09-18T08:04:41.823 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 2.3728999999999999e-05 2024-09-18T08:04:41.823 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:04:41.823 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:04:41.823 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "dispatched", 2024-09-18T08:04:41.823 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:04:41.261593+0000", 2024-09-18T08:04:41.823 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 8.2099999999999993e-06 2024-09-18T08:04:41.823 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:04:41.823 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:04:41.823 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "queued_for_pg", 2024-09-18T08:04:41.824 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:04:41.261599+0000", 2024-09-18T08:04:41.824 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 5.6289999999999998e-06 2024-09-18T08:04:41.824 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:04:41.824 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:04:41.824 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-09-18T08:04:41.824 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:04:41.272330+0000", 2024-09-18T08:04:41.824 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.010731594000000001 2024-09-18T08:04:41.824 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:04:41.824 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:04:41.824 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "started", 2024-09-18T08:04:41.824 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:04:41.272355+0000", 2024-09-18T08:04:41.824 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 2.4530999999999999e-05 2024-09-18T08:04:41.824 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:04:41.824 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:04:41.825 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "commit_sent", 2024-09-18T08:04:41.825 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:04:41.294491+0000", 2024-09-18T08:04:41.825 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.022136474999999999 2024-09-18T08:04:41.825 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:04:41.825 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:04:41.825 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "done", 2024-09-18T08:04:41.825 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:04:41.294534+0000", 2024-09-18T08:04:41.825 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 4.2157000000000003e-05 2024-09-18T08:04:41.825 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-09-18T08:04:41.825 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-09-18T08:04:41.825 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-09-18T08:04:41.825 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:04:41.825 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:04:41.825 INFO:teuthology.orchestra.run.smithi007.stdout: "description": "osd_op(client.14805.0:63967 7.8 7:1368f06b:::benchmark_data_smithi007_73155_object63966:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e100)", 2024-09-18T08:04:41.826 INFO:teuthology.orchestra.run.smithi007.stdout: "initiated_at": "2024-09-18T08:04:41.273998+0000", 2024-09-18T08:04:41.826 INFO:teuthology.orchestra.run.smithi007.stdout: "age": 0.16612210199999999, 2024-09-18T08:04:41.826 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.034514753000000002, 2024-09-18T08:04:41.826 INFO:teuthology.orchestra.run.smithi007.stdout: "continuous": false, 2024-09-18T08:04:41.826 INFO:teuthology.orchestra.run.smithi007.stdout: "type_data": { 2024-09-18T08:04:41.826 INFO:teuthology.orchestra.run.smithi007.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T08:04:41.826 INFO:teuthology.orchestra.run.smithi007.stdout: "client_info": { 2024-09-18T08:04:41.826 INFO:teuthology.orchestra.run.smithi007.stdout: "client": "client.14805", 2024-09-18T08:04:41.826 INFO:teuthology.orchestra.run.smithi007.stdout: "client_addr": "172.21.15.7:0/640320436", 2024-09-18T08:04:41.826 INFO:teuthology.orchestra.run.smithi007.stdout: "tid": 63967 2024-09-18T08:04:41.826 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:04:41.826 INFO:teuthology.orchestra.run.smithi007.stdout: "events": [ 2024-09-18T08:04:41.826 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:04:41.826 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "initiated", 2024-09-18T08:04:41.826 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:04:41.273998+0000", 2024-09-18T08:04:41.827 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-09-18T08:04:41.827 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:04:41.827 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:04:41.827 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "header_read", 2024-09-18T08:04:41.827 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:04:41.273998+0000", 2024-09-18T08:04:41.827 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-09-18T08:04:41.827 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:04:41.827 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:04:41.827 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "throttled", 2024-09-18T08:04:41.827 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:04:41.274001+0000", 2024-09-18T08:04:41.827 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 2.5339999999999998e-06 2024-09-18T08:04:41.827 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:04:41.827 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:04:41.827 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "all_read", 2024-09-18T08:04:41.828 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:04:41.274027+0000", 2024-09-18T08:04:41.828 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 2.5965999999999998e-05 2024-09-18T08:04:41.828 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:04:41.828 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:04:41.828 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "dispatched", 2024-09-18T08:04:41.828 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:04:41.274042+0000", 2024-09-18T08:04:41.828 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.4946000000000001e-05 2024-09-18T08:04:41.828 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:04:41.828 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:04:41.828 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "queued_for_pg", 2024-09-18T08:04:41.828 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:04:41.274052+0000", 2024-09-18T08:04:41.828 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.0531e-05 2024-09-18T08:04:41.828 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:04:41.828 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:04:41.829 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-09-18T08:04:41.829 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:04:41.288064+0000", 2024-09-18T08:04:41.829 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.014011363000000001 2024-09-18T08:04:41.829 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:04:41.829 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:04:41.829 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "started", 2024-09-18T08:04:41.829 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:04:41.288213+0000", 2024-09-18T08:04:41.829 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.00014963899999999999 2024-09-18T08:04:41.829 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:04:41.829 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:04:41.829 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "waiting for subops from 4,7", 2024-09-18T08:04:41.829 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:04:41.288381+0000", 2024-09-18T08:04:41.829 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.00016777799999999999 2024-09-18T08:04:41.829 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:04:41.830 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:04:41.830 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:04:41.830 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:04:41.291029+0000", 2024-09-18T08:04:41.830 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.0026480359999999999 2024-09-18T08:04:41.830 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:04:41.830 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:04:41.830 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:04:41.830 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:04:41.291265+0000", 2024-09-18T08:04:41.830 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.00023642499999999999 2024-09-18T08:04:41.830 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:04:41.832 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:04:41.832 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "op_commit", 2024-09-18T08:04:41.832 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:04:41.308296+0000", 2024-09-18T08:04:41.832 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.017030474 2024-09-18T08:04:41.832 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:04:41.832 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:04:41.832 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "commit_sent", 2024-09-18T08:04:41.832 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:04:41.308460+0000", 2024-09-18T08:04:41.832 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.00016374 2024-09-18T08:04:41.832 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:04:41.832 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:04:41.832 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "done", 2024-09-18T08:04:41.832 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:04:41.308513+0000", 2024-09-18T08:04:41.833 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 5.3321e-05 2024-09-18T08:04:41.833 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-09-18T08:04:41.833 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-09-18T08:04:41.833 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-09-18T08:04:41.833 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:04:41.833 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:04:41.833 INFO:teuthology.orchestra.run.smithi007.stdout: "description": "osd_op(client.14805.0:63973 7.8 7:190b70d4:::benchmark_data_smithi007_73155_object63972:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e100)", 2024-09-18T08:04:41.833 INFO:teuthology.orchestra.run.smithi007.stdout: "initiated_at": "2024-09-18T08:04:41.297476+0000", 2024-09-18T08:04:41.833 INFO:teuthology.orchestra.run.smithi007.stdout: "age": 0.142644466, 2024-09-18T08:04:41.833 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.033414115000000001, 2024-09-18T08:04:41.833 INFO:teuthology.orchestra.run.smithi007.stdout: "continuous": false, 2024-09-18T08:04:41.833 INFO:teuthology.orchestra.run.smithi007.stdout: "type_data": { 2024-09-18T08:04:41.833 INFO:teuthology.orchestra.run.smithi007.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T08:04:41.833 INFO:teuthology.orchestra.run.smithi007.stdout: "client_info": { 2024-09-18T08:04:41.834 INFO:teuthology.orchestra.run.smithi007.stdout: "client": "client.14805", 2024-09-18T08:04:41.834 INFO:teuthology.orchestra.run.smithi007.stdout: "client_addr": "172.21.15.7:0/640320436", 2024-09-18T08:04:41.834 INFO:teuthology.orchestra.run.smithi007.stdout: "tid": 63973 2024-09-18T08:04:41.834 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:04:41.834 INFO:teuthology.orchestra.run.smithi007.stdout: "events": [ 2024-09-18T08:04:41.834 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:04:41.834 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "initiated", 2024-09-18T08:04:41.834 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:04:41.297476+0000", 2024-09-18T08:04:41.834 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-09-18T08:04:41.834 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:04:41.834 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:04:41.834 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "header_read", 2024-09-18T08:04:41.834 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:04:41.297476+0000", 2024-09-18T08:04:41.834 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-09-18T08:04:41.834 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:04:41.835 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:04:41.835 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "throttled", 2024-09-18T08:04:41.835 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:04:41.297478+0000", 2024-09-18T08:04:41.835 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 2.1739999999999999e-06 2024-09-18T08:04:41.835 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:04:41.835 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:04:41.835 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "all_read", 2024-09-18T08:04:41.835 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:04:41.297507+0000", 2024-09-18T08:04:41.835 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 2.8923000000000001e-05 2024-09-18T08:04:41.835 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:04:41.835 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:04:41.835 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "dispatched", 2024-09-18T08:04:41.835 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:04:41.297522+0000", 2024-09-18T08:04:41.835 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.4678000000000001e-05 2024-09-18T08:04:41.836 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:04:41.836 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:04:41.836 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "queued_for_pg", 2024-09-18T08:04:41.836 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:04:41.297531+0000", 2024-09-18T08:04:41.836 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 9.8139999999999995e-06 2024-09-18T08:04:41.836 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:04:41.836 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:04:41.836 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-09-18T08:04:41.836 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:04:41.299251+0000", 2024-09-18T08:04:41.836 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.001719897 2024-09-18T08:04:41.836 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:04:41.836 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:04:41.836 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "started", 2024-09-18T08:04:41.836 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:04:41.299383+0000", 2024-09-18T08:04:41.837 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.00013170400000000001 2024-09-18T08:04:41.837 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:04:41.837 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:04:41.837 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "waiting for subops from 4,7", 2024-09-18T08:04:41.837 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:04:41.299528+0000", 2024-09-18T08:04:41.837 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.00014523100000000001 2024-09-18T08:04:41.837 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:04:41.837 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:04:41.837 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:04:41.837 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:04:41.306716+0000", 2024-09-18T08:04:41.837 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.0071874260000000002 2024-09-18T08:04:41.837 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:04:41.837 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:04:41.837 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:04:41.838 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:04:41.306756+0000", 2024-09-18T08:04:41.838 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 4.0435000000000002e-05 2024-09-18T08:04:41.838 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:04:41.838 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:04:41.838 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "op_commit", 2024-09-18T08:04:41.838 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:04:41.330772+0000", 2024-09-18T08:04:41.838 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.024015924000000001 2024-09-18T08:04:41.838 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:04:41.838 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:04:41.838 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "commit_sent", 2024-09-18T08:04:41.838 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:04:41.330832+0000", 2024-09-18T08:04:41.838 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 5.9540999999999998e-05 2024-09-18T08:04:41.838 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:04:41.838 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:04:41.838 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "done", 2024-09-18T08:04:41.839 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:04:41.330890+0000", 2024-09-18T08:04:41.839 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 5.8368000000000001e-05 2024-09-18T08:04:41.839 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-09-18T08:04:41.839 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-09-18T08:04:41.839 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-09-18T08:04:41.839 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-09-18T08:04:41.839 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-09-18T08:04:41.839 INFO:teuthology.orchestra.run.smithi007.stdout:} 2024-09-18T08:04:41.839 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5 2024-09-18T08:04:41.871 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:04:41.871 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:04:41.872 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5' 2024-09-18T08:04:41.972 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3 2024-09-18T08:04:42.014 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:04:42.015 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:04:42.015 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3' 2024-09-18T08:04:42.116 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7 2024-09-18T08:04:42.145 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:04:42.146 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:04:42.146 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7' 2024-09-18T08:04:42.247 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0 2024-09-18T08:04:42.269 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:04:42.270 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:04:42.270 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0' 2024-09-18T08:04:42.284 INFO:teuthology.orchestra.run.smithi007.stderr:set osd.6 primary-affinity to 0 (802) 2024-09-18T08:04:42.370 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1 2024-09-18T08:04:42.393 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:04:42.393 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:04:42.394 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1' 2024-09-18T08:04:42.495 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6 2024-09-18T08:04:42.517 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:04:42.518 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:04:42.518 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6' 2024-09-18T08:04:42.557 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:04:42 smithi050 ceph-mon[32371]: pgmap v72: 148 pgs: 148 active+clean; 3.7 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 639 B/s rd, 85 MiB/s wr, 1.36k op/s 2024-09-18T08:04:42.557 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:04:42 smithi050 ceph-mon[32371]: from='client.? 172.21.15.7:0/2053842371' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 6, "weight": 0.0} : dispatch 2024-09-18T08:04:42.581 INFO:tasks.radosbench.radosbench.0.smithi007.stdout: 37 16 65122 65106 109.934 54.125 0.00221113 0.00902595 2024-09-18T08:04:42.619 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4 2024-09-18T08:04:42.633 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:04:42 smithi007 ceph-mon[29459]: pgmap v72: 148 pgs: 148 active+clean; 3.7 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 639 B/s rd, 85 MiB/s wr, 1.36k op/s 2024-09-18T08:04:42.633 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:04:42 smithi007 ceph-mon[29459]: from='client.? 172.21.15.7:0/2053842371' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 6, "weight": 0.0} : dispatch 2024-09-18T08:04:42.633 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:04:42 smithi007 ceph-mon[39249]: pgmap v72: 148 pgs: 148 active+clean; 3.7 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 639 B/s rd, 85 MiB/s wr, 1.36k op/s 2024-09-18T08:04:42.633 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:04:42 smithi007 ceph-mon[39249]: from='client.? 172.21.15.7:0/2053842371' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 6, "weight": 0.0} : dispatch 2024-09-18T08:04:42.642 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:04:42.643 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:04:42.643 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4' 2024-09-18T08:04:42.744 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7 2024-09-18T08:04:42.766 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:04:42.766 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:04:42.767 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7' 2024-09-18T08:04:42.867 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2 2024-09-18T08:04:42.892 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:04:42.893 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:04:42.893 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2' 2024-09-18T08:04:42.994 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2 2024-09-18T08:04:43.019 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:04:43.019 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:04:43.020 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2' 2024-09-18T08:04:43.120 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4 2024-09-18T08:04:43.143 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:04:43.143 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:04:43.144 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4' 2024-09-18T08:04:43.245 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7 2024-09-18T08:04:43.269 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:04:43.269 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:04:43.269 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7' 2024-09-18T08:04:43.370 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1 2024-09-18T08:04:43.411 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:04:43.411 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:04:43.412 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1' 2024-09-18T08:04:43.480 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:04:43 smithi007 ceph-mon[29459]: from='client.? 172.21.15.7:0/2053842371' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 6, "weight": 0.0}]': finished 2024-09-18T08:04:43.480 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:04:43 smithi007 ceph-mon[29459]: osdmap e101: 8 total, 8 up, 8 in 2024-09-18T08:04:43.481 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:04:43 smithi007 ceph-mon[39249]: from='client.? 172.21.15.7:0/2053842371' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 6, "weight": 0.0}]': finished 2024-09-18T08:04:43.481 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:04:43 smithi007 ceph-mon[39249]: osdmap e101: 8 total, 8 up, 8 in 2024-09-18T08:04:43.513 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7 2024-09-18T08:04:43.545 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:04:43.546 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:04:43.546 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7' 2024-09-18T08:04:43.581 INFO:tasks.radosbench.radosbench.0.smithi007.stdout: 38 16 65499 65483 107.662 23.5625 0.00794614 0.00928308 2024-09-18T08:04:43.647 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5 2024-09-18T08:04:43.655 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:04:43 smithi050 ceph-mon[32371]: from='client.? 172.21.15.7:0/2053842371' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 6, "weight": 0.0}]': finished 2024-09-18T08:04:43.656 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:04:43 smithi050 ceph-mon[32371]: osdmap e101: 8 total, 8 up, 8 in 2024-09-18T08:04:43.677 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:04:43.678 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:04:43.678 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5' 2024-09-18T08:04:43.779 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7 2024-09-18T08:04:43.808 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:04:43.809 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:04:43.809 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7' 2024-09-18T08:04:43.910 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3 2024-09-18T08:04:43.952 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:04:43.953 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:04:43.953 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3' 2024-09-18T08:04:44.054 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6 2024-09-18T08:04:44.085 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:04:44.086 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:04:44.086 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6' 2024-09-18T08:04:44.187 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0 2024-09-18T08:04:44.229 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:04:44.230 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:04:44.230 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0' 2024-09-18T08:04:44.331 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3 2024-09-18T08:04:44.368 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:04:44.369 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:04:44.369 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3' 2024-09-18T08:04:44.406 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:04:44 smithi050 ceph-mon[32371]: pgmap v74: 148 pgs: 5 peering, 143 active+clean; 3.7 GiB data, 12 GiB used, 704 GiB / 715 GiB avail; 1.0 KiB/s rd, 58 MiB/s wr, 922 op/s 2024-09-18T08:04:44.406 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:04:44 smithi050 ceph-mon[32371]: Health check failed: Reduced data availability: 2 pgs inactive, 5 pgs peering (PG_AVAILABILITY) 2024-09-18T08:04:44.406 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:04:44 smithi050 ceph-mon[32371]: osdmap e102: 8 total, 8 up, 8 in 2024-09-18T08:04:44.470 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3 2024-09-18T08:04:44.515 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:04:44.515 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:04:44.516 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3' 2024-09-18T08:04:44.555 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:04:44 smithi007 ceph-mon[29459]: pgmap v74: 148 pgs: 5 peering, 143 active+clean; 3.7 GiB data, 12 GiB used, 704 GiB / 715 GiB avail; 1.0 KiB/s rd, 58 MiB/s wr, 922 op/s 2024-09-18T08:04:44.556 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:04:44 smithi007 ceph-mon[29459]: Health check failed: Reduced data availability: 2 pgs inactive, 5 pgs peering (PG_AVAILABILITY) 2024-09-18T08:04:44.556 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:04:44 smithi007 ceph-mon[29459]: osdmap e102: 8 total, 8 up, 8 in 2024-09-18T08:04:44.556 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:04:44 smithi007 ceph-mon[39249]: pgmap v74: 148 pgs: 5 peering, 143 active+clean; 3.7 GiB data, 12 GiB used, 704 GiB / 715 GiB avail; 1.0 KiB/s rd, 58 MiB/s wr, 922 op/s 2024-09-18T08:04:44.556 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:04:44 smithi007 ceph-mon[39249]: Health check failed: Reduced data availability: 2 pgs inactive, 5 pgs peering (PG_AVAILABILITY) 2024-09-18T08:04:44.556 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:04:44 smithi007 ceph-mon[39249]: osdmap e102: 8 total, 8 up, 8 in 2024-09-18T08:04:44.581 INFO:tasks.radosbench.radosbench.0.smithi007.stdout: 39 15 66770 66755 106.939 79.5 0.0175107 0.00934566 2024-09-18T08:04:44.617 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6 2024-09-18T08:04:44.645 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:04:44.645 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:04:44.646 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6' 2024-09-18T08:04:44.747 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2 2024-09-18T08:04:44.788 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:04:44.789 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:04:44.789 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2' 2024-09-18T08:04:44.890 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3 2024-09-18T08:04:44.928 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:04:44.929 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:04:44.929 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3' 2024-09-18T08:04:45.030 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5 2024-09-18T08:04:45.058 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:04:45.058 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:04:45.059 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5' 2024-09-18T08:04:45.156 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid 779349ca-7593-11ef-bceb-c7b262605968 -- ceph daemon osd.3 dump_ops_in_flight 2024-09-18T08:04:45.160 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5 2024-09-18T08:04:45.185 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:04:45.186 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:04:45.187 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5' 2024-09-18T08:04:45.287 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0 2024-09-18T08:04:45.347 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:04:45.348 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:04:45.349 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0' 2024-09-18T08:04:45.449 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1 2024-09-18T08:04:45.493 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:04:45.493 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:04:45.494 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1' 2024-09-18T08:04:45.582 INFO:tasks.radosbench.radosbench.0.smithi007.stdout:2024-09-18T08:04:45.580780+0000 min lat: 0.00117713 max lat: 1.0787 avg lat: 0.0094061 2024-09-18T08:04:45.582 INFO:tasks.radosbench.radosbench.0.smithi007.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-09-18T08:04:45.582 INFO:tasks.radosbench.radosbench.0.smithi007.stdout: 40 16 68041 68025 106.25 79.375 0.00768033 0.0094061 2024-09-18T08:04:45.594 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4 2024-09-18T08:04:45.620 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:04:45.620 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:04:45.621 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4' 2024-09-18T08:04:45.724 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2 2024-09-18T08:04:45.770 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/779349ca-7593-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T08:04:45.798 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:04:45.798 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:04:45.799 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2' 2024-09-18T08:04:45.900 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5 2024-09-18T08:04:45.925 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/779349ca-7593-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T08:04:45.926 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:04:45.927 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:04:45.927 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5' 2024-09-18T08:04:45.980 INFO:journalctl@ceph.alertmanager.a.smithi007.stdout:Sep 18 08:04:45 smithi007 ceph-779349ca-7593-11ef-bceb-c7b262605968-alertmanager-a[70326]: ts=2024-09-18T08:04:45.913Z caller=main.go:542 level=info msg="Received SIGTERM, exiting gracefully..." 2024-09-18T08:04:46.028 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3 2024-09-18T08:04:46.079 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:04:46.080 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:04:46.080 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3' 2024-09-18T08:04:46.181 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2 2024-09-18T08:04:46.223 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:04:46.223 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:04:46.224 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2' 2024-09-18T08:04:46.303 INFO:journalctl@ceph.alertmanager.a.smithi007.stdout:Sep 18 08:04:46 smithi007 podman[77870]: 2024-09-18 08:04:46.065591786 +0000 UTC m=+0.323534367 container died 5fd76c77cedf61a73045fde95d31e55a84b78edda9fe5b68ede971e9ff30b7a3 (image=quay.io/prometheus/alertmanager:v0.27.0, name=ceph-779349ca-7593-11ef-bceb-c7b262605968-alertmanager-a, maintainer=The Prometheus Authors ) 2024-09-18T08:04:46.325 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4 2024-09-18T08:04:46.352 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:04:46.353 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:04:46.353 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4' 2024-09-18T08:04:46.454 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6 2024-09-18T08:04:46.483 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:04:46.483 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:04:46.484 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6' 2024-09-18T08:04:46.582 INFO:tasks.radosbench.radosbench.0.smithi007.stdout: 41 16 69319 69303 105.607 79.875 0.00761925 0.00946417 2024-09-18T08:04:46.584 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2 2024-09-18T08:04:46.594 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:04:46 smithi007 ceph-mon[29459]: pgmap v76: 148 pgs: 11 peering, 137 active+clean; 4.0 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 914 B/s rd, 91 MiB/s wr, 1.46k op/s 2024-09-18T08:04:46.595 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:04:46 smithi007 ceph-mon[39249]: pgmap v76: 148 pgs: 11 peering, 137 active+clean; 4.0 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 914 B/s rd, 91 MiB/s wr, 1.46k op/s 2024-09-18T08:04:46.631 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:04:46.631 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:04:46.631 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2' 2024-09-18T08:04:46.656 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:04:46 smithi050 ceph-mon[32371]: pgmap v76: 148 pgs: 11 peering, 137 active+clean; 4.0 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 914 B/s rd, 91 MiB/s wr, 1.46k op/s 2024-09-18T08:04:46.732 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4 2024-09-18T08:04:46.757 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:04:46.758 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:04:46.759 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4' 2024-09-18T08:04:46.856 INFO:journalctl@ceph.alertmanager.a.smithi007.stdout:Sep 18 08:04:46 smithi007 podman[77870]: 2024-09-18 08:04:46.593476477 +0000 UTC m=+0.851419053 container cleanup 5fd76c77cedf61a73045fde95d31e55a84b78edda9fe5b68ede971e9ff30b7a3 (image=quay.io/prometheus/alertmanager:v0.27.0, name=ceph-779349ca-7593-11ef-bceb-c7b262605968-alertmanager-a, maintainer=The Prometheus Authors ) 2024-09-18T08:04:46.856 INFO:journalctl@ceph.alertmanager.a.smithi007.stdout:Sep 18 08:04:46 smithi007 bash[77870]: ceph-779349ca-7593-11ef-bceb-c7b262605968-alertmanager-a 2024-09-18T08:04:46.857 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Sep 18 08:04:46 smithi007 ceph-779349ca-7593-11ef-bceb-c7b262605968-mgr-y[29688]: ::ffff:172.21.15.50 - - [18/Sep/2024:08:04:46] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-09-18T08:04:46.859 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4 2024-09-18T08:04:46.886 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:04:46.887 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:04:46.887 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4' 2024-09-18T08:04:46.988 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5 2024-09-18T08:04:47.014 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:04:47.014 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:04:47.015 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5' 2024-09-18T08:04:47.044 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid 779349ca-7593-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-18T08:04:47.116 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0 2024-09-18T08:04:47.159 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:04:47.159 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:04:47.160 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0' 2024-09-18T08:04:47.260 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7 2024-09-18T08:04:47.286 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:04:47.287 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:04:47.287 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7' 2024-09-18T08:04:47.388 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1 2024-09-18T08:04:47.446 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:04:47.447 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:04:47.448 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1' 2024-09-18T08:04:47.549 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3 2024-09-18T08:04:47.582 INFO:tasks.radosbench.radosbench.0.smithi007.stdout: 42 16 70395 70379 104.694 67.25 0.0109777 0.00954624 2024-09-18T08:04:47.601 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:04:47.601 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:04:47.602 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3' 2024-09-18T08:04:47.684 INFO:journalctl@ceph.alertmanager.a.smithi007.stdout:Sep 18 08:04:47 smithi007 podman[77930]: 2024-09-18 08:04:47.302081626 +0000 UTC m=+1.334489277 container remove 5fd76c77cedf61a73045fde95d31e55a84b78edda9fe5b68ede971e9ff30b7a3 (image=quay.io/prometheus/alertmanager:v0.27.0, name=ceph-779349ca-7593-11ef-bceb-c7b262605968-alertmanager-a, maintainer=The Prometheus Authors ) 2024-09-18T08:04:47.685 INFO:journalctl@ceph.alertmanager.a.smithi007.stdout:Sep 18 08:04:47 smithi007 podman[77930]: 2024-09-18 08:04:47.395125238 +0000 UTC m=+1.427532890 volume remove 1d799220f78d0a5081bb972ed6edb172e3eb23d5c0d107ce2b39e3b2af71480c 2024-09-18T08:04:47.702 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3 2024-09-18T08:04:47.745 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:04:47.746 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:04:47.746 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3' 2024-09-18T08:04:47.847 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7 2024-09-18T08:04:47.871 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:04:47.872 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:04:47.872 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7' 2024-09-18T08:04:47.880 INFO:teuthology.orchestra.run.smithi007.stdout:{ 2024-09-18T08:04:47.880 INFO:teuthology.orchestra.run.smithi007.stdout: "ops": [ 2024-09-18T08:04:47.880 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:04:47.880 INFO:teuthology.orchestra.run.smithi007.stdout: "description": "osd_op(client.14805.0:70675 7.3 7:cacca754:::benchmark_data_smithi007_73155_object70674:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536 in=65536b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e102)", 2024-09-18T08:04:47.880 INFO:teuthology.orchestra.run.smithi007.stdout: "initiated_at": "2024-09-18T08:04:47.847558+0000", 2024-09-18T08:04:47.880 INFO:teuthology.orchestra.run.smithi007.stdout: "age": 0.012758693999999999, 2024-09-18T08:04:47.881 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.012791248, 2024-09-18T08:04:47.881 INFO:teuthology.orchestra.run.smithi007.stdout: "continuous": false, 2024-09-18T08:04:47.881 INFO:teuthology.orchestra.run.smithi007.stdout: "type_data": { 2024-09-18T08:04:47.881 INFO:teuthology.orchestra.run.smithi007.stdout: "flag_point": "waiting for sub ops", 2024-09-18T08:04:47.881 INFO:teuthology.orchestra.run.smithi007.stdout: "client_info": { 2024-09-18T08:04:47.881 INFO:teuthology.orchestra.run.smithi007.stdout: "client": "client.14805", 2024-09-18T08:04:47.881 INFO:teuthology.orchestra.run.smithi007.stdout: "client_addr": "172.21.15.7:0/640320436", 2024-09-18T08:04:47.881 INFO:teuthology.orchestra.run.smithi007.stdout: "tid": 70675 2024-09-18T08:04:47.881 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:04:47.882 INFO:teuthology.orchestra.run.smithi007.stdout: "events": [ 2024-09-18T08:04:47.882 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:04:47.882 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "initiated", 2024-09-18T08:04:47.882 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:04:47.847558+0000", 2024-09-18T08:04:47.882 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-09-18T08:04:47.882 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:04:47.882 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:04:47.882 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "header_read", 2024-09-18T08:04:47.882 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:04:47.847558+0000", 2024-09-18T08:04:47.882 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-09-18T08:04:47.882 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:04:47.882 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:04:47.883 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "throttled", 2024-09-18T08:04:47.883 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:04:47.847558+0000", 2024-09-18T08:04:47.883 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 6.13e-07 2024-09-18T08:04:47.883 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:04:47.883 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:04:47.883 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "all_read", 2024-09-18T08:04:47.883 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:04:47.847594+0000", 2024-09-18T08:04:47.883 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 3.5280000000000001e-05 2024-09-18T08:04:47.883 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:04:47.883 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:04:47.883 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "dispatched", 2024-09-18T08:04:47.884 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:04:47.847608+0000", 2024-09-18T08:04:47.884 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.4545999999999999e-05 2024-09-18T08:04:47.884 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:04:47.884 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:04:47.884 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "queued_for_pg", 2024-09-18T08:04:47.884 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:04:47.847615+0000", 2024-09-18T08:04:47.884 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 7.075e-06 2024-09-18T08:04:47.884 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:04:47.884 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:04:47.884 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-09-18T08:04:47.884 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:04:47.851643+0000", 2024-09-18T08:04:47.884 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.0040280230000000004 2024-09-18T08:04:47.885 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:04:47.885 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:04:47.885 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "started", 2024-09-18T08:04:47.885 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:04:47.851763+0000", 2024-09-18T08:04:47.885 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.00011969899999999999 2024-09-18T08:04:47.885 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:04:47.885 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:04:47.885 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "waiting for subops from 2,4", 2024-09-18T08:04:47.885 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:04:47.851915+0000", 2024-09-18T08:04:47.885 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.00015196 2024-09-18T08:04:47.885 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:04:47.885 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:04:47.885 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "op_commit", 2024-09-18T08:04:47.886 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:04:47.855710+0000", 2024-09-18T08:04:47.886 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.0037952770000000001 2024-09-18T08:04:47.886 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:04:47.886 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:04:47.886 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:04:47.886 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:04:47.860159+0000", 2024-09-18T08:04:47.886 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.00444883 2024-09-18T08:04:47.886 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-09-18T08:04:47.886 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-09-18T08:04:47.886 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-09-18T08:04:47.886 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:04:47.886 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:04:47.886 INFO:teuthology.orchestra.run.smithi007.stdout: "description": "osd_op(client.14805.0:70676 7.3 7:c39c8cd3:::benchmark_data_smithi007_73155_object70675:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536 in=65536b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e102)", 2024-09-18T08:04:47.887 INFO:teuthology.orchestra.run.smithi007.stdout: "initiated_at": "2024-09-18T08:04:47.848190+0000", 2024-09-18T08:04:47.887 INFO:teuthology.orchestra.run.smithi007.stdout: "age": 0.012126878000000001, 2024-09-18T08:04:47.887 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.012240620000000001, 2024-09-18T08:04:47.887 INFO:teuthology.orchestra.run.smithi007.stdout: "continuous": false, 2024-09-18T08:04:47.887 INFO:teuthology.orchestra.run.smithi007.stdout: "type_data": { 2024-09-18T08:04:47.887 INFO:teuthology.orchestra.run.smithi007.stdout: "flag_point": "waiting for sub ops", 2024-09-18T08:04:47.887 INFO:teuthology.orchestra.run.smithi007.stdout: "client_info": { 2024-09-18T08:04:47.887 INFO:teuthology.orchestra.run.smithi007.stdout: "client": "client.14805", 2024-09-18T08:04:47.887 INFO:teuthology.orchestra.run.smithi007.stdout: "client_addr": "172.21.15.7:0/640320436", 2024-09-18T08:04:47.887 INFO:teuthology.orchestra.run.smithi007.stdout: "tid": 70676 2024-09-18T08:04:47.887 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:04:47.887 INFO:teuthology.orchestra.run.smithi007.stdout: "events": [ 2024-09-18T08:04:47.888 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:04:47.888 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "initiated", 2024-09-18T08:04:47.888 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:04:47.848190+0000", 2024-09-18T08:04:47.888 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-09-18T08:04:47.888 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:04:47.888 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:04:47.888 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "header_read", 2024-09-18T08:04:47.888 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:04:47.848190+0000", 2024-09-18T08:04:47.888 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-09-18T08:04:47.888 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:04:47.888 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:04:47.888 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "throttled", 2024-09-18T08:04:47.888 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:04:47.848191+0000", 2024-09-18T08:04:47.889 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.2780000000000001e-06 2024-09-18T08:04:47.889 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:04:47.889 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:04:47.889 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "all_read", 2024-09-18T08:04:47.889 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:04:47.848216+0000", 2024-09-18T08:04:47.889 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 2.4712e-05 2024-09-18T08:04:47.889 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:04:47.889 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:04:47.889 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "dispatched", 2024-09-18T08:04:47.889 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:04:47.848228+0000", 2024-09-18T08:04:47.889 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.2e-05 2024-09-18T08:04:47.889 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:04:47.889 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:04:47.890 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "queued_for_pg", 2024-09-18T08:04:47.890 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:04:47.848236+0000", 2024-09-18T08:04:47.890 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 8.2979999999999992e-06 2024-09-18T08:04:47.890 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:04:47.890 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:04:47.890 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-09-18T08:04:47.890 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:04:47.856243+0000", 2024-09-18T08:04:47.890 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.0080068359999999998 2024-09-18T08:04:47.890 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:04:47.890 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:04:47.890 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "started", 2024-09-18T08:04:47.891 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:04:47.856384+0000", 2024-09-18T08:04:47.891 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.000141414 2024-09-18T08:04:47.891 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:04:47.891 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:04:47.891 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "waiting for subops from 2,4", 2024-09-18T08:04:47.891 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:04:47.856539+0000", 2024-09-18T08:04:47.891 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.00015507800000000001 2024-09-18T08:04:47.891 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-09-18T08:04:47.891 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-09-18T08:04:47.891 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-09-18T08:04:47.891 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:04:47.892 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:04:47.892 INFO:teuthology.orchestra.run.smithi007.stdout: "description": "osd_op(client.14805.0:70682 7.2 7:482795d0:::benchmark_data_smithi007_73155_object70681:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536 in=65536b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e102)", 2024-09-18T08:04:47.892 INFO:teuthology.orchestra.run.smithi007.stdout: "initiated_at": "2024-09-18T08:04:47.854552+0000", 2024-09-18T08:04:47.892 INFO:teuthology.orchestra.run.smithi007.stdout: "age": 0.0057641319999999999, 2024-09-18T08:04:47.892 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.0059435759999999999, 2024-09-18T08:04:47.892 INFO:teuthology.orchestra.run.smithi007.stdout: "continuous": false, 2024-09-18T08:04:47.892 INFO:teuthology.orchestra.run.smithi007.stdout: "type_data": { 2024-09-18T08:04:47.892 INFO:teuthology.orchestra.run.smithi007.stdout: "flag_point": "waiting for sub ops", 2024-09-18T08:04:47.892 INFO:teuthology.orchestra.run.smithi007.stdout: "client_info": { 2024-09-18T08:04:47.892 INFO:teuthology.orchestra.run.smithi007.stdout: "client": "client.14805", 2024-09-18T08:04:47.892 INFO:teuthology.orchestra.run.smithi007.stdout: "client_addr": "172.21.15.7:0/640320436", 2024-09-18T08:04:47.892 INFO:teuthology.orchestra.run.smithi007.stdout: "tid": 70682 2024-09-18T08:04:47.893 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:04:47.893 INFO:teuthology.orchestra.run.smithi007.stdout: "events": [ 2024-09-18T08:04:47.893 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:04:47.893 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "initiated", 2024-09-18T08:04:47.893 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:04:47.854552+0000", 2024-09-18T08:04:47.893 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-09-18T08:04:47.893 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:04:47.893 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:04:47.893 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "header_read", 2024-09-18T08:04:47.894 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:04:47.854552+0000", 2024-09-18T08:04:47.894 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-09-18T08:04:47.894 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:04:47.894 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:04:47.894 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "throttled", 2024-09-18T08:04:47.894 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:04:47.854554+0000", 2024-09-18T08:04:47.895 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.4449999999999999e-06 2024-09-18T08:04:47.895 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:04:47.895 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:04:47.895 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "all_read", 2024-09-18T08:04:47.895 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:04:47.854584+0000", 2024-09-18T08:04:47.895 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 3.0682999999999997e-05 2024-09-18T08:04:47.895 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:04:47.895 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:04:47.895 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "dispatched", 2024-09-18T08:04:47.895 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:04:47.854599+0000", 2024-09-18T08:04:47.895 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.4780000000000001e-05 2024-09-18T08:04:47.895 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:04:47.895 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:04:47.896 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "queued_for_pg", 2024-09-18T08:04:47.896 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:04:47.854607+0000", 2024-09-18T08:04:47.896 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 7.9510000000000006e-06 2024-09-18T08:04:47.896 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:04:47.896 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:04:47.896 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-09-18T08:04:47.896 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:04:47.854688+0000", 2024-09-18T08:04:47.896 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 8.0475000000000005e-05 2024-09-18T08:04:47.896 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:04:47.896 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:04:47.896 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "started", 2024-09-18T08:04:47.896 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:04:47.854825+0000", 2024-09-18T08:04:47.896 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.000137692 2024-09-18T08:04:47.897 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:04:47.897 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:04:47.897 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "waiting for subops from 1,2", 2024-09-18T08:04:47.897 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:04:47.854978+0000", 2024-09-18T08:04:47.897 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.000152688 2024-09-18T08:04:47.897 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:04:47.897 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:04:47.897 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "op_commit", 2024-09-18T08:04:47.897 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:04:47.857986+0000", 2024-09-18T08:04:47.897 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.003008153 2024-09-18T08:04:47.897 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-09-18T08:04:47.897 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-09-18T08:04:47.897 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-09-18T08:04:47.898 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:04:47.898 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:04:47.898 INFO:teuthology.orchestra.run.smithi007.stdout: "description": "osd_repop(client.14805.0:70685 7.1 e102/83 7:827e2993:::benchmark_data_smithi007_73155_object70684:head v 102'4474, mlcod=102'4472)", 2024-09-18T08:04:47.898 INFO:teuthology.orchestra.run.smithi007.stdout: "initiated_at": "2024-09-18T08:04:47.855082+0000", 2024-09-18T08:04:47.898 INFO:teuthology.orchestra.run.smithi007.stdout: "age": 0.0052345539999999998, 2024-09-18T08:04:47.898 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.0054791969999999999, 2024-09-18T08:04:47.898 INFO:teuthology.orchestra.run.smithi007.stdout: "continuous": false, 2024-09-18T08:04:47.898 INFO:teuthology.orchestra.run.smithi007.stdout: "type_data": { 2024-09-18T08:04:47.898 INFO:teuthology.orchestra.run.smithi007.stdout: "flag_point": "started", 2024-09-18T08:04:47.898 INFO:teuthology.orchestra.run.smithi007.stdout: "events": [ 2024-09-18T08:04:47.898 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:04:47.899 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "initiated", 2024-09-18T08:04:47.899 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:04:47.855082+0000", 2024-09-18T08:04:47.899 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-09-18T08:04:47.899 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:04:47.899 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:04:47.899 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "header_read", 2024-09-18T08:04:47.899 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:04:47.855082+0000", 2024-09-18T08:04:47.899 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-09-18T08:04:47.899 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:04:47.899 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:04:47.899 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "throttled", 2024-09-18T08:04:47.899 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:04:47.855082+0000", 2024-09-18T08:04:47.899 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 6.2900000000000003e-07 2024-09-18T08:04:47.900 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:04:47.900 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:04:47.900 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "all_read", 2024-09-18T08:04:47.900 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:04:47.855338+0000", 2024-09-18T08:04:47.900 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.00025527299999999998 2024-09-18T08:04:47.900 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:04:47.900 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:04:47.900 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "dispatched", 2024-09-18T08:04:47.900 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:04:47.855348+0000", 2024-09-18T08:04:47.900 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.0638e-05 2024-09-18T08:04:47.900 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:04:47.900 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:04:47.900 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "queued_for_pg", 2024-09-18T08:04:47.901 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:04:47.855355+0000", 2024-09-18T08:04:47.901 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 6.7259999999999996e-06 2024-09-18T08:04:47.901 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:04:47.901 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:04:47.901 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-09-18T08:04:47.901 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:04:47.855428+0000", 2024-09-18T08:04:47.901 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 7.3044000000000001e-05 2024-09-18T08:04:47.901 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:04:47.901 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:04:47.901 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "started", 2024-09-18T08:04:47.901 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:04:47.855450+0000", 2024-09-18T08:04:47.901 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 2.1464e-05 2024-09-18T08:04:47.901 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-09-18T08:04:47.901 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-09-18T08:04:47.902 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-09-18T08:04:47.902 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:04:47.902 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:04:47.902 INFO:teuthology.orchestra.run.smithi007.stdout: "description": "osd_repop(client.14805.0:70683 7.7 e102/101 7:ee5f4f41:::benchmark_data_smithi007_73155_object70682:head v 102'4411, mlcod=102'4409)", 2024-09-18T08:04:47.902 INFO:teuthology.orchestra.run.smithi007.stdout: "initiated_at": "2024-09-18T08:04:47.857750+0000", 2024-09-18T08:04:47.902 INFO:teuthology.orchestra.run.smithi007.stdout: "age": 0.0025665060000000001, 2024-09-18T08:04:47.902 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.0028579550000000001, 2024-09-18T08:04:47.902 INFO:teuthology.orchestra.run.smithi007.stdout: "continuous": false, 2024-09-18T08:04:47.902 INFO:teuthology.orchestra.run.smithi007.stdout: "type_data": { 2024-09-18T08:04:47.902 INFO:teuthology.orchestra.run.smithi007.stdout: "flag_point": "started", 2024-09-18T08:04:47.902 INFO:teuthology.orchestra.run.smithi007.stdout: "events": [ 2024-09-18T08:04:47.902 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:04:47.902 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "initiated", 2024-09-18T08:04:47.902 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:04:47.857750+0000", 2024-09-18T08:04:47.903 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-09-18T08:04:47.903 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:04:47.903 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:04:47.903 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "header_read", 2024-09-18T08:04:47.903 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:04:47.857750+0000", 2024-09-18T08:04:47.903 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-09-18T08:04:47.903 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:04:47.903 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:04:47.903 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "throttled", 2024-09-18T08:04:47.903 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:04:47.857751+0000", 2024-09-18T08:04:47.903 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 6.6899999999999997e-07 2024-09-18T08:04:47.903 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:04:47.903 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:04:47.904 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "all_read", 2024-09-18T08:04:47.904 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:04:47.859687+0000", 2024-09-18T08:04:47.904 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.00193615 2024-09-18T08:04:47.904 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:04:47.904 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:04:47.904 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "dispatched", 2024-09-18T08:04:47.905 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:04:47.859701+0000", 2024-09-18T08:04:47.905 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.3811e-05 2024-09-18T08:04:47.905 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:04:47.905 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:04:47.905 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "queued_for_pg", 2024-09-18T08:04:47.905 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:04:47.859708+0000", 2024-09-18T08:04:47.905 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 7.294e-06 2024-09-18T08:04:47.905 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:04:47.905 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:04:47.905 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-09-18T08:04:47.905 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:04:47.859805+0000", 2024-09-18T08:04:47.905 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 9.7119000000000003e-05 2024-09-18T08:04:47.906 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:04:47.906 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:04:47.906 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "started", 2024-09-18T08:04:47.906 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:04:47.859823+0000", 2024-09-18T08:04:47.906 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.7697999999999999e-05 2024-09-18T08:04:47.906 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-09-18T08:04:47.906 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-09-18T08:04:47.906 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-09-18T08:04:47.906 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:04:47.906 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:04:47.906 INFO:teuthology.orchestra.run.smithi007.stdout: "description": "osd_repop_reply(client.14805.0:70676 7.3 e102/83)", 2024-09-18T08:04:47.906 INFO:teuthology.orchestra.run.smithi007.stdout: "initiated_at": "2024-09-18T08:04:47.860064+0000", 2024-09-18T08:04:47.906 INFO:teuthology.orchestra.run.smithi007.stdout: "age": 0.00025238099999999998, 2024-09-18T08:04:47.906 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.00058794399999999997, 2024-09-18T08:04:47.907 INFO:teuthology.orchestra.run.smithi007.stdout: "continuous": false, 2024-09-18T08:04:47.907 INFO:teuthology.orchestra.run.smithi007.stdout: "type_data": { 2024-09-18T08:04:47.907 INFO:teuthology.orchestra.run.smithi007.stdout: "flag_point": "queued for pg", 2024-09-18T08:04:47.907 INFO:teuthology.orchestra.run.smithi007.stdout: "events": [ 2024-09-18T08:04:47.907 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:04:47.907 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "initiated", 2024-09-18T08:04:47.907 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:04:47.860064+0000", 2024-09-18T08:04:47.907 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-09-18T08:04:47.907 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:04:47.907 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:04:47.907 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "header_read", 2024-09-18T08:04:47.907 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:04:47.860064+0000", 2024-09-18T08:04:47.907 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-09-18T08:04:47.908 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:04:47.908 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:04:47.908 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "throttled", 2024-09-18T08:04:47.908 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:04:47.860065+0000", 2024-09-18T08:04:47.908 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 7.8899999999999998e-07 2024-09-18T08:04:47.908 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:04:47.908 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:04:47.908 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "all_read", 2024-09-18T08:04:47.908 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:04:47.860068+0000", 2024-09-18T08:04:47.908 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 3.3340000000000002e-06 2024-09-18T08:04:47.908 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:04:47.908 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:04:47.909 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "dispatched", 2024-09-18T08:04:47.909 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:04:47.860078+0000", 2024-09-18T08:04:47.909 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 9.4099999999999997e-06 2024-09-18T08:04:47.909 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:04:47.909 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:04:47.909 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "queued_for_pg", 2024-09-18T08:04:47.910 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:04:47.860084+0000", 2024-09-18T08:04:47.910 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 6.127e-06 2024-09-18T08:04:47.910 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-09-18T08:04:47.910 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-09-18T08:04:47.910 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-09-18T08:04:47.910 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-09-18T08:04:47.910 INFO:teuthology.orchestra.run.smithi007.stdout: ], 2024-09-18T08:04:47.910 INFO:teuthology.orchestra.run.smithi007.stdout: "num_ops": 6 2024-09-18T08:04:47.910 INFO:teuthology.orchestra.run.smithi007.stdout:} 2024-09-18T08:04:47.973 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1 2024-09-18T08:04:48.016 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:04:48.017 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:04:48.018 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1' 2024-09-18T08:04:48.119 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7 2024-09-18T08:04:48.147 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:04:48.148 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:04:48.148 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7' 2024-09-18T08:04:48.229 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/779349ca-7593-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T08:04:48.250 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7 2024-09-18T08:04:48.274 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:04:48.274 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:04:48.275 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7' 2024-09-18T08:04:48.376 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4 2024-09-18T08:04:48.402 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:04:48.403 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:04:48.404 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4' 2024-09-18T08:04:48.505 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4 2024-09-18T08:04:48.535 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:04:48.536 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:04:48.536 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4' 2024-09-18T08:04:48.582 INFO:tasks.radosbench.radosbench.0.smithi007.stdout: 43 16 71585 71569 103.989 74.375 0.00396313 0.00961147 2024-09-18T08:04:48.637 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7 2024-09-18T08:04:48.643 INFO:teuthology.orchestra.run.smithi007.stderr:nodeep-scrub is set 2024-09-18T08:04:48.645 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:04:48 smithi007 ceph-mon[29459]: pgmap v77: 148 pgs: 11 peering, 137 active+clean; 4.0 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 255 B/s rd, 40 MiB/s wr, 642 op/s 2024-09-18T08:04:48.645 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:04:48 smithi007 ceph-mon[29459]: from='mgr.24470 172.21.15.7:0/2594390261' entity='mgr.y' 2024-09-18T08:04:48.645 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:04:48 smithi007 ceph-mon[29459]: from='mgr.24470 172.21.15.7:0/2594390261' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-18T08:04:48.645 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:04:48 smithi007 ceph-mon[29459]: from='client.? 172.21.15.7:0/680006939' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-18T08:04:48.645 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:04:48 smithi007 ceph-mon[29459]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-18T08:04:48.645 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:04:48 smithi007 ceph-mon[39249]: pgmap v77: 148 pgs: 11 peering, 137 active+clean; 4.0 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 255 B/s rd, 40 MiB/s wr, 642 op/s 2024-09-18T08:04:48.646 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:04:48 smithi007 ceph-mon[39249]: from='mgr.24470 172.21.15.7:0/2594390261' entity='mgr.y' 2024-09-18T08:04:48.646 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:04:48 smithi007 ceph-mon[39249]: from='mgr.24470 172.21.15.7:0/2594390261' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-18T08:04:48.646 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:04:48 smithi007 ceph-mon[39249]: from='client.? 172.21.15.7:0/680006939' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-18T08:04:48.646 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:04:48 smithi007 ceph-mon[39249]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-18T08:04:48.655 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:04:48 smithi050 ceph-mon[32371]: pgmap v77: 148 pgs: 11 peering, 137 active+clean; 4.0 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 255 B/s rd, 40 MiB/s wr, 642 op/s 2024-09-18T08:04:48.656 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:04:48 smithi050 ceph-mon[32371]: from='mgr.24470 172.21.15.7:0/2594390261' entity='mgr.y' 2024-09-18T08:04:48.656 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:04:48 smithi050 ceph-mon[32371]: from='mgr.24470 172.21.15.7:0/2594390261' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-18T08:04:48.656 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:04:48 smithi050 ceph-mon[32371]: from='client.? 172.21.15.7:0/680006939' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-18T08:04:48.656 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:04:48 smithi050 ceph-mon[32371]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-18T08:04:48.665 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:04:48.665 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:04:48.666 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7' 2024-09-18T08:04:48.766 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6 2024-09-18T08:04:48.794 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid 779349ca-7593-11ef-bceb-c7b262605968 -- ceph daemon osd.3 dump_blocked_ops 2024-09-18T08:04:48.797 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:04:48.798 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:04:48.798 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6' 2024-09-18T08:04:48.899 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4 2024-09-18T08:04:48.928 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:04:48.929 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:04:48.929 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4' 2024-09-18T08:04:49.030 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3 2024-09-18T08:04:49.096 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:04:49.097 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:04:49.097 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3' 2024-09-18T08:04:49.156 INFO:journalctl@ceph.iscsi.iscsi.a.smithi050.stdout:Sep 18 08:04:48 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-iscsi-iscsi-a[57782]: debug there is no tcmu-runner data available 2024-09-18T08:04:49.198 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5 2024-09-18T08:04:49.226 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:04:49.226 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:04:49.227 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5' 2024-09-18T08:04:49.327 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4 2024-09-18T08:04:49.354 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:04:49.388 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:04:49 smithi007 ceph-mon[29459]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-18T08:04:49.388 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:04:49 smithi007 ceph-mon[29459]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-18T08:04:49.388 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:04:49 smithi007 ceph-mon[29459]: osdmap e103: 8 total, 8 up, 8 in 2024-09-18T08:04:49.388 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:04:49 smithi007 ceph-mon[29459]: Health check update: Reduced data availability: 4 pgs inactive, 8 pgs peering (PG_AVAILABILITY) 2024-09-18T08:04:49.388 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:04:49 smithi007 ceph-mon[29459]: osdmap e104: 8 total, 8 up, 8 in 2024-09-18T08:04:49.389 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:04:49 smithi007 ceph-mon[39249]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-18T08:04:49.389 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:04:49 smithi007 ceph-mon[39249]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-18T08:04:49.389 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:04:49 smithi007 ceph-mon[39249]: osdmap e103: 8 total, 8 up, 8 in 2024-09-18T08:04:49.389 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:04:49 smithi007 ceph-mon[39249]: Health check update: Reduced data availability: 4 pgs inactive, 8 pgs peering (PG_AVAILABILITY) 2024-09-18T08:04:49.389 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:04:49 smithi007 ceph-mon[39249]: osdmap e104: 8 total, 8 up, 8 in 2024-09-18T08:04:49.389 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:04:49.390 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4' 2024-09-18T08:04:49.491 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0 2024-09-18T08:04:49.532 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:04:49.532 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:04:49.533 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0' 2024-09-18T08:04:49.582 INFO:tasks.radosbench.radosbench.0.smithi007.stdout: 44 16 72752 72736 103.283 72.9375 0.00667183 0.00967621 2024-09-18T08:04:49.634 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7 2024-09-18T08:04:49.656 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:04:49 smithi050 ceph-mon[32371]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-18T08:04:49.656 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:04:49 smithi050 ceph-mon[32371]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-18T08:04:49.656 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:04:49 smithi050 ceph-mon[32371]: osdmap e103: 8 total, 8 up, 8 in 2024-09-18T08:04:49.656 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:04:49 smithi050 ceph-mon[32371]: Health check update: Reduced data availability: 4 pgs inactive, 8 pgs peering (PG_AVAILABILITY) 2024-09-18T08:04:49.656 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:04:49 smithi050 ceph-mon[32371]: osdmap e104: 8 total, 8 up, 8 in 2024-09-18T08:04:49.662 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:04:49.663 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:04:49.663 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7' 2024-09-18T08:04:49.731 INFO:journalctl@ceph.alertmanager.a.smithi007.stdout:Sep 18 08:04:49 smithi007 systemd[1]: ceph-779349ca-7593-11ef-bceb-c7b262605968@alertmanager.a.service: Deactivated successfully. 2024-09-18T08:04:49.731 INFO:journalctl@ceph.alertmanager.a.smithi007.stdout:Sep 18 08:04:49 smithi007 systemd[1]: Stopped Ceph alertmanager.a for 779349ca-7593-11ef-bceb-c7b262605968. 2024-09-18T08:04:49.731 INFO:journalctl@ceph.alertmanager.a.smithi007.stdout:Sep 18 08:04:49 smithi007 systemd[1]: ceph-779349ca-7593-11ef-bceb-c7b262605968@alertmanager.a.service: Consumed 1.261s CPU time. 2024-09-18T08:04:49.731 INFO:journalctl@ceph.alertmanager.a.smithi007.stdout:Sep 18 08:04:49 smithi007 systemd[1]: Starting Ceph alertmanager.a for 779349ca-7593-11ef-bceb-c7b262605968... 2024-09-18T08:04:49.764 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7 2024-09-18T08:04:49.794 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:04:49.795 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:04:49.795 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7' 2024-09-18T08:04:49.896 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4 2024-09-18T08:04:49.924 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:04:49.924 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:04:49.924 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4' 2024-09-18T08:04:50.026 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2 2024-09-18T08:04:50.071 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:04:50.072 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:04:50.072 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2' 2024-09-18T08:04:50.173 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7 2024-09-18T08:04:50.199 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:04:50.199 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:04:50.200 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7' 2024-09-18T08:04:50.218 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/779349ca-7593-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T08:04:50.252 INFO:teuthology.orchestra.run.smithi007.stdout:osd.0: {} 2024-09-18T08:04:50.252 INFO:teuthology.orchestra.run.smithi007.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-18T08:04:50.289 INFO:teuthology.orchestra.run.smithi007.stdout:osd.1: {} 2024-09-18T08:04:50.290 INFO:teuthology.orchestra.run.smithi007.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-18T08:04:50.300 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2 2024-09-18T08:04:50.330 INFO:teuthology.orchestra.run.smithi007.stdout:osd.2: {} 2024-09-18T08:04:50.330 INFO:teuthology.orchestra.run.smithi007.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-18T08:04:50.351 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:04:50.352 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:04:50.352 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2' 2024-09-18T08:04:50.368 INFO:teuthology.orchestra.run.smithi007.stdout:osd.3: {} 2024-09-18T08:04:50.368 INFO:teuthology.orchestra.run.smithi007.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-18T08:04:50.408 INFO:teuthology.orchestra.run.smithi007.stdout:osd.4: {} 2024-09-18T08:04:50.409 INFO:teuthology.orchestra.run.smithi007.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-18T08:04:50.409 INFO:journalctl@ceph.alertmanager.a.smithi007.stdout:Sep 18 08:04:50 smithi007 podman[78368]: 2024-09-18 08:04:50.223417139 +0000 UTC m=+0.047445344 image pull 11f11916f8cdfb17c1da74b20deb2075d778357b4d60474d82453303f8d461f5 quay.io/prometheus/alertmanager:v0.27.0 2024-09-18T08:04:50.409 INFO:journalctl@ceph.alertmanager.a.smithi007.stdout:Sep 18 08:04:50 smithi007 podman[78368]: 2024-09-18 08:04:50.336662042 +0000 UTC m=+0.160690226 volume create ad7ae1d96035a9fc08584d046b3eaa81b91f1bc84dc81fa9ef98adc7be4302b9 2024-09-18T08:04:50.410 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:04:50 smithi007 ceph-mon[29459]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-18T08:04:50.410 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:04:50 smithi007 ceph-mon[29459]: pgmap v80: 148 pgs: 148 active+clean; 4.3 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 682 B/s rd, 105 MiB/s wr, 1.68k op/s 2024-09-18T08:04:50.410 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:04:50 smithi007 ceph-mon[29459]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 4 pgs inactive, 8 pgs peering) 2024-09-18T08:04:50.410 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:04:50 smithi007 ceph-mon[29459]: from='client.? 172.21.15.7:0/429459757' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-18T08:04:50.410 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:04:50 smithi007 ceph-mon[39249]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-18T08:04:50.410 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:04:50 smithi007 ceph-mon[39249]: pgmap v80: 148 pgs: 148 active+clean; 4.3 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 682 B/s rd, 105 MiB/s wr, 1.68k op/s 2024-09-18T08:04:50.410 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:04:50 smithi007 ceph-mon[39249]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 4 pgs inactive, 8 pgs peering) 2024-09-18T08:04:50.410 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:04:50 smithi007 ceph-mon[39249]: from='client.? 172.21.15.7:0/429459757' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-18T08:04:50.440 INFO:teuthology.orchestra.run.smithi007.stdout:osd.5: {} 2024-09-18T08:04:50.440 INFO:teuthology.orchestra.run.smithi007.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-18T08:04:50.453 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1 2024-09-18T08:04:50.475 INFO:teuthology.orchestra.run.smithi007.stdout:osd.6: {} 2024-09-18T08:04:50.475 INFO:teuthology.orchestra.run.smithi007.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-18T08:04:50.495 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:04:50.496 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:04:50.496 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1' 2024-09-18T08:04:50.507 INFO:teuthology.orchestra.run.smithi007.stdout:osd.7: {} 2024-09-18T08:04:50.508 INFO:teuthology.orchestra.run.smithi007.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-18T08:04:50.582 INFO:tasks.radosbench.radosbench.0.smithi007.stdout: 45 16 73885 73869 102.561 70.8125 0.0116589 0.00974572 2024-09-18T08:04:50.598 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1 2024-09-18T08:04:50.637 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:04:50.641 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:04:50.642 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1' 2024-09-18T08:04:50.656 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:04:50 smithi050 ceph-mon[32371]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-18T08:04:50.656 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:04:50 smithi050 ceph-mon[32371]: pgmap v80: 148 pgs: 148 active+clean; 4.3 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 682 B/s rd, 105 MiB/s wr, 1.68k op/s 2024-09-18T08:04:50.656 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:04:50 smithi050 ceph-mon[32371]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 4 pgs inactive, 8 pgs peering) 2024-09-18T08:04:50.656 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:04:50 smithi050 ceph-mon[32371]: from='client.? 172.21.15.7:0/429459757' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-18T08:04:50.659 INFO:journalctl@ceph.alertmanager.a.smithi007.stdout:Sep 18 08:04:50 smithi007 podman[78368]: 2024-09-18 08:04:50.454229486 +0000 UTC m=+0.278257672 container create 352fb30367d166924c7334a172aabaddea0970e8e53e53e52c2f103935229846 (image=quay.io/prometheus/alertmanager:v0.27.0, name=ceph-779349ca-7593-11ef-bceb-c7b262605968-alertmanager-a, maintainer=The Prometheus Authors ) 2024-09-18T08:04:50.743 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1 2024-09-18T08:04:50.744 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-09-18T08:04:50.744 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-18T08:04:50.745 INFO:tasks.thrashosds.thrasher:inject_pause on osd.7 2024-09-18T08:04:50.745 INFO:tasks.thrashosds.thrasher:Testing filestore_inject_stall pause injection for duration 3 2024-09-18T08:04:50.745 INFO:tasks.thrashosds.thrasher:Checking after 0, should_be_down=False 2024-09-18T08:04:50.745 DEBUG:teuthology.orchestra.run.smithi050:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid 779349ca-7593-11ef-bceb-c7b262605968 -- ceph daemon osd.7 config set filestore_inject_stall 3 2024-09-18T08:04:50.781 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:04:50.782 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:04:50.783 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1' 2024-09-18T08:04:50.884 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1 2024-09-18T08:04:50.929 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:04:50.930 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:04:50.930 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1' 2024-09-18T08:04:51.031 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0 2024-09-18T08:04:51.071 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:04:51.072 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:04:51.073 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0' 2024-09-18T08:04:51.104 INFO:teuthology.orchestra.run.smithi050.stderr:Inferring config /var/lib/ceph/779349ca-7593-11ef-bceb-c7b262605968/mon.b/config 2024-09-18T08:04:51.174 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5 2024-09-18T08:04:51.203 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:04:51.204 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:04:51.204 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5' 2024-09-18T08:04:51.305 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6 2024-09-18T08:04:51.331 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:04:51.332 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:04:51.332 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6' 2024-09-18T08:04:51.433 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2 2024-09-18T08:04:51.480 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:04:51.481 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:04:51.481 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2' 2024-09-18T08:04:51.582 INFO:tasks.radosbench.radosbench.0.smithi007.stdout: 46 16 75091 75075 101.97 75.375 0.00626555 0.00980035 2024-09-18T08:04:51.582 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1 2024-09-18T08:04:51.622 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:04:51.623 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:04:51.623 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1' 2024-09-18T08:04:51.659 INFO:journalctl@ceph.alertmanager.a.smithi007.stdout:Sep 18 08:04:51 smithi007 podman[78368]: 2024-09-18 08:04:51.405503017 +0000 UTC m=+1.229531208 container init 352fb30367d166924c7334a172aabaddea0970e8e53e53e52c2f103935229846 (image=quay.io/prometheus/alertmanager:v0.27.0, name=ceph-779349ca-7593-11ef-bceb-c7b262605968-alertmanager-a, maintainer=The Prometheus Authors ) 2024-09-18T08:04:51.659 INFO:journalctl@ceph.alertmanager.a.smithi007.stdout:Sep 18 08:04:51 smithi007 podman[78368]: 2024-09-18 08:04:51.417913897 +0000 UTC m=+1.241942079 container start 352fb30367d166924c7334a172aabaddea0970e8e53e53e52c2f103935229846 (image=quay.io/prometheus/alertmanager:v0.27.0, name=ceph-779349ca-7593-11ef-bceb-c7b262605968-alertmanager-a, maintainer=The Prometheus Authors ) 2024-09-18T08:04:51.659 INFO:journalctl@ceph.alertmanager.a.smithi007.stdout:Sep 18 08:04:51 smithi007 ceph-779349ca-7593-11ef-bceb-c7b262605968-alertmanager-a[78486]: ts=2024-09-18T08:04:51.441Z caller=main.go:181 level=info msg="Starting Alertmanager" version="(version=0.27.0, branch=HEAD, revision=0aa3c2aad14cff039931923ab16b26b7481783b5)" 2024-09-18T08:04:51.660 INFO:journalctl@ceph.alertmanager.a.smithi007.stdout:Sep 18 08:04:51 smithi007 ceph-779349ca-7593-11ef-bceb-c7b262605968-alertmanager-a[78486]: ts=2024-09-18T08:04:51.441Z caller=main.go:182 level=info build_context="(go=go1.21.7, platform=linux/amd64, user=root@22cd11f671e9, date=20240228-11:51:20, tags=netgo)" 2024-09-18T08:04:51.660 INFO:journalctl@ceph.alertmanager.a.smithi007.stdout:Sep 18 08:04:51 smithi007 ceph-779349ca-7593-11ef-bceb-c7b262605968-alertmanager-a[78486]: ts=2024-09-18T08:04:51.445Z caller=cluster.go:186 level=info component=cluster msg="setting advertise address explicitly" addr=172.21.15.7 port=9094 2024-09-18T08:04:51.660 INFO:journalctl@ceph.alertmanager.a.smithi007.stdout:Sep 18 08:04:51 smithi007 ceph-779349ca-7593-11ef-bceb-c7b262605968-alertmanager-a[78486]: ts=2024-09-18T08:04:51.448Z caller=cluster.go:683 level=info component=cluster msg="Waiting for gossip to settle..." interval=2s 2024-09-18T08:04:51.660 INFO:journalctl@ceph.alertmanager.a.smithi007.stdout:Sep 18 08:04:51 smithi007 ceph-779349ca-7593-11ef-bceb-c7b262605968-alertmanager-a[78486]: ts=2024-09-18T08:04:51.496Z caller=coordinator.go:113 level=info component=configuration msg="Loading configuration file" file=/etc/alertmanager/alertmanager.yml 2024-09-18T08:04:51.660 INFO:journalctl@ceph.alertmanager.a.smithi007.stdout:Sep 18 08:04:51 smithi007 ceph-779349ca-7593-11ef-bceb-c7b262605968-alertmanager-a[78486]: ts=2024-09-18T08:04:51.497Z caller=coordinator.go:126 level=info component=configuration msg="Completed loading of configuration file" file=/etc/alertmanager/alertmanager.yml 2024-09-18T08:04:51.660 INFO:journalctl@ceph.alertmanager.a.smithi007.stdout:Sep 18 08:04:51 smithi007 ceph-779349ca-7593-11ef-bceb-c7b262605968-alertmanager-a[78486]: ts=2024-09-18T08:04:51.500Z caller=tls_config.go:313 level=info msg="Listening on" address=[::]:9093 2024-09-18T08:04:51.660 INFO:journalctl@ceph.alertmanager.a.smithi007.stdout:Sep 18 08:04:51 smithi007 ceph-779349ca-7593-11ef-bceb-c7b262605968-alertmanager-a[78486]: ts=2024-09-18T08:04:51.500Z caller=tls_config.go:316 level=info msg="TLS is disabled." http2=false address=[::]:9093 2024-09-18T08:04:51.660 INFO:journalctl@ceph.alertmanager.a.smithi007.stdout:Sep 18 08:04:51 smithi007 bash[78368]: 352fb30367d166924c7334a172aabaddea0970e8e53e53e52c2f103935229846 2024-09-18T08:04:51.660 INFO:journalctl@ceph.alertmanager.a.smithi007.stdout:Sep 18 08:04:51 smithi007 systemd[1]: Started Ceph alertmanager.a for 779349ca-7593-11ef-bceb-c7b262605968. 2024-09-18T08:04:51.704 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2024-09-18T08:04:51.704 INFO:teuthology.orchestra.run.smithi050.stdout: "success": "filestore_inject_stall = '' (not observed, change may require restart) " 2024-09-18T08:04:51.704 INFO:teuthology.orchestra.run.smithi050.stdout:} 2024-09-18T08:04:51.725 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2 2024-09-18T08:04:51.768 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:04:51.769 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:04:51.770 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2' 2024-09-18T08:04:51.872 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3 2024-09-18T08:04:51.900 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid 779349ca-7593-11ef-bceb-c7b262605968 -- ceph osd unset noscrub 2024-09-18T08:04:51.930 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:04:51.930 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:04:51.931 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3' 2024-09-18T08:04:52.031 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0 2024-09-18T08:04:52.072 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:04:52.076 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:04:52.076 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0' 2024-09-18T08:04:52.178 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1 2024-09-18T08:04:52.182 INFO:teuthology.orchestra.run.smithi007.stdout:ERROR: (22) Invalid argument 2024-09-18T08:04:52.183 INFO:teuthology.orchestra.run.smithi007.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-18T08:04:52.232 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:04:52.233 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:04:52.233 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1' 2024-09-18T08:04:52.334 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2 2024-09-18T08:04:52.378 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:04:52.379 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:04:52.379 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2' 2024-09-18T08:04:52.480 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1 2024-09-18T08:04:52.518 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:04:52.518 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:04:52.519 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1' 2024-09-18T08:04:52.582 INFO:tasks.radosbench.radosbench.0.smithi007.stdout: 47 16 76154 76138 101.214 66.4375 0.0160996 0.00987472 2024-09-18T08:04:52.620 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5 2024-09-18T08:04:52.649 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:04:52.649 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:04:52.650 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5' 2024-09-18T08:04:52.655 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:04:52 smithi050 ceph-mon[32371]: pgmap v81: 148 pgs: 148 active+clean; 4.3 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 656 B/s rd, 47 MiB/s wr, 755 op/s 2024-09-18T08:04:52.656 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:04:52 smithi050 ceph-mon[32371]: from='mgr.24470 172.21.15.7:0/2594390261' entity='mgr.y' 2024-09-18T08:04:52.656 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:04:52 smithi050 ceph-mon[32371]: from='mgr.24470 172.21.15.7:0/2594390261' entity='mgr.y' 2024-09-18T08:04:52.730 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:04:52 smithi007 ceph-mon[39249]: pgmap v81: 148 pgs: 148 active+clean; 4.3 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 656 B/s rd, 47 MiB/s wr, 755 op/s 2024-09-18T08:04:52.731 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:04:52 smithi007 ceph-mon[39249]: from='mgr.24470 172.21.15.7:0/2594390261' entity='mgr.y' 2024-09-18T08:04:52.731 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:04:52 smithi007 ceph-mon[39249]: from='mgr.24470 172.21.15.7:0/2594390261' entity='mgr.y' 2024-09-18T08:04:52.731 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:04:52 smithi007 ceph-mon[29459]: pgmap v81: 148 pgs: 148 active+clean; 4.3 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 656 B/s rd, 47 MiB/s wr, 755 op/s 2024-09-18T08:04:52.731 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:04:52 smithi007 ceph-mon[29459]: from='mgr.24470 172.21.15.7:0/2594390261' entity='mgr.y' 2024-09-18T08:04:52.731 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:04:52 smithi007 ceph-mon[29459]: from='mgr.24470 172.21.15.7:0/2594390261' entity='mgr.y' 2024-09-18T08:04:52.750 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2 2024-09-18T08:04:52.791 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:04:52.791 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:04:52.792 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2' 2024-09-18T08:04:52.892 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3 2024-09-18T08:04:52.941 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:04:52.942 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:04:52.942 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3' 2024-09-18T08:04:53.043 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4 2024-09-18T08:04:53.049 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid 779349ca-7593-11ef-bceb-c7b262605968 -- ceph daemon osd.3 dump_historic_ops 2024-09-18T08:04:53.069 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:04:53.069 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:04:53.070 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4' 2024-09-18T08:04:53.140 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/779349ca-7593-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T08:04:53.170 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1 2024-09-18T08:04:53.237 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:04:53.237 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:04:53.238 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1' 2024-09-18T08:04:53.338 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3 2024-09-18T08:04:53.351 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid 779349ca-7593-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-18T08:04:53.405 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:04:53 smithi050 ceph-mon[32371]: Reconfiguring prometheus.a (dependencies changed)... 2024-09-18T08:04:53.406 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:04:53 smithi050 ceph-mon[32371]: Reconfiguring daemon prometheus.a on smithi050 2024-09-18T08:04:53.415 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:04:53.416 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:04:53.416 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3' 2024-09-18T08:04:53.517 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4 2024-09-18T08:04:53.544 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:04:53.545 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:04:53.545 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4' 2024-09-18T08:04:53.582 INFO:tasks.radosbench.radosbench.0.smithi007.stdout: 48 16 77253 77237 100.537 68.6875 0.0246446 0.00994172 2024-09-18T08:04:53.646 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2 2024-09-18T08:04:53.709 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:04:53.710 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:04:53.710 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2' 2024-09-18T08:04:53.710 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:04:53 smithi007 ceph-mon[29459]: Reconfiguring prometheus.a (dependencies changed)... 2024-09-18T08:04:53.711 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:04:53 smithi007 ceph-mon[29459]: Reconfiguring daemon prometheus.a on smithi050 2024-09-18T08:04:53.711 INFO:journalctl@ceph.alertmanager.a.smithi007.stdout:Sep 18 08:04:53 smithi007 ceph-779349ca-7593-11ef-bceb-c7b262605968-alertmanager-a[78486]: ts=2024-09-18T08:04:53.449Z caller=cluster.go:708 level=info component=cluster msg="gossip not settled" polls=0 before=0 now=1 elapsed=2.000909037s 2024-09-18T08:04:53.711 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:04:53 smithi007 ceph-mon[39249]: Reconfiguring prometheus.a (dependencies changed)... 2024-09-18T08:04:53.711 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:04:53 smithi007 ceph-mon[39249]: Reconfiguring daemon prometheus.a on smithi050 2024-09-18T08:04:53.811 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4 2024-09-18T08:04:53.843 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:04:53.844 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:04:53.845 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4' 2024-09-18T08:04:53.945 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6 2024-09-18T08:04:53.975 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:04:53.976 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:04:53.976 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6' 2024-09-18T08:04:54.077 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5 2024-09-18T08:04:54.100 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/779349ca-7593-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T08:04:54.104 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:04:54.104 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:04:54.105 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5' 2024-09-18T08:04:54.132 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/779349ca-7593-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T08:04:54.205 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1 2024-09-18T08:04:54.253 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:04:54.254 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:04:54.254 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1' 2024-09-18T08:04:54.355 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2 2024-09-18T08:04:54.404 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:04:54.405 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:04:54.405 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2' 2024-09-18T08:04:54.506 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3 2024-09-18T08:04:54.509 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:04:54 smithi007 ceph-mon[29459]: pgmap v82: 148 pgs: 148 active+clean; 4.4 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 1.2 KiB/s rd, 54 MiB/s wr, 870 op/s 2024-09-18T08:04:54.509 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:04:54 smithi007 ceph-mon[39249]: pgmap v82: 148 pgs: 148 active+clean; 4.4 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 1.2 KiB/s rd, 54 MiB/s wr, 870 op/s 2024-09-18T08:04:54.542 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:04:54.543 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:04:54.543 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3' 2024-09-18T08:04:54.582 INFO:tasks.radosbench.radosbench.0.smithi007.stdout: 49 16 78262 78246 99.772 63.0625 0.0159331 0.0100177 2024-09-18T08:04:54.606 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:04:54 smithi050 ceph-mon[32371]: pgmap v82: 148 pgs: 148 active+clean; 4.4 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 1.2 KiB/s rd, 54 MiB/s wr, 870 op/s 2024-09-18T08:04:54.644 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1 2024-09-18T08:04:54.689 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:04:54.690 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:04:54.690 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1' 2024-09-18T08:04:54.791 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2 2024-09-18T08:04:54.821 INFO:teuthology.orchestra.run.smithi007.stdout:osd.0: {} 2024-09-18T08:04:54.821 INFO:teuthology.orchestra.run.smithi007.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-18T08:04:54.838 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:04:54.839 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:04:54.839 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2' 2024-09-18T08:04:54.859 INFO:teuthology.orchestra.run.smithi007.stdout:osd.1: {} 2024-09-18T08:04:54.859 INFO:teuthology.orchestra.run.smithi007.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-18T08:04:54.896 INFO:teuthology.orchestra.run.smithi007.stdout:osd.2: {} 2024-09-18T08:04:54.897 INFO:teuthology.orchestra.run.smithi007.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-18T08:04:54.931 INFO:teuthology.orchestra.run.smithi007.stdout:osd.3: {} 2024-09-18T08:04:54.931 INFO:teuthology.orchestra.run.smithi007.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-18T08:04:54.940 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0 2024-09-18T08:04:54.968 INFO:teuthology.orchestra.run.smithi007.stdout:osd.4: {} 2024-09-18T08:04:54.968 INFO:teuthology.orchestra.run.smithi007.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-18T08:04:54.983 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:04:54.988 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:04:54.989 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0' 2024-09-18T08:04:55.005 INFO:teuthology.orchestra.run.smithi007.stdout:osd.5: {} 2024-09-18T08:04:55.005 INFO:teuthology.orchestra.run.smithi007.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-18T08:04:55.039 INFO:teuthology.orchestra.run.smithi007.stdout:osd.6: {} 2024-09-18T08:04:55.040 INFO:teuthology.orchestra.run.smithi007.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-18T08:04:55.072 INFO:teuthology.orchestra.run.smithi007.stdout:osd.7: {} 2024-09-18T08:04:55.072 INFO:teuthology.orchestra.run.smithi007.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-18T08:04:55.090 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0 2024-09-18T08:04:55.133 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:04:55.134 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:04:55.134 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0' 2024-09-18T08:04:55.235 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5 2024-09-18T08:04:55.263 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:04:55.263 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:04:55.264 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5' 2024-09-18T08:04:55.361 INFO:teuthology.orchestra.run.smithi007.stderr:noscrub is unset 2024-09-18T08:04:55.364 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4 2024-09-18T08:04:55.391 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:04:55.392 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:04:55.392 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4' 2024-09-18T08:04:55.406 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:04:55 smithi050 ceph-mon[32371]: from='client.? 172.21.15.7:0/2150477428' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-18T08:04:55.406 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:04:55 smithi050 ceph-mon[32371]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-18T08:04:55.406 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:04:55 smithi050 ceph-mon[32371]: from='client.? 172.21.15.7:0/2432438528' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-18T08:04:55.493 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4 2024-09-18T08:04:55.522 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:04:55.523 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:04:55.523 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4' 2024-09-18T08:04:55.582 INFO:tasks.radosbench.radosbench.0.smithi007.stdout: 50 16 79357 79341 99.1453 68.4375 0.0201083 0.0100807 2024-09-18T08:04:55.624 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6 2024-09-18T08:04:55.636 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:04:55 smithi007 ceph-mon[39249]: from='client.? 172.21.15.7:0/2150477428' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-18T08:04:55.636 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:04:55 smithi007 ceph-mon[39249]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-18T08:04:55.636 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:04:55 smithi007 ceph-mon[39249]: from='client.? 172.21.15.7:0/2432438528' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-18T08:04:55.636 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:04:55 smithi007 ceph-mon[29459]: from='client.? 172.21.15.7:0/2150477428' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-18T08:04:55.636 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:04:55 smithi007 ceph-mon[29459]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-18T08:04:55.636 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:04:55 smithi007 ceph-mon[29459]: from='client.? 172.21.15.7:0/2432438528' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-18T08:04:55.653 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:04:55.654 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:04:55.654 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6' 2024-09-18T08:04:55.755 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4 2024-09-18T08:04:55.783 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:04:55.784 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:04:55.785 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4' 2024-09-18T08:04:55.885 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5 2024-09-18T08:04:55.915 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:04:55.916 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:04:55.916 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5' 2024-09-18T08:04:56.017 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0 2024-09-18T08:04:56.058 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:04:56.058 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:04:56.059 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0' 2024-09-18T08:04:56.160 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5 2024-09-18T08:04:56.187 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:04:56.188 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:04:56.188 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5' 2024-09-18T08:04:56.289 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4 2024-09-18T08:04:56.323 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:04:56.324 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:04:56.324 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4' 2024-09-18T08:04:56.425 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3 2024-09-18T08:04:56.462 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:04:56.463 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:04:56.464 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3' 2024-09-18T08:04:56.565 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6 2024-09-18T08:04:56.582 INFO:tasks.radosbench.radosbench.0.smithi007.stdout: 51 16 80520 80504 98.6265 72.6875 0.00647403 0.0101339 2024-09-18T08:04:56.592 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:04:56 smithi050 ceph-mon[32371]: pgmap v83: 148 pgs: 148 active+clean; 4.7 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 1.2 KiB/s rd, 91 MiB/s wr, 1.45k op/s 2024-09-18T08:04:56.592 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:04:56 smithi050 ceph-mon[32371]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-18T08:04:56.593 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:04:56 smithi050 ceph-mon[32371]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-18T08:04:56.593 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:04:56 smithi050 ceph-mon[32371]: osdmap e105: 8 total, 8 up, 8 in 2024-09-18T08:04:56.593 INFO:journalctl@ceph.prometheus.a.smithi050.stdout:Sep 18 08:04:56 smithi050 systemd[1]: Stopping Ceph prometheus.a for 779349ca-7593-11ef-bceb-c7b262605968... 2024-09-18T08:04:56.598 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:04:56.598 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:04:56.599 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6' 2024-09-18T08:04:56.617 INFO:teuthology.orchestra.run.smithi007.stdout:{ 2024-09-18T08:04:56.617 INFO:teuthology.orchestra.run.smithi007.stdout: "size": 20, 2024-09-18T08:04:56.617 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 600, 2024-09-18T08:04:56.617 INFO:teuthology.orchestra.run.smithi007.stdout: "ops": [ 2024-09-18T08:04:56.617 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:04:56.618 INFO:teuthology.orchestra.run.smithi007.stdout: "description": "osd_op(client.14805.0:39800 7.3 7:c4a88bd1:::benchmark_data_smithi007_73155_object39799:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e95)", 2024-09-18T08:04:56.618 INFO:teuthology.orchestra.run.smithi007.stdout: "initiated_at": "2024-09-18T08:04:24.060835+0000", 2024-09-18T08:04:56.618 INFO:teuthology.orchestra.run.smithi007.stdout: "age": 31.571255399000002, 2024-09-18T08:04:56.618 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.056851758000000002, 2024-09-18T08:04:56.618 INFO:teuthology.orchestra.run.smithi007.stdout: "continuous": false, 2024-09-18T08:04:56.618 INFO:teuthology.orchestra.run.smithi007.stdout: "type_data": { 2024-09-18T08:04:56.619 INFO:teuthology.orchestra.run.smithi007.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T08:04:56.619 INFO:teuthology.orchestra.run.smithi007.stdout: "client_info": { 2024-09-18T08:04:56.619 INFO:teuthology.orchestra.run.smithi007.stdout: "client": "client.14805", 2024-09-18T08:04:56.619 INFO:teuthology.orchestra.run.smithi007.stdout: "client_addr": "172.21.15.7:0/640320436", 2024-09-18T08:04:56.619 INFO:teuthology.orchestra.run.smithi007.stdout: "tid": 39800 2024-09-18T08:04:56.619 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:04:56.619 INFO:teuthology.orchestra.run.smithi007.stdout: "events": [ 2024-09-18T08:04:56.619 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:04:56.619 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "initiated", 2024-09-18T08:04:56.619 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:04:24.060835+0000", 2024-09-18T08:04:56.619 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-09-18T08:04:56.619 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:04:56.620 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:04:56.620 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "header_read", 2024-09-18T08:04:56.620 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:04:24.060835+0000", 2024-09-18T08:04:56.620 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-09-18T08:04:56.620 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:04:56.620 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:04:56.620 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "throttled", 2024-09-18T08:04:56.620 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:04:24.060836+0000", 2024-09-18T08:04:56.620 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 5.2200000000000004e-07 2024-09-18T08:04:56.620 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:04:56.620 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:04:56.620 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "all_read", 2024-09-18T08:04:56.620 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:04:24.060868+0000", 2024-09-18T08:04:56.621 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 3.1736999999999998e-05 2024-09-18T08:04:56.621 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:04:56.621 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:04:56.621 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "dispatched", 2024-09-18T08:04:56.621 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:04:24.060875+0000", 2024-09-18T08:04:56.621 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 7.9000000000000006e-06 2024-09-18T08:04:56.621 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:04:56.621 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:04:56.621 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "queued_for_pg", 2024-09-18T08:04:56.621 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:04:24.060881+0000", 2024-09-18T08:04:56.621 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 5.7139999999999998e-06 2024-09-18T08:04:56.621 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:04:56.621 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:04:56.621 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-09-18T08:04:56.622 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:04:24.065025+0000", 2024-09-18T08:04:56.622 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.0041435619999999999 2024-09-18T08:04:56.622 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:04:56.622 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:04:56.622 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "started", 2024-09-18T08:04:56.622 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:04:24.065139+0000", 2024-09-18T08:04:56.622 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.000114731 2024-09-18T08:04:56.622 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:04:56.622 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:04:56.622 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "waiting for subops from 2,4", 2024-09-18T08:04:56.622 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:04:24.065277+0000", 2024-09-18T08:04:56.622 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.000137448 2024-09-18T08:04:56.622 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:04:56.622 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:04:56.622 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "op_commit", 2024-09-18T08:04:56.623 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:04:24.076095+0000", 2024-09-18T08:04:56.623 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.010817970999999999 2024-09-18T08:04:56.623 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:04:56.623 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:04:56.623 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:04:56.623 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:04:24.076886+0000", 2024-09-18T08:04:56.623 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.00079133299999999997 2024-09-18T08:04:56.623 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:04:56.623 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:04:56.624 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:04:56.624 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:04:24.117588+0000", 2024-09-18T08:04:56.624 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.040702019999999998 2024-09-18T08:04:56.624 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:04:56.624 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:04:56.624 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "commit_sent", 2024-09-18T08:04:56.624 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:04:24.117644+0000", 2024-09-18T08:04:56.624 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 5.5760999999999997e-05 2024-09-18T08:04:56.624 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:04:56.624 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:04:56.624 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "done", 2024-09-18T08:04:56.624 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:04:24.117687+0000", 2024-09-18T08:04:56.624 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 4.3059000000000002e-05 2024-09-18T08:04:56.625 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-09-18T08:04:56.625 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-09-18T08:04:56.625 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-09-18T08:04:56.625 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:04:56.625 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:04:56.625 INFO:teuthology.orchestra.run.smithi007.stdout: "description": "osd_op(client.14805.0:39806 7.3 7:cc701fdd:::benchmark_data_smithi007_73155_object39805:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e95)", 2024-09-18T08:04:56.625 INFO:teuthology.orchestra.run.smithi007.stdout: "initiated_at": "2024-09-18T08:04:24.063056+0000", 2024-09-18T08:04:56.625 INFO:teuthology.orchestra.run.smithi007.stdout: "age": 31.569035105000001, 2024-09-18T08:04:56.625 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.055528051000000002, 2024-09-18T08:04:56.625 INFO:teuthology.orchestra.run.smithi007.stdout: "continuous": false, 2024-09-18T08:04:56.625 INFO:teuthology.orchestra.run.smithi007.stdout: "type_data": { 2024-09-18T08:04:56.625 INFO:teuthology.orchestra.run.smithi007.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T08:04:56.625 INFO:teuthology.orchestra.run.smithi007.stdout: "client_info": { 2024-09-18T08:04:56.626 INFO:teuthology.orchestra.run.smithi007.stdout: "client": "client.14805", 2024-09-18T08:04:56.626 INFO:teuthology.orchestra.run.smithi007.stdout: "client_addr": "172.21.15.7:0/640320436", 2024-09-18T08:04:56.626 INFO:teuthology.orchestra.run.smithi007.stdout: "tid": 39806 2024-09-18T08:04:56.626 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:04:56.626 INFO:teuthology.orchestra.run.smithi007.stdout: "events": [ 2024-09-18T08:04:56.626 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:04:56.626 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "initiated", 2024-09-18T08:04:56.626 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:04:24.063056+0000", 2024-09-18T08:04:56.626 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-09-18T08:04:56.626 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:04:56.626 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:04:56.626 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "header_read", 2024-09-18T08:04:56.626 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:04:24.063056+0000", 2024-09-18T08:04:56.626 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-09-18T08:04:56.627 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:04:56.627 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:04:56.627 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "throttled", 2024-09-18T08:04:56.627 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:04:24.063057+0000", 2024-09-18T08:04:56.627 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.7689999999999999e-06 2024-09-18T08:04:56.627 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:04:56.627 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:04:56.627 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "all_read", 2024-09-18T08:04:56.627 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:04:24.063087+0000", 2024-09-18T08:04:56.627 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 2.9617000000000001e-05 2024-09-18T08:04:56.627 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:04:56.627 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:04:56.627 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "dispatched", 2024-09-18T08:04:56.628 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:04:24.063102+0000", 2024-09-18T08:04:56.628 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.4649e-05 2024-09-18T08:04:56.628 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:04:56.629 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:04:56.629 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "queued_for_pg", 2024-09-18T08:04:56.629 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:04:24.063110+0000", 2024-09-18T08:04:56.629 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 8.6549999999999993e-06 2024-09-18T08:04:56.629 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:04:56.629 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:04:56.629 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-09-18T08:04:56.629 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:04:24.072438+0000", 2024-09-18T08:04:56.630 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.009327541 2024-09-18T08:04:56.630 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:04:56.630 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:04:56.630 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "started", 2024-09-18T08:04:56.630 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:04:24.072569+0000", 2024-09-18T08:04:56.630 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.00013085700000000001 2024-09-18T08:04:56.630 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:04:56.630 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:04:56.630 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "waiting for subops from 2,4", 2024-09-18T08:04:56.630 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:04:24.072717+0000", 2024-09-18T08:04:56.630 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.000148546 2024-09-18T08:04:56.631 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:04:56.631 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:04:56.631 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "op_commit", 2024-09-18T08:04:56.631 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:04:24.076996+0000", 2024-09-18T08:04:56.631 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.0042791920000000002 2024-09-18T08:04:56.631 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:04:56.631 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:04:56.631 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:04:56.631 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:04:24.077716+0000", 2024-09-18T08:04:56.631 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.00071948900000000002 2024-09-18T08:04:56.631 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:04:56.631 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:04:56.631 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:04:56.631 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:04:24.118489+0000", 2024-09-18T08:04:56.632 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.040772663000000001 2024-09-18T08:04:56.632 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:04:56.632 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:04:56.632 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "commit_sent", 2024-09-18T08:04:56.632 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:04:24.118544+0000", 2024-09-18T08:04:56.632 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 5.5825000000000002e-05 2024-09-18T08:04:56.632 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:04:56.632 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:04:56.632 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "done", 2024-09-18T08:04:56.633 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:04:24.118584+0000", 2024-09-18T08:04:56.633 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 3.9248000000000001e-05 2024-09-18T08:04:56.633 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-09-18T08:04:56.633 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-09-18T08:04:56.633 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-09-18T08:04:56.633 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:04:56.633 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:04:56.633 INFO:teuthology.orchestra.run.smithi007.stdout: "description": "osd_op(client.14805.0:63781 7.2 7:4105de0f:::benchmark_data_smithi007_73155_object63780:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e100)", 2024-09-18T08:04:56.633 INFO:teuthology.orchestra.run.smithi007.stdout: "initiated_at": "2024-09-18T08:04:41.022122+0000", 2024-09-18T08:04:56.633 INFO:teuthology.orchestra.run.smithi007.stdout: "age": 14.6099684, 2024-09-18T08:04:56.633 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.042708536999999998, 2024-09-18T08:04:56.633 INFO:teuthology.orchestra.run.smithi007.stdout: "continuous": false, 2024-09-18T08:04:56.633 INFO:teuthology.orchestra.run.smithi007.stdout: "type_data": { 2024-09-18T08:04:56.633 INFO:teuthology.orchestra.run.smithi007.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T08:04:56.634 INFO:teuthology.orchestra.run.smithi007.stdout: "client_info": { 2024-09-18T08:04:56.634 INFO:teuthology.orchestra.run.smithi007.stdout: "client": "client.14805", 2024-09-18T08:04:56.634 INFO:teuthology.orchestra.run.smithi007.stdout: "client_addr": "172.21.15.7:0/640320436", 2024-09-18T08:04:56.634 INFO:teuthology.orchestra.run.smithi007.stdout: "tid": 63781 2024-09-18T08:04:56.634 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:04:56.634 INFO:teuthology.orchestra.run.smithi007.stdout: "events": [ 2024-09-18T08:04:56.634 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:04:56.634 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "initiated", 2024-09-18T08:04:56.634 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:04:41.022122+0000", 2024-09-18T08:04:56.634 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-09-18T08:04:56.634 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:04:56.634 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:04:56.634 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "header_read", 2024-09-18T08:04:56.635 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:04:41.022122+0000", 2024-09-18T08:04:56.635 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-09-18T08:04:56.635 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:04:56.635 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:04:56.635 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "throttled", 2024-09-18T08:04:56.635 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:04:41.022124+0000", 2024-09-18T08:04:56.635 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.3659999999999999e-06 2024-09-18T08:04:56.635 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:04:56.635 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:04:56.635 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "all_read", 2024-09-18T08:04:56.635 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:04:41.022149+0000", 2024-09-18T08:04:56.635 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 2.4984e-05 2024-09-18T08:04:56.635 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:04:56.635 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:04:56.635 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "dispatched", 2024-09-18T08:04:56.635 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:04:41.022161+0000", 2024-09-18T08:04:56.636 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.2139e-05 2024-09-18T08:04:56.636 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:04:56.636 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:04:56.636 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "queued_for_pg", 2024-09-18T08:04:56.636 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:04:41.022168+0000", 2024-09-18T08:04:56.636 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 7.4560000000000003e-06 2024-09-18T08:04:56.636 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:04:56.636 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:04:56.636 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-09-18T08:04:56.636 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:04:41.022365+0000", 2024-09-18T08:04:56.637 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.000196458 2024-09-18T08:04:56.637 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:04:56.637 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:04:56.637 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "started", 2024-09-18T08:04:56.637 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:04:41.022496+0000", 2024-09-18T08:04:56.637 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.00013134799999999999 2024-09-18T08:04:56.637 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:04:56.637 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:04:56.637 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "waiting for subops from 1,2", 2024-09-18T08:04:56.637 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:04:41.022638+0000", 2024-09-18T08:04:56.638 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.00014222200000000001 2024-09-18T08:04:56.638 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:04:56.638 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:04:56.638 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "op_commit", 2024-09-18T08:04:56.638 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:04:41.026801+0000", 2024-09-18T08:04:56.638 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.0041631970000000004 2024-09-18T08:04:56.638 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:04:56.638 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:04:56.638 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:04:56.639 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:04:41.034673+0000", 2024-09-18T08:04:56.639 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.0078714340000000001 2024-09-18T08:04:56.639 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:04:56.639 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:04:56.639 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:04:56.639 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:04:41.056261+0000", 2024-09-18T08:04:56.639 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.021587789 2024-09-18T08:04:56.639 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:04:56.640 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:04:56.640 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "commit_sent", 2024-09-18T08:04:56.640 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:04:41.064785+0000", 2024-09-18T08:04:56.640 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.0085238120000000004 2024-09-18T08:04:56.640 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:04:56.640 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:04:56.640 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "done", 2024-09-18T08:04:56.640 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:04:41.064831+0000", 2024-09-18T08:04:56.640 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 4.6332000000000003e-05 2024-09-18T08:04:56.640 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-09-18T08:04:56.640 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-09-18T08:04:56.641 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-09-18T08:04:56.641 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:04:56.641 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:04:56.641 INFO:teuthology.orchestra.run.smithi007.stdout: "description": "osd_op(client.14805.0:63943 7.2 7:46835086:::benchmark_data_smithi007_73155_object63942:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e100)", 2024-09-18T08:04:56.641 INFO:teuthology.orchestra.run.smithi007.stdout: "initiated_at": "2024-09-18T08:04:41.235152+0000", 2024-09-18T08:04:56.641 INFO:teuthology.orchestra.run.smithi007.stdout: "age": 14.39693913, 2024-09-18T08:04:56.641 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.044555504000000003, 2024-09-18T08:04:56.641 INFO:teuthology.orchestra.run.smithi007.stdout: "continuous": false, 2024-09-18T08:04:56.641 INFO:teuthology.orchestra.run.smithi007.stdout: "type_data": { 2024-09-18T08:04:56.641 INFO:teuthology.orchestra.run.smithi007.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T08:04:56.641 INFO:teuthology.orchestra.run.smithi007.stdout: "client_info": { 2024-09-18T08:04:56.641 INFO:teuthology.orchestra.run.smithi007.stdout: "client": "client.14805", 2024-09-18T08:04:56.641 INFO:teuthology.orchestra.run.smithi007.stdout: "client_addr": "172.21.15.7:0/640320436", 2024-09-18T08:04:56.642 INFO:teuthology.orchestra.run.smithi007.stdout: "tid": 63943 2024-09-18T08:04:56.642 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:04:56.642 INFO:teuthology.orchestra.run.smithi007.stdout: "events": [ 2024-09-18T08:04:56.642 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:04:56.642 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "initiated", 2024-09-18T08:04:56.642 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:04:41.235152+0000", 2024-09-18T08:04:56.642 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-09-18T08:04:56.642 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:04:56.642 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:04:56.642 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "header_read", 2024-09-18T08:04:56.642 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:04:41.235152+0000", 2024-09-18T08:04:56.642 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-09-18T08:04:56.642 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:04:56.642 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:04:56.643 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "throttled", 2024-09-18T08:04:56.643 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:04:41.235154+0000", 2024-09-18T08:04:56.643 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.9829999999999999e-06 2024-09-18T08:04:56.643 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:04:56.643 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:04:56.643 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "all_read", 2024-09-18T08:04:56.643 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:04:41.235183+0000", 2024-09-18T08:04:56.643 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 2.9005000000000001e-05 2024-09-18T08:04:56.643 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:04:56.643 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:04:56.643 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "dispatched", 2024-09-18T08:04:56.643 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:04:41.235194+0000", 2024-09-18T08:04:56.643 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.1875000000000001e-05 2024-09-18T08:04:56.643 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:04:56.643 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:04:56.643 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "queued_for_pg", 2024-09-18T08:04:56.644 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:04:41.235202+0000", 2024-09-18T08:04:56.644 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 7.9999999999999996e-06 2024-09-18T08:04:56.644 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:04:56.644 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:04:56.644 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-09-18T08:04:56.644 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:04:41.236317+0000", 2024-09-18T08:04:56.644 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.001114128 2024-09-18T08:04:56.644 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:04:56.644 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:04:56.644 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "started", 2024-09-18T08:04:56.644 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:04:41.236449+0000", 2024-09-18T08:04:56.644 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.00013244299999999999 2024-09-18T08:04:56.644 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:04:56.644 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:04:56.645 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "waiting for subops from 1,2", 2024-09-18T08:04:56.645 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:04:41.236586+0000", 2024-09-18T08:04:56.645 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.00013742800000000001 2024-09-18T08:04:56.645 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:04:56.645 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:04:56.645 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "op_commit", 2024-09-18T08:04:56.645 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:04:41.245403+0000", 2024-09-18T08:04:56.645 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.0088161149999999994 2024-09-18T08:04:56.645 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:04:56.646 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:04:56.646 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:04:56.646 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:04:41.263785+0000", 2024-09-18T08:04:56.646 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.018381959 2024-09-18T08:04:56.646 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:04:56.646 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:04:56.646 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:04:56.646 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:04:41.265963+0000", 2024-09-18T08:04:56.646 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.0021786000000000002 2024-09-18T08:04:56.646 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:04:56.647 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:04:56.647 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "commit_sent", 2024-09-18T08:04:56.647 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:04:41.279508+0000", 2024-09-18T08:04:56.647 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.013545128 2024-09-18T08:04:56.647 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:04:56.647 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:04:56.647 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "done", 2024-09-18T08:04:56.647 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:04:41.279707+0000", 2024-09-18T08:04:56.647 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.00019883999999999999 2024-09-18T08:04:56.647 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-09-18T08:04:56.647 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-09-18T08:04:56.647 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-09-18T08:04:56.647 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:04:56.647 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:04:56.648 INFO:teuthology.orchestra.run.smithi007.stdout: "description": "osd_op(client.14805.0:63948 7.a 7:5ccd60b2:::benchmark_data_smithi007_73155_object63947:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e100)", 2024-09-18T08:04:56.648 INFO:teuthology.orchestra.run.smithi007.stdout: "initiated_at": "2024-09-18T08:04:41.239122+0000", 2024-09-18T08:04:56.648 INFO:teuthology.orchestra.run.smithi007.stdout: "age": 14.392968487999999, 2024-09-18T08:04:56.648 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.052602836, 2024-09-18T08:04:56.648 INFO:teuthology.orchestra.run.smithi007.stdout: "continuous": false, 2024-09-18T08:04:56.648 INFO:teuthology.orchestra.run.smithi007.stdout: "type_data": { 2024-09-18T08:04:56.648 INFO:teuthology.orchestra.run.smithi007.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T08:04:56.648 INFO:teuthology.orchestra.run.smithi007.stdout: "client_info": { 2024-09-18T08:04:56.648 INFO:teuthology.orchestra.run.smithi007.stdout: "client": "client.14805", 2024-09-18T08:04:56.648 INFO:teuthology.orchestra.run.smithi007.stdout: "client_addr": "172.21.15.7:0/640320436", 2024-09-18T08:04:56.648 INFO:teuthology.orchestra.run.smithi007.stdout: "tid": 63948 2024-09-18T08:04:56.648 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:04:56.648 INFO:teuthology.orchestra.run.smithi007.stdout: "events": [ 2024-09-18T08:04:56.648 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:04:56.649 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "initiated", 2024-09-18T08:04:56.649 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:04:41.239122+0000", 2024-09-18T08:04:56.649 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-09-18T08:04:56.649 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:04:56.649 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:04:56.649 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "header_read", 2024-09-18T08:04:56.649 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:04:41.239122+0000", 2024-09-18T08:04:56.649 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-09-18T08:04:56.649 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:04:56.649 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:04:56.650 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "throttled", 2024-09-18T08:04:56.650 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:04:41.239124+0000", 2024-09-18T08:04:56.650 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.4810000000000001e-06 2024-09-18T08:04:56.650 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:04:56.650 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:04:56.650 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "all_read", 2024-09-18T08:04:56.650 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:04:41.239147+0000", 2024-09-18T08:04:56.650 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 2.3053000000000001e-05 2024-09-18T08:04:56.650 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:04:56.651 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:04:56.651 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "dispatched", 2024-09-18T08:04:56.651 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:04:41.239162+0000", 2024-09-18T08:04:56.651 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.5308999999999999e-05 2024-09-18T08:04:56.651 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:04:56.651 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:04:56.651 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "queued_for_pg", 2024-09-18T08:04:56.651 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:04:41.239170+0000", 2024-09-18T08:04:56.651 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 8.0430000000000008e-06 2024-09-18T08:04:56.651 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:04:56.651 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:04:56.651 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-09-18T08:04:56.651 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:04:41.239355+0000", 2024-09-18T08:04:56.651 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.00018490600000000001 2024-09-18T08:04:56.652 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:04:56.652 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:04:56.652 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "started", 2024-09-18T08:04:56.652 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:04:41.240057+0000", 2024-09-18T08:04:56.652 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.000702376 2024-09-18T08:04:56.652 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:04:56.652 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:04:56.652 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "waiting for subops from 2,5", 2024-09-18T08:04:56.652 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:04:41.240196+0000", 2024-09-18T08:04:56.652 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.000138448 2024-09-18T08:04:56.652 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:04:56.652 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:04:56.652 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:04:56.652 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:04:41.250987+0000", 2024-09-18T08:04:56.653 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.010791479 2024-09-18T08:04:56.653 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:04:56.653 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:04:56.653 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "op_commit", 2024-09-18T08:04:56.653 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:04:41.266254+0000", 2024-09-18T08:04:56.653 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.015266873 2024-09-18T08:04:56.653 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:04:56.653 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:04:56.653 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:04:56.653 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:04:41.291512+0000", 2024-09-18T08:04:56.653 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.025257617999999999 2024-09-18T08:04:56.653 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:04:56.653 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:04:56.653 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "commit_sent", 2024-09-18T08:04:56.654 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:04:41.291677+0000", 2024-09-18T08:04:56.654 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.000164723 2024-09-18T08:04:56.654 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:04:56.654 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:04:56.654 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "done", 2024-09-18T08:04:56.654 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:04:41.291725+0000", 2024-09-18T08:04:56.654 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 4.8526999999999998e-05 2024-09-18T08:04:56.654 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-09-18T08:04:56.654 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-09-18T08:04:56.654 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-09-18T08:04:56.654 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:04:56.654 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:04:56.654 INFO:teuthology.orchestra.run.smithi007.stdout: "description": "osd_op(client.14805.0:63950 7.a 7:5df2c2fd:::benchmark_data_smithi007_73155_object63949:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e100)", 2024-09-18T08:04:56.654 INFO:teuthology.orchestra.run.smithi007.stdout: "initiated_at": "2024-09-18T08:04:41.251868+0000", 2024-09-18T08:04:56.654 INFO:teuthology.orchestra.run.smithi007.stdout: "age": 14.380222448, 2024-09-18T08:04:56.654 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.050368929999999999, 2024-09-18T08:04:56.655 INFO:teuthology.orchestra.run.smithi007.stdout: "continuous": false, 2024-09-18T08:04:56.655 INFO:teuthology.orchestra.run.smithi007.stdout: "type_data": { 2024-09-18T08:04:56.655 INFO:teuthology.orchestra.run.smithi007.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T08:04:56.655 INFO:teuthology.orchestra.run.smithi007.stdout: "client_info": { 2024-09-18T08:04:56.655 INFO:teuthology.orchestra.run.smithi007.stdout: "client": "client.14805", 2024-09-18T08:04:56.655 INFO:teuthology.orchestra.run.smithi007.stdout: "client_addr": "172.21.15.7:0/640320436", 2024-09-18T08:04:56.655 INFO:teuthology.orchestra.run.smithi007.stdout: "tid": 63950 2024-09-18T08:04:56.655 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:04:56.655 INFO:teuthology.orchestra.run.smithi007.stdout: "events": [ 2024-09-18T08:04:56.655 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:04:56.655 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "initiated", 2024-09-18T08:04:56.655 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:04:41.251868+0000", 2024-09-18T08:04:56.655 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-09-18T08:04:56.655 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:04:56.655 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:04:56.656 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "header_read", 2024-09-18T08:04:56.656 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:04:41.251868+0000", 2024-09-18T08:04:56.656 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-09-18T08:04:56.656 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:04:56.656 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:04:56.656 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "throttled", 2024-09-18T08:04:56.656 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:04:41.251870+0000", 2024-09-18T08:04:56.656 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.748e-06 2024-09-18T08:04:56.656 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:04:56.656 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:04:56.656 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "all_read", 2024-09-18T08:04:56.657 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:04:41.251899+0000", 2024-09-18T08:04:56.657 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 2.8761000000000001e-05 2024-09-18T08:04:56.657 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:04:56.657 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:04:56.657 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "dispatched", 2024-09-18T08:04:56.657 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:04:41.251917+0000", 2024-09-18T08:04:56.657 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.8578999999999999e-05 2024-09-18T08:04:56.657 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:04:56.657 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:04:56.657 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "queued_for_pg", 2024-09-18T08:04:56.657 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:04:41.251928+0000", 2024-09-18T08:04:56.657 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.0849999999999999e-05 2024-09-18T08:04:56.657 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:04:56.657 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:04:56.657 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-09-18T08:04:56.658 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:04:41.251995+0000", 2024-09-18T08:04:56.658 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 6.7117000000000001e-05 2024-09-18T08:04:56.658 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:04:56.658 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:04:56.658 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "started", 2024-09-18T08:04:56.658 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:04:41.252131+0000", 2024-09-18T08:04:56.658 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.000136105 2024-09-18T08:04:56.658 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:04:56.658 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:04:56.658 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "waiting for subops from 2,5", 2024-09-18T08:04:56.658 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:04:41.252289+0000", 2024-09-18T08:04:56.658 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.000157169 2024-09-18T08:04:56.658 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:04:56.659 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:04:56.659 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:04:56.659 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:04:41.266242+0000", 2024-09-18T08:04:56.659 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.013952915999999999 2024-09-18T08:04:56.659 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:04:56.659 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:04:56.659 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "op_commit", 2024-09-18T08:04:56.659 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:04:41.290023+0000", 2024-09-18T08:04:56.659 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.023781646 2024-09-18T08:04:56.659 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:04:56.659 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:04:56.659 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:04:56.660 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:04:41.302023+0000", 2024-09-18T08:04:56.660 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.011999591 2024-09-18T08:04:56.660 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:04:56.660 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:04:56.660 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "commit_sent", 2024-09-18T08:04:56.660 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:04:41.302187+0000", 2024-09-18T08:04:56.660 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.000164341 2024-09-18T08:04:56.660 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:04:56.660 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:04:56.661 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "done", 2024-09-18T08:04:56.661 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:04:41.302237+0000", 2024-09-18T08:04:56.661 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 5.0107e-05 2024-09-18T08:04:56.661 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-09-18T08:04:56.661 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-09-18T08:04:56.661 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-09-18T08:04:56.661 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:04:56.661 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:04:56.661 INFO:teuthology.orchestra.run.smithi007.stdout: "description": "osd_op(client.14805.0:63953 7.3 7:c0f9fc5d:::benchmark_data_smithi007_73155_object63952:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e100)", 2024-09-18T08:04:56.661 INFO:teuthology.orchestra.run.smithi007.stdout: "initiated_at": "2024-09-18T08:04:41.252014+0000", 2024-09-18T08:04:56.661 INFO:teuthology.orchestra.run.smithi007.stdout: "age": 14.380077012999999, 2024-09-18T08:04:56.661 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.039733352999999999, 2024-09-18T08:04:56.661 INFO:teuthology.orchestra.run.smithi007.stdout: "continuous": false, 2024-09-18T08:04:56.662 INFO:teuthology.orchestra.run.smithi007.stdout: "type_data": { 2024-09-18T08:04:56.662 INFO:teuthology.orchestra.run.smithi007.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T08:04:56.662 INFO:teuthology.orchestra.run.smithi007.stdout: "client_info": { 2024-09-18T08:04:56.662 INFO:teuthology.orchestra.run.smithi007.stdout: "client": "client.14805", 2024-09-18T08:04:56.662 INFO:teuthology.orchestra.run.smithi007.stdout: "client_addr": "172.21.15.7:0/640320436", 2024-09-18T08:04:56.662 INFO:teuthology.orchestra.run.smithi007.stdout: "tid": 63953 2024-09-18T08:04:56.662 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:04:56.662 INFO:teuthology.orchestra.run.smithi007.stdout: "events": [ 2024-09-18T08:04:56.662 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:04:56.662 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "initiated", 2024-09-18T08:04:56.662 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:04:41.252014+0000", 2024-09-18T08:04:56.662 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-09-18T08:04:56.662 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:04:56.662 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:04:56.663 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "header_read", 2024-09-18T08:04:56.663 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:04:41.252014+0000", 2024-09-18T08:04:56.663 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-09-18T08:04:56.663 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:04:56.663 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:04:56.663 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "throttled", 2024-09-18T08:04:56.663 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:04:41.252014+0000", 2024-09-18T08:04:56.663 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 6.2600000000000002e-07 2024-09-18T08:04:56.663 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:04:56.663 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:04:56.663 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "all_read", 2024-09-18T08:04:56.663 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:04:41.252041+0000", 2024-09-18T08:04:56.663 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 2.6210000000000001e-05 2024-09-18T08:04:56.664 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:04:56.664 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:04:56.664 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "dispatched", 2024-09-18T08:04:56.664 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:04:41.252048+0000", 2024-09-18T08:04:56.664 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 7.6059999999999997e-06 2024-09-18T08:04:56.664 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:04:56.664 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:04:56.664 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "queued_for_pg", 2024-09-18T08:04:56.664 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:04:41.252055+0000", 2024-09-18T08:04:56.664 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 6.5479999999999998e-06 2024-09-18T08:04:56.664 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:04:56.664 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:04:56.664 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-09-18T08:04:56.665 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:04:41.255074+0000", 2024-09-18T08:04:56.665 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.0030190669999999998 2024-09-18T08:04:56.665 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:04:56.665 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:04:56.665 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "started", 2024-09-18T08:04:56.665 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:04:41.255222+0000", 2024-09-18T08:04:56.665 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.00014779699999999999 2024-09-18T08:04:56.665 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:04:56.665 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:04:56.665 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "waiting for subops from 2,4", 2024-09-18T08:04:56.665 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:04:41.255513+0000", 2024-09-18T08:04:56.665 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.00029153600000000001 2024-09-18T08:04:56.665 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:04:56.665 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:04:56.666 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "op_commit", 2024-09-18T08:04:56.666 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:04:41.279961+0000", 2024-09-18T08:04:56.666 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.024447472000000001 2024-09-18T08:04:56.666 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:04:56.666 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:04:56.666 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:04:56.666 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:04:41.280015+0000", 2024-09-18T08:04:56.666 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 5.3980000000000002e-05 2024-09-18T08:04:56.666 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:04:56.666 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:04:56.666 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:04:56.666 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:04:41.291632+0000", 2024-09-18T08:04:56.666 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.011617943 2024-09-18T08:04:56.666 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:04:56.666 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:04:56.667 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "commit_sent", 2024-09-18T08:04:56.667 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:04:41.291709+0000", 2024-09-18T08:04:56.667 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 7.6289000000000001e-05 2024-09-18T08:04:56.667 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:04:56.667 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:04:56.667 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "done", 2024-09-18T08:04:56.667 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:04:41.291747+0000", 2024-09-18T08:04:56.667 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 3.8278999999999997e-05 2024-09-18T08:04:56.667 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-09-18T08:04:56.667 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-09-18T08:04:56.667 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-09-18T08:04:56.667 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:04:56.667 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:04:56.667 INFO:teuthology.orchestra.run.smithi007.stdout: "description": "osd_op(client.14805.0:63956 7.a 7:5e928473:::benchmark_data_smithi007_73155_object63955:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e100)", 2024-09-18T08:04:56.667 INFO:teuthology.orchestra.run.smithi007.stdout: "initiated_at": "2024-09-18T08:04:41.252065+0000", 2024-09-18T08:04:56.668 INFO:teuthology.orchestra.run.smithi007.stdout: "age": 14.380025553999999, 2024-09-18T08:04:56.668 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.050395849999999999, 2024-09-18T08:04:56.668 INFO:teuthology.orchestra.run.smithi007.stdout: "continuous": false, 2024-09-18T08:04:56.668 INFO:teuthology.orchestra.run.smithi007.stdout: "type_data": { 2024-09-18T08:04:56.668 INFO:teuthology.orchestra.run.smithi007.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T08:04:56.668 INFO:teuthology.orchestra.run.smithi007.stdout: "client_info": { 2024-09-18T08:04:56.668 INFO:teuthology.orchestra.run.smithi007.stdout: "client": "client.14805", 2024-09-18T08:04:56.668 INFO:teuthology.orchestra.run.smithi007.stdout: "client_addr": "172.21.15.7:0/640320436", 2024-09-18T08:04:56.668 INFO:teuthology.orchestra.run.smithi007.stdout: "tid": 63956 2024-09-18T08:04:56.668 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:04:56.668 INFO:teuthology.orchestra.run.smithi007.stdout: "events": [ 2024-09-18T08:04:56.668 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:04:56.668 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "initiated", 2024-09-18T08:04:56.668 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:04:41.252065+0000", 2024-09-18T08:04:56.668 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-09-18T08:04:56.668 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:04:56.669 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:04:56.669 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "header_read", 2024-09-18T08:04:56.669 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:04:41.252065+0000", 2024-09-18T08:04:56.669 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-09-18T08:04:56.669 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:04:56.669 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:04:56.669 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "throttled", 2024-09-18T08:04:56.669 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:04:41.252066+0000", 2024-09-18T08:04:56.669 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 4.8800000000000003e-07 2024-09-18T08:04:56.669 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:04:56.669 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:04:56.670 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "all_read", 2024-09-18T08:04:56.670 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:04:41.252126+0000", 2024-09-18T08:04:56.670 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 6.0408999999999997e-05 2024-09-18T08:04:56.670 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:04:56.670 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:04:56.670 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "dispatched", 2024-09-18T08:04:56.670 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:04:41.252132+0000", 2024-09-18T08:04:56.670 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 6.0750000000000003e-06 2024-09-18T08:04:56.670 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:04:56.670 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:04:56.670 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "queued_for_pg", 2024-09-18T08:04:56.671 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:04:41.252138+0000", 2024-09-18T08:04:56.671 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 6.2639999999999997e-06 2024-09-18T08:04:56.671 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:04:56.671 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:04:56.671 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-09-18T08:04:56.671 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:04:41.266282+0000", 2024-09-18T08:04:56.671 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.014143687 2024-09-18T08:04:56.671 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:04:56.671 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:04:56.671 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "started", 2024-09-18T08:04:56.671 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:04:41.266411+0000", 2024-09-18T08:04:56.672 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.000129338 2024-09-18T08:04:56.672 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:04:56.672 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:04:56.672 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "waiting for subops from 2,5", 2024-09-18T08:04:56.672 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:04:41.266553+0000", 2024-09-18T08:04:56.672 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.000141805 2024-09-18T08:04:56.672 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:04:56.672 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:04:56.672 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:04:56.672 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:04:41.278495+0000", 2024-09-18T08:04:56.672 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.011941887999999999 2024-09-18T08:04:56.672 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:04:56.672 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:04:56.673 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "op_commit", 2024-09-18T08:04:56.673 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:04:41.290047+0000", 2024-09-18T08:04:56.673 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.011552191 2024-09-18T08:04:56.673 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:04:56.673 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:04:56.673 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:04:56.673 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:04:41.302310+0000", 2024-09-18T08:04:56.673 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.012262363 2024-09-18T08:04:56.673 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:04:56.673 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:04:56.673 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "commit_sent", 2024-09-18T08:04:56.674 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:04:41.302421+0000", 2024-09-18T08:04:56.674 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.000111129 2024-09-18T08:04:56.674 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:04:56.674 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:04:56.674 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "done", 2024-09-18T08:04:56.674 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:04:41.302461+0000", 2024-09-18T08:04:56.674 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 4.0213000000000003e-05 2024-09-18T08:04:56.674 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-09-18T08:04:56.674 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-09-18T08:04:56.674 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-09-18T08:04:56.674 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:04:56.674 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:04:56.674 INFO:teuthology.orchestra.run.smithi007.stdout: "description": "osd_op(client.14805.0:64105 7.2 7:4d5ba537:::benchmark_data_smithi007_73155_object64104:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e100)", 2024-09-18T08:04:56.674 INFO:teuthology.orchestra.run.smithi007.stdout: "initiated_at": "2024-09-18T08:04:41.430085+0000", 2024-09-18T08:04:56.675 INFO:teuthology.orchestra.run.smithi007.stdout: "age": 14.202005228000001, 2024-09-18T08:04:56.675 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.040177125000000001, 2024-09-18T08:04:56.675 INFO:teuthology.orchestra.run.smithi007.stdout: "continuous": false, 2024-09-18T08:04:56.675 INFO:teuthology.orchestra.run.smithi007.stdout: "type_data": { 2024-09-18T08:04:56.675 INFO:teuthology.orchestra.run.smithi007.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T08:04:56.675 INFO:teuthology.orchestra.run.smithi007.stdout: "client_info": { 2024-09-18T08:04:56.675 INFO:teuthology.orchestra.run.smithi007.stdout: "client": "client.14805", 2024-09-18T08:04:56.675 INFO:teuthology.orchestra.run.smithi007.stdout: "client_addr": "172.21.15.7:0/640320436", 2024-09-18T08:04:56.675 INFO:teuthology.orchestra.run.smithi007.stdout: "tid": 64105 2024-09-18T08:04:56.675 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:04:56.675 INFO:teuthology.orchestra.run.smithi007.stdout: "events": [ 2024-09-18T08:04:56.676 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:04:56.676 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "initiated", 2024-09-18T08:04:56.676 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:04:41.430085+0000", 2024-09-18T08:04:56.676 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-09-18T08:04:56.676 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:04:56.676 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:04:56.676 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "header_read", 2024-09-18T08:04:56.676 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:04:41.430085+0000", 2024-09-18T08:04:56.676 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-09-18T08:04:56.676 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:04:56.676 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:04:56.677 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "throttled", 2024-09-18T08:04:56.677 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:04:41.430087+0000", 2024-09-18T08:04:56.677 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.158e-06 2024-09-18T08:04:56.677 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:04:56.677 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:04:56.677 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "all_read", 2024-09-18T08:04:56.677 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:04:41.430109+0000", 2024-09-18T08:04:56.677 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 2.2493999999999998e-05 2024-09-18T08:04:56.677 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:04:56.677 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:04:56.677 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "dispatched", 2024-09-18T08:04:56.677 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:04:41.430121+0000", 2024-09-18T08:04:56.678 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.1771e-05 2024-09-18T08:04:56.678 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:04:56.678 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:04:56.678 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "queued_for_pg", 2024-09-18T08:04:56.678 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:04:41.430129+0000", 2024-09-18T08:04:56.678 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 7.8869999999999992e-06 2024-09-18T08:04:56.678 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:04:56.678 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:04:56.678 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-09-18T08:04:56.679 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:04:41.438488+0000", 2024-09-18T08:04:56.679 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.0083592530000000005 2024-09-18T08:04:56.679 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:04:56.679 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:04:56.679 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "started", 2024-09-18T08:04:56.679 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:04:41.438597+0000", 2024-09-18T08:04:56.679 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.000108837 2024-09-18T08:04:56.679 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:04:56.679 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:04:56.679 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "waiting for subops from 1,2", 2024-09-18T08:04:56.679 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:04:41.444815+0000", 2024-09-18T08:04:56.679 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.0062176660000000002 2024-09-18T08:04:56.679 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:04:56.679 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:04:56.680 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "op_commit", 2024-09-18T08:04:56.680 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:04:41.447132+0000", 2024-09-18T08:04:56.680 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.0023179059999999998 2024-09-18T08:04:56.680 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:04:56.680 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:04:56.680 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:04:56.680 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:04:41.458054+0000", 2024-09-18T08:04:56.680 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.010921896 2024-09-18T08:04:56.680 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:04:56.680 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:04:56.681 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:04:56.681 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:04:41.470154+0000", 2024-09-18T08:04:56.681 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.012100029999999999 2024-09-18T08:04:56.681 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:04:56.681 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:04:56.681 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "commit_sent", 2024-09-18T08:04:56.681 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:04:41.470217+0000", 2024-09-18T08:04:56.681 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 6.2119000000000006e-05 2024-09-18T08:04:56.681 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:04:56.681 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:04:56.682 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "done", 2024-09-18T08:04:56.682 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:04:41.470263+0000", 2024-09-18T08:04:56.682 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 4.6108e-05 2024-09-18T08:04:56.682 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-09-18T08:04:56.682 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-09-18T08:04:56.682 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-09-18T08:04:56.682 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:04:56.682 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:04:56.682 INFO:teuthology.orchestra.run.smithi007.stdout: "description": "osd_op(client.14805.0:64107 7.2 7:4c13e0d3:::benchmark_data_smithi007_73155_object64106:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e100)", 2024-09-18T08:04:56.682 INFO:teuthology.orchestra.run.smithi007.stdout: "initiated_at": "2024-09-18T08:04:41.430142+0000", 2024-09-18T08:04:56.682 INFO:teuthology.orchestra.run.smithi007.stdout: "age": 14.201949106000001, 2024-09-18T08:04:56.682 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.041301053999999997, 2024-09-18T08:04:56.682 INFO:teuthology.orchestra.run.smithi007.stdout: "continuous": false, 2024-09-18T08:04:56.682 INFO:teuthology.orchestra.run.smithi007.stdout: "type_data": { 2024-09-18T08:04:56.683 INFO:teuthology.orchestra.run.smithi007.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T08:04:56.683 INFO:teuthology.orchestra.run.smithi007.stdout: "client_info": { 2024-09-18T08:04:56.683 INFO:teuthology.orchestra.run.smithi007.stdout: "client": "client.14805", 2024-09-18T08:04:56.683 INFO:teuthology.orchestra.run.smithi007.stdout: "client_addr": "172.21.15.7:0/640320436", 2024-09-18T08:04:56.683 INFO:teuthology.orchestra.run.smithi007.stdout: "tid": 64107 2024-09-18T08:04:56.683 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:04:56.683 INFO:teuthology.orchestra.run.smithi007.stdout: "events": [ 2024-09-18T08:04:56.683 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:04:56.683 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "initiated", 2024-09-18T08:04:56.683 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:04:41.430142+0000", 2024-09-18T08:04:56.683 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-09-18T08:04:56.683 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:04:56.683 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:04:56.684 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "header_read", 2024-09-18T08:04:56.684 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:04:41.430142+0000", 2024-09-18T08:04:56.684 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-09-18T08:04:56.684 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:04:56.684 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:04:56.684 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "throttled", 2024-09-18T08:04:56.684 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:04:41.430142+0000", 2024-09-18T08:04:56.684 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 5.3499999999999996e-07 2024-09-18T08:04:56.684 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:04:56.685 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:04:56.685 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "all_read", 2024-09-18T08:04:56.685 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:04:41.430164+0000", 2024-09-18T08:04:56.685 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 2.1671e-05 2024-09-18T08:04:56.685 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:04:56.685 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:04:56.685 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "dispatched", 2024-09-18T08:04:56.685 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:04:41.430172+0000", 2024-09-18T08:04:56.685 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 7.7540000000000006e-06 2024-09-18T08:04:56.685 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:04:56.685 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:04:56.685 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "queued_for_pg", 2024-09-18T08:04:56.685 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:04:41.430178+0000", 2024-09-18T08:04:56.686 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 6.1940000000000003e-06 2024-09-18T08:04:56.686 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:04:56.686 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:04:56.686 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-09-18T08:04:56.686 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:04:41.447382+0000", 2024-09-18T08:04:56.686 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.017204276000000001 2024-09-18T08:04:56.686 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:04:56.686 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:04:56.686 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "started", 2024-09-18T08:04:56.686 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:04:41.447522+0000", 2024-09-18T08:04:56.686 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.00013984800000000001 2024-09-18T08:04:56.686 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:04:56.686 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:04:56.686 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "waiting for subops from 1,2", 2024-09-18T08:04:56.686 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:04:41.447673+0000", 2024-09-18T08:04:56.687 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.000150641 2024-09-18T08:04:56.687 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:04:56.687 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:04:56.687 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "op_commit", 2024-09-18T08:04:56.687 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:04:41.453057+0000", 2024-09-18T08:04:56.687 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.0053843609999999998 2024-09-18T08:04:56.687 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:04:56.687 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:04:56.687 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:04:56.687 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:04:41.458102+0000", 2024-09-18T08:04:56.687 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.0050453039999999996 2024-09-18T08:04:56.687 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:04:56.687 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:04:56.688 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:04:56.688 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:04:41.471155+0000", 2024-09-18T08:04:56.688 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.01305276 2024-09-18T08:04:56.688 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:04:56.688 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:04:56.688 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "commit_sent", 2024-09-18T08:04:56.688 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:04:41.471215+0000", 2024-09-18T08:04:56.688 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 5.9963999999999997e-05 2024-09-18T08:04:56.688 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:04:56.688 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:04:56.688 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "done", 2024-09-18T08:04:56.688 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:04:41.471443+0000", 2024-09-18T08:04:56.688 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.000227746 2024-09-18T08:04:56.688 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-09-18T08:04:56.689 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-09-18T08:04:56.689 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-09-18T08:04:56.689 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:04:56.689 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:04:56.689 INFO:teuthology.orchestra.run.smithi007.stdout: "description": "osd_op(client.14805.0:65066 7.c 7:3fc93c93:::benchmark_data_smithi007_73155_object65065:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e101)", 2024-09-18T08:04:56.689 INFO:teuthology.orchestra.run.smithi007.stdout: "initiated_at": "2024-09-18T08:04:42.307272+0000", 2024-09-18T08:04:56.689 INFO:teuthology.orchestra.run.smithi007.stdout: "age": 13.324818252, 2024-09-18T08:04:56.689 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.011528593, 2024-09-18T08:04:56.689 INFO:teuthology.orchestra.run.smithi007.stdout: "continuous": false, 2024-09-18T08:04:56.689 INFO:teuthology.orchestra.run.smithi007.stdout: "type_data": { 2024-09-18T08:04:56.689 INFO:teuthology.orchestra.run.smithi007.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T08:04:56.689 INFO:teuthology.orchestra.run.smithi007.stdout: "client_info": { 2024-09-18T08:04:56.689 INFO:teuthology.orchestra.run.smithi007.stdout: "client": "client.14805", 2024-09-18T08:04:56.689 INFO:teuthology.orchestra.run.smithi007.stdout: "client_addr": "172.21.15.7:0/640320436", 2024-09-18T08:04:56.690 INFO:teuthology.orchestra.run.smithi007.stdout: "tid": 65066 2024-09-18T08:04:56.690 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:04:56.690 INFO:teuthology.orchestra.run.smithi007.stdout: "events": [ 2024-09-18T08:04:56.690 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:04:56.690 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "initiated", 2024-09-18T08:04:56.690 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:04:42.307272+0000", 2024-09-18T08:04:56.690 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-09-18T08:04:56.690 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:04:56.690 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:04:56.690 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "header_read", 2024-09-18T08:04:56.690 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:04:42.307272+0000", 2024-09-18T08:04:56.690 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-09-18T08:04:56.690 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:04:56.691 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:04:56.691 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "throttled", 2024-09-18T08:04:56.691 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:04:42.307273+0000", 2024-09-18T08:04:56.691 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 9.8899999999999998e-07 2024-09-18T08:04:56.691 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:04:56.691 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:04:56.691 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "all_read", 2024-09-18T08:04:56.691 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:04:42.307296+0000", 2024-09-18T08:04:56.691 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 2.2597000000000001e-05 2024-09-18T08:04:56.691 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:04:56.691 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:04:56.691 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "dispatched", 2024-09-18T08:04:56.692 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:04:42.307305+0000", 2024-09-18T08:04:56.692 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 8.8810000000000002e-06 2024-09-18T08:04:56.692 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:04:56.692 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:04:56.692 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "queued_for_pg", 2024-09-18T08:04:56.692 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:04:42.307312+0000", 2024-09-18T08:04:56.692 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 6.6810000000000001e-06 2024-09-18T08:04:56.692 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:04:56.692 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:04:56.692 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-09-18T08:04:56.692 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:04:42.307372+0000", 2024-09-18T08:04:56.693 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 6.0723000000000001e-05 2024-09-18T08:04:56.693 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:04:56.693 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:04:56.693 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "waiting for peered", 2024-09-18T08:04:56.693 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:04:42.307381+0000", 2024-09-18T08:04:56.693 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 8.8189999999999998e-06 2024-09-18T08:04:56.693 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:04:56.693 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:04:56.693 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-09-18T08:04:56.693 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:04:43.304115+0000", 2024-09-18T08:04:56.693 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.996733435 2024-09-18T08:04:56.693 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:04:56.693 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:04:56.693 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "started", 2024-09-18T08:04:56.694 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:04:43.305386+0000", 2024-09-18T08:04:56.694 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.001270972 2024-09-18T08:04:56.694 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:04:56.694 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:04:56.694 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "waiting for subops from 6,7", 2024-09-18T08:04:56.694 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:04:43.305509+0000", 2024-09-18T08:04:56.694 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.000123113 2024-09-18T08:04:56.694 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:04:56.694 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:04:56.694 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:04:56.694 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:04:43.313755+0000", 2024-09-18T08:04:56.695 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.0082463629999999996 2024-09-18T08:04:56.695 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:04:56.695 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:04:56.695 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:04:56.695 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:04:43.313809+0000", 2024-09-18T08:04:56.695 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 5.3902e-05 2024-09-18T08:04:56.695 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:04:56.695 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:04:56.695 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "op_commit", 2024-09-18T08:04:56.696 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:04:43.318654+0000", 2024-09-18T08:04:56.696 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.004845172 2024-09-18T08:04:56.696 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:04:56.696 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:04:56.696 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "commit_sent", 2024-09-18T08:04:56.696 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:04:43.318752+0000", 2024-09-18T08:04:56.696 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 9.8005999999999996e-05 2024-09-18T08:04:56.696 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:04:56.696 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:04:56.696 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "done", 2024-09-18T08:04:56.697 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:04:43.318801+0000", 2024-09-18T08:04:56.697 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 4.8940000000000002e-05 2024-09-18T08:04:56.697 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-09-18T08:04:56.697 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-09-18T08:04:56.697 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-09-18T08:04:56.697 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:04:56.697 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:04:56.697 INFO:teuthology.orchestra.run.smithi007.stdout: "description": "osd_op(client.14805.0:65075 7.c 7:3d93ab73:::benchmark_data_smithi007_73155_object65074:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e101)", 2024-09-18T08:04:56.697 INFO:teuthology.orchestra.run.smithi007.stdout: "initiated_at": "2024-09-18T08:04:42.315080+0000", 2024-09-18T08:04:56.697 INFO:teuthology.orchestra.run.smithi007.stdout: "age": 13.317010632000001, 2024-09-18T08:04:56.697 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.003851802, 2024-09-18T08:04:56.698 INFO:teuthology.orchestra.run.smithi007.stdout: "continuous": false, 2024-09-18T08:04:56.698 INFO:teuthology.orchestra.run.smithi007.stdout: "type_data": { 2024-09-18T08:04:56.698 INFO:teuthology.orchestra.run.smithi007.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T08:04:56.698 INFO:teuthology.orchestra.run.smithi007.stdout: "client_info": { 2024-09-18T08:04:56.698 INFO:teuthology.orchestra.run.smithi007.stdout: "client": "client.14805", 2024-09-18T08:04:56.698 INFO:teuthology.orchestra.run.smithi007.stdout: "client_addr": "172.21.15.7:0/640320436", 2024-09-18T08:04:56.698 INFO:teuthology.orchestra.run.smithi007.stdout: "tid": 65075 2024-09-18T08:04:56.698 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:04:56.698 INFO:teuthology.orchestra.run.smithi007.stdout: "events": [ 2024-09-18T08:04:56.698 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:04:56.698 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "initiated", 2024-09-18T08:04:56.698 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:04:42.315080+0000", 2024-09-18T08:04:56.698 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-09-18T08:04:56.698 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:04:56.699 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:04:56.699 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "header_read", 2024-09-18T08:04:56.699 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:04:42.315080+0000", 2024-09-18T08:04:56.699 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-09-18T08:04:56.699 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:04:56.699 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:04:56.699 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "throttled", 2024-09-18T08:04:56.699 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:04:42.315082+0000", 2024-09-18T08:04:56.699 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.499e-06 2024-09-18T08:04:56.699 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:04:56.699 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:04:56.699 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "all_read", 2024-09-18T08:04:56.699 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:04:42.315105+0000", 2024-09-18T08:04:56.700 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 2.3039000000000001e-05 2024-09-18T08:04:56.700 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:04:56.700 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:04:56.700 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "dispatched", 2024-09-18T08:04:56.700 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:04:42.315121+0000", 2024-09-18T08:04:56.700 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.6155999999999999e-05 2024-09-18T08:04:56.700 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:04:56.700 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:04:56.700 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "queued_for_pg", 2024-09-18T08:04:56.700 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:04:42.315129+0000", 2024-09-18T08:04:56.700 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 8.0870000000000007e-06 2024-09-18T08:04:56.701 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:04:56.701 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:04:56.701 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-09-18T08:04:56.701 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:04:42.315194+0000", 2024-09-18T08:04:56.701 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 6.4966000000000001e-05 2024-09-18T08:04:56.701 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:04:56.701 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:04:56.701 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "waiting for peered", 2024-09-18T08:04:56.701 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:04:42.315204+0000", 2024-09-18T08:04:56.702 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.031e-05 2024-09-18T08:04:56.702 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:04:56.702 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:04:56.702 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-09-18T08:04:56.702 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:04:43.308149+0000", 2024-09-18T08:04:56.702 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.99294446599999997 2024-09-18T08:04:56.702 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:04:56.702 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:04:56.702 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "started", 2024-09-18T08:04:56.703 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:04:43.308252+0000", 2024-09-18T08:04:56.703 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.00010303500000000001 2024-09-18T08:04:56.703 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:04:56.703 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:04:56.703 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "waiting for subops from 6,7", 2024-09-18T08:04:56.703 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:04:43.308374+0000", 2024-09-18T08:04:56.703 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.000121981 2024-09-18T08:04:56.703 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:04:56.703 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:04:56.703 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:04:56.703 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:04:43.313865+0000", 2024-09-18T08:04:56.703 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.0054910640000000004 2024-09-18T08:04:56.703 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:04:56.703 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:04:56.704 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:04:56.704 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:04:43.313912+0000", 2024-09-18T08:04:56.704 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 4.7299000000000002e-05 2024-09-18T08:04:56.704 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:04:56.704 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:04:56.704 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "op_commit", 2024-09-18T08:04:56.704 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:04:43.318840+0000", 2024-09-18T08:04:56.704 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.0049279720000000001 2024-09-18T08:04:56.704 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:04:56.704 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:04:56.704 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "commit_sent", 2024-09-18T08:04:56.704 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:04:43.318896+0000", 2024-09-18T08:04:56.704 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 5.6047000000000001e-05 2024-09-18T08:04:56.704 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:04:56.705 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:04:56.705 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "done", 2024-09-18T08:04:56.705 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:04:43.318932+0000", 2024-09-18T08:04:56.705 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 3.5880999999999997e-05 2024-09-18T08:04:56.705 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-09-18T08:04:56.705 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-09-18T08:04:56.705 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-09-18T08:04:56.705 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:04:56.705 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:04:56.705 INFO:teuthology.orchestra.run.smithi007.stdout: "description": "osd_op(client.14805.0:65116 7.c 7:3d4b688b:::benchmark_data_smithi007_73155_object65115:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e101)", 2024-09-18T08:04:56.705 INFO:teuthology.orchestra.run.smithi007.stdout: "initiated_at": "2024-09-18T08:04:42.353168+0000", 2024-09-18T08:04:56.706 INFO:teuthology.orchestra.run.smithi007.stdout: "age": 13.278922455, 2024-09-18T08:04:56.706 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.96584373300000004, 2024-09-18T08:04:56.706 INFO:teuthology.orchestra.run.smithi007.stdout: "continuous": false, 2024-09-18T08:04:56.706 INFO:teuthology.orchestra.run.smithi007.stdout: "type_data": { 2024-09-18T08:04:56.706 INFO:teuthology.orchestra.run.smithi007.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T08:04:56.706 INFO:teuthology.orchestra.run.smithi007.stdout: "client_info": { 2024-09-18T08:04:56.706 INFO:teuthology.orchestra.run.smithi007.stdout: "client": "client.14805", 2024-09-18T08:04:56.706 INFO:teuthology.orchestra.run.smithi007.stdout: "client_addr": "172.21.15.7:0/640320436", 2024-09-18T08:04:56.706 INFO:teuthology.orchestra.run.smithi007.stdout: "tid": 65116 2024-09-18T08:04:56.706 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:04:56.706 INFO:teuthology.orchestra.run.smithi007.stdout: "events": [ 2024-09-18T08:04:56.706 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:04:56.706 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "initiated", 2024-09-18T08:04:56.706 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:04:42.353168+0000", 2024-09-18T08:04:56.707 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-09-18T08:04:56.707 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:04:56.707 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:04:56.707 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "header_read", 2024-09-18T08:04:56.707 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:04:42.353168+0000", 2024-09-18T08:04:56.707 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-09-18T08:04:56.707 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:04:56.707 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:04:56.707 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "throttled", 2024-09-18T08:04:56.707 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:04:42.353170+0000", 2024-09-18T08:04:56.707 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.77e-06 2024-09-18T08:04:56.707 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:04:56.707 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:04:56.707 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "all_read", 2024-09-18T08:04:56.707 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:04:42.353195+0000", 2024-09-18T08:04:56.708 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 2.4859999999999999e-05 2024-09-18T08:04:56.708 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:04:56.708 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:04:56.708 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "dispatched", 2024-09-18T08:04:56.708 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:04:42.353206+0000", 2024-09-18T08:04:56.708 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.1263999999999999e-05 2024-09-18T08:04:56.708 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:04:56.708 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:04:56.708 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "queued_for_pg", 2024-09-18T08:04:56.708 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:04:42.353213+0000", 2024-09-18T08:04:56.708 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 7.2099999999999996e-06 2024-09-18T08:04:56.708 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:04:56.708 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:04:56.708 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-09-18T08:04:56.709 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:04:42.353274+0000", 2024-09-18T08:04:56.709 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 6.0185000000000001e-05 2024-09-18T08:04:56.709 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:04:56.709 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:04:56.709 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "waiting for peered", 2024-09-18T08:04:56.709 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:04:42.353280+0000", 2024-09-18T08:04:56.709 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 6.4400000000000002e-06 2024-09-18T08:04:56.709 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:04:56.709 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:04:56.709 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-09-18T08:04:56.709 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:04:43.310990+0000", 2024-09-18T08:04:56.709 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.95770993599999998 2024-09-18T08:04:56.709 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:04:56.709 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:04:56.709 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "started", 2024-09-18T08:04:56.709 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:04:43.311087+0000", 2024-09-18T08:04:56.710 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 9.7320999999999999e-05 2024-09-18T08:04:56.710 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:04:56.710 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:04:56.710 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "waiting for subops from 6,7", 2024-09-18T08:04:56.710 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:04:43.311209+0000", 2024-09-18T08:04:56.710 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.00012188099999999999 2024-09-18T08:04:56.710 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:04:56.710 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:04:56.710 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:04:56.710 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:04:43.318599+0000", 2024-09-18T08:04:56.710 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.0073900429999999998 2024-09-18T08:04:56.710 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:04:56.710 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:04:56.710 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:04:56.710 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:04:43.318643+0000", 2024-09-18T08:04:56.711 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 4.3689000000000001e-05 2024-09-18T08:04:56.711 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:04:56.711 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:04:56.711 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "op_commit", 2024-09-18T08:04:56.711 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:04:43.318939+0000", 2024-09-18T08:04:56.711 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.000296266 2024-09-18T08:04:56.711 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:04:56.711 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:04:56.711 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "commit_sent", 2024-09-18T08:04:56.711 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:04:43.318979+0000", 2024-09-18T08:04:56.711 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 4.0068000000000003e-05 2024-09-18T08:04:56.711 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:04:56.712 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:04:56.712 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "done", 2024-09-18T08:04:56.712 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:04:43.319012+0000", 2024-09-18T08:04:56.712 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 3.2799999999999998e-05 2024-09-18T08:04:56.712 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-09-18T08:04:56.712 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-09-18T08:04:56.712 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-09-18T08:04:56.712 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:04:56.712 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:04:56.712 INFO:teuthology.orchestra.run.smithi007.stdout: "description": "osd_op(client.14805.0:65705 7.2 7:43d9ffa2:::benchmark_data_smithi007_73155_object65704:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e102)", 2024-09-18T08:04:56.712 INFO:teuthology.orchestra.run.smithi007.stdout: "initiated_at": "2024-09-18T08:04:43.720835+0000", 2024-09-18T08:04:56.712 INFO:teuthology.orchestra.run.smithi007.stdout: "age": 11.911255379, 2024-09-18T08:04:56.713 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.039882449, 2024-09-18T08:04:56.713 INFO:teuthology.orchestra.run.smithi007.stdout: "continuous": false, 2024-09-18T08:04:56.713 INFO:teuthology.orchestra.run.smithi007.stdout: "type_data": { 2024-09-18T08:04:56.713 INFO:teuthology.orchestra.run.smithi007.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T08:04:56.713 INFO:teuthology.orchestra.run.smithi007.stdout: "client_info": { 2024-09-18T08:04:56.713 INFO:teuthology.orchestra.run.smithi007.stdout: "client": "client.14805", 2024-09-18T08:04:56.713 INFO:teuthology.orchestra.run.smithi007.stdout: "client_addr": "172.21.15.7:0/640320436", 2024-09-18T08:04:56.713 INFO:teuthology.orchestra.run.smithi007.stdout: "tid": 65705 2024-09-18T08:04:56.713 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:04:56.713 INFO:teuthology.orchestra.run.smithi007.stdout: "events": [ 2024-09-18T08:04:56.713 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:04:56.713 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "initiated", 2024-09-18T08:04:56.713 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:04:43.720835+0000", 2024-09-18T08:04:56.714 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-09-18T08:04:56.714 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:04:56.714 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:04:56.714 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "header_read", 2024-09-18T08:04:56.714 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:04:43.720835+0000", 2024-09-18T08:04:56.714 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-09-18T08:04:56.714 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:04:56.714 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:04:56.714 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "throttled", 2024-09-18T08:04:56.714 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:04:43.720837+0000", 2024-09-18T08:04:56.714 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.3459999999999999e-06 2024-09-18T08:04:56.714 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:04:56.714 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:04:56.714 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "all_read", 2024-09-18T08:04:56.714 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:04:43.720866+0000", 2024-09-18T08:04:56.715 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 2.9581999999999999e-05 2024-09-18T08:04:56.715 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:04:56.715 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:04:56.715 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "dispatched", 2024-09-18T08:04:56.715 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:04:43.720881+0000", 2024-09-18T08:04:56.715 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.4905e-05 2024-09-18T08:04:56.715 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:04:56.715 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:04:56.715 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "queued_for_pg", 2024-09-18T08:04:56.715 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:04:43.720888+0000", 2024-09-18T08:04:56.715 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 7.0589999999999997e-06 2024-09-18T08:04:56.715 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:04:56.715 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:04:56.716 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-09-18T08:04:56.716 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:04:43.728294+0000", 2024-09-18T08:04:56.716 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.0074053610000000001 2024-09-18T08:04:56.716 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:04:56.716 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:04:56.716 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "started", 2024-09-18T08:04:56.716 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:04:43.728714+0000", 2024-09-18T08:04:56.716 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.00042031700000000002 2024-09-18T08:04:56.716 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:04:56.716 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:04:56.716 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "waiting for subops from 1,2", 2024-09-18T08:04:56.717 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:04:43.728867+0000", 2024-09-18T08:04:56.717 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.00015336599999999999 2024-09-18T08:04:56.717 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:04:56.717 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:04:56.717 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:04:56.717 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:04:43.738081+0000", 2024-09-18T08:04:56.717 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.0092138250000000001 2024-09-18T08:04:56.717 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:04:56.717 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:04:56.718 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:04:56.718 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:04:43.738418+0000", 2024-09-18T08:04:56.718 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.00033718600000000001 2024-09-18T08:04:56.718 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:04:56.718 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:04:56.718 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "op_commit", 2024-09-18T08:04:56.718 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:04:43.760609+0000", 2024-09-18T08:04:56.718 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.022190754 2024-09-18T08:04:56.718 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:04:56.718 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:04:56.718 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "commit_sent", 2024-09-18T08:04:56.718 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:04:43.760671+0000", 2024-09-18T08:04:56.718 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 6.2226999999999999e-05 2024-09-18T08:04:56.718 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:04:56.718 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:04:56.719 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "done", 2024-09-18T08:04:56.719 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:04:43.760718+0000", 2024-09-18T08:04:56.719 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 4.6520999999999997e-05 2024-09-18T08:04:56.719 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-09-18T08:04:56.719 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-09-18T08:04:56.719 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-09-18T08:04:56.719 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:04:56.719 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:04:56.719 INFO:teuthology.orchestra.run.smithi007.stdout: "description": "osd_op(client.14805.0:68197 7.a 7:559455ad:::benchmark_data_smithi007_73155_object68196:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e102)", 2024-09-18T08:04:56.719 INFO:teuthology.orchestra.run.smithi007.stdout: "initiated_at": "2024-09-18T08:04:45.695798+0000", 2024-09-18T08:04:56.719 INFO:teuthology.orchestra.run.smithi007.stdout: "age": 9.9362922810000001, 2024-09-18T08:04:56.719 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.041888770999999998, 2024-09-18T08:04:56.719 INFO:teuthology.orchestra.run.smithi007.stdout: "continuous": false, 2024-09-18T08:04:56.719 INFO:teuthology.orchestra.run.smithi007.stdout: "type_data": { 2024-09-18T08:04:56.720 INFO:teuthology.orchestra.run.smithi007.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T08:04:56.720 INFO:teuthology.orchestra.run.smithi007.stdout: "client_info": { 2024-09-18T08:04:56.720 INFO:teuthology.orchestra.run.smithi007.stdout: "client": "client.14805", 2024-09-18T08:04:56.720 INFO:teuthology.orchestra.run.smithi007.stdout: "client_addr": "172.21.15.7:0/640320436", 2024-09-18T08:04:56.720 INFO:teuthology.orchestra.run.smithi007.stdout: "tid": 68197 2024-09-18T08:04:56.720 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:04:56.720 INFO:teuthology.orchestra.run.smithi007.stdout: "events": [ 2024-09-18T08:04:56.720 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:04:56.720 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "initiated", 2024-09-18T08:04:56.720 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:04:45.695798+0000", 2024-09-18T08:04:56.720 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-09-18T08:04:56.720 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:04:56.720 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:04:56.721 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "header_read", 2024-09-18T08:04:56.721 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:04:45.695798+0000", 2024-09-18T08:04:56.721 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-09-18T08:04:56.721 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:04:56.721 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:04:56.721 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "throttled", 2024-09-18T08:04:56.721 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:04:45.695800+0000", 2024-09-18T08:04:56.721 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.4160000000000001e-06 2024-09-18T08:04:56.721 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:04:56.721 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:04:56.721 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "all_read", 2024-09-18T08:04:56.721 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:04:45.695822+0000", 2024-09-18T08:04:56.721 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 2.1970000000000001e-05 2024-09-18T08:04:56.722 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:04:56.722 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:04:56.722 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "dispatched", 2024-09-18T08:04:56.722 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:04:45.695833+0000", 2024-09-18T08:04:56.722 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.1031e-05 2024-09-18T08:04:56.722 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:04:56.722 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:04:56.722 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "queued_for_pg", 2024-09-18T08:04:56.722 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:04:45.695840+0000", 2024-09-18T08:04:56.722 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 6.8979999999999999e-06 2024-09-18T08:04:56.723 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:04:56.723 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:04:56.723 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-09-18T08:04:56.723 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:04:45.711257+0000", 2024-09-18T08:04:56.723 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.015417357 2024-09-18T08:04:56.723 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:04:56.723 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:04:56.723 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "started", 2024-09-18T08:04:56.723 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:04:45.711384+0000", 2024-09-18T08:04:56.724 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.000126933 2024-09-18T08:04:56.724 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:04:56.724 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:04:56.724 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "waiting for subops from 2,5", 2024-09-18T08:04:56.724 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:04:45.711530+0000", 2024-09-18T08:04:56.724 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.00014553 2024-09-18T08:04:56.724 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:04:56.724 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:04:56.724 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:04:56.724 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:04:45.715856+0000", 2024-09-18T08:04:56.724 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.0043261439999999996 2024-09-18T08:04:56.725 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:04:56.725 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:04:56.725 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "op_commit", 2024-09-18T08:04:56.725 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:04:45.717054+0000", 2024-09-18T08:04:56.725 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.0011982340000000001 2024-09-18T08:04:56.725 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:04:56.725 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:04:56.725 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:04:56.725 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:04:45.737594+0000", 2024-09-18T08:04:56.725 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.020539598999999999 2024-09-18T08:04:56.725 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:04:56.725 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:04:56.725 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "commit_sent", 2024-09-18T08:04:56.726 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:04:45.737645+0000", 2024-09-18T08:04:56.726 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 5.164e-05 2024-09-18T08:04:56.726 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:04:56.726 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:04:56.726 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "done", 2024-09-18T08:04:56.726 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:04:45.737687+0000", 2024-09-18T08:04:56.726 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 4.2018999999999999e-05 2024-09-18T08:04:56.726 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-09-18T08:04:56.726 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-09-18T08:04:56.726 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-09-18T08:04:56.726 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:04:56.726 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:04:56.726 INFO:teuthology.orchestra.run.smithi007.stdout: "description": "osd_op(client.14805.0:73312 7.2 7:4e7444c3:::benchmark_data_smithi007_73155_object73311:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e104)", 2024-09-18T08:04:56.726 INFO:teuthology.orchestra.run.smithi007.stdout: "initiated_at": "2024-09-18T08:04:50.037142+0000", 2024-09-18T08:04:56.727 INFO:teuthology.orchestra.run.smithi007.stdout: "age": 5.5949486310000003, 2024-09-18T08:04:56.727 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.043936081000000002, 2024-09-18T08:04:56.727 INFO:teuthology.orchestra.run.smithi007.stdout: "continuous": false, 2024-09-18T08:04:56.727 INFO:teuthology.orchestra.run.smithi007.stdout: "type_data": { 2024-09-18T08:04:56.727 INFO:teuthology.orchestra.run.smithi007.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T08:04:56.727 INFO:teuthology.orchestra.run.smithi007.stdout: "client_info": { 2024-09-18T08:04:56.727 INFO:teuthology.orchestra.run.smithi007.stdout: "client": "client.14805", 2024-09-18T08:04:56.727 INFO:teuthology.orchestra.run.smithi007.stdout: "client_addr": "172.21.15.7:0/640320436", 2024-09-18T08:04:56.727 INFO:teuthology.orchestra.run.smithi007.stdout: "tid": 73312 2024-09-18T08:04:56.727 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:04:56.727 INFO:teuthology.orchestra.run.smithi007.stdout: "events": [ 2024-09-18T08:04:56.727 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:04:56.727 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "initiated", 2024-09-18T08:04:56.728 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:04:50.037142+0000", 2024-09-18T08:04:56.728 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-09-18T08:04:56.728 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:04:56.728 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:04:56.728 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "header_read", 2024-09-18T08:04:56.728 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:04:50.037142+0000", 2024-09-18T08:04:56.728 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-09-18T08:04:56.728 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:04:56.728 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:04:56.728 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "throttled", 2024-09-18T08:04:56.728 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:04:50.037144+0000", 2024-09-18T08:04:56.729 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.6789999999999999e-06 2024-09-18T08:04:56.729 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:04:56.729 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:04:56.729 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "all_read", 2024-09-18T08:04:56.729 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:04:50.037167+0000", 2024-09-18T08:04:56.729 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 2.2739999999999999e-05 2024-09-18T08:04:56.729 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:04:56.729 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:04:56.729 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "dispatched", 2024-09-18T08:04:56.729 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:04:50.037177+0000", 2024-09-18T08:04:56.729 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.0457e-05 2024-09-18T08:04:56.729 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:04:56.730 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:04:56.730 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "queued_for_pg", 2024-09-18T08:04:56.730 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:04:50.037185+0000", 2024-09-18T08:04:56.730 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 8.1070000000000005e-06 2024-09-18T08:04:56.730 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:04:56.730 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:04:56.730 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-09-18T08:04:56.730 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:04:50.044770+0000", 2024-09-18T08:04:56.730 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.0075850149999999996 2024-09-18T08:04:56.730 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:04:56.730 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:04:56.730 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "started", 2024-09-18T08:04:56.730 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:04:50.044907+0000", 2024-09-18T08:04:56.731 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.000137205 2024-09-18T08:04:56.731 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:04:56.731 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:04:56.731 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "waiting for subops from 1,2", 2024-09-18T08:04:56.731 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:04:50.045047+0000", 2024-09-18T08:04:56.731 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.00013929400000000001 2024-09-18T08:04:56.731 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:04:56.731 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:04:56.731 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "op_commit", 2024-09-18T08:04:56.732 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:04:50.051075+0000", 2024-09-18T08:04:56.732 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.0060279849999999996 2024-09-18T08:04:56.732 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:04:56.732 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:04:56.733 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:04:56.733 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:04:50.051934+0000", 2024-09-18T08:04:56.733 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.00085901 2024-09-18T08:04:56.733 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:04:56.733 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:04:56.733 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:04:56.734 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:04:50.073081+0000", 2024-09-18T08:04:56.734 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.021147019 2024-09-18T08:04:56.734 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:04:56.734 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:04:56.734 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "commit_sent", 2024-09-18T08:04:56.734 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:04:50.081021+0000", 2024-09-18T08:04:56.735 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.0079404620000000006 2024-09-18T08:04:56.735 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:04:56.735 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:04:56.735 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "done", 2024-09-18T08:04:56.735 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:04:50.081078+0000", 2024-09-18T08:04:56.735 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 5.7108000000000003e-05 2024-09-18T08:04:56.736 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-09-18T08:04:56.736 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-09-18T08:04:56.736 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-09-18T08:04:56.736 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:04:56.736 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:04:56.737 INFO:teuthology.orchestra.run.smithi007.stdout: "description": "osd_op(client.14805.0:73349 7.a 7:590f9ad0:::benchmark_data_smithi007_73155_object73348:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e104)", 2024-09-18T08:04:56.737 INFO:teuthology.orchestra.run.smithi007.stdout: "initiated_at": "2024-09-18T08:04:50.080359+0000", 2024-09-18T08:04:56.737 INFO:teuthology.orchestra.run.smithi007.stdout: "age": 5.5517318749999998, 2024-09-18T08:04:56.737 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.046959424, 2024-09-18T08:04:56.737 INFO:teuthology.orchestra.run.smithi007.stdout: "continuous": false, 2024-09-18T08:04:56.737 INFO:teuthology.orchestra.run.smithi007.stdout: "type_data": { 2024-09-18T08:04:56.738 INFO:teuthology.orchestra.run.smithi007.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T08:04:56.738 INFO:teuthology.orchestra.run.smithi007.stdout: "client_info": { 2024-09-18T08:04:56.738 INFO:teuthology.orchestra.run.smithi007.stdout: "client": "client.14805", 2024-09-18T08:04:56.738 INFO:teuthology.orchestra.run.smithi007.stdout: "client_addr": "172.21.15.7:0/640320436", 2024-09-18T08:04:56.738 INFO:teuthology.orchestra.run.smithi007.stdout: "tid": 73349 2024-09-18T08:04:56.738 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:04:56.739 INFO:teuthology.orchestra.run.smithi007.stdout: "events": [ 2024-09-18T08:04:56.739 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:04:56.739 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "initiated", 2024-09-18T08:04:56.739 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:04:50.080359+0000", 2024-09-18T08:04:56.739 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-09-18T08:04:56.740 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:04:56.740 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:04:56.740 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "header_read", 2024-09-18T08:04:56.740 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:04:50.080359+0000", 2024-09-18T08:04:56.740 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-09-18T08:04:56.740 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:04:56.741 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:04:56.741 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "throttled", 2024-09-18T08:04:56.741 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:04:50.080361+0000", 2024-09-18T08:04:56.741 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 2.0109999999999999e-06 2024-09-18T08:04:56.741 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:04:56.741 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:04:56.742 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "all_read", 2024-09-18T08:04:56.742 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:04:50.080390+0000", 2024-09-18T08:04:56.742 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 2.8912e-05 2024-09-18T08:04:56.742 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:04:56.742 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:04:56.742 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "dispatched", 2024-09-18T08:04:56.743 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:04:50.080412+0000", 2024-09-18T08:04:56.743 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 2.1787999999999999e-05 2024-09-18T08:04:56.743 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:04:56.743 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:04:56.743 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "queued_for_pg", 2024-09-18T08:04:56.743 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:04:50.080423+0000", 2024-09-18T08:04:56.744 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.137e-05 2024-09-18T08:04:56.744 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:04:56.744 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:04:56.744 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-09-18T08:04:56.744 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:04:50.104109+0000", 2024-09-18T08:04:56.745 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.023686182 2024-09-18T08:04:56.745 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:04:56.745 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:04:56.745 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "started", 2024-09-18T08:04:56.745 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:04:50.104179+0000", 2024-09-18T08:04:56.745 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 6.9441000000000001e-05 2024-09-18T08:04:56.746 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:04:56.746 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:04:56.746 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "waiting for subops from 2,5", 2024-09-18T08:04:56.746 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:04:50.104409+0000", 2024-09-18T08:04:56.746 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.000230509 2024-09-18T08:04:56.746 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:04:56.747 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:04:56.747 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:04:56.747 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:04:50.127134+0000", 2024-09-18T08:04:56.747 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.022724846999999999 2024-09-18T08:04:56.747 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:04:56.747 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:04:56.748 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:04:56.748 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:04:50.127175+0000", 2024-09-18T08:04:56.748 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 4.1193999999999999e-05 2024-09-18T08:04:56.748 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:04:56.748 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:04:56.748 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "op_commit", 2024-09-18T08:04:56.748 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:04:50.127191+0000", 2024-09-18T08:04:56.749 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.5812e-05 2024-09-18T08:04:56.749 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:04:56.749 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:04:56.749 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "commit_sent", 2024-09-18T08:04:56.749 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:04:50.127278+0000", 2024-09-18T08:04:56.749 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 8.7297000000000001e-05 2024-09-18T08:04:56.750 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:04:56.750 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:04:56.750 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "done", 2024-09-18T08:04:56.751 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:04:56 smithi007 ceph-mon[29459]: pgmap v83: 148 pgs: 148 active+clean; 4.7 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 1.2 KiB/s rd, 91 MiB/s wr, 1.45k op/s 2024-09-18T08:04:56.751 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:04:56 smithi007 ceph-mon[29459]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-18T08:04:56.751 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:04:56 smithi007 ceph-mon[29459]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-18T08:04:56.751 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:04:56 smithi007 ceph-mon[29459]: osdmap e105: 8 total, 8 up, 8 in 2024-09-18T08:04:56.752 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:04:56 smithi007 ceph-mon[39249]: pgmap v83: 148 pgs: 148 active+clean; 4.7 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 1.2 KiB/s rd, 91 MiB/s wr, 1.45k op/s 2024-09-18T08:04:56.752 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:04:56 smithi007 ceph-mon[39249]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-18T08:04:56.752 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:04:56 smithi007 ceph-mon[39249]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-18T08:04:56.752 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:04:56 smithi007 ceph-mon[39249]: osdmap e105: 8 total, 8 up, 8 in 2024-09-18T08:04:56.752 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:04:50.127318+0000", 2024-09-18T08:04:56.753 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 4.0061000000000002e-05 2024-09-18T08:04:56.753 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-09-18T08:04:56.753 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-09-18T08:04:56.753 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-09-18T08:04:56.753 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:04:56.753 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:04:56.754 INFO:teuthology.orchestra.run.smithi007.stdout: "description": "osd_op(client.14805.0:73352 7.2 7:4a3308fc:::benchmark_data_smithi007_73155_object73351:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e104)", 2024-09-18T08:04:56.754 INFO:teuthology.orchestra.run.smithi007.stdout: "initiated_at": "2024-09-18T08:04:50.081574+0000", 2024-09-18T08:04:56.754 INFO:teuthology.orchestra.run.smithi007.stdout: "age": 5.5505171400000002, 2024-09-18T08:04:56.754 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.052170671000000002, 2024-09-18T08:04:56.754 INFO:teuthology.orchestra.run.smithi007.stdout: "continuous": false, 2024-09-18T08:04:56.754 INFO:teuthology.orchestra.run.smithi007.stdout: "type_data": { 2024-09-18T08:04:56.755 INFO:teuthology.orchestra.run.smithi007.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T08:04:56.755 INFO:teuthology.orchestra.run.smithi007.stdout: "client_info": { 2024-09-18T08:04:56.755 INFO:teuthology.orchestra.run.smithi007.stdout: "client": "client.14805", 2024-09-18T08:04:56.755 INFO:teuthology.orchestra.run.smithi007.stdout: "client_addr": "172.21.15.7:0/640320436", 2024-09-18T08:04:56.755 INFO:teuthology.orchestra.run.smithi007.stdout: "tid": 73352 2024-09-18T08:04:56.755 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:04:56.756 INFO:teuthology.orchestra.run.smithi007.stdout: "events": [ 2024-09-18T08:04:56.756 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:04:56.756 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "initiated", 2024-09-18T08:04:56.756 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:04:50.081574+0000", 2024-09-18T08:04:56.757 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-09-18T08:04:56.757 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:04:56.757 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:04:56.757 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "header_read", 2024-09-18T08:04:56.757 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:04:50.081574+0000", 2024-09-18T08:04:56.757 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-09-18T08:04:56.757 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:04:56.758 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:04:56.758 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "throttled", 2024-09-18T08:04:56.758 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:04:50.081575+0000", 2024-09-18T08:04:56.758 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.1739999999999999e-06 2024-09-18T08:04:56.758 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:04:56.758 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:04:56.758 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "all_read", 2024-09-18T08:04:56.758 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:04:50.081608+0000", 2024-09-18T08:04:56.759 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 3.2934e-05 2024-09-18T08:04:56.759 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:04:56.759 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:04:56.759 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "dispatched", 2024-09-18T08:04:56.759 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:04:50.081617+0000", 2024-09-18T08:04:56.759 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 9.6549999999999999e-06 2024-09-18T08:04:56.759 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:04:56.760 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:04:56.760 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "queued_for_pg", 2024-09-18T08:04:56.760 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:04:50.081624+0000", 2024-09-18T08:04:56.760 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 7.1280000000000002e-06 2024-09-18T08:04:56.760 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:04:56.760 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:04:56.760 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-09-18T08:04:56.760 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:04:50.104384+0000", 2024-09-18T08:04:56.760 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.022759610999999999 2024-09-18T08:04:56.760 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:04:56.760 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:04:56.760 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "started", 2024-09-18T08:04:56.761 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:04:50.104500+0000", 2024-09-18T08:04:56.761 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.00011586099999999999 2024-09-18T08:04:56.761 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:04:56.761 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:04:56.761 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "waiting for subops from 1,2", 2024-09-18T08:04:56.761 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:04:50.104617+0000", 2024-09-18T08:04:56.761 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.000117353 2024-09-18T08:04:56.761 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:04:56.761 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:04:56.761 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "op_commit", 2024-09-18T08:04:56.761 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:04:50.127186+0000", 2024-09-18T08:04:56.761 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.022569170999999999 2024-09-18T08:04:56.761 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:04:56.762 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:04:56.762 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:04:56.762 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:04:50.127534+0000", 2024-09-18T08:04:56.762 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.00034789199999999998 2024-09-18T08:04:56.762 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:04:56.762 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:04:56.762 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:04:56.762 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:04:50.133519+0000", 2024-09-18T08:04:56.762 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.0059841720000000003 2024-09-18T08:04:56.762 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:04:56.762 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:04:56.762 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "commit_sent", 2024-09-18T08:04:56.762 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:04:50.133699+0000", 2024-09-18T08:04:56.763 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.000180893 2024-09-18T08:04:56.763 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:04:56.763 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:04:56.763 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "done", 2024-09-18T08:04:56.763 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:04:50.133744+0000", 2024-09-18T08:04:56.763 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 4.4827000000000003e-05 2024-09-18T08:04:56.763 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-09-18T08:04:56.763 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-09-18T08:04:56.763 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-09-18T08:04:56.764 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:04:56.764 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:04:56.764 INFO:teuthology.orchestra.run.smithi007.stdout: "description": "osd_op(client.14805.0:79079 7.a 7:50f8b4d8:::benchmark_data_smithi007_73155_object79078:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e104)", 2024-09-18T08:04:56.764 INFO:teuthology.orchestra.run.smithi007.stdout: "initiated_at": "2024-09-18T08:04:55.324119+0000", 2024-09-18T08:04:56.764 INFO:teuthology.orchestra.run.smithi007.stdout: "age": 0.30797184100000002, 2024-09-18T08:04:56.764 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.040254835000000003, 2024-09-18T08:04:56.765 INFO:teuthology.orchestra.run.smithi007.stdout: "continuous": false, 2024-09-18T08:04:56.765 INFO:teuthology.orchestra.run.smithi007.stdout: "type_data": { 2024-09-18T08:04:56.765 INFO:teuthology.orchestra.run.smithi007.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T08:04:56.765 INFO:teuthology.orchestra.run.smithi007.stdout: "client_info": { 2024-09-18T08:04:56.765 INFO:teuthology.orchestra.run.smithi007.stdout: "client": "client.14805", 2024-09-18T08:04:56.765 INFO:teuthology.orchestra.run.smithi007.stdout: "client_addr": "172.21.15.7:0/640320436", 2024-09-18T08:04:56.765 INFO:teuthology.orchestra.run.smithi007.stdout: "tid": 79079 2024-09-18T08:04:56.765 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:04:56.765 INFO:teuthology.orchestra.run.smithi007.stdout: "events": [ 2024-09-18T08:04:56.765 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:04:56.765 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "initiated", 2024-09-18T08:04:56.765 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:04:55.324119+0000", 2024-09-18T08:04:56.765 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-09-18T08:04:56.766 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:04:56.766 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:04:56.766 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "header_read", 2024-09-18T08:04:56.766 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:04:55.324119+0000", 2024-09-18T08:04:56.766 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-09-18T08:04:56.766 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:04:56.766 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:04:56.766 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "throttled", 2024-09-18T08:04:56.766 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:04:55.324120+0000", 2024-09-18T08:04:56.766 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.1149999999999999e-06 2024-09-18T08:04:56.766 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:04:56.766 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:04:56.766 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "all_read", 2024-09-18T08:04:56.767 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:04:55.324264+0000", 2024-09-18T08:04:56.767 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.00014361300000000001 2024-09-18T08:04:56.767 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:04:56.767 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:04:56.767 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "dispatched", 2024-09-18T08:04:56.767 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:04:55.324274+0000", 2024-09-18T08:04:56.767 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.0594999999999999e-05 2024-09-18T08:04:56.767 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:04:56.767 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:04:56.767 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "queued_for_pg", 2024-09-18T08:04:56.767 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:04:55.324281+0000", 2024-09-18T08:04:56.767 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 6.9650000000000002e-06 2024-09-18T08:04:56.767 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:04:56.768 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:04:56.768 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-09-18T08:04:56.768 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:04:55.340639+0000", 2024-09-18T08:04:56.768 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.016357619 2024-09-18T08:04:56.768 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:04:56.768 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:04:56.768 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "started", 2024-09-18T08:04:56.768 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:04:55.340747+0000", 2024-09-18T08:04:56.768 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.000108276 2024-09-18T08:04:56.768 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:04:56.768 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:04:56.768 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "waiting for subops from 2,5", 2024-09-18T08:04:56.768 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:04:55.340889+0000", 2024-09-18T08:04:56.769 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.00014230800000000001 2024-09-18T08:04:56.769 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:04:56.769 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:04:56.769 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:04:56.769 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:04:55.346150+0000", 2024-09-18T08:04:56.769 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.0052606040000000003 2024-09-18T08:04:56.769 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:04:56.769 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:04:56.769 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:04:56.769 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:04:55.346189+0000", 2024-09-18T08:04:56.769 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 3.9325e-05 2024-09-18T08:04:56.769 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:04:56.769 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:04:56.770 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "op_commit", 2024-09-18T08:04:56.770 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:04:55.364075+0000", 2024-09-18T08:04:56.770 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.017885950000000001 2024-09-18T08:04:56.770 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:04:56.770 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:04:56.770 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "commit_sent", 2024-09-18T08:04:56.770 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:04:55.364325+0000", 2024-09-18T08:04:56.770 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.00024976700000000002 2024-09-18T08:04:56.770 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:04:56.770 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:04:56.770 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "done", 2024-09-18T08:04:56.770 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:04:55.364374+0000", 2024-09-18T08:04:56.770 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 4.8698e-05 2024-09-18T08:04:56.770 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-09-18T08:04:56.771 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-09-18T08:04:56.771 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-09-18T08:04:56.771 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:04:56.771 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:04:56.771 INFO:teuthology.orchestra.run.smithi007.stdout: "description": "osd_op(client.14805.0:79084 7.2 7:4e221ac6:::benchmark_data_smithi007_73155_object79083:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e104)", 2024-09-18T08:04:56.771 INFO:teuthology.orchestra.run.smithi007.stdout: "initiated_at": "2024-09-18T08:04:55.328010+0000", 2024-09-18T08:04:56.771 INFO:teuthology.orchestra.run.smithi007.stdout: "age": 0.30408054299999998, 2024-09-18T08:04:56.771 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.043559298000000003, 2024-09-18T08:04:56.771 INFO:teuthology.orchestra.run.smithi007.stdout: "continuous": false, 2024-09-18T08:04:56.771 INFO:teuthology.orchestra.run.smithi007.stdout: "type_data": { 2024-09-18T08:04:56.771 INFO:teuthology.orchestra.run.smithi007.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T08:04:56.772 INFO:teuthology.orchestra.run.smithi007.stdout: "client_info": { 2024-09-18T08:04:56.772 INFO:teuthology.orchestra.run.smithi007.stdout: "client": "client.14805", 2024-09-18T08:04:56.772 INFO:teuthology.orchestra.run.smithi007.stdout: "client_addr": "172.21.15.7:0/640320436", 2024-09-18T08:04:56.772 INFO:teuthology.orchestra.run.smithi007.stdout: "tid": 79084 2024-09-18T08:04:56.772 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:04:56.772 INFO:teuthology.orchestra.run.smithi007.stdout: "events": [ 2024-09-18T08:04:56.772 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:04:56.772 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "initiated", 2024-09-18T08:04:56.772 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:04:55.328010+0000", 2024-09-18T08:04:56.772 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-09-18T08:04:56.772 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:04:56.772 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:04:56.772 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "header_read", 2024-09-18T08:04:56.772 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:04:55.328010+0000", 2024-09-18T08:04:56.772 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-09-18T08:04:56.773 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:04:56.773 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:04:56.773 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "throttled", 2024-09-18T08:04:56.773 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:04:55.328011+0000", 2024-09-18T08:04:56.773 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.108e-06 2024-09-18T08:04:56.773 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:04:56.773 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:04:56.773 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "all_read", 2024-09-18T08:04:56.773 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:04:55.328032+0000", 2024-09-18T08:04:56.773 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 2.0979e-05 2024-09-18T08:04:56.773 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:04:56.774 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:04:56.774 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "dispatched", 2024-09-18T08:04:56.774 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:04:55.328045+0000", 2024-09-18T08:04:56.774 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.2672e-05 2024-09-18T08:04:56.774 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:04:56.774 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:04:56.774 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "queued_for_pg", 2024-09-18T08:04:56.774 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:04:55.328052+0000", 2024-09-18T08:04:56.774 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 7.4150000000000002e-06 2024-09-18T08:04:56.774 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:04:56.774 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:04:56.775 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-09-18T08:04:56.775 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:04:55.343325+0000", 2024-09-18T08:04:56.775 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.015273042000000001 2024-09-18T08:04:56.775 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:04:56.775 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:04:56.775 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "started", 2024-09-18T08:04:56.775 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:04:55.343468+0000", 2024-09-18T08:04:56.775 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.00014293199999999999 2024-09-18T08:04:56.775 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:04:56.775 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:04:56.775 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "waiting for subops from 1,2", 2024-09-18T08:04:56.775 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:04:55.343627+0000", 2024-09-18T08:04:56.776 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.00015910699999999999 2024-09-18T08:04:56.776 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:04:56.776 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:04:56.776 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:04:56.776 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:04:55.359801+0000", 2024-09-18T08:04:56.776 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.016173237 2024-09-18T08:04:56.776 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:04:56.776 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:04:56.776 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:04:56.776 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:04:55.363655+0000", 2024-09-18T08:04:56.776 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.0038546029999999998 2024-09-18T08:04:56.776 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:04:56.776 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:04:56.777 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "op_commit", 2024-09-18T08:04:56.777 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:04:55.371474+0000", 2024-09-18T08:04:56.777 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.0078182740000000001 2024-09-18T08:04:56.777 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:04:56.777 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:04:56.777 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "commit_sent", 2024-09-18T08:04:56.777 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:04:55.371524+0000", 2024-09-18T08:04:56.777 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 4.9972000000000002e-05 2024-09-18T08:04:56.777 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:04:56.777 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:04:56.777 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "done", 2024-09-18T08:04:56.778 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:04:55.371569+0000", 2024-09-18T08:04:56.778 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 4.5957000000000001e-05 2024-09-18T08:04:56.778 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-09-18T08:04:56.778 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-09-18T08:04:56.778 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-09-18T08:04:56.778 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-09-18T08:04:56.778 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-09-18T08:04:56.778 INFO:teuthology.orchestra.run.smithi007.stdout:} 2024-09-18T08:04:56.779 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6 2024-09-18T08:04:56.809 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:04:56.809 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:04:56.810 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6' 2024-09-18T08:04:56.906 INFO:journalctl@ceph.prometheus.a.smithi050.stdout:Sep 18 08:04:56 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-prometheus-a[58694]: ts=2024-09-18T08:04:56.590Z caller=main.go:964 level=warn msg="Received SIGTERM, exiting gracefully..." 2024-09-18T08:04:56.906 INFO:journalctl@ceph.prometheus.a.smithi050.stdout:Sep 18 08:04:56 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-prometheus-a[58694]: ts=2024-09-18T08:04:56.590Z caller=main.go:988 level=info msg="Stopping scrape discovery manager..." 2024-09-18T08:04:56.906 INFO:journalctl@ceph.prometheus.a.smithi050.stdout:Sep 18 08:04:56 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-prometheus-a[58694]: ts=2024-09-18T08:04:56.590Z caller=main.go:1002 level=info msg="Stopping notify discovery manager..." 2024-09-18T08:04:56.907 INFO:journalctl@ceph.prometheus.a.smithi050.stdout:Sep 18 08:04:56 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-prometheus-a[58694]: ts=2024-09-18T08:04:56.590Z caller=manager.go:177 level=info component="rule manager" msg="Stopping rule manager..." 2024-09-18T08:04:56.907 INFO:journalctl@ceph.prometheus.a.smithi050.stdout:Sep 18 08:04:56 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-prometheus-a[58694]: ts=2024-09-18T08:04:56.590Z caller=manager.go:187 level=info component="rule manager" msg="Rule manager stopped" 2024-09-18T08:04:56.907 INFO:journalctl@ceph.prometheus.a.smithi050.stdout:Sep 18 08:04:56 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-prometheus-a[58694]: ts=2024-09-18T08:04:56.590Z caller=main.go:1039 level=info msg="Stopping scrape manager..." 2024-09-18T08:04:56.907 INFO:journalctl@ceph.prometheus.a.smithi050.stdout:Sep 18 08:04:56 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-prometheus-a[58694]: ts=2024-09-18T08:04:56.590Z caller=main.go:998 level=info msg="Notify discovery manager stopped" 2024-09-18T08:04:56.907 INFO:journalctl@ceph.prometheus.a.smithi050.stdout:Sep 18 08:04:56 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-prometheus-a[58694]: ts=2024-09-18T08:04:56.590Z caller=main.go:984 level=info msg="Scrape discovery manager stopped" 2024-09-18T08:04:56.907 INFO:journalctl@ceph.prometheus.a.smithi050.stdout:Sep 18 08:04:56 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-prometheus-a[58694]: ts=2024-09-18T08:04:56.590Z caller=main.go:1031 level=info msg="Scrape manager stopped" 2024-09-18T08:04:56.907 INFO:journalctl@ceph.prometheus.a.smithi050.stdout:Sep 18 08:04:56 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-prometheus-a[58694]: ts=2024-09-18T08:04:56.591Z caller=notifier.go:618 level=info component=notifier msg="Stopping notification manager..." 2024-09-18T08:04:56.907 INFO:journalctl@ceph.prometheus.a.smithi050.stdout:Sep 18 08:04:56 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-prometheus-a[58694]: ts=2024-09-18T08:04:56.591Z caller=main.go:1261 level=info msg="Notifier manager stopped" 2024-09-18T08:04:56.907 INFO:journalctl@ceph.prometheus.a.smithi050.stdout:Sep 18 08:04:56 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-prometheus-a[58694]: ts=2024-09-18T08:04:56.591Z caller=main.go:1273 level=info msg="See you next time!" 2024-09-18T08:04:56.907 INFO:journalctl@ceph.prometheus.a.smithi050.stdout:Sep 18 08:04:56 smithi050 podman[65350]: 2024-09-18 08:04:56.600514202 +0000 UTC m=+0.137333621 container died 4f01531f2bb3ce7e6b7b5bb3adeb4fd754108415006149e95fe4eb669369892d (image=quay.io/prometheus/prometheus:v2.51.0, name=ceph-779349ca-7593-11ef-bceb-c7b262605968-prometheus-a, maintainer=The Prometheus Authors ) 2024-09-18T08:04:56.911 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0 2024-09-18T08:04:56.948 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:04:56.948 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:04:56.949 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0' 2024-09-18T08:04:57.050 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4 2024-09-18T08:04:57.081 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:04:57.082 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:04:57.082 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4' 2024-09-18T08:04:57.183 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6 2024-09-18T08:04:57.211 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:04:57.211 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:04:57.212 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6' 2024-09-18T08:04:57.276 INFO:journalctl@ceph.prometheus.a.smithi050.stdout:Sep 18 08:04:56 smithi050 podman[65350]: 2024-09-18 08:04:56.997707738 +0000 UTC m=+0.534527150 container cleanup 4f01531f2bb3ce7e6b7b5bb3adeb4fd754108415006149e95fe4eb669369892d (image=quay.io/prometheus/prometheus:v2.51.0, name=ceph-779349ca-7593-11ef-bceb-c7b262605968-prometheus-a, maintainer=The Prometheus Authors ) 2024-09-18T08:04:57.276 INFO:journalctl@ceph.prometheus.a.smithi050.stdout:Sep 18 08:04:56 smithi050 bash[65350]: ceph-779349ca-7593-11ef-bceb-c7b262605968-prometheus-a 2024-09-18T08:04:57.312 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5 2024-09-18T08:04:57.343 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:04:57.343 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:04:57.344 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5' 2024-09-18T08:04:57.445 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4 2024-09-18T08:04:57.471 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:04:57.472 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:04:57.472 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4' 2024-09-18T08:04:57.573 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5 2024-09-18T08:04:57.583 INFO:tasks.radosbench.radosbench.0.smithi007.stdout: 52 16 81691 81675 98.1373 73.1875 0.00780955 0.0101842 2024-09-18T08:04:57.603 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:04:57.603 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:04:57.604 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5' 2024-09-18T08:04:57.619 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:04:57 smithi007 ceph-mon[29459]: purged_snaps scrub starts 2024-09-18T08:04:57.619 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:04:57 smithi007 ceph-mon[29459]: purged_snaps scrub ok 2024-09-18T08:04:57.619 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:04:57 smithi007 ceph-mon[29459]: 5.b scrub starts 2024-09-18T08:04:57.619 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:04:57 smithi007 ceph-mon[29459]: 5.b scrub ok 2024-09-18T08:04:57.619 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:04:57 smithi007 ceph-mon[29459]: osdmap e106: 8 total, 8 up, 8 in 2024-09-18T08:04:57.619 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:04:57 smithi007 ceph-mon[29459]: 4.1 scrub starts 2024-09-18T08:04:57.619 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:04:57 smithi007 ceph-mon[29459]: 4.1 scrub ok 2024-09-18T08:04:57.622 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:04:57 smithi007 ceph-mon[39249]: purged_snaps scrub starts 2024-09-18T08:04:57.622 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:04:57 smithi007 ceph-mon[39249]: purged_snaps scrub ok 2024-09-18T08:04:57.622 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:04:57 smithi007 ceph-mon[39249]: 5.b scrub starts 2024-09-18T08:04:57.622 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:04:57 smithi007 ceph-mon[39249]: 5.b scrub ok 2024-09-18T08:04:57.622 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:04:57 smithi007 ceph-mon[39249]: osdmap e106: 8 total, 8 up, 8 in 2024-09-18T08:04:57.622 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:04:57 smithi007 ceph-mon[39249]: 4.1 scrub starts 2024-09-18T08:04:57.622 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:04:57 smithi007 ceph-mon[39249]: 4.1 scrub ok 2024-09-18T08:04:57.656 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:04:57 smithi050 ceph-mon[32371]: purged_snaps scrub starts 2024-09-18T08:04:57.656 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:04:57 smithi050 ceph-mon[32371]: purged_snaps scrub ok 2024-09-18T08:04:57.656 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:04:57 smithi050 ceph-mon[32371]: 5.b scrub starts 2024-09-18T08:04:57.656 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:04:57 smithi050 ceph-mon[32371]: 5.b scrub ok 2024-09-18T08:04:57.656 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:04:57 smithi050 ceph-mon[32371]: osdmap e106: 8 total, 8 up, 8 in 2024-09-18T08:04:57.656 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:04:57 smithi050 ceph-mon[32371]: 4.1 scrub starts 2024-09-18T08:04:57.656 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:04:57 smithi050 ceph-mon[32371]: 4.1 scrub ok 2024-09-18T08:04:57.656 INFO:journalctl@ceph.prometheus.a.smithi050.stdout:Sep 18 08:04:57 smithi050 podman[65379]: 2024-09-18 08:04:57.274939438 +0000 UTC m=+0.670645524 container remove 4f01531f2bb3ce7e6b7b5bb3adeb4fd754108415006149e95fe4eb669369892d (image=quay.io/prometheus/prometheus:v2.51.0, name=ceph-779349ca-7593-11ef-bceb-c7b262605968-prometheus-a, maintainer=The Prometheus Authors ) 2024-09-18T08:04:57.657 INFO:journalctl@ceph.prometheus.a.smithi050.stdout:Sep 18 08:04:57 smithi050 systemd[1]: ceph-779349ca-7593-11ef-bceb-c7b262605968@prometheus.a.service: Deactivated successfully. 2024-09-18T08:04:57.657 INFO:journalctl@ceph.prometheus.a.smithi050.stdout:Sep 18 08:04:57 smithi050 systemd[1]: Stopped Ceph prometheus.a for 779349ca-7593-11ef-bceb-c7b262605968. 2024-09-18T08:04:57.657 INFO:journalctl@ceph.prometheus.a.smithi050.stdout:Sep 18 08:04:57 smithi050 systemd[1]: ceph-779349ca-7593-11ef-bceb-c7b262605968@prometheus.a.service: Consumed 1.245s CPU time. 2024-09-18T08:04:57.657 INFO:journalctl@ceph.prometheus.a.smithi050.stdout:Sep 18 08:04:57 smithi050 systemd[1]: Starting Ceph prometheus.a for 779349ca-7593-11ef-bceb-c7b262605968... 2024-09-18T08:04:57.705 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0 2024-09-18T08:04:57.742 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:04:57.742 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:04:57.743 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0' 2024-09-18T08:04:57.844 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3 2024-09-18T08:04:57.887 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:04:57.888 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:04:57.888 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3' 2024-09-18T08:04:57.963 INFO:journalctl@ceph.prometheus.a.smithi050.stdout:Sep 18 08:04:57 smithi050 podman[65551]: 2024-09-18 08:04:57.607475744 +0000 UTC m=+0.026782190 image pull 1d3b7f56885b6dd623f1785be963aa9c195f86bc256ea454e8d02a7980b79c53 quay.io/prometheus/prometheus:v2.51.0 2024-09-18T08:04:57.963 INFO:journalctl@ceph.prometheus.a.smithi050.stdout:Sep 18 08:04:57 smithi050 podman[65551]: 2024-09-18 08:04:57.743092561 +0000 UTC m=+0.162399008 container create 32d3202e5a25373bca086a0ca9811cc9215c296997ae96fb8f32b6507b0bb3a6 (image=quay.io/prometheus/prometheus:v2.51.0, name=ceph-779349ca-7593-11ef-bceb-c7b262605968-prometheus-a, maintainer=The Prometheus Authors ) 2024-09-18T08:04:57.963 INFO:journalctl@ceph.prometheus.a.smithi050.stdout:Sep 18 08:04:57 smithi050 podman[65551]: 2024-09-18 08:04:57.962340547 +0000 UTC m=+0.381647002 container init 32d3202e5a25373bca086a0ca9811cc9215c296997ae96fb8f32b6507b0bb3a6 (image=quay.io/prometheus/prometheus:v2.51.0, name=ceph-779349ca-7593-11ef-bceb-c7b262605968-prometheus-a, maintainer=The Prometheus Authors ) 2024-09-18T08:04:57.989 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1 2024-09-18T08:04:57.993 DEBUG:teuthology.orchestra.run.smithi050:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid 779349ca-7593-11ef-bceb-c7b262605968 -- ceph daemon osd.4 dump_ops_in_flight 2024-09-18T08:04:58.030 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:04:58.031 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:04:58.031 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1' 2024-09-18T08:04:58.132 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7 2024-09-18T08:04:58.168 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:04:58.169 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:04:58.169 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7' 2024-09-18T08:04:58.195 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-09-18T08:04:58.195 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-18T08:04:58.196 INFO:tasks.thrashosds.thrasher:inject_pause on osd.5 2024-09-18T08:04:58.196 INFO:tasks.thrashosds.thrasher:Testing filestore_inject_stall pause injection for duration 3 2024-09-18T08:04:58.196 INFO:tasks.thrashosds.thrasher:Checking after 0, should_be_down=False 2024-09-18T08:04:58.196 DEBUG:teuthology.orchestra.run.smithi050:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid 779349ca-7593-11ef-bceb-c7b262605968 -- ceph daemon osd.5 config set filestore_inject_stall 3 2024-09-18T08:04:58.243 INFO:journalctl@ceph.prometheus.a.smithi050.stdout:Sep 18 08:04:57 smithi050 podman[65551]: 2024-09-18 08:04:57.966009702 +0000 UTC m=+0.385316150 container start 32d3202e5a25373bca086a0ca9811cc9215c296997ae96fb8f32b6507b0bb3a6 (image=quay.io/prometheus/prometheus:v2.51.0, name=ceph-779349ca-7593-11ef-bceb-c7b262605968-prometheus-a, maintainer=The Prometheus Authors ) 2024-09-18T08:04:58.243 INFO:journalctl@ceph.prometheus.a.smithi050.stdout:Sep 18 08:04:57 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-prometheus-a[65581]: ts=2024-09-18T08:04:57.991Z caller=main.go:617 level=info msg="Starting Prometheus Server" mode=server version="(version=2.51.0, branch=HEAD, revision=c05c15512acb675e3f6cd662a6727854e93fc024)" 2024-09-18T08:04:58.243 INFO:journalctl@ceph.prometheus.a.smithi050.stdout:Sep 18 08:04:57 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-prometheus-a[65581]: ts=2024-09-18T08:04:57.991Z caller=main.go:622 level=info build_context="(go=go1.22.1, platform=linux/amd64, user=root@b5723e458358, date=20240319-10:54:45, tags=netgo,builtinassets,stringlabels)" 2024-09-18T08:04:58.243 INFO:journalctl@ceph.prometheus.a.smithi050.stdout:Sep 18 08:04:57 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-prometheus-a[65581]: ts=2024-09-18T08:04:57.991Z caller=main.go:623 level=info host_details="(Linux 5.14.0-508.el9.x86_64 #1 SMP PREEMPT_DYNAMIC Thu Sep 12 15:49:37 UTC 2024 x86_64 smithi050 (none))" 2024-09-18T08:04:58.244 INFO:journalctl@ceph.prometheus.a.smithi050.stdout:Sep 18 08:04:57 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-prometheus-a[65581]: ts=2024-09-18T08:04:57.991Z caller=main.go:624 level=info fd_limits="(soft=1048576, hard=1048576)" 2024-09-18T08:04:58.244 INFO:journalctl@ceph.prometheus.a.smithi050.stdout:Sep 18 08:04:57 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-prometheus-a[65581]: ts=2024-09-18T08:04:57.991Z caller=main.go:625 level=info vm_limits="(soft=unlimited, hard=unlimited)" 2024-09-18T08:04:58.244 INFO:journalctl@ceph.prometheus.a.smithi050.stdout:Sep 18 08:04:57 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-prometheus-a[65581]: ts=2024-09-18T08:04:57.993Z caller=web.go:568 level=info component=web msg="Start listening for connections" address=:9095 2024-09-18T08:04:58.244 INFO:journalctl@ceph.prometheus.a.smithi050.stdout:Sep 18 08:04:57 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-prometheus-a[65581]: ts=2024-09-18T08:04:57.994Z caller=main.go:1129 level=info msg="Starting TSDB ..." 2024-09-18T08:04:58.244 INFO:journalctl@ceph.prometheus.a.smithi050.stdout:Sep 18 08:04:57 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-prometheus-a[65581]: ts=2024-09-18T08:04:57.997Z caller=tls_config.go:313 level=info component=web msg="Listening on" address=[::]:9095 2024-09-18T08:04:58.244 INFO:journalctl@ceph.prometheus.a.smithi050.stdout:Sep 18 08:04:57 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-prometheus-a[65581]: ts=2024-09-18T08:04:57.997Z caller=tls_config.go:316 level=info component=web msg="TLS is disabled." http2=false address=[::]:9095 2024-09-18T08:04:58.244 INFO:journalctl@ceph.prometheus.a.smithi050.stdout:Sep 18 08:04:57 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-prometheus-a[65581]: ts=2024-09-18T08:04:57.999Z caller=head.go:616 level=info component=tsdb msg="Replaying on-disk memory mappable chunks if any" 2024-09-18T08:04:58.244 INFO:journalctl@ceph.prometheus.a.smithi050.stdout:Sep 18 08:04:58 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-prometheus-a[65581]: ts=2024-09-18T08:04:57.999Z caller=head.go:698 level=info component=tsdb msg="On-disk memory mappable chunks replay completed" duration=2.772µs 2024-09-18T08:04:58.244 INFO:journalctl@ceph.prometheus.a.smithi050.stdout:Sep 18 08:04:58 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-prometheus-a[65581]: ts=2024-09-18T08:04:57.999Z caller=head.go:706 level=info component=tsdb msg="Replaying WAL, this may take a while" 2024-09-18T08:04:58.244 INFO:journalctl@ceph.prometheus.a.smithi050.stdout:Sep 18 08:04:58 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-prometheus-a[65581]: ts=2024-09-18T08:04:58.000Z caller=head.go:778 level=info component=tsdb msg="WAL segment loaded" segment=0 maxSegment=1 2024-09-18T08:04:58.244 INFO:journalctl@ceph.prometheus.a.smithi050.stdout:Sep 18 08:04:58 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-prometheus-a[65581]: ts=2024-09-18T08:04:58.000Z caller=head.go:778 level=info component=tsdb msg="WAL segment loaded" segment=1 maxSegment=1 2024-09-18T08:04:58.244 INFO:journalctl@ceph.prometheus.a.smithi050.stdout:Sep 18 08:04:58 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-prometheus-a[65581]: ts=2024-09-18T08:04:58.000Z caller=head.go:815 level=info component=tsdb msg="WAL replay completed" checkpoint_replay_duration=40.708µs wal_replay_duration=692.92µs wbl_replay_duration=80ns total_replay_duration=780.814µs 2024-09-18T08:04:58.244 INFO:journalctl@ceph.prometheus.a.smithi050.stdout:Sep 18 08:04:58 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-prometheus-a[65581]: ts=2024-09-18T08:04:58.003Z caller=main.go:1150 level=info fs_type=XFS_SUPER_MAGIC 2024-09-18T08:04:58.245 INFO:journalctl@ceph.prometheus.a.smithi050.stdout:Sep 18 08:04:58 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-prometheus-a[65581]: ts=2024-09-18T08:04:58.003Z caller=main.go:1153 level=info msg="TSDB started" 2024-09-18T08:04:58.245 INFO:journalctl@ceph.prometheus.a.smithi050.stdout:Sep 18 08:04:58 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-prometheus-a[65581]: ts=2024-09-18T08:04:58.003Z caller=main.go:1335 level=info msg="Loading configuration file" filename=/etc/prometheus/prometheus.yml 2024-09-18T08:04:58.245 INFO:journalctl@ceph.prometheus.a.smithi050.stdout:Sep 18 08:04:58 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-prometheus-a[65581]: ts=2024-09-18T08:04:58.026Z caller=main.go:1372 level=info msg="Completed loading of configuration file" filename=/etc/prometheus/prometheus.yml totalDuration=22.984563ms db_storage=902ns remote_storage=1.216µs web_handler=364ns query_engine=621ns scrape=555.036µs scrape_sd=171.269µs notify=19.186µs notify_sd=14.568µs rules=21.607777ms tracing=9.843µs 2024-09-18T08:04:58.245 INFO:journalctl@ceph.prometheus.a.smithi050.stdout:Sep 18 08:04:58 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-prometheus-a[65581]: ts=2024-09-18T08:04:58.026Z caller=manager.go:163 level=info component="rule manager" msg="Starting rule manager..." 2024-09-18T08:04:58.245 INFO:journalctl@ceph.prometheus.a.smithi050.stdout:Sep 18 08:04:58 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-prometheus-a[65581]: ts=2024-09-18T08:04:58.026Z caller=main.go:1114 level=info msg="Server is ready to receive web requests." 2024-09-18T08:04:58.245 INFO:journalctl@ceph.prometheus.a.smithi050.stdout:Sep 18 08:04:58 smithi050 bash[65551]: 32d3202e5a25373bca086a0ca9811cc9215c296997ae96fb8f32b6507b0bb3a6 2024-09-18T08:04:58.245 INFO:journalctl@ceph.prometheus.a.smithi050.stdout:Sep 18 08:04:58 smithi050 systemd[1]: Started Ceph prometheus.a for 779349ca-7593-11ef-bceb-c7b262605968. 2024-09-18T08:04:58.269 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3 2024-09-18T08:04:58.314 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:04:58.314 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:04:58.315 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3' 2024-09-18T08:04:58.405 INFO:teuthology.orchestra.run.smithi050.stderr:Inferring config /var/lib/ceph/779349ca-7593-11ef-bceb-c7b262605968/mon.b/config 2024-09-18T08:04:58.415 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4 2024-09-18T08:04:58.457 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:04:58.457 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:04:58.458 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4' 2024-09-18T08:04:58.481 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:04:58 smithi007 ceph-mon[29459]: 3.1e scrub starts 2024-09-18T08:04:58.481 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:04:58 smithi007 ceph-mon[29459]: 3.1e scrub ok 2024-09-18T08:04:58.481 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:04:58 smithi007 ceph-mon[29459]: pgmap v86: 148 pgs: 148 active+clean; 4.7 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 639 B/s rd, 45 MiB/s wr, 718 op/s 2024-09-18T08:04:58.481 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:04:58 smithi007 ceph-mon[29459]: from='mgr.24470 172.21.15.7:0/2594390261' entity='mgr.y' 2024-09-18T08:04:58.481 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:04:58 smithi007 ceph-mon[29459]: from='mgr.24470 172.21.15.7:0/2594390261' entity='mgr.y' 2024-09-18T08:04:58.481 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:04:58 smithi007 ceph-mon[29459]: from='mgr.24470 172.21.15.7:0/2594390261' entity='mgr.y' cmd={"prefix": "dashboard get-alertmanager-api-host"} : dispatch 2024-09-18T08:04:58.481 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:04:58 smithi007 ceph-mon[29459]: from='mgr.24470 172.21.15.7:0/2594390261' entity='mgr.y' cmd={"prefix": "dashboard set-alertmanager-api-host", "value": "http://smithi007.front.sepia.ceph.com:9093"} : dispatch 2024-09-18T08:04:58.481 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:04:58 smithi007 ceph-mon[29459]: from='mgr.24470 172.21.15.7:0/2594390261' entity='mgr.y' 2024-09-18T08:04:58.481 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:04:58 smithi007 ceph-mon[29459]: from='mgr.24470 172.21.15.7:0/2594390261' entity='mgr.y' cmd={"prefix": "dashboard get-prometheus-api-host"} : dispatch 2024-09-18T08:04:58.481 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:04:58 smithi007 ceph-mon[29459]: from='mgr.24470 172.21.15.7:0/2594390261' entity='mgr.y' cmd={"prefix": "dashboard set-prometheus-api-host", "value": "http://smithi050.front.sepia.ceph.com:9095"} : dispatch 2024-09-18T08:04:58.481 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:04:58 smithi007 ceph-mon[29459]: from='mgr.24470 172.21.15.7:0/2594390261' entity='mgr.y' 2024-09-18T08:04:58.481 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:04:58 smithi007 ceph-mon[29459]: from='mgr.24470 172.21.15.7:0/2594390261' entity='mgr.y' cmd={"prefix": "dashboard get-grafana-api-url"} : dispatch 2024-09-18T08:04:58.482 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:04:58 smithi007 ceph-mon[29459]: from='mgr.24470 172.21.15.7:0/2594390261' entity='mgr.y' cmd={"prefix": "dashboard set-grafana-api-url", "value": "https://smithi050.front.sepia.ceph.com:3000"} : dispatch 2024-09-18T08:04:58.482 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:04:58 smithi007 ceph-mon[29459]: from='mgr.24470 172.21.15.7:0/2594390261' entity='mgr.y' 2024-09-18T08:04:58.482 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Sep 18 08:04:58 smithi007 ceph-779349ca-7593-11ef-bceb-c7b262605968-mgr-y[29688]: [18/Sep/2024:08:04:58] ENGINE Bus STOPPING 2024-09-18T08:04:58.482 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:04:58 smithi007 ceph-mon[39249]: 3.1e scrub starts 2024-09-18T08:04:58.482 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:04:58 smithi007 ceph-mon[39249]: 3.1e scrub ok 2024-09-18T08:04:58.482 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:04:58 smithi007 ceph-mon[39249]: pgmap v86: 148 pgs: 148 active+clean; 4.7 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 639 B/s rd, 45 MiB/s wr, 718 op/s 2024-09-18T08:04:58.482 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:04:58 smithi007 ceph-mon[39249]: from='mgr.24470 172.21.15.7:0/2594390261' entity='mgr.y' 2024-09-18T08:04:58.483 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:04:58 smithi007 ceph-mon[39249]: from='mgr.24470 172.21.15.7:0/2594390261' entity='mgr.y' 2024-09-18T08:04:58.483 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:04:58 smithi007 ceph-mon[39249]: from='mgr.24470 172.21.15.7:0/2594390261' entity='mgr.y' cmd={"prefix": "dashboard get-alertmanager-api-host"} : dispatch 2024-09-18T08:04:58.483 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:04:58 smithi007 ceph-mon[39249]: from='mgr.24470 172.21.15.7:0/2594390261' entity='mgr.y' cmd={"prefix": "dashboard set-alertmanager-api-host", "value": "http://smithi007.front.sepia.ceph.com:9093"} : dispatch 2024-09-18T08:04:58.483 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:04:58 smithi007 ceph-mon[39249]: from='mgr.24470 172.21.15.7:0/2594390261' entity='mgr.y' 2024-09-18T08:04:58.483 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:04:58 smithi007 ceph-mon[39249]: from='mgr.24470 172.21.15.7:0/2594390261' entity='mgr.y' cmd={"prefix": "dashboard get-prometheus-api-host"} : dispatch 2024-09-18T08:04:58.483 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:04:58 smithi007 ceph-mon[39249]: from='mgr.24470 172.21.15.7:0/2594390261' entity='mgr.y' cmd={"prefix": "dashboard set-prometheus-api-host", "value": "http://smithi050.front.sepia.ceph.com:9095"} : dispatch 2024-09-18T08:04:58.483 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:04:58 smithi007 ceph-mon[39249]: from='mgr.24470 172.21.15.7:0/2594390261' entity='mgr.y' 2024-09-18T08:04:58.483 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:04:58 smithi007 ceph-mon[39249]: from='mgr.24470 172.21.15.7:0/2594390261' entity='mgr.y' cmd={"prefix": "dashboard get-grafana-api-url"} : dispatch 2024-09-18T08:04:58.483 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:04:58 smithi007 ceph-mon[39249]: from='mgr.24470 172.21.15.7:0/2594390261' entity='mgr.y' cmd={"prefix": "dashboard set-grafana-api-url", "value": "https://smithi050.front.sepia.ceph.com:3000"} : dispatch 2024-09-18T08:04:58.483 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:04:58 smithi007 ceph-mon[39249]: from='mgr.24470 172.21.15.7:0/2594390261' entity='mgr.y' 2024-09-18T08:04:58.514 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid 779349ca-7593-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-18T08:04:58.536 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:04:58 smithi050 ceph-mon[32371]: 3.1e scrub starts 2024-09-18T08:04:58.537 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:04:58 smithi050 ceph-mon[32371]: 3.1e scrub ok 2024-09-18T08:04:58.537 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:04:58 smithi050 ceph-mon[32371]: pgmap v86: 148 pgs: 148 active+clean; 4.7 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 639 B/s rd, 45 MiB/s wr, 718 op/s 2024-09-18T08:04:58.537 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:04:58 smithi050 ceph-mon[32371]: from='mgr.24470 172.21.15.7:0/2594390261' entity='mgr.y' 2024-09-18T08:04:58.537 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:04:58 smithi050 ceph-mon[32371]: from='mgr.24470 172.21.15.7:0/2594390261' entity='mgr.y' 2024-09-18T08:04:58.537 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:04:58 smithi050 ceph-mon[32371]: from='mgr.24470 172.21.15.7:0/2594390261' entity='mgr.y' cmd={"prefix": "dashboard get-alertmanager-api-host"} : dispatch 2024-09-18T08:04:58.537 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:04:58 smithi050 ceph-mon[32371]: from='mgr.24470 172.21.15.7:0/2594390261' entity='mgr.y' cmd={"prefix": "dashboard set-alertmanager-api-host", "value": "http://smithi007.front.sepia.ceph.com:9093"} : dispatch 2024-09-18T08:04:58.537 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:04:58 smithi050 ceph-mon[32371]: from='mgr.24470 172.21.15.7:0/2594390261' entity='mgr.y' 2024-09-18T08:04:58.537 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:04:58 smithi050 ceph-mon[32371]: from='mgr.24470 172.21.15.7:0/2594390261' entity='mgr.y' cmd={"prefix": "dashboard get-prometheus-api-host"} : dispatch 2024-09-18T08:04:58.537 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:04:58 smithi050 ceph-mon[32371]: from='mgr.24470 172.21.15.7:0/2594390261' entity='mgr.y' cmd={"prefix": "dashboard set-prometheus-api-host", "value": "http://smithi050.front.sepia.ceph.com:9095"} : dispatch 2024-09-18T08:04:58.537 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:04:58 smithi050 ceph-mon[32371]: from='mgr.24470 172.21.15.7:0/2594390261' entity='mgr.y' 2024-09-18T08:04:58.537 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:04:58 smithi050 ceph-mon[32371]: from='mgr.24470 172.21.15.7:0/2594390261' entity='mgr.y' cmd={"prefix": "dashboard get-grafana-api-url"} : dispatch 2024-09-18T08:04:58.537 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:04:58 smithi050 ceph-mon[32371]: from='mgr.24470 172.21.15.7:0/2594390261' entity='mgr.y' cmd={"prefix": "dashboard set-grafana-api-url", "value": "https://smithi050.front.sepia.ceph.com:3000"} : dispatch 2024-09-18T08:04:58.538 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:04:58 smithi050 ceph-mon[32371]: from='mgr.24470 172.21.15.7:0/2594390261' entity='mgr.y' 2024-09-18T08:04:58.559 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4 2024-09-18T08:04:58.583 INFO:tasks.radosbench.radosbench.0.smithi007.stdout: 53 16 82634 82618 97.3977 58.9375 0.0426388 0.0102612 2024-09-18T08:04:58.585 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:04:58.586 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:04:58.586 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4' 2024-09-18T08:04:58.687 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6 2024-09-18T08:04:58.718 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:04:58.718 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:04:58.719 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6' 2024-09-18T08:04:58.820 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7 2024-09-18T08:04:58.847 INFO:journalctl@ceph.iscsi.iscsi.a.smithi050.stdout:Sep 18 08:04:58 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-iscsi-iscsi-a[57782]: debug there is no tcmu-runner data available 2024-09-18T08:04:58.848 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:04:58.849 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:04:58.849 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7' 2024-09-18T08:04:58.950 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0 2024-09-18T08:04:58.980 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Sep 18 08:04:58 smithi007 ceph-779349ca-7593-11ef-bceb-c7b262605968-mgr-y[29688]: [18/Sep/2024:08:04:58] ENGINE HTTP Server cherrypy._cpwsgi_server.CPWSGIServer(('::', 9283)) shut down 2024-09-18T08:04:58.981 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Sep 18 08:04:58 smithi007 ceph-779349ca-7593-11ef-bceb-c7b262605968-mgr-y[29688]: [18/Sep/2024:08:04:58] ENGINE Bus STOPPED 2024-09-18T08:04:58.981 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Sep 18 08:04:58 smithi007 ceph-779349ca-7593-11ef-bceb-c7b262605968-mgr-y[29688]: [18/Sep/2024:08:04:58] ENGINE Bus STARTING 2024-09-18T08:04:58.981 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Sep 18 08:04:58 smithi007 ceph-779349ca-7593-11ef-bceb-c7b262605968-mgr-y[29688]: [18/Sep/2024:08:04:58] ENGINE Serving on http://:::9283 2024-09-18T08:04:58.981 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Sep 18 08:04:58 smithi007 ceph-779349ca-7593-11ef-bceb-c7b262605968-mgr-y[29688]: [18/Sep/2024:08:04:58] ENGINE Bus STARTED 2024-09-18T08:04:58.981 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Sep 18 08:04:58 smithi007 ceph-779349ca-7593-11ef-bceb-c7b262605968-mgr-y[29688]: [18/Sep/2024:08:04:58] ENGINE Bus STOPPING 2024-09-18T08:04:58.990 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2024-09-18T08:04:58.990 INFO:teuthology.orchestra.run.smithi050.stdout: "ops": [ 2024-09-18T08:04:58.990 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:04:58.991 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.14805.0:82943 7.1 7:8203ff76:::benchmark_data_smithi007_73155_object82942:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536 in=65536b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e106)", 2024-09-18T08:04:58.991 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-09-18T08:04:58.974032+0000", 2024-09-18T08:04:58.991 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 0.013037081000000001, 2024-09-18T08:04:58.991 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.013061649, 2024-09-18T08:04:58.991 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-09-18T08:04:58.991 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-09-18T08:04:58.991 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "waiting for sub ops", 2024-09-18T08:04:58.991 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2024-09-18T08:04:58.991 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.14805", 2024-09-18T08:04:58.991 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "172.21.15.7:0/640320436", 2024-09-18T08:04:58.991 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 82943 2024-09-18T08:04:58.992 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:04:58.992 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-09-18T08:04:58.992 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:04:58.992 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-09-18T08:04:58.992 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:58.974032+0000", 2024-09-18T08:04:58.992 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-18T08:04:58.992 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:04:58.992 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:04:58.992 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-09-18T08:04:58.992 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:58.974032+0000", 2024-09-18T08:04:58.993 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-18T08:04:58.993 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:04:58.993 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:04:58.993 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-09-18T08:04:58.993 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:58.974033+0000", 2024-09-18T08:04:58.993 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4.7800000000000002e-07 2024-09-18T08:04:58.994 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:04:58.994 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:04:58.994 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-09-18T08:04:58.994 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:58.974066+0000", 2024-09-18T08:04:58.994 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 3.3769999999999997e-05 2024-09-18T08:04:58.994 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:04:58.995 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:04:58.995 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-09-18T08:04:58.995 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:58.974073+0000", 2024-09-18T08:04:58.995 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 6.7819999999999996e-06 2024-09-18T08:04:58.995 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:04:58.996 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:04:58.996 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-09-18T08:04:58.997 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:58.974079+0000", 2024-09-18T08:04:58.997 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 5.4410000000000001e-06 2024-09-18T08:04:58.997 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:04:58.997 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:04:58.997 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-18T08:04:58.997 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:58.976171+0000", 2024-09-18T08:04:58.997 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0020922990000000002 2024-09-18T08:04:58.997 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:04:58.997 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:04:58.997 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-09-18T08:04:58.998 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:58.976317+0000", 2024-09-18T08:04:58.998 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00014643699999999999 2024-09-18T08:04:58.998 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:04:58.998 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:04:58.998 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "waiting for subops from 3,6", 2024-09-18T08:04:58.998 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:58.976488+0000", 2024-09-18T08:04:58.998 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00017118 2024-09-18T08:04:58.998 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:04:58.998 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:04:58.998 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "op_commit", 2024-09-18T08:04:58.998 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:58.980420+0000", 2024-09-18T08:04:58.998 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.003931286 2024-09-18T08:04:58.998 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:04:58.998 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:04:58.999 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:04:58.999 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:58.980605+0000", 2024-09-18T08:04:58.999 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000185546 2024-09-18T08:04:58.999 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-18T08:04:58.999 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-09-18T08:04:58.999 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-18T08:04:58.999 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:04:58.999 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:04:58.999 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_repop(client.14805.0:82945 7.3 e106/83 7:c5d7cbbd:::benchmark_data_smithi007_73155_object82944:head v 106'5235, mlcod=106'5232)", 2024-09-18T08:04:58.999 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-09-18T08:04:58.986407+0000", 2024-09-18T08:04:58.999 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 0.00066163599999999995, 2024-09-18T08:04:58.999 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00073269600000000002, 2024-09-18T08:04:58.999 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-09-18T08:04:58.999 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-09-18T08:04:59.000 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "started", 2024-09-18T08:04:59.000 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-09-18T08:04:59.000 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:04:59.000 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-09-18T08:04:59.000 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:58.986407+0000", 2024-09-18T08:04:59.000 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-18T08:04:59.000 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:04:59.000 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:04:59.000 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-09-18T08:04:59.000 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:58.986407+0000", 2024-09-18T08:04:59.000 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-18T08:04:59.000 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:04:59.000 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:04:59.000 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-09-18T08:04:59.001 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:58.986408+0000", 2024-09-18T08:04:59.001 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 6.3499999999999996e-07 2024-09-18T08:04:59.001 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:04:59.001 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:04:59.001 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-09-18T08:04:59.001 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:58.986446+0000", 2024-09-18T08:04:59.001 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 3.7493999999999997e-05 2024-09-18T08:04:59.001 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:04:59.001 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:04:59.001 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-09-18T08:04:59.001 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:58.986454+0000", 2024-09-18T08:04:59.001 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 8.3399999999999998e-06 2024-09-18T08:04:59.001 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:04:59.001 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:04:59.002 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-09-18T08:04:59.002 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:58.986459+0000", 2024-09-18T08:04:59.002 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4.9910000000000002e-06 2024-09-18T08:04:59.002 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:04:59.002 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:04:59.002 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-18T08:04:59.002 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:58.986508+0000", 2024-09-18T08:04:59.002 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4.8738e-05 2024-09-18T08:04:59.002 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:04:59.002 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:04:59.002 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-09-18T08:04:59.002 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:58.986522+0000", 2024-09-18T08:04:59.002 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.4354e-05 2024-09-18T08:04:59.002 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-18T08:04:59.003 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-09-18T08:04:59.003 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-18T08:04:59.003 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-18T08:04:59.003 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2024-09-18T08:04:59.003 INFO:teuthology.orchestra.run.smithi050.stdout: "num_ops": 2 2024-09-18T08:04:59.003 INFO:teuthology.orchestra.run.smithi050.stdout:} 2024-09-18T08:04:59.003 INFO:teuthology.orchestra.run.smithi050.stderr:Inferring config /var/lib/ceph/779349ca-7593-11ef-bceb-c7b262605968/mon.b/config 2024-09-18T08:04:59.008 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/779349ca-7593-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T08:04:59.011 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:04:59.012 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:04:59.012 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0' 2024-09-18T08:04:59.113 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7 2024-09-18T08:04:59.142 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:04:59.143 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:04:59.143 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7' 2024-09-18T08:04:59.244 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6 2024-09-18T08:04:59.285 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:04:59.286 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:04:59.286 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6' 2024-09-18T08:04:59.387 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2 2024-09-18T08:04:59.435 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:04:59.435 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:04:59.436 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2' 2024-09-18T08:04:59.450 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:04:59 smithi007 ceph-mon[29459]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-09-18T08:04:59.450 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:04:59 smithi007 ceph-mon[29459]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://smithi007.front.sepia.ceph.com:9093"}]: dispatch 2024-09-18T08:04:59.450 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:04:59 smithi007 ceph-mon[29459]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-09-18T08:04:59.450 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:04:59 smithi007 ceph-mon[29459]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://smithi050.front.sepia.ceph.com:9095"}]: dispatch 2024-09-18T08:04:59.450 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:04:59 smithi007 ceph-mon[29459]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-09-18T08:04:59.450 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:04:59 smithi007 ceph-mon[29459]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://smithi050.front.sepia.ceph.com:3000"}]: dispatch 2024-09-18T08:04:59.450 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:04:59 smithi007 ceph-mon[29459]: from='mgr.24470 172.21.15.7:0/2594390261' entity='mgr.y' cmd={"prefix": "orch prometheus get-credentials"} : dispatch 2024-09-18T08:04:59.450 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:04:59 smithi007 ceph-mon[29459]: from='mgr.24470 172.21.15.7:0/2594390261' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-18T08:04:59.451 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:04:59 smithi007 ceph-mon[29459]: from='mgr.24470 172.21.15.7:0/2594390261' entity='mgr.y' cmd={"prefix": "orch prometheus get-credentials"} : dispatch 2024-09-18T08:04:59.451 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Sep 18 08:04:59 smithi007 ceph-779349ca-7593-11ef-bceb-c7b262605968-mgr-y[29688]: [18/Sep/2024:08:04:59] ENGINE HTTP Server cherrypy._cpwsgi_server.CPWSGIServer(('::', 9283)) shut down 2024-09-18T08:04:59.451 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Sep 18 08:04:59 smithi007 ceph-779349ca-7593-11ef-bceb-c7b262605968-mgr-y[29688]: [18/Sep/2024:08:04:59] ENGINE Bus STOPPED 2024-09-18T08:04:59.451 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Sep 18 08:04:59 smithi007 ceph-779349ca-7593-11ef-bceb-c7b262605968-mgr-y[29688]: [18/Sep/2024:08:04:59] ENGINE Bus STARTING 2024-09-18T08:04:59.451 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Sep 18 08:04:59 smithi007 ceph-779349ca-7593-11ef-bceb-c7b262605968-mgr-y[29688]: [18/Sep/2024:08:04:59] ENGINE Serving on http://:::9283 2024-09-18T08:04:59.451 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Sep 18 08:04:59 smithi007 ceph-779349ca-7593-11ef-bceb-c7b262605968-mgr-y[29688]: [18/Sep/2024:08:04:59] ENGINE Bus STARTED 2024-09-18T08:04:59.451 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Sep 18 08:04:59 smithi007 ceph-779349ca-7593-11ef-bceb-c7b262605968-mgr-y[29688]: [18/Sep/2024:08:04:59] ENGINE Bus STOPPING 2024-09-18T08:04:59.452 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:04:59 smithi007 ceph-mon[39249]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-09-18T08:04:59.452 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:04:59 smithi007 ceph-mon[39249]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://smithi007.front.sepia.ceph.com:9093"}]: dispatch 2024-09-18T08:04:59.452 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:04:59 smithi007 ceph-mon[39249]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-09-18T08:04:59.452 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:04:59 smithi007 ceph-mon[39249]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://smithi050.front.sepia.ceph.com:9095"}]: dispatch 2024-09-18T08:04:59.452 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:04:59 smithi007 ceph-mon[39249]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-09-18T08:04:59.452 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:04:59 smithi007 ceph-mon[39249]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://smithi050.front.sepia.ceph.com:3000"}]: dispatch 2024-09-18T08:04:59.452 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:04:59 smithi007 ceph-mon[39249]: from='mgr.24470 172.21.15.7:0/2594390261' entity='mgr.y' cmd={"prefix": "orch prometheus get-credentials"} : dispatch 2024-09-18T08:04:59.452 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:04:59 smithi007 ceph-mon[39249]: from='mgr.24470 172.21.15.7:0/2594390261' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-18T08:04:59.452 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:04:59 smithi007 ceph-mon[39249]: from='mgr.24470 172.21.15.7:0/2594390261' entity='mgr.y' cmd={"prefix": "orch prometheus get-credentials"} : dispatch 2024-09-18T08:04:59.483 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid 779349ca-7593-11ef-bceb-c7b262605968 -- ceph osd unset nodeep-scrub 2024-09-18T08:04:59.537 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2 2024-09-18T08:04:59.580 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:04:59.581 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:04:59.581 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2' 2024-09-18T08:04:59.583 INFO:tasks.radosbench.radosbench.0.smithi007.stdout: 54 16 83480 83464 96.5732 52.875 0.0104676 0.0103488 2024-09-18T08:04:59.656 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:04:59 smithi050 ceph-mon[32371]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-09-18T08:04:59.656 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:04:59 smithi050 ceph-mon[32371]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://smithi007.front.sepia.ceph.com:9093"}]: dispatch 2024-09-18T08:04:59.656 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:04:59 smithi050 ceph-mon[32371]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-09-18T08:04:59.656 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:04:59 smithi050 ceph-mon[32371]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://smithi050.front.sepia.ceph.com:9095"}]: dispatch 2024-09-18T08:04:59.656 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:04:59 smithi050 ceph-mon[32371]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-09-18T08:04:59.656 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:04:59 smithi050 ceph-mon[32371]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://smithi050.front.sepia.ceph.com:3000"}]: dispatch 2024-09-18T08:04:59.657 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:04:59 smithi050 ceph-mon[32371]: from='mgr.24470 172.21.15.7:0/2594390261' entity='mgr.y' cmd={"prefix": "orch prometheus get-credentials"} : dispatch 2024-09-18T08:04:59.657 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:04:59 smithi050 ceph-mon[32371]: from='mgr.24470 172.21.15.7:0/2594390261' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-18T08:04:59.657 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:04:59 smithi050 ceph-mon[32371]: from='mgr.24470 172.21.15.7:0/2594390261' entity='mgr.y' cmd={"prefix": "orch prometheus get-credentials"} : dispatch 2024-09-18T08:04:59.682 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1 2024-09-18T08:04:59.732 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:04:59.732 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:04:59.733 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1' 2024-09-18T08:04:59.740 DEBUG:teuthology.orchestra.run.smithi050:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid 779349ca-7593-11ef-bceb-c7b262605968 -- ceph daemon osd.4 dump_blocked_ops 2024-09-18T08:04:59.834 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0 2024-09-18T08:04:59.892 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:04:59.893 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:04:59.893 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0' 2024-09-18T08:04:59.994 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6 2024-09-18T08:05:00.038 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:05:00.039 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:05:00.039 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6' 2024-09-18T08:05:00.140 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4 2024-09-18T08:05:00.175 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:05:00.176 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:05:00.176 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4' 2024-09-18T08:05:00.230 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Sep 18 08:04:59 smithi007 ceph-779349ca-7593-11ef-bceb-c7b262605968-mgr-y[29688]: [18/Sep/2024:08:04:59] ENGINE HTTP Server cherrypy._cpwsgi_server.CPWSGIServer(('::', 9283)) shut down 2024-09-18T08:05:00.231 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Sep 18 08:04:59 smithi007 ceph-779349ca-7593-11ef-bceb-c7b262605968-mgr-y[29688]: [18/Sep/2024:08:04:59] ENGINE Bus STOPPED 2024-09-18T08:05:00.231 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Sep 18 08:04:59 smithi007 ceph-779349ca-7593-11ef-bceb-c7b262605968-mgr-y[29688]: [18/Sep/2024:08:04:59] ENGINE Bus STARTING 2024-09-18T08:05:00.231 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Sep 18 08:04:59 smithi007 ceph-779349ca-7593-11ef-bceb-c7b262605968-mgr-y[29688]: [18/Sep/2024:08:04:59] ENGINE Serving on http://:::9283 2024-09-18T08:05:00.231 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Sep 18 08:04:59 smithi007 ceph-779349ca-7593-11ef-bceb-c7b262605968-mgr-y[29688]: [18/Sep/2024:08:04:59] ENGINE Bus STARTED 2024-09-18T08:05:00.262 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2024-09-18T08:05:00.262 INFO:teuthology.orchestra.run.smithi050.stdout: "success": "filestore_inject_stall = '' (not observed, change may require restart) " 2024-09-18T08:05:00.262 INFO:teuthology.orchestra.run.smithi050.stdout:} 2024-09-18T08:05:00.278 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5 2024-09-18T08:05:00.325 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:05:00.326 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:05:00.327 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5' 2024-09-18T08:05:00.428 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3 2024-09-18T08:05:00.471 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:05:00.472 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:05:00.472 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3' 2024-09-18T08:05:00.573 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1 2024-09-18T08:05:00.585 INFO:tasks.radosbench.radosbench.0.smithi007.stdout: 55 16 84508 84492 95.9825 64.25 0.0338049 0.0104091 2024-09-18T08:05:00.603 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:05:00 smithi050 ceph-mon[32371]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "orch prometheus get-credentials"}]: dispatch 2024-09-18T08:05:00.603 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:05:00 smithi050 ceph-mon[32371]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-18T08:05:00.603 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:05:00 smithi050 ceph-mon[32371]: 3.c scrub starts 2024-09-18T08:05:00.603 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:05:00 smithi050 ceph-mon[32371]: 3.c scrub ok 2024-09-18T08:05:00.603 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:05:00 smithi050 ceph-mon[32371]: pgmap v87: 148 pgs: 148 active+clean; 5.0 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 1.2 KiB/s rd, 88 MiB/s wr, 1.40k op/s 2024-09-18T08:05:00.603 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:05:00 smithi050 ceph-mon[32371]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "orch prometheus get-credentials"}]: dispatch 2024-09-18T08:05:00.603 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:05:00 smithi050 ceph-mon[32371]: 3.1 scrub starts 2024-09-18T08:05:00.603 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:05:00 smithi050 ceph-mon[32371]: 3.1 scrub ok 2024-09-18T08:05:00.603 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:05:00 smithi050 ceph-mon[32371]: purged_snaps scrub starts 2024-09-18T08:05:00.603 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:05:00 smithi050 ceph-mon[32371]: purged_snaps scrub ok 2024-09-18T08:05:00.604 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:05:00 smithi050 ceph-mon[32371]: from='mgr.24470 172.21.15.7:0/2594390261' entity='mgr.y' cmd={"prefix": "orch prometheus get-credentials"} : dispatch 2024-09-18T08:05:00.612 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:05:00.613 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:05:00.613 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1' 2024-09-18T08:05:00.714 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3 2024-09-18T08:05:00.730 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:05:00 smithi007 ceph-mon[29459]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "orch prometheus get-credentials"}]: dispatch 2024-09-18T08:05:00.730 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:05:00 smithi007 ceph-mon[29459]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-18T08:05:00.730 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:05:00 smithi007 ceph-mon[29459]: 3.c scrub starts 2024-09-18T08:05:00.730 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:05:00 smithi007 ceph-mon[29459]: 3.c scrub ok 2024-09-18T08:05:00.731 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:05:00 smithi007 ceph-mon[29459]: pgmap v87: 148 pgs: 148 active+clean; 5.0 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 1.2 KiB/s rd, 88 MiB/s wr, 1.40k op/s 2024-09-18T08:05:00.731 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:05:00 smithi007 ceph-mon[29459]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "orch prometheus get-credentials"}]: dispatch 2024-09-18T08:05:00.731 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:05:00 smithi007 ceph-mon[29459]: 3.1 scrub starts 2024-09-18T08:05:00.731 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:05:00 smithi007 ceph-mon[29459]: 3.1 scrub ok 2024-09-18T08:05:00.731 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:05:00 smithi007 ceph-mon[29459]: purged_snaps scrub starts 2024-09-18T08:05:00.731 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:05:00 smithi007 ceph-mon[29459]: purged_snaps scrub ok 2024-09-18T08:05:00.731 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:05:00 smithi007 ceph-mon[29459]: from='mgr.24470 172.21.15.7:0/2594390261' entity='mgr.y' cmd={"prefix": "orch prometheus get-credentials"} : dispatch 2024-09-18T08:05:00.731 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:05:00 smithi007 ceph-mon[39249]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "orch prometheus get-credentials"}]: dispatch 2024-09-18T08:05:00.731 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:05:00 smithi007 ceph-mon[39249]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-18T08:05:00.731 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:05:00 smithi007 ceph-mon[39249]: 3.c scrub starts 2024-09-18T08:05:00.731 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:05:00 smithi007 ceph-mon[39249]: 3.c scrub ok 2024-09-18T08:05:00.731 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:05:00 smithi007 ceph-mon[39249]: pgmap v87: 148 pgs: 148 active+clean; 5.0 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 1.2 KiB/s rd, 88 MiB/s wr, 1.40k op/s 2024-09-18T08:05:00.732 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:05:00 smithi007 ceph-mon[39249]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "orch prometheus get-credentials"}]: dispatch 2024-09-18T08:05:00.732 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:05:00 smithi007 ceph-mon[39249]: 3.1 scrub starts 2024-09-18T08:05:00.732 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:05:00 smithi007 ceph-mon[39249]: 3.1 scrub ok 2024-09-18T08:05:00.732 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:05:00 smithi007 ceph-mon[39249]: purged_snaps scrub starts 2024-09-18T08:05:00.732 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:05:00 smithi007 ceph-mon[39249]: purged_snaps scrub ok 2024-09-18T08:05:00.732 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:05:00 smithi007 ceph-mon[39249]: from='mgr.24470 172.21.15.7:0/2594390261' entity='mgr.y' cmd={"prefix": "orch prometheus get-credentials"} : dispatch 2024-09-18T08:05:00.736 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:05:00.736 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:05:00.737 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3' 2024-09-18T08:05:00.838 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5 2024-09-18T08:05:00.868 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:05:00.869 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:05:00.869 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5' 2024-09-18T08:05:00.970 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7 2024-09-18T08:05:01.005 INFO:teuthology.orchestra.run.smithi050.stderr:Inferring config /var/lib/ceph/779349ca-7593-11ef-bceb-c7b262605968/mon.b/config 2024-09-18T08:05:01.022 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:05:01.023 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:05:01.023 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7' 2024-09-18T08:05:01.124 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0 2024-09-18T08:05:01.188 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:05:01.188 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:05:01.189 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0' 2024-09-18T08:05:01.290 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5 2024-09-18T08:05:01.324 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:05:01.325 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:05:01.326 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5' 2024-09-18T08:05:01.427 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3 2024-09-18T08:05:01.465 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:05:01 smithi050 ceph-mon[32371]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "orch prometheus get-credentials"}]: dispatch 2024-09-18T08:05:01.465 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:05:01 smithi050 ceph-mon[32371]: 7.0 scrub starts 2024-09-18T08:05:01.465 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:05:01 smithi050 ceph-mon[32371]: 4.16 scrub starts 2024-09-18T08:05:01.465 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:05:01 smithi050 ceph-mon[32371]: 4.16 scrub ok 2024-09-18T08:05:01.465 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:05:01 smithi050 ceph-mon[32371]: osdmap e107: 8 total, 8 up, 8 in 2024-09-18T08:05:01.465 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:05:01 smithi050 ceph-mon[32371]: 3.14 scrub starts 2024-09-18T08:05:01.465 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:05:01 smithi050 ceph-mon[32371]: 3.14 scrub ok 2024-09-18T08:05:01.483 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:05:01.485 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:05:01.485 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3' 2024-09-18T08:05:01.486 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/779349ca-7593-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T08:05:01.504 INFO:teuthology.orchestra.run.smithi007.stdout:osd.0: {} 2024-09-18T08:05:01.505 INFO:teuthology.orchestra.run.smithi007.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-18T08:05:01.540 INFO:teuthology.orchestra.run.smithi007.stdout:osd.1: {} 2024-09-18T08:05:01.540 INFO:teuthology.orchestra.run.smithi007.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-18T08:05:01.580 INFO:teuthology.orchestra.run.smithi007.stdout:osd.2: {} 2024-09-18T08:05:01.581 INFO:teuthology.orchestra.run.smithi007.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-18T08:05:01.585 INFO:tasks.radosbench.radosbench.0.smithi007.stdout: 56 16 85479 85463 95.352 60.6875 0.0110236 0.010481 2024-09-18T08:05:01.586 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3 2024-09-18T08:05:01.593 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2024-09-18T08:05:01.593 INFO:teuthology.orchestra.run.smithi050.stdout: "ops": [], 2024-09-18T08:05:01.593 INFO:teuthology.orchestra.run.smithi050.stdout: "complaint_time": 30, 2024-09-18T08:05:01.593 INFO:teuthology.orchestra.run.smithi050.stdout: "num_blocked_ops": 0 2024-09-18T08:05:01.593 INFO:teuthology.orchestra.run.smithi050.stdout:} 2024-09-18T08:05:01.618 INFO:teuthology.orchestra.run.smithi007.stdout:osd.3: {} 2024-09-18T08:05:01.618 INFO:teuthology.orchestra.run.smithi007.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-18T08:05:01.644 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:05:01 smithi007 ceph-mon[29459]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "orch prometheus get-credentials"}]: dispatch 2024-09-18T08:05:01.644 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:05:01 smithi007 ceph-mon[29459]: 7.0 scrub starts 2024-09-18T08:05:01.645 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:05:01 smithi007 ceph-mon[29459]: 4.16 scrub starts 2024-09-18T08:05:01.645 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:05:01 smithi007 ceph-mon[29459]: 4.16 scrub ok 2024-09-18T08:05:01.645 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:05:01 smithi007 ceph-mon[29459]: osdmap e107: 8 total, 8 up, 8 in 2024-09-18T08:05:01.645 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:05:01 smithi007 ceph-mon[29459]: 3.14 scrub starts 2024-09-18T08:05:01.645 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:05:01 smithi007 ceph-mon[29459]: 3.14 scrub ok 2024-09-18T08:05:01.646 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:05:01 smithi007 ceph-mon[39249]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "orch prometheus get-credentials"}]: dispatch 2024-09-18T08:05:01.646 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:05:01 smithi007 ceph-mon[39249]: 7.0 scrub starts 2024-09-18T08:05:01.646 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:05:01 smithi007 ceph-mon[39249]: 4.16 scrub starts 2024-09-18T08:05:01.646 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:05:01 smithi007 ceph-mon[39249]: 4.16 scrub ok 2024-09-18T08:05:01.647 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:05:01 smithi007 ceph-mon[39249]: osdmap e107: 8 total, 8 up, 8 in 2024-09-18T08:05:01.647 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:05:01 smithi007 ceph-mon[39249]: 3.14 scrub starts 2024-09-18T08:05:01.647 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:05:01 smithi007 ceph-mon[39249]: 3.14 scrub ok 2024-09-18T08:05:01.647 INFO:journalctl@ceph.alertmanager.a.smithi007.stdout:Sep 18 08:05:01 smithi007 ceph-779349ca-7593-11ef-bceb-c7b262605968-alertmanager-a[78486]: ts=2024-09-18T08:05:01.452Z caller=cluster.go:700 level=info component=cluster msg="gossip settled; proceeding" elapsed=10.003903879s 2024-09-18T08:05:01.649 INFO:teuthology.orchestra.run.smithi007.stdout:osd.4: {} 2024-09-18T08:05:01.649 INFO:teuthology.orchestra.run.smithi007.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-18T08:05:01.652 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:05:01.653 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:05:01.654 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3' 2024-09-18T08:05:01.683 INFO:teuthology.orchestra.run.smithi007.stdout:osd.5: {} 2024-09-18T08:05:01.683 INFO:teuthology.orchestra.run.smithi007.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-18T08:05:01.718 INFO:teuthology.orchestra.run.smithi007.stdout:osd.6: {} 2024-09-18T08:05:01.718 INFO:teuthology.orchestra.run.smithi007.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-18T08:05:01.753 INFO:teuthology.orchestra.run.smithi007.stdout:osd.7: {} 2024-09-18T08:05:01.753 INFO:teuthology.orchestra.run.smithi007.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-18T08:05:01.755 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2 2024-09-18T08:05:01.805 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:05:01.806 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:05:01.806 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2' 2024-09-18T08:05:01.907 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1 2024-09-18T08:05:01.954 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:05:01.954 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:05:01.955 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1' 2024-09-18T08:05:02.056 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3 2024-09-18T08:05:02.095 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:05:02.096 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:05:02.096 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3' 2024-09-18T08:05:02.197 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2 2024-09-18T08:05:02.236 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:05:02.237 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:05:02.237 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2' 2024-09-18T08:05:02.338 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5 2024-09-18T08:05:02.367 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:05:02.368 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:05:02.368 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5' 2024-09-18T08:05:02.469 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6 2024-09-18T08:05:02.498 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:05:02.499 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:05:02.499 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6' 2024-09-18T08:05:02.585 INFO:tasks.radosbench.radosbench.0.smithi007.stdout: 57 16 86601 86585 94.9094 70.125 0.00633315 0.0105305 2024-09-18T08:05:02.590 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:05:02 smithi050 ceph-mon[32371]: 6.15 scrub starts 2024-09-18T08:05:02.591 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:05:02 smithi050 ceph-mon[32371]: 6.15 scrub ok 2024-09-18T08:05:02.591 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:05:02 smithi050 ceph-mon[32371]: 5.14 scrub starts 2024-09-18T08:05:02.591 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:05:02 smithi050 ceph-mon[32371]: 5.14 scrub ok 2024-09-18T08:05:02.591 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:05:02 smithi050 ceph-mon[32371]: pgmap v89: 148 pgs: 148 active+clean; 5.0 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 853 B/s rd, 57 MiB/s wr, 911 op/s 2024-09-18T08:05:02.591 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:05:02 smithi050 ceph-mon[32371]: 6.18 scrub starts 2024-09-18T08:05:02.591 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:05:02 smithi050 ceph-mon[32371]: 6.18 scrub ok 2024-09-18T08:05:02.591 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:05:02 smithi050 ceph-mon[32371]: 7.0 scrub ok 2024-09-18T08:05:02.591 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:05:02 smithi050 ceph-mon[32371]: from='client.? 172.21.15.7:0/3732920322' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-18T08:05:02.600 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2 2024-09-18T08:05:02.630 DEBUG:teuthology.orchestra.run.smithi050:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid 779349ca-7593-11ef-bceb-c7b262605968 -- ceph daemon osd.4 dump_historic_ops 2024-09-18T08:05:02.638 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:05:02.639 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:05:02.639 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2' 2024-09-18T08:05:02.684 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:05:02 smithi007 ceph-mon[29459]: 6.15 scrub starts 2024-09-18T08:05:02.684 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:05:02 smithi007 ceph-mon[29459]: 6.15 scrub ok 2024-09-18T08:05:02.684 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:05:02 smithi007 ceph-mon[29459]: 5.14 scrub starts 2024-09-18T08:05:02.684 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:05:02 smithi007 ceph-mon[29459]: 5.14 scrub ok 2024-09-18T08:05:02.684 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:05:02 smithi007 ceph-mon[29459]: pgmap v89: 148 pgs: 148 active+clean; 5.0 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 853 B/s rd, 57 MiB/s wr, 911 op/s 2024-09-18T08:05:02.684 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:05:02 smithi007 ceph-mon[29459]: 6.18 scrub starts 2024-09-18T08:05:02.684 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:05:02 smithi007 ceph-mon[29459]: 6.18 scrub ok 2024-09-18T08:05:02.684 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:05:02 smithi007 ceph-mon[29459]: 7.0 scrub ok 2024-09-18T08:05:02.685 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:05:02 smithi007 ceph-mon[29459]: from='client.? 172.21.15.7:0/3732920322' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-18T08:05:02.685 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:05:02 smithi007 ceph-mon[39249]: 6.15 scrub starts 2024-09-18T08:05:02.685 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:05:02 smithi007 ceph-mon[39249]: 6.15 scrub ok 2024-09-18T08:05:02.685 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:05:02 smithi007 ceph-mon[39249]: 5.14 scrub starts 2024-09-18T08:05:02.685 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:05:02 smithi007 ceph-mon[39249]: 5.14 scrub ok 2024-09-18T08:05:02.685 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:05:02 smithi007 ceph-mon[39249]: pgmap v89: 148 pgs: 148 active+clean; 5.0 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 853 B/s rd, 57 MiB/s wr, 911 op/s 2024-09-18T08:05:02.685 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:05:02 smithi007 ceph-mon[39249]: 6.18 scrub starts 2024-09-18T08:05:02.685 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:05:02 smithi007 ceph-mon[39249]: 6.18 scrub ok 2024-09-18T08:05:02.685 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:05:02 smithi007 ceph-mon[39249]: 7.0 scrub ok 2024-09-18T08:05:02.686 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:05:02 smithi007 ceph-mon[39249]: from='client.? 172.21.15.7:0/3732920322' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-18T08:05:02.740 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2 2024-09-18T08:05:02.787 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:05:02.788 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:05:02.788 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2' 2024-09-18T08:05:02.889 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5 2024-09-18T08:05:02.918 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:05:02.918 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:05:02.920 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5' 2024-09-18T08:05:03.020 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6 2024-09-18T08:05:03.056 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:05:03.057 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:05:03.058 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6' 2024-09-18T08:05:03.104 INFO:teuthology.orchestra.run.smithi050.stderr:Inferring config /var/lib/ceph/779349ca-7593-11ef-bceb-c7b262605968/mon.b/config 2024-09-18T08:05:03.159 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7 2024-09-18T08:05:03.187 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:05:03.188 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:05:03.188 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7' 2024-09-18T08:05:03.289 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7 2024-09-18T08:05:03.314 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:05:03.315 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:05:03.315 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7' 2024-09-18T08:05:03.416 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4 2024-09-18T08:05:03.450 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:05:03.450 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:05:03.450 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4' 2024-09-18T08:05:03.551 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1 2024-09-18T08:05:03.585 INFO:tasks.radosbench.radosbench.0.smithi007.stdout: 58 16 87642 87626 94.3948 65.0625 0.0205495 0.0105882 2024-09-18T08:05:03.592 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:05:03.593 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:05:03.593 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1' 2024-09-18T08:05:03.655 INFO:teuthology.orchestra.run.smithi050.stdout:ERROR: (22) Invalid argument 2024-09-18T08:05:03.655 INFO:teuthology.orchestra.run.smithi050.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-18T08:05:03.670 INFO:teuthology.orchestra.run.smithi007.stderr:nodeep-scrub is unset 2024-09-18T08:05:03.694 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6 2024-09-18T08:05:03.727 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:05:03.728 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:05:03.728 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6' 2024-09-18T08:05:03.829 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1 2024-09-18T08:05:03.852 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:05:03.853 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:05:03.853 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1' 2024-09-18T08:05:03.906 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:05:03 smithi050 ceph-mon[32371]: 3.18 scrub starts 2024-09-18T08:05:03.906 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:05:03 smithi050 ceph-mon[32371]: 3.18 scrub ok 2024-09-18T08:05:03.906 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:05:03 smithi050 ceph-mon[32371]: from='mgr.24470 172.21.15.7:0/2594390261' entity='mgr.y' 2024-09-18T08:05:03.906 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:05:03 smithi050 ceph-mon[32371]: from='mgr.24470 172.21.15.7:0/2594390261' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-18T08:05:03.906 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:05:03 smithi050 ceph-mon[32371]: pgmap v90: 148 pgs: 148 active+clean; 5.1 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 1.3 KiB/s rd, 51 MiB/s wr, 820 op/s 2024-09-18T08:05:03.906 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:05:03 smithi050 ceph-mon[32371]: from='client.? 172.21.15.7:0/3054817171' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-18T08:05:03.954 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1 2024-09-18T08:05:03.980 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:05:03 smithi007 ceph-mon[29459]: 3.18 scrub starts 2024-09-18T08:05:03.981 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:05:03 smithi007 ceph-mon[29459]: 3.18 scrub ok 2024-09-18T08:05:03.981 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:05:03 smithi007 ceph-mon[29459]: from='mgr.24470 172.21.15.7:0/2594390261' entity='mgr.y' 2024-09-18T08:05:03.981 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:05:03 smithi007 ceph-mon[29459]: from='mgr.24470 172.21.15.7:0/2594390261' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-18T08:05:03.981 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:05:03 smithi007 ceph-mon[29459]: pgmap v90: 148 pgs: 148 active+clean; 5.1 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 1.3 KiB/s rd, 51 MiB/s wr, 820 op/s 2024-09-18T08:05:03.981 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:05:03 smithi007 ceph-mon[29459]: from='client.? 172.21.15.7:0/3054817171' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-18T08:05:03.981 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:05:03 smithi007 ceph-mon[39249]: 3.18 scrub starts 2024-09-18T08:05:03.981 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:05:03 smithi007 ceph-mon[39249]: 3.18 scrub ok 2024-09-18T08:05:03.981 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:05:03 smithi007 ceph-mon[39249]: from='mgr.24470 172.21.15.7:0/2594390261' entity='mgr.y' 2024-09-18T08:05:03.981 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:05:03 smithi007 ceph-mon[39249]: from='mgr.24470 172.21.15.7:0/2594390261' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-18T08:05:03.981 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:05:03 smithi007 ceph-mon[39249]: pgmap v90: 148 pgs: 148 active+clean; 5.1 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 1.3 KiB/s rd, 51 MiB/s wr, 820 op/s 2024-09-18T08:05:03.981 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:05:03 smithi007 ceph-mon[39249]: from='client.? 172.21.15.7:0/3054817171' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-18T08:05:04.009 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:05:04.010 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:05:04.010 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1' 2024-09-18T08:05:04.111 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7 2024-09-18T08:05:04.139 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:05:04.139 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:05:04.140 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7' 2024-09-18T08:05:04.240 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2 2024-09-18T08:05:04.280 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:05:04.281 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:05:04.281 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2' 2024-09-18T08:05:04.382 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4 2024-09-18T08:05:04.409 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:05:04.410 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:05:04.410 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4' 2024-09-18T08:05:04.456 DEBUG:teuthology.orchestra.run.smithi050:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid 779349ca-7593-11ef-bceb-c7b262605968 -- ceph daemon osd.5 dump_ops_in_flight 2024-09-18T08:05:04.511 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1 2024-09-18T08:05:04.539 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:05:04.539 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:05:04.540 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1' 2024-09-18T08:05:04.585 INFO:tasks.radosbench.radosbench.0.smithi007.stdout: 59 16 88713 88697 93.9294 66.9375 0.0122348 0.0106406 2024-09-18T08:05:04.641 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0 2024-09-18T08:05:04.681 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:05:04.682 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:05:04.682 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0' 2024-09-18T08:05:04.684 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid 779349ca-7593-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-18T08:05:04.783 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1 2024-09-18T08:05:04.848 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:05:04.849 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:05:04.849 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1' 2024-09-18T08:05:04.950 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3 2024-09-18T08:05:04.981 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:05:04 smithi007 ceph-mon[29459]: 6.1b scrub starts 2024-09-18T08:05:04.981 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:05:04 smithi007 ceph-mon[29459]: 6.1b scrub ok 2024-09-18T08:05:04.981 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:05:04 smithi007 ceph-mon[29459]: 6.16 scrub starts 2024-09-18T08:05:04.981 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:05:04 smithi007 ceph-mon[29459]: 6.16 scrub ok 2024-09-18T08:05:04.981 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:05:04 smithi007 ceph-mon[29459]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-18T08:05:04.981 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:05:04 smithi007 ceph-mon[29459]: Cluster is now healthy 2024-09-18T08:05:04.981 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:05:04 smithi007 ceph-mon[29459]: from='client.? 172.21.15.7:0/3054817171' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-18T08:05:04.981 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:05:04 smithi007 ceph-mon[29459]: osdmap e108: 8 total, 8 up, 8 in 2024-09-18T08:05:04.985 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:05:04 smithi007 ceph-mon[39249]: 6.1b scrub starts 2024-09-18T08:05:04.985 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:05:04 smithi007 ceph-mon[39249]: 6.1b scrub ok 2024-09-18T08:05:04.985 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:05:04 smithi007 ceph-mon[39249]: 6.16 scrub starts 2024-09-18T08:05:04.985 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:05:04 smithi007 ceph-mon[39249]: 6.16 scrub ok 2024-09-18T08:05:04.985 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:05:04 smithi007 ceph-mon[39249]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-18T08:05:04.985 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:05:04 smithi007 ceph-mon[39249]: Cluster is now healthy 2024-09-18T08:05:04.985 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:05:04 smithi007 ceph-mon[39249]: from='client.? 172.21.15.7:0/3054817171' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-18T08:05:04.986 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:05:04 smithi007 ceph-mon[39249]: osdmap e108: 8 total, 8 up, 8 in 2024-09-18T08:05:05.014 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:05:05.015 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:05:05.015 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3' 2024-09-18T08:05:05.116 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2 2024-09-18T08:05:05.125 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:05:04 smithi050 ceph-mon[32371]: 6.1b scrub starts 2024-09-18T08:05:05.125 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:05:04 smithi050 ceph-mon[32371]: 6.1b scrub ok 2024-09-18T08:05:05.125 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:05:04 smithi050 ceph-mon[32371]: 6.16 scrub starts 2024-09-18T08:05:05.125 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:05:04 smithi050 ceph-mon[32371]: 6.16 scrub ok 2024-09-18T08:05:05.125 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:05:04 smithi050 ceph-mon[32371]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-18T08:05:05.125 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:05:04 smithi050 ceph-mon[32371]: Cluster is now healthy 2024-09-18T08:05:05.125 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:05:04 smithi050 ceph-mon[32371]: from='client.? 172.21.15.7:0/3054817171' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-18T08:05:05.126 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:05:04 smithi050 ceph-mon[32371]: osdmap e108: 8 total, 8 up, 8 in 2024-09-18T08:05:05.162 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:05:05.162 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:05:05.162 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2' 2024-09-18T08:05:05.263 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7 2024-09-18T08:05:05.292 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:05:05.292 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:05:05.293 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7' 2024-09-18T08:05:05.394 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7 2024-09-18T08:05:05.413 INFO:teuthology.orchestra.run.smithi050.stderr:Inferring config /var/lib/ceph/779349ca-7593-11ef-bceb-c7b262605968/mon.b/config 2024-09-18T08:05:05.436 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:05:05.437 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:05:05.437 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7' 2024-09-18T08:05:05.538 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1 2024-09-18T08:05:05.575 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:05:05.576 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:05:05.576 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1' 2024-09-18T08:05:05.585 INFO:tasks.radosbench.radosbench.0.smithi007.stdout:2024-09-18T08:05:05.584566+0000 min lat: 0.00117713 max lat: 1.0787 avg lat: 0.0106962 2024-09-18T08:05:05.585 INFO:tasks.radosbench.radosbench.0.smithi007.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-09-18T08:05:05.585 INFO:tasks.radosbench.radosbench.0.smithi007.stdout: 60 16 89750 89734 93.4442 64.8125 0.00953702 0.0106962 2024-09-18T08:05:05.677 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5 2024-09-18T08:05:05.707 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:05:05.707 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:05:05.708 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5' 2024-09-18T08:05:05.808 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2 2024-09-18T08:05:05.902 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:05:05.902 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:05:05.902 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2' 2024-09-18T08:05:05.905 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:05:05 smithi050 ceph-mon[32371]: 6.1 scrub starts 2024-09-18T08:05:05.906 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:05:05 smithi050 ceph-mon[32371]: 6.1 scrub ok 2024-09-18T08:05:05.906 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:05:05 smithi050 ceph-mon[32371]: 6.6 scrub starts 2024-09-18T08:05:05.906 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:05:05 smithi050 ceph-mon[32371]: 6.6 scrub ok 2024-09-18T08:05:05.906 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:05:05 smithi050 ceph-mon[32371]: pgmap v92: 148 pgs: 148 active+clean; 5.3 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 1.2 KiB/s rd, 81 MiB/s wr, 1.30k op/s 2024-09-18T08:05:05.968 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-09-18T08:05:05.968 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid 779349ca-7593-11ef-bceb-c7b262605968 -- ceph osd reweight 0 1 2024-09-18T08:05:05.980 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:05:05 smithi007 ceph-mon[29459]: 6.1 scrub starts 2024-09-18T08:05:05.980 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:05:05 smithi007 ceph-mon[29459]: 6.1 scrub ok 2024-09-18T08:05:05.980 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:05:05 smithi007 ceph-mon[29459]: 6.6 scrub starts 2024-09-18T08:05:05.981 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:05:05 smithi007 ceph-mon[29459]: 6.6 scrub ok 2024-09-18T08:05:05.981 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:05:05 smithi007 ceph-mon[29459]: pgmap v92: 148 pgs: 148 active+clean; 5.3 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 1.2 KiB/s rd, 81 MiB/s wr, 1.30k op/s 2024-09-18T08:05:05.984 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:05:05 smithi007 ceph-mon[39249]: 6.1 scrub starts 2024-09-18T08:05:05.984 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:05:05 smithi007 ceph-mon[39249]: 6.1 scrub ok 2024-09-18T08:05:05.984 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:05:05 smithi007 ceph-mon[39249]: 6.6 scrub starts 2024-09-18T08:05:05.984 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:05:05 smithi007 ceph-mon[39249]: 6.6 scrub ok 2024-09-18T08:05:05.984 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:05:05 smithi007 ceph-mon[39249]: pgmap v92: 148 pgs: 148 active+clean; 5.3 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 1.2 KiB/s rd, 81 MiB/s wr, 1.30k op/s 2024-09-18T08:05:06.003 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4 2024-09-18T08:05:06.029 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:05:06.030 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:05:06.030 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4' 2024-09-18T08:05:06.114 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/779349ca-7593-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T08:05:06.131 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5 2024-09-18T08:05:06.157 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:05:06.157 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:05:06.158 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5' 2024-09-18T08:05:06.259 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2 2024-09-18T08:05:06.323 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:05:06.323 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:05:06.324 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2' 2024-09-18T08:05:06.424 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4 2024-09-18T08:05:06.448 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:05:06.449 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:05:06.449 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4' 2024-09-18T08:05:06.550 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4 2024-09-18T08:05:06.577 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:05:06.578 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:05:06.579 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4' 2024-09-18T08:05:06.585 INFO:tasks.radosbench.radosbench.0.smithi007.stdout: 61 16 90675 90659 92.86 57.8125 0.00964324 0.0107637 2024-09-18T08:05:06.679 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1 2024-09-18T08:05:06.721 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:05:06.722 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:05:06.722 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1' 2024-09-18T08:05:06.751 INFO:teuthology.orchestra.run.smithi050.stdout:ERROR: (22) Invalid argument 2024-09-18T08:05:06.751 INFO:teuthology.orchestra.run.smithi050.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-18T08:05:06.823 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5 2024-09-18T08:05:06.852 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:05:06.852 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:05:06.853 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5' 2024-09-18T08:05:06.906 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:05:06 smithi050 ceph-mon[32371]: 4.b scrub starts 2024-09-18T08:05:06.906 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:05:06 smithi050 ceph-mon[32371]: 4.b scrub ok 2024-09-18T08:05:06.953 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7 2024-09-18T08:05:06.976 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:05:06.977 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:05:06.977 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7' 2024-09-18T08:05:06.980 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:05:06 smithi007 ceph-mon[29459]: 4.b scrub starts 2024-09-18T08:05:06.980 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:05:06 smithi007 ceph-mon[29459]: 4.b scrub ok 2024-09-18T08:05:06.981 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Sep 18 08:05:06 smithi007 ceph-779349ca-7593-11ef-bceb-c7b262605968-mgr-y[29688]: ::ffff:172.21.15.50 - - [18/Sep/2024:08:05:06] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-09-18T08:05:06.981 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:05:06 smithi007 ceph-mon[39249]: 4.b scrub starts 2024-09-18T08:05:06.981 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:05:06 smithi007 ceph-mon[39249]: 4.b scrub ok 2024-09-18T08:05:07.078 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2 2024-09-18T08:05:07.115 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:05:07.115 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:05:07.116 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2' 2024-09-18T08:05:07.216 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6 2024-09-18T08:05:07.249 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:05:07.250 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:05:07.251 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6' 2024-09-18T08:05:07.351 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3 2024-09-18T08:05:07.389 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:05:07.390 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:05:07.390 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3' 2024-09-18T08:05:07.491 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1 2024-09-18T08:05:07.531 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:05:07.532 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:05:07.532 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1' 2024-09-18T08:05:07.586 INFO:tasks.radosbench.radosbench.0.smithi007.stdout: 62 16 91785 91769 92.4808 69.375 0.0102246 0.0108067 2024-09-18T08:05:07.633 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2 2024-09-18T08:05:07.687 DEBUG:teuthology.orchestra.run.smithi050:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid 779349ca-7593-11ef-bceb-c7b262605968 -- ceph daemon osd.5 dump_blocked_ops 2024-09-18T08:05:07.690 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:05:07.691 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:05:07.691 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2' 2024-09-18T08:05:07.792 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5 2024-09-18T08:05:07.803 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid 779349ca-7593-11ef-bceb-c7b262605968 -- ceph osd set noscrub 2024-09-18T08:05:07.821 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:05:07.821 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:05:07.822 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5' 2024-09-18T08:05:07.898 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:05:07 smithi050 ceph-mon[32371]: 3.19 scrub starts 2024-09-18T08:05:07.898 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:05:07 smithi050 ceph-mon[32371]: 3.19 scrub ok 2024-09-18T08:05:07.898 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:05:07 smithi050 ceph-mon[32371]: 6.3 scrub starts 2024-09-18T08:05:07.898 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:05:07 smithi050 ceph-mon[32371]: 6.3 scrub ok 2024-09-18T08:05:07.898 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:05:07 smithi050 ceph-mon[32371]: pgmap v93: 148 pgs: 148 active+clean; 5.3 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 639 B/s rd, 39 MiB/s wr, 617 op/s 2024-09-18T08:05:07.923 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2 2024-09-18T08:05:07.960 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/779349ca-7593-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T08:05:07.964 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:05:07 smithi007 ceph-mon[39249]: 3.19 scrub starts 2024-09-18T08:05:07.964 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:05:07 smithi007 ceph-mon[39249]: 3.19 scrub ok 2024-09-18T08:05:07.964 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:05:07 smithi007 ceph-mon[39249]: 6.3 scrub starts 2024-09-18T08:05:07.964 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:05:07 smithi007 ceph-mon[39249]: 6.3 scrub ok 2024-09-18T08:05:07.964 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:05:07 smithi007 ceph-mon[39249]: pgmap v93: 148 pgs: 148 active+clean; 5.3 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 639 B/s rd, 39 MiB/s wr, 617 op/s 2024-09-18T08:05:07.964 INFO:teuthology.orchestra.run.smithi007.stdout:osd.0: {} 2024-09-18T08:05:07.965 INFO:teuthology.orchestra.run.smithi007.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-18T08:05:07.966 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:05:07 smithi007 ceph-mon[29459]: 3.19 scrub starts 2024-09-18T08:05:07.966 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:05:07 smithi007 ceph-mon[29459]: 3.19 scrub ok 2024-09-18T08:05:07.966 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:05:07 smithi007 ceph-mon[29459]: 6.3 scrub starts 2024-09-18T08:05:07.967 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:05:07 smithi007 ceph-mon[29459]: 6.3 scrub ok 2024-09-18T08:05:07.967 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:05:07 smithi007 ceph-mon[29459]: pgmap v93: 148 pgs: 148 active+clean; 5.3 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 639 B/s rd, 39 MiB/s wr, 617 op/s 2024-09-18T08:05:08.009 INFO:teuthology.orchestra.run.smithi007.stdout:osd.1: {} 2024-09-18T08:05:08.009 INFO:teuthology.orchestra.run.smithi007.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-18T08:05:08.010 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:05:08.012 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:05:08.012 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2' 2024-09-18T08:05:08.062 INFO:teuthology.orchestra.run.smithi007.stdout:osd.2: {} 2024-09-18T08:05:08.062 INFO:teuthology.orchestra.run.smithi007.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-18T08:05:08.102 INFO:teuthology.orchestra.run.smithi007.stdout:osd.3: {} 2024-09-18T08:05:08.102 INFO:teuthology.orchestra.run.smithi007.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-18T08:05:08.113 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4 2024-09-18T08:05:08.137 INFO:teuthology.orchestra.run.smithi007.stdout:osd.4: {} 2024-09-18T08:05:08.137 INFO:teuthology.orchestra.run.smithi007.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-18T08:05:08.140 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:05:08.141 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:05:08.141 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4' 2024-09-18T08:05:08.170 INFO:teuthology.orchestra.run.smithi007.stdout:osd.5: {} 2024-09-18T08:05:08.170 INFO:teuthology.orchestra.run.smithi007.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-18T08:05:08.208 INFO:teuthology.orchestra.run.smithi007.stdout:osd.6: {} 2024-09-18T08:05:08.208 INFO:teuthology.orchestra.run.smithi007.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-18T08:05:08.241 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1 2024-09-18T08:05:08.244 INFO:teuthology.orchestra.run.smithi007.stdout:osd.7: {} 2024-09-18T08:05:08.244 INFO:teuthology.orchestra.run.smithi007.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-18T08:05:08.286 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:05:08.286 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:05:08.286 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1' 2024-09-18T08:05:08.363 INFO:teuthology.orchestra.run.smithi050.stderr:Inferring config /var/lib/ceph/779349ca-7593-11ef-bceb-c7b262605968/mon.b/config 2024-09-18T08:05:08.387 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1 2024-09-18T08:05:08.436 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:05:08.437 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:05:08.437 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1' 2024-09-18T08:05:08.538 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5 2024-09-18T08:05:08.567 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:05:08.568 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:05:08.568 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5' 2024-09-18T08:05:08.586 INFO:tasks.radosbench.radosbench.0.smithi007.stdout: 63 16 92735 92719 91.9554 59.375 0.0114332 0.0108691 2024-09-18T08:05:08.669 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2 2024-09-18T08:05:08.723 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:05:08.723 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:05:08.723 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2' 2024-09-18T08:05:08.746 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:05:08 smithi050 ceph-mon[32371]: 5.8 scrub starts 2024-09-18T08:05:08.747 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:05:08 smithi050 ceph-mon[32371]: 5.8 scrub ok 2024-09-18T08:05:08.747 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:05:08 smithi050 ceph-mon[32371]: 6.14 scrub starts 2024-09-18T08:05:08.747 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:05:08 smithi050 ceph-mon[32371]: 6.14 scrub ok 2024-09-18T08:05:08.747 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:05:08 smithi050 ceph-mon[32371]: 6.a scrub starts 2024-09-18T08:05:08.747 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:05:08 smithi050 ceph-mon[32371]: 6.a scrub ok 2024-09-18T08:05:08.747 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:05:08 smithi050 ceph-mon[32371]: from='client.? 172.21.15.7:0/2616655797' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-18T08:05:08.825 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7 2024-09-18T08:05:08.854 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:05:08.855 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:05:08.855 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7' 2024-09-18T08:05:08.956 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1 2024-09-18T08:05:08.980 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:05:08 smithi007 ceph-mon[39249]: 5.8 scrub starts 2024-09-18T08:05:08.981 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:05:08 smithi007 ceph-mon[39249]: 5.8 scrub ok 2024-09-18T08:05:08.981 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:05:08 smithi007 ceph-mon[39249]: 6.14 scrub starts 2024-09-18T08:05:08.981 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:05:08 smithi007 ceph-mon[39249]: 6.14 scrub ok 2024-09-18T08:05:08.981 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:05:08 smithi007 ceph-mon[39249]: 6.a scrub starts 2024-09-18T08:05:08.981 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:05:08 smithi007 ceph-mon[39249]: 6.a scrub ok 2024-09-18T08:05:08.981 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:05:08 smithi007 ceph-mon[39249]: from='client.? 172.21.15.7:0/2616655797' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-18T08:05:08.981 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:05:08 smithi007 ceph-mon[29459]: 5.8 scrub starts 2024-09-18T08:05:08.981 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:05:08 smithi007 ceph-mon[29459]: 5.8 scrub ok 2024-09-18T08:05:08.982 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:05:08 smithi007 ceph-mon[29459]: 6.14 scrub starts 2024-09-18T08:05:08.982 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:05:08 smithi007 ceph-mon[29459]: 6.14 scrub ok 2024-09-18T08:05:08.982 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:05:08 smithi007 ceph-mon[29459]: 6.a scrub starts 2024-09-18T08:05:08.982 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:05:08 smithi007 ceph-mon[29459]: 6.a scrub ok 2024-09-18T08:05:08.982 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:05:08 smithi007 ceph-mon[29459]: from='client.? 172.21.15.7:0/2616655797' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-18T08:05:08.994 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:05:08.995 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:05:08.996 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1' 2024-09-18T08:05:09.085 INFO:journalctl@ceph.iscsi.iscsi.a.smithi050.stdout:Sep 18 08:05:08 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-iscsi-iscsi-a[57782]: debug there is no tcmu-runner data available 2024-09-18T08:05:09.097 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7 2024-09-18T08:05:09.124 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:05:09.125 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:05:09.125 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7' 2024-09-18T08:05:09.226 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0 2024-09-18T08:05:09.271 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:05:09.272 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:05:09.272 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0' 2024-09-18T08:05:09.373 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3 2024-09-18T08:05:09.413 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:05:09.414 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:05:09.414 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3' 2024-09-18T08:05:09.515 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5 2024-09-18T08:05:09.542 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:05:09.543 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:05:09.543 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5' 2024-09-18T08:05:09.586 INFO:tasks.radosbench.radosbench.0.smithi007.stdout: 64 16 93779 93763 91.5381 65.25 0.0091856 0.0109187 2024-09-18T08:05:09.644 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4 2024-09-18T08:05:09.682 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:05:09.683 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:05:09.684 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4' 2024-09-18T08:05:09.785 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7 2024-09-18T08:05:09.788 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:05:09 smithi050 ceph-mon[32371]: 4.18 scrub starts 2024-09-18T08:05:09.788 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:05:09 smithi050 ceph-mon[32371]: 4.18 scrub ok 2024-09-18T08:05:09.788 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:05:09 smithi050 ceph-mon[32371]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-18T08:05:09.788 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:05:09 smithi050 ceph-mon[32371]: 5.f scrub starts 2024-09-18T08:05:09.788 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:05:09 smithi050 ceph-mon[32371]: 5.f scrub ok 2024-09-18T08:05:09.788 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:05:09 smithi050 ceph-mon[32371]: 3.13 scrub starts 2024-09-18T08:05:09.788 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:05:09 smithi050 ceph-mon[32371]: 3.13 scrub ok 2024-09-18T08:05:09.789 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:05:09 smithi050 ceph-mon[32371]: pgmap v94: 148 pgs: 148 active+clean; 5.6 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 1.1 KiB/s rd, 72 MiB/s wr, 1.15k op/s 2024-09-18T08:05:09.789 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:05:09 smithi050 ceph-mon[32371]: from='mgr.24470 172.21.15.7:0/2594390261' entity='mgr.y' 2024-09-18T08:05:09.789 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:05:09 smithi050 ceph-mon[32371]: from='mgr.24470 172.21.15.7:0/2594390261' entity='mgr.y' 2024-09-18T08:05:09.792 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2024-09-18T08:05:09.792 INFO:teuthology.orchestra.run.smithi050.stdout: "ops": [], 2024-09-18T08:05:09.792 INFO:teuthology.orchestra.run.smithi050.stdout: "complaint_time": 30, 2024-09-18T08:05:09.793 INFO:teuthology.orchestra.run.smithi050.stdout: "num_blocked_ops": 0 2024-09-18T08:05:09.793 INFO:teuthology.orchestra.run.smithi050.stdout:} 2024-09-18T08:05:09.818 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:05:09.819 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:05:09.819 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7' 2024-09-18T08:05:09.920 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5 2024-09-18T08:05:09.946 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:05:09.947 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:05:09.947 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5' 2024-09-18T08:05:09.980 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:05:09 smithi007 ceph-mon[29459]: 4.18 scrub starts 2024-09-18T08:05:09.981 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:05:09 smithi007 ceph-mon[29459]: 4.18 scrub ok 2024-09-18T08:05:09.981 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:05:09 smithi007 ceph-mon[29459]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-18T08:05:09.981 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:05:09 smithi007 ceph-mon[29459]: 5.f scrub starts 2024-09-18T08:05:09.981 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:05:09 smithi007 ceph-mon[29459]: 5.f scrub ok 2024-09-18T08:05:09.981 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:05:09 smithi007 ceph-mon[29459]: 3.13 scrub starts 2024-09-18T08:05:09.981 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:05:09 smithi007 ceph-mon[29459]: 3.13 scrub ok 2024-09-18T08:05:09.981 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:05:09 smithi007 ceph-mon[29459]: pgmap v94: 148 pgs: 148 active+clean; 5.6 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 1.1 KiB/s rd, 72 MiB/s wr, 1.15k op/s 2024-09-18T08:05:09.981 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:05:09 smithi007 ceph-mon[29459]: from='mgr.24470 172.21.15.7:0/2594390261' entity='mgr.y' 2024-09-18T08:05:09.981 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:05:09 smithi007 ceph-mon[29459]: from='mgr.24470 172.21.15.7:0/2594390261' entity='mgr.y' 2024-09-18T08:05:09.981 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:05:09 smithi007 ceph-mon[39249]: 4.18 scrub starts 2024-09-18T08:05:09.982 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:05:09 smithi007 ceph-mon[39249]: 4.18 scrub ok 2024-09-18T08:05:09.982 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:05:09 smithi007 ceph-mon[39249]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-18T08:05:09.982 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:05:09 smithi007 ceph-mon[39249]: 5.f scrub starts 2024-09-18T08:05:09.982 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:05:09 smithi007 ceph-mon[39249]: 5.f scrub ok 2024-09-18T08:05:09.982 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:05:09 smithi007 ceph-mon[39249]: 3.13 scrub starts 2024-09-18T08:05:09.982 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:05:09 smithi007 ceph-mon[39249]: 3.13 scrub ok 2024-09-18T08:05:09.982 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:05:09 smithi007 ceph-mon[39249]: pgmap v94: 148 pgs: 148 active+clean; 5.6 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 1.1 KiB/s rd, 72 MiB/s wr, 1.15k op/s 2024-09-18T08:05:09.982 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:05:09 smithi007 ceph-mon[39249]: from='mgr.24470 172.21.15.7:0/2594390261' entity='mgr.y' 2024-09-18T08:05:09.982 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:05:09 smithi007 ceph-mon[39249]: from='mgr.24470 172.21.15.7:0/2594390261' entity='mgr.y' 2024-09-18T08:05:10.049 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3 2024-09-18T08:05:10.075 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:05:10.076 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:05:10.076 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3' 2024-09-18T08:05:10.177 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1 2024-09-18T08:05:10.214 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:05:10.215 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:05:10.215 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1' 2024-09-18T08:05:10.316 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3 2024-09-18T08:05:10.354 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:05:10.355 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:05:10.356 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3' 2024-09-18T08:05:10.456 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1 2024-09-18T08:05:10.499 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:05:10.500 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:05:10.500 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1' 2024-09-18T08:05:10.586 INFO:tasks.radosbench.radosbench.0.smithi007.stdout: 65 16 94782 94766 91.0942 62.6875 0.01175 0.0109725 2024-09-18T08:05:10.602 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4 2024-09-18T08:05:10.632 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:05:10.632 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:05:10.633 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4' 2024-09-18T08:05:10.734 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2 2024-09-18T08:05:10.775 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:05:10.776 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:05:10.776 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2' 2024-09-18T08:05:10.877 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5 2024-09-18T08:05:10.905 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:05:10.906 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:05:10 smithi050 ceph-mon[32371]: 6.c scrub starts 2024-09-18T08:05:10.906 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:05:10 smithi050 ceph-mon[32371]: 6.c scrub ok 2024-09-18T08:05:10.906 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:05:10 smithi050 ceph-mon[32371]: 6.9 scrub starts 2024-09-18T08:05:10.906 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:05:10 smithi050 ceph-mon[32371]: 6.9 scrub ok 2024-09-18T08:05:10.906 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:05:10 smithi050 ceph-mon[32371]: 6.17 scrub starts 2024-09-18T08:05:10.906 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:05:10 smithi050 ceph-mon[32371]: 6.17 scrub ok 2024-09-18T08:05:10.906 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:05:10.907 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5' 2024-09-18T08:05:10.949 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/779349ca-7593-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T08:05:10.963 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:05:10 smithi007 ceph-mon[29459]: 6.c scrub starts 2024-09-18T08:05:10.963 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:05:10 smithi007 ceph-mon[29459]: 6.c scrub ok 2024-09-18T08:05:10.963 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:05:10 smithi007 ceph-mon[29459]: 6.9 scrub starts 2024-09-18T08:05:10.963 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:05:10 smithi007 ceph-mon[29459]: 6.9 scrub ok 2024-09-18T08:05:10.963 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:05:10 smithi007 ceph-mon[29459]: 6.17 scrub starts 2024-09-18T08:05:10.963 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:05:10 smithi007 ceph-mon[29459]: 6.17 scrub ok 2024-09-18T08:05:10.963 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:05:10 smithi007 ceph-mon[39249]: 6.c scrub starts 2024-09-18T08:05:10.963 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:05:10 smithi007 ceph-mon[39249]: 6.c scrub ok 2024-09-18T08:05:10.964 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:05:10 smithi007 ceph-mon[39249]: 6.9 scrub starts 2024-09-18T08:05:10.964 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:05:10 smithi007 ceph-mon[39249]: 6.9 scrub ok 2024-09-18T08:05:10.964 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:05:10 smithi007 ceph-mon[39249]: 6.17 scrub starts 2024-09-18T08:05:10.964 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:05:10 smithi007 ceph-mon[39249]: 6.17 scrub ok 2024-09-18T08:05:11.007 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7 2024-09-18T08:05:11.033 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:05:11.033 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:05:11.034 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7' 2024-09-18T08:05:11.134 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3 2024-09-18T08:05:11.196 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:05:11.197 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:05:11.197 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3' 2024-09-18T08:05:11.264 DEBUG:teuthology.orchestra.run.smithi050:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid 779349ca-7593-11ef-bceb-c7b262605968 -- ceph daemon osd.5 dump_historic_ops 2024-09-18T08:05:11.298 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2 2024-09-18T08:05:11.350 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:05:11.350 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:05:11.350 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2' 2024-09-18T08:05:11.451 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6 2024-09-18T08:05:11.478 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:05:11.478 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:05:11.479 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6' 2024-09-18T08:05:11.579 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5 2024-09-18T08:05:11.587 INFO:tasks.radosbench.radosbench.0.smithi007.stdout: 66 16 95586 95570 90.4738 50.25 0.0146272 0.011046 2024-09-18T08:05:11.602 INFO:teuthology.orchestra.run.smithi050.stderr:Inferring config /var/lib/ceph/779349ca-7593-11ef-bceb-c7b262605968/mon.b/config 2024-09-18T08:05:11.607 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:05:11.608 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:05:11.608 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5' 2024-09-18T08:05:11.709 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6 2024-09-18T08:05:11.717 INFO:teuthology.orchestra.run.smithi007.stderr:reweighted osd.0 to 1 (10000) 2024-09-18T08:05:11.737 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:05:11.737 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:05:11.738 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6' 2024-09-18T08:05:11.839 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4 2024-09-18T08:05:11.865 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:05:11.866 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:05:11.866 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4' 2024-09-18T08:05:11.967 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3 2024-09-18T08:05:11.981 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:05:11 smithi007 ceph-mon[39249]: 7.f scrub starts 2024-09-18T08:05:11.981 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:05:11 smithi007 ceph-mon[39249]: 7.e scrub starts 2024-09-18T08:05:11.981 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:05:11 smithi007 ceph-mon[39249]: pgmap v95: 148 pgs: 148 active+clean; 5.6 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 1023 B/s rd, 63 MiB/s wr, 1.00k op/s 2024-09-18T08:05:11.981 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:05:11 smithi007 ceph-mon[39249]: from='client.? 172.21.15.7:0/854706864' entity='client.admin' cmd={"prefix": "osd reweight", "id": 0, "weight": 1.0} : dispatch 2024-09-18T08:05:11.981 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:05:11 smithi007 ceph-mon[39249]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 0, "weight": 1.0} : dispatch 2024-09-18T08:05:11.981 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:05:11 smithi007 ceph-mon[29459]: 7.f scrub starts 2024-09-18T08:05:11.981 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:05:11 smithi007 ceph-mon[29459]: 7.e scrub starts 2024-09-18T08:05:11.981 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:05:11 smithi007 ceph-mon[29459]: pgmap v95: 148 pgs: 148 active+clean; 5.6 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 1023 B/s rd, 63 MiB/s wr, 1.00k op/s 2024-09-18T08:05:11.982 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:05:11 smithi007 ceph-mon[29459]: from='client.? 172.21.15.7:0/854706864' entity='client.admin' cmd={"prefix": "osd reweight", "id": 0, "weight": 1.0} : dispatch 2024-09-18T08:05:11.982 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:05:11 smithi007 ceph-mon[29459]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 0, "weight": 1.0} : dispatch 2024-09-18T08:05:11.982 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:05:11 smithi050 ceph-mon[32371]: 7.f scrub starts 2024-09-18T08:05:11.982 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:05:11 smithi050 ceph-mon[32371]: 7.e scrub starts 2024-09-18T08:05:11.982 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:05:11 smithi050 ceph-mon[32371]: pgmap v95: 148 pgs: 148 active+clean; 5.6 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 1023 B/s rd, 63 MiB/s wr, 1.00k op/s 2024-09-18T08:05:11.982 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:05:11 smithi050 ceph-mon[32371]: from='client.? 172.21.15.7:0/854706864' entity='client.admin' cmd={"prefix": "osd reweight", "id": 0, "weight": 1.0} : dispatch 2024-09-18T08:05:11.982 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:05:11 smithi050 ceph-mon[32371]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 0, "weight": 1.0} : dispatch 2024-09-18T08:05:12.009 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:05:12.010 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:05:12.010 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3' 2024-09-18T08:05:12.111 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5 2024-09-18T08:05:12.137 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:05:12.138 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:05:12.138 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5' 2024-09-18T08:05:12.239 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6 2024-09-18T08:05:12.265 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:05:12.265 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2024-09-18T08:05:12.266 INFO:teuthology.orchestra.run.smithi050.stdout: "size": 20, 2024-09-18T08:05:12.266 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 600, 2024-09-18T08:05:12.266 INFO:teuthology.orchestra.run.smithi050.stdout: "ops": [ 2024-09-18T08:05:12.266 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:05:12.266 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.14805.0:78863 7.0 7:0ec89ba8:::benchmark_data_smithi007_73155_object78862:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e104)", 2024-09-18T08:05:12.266 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-09-18T08:04:55.147639+0000", 2024-09-18T08:05:12.266 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 17.113019756, 2024-09-18T08:05:12.266 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.034894189999999999, 2024-09-18T08:05:12.266 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-09-18T08:05:12.266 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-09-18T08:05:12.266 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T08:05:12.266 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2024-09-18T08:05:12.266 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.14805", 2024-09-18T08:05:12.266 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "172.21.15.7:0/640320436", 2024-09-18T08:05:12.267 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 78863 2024-09-18T08:05:12.267 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:05:12.267 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-09-18T08:05:12.267 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:05:12.267 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-09-18T08:05:12.267 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:55.147639+0000", 2024-09-18T08:05:12.267 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-18T08:05:12.267 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:05:12.267 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:05:12.267 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-09-18T08:05:12.268 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:55.147639+0000", 2024-09-18T08:05:12.268 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-18T08:05:12.268 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:05:12.268 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:05:12.268 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-09-18T08:05:12.268 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:55.147640+0000", 2024-09-18T08:05:12.268 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 8.6099999999999999e-07 2024-09-18T08:05:12.268 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:05:12.268 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:05:12.268 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-09-18T08:05:12.268 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:55.147674+0000", 2024-09-18T08:05:12.268 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 3.3989999999999998e-05 2024-09-18T08:05:12.268 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:05:12.269 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:05:12.269 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-09-18T08:05:12.269 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:55.147685+0000", 2024-09-18T08:05:12.269 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.1409999999999999e-05 2024-09-18T08:05:12.269 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:05:12.269 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:05:12.269 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-09-18T08:05:12.269 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:55.147693+0000", 2024-09-18T08:05:12.269 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 7.4919999999999996e-06 2024-09-18T08:05:12.269 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:05:12.269 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:05:12.269 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-18T08:05:12.270 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:55.149612+0000", 2024-09-18T08:05:12.270 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.001919752 2024-09-18T08:05:12.270 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:05:12.270 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:05:12.270 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-09-18T08:05:12.270 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:55.149755+0000", 2024-09-18T08:05:12.270 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00014244600000000001 2024-09-18T08:05:12.270 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:05:12.270 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:05:12.270 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "waiting for subops from 1,3", 2024-09-18T08:05:12.270 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:55.149941+0000", 2024-09-18T08:05:12.271 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00018651 2024-09-18T08:05:12.271 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:05:12.271 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:05:12.271 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "op_commit", 2024-09-18T08:05:12.271 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:55.153206+0000", 2024-09-18T08:05:12.271 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0032643960000000001 2024-09-18T08:05:12.271 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:05:12.271 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:05:12.271 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:05:12.271 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:55.154045+0000", 2024-09-18T08:05:12.271 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00083944300000000005 2024-09-18T08:05:12.271 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:05:12.272 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:05:12.272 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:05:12.272 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:55.182448+0000", 2024-09-18T08:05:12.272 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.028402502 2024-09-18T08:05:12.272 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:05:12.272 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:05:12.272 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "commit_sent", 2024-09-18T08:05:12.272 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:55.182484+0000", 2024-09-18T08:05:12.272 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 3.6708999999999997e-05 2024-09-18T08:05:12.272 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:05:12.272 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:05:12.273 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-09-18T08:05:12.273 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:55.182533+0000", 2024-09-18T08:05:12.273 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4.8678999999999999e-05 2024-09-18T08:05:12.273 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-18T08:05:12.273 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-09-18T08:05:12.273 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-18T08:05:12.273 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:05:12.273 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:05:12.273 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.14805.0:78871 7.0 7:080069fb:::benchmark_data_smithi007_73155_object78870:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e104)", 2024-09-18T08:05:12.273 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-09-18T08:04:55.156246+0000", 2024-09-18T08:05:12.273 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 17.104412623000002, 2024-09-18T08:05:12.274 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.045731885999999999, 2024-09-18T08:05:12.274 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-09-18T08:05:12.274 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-09-18T08:05:12.274 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T08:05:12.274 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2024-09-18T08:05:12.274 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.14805", 2024-09-18T08:05:12.274 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "172.21.15.7:0/640320436", 2024-09-18T08:05:12.274 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 78871 2024-09-18T08:05:12.274 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:05:12.274 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-09-18T08:05:12.274 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:05:12.274 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-09-18T08:05:12.275 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:55.156246+0000", 2024-09-18T08:05:12.275 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-18T08:05:12.275 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:05:12.275 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:05:12.275 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-09-18T08:05:12.275 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:55.156246+0000", 2024-09-18T08:05:12.275 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-18T08:05:12.275 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:05:12.275 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:05:12.276 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-09-18T08:05:12.276 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:55.156248+0000", 2024-09-18T08:05:12.276 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.592e-06 2024-09-18T08:05:12.276 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:05:12.276 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:05:12.276 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-09-18T08:05:12.276 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:55.156298+0000", 2024-09-18T08:05:12.276 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 5.0015999999999997e-05 2024-09-18T08:05:12.276 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:05:12.276 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:05:12.276 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-09-18T08:05:12.277 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:55.156310+0000", 2024-09-18T08:05:12.277 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.2099000000000001e-05 2024-09-18T08:05:12.277 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:05:12.277 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:05:12.277 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-09-18T08:05:12.277 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:55.156318+0000", 2024-09-18T08:05:12.278 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 8.6090000000000001e-06 2024-09-18T08:05:12.278 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:05:12.278 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:05:12.278 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-18T08:05:12.278 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:55.156365+0000", 2024-09-18T08:05:12.278 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4.6585999999999997e-05 2024-09-18T08:05:12.278 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:05:12.278 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:05:12.278 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-09-18T08:05:12.278 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:55.156469+0000", 2024-09-18T08:05:12.278 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000103985 2024-09-18T08:05:12.278 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:05:12.278 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:05:12.278 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "waiting for subops from 1,3", 2024-09-18T08:05:12.279 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:55.156563+0000", 2024-09-18T08:05:12.279 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 9.3677000000000005e-05 2024-09-18T08:05:12.279 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:05:12.279 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:05:12.279 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "op_commit", 2024-09-18T08:05:12.279 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:55.160979+0000", 2024-09-18T08:05:12.279 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0044160500000000004 2024-09-18T08:05:12.279 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:05:12.279 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:05:12.279 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:05:12.279 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:55.161191+0000", 2024-09-18T08:05:12.279 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000212248 2024-09-18T08:05:12.279 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:05:12.280 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:05:12.280 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:05:12.280 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:55.201868+0000", 2024-09-18T08:05:12.280 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.040676599000000001 2024-09-18T08:05:12.280 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:05:12.280 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:05:12.280 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "commit_sent", 2024-09-18T08:05:12.280 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:55.201935+0000", 2024-09-18T08:05:12.280 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 6.7688999999999995e-05 2024-09-18T08:05:12.281 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:05:12.281 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:05:12.281 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-09-18T08:05:12.281 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:55.201978+0000", 2024-09-18T08:05:12.281 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4.2735999999999999e-05 2024-09-18T08:05:12.281 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-18T08:05:12.281 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-09-18T08:05:12.281 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-18T08:05:12.281 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:05:12.281 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:05:12.281 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.14805.0:78873 7.0 7:089fc6f7:::benchmark_data_smithi007_73155_object78872:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e104)", 2024-09-18T08:05:12.281 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-09-18T08:04:55.156506+0000", 2024-09-18T08:05:12.281 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 17.104153106999998, 2024-09-18T08:05:12.282 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.048621722999999999, 2024-09-18T08:05:12.282 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-09-18T08:05:12.282 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-09-18T08:05:12.282 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T08:05:12.282 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:05:12.283 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6' 2024-09-18T08:05:12.283 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2024-09-18T08:05:12.283 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.14805", 2024-09-18T08:05:12.284 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "172.21.15.7:0/640320436", 2024-09-18T08:05:12.284 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 78873 2024-09-18T08:05:12.284 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:05:12.284 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-09-18T08:05:12.284 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:05:12.284 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-09-18T08:05:12.284 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:55.156506+0000", 2024-09-18T08:05:12.284 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-18T08:05:12.284 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:05:12.284 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:05:12.284 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-09-18T08:05:12.284 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:55.156506+0000", 2024-09-18T08:05:12.284 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-18T08:05:12.285 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:05:12.285 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:05:12.285 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-09-18T08:05:12.285 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:55.156506+0000", 2024-09-18T08:05:12.285 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 7.3300000000000001e-07 2024-09-18T08:05:12.285 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:05:12.285 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:05:12.285 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-09-18T08:05:12.286 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:55.156586+0000", 2024-09-18T08:05:12.286 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 7.9907999999999995e-05 2024-09-18T08:05:12.286 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:05:12.286 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:05:12.286 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-09-18T08:05:12.286 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:55.156596+0000", 2024-09-18T08:05:12.286 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 9.6840000000000008e-06 2024-09-18T08:05:12.286 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:05:12.286 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:05:12.286 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-09-18T08:05:12.286 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:55.156603+0000", 2024-09-18T08:05:12.286 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 7.1550000000000001e-06 2024-09-18T08:05:12.286 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:05:12.286 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:05:12.286 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-18T08:05:12.287 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:55.158535+0000", 2024-09-18T08:05:12.287 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0019323980000000001 2024-09-18T08:05:12.287 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:05:12.287 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:05:12.287 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-09-18T08:05:12.287 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:55.158671+0000", 2024-09-18T08:05:12.287 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00013509000000000001 2024-09-18T08:05:12.287 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:05:12.287 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:05:12.287 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "waiting for subops from 1,3", 2024-09-18T08:05:12.288 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:55.158801+0000", 2024-09-18T08:05:12.288 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00013068199999999999 2024-09-18T08:05:12.288 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:05:12.288 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:05:12.288 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "op_commit", 2024-09-18T08:05:12.288 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:55.161028+0000", 2024-09-18T08:05:12.288 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0022270580000000001 2024-09-18T08:05:12.288 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:05:12.288 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:05:12.288 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:05:12.289 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:55.165591+0000", 2024-09-18T08:05:12.289 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0045623189999999996 2024-09-18T08:05:12.289 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:05:12.289 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:05:12.289 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:05:12.289 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:55.205032+0000", 2024-09-18T08:05:12.289 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.039441468 2024-09-18T08:05:12.289 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:05:12.289 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:05:12.289 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "commit_sent", 2024-09-18T08:05:12.289 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:55.205096+0000", 2024-09-18T08:05:12.289 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 6.3904000000000003e-05 2024-09-18T08:05:12.289 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:05:12.289 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:05:12.290 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-09-18T08:05:12.290 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:55.205127+0000", 2024-09-18T08:05:12.290 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 3.1324e-05 2024-09-18T08:05:12.290 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-18T08:05:12.290 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-09-18T08:05:12.290 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-18T08:05:12.290 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:05:12.290 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:05:12.290 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.14805.0:83596 7.0 7:0f570868:::benchmark_data_smithi007_73155_object83595:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e106)", 2024-09-18T08:05:12.290 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-09-18T08:04:59.687170+0000", 2024-09-18T08:05:12.290 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 12.573488687999999, 2024-09-18T08:05:12.291 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.033623657000000001, 2024-09-18T08:05:12.291 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-09-18T08:05:12.291 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-09-18T08:05:12.291 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T08:05:12.291 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2024-09-18T08:05:12.291 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.14805", 2024-09-18T08:05:12.291 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "172.21.15.7:0/640320436", 2024-09-18T08:05:12.291 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 83596 2024-09-18T08:05:12.291 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:05:12.292 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-09-18T08:05:12.292 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:05:12.292 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-09-18T08:05:12.292 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:59.687170+0000", 2024-09-18T08:05:12.292 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-18T08:05:12.292 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:05:12.292 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:05:12.292 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-09-18T08:05:12.293 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:59.687170+0000", 2024-09-18T08:05:12.293 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-18T08:05:12.293 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:05:12.293 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:05:12.293 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-09-18T08:05:12.293 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:59.687172+0000", 2024-09-18T08:05:12.293 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 2.0870000000000001e-06 2024-09-18T08:05:12.293 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:05:12.293 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:05:12.293 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-09-18T08:05:12.294 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:59.687216+0000", 2024-09-18T08:05:12.294 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4.4109000000000001e-05 2024-09-18T08:05:12.294 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:05:12.294 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:05:12.294 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-09-18T08:05:12.294 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:59.687231+0000", 2024-09-18T08:05:12.294 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.5038000000000001e-05 2024-09-18T08:05:12.294 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:05:12.295 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:05:12.295 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-09-18T08:05:12.295 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:59.687239+0000", 2024-09-18T08:05:12.295 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 8.0129999999999993e-06 2024-09-18T08:05:12.295 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:05:12.295 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:05:12.295 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-18T08:05:12.295 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:59.687292+0000", 2024-09-18T08:05:12.295 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 5.2580999999999997e-05 2024-09-18T08:05:12.295 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:05:12.295 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:05:12.295 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-09-18T08:05:12.296 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:59.687400+0000", 2024-09-18T08:05:12.296 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00010808500000000001 2024-09-18T08:05:12.296 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:05:12.296 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:05:12.296 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "waiting for subops from 1,3", 2024-09-18T08:05:12.296 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:59.687507+0000", 2024-09-18T08:05:12.296 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0001069 2024-09-18T08:05:12.296 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:05:12.296 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:05:12.296 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "op_commit", 2024-09-18T08:05:12.297 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:59.689475+0000", 2024-09-18T08:05:12.297 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0019681519999999999 2024-09-18T08:05:12.297 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:05:12.297 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:05:12.297 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:05:12.297 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:59.697150+0000", 2024-09-18T08:05:12.297 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.007675248 2024-09-18T08:05:12.297 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:05:12.297 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:05:12.297 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:05:12.297 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:59.720647+0000", 2024-09-18T08:05:12.297 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.023496547 2024-09-18T08:05:12.297 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:05:12.298 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:05:12.298 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "commit_sent", 2024-09-18T08:05:12.298 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:59.720744+0000", 2024-09-18T08:05:12.298 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 9.7063000000000002e-05 2024-09-18T08:05:12.298 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:05:12.298 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:05:12.298 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-09-18T08:05:12.298 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:59.720794+0000", 2024-09-18T08:05:12.298 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4.9833999999999998e-05 2024-09-18T08:05:12.298 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-18T08:05:12.298 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-09-18T08:05:12.298 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-18T08:05:12.298 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:05:12.299 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:05:12.299 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.14805.0:83840 7.0 7:0bfbbf38:::benchmark_data_smithi007_73155_object83839:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e106)", 2024-09-18T08:05:12.299 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-09-18T08:04:59.913304+0000", 2024-09-18T08:05:12.299 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 12.347354425000001, 2024-09-18T08:05:12.299 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.050200109999999999, 2024-09-18T08:05:12.299 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-09-18T08:05:12.299 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-09-18T08:05:12.299 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T08:05:12.299 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2024-09-18T08:05:12.299 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.14805", 2024-09-18T08:05:12.299 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "172.21.15.7:0/640320436", 2024-09-18T08:05:12.299 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 83840 2024-09-18T08:05:12.299 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:05:12.299 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-09-18T08:05:12.300 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:05:12.300 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-09-18T08:05:12.300 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:59.913304+0000", 2024-09-18T08:05:12.300 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-18T08:05:12.300 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:05:12.300 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:05:12.300 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-09-18T08:05:12.300 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:59.913304+0000", 2024-09-18T08:05:12.300 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-18T08:05:12.300 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:05:12.300 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:05:12.300 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-09-18T08:05:12.300 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:59.913306+0000", 2024-09-18T08:05:12.301 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.556e-06 2024-09-18T08:05:12.301 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:05:12.301 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:05:12.301 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-09-18T08:05:12.301 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:59.913339+0000", 2024-09-18T08:05:12.301 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 3.2993000000000001e-05 2024-09-18T08:05:12.301 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:05:12.301 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:05:12.301 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-09-18T08:05:12.301 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:59.913371+0000", 2024-09-18T08:05:12.302 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 3.2468999999999997e-05 2024-09-18T08:05:12.302 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:05:12.302 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:05:12.302 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-09-18T08:05:12.302 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:59.913395+0000", 2024-09-18T08:05:12.302 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 2.4119999999999999e-05 2024-09-18T08:05:12.302 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:05:12.302 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:05:12.302 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-18T08:05:12.302 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:59.913451+0000", 2024-09-18T08:05:12.302 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 5.5958000000000002e-05 2024-09-18T08:05:12.302 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:05:12.303 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:05:12.303 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-09-18T08:05:12.303 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:59.913718+0000", 2024-09-18T08:05:12.303 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00026679699999999999 2024-09-18T08:05:12.303 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:05:12.303 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:05:12.303 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "waiting for subops from 1,3", 2024-09-18T08:05:12.303 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:59.913851+0000", 2024-09-18T08:05:12.303 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000133065 2024-09-18T08:05:12.303 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:05:12.303 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:05:12.303 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "op_commit", 2024-09-18T08:05:12.303 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:59.920394+0000", 2024-09-18T08:05:12.304 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.006542918 2024-09-18T08:05:12.304 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:05:12.304 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:05:12.304 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:05:12.304 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:59.926438+0000", 2024-09-18T08:05:12.304 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0060434549999999997 2024-09-18T08:05:12.304 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:05:12.304 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:05:12.304 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:05:12.304 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:59.963422+0000", 2024-09-18T08:05:12.305 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.036984224000000003 2024-09-18T08:05:12.305 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:05:12.305 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:05:12.305 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "commit_sent", 2024-09-18T08:05:12.305 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:59.963467+0000", 2024-09-18T08:05:12.305 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4.4724000000000001e-05 2024-09-18T08:05:12.305 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:05:12.305 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:05:12.305 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-09-18T08:05:12.305 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:59.963504+0000", 2024-09-18T08:05:12.305 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 3.7830999999999997e-05 2024-09-18T08:05:12.305 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-18T08:05:12.305 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-09-18T08:05:12.306 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-18T08:05:12.306 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:05:12.306 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:05:12.306 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.14805.0:83844 7.0 7:094de7b0:::benchmark_data_smithi007_73155_object83843:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e106)", 2024-09-18T08:05:12.306 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-09-18T08:04:59.914155+0000", 2024-09-18T08:05:12.306 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 12.346503907000001, 2024-09-18T08:05:12.306 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.050666481999999999, 2024-09-18T08:05:12.306 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-09-18T08:05:12.307 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-09-18T08:05:12.307 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T08:05:12.307 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2024-09-18T08:05:12.307 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.14805", 2024-09-18T08:05:12.307 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "172.21.15.7:0/640320436", 2024-09-18T08:05:12.307 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 83844 2024-09-18T08:05:12.307 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:05:12.307 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-09-18T08:05:12.307 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:05:12.307 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-09-18T08:05:12.307 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:59.914155+0000", 2024-09-18T08:05:12.308 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-18T08:05:12.308 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:05:12.308 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:05:12.308 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-09-18T08:05:12.308 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:59.914155+0000", 2024-09-18T08:05:12.308 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-18T08:05:12.308 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:05:12.308 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:05:12.308 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-09-18T08:05:12.308 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:59.914156+0000", 2024-09-18T08:05:12.308 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 8.8700000000000004e-07 2024-09-18T08:05:12.308 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:05:12.308 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:05:12.309 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-09-18T08:05:12.309 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:59.914219+0000", 2024-09-18T08:05:12.309 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 6.3171999999999998e-05 2024-09-18T08:05:12.309 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:05:12.309 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:05:12.309 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-09-18T08:05:12.309 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:59.914232+0000", 2024-09-18T08:05:12.309 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.3512999999999999e-05 2024-09-18T08:05:12.309 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:05:12.309 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:05:12.309 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-09-18T08:05:12.309 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:59.914240+0000", 2024-09-18T08:05:12.309 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 7.43e-06 2024-09-18T08:05:12.310 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:05:12.310 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:05:12.310 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-18T08:05:12.310 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:59.917447+0000", 2024-09-18T08:05:12.310 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00320704 2024-09-18T08:05:12.310 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:05:12.310 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:05:12.310 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-09-18T08:05:12.310 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:59.917581+0000", 2024-09-18T08:05:12.310 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00013440500000000001 2024-09-18T08:05:12.310 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:05:12.310 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:05:12.310 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "waiting for subops from 1,3", 2024-09-18T08:05:12.310 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:59.917733+0000", 2024-09-18T08:05:12.311 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000151664 2024-09-18T08:05:12.311 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:05:12.311 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:05:12.311 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "op_commit", 2024-09-18T08:05:12.311 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:59.926449+0000", 2024-09-18T08:05:12.311 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0087161979999999997 2024-09-18T08:05:12.311 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:05:12.311 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:05:12.311 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:05:12.311 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:59.964175+0000", 2024-09-18T08:05:12.312 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.037726350999999998 2024-09-18T08:05:12.312 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:05:12.312 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:05:12.312 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:05:12.312 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:59.964745+0000", 2024-09-18T08:05:12.312 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00056970199999999995 2024-09-18T08:05:12.312 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:05:12.312 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:05:12.312 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "commit_sent", 2024-09-18T08:05:12.312 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:59.964788+0000", 2024-09-18T08:05:12.312 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4.2661999999999999e-05 2024-09-18T08:05:12.312 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:05:12.312 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:05:12.313 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-09-18T08:05:12.313 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:59.964821+0000", 2024-09-18T08:05:12.313 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 3.3457999999999997e-05 2024-09-18T08:05:12.313 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-18T08:05:12.313 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-09-18T08:05:12.313 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-18T08:05:12.313 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:05:12.313 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:05:12.313 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.14805.0:83848 7.0 7:06de8668:::benchmark_data_smithi007_73155_object83847:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e106)", 2024-09-18T08:05:12.313 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-09-18T08:04:59.916621+0000", 2024-09-18T08:05:12.313 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 12.344037939, 2024-09-18T08:05:12.313 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.057821658999999997, 2024-09-18T08:05:12.314 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-09-18T08:05:12.314 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-09-18T08:05:12.314 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T08:05:12.314 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2024-09-18T08:05:12.314 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.14805", 2024-09-18T08:05:12.314 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "172.21.15.7:0/640320436", 2024-09-18T08:05:12.314 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 83848 2024-09-18T08:05:12.314 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:05:12.314 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-09-18T08:05:12.314 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:05:12.314 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-09-18T08:05:12.314 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:59.916621+0000", 2024-09-18T08:05:12.314 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-18T08:05:12.314 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:05:12.315 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:05:12.315 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-09-18T08:05:12.315 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:59.916621+0000", 2024-09-18T08:05:12.315 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-18T08:05:12.315 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:05:12.315 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:05:12.315 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-09-18T08:05:12.315 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:59.916622+0000", 2024-09-18T08:05:12.315 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.3230000000000001e-06 2024-09-18T08:05:12.315 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:05:12.315 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:05:12.315 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-09-18T08:05:12.315 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:59.916648+0000", 2024-09-18T08:05:12.316 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 2.6378999999999999e-05 2024-09-18T08:05:12.316 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:05:12.316 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:05:12.316 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-09-18T08:05:12.316 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:59.916657+0000", 2024-09-18T08:05:12.316 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 8.884e-06 2024-09-18T08:05:12.316 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:05:12.316 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:05:12.316 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-09-18T08:05:12.316 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:59.916665+0000", 2024-09-18T08:05:12.316 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 7.5469999999999999e-06 2024-09-18T08:05:12.317 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:05:12.317 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:05:12.317 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-18T08:05:12.317 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:59.920431+0000", 2024-09-18T08:05:12.317 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00376589 2024-09-18T08:05:12.317 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:05:12.317 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:05:12.317 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-09-18T08:05:12.317 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:59.920578+0000", 2024-09-18T08:05:12.317 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00014743100000000001 2024-09-18T08:05:12.317 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:05:12.318 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:05:12.318 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "waiting for subops from 1,3", 2024-09-18T08:05:12.318 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:59.920971+0000", 2024-09-18T08:05:12.318 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00039322199999999998 2024-09-18T08:05:12.318 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:05:12.318 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:05:12.318 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "op_commit", 2024-09-18T08:05:12.318 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:59.926452+0000", 2024-09-18T08:05:12.318 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.005480581 2024-09-18T08:05:12.318 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:05:12.319 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:05:12.319 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:05:12.319 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:59.974303+0000", 2024-09-18T08:05:12.319 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.047851298 2024-09-18T08:05:12.319 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:05:12.319 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:05:12.319 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:05:12.319 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:59.974341+0000", 2024-09-18T08:05:12.319 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 3.7558000000000002e-05 2024-09-18T08:05:12.319 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:05:12.319 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:05:12.320 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "commit_sent", 2024-09-18T08:05:12.320 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:59.974396+0000", 2024-09-18T08:05:12.320 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 5.4988999999999998e-05 2024-09-18T08:05:12.320 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:05:12.320 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:05:12.320 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-09-18T08:05:12.320 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:59.974442+0000", 2024-09-18T08:05:12.320 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4.6557000000000002e-05 2024-09-18T08:05:12.320 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-18T08:05:12.320 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-09-18T08:05:12.320 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-18T08:05:12.320 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:05:12.320 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:05:12.321 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.14805.0:83852 7.0 7:0f289add:::benchmark_data_smithi007_73155_object83851:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e106)", 2024-09-18T08:05:12.321 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-09-18T08:04:59.920741+0000", 2024-09-18T08:05:12.321 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 12.339917894999999, 2024-09-18T08:05:12.321 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.054000911999999998, 2024-09-18T08:05:12.321 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-09-18T08:05:12.321 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-09-18T08:05:12.321 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T08:05:12.321 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2024-09-18T08:05:12.321 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.14805", 2024-09-18T08:05:12.321 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "172.21.15.7:0/640320436", 2024-09-18T08:05:12.321 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 83852 2024-09-18T08:05:12.321 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:05:12.322 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-09-18T08:05:12.322 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:05:12.322 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-09-18T08:05:12.322 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:59.920741+0000", 2024-09-18T08:05:12.322 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-18T08:05:12.323 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:05:12.323 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:05:12.324 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-09-18T08:05:12.324 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:59.920741+0000", 2024-09-18T08:05:12.324 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-18T08:05:12.324 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:05:12.324 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:05:12.324 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-09-18T08:05:12.324 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:59.920742+0000", 2024-09-18T08:05:12.324 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.601e-06 2024-09-18T08:05:12.324 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:05:12.324 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:05:12.324 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-09-18T08:05:12.324 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:59.920800+0000", 2024-09-18T08:05:12.325 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 5.7713999999999997e-05 2024-09-18T08:05:12.325 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:05:12.325 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:05:12.325 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-09-18T08:05:12.325 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:59.920812+0000", 2024-09-18T08:05:12.325 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.1499e-05 2024-09-18T08:05:12.325 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:05:12.325 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:05:12.325 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-09-18T08:05:12.325 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:59.920820+0000", 2024-09-18T08:05:12.325 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 8.3229999999999998e-06 2024-09-18T08:05:12.325 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:05:12.326 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:05:12.326 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-18T08:05:12.326 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:59.923610+0000", 2024-09-18T08:05:12.326 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.002789733 2024-09-18T08:05:12.326 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:05:12.326 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:05:12.326 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-09-18T08:05:12.326 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:59.923732+0000", 2024-09-18T08:05:12.326 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00012276600000000001 2024-09-18T08:05:12.326 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:05:12.327 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:05:12.327 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "waiting for subops from 1,3", 2024-09-18T08:05:12.327 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:59.923887+0000", 2024-09-18T08:05:12.327 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00015416400000000001 2024-09-18T08:05:12.327 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:05:12.327 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:05:12.327 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "op_commit", 2024-09-18T08:05:12.328 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:59.926508+0000", 2024-09-18T08:05:12.328 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0026218610000000001 2024-09-18T08:05:12.328 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:05:12.328 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:05:12.328 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:05:12.328 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:59.974510+0000", 2024-09-18T08:05:12.328 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.048001180999999997 2024-09-18T08:05:12.328 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:05:12.329 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:05:12.329 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:05:12.329 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:59.974590+0000", 2024-09-18T08:05:12.329 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 8.0632999999999999e-05 2024-09-18T08:05:12.329 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:05:12.329 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:05:12.329 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "commit_sent", 2024-09-18T08:05:12.329 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:59.974685+0000", 2024-09-18T08:05:12.329 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 9.4512000000000006e-05 2024-09-18T08:05:12.329 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:05:12.330 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:05:12.330 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-09-18T08:05:12.330 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:59.974742+0000", 2024-09-18T08:05:12.330 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 5.6925000000000001e-05 2024-09-18T08:05:12.330 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-18T08:05:12.330 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-09-18T08:05:12.330 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-18T08:05:12.330 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:05:12.330 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:05:12.330 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.14805.0:84124 7.0 7:058e82c9:::benchmark_data_smithi007_73155_object84123:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e106)", 2024-09-18T08:05:12.330 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-09-18T08:05:00.189181+0000", 2024-09-18T08:05:12.330 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 12.071478036, 2024-09-18T08:05:12.330 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.046225779000000002, 2024-09-18T08:05:12.330 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-09-18T08:05:12.330 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-09-18T08:05:12.331 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T08:05:12.331 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2024-09-18T08:05:12.331 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.14805", 2024-09-18T08:05:12.331 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "172.21.15.7:0/640320436", 2024-09-18T08:05:12.331 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 84124 2024-09-18T08:05:12.331 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:05:12.331 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-09-18T08:05:12.331 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:05:12.331 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-09-18T08:05:12.331 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:05:00.189181+0000", 2024-09-18T08:05:12.331 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-18T08:05:12.331 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:05:12.331 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:05:12.331 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-09-18T08:05:12.331 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:05:00.189181+0000", 2024-09-18T08:05:12.332 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-18T08:05:12.332 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:05:12.332 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:05:12.332 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-09-18T08:05:12.332 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:05:00.189182+0000", 2024-09-18T08:05:12.332 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.7060000000000001e-06 2024-09-18T08:05:12.332 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:05:12.332 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:05:12.332 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-09-18T08:05:12.332 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:05:00.189240+0000", 2024-09-18T08:05:12.332 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 5.7469000000000002e-05 2024-09-18T08:05:12.332 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:05:12.332 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:05:12.332 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-09-18T08:05:12.333 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:05:00.189252+0000", 2024-09-18T08:05:12.333 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.2604e-05 2024-09-18T08:05:12.333 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:05:12.333 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:05:12.333 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-09-18T08:05:12.333 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:05:00.189261+0000", 2024-09-18T08:05:12.333 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 8.4030000000000007e-06 2024-09-18T08:05:12.333 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:05:12.333 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:05:12.333 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-18T08:05:12.333 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:05:00.201890+0000", 2024-09-18T08:05:12.333 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.01262869 2024-09-18T08:05:12.333 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:05:12.333 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:05:12.334 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-09-18T08:05:12.334 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:05:00.202046+0000", 2024-09-18T08:05:12.335 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000156057 2024-09-18T08:05:12.335 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:05:12.335 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:05:12.335 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "waiting for subops from 1,3", 2024-09-18T08:05:12.335 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:05:00.202205+0000", 2024-09-18T08:05:12.336 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000159841 2024-09-18T08:05:12.336 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:05:12.336 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:05:12.336 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "op_commit", 2024-09-18T08:05:12.336 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:05:00.205188+0000", 2024-09-18T08:05:12.336 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0029820979999999999 2024-09-18T08:05:12.336 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:05:12.336 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:05:12.336 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:05:12.336 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:05:00.209218+0000", 2024-09-18T08:05:12.336 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0040303930000000002 2024-09-18T08:05:12.337 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:05:12.337 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:05:12.337 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:05:12.337 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:05:00.235289+0000", 2024-09-18T08:05:12.337 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.026071238 2024-09-18T08:05:12.337 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:05:12.337 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:05:12.337 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "commit_sent", 2024-09-18T08:05:12.337 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:05:00.235333+0000", 2024-09-18T08:05:12.337 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4.4234999999999999e-05 2024-09-18T08:05:12.338 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:05:12.338 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:05:12.338 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-09-18T08:05:12.338 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:05:00.235406+0000", 2024-09-18T08:05:12.338 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 7.3045000000000003e-05 2024-09-18T08:05:12.338 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-18T08:05:12.338 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-09-18T08:05:12.338 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-18T08:05:12.338 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:05:12.338 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:05:12.338 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.14805.0:84222 7.0 7:07cd4bf7:::benchmark_data_smithi007_73155_object84221:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e106)", 2024-09-18T08:05:12.338 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-09-18T08:05:00.296329+0000", 2024-09-18T08:05:12.338 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 11.964329329, 2024-09-18T08:05:12.339 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.033453778000000003, 2024-09-18T08:05:12.339 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-09-18T08:05:12.339 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-09-18T08:05:12.339 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T08:05:12.339 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2024-09-18T08:05:12.339 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.14805", 2024-09-18T08:05:12.339 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "172.21.15.7:0/640320436", 2024-09-18T08:05:12.339 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 84222 2024-09-18T08:05:12.340 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:05:12.340 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-09-18T08:05:12.340 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:05:12.340 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-09-18T08:05:12.340 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:05:00.296329+0000", 2024-09-18T08:05:12.340 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-18T08:05:12.340 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:05:12.340 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:05:12.340 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-09-18T08:05:12.340 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:05:00.296329+0000", 2024-09-18T08:05:12.341 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-18T08:05:12.341 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:05:12.341 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:05:12.341 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-09-18T08:05:12.341 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:05:00.296331+0000", 2024-09-18T08:05:12.341 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.5880000000000001e-06 2024-09-18T08:05:12.341 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:05:12.341 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:05:12.341 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-09-18T08:05:12.341 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:05:00.296363+0000", 2024-09-18T08:05:12.341 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 3.2461e-05 2024-09-18T08:05:12.341 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:05:12.341 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:05:12.342 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-09-18T08:05:12.342 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:05:00.296374+0000", 2024-09-18T08:05:12.342 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.0896999999999999e-05 2024-09-18T08:05:12.342 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:05:12.342 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:05:12.342 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-09-18T08:05:12.342 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:05:00.296381+0000", 2024-09-18T08:05:12.342 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 6.985e-06 2024-09-18T08:05:12.342 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:05:12.343 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:05:12.343 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-18T08:05:12.343 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:05:00.307988+0000", 2024-09-18T08:05:12.343 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.011607105 2024-09-18T08:05:12.343 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:05:12.343 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:05:12.343 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-09-18T08:05:12.343 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:05:00.308157+0000", 2024-09-18T08:05:12.343 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00016876099999999999 2024-09-18T08:05:12.343 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:05:12.343 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:05:12.343 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "waiting for subops from 1,3", 2024-09-18T08:05:12.344 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:05:00.308304+0000", 2024-09-18T08:05:12.344 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00014726100000000001 2024-09-18T08:05:12.344 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:05:12.344 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:05:12.344 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "op_commit", 2024-09-18T08:05:12.344 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:05:00.315850+0000", 2024-09-18T08:05:12.344 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.007545782 2024-09-18T08:05:12.344 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:05:12.344 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:05:12.344 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:05:12.344 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:05:00.318444+0000", 2024-09-18T08:05:12.344 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0025941430000000001 2024-09-18T08:05:12.345 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:05:12.345 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:05:12.345 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:05:12.345 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:05:00.329702+0000", 2024-09-18T08:05:12.345 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.011257728999999999 2024-09-18T08:05:12.345 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:05:12.345 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:05:12.346 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "commit_sent", 2024-09-18T08:05:12.346 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:05:00.329741+0000", 2024-09-18T08:05:12.346 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 3.8526999999999999e-05 2024-09-18T08:05:12.346 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:05:12.346 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:05:12.347 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-09-18T08:05:12.347 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:05:00.329783+0000", 2024-09-18T08:05:12.347 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4.2539e-05 2024-09-18T08:05:12.347 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-18T08:05:12.347 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-09-18T08:05:12.347 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-18T08:05:12.348 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:05:12.348 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:05:12.348 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.14805.0:84494 7.0 7:0e862e17:::benchmark_data_smithi007_73155_object84493:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e107)", 2024-09-18T08:05:12.348 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-09-18T08:05:00.544247+0000", 2024-09-18T08:05:12.348 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 11.716411233000001, 2024-09-18T08:05:12.348 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.043677137999999997, 2024-09-18T08:05:12.349 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-09-18T08:05:12.349 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-09-18T08:05:12.349 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T08:05:12.349 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2024-09-18T08:05:12.349 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.14805", 2024-09-18T08:05:12.349 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "172.21.15.7:0/640320436", 2024-09-18T08:05:12.349 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 84494 2024-09-18T08:05:12.349 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:05:12.349 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-09-18T08:05:12.349 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:05:12.349 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-09-18T08:05:12.349 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:05:00.544247+0000", 2024-09-18T08:05:12.350 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-18T08:05:12.350 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:05:12.350 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:05:12.350 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-09-18T08:05:12.350 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:05:00.544247+0000", 2024-09-18T08:05:12.350 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-18T08:05:12.350 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:05:12.350 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:05:12.351 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-09-18T08:05:12.351 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:05:00.544249+0000", 2024-09-18T08:05:12.351 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.7010000000000001e-06 2024-09-18T08:05:12.351 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:05:12.351 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:05:12.351 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-09-18T08:05:12.351 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:05:00.544279+0000", 2024-09-18T08:05:12.351 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 2.9510999999999999e-05 2024-09-18T08:05:12.351 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:05:12.351 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:05:12.351 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-09-18T08:05:12.352 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:05:00.544291+0000", 2024-09-18T08:05:12.352 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.186e-05 2024-09-18T08:05:12.352 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:05:12.352 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:05:12.352 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-09-18T08:05:12.352 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:05:00.544298+0000", 2024-09-18T08:05:12.352 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 7.3889999999999999e-06 2024-09-18T08:05:12.352 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:05:12.352 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:05:12.352 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-18T08:05:12.352 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:05:00.549814+0000", 2024-09-18T08:05:12.353 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0055157829999999998 2024-09-18T08:05:12.353 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:05:12.353 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:05:12.353 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-09-18T08:05:12.353 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:05:00.549959+0000", 2024-09-18T08:05:12.353 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00014519399999999999 2024-09-18T08:05:12.353 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:05:12.353 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:05:12.353 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "waiting for subops from 1,3", 2024-09-18T08:05:12.353 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:05:00.550115+0000", 2024-09-18T08:05:12.353 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000156389 2024-09-18T08:05:12.353 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:05:12.353 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:05:12.353 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "op_commit", 2024-09-18T08:05:12.354 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:05:00.553252+0000", 2024-09-18T08:05:12.354 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0031363480000000002 2024-09-18T08:05:12.354 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:05:12.354 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:05:12.354 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:05:12.354 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:05:00.553301+0000", 2024-09-18T08:05:12.354 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4.9706000000000001e-05 2024-09-18T08:05:12.354 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:05:12.354 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:05:12.354 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:05:12.354 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:05:00.587779+0000", 2024-09-18T08:05:12.354 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.034477269999999997 2024-09-18T08:05:12.355 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:05:12.355 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:05:12.355 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "commit_sent", 2024-09-18T08:05:12.355 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:05:00.587855+0000", 2024-09-18T08:05:12.356 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 7.5970999999999995e-05 2024-09-18T08:05:12.356 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:05:12.356 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:05:12.356 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-09-18T08:05:12.357 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:05:00.587925+0000", 2024-09-18T08:05:12.357 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 7.0016000000000002e-05 2024-09-18T08:05:12.357 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-18T08:05:12.357 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-09-18T08:05:12.357 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-18T08:05:12.357 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:05:12.357 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:05:12.357 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.14805.0:84641 7.0 7:05422c2c:::benchmark_data_smithi007_73155_object84640:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e107)", 2024-09-18T08:05:12.357 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-09-18T08:05:00.695806+0000", 2024-09-18T08:05:12.357 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 11.564852568999999, 2024-09-18T08:05:12.357 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.073769931999999996, 2024-09-18T08:05:12.357 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-09-18T08:05:12.357 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-09-18T08:05:12.357 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T08:05:12.358 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2024-09-18T08:05:12.358 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.14805", 2024-09-18T08:05:12.358 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "172.21.15.7:0/640320436", 2024-09-18T08:05:12.358 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 84641 2024-09-18T08:05:12.358 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:05:12.358 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-09-18T08:05:12.358 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:05:12.358 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-09-18T08:05:12.358 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:05:00.695806+0000", 2024-09-18T08:05:12.358 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-18T08:05:12.359 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:05:12.359 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:05:12.359 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-09-18T08:05:12.359 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:05:00.695806+0000", 2024-09-18T08:05:12.359 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-18T08:05:12.359 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:05:12.359 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:05:12.359 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-09-18T08:05:12.359 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:05:00.695808+0000", 2024-09-18T08:05:12.359 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 2.2350000000000002e-06 2024-09-18T08:05:12.359 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:05:12.359 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:05:12.360 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-09-18T08:05:12.360 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:05:00.695836+0000", 2024-09-18T08:05:12.360 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 2.8124999999999999e-05 2024-09-18T08:05:12.360 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:05:12.360 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:05:12.360 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-09-18T08:05:12.360 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:05:00.695849+0000", 2024-09-18T08:05:12.360 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.2694000000000001e-05 2024-09-18T08:05:12.361 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:05:12.361 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:05:12.361 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-09-18T08:05:12.361 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:05:00.695858+0000", 2024-09-18T08:05:12.361 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 8.3610000000000001e-06 2024-09-18T08:05:12.361 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:05:12.361 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:05:12.361 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-18T08:05:12.361 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:05:00.696681+0000", 2024-09-18T08:05:12.361 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00082367300000000003 2024-09-18T08:05:12.361 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:05:12.361 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:05:12.361 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-09-18T08:05:12.361 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:05:00.696777+0000", 2024-09-18T08:05:12.362 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 9.5497999999999999e-05 2024-09-18T08:05:12.362 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:05:12.362 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:05:12.362 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "waiting for subops from 1,3", 2024-09-18T08:05:12.362 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:05:00.696877+0000", 2024-09-18T08:05:12.362 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000100627 2024-09-18T08:05:12.362 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:05:12.362 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:05:12.362 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "op_commit", 2024-09-18T08:05:12.362 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:05:00.700266+0000", 2024-09-18T08:05:12.362 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0033887570000000001 2024-09-18T08:05:12.362 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:05:12.363 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:05:12.363 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:05:12.363 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:05:00.701934+0000", 2024-09-18T08:05:12.363 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.001667604 2024-09-18T08:05:12.363 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:05:12.363 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:05:12.363 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:05:12.363 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:05:00.769482+0000", 2024-09-18T08:05:12.363 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.067548514000000004 2024-09-18T08:05:12.364 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:05:12.364 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:05:12.364 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "commit_sent", 2024-09-18T08:05:12.364 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:05:00.769528+0000", 2024-09-18T08:05:12.364 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4.5624999999999998e-05 2024-09-18T08:05:12.364 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:05:12.365 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:05:12.365 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-09-18T08:05:12.365 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:05:00.769576+0000", 2024-09-18T08:05:12.365 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4.8219e-05 2024-09-18T08:05:12.365 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-18T08:05:12.365 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-09-18T08:05:12.366 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-18T08:05:12.366 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:05:12.366 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:05:12.366 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.14805.0:84877 7.0 7:0a60ed0f:::benchmark_data_smithi007_73155_object84876:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e107)", 2024-09-18T08:05:12.366 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-09-18T08:05:00.954473+0000", 2024-09-18T08:05:12.366 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 11.306185387999999, 2024-09-18T08:05:12.366 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.033399138000000002, 2024-09-18T08:05:12.367 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-09-18T08:05:12.367 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-09-18T08:05:12.367 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T08:05:12.367 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2024-09-18T08:05:12.367 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.14805", 2024-09-18T08:05:12.367 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "172.21.15.7:0/640320436", 2024-09-18T08:05:12.368 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 84877 2024-09-18T08:05:12.368 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:05:12.368 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-09-18T08:05:12.368 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:05:12.368 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-09-18T08:05:12.368 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:05:00.954473+0000", 2024-09-18T08:05:12.368 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-18T08:05:12.368 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:05:12.368 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:05:12.369 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-09-18T08:05:12.369 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:05:00.954473+0000", 2024-09-18T08:05:12.369 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-18T08:05:12.369 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:05:12.369 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:05:12.369 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-09-18T08:05:12.369 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:05:00.954476+0000", 2024-09-18T08:05:12.369 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 2.3020000000000002e-06 2024-09-18T08:05:12.369 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:05:12.370 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:05:12.370 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-09-18T08:05:12.370 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:05:00.954532+0000", 2024-09-18T08:05:12.370 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 5.6648999999999999e-05 2024-09-18T08:05:12.370 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:05:12.370 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:05:12.370 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-09-18T08:05:12.370 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:05:00.954550+0000", 2024-09-18T08:05:12.370 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.7683e-05 2024-09-18T08:05:12.370 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:05:12.370 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:05:12.371 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-09-18T08:05:12.371 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:05:00.954560+0000", 2024-09-18T08:05:12.371 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.0379e-05 2024-09-18T08:05:12.371 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:05:12.371 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:05:12.371 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-18T08:05:12.371 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:05:00.962860+0000", 2024-09-18T08:05:12.371 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0082993259999999992 2024-09-18T08:05:12.371 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:05:12.371 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:05:12.371 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-09-18T08:05:12.371 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:05:00.963014+0000", 2024-09-18T08:05:12.372 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00015439800000000001 2024-09-18T08:05:12.372 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:05:12.372 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:05:12.372 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "waiting for subops from 1,3", 2024-09-18T08:05:12.372 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:05:00.963158+0000", 2024-09-18T08:05:12.372 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00014370799999999999 2024-09-18T08:05:12.372 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:05:12.372 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:05:12.372 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "op_commit", 2024-09-18T08:05:12.372 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:05:00.966550+0000", 2024-09-18T08:05:12.372 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0033923650000000001 2024-09-18T08:05:12.372 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:05:12.373 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:05:12.373 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:05:12.373 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:05:00.972276+0000", 2024-09-18T08:05:12.373 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0057263599999999998 2024-09-18T08:05:12.373 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:05:12.373 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:05:12.373 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:05:12.373 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:05:00.987702+0000", 2024-09-18T08:05:12.373 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.015425345 2024-09-18T08:05:12.373 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:05:12.373 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:05:12.373 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "commit_sent", 2024-09-18T08:05:12.374 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:05:00.987809+0000", 2024-09-18T08:05:12.374 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00010739999999999999 2024-09-18T08:05:12.374 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:05:12.374 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:05:12.374 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-09-18T08:05:12.374 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:05:00.987872+0000", 2024-09-18T08:05:12.374 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 6.3223000000000001e-05 2024-09-18T08:05:12.374 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-18T08:05:12.374 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-09-18T08:05:12.374 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-18T08:05:12.374 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:05:12.374 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:05:12.374 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.14805.0:93942 7.0 7:0792ef39:::benchmark_data_smithi007_73155_object93941:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e108)", 2024-09-18T08:05:12.375 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-09-18T08:05:09.718766+0000", 2024-09-18T08:05:12.375 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 2.5418923609999999, 2024-09-18T08:05:12.375 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.053719203, 2024-09-18T08:05:12.375 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-09-18T08:05:12.375 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-09-18T08:05:12.375 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T08:05:12.375 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2024-09-18T08:05:12.375 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.14805", 2024-09-18T08:05:12.375 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "172.21.15.7:0/640320436", 2024-09-18T08:05:12.375 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 93942 2024-09-18T08:05:12.375 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:05:12.375 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-09-18T08:05:12.376 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:05:12.376 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-09-18T08:05:12.376 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:05:09.718766+0000", 2024-09-18T08:05:12.376 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-18T08:05:12.376 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:05:12.376 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:05:12.376 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-09-18T08:05:12.376 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:05:09.718766+0000", 2024-09-18T08:05:12.376 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-18T08:05:12.376 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:05:12.376 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:05:12.376 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-09-18T08:05:12.377 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:05:09.718768+0000", 2024-09-18T08:05:12.377 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.869e-06 2024-09-18T08:05:12.377 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:05:12.377 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:05:12.377 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-09-18T08:05:12.377 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:05:09.718802+0000", 2024-09-18T08:05:12.377 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 3.3791000000000002e-05 2024-09-18T08:05:12.377 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:05:12.377 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:05:12.377 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-09-18T08:05:12.377 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:05:09.718821+0000", 2024-09-18T08:05:12.377 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.8697000000000001e-05 2024-09-18T08:05:12.377 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:05:12.378 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:05:12.378 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-09-18T08:05:12.378 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:05:09.718831+0000", 2024-09-18T08:05:12.378 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 9.9310000000000003e-06 2024-09-18T08:05:12.378 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:05:12.378 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:05:12.378 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-18T08:05:12.378 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:05:09.718891+0000", 2024-09-18T08:05:12.378 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 6.0194e-05 2024-09-18T08:05:12.378 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:05:12.379 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:05:12.379 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-09-18T08:05:12.379 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:05:09.719034+0000", 2024-09-18T08:05:12.379 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00014299399999999999 2024-09-18T08:05:12.379 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:05:12.379 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:05:12.379 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "waiting for subops from 1,3", 2024-09-18T08:05:12.379 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:05:09.719175+0000", 2024-09-18T08:05:12.379 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00014167499999999999 2024-09-18T08:05:12.379 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:05:12.380 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:05:12.380 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "op_commit", 2024-09-18T08:05:12.380 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:05:09.722195+0000", 2024-09-18T08:05:12.380 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0030196070000000001 2024-09-18T08:05:12.380 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:05:12.380 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:05:12.380 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:05:12.380 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:05:09.722340+0000", 2024-09-18T08:05:12.380 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00014459100000000001 2024-09-18T08:05:12.380 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:05:12.381 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:05:12.381 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:05:12.381 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:05:09.772362+0000", 2024-09-18T08:05:12.381 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.050022412000000002 2024-09-18T08:05:12.381 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:05:12.381 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:05:12.381 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "commit_sent", 2024-09-18T08:05:12.381 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:05:09.772425+0000", 2024-09-18T08:05:12.381 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 6.2594000000000004e-05 2024-09-18T08:05:12.381 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:05:12.381 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:05:12.382 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-09-18T08:05:12.382 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:05:09.772486+0000", 2024-09-18T08:05:12.382 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 6.0847999999999997e-05 2024-09-18T08:05:12.382 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-18T08:05:12.382 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-09-18T08:05:12.382 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-18T08:05:12.382 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:05:12.382 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:05:12.382 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.14805.0:93945 7.0 7:02a19226:::benchmark_data_smithi007_73155_object93944:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e108)", 2024-09-18T08:05:12.382 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-09-18T08:05:09.723432+0000", 2024-09-18T08:05:12.382 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 2.5372268249999999, 2024-09-18T08:05:12.382 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.052429371000000002, 2024-09-18T08:05:12.382 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-09-18T08:05:12.383 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-09-18T08:05:12.383 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T08:05:12.383 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2024-09-18T08:05:12.383 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.14805", 2024-09-18T08:05:12.383 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "172.21.15.7:0/640320436", 2024-09-18T08:05:12.383 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 93945 2024-09-18T08:05:12.383 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:05:12.383 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-09-18T08:05:12.383 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:05:12.383 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-09-18T08:05:12.383 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:05:09.723432+0000", 2024-09-18T08:05:12.384 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-18T08:05:12.384 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:05:12.384 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:05:12.384 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-09-18T08:05:12.384 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:05:09.723432+0000", 2024-09-18T08:05:12.384 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-18T08:05:12.384 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:05:12.384 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:05:12.385 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-09-18T08:05:12.385 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:05:09.723433+0000", 2024-09-18T08:05:12.385 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.3909999999999999e-06 2024-09-18T08:05:12.385 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:05:12.385 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:05:12.385 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-09-18T08:05:12.385 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:05:09.723484+0000", 2024-09-18T08:05:12.385 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 5.0256999999999997e-05 2024-09-18T08:05:12.386 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:05:12.386 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:05:12.386 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-09-18T08:05:12.386 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:05:09.723496+0000", 2024-09-18T08:05:12.386 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.2329e-05 2024-09-18T08:05:12.386 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:05:12.386 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:05:12.386 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-09-18T08:05:12.387 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:05:09.723502+0000", 2024-09-18T08:05:12.387 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 6.0809999999999999e-06 2024-09-18T08:05:12.387 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:05:12.387 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:05:12.387 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-18T08:05:12.387 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:05:09.723549+0000", 2024-09-18T08:05:12.387 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4.7475999999999997e-05 2024-09-18T08:05:12.387 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:05:12.388 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:05:12.388 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-09-18T08:05:12.388 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:05:09.723707+0000", 2024-09-18T08:05:12.388 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00015794 2024-09-18T08:05:12.388 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:05:12.388 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:05:12.388 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "waiting for subops from 1,3", 2024-09-18T08:05:12.388 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:05:09.723889+0000", 2024-09-18T08:05:12.389 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000181521 2024-09-18T08:05:12.389 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:05:12.389 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:05:12.389 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "op_commit", 2024-09-18T08:05:12.389 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:05:09.726881+0000", 2024-09-18T08:05:12.389 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0029921209999999999 2024-09-18T08:05:12.389 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:05:12.389 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:05:12.390 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:05:12.390 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:05:09.727043+0000", 2024-09-18T08:05:12.390 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000161776 2024-09-18T08:05:12.390 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:05:12.390 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:05:12.390 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:05:12.390 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:05:09.775777+0000", 2024-09-18T08:05:12.390 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.048734054999999998 2024-09-18T08:05:12.391 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:05:12.391 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:05:12.391 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "commit_sent", 2024-09-18T08:05:12.391 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:05:09.775817+0000", 2024-09-18T08:05:12.391 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 3.9949e-05 2024-09-18T08:05:12.391 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:05:12.391 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:05:12.391 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-09-18T08:05:12.392 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:05:09.775861+0000", 2024-09-18T08:05:12.392 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4.4474999999999997e-05 2024-09-18T08:05:12.392 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-18T08:05:12.392 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-09-18T08:05:12.392 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-18T08:05:12.392 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:05:12.392 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:05:12.393 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.14805.0:94177 7.0 7:020e8657:::benchmark_data_smithi007_73155_object94176:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e108)", 2024-09-18T08:05:12.393 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-09-18T08:05:09.957962+0000", 2024-09-18T08:05:12.393 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 2.3026965119999998, 2024-09-18T08:05:12.393 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.072811827999999995, 2024-09-18T08:05:12.393 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-09-18T08:05:12.393 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-09-18T08:05:12.393 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T08:05:12.393 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2024-09-18T08:05:12.393 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.14805", 2024-09-18T08:05:12.393 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "172.21.15.7:0/640320436", 2024-09-18T08:05:12.394 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 94177 2024-09-18T08:05:12.394 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:05:12.394 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-09-18T08:05:12.394 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:05:12.394 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-09-18T08:05:12.394 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:05:09.957962+0000", 2024-09-18T08:05:12.394 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-18T08:05:12.394 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:05:12.394 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:05:12.395 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-09-18T08:05:12.395 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:05:09.957962+0000", 2024-09-18T08:05:12.395 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-18T08:05:12.395 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:05:12.395 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:05:12.395 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-09-18T08:05:12.395 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:05:09.957964+0000", 2024-09-18T08:05:12.395 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 2.0930000000000001e-06 2024-09-18T08:05:12.395 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:05:12.395 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:05:12.395 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-09-18T08:05:12.395 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:05:09.957998+0000", 2024-09-18T08:05:12.396 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 3.3964000000000002e-05 2024-09-18T08:05:12.396 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:05:12.396 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:05:12.396 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-09-18T08:05:12.396 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:05:09.958013+0000", 2024-09-18T08:05:12.396 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.482e-05 2024-09-18T08:05:12.396 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:05:12.396 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:05:12.396 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-09-18T08:05:12.396 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:05:09.958021+0000", 2024-09-18T08:05:12.396 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 8.1689999999999992e-06 2024-09-18T08:05:12.396 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:05:12.396 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:05:12.397 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-18T08:05:12.397 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:05:09.958093+0000", 2024-09-18T08:05:12.397 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 7.2188999999999996e-05 2024-09-18T08:05:12.397 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:05:12.397 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:05:12.397 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-09-18T08:05:12.397 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:05:09.958231+0000", 2024-09-18T08:05:12.397 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00013764499999999999 2024-09-18T08:05:12.397 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:05:12.397 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:05:12.397 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "waiting for subops from 1,3", 2024-09-18T08:05:12.397 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:05:09.958382+0000", 2024-09-18T08:05:12.398 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00015097400000000001 2024-09-18T08:05:12.398 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:05:12.398 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:05:12.398 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "op_commit", 2024-09-18T08:05:12.398 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:05:09.961387+0000", 2024-09-18T08:05:12.398 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.003005158 2024-09-18T08:05:12.398 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:05:12.398 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:05:12.398 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:05:12.398 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:05:09.967952+0000", 2024-09-18T08:05:12.398 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0065646999999999997 2024-09-18T08:05:12.398 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:05:12.398 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:05:12.399 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:05:12.399 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:05:10.030695+0000", 2024-09-18T08:05:12.399 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.062743423000000006 2024-09-18T08:05:12.399 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:05:12.399 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:05:12.399 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "commit_sent", 2024-09-18T08:05:12.399 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:05:10.030739+0000", 2024-09-18T08:05:12.399 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4.3467999999999998e-05 2024-09-18T08:05:12.399 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:05:12.399 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:05:12.399 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-09-18T08:05:12.399 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:05:10.030774+0000", 2024-09-18T08:05:12.399 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 3.5225000000000002e-05 2024-09-18T08:05:12.400 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-18T08:05:12.400 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-09-18T08:05:12.400 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-18T08:05:12.400 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:05:12.400 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:05:12.400 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.14805.0:95774 7.0 7:03ba3cb9:::benchmark_data_smithi007_73155_object95773:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e109)", 2024-09-18T08:05:12.400 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-09-18T08:05:11.851771+0000", 2024-09-18T08:05:12.400 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 0.40888739200000002, 2024-09-18T08:05:12.400 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.038186454000000002, 2024-09-18T08:05:12.400 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-09-18T08:05:12.400 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-09-18T08:05:12.400 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T08:05:12.401 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2024-09-18T08:05:12.401 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.14805", 2024-09-18T08:05:12.401 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "172.21.15.7:0/640320436", 2024-09-18T08:05:12.401 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 95774 2024-09-18T08:05:12.401 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:05:12.401 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-09-18T08:05:12.401 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:05:12.401 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-09-18T08:05:12.401 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:05:11.851771+0000", 2024-09-18T08:05:12.401 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-18T08:05:12.401 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:05:12.401 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:05:12.401 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-09-18T08:05:12.402 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:05:11.851771+0000", 2024-09-18T08:05:12.402 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-18T08:05:12.402 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:05:12.402 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:05:12.402 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-09-18T08:05:12.402 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:05:11.851773+0000", 2024-09-18T08:05:12.402 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.9199999999999998e-06 2024-09-18T08:05:12.402 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:05:12.402 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:05:12.402 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-09-18T08:05:12.402 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:05:11.851810+0000", 2024-09-18T08:05:12.403 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 3.6990000000000003e-05 2024-09-18T08:05:12.403 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:05:12.403 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:05:12.403 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-09-18T08:05:12.403 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:05:11.851827+0000", 2024-09-18T08:05:12.403 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.7224999999999999e-05 2024-09-18T08:05:12.403 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:05:12.403 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:05:12.403 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-09-18T08:05:12.403 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:05:11.851836+0000", 2024-09-18T08:05:12.403 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 8.6149999999999997e-06 2024-09-18T08:05:12.403 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:05:12.403 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:05:12.404 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-18T08:05:12.404 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:05:11.851908+0000", 2024-09-18T08:05:12.404 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 7.1464999999999994e-05 2024-09-18T08:05:12.404 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:05:12.404 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:05:12.404 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-09-18T08:05:12.404 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:05:11.852064+0000", 2024-09-18T08:05:12.404 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00015692700000000001 2024-09-18T08:05:12.404 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:05:12.404 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:05:12.404 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "waiting for subops from 1,3", 2024-09-18T08:05:12.404 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:05:11.852233+0000", 2024-09-18T08:05:12.405 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00016856799999999999 2024-09-18T08:05:12.405 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:05:12.405 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:05:12.405 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "op_commit", 2024-09-18T08:05:12.405 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:05:11.854392+0000", 2024-09-18T08:05:12.405 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0021593990000000002 2024-09-18T08:05:12.405 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:05:12.405 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:05:12.405 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:05:12.405 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:05:11.864334+0000", 2024-09-18T08:05:12.405 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0099410990000000001 2024-09-18T08:05:12.405 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:05:12.405 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:05:12.406 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:05:12.406 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:05:11.889883+0000", 2024-09-18T08:05:12.406 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.025549736 2024-09-18T08:05:12.406 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:05:12.406 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:05:12.406 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "commit_sent", 2024-09-18T08:05:12.406 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:05:11.889930+0000", 2024-09-18T08:05:12.406 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4.6912000000000001e-05 2024-09-18T08:05:12.406 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:05:12.406 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:05:12.406 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-09-18T08:05:12.406 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:05:11.889958+0000", 2024-09-18T08:05:12.406 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 2.7597999999999999e-05 2024-09-18T08:05:12.407 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-18T08:05:12.407 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-09-18T08:05:12.407 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-18T08:05:12.407 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:05:12.407 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:05:12.407 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.14805.0:95798 7.0 7:09d4ded4:::benchmark_data_smithi007_73155_object95797:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e109)", 2024-09-18T08:05:12.407 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-09-18T08:05:11.888912+0000", 2024-09-18T08:05:12.407 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 0.37174702599999998, 2024-09-18T08:05:12.407 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.037853043000000003, 2024-09-18T08:05:12.407 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-09-18T08:05:12.407 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-09-18T08:05:12.408 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T08:05:12.408 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2024-09-18T08:05:12.408 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.14805", 2024-09-18T08:05:12.408 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "172.21.15.7:0/640320436", 2024-09-18T08:05:12.408 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 95798 2024-09-18T08:05:12.408 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:05:12.408 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-09-18T08:05:12.408 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:05:12.408 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-09-18T08:05:12.408 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:05:11.888912+0000", 2024-09-18T08:05:12.408 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-18T08:05:12.408 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:05:12.408 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:05:12.409 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-09-18T08:05:12.409 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:05:11.888912+0000", 2024-09-18T08:05:12.409 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-18T08:05:12.409 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:05:12.409 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:05:12.409 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-09-18T08:05:12.409 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:05:11.888912+0000", 2024-09-18T08:05:12.409 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 7.8999999999999995e-07 2024-09-18T08:05:12.409 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:05:12.409 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:05:12.409 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-09-18T08:05:12.409 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:05:11.888975+0000", 2024-09-18T08:05:12.409 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 6.2118000000000004e-05 2024-09-18T08:05:12.409 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:05:12.410 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:05:12.410 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-09-18T08:05:12.410 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:05:11.888985+0000", 2024-09-18T08:05:12.410 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.0122999999999999e-05 2024-09-18T08:05:12.410 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:05:12.410 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:05:12.410 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-09-18T08:05:12.410 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:05:11.888990+0000", 2024-09-18T08:05:12.410 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 5.2390000000000002e-06 2024-09-18T08:05:12.410 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:05:12.410 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:05:12.410 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-18T08:05:12.410 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:05:11.889991+0000", 2024-09-18T08:05:12.411 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0010013839999999999 2024-09-18T08:05:12.411 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:05:12.411 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:05:12.411 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-09-18T08:05:12.411 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:05:11.890088+0000", 2024-09-18T08:05:12.411 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 9.6867999999999995e-05 2024-09-18T08:05:12.411 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:05:12.411 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:05:12.411 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "waiting for subops from 1,3", 2024-09-18T08:05:12.411 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:05:11.890186+0000", 2024-09-18T08:05:12.411 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 9.8165999999999995e-05 2024-09-18T08:05:12.411 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:05:12.412 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:05:12.412 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "op_commit", 2024-09-18T08:05:12.412 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:05:11.895127+0000", 2024-09-18T08:05:12.412 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.004940977 2024-09-18T08:05:12.412 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:05:12.412 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:05:12.412 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:05:12.413 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:05:11.909857+0000", 2024-09-18T08:05:12.413 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.014729559 2024-09-18T08:05:12.413 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:05:12.413 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:05:12.413 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:05:12.413 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:05:11.926662+0000", 2024-09-18T08:05:12.413 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.016805364 2024-09-18T08:05:12.413 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:05:12.413 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:05:12.413 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "commit_sent", 2024-09-18T08:05:12.413 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:05:11.926718+0000", 2024-09-18T08:05:12.414 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 5.5693999999999999e-05 2024-09-18T08:05:12.414 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:05:12.414 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:05:12.414 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-09-18T08:05:12.414 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:05:11.926765+0000", 2024-09-18T08:05:12.414 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4.6761000000000002e-05 2024-09-18T08:05:12.414 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-18T08:05:12.414 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-09-18T08:05:12.414 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-18T08:05:12.414 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:05:12.414 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:05:12.414 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.14805.0:95800 7.0 7:0e8d0ff7:::benchmark_data_smithi007_73155_object95799:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e109)", 2024-09-18T08:05:12.415 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-09-18T08:05:11.890189+0000", 2024-09-18T08:05:12.415 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 0.37046931399999999, 2024-09-18T08:05:12.415 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.036692137, 2024-09-18T08:05:12.415 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-09-18T08:05:12.415 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-09-18T08:05:12.415 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T08:05:12.415 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2024-09-18T08:05:12.415 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.14805", 2024-09-18T08:05:12.415 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "172.21.15.7:0/640320436", 2024-09-18T08:05:12.415 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 95800 2024-09-18T08:05:12.415 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:05:12.415 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-09-18T08:05:12.415 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:05:12.416 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-09-18T08:05:12.416 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:05:11.890189+0000", 2024-09-18T08:05:12.416 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-18T08:05:12.416 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:05:12.416 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:05:12.416 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-09-18T08:05:12.416 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:05:11.890189+0000", 2024-09-18T08:05:12.416 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-18T08:05:12.416 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:05:12.416 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:05:12.416 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-09-18T08:05:12.416 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:05:11.890190+0000", 2024-09-18T08:05:12.416 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 9.4900000000000004e-07 2024-09-18T08:05:12.416 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:05:12.417 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:05:12.417 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-09-18T08:05:12.417 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:05:11.890272+0000", 2024-09-18T08:05:12.417 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 8.1477999999999995e-05 2024-09-18T08:05:12.417 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:05:12.417 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:05:12.417 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-09-18T08:05:12.417 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:05:11.890283+0000", 2024-09-18T08:05:12.417 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.0699999999999999e-05 2024-09-18T08:05:12.417 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:05:12.417 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:05:12.417 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-09-18T08:05:12.417 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:05:11.890290+0000", 2024-09-18T08:05:12.418 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 7.418e-06 2024-09-18T08:05:12.418 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:05:12.418 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:05:12.418 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-18T08:05:12.418 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:05:11.892514+0000", 2024-09-18T08:05:12.418 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0022235919999999999 2024-09-18T08:05:12.418 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:05:12.418 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:05:12.418 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-09-18T08:05:12.418 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:05:11.892644+0000", 2024-09-18T08:05:12.418 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00013051 2024-09-18T08:05:12.418 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:05:12.418 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:05:12.419 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "waiting for subops from 1,3", 2024-09-18T08:05:12.419 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:05:11.892803+0000", 2024-09-18T08:05:12.419 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000158803 2024-09-18T08:05:12.419 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:05:12.419 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:05:12.419 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "op_commit", 2024-09-18T08:05:12.419 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:05:11.897853+0000", 2024-09-18T08:05:12.419 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0050504820000000002 2024-09-18T08:05:12.419 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:05:12.419 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:05:12.419 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:05:12.419 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:05:11.910404+0000", 2024-09-18T08:05:12.419 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.012550875 2024-09-18T08:05:12.420 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:05:12.420 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:05:12.420 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:05:12.420 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:05:11.926808+0000", 2024-09-18T08:05:12.420 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.016404102 2024-09-18T08:05:12.420 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:05:12.420 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:05:12.420 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "commit_sent", 2024-09-18T08:05:12.420 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:05:11.926856+0000", 2024-09-18T08:05:12.420 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4.7419000000000001e-05 2024-09-18T08:05:12.420 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:05:12.420 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:05:12.421 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-09-18T08:05:12.421 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:05:11.926882+0000", 2024-09-18T08:05:12.421 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 2.5809e-05 2024-09-18T08:05:12.421 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-18T08:05:12.421 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-09-18T08:05:12.421 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-18T08:05:12.421 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:05:12.421 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:05:12.421 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.14805.0:95802 7.0 7:02f42dc7:::benchmark_data_smithi007_73155_object95801:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e109)", 2024-09-18T08:05:12.421 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-09-18T08:05:11.893053+0000", 2024-09-18T08:05:12.421 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 0.367606127, 2024-09-18T08:05:12.421 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.033943078000000002, 2024-09-18T08:05:12.421 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-09-18T08:05:12.422 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-09-18T08:05:12.422 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T08:05:12.422 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2024-09-18T08:05:12.422 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.14805", 2024-09-18T08:05:12.422 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "172.21.15.7:0/640320436", 2024-09-18T08:05:12.422 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 95802 2024-09-18T08:05:12.422 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:05:12.422 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-09-18T08:05:12.422 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:05:12.422 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-09-18T08:05:12.422 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:05:11.893053+0000", 2024-09-18T08:05:12.422 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-18T08:05:12.422 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:05:12.422 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:05:12.422 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-09-18T08:05:12.422 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:05:11.893053+0000", 2024-09-18T08:05:12.423 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-18T08:05:12.423 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:05:12.423 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:05:12.423 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-09-18T08:05:12.423 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:05:11.893054+0000", 2024-09-18T08:05:12.423 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.46e-06 2024-09-18T08:05:12.423 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:05:12.423 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:05:12.423 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-09-18T08:05:12.423 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:05:11.893101+0000", 2024-09-18T08:05:12.423 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4.7184000000000001e-05 2024-09-18T08:05:12.423 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:05:12.423 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:05:12.423 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-09-18T08:05:12.423 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:05:11.893115+0000", 2024-09-18T08:05:12.424 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.3365e-05 2024-09-18T08:05:12.424 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:05:12.424 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:05:12.424 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-09-18T08:05:12.424 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:05:11.893125+0000", 2024-09-18T08:05:12.424 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.0859999999999999e-05 2024-09-18T08:05:12.424 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:05:12.424 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:05:12.424 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-18T08:05:12.424 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:05:11.895160+0000", 2024-09-18T08:05:12.424 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0020344819999999998 2024-09-18T08:05:12.425 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:05:12.425 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:05:12.425 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-09-18T08:05:12.425 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:05:11.895301+0000", 2024-09-18T08:05:12.425 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00014154700000000001 2024-09-18T08:05:12.425 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:05:12.425 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:05:12.425 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "waiting for subops from 1,3", 2024-09-18T08:05:12.425 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:05:11.895477+0000", 2024-09-18T08:05:12.425 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000175599 2024-09-18T08:05:12.425 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:05:12.425 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:05:12.425 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "op_commit", 2024-09-18T08:05:12.426 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:05:11.898025+0000", 2024-09-18T08:05:12.426 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0025479970000000002 2024-09-18T08:05:12.426 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:05:12.426 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:05:12.426 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:05:12.426 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:05:11.916394+0000", 2024-09-18T08:05:12.426 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.018369441 2024-09-18T08:05:12.426 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:05:12.426 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:05:12.426 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:05:12.426 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:05:11.926922+0000", 2024-09-18T08:05:12.426 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.010527797 2024-09-18T08:05:12.426 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:05:12.426 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:05:12.426 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "commit_sent", 2024-09-18T08:05:12.427 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:05:11.926957+0000", 2024-09-18T08:05:12.427 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 3.4569000000000001e-05 2024-09-18T08:05:12.427 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:05:12.427 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:05:12.427 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-09-18T08:05:12.427 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:05:11.926996+0000", 2024-09-18T08:05:12.427 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 3.8776999999999998e-05 2024-09-18T08:05:12.427 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-18T08:05:12.427 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-09-18T08:05:12.427 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-18T08:05:12.427 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-18T08:05:12.427 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-09-18T08:05:12.427 INFO:teuthology.orchestra.run.smithi050.stdout:} 2024-09-18T08:05:12.428 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5 2024-09-18T08:05:12.453 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:05:12.454 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:05:12.454 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5' 2024-09-18T08:05:12.555 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5 2024-09-18T08:05:12.583 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:05:12.583 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:05:12.584 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5' 2024-09-18T08:05:12.587 INFO:tasks.radosbench.radosbench.0.smithi007.stdout: 67 16 96285 96269 89.7756 43.6875 0.0136092 0.011133 2024-09-18T08:05:12.685 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1 2024-09-18T08:05:12.745 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:05:12.746 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:05:12.746 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1' 2024-09-18T08:05:12.845 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:05:12 smithi050 ceph-mon[32371]: 7.6 scrub starts 2024-09-18T08:05:12.845 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:05:12 smithi050 ceph-mon[32371]: 5.11 scrub starts 2024-09-18T08:05:12.845 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:05:12 smithi050 ceph-mon[32371]: 5.11 scrub ok 2024-09-18T08:05:12.846 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:05:12 smithi050 ceph-mon[32371]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]': finished 2024-09-18T08:05:12.846 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:05:12 smithi050 ceph-mon[32371]: osdmap e109: 8 total, 8 up, 8 in 2024-09-18T08:05:12.846 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:05:12 smithi050 ceph-mon[32371]: 3.f scrub starts 2024-09-18T08:05:12.846 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:05:12 smithi050 ceph-mon[32371]: 3.f scrub ok 2024-09-18T08:05:12.846 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:05:12 smithi050 ceph-mon[32371]: from='mgr.24470 172.21.15.7:0/2594390261' entity='mgr.y' 2024-09-18T08:05:12.846 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:05:12 smithi050 ceph-mon[32371]: from='mgr.24470 172.21.15.7:0/2594390261' entity='mgr.y' 2024-09-18T08:05:12.846 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:05:12 smithi050 ceph-mon[32371]: from='client.? 172.21.15.7:0/2993149196' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-18T08:05:12.846 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:05:12 smithi050 ceph-mon[32371]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-18T08:05:12.848 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7 2024-09-18T08:05:12.880 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:05:12.880 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:05:12.881 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7' 2024-09-18T08:05:12.982 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:05:12 smithi007 ceph-mon[29459]: 7.6 scrub starts 2024-09-18T08:05:12.982 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:05:12 smithi007 ceph-mon[29459]: 5.11 scrub starts 2024-09-18T08:05:12.982 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:05:12 smithi007 ceph-mon[29459]: 5.11 scrub ok 2024-09-18T08:05:12.982 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:05:12 smithi007 ceph-mon[29459]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]': finished 2024-09-18T08:05:12.982 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:05:12 smithi007 ceph-mon[29459]: osdmap e109: 8 total, 8 up, 8 in 2024-09-18T08:05:12.982 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:05:12 smithi007 ceph-mon[29459]: 3.f scrub starts 2024-09-18T08:05:12.982 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:05:12 smithi007 ceph-mon[29459]: 3.f scrub ok 2024-09-18T08:05:12.982 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:05:12 smithi007 ceph-mon[29459]: from='mgr.24470 172.21.15.7:0/2594390261' entity='mgr.y' 2024-09-18T08:05:12.982 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:05:12 smithi007 ceph-mon[29459]: from='mgr.24470 172.21.15.7:0/2594390261' entity='mgr.y' 2024-09-18T08:05:12.983 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:05:12 smithi007 ceph-mon[29459]: from='client.? 172.21.15.7:0/2993149196' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-18T08:05:12.983 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:05:12 smithi007 ceph-mon[29459]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-18T08:05:12.983 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:05:12 smithi007 ceph-mon[39249]: 7.6 scrub starts 2024-09-18T08:05:12.983 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:05:12 smithi007 ceph-mon[39249]: 5.11 scrub starts 2024-09-18T08:05:12.983 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:05:12 smithi007 ceph-mon[39249]: 5.11 scrub ok 2024-09-18T08:05:12.983 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:05:12 smithi007 ceph-mon[39249]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]': finished 2024-09-18T08:05:12.983 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:05:12 smithi007 ceph-mon[39249]: osdmap e109: 8 total, 8 up, 8 in 2024-09-18T08:05:12.983 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:05:12 smithi007 ceph-mon[39249]: 3.f scrub starts 2024-09-18T08:05:12.983 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:05:12 smithi007 ceph-mon[39249]: 3.f scrub ok 2024-09-18T08:05:12.983 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:05:12 smithi007 ceph-mon[39249]: from='mgr.24470 172.21.15.7:0/2594390261' entity='mgr.y' 2024-09-18T08:05:12.983 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:05:12 smithi007 ceph-mon[39249]: from='mgr.24470 172.21.15.7:0/2594390261' entity='mgr.y' 2024-09-18T08:05:12.984 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:05:12 smithi007 ceph-mon[39249]: from='client.? 172.21.15.7:0/2993149196' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-18T08:05:12.984 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:05:12 smithi007 ceph-mon[39249]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-18T08:05:12.984 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6 2024-09-18T08:05:12.984 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid 779349ca-7593-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-18T08:05:13.013 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:05:13.014 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:05:13.014 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6' 2024-09-18T08:05:13.115 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1 2024-09-18T08:05:13.151 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid 779349ca-7593-11ef-bceb-c7b262605968 -- ceph osd reweight 1 1 2024-09-18T08:05:13.203 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:05:13.203 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:05:13.204 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1' 2024-09-18T08:05:13.304 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0 2024-09-18T08:05:13.420 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:05:13.421 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:05:13.422 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0' 2024-09-18T08:05:13.522 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5 2024-09-18T08:05:13.550 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:05:13.551 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:05:13.551 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5' 2024-09-18T08:05:13.567 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/779349ca-7593-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T08:05:13.589 INFO:tasks.radosbench.radosbench.0.smithi007.stdout: 68 16 96867 96851 88.989 36.375 0.0158672 0.0112299 2024-09-18T08:05:13.626 INFO:teuthology.orchestra.run.smithi007.stderr:noscrub is set 2024-09-18T08:05:13.652 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4 2024-09-18T08:05:13.679 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:05:13.680 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:05:13.681 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4' 2024-09-18T08:05:13.781 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6 2024-09-18T08:05:13.809 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:05:13.810 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:05:13.810 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6' 2024-09-18T08:05:13.911 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3 2024-09-18T08:05:13.955 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:05:13.956 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:05:13.956 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3' 2024-09-18T08:05:14.018 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:05:13 smithi050 ceph-mon[32371]: 3.a scrub starts 2024-09-18T08:05:14.018 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:05:13 smithi050 ceph-mon[32371]: 3.a scrub ok 2024-09-18T08:05:14.018 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:05:13 smithi050 ceph-mon[32371]: pgmap v97: 148 pgs: 148 active+clean; 5.7 GiB data, 18 GiB used, 698 GiB / 715 GiB avail; 1.1 KiB/s rd, 39 MiB/s wr, 624 op/s 2024-09-18T08:05:14.018 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:05:13 smithi050 ceph-mon[32371]: from='mgr.24470 172.21.15.7:0/2594390261' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-18T08:05:14.018 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:05:13 smithi050 ceph-mon[32371]: from='mgr.24470 172.21.15.7:0/2594390261' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-18T08:05:14.019 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:05:13 smithi050 ceph-mon[32371]: from='mgr.24470 172.21.15.7:0/2594390261' entity='mgr.y' 2024-09-18T08:05:14.019 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:05:13 smithi050 ceph-mon[32371]: purged_snaps scrub starts 2024-09-18T08:05:14.019 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:05:13 smithi050 ceph-mon[32371]: purged_snaps scrub ok 2024-09-18T08:05:14.019 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:05:13 smithi050 ceph-mon[32371]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-18T08:05:14.019 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:05:13 smithi050 ceph-mon[32371]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-18T08:05:14.019 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:05:13 smithi050 ceph-mon[32371]: osdmap e110: 8 total, 8 up, 8 in 2024-09-18T08:05:14.020 DEBUG:teuthology.orchestra.run.smithi050:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid 779349ca-7593-11ef-bceb-c7b262605968 -- ceph daemon osd.6 dump_ops_in_flight 2024-09-18T08:05:14.043 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:05:13 smithi007 ceph-mon[29459]: 3.a scrub starts 2024-09-18T08:05:14.043 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:05:13 smithi007 ceph-mon[29459]: 3.a scrub ok 2024-09-18T08:05:14.043 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:05:13 smithi007 ceph-mon[29459]: pgmap v97: 148 pgs: 148 active+clean; 5.7 GiB data, 18 GiB used, 698 GiB / 715 GiB avail; 1.1 KiB/s rd, 39 MiB/s wr, 624 op/s 2024-09-18T08:05:14.044 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:05:13 smithi007 ceph-mon[29459]: from='mgr.24470 172.21.15.7:0/2594390261' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-18T08:05:14.044 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:05:13 smithi007 ceph-mon[29459]: from='mgr.24470 172.21.15.7:0/2594390261' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-18T08:05:14.044 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:05:13 smithi007 ceph-mon[29459]: from='mgr.24470 172.21.15.7:0/2594390261' entity='mgr.y' 2024-09-18T08:05:14.044 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:05:13 smithi007 ceph-mon[29459]: purged_snaps scrub starts 2024-09-18T08:05:14.044 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:05:13 smithi007 ceph-mon[29459]: purged_snaps scrub ok 2024-09-18T08:05:14.044 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:05:13 smithi007 ceph-mon[29459]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-18T08:05:14.044 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:05:13 smithi007 ceph-mon[29459]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-18T08:05:14.044 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:05:13 smithi007 ceph-mon[29459]: osdmap e110: 8 total, 8 up, 8 in 2024-09-18T08:05:14.044 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:05:13 smithi007 ceph-mon[39249]: 3.a scrub starts 2024-09-18T08:05:14.044 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:05:13 smithi007 ceph-mon[39249]: 3.a scrub ok 2024-09-18T08:05:14.044 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:05:13 smithi007 ceph-mon[39249]: pgmap v97: 148 pgs: 148 active+clean; 5.7 GiB data, 18 GiB used, 698 GiB / 715 GiB avail; 1.1 KiB/s rd, 39 MiB/s wr, 624 op/s 2024-09-18T08:05:14.045 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:05:13 smithi007 ceph-mon[39249]: from='mgr.24470 172.21.15.7:0/2594390261' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-18T08:05:14.045 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:05:13 smithi007 ceph-mon[39249]: from='mgr.24470 172.21.15.7:0/2594390261' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-18T08:05:14.045 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:05:13 smithi007 ceph-mon[39249]: from='mgr.24470 172.21.15.7:0/2594390261' entity='mgr.y' 2024-09-18T08:05:14.045 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:05:13 smithi007 ceph-mon[39249]: purged_snaps scrub starts 2024-09-18T08:05:14.045 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:05:13 smithi007 ceph-mon[39249]: purged_snaps scrub ok 2024-09-18T08:05:14.045 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:05:13 smithi007 ceph-mon[39249]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-18T08:05:14.045 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:05:13 smithi007 ceph-mon[39249]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-18T08:05:14.045 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:05:13 smithi007 ceph-mon[39249]: osdmap e110: 8 total, 8 up, 8 in 2024-09-18T08:05:14.057 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2 2024-09-18T08:05:14.092 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:05:14.092 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:05:14.092 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2' 2024-09-18T08:05:14.193 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0 2024-09-18T08:05:14.236 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:05:14.237 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:05:14.237 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0' 2024-09-18T08:05:14.338 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3 2024-09-18T08:05:14.369 INFO:teuthology.orchestra.run.smithi050.stderr:Inferring config /var/lib/ceph/779349ca-7593-11ef-bceb-c7b262605968/mon.b/config 2024-09-18T08:05:14.381 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:05:14.382 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:05:14.382 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3' 2024-09-18T08:05:14.483 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0 2024-09-18T08:05:14.524 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:05:14.525 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:05:14.526 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0' 2024-09-18T08:05:14.589 INFO:tasks.radosbench.radosbench.0.smithi007.stdout: 69 16 97850 97834 88.5897 61.4375 0.0113218 0.0112825 2024-09-18T08:05:14.626 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6 2024-09-18T08:05:14.660 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:05:14.661 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:05:14.661 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6' 2024-09-18T08:05:14.762 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2 2024-09-18T08:05:14.810 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:05:14.810 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:05:14.811 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2' 2024-09-18T08:05:14.813 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:05:14 smithi050 ceph-mon[32371]: 4.11 scrub starts 2024-09-18T08:05:14.813 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:05:14 smithi050 ceph-mon[32371]: 4.11 scrub ok 2024-09-18T08:05:14.814 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:05:14 smithi050 ceph-mon[32371]: 7.7 scrub starts 2024-09-18T08:05:14.911 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0 2024-09-18T08:05:14.954 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2024-09-18T08:05:14.954 INFO:teuthology.orchestra.run.smithi050.stdout: "ops": [ 2024-09-18T08:05:14.954 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:05:14.954 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_repop(client.14805.0:98243 7.b e110/83 7:dc8b3d46:::benchmark_data_smithi007_73155_object98242:head v 110'6073, mlcod=110'6071)", 2024-09-18T08:05:14.954 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-09-18T08:05:14.949032+0000", 2024-09-18T08:05:14.955 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 0.0024815610000000002, 2024-09-18T08:05:14.955 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0025072390000000001, 2024-09-18T08:05:14.955 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-09-18T08:05:14.955 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-09-18T08:05:14.955 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "started", 2024-09-18T08:05:14.955 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-09-18T08:05:14.955 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:05:14.955 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-09-18T08:05:14.955 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:05:14.949032+0000", 2024-09-18T08:05:14.955 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-18T08:05:14.955 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:05:14.955 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:05:14.955 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-09-18T08:05:14.956 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:05:14.949032+0000", 2024-09-18T08:05:14.956 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-18T08:05:14.956 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:05:14.956 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:05:14.956 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-09-18T08:05:14.956 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:05:14.949033+0000", 2024-09-18T08:05:14.956 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 9.4600000000000003e-07 2024-09-18T08:05:14.956 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:05:14.956 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:05:14.956 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-09-18T08:05:14.956 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:05:14.949088+0000", 2024-09-18T08:05:14.956 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 5.4832e-05 2024-09-18T08:05:14.957 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:05:14.957 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:05:14.957 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-09-18T08:05:14.957 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:05:14.949102+0000", 2024-09-18T08:05:14.957 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.3832000000000001e-05 2024-09-18T08:05:14.957 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:05:14.957 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:05:14.957 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-09-18T08:05:14.957 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:05:14.949110+0000", 2024-09-18T08:05:14.957 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 8.5960000000000004e-06 2024-09-18T08:05:14.957 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:05:14.957 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:05:14.957 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-18T08:05:14.957 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:05:14.949168+0000", 2024-09-18T08:05:14.958 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 5.8245999999999998e-05 2024-09-18T08:05:14.958 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:05:14.958 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:05:14.958 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-09-18T08:05:14.958 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:05:14.949192+0000", 2024-09-18T08:05:14.958 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 2.3918999999999999e-05 2024-09-18T08:05:14.958 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-18T08:05:14.958 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-09-18T08:05:14.958 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-18T08:05:14.958 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-18T08:05:14.958 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2024-09-18T08:05:14.958 INFO:teuthology.orchestra.run.smithi050.stdout: "num_ops": 1 2024-09-18T08:05:14.958 INFO:teuthology.orchestra.run.smithi050.stdout:} 2024-09-18T08:05:14.959 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:05:14.959 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:05:14.960 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0' 2024-09-18T08:05:15.061 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5 2024-09-18T08:05:15.072 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:05:14 smithi007 ceph-mon[29459]: 4.11 scrub starts 2024-09-18T08:05:15.072 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:05:14 smithi007 ceph-mon[29459]: 4.11 scrub ok 2024-09-18T08:05:15.072 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:05:14 smithi007 ceph-mon[29459]: 7.7 scrub starts 2024-09-18T08:05:15.072 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:05:14 smithi007 ceph-mon[39249]: 4.11 scrub starts 2024-09-18T08:05:15.072 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:05:14 smithi007 ceph-mon[39249]: 4.11 scrub ok 2024-09-18T08:05:15.072 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:05:14 smithi007 ceph-mon[39249]: 7.7 scrub starts 2024-09-18T08:05:15.087 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:05:15.087 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:05:15.088 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5' 2024-09-18T08:05:15.163 INFO:teuthology.orchestra.run.smithi007.stdout:osd.0: {} 2024-09-18T08:05:15.164 INFO:teuthology.orchestra.run.smithi007.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-18T08:05:15.177 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/779349ca-7593-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T08:05:15.189 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0 2024-09-18T08:05:15.202 INFO:teuthology.orchestra.run.smithi007.stdout:osd.1: {} 2024-09-18T08:05:15.202 INFO:teuthology.orchestra.run.smithi007.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-18T08:05:15.244 INFO:teuthology.orchestra.run.smithi007.stdout:osd.2: {} 2024-09-18T08:05:15.244 INFO:teuthology.orchestra.run.smithi007.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-18T08:05:15.250 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:05:15.251 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:05:15.251 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0' 2024-09-18T08:05:15.281 INFO:teuthology.orchestra.run.smithi007.stdout:osd.3: {} 2024-09-18T08:05:15.282 INFO:teuthology.orchestra.run.smithi007.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-18T08:05:15.316 INFO:teuthology.orchestra.run.smithi007.stdout:osd.4: {} 2024-09-18T08:05:15.317 INFO:teuthology.orchestra.run.smithi007.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-18T08:05:15.350 INFO:teuthology.orchestra.run.smithi007.stdout:osd.5: {} 2024-09-18T08:05:15.351 INFO:teuthology.orchestra.run.smithi007.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-18T08:05:15.352 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6 2024-09-18T08:05:15.379 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:05:15.380 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:05:15.380 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6' 2024-09-18T08:05:15.386 INFO:teuthology.orchestra.run.smithi007.stdout:osd.6: {} 2024-09-18T08:05:15.386 INFO:teuthology.orchestra.run.smithi007.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-18T08:05:15.420 INFO:teuthology.orchestra.run.smithi007.stdout:osd.7: {} 2024-09-18T08:05:15.421 INFO:teuthology.orchestra.run.smithi007.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-18T08:05:15.480 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7 2024-09-18T08:05:15.510 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:05:15.510 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:05:15.511 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7' 2024-09-18T08:05:15.589 INFO:tasks.radosbench.radosbench.0.smithi007.stdout: 70 16 98883 98867 88.2464 64.5625 0.00625862 0.0113263 2024-09-18T08:05:15.611 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5 2024-09-18T08:05:15.640 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:05:15.641 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:05:15.641 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5' 2024-09-18T08:05:15.688 DEBUG:teuthology.orchestra.run.smithi050:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid 779349ca-7593-11ef-bceb-c7b262605968 -- ceph daemon osd.6 dump_blocked_ops 2024-09-18T08:05:15.742 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2 2024-09-18T08:05:15.782 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:05:15.782 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:05:15.783 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2' 2024-09-18T08:05:15.884 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0 2024-09-18T08:05:15.905 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:05:15 smithi050 ceph-mon[32371]: pgmap v99: 148 pgs: 3 active+clean+scrubbing, 145 active+clean; 5.9 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 1.2 KiB/s rd, 74 MiB/s wr, 1.19k op/s 2024-09-18T08:05:15.906 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:05:15 smithi050 ceph-mon[32371]: from='client.? 172.21.15.7:0/2276321166' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-18T08:05:15.924 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:05:15.925 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:05:15.925 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0' 2024-09-18T08:05:16.026 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4 2024-09-18T08:05:16.051 INFO:teuthology.orchestra.run.smithi050.stderr:Inferring config /var/lib/ceph/779349ca-7593-11ef-bceb-c7b262605968/mon.b/config 2024-09-18T08:05:16.064 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:05:16.064 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:05:16.065 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4' 2024-09-18T08:05:16.067 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:05:15 smithi007 ceph-mon[29459]: pgmap v99: 148 pgs: 3 active+clean+scrubbing, 145 active+clean; 5.9 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 1.2 KiB/s rd, 74 MiB/s wr, 1.19k op/s 2024-09-18T08:05:16.068 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:05:15 smithi007 ceph-mon[29459]: from='client.? 172.21.15.7:0/2276321166' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-18T08:05:16.068 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:05:15 smithi007 ceph-mon[39249]: pgmap v99: 148 pgs: 3 active+clean+scrubbing, 145 active+clean; 5.9 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 1.2 KiB/s rd, 74 MiB/s wr, 1.19k op/s 2024-09-18T08:05:16.068 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:05:15 smithi007 ceph-mon[39249]: from='client.? 172.21.15.7:0/2276321166' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-18T08:05:16.165 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6 2024-09-18T08:05:16.196 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:05:16.197 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:05:16.198 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6' 2024-09-18T08:05:16.298 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2 2024-09-18T08:05:16.342 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:05:16.342 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:05:16.343 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2' 2024-09-18T08:05:16.443 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4 2024-09-18T08:05:16.472 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:05:16.473 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:05:16.473 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4' 2024-09-18T08:05:16.513 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid 779349ca-7593-11ef-bceb-c7b262605968 -- ceph osd set nodeep-scrub 2024-09-18T08:05:16.574 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7 2024-09-18T08:05:16.590 INFO:tasks.radosbench.radosbench.0.smithi007.stdout: 71 16 99987 99971 87.9743 69 0.015361 0.0113612 2024-09-18T08:05:16.600 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:05:16.600 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:05:16.601 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7' 2024-09-18T08:05:16.639 INFO:teuthology.orchestra.run.smithi050.stdout:ERROR: (22) Invalid argument 2024-09-18T08:05:16.639 INFO:teuthology.orchestra.run.smithi050.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-18T08:05:16.702 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7 2024-09-18T08:05:16.730 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:05:16.731 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Sep 18 08:05:16 smithi007 ceph-779349ca-7593-11ef-bceb-c7b262605968-mgr-y[29688]: ::ffff:172.21.15.50 - - [18/Sep/2024:08:05:16] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-09-18T08:05:16.731 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:05:16.731 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7' 2024-09-18T08:05:16.833 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3 2024-09-18T08:05:16.879 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:05:16.880 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:05:16.880 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3' 2024-09-18T08:05:16.888 INFO:teuthology.orchestra.run.smithi007.stderr:reweighted osd.1 to 1 (10000) 2024-09-18T08:05:16.981 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0 2024-09-18T08:05:17.032 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:05:17.032 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:05:17.033 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0' 2024-09-18T08:05:17.133 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1 2024-09-18T08:05:17.174 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/779349ca-7593-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T08:05:17.194 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:05:17.194 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:05:17.195 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1' 2024-09-18T08:05:17.230 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:05:16 smithi007 ceph-mon[29459]: from='client.? 172.21.15.7:0/2040824314' entity='client.admin' cmd={"prefix": "osd reweight", "id": 1, "weight": 1.0} : dispatch 2024-09-18T08:05:17.231 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:05:16 smithi007 ceph-mon[39249]: from='client.? 172.21.15.7:0/2040824314' entity='client.admin' cmd={"prefix": "osd reweight", "id": 1, "weight": 1.0} : dispatch 2024-09-18T08:05:17.295 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3 2024-09-18T08:05:17.337 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:05:17.338 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:05:17.338 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3' 2024-09-18T08:05:17.406 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:05:16 smithi050 ceph-mon[32371]: from='client.? 172.21.15.7:0/2040824314' entity='client.admin' cmd={"prefix": "osd reweight", "id": 1, "weight": 1.0} : dispatch 2024-09-18T08:05:17.439 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3 2024-09-18T08:05:17.485 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:05:17.486 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:05:17.486 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3' 2024-09-18T08:05:17.587 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0 2024-09-18T08:05:17.590 INFO:tasks.radosbench.radosbench.0.smithi007.stdout: 72 16 100974 100958 87.6092 61.6875 0.0208425 0.011409 2024-09-18T08:05:17.630 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:05:17.631 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:05:17.632 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0' 2024-09-18T08:05:17.732 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4 2024-09-18T08:05:17.758 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:05:17.759 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:05:17.760 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4' 2024-09-18T08:05:17.780 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid 779349ca-7593-11ef-bceb-c7b262605968 -- ceph osd reweight 2 1 2024-09-18T08:05:17.860 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5 2024-09-18T08:05:17.888 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:05:17.889 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:05:17.889 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5' 2024-09-18T08:05:17.980 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:05:17 smithi007 ceph-mon[29459]: from='client.? 172.21.15.7:0/2040824314' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]': finished 2024-09-18T08:05:17.981 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:05:17 smithi007 ceph-mon[29459]: osdmap e111: 8 total, 8 up, 8 in 2024-09-18T08:05:17.981 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:05:17 smithi007 ceph-mon[29459]: pgmap v101: 148 pgs: 3 active+clean+scrubbing, 145 active+clean; 5.9 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 852 B/s rd, 46 MiB/s wr, 733 op/s 2024-09-18T08:05:17.981 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:05:17 smithi007 ceph-mon[29459]: from='mgr.24470 172.21.15.7:0/2594390261' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-18T08:05:17.981 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:05:17 smithi007 ceph-mon[39249]: from='client.? 172.21.15.7:0/2040824314' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]': finished 2024-09-18T08:05:17.981 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:05:17 smithi007 ceph-mon[39249]: osdmap e111: 8 total, 8 up, 8 in 2024-09-18T08:05:17.981 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:05:17 smithi007 ceph-mon[39249]: pgmap v101: 148 pgs: 3 active+clean+scrubbing, 145 active+clean; 5.9 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 852 B/s rd, 46 MiB/s wr, 733 op/s 2024-09-18T08:05:17.981 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:05:17 smithi007 ceph-mon[39249]: from='mgr.24470 172.21.15.7:0/2594390261' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-18T08:05:17.990 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2 2024-09-18T08:05:18.059 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:05:18.060 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:05:18.060 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2' 2024-09-18T08:05:18.123 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid 779349ca-7593-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-18T08:05:18.156 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:05:17 smithi050 ceph-mon[32371]: from='client.? 172.21.15.7:0/2040824314' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]': finished 2024-09-18T08:05:18.156 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:05:17 smithi050 ceph-mon[32371]: osdmap e111: 8 total, 8 up, 8 in 2024-09-18T08:05:18.156 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:05:17 smithi050 ceph-mon[32371]: pgmap v101: 148 pgs: 3 active+clean+scrubbing, 145 active+clean; 5.9 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 852 B/s rd, 46 MiB/s wr, 733 op/s 2024-09-18T08:05:18.156 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:05:17 smithi050 ceph-mon[32371]: from='mgr.24470 172.21.15.7:0/2594390261' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-18T08:05:18.161 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0 2024-09-18T08:05:18.195 DEBUG:teuthology.orchestra.run.smithi050:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid 779349ca-7593-11ef-bceb-c7b262605968 -- ceph daemon osd.6 dump_historic_ops 2024-09-18T08:05:18.226 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:05:18.227 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:05:18.227 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0' 2024-09-18T08:05:18.328 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3 2024-09-18T08:05:18.382 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:05:18.383 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:05:18.383 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3' 2024-09-18T08:05:18.412 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/779349ca-7593-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T08:05:18.484 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2 2024-09-18T08:05:18.507 INFO:teuthology.orchestra.run.smithi050.stderr:Inferring config /var/lib/ceph/779349ca-7593-11ef-bceb-c7b262605968/mon.b/config 2024-09-18T08:05:18.532 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:05:18.532 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:05:18.533 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2' 2024-09-18T08:05:18.591 INFO:tasks.radosbench.radosbench.0.smithi007.stdout: 73 16 101882 101866 87.1857 56.75 0.015381 0.011464 2024-09-18T08:05:18.634 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5 2024-09-18T08:05:18.661 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:05:18.661 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:05:18.662 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5' 2024-09-18T08:05:18.763 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1 2024-09-18T08:05:18.800 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:05:18.801 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:05:18.801 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1' 2024-09-18T08:05:18.902 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4 2024-09-18T08:05:18.906 INFO:journalctl@ceph.iscsi.iscsi.a.smithi050.stdout:Sep 18 08:05:18 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-iscsi-iscsi-a[57782]: debug there is no tcmu-runner data available 2024-09-18T08:05:18.927 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:05:18.927 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:05:18.928 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4' 2024-09-18T08:05:18.952 INFO:teuthology.orchestra.run.smithi007.stderr:nodeep-scrub is set 2024-09-18T08:05:19.029 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1 2024-09-18T08:05:19.075 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:05:19.076 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:05:19.076 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1' 2024-09-18T08:05:19.177 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0 2024-09-18T08:05:19.194 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:05:18 smithi050 ceph-mon[32371]: purged_snaps scrub starts 2024-09-18T08:05:19.194 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:05:18 smithi050 ceph-mon[32371]: purged_snaps scrub ok 2024-09-18T08:05:19.194 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:05:18 smithi050 ceph-mon[32371]: osdmap e112: 8 total, 8 up, 8 in 2024-09-18T08:05:19.194 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:05:18 smithi050 ceph-mon[32371]: from='client.? 172.21.15.7:0/3064611272' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-18T08:05:19.194 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:05:18 smithi050 ceph-mon[32371]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-18T08:05:19.220 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:05:19.220 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:05:19.221 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0' 2024-09-18T08:05:19.230 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:05:18 smithi007 ceph-mon[29459]: purged_snaps scrub starts 2024-09-18T08:05:19.231 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:05:18 smithi007 ceph-mon[29459]: purged_snaps scrub ok 2024-09-18T08:05:19.231 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:05:18 smithi007 ceph-mon[29459]: osdmap e112: 8 total, 8 up, 8 in 2024-09-18T08:05:19.231 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:05:18 smithi007 ceph-mon[29459]: from='client.? 172.21.15.7:0/3064611272' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-18T08:05:19.231 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:05:18 smithi007 ceph-mon[29459]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-18T08:05:19.232 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:05:18 smithi007 ceph-mon[39249]: purged_snaps scrub starts 2024-09-18T08:05:19.232 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:05:18 smithi007 ceph-mon[39249]: purged_snaps scrub ok 2024-09-18T08:05:19.232 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:05:18 smithi007 ceph-mon[39249]: osdmap e112: 8 total, 8 up, 8 in 2024-09-18T08:05:19.232 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:05:18 smithi007 ceph-mon[39249]: from='client.? 172.21.15.7:0/3064611272' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-18T08:05:19.232 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:05:18 smithi007 ceph-mon[39249]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-18T08:05:19.322 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6 2024-09-18T08:05:19.348 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:05:19.349 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:05:19.349 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6' 2024-09-18T08:05:19.450 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2 2024-09-18T08:05:19.496 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:05:19.497 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:05:19.497 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2' 2024-09-18T08:05:19.591 INFO:tasks.radosbench.radosbench.0.smithi007.stdout: 74 16 102851 102835 86.8259 60.5625 0.00841915 0.0115124 2024-09-18T08:05:19.598 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6 2024-09-18T08:05:19.617 INFO:teuthology.orchestra.run.smithi050.stdout:ERROR: (22) Invalid argument 2024-09-18T08:05:19.617 INFO:teuthology.orchestra.run.smithi050.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-18T08:05:19.633 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:05:19.633 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:05:19.634 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6' 2024-09-18T08:05:19.734 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4 2024-09-18T08:05:19.761 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:05:19.762 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:05:19.762 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4' 2024-09-18T08:05:19.775 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/779349ca-7593-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T08:05:19.863 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7 2024-09-18T08:05:19.897 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:05:19.898 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:05:19.898 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7' 2024-09-18T08:05:19.999 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0 2024-09-18T08:05:20.052 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:05:20.053 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:05:20.053 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0' 2024-09-18T08:05:20.101 INFO:teuthology.orchestra.run.smithi007.stderr:reweighted osd.2 to 1 (10000) 2024-09-18T08:05:20.155 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5 2024-09-18T08:05:20.182 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:05:20.183 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:05:20.183 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5' 2024-09-18T08:05:20.230 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:05:19 smithi007 ceph-mon[29459]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-18T08:05:20.231 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:05:19 smithi007 ceph-mon[29459]: purged_snaps scrub starts 2024-09-18T08:05:20.231 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:05:19 smithi007 ceph-mon[29459]: purged_snaps scrub ok 2024-09-18T08:05:20.231 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:05:19 smithi007 ceph-mon[29459]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-18T08:05:20.231 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:05:19 smithi007 ceph-mon[29459]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-18T08:05:20.231 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:05:19 smithi007 ceph-mon[29459]: osdmap e113: 8 total, 8 up, 8 in 2024-09-18T08:05:20.231 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:05:19 smithi007 ceph-mon[29459]: osdmap e114: 8 total, 8 up, 8 in 2024-09-18T08:05:20.231 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:05:19 smithi007 ceph-mon[29459]: pgmap v105: 148 pgs: 148 active+clean; 6.2 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 1.2 KiB/s rd, 77 MiB/s wr, 1.23k op/s 2024-09-18T08:05:20.231 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:05:19 smithi007 ceph-mon[29459]: from='client.? 172.21.15.7:0/2569040549' entity='client.admin' cmd={"prefix": "osd reweight", "id": 2, "weight": 1.0} : dispatch 2024-09-18T08:05:20.231 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:05:19 smithi007 ceph-mon[29459]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 2, "weight": 1.0} : dispatch 2024-09-18T08:05:20.231 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:05:19 smithi007 ceph-mon[39249]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-18T08:05:20.232 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:05:19 smithi007 ceph-mon[39249]: purged_snaps scrub starts 2024-09-18T08:05:20.232 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:05:19 smithi007 ceph-mon[39249]: purged_snaps scrub ok 2024-09-18T08:05:20.232 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:05:19 smithi007 ceph-mon[39249]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-18T08:05:20.232 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:05:19 smithi007 ceph-mon[39249]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-18T08:05:20.232 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:05:19 smithi007 ceph-mon[39249]: osdmap e113: 8 total, 8 up, 8 in 2024-09-18T08:05:20.232 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:05:19 smithi007 ceph-mon[39249]: osdmap e114: 8 total, 8 up, 8 in 2024-09-18T08:05:20.232 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:05:19 smithi007 ceph-mon[39249]: pgmap v105: 148 pgs: 148 active+clean; 6.2 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 1.2 KiB/s rd, 77 MiB/s wr, 1.23k op/s 2024-09-18T08:05:20.232 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:05:19 smithi007 ceph-mon[39249]: from='client.? 172.21.15.7:0/2569040549' entity='client.admin' cmd={"prefix": "osd reweight", "id": 2, "weight": 1.0} : dispatch 2024-09-18T08:05:20.232 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:05:19 smithi007 ceph-mon[39249]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 2, "weight": 1.0} : dispatch 2024-09-18T08:05:20.284 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7 2024-09-18T08:05:20.313 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:05:20.314 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:05:20.314 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7' 2024-09-18T08:05:20.353 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:05:19 smithi050 ceph-mon[32371]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-18T08:05:20.353 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:05:19 smithi050 ceph-mon[32371]: purged_snaps scrub starts 2024-09-18T08:05:20.353 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:05:19 smithi050 ceph-mon[32371]: purged_snaps scrub ok 2024-09-18T08:05:20.353 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:05:19 smithi050 ceph-mon[32371]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-18T08:05:20.354 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:05:19 smithi050 ceph-mon[32371]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-18T08:05:20.354 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:05:19 smithi050 ceph-mon[32371]: osdmap e113: 8 total, 8 up, 8 in 2024-09-18T08:05:20.354 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:05:19 smithi050 ceph-mon[32371]: osdmap e114: 8 total, 8 up, 8 in 2024-09-18T08:05:20.354 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:05:19 smithi050 ceph-mon[32371]: pgmap v105: 148 pgs: 148 active+clean; 6.2 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 1.2 KiB/s rd, 77 MiB/s wr, 1.23k op/s 2024-09-18T08:05:20.354 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:05:19 smithi050 ceph-mon[32371]: from='client.? 172.21.15.7:0/2569040549' entity='client.admin' cmd={"prefix": "osd reweight", "id": 2, "weight": 1.0} : dispatch 2024-09-18T08:05:20.354 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:05:19 smithi050 ceph-mon[32371]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 2, "weight": 1.0} : dispatch 2024-09-18T08:05:20.399 DEBUG:teuthology.orchestra.run.smithi050:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid 779349ca-7593-11ef-bceb-c7b262605968 -- ceph daemon osd.7 dump_ops_in_flight 2024-09-18T08:05:20.415 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1 2024-09-18T08:05:20.454 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:05:20.454 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:05:20.455 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1' 2024-09-18T08:05:20.556 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4 2024-09-18T08:05:20.584 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:05:20.585 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:05:20.586 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4' 2024-09-18T08:05:20.591 INFO:tasks.radosbench.radosbench.0.smithi007.stdout: 75 16 103954 103938 86.5874 68.9375 0.0192625 0.0115433 2024-09-18T08:05:20.686 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3 2024-09-18T08:05:20.730 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:05:20.731 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:05:20.731 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3' 2024-09-18T08:05:20.769 INFO:teuthology.orchestra.run.smithi050.stderr:Inferring config /var/lib/ceph/779349ca-7593-11ef-bceb-c7b262605968/mon.b/config 2024-09-18T08:05:20.777 INFO:teuthology.orchestra.run.smithi007.stdout:osd.0: {} 2024-09-18T08:05:20.778 INFO:teuthology.orchestra.run.smithi007.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-18T08:05:20.830 INFO:teuthology.orchestra.run.smithi007.stdout:osd.1: {} 2024-09-18T08:05:20.830 INFO:teuthology.orchestra.run.smithi007.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-18T08:05:20.832 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7 2024-09-18T08:05:20.860 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:05:20.861 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:05:20.862 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7' 2024-09-18T08:05:20.871 INFO:teuthology.orchestra.run.smithi007.stdout:osd.2: {} 2024-09-18T08:05:20.871 INFO:teuthology.orchestra.run.smithi007.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-18T08:05:20.917 INFO:teuthology.orchestra.run.smithi007.stdout:osd.3: {} 2024-09-18T08:05:20.917 INFO:teuthology.orchestra.run.smithi007.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-18T08:05:20.951 INFO:teuthology.orchestra.run.smithi007.stdout:osd.4: {} 2024-09-18T08:05:20.951 INFO:teuthology.orchestra.run.smithi007.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-18T08:05:20.963 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6 2024-09-18T08:05:20.987 INFO:teuthology.orchestra.run.smithi007.stdout:osd.5: {} 2024-09-18T08:05:20.988 INFO:teuthology.orchestra.run.smithi007.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-18T08:05:20.988 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:05:20.989 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:05:20.989 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6' 2024-09-18T08:05:21.024 INFO:teuthology.orchestra.run.smithi007.stdout:osd.6: {} 2024-09-18T08:05:21.024 INFO:teuthology.orchestra.run.smithi007.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-18T08:05:21.061 INFO:teuthology.orchestra.run.smithi007.stdout:osd.7: {} 2024-09-18T08:05:21.061 INFO:teuthology.orchestra.run.smithi007.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-18T08:05:21.090 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5 2024-09-18T08:05:21.119 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:05:21.120 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:05:21.120 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5' 2024-09-18T08:05:21.156 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:05:21 smithi050 ceph-mon[32371]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]': finished 2024-09-18T08:05:21.156 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:05:21 smithi050 ceph-mon[32371]: osdmap e115: 8 total, 8 up, 8 in 2024-09-18T08:05:21.156 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:05:21 smithi050 ceph-mon[32371]: from='client.? 172.21.15.7:0/4065055551' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-18T08:05:21.222 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0 2024-09-18T08:05:21.265 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:05:21.266 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:05:21.266 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0' 2024-09-18T08:05:21.319 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2024-09-18T08:05:21.319 INFO:teuthology.orchestra.run.smithi050.stdout: "ops": [ 2024-09-18T08:05:21.320 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:05:21.320 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_repop(client.14805.0:104709 7.6 e115/83 7:6e233322:::benchmark_data_smithi007_73155_object104708:head v 115'6481, mlcod=115'6477)", 2024-09-18T08:05:21.320 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-09-18T08:05:21.314994+0000", 2024-09-18T08:05:21.320 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 0.0018928790000000001, 2024-09-18T08:05:21.320 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.001931828, 2024-09-18T08:05:21.320 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-09-18T08:05:21.320 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-09-18T08:05:21.320 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "started", 2024-09-18T08:05:21.320 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-09-18T08:05:21.320 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:05:21.320 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-09-18T08:05:21.320 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:05:21.314994+0000", 2024-09-18T08:05:21.320 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-18T08:05:21.321 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:05:21.321 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:05:21.321 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-09-18T08:05:21.321 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:05:21.314994+0000", 2024-09-18T08:05:21.321 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-18T08:05:21.321 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:05:21.321 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:05:21.321 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-09-18T08:05:21.321 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:05:21.314994+0000", 2024-09-18T08:05:21.321 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 6.4499999999999997e-07 2024-09-18T08:05:21.321 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:05:21.321 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:05:21.321 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-09-18T08:05:21.322 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:05:21.315069+0000", 2024-09-18T08:05:21.322 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 7.4665000000000004e-05 2024-09-18T08:05:21.322 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:05:21.322 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:05:21.322 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-09-18T08:05:21.322 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:05:21.315084+0000", 2024-09-18T08:05:21.322 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.4618e-05 2024-09-18T08:05:21.322 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:05:21.322 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:05:21.322 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-09-18T08:05:21.322 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:05:21.315091+0000", 2024-09-18T08:05:21.322 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 7.1230000000000002e-06 2024-09-18T08:05:21.322 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:05:21.322 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:05:21.322 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-18T08:05:21.323 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:05:21.315155+0000", 2024-09-18T08:05:21.323 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 6.4571000000000002e-05 2024-09-18T08:05:21.323 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:05:21.323 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:05:21.323 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-09-18T08:05:21.323 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:05:21.315177+0000", 2024-09-18T08:05:21.323 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 2.1562000000000001e-05 2024-09-18T08:05:21.323 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-18T08:05:21.323 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-09-18T08:05:21.323 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-18T08:05:21.323 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-18T08:05:21.323 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2024-09-18T08:05:21.323 INFO:teuthology.orchestra.run.smithi050.stdout: "num_ops": 1 2024-09-18T08:05:21.323 INFO:teuthology.orchestra.run.smithi050.stdout:} 2024-09-18T08:05:21.367 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3 2024-09-18T08:05:21.401 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:05:21.402 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:05:21.402 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3' 2024-09-18T08:05:21.412 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:05:21 smithi007 ceph-mon[29459]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]': finished 2024-09-18T08:05:21.412 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:05:21 smithi007 ceph-mon[29459]: osdmap e115: 8 total, 8 up, 8 in 2024-09-18T08:05:21.412 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:05:21 smithi007 ceph-mon[29459]: from='client.? 172.21.15.7:0/4065055551' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-18T08:05:21.413 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:05:21 smithi007 ceph-mon[39249]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]': finished 2024-09-18T08:05:21.413 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:05:21 smithi007 ceph-mon[39249]: osdmap e115: 8 total, 8 up, 8 in 2024-09-18T08:05:21.413 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:05:21 smithi007 ceph-mon[39249]: from='client.? 172.21.15.7:0/4065055551' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-18T08:05:21.465 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid 779349ca-7593-11ef-bceb-c7b262605968 -- ceph osd reweight 3 1 2024-09-18T08:05:21.504 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3 2024-09-18T08:05:21.557 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:05:21.558 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:05:21.559 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3' 2024-09-18T08:05:21.591 INFO:tasks.radosbench.radosbench.0.smithi007.stdout: 76 16 104990 104974 86.3 64.75 0.00418524 0.0115819 2024-09-18T08:05:21.660 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6 2024-09-18T08:05:21.690 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:05:21.691 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:05:21.691 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6' 2024-09-18T08:05:21.728 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid 779349ca-7593-11ef-bceb-c7b262605968 -- ceph osd unset noscrub 2024-09-18T08:05:21.792 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2 2024-09-18T08:05:21.859 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:05:21.859 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:05:21.859 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2' 2024-09-18T08:05:21.960 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7 2024-09-18T08:05:21.989 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:05:21.989 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:05:21.990 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7' 2024-09-18T08:05:22.027 DEBUG:teuthology.orchestra.run.smithi050:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid 779349ca-7593-11ef-bceb-c7b262605968 -- ceph daemon osd.7 dump_blocked_ops 2024-09-18T08:05:22.091 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7 2024-09-18T08:05:22.139 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:05:22.139 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:05:22.140 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7' 2024-09-18T08:05:22.241 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4 2024-09-18T08:05:22.263 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:05:22.264 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:05:22.264 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4' 2024-09-18T08:05:22.365 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7 2024-09-18T08:05:22.405 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:05:22 smithi050 ceph-mon[32371]: pgmap v107: 148 pgs: 148 active+clean; 6.2 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 1.2 KiB/s rd, 77 MiB/s wr, 1.23k op/s 2024-09-18T08:05:22.406 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:05:22.407 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:05:22.407 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7' 2024-09-18T08:05:22.419 INFO:teuthology.orchestra.run.smithi050.stderr:Inferring config /var/lib/ceph/779349ca-7593-11ef-bceb-c7b262605968/mon.b/config 2024-09-18T08:05:22.481 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:05:22 smithi007 ceph-mon[29459]: pgmap v107: 148 pgs: 148 active+clean; 6.2 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 1.2 KiB/s rd, 77 MiB/s wr, 1.23k op/s 2024-09-18T08:05:22.481 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:05:22 smithi007 ceph-mon[39249]: pgmap v107: 148 pgs: 148 active+clean; 6.2 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 1.2 KiB/s rd, 77 MiB/s wr, 1.23k op/s 2024-09-18T08:05:22.508 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0 2024-09-18T08:05:22.532 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:05:22.533 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:05:22.533 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0' 2024-09-18T08:05:22.591 INFO:tasks.radosbench.radosbench.0.smithi007.stdout: 77 16 105944 105928 85.9536 59.625 0.0319142 0.0116282 2024-09-18T08:05:22.634 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0 2024-09-18T08:05:22.679 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:05:22.680 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:05:22.680 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0' 2024-09-18T08:05:22.781 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5 2024-09-18T08:05:22.812 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:05:22.812 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:05:22.813 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5' 2024-09-18T08:05:22.914 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2 2024-09-18T08:05:22.951 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:05:22.952 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:05:22.953 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2' 2024-09-18T08:05:22.990 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2024-09-18T08:05:22.990 INFO:teuthology.orchestra.run.smithi050.stdout: "ops": [], 2024-09-18T08:05:22.990 INFO:teuthology.orchestra.run.smithi050.stdout: "complaint_time": 30, 2024-09-18T08:05:22.990 INFO:teuthology.orchestra.run.smithi050.stdout: "num_blocked_ops": 0 2024-09-18T08:05:22.990 INFO:teuthology.orchestra.run.smithi050.stdout:} 2024-09-18T08:05:23.053 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4 2024-09-18T08:05:23.083 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:05:23.083 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:05:23.084 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4' 2024-09-18T08:05:23.185 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3 2024-09-18T08:05:23.227 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:05:23.228 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:05:23.229 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3' 2024-09-18T08:05:23.329 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7 2024-09-18T08:05:23.354 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:05:23.355 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:05:23.355 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7' 2024-09-18T08:05:23.394 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/779349ca-7593-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T08:05:23.398 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/779349ca-7593-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T08:05:23.456 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3 2024-09-18T08:05:23.506 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:05:23.506 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:05:23.507 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3' 2024-09-18T08:05:23.591 INFO:tasks.radosbench.radosbench.0.smithi007.stdout: 78 16 107006 106990 85.7026 66.375 0.0150396 0.0116621 2024-09-18T08:05:23.608 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3 2024-09-18T08:05:23.652 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:05:23.652 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:05:23.653 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3' 2024-09-18T08:05:23.753 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7 2024-09-18T08:05:23.766 DEBUG:teuthology.orchestra.run.smithi050:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid 779349ca-7593-11ef-bceb-c7b262605968 -- ceph daemon osd.7 dump_historic_ops 2024-09-18T08:05:23.785 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:05:23.786 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:05:23.786 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7' 2024-09-18T08:05:23.887 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1 2024-09-18T08:05:23.928 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:05:23.929 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:05:23.929 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1' 2024-09-18T08:05:24.030 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6 2024-09-18T08:05:24.064 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:05:24.064 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:05:24.065 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6' 2024-09-18T08:05:24.159 INFO:teuthology.orchestra.run.smithi050.stderr:Inferring config /var/lib/ceph/779349ca-7593-11ef-bceb-c7b262605968/mon.b/config 2024-09-18T08:05:24.166 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7 2024-09-18T08:05:24.198 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:05:24.199 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:05:24.199 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7' 2024-09-18T08:05:24.300 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2 2024-09-18T08:05:24.343 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:05:24.344 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:05:24.344 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2' 2024-09-18T08:05:24.363 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:05:24 smithi007 ceph-mon[29459]: pgmap v108: 148 pgs: 148 active+clean; 6.3 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 1.9 KiB/s rd, 71 MiB/s wr, 1.13k op/s 2024-09-18T08:05:24.363 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:05:24 smithi007 ceph-mon[29459]: osdmap e116: 8 total, 8 up, 8 in 2024-09-18T08:05:24.363 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:05:24 smithi007 ceph-mon[39249]: pgmap v108: 148 pgs: 148 active+clean; 6.3 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 1.9 KiB/s rd, 71 MiB/s wr, 1.13k op/s 2024-09-18T08:05:24.363 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:05:24 smithi007 ceph-mon[39249]: osdmap e116: 8 total, 8 up, 8 in 2024-09-18T08:05:24.406 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:05:24 smithi050 ceph-mon[32371]: pgmap v108: 148 pgs: 148 active+clean; 6.3 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 1.9 KiB/s rd, 71 MiB/s wr, 1.13k op/s 2024-09-18T08:05:24.406 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:05:24 smithi050 ceph-mon[32371]: osdmap e116: 8 total, 8 up, 8 in 2024-09-18T08:05:24.445 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2 2024-09-18T08:05:24.481 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:05:24.482 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:05:24.482 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2' 2024-09-18T08:05:24.583 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5 2024-09-18T08:05:24.591 INFO:tasks.radosbench.radosbench.0.smithi007.stdout: 79 16 108089 108073 85.4745 67.6875 0.0045733 0.0116936 2024-09-18T08:05:24.613 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:05:24.614 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:05:24.614 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5' 2024-09-18T08:05:24.715 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3 2024-09-18T08:05:24.764 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:05:24.765 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:05:24.765 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3' 2024-09-18T08:05:24.866 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4 2024-09-18T08:05:24.897 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:05:24.898 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:05:24.898 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4' 2024-09-18T08:05:24.957 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2024-09-18T08:05:24.957 INFO:teuthology.orchestra.run.smithi050.stdout: "size": 20, 2024-09-18T08:05:24.957 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 600, 2024-09-18T08:05:24.957 INFO:teuthology.orchestra.run.smithi050.stdout: "ops": [ 2024-09-18T08:05:24.957 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:05:24.957 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.14805.0:2069 7.5 7:ab5451f1:::benchmark_data_smithi007_73155_object2068:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e88)", 2024-09-18T08:05:24.957 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-09-18T08:04:06.382124+0000", 2024-09-18T08:05:24.957 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 78.570802920000006, 2024-09-18T08:05:24.957 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.038389829, 2024-09-18T08:05:24.958 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-09-18T08:05:24.958 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-09-18T08:05:24.958 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T08:05:24.958 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2024-09-18T08:05:24.958 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.14805", 2024-09-18T08:05:24.958 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "172.21.15.7:0/640320436", 2024-09-18T08:05:24.958 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 2069 2024-09-18T08:05:24.958 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:05:24.958 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-09-18T08:05:24.958 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:05:24.958 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-09-18T08:05:24.958 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:06.382124+0000", 2024-09-18T08:05:24.958 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-18T08:05:24.959 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:05:24.959 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:05:24.959 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-09-18T08:05:24.959 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:06.382124+0000", 2024-09-18T08:05:24.959 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-18T08:05:24.959 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:05:24.959 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:05:24.959 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-09-18T08:05:24.959 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:06.382126+0000", 2024-09-18T08:05:24.959 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.5510000000000001e-06 2024-09-18T08:05:24.959 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:05:24.959 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:05:24.959 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-09-18T08:05:24.959 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:06.382210+0000", 2024-09-18T08:05:24.960 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 8.4648000000000007e-05 2024-09-18T08:05:24.960 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:05:24.960 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:05:24.960 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-09-18T08:05:24.960 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:06.382222+0000", 2024-09-18T08:05:24.960 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.1421e-05 2024-09-18T08:05:24.960 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:05:24.960 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:05:24.960 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-09-18T08:05:24.960 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:06.382230+0000", 2024-09-18T08:05:24.960 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 7.7470000000000005e-06 2024-09-18T08:05:24.960 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:05:24.960 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:05:24.961 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-18T08:05:24.961 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:06.382396+0000", 2024-09-18T08:05:24.961 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00016597399999999999 2024-09-18T08:05:24.961 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:05:24.961 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:05:24.961 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-09-18T08:05:24.961 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:06.382490+0000", 2024-09-18T08:05:24.961 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 9.4170000000000001e-05 2024-09-18T08:05:24.961 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:05:24.961 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:05:24.961 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "waiting for subops from 0,2", 2024-09-18T08:05:24.961 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:06.382620+0000", 2024-09-18T08:05:24.961 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00013054599999999999 2024-09-18T08:05:24.962 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:05:24.962 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:05:24.962 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "op_commit", 2024-09-18T08:05:24.962 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:06.383044+0000", 2024-09-18T08:05:24.962 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00042359 2024-09-18T08:05:24.962 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:05:24.962 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:05:24.962 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:05:24.962 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:06.386704+0000", 2024-09-18T08:05:24.962 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0036601400000000001 2024-09-18T08:05:24.962 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:05:24.962 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:05:24.962 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:05:24.963 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:06.420445+0000", 2024-09-18T08:05:24.963 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.033740745000000003 2024-09-18T08:05:24.963 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:05:24.963 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:05:24.963 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "commit_sent", 2024-09-18T08:05:24.963 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:06.420481+0000", 2024-09-18T08:05:24.963 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 3.6677999999999997e-05 2024-09-18T08:05:24.963 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:05:24.963 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:05:24.963 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-09-18T08:05:24.963 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:06.420514+0000", 2024-09-18T08:05:24.963 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 3.2619000000000001e-05 2024-09-18T08:05:24.963 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-18T08:05:24.964 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-09-18T08:05:24.964 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-18T08:05:24.964 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:05:24.964 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:05:24.964 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.14805.0:2077 7.5 7:a54e3a91:::benchmark_data_smithi007_73155_object2076:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e88)", 2024-09-18T08:05:24.964 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-09-18T08:04:06.384665+0000", 2024-09-18T08:05:24.964 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 78.568261913000001, 2024-09-18T08:05:24.964 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.036025275000000002, 2024-09-18T08:05:24.964 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-09-18T08:05:24.964 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-09-18T08:05:24.964 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T08:05:24.964 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2024-09-18T08:05:24.965 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.14805", 2024-09-18T08:05:24.965 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "172.21.15.7:0/640320436", 2024-09-18T08:05:24.965 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 2077 2024-09-18T08:05:24.965 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:05:24.965 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-09-18T08:05:24.965 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:05:24.965 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-09-18T08:05:24.965 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:06.384665+0000", 2024-09-18T08:05:24.965 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-18T08:05:24.965 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:05:24.965 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:05:24.965 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-09-18T08:05:24.965 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:06.384665+0000", 2024-09-18T08:05:24.966 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-18T08:05:24.966 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:05:24.966 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:05:24.966 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-09-18T08:05:24.966 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:06.384666+0000", 2024-09-18T08:05:24.966 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 7.4099999999999998e-07 2024-09-18T08:05:24.966 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:05:24.966 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:05:24.966 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-09-18T08:05:24.966 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:06.384692+0000", 2024-09-18T08:05:24.966 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 2.6256999999999999e-05 2024-09-18T08:05:24.966 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:05:24.966 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:05:24.967 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-09-18T08:05:24.967 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:06.384701+0000", 2024-09-18T08:05:24.967 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 9.2250000000000006e-06 2024-09-18T08:05:24.967 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:05:24.967 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:05:24.967 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-09-18T08:05:24.967 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:06.384709+0000", 2024-09-18T08:05:24.967 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 7.3529999999999997e-06 2024-09-18T08:05:24.967 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:05:24.967 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:05:24.967 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-18T08:05:24.967 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:06.384763+0000", 2024-09-18T08:05:24.967 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 5.4104000000000002e-05 2024-09-18T08:05:24.967 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:05:24.967 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:05:24.968 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-09-18T08:05:24.968 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:06.384893+0000", 2024-09-18T08:05:24.968 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00013003099999999999 2024-09-18T08:05:24.968 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:05:24.968 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:05:24.968 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "waiting for subops from 0,2", 2024-09-18T08:05:24.968 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:06.385047+0000", 2024-09-18T08:05:24.968 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00015392699999999999 2024-09-18T08:05:24.968 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:05:24.968 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:05:24.968 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "op_commit", 2024-09-18T08:05:24.968 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:06.385823+0000", 2024-09-18T08:05:24.968 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00077649800000000003 2024-09-18T08:05:24.969 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:05:24.969 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:05:24.969 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:05:24.969 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:06.386738+0000", 2024-09-18T08:05:24.969 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00091412399999999999 2024-09-18T08:05:24.969 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:05:24.969 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:05:24.969 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:05:24.969 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:06.420590+0000", 2024-09-18T08:05:24.969 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.033852776000000001 2024-09-18T08:05:24.969 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:05:24.969 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:05:24.969 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "commit_sent", 2024-09-18T08:05:24.970 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:06.420658+0000", 2024-09-18T08:05:24.970 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 6.7762000000000006e-05 2024-09-18T08:05:24.970 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:05:24.970 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:05:24.970 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-09-18T08:05:24.970 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:06.420691+0000", 2024-09-18T08:05:24.970 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 3.2477000000000001e-05 2024-09-18T08:05:24.970 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-18T08:05:24.970 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-09-18T08:05:24.970 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-18T08:05:24.970 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:05:24.970 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:05:24.970 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.14805.0:2080 7.5 7:a35ff972:::benchmark_data_smithi007_73155_object2079:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e88)", 2024-09-18T08:05:24.970 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-09-18T08:04:06.385258+0000", 2024-09-18T08:05:24.971 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 78.567669659000003, 2024-09-18T08:05:24.971 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.036596191, 2024-09-18T08:05:24.971 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-09-18T08:05:24.971 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-09-18T08:05:24.971 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T08:05:24.971 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2024-09-18T08:05:24.971 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.14805", 2024-09-18T08:05:24.971 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "172.21.15.7:0/640320436", 2024-09-18T08:05:24.971 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 2080 2024-09-18T08:05:24.971 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:05:24.971 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-09-18T08:05:24.971 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:05:24.971 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-09-18T08:05:24.972 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:06.385258+0000", 2024-09-18T08:05:24.972 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-18T08:05:24.972 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:05:24.972 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:05:24.972 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-09-18T08:05:24.972 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:06.385258+0000", 2024-09-18T08:05:24.972 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-18T08:05:24.972 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:05:24.972 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:05:24.972 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-09-18T08:05:24.972 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:06.385258+0000", 2024-09-18T08:05:24.972 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 9.2099999999999995e-07 2024-09-18T08:05:24.972 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:05:24.973 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:05:24.973 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-09-18T08:05:24.973 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:06.385312+0000", 2024-09-18T08:05:24.973 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 5.3919000000000003e-05 2024-09-18T08:05:24.973 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:05:24.973 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:05:24.973 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-09-18T08:05:24.973 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:06.385324+0000", 2024-09-18T08:05:24.973 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.1943000000000001e-05 2024-09-18T08:05:24.973 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:05:24.973 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:05:24.973 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-09-18T08:05:24.973 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:06.385332+0000", 2024-09-18T08:05:24.974 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 7.2910000000000002e-06 2024-09-18T08:05:24.974 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:05:24.974 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:05:24.974 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-18T08:05:24.974 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:06.385393+0000", 2024-09-18T08:05:24.974 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 6.1614999999999999e-05 2024-09-18T08:05:24.974 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:05:24.974 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:05:24.974 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-09-18T08:05:24.974 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:06.385533+0000", 2024-09-18T08:05:24.974 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00014014400000000001 2024-09-18T08:05:24.975 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:05:24.975 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:05:24.975 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "waiting for subops from 0,2", 2024-09-18T08:05:24.975 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:06.385636+0000", 2024-09-18T08:05:24.975 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00010314099999999999 2024-09-18T08:05:24.975 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:05:24.975 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:05:24.975 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "op_commit", 2024-09-18T08:05:24.975 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:06.385955+0000", 2024-09-18T08:05:24.975 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00031824300000000001 2024-09-18T08:05:24.975 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:05:24.975 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:05:24.975 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:05:24.976 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:06.388660+0000", 2024-09-18T08:05:24.976 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0027055149999999999 2024-09-18T08:05:24.976 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:05:24.976 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:05:24.976 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:05:24.976 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:06.421752+0000", 2024-09-18T08:05:24.976 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.033091975000000003 2024-09-18T08:05:24.976 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:05:24.976 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:05:24.976 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "commit_sent", 2024-09-18T08:05:24.976 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:06.421816+0000", 2024-09-18T08:05:24.976 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 6.334e-05 2024-09-18T08:05:24.976 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:05:24.977 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:05:24.977 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-09-18T08:05:24.977 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:06.421854+0000", 2024-09-18T08:05:24.977 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 3.8143999999999999e-05 2024-09-18T08:05:24.977 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-18T08:05:24.977 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-09-18T08:05:24.977 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-18T08:05:24.977 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:05:24.977 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:05:24.977 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.14805.0:2087 7.5 7:a4fcdbed:::benchmark_data_smithi007_73155_object2086:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e88)", 2024-09-18T08:05:24.977 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-09-18T08:04:06.388110+0000", 2024-09-18T08:05:24.977 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 78.564817074000004, 2024-09-18T08:05:24.977 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.034665346999999999, 2024-09-18T08:05:24.977 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-09-18T08:05:24.978 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-09-18T08:05:24.978 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T08:05:24.978 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2024-09-18T08:05:24.978 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.14805", 2024-09-18T08:05:24.978 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "172.21.15.7:0/640320436", 2024-09-18T08:05:24.978 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 2087 2024-09-18T08:05:24.978 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:05:24.978 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-09-18T08:05:24.978 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:05:24.978 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-09-18T08:05:24.978 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:06.388110+0000", 2024-09-18T08:05:24.978 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-18T08:05:24.978 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:05:24.978 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:05:24.979 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-09-18T08:05:24.979 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:06.388110+0000", 2024-09-18T08:05:24.979 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-18T08:05:24.979 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:05:24.979 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:05:24.979 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-09-18T08:05:24.979 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:06.388111+0000", 2024-09-18T08:05:24.979 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 7.8899999999999998e-07 2024-09-18T08:05:24.979 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:05:24.979 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:05:24.979 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-09-18T08:05:24.979 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:06.388170+0000", 2024-09-18T08:05:24.979 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 5.9299999999999998e-05 2024-09-18T08:05:24.979 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:05:24.980 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:05:24.980 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-09-18T08:05:24.980 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:06.388180+0000", 2024-09-18T08:05:24.980 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.0210000000000001e-05 2024-09-18T08:05:24.980 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:05:24.980 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:05:24.980 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-09-18T08:05:24.980 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:06.388186+0000", 2024-09-18T08:05:24.980 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 5.3879999999999999e-06 2024-09-18T08:05:24.980 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:05:24.980 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:05:24.980 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-18T08:05:24.980 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:06.388234+0000", 2024-09-18T08:05:24.981 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4.8078999999999998e-05 2024-09-18T08:05:24.981 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:05:24.981 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:05:24.981 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-09-18T08:05:24.981 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:06.388356+0000", 2024-09-18T08:05:24.981 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00012185599999999999 2024-09-18T08:05:24.981 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:05:24.981 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:05:24.981 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "waiting for subops from 0,2", 2024-09-18T08:05:24.981 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:06.388445+0000", 2024-09-18T08:05:24.981 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 8.9307000000000004e-05 2024-09-18T08:05:24.981 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:05:24.981 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:05:24.982 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "op_commit", 2024-09-18T08:05:24.982 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:06.389240+0000", 2024-09-18T08:05:24.982 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00079541799999999999 2024-09-18T08:05:24.982 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:05:24.982 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:05:24.982 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:05:24.982 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:06.389673+0000", 2024-09-18T08:05:24.982 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00043252399999999999 2024-09-18T08:05:24.982 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:05:24.982 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:05:24.982 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:05:24.982 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:06.422611+0000", 2024-09-18T08:05:24.982 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.032937717999999998 2024-09-18T08:05:24.983 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:05:24.983 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:05:24.983 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "commit_sent", 2024-09-18T08:05:24.983 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:06.422687+0000", 2024-09-18T08:05:24.983 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 7.6046999999999999e-05 2024-09-18T08:05:24.983 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:05:24.983 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:05:24.983 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-09-18T08:05:24.983 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:06.422775+0000", 2024-09-18T08:05:24.983 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 8.8710999999999998e-05 2024-09-18T08:05:24.983 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-18T08:05:24.983 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-09-18T08:05:24.983 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-18T08:05:24.983 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:05:24.984 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:05:24.984 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.14805.0:2088 7.5 7:a738a874:::benchmark_data_smithi007_73155_object2087:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e88)", 2024-09-18T08:05:24.984 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-09-18T08:04:06.388524+0000", 2024-09-18T08:05:24.984 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 78.564402838000007, 2024-09-18T08:05:24.984 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.034404785, 2024-09-18T08:05:24.984 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-09-18T08:05:24.984 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-09-18T08:05:24.984 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T08:05:24.984 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2024-09-18T08:05:24.984 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.14805", 2024-09-18T08:05:24.984 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "172.21.15.7:0/640320436", 2024-09-18T08:05:24.984 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 2088 2024-09-18T08:05:24.984 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:05:24.984 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-09-18T08:05:24.985 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:05:24.985 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-09-18T08:05:24.985 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:06.388524+0000", 2024-09-18T08:05:24.985 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-18T08:05:24.985 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:05:24.985 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:05:24.985 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-09-18T08:05:24.985 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:06.388524+0000", 2024-09-18T08:05:24.986 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-18T08:05:24.986 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:05:24.986 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:05:24.986 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-09-18T08:05:24.986 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:06.388525+0000", 2024-09-18T08:05:24.986 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 5.9800000000000003e-07 2024-09-18T08:05:24.986 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:05:24.987 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:05:24.987 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-09-18T08:05:24.987 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:06.388632+0000", 2024-09-18T08:05:24.987 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00010676200000000001 2024-09-18T08:05:24.987 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:05:24.987 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:05:24.987 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-09-18T08:05:24.987 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:06.388639+0000", 2024-09-18T08:05:24.987 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 7.3989999999999998e-06 2024-09-18T08:05:24.987 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:05:24.987 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:05:24.987 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-09-18T08:05:24.987 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:06.388643+0000", 2024-09-18T08:05:24.987 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 3.7629999999999998e-06 2024-09-18T08:05:24.988 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:05:24.988 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:05:24.988 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-18T08:05:24.988 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:06.388705+0000", 2024-09-18T08:05:24.988 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 6.2156999999999995e-05 2024-09-18T08:05:24.988 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:05:24.988 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:05:24.988 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-09-18T08:05:24.988 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:06.388817+0000", 2024-09-18T08:05:24.988 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000112348 2024-09-18T08:05:24.988 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:05:24.988 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:05:24.989 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "waiting for subops from 0,2", 2024-09-18T08:05:24.989 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:06.388968+0000", 2024-09-18T08:05:24.989 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00015044 2024-09-18T08:05:24.989 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:05:24.989 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:05:24.989 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "op_commit", 2024-09-18T08:05:24.989 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:06.389389+0000", 2024-09-18T08:05:24.989 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00042139599999999999 2024-09-18T08:05:24.989 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:05:24.989 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:05:24.989 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:05:24.989 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:06.391166+0000", 2024-09-18T08:05:24.989 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0017768770000000001 2024-09-18T08:05:24.990 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:05:24.990 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:05:24.990 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:05:24.990 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:06.422823+0000", 2024-09-18T08:05:24.990 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.031656922999999997 2024-09-18T08:05:24.990 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:05:24.990 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:05:24.990 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "commit_sent", 2024-09-18T08:05:24.990 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:06.422875+0000", 2024-09-18T08:05:24.991 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 5.2456000000000001e-05 2024-09-18T08:05:24.991 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:05:24.991 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:05:24.991 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-09-18T08:05:24.991 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:06.422929+0000", 2024-09-18T08:05:24.991 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 5.3665999999999997e-05 2024-09-18T08:05:24.991 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-18T08:05:24.991 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-09-18T08:05:24.991 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-18T08:05:24.991 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:05:24.991 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:05:24.991 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.14805.0:65035 7.5 7:a41a2ff4:::benchmark_data_smithi007_73155_object65034:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e100)", 2024-09-18T08:05:24.991 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-09-18T08:04:42.217482+0000", 2024-09-18T08:05:24.992 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 42.735444729999998, 2024-09-18T08:05:24.992 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.081247141999999994, 2024-09-18T08:05:24.992 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-09-18T08:05:24.992 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-09-18T08:05:24.992 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T08:05:24.992 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2024-09-18T08:05:24.992 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.14805", 2024-09-18T08:05:24.992 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "172.21.15.7:0/640320436", 2024-09-18T08:05:24.992 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 65035 2024-09-18T08:05:24.992 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:05:24.992 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-09-18T08:05:24.992 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:05:24.992 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-09-18T08:05:24.993 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:42.217482+0000", 2024-09-18T08:05:24.993 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-18T08:05:24.993 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:05:24.993 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:05:24.993 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-09-18T08:05:24.993 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:42.217482+0000", 2024-09-18T08:05:24.993 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-18T08:05:24.993 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:05:24.993 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:05:24.993 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-09-18T08:05:24.993 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:42.217483+0000", 2024-09-18T08:05:24.993 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4.2800000000000002e-07 2024-09-18T08:05:24.993 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:05:24.994 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:05:24.994 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-09-18T08:05:24.994 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:42.217510+0000", 2024-09-18T08:05:24.994 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 2.7073e-05 2024-09-18T08:05:24.994 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:05:24.994 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:05:24.994 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-09-18T08:05:24.994 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:42.217514+0000", 2024-09-18T08:05:24.994 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4.279e-06 2024-09-18T08:05:24.994 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:05:24.994 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:05:24.994 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-09-18T08:05:24.994 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:42.217518+0000", 2024-09-18T08:05:24.994 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 3.7670000000000001e-06 2024-09-18T08:05:24.995 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:05:24.995 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:05:24.995 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-18T08:05:24.995 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:42.219467+0000", 2024-09-18T08:05:24.995 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0019486029999999999 2024-09-18T08:05:24.995 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:05:24.995 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:05:24.995 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-09-18T08:05:24.995 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:42.219552+0000", 2024-09-18T08:05:24.995 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 8.5148000000000005e-05 2024-09-18T08:05:24.995 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:05:24.995 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:05:24.996 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "waiting for subops from 0,2", 2024-09-18T08:05:24.996 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:42.219699+0000", 2024-09-18T08:05:24.996 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00014678900000000001 2024-09-18T08:05:24.996 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:05:24.996 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:05:24.996 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "op_commit", 2024-09-18T08:05:24.996 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:42.223199+0000", 2024-09-18T08:05:24.996 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0035005140000000001 2024-09-18T08:05:24.996 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:05:24.996 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:05:24.996 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:05:24.996 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:42.227302+0000", 2024-09-18T08:05:24.996 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0041028749999999998 2024-09-18T08:05:24.996 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:05:24.997 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:05:24.997 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:05:24.997 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:42.298617+0000", 2024-09-18T08:05:24.997 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.071314849999999999 2024-09-18T08:05:24.997 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:05:24.997 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:05:24.997 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "commit_sent", 2024-09-18T08:05:24.997 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:42.298685+0000", 2024-09-18T08:05:24.997 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 6.8138000000000004e-05 2024-09-18T08:05:24.997 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:05:24.997 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:05:24.997 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-09-18T08:05:24.997 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:42.298730+0000", 2024-09-18T08:05:24.997 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4.4678000000000002e-05 2024-09-18T08:05:24.998 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-18T08:05:24.998 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-09-18T08:05:24.998 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-18T08:05:24.998 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:05:24.998 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:05:24.998 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.14805.0:65042 7.5 7:a8688cbb:::benchmark_data_smithi007_73155_object65041:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e100)", 2024-09-18T08:05:24.998 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-09-18T08:04:42.221240+0000", 2024-09-18T08:05:24.998 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 42.731686889999999, 2024-09-18T08:05:24.998 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.077641613999999998, 2024-09-18T08:05:24.998 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-09-18T08:05:24.998 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-09-18T08:05:24.998 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T08:05:24.998 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2024-09-18T08:05:24.998 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.14805", 2024-09-18T08:05:24.999 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "172.21.15.7:0/640320436", 2024-09-18T08:05:24.999 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 65042 2024-09-18T08:05:24.999 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:05:24.999 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-09-18T08:05:24.999 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:05:24.999 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-09-18T08:05:24.999 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:42.221240+0000", 2024-09-18T08:05:24.999 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-18T08:05:24.999 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:05:24.999 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:05:24.999 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-09-18T08:05:24.999 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:42.221240+0000", 2024-09-18T08:05:24.999 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-18T08:05:24.999 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:05:24.999 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:05:25.000 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-09-18T08:05:25.000 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:42.221242+0000", 2024-09-18T08:05:25.000 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.291e-06 2024-09-18T08:05:25.000 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:05:25.000 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:05:25.000 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-09-18T08:05:25.000 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:42.221291+0000", 2024-09-18T08:05:25.000 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4.9774000000000002e-05 2024-09-18T08:05:25.000 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:05:25.000 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:05:25.000 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-09-18T08:05:25.000 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:42.221301+0000", 2024-09-18T08:05:25.001 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 9.4730000000000006e-06 2024-09-18T08:05:25.001 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:05:25.001 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:05:25.001 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-09-18T08:05:25.001 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:42.221308+0000", 2024-09-18T08:05:25.001 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 6.9480000000000002e-06 2024-09-18T08:05:25.001 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:05:25.001 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:05:25.001 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-18T08:05:25.001 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:42.221362+0000", 2024-09-18T08:05:25.001 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 5.4608999999999998e-05 2024-09-18T08:05:25.001 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:05:25.001 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:05:25.002 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-09-18T08:05:25.002 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:42.221521+0000", 2024-09-18T08:05:25.002 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00015870000000000001 2024-09-18T08:05:25.002 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:05:25.002 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:05:25.002 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "waiting for subops from 0,2", 2024-09-18T08:05:25.002 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:42.221680+0000", 2024-09-18T08:05:25.002 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00015865200000000001 2024-09-18T08:05:25.002 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:05:25.002 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:05:25.002 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "op_commit", 2024-09-18T08:05:25.002 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:42.223368+0000", 2024-09-18T08:05:25.002 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00168793 2024-09-18T08:05:25.003 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:05:25.003 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:05:25.003 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:05:25.003 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:42.227333+0000", 2024-09-18T08:05:25.003 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0039657939999999999 2024-09-18T08:05:25.003 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:05:25.003 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:05:25.003 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:05:25.003 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:42.298788+0000", 2024-09-18T08:05:25.003 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.071454675999999995 2024-09-18T08:05:25.003 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:05:25.003 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:05:25.003 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "commit_sent", 2024-09-18T08:05:25.003 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:42.298843+0000", 2024-09-18T08:05:25.003 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 5.4537999999999999e-05 2024-09-18T08:05:25.004 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:05:25.004 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:05:25.004 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-09-18T08:05:25.004 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:42.298882+0000", 2024-09-18T08:05:25.004 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 3.9229e-05 2024-09-18T08:05:25.004 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-18T08:05:25.004 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-09-18T08:05:25.004 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-18T08:05:25.004 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:05:25.004 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:05:25.004 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.14805.0:65046 7.5 7:a27c5abc:::benchmark_data_smithi007_73155_object65045:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e100)", 2024-09-18T08:05:25.004 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-09-18T08:04:42.223529+0000", 2024-09-18T08:05:25.004 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 42.729398324000002, 2024-09-18T08:05:25.004 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.075501960000000007, 2024-09-18T08:05:25.005 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-09-18T08:05:25.005 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-09-18T08:05:25.005 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T08:05:25.005 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2024-09-18T08:05:25.005 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.14805", 2024-09-18T08:05:25.005 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "172.21.15.7:0/640320436", 2024-09-18T08:05:25.005 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 65046 2024-09-18T08:05:25.005 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:05:25.005 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-09-18T08:05:25.005 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:05:25.005 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-09-18T08:05:25.005 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:42.223529+0000", 2024-09-18T08:05:25.005 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-18T08:05:25.006 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:05:25.006 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:05:25.006 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-09-18T08:05:25.006 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:42.223529+0000", 2024-09-18T08:05:25.006 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-18T08:05:25.006 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:05:25.006 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:05:25.006 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-09-18T08:05:25.006 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:42.223530+0000", 2024-09-18T08:05:25.006 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.1310000000000001e-06 2024-09-18T08:05:25.006 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:05:25.006 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:05:25.007 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-09-18T08:05:25.007 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:42.223581+0000", 2024-09-18T08:05:25.007 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 5.0769000000000001e-05 2024-09-18T08:05:25.007 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:05:25.007 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:05:25.007 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-09-18T08:05:25.007 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:42.223593+0000", 2024-09-18T08:05:25.007 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.2354000000000001e-05 2024-09-18T08:05:25.007 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:05:25.007 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:05:25.007 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-09-18T08:05:25.007 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:42.223601+0000", 2024-09-18T08:05:25.007 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 7.5560000000000002e-06 2024-09-18T08:05:25.007 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:05:25.008 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:05:25.008 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-18T08:05:25.008 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:42.223757+0000", 2024-09-18T08:05:25.008 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000156451 2024-09-18T08:05:25.008 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:05:25.008 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:05:25.008 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-09-18T08:05:25.008 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:42.223863+0000", 2024-09-18T08:05:25.008 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000105923 2024-09-18T08:05:25.008 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:05:25.008 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:05:25.008 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "waiting for subops from 0,2", 2024-09-18T08:05:25.008 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:42.223992+0000", 2024-09-18T08:05:25.009 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000129357 2024-09-18T08:05:25.009 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:05:25.009 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:05:25.009 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "op_commit", 2024-09-18T08:05:25.009 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:42.225908+0000", 2024-09-18T08:05:25.009 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0019158510000000001 2024-09-18T08:05:25.009 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:05:25.009 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:05:25.009 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:05:25.009 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:42.233204+0000", 2024-09-18T08:05:25.009 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.007296131 2024-09-18T08:05:25.009 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:05:25.009 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:05:25.009 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:05:25.010 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:42.298932+0000", 2024-09-18T08:05:25.010 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.065727455000000004 2024-09-18T08:05:25.010 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:05:25.010 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:05:25.010 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "commit_sent", 2024-09-18T08:05:25.010 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:42.298989+0000", 2024-09-18T08:05:25.010 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 5.6771000000000002e-05 2024-09-18T08:05:25.010 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:05:25.010 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:05:25.010 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-09-18T08:05:25.010 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:42.299031+0000", 2024-09-18T08:05:25.010 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4.2211e-05 2024-09-18T08:05:25.010 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-18T08:05:25.010 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-09-18T08:05:25.011 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-18T08:05:25.011 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:05:25.011 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:05:25.011 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.14805.0:65059 7.5 7:ac3787be:::benchmark_data_smithi007_73155_object65058:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e100)", 2024-09-18T08:05:25.011 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-09-18T08:04:42.235840+0000", 2024-09-18T08:05:25.011 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 42.717086977000001, 2024-09-18T08:05:25.011 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.074652292999999995, 2024-09-18T08:05:25.011 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-09-18T08:05:25.011 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-09-18T08:05:25.011 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T08:05:25.011 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2024-09-18T08:05:25.011 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.14805", 2024-09-18T08:05:25.011 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "172.21.15.7:0/640320436", 2024-09-18T08:05:25.011 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 65059 2024-09-18T08:05:25.011 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:05:25.012 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-09-18T08:05:25.012 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:05:25.012 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-09-18T08:05:25.012 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:42.235840+0000", 2024-09-18T08:05:25.012 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-18T08:05:25.012 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:05:25.012 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:05:25.012 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-09-18T08:05:25.012 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:42.235840+0000", 2024-09-18T08:05:25.012 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-18T08:05:25.012 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:05:25.012 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:05:25.012 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-09-18T08:05:25.012 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:42.235842+0000", 2024-09-18T08:05:25.012 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.4219999999999999e-06 2024-09-18T08:05:25.013 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:05:25.013 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:05:25.013 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-09-18T08:05:25.013 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:42.235894+0000", 2024-09-18T08:05:25.013 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 5.2539e-05 2024-09-18T08:05:25.013 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:05:25.013 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:05:25.013 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-09-18T08:05:25.013 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:42.235912+0000", 2024-09-18T08:05:25.013 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.8119000000000001e-05 2024-09-18T08:05:25.013 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:05:25.013 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:05:25.013 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-09-18T08:05:25.014 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:42.235918+0000", 2024-09-18T08:05:25.014 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 5.7509999999999997e-06 2024-09-18T08:05:25.014 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:05:25.014 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:05:25.014 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-18T08:05:25.014 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:42.235958+0000", 2024-09-18T08:05:25.014 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4.0210000000000003e-05 2024-09-18T08:05:25.014 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:05:25.014 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:05:25.014 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-09-18T08:05:25.014 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:42.236051+0000", 2024-09-18T08:05:25.014 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 9.2788000000000007e-05 2024-09-18T08:05:25.014 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:05:25.014 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:05:25.014 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "waiting for subops from 0,2", 2024-09-18T08:05:25.015 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:42.236141+0000", 2024-09-18T08:05:25.015 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 9.0458999999999996e-05 2024-09-18T08:05:25.015 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:05:25.015 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:05:25.015 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "op_commit", 2024-09-18T08:05:25.015 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:42.237853+0000", 2024-09-18T08:05:25.015 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.001711588 2024-09-18T08:05:25.015 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:05:25.015 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:05:25.015 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:05:25.015 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:42.238325+0000", 2024-09-18T08:05:25.015 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000471905 2024-09-18T08:05:25.015 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:05:25.015 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:05:25.015 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:05:25.016 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:42.310431+0000", 2024-09-18T08:05:25.016 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.072105662000000001 2024-09-18T08:05:25.016 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:05:25.016 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:05:25.016 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "commit_sent", 2024-09-18T08:05:25.016 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:42.310465+0000", 2024-09-18T08:05:25.016 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 3.4010000000000001e-05 2024-09-18T08:05:25.016 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:05:25.016 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:05:25.016 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-09-18T08:05:25.016 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:42.310492+0000", 2024-09-18T08:05:25.016 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 2.7840000000000001e-05 2024-09-18T08:05:25.016 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-18T08:05:25.016 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-09-18T08:05:25.017 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-18T08:05:25.017 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:05:25.017 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:05:25.017 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.14805.0:66018 7.5 7:af3f4dab:::benchmark_data_smithi007_73155_object66017:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e102)", 2024-09-18T08:05:25.017 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-09-18T08:04:43.951058+0000", 2024-09-18T08:05:25.017 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 41.001869651, 2024-09-18T08:05:25.017 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.090281823999999997, 2024-09-18T08:05:25.017 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-09-18T08:05:25.017 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-09-18T08:05:25.017 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T08:05:25.017 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2024-09-18T08:05:25.017 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.14805", 2024-09-18T08:05:25.017 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "172.21.15.7:0/640320436", 2024-09-18T08:05:25.017 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 66018 2024-09-18T08:05:25.018 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:05:25.018 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-09-18T08:05:25.018 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:05:25.018 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-09-18T08:05:25.018 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:43.951058+0000", 2024-09-18T08:05:25.018 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-18T08:05:25.018 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:05:25.018 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:05:25.018 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-09-18T08:05:25.018 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:43.951058+0000", 2024-09-18T08:05:25.018 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-18T08:05:25.018 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:05:25.018 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:05:25.018 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-09-18T08:05:25.018 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:43.951059+0000", 2024-09-18T08:05:25.018 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.13e-06 2024-09-18T08:05:25.019 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:05:25.019 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:05:25.019 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-09-18T08:05:25.019 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:43.951136+0000", 2024-09-18T08:05:25.019 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 7.7760000000000001e-05 2024-09-18T08:05:25.019 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:05:25.019 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:05:25.019 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-09-18T08:05:25.019 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:43.951147+0000", 2024-09-18T08:05:25.019 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.0807e-05 2024-09-18T08:05:25.019 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:05:25.019 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:05:25.019 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-09-18T08:05:25.019 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:43.951155+0000", 2024-09-18T08:05:25.019 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 7.644e-06 2024-09-18T08:05:25.020 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:05:25.020 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:05:25.020 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-18T08:05:25.020 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:43.952919+0000", 2024-09-18T08:05:25.020 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.001764109 2024-09-18T08:05:25.020 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:05:25.020 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:05:25.020 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-09-18T08:05:25.020 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:43.953039+0000", 2024-09-18T08:05:25.020 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000120265 2024-09-18T08:05:25.020 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:05:25.020 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:05:25.020 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "waiting for subops from 0,2", 2024-09-18T08:05:25.020 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:43.953189+0000", 2024-09-18T08:05:25.021 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000149386 2024-09-18T08:05:25.021 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:05:25.021 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:05:25.021 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "op_commit", 2024-09-18T08:05:25.021 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:43.954700+0000", 2024-09-18T08:05:25.021 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.001511507 2024-09-18T08:05:25.021 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:05:25.021 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:05:25.021 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:05:25.021 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:43.959976+0000", 2024-09-18T08:05:25.021 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0052754380000000003 2024-09-18T08:05:25.021 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:05:25.021 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:05:25.021 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:05:25.022 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:44.041213+0000", 2024-09-18T08:05:25.022 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.081237023000000005 2024-09-18T08:05:25.022 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:05:25.022 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:05:25.022 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "commit_sent", 2024-09-18T08:05:25.022 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:44.041277+0000", 2024-09-18T08:05:25.022 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 6.4898000000000001e-05 2024-09-18T08:05:25.022 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:05:25.022 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:05:25.022 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-09-18T08:05:25.022 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:44.041339+0000", 2024-09-18T08:05:25.022 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 6.1857000000000001e-05 2024-09-18T08:05:25.022 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-18T08:05:25.022 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-09-18T08:05:25.023 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-18T08:05:25.023 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:05:25.023 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:05:25.023 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.14805.0:66022 7.5 7:a793a645:::benchmark_data_smithi007_73155_object66021:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e102)", 2024-09-18T08:05:25.023 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-09-18T08:04:43.956399+0000", 2024-09-18T08:05:25.023 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 40.996528066000003, 2024-09-18T08:05:25.023 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.085062251000000005, 2024-09-18T08:05:25.023 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-09-18T08:05:25.023 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-09-18T08:05:25.023 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T08:05:25.023 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2024-09-18T08:05:25.023 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.14805", 2024-09-18T08:05:25.023 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "172.21.15.7:0/640320436", 2024-09-18T08:05:25.023 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 66022 2024-09-18T08:05:25.024 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:05:25.024 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-09-18T08:05:25.024 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:05:25.024 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-09-18T08:05:25.024 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:43.956399+0000", 2024-09-18T08:05:25.024 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-18T08:05:25.024 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:05:25.024 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:05:25.024 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-09-18T08:05:25.024 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:43.956399+0000", 2024-09-18T08:05:25.024 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-18T08:05:25.024 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:05:25.024 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:05:25.024 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-09-18T08:05:25.024 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:43.956401+0000", 2024-09-18T08:05:25.025 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 2.0660000000000002e-06 2024-09-18T08:05:25.025 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:05:25.025 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:05:25.025 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-09-18T08:05:25.025 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:43.956434+0000", 2024-09-18T08:05:25.025 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 3.2913999999999997e-05 2024-09-18T08:05:25.025 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:05:25.025 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:05:25.025 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-09-18T08:05:25.025 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:43.956450+0000", 2024-09-18T08:05:25.025 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.5688999999999998e-05 2024-09-18T08:05:25.025 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:05:25.025 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:05:25.025 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-09-18T08:05:25.026 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:43.956458+0000", 2024-09-18T08:05:25.026 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 8.1810000000000001e-06 2024-09-18T08:05:25.026 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:05:25.026 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:05:25.026 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-18T08:05:25.026 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:43.956515+0000", 2024-09-18T08:05:25.026 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 5.6974e-05 2024-09-18T08:05:25.026 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:05:25.026 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:05:25.026 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-09-18T08:05:25.026 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:43.956655+0000", 2024-09-18T08:05:25.026 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00014009100000000001 2024-09-18T08:05:25.026 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:05:25.026 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:05:25.026 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "waiting for subops from 0,2", 2024-09-18T08:05:25.027 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:43.956819+0000", 2024-09-18T08:05:25.027 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00016355499999999999 2024-09-18T08:05:25.027 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:05:25.027 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:05:25.027 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:05:25.027 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:43.960008+0000", 2024-09-18T08:05:25.027 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.003189618 2024-09-18T08:05:25.027 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:05:25.027 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:05:25.027 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "op_commit", 2024-09-18T08:05:25.028 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:43.960387+0000", 2024-09-18T08:05:25.028 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00037857500000000003 2024-09-18T08:05:25.028 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:05:25.028 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:05:25.028 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:05:25.028 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:44.041387+0000", 2024-09-18T08:05:25.028 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.081000366000000004 2024-09-18T08:05:25.028 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:05:25.028 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:05:25.028 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "commit_sent", 2024-09-18T08:05:25.028 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:44.041422+0000", 2024-09-18T08:05:25.029 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 3.4962999999999997e-05 2024-09-18T08:05:25.029 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:05:25.029 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:05:25.029 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-09-18T08:05:25.029 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:44.041461+0000", 2024-09-18T08:05:25.029 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 3.9258999999999998e-05 2024-09-18T08:05:25.029 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-18T08:05:25.029 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-09-18T08:05:25.029 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-18T08:05:25.029 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:05:25.029 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:05:25.029 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.14805.0:66032 7.5 7:a3c82aed:::benchmark_data_smithi007_73155_object66031:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e102)", 2024-09-18T08:05:25.029 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-09-18T08:04:43.962345+0000", 2024-09-18T08:05:25.029 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 40.990581738000003, 2024-09-18T08:05:25.030 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.079265130000000003, 2024-09-18T08:05:25.030 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-09-18T08:05:25.030 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-09-18T08:05:25.030 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T08:05:25.030 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2024-09-18T08:05:25.030 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.14805", 2024-09-18T08:05:25.030 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "172.21.15.7:0/640320436", 2024-09-18T08:05:25.030 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 66032 2024-09-18T08:05:25.030 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:05:25.030 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-09-18T08:05:25.030 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:05:25.030 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-09-18T08:05:25.030 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:43.962345+0000", 2024-09-18T08:05:25.030 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-18T08:05:25.031 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:05:25.031 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:05:25.031 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-09-18T08:05:25.031 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:43.962345+0000", 2024-09-18T08:05:25.031 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-18T08:05:25.031 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:05:25.031 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:05:25.031 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-09-18T08:05:25.031 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:43.962347+0000", 2024-09-18T08:05:25.031 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.248e-06 2024-09-18T08:05:25.031 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:05:25.031 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:05:25.031 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-09-18T08:05:25.031 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:43.962396+0000", 2024-09-18T08:05:25.032 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4.8967000000000001e-05 2024-09-18T08:05:25.032 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:05:25.032 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:05:25.032 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-09-18T08:05:25.032 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:43.962410+0000", 2024-09-18T08:05:25.032 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.4100000000000001e-05 2024-09-18T08:05:25.032 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:05:25.032 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:05:25.032 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-09-18T08:05:25.032 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:43.962417+0000", 2024-09-18T08:05:25.032 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 7.729e-06 2024-09-18T08:05:25.032 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:05:25.033 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:05:25.033 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-18T08:05:25.033 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:43.962471+0000", 2024-09-18T08:05:25.033 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 5.3706999999999999e-05 2024-09-18T08:05:25.033 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:05:25.033 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:05:25.033 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-09-18T08:05:25.033 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:43.962572+0000", 2024-09-18T08:05:25.033 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00010068699999999999 2024-09-18T08:05:25.033 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:05:25.033 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:05:25.033 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "waiting for subops from 0,2", 2024-09-18T08:05:25.033 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:43.962667+0000", 2024-09-18T08:05:25.033 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 9.5328000000000006e-05 2024-09-18T08:05:25.034 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:05:25.034 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:05:25.034 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "op_commit", 2024-09-18T08:05:25.034 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:43.964331+0000", 2024-09-18T08:05:25.034 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0016642919999999999 2024-09-18T08:05:25.034 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:05:25.034 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:05:25.034 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:05:25.034 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:43.968740+0000", 2024-09-18T08:05:25.034 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0044080660000000004 2024-09-18T08:05:25.034 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:05:25.034 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:05:25.034 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:05:25.035 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:44.041517+0000", 2024-09-18T08:05:25.035 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.072777122 2024-09-18T08:05:25.035 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:05:25.035 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:05:25.035 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "commit_sent", 2024-09-18T08:05:25.035 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:44.041571+0000", 2024-09-18T08:05:25.035 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 5.4797999999999999e-05 2024-09-18T08:05:25.035 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:05:25.035 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:05:25.035 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-09-18T08:05:25.035 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:44.041611+0000", 2024-09-18T08:05:25.035 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 3.9085999999999998e-05 2024-09-18T08:05:25.035 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-18T08:05:25.035 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-09-18T08:05:25.036 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-18T08:05:25.036 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:05:25.036 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:05:25.036 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.14805.0:66046 7.5 7:adda08cf:::benchmark_data_smithi007_73155_object66045:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e102)", 2024-09-18T08:05:25.036 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-09-18T08:04:43.973767+0000", 2024-09-18T08:05:25.036 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 40.979160276000002, 2024-09-18T08:05:25.036 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.078123282000000002, 2024-09-18T08:05:25.036 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-09-18T08:05:25.036 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-09-18T08:05:25.036 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T08:05:25.036 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2024-09-18T08:05:25.036 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.14805", 2024-09-18T08:05:25.036 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "172.21.15.7:0/640320436", 2024-09-18T08:05:25.037 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 66046 2024-09-18T08:05:25.037 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:05:25.037 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-09-18T08:05:25.037 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:05:25.037 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-09-18T08:05:25.037 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:43.973767+0000", 2024-09-18T08:05:25.037 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-18T08:05:25.037 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:05:25.037 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:05:25.037 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-09-18T08:05:25.037 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:43.973767+0000", 2024-09-18T08:05:25.037 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-18T08:05:25.037 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:05:25.037 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:05:25.038 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-09-18T08:05:25.038 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:43.973769+0000", 2024-09-18T08:05:25.038 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.674e-06 2024-09-18T08:05:25.038 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:05:25.038 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:05:25.038 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-09-18T08:05:25.038 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:43.973824+0000", 2024-09-18T08:05:25.038 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 5.5831999999999997e-05 2024-09-18T08:05:25.038 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:05:25.038 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:05:25.038 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-09-18T08:05:25.038 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:43.973838+0000", 2024-09-18T08:05:25.038 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.3951000000000001e-05 2024-09-18T08:05:25.038 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:05:25.038 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:05:25.038 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-09-18T08:05:25.039 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:43.973846+0000", 2024-09-18T08:05:25.039 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 7.8469999999999996e-06 2024-09-18T08:05:25.039 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:05:25.039 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:05:25.039 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-18T08:05:25.039 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:43.973906+0000", 2024-09-18T08:05:25.039 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 6.0192000000000002e-05 2024-09-18T08:05:25.039 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:05:25.039 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:05:25.039 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-09-18T08:05:25.039 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:43.974010+0000", 2024-09-18T08:05:25.039 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000103706 2024-09-18T08:05:25.039 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:05:25.039 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:05:25.039 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "waiting for subops from 0,2", 2024-09-18T08:05:25.040 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:43.974110+0000", 2024-09-18T08:05:25.040 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 9.9438000000000006e-05 2024-09-18T08:05:25.040 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:05:25.040 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:05:25.040 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "op_commit", 2024-09-18T08:05:25.040 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:43.975845+0000", 2024-09-18T08:05:25.040 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.001735143 2024-09-18T08:05:25.040 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:05:25.040 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:05:25.040 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:05:25.040 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:43.976592+0000", 2024-09-18T08:05:25.040 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00074738199999999999 2024-09-18T08:05:25.040 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:05:25.041 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:05:25.041 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:05:25.041 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:44.051807+0000", 2024-09-18T08:05:25.041 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.075215248999999998 2024-09-18T08:05:25.041 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:05:25.041 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:05:25.041 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "commit_sent", 2024-09-18T08:05:25.041 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:44.051844+0000", 2024-09-18T08:05:25.041 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 3.6248000000000003e-05 2024-09-18T08:05:25.041 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:05:25.041 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6 2024-09-18T08:05:25.043 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:05:25.043 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-09-18T08:05:25.043 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:44.051890+0000", 2024-09-18T08:05:25.043 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4.6619999999999997e-05 2024-09-18T08:05:25.043 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-18T08:05:25.043 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-09-18T08:05:25.043 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-18T08:05:25.043 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:05:25.043 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:05:25.043 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.14805.0:82965 7.5 7:a4b26842:::benchmark_data_smithi007_73155_object82964:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e106)", 2024-09-18T08:05:25.043 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-09-18T08:04:59.005184+0000", 2024-09-18T08:05:25.043 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 25.947743329000001, 2024-09-18T08:05:25.044 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.034987136000000002, 2024-09-18T08:05:25.044 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-09-18T08:05:25.044 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-09-18T08:05:25.044 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T08:05:25.044 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2024-09-18T08:05:25.044 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.14805", 2024-09-18T08:05:25.044 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "172.21.15.7:0/640320436", 2024-09-18T08:05:25.044 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 82965 2024-09-18T08:05:25.044 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:05:25.044 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-09-18T08:05:25.044 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:05:25.044 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-09-18T08:05:25.044 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:59.005184+0000", 2024-09-18T08:05:25.045 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-18T08:05:25.045 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:05:25.045 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:05:25.045 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-09-18T08:05:25.045 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:59.005184+0000", 2024-09-18T08:05:25.045 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-18T08:05:25.045 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:05:25.045 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:05:25.045 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-09-18T08:05:25.045 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:59.005186+0000", 2024-09-18T08:05:25.045 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.7260000000000001e-06 2024-09-18T08:05:25.045 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:05:25.045 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:05:25.045 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-09-18T08:05:25.046 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:59.005242+0000", 2024-09-18T08:05:25.046 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 5.6336999999999999e-05 2024-09-18T08:05:25.046 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:05:25.046 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:05:25.046 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-09-18T08:05:25.046 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:59.005254+0000", 2024-09-18T08:05:25.046 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.2058e-05 2024-09-18T08:05:25.046 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:05:25.046 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:05:25.046 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-09-18T08:05:25.046 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:59.005262+0000", 2024-09-18T08:05:25.046 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 8.4889999999999995e-06 2024-09-18T08:05:25.046 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:05:25.046 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:05:25.046 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-18T08:05:25.047 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:59.005334+0000", 2024-09-18T08:05:25.047 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 7.1328000000000006e-05 2024-09-18T08:05:25.047 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:05:25.047 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:05:25.047 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-09-18T08:05:25.047 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:59.005825+0000", 2024-09-18T08:05:25.047 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00049144900000000003 2024-09-18T08:05:25.047 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:05:25.047 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:05:25.047 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "waiting for subops from 0,2", 2024-09-18T08:05:25.047 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:59.005998+0000", 2024-09-18T08:05:25.047 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00017314500000000001 2024-09-18T08:05:25.047 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:05:25.048 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:05:25.048 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "op_commit", 2024-09-18T08:05:25.048 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:59.009089+0000", 2024-09-18T08:05:25.048 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0030910289999999999 2024-09-18T08:05:25.048 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:05:25.048 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:05:25.048 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:05:25.048 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:59.018736+0000", 2024-09-18T08:05:25.048 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0096462240000000001 2024-09-18T08:05:25.048 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:05:25.048 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:05:25.048 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:05:25.048 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:59.040096+0000", 2024-09-18T08:05:25.048 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.021360308000000001 2024-09-18T08:05:25.049 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:05:25.049 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:05:25.049 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "commit_sent", 2024-09-18T08:05:25.049 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:59.040132+0000", 2024-09-18T08:05:25.049 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 3.6554999999999998e-05 2024-09-18T08:05:25.049 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:05:25.049 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:05:25.049 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-09-18T08:05:25.049 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:59.040171+0000", 2024-09-18T08:05:25.049 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 3.8488000000000001e-05 2024-09-18T08:05:25.049 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-18T08:05:25.049 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-09-18T08:05:25.049 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-18T08:05:25.050 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:05:25.050 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:05:25.050 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.14805.0:83847 7.5 7:a4f0e703:::benchmark_data_smithi007_73155_object83846:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e106)", 2024-09-18T08:05:25.050 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-09-18T08:04:59.916475+0000", 2024-09-18T08:05:25.050 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 25.036452322999999, 2024-09-18T08:05:25.050 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.054575249999999999, 2024-09-18T08:05:25.050 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-09-18T08:05:25.050 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-09-18T08:05:25.050 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T08:05:25.050 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2024-09-18T08:05:25.050 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.14805", 2024-09-18T08:05:25.050 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "172.21.15.7:0/640320436", 2024-09-18T08:05:25.050 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 83847 2024-09-18T08:05:25.050 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:05:25.051 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-09-18T08:05:25.051 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:05:25.051 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-09-18T08:05:25.051 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:59.916475+0000", 2024-09-18T08:05:25.051 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-18T08:05:25.051 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:05:25.051 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:05:25.051 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-09-18T08:05:25.051 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:59.916475+0000", 2024-09-18T08:05:25.051 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-18T08:05:25.051 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:05:25.051 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:05:25.051 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-09-18T08:05:25.051 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:59.916476+0000", 2024-09-18T08:05:25.052 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.3e-06 2024-09-18T08:05:25.052 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:05:25.052 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:05:25.052 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-09-18T08:05:25.052 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:59.916531+0000", 2024-09-18T08:05:25.052 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 5.5324000000000001e-05 2024-09-18T08:05:25.052 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:05:25.052 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:05:25.052 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-09-18T08:05:25.052 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:59.916552+0000", 2024-09-18T08:05:25.052 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 2.0780000000000001e-05 2024-09-18T08:05:25.052 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:05:25.052 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:05:25.053 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-09-18T08:05:25.053 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:59.916559+0000", 2024-09-18T08:05:25.053 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 6.7100000000000001e-06 2024-09-18T08:05:25.053 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:05:25.053 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:05:25.053 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-18T08:05:25.053 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:59.916717+0000", 2024-09-18T08:05:25.053 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000157868 2024-09-18T08:05:25.053 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:05:25.053 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:05:25.053 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-09-18T08:05:25.053 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:59.917002+0000", 2024-09-18T08:05:25.053 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00028470299999999997 2024-09-18T08:05:25.053 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:05:25.054 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:05:25.054 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "waiting for subops from 0,2", 2024-09-18T08:05:25.054 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:59.917321+0000", 2024-09-18T08:05:25.054 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00031942899999999998 2024-09-18T08:05:25.054 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:05:25.054 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:05:25.054 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:05:25.054 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:59.920920+0000", 2024-09-18T08:05:25.054 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0035991590000000002 2024-09-18T08:05:25.054 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:05:25.054 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:05:25.054 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "op_commit", 2024-09-18T08:05:25.054 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:59.920949+0000", 2024-09-18T08:05:25.054 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 2.8747000000000001e-05 2024-09-18T08:05:25.055 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:05:25.055 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:05:25.055 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:05:25.055 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:59.970841+0000", 2024-09-18T08:05:25.055 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.049892261 2024-09-18T08:05:25.055 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:05:25.055 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:05:25.055 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "commit_sent", 2024-09-18T08:05:25.055 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:59.970998+0000", 2024-09-18T08:05:25.055 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00015695699999999999 2024-09-18T08:05:25.055 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:05:25.055 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:05:25.055 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-09-18T08:05:25.055 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:59.971050+0000", 2024-09-18T08:05:25.055 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 5.2012000000000003e-05 2024-09-18T08:05:25.056 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-18T08:05:25.056 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-09-18T08:05:25.056 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-18T08:05:25.056 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:05:25.056 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:05:25.056 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.14805.0:83854 7.5 7:a9ebab2a:::benchmark_data_smithi007_73155_object83853:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e106)", 2024-09-18T08:05:25.056 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-09-18T08:04:59.922842+0000", 2024-09-18T08:05:25.056 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 25.030084821999999, 2024-09-18T08:05:25.056 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.048325126000000003, 2024-09-18T08:05:25.056 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-09-18T08:05:25.056 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-09-18T08:05:25.056 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T08:05:25.056 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2024-09-18T08:05:25.056 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.14805", 2024-09-18T08:05:25.057 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "172.21.15.7:0/640320436", 2024-09-18T08:05:25.057 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 83854 2024-09-18T08:05:25.057 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:05:25.057 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-09-18T08:05:25.057 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:05:25.057 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-09-18T08:05:25.057 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:59.922842+0000", 2024-09-18T08:05:25.057 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-18T08:05:25.057 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:05:25.057 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:05:25.057 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-09-18T08:05:25.057 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:59.922842+0000", 2024-09-18T08:05:25.057 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-18T08:05:25.057 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:05:25.058 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:05:25.058 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-09-18T08:05:25.058 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:59.922862+0000", 2024-09-18T08:05:25.058 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.9962999999999998e-05 2024-09-18T08:05:25.058 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:05:25.058 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:05:25.058 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-09-18T08:05:25.058 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:59.922915+0000", 2024-09-18T08:05:25.058 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 5.3023999999999999e-05 2024-09-18T08:05:25.058 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:05:25.058 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:05:25.058 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-09-18T08:05:25.058 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:59.922929+0000", 2024-09-18T08:05:25.058 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.3932e-05 2024-09-18T08:05:25.058 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:05:25.059 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:05:25.059 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-09-18T08:05:25.059 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:59.922937+0000", 2024-09-18T08:05:25.059 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 8.1610000000000003e-06 2024-09-18T08:05:25.059 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:05:25.059 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:05:25.059 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-18T08:05:25.059 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:59.922991+0000", 2024-09-18T08:05:25.059 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 5.3517999999999998e-05 2024-09-18T08:05:25.059 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:05:25.059 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:05:25.059 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-09-18T08:05:25.059 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:59.923121+0000", 2024-09-18T08:05:25.059 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000130476 2024-09-18T08:05:25.059 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:05:25.060 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:05:25.060 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "waiting for subops from 0,2", 2024-09-18T08:05:25.060 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:59.923262+0000", 2024-09-18T08:05:25.060 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00014012399999999999 2024-09-18T08:05:25.060 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:05:25.060 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:05:25.060 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "op_commit", 2024-09-18T08:05:25.060 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:59.925379+0000", 2024-09-18T08:05:25.060 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0021170820000000002 2024-09-18T08:05:25.060 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:05:25.060 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:05:25.060 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:05:25.060 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:59.926050+0000", 2024-09-18T08:05:25.061 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00067096400000000002 2024-09-18T08:05:25.061 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:05:25.061 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:05:25.061 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:05:25.061 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:59.971088+0000", 2024-09-18T08:05:25.061 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.045038488000000002 2024-09-18T08:05:25.061 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:05:25.061 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:05:25.061 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "commit_sent", 2024-09-18T08:05:25.061 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:59.971136+0000", 2024-09-18T08:05:25.061 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4.7676000000000002e-05 2024-09-18T08:05:25.061 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:05:25.061 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:05:25.061 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-09-18T08:05:25.062 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:59.971167+0000", 2024-09-18T08:05:25.062 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 3.1717999999999997e-05 2024-09-18T08:05:25.062 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-18T08:05:25.062 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-09-18T08:05:25.062 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-18T08:05:25.062 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:05:25.062 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:05:25.062 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.14805.0:93504 7.5 7:add24f38:::benchmark_data_smithi007_73155_object93503:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e108)", 2024-09-18T08:05:25.062 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-09-18T08:05:09.305584+0000", 2024-09-18T08:05:25.062 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 15.647343640999999, 2024-09-18T08:05:25.062 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.040024166, 2024-09-18T08:05:25.062 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-09-18T08:05:25.063 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-09-18T08:05:25.063 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T08:05:25.063 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2024-09-18T08:05:25.063 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.14805", 2024-09-18T08:05:25.063 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "172.21.15.7:0/640320436", 2024-09-18T08:05:25.063 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 93504 2024-09-18T08:05:25.063 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:05:25.063 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-09-18T08:05:25.063 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:05:25.063 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-09-18T08:05:25.063 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:05:09.305584+0000", 2024-09-18T08:05:25.063 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-18T08:05:25.063 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:05:25.063 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:05:25.064 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-09-18T08:05:25.064 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:05:09.305584+0000", 2024-09-18T08:05:25.064 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-18T08:05:25.064 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:05:25.064 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:05:25.064 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-09-18T08:05:25.064 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:05:09.305585+0000", 2024-09-18T08:05:25.064 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.6610000000000001e-06 2024-09-18T08:05:25.064 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:05:25.064 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:05:25.064 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-09-18T08:05:25.064 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:05:09.305756+0000", 2024-09-18T08:05:25.064 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00017088699999999999 2024-09-18T08:05:25.065 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:05:25.065 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:05:25.065 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-09-18T08:05:25.065 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:05:09.305771+0000", 2024-09-18T08:05:25.065 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.5349000000000002e-05 2024-09-18T08:05:25.065 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:05:25.065 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:05:25.065 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-09-18T08:05:25.065 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:05:09.305779+0000", 2024-09-18T08:05:25.065 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 7.238e-06 2024-09-18T08:05:25.065 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:05:25.065 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:05:25.065 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-18T08:05:25.065 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:05:09.305854+0000", 2024-09-18T08:05:25.066 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 7.5202000000000003e-05 2024-09-18T08:05:25.066 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:05:25.066 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:05:25.066 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-09-18T08:05:25.066 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:05:09.305977+0000", 2024-09-18T08:05:25.066 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00012287499999999999 2024-09-18T08:05:25.066 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:05:25.066 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:05:25.066 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "waiting for subops from 0,2", 2024-09-18T08:05:25.066 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:05:09.306109+0000", 2024-09-18T08:05:25.066 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00013263199999999999 2024-09-18T08:05:25.066 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:05:25.066 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:05:25.067 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:05:25.067 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:05:09.309156+0000", 2024-09-18T08:05:25.067 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0030469360000000001 2024-09-18T08:05:25.067 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:05:25.067 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:05:25.067 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "op_commit", 2024-09-18T08:05:25.067 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:05:09.309199+0000", 2024-09-18T08:05:25.067 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4.2762999999999997e-05 2024-09-18T08:05:25.067 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:05:25.067 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:05:25.067 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:05:25.067 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:05:09.345524+0000", 2024-09-18T08:05:25.067 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.036324816000000003 2024-09-18T08:05:25.067 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:05:25.068 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:05:25.068 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "commit_sent", 2024-09-18T08:05:25.068 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:05:09.345565+0000", 2024-09-18T08:05:25.068 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4.0688000000000001e-05 2024-09-18T08:05:25.068 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:05:25.068 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:05:25.068 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-09-18T08:05:25.068 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:05:09.345608+0000", 2024-09-18T08:05:25.068 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4.3118999999999998e-05 2024-09-18T08:05:25.068 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-18T08:05:25.068 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-09-18T08:05:25.068 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-18T08:05:25.068 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:05:25.068 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:05:25.069 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.14805.0:105623 7.5 7:a9038605:::benchmark_data_smithi007_73155_object105622:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e115)", 2024-09-18T08:05:25.069 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-09-18T08:05:22.233147+0000", 2024-09-18T08:05:25.069 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 2.719780605, 2024-09-18T08:05:25.069 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.050722341999999997, 2024-09-18T08:05:25.069 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-09-18T08:05:25.069 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-09-18T08:05:25.069 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T08:05:25.069 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2024-09-18T08:05:25.069 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.14805", 2024-09-18T08:05:25.069 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "172.21.15.7:0/640320436", 2024-09-18T08:05:25.069 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 105623 2024-09-18T08:05:25.069 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:05:25.069 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-09-18T08:05:25.069 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:05:25.069 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-09-18T08:05:25.070 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:05:22.233147+0000", 2024-09-18T08:05:25.070 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-18T08:05:25.070 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:05:25.070 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:05:25.070 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-09-18T08:05:25.070 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:05:22.233147+0000", 2024-09-18T08:05:25.070 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-18T08:05:25.070 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:05:25.070 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:05:25.070 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-09-18T08:05:25.070 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:05:22.233148+0000", 2024-09-18T08:05:25.070 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.3829999999999999e-06 2024-09-18T08:05:25.070 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:05:25.071 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:05:25.071 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-09-18T08:05:25.071 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:05:22.233197+0000", 2024-09-18T08:05:25.071 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4.8918000000000002e-05 2024-09-18T08:05:25.071 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:05:25.071 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:05:25.071 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-09-18T08:05:25.071 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:05:22.233208+0000", 2024-09-18T08:05:25.071 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.1578e-05 2024-09-18T08:05:25.071 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:05:25.071 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:05:25.071 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-09-18T08:05:25.071 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:05:22.233214+0000", 2024-09-18T08:05:25.072 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 5.4029999999999998e-06 2024-09-18T08:05:25.072 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:05:25.072 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:05:25.072 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-18T08:05:25.072 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:05:22.233269+0000", 2024-09-18T08:05:25.072 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 5.4802999999999997e-05 2024-09-18T08:05:25.072 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:05:25.072 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:05:25.072 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-09-18T08:05:25.072 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:05:22.233382+0000", 2024-09-18T08:05:25.072 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00011301 2024-09-18T08:05:25.072 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:05:25.072 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:05:25.072 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "waiting for subops from 0,2", 2024-09-18T08:05:25.073 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:05:22.233542+0000", 2024-09-18T08:05:25.073 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000160354 2024-09-18T08:05:25.073 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:05:25.073 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:05:25.073 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:05:25.073 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:05:22.237194+0000", 2024-09-18T08:05:25.073 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0036518599999999998 2024-09-18T08:05:25.073 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:05:25.073 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:05:25.073 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "op_commit", 2024-09-18T08:05:25.073 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:05:22.237303+0000", 2024-09-18T08:05:25.073 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000108638 2024-09-18T08:05:25.073 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:05:25.074 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:05:25.074 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:05:25.074 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:05:22.283791+0000", 2024-09-18T08:05:25.074 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.046488574999999997 2024-09-18T08:05:25.074 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:05:25.074 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:05:25.074 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "commit_sent", 2024-09-18T08:05:25.074 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:05:22.283834+0000", 2024-09-18T08:05:25.074 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4.3167999999999997e-05 2024-09-18T08:05:25.074 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:05:25.074 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:05:25.074 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-09-18T08:05:25.074 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:05:22.283869+0000", 2024-09-18T08:05:25.074 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 3.4652e-05 2024-09-18T08:05:25.075 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-18T08:05:25.075 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-09-18T08:05:25.075 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-18T08:05:25.075 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:05:25.075 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:05:25.075 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.14805.0:106171 7.5 7:a1841588:::benchmark_data_smithi007_73155_object106170:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e115)", 2024-09-18T08:05:25.075 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-09-18T08:05:22.798206+0000", 2024-09-18T08:05:25.075 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 2.1547216059999998, 2024-09-18T08:05:25.075 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.039393393999999998, 2024-09-18T08:05:25.075 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-09-18T08:05:25.075 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-09-18T08:05:25.075 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T08:05:25.075 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2024-09-18T08:05:25.076 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.14805", 2024-09-18T08:05:25.076 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "172.21.15.7:0/640320436", 2024-09-18T08:05:25.076 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 106171 2024-09-18T08:05:25.076 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:05:25.076 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-09-18T08:05:25.076 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:05:25.076 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-09-18T08:05:25.076 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:05:22.798206+0000", 2024-09-18T08:05:25.076 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-18T08:05:25.076 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:05:25.076 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:05:25.076 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-09-18T08:05:25.076 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:05:22.798206+0000", 2024-09-18T08:05:25.077 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-18T08:05:25.077 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:05:25.077 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:05:25.077 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-09-18T08:05:25.077 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:05:22.798206+0000", 2024-09-18T08:05:25.077 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 5.2300000000000001e-07 2024-09-18T08:05:25.077 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:05:25.077 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:05:25.077 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-09-18T08:05:25.077 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:05:22.798239+0000", 2024-09-18T08:05:25.077 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 3.2845000000000001e-05 2024-09-18T08:05:25.077 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:05:25.077 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:05:25.078 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-09-18T08:05:25.078 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:05:22.798247+0000", 2024-09-18T08:05:25.078 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 7.7729999999999999e-06 2024-09-18T08:05:25.078 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:05:25.078 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:05:25.078 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-09-18T08:05:25.078 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:05:22.798253+0000", 2024-09-18T08:05:25.078 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 6.2890000000000003e-06 2024-09-18T08:05:25.078 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:05:25.078 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:05:25.078 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-18T08:05:25.078 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:05:22.803353+0000", 2024-09-18T08:05:25.078 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0051002169999999998 2024-09-18T08:05:25.078 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:05:25.079 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:05:25.079 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-09-18T08:05:25.079 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:05:22.803466+0000", 2024-09-18T08:05:25.079 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00011266500000000001 2024-09-18T08:05:25.079 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:05:25.079 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:05:25.079 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "waiting for subops from 0,2", 2024-09-18T08:05:25.079 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:05:22.803551+0000", 2024-09-18T08:05:25.079 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 8.4870000000000006e-05 2024-09-18T08:05:25.079 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:05:25.079 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:05:25.079 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "op_commit", 2024-09-18T08:05:25.079 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:05:22.807625+0000", 2024-09-18T08:05:25.079 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0040744700000000002 2024-09-18T08:05:25.080 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:05:25.080 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:05:25.080 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:05:25.080 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:05:22.809655+0000", 2024-09-18T08:05:25.080 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0020296300000000001 2024-09-18T08:05:25.080 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:05:25.080 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:05:25.080 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:05:25.080 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:05:22.837502+0000", 2024-09-18T08:05:25.080 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.027847428 2024-09-18T08:05:25.080 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:05:25.080 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:05:25.080 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "commit_sent", 2024-09-18T08:05:25.081 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:05:22.837555+0000", 2024-09-18T08:05:25.081 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 5.2679000000000002e-05 2024-09-18T08:05:25.081 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:05:25.081 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:05:25.081 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-09-18T08:05:25.081 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:05:22.837599+0000", 2024-09-18T08:05:25.081 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4.4005000000000003e-05 2024-09-18T08:05:25.081 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-18T08:05:25.081 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-09-18T08:05:25.081 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-18T08:05:25.081 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:05:25.081 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:05:25.081 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.14805.0:106172 7.5 7:ac72c295:::benchmark_data_smithi007_73155_object106171:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e115)", 2024-09-18T08:05:25.081 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-09-18T08:05:22.798263+0000", 2024-09-18T08:05:25.082 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 2.1546642440000001, 2024-09-18T08:05:25.082 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.039441602999999999, 2024-09-18T08:05:25.082 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-09-18T08:05:25.082 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-09-18T08:05:25.082 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T08:05:25.082 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2024-09-18T08:05:25.082 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.14805", 2024-09-18T08:05:25.082 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "172.21.15.7:0/640320436", 2024-09-18T08:05:25.082 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 106172 2024-09-18T08:05:25.082 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:05:25.082 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-09-18T08:05:25.082 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:05:25.082 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-09-18T08:05:25.082 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:05:22.798263+0000", 2024-09-18T08:05:25.083 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-18T08:05:25.083 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:05:25.083 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:05:25.083 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-09-18T08:05:25.083 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:05:22.798263+0000", 2024-09-18T08:05:25.083 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-18T08:05:25.083 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:05:25.083 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:05:25.083 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-09-18T08:05:25.083 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:05:22.798263+0000", 2024-09-18T08:05:25.083 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 5.2900000000000004e-07 2024-09-18T08:05:25.083 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:05:25.083 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:05:25.083 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-09-18T08:05:25.084 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:05:22.798292+0000", 2024-09-18T08:05:25.084 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 2.8588000000000001e-05 2024-09-18T08:05:25.084 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:05:25.084 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:05:25.084 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-09-18T08:05:25.084 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:05:22.798299+0000", 2024-09-18T08:05:25.084 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 7.0090000000000001e-06 2024-09-18T08:05:25.084 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:05:25.084 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:05:25.084 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-09-18T08:05:25.084 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:05:22.798305+0000", 2024-09-18T08:05:25.084 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 6.1750000000000002e-06 2024-09-18T08:05:25.084 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:05:25.085 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:05:25.085 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-18T08:05:25.085 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:05:22.805565+0000", 2024-09-18T08:05:25.085 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0072596869999999999 2024-09-18T08:05:25.085 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:05:25.085 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:05:25.085 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-09-18T08:05:25.085 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:05:22.805689+0000", 2024-09-18T08:05:25.085 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000124213 2024-09-18T08:05:25.085 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:05:25.085 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:05:25.085 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "waiting for subops from 0,2", 2024-09-18T08:05:25.085 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:05:22.805784+0000", 2024-09-18T08:05:25.086 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 9.4885999999999999e-05 2024-09-18T08:05:25.086 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:05:25.086 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:05:25.086 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "op_commit", 2024-09-18T08:05:25.086 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:05:22.807895+0000", 2024-09-18T08:05:25.086 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0021114369999999999 2024-09-18T08:05:25.086 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:05:25.086 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:05:25.086 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:05:25.086 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:05:22.817578+0000", 2024-09-18T08:05:25.086 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0096826470000000008 2024-09-18T08:05:25.086 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:05:25.086 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:05:25.086 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:05:25.086 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:05:22.837633+0000", 2024-09-18T08:05:25.087 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.020054969999999998 2024-09-18T08:05:25.087 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:05:25.087 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:05:25.087 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "commit_sent", 2024-09-18T08:05:25.087 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:05:22.837674+0000", 2024-09-18T08:05:25.087 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4.1048999999999999e-05 2024-09-18T08:05:25.087 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:05:25.087 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:05:25.087 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-09-18T08:05:25.087 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:05:22.837705+0000", 2024-09-18T08:05:25.087 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 3.0412999999999998e-05 2024-09-18T08:05:25.087 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-18T08:05:25.087 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-09-18T08:05:25.087 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-18T08:05:25.088 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-18T08:05:25.088 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-09-18T08:05:25.088 INFO:teuthology.orchestra.run.smithi050.stdout:} 2024-09-18T08:05:25.115 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:05:25.115 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:05:25.116 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6' 2024-09-18T08:05:25.129 INFO:teuthology.orchestra.run.smithi007.stderr:reweighted osd.3 to 1 (10000) 2024-09-18T08:05:25.216 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4 2024-09-18T08:05:25.247 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:05:25.247 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:05:25.248 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4' 2024-09-18T08:05:25.348 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0 2024-09-18T08:05:25.349 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid 779349ca-7593-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-18T08:05:25.395 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:05:25.396 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:05:25.396 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0' 2024-09-18T08:05:25.397 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:05:25 smithi007 ceph-mon[29459]: from='client.? 172.21.15.7:0/1811104688' entity='client.admin' cmd={"prefix": "osd reweight", "id": 3, "weight": 1.0} : dispatch 2024-09-18T08:05:25.397 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:05:25 smithi007 ceph-mon[39249]: from='client.? 172.21.15.7:0/1811104688' entity='client.admin' cmd={"prefix": "osd reweight", "id": 3, "weight": 1.0} : dispatch 2024-09-18T08:05:25.405 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:05:25 smithi050 ceph-mon[32371]: from='client.? 172.21.15.7:0/1811104688' entity='client.admin' cmd={"prefix": "osd reweight", "id": 3, "weight": 1.0} : dispatch 2024-09-18T08:05:25.497 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0 2024-09-18T08:05:25.537 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:05:25.537 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:05:25.538 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0' 2024-09-18T08:05:25.591 INFO:tasks.radosbench.radosbench.0.smithi007.stdout:2024-09-18T08:05:25.590473+0000 min lat: 0.00117713 max lat: 1.0787 avg lat: 0.011742 2024-09-18T08:05:25.591 INFO:tasks.radosbench.radosbench.0.smithi007.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-09-18T08:05:25.592 INFO:tasks.radosbench.radosbench.0.smithi007.stdout: 80 16 109008 108992 85.1241 57.4375 0.0274006 0.011742 2024-09-18T08:05:25.638 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4 2024-09-18T08:05:25.667 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:05:25.667 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:05:25.667 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4' 2024-09-18T08:05:25.768 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2 2024-09-18T08:05:25.788 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid 779349ca-7593-11ef-bceb-c7b262605968 -- ceph daemon osd.0 dump_ops_in_flight 2024-09-18T08:05:25.809 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:05:25.810 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:05:25.810 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2' 2024-09-18T08:05:25.911 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0 2024-09-18T08:05:25.966 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:05:25.967 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:05:25.967 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0' 2024-09-18T08:05:26.068 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2 2024-09-18T08:05:26.113 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:05:26.114 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:05:26.114 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2' 2024-09-18T08:05:26.155 INFO:teuthology.orchestra.run.smithi007.stderr:noscrub is unset 2024-09-18T08:05:26.215 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3 2024-09-18T08:05:26.262 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:05:26.263 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:05:26.263 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3' 2024-09-18T08:05:26.364 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0 2024-09-18T08:05:26.403 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:05:26.404 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:05:26.404 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0' 2024-09-18T08:05:26.481 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:05:26 smithi007 ceph-mon[29459]: pgmap v110: 148 pgs: 148 active+clean; 6.5 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 851 B/s rd, 53 MiB/s wr, 842 op/s 2024-09-18T08:05:26.481 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:05:26 smithi007 ceph-mon[29459]: from='client.? 172.21.15.7:0/1811104688' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]': finished 2024-09-18T08:05:26.481 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:05:26 smithi007 ceph-mon[29459]: osdmap e117: 8 total, 8 up, 8 in 2024-09-18T08:05:26.481 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:05:26 smithi007 ceph-mon[29459]: from='client.? 172.21.15.7:0/3229299872' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-18T08:05:26.481 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:05:26 smithi007 ceph-mon[39249]: pgmap v110: 148 pgs: 148 active+clean; 6.5 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 851 B/s rd, 53 MiB/s wr, 842 op/s 2024-09-18T08:05:26.481 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:05:26 smithi007 ceph-mon[39249]: from='client.? 172.21.15.7:0/1811104688' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]': finished 2024-09-18T08:05:26.482 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:05:26 smithi007 ceph-mon[39249]: osdmap e117: 8 total, 8 up, 8 in 2024-09-18T08:05:26.482 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:05:26 smithi007 ceph-mon[39249]: from='client.? 172.21.15.7:0/3229299872' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-18T08:05:26.505 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3 2024-09-18T08:05:26.550 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:05:26.550 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:05:26.551 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3' 2024-09-18T08:05:26.591 INFO:tasks.radosbench.radosbench.0.smithi007.stdout: 81 16 109959 109943 84.8069 59.4375 0.0107584 0.0117863 2024-09-18T08:05:26.652 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6 2024-09-18T08:05:26.655 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:05:26 smithi050 ceph-mon[32371]: pgmap v110: 148 pgs: 148 active+clean; 6.5 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 851 B/s rd, 53 MiB/s wr, 842 op/s 2024-09-18T08:05:26.656 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:05:26 smithi050 ceph-mon[32371]: from='client.? 172.21.15.7:0/1811104688' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]': finished 2024-09-18T08:05:26.656 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:05:26 smithi050 ceph-mon[32371]: osdmap e117: 8 total, 8 up, 8 in 2024-09-18T08:05:26.656 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:05:26 smithi050 ceph-mon[32371]: from='client.? 172.21.15.7:0/3229299872' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-18T08:05:26.680 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:05:26.681 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:05:26.682 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6' 2024-09-18T08:05:26.783 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2 2024-09-18T08:05:26.819 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:05:26.820 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:05:26.820 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2' 2024-09-18T08:05:26.921 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6 2024-09-18T08:05:26.947 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:05:26.948 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:05:26.948 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6' 2024-09-18T08:05:26.980 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Sep 18 08:05:26 smithi007 ceph-779349ca-7593-11ef-bceb-c7b262605968-mgr-y[29688]: ::ffff:172.21.15.50 - - [18/Sep/2024:08:05:26] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-09-18T08:05:27.049 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5 2024-09-18T08:05:27.074 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:05:27.075 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:05:27.075 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5' 2024-09-18T08:05:27.176 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5 2024-09-18T08:05:27.204 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:05:27.205 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:05:27.205 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5' 2024-09-18T08:05:27.306 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0 2024-09-18T08:05:27.346 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:05:27.347 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:05:27.347 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0' 2024-09-18T08:05:27.448 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3 2024-09-18T08:05:27.481 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:05:27 smithi007 ceph-mon[29459]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-18T08:05:27.481 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:05:27 smithi007 ceph-mon[29459]: from='client.? 172.21.15.7:0/3229299872' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-18T08:05:27.481 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:05:27 smithi007 ceph-mon[29459]: osdmap e118: 8 total, 8 up, 8 in 2024-09-18T08:05:27.481 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:05:27 smithi007 ceph-mon[29459]: 4.17 scrub starts 2024-09-18T08:05:27.481 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:05:27 smithi007 ceph-mon[29459]: 4.17 scrub ok 2024-09-18T08:05:27.483 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:05:27 smithi007 ceph-mon[39249]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-18T08:05:27.483 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:05:27 smithi007 ceph-mon[39249]: from='client.? 172.21.15.7:0/3229299872' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-18T08:05:27.483 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:05:27 smithi007 ceph-mon[39249]: osdmap e118: 8 total, 8 up, 8 in 2024-09-18T08:05:27.483 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:05:27 smithi007 ceph-mon[39249]: 4.17 scrub starts 2024-09-18T08:05:27.483 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:05:27 smithi007 ceph-mon[39249]: 4.17 scrub ok 2024-09-18T08:05:27.493 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:05:27.493 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:05:27.494 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3' 2024-09-18T08:05:27.583 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid 779349ca-7593-11ef-bceb-c7b262605968 -- ceph osd reweight 4 1 2024-09-18T08:05:27.591 INFO:tasks.radosbench.radosbench.0.smithi007.stdout: 82 16 110863 110847 84.4615 56.5 0.0258171 0.0118342 2024-09-18T08:05:27.595 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2 2024-09-18T08:05:27.656 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:05:27 smithi050 ceph-mon[32371]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-18T08:05:27.656 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:05:27 smithi050 ceph-mon[32371]: from='client.? 172.21.15.7:0/3229299872' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-18T08:05:27.656 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:05:27 smithi050 ceph-mon[32371]: osdmap e118: 8 total, 8 up, 8 in 2024-09-18T08:05:27.656 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:05:27 smithi050 ceph-mon[32371]: 4.17 scrub starts 2024-09-18T08:05:27.656 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:05:27 smithi050 ceph-mon[32371]: 4.17 scrub ok 2024-09-18T08:05:27.665 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:05:27.666 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:05:27.666 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2' 2024-09-18T08:05:27.768 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1 2024-09-18T08:05:27.821 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:05:27.822 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:05:27.822 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1' 2024-09-18T08:05:27.923 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3 2024-09-18T08:05:27.983 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:05:27.984 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:05:27.984 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3' 2024-09-18T08:05:28.085 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0 2024-09-18T08:05:28.135 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:05:28.136 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:05:28.136 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0' 2024-09-18T08:05:28.237 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6 2024-09-18T08:05:28.262 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:05:28.263 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:05:28.263 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6' 2024-09-18T08:05:28.362 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/779349ca-7593-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T08:05:28.363 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5 2024-09-18T08:05:28.375 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/779349ca-7593-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T08:05:28.390 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:05:28.390 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:05:28.391 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5' 2024-09-18T08:05:28.480 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:05:28 smithi007 ceph-mon[29459]: 7.b scrub starts 2024-09-18T08:05:28.480 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:05:28 smithi007 ceph-mon[29459]: pgmap v113: 148 pgs: 148 active+clean; 6.5 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 853 B/s rd, 53 MiB/s wr, 844 op/s 2024-09-18T08:05:28.481 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:05:28 smithi007 ceph-mon[39249]: 7.b scrub starts 2024-09-18T08:05:28.481 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:05:28 smithi007 ceph-mon[39249]: pgmap v113: 148 pgs: 148 active+clean; 6.5 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 853 B/s rd, 53 MiB/s wr, 844 op/s 2024-09-18T08:05:28.491 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6 2024-09-18T08:05:28.518 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:05:28.519 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:05:28.519 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6' 2024-09-18T08:05:28.592 INFO:tasks.radosbench.radosbench.0.smithi007.stdout: 83 16 111671 111655 84.0524 50.5 0.0149263 0.0118924 2024-09-18T08:05:28.620 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6 2024-09-18T08:05:28.645 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:05:28.646 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:05:28.647 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6' 2024-09-18T08:05:28.655 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:05:28 smithi050 ceph-mon[32371]: 7.b scrub starts 2024-09-18T08:05:28.656 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:05:28 smithi050 ceph-mon[32371]: pgmap v113: 148 pgs: 148 active+clean; 6.5 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 853 B/s rd, 53 MiB/s wr, 844 op/s 2024-09-18T08:05:28.747 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7 2024-09-18T08:05:28.775 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:05:28.775 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:05:28.775 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7' 2024-09-18T08:05:28.876 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4 2024-09-18T08:05:28.903 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:05:28.904 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:05:28.905 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4' 2024-09-18T08:05:29.006 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2 2024-09-18T08:05:29.047 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:05:29.048 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:05:29.048 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2' 2024-09-18T08:05:29.149 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0 2024-09-18T08:05:29.155 INFO:journalctl@ceph.iscsi.iscsi.a.smithi050.stdout:Sep 18 08:05:28 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-iscsi-iscsi-a[57782]: debug there is no tcmu-runner data available 2024-09-18T08:05:29.171 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:05:29.172 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:05:29.172 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0' 2024-09-18T08:05:29.273 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4 2024-09-18T08:05:29.299 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:05:29.300 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:05:29.300 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4' 2024-09-18T08:05:29.401 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2 2024-09-18T08:05:29.441 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:05:29.442 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:05:29.442 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2' 2024-09-18T08:05:29.481 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:05:29 smithi007 ceph-mon[29459]: 3.1b scrub starts 2024-09-18T08:05:29.481 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:05:29 smithi007 ceph-mon[29459]: 3.1b scrub ok 2024-09-18T08:05:29.481 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:05:29 smithi007 ceph-mon[29459]: 5.15 scrub starts 2024-09-18T08:05:29.481 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:05:29 smithi007 ceph-mon[29459]: 5.15 scrub ok 2024-09-18T08:05:29.481 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:05:29 smithi007 ceph-mon[29459]: 4.1c scrub starts 2024-09-18T08:05:29.481 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:05:29 smithi007 ceph-mon[29459]: 4.1c scrub ok 2024-09-18T08:05:29.481 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:05:29 smithi007 ceph-mon[29459]: purged_snaps scrub starts 2024-09-18T08:05:29.481 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:05:29 smithi007 ceph-mon[29459]: purged_snaps scrub ok 2024-09-18T08:05:29.481 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:05:29 smithi007 ceph-mon[39249]: 3.1b scrub starts 2024-09-18T08:05:29.482 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:05:29 smithi007 ceph-mon[39249]: 3.1b scrub ok 2024-09-18T08:05:29.482 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:05:29 smithi007 ceph-mon[39249]: 5.15 scrub starts 2024-09-18T08:05:29.482 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:05:29 smithi007 ceph-mon[39249]: 5.15 scrub ok 2024-09-18T08:05:29.482 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:05:29 smithi007 ceph-mon[39249]: 4.1c scrub starts 2024-09-18T08:05:29.482 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:05:29 smithi007 ceph-mon[39249]: 4.1c scrub ok 2024-09-18T08:05:29.482 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:05:29 smithi007 ceph-mon[39249]: purged_snaps scrub starts 2024-09-18T08:05:29.482 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:05:29 smithi007 ceph-mon[39249]: purged_snaps scrub ok 2024-09-18T08:05:29.543 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2 2024-09-18T08:05:29.587 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:05:29.588 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:05:29.588 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2' 2024-09-18T08:05:29.592 INFO:tasks.radosbench.radosbench.0.smithi007.stdout: 84 16 112508 112492 83.6746 52.3125 0.0100054 0.0119455 2024-09-18T08:05:29.656 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:05:29 smithi050 ceph-mon[32371]: 3.1b scrub starts 2024-09-18T08:05:29.656 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:05:29 smithi050 ceph-mon[32371]: 3.1b scrub ok 2024-09-18T08:05:29.656 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:05:29 smithi050 ceph-mon[32371]: 5.15 scrub starts 2024-09-18T08:05:29.656 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:05:29 smithi050 ceph-mon[32371]: 5.15 scrub ok 2024-09-18T08:05:29.656 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:05:29 smithi050 ceph-mon[32371]: 4.1c scrub starts 2024-09-18T08:05:29.656 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:05:29 smithi050 ceph-mon[32371]: 4.1c scrub ok 2024-09-18T08:05:29.656 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:05:29 smithi050 ceph-mon[32371]: purged_snaps scrub starts 2024-09-18T08:05:29.656 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:05:29 smithi050 ceph-mon[32371]: purged_snaps scrub ok 2024-09-18T08:05:29.689 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7 2024-09-18T08:05:29.716 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:05:29.717 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:05:29.717 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7' 2024-09-18T08:05:29.818 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3 2024-09-18T08:05:29.861 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:05:29.862 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:05:29.862 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3' 2024-09-18T08:05:29.963 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0 2024-09-18T08:05:29.997 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:05:29.997 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:05:29.998 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0' 2024-09-18T08:05:30.099 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6 2024-09-18T08:05:30.125 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:05:30.125 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:05:30.126 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6' 2024-09-18T08:05:30.222 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid 779349ca-7593-11ef-bceb-c7b262605968 -- ceph osd unset nodeep-scrub 2024-09-18T08:05:30.227 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7 2024-09-18T08:05:30.249 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:05:30.249 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:05:30.250 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7' 2024-09-18T08:05:30.350 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4 2024-09-18T08:05:30.377 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:05:30.378 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:05:30.378 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4' 2024-09-18T08:05:30.479 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0 2024-09-18T08:05:30.480 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:05:30 smithi007 ceph-mon[29459]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-18T08:05:30.480 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:05:30 smithi007 ceph-mon[29459]: 7.b scrub ok 2024-09-18T08:05:30.480 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:05:30 smithi007 ceph-mon[29459]: pgmap v114: 148 pgs: 1 active+clean+scrubbing, 147 active+clean; 6.8 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 853 B/s rd, 93 MiB/s wr, 1.48k op/s 2024-09-18T08:05:30.480 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:05:30 smithi007 ceph-mon[29459]: osdmap e119: 8 total, 8 up, 8 in 2024-09-18T08:05:30.482 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:05:30 smithi007 ceph-mon[39249]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-18T08:05:30.482 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:05:30 smithi007 ceph-mon[39249]: 7.b scrub ok 2024-09-18T08:05:30.482 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:05:30 smithi007 ceph-mon[39249]: pgmap v114: 148 pgs: 1 active+clean+scrubbing, 147 active+clean; 6.8 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 853 B/s rd, 93 MiB/s wr, 1.48k op/s 2024-09-18T08:05:30.483 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:05:30 smithi007 ceph-mon[39249]: osdmap e119: 8 total, 8 up, 8 in 2024-09-18T08:05:30.526 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:05:30.526 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:05:30.527 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0' 2024-09-18T08:05:30.592 INFO:tasks.radosbench.radosbench.0.smithi007.stdout: 85 16 113374 113358 83.3269 54.125 0.0128181 0.0119878 2024-09-18T08:05:30.627 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2 2024-09-18T08:05:30.656 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:05:30 smithi050 ceph-mon[32371]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-18T08:05:30.656 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:05:30 smithi050 ceph-mon[32371]: 7.b scrub ok 2024-09-18T08:05:30.656 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:05:30 smithi050 ceph-mon[32371]: pgmap v114: 148 pgs: 1 active+clean+scrubbing, 147 active+clean; 6.8 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 853 B/s rd, 93 MiB/s wr, 1.48k op/s 2024-09-18T08:05:30.656 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:05:30 smithi050 ceph-mon[32371]: osdmap e119: 8 total, 8 up, 8 in 2024-09-18T08:05:30.668 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:05:30.669 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:05:30.669 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2' 2024-09-18T08:05:30.770 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7 2024-09-18T08:05:30.793 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:05:30.794 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:05:30.794 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7' 2024-09-18T08:05:30.895 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6 2024-09-18T08:05:30.925 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:05:30.926 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:05:30.926 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6' 2024-09-18T08:05:31.027 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7 2024-09-18T08:05:31.056 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:05:31.056 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:05:31.057 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7' 2024-09-18T08:05:31.158 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3 2024-09-18T08:05:31.202 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:05:31.202 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:05:31.203 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3' 2024-09-18T08:05:31.304 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2 2024-09-18T08:05:31.362 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:05:31.363 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:05:31.363 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2' 2024-09-18T08:05:31.390 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:05:31 smithi007 ceph-mon[39249]: 5.12 scrub starts 2024-09-18T08:05:31.391 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:05:31 smithi007 ceph-mon[39249]: 5.12 scrub ok 2024-09-18T08:05:31.391 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:05:31 smithi007 ceph-mon[29459]: 5.12 scrub starts 2024-09-18T08:05:31.391 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:05:31 smithi007 ceph-mon[29459]: 5.12 scrub ok 2024-09-18T08:05:31.464 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6 2024-09-18T08:05:31.492 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:05:31.493 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:05:31.493 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6' 2024-09-18T08:05:31.592 INFO:tasks.radosbench.radosbench.0.smithi007.stdout: 86 16 114363 114347 83.0767 61.8125 0.00341786 0.0120291 2024-09-18T08:05:31.593 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1 2024-09-18T08:05:31.622 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:05:31.623 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:05:31.623 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1' 2024-09-18T08:05:31.656 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:05:31 smithi050 ceph-mon[32371]: 5.12 scrub starts 2024-09-18T08:05:31.656 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:05:31 smithi050 ceph-mon[32371]: 5.12 scrub ok 2024-09-18T08:05:31.724 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6 2024-09-18T08:05:31.752 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:05:31.753 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:05:31.753 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6' 2024-09-18T08:05:31.854 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0 2024-09-18T08:05:31.893 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:05:31.893 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:05:31.894 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0' 2024-09-18T08:05:31.995 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3 2024-09-18T08:05:32.039 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:05:32.039 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:05:32.040 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3' 2024-09-18T08:05:32.140 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2 2024-09-18T08:05:32.182 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:05:32.183 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:05:32.183 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2' 2024-09-18T08:05:32.255 INFO:teuthology.orchestra.run.smithi007.stdout:{ 2024-09-18T08:05:32.256 INFO:teuthology.orchestra.run.smithi007.stdout: "ops": [ 2024-09-18T08:05:32.256 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:05:32.256 INFO:teuthology.orchestra.run.smithi007.stdout: "description": "osd_op(client.14805.0:114094 7.9 7:967e8ec2:::benchmark_data_smithi007_73155_object114093:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536 in=65536b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e119)", 2024-09-18T08:05:32.256 INFO:teuthology.orchestra.run.smithi007.stdout: "initiated_at": "2024-09-18T08:05:31.305736+0000", 2024-09-18T08:05:32.256 INFO:teuthology.orchestra.run.smithi007.stdout: "age": 0.002081271, 2024-09-18T08:05:32.256 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.0021251680000000002, 2024-09-18T08:05:32.256 INFO:teuthology.orchestra.run.smithi007.stdout: "continuous": false, 2024-09-18T08:05:32.256 INFO:teuthology.orchestra.run.smithi007.stdout: "type_data": { 2024-09-18T08:05:32.256 INFO:teuthology.orchestra.run.smithi007.stdout: "flag_point": "waiting for sub ops", 2024-09-18T08:05:32.256 INFO:teuthology.orchestra.run.smithi007.stdout: "client_info": { 2024-09-18T08:05:32.256 INFO:teuthology.orchestra.run.smithi007.stdout: "client": "client.14805", 2024-09-18T08:05:32.257 INFO:teuthology.orchestra.run.smithi007.stdout: "client_addr": "172.21.15.7:0/640320436", 2024-09-18T08:05:32.257 INFO:teuthology.orchestra.run.smithi007.stdout: "tid": 114094 2024-09-18T08:05:32.257 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:05:32.257 INFO:teuthology.orchestra.run.smithi007.stdout: "events": [ 2024-09-18T08:05:32.257 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:05:32.257 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "initiated", 2024-09-18T08:05:32.257 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:05:31.305736+0000", 2024-09-18T08:05:32.257 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-09-18T08:05:32.257 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:05:32.257 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:05:32.257 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "header_read", 2024-09-18T08:05:32.258 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:05:31.305736+0000", 2024-09-18T08:05:32.258 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-09-18T08:05:32.258 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:05:32.258 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:05:32.258 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "throttled", 2024-09-18T08:05:32.258 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:05:31.305738+0000", 2024-09-18T08:05:32.258 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.53e-06 2024-09-18T08:05:32.258 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:05:32.258 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:05:32.258 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "all_read", 2024-09-18T08:05:32.258 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:05:31.305762+0000", 2024-09-18T08:05:32.258 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 2.3886000000000001e-05 2024-09-18T08:05:32.259 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:05:32.259 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:05:32.259 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "dispatched", 2024-09-18T08:05:32.259 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:05:31.305773+0000", 2024-09-18T08:05:32.259 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.1695000000000001e-05 2024-09-18T08:05:32.259 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:05:32.259 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:05:32.259 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "queued_for_pg", 2024-09-18T08:05:32.259 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:05:31.305781+0000", 2024-09-18T08:05:32.259 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 7.2459999999999997e-06 2024-09-18T08:05:32.259 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:05:32.259 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:05:32.259 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-09-18T08:05:32.259 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:05:31.306671+0000", 2024-09-18T08:05:32.260 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.00089042199999999998 2024-09-18T08:05:32.260 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:05:32.260 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:05:32.260 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "started", 2024-09-18T08:05:32.260 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:05:31.306795+0000", 2024-09-18T08:05:32.260 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.00012370799999999999 2024-09-18T08:05:32.260 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:05:32.260 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:05:32.260 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "waiting for subops from 1,4", 2024-09-18T08:05:32.260 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:05:31.307148+0000", 2024-09-18T08:05:32.260 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.00035307699999999998 2024-09-18T08:05:32.260 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-09-18T08:05:32.260 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-09-18T08:05:32.261 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-09-18T08:05:32.261 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:05:32.261 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:05:32.261 INFO:teuthology.orchestra.run.smithi007.stdout: "description": "osd_repop_reply(client.14805.0:114086 7.9 e119/83)", 2024-09-18T08:05:32.261 INFO:teuthology.orchestra.run.smithi007.stdout: "initiated_at": "2024-09-18T08:05:31.306986+0000", 2024-09-18T08:05:32.261 INFO:teuthology.orchestra.run.smithi007.stdout: "age": 0.00083133899999999995, 2024-09-18T08:05:32.261 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.00099744700000000005, 2024-09-18T08:05:32.261 INFO:teuthology.orchestra.run.smithi007.stdout: "continuous": false, 2024-09-18T08:05:32.261 INFO:teuthology.orchestra.run.smithi007.stdout: "type_data": { 2024-09-18T08:05:32.261 INFO:teuthology.orchestra.run.smithi007.stdout: "flag_point": "queued for pg", 2024-09-18T08:05:32.261 INFO:teuthology.orchestra.run.smithi007.stdout: "events": [ 2024-09-18T08:05:32.261 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:05:32.262 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "initiated", 2024-09-18T08:05:32.262 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:05:31.306986+0000", 2024-09-18T08:05:32.262 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-09-18T08:05:32.262 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:05:32.262 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:05:32.262 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "header_read", 2024-09-18T08:05:32.262 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:05:31.306986+0000", 2024-09-18T08:05:32.262 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-09-18T08:05:32.262 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:05:32.262 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:05:32.262 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "throttled", 2024-09-18T08:05:32.262 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:05:31.306986+0000", 2024-09-18T08:05:32.263 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 2.41e-07 2024-09-18T08:05:32.263 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:05:32.263 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:05:32.263 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "all_read", 2024-09-18T08:05:32.263 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:05:31.306988+0000", 2024-09-18T08:05:32.263 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.809e-06 2024-09-18T08:05:32.263 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:05:32.264 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:05:32.264 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "dispatched", 2024-09-18T08:05:32.264 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:05:31.306993+0000", 2024-09-18T08:05:32.264 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 5.2220000000000001e-06 2024-09-18T08:05:32.264 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:05:32.264 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:05:32.264 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "queued_for_pg", 2024-09-18T08:05:32.264 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:05:31.306997+0000", 2024-09-18T08:05:32.264 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 3.9870000000000001e-06 2024-09-18T08:05:32.264 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-09-18T08:05:32.264 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-09-18T08:05:32.264 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-09-18T08:05:32.264 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:05:32.265 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:05:32.265 INFO:teuthology.orchestra.run.smithi007.stdout: "description": "osd_repop(client.14805.0:114085 7.e e119/83)", 2024-09-18T08:05:32.265 INFO:teuthology.orchestra.run.smithi007.stdout: "initiated_at": "2024-09-18T08:05:31.307837+0000", 2024-09-18T08:05:32.265 INFO:teuthology.orchestra.run.smithi007.stdout: "age": 4294967295.9999804, 2024-09-18T08:05:32.265 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.000186047, 2024-09-18T08:05:32.265 INFO:teuthology.orchestra.run.smithi007.stdout: "continuous": false, 2024-09-18T08:05:32.265 INFO:teuthology.orchestra.run.smithi007.stdout: "type_data": { 2024-09-18T08:05:32.265 INFO:teuthology.orchestra.run.smithi007.stdout: "flag_point": "queued for pg", 2024-09-18T08:05:32.265 INFO:teuthology.orchestra.run.smithi007.stdout: "events": [ 2024-09-18T08:05:32.265 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:05:32.265 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "initiated", 2024-09-18T08:05:32.266 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:05:31.307837+0000", 2024-09-18T08:05:32.266 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-09-18T08:05:32.266 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:05:32.266 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:05:32.266 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "header_read", 2024-09-18T08:05:32.266 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:05:31.307837+0000", 2024-09-18T08:05:32.266 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-09-18T08:05:32.266 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:05:32.266 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:05:32.266 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "throttled", 2024-09-18T08:05:32.266 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:05:31.307838+0000", 2024-09-18T08:05:32.266 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 8.47e-07 2024-09-18T08:05:32.266 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:05:32.267 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:05:32.267 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "all_read", 2024-09-18T08:05:32.267 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:05:31.307870+0000", 2024-09-18T08:05:32.267 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 3.2404999999999999e-05 2024-09-18T08:05:32.267 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:05:32.267 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:05:32.267 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "dispatched", 2024-09-18T08:05:32.267 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:05:31.307884+0000", 2024-09-18T08:05:32.267 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.3406e-05 2024-09-18T08:05:32.267 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:05:32.267 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:05:32.267 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "queued_for_pg", 2024-09-18T08:05:32.267 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:05:31.307891+0000", 2024-09-18T08:05:32.267 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 7.5780000000000001e-06 2024-09-18T08:05:32.268 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-09-18T08:05:32.268 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-09-18T08:05:32.268 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-09-18T08:05:32.268 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:05:32.268 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:05:32.268 INFO:teuthology.orchestra.run.smithi007.stdout: "description": "osd_op(client.14805.0:114082 7.7 7:e73ac9ec:::benchmark_data_smithi007_73155_object114081:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536 in=65536b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e119)", 2024-09-18T08:05:32.268 INFO:teuthology.orchestra.run.smithi007.stdout: "initiated_at": "2024-09-18T08:05:31.293132+0000", 2024-09-18T08:05:32.268 INFO:teuthology.orchestra.run.smithi007.stdout: "age": 0.014685718, 2024-09-18T08:05:32.269 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.014940332000000001, 2024-09-18T08:05:32.269 INFO:teuthology.orchestra.run.smithi007.stdout: "continuous": false, 2024-09-18T08:05:32.269 INFO:teuthology.orchestra.run.smithi007.stdout: "type_data": { 2024-09-18T08:05:32.269 INFO:teuthology.orchestra.run.smithi007.stdout: "flag_point": "waiting for sub ops", 2024-09-18T08:05:32.269 INFO:teuthology.orchestra.run.smithi007.stdout: "client_info": { 2024-09-18T08:05:32.269 INFO:teuthology.orchestra.run.smithi007.stdout: "client": "client.14805", 2024-09-18T08:05:32.269 INFO:teuthology.orchestra.run.smithi007.stdout: "client_addr": "172.21.15.7:0/640320436", 2024-09-18T08:05:32.269 INFO:teuthology.orchestra.run.smithi007.stdout: "tid": 114082 2024-09-18T08:05:32.269 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:05:32.269 INFO:teuthology.orchestra.run.smithi007.stdout: "events": [ 2024-09-18T08:05:32.269 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:05:32.270 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "initiated", 2024-09-18T08:05:32.270 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:05:31.293132+0000", 2024-09-18T08:05:32.270 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-09-18T08:05:32.270 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:05:32.270 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:05:32.270 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "header_read", 2024-09-18T08:05:32.271 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:05:31.293132+0000", 2024-09-18T08:05:32.271 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-09-18T08:05:32.271 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:05:32.271 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:05:32.271 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "throttled", 2024-09-18T08:05:32.271 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:05:31.293134+0000", 2024-09-18T08:05:32.271 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.8840000000000001e-06 2024-09-18T08:05:32.271 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:05:32.271 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:05:32.272 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "all_read", 2024-09-18T08:05:32.272 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:05:31.293160+0000", 2024-09-18T08:05:32.272 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 2.6472999999999999e-05 2024-09-18T08:05:32.272 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:05:32.272 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:05:32.272 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "dispatched", 2024-09-18T08:05:32.272 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:05:31.293179+0000", 2024-09-18T08:05:32.272 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.8661999999999999e-05 2024-09-18T08:05:32.272 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:05:32.272 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:05:32.272 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "queued_for_pg", 2024-09-18T08:05:32.273 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:05:31.293188+0000", 2024-09-18T08:05:32.273 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 9.6339999999999996e-06 2024-09-18T08:05:32.273 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:05:32.273 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:05:32.273 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-09-18T08:05:32.273 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:05:31.293362+0000", 2024-09-18T08:05:32.273 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.000173396 2024-09-18T08:05:32.273 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:05:32.273 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:05:32.274 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "started", 2024-09-18T08:05:32.274 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:05:31.293488+0000", 2024-09-18T08:05:32.274 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.00012573900000000001 2024-09-18T08:05:32.274 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:05:32.274 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:05:32.274 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "waiting for subops from 3,6", 2024-09-18T08:05:32.274 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:05:31.295566+0000", 2024-09-18T08:05:32.274 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.0020786720000000002 2024-09-18T08:05:32.274 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:05:32.274 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:05:32.274 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "op_commit", 2024-09-18T08:05:32.274 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:05:31.298673+0000", 2024-09-18T08:05:32.274 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.0031064529999999999 2024-09-18T08:05:32.274 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:05:32.275 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:05:32.275 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:05:32.275 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:05:31.299749+0000", 2024-09-18T08:05:32.275 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.0010760430000000001 2024-09-18T08:05:32.275 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-09-18T08:05:32.275 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-09-18T08:05:32.275 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-09-18T08:05:32.275 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:05:32.275 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:05:32.275 INFO:teuthology.orchestra.run.smithi007.stdout: "description": "osd_op(client.14805.0:114083 7.f 7:f6986497:::benchmark_data_smithi007_73155_object114082:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536 in=65536b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e119)", 2024-09-18T08:05:32.275 INFO:teuthology.orchestra.run.smithi007.stdout: "initiated_at": "2024-09-18T08:05:31.293212+0000", 2024-09-18T08:05:32.275 INFO:teuthology.orchestra.run.smithi007.stdout: "age": 0.014605068000000001, 2024-09-18T08:05:32.275 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.014942171000000001, 2024-09-18T08:05:32.275 INFO:teuthology.orchestra.run.smithi007.stdout: "continuous": false, 2024-09-18T08:05:32.276 INFO:teuthology.orchestra.run.smithi007.stdout: "type_data": { 2024-09-18T08:05:32.276 INFO:teuthology.orchestra.run.smithi007.stdout: "flag_point": "waiting for sub ops", 2024-09-18T08:05:32.276 INFO:teuthology.orchestra.run.smithi007.stdout: "client_info": { 2024-09-18T08:05:32.276 INFO:teuthology.orchestra.run.smithi007.stdout: "client": "client.14805", 2024-09-18T08:05:32.276 INFO:teuthology.orchestra.run.smithi007.stdout: "client_addr": "172.21.15.7:0/640320436", 2024-09-18T08:05:32.276 INFO:teuthology.orchestra.run.smithi007.stdout: "tid": 114083 2024-09-18T08:05:32.276 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:05:32.276 INFO:teuthology.orchestra.run.smithi007.stdout: "events": [ 2024-09-18T08:05:32.276 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:05:32.276 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "initiated", 2024-09-18T08:05:32.276 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:05:31.293212+0000", 2024-09-18T08:05:32.276 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-09-18T08:05:32.276 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:05:32.277 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:05:32.277 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "header_read", 2024-09-18T08:05:32.277 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:05:31.293212+0000", 2024-09-18T08:05:32.277 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-09-18T08:05:32.277 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:05:32.277 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:05:32.277 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "throttled", 2024-09-18T08:05:32.277 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:05:31.293213+0000", 2024-09-18T08:05:32.277 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 6.9800000000000003e-07 2024-09-18T08:05:32.277 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:05:32.278 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:05:32.278 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "all_read", 2024-09-18T08:05:32.278 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:05:31.293239+0000", 2024-09-18T08:05:32.278 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 2.6284000000000001e-05 2024-09-18T08:05:32.278 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:05:32.278 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:05:32.278 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "dispatched", 2024-09-18T08:05:32.278 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:05:31.293249+0000", 2024-09-18T08:05:32.278 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 9.0699999999999996e-06 2024-09-18T08:05:32.278 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:05:32.278 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:05:32.278 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "queued_for_pg", 2024-09-18T08:05:32.278 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:05:31.293255+0000", 2024-09-18T08:05:32.279 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 6.8750000000000002e-06 2024-09-18T08:05:32.279 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:05:32.279 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:05:32.279 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-09-18T08:05:32.279 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:05:31.298416+0000", 2024-09-18T08:05:32.279 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.0051604789999999999 2024-09-18T08:05:32.279 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:05:32.279 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:05:32.279 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "started", 2024-09-18T08:05:32.279 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:05:31.298641+0000", 2024-09-18T08:05:32.280 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.000225295 2024-09-18T08:05:32.280 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:05:32.280 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:05:32.280 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "waiting for subops from 3,7", 2024-09-18T08:05:32.280 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:05:31.298786+0000", 2024-09-18T08:05:32.280 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.00014512100000000001 2024-09-18T08:05:32.281 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:05:32.281 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:05:32.281 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:05:32.281 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:05:31.305062+0000", 2024-09-18T08:05:32.281 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.0062756499999999998 2024-09-18T08:05:32.281 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:05:32.281 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:05:32.281 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "op_commit", 2024-09-18T08:05:32.281 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:05:31.306637+0000", 2024-09-18T08:05:32.281 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.0015751529999999999 2024-09-18T08:05:32.281 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-09-18T08:05:32.282 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-09-18T08:05:32.282 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-09-18T08:05:32.282 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:05:32.282 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:05:32.282 INFO:teuthology.orchestra.run.smithi007.stdout: "description": "osd_op(client.14805.0:114086 7.9 7:94bc93e1:::benchmark_data_smithi007_73155_object114085:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536 in=65536b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e119)", 2024-09-18T08:05:32.283 INFO:teuthology.orchestra.run.smithi007.stdout: "initiated_at": "2024-09-18T08:05:31.293267+0000", 2024-09-18T08:05:32.283 INFO:teuthology.orchestra.run.smithi007.stdout: "age": 0.014550356, 2024-09-18T08:05:32.283 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.014967703000000001, 2024-09-18T08:05:32.283 INFO:teuthology.orchestra.run.smithi007.stdout: "continuous": false, 2024-09-18T08:05:32.283 INFO:teuthology.orchestra.run.smithi007.stdout: "type_data": { 2024-09-18T08:05:32.283 INFO:teuthology.orchestra.run.smithi007.stdout: "flag_point": "waiting for sub ops", 2024-09-18T08:05:32.283 INFO:teuthology.orchestra.run.smithi007.stdout: "client_info": { 2024-09-18T08:05:32.283 INFO:teuthology.orchestra.run.smithi007.stdout: "client": "client.14805", 2024-09-18T08:05:32.283 INFO:teuthology.orchestra.run.smithi007.stdout: "client_addr": "172.21.15.7:0/640320436", 2024-09-18T08:05:32.283 INFO:teuthology.orchestra.run.smithi007.stdout: "tid": 114086 2024-09-18T08:05:32.283 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:05:32.283 INFO:teuthology.orchestra.run.smithi007.stdout: "events": [ 2024-09-18T08:05:32.283 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:05:32.284 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "initiated", 2024-09-18T08:05:32.284 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:05:31.293267+0000", 2024-09-18T08:05:32.284 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-09-18T08:05:32.284 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:05:32.284 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:05:32.284 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "header_read", 2024-09-18T08:05:32.284 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:05:31.293267+0000", 2024-09-18T08:05:32.284 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-09-18T08:05:32.284 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:05:32.284 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:05:32.285 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "throttled", 2024-09-18T08:05:32.285 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:05:31.293268+0000", 2024-09-18T08:05:32.285 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 6.75e-07 2024-09-18T08:05:32.285 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:05:32.285 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:05:32.285 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "all_read", 2024-09-18T08:05:32.285 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:05:31.293293+0000", 2024-09-18T08:05:32.285 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 2.5483e-05 2024-09-18T08:05:32.285 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:05:32.285 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:05:32.285 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "dispatched", 2024-09-18T08:05:32.285 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:05:31.293301+0000", 2024-09-18T08:05:32.285 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 7.588e-06 2024-09-18T08:05:32.286 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:05:32.286 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:05:32.286 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "queued_for_pg", 2024-09-18T08:05:32.286 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:05:31.293307+0000", 2024-09-18T08:05:32.286 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 6.3130000000000004e-06 2024-09-18T08:05:32.286 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:05:32.286 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:05:32.286 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-09-18T08:05:32.286 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:05:31.299384+0000", 2024-09-18T08:05:32.286 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.0060763609999999997 2024-09-18T08:05:32.286 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:05:32.286 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:05:32.286 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "started", 2024-09-18T08:05:32.286 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:05:31.299875+0000", 2024-09-18T08:05:32.286 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.00049125299999999998 2024-09-18T08:05:32.287 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:05:32.287 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:05:32.287 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "waiting for subops from 1,4", 2024-09-18T08:05:32.287 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:05:31.300014+0000", 2024-09-18T08:05:32.287 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.00013916300000000001 2024-09-18T08:05:32.287 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:05:32.287 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:05:32.287 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "op_commit", 2024-09-18T08:05:32.287 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:05:31.304665+0000", 2024-09-18T08:05:32.287 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.0046511540000000002 2024-09-18T08:05:32.287 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-09-18T08:05:32.287 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-09-18T08:05:32.287 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-09-18T08:05:32.287 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:05:32.288 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:05:32.288 INFO:teuthology.orchestra.run.smithi007.stdout: "description": "osd_repop(client.14805.0:114079 7.e e119/83 7:75e9acf7:::benchmark_data_smithi007_73155_object114078:head v 119'7120, mlcod=119'7117)", 2024-09-18T08:05:32.288 INFO:teuthology.orchestra.run.smithi007.stdout: "initiated_at": "2024-09-18T08:05:31.295035+0000", 2024-09-18T08:05:32.288 INFO:teuthology.orchestra.run.smithi007.stdout: "age": 0.012782731, 2024-09-18T08:05:32.288 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.013268461, 2024-09-18T08:05:32.288 INFO:teuthology.orchestra.run.smithi007.stdout: "continuous": false, 2024-09-18T08:05:32.288 INFO:teuthology.orchestra.run.smithi007.stdout: "type_data": { 2024-09-18T08:05:32.288 INFO:teuthology.orchestra.run.smithi007.stdout: "flag_point": "started", 2024-09-18T08:05:32.288 INFO:teuthology.orchestra.run.smithi007.stdout: "events": [ 2024-09-18T08:05:32.288 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:05:32.288 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "initiated", 2024-09-18T08:05:32.288 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:05:31.295035+0000", 2024-09-18T08:05:32.288 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-09-18T08:05:32.288 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:05:32.289 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:05:32.289 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "header_read", 2024-09-18T08:05:32.289 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:05:31.295035+0000", 2024-09-18T08:05:32.289 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-09-18T08:05:32.289 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:05:32.289 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:05:32.289 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "throttled", 2024-09-18T08:05:32.289 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:05:31.295036+0000", 2024-09-18T08:05:32.289 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 7.7899999999999997e-07 2024-09-18T08:05:32.289 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:05:32.289 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:05:32.289 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "all_read", 2024-09-18T08:05:32.289 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:05:31.295061+0000", 2024-09-18T08:05:32.289 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 2.5658000000000001e-05 2024-09-18T08:05:32.290 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:05:32.290 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:05:32.290 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "dispatched", 2024-09-18T08:05:32.290 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:05:31.295635+0000", 2024-09-18T08:05:32.290 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.00057329600000000005 2024-09-18T08:05:32.290 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:05:32.290 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:05:32.290 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "queued_for_pg", 2024-09-18T08:05:32.290 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:05:31.295645+0000", 2024-09-18T08:05:32.290 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.0043e-05 2024-09-18T08:05:32.290 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:05:32.290 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:05:32.290 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-09-18T08:05:32.290 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:05:31.295793+0000", 2024-09-18T08:05:32.291 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.000148715 2024-09-18T08:05:32.291 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:05:32.291 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:05:32.291 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "started", 2024-09-18T08:05:32.291 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:05:31.295813+0000", 2024-09-18T08:05:32.291 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.9253999999999999e-05 2024-09-18T08:05:32.291 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-09-18T08:05:32.291 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-09-18T08:05:32.291 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-09-18T08:05:32.291 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:05:32.291 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:05:32.291 INFO:teuthology.orchestra.run.smithi007.stdout: "description": "osd_op(client.14805.0:114092 7.4 7:2fb7af12:::benchmark_data_smithi007_73155_object114091:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536 in=65536b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e119)", 2024-09-18T08:05:32.291 INFO:teuthology.orchestra.run.smithi007.stdout: "initiated_at": "2024-09-18T08:05:31.299463+0000", 2024-09-18T08:05:32.291 INFO:teuthology.orchestra.run.smithi007.stdout: "age": 0.0083543639999999995, 2024-09-18T08:05:32.292 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.0089006929999999995, 2024-09-18T08:05:32.292 INFO:teuthology.orchestra.run.smithi007.stdout: "continuous": false, 2024-09-18T08:05:32.292 INFO:teuthology.orchestra.run.smithi007.stdout: "type_data": { 2024-09-18T08:05:32.292 INFO:teuthology.orchestra.run.smithi007.stdout: "flag_point": "waiting for sub ops", 2024-09-18T08:05:32.292 INFO:teuthology.orchestra.run.smithi007.stdout: "client_info": { 2024-09-18T08:05:32.292 INFO:teuthology.orchestra.run.smithi007.stdout: "client": "client.14805", 2024-09-18T08:05:32.292 INFO:teuthology.orchestra.run.smithi007.stdout: "client_addr": "172.21.15.7:0/640320436", 2024-09-18T08:05:32.292 INFO:teuthology.orchestra.run.smithi007.stdout: "tid": 114092 2024-09-18T08:05:32.292 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:05:32.292 INFO:teuthology.orchestra.run.smithi007.stdout: "events": [ 2024-09-18T08:05:32.292 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:05:32.292 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "initiated", 2024-09-18T08:05:32.292 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:05:31.299463+0000", 2024-09-18T08:05:32.293 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-09-18T08:05:32.293 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:05:32.293 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:05:32.293 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "header_read", 2024-09-18T08:05:32.293 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:05:31.299463+0000", 2024-09-18T08:05:32.293 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-09-18T08:05:32.293 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:05:32.293 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:05:32.293 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "throttled", 2024-09-18T08:05:32.293 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:05:31.299465+0000", 2024-09-18T08:05:32.293 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.3909999999999999e-06 2024-09-18T08:05:32.293 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:05:32.293 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:05:32.294 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "all_read", 2024-09-18T08:05:32.294 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:05:31.299485+0000", 2024-09-18T08:05:32.294 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 2.0554999999999999e-05 2024-09-18T08:05:32.294 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:05:32.294 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:05:32.294 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "dispatched", 2024-09-18T08:05:32.294 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:05:31.299496+0000", 2024-09-18T08:05:32.294 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.0645999999999999e-05 2024-09-18T08:05:32.294 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:05:32.295 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:05:32.295 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "queued_for_pg", 2024-09-18T08:05:32.295 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:05:31.299503+0000", 2024-09-18T08:05:32.295 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 6.8990000000000004e-06 2024-09-18T08:05:32.295 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:05:32.295 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:05:32.295 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-09-18T08:05:32.295 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:05:31.302938+0000", 2024-09-18T08:05:32.295 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.003435647 2024-09-18T08:05:32.295 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:05:32.295 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:05:32.295 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "started", 2024-09-18T08:05:32.295 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:05:31.303054+0000", 2024-09-18T08:05:32.296 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.000115543 2024-09-18T08:05:32.296 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:05:32.296 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:05:32.296 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "waiting for subops from 1,4", 2024-09-18T08:05:32.296 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:05:31.303191+0000", 2024-09-18T08:05:32.296 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.00013753999999999999 2024-09-18T08:05:32.296 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:05:32.296 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:05:32.296 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "op_commit", 2024-09-18T08:05:32.297 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:05:31.306623+0000", 2024-09-18T08:05:32.297 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.003431637 2024-09-18T08:05:32.297 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:05:32.297 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:05:32.297 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:05:32.297 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:05:31.307059+0000", 2024-09-18T08:05:32.298 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.00043621699999999997 2024-09-18T08:05:32.298 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-09-18T08:05:32.298 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-09-18T08:05:32.298 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-09-18T08:05:32.298 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:05:32.298 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:05:32.298 INFO:teuthology.orchestra.run.smithi007.stdout: "description": "osd_repop(client.14805.0:114081 7.e e119/83 7:7c24577d:::benchmark_data_smithi007_73155_object114080:head v 119'7121, mlcod=119'7117)", 2024-09-18T08:05:32.298 INFO:teuthology.orchestra.run.smithi007.stdout: "initiated_at": "2024-09-18T08:05:31.302671+0000", 2024-09-18T08:05:32.298 INFO:teuthology.orchestra.run.smithi007.stdout: "age": 0.0051462019999999999, 2024-09-18T08:05:32.298 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.0057692070000000002, 2024-09-18T08:05:32.299 INFO:teuthology.orchestra.run.smithi007.stdout: "continuous": false, 2024-09-18T08:05:32.299 INFO:teuthology.orchestra.run.smithi007.stdout: "type_data": { 2024-09-18T08:05:32.299 INFO:teuthology.orchestra.run.smithi007.stdout: "flag_point": "started", 2024-09-18T08:05:32.299 INFO:teuthology.orchestra.run.smithi007.stdout: "events": [ 2024-09-18T08:05:32.299 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:05:32.299 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "initiated", 2024-09-18T08:05:32.299 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:05:31.302671+0000", 2024-09-18T08:05:32.299 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-09-18T08:05:32.300 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:05:32.300 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:05:32.300 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "header_read", 2024-09-18T08:05:32.300 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:05:31.302671+0000", 2024-09-18T08:05:32.300 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-09-18T08:05:32.300 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:05:32.300 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:05:32.300 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "throttled", 2024-09-18T08:05:32.300 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:05:31.302672+0000", 2024-09-18T08:05:32.300 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.096e-06 2024-09-18T08:05:32.300 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:05:32.300 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:05:32.301 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "all_read", 2024-09-18T08:05:32.301 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:05:31.302695+0000", 2024-09-18T08:05:32.301 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 2.3003e-05 2024-09-18T08:05:32.301 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:05:32.301 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:05:32.301 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "dispatched", 2024-09-18T08:05:32.301 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:05:31.302714+0000", 2024-09-18T08:05:32.301 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.8950999999999999e-05 2024-09-18T08:05:32.301 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:05:32.301 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:05:32.301 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "queued_for_pg", 2024-09-18T08:05:32.301 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:05:31.302724+0000", 2024-09-18T08:05:32.302 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 9.1940000000000004e-06 2024-09-18T08:05:32.302 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:05:32.302 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:05:32.302 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-09-18T08:05:32.302 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:05:31.302807+0000", 2024-09-18T08:05:32.302 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 8.3104999999999998e-05 2024-09-18T08:05:32.302 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:05:32.302 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:05:32.302 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "started", 2024-09-18T08:05:32.302 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:05:31.302832+0000", 2024-09-18T08:05:32.302 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 2.5409e-05 2024-09-18T08:05:32.302 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-09-18T08:05:32.302 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-09-18T08:05:32.302 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-09-18T08:05:32.303 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-09-18T08:05:32.303 INFO:teuthology.orchestra.run.smithi007.stdout: ], 2024-09-18T08:05:32.303 INFO:teuthology.orchestra.run.smithi007.stdout: "num_ops": 9 2024-09-18T08:05:32.303 INFO:teuthology.orchestra.run.smithi007.stdout:} 2024-09-18T08:05:32.303 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1 2024-09-18T08:05:32.345 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:05:32.346 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:05:32.347 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1' 2024-09-18T08:05:32.447 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7 2024-09-18T08:05:32.477 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:05:32.477 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:05:32.477 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7' 2024-09-18T08:05:32.480 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:05:32 smithi007 ceph-mon[29459]: 3.15 scrub starts 2024-09-18T08:05:32.480 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:05:32 smithi007 ceph-mon[29459]: 3.15 scrub ok 2024-09-18T08:05:32.480 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:05:32 smithi007 ceph-mon[29459]: pgmap v116: 148 pgs: 1 active+clean+scrubbing, 147 active+clean; 6.8 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 853 B/s rd, 50 MiB/s wr, 796 op/s 2024-09-18T08:05:32.481 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:05:32 smithi007 ceph-mon[39249]: 3.15 scrub starts 2024-09-18T08:05:32.481 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:05:32 smithi007 ceph-mon[39249]: 3.15 scrub ok 2024-09-18T08:05:32.481 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:05:32 smithi007 ceph-mon[39249]: pgmap v116: 148 pgs: 1 active+clean+scrubbing, 147 active+clean; 6.8 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 853 B/s rd, 50 MiB/s wr, 796 op/s 2024-09-18T08:05:32.578 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4 2024-09-18T08:05:32.592 INFO:tasks.radosbench.radosbench.0.smithi007.stdout: 87 16 115408 115392 82.8724 65.3125 0.00811215 0.0120622 2024-09-18T08:05:32.607 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:05:32.607 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:05:32.608 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4' 2024-09-18T08:05:32.656 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:05:32 smithi050 ceph-mon[32371]: 3.15 scrub starts 2024-09-18T08:05:32.656 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:05:32 smithi050 ceph-mon[32371]: 3.15 scrub ok 2024-09-18T08:05:32.656 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:05:32 smithi050 ceph-mon[32371]: pgmap v116: 148 pgs: 1 active+clean+scrubbing, 147 active+clean; 6.8 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 853 B/s rd, 50 MiB/s wr, 796 op/s 2024-09-18T08:05:32.709 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3 2024-09-18T08:05:32.750 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:05:32.751 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:05:32.751 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3' 2024-09-18T08:05:32.852 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1 2024-09-18T08:05:32.893 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:05:32.894 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:05:32.895 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1' 2024-09-18T08:05:32.996 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7 2024-09-18T08:05:33.024 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:05:33.026 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:05:33.026 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7' 2024-09-18T08:05:33.127 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7 2024-09-18T08:05:33.157 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:05:33.158 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:05:33.159 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7' 2024-09-18T08:05:33.226 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid 779349ca-7593-11ef-bceb-c7b262605968 -- ceph daemon osd.0 dump_blocked_ops 2024-09-18T08:05:33.259 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1 2024-09-18T08:05:33.320 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:05:33.321 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:05:33.321 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1' 2024-09-18T08:05:33.424 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6 2024-09-18T08:05:33.453 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:05:33.453 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:05:33.454 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6' 2024-09-18T08:05:33.480 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:05:33 smithi007 ceph-mon[29459]: 5.c scrub starts 2024-09-18T08:05:33.481 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:05:33 smithi007 ceph-mon[29459]: 5.c scrub ok 2024-09-18T08:05:33.481 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:05:33 smithi007 ceph-mon[29459]: from='mgr.24470 172.21.15.7:0/2594390261' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-18T08:05:33.481 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:05:33 smithi007 ceph-mon[39249]: 5.c scrub starts 2024-09-18T08:05:33.481 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:05:33 smithi007 ceph-mon[39249]: 5.c scrub ok 2024-09-18T08:05:33.481 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:05:33 smithi007 ceph-mon[39249]: from='mgr.24470 172.21.15.7:0/2594390261' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-18T08:05:33.554 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2 2024-09-18T08:05:33.570 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/779349ca-7593-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T08:05:33.593 INFO:tasks.radosbench.radosbench.0.smithi007.stdout: 88 16 116434 116418 82.659 64.125 0.0127913 0.0120918 2024-09-18T08:05:33.632 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:05:33.633 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:05:33.633 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2' 2024-09-18T08:05:33.656 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:05:33 smithi050 ceph-mon[32371]: 5.c scrub starts 2024-09-18T08:05:33.656 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:05:33 smithi050 ceph-mon[32371]: 5.c scrub ok 2024-09-18T08:05:33.656 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:05:33 smithi050 ceph-mon[32371]: from='mgr.24470 172.21.15.7:0/2594390261' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-18T08:05:33.734 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7 2024-09-18T08:05:33.762 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:05:33.762 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:05:33.763 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7' 2024-09-18T08:05:33.864 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2 2024-09-18T08:05:33.909 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:05:33.910 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:05:33.910 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2' 2024-09-18T08:05:34.011 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0 2024-09-18T08:05:34.051 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:05:34.053 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:05:34.054 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0' 2024-09-18T08:05:34.155 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7 2024-09-18T08:05:34.181 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:05:34.182 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:05:34.183 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7' 2024-09-18T08:05:34.201 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/779349ca-7593-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T08:05:34.205 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/779349ca-7593-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T08:05:34.244 INFO:teuthology.orchestra.run.smithi007.stderr:reweighted osd.4 to 1 (10000) 2024-09-18T08:05:34.284 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7 2024-09-18T08:05:34.309 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:05:34.310 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:05:34.310 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7' 2024-09-18T08:05:34.317 INFO:teuthology.orchestra.run.smithi007.stdout:osd.0: {} 2024-09-18T08:05:34.317 INFO:teuthology.orchestra.run.smithi007.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-18T08:05:34.345 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:05:34 smithi007 ceph-mon[29459]: 6.12 scrub starts 2024-09-18T08:05:34.345 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:05:34 smithi007 ceph-mon[29459]: 6.12 scrub ok 2024-09-18T08:05:34.345 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:05:34 smithi007 ceph-mon[29459]: 5.d scrub starts 2024-09-18T08:05:34.345 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:05:34 smithi007 ceph-mon[29459]: 5.d scrub ok 2024-09-18T08:05:34.345 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:05:34 smithi007 ceph-mon[29459]: pgmap v117: 148 pgs: 1 active+clean+scrubbing, 147 active+clean; 6.9 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 1.3 KiB/s rd, 44 MiB/s wr, 707 op/s 2024-09-18T08:05:34.345 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:05:34 smithi007 ceph-mon[29459]: from='client.? 172.21.15.7:0/576457400' entity='client.admin' cmd={"prefix": "osd reweight", "id": 4, "weight": 1.0} : dispatch 2024-09-18T08:05:34.346 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:05:34 smithi007 ceph-mon[29459]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 4, "weight": 1.0} : dispatch 2024-09-18T08:05:34.346 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:05:34 smithi007 ceph-mon[39249]: 6.12 scrub starts 2024-09-18T08:05:34.346 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:05:34 smithi007 ceph-mon[39249]: 6.12 scrub ok 2024-09-18T08:05:34.346 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:05:34 smithi007 ceph-mon[39249]: 5.d scrub starts 2024-09-18T08:05:34.346 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:05:34 smithi007 ceph-mon[39249]: 5.d scrub ok 2024-09-18T08:05:34.346 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:05:34 smithi007 ceph-mon[39249]: pgmap v117: 148 pgs: 1 active+clean+scrubbing, 147 active+clean; 6.9 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 1.3 KiB/s rd, 44 MiB/s wr, 707 op/s 2024-09-18T08:05:34.346 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:05:34 smithi007 ceph-mon[39249]: from='client.? 172.21.15.7:0/576457400' entity='client.admin' cmd={"prefix": "osd reweight", "id": 4, "weight": 1.0} : dispatch 2024-09-18T08:05:34.346 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:05:34 smithi007 ceph-mon[39249]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 4, "weight": 1.0} : dispatch 2024-09-18T08:05:34.357 INFO:teuthology.orchestra.run.smithi007.stdout:osd.1: {} 2024-09-18T08:05:34.357 INFO:teuthology.orchestra.run.smithi007.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-18T08:05:34.393 INFO:teuthology.orchestra.run.smithi007.stdout:osd.2: {} 2024-09-18T08:05:34.393 INFO:teuthology.orchestra.run.smithi007.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-18T08:05:34.411 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3 2024-09-18T08:05:34.434 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:05:34.435 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:05:34.436 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3' 2024-09-18T08:05:34.440 INFO:teuthology.orchestra.run.smithi007.stdout:osd.3: {} 2024-09-18T08:05:34.440 INFO:teuthology.orchestra.run.smithi007.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-18T08:05:34.467 INFO:teuthology.orchestra.run.smithi007.stdout:osd.4: {} 2024-09-18T08:05:34.467 INFO:teuthology.orchestra.run.smithi007.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-18T08:05:34.493 INFO:teuthology.orchestra.run.smithi007.stdout:osd.5: {} 2024-09-18T08:05:34.493 INFO:teuthology.orchestra.run.smithi007.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-18T08:05:34.527 INFO:teuthology.orchestra.run.smithi007.stdout:osd.6: {} 2024-09-18T08:05:34.527 INFO:teuthology.orchestra.run.smithi007.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-18T08:05:34.537 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1 2024-09-18T08:05:34.561 INFO:teuthology.orchestra.run.smithi007.stdout:osd.7: {} 2024-09-18T08:05:34.562 INFO:teuthology.orchestra.run.smithi007.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-18T08:05:34.578 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:05:34.579 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:05:34.580 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1' 2024-09-18T08:05:34.593 INFO:tasks.radosbench.radosbench.0.smithi007.stdout: 89 16 117312 117296 82.3468 54.875 0.00622564 0.0121387 2024-09-18T08:05:34.656 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:05:34 smithi050 ceph-mon[32371]: 6.12 scrub starts 2024-09-18T08:05:34.656 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:05:34 smithi050 ceph-mon[32371]: 6.12 scrub ok 2024-09-18T08:05:34.656 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:05:34 smithi050 ceph-mon[32371]: 5.d scrub starts 2024-09-18T08:05:34.656 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:05:34 smithi050 ceph-mon[32371]: 5.d scrub ok 2024-09-18T08:05:34.656 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:05:34 smithi050 ceph-mon[32371]: pgmap v117: 148 pgs: 1 active+clean+scrubbing, 147 active+clean; 6.9 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 1.3 KiB/s rd, 44 MiB/s wr, 707 op/s 2024-09-18T08:05:34.656 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:05:34 smithi050 ceph-mon[32371]: from='client.? 172.21.15.7:0/576457400' entity='client.admin' cmd={"prefix": "osd reweight", "id": 4, "weight": 1.0} : dispatch 2024-09-18T08:05:34.656 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:05:34 smithi050 ceph-mon[32371]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 4, "weight": 1.0} : dispatch 2024-09-18T08:05:34.681 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0 2024-09-18T08:05:34.716 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:05:34.717 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:05:34.717 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0' 2024-09-18T08:05:34.818 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2 2024-09-18T08:05:34.854 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:05:34.855 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:05:34.856 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2' 2024-09-18T08:05:34.956 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3 2024-09-18T08:05:34.994 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:05:34.994 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:05:34.995 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3' 2024-09-18T08:05:35.095 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1 2024-09-18T08:05:35.133 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:05:35.134 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:05:35.135 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1' 2024-09-18T08:05:35.235 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2 2024-09-18T08:05:35.274 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:05:35.275 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:05:35.275 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2' 2024-09-18T08:05:35.375 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:05:35 smithi007 ceph-mon[29459]: 4.9 scrub starts 2024-09-18T08:05:35.376 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:05:35 smithi007 ceph-mon[29459]: 4.9 scrub ok 2024-09-18T08:05:35.376 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:05:35 smithi007 ceph-mon[29459]: 5.1b scrub starts 2024-09-18T08:05:35.376 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:05:35 smithi007 ceph-mon[29459]: 5.1b scrub ok 2024-09-18T08:05:35.376 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:05:35 smithi007 ceph-mon[29459]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]': finished 2024-09-18T08:05:35.376 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:05:35 smithi007 ceph-mon[29459]: from='client.? 172.21.15.7:0/3634960675' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-18T08:05:35.376 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:05:35 smithi007 ceph-mon[29459]: osdmap e120: 8 total, 8 up, 8 in 2024-09-18T08:05:35.376 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:05:35 smithi007 ceph-mon[29459]: 3.7 scrub starts 2024-09-18T08:05:35.376 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:05:35 smithi007 ceph-mon[29459]: 3.7 scrub ok 2024-09-18T08:05:35.376 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:05:35 smithi007 ceph-mon[39249]: 4.9 scrub starts 2024-09-18T08:05:35.376 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:05:35 smithi007 ceph-mon[39249]: 4.9 scrub ok 2024-09-18T08:05:35.377 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:05:35 smithi007 ceph-mon[39249]: 5.1b scrub starts 2024-09-18T08:05:35.377 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:05:35 smithi007 ceph-mon[39249]: 5.1b scrub ok 2024-09-18T08:05:35.377 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:05:35 smithi007 ceph-mon[39249]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]': finished 2024-09-18T08:05:35.377 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:05:35 smithi007 ceph-mon[39249]: from='client.? 172.21.15.7:0/3634960675' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-18T08:05:35.377 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:05:35 smithi007 ceph-mon[39249]: osdmap e120: 8 total, 8 up, 8 in 2024-09-18T08:05:35.377 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:05:35 smithi007 ceph-mon[39249]: 3.7 scrub starts 2024-09-18T08:05:35.377 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:05:35 smithi007 ceph-mon[39249]: 3.7 scrub ok 2024-09-18T08:05:35.377 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1 2024-09-18T08:05:35.414 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:05:35.415 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:05:35.415 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1' 2024-09-18T08:05:35.516 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5 2024-09-18T08:05:35.567 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:05:35.567 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:05:35.568 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5' 2024-09-18T08:05:35.593 INFO:tasks.radosbench.radosbench.0.smithi007.stdout:Total time run: 90.0115 2024-09-18T08:05:35.593 INFO:tasks.radosbench.radosbench.0.smithi007.stdout:Total writes made: 118288 2024-09-18T08:05:35.593 INFO:tasks.radosbench.radosbench.0.smithi007.stdout:Write size: 65536 2024-09-18T08:05:35.593 INFO:tasks.radosbench.radosbench.0.smithi007.stdout:Object size: 65536 2024-09-18T08:05:35.593 INFO:tasks.radosbench.radosbench.0.smithi007.stdout:Bandwidth (MB/sec): 82.1339 2024-09-18T08:05:35.593 INFO:tasks.radosbench.radosbench.0.smithi007.stdout:Stddev Bandwidth: 32.7209 2024-09-18T08:05:35.593 INFO:tasks.radosbench.radosbench.0.smithi007.stdout:Max bandwidth (MB/sec): 176.5 2024-09-18T08:05:35.594 INFO:tasks.radosbench.radosbench.0.smithi007.stdout:Min bandwidth (MB/sec): 23.5625 2024-09-18T08:05:35.594 INFO:tasks.radosbench.radosbench.0.smithi007.stdout:Average IOPS: 1314 2024-09-18T08:05:35.594 INFO:tasks.radosbench.radosbench.0.smithi007.stdout:Stddev IOPS: 523.534 2024-09-18T08:05:35.594 INFO:tasks.radosbench.radosbench.0.smithi007.stdout:Max IOPS: 2824 2024-09-18T08:05:35.594 INFO:tasks.radosbench.radosbench.0.smithi007.stdout:Min IOPS: 377 2024-09-18T08:05:35.594 INFO:tasks.radosbench.radosbench.0.smithi007.stdout:Average Latency(s): 0.012171 2024-09-18T08:05:35.594 INFO:tasks.radosbench.radosbench.0.smithi007.stdout:Stddev Latency(s): 0.0149728 2024-09-18T08:05:35.594 INFO:tasks.radosbench.radosbench.0.smithi007.stdout:Max latency(s): 1.0787 2024-09-18T08:05:35.594 INFO:tasks.radosbench.radosbench.0.smithi007.stdout:Min latency(s): 0.00117713 2024-09-18T08:05:35.596 INFO:tasks.radosbench.radosbench.0.smithi007.stdout:Cleaning up (deleting benchmark objects) 2024-09-18T08:05:35.656 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:05:35 smithi050 ceph-mon[32371]: 4.9 scrub starts 2024-09-18T08:05:35.656 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:05:35 smithi050 ceph-mon[32371]: 4.9 scrub ok 2024-09-18T08:05:35.656 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:05:35 smithi050 ceph-mon[32371]: 5.1b scrub starts 2024-09-18T08:05:35.656 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:05:35 smithi050 ceph-mon[32371]: 5.1b scrub ok 2024-09-18T08:05:35.656 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:05:35 smithi050 ceph-mon[32371]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]': finished 2024-09-18T08:05:35.656 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:05:35 smithi050 ceph-mon[32371]: from='client.? 172.21.15.7:0/3634960675' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-18T08:05:35.656 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:05:35 smithi050 ceph-mon[32371]: osdmap e120: 8 total, 8 up, 8 in 2024-09-18T08:05:35.657 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:05:35 smithi050 ceph-mon[32371]: 3.7 scrub starts 2024-09-18T08:05:35.657 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:05:35 smithi050 ceph-mon[32371]: 3.7 scrub ok 2024-09-18T08:05:35.669 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0 2024-09-18T08:05:35.709 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:05:35.710 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:05:35.710 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0' 2024-09-18T08:05:35.811 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3 2024-09-18T08:05:35.852 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:05:35.853 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:05:35.853 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3' 2024-09-18T08:05:35.955 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0 2024-09-18T08:05:35.996 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:05:35.996 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:05:35.997 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0' 2024-09-18T08:05:36.038 INFO:teuthology.orchestra.run.smithi007.stdout:ERROR: (22) Invalid argument 2024-09-18T08:05:36.038 INFO:teuthology.orchestra.run.smithi007.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-18T08:05:36.098 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1 2024-09-18T08:05:36.131 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:05:36.132 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:05:36.132 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1' 2024-09-18T08:05:36.233 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3 2024-09-18T08:05:36.278 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:05:36.279 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:05:36.279 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3' 2024-09-18T08:05:36.378 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:05:36 smithi007 ceph-mon[29459]: 6.d scrub starts 2024-09-18T08:05:36.378 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:05:36 smithi007 ceph-mon[29459]: 6.d scrub ok 2024-09-18T08:05:36.378 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:05:36 smithi007 ceph-mon[29459]: pgmap v119: 148 pgs: 148 active+clean; 7.1 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 1.2 KiB/s rd, 74 MiB/s wr, 1.19k op/s 2024-09-18T08:05:36.378 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:05:36 smithi007 ceph-mon[39249]: 6.d scrub starts 2024-09-18T08:05:36.378 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:05:36 smithi007 ceph-mon[39249]: 6.d scrub ok 2024-09-18T08:05:36.378 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:05:36 smithi007 ceph-mon[39249]: pgmap v119: 148 pgs: 148 active+clean; 7.1 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 1.2 KiB/s rd, 74 MiB/s wr, 1.19k op/s 2024-09-18T08:05:36.380 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2 2024-09-18T08:05:36.416 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:05:36.417 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:05:36.417 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2' 2024-09-18T08:05:36.518 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5 2024-09-18T08:05:36.545 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:05:36.546 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:05:36.546 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5' 2024-09-18T08:05:36.647 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2 2024-09-18T08:05:36.655 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:05:36 smithi050 ceph-mon[32371]: 6.d scrub starts 2024-09-18T08:05:36.656 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:05:36 smithi050 ceph-mon[32371]: 6.d scrub ok 2024-09-18T08:05:36.656 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:05:36 smithi050 ceph-mon[32371]: pgmap v119: 148 pgs: 148 active+clean; 7.1 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 1.2 KiB/s rd, 74 MiB/s wr, 1.19k op/s 2024-09-18T08:05:36.684 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:05:36.685 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:05:36.685 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2' 2024-09-18T08:05:36.786 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4 2024-09-18T08:05:36.814 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:05:36.815 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:05:36.815 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4' 2024-09-18T08:05:36.853 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid 779349ca-7593-11ef-bceb-c7b262605968 -- ceph osd reweight 5 1 2024-09-18T08:05:36.916 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4 2024-09-18T08:05:36.941 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:05:36.942 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:05:36.942 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4' 2024-09-18T08:05:36.980 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Sep 18 08:05:36 smithi007 ceph-779349ca-7593-11ef-bceb-c7b262605968-mgr-y[29688]: ::ffff:172.21.15.50 - - [18/Sep/2024:08:05:36] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-09-18T08:05:37.043 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5 2024-09-18T08:05:37.067 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:05:37.067 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:05:37.068 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5' 2024-09-18T08:05:37.169 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6 2024-09-18T08:05:37.195 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:05:37.196 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:05:37.196 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6' 2024-09-18T08:05:37.297 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6 2024-09-18T08:05:37.322 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:05:37.323 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:05:37.323 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6' 2024-09-18T08:05:37.424 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2 2024-09-18T08:05:37.465 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:05:37.465 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:05:37.466 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2' 2024-09-18T08:05:37.567 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0 2024-09-18T08:05:37.609 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:05:37.609 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:05:37.610 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0' 2024-09-18T08:05:37.710 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4 2024-09-18T08:05:37.738 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:05:37.739 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:05:37.739 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4' 2024-09-18T08:05:37.840 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3 2024-09-18T08:05:37.878 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:05:37.879 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:05:37.879 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3' 2024-09-18T08:05:37.980 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4 2024-09-18T08:05:38.008 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:05:38.009 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:05:38.009 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4' 2024-09-18T08:05:38.110 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3 2024-09-18T08:05:38.146 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:05:38.148 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:05:38.148 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3' 2024-09-18T08:05:38.249 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0 2024-09-18T08:05:38.297 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:05:38.298 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:05:38.298 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0' 2024-09-18T08:05:38.399 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6 2024-09-18T08:05:38.424 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:05:38.425 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:05:38.425 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6' 2024-09-18T08:05:38.448 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:05:38 smithi007 ceph-mon[39249]: 6.1c scrub starts 2024-09-18T08:05:38.448 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:05:38 smithi007 ceph-mon[39249]: 6.1c scrub ok 2024-09-18T08:05:38.448 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:05:38 smithi007 ceph-mon[39249]: 4.15 scrub starts 2024-09-18T08:05:38.448 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:05:38 smithi007 ceph-mon[39249]: 4.15 scrub ok 2024-09-18T08:05:38.448 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:05:38 smithi007 ceph-mon[39249]: pgmap v120: 148 pgs: 148 active+clean; 7.1 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 639 B/s rd, 37 MiB/s wr, 587 op/s 2024-09-18T08:05:38.448 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:05:38 smithi007 ceph-mon[39249]: 3.2 scrub starts 2024-09-18T08:05:38.448 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:05:38 smithi007 ceph-mon[39249]: 3.2 scrub ok 2024-09-18T08:05:38.449 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:05:38 smithi007 ceph-mon[29459]: 6.1c scrub starts 2024-09-18T08:05:38.449 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:05:38 smithi007 ceph-mon[29459]: 6.1c scrub ok 2024-09-18T08:05:38.449 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:05:38 smithi007 ceph-mon[29459]: 4.15 scrub starts 2024-09-18T08:05:38.449 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:05:38 smithi007 ceph-mon[29459]: 4.15 scrub ok 2024-09-18T08:05:38.449 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:05:38 smithi007 ceph-mon[29459]: pgmap v120: 148 pgs: 148 active+clean; 7.1 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 639 B/s rd, 37 MiB/s wr, 587 op/s 2024-09-18T08:05:38.449 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:05:38 smithi007 ceph-mon[29459]: 3.2 scrub starts 2024-09-18T08:05:38.449 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:05:38 smithi007 ceph-mon[29459]: 3.2 scrub ok 2024-09-18T08:05:38.526 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4 2024-09-18T08:05:38.552 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:05:38.552 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:05:38.553 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4' 2024-09-18T08:05:38.653 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6 2024-09-18T08:05:38.656 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:05:38 smithi050 ceph-mon[32371]: 6.1c scrub starts 2024-09-18T08:05:38.656 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:05:38 smithi050 ceph-mon[32371]: 6.1c scrub ok 2024-09-18T08:05:38.657 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:05:38 smithi050 ceph-mon[32371]: 4.15 scrub starts 2024-09-18T08:05:38.657 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:05:38 smithi050 ceph-mon[32371]: 4.15 scrub ok 2024-09-18T08:05:38.657 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:05:38 smithi050 ceph-mon[32371]: pgmap v120: 148 pgs: 148 active+clean; 7.1 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 639 B/s rd, 37 MiB/s wr, 587 op/s 2024-09-18T08:05:38.657 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:05:38 smithi050 ceph-mon[32371]: 3.2 scrub starts 2024-09-18T08:05:38.657 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:05:38 smithi050 ceph-mon[32371]: 3.2 scrub ok 2024-09-18T08:05:38.681 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:05:38.682 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:05:38.682 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6' 2024-09-18T08:05:38.783 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3 2024-09-18T08:05:38.821 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:05:38.822 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:05:38.822 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3' 2024-09-18T08:05:38.924 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2 2024-09-18T08:05:38.938 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid 779349ca-7593-11ef-bceb-c7b262605968 -- ceph daemon osd.0 dump_historic_ops 2024-09-18T08:05:38.970 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/779349ca-7593-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T08:05:38.984 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:05:38.985 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:05:38.985 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2' 2024-09-18T08:05:39.086 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1 2024-09-18T08:05:39.153 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:05:39.153 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:05:39.154 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1' 2024-09-18T08:05:39.155 INFO:journalctl@ceph.iscsi.iscsi.a.smithi050.stdout:Sep 18 08:05:38 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-iscsi-iscsi-a[57782]: debug there is no tcmu-runner data available 2024-09-18T08:05:39.254 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0 2024-09-18T08:05:39.303 INFO:teuthology.orchestra.run.smithi007.stderr:nodeep-scrub is unset 2024-09-18T08:05:39.308 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:05:39.308 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:05:39.309 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0' 2024-09-18T08:05:39.410 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2 2024-09-18T08:05:39.451 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:05:39.452 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:05:39.452 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2' 2024-09-18T08:05:39.480 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:05:39 smithi007 ceph-mon[29459]: from='client.? 172.21.15.7:0/1487519869' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-18T08:05:39.481 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:05:39 smithi007 ceph-mon[39249]: from='client.? 172.21.15.7:0/1487519869' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-18T08:05:39.553 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4 2024-09-18T08:05:39.582 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:05:39.583 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:05:39.583 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4' 2024-09-18T08:05:39.656 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:05:39 smithi050 ceph-mon[32371]: from='client.? 172.21.15.7:0/1487519869' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-18T08:05:39.685 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2 2024-09-18T08:05:39.729 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:05:39.730 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:05:39.730 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2' 2024-09-18T08:05:39.831 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2 2024-09-18T08:05:39.884 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:05:39.884 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:05:39.885 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2' 2024-09-18T08:05:39.985 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5 2024-09-18T08:05:40.010 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:05:40.011 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:05:40.011 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5' 2024-09-18T08:05:40.088 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid 779349ca-7593-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-18T08:05:40.112 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7 2024-09-18T08:05:40.136 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:05:40.137 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:05:40.137 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7' 2024-09-18T08:05:40.238 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5 2024-09-18T08:05:40.264 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:05:40.265 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:05:40.265 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5' 2024-09-18T08:05:40.305 INFO:teuthology.orchestra.run.smithi007.stderr:reweighted osd.5 to 1 (10000) 2024-09-18T08:05:40.366 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6 2024-09-18T08:05:40.391 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:05:40.391 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:05:40.392 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6' 2024-09-18T08:05:40.392 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:05:40 smithi007 ceph-mon[29459]: 4.6 scrub starts 2024-09-18T08:05:40.393 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:05:40 smithi007 ceph-mon[29459]: 4.6 scrub ok 2024-09-18T08:05:40.393 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:05:40 smithi007 ceph-mon[29459]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-18T08:05:40.393 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:05:40 smithi007 ceph-mon[29459]: 5.1a scrub starts 2024-09-18T08:05:40.393 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:05:40 smithi007 ceph-mon[29459]: 5.1a scrub ok 2024-09-18T08:05:40.393 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:05:40 smithi007 ceph-mon[29459]: pgmap v121: 148 pgs: 148 active+clean; 7.0 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 1.1 KiB/s rd, 44 MiB/s wr, 995 op/s 2024-09-18T08:05:40.393 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:05:40 smithi007 ceph-mon[29459]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-18T08:05:40.393 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:05:40 smithi007 ceph-mon[29459]: Cluster is now healthy 2024-09-18T08:05:40.393 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:05:40 smithi007 ceph-mon[29459]: from='client.? 172.21.15.7:0/1487519869' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-18T08:05:40.393 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:05:40 smithi007 ceph-mon[29459]: osdmap e121: 8 total, 8 up, 8 in 2024-09-18T08:05:40.393 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:05:40 smithi007 ceph-mon[29459]: from='client.? 172.21.15.7:0/2707111498' entity='client.admin' cmd={"prefix": "osd reweight", "id": 5, "weight": 1.0} : dispatch 2024-09-18T08:05:40.394 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:05:40 smithi007 ceph-mon[39249]: 4.6 scrub starts 2024-09-18T08:05:40.394 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:05:40 smithi007 ceph-mon[39249]: 4.6 scrub ok 2024-09-18T08:05:40.394 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:05:40 smithi007 ceph-mon[39249]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-18T08:05:40.394 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:05:40 smithi007 ceph-mon[39249]: 5.1a scrub starts 2024-09-18T08:05:40.394 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:05:40 smithi007 ceph-mon[39249]: 5.1a scrub ok 2024-09-18T08:05:40.394 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:05:40 smithi007 ceph-mon[39249]: pgmap v121: 148 pgs: 148 active+clean; 7.0 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 1.1 KiB/s rd, 44 MiB/s wr, 995 op/s 2024-09-18T08:05:40.394 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:05:40 smithi007 ceph-mon[39249]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-18T08:05:40.394 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:05:40 smithi007 ceph-mon[39249]: Cluster is now healthy 2024-09-18T08:05:40.394 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:05:40 smithi007 ceph-mon[39249]: from='client.? 172.21.15.7:0/1487519869' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-18T08:05:40.394 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:05:40 smithi007 ceph-mon[39249]: osdmap e121: 8 total, 8 up, 8 in 2024-09-18T08:05:40.394 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:05:40 smithi007 ceph-mon[39249]: from='client.? 172.21.15.7:0/2707111498' entity='client.admin' cmd={"prefix": "osd reweight", "id": 5, "weight": 1.0} : dispatch 2024-09-18T08:05:40.493 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4 2024-09-18T08:05:40.520 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:05:40.521 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:05:40.521 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4' 2024-09-18T08:05:40.622 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7 2024-09-18T08:05:40.648 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:05:40.649 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:05:40.649 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7' 2024-09-18T08:05:40.656 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:05:40 smithi050 ceph-mon[32371]: 4.6 scrub starts 2024-09-18T08:05:40.656 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:05:40 smithi050 ceph-mon[32371]: 4.6 scrub ok 2024-09-18T08:05:40.656 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:05:40 smithi050 ceph-mon[32371]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-18T08:05:40.656 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:05:40 smithi050 ceph-mon[32371]: 5.1a scrub starts 2024-09-18T08:05:40.656 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:05:40 smithi050 ceph-mon[32371]: 5.1a scrub ok 2024-09-18T08:05:40.656 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:05:40 smithi050 ceph-mon[32371]: pgmap v121: 148 pgs: 148 active+clean; 7.0 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 1.1 KiB/s rd, 44 MiB/s wr, 995 op/s 2024-09-18T08:05:40.656 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:05:40 smithi050 ceph-mon[32371]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-18T08:05:40.656 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:05:40 smithi050 ceph-mon[32371]: Cluster is now healthy 2024-09-18T08:05:40.656 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:05:40 smithi050 ceph-mon[32371]: from='client.? 172.21.15.7:0/1487519869' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-18T08:05:40.656 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:05:40 smithi050 ceph-mon[32371]: osdmap e121: 8 total, 8 up, 8 in 2024-09-18T08:05:40.656 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:05:40 smithi050 ceph-mon[32371]: from='client.? 172.21.15.7:0/2707111498' entity='client.admin' cmd={"prefix": "osd reweight", "id": 5, "weight": 1.0} : dispatch 2024-09-18T08:05:40.750 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5 2024-09-18T08:05:40.776 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:05:40.777 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:05:40.777 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5' 2024-09-18T08:05:40.878 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6 2024-09-18T08:05:40.906 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:05:40.907 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:05:40.907 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6' 2024-09-18T08:05:41.008 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0 2024-09-18T08:05:41.046 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:05:41.047 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:05:41.047 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0' 2024-09-18T08:05:41.148 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5 2024-09-18T08:05:41.150 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid 779349ca-7593-11ef-bceb-c7b262605968 -- ceph osd reweight 6 1 2024-09-18T08:05:41.174 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:05:41.175 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:05:41.175 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5' 2024-09-18T08:05:41.185 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/779349ca-7593-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T08:05:41.246 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/779349ca-7593-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T08:05:41.275 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2 2024-09-18T08:05:41.335 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:05:41.335 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:05:41.335 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2' 2024-09-18T08:05:41.371 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:05:41 smithi007 ceph-mon[39249]: 3.3 scrub starts 2024-09-18T08:05:41.372 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:05:41 smithi007 ceph-mon[39249]: 3.3 scrub ok 2024-09-18T08:05:41.372 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:05:41 smithi007 ceph-mon[39249]: 4.2 scrub starts 2024-09-18T08:05:41.372 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:05:41 smithi007 ceph-mon[39249]: 4.2 scrub ok 2024-09-18T08:05:41.372 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:05:41 smithi007 ceph-mon[39249]: 6.19 scrub starts 2024-09-18T08:05:41.372 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:05:41 smithi007 ceph-mon[39249]: 6.19 scrub ok 2024-09-18T08:05:41.372 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:05:41 smithi007 ceph-mon[39249]: 3.e scrub starts 2024-09-18T08:05:41.372 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:05:41 smithi007 ceph-mon[39249]: 3.e scrub ok 2024-09-18T08:05:41.372 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:05:41 smithi007 ceph-mon[39249]: from='client.? 172.21.15.7:0/2707111498' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]': finished 2024-09-18T08:05:41.372 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:05:41 smithi007 ceph-mon[39249]: osdmap e122: 8 total, 8 up, 8 in 2024-09-18T08:05:41.372 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:05:41 smithi007 ceph-mon[39249]: 6.1e scrub starts 2024-09-18T08:05:41.372 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:05:41 smithi007 ceph-mon[39249]: 6.1e scrub ok 2024-09-18T08:05:41.373 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:05:41 smithi007 ceph-mon[29459]: 3.3 scrub starts 2024-09-18T08:05:41.373 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:05:41 smithi007 ceph-mon[29459]: 3.3 scrub ok 2024-09-18T08:05:41.373 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:05:41 smithi007 ceph-mon[29459]: 4.2 scrub starts 2024-09-18T08:05:41.373 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:05:41 smithi007 ceph-mon[29459]: 4.2 scrub ok 2024-09-18T08:05:41.373 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:05:41 smithi007 ceph-mon[29459]: 6.19 scrub starts 2024-09-18T08:05:41.373 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:05:41 smithi007 ceph-mon[29459]: 6.19 scrub ok 2024-09-18T08:05:41.373 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:05:41 smithi007 ceph-mon[29459]: 3.e scrub starts 2024-09-18T08:05:41.373 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:05:41 smithi007 ceph-mon[29459]: 3.e scrub ok 2024-09-18T08:05:41.373 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:05:41 smithi007 ceph-mon[29459]: from='client.? 172.21.15.7:0/2707111498' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]': finished 2024-09-18T08:05:41.373 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:05:41 smithi007 ceph-mon[29459]: osdmap e122: 8 total, 8 up, 8 in 2024-09-18T08:05:41.373 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:05:41 smithi007 ceph-mon[29459]: 6.1e scrub starts 2024-09-18T08:05:41.374 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:05:41 smithi007 ceph-mon[29459]: 6.1e scrub ok 2024-09-18T08:05:41.436 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1 2024-09-18T08:05:41.498 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:05:41.499 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:05:41.499 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1' 2024-09-18T08:05:41.600 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1 2024-09-18T08:05:41.623 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:05:41.623 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:05:41.623 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1' 2024-09-18T08:05:41.656 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:05:41 smithi050 ceph-mon[32371]: 3.3 scrub starts 2024-09-18T08:05:41.656 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:05:41 smithi050 ceph-mon[32371]: 3.3 scrub ok 2024-09-18T08:05:41.656 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:05:41 smithi050 ceph-mon[32371]: 4.2 scrub starts 2024-09-18T08:05:41.656 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:05:41 smithi050 ceph-mon[32371]: 4.2 scrub ok 2024-09-18T08:05:41.657 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:05:41 smithi050 ceph-mon[32371]: 6.19 scrub starts 2024-09-18T08:05:41.657 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:05:41 smithi050 ceph-mon[32371]: 6.19 scrub ok 2024-09-18T08:05:41.657 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:05:41 smithi050 ceph-mon[32371]: 3.e scrub starts 2024-09-18T08:05:41.657 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:05:41 smithi050 ceph-mon[32371]: 3.e scrub ok 2024-09-18T08:05:41.657 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:05:41 smithi050 ceph-mon[32371]: from='client.? 172.21.15.7:0/2707111498' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]': finished 2024-09-18T08:05:41.657 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:05:41 smithi050 ceph-mon[32371]: osdmap e122: 8 total, 8 up, 8 in 2024-09-18T08:05:41.657 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:05:41 smithi050 ceph-mon[32371]: 6.1e scrub starts 2024-09-18T08:05:41.658 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:05:41 smithi050 ceph-mon[32371]: 6.1e scrub ok 2024-09-18T08:05:41.724 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0 2024-09-18T08:05:41.761 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:05:41.762 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:05:41.762 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0' 2024-09-18T08:05:41.863 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4 2024-09-18T08:05:41.888 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:05:41.889 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:05:41.889 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4' 2024-09-18T08:05:41.990 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6 2024-09-18T08:05:42.019 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:05:42.019 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:05:42.020 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6' 2024-09-18T08:05:42.120 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6 2024-09-18T08:05:42.149 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:05:42.150 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:05:42.151 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6' 2024-09-18T08:05:42.252 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5 2024-09-18T08:05:42.273 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:05:42.274 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:05:42.274 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5' 2024-09-18T08:05:42.375 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2 2024-09-18T08:05:42.419 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:05:42.420 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:05:42.420 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2' 2024-09-18T08:05:42.437 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid 779349ca-7593-11ef-bceb-c7b262605968 -- ceph osd set noscrub 2024-09-18T08:05:42.521 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7 2024-09-18T08:05:42.546 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:05:42.547 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:05:42.547 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7' 2024-09-18T08:05:42.648 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4 2024-09-18T08:05:42.655 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:05:42 smithi050 ceph-mon[32371]: 5.1d scrub starts 2024-09-18T08:05:42.656 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:05:42 smithi050 ceph-mon[32371]: 5.1d scrub ok 2024-09-18T08:05:42.656 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:05:42 smithi050 ceph-mon[32371]: 6.5 scrub starts 2024-09-18T08:05:42.656 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:05:42 smithi050 ceph-mon[32371]: 6.5 scrub ok 2024-09-18T08:05:42.656 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:05:42 smithi050 ceph-mon[32371]: 5.6 scrub starts 2024-09-18T08:05:42.656 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:05:42 smithi050 ceph-mon[32371]: 5.6 scrub ok 2024-09-18T08:05:42.656 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:05:42 smithi050 ceph-mon[32371]: pgmap v124: 148 pgs: 148 active+clean; 7.0 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 893 B/s rd, 20 MiB/s wr, 749 op/s 2024-09-18T08:05:42.656 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:05:42 smithi050 ceph-mon[32371]: 4.c scrub starts 2024-09-18T08:05:42.656 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:05:42 smithi050 ceph-mon[32371]: 4.c scrub ok 2024-09-18T08:05:42.656 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:05:42 smithi050 ceph-mon[32371]: 5.17 scrub starts 2024-09-18T08:05:42.656 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:05:42 smithi050 ceph-mon[32371]: 5.17 scrub ok 2024-09-18T08:05:42.676 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:05:42.677 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:05:42.677 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4' 2024-09-18T08:05:42.731 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:05:42 smithi007 ceph-mon[29459]: 5.1d scrub starts 2024-09-18T08:05:42.731 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:05:42 smithi007 ceph-mon[29459]: 5.1d scrub ok 2024-09-18T08:05:42.731 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:05:42 smithi007 ceph-mon[29459]: 6.5 scrub starts 2024-09-18T08:05:42.731 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:05:42 smithi007 ceph-mon[29459]: 6.5 scrub ok 2024-09-18T08:05:42.731 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:05:42 smithi007 ceph-mon[29459]: 5.6 scrub starts 2024-09-18T08:05:42.731 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:05:42 smithi007 ceph-mon[29459]: 5.6 scrub ok 2024-09-18T08:05:42.731 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:05:42 smithi007 ceph-mon[29459]: pgmap v124: 148 pgs: 148 active+clean; 7.0 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 893 B/s rd, 20 MiB/s wr, 749 op/s 2024-09-18T08:05:42.731 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:05:42 smithi007 ceph-mon[29459]: 4.c scrub starts 2024-09-18T08:05:42.732 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:05:42 smithi007 ceph-mon[29459]: 4.c scrub ok 2024-09-18T08:05:42.732 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:05:42 smithi007 ceph-mon[29459]: 5.17 scrub starts 2024-09-18T08:05:42.732 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:05:42 smithi007 ceph-mon[29459]: 5.17 scrub ok 2024-09-18T08:05:42.732 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:05:42 smithi007 ceph-mon[39249]: 5.1d scrub starts 2024-09-18T08:05:42.732 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:05:42 smithi007 ceph-mon[39249]: 5.1d scrub ok 2024-09-18T08:05:42.732 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:05:42 smithi007 ceph-mon[39249]: 6.5 scrub starts 2024-09-18T08:05:42.732 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:05:42 smithi007 ceph-mon[39249]: 6.5 scrub ok 2024-09-18T08:05:42.732 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:05:42 smithi007 ceph-mon[39249]: 5.6 scrub starts 2024-09-18T08:05:42.733 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:05:42 smithi007 ceph-mon[39249]: 5.6 scrub ok 2024-09-18T08:05:42.733 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:05:42 smithi007 ceph-mon[39249]: pgmap v124: 148 pgs: 148 active+clean; 7.0 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 893 B/s rd, 20 MiB/s wr, 749 op/s 2024-09-18T08:05:42.733 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:05:42 smithi007 ceph-mon[39249]: 4.c scrub starts 2024-09-18T08:05:42.733 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:05:42 smithi007 ceph-mon[39249]: 4.c scrub ok 2024-09-18T08:05:42.733 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:05:42 smithi007 ceph-mon[39249]: 5.17 scrub starts 2024-09-18T08:05:42.733 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:05:42 smithi007 ceph-mon[39249]: 5.17 scrub ok 2024-09-18T08:05:42.778 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4 2024-09-18T08:05:42.806 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:05:42.806 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:05:42.807 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4' 2024-09-18T08:05:42.907 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4 2024-09-18T08:05:42.935 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:05:42.936 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:05:42.936 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4' 2024-09-18T08:05:43.037 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7 2024-09-18T08:05:43.064 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:05:43.065 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:05:43.065 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7' 2024-09-18T08:05:43.166 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4 2024-09-18T08:05:43.196 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:05:43.197 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:05:43.197 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4' 2024-09-18T08:05:43.294 INFO:teuthology.orchestra.run.smithi007.stdout:ERROR: (22) Invalid argument 2024-09-18T08:05:43.294 INFO:teuthology.orchestra.run.smithi007.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-18T08:05:43.298 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2 2024-09-18T08:05:43.347 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:05:43.347 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:05:43.348 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2' 2024-09-18T08:05:43.449 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0 2024-09-18T08:05:43.492 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:05:43.493 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:05:43.493 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0' 2024-09-18T08:05:43.594 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7 2024-09-18T08:05:43.625 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:05:43.626 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:05:43.626 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7' 2024-09-18T08:05:43.655 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:05:43 smithi050 ceph-mon[32371]: 5.7 scrub starts 2024-09-18T08:05:43.656 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:05:43 smithi050 ceph-mon[32371]: 5.7 scrub ok 2024-09-18T08:05:43.656 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:05:43 smithi050 ceph-mon[32371]: 7.d scrub starts 2024-09-18T08:05:43.656 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:05:43 smithi050 ceph-mon[32371]: 5.13 scrub starts 2024-09-18T08:05:43.656 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:05:43 smithi050 ceph-mon[32371]: 5.13 scrub ok 2024-09-18T08:05:43.726 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3 2024-09-18T08:05:43.730 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:05:43 smithi007 ceph-mon[29459]: 5.7 scrub starts 2024-09-18T08:05:43.730 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:05:43 smithi007 ceph-mon[29459]: 5.7 scrub ok 2024-09-18T08:05:43.730 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:05:43 smithi007 ceph-mon[29459]: 7.d scrub starts 2024-09-18T08:05:43.730 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:05:43 smithi007 ceph-mon[29459]: 5.13 scrub starts 2024-09-18T08:05:43.730 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:05:43 smithi007 ceph-mon[29459]: 5.13 scrub ok 2024-09-18T08:05:43.731 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:05:43 smithi007 ceph-mon[39249]: 5.7 scrub starts 2024-09-18T08:05:43.732 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:05:43 smithi007 ceph-mon[39249]: 5.7 scrub ok 2024-09-18T08:05:43.732 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:05:43 smithi007 ceph-mon[39249]: 7.d scrub starts 2024-09-18T08:05:43.732 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:05:43 smithi007 ceph-mon[39249]: 5.13 scrub starts 2024-09-18T08:05:43.732 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:05:43 smithi007 ceph-mon[39249]: 5.13 scrub ok 2024-09-18T08:05:43.767 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:05:43.767 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:05:43.768 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3' 2024-09-18T08:05:43.869 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0 2024-09-18T08:05:43.936 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:05:43.937 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:05:43.937 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0' 2024-09-18T08:05:44.038 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3 2024-09-18T08:05:44.074 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:05:44.075 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:05:44.076 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3' 2024-09-18T08:05:44.177 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0 2024-09-18T08:05:44.213 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:05:44.214 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:05:44.214 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0' 2024-09-18T08:05:44.315 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6 2024-09-18T08:05:44.325 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid 779349ca-7593-11ef-bceb-c7b262605968 -- ceph daemon osd.1 dump_ops_in_flight 2024-09-18T08:05:44.339 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:05:44.339 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:05:44.339 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6' 2024-09-18T08:05:44.440 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0 2024-09-18T08:05:44.485 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:05:44.486 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:05:44.486 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0' 2024-09-18T08:05:44.572 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:05:44 smithi007 ceph-mon[39249]: 5.4 scrub starts 2024-09-18T08:05:44.572 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:05:44 smithi007 ceph-mon[39249]: 5.4 scrub ok 2024-09-18T08:05:44.572 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:05:44 smithi007 ceph-mon[39249]: purged_snaps scrub starts 2024-09-18T08:05:44.572 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:05:44 smithi007 ceph-mon[39249]: purged_snaps scrub ok 2024-09-18T08:05:44.572 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:05:44 smithi007 ceph-mon[39249]: 5.a scrub starts 2024-09-18T08:05:44.572 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:05:44 smithi007 ceph-mon[39249]: 5.a scrub ok 2024-09-18T08:05:44.572 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:05:44 smithi007 ceph-mon[39249]: pgmap v125: 148 pgs: 148 active+clean; 7.0 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 1.4 KiB/s rd, 17 MiB/s wr, 757 op/s 2024-09-18T08:05:44.572 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:05:44 smithi007 ceph-mon[39249]: osdmap e123: 8 total, 8 up, 8 in 2024-09-18T08:05:44.572 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:05:44 smithi007 ceph-mon[39249]: 5.19 scrub starts 2024-09-18T08:05:44.573 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:05:44 smithi007 ceph-mon[39249]: 5.19 scrub ok 2024-09-18T08:05:44.573 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:05:44 smithi007 ceph-mon[39249]: 6.b scrub starts 2024-09-18T08:05:44.573 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:05:44 smithi007 ceph-mon[39249]: 6.b scrub ok 2024-09-18T08:05:44.573 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:05:44 smithi007 ceph-mon[29459]: 5.4 scrub starts 2024-09-18T08:05:44.573 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:05:44 smithi007 ceph-mon[29459]: 5.4 scrub ok 2024-09-18T08:05:44.573 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:05:44 smithi007 ceph-mon[29459]: purged_snaps scrub starts 2024-09-18T08:05:44.573 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:05:44 smithi007 ceph-mon[29459]: purged_snaps scrub ok 2024-09-18T08:05:44.573 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:05:44 smithi007 ceph-mon[29459]: 5.a scrub starts 2024-09-18T08:05:44.573 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:05:44 smithi007 ceph-mon[29459]: 5.a scrub ok 2024-09-18T08:05:44.573 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:05:44 smithi007 ceph-mon[29459]: pgmap v125: 148 pgs: 148 active+clean; 7.0 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 1.4 KiB/s rd, 17 MiB/s wr, 757 op/s 2024-09-18T08:05:44.574 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:05:44 smithi007 ceph-mon[29459]: osdmap e123: 8 total, 8 up, 8 in 2024-09-18T08:05:44.574 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:05:44 smithi007 ceph-mon[29459]: 5.19 scrub starts 2024-09-18T08:05:44.574 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:05:44 smithi007 ceph-mon[29459]: 5.19 scrub ok 2024-09-18T08:05:44.574 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:05:44 smithi007 ceph-mon[29459]: 6.b scrub starts 2024-09-18T08:05:44.574 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:05:44 smithi007 ceph-mon[29459]: 6.b scrub ok 2024-09-18T08:05:44.587 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3 2024-09-18T08:05:44.656 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:05:44 smithi050 ceph-mon[32371]: 5.4 scrub starts 2024-09-18T08:05:44.656 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:05:44 smithi050 ceph-mon[32371]: 5.4 scrub ok 2024-09-18T08:05:44.656 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:05:44 smithi050 ceph-mon[32371]: purged_snaps scrub starts 2024-09-18T08:05:44.656 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:05:44 smithi050 ceph-mon[32371]: purged_snaps scrub ok 2024-09-18T08:05:44.656 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:05:44 smithi050 ceph-mon[32371]: 5.a scrub starts 2024-09-18T08:05:44.656 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:05:44 smithi050 ceph-mon[32371]: 5.a scrub ok 2024-09-18T08:05:44.656 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:05:44 smithi050 ceph-mon[32371]: pgmap v125: 148 pgs: 148 active+clean; 7.0 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 1.4 KiB/s rd, 17 MiB/s wr, 757 op/s 2024-09-18T08:05:44.656 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:05:44 smithi050 ceph-mon[32371]: osdmap e123: 8 total, 8 up, 8 in 2024-09-18T08:05:44.657 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:05:44 smithi050 ceph-mon[32371]: 5.19 scrub starts 2024-09-18T08:05:44.657 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:05:44 smithi050 ceph-mon[32371]: 5.19 scrub ok 2024-09-18T08:05:44.657 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:05:44 smithi050 ceph-mon[32371]: 6.b scrub starts 2024-09-18T08:05:44.657 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:05:44 smithi050 ceph-mon[32371]: 6.b scrub ok 2024-09-18T08:05:44.659 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:05:44.659 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:05:44.660 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3' 2024-09-18T08:05:44.671 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/779349ca-7593-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T08:05:44.697 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/779349ca-7593-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T08:05:44.701 INFO:teuthology.orchestra.run.smithi007.stdout:osd.0: {} 2024-09-18T08:05:44.701 INFO:teuthology.orchestra.run.smithi007.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-18T08:05:44.748 INFO:teuthology.orchestra.run.smithi007.stdout:osd.1: {} 2024-09-18T08:05:44.749 INFO:teuthology.orchestra.run.smithi007.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-18T08:05:44.760 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4 2024-09-18T08:05:44.781 INFO:teuthology.orchestra.run.smithi007.stdout:osd.2: {} 2024-09-18T08:05:44.782 INFO:teuthology.orchestra.run.smithi007.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-18T08:05:44.787 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:05:44.788 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:05:44.789 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4' 2024-09-18T08:05:44.823 INFO:teuthology.orchestra.run.smithi007.stdout:osd.3: {} 2024-09-18T08:05:44.823 INFO:teuthology.orchestra.run.smithi007.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-18T08:05:44.857 INFO:teuthology.orchestra.run.smithi007.stdout:osd.4: {} 2024-09-18T08:05:44.857 INFO:teuthology.orchestra.run.smithi007.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-18T08:05:44.889 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0 2024-09-18T08:05:44.890 INFO:teuthology.orchestra.run.smithi007.stdout:osd.5: {} 2024-09-18T08:05:44.891 INFO:teuthology.orchestra.run.smithi007.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-18T08:05:44.924 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:05:44.924 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:05:44.925 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0' 2024-09-18T08:05:44.927 INFO:teuthology.orchestra.run.smithi007.stdout:osd.6: {} 2024-09-18T08:05:44.927 INFO:teuthology.orchestra.run.smithi007.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-18T08:05:44.957 INFO:teuthology.orchestra.run.smithi007.stdout:osd.7: {} 2024-09-18T08:05:44.957 INFO:teuthology.orchestra.run.smithi007.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-18T08:05:45.026 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1 2024-09-18T08:05:45.065 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:05:45.066 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:05:45.067 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1' 2024-09-18T08:05:45.167 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7 2024-09-18T08:05:45.193 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:05:45.194 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:05:45.194 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7' 2024-09-18T08:05:45.295 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3 2024-09-18T08:05:45.341 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:05:45.342 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:05:45.342 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3' 2024-09-18T08:05:45.443 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6 2024-09-18T08:05:45.469 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:05:45.469 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:05:45.470 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6' 2024-09-18T08:05:45.571 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5 2024-09-18T08:05:45.597 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:05:45.598 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:05:45.599 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5' 2024-09-18T08:05:45.656 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:05:45 smithi050 ceph-mon[32371]: 6.0 scrub starts 2024-09-18T08:05:45.656 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:05:45 smithi050 ceph-mon[32371]: 6.0 scrub ok 2024-09-18T08:05:45.656 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:05:45 smithi050 ceph-mon[32371]: 5.1f scrub starts 2024-09-18T08:05:45.657 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:05:45 smithi050 ceph-mon[32371]: 5.1f scrub ok 2024-09-18T08:05:45.657 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:05:45 smithi050 ceph-mon[32371]: 7.d scrub ok 2024-09-18T08:05:45.657 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:05:45 smithi050 ceph-mon[32371]: from='client.? 172.21.15.7:0/3181545267' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-18T08:05:45.699 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7 2024-09-18T08:05:45.725 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:05:45.725 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:05:45.726 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7' 2024-09-18T08:05:45.730 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:05:45 smithi007 ceph-mon[39249]: 6.0 scrub starts 2024-09-18T08:05:45.730 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:05:45 smithi007 ceph-mon[39249]: 6.0 scrub ok 2024-09-18T08:05:45.731 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:05:45 smithi007 ceph-mon[39249]: 5.1f scrub starts 2024-09-18T08:05:45.731 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:05:45 smithi007 ceph-mon[39249]: 5.1f scrub ok 2024-09-18T08:05:45.731 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:05:45 smithi007 ceph-mon[39249]: 7.d scrub ok 2024-09-18T08:05:45.731 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:05:45 smithi007 ceph-mon[39249]: from='client.? 172.21.15.7:0/3181545267' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-18T08:05:45.731 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:05:45 smithi007 ceph-mon[29459]: 6.0 scrub starts 2024-09-18T08:05:45.731 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:05:45 smithi007 ceph-mon[29459]: 6.0 scrub ok 2024-09-18T08:05:45.731 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:05:45 smithi007 ceph-mon[29459]: 5.1f scrub starts 2024-09-18T08:05:45.732 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:05:45 smithi007 ceph-mon[29459]: 5.1f scrub ok 2024-09-18T08:05:45.732 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:05:45 smithi007 ceph-mon[29459]: 7.d scrub ok 2024-09-18T08:05:45.732 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:05:45 smithi007 ceph-mon[29459]: from='client.? 172.21.15.7:0/3181545267' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-18T08:05:45.827 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6 2024-09-18T08:05:45.853 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:05:45.853 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:05:45.854 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6' 2024-09-18T08:05:45.955 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2 2024-09-18T08:05:46.001 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:05:46.002 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:05:46.002 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2' 2024-09-18T08:05:46.103 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3 2024-09-18T08:05:46.152 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:05:46.152 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:05:46.153 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3' 2024-09-18T08:05:46.254 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7 2024-09-18T08:05:46.278 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:05:46.279 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:05:46.279 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7' 2024-09-18T08:05:46.360 INFO:teuthology.orchestra.run.smithi007.stderr:reweighted osd.6 to 1 (10000) 2024-09-18T08:05:46.380 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5 2024-09-18T08:05:46.408 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:05:46.409 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:05:46.409 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5' 2024-09-18T08:05:46.510 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2 2024-09-18T08:05:46.548 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:05:46.549 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:05:46.549 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2' 2024-09-18T08:05:46.650 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2 2024-09-18T08:05:46.656 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:05:46 smithi050 ceph-mon[32371]: 2.2 scrub starts 2024-09-18T08:05:46.656 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:05:46 smithi050 ceph-mon[32371]: 2.2 scrub ok 2024-09-18T08:05:46.656 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:05:46 smithi050 ceph-mon[32371]: 7.e scrub starts 2024-09-18T08:05:46.656 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:05:46 smithi050 ceph-mon[32371]: pgmap v127: 148 pgs: 1 active+clean+scrubbing, 147 active+clean; 6.7 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 852 B/s rd, 0 B/s wr, 800 op/s 2024-09-18T08:05:46.656 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:05:46 smithi050 ceph-mon[32371]: from='client.? 172.21.15.7:0/2609103595' entity='client.admin' cmd={"prefix": "osd reweight", "id": 6, "weight": 1.0} : dispatch 2024-09-18T08:05:46.656 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:05:46 smithi050 ceph-mon[32371]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 6, "weight": 1.0} : dispatch 2024-09-18T08:05:46.692 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:05:46.693 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:05:46.693 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2' 2024-09-18T08:05:46.698 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:05:46 smithi007 ceph-mon[39249]: 2.2 scrub starts 2024-09-18T08:05:46.698 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:05:46 smithi007 ceph-mon[39249]: 2.2 scrub ok 2024-09-18T08:05:46.698 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:05:46 smithi007 ceph-mon[39249]: 7.e scrub starts 2024-09-18T08:05:46.698 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:05:46 smithi007 ceph-mon[39249]: pgmap v127: 148 pgs: 1 active+clean+scrubbing, 147 active+clean; 6.7 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 852 B/s rd, 0 B/s wr, 800 op/s 2024-09-18T08:05:46.698 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:05:46 smithi007 ceph-mon[39249]: from='client.? 172.21.15.7:0/2609103595' entity='client.admin' cmd={"prefix": "osd reweight", "id": 6, "weight": 1.0} : dispatch 2024-09-18T08:05:46.698 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:05:46 smithi007 ceph-mon[39249]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 6, "weight": 1.0} : dispatch 2024-09-18T08:05:46.698 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:05:46 smithi007 ceph-mon[29459]: 2.2 scrub starts 2024-09-18T08:05:46.699 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:05:46 smithi007 ceph-mon[29459]: 2.2 scrub ok 2024-09-18T08:05:46.699 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:05:46 smithi007 ceph-mon[29459]: 7.e scrub starts 2024-09-18T08:05:46.699 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:05:46 smithi007 ceph-mon[29459]: pgmap v127: 148 pgs: 1 active+clean+scrubbing, 147 active+clean; 6.7 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 852 B/s rd, 0 B/s wr, 800 op/s 2024-09-18T08:05:46.699 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:05:46 smithi007 ceph-mon[29459]: from='client.? 172.21.15.7:0/2609103595' entity='client.admin' cmd={"prefix": "osd reweight", "id": 6, "weight": 1.0} : dispatch 2024-09-18T08:05:46.699 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:05:46 smithi007 ceph-mon[29459]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 6, "weight": 1.0} : dispatch 2024-09-18T08:05:46.774 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/779349ca-7593-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T08:05:46.794 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2 2024-09-18T08:05:46.844 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:05:46.844 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:05:46.845 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2' 2024-09-18T08:05:46.949 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5 2024-09-18T08:05:46.950 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Sep 18 08:05:46 smithi007 ceph-779349ca-7593-11ef-bceb-c7b262605968-mgr-y[29688]: ::ffff:172.21.15.50 - - [18/Sep/2024:08:05:46] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-09-18T08:05:46.975 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:05:46.976 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:05:46.976 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5' 2024-09-18T08:05:47.077 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3 2024-09-18T08:05:47.129 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:05:47.130 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:05:47.130 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3' 2024-09-18T08:05:47.231 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0 2024-09-18T08:05:47.266 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:05:47.267 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:05:47.267 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0' 2024-09-18T08:05:47.368 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0 2024-09-18T08:05:47.380 INFO:teuthology.orchestra.run.smithi007.stderr:noscrub is set 2024-09-18T08:05:47.406 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:05:47.407 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:05:47.407 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0' 2024-09-18T08:05:47.508 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2 2024-09-18T08:05:47.549 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:05:47.550 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:05:47.551 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2' 2024-09-18T08:05:47.577 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid 779349ca-7593-11ef-bceb-c7b262605968 -- ceph osd reweight 7 1 2024-09-18T08:05:47.652 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5 2024-09-18T08:05:47.653 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:05:47 smithi007 ceph-mon[39249]: 5.16 scrub starts 2024-09-18T08:05:47.654 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:05:47 smithi007 ceph-mon[39249]: 5.16 scrub ok 2024-09-18T08:05:47.654 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:05:47 smithi007 ceph-mon[39249]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]': finished 2024-09-18T08:05:47.654 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:05:47 smithi007 ceph-mon[39249]: osdmap e124: 8 total, 8 up, 8 in 2024-09-18T08:05:47.654 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:05:47 smithi007 ceph-mon[39249]: from='client.? 172.21.15.7:0/2422556861' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-18T08:05:47.654 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:05:47 smithi007 ceph-mon[39249]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-18T08:05:47.654 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:05:47 smithi007 ceph-mon[39249]: 4.1d scrub starts 2024-09-18T08:05:47.654 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:05:47 smithi007 ceph-mon[39249]: 4.1d scrub ok 2024-09-18T08:05:47.654 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:05:47 smithi007 ceph-mon[29459]: 5.16 scrub starts 2024-09-18T08:05:47.654 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:05:47 smithi007 ceph-mon[29459]: 5.16 scrub ok 2024-09-18T08:05:47.654 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:05:47 smithi007 ceph-mon[29459]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]': finished 2024-09-18T08:05:47.654 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:05:47 smithi007 ceph-mon[29459]: osdmap e124: 8 total, 8 up, 8 in 2024-09-18T08:05:47.654 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:05:47 smithi007 ceph-mon[29459]: from='client.? 172.21.15.7:0/2422556861' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-18T08:05:47.655 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:05:47 smithi007 ceph-mon[29459]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-18T08:05:47.655 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:05:47 smithi007 ceph-mon[29459]: 4.1d scrub starts 2024-09-18T08:05:47.655 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:05:47 smithi007 ceph-mon[29459]: 4.1d scrub ok 2024-09-18T08:05:47.656 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:05:47 smithi050 ceph-mon[32371]: 5.16 scrub starts 2024-09-18T08:05:47.656 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:05:47 smithi050 ceph-mon[32371]: 5.16 scrub ok 2024-09-18T08:05:47.656 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:05:47 smithi050 ceph-mon[32371]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]': finished 2024-09-18T08:05:47.656 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:05:47 smithi050 ceph-mon[32371]: osdmap e124: 8 total, 8 up, 8 in 2024-09-18T08:05:47.656 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:05:47 smithi050 ceph-mon[32371]: from='client.? 172.21.15.7:0/2422556861' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-18T08:05:47.656 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:05:47 smithi050 ceph-mon[32371]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-18T08:05:47.656 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:05:47 smithi050 ceph-mon[32371]: 4.1d scrub starts 2024-09-18T08:05:47.656 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:05:47 smithi050 ceph-mon[32371]: 4.1d scrub ok 2024-09-18T08:05:47.679 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:05:47.680 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:05:47.680 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5' 2024-09-18T08:05:47.781 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6 2024-09-18T08:05:47.807 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:05:47.808 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:05:47.809 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6' 2024-09-18T08:05:47.909 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6 2024-09-18T08:05:47.937 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:05:47.938 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:05:47.938 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6' 2024-09-18T08:05:47.971 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid 779349ca-7593-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-18T08:05:48.017 INFO:teuthology.orchestra.run.smithi007.stdout:{ 2024-09-18T08:05:48.017 INFO:teuthology.orchestra.run.smithi007.stdout: "ops": [ 2024-09-18T08:05:48.017 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:05:48.017 INFO:teuthology.orchestra.run.smithi007.stdout: "description": "osd_op(client.14805.0:130566 7.d 7:bc8b72fd:::benchmark_data_smithi007_73155_object12275:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e125)", 2024-09-18T08:05:48.017 INFO:teuthology.orchestra.run.smithi007.stdout: "initiated_at": "2024-09-18T08:05:47.989052+0000", 2024-09-18T08:05:48.017 INFO:teuthology.orchestra.run.smithi007.stdout: "age": 0.025509469, 2024-09-18T08:05:48.018 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.025546571000000001, 2024-09-18T08:05:48.018 INFO:teuthology.orchestra.run.smithi007.stdout: "continuous": false, 2024-09-18T08:05:48.018 INFO:teuthology.orchestra.run.smithi007.stdout: "type_data": { 2024-09-18T08:05:48.018 INFO:teuthology.orchestra.run.smithi007.stdout: "flag_point": "waiting for sub ops", 2024-09-18T08:05:48.018 INFO:teuthology.orchestra.run.smithi007.stdout: "client_info": { 2024-09-18T08:05:48.018 INFO:teuthology.orchestra.run.smithi007.stdout: "client": "client.14805", 2024-09-18T08:05:48.018 INFO:teuthology.orchestra.run.smithi007.stdout: "client_addr": "172.21.15.7:0/640320436", 2024-09-18T08:05:48.018 INFO:teuthology.orchestra.run.smithi007.stdout: "tid": 130566 2024-09-18T08:05:48.018 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:05:48.018 INFO:teuthology.orchestra.run.smithi007.stdout: "events": [ 2024-09-18T08:05:48.018 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:05:48.019 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "initiated", 2024-09-18T08:05:48.019 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:05:47.989052+0000", 2024-09-18T08:05:48.019 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-09-18T08:05:48.019 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:05:48.019 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:05:48.019 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "header_read", 2024-09-18T08:05:48.019 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:05:47.989052+0000", 2024-09-18T08:05:48.019 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-09-18T08:05:48.019 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:05:48.019 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:05:48.019 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "throttled", 2024-09-18T08:05:48.019 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:05:47.989055+0000", 2024-09-18T08:05:48.020 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 2.4770000000000002e-06 2024-09-18T08:05:48.020 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:05:48.020 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:05:48.020 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "all_read", 2024-09-18T08:05:48.020 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:05:47.989063+0000", 2024-09-18T08:05:48.020 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 8.2139999999999996e-06 2024-09-18T08:05:48.020 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:05:48.020 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:05:48.020 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "dispatched", 2024-09-18T08:05:48.020 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:05:47.989077+0000", 2024-09-18T08:05:48.020 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.4245e-05 2024-09-18T08:05:48.020 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:05:48.020 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:05:48.021 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "queued_for_pg", 2024-09-18T08:05:48.021 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:05:47.989087+0000", 2024-09-18T08:05:48.021 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 9.9399999999999997e-06 2024-09-18T08:05:48.021 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:05:48.021 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:05:48.021 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-09-18T08:05:48.021 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:05:47.990885+0000", 2024-09-18T08:05:48.021 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.0017976579999999999 2024-09-18T08:05:48.021 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:05:48.021 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:05:48.021 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "started", 2024-09-18T08:05:48.021 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:05:47.991015+0000", 2024-09-18T08:05:48.022 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.000130424 2024-09-18T08:05:48.022 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:05:48.022 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:05:48.022 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "waiting for subops from 0,6", 2024-09-18T08:05:48.022 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:05:47.991140+0000", 2024-09-18T08:05:48.022 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.00012475999999999999 2024-09-18T08:05:48.022 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:05:48.022 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:05:48.022 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:05:48.022 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:05:47.998319+0000", 2024-09-18T08:05:48.022 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.0071788629999999997 2024-09-18T08:05:48.022 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:05:48.023 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:05:48.023 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "op_commit", 2024-09-18T08:05:48.023 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:05:48.004732+0000", 2024-09-18T08:05:48.023 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.0064137380000000004 2024-09-18T08:05:48.023 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-09-18T08:05:48.023 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-09-18T08:05:48.023 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-09-18T08:05:48.023 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:05:48.023 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:05:48.023 INFO:teuthology.orchestra.run.smithi007.stdout: "description": "osd_op(client.14805.0:130570 7.d 7:bac70a7b:::benchmark_data_smithi007_73155_object12279:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e125)", 2024-09-18T08:05:48.023 INFO:teuthology.orchestra.run.smithi007.stdout: "initiated_at": "2024-09-18T08:05:47.993342+0000", 2024-09-18T08:05:48.023 INFO:teuthology.orchestra.run.smithi007.stdout: "age": 0.021219298000000001, 2024-09-18T08:05:48.023 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.021340810000000002, 2024-09-18T08:05:48.024 INFO:teuthology.orchestra.run.smithi007.stdout: "continuous": false, 2024-09-18T08:05:48.024 INFO:teuthology.orchestra.run.smithi007.stdout: "type_data": { 2024-09-18T08:05:48.024 INFO:teuthology.orchestra.run.smithi007.stdout: "flag_point": "waiting for sub ops", 2024-09-18T08:05:48.024 INFO:teuthology.orchestra.run.smithi007.stdout: "client_info": { 2024-09-18T08:05:48.024 INFO:teuthology.orchestra.run.smithi007.stdout: "client": "client.14805", 2024-09-18T08:05:48.024 INFO:teuthology.orchestra.run.smithi007.stdout: "client_addr": "172.21.15.7:0/640320436", 2024-09-18T08:05:48.024 INFO:teuthology.orchestra.run.smithi007.stdout: "tid": 130570 2024-09-18T08:05:48.024 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:05:48.024 INFO:teuthology.orchestra.run.smithi007.stdout: "events": [ 2024-09-18T08:05:48.024 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:05:48.024 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "initiated", 2024-09-18T08:05:48.025 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:05:47.993342+0000", 2024-09-18T08:05:48.025 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-09-18T08:05:48.025 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:05:48.025 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:05:48.025 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "header_read", 2024-09-18T08:05:48.025 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:05:47.993342+0000", 2024-09-18T08:05:48.025 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-09-18T08:05:48.025 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:05:48.025 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:05:48.025 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "throttled", 2024-09-18T08:05:48.025 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:05:47.993344+0000", 2024-09-18T08:05:48.025 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 2.2259999999999999e-06 2024-09-18T08:05:48.025 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:05:48.026 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:05:48.026 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "all_read", 2024-09-18T08:05:48.026 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:05:47.993352+0000", 2024-09-18T08:05:48.026 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 7.8679999999999999e-06 2024-09-18T08:05:48.026 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:05:48.026 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:05:48.026 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "dispatched", 2024-09-18T08:05:48.026 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:05:47.993366+0000", 2024-09-18T08:05:48.026 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.4146e-05 2024-09-18T08:05:48.026 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:05:48.026 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:05:48.026 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "queued_for_pg", 2024-09-18T08:05:48.026 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:05:47.993376+0000", 2024-09-18T08:05:48.026 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 9.5279999999999992e-06 2024-09-18T08:05:48.027 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:05:48.027 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:05:48.027 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-09-18T08:05:48.027 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:05:47.998348+0000", 2024-09-18T08:05:48.027 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.0049722500000000001 2024-09-18T08:05:48.027 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:05:48.027 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:05:48.027 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "started", 2024-09-18T08:05:48.027 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:05:47.998482+0000", 2024-09-18T08:05:48.027 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.000133786 2024-09-18T08:05:48.027 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:05:48.027 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:05:48.027 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "waiting for subops from 0,6", 2024-09-18T08:05:48.028 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:05:47.998614+0000", 2024-09-18T08:05:48.028 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.0001315 2024-09-18T08:05:48.028 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:05:48.028 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:05:48.028 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:05:48.028 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:05:48.001455+0000", 2024-09-18T08:05:48.028 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.002840955 2024-09-18T08:05:48.028 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:05:48.028 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:05:48.028 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "op_commit", 2024-09-18T08:05:48.028 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:05:48.004740+0000", 2024-09-18T08:05:48.028 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.0032852519999999998 2024-09-18T08:05:48.029 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-09-18T08:05:48.029 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-09-18T08:05:48.029 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-09-18T08:05:48.029 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:05:48.029 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:05:48.029 INFO:teuthology.orchestra.run.smithi007.stdout: "description": "osd_repop(client.14805.0:130584 7.0 e125/83 7:06f8c8b6:::benchmark_data_smithi007_73155_object12293:head v 125'8088, mlcod=125'8085)", 2024-09-18T08:05:48.029 INFO:teuthology.orchestra.run.smithi007.stdout: "initiated_at": "2024-09-18T08:05:48.011939+0000", 2024-09-18T08:05:48.029 INFO:teuthology.orchestra.run.smithi007.stdout: "age": 0.0026225469999999998, 2024-09-18T08:05:48.029 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.0028126129999999998, 2024-09-18T08:05:48.029 INFO:teuthology.orchestra.run.smithi007.stdout: "continuous": false, 2024-09-18T08:05:48.029 INFO:teuthology.orchestra.run.smithi007.stdout: "type_data": { 2024-09-18T08:05:48.029 INFO:teuthology.orchestra.run.smithi007.stdout: "flag_point": "started", 2024-09-18T08:05:48.029 INFO:teuthology.orchestra.run.smithi007.stdout: "events": [ 2024-09-18T08:05:48.030 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:05:48.030 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "initiated", 2024-09-18T08:05:48.030 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:05:48.011939+0000", 2024-09-18T08:05:48.030 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-09-18T08:05:48.030 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:05:48.030 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:05:48.030 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "header_read", 2024-09-18T08:05:48.030 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:05:48.011939+0000", 2024-09-18T08:05:48.030 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-09-18T08:05:48.030 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:05:48.030 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:05:48.030 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "throttled", 2024-09-18T08:05:48.030 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:05:48.011940+0000", 2024-09-18T08:05:48.031 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 7.3e-07 2024-09-18T08:05:48.031 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:05:48.031 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:05:48.031 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "all_read", 2024-09-18T08:05:48.031 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:05:48.011945+0000", 2024-09-18T08:05:48.031 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 5.4210000000000003e-06 2024-09-18T08:05:48.031 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:05:48.031 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:05:48.031 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "dispatched", 2024-09-18T08:05:48.031 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:05:48.011961+0000", 2024-09-18T08:05:48.031 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.5882000000000001e-05 2024-09-18T08:05:48.031 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:05:48.031 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:05:48.032 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "queued_for_pg", 2024-09-18T08:05:48.032 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:05:48.011969+0000", 2024-09-18T08:05:48.032 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 7.6079999999999998e-06 2024-09-18T08:05:48.032 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:05:48.032 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:05:48.032 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-09-18T08:05:48.032 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:05:48.012043+0000", 2024-09-18T08:05:48.032 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 7.4400000000000006e-05 2024-09-18T08:05:48.032 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:05:48.032 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:05:48.032 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "started", 2024-09-18T08:05:48.032 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:05:48.012063+0000", 2024-09-18T08:05:48.032 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.9794999999999999e-05 2024-09-18T08:05:48.033 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-09-18T08:05:48.033 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-09-18T08:05:48.033 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-09-18T08:05:48.033 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-09-18T08:05:48.033 INFO:teuthology.orchestra.run.smithi007.stdout: ], 2024-09-18T08:05:48.033 INFO:teuthology.orchestra.run.smithi007.stdout: "num_ops": 3 2024-09-18T08:05:48.033 INFO:teuthology.orchestra.run.smithi007.stdout:} 2024-09-18T08:05:48.040 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7 2024-09-18T08:05:48.065 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:05:48.066 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:05:48.066 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7' 2024-09-18T08:05:48.167 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2 2024-09-18T08:05:48.222 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:05:48.223 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:05:48.223 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2' 2024-09-18T08:05:48.324 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7 2024-09-18T08:05:48.350 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:05:48.351 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:05:48.351 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7' 2024-09-18T08:05:48.452 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6 2024-09-18T08:05:48.478 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:05:48.479 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:05:48.479 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6' 2024-09-18T08:05:48.580 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7 2024-09-18T08:05:48.608 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:05:48.608 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:05:48.609 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7' 2024-09-18T08:05:48.656 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:05:48 smithi050 ceph-mon[32371]: pgmap v129: 148 pgs: 1 active+clean+scrubbing, 147 active+clean; 6.7 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 751 B/s rd, 0 B/s wr, 706 op/s 2024-09-18T08:05:48.656 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:05:48 smithi050 ceph-mon[32371]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-18T08:05:48.656 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:05:48 smithi050 ceph-mon[32371]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-18T08:05:48.656 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:05:48 smithi050 ceph-mon[32371]: osdmap e125: 8 total, 8 up, 8 in 2024-09-18T08:05:48.656 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:05:48 smithi050 ceph-mon[32371]: from='mgr.24470 172.21.15.7:0/2594390261' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-18T08:05:48.710 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0 2024-09-18T08:05:48.730 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:05:48 smithi007 ceph-mon[29459]: pgmap v129: 148 pgs: 1 active+clean+scrubbing, 147 active+clean; 6.7 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 751 B/s rd, 0 B/s wr, 706 op/s 2024-09-18T08:05:48.730 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:05:48 smithi007 ceph-mon[29459]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-18T08:05:48.730 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:05:48 smithi007 ceph-mon[29459]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-18T08:05:48.731 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:05:48 smithi007 ceph-mon[29459]: osdmap e125: 8 total, 8 up, 8 in 2024-09-18T08:05:48.731 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:05:48 smithi007 ceph-mon[29459]: from='mgr.24470 172.21.15.7:0/2594390261' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-18T08:05:48.731 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:05:48 smithi007 ceph-mon[39249]: pgmap v129: 148 pgs: 1 active+clean+scrubbing, 147 active+clean; 6.7 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 751 B/s rd, 0 B/s wr, 706 op/s 2024-09-18T08:05:48.731 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:05:48 smithi007 ceph-mon[39249]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-18T08:05:48.731 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:05:48 smithi007 ceph-mon[39249]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-18T08:05:48.731 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:05:48 smithi007 ceph-mon[39249]: osdmap e125: 8 total, 8 up, 8 in 2024-09-18T08:05:48.731 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:05:48 smithi007 ceph-mon[39249]: from='mgr.24470 172.21.15.7:0/2594390261' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-18T08:05:48.753 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:05:48.754 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:05:48.754 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0' 2024-09-18T08:05:48.855 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2 2024-09-18T08:05:48.891 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:05:48.891 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:05:48.892 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2' 2024-09-18T08:05:48.993 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0 2024-09-18T08:05:49.036 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:05:49.037 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:05:49.037 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0' 2024-09-18T08:05:49.138 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5 2024-09-18T08:05:49.156 INFO:journalctl@ceph.iscsi.iscsi.a.smithi050.stdout:Sep 18 08:05:48 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-iscsi-iscsi-a[57782]: debug there is no tcmu-runner data available 2024-09-18T08:05:49.164 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:05:49.165 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:05:49.165 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5' 2024-09-18T08:05:49.266 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7 2024-09-18T08:05:49.291 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:05:49.291 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:05:49.292 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7' 2024-09-18T08:05:49.303 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid 779349ca-7593-11ef-bceb-c7b262605968 -- ceph daemon osd.1 dump_blocked_ops 2024-09-18T08:05:49.385 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/779349ca-7593-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T08:05:49.392 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1 2024-09-18T08:05:49.402 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/779349ca-7593-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T08:05:49.466 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:05:49.466 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:05:49.467 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1' 2024-09-18T08:05:49.553 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:05:49 smithi007 ceph-mon[29459]: 6.10 scrub starts 2024-09-18T08:05:49.553 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:05:49 smithi007 ceph-mon[39249]: 6.10 scrub starts 2024-09-18T08:05:49.568 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0 2024-09-18T08:05:49.618 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:05:49.618 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:05:49.619 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0' 2024-09-18T08:05:49.656 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:05:49 smithi050 ceph-mon[32371]: 6.10 scrub starts 2024-09-18T08:05:49.720 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2 2024-09-18T08:05:49.801 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:05:49.820 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:05:49.820 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2' 2024-09-18T08:05:49.921 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0 2024-09-18T08:05:49.960 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:05:49.961 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:05:49.961 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0' 2024-09-18T08:05:50.062 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3 2024-09-18T08:05:50.097 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:05:50.098 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:05:50.098 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3' 2024-09-18T08:05:50.199 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0 2024-09-18T08:05:50.246 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:05:50.247 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:05:50.247 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0' 2024-09-18T08:05:50.303 INFO:teuthology.orchestra.run.smithi007.stdout:osd.0: {} 2024-09-18T08:05:50.303 INFO:teuthology.orchestra.run.smithi007.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-18T08:05:50.342 INFO:teuthology.orchestra.run.smithi007.stdout:osd.1: {} 2024-09-18T08:05:50.343 INFO:teuthology.orchestra.run.smithi007.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-18T08:05:50.348 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3 2024-09-18T08:05:50.378 INFO:teuthology.orchestra.run.smithi007.stdout:osd.2: {} 2024-09-18T08:05:50.378 INFO:teuthology.orchestra.run.smithi007.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-18T08:05:50.388 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:05:50.388 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:05:50.389 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3' 2024-09-18T08:05:50.418 INFO:teuthology.orchestra.run.smithi007.stdout:osd.3: {} 2024-09-18T08:05:50.418 INFO:teuthology.orchestra.run.smithi007.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-18T08:05:50.462 INFO:teuthology.orchestra.run.smithi007.stdout:osd.4: {} 2024-09-18T08:05:50.463 INFO:teuthology.orchestra.run.smithi007.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-18T08:05:50.489 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5 2024-09-18T08:05:50.508 INFO:teuthology.orchestra.run.smithi007.stdout:osd.5: {} 2024-09-18T08:05:50.508 INFO:teuthology.orchestra.run.smithi007.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-18T08:05:50.516 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:05:50.517 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:05:50.517 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5' 2024-09-18T08:05:50.530 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid 779349ca-7593-11ef-bceb-c7b262605968 -- ceph osd set nodeep-scrub 2024-09-18T08:05:50.550 INFO:teuthology.orchestra.run.smithi007.stdout:osd.6: {} 2024-09-18T08:05:50.550 INFO:teuthology.orchestra.run.smithi007.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-18T08:05:50.593 INFO:teuthology.orchestra.run.smithi007.stdout:osd.7: {} 2024-09-18T08:05:50.593 INFO:teuthology.orchestra.run.smithi007.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-18T08:05:50.618 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6 2024-09-18T08:05:50.643 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:05:50.643 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:05:50.644 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6' 2024-09-18T08:05:50.655 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:05:50 smithi050 ceph-mon[32371]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-18T08:05:50.656 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:05:50 smithi050 ceph-mon[32371]: pgmap v131: 148 pgs: 1 active+clean+scrubbing, 147 active+clean; 6.5 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 852 B/s rd, 0 B/s wr, 1.44k op/s 2024-09-18T08:05:50.656 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:05:50 smithi050 ceph-mon[32371]: from='client.? 172.21.15.7:0/261381134' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-18T08:05:50.730 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:05:50 smithi007 ceph-mon[29459]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-18T08:05:50.730 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:05:50 smithi007 ceph-mon[29459]: pgmap v131: 148 pgs: 1 active+clean+scrubbing, 147 active+clean; 6.5 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 852 B/s rd, 0 B/s wr, 1.44k op/s 2024-09-18T08:05:50.731 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:05:50 smithi007 ceph-mon[29459]: from='client.? 172.21.15.7:0/261381134' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-18T08:05:50.731 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:05:50 smithi007 ceph-mon[39249]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-18T08:05:50.731 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:05:50 smithi007 ceph-mon[39249]: pgmap v131: 148 pgs: 1 active+clean+scrubbing, 147 active+clean; 6.5 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 852 B/s rd, 0 B/s wr, 1.44k op/s 2024-09-18T08:05:50.731 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:05:50 smithi007 ceph-mon[39249]: from='client.? 172.21.15.7:0/261381134' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-18T08:05:50.744 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5 2024-09-18T08:05:50.766 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:05:50.767 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:05:50.768 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5' 2024-09-18T08:05:50.869 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0 2024-09-18T08:05:50.910 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:05:50.911 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:05:50.911 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0' 2024-09-18T08:05:51.012 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0 2024-09-18T08:05:51.048 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:05:51.049 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:05:51.050 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0' 2024-09-18T08:05:51.150 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5 2024-09-18T08:05:51.179 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:05:51.180 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:05:51.180 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5' 2024-09-18T08:05:51.281 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3 2024-09-18T08:05:51.322 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:05:51.323 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:05:51.323 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3' 2024-09-18T08:05:51.424 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3 2024-09-18T08:05:51.425 INFO:teuthology.orchestra.run.smithi007.stderr:reweighted osd.7 to 1 (10000) 2024-09-18T08:05:51.470 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:05:51.471 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:05:51.471 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3' 2024-09-18T08:05:51.572 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5 2024-09-18T08:05:51.601 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:05:51.602 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:05:51.602 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5' 2024-09-18T08:05:51.657 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:05:51 smithi007 ceph-mon[29459]: from='client.? 172.21.15.7:0/1113049242' entity='client.admin' cmd={"prefix": "osd reweight", "id": 7, "weight": 1.0} : dispatch 2024-09-18T08:05:51.658 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:05:51 smithi007 ceph-mon[39249]: from='client.? 172.21.15.7:0/1113049242' entity='client.admin' cmd={"prefix": "osd reweight", "id": 7, "weight": 1.0} : dispatch 2024-09-18T08:05:51.703 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5 2024-09-18T08:05:51.731 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:05:51.732 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:05:51.733 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5' 2024-09-18T08:05:51.833 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7 2024-09-18T08:05:51.863 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:05:51.863 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:05:51.863 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7' 2024-09-18T08:05:51.887 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/779349ca-7593-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T08:05:51.905 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:05:51 smithi050 ceph-mon[32371]: from='client.? 172.21.15.7:0/1113049242' entity='client.admin' cmd={"prefix": "osd reweight", "id": 7, "weight": 1.0} : dispatch 2024-09-18T08:05:51.964 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0 2024-09-18T08:05:52.016 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:05:52.017 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:05:52.017 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0' 2024-09-18T08:05:52.118 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1 2024-09-18T08:05:52.154 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:05:52.154 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:05:52.155 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1' 2024-09-18T08:05:52.255 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3 2024-09-18T08:05:52.291 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:05:52.291 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:05:52.292 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3' 2024-09-18T08:05:52.360 INFO:tasks.cephadm.ceph_manager.ceph:waiting for recovery to complete 2024-09-18T08:05:52.360 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid 779349ca-7593-11ef-bceb-c7b262605968 -- ceph pg dump --format=json 2024-09-18T08:05:52.393 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6 2024-09-18T08:05:52.418 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:05:52.419 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:05:52.419 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6' 2024-09-18T08:05:52.520 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5 2024-09-18T08:05:52.548 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:05:52.549 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:05:52.550 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5' 2024-09-18T08:05:52.650 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1 2024-09-18T08:05:52.652 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:05:52 smithi007 ceph-mon[39249]: pgmap v132: 148 pgs: 1 active+clean+scrubbing, 147 active+clean; 6.5 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 657 B/s rd, 0 B/s wr, 612 op/s 2024-09-18T08:05:52.652 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:05:52 smithi007 ceph-mon[39249]: purged_snaps scrub starts 2024-09-18T08:05:52.652 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:05:52 smithi007 ceph-mon[39249]: purged_snaps scrub ok 2024-09-18T08:05:52.652 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:05:52 smithi007 ceph-mon[39249]: from='client.? 172.21.15.7:0/1113049242' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]': finished 2024-09-18T08:05:52.652 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:05:52 smithi007 ceph-mon[39249]: osdmap e126: 8 total, 8 up, 8 in 2024-09-18T08:05:52.653 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:05:52 smithi007 ceph-mon[29459]: pgmap v132: 148 pgs: 1 active+clean+scrubbing, 147 active+clean; 6.5 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 657 B/s rd, 0 B/s wr, 612 op/s 2024-09-18T08:05:52.653 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:05:52 smithi007 ceph-mon[29459]: purged_snaps scrub starts 2024-09-18T08:05:52.653 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:05:52 smithi007 ceph-mon[29459]: purged_snaps scrub ok 2024-09-18T08:05:52.653 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:05:52 smithi007 ceph-mon[29459]: from='client.? 172.21.15.7:0/1113049242' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]': finished 2024-09-18T08:05:52.653 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:05:52 smithi007 ceph-mon[29459]: osdmap e126: 8 total, 8 up, 8 in 2024-09-18T08:05:52.716 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:05:52.716 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:05:52.717 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1' 2024-09-18T08:05:52.818 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7 2024-09-18T08:05:52.845 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:05:52.845 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:05:52.846 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7' 2024-09-18T08:05:52.906 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:05:52 smithi050 ceph-mon[32371]: pgmap v132: 148 pgs: 1 active+clean+scrubbing, 147 active+clean; 6.5 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 657 B/s rd, 0 B/s wr, 612 op/s 2024-09-18T08:05:52.906 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:05:52 smithi050 ceph-mon[32371]: purged_snaps scrub starts 2024-09-18T08:05:52.906 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:05:52 smithi050 ceph-mon[32371]: purged_snaps scrub ok 2024-09-18T08:05:52.906 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:05:52 smithi050 ceph-mon[32371]: from='client.? 172.21.15.7:0/1113049242' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]': finished 2024-09-18T08:05:52.906 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:05:52 smithi050 ceph-mon[32371]: osdmap e126: 8 total, 8 up, 8 in 2024-09-18T08:05:52.944 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/779349ca-7593-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T08:05:52.946 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7 2024-09-18T08:05:52.971 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:05:52.972 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:05:52.972 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7' 2024-09-18T08:05:52.996 INFO:teuthology.orchestra.run.smithi007.stdout:ERROR: (22) Invalid argument 2024-09-18T08:05:52.996 INFO:teuthology.orchestra.run.smithi007.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-18T08:05:53.073 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3 2024-09-18T08:05:53.112 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:05:53.113 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:05:53.113 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3' 2024-09-18T08:05:53.214 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4 2024-09-18T08:05:53.240 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:05:53.241 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:05:53.241 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4' 2024-09-18T08:05:53.257 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/779349ca-7593-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T08:05:53.342 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4 2024-09-18T08:05:53.371 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:05:53.372 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:05:53.372 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4' 2024-09-18T08:05:53.473 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2 2024-09-18T08:05:53.517 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:05:53.518 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:05:53.518 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2' 2024-09-18T08:05:53.619 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5 2024-09-18T08:05:53.647 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:05:53.647 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:05:53.648 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5' 2024-09-18T08:05:53.707 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid 779349ca-7593-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-18T08:05:53.748 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0 2024-09-18T08:05:53.803 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:05:53.804 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:05:53.804 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0' 2024-09-18T08:05:53.905 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7 2024-09-18T08:05:53.931 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:05:53.932 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:05:53.933 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7' 2024-09-18T08:05:54.034 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1 2024-09-18T08:05:54.058 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid 779349ca-7593-11ef-bceb-c7b262605968 -- ceph daemon osd.1 dump_historic_ops 2024-09-18T08:05:54.082 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:05:54.083 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:05:54.083 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1' 2024-09-18T08:05:54.184 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2 2024-09-18T08:05:54.228 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:05:54.229 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:05:54.229 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2' 2024-09-18T08:05:54.330 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6 2024-09-18T08:05:54.356 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:05:54.357 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:05:54.357 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6' 2024-09-18T08:05:54.458 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3 2024-09-18T08:05:54.513 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:05:54.513 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:05:54.514 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3' 2024-09-18T08:05:54.570 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:05:54 smithi007 ceph-mon[29459]: pgmap v134: 148 pgs: 1 active+clean+scrubbing, 147 active+clean; 6.4 GiB data, 20 GiB used, 696 GiB / 715 GiB avail; 1.5 KiB/s rd, 0 B/s wr, 844 op/s 2024-09-18T08:05:54.571 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:05:54 smithi007 ceph-mon[39249]: pgmap v134: 148 pgs: 1 active+clean+scrubbing, 147 active+clean; 6.4 GiB data, 20 GiB used, 696 GiB / 715 GiB avail; 1.5 KiB/s rd, 0 B/s wr, 844 op/s 2024-09-18T08:05:54.615 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6 2024-09-18T08:05:54.643 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:05:54.644 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:05:54.644 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6' 2024-09-18T08:05:54.655 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:05:54 smithi050 ceph-mon[32371]: pgmap v134: 148 pgs: 1 active+clean+scrubbing, 147 active+clean; 6.4 GiB data, 20 GiB used, 696 GiB / 715 GiB avail; 1.5 KiB/s rd, 0 B/s wr, 844 op/s 2024-09-18T08:05:54.745 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4 2024-09-18T08:05:54.770 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:05:54.771 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:05:54.771 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4' 2024-09-18T08:05:54.872 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7 2024-09-18T08:05:54.898 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:05:54.899 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:05:54.899 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7' 2024-09-18T08:05:55.000 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4 2024-09-18T08:05:55.020 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/779349ca-7593-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T08:05:55.025 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:05:55.026 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:05:55.026 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4' 2024-09-18T08:05:55.103 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/779349ca-7593-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T08:05:55.126 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2 2024-09-18T08:05:55.180 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:05:55.180 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:05:55.181 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2' 2024-09-18T08:05:55.246 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-09-18T08:05:55.248 INFO:teuthology.orchestra.run.smithi007.stderr:dumped all 2024-09-18T08:05:55.281 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6 2024-09-18T08:05:55.308 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:05:55.309 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:05:55.309 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6' 2024-09-18T08:05:55.410 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0 2024-09-18T08:05:55.450 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:05:55.451 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:05:55.451 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0' 2024-09-18T08:05:55.455 INFO:teuthology.orchestra.run.smithi007.stderr:nodeep-scrub is set 2024-09-18T08:05:55.552 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3 2024-09-18T08:05:55.595 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:05:55.596 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:05:55.596 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3' 2024-09-18T08:05:55.697 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5 2024-09-18T08:05:55.725 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:05:55.726 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:05:55.726 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5' 2024-09-18T08:05:55.730 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:05:55 smithi007 ceph-mon[29459]: from='client.? 172.21.15.7:0/2337631247' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-18T08:05:55.730 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:05:55 smithi007 ceph-mon[39249]: from='client.? 172.21.15.7:0/2337631247' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-18T08:05:55.827 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3 2024-09-18T08:05:55.861 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:05:55.862 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:05:55.862 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3' 2024-09-18T08:05:55.905 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:05:55 smithi050 ceph-mon[32371]: from='client.? 172.21.15.7:0/2337631247' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-18T08:05:55.963 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0 2024-09-18T08:05:56.000 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:05:56.001 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:05:56.001 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0' 2024-09-18T08:05:56.102 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7 2024-09-18T08:05:56.130 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:05:56.131 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:05:56.131 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7' 2024-09-18T08:05:56.232 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0 2024-09-18T08:05:56.273 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:05:56.273 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:05:56.274 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0' 2024-09-18T08:05:56.320 INFO:teuthology.orchestra.run.smithi007.stdout:{"pg_ready":true,"pg_map":{"version":135,"stamp":"2024-09-18T08:05:55.124012+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":6588471859,"num_objects":100723,"num_object_clones":0,"num_object_copies":302169,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":100723,"num_whiteouts":0,"num_read":1025,"num_read_kb":882,"num_write":136603,"num_write_kb":7572106,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":590808,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":86908,"ondisk_log_size":86908,"up":444,"acting":444,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":444,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":20067288,"kb_used_data":19416328,"kb_used_omap":12,"kb_used_meta":650547,"kb_avail":729893928,"statfs":{"total":767960285184,"available":747411382272,"internally_reserved":0,"allocated":19882319872,"data_stored":19879104441,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12750,"internal_metadata":666160690},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3],"upper_bound":2},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-656080896,"num_objects":-10011,"num_object_clones":0,"num_object_copies":-30033,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-10011,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":10011,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.002603"},"pg_stats":[{"pgid":"3.1f","version":"0'0","reported_seq":149,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-09-18T08:05:51.431367+0000","last_change":"2024-09-18T08:04:34.226236+0000","last_active":"2024-09-18T08:05:51.431367+0000","last_peered":"2024-09-18T08:05:51.431367+0000","last_clean":"2024-09-18T08:05:51.431367+0000","last_became_active":"2024-09-18T08:02:47.877698+0000","last_became_peered":"2024-09-18T08:02:47.877698+0000","last_unstale":"2024-09-18T08:05:51.431367+0000","last_undegraded":"2024-09-18T08:05:51.431367+0000","last_fullsized":"2024-09-18T08:05:51.431367+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:04:34.226054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:46.860092+0000","last_clean_scrub_stamp":"2024-09-18T08:04:34.226054+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:05:52.203668+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"69'9","reported_seq":161,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-09-18T08:05:51.420701+0000","last_change":"2024-09-18T08:05:08.697007+0000","last_active":"2024-09-18T08:05:51.420701+0000","last_peered":"2024-09-18T08:05:51.420701+0000","last_clean":"2024-09-18T08:05:51.420701+0000","last_became_active":"2024-09-18T08:02:49.894557+0000","last_became_peered":"2024-09-18T08:02:49.894557+0000","last_unstale":"2024-09-18T08:05:51.420701+0000","last_undegraded":"2024-09-18T08:05:51.420701+0000","last_fullsized":"2024-09-18T08:05:51.420701+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'9","last_scrub_stamp":"2024-09-18T08:05:08.696796+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_clean_scrub_stamp":"2024-09-18T08:05:08.696796+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:06:09.954040+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":149,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-09-18T08:05:51.452485+0000","last_change":"2024-09-18T08:05:43.668557+0000","last_active":"2024-09-18T08:05:51.452485+0000","last_peered":"2024-09-18T08:05:51.452485+0000","last_clean":"2024-09-18T08:05:51.452485+0000","last_became_active":"2024-09-18T08:02:51.907140+0000","last_became_peered":"2024-09-18T08:02:51.907140+0000","last_unstale":"2024-09-18T08:05:51.452485+0000","last_undegraded":"2024-09-18T08:05:51.452485+0000","last_fullsized":"2024-09-18T08:05:51.452485+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:05:43.668271+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:50.886830+0000","last_clean_scrub_stamp":"2024-09-18T08:05:43.668271+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:07:02.226478+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":130,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-09-18T08:05:51.420872+0000","last_change":"2024-09-18T08:02:53.906914+0000","last_active":"2024-09-18T08:05:51.420872+0000","last_peered":"2024-09-18T08:05:51.420872+0000","last_clean":"2024-09-18T08:05:51.420872+0000","last_became_active":"2024-09-18T08:02:53.906622+0000","last_became_peered":"2024-09-18T08:02:53.906622+0000","last_unstale":"2024-09-18T08:05:51.420872+0000","last_undegraded":"2024-09-18T08:05:51.420872+0000","last_fullsized":"2024-09-18T08:05:51.420872+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_clean_scrub_stamp":"2024-09-18T08:02:52.890475+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:05:51.546018+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":137,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-09-18T08:05:51.441042+0000","last_change":"2024-09-18T08:05:03.047185+0000","last_active":"2024-09-18T08:05:51.441042+0000","last_peered":"2024-09-18T08:05:51.441042+0000","last_clean":"2024-09-18T08:05:51.441042+0000","last_became_active":"2024-09-18T08:02:54.047640+0000","last_became_peered":"2024-09-18T08:02:54.047640+0000","last_unstale":"2024-09-18T08:05:51.441042+0000","last_undegraded":"2024-09-18T08:05:51.441042+0000","last_fullsized":"2024-09-18T08:05:51.441042+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:05:03.047120+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_clean_scrub_stamp":"2024-09-18T08:05:03.047120+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:06:25.111306+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":149,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-09-18T08:05:51.443247+0000","last_change":"2024-09-18T08:04:57.047972+0000","last_active":"2024-09-18T08:05:51.443247+0000","last_peered":"2024-09-18T08:05:51.443247+0000","last_clean":"2024-09-18T08:05:51.443247+0000","last_became_active":"2024-09-18T08:02:47.872316+0000","last_became_peered":"2024-09-18T08:02:47.872316+0000","last_unstale":"2024-09-18T08:05:51.443247+0000","last_undegraded":"2024-09-18T08:05:51.443247+0000","last_fullsized":"2024-09-18T08:05:51.443247+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:04:57.047904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:46.860092+0000","last_clean_scrub_stamp":"2024-09-18T08:04:57.047904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:06:26.082005+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"69'15","reported_seq":162,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-09-18T08:05:51.440034+0000","last_change":"2024-09-18T08:02:49.887111+0000","last_active":"2024-09-18T08:05:51.440034+0000","last_peered":"2024-09-18T08:05:51.440034+0000","last_clean":"2024-09-18T08:05:51.440034+0000","last_became_active":"2024-09-18T08:02:49.886106+0000","last_became_peered":"2024-09-18T08:02:49.886106+0000","last_unstale":"2024-09-18T08:05:51.440034+0000","last_undegraded":"2024-09-18T08:05:51.440034+0000","last_fullsized":"2024-09-18T08:05:51.440034+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_clean_scrub_stamp":"2024-09-18T08:02:48.870718+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:06:51.053554+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":141,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-09-18T08:05:51.420619+0000","last_change":"2024-09-18T08:04:29.435750+0000","last_active":"2024-09-18T08:05:51.420619+0000","last_peered":"2024-09-18T08:05:51.420619+0000","last_clean":"2024-09-18T08:05:51.420619+0000","last_became_active":"2024-09-18T08:02:51.904184+0000","last_became_peered":"2024-09-18T08:02:51.904184+0000","last_unstale":"2024-09-18T08:05:51.420619+0000","last_undegraded":"2024-09-18T08:05:51.420619+0000","last_fullsized":"2024-09-18T08:05:51.420619+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:04:29.435669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:50.886830+0000","last_clean_scrub_stamp":"2024-09-18T08:04:29.435669+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:05:57.174217+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":149,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-09-18T08:05:51.439510+0000","last_change":"2024-09-18T08:03:53.821133+0000","last_active":"2024-09-18T08:05:51.439510+0000","last_peered":"2024-09-18T08:05:51.439510+0000","last_clean":"2024-09-18T08:05:51.439510+0000","last_became_active":"2024-09-18T08:02:47.875124+0000","last_became_peered":"2024-09-18T08:02:47.875124+0000","last_unstale":"2024-09-18T08:05:51.439510+0000","last_undegraded":"2024-09-18T08:05:51.439510+0000","last_fullsized":"2024-09-18T08:05:51.439510+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:03:53.821070+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:46.860092+0000","last_clean_scrub_stamp":"2024-09-18T08:03:53.821070+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:06:20.018150+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"69'9","reported_seq":153,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-09-18T08:05:51.421186+0000","last_change":"2024-09-18T08:02:49.887589+0000","last_active":"2024-09-18T08:05:51.421186+0000","last_peered":"2024-09-18T08:05:51.421186+0000","last_clean":"2024-09-18T08:05:51.421186+0000","last_became_active":"2024-09-18T08:02:49.887161+0000","last_became_peered":"2024-09-18T08:02:49.887161+0000","last_unstale":"2024-09-18T08:05:51.421186+0000","last_undegraded":"2024-09-18T08:05:51.421186+0000","last_fullsized":"2024-09-18T08:05:51.421186+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_clean_scrub_stamp":"2024-09-18T08:02:48.870718+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:05:54.567031+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":149,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-09-18T08:05:51.439050+0000","last_change":"2024-09-18T08:05:33.935844+0000","last_active":"2024-09-18T08:05:51.439050+0000","last_peered":"2024-09-18T08:05:51.439050+0000","last_clean":"2024-09-18T08:05:51.439050+0000","last_became_active":"2024-09-18T08:02:51.907296+0000","last_became_peered":"2024-09-18T08:02:51.907296+0000","last_unstale":"2024-09-18T08:05:51.439050+0000","last_undegraded":"2024-09-18T08:05:51.439050+0000","last_fullsized":"2024-09-18T08:05:51.439050+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:05:33.935798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:50.886830+0000","last_clean_scrub_stamp":"2024-09-18T08:05:33.935798+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:06:48.799006+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":138,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-09-18T08:05:51.429804+0000","last_change":"2024-09-18T08:05:01.314402+0000","last_active":"2024-09-18T08:05:51.429804+0000","last_peered":"2024-09-18T08:05:51.429804+0000","last_clean":"2024-09-18T08:05:51.429804+0000","last_became_active":"2024-09-18T08:02:53.913519+0000","last_became_peered":"2024-09-18T08:02:53.913519+0000","last_unstale":"2024-09-18T08:05:51.429804+0000","last_undegraded":"2024-09-18T08:05:51.429804+0000","last_fullsized":"2024-09-18T08:05:51.429804+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:05:01.314330+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_clean_scrub_stamp":"2024-09-18T08:05:01.314330+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:06:05.657205+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":149,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-09-18T08:05:51.439291+0000","last_change":"2024-09-18T08:04:32.849176+0000","last_active":"2024-09-18T08:05:51.439291+0000","last_peered":"2024-09-18T08:05:51.439291+0000","last_clean":"2024-09-18T08:05:51.439291+0000","last_became_active":"2024-09-18T08:02:47.875042+0000","last_became_peered":"2024-09-18T08:02:47.875042+0000","last_unstale":"2024-09-18T08:05:51.439291+0000","last_undegraded":"2024-09-18T08:05:51.439291+0000","last_fullsized":"2024-09-18T08:05:51.439291+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:04:32.849119+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:46.860092+0000","last_clean_scrub_stamp":"2024-09-18T08:04:32.849119+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:05:56.230469+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1b","version":"69'5","reported_seq":155,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-09-18T08:05:51.421615+0000","last_change":"2024-09-18T08:04:31.380754+0000","last_active":"2024-09-18T08:05:51.421615+0000","last_peered":"2024-09-18T08:05:51.421615+0000","last_clean":"2024-09-18T08:05:51.421615+0000","last_became_active":"2024-09-18T08:02:49.895344+0000","last_became_peered":"2024-09-18T08:02:49.895344+0000","last_unstale":"2024-09-18T08:05:51.421615+0000","last_undegraded":"2024-09-18T08:05:51.421615+0000","last_fullsized":"2024-09-18T08:05:51.421615+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'5","last_scrub_stamp":"2024-09-18T08:04:31.380667+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_clean_scrub_stamp":"2024-09-18T08:04:31.380667+0000","objects_scrubbed":2,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":11,"num_read_kb":7,"num_write":6,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":149,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-09-18T08:05:51.434630+0000","last_change":"2024-09-18T08:05:38.866695+0000","last_active":"2024-09-18T08:05:51.434630+0000","last_peered":"2024-09-18T08:05:51.434630+0000","last_clean":"2024-09-18T08:05:51.434630+0000","last_became_active":"2024-09-18T08:02:51.905776+0000","last_became_peered":"2024-09-18T08:02:51.905776+0000","last_unstale":"2024-09-18T08:05:51.434630+0000","last_undegraded":"2024-09-18T08:05:51.434630+0000","last_fullsized":"2024-09-18T08:05:51.434630+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:05:38.866651+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:50.886830+0000","last_clean_scrub_stamp":"2024-09-18T08:05:38.866651+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:06:53.681027+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":138,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-09-18T08:05:51.438889+0000","last_change":"2024-09-18T08:05:39.843588+0000","last_active":"2024-09-18T08:05:51.438889+0000","last_peered":"2024-09-18T08:05:51.438889+0000","last_clean":"2024-09-18T08:05:51.438889+0000","last_became_active":"2024-09-18T08:02:53.911648+0000","last_became_peered":"2024-09-18T08:02:53.911648+0000","last_unstale":"2024-09-18T08:05:51.438889+0000","last_undegraded":"2024-09-18T08:05:51.438889+0000","last_fullsized":"2024-09-18T08:05:51.438889+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:05:39.843536+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_clean_scrub_stamp":"2024-09-18T08:05:39.843536+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:06:40.362817+0000","scrub_duration":20,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":137,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-09-18T08:05:51.422761+0000","last_change":"2024-09-18T08:05:40.478025+0000","last_active":"2024-09-18T08:05:51.422761+0000","last_peered":"2024-09-18T08:05:51.422761+0000","last_clean":"2024-09-18T08:05:51.422761+0000","last_became_active":"2024-09-18T08:02:54.046776+0000","last_became_peered":"2024-09-18T08:02:54.046776+0000","last_unstale":"2024-09-18T08:05:51.422761+0000","last_undegraded":"2024-09-18T08:05:51.422761+0000","last_fullsized":"2024-09-18T08:05:51.422761+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:05:40.477965+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_clean_scrub_stamp":"2024-09-18T08:05:40.477965+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:06:54.789197+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":157,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-09-18T08:05:51.430110+0000","last_change":"2024-09-18T08:05:27.476211+0000","last_active":"2024-09-18T08:05:51.430110+0000","last_peered":"2024-09-18T08:05:51.430110+0000","last_clean":"2024-09-18T08:05:51.430110+0000","last_became_active":"2024-09-18T08:02:47.875963+0000","last_became_peered":"2024-09-18T08:02:47.875963+0000","last_unstale":"2024-09-18T08:05:51.430110+0000","last_undegraded":"2024-09-18T08:05:51.430110+0000","last_fullsized":"2024-09-18T08:05:51.430110+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:05:27.476144+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:46.860092+0000","last_clean_scrub_stamp":"2024-09-18T08:05:27.476144+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:06:42.199865+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1c","version":"69'15","reported_seq":201,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-09-18T08:05:51.433123+0000","last_change":"2024-09-18T08:05:28.084324+0000","last_active":"2024-09-18T08:05:51.433123+0000","last_peered":"2024-09-18T08:05:51.433123+0000","last_clean":"2024-09-18T08:05:51.433123+0000","last_became_active":"2024-09-18T08:04:18.147907+0000","last_became_peered":"2024-09-18T08:04:18.147907+0000","last_unstale":"2024-09-18T08:05:51.433123+0000","last_undegraded":"2024-09-18T08:05:51.433123+0000","last_fullsized":"2024-09-18T08:05:51.433123+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'15","last_scrub_stamp":"2024-09-18T08:05:28.084220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_clean_scrub_stamp":"2024-09-18T08:05:28.084220+0000","objects_scrubbed":7,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:06:39.486316+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,3],"acting":[2,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":149,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-09-18T08:05:51.443277+0000","last_change":"2024-09-18T08:05:40.779422+0000","last_active":"2024-09-18T08:05:51.443277+0000","last_peered":"2024-09-18T08:05:51.443277+0000","last_clean":"2024-09-18T08:05:51.443277+0000","last_became_active":"2024-09-18T08:02:51.908430+0000","last_became_peered":"2024-09-18T08:02:51.908430+0000","last_unstale":"2024-09-18T08:05:51.443277+0000","last_undegraded":"2024-09-18T08:05:51.443277+0000","last_fullsized":"2024-09-18T08:05:51.443277+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:05:40.779230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:50.886830+0000","last_clean_scrub_stamp":"2024-09-18T08:05:40.779230+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:07:00.025019+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":137,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-09-18T08:05:51.450102+0000","last_change":"2024-09-18T08:04:29.956037+0000","last_active":"2024-09-18T08:05:51.450102+0000","last_peered":"2024-09-18T08:05:51.450102+0000","last_clean":"2024-09-18T08:05:51.450102+0000","last_became_active":"2024-09-18T08:02:54.047162+0000","last_became_peered":"2024-09-18T08:02:54.047162+0000","last_unstale":"2024-09-18T08:05:51.450102+0000","last_undegraded":"2024-09-18T08:05:51.450102+0000","last_fullsized":"2024-09-18T08:05:51.450102+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:04:29.955822+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_clean_scrub_stamp":"2024-09-18T08:04:29.955822+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:05:51.840369+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":149,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-09-18T08:05:51.421339+0000","last_change":"2024-09-18T08:04:22.420158+0000","last_active":"2024-09-18T08:05:51.421339+0000","last_peered":"2024-09-18T08:05:51.421339+0000","last_clean":"2024-09-18T08:05:51.421339+0000","last_became_active":"2024-09-18T08:02:47.871523+0000","last_became_peered":"2024-09-18T08:02:47.871523+0000","last_unstale":"2024-09-18T08:05:51.421339+0000","last_undegraded":"2024-09-18T08:05:51.421339+0000","last_fullsized":"2024-09-18T08:05:51.421339+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:04:22.420112+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:46.860092+0000","last_clean_scrub_stamp":"2024-09-18T08:04:22.420112+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"69'12","reported_seq":168,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-09-18T08:05:51.441109+0000","last_change":"2024-09-18T08:05:47.068485+0000","last_active":"2024-09-18T08:05:51.441109+0000","last_peered":"2024-09-18T08:05:51.441109+0000","last_clean":"2024-09-18T08:05:51.441109+0000","last_became_active":"2024-09-18T08:02:49.888831+0000","last_became_peered":"2024-09-18T08:02:49.888831+0000","last_unstale":"2024-09-18T08:05:51.441109+0000","last_undegraded":"2024-09-18T08:05:51.441109+0000","last_fullsized":"2024-09-18T08:05:51.441109+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'12","last_scrub_stamp":"2024-09-18T08:05:47.068375+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_clean_scrub_stamp":"2024-09-18T08:05:47.068375+0000","objects_scrubbed":5,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:06:50.263718+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":25,"num_read_kb":16,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":141,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-09-18T08:05:51.421989+0000","last_change":"2024-09-18T08:04:27.481522+0000","last_active":"2024-09-18T08:05:51.421989+0000","last_peered":"2024-09-18T08:05:51.421989+0000","last_clean":"2024-09-18T08:05:51.421989+0000","last_became_active":"2024-09-18T08:02:51.910316+0000","last_became_peered":"2024-09-18T08:02:51.910316+0000","last_unstale":"2024-09-18T08:05:51.421989+0000","last_undegraded":"2024-09-18T08:05:51.421989+0000","last_fullsized":"2024-09-18T08:05:51.421989+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:04:27.481457+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:50.886830+0000","last_clean_scrub_stamp":"2024-09-18T08:04:27.481457+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1c","version":"69'1","reported_seq":139,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-09-18T08:05:51.434915+0000","last_change":"2024-09-18T08:05:36.776441+0000","last_active":"2024-09-18T08:05:51.434915+0000","last_peered":"2024-09-18T08:05:51.434915+0000","last_clean":"2024-09-18T08:05:51.434915+0000","last_became_active":"2024-09-18T08:02:53.910589+0000","last_became_peered":"2024-09-18T08:02:53.910589+0000","last_unstale":"2024-09-18T08:05:51.434915+0000","last_undegraded":"2024-09-18T08:05:51.434915+0000","last_fullsized":"2024-09-18T08:05:51.434915+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'1","last_scrub_stamp":"2024-09-18T08:05:36.776369+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_clean_scrub_stamp":"2024-09-18T08:05:36.776369+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:06:36.944612+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":403,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.19","version":"62'1","reported_seq":150,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-09-18T08:05:51.452612+0000","last_change":"2024-09-18T08:05:06.819830+0000","last_active":"2024-09-18T08:05:51.452612+0000","last_peered":"2024-09-18T08:05:51.452612+0000","last_clean":"2024-09-18T08:05:51.452612+0000","last_became_active":"2024-09-18T08:02:47.880998+0000","last_became_peered":"2024-09-18T08:02:47.880998+0000","last_unstale":"2024-09-18T08:05:51.452612+0000","last_undegraded":"2024-09-18T08:05:51.452612+0000","last_fullsized":"2024-09-18T08:05:51.452612+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"62'1","last_scrub_stamp":"2024-09-18T08:05:06.819539+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:46.860092+0000","last_clean_scrub_stamp":"2024-09-18T08:05:06.819539+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:06:27.260153+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1e","version":"69'10","reported_seq":152,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-09-18T08:05:51.431638+0000","last_change":"2024-09-18T08:02:49.892755+0000","last_active":"2024-09-18T08:05:51.431638+0000","last_peered":"2024-09-18T08:05:51.431638+0000","last_clean":"2024-09-18T08:05:51.431638+0000","last_became_active":"2024-09-18T08:02:49.892024+0000","last_became_peered":"2024-09-18T08:02:49.892024+0000","last_unstale":"2024-09-18T08:05:51.431638+0000","last_undegraded":"2024-09-18T08:05:51.431638+0000","last_fullsized":"2024-09-18T08:05:51.431638+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_clean_scrub_stamp":"2024-09-18T08:02:48.870718+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1f","version":"102'9","reported_seq":113,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-09-18T08:05:51.423077+0000","last_change":"2024-09-18T08:05:44.463855+0000","last_active":"2024-09-18T08:05:51.423077+0000","last_peered":"2024-09-18T08:05:51.423077+0000","last_clean":"2024-09-18T08:05:51.423077+0000","last_became_active":"2024-09-18T08:04:43.299359+0000","last_became_peered":"2024-09-18T08:04:43.299359+0000","last_unstale":"2024-09-18T08:05:51.423077+0000","last_undegraded":"2024-09-18T08:05:51.423077+0000","last_fullsized":"2024-09-18T08:05:51.423077+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"102'9","last_scrub_stamp":"2024-09-18T08:05:44.463780+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:50.886830+0000","last_clean_scrub_stamp":"2024-09-18T08:05:44.463780+0000","objects_scrubbed":1,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:07:10.059311+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,7],"acting":[4,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.c","version":"126'8650","reported_seq":8753,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-09-18T08:05:53.304090+0000","last_change":"2024-09-18T08:04:43.307818+0000","last_active":"2024-09-18T08:05:53.304090+0000","last_peered":"2024-09-18T08:05:53.304090+0000","last_clean":"2024-09-18T08:05:53.304090+0000","last_became_active":"2024-09-18T08:04:43.300310+0000","last_became_peered":"2024-09-18T08:04:43.300310+0000","last_unstale":"2024-09-18T08:05:53.304090+0000","last_undegraded":"2024-09-18T08:05:53.304090+0000","last_fullsized":"2024-09-18T08:05:53.304090+0000","mapping_epoch":101,"log_start":"100'4000","ondisk_log_start":"100'4000","created":83,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:03:56.449957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:03:56.449957+0000","last_clean_scrub_stamp":"2024-09-18T08:03:56.449957+0000","objects_scrubbed":0,"log_size":4650,"log_dups_size":0,"ondisk_log_size":4650,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":416808960,"num_objects":6360,"num_object_clones":0,"num_object_copies":19080,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6360,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8650,"num_write_kb":480320,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,7],"acting":[3,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.f","version":"69'15","reported_seq":170,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-09-18T08:05:51.442676+0000","last_change":"2024-09-18T08:04:22.741342+0000","last_active":"2024-09-18T08:05:51.442676+0000","last_peered":"2024-09-18T08:05:51.442676+0000","last_clean":"2024-09-18T08:05:51.442676+0000","last_became_active":"2024-09-18T08:02:49.899083+0000","last_became_peered":"2024-09-18T08:02:49.899083+0000","last_unstale":"2024-09-18T08:05:51.442676+0000","last_undegraded":"2024-09-18T08:05:51.442676+0000","last_fullsized":"2024-09-18T08:05:51.442676+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'15","last_scrub_stamp":"2024-09-18T08:04:22.741261+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_clean_scrub_stamp":"2024-09-18T08:04:22.741261+0000","objects_scrubbed":7,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":149,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-09-18T08:05:51.440490+0000","last_change":"2024-09-18T08:04:08.982535+0000","last_active":"2024-09-18T08:05:51.440490+0000","last_peered":"2024-09-18T08:05:51.440490+0000","last_clean":"2024-09-18T08:05:51.440490+0000","last_became_active":"2024-09-18T08:02:47.879328+0000","last_became_peered":"2024-09-18T08:02:47.879328+0000","last_unstale":"2024-09-18T08:05:51.440490+0000","last_undegraded":"2024-09-18T08:05:51.440490+0000","last_fullsized":"2024-09-18T08:05:51.440490+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:04:08.982469+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:46.860092+0000","last_clean_scrub_stamp":"2024-09-18T08:04:08.982469+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.e","version":"69'8","reported_seq":187,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-09-18T08:05:51.423167+0000","last_change":"2024-09-18T08:03:51.342453+0000","last_active":"2024-09-18T08:05:51.423167+0000","last_peered":"2024-09-18T08:05:51.423167+0000","last_clean":"2024-09-18T08:05:51.423167+0000","last_became_active":"2024-09-18T08:02:51.903369+0000","last_became_peered":"2024-09-18T08:02:51.903369+0000","last_unstale":"2024-09-18T08:05:51.423167+0000","last_undegraded":"2024-09-18T08:05:51.423167+0000","last_fullsized":"2024-09-18T08:05:51.423167+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'8","last_scrub_stamp":"2024-09-18T08:03:51.342402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:50.886830+0000","last_clean_scrub_stamp":"2024-09-18T08:03:51.342402+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:06:18.565571+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":146,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-09-18T08:05:51.439328+0000","last_change":"2024-09-18T08:05:34.905147+0000","last_active":"2024-09-18T08:05:51.439328+0000","last_peered":"2024-09-18T08:05:51.439328+0000","last_clean":"2024-09-18T08:05:51.439328+0000","last_became_active":"2024-09-18T08:02:53.909834+0000","last_became_peered":"2024-09-18T08:02:53.909834+0000","last_unstale":"2024-09-18T08:05:51.439328+0000","last_undegraded":"2024-09-18T08:05:51.439328+0000","last_fullsized":"2024-09-18T08:05:51.439328+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:05:34.905083+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_clean_scrub_stamp":"2024-09-18T08:05:34.905083+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:06:53.452264+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.3","version":"126'8456","reported_seq":8553,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-09-18T08:05:53.303793+0000","last_change":"2024-09-18T08:03:57.462489+0000","last_active":"2024-09-18T08:05:53.303793+0000","last_peered":"2024-09-18T08:05:53.303793+0000","last_clean":"2024-09-18T08:05:53.303793+0000","last_became_active":"2024-09-18T08:03:57.462210+0000","last_became_peered":"2024-09-18T08:03:57.462210+0000","last_unstale":"2024-09-18T08:05:53.303793+0000","last_undegraded":"2024-09-18T08:05:53.303793+0000","last_fullsized":"2024-09-18T08:05:53.303793+0000","mapping_epoch":83,"log_start":"100'3800","ondisk_log_start":"100'3800","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:03:56.449957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:03:56.449957+0000","last_clean_scrub_stamp":"2024-09-18T08:03:56.449957+0000","objects_scrubbed":0,"log_size":4656,"log_dups_size":0,"ondisk_log_size":4656,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:06:20.115988+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":407502848,"num_objects":6218,"num_object_clones":0,"num_object_copies":18654,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6218,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8456,"num_write_kb":469568,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,4],"acting":[3,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.0","version":"69'18","reported_seq":177,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-09-18T08:05:51.440326+0000","last_change":"2024-09-18T08:04:14.969799+0000","last_active":"2024-09-18T08:05:51.440326+0000","last_peered":"2024-09-18T08:05:51.440326+0000","last_clean":"2024-09-18T08:05:51.440326+0000","last_became_active":"2024-09-18T08:02:49.886818+0000","last_became_peered":"2024-09-18T08:02:49.886818+0000","last_unstale":"2024-09-18T08:05:51.440326+0000","last_undegraded":"2024-09-18T08:05:51.440326+0000","last_fullsized":"2024-09-18T08:05:51.440326+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'18","last_scrub_stamp":"2024-09-18T08:04:14.969716+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_clean_scrub_stamp":"2024-09-18T08:04:14.969716+0000","objects_scrubbed":8,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":34,"num_read_kb":22,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":157,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-09-18T08:05:51.452090+0000","last_change":"2024-09-18T08:05:35.100407+0000","last_active":"2024-09-18T08:05:51.452090+0000","last_peered":"2024-09-18T08:05:51.452090+0000","last_clean":"2024-09-18T08:05:51.452090+0000","last_became_active":"2024-09-18T08:02:47.880225+0000","last_became_peered":"2024-09-18T08:02:47.880225+0000","last_unstale":"2024-09-18T08:05:51.452090+0000","last_undegraded":"2024-09-18T08:05:51.452090+0000","last_fullsized":"2024-09-18T08:05:51.452090+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:05:35.100332+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:46.860092+0000","last_clean_scrub_stamp":"2024-09-18T08:05:35.100332+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:06:57.779942+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":141,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-09-18T08:05:51.421028+0000","last_change":"2024-09-18T08:04:14.442378+0000","last_active":"2024-09-18T08:05:51.421028+0000","last_peered":"2024-09-18T08:05:51.421028+0000","last_clean":"2024-09-18T08:05:51.421028+0000","last_became_active":"2024-09-18T08:02:51.899594+0000","last_became_peered":"2024-09-18T08:02:51.899594+0000","last_unstale":"2024-09-18T08:05:51.421028+0000","last_undegraded":"2024-09-18T08:05:51.421028+0000","last_fullsized":"2024-09-18T08:05:51.421028+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:04:14.442331+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:50.886830+0000","last_clean_scrub_stamp":"2024-09-18T08:04:14.442331+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":138,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-09-18T08:05:51.421100+0000","last_change":"2024-09-18T08:04:16.427470+0000","last_active":"2024-09-18T08:05:51.421100+0000","last_peered":"2024-09-18T08:05:51.421100+0000","last_clean":"2024-09-18T08:05:51.421100+0000","last_became_active":"2024-09-18T08:02:53.903957+0000","last_became_peered":"2024-09-18T08:02:53.903957+0000","last_unstale":"2024-09-18T08:05:51.421100+0000","last_undegraded":"2024-09-18T08:05:51.421100+0000","last_fullsized":"2024-09-18T08:05:51.421100+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:04:16.427390+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_clean_scrub_stamp":"2024-09-18T08:04:16.427390+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:06:24.577043+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.2","version":"126'8519","reported_seq":8614,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-09-18T08:05:53.318865+0000","last_change":"2024-09-18T08:03:57.463750+0000","last_active":"2024-09-18T08:05:53.318865+0000","last_peered":"2024-09-18T08:05:53.318865+0000","last_clean":"2024-09-18T08:05:53.318865+0000","last_became_active":"2024-09-18T08:03:57.463459+0000","last_became_peered":"2024-09-18T08:03:57.463459+0000","last_unstale":"2024-09-18T08:05:53.318865+0000","last_undegraded":"2024-09-18T08:05:53.318865+0000","last_fullsized":"2024-09-18T08:05:53.318865+0000","mapping_epoch":83,"log_start":"100'3900","ondisk_log_start":"100'3900","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:03:56.449957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:03:56.449957+0000","last_clean_scrub_stamp":"2024-09-18T08:03:56.449957+0000","objects_scrubbed":0,"log_size":4619,"log_dups_size":0,"ondisk_log_size":4619,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":413073408,"num_objects":6303,"num_object_clones":0,"num_object_copies":18909,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6303,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8519,"num_write_kb":474304,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"69'14","reported_seq":174,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-09-18T08:05:51.421103+0000","last_change":"2024-09-18T08:04:56.545313+0000","last_active":"2024-09-18T08:05:51.421103+0000","last_peered":"2024-09-18T08:05:51.421103+0000","last_clean":"2024-09-18T08:05:51.421103+0000","last_became_active":"2024-09-18T08:02:49.893830+0000","last_became_peered":"2024-09-18T08:02:49.893830+0000","last_unstale":"2024-09-18T08:05:51.421103+0000","last_undegraded":"2024-09-18T08:05:51.421103+0000","last_fullsized":"2024-09-18T08:05:51.421103+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'14","last_scrub_stamp":"2024-09-18T08:04:56.545262+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_clean_scrub_stamp":"2024-09-18T08:04:56.545262+0000","objects_scrubbed":7,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:06:17.884382+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":21,"num_read_kb":14,"num_write":14,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"62'2","reported_seq":156,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-09-18T08:05:51.431828+0000","last_change":"2024-09-18T08:04:09.240465+0000","last_active":"2024-09-18T08:05:51.431828+0000","last_peered":"2024-09-18T08:05:51.431828+0000","last_clean":"2024-09-18T08:05:51.431828+0000","last_became_active":"2024-09-18T08:02:47.872857+0000","last_became_peered":"2024-09-18T08:02:47.872857+0000","last_unstale":"2024-09-18T08:05:51.431828+0000","last_undegraded":"2024-09-18T08:05:51.431828+0000","last_fullsized":"2024-09-18T08:05:51.431828+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"62'2","last_scrub_stamp":"2024-09-18T08:04:09.240392+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:46.860092+0000","last_clean_scrub_stamp":"2024-09-18T08:04:09.240392+0000","objects_scrubbed":2,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:06:24.534983+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1039,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.0","version":"69'8","reported_seq":187,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-09-18T08:05:51.440126+0000","last_change":"2024-09-18T08:04:23.863573+0000","last_active":"2024-09-18T08:05:51.440126+0000","last_peered":"2024-09-18T08:05:51.440126+0000","last_clean":"2024-09-18T08:05:51.440126+0000","last_became_active":"2024-09-18T08:02:51.909680+0000","last_became_peered":"2024-09-18T08:02:51.909680+0000","last_unstale":"2024-09-18T08:05:51.440126+0000","last_undegraded":"2024-09-18T08:05:51.440126+0000","last_fullsized":"2024-09-18T08:05:51.440126+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'8","last_scrub_stamp":"2024-09-18T08:04:23.863330+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:50.886830+0000","last_clean_scrub_stamp":"2024-09-18T08:04:23.863330+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":138,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-09-18T08:05:51.434984+0000","last_change":"2024-09-18T08:05:06.885362+0000","last_active":"2024-09-18T08:05:51.434984+0000","last_peered":"2024-09-18T08:05:51.434984+0000","last_clean":"2024-09-18T08:05:51.434984+0000","last_became_active":"2024-09-18T08:02:54.047039+0000","last_became_peered":"2024-09-18T08:02:54.047039+0000","last_unstale":"2024-09-18T08:05:51.434984+0000","last_undegraded":"2024-09-18T08:05:51.434984+0000","last_fullsized":"2024-09-18T08:05:51.434984+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:05:06.885311+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_clean_scrub_stamp":"2024-09-18T08:05:06.885311+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:06:29.893451+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.1","version":"126'8498","reported_seq":8594,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-09-18T08:05:53.084053+0000","last_change":"2024-09-18T08:03:57.465538+0000","last_active":"2024-09-18T08:05:53.084053+0000","last_peered":"2024-09-18T08:05:53.084053+0000","last_clean":"2024-09-18T08:05:53.084053+0000","last_became_active":"2024-09-18T08:03:57.465253+0000","last_became_peered":"2024-09-18T08:03:57.465253+0000","last_unstale":"2024-09-18T08:05:53.084053+0000","last_undegraded":"2024-09-18T08:05:53.084053+0000","last_fullsized":"2024-09-18T08:05:53.084053+0000","mapping_epoch":83,"log_start":"99'3700","ondisk_log_start":"99'3700","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:03:56.449957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:03:56.449957+0000","last_clean_scrub_stamp":"2024-09-18T08:03:56.449957+0000","objects_scrubbed":0,"log_size":4798,"log_dups_size":0,"ondisk_log_size":4798,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:06:15.615182+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":416153600,"num_objects":6350,"num_object_clones":0,"num_object_copies":19050,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6350,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8498,"num_write_kb":475136,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.2","version":"69'10","reported_seq":168,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-09-18T08:05:51.447107+0000","last_change":"2024-09-18T08:05:39.769686+0000","last_active":"2024-09-18T08:05:51.447107+0000","last_peered":"2024-09-18T08:05:51.447107+0000","last_clean":"2024-09-18T08:05:51.447107+0000","last_became_active":"2024-09-18T08:02:49.895265+0000","last_became_peered":"2024-09-18T08:02:49.895265+0000","last_unstale":"2024-09-18T08:05:51.447107+0000","last_undegraded":"2024-09-18T08:05:51.447107+0000","last_fullsized":"2024-09-18T08:05:51.447107+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'10","last_scrub_stamp":"2024-09-18T08:05:39.768393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_clean_scrub_stamp":"2024-09-18T08:05:39.768393+0000","objects_scrubbed":5,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:07:07.361416+0000","scrub_duration":17,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":149,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-09-18T08:05:51.439591+0000","last_change":"2024-09-18T08:03:48.009689+0000","last_active":"2024-09-18T08:05:51.439591+0000","last_peered":"2024-09-18T08:05:51.439591+0000","last_clean":"2024-09-18T08:05:51.439591+0000","last_became_active":"2024-09-18T08:02:47.874727+0000","last_became_peered":"2024-09-18T08:02:47.874727+0000","last_unstale":"2024-09-18T08:05:51.439591+0000","last_undegraded":"2024-09-18T08:05:51.439591+0000","last_fullsized":"2024-09-18T08:05:51.439591+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:03:48.009611+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:46.860092+0000","last_clean_scrub_stamp":"2024-09-18T08:03:48.009611+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:06:18.040455+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.3","version":"69'8","reported_seq":179,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-09-18T08:05:51.429703+0000","last_change":"2024-09-18T08:02:51.906369+0000","last_active":"2024-09-18T08:05:51.429703+0000","last_peered":"2024-09-18T08:05:51.429703+0000","last_clean":"2024-09-18T08:05:51.429703+0000","last_became_active":"2024-09-18T08:02:51.906055+0000","last_became_peered":"2024-09-18T08:02:51.906055+0000","last_unstale":"2024-09-18T08:05:51.429703+0000","last_undegraded":"2024-09-18T08:05:51.429703+0000","last_fullsized":"2024-09-18T08:05:51.429703+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:02:50.886830+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:50.886830+0000","last_clean_scrub_stamp":"2024-09-18T08:02:50.886830+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:05:53.303295+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":146,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-09-18T08:05:51.429970+0000","last_change":"2024-09-18T08:05:43.440713+0000","last_active":"2024-09-18T08:05:51.429970+0000","last_peered":"2024-09-18T08:05:51.429970+0000","last_clean":"2024-09-18T08:05:51.429970+0000","last_became_active":"2024-09-18T08:02:53.909583+0000","last_became_peered":"2024-09-18T08:02:53.909583+0000","last_unstale":"2024-09-18T08:05:51.429970+0000","last_undegraded":"2024-09-18T08:05:51.429970+0000","last_fullsized":"2024-09-18T08:05:51.429970+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:05:43.440642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_clean_scrub_stamp":"2024-09-18T08:05:43.440642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:07:02.845583+0000","scrub_duration":35,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.0","version":"126'8418","reported_seq":8576,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-09-18T08:05:53.080027+0000","last_change":"2024-09-18T08:05:01.419397+0000","last_active":"2024-09-18T08:05:53.080027+0000","last_peered":"2024-09-18T08:05:53.080027+0000","last_clean":"2024-09-18T08:05:53.080027+0000","last_became_active":"2024-09-18T08:03:57.468985+0000","last_became_peered":"2024-09-18T08:03:57.468985+0000","last_unstale":"2024-09-18T08:05:53.080027+0000","last_undegraded":"2024-09-18T08:05:53.080027+0000","last_fullsized":"2024-09-18T08:05:53.080027+0000","mapping_epoch":83,"log_start":"96'2600","ondisk_log_start":"96'2600","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"107'5286","last_scrub_stamp":"2024-09-18T08:05:01.419332+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:03:56.449957+0000","last_clean_scrub_stamp":"2024-09-18T08:05:01.419332+0000","objects_scrubbed":5263,"log_size":5818,"log_dups_size":0,"ondisk_log_size":5818,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:06:20.999535+0000","scrub_duration":1313,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":409468928,"num_objects":6248,"num_object_clones":0,"num_object_copies":18744,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6248,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8418,"num_write_kb":469312,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.3","version":"69'19","reported_seq":181,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-09-18T08:05:51.429853+0000","last_change":"2024-09-18T08:04:07.231406+0000","last_active":"2024-09-18T08:05:51.429853+0000","last_peered":"2024-09-18T08:05:51.429853+0000","last_clean":"2024-09-18T08:05:51.429853+0000","last_became_active":"2024-09-18T08:02:49.890318+0000","last_became_peered":"2024-09-18T08:02:49.890318+0000","last_unstale":"2024-09-18T08:05:51.429853+0000","last_undegraded":"2024-09-18T08:05:51.429853+0000","last_fullsized":"2024-09-18T08:05:51.429853+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'19","last_scrub_stamp":"2024-09-18T08:04:07.229846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_clean_scrub_stamp":"2024-09-18T08:04:07.229846+0000","objects_scrubbed":8,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:06:15.339152+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":39,"num_read_kb":25,"num_write":22,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.4","version":"62'1","reported_seq":155,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-09-18T08:05:51.445283+0000","last_change":"2024-09-18T08:04:28.827545+0000","last_active":"2024-09-18T08:05:51.445283+0000","last_peered":"2024-09-18T08:05:51.445283+0000","last_clean":"2024-09-18T08:05:51.445283+0000","last_became_active":"2024-09-18T08:02:47.874764+0000","last_became_peered":"2024-09-18T08:02:47.874764+0000","last_unstale":"2024-09-18T08:05:51.445283+0000","last_undegraded":"2024-09-18T08:05:51.445283+0000","last_fullsized":"2024-09-18T08:05:51.445283+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"62'1","last_scrub_stamp":"2024-09-18T08:04:28.827464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:46.860092+0000","last_clean_scrub_stamp":"2024-09-18T08:04:28.827464+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":436,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":61,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-09-18T08:05:51.430453+0000","last_change":"2024-09-18T08:04:43.304561+0000","last_active":"2024-09-18T08:05:51.430453+0000","last_peered":"2024-09-18T08:05:51.430453+0000","last_clean":"2024-09-18T08:05:51.430453+0000","last_became_active":"2024-09-18T08:04:43.304002+0000","last_became_peered":"2024-09-18T08:04:43.304002+0000","last_unstale":"2024-09-18T08:05:51.430453+0000","last_undegraded":"2024-09-18T08:05:51.430453+0000","last_fullsized":"2024-09-18T08:05:51.430453+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:04:13.377561+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:50.886830+0000","last_clean_scrub_stamp":"2024-09-18T08:04:13.377561+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":137,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-09-18T08:05:51.452400+0000","last_change":"2024-09-18T08:05:03.874423+0000","last_active":"2024-09-18T08:05:51.452400+0000","last_peered":"2024-09-18T08:05:51.452400+0000","last_clean":"2024-09-18T08:05:51.452400+0000","last_became_active":"2024-09-18T08:02:54.045726+0000","last_became_peered":"2024-09-18T08:02:54.045726+0000","last_unstale":"2024-09-18T08:05:51.452400+0000","last_undegraded":"2024-09-18T08:05:51.452400+0000","last_fullsized":"2024-09-18T08:05:51.452400+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:05:03.874339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_clean_scrub_stamp":"2024-09-18T08:05:03.874339+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:06:08.270559+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.7","version":"126'8521","reported_seq":8629,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-09-18T08:05:53.319132+0000","last_change":"2024-09-18T08:05:13.708697+0000","last_active":"2024-09-18T08:05:53.319132+0000","last_peered":"2024-09-18T08:05:53.319132+0000","last_clean":"2024-09-18T08:05:53.319132+0000","last_became_active":"2024-09-18T08:04:43.300866+0000","last_became_peered":"2024-09-18T08:04:43.300866+0000","last_unstale":"2024-09-18T08:05:53.319132+0000","last_undegraded":"2024-09-18T08:05:53.319132+0000","last_fullsized":"2024-09-18T08:05:53.319132+0000","mapping_epoch":101,"log_start":"97'2900","ondisk_log_start":"97'2900","created":83,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:03:56.449957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:03:56.449957+0000","last_clean_scrub_stamp":"2024-09-18T08:03:56.449957+0000","objects_scrubbed":399,"log_size":5621,"log_dups_size":0,"ondisk_log_size":5621,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:05:54.631049+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":416481280,"num_objects":6355,"num_object_clones":0,"num_object_copies":19065,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6355,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8521,"num_write_kb":476032,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,3],"acting":[0,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.4","version":"69'28","reported_seq":231,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-09-18T08:05:51.445226+0000","last_change":"2024-09-18T08:04:12.737618+0000","last_active":"2024-09-18T08:05:51.445226+0000","last_peered":"2024-09-18T08:05:51.445226+0000","last_clean":"2024-09-18T08:05:51.445226+0000","last_became_active":"2024-09-18T08:02:49.897221+0000","last_became_peered":"2024-09-18T08:02:49.897221+0000","last_unstale":"2024-09-18T08:05:51.445226+0000","last_undegraded":"2024-09-18T08:05:51.445226+0000","last_fullsized":"2024-09-18T08:05:51.445226+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'28","last_scrub_stamp":"2024-09-18T08:04:12.737528+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_clean_scrub_stamp":"2024-09-18T08:04:12.737528+0000","objects_scrubbed":10,"log_size":28,"log_dups_size":0,"ondisk_log_size":28,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:06:50.593181+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":48,"num_read_kb":33,"num_write":26,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":149,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-09-18T08:05:51.421823+0000","last_change":"2024-09-18T08:05:39.490204+0000","last_active":"2024-09-18T08:05:51.421823+0000","last_peered":"2024-09-18T08:05:51.421823+0000","last_clean":"2024-09-18T08:05:51.421823+0000","last_became_active":"2024-09-18T08:02:47.877085+0000","last_became_peered":"2024-09-18T08:02:47.877085+0000","last_unstale":"2024-09-18T08:05:51.421823+0000","last_undegraded":"2024-09-18T08:05:51.421823+0000","last_fullsized":"2024-09-18T08:05:51.421823+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:05:39.490141+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:46.860092+0000","last_clean_scrub_stamp":"2024-09-18T08:05:39.490141+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:06:51.573012+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"64'2","reported_seq":167,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-09-18T08:05:51.439189+0000","last_change":"2024-09-18T08:05:44.803021+0000","last_active":"2024-09-18T08:05:51.439189+0000","last_peered":"2024-09-18T08:05:51.439189+0000","last_clean":"2024-09-18T08:05:51.439189+0000","last_became_active":"2024-09-18T08:02:46.867802+0000","last_became_peered":"2024-09-18T08:02:46.867802+0000","last_unstale":"2024-09-18T08:05:51.439189+0000","last_undegraded":"2024-09-18T08:05:51.439189+0000","last_fullsized":"2024-09-18T08:05:51.439189+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"64'2","last_scrub_stamp":"2024-09-18T08:05:44.802942+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:45.855979+0000","last_clean_scrub_stamp":"2024-09-18T08:05:44.802942+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:07:03.506811+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.001348349,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":133,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-09-18T08:05:51.431273+0000","last_change":"2024-09-18T08:02:51.905694+0000","last_active":"2024-09-18T08:05:51.431273+0000","last_peered":"2024-09-18T08:05:51.431273+0000","last_clean":"2024-09-18T08:05:51.431273+0000","last_became_active":"2024-09-18T08:02:51.905357+0000","last_became_peered":"2024-09-18T08:02:51.905357+0000","last_unstale":"2024-09-18T08:05:51.431273+0000","last_undegraded":"2024-09-18T08:05:51.431273+0000","last_fullsized":"2024-09-18T08:05:51.431273+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:02:50.886830+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:50.886830+0000","last_clean_scrub_stamp":"2024-09-18T08:02:50.886830+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:06:51.299235+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":138,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-09-18T08:05:51.442996+0000","last_change":"2024-09-18T08:05:04.962900+0000","last_active":"2024-09-18T08:05:51.442996+0000","last_peered":"2024-09-18T08:05:51.442996+0000","last_clean":"2024-09-18T08:05:51.442996+0000","last_became_active":"2024-09-18T08:02:53.911036+0000","last_became_peered":"2024-09-18T08:02:53.911036+0000","last_unstale":"2024-09-18T08:05:51.442996+0000","last_undegraded":"2024-09-18T08:05:51.442996+0000","last_fullsized":"2024-09-18T08:05:51.442996+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:05:04.962686+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_clean_scrub_stamp":"2024-09-18T08:05:04.962686+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:06:08.603603+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.4","version":"126'8568","reported_seq":8665,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-09-18T08:05:53.326602+0000","last_change":"2024-09-18T08:03:57.466551+0000","last_active":"2024-09-18T08:05:53.326602+0000","last_peered":"2024-09-18T08:05:53.326602+0000","last_clean":"2024-09-18T08:05:53.326602+0000","last_became_active":"2024-09-18T08:03:57.466232+0000","last_became_peered":"2024-09-18T08:03:57.466232+0000","last_unstale":"2024-09-18T08:05:53.326602+0000","last_undegraded":"2024-09-18T08:05:53.326602+0000","last_fullsized":"2024-09-18T08:05:53.326602+0000","mapping_epoch":83,"log_start":"98'3000","ondisk_log_start":"98'3000","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:03:56.449957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:03:56.449957+0000","last_clean_scrub_stamp":"2024-09-18T08:03:56.449957+0000","objects_scrubbed":0,"log_size":5568,"log_dups_size":0,"ondisk_log_size":5568,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:06:25.555609+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":412614656,"num_objects":6296,"num_object_clones":0,"num_object_copies":18888,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6296,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8568,"num_write_kb":475648,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,1],"acting":[0,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.7","version":"69'13","reported_seq":90,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-09-18T08:05:51.433012+0000","last_change":"2024-09-18T08:04:18.164662+0000","last_active":"2024-09-18T08:05:51.433012+0000","last_peered":"2024-09-18T08:05:51.433012+0000","last_clean":"2024-09-18T08:05:51.433012+0000","last_became_active":"2024-09-18T08:04:18.144925+0000","last_became_peered":"2024-09-18T08:04:18.144925+0000","last_unstale":"2024-09-18T08:05:51.433012+0000","last_undegraded":"2024-09-18T08:05:51.433012+0000","last_fullsized":"2024-09-18T08:05:51.433012+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'13","last_scrub_stamp":"2024-09-18T08:03:50.700818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_clean_scrub_stamp":"2024-09-18T08:03:50.700818+0000","objects_scrubbed":0,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:06:19.016400+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":29,"num_read_kb":18,"num_write":16,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,0],"acting":[2,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":149,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-09-18T08:05:51.442747+0000","last_change":"2024-09-18T08:03:49.704071+0000","last_active":"2024-09-18T08:05:51.442747+0000","last_peered":"2024-09-18T08:05:51.442747+0000","last_clean":"2024-09-18T08:05:51.442747+0000","last_became_active":"2024-09-18T08:02:47.871416+0000","last_became_peered":"2024-09-18T08:02:47.871416+0000","last_unstale":"2024-09-18T08:05:51.442747+0000","last_undegraded":"2024-09-18T08:05:51.442747+0000","last_fullsized":"2024-09-18T08:05:51.442747+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:03:49.704004+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:46.860092+0000","last_clean_scrub_stamp":"2024-09-18T08:03:49.704004+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:06:21.766689+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"62'1","reported_seq":158,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-09-18T08:05:51.432676+0000","last_change":"2024-09-18T08:04:09.732724+0000","last_active":"2024-09-18T08:05:51.432676+0000","last_peered":"2024-09-18T08:05:51.432676+0000","last_clean":"2024-09-18T08:05:51.432676+0000","last_became_active":"2024-09-18T08:02:46.868175+0000","last_became_peered":"2024-09-18T08:02:46.868175+0000","last_unstale":"2024-09-18T08:05:51.432676+0000","last_undegraded":"2024-09-18T08:05:51.432676+0000","last_fullsized":"2024-09-18T08:05:51.432676+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"62'1","last_scrub_stamp":"2024-09-18T08:04:09.732645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:45.855979+0000","last_clean_scrub_stamp":"2024-09-18T08:04:09.732645+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:06:24.986709+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0.00066785499999999997,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":149,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-09-18T08:05:51.434945+0000","last_change":"2024-09-18T08:05:41.059426+0000","last_active":"2024-09-18T08:05:51.434945+0000","last_peered":"2024-09-18T08:05:51.434945+0000","last_clean":"2024-09-18T08:05:51.434945+0000","last_became_active":"2024-09-18T08:02:51.908156+0000","last_became_peered":"2024-09-18T08:02:51.908156+0000","last_unstale":"2024-09-18T08:05:51.434945+0000","last_undegraded":"2024-09-18T08:05:51.434945+0000","last_fullsized":"2024-09-18T08:05:51.434945+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:05:41.059230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:50.886830+0000","last_clean_scrub_stamp":"2024-09-18T08:05:41.059230+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:06:45.166099+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":138,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-09-18T08:05:51.435299+0000","last_change":"2024-09-18T08:05:40.929414+0000","last_active":"2024-09-18T08:05:51.435299+0000","last_peered":"2024-09-18T08:05:51.435299+0000","last_clean":"2024-09-18T08:05:51.435299+0000","last_became_active":"2024-09-18T08:02:54.048004+0000","last_became_peered":"2024-09-18T08:02:54.048004+0000","last_unstale":"2024-09-18T08:05:51.435299+0000","last_undegraded":"2024-09-18T08:05:51.435299+0000","last_fullsized":"2024-09-18T08:05:51.435299+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:05:40.929345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_clean_scrub_stamp":"2024-09-18T08:05:40.929345+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:06:57.162980+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.5","version":"126'8452","reported_seq":8549,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-09-18T08:05:53.064778+0000","last_change":"2024-09-18T08:03:57.469151+0000","last_active":"2024-09-18T08:05:53.064778+0000","last_peered":"2024-09-18T08:05:53.064778+0000","last_clean":"2024-09-18T08:05:53.064778+0000","last_became_active":"2024-09-18T08:03:57.468853+0000","last_became_peered":"2024-09-18T08:03:57.468853+0000","last_unstale":"2024-09-18T08:05:53.064778+0000","last_undegraded":"2024-09-18T08:05:53.064778+0000","last_fullsized":"2024-09-18T08:05:53.064778+0000","mapping_epoch":83,"log_start":"98'3200","ondisk_log_start":"98'3200","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:03:56.449957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:03:56.449957+0000","last_clean_scrub_stamp":"2024-09-18T08:03:56.449957+0000","objects_scrubbed":0,"log_size":5252,"log_dups_size":0,"ondisk_log_size":5252,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:06:20.730047+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":412614656,"num_objects":6296,"num_object_clones":0,"num_object_copies":18888,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6296,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8452,"num_write_kb":471936,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,0],"acting":[7,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"69'12","reported_seq":163,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-09-18T08:05:51.431737+0000","last_change":"2024-09-18T08:05:38.471012+0000","last_active":"2024-09-18T08:05:51.431737+0000","last_peered":"2024-09-18T08:05:51.431737+0000","last_clean":"2024-09-18T08:05:51.431737+0000","last_became_active":"2024-09-18T08:02:49.898079+0000","last_became_peered":"2024-09-18T08:02:49.898079+0000","last_unstale":"2024-09-18T08:05:51.431737+0000","last_undegraded":"2024-09-18T08:05:51.431737+0000","last_fullsized":"2024-09-18T08:05:51.431737+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'12","last_scrub_stamp":"2024-09-18T08:05:38.470713+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_clean_scrub_stamp":"2024-09-18T08:05:38.470713+0000","objects_scrubbed":6,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:06:50.866624+0000","scrub_duration":28,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":18,"num_read_kb":12,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":149,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-09-18T08:05:51.432950+0000","last_change":"2024-09-18T08:04:59.298230+0000","last_active":"2024-09-18T08:05:51.432950+0000","last_peered":"2024-09-18T08:05:51.432950+0000","last_clean":"2024-09-18T08:05:51.432950+0000","last_became_active":"2024-09-18T08:02:47.873809+0000","last_became_peered":"2024-09-18T08:02:47.873809+0000","last_unstale":"2024-09-18T08:05:51.432950+0000","last_undegraded":"2024-09-18T08:05:51.432950+0000","last_fullsized":"2024-09-18T08:05:51.432950+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:04:59.298029+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:46.860092+0000","last_clean_scrub_stamp":"2024-09-18T08:04:59.298029+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:06:19.180037+0000","scrub_duration":31,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"70'5","reported_seq":340,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-09-18T08:05:51.873231+0000","last_change":"2024-09-18T08:04:06.814615+0000","last_active":"2024-09-18T08:05:51.873231+0000","last_peered":"2024-09-18T08:05:51.873231+0000","last_clean":"2024-09-18T08:05:51.873231+0000","last_became_active":"2024-09-18T08:02:46.872144+0000","last_became_peered":"2024-09-18T08:02:46.872144+0000","last_unstale":"2024-09-18T08:05:51.873231+0000","last_undegraded":"2024-09-18T08:05:51.873231+0000","last_fullsized":"2024-09-18T08:05:51.873231+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"70'5","last_scrub_stamp":"2024-09-18T08:04:06.814540+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:45.855979+0000","last_clean_scrub_stamp":"2024-09-18T08:04:06.814540+0000","objects_scrubbed":1,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:06:24.789846+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.000504055,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":180,"num_read_kb":175,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":149,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-09-18T08:05:51.439757+0000","last_change":"2024-09-18T08:05:41.750515+0000","last_active":"2024-09-18T08:05:51.439757+0000","last_peered":"2024-09-18T08:05:51.439757+0000","last_clean":"2024-09-18T08:05:51.439757+0000","last_became_active":"2024-09-18T08:02:51.913112+0000","last_became_peered":"2024-09-18T08:02:51.913112+0000","last_unstale":"2024-09-18T08:05:51.439757+0000","last_undegraded":"2024-09-18T08:05:51.439757+0000","last_fullsized":"2024-09-18T08:05:51.439757+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:05:41.750446+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:50.886830+0000","last_clean_scrub_stamp":"2024-09-18T08:05:41.750446+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:06:55.866373+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":130,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-09-18T08:05:51.442282+0000","last_change":"2024-09-18T08:02:53.907689+0000","last_active":"2024-09-18T08:05:51.442282+0000","last_peered":"2024-09-18T08:05:51.442282+0000","last_clean":"2024-09-18T08:05:51.442282+0000","last_became_active":"2024-09-18T08:02:53.907404+0000","last_became_peered":"2024-09-18T08:02:53.907404+0000","last_unstale":"2024-09-18T08:05:51.442282+0000","last_undegraded":"2024-09-18T08:05:51.442282+0000","last_fullsized":"2024-09-18T08:05:51.442282+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_clean_scrub_stamp":"2024-09-18T08:02:52.890475+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:06:18.732348+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.6","version":"126'8482","reported_seq":8622,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-09-18T08:05:53.301793+0000","last_change":"2024-09-18T08:05:13.657923+0000","last_active":"2024-09-18T08:05:53.301793+0000","last_peered":"2024-09-18T08:05:53.301793+0000","last_clean":"2024-09-18T08:05:53.301793+0000","last_became_active":"2024-09-18T08:03:57.466391+0000","last_became_peered":"2024-09-18T08:03:57.466391+0000","last_unstale":"2024-09-18T08:05:53.301793+0000","last_undegraded":"2024-09-18T08:05:53.301793+0000","last_fullsized":"2024-09-18T08:05:53.301793+0000","mapping_epoch":83,"log_start":"97'2900","ondisk_log_start":"97'2900","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:03:56.449957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:03:56.449957+0000","last_clean_scrub_stamp":"2024-09-18T08:03:56.449957+0000","objects_scrubbed":3899,"log_size":5582,"log_dups_size":0,"ondisk_log_size":5582,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":412352512,"num_objects":6292,"num_object_clones":0,"num_object_copies":18876,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6292,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8482,"num_write_kb":472768,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.5","version":"102'17","reported_seq":120,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-09-18T08:05:51.421874+0000","last_change":"2024-09-18T08:04:43.301741+0000","last_active":"2024-09-18T08:05:51.421874+0000","last_peered":"2024-09-18T08:05:51.421874+0000","last_clean":"2024-09-18T08:05:51.421874+0000","last_became_active":"2024-09-18T08:04:43.301400+0000","last_became_peered":"2024-09-18T08:04:43.301400+0000","last_unstale":"2024-09-18T08:05:51.421874+0000","last_undegraded":"2024-09-18T08:05:51.421874+0000","last_fullsized":"2024-09-18T08:05:51.421874+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'16","last_scrub_stamp":"2024-09-18T08:04:14.358816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_clean_scrub_stamp":"2024-09-18T08:04:14.358816+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:06:21.649266+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":24,"num_read_kb":14,"num_write":13,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,7],"acting":[4,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":149,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-09-18T08:05:51.443331+0000","last_change":"2024-09-18T08:05:38.081633+0000","last_active":"2024-09-18T08:05:51.443331+0000","last_peered":"2024-09-18T08:05:51.443331+0000","last_clean":"2024-09-18T08:05:51.443331+0000","last_became_active":"2024-09-18T08:02:47.875950+0000","last_became_peered":"2024-09-18T08:02:47.875950+0000","last_unstale":"2024-09-18T08:05:51.443331+0000","last_undegraded":"2024-09-18T08:05:51.443331+0000","last_fullsized":"2024-09-18T08:05:51.443331+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:05:38.080993+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:46.860092+0000","last_clean_scrub_stamp":"2024-09-18T08:05:38.080993+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:06:45.582435+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"74'79","reported_seq":196,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-09-18T08:05:51.435433+0000","last_change":"2024-09-18T08:04:33.791905+0000","last_active":"2024-09-18T08:05:51.435433+0000","last_peered":"2024-09-18T08:05:51.435433+0000","last_clean":"2024-09-18T08:05:51.435433+0000","last_became_active":"2024-09-18T08:02:39.446743+0000","last_became_peered":"2024-09-18T08:02:39.446743+0000","last_unstale":"2024-09-18T08:05:51.435433+0000","last_undegraded":"2024-09-18T08:05:51.435433+0000","last_fullsized":"2024-09-18T08:05:51.435433+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'79","last_scrub_stamp":"2024-09-18T08:04:33.791841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:00:11.893742+0000","last_clean_scrub_stamp":"2024-09-18T08:04:33.791841+0000","objects_scrubbed":2,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:06:01.148512+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":231,"num_write":113,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":590368,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":149,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-09-18T08:05:51.434995+0000","last_change":"2024-09-18T08:05:42.929394+0000","last_active":"2024-09-18T08:05:51.434995+0000","last_peered":"2024-09-18T08:05:51.434995+0000","last_clean":"2024-09-18T08:05:51.434995+0000","last_became_active":"2024-09-18T08:02:51.905516+0000","last_became_peered":"2024-09-18T08:02:51.905516+0000","last_unstale":"2024-09-18T08:05:51.434995+0000","last_undegraded":"2024-09-18T08:05:51.434995+0000","last_fullsized":"2024-09-18T08:05:51.434995+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:05:42.929311+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:50.886830+0000","last_clean_scrub_stamp":"2024-09-18T08:05:42.929311+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:06:48.900458+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":138,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-09-18T08:05:51.439649+0000","last_change":"2024-09-18T08:04:11.930843+0000","last_active":"2024-09-18T08:05:51.439649+0000","last_peered":"2024-09-18T08:05:51.439649+0000","last_clean":"2024-09-18T08:05:51.439649+0000","last_became_active":"2024-09-18T08:02:53.908668+0000","last_became_peered":"2024-09-18T08:02:53.908668+0000","last_unstale":"2024-09-18T08:05:51.439649+0000","last_undegraded":"2024-09-18T08:05:51.439649+0000","last_fullsized":"2024-09-18T08:05:51.439649+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:04:11.930797+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_clean_scrub_stamp":"2024-09-18T08:04:11.930797+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:06:49.753536+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.d","version":"126'8527","reported_seq":8707,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-09-18T08:05:53.128794+0000","last_change":"2024-09-18T08:05:44.591209+0000","last_active":"2024-09-18T08:05:53.128794+0000","last_peered":"2024-09-18T08:05:53.128794+0000","last_clean":"2024-09-18T08:05:53.128794+0000","last_became_active":"2024-09-18T08:04:43.300869+0000","last_became_peered":"2024-09-18T08:04:43.300869+0000","last_unstale":"2024-09-18T08:05:53.128794+0000","last_undegraded":"2024-09-18T08:05:53.128794+0000","last_fullsized":"2024-09-18T08:05:53.128794+0000","mapping_epoch":101,"log_start":"98'3500","ondisk_log_start":"98'3500","created":83,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"123'7982","last_scrub_stamp":"2024-09-18T08:05:44.590884+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:03:56.449957+0000","last_clean_scrub_stamp":"2024-09-18T08:05:44.590884+0000","objects_scrubbed":6877,"log_size":5027,"log_dups_size":0,"ondisk_log_size":5027,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:07:10.003353+0000","scrub_duration":2853,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":409075736,"num_objects":6243,"num_object_clones":0,"num_object_copies":18729,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6243,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":8527,"num_write_kb":472577,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,0],"acting":[1,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.e","version":"69'11","reported_seq":156,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-09-18T08:05:51.422855+0000","last_change":"2024-09-18T08:02:49.895676+0000","last_active":"2024-09-18T08:05:51.422855+0000","last_peered":"2024-09-18T08:05:51.422855+0000","last_clean":"2024-09-18T08:05:51.422855+0000","last_became_active":"2024-09-18T08:02:49.895303+0000","last_became_peered":"2024-09-18T08:02:49.895303+0000","last_unstale":"2024-09-18T08:05:51.422855+0000","last_undegraded":"2024-09-18T08:05:51.422855+0000","last_fullsized":"2024-09-18T08:05:51.422855+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_clean_scrub_stamp":"2024-09-18T08:02:48.870718+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:06:48.446771+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":141,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-09-18T08:05:51.421411+0000","last_change":"2024-09-18T08:02:47.880265+0000","last_active":"2024-09-18T08:05:51.421411+0000","last_peered":"2024-09-18T08:05:51.421411+0000","last_clean":"2024-09-18T08:05:51.421411+0000","last_became_active":"2024-09-18T08:02:47.879764+0000","last_became_peered":"2024-09-18T08:02:47.879764+0000","last_unstale":"2024-09-18T08:05:51.421411+0000","last_undegraded":"2024-09-18T08:05:51.421411+0000","last_fullsized":"2024-09-18T08:05:51.421411+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:02:46.860092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:46.860092+0000","last_clean_scrub_stamp":"2024-09-18T08:02:46.860092+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":141,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-09-18T08:05:51.439547+0000","last_change":"2024-09-18T08:05:08.839711+0000","last_active":"2024-09-18T08:05:51.439547+0000","last_peered":"2024-09-18T08:05:51.439547+0000","last_clean":"2024-09-18T08:05:51.439547+0000","last_became_active":"2024-09-18T08:02:51.910649+0000","last_became_peered":"2024-09-18T08:02:51.910649+0000","last_unstale":"2024-09-18T08:05:51.439547+0000","last_undegraded":"2024-09-18T08:05:51.439547+0000","last_fullsized":"2024-09-18T08:05:51.439547+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:05:08.839644+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:50.886830+0000","last_clean_scrub_stamp":"2024-09-18T08:05:08.839644+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:06:13.625868+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":137,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-09-18T08:05:51.442914+0000","last_change":"2024-09-18T08:05:09.038204+0000","last_active":"2024-09-18T08:05:51.442914+0000","last_peered":"2024-09-18T08:05:51.442914+0000","last_clean":"2024-09-18T08:05:51.442914+0000","last_became_active":"2024-09-18T08:02:54.045091+0000","last_became_peered":"2024-09-18T08:02:54.045091+0000","last_unstale":"2024-09-18T08:05:51.442914+0000","last_undegraded":"2024-09-18T08:05:51.442914+0000","last_fullsized":"2024-09-18T08:05:51.442914+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:05:09.038115+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_clean_scrub_stamp":"2024-09-18T08:05:09.038115+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:06:18.302588+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.e","version":"126'8494","reported_seq":8710,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-09-18T08:05:53.213424+0000","last_change":"2024-09-18T08:05:47.424879+0000","last_active":"2024-09-18T08:05:53.213424+0000","last_peered":"2024-09-18T08:05:53.213424+0000","last_clean":"2024-09-18T08:05:53.213424+0000","last_became_active":"2024-09-18T08:03:57.464311+0000","last_became_peered":"2024-09-18T08:03:57.464311+0000","last_unstale":"2024-09-18T08:05:53.213424+0000","last_undegraded":"2024-09-18T08:05:53.213424+0000","last_fullsized":"2024-09-18T08:05:53.213424+0000","mapping_epoch":83,"log_start":"91'1600","ondisk_log_start":"91'1600","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:03:56.449957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:03:56.449957+0000","last_clean_scrub_stamp":"2024-09-18T08:03:56.449957+0000","objects_scrubbed":5849,"log_size":6894,"log_dups_size":0,"ondisk_log_size":6894,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:06:17.403222+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":411566080,"num_objects":6280,"num_object_clones":0,"num_object_copies":18840,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6280,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8494,"num_write_kb":472768,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,0],"acting":[2,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.d","version":"69'17","reported_seq":165,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-09-18T08:05:51.422525+0000","last_change":"2024-09-18T08:02:49.895971+0000","last_active":"2024-09-18T08:05:51.422525+0000","last_peered":"2024-09-18T08:05:51.422525+0000","last_clean":"2024-09-18T08:05:51.422525+0000","last_became_active":"2024-09-18T08:02:49.895578+0000","last_became_peered":"2024-09-18T08:02:49.895578+0000","last_unstale":"2024-09-18T08:05:51.422525+0000","last_undegraded":"2024-09-18T08:05:51.422525+0000","last_fullsized":"2024-09-18T08:05:51.422525+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_clean_scrub_stamp":"2024-09-18T08:02:48.870718+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":29,"num_read_kb":19,"num_write":18,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,1],"acting":[4,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":69,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-09-18T08:05:51.421266+0000","last_change":"2024-09-18T08:05:12.622971+0000","last_active":"2024-09-18T08:05:51.421266+0000","last_peered":"2024-09-18T08:05:51.421266+0000","last_clean":"2024-09-18T08:05:51.421266+0000","last_became_active":"2024-09-18T08:04:43.302507+0000","last_became_peered":"2024-09-18T08:04:43.302507+0000","last_unstale":"2024-09-18T08:05:51.421266+0000","last_undegraded":"2024-09-18T08:05:51.421266+0000","last_fullsized":"2024-09-18T08:05:51.421266+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:05:12.622901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:46.860092+0000","last_clean_scrub_stamp":"2024-09-18T08:05:12.622901+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:06:36.771484+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":149,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-09-18T08:05:51.445083+0000","last_change":"2024-09-18T08:05:31.851989+0000","last_active":"2024-09-18T08:05:51.445083+0000","last_peered":"2024-09-18T08:05:51.445083+0000","last_clean":"2024-09-18T08:05:51.445083+0000","last_became_active":"2024-09-18T08:02:51.907196+0000","last_became_peered":"2024-09-18T08:02:51.907196+0000","last_unstale":"2024-09-18T08:05:51.445083+0000","last_undegraded":"2024-09-18T08:05:51.445083+0000","last_fullsized":"2024-09-18T08:05:51.445083+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:05:31.851797+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:50.886830+0000","last_clean_scrub_stamp":"2024-09-18T08:05:31.851797+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:06:59.697560+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":138,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-09-18T08:05:51.433150+0000","last_change":"2024-09-18T08:04:06.709703+0000","last_active":"2024-09-18T08:05:51.433150+0000","last_peered":"2024-09-18T08:05:51.433150+0000","last_clean":"2024-09-18T08:05:51.433150+0000","last_became_active":"2024-09-18T08:02:53.904144+0000","last_became_peered":"2024-09-18T08:02:53.904144+0000","last_unstale":"2024-09-18T08:05:51.433150+0000","last_undegraded":"2024-09-18T08:05:51.433150+0000","last_fullsized":"2024-09-18T08:05:51.433150+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:04:06.709631+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_clean_scrub_stamp":"2024-09-18T08:04:06.709631+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:06:14.883587+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.f","version":"126'8446","reported_seq":8604,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-09-18T08:05:53.307704+0000","last_change":"2024-09-18T08:05:13.677385+0000","last_active":"2024-09-18T08:05:53.307704+0000","last_peered":"2024-09-18T08:05:53.307704+0000","last_clean":"2024-09-18T08:05:53.307704+0000","last_became_active":"2024-09-18T08:03:57.467980+0000","last_became_peered":"2024-09-18T08:03:57.467980+0000","last_unstale":"2024-09-18T08:05:53.307704+0000","last_undegraded":"2024-09-18T08:05:53.307704+0000","last_fullsized":"2024-09-18T08:05:53.307704+0000","mapping_epoch":83,"log_start":"97'2800","ondisk_log_start":"97'2800","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:03:56.449957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:03:56.449957+0000","last_clean_scrub_stamp":"2024-09-18T08:03:56.449957+0000","objects_scrubbed":5349,"log_size":5646,"log_dups_size":0,"ondisk_log_size":5646,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:06:48.316140+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":407896064,"num_objects":6224,"num_object_clones":0,"num_object_copies":18672,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6224,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8446,"num_write_kb":469440,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.c","version":"69'10","reported_seq":160,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-09-18T08:05:51.422065+0000","last_change":"2024-09-18T08:05:41.474883+0000","last_active":"2024-09-18T08:05:51.422065+0000","last_peered":"2024-09-18T08:05:51.422065+0000","last_clean":"2024-09-18T08:05:51.422065+0000","last_became_active":"2024-09-18T08:02:49.892527+0000","last_became_peered":"2024-09-18T08:02:49.892527+0000","last_unstale":"2024-09-18T08:05:51.422065+0000","last_undegraded":"2024-09-18T08:05:51.422065+0000","last_fullsized":"2024-09-18T08:05:51.422065+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'10","last_scrub_stamp":"2024-09-18T08:05:41.474813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_clean_scrub_stamp":"2024-09-18T08:05:41.474813+0000","objects_scrubbed":5,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:06:51.749825+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":149,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-09-18T08:05:51.441469+0000","last_change":"2024-09-18T08:04:24.892153+0000","last_active":"2024-09-18T08:05:51.441469+0000","last_peered":"2024-09-18T08:05:51.441469+0000","last_clean":"2024-09-18T08:05:51.441469+0000","last_became_active":"2024-09-18T08:02:47.881512+0000","last_became_peered":"2024-09-18T08:02:47.881512+0000","last_unstale":"2024-09-18T08:05:51.441469+0000","last_undegraded":"2024-09-18T08:05:51.441469+0000","last_fullsized":"2024-09-18T08:05:51.441469+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:04:24.892079+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:46.860092+0000","last_clean_scrub_stamp":"2024-09-18T08:04:24.892079+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.d","version":"69'8","reported_seq":195,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-09-18T08:05:51.433342+0000","last_change":"2024-09-18T08:05:33.027988+0000","last_active":"2024-09-18T08:05:51.433342+0000","last_peered":"2024-09-18T08:05:51.433342+0000","last_clean":"2024-09-18T08:05:51.433342+0000","last_became_active":"2024-09-18T08:02:51.909011+0000","last_became_peered":"2024-09-18T08:02:51.909011+0000","last_unstale":"2024-09-18T08:05:51.433342+0000","last_undegraded":"2024-09-18T08:05:51.433342+0000","last_fullsized":"2024-09-18T08:05:51.433342+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'8","last_scrub_stamp":"2024-09-18T08:05:33.027783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:50.886830+0000","last_clean_scrub_stamp":"2024-09-18T08:05:33.027783+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:06:41.650338+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":138,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-09-18T08:05:51.422633+0000","last_change":"2024-09-18T08:04:32.377211+0000","last_active":"2024-09-18T08:05:51.422633+0000","last_peered":"2024-09-18T08:05:51.422633+0000","last_clean":"2024-09-18T08:05:51.422633+0000","last_became_active":"2024-09-18T08:02:53.901858+0000","last_became_peered":"2024-09-18T08:02:53.901858+0000","last_unstale":"2024-09-18T08:05:51.422633+0000","last_undegraded":"2024-09-18T08:05:51.422633+0000","last_fullsized":"2024-09-18T08:05:51.422633+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:04:32.377157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_clean_scrub_stamp":"2024-09-18T08:04:32.377157+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.8","version":"126'8506","reported_seq":8603,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-09-18T08:05:53.208410+0000","last_change":"2024-09-18T08:03:57.465051+0000","last_active":"2024-09-18T08:05:53.208410+0000","last_peered":"2024-09-18T08:05:53.208410+0000","last_clean":"2024-09-18T08:05:53.208410+0000","last_became_active":"2024-09-18T08:03:57.464738+0000","last_became_peered":"2024-09-18T08:03:57.464738+0000","last_unstale":"2024-09-18T08:05:53.208410+0000","last_undegraded":"2024-09-18T08:05:53.208410+0000","last_fullsized":"2024-09-18T08:05:53.208410+0000","mapping_epoch":83,"log_start":"91'1600","ondisk_log_start":"91'1600","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:03:56.449957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:03:56.449957+0000","last_clean_scrub_stamp":"2024-09-18T08:03:56.449957+0000","objects_scrubbed":0,"log_size":6906,"log_dups_size":0,"ondisk_log_size":6906,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:06:21.984124+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":415760384,"num_objects":6344,"num_object_clones":0,"num_object_copies":19032,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6344,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8506,"num_write_kb":475200,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,4],"acting":[2,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.b","version":"69'9","reported_seq":161,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-09-18T08:05:51.429751+0000","last_change":"2024-09-18T08:05:05.294031+0000","last_active":"2024-09-18T08:05:51.429751+0000","last_peered":"2024-09-18T08:05:51.429751+0000","last_clean":"2024-09-18T08:05:51.429751+0000","last_became_active":"2024-09-18T08:02:49.897998+0000","last_became_peered":"2024-09-18T08:02:49.897998+0000","last_unstale":"2024-09-18T08:05:51.429751+0000","last_undegraded":"2024-09-18T08:05:51.429751+0000","last_fullsized":"2024-09-18T08:05:51.429751+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'9","last_scrub_stamp":"2024-09-18T08:05:05.293936+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_clean_scrub_stamp":"2024-09-18T08:05:05.293936+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:06:11.067729+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":149,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-09-18T08:05:51.439330+0000","last_change":"2024-09-18T08:04:59.080170+0000","last_active":"2024-09-18T08:05:51.439330+0000","last_peered":"2024-09-18T08:05:51.439330+0000","last_clean":"2024-09-18T08:05:51.439330+0000","last_became_active":"2024-09-18T08:02:47.877161+0000","last_became_peered":"2024-09-18T08:02:47.877161+0000","last_unstale":"2024-09-18T08:05:51.439330+0000","last_undegraded":"2024-09-18T08:05:51.439330+0000","last_fullsized":"2024-09-18T08:05:51.439330+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:04:59.080115+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:46.860092+0000","last_clean_scrub_stamp":"2024-09-18T08:04:59.080115+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:06:27.394180+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":149,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-09-18T08:05:51.435735+0000","last_change":"2024-09-18T08:05:43.068256+0000","last_active":"2024-09-18T08:05:51.435735+0000","last_peered":"2024-09-18T08:05:51.435735+0000","last_clean":"2024-09-18T08:05:51.435735+0000","last_became_active":"2024-09-18T08:02:51.901330+0000","last_became_peered":"2024-09-18T08:02:51.901330+0000","last_unstale":"2024-09-18T08:05:51.435735+0000","last_undegraded":"2024-09-18T08:05:51.435735+0000","last_fullsized":"2024-09-18T08:05:51.435735+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:05:43.067057+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:50.886830+0000","last_clean_scrub_stamp":"2024-09-18T08:05:43.067057+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:07:10.458536+0000","scrub_duration":40,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,3],"acting":[2,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":138,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-09-18T08:05:51.432871+0000","last_change":"2024-09-18T08:05:09.251072+0000","last_active":"2024-09-18T08:05:51.432871+0000","last_peered":"2024-09-18T08:05:51.432871+0000","last_clean":"2024-09-18T08:05:51.432871+0000","last_became_active":"2024-09-18T08:02:53.911068+0000","last_became_peered":"2024-09-18T08:02:53.911068+0000","last_unstale":"2024-09-18T08:05:51.432871+0000","last_undegraded":"2024-09-18T08:05:51.432871+0000","last_fullsized":"2024-09-18T08:05:51.432871+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:05:09.250843+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_clean_scrub_stamp":"2024-09-18T08:05:09.250843+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:06:22.459347+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.9","version":"126'8471","reported_seq":8568,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-09-18T08:05:53.311124+0000","last_change":"2024-09-18T08:03:57.467481+0000","last_active":"2024-09-18T08:05:53.311124+0000","last_peered":"2024-09-18T08:05:53.311124+0000","last_clean":"2024-09-18T08:05:53.311124+0000","last_became_active":"2024-09-18T08:03:57.467065+0000","last_became_peered":"2024-09-18T08:03:57.467065+0000","last_unstale":"2024-09-18T08:05:53.311124+0000","last_undegraded":"2024-09-18T08:05:53.311124+0000","last_fullsized":"2024-09-18T08:05:53.311124+0000","mapping_epoch":83,"log_start":"97'2900","ondisk_log_start":"97'2900","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:03:56.449957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:03:56.449957+0000","last_clean_scrub_stamp":"2024-09-18T08:03:56.449957+0000","objects_scrubbed":0,"log_size":5571,"log_dups_size":0,"ondisk_log_size":5571,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:06:23.505530+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":411500544,"num_objects":6279,"num_object_clones":0,"num_object_copies":18837,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6279,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8471,"num_write_kb":472000,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.a","version":"69'19","reported_seq":112,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-09-18T08:05:51.447009+0000","last_change":"2024-09-18T08:04:43.303845+0000","last_active":"2024-09-18T08:05:51.447009+0000","last_peered":"2024-09-18T08:05:51.447009+0000","last_clean":"2024-09-18T08:05:51.447009+0000","last_became_active":"2024-09-18T08:04:43.303450+0000","last_became_peered":"2024-09-18T08:04:43.303450+0000","last_unstale":"2024-09-18T08:05:51.447009+0000","last_undegraded":"2024-09-18T08:05:51.447009+0000","last_fullsized":"2024-09-18T08:05:51.447009+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'19","last_scrub_stamp":"2024-09-18T08:04:20.391859+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_clean_scrub_stamp":"2024-09-18T08:04:20.391859+0000","objects_scrubbed":0,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:06:49.630107+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":31,"num_read_kb":20,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,7],"acting":[1,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":149,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-09-18T08:05:51.435568+0000","last_change":"2024-09-18T08:04:15.812374+0000","last_active":"2024-09-18T08:05:51.435568+0000","last_peered":"2024-09-18T08:05:51.435568+0000","last_clean":"2024-09-18T08:05:51.435568+0000","last_became_active":"2024-09-18T08:02:47.880645+0000","last_became_peered":"2024-09-18T08:02:47.880645+0000","last_unstale":"2024-09-18T08:05:51.435568+0000","last_undegraded":"2024-09-18T08:05:51.435568+0000","last_fullsized":"2024-09-18T08:05:51.435568+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:04:15.812306+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:46.860092+0000","last_clean_scrub_stamp":"2024-09-18T08:04:15.812306+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:06:48.868926+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":141,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-09-18T08:05:51.435200+0000","last_change":"2024-09-18T08:04:56.004261+0000","last_active":"2024-09-18T08:05:51.435200+0000","last_peered":"2024-09-18T08:05:51.435200+0000","last_clean":"2024-09-18T08:05:51.435200+0000","last_became_active":"2024-09-18T08:02:51.903439+0000","last_became_peered":"2024-09-18T08:02:51.903439+0000","last_unstale":"2024-09-18T08:05:51.435200+0000","last_undegraded":"2024-09-18T08:05:51.435200+0000","last_fullsized":"2024-09-18T08:05:51.435200+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:04:56.004150+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:50.886830+0000","last_clean_scrub_stamp":"2024-09-18T08:04:56.004150+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:05:57.194891+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":138,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-09-18T08:05:51.435345+0000","last_change":"2024-09-18T08:04:17.811662+0000","last_active":"2024-09-18T08:05:51.435345+0000","last_peered":"2024-09-18T08:05:51.435345+0000","last_clean":"2024-09-18T08:05:51.435345+0000","last_became_active":"2024-09-18T08:02:53.912762+0000","last_became_peered":"2024-09-18T08:02:53.912762+0000","last_unstale":"2024-09-18T08:05:51.435345+0000","last_undegraded":"2024-09-18T08:05:51.435345+0000","last_fullsized":"2024-09-18T08:05:51.435345+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:04:17.811597+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_clean_scrub_stamp":"2024-09-18T08:04:17.811597+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.a","version":"126'8614","reported_seq":8711,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-09-18T08:05:53.290852+0000","last_change":"2024-09-18T08:03:57.466483+0000","last_active":"2024-09-18T08:05:53.290852+0000","last_peered":"2024-09-18T08:05:53.290852+0000","last_clean":"2024-09-18T08:05:53.290852+0000","last_became_active":"2024-09-18T08:03:57.466168+0000","last_became_peered":"2024-09-18T08:03:57.466168+0000","last_unstale":"2024-09-18T08:05:53.290852+0000","last_undegraded":"2024-09-18T08:05:53.290852+0000","last_fullsized":"2024-09-18T08:05:53.290852+0000","mapping_epoch":83,"log_start":"100'3900","ondisk_log_start":"100'3900","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:03:56.449957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:03:56.449957+0000","last_clean_scrub_stamp":"2024-09-18T08:03:56.449957+0000","objects_scrubbed":0,"log_size":4714,"log_dups_size":0,"ondisk_log_size":4714,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:06:21.099292+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":413138944,"num_objects":6304,"num_object_clones":0,"num_object_copies":18912,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6304,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8614,"num_write_kb":477376,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.9","version":"69'12","reported_seq":176,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-09-18T08:05:51.421231+0000","last_change":"2024-09-18T08:05:33.544197+0000","last_active":"2024-09-18T08:05:51.421231+0000","last_peered":"2024-09-18T08:05:51.421231+0000","last_clean":"2024-09-18T08:05:51.421231+0000","last_became_active":"2024-09-18T08:02:49.894762+0000","last_became_peered":"2024-09-18T08:02:49.894762+0000","last_unstale":"2024-09-18T08:05:51.421231+0000","last_undegraded":"2024-09-18T08:05:51.421231+0000","last_fullsized":"2024-09-18T08:05:51.421231+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'12","last_scrub_stamp":"2024-09-18T08:05:33.544135+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_clean_scrub_stamp":"2024-09-18T08:05:33.544135+0000","objects_scrubbed":5,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:06:52.810406+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":25,"num_read_kb":16,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":157,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-09-18T08:05:51.436485+0000","last_change":"2024-09-18T08:05:39.906123+0000","last_active":"2024-09-18T08:05:51.436485+0000","last_peered":"2024-09-18T08:05:51.436485+0000","last_clean":"2024-09-18T08:05:51.436485+0000","last_became_active":"2024-09-18T08:02:47.884052+0000","last_became_peered":"2024-09-18T08:02:47.884052+0000","last_unstale":"2024-09-18T08:05:51.436485+0000","last_undegraded":"2024-09-18T08:05:51.436485+0000","last_fullsized":"2024-09-18T08:05:51.436485+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:05:39.906062+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:46.860092+0000","last_clean_scrub_stamp":"2024-09-18T08:05:39.906062+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:07:03.409405+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":141,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-09-18T08:05:51.433529+0000","last_change":"2024-09-18T08:05:06.828638+0000","last_active":"2024-09-18T08:05:51.433529+0000","last_peered":"2024-09-18T08:05:51.433529+0000","last_clean":"2024-09-18T08:05:51.433529+0000","last_became_active":"2024-09-18T08:02:51.902163+0000","last_became_peered":"2024-09-18T08:02:51.902163+0000","last_unstale":"2024-09-18T08:05:51.433529+0000","last_undegraded":"2024-09-18T08:05:51.433529+0000","last_fullsized":"2024-09-18T08:05:51.433529+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:05:06.828408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:50.886830+0000","last_clean_scrub_stamp":"2024-09-18T08:05:06.828408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:06:17.532271+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":138,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-09-18T08:05:51.441160+0000","last_change":"2024-09-18T08:05:44.074994+0000","last_active":"2024-09-18T08:05:51.441160+0000","last_peered":"2024-09-18T08:05:51.441160+0000","last_clean":"2024-09-18T08:05:51.441160+0000","last_became_active":"2024-09-18T08:02:53.907412+0000","last_became_peered":"2024-09-18T08:02:53.907412+0000","last_unstale":"2024-09-18T08:05:51.441160+0000","last_undegraded":"2024-09-18T08:05:51.441160+0000","last_fullsized":"2024-09-18T08:05:51.441160+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:05:44.074555+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_clean_scrub_stamp":"2024-09-18T08:05:44.074555+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:07:10.508813+0000","scrub_duration":23,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.b","version":"126'8432","reported_seq":8606,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-09-18T08:05:53.152975+0000","last_change":"2024-09-18T08:05:28.826831+0000","last_active":"2024-09-18T08:05:53.152975+0000","last_peered":"2024-09-18T08:05:53.152975+0000","last_clean":"2024-09-18T08:05:53.152975+0000","last_became_active":"2024-09-18T08:03:57.466106+0000","last_became_peered":"2024-09-18T08:03:57.466106+0000","last_unstale":"2024-09-18T08:05:53.152975+0000","last_undegraded":"2024-09-18T08:05:53.152975+0000","last_fullsized":"2024-09-18T08:05:53.152975+0000","mapping_epoch":83,"log_start":"98'3400","ondisk_log_start":"98'3400","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"118'6870","last_scrub_stamp":"2024-09-18T08:05:28.826628+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:03:56.449957+0000","last_clean_scrub_stamp":"2024-09-18T08:05:28.826628+0000","objects_scrubbed":6831,"log_size":5032,"log_dups_size":0,"ondisk_log_size":5032,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:06:33.253487+0000","scrub_duration":1959,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":401866752,"num_objects":6132,"num_object_clones":0,"num_object_copies":18396,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6132,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8432,"num_write_kb":466048,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6],"acting":[1,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.8","version":"69'15","reported_seq":170,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-09-18T08:05:51.438814+0000","last_change":"2024-09-18T08:04:26.859424+0000","last_active":"2024-09-18T08:05:51.438814+0000","last_peered":"2024-09-18T08:05:51.438814+0000","last_clean":"2024-09-18T08:05:51.438814+0000","last_became_active":"2024-09-18T08:02:49.888799+0000","last_became_peered":"2024-09-18T08:02:49.888799+0000","last_unstale":"2024-09-18T08:05:51.438814+0000","last_undegraded":"2024-09-18T08:05:51.438814+0000","last_fullsized":"2024-09-18T08:05:51.438814+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'15","last_scrub_stamp":"2024-09-18T08:04:26.859044+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_clean_scrub_stamp":"2024-09-18T08:04:26.859044+0000","objects_scrubbed":7,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.f","version":"62'2","reported_seq":169,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-09-18T08:05:51.435308+0000","last_change":"2024-09-18T08:05:11.906358+0000","last_active":"2024-09-18T08:05:51.435308+0000","last_peered":"2024-09-18T08:05:51.435308+0000","last_clean":"2024-09-18T08:05:51.435308+0000","last_became_active":"2024-09-18T08:02:47.881165+0000","last_became_peered":"2024-09-18T08:02:47.881165+0000","last_unstale":"2024-09-18T08:05:51.435308+0000","last_undegraded":"2024-09-18T08:05:51.435308+0000","last_fullsized":"2024-09-18T08:05:51.435308+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"62'2","last_scrub_stamp":"2024-09-18T08:05:11.906308+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:46.860092+0000","last_clean_scrub_stamp":"2024-09-18T08:05:11.906308+0000","objects_scrubbed":2,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:06:20.720016+0000","scrub_duration":24,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":4,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.9","version":"69'8","reported_seq":190,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-09-18T08:05:51.434819+0000","last_change":"2024-09-18T08:03:52.572824+0000","last_active":"2024-09-18T08:05:51.434819+0000","last_peered":"2024-09-18T08:05:51.434819+0000","last_clean":"2024-09-18T08:05:51.434819+0000","last_became_active":"2024-09-18T08:02:51.905059+0000","last_became_peered":"2024-09-18T08:02:51.905059+0000","last_unstale":"2024-09-18T08:05:51.434819+0000","last_undegraded":"2024-09-18T08:05:51.434819+0000","last_fullsized":"2024-09-18T08:05:51.434819+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'8","last_scrub_stamp":"2024-09-18T08:03:52.572757+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:50.886830+0000","last_clean_scrub_stamp":"2024-09-18T08:03:52.572757+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:06:15.853616+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":137,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-09-18T08:05:51.439147+0000","last_change":"2024-09-18T08:05:07.879565+0000","last_active":"2024-09-18T08:05:51.439147+0000","last_peered":"2024-09-18T08:05:51.439147+0000","last_clean":"2024-09-18T08:05:51.439147+0000","last_became_active":"2024-09-18T08:02:54.045845+0000","last_became_peered":"2024-09-18T08:02:54.045845+0000","last_unstale":"2024-09-18T08:05:51.439147+0000","last_undegraded":"2024-09-18T08:05:51.439147+0000","last_fullsized":"2024-09-18T08:05:51.439147+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:05:07.879501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_clean_scrub_stamp":"2024-09-18T08:05:07.879501+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:06:17.801897+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":61,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-09-18T08:05:51.430088+0000","last_change":"2024-09-18T08:04:43.302438+0000","last_active":"2024-09-18T08:05:51.430088+0000","last_peered":"2024-09-18T08:05:51.430088+0000","last_clean":"2024-09-18T08:05:51.430088+0000","last_became_active":"2024-09-18T08:04:43.302022+0000","last_became_peered":"2024-09-18T08:04:43.302022+0000","last_unstale":"2024-09-18T08:05:51.430088+0000","last_undegraded":"2024-09-18T08:05:51.430088+0000","last_fullsized":"2024-09-18T08:05:51.430088+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:04:16.378205+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:46.860092+0000","last_clean_scrub_stamp":"2024-09-18T08:04:16.378205+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.17","version":"69'6","reported_seq":162,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-09-18T08:05:51.450367+0000","last_change":"2024-09-18T08:05:26.228344+0000","last_active":"2024-09-18T08:05:51.450367+0000","last_peered":"2024-09-18T08:05:51.450367+0000","last_clean":"2024-09-18T08:05:51.450367+0000","last_became_active":"2024-09-18T08:02:49.895067+0000","last_became_peered":"2024-09-18T08:02:49.895067+0000","last_unstale":"2024-09-18T08:05:51.450367+0000","last_undegraded":"2024-09-18T08:05:51.450367+0000","last_fullsized":"2024-09-18T08:05:51.450367+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'6","last_scrub_stamp":"2024-09-18T08:05:26.227814+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_clean_scrub_stamp":"2024-09-18T08:05:26.227814+0000","objects_scrubbed":3,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:06:36.064005+0000","scrub_duration":18,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":9,"num_read_kb":6,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":149,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-09-18T08:05:51.440785+0000","last_change":"2024-09-18T08:05:45.812436+0000","last_active":"2024-09-18T08:05:51.440785+0000","last_peered":"2024-09-18T08:05:51.440785+0000","last_clean":"2024-09-18T08:05:51.440785+0000","last_became_active":"2024-09-18T08:02:51.912557+0000","last_became_peered":"2024-09-18T08:02:51.912557+0000","last_unstale":"2024-09-18T08:05:51.440785+0000","last_undegraded":"2024-09-18T08:05:51.440785+0000","last_fullsized":"2024-09-18T08:05:51.440785+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:05:45.812389+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:50.886830+0000","last_clean_scrub_stamp":"2024-09-18T08:05:45.812389+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:07:01.523740+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,1],"acting":[5,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":138,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-09-18T08:05:51.435197+0000","last_change":"2024-09-18T08:05:00.883597+0000","last_active":"2024-09-18T08:05:51.435197+0000","last_peered":"2024-09-18T08:05:51.435197+0000","last_clean":"2024-09-18T08:05:51.435197+0000","last_became_active":"2024-09-18T08:02:54.048959+0000","last_became_peered":"2024-09-18T08:02:54.048959+0000","last_unstale":"2024-09-18T08:05:51.435197+0000","last_undegraded":"2024-09-18T08:05:51.435197+0000","last_fullsized":"2024-09-18T08:05:51.435197+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:05:00.883555+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_clean_scrub_stamp":"2024-09-18T08:05:00.883555+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:06:25.986166+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"69'9","reported_seq":161,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-09-18T08:05:51.431804+0000","last_change":"2024-09-18T08:05:00.265939+0000","last_active":"2024-09-18T08:05:51.431804+0000","last_peered":"2024-09-18T08:05:51.431804+0000","last_clean":"2024-09-18T08:05:51.431804+0000","last_became_active":"2024-09-18T08:02:49.891567+0000","last_became_peered":"2024-09-18T08:02:49.891567+0000","last_unstale":"2024-09-18T08:05:51.431804+0000","last_undegraded":"2024-09-18T08:05:51.431804+0000","last_fullsized":"2024-09-18T08:05:51.431804+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'9","last_scrub_stamp":"2024-09-18T08:05:00.265738+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_clean_scrub_stamp":"2024-09-18T08:05:00.265738+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:06:27.928037+0000","scrub_duration":17,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":149,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-09-18T08:05:51.434876+0000","last_change":"2024-09-18T08:03:49.591148+0000","last_active":"2024-09-18T08:05:51.434876+0000","last_peered":"2024-09-18T08:05:51.434876+0000","last_clean":"2024-09-18T08:05:51.434876+0000","last_became_active":"2024-09-18T08:02:47.885790+0000","last_became_peered":"2024-09-18T08:02:47.885790+0000","last_unstale":"2024-09-18T08:05:51.434876+0000","last_undegraded":"2024-09-18T08:05:51.434876+0000","last_fullsized":"2024-09-18T08:05:51.434876+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:03:49.591068+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:46.860092+0000","last_clean_scrub_stamp":"2024-09-18T08:03:49.591068+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:06:17.821808+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":141,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-09-18T08:05:51.450241+0000","last_change":"2024-09-18T08:05:42.061649+0000","last_active":"2024-09-18T08:05:51.450241+0000","last_peered":"2024-09-18T08:05:51.450241+0000","last_clean":"2024-09-18T08:05:51.450241+0000","last_became_active":"2024-09-18T08:02:51.907361+0000","last_became_peered":"2024-09-18T08:02:51.907361+0000","last_unstale":"2024-09-18T08:05:51.450241+0000","last_undegraded":"2024-09-18T08:05:51.450241+0000","last_fullsized":"2024-09-18T08:05:51.450241+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:05:42.061452+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:50.886830+0000","last_clean_scrub_stamp":"2024-09-18T08:05:42.061452+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:07:05.139088+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":138,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-09-18T08:05:51.432780+0000","last_change":"2024-09-18T08:05:07.780899+0000","last_active":"2024-09-18T08:05:51.432780+0000","last_peered":"2024-09-18T08:05:51.432780+0000","last_clean":"2024-09-18T08:05:51.432780+0000","last_became_active":"2024-09-18T08:02:53.908917+0000","last_became_peered":"2024-09-18T08:02:53.908917+0000","last_unstale":"2024-09-18T08:05:51.432780+0000","last_undegraded":"2024-09-18T08:05:51.432780+0000","last_fullsized":"2024-09-18T08:05:51.432780+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:05:07.780816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_clean_scrub_stamp":"2024-09-18T08:05:07.780816+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:06:31.045662+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.15","version":"69'9","reported_seq":169,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-09-18T08:05:51.439507+0000","last_change":"2024-09-18T08:05:36.896299+0000","last_active":"2024-09-18T08:05:51.439507+0000","last_peered":"2024-09-18T08:05:51.439507+0000","last_clean":"2024-09-18T08:05:51.439507+0000","last_became_active":"2024-09-18T08:02:49.890497+0000","last_became_peered":"2024-09-18T08:02:49.890497+0000","last_unstale":"2024-09-18T08:05:51.439507+0000","last_undegraded":"2024-09-18T08:05:51.439507+0000","last_fullsized":"2024-09-18T08:05:51.439507+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'9","last_scrub_stamp":"2024-09-18T08:05:36.896238+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_clean_scrub_stamp":"2024-09-18T08:05:36.896238+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:06:52.292485+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":149,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-09-18T08:05:51.430599+0000","last_change":"2024-09-18T08:04:23.160350+0000","last_active":"2024-09-18T08:05:51.430599+0000","last_peered":"2024-09-18T08:05:51.430599+0000","last_clean":"2024-09-18T08:05:51.430599+0000","last_became_active":"2024-09-18T08:02:47.875983+0000","last_became_peered":"2024-09-18T08:02:47.875983+0000","last_unstale":"2024-09-18T08:05:51.430599+0000","last_undegraded":"2024-09-18T08:05:51.430599+0000","last_fullsized":"2024-09-18T08:05:51.430599+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:04:23.160277+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:46.860092+0000","last_clean_scrub_stamp":"2024-09-18T08:04:23.160277+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.14","version":"69'8","reported_seq":187,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-09-18T08:05:51.443058+0000","last_change":"2024-09-18T08:05:01.083803+0000","last_active":"2024-09-18T08:05:51.443058+0000","last_peered":"2024-09-18T08:05:51.443058+0000","last_clean":"2024-09-18T08:05:51.443058+0000","last_became_active":"2024-09-18T08:02:51.909513+0000","last_became_peered":"2024-09-18T08:02:51.909513+0000","last_unstale":"2024-09-18T08:05:51.443058+0000","last_undegraded":"2024-09-18T08:05:51.443058+0000","last_fullsized":"2024-09-18T08:05:51.443058+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'8","last_scrub_stamp":"2024-09-18T08:05:01.082570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:50.886830+0000","last_clean_scrub_stamp":"2024-09-18T08:05:01.082570+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:06:22.559429+0000","scrub_duration":17,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":138,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-09-18T08:05:51.422887+0000","last_change":"2024-09-18T08:05:09.654918+0000","last_active":"2024-09-18T08:05:51.422887+0000","last_peered":"2024-09-18T08:05:51.422887+0000","last_clean":"2024-09-18T08:05:51.422887+0000","last_became_active":"2024-09-18T08:02:53.907428+0000","last_became_peered":"2024-09-18T08:02:53.907428+0000","last_unstale":"2024-09-18T08:05:51.422887+0000","last_undegraded":"2024-09-18T08:05:51.422887+0000","last_fullsized":"2024-09-18T08:05:51.422887+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:05:09.654832+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_clean_scrub_stamp":"2024-09-18T08:05:09.654832+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:06:19.204016+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.14","version":"69'10","reported_seq":179,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-09-18T08:05:51.443422+0000","last_change":"2024-09-18T08:04:18.172364+0000","last_active":"2024-09-18T08:05:51.443422+0000","last_peered":"2024-09-18T08:05:51.443422+0000","last_clean":"2024-09-18T08:05:51.443422+0000","last_became_active":"2024-09-18T08:04:18.150968+0000","last_became_peered":"2024-09-18T08:04:18.150968+0000","last_unstale":"2024-09-18T08:05:51.443422+0000","last_undegraded":"2024-09-18T08:05:51.443422+0000","last_fullsized":"2024-09-18T08:05:51.443422+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'10","last_scrub_stamp":"2024-09-18T08:04:16.940778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_clean_scrub_stamp":"2024-09-18T08:04:16.940778+0000","objects_scrubbed":5,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,7],"acting":[3,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":149,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-09-18T08:05:51.435292+0000","last_change":"2024-09-18T08:05:08.855132+0000","last_active":"2024-09-18T08:05:51.435292+0000","last_peered":"2024-09-18T08:05:51.435292+0000","last_clean":"2024-09-18T08:05:51.435292+0000","last_became_active":"2024-09-18T08:02:47.882065+0000","last_became_peered":"2024-09-18T08:02:47.882065+0000","last_unstale":"2024-09-18T08:05:51.435292+0000","last_undegraded":"2024-09-18T08:05:51.435292+0000","last_fullsized":"2024-09-18T08:05:51.435292+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:05:08.855064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:46.860092+0000","last_clean_scrub_stamp":"2024-09-18T08:05:08.855064+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:06:28.592575+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.15","version":"69'8","reported_seq":195,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-09-18T08:05:51.439440+0000","last_change":"2024-09-18T08:05:27.998896+0000","last_active":"2024-09-18T08:05:51.439440+0000","last_peered":"2024-09-18T08:05:51.439440+0000","last_clean":"2024-09-18T08:05:51.439440+0000","last_became_active":"2024-09-18T08:02:51.914135+0000","last_became_peered":"2024-09-18T08:02:51.914135+0000","last_unstale":"2024-09-18T08:05:51.439440+0000","last_undegraded":"2024-09-18T08:05:51.439440+0000","last_fullsized":"2024-09-18T08:05:51.439440+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'8","last_scrub_stamp":"2024-09-18T08:05:27.998833+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:50.886830+0000","last_clean_scrub_stamp":"2024-09-18T08:05:27.998833+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:06:41.603719+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":138,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-09-18T08:05:51.431368+0000","last_change":"2024-09-18T08:05:03.302755+0000","last_active":"2024-09-18T08:05:51.431368+0000","last_peered":"2024-09-18T08:05:51.431368+0000","last_clean":"2024-09-18T08:05:51.431368+0000","last_became_active":"2024-09-18T08:02:53.912112+0000","last_became_peered":"2024-09-18T08:02:53.912112+0000","last_unstale":"2024-09-18T08:05:51.431368+0000","last_undegraded":"2024-09-18T08:05:51.431368+0000","last_fullsized":"2024-09-18T08:05:51.431368+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:05:03.302444+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_clean_scrub_stamp":"2024-09-18T08:05:03.302444+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:06:25.825825+0000","scrub_duration":16,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.13","version":"69'11","reported_seq":164,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-09-18T08:05:51.421669+0000","last_change":"2024-09-18T08:04:34.451463+0000","last_active":"2024-09-18T08:05:51.421669+0000","last_peered":"2024-09-18T08:05:51.421669+0000","last_clean":"2024-09-18T08:05:51.421669+0000","last_became_active":"2024-09-18T08:02:49.896949+0000","last_became_peered":"2024-09-18T08:02:49.896949+0000","last_unstale":"2024-09-18T08:05:51.421669+0000","last_undegraded":"2024-09-18T08:05:51.421669+0000","last_fullsized":"2024-09-18T08:05:51.421669+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'11","last_scrub_stamp":"2024-09-18T08:04:34.451379+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_clean_scrub_stamp":"2024-09-18T08:04:34.451379+0000","objects_scrubbed":5,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:06:00.685349+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":149,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-09-18T08:05:51.422238+0000","last_change":"2024-09-18T08:05:00.588793+0000","last_active":"2024-09-18T08:05:51.422238+0000","last_peered":"2024-09-18T08:05:51.422238+0000","last_clean":"2024-09-18T08:05:51.422238+0000","last_became_active":"2024-09-18T08:02:47.879771+0000","last_became_peered":"2024-09-18T08:02:47.879771+0000","last_unstale":"2024-09-18T08:05:51.422238+0000","last_undegraded":"2024-09-18T08:05:51.422238+0000","last_fullsized":"2024-09-18T08:05:51.422238+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:05:00.588728+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:46.860092+0000","last_clean_scrub_stamp":"2024-09-18T08:05:00.588728+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:06:15.909247+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":149,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-09-18T08:05:51.446920+0000","last_change":"2024-09-18T08:05:29.896334+0000","last_active":"2024-09-18T08:05:51.446920+0000","last_peered":"2024-09-18T08:05:51.446920+0000","last_clean":"2024-09-18T08:05:51.446920+0000","last_became_active":"2024-09-18T08:02:51.910752+0000","last_became_peered":"2024-09-18T08:02:51.910752+0000","last_unstale":"2024-09-18T08:05:51.446920+0000","last_undegraded":"2024-09-18T08:05:51.446920+0000","last_fullsized":"2024-09-18T08:05:51.446920+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:05:29.888927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:50.886830+0000","last_clean_scrub_stamp":"2024-09-18T08:05:29.888927+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:06:47.803144+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":138,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-09-18T08:05:51.439639+0000","last_change":"2024-09-18T08:04:18.998630+0000","last_active":"2024-09-18T08:05:51.439639+0000","last_peered":"2024-09-18T08:05:51.439639+0000","last_clean":"2024-09-18T08:05:51.439639+0000","last_became_active":"2024-09-18T08:02:53.911429+0000","last_became_peered":"2024-09-18T08:02:53.911429+0000","last_unstale":"2024-09-18T08:05:51.439639+0000","last_undegraded":"2024-09-18T08:05:51.439639+0000","last_fullsized":"2024-09-18T08:05:51.439639+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:04:18.998565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_clean_scrub_stamp":"2024-09-18T08:04:18.998565+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":84,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.12","version":"69'9","reported_seq":153,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-09-18T08:05:51.447200+0000","last_change":"2024-09-18T08:02:49.893472+0000","last_active":"2024-09-18T08:05:51.447200+0000","last_peered":"2024-09-18T08:05:51.447200+0000","last_clean":"2024-09-18T08:05:51.447200+0000","last_became_active":"2024-09-18T08:02:49.885950+0000","last_became_peered":"2024-09-18T08:02:49.885950+0000","last_unstale":"2024-09-18T08:05:51.447200+0000","last_undegraded":"2024-09-18T08:05:51.447200+0000","last_fullsized":"2024-09-18T08:05:51.447200+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_clean_scrub_stamp":"2024-09-18T08:02:48.870718+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:06:15.746930+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":157,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-09-18T08:05:51.435520+0000","last_change":"2024-09-18T08:05:30.820244+0000","last_active":"2024-09-18T08:05:51.435520+0000","last_peered":"2024-09-18T08:05:51.435520+0000","last_clean":"2024-09-18T08:05:51.435520+0000","last_became_active":"2024-09-18T08:02:47.882096+0000","last_became_peered":"2024-09-18T08:02:47.882096+0000","last_unstale":"2024-09-18T08:05:51.435520+0000","last_undegraded":"2024-09-18T08:05:51.435520+0000","last_fullsized":"2024-09-18T08:05:51.435520+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:05:30.820200+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:46.860092+0000","last_clean_scrub_stamp":"2024-09-18T08:05:30.820200+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:06:54.287921+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":141,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-09-18T08:05:51.441252+0000","last_change":"2024-09-18T08:05:43.165522+0000","last_active":"2024-09-18T08:05:51.441252+0000","last_peered":"2024-09-18T08:05:51.441252+0000","last_clean":"2024-09-18T08:05:51.441252+0000","last_became_active":"2024-09-18T08:02:51.906277+0000","last_became_peered":"2024-09-18T08:02:51.906277+0000","last_unstale":"2024-09-18T08:05:51.441252+0000","last_undegraded":"2024-09-18T08:05:51.441252+0000","last_fullsized":"2024-09-18T08:05:51.441252+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:05:43.165258+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:50.886830+0000","last_clean_scrub_stamp":"2024-09-18T08:05:43.165258+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:07:08.715609+0000","scrub_duration":89,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.10","version":"69'1","reported_seq":134,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-09-18T08:05:51.429631+0000","last_change":"2024-09-18T08:05:51.429631+0000","last_active":"2024-09-18T08:05:51.429631+0000","last_peered":"2024-09-18T08:05:51.429631+0000","last_clean":"2024-09-18T08:05:51.429631+0000","last_became_active":"2024-09-18T08:02:53.912699+0000","last_became_peered":"2024-09-18T08:02:53.912699+0000","last_unstale":"2024-09-18T08:05:51.429631+0000","last_undegraded":"2024-09-18T08:05:51.429631+0000","last_fullsized":"2024-09-18T08:05:51.429631+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_clean_scrub_stamp":"2024-09-18T08:02:52.890475+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:06:17.399772+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.11","version":"69'11","reported_seq":172,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-09-18T08:05:51.439772+0000","last_change":"2024-09-18T08:05:13.037686+0000","last_active":"2024-09-18T08:05:51.439772+0000","last_peered":"2024-09-18T08:05:51.439772+0000","last_clean":"2024-09-18T08:05:51.439772+0000","last_became_active":"2024-09-18T08:02:49.889611+0000","last_became_peered":"2024-09-18T08:02:49.889611+0000","last_unstale":"2024-09-18T08:05:51.439772+0000","last_undegraded":"2024-09-18T08:05:51.439772+0000","last_fullsized":"2024-09-18T08:05:51.439772+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'11","last_scrub_stamp":"2024-09-18T08:05:13.037615+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_clean_scrub_stamp":"2024-09-18T08:05:13.037615+0000","objects_scrubbed":5,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:06:30.232726+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":149,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-09-18T08:05:51.440823+0000","last_change":"2024-09-18T08:03:49.960690+0000","last_active":"2024-09-18T08:05:51.440823+0000","last_peered":"2024-09-18T08:05:51.440823+0000","last_clean":"2024-09-18T08:05:51.440823+0000","last_became_active":"2024-09-18T08:02:47.878886+0000","last_became_peered":"2024-09-18T08:02:47.878886+0000","last_unstale":"2024-09-18T08:05:51.440823+0000","last_undegraded":"2024-09-18T08:05:51.440823+0000","last_fullsized":"2024-09-18T08:05:51.440823+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:03:49.960617+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:46.860092+0000","last_clean_scrub_stamp":"2024-09-18T08:03:49.960617+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:06:13.978672+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":141,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-09-18T08:05:51.434484+0000","last_change":"2024-09-18T08:04:12.846215+0000","last_active":"2024-09-18T08:05:51.434484+0000","last_peered":"2024-09-18T08:05:51.434484+0000","last_clean":"2024-09-18T08:05:51.434484+0000","last_became_active":"2024-09-18T08:02:51.907762+0000","last_became_peered":"2024-09-18T08:02:51.907762+0000","last_unstale":"2024-09-18T08:05:51.434484+0000","last_undegraded":"2024-09-18T08:05:51.434484+0000","last_fullsized":"2024-09-18T08:05:51.434484+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:04:12.846141+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:50.886830+0000","last_clean_scrub_stamp":"2024-09-18T08:04:12.846141+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:06:18.783707+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":129,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-09-18T08:05:51.441389+0000","last_change":"2024-09-18T08:02:54.047687+0000","last_active":"2024-09-18T08:05:51.441389+0000","last_peered":"2024-09-18T08:05:51.441389+0000","last_clean":"2024-09-18T08:05:51.441389+0000","last_became_active":"2024-09-18T08:02:54.047181+0000","last_became_peered":"2024-09-18T08:02:54.047181+0000","last_unstale":"2024-09-18T08:05:51.441389+0000","last_undegraded":"2024-09-18T08:05:51.441389+0000","last_fullsized":"2024-09-18T08:05:51.441389+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_clean_scrub_stamp":"2024-09-18T08:02:52.890475+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:06:49.067042+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.10","version":"69'4","reported_seq":151,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-09-18T08:05:51.440238+0000","last_change":"2024-09-18T08:04:13.000124+0000","last_active":"2024-09-18T08:05:51.440238+0000","last_peered":"2024-09-18T08:05:51.440238+0000","last_clean":"2024-09-18T08:05:51.440238+0000","last_became_active":"2024-09-18T08:02:49.890126+0000","last_became_peered":"2024-09-18T08:02:49.890126+0000","last_unstale":"2024-09-18T08:05:51.440238+0000","last_undegraded":"2024-09-18T08:05:51.440238+0000","last_fullsized":"2024-09-18T08:05:51.440238+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'4","last_scrub_stamp":"2024-09-18T08:04:12.999978+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_clean_scrub_stamp":"2024-09-18T08:04:12.999978+0000","objects_scrubbed":2,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":6,"num_read_kb":4,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":149,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-09-18T08:05:51.431450+0000","last_change":"2024-09-18T08:04:10.234061+0000","last_active":"2024-09-18T08:05:51.431450+0000","last_peered":"2024-09-18T08:05:51.431450+0000","last_clean":"2024-09-18T08:05:51.431450+0000","last_became_active":"2024-09-18T08:02:47.877616+0000","last_became_peered":"2024-09-18T08:02:47.877616+0000","last_unstale":"2024-09-18T08:05:51.431450+0000","last_undegraded":"2024-09-18T08:05:51.431450+0000","last_fullsized":"2024-09-18T08:05:51.431450+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:04:10.233969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:46.860092+0000","last_clean_scrub_stamp":"2024-09-18T08:04:10.233969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":69,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-09-18T08:05:51.420887+0000","last_change":"2024-09-18T08:05:11.591350+0000","last_active":"2024-09-18T08:05:51.420887+0000","last_peered":"2024-09-18T08:05:51.420887+0000","last_clean":"2024-09-18T08:05:51.420887+0000","last_became_active":"2024-09-18T08:04:43.299415+0000","last_became_peered":"2024-09-18T08:04:43.299415+0000","last_unstale":"2024-09-18T08:05:51.420887+0000","last_undegraded":"2024-09-18T08:05:51.420887+0000","last_fullsized":"2024-09-18T08:05:51.420887+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:05:11.591275+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:50.886830+0000","last_clean_scrub_stamp":"2024-09-18T08:05:11.591275+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:06:32.194148+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,7],"acting":[4,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":146,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-09-18T08:05:51.434613+0000","last_change":"2024-09-18T08:05:32.798628+0000","last_active":"2024-09-18T08:05:51.434613+0000","last_peered":"2024-09-18T08:05:51.434613+0000","last_clean":"2024-09-18T08:05:51.434613+0000","last_became_active":"2024-09-18T08:02:53.909146+0000","last_became_peered":"2024-09-18T08:02:53.909146+0000","last_unstale":"2024-09-18T08:05:51.434613+0000","last_undegraded":"2024-09-18T08:05:51.434613+0000","last_fullsized":"2024-09-18T08:05:51.434613+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:05:32.798564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_clean_scrub_stamp":"2024-09-18T08:05:32.798564+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:06:38.927572+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,4],"acting":[7,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":130,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-09-18T08:05:51.443071+0000","last_change":"2024-09-18T08:02:53.907055+0000","last_active":"2024-09-18T08:05:51.443071+0000","last_peered":"2024-09-18T08:05:51.443071+0000","last_clean":"2024-09-18T08:05:51.443071+0000","last_became_active":"2024-09-18T08:02:53.906726+0000","last_became_peered":"2024-09-18T08:02:53.906726+0000","last_unstale":"2024-09-18T08:05:51.443071+0000","last_undegraded":"2024-09-18T08:05:51.443071+0000","last_fullsized":"2024-09-18T08:05:51.443071+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_clean_scrub_stamp":"2024-09-18T08:02:52.890475+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:06:19.752175+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":149,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-09-18T08:05:51.440412+0000","last_change":"2024-09-18T08:05:02.063591+0000","last_active":"2024-09-18T08:05:51.440412+0000","last_peered":"2024-09-18T08:05:51.440412+0000","last_clean":"2024-09-18T08:05:51.440412+0000","last_became_active":"2024-09-18T08:02:47.878922+0000","last_became_peered":"2024-09-18T08:02:47.878922+0000","last_unstale":"2024-09-18T08:05:51.440412+0000","last_undegraded":"2024-09-18T08:05:51.440412+0000","last_fullsized":"2024-09-18T08:05:51.440412+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:05:02.063478+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:46.860092+0000","last_clean_scrub_stamp":"2024-09-18T08:05:02.063478+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:06:29.652021+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"69'11","reported_seq":100,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-09-18T08:05:51.439652+0000","last_change":"2024-09-18T08:04:43.304006+0000","last_active":"2024-09-18T08:05:51.439652+0000","last_peered":"2024-09-18T08:05:51.439652+0000","last_clean":"2024-09-18T08:05:51.439652+0000","last_became_active":"2024-09-18T08:04:43.303640+0000","last_became_peered":"2024-09-18T08:04:43.303640+0000","last_unstale":"2024-09-18T08:05:51.439652+0000","last_undegraded":"2024-09-18T08:05:51.439652+0000","last_fullsized":"2024-09-18T08:05:51.439652+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_clean_scrub_stamp":"2024-09-18T08:02:48.870718+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:06:47.748355+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":19,"num_read_kb":12,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,1],"acting":[5,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":141,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-09-18T08:05:51.431688+0000","last_change":"2024-09-18T08:04:26.117117+0000","last_active":"2024-09-18T08:05:51.431688+0000","last_peered":"2024-09-18T08:05:51.431688+0000","last_clean":"2024-09-18T08:05:51.431688+0000","last_became_active":"2024-09-18T08:02:51.900300+0000","last_became_peered":"2024-09-18T08:02:51.900300+0000","last_unstale":"2024-09-18T08:05:51.431688+0000","last_undegraded":"2024-09-18T08:05:51.431688+0000","last_fullsized":"2024-09-18T08:05:51.431688+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:04:26.117051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:50.886830+0000","last_clean_scrub_stamp":"2024-09-18T08:04:26.117051+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:05:55.198838+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":7,"num_pg":16,"stat_sum":{"num_bytes":6587875352,"num_objects":100524,"num_object_clones":0,"num_object_copies":301572,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":100524,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":136054,"num_write_kb":7570433,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":19773468672,"data_stored":19773456456,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":86354,"ondisk_log_size":86354,"up":48,"acting":48,"num_store_stats":8},{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":416,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":65,"ondisk_log_size":65,"up":96,"acting":96,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":178,"num_object_clones":0,"num_object_copies":534,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":178,"num_whiteouts":0,"num_read":694,"num_read_kb":451,"num_write":417,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":17,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":417792,"data_stored":11106,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":394,"ondisk_log_size":394,"up":96,"acting":96,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":1613,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":24,"num_read_kb":24,"num_write":10,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":4839,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":6,"ondisk_log_size":6,"up":96,"acting":96,"num_store_stats":8},{"poolid":2,"num_pg":3,"stat_sum":{"num_bytes":408,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":180,"num_read_kb":175,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":8,"ondisk_log_size":8,"up":9,"acting":9,"num_store_stats":7},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":231,"num_write":113,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":590368,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":79,"ondisk_log_size":79,"up":3,"acting":3,"num_store_stats":3}],"osd_stats":[{"osd":7,"up_from":56,"seq":240518168617,"num_pgs":58,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2091168,"kb_used_data":2023884,"kb_used_omap":1,"kb_used_meta":67262,"kb_avail":91653984,"statfs":{"total":95995035648,"available":93853679616,"internally_reserved":0,"allocated":2072457216,"data_stored":2072060574,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":68876746},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":47,"seq":201863462959,"num_pgs":48,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2091856,"kb_used_data":2024572,"kb_used_omap":1,"kb_used_meta":67262,"kb_avail":91653296,"statfs":{"total":95995035648,"available":93852975104,"internally_reserved":0,"allocated":2073161728,"data_stored":2072780409,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":68876742},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1],"upper_bound":2},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":39,"seq":167503724597,"num_pgs":52,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1674776,"kb_used_data":1607492,"kb_used_omap":1,"kb_used_meta":67262,"kb_avail":92070376,"statfs":{"total":95995035648,"available":94280065024,"internally_reserved":0,"allocated":1646071808,"data_stored":1645682647,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":68876742},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2],"upper_bound":2},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":32,"seq":137438953530,"num_pgs":63,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2101928,"kb_used_data":2034644,"kb_used_omap":1,"kb_used_meta":67262,"kb_avail":91643224,"statfs":{"total":95995035648,"available":93842661376,"internally_reserved":0,"allocated":2083475456,"data_stored":2083070591,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":68876741},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":26,"seq":111669149761,"num_pgs":65,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":3738792,"kb_used_data":3635352,"kb_used_omap":1,"kb_used_meta":103358,"kb_avail":90006360,"statfs":{"total":95995035648,"available":92166512640,"internally_reserved":0,"allocated":3722600448,"data_stored":3722190780,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":105839048},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411399,"num_pgs":44,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2504656,"kb_used_data":2423424,"kb_used_omap":1,"kb_used_meta":81150,"kb_avail":91240496,"statfs":{"total":95995035648,"available":93430267904,"internally_reserved":0,"allocated":2481586176,"data_stored":2481201107,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":83098053},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574923,"num_pgs":60,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2514308,"kb_used_data":2420596,"kb_used_omap":1,"kb_used_meta":93630,"kb_avail":91230844,"statfs":{"total":95995035648,"available":93420384256,"internally_reserved":0,"allocated":2478690304,"data_stored":2478253884,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":95877573},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738450,"num_pgs":54,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":3349804,"kb_used_data":3246364,"kb_used_omap":1,"kb_used_meta":103358,"kb_avail":90395348,"statfs":{"total":95995035648,"available":92564836352,"internally_reserved":0,"allocated":3324276736,"data_stored":3323864449,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":105839045},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1131,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1521,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":436,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":46,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1177,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":436,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":1898,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1128,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":1898,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1474,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":1100,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1034,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1254,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":403,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":416,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":403,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":3291549696,"data_stored":3291545624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":2459635712,"data_stored":2459631640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":2474115072,"data_stored":2474115072,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":3709796352,"data_stored":3709796352,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":2067595264,"data_stored":2067595264,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":1638334464,"data_stored":1638334464,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":2063470592,"data_stored":2063466520,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":2068971520,"data_stored":2068971520,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T08:05:56.328 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid 779349ca-7593-11ef-bceb-c7b262605968 -- ceph pg dump --format=json 2024-09-18T08:05:56.375 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6 2024-09-18T08:05:56.401 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:05:56.401 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:05:56.402 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6' 2024-09-18T08:05:56.503 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5 2024-09-18T08:05:56.531 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:05:56.532 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:05:56.533 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5' 2024-09-18T08:05:56.633 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5 2024-09-18T08:05:56.655 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:05:56.656 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:05:56.656 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5' 2024-09-18T08:05:56.698 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:05:56 smithi007 ceph-mon[29459]: pgmap v135: 148 pgs: 148 active+clean; 6.1 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.25k op/s 2024-09-18T08:05:56.698 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:05:56 smithi007 ceph-mon[29459]: from='client.14973 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-18T08:05:56.699 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:05:56 smithi007 ceph-mon[29459]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-18T08:05:56.699 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:05:56 smithi007 ceph-mon[29459]: from='client.? 172.21.15.7:0/2337631247' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-18T08:05:56.699 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:05:56 smithi007 ceph-mon[29459]: osdmap e127: 8 total, 8 up, 8 in 2024-09-18T08:05:56.699 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:05:56 smithi007 ceph-mon[39249]: pgmap v135: 148 pgs: 148 active+clean; 6.1 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.25k op/s 2024-09-18T08:05:56.699 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:05:56 smithi007 ceph-mon[39249]: from='client.14973 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-18T08:05:56.699 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:05:56 smithi007 ceph-mon[39249]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-18T08:05:56.699 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:05:56 smithi007 ceph-mon[39249]: from='client.? 172.21.15.7:0/2337631247' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-18T08:05:56.699 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:05:56 smithi007 ceph-mon[39249]: osdmap e127: 8 total, 8 up, 8 in 2024-09-18T08:05:56.757 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5 2024-09-18T08:05:56.786 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:05:56.787 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:05:56.787 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5' 2024-09-18T08:05:56.888 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5 2024-09-18T08:05:56.906 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:05:56 smithi050 ceph-mon[32371]: pgmap v135: 148 pgs: 148 active+clean; 6.1 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.25k op/s 2024-09-18T08:05:56.906 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:05:56 smithi050 ceph-mon[32371]: from='client.14973 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-18T08:05:56.906 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:05:56 smithi050 ceph-mon[32371]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-18T08:05:56.906 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:05:56 smithi050 ceph-mon[32371]: from='client.? 172.21.15.7:0/2337631247' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-18T08:05:56.906 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:05:56 smithi050 ceph-mon[32371]: osdmap e127: 8 total, 8 up, 8 in 2024-09-18T08:05:56.913 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:05:56.914 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:05:56.915 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5' 2024-09-18T08:05:56.980 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Sep 18 08:05:56 smithi007 ceph-779349ca-7593-11ef-bceb-c7b262605968-mgr-y[29688]: ::ffff:172.21.15.50 - - [18/Sep/2024:08:05:56] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-09-18T08:05:57.016 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5 2024-09-18T08:05:57.044 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:05:57.044 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:05:57.045 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5' 2024-09-18T08:05:57.146 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5 2024-09-18T08:05:57.173 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:05:57.174 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:05:57.175 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5' 2024-09-18T08:05:57.275 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4 2024-09-18T08:05:57.303 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:05:57.304 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:05:57.304 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4' 2024-09-18T08:05:57.405 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5 2024-09-18T08:05:57.433 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:05:57.434 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:05:57.435 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5' 2024-09-18T08:05:57.535 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7 2024-09-18T08:05:57.561 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:05:57.562 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:05:57.562 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7' 2024-09-18T08:05:57.663 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5 2024-09-18T08:05:57.690 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:05:57.690 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:05:57.691 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5' 2024-09-18T08:05:57.791 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3 2024-09-18T08:05:57.832 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:05:57.832 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:05:57.833 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3' 2024-09-18T08:05:57.933 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2 2024-09-18T08:05:57.979 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:05:57.980 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:05:57.980 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2' 2024-09-18T08:05:58.081 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3 2024-09-18T08:05:58.119 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:05:58.119 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:05:58.120 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3' 2024-09-18T08:05:58.220 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3 2024-09-18T08:05:58.265 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:05:58.266 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:05:58.266 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3' 2024-09-18T08:05:58.367 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5 2024-09-18T08:05:58.392 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:05:58.393 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:05:58.393 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5' 2024-09-18T08:05:58.411 INFO:teuthology.orchestra.run.smithi007.stdout:ERROR: (22) Invalid argument 2024-09-18T08:05:58.411 INFO:teuthology.orchestra.run.smithi007.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-18T08:05:58.495 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0 2024-09-18T08:05:58.540 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:05:58.541 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:05:58.542 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0' 2024-09-18T08:05:58.595 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:05:58 smithi007 ceph-mon[29459]: pgmap v137: 148 pgs: 148 active+clean; 6.1 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 656 op/s 2024-09-18T08:05:58.595 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:05:58 smithi007 ceph-mon[39249]: pgmap v137: 148 pgs: 148 active+clean; 6.1 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 656 op/s 2024-09-18T08:05:58.643 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4 2024-09-18T08:05:58.668 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:05:58.669 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:05:58.669 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4' 2024-09-18T08:05:58.671 INFO:teuthology.orchestra.run.smithi007.stdout:osd.0: {} 2024-09-18T08:05:58.671 INFO:teuthology.orchestra.run.smithi007.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-18T08:05:58.709 INFO:teuthology.orchestra.run.smithi007.stdout:osd.1: {} 2024-09-18T08:05:58.710 INFO:teuthology.orchestra.run.smithi007.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-18T08:05:58.752 INFO:teuthology.orchestra.run.smithi007.stdout:osd.2: {} 2024-09-18T08:05:58.752 INFO:teuthology.orchestra.run.smithi007.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-18T08:05:58.770 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7 2024-09-18T08:05:58.793 INFO:teuthology.orchestra.run.smithi007.stdout:osd.3: {} 2024-09-18T08:05:58.794 INFO:teuthology.orchestra.run.smithi007.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-18T08:05:58.795 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:05:58.796 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:05:58.796 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7' 2024-09-18T08:05:58.829 INFO:teuthology.orchestra.run.smithi007.stdout:osd.4: {} 2024-09-18T08:05:58.829 INFO:teuthology.orchestra.run.smithi007.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-18T08:05:58.840 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:05:58 smithi050 ceph-mon[32371]: pgmap v137: 148 pgs: 148 active+clean; 6.1 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 656 op/s 2024-09-18T08:05:58.864 INFO:teuthology.orchestra.run.smithi007.stdout:osd.5: {} 2024-09-18T08:05:58.864 INFO:teuthology.orchestra.run.smithi007.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-18T08:05:58.897 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4 2024-09-18T08:05:58.901 INFO:teuthology.orchestra.run.smithi007.stdout:osd.6: {} 2024-09-18T08:05:58.902 INFO:teuthology.orchestra.run.smithi007.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-18T08:05:58.924 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:05:58.925 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:05:58.925 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4' 2024-09-18T08:05:58.937 INFO:teuthology.orchestra.run.smithi007.stdout:osd.7: {} 2024-09-18T08:05:58.937 INFO:teuthology.orchestra.run.smithi007.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-18T08:05:59.026 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2 2024-09-18T08:05:59.071 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:05:59.072 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:05:59.072 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2' 2024-09-18T08:05:59.156 INFO:journalctl@ceph.iscsi.iscsi.a.smithi050.stdout:Sep 18 08:05:58 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-iscsi-iscsi-a[57782]: debug there is no tcmu-runner data available 2024-09-18T08:05:59.172 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7 2024-09-18T08:05:59.198 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:05:59.199 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:05:59.199 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7' 2024-09-18T08:05:59.300 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6 2024-09-18T08:05:59.326 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:05:59.327 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:05:59.327 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6' 2024-09-18T08:05:59.431 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0 2024-09-18T08:05:59.455 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:05:59.456 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:05:59.456 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0' 2024-09-18T08:05:59.557 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7 2024-09-18T08:05:59.583 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:05:59.584 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:05:59.584 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7' 2024-09-18T08:05:59.685 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1 2024-09-18T08:05:59.730 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:05:59 smithi007 ceph-mon[29459]: from='client.? 172.21.15.7:0/1687080619' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-18T08:05:59.731 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:05:59 smithi007 ceph-mon[39249]: from='client.? 172.21.15.7:0/1687080619' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-18T08:05:59.737 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:05:59.738 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:05:59.738 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1' 2024-09-18T08:05:59.839 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4 2024-09-18T08:05:59.866 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:05:59.867 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:05:59.867 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4' 2024-09-18T08:05:59.905 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:05:59 smithi050 ceph-mon[32371]: from='client.? 172.21.15.7:0/1687080619' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-18T08:05:59.968 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2 2024-09-18T08:06:00.006 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:06:00.007 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:06:00.007 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2' 2024-09-18T08:06:00.108 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0 2024-09-18T08:06:00.145 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:06:00.146 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:06:00.146 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0' 2024-09-18T08:06:00.247 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6 2024-09-18T08:06:00.275 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:06:00.276 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:06:00.276 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6' 2024-09-18T08:06:00.377 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4 2024-09-18T08:06:00.406 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:06:00.406 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:06:00.406 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4' 2024-09-18T08:06:00.507 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6 2024-09-18T08:06:00.536 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:06:00.536 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:06:00.536 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6' 2024-09-18T08:06:00.637 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5 2024-09-18T08:06:00.661 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:06:00.662 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:06:00.662 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5' 2024-09-18T08:06:00.670 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid 779349ca-7593-11ef-bceb-c7b262605968 -- ceph daemon osd.2 dump_ops_in_flight 2024-09-18T08:06:00.709 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:06:00 smithi007 ceph-mon[29459]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-18T08:06:00.709 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:06:00 smithi007 ceph-mon[29459]: pgmap v138: 148 pgs: 148 active+clean; 5.8 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.29k op/s 2024-09-18T08:06:00.709 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:06:00 smithi007 ceph-mon[39249]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-18T08:06:00.709 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:06:00 smithi007 ceph-mon[39249]: pgmap v138: 148 pgs: 148 active+clean; 5.8 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.29k op/s 2024-09-18T08:06:00.763 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2 2024-09-18T08:06:00.803 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:06:00.804 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:06:00.804 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2' 2024-09-18T08:06:00.905 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5 2024-09-18T08:06:00.906 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:06:00 smithi050 ceph-mon[32371]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-18T08:06:00.906 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:06:00 smithi050 ceph-mon[32371]: pgmap v138: 148 pgs: 148 active+clean; 5.8 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.29k op/s 2024-09-18T08:06:00.932 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:06:00.933 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:06:00.933 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5' 2024-09-18T08:06:01.034 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6 2024-09-18T08:06:01.060 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:06:01.061 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:06:01.061 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6' 2024-09-18T08:06:01.162 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6 2024-09-18T08:06:01.188 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:06:01.189 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:06:01.189 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6' 2024-09-18T08:06:01.290 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6 2024-09-18T08:06:01.317 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:06:01.317 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:06:01.318 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6' 2024-09-18T08:06:01.419 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2 2024-09-18T08:06:01.473 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:06:01.476 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:06:01.476 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2' 2024-09-18T08:06:01.516 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/779349ca-7593-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T08:06:01.577 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3 2024-09-18T08:06:01.621 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:06:01.622 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:06:01.622 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3' 2024-09-18T08:06:01.723 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4 2024-09-18T08:06:01.749 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:06:01.750 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:06:01.750 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4' 2024-09-18T08:06:01.851 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1 2024-09-18T08:06:01.891 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:06:01.892 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:06:01.892 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1' 2024-09-18T08:06:01.993 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0 2024-09-18T08:06:02.009 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/779349ca-7593-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T08:06:02.049 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:06:02.050 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:06:02.050 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0' 2024-09-18T08:06:02.151 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0 2024-09-18T08:06:02.198 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:06:02.198 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:06:02.199 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0' 2024-09-18T08:06:02.300 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5 2024-09-18T08:06:02.325 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:06:02.325 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:06:02.326 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5' 2024-09-18T08:06:02.427 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6 2024-09-18T08:06:02.453 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:06:02.453 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:06:02.454 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6' 2024-09-18T08:06:02.554 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6 2024-09-18T08:06:02.578 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:06:02 smithi007 ceph-mon[29459]: pgmap v139: 148 pgs: 148 active+clean; 5.8 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 1.0 KiB/s rd, 0 B/s wr, 1.06k op/s 2024-09-18T08:06:02.578 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:06:02 smithi007 ceph-mon[39249]: pgmap v139: 148 pgs: 148 active+clean; 5.8 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 1.0 KiB/s rd, 0 B/s wr, 1.06k op/s 2024-09-18T08:06:02.579 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:06:02.579 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:06:02.579 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6' 2024-09-18T08:06:02.680 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2 2024-09-18T08:06:02.725 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:06:02.726 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:06:02.727 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2' 2024-09-18T08:06:02.827 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5 2024-09-18T08:06:02.860 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:06:02.861 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:06:02.861 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5' 2024-09-18T08:06:02.905 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:06:02 smithi050 ceph-mon[32371]: pgmap v139: 148 pgs: 148 active+clean; 5.8 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 1.0 KiB/s rd, 0 B/s wr, 1.06k op/s 2024-09-18T08:06:02.962 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7 2024-09-18T08:06:02.987 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:06:02.988 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:06:02.988 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7' 2024-09-18T08:06:03.089 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1 2024-09-18T08:06:03.134 INFO:teuthology.orchestra.run.smithi007.stdout:{ 2024-09-18T08:06:03.134 INFO:teuthology.orchestra.run.smithi007.stdout: "ops": [ 2024-09-18T08:06:03.134 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:06:03.135 INFO:teuthology.orchestra.run.smithi007.stdout: "description": "osd_op(client.14805.0:146173 7.e 7:70d2a36e:::benchmark_data_smithi007_73155_object27882:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e127)", 2024-09-18T08:06:03.135 INFO:teuthology.orchestra.run.smithi007.stdout: "initiated_at": "2024-09-18T08:06:03.111527+0000", 2024-09-18T08:06:03.135 INFO:teuthology.orchestra.run.smithi007.stdout: "age": 0.020085072999999998, 2024-09-18T08:06:03.135 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.020110494, 2024-09-18T08:06:03.135 INFO:teuthology.orchestra.run.smithi007.stdout: "continuous": false, 2024-09-18T08:06:03.135 INFO:teuthology.orchestra.run.smithi007.stdout: "type_data": { 2024-09-18T08:06:03.135 INFO:teuthology.orchestra.run.smithi007.stdout: "flag_point": "waiting for sub ops", 2024-09-18T08:06:03.135 INFO:teuthology.orchestra.run.smithi007.stdout: "client_info": { 2024-09-18T08:06:03.135 INFO:teuthology.orchestra.run.smithi007.stdout: "client": "client.14805", 2024-09-18T08:06:03.135 INFO:teuthology.orchestra.run.smithi007.stdout: "client_addr": "172.21.15.7:0/640320436", 2024-09-18T08:06:03.135 INFO:teuthology.orchestra.run.smithi007.stdout: "tid": 146173 2024-09-18T08:06:03.135 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:06:03.135 INFO:teuthology.orchestra.run.smithi007.stdout: "events": [ 2024-09-18T08:06:03.135 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:06:03.136 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "initiated", 2024-09-18T08:06:03.136 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:06:03.111527+0000", 2024-09-18T08:06:03.136 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-09-18T08:06:03.136 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:06:03.136 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:06:03.136 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "header_read", 2024-09-18T08:06:03.136 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:06:03.111527+0000", 2024-09-18T08:06:03.136 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-09-18T08:06:03.136 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:06:03.136 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:06:03.136 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "throttled", 2024-09-18T08:06:03.136 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:06:03.111529+0000", 2024-09-18T08:06:03.136 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.556e-06 2024-09-18T08:06:03.137 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:06:03.137 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:06:03.137 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "all_read", 2024-09-18T08:06:03.137 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:06:03.111534+0000", 2024-09-18T08:06:03.137 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 5.3469999999999998e-06 2024-09-18T08:06:03.137 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:06:03.137 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:06:03.137 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "dispatched", 2024-09-18T08:06:03.137 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:06:03.111575+0000", 2024-09-18T08:06:03.137 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 4.0751000000000003e-05 2024-09-18T08:06:03.137 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:06:03.137 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:06:03.137 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "queued_for_pg", 2024-09-18T08:06:03.137 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:06:03.111583+0000", 2024-09-18T08:06:03.138 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 8.2660000000000001e-06 2024-09-18T08:06:03.138 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:06:03.138 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:06:03.138 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-09-18T08:06:03.138 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:06:03.118094+0000", 2024-09-18T08:06:03.138 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.0065103239999999996 2024-09-18T08:06:03.138 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:06:03.138 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:06:03.138 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "started", 2024-09-18T08:06:03.138 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:06:03.118992+0000", 2024-09-18T08:06:03.138 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.00089879200000000004 2024-09-18T08:06:03.138 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:06:03.138 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:06:03.138 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "waiting for subops from 0,5", 2024-09-18T08:06:03.139 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:06:03.119097+0000", 2024-09-18T08:06:03.139 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.000104894 2024-09-18T08:06:03.139 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:06:03.139 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:06:03.139 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "op_commit", 2024-09-18T08:06:03.139 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:06:03.122967+0000", 2024-09-18T08:06:03.139 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.0038701019999999998 2024-09-18T08:06:03.139 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:06:03.139 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:06:03.139 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:06:03.139 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:06:03.127825+0000", 2024-09-18T08:06:03.139 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.0048578680000000004 2024-09-18T08:06:03.139 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-09-18T08:06:03.140 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-09-18T08:06:03.140 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-09-18T08:06:03.140 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:06:03.140 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:06:03.140 INFO:teuthology.orchestra.run.smithi007.stdout: "description": "osd_op(client.14805.0:146188 7.8 7:14ae9e6c:::benchmark_data_smithi007_73155_object27897:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e127)", 2024-09-18T08:06:03.140 INFO:teuthology.orchestra.run.smithi007.stdout: "initiated_at": "2024-09-18T08:06:03.127401+0000", 2024-09-18T08:06:03.140 INFO:teuthology.orchestra.run.smithi007.stdout: "age": 0.0042116690000000003, 2024-09-18T08:06:03.140 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.0043255769999999997, 2024-09-18T08:06:03.140 INFO:teuthology.orchestra.run.smithi007.stdout: "continuous": false, 2024-09-18T08:06:03.140 INFO:teuthology.orchestra.run.smithi007.stdout: "type_data": { 2024-09-18T08:06:03.140 INFO:teuthology.orchestra.run.smithi007.stdout: "flag_point": "waiting for sub ops", 2024-09-18T08:06:03.140 INFO:teuthology.orchestra.run.smithi007.stdout: "client_info": { 2024-09-18T08:06:03.140 INFO:teuthology.orchestra.run.smithi007.stdout: "client": "client.14805", 2024-09-18T08:06:03.141 INFO:teuthology.orchestra.run.smithi007.stdout: "client_addr": "172.21.15.7:0/640320436", 2024-09-18T08:06:03.141 INFO:teuthology.orchestra.run.smithi007.stdout: "tid": 146188 2024-09-18T08:06:03.141 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:06:03.141 INFO:teuthology.orchestra.run.smithi007.stdout: "events": [ 2024-09-18T08:06:03.141 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:06:03.141 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "initiated", 2024-09-18T08:06:03.141 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:06:03.127401+0000", 2024-09-18T08:06:03.141 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-09-18T08:06:03.141 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:06:03.141 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:06:03.141 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "header_read", 2024-09-18T08:06:03.141 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:06:03.127401+0000", 2024-09-18T08:06:03.141 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-09-18T08:06:03.141 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:06:03.142 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:06:03.142 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "throttled", 2024-09-18T08:06:03.142 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:06:03.127402+0000", 2024-09-18T08:06:03.142 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.5e-06 2024-09-18T08:06:03.142 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:06:03.142 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:06:03.142 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "all_read", 2024-09-18T08:06:03.142 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:06:03.127409+0000", 2024-09-18T08:06:03.142 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 7.0079999999999996e-06 2024-09-18T08:06:03.142 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:06:03.142 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:06:03.142 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "dispatched", 2024-09-18T08:06:03.142 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:06:03.127427+0000", 2024-09-18T08:06:03.143 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.7904e-05 2024-09-18T08:06:03.143 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:06:03.143 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:06:03.143 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "queued_for_pg", 2024-09-18T08:06:03.143 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:06:03.127437+0000", 2024-09-18T08:06:03.143 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 9.5070000000000006e-06 2024-09-18T08:06:03.143 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:06:03.143 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:06:03.143 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-09-18T08:06:03.143 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:06:03.127560+0000", 2024-09-18T08:06:03.144 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.000123657 2024-09-18T08:06:03.144 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:06:03.144 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:06:03.144 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "started", 2024-09-18T08:06:03.144 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:06:03.127677+0000", 2024-09-18T08:06:03.144 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.000117043 2024-09-18T08:06:03.144 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:06:03.144 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:06:03.144 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "waiting for subops from 4,7", 2024-09-18T08:06:03.144 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:06:03.127798+0000", 2024-09-18T08:06:03.144 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.000121135 2024-09-18T08:06:03.144 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-09-18T08:06:03.144 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-09-18T08:06:03.144 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-09-18T08:06:03.145 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:06:03.145 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:06:03.145 INFO:teuthology.orchestra.run.smithi007.stdout: "description": "osd_op(client.14805.0:146191 7.8 7:1ffbe5ff:::benchmark_data_smithi007_73155_object27900:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e127)", 2024-09-18T08:06:03.145 INFO:teuthology.orchestra.run.smithi007.stdout: "initiated_at": "2024-09-18T08:06:03.127462+0000", 2024-09-18T08:06:03.145 INFO:teuthology.orchestra.run.smithi007.stdout: "age": 0.0041505930000000002, 2024-09-18T08:06:03.145 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.0043265810000000004, 2024-09-18T08:06:03.145 INFO:teuthology.orchestra.run.smithi007.stdout: "continuous": false, 2024-09-18T08:06:03.145 INFO:teuthology.orchestra.run.smithi007.stdout: "type_data": { 2024-09-18T08:06:03.145 INFO:teuthology.orchestra.run.smithi007.stdout: "flag_point": "waiting for sub ops", 2024-09-18T08:06:03.145 INFO:teuthology.orchestra.run.smithi007.stdout: "client_info": { 2024-09-18T08:06:03.145 INFO:teuthology.orchestra.run.smithi007.stdout: "client": "client.14805", 2024-09-18T08:06:03.145 INFO:teuthology.orchestra.run.smithi007.stdout: "client_addr": "172.21.15.7:0/640320436", 2024-09-18T08:06:03.145 INFO:teuthology.orchestra.run.smithi007.stdout: "tid": 146191 2024-09-18T08:06:03.145 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:06:03.146 INFO:teuthology.orchestra.run.smithi007.stdout: "events": [ 2024-09-18T08:06:03.146 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:06:03.146 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "initiated", 2024-09-18T08:06:03.146 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:06:03.127462+0000", 2024-09-18T08:06:03.146 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-09-18T08:06:03.146 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:06:03.146 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:06:03.146 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "header_read", 2024-09-18T08:06:03.146 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:06:03.127462+0000", 2024-09-18T08:06:03.146 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-09-18T08:06:03.146 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:06:03.146 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:06:03.146 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "throttled", 2024-09-18T08:06:03.146 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:06:03.127462+0000", 2024-09-18T08:06:03.147 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 5.5400000000000001e-07 2024-09-18T08:06:03.147 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:06:03.147 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:06:03.147 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "all_read", 2024-09-18T08:06:03.147 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:06:03.127465+0000", 2024-09-18T08:06:03.147 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 3.016e-06 2024-09-18T08:06:03.147 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:06:03.147 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:06:03.147 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "dispatched", 2024-09-18T08:06:03.147 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:06:03.127473+0000", 2024-09-18T08:06:03.147 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 7.3989999999999998e-06 2024-09-18T08:06:03.147 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:06:03.147 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:06:03.147 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "queued_for_pg", 2024-09-18T08:06:03.148 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:06:03.127479+0000", 2024-09-18T08:06:03.148 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 5.8629999999999996e-06 2024-09-18T08:06:03.148 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:06:03.148 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:06:03.148 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-09-18T08:06:03.148 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:06:03.131251+0000", 2024-09-18T08:06:03.148 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.0037722950000000002 2024-09-18T08:06:03.148 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:06:03.148 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:06:03.148 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "started", 2024-09-18T08:06:03.148 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:06:03.131367+0000", 2024-09-18T08:06:03.148 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.000116091 2024-09-18T08:06:03.148 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:06:03.148 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:06:03.149 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "waiting for subops from 4,7", 2024-09-18T08:06:03.149 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:06:03.131482+0000", 2024-09-18T08:06:03.149 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.00011455000000000001 2024-09-18T08:06:03.149 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-09-18T08:06:03.149 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-09-18T08:06:03.149 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-09-18T08:06:03.149 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:06:03.150 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:06:03.150 INFO:teuthology.orchestra.run.smithi007.stdout: "description": "osd_repop(client.14805.0:146179 7.3 e127/83 7:cd16b574:::benchmark_data_smithi007_73155_object27888:head v 127'9099, mlcod=127'9096)", 2024-09-18T08:06:03.150 INFO:teuthology.orchestra.run.smithi007.stdout: "initiated_at": "2024-09-18T08:06:03.131096+0000", 2024-09-18T08:06:03.150 INFO:teuthology.orchestra.run.smithi007.stdout: "age": 0.00051587900000000001, 2024-09-18T08:06:03.150 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.00076048800000000003, 2024-09-18T08:06:03.150 INFO:teuthology.orchestra.run.smithi007.stdout: "continuous": false, 2024-09-18T08:06:03.150 INFO:teuthology.orchestra.run.smithi007.stdout: "type_data": { 2024-09-18T08:06:03.150 INFO:teuthology.orchestra.run.smithi007.stdout: "flag_point": "started", 2024-09-18T08:06:03.150 INFO:teuthology.orchestra.run.smithi007.stdout: "events": [ 2024-09-18T08:06:03.150 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:06:03.150 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "initiated", 2024-09-18T08:06:03.150 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:06:03.131096+0000", 2024-09-18T08:06:03.150 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-09-18T08:06:03.150 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:06:03.151 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:06:03.151 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "header_read", 2024-09-18T08:06:03.151 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:06:03.131096+0000", 2024-09-18T08:06:03.151 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-09-18T08:06:03.151 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:06:03.151 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:06:03.151 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "throttled", 2024-09-18T08:06:03.151 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:06:03.131098+0000", 2024-09-18T08:06:03.151 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.415e-06 2024-09-18T08:06:03.151 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:06:03.151 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:06:03.151 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "all_read", 2024-09-18T08:06:03.151 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:06:03.131106+0000", 2024-09-18T08:06:03.151 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 7.8660000000000006e-06 2024-09-18T08:06:03.152 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:06:03.152 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:06:03.152 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "dispatched", 2024-09-18T08:06:03.152 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:06:03.131122+0000", 2024-09-18T08:06:03.152 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.6693000000000001e-05 2024-09-18T08:06:03.152 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:06:03.152 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:06:03.152 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "queued_for_pg", 2024-09-18T08:06:03.152 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:06:03.131131+0000", 2024-09-18T08:06:03.152 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 8.2109999999999998e-06 2024-09-18T08:06:03.152 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:06:03.152 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:06:03.152 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-09-18T08:06:03.152 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:06:03.131523+0000", 2024-09-18T08:06:03.153 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.00039246600000000001 2024-09-18T08:06:03.153 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:06:03.153 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:06:03.153 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "started", 2024-09-18T08:06:03.153 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:06:03.131542+0000", 2024-09-18T08:06:03.153 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.8661999999999999e-05 2024-09-18T08:06:03.153 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-09-18T08:06:03.153 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-09-18T08:06:03.153 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-09-18T08:06:03.153 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:06:03.153 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:06:03.154 INFO:teuthology.orchestra.run.smithi007.stdout: "description": "osd_repop(client.14805.0:146185 7.a e127/83)", 2024-09-18T08:06:03.154 INFO:teuthology.orchestra.run.smithi007.stdout: "initiated_at": "2024-09-18T08:06:03.131157+0000", 2024-09-18T08:06:03.154 INFO:teuthology.orchestra.run.smithi007.stdout: "age": 0.00045555500000000001, 2024-09-18T08:06:03.154 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.00074903399999999998, 2024-09-18T08:06:03.154 INFO:teuthology.orchestra.run.smithi007.stdout: "continuous": false, 2024-09-18T08:06:03.154 INFO:teuthology.orchestra.run.smithi007.stdout: "type_data": { 2024-09-18T08:06:03.154 INFO:teuthology.orchestra.run.smithi007.stdout: "flag_point": "queued for pg", 2024-09-18T08:06:03.154 INFO:teuthology.orchestra.run.smithi007.stdout: "events": [ 2024-09-18T08:06:03.154 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:06:03.154 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "initiated", 2024-09-18T08:06:03.154 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:06:03.131157+0000", 2024-09-18T08:06:03.154 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-09-18T08:06:03.154 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:06:03.154 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:06:03.155 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "header_read", 2024-09-18T08:06:03.155 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:06:03.131157+0000", 2024-09-18T08:06:03.155 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-09-18T08:06:03.155 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:06:03.155 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:06:03.155 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "throttled", 2024-09-18T08:06:03.155 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:06:03.131157+0000", 2024-09-18T08:06:03.155 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 3.4299999999999999e-07 2024-09-18T08:06:03.155 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:06:03.155 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:06:03.155 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "all_read", 2024-09-18T08:06:03.155 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:06:03.131161+0000", 2024-09-18T08:06:03.155 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 3.7479999999999999e-06 2024-09-18T08:06:03.155 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:06:03.156 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:06:03.156 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "dispatched", 2024-09-18T08:06:03.156 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:06:03.131167+0000", 2024-09-18T08:06:03.156 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 5.8549999999999998e-06 2024-09-18T08:06:03.156 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:06:03.156 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:06:03.156 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "queued_for_pg", 2024-09-18T08:06:03.156 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:06:03.131172+0000", 2024-09-18T08:06:03.156 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 4.9540000000000003e-06 2024-09-18T08:06:03.156 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-09-18T08:06:03.156 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-09-18T08:06:03.156 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-09-18T08:06:03.156 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-09-18T08:06:03.156 INFO:teuthology.orchestra.run.smithi007.stdout: ], 2024-09-18T08:06:03.157 INFO:teuthology.orchestra.run.smithi007.stdout: "num_ops": 5 2024-09-18T08:06:03.157 INFO:teuthology.orchestra.run.smithi007.stdout:} 2024-09-18T08:06:03.157 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:06:03.157 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:06:03.158 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1' 2024-09-18T08:06:03.228 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-09-18T08:06:03.230 INFO:teuthology.orchestra.run.smithi007.stderr:dumped all 2024-09-18T08:06:03.259 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7 2024-09-18T08:06:03.285 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:06:03.285 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:06:03.286 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7' 2024-09-18T08:06:03.387 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7 2024-09-18T08:06:03.414 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:06:03.415 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:06:03.415 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7' 2024-09-18T08:06:03.423 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid 779349ca-7593-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-18T08:06:03.516 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7 2024-09-18T08:06:03.539 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:06:03.540 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:06:03.540 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7' 2024-09-18T08:06:03.641 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6 2024-09-18T08:06:03.670 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:06:03.670 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:06:03.671 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6' 2024-09-18T08:06:03.730 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:06:03 smithi007 ceph-mon[29459]: from='mgr.24470 172.21.15.7:0/2594390261' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-18T08:06:03.731 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:06:03 smithi007 ceph-mon[39249]: from='mgr.24470 172.21.15.7:0/2594390261' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-18T08:06:03.771 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2 2024-09-18T08:06:03.808 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:06:03.809 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:06:03.810 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2' 2024-09-18T08:06:03.906 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:06:03 smithi050 ceph-mon[32371]: from='mgr.24470 172.21.15.7:0/2594390261' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-18T08:06:03.910 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7 2024-09-18T08:06:03.939 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:06:03.940 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:06:03.940 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7' 2024-09-18T08:06:03.976 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid 779349ca-7593-11ef-bceb-c7b262605968 -- ceph osd unset noscrub 2024-09-18T08:06:04.041 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7 2024-09-18T08:06:04.066 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:06:04.067 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:06:04.067 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7' 2024-09-18T08:06:04.168 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2 2024-09-18T08:06:04.204 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:06:04.205 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:06:04.205 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2' 2024-09-18T08:06:04.306 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1 2024-09-18T08:06:04.357 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:06:04.357 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:06:04.358 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1' 2024-09-18T08:06:04.459 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6 2024-09-18T08:06:04.489 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:06:04.490 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:06:04.490 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6' 2024-09-18T08:06:04.591 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3 2024-09-18T08:06:04.626 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:06:04.627 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:06:04.627 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3' 2024-09-18T08:06:04.728 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2 2024-09-18T08:06:04.731 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:06:04 smithi007 ceph-mon[29459]: pgmap v140: 148 pgs: 148 active+clean; 5.8 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.03k op/s 2024-09-18T08:06:04.731 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:06:04 smithi007 ceph-mon[29459]: from='client.24814 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-18T08:06:04.731 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:06:04 smithi007 ceph-mon[39249]: pgmap v140: 148 pgs: 148 active+clean; 5.8 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.03k op/s 2024-09-18T08:06:04.732 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:06:04 smithi007 ceph-mon[39249]: from='client.24814 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-18T08:06:04.750 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:06:04.751 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:06:04.751 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2' 2024-09-18T08:06:04.852 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5 2024-09-18T08:06:04.879 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:06:04.880 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:06:04.880 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5' 2024-09-18T08:06:04.906 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:06:04 smithi050 ceph-mon[32371]: pgmap v140: 148 pgs: 148 active+clean; 5.8 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.03k op/s 2024-09-18T08:06:04.906 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:06:04 smithi050 ceph-mon[32371]: from='client.24814 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-18T08:06:04.981 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5 2024-09-18T08:06:05.008 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:06:05.009 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:06:05.009 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5' 2024-09-18T08:06:05.110 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7 2024-09-18T08:06:05.137 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:06:05.138 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:06:05.138 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7' 2024-09-18T08:06:05.239 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0 2024-09-18T08:06:05.277 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:06:05.277 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:06:05.277 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0' 2024-09-18T08:06:05.378 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6 2024-09-18T08:06:05.407 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:06:05.408 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:06:05.408 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6' 2024-09-18T08:06:05.509 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1 2024-09-18T08:06:05.548 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:06:05.549 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:06:05.549 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1' 2024-09-18T08:06:05.650 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7 2024-09-18T08:06:05.677 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:06:05.677 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:06:05.678 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7' 2024-09-18T08:06:05.778 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0 2024-09-18T08:06:05.823 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:06:05.823 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:06:05.824 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0' 2024-09-18T08:06:05.924 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6 2024-09-18T08:06:05.953 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:06:05.954 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:06:05.954 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6' 2024-09-18T08:06:06.055 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1 2024-09-18T08:06:06.092 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:06:06.093 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:06:06.094 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1' 2024-09-18T08:06:06.194 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4 2024-09-18T08:06:06.221 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:06:06.222 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:06:06.222 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4' 2024-09-18T08:06:06.323 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3 2024-09-18T08:06:06.358 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:06:06.359 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:06:06.360 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3' 2024-09-18T08:06:06.448 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid 779349ca-7593-11ef-bceb-c7b262605968 -- ceph daemon osd.2 dump_blocked_ops 2024-09-18T08:06:06.460 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3 2024-09-18T08:06:06.552 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:06:06.553 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:06:06.553 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3' 2024-09-18T08:06:06.654 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6 2024-09-18T08:06:06.680 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:06:06.681 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:06:06.681 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6' 2024-09-18T08:06:06.698 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:06:06 smithi007 ceph-mon[29459]: pgmap v141: 148 pgs: 148 active+clean; 5.5 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.03k op/s 2024-09-18T08:06:06.698 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:06:06 smithi007 ceph-mon[39249]: pgmap v141: 148 pgs: 148 active+clean; 5.5 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.03k op/s 2024-09-18T08:06:06.782 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3 2024-09-18T08:06:06.831 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:06:06.831 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:06:06.832 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3' 2024-09-18T08:06:06.906 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:06:06 smithi050 ceph-mon[32371]: pgmap v141: 148 pgs: 148 active+clean; 5.5 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.03k op/s 2024-09-18T08:06:06.932 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3 2024-09-18T08:06:06.980 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Sep 18 08:06:06 smithi007 ceph-779349ca-7593-11ef-bceb-c7b262605968-mgr-y[29688]: ::ffff:172.21.15.50 - - [18/Sep/2024:08:06:06] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-09-18T08:06:06.983 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:06:06.984 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:06:06.984 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3' 2024-09-18T08:06:07.086 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0 2024-09-18T08:06:07.138 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:06:07.139 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:06:07.139 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0' 2024-09-18T08:06:07.240 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7 2024-09-18T08:06:07.265 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:06:07.266 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:06:07.266 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7' 2024-09-18T08:06:07.305 INFO:teuthology.orchestra.run.smithi007.stdout:{"pg_ready":true,"pg_map":{"version":140,"stamp":"2024-09-18T08:06:03.128573+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":6192831027,"num_objects":94686,"num_object_clones":0,"num_object_copies":284058,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":94686,"num_whiteouts":0,"num_read":1035,"num_read_kb":892,"num_write":142640,"num_write_kb":7572106,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":590808,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":86945,"ondisk_log_size":86945,"up":444,"acting":444,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":444,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":18676148,"kb_used_data":18025320,"kb_used_omap":12,"kb_used_meta":650547,"kb_avail":731285068,"statfs":{"total":767960285184,"available":748835909632,"internally_reserved":0,"allocated":18457927680,"data_stored":18454668793,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12750,"internal_metadata":666160690},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,1,1],"upper_bound":8},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-677707776,"num_objects":-10341,"num_object_clones":0,"num_object_copies":-31023,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-10341,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":10341,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.005188"},"pg_stats":[{"pgid":"3.1f","version":"0'0","reported_seq":151,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-09-18T08:05:55.472932+0000","last_change":"2024-09-18T08:04:34.226236+0000","last_active":"2024-09-18T08:05:55.472932+0000","last_peered":"2024-09-18T08:05:55.472932+0000","last_clean":"2024-09-18T08:05:55.472932+0000","last_became_active":"2024-09-18T08:02:47.877698+0000","last_became_peered":"2024-09-18T08:02:47.877698+0000","last_unstale":"2024-09-18T08:05:55.472932+0000","last_undegraded":"2024-09-18T08:05:55.472932+0000","last_fullsized":"2024-09-18T08:05:55.472932+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:04:34.226054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:46.860092+0000","last_clean_scrub_stamp":"2024-09-18T08:04:34.226054+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"69'9","reported_seq":163,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-09-18T08:05:55.470140+0000","last_change":"2024-09-18T08:05:08.697007+0000","last_active":"2024-09-18T08:05:55.470140+0000","last_peered":"2024-09-18T08:05:55.470140+0000","last_clean":"2024-09-18T08:05:55.470140+0000","last_became_active":"2024-09-18T08:02:49.894557+0000","last_became_peered":"2024-09-18T08:02:49.894557+0000","last_unstale":"2024-09-18T08:05:55.470140+0000","last_undegraded":"2024-09-18T08:05:55.470140+0000","last_fullsized":"2024-09-18T08:05:55.470140+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'9","last_scrub_stamp":"2024-09-18T08:05:08.696796+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_clean_scrub_stamp":"2024-09-18T08:05:08.696796+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:06:09.954040+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":151,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-09-18T08:05:55.470265+0000","last_change":"2024-09-18T08:05:43.668557+0000","last_active":"2024-09-18T08:05:55.470265+0000","last_peered":"2024-09-18T08:05:55.470265+0000","last_clean":"2024-09-18T08:05:55.470265+0000","last_became_active":"2024-09-18T08:02:51.907140+0000","last_became_peered":"2024-09-18T08:02:51.907140+0000","last_unstale":"2024-09-18T08:05:55.470265+0000","last_undegraded":"2024-09-18T08:05:55.470265+0000","last_fullsized":"2024-09-18T08:05:55.470265+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:05:43.668271+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:50.886830+0000","last_clean_scrub_stamp":"2024-09-18T08:05:43.668271+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:07:02.226478+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":132,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-09-18T08:05:55.470221+0000","last_change":"2024-09-18T08:02:53.906914+0000","last_active":"2024-09-18T08:05:55.470221+0000","last_peered":"2024-09-18T08:05:55.470221+0000","last_clean":"2024-09-18T08:05:55.470221+0000","last_became_active":"2024-09-18T08:02:53.906622+0000","last_became_peered":"2024-09-18T08:02:53.906622+0000","last_unstale":"2024-09-18T08:05:55.470221+0000","last_undegraded":"2024-09-18T08:05:55.470221+0000","last_fullsized":"2024-09-18T08:05:55.470221+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_clean_scrub_stamp":"2024-09-18T08:02:52.890475+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":139,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-09-18T08:05:55.459931+0000","last_change":"2024-09-18T08:05:03.047185+0000","last_active":"2024-09-18T08:05:55.459931+0000","last_peered":"2024-09-18T08:05:55.459931+0000","last_clean":"2024-09-18T08:05:55.459931+0000","last_became_active":"2024-09-18T08:02:54.047640+0000","last_became_peered":"2024-09-18T08:02:54.047640+0000","last_unstale":"2024-09-18T08:05:55.459931+0000","last_undegraded":"2024-09-18T08:05:55.459931+0000","last_fullsized":"2024-09-18T08:05:55.459931+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:05:03.047120+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_clean_scrub_stamp":"2024-09-18T08:05:03.047120+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:06:25.111306+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":151,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-09-18T08:05:55.460680+0000","last_change":"2024-09-18T08:04:57.047972+0000","last_active":"2024-09-18T08:05:55.460680+0000","last_peered":"2024-09-18T08:05:55.460680+0000","last_clean":"2024-09-18T08:05:55.460680+0000","last_became_active":"2024-09-18T08:02:47.872316+0000","last_became_peered":"2024-09-18T08:02:47.872316+0000","last_unstale":"2024-09-18T08:05:55.460680+0000","last_undegraded":"2024-09-18T08:05:55.460680+0000","last_fullsized":"2024-09-18T08:05:55.460680+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:04:57.047904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:46.860092+0000","last_clean_scrub_stamp":"2024-09-18T08:04:57.047904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:06:26.082005+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"69'15","reported_seq":164,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-09-18T08:05:55.460306+0000","last_change":"2024-09-18T08:02:49.887111+0000","last_active":"2024-09-18T08:05:55.460306+0000","last_peered":"2024-09-18T08:05:55.460306+0000","last_clean":"2024-09-18T08:05:55.460306+0000","last_became_active":"2024-09-18T08:02:49.886106+0000","last_became_peered":"2024-09-18T08:02:49.886106+0000","last_unstale":"2024-09-18T08:05:55.460306+0000","last_undegraded":"2024-09-18T08:05:55.460306+0000","last_fullsized":"2024-09-18T08:05:55.460306+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_clean_scrub_stamp":"2024-09-18T08:02:48.870718+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:06:51.053554+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":143,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-09-18T08:05:55.470059+0000","last_change":"2024-09-18T08:04:29.435750+0000","last_active":"2024-09-18T08:05:55.470059+0000","last_peered":"2024-09-18T08:05:55.470059+0000","last_clean":"2024-09-18T08:05:55.470059+0000","last_became_active":"2024-09-18T08:02:51.904184+0000","last_became_peered":"2024-09-18T08:02:51.904184+0000","last_unstale":"2024-09-18T08:05:55.470059+0000","last_undegraded":"2024-09-18T08:05:55.470059+0000","last_fullsized":"2024-09-18T08:05:55.470059+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:04:29.435669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:50.886830+0000","last_clean_scrub_stamp":"2024-09-18T08:04:29.435669+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:05:57.174217+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":151,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-09-18T08:05:55.465978+0000","last_change":"2024-09-18T08:03:53.821133+0000","last_active":"2024-09-18T08:05:55.465978+0000","last_peered":"2024-09-18T08:05:55.465978+0000","last_clean":"2024-09-18T08:05:55.465978+0000","last_became_active":"2024-09-18T08:02:47.875124+0000","last_became_peered":"2024-09-18T08:02:47.875124+0000","last_unstale":"2024-09-18T08:05:55.465978+0000","last_undegraded":"2024-09-18T08:05:55.465978+0000","last_fullsized":"2024-09-18T08:05:55.465978+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:03:53.821070+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:46.860092+0000","last_clean_scrub_stamp":"2024-09-18T08:03:53.821070+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:06:20.018150+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"69'9","reported_seq":155,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-09-18T08:05:55.470407+0000","last_change":"2024-09-18T08:02:49.887589+0000","last_active":"2024-09-18T08:05:55.470407+0000","last_peered":"2024-09-18T08:05:55.470407+0000","last_clean":"2024-09-18T08:05:55.470407+0000","last_became_active":"2024-09-18T08:02:49.887161+0000","last_became_peered":"2024-09-18T08:02:49.887161+0000","last_unstale":"2024-09-18T08:05:55.470407+0000","last_undegraded":"2024-09-18T08:05:55.470407+0000","last_fullsized":"2024-09-18T08:05:55.470407+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_clean_scrub_stamp":"2024-09-18T08:02:48.870718+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":151,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-09-18T08:05:55.465566+0000","last_change":"2024-09-18T08:05:33.935844+0000","last_active":"2024-09-18T08:05:55.465566+0000","last_peered":"2024-09-18T08:05:55.465566+0000","last_clean":"2024-09-18T08:05:55.465566+0000","last_became_active":"2024-09-18T08:02:51.907296+0000","last_became_peered":"2024-09-18T08:02:51.907296+0000","last_unstale":"2024-09-18T08:05:55.465566+0000","last_undegraded":"2024-09-18T08:05:55.465566+0000","last_fullsized":"2024-09-18T08:05:55.465566+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:05:33.935798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:50.886830+0000","last_clean_scrub_stamp":"2024-09-18T08:05:33.935798+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:06:48.799006+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":140,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-09-18T08:05:55.471836+0000","last_change":"2024-09-18T08:05:01.314402+0000","last_active":"2024-09-18T08:05:55.471836+0000","last_peered":"2024-09-18T08:05:55.471836+0000","last_clean":"2024-09-18T08:05:55.471836+0000","last_became_active":"2024-09-18T08:02:53.913519+0000","last_became_peered":"2024-09-18T08:02:53.913519+0000","last_unstale":"2024-09-18T08:05:55.471836+0000","last_undegraded":"2024-09-18T08:05:55.471836+0000","last_fullsized":"2024-09-18T08:05:55.471836+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:05:01.314330+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_clean_scrub_stamp":"2024-09-18T08:05:01.314330+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:06:05.657205+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":151,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-09-18T08:05:55.465675+0000","last_change":"2024-09-18T08:04:32.849176+0000","last_active":"2024-09-18T08:05:55.465675+0000","last_peered":"2024-09-18T08:05:55.465675+0000","last_clean":"2024-09-18T08:05:55.465675+0000","last_became_active":"2024-09-18T08:02:47.875042+0000","last_became_peered":"2024-09-18T08:02:47.875042+0000","last_unstale":"2024-09-18T08:05:55.465675+0000","last_undegraded":"2024-09-18T08:05:55.465675+0000","last_fullsized":"2024-09-18T08:05:55.465675+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:04:32.849119+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:46.860092+0000","last_clean_scrub_stamp":"2024-09-18T08:04:32.849119+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:05:56.230469+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1b","version":"69'5","reported_seq":157,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-09-18T08:05:55.471449+0000","last_change":"2024-09-18T08:04:31.380754+0000","last_active":"2024-09-18T08:05:55.471449+0000","last_peered":"2024-09-18T08:05:55.471449+0000","last_clean":"2024-09-18T08:05:55.471449+0000","last_became_active":"2024-09-18T08:02:49.895344+0000","last_became_peered":"2024-09-18T08:02:49.895344+0000","last_unstale":"2024-09-18T08:05:55.471449+0000","last_undegraded":"2024-09-18T08:05:55.471449+0000","last_fullsized":"2024-09-18T08:05:55.471449+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'5","last_scrub_stamp":"2024-09-18T08:04:31.380667+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_clean_scrub_stamp":"2024-09-18T08:04:31.380667+0000","objects_scrubbed":2,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":11,"num_read_kb":7,"num_write":6,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":151,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-09-18T08:05:55.461836+0000","last_change":"2024-09-18T08:05:38.866695+0000","last_active":"2024-09-18T08:05:55.461836+0000","last_peered":"2024-09-18T08:05:55.461836+0000","last_clean":"2024-09-18T08:05:55.461836+0000","last_became_active":"2024-09-18T08:02:51.905776+0000","last_became_peered":"2024-09-18T08:02:51.905776+0000","last_unstale":"2024-09-18T08:05:55.461836+0000","last_undegraded":"2024-09-18T08:05:55.461836+0000","last_fullsized":"2024-09-18T08:05:55.461836+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:05:38.866651+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:50.886830+0000","last_clean_scrub_stamp":"2024-09-18T08:05:38.866651+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:06:53.681027+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":140,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-09-18T08:05:55.465392+0000","last_change":"2024-09-18T08:05:39.843588+0000","last_active":"2024-09-18T08:05:55.465392+0000","last_peered":"2024-09-18T08:05:55.465392+0000","last_clean":"2024-09-18T08:05:55.465392+0000","last_became_active":"2024-09-18T08:02:53.911648+0000","last_became_peered":"2024-09-18T08:02:53.911648+0000","last_unstale":"2024-09-18T08:05:55.465392+0000","last_undegraded":"2024-09-18T08:05:55.465392+0000","last_fullsized":"2024-09-18T08:05:55.465392+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:05:39.843536+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_clean_scrub_stamp":"2024-09-18T08:05:39.843536+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:06:40.362817+0000","scrub_duration":20,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":139,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-09-18T08:05:55.470627+0000","last_change":"2024-09-18T08:05:40.478025+0000","last_active":"2024-09-18T08:05:55.470627+0000","last_peered":"2024-09-18T08:05:55.470627+0000","last_clean":"2024-09-18T08:05:55.470627+0000","last_became_active":"2024-09-18T08:02:54.046776+0000","last_became_peered":"2024-09-18T08:02:54.046776+0000","last_unstale":"2024-09-18T08:05:55.470627+0000","last_undegraded":"2024-09-18T08:05:55.470627+0000","last_fullsized":"2024-09-18T08:05:55.470627+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:05:40.477965+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_clean_scrub_stamp":"2024-09-18T08:05:40.477965+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:06:54.789197+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":159,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-09-18T08:05:55.472892+0000","last_change":"2024-09-18T08:05:27.476211+0000","last_active":"2024-09-18T08:05:55.472892+0000","last_peered":"2024-09-18T08:05:55.472892+0000","last_clean":"2024-09-18T08:05:55.472892+0000","last_became_active":"2024-09-18T08:02:47.875963+0000","last_became_peered":"2024-09-18T08:02:47.875963+0000","last_unstale":"2024-09-18T08:05:55.472892+0000","last_undegraded":"2024-09-18T08:05:55.472892+0000","last_fullsized":"2024-09-18T08:05:55.472892+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:05:27.476144+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:46.860092+0000","last_clean_scrub_stamp":"2024-09-18T08:05:27.476144+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:06:42.199865+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1c","version":"69'15","reported_seq":203,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-09-18T08:05:55.468314+0000","last_change":"2024-09-18T08:05:28.084324+0000","last_active":"2024-09-18T08:05:55.468314+0000","last_peered":"2024-09-18T08:05:55.468314+0000","last_clean":"2024-09-18T08:05:55.468314+0000","last_became_active":"2024-09-18T08:04:18.147907+0000","last_became_peered":"2024-09-18T08:04:18.147907+0000","last_unstale":"2024-09-18T08:05:55.468314+0000","last_undegraded":"2024-09-18T08:05:55.468314+0000","last_fullsized":"2024-09-18T08:05:55.468314+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'15","last_scrub_stamp":"2024-09-18T08:05:28.084220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_clean_scrub_stamp":"2024-09-18T08:05:28.084220+0000","objects_scrubbed":7,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:06:39.486316+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,3],"acting":[2,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":151,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-09-18T08:05:55.470923+0000","last_change":"2024-09-18T08:05:40.779422+0000","last_active":"2024-09-18T08:05:55.470923+0000","last_peered":"2024-09-18T08:05:55.470923+0000","last_clean":"2024-09-18T08:05:55.470923+0000","last_became_active":"2024-09-18T08:02:51.908430+0000","last_became_peered":"2024-09-18T08:02:51.908430+0000","last_unstale":"2024-09-18T08:05:55.470923+0000","last_undegraded":"2024-09-18T08:05:55.470923+0000","last_fullsized":"2024-09-18T08:05:55.470923+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:05:40.779230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:50.886830+0000","last_clean_scrub_stamp":"2024-09-18T08:05:40.779230+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:07:00.025019+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":139,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-09-18T08:05:55.460912+0000","last_change":"2024-09-18T08:04:29.956037+0000","last_active":"2024-09-18T08:05:55.460912+0000","last_peered":"2024-09-18T08:05:55.460912+0000","last_clean":"2024-09-18T08:05:55.460912+0000","last_became_active":"2024-09-18T08:02:54.047162+0000","last_became_peered":"2024-09-18T08:02:54.047162+0000","last_unstale":"2024-09-18T08:05:55.460912+0000","last_undegraded":"2024-09-18T08:05:55.460912+0000","last_fullsized":"2024-09-18T08:05:55.460912+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:04:29.955822+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_clean_scrub_stamp":"2024-09-18T08:04:29.955822+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":151,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-09-18T08:05:55.470527+0000","last_change":"2024-09-18T08:04:22.420158+0000","last_active":"2024-09-18T08:05:55.470527+0000","last_peered":"2024-09-18T08:05:55.470527+0000","last_clean":"2024-09-18T08:05:55.470527+0000","last_became_active":"2024-09-18T08:02:47.871523+0000","last_became_peered":"2024-09-18T08:02:47.871523+0000","last_unstale":"2024-09-18T08:05:55.470527+0000","last_undegraded":"2024-09-18T08:05:55.470527+0000","last_fullsized":"2024-09-18T08:05:55.470527+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:04:22.420112+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:46.860092+0000","last_clean_scrub_stamp":"2024-09-18T08:04:22.420112+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"69'12","reported_seq":170,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-09-18T08:05:55.460185+0000","last_change":"2024-09-18T08:05:47.068485+0000","last_active":"2024-09-18T08:05:55.460185+0000","last_peered":"2024-09-18T08:05:55.460185+0000","last_clean":"2024-09-18T08:05:55.460185+0000","last_became_active":"2024-09-18T08:02:49.888831+0000","last_became_peered":"2024-09-18T08:02:49.888831+0000","last_unstale":"2024-09-18T08:05:55.460185+0000","last_undegraded":"2024-09-18T08:05:55.460185+0000","last_fullsized":"2024-09-18T08:05:55.460185+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'12","last_scrub_stamp":"2024-09-18T08:05:47.068375+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_clean_scrub_stamp":"2024-09-18T08:05:47.068375+0000","objects_scrubbed":5,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:06:50.263718+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":25,"num_read_kb":16,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":143,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-09-18T08:05:55.470457+0000","last_change":"2024-09-18T08:04:27.481522+0000","last_active":"2024-09-18T08:05:55.470457+0000","last_peered":"2024-09-18T08:05:55.470457+0000","last_clean":"2024-09-18T08:05:55.470457+0000","last_became_active":"2024-09-18T08:02:51.910316+0000","last_became_peered":"2024-09-18T08:02:51.910316+0000","last_unstale":"2024-09-18T08:05:55.470457+0000","last_undegraded":"2024-09-18T08:05:55.470457+0000","last_fullsized":"2024-09-18T08:05:55.470457+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:04:27.481457+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:50.886830+0000","last_clean_scrub_stamp":"2024-09-18T08:04:27.481457+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1c","version":"69'1","reported_seq":141,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-09-18T08:05:55.462941+0000","last_change":"2024-09-18T08:05:36.776441+0000","last_active":"2024-09-18T08:05:55.462941+0000","last_peered":"2024-09-18T08:05:55.462941+0000","last_clean":"2024-09-18T08:05:55.462941+0000","last_became_active":"2024-09-18T08:02:53.910589+0000","last_became_peered":"2024-09-18T08:02:53.910589+0000","last_unstale":"2024-09-18T08:05:55.462941+0000","last_undegraded":"2024-09-18T08:05:55.462941+0000","last_fullsized":"2024-09-18T08:05:55.462941+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'1","last_scrub_stamp":"2024-09-18T08:05:36.776369+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_clean_scrub_stamp":"2024-09-18T08:05:36.776369+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:06:36.944612+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":403,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.19","version":"62'1","reported_seq":152,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-09-18T08:05:55.470403+0000","last_change":"2024-09-18T08:05:06.819830+0000","last_active":"2024-09-18T08:05:55.470403+0000","last_peered":"2024-09-18T08:05:55.470403+0000","last_clean":"2024-09-18T08:05:55.470403+0000","last_became_active":"2024-09-18T08:02:47.880998+0000","last_became_peered":"2024-09-18T08:02:47.880998+0000","last_unstale":"2024-09-18T08:05:55.470403+0000","last_undegraded":"2024-09-18T08:05:55.470403+0000","last_fullsized":"2024-09-18T08:05:55.470403+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"62'1","last_scrub_stamp":"2024-09-18T08:05:06.819539+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:46.860092+0000","last_clean_scrub_stamp":"2024-09-18T08:05:06.819539+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:06:27.260153+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1e","version":"69'10","reported_seq":154,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-09-18T08:05:55.472277+0000","last_change":"2024-09-18T08:02:49.892755+0000","last_active":"2024-09-18T08:05:55.472277+0000","last_peered":"2024-09-18T08:05:55.472277+0000","last_clean":"2024-09-18T08:05:55.472277+0000","last_became_active":"2024-09-18T08:02:49.892024+0000","last_became_peered":"2024-09-18T08:02:49.892024+0000","last_unstale":"2024-09-18T08:05:55.472277+0000","last_undegraded":"2024-09-18T08:05:55.472277+0000","last_fullsized":"2024-09-18T08:05:55.472277+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_clean_scrub_stamp":"2024-09-18T08:02:48.870718+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1f","version":"102'9","reported_seq":117,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-09-18T08:05:57.966278+0000","last_change":"2024-09-18T08:05:44.463855+0000","last_active":"2024-09-18T08:05:57.966278+0000","last_peered":"2024-09-18T08:05:57.966278+0000","last_clean":"2024-09-18T08:05:57.966278+0000","last_became_active":"2024-09-18T08:04:43.299359+0000","last_became_peered":"2024-09-18T08:04:43.299359+0000","last_unstale":"2024-09-18T08:05:57.966278+0000","last_undegraded":"2024-09-18T08:05:57.966278+0000","last_fullsized":"2024-09-18T08:05:57.966278+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"102'9","last_scrub_stamp":"2024-09-18T08:05:44.463780+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:50.886830+0000","last_clean_scrub_stamp":"2024-09-18T08:05:44.463780+0000","objects_scrubbed":1,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:07:10.059311+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,7],"acting":[4,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.c","version":"127'8954","reported_seq":9059,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-09-18T08:05:58.305489+0000","last_change":"2024-09-18T08:04:43.307818+0000","last_active":"2024-09-18T08:05:58.305489+0000","last_peered":"2024-09-18T08:05:58.305489+0000","last_clean":"2024-09-18T08:05:58.305489+0000","last_became_active":"2024-09-18T08:04:43.300310+0000","last_became_peered":"2024-09-18T08:04:43.300310+0000","last_unstale":"2024-09-18T08:05:58.305489+0000","last_undegraded":"2024-09-18T08:05:58.305489+0000","last_fullsized":"2024-09-18T08:05:58.305489+0000","mapping_epoch":101,"log_start":"102'4300","ondisk_log_start":"102'4300","created":83,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:03:56.449957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:03:56.449957+0000","last_clean_scrub_stamp":"2024-09-18T08:03:56.449957+0000","objects_scrubbed":0,"log_size":4654,"log_dups_size":0,"ondisk_log_size":4654,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":396886016,"num_objects":6056,"num_object_clones":0,"num_object_copies":18168,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6056,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8954,"num_write_kb":480320,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,7],"acting":[3,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.f","version":"69'15","reported_seq":172,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-09-18T08:05:55.469904+0000","last_change":"2024-09-18T08:04:22.741342+0000","last_active":"2024-09-18T08:05:55.469904+0000","last_peered":"2024-09-18T08:05:55.469904+0000","last_clean":"2024-09-18T08:05:55.469904+0000","last_became_active":"2024-09-18T08:02:49.899083+0000","last_became_peered":"2024-09-18T08:02:49.899083+0000","last_unstale":"2024-09-18T08:05:55.469904+0000","last_undegraded":"2024-09-18T08:05:55.469904+0000","last_fullsized":"2024-09-18T08:05:55.469904+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'15","last_scrub_stamp":"2024-09-18T08:04:22.741261+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_clean_scrub_stamp":"2024-09-18T08:04:22.741261+0000","objects_scrubbed":7,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:06:53.535126+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":151,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-09-18T08:05:55.466412+0000","last_change":"2024-09-18T08:04:08.982535+0000","last_active":"2024-09-18T08:05:55.466412+0000","last_peered":"2024-09-18T08:05:55.466412+0000","last_clean":"2024-09-18T08:05:55.466412+0000","last_became_active":"2024-09-18T08:02:47.879328+0000","last_became_peered":"2024-09-18T08:02:47.879328+0000","last_unstale":"2024-09-18T08:05:55.466412+0000","last_undegraded":"2024-09-18T08:05:55.466412+0000","last_fullsized":"2024-09-18T08:05:55.466412+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:04:08.982469+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:46.860092+0000","last_clean_scrub_stamp":"2024-09-18T08:04:08.982469+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.e","version":"69'8","reported_seq":191,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-09-18T08:05:57.966286+0000","last_change":"2024-09-18T08:03:51.342453+0000","last_active":"2024-09-18T08:05:57.966286+0000","last_peered":"2024-09-18T08:05:57.966286+0000","last_clean":"2024-09-18T08:05:57.966286+0000","last_became_active":"2024-09-18T08:02:51.903369+0000","last_became_peered":"2024-09-18T08:02:51.903369+0000","last_unstale":"2024-09-18T08:05:57.966286+0000","last_undegraded":"2024-09-18T08:05:57.966286+0000","last_fullsized":"2024-09-18T08:05:57.966286+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'8","last_scrub_stamp":"2024-09-18T08:03:51.342402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:50.886830+0000","last_clean_scrub_stamp":"2024-09-18T08:03:51.342402+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:06:18.565571+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":148,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-09-18T08:05:55.465701+0000","last_change":"2024-09-18T08:05:34.905147+0000","last_active":"2024-09-18T08:05:55.465701+0000","last_peered":"2024-09-18T08:05:55.465701+0000","last_clean":"2024-09-18T08:05:55.465701+0000","last_became_active":"2024-09-18T08:02:53.909834+0000","last_became_peered":"2024-09-18T08:02:53.909834+0000","last_unstale":"2024-09-18T08:05:55.465701+0000","last_undegraded":"2024-09-18T08:05:55.465701+0000","last_fullsized":"2024-09-18T08:05:55.465701+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:05:34.905083+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_clean_scrub_stamp":"2024-09-18T08:05:34.905083+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:06:53.452264+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.3","version":"127'8781","reported_seq":8880,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-09-18T08:05:58.334966+0000","last_change":"2024-09-18T08:03:57.462489+0000","last_active":"2024-09-18T08:05:58.334966+0000","last_peered":"2024-09-18T08:05:58.334966+0000","last_clean":"2024-09-18T08:05:58.334966+0000","last_became_active":"2024-09-18T08:03:57.462210+0000","last_became_peered":"2024-09-18T08:03:57.462210+0000","last_unstale":"2024-09-18T08:05:58.334966+0000","last_undegraded":"2024-09-18T08:05:58.334966+0000","last_fullsized":"2024-09-18T08:05:58.334966+0000","mapping_epoch":83,"log_start":"102'4100","ondisk_log_start":"102'4100","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:03:56.449957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:03:56.449957+0000","last_clean_scrub_stamp":"2024-09-18T08:03:56.449957+0000","objects_scrubbed":0,"log_size":4681,"log_dups_size":0,"ondisk_log_size":4681,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:06:20.115988+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":386203648,"num_objects":5893,"num_object_clones":0,"num_object_copies":17679,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5893,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8781,"num_write_kb":469568,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,4],"acting":[3,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.0","version":"69'18","reported_seq":179,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-09-18T08:05:55.466186+0000","last_change":"2024-09-18T08:04:14.969799+0000","last_active":"2024-09-18T08:05:55.466186+0000","last_peered":"2024-09-18T08:05:55.466186+0000","last_clean":"2024-09-18T08:05:55.466186+0000","last_became_active":"2024-09-18T08:02:49.886818+0000","last_became_peered":"2024-09-18T08:02:49.886818+0000","last_unstale":"2024-09-18T08:05:55.466186+0000","last_undegraded":"2024-09-18T08:05:55.466186+0000","last_fullsized":"2024-09-18T08:05:55.466186+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'18","last_scrub_stamp":"2024-09-18T08:04:14.969716+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_clean_scrub_stamp":"2024-09-18T08:04:14.969716+0000","objects_scrubbed":8,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":34,"num_read_kb":22,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":159,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-09-18T08:05:55.461216+0000","last_change":"2024-09-18T08:05:35.100407+0000","last_active":"2024-09-18T08:05:55.461216+0000","last_peered":"2024-09-18T08:05:55.461216+0000","last_clean":"2024-09-18T08:05:55.461216+0000","last_became_active":"2024-09-18T08:02:47.880225+0000","last_became_peered":"2024-09-18T08:02:47.880225+0000","last_unstale":"2024-09-18T08:05:55.461216+0000","last_undegraded":"2024-09-18T08:05:55.461216+0000","last_fullsized":"2024-09-18T08:05:55.461216+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:05:35.100332+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:46.860092+0000","last_clean_scrub_stamp":"2024-09-18T08:05:35.100332+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:06:57.779942+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":143,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-09-18T08:05:55.470247+0000","last_change":"2024-09-18T08:04:14.442378+0000","last_active":"2024-09-18T08:05:55.470247+0000","last_peered":"2024-09-18T08:05:55.470247+0000","last_clean":"2024-09-18T08:05:55.470247+0000","last_became_active":"2024-09-18T08:02:51.899594+0000","last_became_peered":"2024-09-18T08:02:51.899594+0000","last_unstale":"2024-09-18T08:05:55.470247+0000","last_undegraded":"2024-09-18T08:05:55.470247+0000","last_fullsized":"2024-09-18T08:05:55.470247+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:04:14.442331+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:50.886830+0000","last_clean_scrub_stamp":"2024-09-18T08:04:14.442331+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":140,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-09-18T08:05:55.470350+0000","last_change":"2024-09-18T08:04:16.427470+0000","last_active":"2024-09-18T08:05:55.470350+0000","last_peered":"2024-09-18T08:05:55.470350+0000","last_clean":"2024-09-18T08:05:55.470350+0000","last_became_active":"2024-09-18T08:02:53.903957+0000","last_became_peered":"2024-09-18T08:02:53.903957+0000","last_unstale":"2024-09-18T08:05:55.470350+0000","last_undegraded":"2024-09-18T08:05:55.470350+0000","last_fullsized":"2024-09-18T08:05:55.470350+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:04:16.427390+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_clean_scrub_stamp":"2024-09-18T08:04:16.427390+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:06:24.577043+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.2","version":"127'8801","reported_seq":8900,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-09-18T08:05:58.305219+0000","last_change":"2024-09-18T08:03:57.463750+0000","last_active":"2024-09-18T08:05:58.305219+0000","last_peered":"2024-09-18T08:05:58.305219+0000","last_clean":"2024-09-18T08:05:58.305219+0000","last_became_active":"2024-09-18T08:03:57.463459+0000","last_became_peered":"2024-09-18T08:03:57.463459+0000","last_unstale":"2024-09-18T08:05:58.305219+0000","last_undegraded":"2024-09-18T08:05:58.305219+0000","last_fullsized":"2024-09-18T08:05:58.305219+0000","mapping_epoch":83,"log_start":"102'4100","ondisk_log_start":"102'4100","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:03:56.449957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:03:56.449957+0000","last_clean_scrub_stamp":"2024-09-18T08:03:56.449957+0000","objects_scrubbed":0,"log_size":4701,"log_dups_size":0,"ondisk_log_size":4701,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":394592256,"num_objects":6021,"num_object_clones":0,"num_object_copies":18063,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6021,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8801,"num_write_kb":474304,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"69'14","reported_seq":176,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-09-18T08:05:55.470257+0000","last_change":"2024-09-18T08:04:56.545313+0000","last_active":"2024-09-18T08:05:55.470257+0000","last_peered":"2024-09-18T08:05:55.470257+0000","last_clean":"2024-09-18T08:05:55.470257+0000","last_became_active":"2024-09-18T08:02:49.893830+0000","last_became_peered":"2024-09-18T08:02:49.893830+0000","last_unstale":"2024-09-18T08:05:55.470257+0000","last_undegraded":"2024-09-18T08:05:55.470257+0000","last_fullsized":"2024-09-18T08:05:55.470257+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'14","last_scrub_stamp":"2024-09-18T08:04:56.545262+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_clean_scrub_stamp":"2024-09-18T08:04:56.545262+0000","objects_scrubbed":7,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:06:17.884382+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":21,"num_read_kb":14,"num_write":14,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"62'2","reported_seq":158,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-09-18T08:05:55.472436+0000","last_change":"2024-09-18T08:04:09.240465+0000","last_active":"2024-09-18T08:05:55.472436+0000","last_peered":"2024-09-18T08:05:55.472436+0000","last_clean":"2024-09-18T08:05:55.472436+0000","last_became_active":"2024-09-18T08:02:47.872857+0000","last_became_peered":"2024-09-18T08:02:47.872857+0000","last_unstale":"2024-09-18T08:05:55.472436+0000","last_undegraded":"2024-09-18T08:05:55.472436+0000","last_fullsized":"2024-09-18T08:05:55.472436+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"62'2","last_scrub_stamp":"2024-09-18T08:04:09.240392+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:46.860092+0000","last_clean_scrub_stamp":"2024-09-18T08:04:09.240392+0000","objects_scrubbed":2,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:06:24.534983+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1039,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.0","version":"69'8","reported_seq":190,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-09-18T08:05:55.463270+0000","last_change":"2024-09-18T08:04:23.863573+0000","last_active":"2024-09-18T08:05:55.463270+0000","last_peered":"2024-09-18T08:05:55.463270+0000","last_clean":"2024-09-18T08:05:55.463270+0000","last_became_active":"2024-09-18T08:02:51.909680+0000","last_became_peered":"2024-09-18T08:02:51.909680+0000","last_unstale":"2024-09-18T08:05:55.463270+0000","last_undegraded":"2024-09-18T08:05:55.463270+0000","last_fullsized":"2024-09-18T08:05:55.463270+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'8","last_scrub_stamp":"2024-09-18T08:04:23.863330+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:50.886830+0000","last_clean_scrub_stamp":"2024-09-18T08:04:23.863330+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":140,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-09-18T08:05:55.462260+0000","last_change":"2024-09-18T08:05:06.885362+0000","last_active":"2024-09-18T08:05:55.462260+0000","last_peered":"2024-09-18T08:05:55.462260+0000","last_clean":"2024-09-18T08:05:55.462260+0000","last_became_active":"2024-09-18T08:02:54.047039+0000","last_became_peered":"2024-09-18T08:02:54.047039+0000","last_unstale":"2024-09-18T08:05:55.462260+0000","last_undegraded":"2024-09-18T08:05:55.462260+0000","last_fullsized":"2024-09-18T08:05:55.462260+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:05:06.885311+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_clean_scrub_stamp":"2024-09-18T08:05:06.885311+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:06:29.893451+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.1","version":"127'9143","reported_seq":9243,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-09-18T08:06:03.041618+0000","last_change":"2024-09-18T08:03:57.465538+0000","last_active":"2024-09-18T08:06:03.041618+0000","last_peered":"2024-09-18T08:06:03.041618+0000","last_clean":"2024-09-18T08:06:03.041618+0000","last_became_active":"2024-09-18T08:03:57.465253+0000","last_became_peered":"2024-09-18T08:03:57.465253+0000","last_unstale":"2024-09-18T08:06:03.041618+0000","last_undegraded":"2024-09-18T08:06:03.041618+0000","last_fullsized":"2024-09-18T08:06:03.041618+0000","mapping_epoch":83,"log_start":"102'4300","ondisk_log_start":"102'4300","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:03:56.449957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:03:56.449957+0000","last_clean_scrub_stamp":"2024-09-18T08:03:56.449957+0000","objects_scrubbed":0,"log_size":4843,"log_dups_size":0,"ondisk_log_size":4843,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:06:15.615182+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":373882880,"num_objects":5705,"num_object_clones":0,"num_object_copies":17115,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5705,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9143,"num_write_kb":475136,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.2","version":"69'10","reported_seq":170,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-09-18T08:05:55.469359+0000","last_change":"2024-09-18T08:05:39.769686+0000","last_active":"2024-09-18T08:05:55.469359+0000","last_peered":"2024-09-18T08:05:55.469359+0000","last_clean":"2024-09-18T08:05:55.469359+0000","last_became_active":"2024-09-18T08:02:49.895265+0000","last_became_peered":"2024-09-18T08:02:49.895265+0000","last_unstale":"2024-09-18T08:05:55.469359+0000","last_undegraded":"2024-09-18T08:05:55.469359+0000","last_fullsized":"2024-09-18T08:05:55.469359+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'10","last_scrub_stamp":"2024-09-18T08:05:39.768393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_clean_scrub_stamp":"2024-09-18T08:05:39.768393+0000","objects_scrubbed":5,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:07:07.361416+0000","scrub_duration":17,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":151,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-09-18T08:05:55.466004+0000","last_change":"2024-09-18T08:03:48.009689+0000","last_active":"2024-09-18T08:05:55.466004+0000","last_peered":"2024-09-18T08:05:55.466004+0000","last_clean":"2024-09-18T08:05:55.466004+0000","last_became_active":"2024-09-18T08:02:47.874727+0000","last_became_peered":"2024-09-18T08:02:47.874727+0000","last_unstale":"2024-09-18T08:05:55.466004+0000","last_undegraded":"2024-09-18T08:05:55.466004+0000","last_fullsized":"2024-09-18T08:05:55.466004+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:03:48.009611+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:46.860092+0000","last_clean_scrub_stamp":"2024-09-18T08:03:48.009611+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:06:18.040455+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.3","version":"69'8","reported_seq":182,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-09-18T08:05:55.472460+0000","last_change":"2024-09-18T08:02:51.906369+0000","last_active":"2024-09-18T08:05:55.472460+0000","last_peered":"2024-09-18T08:05:55.472460+0000","last_clean":"2024-09-18T08:05:55.472460+0000","last_became_active":"2024-09-18T08:02:51.906055+0000","last_became_peered":"2024-09-18T08:02:51.906055+0000","last_unstale":"2024-09-18T08:05:55.472460+0000","last_undegraded":"2024-09-18T08:05:55.472460+0000","last_fullsized":"2024-09-18T08:05:55.472460+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:02:50.886830+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:50.886830+0000","last_clean_scrub_stamp":"2024-09-18T08:02:50.886830+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":148,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-09-18T08:05:55.471988+0000","last_change":"2024-09-18T08:05:43.440713+0000","last_active":"2024-09-18T08:05:55.471988+0000","last_peered":"2024-09-18T08:05:55.471988+0000","last_clean":"2024-09-18T08:05:55.471988+0000","last_became_active":"2024-09-18T08:02:53.909583+0000","last_became_peered":"2024-09-18T08:02:53.909583+0000","last_unstale":"2024-09-18T08:05:55.471988+0000","last_undegraded":"2024-09-18T08:05:55.471988+0000","last_fullsized":"2024-09-18T08:05:55.471988+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:05:43.440642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_clean_scrub_stamp":"2024-09-18T08:05:43.440642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:07:02.845583+0000","scrub_duration":35,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.0","version":"127'9062","reported_seq":9223,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-09-18T08:06:03.076648+0000","last_change":"2024-09-18T08:05:01.419397+0000","last_active":"2024-09-18T08:06:03.076648+0000","last_peered":"2024-09-18T08:06:03.076648+0000","last_clean":"2024-09-18T08:06:03.076648+0000","last_became_active":"2024-09-18T08:03:57.468985+0000","last_became_peered":"2024-09-18T08:03:57.468985+0000","last_unstale":"2024-09-18T08:06:03.076648+0000","last_undegraded":"2024-09-18T08:06:03.076648+0000","last_fullsized":"2024-09-18T08:06:03.076648+0000","mapping_epoch":83,"log_start":"98'3200","ondisk_log_start":"98'3200","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"107'5286","last_scrub_stamp":"2024-09-18T08:05:01.419332+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:03:56.449957+0000","last_clean_scrub_stamp":"2024-09-18T08:05:01.419332+0000","objects_scrubbed":5263,"log_size":5862,"log_dups_size":0,"ondisk_log_size":5862,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:06:20.999535+0000","scrub_duration":1313,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":367263744,"num_objects":5604,"num_object_clones":0,"num_object_copies":16812,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5604,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9062,"num_write_kb":469312,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.3","version":"69'19","reported_seq":183,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-09-18T08:05:55.472628+0000","last_change":"2024-09-18T08:04:07.231406+0000","last_active":"2024-09-18T08:05:55.472628+0000","last_peered":"2024-09-18T08:05:55.472628+0000","last_clean":"2024-09-18T08:05:55.472628+0000","last_became_active":"2024-09-18T08:02:49.890318+0000","last_became_peered":"2024-09-18T08:02:49.890318+0000","last_unstale":"2024-09-18T08:05:55.472628+0000","last_undegraded":"2024-09-18T08:05:55.472628+0000","last_fullsized":"2024-09-18T08:05:55.472628+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'19","last_scrub_stamp":"2024-09-18T08:04:07.229846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_clean_scrub_stamp":"2024-09-18T08:04:07.229846+0000","objects_scrubbed":8,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:06:15.339152+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":39,"num_read_kb":25,"num_write":22,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.4","version":"62'1","reported_seq":157,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-09-18T08:05:55.469711+0000","last_change":"2024-09-18T08:04:28.827545+0000","last_active":"2024-09-18T08:05:55.469711+0000","last_peered":"2024-09-18T08:05:55.469711+0000","last_clean":"2024-09-18T08:05:55.469711+0000","last_became_active":"2024-09-18T08:02:47.874764+0000","last_became_peered":"2024-09-18T08:02:47.874764+0000","last_unstale":"2024-09-18T08:05:55.469711+0000","last_undegraded":"2024-09-18T08:05:55.469711+0000","last_fullsized":"2024-09-18T08:05:55.469711+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"62'1","last_scrub_stamp":"2024-09-18T08:04:28.827464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:46.860092+0000","last_clean_scrub_stamp":"2024-09-18T08:04:28.827464+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:06:52.521799+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":436,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":63,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-09-18T08:05:55.472405+0000","last_change":"2024-09-18T08:04:43.304561+0000","last_active":"2024-09-18T08:05:55.472405+0000","last_peered":"2024-09-18T08:05:55.472405+0000","last_clean":"2024-09-18T08:05:55.472405+0000","last_became_active":"2024-09-18T08:04:43.304002+0000","last_became_peered":"2024-09-18T08:04:43.304002+0000","last_unstale":"2024-09-18T08:05:55.472405+0000","last_undegraded":"2024-09-18T08:05:55.472405+0000","last_fullsized":"2024-09-18T08:05:55.472405+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:04:13.377561+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:50.886830+0000","last_clean_scrub_stamp":"2024-09-18T08:04:13.377561+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":139,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-09-18T08:05:55.469614+0000","last_change":"2024-09-18T08:05:03.874423+0000","last_active":"2024-09-18T08:05:55.469614+0000","last_peered":"2024-09-18T08:05:55.469614+0000","last_clean":"2024-09-18T08:05:55.469614+0000","last_became_active":"2024-09-18T08:02:54.045726+0000","last_became_peered":"2024-09-18T08:02:54.045726+0000","last_unstale":"2024-09-18T08:05:55.469614+0000","last_undegraded":"2024-09-18T08:05:55.469614+0000","last_fullsized":"2024-09-18T08:05:55.469614+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:05:03.874339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_clean_scrub_stamp":"2024-09-18T08:05:03.874339+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:06:08.270559+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.7","version":"127'8837","reported_seq":8947,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-09-18T08:05:58.362295+0000","last_change":"2024-09-18T08:05:13.708697+0000","last_active":"2024-09-18T08:05:58.362295+0000","last_peered":"2024-09-18T08:05:58.362295+0000","last_clean":"2024-09-18T08:05:58.362295+0000","last_became_active":"2024-09-18T08:04:43.300866+0000","last_became_peered":"2024-09-18T08:04:43.300866+0000","last_unstale":"2024-09-18T08:05:58.362295+0000","last_undegraded":"2024-09-18T08:05:58.362295+0000","last_fullsized":"2024-09-18T08:05:58.362295+0000","mapping_epoch":101,"log_start":"98'3200","ondisk_log_start":"98'3200","created":83,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:03:56.449957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:03:56.449957+0000","last_clean_scrub_stamp":"2024-09-18T08:03:56.449957+0000","objects_scrubbed":399,"log_size":5637,"log_dups_size":0,"ondisk_log_size":5637,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":395771904,"num_objects":6039,"num_object_clones":0,"num_object_copies":18117,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6039,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8837,"num_write_kb":476032,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,3],"acting":[0,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.4","version":"69'28","reported_seq":234,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-09-18T08:05:55.469598+0000","last_change":"2024-09-18T08:04:12.737618+0000","last_active":"2024-09-18T08:05:55.469598+0000","last_peered":"2024-09-18T08:05:55.469598+0000","last_clean":"2024-09-18T08:05:55.469598+0000","last_became_active":"2024-09-18T08:02:49.897221+0000","last_became_peered":"2024-09-18T08:02:49.897221+0000","last_unstale":"2024-09-18T08:05:55.469598+0000","last_undegraded":"2024-09-18T08:05:55.469598+0000","last_fullsized":"2024-09-18T08:05:55.469598+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'28","last_scrub_stamp":"2024-09-18T08:04:12.737528+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_clean_scrub_stamp":"2024-09-18T08:04:12.737528+0000","objects_scrubbed":10,"log_size":28,"log_dups_size":0,"ondisk_log_size":28,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:06:50.593181+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":48,"num_read_kb":33,"num_write":26,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":151,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-09-18T08:05:55.471565+0000","last_change":"2024-09-18T08:05:39.490204+0000","last_active":"2024-09-18T08:05:55.471565+0000","last_peered":"2024-09-18T08:05:55.471565+0000","last_clean":"2024-09-18T08:05:55.471565+0000","last_became_active":"2024-09-18T08:02:47.877085+0000","last_became_peered":"2024-09-18T08:02:47.877085+0000","last_unstale":"2024-09-18T08:05:55.471565+0000","last_undegraded":"2024-09-18T08:05:55.471565+0000","last_fullsized":"2024-09-18T08:05:55.471565+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:05:39.490141+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:46.860092+0000","last_clean_scrub_stamp":"2024-09-18T08:05:39.490141+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:06:51.573012+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"64'2","reported_seq":169,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-09-18T08:05:55.465597+0000","last_change":"2024-09-18T08:05:44.803021+0000","last_active":"2024-09-18T08:05:55.465597+0000","last_peered":"2024-09-18T08:05:55.465597+0000","last_clean":"2024-09-18T08:05:55.465597+0000","last_became_active":"2024-09-18T08:02:46.867802+0000","last_became_peered":"2024-09-18T08:02:46.867802+0000","last_unstale":"2024-09-18T08:05:55.465597+0000","last_undegraded":"2024-09-18T08:05:55.465597+0000","last_fullsized":"2024-09-18T08:05:55.465597+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"64'2","last_scrub_stamp":"2024-09-18T08:05:44.802942+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:45.855979+0000","last_clean_scrub_stamp":"2024-09-18T08:05:44.802942+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:07:03.506811+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.001348349,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":135,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-09-18T08:05:55.471989+0000","last_change":"2024-09-18T08:02:51.905694+0000","last_active":"2024-09-18T08:05:55.471989+0000","last_peered":"2024-09-18T08:05:55.471989+0000","last_clean":"2024-09-18T08:05:55.471989+0000","last_became_active":"2024-09-18T08:02:51.905357+0000","last_became_peered":"2024-09-18T08:02:51.905357+0000","last_unstale":"2024-09-18T08:05:55.471989+0000","last_undegraded":"2024-09-18T08:05:55.471989+0000","last_fullsized":"2024-09-18T08:05:55.471989+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:02:50.886830+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:50.886830+0000","last_clean_scrub_stamp":"2024-09-18T08:02:50.886830+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:06:51.299235+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":140,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-09-18T08:05:55.460455+0000","last_change":"2024-09-18T08:05:04.962900+0000","last_active":"2024-09-18T08:05:55.460455+0000","last_peered":"2024-09-18T08:05:55.460455+0000","last_clean":"2024-09-18T08:05:55.460455+0000","last_became_active":"2024-09-18T08:02:53.911036+0000","last_became_peered":"2024-09-18T08:02:53.911036+0000","last_unstale":"2024-09-18T08:05:55.460455+0000","last_undegraded":"2024-09-18T08:05:55.460455+0000","last_fullsized":"2024-09-18T08:05:55.460455+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:05:04.962686+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_clean_scrub_stamp":"2024-09-18T08:05:04.962686+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:06:08.603603+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.4","version":"127'8896","reported_seq":8995,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-09-18T08:05:58.303619+0000","last_change":"2024-09-18T08:03:57.466551+0000","last_active":"2024-09-18T08:05:58.303619+0000","last_peered":"2024-09-18T08:05:58.303619+0000","last_clean":"2024-09-18T08:05:58.303619+0000","last_became_active":"2024-09-18T08:03:57.466232+0000","last_became_peered":"2024-09-18T08:03:57.466232+0000","last_unstale":"2024-09-18T08:05:58.303619+0000","last_undegraded":"2024-09-18T08:05:58.303619+0000","last_fullsized":"2024-09-18T08:05:58.303619+0000","mapping_epoch":83,"log_start":"98'3300","ondisk_log_start":"98'3300","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:03:56.449957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:03:56.449957+0000","last_clean_scrub_stamp":"2024-09-18T08:03:56.449957+0000","objects_scrubbed":0,"log_size":5596,"log_dups_size":0,"ondisk_log_size":5596,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:06:25.555609+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":391118848,"num_objects":5968,"num_object_clones":0,"num_object_copies":17904,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5968,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8896,"num_write_kb":475648,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,1],"acting":[0,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.7","version":"69'13","reported_seq":92,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-09-18T08:05:55.468173+0000","last_change":"2024-09-18T08:04:18.164662+0000","last_active":"2024-09-18T08:05:55.468173+0000","last_peered":"2024-09-18T08:05:55.468173+0000","last_clean":"2024-09-18T08:05:55.468173+0000","last_became_active":"2024-09-18T08:04:18.144925+0000","last_became_peered":"2024-09-18T08:04:18.144925+0000","last_unstale":"2024-09-18T08:05:55.468173+0000","last_undegraded":"2024-09-18T08:05:55.468173+0000","last_fullsized":"2024-09-18T08:05:55.468173+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'13","last_scrub_stamp":"2024-09-18T08:03:50.700818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_clean_scrub_stamp":"2024-09-18T08:03:50.700818+0000","objects_scrubbed":0,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:06:19.016400+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":29,"num_read_kb":18,"num_write":16,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,0],"acting":[2,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":151,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-09-18T08:05:55.471425+0000","last_change":"2024-09-18T08:03:49.704071+0000","last_active":"2024-09-18T08:05:55.471425+0000","last_peered":"2024-09-18T08:05:55.471425+0000","last_clean":"2024-09-18T08:05:55.471425+0000","last_became_active":"2024-09-18T08:02:47.871416+0000","last_became_peered":"2024-09-18T08:02:47.871416+0000","last_unstale":"2024-09-18T08:05:55.471425+0000","last_undegraded":"2024-09-18T08:05:55.471425+0000","last_fullsized":"2024-09-18T08:05:55.471425+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:03:49.704004+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:46.860092+0000","last_clean_scrub_stamp":"2024-09-18T08:03:49.704004+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:06:21.766689+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"62'1","reported_seq":160,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-09-18T08:05:55.465976+0000","last_change":"2024-09-18T08:04:09.732724+0000","last_active":"2024-09-18T08:05:55.465976+0000","last_peered":"2024-09-18T08:05:55.465976+0000","last_clean":"2024-09-18T08:05:55.465976+0000","last_became_active":"2024-09-18T08:02:46.868175+0000","last_became_peered":"2024-09-18T08:02:46.868175+0000","last_unstale":"2024-09-18T08:05:55.465976+0000","last_undegraded":"2024-09-18T08:05:55.465976+0000","last_fullsized":"2024-09-18T08:05:55.465976+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"62'1","last_scrub_stamp":"2024-09-18T08:04:09.732645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:45.855979+0000","last_clean_scrub_stamp":"2024-09-18T08:04:09.732645+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:06:24.986709+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0.00066785499999999997,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":151,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-09-18T08:05:55.468194+0000","last_change":"2024-09-18T08:05:41.059426+0000","last_active":"2024-09-18T08:05:55.468194+0000","last_peered":"2024-09-18T08:05:55.468194+0000","last_clean":"2024-09-18T08:05:55.468194+0000","last_became_active":"2024-09-18T08:02:51.908156+0000","last_became_peered":"2024-09-18T08:02:51.908156+0000","last_unstale":"2024-09-18T08:05:55.468194+0000","last_undegraded":"2024-09-18T08:05:55.468194+0000","last_fullsized":"2024-09-18T08:05:55.468194+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:05:41.059230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:50.886830+0000","last_clean_scrub_stamp":"2024-09-18T08:05:41.059230+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:06:45.166099+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":140,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-09-18T08:05:55.462302+0000","last_change":"2024-09-18T08:05:40.929414+0000","last_active":"2024-09-18T08:05:55.462302+0000","last_peered":"2024-09-18T08:05:55.462302+0000","last_clean":"2024-09-18T08:05:55.462302+0000","last_became_active":"2024-09-18T08:02:54.048004+0000","last_became_peered":"2024-09-18T08:02:54.048004+0000","last_unstale":"2024-09-18T08:05:55.462302+0000","last_undegraded":"2024-09-18T08:05:55.462302+0000","last_fullsized":"2024-09-18T08:05:55.462302+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:05:40.929345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_clean_scrub_stamp":"2024-09-18T08:05:40.929345+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:06:57.162980+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.5","version":"127'9090","reported_seq":9189,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-09-18T08:06:03.083161+0000","last_change":"2024-09-18T08:03:57.469151+0000","last_active":"2024-09-18T08:06:03.083161+0000","last_peered":"2024-09-18T08:06:03.083161+0000","last_clean":"2024-09-18T08:06:03.083161+0000","last_became_active":"2024-09-18T08:03:57.468853+0000","last_became_peered":"2024-09-18T08:03:57.468853+0000","last_unstale":"2024-09-18T08:06:03.083161+0000","last_undegraded":"2024-09-18T08:06:03.083161+0000","last_fullsized":"2024-09-18T08:06:03.083161+0000","mapping_epoch":83,"log_start":"100'3900","ondisk_log_start":"100'3900","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:03:56.449957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:03:56.449957+0000","last_clean_scrub_stamp":"2024-09-18T08:03:56.449957+0000","objects_scrubbed":0,"log_size":5190,"log_dups_size":0,"ondisk_log_size":5190,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:06:20.730047+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":370802688,"num_objects":5658,"num_object_clones":0,"num_object_copies":16974,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5658,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9090,"num_write_kb":471936,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,0],"acting":[7,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"69'12","reported_seq":165,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-09-18T08:05:55.472338+0000","last_change":"2024-09-18T08:05:38.471012+0000","last_active":"2024-09-18T08:05:55.472338+0000","last_peered":"2024-09-18T08:05:55.472338+0000","last_clean":"2024-09-18T08:05:55.472338+0000","last_became_active":"2024-09-18T08:02:49.898079+0000","last_became_peered":"2024-09-18T08:02:49.898079+0000","last_unstale":"2024-09-18T08:05:55.472338+0000","last_undegraded":"2024-09-18T08:05:55.472338+0000","last_fullsized":"2024-09-18T08:05:55.472338+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'12","last_scrub_stamp":"2024-09-18T08:05:38.470713+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_clean_scrub_stamp":"2024-09-18T08:05:38.470713+0000","objects_scrubbed":6,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:06:50.866624+0000","scrub_duration":28,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":18,"num_read_kb":12,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":151,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-09-18T08:05:55.473394+0000","last_change":"2024-09-18T08:04:59.298230+0000","last_active":"2024-09-18T08:05:55.473394+0000","last_peered":"2024-09-18T08:05:55.473394+0000","last_clean":"2024-09-18T08:05:55.473394+0000","last_became_active":"2024-09-18T08:02:47.873809+0000","last_became_peered":"2024-09-18T08:02:47.873809+0000","last_unstale":"2024-09-18T08:05:55.473394+0000","last_undegraded":"2024-09-18T08:05:55.473394+0000","last_fullsized":"2024-09-18T08:05:55.473394+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:04:59.298029+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:46.860092+0000","last_clean_scrub_stamp":"2024-09-18T08:04:59.298029+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:06:19.180037+0000","scrub_duration":31,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"70'5","reported_seq":352,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-09-18T08:06:01.884264+0000","last_change":"2024-09-18T08:04:06.814615+0000","last_active":"2024-09-18T08:06:01.884264+0000","last_peered":"2024-09-18T08:06:01.884264+0000","last_clean":"2024-09-18T08:06:01.884264+0000","last_became_active":"2024-09-18T08:02:46.872144+0000","last_became_peered":"2024-09-18T08:02:46.872144+0000","last_unstale":"2024-09-18T08:06:01.884264+0000","last_undegraded":"2024-09-18T08:06:01.884264+0000","last_fullsized":"2024-09-18T08:06:01.884264+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"70'5","last_scrub_stamp":"2024-09-18T08:04:06.814540+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:45.855979+0000","last_clean_scrub_stamp":"2024-09-18T08:04:06.814540+0000","objects_scrubbed":1,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:06:24.789846+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.000504055,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":190,"num_read_kb":185,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":151,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-09-18T08:05:55.466250+0000","last_change":"2024-09-18T08:05:41.750515+0000","last_active":"2024-09-18T08:05:55.466250+0000","last_peered":"2024-09-18T08:05:55.466250+0000","last_clean":"2024-09-18T08:05:55.466250+0000","last_became_active":"2024-09-18T08:02:51.913112+0000","last_became_peered":"2024-09-18T08:02:51.913112+0000","last_unstale":"2024-09-18T08:05:55.466250+0000","last_undegraded":"2024-09-18T08:05:55.466250+0000","last_fullsized":"2024-09-18T08:05:55.466250+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:05:41.750446+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:50.886830+0000","last_clean_scrub_stamp":"2024-09-18T08:05:41.750446+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:06:55.866373+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":132,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-09-18T08:05:55.469068+0000","last_change":"2024-09-18T08:02:53.907689+0000","last_active":"2024-09-18T08:05:55.469068+0000","last_peered":"2024-09-18T08:05:55.469068+0000","last_clean":"2024-09-18T08:05:55.469068+0000","last_became_active":"2024-09-18T08:02:53.907404+0000","last_became_peered":"2024-09-18T08:02:53.907404+0000","last_unstale":"2024-09-18T08:05:55.469068+0000","last_undegraded":"2024-09-18T08:05:55.469068+0000","last_fullsized":"2024-09-18T08:05:55.469068+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_clean_scrub_stamp":"2024-09-18T08:02:52.890475+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:06:18.732348+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.6","version":"127'8774","reported_seq":8916,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-09-18T08:05:58.346706+0000","last_change":"2024-09-18T08:05:13.657923+0000","last_active":"2024-09-18T08:05:58.346706+0000","last_peered":"2024-09-18T08:05:58.346706+0000","last_clean":"2024-09-18T08:05:58.346706+0000","last_became_active":"2024-09-18T08:03:57.466391+0000","last_became_peered":"2024-09-18T08:03:57.466391+0000","last_unstale":"2024-09-18T08:05:58.346706+0000","last_undegraded":"2024-09-18T08:05:58.346706+0000","last_fullsized":"2024-09-18T08:05:58.346706+0000","mapping_epoch":83,"log_start":"98'3200","ondisk_log_start":"98'3200","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:03:56.449957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:03:56.449957+0000","last_clean_scrub_stamp":"2024-09-18T08:03:56.449957+0000","objects_scrubbed":3899,"log_size":5574,"log_dups_size":0,"ondisk_log_size":5574,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":393216000,"num_objects":6000,"num_object_clones":0,"num_object_copies":18000,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6000,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8774,"num_write_kb":472768,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.5","version":"102'17","reported_seq":124,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-09-18T08:05:57.966336+0000","last_change":"2024-09-18T08:04:43.301741+0000","last_active":"2024-09-18T08:05:57.966336+0000","last_peered":"2024-09-18T08:05:57.966336+0000","last_clean":"2024-09-18T08:05:57.966336+0000","last_became_active":"2024-09-18T08:04:43.301400+0000","last_became_peered":"2024-09-18T08:04:43.301400+0000","last_unstale":"2024-09-18T08:05:57.966336+0000","last_undegraded":"2024-09-18T08:05:57.966336+0000","last_fullsized":"2024-09-18T08:05:57.966336+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'16","last_scrub_stamp":"2024-09-18T08:04:14.358816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_clean_scrub_stamp":"2024-09-18T08:04:14.358816+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:06:21.649266+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":24,"num_read_kb":14,"num_write":13,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,7],"acting":[4,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":151,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-09-18T08:05:55.464092+0000","last_change":"2024-09-18T08:05:38.081633+0000","last_active":"2024-09-18T08:05:55.464092+0000","last_peered":"2024-09-18T08:05:55.464092+0000","last_clean":"2024-09-18T08:05:55.464092+0000","last_became_active":"2024-09-18T08:02:47.875950+0000","last_became_peered":"2024-09-18T08:02:47.875950+0000","last_unstale":"2024-09-18T08:05:55.464092+0000","last_undegraded":"2024-09-18T08:05:55.464092+0000","last_fullsized":"2024-09-18T08:05:55.464092+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:05:38.080993+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:46.860092+0000","last_clean_scrub_stamp":"2024-09-18T08:05:38.080993+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:06:45.582435+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"74'79","reported_seq":198,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-09-18T08:05:55.461915+0000","last_change":"2024-09-18T08:04:33.791905+0000","last_active":"2024-09-18T08:05:55.461915+0000","last_peered":"2024-09-18T08:05:55.461915+0000","last_clean":"2024-09-18T08:05:55.461915+0000","last_became_active":"2024-09-18T08:02:39.446743+0000","last_became_peered":"2024-09-18T08:02:39.446743+0000","last_unstale":"2024-09-18T08:05:55.461915+0000","last_undegraded":"2024-09-18T08:05:55.461915+0000","last_fullsized":"2024-09-18T08:05:55.461915+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'79","last_scrub_stamp":"2024-09-18T08:04:33.791841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:00:11.893742+0000","last_clean_scrub_stamp":"2024-09-18T08:04:33.791841+0000","objects_scrubbed":2,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:06:01.148512+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":231,"num_write":113,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":590368,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":151,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-09-18T08:05:55.463099+0000","last_change":"2024-09-18T08:05:42.929394+0000","last_active":"2024-09-18T08:05:55.463099+0000","last_peered":"2024-09-18T08:05:55.463099+0000","last_clean":"2024-09-18T08:05:55.463099+0000","last_became_active":"2024-09-18T08:02:51.905516+0000","last_became_peered":"2024-09-18T08:02:51.905516+0000","last_unstale":"2024-09-18T08:05:55.463099+0000","last_undegraded":"2024-09-18T08:05:55.463099+0000","last_fullsized":"2024-09-18T08:05:55.463099+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:05:42.929311+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:50.886830+0000","last_clean_scrub_stamp":"2024-09-18T08:05:42.929311+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:06:48.900458+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":140,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-09-18T08:05:55.466198+0000","last_change":"2024-09-18T08:04:11.930843+0000","last_active":"2024-09-18T08:05:55.466198+0000","last_peered":"2024-09-18T08:05:55.466198+0000","last_clean":"2024-09-18T08:05:55.466198+0000","last_became_active":"2024-09-18T08:02:53.908668+0000","last_became_peered":"2024-09-18T08:02:53.908668+0000","last_unstale":"2024-09-18T08:05:55.466198+0000","last_undegraded":"2024-09-18T08:05:55.466198+0000","last_fullsized":"2024-09-18T08:05:55.466198+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:04:11.930797+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_clean_scrub_stamp":"2024-09-18T08:04:11.930797+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:06:49.753536+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.d","version":"127'8857","reported_seq":9038,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-09-18T08:05:58.152585+0000","last_change":"2024-09-18T08:05:44.591209+0000","last_active":"2024-09-18T08:05:58.152585+0000","last_peered":"2024-09-18T08:05:58.152585+0000","last_clean":"2024-09-18T08:05:58.152585+0000","last_became_active":"2024-09-18T08:04:43.300869+0000","last_became_peered":"2024-09-18T08:04:43.300869+0000","last_unstale":"2024-09-18T08:05:58.152585+0000","last_undegraded":"2024-09-18T08:05:58.152585+0000","last_fullsized":"2024-09-18T08:05:58.152585+0000","mapping_epoch":101,"log_start":"100'3800","ondisk_log_start":"100'3800","created":83,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"123'7982","last_scrub_stamp":"2024-09-18T08:05:44.590884+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:03:56.449957+0000","last_clean_scrub_stamp":"2024-09-18T08:05:44.590884+0000","objects_scrubbed":6877,"log_size":5057,"log_dups_size":0,"ondisk_log_size":5057,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:07:10.003353+0000","scrub_duration":2853,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":387448856,"num_objects":5913,"num_object_clones":0,"num_object_copies":17739,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5913,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":8857,"num_write_kb":472577,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,0],"acting":[1,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.e","version":"69'11","reported_seq":158,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-09-18T08:05:55.470818+0000","last_change":"2024-09-18T08:02:49.895676+0000","last_active":"2024-09-18T08:05:55.470818+0000","last_peered":"2024-09-18T08:05:55.470818+0000","last_clean":"2024-09-18T08:05:55.470818+0000","last_became_active":"2024-09-18T08:02:49.895303+0000","last_became_peered":"2024-09-18T08:02:49.895303+0000","last_unstale":"2024-09-18T08:05:55.470818+0000","last_undegraded":"2024-09-18T08:05:55.470818+0000","last_fullsized":"2024-09-18T08:05:55.470818+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_clean_scrub_stamp":"2024-09-18T08:02:48.870718+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:06:48.446771+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":143,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-09-18T08:05:55.470445+0000","last_change":"2024-09-18T08:02:47.880265+0000","last_active":"2024-09-18T08:05:55.470445+0000","last_peered":"2024-09-18T08:05:55.470445+0000","last_clean":"2024-09-18T08:05:55.470445+0000","last_became_active":"2024-09-18T08:02:47.879764+0000","last_became_peered":"2024-09-18T08:02:47.879764+0000","last_unstale":"2024-09-18T08:05:55.470445+0000","last_undegraded":"2024-09-18T08:05:55.470445+0000","last_fullsized":"2024-09-18T08:05:55.470445+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:02:46.860092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:46.860092+0000","last_clean_scrub_stamp":"2024-09-18T08:02:46.860092+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:06:53.348536+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":143,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-09-18T08:05:55.466138+0000","last_change":"2024-09-18T08:05:08.839711+0000","last_active":"2024-09-18T08:05:55.466138+0000","last_peered":"2024-09-18T08:05:55.466138+0000","last_clean":"2024-09-18T08:05:55.466138+0000","last_became_active":"2024-09-18T08:02:51.910649+0000","last_became_peered":"2024-09-18T08:02:51.910649+0000","last_unstale":"2024-09-18T08:05:55.466138+0000","last_undegraded":"2024-09-18T08:05:55.466138+0000","last_fullsized":"2024-09-18T08:05:55.466138+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:05:08.839644+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:50.886830+0000","last_clean_scrub_stamp":"2024-09-18T08:05:08.839644+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:06:13.625868+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":139,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-09-18T08:05:55.459812+0000","last_change":"2024-09-18T08:05:09.038204+0000","last_active":"2024-09-18T08:05:55.459812+0000","last_peered":"2024-09-18T08:05:55.459812+0000","last_clean":"2024-09-18T08:05:55.459812+0000","last_became_active":"2024-09-18T08:02:54.045091+0000","last_became_peered":"2024-09-18T08:02:54.045091+0000","last_unstale":"2024-09-18T08:05:55.459812+0000","last_undegraded":"2024-09-18T08:05:55.459812+0000","last_fullsized":"2024-09-18T08:05:55.459812+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:05:09.038115+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_clean_scrub_stamp":"2024-09-18T08:05:09.038115+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:06:18.302588+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.e","version":"127'8812","reported_seq":9030,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-09-18T08:05:58.231640+0000","last_change":"2024-09-18T08:05:47.424879+0000","last_active":"2024-09-18T08:05:58.231640+0000","last_peered":"2024-09-18T08:05:58.231640+0000","last_clean":"2024-09-18T08:05:58.231640+0000","last_became_active":"2024-09-18T08:03:57.464311+0000","last_became_peered":"2024-09-18T08:03:57.464311+0000","last_unstale":"2024-09-18T08:05:58.231640+0000","last_undegraded":"2024-09-18T08:05:58.231640+0000","last_fullsized":"2024-09-18T08:05:58.231640+0000","mapping_epoch":83,"log_start":"93'1900","ondisk_log_start":"93'1900","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:03:56.449957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:03:56.449957+0000","last_clean_scrub_stamp":"2024-09-18T08:03:56.449957+0000","objects_scrubbed":5849,"log_size":6912,"log_dups_size":0,"ondisk_log_size":6912,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:06:17.403222+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":390725632,"num_objects":5962,"num_object_clones":0,"num_object_copies":17886,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5962,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8812,"num_write_kb":472768,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,0],"acting":[2,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.d","version":"69'17","reported_seq":167,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-09-18T08:05:55.470840+0000","last_change":"2024-09-18T08:02:49.895971+0000","last_active":"2024-09-18T08:05:55.470840+0000","last_peered":"2024-09-18T08:05:55.470840+0000","last_clean":"2024-09-18T08:05:55.470840+0000","last_became_active":"2024-09-18T08:02:49.895578+0000","last_became_peered":"2024-09-18T08:02:49.895578+0000","last_unstale":"2024-09-18T08:05:55.470840+0000","last_undegraded":"2024-09-18T08:05:55.470840+0000","last_fullsized":"2024-09-18T08:05:55.470840+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_clean_scrub_stamp":"2024-09-18T08:02:48.870718+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":29,"num_read_kb":19,"num_write":18,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,1],"acting":[4,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":71,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-09-18T08:05:55.470447+0000","last_change":"2024-09-18T08:05:12.622971+0000","last_active":"2024-09-18T08:05:55.470447+0000","last_peered":"2024-09-18T08:05:55.470447+0000","last_clean":"2024-09-18T08:05:55.470447+0000","last_became_active":"2024-09-18T08:04:43.302507+0000","last_became_peered":"2024-09-18T08:04:43.302507+0000","last_unstale":"2024-09-18T08:05:55.470447+0000","last_undegraded":"2024-09-18T08:05:55.470447+0000","last_fullsized":"2024-09-18T08:05:55.470447+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:05:12.622901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:46.860092+0000","last_clean_scrub_stamp":"2024-09-18T08:05:12.622901+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:06:36.771484+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":151,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-09-18T08:05:55.470166+0000","last_change":"2024-09-18T08:05:31.851989+0000","last_active":"2024-09-18T08:05:55.470166+0000","last_peered":"2024-09-18T08:05:55.470166+0000","last_clean":"2024-09-18T08:05:55.470166+0000","last_became_active":"2024-09-18T08:02:51.907196+0000","last_became_peered":"2024-09-18T08:02:51.907196+0000","last_unstale":"2024-09-18T08:05:55.470166+0000","last_undegraded":"2024-09-18T08:05:55.470166+0000","last_fullsized":"2024-09-18T08:05:55.470166+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:05:31.851797+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:50.886830+0000","last_clean_scrub_stamp":"2024-09-18T08:05:31.851797+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:06:59.697560+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":140,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-09-18T08:05:55.468348+0000","last_change":"2024-09-18T08:04:06.709703+0000","last_active":"2024-09-18T08:05:55.468348+0000","last_peered":"2024-09-18T08:05:55.468348+0000","last_clean":"2024-09-18T08:05:55.468348+0000","last_became_active":"2024-09-18T08:02:53.904144+0000","last_became_peered":"2024-09-18T08:02:53.904144+0000","last_unstale":"2024-09-18T08:05:55.468348+0000","last_undegraded":"2024-09-18T08:05:55.468348+0000","last_fullsized":"2024-09-18T08:05:55.468348+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:04:06.709631+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_clean_scrub_stamp":"2024-09-18T08:04:06.709631+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:06:14.883587+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.f","version":"127'8791","reported_seq":8951,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-09-18T08:05:58.304252+0000","last_change":"2024-09-18T08:05:13.677385+0000","last_active":"2024-09-18T08:05:58.304252+0000","last_peered":"2024-09-18T08:05:58.304252+0000","last_clean":"2024-09-18T08:05:58.304252+0000","last_became_active":"2024-09-18T08:03:57.467980+0000","last_became_peered":"2024-09-18T08:03:57.467980+0000","last_unstale":"2024-09-18T08:05:58.304252+0000","last_undegraded":"2024-09-18T08:05:58.304252+0000","last_fullsized":"2024-09-18T08:05:58.304252+0000","mapping_epoch":83,"log_start":"98'3200","ondisk_log_start":"98'3200","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:03:56.449957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:03:56.449957+0000","last_clean_scrub_stamp":"2024-09-18T08:03:56.449957+0000","objects_scrubbed":5349,"log_size":5591,"log_dups_size":0,"ondisk_log_size":5591,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:06:48.316140+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":385286144,"num_objects":5879,"num_object_clones":0,"num_object_copies":17637,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5879,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8791,"num_write_kb":469440,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.c","version":"69'10","reported_seq":162,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-09-18T08:05:55.470555+0000","last_change":"2024-09-18T08:05:41.474883+0000","last_active":"2024-09-18T08:05:55.470555+0000","last_peered":"2024-09-18T08:05:55.470555+0000","last_clean":"2024-09-18T08:05:55.470555+0000","last_became_active":"2024-09-18T08:02:49.892527+0000","last_became_peered":"2024-09-18T08:02:49.892527+0000","last_unstale":"2024-09-18T08:05:55.470555+0000","last_undegraded":"2024-09-18T08:05:55.470555+0000","last_fullsized":"2024-09-18T08:05:55.470555+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'10","last_scrub_stamp":"2024-09-18T08:05:41.474813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_clean_scrub_stamp":"2024-09-18T08:05:41.474813+0000","objects_scrubbed":5,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:06:51.749825+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":151,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-09-18T08:05:55.460394+0000","last_change":"2024-09-18T08:04:24.892153+0000","last_active":"2024-09-18T08:05:55.460394+0000","last_peered":"2024-09-18T08:05:55.460394+0000","last_clean":"2024-09-18T08:05:55.460394+0000","last_became_active":"2024-09-18T08:02:47.881512+0000","last_became_peered":"2024-09-18T08:02:47.881512+0000","last_unstale":"2024-09-18T08:05:55.460394+0000","last_undegraded":"2024-09-18T08:05:55.460394+0000","last_fullsized":"2024-09-18T08:05:55.460394+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:04:24.892079+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:46.860092+0000","last_clean_scrub_stamp":"2024-09-18T08:04:24.892079+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.d","version":"69'8","reported_seq":198,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-09-18T08:05:55.468052+0000","last_change":"2024-09-18T08:05:33.027988+0000","last_active":"2024-09-18T08:05:55.468052+0000","last_peered":"2024-09-18T08:05:55.468052+0000","last_clean":"2024-09-18T08:05:55.468052+0000","last_became_active":"2024-09-18T08:02:51.909011+0000","last_became_peered":"2024-09-18T08:02:51.909011+0000","last_unstale":"2024-09-18T08:05:55.468052+0000","last_undegraded":"2024-09-18T08:05:55.468052+0000","last_fullsized":"2024-09-18T08:05:55.468052+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'8","last_scrub_stamp":"2024-09-18T08:05:33.027783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:50.886830+0000","last_clean_scrub_stamp":"2024-09-18T08:05:33.027783+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:06:41.650338+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":140,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-09-18T08:05:55.470522+0000","last_change":"2024-09-18T08:04:32.377211+0000","last_active":"2024-09-18T08:05:55.470522+0000","last_peered":"2024-09-18T08:05:55.470522+0000","last_clean":"2024-09-18T08:05:55.470522+0000","last_became_active":"2024-09-18T08:02:53.901858+0000","last_became_peered":"2024-09-18T08:02:53.901858+0000","last_unstale":"2024-09-18T08:05:55.470522+0000","last_undegraded":"2024-09-18T08:05:55.470522+0000","last_fullsized":"2024-09-18T08:05:55.470522+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:04:32.377157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_clean_scrub_stamp":"2024-09-18T08:04:32.377157+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.8","version":"127'8836","reported_seq":8935,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-09-18T08:05:58.223016+0000","last_change":"2024-09-18T08:03:57.465051+0000","last_active":"2024-09-18T08:05:58.223016+0000","last_peered":"2024-09-18T08:05:58.223016+0000","last_clean":"2024-09-18T08:05:58.223016+0000","last_became_active":"2024-09-18T08:03:57.464738+0000","last_became_peered":"2024-09-18T08:03:57.464738+0000","last_unstale":"2024-09-18T08:05:58.223016+0000","last_undegraded":"2024-09-18T08:05:58.223016+0000","last_fullsized":"2024-09-18T08:05:58.223016+0000","mapping_epoch":83,"log_start":"94'2000","ondisk_log_start":"94'2000","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:03:56.449957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:03:56.449957+0000","last_clean_scrub_stamp":"2024-09-18T08:03:56.449957+0000","objects_scrubbed":0,"log_size":6836,"log_dups_size":0,"ondisk_log_size":6836,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:06:21.984124+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":394133504,"num_objects":6014,"num_object_clones":0,"num_object_copies":18042,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6014,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8836,"num_write_kb":475200,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,4],"acting":[2,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.b","version":"69'9","reported_seq":163,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-09-18T08:05:55.472519+0000","last_change":"2024-09-18T08:05:05.294031+0000","last_active":"2024-09-18T08:05:55.472519+0000","last_peered":"2024-09-18T08:05:55.472519+0000","last_clean":"2024-09-18T08:05:55.472519+0000","last_became_active":"2024-09-18T08:02:49.897998+0000","last_became_peered":"2024-09-18T08:02:49.897998+0000","last_unstale":"2024-09-18T08:05:55.472519+0000","last_undegraded":"2024-09-18T08:05:55.472519+0000","last_fullsized":"2024-09-18T08:05:55.472519+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'9","last_scrub_stamp":"2024-09-18T08:05:05.293936+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_clean_scrub_stamp":"2024-09-18T08:05:05.293936+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:06:11.067729+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":151,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-09-18T08:05:55.465741+0000","last_change":"2024-09-18T08:04:59.080170+0000","last_active":"2024-09-18T08:05:55.465741+0000","last_peered":"2024-09-18T08:05:55.465741+0000","last_clean":"2024-09-18T08:05:55.465741+0000","last_became_active":"2024-09-18T08:02:47.877161+0000","last_became_peered":"2024-09-18T08:02:47.877161+0000","last_unstale":"2024-09-18T08:05:55.465741+0000","last_undegraded":"2024-09-18T08:05:55.465741+0000","last_fullsized":"2024-09-18T08:05:55.465741+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:04:59.080115+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:46.860092+0000","last_clean_scrub_stamp":"2024-09-18T08:04:59.080115+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:06:27.394180+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":151,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-09-18T08:05:55.468118+0000","last_change":"2024-09-18T08:05:43.068256+0000","last_active":"2024-09-18T08:05:55.468118+0000","last_peered":"2024-09-18T08:05:55.468118+0000","last_clean":"2024-09-18T08:05:55.468118+0000","last_became_active":"2024-09-18T08:02:51.901330+0000","last_became_peered":"2024-09-18T08:02:51.901330+0000","last_unstale":"2024-09-18T08:05:55.468118+0000","last_undegraded":"2024-09-18T08:05:55.468118+0000","last_fullsized":"2024-09-18T08:05:55.468118+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:05:43.067057+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:50.886830+0000","last_clean_scrub_stamp":"2024-09-18T08:05:43.067057+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:07:10.458536+0000","scrub_duration":40,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,3],"acting":[2,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":140,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-09-18T08:05:55.473315+0000","last_change":"2024-09-18T08:05:09.251072+0000","last_active":"2024-09-18T08:05:55.473315+0000","last_peered":"2024-09-18T08:05:55.473315+0000","last_clean":"2024-09-18T08:05:55.473315+0000","last_became_active":"2024-09-18T08:02:53.911068+0000","last_became_peered":"2024-09-18T08:02:53.911068+0000","last_unstale":"2024-09-18T08:05:55.473315+0000","last_undegraded":"2024-09-18T08:05:55.473315+0000","last_fullsized":"2024-09-18T08:05:55.473315+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:05:09.250843+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_clean_scrub_stamp":"2024-09-18T08:05:09.250843+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:06:22.459347+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.9","version":"127'8767","reported_seq":8866,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-09-18T08:05:58.319861+0000","last_change":"2024-09-18T08:03:57.467481+0000","last_active":"2024-09-18T08:05:58.319861+0000","last_peered":"2024-09-18T08:05:58.319861+0000","last_clean":"2024-09-18T08:05:58.319861+0000","last_became_active":"2024-09-18T08:03:57.467065+0000","last_became_peered":"2024-09-18T08:03:57.467065+0000","last_unstale":"2024-09-18T08:05:58.319861+0000","last_undegraded":"2024-09-18T08:05:58.319861+0000","last_fullsized":"2024-09-18T08:05:58.319861+0000","mapping_epoch":83,"log_start":"98'3200","ondisk_log_start":"98'3200","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:03:56.449957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:03:56.449957+0000","last_clean_scrub_stamp":"2024-09-18T08:03:56.449957+0000","objects_scrubbed":0,"log_size":5567,"log_dups_size":0,"ondisk_log_size":5567,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:06:23.505530+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":392101888,"num_objects":5983,"num_object_clones":0,"num_object_copies":17949,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5983,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8767,"num_write_kb":472000,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.a","version":"69'19","reported_seq":114,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-09-18T08:05:55.469356+0000","last_change":"2024-09-18T08:04:43.303845+0000","last_active":"2024-09-18T08:05:55.469356+0000","last_peered":"2024-09-18T08:05:55.469356+0000","last_clean":"2024-09-18T08:05:55.469356+0000","last_became_active":"2024-09-18T08:04:43.303450+0000","last_became_peered":"2024-09-18T08:04:43.303450+0000","last_unstale":"2024-09-18T08:05:55.469356+0000","last_undegraded":"2024-09-18T08:05:55.469356+0000","last_fullsized":"2024-09-18T08:05:55.469356+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'19","last_scrub_stamp":"2024-09-18T08:04:20.391859+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_clean_scrub_stamp":"2024-09-18T08:04:20.391859+0000","objects_scrubbed":0,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:06:49.630107+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":31,"num_read_kb":20,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,7],"acting":[1,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":151,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-09-18T08:05:55.462601+0000","last_change":"2024-09-18T08:04:15.812374+0000","last_active":"2024-09-18T08:05:55.462601+0000","last_peered":"2024-09-18T08:05:55.462601+0000","last_clean":"2024-09-18T08:05:55.462601+0000","last_became_active":"2024-09-18T08:02:47.880645+0000","last_became_peered":"2024-09-18T08:02:47.880645+0000","last_unstale":"2024-09-18T08:05:55.462601+0000","last_undegraded":"2024-09-18T08:05:55.462601+0000","last_fullsized":"2024-09-18T08:05:55.462601+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:04:15.812306+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:46.860092+0000","last_clean_scrub_stamp":"2024-09-18T08:04:15.812306+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:06:48.868926+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":143,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-09-18T08:05:55.467935+0000","last_change":"2024-09-18T08:04:56.004261+0000","last_active":"2024-09-18T08:05:55.467935+0000","last_peered":"2024-09-18T08:05:55.467935+0000","last_clean":"2024-09-18T08:05:55.467935+0000","last_became_active":"2024-09-18T08:02:51.903439+0000","last_became_peered":"2024-09-18T08:02:51.903439+0000","last_unstale":"2024-09-18T08:05:55.467935+0000","last_undegraded":"2024-09-18T08:05:55.467935+0000","last_fullsized":"2024-09-18T08:05:55.467935+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:04:56.004150+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:50.886830+0000","last_clean_scrub_stamp":"2024-09-18T08:04:56.004150+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:05:57.194891+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":140,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-09-18T08:05:55.461810+0000","last_change":"2024-09-18T08:04:17.811662+0000","last_active":"2024-09-18T08:05:55.461810+0000","last_peered":"2024-09-18T08:05:55.461810+0000","last_clean":"2024-09-18T08:05:55.461810+0000","last_became_active":"2024-09-18T08:02:53.912762+0000","last_became_peered":"2024-09-18T08:02:53.912762+0000","last_unstale":"2024-09-18T08:05:55.461810+0000","last_undegraded":"2024-09-18T08:05:55.461810+0000","last_fullsized":"2024-09-18T08:05:55.461810+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:04:17.811597+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_clean_scrub_stamp":"2024-09-18T08:04:17.811597+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:06:52.934540+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.a","version":"127'8946","reported_seq":9043,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-09-18T08:05:58.347834+0000","last_change":"2024-09-18T08:03:57.466483+0000","last_active":"2024-09-18T08:05:58.347834+0000","last_peered":"2024-09-18T08:05:58.347834+0000","last_clean":"2024-09-18T08:05:58.347834+0000","last_became_active":"2024-09-18T08:03:57.466168+0000","last_became_peered":"2024-09-18T08:03:57.466168+0000","last_unstale":"2024-09-18T08:05:58.347834+0000","last_undegraded":"2024-09-18T08:05:58.347834+0000","last_fullsized":"2024-09-18T08:05:58.347834+0000","mapping_epoch":83,"log_start":"102'4300","ondisk_log_start":"102'4300","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:03:56.449957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:03:56.449957+0000","last_clean_scrub_stamp":"2024-09-18T08:03:56.449957+0000","objects_scrubbed":0,"log_size":4646,"log_dups_size":0,"ondisk_log_size":4646,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:06:21.099292+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":391380992,"num_objects":5972,"num_object_clones":0,"num_object_copies":17916,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5972,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8946,"num_write_kb":477376,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.9","version":"69'12","reported_seq":178,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-09-18T08:05:55.470360+0000","last_change":"2024-09-18T08:05:33.544197+0000","last_active":"2024-09-18T08:05:55.470360+0000","last_peered":"2024-09-18T08:05:55.470360+0000","last_clean":"2024-09-18T08:05:55.470360+0000","last_became_active":"2024-09-18T08:02:49.894762+0000","last_became_peered":"2024-09-18T08:02:49.894762+0000","last_unstale":"2024-09-18T08:05:55.470360+0000","last_undegraded":"2024-09-18T08:05:55.470360+0000","last_fullsized":"2024-09-18T08:05:55.470360+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'12","last_scrub_stamp":"2024-09-18T08:05:33.544135+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_clean_scrub_stamp":"2024-09-18T08:05:33.544135+0000","objects_scrubbed":5,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:06:52.810406+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":25,"num_read_kb":16,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":159,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-09-18T08:05:55.463084+0000","last_change":"2024-09-18T08:05:39.906123+0000","last_active":"2024-09-18T08:05:55.463084+0000","last_peered":"2024-09-18T08:05:55.463084+0000","last_clean":"2024-09-18T08:05:55.463084+0000","last_became_active":"2024-09-18T08:02:47.884052+0000","last_became_peered":"2024-09-18T08:02:47.884052+0000","last_unstale":"2024-09-18T08:05:55.463084+0000","last_undegraded":"2024-09-18T08:05:55.463084+0000","last_fullsized":"2024-09-18T08:05:55.463084+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:05:39.906062+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:46.860092+0000","last_clean_scrub_stamp":"2024-09-18T08:05:39.906062+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:07:03.409405+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":143,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-09-18T08:05:55.472141+0000","last_change":"2024-09-18T08:05:06.828638+0000","last_active":"2024-09-18T08:05:55.472141+0000","last_peered":"2024-09-18T08:05:55.472141+0000","last_clean":"2024-09-18T08:05:55.472141+0000","last_became_active":"2024-09-18T08:02:51.902163+0000","last_became_peered":"2024-09-18T08:02:51.902163+0000","last_unstale":"2024-09-18T08:05:55.472141+0000","last_undegraded":"2024-09-18T08:05:55.472141+0000","last_fullsized":"2024-09-18T08:05:55.472141+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:05:06.828408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:50.886830+0000","last_clean_scrub_stamp":"2024-09-18T08:05:06.828408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:06:17.532271+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":140,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-09-18T08:05:55.460024+0000","last_change":"2024-09-18T08:05:44.074994+0000","last_active":"2024-09-18T08:05:55.460024+0000","last_peered":"2024-09-18T08:05:55.460024+0000","last_clean":"2024-09-18T08:05:55.460024+0000","last_became_active":"2024-09-18T08:02:53.907412+0000","last_became_peered":"2024-09-18T08:02:53.907412+0000","last_unstale":"2024-09-18T08:05:55.460024+0000","last_undegraded":"2024-09-18T08:05:55.460024+0000","last_fullsized":"2024-09-18T08:05:55.460024+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:05:44.074555+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_clean_scrub_stamp":"2024-09-18T08:05:44.074555+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:07:10.508813+0000","scrub_duration":23,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.b","version":"127'8744","reported_seq":8920,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-09-18T08:05:58.133776+0000","last_change":"2024-09-18T08:05:28.826831+0000","last_active":"2024-09-18T08:05:58.133776+0000","last_peered":"2024-09-18T08:05:58.133776+0000","last_clean":"2024-09-18T08:05:58.133776+0000","last_became_active":"2024-09-18T08:03:57.466106+0000","last_became_peered":"2024-09-18T08:03:57.466106+0000","last_unstale":"2024-09-18T08:05:58.133776+0000","last_undegraded":"2024-09-18T08:05:58.133776+0000","last_fullsized":"2024-09-18T08:05:58.133776+0000","mapping_epoch":83,"log_start":"99'3700","ondisk_log_start":"99'3700","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"118'6870","last_scrub_stamp":"2024-09-18T08:05:28.826628+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:03:56.449957+0000","last_clean_scrub_stamp":"2024-09-18T08:05:28.826628+0000","objects_scrubbed":6831,"log_size":5044,"log_dups_size":0,"ondisk_log_size":5044,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:06:33.253487+0000","scrub_duration":1959,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":381419520,"num_objects":5820,"num_object_clones":0,"num_object_copies":17460,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5820,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8744,"num_write_kb":466048,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6],"acting":[1,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.8","version":"69'15","reported_seq":172,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-09-18T08:05:55.465382+0000","last_change":"2024-09-18T08:04:26.859424+0000","last_active":"2024-09-18T08:05:55.465382+0000","last_peered":"2024-09-18T08:05:55.465382+0000","last_clean":"2024-09-18T08:05:55.465382+0000","last_became_active":"2024-09-18T08:02:49.888799+0000","last_became_peered":"2024-09-18T08:02:49.888799+0000","last_unstale":"2024-09-18T08:05:55.465382+0000","last_undegraded":"2024-09-18T08:05:55.465382+0000","last_fullsized":"2024-09-18T08:05:55.465382+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'15","last_scrub_stamp":"2024-09-18T08:04:26.859044+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_clean_scrub_stamp":"2024-09-18T08:04:26.859044+0000","objects_scrubbed":7,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:06:52.740633+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.f","version":"62'2","reported_seq":171,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-09-18T08:05:55.463977+0000","last_change":"2024-09-18T08:05:11.906358+0000","last_active":"2024-09-18T08:05:55.463977+0000","last_peered":"2024-09-18T08:05:55.463977+0000","last_clean":"2024-09-18T08:05:55.463977+0000","last_became_active":"2024-09-18T08:02:47.881165+0000","last_became_peered":"2024-09-18T08:02:47.881165+0000","last_unstale":"2024-09-18T08:05:55.463977+0000","last_undegraded":"2024-09-18T08:05:55.463977+0000","last_fullsized":"2024-09-18T08:05:55.463977+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"62'2","last_scrub_stamp":"2024-09-18T08:05:11.906308+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:46.860092+0000","last_clean_scrub_stamp":"2024-09-18T08:05:11.906308+0000","objects_scrubbed":2,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:06:20.720016+0000","scrub_duration":24,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":4,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.9","version":"69'8","reported_seq":194,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-09-18T08:05:57.966411+0000","last_change":"2024-09-18T08:03:52.572824+0000","last_active":"2024-09-18T08:05:57.966411+0000","last_peered":"2024-09-18T08:05:57.966411+0000","last_clean":"2024-09-18T08:05:57.966411+0000","last_became_active":"2024-09-18T08:02:51.905059+0000","last_became_peered":"2024-09-18T08:02:51.905059+0000","last_unstale":"2024-09-18T08:05:57.966411+0000","last_undegraded":"2024-09-18T08:05:57.966411+0000","last_fullsized":"2024-09-18T08:05:57.966411+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'8","last_scrub_stamp":"2024-09-18T08:03:52.572757+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:50.886830+0000","last_clean_scrub_stamp":"2024-09-18T08:03:52.572757+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:06:15.853616+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":139,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-09-18T08:05:55.465382+0000","last_change":"2024-09-18T08:05:07.879565+0000","last_active":"2024-09-18T08:05:55.465382+0000","last_peered":"2024-09-18T08:05:55.465382+0000","last_clean":"2024-09-18T08:05:55.465382+0000","last_became_active":"2024-09-18T08:02:54.045845+0000","last_became_peered":"2024-09-18T08:02:54.045845+0000","last_unstale":"2024-09-18T08:05:55.465382+0000","last_undegraded":"2024-09-18T08:05:55.465382+0000","last_fullsized":"2024-09-18T08:05:55.465382+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:05:07.879501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_clean_scrub_stamp":"2024-09-18T08:05:07.879501+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:06:17.801897+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":63,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-09-18T08:05:55.472070+0000","last_change":"2024-09-18T08:04:43.302438+0000","last_active":"2024-09-18T08:05:55.472070+0000","last_peered":"2024-09-18T08:05:55.472070+0000","last_clean":"2024-09-18T08:05:55.472070+0000","last_became_active":"2024-09-18T08:04:43.302022+0000","last_became_peered":"2024-09-18T08:04:43.302022+0000","last_unstale":"2024-09-18T08:05:55.472070+0000","last_undegraded":"2024-09-18T08:05:55.472070+0000","last_fullsized":"2024-09-18T08:05:55.472070+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:04:16.378205+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:46.860092+0000","last_clean_scrub_stamp":"2024-09-18T08:04:16.378205+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.17","version":"69'6","reported_seq":164,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-09-18T08:05:55.461093+0000","last_change":"2024-09-18T08:05:26.228344+0000","last_active":"2024-09-18T08:05:55.461093+0000","last_peered":"2024-09-18T08:05:55.461093+0000","last_clean":"2024-09-18T08:05:55.461093+0000","last_became_active":"2024-09-18T08:02:49.895067+0000","last_became_peered":"2024-09-18T08:02:49.895067+0000","last_unstale":"2024-09-18T08:05:55.461093+0000","last_undegraded":"2024-09-18T08:05:55.461093+0000","last_fullsized":"2024-09-18T08:05:55.461093+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'6","last_scrub_stamp":"2024-09-18T08:05:26.227814+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_clean_scrub_stamp":"2024-09-18T08:05:26.227814+0000","objects_scrubbed":3,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:06:36.064005+0000","scrub_duration":18,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":9,"num_read_kb":6,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":151,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-09-18T08:05:55.466108+0000","last_change":"2024-09-18T08:05:45.812436+0000","last_active":"2024-09-18T08:05:55.466108+0000","last_peered":"2024-09-18T08:05:55.466108+0000","last_clean":"2024-09-18T08:05:55.466108+0000","last_became_active":"2024-09-18T08:02:51.912557+0000","last_became_peered":"2024-09-18T08:02:51.912557+0000","last_unstale":"2024-09-18T08:05:55.466108+0000","last_undegraded":"2024-09-18T08:05:55.466108+0000","last_fullsized":"2024-09-18T08:05:55.466108+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:05:45.812389+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:50.886830+0000","last_clean_scrub_stamp":"2024-09-18T08:05:45.812389+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:07:01.523740+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,1],"acting":[5,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":140,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-09-18T08:05:55.462217+0000","last_change":"2024-09-18T08:05:00.883597+0000","last_active":"2024-09-18T08:05:55.462217+0000","last_peered":"2024-09-18T08:05:55.462217+0000","last_clean":"2024-09-18T08:05:55.462217+0000","last_became_active":"2024-09-18T08:02:54.048959+0000","last_became_peered":"2024-09-18T08:02:54.048959+0000","last_unstale":"2024-09-18T08:05:55.462217+0000","last_undegraded":"2024-09-18T08:05:55.462217+0000","last_fullsized":"2024-09-18T08:05:55.462217+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:05:00.883555+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_clean_scrub_stamp":"2024-09-18T08:05:00.883555+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:06:25.986166+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"69'9","reported_seq":163,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-09-18T08:05:55.472373+0000","last_change":"2024-09-18T08:05:00.265939+0000","last_active":"2024-09-18T08:05:55.472373+0000","last_peered":"2024-09-18T08:05:55.472373+0000","last_clean":"2024-09-18T08:05:55.472373+0000","last_became_active":"2024-09-18T08:02:49.891567+0000","last_became_peered":"2024-09-18T08:02:49.891567+0000","last_unstale":"2024-09-18T08:05:55.472373+0000","last_undegraded":"2024-09-18T08:05:55.472373+0000","last_fullsized":"2024-09-18T08:05:55.472373+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'9","last_scrub_stamp":"2024-09-18T08:05:00.265738+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_clean_scrub_stamp":"2024-09-18T08:05:00.265738+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:06:27.928037+0000","scrub_duration":17,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":151,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-09-18T08:05:55.461870+0000","last_change":"2024-09-18T08:03:49.591148+0000","last_active":"2024-09-18T08:05:55.461870+0000","last_peered":"2024-09-18T08:05:55.461870+0000","last_clean":"2024-09-18T08:05:55.461870+0000","last_became_active":"2024-09-18T08:02:47.885790+0000","last_became_peered":"2024-09-18T08:02:47.885790+0000","last_unstale":"2024-09-18T08:05:55.461870+0000","last_undegraded":"2024-09-18T08:05:55.461870+0000","last_fullsized":"2024-09-18T08:05:55.461870+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:03:49.591068+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:46.860092+0000","last_clean_scrub_stamp":"2024-09-18T08:03:49.591068+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:06:17.821808+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":143,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-09-18T08:05:55.460981+0000","last_change":"2024-09-18T08:05:42.061649+0000","last_active":"2024-09-18T08:05:55.460981+0000","last_peered":"2024-09-18T08:05:55.460981+0000","last_clean":"2024-09-18T08:05:55.460981+0000","last_became_active":"2024-09-18T08:02:51.907361+0000","last_became_peered":"2024-09-18T08:02:51.907361+0000","last_unstale":"2024-09-18T08:05:55.460981+0000","last_undegraded":"2024-09-18T08:05:55.460981+0000","last_fullsized":"2024-09-18T08:05:55.460981+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:05:42.061452+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:50.886830+0000","last_clean_scrub_stamp":"2024-09-18T08:05:42.061452+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:07:05.139088+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":140,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-09-18T08:05:55.467941+0000","last_change":"2024-09-18T08:05:07.780899+0000","last_active":"2024-09-18T08:05:55.467941+0000","last_peered":"2024-09-18T08:05:55.467941+0000","last_clean":"2024-09-18T08:05:55.467941+0000","last_became_active":"2024-09-18T08:02:53.908917+0000","last_became_peered":"2024-09-18T08:02:53.908917+0000","last_unstale":"2024-09-18T08:05:55.467941+0000","last_undegraded":"2024-09-18T08:05:55.467941+0000","last_fullsized":"2024-09-18T08:05:55.467941+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:05:07.780816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_clean_scrub_stamp":"2024-09-18T08:05:07.780816+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:06:31.045662+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.15","version":"69'9","reported_seq":171,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-09-18T08:05:55.465864+0000","last_change":"2024-09-18T08:05:36.896299+0000","last_active":"2024-09-18T08:05:55.465864+0000","last_peered":"2024-09-18T08:05:55.465864+0000","last_clean":"2024-09-18T08:05:55.465864+0000","last_became_active":"2024-09-18T08:02:49.890497+0000","last_became_peered":"2024-09-18T08:02:49.890497+0000","last_unstale":"2024-09-18T08:05:55.465864+0000","last_undegraded":"2024-09-18T08:05:55.465864+0000","last_fullsized":"2024-09-18T08:05:55.465864+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'9","last_scrub_stamp":"2024-09-18T08:05:36.896238+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_clean_scrub_stamp":"2024-09-18T08:05:36.896238+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:06:52.292485+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":151,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-09-18T08:05:55.472518+0000","last_change":"2024-09-18T08:04:23.160350+0000","last_active":"2024-09-18T08:05:55.472518+0000","last_peered":"2024-09-18T08:05:55.472518+0000","last_clean":"2024-09-18T08:05:55.472518+0000","last_became_active":"2024-09-18T08:02:47.875983+0000","last_became_peered":"2024-09-18T08:02:47.875983+0000","last_unstale":"2024-09-18T08:05:55.472518+0000","last_undegraded":"2024-09-18T08:05:55.472518+0000","last_fullsized":"2024-09-18T08:05:55.472518+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:04:23.160277+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:46.860092+0000","last_clean_scrub_stamp":"2024-09-18T08:04:23.160277+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.14","version":"69'8","reported_seq":190,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-09-18T08:05:55.459971+0000","last_change":"2024-09-18T08:05:01.083803+0000","last_active":"2024-09-18T08:05:55.459971+0000","last_peered":"2024-09-18T08:05:55.459971+0000","last_clean":"2024-09-18T08:05:55.459971+0000","last_became_active":"2024-09-18T08:02:51.909513+0000","last_became_peered":"2024-09-18T08:02:51.909513+0000","last_unstale":"2024-09-18T08:05:55.459971+0000","last_undegraded":"2024-09-18T08:05:55.459971+0000","last_fullsized":"2024-09-18T08:05:55.459971+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'8","last_scrub_stamp":"2024-09-18T08:05:01.082570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:50.886830+0000","last_clean_scrub_stamp":"2024-09-18T08:05:01.082570+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:06:22.559429+0000","scrub_duration":17,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":140,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-09-18T08:05:55.470720+0000","last_change":"2024-09-18T08:05:09.654918+0000","last_active":"2024-09-18T08:05:55.470720+0000","last_peered":"2024-09-18T08:05:55.470720+0000","last_clean":"2024-09-18T08:05:55.470720+0000","last_became_active":"2024-09-18T08:02:53.907428+0000","last_became_peered":"2024-09-18T08:02:53.907428+0000","last_unstale":"2024-09-18T08:05:55.470720+0000","last_undegraded":"2024-09-18T08:05:55.470720+0000","last_fullsized":"2024-09-18T08:05:55.470720+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:05:09.654832+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_clean_scrub_stamp":"2024-09-18T08:05:09.654832+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:06:19.204016+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.14","version":"69'10","reported_seq":181,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-09-18T08:05:55.460193+0000","last_change":"2024-09-18T08:04:18.172364+0000","last_active":"2024-09-18T08:05:55.460193+0000","last_peered":"2024-09-18T08:05:55.460193+0000","last_clean":"2024-09-18T08:05:55.460193+0000","last_became_active":"2024-09-18T08:04:18.150968+0000","last_became_peered":"2024-09-18T08:04:18.150968+0000","last_unstale":"2024-09-18T08:05:55.460193+0000","last_undegraded":"2024-09-18T08:05:55.460193+0000","last_fullsized":"2024-09-18T08:05:55.460193+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'10","last_scrub_stamp":"2024-09-18T08:04:16.940778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_clean_scrub_stamp":"2024-09-18T08:04:16.940778+0000","objects_scrubbed":5,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:06:54.101039+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,7],"acting":[3,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":151,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-09-18T08:05:55.462538+0000","last_change":"2024-09-18T08:05:08.855132+0000","last_active":"2024-09-18T08:05:55.462538+0000","last_peered":"2024-09-18T08:05:55.462538+0000","last_clean":"2024-09-18T08:05:55.462538+0000","last_became_active":"2024-09-18T08:02:47.882065+0000","last_became_peered":"2024-09-18T08:02:47.882065+0000","last_unstale":"2024-09-18T08:05:55.462538+0000","last_undegraded":"2024-09-18T08:05:55.462538+0000","last_fullsized":"2024-09-18T08:05:55.462538+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:05:08.855064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:46.860092+0000","last_clean_scrub_stamp":"2024-09-18T08:05:08.855064+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:06:28.592575+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.15","version":"69'8","reported_seq":199,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-09-18T08:05:57.966329+0000","last_change":"2024-09-18T08:05:27.998896+0000","last_active":"2024-09-18T08:05:57.966329+0000","last_peered":"2024-09-18T08:05:57.966329+0000","last_clean":"2024-09-18T08:05:57.966329+0000","last_became_active":"2024-09-18T08:02:51.914135+0000","last_became_peered":"2024-09-18T08:02:51.914135+0000","last_unstale":"2024-09-18T08:05:57.966329+0000","last_undegraded":"2024-09-18T08:05:57.966329+0000","last_fullsized":"2024-09-18T08:05:57.966329+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'8","last_scrub_stamp":"2024-09-18T08:05:27.998833+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:50.886830+0000","last_clean_scrub_stamp":"2024-09-18T08:05:27.998833+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:06:41.603719+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":140,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-09-18T08:05:55.472055+0000","last_change":"2024-09-18T08:05:03.302755+0000","last_active":"2024-09-18T08:05:55.472055+0000","last_peered":"2024-09-18T08:05:55.472055+0000","last_clean":"2024-09-18T08:05:55.472055+0000","last_became_active":"2024-09-18T08:02:53.912112+0000","last_became_peered":"2024-09-18T08:02:53.912112+0000","last_unstale":"2024-09-18T08:05:55.472055+0000","last_undegraded":"2024-09-18T08:05:55.472055+0000","last_fullsized":"2024-09-18T08:05:55.472055+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:05:03.302444+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_clean_scrub_stamp":"2024-09-18T08:05:03.302444+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:06:25.825825+0000","scrub_duration":16,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.13","version":"69'11","reported_seq":166,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-09-18T08:05:55.471498+0000","last_change":"2024-09-18T08:04:34.451463+0000","last_active":"2024-09-18T08:05:55.471498+0000","last_peered":"2024-09-18T08:05:55.471498+0000","last_clean":"2024-09-18T08:05:55.471498+0000","last_became_active":"2024-09-18T08:02:49.896949+0000","last_became_peered":"2024-09-18T08:02:49.896949+0000","last_unstale":"2024-09-18T08:05:55.471498+0000","last_undegraded":"2024-09-18T08:05:55.471498+0000","last_fullsized":"2024-09-18T08:05:55.471498+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'11","last_scrub_stamp":"2024-09-18T08:04:34.451379+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_clean_scrub_stamp":"2024-09-18T08:04:34.451379+0000","objects_scrubbed":5,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:06:00.685349+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":151,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-09-18T08:05:55.470714+0000","last_change":"2024-09-18T08:05:00.588793+0000","last_active":"2024-09-18T08:05:55.470714+0000","last_peered":"2024-09-18T08:05:55.470714+0000","last_clean":"2024-09-18T08:05:55.470714+0000","last_became_active":"2024-09-18T08:02:47.879771+0000","last_became_peered":"2024-09-18T08:02:47.879771+0000","last_unstale":"2024-09-18T08:05:55.470714+0000","last_undegraded":"2024-09-18T08:05:55.470714+0000","last_fullsized":"2024-09-18T08:05:55.470714+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:05:00.588728+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:46.860092+0000","last_clean_scrub_stamp":"2024-09-18T08:05:00.588728+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:06:15.909247+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":151,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-09-18T08:05:55.469273+0000","last_change":"2024-09-18T08:05:29.896334+0000","last_active":"2024-09-18T08:05:55.469273+0000","last_peered":"2024-09-18T08:05:55.469273+0000","last_clean":"2024-09-18T08:05:55.469273+0000","last_became_active":"2024-09-18T08:02:51.910752+0000","last_became_peered":"2024-09-18T08:02:51.910752+0000","last_unstale":"2024-09-18T08:05:55.469273+0000","last_undegraded":"2024-09-18T08:05:55.469273+0000","last_fullsized":"2024-09-18T08:05:55.469273+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:05:29.888927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:50.886830+0000","last_clean_scrub_stamp":"2024-09-18T08:05:29.888927+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:06:47.803144+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":140,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-09-18T08:05:55.459824+0000","last_change":"2024-09-18T08:04:18.998630+0000","last_active":"2024-09-18T08:05:55.459824+0000","last_peered":"2024-09-18T08:05:55.459824+0000","last_clean":"2024-09-18T08:05:55.459824+0000","last_became_active":"2024-09-18T08:02:53.911429+0000","last_became_peered":"2024-09-18T08:02:53.911429+0000","last_unstale":"2024-09-18T08:05:55.459824+0000","last_undegraded":"2024-09-18T08:05:55.459824+0000","last_fullsized":"2024-09-18T08:05:55.459824+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:04:18.998565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_clean_scrub_stamp":"2024-09-18T08:04:18.998565+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":84,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.12","version":"69'9","reported_seq":155,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-09-18T08:05:55.469462+0000","last_change":"2024-09-18T08:02:49.893472+0000","last_active":"2024-09-18T08:05:55.469462+0000","last_peered":"2024-09-18T08:05:55.469462+0000","last_clean":"2024-09-18T08:05:55.469462+0000","last_became_active":"2024-09-18T08:02:49.885950+0000","last_became_peered":"2024-09-18T08:02:49.885950+0000","last_unstale":"2024-09-18T08:05:55.469462+0000","last_undegraded":"2024-09-18T08:05:55.469462+0000","last_fullsized":"2024-09-18T08:05:55.469462+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_clean_scrub_stamp":"2024-09-18T08:02:48.870718+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:06:15.746930+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":159,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-09-18T08:05:55.462478+0000","last_change":"2024-09-18T08:05:30.820244+0000","last_active":"2024-09-18T08:05:55.462478+0000","last_peered":"2024-09-18T08:05:55.462478+0000","last_clean":"2024-09-18T08:05:55.462478+0000","last_became_active":"2024-09-18T08:02:47.882096+0000","last_became_peered":"2024-09-18T08:02:47.882096+0000","last_unstale":"2024-09-18T08:05:55.462478+0000","last_undegraded":"2024-09-18T08:05:55.462478+0000","last_fullsized":"2024-09-18T08:05:55.462478+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:05:30.820200+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:46.860092+0000","last_clean_scrub_stamp":"2024-09-18T08:05:30.820200+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:06:54.287921+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":143,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-09-18T08:05:55.460157+0000","last_change":"2024-09-18T08:05:43.165522+0000","last_active":"2024-09-18T08:05:55.460157+0000","last_peered":"2024-09-18T08:05:55.460157+0000","last_clean":"2024-09-18T08:05:55.460157+0000","last_became_active":"2024-09-18T08:02:51.906277+0000","last_became_peered":"2024-09-18T08:02:51.906277+0000","last_unstale":"2024-09-18T08:05:55.460157+0000","last_undegraded":"2024-09-18T08:05:55.460157+0000","last_fullsized":"2024-09-18T08:05:55.460157+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:05:43.165258+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:50.886830+0000","last_clean_scrub_stamp":"2024-09-18T08:05:43.165258+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:07:08.715609+0000","scrub_duration":89,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.10","version":"69'1","reported_seq":136,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-09-18T08:05:55.471653+0000","last_change":"2024-09-18T08:05:51.429631+0000","last_active":"2024-09-18T08:05:55.471653+0000","last_peered":"2024-09-18T08:05:55.471653+0000","last_clean":"2024-09-18T08:05:55.471653+0000","last_became_active":"2024-09-18T08:02:53.912699+0000","last_became_peered":"2024-09-18T08:02:53.912699+0000","last_unstale":"2024-09-18T08:05:55.471653+0000","last_undegraded":"2024-09-18T08:05:55.471653+0000","last_fullsized":"2024-09-18T08:05:55.471653+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_clean_scrub_stamp":"2024-09-18T08:02:52.890475+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:06:17.399772+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.11","version":"69'11","reported_seq":174,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-09-18T08:05:55.459976+0000","last_change":"2024-09-18T08:05:13.037686+0000","last_active":"2024-09-18T08:05:55.459976+0000","last_peered":"2024-09-18T08:05:55.459976+0000","last_clean":"2024-09-18T08:05:55.459976+0000","last_became_active":"2024-09-18T08:02:49.889611+0000","last_became_peered":"2024-09-18T08:02:49.889611+0000","last_unstale":"2024-09-18T08:05:55.459976+0000","last_undegraded":"2024-09-18T08:05:55.459976+0000","last_fullsized":"2024-09-18T08:05:55.459976+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'11","last_scrub_stamp":"2024-09-18T08:05:13.037615+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_clean_scrub_stamp":"2024-09-18T08:05:13.037615+0000","objects_scrubbed":5,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:06:30.232726+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":151,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-09-18T08:05:55.466158+0000","last_change":"2024-09-18T08:03:49.960690+0000","last_active":"2024-09-18T08:05:55.466158+0000","last_peered":"2024-09-18T08:05:55.466158+0000","last_clean":"2024-09-18T08:05:55.466158+0000","last_became_active":"2024-09-18T08:02:47.878886+0000","last_became_peered":"2024-09-18T08:02:47.878886+0000","last_unstale":"2024-09-18T08:05:55.466158+0000","last_undegraded":"2024-09-18T08:05:55.466158+0000","last_fullsized":"2024-09-18T08:05:55.466158+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:03:49.960617+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:46.860092+0000","last_clean_scrub_stamp":"2024-09-18T08:03:49.960617+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:06:13.978672+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":143,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-09-18T08:05:55.461521+0000","last_change":"2024-09-18T08:04:12.846215+0000","last_active":"2024-09-18T08:05:55.461521+0000","last_peered":"2024-09-18T08:05:55.461521+0000","last_clean":"2024-09-18T08:05:55.461521+0000","last_became_active":"2024-09-18T08:02:51.907762+0000","last_became_peered":"2024-09-18T08:02:51.907762+0000","last_unstale":"2024-09-18T08:05:55.461521+0000","last_undegraded":"2024-09-18T08:05:55.461521+0000","last_fullsized":"2024-09-18T08:05:55.461521+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:04:12.846141+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:50.886830+0000","last_clean_scrub_stamp":"2024-09-18T08:04:12.846141+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:06:18.783707+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":131,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-09-18T08:05:55.460313+0000","last_change":"2024-09-18T08:02:54.047687+0000","last_active":"2024-09-18T08:05:55.460313+0000","last_peered":"2024-09-18T08:05:55.460313+0000","last_clean":"2024-09-18T08:05:55.460313+0000","last_became_active":"2024-09-18T08:02:54.047181+0000","last_became_peered":"2024-09-18T08:02:54.047181+0000","last_unstale":"2024-09-18T08:05:55.460313+0000","last_undegraded":"2024-09-18T08:05:55.460313+0000","last_fullsized":"2024-09-18T08:05:55.460313+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_clean_scrub_stamp":"2024-09-18T08:02:52.890475+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:06:49.067042+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.10","version":"69'4","reported_seq":153,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-09-18T08:05:55.466092+0000","last_change":"2024-09-18T08:04:13.000124+0000","last_active":"2024-09-18T08:05:55.466092+0000","last_peered":"2024-09-18T08:05:55.466092+0000","last_clean":"2024-09-18T08:05:55.466092+0000","last_became_active":"2024-09-18T08:02:49.890126+0000","last_became_peered":"2024-09-18T08:02:49.890126+0000","last_unstale":"2024-09-18T08:05:55.466092+0000","last_undegraded":"2024-09-18T08:05:55.466092+0000","last_fullsized":"2024-09-18T08:05:55.466092+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'4","last_scrub_stamp":"2024-09-18T08:04:12.999978+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_clean_scrub_stamp":"2024-09-18T08:04:12.999978+0000","objects_scrubbed":2,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:06:53.064183+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":6,"num_read_kb":4,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":151,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-09-18T08:05:55.473026+0000","last_change":"2024-09-18T08:04:10.234061+0000","last_active":"2024-09-18T08:05:55.473026+0000","last_peered":"2024-09-18T08:05:55.473026+0000","last_clean":"2024-09-18T08:05:55.473026+0000","last_became_active":"2024-09-18T08:02:47.877616+0000","last_became_peered":"2024-09-18T08:02:47.877616+0000","last_unstale":"2024-09-18T08:05:55.473026+0000","last_undegraded":"2024-09-18T08:05:55.473026+0000","last_fullsized":"2024-09-18T08:05:55.473026+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:04:10.233969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:46.860092+0000","last_clean_scrub_stamp":"2024-09-18T08:04:10.233969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":71,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-09-18T08:05:55.470116+0000","last_change":"2024-09-18T08:05:11.591350+0000","last_active":"2024-09-18T08:05:55.470116+0000","last_peered":"2024-09-18T08:05:55.470116+0000","last_clean":"2024-09-18T08:05:55.470116+0000","last_became_active":"2024-09-18T08:04:43.299415+0000","last_became_peered":"2024-09-18T08:04:43.299415+0000","last_unstale":"2024-09-18T08:05:55.470116+0000","last_undegraded":"2024-09-18T08:05:55.470116+0000","last_fullsized":"2024-09-18T08:05:55.470116+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:05:11.591275+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:50.886830+0000","last_clean_scrub_stamp":"2024-09-18T08:05:11.591275+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:06:32.194148+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,7],"acting":[4,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":148,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-09-18T08:05:55.461743+0000","last_change":"2024-09-18T08:05:32.798628+0000","last_active":"2024-09-18T08:05:55.461743+0000","last_peered":"2024-09-18T08:05:55.461743+0000","last_clean":"2024-09-18T08:05:55.461743+0000","last_became_active":"2024-09-18T08:02:53.909146+0000","last_became_peered":"2024-09-18T08:02:53.909146+0000","last_unstale":"2024-09-18T08:05:55.461743+0000","last_undegraded":"2024-09-18T08:05:55.461743+0000","last_fullsized":"2024-09-18T08:05:55.461743+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:05:32.798564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_clean_scrub_stamp":"2024-09-18T08:05:32.798564+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:06:38.927572+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,4],"acting":[7,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":132,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-09-18T08:05:55.470738+0000","last_change":"2024-09-18T08:02:53.907055+0000","last_active":"2024-09-18T08:05:55.470738+0000","last_peered":"2024-09-18T08:05:55.470738+0000","last_clean":"2024-09-18T08:05:55.470738+0000","last_became_active":"2024-09-18T08:02:53.906726+0000","last_became_peered":"2024-09-18T08:02:53.906726+0000","last_unstale":"2024-09-18T08:05:55.470738+0000","last_undegraded":"2024-09-18T08:05:55.470738+0000","last_fullsized":"2024-09-18T08:05:55.470738+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_clean_scrub_stamp":"2024-09-18T08:02:52.890475+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:06:19.752175+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":151,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-09-18T08:05:55.466319+0000","last_change":"2024-09-18T08:05:02.063591+0000","last_active":"2024-09-18T08:05:55.466319+0000","last_peered":"2024-09-18T08:05:55.466319+0000","last_clean":"2024-09-18T08:05:55.466319+0000","last_became_active":"2024-09-18T08:02:47.878922+0000","last_became_peered":"2024-09-18T08:02:47.878922+0000","last_unstale":"2024-09-18T08:05:55.466319+0000","last_undegraded":"2024-09-18T08:05:55.466319+0000","last_fullsized":"2024-09-18T08:05:55.466319+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:05:02.063478+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:46.860092+0000","last_clean_scrub_stamp":"2024-09-18T08:05:02.063478+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:06:29.652021+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"69'11","reported_seq":102,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-09-18T08:05:55.466212+0000","last_change":"2024-09-18T08:04:43.304006+0000","last_active":"2024-09-18T08:05:55.466212+0000","last_peered":"2024-09-18T08:05:55.466212+0000","last_clean":"2024-09-18T08:05:55.466212+0000","last_became_active":"2024-09-18T08:04:43.303640+0000","last_became_peered":"2024-09-18T08:04:43.303640+0000","last_unstale":"2024-09-18T08:05:55.466212+0000","last_undegraded":"2024-09-18T08:05:55.466212+0000","last_fullsized":"2024-09-18T08:05:55.466212+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_clean_scrub_stamp":"2024-09-18T08:02:48.870718+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:06:47.748355+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":19,"num_read_kb":12,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,1],"acting":[5,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":143,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-09-18T08:05:55.472251+0000","last_change":"2024-09-18T08:04:26.117117+0000","last_active":"2024-09-18T08:05:55.472251+0000","last_peered":"2024-09-18T08:05:55.472251+0000","last_clean":"2024-09-18T08:05:55.472251+0000","last_became_active":"2024-09-18T08:02:51.900300+0000","last_became_peered":"2024-09-18T08:02:51.900300+0000","last_unstale":"2024-09-18T08:05:55.472251+0000","last_undegraded":"2024-09-18T08:05:55.472251+0000","last_fullsized":"2024-09-18T08:05:55.472251+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:04:26.117051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:50.886830+0000","last_clean_scrub_stamp":"2024-09-18T08:04:26.117051+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":7,"num_pg":16,"stat_sum":{"num_bytes":6192234520,"num_objects":94487,"num_object_clones":0,"num_object_copies":283461,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":94487,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":142091,"num_write_kb":7570433,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":18369687552,"data_stored":18369675336,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":86391,"ondisk_log_size":86391,"up":48,"acting":48,"num_store_stats":8},{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":416,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":65,"ondisk_log_size":65,"up":96,"acting":96,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":178,"num_object_clones":0,"num_object_copies":534,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":178,"num_whiteouts":0,"num_read":694,"num_read_kb":451,"num_write":417,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":17,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":417792,"data_stored":11106,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":394,"ondisk_log_size":394,"up":96,"acting":96,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":1613,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":24,"num_read_kb":24,"num_write":10,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":4839,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":6,"ondisk_log_size":6,"up":96,"acting":96,"num_store_stats":8},{"poolid":2,"num_pg":3,"stat_sum":{"num_bytes":408,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":190,"num_read_kb":185,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":8,"ondisk_log_size":8,"up":9,"acting":9,"num_store_stats":7},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":231,"num_write":113,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":590368,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":79,"ondisk_log_size":79,"up":3,"acting":3,"num_store_stats":3}],"osd_stats":[{"osd":7,"up_from":56,"seq":240518168619,"num_pgs":58,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1885224,"kb_used_data":1822104,"kb_used_omap":1,"kb_used_meta":63102,"kb_avail":91859928,"statfs":{"total":95995035648,"available":94064566272,"internally_reserved":0,"allocated":1865834496,"data_stored":1865432422,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":64616906},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1],"upper_bound":8},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":47,"seq":201863462961,"num_pgs":48,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1877784,"kb_used_data":1814536,"kb_used_omap":1,"kb_used_meta":63102,"kb_avail":91867368,"statfs":{"total":95995035648,"available":94072184832,"internally_reserved":0,"allocated":1858084864,"data_stored":1857698113,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":64616902},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1],"upper_bound":2},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":39,"seq":167503724599,"num_pgs":52,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1499424,"kb_used_data":1436304,"kb_used_omap":1,"kb_used_meta":63102,"kb_avail":92245728,"statfs":{"total":95995035648,"available":94459625472,"internally_reserved":0,"allocated":1470775296,"data_stored":1470380703,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":64616902},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1],"upper_bound":4},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":32,"seq":137438953532,"num_pgs":63,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1890928,"kb_used_data":1827808,"kb_used_omap":1,"kb_used_meta":63102,"kb_avail":91854224,"statfs":{"total":95995035648,"available":94058725376,"internally_reserved":0,"allocated":1871675392,"data_stored":1871265095,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":64616901},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":26,"seq":111669149762,"num_pgs":65,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":3561652,"kb_used_data":3449956,"kb_used_omap":1,"kb_used_meta":111678,"kb_avail":90183500,"statfs":{"total":95995035648,"available":92347904000,"internally_reserved":0,"allocated":3532754944,"data_stored":3532339844,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":114358728},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411400,"num_pgs":44,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2388576,"kb_used_data":2299084,"kb_used_omap":1,"kb_used_meta":89470,"kb_avail":91356576,"statfs":{"total":95995035648,"available":93549133824,"internally_reserved":0,"allocated":2354262016,"data_stored":2353871515,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":91617733},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574924,"num_pgs":60,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2384532,"kb_used_data":2299200,"kb_used_omap":1,"kb_used_meta":85310,"kb_avail":91360620,"statfs":{"total":95995035648,"available":93553274880,"internally_reserved":0,"allocated":2354380800,"data_stored":2353938948,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":87357893},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738451,"num_pgs":54,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":3188028,"kb_used_data":3076328,"kb_used_omap":1,"kb_used_meta":111678,"kb_avail":90557124,"statfs":{"total":95995035648,"available":92730494976,"internally_reserved":0,"allocated":3150159872,"data_stored":3149742153,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":114358725},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1131,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1521,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":436,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":46,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1177,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":436,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":1898,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1128,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":1898,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1474,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":1100,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1034,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1254,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":403,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":416,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":403,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":3123318784,"data_stored":3123314712,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":2337280000,"data_stored":2337275928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":2347958272,"data_stored":2347958272,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":3525115904,"data_stored":3525115904,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":1856176128,"data_stored":1856176128,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":1467154432,"data_stored":1467154432,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":1851002880,"data_stored":1850998808,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":1861681152,"data_stored":1861681152,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T08:06:07.311 INFO:tasks.cephadm.ceph_manager.ceph:recovered! 2024-09-18T08:06:07.311 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-18T08:06:07.312 INFO:tasks.thrashosds.thrasher:Removing osd 6, in_osds are: [0, 1, 2, 3, 4, 5, 6, 7] 2024-09-18T08:06:07.312 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid 779349ca-7593-11ef-bceb-c7b262605968 -- ceph osd out 6 2024-09-18T08:06:07.349 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/779349ca-7593-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T08:06:07.367 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4 2024-09-18T08:06:07.377 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/779349ca-7593-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T08:06:07.388 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/779349ca-7593-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T08:06:07.391 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:06:07.392 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:06:07.392 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4' 2024-09-18T08:06:07.493 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3 2024-09-18T08:06:07.560 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:06:07.561 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:06:07.561 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3' 2024-09-18T08:06:07.662 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1 2024-09-18T08:06:07.722 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:06:07.723 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:06:07.723 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1' 2024-09-18T08:06:07.824 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5 2024-09-18T08:06:07.852 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:06:07.853 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:06:07.853 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5' 2024-09-18T08:06:07.954 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6 2024-09-18T08:06:07.980 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:06:07.981 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:06:07.981 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6' 2024-09-18T08:06:08.082 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0 2024-09-18T08:06:08.084 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/779349ca-7593-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T08:06:08.135 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:06:08.136 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:06:08.137 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0' 2024-09-18T08:06:08.237 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5 2024-09-18T08:06:08.260 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:06:08.261 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:06:08.261 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5' 2024-09-18T08:06:08.362 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0 2024-09-18T08:06:08.401 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:06:08.402 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:06:08.403 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0' 2024-09-18T08:06:08.494 INFO:teuthology.orchestra.run.smithi007.stderr:noscrub is unset 2024-09-18T08:06:08.504 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6 2024-09-18T08:06:08.534 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:06:08.534 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:06:08.535 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6' 2024-09-18T08:06:08.636 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7 2024-09-18T08:06:08.661 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:06:08.661 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:06:08.662 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7' 2024-09-18T08:06:08.763 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0 2024-09-18T08:06:08.802 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:06:08.803 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:06:08.803 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0' 2024-09-18T08:06:08.850 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:06:08 smithi050 ceph-mon[32371]: pgmap v142: 148 pgs: 148 active+clean; 5.5 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 877 B/s rd, 0 B/s wr, 883 op/s 2024-09-18T08:06:08.851 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:06:08 smithi050 ceph-mon[32371]: from='client.? 172.21.15.7:0/860955804' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-18T08:06:08.904 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6 2024-09-18T08:06:08.933 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:06:08.934 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:06:08.934 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6' 2024-09-18T08:06:08.980 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:06:08 smithi007 ceph-mon[29459]: pgmap v142: 148 pgs: 148 active+clean; 5.5 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 877 B/s rd, 0 B/s wr, 883 op/s 2024-09-18T08:06:08.980 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:06:08 smithi007 ceph-mon[29459]: from='client.? 172.21.15.7:0/860955804' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-18T08:06:08.981 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:06:08 smithi007 ceph-mon[39249]: pgmap v142: 148 pgs: 148 active+clean; 5.5 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 877 B/s rd, 0 B/s wr, 883 op/s 2024-09-18T08:06:08.981 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:06:08 smithi007 ceph-mon[39249]: from='client.? 172.21.15.7:0/860955804' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-18T08:06:09.034 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0 2024-09-18T08:06:09.079 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:06:09.080 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:06:09.080 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0' 2024-09-18T08:06:09.156 INFO:journalctl@ceph.iscsi.iscsi.a.smithi050.stdout:Sep 18 08:06:08 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-iscsi-iscsi-a[57782]: debug there is no tcmu-runner data available 2024-09-18T08:06:09.181 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4 2024-09-18T08:06:09.208 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:06:09.208 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:06:09.209 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4' 2024-09-18T08:06:09.310 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2 2024-09-18T08:06:09.350 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:06:09.351 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:06:09.351 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2' 2024-09-18T08:06:09.452 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2 2024-09-18T08:06:09.499 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:06:09.500 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:06:09.500 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2' 2024-09-18T08:06:09.599 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:06:09 smithi007 ceph-mon[29459]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-18T08:06:09.599 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:06:09 smithi007 ceph-mon[29459]: from='client.? 172.21.15.7:0/860955804' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-18T08:06:09.599 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:06:09 smithi007 ceph-mon[29459]: osdmap e128: 8 total, 8 up, 8 in 2024-09-18T08:06:09.599 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:06:09 smithi007 ceph-mon[29459]: from='client.? 172.21.15.7:0/1648764002' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-18T08:06:09.599 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:06:09 smithi007 ceph-mon[39249]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-18T08:06:09.600 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:06:09 smithi007 ceph-mon[39249]: from='client.? 172.21.15.7:0/860955804' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-18T08:06:09.600 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:06:09 smithi007 ceph-mon[39249]: osdmap e128: 8 total, 8 up, 8 in 2024-09-18T08:06:09.600 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:06:09 smithi007 ceph-mon[39249]: from='client.? 172.21.15.7:0/1648764002' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-18T08:06:09.601 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1 2024-09-18T08:06:09.642 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:06:09.642 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:06:09.643 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1' 2024-09-18T08:06:09.743 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4 2024-09-18T08:06:09.772 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:06:09.773 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:06:09.773 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4' 2024-09-18T08:06:09.874 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1 2024-09-18T08:06:09.905 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:06:09 smithi050 ceph-mon[32371]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-18T08:06:09.906 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:06:09 smithi050 ceph-mon[32371]: from='client.? 172.21.15.7:0/860955804' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-18T08:06:09.906 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:06:09 smithi050 ceph-mon[32371]: osdmap e128: 8 total, 8 up, 8 in 2024-09-18T08:06:09.906 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:06:09 smithi050 ceph-mon[32371]: from='client.? 172.21.15.7:0/1648764002' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-18T08:06:09.910 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:06:09.911 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:06:09.911 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1' 2024-09-18T08:06:10.012 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5 2024-09-18T08:06:10.037 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:06:10.038 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:06:10.038 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5' 2024-09-18T08:06:10.140 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7 2024-09-18T08:06:10.165 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:06:10.166 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:06:10.166 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7' 2024-09-18T08:06:10.267 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7 2024-09-18T08:06:10.295 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:06:10.295 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:06:10.296 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7' 2024-09-18T08:06:10.396 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2 2024-09-18T08:06:10.435 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:06:10.435 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:06:10.436 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2' 2024-09-18T08:06:10.536 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0 2024-09-18T08:06:10.583 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:06:10.583 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:06:10.584 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0' 2024-09-18T08:06:10.637 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:06:10 smithi007 ceph-mon[29459]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-18T08:06:10.637 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:06:10 smithi007 ceph-mon[29459]: pgmap v144: 148 pgs: 148 active+clean; 5.2 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.03k op/s 2024-09-18T08:06:10.637 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:06:10 smithi007 ceph-mon[29459]: 4.1b scrub starts 2024-09-18T08:06:10.637 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:06:10 smithi007 ceph-mon[29459]: 4.1b scrub ok 2024-09-18T08:06:10.637 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:06:10 smithi007 ceph-mon[29459]: 7.7 scrub starts 2024-09-18T08:06:10.638 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:06:10 smithi007 ceph-mon[39249]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-18T08:06:10.638 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:06:10 smithi007 ceph-mon[39249]: pgmap v144: 148 pgs: 148 active+clean; 5.2 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.03k op/s 2024-09-18T08:06:10.638 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:06:10 smithi007 ceph-mon[39249]: 4.1b scrub starts 2024-09-18T08:06:10.638 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:06:10 smithi007 ceph-mon[39249]: 4.1b scrub ok 2024-09-18T08:06:10.638 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:06:10 smithi007 ceph-mon[39249]: 7.7 scrub starts 2024-09-18T08:06:10.684 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5 2024-09-18T08:06:10.712 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:06:10.712 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:06:10.713 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5' 2024-09-18T08:06:10.814 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1 2024-09-18T08:06:10.857 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:06:10.857 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:06:10.858 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1' 2024-09-18T08:06:10.890 INFO:teuthology.orchestra.run.smithi007.stdout:osd.0: {} 2024-09-18T08:06:10.890 INFO:teuthology.orchestra.run.smithi007.stdout:osd.1: {} 2024-09-18T08:06:10.890 INFO:teuthology.orchestra.run.smithi007.stdout:osd.2: {} 2024-09-18T08:06:10.890 INFO:teuthology.orchestra.run.smithi007.stdout:osd.3: {} 2024-09-18T08:06:10.890 INFO:teuthology.orchestra.run.smithi007.stdout:osd.4: {} 2024-09-18T08:06:10.890 INFO:teuthology.orchestra.run.smithi007.stdout:osd.5: {} 2024-09-18T08:06:10.890 INFO:teuthology.orchestra.run.smithi007.stdout:osd.6: {} 2024-09-18T08:06:10.890 INFO:teuthology.orchestra.run.smithi007.stdout:osd.7: {} 2024-09-18T08:06:10.890 INFO:teuthology.orchestra.run.smithi007.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-18T08:06:10.891 INFO:teuthology.orchestra.run.smithi007.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-18T08:06:10.891 INFO:teuthology.orchestra.run.smithi007.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-18T08:06:10.891 INFO:teuthology.orchestra.run.smithi007.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-18T08:06:10.891 INFO:teuthology.orchestra.run.smithi007.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-18T08:06:10.891 INFO:teuthology.orchestra.run.smithi007.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-18T08:06:10.891 INFO:teuthology.orchestra.run.smithi007.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-18T08:06:10.891 INFO:teuthology.orchestra.run.smithi007.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-18T08:06:10.905 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:06:10 smithi050 ceph-mon[32371]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-18T08:06:10.906 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:06:10 smithi050 ceph-mon[32371]: pgmap v144: 148 pgs: 148 active+clean; 5.2 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.03k op/s 2024-09-18T08:06:10.906 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:06:10 smithi050 ceph-mon[32371]: 4.1b scrub starts 2024-09-18T08:06:10.906 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:06:10 smithi050 ceph-mon[32371]: 4.1b scrub ok 2024-09-18T08:06:10.906 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:06:10 smithi050 ceph-mon[32371]: 7.7 scrub starts 2024-09-18T08:06:10.958 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7 2024-09-18T08:06:10.986 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:06:10.987 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:06:10.987 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7' 2024-09-18T08:06:11.088 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6 2024-09-18T08:06:11.117 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:06:11.118 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:06:11.118 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6' 2024-09-18T08:06:11.219 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7 2024-09-18T08:06:11.230 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:06:11 smithi007 ceph-779349ca-7593-11ef-bceb-c7b262605968-mon-a[29455]: 2024-09-18T08:06:11.088+0000 7fcd2b595640 -1 mon.a@0(leader).osd e128 definitely_dead 0 2024-09-18T08:06:11.248 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:06:11.248 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:06:11.249 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7' 2024-09-18T08:06:11.350 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7 2024-09-18T08:06:11.380 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:06:11.381 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:06:11.381 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7' 2024-09-18T08:06:11.482 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1 2024-09-18T08:06:11.513 INFO:teuthology.orchestra.run.smithi007.stderr:marked out osd.6. 2024-09-18T08:06:11.519 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:06:11.520 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:06:11.521 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1' 2024-09-18T08:06:11.621 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2 2024-09-18T08:06:11.643 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:06:11.644 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:06:11.644 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2' 2024-09-18T08:06:11.745 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5 2024-09-18T08:06:11.768 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:06:11.768 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:06:11.768 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5' 2024-09-18T08:06:11.869 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7 2024-09-18T08:06:11.891 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:06:11.892 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:06:11.893 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7' 2024-09-18T08:06:11.899 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:06:11 smithi007 ceph-mon[29459]: 3.17 scrub starts 2024-09-18T08:06:11.900 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:06:11 smithi007 ceph-mon[29459]: 3.17 scrub ok 2024-09-18T08:06:11.900 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:06:11 smithi007 ceph-mon[29459]: 5.1 scrub starts 2024-09-18T08:06:11.900 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:06:11 smithi007 ceph-mon[29459]: 5.1 scrub ok 2024-09-18T08:06:11.900 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:06:11 smithi007 ceph-mon[29459]: 6.1 scrub starts 2024-09-18T08:06:11.900 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:06:11 smithi007 ceph-mon[29459]: 6.1 scrub ok 2024-09-18T08:06:11.900 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:06:11 smithi007 ceph-mon[29459]: from='client.? 172.21.15.7:0/488598916' entity='client.admin' cmd={"prefix": "osd out", "ids": ["6"]} : dispatch 2024-09-18T08:06:11.900 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:06:11 smithi007 ceph-mon[29459]: from='client.? ' entity='client.admin' cmd={"prefix": "osd out", "ids": ["6"]} : dispatch 2024-09-18T08:06:11.900 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:06:11 smithi007 ceph-mon[29459]: Client client.admin marked osd.6 out, while it was still marked up 2024-09-18T08:06:11.900 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:06:11 smithi007 ceph-mon[39249]: 3.17 scrub starts 2024-09-18T08:06:11.900 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:06:11 smithi007 ceph-mon[39249]: 3.17 scrub ok 2024-09-18T08:06:11.900 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:06:11 smithi007 ceph-mon[39249]: 5.1 scrub starts 2024-09-18T08:06:11.901 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:06:11 smithi007 ceph-mon[39249]: 5.1 scrub ok 2024-09-18T08:06:11.901 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:06:11 smithi007 ceph-mon[39249]: 6.1 scrub starts 2024-09-18T08:06:11.901 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:06:11 smithi007 ceph-mon[39249]: 6.1 scrub ok 2024-09-18T08:06:11.901 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:06:11 smithi007 ceph-mon[39249]: from='client.? 172.21.15.7:0/488598916' entity='client.admin' cmd={"prefix": "osd out", "ids": ["6"]} : dispatch 2024-09-18T08:06:11.901 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:06:11 smithi007 ceph-mon[39249]: from='client.? ' entity='client.admin' cmd={"prefix": "osd out", "ids": ["6"]} : dispatch 2024-09-18T08:06:11.901 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:06:11 smithi007 ceph-mon[39249]: Client client.admin marked osd.6 out, while it was still marked up 2024-09-18T08:06:11.905 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:06:11 smithi050 ceph-mon[32371]: 3.17 scrub starts 2024-09-18T08:06:11.906 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:06:11 smithi050 ceph-mon[32371]: 3.17 scrub ok 2024-09-18T08:06:11.906 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:06:11 smithi050 ceph-mon[32371]: 5.1 scrub starts 2024-09-18T08:06:11.906 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:06:11 smithi050 ceph-mon[32371]: 5.1 scrub ok 2024-09-18T08:06:11.906 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:06:11 smithi050 ceph-mon[32371]: 6.1 scrub starts 2024-09-18T08:06:11.906 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:06:11 smithi050 ceph-mon[32371]: 6.1 scrub ok 2024-09-18T08:06:11.906 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:06:11 smithi050 ceph-mon[32371]: from='client.? 172.21.15.7:0/488598916' entity='client.admin' cmd={"prefix": "osd out", "ids": ["6"]} : dispatch 2024-09-18T08:06:11.906 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:06:11 smithi050 ceph-mon[32371]: from='client.? ' entity='client.admin' cmd={"prefix": "osd out", "ids": ["6"]} : dispatch 2024-09-18T08:06:11.906 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:06:11 smithi050 ceph-mon[32371]: Client client.admin marked osd.6 out, while it was still marked up 2024-09-18T08:06:11.994 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5 2024-09-18T08:06:12.016 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:06:12.016 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:06:12.017 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5' 2024-09-18T08:06:12.052 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid 779349ca-7593-11ef-bceb-c7b262605968 -- ceph osd unset nodeep-scrub 2024-09-18T08:06:12.118 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1 2024-09-18T08:06:12.140 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:06:12.141 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:06:12.141 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1' 2024-09-18T08:06:12.242 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3 2024-09-18T08:06:12.264 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:06:12.265 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:06:12.265 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3' 2024-09-18T08:06:12.366 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1 2024-09-18T08:06:12.388 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:06:12.388 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:06:12.389 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1' 2024-09-18T08:06:12.459 INFO:teuthology.orchestra.run.smithi007.stdout:ERROR: (22) Invalid argument 2024-09-18T08:06:12.460 INFO:teuthology.orchestra.run.smithi007.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-18T08:06:12.490 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5 2024-09-18T08:06:12.518 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:06:12.518 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:06:12.518 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5' 2024-09-18T08:06:12.619 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6 2024-09-18T08:06:12.642 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:06:12.642 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:06:12.643 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6' 2024-09-18T08:06:12.730 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:06:12 smithi007 ceph-mon[29459]: pgmap v145: 148 pgs: 148 active+clean; 5.2 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.03k op/s 2024-09-18T08:06:12.731 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:06:12 smithi007 ceph-mon[29459]: 6.11 scrub starts 2024-09-18T08:06:12.731 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:06:12 smithi007 ceph-mon[29459]: 6.11 scrub ok 2024-09-18T08:06:12.731 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:06:12 smithi007 ceph-mon[29459]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd out", "ids": ["6"]}]': finished 2024-09-18T08:06:12.731 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:06:12 smithi007 ceph-mon[29459]: osdmap e129: 8 total, 8 up, 7 in 2024-09-18T08:06:12.731 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:06:12 smithi007 ceph-mon[29459]: osdmap e130: 8 total, 8 up, 7 in 2024-09-18T08:06:12.731 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:06:12 smithi007 ceph-mon[39249]: pgmap v145: 148 pgs: 148 active+clean; 5.2 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.03k op/s 2024-09-18T08:06:12.731 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:06:12 smithi007 ceph-mon[39249]: 6.11 scrub starts 2024-09-18T08:06:12.731 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:06:12 smithi007 ceph-mon[39249]: 6.11 scrub ok 2024-09-18T08:06:12.731 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:06:12 smithi007 ceph-mon[39249]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd out", "ids": ["6"]}]': finished 2024-09-18T08:06:12.731 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:06:12 smithi007 ceph-mon[39249]: osdmap e129: 8 total, 8 up, 7 in 2024-09-18T08:06:12.732 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:06:12 smithi007 ceph-mon[39249]: osdmap e130: 8 total, 8 up, 7 in 2024-09-18T08:06:12.744 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4 2024-09-18T08:06:12.766 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:06:12.766 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:06:12.767 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4' 2024-09-18T08:06:12.807 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:06:12 smithi050 ceph-mon[32371]: pgmap v145: 148 pgs: 148 active+clean; 5.2 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.03k op/s 2024-09-18T08:06:12.807 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:06:12 smithi050 ceph-mon[32371]: 6.11 scrub starts 2024-09-18T08:06:12.807 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:06:12 smithi050 ceph-mon[32371]: 6.11 scrub ok 2024-09-18T08:06:12.808 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:06:12 smithi050 ceph-mon[32371]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd out", "ids": ["6"]}]': finished 2024-09-18T08:06:12.808 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:06:12 smithi050 ceph-mon[32371]: osdmap e129: 8 total, 8 up, 7 in 2024-09-18T08:06:12.808 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:06:12 smithi050 ceph-mon[32371]: osdmap e130: 8 total, 8 up, 7 in 2024-09-18T08:06:12.868 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4 2024-09-18T08:06:12.890 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:06:12.891 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:06:12.891 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4' 2024-09-18T08:06:12.991 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7 2024-09-18T08:06:13.013 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:06:13.014 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:06:13.014 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7' 2024-09-18T08:06:13.115 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4 2024-09-18T08:06:13.137 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:06:13.138 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:06:13.138 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4' 2024-09-18T08:06:13.142 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid 779349ca-7593-11ef-bceb-c7b262605968 -- ceph daemon osd.2 dump_historic_ops 2024-09-18T08:06:13.184 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/779349ca-7593-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T08:06:13.239 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2 2024-09-18T08:06:13.264 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:06:13.264 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:06:13.264 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2' 2024-09-18T08:06:13.365 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1 2024-09-18T08:06:13.387 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:06:13.388 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:06:13.388 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1' 2024-09-18T08:06:13.489 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1 2024-09-18T08:06:13.516 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:06:13.517 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:06:13.517 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1' 2024-09-18T08:06:13.602 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid 779349ca-7593-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-18T08:06:13.618 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3 2024-09-18T08:06:13.724 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:06:13.725 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:06:13.725 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3' 2024-09-18T08:06:13.730 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:06:13 smithi007 ceph-mon[29459]: 5.1c scrub starts 2024-09-18T08:06:13.730 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:06:13 smithi007 ceph-mon[29459]: 5.1c scrub ok 2024-09-18T08:06:13.730 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:06:13 smithi007 ceph-mon[29459]: 3.12 scrub starts 2024-09-18T08:06:13.730 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:06:13 smithi007 ceph-mon[29459]: 3.12 scrub ok 2024-09-18T08:06:13.731 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:06:13 smithi007 ceph-mon[29459]: pgmap v148: 148 pgs: 1 remapped+peering, 147 active+clean; 5.2 GiB data, 14 GiB used, 611 GiB / 626 GiB avail; 1.7 KiB/s rd, 0 B/s wr, 971 op/s; 96 KiB/s, 8 objects/s recovering 2024-09-18T08:06:13.731 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:06:13 smithi007 ceph-mon[29459]: 3.11 scrub starts 2024-09-18T08:06:13.731 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:06:13 smithi007 ceph-mon[29459]: 3.11 scrub ok 2024-09-18T08:06:13.731 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:06:13 smithi007 ceph-mon[29459]: osdmap e131: 8 total, 8 up, 7 in 2024-09-18T08:06:13.731 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:06:13 smithi007 ceph-mon[39249]: 5.1c scrub starts 2024-09-18T08:06:13.731 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:06:13 smithi007 ceph-mon[39249]: 5.1c scrub ok 2024-09-18T08:06:13.731 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:06:13 smithi007 ceph-mon[39249]: 3.12 scrub starts 2024-09-18T08:06:13.732 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:06:13 smithi007 ceph-mon[39249]: 3.12 scrub ok 2024-09-18T08:06:13.732 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:06:13 smithi007 ceph-mon[39249]: pgmap v148: 148 pgs: 1 remapped+peering, 147 active+clean; 5.2 GiB data, 14 GiB used, 611 GiB / 626 GiB avail; 1.7 KiB/s rd, 0 B/s wr, 971 op/s; 96 KiB/s, 8 objects/s recovering 2024-09-18T08:06:13.732 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:06:13 smithi007 ceph-mon[39249]: 3.11 scrub starts 2024-09-18T08:06:13.732 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:06:13 smithi007 ceph-mon[39249]: 3.11 scrub ok 2024-09-18T08:06:13.732 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:06:13 smithi007 ceph-mon[39249]: osdmap e131: 8 total, 8 up, 7 in 2024-09-18T08:06:13.784 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/779349ca-7593-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T08:06:13.826 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2 2024-09-18T08:06:13.886 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:06:13.887 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:06:13.887 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2' 2024-09-18T08:06:13.906 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:06:13 smithi050 ceph-mon[32371]: 5.1c scrub starts 2024-09-18T08:06:13.906 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:06:13 smithi050 ceph-mon[32371]: 5.1c scrub ok 2024-09-18T08:06:13.906 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:06:13 smithi050 ceph-mon[32371]: 3.12 scrub starts 2024-09-18T08:06:13.906 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:06:13 smithi050 ceph-mon[32371]: 3.12 scrub ok 2024-09-18T08:06:13.906 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:06:13 smithi050 ceph-mon[32371]: pgmap v148: 148 pgs: 1 remapped+peering, 147 active+clean; 5.2 GiB data, 14 GiB used, 611 GiB / 626 GiB avail; 1.7 KiB/s rd, 0 B/s wr, 971 op/s; 96 KiB/s, 8 objects/s recovering 2024-09-18T08:06:13.906 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:06:13 smithi050 ceph-mon[32371]: 3.11 scrub starts 2024-09-18T08:06:13.906 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:06:13 smithi050 ceph-mon[32371]: 3.11 scrub ok 2024-09-18T08:06:13.906 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:06:13 smithi050 ceph-mon[32371]: osdmap e131: 8 total, 8 up, 7 in 2024-09-18T08:06:13.988 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7 2024-09-18T08:06:14.013 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:06:14.014 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:06:14.014 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7' 2024-09-18T08:06:14.115 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7 2024-09-18T08:06:14.139 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:06:14.140 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:06:14.140 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7' 2024-09-18T08:06:14.241 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5 2024-09-18T08:06:14.266 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:06:14.267 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:06:14.267 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5' 2024-09-18T08:06:14.368 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6 2024-09-18T08:06:14.396 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:06:14.396 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:06:14.396 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6' 2024-09-18T08:06:14.482 INFO:teuthology.orchestra.run.smithi007.stdout:ERROR: (22) Invalid argument 2024-09-18T08:06:14.482 INFO:teuthology.orchestra.run.smithi007.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-18T08:06:14.497 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3 2024-09-18T08:06:14.548 INFO:teuthology.orchestra.run.smithi007.stderr:nodeep-scrub is unset 2024-09-18T08:06:14.554 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:06:14.555 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:06:14.556 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3' 2024-09-18T08:06:14.656 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6 2024-09-18T08:06:14.683 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:06:14.683 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:06:14.684 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6' 2024-09-18T08:06:14.715 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:06:14 smithi007 ceph-mon[29459]: purged_snaps scrub starts 2024-09-18T08:06:14.715 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:06:14 smithi007 ceph-mon[29459]: purged_snaps scrub ok 2024-09-18T08:06:14.715 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:06:14 smithi007 ceph-mon[29459]: 4.1f scrub starts 2024-09-18T08:06:14.715 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:06:14 smithi007 ceph-mon[29459]: 4.1f scrub ok 2024-09-18T08:06:14.715 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:06:14 smithi007 ceph-mon[29459]: 6.13 scrub starts 2024-09-18T08:06:14.715 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:06:14 smithi007 ceph-mon[29459]: 6.13 scrub ok 2024-09-18T08:06:14.715 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:06:14 smithi007 ceph-mon[29459]: 5.3 scrub starts 2024-09-18T08:06:14.715 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:06:14 smithi007 ceph-mon[29459]: 5.3 scrub ok 2024-09-18T08:06:14.715 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:06:14 smithi007 ceph-mon[29459]: 7.1 starting backfill to osd.1 from (0'0,0'0] MAX to 128'9702 2024-09-18T08:06:14.715 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:06:14 smithi007 ceph-mon[29459]: 7.b starting backfill to osd.2 from (0'0,0'0] MAX to 128'9592 2024-09-18T08:06:14.715 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:06:14 smithi007 ceph-mon[29459]: 7.c starting backfill to osd.2 from (0'0,0'0] MAX to 128'9827 2024-09-18T08:06:14.715 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:06:14 smithi007 ceph-mon[29459]: 7.d starting backfill to osd.3 from (0'0,0'0] MAX to 128'9703 2024-09-18T08:06:14.716 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:06:14 smithi007 ceph-mon[29459]: 7.7 starting backfill to osd.1 from (0'0,0'0] MAX to 128'9727 2024-09-18T08:06:14.716 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:06:14 smithi007 ceph-mon[29459]: from='client.? 172.21.15.7:0/3050990700' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-18T08:06:14.716 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:06:14 smithi007 ceph-mon[29459]: 5.9 scrub starts 2024-09-18T08:06:14.716 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:06:14 smithi007 ceph-mon[29459]: 5.9 scrub ok 2024-09-18T08:06:14.716 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:06:14 smithi007 ceph-mon[29459]: from='mgr.24470 172.21.15.7:0/2594390261' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-18T08:06:14.716 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:06:14 smithi007 ceph-mon[39249]: purged_snaps scrub starts 2024-09-18T08:06:14.717 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:06:14 smithi007 ceph-mon[39249]: purged_snaps scrub ok 2024-09-18T08:06:14.717 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:06:14 smithi007 ceph-mon[39249]: 4.1f scrub starts 2024-09-18T08:06:14.717 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:06:14 smithi007 ceph-mon[39249]: 4.1f scrub ok 2024-09-18T08:06:14.717 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:06:14 smithi007 ceph-mon[39249]: 6.13 scrub starts 2024-09-18T08:06:14.717 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:06:14 smithi007 ceph-mon[39249]: 6.13 scrub ok 2024-09-18T08:06:14.717 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:06:14 smithi007 ceph-mon[39249]: 5.3 scrub starts 2024-09-18T08:06:14.717 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:06:14 smithi007 ceph-mon[39249]: 5.3 scrub ok 2024-09-18T08:06:14.717 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:06:14 smithi007 ceph-mon[39249]: 7.1 starting backfill to osd.1 from (0'0,0'0] MAX to 128'9702 2024-09-18T08:06:14.717 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:06:14 smithi007 ceph-mon[39249]: 7.b starting backfill to osd.2 from (0'0,0'0] MAX to 128'9592 2024-09-18T08:06:14.717 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:06:14 smithi007 ceph-mon[39249]: 7.c starting backfill to osd.2 from (0'0,0'0] MAX to 128'9827 2024-09-18T08:06:14.717 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:06:14 smithi007 ceph-mon[39249]: 7.d starting backfill to osd.3 from (0'0,0'0] MAX to 128'9703 2024-09-18T08:06:14.717 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:06:14 smithi007 ceph-mon[39249]: 7.7 starting backfill to osd.1 from (0'0,0'0] MAX to 128'9727 2024-09-18T08:06:14.717 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:06:14 smithi007 ceph-mon[39249]: from='client.? 172.21.15.7:0/3050990700' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-18T08:06:14.717 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:06:14 smithi007 ceph-mon[39249]: 5.9 scrub starts 2024-09-18T08:06:14.717 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:06:14 smithi007 ceph-mon[39249]: 5.9 scrub ok 2024-09-18T08:06:14.718 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:06:14 smithi007 ceph-mon[39249]: from='mgr.24470 172.21.15.7:0/2594390261' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-18T08:06:14.784 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2 2024-09-18T08:06:14.836 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:06:14.837 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:06:14.837 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2' 2024-09-18T08:06:14.906 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:06:14 smithi050 ceph-mon[32371]: purged_snaps scrub starts 2024-09-18T08:06:14.906 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:06:14 smithi050 ceph-mon[32371]: purged_snaps scrub ok 2024-09-18T08:06:14.906 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:06:14 smithi050 ceph-mon[32371]: 4.1f scrub starts 2024-09-18T08:06:14.906 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:06:14 smithi050 ceph-mon[32371]: 4.1f scrub ok 2024-09-18T08:06:14.906 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:06:14 smithi050 ceph-mon[32371]: 6.13 scrub starts 2024-09-18T08:06:14.906 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:06:14 smithi050 ceph-mon[32371]: 6.13 scrub ok 2024-09-18T08:06:14.906 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:06:14 smithi050 ceph-mon[32371]: 5.3 scrub starts 2024-09-18T08:06:14.906 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:06:14 smithi050 ceph-mon[32371]: 5.3 scrub ok 2024-09-18T08:06:14.907 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:06:14 smithi050 ceph-mon[32371]: 7.1 starting backfill to osd.1 from (0'0,0'0] MAX to 128'9702 2024-09-18T08:06:14.907 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:06:14 smithi050 ceph-mon[32371]: 7.b starting backfill to osd.2 from (0'0,0'0] MAX to 128'9592 2024-09-18T08:06:14.907 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:06:14 smithi050 ceph-mon[32371]: 7.c starting backfill to osd.2 from (0'0,0'0] MAX to 128'9827 2024-09-18T08:06:14.907 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:06:14 smithi050 ceph-mon[32371]: 7.d starting backfill to osd.3 from (0'0,0'0] MAX to 128'9703 2024-09-18T08:06:14.907 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:06:14 smithi050 ceph-mon[32371]: 7.7 starting backfill to osd.1 from (0'0,0'0] MAX to 128'9727 2024-09-18T08:06:14.907 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:06:14 smithi050 ceph-mon[32371]: from='client.? 172.21.15.7:0/3050990700' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-18T08:06:14.907 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:06:14 smithi050 ceph-mon[32371]: 5.9 scrub starts 2024-09-18T08:06:14.907 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:06:14 smithi050 ceph-mon[32371]: 5.9 scrub ok 2024-09-18T08:06:14.907 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:06:14 smithi050 ceph-mon[32371]: from='mgr.24470 172.21.15.7:0/2594390261' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-18T08:06:14.937 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7 2024-09-18T08:06:14.961 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/779349ca-7593-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T08:06:14.964 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:06:14.965 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:06:14.965 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7' 2024-09-18T08:06:15.066 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7 2024-09-18T08:06:15.092 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:06:15.092 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:06:15.093 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7' 2024-09-18T08:06:15.194 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1 2024-09-18T08:06:15.239 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:06:15.239 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:06:15.240 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1' 2024-09-18T08:06:15.342 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0 2024-09-18T08:06:15.383 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:06:15.384 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:06:15.384 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0' 2024-09-18T08:06:15.485 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4 2024-09-18T08:06:15.515 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:06:15.517 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:06:15.517 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4' 2024-09-18T08:06:15.618 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2 2024-09-18T08:06:15.654 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:06:15.655 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:06:15.655 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2' 2024-09-18T08:06:15.731 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:06:15 smithi007 ceph-mon[29459]: 3.1d scrub starts 2024-09-18T08:06:15.732 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:06:15 smithi007 ceph-mon[29459]: 3.1d scrub ok 2024-09-18T08:06:15.732 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:06:15 smithi007 ceph-mon[29459]: 4.e scrub starts 2024-09-18T08:06:15.732 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:06:15 smithi007 ceph-mon[29459]: 4.e scrub ok 2024-09-18T08:06:15.732 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:06:15 smithi007 ceph-mon[29459]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-18T08:06:15.732 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:06:15 smithi007 ceph-mon[29459]: Cluster is now healthy 2024-09-18T08:06:15.732 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:06:15 smithi007 ceph-mon[29459]: from='client.? 172.21.15.7:0/3050990700' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-18T08:06:15.732 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:06:15 smithi007 ceph-mon[29459]: osdmap e132: 8 total, 8 up, 7 in 2024-09-18T08:06:15.732 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:06:15 smithi007 ceph-mon[29459]: purged_snaps scrub starts 2024-09-18T08:06:15.732 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:06:15 smithi007 ceph-mon[29459]: purged_snaps scrub ok 2024-09-18T08:06:15.732 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:06:15 smithi007 ceph-mon[29459]: pgmap v151: 148 pgs: 5 remapped+peering, 143 active+clean; 5.0 GiB data, 14 GiB used, 612 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 837 op/s; 144 KiB/s, 17 objects/s recovering 2024-09-18T08:06:15.732 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:06:15 smithi007 ceph-mon[29459]: 5.10 scrub starts 2024-09-18T08:06:15.732 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:06:15 smithi007 ceph-mon[29459]: 5.10 scrub ok 2024-09-18T08:06:15.733 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:06:15 smithi007 ceph-mon[39249]: 3.1d scrub starts 2024-09-18T08:06:15.733 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:06:15 smithi007 ceph-mon[39249]: 3.1d scrub ok 2024-09-18T08:06:15.733 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:06:15 smithi007 ceph-mon[39249]: 4.e scrub starts 2024-09-18T08:06:15.733 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:06:15 smithi007 ceph-mon[39249]: 4.e scrub ok 2024-09-18T08:06:15.733 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:06:15 smithi007 ceph-mon[39249]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-18T08:06:15.733 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:06:15 smithi007 ceph-mon[39249]: Cluster is now healthy 2024-09-18T08:06:15.733 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:06:15 smithi007 ceph-mon[39249]: from='client.? 172.21.15.7:0/3050990700' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-18T08:06:15.733 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:06:15 smithi007 ceph-mon[39249]: osdmap e132: 8 total, 8 up, 7 in 2024-09-18T08:06:15.734 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:06:15 smithi007 ceph-mon[39249]: purged_snaps scrub starts 2024-09-18T08:06:15.734 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:06:15 smithi007 ceph-mon[39249]: purged_snaps scrub ok 2024-09-18T08:06:15.734 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:06:15 smithi007 ceph-mon[39249]: pgmap v151: 148 pgs: 5 remapped+peering, 143 active+clean; 5.0 GiB data, 14 GiB used, 612 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 837 op/s; 144 KiB/s, 17 objects/s recovering 2024-09-18T08:06:15.734 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:06:15 smithi007 ceph-mon[39249]: 5.10 scrub starts 2024-09-18T08:06:15.734 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:06:15 smithi007 ceph-mon[39249]: 5.10 scrub ok 2024-09-18T08:06:15.756 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6 2024-09-18T08:06:15.784 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:06:15.785 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:06:15.786 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6' 2024-09-18T08:06:15.887 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7 2024-09-18T08:06:15.906 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:06:15 smithi050 ceph-mon[32371]: 3.1d scrub starts 2024-09-18T08:06:15.906 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:06:15 smithi050 ceph-mon[32371]: 3.1d scrub ok 2024-09-18T08:06:15.906 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:06:15 smithi050 ceph-mon[32371]: 4.e scrub starts 2024-09-18T08:06:15.906 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:06:15 smithi050 ceph-mon[32371]: 4.e scrub ok 2024-09-18T08:06:15.906 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:06:15 smithi050 ceph-mon[32371]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-18T08:06:15.907 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:06:15 smithi050 ceph-mon[32371]: Cluster is now healthy 2024-09-18T08:06:15.907 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:06:15 smithi050 ceph-mon[32371]: from='client.? 172.21.15.7:0/3050990700' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-18T08:06:15.907 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:06:15 smithi050 ceph-mon[32371]: osdmap e132: 8 total, 8 up, 7 in 2024-09-18T08:06:15.907 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:06:15 smithi050 ceph-mon[32371]: purged_snaps scrub starts 2024-09-18T08:06:15.907 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:06:15 smithi050 ceph-mon[32371]: purged_snaps scrub ok 2024-09-18T08:06:15.907 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:06:15 smithi050 ceph-mon[32371]: pgmap v151: 148 pgs: 5 remapped+peering, 143 active+clean; 5.0 GiB data, 14 GiB used, 612 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 837 op/s; 144 KiB/s, 17 objects/s recovering 2024-09-18T08:06:15.907 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:06:15 smithi050 ceph-mon[32371]: 5.10 scrub starts 2024-09-18T08:06:15.907 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:06:15 smithi050 ceph-mon[32371]: 5.10 scrub ok 2024-09-18T08:06:15.912 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:06:15.912 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:06:15.913 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7' 2024-09-18T08:06:15.921 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid 779349ca-7593-11ef-bceb-c7b262605968 -- ceph daemon osd.3 dump_ops_in_flight 2024-09-18T08:06:16.016 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2 2024-09-18T08:06:16.078 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:06:16.078 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:06:16.079 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2' 2024-09-18T08:06:16.179 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2 2024-09-18T08:06:16.234 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:06:16.235 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:06:16.235 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2' 2024-09-18T08:06:16.336 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0 2024-09-18T08:06:16.381 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:06:16.382 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:06:16.382 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0' 2024-09-18T08:06:16.483 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0 2024-09-18T08:06:16.537 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:06:16.537 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:06:16.537 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0' 2024-09-18T08:06:16.638 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3 2024-09-18T08:06:16.673 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:06:16.674 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:06:16.674 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3' 2024-09-18T08:06:16.775 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1 2024-09-18T08:06:16.815 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:06:16.815 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:06:16.816 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1' 2024-09-18T08:06:16.906 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:06:16 smithi050 ceph-mon[32371]: 4.5 scrub starts 2024-09-18T08:06:16.906 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:06:16 smithi050 ceph-mon[32371]: 4.5 scrub ok 2024-09-18T08:06:16.906 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:06:16 smithi050 ceph-mon[32371]: osdmap e133: 8 total, 8 up, 7 in 2024-09-18T08:06:16.916 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2 2024-09-18T08:06:16.961 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:06:16.963 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:06:16.964 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2' 2024-09-18T08:06:16.980 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:06:16 smithi007 ceph-mon[29459]: 4.5 scrub starts 2024-09-18T08:06:16.980 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:06:16 smithi007 ceph-mon[29459]: 4.5 scrub ok 2024-09-18T08:06:16.980 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:06:16 smithi007 ceph-mon[29459]: osdmap e133: 8 total, 8 up, 7 in 2024-09-18T08:06:16.981 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Sep 18 08:06:16 smithi007 ceph-779349ca-7593-11ef-bceb-c7b262605968-mgr-y[29688]: ::ffff:172.21.15.50 - - [18/Sep/2024:08:06:16] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-09-18T08:06:16.981 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:06:16 smithi007 ceph-mon[39249]: 4.5 scrub starts 2024-09-18T08:06:16.981 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:06:16 smithi007 ceph-mon[39249]: 4.5 scrub ok 2024-09-18T08:06:16.981 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:06:16 smithi007 ceph-mon[39249]: osdmap e133: 8 total, 8 up, 7 in 2024-09-18T08:06:17.065 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2 2024-09-18T08:06:17.107 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:06:17.108 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:06:17.108 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2' 2024-09-18T08:06:17.209 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5 2024-09-18T08:06:17.214 INFO:teuthology.orchestra.run.smithi007.stdout:osd.0: {} 2024-09-18T08:06:17.214 INFO:teuthology.orchestra.run.smithi007.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-18T08:06:17.229 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 7] out_osds: [6] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-09-18T08:06:17.230 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-18T08:06:17.230 INFO:tasks.thrashosds.thrasher:Adding osd 6 2024-09-18T08:06:17.230 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid 779349ca-7593-11ef-bceb-c7b262605968 -- ceph osd in 6 2024-09-18T08:06:17.234 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:06:17.234 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:06:17.235 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5' 2024-09-18T08:06:17.254 INFO:teuthology.orchestra.run.smithi007.stdout:osd.1: {} 2024-09-18T08:06:17.254 INFO:teuthology.orchestra.run.smithi007.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-18T08:06:17.298 INFO:teuthology.orchestra.run.smithi007.stdout:osd.2: {} 2024-09-18T08:06:17.298 INFO:teuthology.orchestra.run.smithi007.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-18T08:06:17.336 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2 2024-09-18T08:06:17.340 INFO:teuthology.orchestra.run.smithi007.stdout:osd.3: {} 2024-09-18T08:06:17.340 INFO:teuthology.orchestra.run.smithi007.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-18T08:06:17.387 INFO:teuthology.orchestra.run.smithi007.stdout:osd.4: {} 2024-09-18T08:06:17.387 INFO:teuthology.orchestra.run.smithi007.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-18T08:06:17.389 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:06:17.390 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:06:17.390 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2' 2024-09-18T08:06:17.423 INFO:teuthology.orchestra.run.smithi007.stdout:osd.5: {} 2024-09-18T08:06:17.423 INFO:teuthology.orchestra.run.smithi007.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-18T08:06:17.458 INFO:teuthology.orchestra.run.smithi007.stdout:osd.6: {} 2024-09-18T08:06:17.458 INFO:teuthology.orchestra.run.smithi007.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-18T08:06:17.462 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid 779349ca-7593-11ef-bceb-c7b262605968 -- ceph osd set noscrub 2024-09-18T08:06:17.491 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7 2024-09-18T08:06:17.495 INFO:teuthology.orchestra.run.smithi007.stdout:osd.7: {} 2024-09-18T08:06:17.495 INFO:teuthology.orchestra.run.smithi007.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-18T08:06:17.516 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:06:17.517 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:06:17.517 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7' 2024-09-18T08:06:17.618 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0 2024-09-18T08:06:17.661 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:06:17.662 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:06:17.662 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0' 2024-09-18T08:06:17.703 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:06:17 smithi050 ceph-mon[32371]: pgmap v153: 148 pgs: 5 remapped+peering, 143 active+clean; 5.0 GiB data, 14 GiB used, 612 GiB / 626 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 724 op/s; 125 KiB/s, 15 objects/s recovering 2024-09-18T08:06:17.703 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:06:17 smithi050 ceph-mon[32371]: from='client.? 172.21.15.7:0/2799182612' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-18T08:06:17.762 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6 2024-09-18T08:06:17.787 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:06:17.788 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:06:17.788 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6' 2024-09-18T08:06:17.882 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/779349ca-7593-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T08:06:17.889 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3 2024-09-18T08:06:17.904 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:06:17 smithi007 ceph-mon[29459]: pgmap v153: 148 pgs: 5 remapped+peering, 143 active+clean; 5.0 GiB data, 14 GiB used, 612 GiB / 626 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 724 op/s; 125 KiB/s, 15 objects/s recovering 2024-09-18T08:06:17.904 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:06:17 smithi007 ceph-mon[29459]: from='client.? 172.21.15.7:0/2799182612' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-18T08:06:17.905 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:06:17 smithi007 ceph-mon[39249]: pgmap v153: 148 pgs: 5 remapped+peering, 143 active+clean; 5.0 GiB data, 14 GiB used, 612 GiB / 626 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 724 op/s; 125 KiB/s, 15 objects/s recovering 2024-09-18T08:06:17.906 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:06:17 smithi007 ceph-mon[39249]: from='client.? 172.21.15.7:0/2799182612' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-18T08:06:17.960 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:06:17.961 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:06:17.961 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3' 2024-09-18T08:06:18.062 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1 2024-09-18T08:06:18.106 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:06:18.106 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:06:18.107 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1' 2024-09-18T08:06:18.207 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4 2024-09-18T08:06:18.236 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:06:18.237 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:06:18.237 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4' 2024-09-18T08:06:18.338 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2 2024-09-18T08:06:18.383 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:06:18.383 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:06:18.384 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2' 2024-09-18T08:06:18.485 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0 2024-09-18T08:06:18.531 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:06:18.532 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:06:18.532 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0' 2024-09-18T08:06:18.633 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4 2024-09-18T08:06:18.658 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:06:18.659 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:06:18.659 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4' 2024-09-18T08:06:18.760 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7 2024-09-18T08:06:18.791 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:06:18.792 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:06:18.792 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7' 2024-09-18T08:06:18.893 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6 2024-09-18T08:06:18.921 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:06:18.922 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:06:18.922 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6' 2024-09-18T08:06:18.948 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:06:18 smithi007 ceph-mon[29459]: from='mgr.24470 172.21.15.7:0/2594390261' entity='mgr.y' 2024-09-18T08:06:18.948 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:06:18 smithi007 ceph-mon[29459]: from='mgr.24470 172.21.15.7:0/2594390261' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-18T08:06:18.949 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:06:18 smithi007 ceph-mon[29459]: 3.16 scrub starts 2024-09-18T08:06:18.949 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:06:18 smithi007 ceph-mon[29459]: 3.16 scrub ok 2024-09-18T08:06:18.949 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:06:18 smithi007 ceph-mon[39249]: from='mgr.24470 172.21.15.7:0/2594390261' entity='mgr.y' 2024-09-18T08:06:18.949 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:06:18 smithi007 ceph-mon[39249]: from='mgr.24470 172.21.15.7:0/2594390261' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-18T08:06:18.949 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:06:18 smithi007 ceph-mon[39249]: 3.16 scrub starts 2024-09-18T08:06:18.949 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:06:18 smithi007 ceph-mon[39249]: 3.16 scrub ok 2024-09-18T08:06:19.022 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0 2024-09-18T08:06:19.073 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:06:19.074 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:06:19.074 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0' 2024-09-18T08:06:19.122 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:06:18 smithi050 ceph-mon[32371]: from='mgr.24470 172.21.15.7:0/2594390261' entity='mgr.y' 2024-09-18T08:06:19.122 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:06:18 smithi050 ceph-mon[32371]: from='mgr.24470 172.21.15.7:0/2594390261' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-18T08:06:19.122 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:06:18 smithi050 ceph-mon[32371]: 3.16 scrub starts 2024-09-18T08:06:19.123 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:06:18 smithi050 ceph-mon[32371]: 3.16 scrub ok 2024-09-18T08:06:19.123 INFO:journalctl@ceph.iscsi.iscsi.a.smithi050.stdout:Sep 18 08:06:18 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-iscsi-iscsi-a[57782]: debug there is no tcmu-runner data available 2024-09-18T08:06:19.175 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6 2024-09-18T08:06:19.201 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:06:19.202 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:06:19.202 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6' 2024-09-18T08:06:19.253 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/779349ca-7593-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T08:06:19.293 INFO:teuthology.orchestra.run.smithi007.stdout:{ 2024-09-18T08:06:19.293 INFO:teuthology.orchestra.run.smithi007.stdout: "ops": [ 2024-09-18T08:06:19.293 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:06:19.293 INFO:teuthology.orchestra.run.smithi007.stdout: "description": "osd_op(client.14805.0:159598 7.c 7:3c65cbaa:::benchmark_data_smithi007_73155_object41307:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e133)", 2024-09-18T08:06:19.293 INFO:teuthology.orchestra.run.smithi007.stdout: "initiated_at": "2024-09-18T08:06:19.273865+0000", 2024-09-18T08:06:19.293 INFO:teuthology.orchestra.run.smithi007.stdout: "age": 0.014229462, 2024-09-18T08:06:19.293 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.014260653, 2024-09-18T08:06:19.293 INFO:teuthology.orchestra.run.smithi007.stdout: "continuous": false, 2024-09-18T08:06:19.294 INFO:teuthology.orchestra.run.smithi007.stdout: "type_data": { 2024-09-18T08:06:19.294 INFO:teuthology.orchestra.run.smithi007.stdout: "flag_point": "waiting for sub ops", 2024-09-18T08:06:19.294 INFO:teuthology.orchestra.run.smithi007.stdout: "client_info": { 2024-09-18T08:06:19.294 INFO:teuthology.orchestra.run.smithi007.stdout: "client": "client.14805", 2024-09-18T08:06:19.294 INFO:teuthology.orchestra.run.smithi007.stdout: "client_addr": "172.21.15.7:0/640320436", 2024-09-18T08:06:19.294 INFO:teuthology.orchestra.run.smithi007.stdout: "tid": 159598 2024-09-18T08:06:19.294 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:06:19.294 INFO:teuthology.orchestra.run.smithi007.stdout: "events": [ 2024-09-18T08:06:19.294 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:06:19.294 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "initiated", 2024-09-18T08:06:19.294 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:06:19.273865+0000", 2024-09-18T08:06:19.294 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-09-18T08:06:19.294 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:06:19.294 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:06:19.295 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "header_read", 2024-09-18T08:06:19.295 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:06:19.273865+0000", 2024-09-18T08:06:19.295 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-09-18T08:06:19.295 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:06:19.295 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:06:19.295 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "throttled", 2024-09-18T08:06:19.295 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:06:19.273866+0000", 2024-09-18T08:06:19.295 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.9319999999999999e-06 2024-09-18T08:06:19.295 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:06:19.295 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:06:19.295 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "all_read", 2024-09-18T08:06:19.295 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:06:19.273874+0000", 2024-09-18T08:06:19.295 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 7.3180000000000001e-06 2024-09-18T08:06:19.296 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:06:19.296 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:06:19.296 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "dispatched", 2024-09-18T08:06:19.296 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:06:19.273886+0000", 2024-09-18T08:06:19.296 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.2602000000000001e-05 2024-09-18T08:06:19.296 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:06:19.296 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:06:19.296 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "queued_for_pg", 2024-09-18T08:06:19.296 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:06:19.273895+0000", 2024-09-18T08:06:19.296 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 8.2060000000000007e-06 2024-09-18T08:06:19.296 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:06:19.296 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:06:19.296 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-09-18T08:06:19.297 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:06:19.274207+0000", 2024-09-18T08:06:19.297 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.00031224699999999998 2024-09-18T08:06:19.297 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:06:19.297 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:06:19.297 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "started", 2024-09-18T08:06:19.297 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:06:19.274329+0000", 2024-09-18T08:06:19.297 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.000122669 2024-09-18T08:06:19.297 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:06:19.297 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:06:19.297 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "waiting for subops from 2,6,7", 2024-09-18T08:06:19.297 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:06:19.275031+0000", 2024-09-18T08:06:19.297 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.00070100299999999996 2024-09-18T08:06:19.297 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-09-18T08:06:19.297 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-09-18T08:06:19.298 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-09-18T08:06:19.298 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:06:19.298 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:06:19.298 INFO:teuthology.orchestra.run.smithi007.stdout: "description": "osd_op(client.14805.0:159602 7.2 7:4476c972:::benchmark_data_smithi007_73155_object41311:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e133)", 2024-09-18T08:06:19.298 INFO:teuthology.orchestra.run.smithi007.stdout: "initiated_at": "2024-09-18T08:06:19.273913+0000", 2024-09-18T08:06:19.298 INFO:teuthology.orchestra.run.smithi007.stdout: "age": 0.014180580999999999, 2024-09-18T08:06:19.298 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.014285068999999999, 2024-09-18T08:06:19.298 INFO:teuthology.orchestra.run.smithi007.stdout: "continuous": false, 2024-09-18T08:06:19.298 INFO:teuthology.orchestra.run.smithi007.stdout: "type_data": { 2024-09-18T08:06:19.298 INFO:teuthology.orchestra.run.smithi007.stdout: "flag_point": "waiting for sub ops", 2024-09-18T08:06:19.298 INFO:teuthology.orchestra.run.smithi007.stdout: "client_info": { 2024-09-18T08:06:19.298 INFO:teuthology.orchestra.run.smithi007.stdout: "client": "client.14805", 2024-09-18T08:06:19.298 INFO:teuthology.orchestra.run.smithi007.stdout: "client_addr": "172.21.15.7:0/640320436", 2024-09-18T08:06:19.298 INFO:teuthology.orchestra.run.smithi007.stdout: "tid": 159602 2024-09-18T08:06:19.299 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:06:19.299 INFO:teuthology.orchestra.run.smithi007.stdout: "events": [ 2024-09-18T08:06:19.299 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:06:19.299 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "initiated", 2024-09-18T08:06:19.299 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:06:19.273913+0000", 2024-09-18T08:06:19.299 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-09-18T08:06:19.299 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:06:19.299 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:06:19.299 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "header_read", 2024-09-18T08:06:19.299 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:06:19.273913+0000", 2024-09-18T08:06:19.299 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-09-18T08:06:19.299 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:06:19.299 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:06:19.299 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "throttled", 2024-09-18T08:06:19.300 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:06:19.273914+0000", 2024-09-18T08:06:19.300 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 5.5000000000000003e-07 2024-09-18T08:06:19.300 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:06:19.300 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:06:19.300 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "all_read", 2024-09-18T08:06:19.300 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:06:19.273917+0000", 2024-09-18T08:06:19.300 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 2.976e-06 2024-09-18T08:06:19.300 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:06:19.300 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:06:19.300 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "dispatched", 2024-09-18T08:06:19.300 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:06:19.273928+0000", 2024-09-18T08:06:19.300 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.0852999999999999e-05 2024-09-18T08:06:19.300 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:06:19.301 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:06:19.301 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "queued_for_pg", 2024-09-18T08:06:19.301 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:06:19.273949+0000", 2024-09-18T08:06:19.301 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 2.101e-05 2024-09-18T08:06:19.301 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:06:19.301 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:06:19.301 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-09-18T08:06:19.301 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:06:19.276687+0000", 2024-09-18T08:06:19.301 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.0027386799999999998 2024-09-18T08:06:19.301 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:06:19.301 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:06:19.301 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "started", 2024-09-18T08:06:19.301 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:06:19.276811+0000", 2024-09-18T08:06:19.302 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.00012306799999999999 2024-09-18T08:06:19.302 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:06:19.302 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:06:19.302 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "waiting for subops from 1,2", 2024-09-18T08:06:19.302 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:06:19.276937+0000", 2024-09-18T08:06:19.302 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.00012612800000000001 2024-09-18T08:06:19.302 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:06:19.302 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:06:19.302 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "op_commit", 2024-09-18T08:06:19.302 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:06:19.281193+0000", 2024-09-18T08:06:19.302 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.0042561300000000003 2024-09-18T08:06:19.302 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-09-18T08:06:19.302 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-09-18T08:06:19.302 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-09-18T08:06:19.303 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:06:19.303 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:06:19.303 INFO:teuthology.orchestra.run.smithi007.stdout: "description": "osd_op(client.14805.0:159603 7.c 7:333e3821:::benchmark_data_smithi007_73155_object41312:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e133)", 2024-09-18T08:06:19.303 INFO:teuthology.orchestra.run.smithi007.stdout: "initiated_at": "2024-09-18T08:06:19.273966+0000", 2024-09-18T08:06:19.303 INFO:teuthology.orchestra.run.smithi007.stdout: "age": 0.014128056, 2024-09-18T08:06:19.303 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.014298955, 2024-09-18T08:06:19.303 INFO:teuthology.orchestra.run.smithi007.stdout: "continuous": false, 2024-09-18T08:06:19.303 INFO:teuthology.orchestra.run.smithi007.stdout: "type_data": { 2024-09-18T08:06:19.303 INFO:teuthology.orchestra.run.smithi007.stdout: "flag_point": "waiting for sub ops", 2024-09-18T08:06:19.303 INFO:teuthology.orchestra.run.smithi007.stdout: "client_info": { 2024-09-18T08:06:19.303 INFO:teuthology.orchestra.run.smithi007.stdout: "client": "client.14805", 2024-09-18T08:06:19.303 INFO:teuthology.orchestra.run.smithi007.stdout: "client_addr": "172.21.15.7:0/640320436", 2024-09-18T08:06:19.303 INFO:teuthology.orchestra.run.smithi007.stdout: "tid": 159603 2024-09-18T08:06:19.304 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:06:19.304 INFO:teuthology.orchestra.run.smithi007.stdout: "events": [ 2024-09-18T08:06:19.304 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:06:19.304 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "initiated", 2024-09-18T08:06:19.304 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:06:19.273966+0000", 2024-09-18T08:06:19.304 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-09-18T08:06:19.304 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:06:19.304 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:06:19.304 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "header_read", 2024-09-18T08:06:19.304 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:06:19.273966+0000", 2024-09-18T08:06:19.304 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-09-18T08:06:19.304 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:06:19.304 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:06:19.304 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "throttled", 2024-09-18T08:06:19.305 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:06:19.273966+0000", 2024-09-18T08:06:19.305 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 5.6100000000000001e-07 2024-09-18T08:06:19.305 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:06:19.305 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:06:19.305 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "all_read", 2024-09-18T08:06:19.305 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:06:19.273970+0000", 2024-09-18T08:06:19.305 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 3.2820000000000001e-06 2024-09-18T08:06:19.305 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:06:19.305 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:06:19.305 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "dispatched", 2024-09-18T08:06:19.305 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:06:19.273977+0000", 2024-09-18T08:06:19.305 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 6.7909999999999999e-06 2024-09-18T08:06:19.305 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:06:19.306 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:06:19.306 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "queued_for_pg", 2024-09-18T08:06:19.306 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:06:19.273982+0000", 2024-09-18T08:06:19.306 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 5.5079999999999996e-06 2024-09-18T08:06:19.306 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:06:19.306 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:06:19.306 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-09-18T08:06:19.306 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:06:19.278715+0000", 2024-09-18T08:06:19.306 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.0047327760000000002 2024-09-18T08:06:19.306 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:06:19.306 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:06:19.306 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "started", 2024-09-18T08:06:19.306 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:06:19.281266+0000", 2024-09-18T08:06:19.306 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.002551205 2024-09-18T08:06:19.307 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:06:19.307 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:06:19.307 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "waiting for subops from 2,6,7", 2024-09-18T08:06:19.307 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:06:19.281395+0000", 2024-09-18T08:06:19.307 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.00012879699999999999 2024-09-18T08:06:19.307 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-09-18T08:06:19.307 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-09-18T08:06:19.307 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-09-18T08:06:19.307 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:06:19.307 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:06:19.307 INFO:teuthology.orchestra.run.smithi007.stdout: "description": "osd_op(client.14805.0:159606 7.3 7:c68d1091:::benchmark_data_smithi007_73155_object41315:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e133)", 2024-09-18T08:06:19.307 INFO:teuthology.orchestra.run.smithi007.stdout: "initiated_at": "2024-09-18T08:06:19.273990+0000", 2024-09-18T08:06:19.307 INFO:teuthology.orchestra.run.smithi007.stdout: "age": 0.014103920000000001, 2024-09-18T08:06:19.307 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.014335517000000001, 2024-09-18T08:06:19.308 INFO:teuthology.orchestra.run.smithi007.stdout: "continuous": false, 2024-09-18T08:06:19.308 INFO:teuthology.orchestra.run.smithi007.stdout: "type_data": { 2024-09-18T08:06:19.308 INFO:teuthology.orchestra.run.smithi007.stdout: "flag_point": "waiting for sub ops", 2024-09-18T08:06:19.308 INFO:teuthology.orchestra.run.smithi007.stdout: "client_info": { 2024-09-18T08:06:19.308 INFO:teuthology.orchestra.run.smithi007.stdout: "client": "client.14805", 2024-09-18T08:06:19.308 INFO:teuthology.orchestra.run.smithi007.stdout: "client_addr": "172.21.15.7:0/640320436", 2024-09-18T08:06:19.308 INFO:teuthology.orchestra.run.smithi007.stdout: "tid": 159606 2024-09-18T08:06:19.308 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:06:19.308 INFO:teuthology.orchestra.run.smithi007.stdout: "events": [ 2024-09-18T08:06:19.308 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:06:19.308 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "initiated", 2024-09-18T08:06:19.308 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:06:19.273990+0000", 2024-09-18T08:06:19.308 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-09-18T08:06:19.308 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:06:19.309 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:06:19.309 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "header_read", 2024-09-18T08:06:19.309 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:06:19.273990+0000", 2024-09-18T08:06:19.309 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-09-18T08:06:19.309 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:06:19.309 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:06:19.309 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "throttled", 2024-09-18T08:06:19.309 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:06:19.273990+0000", 2024-09-18T08:06:19.309 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 4.1399999999999997e-07 2024-09-18T08:06:19.309 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:06:19.309 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:06:19.309 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "all_read", 2024-09-18T08:06:19.309 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:06:19.273993+0000", 2024-09-18T08:06:19.309 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 2.1330000000000002e-06 2024-09-18T08:06:19.310 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:06:19.310 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:06:19.310 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "dispatched", 2024-09-18T08:06:19.310 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:06:19.273998+0000", 2024-09-18T08:06:19.310 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 5.6289999999999998e-06 2024-09-18T08:06:19.310 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:06:19.310 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:06:19.310 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "queued_for_pg", 2024-09-18T08:06:19.310 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:06:19.274004+0000", 2024-09-18T08:06:19.310 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 5.3800000000000002e-06 2024-09-18T08:06:19.310 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:06:19.310 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:06:19.310 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-09-18T08:06:19.310 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:06:19.274059+0000", 2024-09-18T08:06:19.310 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 5.5676e-05 2024-09-18T08:06:19.311 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:06:19.311 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:06:19.311 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "started", 2024-09-18T08:06:19.311 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:06:19.274182+0000", 2024-09-18T08:06:19.311 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.00012286999999999999 2024-09-18T08:06:19.311 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:06:19.311 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:06:19.311 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "waiting for subops from 2,4", 2024-09-18T08:06:19.311 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:06:19.274312+0000", 2024-09-18T08:06:19.311 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.00012998 2024-09-18T08:06:19.311 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:06:19.311 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:06:19.311 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:06:19.311 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:06:19.280956+0000", 2024-09-18T08:06:19.312 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.0066440900000000001 2024-09-18T08:06:19.312 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:06:19.312 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:06:19.312 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "op_commit", 2024-09-18T08:06:19.312 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:06:19.281106+0000", 2024-09-18T08:06:19.312 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.00014930299999999999 2024-09-18T08:06:19.312 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-09-18T08:06:19.312 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-09-18T08:06:19.312 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-09-18T08:06:19.312 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:06:19.312 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:06:19.312 INFO:teuthology.orchestra.run.smithi007.stdout: "description": "osd_repop_reply(client.14805.0:159598 7.c e133/130)", 2024-09-18T08:06:19.312 INFO:teuthology.orchestra.run.smithi007.stdout: "initiated_at": "2024-09-18T08:06:19.278945+0000", 2024-09-18T08:06:19.312 INFO:teuthology.orchestra.run.smithi007.stdout: "age": 0.0091486899999999993, 2024-09-18T08:06:19.313 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.0094435470000000001, 2024-09-18T08:06:19.313 INFO:teuthology.orchestra.run.smithi007.stdout: "continuous": false, 2024-09-18T08:06:19.313 INFO:teuthology.orchestra.run.smithi007.stdout: "type_data": { 2024-09-18T08:06:19.313 INFO:teuthology.orchestra.run.smithi007.stdout: "flag_point": "queued for pg", 2024-09-18T08:06:19.313 INFO:teuthology.orchestra.run.smithi007.stdout: "events": [ 2024-09-18T08:06:19.313 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:06:19.313 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "initiated", 2024-09-18T08:06:19.313 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:06:19.278945+0000", 2024-09-18T08:06:19.313 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-09-18T08:06:19.313 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:06:19.313 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:06:19.313 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "header_read", 2024-09-18T08:06:19.313 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:06:19.278945+0000", 2024-09-18T08:06:19.314 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-09-18T08:06:19.314 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:06:19.314 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:06:19.314 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "throttled", 2024-09-18T08:06:19.314 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:06:19.278946+0000", 2024-09-18T08:06:19.314 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 9.6800000000000009e-07 2024-09-18T08:06:19.314 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:06:19.314 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:06:19.314 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "all_read", 2024-09-18T08:06:19.314 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:06:19.278950+0000", 2024-09-18T08:06:19.314 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 3.7519999999999998e-06 2024-09-18T08:06:19.314 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:06:19.314 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:06:19.314 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "dispatched", 2024-09-18T08:06:19.315 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:06:19.278959+0000", 2024-09-18T08:06:19.315 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 8.8470000000000002e-06 2024-09-18T08:06:19.315 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:06:19.315 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:06:19.315 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "queued_for_pg", 2024-09-18T08:06:19.315 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:06:19.278966+0000", 2024-09-18T08:06:19.315 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 7.3289999999999996e-06 2024-09-18T08:06:19.315 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-09-18T08:06:19.315 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-09-18T08:06:19.315 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-09-18T08:06:19.315 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:06:19.315 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:06:19.315 INFO:teuthology.orchestra.run.smithi007.stdout: "description": "osd_repop_reply(client.14805.0:159598 7.c e133/130)", 2024-09-18T08:06:19.315 INFO:teuthology.orchestra.run.smithi007.stdout: "initiated_at": "2024-09-18T08:06:19.280636+0000", 2024-09-18T08:06:19.315 INFO:teuthology.orchestra.run.smithi007.stdout: "age": 0.0074581630000000003, 2024-09-18T08:06:19.316 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.0077886689999999998, 2024-09-18T08:06:19.316 INFO:teuthology.orchestra.run.smithi007.stdout: "continuous": false, 2024-09-18T08:06:19.316 INFO:teuthology.orchestra.run.smithi007.stdout: "type_data": { 2024-09-18T08:06:19.316 INFO:teuthology.orchestra.run.smithi007.stdout: "flag_point": "queued for pg", 2024-09-18T08:06:19.316 INFO:teuthology.orchestra.run.smithi007.stdout: "events": [ 2024-09-18T08:06:19.316 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:06:19.316 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "initiated", 2024-09-18T08:06:19.316 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:06:19.280636+0000", 2024-09-18T08:06:19.316 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-09-18T08:06:19.316 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:06:19.316 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:06:19.316 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "header_read", 2024-09-18T08:06:19.316 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:06:19.280636+0000", 2024-09-18T08:06:19.316 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-09-18T08:06:19.317 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:06:19.317 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:06:19.317 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "throttled", 2024-09-18T08:06:19.317 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:06:19.280636+0000", 2024-09-18T08:06:19.317 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 3.6899999999999998e-07 2024-09-18T08:06:19.317 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:06:19.317 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:06:19.317 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "all_read", 2024-09-18T08:06:19.317 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:06:19.280640+0000", 2024-09-18T08:06:19.317 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 3.8070000000000001e-06 2024-09-18T08:06:19.317 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:06:19.317 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:06:19.317 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "dispatched", 2024-09-18T08:06:19.317 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:06:19.280647+0000", 2024-09-18T08:06:19.318 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 7.1319999999999996e-06 2024-09-18T08:06:19.318 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:06:19.318 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:06:19.318 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "queued_for_pg", 2024-09-18T08:06:19.318 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:06:19.280653+0000", 2024-09-18T08:06:19.318 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 5.7219999999999996e-06 2024-09-18T08:06:19.318 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-09-18T08:06:19.318 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-09-18T08:06:19.318 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-09-18T08:06:19.318 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:06:19.318 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:06:19.318 INFO:teuthology.orchestra.run.smithi007.stdout: "description": "osd_repop_reply(client.14805.0:159598 7.c e133/130)", 2024-09-18T08:06:19.318 INFO:teuthology.orchestra.run.smithi007.stdout: "initiated_at": "2024-09-18T08:06:19.280670+0000", 2024-09-18T08:06:19.318 INFO:teuthology.orchestra.run.smithi007.stdout: "age": 0.0074235179999999996, 2024-09-18T08:06:19.319 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.0077890670000000002, 2024-09-18T08:06:19.319 INFO:teuthology.orchestra.run.smithi007.stdout: "continuous": false, 2024-09-18T08:06:19.319 INFO:teuthology.orchestra.run.smithi007.stdout: "type_data": { 2024-09-18T08:06:19.319 INFO:teuthology.orchestra.run.smithi007.stdout: "flag_point": "queued for pg", 2024-09-18T08:06:19.319 INFO:teuthology.orchestra.run.smithi007.stdout: "events": [ 2024-09-18T08:06:19.319 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:06:19.319 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "initiated", 2024-09-18T08:06:19.319 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:06:19.280670+0000", 2024-09-18T08:06:19.319 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-09-18T08:06:19.319 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:06:19.319 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:06:19.319 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "header_read", 2024-09-18T08:06:19.319 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:06:19.280670+0000", 2024-09-18T08:06:19.319 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-09-18T08:06:19.320 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:06:19.320 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:06:19.320 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "throttled", 2024-09-18T08:06:19.320 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:06:19.280671+0000", 2024-09-18T08:06:19.320 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 3.4900000000000001e-07 2024-09-18T08:06:19.320 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:06:19.320 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:06:19.320 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "all_read", 2024-09-18T08:06:19.320 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:06:19.280673+0000", 2024-09-18T08:06:19.320 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 2.379e-06 2024-09-18T08:06:19.320 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:06:19.320 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:06:19.320 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "dispatched", 2024-09-18T08:06:19.320 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:06:19.280679+0000", 2024-09-18T08:06:19.320 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 5.4040000000000003e-06 2024-09-18T08:06:19.321 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:06:19.321 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:06:19.321 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "queued_for_pg", 2024-09-18T08:06:19.321 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:06:19.280683+0000", 2024-09-18T08:06:19.322 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 4.5700000000000003e-06 2024-09-18T08:06:19.322 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-09-18T08:06:19.322 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-09-18T08:06:19.322 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-09-18T08:06:19.322 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:06:19.322 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:06:19.322 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3 2024-09-18T08:06:19.324 INFO:teuthology.orchestra.run.smithi007.stdout: "description": "osd_op(client.14805.0:159609 7.a 7.9ee291aa (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e133)", 2024-09-18T08:06:19.324 INFO:teuthology.orchestra.run.smithi007.stdout: "initiated_at": "2024-09-18T08:06:19.287814+0000", 2024-09-18T08:06:19.324 INFO:teuthology.orchestra.run.smithi007.stdout: "age": 0.00027995400000000001, 2024-09-18T08:06:19.324 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.00068477100000000003, 2024-09-18T08:06:19.324 INFO:teuthology.orchestra.run.smithi007.stdout: "continuous": false, 2024-09-18T08:06:19.324 INFO:teuthology.orchestra.run.smithi007.stdout: "type_data": { 2024-09-18T08:06:19.324 INFO:teuthology.orchestra.run.smithi007.stdout: "flag_point": "queued for pg", 2024-09-18T08:06:19.324 INFO:teuthology.orchestra.run.smithi007.stdout: "client_info": { 2024-09-18T08:06:19.324 INFO:teuthology.orchestra.run.smithi007.stdout: "client": "client.14805", 2024-09-18T08:06:19.324 INFO:teuthology.orchestra.run.smithi007.stdout: "client_addr": "172.21.15.7:0/640320436", 2024-09-18T08:06:19.325 INFO:teuthology.orchestra.run.smithi007.stdout: "tid": 159609 2024-09-18T08:06:19.325 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:06:19.325 INFO:teuthology.orchestra.run.smithi007.stdout: "events": [ 2024-09-18T08:06:19.325 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:06:19.325 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "initiated", 2024-09-18T08:06:19.325 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:06:19.287814+0000", 2024-09-18T08:06:19.325 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-09-18T08:06:19.325 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:06:19.325 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:06:19.325 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "header_read", 2024-09-18T08:06:19.325 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:06:19.287814+0000", 2024-09-18T08:06:19.325 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-09-18T08:06:19.325 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:06:19.325 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:06:19.326 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "throttled", 2024-09-18T08:06:19.326 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:06:19.287816+0000", 2024-09-18T08:06:19.326 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.8920000000000001e-06 2024-09-18T08:06:19.326 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:06:19.326 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:06:19.326 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "all_read", 2024-09-18T08:06:19.326 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:06:19.287824+0000", 2024-09-18T08:06:19.326 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 7.6879999999999999e-06 2024-09-18T08:06:19.326 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:06:19.326 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:06:19.326 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "dispatched", 2024-09-18T08:06:19.326 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:06:19.287843+0000", 2024-09-18T08:06:19.326 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.9647999999999999e-05 2024-09-18T08:06:19.326 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:06:19.327 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:06:19.327 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "queued_for_pg", 2024-09-18T08:06:19.327 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:06:19.287861+0000", 2024-09-18T08:06:19.327 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.7561e-05 2024-09-18T08:06:19.327 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-09-18T08:06:19.327 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-09-18T08:06:19.327 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-09-18T08:06:19.327 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:06:19.327 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:06:19.327 INFO:teuthology.orchestra.run.smithi007.stdout: "description": "osd_op(client.14805.0:159610 7.3 7.349c73 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e133)", 2024-09-18T08:06:19.327 INFO:teuthology.orchestra.run.smithi007.stdout: "initiated_at": "2024-09-18T08:06:19.287886+0000", 2024-09-18T08:06:19.327 INFO:teuthology.orchestra.run.smithi007.stdout: "age": 0.00020788800000000001, 2024-09-18T08:06:19.327 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.000654994, 2024-09-18T08:06:19.327 INFO:teuthology.orchestra.run.smithi007.stdout: "continuous": false, 2024-09-18T08:06:19.328 INFO:teuthology.orchestra.run.smithi007.stdout: "type_data": { 2024-09-18T08:06:19.328 INFO:teuthology.orchestra.run.smithi007.stdout: "flag_point": "queued for pg", 2024-09-18T08:06:19.328 INFO:teuthology.orchestra.run.smithi007.stdout: "client_info": { 2024-09-18T08:06:19.328 INFO:teuthology.orchestra.run.smithi007.stdout: "client": "client.14805", 2024-09-18T08:06:19.328 INFO:teuthology.orchestra.run.smithi007.stdout: "client_addr": "172.21.15.7:0/640320436", 2024-09-18T08:06:19.328 INFO:teuthology.orchestra.run.smithi007.stdout: "tid": 159610 2024-09-18T08:06:19.328 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:06:19.328 INFO:teuthology.orchestra.run.smithi007.stdout: "events": [ 2024-09-18T08:06:19.328 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:06:19.328 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "initiated", 2024-09-18T08:06:19.328 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:06:19.287886+0000", 2024-09-18T08:06:19.328 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-09-18T08:06:19.328 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:06:19.328 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:06:19.329 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "header_read", 2024-09-18T08:06:19.329 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:06:19.287886+0000", 2024-09-18T08:06:19.329 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-09-18T08:06:19.329 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:06:19.329 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:06:19.329 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "throttled", 2024-09-18T08:06:19.329 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:06:19.287887+0000", 2024-09-18T08:06:19.329 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 5.4000000000000002e-07 2024-09-18T08:06:19.329 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:06:19.329 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:06:19.329 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "all_read", 2024-09-18T08:06:19.329 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:06:19.287889+0000", 2024-09-18T08:06:19.329 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 2.779e-06 2024-09-18T08:06:19.330 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:06:19.330 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:06:19.330 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "dispatched", 2024-09-18T08:06:19.330 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:06:19.287895+0000", 2024-09-18T08:06:19.330 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 6.0240000000000003e-06 2024-09-18T08:06:19.330 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:06:19.330 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:06:19.330 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "queued_for_pg", 2024-09-18T08:06:19.330 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:06:19.287901+0000", 2024-09-18T08:06:19.330 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 5.1749999999999996e-06 2024-09-18T08:06:19.330 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-09-18T08:06:19.330 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-09-18T08:06:19.330 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-09-18T08:06:19.330 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:06:19.331 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:06:19.331 INFO:teuthology.orchestra.run.smithi007.stdout: "description": "osd_repop_reply(client.14805.0:159603 7.c e133/130)", 2024-09-18T08:06:19.331 INFO:teuthology.orchestra.run.smithi007.stdout: "initiated_at": "2024-09-18T08:06:19.287946+0000", 2024-09-18T08:06:19.331 INFO:teuthology.orchestra.run.smithi007.stdout: "age": 0.00014762799999999999, 2024-09-18T08:06:19.331 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.00063558400000000002, 2024-09-18T08:06:19.331 INFO:teuthology.orchestra.run.smithi007.stdout: "continuous": false, 2024-09-18T08:06:19.331 INFO:teuthology.orchestra.run.smithi007.stdout: "type_data": { 2024-09-18T08:06:19.331 INFO:teuthology.orchestra.run.smithi007.stdout: "flag_point": "queued for pg", 2024-09-18T08:06:19.331 INFO:teuthology.orchestra.run.smithi007.stdout: "events": [ 2024-09-18T08:06:19.331 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:06:19.331 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "initiated", 2024-09-18T08:06:19.331 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:06:19.287946+0000", 2024-09-18T08:06:19.331 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-09-18T08:06:19.331 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:06:19.332 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:06:19.332 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "header_read", 2024-09-18T08:06:19.332 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:06:19.287946+0000", 2024-09-18T08:06:19.332 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-09-18T08:06:19.332 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:06:19.332 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:06:19.332 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "throttled", 2024-09-18T08:06:19.332 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:06:19.287947+0000", 2024-09-18T08:06:19.332 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 6.7899999999999998e-07 2024-09-18T08:06:19.332 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:06:19.332 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:06:19.332 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "all_read", 2024-09-18T08:06:19.332 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:06:19.287950+0000", 2024-09-18T08:06:19.333 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 2.9500000000000001e-06 2024-09-18T08:06:19.333 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:06:19.333 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:06:19.333 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "dispatched", 2024-09-18T08:06:19.333 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:06:19.287958+0000", 2024-09-18T08:06:19.333 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 7.8990000000000001e-06 2024-09-18T08:06:19.333 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:06:19.333 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:06:19.333 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "queued_for_pg", 2024-09-18T08:06:19.333 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:06:19.287963+0000", 2024-09-18T08:06:19.333 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 5.2850000000000002e-06 2024-09-18T08:06:19.333 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-09-18T08:06:19.333 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-09-18T08:06:19.334 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-09-18T08:06:19.334 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:06:19.334 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:06:19.334 INFO:teuthology.orchestra.run.smithi007.stdout: "description": "osd_repop_reply(client.14805.0:159603 7.c e133/130)", 2024-09-18T08:06:19.334 INFO:teuthology.orchestra.run.smithi007.stdout: "initiated_at": "2024-09-18T08:06:19.287991+0000", 2024-09-18T08:06:19.334 INFO:teuthology.orchestra.run.smithi007.stdout: "age": 0.000102675, 2024-09-18T08:06:19.334 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.00062565699999999995, 2024-09-18T08:06:19.334 INFO:teuthology.orchestra.run.smithi007.stdout: "continuous": false, 2024-09-18T08:06:19.334 INFO:teuthology.orchestra.run.smithi007.stdout: "type_data": { 2024-09-18T08:06:19.334 INFO:teuthology.orchestra.run.smithi007.stdout: "flag_point": "queued for pg", 2024-09-18T08:06:19.334 INFO:teuthology.orchestra.run.smithi007.stdout: "events": [ 2024-09-18T08:06:19.335 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:06:19.335 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "initiated", 2024-09-18T08:06:19.335 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:06:19.287991+0000", 2024-09-18T08:06:19.335 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-09-18T08:06:19.335 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:06:19.335 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:06:19.335 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "header_read", 2024-09-18T08:06:19.335 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:06:19.287991+0000", 2024-09-18T08:06:19.335 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-09-18T08:06:19.335 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:06:19.335 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:06:19.335 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "throttled", 2024-09-18T08:06:19.335 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:06:19.287992+0000", 2024-09-18T08:06:19.336 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 4.2399999999999999e-07 2024-09-18T08:06:19.336 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:06:19.336 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:06:19.336 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "all_read", 2024-09-18T08:06:19.336 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:06:19.287994+0000", 2024-09-18T08:06:19.336 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 2.0870000000000001e-06 2024-09-18T08:06:19.336 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:06:19.336 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:06:19.336 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "dispatched", 2024-09-18T08:06:19.336 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:06:19.287999+0000", 2024-09-18T08:06:19.336 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 5.6570000000000002e-06 2024-09-18T08:06:19.336 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:06:19.336 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:06:19.336 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "queued_for_pg", 2024-09-18T08:06:19.337 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:06:19.288004+0000", 2024-09-18T08:06:19.337 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 4.8319999999999996e-06 2024-09-18T08:06:19.337 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-09-18T08:06:19.337 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-09-18T08:06:19.337 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-09-18T08:06:19.337 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:06:19.337 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:06:19.337 INFO:teuthology.orchestra.run.smithi007.stdout: "description": "osd_repop(client.14805.0:159605 7.d e133/130 7:b3e3c7d0:::benchmark_data_smithi007_73155_object41314:head v 133'10001, mlcod=133'9999)", 2024-09-18T08:06:19.337 INFO:teuthology.orchestra.run.smithi007.stdout: "initiated_at": "2024-09-18T08:06:19.288029+0000", 2024-09-18T08:06:19.337 INFO:teuthology.orchestra.run.smithi007.stdout: "age": 6.4997000000000001e-05, 2024-09-18T08:06:19.337 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.00062688700000000002, 2024-09-18T08:06:19.337 INFO:teuthology.orchestra.run.smithi007.stdout: "continuous": false, 2024-09-18T08:06:19.337 INFO:teuthology.orchestra.run.smithi007.stdout: "type_data": { 2024-09-18T08:06:19.337 INFO:teuthology.orchestra.run.smithi007.stdout: "flag_point": "started", 2024-09-18T08:06:19.338 INFO:teuthology.orchestra.run.smithi007.stdout: "events": [ 2024-09-18T08:06:19.338 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:06:19.338 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "initiated", 2024-09-18T08:06:19.338 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:06:19.288029+0000", 2024-09-18T08:06:19.338 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-09-18T08:06:19.338 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:06:19.338 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:06:19.338 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "header_read", 2024-09-18T08:06:19.338 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:06:19.288029+0000", 2024-09-18T08:06:19.338 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-09-18T08:06:19.338 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:06:19.338 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:06:19.338 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "throttled", 2024-09-18T08:06:19.338 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:06:19.288029+0000", 2024-09-18T08:06:19.338 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 3.7099999999999997e-07 2024-09-18T08:06:19.339 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:06:19.339 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:06:19.339 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "all_read", 2024-09-18T08:06:19.339 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:06:19.288033+0000", 2024-09-18T08:06:19.339 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 3.3160000000000001e-06 2024-09-18T08:06:19.339 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:06:19.339 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:06:19.339 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "dispatched", 2024-09-18T08:06:19.339 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:06:19.288039+0000", 2024-09-18T08:06:19.339 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 6.5490000000000003e-06 2024-09-18T08:06:19.339 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:06:19.339 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:06:19.339 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "queued_for_pg", 2024-09-18T08:06:19.339 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:06:19.288044+0000", 2024-09-18T08:06:19.340 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 5.096e-06 2024-09-18T08:06:19.340 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:06:19.340 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:06:19.340 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-09-18T08:06:19.340 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:06:19.288172+0000", 2024-09-18T08:06:19.340 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.00012798599999999999 2024-09-18T08:06:19.340 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:06:19.340 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:06:19.340 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "started", 2024-09-18T08:06:19.340 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:06:19.288194+0000", 2024-09-18T08:06:19.340 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 2.1980999999999998e-05 2024-09-18T08:06:19.340 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-09-18T08:06:19.340 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-09-18T08:06:19.341 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-09-18T08:06:19.341 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:06:19.341 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:06:19.341 INFO:teuthology.orchestra.run.smithi007.stdout: "description": "osd_repop(client.14805.0:159611 7.1 e133/130)", 2024-09-18T08:06:19.341 INFO:teuthology.orchestra.run.smithi007.stdout: "initiated_at": "2024-09-18T08:06:19.288065+0000", 2024-09-18T08:06:19.341 INFO:teuthology.orchestra.run.smithi007.stdout: "age": 2.8707000000000001e-05, 2024-09-18T08:06:19.341 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.000635638, 2024-09-18T08:06:19.341 INFO:teuthology.orchestra.run.smithi007.stdout: "continuous": false, 2024-09-18T08:06:19.341 INFO:teuthology.orchestra.run.smithi007.stdout: "type_data": { 2024-09-18T08:06:19.341 INFO:teuthology.orchestra.run.smithi007.stdout: "flag_point": "queued for pg", 2024-09-18T08:06:19.341 INFO:teuthology.orchestra.run.smithi007.stdout: "events": [ 2024-09-18T08:06:19.341 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:06:19.341 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "initiated", 2024-09-18T08:06:19.341 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:06:19.288065+0000", 2024-09-18T08:06:19.342 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-09-18T08:06:19.342 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:06:19.342 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:06:19.342 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "header_read", 2024-09-18T08:06:19.342 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:06:19.288065+0000", 2024-09-18T08:06:19.342 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-09-18T08:06:19.342 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:06:19.342 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:06:19.342 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "throttled", 2024-09-18T08:06:19.342 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:06:19.288066+0000", 2024-09-18T08:06:19.342 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 3.0800000000000001e-07 2024-09-18T08:06:19.342 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:06:19.342 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:06:19.342 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "all_read", 2024-09-18T08:06:19.343 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:06:19.288069+0000", 2024-09-18T08:06:19.343 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 3.1070000000000001e-06 2024-09-18T08:06:19.343 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:06:19.343 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:06:19.343 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "dispatched", 2024-09-18T08:06:19.343 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:06:19.288074+0000", 2024-09-18T08:06:19.343 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 5.8000000000000004e-06 2024-09-18T08:06:19.343 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:06:19.343 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:06:19.343 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "queued_for_pg", 2024-09-18T08:06:19.343 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:06:19.288079+0000", 2024-09-18T08:06:19.343 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 4.5469999999999998e-06 2024-09-18T08:06:19.343 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-09-18T08:06:19.343 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-09-18T08:06:19.344 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-09-18T08:06:19.344 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-09-18T08:06:19.344 INFO:teuthology.orchestra.run.smithi007.stdout: ], 2024-09-18T08:06:19.344 INFO:teuthology.orchestra.run.smithi007.stdout: "num_ops": 13 2024-09-18T08:06:19.344 INFO:teuthology.orchestra.run.smithi007.stdout:} 2024-09-18T08:06:19.392 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:06:19.393 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:06:19.393 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3' 2024-09-18T08:06:19.496 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3 2024-09-18T08:06:19.547 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:06:19.548 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:06:19.548 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3' 2024-09-18T08:06:19.649 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4 2024-09-18T08:06:19.678 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:06:19.679 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:06:19.679 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4' 2024-09-18T08:06:19.687 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/779349ca-7593-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T08:06:19.780 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4 2024-09-18T08:06:19.811 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:06:19.811 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:06:19.812 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4' 2024-09-18T08:06:19.913 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0 2024-09-18T08:06:19.960 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:06:19.960 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:06:19.961 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0' 2024-09-18T08:06:20.062 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0 2024-09-18T08:06:20.098 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:06:20.099 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:06:20.100 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0' 2024-09-18T08:06:20.200 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1 2024-09-18T08:06:20.214 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid 779349ca-7593-11ef-bceb-c7b262605968 -- ceph daemon osd.3 dump_blocked_ops 2024-09-18T08:06:20.237 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:06:20.238 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:06:20.238 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1' 2024-09-18T08:06:20.339 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0 2024-09-18T08:06:20.426 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:06:20.426 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:06:20.427 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0' 2024-09-18T08:06:20.528 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2 2024-09-18T08:06:20.602 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:06:20.602 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:06:20.603 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2' 2024-09-18T08:06:20.689 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid 779349ca-7593-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-18T08:06:20.704 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5 2024-09-18T08:06:20.731 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:06:20 smithi007 ceph-mon[29459]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-18T08:06:20.731 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:06:20 smithi007 ceph-mon[29459]: pgmap v154: 148 pgs: 3 active+remapped+backfilling, 2 active+remapped+backfill_toofull, 143 active+clean; 4.8 GiB data, 14 GiB used, 612 GiB / 626 GiB avail; 853 B/s rd, 0 B/s wr, 1.08k op/s; 23198/234378 objects misplaced (9.898%); 12 MiB/s, 199 objects/s recovering 2024-09-18T08:06:20.731 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:06:20 smithi007 ceph-mon[29459]: 5.e scrub starts 2024-09-18T08:06:20.732 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:06:20 smithi007 ceph-mon[29459]: 5.e scrub ok 2024-09-18T08:06:20.732 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:06:20 smithi007 ceph-mon[29459]: from='mgr.24470 172.21.15.7:0/2594390261' entity='mgr.y' 2024-09-18T08:06:20.732 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:06:20 smithi007 ceph-mon[29459]: from='mgr.24470 172.21.15.7:0/2594390261' entity='mgr.y' 2024-09-18T08:06:20.732 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:06:20 smithi007 ceph-mon[29459]: purged_snaps scrub starts 2024-09-18T08:06:20.732 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:06:20 smithi007 ceph-mon[29459]: purged_snaps scrub ok 2024-09-18T08:06:20.732 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:06:20 smithi007 ceph-mon[29459]: Health check failed: Low space hindering backfill (add storage if this doesn't resolve itself): 2 pgs backfill_toofull (PG_BACKFILL_FULL) 2024-09-18T08:06:20.733 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:06:20 smithi007 ceph-mon[39249]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-18T08:06:20.733 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:06:20 smithi007 ceph-mon[39249]: pgmap v154: 148 pgs: 3 active+remapped+backfilling, 2 active+remapped+backfill_toofull, 143 active+clean; 4.8 GiB data, 14 GiB used, 612 GiB / 626 GiB avail; 853 B/s rd, 0 B/s wr, 1.08k op/s; 23198/234378 objects misplaced (9.898%); 12 MiB/s, 199 objects/s recovering 2024-09-18T08:06:20.733 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:06:20 smithi007 ceph-mon[39249]: 5.e scrub starts 2024-09-18T08:06:20.733 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:06:20 smithi007 ceph-mon[39249]: 5.e scrub ok 2024-09-18T08:06:20.733 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:06:20 smithi007 ceph-mon[39249]: from='mgr.24470 172.21.15.7:0/2594390261' entity='mgr.y' 2024-09-18T08:06:20.733 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:06:20 smithi007 ceph-mon[39249]: from='mgr.24470 172.21.15.7:0/2594390261' entity='mgr.y' 2024-09-18T08:06:20.733 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:06:20 smithi007 ceph-mon[39249]: purged_snaps scrub starts 2024-09-18T08:06:20.733 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:06:20 smithi007 ceph-mon[39249]: purged_snaps scrub ok 2024-09-18T08:06:20.734 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:06:20 smithi007 ceph-mon[39249]: Health check failed: Low space hindering backfill (add storage if this doesn't resolve itself): 2 pgs backfill_toofull (PG_BACKFILL_FULL) 2024-09-18T08:06:20.734 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:06:20.735 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:06:20.735 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5' 2024-09-18T08:06:20.836 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2 2024-09-18T08:06:20.906 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:06:20 smithi050 ceph-mon[32371]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-18T08:06:20.906 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:06:20 smithi050 ceph-mon[32371]: pgmap v154: 148 pgs: 3 active+remapped+backfilling, 2 active+remapped+backfill_toofull, 143 active+clean; 4.8 GiB data, 14 GiB used, 612 GiB / 626 GiB avail; 853 B/s rd, 0 B/s wr, 1.08k op/s; 23198/234378 objects misplaced (9.898%); 12 MiB/s, 199 objects/s recovering 2024-09-18T08:06:20.906 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:06:20 smithi050 ceph-mon[32371]: 5.e scrub starts 2024-09-18T08:06:20.906 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:06:20 smithi050 ceph-mon[32371]: 5.e scrub ok 2024-09-18T08:06:20.906 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:06:20 smithi050 ceph-mon[32371]: from='mgr.24470 172.21.15.7:0/2594390261' entity='mgr.y' 2024-09-18T08:06:20.906 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:06:20 smithi050 ceph-mon[32371]: from='mgr.24470 172.21.15.7:0/2594390261' entity='mgr.y' 2024-09-18T08:06:20.906 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:06:20 smithi050 ceph-mon[32371]: purged_snaps scrub starts 2024-09-18T08:06:20.906 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:06:20 smithi050 ceph-mon[32371]: purged_snaps scrub ok 2024-09-18T08:06:20.906 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:06:20 smithi050 ceph-mon[32371]: Health check failed: Low space hindering backfill (add storage if this doesn't resolve itself): 2 pgs backfill_toofull (PG_BACKFILL_FULL) 2024-09-18T08:06:20.907 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:06:20.908 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:06:20.908 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2' 2024-09-18T08:06:21.009 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2 2024-09-18T08:06:21.039 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:06:21.040 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:06:21.040 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2' 2024-09-18T08:06:21.141 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3 2024-09-18T08:06:21.179 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:06:21.180 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:06:21.180 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3' 2024-09-18T08:06:21.281 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6 2024-09-18T08:06:21.306 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:06:21.306 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:06:21.307 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6' 2024-09-18T08:06:21.407 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2 2024-09-18T08:06:21.454 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:06:21.455 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:06:21.455 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2' 2024-09-18T08:06:21.556 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4 2024-09-18T08:06:21.583 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:06:21.584 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:06:21.584 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4' 2024-09-18T08:06:21.625 INFO:teuthology.orchestra.run.smithi007.stderr:noscrub is set 2024-09-18T08:06:21.685 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2 2024-09-18T08:06:21.727 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:06:21.728 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:06:21.729 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2' 2024-09-18T08:06:21.830 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5 2024-09-18T08:06:21.854 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:06:21.855 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:06:21.856 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5' 2024-09-18T08:06:21.906 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:06:21 smithi050 ceph-mon[32371]: osdmap e134: 8 total, 8 up, 7 in 2024-09-18T08:06:21.906 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:06:21 smithi050 ceph-mon[32371]: from='client.? 172.21.15.7:0/3307807491' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-18T08:06:21.906 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:06:21 smithi050 ceph-mon[32371]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-18T08:06:21.957 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7 2024-09-18T08:06:21.980 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:06:21 smithi007 ceph-mon[39249]: osdmap e134: 8 total, 8 up, 7 in 2024-09-18T08:06:21.980 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:06:21 smithi007 ceph-mon[39249]: from='client.? 172.21.15.7:0/3307807491' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-18T08:06:21.981 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:06:21 smithi007 ceph-mon[39249]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-18T08:06:21.981 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:06:21 smithi007 ceph-mon[29459]: osdmap e134: 8 total, 8 up, 7 in 2024-09-18T08:06:21.981 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:06:21 smithi007 ceph-mon[29459]: from='client.? 172.21.15.7:0/3307807491' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-18T08:06:21.981 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:06:21 smithi007 ceph-mon[29459]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-18T08:06:21.985 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:06:21.986 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:06:21.986 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7' 2024-09-18T08:06:22.087 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3 2024-09-18T08:06:22.140 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:06:22.141 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:06:22.141 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3' 2024-09-18T08:06:22.242 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3 2024-09-18T08:06:22.311 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:06:22.312 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:06:22.312 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3' 2024-09-18T08:06:22.413 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1 2024-09-18T08:06:22.471 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:06:22.472 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:06:22.473 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1' 2024-09-18T08:06:22.574 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3 2024-09-18T08:06:22.616 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:06:22.617 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:06:22.618 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3' 2024-09-18T08:06:22.684 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:06:22 smithi007 ceph-mon[29459]: pgmap v156: 148 pgs: 3 active+remapped+backfilling, 2 active+remapped+backfill_toofull, 143 active+clean; 4.8 GiB data, 14 GiB used, 612 GiB / 626 GiB avail; 777 B/s rd, 0 B/s wr, 580 op/s; 23198/234378 objects misplaced (9.898%); 11 MiB/s, 178 objects/s recovering 2024-09-18T08:06:22.684 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:06:22 smithi007 ceph-mon[29459]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-18T08:06:22.684 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:06:22 smithi007 ceph-mon[29459]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-18T08:06:22.684 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:06:22 smithi007 ceph-mon[29459]: osdmap e135: 8 total, 8 up, 7 in 2024-09-18T08:06:22.684 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:06:22 smithi007 ceph-779349ca-7593-11ef-bceb-c7b262605968-mon-a[29455]: 2024-09-18T08:06:22.586+0000 7fcd2b595640 -1 mon.a@0(leader).osd e135 definitely_dead 0 2024-09-18T08:06:22.684 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:06:22 smithi007 ceph-mon[39249]: pgmap v156: 148 pgs: 3 active+remapped+backfilling, 2 active+remapped+backfill_toofull, 143 active+clean; 4.8 GiB data, 14 GiB used, 612 GiB / 626 GiB avail; 777 B/s rd, 0 B/s wr, 580 op/s; 23198/234378 objects misplaced (9.898%); 11 MiB/s, 178 objects/s recovering 2024-09-18T08:06:22.684 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:06:22 smithi007 ceph-mon[39249]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-18T08:06:22.685 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:06:22 smithi007 ceph-mon[39249]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-18T08:06:22.685 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:06:22 smithi007 ceph-mon[39249]: osdmap e135: 8 total, 8 up, 7 in 2024-09-18T08:06:22.718 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1 2024-09-18T08:06:22.767 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:06:22.768 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:06:22.768 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1' 2024-09-18T08:06:22.810 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/779349ca-7593-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T08:06:22.828 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/779349ca-7593-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T08:06:22.868 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5 2024-09-18T08:06:22.892 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:06:22.893 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:06:22.893 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5' 2024-09-18T08:06:22.905 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:06:22 smithi050 ceph-mon[32371]: pgmap v156: 148 pgs: 3 active+remapped+backfilling, 2 active+remapped+backfill_toofull, 143 active+clean; 4.8 GiB data, 14 GiB used, 612 GiB / 626 GiB avail; 777 B/s rd, 0 B/s wr, 580 op/s; 23198/234378 objects misplaced (9.898%); 11 MiB/s, 178 objects/s recovering 2024-09-18T08:06:22.906 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:06:22 smithi050 ceph-mon[32371]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-18T08:06:22.906 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:06:22 smithi050 ceph-mon[32371]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-18T08:06:22.906 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:06:22 smithi050 ceph-mon[32371]: osdmap e135: 8 total, 8 up, 7 in 2024-09-18T08:06:22.995 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2 2024-09-18T08:06:23.042 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:06:23.043 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:06:23.043 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2' 2024-09-18T08:06:23.144 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4 2024-09-18T08:06:23.169 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:06:23.169 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:06:23.170 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4' 2024-09-18T08:06:23.271 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0 2024-09-18T08:06:23.319 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:06:23.320 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:06:23.320 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0' 2024-09-18T08:06:23.421 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6 2024-09-18T08:06:23.444 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:06:23.445 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:06:23.445 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6' 2024-09-18T08:06:23.519 INFO:teuthology.orchestra.run.smithi007.stderr:marked in osd.6. 2024-09-18T08:06:23.546 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:06:23 smithi007 ceph-mon[29459]: from='client.? 172.21.15.7:0/1682764964' entity='client.admin' cmd={"prefix": "osd in", "ids": ["6"]} : dispatch 2024-09-18T08:06:23.546 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:06:23 smithi007 ceph-mon[39249]: from='client.? 172.21.15.7:0/1682764964' entity='client.admin' cmd={"prefix": "osd in", "ids": ["6"]} : dispatch 2024-09-18T08:06:23.547 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5 2024-09-18T08:06:23.571 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:06:23.571 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:06:23.572 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5' 2024-09-18T08:06:23.672 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5 2024-09-18T08:06:23.695 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:06:23.695 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:06:23.696 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5' 2024-09-18T08:06:23.745 INFO:teuthology.orchestra.run.smithi007.stdout:ERROR: (22) Invalid argument 2024-09-18T08:06:23.746 INFO:teuthology.orchestra.run.smithi007.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-18T08:06:23.796 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2 2024-09-18T08:06:23.818 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:06:23.819 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:06:23.819 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2' 2024-09-18T08:06:23.906 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:06:23 smithi050 ceph-mon[32371]: from='client.? 172.21.15.7:0/1682764964' entity='client.admin' cmd={"prefix": "osd in", "ids": ["6"]} : dispatch 2024-09-18T08:06:23.920 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6 2024-09-18T08:06:23.943 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:06:23.944 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:06:23.944 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6' 2024-09-18T08:06:24.045 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1 2024-09-18T08:06:24.068 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:06:24.068 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:06:24.069 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1' 2024-09-18T08:06:24.170 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2 2024-09-18T08:06:24.204 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:06:24.205 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:06:24.205 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2' 2024-09-18T08:06:24.306 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4 2024-09-18T08:06:24.328 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:06:24.329 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:06:24.329 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4' 2024-09-18T08:06:24.430 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5 2024-09-18T08:06:24.453 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:06:24.453 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:06:24.453 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5' 2024-09-18T08:06:24.554 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1 2024-09-18T08:06:24.577 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:06:24.578 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:06:24.578 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1' 2024-09-18T08:06:24.679 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4 2024-09-18T08:06:24.703 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:06:24.703 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:06:24.704 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4' 2024-09-18T08:06:24.730 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:06:24 smithi007 ceph-mon[29459]: pgmap v158: 148 pgs: 3 active+remapped+backfilling, 2 active+remapped+backfill_toofull, 143 active+clean; 4.7 GiB data, 13 GiB used, 612 GiB / 626 GiB avail; 1.3 KiB/s rd, 0 B/s wr, 625 op/s; 22903/231678 objects misplaced (9.886%); 9.7 MiB/s, 155 objects/s recovering 2024-09-18T08:06:24.730 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:06:24 smithi007 ceph-mon[29459]: from='client.? 172.21.15.7:0/1682764964' entity='client.admin' cmd='[{"prefix": "osd in", "ids": ["6"]}]': finished 2024-09-18T08:06:24.731 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:06:24 smithi007 ceph-mon[29459]: osdmap e136: 8 total, 8 up, 8 in 2024-09-18T08:06:24.731 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:06:24 smithi007 ceph-mon[29459]: from='client.? 172.21.15.7:0/1928057422' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-18T08:06:24.731 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:06:24 smithi007 ceph-mon[29459]: osdmap e137: 8 total, 8 up, 8 in 2024-09-18T08:06:24.731 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:06:24 smithi007 ceph-mon[39249]: pgmap v158: 148 pgs: 3 active+remapped+backfilling, 2 active+remapped+backfill_toofull, 143 active+clean; 4.7 GiB data, 13 GiB used, 612 GiB / 626 GiB avail; 1.3 KiB/s rd, 0 B/s wr, 625 op/s; 22903/231678 objects misplaced (9.886%); 9.7 MiB/s, 155 objects/s recovering 2024-09-18T08:06:24.731 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:06:24 smithi007 ceph-mon[39249]: from='client.? 172.21.15.7:0/1682764964' entity='client.admin' cmd='[{"prefix": "osd in", "ids": ["6"]}]': finished 2024-09-18T08:06:24.731 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:06:24 smithi007 ceph-mon[39249]: osdmap e136: 8 total, 8 up, 8 in 2024-09-18T08:06:24.731 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:06:24 smithi007 ceph-mon[39249]: from='client.? 172.21.15.7:0/1928057422' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-18T08:06:24.731 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:06:24 smithi007 ceph-mon[39249]: osdmap e137: 8 total, 8 up, 8 in 2024-09-18T08:06:24.735 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid 779349ca-7593-11ef-bceb-c7b262605968 -- ceph osd set nodeep-scrub 2024-09-18T08:06:24.804 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4 2024-09-18T08:06:24.827 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:06:24.828 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:06:24.828 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4' 2024-09-18T08:06:24.900 INFO:teuthology.orchestra.run.smithi007.stdout:osd.0: {} 2024-09-18T08:06:24.900 INFO:teuthology.orchestra.run.smithi007.stdout:osd.1: {} 2024-09-18T08:06:24.900 INFO:teuthology.orchestra.run.smithi007.stdout:osd.2: {} 2024-09-18T08:06:24.900 INFO:teuthology.orchestra.run.smithi007.stdout:osd.3: {} 2024-09-18T08:06:24.900 INFO:teuthology.orchestra.run.smithi007.stdout:osd.4: {} 2024-09-18T08:06:24.900 INFO:teuthology.orchestra.run.smithi007.stdout:osd.5: {} 2024-09-18T08:06:24.901 INFO:teuthology.orchestra.run.smithi007.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-18T08:06:24.901 INFO:teuthology.orchestra.run.smithi007.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-18T08:06:24.901 INFO:teuthology.orchestra.run.smithi007.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-18T08:06:24.901 INFO:teuthology.orchestra.run.smithi007.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-18T08:06:24.901 INFO:teuthology.orchestra.run.smithi007.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-18T08:06:24.901 INFO:teuthology.orchestra.run.smithi007.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-18T08:06:24.902 INFO:teuthology.orchestra.run.smithi007.stdout:osd.6: {} 2024-09-18T08:06:24.902 INFO:teuthology.orchestra.run.smithi007.stdout:osd.7: {} 2024-09-18T08:06:24.902 INFO:teuthology.orchestra.run.smithi007.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-18T08:06:24.902 INFO:teuthology.orchestra.run.smithi007.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-18T08:06:24.906 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:06:24 smithi050 ceph-mon[32371]: pgmap v158: 148 pgs: 3 active+remapped+backfilling, 2 active+remapped+backfill_toofull, 143 active+clean; 4.7 GiB data, 13 GiB used, 612 GiB / 626 GiB avail; 1.3 KiB/s rd, 0 B/s wr, 625 op/s; 22903/231678 objects misplaced (9.886%); 9.7 MiB/s, 155 objects/s recovering 2024-09-18T08:06:24.906 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:06:24 smithi050 ceph-mon[32371]: from='client.? 172.21.15.7:0/1682764964' entity='client.admin' cmd='[{"prefix": "osd in", "ids": ["6"]}]': finished 2024-09-18T08:06:24.906 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:06:24 smithi050 ceph-mon[32371]: osdmap e136: 8 total, 8 up, 8 in 2024-09-18T08:06:24.906 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:06:24 smithi050 ceph-mon[32371]: from='client.? 172.21.15.7:0/1928057422' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-18T08:06:24.906 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:06:24 smithi050 ceph-mon[32371]: osdmap e137: 8 total, 8 up, 8 in 2024-09-18T08:06:24.930 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1 2024-09-18T08:06:24.951 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:06:24.952 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:06:24.953 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1' 2024-09-18T08:06:25.053 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3 2024-09-18T08:06:25.076 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:06:25.077 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:06:25.077 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3' 2024-09-18T08:06:25.178 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1 2024-09-18T08:06:25.223 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:06:25.225 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:06:25.225 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1' 2024-09-18T08:06:25.278 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid 779349ca-7593-11ef-bceb-c7b262605968 -- ceph daemon osd.3 dump_historic_ops 2024-09-18T08:06:25.326 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3 2024-09-18T08:06:25.369 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:06:25.370 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:06:25.370 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3' 2024-09-18T08:06:25.471 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2 2024-09-18T08:06:25.532 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:06:25.532 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:06:25.533 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2' 2024-09-18T08:06:25.634 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1 2024-09-18T08:06:25.682 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:06:25.682 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:06:25.682 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1' 2024-09-18T08:06:25.783 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7 2024-09-18T08:06:25.810 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:06:25.811 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:06:25.811 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7' 2024-09-18T08:06:25.906 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:06:25 smithi050 ceph-mon[32371]: purged_snaps scrub starts 2024-09-18T08:06:25.906 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:06:25 smithi050 ceph-mon[32371]: purged_snaps scrub ok 2024-09-18T08:06:25.906 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:06:25 smithi050 ceph-mon[32371]: osdmap e138: 8 total, 8 up, 8 in 2024-09-18T08:06:25.912 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0 2024-09-18T08:06:25.953 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:06:25.954 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:06:25.954 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0' 2024-09-18T08:06:25.980 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:06:25 smithi007 ceph-mon[39249]: purged_snaps scrub starts 2024-09-18T08:06:25.980 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:06:25 smithi007 ceph-mon[39249]: purged_snaps scrub ok 2024-09-18T08:06:25.981 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:06:25 smithi007 ceph-mon[39249]: osdmap e138: 8 total, 8 up, 8 in 2024-09-18T08:06:25.981 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:06:25 smithi007 ceph-mon[29459]: purged_snaps scrub starts 2024-09-18T08:06:25.981 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:06:25 smithi007 ceph-mon[29459]: purged_snaps scrub ok 2024-09-18T08:06:25.981 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:06:25 smithi007 ceph-mon[29459]: osdmap e138: 8 total, 8 up, 8 in 2024-09-18T08:06:26.056 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5 2024-09-18T08:06:26.083 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:06:26.083 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:06:26.084 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5' 2024-09-18T08:06:26.184 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3 2024-09-18T08:06:26.223 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:06:26.223 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:06:26.224 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3' 2024-09-18T08:06:26.324 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0 2024-09-18T08:06:26.354 INFO:tasks.thrashosds.thrasher:Added osd 6 2024-09-18T08:06:26.390 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/779349ca-7593-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T08:06:26.397 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/779349ca-7593-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T08:06:26.410 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:06:26.410 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:06:26.411 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0' 2024-09-18T08:06:26.512 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6 2024-09-18T08:06:26.540 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:06:26.540 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:06:26.540 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6' 2024-09-18T08:06:26.541 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:06:26 smithi007 ceph-mon[39249]: pgmap v162: 148 pgs: 3 active+remapped+backfilling, 2 active+remapped+backfill_toofull, 143 active+clean; 4.5 GiB data, 13 GiB used, 613 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.15k op/s; 20493/220638 objects misplaced (9.288%); 22 MiB/s, 349 objects/s recovering 2024-09-18T08:06:26.541 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:06:26 smithi007 ceph-mon[29459]: pgmap v162: 148 pgs: 3 active+remapped+backfilling, 2 active+remapped+backfill_toofull, 143 active+clean; 4.5 GiB data, 13 GiB used, 613 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.15k op/s; 20493/220638 objects misplaced (9.288%); 22 MiB/s, 349 objects/s recovering 2024-09-18T08:06:26.641 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1 2024-09-18T08:06:26.685 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:06:26.686 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:06:26.686 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1' 2024-09-18T08:06:26.787 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2 2024-09-18T08:06:26.830 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:06:26.830 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:06:26.831 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2' 2024-09-18T08:06:26.906 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:06:26 smithi050 ceph-mon[32371]: pgmap v162: 148 pgs: 3 active+remapped+backfilling, 2 active+remapped+backfill_toofull, 143 active+clean; 4.5 GiB data, 13 GiB used, 613 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.15k op/s; 20493/220638 objects misplaced (9.288%); 22 MiB/s, 349 objects/s recovering 2024-09-18T08:06:26.931 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6 2024-09-18T08:06:26.957 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:06:26.958 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:06:26.958 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6' 2024-09-18T08:06:26.980 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Sep 18 08:06:26 smithi007 ceph-779349ca-7593-11ef-bceb-c7b262605968-mgr-y[29688]: ::ffff:172.21.15.50 - - [18/Sep/2024:08:06:26] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-09-18T08:06:27.059 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7 2024-09-18T08:06:27.089 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:06:27.089 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:06:27.089 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7' 2024-09-18T08:06:27.190 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2 2024-09-18T08:06:27.228 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:06:27.229 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:06:27.229 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2' 2024-09-18T08:06:27.330 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4 2024-09-18T08:06:27.354 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:06:27.355 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:06:27.355 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4' 2024-09-18T08:06:27.456 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4 2024-09-18T08:06:27.478 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:06:27.478 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:06:27.479 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4' 2024-09-18T08:06:27.580 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2 2024-09-18T08:06:27.616 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:06:27.616 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:06:27.617 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2' 2024-09-18T08:06:27.718 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4 2024-09-18T08:06:27.743 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:06:27.744 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:06:27.744 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4' 2024-09-18T08:06:27.771 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid 779349ca-7593-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-18T08:06:27.845 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3 2024-09-18T08:06:27.891 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:06:27.892 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:06:27.892 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3' 2024-09-18T08:06:27.993 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7 2024-09-18T08:06:28.015 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:06:28.015 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:06:28.016 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7' 2024-09-18T08:06:28.117 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1 2024-09-18T08:06:28.159 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:06:28.160 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:06:28.160 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1' 2024-09-18T08:06:28.261 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7 2024-09-18T08:06:28.285 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:06:28.286 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:06:28.286 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7' 2024-09-18T08:06:28.387 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5 2024-09-18T08:06:28.414 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:06:28.414 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:06:28.415 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5' 2024-09-18T08:06:28.516 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2 2024-09-18T08:06:28.556 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:06:28.557 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:06:28.557 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2' 2024-09-18T08:06:28.658 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0 2024-09-18T08:06:28.698 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:06:28.699 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:06:28.699 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0' 2024-09-18T08:06:28.800 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5 2024-09-18T08:06:28.824 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:06:28.824 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:06:28.825 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5' 2024-09-18T08:06:28.872 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:06:28 smithi050 ceph-mon[32371]: pgmap v163: 148 pgs: 3 active+remapped+backfilling, 2 active+remapped+backfill_toofull, 143 active+clean; 4.5 GiB data, 13 GiB used, 613 GiB / 626 GiB avail; 908 B/s rd, 0 B/s wr, 813 op/s; 20493/220638 objects misplaced (9.288%); 16 MiB/s, 248 objects/s recovering 2024-09-18T08:06:28.926 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3 2024-09-18T08:06:28.967 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:06:28.967 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:06:28.968 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3' 2024-09-18T08:06:28.980 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:06:28 smithi007 ceph-mon[39249]: pgmap v163: 148 pgs: 3 active+remapped+backfilling, 2 active+remapped+backfill_toofull, 143 active+clean; 4.5 GiB data, 13 GiB used, 613 GiB / 626 GiB avail; 908 B/s rd, 0 B/s wr, 813 op/s; 20493/220638 objects misplaced (9.288%); 16 MiB/s, 248 objects/s recovering 2024-09-18T08:06:28.981 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:06:28 smithi007 ceph-mon[29459]: pgmap v163: 148 pgs: 3 active+remapped+backfilling, 2 active+remapped+backfill_toofull, 143 active+clean; 4.5 GiB data, 13 GiB used, 613 GiB / 626 GiB avail; 908 B/s rd, 0 B/s wr, 813 op/s; 20493/220638 objects misplaced (9.288%); 16 MiB/s, 248 objects/s recovering 2024-09-18T08:06:29.069 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1 2024-09-18T08:06:29.114 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:06:29.116 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:06:29.117 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1' 2024-09-18T08:06:29.156 INFO:journalctl@ceph.iscsi.iscsi.a.smithi050.stdout:Sep 18 08:06:28 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-iscsi-iscsi-a[57782]: debug there is no tcmu-runner data available 2024-09-18T08:06:29.217 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4 2024-09-18T08:06:29.244 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:06:29.245 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:06:29.245 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4' 2024-09-18T08:06:29.346 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1 2024-09-18T08:06:29.370 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:06:29.370 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:06:29.371 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1' 2024-09-18T08:06:29.472 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7 2024-09-18T08:06:29.496 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:06:29.497 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:06:29.498 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7' 2024-09-18T08:06:29.599 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6 2024-09-18T08:06:29.624 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:06:29.625 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:06:29.625 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6' 2024-09-18T08:06:29.726 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4 2024-09-18T08:06:29.757 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:06:29.757 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:06:29.758 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4' 2024-09-18T08:06:29.859 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3 2024-09-18T08:06:29.898 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:06:29.899 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:06:29.899 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3' 2024-09-18T08:06:30.000 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0 2024-09-18T08:06:30.049 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:06:30.049 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:06:30.049 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0' 2024-09-18T08:06:30.152 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1 2024-09-18T08:06:30.196 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:06:30.196 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:06:30.197 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1' 2024-09-18T08:06:30.297 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1 2024-09-18T08:06:30.336 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:06:30.336 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:06:30.337 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1' 2024-09-18T08:06:30.406 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:06:30 smithi050 ceph-mon[32371]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-18T08:06:30.406 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:06:30 smithi050 ceph-mon[32371]: pgmap v164: 148 pgs: 148 active+clean; 4.3 GiB data, 14 GiB used, 702 GiB / 715 GiB avail; 853 B/s rd, 0 B/s wr, 1.20k op/s; 15 MiB/s, 253 objects/s recovering 2024-09-18T08:06:30.437 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2 2024-09-18T08:06:30.465 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:06:30.466 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:06:30.466 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2' 2024-09-18T08:06:30.480 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:06:30 smithi007 ceph-mon[29459]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-18T08:06:30.480 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:06:30 smithi007 ceph-mon[29459]: pgmap v164: 148 pgs: 148 active+clean; 4.3 GiB data, 14 GiB used, 702 GiB / 715 GiB avail; 853 B/s rd, 0 B/s wr, 1.20k op/s; 15 MiB/s, 253 objects/s recovering 2024-09-18T08:06:30.480 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:06:30 smithi007 ceph-mon[39249]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-18T08:06:30.480 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:06:30 smithi007 ceph-mon[39249]: pgmap v164: 148 pgs: 148 active+clean; 4.3 GiB data, 14 GiB used, 702 GiB / 715 GiB avail; 853 B/s rd, 0 B/s wr, 1.20k op/s; 15 MiB/s, 253 objects/s recovering 2024-09-18T08:06:30.567 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0 2024-09-18T08:06:30.609 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:06:30.609 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:06:30.610 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0' 2024-09-18T08:06:30.711 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1 2024-09-18T08:06:30.750 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:06:30.750 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:06:30.751 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1' 2024-09-18T08:06:30.852 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7 2024-09-18T08:06:30.880 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:06:30.881 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:06:30.881 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7' 2024-09-18T08:06:30.982 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1 2024-09-18T08:06:31.024 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:06:31.024 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:06:31.025 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1' 2024-09-18T08:06:31.126 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5 2024-09-18T08:06:31.154 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:06:31.154 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:06:31.155 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5' 2024-09-18T08:06:31.255 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2 2024-09-18T08:06:31.300 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:06:31.301 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:06:31.301 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2' 2024-09-18T08:06:31.355 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 7, 6] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-09-18T08:06:31.355 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-18T08:06:31.356 INFO:tasks.thrashosds.thrasher:Shrinking pool unique_pool_0 2024-09-18T08:06:31.356 INFO:tasks.cephadm.ceph_manager.ceph:contract_pool unique_pool_0 by 10 min 8 2024-09-18T08:06:31.356 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid 779349ca-7593-11ef-bceb-c7b262605968 -- ceph pg dump --format=json 2024-09-18T08:06:31.397 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:06:31 smithi007 ceph-mon[29459]: Health check cleared: PG_BACKFILL_FULL (was: Low space hindering backfill (add storage if this doesn't resolve itself): 2 pgs backfill_toofull) 2024-09-18T08:06:31.397 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:06:31 smithi007 ceph-mon[39249]: Health check cleared: PG_BACKFILL_FULL (was: Low space hindering backfill (add storage if this doesn't resolve itself): 2 pgs backfill_toofull) 2024-09-18T08:06:31.401 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7 2024-09-18T08:06:31.406 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:06:31 smithi050 ceph-mon[32371]: Health check cleared: PG_BACKFILL_FULL (was: Low space hindering backfill (add storage if this doesn't resolve itself): 2 pgs backfill_toofull) 2024-09-18T08:06:31.423 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:06:31.424 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:06:31.424 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7' 2024-09-18T08:06:31.525 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7 2024-09-18T08:06:31.553 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:06:31.553 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:06:31.554 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7' 2024-09-18T08:06:31.654 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0 2024-09-18T08:06:31.685 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:06:31.685 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:06:31.685 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0' 2024-09-18T08:06:31.788 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1 2024-09-18T08:06:31.815 INFO:teuthology.orchestra.run.smithi007.stdout:ERROR: (22) Invalid argument 2024-09-18T08:06:31.815 INFO:teuthology.orchestra.run.smithi007.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-18T08:06:31.828 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:06:31.829 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:06:31.830 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1' 2024-09-18T08:06:31.931 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3 2024-09-18T08:06:31.970 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:06:31.970 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:06:31.971 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3' 2024-09-18T08:06:32.072 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7 2024-09-18T08:06:32.100 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:06:32.101 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:06:32.101 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7' 2024-09-18T08:06:32.202 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0 2024-09-18T08:06:32.242 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:06:32.243 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:06:32.243 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0' 2024-09-18T08:06:32.344 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6 2024-09-18T08:06:32.370 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:06:32.371 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:06:32.371 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6' 2024-09-18T08:06:32.406 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:06:32 smithi050 ceph-mon[32371]: pgmap v165: 148 pgs: 148 active+clean; 4.3 GiB data, 14 GiB used, 702 GiB / 715 GiB avail; 672 B/s rd, 0 B/s wr, 466 op/s; 505 KiB/s, 16 objects/s recovering 2024-09-18T08:06:32.472 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5 2024-09-18T08:06:32.480 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:06:32 smithi007 ceph-mon[29459]: pgmap v165: 148 pgs: 148 active+clean; 4.3 GiB data, 14 GiB used, 702 GiB / 715 GiB avail; 672 B/s rd, 0 B/s wr, 466 op/s; 505 KiB/s, 16 objects/s recovering 2024-09-18T08:06:32.480 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:06:32 smithi007 ceph-mon[39249]: pgmap v165: 148 pgs: 148 active+clean; 4.3 GiB data, 14 GiB used, 702 GiB / 715 GiB avail; 672 B/s rd, 0 B/s wr, 466 op/s; 505 KiB/s, 16 objects/s recovering 2024-09-18T08:06:32.498 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:06:32.499 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:06:32.499 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5' 2024-09-18T08:06:32.600 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0 2024-09-18T08:06:32.648 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:06:32.649 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:06:32.649 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0' 2024-09-18T08:06:32.750 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6 2024-09-18T08:06:32.776 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:06:32.777 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:06:32.778 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6' 2024-09-18T08:06:32.878 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4 2024-09-18T08:06:32.905 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:06:32.906 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:06:32.906 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4' 2024-09-18T08:06:33.007 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2 2024-09-18T08:06:33.050 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:06:33.050 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:06:33.050 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2' 2024-09-18T08:06:33.151 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6 2024-09-18T08:06:33.172 INFO:teuthology.orchestra.run.smithi007.stderr:nodeep-scrub is set 2024-09-18T08:06:33.172 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:06:33 smithi007 ceph-mon[29459]: from='client.? 172.21.15.7:0/2198235456' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-18T08:06:33.173 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:06:33 smithi007 ceph-mon[29459]: from='mgr.24470 172.21.15.7:0/2594390261' entity='mgr.y' 2024-09-18T08:06:33.173 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:06:33 smithi007 ceph-mon[29459]: from='mgr.24470 172.21.15.7:0/2594390261' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-18T08:06:33.173 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:06:33 smithi007 ceph-mon[39249]: from='client.? 172.21.15.7:0/2198235456' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-18T08:06:33.173 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:06:33 smithi007 ceph-mon[39249]: from='mgr.24470 172.21.15.7:0/2594390261' entity='mgr.y' 2024-09-18T08:06:33.173 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:06:33 smithi007 ceph-mon[39249]: from='mgr.24470 172.21.15.7:0/2594390261' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-18T08:06:33.179 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:06:33.180 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:06:33.180 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6' 2024-09-18T08:06:33.281 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7 2024-09-18T08:06:33.309 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:06:33.310 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:06:33.310 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7' 2024-09-18T08:06:33.375 DEBUG:teuthology.orchestra.run.smithi050:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid 779349ca-7593-11ef-bceb-c7b262605968 -- ceph daemon osd.4 dump_ops_in_flight 2024-09-18T08:06:33.402 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:06:33 smithi050 ceph-mon[32371]: from='client.? 172.21.15.7:0/2198235456' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-18T08:06:33.402 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:06:33 smithi050 ceph-mon[32371]: from='mgr.24470 172.21.15.7:0/2594390261' entity='mgr.y' 2024-09-18T08:06:33.402 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:06:33 smithi050 ceph-mon[32371]: from='mgr.24470 172.21.15.7:0/2594390261' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-18T08:06:33.410 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0 2024-09-18T08:06:33.446 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:06:33.447 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:06:33.448 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0' 2024-09-18T08:06:33.549 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6 2024-09-18T08:06:33.580 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:06:33.580 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:06:33.581 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6' 2024-09-18T08:06:33.682 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3 2024-09-18T08:06:33.721 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:06:33.721 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:06:33.722 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3' 2024-09-18T08:06:33.729 INFO:teuthology.orchestra.run.smithi050.stderr:Inferring config /var/lib/ceph/779349ca-7593-11ef-bceb-c7b262605968/mon.b/config 2024-09-18T08:06:33.823 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3 2024-09-18T08:06:33.884 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:06:33.885 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:06:33.885 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3' 2024-09-18T08:06:33.986 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0 2024-09-18T08:06:34.028 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:06:34.028 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:06:34.029 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0' 2024-09-18T08:06:34.129 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7 2024-09-18T08:06:34.161 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:06:34.161 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:06:34.162 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7' 2024-09-18T08:06:34.263 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5 2024-09-18T08:06:34.295 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:06:34.295 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:06:34.295 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5' 2024-09-18T08:06:34.396 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6 2024-09-18T08:06:34.406 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:06:34 smithi050 ceph-mon[32371]: pgmap v166: 148 pgs: 148 active+clean; 4.2 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 502 op/s; 426 KiB/s, 13 objects/s recovering 2024-09-18T08:06:34.406 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:06:34 smithi050 ceph-mon[32371]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-18T08:06:34.406 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:06:34 smithi050 ceph-mon[32371]: from='client.? 172.21.15.7:0/2198235456' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-18T08:06:34.406 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:06:34 smithi050 ceph-mon[32371]: osdmap e139: 8 total, 8 up, 8 in 2024-09-18T08:06:34.431 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:06:34.431 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:06:34.432 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6' 2024-09-18T08:06:34.440 INFO:teuthology.orchestra.run.smithi050.stdout:ERROR: (22) Invalid argument 2024-09-18T08:06:34.440 INFO:teuthology.orchestra.run.smithi050.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-18T08:06:34.445 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:06:34 smithi007 ceph-mon[29459]: pgmap v166: 148 pgs: 148 active+clean; 4.2 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 502 op/s; 426 KiB/s, 13 objects/s recovering 2024-09-18T08:06:34.448 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:06:34 smithi007 ceph-mon[29459]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-18T08:06:34.448 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:06:34 smithi007 ceph-mon[29459]: from='client.? 172.21.15.7:0/2198235456' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-18T08:06:34.449 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:06:34 smithi007 ceph-mon[29459]: osdmap e139: 8 total, 8 up, 8 in 2024-09-18T08:06:34.449 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:06:34 smithi007 ceph-mon[39249]: pgmap v166: 148 pgs: 148 active+clean; 4.2 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 502 op/s; 426 KiB/s, 13 objects/s recovering 2024-09-18T08:06:34.449 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:06:34 smithi007 ceph-mon[39249]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-18T08:06:34.449 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:06:34 smithi007 ceph-mon[39249]: from='client.? 172.21.15.7:0/2198235456' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-18T08:06:34.449 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:06:34 smithi007 ceph-mon[39249]: osdmap e139: 8 total, 8 up, 8 in 2024-09-18T08:06:34.532 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6 2024-09-18T08:06:34.555 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/779349ca-7593-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T08:06:34.562 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:06:34.563 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:06:34.563 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6' 2024-09-18T08:06:34.566 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/779349ca-7593-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T08:06:34.664 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4 2024-09-18T08:06:34.698 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:06:34.699 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:06:34.699 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4' 2024-09-18T08:06:34.800 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1 2024-09-18T08:06:34.846 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:06:34.847 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:06:34.848 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1' 2024-09-18T08:06:34.948 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2 2024-09-18T08:06:34.987 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:06:34.987 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:06:34.988 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2' 2024-09-18T08:06:35.088 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6 2024-09-18T08:06:35.115 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:06:35.116 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:06:35.116 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6' 2024-09-18T08:06:35.217 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7 2024-09-18T08:06:35.241 DEBUG:teuthology.orchestra.run.smithi050:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid 779349ca-7593-11ef-bceb-c7b262605968 -- ceph daemon osd.4 dump_blocked_ops 2024-09-18T08:06:35.256 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:06:35.256 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:06:35.257 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7' 2024-09-18T08:06:35.357 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3 2024-09-18T08:06:35.392 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:06:35.393 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:06:35.393 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3' 2024-09-18T08:06:35.494 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0 2024-09-18T08:06:35.534 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:06:35.535 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:06:35.535 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0' 2024-09-18T08:06:35.636 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0 2024-09-18T08:06:35.650 INFO:teuthology.orchestra.run.smithi050.stderr:Inferring config /var/lib/ceph/779349ca-7593-11ef-bceb-c7b262605968/mon.b/config 2024-09-18T08:06:35.692 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:06:35.692 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:06:35.693 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0' 2024-09-18T08:06:35.794 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3 2024-09-18T08:06:35.846 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:06:35.846 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:06:35.847 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3' 2024-09-18T08:06:35.947 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4 2024-09-18T08:06:35.972 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:06:35.973 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:06:35.973 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4' 2024-09-18T08:06:36.074 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0 2024-09-18T08:06:36.114 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:06:36.115 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:06:36.115 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0' 2024-09-18T08:06:36.215 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-09-18T08:06:36.216 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5 2024-09-18T08:06:36.224 INFO:teuthology.orchestra.run.smithi007.stderr:dumped all 2024-09-18T08:06:36.244 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:06:36.244 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:06:36.245 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5' 2024-09-18T08:06:36.246 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:06:36 smithi007 ceph-mon[29459]: pgmap v168: 148 pgs: 148 active+clean; 3.9 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 892 op/s; 384 KiB/s, 12 objects/s recovering 2024-09-18T08:06:36.246 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:06:36 smithi007 ceph-mon[29459]: from='mgr.24470 172.21.15.7:0/2594390261' entity='mgr.y' 2024-09-18T08:06:36.246 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:06:36 smithi007 ceph-mon[29459]: from='mgr.24470 172.21.15.7:0/2594390261' entity='mgr.y' 2024-09-18T08:06:36.247 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:06:36 smithi007 ceph-mon[39249]: pgmap v168: 148 pgs: 148 active+clean; 3.9 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 892 op/s; 384 KiB/s, 12 objects/s recovering 2024-09-18T08:06:36.247 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:06:36 smithi007 ceph-mon[39249]: from='mgr.24470 172.21.15.7:0/2594390261' entity='mgr.y' 2024-09-18T08:06:36.247 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:06:36 smithi007 ceph-mon[39249]: from='mgr.24470 172.21.15.7:0/2594390261' entity='mgr.y' 2024-09-18T08:06:36.345 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5 2024-09-18T08:06:36.371 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:06:36.372 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:06:36.372 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5' 2024-09-18T08:06:36.391 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:06:36 smithi050 ceph-mon[32371]: pgmap v168: 148 pgs: 148 active+clean; 3.9 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 892 op/s; 384 KiB/s, 12 objects/s recovering 2024-09-18T08:06:36.391 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:06:36 smithi050 ceph-mon[32371]: from='mgr.24470 172.21.15.7:0/2594390261' entity='mgr.y' 2024-09-18T08:06:36.391 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:06:36 smithi050 ceph-mon[32371]: from='mgr.24470 172.21.15.7:0/2594390261' entity='mgr.y' 2024-09-18T08:06:36.473 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2 2024-09-18T08:06:36.503 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid 779349ca-7593-11ef-bceb-c7b262605968 -- ceph osd unset noscrub 2024-09-18T08:06:36.515 INFO:teuthology.orchestra.run.smithi007.stdout:osd.0: {} 2024-09-18T08:06:36.516 INFO:teuthology.orchestra.run.smithi007.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-18T08:06:36.516 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:06:36.520 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:06:36.520 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2' 2024-09-18T08:06:36.525 INFO:teuthology.orchestra.run.smithi050.stdout:ERROR: (22) Invalid argument 2024-09-18T08:06:36.525 INFO:teuthology.orchestra.run.smithi050.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-18T08:06:36.554 INFO:teuthology.orchestra.run.smithi007.stdout:osd.1: {} 2024-09-18T08:06:36.554 INFO:teuthology.orchestra.run.smithi007.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-18T08:06:36.596 INFO:teuthology.orchestra.run.smithi007.stdout:osd.2: {} 2024-09-18T08:06:36.596 INFO:teuthology.orchestra.run.smithi007.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-18T08:06:36.621 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4 2024-09-18T08:06:36.640 INFO:teuthology.orchestra.run.smithi007.stdout:osd.3: {} 2024-09-18T08:06:36.640 INFO:teuthology.orchestra.run.smithi007.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-18T08:06:36.647 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:06:36.647 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:06:36.648 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4' 2024-09-18T08:06:36.676 INFO:teuthology.orchestra.run.smithi007.stdout:osd.4: {} 2024-09-18T08:06:36.676 INFO:teuthology.orchestra.run.smithi007.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-18T08:06:36.720 INFO:teuthology.orchestra.run.smithi007.stdout:osd.5: {} 2024-09-18T08:06:36.720 INFO:teuthology.orchestra.run.smithi007.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-18T08:06:36.748 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0 2024-09-18T08:06:36.755 INFO:teuthology.orchestra.run.smithi007.stdout:osd.6: {} 2024-09-18T08:06:36.755 INFO:teuthology.orchestra.run.smithi007.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-18T08:06:36.802 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Sep 18 08:06:36 smithi007 ceph-779349ca-7593-11ef-bceb-c7b262605968-mgr-y[29688]: ::ffff:172.21.15.50 - - [18/Sep/2024:08:06:36] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-09-18T08:06:36.802 INFO:teuthology.orchestra.run.smithi007.stdout:osd.7: {} 2024-09-18T08:06:36.802 INFO:teuthology.orchestra.run.smithi007.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-18T08:06:36.805 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:06:36.806 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:06:36.806 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0' 2024-09-18T08:06:36.907 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0 2024-09-18T08:06:36.981 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:06:36.982 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:06:36.982 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0' 2024-09-18T08:06:37.082 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3 2024-09-18T08:06:37.125 INFO:teuthology.orchestra.run.smithi007.stdout:{"pg_ready":true,"pg_map":{"version":168,"stamp":"2024-09-18T08:06:35.137572+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":4222818867,"num_objects":64626,"num_object_clones":0,"num_object_copies":193878,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":64626,"num_whiteouts":0,"num_read":1065,"num_read_kb":922,"num_write":172700,"num_write_kb":7572106,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2792,"num_bytes_recovered":173871186,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":83562,"ondisk_log_size":83562,"up":444,"acting":444,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":445,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":13368664,"kb_used_data":12546908,"kb_used_omap":12,"kb_used_meta":821299,"kb_avail":736592552,"statfs":{"total":767960285184,"available":754270773248,"internally_reserved":0,"allocated":12848033792,"data_stored":12844848346,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12750,"internal_metadata":841010738},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-584581120,"num_objects":-8920,"num_object_clones":0,"num_object_copies":-26760,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-20493,"num_objects_unfound":0,"num_objects_dirty":-8920,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":8920,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":125,"num_bytes_recovered":3933757,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.003046"},"pg_stats":[{"pgid":"3.1f","version":"0'0","reported_seq":175,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-09-18T08:06:33.180723+0000","last_change":"2024-09-18T08:04:34.226236+0000","last_active":"2024-09-18T08:06:33.180723+0000","last_peered":"2024-09-18T08:06:33.180723+0000","last_clean":"2024-09-18T08:06:33.180723+0000","last_became_active":"2024-09-18T08:02:47.877698+0000","last_became_peered":"2024-09-18T08:02:47.877698+0000","last_unstale":"2024-09-18T08:06:33.180723+0000","last_undegraded":"2024-09-18T08:06:33.180723+0000","last_fullsized":"2024-09-18T08:06:33.180723+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:04:34.226054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:46.860092+0000","last_clean_scrub_stamp":"2024-09-18T08:04:34.226054+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"69'9","reported_seq":219,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-09-18T08:06:25.117517+0000","last_change":"2024-09-18T08:06:24.171122+0000","last_active":"2024-09-18T08:06:25.117517+0000","last_peered":"2024-09-18T08:06:25.117517+0000","last_clean":"2024-09-18T08:06:25.117517+0000","last_became_active":"2024-09-18T08:06:24.140103+0000","last_became_peered":"2024-09-18T08:06:24.140103+0000","last_unstale":"2024-09-18T08:06:25.117517+0000","last_undegraded":"2024-09-18T08:06:25.117517+0000","last_fullsized":"2024-09-18T08:06:25.117517+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'9","last_scrub_stamp":"2024-09-18T08:05:08.696796+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_clean_scrub_stamp":"2024-09-18T08:05:08.696796+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:06:33.226832+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":175,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-09-18T08:06:33.194009+0000","last_change":"2024-09-18T08:05:43.668557+0000","last_active":"2024-09-18T08:06:33.194009+0000","last_peered":"2024-09-18T08:06:33.194009+0000","last_clean":"2024-09-18T08:06:33.194009+0000","last_became_active":"2024-09-18T08:02:51.907140+0000","last_became_peered":"2024-09-18T08:02:51.907140+0000","last_unstale":"2024-09-18T08:06:33.194009+0000","last_undegraded":"2024-09-18T08:06:33.194009+0000","last_fullsized":"2024-09-18T08:06:33.194009+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:05:43.668271+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:50.886830+0000","last_clean_scrub_stamp":"2024-09-18T08:05:43.668271+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:07:02.226478+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":154,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-09-18T08:06:25.116937+0000","last_change":"2024-09-18T08:02:53.906914+0000","last_active":"2024-09-18T08:06:25.116937+0000","last_peered":"2024-09-18T08:06:25.116937+0000","last_clean":"2024-09-18T08:06:25.116937+0000","last_became_active":"2024-09-18T08:02:53.906622+0000","last_became_peered":"2024-09-18T08:02:53.906622+0000","last_unstale":"2024-09-18T08:06:25.116937+0000","last_undegraded":"2024-09-18T08:06:25.116937+0000","last_fullsized":"2024-09-18T08:06:25.116937+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_clean_scrub_stamp":"2024-09-18T08:02:52.890475+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:07:04.364627+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":177,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-09-18T08:06:33.179011+0000","last_change":"2024-09-18T08:06:24.128885+0000","last_active":"2024-09-18T08:06:33.179011+0000","last_peered":"2024-09-18T08:06:33.179011+0000","last_clean":"2024-09-18T08:06:33.179011+0000","last_became_active":"2024-09-18T08:06:24.128511+0000","last_became_peered":"2024-09-18T08:06:24.128511+0000","last_unstale":"2024-09-18T08:06:33.179011+0000","last_undegraded":"2024-09-18T08:06:33.179011+0000","last_fullsized":"2024-09-18T08:06:33.179011+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:05:03.047120+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_clean_scrub_stamp":"2024-09-18T08:05:03.047120+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":189,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-09-18T08:06:33.183174+0000","last_change":"2024-09-18T08:06:24.126544+0000","last_active":"2024-09-18T08:06:33.183174+0000","last_peered":"2024-09-18T08:06:33.183174+0000","last_clean":"2024-09-18T08:06:33.183174+0000","last_became_active":"2024-09-18T08:06:24.126194+0000","last_became_peered":"2024-09-18T08:06:24.126194+0000","last_unstale":"2024-09-18T08:06:33.183174+0000","last_undegraded":"2024-09-18T08:06:33.183174+0000","last_fullsized":"2024-09-18T08:06:33.183174+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:04:57.047904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:46.860092+0000","last_clean_scrub_stamp":"2024-09-18T08:04:57.047904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"69'15","reported_seq":188,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-09-18T08:06:33.180127+0000","last_change":"2024-09-18T08:02:49.887111+0000","last_active":"2024-09-18T08:06:33.180127+0000","last_peered":"2024-09-18T08:06:33.180127+0000","last_clean":"2024-09-18T08:06:33.180127+0000","last_became_active":"2024-09-18T08:02:49.886106+0000","last_became_peered":"2024-09-18T08:02:49.886106+0000","last_unstale":"2024-09-18T08:06:33.180127+0000","last_undegraded":"2024-09-18T08:06:33.180127+0000","last_fullsized":"2024-09-18T08:06:33.180127+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_clean_scrub_stamp":"2024-09-18T08:02:48.870718+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:06:51.053554+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":179,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-09-18T08:06:25.116856+0000","last_change":"2024-09-18T08:06:24.146536+0000","last_active":"2024-09-18T08:06:25.116856+0000","last_peered":"2024-09-18T08:06:25.116856+0000","last_clean":"2024-09-18T08:06:25.116856+0000","last_became_active":"2024-09-18T08:06:24.144841+0000","last_became_peered":"2024-09-18T08:06:24.144841+0000","last_unstale":"2024-09-18T08:06:25.116856+0000","last_undegraded":"2024-09-18T08:06:25.116856+0000","last_fullsized":"2024-09-18T08:06:25.116856+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:04:29.435669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:50.886830+0000","last_clean_scrub_stamp":"2024-09-18T08:04:29.435669+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":195,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-09-18T08:06:25.162269+0000","last_change":"2024-09-18T08:06:24.132542+0000","last_active":"2024-09-18T08:06:25.162269+0000","last_peered":"2024-09-18T08:06:25.162269+0000","last_clean":"2024-09-18T08:06:25.162269+0000","last_became_active":"2024-09-18T08:06:24.132206+0000","last_became_peered":"2024-09-18T08:06:24.132206+0000","last_unstale":"2024-09-18T08:06:25.162269+0000","last_undegraded":"2024-09-18T08:06:25.162269+0000","last_fullsized":"2024-09-18T08:06:25.162269+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:06:14.027077+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:46.860092+0000","last_clean_scrub_stamp":"2024-09-18T08:06:14.027077+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:07:24.620825+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"69'9","reported_seq":177,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-09-18T08:06:25.117138+0000","last_change":"2024-09-18T08:02:49.887589+0000","last_active":"2024-09-18T08:06:25.117138+0000","last_peered":"2024-09-18T08:06:25.117138+0000","last_clean":"2024-09-18T08:06:25.117138+0000","last_became_active":"2024-09-18T08:02:49.887161+0000","last_became_peered":"2024-09-18T08:02:49.887161+0000","last_unstale":"2024-09-18T08:06:25.117138+0000","last_undegraded":"2024-09-18T08:06:25.117138+0000","last_fullsized":"2024-09-18T08:06:25.117138+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_clean_scrub_stamp":"2024-09-18T08:02:48.870718+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:06:59.372043+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":173,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-09-18T08:06:25.161734+0000","last_change":"2024-09-18T08:05:33.935844+0000","last_active":"2024-09-18T08:06:25.161734+0000","last_peered":"2024-09-18T08:06:25.161734+0000","last_clean":"2024-09-18T08:06:25.161734+0000","last_became_active":"2024-09-18T08:02:51.907296+0000","last_became_peered":"2024-09-18T08:02:51.907296+0000","last_unstale":"2024-09-18T08:06:25.161734+0000","last_undegraded":"2024-09-18T08:06:25.161734+0000","last_fullsized":"2024-09-18T08:06:25.161734+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:05:33.935798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:50.886830+0000","last_clean_scrub_stamp":"2024-09-18T08:05:33.935798+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:06:48.799006+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":164,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-09-18T08:06:33.177413+0000","last_change":"2024-09-18T08:05:01.314402+0000","last_active":"2024-09-18T08:06:33.177413+0000","last_peered":"2024-09-18T08:06:33.177413+0000","last_clean":"2024-09-18T08:06:33.177413+0000","last_became_active":"2024-09-18T08:02:53.913519+0000","last_became_peered":"2024-09-18T08:02:53.913519+0000","last_unstale":"2024-09-18T08:06:33.177413+0000","last_undegraded":"2024-09-18T08:06:33.177413+0000","last_fullsized":"2024-09-18T08:06:33.177413+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:05:01.314330+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_clean_scrub_stamp":"2024-09-18T08:05:01.314330+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":173,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-09-18T08:06:25.162320+0000","last_change":"2024-09-18T08:04:32.849176+0000","last_active":"2024-09-18T08:06:25.162320+0000","last_peered":"2024-09-18T08:06:25.162320+0000","last_clean":"2024-09-18T08:06:25.162320+0000","last_became_active":"2024-09-18T08:02:47.875042+0000","last_became_peered":"2024-09-18T08:02:47.875042+0000","last_unstale":"2024-09-18T08:06:25.162320+0000","last_undegraded":"2024-09-18T08:06:25.162320+0000","last_fullsized":"2024-09-18T08:06:25.162320+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:04:32.849119+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:46.860092+0000","last_clean_scrub_stamp":"2024-09-18T08:04:32.849119+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:07:03.937944+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1b","version":"69'5","reported_seq":187,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-09-18T08:06:25.117178+0000","last_change":"2024-09-18T08:06:09.466075+0000","last_active":"2024-09-18T08:06:25.117178+0000","last_peered":"2024-09-18T08:06:25.117178+0000","last_clean":"2024-09-18T08:06:25.117178+0000","last_became_active":"2024-09-18T08:02:49.895344+0000","last_became_peered":"2024-09-18T08:02:49.895344+0000","last_unstale":"2024-09-18T08:06:25.117178+0000","last_undegraded":"2024-09-18T08:06:25.117178+0000","last_fullsized":"2024-09-18T08:06:25.117178+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'5","last_scrub_stamp":"2024-09-18T08:06:09.465979+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_clean_scrub_stamp":"2024-09-18T08:06:09.465979+0000","objects_scrubbed":2,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:07:37.799450+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":11,"num_read_kb":7,"num_write":6,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":173,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-09-18T08:06:25.120921+0000","last_change":"2024-09-18T08:05:38.866695+0000","last_active":"2024-09-18T08:06:25.120921+0000","last_peered":"2024-09-18T08:06:25.120921+0000","last_clean":"2024-09-18T08:06:25.120921+0000","last_became_active":"2024-09-18T08:02:51.905776+0000","last_became_peered":"2024-09-18T08:02:51.905776+0000","last_unstale":"2024-09-18T08:06:25.120921+0000","last_undegraded":"2024-09-18T08:06:25.120921+0000","last_fullsized":"2024-09-18T08:06:25.120921+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:05:38.866651+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:50.886830+0000","last_clean_scrub_stamp":"2024-09-18T08:05:38.866651+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:06:53.681027+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":162,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-09-18T08:06:25.161491+0000","last_change":"2024-09-18T08:05:39.843588+0000","last_active":"2024-09-18T08:06:25.161491+0000","last_peered":"2024-09-18T08:06:25.161491+0000","last_clean":"2024-09-18T08:06:25.161491+0000","last_became_active":"2024-09-18T08:02:53.911648+0000","last_became_peered":"2024-09-18T08:02:53.911648+0000","last_unstale":"2024-09-18T08:06:25.161491+0000","last_undegraded":"2024-09-18T08:06:25.161491+0000","last_fullsized":"2024-09-18T08:06:25.161491+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:05:39.843536+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_clean_scrub_stamp":"2024-09-18T08:05:39.843536+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:06:40.362817+0000","scrub_duration":20,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":175,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-09-18T08:06:25.117721+0000","last_change":"2024-09-18T08:06:24.132553+0000","last_active":"2024-09-18T08:06:25.117721+0000","last_peered":"2024-09-18T08:06:25.117721+0000","last_clean":"2024-09-18T08:06:25.117721+0000","last_became_active":"2024-09-18T08:06:24.132228+0000","last_became_peered":"2024-09-18T08:06:24.132228+0000","last_unstale":"2024-09-18T08:06:25.117721+0000","last_undegraded":"2024-09-18T08:06:25.117721+0000","last_fullsized":"2024-09-18T08:06:25.117721+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:05:40.477965+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_clean_scrub_stamp":"2024-09-18T08:05:40.477965+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:06:55.439381+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":183,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-09-18T08:06:33.181459+0000","last_change":"2024-09-18T08:05:27.476211+0000","last_active":"2024-09-18T08:06:33.181459+0000","last_peered":"2024-09-18T08:06:33.181459+0000","last_clean":"2024-09-18T08:06:33.181459+0000","last_became_active":"2024-09-18T08:02:47.875963+0000","last_became_peered":"2024-09-18T08:02:47.875963+0000","last_unstale":"2024-09-18T08:06:33.181459+0000","last_undegraded":"2024-09-18T08:06:33.181459+0000","last_fullsized":"2024-09-18T08:06:33.181459+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:05:27.476144+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:46.860092+0000","last_clean_scrub_stamp":"2024-09-18T08:05:27.476144+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:06:42.199865+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1c","version":"69'15","reported_seq":227,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-09-18T08:06:33.170147+0000","last_change":"2024-09-18T08:05:28.084324+0000","last_active":"2024-09-18T08:06:33.170147+0000","last_peered":"2024-09-18T08:06:33.170147+0000","last_clean":"2024-09-18T08:06:33.170147+0000","last_became_active":"2024-09-18T08:04:18.147907+0000","last_became_peered":"2024-09-18T08:04:18.147907+0000","last_unstale":"2024-09-18T08:06:33.170147+0000","last_undegraded":"2024-09-18T08:06:33.170147+0000","last_fullsized":"2024-09-18T08:06:33.170147+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'15","last_scrub_stamp":"2024-09-18T08:05:28.084220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_clean_scrub_stamp":"2024-09-18T08:05:28.084220+0000","objects_scrubbed":7,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:06:39.486316+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,3],"acting":[2,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":175,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-09-18T08:06:33.188342+0000","last_change":"2024-09-18T08:05:40.779422+0000","last_active":"2024-09-18T08:06:33.188342+0000","last_peered":"2024-09-18T08:06:33.188342+0000","last_clean":"2024-09-18T08:06:33.188342+0000","last_became_active":"2024-09-18T08:02:51.908430+0000","last_became_peered":"2024-09-18T08:02:51.908430+0000","last_unstale":"2024-09-18T08:06:33.188342+0000","last_undegraded":"2024-09-18T08:06:33.188342+0000","last_fullsized":"2024-09-18T08:06:33.188342+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:05:40.779230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:50.886830+0000","last_clean_scrub_stamp":"2024-09-18T08:05:40.779230+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:07:00.025019+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":177,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-09-18T08:06:33.179600+0000","last_change":"2024-09-18T08:06:24.129031+0000","last_active":"2024-09-18T08:06:33.179600+0000","last_peered":"2024-09-18T08:06:33.179600+0000","last_clean":"2024-09-18T08:06:33.179600+0000","last_became_active":"2024-09-18T08:06:24.128659+0000","last_became_peered":"2024-09-18T08:06:24.128659+0000","last_unstale":"2024-09-18T08:06:33.179600+0000","last_undegraded":"2024-09-18T08:06:33.179600+0000","last_fullsized":"2024-09-18T08:06:33.179600+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:04:29.955822+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_clean_scrub_stamp":"2024-09-18T08:04:29.955822+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":173,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-09-18T08:06:25.117277+0000","last_change":"2024-09-18T08:04:22.420158+0000","last_active":"2024-09-18T08:06:25.117277+0000","last_peered":"2024-09-18T08:06:25.117277+0000","last_clean":"2024-09-18T08:06:25.117277+0000","last_became_active":"2024-09-18T08:02:47.871523+0000","last_became_peered":"2024-09-18T08:02:47.871523+0000","last_unstale":"2024-09-18T08:06:25.117277+0000","last_undegraded":"2024-09-18T08:06:25.117277+0000","last_fullsized":"2024-09-18T08:06:25.117277+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:04:22.420112+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:46.860092+0000","last_clean_scrub_stamp":"2024-09-18T08:04:22.420112+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:07:06.346239+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"69'12","reported_seq":194,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-09-18T08:06:33.179406+0000","last_change":"2024-09-18T08:05:47.068485+0000","last_active":"2024-09-18T08:06:33.179406+0000","last_peered":"2024-09-18T08:06:33.179406+0000","last_clean":"2024-09-18T08:06:33.179406+0000","last_became_active":"2024-09-18T08:02:49.888831+0000","last_became_peered":"2024-09-18T08:02:49.888831+0000","last_unstale":"2024-09-18T08:06:33.179406+0000","last_undegraded":"2024-09-18T08:06:33.179406+0000","last_fullsized":"2024-09-18T08:06:33.179406+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'12","last_scrub_stamp":"2024-09-18T08:05:47.068375+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_clean_scrub_stamp":"2024-09-18T08:05:47.068375+0000","objects_scrubbed":5,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:06:50.263718+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":25,"num_read_kb":16,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":173,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-09-18T08:06:25.117391+0000","last_change":"2024-09-18T08:06:12.434234+0000","last_active":"2024-09-18T08:06:25.117391+0000","last_peered":"2024-09-18T08:06:25.117391+0000","last_clean":"2024-09-18T08:06:25.117391+0000","last_became_active":"2024-09-18T08:02:51.910316+0000","last_became_peered":"2024-09-18T08:02:51.910316+0000","last_unstale":"2024-09-18T08:06:25.117391+0000","last_undegraded":"2024-09-18T08:06:25.117391+0000","last_fullsized":"2024-09-18T08:06:25.117391+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:06:12.434179+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:50.886830+0000","last_clean_scrub_stamp":"2024-09-18T08:06:12.434179+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:07:35.662786+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1c","version":"69'1","reported_seq":163,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-09-18T08:06:25.120385+0000","last_change":"2024-09-18T08:05:36.776441+0000","last_active":"2024-09-18T08:06:25.120385+0000","last_peered":"2024-09-18T08:06:25.120385+0000","last_clean":"2024-09-18T08:06:25.120385+0000","last_became_active":"2024-09-18T08:02:53.910589+0000","last_became_peered":"2024-09-18T08:02:53.910589+0000","last_unstale":"2024-09-18T08:06:25.120385+0000","last_undegraded":"2024-09-18T08:06:25.120385+0000","last_fullsized":"2024-09-18T08:06:25.120385+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'1","last_scrub_stamp":"2024-09-18T08:05:36.776369+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_clean_scrub_stamp":"2024-09-18T08:05:36.776369+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:06:36.944612+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":403,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.19","version":"62'1","reported_seq":176,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-09-18T08:06:33.194151+0000","last_change":"2024-09-18T08:05:06.819830+0000","last_active":"2024-09-18T08:06:33.194151+0000","last_peered":"2024-09-18T08:06:33.194151+0000","last_clean":"2024-09-18T08:06:33.194151+0000","last_became_active":"2024-09-18T08:02:47.880998+0000","last_became_peered":"2024-09-18T08:02:47.880998+0000","last_unstale":"2024-09-18T08:06:33.194151+0000","last_undegraded":"2024-09-18T08:06:33.194151+0000","last_fullsized":"2024-09-18T08:06:33.194151+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"62'1","last_scrub_stamp":"2024-09-18T08:05:06.819539+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:46.860092+0000","last_clean_scrub_stamp":"2024-09-18T08:05:06.819539+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1e","version":"69'10","reported_seq":178,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-09-18T08:06:33.191307+0000","last_change":"2024-09-18T08:02:49.892755+0000","last_active":"2024-09-18T08:06:33.191307+0000","last_peered":"2024-09-18T08:06:33.191307+0000","last_clean":"2024-09-18T08:06:33.191307+0000","last_became_active":"2024-09-18T08:02:49.892024+0000","last_became_peered":"2024-09-18T08:02:49.892024+0000","last_unstale":"2024-09-18T08:06:33.191307+0000","last_undegraded":"2024-09-18T08:06:33.191307+0000","last_fullsized":"2024-09-18T08:06:33.191307+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_clean_scrub_stamp":"2024-09-18T08:02:48.870718+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:06:59.489315+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1f","version":"138'11","reported_seq":168,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-09-18T08:06:27.973440+0000","last_change":"2024-09-18T08:06:24.180060+0000","last_active":"2024-09-18T08:06:27.973440+0000","last_peered":"2024-09-18T08:06:27.973440+0000","last_clean":"2024-09-18T08:06:27.973440+0000","last_became_active":"2024-09-18T08:06:24.145360+0000","last_became_peered":"2024-09-18T08:06:24.145360+0000","last_unstale":"2024-09-18T08:06:27.973440+0000","last_undegraded":"2024-09-18T08:06:27.973440+0000","last_fullsized":"2024-09-18T08:06:27.973440+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"102'9","last_scrub_stamp":"2024-09-18T08:05:44.463780+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:50.886830+0000","last_clean_scrub_stamp":"2024-09-18T08:05:44.463780+0000","objects_scrubbed":1,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:07:07.778823+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,7],"acting":[4,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.c","version":"139'10906","reported_seq":12927,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-09-18T08:06:33.438451+0000","last_change":"2024-09-18T08:06:25.149370+0000","last_active":"2024-09-18T08:06:33.438451+0000","last_peered":"2024-09-18T08:06:33.438451+0000","last_clean":"2024-09-18T08:06:33.438451+0000","last_became_active":"2024-09-18T08:06:25.123363+0000","last_became_peered":"2024-09-18T08:06:25.123363+0000","last_unstale":"2024-09-18T08:06:33.438451+0000","last_undegraded":"2024-09-18T08:06:33.438451+0000","last_fullsized":"2024-09-18T08:06:33.438451+0000","mapping_epoch":137,"log_start":"110'6218","ondisk_log_start":"110'6218","created":83,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:03:56.449957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:03:56.449957+0000","last_clean_scrub_stamp":"2024-09-18T08:03:56.449957+0000","objects_scrubbed":0,"log_size":4688,"log_dups_size":0,"ondisk_log_size":4688,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:07:33.267946+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":268959744,"num_objects":4104,"num_object_clones":0,"num_object_copies":12312,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4104,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10906,"num_write_kb":480320,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":940,"num_bytes_recovered":61603840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,7],"acting":[3,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.f","version":"69'15","reported_seq":196,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-09-18T08:06:33.188623+0000","last_change":"2024-09-18T08:04:22.741342+0000","last_active":"2024-09-18T08:06:33.188623+0000","last_peered":"2024-09-18T08:06:33.188623+0000","last_clean":"2024-09-18T08:06:33.188623+0000","last_became_active":"2024-09-18T08:02:49.899083+0000","last_became_peered":"2024-09-18T08:02:49.899083+0000","last_unstale":"2024-09-18T08:06:33.188623+0000","last_undegraded":"2024-09-18T08:06:33.188623+0000","last_fullsized":"2024-09-18T08:06:33.188623+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'15","last_scrub_stamp":"2024-09-18T08:04:22.741261+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_clean_scrub_stamp":"2024-09-18T08:04:22.741261+0000","objects_scrubbed":7,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:06:53.535126+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":175,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-09-18T08:06:33.180731+0000","last_change":"2024-09-18T08:04:08.982535+0000","last_active":"2024-09-18T08:06:33.180731+0000","last_peered":"2024-09-18T08:06:33.180731+0000","last_clean":"2024-09-18T08:06:33.180731+0000","last_became_active":"2024-09-18T08:02:47.879328+0000","last_became_peered":"2024-09-18T08:02:47.879328+0000","last_unstale":"2024-09-18T08:06:33.180731+0000","last_undegraded":"2024-09-18T08:06:33.180731+0000","last_fullsized":"2024-09-18T08:06:33.180731+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:04:08.982469+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:46.860092+0000","last_clean_scrub_stamp":"2024-09-18T08:04:08.982469+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:06:57.148595+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.e","version":"69'8","reported_seq":227,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-09-18T08:06:27.973510+0000","last_change":"2024-09-18T08:06:19.459452+0000","last_active":"2024-09-18T08:06:27.973510+0000","last_peered":"2024-09-18T08:06:27.973510+0000","last_clean":"2024-09-18T08:06:27.973510+0000","last_became_active":"2024-09-18T08:02:51.903369+0000","last_became_peered":"2024-09-18T08:02:51.903369+0000","last_unstale":"2024-09-18T08:06:27.973510+0000","last_undegraded":"2024-09-18T08:06:27.973510+0000","last_fullsized":"2024-09-18T08:06:27.973510+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'8","last_scrub_stamp":"2024-09-18T08:06:19.459375+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:50.886830+0000","last_clean_scrub_stamp":"2024-09-18T08:06:19.459375+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:07:30.186221+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":170,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-09-18T08:06:25.161876+0000","last_change":"2024-09-18T08:05:34.905147+0000","last_active":"2024-09-18T08:06:25.161876+0000","last_peered":"2024-09-18T08:06:25.161876+0000","last_clean":"2024-09-18T08:06:25.161876+0000","last_became_active":"2024-09-18T08:02:53.909834+0000","last_became_peered":"2024-09-18T08:02:53.909834+0000","last_unstale":"2024-09-18T08:06:25.161876+0000","last_undegraded":"2024-09-18T08:06:25.161876+0000","last_fullsized":"2024-09-18T08:06:25.161876+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:05:34.905083+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_clean_scrub_stamp":"2024-09-18T08:05:34.905083+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:06:53.452264+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.3","version":"139'10673","reported_seq":10797,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-09-18T08:06:33.421325+0000","last_change":"2024-09-18T08:03:57.462489+0000","last_active":"2024-09-18T08:06:33.421325+0000","last_peered":"2024-09-18T08:06:33.421325+0000","last_clean":"2024-09-18T08:06:33.421325+0000","last_became_active":"2024-09-18T08:03:57.462210+0000","last_became_peered":"2024-09-18T08:03:57.462210+0000","last_unstale":"2024-09-18T08:06:33.421325+0000","last_undegraded":"2024-09-18T08:06:33.421325+0000","last_fullsized":"2024-09-18T08:06:33.421325+0000","mapping_epoch":83,"log_start":"109'6022","ondisk_log_start":"109'6022","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:03:56.449957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:03:56.449957+0000","last_clean_scrub_stamp":"2024-09-18T08:03:56.449957+0000","objects_scrubbed":0,"log_size":4651,"log_dups_size":0,"ondisk_log_size":4651,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:07:30.236230+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262209536,"num_objects":4001,"num_object_clones":0,"num_object_copies":12003,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4001,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10673,"num_write_kb":469568,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,4],"acting":[3,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.0","version":"69'18","reported_seq":203,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-09-18T08:06:33.180608+0000","last_change":"2024-09-18T08:04:14.969799+0000","last_active":"2024-09-18T08:06:33.180608+0000","last_peered":"2024-09-18T08:06:33.180608+0000","last_clean":"2024-09-18T08:06:33.180608+0000","last_became_active":"2024-09-18T08:02:49.886818+0000","last_became_peered":"2024-09-18T08:02:49.886818+0000","last_unstale":"2024-09-18T08:06:33.180608+0000","last_undegraded":"2024-09-18T08:06:33.180608+0000","last_fullsized":"2024-09-18T08:06:33.180608+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'18","last_scrub_stamp":"2024-09-18T08:04:14.969716+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_clean_scrub_stamp":"2024-09-18T08:04:14.969716+0000","objects_scrubbed":8,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":34,"num_read_kb":22,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":183,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-09-18T08:06:33.180072+0000","last_change":"2024-09-18T08:05:35.100407+0000","last_active":"2024-09-18T08:06:33.180072+0000","last_peered":"2024-09-18T08:06:33.180072+0000","last_clean":"2024-09-18T08:06:33.180072+0000","last_became_active":"2024-09-18T08:02:47.880225+0000","last_became_peered":"2024-09-18T08:02:47.880225+0000","last_unstale":"2024-09-18T08:06:33.180072+0000","last_undegraded":"2024-09-18T08:06:33.180072+0000","last_fullsized":"2024-09-18T08:06:33.180072+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:05:35.100332+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:46.860092+0000","last_clean_scrub_stamp":"2024-09-18T08:05:35.100332+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:06:57.779942+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":173,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-09-18T08:06:25.116946+0000","last_change":"2024-09-18T08:06:10.468172+0000","last_active":"2024-09-18T08:06:25.116946+0000","last_peered":"2024-09-18T08:06:25.116946+0000","last_clean":"2024-09-18T08:06:25.116946+0000","last_became_active":"2024-09-18T08:02:51.899594+0000","last_became_peered":"2024-09-18T08:02:51.899594+0000","last_unstale":"2024-09-18T08:06:25.116946+0000","last_undegraded":"2024-09-18T08:06:25.116946+0000","last_fullsized":"2024-09-18T08:06:25.116946+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:06:10.468119+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:50.886830+0000","last_clean_scrub_stamp":"2024-09-18T08:06:10.468119+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:07:23.534443+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":162,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-09-18T08:06:25.117085+0000","last_change":"2024-09-18T08:04:16.427470+0000","last_active":"2024-09-18T08:06:25.117085+0000","last_peered":"2024-09-18T08:06:25.117085+0000","last_clean":"2024-09-18T08:06:25.117085+0000","last_became_active":"2024-09-18T08:02:53.903957+0000","last_became_peered":"2024-09-18T08:02:53.903957+0000","last_unstale":"2024-09-18T08:06:25.117085+0000","last_undegraded":"2024-09-18T08:06:25.117085+0000","last_fullsized":"2024-09-18T08:06:25.117085+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:04:16.427390+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_clean_scrub_stamp":"2024-09-18T08:04:16.427390+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.2","version":"139'10775","reported_seq":10899,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-09-18T08:06:33.394794+0000","last_change":"2024-09-18T08:03:57.463750+0000","last_active":"2024-09-18T08:06:33.394794+0000","last_peered":"2024-09-18T08:06:33.394794+0000","last_clean":"2024-09-18T08:06:33.394794+0000","last_became_active":"2024-09-18T08:03:57.463459+0000","last_became_peered":"2024-09-18T08:03:57.463459+0000","last_unstale":"2024-09-18T08:06:33.394794+0000","last_undegraded":"2024-09-18T08:06:33.394794+0000","last_fullsized":"2024-09-18T08:06:33.394794+0000","mapping_epoch":83,"log_start":"110'6143","ondisk_log_start":"110'6143","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:03:56.449957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:03:56.449957+0000","last_clean_scrub_stamp":"2024-09-18T08:03:56.449957+0000","objects_scrubbed":0,"log_size":4632,"log_dups_size":0,"ondisk_log_size":4632,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:06:59.160354+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":265224192,"num_objects":4047,"num_object_clones":0,"num_object_copies":12141,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4047,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10775,"num_write_kb":474304,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"69'14","reported_seq":244,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-09-18T08:06:25.117003+0000","last_change":"2024-09-18T08:06:24.184801+0000","last_active":"2024-09-18T08:06:25.117003+0000","last_peered":"2024-09-18T08:06:25.117003+0000","last_clean":"2024-09-18T08:06:25.117003+0000","last_became_active":"2024-09-18T08:06:24.133949+0000","last_became_peered":"2024-09-18T08:06:24.133949+0000","last_unstale":"2024-09-18T08:06:25.117003+0000","last_undegraded":"2024-09-18T08:06:25.117003+0000","last_fullsized":"2024-09-18T08:06:25.117003+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'14","last_scrub_stamp":"2024-09-18T08:04:56.545262+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_clean_scrub_stamp":"2024-09-18T08:04:56.545262+0000","objects_scrubbed":7,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":21,"num_read_kb":14,"num_write":14,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"62'2","reported_seq":182,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-09-18T08:06:33.191637+0000","last_change":"2024-09-18T08:04:09.240465+0000","last_active":"2024-09-18T08:06:33.191637+0000","last_peered":"2024-09-18T08:06:33.191637+0000","last_clean":"2024-09-18T08:06:33.191637+0000","last_became_active":"2024-09-18T08:02:47.872857+0000","last_became_peered":"2024-09-18T08:02:47.872857+0000","last_unstale":"2024-09-18T08:06:33.191637+0000","last_undegraded":"2024-09-18T08:06:33.191637+0000","last_fullsized":"2024-09-18T08:06:33.191637+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"62'2","last_scrub_stamp":"2024-09-18T08:04:09.240392+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:46.860092+0000","last_clean_scrub_stamp":"2024-09-18T08:04:09.240392+0000","objects_scrubbed":2,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1039,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.0","version":"69'8","reported_seq":221,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-09-18T08:06:33.180375+0000","last_change":"2024-09-18T08:04:23.863573+0000","last_active":"2024-09-18T08:06:33.180375+0000","last_peered":"2024-09-18T08:06:33.180375+0000","last_clean":"2024-09-18T08:06:33.180375+0000","last_became_active":"2024-09-18T08:02:51.909680+0000","last_became_peered":"2024-09-18T08:02:51.909680+0000","last_unstale":"2024-09-18T08:06:33.180375+0000","last_undegraded":"2024-09-18T08:06:33.180375+0000","last_fullsized":"2024-09-18T08:06:33.180375+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'8","last_scrub_stamp":"2024-09-18T08:04:23.863330+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:50.886830+0000","last_clean_scrub_stamp":"2024-09-18T08:04:23.863330+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":176,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-09-18T08:06:25.120390+0000","last_change":"2024-09-18T08:06:24.144574+0000","last_active":"2024-09-18T08:06:25.120390+0000","last_peered":"2024-09-18T08:06:25.120390+0000","last_clean":"2024-09-18T08:06:25.120390+0000","last_became_active":"2024-09-18T08:06:24.140303+0000","last_became_peered":"2024-09-18T08:06:24.140303+0000","last_unstale":"2024-09-18T08:06:25.120390+0000","last_undegraded":"2024-09-18T08:06:25.120390+0000","last_fullsized":"2024-09-18T08:06:25.120390+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:05:06.885311+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_clean_scrub_stamp":"2024-09-18T08:05:06.885311+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:06:28.219450+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.1","version":"138'10804","reported_seq":10938,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-09-18T08:06:33.093580+0000","last_change":"2024-09-18T08:06:24.166578+0000","last_active":"2024-09-18T08:06:33.093580+0000","last_peered":"2024-09-18T08:06:33.093580+0000","last_clean":"2024-09-18T08:06:33.093580+0000","last_became_active":"2024-09-18T08:06:24.134315+0000","last_became_peered":"2024-09-18T08:06:24.134315+0000","last_unstale":"2024-09-18T08:06:33.093580+0000","last_undegraded":"2024-09-18T08:06:33.093580+0000","last_fullsized":"2024-09-18T08:06:33.093580+0000","mapping_epoch":136,"log_start":"108'5957","ondisk_log_start":"108'5957","created":83,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:03:56.449957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:03:56.449957+0000","last_clean_scrub_stamp":"2024-09-18T08:03:56.449957+0000","objects_scrubbed":0,"log_size":4847,"log_dups_size":0,"ondisk_log_size":4847,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:07:27.452504+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":265027584,"num_objects":4044,"num_object_clones":0,"num_object_copies":12132,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4044,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10804,"num_write_kb":475136,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.2","version":"69'10","reported_seq":194,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-09-18T08:06:33.187986+0000","last_change":"2024-09-18T08:05:39.769686+0000","last_active":"2024-09-18T08:06:33.187986+0000","last_peered":"2024-09-18T08:06:33.187986+0000","last_clean":"2024-09-18T08:06:33.187986+0000","last_became_active":"2024-09-18T08:02:49.895265+0000","last_became_peered":"2024-09-18T08:02:49.895265+0000","last_unstale":"2024-09-18T08:06:33.187986+0000","last_undegraded":"2024-09-18T08:06:33.187986+0000","last_fullsized":"2024-09-18T08:06:33.187986+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'10","last_scrub_stamp":"2024-09-18T08:05:39.768393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_clean_scrub_stamp":"2024-09-18T08:05:39.768393+0000","objects_scrubbed":5,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:07:07.361416+0000","scrub_duration":17,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":173,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-09-18T08:06:25.162353+0000","last_change":"2024-09-18T08:03:48.009689+0000","last_active":"2024-09-18T08:06:25.162353+0000","last_peered":"2024-09-18T08:06:25.162353+0000","last_clean":"2024-09-18T08:06:25.162353+0000","last_became_active":"2024-09-18T08:02:47.874727+0000","last_became_peered":"2024-09-18T08:02:47.874727+0000","last_unstale":"2024-09-18T08:06:25.162353+0000","last_undegraded":"2024-09-18T08:06:25.162353+0000","last_fullsized":"2024-09-18T08:06:25.162353+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:03:48.009611+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:46.860092+0000","last_clean_scrub_stamp":"2024-09-18T08:03:48.009611+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:07:24.213853+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.3","version":"138'10","reported_seq":244,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-09-18T08:06:33.178092+0000","last_change":"2024-09-18T08:06:24.157627+0000","last_active":"2024-09-18T08:06:33.178092+0000","last_peered":"2024-09-18T08:06:33.178092+0000","last_clean":"2024-09-18T08:06:33.178092+0000","last_became_active":"2024-09-18T08:06:24.134194+0000","last_became_peered":"2024-09-18T08:06:24.134194+0000","last_unstale":"2024-09-18T08:06:33.178092+0000","last_undegraded":"2024-09-18T08:06:33.178092+0000","last_fullsized":"2024-09-18T08:06:33.178092+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"130'9","last_scrub_stamp":"2024-09-18T08:06:13.462610+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:50.886830+0000","last_clean_scrub_stamp":"2024-09-18T08:06:13.462610+0000","objects_scrubbed":1,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:07:27.049204+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":172,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-09-18T08:06:33.177571+0000","last_change":"2024-09-18T08:05:43.440713+0000","last_active":"2024-09-18T08:06:33.177571+0000","last_peered":"2024-09-18T08:06:33.177571+0000","last_clean":"2024-09-18T08:06:33.177571+0000","last_became_active":"2024-09-18T08:02:53.909583+0000","last_became_peered":"2024-09-18T08:02:53.909583+0000","last_unstale":"2024-09-18T08:06:33.177571+0000","last_undegraded":"2024-09-18T08:06:33.177571+0000","last_fullsized":"2024-09-18T08:06:33.177571+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:05:43.440642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_clean_scrub_stamp":"2024-09-18T08:05:43.440642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:07:02.845583+0000","scrub_duration":35,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.0","version":"138'10668","reported_seq":10851,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-09-18T08:06:33.080595+0000","last_change":"2024-09-18T08:05:01.419397+0000","last_active":"2024-09-18T08:06:33.080595+0000","last_peered":"2024-09-18T08:06:33.080595+0000","last_clean":"2024-09-18T08:06:33.080595+0000","last_became_active":"2024-09-18T08:03:57.468985+0000","last_became_peered":"2024-09-18T08:03:57.468985+0000","last_unstale":"2024-09-18T08:06:33.080595+0000","last_undegraded":"2024-09-18T08:06:33.080595+0000","last_fullsized":"2024-09-18T08:06:33.080595+0000","mapping_epoch":83,"log_start":"104'4832","ondisk_log_start":"104'4832","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"107'5286","last_scrub_stamp":"2024-09-18T08:05:01.419332+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:03:56.449957+0000","last_clean_scrub_stamp":"2024-09-18T08:05:01.419332+0000","objects_scrubbed":5263,"log_size":5836,"log_dups_size":0,"ondisk_log_size":5836,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:07:27.313345+0000","scrub_duration":1313,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262012928,"num_objects":3998,"num_object_clones":0,"num_object_copies":11994,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3998,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10668,"num_write_kb":469312,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.3","version":"69'19","reported_seq":207,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-09-18T08:06:33.181193+0000","last_change":"2024-09-18T08:04:07.231406+0000","last_active":"2024-09-18T08:06:33.181193+0000","last_peered":"2024-09-18T08:06:33.181193+0000","last_clean":"2024-09-18T08:06:33.181193+0000","last_became_active":"2024-09-18T08:02:49.890318+0000","last_became_peered":"2024-09-18T08:02:49.890318+0000","last_unstale":"2024-09-18T08:06:33.181193+0000","last_undegraded":"2024-09-18T08:06:33.181193+0000","last_fullsized":"2024-09-18T08:06:33.181193+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'19","last_scrub_stamp":"2024-09-18T08:04:07.229846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_clean_scrub_stamp":"2024-09-18T08:04:07.229846+0000","objects_scrubbed":8,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":39,"num_read_kb":25,"num_write":22,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.4","version":"62'1","reported_seq":181,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-09-18T08:06:33.188289+0000","last_change":"2024-09-18T08:04:28.827545+0000","last_active":"2024-09-18T08:06:33.188289+0000","last_peered":"2024-09-18T08:06:33.188289+0000","last_clean":"2024-09-18T08:06:33.188289+0000","last_became_active":"2024-09-18T08:02:47.874764+0000","last_became_peered":"2024-09-18T08:02:47.874764+0000","last_unstale":"2024-09-18T08:06:33.188289+0000","last_undegraded":"2024-09-18T08:06:33.188289+0000","last_fullsized":"2024-09-18T08:06:33.188289+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"62'1","last_scrub_stamp":"2024-09-18T08:04:28.827464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:46.860092+0000","last_clean_scrub_stamp":"2024-09-18T08:04:28.827464+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:06:52.521799+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":436,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":101,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-09-18T08:06:33.177664+0000","last_change":"2024-09-18T08:06:24.133954+0000","last_active":"2024-09-18T08:06:33.177664+0000","last_peered":"2024-09-18T08:06:33.177664+0000","last_clean":"2024-09-18T08:06:33.177664+0000","last_became_active":"2024-09-18T08:06:24.133262+0000","last_became_peered":"2024-09-18T08:06:24.133262+0000","last_unstale":"2024-09-18T08:06:33.177664+0000","last_undegraded":"2024-09-18T08:06:33.177664+0000","last_fullsized":"2024-09-18T08:06:33.177664+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:04:13.377561+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:50.886830+0000","last_clean_scrub_stamp":"2024-09-18T08:04:13.377561+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":185,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-09-18T08:06:33.193870+0000","last_change":"2024-09-18T08:06:24.139017+0000","last_active":"2024-09-18T08:06:33.193870+0000","last_peered":"2024-09-18T08:06:33.193870+0000","last_clean":"2024-09-18T08:06:33.193870+0000","last_became_active":"2024-09-18T08:06:24.138652+0000","last_became_peered":"2024-09-18T08:06:24.138652+0000","last_unstale":"2024-09-18T08:06:33.193870+0000","last_undegraded":"2024-09-18T08:06:33.193870+0000","last_fullsized":"2024-09-18T08:06:33.193870+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:06:10.507191+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_clean_scrub_stamp":"2024-09-18T08:06:10.507191+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:07:38.239809+0000","scrub_duration":25,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.7","version":"139'10808","reported_seq":12579,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-09-18T08:06:33.446761+0000","last_change":"2024-09-18T08:06:25.150134+0000","last_active":"2024-09-18T08:06:33.446761+0000","last_peered":"2024-09-18T08:06:33.446761+0000","last_clean":"2024-09-18T08:06:33.446761+0000","last_became_active":"2024-09-18T08:06:25.123728+0000","last_became_peered":"2024-09-18T08:06:25.123728+0000","last_unstale":"2024-09-18T08:06:33.446761+0000","last_undegraded":"2024-09-18T08:06:33.446761+0000","last_fullsized":"2024-09-18T08:06:33.446761+0000","mapping_epoch":137,"log_start":"107'5466","ondisk_log_start":"107'5466","created":83,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:03:56.449957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:03:56.449957+0000","last_clean_scrub_stamp":"2024-09-18T08:03:56.449957+0000","objects_scrubbed":4174,"log_size":5342,"log_dups_size":0,"ondisk_log_size":5342,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:07:31.458495+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":266600448,"num_objects":4068,"num_object_clones":0,"num_object_copies":12204,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4068,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10808,"num_write_kb":476032,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":787,"num_bytes_recovered":51511296,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,3],"acting":[0,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.4","version":"69'28","reported_seq":265,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-09-18T08:06:33.188178+0000","last_change":"2024-09-18T08:04:12.737618+0000","last_active":"2024-09-18T08:06:33.188178+0000","last_peered":"2024-09-18T08:06:33.188178+0000","last_clean":"2024-09-18T08:06:33.188178+0000","last_became_active":"2024-09-18T08:02:49.897221+0000","last_became_peered":"2024-09-18T08:02:49.897221+0000","last_unstale":"2024-09-18T08:06:33.188178+0000","last_undegraded":"2024-09-18T08:06:33.188178+0000","last_fullsized":"2024-09-18T08:06:33.188178+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'28","last_scrub_stamp":"2024-09-18T08:04:12.737528+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_clean_scrub_stamp":"2024-09-18T08:04:12.737528+0000","objects_scrubbed":10,"log_size":28,"log_dups_size":0,"ondisk_log_size":28,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:06:50.593181+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":48,"num_read_kb":33,"num_write":26,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":187,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-09-18T08:06:25.117506+0000","last_change":"2024-09-18T08:06:24.143321+0000","last_active":"2024-09-18T08:06:25.117506+0000","last_peered":"2024-09-18T08:06:25.117506+0000","last_clean":"2024-09-18T08:06:25.117506+0000","last_became_active":"2024-09-18T08:06:24.143019+0000","last_became_peered":"2024-09-18T08:06:24.143019+0000","last_unstale":"2024-09-18T08:06:25.117506+0000","last_undegraded":"2024-09-18T08:06:25.117506+0000","last_fullsized":"2024-09-18T08:06:25.117506+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:05:39.490141+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:46.860092+0000","last_clean_scrub_stamp":"2024-09-18T08:05:39.490141+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:06:48.790845+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"64'2","reported_seq":213,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-09-18T08:06:25.161854+0000","last_change":"2024-09-18T08:06:24.165227+0000","last_active":"2024-09-18T08:06:25.161854+0000","last_peered":"2024-09-18T08:06:25.161854+0000","last_clean":"2024-09-18T08:06:25.161854+0000","last_became_active":"2024-09-18T08:06:24.136080+0000","last_became_peered":"2024-09-18T08:06:24.136080+0000","last_unstale":"2024-09-18T08:06:25.161854+0000","last_undegraded":"2024-09-18T08:06:25.161854+0000","last_fullsized":"2024-09-18T08:06:25.161854+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"64'2","last_scrub_stamp":"2024-09-18T08:05:44.802942+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:45.855979+0000","last_clean_scrub_stamp":"2024-09-18T08:05:44.802942+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:06:50.488908+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.001348349,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":38,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":159,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-09-18T08:06:33.187899+0000","last_change":"2024-09-18T08:02:51.905694+0000","last_active":"2024-09-18T08:06:33.187899+0000","last_peered":"2024-09-18T08:06:33.187899+0000","last_clean":"2024-09-18T08:06:33.187899+0000","last_became_active":"2024-09-18T08:02:51.905357+0000","last_became_peered":"2024-09-18T08:02:51.905357+0000","last_unstale":"2024-09-18T08:06:33.187899+0000","last_undegraded":"2024-09-18T08:06:33.187899+0000","last_fullsized":"2024-09-18T08:06:33.187899+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:02:50.886830+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:50.886830+0000","last_clean_scrub_stamp":"2024-09-18T08:02:50.886830+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:06:51.299235+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":164,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-09-18T08:06:33.182914+0000","last_change":"2024-09-18T08:05:04.962900+0000","last_active":"2024-09-18T08:06:33.182914+0000","last_peered":"2024-09-18T08:06:33.182914+0000","last_clean":"2024-09-18T08:06:33.182914+0000","last_became_active":"2024-09-18T08:02:53.911036+0000","last_became_peered":"2024-09-18T08:02:53.911036+0000","last_unstale":"2024-09-18T08:06:33.182914+0000","last_undegraded":"2024-09-18T08:06:33.182914+0000","last_fullsized":"2024-09-18T08:06:33.182914+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:05:04.962686+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_clean_scrub_stamp":"2024-09-18T08:05:04.962686+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.4","version":"139'10831","reported_seq":10955,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-09-18T08:06:33.458733+0000","last_change":"2024-09-18T08:03:57.466551+0000","last_active":"2024-09-18T08:06:33.458733+0000","last_peered":"2024-09-18T08:06:33.458733+0000","last_clean":"2024-09-18T08:06:33.458733+0000","last_became_active":"2024-09-18T08:03:57.466232+0000","last_became_peered":"2024-09-18T08:03:57.466232+0000","last_unstale":"2024-09-18T08:06:33.458733+0000","last_undegraded":"2024-09-18T08:06:33.458733+0000","last_fullsized":"2024-09-18T08:06:33.458733+0000","mapping_epoch":83,"log_start":"107'5485","ondisk_log_start":"107'5485","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:03:56.449957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:03:56.449957+0000","last_clean_scrub_stamp":"2024-09-18T08:03:56.449957+0000","objects_scrubbed":0,"log_size":5346,"log_dups_size":0,"ondisk_log_size":5346,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":264306688,"num_objects":4033,"num_object_clones":0,"num_object_copies":12099,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4033,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10831,"num_write_kb":475648,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,1],"acting":[0,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.7","version":"69'13","reported_seq":117,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-09-18T08:06:33.168404+0000","last_change":"2024-09-18T08:04:18.164662+0000","last_active":"2024-09-18T08:06:33.168404+0000","last_peered":"2024-09-18T08:06:33.168404+0000","last_clean":"2024-09-18T08:06:33.168404+0000","last_became_active":"2024-09-18T08:04:18.144925+0000","last_became_peered":"2024-09-18T08:04:18.144925+0000","last_unstale":"2024-09-18T08:06:33.168404+0000","last_undegraded":"2024-09-18T08:06:33.168404+0000","last_fullsized":"2024-09-18T08:06:33.168404+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'13","last_scrub_stamp":"2024-09-18T08:03:50.700818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_clean_scrub_stamp":"2024-09-18T08:03:50.700818+0000","objects_scrubbed":0,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:07:29.027985+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":29,"num_read_kb":18,"num_write":16,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,0],"acting":[2,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":189,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-09-18T08:06:33.192944+0000","last_change":"2024-09-18T08:06:24.130858+0000","last_active":"2024-09-18T08:06:33.192944+0000","last_peered":"2024-09-18T08:06:33.192944+0000","last_clean":"2024-09-18T08:06:33.192944+0000","last_became_active":"2024-09-18T08:06:24.126334+0000","last_became_peered":"2024-09-18T08:06:24.126334+0000","last_unstale":"2024-09-18T08:06:33.192944+0000","last_undegraded":"2024-09-18T08:06:33.192944+0000","last_fullsized":"2024-09-18T08:06:33.192944+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:03:49.704004+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:46.860092+0000","last_clean_scrub_stamp":"2024-09-18T08:03:49.704004+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:07:31.622951+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"62'1","reported_seq":184,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-09-18T08:06:33.168249+0000","last_change":"2024-09-18T08:04:09.732724+0000","last_active":"2024-09-18T08:06:33.168249+0000","last_peered":"2024-09-18T08:06:33.168249+0000","last_clean":"2024-09-18T08:06:33.168249+0000","last_became_active":"2024-09-18T08:02:46.868175+0000","last_became_peered":"2024-09-18T08:02:46.868175+0000","last_unstale":"2024-09-18T08:06:33.168249+0000","last_undegraded":"2024-09-18T08:06:33.168249+0000","last_fullsized":"2024-09-18T08:06:33.168249+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"62'1","last_scrub_stamp":"2024-09-18T08:04:09.732645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:45.855979+0000","last_clean_scrub_stamp":"2024-09-18T08:04:09.732645+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0.00066785499999999997,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":175,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-09-18T08:06:33.173930+0000","last_change":"2024-09-18T08:05:41.059426+0000","last_active":"2024-09-18T08:06:33.173930+0000","last_peered":"2024-09-18T08:06:33.173930+0000","last_clean":"2024-09-18T08:06:33.173930+0000","last_became_active":"2024-09-18T08:02:51.908156+0000","last_became_peered":"2024-09-18T08:02:51.908156+0000","last_unstale":"2024-09-18T08:06:33.173930+0000","last_undegraded":"2024-09-18T08:06:33.173930+0000","last_fullsized":"2024-09-18T08:06:33.173930+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:05:41.059230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:50.886830+0000","last_clean_scrub_stamp":"2024-09-18T08:05:41.059230+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:06:45.166099+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":176,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-09-18T08:06:25.121145+0000","last_change":"2024-09-18T08:06:24.146209+0000","last_active":"2024-09-18T08:06:25.121145+0000","last_peered":"2024-09-18T08:06:25.121145+0000","last_clean":"2024-09-18T08:06:25.121145+0000","last_became_active":"2024-09-18T08:06:24.145382+0000","last_became_peered":"2024-09-18T08:06:24.145382+0000","last_unstale":"2024-09-18T08:06:25.121145+0000","last_undegraded":"2024-09-18T08:06:25.121145+0000","last_fullsized":"2024-09-18T08:06:25.121145+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:05:40.929345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_clean_scrub_stamp":"2024-09-18T08:05:40.929345+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:07:08.964083+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.5","version":"138'10705","reported_seq":10827,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-09-18T08:06:33.083964+0000","last_change":"2024-09-18T08:03:57.469151+0000","last_active":"2024-09-18T08:06:33.083964+0000","last_peered":"2024-09-18T08:06:33.083964+0000","last_clean":"2024-09-18T08:06:33.083964+0000","last_became_active":"2024-09-18T08:03:57.468853+0000","last_became_peered":"2024-09-18T08:03:57.468853+0000","last_unstale":"2024-09-18T08:06:33.083964+0000","last_undegraded":"2024-09-18T08:06:33.083964+0000","last_fullsized":"2024-09-18T08:06:33.083964+0000","mapping_epoch":83,"log_start":"108'5523","ondisk_log_start":"108'5523","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:03:56.449957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:03:56.449957+0000","last_clean_scrub_stamp":"2024-09-18T08:03:56.449957+0000","objects_scrubbed":0,"log_size":5182,"log_dups_size":0,"ondisk_log_size":5182,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:07:24.139200+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":264962048,"num_objects":4043,"num_object_clones":0,"num_object_copies":12129,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4043,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10705,"num_write_kb":471936,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,0],"acting":[7,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"69'12","reported_seq":189,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-09-18T08:06:33.191395+0000","last_change":"2024-09-18T08:05:38.471012+0000","last_active":"2024-09-18T08:06:33.191395+0000","last_peered":"2024-09-18T08:06:33.191395+0000","last_clean":"2024-09-18T08:06:33.191395+0000","last_became_active":"2024-09-18T08:02:49.898079+0000","last_became_peered":"2024-09-18T08:02:49.898079+0000","last_unstale":"2024-09-18T08:06:33.191395+0000","last_undegraded":"2024-09-18T08:06:33.191395+0000","last_fullsized":"2024-09-18T08:06:33.191395+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'12","last_scrub_stamp":"2024-09-18T08:05:38.470713+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_clean_scrub_stamp":"2024-09-18T08:05:38.470713+0000","objects_scrubbed":6,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:06:50.866624+0000","scrub_duration":28,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":18,"num_read_kb":12,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":175,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-09-18T08:06:33.177575+0000","last_change":"2024-09-18T08:04:59.298230+0000","last_active":"2024-09-18T08:06:33.177575+0000","last_peered":"2024-09-18T08:06:33.177575+0000","last_clean":"2024-09-18T08:06:33.177575+0000","last_became_active":"2024-09-18T08:02:47.873809+0000","last_became_peered":"2024-09-18T08:02:47.873809+0000","last_unstale":"2024-09-18T08:06:33.177575+0000","last_undegraded":"2024-09-18T08:06:33.177575+0000","last_fullsized":"2024-09-18T08:06:33.177575+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:04:59.298029+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:46.860092+0000","last_clean_scrub_stamp":"2024-09-18T08:04:59.298029+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":31,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"70'5","reported_seq":404,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-09-18T08:06:31.914189+0000","last_change":"2024-09-18T08:04:06.814615+0000","last_active":"2024-09-18T08:06:31.914189+0000","last_peered":"2024-09-18T08:06:31.914189+0000","last_clean":"2024-09-18T08:06:31.914189+0000","last_became_active":"2024-09-18T08:02:46.872144+0000","last_became_peered":"2024-09-18T08:02:46.872144+0000","last_unstale":"2024-09-18T08:06:31.914189+0000","last_undegraded":"2024-09-18T08:06:31.914189+0000","last_fullsized":"2024-09-18T08:06:31.914189+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"70'5","last_scrub_stamp":"2024-09-18T08:04:06.814540+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:45.855979+0000","last_clean_scrub_stamp":"2024-09-18T08:04:06.814540+0000","objects_scrubbed":1,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:07:29.176232+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.000504055,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":220,"num_read_kb":215,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":173,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-09-18T08:06:25.162881+0000","last_change":"2024-09-18T08:05:41.750515+0000","last_active":"2024-09-18T08:06:25.162881+0000","last_peered":"2024-09-18T08:06:25.162881+0000","last_clean":"2024-09-18T08:06:25.162881+0000","last_became_active":"2024-09-18T08:02:51.913112+0000","last_became_peered":"2024-09-18T08:02:51.913112+0000","last_unstale":"2024-09-18T08:06:25.162881+0000","last_undegraded":"2024-09-18T08:06:25.162881+0000","last_fullsized":"2024-09-18T08:06:25.162881+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:05:41.750446+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:50.886830+0000","last_clean_scrub_stamp":"2024-09-18T08:05:41.750446+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:06:55.866373+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":156,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-09-18T08:06:33.187905+0000","last_change":"2024-09-18T08:02:53.907689+0000","last_active":"2024-09-18T08:06:33.187905+0000","last_peered":"2024-09-18T08:06:33.187905+0000","last_clean":"2024-09-18T08:06:33.187905+0000","last_became_active":"2024-09-18T08:02:53.907404+0000","last_became_peered":"2024-09-18T08:02:53.907404+0000","last_unstale":"2024-09-18T08:06:33.187905+0000","last_undegraded":"2024-09-18T08:06:33.187905+0000","last_fullsized":"2024-09-18T08:06:33.187905+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_clean_scrub_stamp":"2024-09-18T08:02:52.890475+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:07:25.591758+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.6","version":"139'10732","reported_seq":10897,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-09-18T08:06:33.472376+0000","last_change":"2024-09-18T08:05:13.657923+0000","last_active":"2024-09-18T08:06:33.472376+0000","last_peered":"2024-09-18T08:06:33.472376+0000","last_clean":"2024-09-18T08:06:33.472376+0000","last_became_active":"2024-09-18T08:03:57.466391+0000","last_became_peered":"2024-09-18T08:03:57.466391+0000","last_unstale":"2024-09-18T08:06:33.472376+0000","last_undegraded":"2024-09-18T08:06:33.472376+0000","last_fullsized":"2024-09-18T08:06:33.472376+0000","mapping_epoch":83,"log_start":"107'5327","ondisk_log_start":"107'5327","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:03:56.449957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:03:56.449957+0000","last_clean_scrub_stamp":"2024-09-18T08:03:56.449957+0000","objects_scrubbed":3899,"log_size":5405,"log_dups_size":0,"ondisk_log_size":5405,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:07:05.494747+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":264896512,"num_objects":4042,"num_object_clones":0,"num_object_copies":12126,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4042,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10732,"num_write_kb":472768,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.5","version":"138'19","reported_seq":211,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-09-18T08:06:27.973518+0000","last_change":"2024-09-18T08:06:24.191102+0000","last_active":"2024-09-18T08:06:27.973518+0000","last_peered":"2024-09-18T08:06:27.973518+0000","last_clean":"2024-09-18T08:06:27.973518+0000","last_became_active":"2024-09-18T08:06:24.146117+0000","last_became_peered":"2024-09-18T08:06:24.146117+0000","last_unstale":"2024-09-18T08:06:27.973518+0000","last_undegraded":"2024-09-18T08:06:27.973518+0000","last_fullsized":"2024-09-18T08:06:27.973518+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"130'18","last_scrub_stamp":"2024-09-18T08:06:15.455503+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_clean_scrub_stamp":"2024-09-18T08:06:15.455503+0000","objects_scrubbed":8,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:07:41.325892+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":24,"num_read_kb":14,"num_write":13,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":308,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,7],"acting":[4,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":189,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-09-18T08:06:33.180271+0000","last_change":"2024-09-18T08:06:24.134043+0000","last_active":"2024-09-18T08:06:33.180271+0000","last_peered":"2024-09-18T08:06:33.180271+0000","last_clean":"2024-09-18T08:06:33.180271+0000","last_became_active":"2024-09-18T08:06:24.133425+0000","last_became_peered":"2024-09-18T08:06:24.133425+0000","last_unstale":"2024-09-18T08:06:33.180271+0000","last_undegraded":"2024-09-18T08:06:33.180271+0000","last_fullsized":"2024-09-18T08:06:33.180271+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:05:38.080993+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:46.860092+0000","last_clean_scrub_stamp":"2024-09-18T08:05:38.080993+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:07:00.431736+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"74'79","reported_seq":246,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-09-18T08:06:25.120267+0000","last_change":"2024-09-18T08:06:24.167946+0000","last_active":"2024-09-18T08:06:25.120267+0000","last_peered":"2024-09-18T08:06:25.120267+0000","last_clean":"2024-09-18T08:06:25.120267+0000","last_became_active":"2024-09-18T08:06:24.146679+0000","last_became_peered":"2024-09-18T08:06:24.146679+0000","last_unstale":"2024-09-18T08:06:25.120267+0000","last_undegraded":"2024-09-18T08:06:25.120267+0000","last_fullsized":"2024-09-18T08:06:25.120267+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'79","last_scrub_stamp":"2024-09-18T08:04:33.791841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:00:11.893742+0000","last_clean_scrub_stamp":"2024-09-18T08:04:33.791841+0000","objects_scrubbed":2,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":231,"num_write":113,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1771104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":173,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-09-18T08:06:25.120667+0000","last_change":"2024-09-18T08:05:42.929394+0000","last_active":"2024-09-18T08:06:25.120667+0000","last_peered":"2024-09-18T08:06:25.120667+0000","last_clean":"2024-09-18T08:06:25.120667+0000","last_became_active":"2024-09-18T08:02:51.905516+0000","last_became_peered":"2024-09-18T08:02:51.905516+0000","last_unstale":"2024-09-18T08:06:25.120667+0000","last_undegraded":"2024-09-18T08:06:25.120667+0000","last_fullsized":"2024-09-18T08:06:25.120667+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:05:42.929311+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:50.886830+0000","last_clean_scrub_stamp":"2024-09-18T08:05:42.929311+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:06:48.900458+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":162,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-09-18T08:06:25.162778+0000","last_change":"2024-09-18T08:04:11.930843+0000","last_active":"2024-09-18T08:06:25.162778+0000","last_peered":"2024-09-18T08:06:25.162778+0000","last_clean":"2024-09-18T08:06:25.162778+0000","last_became_active":"2024-09-18T08:02:53.908668+0000","last_became_peered":"2024-09-18T08:02:53.908668+0000","last_unstale":"2024-09-18T08:06:25.162778+0000","last_undegraded":"2024-09-18T08:06:25.162778+0000","last_fullsized":"2024-09-18T08:06:25.162778+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:04:11.930797+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_clean_scrub_stamp":"2024-09-18T08:04:11.930797+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:06:49.753536+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.d","version":"139'10727","reported_seq":10945,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-09-18T08:06:33.268713+0000","last_change":"2024-09-18T08:06:25.146940+0000","last_active":"2024-09-18T08:06:33.268713+0000","last_peered":"2024-09-18T08:06:33.268713+0000","last_clean":"2024-09-18T08:06:33.268713+0000","last_became_active":"2024-09-18T08:06:25.123627+0000","last_became_peered":"2024-09-18T08:06:25.123627+0000","last_unstale":"2024-09-18T08:06:33.268713+0000","last_undegraded":"2024-09-18T08:06:33.268713+0000","last_fullsized":"2024-09-18T08:06:33.268713+0000","mapping_epoch":137,"log_start":"109'6037","ondisk_log_start":"109'6037","created":83,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"123'7982","last_scrub_stamp":"2024-09-18T08:05:44.590884+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:03:56.449957+0000","last_clean_scrub_stamp":"2024-09-18T08:05:44.590884+0000","objects_scrubbed":6877,"log_size":4690,"log_dups_size":0,"ondisk_log_size":4690,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:07:00.648945+0000","scrub_duration":2853,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":264896536,"num_objects":4043,"num_object_clones":0,"num_object_copies":12129,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4043,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10727,"num_write_kb":472577,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,0],"acting":[1,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.e","version":"69'11","reported_seq":226,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-09-18T08:06:25.117877+0000","last_change":"2024-09-18T08:06:24.173542+0000","last_active":"2024-09-18T08:06:25.117877+0000","last_peered":"2024-09-18T08:06:25.117877+0000","last_clean":"2024-09-18T08:06:25.117877+0000","last_became_active":"2024-09-18T08:06:24.143839+0000","last_became_peered":"2024-09-18T08:06:24.143839+0000","last_unstale":"2024-09-18T08:06:25.117877+0000","last_undegraded":"2024-09-18T08:06:25.117877+0000","last_fullsized":"2024-09-18T08:06:25.117877+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'11","last_scrub_stamp":"2024-09-18T08:06:14.472168+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_clean_scrub_stamp":"2024-09-18T08:06:14.472168+0000","objects_scrubbed":5,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:07:20.674211+0000","scrub_duration":23,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":165,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-09-18T08:06:25.117342+0000","last_change":"2024-09-18T08:02:47.880265+0000","last_active":"2024-09-18T08:06:25.117342+0000","last_peered":"2024-09-18T08:06:25.117342+0000","last_clean":"2024-09-18T08:06:25.117342+0000","last_became_active":"2024-09-18T08:02:47.879764+0000","last_became_peered":"2024-09-18T08:02:47.879764+0000","last_unstale":"2024-09-18T08:06:25.117342+0000","last_undegraded":"2024-09-18T08:06:25.117342+0000","last_fullsized":"2024-09-18T08:06:25.117342+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:02:46.860092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:46.860092+0000","last_clean_scrub_stamp":"2024-09-18T08:02:46.860092+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:06:53.348536+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":179,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-09-18T08:06:25.162641+0000","last_change":"2024-09-18T08:06:24.132512+0000","last_active":"2024-09-18T08:06:25.162641+0000","last_peered":"2024-09-18T08:06:25.162641+0000","last_clean":"2024-09-18T08:06:25.162641+0000","last_became_active":"2024-09-18T08:06:24.132140+0000","last_became_peered":"2024-09-18T08:06:24.132140+0000","last_unstale":"2024-09-18T08:06:25.162641+0000","last_undegraded":"2024-09-18T08:06:25.162641+0000","last_fullsized":"2024-09-18T08:06:25.162641+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:05:08.839644+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:50.886830+0000","last_clean_scrub_stamp":"2024-09-18T08:05:08.839644+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:06:34.763900+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":177,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-09-18T08:06:33.179476+0000","last_change":"2024-09-18T08:06:24.130609+0000","last_active":"2024-09-18T08:06:33.179476+0000","last_peered":"2024-09-18T08:06:33.179476+0000","last_clean":"2024-09-18T08:06:33.179476+0000","last_became_active":"2024-09-18T08:06:24.130266+0000","last_became_peered":"2024-09-18T08:06:24.130266+0000","last_unstale":"2024-09-18T08:06:33.179476+0000","last_undegraded":"2024-09-18T08:06:33.179476+0000","last_fullsized":"2024-09-18T08:06:33.179476+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:05:09.038115+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_clean_scrub_stamp":"2024-09-18T08:05:09.038115+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.e","version":"139'10781","reported_seq":11023,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-09-18T08:06:33.269462+0000","last_change":"2024-09-18T08:05:47.424879+0000","last_active":"2024-09-18T08:06:33.269462+0000","last_peered":"2024-09-18T08:06:33.269462+0000","last_clean":"2024-09-18T08:06:33.269462+0000","last_became_active":"2024-09-18T08:03:57.464311+0000","last_became_peered":"2024-09-18T08:03:57.464311+0000","last_unstale":"2024-09-18T08:06:33.269462+0000","last_undegraded":"2024-09-18T08:06:33.269462+0000","last_fullsized":"2024-09-18T08:06:33.269462+0000","mapping_epoch":83,"log_start":"104'4639","ondisk_log_start":"104'4639","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:03:56.449957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:03:56.449957+0000","last_clean_scrub_stamp":"2024-09-18T08:03:56.449957+0000","objects_scrubbed":5849,"log_size":6142,"log_dups_size":0,"ondisk_log_size":6142,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":261685248,"num_objects":3993,"num_object_clones":0,"num_object_copies":11979,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3993,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10781,"num_write_kb":472768,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,0],"acting":[2,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.d","version":"69'17","reported_seq":189,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-09-18T08:06:25.117860+0000","last_change":"2024-09-18T08:02:49.895971+0000","last_active":"2024-09-18T08:06:25.117860+0000","last_peered":"2024-09-18T08:06:25.117860+0000","last_clean":"2024-09-18T08:06:25.117860+0000","last_became_active":"2024-09-18T08:02:49.895578+0000","last_became_peered":"2024-09-18T08:02:49.895578+0000","last_unstale":"2024-09-18T08:06:25.117860+0000","last_undegraded":"2024-09-18T08:06:25.117860+0000","last_fullsized":"2024-09-18T08:06:25.117860+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_clean_scrub_stamp":"2024-09-18T08:02:48.870718+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:07:02.375027+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":29,"num_read_kb":19,"num_write":18,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,1],"acting":[4,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":107,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-09-18T08:06:25.117212+0000","last_change":"2024-09-18T08:06:24.144485+0000","last_active":"2024-09-18T08:06:25.117212+0000","last_peered":"2024-09-18T08:06:25.117212+0000","last_clean":"2024-09-18T08:06:25.117212+0000","last_became_active":"2024-09-18T08:06:24.143152+0000","last_became_peered":"2024-09-18T08:06:24.143152+0000","last_unstale":"2024-09-18T08:06:25.117212+0000","last_undegraded":"2024-09-18T08:06:25.117212+0000","last_fullsized":"2024-09-18T08:06:25.117212+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:05:12.622901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:46.860092+0000","last_clean_scrub_stamp":"2024-09-18T08:05:12.622901+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:06:30.932406+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":175,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-09-18T08:06:33.187999+0000","last_change":"2024-09-18T08:05:31.851989+0000","last_active":"2024-09-18T08:06:33.187999+0000","last_peered":"2024-09-18T08:06:33.187999+0000","last_clean":"2024-09-18T08:06:33.187999+0000","last_became_active":"2024-09-18T08:02:51.907196+0000","last_became_peered":"2024-09-18T08:02:51.907196+0000","last_unstale":"2024-09-18T08:06:33.187999+0000","last_undegraded":"2024-09-18T08:06:33.187999+0000","last_fullsized":"2024-09-18T08:06:33.187999+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:05:31.851797+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:50.886830+0000","last_clean_scrub_stamp":"2024-09-18T08:05:31.851797+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:06:59.697560+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":164,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-09-18T08:06:33.168575+0000","last_change":"2024-09-18T08:04:06.709703+0000","last_active":"2024-09-18T08:06:33.168575+0000","last_peered":"2024-09-18T08:06:33.168575+0000","last_clean":"2024-09-18T08:06:33.168575+0000","last_became_active":"2024-09-18T08:02:53.904144+0000","last_became_peered":"2024-09-18T08:02:53.904144+0000","last_unstale":"2024-09-18T08:06:33.168575+0000","last_undegraded":"2024-09-18T08:06:33.168575+0000","last_fullsized":"2024-09-18T08:06:33.168575+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:04:06.709631+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_clean_scrub_stamp":"2024-09-18T08:04:06.709631+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:07:24.993615+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.f","version":"139'10705","reported_seq":10889,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-09-18T08:06:33.476090+0000","last_change":"2024-09-18T08:05:13.677385+0000","last_active":"2024-09-18T08:06:33.476090+0000","last_peered":"2024-09-18T08:06:33.476090+0000","last_clean":"2024-09-18T08:06:33.476090+0000","last_became_active":"2024-09-18T08:03:57.467980+0000","last_became_peered":"2024-09-18T08:03:57.467980+0000","last_unstale":"2024-09-18T08:06:33.476090+0000","last_undegraded":"2024-09-18T08:06:33.476090+0000","last_fullsized":"2024-09-18T08:06:33.476090+0000","mapping_epoch":83,"log_start":"107'5363","ondisk_log_start":"107'5363","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:03:56.449957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:03:56.449957+0000","last_clean_scrub_stamp":"2024-09-18T08:03:56.449957+0000","objects_scrubbed":5349,"log_size":5342,"log_dups_size":0,"ondisk_log_size":5342,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:06:48.316140+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":259850240,"num_objects":3965,"num_object_clones":0,"num_object_copies":11895,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3965,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10705,"num_write_kb":469440,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.c","version":"69'10","reported_seq":222,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-09-18T08:06:25.117504+0000","last_change":"2024-09-18T08:06:24.180438+0000","last_active":"2024-09-18T08:06:25.117504+0000","last_peered":"2024-09-18T08:06:25.117504+0000","last_clean":"2024-09-18T08:06:25.117504+0000","last_became_active":"2024-09-18T08:06:24.145005+0000","last_became_peered":"2024-09-18T08:06:24.145005+0000","last_unstale":"2024-09-18T08:06:25.117504+0000","last_undegraded":"2024-09-18T08:06:25.117504+0000","last_fullsized":"2024-09-18T08:06:25.117504+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'10","last_scrub_stamp":"2024-09-18T08:05:41.474813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_clean_scrub_stamp":"2024-09-18T08:05:41.474813+0000","objects_scrubbed":5,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:07:01.198872+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":175,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-09-18T08:06:33.179907+0000","last_change":"2024-09-18T08:04:24.892153+0000","last_active":"2024-09-18T08:06:33.179907+0000","last_peered":"2024-09-18T08:06:33.179907+0000","last_clean":"2024-09-18T08:06:33.179907+0000","last_became_active":"2024-09-18T08:02:47.881512+0000","last_became_peered":"2024-09-18T08:02:47.881512+0000","last_unstale":"2024-09-18T08:06:33.179907+0000","last_undegraded":"2024-09-18T08:06:33.179907+0000","last_fullsized":"2024-09-18T08:06:33.179907+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:04:24.892079+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:46.860092+0000","last_clean_scrub_stamp":"2024-09-18T08:04:24.892079+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.d","version":"69'8","reported_seq":229,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-09-18T08:06:33.168426+0000","last_change":"2024-09-18T08:05:33.027988+0000","last_active":"2024-09-18T08:06:33.168426+0000","last_peered":"2024-09-18T08:06:33.168426+0000","last_clean":"2024-09-18T08:06:33.168426+0000","last_became_active":"2024-09-18T08:02:51.909011+0000","last_became_peered":"2024-09-18T08:02:51.909011+0000","last_unstale":"2024-09-18T08:06:33.168426+0000","last_undegraded":"2024-09-18T08:06:33.168426+0000","last_fullsized":"2024-09-18T08:06:33.168426+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'8","last_scrub_stamp":"2024-09-18T08:05:33.027783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:50.886830+0000","last_clean_scrub_stamp":"2024-09-18T08:05:33.027783+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:06:41.650338+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":162,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-09-18T08:06:25.117656+0000","last_change":"2024-09-18T08:04:32.377211+0000","last_active":"2024-09-18T08:06:25.117656+0000","last_peered":"2024-09-18T08:06:25.117656+0000","last_clean":"2024-09-18T08:06:25.117656+0000","last_became_active":"2024-09-18T08:02:53.901858+0000","last_became_peered":"2024-09-18T08:02:53.901858+0000","last_unstale":"2024-09-18T08:06:25.117656+0000","last_undegraded":"2024-09-18T08:06:25.117656+0000","last_fullsized":"2024-09-18T08:06:25.117656+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:04:32.377157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_clean_scrub_stamp":"2024-09-18T08:04:32.377157+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:07:07.380764+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.8","version":"139'10760","reported_seq":10884,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-09-18T08:06:33.279396+0000","last_change":"2024-09-18T08:03:57.465051+0000","last_active":"2024-09-18T08:06:33.279396+0000","last_peered":"2024-09-18T08:06:33.279396+0000","last_clean":"2024-09-18T08:06:33.279396+0000","last_became_active":"2024-09-18T08:03:57.464738+0000","last_became_peered":"2024-09-18T08:03:57.464738+0000","last_unstale":"2024-09-18T08:06:33.279396+0000","last_undegraded":"2024-09-18T08:06:33.279396+0000","last_fullsized":"2024-09-18T08:06:33.279396+0000","mapping_epoch":83,"log_start":"104'4627","ondisk_log_start":"104'4627","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:03:56.449957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:03:56.449957+0000","last_clean_scrub_stamp":"2024-09-18T08:03:56.449957+0000","objects_scrubbed":0,"log_size":6133,"log_dups_size":0,"ondisk_log_size":6133,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":268042240,"num_objects":4090,"num_object_clones":0,"num_object_copies":12270,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4090,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10760,"num_write_kb":475200,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,4],"acting":[2,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.b","version":"69'9","reported_seq":187,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-09-18T08:06:33.180885+0000","last_change":"2024-09-18T08:05:05.294031+0000","last_active":"2024-09-18T08:06:33.180885+0000","last_peered":"2024-09-18T08:06:33.180885+0000","last_clean":"2024-09-18T08:06:33.180885+0000","last_became_active":"2024-09-18T08:02:49.897998+0000","last_became_peered":"2024-09-18T08:02:49.897998+0000","last_unstale":"2024-09-18T08:06:33.180885+0000","last_undegraded":"2024-09-18T08:06:33.180885+0000","last_fullsized":"2024-09-18T08:06:33.180885+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'9","last_scrub_stamp":"2024-09-18T08:05:05.293936+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_clean_scrub_stamp":"2024-09-18T08:05:05.293936+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":187,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-09-18T08:06:25.162479+0000","last_change":"2024-09-18T08:06:24.135338+0000","last_active":"2024-09-18T08:06:25.162479+0000","last_peered":"2024-09-18T08:06:25.162479+0000","last_clean":"2024-09-18T08:06:25.162479+0000","last_became_active":"2024-09-18T08:06:24.135005+0000","last_became_peered":"2024-09-18T08:06:24.135005+0000","last_unstale":"2024-09-18T08:06:25.162479+0000","last_undegraded":"2024-09-18T08:06:25.162479+0000","last_fullsized":"2024-09-18T08:06:25.162479+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:04:59.080115+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:46.860092+0000","last_clean_scrub_stamp":"2024-09-18T08:04:59.080115+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:06:28.803281+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":175,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-09-18T08:06:33.175132+0000","last_change":"2024-09-18T08:05:43.068256+0000","last_active":"2024-09-18T08:06:33.175132+0000","last_peered":"2024-09-18T08:06:33.175132+0000","last_clean":"2024-09-18T08:06:33.175132+0000","last_became_active":"2024-09-18T08:02:51.901330+0000","last_became_peered":"2024-09-18T08:02:51.901330+0000","last_unstale":"2024-09-18T08:06:33.175132+0000","last_undegraded":"2024-09-18T08:06:33.175132+0000","last_fullsized":"2024-09-18T08:06:33.175132+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:05:43.067057+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:50.886830+0000","last_clean_scrub_stamp":"2024-09-18T08:05:43.067057+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:07:10.458536+0000","scrub_duration":40,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,3],"acting":[2,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":164,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-09-18T08:06:33.177491+0000","last_change":"2024-09-18T08:05:09.251072+0000","last_active":"2024-09-18T08:06:33.177491+0000","last_peered":"2024-09-18T08:06:33.177491+0000","last_clean":"2024-09-18T08:06:33.177491+0000","last_became_active":"2024-09-18T08:02:53.911068+0000","last_became_peered":"2024-09-18T08:02:53.911068+0000","last_unstale":"2024-09-18T08:06:33.177491+0000","last_undegraded":"2024-09-18T08:06:33.177491+0000","last_fullsized":"2024-09-18T08:06:33.177491+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:05:09.250843+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_clean_scrub_stamp":"2024-09-18T08:05:09.250843+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.9","version":"139'10735","reported_seq":10859,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-09-18T08:06:33.458584+0000","last_change":"2024-09-18T08:03:57.467481+0000","last_active":"2024-09-18T08:06:33.458584+0000","last_peered":"2024-09-18T08:06:33.458584+0000","last_clean":"2024-09-18T08:06:33.458584+0000","last_became_active":"2024-09-18T08:03:57.467065+0000","last_became_peered":"2024-09-18T08:03:57.467065+0000","last_unstale":"2024-09-18T08:06:33.458584+0000","last_undegraded":"2024-09-18T08:06:33.458584+0000","last_fullsized":"2024-09-18T08:06:33.458584+0000","mapping_epoch":83,"log_start":"107'5380","ondisk_log_start":"107'5380","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:03:56.449957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:03:56.449957+0000","last_clean_scrub_stamp":"2024-09-18T08:03:56.449957+0000","objects_scrubbed":0,"log_size":5355,"log_dups_size":0,"ondisk_log_size":5355,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":263127040,"num_objects":4015,"num_object_clones":0,"num_object_copies":12045,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4015,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10735,"num_write_kb":472000,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.a","version":"69'19","reported_seq":192,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-09-18T08:06:33.187884+0000","last_change":"2024-09-18T08:06:24.162464+0000","last_active":"2024-09-18T08:06:33.187884+0000","last_peered":"2024-09-18T08:06:33.187884+0000","last_clean":"2024-09-18T08:06:33.187884+0000","last_became_active":"2024-09-18T08:06:24.139909+0000","last_became_peered":"2024-09-18T08:06:24.139909+0000","last_unstale":"2024-09-18T08:06:33.187884+0000","last_undegraded":"2024-09-18T08:06:33.187884+0000","last_fullsized":"2024-09-18T08:06:33.187884+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'19","last_scrub_stamp":"2024-09-18T08:04:20.391859+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_clean_scrub_stamp":"2024-09-18T08:04:20.391859+0000","objects_scrubbed":0,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:07:32.660944+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":31,"num_read_kb":20,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,7],"acting":[1,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":187,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-09-18T08:06:25.121338+0000","last_change":"2024-09-18T08:06:24.148358+0000","last_active":"2024-09-18T08:06:25.121338+0000","last_peered":"2024-09-18T08:06:25.121338+0000","last_clean":"2024-09-18T08:06:25.121338+0000","last_became_active":"2024-09-18T08:06:24.146699+0000","last_became_peered":"2024-09-18T08:06:24.146699+0000","last_unstale":"2024-09-18T08:06:25.121338+0000","last_undegraded":"2024-09-18T08:06:25.121338+0000","last_fullsized":"2024-09-18T08:06:25.121338+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:04:15.812306+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:46.860092+0000","last_clean_scrub_stamp":"2024-09-18T08:04:15.812306+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":167,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-09-18T08:06:33.170158+0000","last_change":"2024-09-18T08:04:56.004261+0000","last_active":"2024-09-18T08:06:33.170158+0000","last_peered":"2024-09-18T08:06:33.170158+0000","last_clean":"2024-09-18T08:06:33.170158+0000","last_became_active":"2024-09-18T08:02:51.903439+0000","last_became_peered":"2024-09-18T08:02:51.903439+0000","last_unstale":"2024-09-18T08:06:33.170158+0000","last_undegraded":"2024-09-18T08:06:33.170158+0000","last_fullsized":"2024-09-18T08:06:33.170158+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:04:56.004150+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:50.886830+0000","last_clean_scrub_stamp":"2024-09-18T08:04:56.004150+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:06:59.006166+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":162,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-09-18T08:06:25.120158+0000","last_change":"2024-09-18T08:04:17.811662+0000","last_active":"2024-09-18T08:06:25.120158+0000","last_peered":"2024-09-18T08:06:25.120158+0000","last_clean":"2024-09-18T08:06:25.120158+0000","last_became_active":"2024-09-18T08:02:53.912762+0000","last_became_peered":"2024-09-18T08:02:53.912762+0000","last_unstale":"2024-09-18T08:06:25.120158+0000","last_undegraded":"2024-09-18T08:06:25.120158+0000","last_fullsized":"2024-09-18T08:06:25.120158+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:04:17.811597+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_clean_scrub_stamp":"2024-09-18T08:04:17.811597+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:06:52.934540+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.a","version":"139'10865","reported_seq":10989,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-09-18T08:06:33.421540+0000","last_change":"2024-09-18T08:03:57.466483+0000","last_active":"2024-09-18T08:06:33.421540+0000","last_peered":"2024-09-18T08:06:33.421540+0000","last_clean":"2024-09-18T08:06:33.421540+0000","last_became_active":"2024-09-18T08:03:57.466168+0000","last_became_peered":"2024-09-18T08:03:57.466168+0000","last_unstale":"2024-09-18T08:06:33.421540+0000","last_undegraded":"2024-09-18T08:06:33.421540+0000","last_fullsized":"2024-09-18T08:06:33.421540+0000","mapping_epoch":83,"log_start":"110'6206","ondisk_log_start":"110'6206","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:03:56.449957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:03:56.449957+0000","last_clean_scrub_stamp":"2024-09-18T08:03:56.449957+0000","objects_scrubbed":0,"log_size":4659,"log_dups_size":0,"ondisk_log_size":4659,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:07:32.247056+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":265617408,"num_objects":4053,"num_object_clones":0,"num_object_copies":12159,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4053,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10865,"num_write_kb":477376,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.9","version":"69'12","reported_seq":200,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-09-18T08:06:25.117111+0000","last_change":"2024-09-18T08:05:33.544197+0000","last_active":"2024-09-18T08:06:25.117111+0000","last_peered":"2024-09-18T08:06:25.117111+0000","last_clean":"2024-09-18T08:06:25.117111+0000","last_became_active":"2024-09-18T08:02:49.894762+0000","last_became_peered":"2024-09-18T08:02:49.894762+0000","last_unstale":"2024-09-18T08:06:25.117111+0000","last_undegraded":"2024-09-18T08:06:25.117111+0000","last_fullsized":"2024-09-18T08:06:25.117111+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'12","last_scrub_stamp":"2024-09-18T08:05:33.544135+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_clean_scrub_stamp":"2024-09-18T08:05:33.544135+0000","objects_scrubbed":5,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:06:52.810406+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":25,"num_read_kb":16,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":181,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-09-18T08:06:25.121465+0000","last_change":"2024-09-18T08:05:39.906123+0000","last_active":"2024-09-18T08:06:25.121465+0000","last_peered":"2024-09-18T08:06:25.121465+0000","last_clean":"2024-09-18T08:06:25.121465+0000","last_became_active":"2024-09-18T08:02:47.884052+0000","last_became_peered":"2024-09-18T08:02:47.884052+0000","last_unstale":"2024-09-18T08:06:25.121465+0000","last_undegraded":"2024-09-18T08:06:25.121465+0000","last_fullsized":"2024-09-18T08:06:25.121465+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:05:39.906062+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:46.860092+0000","last_clean_scrub_stamp":"2024-09-18T08:05:39.906062+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:07:03.409405+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":167,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-09-18T08:06:33.174218+0000","last_change":"2024-09-18T08:05:06.828638+0000","last_active":"2024-09-18T08:06:33.174218+0000","last_peered":"2024-09-18T08:06:33.174218+0000","last_clean":"2024-09-18T08:06:33.174218+0000","last_became_active":"2024-09-18T08:02:51.902163+0000","last_became_peered":"2024-09-18T08:02:51.902163+0000","last_unstale":"2024-09-18T08:06:33.174218+0000","last_undegraded":"2024-09-18T08:06:33.174218+0000","last_fullsized":"2024-09-18T08:06:33.174218+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:05:06.828408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:50.886830+0000","last_clean_scrub_stamp":"2024-09-18T08:05:06.828408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":164,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-09-18T08:06:33.179600+0000","last_change":"2024-09-18T08:05:44.074994+0000","last_active":"2024-09-18T08:06:33.179600+0000","last_peered":"2024-09-18T08:06:33.179600+0000","last_clean":"2024-09-18T08:06:33.179600+0000","last_became_active":"2024-09-18T08:02:53.907412+0000","last_became_peered":"2024-09-18T08:02:53.907412+0000","last_unstale":"2024-09-18T08:06:33.179600+0000","last_undegraded":"2024-09-18T08:06:33.179600+0000","last_fullsized":"2024-09-18T08:06:33.179600+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:05:44.074555+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_clean_scrub_stamp":"2024-09-18T08:05:44.074555+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:07:10.508813+0000","scrub_duration":23,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.b","version":"139'10676","reported_seq":12694,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-09-18T08:06:33.261315+0000","last_change":"2024-09-18T08:06:24.183329+0000","last_active":"2024-09-18T08:06:33.261315+0000","last_peered":"2024-09-18T08:06:33.261315+0000","last_clean":"2024-09-18T08:06:33.261315+0000","last_became_active":"2024-09-18T08:06:24.157344+0000","last_became_peered":"2024-09-18T08:06:24.157344+0000","last_unstale":"2024-09-18T08:06:33.261315+0000","last_undegraded":"2024-09-18T08:06:33.261315+0000","last_fullsized":"2024-09-18T08:06:33.261315+0000","mapping_epoch":136,"log_start":"109'5926","ondisk_log_start":"109'5926","created":83,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"118'6870","last_scrub_stamp":"2024-09-18T08:05:28.826628+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:03:56.449957+0000","last_clean_scrub_stamp":"2024-09-18T08:05:28.826628+0000","objects_scrubbed":6831,"log_size":4750,"log_dups_size":0,"ondisk_log_size":4750,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"queued for scrub","scrub_duration":1959,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":254803968,"num_objects":3888,"num_object_clones":0,"num_object_copies":11664,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3888,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10676,"num_write_kb":466048,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":902,"num_bytes_recovered":58982400,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6],"acting":[1,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.8","version":"69'15","reported_seq":240,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-09-18T08:06:25.161736+0000","last_change":"2024-09-18T08:06:24.162625+0000","last_active":"2024-09-18T08:06:25.161736+0000","last_peered":"2024-09-18T08:06:25.161736+0000","last_clean":"2024-09-18T08:06:25.161736+0000","last_became_active":"2024-09-18T08:06:24.138259+0000","last_became_peered":"2024-09-18T08:06:24.138259+0000","last_unstale":"2024-09-18T08:06:25.161736+0000","last_undegraded":"2024-09-18T08:06:25.161736+0000","last_fullsized":"2024-09-18T08:06:25.161736+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'15","last_scrub_stamp":"2024-09-18T08:04:26.859044+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_clean_scrub_stamp":"2024-09-18T08:04:26.859044+0000","objects_scrubbed":7,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.f","version":"62'2","reported_seq":193,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-09-18T08:06:25.121590+0000","last_change":"2024-09-18T08:05:11.906358+0000","last_active":"2024-09-18T08:06:25.121590+0000","last_peered":"2024-09-18T08:06:25.121590+0000","last_clean":"2024-09-18T08:06:25.121590+0000","last_became_active":"2024-09-18T08:02:47.881165+0000","last_became_peered":"2024-09-18T08:02:47.881165+0000","last_unstale":"2024-09-18T08:06:25.121590+0000","last_undegraded":"2024-09-18T08:06:25.121590+0000","last_fullsized":"2024-09-18T08:06:25.121590+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"62'2","last_scrub_stamp":"2024-09-18T08:05:11.906308+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:46.860092+0000","last_clean_scrub_stamp":"2024-09-18T08:05:11.906308+0000","objects_scrubbed":2,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":24,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":4,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.9","version":"138'10","reported_seq":253,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-09-18T08:06:27.973515+0000","last_change":"2024-09-18T08:06:24.176322+0000","last_active":"2024-09-18T08:06:27.973515+0000","last_peered":"2024-09-18T08:06:27.973515+0000","last_clean":"2024-09-18T08:06:27.973515+0000","last_became_active":"2024-09-18T08:06:24.170165+0000","last_became_peered":"2024-09-18T08:06:24.170165+0000","last_unstale":"2024-09-18T08:06:27.973515+0000","last_undegraded":"2024-09-18T08:06:27.973515+0000","last_fullsized":"2024-09-18T08:06:27.973515+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"130'9","last_scrub_stamp":"2024-09-18T08:06:14.178538+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:50.886830+0000","last_clean_scrub_stamp":"2024-09-18T08:06:14.178538+0000","objects_scrubbed":1,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:07:19.547201+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":175,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-09-18T08:06:25.161435+0000","last_change":"2024-09-18T08:06:24.135990+0000","last_active":"2024-09-18T08:06:25.161435+0000","last_peered":"2024-09-18T08:06:25.161435+0000","last_clean":"2024-09-18T08:06:25.161435+0000","last_became_active":"2024-09-18T08:06:24.133119+0000","last_became_peered":"2024-09-18T08:06:24.133119+0000","last_unstale":"2024-09-18T08:06:25.161435+0000","last_undegraded":"2024-09-18T08:06:25.161435+0000","last_fullsized":"2024-09-18T08:06:25.161435+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:05:07.879501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_clean_scrub_stamp":"2024-09-18T08:05:07.879501+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:06:29.784165+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":101,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-09-18T08:06:33.180948+0000","last_change":"2024-09-18T08:06:24.133955+0000","last_active":"2024-09-18T08:06:33.180948+0000","last_peered":"2024-09-18T08:06:33.180948+0000","last_clean":"2024-09-18T08:06:33.180948+0000","last_became_active":"2024-09-18T08:06:24.133644+0000","last_became_peered":"2024-09-18T08:06:24.133644+0000","last_unstale":"2024-09-18T08:06:33.180948+0000","last_undegraded":"2024-09-18T08:06:33.180948+0000","last_fullsized":"2024-09-18T08:06:33.180948+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:04:16.378205+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:46.860092+0000","last_clean_scrub_stamp":"2024-09-18T08:04:16.378205+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.17","version":"69'6","reported_seq":188,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-09-18T08:06:33.179928+0000","last_change":"2024-09-18T08:05:26.228344+0000","last_active":"2024-09-18T08:06:33.179928+0000","last_peered":"2024-09-18T08:06:33.179928+0000","last_clean":"2024-09-18T08:06:33.179928+0000","last_became_active":"2024-09-18T08:02:49.895067+0000","last_became_peered":"2024-09-18T08:02:49.895067+0000","last_unstale":"2024-09-18T08:06:33.179928+0000","last_undegraded":"2024-09-18T08:06:33.179928+0000","last_fullsized":"2024-09-18T08:06:33.179928+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'6","last_scrub_stamp":"2024-09-18T08:05:26.227814+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_clean_scrub_stamp":"2024-09-18T08:05:26.227814+0000","objects_scrubbed":3,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:06:36.064005+0000","scrub_duration":18,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":9,"num_read_kb":6,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":173,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-09-18T08:06:25.162546+0000","last_change":"2024-09-18T08:05:45.812436+0000","last_active":"2024-09-18T08:06:25.162546+0000","last_peered":"2024-09-18T08:06:25.162546+0000","last_clean":"2024-09-18T08:06:25.162546+0000","last_became_active":"2024-09-18T08:02:51.912557+0000","last_became_peered":"2024-09-18T08:02:51.912557+0000","last_unstale":"2024-09-18T08:06:25.162546+0000","last_undegraded":"2024-09-18T08:06:25.162546+0000","last_fullsized":"2024-09-18T08:06:25.162546+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:05:45.812389+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:50.886830+0000","last_clean_scrub_stamp":"2024-09-18T08:05:45.812389+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:07:01.523740+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,1],"acting":[5,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":176,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-09-18T08:06:25.120395+0000","last_change":"2024-09-18T08:06:24.147538+0000","last_active":"2024-09-18T08:06:25.120395+0000","last_peered":"2024-09-18T08:06:25.120395+0000","last_clean":"2024-09-18T08:06:25.120395+0000","last_became_active":"2024-09-18T08:06:24.145909+0000","last_became_peered":"2024-09-18T08:06:24.145909+0000","last_unstale":"2024-09-18T08:06:25.120395+0000","last_undegraded":"2024-09-18T08:06:25.120395+0000","last_fullsized":"2024-09-18T08:06:25.120395+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:05:00.883555+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_clean_scrub_stamp":"2024-09-18T08:05:00.883555+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"69'9","reported_seq":187,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-09-18T08:06:33.191483+0000","last_change":"2024-09-18T08:05:00.265939+0000","last_active":"2024-09-18T08:06:33.191483+0000","last_peered":"2024-09-18T08:06:33.191483+0000","last_clean":"2024-09-18T08:06:33.191483+0000","last_became_active":"2024-09-18T08:02:49.891567+0000","last_became_peered":"2024-09-18T08:02:49.891567+0000","last_unstale":"2024-09-18T08:06:33.191483+0000","last_undegraded":"2024-09-18T08:06:33.191483+0000","last_fullsized":"2024-09-18T08:06:33.191483+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'9","last_scrub_stamp":"2024-09-18T08:05:00.265738+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_clean_scrub_stamp":"2024-09-18T08:05:00.265738+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":17,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":195,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-09-18T08:06:25.120183+0000","last_change":"2024-09-18T08:06:24.170084+0000","last_active":"2024-09-18T08:06:25.120183+0000","last_peered":"2024-09-18T08:06:25.120183+0000","last_clean":"2024-09-18T08:06:25.120183+0000","last_became_active":"2024-09-18T08:06:24.169777+0000","last_became_peered":"2024-09-18T08:06:24.169777+0000","last_unstale":"2024-09-18T08:06:25.120183+0000","last_undegraded":"2024-09-18T08:06:25.120183+0000","last_fullsized":"2024-09-18T08:06:25.120183+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:06:13.214705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:46.860092+0000","last_clean_scrub_stamp":"2024-09-18T08:06:13.214705+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:07:26.133630+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":167,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-09-18T08:06:33.179772+0000","last_change":"2024-09-18T08:05:42.061649+0000","last_active":"2024-09-18T08:06:33.179772+0000","last_peered":"2024-09-18T08:06:33.179772+0000","last_clean":"2024-09-18T08:06:33.179772+0000","last_became_active":"2024-09-18T08:02:51.907361+0000","last_became_peered":"2024-09-18T08:02:51.907361+0000","last_unstale":"2024-09-18T08:06:33.179772+0000","last_undegraded":"2024-09-18T08:06:33.179772+0000","last_fullsized":"2024-09-18T08:06:33.179772+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:05:42.061452+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:50.886830+0000","last_clean_scrub_stamp":"2024-09-18T08:05:42.061452+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:07:05.139088+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":164,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-09-18T08:06:33.168427+0000","last_change":"2024-09-18T08:05:07.780899+0000","last_active":"2024-09-18T08:06:33.168427+0000","last_peered":"2024-09-18T08:06:33.168427+0000","last_clean":"2024-09-18T08:06:33.168427+0000","last_became_active":"2024-09-18T08:02:53.908917+0000","last_became_peered":"2024-09-18T08:02:53.908917+0000","last_unstale":"2024-09-18T08:06:33.168427+0000","last_undegraded":"2024-09-18T08:06:33.168427+0000","last_fullsized":"2024-09-18T08:06:33.168427+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:05:07.780816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_clean_scrub_stamp":"2024-09-18T08:05:07.780816+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.15","version":"69'9","reported_seq":193,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-09-18T08:06:25.162166+0000","last_change":"2024-09-18T08:05:36.896299+0000","last_active":"2024-09-18T08:06:25.162166+0000","last_peered":"2024-09-18T08:06:25.162166+0000","last_clean":"2024-09-18T08:06:25.162166+0000","last_became_active":"2024-09-18T08:02:49.890497+0000","last_became_peered":"2024-09-18T08:02:49.890497+0000","last_unstale":"2024-09-18T08:06:25.162166+0000","last_undegraded":"2024-09-18T08:06:25.162166+0000","last_fullsized":"2024-09-18T08:06:25.162166+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'9","last_scrub_stamp":"2024-09-18T08:05:36.896238+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_clean_scrub_stamp":"2024-09-18T08:05:36.896238+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:06:52.292485+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":183,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-09-18T08:06:33.177817+0000","last_change":"2024-09-18T08:06:12.437630+0000","last_active":"2024-09-18T08:06:33.177817+0000","last_peered":"2024-09-18T08:06:33.177817+0000","last_clean":"2024-09-18T08:06:33.177817+0000","last_became_active":"2024-09-18T08:02:47.875983+0000","last_became_peered":"2024-09-18T08:02:47.875983+0000","last_unstale":"2024-09-18T08:06:33.177817+0000","last_undegraded":"2024-09-18T08:06:33.177817+0000","last_fullsized":"2024-09-18T08:06:33.177817+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:06:12.437584+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:46.860092+0000","last_clean_scrub_stamp":"2024-09-18T08:06:12.437584+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:07:30.735772+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.14","version":"69'8","reported_seq":221,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-09-18T08:06:33.178952+0000","last_change":"2024-09-18T08:05:01.083803+0000","last_active":"2024-09-18T08:06:33.178952+0000","last_peered":"2024-09-18T08:06:33.178952+0000","last_clean":"2024-09-18T08:06:33.178952+0000","last_became_active":"2024-09-18T08:02:51.909513+0000","last_became_peered":"2024-09-18T08:02:51.909513+0000","last_unstale":"2024-09-18T08:06:33.178952+0000","last_undegraded":"2024-09-18T08:06:33.178952+0000","last_fullsized":"2024-09-18T08:06:33.178952+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'8","last_scrub_stamp":"2024-09-18T08:05:01.082570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:50.886830+0000","last_clean_scrub_stamp":"2024-09-18T08:05:01.082570+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":17,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":162,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-09-18T08:06:25.117931+0000","last_change":"2024-09-18T08:05:09.654918+0000","last_active":"2024-09-18T08:06:25.117931+0000","last_peered":"2024-09-18T08:06:25.117931+0000","last_clean":"2024-09-18T08:06:25.117931+0000","last_became_active":"2024-09-18T08:02:53.907428+0000","last_became_peered":"2024-09-18T08:02:53.907428+0000","last_unstale":"2024-09-18T08:06:25.117931+0000","last_undegraded":"2024-09-18T08:06:25.117931+0000","last_fullsized":"2024-09-18T08:06:25.117931+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:05:09.654832+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_clean_scrub_stamp":"2024-09-18T08:05:09.654832+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.14","version":"69'10","reported_seq":205,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-09-18T08:06:33.179728+0000","last_change":"2024-09-18T08:04:18.172364+0000","last_active":"2024-09-18T08:06:33.179728+0000","last_peered":"2024-09-18T08:06:33.179728+0000","last_clean":"2024-09-18T08:06:33.179728+0000","last_became_active":"2024-09-18T08:04:18.150968+0000","last_became_peered":"2024-09-18T08:04:18.150968+0000","last_unstale":"2024-09-18T08:06:33.179728+0000","last_undegraded":"2024-09-18T08:06:33.179728+0000","last_fullsized":"2024-09-18T08:06:33.179728+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'10","last_scrub_stamp":"2024-09-18T08:04:16.940778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_clean_scrub_stamp":"2024-09-18T08:04:16.940778+0000","objects_scrubbed":5,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:06:54.101039+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,7],"acting":[3,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":173,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-09-18T08:06:25.120730+0000","last_change":"2024-09-18T08:05:08.855132+0000","last_active":"2024-09-18T08:06:25.120730+0000","last_peered":"2024-09-18T08:06:25.120730+0000","last_clean":"2024-09-18T08:06:25.120730+0000","last_became_active":"2024-09-18T08:02:47.882065+0000","last_became_peered":"2024-09-18T08:02:47.882065+0000","last_unstale":"2024-09-18T08:06:25.120730+0000","last_undegraded":"2024-09-18T08:06:25.120730+0000","last_fullsized":"2024-09-18T08:06:25.120730+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:05:08.855064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:46.860092+0000","last_clean_scrub_stamp":"2024-09-18T08:05:08.855064+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:06:28.592575+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.15","version":"69'8","reported_seq":227,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-09-18T08:06:27.973525+0000","last_change":"2024-09-18T08:05:27.998896+0000","last_active":"2024-09-18T08:06:27.973525+0000","last_peered":"2024-09-18T08:06:27.973525+0000","last_clean":"2024-09-18T08:06:27.973525+0000","last_became_active":"2024-09-18T08:02:51.914135+0000","last_became_peered":"2024-09-18T08:02:51.914135+0000","last_unstale":"2024-09-18T08:06:27.973525+0000","last_undegraded":"2024-09-18T08:06:27.973525+0000","last_fullsized":"2024-09-18T08:06:27.973525+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'8","last_scrub_stamp":"2024-09-18T08:05:27.998833+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:50.886830+0000","last_clean_scrub_stamp":"2024-09-18T08:05:27.998833+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:06:41.603719+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":164,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-09-18T08:06:33.177730+0000","last_change":"2024-09-18T08:05:03.302755+0000","last_active":"2024-09-18T08:06:33.177730+0000","last_peered":"2024-09-18T08:06:33.177730+0000","last_clean":"2024-09-18T08:06:33.177730+0000","last_became_active":"2024-09-18T08:02:53.912112+0000","last_became_peered":"2024-09-18T08:02:53.912112+0000","last_unstale":"2024-09-18T08:06:33.177730+0000","last_undegraded":"2024-09-18T08:06:33.177730+0000","last_fullsized":"2024-09-18T08:06:33.177730+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:05:03.302444+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_clean_scrub_stamp":"2024-09-18T08:05:03.302444+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":16,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.13","version":"69'11","reported_seq":226,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-09-18T08:06:25.117237+0000","last_change":"2024-09-18T08:06:24.189911+0000","last_active":"2024-09-18T08:06:25.117237+0000","last_peered":"2024-09-18T08:06:25.117237+0000","last_clean":"2024-09-18T08:06:25.117237+0000","last_became_active":"2024-09-18T08:06:24.150800+0000","last_became_peered":"2024-09-18T08:06:24.150800+0000","last_unstale":"2024-09-18T08:06:25.117237+0000","last_undegraded":"2024-09-18T08:06:25.117237+0000","last_fullsized":"2024-09-18T08:06:25.117237+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'11","last_scrub_stamp":"2024-09-18T08:04:34.451379+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_clean_scrub_stamp":"2024-09-18T08:04:34.451379+0000","objects_scrubbed":5,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":187,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-09-18T08:06:25.117737+0000","last_change":"2024-09-18T08:06:24.145787+0000","last_active":"2024-09-18T08:06:25.117737+0000","last_peered":"2024-09-18T08:06:25.117737+0000","last_clean":"2024-09-18T08:06:25.117737+0000","last_became_active":"2024-09-18T08:06:24.145458+0000","last_became_peered":"2024-09-18T08:06:24.145458+0000","last_unstale":"2024-09-18T08:06:25.117737+0000","last_undegraded":"2024-09-18T08:06:25.117737+0000","last_fullsized":"2024-09-18T08:06:25.117737+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:05:00.588728+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:46.860092+0000","last_clean_scrub_stamp":"2024-09-18T08:05:00.588728+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:06:26.107020+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":175,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-09-18T08:06:33.187775+0000","last_change":"2024-09-18T08:05:29.896334+0000","last_active":"2024-09-18T08:06:33.187775+0000","last_peered":"2024-09-18T08:06:33.187775+0000","last_clean":"2024-09-18T08:06:33.187775+0000","last_became_active":"2024-09-18T08:02:51.910752+0000","last_became_peered":"2024-09-18T08:02:51.910752+0000","last_unstale":"2024-09-18T08:06:33.187775+0000","last_undegraded":"2024-09-18T08:06:33.187775+0000","last_fullsized":"2024-09-18T08:06:33.187775+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:05:29.888927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:50.886830+0000","last_clean_scrub_stamp":"2024-09-18T08:05:29.888927+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:06:47.803144+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":172,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-09-18T08:06:33.179654+0000","last_change":"2024-09-18T08:06:11.249403+0000","last_active":"2024-09-18T08:06:33.179654+0000","last_peered":"2024-09-18T08:06:33.179654+0000","last_clean":"2024-09-18T08:06:33.179654+0000","last_became_active":"2024-09-18T08:02:53.911429+0000","last_became_peered":"2024-09-18T08:02:53.911429+0000","last_unstale":"2024-09-18T08:06:33.179654+0000","last_undegraded":"2024-09-18T08:06:33.179654+0000","last_fullsized":"2024-09-18T08:06:33.179654+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:06:11.249317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_clean_scrub_stamp":"2024-09-18T08:06:11.249317+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:07:33.181678+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.12","version":"69'9","reported_seq":213,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-09-18T08:06:33.188138+0000","last_change":"2024-09-18T08:06:24.152803+0000","last_active":"2024-09-18T08:06:33.188138+0000","last_peered":"2024-09-18T08:06:33.188138+0000","last_clean":"2024-09-18T08:06:33.188138+0000","last_became_active":"2024-09-18T08:06:24.132270+0000","last_became_peered":"2024-09-18T08:06:24.132270+0000","last_unstale":"2024-09-18T08:06:33.188138+0000","last_undegraded":"2024-09-18T08:06:33.188138+0000","last_fullsized":"2024-09-18T08:06:33.188138+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_clean_scrub_stamp":"2024-09-18T08:02:48.870718+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:07:28.580662+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":181,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-09-18T08:06:25.121260+0000","last_change":"2024-09-18T08:05:30.820244+0000","last_active":"2024-09-18T08:06:25.121260+0000","last_peered":"2024-09-18T08:06:25.121260+0000","last_clean":"2024-09-18T08:06:25.121260+0000","last_became_active":"2024-09-18T08:02:47.882096+0000","last_became_peered":"2024-09-18T08:02:47.882096+0000","last_unstale":"2024-09-18T08:06:25.121260+0000","last_undegraded":"2024-09-18T08:06:25.121260+0000","last_fullsized":"2024-09-18T08:06:25.121260+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:05:30.820200+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:46.860092+0000","last_clean_scrub_stamp":"2024-09-18T08:05:30.820200+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:06:54.287921+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":167,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-09-18T08:06:33.179644+0000","last_change":"2024-09-18T08:05:43.165522+0000","last_active":"2024-09-18T08:06:33.179644+0000","last_peered":"2024-09-18T08:06:33.179644+0000","last_clean":"2024-09-18T08:06:33.179644+0000","last_became_active":"2024-09-18T08:02:51.906277+0000","last_became_peered":"2024-09-18T08:02:51.906277+0000","last_unstale":"2024-09-18T08:06:33.179644+0000","last_undegraded":"2024-09-18T08:06:33.179644+0000","last_fullsized":"2024-09-18T08:06:33.179644+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:05:43.165258+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:50.886830+0000","last_clean_scrub_stamp":"2024-09-18T08:05:43.165258+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:07:08.715609+0000","scrub_duration":89,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.10","version":"69'1","reported_seq":160,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-09-18T08:06:33.177215+0000","last_change":"2024-09-18T08:05:51.429631+0000","last_active":"2024-09-18T08:06:33.177215+0000","last_peered":"2024-09-18T08:06:33.177215+0000","last_clean":"2024-09-18T08:06:33.177215+0000","last_became_active":"2024-09-18T08:02:53.912699+0000","last_became_peered":"2024-09-18T08:02:53.912699+0000","last_unstale":"2024-09-18T08:06:33.177215+0000","last_undegraded":"2024-09-18T08:06:33.177215+0000","last_fullsized":"2024-09-18T08:06:33.177215+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_clean_scrub_stamp":"2024-09-18T08:02:52.890475+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:07:25.467898+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.11","version":"69'11","reported_seq":236,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-09-18T08:06:33.179840+0000","last_change":"2024-09-18T08:06:24.154365+0000","last_active":"2024-09-18T08:06:33.179840+0000","last_peered":"2024-09-18T08:06:33.179840+0000","last_clean":"2024-09-18T08:06:33.179840+0000","last_became_active":"2024-09-18T08:06:24.132416+0000","last_became_peered":"2024-09-18T08:06:24.132416+0000","last_unstale":"2024-09-18T08:06:33.179840+0000","last_undegraded":"2024-09-18T08:06:33.179840+0000","last_fullsized":"2024-09-18T08:06:33.179840+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'11","last_scrub_stamp":"2024-09-18T08:05:13.037615+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_clean_scrub_stamp":"2024-09-18T08:05:13.037615+0000","objects_scrubbed":5,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":181,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-09-18T08:06:25.162634+0000","last_change":"2024-09-18T08:06:18.168254+0000","last_active":"2024-09-18T08:06:25.162634+0000","last_peered":"2024-09-18T08:06:25.162634+0000","last_clean":"2024-09-18T08:06:25.162634+0000","last_became_active":"2024-09-18T08:02:47.878886+0000","last_became_peered":"2024-09-18T08:02:47.878886+0000","last_unstale":"2024-09-18T08:06:25.162634+0000","last_undegraded":"2024-09-18T08:06:25.162634+0000","last_fullsized":"2024-09-18T08:06:25.162634+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:06:18.168206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:46.860092+0000","last_clean_scrub_stamp":"2024-09-18T08:06:18.168206+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:07:28.641116+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":187,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-09-18T08:06:25.119672+0000","last_change":"2024-09-18T08:06:24.148922+0000","last_active":"2024-09-18T08:06:25.119672+0000","last_peered":"2024-09-18T08:06:25.119672+0000","last_clean":"2024-09-18T08:06:25.119672+0000","last_became_active":"2024-09-18T08:06:24.148302+0000","last_became_peered":"2024-09-18T08:06:24.148302+0000","last_unstale":"2024-09-18T08:06:25.119672+0000","last_undegraded":"2024-09-18T08:06:25.119672+0000","last_fullsized":"2024-09-18T08:06:25.119672+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:06:15.177302+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:50.886830+0000","last_clean_scrub_stamp":"2024-09-18T08:06:15.177302+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:07:19.110873+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":177,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-09-18T08:06:33.179800+0000","last_change":"2024-09-18T08:06:24.128970+0000","last_active":"2024-09-18T08:06:33.179800+0000","last_peered":"2024-09-18T08:06:33.179800+0000","last_clean":"2024-09-18T08:06:33.179800+0000","last_became_active":"2024-09-18T08:06:24.128582+0000","last_became_peered":"2024-09-18T08:06:24.128582+0000","last_unstale":"2024-09-18T08:06:33.179800+0000","last_undegraded":"2024-09-18T08:06:33.179800+0000","last_fullsized":"2024-09-18T08:06:33.179800+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:06:13.241940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_clean_scrub_stamp":"2024-09-18T08:06:13.241940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:07:27.180968+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.10","version":"69'4","reported_seq":203,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-09-18T08:06:33.180515+0000","last_change":"2024-09-18T08:06:24.151823+0000","last_active":"2024-09-18T08:06:33.180515+0000","last_peered":"2024-09-18T08:06:33.180515+0000","last_clean":"2024-09-18T08:06:33.180515+0000","last_became_active":"2024-09-18T08:06:24.131309+0000","last_became_peered":"2024-09-18T08:06:24.131309+0000","last_unstale":"2024-09-18T08:06:33.180515+0000","last_undegraded":"2024-09-18T08:06:33.180515+0000","last_fullsized":"2024-09-18T08:06:33.180515+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'4","last_scrub_stamp":"2024-09-18T08:04:12.999978+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_clean_scrub_stamp":"2024-09-18T08:04:12.999978+0000","objects_scrubbed":2,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":6,"num_read_kb":4,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":183,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-09-18T08:06:33.180917+0000","last_change":"2024-09-18T08:06:10.457663+0000","last_active":"2024-09-18T08:06:33.180917+0000","last_peered":"2024-09-18T08:06:33.180917+0000","last_clean":"2024-09-18T08:06:33.180917+0000","last_became_active":"2024-09-18T08:02:47.877616+0000","last_became_peered":"2024-09-18T08:02:47.877616+0000","last_unstale":"2024-09-18T08:06:33.180917+0000","last_undegraded":"2024-09-18T08:06:33.180917+0000","last_fullsized":"2024-09-18T08:06:33.180917+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:06:10.457502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:46.860092+0000","last_clean_scrub_stamp":"2024-09-18T08:06:10.457502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:07:37.803172+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":107,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-09-18T08:06:25.116812+0000","last_change":"2024-09-18T08:06:24.168175+0000","last_active":"2024-09-18T08:06:25.116812+0000","last_peered":"2024-09-18T08:06:25.116812+0000","last_clean":"2024-09-18T08:06:25.116812+0000","last_became_active":"2024-09-18T08:06:24.166698+0000","last_became_peered":"2024-09-18T08:06:24.166698+0000","last_unstale":"2024-09-18T08:06:25.116812+0000","last_undegraded":"2024-09-18T08:06:25.116812+0000","last_fullsized":"2024-09-18T08:06:25.116812+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:05:11.591275+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:50.886830+0000","last_clean_scrub_stamp":"2024-09-18T08:05:11.591275+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:06:39.757348+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,7],"acting":[4,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":170,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-09-18T08:06:25.120076+0000","last_change":"2024-09-18T08:05:32.798628+0000","last_active":"2024-09-18T08:06:25.120076+0000","last_peered":"2024-09-18T08:06:25.120076+0000","last_clean":"2024-09-18T08:06:25.120076+0000","last_became_active":"2024-09-18T08:02:53.909146+0000","last_became_peered":"2024-09-18T08:02:53.909146+0000","last_unstale":"2024-09-18T08:06:25.120076+0000","last_undegraded":"2024-09-18T08:06:25.120076+0000","last_fullsized":"2024-09-18T08:06:25.120076+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:05:32.798564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_clean_scrub_stamp":"2024-09-18T08:05:32.798564+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:06:38.927572+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,4],"acting":[7,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":156,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-09-18T08:06:33.188172+0000","last_change":"2024-09-18T08:02:53.907055+0000","last_active":"2024-09-18T08:06:33.188172+0000","last_peered":"2024-09-18T08:06:33.188172+0000","last_clean":"2024-09-18T08:06:33.188172+0000","last_became_active":"2024-09-18T08:02:53.906726+0000","last_became_peered":"2024-09-18T08:02:53.906726+0000","last_unstale":"2024-09-18T08:06:33.188172+0000","last_undegraded":"2024-09-18T08:06:33.188172+0000","last_fullsized":"2024-09-18T08:06:33.188172+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_clean_scrub_stamp":"2024-09-18T08:02:52.890475+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:07:27.599829+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":175,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-09-18T08:06:33.180659+0000","last_change":"2024-09-18T08:05:02.063591+0000","last_active":"2024-09-18T08:06:33.180659+0000","last_peered":"2024-09-18T08:06:33.180659+0000","last_clean":"2024-09-18T08:06:33.180659+0000","last_became_active":"2024-09-18T08:02:47.878922+0000","last_became_peered":"2024-09-18T08:02:47.878922+0000","last_unstale":"2024-09-18T08:06:33.180659+0000","last_undegraded":"2024-09-18T08:06:33.180659+0000","last_fullsized":"2024-09-18T08:06:33.180659+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:05:02.063478+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:46.860092+0000","last_clean_scrub_stamp":"2024-09-18T08:05:02.063478+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"69'11","reported_seq":170,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-09-18T08:06:25.162847+0000","last_change":"2024-09-18T08:06:24.164086+0000","last_active":"2024-09-18T08:06:25.162847+0000","last_peered":"2024-09-18T08:06:25.162847+0000","last_clean":"2024-09-18T08:06:25.162847+0000","last_became_active":"2024-09-18T08:06:24.135232+0000","last_became_peered":"2024-09-18T08:06:24.135232+0000","last_unstale":"2024-09-18T08:06:25.162847+0000","last_undegraded":"2024-09-18T08:06:25.162847+0000","last_fullsized":"2024-09-18T08:06:25.162847+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'11","last_scrub_stamp":"2024-09-18T08:06:12.967462+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_clean_scrub_stamp":"2024-09-18T08:06:12.967462+0000","objects_scrubbed":5,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:07:25.928591+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":19,"num_read_kb":12,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,1],"acting":[5,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":167,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-09-18T08:06:33.191229+0000","last_change":"2024-09-18T08:04:26.117117+0000","last_active":"2024-09-18T08:06:33.191229+0000","last_peered":"2024-09-18T08:06:33.191229+0000","last_clean":"2024-09-18T08:06:33.191229+0000","last_became_active":"2024-09-18T08:02:51.900300+0000","last_became_peered":"2024-09-18T08:02:51.900300+0000","last_unstale":"2024-09-18T08:06:33.191229+0000","last_undegraded":"2024-09-18T08:06:33.191229+0000","last_fullsized":"2024-09-18T08:06:33.191229+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:04:26.117051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:50.886830+0000","last_clean_scrub_stamp":"2024-09-18T08:04:26.117051+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":7,"num_pg":16,"stat_sum":{"num_bytes":4222222360,"num_objects":64427,"num_object_clones":0,"num_object_copies":193281,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":64427,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":172151,"num_write_kb":7570433,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2629,"num_bytes_recovered":172097536,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12714717184,"data_stored":12714704968,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":83000,"ondisk_log_size":83000,"up":48,"acting":48,"num_store_stats":8},{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":416,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":71,"ondisk_log_size":71,"up":96,"acting":96,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":178,"num_object_clones":0,"num_object_copies":534,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":178,"num_whiteouts":0,"num_read":694,"num_read_kb":451,"num_write":417,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":149,"num_bytes_recovered":2508,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":417792,"data_stored":11106,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":396,"ondisk_log_size":396,"up":96,"acting":96,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":1613,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":24,"num_read_kb":24,"num_write":10,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":4839,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":6,"ondisk_log_size":6,"up":96,"acting":96,"num_store_stats":8},{"poolid":2,"num_pg":3,"stat_sum":{"num_bytes":408,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":220,"num_read_kb":215,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":38,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":8,"ondisk_log_size":8,"up":9,"acting":9,"num_store_stats":7},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":231,"num_write":113,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1771104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":2375680,"data_stored":2361472,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":79,"ondisk_log_size":79,"up":3,"acting":3,"num_store_stats":4}],"osd_stats":[{"osd":7,"up_from":56,"seq":240518168625,"num_pgs":58,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1406004,"kb_used_data":1324836,"kb_used_omap":1,"kb_used_meta":81150,"kb_avail":92339148,"statfs":{"total":95995035648,"available":94555287552,"internally_reserved":0,"allocated":1356632064,"data_stored":1356232792,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":83098058},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":47,"seq":201863462967,"num_pgs":48,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1406116,"kb_used_data":1306900,"kb_used_omap":1,"kb_used_meta":99198,"kb_avail":92339036,"statfs":{"total":95995035648,"available":94555172864,"internally_reserved":0,"allocated":1338265600,"data_stored":1337881651,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":101579206},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":39,"seq":167503724605,"num_pgs":52,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1106348,"kb_used_data":1034908,"kb_used_omap":1,"kb_used_meta":71422,"kb_avail":92638804,"statfs":{"total":95995035648,"available":94862135296,"internally_reserved":0,"allocated":1059745792,"data_stored":1059354001,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":73136582},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":32,"seq":137438953538,"num_pgs":63,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1392444,"kb_used_data":1311276,"kb_used_omap":1,"kb_used_meta":81150,"kb_avail":92352708,"statfs":{"total":95995035648,"available":94569172992,"internally_reserved":0,"allocated":1342746624,"data_stored":1342339129,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":83098053},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":26,"seq":111669149769,"num_pgs":65,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2473360,"kb_used_data":2329476,"kb_used_omap":1,"kb_used_meta":143614,"kb_avail":91271792,"statfs":{"total":95995035648,"available":93462315008,"internally_reserved":0,"allocated":2385383424,"data_stored":2385022151,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":147061192},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411407,"num_pgs":44,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1669416,"kb_used_data":1557656,"kb_used_omap":1,"kb_used_meta":111678,"kb_avail":92075736,"statfs":{"total":95995035648,"available":94285553664,"internally_reserved":0,"allocated":1595039744,"data_stored":1594652045,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":114358725},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574931,"num_pgs":61,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1691868,"kb_used_data":1588492,"kb_used_omap":1,"kb_used_meta":103358,"kb_avail":92053284,"statfs":{"total":95995035648,"available":94262562816,"internally_reserved":0,"allocated":1626615808,"data_stored":1626176758,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":105839045},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738458,"num_pgs":54,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2223108,"kb_used_data":2093364,"kb_used_omap":1,"kb_used_meta":129726,"kb_avail":91522044,"statfs":{"total":95995035648,"available":93718573056,"internally_reserved":0,"allocated":2143604736,"data_stored":2143189819,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":132839877},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1131,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1521,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":436,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":46,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1177,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":436,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":1898,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1128,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":1898,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1474,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":1100,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1034,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1254,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":403,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":416,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":403,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":2106003456,"data_stored":2105999384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":1609699328,"data_stored":1609695256,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":1588592640,"data_stored":1588592640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":2377646080,"data_stored":2377646080,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":1328742400,"data_stored":1328742400,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":1046806528,"data_stored":1046806528,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":1324617728,"data_stored":1324613656,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":1332609024,"data_stored":1332609024,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T08:06:37.133 INFO:tasks.cephadm.ceph_manager.ceph:would drop below min_pgs, proj 6, currently 16 2024-09-18T08:06:37.134 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:06:37.135 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:06:37.136 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3' 2024-09-18T08:06:37.237 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2 2024-09-18T08:06:37.277 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:06:37.277 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:06:37.278 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2' 2024-09-18T08:06:37.379 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5 2024-09-18T08:06:37.406 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:06:37.406 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:06:37 smithi050 ceph-mon[32371]: from='client.15039 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-18T08:06:37.406 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:06:37 smithi050 ceph-mon[32371]: from='mgr.24470 172.21.15.7:0/2594390261' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-18T08:06:37.406 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:06:37 smithi050 ceph-mon[32371]: from='mgr.24470 172.21.15.7:0/2594390261' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-18T08:06:37.406 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:06:37 smithi050 ceph-mon[32371]: from='mgr.24470 172.21.15.7:0/2594390261' entity='mgr.y' 2024-09-18T08:06:37.406 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:06:37 smithi050 ceph-mon[32371]: from='client.? 172.21.15.7:0/4017678982' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-18T08:06:37.407 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:06:37.407 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5' 2024-09-18T08:06:37.481 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:06:37 smithi007 ceph-mon[29459]: from='client.15039 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-18T08:06:37.481 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:06:37 smithi007 ceph-mon[29459]: from='mgr.24470 172.21.15.7:0/2594390261' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-18T08:06:37.481 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:06:37 smithi007 ceph-mon[29459]: from='mgr.24470 172.21.15.7:0/2594390261' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-18T08:06:37.481 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:06:37 smithi007 ceph-mon[29459]: from='mgr.24470 172.21.15.7:0/2594390261' entity='mgr.y' 2024-09-18T08:06:37.481 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:06:37 smithi007 ceph-mon[29459]: from='client.? 172.21.15.7:0/4017678982' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-18T08:06:37.481 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:06:37 smithi007 ceph-mon[39249]: from='client.15039 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-18T08:06:37.481 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:06:37 smithi007 ceph-mon[39249]: from='mgr.24470 172.21.15.7:0/2594390261' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-18T08:06:37.481 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:06:37 smithi007 ceph-mon[39249]: from='mgr.24470 172.21.15.7:0/2594390261' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-18T08:06:37.482 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:06:37 smithi007 ceph-mon[39249]: from='mgr.24470 172.21.15.7:0/2594390261' entity='mgr.y' 2024-09-18T08:06:37.482 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:06:37 smithi007 ceph-mon[39249]: from='client.? 172.21.15.7:0/4017678982' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-18T08:06:37.508 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0 2024-09-18T08:06:37.545 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:06:37.546 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:06:37.546 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0' 2024-09-18T08:06:37.645 DEBUG:teuthology.orchestra.run.smithi050:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid 779349ca-7593-11ef-bceb-c7b262605968 -- ceph daemon osd.4 dump_historic_ops 2024-09-18T08:06:37.647 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1 2024-09-18T08:06:37.689 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:06:37.689 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:06:37.690 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1' 2024-09-18T08:06:37.791 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3 2024-09-18T08:06:37.827 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:06:37.828 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:06:37.828 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3' 2024-09-18T08:06:37.929 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3 2024-09-18T08:06:37.993 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/779349ca-7593-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T08:06:37.994 INFO:teuthology.orchestra.run.smithi050.stderr:Inferring config /var/lib/ceph/779349ca-7593-11ef-bceb-c7b262605968/mon.b/config 2024-09-18T08:06:38.003 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:06:38.005 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:06:38.006 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3' 2024-09-18T08:06:38.107 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6 2024-09-18T08:06:38.138 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:06:38.139 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:06:38.140 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6' 2024-09-18T08:06:38.240 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4 2024-09-18T08:06:38.271 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:06:38.272 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:06:38.272 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4' 2024-09-18T08:06:38.373 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6 2024-09-18T08:06:38.403 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:06:38.403 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:06:38.404 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6' 2024-09-18T08:06:38.480 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:06:38 smithi007 ceph-mon[39249]: pgmap v169: 148 pgs: 148 active+clean; 3.9 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 892 op/s; 384 KiB/s, 12 objects/s recovering 2024-09-18T08:06:38.481 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:06:38 smithi007 ceph-mon[29459]: pgmap v169: 148 pgs: 148 active+clean; 3.9 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 892 op/s; 384 KiB/s, 12 objects/s recovering 2024-09-18T08:06:38.504 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6 2024-09-18T08:06:38.533 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:06:38.534 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:06:38.535 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6' 2024-09-18T08:06:38.587 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2024-09-18T08:06:38.587 INFO:teuthology.orchestra.run.smithi050.stdout: "size": 20, 2024-09-18T08:06:38.587 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 600, 2024-09-18T08:06:38.588 INFO:teuthology.orchestra.run.smithi050.stdout: "ops": [ 2024-09-18T08:06:38.588 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:06:38.588 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_repop(client.14805.0:39798 7.9 e95/83 7:93915262:::benchmark_data_smithi007_73155_object39797:head v 95'2483, mlcod=95'2480)", 2024-09-18T08:06:38.588 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-09-18T08:04:24.066023+0000", 2024-09-18T08:06:38.588 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 134.51829066299999, 2024-09-18T08:06:38.588 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.052629331000000001, 2024-09-18T08:06:38.588 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-09-18T08:06:38.588 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-09-18T08:06:38.588 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T08:06:38.588 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-09-18T08:06:38.588 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:06:38.588 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-09-18T08:06:38.588 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:24.066023+0000", 2024-09-18T08:06:38.589 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-18T08:06:38.589 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:06:38.589 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:06:38.589 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-09-18T08:06:38.589 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:24.066023+0000", 2024-09-18T08:06:38.589 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-18T08:06:38.589 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:06:38.589 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:06:38.589 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-09-18T08:06:38.589 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:24.066024+0000", 2024-09-18T08:06:38.589 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.0780000000000001e-06 2024-09-18T08:06:38.589 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:06:38.589 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:06:38.590 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-09-18T08:06:38.590 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:24.066080+0000", 2024-09-18T08:06:38.590 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 5.6041000000000001e-05 2024-09-18T08:06:38.590 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:06:38.590 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:06:38.590 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-09-18T08:06:38.590 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:24.112425+0000", 2024-09-18T08:06:38.590 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.046345056000000003 2024-09-18T08:06:38.590 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:06:38.590 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:06:38.590 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-09-18T08:06:38.590 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:24.112528+0000", 2024-09-18T08:06:38.591 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00010244 2024-09-18T08:06:38.591 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:06:38.591 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:06:38.591 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-18T08:06:38.591 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:24.114378+0000", 2024-09-18T08:06:38.591 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0018503759999999999 2024-09-18T08:06:38.591 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:06:38.591 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:06:38.591 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-09-18T08:06:38.591 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:24.114396+0000", 2024-09-18T08:06:38.591 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.8238999999999999e-05 2024-09-18T08:06:38.591 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:06:38.591 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:06:38.592 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "commit_sent", 2024-09-18T08:06:38.592 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:24.118608+0000", 2024-09-18T08:06:38.592 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0042119289999999997 2024-09-18T08:06:38.592 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:06:38.592 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:06:38.592 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-09-18T08:06:38.592 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:24.118652+0000", 2024-09-18T08:06:38.592 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4.4172000000000003e-05 2024-09-18T08:06:38.592 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-18T08:06:38.592 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-09-18T08:06:38.592 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-18T08:06:38.592 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:06:38.592 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:06:38.593 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.24356.0:1183 5.1f 5:f95f44c2:::notify.0:head [watch reconnect cookie 94700888679552 gen 1] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e101)", 2024-09-18T08:06:38.593 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-09-18T08:04:42.950623+0000", 2024-09-18T08:06:38.593 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 115.633690601, 2024-09-18T08:06:38.593 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.35355955700000002, 2024-09-18T08:06:38.593 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-09-18T08:06:38.593 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-09-18T08:06:38.593 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "started", 2024-09-18T08:06:38.593 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2024-09-18T08:06:38.593 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.24356", 2024-09-18T08:06:38.593 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "172.21.15.7:0/219168083", 2024-09-18T08:06:38.593 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 1183 2024-09-18T08:06:38.593 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:06:38.593 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-09-18T08:06:38.593 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:06:38.594 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-09-18T08:06:38.594 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:42.950623+0000", 2024-09-18T08:06:38.594 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-18T08:06:38.594 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:06:38.594 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:06:38.594 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-09-18T08:06:38.594 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:42.950623+0000", 2024-09-18T08:06:38.594 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-18T08:06:38.594 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:06:38.594 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:06:38.594 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-09-18T08:06:38.594 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:42.950624+0000", 2024-09-18T08:06:38.594 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 8.6499999999999998e-07 2024-09-18T08:06:38.595 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:06:38.595 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:06:38.595 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-09-18T08:06:38.595 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:42.950628+0000", 2024-09-18T08:06:38.595 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4.0879999999999997e-06 2024-09-18T08:06:38.595 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:06:38.595 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:06:38.595 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-09-18T08:06:38.595 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:42.950644+0000", 2024-09-18T08:06:38.595 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.6507e-05 2024-09-18T08:06:38.595 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:06:38.595 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:06:38.595 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-09-18T08:06:38.596 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:42.950651+0000", 2024-09-18T08:06:38.596 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 6.5599999999999999e-06 2024-09-18T08:06:38.596 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:06:38.596 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:06:38.596 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-18T08:06:38.596 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:42.950713+0000", 2024-09-18T08:06:38.596 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 6.2360000000000006e-05 2024-09-18T08:06:38.596 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:06:38.596 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:06:38.596 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "waiting for peered", 2024-09-18T08:06:38.596 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:42.950724+0000", 2024-09-18T08:06:38.596 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.0329999999999999e-05 2024-09-18T08:06:38.596 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:06:38.596 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:06:38.597 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-18T08:06:38.597 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:43.301407+0000", 2024-09-18T08:06:38.597 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.35068303499999998 2024-09-18T08:06:38.597 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:06:38.597 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:06:38.597 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-09-18T08:06:38.597 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:43.301697+0000", 2024-09-18T08:06:38.597 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000289938 2024-09-18T08:06:38.597 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:06:38.597 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:06:38.597 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-09-18T08:06:38.597 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:43.304183+0000", 2024-09-18T08:06:38.597 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0024858739999999999 2024-09-18T08:06:38.597 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-18T08:06:38.598 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-09-18T08:06:38.598 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-18T08:06:38.598 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:06:38.598 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:06:38.598 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.24356.0:1184 4.5 4:a60f9269:lc::lua_package_allowlist:head [watch reconnect cookie 94700889442560 gen 1] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e101)", 2024-09-18T08:06:38.598 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-09-18T08:04:42.950665+0000", 2024-09-18T08:06:38.598 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 115.63364822200001, 2024-09-18T08:06:38.598 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.35372086800000002, 2024-09-18T08:06:38.598 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-09-18T08:06:38.598 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-09-18T08:06:38.598 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "started", 2024-09-18T08:06:38.598 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2024-09-18T08:06:38.598 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.24356", 2024-09-18T08:06:38.598 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "172.21.15.7:0/219168083", 2024-09-18T08:06:38.599 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 1184 2024-09-18T08:06:38.599 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:06:38.599 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-09-18T08:06:38.599 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:06:38.599 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-09-18T08:06:38.599 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:42.950665+0000", 2024-09-18T08:06:38.599 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-18T08:06:38.599 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:06:38.599 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:06:38.599 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-09-18T08:06:38.599 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:42.950665+0000", 2024-09-18T08:06:38.599 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-18T08:06:38.599 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:06:38.599 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:06:38.600 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-09-18T08:06:38.600 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:42.950666+0000", 2024-09-18T08:06:38.600 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 6.1099999999999995e-07 2024-09-18T08:06:38.600 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:06:38.600 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:06:38.600 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-09-18T08:06:38.600 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:42.950668+0000", 2024-09-18T08:06:38.600 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 2.514e-06 2024-09-18T08:06:38.600 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:06:38.600 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:06:38.600 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-09-18T08:06:38.600 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:42.950683+0000", 2024-09-18T08:06:38.600 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.4827000000000001e-05 2024-09-18T08:06:38.600 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:06:38.601 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:06:38.601 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-09-18T08:06:38.601 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:42.950689+0000", 2024-09-18T08:06:38.601 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 6.0789999999999997e-06 2024-09-18T08:06:38.601 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:06:38.601 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:06:38.601 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-18T08:06:38.601 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:42.950753+0000", 2024-09-18T08:06:38.601 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 6.3466999999999994e-05 2024-09-18T08:06:38.601 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:06:38.601 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:06:38.601 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "waiting for peered", 2024-09-18T08:06:38.601 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:42.950763+0000", 2024-09-18T08:06:38.602 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.0193e-05 2024-09-18T08:06:38.602 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:06:38.602 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:06:38.602 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-18T08:06:38.603 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:43.301959+0000", 2024-09-18T08:06:38.603 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.35119641299999999 2024-09-18T08:06:38.603 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:06:38.603 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:06:38.603 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-09-18T08:06:38.603 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:43.302091+0000", 2024-09-18T08:06:38.603 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00013126500000000001 2024-09-18T08:06:38.603 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:06:38.603 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:06:38.603 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-09-18T08:06:38.603 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:43.304386+0000", 2024-09-18T08:06:38.603 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0022954989999999999 2024-09-18T08:06:38.603 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-18T08:06:38.604 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-09-18T08:06:38.604 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-18T08:06:38.604 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:06:38.604 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:06:38.604 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.14805.0:84652 7.1 7:8d4001d4:::benchmark_data_smithi007_73155_object84651:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e107)", 2024-09-18T08:06:38.604 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-09-18T08:05:00.706304+0000", 2024-09-18T08:06:38.604 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 97.878010037999999, 2024-09-18T08:06:38.604 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.066859704000000006, 2024-09-18T08:06:38.604 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-09-18T08:06:38.604 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-09-18T08:06:38.604 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T08:06:38.604 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2024-09-18T08:06:38.604 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.14805", 2024-09-18T08:06:38.604 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "172.21.15.7:0/640320436", 2024-09-18T08:06:38.605 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 84652 2024-09-18T08:06:38.605 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:06:38.605 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-09-18T08:06:38.605 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:06:38.605 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-09-18T08:06:38.605 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:05:00.706304+0000", 2024-09-18T08:06:38.605 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-18T08:06:38.605 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:06:38.605 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:06:38.605 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-09-18T08:06:38.605 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:05:00.706304+0000", 2024-09-18T08:06:38.605 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-18T08:06:38.605 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:06:38.606 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:06:38.606 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-09-18T08:06:38.606 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:05:00.706305+0000", 2024-09-18T08:06:38.606 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.116e-06 2024-09-18T08:06:38.606 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:06:38.606 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:06:38.606 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-09-18T08:06:38.606 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:05:00.706355+0000", 2024-09-18T08:06:38.606 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 5.0302000000000001e-05 2024-09-18T08:06:38.606 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:06:38.606 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:06:38.606 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-09-18T08:06:38.606 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:05:00.706364+0000", 2024-09-18T08:06:38.606 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 9.4609999999999997e-06 2024-09-18T08:06:38.607 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:06:38.607 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:06:38.607 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-09-18T08:06:38.607 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:05:00.706369+0000", 2024-09-18T08:06:38.607 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 5.0889999999999999e-06 2024-09-18T08:06:38.607 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:06:38.607 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:06:38.607 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-18T08:06:38.607 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:05:00.706412+0000", 2024-09-18T08:06:38.607 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4.2211e-05 2024-09-18T08:06:38.607 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:06:38.607 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:06:38.607 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-09-18T08:06:38.607 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:05:00.706513+0000", 2024-09-18T08:06:38.607 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000100887 2024-09-18T08:06:38.608 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:06:38.608 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:06:38.608 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "waiting for subops from 3,6", 2024-09-18T08:06:38.608 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:05:00.706613+0000", 2024-09-18T08:06:38.608 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 9.9984999999999999e-05 2024-09-18T08:06:38.608 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:06:38.608 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:06:38.608 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "op_commit", 2024-09-18T08:06:38.608 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:05:00.708644+0000", 2024-09-18T08:06:38.608 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0020316119999999999 2024-09-18T08:06:38.608 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:06:38.608 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:06:38.608 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:06:38.609 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:05:00.708857+0000", 2024-09-18T08:06:38.609 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000212494 2024-09-18T08:06:38.609 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:06:38.609 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:06:38.609 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:06:38.609 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:05:00.773039+0000", 2024-09-18T08:06:38.609 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.064182634000000002 2024-09-18T08:06:38.609 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:06:38.609 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:06:38.609 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "commit_sent", 2024-09-18T08:06:38.609 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:05:00.773110+0000", 2024-09-18T08:06:38.609 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 7.0203999999999994e-05 2024-09-18T08:06:38.609 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:06:38.610 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:06:38.610 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-09-18T08:06:38.610 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:05:00.773163+0000", 2024-09-18T08:06:38.610 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 5.3708999999999997e-05 2024-09-18T08:06:38.610 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-18T08:06:38.610 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-09-18T08:06:38.610 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-18T08:06:38.610 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:06:38.610 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:06:38.610 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.14805.0:84658 7.1 7:893b354a:::benchmark_data_smithi007_73155_object84657:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e107)", 2024-09-18T08:06:38.610 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-09-18T08:05:00.713288+0000", 2024-09-18T08:06:38.610 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 97.871025337999995, 2024-09-18T08:06:38.610 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.060005970999999998, 2024-09-18T08:06:38.611 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-09-18T08:06:38.611 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-09-18T08:06:38.611 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T08:06:38.611 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2024-09-18T08:06:38.611 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.14805", 2024-09-18T08:06:38.611 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "172.21.15.7:0/640320436", 2024-09-18T08:06:38.611 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 84658 2024-09-18T08:06:38.611 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:06:38.611 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-09-18T08:06:38.611 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:06:38.611 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-09-18T08:06:38.611 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:05:00.713288+0000", 2024-09-18T08:06:38.611 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-18T08:06:38.611 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:06:38.611 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:06:38.612 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-09-18T08:06:38.612 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:05:00.713288+0000", 2024-09-18T08:06:38.612 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-18T08:06:38.612 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:06:38.612 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:06:38.612 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-09-18T08:06:38.612 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:05:00.713289+0000", 2024-09-18T08:06:38.612 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.2219999999999999e-06 2024-09-18T08:06:38.612 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:06:38.612 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:06:38.612 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-09-18T08:06:38.612 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:05:00.713317+0000", 2024-09-18T08:06:38.612 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 2.7093e-05 2024-09-18T08:06:38.612 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:06:38.613 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:06:38.613 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-09-18T08:06:38.613 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:05:00.713328+0000", 2024-09-18T08:06:38.613 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.1313e-05 2024-09-18T08:06:38.613 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:06:38.613 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:06:38.613 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-09-18T08:06:38.613 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:05:00.713333+0000", 2024-09-18T08:06:38.613 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 5.5300000000000004e-06 2024-09-18T08:06:38.613 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:06:38.614 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:06:38.614 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-18T08:06:38.614 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:05:00.713374+0000", 2024-09-18T08:06:38.614 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4.0632e-05 2024-09-18T08:06:38.614 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:06:38.614 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:06:38.614 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-09-18T08:06:38.614 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:05:00.713460+0000", 2024-09-18T08:06:38.614 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 8.6366000000000006e-05 2024-09-18T08:06:38.614 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:06:38.614 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:06:38.614 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "waiting for subops from 3,6", 2024-09-18T08:06:38.614 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:05:00.713557+0000", 2024-09-18T08:06:38.615 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 9.6329000000000006e-05 2024-09-18T08:06:38.615 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:06:38.615 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:06:38.615 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "op_commit", 2024-09-18T08:06:38.615 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:05:00.715531+0000", 2024-09-18T08:06:38.615 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.001974066 2024-09-18T08:06:38.615 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:06:38.615 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:06:38.615 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:06:38.615 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:05:00.715621+0000", 2024-09-18T08:06:38.615 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 9.0563999999999996e-05 2024-09-18T08:06:38.615 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:06:38.615 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:06:38.615 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:06:38.615 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:05:00.773245+0000", 2024-09-18T08:06:38.616 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.057623665999999997 2024-09-18T08:06:38.616 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:06:38.616 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:06:38.616 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "commit_sent", 2024-09-18T08:06:38.616 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:05:00.773273+0000", 2024-09-18T08:06:38.616 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 2.8470999999999999e-05 2024-09-18T08:06:38.616 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:06:38.616 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:06:38.616 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-09-18T08:06:38.616 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:05:00.773294+0000", 2024-09-18T08:06:38.616 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 2.0718999999999999e-05 2024-09-18T08:06:38.616 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-18T08:06:38.616 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-09-18T08:06:38.617 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-18T08:06:38.617 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:06:38.617 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:06:38.617 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.14805.0:93941 7.1 7:85471127:::benchmark_data_smithi007_73155_object93940:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e108)", 2024-09-18T08:06:38.617 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-09-18T08:05:09.718257+0000", 2024-09-18T08:06:38.617 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 88.866056583000002, 2024-09-18T08:06:38.617 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.052045593000000001, 2024-09-18T08:06:38.617 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-09-18T08:06:38.617 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-09-18T08:06:38.617 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T08:06:38.617 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2024-09-18T08:06:38.617 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.14805", 2024-09-18T08:06:38.617 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "172.21.15.7:0/640320436", 2024-09-18T08:06:38.618 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 93941 2024-09-18T08:06:38.618 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:06:38.618 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-09-18T08:06:38.618 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:06:38.618 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-09-18T08:06:38.618 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:05:09.718257+0000", 2024-09-18T08:06:38.618 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-18T08:06:38.618 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:06:38.618 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:06:38.618 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-09-18T08:06:38.618 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:05:09.718257+0000", 2024-09-18T08:06:38.618 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-18T08:06:38.618 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:06:38.618 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:06:38.619 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-09-18T08:06:38.619 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:05:09.718259+0000", 2024-09-18T08:06:38.619 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.838e-06 2024-09-18T08:06:38.619 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:06:38.619 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:06:38.619 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-09-18T08:06:38.619 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:05:09.718298+0000", 2024-09-18T08:06:38.619 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 3.8931999999999999e-05 2024-09-18T08:06:38.619 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:06:38.619 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:06:38.619 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-09-18T08:06:38.619 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:05:09.718315+0000", 2024-09-18T08:06:38.619 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.6847e-05 2024-09-18T08:06:38.619 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:06:38.620 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:06:38.620 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-09-18T08:06:38.620 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:05:09.718322+0000", 2024-09-18T08:06:38.620 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 7.4379999999999998e-06 2024-09-18T08:06:38.620 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:06:38.620 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:06:38.620 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-18T08:06:38.620 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:05:09.718721+0000", 2024-09-18T08:06:38.620 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00039854099999999998 2024-09-18T08:06:38.621 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:06:38.621 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:06:38.621 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-09-18T08:06:38.621 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:05:09.719044+0000", 2024-09-18T08:06:38.621 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00032366200000000001 2024-09-18T08:06:38.621 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:06:38.621 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:06:38.621 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "waiting for subops from 3,6", 2024-09-18T08:06:38.621 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:05:09.719538+0000", 2024-09-18T08:06:38.621 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00049423200000000005 2024-09-18T08:06:38.621 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:06:38.621 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:06:38.621 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "op_commit", 2024-09-18T08:06:38.621 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:05:09.722336+0000", 2024-09-18T08:06:38.622 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0027971929999999999 2024-09-18T08:06:38.622 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:06:38.622 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:06:38.622 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:06:38.622 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:05:09.722645+0000", 2024-09-18T08:06:38.622 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000309649 2024-09-18T08:06:38.622 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:06:38.622 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:06:38.622 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:06:38.622 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:05:09.770206+0000", 2024-09-18T08:06:38.622 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.047561009000000001 2024-09-18T08:06:38.622 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:06:38.622 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:06:38.623 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "commit_sent", 2024-09-18T08:06:38.623 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:05:09.770254+0000", 2024-09-18T08:06:38.623 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4.7861999999999997e-05 2024-09-18T08:06:38.623 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:06:38.623 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:06:38.623 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-09-18T08:06:38.623 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:05:09.770303+0000", 2024-09-18T08:06:38.623 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4.8390000000000003e-05 2024-09-18T08:06:38.623 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-18T08:06:38.623 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-09-18T08:06:38.623 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-18T08:06:38.623 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:06:38.623 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:06:38.624 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.14805.0:113361 7.1 7:807e3ad4:::benchmark_data_smithi007_73155_object113360:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e119)", 2024-09-18T08:06:38.624 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-09-18T08:05:30.525448+0000", 2024-09-18T08:06:38.624 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 68.058865642000001, 2024-09-18T08:06:38.624 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.069507528999999998, 2024-09-18T08:06:38.624 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-09-18T08:06:38.624 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-09-18T08:06:38.624 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T08:06:38.624 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2024-09-18T08:06:38.624 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.14805", 2024-09-18T08:06:38.624 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "172.21.15.7:0/640320436", 2024-09-18T08:06:38.624 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 113361 2024-09-18T08:06:38.624 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:06:38.624 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-09-18T08:06:38.624 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:06:38.625 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-09-18T08:06:38.625 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:05:30.525448+0000", 2024-09-18T08:06:38.625 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-18T08:06:38.625 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:06:38.625 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:06:38.625 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-09-18T08:06:38.625 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:05:30.525448+0000", 2024-09-18T08:06:38.625 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-18T08:06:38.625 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:06:38.625 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:06:38.625 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-09-18T08:06:38.625 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:05:30.525450+0000", 2024-09-18T08:06:38.625 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.8589999999999999e-06 2024-09-18T08:06:38.625 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:06:38.626 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:06:38.626 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-09-18T08:06:38.626 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:05:30.525500+0000", 2024-09-18T08:06:38.626 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4.9812999999999999e-05 2024-09-18T08:06:38.626 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:06:38.626 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:06:38.626 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-09-18T08:06:38.626 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:05:30.525513+0000", 2024-09-18T08:06:38.626 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.3337e-05 2024-09-18T08:06:38.626 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:06:38.626 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:06:38.626 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-09-18T08:06:38.626 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:05:30.525521+0000", 2024-09-18T08:06:38.626 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 7.7100000000000007e-06 2024-09-18T08:06:38.627 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:06:38.627 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:06:38.627 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-18T08:06:38.627 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:05:30.525586+0000", 2024-09-18T08:06:38.627 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 6.5506000000000006e-05 2024-09-18T08:06:38.627 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:06:38.627 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:06:38.627 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-09-18T08:06:38.627 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:05:30.525724+0000", 2024-09-18T08:06:38.627 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00013775699999999999 2024-09-18T08:06:38.627 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:06:38.627 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:06:38.627 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "waiting for subops from 3,6", 2024-09-18T08:06:38.627 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:05:30.525875+0000", 2024-09-18T08:06:38.628 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00015107799999999999 2024-09-18T08:06:38.628 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:06:38.628 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:06:38.628 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:06:38.628 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:05:30.528390+0000", 2024-09-18T08:06:38.628 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0025149479999999999 2024-09-18T08:06:38.628 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:06:38.628 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:06:38.628 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "op_commit", 2024-09-18T08:06:38.628 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:05:30.528441+0000", 2024-09-18T08:06:38.628 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 5.0591999999999999e-05 2024-09-18T08:06:38.628 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:06:38.628 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:06:38.628 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:06:38.628 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:05:30.594832+0000", 2024-09-18T08:06:38.629 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.066391475000000005 2024-09-18T08:06:38.629 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:06:38.629 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:06:38.629 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "commit_sent", 2024-09-18T08:06:38.629 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:05:30.594907+0000", 2024-09-18T08:06:38.629 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 7.4995999999999998e-05 2024-09-18T08:06:38.629 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:06:38.629 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:06:38.629 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-09-18T08:06:38.629 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:05:30.594955+0000", 2024-09-18T08:06:38.629 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4.8458000000000003e-05 2024-09-18T08:06:38.629 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-18T08:06:38.629 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-09-18T08:06:38.629 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-18T08:06:38.630 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:06:38.630 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:06:38.630 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.14805.0:113528 7.1 7:8ab97e16:::benchmark_data_smithi007_73155_object113527:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e119)", 2024-09-18T08:06:38.630 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-09-18T08:05:30.734120+0000", 2024-09-18T08:06:38.630 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 67.850193508999993, 2024-09-18T08:06:38.630 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.056096998000000002, 2024-09-18T08:06:38.630 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-09-18T08:06:38.630 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-09-18T08:06:38.630 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T08:06:38.630 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2024-09-18T08:06:38.630 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.14805", 2024-09-18T08:06:38.630 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "172.21.15.7:0/640320436", 2024-09-18T08:06:38.630 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 113528 2024-09-18T08:06:38.631 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:06:38.631 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-09-18T08:06:38.631 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:06:38.631 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-09-18T08:06:38.631 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:05:30.734120+0000", 2024-09-18T08:06:38.631 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-18T08:06:38.631 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:06:38.631 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:06:38.631 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-09-18T08:06:38.631 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:05:30.734120+0000", 2024-09-18T08:06:38.631 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-18T08:06:38.631 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:06:38.631 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:06:38.631 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-09-18T08:06:38.632 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:05:30.734121+0000", 2024-09-18T08:06:38.632 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 7.6700000000000003e-07 2024-09-18T08:06:38.632 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:06:38.632 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:06:38.632 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-09-18T08:06:38.632 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:05:30.734198+0000", 2024-09-18T08:06:38.632 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 7.6809000000000003e-05 2024-09-18T08:06:38.632 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:06:38.632 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:06:38.632 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-09-18T08:06:38.632 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:05:30.734208+0000", 2024-09-18T08:06:38.632 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.0485999999999999e-05 2024-09-18T08:06:38.632 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:06:38.633 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:06:38.633 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-09-18T08:06:38.633 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:05:30.734214+0000", 2024-09-18T08:06:38.633 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 5.626e-06 2024-09-18T08:06:38.633 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:06:38.633 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:06:38.633 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-18T08:06:38.633 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:05:30.735773+0000", 2024-09-18T08:06:38.633 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.001559178 2024-09-18T08:06:38.633 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:06:38.633 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:06:38.633 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-09-18T08:06:38.633 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:05:30.735914+0000", 2024-09-18T08:06:38.633 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00014068699999999999 2024-09-18T08:06:38.634 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:06:38.634 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:06:38.634 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "waiting for subops from 3,6", 2024-09-18T08:06:38.634 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:05:30.736006+0000", 2024-09-18T08:06:38.634 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 9.2829000000000002e-05 2024-09-18T08:06:38.634 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:06:38.634 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:06:38.634 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:06:38.634 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:05:30.738536+0000", 2024-09-18T08:06:38.634 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0025297399999999999 2024-09-18T08:06:38.634 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:06:38.634 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:06:38.634 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "op_commit", 2024-09-18T08:06:38.635 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:05:30.738550+0000", 2024-09-18T08:06:38.635 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.3509999999999999e-05 2024-09-18T08:06:38.635 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:06:38.635 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:06:38.635 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:06:38.635 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:05:30.790151+0000", 2024-09-18T08:06:38.635 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.051601683000000002 2024-09-18T08:06:38.635 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:06:38.635 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:06:38.635 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "commit_sent", 2024-09-18T08:06:38.635 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:05:30.790188+0000", 2024-09-18T08:06:38.635 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 3.6371999999999997e-05 2024-09-18T08:06:38.635 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:06:38.636 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:06:38.636 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-09-18T08:06:38.636 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:05:30.790217+0000", 2024-09-18T08:06:38.636 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 2.9311000000000001e-05 2024-09-18T08:06:38.636 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-18T08:06:38.636 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-09-18T08:06:38.636 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-18T08:06:38.636 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:06:38.636 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:06:38.636 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.14805.0:141888 7.1 7:82d8ae08:::benchmark_data_smithi007_73155_object23597:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e127)", 2024-09-18T08:06:38.636 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-09-18T08:05:59.090389+0000", 2024-09-18T08:06:38.636 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 39.493924077999999, 2024-09-18T08:06:38.636 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.070629848999999995, 2024-09-18T08:06:38.636 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-09-18T08:06:38.637 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-09-18T08:06:38.637 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T08:06:38.637 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2024-09-18T08:06:38.637 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.14805", 2024-09-18T08:06:38.637 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "172.21.15.7:0/640320436", 2024-09-18T08:06:38.637 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 141888 2024-09-18T08:06:38.637 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:06:38.637 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-09-18T08:06:38.637 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:06:38.637 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-09-18T08:06:38.637 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:05:59.090389+0000", 2024-09-18T08:06:38.637 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-18T08:06:38.637 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:06:38.637 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:06:38.638 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-09-18T08:06:38.638 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:05:59.090389+0000", 2024-09-18T08:06:38.638 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-18T08:06:38.638 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:06:38.638 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:06:38.638 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-09-18T08:06:38.638 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:05:59.090391+0000", 2024-09-18T08:06:38.638 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.1689999999999999e-06 2024-09-18T08:06:38.638 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:06:38.638 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:06:38.638 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-09-18T08:06:38.638 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:05:59.090395+0000", 2024-09-18T08:06:38.639 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4.5650000000000003e-06 2024-09-18T08:06:38.639 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:06:38.639 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:06:38.639 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-09-18T08:06:38.639 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:05:59.090406+0000", 2024-09-18T08:06:38.639 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.0407999999999999e-05 2024-09-18T08:06:38.639 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:06:38.639 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:06:38.639 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-09-18T08:06:38.639 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:05:59.090413+0000", 2024-09-18T08:06:38.639 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 7.7510000000000008e-06 2024-09-18T08:06:38.639 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:06:38.639 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:06:38.640 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-18T08:06:38.640 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:05:59.090468+0000", 2024-09-18T08:06:38.640 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 5.4684e-05 2024-09-18T08:06:38.640 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:06:38.640 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:06:38.640 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-09-18T08:06:38.640 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:05:59.090560+0000", 2024-09-18T08:06:38.640 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 9.1528999999999997e-05 2024-09-18T08:06:38.640 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:06:38.640 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:06:38.640 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "waiting for subops from 3,6", 2024-09-18T08:06:38.640 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:05:59.090626+0000", 2024-09-18T08:06:38.640 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 6.6403999999999996e-05 2024-09-18T08:06:38.640 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:06:38.641 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:06:38.641 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "op_commit", 2024-09-18T08:06:38.641 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:05:59.092907+0000", 2024-09-18T08:06:38.641 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.002280528 2024-09-18T08:06:38.641 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:06:38.641 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:06:38.641 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:06:38.641 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:05:59.093035+0000", 2024-09-18T08:06:38.641 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00012871900000000001 2024-09-18T08:06:38.641 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:06:38.641 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:06:38.641 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:06:38.641 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:05:59.160952+0000", 2024-09-18T08:06:38.641 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.067916502000000004 2024-09-18T08:06:38.642 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:06:38.642 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:06:38.642 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "commit_sent", 2024-09-18T08:06:38.642 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:05:59.160987+0000", 2024-09-18T08:06:38.642 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 3.5267e-05 2024-09-18T08:06:38.642 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:06:38.642 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:06:38.642 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-09-18T08:06:38.642 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:05:59.161019+0000", 2024-09-18T08:06:38.642 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 3.2323000000000002e-05 2024-09-18T08:06:38.642 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-18T08:06:38.642 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-09-18T08:06:38.642 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-18T08:06:38.642 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:06:38.642 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:06:38.643 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.14805.0:142145 7.1 7:85ef7aca:::benchmark_data_smithi007_73155_object23854:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e127)", 2024-09-18T08:06:38.643 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-09-18T08:05:59.413428+0000", 2024-09-18T08:06:38.643 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 39.170885689999999, 2024-09-18T08:06:38.643 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.061856467999999998, 2024-09-18T08:06:38.643 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-09-18T08:06:38.643 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-09-18T08:06:38.643 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T08:06:38.643 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2024-09-18T08:06:38.643 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.14805", 2024-09-18T08:06:38.643 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "172.21.15.7:0/640320436", 2024-09-18T08:06:38.643 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 142145 2024-09-18T08:06:38.643 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:06:38.643 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-09-18T08:06:38.643 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:06:38.643 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-09-18T08:06:38.643 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:05:59.413428+0000", 2024-09-18T08:06:38.644 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-18T08:06:38.644 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:06:38.644 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:06:38.644 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-09-18T08:06:38.644 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:05:59.413428+0000", 2024-09-18T08:06:38.644 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-18T08:06:38.644 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:06:38.644 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:06:38.644 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-09-18T08:06:38.644 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:05:59.413429+0000", 2024-09-18T08:06:38.644 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.4759999999999999e-06 2024-09-18T08:06:38.644 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:06:38.644 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:06:38.644 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-09-18T08:06:38.644 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:05:59.413435+0000", 2024-09-18T08:06:38.645 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 6.0349999999999998e-06 2024-09-18T08:06:38.645 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:06:38.645 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:06:38.645 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-09-18T08:06:38.645 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:05:59.413449+0000", 2024-09-18T08:06:38.645 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.397e-05 2024-09-18T08:06:38.645 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:06:38.645 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:06:38.645 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-09-18T08:06:38.645 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:05:59.413458+0000", 2024-09-18T08:06:38.645 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 8.2779999999999993e-06 2024-09-18T08:06:38.645 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:06:38.645 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:06:38.646 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-18T08:06:38.646 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:05:59.413520+0000", 2024-09-18T08:06:38.646 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 6.2385000000000007e-05 2024-09-18T08:06:38.646 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:06:38.646 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:06:38.646 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-09-18T08:06:38.646 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:05:59.413650+0000", 2024-09-18T08:06:38.646 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00012997599999999999 2024-09-18T08:06:38.646 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:06:38.646 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:06:38.646 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "waiting for subops from 3,6", 2024-09-18T08:06:38.646 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:05:59.413777+0000", 2024-09-18T08:06:38.646 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000127133 2024-09-18T08:06:38.646 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:06:38.647 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:06:38.647 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "op_commit", 2024-09-18T08:06:38.647 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:05:59.416223+0000", 2024-09-18T08:06:38.647 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.002445773 2024-09-18T08:06:38.647 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:06:38.647 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:06:38.647 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:06:38.647 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:05:59.416586+0000", 2024-09-18T08:06:38.647 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00036319999999999999 2024-09-18T08:06:38.647 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:06:38.647 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:06:38.647 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:06:38.647 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:05:59.475199+0000", 2024-09-18T08:06:38.648 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.058613382999999998 2024-09-18T08:06:38.648 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:06:38.648 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:06:38.648 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "commit_sent", 2024-09-18T08:06:38.648 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:05:59.475239+0000", 2024-09-18T08:06:38.648 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 3.9280999999999999e-05 2024-09-18T08:06:38.648 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:06:38.648 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:06:38.648 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-09-18T08:06:38.648 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:05:59.475284+0000", 2024-09-18T08:06:38.648 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4.5578000000000003e-05 2024-09-18T08:06:38.648 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-18T08:06:38.648 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-09-18T08:06:38.649 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-18T08:06:38.649 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:06:38.649 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:06:38.649 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.14805.0:147845 7.1 7:8a0a44bd:::benchmark_data_smithi007_73155_object29554:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e127)", 2024-09-18T08:06:38.649 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-09-18T08:06:04.708313+0000", 2024-09-18T08:06:38.649 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 33.876000791999999, 2024-09-18T08:06:38.649 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.073618763000000004, 2024-09-18T08:06:38.649 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-09-18T08:06:38.649 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-09-18T08:06:38.649 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T08:06:38.649 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2024-09-18T08:06:38.649 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.14805", 2024-09-18T08:06:38.649 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "172.21.15.7:0/640320436", 2024-09-18T08:06:38.649 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 147845 2024-09-18T08:06:38.650 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:06:38.650 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-09-18T08:06:38.650 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:06:38.650 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-09-18T08:06:38.650 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:06:04.708313+0000", 2024-09-18T08:06:38.650 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-18T08:06:38.650 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:06:38.650 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:06:38.650 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-09-18T08:06:38.650 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:06:04.708313+0000", 2024-09-18T08:06:38.650 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-18T08:06:38.650 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:06:38.650 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:06:38.650 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-09-18T08:06:38.650 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:06:04.708313+0000", 2024-09-18T08:06:38.651 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 6.9999999999999997e-07 2024-09-18T08:06:38.651 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:06:38.651 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:06:38.651 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-09-18T08:06:38.651 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:06:04.708317+0000", 2024-09-18T08:06:38.651 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 3.36e-06 2024-09-18T08:06:38.651 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:06:38.651 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:06:38.651 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-09-18T08:06:38.651 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:06:04.708324+0000", 2024-09-18T08:06:38.651 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 7.5739999999999998e-06 2024-09-18T08:06:38.651 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:06:38.651 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:06:38.651 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-09-18T08:06:38.652 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:06:04.708329+0000", 2024-09-18T08:06:38.652 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4.4209999999999997e-06 2024-09-18T08:06:38.652 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:06:38.652 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:06:38.652 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-18T08:06:38.652 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:06:04.709653+0000", 2024-09-18T08:06:38.652 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0013239090000000001 2024-09-18T08:06:38.652 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:06:38.652 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:06:38.652 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-09-18T08:06:38.652 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:06:04.710395+0000", 2024-09-18T08:06:38.652 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00074240000000000005 2024-09-18T08:06:38.652 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:06:38.652 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:06:38.653 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "waiting for subops from 3,6", 2024-09-18T08:06:38.653 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:06:04.710508+0000", 2024-09-18T08:06:38.653 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000113248 2024-09-18T08:06:38.653 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:06:38.653 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:06:38.653 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "op_commit", 2024-09-18T08:06:38.653 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:06:04.712970+0000", 2024-09-18T08:06:38.653 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0024617160000000001 2024-09-18T08:06:38.653 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:06:38.653 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:06:38.653 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:06:38.653 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:06:04.714318+0000", 2024-09-18T08:06:38.653 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0013482629999999999 2024-09-18T08:06:38.654 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:06:38.654 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:06:38.654 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:06:38.654 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:06:04.781826+0000", 2024-09-18T08:06:38.654 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.067507519000000002 2024-09-18T08:06:38.654 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:06:38.654 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:06:38.654 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "commit_sent", 2024-09-18T08:06:38.654 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:06:04.781893+0000", 2024-09-18T08:06:38.654 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 6.7374999999999998e-05 2024-09-18T08:06:38.654 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:06:38.654 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:06:38.654 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-09-18T08:06:38.654 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:06:04.781932+0000", 2024-09-18T08:06:38.654 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 3.8278000000000002e-05 2024-09-18T08:06:38.655 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-18T08:06:38.655 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-09-18T08:06:38.655 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-18T08:06:38.655 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:06:38.655 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:06:38.655 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.14805.0:148169 7.1 7:8d71082f:::benchmark_data_smithi007_73155_object29878:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e127)", 2024-09-18T08:06:38.655 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-09-18T08:06:05.041000+0000", 2024-09-18T08:06:38.655 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 33.543313685999998, 2024-09-18T08:06:38.655 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.062830021, 2024-09-18T08:06:38.655 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-09-18T08:06:38.655 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-09-18T08:06:38.655 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T08:06:38.655 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2024-09-18T08:06:38.655 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.14805", 2024-09-18T08:06:38.656 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "172.21.15.7:0/640320436", 2024-09-18T08:06:38.656 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 148169 2024-09-18T08:06:38.656 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:06:38.656 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-09-18T08:06:38.656 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:06:38.656 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-09-18T08:06:38.656 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:06:05.041000+0000", 2024-09-18T08:06:38.656 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-18T08:06:38.656 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:06:38.656 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:06:38.656 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-09-18T08:06:38.656 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:06:05.041000+0000", 2024-09-18T08:06:38.656 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-18T08:06:38.657 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:06:38.657 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:06:38.657 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-09-18T08:06:38.657 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:06:05.041001+0000", 2024-09-18T08:06:38.657 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.575e-06 2024-09-18T08:06:38.657 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:06:38.657 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:06:38.657 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-09-18T08:06:38.657 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:06:05.041007+0000", 2024-09-18T08:06:38.657 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 5.0839999999999999e-06 2024-09-18T08:06:38.657 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:06:38.657 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:06:38.657 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-09-18T08:06:38.657 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:06:05.041019+0000", 2024-09-18T08:06:38.657 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.2668e-05 2024-09-18T08:06:38.658 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:06:38.658 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:06:38.658 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-09-18T08:06:38.658 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:06:05.041026+0000", 2024-09-18T08:06:38.658 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 7.0910000000000004e-06 2024-09-18T08:06:38.658 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:06:38.658 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2 2024-09-18T08:06:38.659 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:06:38.659 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-18T08:06:38.659 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:06:05.041070+0000", 2024-09-18T08:06:38.659 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4.4030999999999999e-05 2024-09-18T08:06:38.660 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:06:38.660 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:06:38.660 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-09-18T08:06:38.660 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:06:05.041178+0000", 2024-09-18T08:06:38.660 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00010746700000000001 2024-09-18T08:06:38.660 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:06:38.660 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:06:38.660 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "waiting for subops from 3,6", 2024-09-18T08:06:38.660 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:06:05.041302+0000", 2024-09-18T08:06:38.660 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000123788 2024-09-18T08:06:38.660 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:06:38.660 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:06:38.660 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "op_commit", 2024-09-18T08:06:38.661 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:06:05.044159+0000", 2024-09-18T08:06:38.661 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.002857264 2024-09-18T08:06:38.661 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:06:38.661 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:06:38.661 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:06:38.661 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:06:05.044286+0000", 2024-09-18T08:06:38.661 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00012713500000000001 2024-09-18T08:06:38.661 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:06:38.661 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:06:38.661 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:06:38.661 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:06:05.103732+0000", 2024-09-18T08:06:38.661 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.059446163000000003 2024-09-18T08:06:38.661 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:06:38.662 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:06:38.662 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "commit_sent", 2024-09-18T08:06:38.662 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:06:05.103772+0000", 2024-09-18T08:06:38.662 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 3.9966000000000003e-05 2024-09-18T08:06:38.662 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:06:38.662 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:06:38.662 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-09-18T08:06:38.662 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:06:05.103830+0000", 2024-09-18T08:06:38.662 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 5.7788999999999998e-05 2024-09-18T08:06:38.662 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-18T08:06:38.662 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-09-18T08:06:38.662 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-18T08:06:38.662 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:06:38.662 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:06:38.662 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.14805.0:148357 7.1 7:8ec47016:::benchmark_data_smithi007_73155_object30066:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e127)", 2024-09-18T08:06:38.663 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-09-18T08:06:05.261852+0000", 2024-09-18T08:06:38.663 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 33.322461265000001, 2024-09-18T08:06:38.663 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.058562336999999999, 2024-09-18T08:06:38.663 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-09-18T08:06:38.663 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-09-18T08:06:38.663 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T08:06:38.663 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2024-09-18T08:06:38.663 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.14805", 2024-09-18T08:06:38.663 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "172.21.15.7:0/640320436", 2024-09-18T08:06:38.663 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 148357 2024-09-18T08:06:38.663 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:06:38.663 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-09-18T08:06:38.663 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:06:38.664 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-09-18T08:06:38.664 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:06:05.261852+0000", 2024-09-18T08:06:38.664 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-18T08:06:38.664 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:06:38.664 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:06:38.664 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-09-18T08:06:38.664 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:06:05.261852+0000", 2024-09-18T08:06:38.664 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-18T08:06:38.664 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:06:38.664 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:06:38.664 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-09-18T08:06:38.664 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:06:05.261854+0000", 2024-09-18T08:06:38.664 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.4789999999999999e-06 2024-09-18T08:06:38.664 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:06:38.665 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:06:38.665 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-09-18T08:06:38.665 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:06:05.261860+0000", 2024-09-18T08:06:38.665 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 5.993e-06 2024-09-18T08:06:38.665 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:06:38.665 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:06:38.665 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-09-18T08:06:38.665 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:06:05.261874+0000", 2024-09-18T08:06:38.665 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.4637000000000001e-05 2024-09-18T08:06:38.665 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:06:38.665 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:06:38.665 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-09-18T08:06:38.665 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:06:05.261882+0000", 2024-09-18T08:06:38.666 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 7.9330000000000001e-06 2024-09-18T08:06:38.666 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:06:38.666 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:06:38.666 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-18T08:06:38.666 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:06:05.261954+0000", 2024-09-18T08:06:38.666 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 7.2095999999999995e-05 2024-09-18T08:06:38.666 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:06:38.666 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:06:38.666 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-09-18T08:06:38.666 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:06:05.262078+0000", 2024-09-18T08:06:38.666 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00012376600000000001 2024-09-18T08:06:38.666 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:06:38.666 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:06:38.666 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "waiting for subops from 3,6", 2024-09-18T08:06:38.667 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:06:05.262201+0000", 2024-09-18T08:06:38.667 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00012270299999999999 2024-09-18T08:06:38.667 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:06:38.667 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:06:38.667 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:06:38.667 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:06:05.264434+0000", 2024-09-18T08:06:38.667 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0022327789999999998 2024-09-18T08:06:38.667 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:06:38.667 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:06:38.668 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "op_commit", 2024-09-18T08:06:38.668 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:06:05.266518+0000", 2024-09-18T08:06:38.668 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0020840049999999999 2024-09-18T08:06:38.668 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:06:38.668 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:06:38.668 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:06:38.668 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:06:05.320298+0000", 2024-09-18T08:06:38.668 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.053779929999999997 2024-09-18T08:06:38.668 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:06:38.668 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:06:38.668 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "commit_sent", 2024-09-18T08:06:38.668 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:06:05.320362+0000", 2024-09-18T08:06:38.668 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 6.4121000000000005e-05 2024-09-18T08:06:38.669 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:06:38.669 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:06:38.669 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-09-18T08:06:38.669 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:06:05.320415+0000", 2024-09-18T08:06:38.669 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 5.2895000000000001e-05 2024-09-18T08:06:38.669 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-18T08:06:38.669 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-09-18T08:06:38.669 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-18T08:06:38.669 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:06:38.669 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:06:38.669 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.14805.0:151467 7.1 7:87478555:::benchmark_data_smithi007_73155_object33176:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e127)", 2024-09-18T08:06:38.669 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-09-18T08:06:08.235608+0000", 2024-09-18T08:06:38.669 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 30.348705913, 2024-09-18T08:06:38.669 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.058523480000000003, 2024-09-18T08:06:38.670 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-09-18T08:06:38.670 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-09-18T08:06:38.670 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T08:06:38.670 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2024-09-18T08:06:38.670 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.14805", 2024-09-18T08:06:38.670 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "172.21.15.7:0/640320436", 2024-09-18T08:06:38.670 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 151467 2024-09-18T08:06:38.670 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:06:38.670 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-09-18T08:06:38.670 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:06:38.670 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-09-18T08:06:38.670 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:06:08.235608+0000", 2024-09-18T08:06:38.670 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-18T08:06:38.670 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:06:38.670 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:06:38.671 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-09-18T08:06:38.671 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:06:08.235608+0000", 2024-09-18T08:06:38.671 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-18T08:06:38.671 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:06:38.671 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:06:38.671 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-09-18T08:06:38.671 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:06:08.235609+0000", 2024-09-18T08:06:38.671 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.514e-06 2024-09-18T08:06:38.671 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:06:38.671 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:06:38.671 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-09-18T08:06:38.671 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:06:08.235615+0000", 2024-09-18T08:06:38.671 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 5.6080000000000003e-06 2024-09-18T08:06:38.672 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:06:38.672 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:06:38.672 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-09-18T08:06:38.672 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:06:08.235629+0000", 2024-09-18T08:06:38.672 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.4343e-05 2024-09-18T08:06:38.672 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:06:38.672 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:06:38.672 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-09-18T08:06:38.672 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:06:08.235637+0000", 2024-09-18T08:06:38.672 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 8.0739999999999993e-06 2024-09-18T08:06:38.672 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:06:38.672 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:06:38.672 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-18T08:06:38.672 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:06:08.235692+0000", 2024-09-18T08:06:38.672 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 5.4851000000000001e-05 2024-09-18T08:06:38.673 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:06:38.673 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:06:38.673 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-09-18T08:06:38.673 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:06:08.235809+0000", 2024-09-18T08:06:38.673 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00011712799999999999 2024-09-18T08:06:38.673 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:06:38.673 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:06:38.673 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "waiting for subops from 3,6", 2024-09-18T08:06:38.673 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:06:08.236204+0000", 2024-09-18T08:06:38.673 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000395103 2024-09-18T08:06:38.673 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:06:38.673 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:06:38.673 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:06:38.674 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:06:08.239329+0000", 2024-09-18T08:06:38.674 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.003124319 2024-09-18T08:06:38.674 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:06:38.674 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:06:38.674 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "op_commit", 2024-09-18T08:06:38.674 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:06:08.240293+0000", 2024-09-18T08:06:38.674 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00096459200000000001 2024-09-18T08:06:38.674 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:06:38.674 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:06:38.674 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:06:38.674 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:06:08.294031+0000", 2024-09-18T08:06:38.674 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.053738068 2024-09-18T08:06:38.674 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:06:38.674 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:06:38.675 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "commit_sent", 2024-09-18T08:06:38.675 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:06:08.294084+0000", 2024-09-18T08:06:38.675 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 5.2902000000000003e-05 2024-09-18T08:06:38.675 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:06:38.675 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:06:38.675 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-09-18T08:06:38.675 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:06:08.294131+0000", 2024-09-18T08:06:38.675 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4.6978000000000003e-05 2024-09-18T08:06:38.675 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-18T08:06:38.675 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-09-18T08:06:38.675 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-18T08:06:38.675 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:06:38.675 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:06:38.675 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.14805.0:151470 7.1 7:8a658e0b:::benchmark_data_smithi007_73155_object33179:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e127)", 2024-09-18T08:06:38.676 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-09-18T08:06:08.241975+0000", 2024-09-18T08:06:38.676 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 30.342338922, 2024-09-18T08:06:38.676 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.062786429000000005, 2024-09-18T08:06:38.676 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-09-18T08:06:38.676 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-09-18T08:06:38.676 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T08:06:38.676 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2024-09-18T08:06:38.676 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.14805", 2024-09-18T08:06:38.676 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "172.21.15.7:0/640320436", 2024-09-18T08:06:38.676 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 151470 2024-09-18T08:06:38.676 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:06:38.676 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-09-18T08:06:38.676 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:06:38.676 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-09-18T08:06:38.677 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:06:08.241975+0000", 2024-09-18T08:06:38.677 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-18T08:06:38.677 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:06:38.677 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:06:38.677 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-09-18T08:06:38.677 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:06:08.241975+0000", 2024-09-18T08:06:38.677 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-18T08:06:38.677 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:06:38.677 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:06:38.677 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-09-18T08:06:38.677 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:06:08.241977+0000", 2024-09-18T08:06:38.677 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 2.3970000000000001e-06 2024-09-18T08:06:38.677 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:06:38.678 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:06:38.678 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-09-18T08:06:38.678 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:06:08.241986+0000", 2024-09-18T08:06:38.678 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 8.7809999999999994e-06 2024-09-18T08:06:38.678 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:06:38.678 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:06:38.678 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-09-18T08:06:38.678 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:06:08.242005+0000", 2024-09-18T08:06:38.678 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.9017000000000001e-05 2024-09-18T08:06:38.678 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:06:38.678 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:06:38.678 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-09-18T08:06:38.678 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:06:08.242013+0000", 2024-09-18T08:06:38.678 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 8.5010000000000004e-06 2024-09-18T08:06:38.679 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:06:38.679 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:06:38.679 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-18T08:06:38.679 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:06:08.242078+0000", 2024-09-18T08:06:38.679 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 6.4772999999999998e-05 2024-09-18T08:06:38.679 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:06:38.679 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:06:38.679 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-09-18T08:06:38.679 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:06:08.242207+0000", 2024-09-18T08:06:38.679 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00012895199999999999 2024-09-18T08:06:38.679 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:06:38.679 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:06:38.679 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "waiting for subops from 3,6", 2024-09-18T08:06:38.680 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:06:08.242281+0000", 2024-09-18T08:06:38.680 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 7.3670000000000004e-05 2024-09-18T08:06:38.680 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:06:38.680 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:06:38.680 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "op_commit", 2024-09-18T08:06:38.680 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:06:08.244516+0000", 2024-09-18T08:06:38.680 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0022348390000000002 2024-09-18T08:06:38.680 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:06:38.680 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:06:38.680 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:06:38.680 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:06:08.244704+0000", 2024-09-18T08:06:38.680 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000188884 2024-09-18T08:06:38.680 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:06:38.680 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:06:38.680 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:06:38.681 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:06:08.304618+0000", 2024-09-18T08:06:38.681 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.059913392000000003 2024-09-18T08:06:38.681 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:06:38.681 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:06:38.681 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "commit_sent", 2024-09-18T08:06:38.681 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:06:08.304704+0000", 2024-09-18T08:06:38.681 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 8.6261000000000006e-05 2024-09-18T08:06:38.681 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:06:38.681 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:06:38.681 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-09-18T08:06:38.681 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:06:08.304761+0000", 2024-09-18T08:06:38.681 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 5.6962000000000001e-05 2024-09-18T08:06:38.681 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-18T08:06:38.682 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-09-18T08:06:38.682 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-18T08:06:38.682 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:06:38.682 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:06:38.682 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.14805.0:151646 7.1 7:8ab88cf8:::benchmark_data_smithi007_73155_object33355:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e127)", 2024-09-18T08:06:38.682 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-09-18T08:06:08.437973+0000", 2024-09-18T08:06:38.682 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 30.146341, 2024-09-18T08:06:38.682 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.063089381, 2024-09-18T08:06:38.682 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-09-18T08:06:38.682 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-09-18T08:06:38.682 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T08:06:38.682 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2024-09-18T08:06:38.682 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.14805", 2024-09-18T08:06:38.682 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "172.21.15.7:0/640320436", 2024-09-18T08:06:38.683 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 151646 2024-09-18T08:06:38.683 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:06:38.683 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-09-18T08:06:38.683 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:06:38.683 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-09-18T08:06:38.683 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:06:08.437973+0000", 2024-09-18T08:06:38.683 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-18T08:06:38.683 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:06:38.683 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:06:38.683 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-09-18T08:06:38.683 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:06:08.437973+0000", 2024-09-18T08:06:38.683 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-18T08:06:38.683 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:06:38.683 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:06:38.684 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-09-18T08:06:38.684 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:06:08.437974+0000", 2024-09-18T08:06:38.684 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.1939999999999999e-06 2024-09-18T08:06:38.684 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:06:38.684 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:06:38.684 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-09-18T08:06:38.684 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:06:08.437978+0000", 2024-09-18T08:06:38.684 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4.5290000000000002e-06 2024-09-18T08:06:38.684 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:06:38.684 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:06:38.684 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-09-18T08:06:38.684 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:06:08.437990+0000", 2024-09-18T08:06:38.684 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.1888e-05 2024-09-18T08:06:38.684 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:06:38.685 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:06:38.685 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-09-18T08:06:38.685 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:06:08.437997+0000", 2024-09-18T08:06:38.685 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 6.9530000000000002e-06 2024-09-18T08:06:38.685 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:06:38.685 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:06:38.685 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-18T08:06:38.685 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:06:08.438043+0000", 2024-09-18T08:06:38.685 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4.5904e-05 2024-09-18T08:06:38.685 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:06:38.685 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:06:38.685 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-09-18T08:06:38.685 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:06:08.438120+0000", 2024-09-18T08:06:38.685 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 7.6612000000000004e-05 2024-09-18T08:06:38.685 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:06:38.686 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:06:38.686 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "waiting for subops from 3,6", 2024-09-18T08:06:38.686 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:06:08.438188+0000", 2024-09-18T08:06:38.686 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 6.8455999999999996e-05 2024-09-18T08:06:38.686 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:06:38.686 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:06:38.686 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "op_commit", 2024-09-18T08:06:38.686 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:06:08.445164+0000", 2024-09-18T08:06:38.686 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.006976192 2024-09-18T08:06:38.686 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:06:38.686 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:06:38.686 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:06:38.686 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:06:08.445222+0000", 2024-09-18T08:06:38.686 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 5.7447000000000001e-05 2024-09-18T08:06:38.687 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:06:38.687 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:06:38.687 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:06:38.687 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:06:08.500981+0000", 2024-09-18T08:06:38.687 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.055759727000000002 2024-09-18T08:06:38.687 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:06:38.687 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:06:38.687 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "commit_sent", 2024-09-18T08:06:38.687 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:06:08.501021+0000", 2024-09-18T08:06:38.687 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 3.9851000000000002e-05 2024-09-18T08:06:38.687 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:06:38.687 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:06:38.687 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-09-18T08:06:38.687 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:06:08.501062+0000", 2024-09-18T08:06:38.688 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4.0627999999999998e-05 2024-09-18T08:06:38.688 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-18T08:06:38.688 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-09-18T08:06:38.688 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-18T08:06:38.688 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:06:38.688 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:06:38.688 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.14805.0:151648 7.1 7:834e324e:::benchmark_data_smithi007_73155_object33357:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e127)", 2024-09-18T08:06:38.688 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-09-18T08:06:08.439698+0000", 2024-09-18T08:06:38.688 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 30.144615149, 2024-09-18T08:06:38.688 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.061456726000000003, 2024-09-18T08:06:38.688 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-09-18T08:06:38.688 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-09-18T08:06:38.688 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T08:06:38.689 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2024-09-18T08:06:38.689 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.14805", 2024-09-18T08:06:38.689 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "172.21.15.7:0/640320436", 2024-09-18T08:06:38.689 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 151648 2024-09-18T08:06:38.689 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:06:38.689 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-09-18T08:06:38.689 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:06:38.689 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-09-18T08:06:38.689 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:06:08.439698+0000", 2024-09-18T08:06:38.689 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-18T08:06:38.689 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:06:38.689 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:06:38.689 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-09-18T08:06:38.690 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:06:08.439698+0000", 2024-09-18T08:06:38.690 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-18T08:06:38.690 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:06:38.690 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:06:38.690 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-09-18T08:06:38.690 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:06:08.439699+0000", 2024-09-18T08:06:38.690 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.057e-06 2024-09-18T08:06:38.690 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:06:38.690 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:06:38.690 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-09-18T08:06:38.690 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:06:08.439707+0000", 2024-09-18T08:06:38.690 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 7.1450000000000002e-06 2024-09-18T08:06:38.690 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:06:38.690 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:06:38.690 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-09-18T08:06:38.691 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:06:08.439719+0000", 2024-09-18T08:06:38.691 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.1946e-05 2024-09-18T08:06:38.691 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:06:38.691 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:06:38.691 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-09-18T08:06:38.691 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:06:08.439725+0000", 2024-09-18T08:06:38.691 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 6.5540000000000003e-06 2024-09-18T08:06:38.691 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:06:38.691 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:06:38.691 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-18T08:06:38.691 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:06:08.439955+0000", 2024-09-18T08:06:38.691 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000229662 2024-09-18T08:06:38.691 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:06:38.691 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:06:38.692 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-09-18T08:06:38.692 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:06:08.440063+0000", 2024-09-18T08:06:38.692 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000108257 2024-09-18T08:06:38.692 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:06:38.692 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:06:38.692 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "waiting for subops from 3,6", 2024-09-18T08:06:38.692 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:06:08.440176+0000", 2024-09-18T08:06:38.692 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000113028 2024-09-18T08:06:38.692 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:06:38.692 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:06:38.692 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "op_commit", 2024-09-18T08:06:38.692 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:06:08.445175+0000", 2024-09-18T08:06:38.692 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0049987900000000004 2024-09-18T08:06:38.693 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:06:38.693 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:06:38.693 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:06:38.693 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:06:08.445415+0000", 2024-09-18T08:06:38.693 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00024047200000000001 2024-09-18T08:06:38.693 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:06:38.693 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:06:38.693 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:06:38.693 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:06:08.501099+0000", 2024-09-18T08:06:38.693 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.055683208999999997 2024-09-18T08:06:38.693 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:06:38.693 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:06:38.693 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "commit_sent", 2024-09-18T08:06:38.694 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:06:08.501127+0000", 2024-09-18T08:06:38.694 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 2.8503000000000001e-05 2024-09-18T08:06:38.694 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:06:38.694 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:06:38.694 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-09-18T08:06:38.694 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:06:08.501155+0000", 2024-09-18T08:06:38.694 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 2.8102999999999998e-05 2024-09-18T08:06:38.694 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-18T08:06:38.694 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-09-18T08:06:38.694 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-18T08:06:38.694 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:06:38.694 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:06:38.694 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.14805.0:151649 7.1 7:87d4917c:::benchmark_data_smithi007_73155_object33358:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e127)", 2024-09-18T08:06:38.694 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-09-18T08:06:08.439739+0000", 2024-09-18T08:06:38.695 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 30.144574962, 2024-09-18T08:06:38.695 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.062102242000000002, 2024-09-18T08:06:38.695 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-09-18T08:06:38.695 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-09-18T08:06:38.695 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T08:06:38.695 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2024-09-18T08:06:38.695 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.14805", 2024-09-18T08:06:38.695 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "172.21.15.7:0/640320436", 2024-09-18T08:06:38.695 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 151649 2024-09-18T08:06:38.695 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:06:38.695 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-09-18T08:06:38.695 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:06:38.695 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-09-18T08:06:38.695 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:06:08.439739+0000", 2024-09-18T08:06:38.696 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-18T08:06:38.696 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:06:38.696 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:06:38.696 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-09-18T08:06:38.696 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:06:08.439739+0000", 2024-09-18T08:06:38.696 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-18T08:06:38.696 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:06:38.696 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:06:38.696 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-09-18T08:06:38.696 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:06:08.439739+0000", 2024-09-18T08:06:38.696 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4.63e-07 2024-09-18T08:06:38.696 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:06:38.696 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:06:38.696 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-09-18T08:06:38.696 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:06:08.439741+0000", 2024-09-18T08:06:38.697 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 2.3360000000000002e-06 2024-09-18T08:06:38.697 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:06:38.697 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:06:38.697 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-09-18T08:06:38.697 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:06:08.439748+0000", 2024-09-18T08:06:38.697 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 6.6900000000000003e-06 2024-09-18T08:06:38.697 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:06:38.697 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:06:38.697 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-09-18T08:06:38.697 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:06:08.439753+0000", 2024-09-18T08:06:38.697 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4.5249999999999999e-06 2024-09-18T08:06:38.697 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:06:38.697 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:06:38.697 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-18T08:06:38.698 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:06:08.442205+0000", 2024-09-18T08:06:38.698 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.002451997 2024-09-18T08:06:38.698 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:06:38.698 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:06:38.698 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-09-18T08:06:38.698 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:06:08.442326+0000", 2024-09-18T08:06:38.698 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000121725 2024-09-18T08:06:38.698 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:06:38.698 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:06:38.698 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "waiting for subops from 3,6", 2024-09-18T08:06:38.698 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:06:08.442444+0000", 2024-09-18T08:06:38.698 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000117799 2024-09-18T08:06:38.698 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:06:38.698 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:06:38.698 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "op_commit", 2024-09-18T08:06:38.699 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:06:08.445294+0000", 2024-09-18T08:06:38.699 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.002849659 2024-09-18T08:06:38.699 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:06:38.699 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:06:38.699 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:06:38.699 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:06:08.445503+0000", 2024-09-18T08:06:38.699 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000209621 2024-09-18T08:06:38.699 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:06:38.699 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:06:38.699 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:06:38.699 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:06:08.501752+0000", 2024-09-18T08:06:38.699 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.056248116000000001 2024-09-18T08:06:38.699 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:06:38.699 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:06:38.700 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "commit_sent", 2024-09-18T08:06:38.700 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:06:08.501803+0000", 2024-09-18T08:06:38.700 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 5.1403000000000002e-05 2024-09-18T08:06:38.700 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:06:38.700 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:06:38.700 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-09-18T08:06:38.700 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:06:08.501841+0000", 2024-09-18T08:06:38.700 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 3.7907999999999997e-05 2024-09-18T08:06:38.700 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-18T08:06:38.700 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-09-18T08:06:38.700 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-18T08:06:38.700 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:06:38.700 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:06:38.700 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.14805.0:151660 7.1 7:845715b2:::benchmark_data_smithi007_73155_object33369:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e127)", 2024-09-18T08:06:38.701 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-09-18T08:06:08.449959+0000", 2024-09-18T08:06:38.701 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 30.134354906999999, 2024-09-18T08:06:38.701 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.052028337000000001, 2024-09-18T08:06:38.701 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-09-18T08:06:38.701 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-09-18T08:06:38.701 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T08:06:38.701 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2024-09-18T08:06:38.701 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.14805", 2024-09-18T08:06:38.701 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "172.21.15.7:0/640320436", 2024-09-18T08:06:38.701 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 151660 2024-09-18T08:06:38.701 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:06:38.701 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-09-18T08:06:38.701 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:06:38.701 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-09-18T08:06:38.702 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:06:08.449959+0000", 2024-09-18T08:06:38.702 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-18T08:06:38.702 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:06:38.702 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:06:38.702 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-09-18T08:06:38.702 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:06:08.449959+0000", 2024-09-18T08:06:38.702 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-18T08:06:38.702 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:06:38.702 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:06:38.702 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-09-18T08:06:38.702 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:06:08.449960+0000", 2024-09-18T08:06:38.702 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.606e-06 2024-09-18T08:06:38.702 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:06:38.702 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:06:38.703 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-09-18T08:06:38.703 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:06:08.449966+0000", 2024-09-18T08:06:38.703 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 6.139e-06 2024-09-18T08:06:38.703 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:06:38.703 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:06:38.703 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-09-18T08:06:38.703 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:06:08.449982+0000", 2024-09-18T08:06:38.703 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.5122e-05 2024-09-18T08:06:38.703 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:06:38.703 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:06:38.703 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-09-18T08:06:38.703 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:06:08.449990+0000", 2024-09-18T08:06:38.703 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 8.337e-06 2024-09-18T08:06:38.704 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:06:38.704 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:06:38.704 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-18T08:06:38.704 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:06:08.450052+0000", 2024-09-18T08:06:38.704 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 6.2224000000000006e-05 2024-09-18T08:06:38.704 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:06:38.704 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:06:38.704 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-09-18T08:06:38.704 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:06:08.450175+0000", 2024-09-18T08:06:38.704 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000123404 2024-09-18T08:06:38.704 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:06:38.704 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:06:38.704 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "waiting for subops from 3,6", 2024-09-18T08:06:38.705 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:06:08.450260+0000", 2024-09-18T08:06:38.705 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 8.4561999999999994e-05 2024-09-18T08:06:38.705 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:06:38.705 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:06:38.705 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "op_commit", 2024-09-18T08:06:38.705 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:06:08.452276+0000", 2024-09-18T08:06:38.705 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0020157550000000002 2024-09-18T08:06:38.705 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:06:38.705 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:06:38.705 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:06:38.705 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:06:08.452413+0000", 2024-09-18T08:06:38.705 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00013727600000000001 2024-09-18T08:06:38.705 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:06:38.706 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:06:38.706 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:06:38.706 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:06:08.501895+0000", 2024-09-18T08:06:38.706 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.049482330999999997 2024-09-18T08:06:38.706 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:06:38.706 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:06:38.706 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "commit_sent", 2024-09-18T08:06:38.706 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:06:08.501954+0000", 2024-09-18T08:06:38.706 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 5.8186000000000002e-05 2024-09-18T08:06:38.706 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:06:38.706 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:06:38.706 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-09-18T08:06:38.706 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:06:08.501987+0000", 2024-09-18T08:06:38.706 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 3.3395000000000001e-05 2024-09-18T08:06:38.707 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-18T08:06:38.707 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-09-18T08:06:38.707 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-18T08:06:38.707 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:06:38.707 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:06:38.707 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.14805.0:161175 7.1 7:8e9a4b6b:::benchmark_data_smithi007_73155_object42884:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e134)", 2024-09-18T08:06:38.707 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-09-18T08:06:21.016529+0000", 2024-09-18T08:06:38.707 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 17.567784397, 2024-09-18T08:06:38.707 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.060141834999999998, 2024-09-18T08:06:38.707 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-09-18T08:06:38.707 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-09-18T08:06:38.707 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T08:06:38.707 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2024-09-18T08:06:38.707 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.14805", 2024-09-18T08:06:38.708 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "172.21.15.7:0/640320436", 2024-09-18T08:06:38.708 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 161175 2024-09-18T08:06:38.708 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:06:38.708 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-09-18T08:06:38.708 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:06:38.708 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-09-18T08:06:38.708 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:06:21.016529+0000", 2024-09-18T08:06:38.708 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-18T08:06:38.708 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:06:38.708 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:06:38.708 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-09-18T08:06:38.708 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:06:21.016529+0000", 2024-09-18T08:06:38.708 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-18T08:06:38.709 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:06:38.709 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:06:38.709 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-09-18T08:06:38.709 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:06:21.016530+0000", 2024-09-18T08:06:38.709 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 8.9199999999999999e-07 2024-09-18T08:06:38.709 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:06:38.709 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:06:38.709 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-09-18T08:06:38.709 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:06:21.016534+0000", 2024-09-18T08:06:38.709 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4.2699999999999998e-06 2024-09-18T08:06:38.709 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:06:38.709 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:06:38.709 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-09-18T08:06:38.709 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:06:21.016543+0000", 2024-09-18T08:06:38.709 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 8.5629999999999992e-06 2024-09-18T08:06:38.710 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:06:38.710 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:06:38.710 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-09-18T08:06:38.710 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:06:21.016548+0000", 2024-09-18T08:06:38.710 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4.6530000000000002e-06 2024-09-18T08:06:38.710 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:06:38.710 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:06:38.710 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-18T08:06:38.710 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:06:21.016595+0000", 2024-09-18T08:06:38.710 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4.7082000000000001e-05 2024-09-18T08:06:38.710 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:06:38.710 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:06:38.710 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-09-18T08:06:38.710 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:06:21.016726+0000", 2024-09-18T08:06:38.710 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00013156999999999999 2024-09-18T08:06:38.711 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:06:38.711 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:06:38.711 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "waiting for subops from 1,3,6", 2024-09-18T08:06:38.711 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:06:21.016873+0000", 2024-09-18T08:06:38.711 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00014691899999999999 2024-09-18T08:06:38.711 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:06:38.711 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:06:38.711 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:06:38.711 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:06:21.019088+0000", 2024-09-18T08:06:38.711 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0022153099999999998 2024-09-18T08:06:38.711 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:06:38.711 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:06:38.711 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "op_commit", 2024-09-18T08:06:38.711 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:06:21.019167+0000", 2024-09-18T08:06:38.712 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 7.8498999999999995e-05 2024-09-18T08:06:38.712 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:06:38.712 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:06:38.712 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:06:38.712 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:06:21.068944+0000", 2024-09-18T08:06:38.712 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.049777228999999999 2024-09-18T08:06:38.712 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:06:38.712 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:06:38.712 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:06:38.712 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:06:21.076480+0000", 2024-09-18T08:06:38.712 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.007535623 2024-09-18T08:06:38.712 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:06:38.712 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:06:38.713 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "commit_sent", 2024-09-18T08:06:38.713 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:06:21.076556+0000", 2024-09-18T08:06:38.713 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 7.5822999999999996e-05 2024-09-18T08:06:38.713 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:06:38.713 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:06:38.713 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-09-18T08:06:38.713 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:06:21.076671+0000", 2024-09-18T08:06:38.713 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000115402 2024-09-18T08:06:38.713 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-18T08:06:38.714 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-09-18T08:06:38.714 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-18T08:06:38.714 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-18T08:06:38.714 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-09-18T08:06:38.714 INFO:teuthology.orchestra.run.smithi050.stdout:} 2024-09-18T08:06:38.714 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:06:38 smithi050 ceph-mon[32371]: pgmap v169: 148 pgs: 148 active+clean; 3.9 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 892 op/s; 384 KiB/s, 12 objects/s recovering 2024-09-18T08:06:38.754 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:06:38.755 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:06:38.755 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2' 2024-09-18T08:06:38.856 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4 2024-09-18T08:06:38.878 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:06:38.879 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:06:38.879 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4' 2024-09-18T08:06:38.980 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4 2024-09-18T08:06:39.009 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:06:39.009 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:06:39.010 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4' 2024-09-18T08:06:39.111 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0 2024-09-18T08:06:39.153 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:06:39.154 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:06:39.154 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0' 2024-09-18T08:06:39.155 INFO:journalctl@ceph.iscsi.iscsi.a.smithi050.stdout:Sep 18 08:06:38 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-iscsi-iscsi-a[57782]: debug there is no tcmu-runner data available 2024-09-18T08:06:39.254 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0 2024-09-18T08:06:39.296 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:06:39.296 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:06:39.296 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0' 2024-09-18T08:06:39.397 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1 2024-09-18T08:06:39.431 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:06:39.432 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:06:39.432 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1' 2024-09-18T08:06:39.533 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2 2024-09-18T08:06:39.569 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:06:39.569 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:06:39.570 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2' 2024-09-18T08:06:39.671 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6 2024-09-18T08:06:39.696 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:06:39.697 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:06:39.697 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6' 2024-09-18T08:06:39.798 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1 2024-09-18T08:06:39.817 DEBUG:teuthology.orchestra.run.smithi050:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid 779349ca-7593-11ef-bceb-c7b262605968 -- ceph daemon osd.5 dump_ops_in_flight 2024-09-18T08:06:39.837 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:06:39.838 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:06:39.838 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1' 2024-09-18T08:06:39.939 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2 2024-09-18T08:06:39.946 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid 779349ca-7593-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-18T08:06:39.983 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:06:39.984 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:06:39.984 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2' 2024-09-18T08:06:40.086 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6 2024-09-18T08:06:40.117 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:06:40.118 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:06:40.118 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6' 2024-09-18T08:06:40.152 INFO:teuthology.orchestra.run.smithi050.stderr:Inferring config /var/lib/ceph/779349ca-7593-11ef-bceb-c7b262605968/mon.b/config 2024-09-18T08:06:40.219 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5 2024-09-18T08:06:40.245 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:06:40.245 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:06:40.246 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5' 2024-09-18T08:06:40.347 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4 2024-09-18T08:06:40.376 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:06:40.377 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:06:40.378 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4' 2024-09-18T08:06:40.397 INFO:teuthology.orchestra.run.smithi007.stderr:noscrub is unset 2024-09-18T08:06:40.478 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0 2024-09-18T08:06:40.523 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:06:40.524 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:06:40.524 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0' 2024-09-18T08:06:40.625 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0 2024-09-18T08:06:40.641 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:06:40 smithi050 ceph-mon[32371]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-18T08:06:40.641 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:06:40 smithi050 ceph-mon[32371]: pgmap v170: 148 pgs: 148 active+clean; 3.6 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.07k op/s 2024-09-18T08:06:40.641 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:06:40 smithi050 ceph-mon[32371]: from='client.? 172.21.15.7:0/1360881604' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-18T08:06:40.641 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:06:40 smithi050 ceph-mon[32371]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-18T08:06:40.666 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:06:40.667 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:06:40.667 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0' 2024-09-18T08:06:40.730 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:06:40 smithi007 ceph-mon[39249]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-18T08:06:40.730 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:06:40 smithi007 ceph-mon[39249]: pgmap v170: 148 pgs: 148 active+clean; 3.6 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.07k op/s 2024-09-18T08:06:40.731 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:06:40 smithi007 ceph-mon[39249]: from='client.? 172.21.15.7:0/1360881604' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-18T08:06:40.731 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:06:40 smithi007 ceph-mon[39249]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-18T08:06:40.731 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:06:40 smithi007 ceph-mon[29459]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-18T08:06:40.731 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:06:40 smithi007 ceph-mon[29459]: pgmap v170: 148 pgs: 148 active+clean; 3.6 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.07k op/s 2024-09-18T08:06:40.731 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:06:40 smithi007 ceph-mon[29459]: from='client.? 172.21.15.7:0/1360881604' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-18T08:06:40.731 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:06:40 smithi007 ceph-mon[29459]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-18T08:06:40.769 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3 2024-09-18T08:06:40.804 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:06:40.804 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:06:40.804 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3' 2024-09-18T08:06:40.905 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2 2024-09-18T08:06:40.926 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2024-09-18T08:06:40.926 INFO:teuthology.orchestra.run.smithi050.stdout: "ops": [], 2024-09-18T08:06:40.926 INFO:teuthology.orchestra.run.smithi050.stdout: "num_ops": 0 2024-09-18T08:06:40.926 INFO:teuthology.orchestra.run.smithi050.stdout:} 2024-09-18T08:06:40.941 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:06:40.942 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:06:40.943 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2' 2024-09-18T08:06:41.043 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0 2024-09-18T08:06:41.081 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:06:41.082 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:06:41.082 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0' 2024-09-18T08:06:41.183 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6 2024-09-18T08:06:41.214 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:06:41.214 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:06:41.215 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6' 2024-09-18T08:06:41.315 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0 2024-09-18T08:06:41.359 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:06:41.360 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:06:41.360 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0' 2024-09-18T08:06:41.461 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6 2024-09-18T08:06:41.491 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:06:41.492 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:06:41.492 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6' 2024-09-18T08:06:41.593 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6 2024-09-18T08:06:41.621 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:06:41.622 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:06:41.622 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6' 2024-09-18T08:06:41.723 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0 2024-09-18T08:06:41.730 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:06:41 smithi007 ceph-mon[39249]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-18T08:06:41.731 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:06:41 smithi007 ceph-mon[39249]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-18T08:06:41.731 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:06:41 smithi007 ceph-mon[39249]: osdmap e140: 8 total, 8 up, 8 in 2024-09-18T08:06:41.731 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:06:41 smithi007 ceph-mon[39249]: purged_snaps scrub starts 2024-09-18T08:06:41.731 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:06:41 smithi007 ceph-mon[39249]: purged_snaps scrub ok 2024-09-18T08:06:41.731 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:06:41 smithi007 ceph-mon[29459]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-18T08:06:41.731 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:06:41 smithi007 ceph-mon[29459]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-18T08:06:41.731 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:06:41 smithi007 ceph-mon[29459]: osdmap e140: 8 total, 8 up, 8 in 2024-09-18T08:06:41.731 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:06:41 smithi007 ceph-mon[29459]: purged_snaps scrub starts 2024-09-18T08:06:41.731 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:06:41 smithi007 ceph-mon[29459]: purged_snaps scrub ok 2024-09-18T08:06:41.763 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:06:41.764 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:06:41.764 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0' 2024-09-18T08:06:41.865 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4 2024-09-18T08:06:41.894 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:06:41.894 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:06:41.895 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4' 2024-09-18T08:06:41.905 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:06:41 smithi050 ceph-mon[32371]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-18T08:06:41.906 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:06:41 smithi050 ceph-mon[32371]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-18T08:06:41.906 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:06:41 smithi050 ceph-mon[32371]: osdmap e140: 8 total, 8 up, 8 in 2024-09-18T08:06:41.906 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:06:41 smithi050 ceph-mon[32371]: purged_snaps scrub starts 2024-09-18T08:06:41.906 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:06:41 smithi050 ceph-mon[32371]: purged_snaps scrub ok 2024-09-18T08:06:41.960 DEBUG:teuthology.orchestra.run.smithi050:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid 779349ca-7593-11ef-bceb-c7b262605968 -- ceph daemon osd.5 dump_blocked_ops 2024-09-18T08:06:41.980 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/779349ca-7593-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T08:06:41.995 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2 2024-09-18T08:06:42.058 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:06:42.058 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:06:42.058 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2' 2024-09-18T08:06:42.133 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 7, 6] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-09-18T08:06:42.133 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-18T08:06:42.134 INFO:tasks.thrashosds.thrasher:primary_affinity 2024-09-18T08:06:42.134 INFO:tasks.thrashosds.thrasher:Setting osd 7 primary_affinity to 0.730265 2024-09-18T08:06:42.134 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid 779349ca-7593-11ef-bceb-c7b262605968 -- ceph osd primary-affinity 7 0.7302645555563375 2024-09-18T08:06:42.159 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1 2024-09-18T08:06:42.237 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:06:42.237 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:06:42.238 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1' 2024-09-18T08:06:42.330 INFO:teuthology.orchestra.run.smithi050.stderr:Inferring config /var/lib/ceph/779349ca-7593-11ef-bceb-c7b262605968/mon.b/config 2024-09-18T08:06:42.339 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5 2024-09-18T08:06:42.369 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:06:42.370 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:06:42.370 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5' 2024-09-18T08:06:42.471 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7 2024-09-18T08:06:42.498 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:06:42.499 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:06:42.499 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7' 2024-09-18T08:06:42.600 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3 2024-09-18T08:06:42.659 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:06:42.660 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:06:42.660 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3' 2024-09-18T08:06:42.667 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:06:42 smithi050 ceph-mon[32371]: pgmap v172: 148 pgs: 148 active+clean; 3.6 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 1.21k op/s 2024-09-18T08:06:42.667 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:06:42 smithi050 ceph-mon[32371]: osdmap e141: 8 total, 8 up, 8 in 2024-09-18T08:06:42.667 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:06:42 smithi050 ceph-mon[32371]: 7.b scrub starts 2024-09-18T08:06:42.685 INFO:teuthology.orchestra.run.smithi007.stdout:osd.0: {} 2024-09-18T08:06:42.685 INFO:teuthology.orchestra.run.smithi007.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-18T08:06:42.686 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:06:42 smithi007 ceph-mon[39249]: pgmap v172: 148 pgs: 148 active+clean; 3.6 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 1.21k op/s 2024-09-18T08:06:42.686 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:06:42 smithi007 ceph-mon[39249]: osdmap e141: 8 total, 8 up, 8 in 2024-09-18T08:06:42.686 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:06:42 smithi007 ceph-mon[39249]: 7.b scrub starts 2024-09-18T08:06:42.686 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:06:42 smithi007 ceph-mon[29459]: pgmap v172: 148 pgs: 148 active+clean; 3.6 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 1.21k op/s 2024-09-18T08:06:42.686 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:06:42 smithi007 ceph-mon[29459]: osdmap e141: 8 total, 8 up, 8 in 2024-09-18T08:06:42.687 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:06:42 smithi007 ceph-mon[29459]: 7.b scrub starts 2024-09-18T08:06:42.726 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/779349ca-7593-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T08:06:42.726 INFO:teuthology.orchestra.run.smithi007.stdout:osd.1: {} 2024-09-18T08:06:42.727 INFO:teuthology.orchestra.run.smithi007.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-18T08:06:42.760 INFO:teuthology.orchestra.run.smithi007.stdout:osd.2: {} 2024-09-18T08:06:42.761 INFO:teuthology.orchestra.run.smithi007.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-18T08:06:42.761 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0 2024-09-18T08:06:42.797 INFO:teuthology.orchestra.run.smithi007.stdout:osd.3: {} 2024-09-18T08:06:42.797 INFO:teuthology.orchestra.run.smithi007.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-18T08:06:42.820 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:06:42.821 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:06:42.821 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0' 2024-09-18T08:06:42.836 INFO:teuthology.orchestra.run.smithi007.stdout:osd.4: {} 2024-09-18T08:06:42.836 INFO:teuthology.orchestra.run.smithi007.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-18T08:06:42.871 INFO:teuthology.orchestra.run.smithi007.stdout:osd.5: {} 2024-09-18T08:06:42.872 INFO:teuthology.orchestra.run.smithi007.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-18T08:06:42.905 INFO:teuthology.orchestra.run.smithi007.stdout:osd.6: {} 2024-09-18T08:06:42.906 INFO:teuthology.orchestra.run.smithi007.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-18T08:06:42.922 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1 2024-09-18T08:06:42.946 INFO:teuthology.orchestra.run.smithi007.stdout:osd.7: {} 2024-09-18T08:06:42.947 INFO:teuthology.orchestra.run.smithi007.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-18T08:06:42.971 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:06:42.971 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:06:42.972 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1' 2024-09-18T08:06:43.072 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6 2024-09-18T08:06:43.102 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:06:43.103 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:06:43.103 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6' 2024-09-18T08:06:43.204 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6 2024-09-18T08:06:43.234 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:06:43.235 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:06:43.235 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6' 2024-09-18T08:06:43.298 INFO:teuthology.orchestra.run.smithi050.stdout:ERROR: (22) Invalid argument 2024-09-18T08:06:43.299 INFO:teuthology.orchestra.run.smithi050.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-18T08:06:43.336 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6 2024-09-18T08:06:43.364 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:06:43.365 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:06:43.365 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6' 2024-09-18T08:06:43.466 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7 2024-09-18T08:06:43.496 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:06:43.496 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:06:43.497 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7' 2024-09-18T08:06:43.500 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:06:43 smithi007 ceph-mon[29459]: 7.8 scrub starts 2024-09-18T08:06:43.500 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:06:43 smithi007 ceph-mon[29459]: purged_snaps scrub starts 2024-09-18T08:06:43.500 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:06:43 smithi007 ceph-mon[29459]: purged_snaps scrub ok 2024-09-18T08:06:43.500 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:06:43 smithi007 ceph-mon[29459]: osdmap e142: 8 total, 8 up, 8 in 2024-09-18T08:06:43.500 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:06:43 smithi007 ceph-mon[29459]: 7.b scrub ok 2024-09-18T08:06:43.500 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:06:43 smithi007 ceph-mon[29459]: from='client.? 172.21.15.7:0/3529624003' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-18T08:06:43.501 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:06:43 smithi007 ceph-mon[39249]: 7.8 scrub starts 2024-09-18T08:06:43.501 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:06:43 smithi007 ceph-mon[39249]: purged_snaps scrub starts 2024-09-18T08:06:43.501 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:06:43 smithi007 ceph-mon[39249]: purged_snaps scrub ok 2024-09-18T08:06:43.501 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:06:43 smithi007 ceph-mon[39249]: osdmap e142: 8 total, 8 up, 8 in 2024-09-18T08:06:43.501 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:06:43 smithi007 ceph-mon[39249]: 7.b scrub ok 2024-09-18T08:06:43.501 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:06:43 smithi007 ceph-mon[39249]: from='client.? 172.21.15.7:0/3529624003' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-18T08:06:43.598 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0 2024-09-18T08:06:43.637 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:06:43.638 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:06:43.638 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0' 2024-09-18T08:06:43.739 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3 2024-09-18T08:06:43.782 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:06:43.783 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:06:43.783 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3' 2024-09-18T08:06:43.884 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6 2024-09-18T08:06:43.906 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:06:43 smithi050 ceph-mon[32371]: 7.8 scrub starts 2024-09-18T08:06:43.906 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:06:43 smithi050 ceph-mon[32371]: purged_snaps scrub starts 2024-09-18T08:06:43.906 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:06:43 smithi050 ceph-mon[32371]: purged_snaps scrub ok 2024-09-18T08:06:43.906 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:06:43 smithi050 ceph-mon[32371]: osdmap e142: 8 total, 8 up, 8 in 2024-09-18T08:06:43.906 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:06:43 smithi050 ceph-mon[32371]: 7.b scrub ok 2024-09-18T08:06:43.906 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:06:43 smithi050 ceph-mon[32371]: from='client.? 172.21.15.7:0/3529624003' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-18T08:06:43.912 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:06:43.913 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:06:43.914 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6' 2024-09-18T08:06:43.939 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid 779349ca-7593-11ef-bceb-c7b262605968 -- ceph osd unset nodeep-scrub 2024-09-18T08:06:44.014 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0 2024-09-18T08:06:44.066 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:06:44.066 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:06:44.066 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0' 2024-09-18T08:06:44.108 DEBUG:teuthology.orchestra.run.smithi050:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid 779349ca-7593-11ef-bceb-c7b262605968 -- ceph daemon osd.5 dump_historic_ops 2024-09-18T08:06:44.167 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4 2024-09-18T08:06:44.192 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:06:44.193 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:06:44.193 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4' 2024-09-18T08:06:44.294 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3 2024-09-18T08:06:44.340 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:06:44.341 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:06:44.341 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3' 2024-09-18T08:06:44.411 INFO:teuthology.orchestra.run.smithi050.stderr:Inferring config /var/lib/ceph/779349ca-7593-11ef-bceb-c7b262605968/mon.b/config 2024-09-18T08:06:44.442 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1 2024-09-18T08:06:44.467 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/779349ca-7593-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T08:06:44.501 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:06:44.501 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:06:44.502 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1' 2024-09-18T08:06:44.603 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4 2024-09-18T08:06:44.629 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:06:44.630 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:06:44.630 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4' 2024-09-18T08:06:44.720 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:06:44 smithi050 ceph-mon[32371]: 7.9 scrub starts 2024-09-18T08:06:44.721 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:06:44 smithi050 ceph-mon[32371]: pgmap v175: 148 pgs: 148 active+clean; 3.6 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 1.7 KiB/s rd, 0 B/s wr, 1.04k op/s 2024-09-18T08:06:44.721 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:06:44 smithi050 ceph-mon[32371]: 7.8 scrub ok 2024-09-18T08:06:44.721 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:06:44 smithi050 ceph-mon[32371]: 6.17 scrub starts 2024-09-18T08:06:44.721 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:06:44 smithi050 ceph-mon[32371]: 6.17 scrub ok 2024-09-18T08:06:44.721 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:06:44 smithi050 ceph-mon[32371]: 4.0 scrub starts 2024-09-18T08:06:44.721 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:06:44 smithi050 ceph-mon[32371]: 4.0 scrub ok 2024-09-18T08:06:44.721 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:06:44 smithi050 ceph-mon[32371]: from='client.? 172.21.15.7:0/1448737084' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 7, "weight": 0.7302645555563375} : dispatch 2024-09-18T08:06:44.721 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:06:44 smithi050 ceph-mon[32371]: from='client.? ' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 7, "weight": 0.7302645555563375} : dispatch 2024-09-18T08:06:44.730 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:06:44 smithi007 ceph-mon[39249]: 7.9 scrub starts 2024-09-18T08:06:44.730 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:06:44 smithi007 ceph-mon[39249]: pgmap v175: 148 pgs: 148 active+clean; 3.6 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 1.7 KiB/s rd, 0 B/s wr, 1.04k op/s 2024-09-18T08:06:44.731 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:06:44 smithi007 ceph-mon[39249]: 7.8 scrub ok 2024-09-18T08:06:44.731 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:06:44 smithi007 ceph-mon[39249]: 6.17 scrub starts 2024-09-18T08:06:44.731 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:06:44 smithi007 ceph-mon[39249]: 6.17 scrub ok 2024-09-18T08:06:44.731 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:06:44 smithi007 ceph-mon[39249]: 4.0 scrub starts 2024-09-18T08:06:44.731 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:06:44 smithi007 ceph-mon[39249]: 4.0 scrub ok 2024-09-18T08:06:44.731 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:06:44 smithi007 ceph-mon[39249]: from='client.? 172.21.15.7:0/1448737084' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 7, "weight": 0.7302645555563375} : dispatch 2024-09-18T08:06:44.731 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:06:44 smithi007 ceph-mon[39249]: from='client.? ' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 7, "weight": 0.7302645555563375} : dispatch 2024-09-18T08:06:44.731 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:06:44 smithi007 ceph-mon[29459]: 7.9 scrub starts 2024-09-18T08:06:44.731 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:06:44 smithi007 ceph-mon[29459]: pgmap v175: 148 pgs: 148 active+clean; 3.6 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 1.7 KiB/s rd, 0 B/s wr, 1.04k op/s 2024-09-18T08:06:44.731 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:06:44 smithi007 ceph-mon[29459]: 7.8 scrub ok 2024-09-18T08:06:44.731 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:06:44 smithi007 ceph-mon[29459]: 6.17 scrub starts 2024-09-18T08:06:44.732 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:06:44 smithi007 ceph-mon[29459]: 6.17 scrub ok 2024-09-18T08:06:44.732 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:06:44 smithi007 ceph-mon[29459]: 4.0 scrub starts 2024-09-18T08:06:44.732 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:06:44 smithi007 ceph-mon[29459]: 4.0 scrub ok 2024-09-18T08:06:44.732 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:06:44 smithi007 ceph-mon[29459]: from='client.? 172.21.15.7:0/1448737084' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 7, "weight": 0.7302645555563375} : dispatch 2024-09-18T08:06:44.732 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:06:44 smithi007 ceph-mon[29459]: from='client.? ' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 7, "weight": 0.7302645555563375} : dispatch 2024-09-18T08:06:44.732 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5 2024-09-18T08:06:44.771 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:06:44.772 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:06:44.772 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5' 2024-09-18T08:06:44.873 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7 2024-09-18T08:06:44.906 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:06:44.906 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:06:44.906 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7' 2024-09-18T08:06:45.003 INFO:teuthology.orchestra.run.smithi050.stdout:ERROR: (22) Invalid argument 2024-09-18T08:06:45.003 INFO:teuthology.orchestra.run.smithi050.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-18T08:06:45.007 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6 2024-09-18T08:06:45.041 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:06:45.042 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:06:45.043 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6' 2024-09-18T08:06:45.143 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4 2024-09-18T08:06:45.144 INFO:teuthology.orchestra.run.smithi007.stderr:set osd.7 primary-affinity to 0.730265 (8478582) 2024-09-18T08:06:45.167 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:06:45.168 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:06:45.168 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4' 2024-09-18T08:06:45.269 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2 2024-09-18T08:06:45.306 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:06:45.307 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:06:45.307 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2' 2024-09-18T08:06:45.408 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5 2024-09-18T08:06:45.431 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:06:45.431 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:06:45.432 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5' 2024-09-18T08:06:45.444 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:06:45 smithi007 ceph-mon[39249]: 3.6 scrub starts 2024-09-18T08:06:45.444 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:06:45 smithi007 ceph-mon[39249]: 3.6 scrub ok 2024-09-18T08:06:45.444 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:06:45 smithi007 ceph-mon[39249]: 6.15 scrub starts 2024-09-18T08:06:45.444 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:06:45 smithi007 ceph-mon[39249]: 6.15 scrub ok 2024-09-18T08:06:45.444 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:06:45 smithi007 ceph-mon[39249]: 7.9 scrub ok 2024-09-18T08:06:45.533 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6 2024-09-18T08:06:45.555 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:06:45.556 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:06:45.556 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6' 2024-09-18T08:06:45.657 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2 2024-09-18T08:06:45.678 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid 779349ca-7593-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-18T08:06:45.679 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:06:45.679 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:06:45.680 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2' 2024-09-18T08:06:45.694 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:06:45 smithi050 ceph-mon[32371]: 3.6 scrub starts 2024-09-18T08:06:45.694 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:06:45 smithi050 ceph-mon[32371]: 3.6 scrub ok 2024-09-18T08:06:45.694 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:06:45 smithi050 ceph-mon[32371]: 6.15 scrub starts 2024-09-18T08:06:45.694 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:06:45 smithi050 ceph-mon[32371]: 6.15 scrub ok 2024-09-18T08:06:45.694 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:06:45 smithi050 ceph-mon[32371]: 7.9 scrub ok 2024-09-18T08:06:45.694 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:06:45 smithi050 ceph-mon[32371]: 6.19 scrub starts 2024-09-18T08:06:45.695 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:06:45 smithi050 ceph-mon[32371]: 6.19 scrub ok 2024-09-18T08:06:45.695 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:06:45 smithi050 ceph-mon[32371]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 7, "weight": 0.7302645555563375}]': finished 2024-09-18T08:06:45.695 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:06:45 smithi050 ceph-mon[32371]: osdmap e143: 8 total, 8 up, 8 in 2024-09-18T08:06:45.700 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:06:45 smithi007 ceph-mon[39249]: 6.19 scrub starts 2024-09-18T08:06:45.700 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:06:45 smithi007 ceph-mon[39249]: 6.19 scrub ok 2024-09-18T08:06:45.700 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:06:45 smithi007 ceph-mon[39249]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 7, "weight": 0.7302645555563375}]': finished 2024-09-18T08:06:45.701 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:06:45 smithi007 ceph-mon[39249]: osdmap e143: 8 total, 8 up, 8 in 2024-09-18T08:06:45.701 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:06:45 smithi007 ceph-mon[29459]: 3.6 scrub starts 2024-09-18T08:06:45.701 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:06:45 smithi007 ceph-mon[29459]: 3.6 scrub ok 2024-09-18T08:06:45.701 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:06:45 smithi007 ceph-mon[29459]: 6.15 scrub starts 2024-09-18T08:06:45.701 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:06:45 smithi007 ceph-mon[29459]: 6.15 scrub ok 2024-09-18T08:06:45.701 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:06:45 smithi007 ceph-mon[29459]: 7.9 scrub ok 2024-09-18T08:06:45.701 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:06:45 smithi007 ceph-mon[29459]: 6.19 scrub starts 2024-09-18T08:06:45.701 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:06:45 smithi007 ceph-mon[29459]: 6.19 scrub ok 2024-09-18T08:06:45.701 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:06:45 smithi007 ceph-mon[29459]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 7, "weight": 0.7302645555563375}]': finished 2024-09-18T08:06:45.701 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:06:45 smithi007 ceph-mon[29459]: osdmap e143: 8 total, 8 up, 8 in 2024-09-18T08:06:45.721 DEBUG:teuthology.orchestra.run.smithi050:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid 779349ca-7593-11ef-bceb-c7b262605968 -- ceph daemon osd.6 dump_ops_in_flight 2024-09-18T08:06:45.781 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4 2024-09-18T08:06:45.803 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:06:45.804 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:06:45.804 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4' 2024-09-18T08:06:45.907 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7 2024-09-18T08:06:45.931 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:06:45.932 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:06:45.932 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7' 2024-09-18T08:06:45.939 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/779349ca-7593-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T08:06:45.977 INFO:teuthology.orchestra.run.smithi050.stderr:Inferring config /var/lib/ceph/779349ca-7593-11ef-bceb-c7b262605968/mon.b/config 2024-09-18T08:06:46.033 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4 2024-09-18T08:06:46.055 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:06:46.056 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:06:46.056 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4' 2024-09-18T08:06:46.134 INFO:teuthology.orchestra.run.smithi007.stderr:nodeep-scrub is unset 2024-09-18T08:06:46.157 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0 2024-09-18T08:06:46.187 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:06:46.188 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:06:46.189 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0' 2024-09-18T08:06:46.289 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4 2024-09-18T08:06:46.327 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:06:46.328 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:06:46.328 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4' 2024-09-18T08:06:46.429 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3 2024-09-18T08:06:46.470 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:06:46.470 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:06:46.471 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3' 2024-09-18T08:06:46.571 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5 2024-09-18T08:06:46.597 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:06:46.597 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:06:46.598 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5' 2024-09-18T08:06:46.697 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:06:46 smithi007 ceph-mon[29459]: 6.3 scrub starts 2024-09-18T08:06:46.698 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:06:46 smithi007 ceph-mon[29459]: 6.3 scrub ok 2024-09-18T08:06:46.698 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:06:46 smithi007 ceph-mon[29459]: pgmap v177: 148 pgs: 2 active+clean+scrubbing, 146 active+clean; 3.3 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 1.0 KiB/s rd, 0 B/s wr, 1.04k op/s 2024-09-18T08:06:46.698 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:06:46 smithi007 ceph-mon[29459]: from='client.? 172.21.15.7:0/1818440908' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-18T08:06:46.698 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:06:46 smithi007 ceph-mon[29459]: 5.15 scrub starts 2024-09-18T08:06:46.698 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:06:46 smithi007 ceph-mon[29459]: 5.15 scrub ok 2024-09-18T08:06:46.698 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:06:46 smithi007 ceph-mon[29459]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-18T08:06:46.698 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:06:46 smithi007 ceph-mon[29459]: Cluster is now healthy 2024-09-18T08:06:46.698 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:06:46 smithi007 ceph-mon[29459]: from='client.? 172.21.15.7:0/1818440908' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-18T08:06:46.698 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:06:46 smithi007 ceph-mon[29459]: osdmap e144: 8 total, 8 up, 8 in 2024-09-18T08:06:46.699 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:06:46 smithi007 ceph-mon[39249]: 6.3 scrub starts 2024-09-18T08:06:46.699 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:06:46 smithi007 ceph-mon[39249]: 6.3 scrub ok 2024-09-18T08:06:46.699 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:06:46 smithi007 ceph-mon[39249]: pgmap v177: 148 pgs: 2 active+clean+scrubbing, 146 active+clean; 3.3 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 1.0 KiB/s rd, 0 B/s wr, 1.04k op/s 2024-09-18T08:06:46.699 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:06:46 smithi007 ceph-mon[39249]: from='client.? 172.21.15.7:0/1818440908' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-18T08:06:46.699 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:06:46 smithi007 ceph-mon[39249]: 5.15 scrub starts 2024-09-18T08:06:46.699 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:06:46 smithi007 ceph-mon[39249]: 5.15 scrub ok 2024-09-18T08:06:46.699 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:06:46 smithi007 ceph-mon[39249]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-18T08:06:46.699 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:06:46 smithi007 ceph-mon[39249]: Cluster is now healthy 2024-09-18T08:06:46.699 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:06:46 smithi007 ceph-mon[39249]: from='client.? 172.21.15.7:0/1818440908' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-18T08:06:46.699 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:06:46 smithi007 ceph-mon[39249]: osdmap e144: 8 total, 8 up, 8 in 2024-09-18T08:06:46.700 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3 2024-09-18T08:06:46.708 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:06:46 smithi050 ceph-mon[32371]: 6.3 scrub starts 2024-09-18T08:06:46.708 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:06:46 smithi050 ceph-mon[32371]: 6.3 scrub ok 2024-09-18T08:06:46.708 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:06:46 smithi050 ceph-mon[32371]: pgmap v177: 148 pgs: 2 active+clean+scrubbing, 146 active+clean; 3.3 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 1.0 KiB/s rd, 0 B/s wr, 1.04k op/s 2024-09-18T08:06:46.708 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:06:46 smithi050 ceph-mon[32371]: from='client.? 172.21.15.7:0/1818440908' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-18T08:06:46.708 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:06:46 smithi050 ceph-mon[32371]: 5.15 scrub starts 2024-09-18T08:06:46.709 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:06:46 smithi050 ceph-mon[32371]: 5.15 scrub ok 2024-09-18T08:06:46.709 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:06:46 smithi050 ceph-mon[32371]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-18T08:06:46.709 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:06:46 smithi050 ceph-mon[32371]: Cluster is now healthy 2024-09-18T08:06:46.709 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:06:46 smithi050 ceph-mon[32371]: from='client.? 172.21.15.7:0/1818440908' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-18T08:06:46.709 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:06:46 smithi050 ceph-mon[32371]: osdmap e144: 8 total, 8 up, 8 in 2024-09-18T08:06:46.721 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:06:46.722 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:06:46.722 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3' 2024-09-18T08:06:46.727 INFO:teuthology.orchestra.run.smithi050.stdout:ERROR: (22) Invalid argument 2024-09-18T08:06:46.727 INFO:teuthology.orchestra.run.smithi050.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-18T08:06:46.823 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3 2024-09-18T08:06:46.864 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:06:46.865 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:06:46.866 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3' 2024-09-18T08:06:46.966 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6 2024-09-18T08:06:46.980 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Sep 18 08:06:46 smithi007 ceph-779349ca-7593-11ef-bceb-c7b262605968-mgr-y[29688]: ::ffff:172.21.15.50 - - [18/Sep/2024:08:06:46] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-09-18T08:06:46.994 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:06:46.994 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:06:46.995 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6' 2024-09-18T08:06:47.095 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2 2024-09-18T08:06:47.125 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:06:47.129 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:06:47.130 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2' 2024-09-18T08:06:47.231 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3 2024-09-18T08:06:47.267 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:06:47.268 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:06:47.268 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3' 2024-09-18T08:06:47.369 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1 2024-09-18T08:06:47.410 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:06:47.411 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:06:47.411 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1' 2024-09-18T08:06:47.479 DEBUG:teuthology.orchestra.run.smithi050:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid 779349ca-7593-11ef-bceb-c7b262605968 -- ceph daemon osd.6 dump_blocked_ops 2024-09-18T08:06:47.512 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2 2024-09-18T08:06:47.552 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:06:47.553 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:06:47.554 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2' 2024-09-18T08:06:47.654 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6 2024-09-18T08:06:47.688 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:06:47.688 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:06:47.689 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6' 2024-09-18T08:06:47.730 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:06:47 smithi007 ceph-mon[29459]: 3.b scrub starts 2024-09-18T08:06:47.730 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:06:47 smithi007 ceph-mon[29459]: 3.b scrub ok 2024-09-18T08:06:47.731 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:06:47 smithi007 ceph-mon[29459]: 3.14 scrub starts 2024-09-18T08:06:47.731 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:06:47 smithi007 ceph-mon[29459]: 3.14 scrub ok 2024-09-18T08:06:47.731 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:06:47 smithi007 ceph-mon[39249]: 3.b scrub starts 2024-09-18T08:06:47.731 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:06:47 smithi007 ceph-mon[39249]: 3.b scrub ok 2024-09-18T08:06:47.731 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:06:47 smithi007 ceph-mon[39249]: 3.14 scrub starts 2024-09-18T08:06:47.731 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:06:47 smithi007 ceph-mon[39249]: 3.14 scrub ok 2024-09-18T08:06:47.790 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6 2024-09-18T08:06:47.824 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:06:47.825 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:06:47.825 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6' 2024-09-18T08:06:47.830 INFO:teuthology.orchestra.run.smithi050.stderr:Inferring config /var/lib/ceph/779349ca-7593-11ef-bceb-c7b262605968/mon.b/config 2024-09-18T08:06:47.906 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:06:47 smithi050 ceph-mon[32371]: 3.b scrub starts 2024-09-18T08:06:47.906 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:06:47 smithi050 ceph-mon[32371]: 3.b scrub ok 2024-09-18T08:06:47.906 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:06:47 smithi050 ceph-mon[32371]: 3.14 scrub starts 2024-09-18T08:06:47.906 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:06:47 smithi050 ceph-mon[32371]: 3.14 scrub ok 2024-09-18T08:06:47.926 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6 2024-09-18T08:06:47.955 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:06:47.956 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:06:47.956 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6' 2024-09-18T08:06:48.006 INFO:teuthology.orchestra.run.smithi007.stdout:osd.0: {} 2024-09-18T08:06:48.006 INFO:teuthology.orchestra.run.smithi007.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-18T08:06:48.044 INFO:teuthology.orchestra.run.smithi007.stdout:osd.1: {} 2024-09-18T08:06:48.044 INFO:teuthology.orchestra.run.smithi007.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-18T08:06:48.057 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0 2024-09-18T08:06:48.081 INFO:teuthology.orchestra.run.smithi007.stdout:osd.2: {} 2024-09-18T08:06:48.081 INFO:teuthology.orchestra.run.smithi007.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-18T08:06:48.097 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:06:48.098 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:06:48.098 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0' 2024-09-18T08:06:48.122 INFO:teuthology.orchestra.run.smithi007.stdout:osd.3: {} 2024-09-18T08:06:48.122 INFO:teuthology.orchestra.run.smithi007.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-18T08:06:48.184 INFO:teuthology.orchestra.run.smithi007.stdout:osd.4: {} 2024-09-18T08:06:48.184 INFO:teuthology.orchestra.run.smithi007.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-18T08:06:48.200 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4 2024-09-18T08:06:48.216 INFO:teuthology.orchestra.run.smithi007.stdout:osd.5: {} 2024-09-18T08:06:48.216 INFO:teuthology.orchestra.run.smithi007.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-18T08:06:48.225 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:06:48.226 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:06:48.226 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4' 2024-09-18T08:06:48.252 INFO:teuthology.orchestra.run.smithi007.stdout:osd.6: {} 2024-09-18T08:06:48.252 INFO:teuthology.orchestra.run.smithi007.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-18T08:06:48.287 INFO:teuthology.orchestra.run.smithi007.stdout:osd.7: {} 2024-09-18T08:06:48.287 INFO:teuthology.orchestra.run.smithi007.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-18T08:06:48.326 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1 2024-09-18T08:06:48.363 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2024-09-18T08:06:48.363 INFO:teuthology.orchestra.run.smithi050.stdout: "ops": [], 2024-09-18T08:06:48.363 INFO:teuthology.orchestra.run.smithi050.stdout: "complaint_time": 30, 2024-09-18T08:06:48.364 INFO:teuthology.orchestra.run.smithi050.stdout: "num_blocked_ops": 0 2024-09-18T08:06:48.364 INFO:teuthology.orchestra.run.smithi050.stdout:} 2024-09-18T08:06:48.368 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:06:48.369 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:06:48.369 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1' 2024-09-18T08:06:48.470 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7 2024-09-18T08:06:48.497 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:06:48.497 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:06:48.498 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7' 2024-09-18T08:06:48.599 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1 2024-09-18T08:06:48.635 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:06:48.636 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:06:48.636 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1' 2024-09-18T08:06:48.656 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:06:48 smithi050 ceph-mon[32371]: 3.13 scrub starts 2024-09-18T08:06:48.656 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:06:48 smithi050 ceph-mon[32371]: 3.13 scrub ok 2024-09-18T08:06:48.656 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:06:48 smithi050 ceph-mon[32371]: 7.5 scrub starts 2024-09-18T08:06:48.656 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:06:48 smithi050 ceph-mon[32371]: pgmap v179: 148 pgs: 2 active+clean+scrubbing, 146 active+clean; 3.3 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 892 B/s rd, 0 B/s wr, 865 op/s 2024-09-18T08:06:48.656 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:06:48 smithi050 ceph-mon[32371]: 3.d scrub starts 2024-09-18T08:06:48.656 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:06:48 smithi050 ceph-mon[32371]: 3.d scrub ok 2024-09-18T08:06:48.656 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:06:48 smithi050 ceph-mon[32371]: from='mgr.24470 172.21.15.7:0/2594390261' entity='mgr.y' 2024-09-18T08:06:48.656 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:06:48 smithi050 ceph-mon[32371]: from='mgr.24470 172.21.15.7:0/2594390261' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-18T08:06:48.656 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:06:48 smithi050 ceph-mon[32371]: from='client.? 172.21.15.7:0/3906338375' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-18T08:06:48.730 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:06:48 smithi007 ceph-mon[39249]: 3.13 scrub starts 2024-09-18T08:06:48.731 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:06:48 smithi007 ceph-mon[39249]: 3.13 scrub ok 2024-09-18T08:06:48.731 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:06:48 smithi007 ceph-mon[39249]: 7.5 scrub starts 2024-09-18T08:06:48.731 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:06:48 smithi007 ceph-mon[39249]: pgmap v179: 148 pgs: 2 active+clean+scrubbing, 146 active+clean; 3.3 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 892 B/s rd, 0 B/s wr, 865 op/s 2024-09-18T08:06:48.731 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:06:48 smithi007 ceph-mon[39249]: 3.d scrub starts 2024-09-18T08:06:48.731 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:06:48 smithi007 ceph-mon[39249]: 3.d scrub ok 2024-09-18T08:06:48.731 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:06:48 smithi007 ceph-mon[39249]: from='mgr.24470 172.21.15.7:0/2594390261' entity='mgr.y' 2024-09-18T08:06:48.731 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:06:48 smithi007 ceph-mon[39249]: from='mgr.24470 172.21.15.7:0/2594390261' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-18T08:06:48.731 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:06:48 smithi007 ceph-mon[39249]: from='client.? 172.21.15.7:0/3906338375' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-18T08:06:48.731 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:06:48 smithi007 ceph-mon[29459]: 3.13 scrub starts 2024-09-18T08:06:48.731 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:06:48 smithi007 ceph-mon[29459]: 3.13 scrub ok 2024-09-18T08:06:48.731 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:06:48 smithi007 ceph-mon[29459]: 7.5 scrub starts 2024-09-18T08:06:48.732 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:06:48 smithi007 ceph-mon[29459]: pgmap v179: 148 pgs: 2 active+clean+scrubbing, 146 active+clean; 3.3 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 892 B/s rd, 0 B/s wr, 865 op/s 2024-09-18T08:06:48.732 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:06:48 smithi007 ceph-mon[29459]: 3.d scrub starts 2024-09-18T08:06:48.732 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:06:48 smithi007 ceph-mon[29459]: 3.d scrub ok 2024-09-18T08:06:48.732 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:06:48 smithi007 ceph-mon[29459]: from='mgr.24470 172.21.15.7:0/2594390261' entity='mgr.y' 2024-09-18T08:06:48.732 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:06:48 smithi007 ceph-mon[29459]: from='mgr.24470 172.21.15.7:0/2594390261' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-18T08:06:48.732 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:06:48 smithi007 ceph-mon[29459]: from='client.? 172.21.15.7:0/3906338375' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-18T08:06:48.737 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1 2024-09-18T08:06:48.778 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:06:48.779 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:06:48.779 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1' 2024-09-18T08:06:48.880 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7 2024-09-18T08:06:48.907 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:06:48.908 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:06:48.908 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7' 2024-09-18T08:06:49.009 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3 2024-09-18T08:06:49.049 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:06:49.050 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:06:49.050 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3' 2024-09-18T08:06:49.151 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0 2024-09-18T08:06:49.155 INFO:journalctl@ceph.iscsi.iscsi.a.smithi050.stdout:Sep 18 08:06:48 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-iscsi-iscsi-a[57782]: debug there is no tcmu-runner data available 2024-09-18T08:06:49.198 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:06:49.199 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:06:49.199 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0' 2024-09-18T08:06:49.300 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6 2024-09-18T08:06:49.327 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:06:49.328 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:06:49.328 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6' 2024-09-18T08:06:49.429 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5 2024-09-18T08:06:49.457 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:06:49.458 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:06:49.459 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5' 2024-09-18T08:06:49.559 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3 2024-09-18T08:06:49.597 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:06:49.597 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:06:49.597 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3' 2024-09-18T08:06:49.635 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:06:49 smithi050 ceph-mon[32371]: 6.12 scrub starts 2024-09-18T08:06:49.636 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:06:49 smithi050 ceph-mon[32371]: 6.12 scrub ok 2024-09-18T08:06:49.636 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:06:49 smithi050 ceph-mon[32371]: 7.5 scrub ok 2024-09-18T08:06:49.675 DEBUG:teuthology.orchestra.run.smithi050:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid 779349ca-7593-11ef-bceb-c7b262605968 -- ceph daemon osd.6 dump_historic_ops 2024-09-18T08:06:49.693 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid 779349ca-7593-11ef-bceb-c7b262605968 -- ceph osd set noscrub 2024-09-18T08:06:49.698 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2 2024-09-18T08:06:49.730 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:06:49 smithi007 ceph-mon[39249]: 6.12 scrub starts 2024-09-18T08:06:49.731 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:06:49 smithi007 ceph-mon[39249]: 6.12 scrub ok 2024-09-18T08:06:49.731 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:06:49 smithi007 ceph-mon[39249]: 7.5 scrub ok 2024-09-18T08:06:49.731 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:06:49 smithi007 ceph-mon[29459]: 6.12 scrub starts 2024-09-18T08:06:49.731 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:06:49 smithi007 ceph-mon[29459]: 6.12 scrub ok 2024-09-18T08:06:49.731 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:06:49 smithi007 ceph-mon[29459]: 7.5 scrub ok 2024-09-18T08:06:49.781 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:06:49.782 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:06:49.782 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2' 2024-09-18T08:06:49.883 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4 2024-09-18T08:06:49.915 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:06:49.916 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:06:49.916 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4' 2024-09-18T08:06:49.998 INFO:teuthology.orchestra.run.smithi050.stderr:Inferring config /var/lib/ceph/779349ca-7593-11ef-bceb-c7b262605968/mon.b/config 2024-09-18T08:06:50.017 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0 2024-09-18T08:06:50.080 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:06:50.081 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:06:50.081 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0' 2024-09-18T08:06:50.163 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/779349ca-7593-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T08:06:50.182 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0 2024-09-18T08:06:50.248 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:06:50.249 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:06:50.249 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0' 2024-09-18T08:06:50.350 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0 2024-09-18T08:06:50.382 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:06:50.383 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:06:50.383 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0' 2024-09-18T08:06:50.484 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4 2024-09-18T08:06:50.515 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:06:50.516 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:06:50.517 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4' 2024-09-18T08:06:50.544 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2024-09-18T08:06:50.544 INFO:teuthology.orchestra.run.smithi050.stdout: "size": 20, 2024-09-18T08:06:50.544 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 600, 2024-09-18T08:06:50.544 INFO:teuthology.orchestra.run.smithi050.stdout: "ops": [ 2024-09-18T08:06:50.544 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:06:50.544 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.14805.0:38000 7.7 7:e509d896:::benchmark_data_smithi007_73155_object37999:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e94)", 2024-09-18T08:06:50.544 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-09-18T08:04:23.049009+0000", 2024-09-18T08:06:50.545 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 147.491354879, 2024-09-18T08:06:50.545 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.027565702000000001, 2024-09-18T08:06:50.545 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-09-18T08:06:50.545 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-09-18T08:06:50.545 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T08:06:50.545 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2024-09-18T08:06:50.545 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.14805", 2024-09-18T08:06:50.545 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "172.21.15.7:0/640320436", 2024-09-18T08:06:50.545 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 38000 2024-09-18T08:06:50.546 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:06:50.546 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-09-18T08:06:50.546 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:06:50.546 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-09-18T08:06:50.546 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:23.049009+0000", 2024-09-18T08:06:50.546 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-18T08:06:50.546 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:06:50.546 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:06:50.546 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-09-18T08:06:50.546 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:23.049009+0000", 2024-09-18T08:06:50.546 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-18T08:06:50.547 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:06:50.547 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:06:50.547 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-09-18T08:06:50.547 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:23.049010+0000", 2024-09-18T08:06:50.547 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.11e-06 2024-09-18T08:06:50.547 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:06:50.547 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:06:50.547 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-09-18T08:06:50.547 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:23.068284+0000", 2024-09-18T08:06:50.547 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.019274264999999999 2024-09-18T08:06:50.547 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:06:50.548 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:06:50.548 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-09-18T08:06:50.548 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:23.068323+0000", 2024-09-18T08:06:50.548 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 3.8167999999999998e-05 2024-09-18T08:06:50.548 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:06:50.548 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:06:50.548 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-09-18T08:06:50.548 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:23.068329+0000", 2024-09-18T08:06:50.548 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 6.331e-06 2024-09-18T08:06:50.548 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:06:50.548 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:06:50.548 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-18T08:06:50.549 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:23.068399+0000", 2024-09-18T08:06:50.549 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 7.0055999999999995e-05 2024-09-18T08:06:50.549 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:06:50.549 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:06:50.549 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-09-18T08:06:50.549 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:23.068503+0000", 2024-09-18T08:06:50.549 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000104206 2024-09-18T08:06:50.549 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:06:50.549 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:06:50.549 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "waiting for subops from 0,3", 2024-09-18T08:06:50.549 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:23.068640+0000", 2024-09-18T08:06:50.549 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00013645000000000001 2024-09-18T08:06:50.550 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:06:50.550 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:06:50.550 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "op_commit", 2024-09-18T08:06:50.550 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:23.073270+0000", 2024-09-18T08:06:50.550 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0046298950000000002 2024-09-18T08:06:50.550 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:06:50.550 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:06:50.550 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:06:50.550 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:23.073311+0000", 2024-09-18T08:06:50.550 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4.1406999999999998e-05 2024-09-18T08:06:50.550 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:06:50.550 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:06:50.551 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:06:50.551 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:23.076521+0000", 2024-09-18T08:06:50.551 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.003209862 2024-09-18T08:06:50.551 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:06:50.551 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:06:50.551 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "commit_sent", 2024-09-18T08:06:50.551 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:23.076552+0000", 2024-09-18T08:06:50.551 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 3.1698e-05 2024-09-18T08:06:50.551 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:06:50.551 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:06:50.551 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-09-18T08:06:50.552 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:23.076575+0000", 2024-09-18T08:06:50.552 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 2.2254000000000001e-05 2024-09-18T08:06:50.552 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-18T08:06:50.552 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-09-18T08:06:50.552 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-18T08:06:50.552 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:06:50.552 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:06:50.552 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.14805.0:39064 7.7 7:ef833039:::benchmark_data_smithi007_73155_object39063:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e95)", 2024-09-18T08:06:50.552 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-09-18T08:04:23.658960+0000", 2024-09-18T08:06:50.552 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 146.88140360700001, 2024-09-18T08:06:50.552 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.029075624000000001, 2024-09-18T08:06:50.553 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-09-18T08:06:50.553 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-09-18T08:06:50.553 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T08:06:50.553 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2024-09-18T08:06:50.553 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.14805", 2024-09-18T08:06:50.553 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "172.21.15.7:0/640320436", 2024-09-18T08:06:50.553 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 39064 2024-09-18T08:06:50.553 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:06:50.553 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-09-18T08:06:50.553 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:06:50.553 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-09-18T08:06:50.553 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:23.658960+0000", 2024-09-18T08:06:50.553 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-18T08:06:50.554 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:06:50.554 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:06:50.554 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-09-18T08:06:50.554 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:23.658960+0000", 2024-09-18T08:06:50.554 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-18T08:06:50.554 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:06:50.554 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:06:50.554 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-09-18T08:06:50.554 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:23.658961+0000", 2024-09-18T08:06:50.554 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 6.7899999999999998e-07 2024-09-18T08:06:50.554 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:06:50.554 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:06:50.555 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-09-18T08:06:50.555 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:23.658991+0000", 2024-09-18T08:06:50.555 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 3.0168999999999999e-05 2024-09-18T08:06:50.555 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:06:50.555 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:06:50.555 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-09-18T08:06:50.555 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:23.659001+0000", 2024-09-18T08:06:50.555 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 9.6220000000000004e-06 2024-09-18T08:06:50.555 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:06:50.555 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:06:50.555 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-09-18T08:06:50.555 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:23.659010+0000", 2024-09-18T08:06:50.556 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 8.9160000000000007e-06 2024-09-18T08:06:50.556 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:06:50.556 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:06:50.556 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-18T08:06:50.556 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:23.661923+0000", 2024-09-18T08:06:50.556 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.002913279 2024-09-18T08:06:50.556 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:06:50.556 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:06:50.556 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-09-18T08:06:50.556 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:23.662030+0000", 2024-09-18T08:06:50.556 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00010665700000000001 2024-09-18T08:06:50.556 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:06:50.557 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:06:50.557 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "waiting for subops from 0,3", 2024-09-18T08:06:50.557 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:23.662167+0000", 2024-09-18T08:06:50.557 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00013728300000000001 2024-09-18T08:06:50.557 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:06:50.557 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:06:50.557 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "op_commit", 2024-09-18T08:06:50.557 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:23.663255+0000", 2024-09-18T08:06:50.557 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0010883900000000001 2024-09-18T08:06:50.557 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:06:50.557 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:06:50.558 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:06:50.558 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:23.671279+0000", 2024-09-18T08:06:50.558 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0080233749999999993 2024-09-18T08:06:50.558 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:06:50.558 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:06:50.558 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:06:50.558 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:23.687965+0000", 2024-09-18T08:06:50.558 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.016686590000000001 2024-09-18T08:06:50.558 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:06:50.558 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:06:50.558 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "commit_sent", 2024-09-18T08:06:50.558 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:23.688005+0000", 2024-09-18T08:06:50.559 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 3.9604000000000002e-05 2024-09-18T08:06:50.559 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:06:50.559 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:06:50.559 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-09-18T08:06:50.559 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:23.688036+0000", 2024-09-18T08:06:50.559 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 3.1059999999999997e-05 2024-09-18T08:06:50.559 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-18T08:06:50.559 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-09-18T08:06:50.559 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-18T08:06:50.559 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:06:50.559 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:06:50.559 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.14805.0:39805 7.7 7:e9cd36b3:::benchmark_data_smithi007_73155_object39804:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e95)", 2024-09-18T08:06:50.560 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-09-18T08:04:24.061655+0000", 2024-09-18T08:06:50.560 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 146.478708926, 2024-09-18T08:06:50.560 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.036883131, 2024-09-18T08:06:50.560 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-09-18T08:06:50.560 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-09-18T08:06:50.560 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T08:06:50.560 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2024-09-18T08:06:50.560 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.14805", 2024-09-18T08:06:50.560 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "172.21.15.7:0/640320436", 2024-09-18T08:06:50.560 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 39805 2024-09-18T08:06:50.561 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:06:50.561 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-09-18T08:06:50.561 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:06:50.561 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-09-18T08:06:50.561 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:24.061655+0000", 2024-09-18T08:06:50.561 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-18T08:06:50.561 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:06:50.561 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:06:50.561 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-09-18T08:06:50.561 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:24.061655+0000", 2024-09-18T08:06:50.561 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-18T08:06:50.562 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:06:50.562 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:06:50.562 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-09-18T08:06:50.562 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:24.061656+0000", 2024-09-18T08:06:50.562 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.4449999999999999e-06 2024-09-18T08:06:50.562 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:06:50.562 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:06:50.562 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-09-18T08:06:50.562 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:24.061686+0000", 2024-09-18T08:06:50.562 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 2.9984e-05 2024-09-18T08:06:50.562 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:06:50.562 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:06:50.562 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-09-18T08:06:50.563 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:24.061699+0000", 2024-09-18T08:06:50.563 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.2850000000000001e-05 2024-09-18T08:06:50.563 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:06:50.563 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:06:50.563 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-09-18T08:06:50.563 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:24.061706+0000", 2024-09-18T08:06:50.563 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 6.8560000000000001e-06 2024-09-18T08:06:50.563 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:06:50.563 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:06:50.563 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-18T08:06:50.563 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:24.061755+0000", 2024-09-18T08:06:50.563 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4.9001000000000001e-05 2024-09-18T08:06:50.564 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:06:50.564 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:06:50.564 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-09-18T08:06:50.564 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:24.061896+0000", 2024-09-18T08:06:50.564 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00014040499999999999 2024-09-18T08:06:50.564 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:06:50.564 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:06:50.564 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "waiting for subops from 0,3", 2024-09-18T08:06:50.564 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:24.062006+0000", 2024-09-18T08:06:50.564 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000110813 2024-09-18T08:06:50.565 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:06:50.565 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:06:50.565 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:06:50.565 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:24.065677+0000", 2024-09-18T08:06:50.565 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0036702800000000002 2024-09-18T08:06:50.565 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:06:50.565 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:06:50.565 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "op_commit", 2024-09-18T08:06:50.565 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:24.065694+0000", 2024-09-18T08:06:50.565 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.7876e-05 2024-09-18T08:06:50.565 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:06:50.565 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:06:50.566 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:06:50.566 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:24.098357+0000", 2024-09-18T08:06:50.566 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.032662140999999999 2024-09-18T08:06:50.566 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:06:50.566 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:06:50.566 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "commit_sent", 2024-09-18T08:06:50.566 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:24.098447+0000", 2024-09-18T08:06:50.566 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 9.0724999999999996e-05 2024-09-18T08:06:50.566 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:06:50.566 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:06:50.566 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-09-18T08:06:50.567 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:24.098538+0000", 2024-09-18T08:06:50.567 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 9.0754999999999994e-05 2024-09-18T08:06:50.567 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-18T08:06:50.567 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-09-18T08:06:50.567 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-18T08:06:50.567 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:06:50.567 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:06:50.567 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.14805.0:39807 7.d 7:bf1e27dc:::benchmark_data_smithi007_73155_object39806:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e95)", 2024-09-18T08:06:50.567 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-09-18T08:04:24.061827+0000", 2024-09-18T08:06:50.567 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 146.478536631, 2024-09-18T08:06:50.568 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.037358672000000002, 2024-09-18T08:06:50.568 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-09-18T08:06:50.568 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-09-18T08:06:50.568 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T08:06:50.568 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2024-09-18T08:06:50.568 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.14805", 2024-09-18T08:06:50.568 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "172.21.15.7:0/640320436", 2024-09-18T08:06:50.568 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 39807 2024-09-18T08:06:50.568 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:06:50.568 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-09-18T08:06:50.569 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:06:50.569 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-09-18T08:06:50.569 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:24.061827+0000", 2024-09-18T08:06:50.569 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-18T08:06:50.569 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:06:50.569 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:06:50.569 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-09-18T08:06:50.569 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:24.061827+0000", 2024-09-18T08:06:50.569 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-18T08:06:50.569 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:06:50.569 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:06:50.569 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-09-18T08:06:50.570 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:24.061828+0000", 2024-09-18T08:06:50.570 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 7.9899999999999999e-07 2024-09-18T08:06:50.570 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:06:50.570 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:06:50.570 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-09-18T08:06:50.570 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:24.061888+0000", 2024-09-18T08:06:50.570 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 5.9497000000000003e-05 2024-09-18T08:06:50.570 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:06:50.570 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:06:50.570 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-09-18T08:06:50.570 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:24.061904+0000", 2024-09-18T08:06:50.570 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.6393999999999999e-05 2024-09-18T08:06:50.571 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:06:50.571 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:06:50.571 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-09-18T08:06:50.571 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:24.061912+0000", 2024-09-18T08:06:50.571 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 8.0909999999999993e-06 2024-09-18T08:06:50.571 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:06:50.571 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:06:50.571 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-18T08:06:50.571 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:24.061968+0000", 2024-09-18T08:06:50.571 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 5.5543e-05 2024-09-18T08:06:50.571 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:06:50.571 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:06:50.572 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-09-18T08:06:50.572 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:24.062061+0000", 2024-09-18T08:06:50.572 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 9.3558000000000001e-05 2024-09-18T08:06:50.572 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:06:50.572 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:06:50.572 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "waiting for subops from 0,1", 2024-09-18T08:06:50.572 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:24.062163+0000", 2024-09-18T08:06:50.572 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000101903 2024-09-18T08:06:50.572 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:06:50.572 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:06:50.572 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "op_commit", 2024-09-18T08:06:50.573 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:24.064018+0000", 2024-09-18T08:06:50.573 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.001855071 2024-09-18T08:06:50.573 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:06:50.573 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:06:50.573 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:06:50.573 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:24.064283+0000", 2024-09-18T08:06:50.573 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000265173 2024-09-18T08:06:50.573 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:06:50.573 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:06:50.573 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:06:50.573 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:24.099043+0000", 2024-09-18T08:06:50.574 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.034760159999999998 2024-09-18T08:06:50.574 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:06:50.574 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:06:50.574 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "commit_sent", 2024-09-18T08:06:50.574 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:24.099100+0000", 2024-09-18T08:06:50.574 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 5.6147000000000003e-05 2024-09-18T08:06:50.574 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:06:50.574 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:06:50.574 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-09-18T08:06:50.574 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:24.099186+0000", 2024-09-18T08:06:50.574 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 8.6335999999999995e-05 2024-09-18T08:06:50.574 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-18T08:06:50.575 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-09-18T08:06:50.575 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-18T08:06:50.575 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:06:50.575 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:06:50.575 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.14805.0:39808 7.d 7:b98cab8f:::benchmark_data_smithi007_73155_object39807:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e95)", 2024-09-18T08:06:50.575 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-09-18T08:04:24.067964+0000", 2024-09-18T08:06:50.575 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 146.47240019399999, 2024-09-18T08:06:50.575 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.034258180999999999, 2024-09-18T08:06:50.575 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-09-18T08:06:50.575 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-09-18T08:06:50.575 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T08:06:50.575 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2024-09-18T08:06:50.576 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.14805", 2024-09-18T08:06:50.576 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "172.21.15.7:0/640320436", 2024-09-18T08:06:50.576 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 39808 2024-09-18T08:06:50.576 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:06:50.576 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-09-18T08:06:50.576 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:06:50.576 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-09-18T08:06:50.576 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:24.067964+0000", 2024-09-18T08:06:50.576 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-18T08:06:50.576 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:06:50.576 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:06:50.577 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-09-18T08:06:50.577 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:24.067964+0000", 2024-09-18T08:06:50.577 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-18T08:06:50.577 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:06:50.577 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:06:50.578 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-09-18T08:06:50.578 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:24.067965+0000", 2024-09-18T08:06:50.578 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.4750000000000001e-06 2024-09-18T08:06:50.579 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:06:50.579 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:06:50.579 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-09-18T08:06:50.579 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:24.067992+0000", 2024-09-18T08:06:50.579 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 2.6760000000000001e-05 2024-09-18T08:06:50.579 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:06:50.579 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:06:50.579 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-09-18T08:06:50.579 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:24.068004+0000", 2024-09-18T08:06:50.579 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.2456000000000001e-05 2024-09-18T08:06:50.579 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:06:50.580 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:06:50.580 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-09-18T08:06:50.580 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:24.068012+0000", 2024-09-18T08:06:50.580 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 8.0779999999999996e-06 2024-09-18T08:06:50.580 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:06:50.580 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:06:50.580 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-18T08:06:50.580 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:24.068072+0000", 2024-09-18T08:06:50.580 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 5.9299000000000003e-05 2024-09-18T08:06:50.580 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:06:50.580 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:06:50.581 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-09-18T08:06:50.581 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:24.068164+0000", 2024-09-18T08:06:50.581 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 9.2053999999999997e-05 2024-09-18T08:06:50.581 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:06:50.581 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:06:50.581 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "waiting for subops from 0,1", 2024-09-18T08:06:50.581 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:24.096945+0000", 2024-09-18T08:06:50.581 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.028781128 2024-09-18T08:06:50.581 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:06:50.582 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:06:50.582 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "op_commit", 2024-09-18T08:06:50.582 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:24.099254+0000", 2024-09-18T08:06:50.582 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.002309218 2024-09-18T08:06:50.582 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:06:50.582 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:06:50.582 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:06:50.582 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:24.100291+0000", 2024-09-18T08:06:50.582 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0010365280000000001 2024-09-18T08:06:50.582 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:06:50.582 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:06:50.583 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:06:50.583 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:24.102125+0000", 2024-09-18T08:06:50.583 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0018346599999999999 2024-09-18T08:06:50.583 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:06:50.583 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:06:50.583 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "commit_sent", 2024-09-18T08:06:50.583 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:24.102184+0000", 2024-09-18T08:06:50.583 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 5.8965000000000002e-05 2024-09-18T08:06:50.583 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:06:50.583 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:06:50.583 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-09-18T08:06:50.583 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:24.102222+0000", 2024-09-18T08:06:50.583 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 3.756e-05 2024-09-18T08:06:50.584 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-18T08:06:50.584 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-09-18T08:06:50.584 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-18T08:06:50.584 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:06:50.584 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:06:50.584 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_repop_reply(client.14805.0:39807 7.d e95/83 ondisk, result = 0)", 2024-09-18T08:06:50.584 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-09-18T08:04:24.069321+0000", 2024-09-18T08:06:50.584 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 146.47104282999999, 2024-09-18T08:06:50.584 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.029872724999999999, 2024-09-18T08:06:50.584 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-09-18T08:06:50.584 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-09-18T08:06:50.585 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "started", 2024-09-18T08:06:50.585 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-09-18T08:06:50.585 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:06:50.585 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-09-18T08:06:50.585 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:24.069321+0000", 2024-09-18T08:06:50.585 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-18T08:06:50.585 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:06:50.585 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:06:50.585 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-09-18T08:06:50.585 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:24.069321+0000", 2024-09-18T08:06:50.585 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-18T08:06:50.586 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:06:50.586 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:06:50.586 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-09-18T08:06:50.586 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:24.069322+0000", 2024-09-18T08:06:50.586 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 8.3799999999999996e-07 2024-09-18T08:06:50.586 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:06:50.586 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:06:50.586 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-09-18T08:06:50.586 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:24.069327+0000", 2024-09-18T08:06:50.586 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4.8679999999999998e-06 2024-09-18T08:06:50.586 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:06:50.586 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:06:50.587 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-09-18T08:06:50.587 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:24.098180+0000", 2024-09-18T08:06:50.587 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.028853407000000001 2024-09-18T08:06:50.587 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:06:50.587 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:06:50.587 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-09-18T08:06:50.587 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:24.098189+0000", 2024-09-18T08:06:50.587 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 8.8850000000000005e-06 2024-09-18T08:06:50.587 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:06:50.587 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:06:50.587 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-18T08:06:50.588 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:24.099029+0000", 2024-09-18T08:06:50.588 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00084036500000000004 2024-09-18T08:06:50.588 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:06:50.588 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:06:50.588 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-09-18T08:06:50.588 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:24.099037+0000", 2024-09-18T08:06:50.588 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 8.0150000000000003e-06 2024-09-18T08:06:50.588 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:06:50.588 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:06:50.588 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-09-18T08:06:50.588 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:24.099194+0000", 2024-09-18T08:06:50.588 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00015634700000000001 2024-09-18T08:06:50.589 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-18T08:06:50.589 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-09-18T08:06:50.589 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-18T08:06:50.589 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:06:50.589 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:06:50.589 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.14805.0:39817 7.c 7:3826d3a8:::benchmark_data_smithi007_73155_object39816:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e96)", 2024-09-18T08:06:50.589 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-09-18T08:04:24.077849+0000", 2024-09-18T08:06:50.589 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 146.46251488199999, 2024-09-18T08:06:50.589 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.028025705000000001, 2024-09-18T08:06:50.589 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-09-18T08:06:50.589 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-09-18T08:06:50.590 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T08:06:50.590 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2024-09-18T08:06:50.590 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.14805", 2024-09-18T08:06:50.590 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "172.21.15.7:0/640320436", 2024-09-18T08:06:50.590 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 39817 2024-09-18T08:06:50.590 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:06:50.590 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-09-18T08:06:50.590 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:06:50.590 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-09-18T08:06:50.590 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:24.077849+0000", 2024-09-18T08:06:50.590 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-18T08:06:50.590 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:06:50.591 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:06:50.591 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-09-18T08:06:50.591 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:24.077849+0000", 2024-09-18T08:06:50.591 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-18T08:06:50.591 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:06:50.591 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:06:50.591 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-09-18T08:06:50.591 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:24.077851+0000", 2024-09-18T08:06:50.591 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 2.322e-06 2024-09-18T08:06:50.591 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:06:50.591 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:06:50.592 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-09-18T08:06:50.592 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:24.077879+0000", 2024-09-18T08:06:50.592 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 2.7596000000000001e-05 2024-09-18T08:06:50.592 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:06:50.592 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:06:50.592 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-09-18T08:06:50.592 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:24.097805+0000", 2024-09-18T08:06:50.592 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.019925552999999999 2024-09-18T08:06:50.592 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:06:50.592 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:06:50.592 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-09-18T08:06:50.592 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:24.097820+0000", 2024-09-18T08:06:50.593 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.5705999999999998e-05 2024-09-18T08:06:50.593 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:06:50.593 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:06:50.593 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-18T08:06:50.593 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:24.097984+0000", 2024-09-18T08:06:50.593 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00016333499999999999 2024-09-18T08:06:50.593 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:06:50.593 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:06:50.593 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "op must wait for map", 2024-09-18T08:06:50.593 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:24.097990+0000", 2024-09-18T08:06:50.593 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 6.81e-06 2024-09-18T08:06:50.593 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:06:50.594 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:06:50.594 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-18T08:06:50.594 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:24.102003+0000", 2024-09-18T08:06:50.594 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0040126160000000001 2024-09-18T08:06:50.594 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:06:50.594 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:06:50.594 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-09-18T08:06:50.594 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:24.102155+0000", 2024-09-18T08:06:50.594 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00015247500000000001 2024-09-18T08:06:50.594 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:06:50.595 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:06:50.595 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "waiting for subops from 3,7", 2024-09-18T08:06:50.595 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:24.102305+0000", 2024-09-18T08:06:50.595 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000149872 2024-09-18T08:06:50.595 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:06:50.595 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:06:50.595 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "op_commit", 2024-09-18T08:06:50.595 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:24.105200+0000", 2024-09-18T08:06:50.595 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0028950730000000002 2024-09-18T08:06:50.595 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:06:50.595 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:06:50.595 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:06:50.596 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:24.105248+0000", 2024-09-18T08:06:50.596 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4.7787999999999997e-05 2024-09-18T08:06:50.596 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:06:50.596 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:06:50.596 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:06:50.596 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:24.105772+0000", 2024-09-18T08:06:50.596 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00052425699999999998 2024-09-18T08:06:50.596 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:06:50.596 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:06:50.596 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "commit_sent", 2024-09-18T08:06:50.596 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:24.105826+0000", 2024-09-18T08:06:50.596 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 5.393e-05 2024-09-18T08:06:50.597 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:06:50.597 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:06:50.597 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-09-18T08:06:50.597 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:24.105875+0000", 2024-09-18T08:06:50.597 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4.8371999999999997e-05 2024-09-18T08:06:50.597 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-18T08:06:50.597 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-09-18T08:06:50.597 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-18T08:06:50.597 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:06:50.597 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:06:50.597 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.14805.0:43273 7.c 7:38821c7c:::benchmark_data_smithi007_73155_object43272:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e96)", 2024-09-18T08:06:50.598 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-09-18T08:04:26.167716+0000", 2024-09-18T08:06:50.598 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 144.37264795300001, 2024-09-18T08:06:50.598 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.027026471999999999, 2024-09-18T08:06:50.598 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-09-18T08:06:50.598 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-09-18T08:06:50.598 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T08:06:50.598 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2024-09-18T08:06:50.598 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.14805", 2024-09-18T08:06:50.598 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "172.21.15.7:0/640320436", 2024-09-18T08:06:50.598 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 43273 2024-09-18T08:06:50.598 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:06:50.599 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-09-18T08:06:50.599 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:06:50.599 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-09-18T08:06:50.599 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:26.167716+0000", 2024-09-18T08:06:50.599 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-18T08:06:50.599 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:06:50.599 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:06:50.599 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-09-18T08:06:50.599 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:26.167716+0000", 2024-09-18T08:06:50.599 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-18T08:06:50.599 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:06:50.599 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:06:50.599 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-09-18T08:06:50.600 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:26.167717+0000", 2024-09-18T08:06:50.600 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.3629999999999999e-06 2024-09-18T08:06:50.600 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:06:50.600 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:06:50.600 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-09-18T08:06:50.600 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:26.187242+0000", 2024-09-18T08:06:50.600 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.019525094999999999 2024-09-18T08:06:50.600 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:06:50.600 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:06:50.600 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-09-18T08:06:50.600 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:26.187255+0000", 2024-09-18T08:06:50.600 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.2669e-05 2024-09-18T08:06:50.601 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:06:50.601 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:06:50.601 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-09-18T08:06:50.601 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:26.187264+0000", 2024-09-18T08:06:50.601 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 9.3770000000000002e-06 2024-09-18T08:06:50.601 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:06:50.601 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:06:50.601 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-18T08:06:50.601 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:26.187321+0000", 2024-09-18T08:06:50.601 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 5.6379999999999999e-05 2024-09-18T08:06:50.601 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:06:50.601 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:06:50.602 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-09-18T08:06:50.602 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:26.187444+0000", 2024-09-18T08:06:50.602 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00012314699999999999 2024-09-18T08:06:50.602 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:06:50.602 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:06:50.602 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "waiting for subops from 3,7", 2024-09-18T08:06:50.602 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:26.187575+0000", 2024-09-18T08:06:50.602 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00013138700000000001 2024-09-18T08:06:50.602 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:06:50.602 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:06:50.602 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "op_commit", 2024-09-18T08:06:50.603 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:26.189195+0000", 2024-09-18T08:06:50.603 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0016195770000000001 2024-09-18T08:06:50.603 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:06:50.603 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:06:50.603 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:06:50.603 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:26.189327+0000", 2024-09-18T08:06:50.603 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00013181099999999999 2024-09-18T08:06:50.603 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:06:50.603 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:06:50.603 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:06:50.603 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:26.194627+0000", 2024-09-18T08:06:50.604 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0052998799999999999 2024-09-18T08:06:50.604 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:06:50.604 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:06:50.604 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "commit_sent", 2024-09-18T08:06:50.604 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:26.194693+0000", 2024-09-18T08:06:50.604 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 6.6748999999999994e-05 2024-09-18T08:06:50.604 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:06:50.604 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:06:50.604 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-09-18T08:06:50.604 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:26.194742+0000", 2024-09-18T08:06:50.604 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4.9036999999999998e-05 2024-09-18T08:06:50.604 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-18T08:06:50.605 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-09-18T08:06:50.605 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-18T08:06:50.605 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:06:50.605 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:06:50.605 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.14805.0:43281 7.c 7:35a814bc:::benchmark_data_smithi007_73155_object43280:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e96)", 2024-09-18T08:06:50.605 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-09-18T08:04:26.187285+0000", 2024-09-18T08:06:50.605 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 144.35307876499999, 2024-09-18T08:06:50.605 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.026194973999999999, 2024-09-18T08:06:50.605 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-09-18T08:06:50.605 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-09-18T08:06:50.605 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T08:06:50.605 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2024-09-18T08:06:50.606 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.14805", 2024-09-18T08:06:50.606 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "172.21.15.7:0/640320436", 2024-09-18T08:06:50.606 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 43281 2024-09-18T08:06:50.606 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:06:50.606 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-09-18T08:06:50.606 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:06:50.606 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-09-18T08:06:50.606 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:26.187285+0000", 2024-09-18T08:06:50.606 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-18T08:06:50.606 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:06:50.607 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:06:50.607 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-09-18T08:06:50.607 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:26.187285+0000", 2024-09-18T08:06:50.607 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-18T08:06:50.607 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:06:50.607 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:06:50.607 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-09-18T08:06:50.607 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:26.187286+0000", 2024-09-18T08:06:50.607 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.2190000000000001e-06 2024-09-18T08:06:50.607 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:06:50.607 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:06:50.608 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-09-18T08:06:50.608 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:26.206523+0000", 2024-09-18T08:06:50.608 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.019236797999999999 2024-09-18T08:06:50.608 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:06:50.608 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:06:50.608 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-09-18T08:06:50.608 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:26.206550+0000", 2024-09-18T08:06:50.608 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 2.6509999999999999e-05 2024-09-18T08:06:50.608 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:06:50.608 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:06:50.608 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-09-18T08:06:50.609 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:26.206567+0000", 2024-09-18T08:06:50.609 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.7401000000000002e-05 2024-09-18T08:06:50.609 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:06:50.609 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:06:50.609 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-18T08:06:50.609 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:26.206657+0000", 2024-09-18T08:06:50.609 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 8.9485000000000001e-05 2024-09-18T08:06:50.609 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:06:50.609 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:06:50.609 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-09-18T08:06:50.609 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:26.206738+0000", 2024-09-18T08:06:50.609 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 8.1077999999999999e-05 2024-09-18T08:06:50.610 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:06:50.610 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:06:50.610 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "waiting for subops from 3,7", 2024-09-18T08:06:50.610 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:26.206832+0000", 2024-09-18T08:06:50.610 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 9.4160000000000006e-05 2024-09-18T08:06:50.610 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:06:50.610 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:06:50.610 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:06:50.610 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:26.213326+0000", 2024-09-18T08:06:50.610 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0064943099999999997 2024-09-18T08:06:50.610 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:06:50.610 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:06:50.610 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "op_commit", 2024-09-18T08:06:50.610 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:26.213342+0000", 2024-09-18T08:06:50.611 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.5415000000000001e-05 2024-09-18T08:06:50.611 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:06:50.611 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:06:50.611 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:06:50.611 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:26.213393+0000", 2024-09-18T08:06:50.611 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 5.1038999999999997e-05 2024-09-18T08:06:50.611 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:06:50.611 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:06:50.611 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "commit_sent", 2024-09-18T08:06:50.611 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:26.213443+0000", 2024-09-18T08:06:50.611 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4.9938999999999997e-05 2024-09-18T08:06:50.611 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:06:50.612 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:06:50.612 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-09-18T08:06:50.612 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:26.213480+0000", 2024-09-18T08:06:50.612 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 3.7620000000000002e-05 2024-09-18T08:06:50.612 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-18T08:06:50.612 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-09-18T08:06:50.612 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-18T08:06:50.612 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:06:50.612 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:06:50.612 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.14805.0:43731 7.d 7:b3fc8faf:::benchmark_data_smithi007_73155_object43730:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e97)", 2024-09-18T08:06:50.612 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-09-18T08:04:26.432079+0000", 2024-09-18T08:06:50.612 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 144.10828460799999, 2024-09-18T08:06:50.613 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.027497293999999999, 2024-09-18T08:06:50.613 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-09-18T08:06:50.613 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-09-18T08:06:50.613 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T08:06:50.613 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2024-09-18T08:06:50.613 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.14805", 2024-09-18T08:06:50.613 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "172.21.15.7:0/640320436", 2024-09-18T08:06:50.613 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 43731 2024-09-18T08:06:50.613 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:06:50.613 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-09-18T08:06:50.613 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:06:50.613 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-09-18T08:06:50.613 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:26.432079+0000", 2024-09-18T08:06:50.614 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-18T08:06:50.614 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:06:50.614 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:06:50.614 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-09-18T08:06:50.614 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:26.432079+0000", 2024-09-18T08:06:50.614 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-18T08:06:50.614 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:06:50.614 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:06:50.614 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-09-18T08:06:50.614 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:26.432080+0000", 2024-09-18T08:06:50.614 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.128e-06 2024-09-18T08:06:50.615 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:06:50.615 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:06:50.615 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-09-18T08:06:50.615 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:26.432153+0000", 2024-09-18T08:06:50.615 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 7.2521999999999994e-05 2024-09-18T08:06:50.615 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:06:50.615 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:06:50.615 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-09-18T08:06:50.615 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:26.432163+0000", 2024-09-18T08:06:50.615 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.0115999999999999e-05 2024-09-18T08:06:50.615 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:06:50.616 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:06:50.616 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-09-18T08:06:50.616 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:26.432170+0000", 2024-09-18T08:06:50.616 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 7.0729999999999999e-06 2024-09-18T08:06:50.616 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:06:50.616 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:06:50.616 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-18T08:06:50.616 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:26.432225+0000", 2024-09-18T08:06:50.616 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 5.4463999999999999e-05 2024-09-18T08:06:50.616 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:06:50.616 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:06:50.617 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-09-18T08:06:50.617 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:26.432342+0000", 2024-09-18T08:06:50.617 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000117118 2024-09-18T08:06:50.617 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:06:50.617 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:06:50.617 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "waiting for subops from 0,1", 2024-09-18T08:06:50.617 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:26.432467+0000", 2024-09-18T08:06:50.617 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00012505700000000001 2024-09-18T08:06:50.617 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:06:50.617 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:06:50.618 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "op_commit", 2024-09-18T08:06:50.618 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:26.433881+0000", 2024-09-18T08:06:50.618 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.001413812 2024-09-18T08:06:50.618 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:06:50.618 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:06:50.618 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:06:50.618 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:26.434589+0000", 2024-09-18T08:06:50.618 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00070883600000000001 2024-09-18T08:06:50.618 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:06:50.618 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:06:50.619 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:06:50.619 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:26.459469+0000", 2024-09-18T08:06:50.619 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.024879951000000001 2024-09-18T08:06:50.619 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:06:50.619 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:06:50.619 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "commit_sent", 2024-09-18T08:06:50.619 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:26.459527+0000", 2024-09-18T08:06:50.619 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 5.7126999999999997e-05 2024-09-18T08:06:50.619 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:06:50.619 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:06:50.619 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-09-18T08:06:50.619 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:26.459577+0000", 2024-09-18T08:06:50.620 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 5.0090000000000003e-05 2024-09-18T08:06:50.620 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-18T08:06:50.620 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-09-18T08:06:50.620 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-18T08:06:50.620 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:06:50.620 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:06:50.620 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.14805.0:63861 7.d 7:bf91474b:::benchmark_data_smithi007_73155_object63860:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e100)", 2024-09-18T08:06:50.620 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-09-18T08:04:41.125989+0000", 2024-09-18T08:06:50.620 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 129.41437531599999, 2024-09-18T08:06:50.620 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.031566575999999999, 2024-09-18T08:06:50.620 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-09-18T08:06:50.621 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-09-18T08:06:50.621 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T08:06:50.621 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2024-09-18T08:06:50.621 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.14805", 2024-09-18T08:06:50.621 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "172.21.15.7:0/640320436", 2024-09-18T08:06:50.621 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 63861 2024-09-18T08:06:50.621 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:06:50.621 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-09-18T08:06:50.621 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:06:50.621 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-09-18T08:06:50.621 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:41.125989+0000", 2024-09-18T08:06:50.622 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-18T08:06:50.622 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:06:50.622 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:06:50.622 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-09-18T08:06:50.622 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:41.125989+0000", 2024-09-18T08:06:50.622 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-18T08:06:50.622 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:06:50.622 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:06:50.622 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-09-18T08:06:50.622 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:41.125990+0000", 2024-09-18T08:06:50.622 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.6109999999999999e-06 2024-09-18T08:06:50.622 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:06:50.623 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:06:50.623 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-09-18T08:06:50.623 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:41.126017+0000", 2024-09-18T08:06:50.623 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 2.6594999999999999e-05 2024-09-18T08:06:50.623 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:06:50.623 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:06:50.623 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-09-18T08:06:50.623 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:41.126035+0000", 2024-09-18T08:06:50.623 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.7842999999999998e-05 2024-09-18T08:06:50.623 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:06:50.623 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:06:50.624 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-09-18T08:06:50.624 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:41.126043+0000", 2024-09-18T08:06:50.624 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 8.8489999999999995e-06 2024-09-18T08:06:50.624 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:06:50.624 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:06:50.624 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-18T08:06:50.624 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:41.126104+0000", 2024-09-18T08:06:50.624 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 6.0448999999999997e-05 2024-09-18T08:06:50.624 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:06:50.624 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:06:50.624 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-09-18T08:06:50.625 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:41.126225+0000", 2024-09-18T08:06:50.625 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000121101 2024-09-18T08:06:50.625 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:06:50.625 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:06:50.625 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "waiting for subops from 0,1", 2024-09-18T08:06:50.625 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:41.126366+0000", 2024-09-18T08:06:50.625 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00014059899999999999 2024-09-18T08:06:50.625 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:06:50.625 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:06:50.625 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "op_commit", 2024-09-18T08:06:50.625 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:41.129750+0000", 2024-09-18T08:06:50.625 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.003383863 2024-09-18T08:06:50.626 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:06:50.626 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:06:50.626 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:06:50.626 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:41.140266+0000", 2024-09-18T08:06:50.626 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.010516266999999999 2024-09-18T08:06:50.626 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:06:50.626 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:06:50.626 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:06:50.626 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:41.157441+0000", 2024-09-18T08:06:50.626 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.017174891000000001 2024-09-18T08:06:50.626 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:06:50.626 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:06:50.627 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "commit_sent", 2024-09-18T08:06:50.627 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:41.157509+0000", 2024-09-18T08:06:50.627 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 6.8627999999999994e-05 2024-09-18T08:06:50.627 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:06:50.627 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:06:50.627 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-09-18T08:06:50.627 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:41.157555+0000", 2024-09-18T08:06:50.627 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4.5880000000000001e-05 2024-09-18T08:06:50.627 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-18T08:06:50.627 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-09-18T08:06:50.627 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-18T08:06:50.627 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:06:50.628 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:06:50.628 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.14805.0:63862 7.d 7:b1f318bc:::benchmark_data_smithi007_73155_object63861:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e100)", 2024-09-18T08:06:50.628 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-09-18T08:04:41.126934+0000", 2024-09-18T08:06:50.628 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 129.41343017899999, 2024-09-18T08:06:50.628 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.030729600999999999, 2024-09-18T08:06:50.628 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-09-18T08:06:50.628 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-09-18T08:06:50.628 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T08:06:50.628 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2024-09-18T08:06:50.628 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.14805", 2024-09-18T08:06:50.629 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "172.21.15.7:0/640320436", 2024-09-18T08:06:50.629 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 63862 2024-09-18T08:06:50.629 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:06:50.629 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-09-18T08:06:50.629 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:06:50.629 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-09-18T08:06:50.629 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:41.126934+0000", 2024-09-18T08:06:50.629 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-18T08:06:50.629 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:06:50.629 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:06:50.629 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-09-18T08:06:50.630 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:41.126934+0000", 2024-09-18T08:06:50.630 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-18T08:06:50.630 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:06:50.630 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:06:50.630 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-09-18T08:06:50.630 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:41.126935+0000", 2024-09-18T08:06:50.630 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 7.9599999999999998e-07 2024-09-18T08:06:50.630 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:06:50.630 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:06:50.630 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-09-18T08:06:50.630 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:41.126989+0000", 2024-09-18T08:06:50.630 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 5.4783e-05 2024-09-18T08:06:50.631 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:06:50.631 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:06:50.631 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-09-18T08:06:50.631 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:41.126999+0000", 2024-09-18T08:06:50.631 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 9.8889999999999997e-06 2024-09-18T08:06:50.631 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:06:50.631 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:06:50.631 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-09-18T08:06:50.631 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:41.127007+0000", 2024-09-18T08:06:50.631 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 7.4399999999999999e-06 2024-09-18T08:06:50.631 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:06:50.631 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:06:50.632 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-18T08:06:50.632 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:41.128171+0000", 2024-09-18T08:06:50.632 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0011642600000000001 2024-09-18T08:06:50.632 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:06:50.632 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:06:50.632 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-09-18T08:06:50.632 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:41.128287+0000", 2024-09-18T08:06:50.632 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000116508 2024-09-18T08:06:50.632 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:06:50.632 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:06:50.633 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "waiting for subops from 0,1", 2024-09-18T08:06:50.633 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:41.128375+0000", 2024-09-18T08:06:50.633 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 8.7300999999999996e-05 2024-09-18T08:06:50.633 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:06:50.633 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:06:50.633 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "op_commit", 2024-09-18T08:06:50.633 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:41.130919+0000", 2024-09-18T08:06:50.633 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0025447450000000002 2024-09-18T08:06:50.633 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:06:50.633 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:06:50.633 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:06:50.634 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:41.140334+0000", 2024-09-18T08:06:50.634 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0094146549999999992 2024-09-18T08:06:50.634 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:06:50.634 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:06:50.634 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:06:50.634 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:41.157602+0000", 2024-09-18T08:06:50.634 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.017267966999999999 2024-09-18T08:06:50.634 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:06:50.634 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:06:50.634 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "commit_sent", 2024-09-18T08:06:50.634 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:41.157636+0000", 2024-09-18T08:06:50.635 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 3.4214000000000001e-05 2024-09-18T08:06:50.635 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:06:50.635 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:06:50.635 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-09-18T08:06:50.635 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:41.157663+0000", 2024-09-18T08:06:50.635 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 2.7042999999999998e-05 2024-09-18T08:06:50.635 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-18T08:06:50.635 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-09-18T08:06:50.635 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-18T08:06:50.635 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:06:50.635 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:06:50.635 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.14805.0:63864 7.d 7:bb7b1761:::benchmark_data_smithi007_73155_object63863:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e100)", 2024-09-18T08:06:50.636 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-09-18T08:04:41.127581+0000", 2024-09-18T08:06:50.636 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 129.41278271600001, 2024-09-18T08:06:50.636 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.030231638000000002, 2024-09-18T08:06:50.636 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-09-18T08:06:50.636 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-09-18T08:06:50.636 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T08:06:50.636 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2024-09-18T08:06:50.636 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.14805", 2024-09-18T08:06:50.636 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "172.21.15.7:0/640320436", 2024-09-18T08:06:50.636 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 63864 2024-09-18T08:06:50.637 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:06:50.637 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-09-18T08:06:50.637 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:06:50.637 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-09-18T08:06:50.637 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:41.127581+0000", 2024-09-18T08:06:50.637 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-18T08:06:50.637 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:06:50.637 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:06:50.637 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-09-18T08:06:50.637 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:41.127581+0000", 2024-09-18T08:06:50.637 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-18T08:06:50.638 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:06:50.638 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:06:50.638 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-09-18T08:06:50.638 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:41.127582+0000", 2024-09-18T08:06:50.638 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 7.8999999999999995e-07 2024-09-18T08:06:50.638 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:06:50.638 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:06:50.638 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-09-18T08:06:50.638 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:41.127648+0000", 2024-09-18T08:06:50.638 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 6.5618999999999996e-05 2024-09-18T08:06:50.638 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:06:50.638 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:06:50.639 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-09-18T08:06:50.639 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:41.127658+0000", 2024-09-18T08:06:50.639 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.0192e-05 2024-09-18T08:06:50.639 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:06:50.639 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:06:50.639 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-09-18T08:06:50.639 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:41.127663+0000", 2024-09-18T08:06:50.639 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 5.2800000000000003e-06 2024-09-18T08:06:50.639 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:06:50.639 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:06:50.639 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-18T08:06:50.640 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:41.129783+0000", 2024-09-18T08:06:50.640 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0021199740000000002 2024-09-18T08:06:50.640 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:06:50.640 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:06:50.640 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-09-18T08:06:50.640 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:41.129885+0000", 2024-09-18T08:06:50.640 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000101987 2024-09-18T08:06:50.640 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:06:50.640 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:06:50.640 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "waiting for subops from 0,1", 2024-09-18T08:06:50.640 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:41.129980+0000", 2024-09-18T08:06:50.640 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 9.4480000000000003e-05 2024-09-18T08:06:50.641 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:06:50.641 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:06:50.641 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "op_commit", 2024-09-18T08:06:50.641 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:41.131035+0000", 2024-09-18T08:06:50.641 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.001055884 2024-09-18T08:06:50.641 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:06:50.641 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:06:50.641 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:06:50.641 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:41.140376+0000", 2024-09-18T08:06:50.641 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0093401439999999999 2024-09-18T08:06:50.641 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:06:50.641 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:06:50.642 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:06:50.642 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:41.157713+0000", 2024-09-18T08:06:50.642 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.017337676999999999 2024-09-18T08:06:50.642 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:06:50.642 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:06:50.642 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "commit_sent", 2024-09-18T08:06:50.642 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:41.157771+0000", 2024-09-18T08:06:50.642 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 5.7674999999999999e-05 2024-09-18T08:06:50.642 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:06:50.642 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:06:50.642 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-09-18T08:06:50.643 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:41.157813+0000", 2024-09-18T08:06:50.643 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4.1936e-05 2024-09-18T08:06:50.643 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-18T08:06:50.643 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-09-18T08:06:50.643 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-18T08:06:50.643 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:06:50.643 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:06:50.643 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.14805.0:63883 7.7 7:ebafa8b8:::benchmark_data_smithi007_73155_object63882:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e100)", 2024-09-18T08:06:50.643 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-09-18T08:04:41.154209+0000", 2024-09-18T08:06:50.643 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 129.38615452400001, 2024-09-18T08:06:50.643 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.042338318999999999, 2024-09-18T08:06:50.643 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-09-18T08:06:50.644 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-09-18T08:06:50.644 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T08:06:50.644 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2024-09-18T08:06:50.644 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.14805", 2024-09-18T08:06:50.644 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "172.21.15.7:0/640320436", 2024-09-18T08:06:50.644 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 63883 2024-09-18T08:06:50.644 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:06:50.644 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-09-18T08:06:50.644 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:06:50.644 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-09-18T08:06:50.644 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:41.154209+0000", 2024-09-18T08:06:50.645 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-18T08:06:50.645 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:06:50.645 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:06:50.645 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-09-18T08:06:50.645 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:41.154209+0000", 2024-09-18T08:06:50.645 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-18T08:06:50.645 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:06:50.645 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:06:50.645 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-09-18T08:06:50.645 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:41.154211+0000", 2024-09-18T08:06:50.645 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.2109999999999999e-06 2024-09-18T08:06:50.646 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:06:50.646 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:06:50.646 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-09-18T08:06:50.646 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:41.154259+0000", 2024-09-18T08:06:50.646 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4.8412000000000003e-05 2024-09-18T08:06:50.646 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:06:50.646 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:06:50.646 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-09-18T08:06:50.646 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:41.154270+0000", 2024-09-18T08:06:50.646 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.0515e-05 2024-09-18T08:06:50.646 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:06:50.647 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:06:50.647 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-09-18T08:06:50.647 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:41.154275+0000", 2024-09-18T08:06:50.647 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 5.4469999999999997e-06 2024-09-18T08:06:50.647 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:06:50.647 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:06:50.647 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-18T08:06:50.647 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:41.154323+0000", 2024-09-18T08:06:50.647 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4.8297999999999997e-05 2024-09-18T08:06:50.647 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:06:50.647 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:06:50.648 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-09-18T08:06:50.648 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:41.154456+0000", 2024-09-18T08:06:50.648 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000132593 2024-09-18T08:06:50.648 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:06:50.648 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:06:50.648 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "waiting for subops from 0,3", 2024-09-18T08:06:50.648 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:41.154615+0000", 2024-09-18T08:06:50.648 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000159049 2024-09-18T08:06:50.648 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:06:50.648 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:06:50.648 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "op_commit", 2024-09-18T08:06:50.648 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:41.156649+0000", 2024-09-18T08:06:50.649 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0020336709999999999 2024-09-18T08:06:50.649 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:06:50.649 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:06:50.649 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:06:50.649 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:41.165163+0000", 2024-09-18T08:06:50.649 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0085141100000000001 2024-09-18T08:06:50.649 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:06:50.649 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:06:50.649 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:06:50.649 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:41.196483+0000", 2024-09-18T08:06:50.649 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.031320300000000002 2024-09-18T08:06:50.650 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:06:50.650 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:06:50.650 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "commit_sent", 2024-09-18T08:06:50.650 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:41.196519+0000", 2024-09-18T08:06:50.650 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 3.6109999999999998e-05 2024-09-18T08:06:50.650 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:06:50.650 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:06:50.650 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-09-18T08:06:50.650 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:41.196548+0000", 2024-09-18T08:06:50.650 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 2.8603e-05 2024-09-18T08:06:50.650 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-18T08:06:50.651 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-09-18T08:06:50.651 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-18T08:06:50.651 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:06:50.651 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:06:50.651 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.14805.0:63908 7.d 7:bd06bac1:::benchmark_data_smithi007_73155_object63907:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e100)", 2024-09-18T08:06:50.651 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-09-18T08:04:41.183375+0000", 2024-09-18T08:06:50.651 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 129.35698876199999, 2024-09-18T08:06:50.651 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.033452764000000003, 2024-09-18T08:06:50.651 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-09-18T08:06:50.651 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-09-18T08:06:50.651 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T08:06:50.652 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2024-09-18T08:06:50.652 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.14805", 2024-09-18T08:06:50.652 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "172.21.15.7:0/640320436", 2024-09-18T08:06:50.652 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 63908 2024-09-18T08:06:50.652 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:06:50.652 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-09-18T08:06:50.652 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:06:50.652 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-09-18T08:06:50.652 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:41.183375+0000", 2024-09-18T08:06:50.652 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-18T08:06:50.652 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:06:50.653 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:06:50.653 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-09-18T08:06:50.653 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:41.183375+0000", 2024-09-18T08:06:50.653 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-18T08:06:50.653 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:06:50.653 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:06:50.653 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-09-18T08:06:50.653 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:41.183376+0000", 2024-09-18T08:06:50.653 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.226e-06 2024-09-18T08:06:50.653 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:06:50.653 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:06:50.653 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-09-18T08:06:50.654 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:41.183403+0000", 2024-09-18T08:06:50.654 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 2.6404e-05 2024-09-18T08:06:50.654 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:06:50.654 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:06:50.654 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-09-18T08:06:50.654 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:41.183417+0000", 2024-09-18T08:06:50.654 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.4399000000000001e-05 2024-09-18T08:06:50.654 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:06:50.654 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:06:50.654 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-09-18T08:06:50.654 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:41.183425+0000", 2024-09-18T08:06:50.654 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 8.1459999999999996e-06 2024-09-18T08:06:50.655 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:06:50.655 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:06:50.655 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-18T08:06:50.655 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:41.183485+0000", 2024-09-18T08:06:50.655 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 5.9673999999999998e-05 2024-09-18T08:06:50.655 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:06:50.655 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:06:50.655 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-09-18T08:06:50.655 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:41.183587+0000", 2024-09-18T08:06:50.655 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000101611 2024-09-18T08:06:50.655 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:06:50.655 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:06:50.655 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "waiting for subops from 0,1", 2024-09-18T08:06:50.656 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:41.183686+0000", 2024-09-18T08:06:50.656 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 9.9823999999999998e-05 2024-09-18T08:06:50.656 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:06:50.656 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:06:50.656 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "op_commit", 2024-09-18T08:06:50.656 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:41.185240+0000", 2024-09-18T08:06:50.656 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0015539340000000001 2024-09-18T08:06:50.656 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:06:50.656 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:06:50.656 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:06:50.656 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:41.191524+0000", 2024-09-18T08:06:50.656 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0062835690000000001 2024-09-18T08:06:50.657 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:06:50.657 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:06:50.657 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:06:50.657 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:41.216746+0000", 2024-09-18T08:06:50.657 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.025222364000000001 2024-09-18T08:06:50.657 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:06:50.657 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:06:50.657 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "commit_sent", 2024-09-18T08:06:50.657 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:41.216782+0000", 2024-09-18T08:06:50.657 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 3.6069999999999999e-05 2024-09-18T08:06:50.657 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:06:50.658 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:06:50.658 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-09-18T08:06:50.658 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:41.216828+0000", 2024-09-18T08:06:50.658 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4.5543000000000001e-05 2024-09-18T08:06:50.658 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-18T08:06:50.658 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-09-18T08:06:50.658 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-18T08:06:50.658 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:06:50.658 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:06:50.658 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.14805.0:63960 7.c 7:30cc2060:::benchmark_data_smithi007_73155_object63959:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e100)", 2024-09-18T08:06:50.658 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-09-18T08:04:41.252452+0000", 2024-09-18T08:06:50.658 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 129.28791203599999, 2024-09-18T08:06:50.659 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.027634619999999999, 2024-09-18T08:06:50.659 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-09-18T08:06:50.659 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-09-18T08:06:50.659 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T08:06:50.659 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2024-09-18T08:06:50.659 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.14805", 2024-09-18T08:06:50.659 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "172.21.15.7:0/640320436", 2024-09-18T08:06:50.659 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 63960 2024-09-18T08:06:50.659 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:06:50.659 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-09-18T08:06:50.659 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:06:50.659 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-09-18T08:06:50.659 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:41.252452+0000", 2024-09-18T08:06:50.660 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-18T08:06:50.660 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:06:50.660 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:06:50.660 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-09-18T08:06:50.660 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:41.252452+0000", 2024-09-18T08:06:50.660 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-18T08:06:50.660 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:06:50.660 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:06:50.660 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-09-18T08:06:50.660 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:41.252453+0000", 2024-09-18T08:06:50.660 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.063e-06 2024-09-18T08:06:50.660 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:06:50.660 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:06:50.661 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7 2024-09-18T08:06:50.662 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-09-18T08:06:50.662 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:41.252503+0000", 2024-09-18T08:06:50.662 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 5.024e-05 2024-09-18T08:06:50.662 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:06:50.662 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:06:50.662 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-09-18T08:06:50.662 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:41.252513+0000", 2024-09-18T08:06:50.663 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.0137e-05 2024-09-18T08:06:50.663 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:06:50.663 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:06:50.663 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-09-18T08:06:50.663 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:41.252519+0000", 2024-09-18T08:06:50.663 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 5.2630000000000003e-06 2024-09-18T08:06:50.663 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:06:50.663 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:06:50.663 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-18T08:06:50.663 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:41.252564+0000", 2024-09-18T08:06:50.663 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4.5528999999999997e-05 2024-09-18T08:06:50.663 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:06:50.664 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:06:50.664 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-09-18T08:06:50.664 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:41.252645+0000", 2024-09-18T08:06:50.664 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 8.1118000000000005e-05 2024-09-18T08:06:50.664 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:06:50.664 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:06:50.664 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "waiting for subops from 3,7", 2024-09-18T08:06:50.664 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:41.252726+0000", 2024-09-18T08:06:50.664 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 8.0912000000000001e-05 2024-09-18T08:06:50.664 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:06:50.664 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:06:50.665 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "op_commit", 2024-09-18T08:06:50.665 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:41.254200+0000", 2024-09-18T08:06:50.665 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0014738010000000001 2024-09-18T08:06:50.665 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:06:50.665 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:06:50.665 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:06:50.665 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:41.254525+0000", 2024-09-18T08:06:50.665 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00032529599999999998 2024-09-18T08:06:50.665 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:06:50.665 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:06:50.665 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:06:50.665 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:41.280019+0000", 2024-09-18T08:06:50.666 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.025493512999999999 2024-09-18T08:06:50.666 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:06:50.666 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:06:50.666 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "commit_sent", 2024-09-18T08:06:50.666 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:41.280057+0000", 2024-09-18T08:06:50.666 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 3.8287000000000001e-05 2024-09-18T08:06:50.666 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:06:50.666 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:06:50.666 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-09-18T08:06:50.666 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:41.280086+0000", 2024-09-18T08:06:50.666 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 2.9461000000000001e-05 2024-09-18T08:06:50.667 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-18T08:06:50.667 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-09-18T08:06:50.667 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-18T08:06:50.667 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:06:50.667 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:06:50.667 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.14805.0:65051 7.d 7:bcc5331c:::benchmark_data_smithi007_73155_object65050:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e100)", 2024-09-18T08:06:50.667 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-09-18T08:04:42.227551+0000", 2024-09-18T08:06:50.667 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 128.312813275, 2024-09-18T08:06:50.667 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.057084449000000002, 2024-09-18T08:06:50.667 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-09-18T08:06:50.667 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-09-18T08:06:50.668 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "waiting for sub ops", 2024-09-18T08:06:50.668 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2024-09-18T08:06:50.668 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.14805", 2024-09-18T08:06:50.668 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "172.21.15.7:0/640320436", 2024-09-18T08:06:50.668 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 65051 2024-09-18T08:06:50.668 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:06:50.668 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-09-18T08:06:50.668 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:06:50.668 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-09-18T08:06:50.668 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:42.227551+0000", 2024-09-18T08:06:50.668 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-18T08:06:50.669 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:06:50.669 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:06:50.669 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-09-18T08:06:50.669 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:42.227551+0000", 2024-09-18T08:06:50.669 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-18T08:06:50.669 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:06:50.669 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:06:50.669 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-09-18T08:06:50.669 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:42.227552+0000", 2024-09-18T08:06:50.669 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.5319999999999999e-06 2024-09-18T08:06:50.669 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:06:50.670 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:06:50.670 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-09-18T08:06:50.670 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:42.227606+0000", 2024-09-18T08:06:50.670 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 5.4290999999999999e-05 2024-09-18T08:06:50.670 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:06:50.670 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:06:50.670 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-09-18T08:06:50.670 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:42.227619+0000", 2024-09-18T08:06:50.670 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.2937e-05 2024-09-18T08:06:50.670 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:06:50.670 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:06:50.670 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-09-18T08:06:50.670 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:42.227627+0000", 2024-09-18T08:06:50.671 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 7.9340000000000006e-06 2024-09-18T08:06:50.671 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:06:50.671 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:06:50.671 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-18T08:06:50.671 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:42.227681+0000", 2024-09-18T08:06:50.671 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 5.3771999999999999e-05 2024-09-18T08:06:50.671 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:06:50.671 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:06:50.671 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-09-18T08:06:50.671 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:42.227809+0000", 2024-09-18T08:06:50.671 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00012750399999999999 2024-09-18T08:06:50.671 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:06:50.672 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:06:50.672 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "waiting for subops from 0,1", 2024-09-18T08:06:50.672 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:42.227961+0000", 2024-09-18T08:06:50.672 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00015280599999999999 2024-09-18T08:06:50.672 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:06:50.672 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:06:50.672 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "op_commit", 2024-09-18T08:06:50.672 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:42.229971+0000", 2024-09-18T08:06:50.672 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0020099179999999999 2024-09-18T08:06:50.672 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:06:50.672 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:06:50.673 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:06:50.673 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:42.232646+0000", 2024-09-18T08:06:50.673 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0026748359999999999 2024-09-18T08:06:50.673 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:06:50.673 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:06:50.673 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-09-18T08:06:50.673 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:42.284635+0000", 2024-09-18T08:06:50.673 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.051988919000000001 2024-09-18T08:06:50.673 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-18T08:06:50.673 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-09-18T08:06:50.674 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-18T08:06:50.674 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:06:50.674 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:06:50.674 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.14805.0:65057 7.d 7:bffbe46c:::benchmark_data_smithi007_73155_object65056:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e100)", 2024-09-18T08:06:50.674 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-09-18T08:04:42.229929+0000", 2024-09-18T08:06:50.674 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 128.310434579, 2024-09-18T08:06:50.674 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.054704183000000003, 2024-09-18T08:06:50.674 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-09-18T08:06:50.674 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-09-18T08:06:50.674 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "waiting for sub ops", 2024-09-18T08:06:50.674 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2024-09-18T08:06:50.674 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.14805", 2024-09-18T08:06:50.674 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "172.21.15.7:0/640320436", 2024-09-18T08:06:50.675 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 65057 2024-09-18T08:06:50.675 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:06:50.675 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-09-18T08:06:50.675 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:06:50.675 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-09-18T08:06:50.675 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:42.229929+0000", 2024-09-18T08:06:50.675 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-18T08:06:50.675 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:06:50.675 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:06:50.675 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-09-18T08:06:50.675 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:42.229929+0000", 2024-09-18T08:06:50.675 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-18T08:06:50.676 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:06:50.676 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:06:50.676 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-09-18T08:06:50.676 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:42.229930+0000", 2024-09-18T08:06:50.676 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.0219999999999999e-06 2024-09-18T08:06:50.676 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:06:50.676 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:06:50.676 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-09-18T08:06:50.676 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:42.229973+0000", 2024-09-18T08:06:50.676 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4.2904000000000001e-05 2024-09-18T08:06:50.676 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:06:50.677 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:06:50.677 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-09-18T08:06:50.677 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:42.229987+0000", 2024-09-18T08:06:50.677 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.3405e-05 2024-09-18T08:06:50.677 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:06:50.677 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:06:50.677 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-09-18T08:06:50.677 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:42.229996+0000", 2024-09-18T08:06:50.677 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 9.6299999999999993e-06 2024-09-18T08:06:50.677 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:06:50.677 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:06:50.678 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-18T08:06:50.678 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:42.230048+0000", 2024-09-18T08:06:50.678 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 5.1839000000000003e-05 2024-09-18T08:06:50.678 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:06:50.678 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:06:50.678 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-09-18T08:06:50.678 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:42.230146+0000", 2024-09-18T08:06:50.678 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 9.8101000000000001e-05 2024-09-18T08:06:50.678 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:06:50.678 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:06:50.678 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "waiting for subops from 0,1", 2024-09-18T08:06:50.679 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:42.230237+0000", 2024-09-18T08:06:50.679 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 9.0634e-05 2024-09-18T08:06:50.679 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:06:50.679 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:06:50.679 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "op_commit", 2024-09-18T08:06:50.679 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:42.231752+0000", 2024-09-18T08:06:50.679 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.001515548 2024-09-18T08:06:50.679 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:06:50.679 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:06:50.679 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:06:50.679 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:42.232681+0000", 2024-09-18T08:06:50.679 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00092884 2024-09-18T08:06:50.680 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:06:50.680 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:06:50.680 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-09-18T08:06:50.680 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:42.284634+0000", 2024-09-18T08:06:50.680 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.05195226 2024-09-18T08:06:50.680 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-18T08:06:50.680 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-09-18T08:06:50.680 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-18T08:06:50.680 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:06:50.680 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:06:50.680 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.14805.0:65062 7.d 7:b1e686a2:::benchmark_data_smithi007_73155_object65061:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e100)", 2024-09-18T08:06:50.680 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-09-18T08:04:42.236979+0000", 2024-09-18T08:06:50.681 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 128.30338526400001, 2024-09-18T08:06:50.681 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.047653305999999999, 2024-09-18T08:06:50.681 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-09-18T08:06:50.681 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-09-18T08:06:50.681 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "waiting for sub ops", 2024-09-18T08:06:50.681 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2024-09-18T08:06:50.681 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.14805", 2024-09-18T08:06:50.681 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "172.21.15.7:0/640320436", 2024-09-18T08:06:50.681 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 65062 2024-09-18T08:06:50.681 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:06:50.681 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-09-18T08:06:50.681 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:06:50.682 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-09-18T08:06:50.682 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:42.236979+0000", 2024-09-18T08:06:50.682 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-18T08:06:50.682 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:06:50.682 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:06:50.682 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-09-18T08:06:50.682 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:42.236979+0000", 2024-09-18T08:06:50.682 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-18T08:06:50.682 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:06:50.682 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:06:50.682 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-09-18T08:06:50.682 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:42.236980+0000", 2024-09-18T08:06:50.682 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.124e-06 2024-09-18T08:06:50.683 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:06:50.683 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:06:50.683 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-09-18T08:06:50.683 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:42.237029+0000", 2024-09-18T08:06:50.683 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4.9450999999999998e-05 2024-09-18T08:06:50.683 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:06:50.683 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:06:50.683 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-09-18T08:06:50.683 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:42.237041+0000", 2024-09-18T08:06:50.683 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.1418e-05 2024-09-18T08:06:50.683 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:06:50.683 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:06:50.683 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-09-18T08:06:50.684 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:42.237049+0000", 2024-09-18T08:06:50.684 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 8.0059999999999992e-06 2024-09-18T08:06:50.684 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:06:50.684 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:06:50.684 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-18T08:06:50.684 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:42.237104+0000", 2024-09-18T08:06:50.684 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 5.4870999999999997e-05 2024-09-18T08:06:50.684 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:06:50.684 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:06:50.684 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-09-18T08:06:50.684 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:42.237258+0000", 2024-09-18T08:06:50.684 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000154857 2024-09-18T08:06:50.685 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:06:50.685 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:06:50.685 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "waiting for subops from 0,1", 2024-09-18T08:06:50.685 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:42.237413+0000", 2024-09-18T08:06:50.685 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000154764 2024-09-18T08:06:50.685 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:06:50.685 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:06:50.685 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "op_commit", 2024-09-18T08:06:50.685 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:42.240054+0000", 2024-09-18T08:06:50.685 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0026404549999999999 2024-09-18T08:06:50.685 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:06:50.686 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:06:50.686 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-09-18T08:06:50.686 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:42.284632+0000", 2024-09-18T08:06:50.686 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.044578359999999997 2024-09-18T08:06:50.686 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-18T08:06:50.686 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-09-18T08:06:50.686 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-18T08:06:50.686 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:06:50.686 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:06:50.686 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.14805.0:65063 7.d 7:b037f784:::benchmark_data_smithi007_73155_object65062:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e100)", 2024-09-18T08:06:50.687 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-09-18T08:04:42.237116+0000", 2024-09-18T08:06:50.687 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 128.303247798, 2024-09-18T08:06:50.687 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.047512190000000003, 2024-09-18T08:06:50.687 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-09-18T08:06:50.687 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-09-18T08:06:50.687 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "waiting for sub ops", 2024-09-18T08:06:50.687 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2024-09-18T08:06:50.687 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.14805", 2024-09-18T08:06:50.687 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "172.21.15.7:0/640320436", 2024-09-18T08:06:50.687 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 65063 2024-09-18T08:06:50.687 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:06:50.688 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-09-18T08:06:50.688 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:06:50.688 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-09-18T08:06:50.688 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:42.237116+0000", 2024-09-18T08:06:50.688 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-18T08:06:50.688 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:06:50.688 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:06:50.688 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-09-18T08:06:50.688 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:42.237116+0000", 2024-09-18T08:06:50.688 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-18T08:06:50.688 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:06:50.689 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:06:50.689 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-09-18T08:06:50.689 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:42.237117+0000", 2024-09-18T08:06:50.689 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 6.7000000000000004e-07 2024-09-18T08:06:50.689 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:06:50.689 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:06:50.689 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-09-18T08:06:50.689 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:42.237190+0000", 2024-09-18T08:06:50.689 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 7.2972000000000005e-05 2024-09-18T08:06:50.689 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:06:50.689 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:06:50.689 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-09-18T08:06:50.690 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:42.237197+0000", 2024-09-18T08:06:50.690 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 7.2729999999999997e-06 2024-09-18T08:06:50.690 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:06:50.690 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:06:50.690 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-09-18T08:06:50.690 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:42.237203+0000", 2024-09-18T08:06:50.690 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 5.5289999999999999e-06 2024-09-18T08:06:50.690 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:06:50.690 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:06:50.690 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-18T08:06:50.690 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:42.238960+0000", 2024-09-18T08:06:50.691 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.001757759 2024-09-18T08:06:50.691 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:06:50.691 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:06:50.691 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-09-18T08:06:50.691 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:42.239038+0000", 2024-09-18T08:06:50.691 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 7.8054999999999997e-05 2024-09-18T08:06:50.691 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:06:50.691 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:06:50.691 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "waiting for subops from 0,1", 2024-09-18T08:06:50.691 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:42.239122+0000", 2024-09-18T08:06:50.692 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 8.3963999999999998e-05 2024-09-18T08:06:50.692 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:06:50.692 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:06:50.692 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "op_commit", 2024-09-18T08:06:50.692 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:42.240157+0000", 2024-09-18T08:06:50.692 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.001034372 2024-09-18T08:06:50.692 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:06:50.692 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:06:50.692 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-09-18T08:06:50.692 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:42.284628+0000", 2024-09-18T08:06:50.692 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.044471596000000002 2024-09-18T08:06:50.692 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-18T08:06:50.693 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-09-18T08:06:50.693 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-18T08:06:50.693 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-18T08:06:50.693 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-09-18T08:06:50.693 INFO:teuthology.orchestra.run.smithi050.stdout:} 2024-09-18T08:06:50.721 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:06:50.721 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:06:50.722 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7' 2024-09-18T08:06:50.730 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:06:50 smithi007 ceph-mon[29459]: 7.f scrub starts 2024-09-18T08:06:50.730 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:06:50 smithi007 ceph-mon[29459]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-18T08:06:50.731 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:06:50 smithi007 ceph-mon[29459]: pgmap v180: 148 pgs: 148 active+clean; 3.1 GiB data, 10 GiB used, 705 GiB / 715 GiB avail; 0 B/s wr, 1.32k op/s 2024-09-18T08:06:50.731 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:06:50 smithi007 ceph-mon[39249]: 7.f scrub starts 2024-09-18T08:06:50.731 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:06:50 smithi007 ceph-mon[39249]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-18T08:06:50.731 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:06:50 smithi007 ceph-mon[39249]: pgmap v180: 148 pgs: 148 active+clean; 3.1 GiB data, 10 GiB used, 705 GiB / 715 GiB avail; 0 B/s wr, 1.32k op/s 2024-09-18T08:06:50.822 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7 2024-09-18T08:06:50.847 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:06:50.848 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 7, 6] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-09-18T08:06:50.848 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-18T08:06:50.848 INFO:tasks.thrashosds.thrasher:fixing pg num pool .mgr 2024-09-18T08:06:50.848 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid 779349ca-7593-11ef-bceb-c7b262605968 -- ceph pg dump --format=json 2024-09-18T08:06:50.849 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:06:50.850 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7' 2024-09-18T08:06:50.904 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:06:50 smithi050 ceph-mon[32371]: 7.f scrub starts 2024-09-18T08:06:50.904 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:06:50 smithi050 ceph-mon[32371]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-18T08:06:50.904 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:06:50 smithi050 ceph-mon[32371]: pgmap v180: 148 pgs: 148 active+clean; 3.1 GiB data, 10 GiB used, 705 GiB / 715 GiB avail; 0 B/s wr, 1.32k op/s 2024-09-18T08:06:50.950 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7 2024-09-18T08:06:50.976 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:06:50.977 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:06:50.977 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7' 2024-09-18T08:06:51.074 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid 779349ca-7593-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-18T08:06:51.078 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0 2024-09-18T08:06:51.169 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:06:51.170 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:06:51.170 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0' 2024-09-18T08:06:51.271 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1 2024-09-18T08:06:51.321 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:06:51.322 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:06:51.322 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1' 2024-09-18T08:06:51.423 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7 2024-09-18T08:06:51.449 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:06:51.450 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:06:51.450 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7' 2024-09-18T08:06:51.540 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:06:51 smithi007 ceph-mon[29459]: 7.4 scrub starts 2024-09-18T08:06:51.541 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:06:51 smithi007 ceph-mon[29459]: 7.f scrub ok 2024-09-18T08:06:51.543 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:06:51 smithi007 ceph-mon[39249]: 7.4 scrub starts 2024-09-18T08:06:51.543 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:06:51 smithi007 ceph-mon[39249]: 7.f scrub ok 2024-09-18T08:06:51.551 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0 2024-09-18T08:06:51.563 DEBUG:teuthology.orchestra.run.smithi050:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid 779349ca-7593-11ef-bceb-c7b262605968 -- ceph daemon osd.7 dump_ops_in_flight 2024-09-18T08:06:51.608 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:06:51.609 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:06:51.609 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0' 2024-09-18T08:06:51.710 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5 2024-09-18T08:06:51.740 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:06:51.741 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:06:51.741 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5' 2024-09-18T08:06:51.799 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/779349ca-7593-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T08:06:51.803 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/779349ca-7593-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T08:06:51.842 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7 2024-09-18T08:06:51.869 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:06:51.870 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:06:51.870 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7' 2024-09-18T08:06:51.883 INFO:teuthology.orchestra.run.smithi050.stderr:Inferring config /var/lib/ceph/779349ca-7593-11ef-bceb-c7b262605968/mon.b/config 2024-09-18T08:06:51.905 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:06:51 smithi050 ceph-mon[32371]: 7.4 scrub starts 2024-09-18T08:06:51.906 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:06:51 smithi050 ceph-mon[32371]: 7.f scrub ok 2024-09-18T08:06:51.971 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5 2024-09-18T08:06:51.998 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:06:51.998 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:06:51.999 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5' 2024-09-18T08:06:52.099 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0 2024-09-18T08:06:52.139 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:06:52.139 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:06:52.140 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0' 2024-09-18T08:06:52.241 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2 2024-09-18T08:06:52.280 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:06:52.281 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:06:52.281 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2' 2024-09-18T08:06:52.382 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7 2024-09-18T08:06:52.442 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:06:52.442 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:06:52.443 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7' 2024-09-18T08:06:52.493 INFO:teuthology.orchestra.run.smithi007.stderr:noscrub is set 2024-09-18T08:06:52.543 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1 2024-09-18T08:06:52.595 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:06:52.596 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:06:52.596 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1' 2024-09-18T08:06:52.614 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:06:52 smithi050 ceph-mon[32371]: 6.7 scrub starts 2024-09-18T08:06:52.614 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:06:52 smithi050 ceph-mon[32371]: 6.7 scrub ok 2024-09-18T08:06:52.614 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:06:52 smithi050 ceph-mon[32371]: 6.8 scrub starts 2024-09-18T08:06:52.614 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:06:52 smithi050 ceph-mon[32371]: 6.8 scrub ok 2024-09-18T08:06:52.614 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:06:52 smithi050 ceph-mon[32371]: 7.4 scrub ok 2024-09-18T08:06:52.614 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:06:52 smithi050 ceph-mon[32371]: pgmap v181: 148 pgs: 148 active+clean; 3.1 GiB data, 10 GiB used, 705 GiB / 715 GiB avail; 0 B/s wr, 991 op/s 2024-09-18T08:06:52.614 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:06:52 smithi050 ceph-mon[32371]: 2.0 scrub starts 2024-09-18T08:06:52.615 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:06:52 smithi050 ceph-mon[32371]: 2.0 scrub ok 2024-09-18T08:06:52.615 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:06:52 smithi050 ceph-mon[32371]: from='client.? 172.21.15.7:0/3004428757' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-18T08:06:52.615 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:06:52 smithi050 ceph-mon[32371]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-18T08:06:52.630 INFO:teuthology.orchestra.run.smithi007.stdout:osd.0: {} 2024-09-18T08:06:52.631 INFO:teuthology.orchestra.run.smithi007.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-18T08:06:52.638 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2024-09-18T08:06:52.638 INFO:teuthology.orchestra.run.smithi050.stdout: "ops": [ 2024-09-18T08:06:52.638 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:06:52.638 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_repop(client.14805.0:190465 7.c e145/137 7:3def6241:::benchmark_data_smithi007_73155_object72174:head v 145'12076, mlcod=145'12074)", 2024-09-18T08:06:52.638 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-09-18T08:06:52.635186+0000", 2024-09-18T08:06:52.639 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 0.0010279, 2024-09-18T08:06:52.639 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0010463530000000001, 2024-09-18T08:06:52.639 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-09-18T08:06:52.639 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-09-18T08:06:52.639 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "started", 2024-09-18T08:06:52.639 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-09-18T08:06:52.639 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:06:52.639 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-09-18T08:06:52.639 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:06:52.635186+0000", 2024-09-18T08:06:52.639 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-18T08:06:52.639 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:06:52.639 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:06:52.639 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-09-18T08:06:52.640 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:06:52.635186+0000", 2024-09-18T08:06:52.640 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-18T08:06:52.640 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:06:52.640 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:06:52.640 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-09-18T08:06:52.640 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:06:52.635187+0000", 2024-09-18T08:06:52.640 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 9.2099999999999995e-07 2024-09-18T08:06:52.640 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:06:52.640 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:06:52.640 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-09-18T08:06:52.640 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:06:52.635194+0000", 2024-09-18T08:06:52.640 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 7.3599999999999998e-06 2024-09-18T08:06:52.640 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:06:52.640 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:06:52.641 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-09-18T08:06:52.641 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:06:52.635211+0000", 2024-09-18T08:06:52.641 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.6778000000000001e-05 2024-09-18T08:06:52.641 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:06:52.641 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:06:52.641 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-09-18T08:06:52.641 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:06:52.635219+0000", 2024-09-18T08:06:52.641 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 8.1459999999999996e-06 2024-09-18T08:06:52.641 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:06:52.641 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:06:52.641 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-18T08:06:52.641 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:06:52.635294+0000", 2024-09-18T08:06:52.641 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 7.5338000000000003e-05 2024-09-18T08:06:52.641 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:06:52.641 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:06:52.642 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-09-18T08:06:52.642 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:06:52.635316+0000", 2024-09-18T08:06:52.642 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 2.1350999999999999e-05 2024-09-18T08:06:52.642 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-18T08:06:52.642 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-09-18T08:06:52.642 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-18T08:06:52.642 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-18T08:06:52.642 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2024-09-18T08:06:52.642 INFO:teuthology.orchestra.run.smithi050.stdout: "num_ops": 1 2024-09-18T08:06:52.642 INFO:teuthology.orchestra.run.smithi050.stdout:} 2024-09-18T08:06:52.664 INFO:teuthology.orchestra.run.smithi007.stdout:osd.1: {} 2024-09-18T08:06:52.664 INFO:teuthology.orchestra.run.smithi007.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-18T08:06:52.697 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5 2024-09-18T08:06:52.699 INFO:teuthology.orchestra.run.smithi007.stdout:osd.2: {} 2024-09-18T08:06:52.699 INFO:teuthology.orchestra.run.smithi007.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-18T08:06:52.725 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:06:52.725 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:06:52.726 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5' 2024-09-18T08:06:52.742 INFO:teuthology.orchestra.run.smithi007.stdout:osd.3: {} 2024-09-18T08:06:52.742 INFO:teuthology.orchestra.run.smithi007.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-18T08:06:52.743 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:06:52 smithi007 ceph-mon[29459]: 6.7 scrub starts 2024-09-18T08:06:52.743 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:06:52 smithi007 ceph-mon[29459]: 6.7 scrub ok 2024-09-18T08:06:52.743 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:06:52 smithi007 ceph-mon[29459]: 6.8 scrub starts 2024-09-18T08:06:52.743 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:06:52 smithi007 ceph-mon[29459]: 6.8 scrub ok 2024-09-18T08:06:52.743 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:06:52 smithi007 ceph-mon[29459]: 7.4 scrub ok 2024-09-18T08:06:52.743 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:06:52 smithi007 ceph-mon[29459]: pgmap v181: 148 pgs: 148 active+clean; 3.1 GiB data, 10 GiB used, 705 GiB / 715 GiB avail; 0 B/s wr, 991 op/s 2024-09-18T08:06:52.743 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:06:52 smithi007 ceph-mon[29459]: 2.0 scrub starts 2024-09-18T08:06:52.743 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:06:52 smithi007 ceph-mon[29459]: 2.0 scrub ok 2024-09-18T08:06:52.743 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:06:52 smithi007 ceph-mon[29459]: from='client.? 172.21.15.7:0/3004428757' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-18T08:06:52.743 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:06:52 smithi007 ceph-mon[29459]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-18T08:06:52.744 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:06:52 smithi007 ceph-mon[39249]: 6.7 scrub starts 2024-09-18T08:06:52.744 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:06:52 smithi007 ceph-mon[39249]: 6.7 scrub ok 2024-09-18T08:06:52.744 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:06:52 smithi007 ceph-mon[39249]: 6.8 scrub starts 2024-09-18T08:06:52.744 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:06:52 smithi007 ceph-mon[39249]: 6.8 scrub ok 2024-09-18T08:06:52.744 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:06:52 smithi007 ceph-mon[39249]: 7.4 scrub ok 2024-09-18T08:06:52.744 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:06:52 smithi007 ceph-mon[39249]: pgmap v181: 148 pgs: 148 active+clean; 3.1 GiB data, 10 GiB used, 705 GiB / 715 GiB avail; 0 B/s wr, 991 op/s 2024-09-18T08:06:52.744 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:06:52 smithi007 ceph-mon[39249]: 2.0 scrub starts 2024-09-18T08:06:52.744 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:06:52 smithi007 ceph-mon[39249]: 2.0 scrub ok 2024-09-18T08:06:52.744 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:06:52 smithi007 ceph-mon[39249]: from='client.? 172.21.15.7:0/3004428757' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-18T08:06:52.745 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:06:52 smithi007 ceph-mon[39249]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-18T08:06:52.778 INFO:teuthology.orchestra.run.smithi007.stdout:osd.4: {} 2024-09-18T08:06:52.778 INFO:teuthology.orchestra.run.smithi007.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-18T08:06:52.813 INFO:teuthology.orchestra.run.smithi007.stdout:osd.5: {} 2024-09-18T08:06:52.813 INFO:teuthology.orchestra.run.smithi007.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-18T08:06:52.826 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6 2024-09-18T08:06:52.851 INFO:teuthology.orchestra.run.smithi007.stdout:osd.6: {} 2024-09-18T08:06:52.851 INFO:teuthology.orchestra.run.smithi007.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-18T08:06:52.854 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:06:52.855 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:06:52.855 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6' 2024-09-18T08:06:52.886 INFO:teuthology.orchestra.run.smithi007.stdout:osd.7: {} 2024-09-18T08:06:52.886 INFO:teuthology.orchestra.run.smithi007.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-18T08:06:52.957 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0 2024-09-18T08:06:53.004 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:06:53.005 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:06:53.005 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0' 2024-09-18T08:06:53.106 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0 2024-09-18T08:06:53.143 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:06:53.144 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:06:53.144 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0' 2024-09-18T08:06:53.245 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1 2024-09-18T08:06:53.289 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:06:53.290 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:06:53.290 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1' 2024-09-18T08:06:53.391 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3 2024-09-18T08:06:53.437 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:06:53.438 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:06:53.438 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3' 2024-09-18T08:06:53.467 DEBUG:teuthology.orchestra.run.smithi050:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid 779349ca-7593-11ef-bceb-c7b262605968 -- ceph daemon osd.7 dump_blocked_ops 2024-09-18T08:06:53.539 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4 2024-09-18T08:06:53.572 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:06:53.573 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:06:53.573 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4' 2024-09-18T08:06:53.674 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2 2024-09-18T08:06:53.708 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:06:53.709 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:06:53.709 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2' 2024-09-18T08:06:53.798 INFO:teuthology.orchestra.run.smithi050.stderr:Inferring config /var/lib/ceph/779349ca-7593-11ef-bceb-c7b262605968/mon.b/config 2024-09-18T08:06:53.809 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1 2024-09-18T08:06:53.849 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:06:53.850 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:06:53.850 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1' 2024-09-18T08:06:53.906 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:06:53 smithi050 ceph-mon[32371]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-18T08:06:53.906 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:06:53 smithi050 ceph-mon[32371]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-18T08:06:53.906 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:06:53 smithi050 ceph-mon[32371]: osdmap e145: 8 total, 8 up, 8 in 2024-09-18T08:06:53.906 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:06:53 smithi050 ceph-mon[32371]: from='client.? 172.21.15.7:0/2138053719' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-18T08:06:53.951 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2 2024-09-18T08:06:53.980 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:06:53 smithi007 ceph-mon[29459]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-18T08:06:53.980 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:06:53 smithi007 ceph-mon[29459]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-18T08:06:53.980 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:06:53 smithi007 ceph-mon[29459]: osdmap e145: 8 total, 8 up, 8 in 2024-09-18T08:06:53.981 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:06:53 smithi007 ceph-mon[29459]: from='client.? 172.21.15.7:0/2138053719' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-18T08:06:53.981 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:06:53 smithi007 ceph-mon[39249]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-18T08:06:53.981 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:06:53 smithi007 ceph-mon[39249]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-18T08:06:53.982 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:06:53 smithi007 ceph-mon[39249]: osdmap e145: 8 total, 8 up, 8 in 2024-09-18T08:06:53.982 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:06:53 smithi007 ceph-mon[39249]: from='client.? 172.21.15.7:0/2138053719' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-18T08:06:53.990 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:06:53.991 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:06:53.991 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2' 2024-09-18T08:06:54.092 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1 2024-09-18T08:06:54.133 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:06:54.133 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:06:54.134 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1' 2024-09-18T08:06:54.235 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3 2024-09-18T08:06:54.274 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:06:54.274 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:06:54.275 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3' 2024-09-18T08:06:54.342 INFO:teuthology.orchestra.run.smithi050.stdout:ERROR: (22) Invalid argument 2024-09-18T08:06:54.342 INFO:teuthology.orchestra.run.smithi050.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-18T08:06:54.376 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6 2024-09-18T08:06:54.407 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:06:54.408 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:06:54.408 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6' 2024-09-18T08:06:54.419 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-09-18T08:06:54.421 INFO:teuthology.orchestra.run.smithi007.stderr:dumped all 2024-09-18T08:06:54.509 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1 2024-09-18T08:06:54.551 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:06:54.552 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:06:54.552 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1' 2024-09-18T08:06:54.653 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4 2024-09-18T08:06:54.680 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:06:54.680 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:06:54.681 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4' 2024-09-18T08:06:54.730 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:06:54 smithi007 ceph-mon[29459]: pgmap v183: 148 pgs: 148 active+clean; 3.0 GiB data, 9.8 GiB used, 705 GiB / 715 GiB avail; 0 B/s wr, 564 op/s 2024-09-18T08:06:54.730 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:06:54 smithi007 ceph-mon[29459]: from='client.24920 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-18T08:06:54.731 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:06:54 smithi007 ceph-mon[39249]: pgmap v183: 148 pgs: 148 active+clean; 3.0 GiB data, 9.8 GiB used, 705 GiB / 715 GiB avail; 0 B/s wr, 564 op/s 2024-09-18T08:06:54.731 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:06:54 smithi007 ceph-mon[39249]: from='client.24920 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-18T08:06:54.781 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6 2024-09-18T08:06:54.811 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:06:54.811 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:06:54.812 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6' 2024-09-18T08:06:54.906 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:06:54 smithi050 ceph-mon[32371]: pgmap v183: 148 pgs: 148 active+clean; 3.0 GiB data, 9.8 GiB used, 705 GiB / 715 GiB avail; 0 B/s wr, 564 op/s 2024-09-18T08:06:54.906 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:06:54 smithi050 ceph-mon[32371]: from='client.24920 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-18T08:06:54.912 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1 2024-09-18T08:06:54.952 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:06:54.953 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:06:54.953 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1' 2024-09-18T08:06:55.054 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7 2024-09-18T08:06:55.082 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:06:55.083 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:06:55.083 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7' 2024-09-18T08:06:55.182 DEBUG:teuthology.orchestra.run.smithi050:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid 779349ca-7593-11ef-bceb-c7b262605968 -- ceph daemon osd.7 dump_historic_ops 2024-09-18T08:06:55.183 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1 2024-09-18T08:06:55.225 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:06:55.226 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:06:55.226 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1' 2024-09-18T08:06:55.327 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2 2024-09-18T08:06:55.365 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:06:55.366 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:06:55.366 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2' 2024-09-18T08:06:55.467 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0 2024-09-18T08:06:55.505 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:06:55.506 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:06:55.506 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0' 2024-09-18T08:06:55.515 INFO:teuthology.orchestra.run.smithi050.stderr:Inferring config /var/lib/ceph/779349ca-7593-11ef-bceb-c7b262605968/mon.b/config 2024-09-18T08:06:55.607 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5 2024-09-18T08:06:55.635 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:06:55.635 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:06:55.636 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5' 2024-09-18T08:06:55.737 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2 2024-09-18T08:06:55.777 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:06:55.778 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:06:55.778 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2' 2024-09-18T08:06:55.879 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7 2024-09-18T08:06:55.908 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:06:55.909 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:06:55.909 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7' 2024-09-18T08:06:56.010 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2 2024-09-18T08:06:56.048 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:06:56.049 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:06:56.049 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2' 2024-09-18T08:06:56.133 INFO:teuthology.orchestra.run.smithi050.stdout:ERROR: (22) Invalid argument 2024-09-18T08:06:56.133 INFO:teuthology.orchestra.run.smithi050.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-18T08:06:56.150 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1 2024-09-18T08:06:56.188 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:06:56.189 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:06:56.189 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1' 2024-09-18T08:06:56.290 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5 2024-09-18T08:06:56.319 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:06:56.320 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:06:56.320 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5' 2024-09-18T08:06:56.420 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3 2024-09-18T08:06:56.459 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:06:56.460 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:06:56.460 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3' 2024-09-18T08:06:56.500 INFO:teuthology.orchestra.run.smithi007.stdout:{"pg_ready":true,"pg_map":{"version":183,"stamp":"2024-09-18T08:06:53.146556+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":3253934643,"num_objects":49842,"num_object_clones":0,"num_object_copies":149526,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":49842,"num_whiteouts":0,"num_read":854,"num_read_kb":711,"num_write":187484,"num_write_kb":7572106,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2792,"num_bytes_recovered":173871186,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":87046,"ondisk_log_size":87046,"up":444,"acting":444,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":445,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":10264968,"kb_used_data":9403192,"kb_used_omap":12,"kb_used_meta":861555,"kb_avail":739696248,"statfs":{"total":767960285184,"available":757448957952,"internally_reserved":0,"allocated":9628868608,"data_stored":9625802449,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12750,"internal_metadata":882232882},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,2,2],"upper_bound":8},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-296222720,"num_objects":-4520,"num_object_clones":0,"num_object_copies":-13560,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-4520,"num_whiteouts":0,"num_read":-221,"num_read_kb":-221,"num_write":4520,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.002353"},"pg_stats":[{"pgid":"3.1f","version":"0'0","reported_seq":185,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-09-18T08:06:46.139242+0000","last_change":"2024-09-18T08:04:34.226236+0000","last_active":"2024-09-18T08:06:46.139242+0000","last_peered":"2024-09-18T08:06:46.139242+0000","last_clean":"2024-09-18T08:06:46.139242+0000","last_became_active":"2024-09-18T08:02:47.877698+0000","last_became_peered":"2024-09-18T08:02:47.877698+0000","last_unstale":"2024-09-18T08:06:46.139242+0000","last_undegraded":"2024-09-18T08:06:46.139242+0000","last_fullsized":"2024-09-18T08:06:46.139242+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:04:34.226054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:46.860092+0000","last_clean_scrub_stamp":"2024-09-18T08:04:34.226054+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"69'9","reported_seq":233,"reported_epoch":145,"state":"active+clean","last_fresh":"2024-09-18T08:06:52.486587+0000","last_change":"2024-09-18T08:06:24.171122+0000","last_active":"2024-09-18T08:06:52.486587+0000","last_peered":"2024-09-18T08:06:52.486587+0000","last_clean":"2024-09-18T08:06:52.486587+0000","last_became_active":"2024-09-18T08:06:24.140103+0000","last_became_peered":"2024-09-18T08:06:24.140103+0000","last_unstale":"2024-09-18T08:06:52.486587+0000","last_undegraded":"2024-09-18T08:06:52.486587+0000","last_fullsized":"2024-09-18T08:06:52.486587+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'9","last_scrub_stamp":"2024-09-18T08:05:08.696796+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_clean_scrub_stamp":"2024-09-18T08:05:08.696796+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":185,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-09-18T08:06:46.138665+0000","last_change":"2024-09-18T08:05:43.668557+0000","last_active":"2024-09-18T08:06:46.138665+0000","last_peered":"2024-09-18T08:06:46.138665+0000","last_clean":"2024-09-18T08:06:46.138665+0000","last_became_active":"2024-09-18T08:02:51.907140+0000","last_became_peered":"2024-09-18T08:02:51.907140+0000","last_unstale":"2024-09-18T08:06:46.138665+0000","last_undegraded":"2024-09-18T08:06:46.138665+0000","last_fullsized":"2024-09-18T08:06:46.138665+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:05:43.668271+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:50.886830+0000","last_clean_scrub_stamp":"2024-09-18T08:05:43.668271+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:07:02.226478+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":168,"reported_epoch":145,"state":"active+clean","last_fresh":"2024-09-18T08:06:52.486593+0000","last_change":"2024-09-18T08:02:53.906914+0000","last_active":"2024-09-18T08:06:52.486593+0000","last_peered":"2024-09-18T08:06:52.486593+0000","last_clean":"2024-09-18T08:06:52.486593+0000","last_became_active":"2024-09-18T08:02:53.906622+0000","last_became_peered":"2024-09-18T08:02:53.906622+0000","last_unstale":"2024-09-18T08:06:52.486593+0000","last_undegraded":"2024-09-18T08:06:52.486593+0000","last_fullsized":"2024-09-18T08:06:52.486593+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_clean_scrub_stamp":"2024-09-18T08:02:52.890475+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:07:04.364627+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":187,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-09-18T08:06:46.135279+0000","last_change":"2024-09-18T08:06:24.128885+0000","last_active":"2024-09-18T08:06:46.135279+0000","last_peered":"2024-09-18T08:06:46.135279+0000","last_clean":"2024-09-18T08:06:46.135279+0000","last_became_active":"2024-09-18T08:06:24.128511+0000","last_became_peered":"2024-09-18T08:06:24.128511+0000","last_unstale":"2024-09-18T08:06:46.135279+0000","last_undegraded":"2024-09-18T08:06:46.135279+0000","last_fullsized":"2024-09-18T08:06:46.135279+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:05:03.047120+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_clean_scrub_stamp":"2024-09-18T08:05:03.047120+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":199,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-09-18T08:06:46.137734+0000","last_change":"2024-09-18T08:06:24.126544+0000","last_active":"2024-09-18T08:06:46.137734+0000","last_peered":"2024-09-18T08:06:46.137734+0000","last_clean":"2024-09-18T08:06:46.137734+0000","last_became_active":"2024-09-18T08:06:24.126194+0000","last_became_peered":"2024-09-18T08:06:24.126194+0000","last_unstale":"2024-09-18T08:06:46.137734+0000","last_undegraded":"2024-09-18T08:06:46.137734+0000","last_fullsized":"2024-09-18T08:06:46.137734+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:04:57.047904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:46.860092+0000","last_clean_scrub_stamp":"2024-09-18T08:04:57.047904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"69'15","reported_seq":198,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-09-18T08:06:46.137641+0000","last_change":"2024-09-18T08:02:49.887111+0000","last_active":"2024-09-18T08:06:46.137641+0000","last_peered":"2024-09-18T08:06:46.137641+0000","last_clean":"2024-09-18T08:06:46.137641+0000","last_became_active":"2024-09-18T08:02:49.886106+0000","last_became_peered":"2024-09-18T08:02:49.886106+0000","last_unstale":"2024-09-18T08:06:46.137641+0000","last_undegraded":"2024-09-18T08:06:46.137641+0000","last_fullsized":"2024-09-18T08:06:46.137641+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_clean_scrub_stamp":"2024-09-18T08:02:48.870718+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:06:51.053554+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":193,"reported_epoch":145,"state":"active+clean","last_fresh":"2024-09-18T08:06:52.486441+0000","last_change":"2024-09-18T08:06:24.146536+0000","last_active":"2024-09-18T08:06:52.486441+0000","last_peered":"2024-09-18T08:06:52.486441+0000","last_clean":"2024-09-18T08:06:52.486441+0000","last_became_active":"2024-09-18T08:06:24.144841+0000","last_became_peered":"2024-09-18T08:06:24.144841+0000","last_unstale":"2024-09-18T08:06:52.486441+0000","last_undegraded":"2024-09-18T08:06:52.486441+0000","last_fullsized":"2024-09-18T08:06:52.486441+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:04:29.435669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:50.886830+0000","last_clean_scrub_stamp":"2024-09-18T08:04:29.435669+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:07:33.513868+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":209,"reported_epoch":145,"state":"active+clean","last_fresh":"2024-09-18T08:06:52.496654+0000","last_change":"2024-09-18T08:06:24.132542+0000","last_active":"2024-09-18T08:06:52.496654+0000","last_peered":"2024-09-18T08:06:52.496654+0000","last_clean":"2024-09-18T08:06:52.496654+0000","last_became_active":"2024-09-18T08:06:24.132206+0000","last_became_peered":"2024-09-18T08:06:24.132206+0000","last_unstale":"2024-09-18T08:06:52.496654+0000","last_undegraded":"2024-09-18T08:06:52.496654+0000","last_fullsized":"2024-09-18T08:06:52.496654+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:06:14.027077+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:46.860092+0000","last_clean_scrub_stamp":"2024-09-18T08:06:14.027077+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:07:24.620825+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"69'9","reported_seq":191,"reported_epoch":145,"state":"active+clean","last_fresh":"2024-09-18T08:06:52.486872+0000","last_change":"2024-09-18T08:02:49.887589+0000","last_active":"2024-09-18T08:06:52.486872+0000","last_peered":"2024-09-18T08:06:52.486872+0000","last_clean":"2024-09-18T08:06:52.486872+0000","last_became_active":"2024-09-18T08:02:49.887161+0000","last_became_peered":"2024-09-18T08:02:49.887161+0000","last_unstale":"2024-09-18T08:06:52.486872+0000","last_undegraded":"2024-09-18T08:06:52.486872+0000","last_fullsized":"2024-09-18T08:06:52.486872+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_clean_scrub_stamp":"2024-09-18T08:02:48.870718+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:06:59.372043+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":187,"reported_epoch":145,"state":"active+clean","last_fresh":"2024-09-18T08:06:52.496003+0000","last_change":"2024-09-18T08:05:33.935844+0000","last_active":"2024-09-18T08:06:52.496003+0000","last_peered":"2024-09-18T08:06:52.496003+0000","last_clean":"2024-09-18T08:06:52.496003+0000","last_became_active":"2024-09-18T08:02:51.907296+0000","last_became_peered":"2024-09-18T08:02:51.907296+0000","last_unstale":"2024-09-18T08:06:52.496003+0000","last_undegraded":"2024-09-18T08:06:52.496003+0000","last_fullsized":"2024-09-18T08:06:52.496003+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:05:33.935798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:50.886830+0000","last_clean_scrub_stamp":"2024-09-18T08:05:33.935798+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":174,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-09-18T08:06:46.137090+0000","last_change":"2024-09-18T08:05:01.314402+0000","last_active":"2024-09-18T08:06:46.137090+0000","last_peered":"2024-09-18T08:06:46.137090+0000","last_clean":"2024-09-18T08:06:46.137090+0000","last_became_active":"2024-09-18T08:02:53.913519+0000","last_became_peered":"2024-09-18T08:02:53.913519+0000","last_unstale":"2024-09-18T08:06:46.137090+0000","last_undegraded":"2024-09-18T08:06:46.137090+0000","last_fullsized":"2024-09-18T08:06:46.137090+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:05:01.314330+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_clean_scrub_stamp":"2024-09-18T08:05:01.314330+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":187,"reported_epoch":145,"state":"active+clean","last_fresh":"2024-09-18T08:06:52.496291+0000","last_change":"2024-09-18T08:04:32.849176+0000","last_active":"2024-09-18T08:06:52.496291+0000","last_peered":"2024-09-18T08:06:52.496291+0000","last_clean":"2024-09-18T08:06:52.496291+0000","last_became_active":"2024-09-18T08:02:47.875042+0000","last_became_peered":"2024-09-18T08:02:47.875042+0000","last_unstale":"2024-09-18T08:06:52.496291+0000","last_undegraded":"2024-09-18T08:06:52.496291+0000","last_fullsized":"2024-09-18T08:06:52.496291+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:04:32.849119+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:46.860092+0000","last_clean_scrub_stamp":"2024-09-18T08:04:32.849119+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:07:03.937944+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1b","version":"69'5","reported_seq":201,"reported_epoch":145,"state":"active+clean","last_fresh":"2024-09-18T08:06:52.489581+0000","last_change":"2024-09-18T08:06:09.466075+0000","last_active":"2024-09-18T08:06:52.489581+0000","last_peered":"2024-09-18T08:06:52.489581+0000","last_clean":"2024-09-18T08:06:52.489581+0000","last_became_active":"2024-09-18T08:02:49.895344+0000","last_became_peered":"2024-09-18T08:02:49.895344+0000","last_unstale":"2024-09-18T08:06:52.489581+0000","last_undegraded":"2024-09-18T08:06:52.489581+0000","last_fullsized":"2024-09-18T08:06:52.489581+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'5","last_scrub_stamp":"2024-09-18T08:06:09.465979+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_clean_scrub_stamp":"2024-09-18T08:06:09.465979+0000","objects_scrubbed":2,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:07:37.799450+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":11,"num_read_kb":7,"num_write":6,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":187,"reported_epoch":145,"state":"active+clean","last_fresh":"2024-09-18T08:06:52.509315+0000","last_change":"2024-09-18T08:05:38.866695+0000","last_active":"2024-09-18T08:06:52.509315+0000","last_peered":"2024-09-18T08:06:52.509315+0000","last_clean":"2024-09-18T08:06:52.509315+0000","last_became_active":"2024-09-18T08:02:51.905776+0000","last_became_peered":"2024-09-18T08:02:51.905776+0000","last_unstale":"2024-09-18T08:06:52.509315+0000","last_undegraded":"2024-09-18T08:06:52.509315+0000","last_fullsized":"2024-09-18T08:06:52.509315+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:05:38.866651+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:50.886830+0000","last_clean_scrub_stamp":"2024-09-18T08:05:38.866651+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:06:53.681027+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":184,"reported_epoch":145,"state":"active+clean","last_fresh":"2024-09-18T08:06:52.495805+0000","last_change":"2024-09-18T08:06:44.518882+0000","last_active":"2024-09-18T08:06:52.495805+0000","last_peered":"2024-09-18T08:06:52.495805+0000","last_clean":"2024-09-18T08:06:52.495805+0000","last_became_active":"2024-09-18T08:02:53.911648+0000","last_became_peered":"2024-09-18T08:02:53.911648+0000","last_unstale":"2024-09-18T08:06:52.495805+0000","last_undegraded":"2024-09-18T08:06:52.495805+0000","last_fullsized":"2024-09-18T08:06:52.495805+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:06:44.518835+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_clean_scrub_stamp":"2024-09-18T08:06:44.518835+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:08:04.920034+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":190,"reported_epoch":145,"state":"active+clean","last_fresh":"2024-09-18T08:06:52.488488+0000","last_change":"2024-09-18T08:06:24.132553+0000","last_active":"2024-09-18T08:06:52.488488+0000","last_peered":"2024-09-18T08:06:52.488488+0000","last_clean":"2024-09-18T08:06:52.488488+0000","last_became_active":"2024-09-18T08:06:24.132228+0000","last_became_peered":"2024-09-18T08:06:24.132228+0000","last_unstale":"2024-09-18T08:06:52.488488+0000","last_undegraded":"2024-09-18T08:06:52.488488+0000","last_fullsized":"2024-09-18T08:06:52.488488+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:05:40.477965+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_clean_scrub_stamp":"2024-09-18T08:05:40.477965+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:06:55.439381+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":193,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-09-18T08:06:46.137781+0000","last_change":"2024-09-18T08:05:27.476211+0000","last_active":"2024-09-18T08:06:46.137781+0000","last_peered":"2024-09-18T08:06:46.137781+0000","last_clean":"2024-09-18T08:06:46.137781+0000","last_became_active":"2024-09-18T08:02:47.875963+0000","last_became_peered":"2024-09-18T08:02:47.875963+0000","last_unstale":"2024-09-18T08:06:46.137781+0000","last_undegraded":"2024-09-18T08:06:46.137781+0000","last_fullsized":"2024-09-18T08:06:46.137781+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:05:27.476144+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:46.860092+0000","last_clean_scrub_stamp":"2024-09-18T08:05:27.476144+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1c","version":"69'15","reported_seq":237,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-09-18T08:06:46.135068+0000","last_change":"2024-09-18T08:05:28.084324+0000","last_active":"2024-09-18T08:06:46.135068+0000","last_peered":"2024-09-18T08:06:46.135068+0000","last_clean":"2024-09-18T08:06:46.135068+0000","last_became_active":"2024-09-18T08:04:18.147907+0000","last_became_peered":"2024-09-18T08:04:18.147907+0000","last_unstale":"2024-09-18T08:06:46.135068+0000","last_undegraded":"2024-09-18T08:06:46.135068+0000","last_fullsized":"2024-09-18T08:06:46.135068+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'15","last_scrub_stamp":"2024-09-18T08:05:28.084220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_clean_scrub_stamp":"2024-09-18T08:05:28.084220+0000","objects_scrubbed":7,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,3],"acting":[2,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":185,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-09-18T08:06:46.139517+0000","last_change":"2024-09-18T08:05:40.779422+0000","last_active":"2024-09-18T08:06:46.139517+0000","last_peered":"2024-09-18T08:06:46.139517+0000","last_clean":"2024-09-18T08:06:46.139517+0000","last_became_active":"2024-09-18T08:02:51.908430+0000","last_became_peered":"2024-09-18T08:02:51.908430+0000","last_unstale":"2024-09-18T08:06:46.139517+0000","last_undegraded":"2024-09-18T08:06:46.139517+0000","last_fullsized":"2024-09-18T08:06:46.139517+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:05:40.779230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:50.886830+0000","last_clean_scrub_stamp":"2024-09-18T08:05:40.779230+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:07:00.025019+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":187,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-09-18T08:06:46.138014+0000","last_change":"2024-09-18T08:06:24.129031+0000","last_active":"2024-09-18T08:06:46.138014+0000","last_peered":"2024-09-18T08:06:46.138014+0000","last_clean":"2024-09-18T08:06:46.138014+0000","last_became_active":"2024-09-18T08:06:24.128659+0000","last_became_peered":"2024-09-18T08:06:24.128659+0000","last_unstale":"2024-09-18T08:06:46.138014+0000","last_undegraded":"2024-09-18T08:06:46.138014+0000","last_fullsized":"2024-09-18T08:06:46.138014+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:04:29.955822+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_clean_scrub_stamp":"2024-09-18T08:04:29.955822+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":187,"reported_epoch":145,"state":"active+clean","last_fresh":"2024-09-18T08:06:52.486963+0000","last_change":"2024-09-18T08:04:22.420158+0000","last_active":"2024-09-18T08:06:52.486963+0000","last_peered":"2024-09-18T08:06:52.486963+0000","last_clean":"2024-09-18T08:06:52.486963+0000","last_became_active":"2024-09-18T08:02:47.871523+0000","last_became_peered":"2024-09-18T08:02:47.871523+0000","last_unstale":"2024-09-18T08:06:52.486963+0000","last_undegraded":"2024-09-18T08:06:52.486963+0000","last_fullsized":"2024-09-18T08:06:52.486963+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:04:22.420112+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:46.860092+0000","last_clean_scrub_stamp":"2024-09-18T08:04:22.420112+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:07:06.346239+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"69'12","reported_seq":204,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-09-18T08:06:46.134307+0000","last_change":"2024-09-18T08:05:47.068485+0000","last_active":"2024-09-18T08:06:46.134307+0000","last_peered":"2024-09-18T08:06:46.134307+0000","last_clean":"2024-09-18T08:06:46.134307+0000","last_became_active":"2024-09-18T08:02:49.888831+0000","last_became_peered":"2024-09-18T08:02:49.888831+0000","last_unstale":"2024-09-18T08:06:46.134307+0000","last_undegraded":"2024-09-18T08:06:46.134307+0000","last_fullsized":"2024-09-18T08:06:46.134307+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'12","last_scrub_stamp":"2024-09-18T08:05:47.068375+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_clean_scrub_stamp":"2024-09-18T08:05:47.068375+0000","objects_scrubbed":5,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:06:50.263718+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":25,"num_read_kb":16,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":187,"reported_epoch":145,"state":"active+clean","last_fresh":"2024-09-18T08:06:52.487187+0000","last_change":"2024-09-18T08:06:12.434234+0000","last_active":"2024-09-18T08:06:52.487187+0000","last_peered":"2024-09-18T08:06:52.487187+0000","last_clean":"2024-09-18T08:06:52.487187+0000","last_became_active":"2024-09-18T08:02:51.910316+0000","last_became_peered":"2024-09-18T08:02:51.910316+0000","last_unstale":"2024-09-18T08:06:52.487187+0000","last_undegraded":"2024-09-18T08:06:52.487187+0000","last_fullsized":"2024-09-18T08:06:52.487187+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:06:12.434179+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:50.886830+0000","last_clean_scrub_stamp":"2024-09-18T08:06:12.434179+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:07:35.662786+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1c","version":"69'1","reported_seq":27,"reported_epoch":145,"state":"active+clean","last_fresh":"2024-09-18T08:06:52.496100+0000","last_change":"2024-09-18T08:06:46.142143+0000","last_active":"2024-09-18T08:06:52.496100+0000","last_peered":"2024-09-18T08:06:52.496100+0000","last_clean":"2024-09-18T08:06:52.496100+0000","last_became_active":"2024-09-18T08:06:46.141661+0000","last_became_peered":"2024-09-18T08:06:46.141661+0000","last_unstale":"2024-09-18T08:06:52.496100+0000","last_undegraded":"2024-09-18T08:06:52.496100+0000","last_fullsized":"2024-09-18T08:06:52.496100+0000","mapping_epoch":143,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":144,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'1","last_scrub_stamp":"2024-09-18T08:05:36.776369+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_clean_scrub_stamp":"2024-09-18T08:05:36.776369+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:07:00.455428+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":403,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,2],"acting":[5,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"62'1","reported_seq":186,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-09-18T08:06:46.138777+0000","last_change":"2024-09-18T08:05:06.819830+0000","last_active":"2024-09-18T08:06:46.138777+0000","last_peered":"2024-09-18T08:06:46.138777+0000","last_clean":"2024-09-18T08:06:46.138777+0000","last_became_active":"2024-09-18T08:02:47.880998+0000","last_became_peered":"2024-09-18T08:02:47.880998+0000","last_unstale":"2024-09-18T08:06:46.138777+0000","last_undegraded":"2024-09-18T08:06:46.138777+0000","last_fullsized":"2024-09-18T08:06:46.138777+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"62'1","last_scrub_stamp":"2024-09-18T08:05:06.819539+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:46.860092+0000","last_clean_scrub_stamp":"2024-09-18T08:05:06.819539+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:07:39.507165+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1e","version":"69'10","reported_seq":188,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-09-18T08:06:46.138100+0000","last_change":"2024-09-18T08:02:49.892755+0000","last_active":"2024-09-18T08:06:46.138100+0000","last_peered":"2024-09-18T08:06:46.138100+0000","last_clean":"2024-09-18T08:06:46.138100+0000","last_became_active":"2024-09-18T08:02:49.892024+0000","last_became_peered":"2024-09-18T08:02:49.892024+0000","last_unstale":"2024-09-18T08:06:46.138100+0000","last_undegraded":"2024-09-18T08:06:46.138100+0000","last_fullsized":"2024-09-18T08:06:46.138100+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_clean_scrub_stamp":"2024-09-18T08:02:48.870718+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:06:59.489315+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1f","version":"138'11","reported_seq":187,"reported_epoch":145,"state":"active+clean","last_fresh":"2024-09-18T08:06:52.488632+0000","last_change":"2024-09-18T08:06:24.180060+0000","last_active":"2024-09-18T08:06:52.488632+0000","last_peered":"2024-09-18T08:06:52.488632+0000","last_clean":"2024-09-18T08:06:52.488632+0000","last_became_active":"2024-09-18T08:06:24.145360+0000","last_became_peered":"2024-09-18T08:06:24.145360+0000","last_unstale":"2024-09-18T08:06:52.488632+0000","last_undegraded":"2024-09-18T08:06:52.488632+0000","last_fullsized":"2024-09-18T08:06:52.488632+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"102'9","last_scrub_stamp":"2024-09-18T08:05:44.463780+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:50.886830+0000","last_clean_scrub_stamp":"2024-09-18T08:05:44.463780+0000","objects_scrubbed":1,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:07:07.778823+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,7],"acting":[4,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.c","version":"144'11829","reported_seq":13860,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-09-18T08:06:48.437131+0000","last_change":"2024-09-18T08:06:25.149370+0000","last_active":"2024-09-18T08:06:48.437131+0000","last_peered":"2024-09-18T08:06:48.437131+0000","last_clean":"2024-09-18T08:06:48.437131+0000","last_became_active":"2024-09-18T08:06:25.123363+0000","last_became_peered":"2024-09-18T08:06:25.123363+0000","last_unstale":"2024-09-18T08:06:48.437131+0000","last_undegraded":"2024-09-18T08:06:48.437131+0000","last_fullsized":"2024-09-18T08:06:48.437131+0000","mapping_epoch":137,"log_start":"118'7118","ondisk_log_start":"118'7118","created":83,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:03:56.449957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:03:56.449957+0000","last_clean_scrub_stamp":"2024-09-18T08:03:56.449957+0000","objects_scrubbed":0,"log_size":4711,"log_dups_size":0,"ondisk_log_size":4711,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:07:33.267946+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":208470016,"num_objects":3181,"num_object_clones":0,"num_object_copies":9543,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3181,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11829,"num_write_kb":480320,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":940,"num_bytes_recovered":61603840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,7],"acting":[3,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.f","version":"69'15","reported_seq":206,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-09-18T08:06:46.139222+0000","last_change":"2024-09-18T08:04:22.741342+0000","last_active":"2024-09-18T08:06:46.139222+0000","last_peered":"2024-09-18T08:06:46.139222+0000","last_clean":"2024-09-18T08:06:46.139222+0000","last_became_active":"2024-09-18T08:02:49.899083+0000","last_became_peered":"2024-09-18T08:02:49.899083+0000","last_unstale":"2024-09-18T08:06:46.139222+0000","last_undegraded":"2024-09-18T08:06:46.139222+0000","last_fullsized":"2024-09-18T08:06:46.139222+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'15","last_scrub_stamp":"2024-09-18T08:04:22.741261+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_clean_scrub_stamp":"2024-09-18T08:04:22.741261+0000","objects_scrubbed":7,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:06:53.535126+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":185,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-09-18T08:06:46.135102+0000","last_change":"2024-09-18T08:04:08.982535+0000","last_active":"2024-09-18T08:06:46.135102+0000","last_peered":"2024-09-18T08:06:46.135102+0000","last_clean":"2024-09-18T08:06:46.135102+0000","last_became_active":"2024-09-18T08:02:47.879328+0000","last_became_peered":"2024-09-18T08:02:47.879328+0000","last_unstale":"2024-09-18T08:06:46.135102+0000","last_undegraded":"2024-09-18T08:06:46.135102+0000","last_fullsized":"2024-09-18T08:06:46.135102+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:04:08.982469+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:46.860092+0000","last_clean_scrub_stamp":"2024-09-18T08:04:08.982469+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:06:57.148595+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.e","version":"69'8","reported_seq":245,"reported_epoch":145,"state":"active+clean","last_fresh":"2024-09-18T08:06:52.488570+0000","last_change":"2024-09-18T08:06:19.459452+0000","last_active":"2024-09-18T08:06:52.488570+0000","last_peered":"2024-09-18T08:06:52.488570+0000","last_clean":"2024-09-18T08:06:52.488570+0000","last_became_active":"2024-09-18T08:02:51.903369+0000","last_became_peered":"2024-09-18T08:02:51.903369+0000","last_unstale":"2024-09-18T08:06:52.488570+0000","last_undegraded":"2024-09-18T08:06:52.488570+0000","last_fullsized":"2024-09-18T08:06:52.488570+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'8","last_scrub_stamp":"2024-09-18T08:06:19.459375+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:50.886830+0000","last_clean_scrub_stamp":"2024-09-18T08:06:19.459375+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:07:30.186221+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":184,"reported_epoch":145,"state":"active+clean","last_fresh":"2024-09-18T08:06:52.496022+0000","last_change":"2024-09-18T08:05:34.905147+0000","last_active":"2024-09-18T08:06:52.496022+0000","last_peered":"2024-09-18T08:06:52.496022+0000","last_clean":"2024-09-18T08:06:52.496022+0000","last_became_active":"2024-09-18T08:02:53.909834+0000","last_became_peered":"2024-09-18T08:02:53.909834+0000","last_unstale":"2024-09-18T08:06:52.496022+0000","last_undegraded":"2024-09-18T08:06:52.496022+0000","last_fullsized":"2024-09-18T08:06:52.496022+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:05:34.905083+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_clean_scrub_stamp":"2024-09-18T08:05:34.905083+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:06:53.452264+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.3","version":"144'11566","reported_seq":11700,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-09-18T08:06:48.436942+0000","last_change":"2024-09-18T08:03:57.462489+0000","last_active":"2024-09-18T08:06:48.436942+0000","last_peered":"2024-09-18T08:06:48.436942+0000","last_clean":"2024-09-18T08:06:48.436942+0000","last_became_active":"2024-09-18T08:03:57.462210+0000","last_became_peered":"2024-09-18T08:03:57.462210+0000","last_unstale":"2024-09-18T08:06:48.436942+0000","last_undegraded":"2024-09-18T08:06:48.436942+0000","last_fullsized":"2024-09-18T08:06:48.436942+0000","mapping_epoch":83,"log_start":"118'6922","ondisk_log_start":"118'6922","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:03:56.449957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:03:56.449957+0000","last_clean_scrub_stamp":"2024-09-18T08:03:56.449957+0000","objects_scrubbed":0,"log_size":4644,"log_dups_size":0,"ondisk_log_size":4644,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:07:30.236230+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":203685888,"num_objects":3108,"num_object_clones":0,"num_object_copies":9324,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3108,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11566,"num_write_kb":469568,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,4],"acting":[3,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.0","version":"69'18","reported_seq":221,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-09-18T08:06:46.134797+0000","last_change":"2024-09-18T08:06:44.301130+0000","last_active":"2024-09-18T08:06:46.134797+0000","last_peered":"2024-09-18T08:06:46.134797+0000","last_clean":"2024-09-18T08:06:46.134797+0000","last_became_active":"2024-09-18T08:02:49.886818+0000","last_became_peered":"2024-09-18T08:02:49.886818+0000","last_unstale":"2024-09-18T08:06:46.134797+0000","last_undegraded":"2024-09-18T08:06:46.134797+0000","last_fullsized":"2024-09-18T08:06:46.134797+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'18","last_scrub_stamp":"2024-09-18T08:06:44.300772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_clean_scrub_stamp":"2024-09-18T08:06:44.300772+0000","objects_scrubbed":8,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:07:52.014378+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":34,"num_read_kb":22,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":193,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-09-18T08:06:46.138443+0000","last_change":"2024-09-18T08:05:35.100407+0000","last_active":"2024-09-18T08:06:46.138443+0000","last_peered":"2024-09-18T08:06:46.138443+0000","last_clean":"2024-09-18T08:06:46.138443+0000","last_became_active":"2024-09-18T08:02:47.880225+0000","last_became_peered":"2024-09-18T08:02:47.880225+0000","last_unstale":"2024-09-18T08:06:46.138443+0000","last_undegraded":"2024-09-18T08:06:46.138443+0000","last_fullsized":"2024-09-18T08:06:46.138443+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:05:35.100332+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:46.860092+0000","last_clean_scrub_stamp":"2024-09-18T08:05:35.100332+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:06:57.779942+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":187,"reported_epoch":145,"state":"active+clean","last_fresh":"2024-09-18T08:06:52.489066+0000","last_change":"2024-09-18T08:06:10.468172+0000","last_active":"2024-09-18T08:06:52.489066+0000","last_peered":"2024-09-18T08:06:52.489066+0000","last_clean":"2024-09-18T08:06:52.489066+0000","last_became_active":"2024-09-18T08:02:51.899594+0000","last_became_peered":"2024-09-18T08:02:51.899594+0000","last_unstale":"2024-09-18T08:06:52.489066+0000","last_undegraded":"2024-09-18T08:06:52.489066+0000","last_fullsized":"2024-09-18T08:06:52.489066+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:06:10.468119+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:50.886830+0000","last_clean_scrub_stamp":"2024-09-18T08:06:10.468119+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:07:23.534443+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":176,"reported_epoch":145,"state":"active+clean","last_fresh":"2024-09-18T08:06:52.486768+0000","last_change":"2024-09-18T08:04:16.427470+0000","last_active":"2024-09-18T08:06:52.486768+0000","last_peered":"2024-09-18T08:06:52.486768+0000","last_clean":"2024-09-18T08:06:52.486768+0000","last_became_active":"2024-09-18T08:02:53.903957+0000","last_became_peered":"2024-09-18T08:02:53.903957+0000","last_unstale":"2024-09-18T08:06:52.486768+0000","last_undegraded":"2024-09-18T08:06:52.486768+0000","last_fullsized":"2024-09-18T08:06:52.486768+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:04:16.427390+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_clean_scrub_stamp":"2024-09-18T08:04:16.427390+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:07:26.444141+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.2","version":"144'11635","reported_seq":11769,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-09-18T08:06:48.447327+0000","last_change":"2024-09-18T08:03:57.463750+0000","last_active":"2024-09-18T08:06:48.447327+0000","last_peered":"2024-09-18T08:06:48.447327+0000","last_clean":"2024-09-18T08:06:48.447327+0000","last_became_active":"2024-09-18T08:03:57.463459+0000","last_became_peered":"2024-09-18T08:03:57.463459+0000","last_unstale":"2024-09-18T08:06:48.447327+0000","last_undegraded":"2024-09-18T08:06:48.447327+0000","last_fullsized":"2024-09-18T08:06:48.447327+0000","mapping_epoch":83,"log_start":"118'6943","ondisk_log_start":"118'6943","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:03:56.449957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:03:56.449957+0000","last_clean_scrub_stamp":"2024-09-18T08:03:56.449957+0000","objects_scrubbed":0,"log_size":4692,"log_dups_size":0,"ondisk_log_size":4692,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:06:59.160354+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":208863232,"num_objects":3187,"num_object_clones":0,"num_object_copies":9561,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3187,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11635,"num_write_kb":474304,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"69'14","reported_seq":258,"reported_epoch":145,"state":"active+clean","last_fresh":"2024-09-18T08:06:52.489169+0000","last_change":"2024-09-18T08:06:24.184801+0000","last_active":"2024-09-18T08:06:52.489169+0000","last_peered":"2024-09-18T08:06:52.489169+0000","last_clean":"2024-09-18T08:06:52.489169+0000","last_became_active":"2024-09-18T08:06:24.133949+0000","last_became_peered":"2024-09-18T08:06:24.133949+0000","last_unstale":"2024-09-18T08:06:52.489169+0000","last_undegraded":"2024-09-18T08:06:52.489169+0000","last_fullsized":"2024-09-18T08:06:52.489169+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'14","last_scrub_stamp":"2024-09-18T08:04:56.545262+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_clean_scrub_stamp":"2024-09-18T08:04:56.545262+0000","objects_scrubbed":7,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:07:37.514965+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":21,"num_read_kb":14,"num_write":14,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"62'2","reported_seq":200,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-09-18T08:06:46.138021+0000","last_change":"2024-09-18T08:06:43.607646+0000","last_active":"2024-09-18T08:06:46.138021+0000","last_peered":"2024-09-18T08:06:46.138021+0000","last_clean":"2024-09-18T08:06:46.138021+0000","last_became_active":"2024-09-18T08:02:47.872857+0000","last_became_peered":"2024-09-18T08:02:47.872857+0000","last_unstale":"2024-09-18T08:06:46.138021+0000","last_undegraded":"2024-09-18T08:06:46.138021+0000","last_fullsized":"2024-09-18T08:06:46.138021+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"62'2","last_scrub_stamp":"2024-09-18T08:06:43.607152+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:46.860092+0000","last_clean_scrub_stamp":"2024-09-18T08:06:43.607152+0000","objects_scrubbed":2,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:07:53.027436+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1039,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.0","version":"69'8","reported_seq":234,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-09-18T08:06:46.134599+0000","last_change":"2024-09-18T08:04:23.863573+0000","last_active":"2024-09-18T08:06:46.134599+0000","last_peered":"2024-09-18T08:06:46.134599+0000","last_clean":"2024-09-18T08:06:46.134599+0000","last_became_active":"2024-09-18T08:02:51.909680+0000","last_became_peered":"2024-09-18T08:02:51.909680+0000","last_unstale":"2024-09-18T08:06:46.134599+0000","last_undegraded":"2024-09-18T08:06:46.134599+0000","last_fullsized":"2024-09-18T08:06:46.134599+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'8","last_scrub_stamp":"2024-09-18T08:04:23.863330+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:50.886830+0000","last_clean_scrub_stamp":"2024-09-18T08:04:23.863330+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":198,"reported_epoch":145,"state":"active+clean","last_fresh":"2024-09-18T08:06:52.509489+0000","last_change":"2024-09-18T08:06:44.781751+0000","last_active":"2024-09-18T08:06:52.509489+0000","last_peered":"2024-09-18T08:06:52.509489+0000","last_clean":"2024-09-18T08:06:52.509489+0000","last_became_active":"2024-09-18T08:06:24.140303+0000","last_became_peered":"2024-09-18T08:06:24.140303+0000","last_unstale":"2024-09-18T08:06:52.509489+0000","last_undegraded":"2024-09-18T08:06:52.509489+0000","last_fullsized":"2024-09-18T08:06:52.509489+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:06:44.781685+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_clean_scrub_stamp":"2024-09-18T08:06:44.781685+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:07:48.361147+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.1","version":"145'12022","reported_seq":12169,"reported_epoch":145,"state":"active+clean","last_fresh":"2024-09-18T08:06:53.089992+0000","last_change":"2024-09-18T08:06:24.166578+0000","last_active":"2024-09-18T08:06:53.089992+0000","last_peered":"2024-09-18T08:06:53.089992+0000","last_clean":"2024-09-18T08:06:53.089992+0000","last_became_active":"2024-09-18T08:06:24.134315+0000","last_became_peered":"2024-09-18T08:06:24.134315+0000","last_unstale":"2024-09-18T08:06:53.089992+0000","last_undegraded":"2024-09-18T08:06:53.089992+0000","last_fullsized":"2024-09-18T08:06:53.089992+0000","mapping_epoch":136,"log_start":"119'7257","ondisk_log_start":"119'7257","created":83,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:03:56.449957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:03:56.449957+0000","last_clean_scrub_stamp":"2024-09-18T08:03:56.449957+0000","objects_scrubbed":0,"log_size":4765,"log_dups_size":0,"ondisk_log_size":4765,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:07:27.452504+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":185204736,"num_objects":2826,"num_object_clones":0,"num_object_copies":8478,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2826,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12022,"num_write_kb":475136,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.2","version":"69'10","reported_seq":204,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-09-18T08:06:46.138677+0000","last_change":"2024-09-18T08:05:39.769686+0000","last_active":"2024-09-18T08:06:46.138677+0000","last_peered":"2024-09-18T08:06:46.138677+0000","last_clean":"2024-09-18T08:06:46.138677+0000","last_became_active":"2024-09-18T08:02:49.895265+0000","last_became_peered":"2024-09-18T08:02:49.895265+0000","last_unstale":"2024-09-18T08:06:46.138677+0000","last_undegraded":"2024-09-18T08:06:46.138677+0000","last_fullsized":"2024-09-18T08:06:46.138677+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'10","last_scrub_stamp":"2024-09-18T08:05:39.768393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_clean_scrub_stamp":"2024-09-18T08:05:39.768393+0000","objects_scrubbed":5,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:07:07.361416+0000","scrub_duration":17,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":187,"reported_epoch":145,"state":"active+clean","last_fresh":"2024-09-18T08:06:52.496677+0000","last_change":"2024-09-18T08:03:48.009689+0000","last_active":"2024-09-18T08:06:52.496677+0000","last_peered":"2024-09-18T08:06:52.496677+0000","last_clean":"2024-09-18T08:06:52.496677+0000","last_became_active":"2024-09-18T08:02:47.874727+0000","last_became_peered":"2024-09-18T08:02:47.874727+0000","last_unstale":"2024-09-18T08:06:52.496677+0000","last_undegraded":"2024-09-18T08:06:52.496677+0000","last_fullsized":"2024-09-18T08:06:52.496677+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:03:48.009611+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:46.860092+0000","last_clean_scrub_stamp":"2024-09-18T08:03:48.009611+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:07:24.213853+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.3","version":"138'10","reported_seq":257,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-09-18T08:06:46.137371+0000","last_change":"2024-09-18T08:06:24.157627+0000","last_active":"2024-09-18T08:06:46.137371+0000","last_peered":"2024-09-18T08:06:46.137371+0000","last_clean":"2024-09-18T08:06:46.137371+0000","last_became_active":"2024-09-18T08:06:24.134194+0000","last_became_peered":"2024-09-18T08:06:24.134194+0000","last_unstale":"2024-09-18T08:06:46.137371+0000","last_undegraded":"2024-09-18T08:06:46.137371+0000","last_fullsized":"2024-09-18T08:06:46.137371+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"130'9","last_scrub_stamp":"2024-09-18T08:06:13.462610+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:50.886830+0000","last_clean_scrub_stamp":"2024-09-18T08:06:13.462610+0000","objects_scrubbed":1,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:07:27.049204+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":182,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-09-18T08:06:46.137269+0000","last_change":"2024-09-18T08:05:43.440713+0000","last_active":"2024-09-18T08:06:46.137269+0000","last_peered":"2024-09-18T08:06:46.137269+0000","last_clean":"2024-09-18T08:06:46.137269+0000","last_became_active":"2024-09-18T08:02:53.909583+0000","last_became_peered":"2024-09-18T08:02:53.909583+0000","last_unstale":"2024-09-18T08:06:46.137269+0000","last_undegraded":"2024-09-18T08:06:46.137269+0000","last_fullsized":"2024-09-18T08:06:46.137269+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:05:43.440642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_clean_scrub_stamp":"2024-09-18T08:05:43.440642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:07:02.845583+0000","scrub_duration":35,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.0","version":"145'11842","reported_seq":12039,"reported_epoch":145,"state":"active+clean","last_fresh":"2024-09-18T08:06:53.082532+0000","last_change":"2024-09-18T08:05:01.419397+0000","last_active":"2024-09-18T08:06:53.082532+0000","last_peered":"2024-09-18T08:06:53.082532+0000","last_clean":"2024-09-18T08:06:53.082532+0000","last_became_active":"2024-09-18T08:03:57.468985+0000","last_became_peered":"2024-09-18T08:03:57.468985+0000","last_unstale":"2024-09-18T08:06:53.082532+0000","last_undegraded":"2024-09-18T08:06:53.082532+0000","last_fullsized":"2024-09-18T08:06:53.082532+0000","mapping_epoch":83,"log_start":"110'6032","ondisk_log_start":"110'6032","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"107'5286","last_scrub_stamp":"2024-09-18T08:05:01.419332+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:03:56.449957+0000","last_clean_scrub_stamp":"2024-09-18T08:05:01.419332+0000","objects_scrubbed":5263,"log_size":5810,"log_dups_size":0,"ondisk_log_size":5810,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:07:27.313345+0000","scrub_duration":1313,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":185073664,"num_objects":2824,"num_object_clones":0,"num_object_copies":8472,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2824,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11842,"num_write_kb":469312,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.3","version":"69'19","reported_seq":217,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-09-18T08:06:46.137520+0000","last_change":"2024-09-18T08:04:07.231406+0000","last_active":"2024-09-18T08:06:46.137520+0000","last_peered":"2024-09-18T08:06:46.137520+0000","last_clean":"2024-09-18T08:06:46.137520+0000","last_became_active":"2024-09-18T08:02:49.890318+0000","last_became_peered":"2024-09-18T08:02:49.890318+0000","last_unstale":"2024-09-18T08:06:46.137520+0000","last_undegraded":"2024-09-18T08:06:46.137520+0000","last_fullsized":"2024-09-18T08:06:46.137520+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'19","last_scrub_stamp":"2024-09-18T08:04:07.229846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_clean_scrub_stamp":"2024-09-18T08:04:07.229846+0000","objects_scrubbed":8,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:07:33.520229+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":39,"num_read_kb":25,"num_write":22,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.4","version":"62'1","reported_seq":191,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-09-18T08:06:46.139619+0000","last_change":"2024-09-18T08:04:28.827545+0000","last_active":"2024-09-18T08:06:46.139619+0000","last_peered":"2024-09-18T08:06:46.139619+0000","last_clean":"2024-09-18T08:06:46.139619+0000","last_became_active":"2024-09-18T08:02:47.874764+0000","last_became_peered":"2024-09-18T08:02:47.874764+0000","last_unstale":"2024-09-18T08:06:46.139619+0000","last_undegraded":"2024-09-18T08:06:46.139619+0000","last_fullsized":"2024-09-18T08:06:46.139619+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"62'1","last_scrub_stamp":"2024-09-18T08:04:28.827464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:46.860092+0000","last_clean_scrub_stamp":"2024-09-18T08:04:28.827464+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:06:52.521799+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":436,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":112,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-09-18T08:06:46.137254+0000","last_change":"2024-09-18T08:06:24.133954+0000","last_active":"2024-09-18T08:06:46.137254+0000","last_peered":"2024-09-18T08:06:46.137254+0000","last_clean":"2024-09-18T08:06:46.137254+0000","last_became_active":"2024-09-18T08:06:24.133262+0000","last_became_peered":"2024-09-18T08:06:24.133262+0000","last_unstale":"2024-09-18T08:06:46.137254+0000","last_undegraded":"2024-09-18T08:06:46.137254+0000","last_fullsized":"2024-09-18T08:06:46.137254+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:04:13.377561+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:50.886830+0000","last_clean_scrub_stamp":"2024-09-18T08:04:13.377561+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":195,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-09-18T08:06:46.138640+0000","last_change":"2024-09-18T08:06:24.139017+0000","last_active":"2024-09-18T08:06:46.138640+0000","last_peered":"2024-09-18T08:06:46.138640+0000","last_clean":"2024-09-18T08:06:46.138640+0000","last_became_active":"2024-09-18T08:06:24.138652+0000","last_became_peered":"2024-09-18T08:06:24.138652+0000","last_unstale":"2024-09-18T08:06:46.138640+0000","last_undegraded":"2024-09-18T08:06:46.138640+0000","last_fullsized":"2024-09-18T08:06:46.138640+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:06:10.507191+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_clean_scrub_stamp":"2024-09-18T08:06:10.507191+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:07:38.239809+0000","scrub_duration":25,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.7","version":"144'11704","reported_seq":13485,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-09-18T08:06:48.498785+0000","last_change":"2024-09-18T08:06:25.150134+0000","last_active":"2024-09-18T08:06:48.498785+0000","last_peered":"2024-09-18T08:06:48.498785+0000","last_clean":"2024-09-18T08:06:48.498785+0000","last_became_active":"2024-09-18T08:06:25.123728+0000","last_became_peered":"2024-09-18T08:06:25.123728+0000","last_unstale":"2024-09-18T08:06:48.498785+0000","last_undegraded":"2024-09-18T08:06:48.498785+0000","last_fullsized":"2024-09-18T08:06:48.498785+0000","mapping_epoch":137,"log_start":"109'6066","ondisk_log_start":"109'6066","created":83,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:03:56.449957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:03:56.449957+0000","last_clean_scrub_stamp":"2024-09-18T08:03:56.449957+0000","objects_scrubbed":4174,"log_size":5638,"log_dups_size":0,"ondisk_log_size":5638,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:07:31.458495+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":207880192,"num_objects":3172,"num_object_clones":0,"num_object_copies":9516,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3172,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11704,"num_write_kb":476032,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":787,"num_bytes_recovered":51511296,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,3],"acting":[0,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.4","version":"69'28","reported_seq":278,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-09-18T08:06:46.139492+0000","last_change":"2024-09-18T08:04:12.737618+0000","last_active":"2024-09-18T08:06:46.139492+0000","last_peered":"2024-09-18T08:06:46.139492+0000","last_clean":"2024-09-18T08:06:46.139492+0000","last_became_active":"2024-09-18T08:02:49.897221+0000","last_became_peered":"2024-09-18T08:02:49.897221+0000","last_unstale":"2024-09-18T08:06:46.139492+0000","last_undegraded":"2024-09-18T08:06:46.139492+0000","last_fullsized":"2024-09-18T08:06:46.139492+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'28","last_scrub_stamp":"2024-09-18T08:04:12.737528+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_clean_scrub_stamp":"2024-09-18T08:04:12.737528+0000","objects_scrubbed":10,"log_size":28,"log_dups_size":0,"ondisk_log_size":28,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:06:50.593181+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":48,"num_read_kb":33,"num_write":26,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":201,"reported_epoch":145,"state":"active+clean","last_fresh":"2024-09-18T08:06:52.489748+0000","last_change":"2024-09-18T08:06:24.143321+0000","last_active":"2024-09-18T08:06:52.489748+0000","last_peered":"2024-09-18T08:06:52.489748+0000","last_clean":"2024-09-18T08:06:52.489748+0000","last_became_active":"2024-09-18T08:06:24.143019+0000","last_became_peered":"2024-09-18T08:06:24.143019+0000","last_unstale":"2024-09-18T08:06:52.489748+0000","last_undegraded":"2024-09-18T08:06:52.489748+0000","last_fullsized":"2024-09-18T08:06:52.489748+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:05:39.490141+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:46.860092+0000","last_clean_scrub_stamp":"2024-09-18T08:05:39.490141+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"64'2","reported_seq":227,"reported_epoch":145,"state":"active+clean","last_fresh":"2024-09-18T08:06:52.496037+0000","last_change":"2024-09-18T08:06:24.165227+0000","last_active":"2024-09-18T08:06:52.496037+0000","last_peered":"2024-09-18T08:06:52.496037+0000","last_clean":"2024-09-18T08:06:52.496037+0000","last_became_active":"2024-09-18T08:06:24.136080+0000","last_became_peered":"2024-09-18T08:06:24.136080+0000","last_unstale":"2024-09-18T08:06:52.496037+0000","last_undegraded":"2024-09-18T08:06:52.496037+0000","last_fullsized":"2024-09-18T08:06:52.496037+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"64'2","last_scrub_stamp":"2024-09-18T08:05:44.802942+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:45.855979+0000","last_clean_scrub_stamp":"2024-09-18T08:05:44.802942+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.001348349,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":38,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":169,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-09-18T08:06:46.140182+0000","last_change":"2024-09-18T08:02:51.905694+0000","last_active":"2024-09-18T08:06:46.140182+0000","last_peered":"2024-09-18T08:06:46.140182+0000","last_clean":"2024-09-18T08:06:46.140182+0000","last_became_active":"2024-09-18T08:02:51.905357+0000","last_became_peered":"2024-09-18T08:02:51.905357+0000","last_unstale":"2024-09-18T08:06:46.140182+0000","last_undegraded":"2024-09-18T08:06:46.140182+0000","last_fullsized":"2024-09-18T08:06:46.140182+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:02:50.886830+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:50.886830+0000","last_clean_scrub_stamp":"2024-09-18T08:02:50.886830+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:06:51.299235+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":174,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-09-18T08:06:46.137462+0000","last_change":"2024-09-18T08:05:04.962900+0000","last_active":"2024-09-18T08:06:46.137462+0000","last_peered":"2024-09-18T08:06:46.137462+0000","last_clean":"2024-09-18T08:06:46.137462+0000","last_became_active":"2024-09-18T08:02:53.911036+0000","last_became_peered":"2024-09-18T08:02:53.911036+0000","last_unstale":"2024-09-18T08:06:46.137462+0000","last_undegraded":"2024-09-18T08:06:46.137462+0000","last_fullsized":"2024-09-18T08:06:46.137462+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:05:04.962686+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_clean_scrub_stamp":"2024-09-18T08:05:04.962686+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.4","version":"144'11738","reported_seq":11872,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-09-18T08:06:48.481310+0000","last_change":"2024-09-18T08:03:57.466551+0000","last_active":"2024-09-18T08:06:48.481310+0000","last_peered":"2024-09-18T08:06:48.481310+0000","last_clean":"2024-09-18T08:06:48.481310+0000","last_became_active":"2024-09-18T08:03:57.466232+0000","last_became_peered":"2024-09-18T08:03:57.466232+0000","last_unstale":"2024-09-18T08:06:48.481310+0000","last_undegraded":"2024-09-18T08:06:48.481310+0000","last_fullsized":"2024-09-18T08:06:48.481310+0000","mapping_epoch":83,"log_start":"109'6085","ondisk_log_start":"109'6085","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:03:56.449957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:03:56.449957+0000","last_clean_scrub_stamp":"2024-09-18T08:03:56.449957+0000","objects_scrubbed":0,"log_size":5653,"log_dups_size":0,"ondisk_log_size":5653,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":204865536,"num_objects":3126,"num_object_clones":0,"num_object_copies":9378,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3126,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11738,"num_write_kb":475648,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,1],"acting":[0,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.7","version":"69'13","reported_seq":127,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-09-18T08:06:46.135358+0000","last_change":"2024-09-18T08:04:18.164662+0000","last_active":"2024-09-18T08:06:46.135358+0000","last_peered":"2024-09-18T08:06:46.135358+0000","last_clean":"2024-09-18T08:06:46.135358+0000","last_became_active":"2024-09-18T08:04:18.144925+0000","last_became_peered":"2024-09-18T08:04:18.144925+0000","last_unstale":"2024-09-18T08:06:46.135358+0000","last_undegraded":"2024-09-18T08:06:46.135358+0000","last_fullsized":"2024-09-18T08:06:46.135358+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'13","last_scrub_stamp":"2024-09-18T08:03:50.700818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_clean_scrub_stamp":"2024-09-18T08:03:50.700818+0000","objects_scrubbed":0,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:07:29.027985+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":29,"num_read_kb":18,"num_write":16,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,0],"acting":[2,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":200,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-09-18T08:06:46.140508+0000","last_change":"2024-09-18T08:06:24.130858+0000","last_active":"2024-09-18T08:06:46.140508+0000","last_peered":"2024-09-18T08:06:46.140508+0000","last_clean":"2024-09-18T08:06:46.140508+0000","last_became_active":"2024-09-18T08:06:24.126334+0000","last_became_peered":"2024-09-18T08:06:24.126334+0000","last_unstale":"2024-09-18T08:06:46.140508+0000","last_undegraded":"2024-09-18T08:06:46.140508+0000","last_fullsized":"2024-09-18T08:06:46.140508+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:03:49.704004+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:46.860092+0000","last_clean_scrub_stamp":"2024-09-18T08:03:49.704004+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:07:31.622951+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"62'1","reported_seq":194,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-09-18T08:06:46.134174+0000","last_change":"2024-09-18T08:04:09.732724+0000","last_active":"2024-09-18T08:06:46.134174+0000","last_peered":"2024-09-18T08:06:46.134174+0000","last_clean":"2024-09-18T08:06:46.134174+0000","last_became_active":"2024-09-18T08:02:46.868175+0000","last_became_peered":"2024-09-18T08:02:46.868175+0000","last_unstale":"2024-09-18T08:06:46.134174+0000","last_undegraded":"2024-09-18T08:06:46.134174+0000","last_fullsized":"2024-09-18T08:06:46.134174+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"62'1","last_scrub_stamp":"2024-09-18T08:04:09.732645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:45.855979+0000","last_clean_scrub_stamp":"2024-09-18T08:04:09.732645+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0.00066785499999999997,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":185,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-09-18T08:06:46.136674+0000","last_change":"2024-09-18T08:05:41.059426+0000","last_active":"2024-09-18T08:06:46.136674+0000","last_peered":"2024-09-18T08:06:46.136674+0000","last_clean":"2024-09-18T08:06:46.136674+0000","last_became_active":"2024-09-18T08:02:51.908156+0000","last_became_peered":"2024-09-18T08:02:51.908156+0000","last_unstale":"2024-09-18T08:06:46.136674+0000","last_undegraded":"2024-09-18T08:06:46.136674+0000","last_fullsized":"2024-09-18T08:06:46.136674+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:05:41.059230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:50.886830+0000","last_clean_scrub_stamp":"2024-09-18T08:05:41.059230+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":190,"reported_epoch":145,"state":"active+clean","last_fresh":"2024-09-18T08:06:52.511312+0000","last_change":"2024-09-18T08:06:24.146209+0000","last_active":"2024-09-18T08:06:52.511312+0000","last_peered":"2024-09-18T08:06:52.511312+0000","last_clean":"2024-09-18T08:06:52.511312+0000","last_became_active":"2024-09-18T08:06:24.145382+0000","last_became_peered":"2024-09-18T08:06:24.145382+0000","last_unstale":"2024-09-18T08:06:52.511312+0000","last_undegraded":"2024-09-18T08:06:52.511312+0000","last_fullsized":"2024-09-18T08:06:52.511312+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:05:40.929345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_clean_scrub_stamp":"2024-09-18T08:05:40.929345+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:07:08.964083+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.5","version":"144'11611","reported_seq":11791,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-09-18T08:06:48.297765+0000","last_change":"2024-09-18T08:06:48.205596+0000","last_active":"2024-09-18T08:06:48.297765+0000","last_peered":"2024-09-18T08:06:48.297765+0000","last_clean":"2024-09-18T08:06:48.297765+0000","last_became_active":"2024-09-18T08:06:46.140892+0000","last_became_peered":"2024-09-18T08:06:46.140892+0000","last_unstale":"2024-09-18T08:06:48.297765+0000","last_undegraded":"2024-09-18T08:06:48.297765+0000","last_fullsized":"2024-09-18T08:06:48.297765+0000","mapping_epoch":143,"log_start":"110'6223","ondisk_log_start":"110'6223","created":83,"last_epoch_clean":144,"parent":"0.0","parent_split_bits":0,"last_scrub":"144'11604","last_scrub_stamp":"2024-09-18T08:06:48.205349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:03:56.449957+0000","last_clean_scrub_stamp":"2024-09-18T08:06:48.205349+0000","objects_scrubbed":3183,"log_size":5388,"log_dups_size":0,"ondisk_log_size":5388,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:08:17.112045+0000","scrub_duration":1267,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":205586432,"num_objects":3137,"num_object_clones":0,"num_object_copies":9411,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3137,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11611,"num_write_kb":471936,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,0],"acting":[2,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.6","version":"69'12","reported_seq":199,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-09-18T08:06:46.137824+0000","last_change":"2024-09-18T08:05:38.471012+0000","last_active":"2024-09-18T08:06:46.137824+0000","last_peered":"2024-09-18T08:06:46.137824+0000","last_clean":"2024-09-18T08:06:46.137824+0000","last_became_active":"2024-09-18T08:02:49.898079+0000","last_became_peered":"2024-09-18T08:02:49.898079+0000","last_unstale":"2024-09-18T08:06:46.137824+0000","last_undegraded":"2024-09-18T08:06:46.137824+0000","last_fullsized":"2024-09-18T08:06:46.137824+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'12","last_scrub_stamp":"2024-09-18T08:05:38.470713+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_clean_scrub_stamp":"2024-09-18T08:05:38.470713+0000","objects_scrubbed":6,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:06:50.866624+0000","scrub_duration":28,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":18,"num_read_kb":12,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":185,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-09-18T08:06:46.138290+0000","last_change":"2024-09-18T08:04:59.298230+0000","last_active":"2024-09-18T08:06:46.138290+0000","last_peered":"2024-09-18T08:06:46.138290+0000","last_clean":"2024-09-18T08:06:46.138290+0000","last_became_active":"2024-09-18T08:02:47.873809+0000","last_became_peered":"2024-09-18T08:02:47.873809+0000","last_unstale":"2024-09-18T08:06:46.138290+0000","last_undegraded":"2024-09-18T08:06:46.138290+0000","last_fullsized":"2024-09-18T08:06:46.138290+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:04:59.298029+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:46.860092+0000","last_clean_scrub_stamp":"2024-09-18T08:04:59.298029+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":31,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"70'5","reported_seq":59,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-09-18T08:06:48.145490+0000","last_change":"2024-09-18T08:06:46.142084+0000","last_active":"2024-09-18T08:06:48.145490+0000","last_peered":"2024-09-18T08:06:48.145490+0000","last_clean":"2024-09-18T08:06:48.145490+0000","last_became_active":"2024-09-18T08:06:46.141667+0000","last_became_peered":"2024-09-18T08:06:46.141667+0000","last_unstale":"2024-09-18T08:06:48.145490+0000","last_undegraded":"2024-09-18T08:06:48.145490+0000","last_fullsized":"2024-09-18T08:06:48.145490+0000","mapping_epoch":143,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":144,"parent":"0.0","parent_split_bits":0,"last_scrub":"70'5","last_scrub_stamp":"2024-09-18T08:04:06.814540+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:45.855979+0000","last_clean_scrub_stamp":"2024-09-18T08:04:06.814540+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000504055,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":9,"num_read_kb":4,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":187,"reported_epoch":145,"state":"active+clean","last_fresh":"2024-09-18T08:06:52.497021+0000","last_change":"2024-09-18T08:05:41.750515+0000","last_active":"2024-09-18T08:06:52.497021+0000","last_peered":"2024-09-18T08:06:52.497021+0000","last_clean":"2024-09-18T08:06:52.497021+0000","last_became_active":"2024-09-18T08:02:51.913112+0000","last_became_peered":"2024-09-18T08:02:51.913112+0000","last_unstale":"2024-09-18T08:06:52.497021+0000","last_undegraded":"2024-09-18T08:06:52.497021+0000","last_fullsized":"2024-09-18T08:06:52.497021+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:05:41.750446+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:50.886830+0000","last_clean_scrub_stamp":"2024-09-18T08:05:41.750446+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:06:55.866373+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":166,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-09-18T08:06:46.139096+0000","last_change":"2024-09-18T08:02:53.907689+0000","last_active":"2024-09-18T08:06:46.139096+0000","last_peered":"2024-09-18T08:06:46.139096+0000","last_clean":"2024-09-18T08:06:46.139096+0000","last_became_active":"2024-09-18T08:02:53.907404+0000","last_became_peered":"2024-09-18T08:02:53.907404+0000","last_unstale":"2024-09-18T08:06:46.139096+0000","last_undegraded":"2024-09-18T08:06:46.139096+0000","last_fullsized":"2024-09-18T08:06:46.139096+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_clean_scrub_stamp":"2024-09-18T08:02:52.890475+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:07:25.591758+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.6","version":"144'11605","reported_seq":11781,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-09-18T08:06:48.443433+0000","last_change":"2024-09-18T08:05:13.657923+0000","last_active":"2024-09-18T08:06:48.443433+0000","last_peered":"2024-09-18T08:06:48.443433+0000","last_clean":"2024-09-18T08:06:48.443433+0000","last_became_active":"2024-09-18T08:03:57.466391+0000","last_became_peered":"2024-09-18T08:03:57.466391+0000","last_unstale":"2024-09-18T08:06:48.443433+0000","last_undegraded":"2024-09-18T08:06:48.443433+0000","last_fullsized":"2024-09-18T08:06:48.443433+0000","mapping_epoch":83,"log_start":"110'6027","ondisk_log_start":"110'6027","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:03:56.449957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:03:56.449957+0000","last_clean_scrub_stamp":"2024-09-18T08:03:56.449957+0000","objects_scrubbed":3899,"log_size":5578,"log_dups_size":0,"ondisk_log_size":5578,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:07:05.494747+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":207683584,"num_objects":3169,"num_object_clones":0,"num_object_copies":9507,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3169,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11605,"num_write_kb":472768,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.5","version":"138'19","reported_seq":230,"reported_epoch":145,"state":"active+clean","last_fresh":"2024-09-18T08:06:52.488060+0000","last_change":"2024-09-18T08:06:24.191102+0000","last_active":"2024-09-18T08:06:52.488060+0000","last_peered":"2024-09-18T08:06:52.488060+0000","last_clean":"2024-09-18T08:06:52.488060+0000","last_became_active":"2024-09-18T08:06:24.146117+0000","last_became_peered":"2024-09-18T08:06:24.146117+0000","last_unstale":"2024-09-18T08:06:52.488060+0000","last_undegraded":"2024-09-18T08:06:52.488060+0000","last_fullsized":"2024-09-18T08:06:52.488060+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"130'18","last_scrub_stamp":"2024-09-18T08:06:15.455503+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_clean_scrub_stamp":"2024-09-18T08:06:15.455503+0000","objects_scrubbed":8,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:07:41.325892+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":24,"num_read_kb":14,"num_write":13,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":308,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,7],"acting":[4,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":199,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-09-18T08:06:46.139189+0000","last_change":"2024-09-18T08:06:24.134043+0000","last_active":"2024-09-18T08:06:46.139189+0000","last_peered":"2024-09-18T08:06:46.139189+0000","last_clean":"2024-09-18T08:06:46.139189+0000","last_became_active":"2024-09-18T08:06:24.133425+0000","last_became_peered":"2024-09-18T08:06:24.133425+0000","last_unstale":"2024-09-18T08:06:46.139189+0000","last_undegraded":"2024-09-18T08:06:46.139189+0000","last_fullsized":"2024-09-18T08:06:46.139189+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:05:38.080993+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:46.860092+0000","last_clean_scrub_stamp":"2024-09-18T08:05:38.080993+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:07:00.431736+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"74'79","reported_seq":261,"reported_epoch":145,"state":"active+clean","last_fresh":"2024-09-18T08:06:52.508989+0000","last_change":"2024-09-18T08:06:24.167946+0000","last_active":"2024-09-18T08:06:52.508989+0000","last_peered":"2024-09-18T08:06:52.508989+0000","last_clean":"2024-09-18T08:06:52.508989+0000","last_became_active":"2024-09-18T08:06:24.146679+0000","last_became_peered":"2024-09-18T08:06:24.146679+0000","last_unstale":"2024-09-18T08:06:52.508989+0000","last_undegraded":"2024-09-18T08:06:52.508989+0000","last_fullsized":"2024-09-18T08:06:52.508989+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'79","last_scrub_stamp":"2024-09-18T08:04:33.791841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:00:11.893742+0000","last_clean_scrub_stamp":"2024-09-18T08:04:33.791841+0000","objects_scrubbed":2,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:07:37.020534+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":231,"num_write":113,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1771104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":13,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-09-18T08:06:46.142055+0000","last_change":"2024-09-18T08:06:46.141364+0000","last_active":"2024-09-18T08:06:46.142055+0000","last_peered":"2024-09-18T08:06:46.142055+0000","last_clean":"2024-09-18T08:06:46.142055+0000","last_became_active":"2024-09-18T08:06:46.141046+0000","last_became_peered":"2024-09-18T08:06:46.141046+0000","last_unstale":"2024-09-18T08:06:46.142055+0000","last_undegraded":"2024-09-18T08:06:46.142055+0000","last_fullsized":"2024-09-18T08:06:46.142055+0000","mapping_epoch":143,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":144,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:05:42.929311+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:50.886830+0000","last_clean_scrub_stamp":"2024-09-18T08:05:42.929311+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:06:56.880704+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":184,"reported_epoch":145,"state":"active+clean","last_fresh":"2024-09-18T08:06:52.496971+0000","last_change":"2024-09-18T08:06:50.710637+0000","last_active":"2024-09-18T08:06:52.496971+0000","last_peered":"2024-09-18T08:06:52.496971+0000","last_clean":"2024-09-18T08:06:52.496971+0000","last_became_active":"2024-09-18T08:02:53.908668+0000","last_became_peered":"2024-09-18T08:02:53.908668+0000","last_unstale":"2024-09-18T08:06:52.496971+0000","last_undegraded":"2024-09-18T08:06:52.496971+0000","last_fullsized":"2024-09-18T08:06:52.496971+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:06:50.710565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_clean_scrub_stamp":"2024-09-18T08:06:50.710565+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:08:16.618092+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.d","version":"144'11589","reported_seq":11817,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-09-18T08:06:48.280389+0000","last_change":"2024-09-18T08:06:25.146940+0000","last_active":"2024-09-18T08:06:48.280389+0000","last_peered":"2024-09-18T08:06:48.280389+0000","last_clean":"2024-09-18T08:06:48.280389+0000","last_became_active":"2024-09-18T08:06:25.123627+0000","last_became_peered":"2024-09-18T08:06:25.123627+0000","last_unstale":"2024-09-18T08:06:48.280389+0000","last_undegraded":"2024-09-18T08:06:48.280389+0000","last_fullsized":"2024-09-18T08:06:48.280389+0000","mapping_epoch":137,"log_start":"115'6637","ondisk_log_start":"115'6637","created":83,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"123'7982","last_scrub_stamp":"2024-09-18T08:05:44.590884+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:03:56.449957+0000","last_clean_scrub_stamp":"2024-09-18T08:05:44.590884+0000","objects_scrubbed":6877,"log_size":4952,"log_dups_size":0,"ondisk_log_size":4952,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:07:00.648945+0000","scrub_duration":2853,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":208404504,"num_objects":3181,"num_object_clones":0,"num_object_copies":9543,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3181,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":11589,"num_write_kb":472577,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,0],"acting":[1,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.e","version":"69'11","reported_seq":240,"reported_epoch":145,"state":"active+clean","last_fresh":"2024-09-18T08:06:52.488624+0000","last_change":"2024-09-18T08:06:24.173542+0000","last_active":"2024-09-18T08:06:52.488624+0000","last_peered":"2024-09-18T08:06:52.488624+0000","last_clean":"2024-09-18T08:06:52.488624+0000","last_became_active":"2024-09-18T08:06:24.143839+0000","last_became_peered":"2024-09-18T08:06:24.143839+0000","last_unstale":"2024-09-18T08:06:52.488624+0000","last_undegraded":"2024-09-18T08:06:52.488624+0000","last_fullsized":"2024-09-18T08:06:52.488624+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'11","last_scrub_stamp":"2024-09-18T08:06:14.472168+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_clean_scrub_stamp":"2024-09-18T08:06:14.472168+0000","objects_scrubbed":5,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:07:20.674211+0000","scrub_duration":23,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":179,"reported_epoch":145,"state":"active+clean","last_fresh":"2024-09-18T08:06:52.489654+0000","last_change":"2024-09-18T08:02:47.880265+0000","last_active":"2024-09-18T08:06:52.489654+0000","last_peered":"2024-09-18T08:06:52.489654+0000","last_clean":"2024-09-18T08:06:52.489654+0000","last_became_active":"2024-09-18T08:02:47.879764+0000","last_became_peered":"2024-09-18T08:02:47.879764+0000","last_unstale":"2024-09-18T08:06:52.489654+0000","last_undegraded":"2024-09-18T08:06:52.489654+0000","last_fullsized":"2024-09-18T08:06:52.489654+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:02:46.860092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:46.860092+0000","last_clean_scrub_stamp":"2024-09-18T08:02:46.860092+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:06:53.348536+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":193,"reported_epoch":145,"state":"active+clean","last_fresh":"2024-09-18T08:06:52.496875+0000","last_change":"2024-09-18T08:06:24.132512+0000","last_active":"2024-09-18T08:06:52.496875+0000","last_peered":"2024-09-18T08:06:52.496875+0000","last_clean":"2024-09-18T08:06:52.496875+0000","last_became_active":"2024-09-18T08:06:24.132140+0000","last_became_peered":"2024-09-18T08:06:24.132140+0000","last_unstale":"2024-09-18T08:06:52.496875+0000","last_undegraded":"2024-09-18T08:06:52.496875+0000","last_fullsized":"2024-09-18T08:06:52.496875+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:05:08.839644+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:50.886830+0000","last_clean_scrub_stamp":"2024-09-18T08:05:08.839644+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:07:35.376189+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":187,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-09-18T08:06:46.136457+0000","last_change":"2024-09-18T08:06:24.130609+0000","last_active":"2024-09-18T08:06:46.136457+0000","last_peered":"2024-09-18T08:06:46.136457+0000","last_clean":"2024-09-18T08:06:46.136457+0000","last_became_active":"2024-09-18T08:06:24.130266+0000","last_became_peered":"2024-09-18T08:06:24.130266+0000","last_unstale":"2024-09-18T08:06:46.136457+0000","last_undegraded":"2024-09-18T08:06:46.136457+0000","last_fullsized":"2024-09-18T08:06:46.136457+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:05:09.038115+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_clean_scrub_stamp":"2024-09-18T08:05:09.038115+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.e","version":"144'11638","reported_seq":11890,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-09-18T08:06:48.232145+0000","last_change":"2024-09-18T08:05:47.424879+0000","last_active":"2024-09-18T08:06:48.232145+0000","last_peered":"2024-09-18T08:06:48.232145+0000","last_clean":"2024-09-18T08:06:48.232145+0000","last_became_active":"2024-09-18T08:03:57.464311+0000","last_became_peered":"2024-09-18T08:03:57.464311+0000","last_unstale":"2024-09-18T08:06:48.232145+0000","last_undegraded":"2024-09-18T08:06:48.232145+0000","last_fullsized":"2024-09-18T08:06:48.232145+0000","mapping_epoch":83,"log_start":"104'4739","ondisk_log_start":"104'4739","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:03:56.449957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:03:56.449957+0000","last_clean_scrub_stamp":"2024-09-18T08:03:56.449957+0000","objects_scrubbed":5849,"log_size":6899,"log_dups_size":0,"ondisk_log_size":6899,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:07:37.027598+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":205520896,"num_objects":3136,"num_object_clones":0,"num_object_copies":9408,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3136,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11638,"num_write_kb":472768,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,0],"acting":[2,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.d","version":"69'17","reported_seq":203,"reported_epoch":145,"state":"active+clean","last_fresh":"2024-09-18T08:06:52.488082+0000","last_change":"2024-09-18T08:02:49.895971+0000","last_active":"2024-09-18T08:06:52.488082+0000","last_peered":"2024-09-18T08:06:52.488082+0000","last_clean":"2024-09-18T08:06:52.488082+0000","last_became_active":"2024-09-18T08:02:49.895578+0000","last_became_peered":"2024-09-18T08:02:49.895578+0000","last_unstale":"2024-09-18T08:06:52.488082+0000","last_undegraded":"2024-09-18T08:06:52.488082+0000","last_fullsized":"2024-09-18T08:06:52.488082+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_clean_scrub_stamp":"2024-09-18T08:02:48.870718+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:07:02.375027+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":29,"num_read_kb":19,"num_write":18,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,1],"acting":[4,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":122,"reported_epoch":145,"state":"active+clean","last_fresh":"2024-09-18T08:06:52.486917+0000","last_change":"2024-09-18T08:06:24.144485+0000","last_active":"2024-09-18T08:06:52.486917+0000","last_peered":"2024-09-18T08:06:52.486917+0000","last_clean":"2024-09-18T08:06:52.486917+0000","last_became_active":"2024-09-18T08:06:24.143152+0000","last_became_peered":"2024-09-18T08:06:24.143152+0000","last_unstale":"2024-09-18T08:06:52.486917+0000","last_undegraded":"2024-09-18T08:06:52.486917+0000","last_fullsized":"2024-09-18T08:06:52.486917+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:05:12.622901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:46.860092+0000","last_clean_scrub_stamp":"2024-09-18T08:05:12.622901+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":185,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-09-18T08:06:46.139348+0000","last_change":"2024-09-18T08:05:31.851989+0000","last_active":"2024-09-18T08:06:46.139348+0000","last_peered":"2024-09-18T08:06:46.139348+0000","last_clean":"2024-09-18T08:06:46.139348+0000","last_became_active":"2024-09-18T08:02:51.907196+0000","last_became_peered":"2024-09-18T08:02:51.907196+0000","last_unstale":"2024-09-18T08:06:46.139348+0000","last_undegraded":"2024-09-18T08:06:46.139348+0000","last_fullsized":"2024-09-18T08:06:46.139348+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:05:31.851797+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:50.886830+0000","last_clean_scrub_stamp":"2024-09-18T08:05:31.851797+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:06:59.697560+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":174,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-09-18T08:06:46.135489+0000","last_change":"2024-09-18T08:04:06.709703+0000","last_active":"2024-09-18T08:06:46.135489+0000","last_peered":"2024-09-18T08:06:46.135489+0000","last_clean":"2024-09-18T08:06:46.135489+0000","last_became_active":"2024-09-18T08:02:53.904144+0000","last_became_peered":"2024-09-18T08:02:53.904144+0000","last_unstale":"2024-09-18T08:06:46.135489+0000","last_undegraded":"2024-09-18T08:06:46.135489+0000","last_fullsized":"2024-09-18T08:06:46.135489+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:04:06.709631+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_clean_scrub_stamp":"2024-09-18T08:04:06.709631+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:07:24.993615+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.f","version":"144'11600","reported_seq":11794,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-09-18T08:06:48.509117+0000","last_change":"2024-09-18T08:05:13.677385+0000","last_active":"2024-09-18T08:06:48.509117+0000","last_peered":"2024-09-18T08:06:48.509117+0000","last_clean":"2024-09-18T08:06:48.509117+0000","last_became_active":"2024-09-18T08:03:57.467980+0000","last_became_peered":"2024-09-18T08:03:57.467980+0000","last_unstale":"2024-09-18T08:06:48.509117+0000","last_undegraded":"2024-09-18T08:06:48.509117+0000","last_fullsized":"2024-09-18T08:06:48.509117+0000","mapping_epoch":83,"log_start":"108'5963","ondisk_log_start":"108'5963","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:03:56.449957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:03:56.449957+0000","last_clean_scrub_stamp":"2024-09-18T08:03:56.449957+0000","objects_scrubbed":5349,"log_size":5637,"log_dups_size":0,"ondisk_log_size":5637,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":201195520,"num_objects":3070,"num_object_clones":0,"num_object_copies":9210,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3070,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11600,"num_write_kb":469440,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.c","version":"69'10","reported_seq":236,"reported_epoch":145,"state":"active+clean","last_fresh":"2024-09-18T08:06:52.487260+0000","last_change":"2024-09-18T08:06:24.180438+0000","last_active":"2024-09-18T08:06:52.487260+0000","last_peered":"2024-09-18T08:06:52.487260+0000","last_clean":"2024-09-18T08:06:52.487260+0000","last_became_active":"2024-09-18T08:06:24.145005+0000","last_became_peered":"2024-09-18T08:06:24.145005+0000","last_unstale":"2024-09-18T08:06:52.487260+0000","last_undegraded":"2024-09-18T08:06:52.487260+0000","last_fullsized":"2024-09-18T08:06:52.487260+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'10","last_scrub_stamp":"2024-09-18T08:05:41.474813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_clean_scrub_stamp":"2024-09-18T08:05:41.474813+0000","objects_scrubbed":5,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:07:01.198872+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":193,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-09-18T08:06:46.339604+0000","last_change":"2024-09-18T08:06:46.339604+0000","last_active":"2024-09-18T08:06:46.339604+0000","last_peered":"2024-09-18T08:06:46.339604+0000","last_clean":"2024-09-18T08:06:46.339604+0000","last_became_active":"2024-09-18T08:02:47.881512+0000","last_became_peered":"2024-09-18T08:02:47.881512+0000","last_unstale":"2024-09-18T08:06:46.339604+0000","last_undegraded":"2024-09-18T08:06:46.339604+0000","last_fullsized":"2024-09-18T08:06:46.339604+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:06:46.339428+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:46.860092+0000","last_clean_scrub_stamp":"2024-09-18T08:06:46.339428+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:07:54.370307+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.d","version":"69'8","reported_seq":242,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-09-18T08:06:46.134238+0000","last_change":"2024-09-18T08:05:33.027988+0000","last_active":"2024-09-18T08:06:46.134238+0000","last_peered":"2024-09-18T08:06:46.134238+0000","last_clean":"2024-09-18T08:06:46.134238+0000","last_became_active":"2024-09-18T08:02:51.909011+0000","last_became_peered":"2024-09-18T08:02:51.909011+0000","last_unstale":"2024-09-18T08:06:46.134238+0000","last_undegraded":"2024-09-18T08:06:46.134238+0000","last_fullsized":"2024-09-18T08:06:46.134238+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'8","last_scrub_stamp":"2024-09-18T08:05:33.027783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:50.886830+0000","last_clean_scrub_stamp":"2024-09-18T08:05:33.027783+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":176,"reported_epoch":145,"state":"active+clean","last_fresh":"2024-09-18T08:06:52.488223+0000","last_change":"2024-09-18T08:04:32.377211+0000","last_active":"2024-09-18T08:06:52.488223+0000","last_peered":"2024-09-18T08:06:52.488223+0000","last_clean":"2024-09-18T08:06:52.488223+0000","last_became_active":"2024-09-18T08:02:53.901858+0000","last_became_peered":"2024-09-18T08:02:53.901858+0000","last_unstale":"2024-09-18T08:06:52.488223+0000","last_undegraded":"2024-09-18T08:06:52.488223+0000","last_fullsized":"2024-09-18T08:06:52.488223+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:04:32.377157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_clean_scrub_stamp":"2024-09-18T08:04:32.377157+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:07:07.380764+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.8","version":"144'11652","reported_seq":11830,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-09-18T08:06:48.269838+0000","last_change":"2024-09-18T08:06:43.340032+0000","last_active":"2024-09-18T08:06:48.269838+0000","last_peered":"2024-09-18T08:06:48.269838+0000","last_clean":"2024-09-18T08:06:48.269838+0000","last_became_active":"2024-09-18T08:03:57.464738+0000","last_became_peered":"2024-09-18T08:03:57.464738+0000","last_unstale":"2024-09-18T08:06:48.269838+0000","last_undegraded":"2024-09-18T08:06:48.269838+0000","last_fullsized":"2024-09-18T08:06:48.269838+0000","mapping_epoch":83,"log_start":"104'4827","ondisk_log_start":"104'4827","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"142'11403","last_scrub_stamp":"2024-09-18T08:06:43.339730+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:03:56.449957+0000","last_clean_scrub_stamp":"2024-09-18T08:06:43.339730+0000","objects_scrubbed":3493,"log_size":6825,"log_dups_size":0,"ondisk_log_size":6825,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:07:53.964339+0000","scrub_duration":1312,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":209584128,"num_objects":3198,"num_object_clones":0,"num_object_copies":9594,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3198,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11652,"num_write_kb":475200,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,4],"acting":[2,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.b","version":"69'9","reported_seq":197,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-09-18T08:06:46.137426+0000","last_change":"2024-09-18T08:05:05.294031+0000","last_active":"2024-09-18T08:06:46.137426+0000","last_peered":"2024-09-18T08:06:46.137426+0000","last_clean":"2024-09-18T08:06:46.137426+0000","last_became_active":"2024-09-18T08:02:49.897998+0000","last_became_peered":"2024-09-18T08:02:49.897998+0000","last_unstale":"2024-09-18T08:06:46.137426+0000","last_undegraded":"2024-09-18T08:06:46.137426+0000","last_fullsized":"2024-09-18T08:06:46.137426+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'9","last_scrub_stamp":"2024-09-18T08:05:05.293936+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_clean_scrub_stamp":"2024-09-18T08:05:05.293936+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":201,"reported_epoch":145,"state":"active+clean","last_fresh":"2024-09-18T08:06:52.496318+0000","last_change":"2024-09-18T08:06:24.135338+0000","last_active":"2024-09-18T08:06:52.496318+0000","last_peered":"2024-09-18T08:06:52.496318+0000","last_clean":"2024-09-18T08:06:52.496318+0000","last_became_active":"2024-09-18T08:06:24.135005+0000","last_became_peered":"2024-09-18T08:06:24.135005+0000","last_unstale":"2024-09-18T08:06:52.496318+0000","last_undegraded":"2024-09-18T08:06:52.496318+0000","last_fullsized":"2024-09-18T08:06:52.496318+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:04:59.080115+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:46.860092+0000","last_clean_scrub_stamp":"2024-09-18T08:04:59.080115+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:07:30.323111+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":185,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-09-18T08:06:46.139692+0000","last_change":"2024-09-18T08:05:43.068256+0000","last_active":"2024-09-18T08:06:46.139692+0000","last_peered":"2024-09-18T08:06:46.139692+0000","last_clean":"2024-09-18T08:06:46.139692+0000","last_became_active":"2024-09-18T08:02:51.901330+0000","last_became_peered":"2024-09-18T08:02:51.901330+0000","last_unstale":"2024-09-18T08:06:46.139692+0000","last_undegraded":"2024-09-18T08:06:46.139692+0000","last_fullsized":"2024-09-18T08:06:46.139692+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:05:43.067057+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:50.886830+0000","last_clean_scrub_stamp":"2024-09-18T08:05:43.067057+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:07:10.458536+0000","scrub_duration":40,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,3],"acting":[2,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":174,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-09-18T08:06:46.138202+0000","last_change":"2024-09-18T08:05:09.251072+0000","last_active":"2024-09-18T08:06:46.138202+0000","last_peered":"2024-09-18T08:06:46.138202+0000","last_clean":"2024-09-18T08:06:46.138202+0000","last_became_active":"2024-09-18T08:02:53.911068+0000","last_became_peered":"2024-09-18T08:02:53.911068+0000","last_unstale":"2024-09-18T08:06:46.138202+0000","last_undegraded":"2024-09-18T08:06:46.138202+0000","last_fullsized":"2024-09-18T08:06:46.138202+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:05:09.250843+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_clean_scrub_stamp":"2024-09-18T08:05:09.250843+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.9","version":"144'11603","reported_seq":11780,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-09-18T08:06:48.463703+0000","last_change":"2024-09-18T08:06:43.822074+0000","last_active":"2024-09-18T08:06:48.463703+0000","last_peered":"2024-09-18T08:06:48.463703+0000","last_clean":"2024-09-18T08:06:48.463703+0000","last_became_active":"2024-09-18T08:03:57.467065+0000","last_became_peered":"2024-09-18T08:03:57.467065+0000","last_unstale":"2024-09-18T08:06:48.463703+0000","last_undegraded":"2024-09-18T08:06:48.463703+0000","last_fullsized":"2024-09-18T08:06:48.463703+0000","mapping_epoch":83,"log_start":"109'5980","ondisk_log_start":"109'5980","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"142'11396","last_scrub_stamp":"2024-09-18T08:06:43.821646+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:03:56.449957+0000","last_clean_scrub_stamp":"2024-09-18T08:06:43.821646+0000","objects_scrubbed":3394,"log_size":5623,"log_dups_size":0,"ondisk_log_size":5623,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:07:54.095340+0000","scrub_duration":1188,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":206241792,"num_objects":3147,"num_object_clones":0,"num_object_copies":9441,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3147,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11603,"num_write_kb":472000,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.a","version":"69'19","reported_seq":202,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-09-18T08:06:46.138556+0000","last_change":"2024-09-18T08:06:24.162464+0000","last_active":"2024-09-18T08:06:46.138556+0000","last_peered":"2024-09-18T08:06:46.138556+0000","last_clean":"2024-09-18T08:06:46.138556+0000","last_became_active":"2024-09-18T08:06:24.139909+0000","last_became_peered":"2024-09-18T08:06:24.139909+0000","last_unstale":"2024-09-18T08:06:46.138556+0000","last_undegraded":"2024-09-18T08:06:46.138556+0000","last_fullsized":"2024-09-18T08:06:46.138556+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'19","last_scrub_stamp":"2024-09-18T08:04:20.391859+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_clean_scrub_stamp":"2024-09-18T08:04:20.391859+0000","objects_scrubbed":0,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:07:32.660944+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":31,"num_read_kb":20,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,7],"acting":[1,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":198,"reported_epoch":145,"state":"active+clean","last_fresh":"2024-09-18T08:06:52.496449+0000","last_change":"2024-09-18T08:06:47.638974+0000","last_active":"2024-09-18T08:06:52.496449+0000","last_peered":"2024-09-18T08:06:52.496449+0000","last_clean":"2024-09-18T08:06:52.496449+0000","last_became_active":"2024-09-18T08:06:46.171298+0000","last_became_peered":"2024-09-18T08:06:46.171298+0000","last_unstale":"2024-09-18T08:06:52.496449+0000","last_undegraded":"2024-09-18T08:06:52.496449+0000","last_fullsized":"2024-09-18T08:06:52.496449+0000","mapping_epoch":143,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":144,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:06:47.638911+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:46.860092+0000","last_clean_scrub_stamp":"2024-09-18T08:06:47.638911+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:07:53.269627+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":177,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-09-18T08:06:46.136043+0000","last_change":"2024-09-18T08:04:56.004261+0000","last_active":"2024-09-18T08:06:46.136043+0000","last_peered":"2024-09-18T08:06:46.136043+0000","last_clean":"2024-09-18T08:06:46.136043+0000","last_became_active":"2024-09-18T08:02:51.903439+0000","last_became_peered":"2024-09-18T08:02:51.903439+0000","last_unstale":"2024-09-18T08:06:46.136043+0000","last_undegraded":"2024-09-18T08:06:46.136043+0000","last_fullsized":"2024-09-18T08:06:46.136043+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:04:56.004150+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:50.886830+0000","last_clean_scrub_stamp":"2024-09-18T08:04:56.004150+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:06:59.006166+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":13,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-09-18T08:06:46.139106+0000","last_change":"2024-09-18T08:06:46.138160+0000","last_active":"2024-09-18T08:06:46.139106+0000","last_peered":"2024-09-18T08:06:46.139106+0000","last_clean":"2024-09-18T08:06:46.139106+0000","last_became_active":"2024-09-18T08:06:46.137872+0000","last_became_peered":"2024-09-18T08:06:46.137872+0000","last_unstale":"2024-09-18T08:06:46.139106+0000","last_undegraded":"2024-09-18T08:06:46.139106+0000","last_fullsized":"2024-09-18T08:06:46.139106+0000","mapping_epoch":143,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":144,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:04:17.811597+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_clean_scrub_stamp":"2024-09-18T08:04:17.811597+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,3],"acting":[2,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.a","version":"144'11764","reported_seq":11898,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-09-18T08:06:48.454414+0000","last_change":"2024-09-18T08:03:57.466483+0000","last_active":"2024-09-18T08:06:48.454414+0000","last_peered":"2024-09-18T08:06:48.454414+0000","last_clean":"2024-09-18T08:06:48.454414+0000","last_became_active":"2024-09-18T08:03:57.466168+0000","last_became_peered":"2024-09-18T08:03:57.466168+0000","last_unstale":"2024-09-18T08:06:48.454414+0000","last_undegraded":"2024-09-18T08:06:48.454414+0000","last_fullsized":"2024-09-18T08:06:48.454414+0000","mapping_epoch":83,"log_start":"119'7106","ondisk_log_start":"119'7106","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:03:56.449957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:03:56.449957+0000","last_clean_scrub_stamp":"2024-09-18T08:03:56.449957+0000","objects_scrubbed":0,"log_size":4658,"log_dups_size":0,"ondisk_log_size":4658,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:07:32.247056+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":206700544,"num_objects":3154,"num_object_clones":0,"num_object_copies":9462,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3154,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11764,"num_write_kb":477376,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.9","version":"69'12","reported_seq":214,"reported_epoch":145,"state":"active+clean","last_fresh":"2024-09-18T08:06:52.489412+0000","last_change":"2024-09-18T08:05:33.544197+0000","last_active":"2024-09-18T08:06:52.489412+0000","last_peered":"2024-09-18T08:06:52.489412+0000","last_clean":"2024-09-18T08:06:52.489412+0000","last_became_active":"2024-09-18T08:02:49.894762+0000","last_became_peered":"2024-09-18T08:02:49.894762+0000","last_unstale":"2024-09-18T08:06:52.489412+0000","last_undegraded":"2024-09-18T08:06:52.489412+0000","last_fullsized":"2024-09-18T08:06:52.489412+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'12","last_scrub_stamp":"2024-09-18T08:05:33.544135+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_clean_scrub_stamp":"2024-09-18T08:05:33.544135+0000","objects_scrubbed":5,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:06:52.810406+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":25,"num_read_kb":16,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":195,"reported_epoch":145,"state":"active+clean","last_fresh":"2024-09-18T08:06:52.510070+0000","last_change":"2024-09-18T08:05:39.906123+0000","last_active":"2024-09-18T08:06:52.510070+0000","last_peered":"2024-09-18T08:06:52.510070+0000","last_clean":"2024-09-18T08:06:52.510070+0000","last_became_active":"2024-09-18T08:02:47.884052+0000","last_became_peered":"2024-09-18T08:02:47.884052+0000","last_unstale":"2024-09-18T08:06:52.510070+0000","last_undegraded":"2024-09-18T08:06:52.510070+0000","last_fullsized":"2024-09-18T08:06:52.510070+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:05:39.906062+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:46.860092+0000","last_clean_scrub_stamp":"2024-09-18T08:05:39.906062+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:07:03.409405+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":177,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-09-18T08:06:46.135276+0000","last_change":"2024-09-18T08:05:06.828638+0000","last_active":"2024-09-18T08:06:46.135276+0000","last_peered":"2024-09-18T08:06:46.135276+0000","last_clean":"2024-09-18T08:06:46.135276+0000","last_became_active":"2024-09-18T08:02:51.902163+0000","last_became_peered":"2024-09-18T08:02:51.902163+0000","last_unstale":"2024-09-18T08:06:46.135276+0000","last_undegraded":"2024-09-18T08:06:46.135276+0000","last_fullsized":"2024-09-18T08:06:46.135276+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:05:06.828408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:50.886830+0000","last_clean_scrub_stamp":"2024-09-18T08:05:06.828408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":174,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-09-18T08:06:46.135360+0000","last_change":"2024-09-18T08:05:44.074994+0000","last_active":"2024-09-18T08:06:46.135360+0000","last_peered":"2024-09-18T08:06:46.135360+0000","last_clean":"2024-09-18T08:06:46.135360+0000","last_became_active":"2024-09-18T08:02:53.907412+0000","last_became_peered":"2024-09-18T08:02:53.907412+0000","last_unstale":"2024-09-18T08:06:46.135360+0000","last_undegraded":"2024-09-18T08:06:46.135360+0000","last_fullsized":"2024-09-18T08:06:46.135360+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:05:44.074555+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_clean_scrub_stamp":"2024-09-18T08:05:44.074555+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:07:10.508813+0000","scrub_duration":23,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.b","version":"144'11537","reported_seq":13607,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-09-18T08:06:48.273617+0000","last_change":"2024-09-18T08:06:42.616279+0000","last_active":"2024-09-18T08:06:48.273617+0000","last_peered":"2024-09-18T08:06:48.273617+0000","last_clean":"2024-09-18T08:06:48.273617+0000","last_became_active":"2024-09-18T08:06:24.157344+0000","last_became_peered":"2024-09-18T08:06:24.157344+0000","last_unstale":"2024-09-18T08:06:48.273617+0000","last_undegraded":"2024-09-18T08:06:48.273617+0000","last_fullsized":"2024-09-18T08:06:48.273617+0000","mapping_epoch":136,"log_start":"115'6526","ondisk_log_start":"115'6526","created":83,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"142'11259","last_scrub_stamp":"2024-09-18T08:06:42.615670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:03:56.449957+0000","last_clean_scrub_stamp":"2024-09-18T08:06:42.615670+0000","objects_scrubbed":3328,"log_size":5011,"log_dups_size":0,"ondisk_log_size":5011,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:08:03.658234+0000","scrub_duration":1074,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":198377472,"num_objects":3027,"num_object_clones":0,"num_object_copies":9081,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3027,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11537,"num_write_kb":466048,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":902,"num_bytes_recovered":58982400,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6],"acting":[1,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.8","version":"69'15","reported_seq":254,"reported_epoch":145,"state":"active+clean","last_fresh":"2024-09-18T08:06:52.495712+0000","last_change":"2024-09-18T08:06:24.162625+0000","last_active":"2024-09-18T08:06:52.495712+0000","last_peered":"2024-09-18T08:06:52.495712+0000","last_clean":"2024-09-18T08:06:52.495712+0000","last_became_active":"2024-09-18T08:06:24.138259+0000","last_became_peered":"2024-09-18T08:06:24.138259+0000","last_unstale":"2024-09-18T08:06:52.495712+0000","last_undegraded":"2024-09-18T08:06:52.495712+0000","last_fullsized":"2024-09-18T08:06:52.495712+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'15","last_scrub_stamp":"2024-09-18T08:04:26.859044+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_clean_scrub_stamp":"2024-09-18T08:04:26.859044+0000","objects_scrubbed":7,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:07:26.277667+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.f","version":"62'2","reported_seq":207,"reported_epoch":145,"state":"active+clean","last_fresh":"2024-09-18T08:06:52.510006+0000","last_change":"2024-09-18T08:05:11.906358+0000","last_active":"2024-09-18T08:06:52.510006+0000","last_peered":"2024-09-18T08:06:52.510006+0000","last_clean":"2024-09-18T08:06:52.510006+0000","last_became_active":"2024-09-18T08:02:47.881165+0000","last_became_peered":"2024-09-18T08:02:47.881165+0000","last_unstale":"2024-09-18T08:06:52.510006+0000","last_undegraded":"2024-09-18T08:06:52.510006+0000","last_fullsized":"2024-09-18T08:06:52.510006+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"62'2","last_scrub_stamp":"2024-09-18T08:05:11.906308+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:46.860092+0000","last_clean_scrub_stamp":"2024-09-18T08:05:11.906308+0000","objects_scrubbed":2,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:07:39.906870+0000","scrub_duration":24,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":4,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.9","version":"138'10","reported_seq":271,"reported_epoch":145,"state":"active+clean","last_fresh":"2024-09-18T08:06:52.509007+0000","last_change":"2024-09-18T08:06:24.176322+0000","last_active":"2024-09-18T08:06:52.509007+0000","last_peered":"2024-09-18T08:06:52.509007+0000","last_clean":"2024-09-18T08:06:52.509007+0000","last_became_active":"2024-09-18T08:06:24.170165+0000","last_became_peered":"2024-09-18T08:06:24.170165+0000","last_unstale":"2024-09-18T08:06:52.509007+0000","last_undegraded":"2024-09-18T08:06:52.509007+0000","last_fullsized":"2024-09-18T08:06:52.509007+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"130'9","last_scrub_stamp":"2024-09-18T08:06:14.178538+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:50.886830+0000","last_clean_scrub_stamp":"2024-09-18T08:06:14.178538+0000","objects_scrubbed":1,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:07:19.547201+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":189,"reported_epoch":145,"state":"active+clean","last_fresh":"2024-09-18T08:06:52.495808+0000","last_change":"2024-09-18T08:06:24.135990+0000","last_active":"2024-09-18T08:06:52.495808+0000","last_peered":"2024-09-18T08:06:52.495808+0000","last_clean":"2024-09-18T08:06:52.495808+0000","last_became_active":"2024-09-18T08:06:24.133119+0000","last_became_peered":"2024-09-18T08:06:24.133119+0000","last_unstale":"2024-09-18T08:06:52.495808+0000","last_undegraded":"2024-09-18T08:06:52.495808+0000","last_fullsized":"2024-09-18T08:06:52.495808+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:05:07.879501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_clean_scrub_stamp":"2024-09-18T08:05:07.879501+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:07:32.342623+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":111,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-09-18T08:06:46.137366+0000","last_change":"2024-09-18T08:06:24.133955+0000","last_active":"2024-09-18T08:06:46.137366+0000","last_peered":"2024-09-18T08:06:46.137366+0000","last_clean":"2024-09-18T08:06:46.137366+0000","last_became_active":"2024-09-18T08:06:24.133644+0000","last_became_peered":"2024-09-18T08:06:24.133644+0000","last_unstale":"2024-09-18T08:06:46.137366+0000","last_undegraded":"2024-09-18T08:06:46.137366+0000","last_fullsized":"2024-09-18T08:06:46.137366+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:04:16.378205+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:46.860092+0000","last_clean_scrub_stamp":"2024-09-18T08:04:16.378205+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:07:35.575551+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.17","version":"69'6","reported_seq":198,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-09-18T08:06:46.138292+0000","last_change":"2024-09-18T08:05:26.228344+0000","last_active":"2024-09-18T08:06:46.138292+0000","last_peered":"2024-09-18T08:06:46.138292+0000","last_clean":"2024-09-18T08:06:46.138292+0000","last_became_active":"2024-09-18T08:02:49.895067+0000","last_became_peered":"2024-09-18T08:02:49.895067+0000","last_unstale":"2024-09-18T08:06:46.138292+0000","last_undegraded":"2024-09-18T08:06:46.138292+0000","last_fullsized":"2024-09-18T08:06:46.138292+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'6","last_scrub_stamp":"2024-09-18T08:05:26.227814+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_clean_scrub_stamp":"2024-09-18T08:05:26.227814+0000","objects_scrubbed":3,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":18,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":9,"num_read_kb":6,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":187,"reported_epoch":145,"state":"active+clean","last_fresh":"2024-09-18T08:06:52.496823+0000","last_change":"2024-09-18T08:05:45.812436+0000","last_active":"2024-09-18T08:06:52.496823+0000","last_peered":"2024-09-18T08:06:52.496823+0000","last_clean":"2024-09-18T08:06:52.496823+0000","last_became_active":"2024-09-18T08:02:51.912557+0000","last_became_peered":"2024-09-18T08:02:51.912557+0000","last_unstale":"2024-09-18T08:06:52.496823+0000","last_undegraded":"2024-09-18T08:06:52.496823+0000","last_fullsized":"2024-09-18T08:06:52.496823+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:05:45.812389+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:50.886830+0000","last_clean_scrub_stamp":"2024-09-18T08:05:45.812389+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:07:01.523740+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,1],"acting":[5,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":198,"reported_epoch":145,"state":"active+clean","last_fresh":"2024-09-18T08:06:52.511219+0000","last_change":"2024-09-18T08:06:43.808497+0000","last_active":"2024-09-18T08:06:52.511219+0000","last_peered":"2024-09-18T08:06:52.511219+0000","last_clean":"2024-09-18T08:06:52.511219+0000","last_became_active":"2024-09-18T08:06:24.145909+0000","last_became_peered":"2024-09-18T08:06:24.145909+0000","last_unstale":"2024-09-18T08:06:52.511219+0000","last_undegraded":"2024-09-18T08:06:52.511219+0000","last_fullsized":"2024-09-18T08:06:52.511219+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:06:43.808433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_clean_scrub_stamp":"2024-09-18T08:06:43.808433+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:07:52.050872+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"69'9","reported_seq":197,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-09-18T08:06:46.137925+0000","last_change":"2024-09-18T08:05:00.265939+0000","last_active":"2024-09-18T08:06:46.137925+0000","last_peered":"2024-09-18T08:06:46.137925+0000","last_clean":"2024-09-18T08:06:46.137925+0000","last_became_active":"2024-09-18T08:02:49.891567+0000","last_became_peered":"2024-09-18T08:02:49.891567+0000","last_unstale":"2024-09-18T08:06:46.137925+0000","last_undegraded":"2024-09-18T08:06:46.137925+0000","last_fullsized":"2024-09-18T08:06:46.137925+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'9","last_scrub_stamp":"2024-09-18T08:05:00.265738+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_clean_scrub_stamp":"2024-09-18T08:05:00.265738+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":17,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":198,"reported_epoch":145,"state":"active+clean","last_fresh":"2024-09-18T08:06:52.489298+0000","last_change":"2024-09-18T08:06:46.171693+0000","last_active":"2024-09-18T08:06:52.489298+0000","last_peered":"2024-09-18T08:06:52.489298+0000","last_clean":"2024-09-18T08:06:52.489298+0000","last_became_active":"2024-09-18T08:06:46.171291+0000","last_became_peered":"2024-09-18T08:06:46.171291+0000","last_unstale":"2024-09-18T08:06:52.489298+0000","last_undegraded":"2024-09-18T08:06:52.489298+0000","last_fullsized":"2024-09-18T08:06:52.489298+0000","mapping_epoch":143,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":144,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:06:13.214705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:46.860092+0000","last_clean_scrub_stamp":"2024-09-18T08:06:13.214705+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:07:42.434590+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":177,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-09-18T08:06:46.138153+0000","last_change":"2024-09-18T08:05:42.061649+0000","last_active":"2024-09-18T08:06:46.138153+0000","last_peered":"2024-09-18T08:06:46.138153+0000","last_clean":"2024-09-18T08:06:46.138153+0000","last_became_active":"2024-09-18T08:02:51.907361+0000","last_became_peered":"2024-09-18T08:02:51.907361+0000","last_unstale":"2024-09-18T08:06:46.138153+0000","last_undegraded":"2024-09-18T08:06:46.138153+0000","last_fullsized":"2024-09-18T08:06:46.138153+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:05:42.061452+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:50.886830+0000","last_clean_scrub_stamp":"2024-09-18T08:05:42.061452+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:07:05.139088+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":174,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-09-18T08:06:46.134365+0000","last_change":"2024-09-18T08:05:07.780899+0000","last_active":"2024-09-18T08:06:46.134365+0000","last_peered":"2024-09-18T08:06:46.134365+0000","last_clean":"2024-09-18T08:06:46.134365+0000","last_became_active":"2024-09-18T08:02:53.908917+0000","last_became_peered":"2024-09-18T08:02:53.908917+0000","last_unstale":"2024-09-18T08:06:46.134365+0000","last_undegraded":"2024-09-18T08:06:46.134365+0000","last_fullsized":"2024-09-18T08:06:46.134365+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:05:07.780816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_clean_scrub_stamp":"2024-09-18T08:05:07.780816+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.15","version":"69'9","reported_seq":207,"reported_epoch":145,"state":"active+clean","last_fresh":"2024-09-18T08:06:52.496540+0000","last_change":"2024-09-18T08:05:36.896299+0000","last_active":"2024-09-18T08:06:52.496540+0000","last_peered":"2024-09-18T08:06:52.496540+0000","last_clean":"2024-09-18T08:06:52.496540+0000","last_became_active":"2024-09-18T08:02:49.890497+0000","last_became_peered":"2024-09-18T08:02:49.890497+0000","last_unstale":"2024-09-18T08:06:52.496540+0000","last_undegraded":"2024-09-18T08:06:52.496540+0000","last_fullsized":"2024-09-18T08:06:52.496540+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'9","last_scrub_stamp":"2024-09-18T08:05:36.896238+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_clean_scrub_stamp":"2024-09-18T08:05:36.896238+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":193,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-09-18T08:06:46.137366+0000","last_change":"2024-09-18T08:06:12.437630+0000","last_active":"2024-09-18T08:06:46.137366+0000","last_peered":"2024-09-18T08:06:46.137366+0000","last_clean":"2024-09-18T08:06:46.137366+0000","last_became_active":"2024-09-18T08:02:47.875983+0000","last_became_peered":"2024-09-18T08:02:47.875983+0000","last_unstale":"2024-09-18T08:06:46.137366+0000","last_undegraded":"2024-09-18T08:06:46.137366+0000","last_fullsized":"2024-09-18T08:06:46.137366+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:06:12.437584+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:46.860092+0000","last_clean_scrub_stamp":"2024-09-18T08:06:12.437584+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:07:30.735772+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.14","version":"69'8","reported_seq":234,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-09-18T08:06:46.136633+0000","last_change":"2024-09-18T08:05:01.083803+0000","last_active":"2024-09-18T08:06:46.136633+0000","last_peered":"2024-09-18T08:06:46.136633+0000","last_clean":"2024-09-18T08:06:46.136633+0000","last_became_active":"2024-09-18T08:02:51.909513+0000","last_became_peered":"2024-09-18T08:02:51.909513+0000","last_unstale":"2024-09-18T08:06:46.136633+0000","last_undegraded":"2024-09-18T08:06:46.136633+0000","last_fullsized":"2024-09-18T08:06:46.136633+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'8","last_scrub_stamp":"2024-09-18T08:05:01.082570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:50.886830+0000","last_clean_scrub_stamp":"2024-09-18T08:05:01.082570+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":17,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":184,"reported_epoch":145,"state":"active+clean","last_fresh":"2024-09-18T08:06:52.488427+0000","last_change":"2024-09-18T08:06:43.510575+0000","last_active":"2024-09-18T08:06:52.488427+0000","last_peered":"2024-09-18T08:06:52.488427+0000","last_clean":"2024-09-18T08:06:52.488427+0000","last_became_active":"2024-09-18T08:02:53.907428+0000","last_became_peered":"2024-09-18T08:02:53.907428+0000","last_unstale":"2024-09-18T08:06:52.488427+0000","last_undegraded":"2024-09-18T08:06:52.488427+0000","last_fullsized":"2024-09-18T08:06:52.488427+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:06:43.510513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_clean_scrub_stamp":"2024-09-18T08:06:43.510513+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:08:09.391404+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.14","version":"69'10","reported_seq":215,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-09-18T08:06:46.137027+0000","last_change":"2024-09-18T08:04:18.172364+0000","last_active":"2024-09-18T08:06:46.137027+0000","last_peered":"2024-09-18T08:06:46.137027+0000","last_clean":"2024-09-18T08:06:46.137027+0000","last_became_active":"2024-09-18T08:04:18.150968+0000","last_became_peered":"2024-09-18T08:04:18.150968+0000","last_unstale":"2024-09-18T08:06:46.137027+0000","last_undegraded":"2024-09-18T08:06:46.137027+0000","last_fullsized":"2024-09-18T08:06:46.137027+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'10","last_scrub_stamp":"2024-09-18T08:04:16.940778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_clean_scrub_stamp":"2024-09-18T08:04:16.940778+0000","objects_scrubbed":5,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:06:54.101039+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,7],"acting":[3,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":195,"reported_epoch":145,"state":"active+clean","last_fresh":"2024-09-18T08:06:52.509650+0000","last_change":"2024-09-18T08:06:46.782184+0000","last_active":"2024-09-18T08:06:52.509650+0000","last_peered":"2024-09-18T08:06:52.509650+0000","last_clean":"2024-09-18T08:06:52.509650+0000","last_became_active":"2024-09-18T08:02:47.882065+0000","last_became_peered":"2024-09-18T08:02:47.882065+0000","last_unstale":"2024-09-18T08:06:52.509650+0000","last_undegraded":"2024-09-18T08:06:52.509650+0000","last_fullsized":"2024-09-18T08:06:52.509650+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:06:46.782109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:46.860092+0000","last_clean_scrub_stamp":"2024-09-18T08:06:46.782109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:08:11.695710+0000","scrub_duration":21,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.15","version":"69'8","reported_seq":253,"reported_epoch":145,"state":"active+clean","last_fresh":"2024-09-18T08:06:52.496330+0000","last_change":"2024-09-18T08:06:45.555513+0000","last_active":"2024-09-18T08:06:52.496330+0000","last_peered":"2024-09-18T08:06:52.496330+0000","last_clean":"2024-09-18T08:06:52.496330+0000","last_became_active":"2024-09-18T08:02:51.914135+0000","last_became_peered":"2024-09-18T08:02:51.914135+0000","last_unstale":"2024-09-18T08:06:52.496330+0000","last_undegraded":"2024-09-18T08:06:52.496330+0000","last_fullsized":"2024-09-18T08:06:52.496330+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'8","last_scrub_stamp":"2024-09-18T08:06:45.555459+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:50.886830+0000","last_clean_scrub_stamp":"2024-09-18T08:06:45.555459+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:07:48.944224+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":174,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-09-18T08:06:46.137411+0000","last_change":"2024-09-18T08:05:03.302755+0000","last_active":"2024-09-18T08:06:46.137411+0000","last_peered":"2024-09-18T08:06:46.137411+0000","last_clean":"2024-09-18T08:06:46.137411+0000","last_became_active":"2024-09-18T08:02:53.912112+0000","last_became_peered":"2024-09-18T08:02:53.912112+0000","last_unstale":"2024-09-18T08:06:46.137411+0000","last_undegraded":"2024-09-18T08:06:46.137411+0000","last_fullsized":"2024-09-18T08:06:46.137411+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:05:03.302444+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_clean_scrub_stamp":"2024-09-18T08:05:03.302444+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":16,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.13","version":"69'11","reported_seq":240,"reported_epoch":145,"state":"active+clean","last_fresh":"2024-09-18T08:06:52.489604+0000","last_change":"2024-09-18T08:06:24.189911+0000","last_active":"2024-09-18T08:06:52.489604+0000","last_peered":"2024-09-18T08:06:52.489604+0000","last_clean":"2024-09-18T08:06:52.489604+0000","last_became_active":"2024-09-18T08:06:24.150800+0000","last_became_peered":"2024-09-18T08:06:24.150800+0000","last_unstale":"2024-09-18T08:06:52.489604+0000","last_undegraded":"2024-09-18T08:06:52.489604+0000","last_fullsized":"2024-09-18T08:06:52.489604+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'11","last_scrub_stamp":"2024-09-18T08:04:34.451379+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_clean_scrub_stamp":"2024-09-18T08:04:34.451379+0000","objects_scrubbed":5,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:07:35.515469+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":209,"reported_epoch":145,"state":"active+clean","last_fresh":"2024-09-18T08:06:52.487421+0000","last_change":"2024-09-18T08:06:46.502473+0000","last_active":"2024-09-18T08:06:52.487421+0000","last_peered":"2024-09-18T08:06:52.487421+0000","last_clean":"2024-09-18T08:06:52.487421+0000","last_became_active":"2024-09-18T08:06:24.145458+0000","last_became_peered":"2024-09-18T08:06:24.145458+0000","last_unstale":"2024-09-18T08:06:52.487421+0000","last_undegraded":"2024-09-18T08:06:52.487421+0000","last_fullsized":"2024-09-18T08:06:52.487421+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:06:46.502422+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:46.860092+0000","last_clean_scrub_stamp":"2024-09-18T08:06:46.502422+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:08:03.119365+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":185,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-09-18T08:06:46.138451+0000","last_change":"2024-09-18T08:05:29.896334+0000","last_active":"2024-09-18T08:06:46.138451+0000","last_peered":"2024-09-18T08:06:46.138451+0000","last_clean":"2024-09-18T08:06:46.138451+0000","last_became_active":"2024-09-18T08:02:51.910752+0000","last_became_peered":"2024-09-18T08:02:51.910752+0000","last_unstale":"2024-09-18T08:06:46.138451+0000","last_undegraded":"2024-09-18T08:06:46.138451+0000","last_fullsized":"2024-09-18T08:06:46.138451+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:05:29.888927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:50.886830+0000","last_clean_scrub_stamp":"2024-09-18T08:05:29.888927+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:06:47.803144+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":182,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-09-18T08:06:46.137205+0000","last_change":"2024-09-18T08:06:11.249403+0000","last_active":"2024-09-18T08:06:46.137205+0000","last_peered":"2024-09-18T08:06:46.137205+0000","last_clean":"2024-09-18T08:06:46.137205+0000","last_became_active":"2024-09-18T08:02:53.911429+0000","last_became_peered":"2024-09-18T08:02:53.911429+0000","last_unstale":"2024-09-18T08:06:46.137205+0000","last_undegraded":"2024-09-18T08:06:46.137205+0000","last_fullsized":"2024-09-18T08:06:46.137205+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:06:11.249317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_clean_scrub_stamp":"2024-09-18T08:06:11.249317+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:07:33.181678+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.12","version":"69'9","reported_seq":223,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-09-18T08:06:46.138780+0000","last_change":"2024-09-18T08:06:24.152803+0000","last_active":"2024-09-18T08:06:46.138780+0000","last_peered":"2024-09-18T08:06:46.138780+0000","last_clean":"2024-09-18T08:06:46.138780+0000","last_became_active":"2024-09-18T08:06:24.132270+0000","last_became_peered":"2024-09-18T08:06:24.132270+0000","last_unstale":"2024-09-18T08:06:46.138780+0000","last_undegraded":"2024-09-18T08:06:46.138780+0000","last_fullsized":"2024-09-18T08:06:46.138780+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_clean_scrub_stamp":"2024-09-18T08:02:48.870718+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:07:28.580662+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":195,"reported_epoch":145,"state":"active+clean","last_fresh":"2024-09-18T08:06:52.511492+0000","last_change":"2024-09-18T08:05:30.820244+0000","last_active":"2024-09-18T08:06:52.511492+0000","last_peered":"2024-09-18T08:06:52.511492+0000","last_clean":"2024-09-18T08:06:52.511492+0000","last_became_active":"2024-09-18T08:02:47.882096+0000","last_became_peered":"2024-09-18T08:02:47.882096+0000","last_unstale":"2024-09-18T08:06:52.511492+0000","last_undegraded":"2024-09-18T08:06:52.511492+0000","last_fullsized":"2024-09-18T08:06:52.511492+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:05:30.820200+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:46.860092+0000","last_clean_scrub_stamp":"2024-09-18T08:05:30.820200+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:06:54.287921+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":177,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-09-18T08:06:46.135438+0000","last_change":"2024-09-18T08:05:43.165522+0000","last_active":"2024-09-18T08:06:46.135438+0000","last_peered":"2024-09-18T08:06:46.135438+0000","last_clean":"2024-09-18T08:06:46.135438+0000","last_became_active":"2024-09-18T08:02:51.906277+0000","last_became_peered":"2024-09-18T08:02:51.906277+0000","last_unstale":"2024-09-18T08:06:46.135438+0000","last_undegraded":"2024-09-18T08:06:46.135438+0000","last_fullsized":"2024-09-18T08:06:46.135438+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:05:43.165258+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:50.886830+0000","last_clean_scrub_stamp":"2024-09-18T08:05:43.165258+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:07:08.715609+0000","scrub_duration":89,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.10","version":"69'1","reported_seq":170,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-09-18T08:06:46.136905+0000","last_change":"2024-09-18T08:05:51.429631+0000","last_active":"2024-09-18T08:06:46.136905+0000","last_peered":"2024-09-18T08:06:46.136905+0000","last_clean":"2024-09-18T08:06:46.136905+0000","last_became_active":"2024-09-18T08:02:53.912699+0000","last_became_peered":"2024-09-18T08:02:53.912699+0000","last_unstale":"2024-09-18T08:06:46.136905+0000","last_undegraded":"2024-09-18T08:06:46.136905+0000","last_fullsized":"2024-09-18T08:06:46.136905+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_clean_scrub_stamp":"2024-09-18T08:02:52.890475+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:07:25.467898+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.11","version":"69'11","reported_seq":246,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-09-18T08:06:46.137359+0000","last_change":"2024-09-18T08:06:24.154365+0000","last_active":"2024-09-18T08:06:46.137359+0000","last_peered":"2024-09-18T08:06:46.137359+0000","last_clean":"2024-09-18T08:06:46.137359+0000","last_became_active":"2024-09-18T08:06:24.132416+0000","last_became_peered":"2024-09-18T08:06:24.132416+0000","last_unstale":"2024-09-18T08:06:46.137359+0000","last_undegraded":"2024-09-18T08:06:46.137359+0000","last_fullsized":"2024-09-18T08:06:46.137359+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'11","last_scrub_stamp":"2024-09-18T08:05:13.037615+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_clean_scrub_stamp":"2024-09-18T08:05:13.037615+0000","objects_scrubbed":5,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":195,"reported_epoch":145,"state":"active+clean","last_fresh":"2024-09-18T08:06:52.496878+0000","last_change":"2024-09-18T08:06:18.168254+0000","last_active":"2024-09-18T08:06:52.496878+0000","last_peered":"2024-09-18T08:06:52.496878+0000","last_clean":"2024-09-18T08:06:52.496878+0000","last_became_active":"2024-09-18T08:02:47.878886+0000","last_became_peered":"2024-09-18T08:02:47.878886+0000","last_unstale":"2024-09-18T08:06:52.496878+0000","last_undegraded":"2024-09-18T08:06:52.496878+0000","last_fullsized":"2024-09-18T08:06:52.496878+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:06:18.168206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:46.860092+0000","last_clean_scrub_stamp":"2024-09-18T08:06:18.168206+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:07:28.641116+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":201,"reported_epoch":145,"state":"active+clean","last_fresh":"2024-09-18T08:06:52.508383+0000","last_change":"2024-09-18T08:06:24.148922+0000","last_active":"2024-09-18T08:06:52.508383+0000","last_peered":"2024-09-18T08:06:52.508383+0000","last_clean":"2024-09-18T08:06:52.508383+0000","last_became_active":"2024-09-18T08:06:24.148302+0000","last_became_peered":"2024-09-18T08:06:24.148302+0000","last_unstale":"2024-09-18T08:06:52.508383+0000","last_undegraded":"2024-09-18T08:06:52.508383+0000","last_fullsized":"2024-09-18T08:06:52.508383+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:06:15.177302+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:50.886830+0000","last_clean_scrub_stamp":"2024-09-18T08:06:15.177302+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:07:19.110873+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":187,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-09-18T08:06:46.135582+0000","last_change":"2024-09-18T08:06:24.128970+0000","last_active":"2024-09-18T08:06:46.135582+0000","last_peered":"2024-09-18T08:06:46.135582+0000","last_clean":"2024-09-18T08:06:46.135582+0000","last_became_active":"2024-09-18T08:06:24.128582+0000","last_became_peered":"2024-09-18T08:06:24.128582+0000","last_unstale":"2024-09-18T08:06:46.135582+0000","last_undegraded":"2024-09-18T08:06:46.135582+0000","last_fullsized":"2024-09-18T08:06:46.135582+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:06:13.241940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_clean_scrub_stamp":"2024-09-18T08:06:13.241940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:07:27.180968+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.10","version":"69'4","reported_seq":213,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-09-18T08:06:46.134706+0000","last_change":"2024-09-18T08:06:24.151823+0000","last_active":"2024-09-18T08:06:46.134706+0000","last_peered":"2024-09-18T08:06:46.134706+0000","last_clean":"2024-09-18T08:06:46.134706+0000","last_became_active":"2024-09-18T08:06:24.131309+0000","last_became_peered":"2024-09-18T08:06:24.131309+0000","last_unstale":"2024-09-18T08:06:46.134706+0000","last_undegraded":"2024-09-18T08:06:46.134706+0000","last_fullsized":"2024-09-18T08:06:46.134706+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'4","last_scrub_stamp":"2024-09-18T08:04:12.999978+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_clean_scrub_stamp":"2024-09-18T08:04:12.999978+0000","objects_scrubbed":2,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:07:35.247320+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":6,"num_read_kb":4,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":193,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-09-18T08:06:46.139319+0000","last_change":"2024-09-18T08:06:10.457663+0000","last_active":"2024-09-18T08:06:46.139319+0000","last_peered":"2024-09-18T08:06:46.139319+0000","last_clean":"2024-09-18T08:06:46.139319+0000","last_became_active":"2024-09-18T08:02:47.877616+0000","last_became_peered":"2024-09-18T08:02:47.877616+0000","last_unstale":"2024-09-18T08:06:46.139319+0000","last_undegraded":"2024-09-18T08:06:46.139319+0000","last_fullsized":"2024-09-18T08:06:46.139319+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:06:10.457502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:46.860092+0000","last_clean_scrub_stamp":"2024-09-18T08:06:10.457502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:07:37.803172+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":121,"reported_epoch":145,"state":"active+clean","last_fresh":"2024-09-18T08:06:52.488895+0000","last_change":"2024-09-18T08:06:24.168175+0000","last_active":"2024-09-18T08:06:52.488895+0000","last_peered":"2024-09-18T08:06:52.488895+0000","last_clean":"2024-09-18T08:06:52.488895+0000","last_became_active":"2024-09-18T08:06:24.166698+0000","last_became_peered":"2024-09-18T08:06:24.166698+0000","last_unstale":"2024-09-18T08:06:52.488895+0000","last_undegraded":"2024-09-18T08:06:52.488895+0000","last_fullsized":"2024-09-18T08:06:52.488895+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:05:11.591275+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:50.886830+0000","last_clean_scrub_stamp":"2024-09-18T08:05:11.591275+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,7],"acting":[4,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":192,"reported_epoch":145,"state":"active+clean","last_fresh":"2024-09-18T08:06:52.509250+0000","last_change":"2024-09-18T08:06:47.791172+0000","last_active":"2024-09-18T08:06:52.509250+0000","last_peered":"2024-09-18T08:06:52.509250+0000","last_clean":"2024-09-18T08:06:52.509250+0000","last_became_active":"2024-09-18T08:02:53.909146+0000","last_became_peered":"2024-09-18T08:02:53.909146+0000","last_unstale":"2024-09-18T08:06:52.509250+0000","last_undegraded":"2024-09-18T08:06:52.509250+0000","last_fullsized":"2024-09-18T08:06:52.509250+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:06:47.791091+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_clean_scrub_stamp":"2024-09-18T08:06:47.791091+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:08:13.139318+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,4],"acting":[7,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":166,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-09-18T08:06:46.139211+0000","last_change":"2024-09-18T08:02:53.907055+0000","last_active":"2024-09-18T08:06:46.139211+0000","last_peered":"2024-09-18T08:06:46.139211+0000","last_clean":"2024-09-18T08:06:46.139211+0000","last_became_active":"2024-09-18T08:02:53.906726+0000","last_became_peered":"2024-09-18T08:02:53.906726+0000","last_unstale":"2024-09-18T08:06:46.139211+0000","last_undegraded":"2024-09-18T08:06:46.139211+0000","last_fullsized":"2024-09-18T08:06:46.139211+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_clean_scrub_stamp":"2024-09-18T08:02:52.890475+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:07:27.599829+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":185,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-09-18T08:06:46.135029+0000","last_change":"2024-09-18T08:05:02.063591+0000","last_active":"2024-09-18T08:06:46.135029+0000","last_peered":"2024-09-18T08:06:46.135029+0000","last_clean":"2024-09-18T08:06:46.135029+0000","last_became_active":"2024-09-18T08:02:47.878922+0000","last_became_peered":"2024-09-18T08:02:47.878922+0000","last_unstale":"2024-09-18T08:06:46.135029+0000","last_undegraded":"2024-09-18T08:06:46.135029+0000","last_fullsized":"2024-09-18T08:06:46.135029+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:05:02.063478+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:46.860092+0000","last_clean_scrub_stamp":"2024-09-18T08:05:02.063478+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"69'11","reported_seq":185,"reported_epoch":145,"state":"active+clean","last_fresh":"2024-09-18T08:06:52.496971+0000","last_change":"2024-09-18T08:06:24.164086+0000","last_active":"2024-09-18T08:06:52.496971+0000","last_peered":"2024-09-18T08:06:52.496971+0000","last_clean":"2024-09-18T08:06:52.496971+0000","last_became_active":"2024-09-18T08:06:24.135232+0000","last_became_peered":"2024-09-18T08:06:24.135232+0000","last_unstale":"2024-09-18T08:06:52.496971+0000","last_undegraded":"2024-09-18T08:06:52.496971+0000","last_fullsized":"2024-09-18T08:06:52.496971+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'11","last_scrub_stamp":"2024-09-18T08:06:12.967462+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_clean_scrub_stamp":"2024-09-18T08:06:12.967462+0000","objects_scrubbed":5,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:07:25.928591+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":19,"num_read_kb":12,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,1],"acting":[5,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":177,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-09-18T08:06:46.137669+0000","last_change":"2024-09-18T08:04:26.117117+0000","last_active":"2024-09-18T08:06:46.137669+0000","last_peered":"2024-09-18T08:06:46.137669+0000","last_clean":"2024-09-18T08:06:46.137669+0000","last_became_active":"2024-09-18T08:02:51.900300+0000","last_became_peered":"2024-09-18T08:02:51.900300+0000","last_unstale":"2024-09-18T08:06:46.137669+0000","last_undegraded":"2024-09-18T08:06:46.137669+0000","last_fullsized":"2024-09-18T08:06:46.137669+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:04:26.117051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:50.886830+0000","last_clean_scrub_stamp":"2024-09-18T08:04:26.117051+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":7,"num_pg":16,"stat_sum":{"num_bytes":3253338136,"num_objects":49643,"num_object_clones":0,"num_object_copies":148929,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":49643,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":186935,"num_write_kb":7570433,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2629,"num_bytes_recovered":172097536,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":9517936640,"data_stored":9517924424,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":86484,"ondisk_log_size":86484,"up":48,"acting":48,"num_store_stats":8},{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":416,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":71,"ondisk_log_size":71,"up":96,"acting":96,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":178,"num_object_clones":0,"num_object_copies":534,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":178,"num_whiteouts":0,"num_read":694,"num_read_kb":451,"num_write":417,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":149,"num_bytes_recovered":2508,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":417792,"data_stored":11106,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":396,"ondisk_log_size":396,"up":96,"acting":96,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":1613,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":24,"num_read_kb":24,"num_write":10,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":4839,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":6,"ondisk_log_size":6,"up":96,"acting":96,"num_store_stats":8},{"poolid":2,"num_pg":3,"stat_sum":{"num_bytes":408,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":9,"num_read_kb":4,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":38,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":8,"ondisk_log_size":8,"up":9,"acting":9,"num_store_stats":7},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":231,"num_write":113,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1771104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":2375680,"data_stored":2361472,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":79,"ondisk_log_size":79,"up":3,"acting":3,"num_store_stats":4}],"osd_stats":[{"osd":7,"up_from":56,"seq":240518168629,"num_pgs":58,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1033364,"kb_used_data":925824,"kb_used_omap":1,"kb_used_meta":107518,"kb_avail":92711788,"statfs":{"total":95995035648,"available":94936870912,"internally_reserved":0,"allocated":948043776,"data_stored":947663077,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":110098890},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1],"upper_bound":4},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":47,"seq":201863462971,"num_pgs":48,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1029956,"kb_used_data":922416,"kb_used_omap":1,"kb_used_meta":107518,"kb_avail":92715196,"statfs":{"total":95995035648,"available":94940360704,"internally_reserved":0,"allocated":944553984,"data_stored":944188608,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":110098886},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1],"upper_bound":4},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":39,"seq":167503724609,"num_pgs":52,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":802828,"kb_used_data":727224,"kb_used_omap":1,"kb_used_meta":75582,"kb_avail":92942324,"statfs":{"total":95995035648,"available":95172939776,"internally_reserved":0,"allocated":744677376,"data_stored":744304158,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":77396422},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":32,"seq":137438953542,"num_pgs":63,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1032156,"kb_used_data":924616,"kb_used_omap":1,"kb_used_meta":107518,"kb_avail":92712996,"statfs":{"total":95995035648,"available":94938107904,"internally_reserved":0,"allocated":946806784,"data_stored":946417862,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":110098885},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2],"upper_bound":8},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":26,"seq":111669149772,"num_pgs":65,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1949456,"kb_used_data":1814144,"kb_used_omap":1,"kb_used_meta":135294,"kb_avail":91795696,"statfs":{"total":95995035648,"available":93998792704,"internally_reserved":0,"allocated":1857683456,"data_stored":1857295163,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":138541512},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411410,"num_pgs":44,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1321464,"kb_used_data":1222248,"kb_used_omap":1,"kb_used_meta":99198,"kb_avail":92423688,"statfs":{"total":95995035648,"available":94641856512,"internally_reserved":0,"allocated":1251581952,"data_stored":1251218258,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":101579205},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574934,"num_pgs":61,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1337260,"kb_used_data":1229660,"kb_used_omap":1,"kb_used_meta":107518,"kb_avail":92407892,"statfs":{"total":95995035648,"available":94625681408,"internally_reserved":0,"allocated":1259171840,"data_stored":1258756795,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":110098885},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738461,"num_pgs":54,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1758484,"kb_used_data":1637060,"kb_used_omap":1,"kb_used_meta":121406,"kb_avail":91986668,"statfs":{"total":95995035648,"available":94194348032,"internally_reserved":0,"allocated":1676349440,"data_stored":1675958528,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":124320197},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1131,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1521,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":436,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":46,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1177,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":436,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":1898,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1128,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":1898,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1474,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":1100,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1034,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1254,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":403,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":416,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":403,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":1643909120,"data_stored":1643905048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":1237716992,"data_stored":1237712920,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":1241907200,"data_stored":1241907200,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":1851260928,"data_stored":1851260928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":931921920,"data_stored":931921920,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":738852864,"data_stored":738852864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":934219776,"data_stored":934215704,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":938147840,"data_stored":938147840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T08:06:56.507 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid 779349ca-7593-11ef-bceb-c7b262605968 -- ceph osd pool set .mgr pgp_num 1 2024-09-18T08:06:56.561 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4 2024-09-18T08:06:56.587 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:06:56.588 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:06:56.589 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4' 2024-09-18T08:06:56.689 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6 2024-09-18T08:06:56.717 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:06:56.717 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:06:56.718 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6' 2024-09-18T08:06:56.730 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:06:56 smithi007 ceph-mon[29459]: pgmap v184: 148 pgs: 148 active+clean; 2.8 GiB data, 9.3 GiB used, 706 GiB / 715 GiB avail; 0 B/s wr, 976 op/s 2024-09-18T08:06:56.730 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:06:56 smithi007 ceph-mon[29459]: purged_snaps scrub starts 2024-09-18T08:06:56.730 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:06:56 smithi007 ceph-mon[29459]: purged_snaps scrub ok 2024-09-18T08:06:56.731 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Sep 18 08:06:56 smithi007 ceph-779349ca-7593-11ef-bceb-c7b262605968-mgr-y[29688]: ::ffff:172.21.15.50 - - [18/Sep/2024:08:06:56] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-09-18T08:06:56.731 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:06:56 smithi007 ceph-mon[39249]: pgmap v184: 148 pgs: 148 active+clean; 2.8 GiB data, 9.3 GiB used, 706 GiB / 715 GiB avail; 0 B/s wr, 976 op/s 2024-09-18T08:06:56.731 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:06:56 smithi007 ceph-mon[39249]: purged_snaps scrub starts 2024-09-18T08:06:56.731 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:06:56 smithi007 ceph-mon[39249]: purged_snaps scrub ok 2024-09-18T08:06:56.780 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:06:56 smithi050 ceph-mon[32371]: pgmap v184: 148 pgs: 148 active+clean; 2.8 GiB data, 9.3 GiB used, 706 GiB / 715 GiB avail; 0 B/s wr, 976 op/s 2024-09-18T08:06:56.780 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:06:56 smithi050 ceph-mon[32371]: purged_snaps scrub starts 2024-09-18T08:06:56.780 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:06:56 smithi050 ceph-mon[32371]: purged_snaps scrub ok 2024-09-18T08:06:56.819 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid 779349ca-7593-11ef-bceb-c7b262605968 -- ceph daemon osd.0 dump_ops_in_flight 2024-09-18T08:06:56.819 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6 2024-09-18T08:06:56.845 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:06:56.845 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:06:56.846 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6' 2024-09-18T08:06:56.947 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2 2024-09-18T08:06:57.006 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:06:57.006 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:06:57.007 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2' 2024-09-18T08:06:57.016 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/779349ca-7593-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T08:06:57.108 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4 2024-09-18T08:06:57.134 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:06:57.134 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:06:57.135 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4' 2024-09-18T08:06:57.140 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid 779349ca-7593-11ef-bceb-c7b262605968 -- ceph osd set nodeep-scrub 2024-09-18T08:06:57.236 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4 2024-09-18T08:06:57.260 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:06:57.260 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:06:57.261 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4' 2024-09-18T08:06:57.362 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5 2024-09-18T08:06:57.387 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:06:57.387 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:06:57.388 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5' 2024-09-18T08:06:57.488 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7 2024-09-18T08:06:57.512 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:06:57.513 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:06:57.513 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7' 2024-09-18T08:06:57.614 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7 2024-09-18T08:06:57.637 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:06:57.638 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:06:57.638 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7' 2024-09-18T08:06:57.739 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7 2024-09-18T08:06:57.766 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:06:57.766 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:06:57.767 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7' 2024-09-18T08:06:57.787 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:06:57 smithi007 ceph-mon[29459]: osdmap e146: 8 total, 8 up, 8 in 2024-09-18T08:06:57.788 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:06:57 smithi007 ceph-mon[39249]: osdmap e146: 8 total, 8 up, 8 in 2024-09-18T08:06:57.820 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid 779349ca-7593-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-18T08:06:57.868 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5 2024-09-18T08:06:57.893 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:06:57.893 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:06:57.894 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5' 2024-09-18T08:06:57.905 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:06:57 smithi050 ceph-mon[32371]: osdmap e146: 8 total, 8 up, 8 in 2024-09-18T08:06:57.995 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4 2024-09-18T08:06:58.011 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/779349ca-7593-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T08:06:58.017 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:06:58.018 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:06:58.018 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4' 2024-09-18T08:06:58.032 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/779349ca-7593-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T08:06:58.119 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1 2024-09-18T08:06:58.175 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:06:58.175 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:06:58.175 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1' 2024-09-18T08:06:58.276 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3 2024-09-18T08:06:58.325 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:06:58.326 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:06:58.326 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3' 2024-09-18T08:06:58.427 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2 2024-09-18T08:06:58.473 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:06:58.473 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:06:58.473 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2' 2024-09-18T08:06:58.545 INFO:teuthology.orchestra.run.smithi007.stderr:set pool 1 pgp_num to 1 2024-09-18T08:06:58.574 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0 2024-09-18T08:06:58.625 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:06:58.625 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:06:58.626 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0' 2024-09-18T08:06:58.727 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2 2024-09-18T08:06:58.730 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:06:58 smithi007 ceph-mon[29459]: pgmap v186: 148 pgs: 148 active+clean; 2.8 GiB data, 9.3 GiB used, 706 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 611 op/s 2024-09-18T08:06:58.730 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:06:58 smithi007 ceph-mon[29459]: from='client.? 172.21.15.7:0/3274242996' entity='client.admin' cmd={"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num", "val": "1"} : dispatch 2024-09-18T08:06:58.730 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:06:58 smithi007 ceph-mon[29459]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num", "val": "1"} : dispatch 2024-09-18T08:06:58.731 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:06:58 smithi007 ceph-mon[39249]: pgmap v186: 148 pgs: 148 active+clean; 2.8 GiB data, 9.3 GiB used, 706 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 611 op/s 2024-09-18T08:06:58.731 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:06:58 smithi007 ceph-mon[39249]: from='client.? 172.21.15.7:0/3274242996' entity='client.admin' cmd={"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num", "val": "1"} : dispatch 2024-09-18T08:06:58.731 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:06:58 smithi007 ceph-mon[39249]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num", "val": "1"} : dispatch 2024-09-18T08:06:58.761 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:06:58.763 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:06:58.763 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2' 2024-09-18T08:06:58.864 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2 2024-09-18T08:06:58.906 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:06:58 smithi050 ceph-mon[32371]: pgmap v186: 148 pgs: 148 active+clean; 2.8 GiB data, 9.3 GiB used, 706 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 611 op/s 2024-09-18T08:06:58.906 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:06:58 smithi050 ceph-mon[32371]: from='client.? 172.21.15.7:0/3274242996' entity='client.admin' cmd={"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num", "val": "1"} : dispatch 2024-09-18T08:06:58.906 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:06:58 smithi050 ceph-mon[32371]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num", "val": "1"} : dispatch 2024-09-18T08:06:58.909 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:06:58.910 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:06:58.911 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2' 2024-09-18T08:06:59.011 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3 2024-09-18T08:06:59.049 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:06:59.050 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:06:59.051 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3' 2024-09-18T08:06:59.151 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1 2024-09-18T08:06:59.188 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:06:59.188 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:06:59.188 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1' 2024-09-18T08:06:59.289 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0 2024-09-18T08:06:59.326 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:06:59.327 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:06:59.328 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0' 2024-09-18T08:06:59.405 INFO:journalctl@ceph.iscsi.iscsi.a.smithi050.stdout:Sep 18 08:06:58 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-iscsi-iscsi-a[57782]: debug there is no tcmu-runner data available 2024-09-18T08:06:59.429 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0 2024-09-18T08:06:59.479 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:06:59.480 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:06:59.480 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0' 2024-09-18T08:06:59.570 INFO:teuthology.orchestra.run.smithi007.stdout:ERROR: (22) Invalid argument 2024-09-18T08:06:59.570 INFO:teuthology.orchestra.run.smithi007.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-18T08:06:59.581 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4 2024-09-18T08:06:59.607 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:06:59.608 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:06:59.608 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4' 2024-09-18T08:06:59.709 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2 2024-09-18T08:06:59.752 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:06:59.753 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:06:59.753 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2' 2024-09-18T08:06:59.768 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:06:59 smithi007 ceph-mon[29459]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num", "val": "1"}]': finished 2024-09-18T08:06:59.768 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:06:59 smithi007 ceph-mon[29459]: osdmap e147: 8 total, 8 up, 8 in 2024-09-18T08:06:59.768 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:06:59 smithi007 ceph-mon[29459]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-18T08:06:59.768 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:06:59 smithi007 ceph-mon[29459]: pgmap v188: 148 pgs: 148 active+clean; 2.5 GiB data, 8.3 GiB used, 707 GiB / 715 GiB avail; 1.5 KiB/s rd, 0 B/s wr, 1.48k op/s 2024-09-18T08:06:59.768 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:06:59 smithi007 ceph-mon[39249]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num", "val": "1"}]': finished 2024-09-18T08:06:59.768 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:06:59 smithi007 ceph-mon[39249]: osdmap e147: 8 total, 8 up, 8 in 2024-09-18T08:06:59.768 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:06:59 smithi007 ceph-mon[39249]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-18T08:06:59.768 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:06:59 smithi007 ceph-mon[39249]: pgmap v188: 148 pgs: 148 active+clean; 2.5 GiB data, 8.3 GiB used, 707 GiB / 715 GiB avail; 1.5 KiB/s rd, 0 B/s wr, 1.48k op/s 2024-09-18T08:06:59.854 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4 2024-09-18T08:06:59.879 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:06:59.880 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:06:59.880 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4' 2024-09-18T08:06:59.906 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:06:59 smithi050 ceph-mon[32371]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num", "val": "1"}]': finished 2024-09-18T08:06:59.906 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:06:59 smithi050 ceph-mon[32371]: osdmap e147: 8 total, 8 up, 8 in 2024-09-18T08:06:59.906 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:06:59 smithi050 ceph-mon[32371]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-18T08:06:59.906 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:06:59 smithi050 ceph-mon[32371]: pgmap v188: 148 pgs: 148 active+clean; 2.5 GiB data, 8.3 GiB used, 707 GiB / 715 GiB avail; 1.5 KiB/s rd, 0 B/s wr, 1.48k op/s 2024-09-18T08:06:59.981 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2 2024-09-18T08:07:00.038 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:07:00.039 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:07:00.039 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2' 2024-09-18T08:07:00.140 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7 2024-09-18T08:07:00.165 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:07:00.166 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:07:00.166 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7' 2024-09-18T08:07:00.267 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1 2024-09-18T08:07:00.304 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:07:00.305 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:07:00.305 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1' 2024-09-18T08:07:00.406 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2 2024-09-18T08:07:00.453 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:07:00.453 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:07:00.454 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2' 2024-09-18T08:07:00.458 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/779349ca-7593-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T08:07:00.469 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid 779349ca-7593-11ef-bceb-c7b262605968 -- ceph daemon osd.0 dump_blocked_ops 2024-09-18T08:07:00.555 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6 2024-09-18T08:07:00.565 INFO:teuthology.orchestra.run.smithi007.stderr:nodeep-scrub is set 2024-09-18T08:07:00.581 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:07:00.581 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:07:00.582 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6' 2024-09-18T08:07:00.682 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4 2024-09-18T08:07:00.708 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:07:00.709 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:07:00.709 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4' 2024-09-18T08:07:00.730 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:07:00 smithi007 ceph-mon[39249]: from='client.? 172.21.15.7:0/2400309436' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-18T08:07:00.730 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:07:00 smithi007 ceph-mon[29459]: from='client.? 172.21.15.7:0/2400309436' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-18T08:07:00.810 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1 2024-09-18T08:07:00.850 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:07:00.851 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:07:00.851 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1' 2024-09-18T08:07:00.905 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:07:00 smithi050 ceph-mon[32371]: from='client.? 172.21.15.7:0/2400309436' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-18T08:07:00.952 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7 2024-09-18T08:07:00.981 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:07:00.982 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:07:00.982 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7' 2024-09-18T08:07:01.083 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5 2024-09-18T08:07:01.113 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:07:01.114 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:07:01.114 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5' 2024-09-18T08:07:01.215 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4 2024-09-18T08:07:01.240 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:07:01.241 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:07:01.241 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4' 2024-09-18T08:07:01.342 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6 2024-09-18T08:07:01.369 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:07:01.370 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:07:01.370 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6' 2024-09-18T08:07:01.471 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1 2024-09-18T08:07:01.513 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:07:01.514 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:07:01.514 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1' 2024-09-18T08:07:01.615 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4 2024-09-18T08:07:01.643 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:07:01.644 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:07:01.644 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4' 2024-09-18T08:07:01.745 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3 2024-09-18T08:07:01.794 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:07:01.794 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:07:01.795 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3' 2024-09-18T08:07:01.891 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:07:01 smithi007 ceph-mon[39249]: purged_snaps scrub starts 2024-09-18T08:07:01.891 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:07:01 smithi007 ceph-mon[39249]: purged_snaps scrub ok 2024-09-18T08:07:01.891 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:07:01 smithi007 ceph-mon[39249]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-18T08:07:01.891 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:07:01 smithi007 ceph-mon[39249]: from='client.? 172.21.15.7:0/2400309436' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-18T08:07:01.891 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:07:01 smithi007 ceph-mon[39249]: osdmap e148: 8 total, 8 up, 8 in 2024-09-18T08:07:01.891 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:07:01 smithi007 ceph-mon[39249]: pgmap v190: 148 pgs: 148 active+clean; 2.5 GiB data, 8.3 GiB used, 707 GiB / 715 GiB avail; 853 B/s rd, 0 B/s wr, 824 op/s 2024-09-18T08:07:01.892 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:07:01 smithi007 ceph-mon[29459]: purged_snaps scrub starts 2024-09-18T08:07:01.892 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:07:01 smithi007 ceph-mon[29459]: purged_snaps scrub ok 2024-09-18T08:07:01.892 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:07:01 smithi007 ceph-mon[29459]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-18T08:07:01.892 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:07:01 smithi007 ceph-mon[29459]: from='client.? 172.21.15.7:0/2400309436' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-18T08:07:01.892 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:07:01 smithi007 ceph-mon[29459]: osdmap e148: 8 total, 8 up, 8 in 2024-09-18T08:07:01.892 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:07:01 smithi007 ceph-mon[29459]: pgmap v190: 148 pgs: 148 active+clean; 2.5 GiB data, 8.3 GiB used, 707 GiB / 715 GiB avail; 853 B/s rd, 0 B/s wr, 824 op/s 2024-09-18T08:07:01.895 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7 2024-09-18T08:07:01.906 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:07:01 smithi050 ceph-mon[32371]: purged_snaps scrub starts 2024-09-18T08:07:01.906 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:07:01 smithi050 ceph-mon[32371]: purged_snaps scrub ok 2024-09-18T08:07:01.906 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:07:01 smithi050 ceph-mon[32371]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-18T08:07:01.906 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:07:01 smithi050 ceph-mon[32371]: from='client.? 172.21.15.7:0/2400309436' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-18T08:07:01.906 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:07:01 smithi050 ceph-mon[32371]: osdmap e148: 8 total, 8 up, 8 in 2024-09-18T08:07:01.906 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:07:01 smithi050 ceph-mon[32371]: pgmap v190: 148 pgs: 148 active+clean; 2.5 GiB data, 8.3 GiB used, 707 GiB / 715 GiB avail; 853 B/s rd, 0 B/s wr, 824 op/s 2024-09-18T08:07:01.920 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:07:01.921 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:07:01.921 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7' 2024-09-18T08:07:02.010 INFO:teuthology.orchestra.run.smithi007.stdout:osd.0: {} 2024-09-18T08:07:02.011 INFO:teuthology.orchestra.run.smithi007.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-18T08:07:02.022 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6 2024-09-18T08:07:02.042 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/779349ca-7593-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T08:07:02.046 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:07:02.047 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:07:02.047 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6' 2024-09-18T08:07:02.049 INFO:teuthology.orchestra.run.smithi007.stdout:osd.1: {} 2024-09-18T08:07:02.050 INFO:teuthology.orchestra.run.smithi007.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-18T08:07:02.082 INFO:teuthology.orchestra.run.smithi007.stdout:osd.2: {} 2024-09-18T08:07:02.083 INFO:teuthology.orchestra.run.smithi007.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-18T08:07:02.124 INFO:teuthology.orchestra.run.smithi007.stdout:osd.3: {} 2024-09-18T08:07:02.124 INFO:teuthology.orchestra.run.smithi007.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-18T08:07:02.148 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1 2024-09-18T08:07:02.155 INFO:teuthology.orchestra.run.smithi007.stdout:osd.4: {} 2024-09-18T08:07:02.155 INFO:teuthology.orchestra.run.smithi007.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-18T08:07:02.194 INFO:teuthology.orchestra.run.smithi007.stdout:osd.5: {} 2024-09-18T08:07:02.195 INFO:teuthology.orchestra.run.smithi007.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-18T08:07:02.196 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:07:02.196 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:07:02.196 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1' 2024-09-18T08:07:02.227 INFO:teuthology.orchestra.run.smithi007.stdout:osd.6: {} 2024-09-18T08:07:02.228 INFO:teuthology.orchestra.run.smithi007.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-18T08:07:02.260 INFO:teuthology.orchestra.run.smithi007.stdout:osd.7: {} 2024-09-18T08:07:02.260 INFO:teuthology.orchestra.run.smithi007.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-18T08:07:02.297 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6 2024-09-18T08:07:02.322 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:07:02.323 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:07:02.323 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6' 2024-09-18T08:07:02.424 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3 2024-09-18T08:07:02.465 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:07:02.511 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:07:02.512 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3' 2024-09-18T08:07:02.613 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1 2024-09-18T08:07:02.657 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:07:02.658 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:07:02.658 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1' 2024-09-18T08:07:02.759 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7 2024-09-18T08:07:02.784 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:07:02.785 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:07:02.785 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7' 2024-09-18T08:07:02.830 INFO:teuthology.orchestra.run.smithi007.stdout:{ 2024-09-18T08:07:02.830 INFO:teuthology.orchestra.run.smithi007.stdout: "ops": [], 2024-09-18T08:07:02.830 INFO:teuthology.orchestra.run.smithi007.stdout: "complaint_time": 30, 2024-09-18T08:07:02.830 INFO:teuthology.orchestra.run.smithi007.stdout: "num_blocked_ops": 0 2024-09-18T08:07:02.830 INFO:teuthology.orchestra.run.smithi007.stdout:} 2024-09-18T08:07:02.886 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5 2024-09-18T08:07:02.905 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:07:02 smithi050 ceph-mon[32371]: from='client.? 172.21.15.7:0/688643415' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-18T08:07:02.916 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:07:02.916 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:07:02.917 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5' 2024-09-18T08:07:02.980 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:07:02 smithi007 ceph-mon[29459]: from='client.? 172.21.15.7:0/688643415' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-18T08:07:02.981 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:07:02 smithi007 ceph-mon[39249]: from='client.? 172.21.15.7:0/688643415' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-18T08:07:03.017 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7 2024-09-18T08:07:03.046 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:07:03.046 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:07:03.047 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7' 2024-09-18T08:07:03.147 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1 2024-09-18T08:07:03.191 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:07:03.192 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:07:03.192 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1' 2024-09-18T08:07:03.293 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2 2024-09-18T08:07:03.323 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:07:03.323 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:07:03.324 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2' 2024-09-18T08:07:03.424 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1 2024-09-18T08:07:03.466 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:07:03.467 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:07:03.467 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1' 2024-09-18T08:07:03.568 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2 2024-09-18T08:07:03.607 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:07:03.607 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:07:03.608 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2' 2024-09-18T08:07:03.690 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid 779349ca-7593-11ef-bceb-c7b262605968 -- ceph osd unset noscrub 2024-09-18T08:07:03.709 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5 2024-09-18T08:07:03.736 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:07:03.737 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:07:03.738 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5' 2024-09-18T08:07:03.838 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2 2024-09-18T08:07:03.897 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:07:03.897 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:07:03.898 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2' 2024-09-18T08:07:03.981 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:07:03 smithi007 ceph-mon[29459]: from='mgr.24470 172.21.15.7:0/2594390261' entity='mgr.y' 2024-09-18T08:07:03.981 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:07:03 smithi007 ceph-mon[29459]: from='mgr.24470 172.21.15.7:0/2594390261' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-18T08:07:03.981 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:07:03 smithi007 ceph-mon[29459]: pgmap v191: 148 pgs: 148 active+clean; 2.4 GiB data, 8.0 GiB used, 707 GiB / 715 GiB avail; 772 B/s rd, 0 B/s wr, 839 op/s 2024-09-18T08:07:03.981 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:07:03 smithi007 ceph-mon[39249]: from='mgr.24470 172.21.15.7:0/2594390261' entity='mgr.y' 2024-09-18T08:07:03.981 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:07:03 smithi007 ceph-mon[39249]: from='mgr.24470 172.21.15.7:0/2594390261' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-18T08:07:03.981 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:07:03 smithi007 ceph-mon[39249]: pgmap v191: 148 pgs: 148 active+clean; 2.4 GiB data, 8.0 GiB used, 707 GiB / 715 GiB avail; 772 B/s rd, 0 B/s wr, 839 op/s 2024-09-18T08:07:03.999 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0 2024-09-18T08:07:04.053 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:07:04.055 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:07:04.055 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0' 2024-09-18T08:07:04.155 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1 2024-09-18T08:07:04.157 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:07:03 smithi050 ceph-mon[32371]: from='mgr.24470 172.21.15.7:0/2594390261' entity='mgr.y' 2024-09-18T08:07:04.157 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:07:03 smithi050 ceph-mon[32371]: from='mgr.24470 172.21.15.7:0/2594390261' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-18T08:07:04.157 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:07:03 smithi050 ceph-mon[32371]: pgmap v191: 148 pgs: 148 active+clean; 2.4 GiB data, 8.0 GiB used, 707 GiB / 715 GiB avail; 772 B/s rd, 0 B/s wr, 839 op/s 2024-09-18T08:07:04.196 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:07:04.196 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:07:04.197 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1' 2024-09-18T08:07:04.298 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7 2024-09-18T08:07:04.322 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:07:04.323 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:07:04.323 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7' 2024-09-18T08:07:04.331 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 7, 6] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-09-18T08:07:04.331 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-18T08:07:04.331 INFO:tasks.thrashosds.thrasher:inject_pause on osd.4 2024-09-18T08:07:04.331 INFO:tasks.thrashosds.thrasher:Testing filestore_inject_stall pause injection for duration 3 2024-09-18T08:07:04.331 INFO:tasks.thrashosds.thrasher:Checking after 0, should_be_down=False 2024-09-18T08:07:04.332 DEBUG:teuthology.orchestra.run.smithi050:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid 779349ca-7593-11ef-bceb-c7b262605968 -- ceph daemon osd.4 config set filestore_inject_stall 3 2024-09-18T08:07:04.424 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0 2024-09-18T08:07:04.462 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:07:04.463 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:07:04.463 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0' 2024-09-18T08:07:04.564 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1 2024-09-18T08:07:04.606 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:07:04.607 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:07:04.607 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1' 2024-09-18T08:07:04.704 INFO:teuthology.orchestra.run.smithi050.stderr:Inferring config /var/lib/ceph/779349ca-7593-11ef-bceb-c7b262605968/mon.b/config 2024-09-18T08:07:04.708 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7 2024-09-18T08:07:04.748 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:07:04.749 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:07:04.749 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7' 2024-09-18T08:07:04.850 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6 2024-09-18T08:07:04.877 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:07:04.878 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:07:04.878 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6' 2024-09-18T08:07:04.980 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2 2024-09-18T08:07:05.019 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:07:05.020 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:07:05.020 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2' 2024-09-18T08:07:05.121 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7 2024-09-18T08:07:05.182 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:07:05.183 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:07:05.184 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7' 2024-09-18T08:07:05.284 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1 2024-09-18T08:07:05.328 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:07:05.329 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:07:05.329 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1' 2024-09-18T08:07:05.382 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2024-09-18T08:07:05.382 INFO:teuthology.orchestra.run.smithi050.stdout: "success": "filestore_inject_stall = '' (not observed, change may require restart) " 2024-09-18T08:07:05.382 INFO:teuthology.orchestra.run.smithi050.stdout:} 2024-09-18T08:07:05.430 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6 2024-09-18T08:07:05.463 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:07:05.464 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:07:05.464 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6' 2024-09-18T08:07:05.565 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0 2024-09-18T08:07:05.608 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:07:05.608 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:07:05.609 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0' 2024-09-18T08:07:05.710 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5 2024-09-18T08:07:05.747 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:07:05.747 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:07:05.748 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5' 2024-09-18T08:07:05.849 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4 2024-09-18T08:07:05.880 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:07:05.881 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:07:05.881 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4' 2024-09-18T08:07:05.982 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6 2024-09-18T08:07:06.013 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:07:06.013 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:07:06.014 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6' 2024-09-18T08:07:06.115 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5 2024-09-18T08:07:06.148 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:07:06.149 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:07:06.149 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5' 2024-09-18T08:07:06.197 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid 779349ca-7593-11ef-bceb-c7b262605968 -- ceph daemon osd.0 dump_historic_ops 2024-09-18T08:07:06.250 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0 2024-09-18T08:07:06.296 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:07:06.297 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:07:06.297 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0' 2024-09-18T08:07:06.307 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:07:06 smithi050 ceph-mon[32371]: pgmap v192: 148 pgs: 148 active+clean; 2.2 GiB data, 7.5 GiB used, 708 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.25k op/s 2024-09-18T08:07:06.341 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/779349ca-7593-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T08:07:06.398 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5 2024-09-18T08:07:06.424 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:07:06.424 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:07:06.425 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5' 2024-09-18T08:07:06.480 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:07:06 smithi007 ceph-mon[29459]: pgmap v192: 148 pgs: 148 active+clean; 2.2 GiB data, 7.5 GiB used, 708 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.25k op/s 2024-09-18T08:07:06.481 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:07:06 smithi007 ceph-mon[39249]: pgmap v192: 148 pgs: 148 active+clean; 2.2 GiB data, 7.5 GiB used, 708 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.25k op/s 2024-09-18T08:07:06.526 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4 2024-09-18T08:07:06.553 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:07:06.554 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:07:06.554 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4' 2024-09-18T08:07:06.655 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6 2024-09-18T08:07:06.683 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:07:06.684 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:07:06.684 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6' 2024-09-18T08:07:06.785 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3 2024-09-18T08:07:06.830 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:07:06.831 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:07:06.832 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3' 2024-09-18T08:07:06.854 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Sep 18 08:07:06 smithi007 ceph-779349ca-7593-11ef-bceb-c7b262605968-mgr-y[29688]: ::ffff:172.21.15.50 - - [18/Sep/2024:08:07:06] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-09-18T08:07:06.932 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0 2024-09-18T08:07:06.972 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:07:06.972 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:07:06.972 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0' 2024-09-18T08:07:07.073 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5 2024-09-18T08:07:07.102 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:07:07.103 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:07:07.104 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5' 2024-09-18T08:07:07.204 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5 2024-09-18T08:07:07.231 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:07:07.232 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:07:07.232 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5' 2024-09-18T08:07:07.333 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2 2024-09-18T08:07:07.340 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/779349ca-7593-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T08:07:07.390 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:07:07.390 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:07:07.391 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2' 2024-09-18T08:07:07.491 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7 2024-09-18T08:07:07.519 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:07:07.520 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:07:07.521 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7' 2024-09-18T08:07:07.621 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5 2024-09-18T08:07:07.649 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:07:07.650 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:07:07.650 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5' 2024-09-18T08:07:07.751 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3 2024-09-18T08:07:07.790 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid 779349ca-7593-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-18T08:07:07.794 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:07:07.795 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:07:07.795 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3' 2024-09-18T08:07:07.895 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5 2024-09-18T08:07:07.921 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:07:07.922 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:07:07.922 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5' 2024-09-18T08:07:07.989 INFO:teuthology.orchestra.run.smithi007.stdout:{ 2024-09-18T08:07:07.989 INFO:teuthology.orchestra.run.smithi007.stdout: "size": 20, 2024-09-18T08:07:07.989 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 600, 2024-09-18T08:07:07.989 INFO:teuthology.orchestra.run.smithi007.stdout: "ops": [ 2024-09-18T08:07:07.989 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:07:07.990 INFO:teuthology.orchestra.run.smithi007.stdout: "description": "osd_op(client.14805.0:65085 7.7 7:e643a565:::benchmark_data_smithi007_73155_object65084:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e101)", 2024-09-18T08:07:07.990 INFO:teuthology.orchestra.run.smithi007.stdout: "initiated_at": "2024-09-18T08:04:42.325315+0000", 2024-09-18T08:07:07.990 INFO:teuthology.orchestra.run.smithi007.stdout: "age": 145.658675746, 2024-09-18T08:07:07.990 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.99419469699999996, 2024-09-18T08:07:07.990 INFO:teuthology.orchestra.run.smithi007.stdout: "continuous": false, 2024-09-18T08:07:07.990 INFO:teuthology.orchestra.run.smithi007.stdout: "type_data": { 2024-09-18T08:07:07.990 INFO:teuthology.orchestra.run.smithi007.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T08:07:07.990 INFO:teuthology.orchestra.run.smithi007.stdout: "client_info": { 2024-09-18T08:07:07.990 INFO:teuthology.orchestra.run.smithi007.stdout: "client": "client.14805", 2024-09-18T08:07:07.990 INFO:teuthology.orchestra.run.smithi007.stdout: "client_addr": "172.21.15.7:0/640320436", 2024-09-18T08:07:07.990 INFO:teuthology.orchestra.run.smithi007.stdout: "tid": 65085 2024-09-18T08:07:07.990 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:07:07.990 INFO:teuthology.orchestra.run.smithi007.stdout: "events": [ 2024-09-18T08:07:07.991 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:07:07.991 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "initiated", 2024-09-18T08:07:07.991 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:04:42.325315+0000", 2024-09-18T08:07:07.991 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-09-18T08:07:07.991 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:07:07.991 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:07:07.991 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "header_read", 2024-09-18T08:07:07.991 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:04:42.325315+0000", 2024-09-18T08:07:07.991 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-09-18T08:07:07.991 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:07:07.991 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:07:07.991 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "throttled", 2024-09-18T08:07:07.991 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:04:42.325316+0000", 2024-09-18T08:07:07.992 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.621e-06 2024-09-18T08:07:07.992 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:07:07.992 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:07:07.992 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "all_read", 2024-09-18T08:07:07.992 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:04:42.325346+0000", 2024-09-18T08:07:07.992 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 2.9215999999999999e-05 2024-09-18T08:07:07.992 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:07:07.992 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:07:07.992 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "dispatched", 2024-09-18T08:07:07.992 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:04:42.325438+0000", 2024-09-18T08:07:07.992 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 9.2672999999999999e-05 2024-09-18T08:07:07.992 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:07:07.992 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:07:07.993 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "queued_for_pg", 2024-09-18T08:07:07.993 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:04:42.325447+0000", 2024-09-18T08:07:07.993 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 8.6989999999999992e-06 2024-09-18T08:07:07.993 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:07:07.993 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:07:07.993 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-09-18T08:07:07.993 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:04:42.325518+0000", 2024-09-18T08:07:07.993 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 7.0924000000000001e-05 2024-09-18T08:07:07.993 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:07:07.993 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:07:07.993 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "waiting for peered", 2024-09-18T08:07:07.993 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:04:42.325529+0000", 2024-09-18T08:07:07.993 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.0774999999999999e-05 2024-09-18T08:07:07.993 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:07:07.994 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:07:07.994 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-09-18T08:07:07.994 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:04:43.305082+0000", 2024-09-18T08:07:07.994 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.97955334900000002 2024-09-18T08:07:07.994 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:07:07.994 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:07:07.994 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "started", 2024-09-18T08:07:07.994 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:04:43.306711+0000", 2024-09-18T08:07:07.994 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.0016285259999999999 2024-09-18T08:07:07.994 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:07:07.994 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:07:07.994 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "waiting for subops from 3,6", 2024-09-18T08:07:07.994 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:04:43.306878+0000", 2024-09-18T08:07:07.994 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.00016743300000000001 2024-09-18T08:07:07.994 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:07:07.994 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:07:07.995 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:07:07.995 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:04:43.312523+0000", 2024-09-18T08:07:07.995 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.0056453880000000003 2024-09-18T08:07:07.995 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:07:07.995 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:07:07.995 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:07:07.995 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:04:43.319352+0000", 2024-09-18T08:07:07.995 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.0068285100000000003 2024-09-18T08:07:07.995 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:07:07.995 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:07:07.995 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "op_commit", 2024-09-18T08:07:07.995 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:04:43.319365+0000", 2024-09-18T08:07:07.995 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.2870999999999999e-05 2024-09-18T08:07:07.995 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:07:07.996 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:07:07.996 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "commit_sent", 2024-09-18T08:07:07.996 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:04:43.319431+0000", 2024-09-18T08:07:07.996 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 6.6431000000000001e-05 2024-09-18T08:07:07.996 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:07:07.996 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:07:07.996 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "done", 2024-09-18T08:07:07.996 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:04:43.319509+0000", 2024-09-18T08:07:07.996 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 7.8281000000000005e-05 2024-09-18T08:07:07.996 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-09-18T08:07:07.996 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-09-18T08:07:07.996 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-09-18T08:07:07.996 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:07:07.996 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:07:07.997 INFO:teuthology.orchestra.run.smithi007.stdout: "description": "osd_op(client.14805.0:65088 7.7 7:ec575e01:::benchmark_data_smithi007_73155_object65087:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e101)", 2024-09-18T08:07:07.997 INFO:teuthology.orchestra.run.smithi007.stdout: "initiated_at": "2024-09-18T08:04:42.325472+0000", 2024-09-18T08:07:07.997 INFO:teuthology.orchestra.run.smithi007.stdout: "age": 145.65851836300001, 2024-09-18T08:07:07.997 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.99450635799999998, 2024-09-18T08:07:07.997 INFO:teuthology.orchestra.run.smithi007.stdout: "continuous": false, 2024-09-18T08:07:07.997 INFO:teuthology.orchestra.run.smithi007.stdout: "type_data": { 2024-09-18T08:07:07.997 INFO:teuthology.orchestra.run.smithi007.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T08:07:07.997 INFO:teuthology.orchestra.run.smithi007.stdout: "client_info": { 2024-09-18T08:07:07.997 INFO:teuthology.orchestra.run.smithi007.stdout: "client": "client.14805", 2024-09-18T08:07:07.997 INFO:teuthology.orchestra.run.smithi007.stdout: "client_addr": "172.21.15.7:0/640320436", 2024-09-18T08:07:07.997 INFO:teuthology.orchestra.run.smithi007.stdout: "tid": 65088 2024-09-18T08:07:07.997 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:07:07.997 INFO:teuthology.orchestra.run.smithi007.stdout: "events": [ 2024-09-18T08:07:07.997 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:07:07.998 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "initiated", 2024-09-18T08:07:07.998 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:04:42.325472+0000", 2024-09-18T08:07:07.998 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-09-18T08:07:07.998 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:07:07.998 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:07:07.998 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "header_read", 2024-09-18T08:07:07.998 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:04:42.325472+0000", 2024-09-18T08:07:07.998 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-09-18T08:07:07.998 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:07:07.998 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:07:07.998 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "throttled", 2024-09-18T08:07:07.998 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:04:42.325473+0000", 2024-09-18T08:07:07.998 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 6.5700000000000002e-07 2024-09-18T08:07:07.998 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:07:07.998 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:07:07.999 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "all_read", 2024-09-18T08:07:07.999 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:04:42.325501+0000", 2024-09-18T08:07:07.999 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 2.7662999999999999e-05 2024-09-18T08:07:07.999 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:07:07.999 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:07:07.999 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "dispatched", 2024-09-18T08:07:07.999 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:04:42.325508+0000", 2024-09-18T08:07:07.999 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 7.0509999999999999e-06 2024-09-18T08:07:07.999 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:07:07.999 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:07:07.999 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "queued_for_pg", 2024-09-18T08:07:07.999 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:04:42.325513+0000", 2024-09-18T08:07:07.999 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 5.8760000000000002e-06 2024-09-18T08:07:07.999 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:07:07.999 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:07:08.000 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-09-18T08:07:08.000 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:04:42.325558+0000", 2024-09-18T08:07:08.000 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 4.4384000000000001e-05 2024-09-18T08:07:08.000 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:07:08.000 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:07:08.000 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "waiting for peered", 2024-09-18T08:07:08.000 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:04:42.325570+0000", 2024-09-18T08:07:08.000 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.1719000000000001e-05 2024-09-18T08:07:08.000 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:07:08.000 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:07:08.000 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-09-18T08:07:08.000 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:04:43.309576+0000", 2024-09-18T08:07:08.000 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.98400614799999997 2024-09-18T08:07:08.000 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:07:08.000 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:07:08.001 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "started", 2024-09-18T08:07:08.001 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:04:43.309740+0000", 2024-09-18T08:07:08.001 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.000164229 2024-09-18T08:07:08.001 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:07:08.001 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:07:08.001 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "waiting for subops from 3,6", 2024-09-18T08:07:08.001 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:04:43.310047+0000", 2024-09-18T08:07:08.001 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.00030731600000000002 2024-09-18T08:07:08.001 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:07:08.001 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:07:08.001 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:07:08.001 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:04:43.312568+0000", 2024-09-18T08:07:08.001 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.0025208069999999999 2024-09-18T08:07:08.001 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:07:08.001 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:07:08.002 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "op_commit", 2024-09-18T08:07:08.002 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:04:43.319825+0000", 2024-09-18T08:07:08.002 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.0072565499999999996 2024-09-18T08:07:08.002 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:07:08.002 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:07:08.002 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:07:08.002 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:04:43.319883+0000", 2024-09-18T08:07:08.002 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 5.8579000000000003e-05 2024-09-18T08:07:08.002 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:07:08.003 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:07:08.003 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "commit_sent", 2024-09-18T08:07:08.003 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:04:43.319940+0000", 2024-09-18T08:07:08.003 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 5.6436e-05 2024-09-18T08:07:08.003 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:07:08.003 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:07:08.003 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "done", 2024-09-18T08:07:08.003 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:04:43.319979+0000", 2024-09-18T08:07:08.003 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 3.8943000000000003e-05 2024-09-18T08:07:08.003 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-09-18T08:07:08.003 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-09-18T08:07:08.003 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-09-18T08:07:08.003 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:07:08.003 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:07:08.003 INFO:teuthology.orchestra.run.smithi007.stdout: "description": "osd_op(client.14805.0:65089 7.7 7:effcefb5:::benchmark_data_smithi007_73155_object65088:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e101)", 2024-09-18T08:07:08.004 INFO:teuthology.orchestra.run.smithi007.stdout: "initiated_at": "2024-09-18T08:04:42.325980+0000", 2024-09-18T08:07:08.004 INFO:teuthology.orchestra.run.smithi007.stdout: "age": 145.658010203, 2024-09-18T08:07:08.004 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.99416539400000004, 2024-09-18T08:07:08.004 INFO:teuthology.orchestra.run.smithi007.stdout: "continuous": false, 2024-09-18T08:07:08.004 INFO:teuthology.orchestra.run.smithi007.stdout: "type_data": { 2024-09-18T08:07:08.004 INFO:teuthology.orchestra.run.smithi007.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T08:07:08.004 INFO:teuthology.orchestra.run.smithi007.stdout: "client_info": { 2024-09-18T08:07:08.004 INFO:teuthology.orchestra.run.smithi007.stdout: "client": "client.14805", 2024-09-18T08:07:08.004 INFO:teuthology.orchestra.run.smithi007.stdout: "client_addr": "172.21.15.7:0/640320436", 2024-09-18T08:07:08.004 INFO:teuthology.orchestra.run.smithi007.stdout: "tid": 65089 2024-09-18T08:07:08.004 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:07:08.004 INFO:teuthology.orchestra.run.smithi007.stdout: "events": [ 2024-09-18T08:07:08.004 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:07:08.005 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "initiated", 2024-09-18T08:07:08.005 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:04:42.325980+0000", 2024-09-18T08:07:08.005 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-09-18T08:07:08.005 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:07:08.005 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:07:08.005 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "header_read", 2024-09-18T08:07:08.006 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:04:42.325980+0000", 2024-09-18T08:07:08.006 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-09-18T08:07:08.006 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:07:08.006 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:07:08.006 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "throttled", 2024-09-18T08:07:08.006 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:04:42.325981+0000", 2024-09-18T08:07:08.006 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.026e-06 2024-09-18T08:07:08.006 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:07:08.006 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:07:08.007 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "all_read", 2024-09-18T08:07:08.007 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:04:42.326003+0000", 2024-09-18T08:07:08.007 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 2.2048999999999998e-05 2024-09-18T08:07:08.007 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:07:08.007 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:07:08.007 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "dispatched", 2024-09-18T08:07:08.007 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:04:42.326014+0000", 2024-09-18T08:07:08.007 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.0560000000000001e-05 2024-09-18T08:07:08.007 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:07:08.007 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:07:08.007 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "queued_for_pg", 2024-09-18T08:07:08.007 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:04:42.326021+0000", 2024-09-18T08:07:08.007 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 6.9990000000000002e-06 2024-09-18T08:07:08.007 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:07:08.007 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:07:08.008 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-09-18T08:07:08.008 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:04:42.326162+0000", 2024-09-18T08:07:08.008 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.000141447 2024-09-18T08:07:08.008 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:07:08.008 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:07:08.008 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "waiting for peered", 2024-09-18T08:07:08.008 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:04:42.326172+0000", 2024-09-18T08:07:08.008 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 9.2459999999999992e-06 2024-09-18T08:07:08.008 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:07:08.008 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:07:08.008 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-09-18T08:07:08.008 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:04:43.312600+0000", 2024-09-18T08:07:08.008 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.98642846200000001 2024-09-18T08:07:08.008 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:07:08.009 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:07:08.009 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "started", 2024-09-18T08:07:08.009 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:04:43.312707+0000", 2024-09-18T08:07:08.009 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.000107225 2024-09-18T08:07:08.009 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:07:08.009 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:07:08.009 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "waiting for subops from 3,6", 2024-09-18T08:07:08.009 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:04:43.312837+0000", 2024-09-18T08:07:08.009 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.00013003900000000001 2024-09-18T08:07:08.009 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:07:08.009 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:07:08.009 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:07:08.009 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:04:43.319777+0000", 2024-09-18T08:07:08.009 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.0069396969999999999 2024-09-18T08:07:08.010 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:07:08.010 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:07:08.010 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "op_commit", 2024-09-18T08:07:08.010 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:04:43.319830+0000", 2024-09-18T08:07:08.010 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 5.2747000000000002e-05 2024-09-18T08:07:08.010 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:07:08.010 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:07:08.010 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:07:08.010 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:04:43.320062+0000", 2024-09-18T08:07:08.010 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.00023171800000000001 2024-09-18T08:07:08.010 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:07:08.010 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:07:08.010 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "commit_sent", 2024-09-18T08:07:08.010 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:04:43.320107+0000", 2024-09-18T08:07:08.010 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 4.5578999999999998e-05 2024-09-18T08:07:08.011 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:07:08.011 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:07:08.011 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "done", 2024-09-18T08:07:08.011 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:04:43.320146+0000", 2024-09-18T08:07:08.011 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 3.8600000000000003e-05 2024-09-18T08:07:08.011 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-09-18T08:07:08.011 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-09-18T08:07:08.011 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-09-18T08:07:08.011 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:07:08.011 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:07:08.012 INFO:teuthology.orchestra.run.smithi007.stdout: "description": "osd_op(client.14805.0:65105 7.7 7:e54b6e92:::benchmark_data_smithi007_73155_object65104:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e101)", 2024-09-18T08:07:08.012 INFO:teuthology.orchestra.run.smithi007.stdout: "initiated_at": "2024-09-18T08:04:42.341572+0000", 2024-09-18T08:07:08.012 INFO:teuthology.orchestra.run.smithi007.stdout: "age": 145.64241830899999, 2024-09-18T08:07:08.012 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.99348088999999995, 2024-09-18T08:07:08.012 INFO:teuthology.orchestra.run.smithi007.stdout: "continuous": false, 2024-09-18T08:07:08.012 INFO:teuthology.orchestra.run.smithi007.stdout: "type_data": { 2024-09-18T08:07:08.012 INFO:teuthology.orchestra.run.smithi007.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T08:07:08.012 INFO:teuthology.orchestra.run.smithi007.stdout: "client_info": { 2024-09-18T08:07:08.012 INFO:teuthology.orchestra.run.smithi007.stdout: "client": "client.14805", 2024-09-18T08:07:08.012 INFO:teuthology.orchestra.run.smithi007.stdout: "client_addr": "172.21.15.7:0/640320436", 2024-09-18T08:07:08.012 INFO:teuthology.orchestra.run.smithi007.stdout: "tid": 65105 2024-09-18T08:07:08.012 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:07:08.013 INFO:teuthology.orchestra.run.smithi007.stdout: "events": [ 2024-09-18T08:07:08.013 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:07:08.013 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "initiated", 2024-09-18T08:07:08.013 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:04:42.341572+0000", 2024-09-18T08:07:08.013 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-09-18T08:07:08.013 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:07:08.013 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:07:08.013 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "header_read", 2024-09-18T08:07:08.013 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:04:42.341572+0000", 2024-09-18T08:07:08.013 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-09-18T08:07:08.013 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:07:08.013 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:07:08.013 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "throttled", 2024-09-18T08:07:08.013 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:04:42.341574+0000", 2024-09-18T08:07:08.014 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.6080000000000001e-06 2024-09-18T08:07:08.014 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:07:08.014 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:07:08.014 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "all_read", 2024-09-18T08:07:08.014 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:04:42.341597+0000", 2024-09-18T08:07:08.014 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 2.2691999999999999e-05 2024-09-18T08:07:08.014 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:07:08.014 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:07:08.014 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "dispatched", 2024-09-18T08:07:08.014 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:04:42.341612+0000", 2024-09-18T08:07:08.014 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.5775000000000001e-05 2024-09-18T08:07:08.014 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:07:08.014 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:07:08.014 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "queued_for_pg", 2024-09-18T08:07:08.015 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:04:42.341622+0000", 2024-09-18T08:07:08.015 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 9.329e-06 2024-09-18T08:07:08.015 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:07:08.015 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:07:08.015 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-09-18T08:07:08.015 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:04:42.341694+0000", 2024-09-18T08:07:08.015 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 7.2757999999999997e-05 2024-09-18T08:07:08.015 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:07:08.015 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:07:08.015 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "waiting for peered", 2024-09-18T08:07:08.015 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:04:42.341706+0000", 2024-09-18T08:07:08.015 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.2075999999999999e-05 2024-09-18T08:07:08.015 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:07:08.015 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:07:08.015 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-09-18T08:07:08.016 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:04:43.315804+0000", 2024-09-18T08:07:08.016 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.97409722899999995 2024-09-18T08:07:08.016 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:07:08.016 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:07:08.016 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "started", 2024-09-18T08:07:08.016 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:04:43.315915+0000", 2024-09-18T08:07:08.016 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.000111437 2024-09-18T08:07:08.016 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:07:08.016 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:07:08.016 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "waiting for subops from 3,6", 2024-09-18T08:07:08.016 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:04:43.316035+0000", 2024-09-18T08:07:08.016 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.00011955000000000001 2024-09-18T08:07:08.016 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:07:08.016 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:07:08.017 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "op_commit", 2024-09-18T08:07:08.017 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:04:43.319834+0000", 2024-09-18T08:07:08.017 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.0037993900000000001 2024-09-18T08:07:08.017 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:07:08.017 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:07:08.017 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:07:08.017 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:04:43.320021+0000", 2024-09-18T08:07:08.017 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.00018648499999999999 2024-09-18T08:07:08.017 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:07:08.017 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:07:08.018 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:07:08.018 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:04:43.334952+0000", 2024-09-18T08:07:08.018 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.014931275000000001 2024-09-18T08:07:08.018 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:07:08.018 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:07:08.018 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "commit_sent", 2024-09-18T08:07:08.018 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:04:43.335007+0000", 2024-09-18T08:07:08.018 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 5.4849000000000003e-05 2024-09-18T08:07:08.019 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:07:08.019 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:07:08.019 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "done", 2024-09-18T08:07:08.019 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:04:43.335053+0000", 2024-09-18T08:07:08.019 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 4.6437000000000003e-05 2024-09-18T08:07:08.019 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-09-18T08:07:08.019 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-09-18T08:07:08.019 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-09-18T08:07:08.019 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:07:08.019 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:07:08.019 INFO:teuthology.orchestra.run.smithi007.stdout: "description": "osd_op(client.14805.0:65108 7.7 7:ef77f963:::benchmark_data_smithi007_73155_object65107:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e101)", 2024-09-18T08:07:08.019 INFO:teuthology.orchestra.run.smithi007.stdout: "initiated_at": "2024-09-18T08:04:42.343962+0000", 2024-09-18T08:07:08.019 INFO:teuthology.orchestra.run.smithi007.stdout: "age": 145.64002888300001, 2024-09-18T08:07:08.019 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.99132680500000003, 2024-09-18T08:07:08.019 INFO:teuthology.orchestra.run.smithi007.stdout: "continuous": false, 2024-09-18T08:07:08.020 INFO:teuthology.orchestra.run.smithi007.stdout: "type_data": { 2024-09-18T08:07:08.020 INFO:teuthology.orchestra.run.smithi007.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T08:07:08.020 INFO:teuthology.orchestra.run.smithi007.stdout: "client_info": { 2024-09-18T08:07:08.020 INFO:teuthology.orchestra.run.smithi007.stdout: "client": "client.14805", 2024-09-18T08:07:08.020 INFO:teuthology.orchestra.run.smithi007.stdout: "client_addr": "172.21.15.7:0/640320436", 2024-09-18T08:07:08.020 INFO:teuthology.orchestra.run.smithi007.stdout: "tid": 65108 2024-09-18T08:07:08.020 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:07:08.020 INFO:teuthology.orchestra.run.smithi007.stdout: "events": [ 2024-09-18T08:07:08.020 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:07:08.020 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "initiated", 2024-09-18T08:07:08.020 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:04:42.343962+0000", 2024-09-18T08:07:08.020 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-09-18T08:07:08.020 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:07:08.020 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:07:08.020 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "header_read", 2024-09-18T08:07:08.021 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:04:42.343962+0000", 2024-09-18T08:07:08.021 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-09-18T08:07:08.021 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:07:08.021 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:07:08.021 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "throttled", 2024-09-18T08:07:08.021 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:04:42.343963+0000", 2024-09-18T08:07:08.021 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.531e-06 2024-09-18T08:07:08.021 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:07:08.021 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:07:08.021 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "all_read", 2024-09-18T08:07:08.021 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:04:42.343989+0000", 2024-09-18T08:07:08.021 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 2.5735999999999999e-05 2024-09-18T08:07:08.021 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:07:08.021 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:07:08.021 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "dispatched", 2024-09-18T08:07:08.022 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:04:42.344001+0000", 2024-09-18T08:07:08.022 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.219e-05 2024-09-18T08:07:08.022 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:07:08.022 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:07:08.022 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "queued_for_pg", 2024-09-18T08:07:08.022 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:04:42.344009+0000", 2024-09-18T08:07:08.022 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 7.6420000000000007e-06 2024-09-18T08:07:08.022 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:07:08.022 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:07:08.022 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-09-18T08:07:08.023 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:04:42.344058+0000", 2024-09-18T08:07:08.023 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 4.9076999999999998e-05 2024-09-18T08:07:08.023 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:07:08.023 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:07:08.023 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "waiting for peered", 2024-09-18T08:07:08.023 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:04:42.344064+0000", 2024-09-18T08:07:08.023 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 6.1240000000000002e-06 2024-09-18T08:07:08.023 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:07:08.023 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:07:08.023 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-09-18T08:07:08.023 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:04:43.320175+0000", 2024-09-18T08:07:08.023 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.97611146900000001 2024-09-18T08:07:08.023 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:07:08.023 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:07:08.024 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "started", 2024-09-18T08:07:08.024 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:04:43.320286+0000", 2024-09-18T08:07:08.024 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.000110221 2024-09-18T08:07:08.024 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:07:08.024 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:07:08.024 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "waiting for subops from 3,6", 2024-09-18T08:07:08.024 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:04:43.320416+0000", 2024-09-18T08:07:08.024 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.00013077100000000001 2024-09-18T08:07:08.024 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:07:08.024 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:07:08.024 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:07:08.024 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:04:43.325654+0000", 2024-09-18T08:07:08.024 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.0052372850000000004 2024-09-18T08:07:08.025 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:07:08.025 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:07:08.025 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "op_commit", 2024-09-18T08:07:08.025 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:04:43.332014+0000", 2024-09-18T08:07:08.025 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.0063605989999999998 2024-09-18T08:07:08.025 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:07:08.025 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:07:08.025 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:07:08.025 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:04:43.335151+0000", 2024-09-18T08:07:08.025 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.0031363960000000001 2024-09-18T08:07:08.026 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:07:08.026 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:07:08.026 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "commit_sent", 2024-09-18T08:07:08.026 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:04:43.335241+0000", 2024-09-18T08:07:08.026 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 9.0564999999999998e-05 2024-09-18T08:07:08.026 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:07:08.026 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:07:08.026 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "done", 2024-09-18T08:07:08.026 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:04:43.335288+0000", 2024-09-18T08:07:08.026 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 4.7199e-05 2024-09-18T08:07:08.026 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-09-18T08:07:08.026 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-09-18T08:07:08.027 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-09-18T08:07:08.027 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:07:08.027 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:07:08.027 INFO:teuthology.orchestra.run.smithi007.stdout: "description": "osd_op(client.14805.0:65121 7.7 7:e3b4a34b:::benchmark_data_smithi007_73155_object65120:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e101)", 2024-09-18T08:07:08.027 INFO:teuthology.orchestra.run.smithi007.stdout: "initiated_at": "2024-09-18T08:04:42.359562+0000", 2024-09-18T08:07:08.027 INFO:teuthology.orchestra.run.smithi007.stdout: "age": 145.62442829, 2024-09-18T08:07:08.027 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.97975838599999998, 2024-09-18T08:07:08.027 INFO:teuthology.orchestra.run.smithi007.stdout: "continuous": false, 2024-09-18T08:07:08.027 INFO:teuthology.orchestra.run.smithi007.stdout: "type_data": { 2024-09-18T08:07:08.027 INFO:teuthology.orchestra.run.smithi007.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T08:07:08.027 INFO:teuthology.orchestra.run.smithi007.stdout: "client_info": { 2024-09-18T08:07:08.027 INFO:teuthology.orchestra.run.smithi007.stdout: "client": "client.14805", 2024-09-18T08:07:08.027 INFO:teuthology.orchestra.run.smithi007.stdout: "client_addr": "172.21.15.7:0/640320436", 2024-09-18T08:07:08.027 INFO:teuthology.orchestra.run.smithi007.stdout: "tid": 65121 2024-09-18T08:07:08.027 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:07:08.027 INFO:teuthology.orchestra.run.smithi007.stdout: "events": [ 2024-09-18T08:07:08.028 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:07:08.028 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "initiated", 2024-09-18T08:07:08.028 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:04:42.359562+0000", 2024-09-18T08:07:08.028 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-09-18T08:07:08.028 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:07:08.028 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:07:08.028 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "header_read", 2024-09-18T08:07:08.028 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:04:42.359562+0000", 2024-09-18T08:07:08.028 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-09-18T08:07:08.028 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:07:08.028 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:07:08.028 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "throttled", 2024-09-18T08:07:08.028 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:04:42.359563+0000", 2024-09-18T08:07:08.028 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 8.6700000000000002e-07 2024-09-18T08:07:08.029 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:07:08.029 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:07:08.029 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "all_read", 2024-09-18T08:07:08.029 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:04:42.359587+0000", 2024-09-18T08:07:08.029 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 2.3782999999999999e-05 2024-09-18T08:07:08.029 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:07:08.029 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:07:08.029 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "dispatched", 2024-09-18T08:07:08.029 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:04:42.359597+0000", 2024-09-18T08:07:08.029 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 9.9660000000000008e-06 2024-09-18T08:07:08.029 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:07:08.029 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:07:08.029 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "queued_for_pg", 2024-09-18T08:07:08.029 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:04:42.359606+0000", 2024-09-18T08:07:08.030 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 9.4120000000000007e-06 2024-09-18T08:07:08.030 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:07:08.030 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:07:08.030 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-09-18T08:07:08.030 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:04:42.359663+0000", 2024-09-18T08:07:08.030 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 5.7113e-05 2024-09-18T08:07:08.030 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:07:08.030 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:07:08.030 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "waiting for peered", 2024-09-18T08:07:08.030 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:04:42.359671+0000", 2024-09-18T08:07:08.030 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 7.5159999999999997e-06 2024-09-18T08:07:08.030 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:07:08.030 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:07:08.030 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-09-18T08:07:08.030 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:04:43.322602+0000", 2024-09-18T08:07:08.031 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.96293140099999996 2024-09-18T08:07:08.031 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:07:08.031 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:07:08.031 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "started", 2024-09-18T08:07:08.031 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:04:43.322715+0000", 2024-09-18T08:07:08.031 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.000113027 2024-09-18T08:07:08.031 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:07:08.031 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:07:08.031 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "waiting for subops from 3,6", 2024-09-18T08:07:08.031 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:04:43.322871+0000", 2024-09-18T08:07:08.031 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.00015569299999999999 2024-09-18T08:07:08.031 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:07:08.031 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:07:08.031 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:07:08.031 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:04:43.327357+0000", 2024-09-18T08:07:08.031 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.0044856920000000003 2024-09-18T08:07:08.032 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:07:08.032 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:07:08.032 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "op_commit", 2024-09-18T08:07:08.032 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:04:43.332022+0000", 2024-09-18T08:07:08.032 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.0046651119999999999 2024-09-18T08:07:08.032 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:07:08.032 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:07:08.032 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:07:08.032 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:04:43.339232+0000", 2024-09-18T08:07:08.032 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.0072099140000000004 2024-09-18T08:07:08.032 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:07:08.032 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:07:08.032 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "commit_sent", 2024-09-18T08:07:08.033 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:04:43.339284+0000", 2024-09-18T08:07:08.033 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 5.1975999999999998e-05 2024-09-18T08:07:08.033 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:07:08.033 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:07:08.033 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "done", 2024-09-18T08:07:08.033 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:04:43.339321+0000", 2024-09-18T08:07:08.033 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 3.6913999999999999e-05 2024-09-18T08:07:08.033 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-09-18T08:07:08.033 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-09-18T08:07:08.033 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-09-18T08:07:08.033 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:07:08.033 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:07:08.033 INFO:teuthology.orchestra.run.smithi007.stdout: "description": "osd_op(client.14805.0:94167 7.f 7:f6221d40:::benchmark_data_smithi007_73155_object94166:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e108)", 2024-09-18T08:07:08.033 INFO:teuthology.orchestra.run.smithi007.stdout: "initiated_at": "2024-09-18T08:05:09.952116+0000", 2024-09-18T08:07:08.034 INFO:teuthology.orchestra.run.smithi007.stdout: "age": 118.031874532, 2024-09-18T08:07:08.034 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.11215270099999999, 2024-09-18T08:07:08.034 INFO:teuthology.orchestra.run.smithi007.stdout: "continuous": false, 2024-09-18T08:07:08.034 INFO:teuthology.orchestra.run.smithi007.stdout: "type_data": { 2024-09-18T08:07:08.034 INFO:teuthology.orchestra.run.smithi007.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T08:07:08.034 INFO:teuthology.orchestra.run.smithi007.stdout: "client_info": { 2024-09-18T08:07:08.034 INFO:teuthology.orchestra.run.smithi007.stdout: "client": "client.14805", 2024-09-18T08:07:08.034 INFO:teuthology.orchestra.run.smithi007.stdout: "client_addr": "172.21.15.7:0/640320436", 2024-09-18T08:07:08.034 INFO:teuthology.orchestra.run.smithi007.stdout: "tid": 94167 2024-09-18T08:07:08.034 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:07:08.034 INFO:teuthology.orchestra.run.smithi007.stdout: "events": [ 2024-09-18T08:07:08.035 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:07:08.035 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "initiated", 2024-09-18T08:07:08.035 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:05:09.952116+0000", 2024-09-18T08:07:08.035 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-09-18T08:07:08.035 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:07:08.035 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:07:08.035 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "header_read", 2024-09-18T08:07:08.035 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:05:09.952116+0000", 2024-09-18T08:07:08.035 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-09-18T08:07:08.035 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:07:08.035 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:07:08.036 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "throttled", 2024-09-18T08:07:08.036 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:05:09.952117+0000", 2024-09-18T08:07:08.036 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.192e-06 2024-09-18T08:07:08.036 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:07:08.036 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:07:08.036 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "all_read", 2024-09-18T08:07:08.036 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:05:09.952140+0000", 2024-09-18T08:07:08.036 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 2.3283000000000001e-05 2024-09-18T08:07:08.036 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:07:08.036 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:07:08.036 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "dispatched", 2024-09-18T08:07:08.036 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:05:09.952151+0000", 2024-09-18T08:07:08.036 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.0098000000000001e-05 2024-09-18T08:07:08.037 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:07:08.037 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:07:08.037 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "queued_for_pg", 2024-09-18T08:07:08.037 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:05:09.952157+0000", 2024-09-18T08:07:08.037 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 6.686e-06 2024-09-18T08:07:08.037 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:07:08.037 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:07:08.037 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-09-18T08:07:08.037 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:05:09.957208+0000", 2024-09-18T08:07:08.037 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.005050354 2024-09-18T08:07:08.037 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:07:08.037 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:07:08.037 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "started", 2024-09-18T08:07:08.037 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:05:09.957347+0000", 2024-09-18T08:07:08.038 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.000139338 2024-09-18T08:07:08.038 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:07:08.038 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:07:08.038 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "waiting for subops from 3,7", 2024-09-18T08:07:08.038 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:05:09.957513+0000", 2024-09-18T08:07:08.038 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.00016620599999999999 2024-09-18T08:07:08.038 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:07:08.038 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:07:08.038 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:07:08.038 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:05:09.961059+0000", 2024-09-18T08:07:08.038 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.00354588 2024-09-18T08:07:08.039 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:07:08.039 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:07:08.039 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "op_commit", 2024-09-18T08:07:08.039 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:05:09.967589+0000", 2024-09-18T08:07:08.039 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.0065294810000000002 2024-09-18T08:07:08.039 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:07:08.039 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:07:08.039 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:07:08.039 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:05:10.064097+0000", 2024-09-18T08:07:08.039 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.096508438000000002 2024-09-18T08:07:08.039 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:07:08.039 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:07:08.039 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "commit_sent", 2024-09-18T08:07:08.039 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:05:10.064225+0000", 2024-09-18T08:07:08.040 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.00012780699999999999 2024-09-18T08:07:08.040 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:07:08.040 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:07:08.040 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "done", 2024-09-18T08:07:08.040 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:05:10.064269+0000", 2024-09-18T08:07:08.040 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 4.3937999999999998e-05 2024-09-18T08:07:08.040 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-09-18T08:07:08.040 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-09-18T08:07:08.040 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-09-18T08:07:08.040 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:07:08.040 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:07:08.040 INFO:teuthology.orchestra.run.smithi007.stdout: "description": "osd_op(client.14805.0:94175 7.4 7:2a0b3e41:::benchmark_data_smithi007_73155_object94174:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e108)", 2024-09-18T08:07:08.040 INFO:teuthology.orchestra.run.smithi007.stdout: "initiated_at": "2024-09-18T08:05:09.957080+0000", 2024-09-18T08:07:08.040 INFO:teuthology.orchestra.run.smithi007.stdout: "age": 118.02691040000001, 2024-09-18T08:07:08.041 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.11815871, 2024-09-18T08:07:08.041 INFO:teuthology.orchestra.run.smithi007.stdout: "continuous": false, 2024-09-18T08:07:08.041 INFO:teuthology.orchestra.run.smithi007.stdout: "type_data": { 2024-09-18T08:07:08.041 INFO:teuthology.orchestra.run.smithi007.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T08:07:08.041 INFO:teuthology.orchestra.run.smithi007.stdout: "client_info": { 2024-09-18T08:07:08.041 INFO:teuthology.orchestra.run.smithi007.stdout: "client": "client.14805", 2024-09-18T08:07:08.041 INFO:teuthology.orchestra.run.smithi007.stdout: "client_addr": "172.21.15.7:0/640320436", 2024-09-18T08:07:08.041 INFO:teuthology.orchestra.run.smithi007.stdout: "tid": 94175 2024-09-18T08:07:08.041 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:07:08.041 INFO:teuthology.orchestra.run.smithi007.stdout: "events": [ 2024-09-18T08:07:08.041 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:07:08.041 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "initiated", 2024-09-18T08:07:08.041 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:05:09.957080+0000", 2024-09-18T08:07:08.041 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-09-18T08:07:08.042 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:07:08.042 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:07:08.042 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "header_read", 2024-09-18T08:07:08.042 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:05:09.957080+0000", 2024-09-18T08:07:08.042 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-09-18T08:07:08.042 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:07:08.042 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:07:08.042 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "throttled", 2024-09-18T08:07:08.042 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:05:09.957091+0000", 2024-09-18T08:07:08.042 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.0399e-05 2024-09-18T08:07:08.042 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:07:08.042 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:07:08.043 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "all_read", 2024-09-18T08:07:08.043 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:05:09.957116+0000", 2024-09-18T08:07:08.043 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 2.5800999999999999e-05 2024-09-18T08:07:08.043 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:07:08.043 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:07:08.043 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "dispatched", 2024-09-18T08:07:08.043 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:05:09.957126+0000", 2024-09-18T08:07:08.043 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 9.6190000000000006e-06 2024-09-18T08:07:08.043 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:07:08.043 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:07:08.043 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "queued_for_pg", 2024-09-18T08:07:08.043 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:05:09.957133+0000", 2024-09-18T08:07:08.043 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 6.8859999999999998e-06 2024-09-18T08:07:08.044 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:07:08.044 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:07:08.044 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-09-18T08:07:08.044 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:05:09.957471+0000", 2024-09-18T08:07:08.044 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.00033765299999999999 2024-09-18T08:07:08.044 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:07:08.044 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:07:08.044 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "started", 2024-09-18T08:07:08.044 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:05:09.957574+0000", 2024-09-18T08:07:08.044 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.000103397 2024-09-18T08:07:08.044 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:07:08.044 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:07:08.044 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "waiting for subops from 1,4", 2024-09-18T08:07:08.044 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:05:09.957709+0000", 2024-09-18T08:07:08.044 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.00013518100000000001 2024-09-18T08:07:08.045 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:07:08.045 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:07:08.045 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:07:08.045 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:05:10.064554+0000", 2024-09-18T08:07:08.045 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.106844622 2024-09-18T08:07:08.045 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:07:08.045 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:07:08.045 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:07:08.045 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:05:10.064665+0000", 2024-09-18T08:07:08.045 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.0001115 2024-09-18T08:07:08.045 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:07:08.045 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:07:08.045 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "op_commit", 2024-09-18T08:07:08.045 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:05:10.074995+0000", 2024-09-18T08:07:08.045 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.010329925 2024-09-18T08:07:08.046 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:07:08.046 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:07:08.046 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "commit_sent", 2024-09-18T08:07:08.046 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:05:10.075190+0000", 2024-09-18T08:07:08.046 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.000194389 2024-09-18T08:07:08.046 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:07:08.046 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3 2024-09-18T08:07:08.047 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:07:08.047 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "done", 2024-09-18T08:07:08.047 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:05:10.075239+0000", 2024-09-18T08:07:08.047 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 4.9338000000000001e-05 2024-09-18T08:07:08.047 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-09-18T08:07:08.047 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-09-18T08:07:08.048 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-09-18T08:07:08.048 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:07:08.048 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:07:08.048 INFO:teuthology.orchestra.run.smithi007.stdout: "description": "osd_op(client.14805.0:94176 7.7 7:e88c7ff5:::benchmark_data_smithi007_73155_object94175:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e108)", 2024-09-18T08:07:08.048 INFO:teuthology.orchestra.run.smithi007.stdout: "initiated_at": "2024-09-18T08:05:09.957889+0000", 2024-09-18T08:07:08.048 INFO:teuthology.orchestra.run.smithi007.stdout: "age": 118.026101442, 2024-09-18T08:07:08.048 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.106492589, 2024-09-18T08:07:08.048 INFO:teuthology.orchestra.run.smithi007.stdout: "continuous": false, 2024-09-18T08:07:08.048 INFO:teuthology.orchestra.run.smithi007.stdout: "type_data": { 2024-09-18T08:07:08.048 INFO:teuthology.orchestra.run.smithi007.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T08:07:08.048 INFO:teuthology.orchestra.run.smithi007.stdout: "client_info": { 2024-09-18T08:07:08.048 INFO:teuthology.orchestra.run.smithi007.stdout: "client": "client.14805", 2024-09-18T08:07:08.048 INFO:teuthology.orchestra.run.smithi007.stdout: "client_addr": "172.21.15.7:0/640320436", 2024-09-18T08:07:08.048 INFO:teuthology.orchestra.run.smithi007.stdout: "tid": 94176 2024-09-18T08:07:08.049 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:07:08.049 INFO:teuthology.orchestra.run.smithi007.stdout: "events": [ 2024-09-18T08:07:08.049 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:07:08.049 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "initiated", 2024-09-18T08:07:08.049 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:05:09.957889+0000", 2024-09-18T08:07:08.049 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-09-18T08:07:08.049 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:07:08.049 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:07:08.050 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "header_read", 2024-09-18T08:07:08.050 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:05:09.957889+0000", 2024-09-18T08:07:08.050 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-09-18T08:07:08.050 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:07:08.050 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:07:08.050 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "throttled", 2024-09-18T08:07:08.050 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:05:09.957890+0000", 2024-09-18T08:07:08.050 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 8.47e-07 2024-09-18T08:07:08.051 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:07:08.051 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:07:08.051 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "all_read", 2024-09-18T08:07:08.051 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:05:09.957920+0000", 2024-09-18T08:07:08.051 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 3.0399000000000002e-05 2024-09-18T08:07:08.051 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:07:08.051 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:07:08.051 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "dispatched", 2024-09-18T08:07:08.051 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:05:09.957930+0000", 2024-09-18T08:07:08.051 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 9.6379999999999999e-06 2024-09-18T08:07:08.051 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:07:08.051 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:07:08.051 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "queued_for_pg", 2024-09-18T08:07:08.052 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:05:09.957936+0000", 2024-09-18T08:07:08.052 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 6.4300000000000003e-06 2024-09-18T08:07:08.052 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:07:08.052 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:07:08.052 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-09-18T08:07:08.052 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:05:09.961137+0000", 2024-09-18T08:07:08.052 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.0032002649999999999 2024-09-18T08:07:08.052 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:07:08.052 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:07:08.053 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "started", 2024-09-18T08:07:08.053 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:05:09.961249+0000", 2024-09-18T08:07:08.053 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.00011235600000000001 2024-09-18T08:07:08.053 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:07:08.053 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:07:08.053 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "waiting for subops from 3,6", 2024-09-18T08:07:08.053 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:05:09.961388+0000", 2024-09-18T08:07:08.053 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.00013849000000000001 2024-09-18T08:07:08.053 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:07:08.053 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:07:08.053 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:07:08.053 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:05:09.964467+0000", 2024-09-18T08:07:08.053 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.0030798240000000001 2024-09-18T08:07:08.054 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:07:08.054 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:07:08.054 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "op_commit", 2024-09-18T08:07:08.054 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:05:10.063981+0000", 2024-09-18T08:07:08.054 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.099513613000000001 2024-09-18T08:07:08.054 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:07:08.054 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:07:08.054 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:07:08.054 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:05:10.064305+0000", 2024-09-18T08:07:08.054 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.00032443199999999999 2024-09-18T08:07:08.054 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:07:08.054 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:07:08.054 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "commit_sent", 2024-09-18T08:07:08.055 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:05:10.064348+0000", 2024-09-18T08:07:08.055 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 4.2828999999999999e-05 2024-09-18T08:07:08.055 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:07:08.055 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:07:08.055 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "done", 2024-09-18T08:07:08.055 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:05:10.064382+0000", 2024-09-18T08:07:08.055 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 3.3466000000000001e-05 2024-09-18T08:07:08.055 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-09-18T08:07:08.055 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-09-18T08:07:08.055 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-09-18T08:07:08.055 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:07:08.055 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:07:08.055 INFO:teuthology.orchestra.run.smithi007.stdout: "description": "osd_op(client.14805.0:94180 7.4 7:220e5307:::benchmark_data_smithi007_73155_object94179:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e108)", 2024-09-18T08:07:08.055 INFO:teuthology.orchestra.run.smithi007.stdout: "initiated_at": "2024-09-18T08:05:09.960662+0000", 2024-09-18T08:07:08.055 INFO:teuthology.orchestra.run.smithi007.stdout: "age": 118.023328369, 2024-09-18T08:07:08.056 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.114766168, 2024-09-18T08:07:08.056 INFO:teuthology.orchestra.run.smithi007.stdout: "continuous": false, 2024-09-18T08:07:08.056 INFO:teuthology.orchestra.run.smithi007.stdout: "type_data": { 2024-09-18T08:07:08.056 INFO:teuthology.orchestra.run.smithi007.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T08:07:08.056 INFO:teuthology.orchestra.run.smithi007.stdout: "client_info": { 2024-09-18T08:07:08.056 INFO:teuthology.orchestra.run.smithi007.stdout: "client": "client.14805", 2024-09-18T08:07:08.056 INFO:teuthology.orchestra.run.smithi007.stdout: "client_addr": "172.21.15.7:0/640320436", 2024-09-18T08:07:08.056 INFO:teuthology.orchestra.run.smithi007.stdout: "tid": 94180 2024-09-18T08:07:08.056 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:07:08.056 INFO:teuthology.orchestra.run.smithi007.stdout: "events": [ 2024-09-18T08:07:08.056 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:07:08.056 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "initiated", 2024-09-18T08:07:08.056 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:05:09.960662+0000", 2024-09-18T08:07:08.056 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-09-18T08:07:08.056 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:07:08.057 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:07:08.057 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "header_read", 2024-09-18T08:07:08.057 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:05:09.960662+0000", 2024-09-18T08:07:08.057 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-09-18T08:07:08.057 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:07:08.057 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:07:08.057 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "throttled", 2024-09-18T08:07:08.057 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:05:09.960664+0000", 2024-09-18T08:07:08.057 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.75e-06 2024-09-18T08:07:08.057 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:07:08.057 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:07:08.057 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "all_read", 2024-09-18T08:07:08.057 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:05:09.960692+0000", 2024-09-18T08:07:08.057 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 2.7991e-05 2024-09-18T08:07:08.058 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:07:08.058 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:07:08.058 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "dispatched", 2024-09-18T08:07:08.058 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:05:09.960704+0000", 2024-09-18T08:07:08.058 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.2009e-05 2024-09-18T08:07:08.058 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:07:08.058 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:07:08.058 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "queued_for_pg", 2024-09-18T08:07:08.058 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:05:09.960711+0000", 2024-09-18T08:07:08.058 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 6.9809999999999997e-06 2024-09-18T08:07:08.058 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:07:08.058 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:07:08.058 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-09-18T08:07:08.058 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:05:09.961829+0000", 2024-09-18T08:07:08.059 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.001118045 2024-09-18T08:07:08.059 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:07:08.059 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:07:08.059 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "started", 2024-09-18T08:07:08.059 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:05:09.961946+0000", 2024-09-18T08:07:08.059 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.00011747999999999999 2024-09-18T08:07:08.059 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:07:08.059 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:07:08.059 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "waiting for subops from 1,4", 2024-09-18T08:07:08.059 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:05:09.962073+0000", 2024-09-18T08:07:08.059 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.000126933 2024-09-18T08:07:08.059 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:07:08.060 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:07:08.060 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:07:08.060 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:05:10.064592+0000", 2024-09-18T08:07:08.060 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.102518363 2024-09-18T08:07:08.060 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:07:08.060 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:07:08.060 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:07:08.060 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:05:10.064835+0000", 2024-09-18T08:07:08.060 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.00024330900000000001 2024-09-18T08:07:08.060 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:07:08.060 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:07:08.060 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "op_commit", 2024-09-18T08:07:08.060 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:05:10.075247+0000", 2024-09-18T08:07:08.061 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.010411767000000001 2024-09-18T08:07:08.061 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:07:08.061 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:07:08.061 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "commit_sent", 2024-09-18T08:07:08.061 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:05:10.075389+0000", 2024-09-18T08:07:08.061 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.00014242500000000001 2024-09-18T08:07:08.061 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:07:08.061 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:07:08.061 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "done", 2024-09-18T08:07:08.061 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:05:10.075428+0000", 2024-09-18T08:07:08.061 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 3.9115000000000001e-05 2024-09-18T08:07:08.061 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-09-18T08:07:08.061 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-09-18T08:07:08.061 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-09-18T08:07:08.062 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:07:08.062 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:07:08.062 INFO:teuthology.orchestra.run.smithi007.stdout: "description": "osd_op(client.14805.0:94184 7.4 7:2349c68d:::benchmark_data_smithi007_73155_object94183:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e108)", 2024-09-18T08:07:08.062 INFO:teuthology.orchestra.run.smithi007.stdout: "initiated_at": "2024-09-18T08:05:09.961496+0000", 2024-09-18T08:07:08.062 INFO:teuthology.orchestra.run.smithi007.stdout: "age": 118.02249427700001, 2024-09-18T08:07:08.062 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.114148087, 2024-09-18T08:07:08.062 INFO:teuthology.orchestra.run.smithi007.stdout: "continuous": false, 2024-09-18T08:07:08.063 INFO:teuthology.orchestra.run.smithi007.stdout: "type_data": { 2024-09-18T08:07:08.063 INFO:teuthology.orchestra.run.smithi007.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T08:07:08.063 INFO:teuthology.orchestra.run.smithi007.stdout: "client_info": { 2024-09-18T08:07:08.063 INFO:teuthology.orchestra.run.smithi007.stdout: "client": "client.14805", 2024-09-18T08:07:08.064 INFO:teuthology.orchestra.run.smithi007.stdout: "client_addr": "172.21.15.7:0/640320436", 2024-09-18T08:07:08.064 INFO:teuthology.orchestra.run.smithi007.stdout: "tid": 94184 2024-09-18T08:07:08.064 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:07:08.064 INFO:teuthology.orchestra.run.smithi007.stdout: "events": [ 2024-09-18T08:07:08.064 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:07:08.064 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "initiated", 2024-09-18T08:07:08.064 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:05:09.961496+0000", 2024-09-18T08:07:08.064 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-09-18T08:07:08.064 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:07:08.064 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:07:08.064 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "header_read", 2024-09-18T08:07:08.064 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:05:09.961496+0000", 2024-09-18T08:07:08.064 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-09-18T08:07:08.065 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:07:08.065 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:07:08.065 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "throttled", 2024-09-18T08:07:08.065 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:05:09.961497+0000", 2024-09-18T08:07:08.065 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 9.47e-07 2024-09-18T08:07:08.065 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:07:08.065 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:07:08.065 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "all_read", 2024-09-18T08:07:08.065 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:05:09.961518+0000", 2024-09-18T08:07:08.065 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 2.0803999999999999e-05 2024-09-18T08:07:08.065 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:07:08.065 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:07:08.065 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "dispatched", 2024-09-18T08:07:08.065 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:05:09.961529+0000", 2024-09-18T08:07:08.066 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.0675e-05 2024-09-18T08:07:08.066 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:07:08.066 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:07:08.066 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "queued_for_pg", 2024-09-18T08:07:08.066 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:05:09.961536+0000", 2024-09-18T08:07:08.066 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 7.1149999999999996e-06 2024-09-18T08:07:08.066 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:07:08.066 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:07:08.066 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-09-18T08:07:08.066 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:05:09.964897+0000", 2024-09-18T08:07:08.066 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.0033611209999999999 2024-09-18T08:07:08.066 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:07:08.066 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:07:08.066 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "started", 2024-09-18T08:07:08.066 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:05:09.965000+0000", 2024-09-18T08:07:08.067 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.000102948 2024-09-18T08:07:08.067 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:07:08.067 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:07:08.067 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "waiting for subops from 1,4", 2024-09-18T08:07:08.067 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:05:09.965125+0000", 2024-09-18T08:07:08.067 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.00012555500000000001 2024-09-18T08:07:08.067 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:07:08.067 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:07:08.067 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:07:08.067 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:05:10.064759+0000", 2024-09-18T08:07:08.067 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.099633304000000006 2024-09-18T08:07:08.067 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:07:08.067 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:07:08.067 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:07:08.067 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:05:10.074877+0000", 2024-09-18T08:07:08.068 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.010118248 2024-09-18T08:07:08.068 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:07:08.068 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:07:08.068 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "op_commit", 2024-09-18T08:07:08.068 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:05:10.075436+0000", 2024-09-18T08:07:08.068 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.00055861599999999995 2024-09-18T08:07:08.068 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:07:08.068 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:07:08.068 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "commit_sent", 2024-09-18T08:07:08.068 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:05:10.075605+0000", 2024-09-18T08:07:08.068 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.00016952800000000001 2024-09-18T08:07:08.068 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:07:08.068 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:07:08.068 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "done", 2024-09-18T08:07:08.068 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:05:10.075644+0000", 2024-09-18T08:07:08.069 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 3.9226e-05 2024-09-18T08:07:08.069 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-09-18T08:07:08.069 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-09-18T08:07:08.069 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-09-18T08:07:08.069 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:07:08.069 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:07:08.069 INFO:teuthology.orchestra.run.smithi007.stdout: "description": "osd_op(client.14805.0:94190 7.6 7:62f26a1c:::benchmark_data_smithi007_73155_object94189:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e108)", 2024-09-18T08:07:08.069 INFO:teuthology.orchestra.run.smithi007.stdout: "initiated_at": "2024-09-18T08:05:09.970773+0000", 2024-09-18T08:07:08.069 INFO:teuthology.orchestra.run.smithi007.stdout: "age": 118.013217618, 2024-09-18T08:07:08.069 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.110747217, 2024-09-18T08:07:08.069 INFO:teuthology.orchestra.run.smithi007.stdout: "continuous": false, 2024-09-18T08:07:08.069 INFO:teuthology.orchestra.run.smithi007.stdout: "type_data": { 2024-09-18T08:07:08.069 INFO:teuthology.orchestra.run.smithi007.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T08:07:08.069 INFO:teuthology.orchestra.run.smithi007.stdout: "client_info": { 2024-09-18T08:07:08.070 INFO:teuthology.orchestra.run.smithi007.stdout: "client": "client.14805", 2024-09-18T08:07:08.070 INFO:teuthology.orchestra.run.smithi007.stdout: "client_addr": "172.21.15.7:0/640320436", 2024-09-18T08:07:08.070 INFO:teuthology.orchestra.run.smithi007.stdout: "tid": 94190 2024-09-18T08:07:08.070 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:07:08.070 INFO:teuthology.orchestra.run.smithi007.stdout: "events": [ 2024-09-18T08:07:08.070 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:07:08.070 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "initiated", 2024-09-18T08:07:08.070 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:05:09.970773+0000", 2024-09-18T08:07:08.070 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-09-18T08:07:08.070 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:07:08.070 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:07:08.070 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "header_read", 2024-09-18T08:07:08.071 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:05:09.970773+0000", 2024-09-18T08:07:08.071 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-09-18T08:07:08.071 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:07:08.071 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:07:08.071 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "throttled", 2024-09-18T08:07:08.071 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:05:09.970774+0000", 2024-09-18T08:07:08.071 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.53e-06 2024-09-18T08:07:08.071 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:07:08.071 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:07:08.071 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "all_read", 2024-09-18T08:07:08.071 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:05:09.970799+0000", 2024-09-18T08:07:08.071 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 2.5014000000000001e-05 2024-09-18T08:07:08.071 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:07:08.072 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:07:08.072 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "dispatched", 2024-09-18T08:07:08.072 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:05:10.064318+0000", 2024-09-18T08:07:08.072 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.093518854999999998 2024-09-18T08:07:08.072 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:07:08.072 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:07:08.072 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "queued_for_pg", 2024-09-18T08:07:08.072 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:05:10.064329+0000", 2024-09-18T08:07:08.072 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.0436e-05 2024-09-18T08:07:08.072 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:07:08.072 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:07:08.072 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-09-18T08:07:08.072 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:05:10.068584+0000", 2024-09-18T08:07:08.072 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.0042548029999999997 2024-09-18T08:07:08.072 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:07:08.073 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:07:08.073 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "started", 2024-09-18T08:07:08.073 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:05:10.068697+0000", 2024-09-18T08:07:08.073 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.000113527 2024-09-18T08:07:08.073 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:07:08.073 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:07:08.073 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "waiting for subops from 3,7", 2024-09-18T08:07:08.073 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:05:10.068828+0000", 2024-09-18T08:07:08.073 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.00013095899999999999 2024-09-18T08:07:08.073 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:07:08.073 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:07:08.073 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:07:08.073 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:05:10.073153+0000", 2024-09-18T08:07:08.073 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.0043253220000000004 2024-09-18T08:07:08.074 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:07:08.074 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:07:08.074 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:07:08.074 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:05:10.080483+0000", 2024-09-18T08:07:08.074 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.0073299599999999999 2024-09-18T08:07:08.074 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:07:08.074 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:07:08.074 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "op_commit", 2024-09-18T08:07:08.074 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:05:10.081293+0000", 2024-09-18T08:07:08.074 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.00081008399999999998 2024-09-18T08:07:08.074 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:07:08.074 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:07:08.074 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "commit_sent", 2024-09-18T08:07:08.074 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:05:10.081473+0000", 2024-09-18T08:07:08.074 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.00017958699999999999 2024-09-18T08:07:08.075 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:07:08.075 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:07:08.075 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "done", 2024-09-18T08:07:08.075 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:05:10.081520+0000", 2024-09-18T08:07:08.075 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 4.7139999999999999e-05 2024-09-18T08:07:08.075 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-09-18T08:07:08.075 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-09-18T08:07:08.075 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-09-18T08:07:08.075 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:07:08.075 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:07:08.075 INFO:teuthology.orchestra.run.smithi007.stdout: "description": "osd_op(client.14805.0:96727 7.7 7:e43d8257:::benchmark_data_smithi007_73155_object96726:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e109)", 2024-09-18T08:07:08.075 INFO:teuthology.orchestra.run.smithi007.stdout: "initiated_at": "2024-09-18T08:05:13.307664+0000", 2024-09-18T08:07:08.075 INFO:teuthology.orchestra.run.smithi007.stdout: "age": 114.676326291, 2024-09-18T08:07:08.075 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.109680052, 2024-09-18T08:07:08.075 INFO:teuthology.orchestra.run.smithi007.stdout: "continuous": false, 2024-09-18T08:07:08.075 INFO:teuthology.orchestra.run.smithi007.stdout: "type_data": { 2024-09-18T08:07:08.076 INFO:teuthology.orchestra.run.smithi007.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T08:07:08.076 INFO:teuthology.orchestra.run.smithi007.stdout: "client_info": { 2024-09-18T08:07:08.076 INFO:teuthology.orchestra.run.smithi007.stdout: "client": "client.14805", 2024-09-18T08:07:08.076 INFO:teuthology.orchestra.run.smithi007.stdout: "client_addr": "172.21.15.7:0/640320436", 2024-09-18T08:07:08.076 INFO:teuthology.orchestra.run.smithi007.stdout: "tid": 96727 2024-09-18T08:07:08.076 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:07:08.076 INFO:teuthology.orchestra.run.smithi007.stdout: "events": [ 2024-09-18T08:07:08.076 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:07:08.076 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "initiated", 2024-09-18T08:07:08.076 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:05:13.307664+0000", 2024-09-18T08:07:08.076 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-09-18T08:07:08.076 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:07:08.076 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:07:08.076 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "header_read", 2024-09-18T08:07:08.077 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:05:13.307664+0000", 2024-09-18T08:07:08.077 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-09-18T08:07:08.077 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:07:08.077 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:07:08.077 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "throttled", 2024-09-18T08:07:08.077 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:05:13.307665+0000", 2024-09-18T08:07:08.077 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 4.3099999999999998e-07 2024-09-18T08:07:08.077 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:07:08.077 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:07:08.077 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "all_read", 2024-09-18T08:07:08.077 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:05:13.307685+0000", 2024-09-18T08:07:08.077 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 2.0608e-05 2024-09-18T08:07:08.078 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:07:08.078 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:07:08.078 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "dispatched", 2024-09-18T08:07:08.078 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:05:13.307691+0000", 2024-09-18T08:07:08.078 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 5.8830000000000003e-06 2024-09-18T08:07:08.078 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:07:08.078 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:07:08.078 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "queued_for_pg", 2024-09-18T08:07:08.078 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:05:13.307696+0000", 2024-09-18T08:07:08.078 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 5.1490000000000002e-06 2024-09-18T08:07:08.078 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:07:08.078 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:07:08.078 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-09-18T08:07:08.078 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:05:13.367326+0000", 2024-09-18T08:07:08.079 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.059629592000000002 2024-09-18T08:07:08.079 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:07:08.079 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:07:08.079 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "started", 2024-09-18T08:07:08.079 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:05:13.367474+0000", 2024-09-18T08:07:08.079 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.00014821699999999999 2024-09-18T08:07:08.079 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:07:08.079 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:07:08.079 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "waiting for subops from 3,6", 2024-09-18T08:07:08.079 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:05:13.367646+0000", 2024-09-18T08:07:08.079 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.00017211599999999999 2024-09-18T08:07:08.079 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:07:08.079 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:07:08.079 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:07:08.080 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:05:13.409889+0000", 2024-09-18T08:07:08.080 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.042242276000000002 2024-09-18T08:07:08.080 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:07:08.080 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:07:08.080 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "op_commit", 2024-09-18T08:07:08.080 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:05:13.409900+0000", 2024-09-18T08:07:08.080 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.1486e-05 2024-09-18T08:07:08.080 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:07:08.080 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:07:08.080 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:07:08.080 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:05:13.417217+0000", 2024-09-18T08:07:08.081 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.0073172649999999999 2024-09-18T08:07:08.081 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:07:08.081 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:07:08.081 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "commit_sent", 2024-09-18T08:07:08.081 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:05:13.417283+0000", 2024-09-18T08:07:08.081 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 6.5679999999999995e-05 2024-09-18T08:07:08.081 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:07:08.081 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:07:08.081 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "done", 2024-09-18T08:07:08.081 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:05:13.417344+0000", 2024-09-18T08:07:08.081 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 6.1348999999999998e-05 2024-09-18T08:07:08.081 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-09-18T08:07:08.081 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-09-18T08:07:08.082 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-09-18T08:07:08.082 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:07:08.082 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:07:08.082 INFO:teuthology.orchestra.run.smithi007.stdout: "description": "osd_op(client.14805.0:110408 7.6 7:63ec65a5:::benchmark_data_smithi007_73155_object110407:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e118)", 2024-09-18T08:07:08.082 INFO:teuthology.orchestra.run.smithi007.stdout: "initiated_at": "2024-09-18T08:05:27.019055+0000", 2024-09-18T08:07:08.082 INFO:teuthology.orchestra.run.smithi007.stdout: "age": 100.964935075, 2024-09-18T08:07:08.082 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.142191123, 2024-09-18T08:07:08.082 INFO:teuthology.orchestra.run.smithi007.stdout: "continuous": false, 2024-09-18T08:07:08.082 INFO:teuthology.orchestra.run.smithi007.stdout: "type_data": { 2024-09-18T08:07:08.082 INFO:teuthology.orchestra.run.smithi007.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T08:07:08.082 INFO:teuthology.orchestra.run.smithi007.stdout: "client_info": { 2024-09-18T08:07:08.082 INFO:teuthology.orchestra.run.smithi007.stdout: "client": "client.14805", 2024-09-18T08:07:08.082 INFO:teuthology.orchestra.run.smithi007.stdout: "client_addr": "172.21.15.7:0/640320436", 2024-09-18T08:07:08.083 INFO:teuthology.orchestra.run.smithi007.stdout: "tid": 110408 2024-09-18T08:07:08.083 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:07:08.083 INFO:teuthology.orchestra.run.smithi007.stdout: "events": [ 2024-09-18T08:07:08.083 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:07:08.083 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "initiated", 2024-09-18T08:07:08.083 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:05:27.019055+0000", 2024-09-18T08:07:08.083 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-09-18T08:07:08.083 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:07:08.083 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:07:08.083 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "header_read", 2024-09-18T08:07:08.083 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:05:27.019055+0000", 2024-09-18T08:07:08.083 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-09-18T08:07:08.083 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:07:08.083 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:07:08.083 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "throttled", 2024-09-18T08:07:08.084 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:05:27.019057+0000", 2024-09-18T08:07:08.084 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.347e-06 2024-09-18T08:07:08.084 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:07:08.084 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:07:08.084 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "all_read", 2024-09-18T08:07:08.084 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:05:27.019077+0000", 2024-09-18T08:07:08.084 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.9704999999999999e-05 2024-09-18T08:07:08.084 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:07:08.084 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:07:08.084 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "dispatched", 2024-09-18T08:07:08.084 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:05:27.089428+0000", 2024-09-18T08:07:08.084 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.070351231 2024-09-18T08:07:08.084 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:07:08.084 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:07:08.085 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "queued_for_pg", 2024-09-18T08:07:08.085 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:05:27.089439+0000", 2024-09-18T08:07:08.085 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.0803e-05 2024-09-18T08:07:08.085 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:07:08.085 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:07:08.085 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-09-18T08:07:08.085 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:05:27.089524+0000", 2024-09-18T08:07:08.085 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 8.5119999999999998e-05 2024-09-18T08:07:08.085 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:07:08.085 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:07:08.085 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "started", 2024-09-18T08:07:08.085 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:05:27.089641+0000", 2024-09-18T08:07:08.085 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.00011754199999999999 2024-09-18T08:07:08.086 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:07:08.086 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:07:08.086 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "waiting for subops from 3,7", 2024-09-18T08:07:08.086 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:05:27.092158+0000", 2024-09-18T08:07:08.086 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.002516903 2024-09-18T08:07:08.086 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:07:08.086 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:07:08.086 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:07:08.086 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:05:27.095183+0000", 2024-09-18T08:07:08.086 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.0030251269999999999 2024-09-18T08:07:08.086 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:07:08.086 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:07:08.086 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "op_commit", 2024-09-18T08:07:08.086 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:05:27.095390+0000", 2024-09-18T08:07:08.086 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.000206818 2024-09-18T08:07:08.087 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:07:08.087 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:07:08.087 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:07:08.087 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:05:27.161053+0000", 2024-09-18T08:07:08.087 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.065662639999999994 2024-09-18T08:07:08.087 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:07:08.087 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:07:08.087 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "commit_sent", 2024-09-18T08:07:08.087 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:05:27.161160+0000", 2024-09-18T08:07:08.087 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.000107382 2024-09-18T08:07:08.087 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:07:08.087 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:07:08.087 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "done", 2024-09-18T08:07:08.087 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:05:27.161247+0000", 2024-09-18T08:07:08.088 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 8.6504999999999999e-05 2024-09-18T08:07:08.088 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-09-18T08:07:08.088 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-09-18T08:07:08.088 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-09-18T08:07:08.088 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:07:08.088 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:07:08.088 INFO:teuthology.orchestra.run.smithi007.stdout: "description": "osd_op(client.14805.0:111939 7.7 7:e3760c3c:::benchmark_data_smithi007_73155_object111938:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e118)", 2024-09-18T08:07:08.088 INFO:teuthology.orchestra.run.smithi007.stdout: "initiated_at": "2024-09-18T08:05:28.892891+0000", 2024-09-18T08:07:08.088 INFO:teuthology.orchestra.run.smithi007.stdout: "age": 99.091099512, 2024-09-18T08:07:08.088 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.10522242, 2024-09-18T08:07:08.088 INFO:teuthology.orchestra.run.smithi007.stdout: "continuous": false, 2024-09-18T08:07:08.088 INFO:teuthology.orchestra.run.smithi007.stdout: "type_data": { 2024-09-18T08:07:08.089 INFO:teuthology.orchestra.run.smithi007.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T08:07:08.089 INFO:teuthology.orchestra.run.smithi007.stdout: "client_info": { 2024-09-18T08:07:08.089 INFO:teuthology.orchestra.run.smithi007.stdout: "client": "client.14805", 2024-09-18T08:07:08.089 INFO:teuthology.orchestra.run.smithi007.stdout: "client_addr": "172.21.15.7:0/640320436", 2024-09-18T08:07:08.089 INFO:teuthology.orchestra.run.smithi007.stdout: "tid": 111939 2024-09-18T08:07:08.089 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:07:08.089 INFO:teuthology.orchestra.run.smithi007.stdout: "events": [ 2024-09-18T08:07:08.089 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:07:08.089 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "initiated", 2024-09-18T08:07:08.089 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:05:28.892891+0000", 2024-09-18T08:07:08.089 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-09-18T08:07:08.089 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:07:08.089 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:07:08.089 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "header_read", 2024-09-18T08:07:08.090 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:05:28.892891+0000", 2024-09-18T08:07:08.090 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-09-18T08:07:08.090 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:07:08.090 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:07:08.090 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "throttled", 2024-09-18T08:07:08.090 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:05:28.892893+0000", 2024-09-18T08:07:08.090 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 2.1150000000000001e-06 2024-09-18T08:07:08.090 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:07:08.090 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:07:08.090 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "all_read", 2024-09-18T08:07:08.090 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:05:28.892919+0000", 2024-09-18T08:07:08.090 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 2.5681e-05 2024-09-18T08:07:08.090 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:07:08.090 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:07:08.090 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "dispatched", 2024-09-18T08:07:08.091 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:05:28.892932+0000", 2024-09-18T08:07:08.091 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.3010000000000001e-05 2024-09-18T08:07:08.091 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:07:08.092 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:07:08.092 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "queued_for_pg", 2024-09-18T08:07:08.092 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:05:28.892940+0000", 2024-09-18T08:07:08.092 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 8.1589999999999993e-06 2024-09-18T08:07:08.092 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:07:08.092 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:07:08.092 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-09-18T08:07:08.092 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:05:28.892995+0000", 2024-09-18T08:07:08.092 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 5.4568000000000003e-05 2024-09-18T08:07:08.092 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:07:08.092 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:07:08.092 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "started", 2024-09-18T08:07:08.093 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:05:28.893163+0000", 2024-09-18T08:07:08.093 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.00016881 2024-09-18T08:07:08.093 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:07:08.093 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:07:08.093 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "waiting for subops from 3,6", 2024-09-18T08:07:08.093 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:05:28.893315+0000", 2024-09-18T08:07:08.093 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.000151791 2024-09-18T08:07:08.093 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:07:08.093 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:07:08.093 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:07:08.093 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:05:28.946160+0000", 2024-09-18T08:07:08.093 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.052844374 2024-09-18T08:07:08.093 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:07:08.094 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:07:08.094 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "op_commit", 2024-09-18T08:07:08.094 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:05:28.946175+0000", 2024-09-18T08:07:08.094 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.5605000000000001e-05 2024-09-18T08:07:08.094 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:07:08.094 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:07:08.094 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:07:08.094 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:05:28.997754+0000", 2024-09-18T08:07:08.094 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.051579125000000003 2024-09-18T08:07:08.094 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:07:08.094 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:07:08.094 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "commit_sent", 2024-09-18T08:07:08.094 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:05:28.998056+0000", 2024-09-18T08:07:08.094 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.00030126600000000001 2024-09-18T08:07:08.095 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:07:08.095 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:07:08.095 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "done", 2024-09-18T08:07:08.095 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:05:28.998113+0000", 2024-09-18T08:07:08.095 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 5.7915999999999999e-05 2024-09-18T08:07:08.095 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-09-18T08:07:08.095 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-09-18T08:07:08.095 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-09-18T08:07:08.095 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:07:08.095 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:07:08.095 INFO:teuthology.orchestra.run.smithi007.stdout: "description": "osd_op(client.14805.0:111941 7.7 7:e7a646d7:::benchmark_data_smithi007_73155_object111940:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e118)", 2024-09-18T08:07:08.095 INFO:teuthology.orchestra.run.smithi007.stdout: "initiated_at": "2024-09-18T08:05:28.894173+0000", 2024-09-18T08:07:08.095 INFO:teuthology.orchestra.run.smithi007.stdout: "age": 99.089817272000005, 2024-09-18T08:07:08.096 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.10430357, 2024-09-18T08:07:08.096 INFO:teuthology.orchestra.run.smithi007.stdout: "continuous": false, 2024-09-18T08:07:08.096 INFO:teuthology.orchestra.run.smithi007.stdout: "type_data": { 2024-09-18T08:07:08.096 INFO:teuthology.orchestra.run.smithi007.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T08:07:08.096 INFO:teuthology.orchestra.run.smithi007.stdout: "client_info": { 2024-09-18T08:07:08.096 INFO:teuthology.orchestra.run.smithi007.stdout: "client": "client.14805", 2024-09-18T08:07:08.096 INFO:teuthology.orchestra.run.smithi007.stdout: "client_addr": "172.21.15.7:0/640320436", 2024-09-18T08:07:08.096 INFO:teuthology.orchestra.run.smithi007.stdout: "tid": 111941 2024-09-18T08:07:08.096 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:07:08.096 INFO:teuthology.orchestra.run.smithi007.stdout: "events": [ 2024-09-18T08:07:08.096 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:07:08.096 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "initiated", 2024-09-18T08:07:08.096 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:05:28.894173+0000", 2024-09-18T08:07:08.097 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-09-18T08:07:08.097 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:07:08.097 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:07:08.097 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "header_read", 2024-09-18T08:07:08.097 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:05:28.894173+0000", 2024-09-18T08:07:08.097 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-09-18T08:07:08.097 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:07:08.097 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:07:08.097 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "throttled", 2024-09-18T08:07:08.098 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:05:28.894175+0000", 2024-09-18T08:07:08.098 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.347e-06 2024-09-18T08:07:08.098 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:07:08.098 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:07:08.098 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "all_read", 2024-09-18T08:07:08.098 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:05:28.894201+0000", 2024-09-18T08:07:08.098 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 2.5899999999999999e-05 2024-09-18T08:07:08.098 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:07:08.098 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:07:08.098 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "dispatched", 2024-09-18T08:07:08.098 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:05:28.894212+0000", 2024-09-18T08:07:08.098 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.1919000000000001e-05 2024-09-18T08:07:08.099 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:07:08.099 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:07:08.099 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "queued_for_pg", 2024-09-18T08:07:08.099 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:05:28.894220+0000", 2024-09-18T08:07:08.099 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 7.6920000000000002e-06 2024-09-18T08:07:08.099 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:07:08.099 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:07:08.099 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-09-18T08:07:08.099 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:05:28.896441+0000", 2024-09-18T08:07:08.099 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.0022211100000000001 2024-09-18T08:07:08.099 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:07:08.099 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:07:08.100 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "started", 2024-09-18T08:07:08.100 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:05:28.896555+0000", 2024-09-18T08:07:08.100 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.00011380899999999999 2024-09-18T08:07:08.100 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:07:08.100 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:07:08.100 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "waiting for subops from 3,6", 2024-09-18T08:07:08.100 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:05:28.896982+0000", 2024-09-18T08:07:08.100 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.000427064 2024-09-18T08:07:08.100 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:07:08.100 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:07:08.100 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "op_commit", 2024-09-18T08:07:08.100 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:05:28.946180+0000", 2024-09-18T08:07:08.100 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.049197850000000001 2024-09-18T08:07:08.101 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:07:08.101 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:07:08.101 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:07:08.101 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:05:28.946305+0000", 2024-09-18T08:07:08.101 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.000125258 2024-09-18T08:07:08.101 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:07:08.101 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:07:08.101 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:07:08.101 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:05:28.998206+0000", 2024-09-18T08:07:08.101 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.051901111 2024-09-18T08:07:08.101 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:07:08.101 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:07:08.102 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "commit_sent", 2024-09-18T08:07:08.102 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:05:28.998378+0000", 2024-09-18T08:07:08.102 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.000171886 2024-09-18T08:07:08.102 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:07:08.102 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:07:08.102 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "done", 2024-09-18T08:07:08.102 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:05:28.998477+0000", 2024-09-18T08:07:08.102 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 9.8623999999999996e-05 2024-09-18T08:07:08.102 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-09-18T08:07:08.102 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-09-18T08:07:08.102 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-09-18T08:07:08.102 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:07:08.102 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:07:08.103 INFO:teuthology.orchestra.run.smithi007.stdout: "description": "osd_op(client.14805.0:111942 7.f 7:ff6785bc:::benchmark_data_smithi007_73155_object111941:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e118)", 2024-09-18T08:07:08.103 INFO:teuthology.orchestra.run.smithi007.stdout: "initiated_at": "2024-09-18T08:05:28.894350+0000", 2024-09-18T08:07:08.103 INFO:teuthology.orchestra.run.smithi007.stdout: "age": 99.089641035, 2024-09-18T08:07:08.103 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.104224172, 2024-09-18T08:07:08.103 INFO:teuthology.orchestra.run.smithi007.stdout: "continuous": false, 2024-09-18T08:07:08.103 INFO:teuthology.orchestra.run.smithi007.stdout: "type_data": { 2024-09-18T08:07:08.103 INFO:teuthology.orchestra.run.smithi007.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T08:07:08.103 INFO:teuthology.orchestra.run.smithi007.stdout: "client_info": { 2024-09-18T08:07:08.103 INFO:teuthology.orchestra.run.smithi007.stdout: "client": "client.14805", 2024-09-18T08:07:08.103 INFO:teuthology.orchestra.run.smithi007.stdout: "client_addr": "172.21.15.7:0/640320436", 2024-09-18T08:07:08.103 INFO:teuthology.orchestra.run.smithi007.stdout: "tid": 111942 2024-09-18T08:07:08.103 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:07:08.103 INFO:teuthology.orchestra.run.smithi007.stdout: "events": [ 2024-09-18T08:07:08.103 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:07:08.104 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "initiated", 2024-09-18T08:07:08.104 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:05:28.894350+0000", 2024-09-18T08:07:08.104 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-09-18T08:07:08.104 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:07:08.104 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:07:08.104 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "header_read", 2024-09-18T08:07:08.104 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:05:28.894350+0000", 2024-09-18T08:07:08.104 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-09-18T08:07:08.104 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:07:08.104 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:07:08.104 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "throttled", 2024-09-18T08:07:08.104 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:05:28.894350+0000", 2024-09-18T08:07:08.104 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 8.78e-07 2024-09-18T08:07:08.104 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:07:08.104 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:07:08.105 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "all_read", 2024-09-18T08:07:08.105 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:05:28.894372+0000", 2024-09-18T08:07:08.105 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 2.1611e-05 2024-09-18T08:07:08.105 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:07:08.105 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:07:08.105 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "dispatched", 2024-09-18T08:07:08.105 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:05:28.894381+0000", 2024-09-18T08:07:08.105 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 8.7790000000000001e-06 2024-09-18T08:07:08.105 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:07:08.105 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:07:08.105 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "queued_for_pg", 2024-09-18T08:07:08.105 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:05:28.894387+0000", 2024-09-18T08:07:08.105 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 6.3090000000000001e-06 2024-09-18T08:07:08.105 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:07:08.106 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:07:08.106 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-09-18T08:07:08.106 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:05:28.946127+0000", 2024-09-18T08:07:08.106 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.051739947000000001 2024-09-18T08:07:08.106 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:07:08.106 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:07:08.106 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "started", 2024-09-18T08:07:08.106 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:05:28.946283+0000", 2024-09-18T08:07:08.106 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.000155728 2024-09-18T08:07:08.106 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:07:08.106 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:07:08.106 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "waiting for subops from 3,7", 2024-09-18T08:07:08.106 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:05:28.946447+0000", 2024-09-18T08:07:08.106 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.000164375 2024-09-18T08:07:08.106 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:07:08.107 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:07:08.107 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:07:08.107 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:05:28.954222+0000", 2024-09-18T08:07:08.107 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.0077748460000000002 2024-09-18T08:07:08.107 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:07:08.107 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:07:08.107 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "op_commit", 2024-09-18T08:07:08.107 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:05:28.957519+0000", 2024-09-18T08:07:08.107 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.0032968440000000002 2024-09-18T08:07:08.107 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:07:08.107 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:07:08.107 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:07:08.108 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:05:28.997653+0000", 2024-09-18T08:07:08.108 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.040134026000000003 2024-09-18T08:07:08.108 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:07:08.108 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:07:08.108 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "commit_sent", 2024-09-18T08:07:08.108 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:05:28.998536+0000", 2024-09-18T08:07:08.108 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.00088349700000000001 2024-09-18T08:07:08.108 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:07:08.108 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:07:08.108 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "done", 2024-09-18T08:07:08.108 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:05:28.998574+0000", 2024-09-18T08:07:08.108 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 3.7332000000000001e-05 2024-09-18T08:07:08.108 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-09-18T08:07:08.108 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-09-18T08:07:08.109 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-09-18T08:07:08.109 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:07:08.109 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:07:08.109 INFO:teuthology.orchestra.run.smithi007.stdout: "description": "osd_op(client.14805.0:113337 7.f 7:ff28b67f:::benchmark_data_smithi007_73155_object113336:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e119)", 2024-09-18T08:07:08.109 INFO:teuthology.orchestra.run.smithi007.stdout: "initiated_at": "2024-09-18T08:05:30.500430+0000", 2024-09-18T08:07:08.109 INFO:teuthology.orchestra.run.smithi007.stdout: "age": 97.483560655000005, 2024-09-18T08:07:08.109 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.112238877, 2024-09-18T08:07:08.109 INFO:teuthology.orchestra.run.smithi007.stdout: "continuous": false, 2024-09-18T08:07:08.109 INFO:teuthology.orchestra.run.smithi007.stdout: "type_data": { 2024-09-18T08:07:08.109 INFO:teuthology.orchestra.run.smithi007.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T08:07:08.109 INFO:teuthology.orchestra.run.smithi007.stdout: "client_info": { 2024-09-18T08:07:08.109 INFO:teuthology.orchestra.run.smithi007.stdout: "client": "client.14805", 2024-09-18T08:07:08.109 INFO:teuthology.orchestra.run.smithi007.stdout: "client_addr": "172.21.15.7:0/640320436", 2024-09-18T08:07:08.109 INFO:teuthology.orchestra.run.smithi007.stdout: "tid": 113337 2024-09-18T08:07:08.110 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:07:08.110 INFO:teuthology.orchestra.run.smithi007.stdout: "events": [ 2024-09-18T08:07:08.110 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:07:08.110 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "initiated", 2024-09-18T08:07:08.110 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:05:30.500430+0000", 2024-09-18T08:07:08.110 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-09-18T08:07:08.110 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:07:08.110 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:07:08.110 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "header_read", 2024-09-18T08:07:08.110 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:05:30.500430+0000", 2024-09-18T08:07:08.110 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-09-18T08:07:08.110 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:07:08.110 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:07:08.110 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "throttled", 2024-09-18T08:07:08.111 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:05:30.500430+0000", 2024-09-18T08:07:08.111 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 4.6600000000000002e-07 2024-09-18T08:07:08.111 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:07:08.111 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:07:08.111 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "all_read", 2024-09-18T08:07:08.111 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:05:30.500451+0000", 2024-09-18T08:07:08.111 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 2.0979999999999999e-05 2024-09-18T08:07:08.111 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:07:08.111 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:07:08.111 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "dispatched", 2024-09-18T08:07:08.111 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:05:30.500458+0000", 2024-09-18T08:07:08.111 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 6.934e-06 2024-09-18T08:07:08.111 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:07:08.111 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:07:08.112 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "queued_for_pg", 2024-09-18T08:07:08.112 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:05:30.500463+0000", 2024-09-18T08:07:08.112 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 5.2100000000000001e-06 2024-09-18T08:07:08.112 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:07:08.112 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:07:08.112 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-09-18T08:07:08.112 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:05:30.504713+0000", 2024-09-18T08:07:08.112 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.0042491530000000003 2024-09-18T08:07:08.112 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:07:08.113 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:07:08.113 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "started", 2024-09-18T08:07:08.113 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:05:30.505108+0000", 2024-09-18T08:07:08.113 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.00039518499999999999 2024-09-18T08:07:08.113 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:07:08.113 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:07:08.113 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "waiting for subops from 3,7", 2024-09-18T08:07:08.113 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:05:30.509160+0000", 2024-09-18T08:07:08.113 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.0040521990000000003 2024-09-18T08:07:08.113 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:07:08.113 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:07:08.114 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:07:08.114 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:05:30.517708+0000", 2024-09-18T08:07:08.114 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.0085477230000000001 2024-09-18T08:07:08.114 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:07:08.114 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:07:08.114 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "op_commit", 2024-09-18T08:07:08.114 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:05:30.524013+0000", 2024-09-18T08:07:08.114 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.00630526 2024-09-18T08:07:08.114 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:07:08.114 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:07:08.114 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:07:08.114 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:05:30.612568+0000", 2024-09-18T08:07:08.114 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.088554747000000003 2024-09-18T08:07:08.115 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:07:08.115 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:07:08.115 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "commit_sent", 2024-09-18T08:07:08.115 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:05:30.612616+0000", 2024-09-18T08:07:08.115 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 4.8602e-05 2024-09-18T08:07:08.115 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:07:08.115 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:07:08.115 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "done", 2024-09-18T08:07:08.115 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:05:30.612669+0000", 2024-09-18T08:07:08.115 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 5.2417999999999999e-05 2024-09-18T08:07:08.115 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-09-18T08:07:08.115 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-09-18T08:07:08.115 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-09-18T08:07:08.115 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:07:08.116 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:07:08.116 INFO:teuthology.orchestra.run.smithi007.stdout: "description": "osd_repop(client.14805.0:177709 7.e e139/83 7:780b878d:::benchmark_data_smithi007_73155_object59418:head v 139'11129, mlcod=139'11125)", 2024-09-18T08:07:08.116 INFO:teuthology.orchestra.run.smithi007.stdout: "initiated_at": "2024-09-18T08:06:38.572705+0000", 2024-09-18T08:07:08.116 INFO:teuthology.orchestra.run.smithi007.stdout: "age": 29.411285681999999, 2024-09-18T08:07:08.116 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.10862872899999999, 2024-09-18T08:07:08.116 INFO:teuthology.orchestra.run.smithi007.stdout: "continuous": false, 2024-09-18T08:07:08.116 INFO:teuthology.orchestra.run.smithi007.stdout: "type_data": { 2024-09-18T08:07:08.116 INFO:teuthology.orchestra.run.smithi007.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T08:07:08.116 INFO:teuthology.orchestra.run.smithi007.stdout: "events": [ 2024-09-18T08:07:08.116 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:07:08.116 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "initiated", 2024-09-18T08:07:08.116 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:06:38.572705+0000", 2024-09-18T08:07:08.116 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-09-18T08:07:08.116 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:07:08.117 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:07:08.117 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "header_read", 2024-09-18T08:07:08.117 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:06:38.572705+0000", 2024-09-18T08:07:08.117 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-09-18T08:07:08.117 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:07:08.117 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:07:08.117 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "throttled", 2024-09-18T08:07:08.117 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:06:38.572706+0000", 2024-09-18T08:07:08.117 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 7.6400000000000001e-07 2024-09-18T08:07:08.117 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:07:08.117 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:07:08.117 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "all_read", 2024-09-18T08:07:08.117 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:06:38.572717+0000", 2024-09-18T08:07:08.117 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.1555e-05 2024-09-18T08:07:08.117 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:07:08.118 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:07:08.118 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "dispatched", 2024-09-18T08:07:08.118 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:06:38.572732+0000", 2024-09-18T08:07:08.118 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.4555e-05 2024-09-18T08:07:08.118 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:07:08.118 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:07:08.118 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "queued_for_pg", 2024-09-18T08:07:08.118 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:06:38.572738+0000", 2024-09-18T08:07:08.118 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 6.4180000000000002e-06 2024-09-18T08:07:08.118 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:07:08.118 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:07:08.118 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-09-18T08:07:08.118 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:06:38.572827+0000", 2024-09-18T08:07:08.118 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 8.8611999999999998e-05 2024-09-18T08:07:08.118 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:07:08.119 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:07:08.119 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "started", 2024-09-18T08:07:08.119 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:06:38.572844+0000", 2024-09-18T08:07:08.119 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.7416000000000001e-05 2024-09-18T08:07:08.119 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:07:08.119 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:07:08.119 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "commit_sent", 2024-09-18T08:07:08.119 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:06:38.681294+0000", 2024-09-18T08:07:08.119 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.108450168 2024-09-18T08:07:08.119 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:07:08.119 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:07:08.119 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "done", 2024-09-18T08:07:08.119 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:06:38.681334+0000", 2024-09-18T08:07:08.119 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 3.9240999999999999e-05 2024-09-18T08:07:08.119 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-09-18T08:07:08.120 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-09-18T08:07:08.120 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-09-18T08:07:08.120 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:07:08.120 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:07:08.120 INFO:teuthology.orchestra.run.smithi007.stdout: "description": "osd_op(client.14805.0:177712 7.6 7:6af3d5cb:::benchmark_data_smithi007_73155_object59421:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e139)", 2024-09-18T08:07:08.120 INFO:teuthology.orchestra.run.smithi007.stdout: "initiated_at": "2024-09-18T08:06:38.573455+0000", 2024-09-18T08:07:08.120 INFO:teuthology.orchestra.run.smithi007.stdout: "age": 29.410535663000001, 2024-09-18T08:07:08.120 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.108223613, 2024-09-18T08:07:08.120 INFO:teuthology.orchestra.run.smithi007.stdout: "continuous": false, 2024-09-18T08:07:08.121 INFO:teuthology.orchestra.run.smithi007.stdout: "type_data": { 2024-09-18T08:07:08.121 INFO:teuthology.orchestra.run.smithi007.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T08:07:08.121 INFO:teuthology.orchestra.run.smithi007.stdout: "client_info": { 2024-09-18T08:07:08.121 INFO:teuthology.orchestra.run.smithi007.stdout: "client": "client.14805", 2024-09-18T08:07:08.121 INFO:teuthology.orchestra.run.smithi007.stdout: "client_addr": "172.21.15.7:0/640320436", 2024-09-18T08:07:08.121 INFO:teuthology.orchestra.run.smithi007.stdout: "tid": 177712 2024-09-18T08:07:08.121 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:07:08.121 INFO:teuthology.orchestra.run.smithi007.stdout: "events": [ 2024-09-18T08:07:08.121 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:07:08.121 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "initiated", 2024-09-18T08:07:08.121 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:06:38.573455+0000", 2024-09-18T08:07:08.121 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-09-18T08:07:08.122 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:07:08.122 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:07:08.122 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "header_read", 2024-09-18T08:07:08.122 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:06:38.573455+0000", 2024-09-18T08:07:08.122 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-09-18T08:07:08.122 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:07:08.122 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:07:08.122 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "throttled", 2024-09-18T08:07:08.122 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:06:38.573456+0000", 2024-09-18T08:07:08.122 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.3319999999999999e-06 2024-09-18T08:07:08.122 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:07:08.123 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:07:08.123 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "all_read", 2024-09-18T08:07:08.123 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:06:38.573462+0000", 2024-09-18T08:07:08.123 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 5.3190000000000002e-06 2024-09-18T08:07:08.123 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:07:08.123 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:07:08.123 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "dispatched", 2024-09-18T08:07:08.123 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:06:38.573471+0000", 2024-09-18T08:07:08.123 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 9.8330000000000006e-06 2024-09-18T08:07:08.123 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:07:08.123 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:07:08.123 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "queued_for_pg", 2024-09-18T08:07:08.123 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:06:38.573479+0000", 2024-09-18T08:07:08.123 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 7.6580000000000002e-06 2024-09-18T08:07:08.123 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:07:08.124 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:07:08.124 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-09-18T08:07:08.124 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:06:38.573578+0000", 2024-09-18T08:07:08.124 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 9.9283000000000005e-05 2024-09-18T08:07:08.124 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:07:08.124 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:07:08.124 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "started", 2024-09-18T08:07:08.124 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:06:38.573677+0000", 2024-09-18T08:07:08.124 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 9.8191999999999998e-05 2024-09-18T08:07:08.124 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:07:08.124 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:07:08.124 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "waiting for subops from 3,7", 2024-09-18T08:07:08.124 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:06:38.574106+0000", 2024-09-18T08:07:08.125 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.00042906800000000001 2024-09-18T08:07:08.125 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:07:08.125 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:07:08.125 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:07:08.125 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:06:38.671566+0000", 2024-09-18T08:07:08.125 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.097460049000000007 2024-09-18T08:07:08.125 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:07:08.125 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:07:08.125 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:07:08.125 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:06:38.671608+0000", 2024-09-18T08:07:08.125 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 4.2558999999999997e-05 2024-09-18T08:07:08.125 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:07:08.126 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:07:08.126 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "op_commit", 2024-09-18T08:07:08.126 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:06:38.681341+0000", 2024-09-18T08:07:08.126 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.0097332440000000003 2024-09-18T08:07:08.126 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:07:08.126 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:07:08.126 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "commit_sent", 2024-09-18T08:07:08.126 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:06:38.681401+0000", 2024-09-18T08:07:08.126 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 5.9629000000000001e-05 2024-09-18T08:07:08.126 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:07:08.126 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:07:08.127 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "done", 2024-09-18T08:07:08.127 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:06:38.681678+0000", 2024-09-18T08:07:08.127 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.000277447 2024-09-18T08:07:08.127 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-09-18T08:07:08.127 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-09-18T08:07:08.127 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-09-18T08:07:08.127 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-09-18T08:07:08.127 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-09-18T08:07:08.127 INFO:teuthology.orchestra.run.smithi007.stdout:} 2024-09-18T08:07:08.147 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:07:08.147 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:07:08.148 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3' 2024-09-18T08:07:08.217 INFO:teuthology.orchestra.run.smithi007.stderr:noscrub is unset 2024-09-18T08:07:08.248 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3 2024-09-18T08:07:08.290 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:07:08.291 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:07:08.291 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3' 2024-09-18T08:07:08.392 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0 2024-09-18T08:07:08.439 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:07:08.439 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:07:08.440 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0' 2024-09-18T08:07:08.462 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:07:08 smithi007 ceph-mon[29459]: pgmap v193: 148 pgs: 148 active+clean; 2.2 GiB data, 7.5 GiB used, 708 GiB / 715 GiB avail; 594 B/s rd, 0 B/s wr, 770 op/s 2024-09-18T08:07:08.462 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:07:08 smithi007 ceph-mon[29459]: from='client.? 172.21.15.7:0/949912570' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-18T08:07:08.462 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:07:08 smithi007 ceph-mon[39249]: pgmap v193: 148 pgs: 148 active+clean; 2.2 GiB data, 7.5 GiB used, 708 GiB / 715 GiB avail; 594 B/s rd, 0 B/s wr, 770 op/s 2024-09-18T08:07:08.463 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:07:08 smithi007 ceph-mon[39249]: from='client.? 172.21.15.7:0/949912570' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-18T08:07:08.540 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4 2024-09-18T08:07:08.567 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:07:08.568 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:07:08.569 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4' 2024-09-18T08:07:08.656 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:07:08 smithi050 ceph-mon[32371]: pgmap v193: 148 pgs: 148 active+clean; 2.2 GiB data, 7.5 GiB used, 708 GiB / 715 GiB avail; 594 B/s rd, 0 B/s wr, 770 op/s 2024-09-18T08:07:08.656 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:07:08 smithi050 ceph-mon[32371]: from='client.? 172.21.15.7:0/949912570' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-18T08:07:08.669 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5 2024-09-18T08:07:08.696 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:07:08.696 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:07:08.697 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5' 2024-09-18T08:07:08.797 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4 2024-09-18T08:07:08.825 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:07:08.826 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:07:08.827 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4' 2024-09-18T08:07:08.927 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5 2024-09-18T08:07:08.955 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:07:08.956 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:07:08.956 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5' 2024-09-18T08:07:09.057 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4 2024-09-18T08:07:09.084 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:07:09.085 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:07:09.085 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4' 2024-09-18T08:07:09.186 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1 2024-09-18T08:07:09.234 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:07:09.234 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:07:09.235 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1' 2024-09-18T08:07:09.241 INFO:journalctl@ceph.iscsi.iscsi.a.smithi050.stdout:Sep 18 08:07:08 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-iscsi-iscsi-a[57782]: debug there is no tcmu-runner data available 2024-09-18T08:07:09.335 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1 2024-09-18T08:07:09.383 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:07:09.383 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:07:09.384 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1' 2024-09-18T08:07:09.485 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6 2024-09-18T08:07:09.511 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:07:09.512 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:07:09.512 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6' 2024-09-18T08:07:09.568 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:07:09 smithi007 ceph-mon[29459]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-18T08:07:09.568 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:07:09 smithi007 ceph-mon[29459]: from='client.? 172.21.15.7:0/949912570' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-18T08:07:09.568 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:07:09 smithi007 ceph-mon[29459]: osdmap e149: 8 total, 8 up, 8 in 2024-09-18T08:07:09.568 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:07:09 smithi007 ceph-mon[29459]: 6.6 scrub starts 2024-09-18T08:07:09.568 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:07:09 smithi007 ceph-mon[29459]: 6.6 scrub ok 2024-09-18T08:07:09.569 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:07:09 smithi007 ceph-mon[29459]: 5.1d scrub starts 2024-09-18T08:07:09.569 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:07:09 smithi007 ceph-mon[29459]: 5.1d scrub ok 2024-09-18T08:07:09.569 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:07:09 smithi007 ceph-mon[29459]: 5.6 scrub starts 2024-09-18T08:07:09.569 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:07:09 smithi007 ceph-mon[29459]: 5.6 scrub ok 2024-09-18T08:07:09.569 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:07:09 smithi007 ceph-mon[39249]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-18T08:07:09.569 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:07:09 smithi007 ceph-mon[39249]: from='client.? 172.21.15.7:0/949912570' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-18T08:07:09.569 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:07:09 smithi007 ceph-mon[39249]: osdmap e149: 8 total, 8 up, 8 in 2024-09-18T08:07:09.569 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:07:09 smithi007 ceph-mon[39249]: 6.6 scrub starts 2024-09-18T08:07:09.569 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:07:09 smithi007 ceph-mon[39249]: 6.6 scrub ok 2024-09-18T08:07:09.569 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:07:09 smithi007 ceph-mon[39249]: 5.1d scrub starts 2024-09-18T08:07:09.569 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:07:09 smithi007 ceph-mon[39249]: 5.1d scrub ok 2024-09-18T08:07:09.569 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:07:09 smithi007 ceph-mon[39249]: 5.6 scrub starts 2024-09-18T08:07:09.570 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:07:09 smithi007 ceph-mon[39249]: 5.6 scrub ok 2024-09-18T08:07:09.613 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0 2024-09-18T08:07:09.649 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:07:09.649 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:07:09.650 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0' 2024-09-18T08:07:09.655 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:07:09 smithi050 ceph-mon[32371]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-18T08:07:09.656 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:07:09 smithi050 ceph-mon[32371]: from='client.? 172.21.15.7:0/949912570' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-18T08:07:09.656 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:07:09 smithi050 ceph-mon[32371]: osdmap e149: 8 total, 8 up, 8 in 2024-09-18T08:07:09.656 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:07:09 smithi050 ceph-mon[32371]: 6.6 scrub starts 2024-09-18T08:07:09.656 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:07:09 smithi050 ceph-mon[32371]: 6.6 scrub ok 2024-09-18T08:07:09.656 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:07:09 smithi050 ceph-mon[32371]: 5.1d scrub starts 2024-09-18T08:07:09.656 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:07:09 smithi050 ceph-mon[32371]: 5.1d scrub ok 2024-09-18T08:07:09.656 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:07:09 smithi050 ceph-mon[32371]: 5.6 scrub starts 2024-09-18T08:07:09.656 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:07:09 smithi050 ceph-mon[32371]: 5.6 scrub ok 2024-09-18T08:07:09.751 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2 2024-09-18T08:07:09.786 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:07:09.787 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:07:09.787 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2' 2024-09-18T08:07:09.888 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1 2024-09-18T08:07:09.928 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:07:09.929 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:07:09.930 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1' 2024-09-18T08:07:10.031 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1 2024-09-18T08:07:10.074 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:07:10.075 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:07:10.075 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1' 2024-09-18T08:07:10.117 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid 779349ca-7593-11ef-bceb-c7b262605968 -- ceph daemon osd.1 dump_ops_in_flight 2024-09-18T08:07:10.176 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1 2024-09-18T08:07:10.220 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/779349ca-7593-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T08:07:10.274 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:07:10.274 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:07:10.275 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1' 2024-09-18T08:07:10.371 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:07:10 smithi007 ceph-mon[29459]: 6.1a scrub starts 2024-09-18T08:07:10.372 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:07:10 smithi007 ceph-mon[29459]: 6.1a scrub ok 2024-09-18T08:07:10.372 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:07:10 smithi007 ceph-mon[29459]: 7.6 scrub starts 2024-09-18T08:07:10.372 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:07:10 smithi007 ceph-mon[29459]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-18T08:07:10.372 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:07:10 smithi007 ceph-mon[29459]: pgmap v195: 148 pgs: 148 active+clean; 1.8 GiB data, 6.5 GiB used, 709 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.18k op/s 2024-09-18T08:07:10.372 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:07:10 smithi007 ceph-mon[29459]: 6.c scrub starts 2024-09-18T08:07:10.372 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:07:10 smithi007 ceph-mon[29459]: 6.c scrub ok 2024-09-18T08:07:10.372 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:07:10 smithi007 ceph-mon[29459]: 5.4 scrub starts 2024-09-18T08:07:10.372 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:07:10 smithi007 ceph-mon[29459]: 5.4 scrub ok 2024-09-18T08:07:10.373 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:07:10 smithi007 ceph-mon[39249]: 6.1a scrub starts 2024-09-18T08:07:10.373 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:07:10 smithi007 ceph-mon[39249]: 6.1a scrub ok 2024-09-18T08:07:10.373 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:07:10 smithi007 ceph-mon[39249]: 7.6 scrub starts 2024-09-18T08:07:10.373 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:07:10 smithi007 ceph-mon[39249]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-18T08:07:10.373 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:07:10 smithi007 ceph-mon[39249]: pgmap v195: 148 pgs: 148 active+clean; 1.8 GiB data, 6.5 GiB used, 709 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.18k op/s 2024-09-18T08:07:10.373 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:07:10 smithi007 ceph-mon[39249]: 6.c scrub starts 2024-09-18T08:07:10.373 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:07:10 smithi007 ceph-mon[39249]: 6.c scrub ok 2024-09-18T08:07:10.373 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:07:10 smithi007 ceph-mon[39249]: 5.4 scrub starts 2024-09-18T08:07:10.373 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:07:10 smithi007 ceph-mon[39249]: 5.4 scrub ok 2024-09-18T08:07:10.376 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5 2024-09-18T08:07:10.402 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:07:10.403 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:07:10.404 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5' 2024-09-18T08:07:10.505 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2 2024-09-18T08:07:10.555 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:07:10.556 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:07:10.556 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2' 2024-09-18T08:07:10.656 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:07:10 smithi050 ceph-mon[32371]: 6.1a scrub starts 2024-09-18T08:07:10.656 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:07:10 smithi050 ceph-mon[32371]: 6.1a scrub ok 2024-09-18T08:07:10.656 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:07:10 smithi050 ceph-mon[32371]: 7.6 scrub starts 2024-09-18T08:07:10.656 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:07:10 smithi050 ceph-mon[32371]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-18T08:07:10.656 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:07:10 smithi050 ceph-mon[32371]: pgmap v195: 148 pgs: 148 active+clean; 1.8 GiB data, 6.5 GiB used, 709 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.18k op/s 2024-09-18T08:07:10.656 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:07:10 smithi050 ceph-mon[32371]: 6.c scrub starts 2024-09-18T08:07:10.656 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:07:10 smithi050 ceph-mon[32371]: 6.c scrub ok 2024-09-18T08:07:10.657 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:07:10 smithi050 ceph-mon[32371]: 5.4 scrub starts 2024-09-18T08:07:10.657 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:07:10 smithi050 ceph-mon[32371]: 5.4 scrub ok 2024-09-18T08:07:10.657 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4 2024-09-18T08:07:10.685 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:07:10.686 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:07:10.686 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4' 2024-09-18T08:07:10.787 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5 2024-09-18T08:07:10.816 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:07:10.816 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:07:10.817 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5' 2024-09-18T08:07:10.918 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6 2024-09-18T08:07:10.945 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:07:10.945 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:07:10.946 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6' 2024-09-18T08:07:11.047 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1 2024-09-18T08:07:11.086 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:07:11.087 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:07:11.087 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1' 2024-09-18T08:07:11.188 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3 2024-09-18T08:07:11.229 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:07:11.229 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:07:11.230 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3' 2024-09-18T08:07:11.287 INFO:teuthology.orchestra.run.smithi007.stdout:osd.0: {} 2024-09-18T08:07:11.288 INFO:teuthology.orchestra.run.smithi007.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-18T08:07:11.301 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/779349ca-7593-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T08:07:11.309 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 7, 6] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-09-18T08:07:11.309 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-18T08:07:11.309 INFO:tasks.thrashosds.thrasher:Growing pool .mgr 2024-09-18T08:07:11.309 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid 779349ca-7593-11ef-bceb-c7b262605968 -- ceph pg dump --format=json 2024-09-18T08:07:11.327 INFO:teuthology.orchestra.run.smithi007.stdout:osd.1: {} 2024-09-18T08:07:11.327 INFO:teuthology.orchestra.run.smithi007.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-18T08:07:11.330 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0 2024-09-18T08:07:11.365 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:07:11 smithi007 ceph-mon[39249]: 7.6 scrub ok 2024-09-18T08:07:11.365 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:07:11 smithi007 ceph-mon[39249]: 3.1e scrub starts 2024-09-18T08:07:11.365 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:07:11 smithi007 ceph-mon[39249]: 3.1e scrub ok 2024-09-18T08:07:11.365 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:07:11 smithi007 ceph-mon[39249]: 5.a scrub starts 2024-09-18T08:07:11.365 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:07:11 smithi007 ceph-mon[39249]: 5.a scrub ok 2024-09-18T08:07:11.366 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:07:11 smithi007 ceph-mon[39249]: from='client.? 172.21.15.7:0/473329352' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-18T08:07:11.366 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:07:11 smithi007 ceph-mon[29459]: 7.6 scrub ok 2024-09-18T08:07:11.366 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:07:11 smithi007 ceph-mon[29459]: 3.1e scrub starts 2024-09-18T08:07:11.366 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:07:11 smithi007 ceph-mon[29459]: 3.1e scrub ok 2024-09-18T08:07:11.366 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:07:11 smithi007 ceph-mon[29459]: 5.a scrub starts 2024-09-18T08:07:11.366 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:07:11 smithi007 ceph-mon[29459]: 5.a scrub ok 2024-09-18T08:07:11.366 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:07:11 smithi007 ceph-mon[29459]: from='client.? 172.21.15.7:0/473329352' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-18T08:07:11.375 INFO:teuthology.orchestra.run.smithi007.stdout:osd.2: {} 2024-09-18T08:07:11.375 INFO:teuthology.orchestra.run.smithi007.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-18T08:07:11.401 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:07:11.401 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:07:11.402 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0' 2024-09-18T08:07:11.418 INFO:teuthology.orchestra.run.smithi007.stdout:osd.3: {} 2024-09-18T08:07:11.418 INFO:teuthology.orchestra.run.smithi007.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-18T08:07:11.452 INFO:teuthology.orchestra.run.smithi007.stdout:osd.4: {} 2024-09-18T08:07:11.452 INFO:teuthology.orchestra.run.smithi007.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-18T08:07:11.484 INFO:teuthology.orchestra.run.smithi007.stdout:osd.5: {} 2024-09-18T08:07:11.484 INFO:teuthology.orchestra.run.smithi007.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-18T08:07:11.503 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0 2024-09-18T08:07:11.523 INFO:teuthology.orchestra.run.smithi007.stdout:osd.6: {} 2024-09-18T08:07:11.524 INFO:teuthology.orchestra.run.smithi007.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-18T08:07:11.557 INFO:teuthology.orchestra.run.smithi007.stdout:osd.7: {} 2024-09-18T08:07:11.558 INFO:teuthology.orchestra.run.smithi007.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-18T08:07:11.566 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:07:11.567 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:07:11.568 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0' 2024-09-18T08:07:11.618 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid 779349ca-7593-11ef-bceb-c7b262605968 -- ceph osd unset nodeep-scrub 2024-09-18T08:07:11.656 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:07:11 smithi050 ceph-mon[32371]: 7.6 scrub ok 2024-09-18T08:07:11.656 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:07:11 smithi050 ceph-mon[32371]: 3.1e scrub starts 2024-09-18T08:07:11.656 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:07:11 smithi050 ceph-mon[32371]: 3.1e scrub ok 2024-09-18T08:07:11.656 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:07:11 smithi050 ceph-mon[32371]: 5.a scrub starts 2024-09-18T08:07:11.656 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:07:11 smithi050 ceph-mon[32371]: 5.a scrub ok 2024-09-18T08:07:11.656 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:07:11 smithi050 ceph-mon[32371]: from='client.? 172.21.15.7:0/473329352' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-18T08:07:11.669 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0 2024-09-18T08:07:11.720 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:07:11.720 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:07:11.720 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0' 2024-09-18T08:07:11.821 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2 2024-09-18T08:07:11.851 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:07:11.852 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:07:11.852 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2' 2024-09-18T08:07:11.953 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4 2024-09-18T08:07:11.978 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:07:11.978 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:07:11.979 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4' 2024-09-18T08:07:12.079 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1 2024-09-18T08:07:12.129 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:07:12.129 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:07:12.130 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1' 2024-09-18T08:07:12.230 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0 2024-09-18T08:07:12.258 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:07:12.259 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:07:12.259 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0' 2024-09-18T08:07:12.360 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0 2024-09-18T08:07:12.400 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:07:12.401 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:07:12.401 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0' 2024-09-18T08:07:12.502 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7 2024-09-18T08:07:12.529 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:07:12.529 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:07:12.530 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7' 2024-09-18T08:07:12.556 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:07:12 smithi050 ceph-mon[32371]: 3.1 scrub starts 2024-09-18T08:07:12.556 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:07:12 smithi050 ceph-mon[32371]: 3.1 scrub ok 2024-09-18T08:07:12.556 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:07:12 smithi050 ceph-mon[32371]: 6.5 scrub starts 2024-09-18T08:07:12.556 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:07:12 smithi050 ceph-mon[32371]: 6.5 scrub ok 2024-09-18T08:07:12.556 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:07:12 smithi050 ceph-mon[32371]: pgmap v196: 148 pgs: 148 active+clean; 1.8 GiB data, 6.5 GiB used, 709 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.01k op/s 2024-09-18T08:07:12.559 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:07:12 smithi007 ceph-mon[29459]: 3.1 scrub starts 2024-09-18T08:07:12.559 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:07:12 smithi007 ceph-mon[29459]: 3.1 scrub ok 2024-09-18T08:07:12.559 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:07:12 smithi007 ceph-mon[29459]: 6.5 scrub starts 2024-09-18T08:07:12.559 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:07:12 smithi007 ceph-mon[29459]: 6.5 scrub ok 2024-09-18T08:07:12.559 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:07:12 smithi007 ceph-mon[29459]: pgmap v196: 148 pgs: 148 active+clean; 1.8 GiB data, 6.5 GiB used, 709 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.01k op/s 2024-09-18T08:07:12.559 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:07:12 smithi007 ceph-mon[39249]: 3.1 scrub starts 2024-09-18T08:07:12.559 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:07:12 smithi007 ceph-mon[39249]: 3.1 scrub ok 2024-09-18T08:07:12.559 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:07:12 smithi007 ceph-mon[39249]: 6.5 scrub starts 2024-09-18T08:07:12.560 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:07:12 smithi007 ceph-mon[39249]: 6.5 scrub ok 2024-09-18T08:07:12.560 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:07:12 smithi007 ceph-mon[39249]: pgmap v196: 148 pgs: 148 active+clean; 1.8 GiB data, 6.5 GiB used, 709 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.01k op/s 2024-09-18T08:07:12.631 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4 2024-09-18T08:07:12.656 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:07:12.657 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:07:12.657 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4' 2024-09-18T08:07:12.758 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2 2024-09-18T08:07:12.795 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:07:12.796 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:07:12.796 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2' 2024-09-18T08:07:12.897 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2 2024-09-18T08:07:12.939 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:07:12.940 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:07:12.940 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2' 2024-09-18T08:07:13.041 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5 2024-09-18T08:07:13.050 INFO:teuthology.orchestra.run.smithi007.stdout:ERROR: (22) Invalid argument 2024-09-18T08:07:13.050 INFO:teuthology.orchestra.run.smithi007.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-18T08:07:13.066 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:07:13.067 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:07:13.067 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5' 2024-09-18T08:07:13.168 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3 2024-09-18T08:07:13.213 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:07:13.214 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:07:13.214 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3' 2024-09-18T08:07:13.315 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7 2024-09-18T08:07:13.316 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:07:13 smithi007 ceph-mon[39249]: 6.9 scrub starts 2024-09-18T08:07:13.316 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:07:13 smithi007 ceph-mon[39249]: 6.9 scrub ok 2024-09-18T08:07:13.316 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:07:13 smithi007 ceph-mon[39249]: 7.d scrub starts 2024-09-18T08:07:13.317 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:07:13 smithi007 ceph-mon[29459]: 6.9 scrub starts 2024-09-18T08:07:13.317 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:07:13 smithi007 ceph-mon[29459]: 6.9 scrub ok 2024-09-18T08:07:13.317 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:07:13 smithi007 ceph-mon[29459]: 7.d scrub starts 2024-09-18T08:07:13.340 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:07:13.340 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:07:13.341 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7' 2024-09-18T08:07:13.442 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1 2024-09-18T08:07:13.482 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:07:13.482 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:07:13.483 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1' 2024-09-18T08:07:13.583 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4 2024-09-18T08:07:13.610 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:07:13.611 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:07:13.611 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4' 2024-09-18T08:07:13.656 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:07:13 smithi050 ceph-mon[32371]: 6.9 scrub starts 2024-09-18T08:07:13.656 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:07:13 smithi050 ceph-mon[32371]: 6.9 scrub ok 2024-09-18T08:07:13.656 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:07:13 smithi050 ceph-mon[32371]: 7.d scrub starts 2024-09-18T08:07:13.712 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2 2024-09-18T08:07:13.747 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:07:13.748 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:07:13.748 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2' 2024-09-18T08:07:13.849 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0 2024-09-18T08:07:13.887 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:07:13.889 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:07:13.889 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0' 2024-09-18T08:07:13.990 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3 2024-09-18T08:07:14.044 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:07:14.045 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:07:14.046 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3' 2024-09-18T08:07:14.147 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7 2024-09-18T08:07:14.178 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:07:14.178 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:07:14.178 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7' 2024-09-18T08:07:14.279 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3 2024-09-18T08:07:14.321 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:07:14.321 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:07:14.322 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3' 2024-09-18T08:07:14.423 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4 2024-09-18T08:07:14.434 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid 779349ca-7593-11ef-bceb-c7b262605968 -- ceph daemon osd.1 dump_blocked_ops 2024-09-18T08:07:14.434 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/779349ca-7593-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T08:07:14.447 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:07:14.448 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:07:14.448 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4' 2024-09-18T08:07:14.494 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/779349ca-7593-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T08:07:14.549 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5 2024-09-18T08:07:14.575 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:07:14.576 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:07:14.577 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5' 2024-09-18T08:07:14.585 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:07:14 smithi007 ceph-mon[29459]: pgmap v197: 148 pgs: 148 active+clean; 1.8 GiB data, 6.2 GiB used, 709 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.02k op/s 2024-09-18T08:07:14.585 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:07:14 smithi007 ceph-mon[29459]: 6.1b scrub starts 2024-09-18T08:07:14.585 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:07:14 smithi007 ceph-mon[29459]: 6.1b scrub ok 2024-09-18T08:07:14.585 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:07:14 smithi007 ceph-mon[29459]: 7.d scrub ok 2024-09-18T08:07:14.585 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:07:14 smithi007 ceph-mon[39249]: pgmap v197: 148 pgs: 148 active+clean; 1.8 GiB data, 6.2 GiB used, 709 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.02k op/s 2024-09-18T08:07:14.585 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:07:14 smithi007 ceph-mon[39249]: 6.1b scrub starts 2024-09-18T08:07:14.585 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:07:14 smithi007 ceph-mon[39249]: 6.1b scrub ok 2024-09-18T08:07:14.586 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:07:14 smithi007 ceph-mon[39249]: 7.d scrub ok 2024-09-18T08:07:14.656 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:07:14 smithi050 ceph-mon[32371]: pgmap v197: 148 pgs: 148 active+clean; 1.8 GiB data, 6.2 GiB used, 709 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.02k op/s 2024-09-18T08:07:14.656 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:07:14 smithi050 ceph-mon[32371]: 6.1b scrub starts 2024-09-18T08:07:14.656 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:07:14 smithi050 ceph-mon[32371]: 6.1b scrub ok 2024-09-18T08:07:14.656 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:07:14 smithi050 ceph-mon[32371]: 7.d scrub ok 2024-09-18T08:07:14.677 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4 2024-09-18T08:07:14.703 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:07:14.704 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:07:14.705 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4' 2024-09-18T08:07:14.806 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0 2024-09-18T08:07:14.847 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:07:14.848 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:07:14.848 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0' 2024-09-18T08:07:14.949 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7 2024-09-18T08:07:14.975 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:07:14.976 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:07:14.976 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7' 2024-09-18T08:07:15.077 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4 2024-09-18T08:07:15.103 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:07:15.103 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:07:15.104 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4' 2024-09-18T08:07:15.204 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4 2024-09-18T08:07:15.230 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:07:15.231 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:07:15.231 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4' 2024-09-18T08:07:15.332 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1 2024-09-18T08:07:15.376 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:07:15.377 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:07:15.378 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1' 2024-09-18T08:07:15.478 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7 2024-09-18T08:07:15.480 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:07:15 smithi007 ceph-mon[29459]: 3.1a scrub starts 2024-09-18T08:07:15.480 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:07:15 smithi007 ceph-mon[29459]: 3.1a scrub ok 2024-09-18T08:07:15.481 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:07:15 smithi007 ceph-mon[39249]: 3.1a scrub starts 2024-09-18T08:07:15.482 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:07:15 smithi007 ceph-mon[39249]: 3.1a scrub ok 2024-09-18T08:07:15.502 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:07:15.502 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:07:15.503 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7' 2024-09-18T08:07:15.604 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7 2024-09-18T08:07:15.627 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:07:15.628 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:07:15.628 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7' 2024-09-18T08:07:15.656 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:07:15 smithi050 ceph-mon[32371]: 3.1a scrub starts 2024-09-18T08:07:15.656 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:07:15 smithi050 ceph-mon[32371]: 3.1a scrub ok 2024-09-18T08:07:15.724 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/779349ca-7593-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T08:07:15.729 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0 2024-09-18T08:07:15.750 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-09-18T08:07:15.752 INFO:teuthology.orchestra.run.smithi007.stderr:dumped all 2024-09-18T08:07:15.787 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:07:15.787 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:07:15.788 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0' 2024-09-18T08:07:15.888 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6 2024-09-18T08:07:15.916 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:07:15.917 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:07:15.917 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6' 2024-09-18T08:07:16.000 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid 779349ca-7593-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-18T08:07:16.018 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5 2024-09-18T08:07:16.043 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:07:16.044 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:07:16.044 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5' 2024-09-18T08:07:16.145 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5 2024-09-18T08:07:16.172 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:07:16.172 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:07:16.173 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5' 2024-09-18T08:07:16.273 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3 2024-09-18T08:07:16.304 INFO:teuthology.orchestra.run.smithi007.stderr:nodeep-scrub is unset 2024-09-18T08:07:16.336 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:07:16.337 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:07:16.338 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3' 2024-09-18T08:07:16.439 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7 2024-09-18T08:07:16.465 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:07:16.466 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:07:16.466 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7' 2024-09-18T08:07:16.567 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1 2024-09-18T08:07:16.602 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:07:16.603 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:07:16.603 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1' 2024-09-18T08:07:16.656 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:07:16 smithi050 ceph-mon[32371]: 6.16 scrub starts 2024-09-18T08:07:16.656 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:07:16 smithi050 ceph-mon[32371]: 6.16 scrub ok 2024-09-18T08:07:16.656 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:07:16 smithi050 ceph-mon[32371]: pgmap v198: 148 pgs: 1 active+clean+scrubbing, 147 active+clean; 1.5 GiB data, 5.6 GiB used, 710 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 998 op/s 2024-09-18T08:07:16.656 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:07:16 smithi050 ceph-mon[32371]: from='client.? 172.21.15.7:0/1205788035' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-18T08:07:16.656 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:07:16 smithi050 ceph-mon[32371]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-18T08:07:16.698 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:07:16 smithi007 ceph-mon[39249]: 6.16 scrub starts 2024-09-18T08:07:16.698 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:07:16 smithi007 ceph-mon[39249]: 6.16 scrub ok 2024-09-18T08:07:16.698 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:07:16 smithi007 ceph-mon[39249]: pgmap v198: 148 pgs: 1 active+clean+scrubbing, 147 active+clean; 1.5 GiB data, 5.6 GiB used, 710 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 998 op/s 2024-09-18T08:07:16.698 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:07:16 smithi007 ceph-mon[39249]: from='client.? 172.21.15.7:0/1205788035' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-18T08:07:16.698 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:07:16 smithi007 ceph-mon[39249]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-18T08:07:16.699 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:07:16 smithi007 ceph-mon[29459]: 6.16 scrub starts 2024-09-18T08:07:16.699 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:07:16 smithi007 ceph-mon[29459]: 6.16 scrub ok 2024-09-18T08:07:16.699 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:07:16 smithi007 ceph-mon[29459]: pgmap v198: 148 pgs: 1 active+clean+scrubbing, 147 active+clean; 1.5 GiB data, 5.6 GiB used, 710 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 998 op/s 2024-09-18T08:07:16.699 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:07:16 smithi007 ceph-mon[29459]: from='client.? 172.21.15.7:0/1205788035' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-18T08:07:16.699 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:07:16 smithi007 ceph-mon[29459]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-18T08:07:16.704 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3 2024-09-18T08:07:16.744 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:07:16.745 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:07:16.745 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3' 2024-09-18T08:07:16.846 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1 2024-09-18T08:07:16.879 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:07:16.881 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:07:16.881 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1' 2024-09-18T08:07:16.956 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Sep 18 08:07:16 smithi007 ceph-779349ca-7593-11ef-bceb-c7b262605968-mgr-y[29688]: ::ffff:172.21.15.50 - - [18/Sep/2024:08:07:16] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-09-18T08:07:16.976 INFO:teuthology.orchestra.run.smithi007.stdout:{"pg_ready":true,"pg_map":{"version":198,"stamp":"2024-09-18T08:07:15.157315+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":1662196275,"num_objects":25554,"num_object_clones":0,"num_object_copies":76662,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":25554,"num_whiteouts":0,"num_read":879,"num_read_kb":736,"num_write":211772,"num_write_kb":7572106,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2792,"num_bytes_recovered":173871186,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":88434,"ondisk_log_size":88434,"up":444,"acting":444,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":444,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":5908264,"kb_used_data":4992360,"kb_used_omap":12,"kb_used_meta":915699,"kb_avail":744052952,"statfs":{"total":767960285184,"available":761910222848,"internally_reserved":0,"allocated":5112176640,"data_stored":5109351289,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12750,"internal_metadata":937676338},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,6,1,1,3,2],"upper_bound":64},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-654508032,"num_objects":-9987,"num_object_clones":0,"num_object_copies":-29961,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-9987,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":9987,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.007523"},"pg_stats":[{"pgid":"3.1f","version":"0'0","reported_seq":196,"reported_epoch":149,"state":"active+clean","last_fresh":"2024-09-18T08:07:08.229147+0000","last_change":"2024-09-18T08:04:34.226236+0000","last_active":"2024-09-18T08:07:08.229147+0000","last_peered":"2024-09-18T08:07:08.229147+0000","last_clean":"2024-09-18T08:07:08.229147+0000","last_became_active":"2024-09-18T08:02:47.877698+0000","last_became_peered":"2024-09-18T08:02:47.877698+0000","last_unstale":"2024-09-18T08:07:08.229147+0000","last_undegraded":"2024-09-18T08:07:08.229147+0000","last_fullsized":"2024-09-18T08:07:08.229147+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:04:34.226054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:46.860092+0000","last_clean_scrub_stamp":"2024-09-18T08:04:34.226054+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:07:54.705201+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"69'9","reported_seq":241,"reported_epoch":149,"state":"active+clean","last_fresh":"2024-09-18T08:07:08.214154+0000","last_change":"2024-09-18T08:06:24.171122+0000","last_active":"2024-09-18T08:07:08.214154+0000","last_peered":"2024-09-18T08:07:08.214154+0000","last_clean":"2024-09-18T08:07:08.214154+0000","last_became_active":"2024-09-18T08:06:24.140103+0000","last_became_peered":"2024-09-18T08:06:24.140103+0000","last_unstale":"2024-09-18T08:07:08.214154+0000","last_undegraded":"2024-09-18T08:07:08.214154+0000","last_fullsized":"2024-09-18T08:07:08.214154+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'9","last_scrub_stamp":"2024-09-18T08:05:08.696796+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_clean_scrub_stamp":"2024-09-18T08:05:08.696796+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:08:01.568014+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":195,"reported_epoch":149,"state":"active+clean","last_fresh":"2024-09-18T08:07:08.232608+0000","last_change":"2024-09-18T08:05:43.668557+0000","last_active":"2024-09-18T08:07:08.232608+0000","last_peered":"2024-09-18T08:07:08.232608+0000","last_clean":"2024-09-18T08:07:08.232608+0000","last_became_active":"2024-09-18T08:02:51.907140+0000","last_became_peered":"2024-09-18T08:02:51.907140+0000","last_unstale":"2024-09-18T08:07:08.232608+0000","last_undegraded":"2024-09-18T08:07:08.232608+0000","last_fullsized":"2024-09-18T08:07:08.232608+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:05:43.668271+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:50.886830+0000","last_clean_scrub_stamp":"2024-09-18T08:05:43.668271+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":184,"reported_epoch":149,"state":"active+clean","last_fresh":"2024-09-18T08:07:08.451937+0000","last_change":"2024-09-18T08:07:08.451937+0000","last_active":"2024-09-18T08:07:08.451937+0000","last_peered":"2024-09-18T08:07:08.451937+0000","last_clean":"2024-09-18T08:07:08.451937+0000","last_became_active":"2024-09-18T08:02:53.906622+0000","last_became_peered":"2024-09-18T08:02:53.906622+0000","last_unstale":"2024-09-18T08:07:08.451937+0000","last_undegraded":"2024-09-18T08:07:08.451937+0000","last_fullsized":"2024-09-18T08:07:08.451937+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:07:08.451865+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_clean_scrub_stamp":"2024-09-18T08:07:08.451865+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:08:20.709312+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":205,"reported_epoch":149,"state":"active+clean","last_fresh":"2024-09-18T08:07:13.244647+0000","last_change":"2024-09-18T08:07:13.244647+0000","last_active":"2024-09-18T08:07:13.244647+0000","last_peered":"2024-09-18T08:07:13.244647+0000","last_clean":"2024-09-18T08:07:13.244647+0000","last_became_active":"2024-09-18T08:06:24.128511+0000","last_became_peered":"2024-09-18T08:06:24.128511+0000","last_unstale":"2024-09-18T08:07:13.244647+0000","last_undegraded":"2024-09-18T08:07:13.244647+0000","last_fullsized":"2024-09-18T08:07:13.244647+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:07:13.244555+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_clean_scrub_stamp":"2024-09-18T08:07:13.244555+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:08:18.290204+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":218,"reported_epoch":149,"state":"active+clean","last_fresh":"2024-09-18T08:07:10.257236+0000","last_change":"2024-09-18T08:07:10.257236+0000","last_active":"2024-09-18T08:07:10.257236+0000","last_peered":"2024-09-18T08:07:10.257236+0000","last_clean":"2024-09-18T08:07:10.257236+0000","last_became_active":"2024-09-18T08:06:24.126194+0000","last_became_peered":"2024-09-18T08:06:24.126194+0000","last_unstale":"2024-09-18T08:07:10.257236+0000","last_undegraded":"2024-09-18T08:07:10.257236+0000","last_fullsized":"2024-09-18T08:07:10.257236+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:07:10.257170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:46.860092+0000","last_clean_scrub_stamp":"2024-09-18T08:07:10.257170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:08:28.333021+0000","scrub_duration":37,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"69'15","reported_seq":208,"reported_epoch":149,"state":"active+clean","last_fresh":"2024-09-18T08:07:08.225952+0000","last_change":"2024-09-18T08:02:49.887111+0000","last_active":"2024-09-18T08:07:08.225952+0000","last_peered":"2024-09-18T08:07:08.225952+0000","last_clean":"2024-09-18T08:07:08.225952+0000","last_became_active":"2024-09-18T08:02:49.886106+0000","last_became_peered":"2024-09-18T08:02:49.886106+0000","last_unstale":"2024-09-18T08:07:08.225952+0000","last_undegraded":"2024-09-18T08:07:08.225952+0000","last_fullsized":"2024-09-18T08:07:08.225952+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_clean_scrub_stamp":"2024-09-18T08:02:48.870718+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:07:55.263331+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":201,"reported_epoch":149,"state":"active+clean","last_fresh":"2024-09-18T08:07:08.214013+0000","last_change":"2024-09-18T08:06:24.146536+0000","last_active":"2024-09-18T08:07:08.214013+0000","last_peered":"2024-09-18T08:07:08.214013+0000","last_clean":"2024-09-18T08:07:08.214013+0000","last_became_active":"2024-09-18T08:06:24.144841+0000","last_became_peered":"2024-09-18T08:06:24.144841+0000","last_unstale":"2024-09-18T08:07:08.214013+0000","last_undegraded":"2024-09-18T08:07:08.214013+0000","last_fullsized":"2024-09-18T08:07:08.214013+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:04:29.435669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:50.886830+0000","last_clean_scrub_stamp":"2024-09-18T08:04:29.435669+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:07:33.513868+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":217,"reported_epoch":149,"state":"active+clean","last_fresh":"2024-09-18T08:07:08.230052+0000","last_change":"2024-09-18T08:06:24.132542+0000","last_active":"2024-09-18T08:07:08.230052+0000","last_peered":"2024-09-18T08:07:08.230052+0000","last_clean":"2024-09-18T08:07:08.230052+0000","last_became_active":"2024-09-18T08:06:24.132206+0000","last_became_peered":"2024-09-18T08:06:24.132206+0000","last_unstale":"2024-09-18T08:07:08.230052+0000","last_undegraded":"2024-09-18T08:07:08.230052+0000","last_fullsized":"2024-09-18T08:07:08.230052+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:06:14.027077+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:46.860092+0000","last_clean_scrub_stamp":"2024-09-18T08:06:14.027077+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:07:24.620825+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"69'9","reported_seq":199,"reported_epoch":149,"state":"active+clean","last_fresh":"2024-09-18T08:07:08.214478+0000","last_change":"2024-09-18T08:02:49.887589+0000","last_active":"2024-09-18T08:07:08.214478+0000","last_peered":"2024-09-18T08:07:08.214478+0000","last_clean":"2024-09-18T08:07:08.214478+0000","last_became_active":"2024-09-18T08:02:49.887161+0000","last_became_peered":"2024-09-18T08:02:49.887161+0000","last_unstale":"2024-09-18T08:07:08.214478+0000","last_undegraded":"2024-09-18T08:07:08.214478+0000","last_fullsized":"2024-09-18T08:07:08.214478+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_clean_scrub_stamp":"2024-09-18T08:02:48.870718+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:07:59.552425+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":195,"reported_epoch":149,"state":"active+clean","last_fresh":"2024-09-18T08:07:08.229594+0000","last_change":"2024-09-18T08:05:33.935844+0000","last_active":"2024-09-18T08:07:08.229594+0000","last_peered":"2024-09-18T08:07:08.229594+0000","last_clean":"2024-09-18T08:07:08.229594+0000","last_became_active":"2024-09-18T08:02:51.907296+0000","last_became_peered":"2024-09-18T08:02:51.907296+0000","last_unstale":"2024-09-18T08:07:08.229594+0000","last_undegraded":"2024-09-18T08:07:08.229594+0000","last_fullsized":"2024-09-18T08:07:08.229594+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:05:33.935798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:50.886830+0000","last_clean_scrub_stamp":"2024-09-18T08:05:33.935798+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:07:55.907350+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":184,"reported_epoch":149,"state":"active+clean","last_fresh":"2024-09-18T08:07:08.225162+0000","last_change":"2024-09-18T08:05:01.314402+0000","last_active":"2024-09-18T08:07:08.225162+0000","last_peered":"2024-09-18T08:07:08.225162+0000","last_clean":"2024-09-18T08:07:08.225162+0000","last_became_active":"2024-09-18T08:02:53.913519+0000","last_became_peered":"2024-09-18T08:02:53.913519+0000","last_unstale":"2024-09-18T08:07:08.225162+0000","last_undegraded":"2024-09-18T08:07:08.225162+0000","last_fullsized":"2024-09-18T08:07:08.225162+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:05:01.314330+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_clean_scrub_stamp":"2024-09-18T08:05:01.314330+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:07:56.780219+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":195,"reported_epoch":149,"state":"active+clean","last_fresh":"2024-09-18T08:07:08.230037+0000","last_change":"2024-09-18T08:04:32.849176+0000","last_active":"2024-09-18T08:07:08.230037+0000","last_peered":"2024-09-18T08:07:08.230037+0000","last_clean":"2024-09-18T08:07:08.230037+0000","last_became_active":"2024-09-18T08:02:47.875042+0000","last_became_peered":"2024-09-18T08:02:47.875042+0000","last_unstale":"2024-09-18T08:07:08.230037+0000","last_undegraded":"2024-09-18T08:07:08.230037+0000","last_fullsized":"2024-09-18T08:07:08.230037+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:04:32.849119+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:46.860092+0000","last_clean_scrub_stamp":"2024-09-18T08:04:32.849119+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:08:06.853060+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1b","version":"69'5","reported_seq":209,"reported_epoch":149,"state":"active+clean","last_fresh":"2024-09-18T08:07:08.214824+0000","last_change":"2024-09-18T08:06:09.466075+0000","last_active":"2024-09-18T08:07:08.214824+0000","last_peered":"2024-09-18T08:07:08.214824+0000","last_clean":"2024-09-18T08:07:08.214824+0000","last_became_active":"2024-09-18T08:02:49.895344+0000","last_became_peered":"2024-09-18T08:02:49.895344+0000","last_unstale":"2024-09-18T08:07:08.214824+0000","last_undegraded":"2024-09-18T08:07:08.214824+0000","last_fullsized":"2024-09-18T08:07:08.214824+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'5","last_scrub_stamp":"2024-09-18T08:06:09.465979+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_clean_scrub_stamp":"2024-09-18T08:06:09.465979+0000","objects_scrubbed":2,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:07:37.799450+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":11,"num_read_kb":7,"num_write":6,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":195,"reported_epoch":149,"state":"active+clean","last_fresh":"2024-09-18T08:07:08.234662+0000","last_change":"2024-09-18T08:05:38.866695+0000","last_active":"2024-09-18T08:07:08.234662+0000","last_peered":"2024-09-18T08:07:08.234662+0000","last_clean":"2024-09-18T08:07:08.234662+0000","last_became_active":"2024-09-18T08:02:51.905776+0000","last_became_peered":"2024-09-18T08:02:51.905776+0000","last_unstale":"2024-09-18T08:07:08.234662+0000","last_undegraded":"2024-09-18T08:07:08.234662+0000","last_fullsized":"2024-09-18T08:07:08.234662+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:05:38.866651+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:50.886830+0000","last_clean_scrub_stamp":"2024-09-18T08:05:38.866651+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:07:53.746353+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":192,"reported_epoch":149,"state":"active+clean","last_fresh":"2024-09-18T08:07:08.229396+0000","last_change":"2024-09-18T08:06:44.518882+0000","last_active":"2024-09-18T08:07:08.229396+0000","last_peered":"2024-09-18T08:07:08.229396+0000","last_clean":"2024-09-18T08:07:08.229396+0000","last_became_active":"2024-09-18T08:02:53.911648+0000","last_became_peered":"2024-09-18T08:02:53.911648+0000","last_unstale":"2024-09-18T08:07:08.229396+0000","last_undegraded":"2024-09-18T08:07:08.229396+0000","last_fullsized":"2024-09-18T08:07:08.229396+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:06:44.518835+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_clean_scrub_stamp":"2024-09-18T08:06:44.518835+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:08:04.920034+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":198,"reported_epoch":149,"state":"active+clean","last_fresh":"2024-09-18T08:07:08.215693+0000","last_change":"2024-09-18T08:06:24.132553+0000","last_active":"2024-09-18T08:07:08.215693+0000","last_peered":"2024-09-18T08:07:08.215693+0000","last_clean":"2024-09-18T08:07:08.215693+0000","last_became_active":"2024-09-18T08:06:24.132228+0000","last_became_peered":"2024-09-18T08:06:24.132228+0000","last_unstale":"2024-09-18T08:07:08.215693+0000","last_undegraded":"2024-09-18T08:07:08.215693+0000","last_fullsized":"2024-09-18T08:07:08.215693+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:05:40.477965+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_clean_scrub_stamp":"2024-09-18T08:05:40.477965+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":203,"reported_epoch":149,"state":"active+clean","last_fresh":"2024-09-18T08:07:08.227038+0000","last_change":"2024-09-18T08:05:27.476211+0000","last_active":"2024-09-18T08:07:08.227038+0000","last_peered":"2024-09-18T08:07:08.227038+0000","last_clean":"2024-09-18T08:07:08.227038+0000","last_became_active":"2024-09-18T08:02:47.875963+0000","last_became_peered":"2024-09-18T08:02:47.875963+0000","last_unstale":"2024-09-18T08:07:08.227038+0000","last_undegraded":"2024-09-18T08:07:08.227038+0000","last_fullsized":"2024-09-18T08:07:08.227038+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:05:27.476144+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:46.860092+0000","last_clean_scrub_stamp":"2024-09-18T08:05:27.476144+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1c","version":"69'15","reported_seq":247,"reported_epoch":149,"state":"active+clean","last_fresh":"2024-09-18T08:07:08.222235+0000","last_change":"2024-09-18T08:05:28.084324+0000","last_active":"2024-09-18T08:07:08.222235+0000","last_peered":"2024-09-18T08:07:08.222235+0000","last_clean":"2024-09-18T08:07:08.222235+0000","last_became_active":"2024-09-18T08:04:18.147907+0000","last_became_peered":"2024-09-18T08:04:18.147907+0000","last_unstale":"2024-09-18T08:07:08.222235+0000","last_undegraded":"2024-09-18T08:07:08.222235+0000","last_fullsized":"2024-09-18T08:07:08.222235+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'15","last_scrub_stamp":"2024-09-18T08:05:28.084220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_clean_scrub_stamp":"2024-09-18T08:05:28.084220+0000","objects_scrubbed":7,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:08:04.766020+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,3],"acting":[2,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":203,"reported_epoch":149,"state":"active+clean","last_fresh":"2024-09-18T08:07:08.743521+0000","last_change":"2024-09-18T08:07:08.743521+0000","last_active":"2024-09-18T08:07:08.743521+0000","last_peered":"2024-09-18T08:07:08.743521+0000","last_clean":"2024-09-18T08:07:08.743521+0000","last_became_active":"2024-09-18T08:02:51.908430+0000","last_became_peered":"2024-09-18T08:02:51.908430+0000","last_unstale":"2024-09-18T08:07:08.743521+0000","last_undegraded":"2024-09-18T08:07:08.743521+0000","last_fullsized":"2024-09-18T08:07:08.743521+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:07:08.743455+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:50.886830+0000","last_clean_scrub_stamp":"2024-09-18T08:07:08.743455+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:08:19.243807+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":198,"reported_epoch":149,"state":"active+clean","last_fresh":"2024-09-18T08:07:08.222086+0000","last_change":"2024-09-18T08:06:24.129031+0000","last_active":"2024-09-18T08:07:08.222086+0000","last_peered":"2024-09-18T08:07:08.222086+0000","last_clean":"2024-09-18T08:07:08.222086+0000","last_became_active":"2024-09-18T08:06:24.128659+0000","last_became_peered":"2024-09-18T08:06:24.128659+0000","last_unstale":"2024-09-18T08:07:08.222086+0000","last_undegraded":"2024-09-18T08:07:08.222086+0000","last_fullsized":"2024-09-18T08:07:08.222086+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:04:29.955822+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_clean_scrub_stamp":"2024-09-18T08:04:29.955822+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:08:05.214492+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":195,"reported_epoch":149,"state":"active+clean","last_fresh":"2024-09-18T08:07:08.214635+0000","last_change":"2024-09-18T08:04:22.420158+0000","last_active":"2024-09-18T08:07:08.214635+0000","last_peered":"2024-09-18T08:07:08.214635+0000","last_clean":"2024-09-18T08:07:08.214635+0000","last_became_active":"2024-09-18T08:02:47.871523+0000","last_became_peered":"2024-09-18T08:02:47.871523+0000","last_unstale":"2024-09-18T08:07:08.214635+0000","last_undegraded":"2024-09-18T08:07:08.214635+0000","last_fullsized":"2024-09-18T08:07:08.214635+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:04:22.420112+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:46.860092+0000","last_clean_scrub_stamp":"2024-09-18T08:04:22.420112+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"69'12","reported_seq":214,"reported_epoch":149,"state":"active+clean","last_fresh":"2024-09-18T08:07:08.221728+0000","last_change":"2024-09-18T08:05:47.068485+0000","last_active":"2024-09-18T08:07:08.221728+0000","last_peered":"2024-09-18T08:07:08.221728+0000","last_clean":"2024-09-18T08:07:08.221728+0000","last_became_active":"2024-09-18T08:02:49.888831+0000","last_became_peered":"2024-09-18T08:02:49.888831+0000","last_unstale":"2024-09-18T08:07:08.221728+0000","last_undegraded":"2024-09-18T08:07:08.221728+0000","last_fullsized":"2024-09-18T08:07:08.221728+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'12","last_scrub_stamp":"2024-09-18T08:05:47.068375+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_clean_scrub_stamp":"2024-09-18T08:05:47.068375+0000","objects_scrubbed":5,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":25,"num_read_kb":16,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":195,"reported_epoch":149,"state":"active+clean","last_fresh":"2024-09-18T08:07:08.215050+0000","last_change":"2024-09-18T08:06:12.434234+0000","last_active":"2024-09-18T08:07:08.215050+0000","last_peered":"2024-09-18T08:07:08.215050+0000","last_clean":"2024-09-18T08:07:08.215050+0000","last_became_active":"2024-09-18T08:02:51.910316+0000","last_became_peered":"2024-09-18T08:02:51.910316+0000","last_unstale":"2024-09-18T08:07:08.215050+0000","last_undegraded":"2024-09-18T08:07:08.215050+0000","last_fullsized":"2024-09-18T08:07:08.215050+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:06:12.434179+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:50.886830+0000","last_clean_scrub_stamp":"2024-09-18T08:06:12.434179+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:07:35.662786+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1c","version":"69'1","reported_seq":35,"reported_epoch":149,"state":"active+clean","last_fresh":"2024-09-18T08:07:08.229715+0000","last_change":"2024-09-18T08:06:46.142143+0000","last_active":"2024-09-18T08:07:08.229715+0000","last_peered":"2024-09-18T08:07:08.229715+0000","last_clean":"2024-09-18T08:07:08.229715+0000","last_became_active":"2024-09-18T08:06:46.141661+0000","last_became_peered":"2024-09-18T08:06:46.141661+0000","last_unstale":"2024-09-18T08:07:08.229715+0000","last_undegraded":"2024-09-18T08:07:08.229715+0000","last_fullsized":"2024-09-18T08:07:08.229715+0000","mapping_epoch":143,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":144,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'1","last_scrub_stamp":"2024-09-18T08:05:36.776369+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_clean_scrub_stamp":"2024-09-18T08:05:36.776369+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":403,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,2],"acting":[5,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"62'1","reported_seq":196,"reported_epoch":149,"state":"active+clean","last_fresh":"2024-09-18T08:07:08.232749+0000","last_change":"2024-09-18T08:05:06.819830+0000","last_active":"2024-09-18T08:07:08.232749+0000","last_peered":"2024-09-18T08:07:08.232749+0000","last_clean":"2024-09-18T08:07:08.232749+0000","last_became_active":"2024-09-18T08:02:47.880998+0000","last_became_peered":"2024-09-18T08:02:47.880998+0000","last_unstale":"2024-09-18T08:07:08.232749+0000","last_undegraded":"2024-09-18T08:07:08.232749+0000","last_fullsized":"2024-09-18T08:07:08.232749+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"62'1","last_scrub_stamp":"2024-09-18T08:05:06.819539+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:46.860092+0000","last_clean_scrub_stamp":"2024-09-18T08:05:06.819539+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:07:39.507165+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1e","version":"69'10","reported_seq":199,"reported_epoch":149,"state":"active+clean","last_fresh":"2024-09-18T08:07:08.233978+0000","last_change":"2024-09-18T08:02:49.892755+0000","last_active":"2024-09-18T08:07:08.233978+0000","last_peered":"2024-09-18T08:07:08.233978+0000","last_clean":"2024-09-18T08:07:08.233978+0000","last_became_active":"2024-09-18T08:02:49.892024+0000","last_became_peered":"2024-09-18T08:02:49.892024+0000","last_unstale":"2024-09-18T08:07:08.233978+0000","last_undegraded":"2024-09-18T08:07:08.233978+0000","last_fullsized":"2024-09-18T08:07:08.233978+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_clean_scrub_stamp":"2024-09-18T08:02:48.870718+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:08:01.836462+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1f","version":"138'11","reported_seq":199,"reported_epoch":149,"state":"active+clean","last_fresh":"2024-09-18T08:07:08.216339+0000","last_change":"2024-09-18T08:06:24.180060+0000","last_active":"2024-09-18T08:07:08.216339+0000","last_peered":"2024-09-18T08:07:08.216339+0000","last_clean":"2024-09-18T08:07:08.216339+0000","last_became_active":"2024-09-18T08:06:24.145360+0000","last_became_peered":"2024-09-18T08:06:24.145360+0000","last_unstale":"2024-09-18T08:07:08.216339+0000","last_undegraded":"2024-09-18T08:07:08.216339+0000","last_fullsized":"2024-09-18T08:07:08.216339+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"102'9","last_scrub_stamp":"2024-09-18T08:05:44.463780+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:50.886830+0000","last_clean_scrub_stamp":"2024-09-18T08:05:44.463780+0000","objects_scrubbed":1,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,7],"acting":[4,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.c","version":"149'13436","reported_seq":15478,"reported_epoch":149,"state":"active+clean","last_fresh":"2024-09-18T08:07:13.516623+0000","last_change":"2024-09-18T08:06:25.149370+0000","last_active":"2024-09-18T08:07:13.516623+0000","last_peered":"2024-09-18T08:07:13.516623+0000","last_clean":"2024-09-18T08:07:13.516623+0000","last_became_active":"2024-09-18T08:06:25.123363+0000","last_became_peered":"2024-09-18T08:06:25.123363+0000","last_unstale":"2024-09-18T08:07:13.516623+0000","last_undegraded":"2024-09-18T08:07:13.516623+0000","last_fullsized":"2024-09-18T08:07:13.516623+0000","mapping_epoch":137,"log_start":"127'8818","ondisk_log_start":"127'8818","created":83,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:03:56.449957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:03:56.449957+0000","last_clean_scrub_stamp":"2024-09-18T08:03:56.449957+0000","objects_scrubbed":0,"log_size":4618,"log_dups_size":0,"ondisk_log_size":4618,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:07:33.267946+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":103153664,"num_objects":1574,"num_object_clones":0,"num_object_copies":4722,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1574,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13436,"num_write_kb":480320,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":940,"num_bytes_recovered":61603840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,7],"acting":[3,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.f","version":"69'15","reported_seq":216,"reported_epoch":149,"state":"active+clean","last_fresh":"2024-09-18T08:07:08.233366+0000","last_change":"2024-09-18T08:04:22.741342+0000","last_active":"2024-09-18T08:07:08.233366+0000","last_peered":"2024-09-18T08:07:08.233366+0000","last_clean":"2024-09-18T08:07:08.233366+0000","last_became_active":"2024-09-18T08:02:49.899083+0000","last_became_peered":"2024-09-18T08:02:49.899083+0000","last_unstale":"2024-09-18T08:07:08.233366+0000","last_undegraded":"2024-09-18T08:07:08.233366+0000","last_fullsized":"2024-09-18T08:07:08.233366+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'15","last_scrub_stamp":"2024-09-18T08:04:22.741261+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_clean_scrub_stamp":"2024-09-18T08:04:22.741261+0000","objects_scrubbed":7,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:07:56.574487+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":196,"reported_epoch":149,"state":"active+clean","last_fresh":"2024-09-18T08:07:08.224068+0000","last_change":"2024-09-18T08:04:08.982535+0000","last_active":"2024-09-18T08:07:08.224068+0000","last_peered":"2024-09-18T08:07:08.224068+0000","last_clean":"2024-09-18T08:07:08.224068+0000","last_became_active":"2024-09-18T08:02:47.879328+0000","last_became_peered":"2024-09-18T08:02:47.879328+0000","last_unstale":"2024-09-18T08:07:08.224068+0000","last_undegraded":"2024-09-18T08:07:08.224068+0000","last_fullsized":"2024-09-18T08:07:08.224068+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:04:08.982469+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:46.860092+0000","last_clean_scrub_stamp":"2024-09-18T08:04:08.982469+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:07:59.169101+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.e","version":"69'8","reported_seq":257,"reported_epoch":149,"state":"active+clean","last_fresh":"2024-09-18T08:07:08.215814+0000","last_change":"2024-09-18T08:06:19.459452+0000","last_active":"2024-09-18T08:07:08.215814+0000","last_peered":"2024-09-18T08:07:08.215814+0000","last_clean":"2024-09-18T08:07:08.215814+0000","last_became_active":"2024-09-18T08:02:51.903369+0000","last_became_peered":"2024-09-18T08:02:51.903369+0000","last_unstale":"2024-09-18T08:07:08.215814+0000","last_undegraded":"2024-09-18T08:07:08.215814+0000","last_fullsized":"2024-09-18T08:07:08.215814+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'8","last_scrub_stamp":"2024-09-18T08:06:19.459375+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:50.886830+0000","last_clean_scrub_stamp":"2024-09-18T08:06:19.459375+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:07:30.186221+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":192,"reported_epoch":149,"state":"active+clean","last_fresh":"2024-09-18T08:07:08.229706+0000","last_change":"2024-09-18T08:05:34.905147+0000","last_active":"2024-09-18T08:07:08.229706+0000","last_peered":"2024-09-18T08:07:08.229706+0000","last_clean":"2024-09-18T08:07:08.229706+0000","last_became_active":"2024-09-18T08:02:53.909834+0000","last_became_peered":"2024-09-18T08:02:53.909834+0000","last_unstale":"2024-09-18T08:07:08.229706+0000","last_undegraded":"2024-09-18T08:07:08.229706+0000","last_fullsized":"2024-09-18T08:07:08.229706+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:05:34.905083+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_clean_scrub_stamp":"2024-09-18T08:05:34.905083+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.3","version":"149'13161","reported_seq":13305,"reported_epoch":149,"state":"active+clean","last_fresh":"2024-09-18T08:07:13.446335+0000","last_change":"2024-09-18T08:03:57.462489+0000","last_active":"2024-09-18T08:07:13.446335+0000","last_peered":"2024-09-18T08:07:13.446335+0000","last_clean":"2024-09-18T08:07:13.446335+0000","last_became_active":"2024-09-18T08:03:57.462210+0000","last_became_peered":"2024-09-18T08:03:57.462210+0000","last_unstale":"2024-09-18T08:07:13.446335+0000","last_undegraded":"2024-09-18T08:07:13.446335+0000","last_fullsized":"2024-09-18T08:07:13.446335+0000","mapping_epoch":83,"log_start":"126'8522","ondisk_log_start":"126'8522","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:03:56.449957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:03:56.449957+0000","last_clean_scrub_stamp":"2024-09-18T08:03:56.449957+0000","objects_scrubbed":0,"log_size":4639,"log_dups_size":0,"ondisk_log_size":4639,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:07:30.236230+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":99155968,"num_objects":1513,"num_object_clones":0,"num_object_copies":4539,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1513,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13161,"num_write_kb":469568,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,4],"acting":[3,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.0","version":"69'18","reported_seq":232,"reported_epoch":149,"state":"active+clean","last_fresh":"2024-09-18T08:07:08.223755+0000","last_change":"2024-09-18T08:06:44.301130+0000","last_active":"2024-09-18T08:07:08.223755+0000","last_peered":"2024-09-18T08:07:08.223755+0000","last_clean":"2024-09-18T08:07:08.223755+0000","last_became_active":"2024-09-18T08:02:49.886818+0000","last_became_peered":"2024-09-18T08:02:49.886818+0000","last_unstale":"2024-09-18T08:07:08.223755+0000","last_undegraded":"2024-09-18T08:07:08.223755+0000","last_fullsized":"2024-09-18T08:07:08.223755+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'18","last_scrub_stamp":"2024-09-18T08:06:44.300772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_clean_scrub_stamp":"2024-09-18T08:06:44.300772+0000","objects_scrubbed":8,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:07:52.014378+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":34,"num_read_kb":22,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":204,"reported_epoch":149,"state":"active+clean","last_fresh":"2024-09-18T08:07:08.230705+0000","last_change":"2024-09-18T08:05:35.100407+0000","last_active":"2024-09-18T08:07:08.230705+0000","last_peered":"2024-09-18T08:07:08.230705+0000","last_clean":"2024-09-18T08:07:08.230705+0000","last_became_active":"2024-09-18T08:02:47.880225+0000","last_became_peered":"2024-09-18T08:02:47.880225+0000","last_unstale":"2024-09-18T08:07:08.230705+0000","last_undegraded":"2024-09-18T08:07:08.230705+0000","last_fullsized":"2024-09-18T08:07:08.230705+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:05:35.100332+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:46.860092+0000","last_clean_scrub_stamp":"2024-09-18T08:05:35.100332+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":196,"reported_epoch":149,"state":"active+clean","last_fresh":"2024-09-18T08:07:08.214532+0000","last_change":"2024-09-18T08:06:10.468172+0000","last_active":"2024-09-18T08:07:08.214532+0000","last_peered":"2024-09-18T08:07:08.214532+0000","last_clean":"2024-09-18T08:07:08.214532+0000","last_became_active":"2024-09-18T08:02:51.899594+0000","last_became_peered":"2024-09-18T08:02:51.899594+0000","last_unstale":"2024-09-18T08:07:08.214532+0000","last_undegraded":"2024-09-18T08:07:08.214532+0000","last_fullsized":"2024-09-18T08:07:08.214532+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:06:10.468119+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:50.886830+0000","last_clean_scrub_stamp":"2024-09-18T08:06:10.468119+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:07:23.534443+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":185,"reported_epoch":149,"state":"active+clean","last_fresh":"2024-09-18T08:07:08.214395+0000","last_change":"2024-09-18T08:04:16.427470+0000","last_active":"2024-09-18T08:07:08.214395+0000","last_peered":"2024-09-18T08:07:08.214395+0000","last_clean":"2024-09-18T08:07:08.214395+0000","last_became_active":"2024-09-18T08:02:53.903957+0000","last_became_peered":"2024-09-18T08:02:53.903957+0000","last_unstale":"2024-09-18T08:07:08.214395+0000","last_undegraded":"2024-09-18T08:07:08.214395+0000","last_fullsized":"2024-09-18T08:07:08.214395+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:04:16.427390+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_clean_scrub_stamp":"2024-09-18T08:04:16.427390+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:07:26.444141+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.2","version":"149'13202","reported_seq":13346,"reported_epoch":149,"state":"active+clean","last_fresh":"2024-09-18T08:07:13.510208+0000","last_change":"2024-09-18T08:03:57.463750+0000","last_active":"2024-09-18T08:07:13.510208+0000","last_peered":"2024-09-18T08:07:13.510208+0000","last_clean":"2024-09-18T08:07:13.510208+0000","last_became_active":"2024-09-18T08:03:57.463459+0000","last_became_peered":"2024-09-18T08:03:57.463459+0000","last_unstale":"2024-09-18T08:07:13.510208+0000","last_undegraded":"2024-09-18T08:07:13.510208+0000","last_fullsized":"2024-09-18T08:07:13.510208+0000","mapping_epoch":83,"log_start":"126'8543","ondisk_log_start":"126'8543","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:03:56.449957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:03:56.449957+0000","last_clean_scrub_stamp":"2024-09-18T08:03:56.449957+0000","objects_scrubbed":0,"log_size":4659,"log_dups_size":0,"ondisk_log_size":4659,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:08:02.118953+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":106168320,"num_objects":1620,"num_object_clones":0,"num_object_copies":4860,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1620,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13202,"num_write_kb":474304,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"69'14","reported_seq":267,"reported_epoch":149,"state":"active+clean","last_fresh":"2024-09-18T08:07:08.214607+0000","last_change":"2024-09-18T08:06:24.184801+0000","last_active":"2024-09-18T08:07:08.214607+0000","last_peered":"2024-09-18T08:07:08.214607+0000","last_clean":"2024-09-18T08:07:08.214607+0000","last_became_active":"2024-09-18T08:06:24.133949+0000","last_became_peered":"2024-09-18T08:06:24.133949+0000","last_unstale":"2024-09-18T08:07:08.214607+0000","last_undegraded":"2024-09-18T08:07:08.214607+0000","last_fullsized":"2024-09-18T08:07:08.214607+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'14","last_scrub_stamp":"2024-09-18T08:04:56.545262+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_clean_scrub_stamp":"2024-09-18T08:04:56.545262+0000","objects_scrubbed":7,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:07:37.514965+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":21,"num_read_kb":14,"num_write":14,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"62'2","reported_seq":211,"reported_epoch":149,"state":"active+clean","last_fresh":"2024-09-18T08:07:08.234286+0000","last_change":"2024-09-18T08:06:43.607646+0000","last_active":"2024-09-18T08:07:08.234286+0000","last_peered":"2024-09-18T08:07:08.234286+0000","last_clean":"2024-09-18T08:07:08.234286+0000","last_became_active":"2024-09-18T08:02:47.872857+0000","last_became_peered":"2024-09-18T08:02:47.872857+0000","last_unstale":"2024-09-18T08:07:08.234286+0000","last_undegraded":"2024-09-18T08:07:08.234286+0000","last_fullsized":"2024-09-18T08:07:08.234286+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"62'2","last_scrub_stamp":"2024-09-18T08:06:43.607152+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:46.860092+0000","last_clean_scrub_stamp":"2024-09-18T08:06:43.607152+0000","objects_scrubbed":2,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:07:53.027436+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1039,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.0","version":"69'8","reported_seq":250,"reported_epoch":149,"state":"active+clean","last_fresh":"2024-09-18T08:07:08.223524+0000","last_change":"2024-09-18T08:04:23.863573+0000","last_active":"2024-09-18T08:07:08.223524+0000","last_peered":"2024-09-18T08:07:08.223524+0000","last_clean":"2024-09-18T08:07:08.223524+0000","last_became_active":"2024-09-18T08:02:51.909680+0000","last_became_peered":"2024-09-18T08:02:51.909680+0000","last_unstale":"2024-09-18T08:07:08.223524+0000","last_undegraded":"2024-09-18T08:07:08.223524+0000","last_fullsized":"2024-09-18T08:07:08.223524+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'8","last_scrub_stamp":"2024-09-18T08:04:23.863330+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:50.886830+0000","last_clean_scrub_stamp":"2024-09-18T08:04:23.863330+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:08:04.168099+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":207,"reported_epoch":149,"state":"active+clean","last_fresh":"2024-09-18T08:07:08.234855+0000","last_change":"2024-09-18T08:06:44.781751+0000","last_active":"2024-09-18T08:07:08.234855+0000","last_peered":"2024-09-18T08:07:08.234855+0000","last_clean":"2024-09-18T08:07:08.234855+0000","last_became_active":"2024-09-18T08:06:24.140303+0000","last_became_peered":"2024-09-18T08:06:24.140303+0000","last_unstale":"2024-09-18T08:07:08.234855+0000","last_undegraded":"2024-09-18T08:07:08.234855+0000","last_fullsized":"2024-09-18T08:07:08.234855+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:06:44.781685+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_clean_scrub_stamp":"2024-09-18T08:06:44.781685+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:07:48.361147+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.1","version":"149'13284","reported_seq":13440,"reported_epoch":149,"state":"active+clean","last_fresh":"2024-09-18T08:07:13.056817+0000","last_change":"2024-09-18T08:06:24.166578+0000","last_active":"2024-09-18T08:07:13.056817+0000","last_peered":"2024-09-18T08:07:13.056817+0000","last_clean":"2024-09-18T08:07:13.056817+0000","last_became_active":"2024-09-18T08:06:24.134315+0000","last_became_peered":"2024-09-18T08:06:24.134315+0000","last_unstale":"2024-09-18T08:07:13.056817+0000","last_undegraded":"2024-09-18T08:07:13.056817+0000","last_fullsized":"2024-09-18T08:07:13.056817+0000","mapping_epoch":136,"log_start":"126'8457","ondisk_log_start":"126'8457","created":83,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:03:56.449957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:03:56.449957+0000","last_clean_scrub_stamp":"2024-09-18T08:03:56.449957+0000","objects_scrubbed":0,"log_size":4827,"log_dups_size":0,"ondisk_log_size":4827,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:07:27.452504+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":102498304,"num_objects":1564,"num_object_clones":0,"num_object_copies":4692,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1564,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13284,"num_write_kb":475136,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.2","version":"69'10","reported_seq":215,"reported_epoch":149,"state":"active+clean","last_fresh":"2024-09-18T08:07:08.233745+0000","last_change":"2024-09-18T08:05:39.769686+0000","last_active":"2024-09-18T08:07:08.233745+0000","last_peered":"2024-09-18T08:07:08.233745+0000","last_clean":"2024-09-18T08:07:08.233745+0000","last_became_active":"2024-09-18T08:02:49.895265+0000","last_became_peered":"2024-09-18T08:02:49.895265+0000","last_unstale":"2024-09-18T08:07:08.233745+0000","last_undegraded":"2024-09-18T08:07:08.233745+0000","last_fullsized":"2024-09-18T08:07:08.233745+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'10","last_scrub_stamp":"2024-09-18T08:05:39.768393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_clean_scrub_stamp":"2024-09-18T08:05:39.768393+0000","objects_scrubbed":5,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":17,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":196,"reported_epoch":149,"state":"active+clean","last_fresh":"2024-09-18T08:07:08.230139+0000","last_change":"2024-09-18T08:03:48.009689+0000","last_active":"2024-09-18T08:07:08.230139+0000","last_peered":"2024-09-18T08:07:08.230139+0000","last_clean":"2024-09-18T08:07:08.230139+0000","last_became_active":"2024-09-18T08:02:47.874727+0000","last_became_peered":"2024-09-18T08:02:47.874727+0000","last_unstale":"2024-09-18T08:07:08.230139+0000","last_undegraded":"2024-09-18T08:07:08.230139+0000","last_fullsized":"2024-09-18T08:07:08.230139+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:03:48.009611+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:46.860092+0000","last_clean_scrub_stamp":"2024-09-18T08:03:48.009611+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:07:24.213853+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.3","version":"138'10","reported_seq":273,"reported_epoch":149,"state":"active+clean","last_fresh":"2024-09-18T08:07:08.226625+0000","last_change":"2024-09-18T08:06:24.157627+0000","last_active":"2024-09-18T08:07:08.226625+0000","last_peered":"2024-09-18T08:07:08.226625+0000","last_clean":"2024-09-18T08:07:08.226625+0000","last_became_active":"2024-09-18T08:06:24.134194+0000","last_became_peered":"2024-09-18T08:06:24.134194+0000","last_unstale":"2024-09-18T08:07:08.226625+0000","last_undegraded":"2024-09-18T08:07:08.226625+0000","last_fullsized":"2024-09-18T08:07:08.226625+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"130'9","last_scrub_stamp":"2024-09-18T08:06:13.462610+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:50.886830+0000","last_clean_scrub_stamp":"2024-09-18T08:06:13.462610+0000","objects_scrubbed":1,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:07:27.049204+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":193,"reported_epoch":149,"state":"active+clean","last_fresh":"2024-09-18T08:07:08.225313+0000","last_change":"2024-09-18T08:05:43.440713+0000","last_active":"2024-09-18T08:07:08.225313+0000","last_peered":"2024-09-18T08:07:08.225313+0000","last_clean":"2024-09-18T08:07:08.225313+0000","last_became_active":"2024-09-18T08:02:53.909583+0000","last_became_peered":"2024-09-18T08:02:53.909583+0000","last_unstale":"2024-09-18T08:07:08.225313+0000","last_undegraded":"2024-09-18T08:07:08.225313+0000","last_fullsized":"2024-09-18T08:07:08.225313+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:05:43.440642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_clean_scrub_stamp":"2024-09-18T08:05:43.440642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":35,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.0","version":"149'13058","reported_seq":13263,"reported_epoch":149,"state":"active+clean","last_fresh":"2024-09-18T08:07:13.108753+0000","last_change":"2024-09-18T08:05:01.419397+0000","last_active":"2024-09-18T08:07:13.108753+0000","last_peered":"2024-09-18T08:07:13.108753+0000","last_clean":"2024-09-18T08:07:13.108753+0000","last_became_active":"2024-09-18T08:03:57.468985+0000","last_became_peered":"2024-09-18T08:03:57.468985+0000","last_unstale":"2024-09-18T08:07:13.108753+0000","last_undegraded":"2024-09-18T08:07:13.108753+0000","last_fullsized":"2024-09-18T08:07:13.108753+0000","mapping_epoch":83,"log_start":"119'7232","ondisk_log_start":"119'7232","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"107'5286","last_scrub_stamp":"2024-09-18T08:05:01.419332+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:03:56.449957+0000","last_clean_scrub_stamp":"2024-09-18T08:05:01.419332+0000","objects_scrubbed":5263,"log_size":5826,"log_dups_size":0,"ondisk_log_size":5826,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:07:27.313345+0000","scrub_duration":1313,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":105381888,"num_objects":1608,"num_object_clones":0,"num_object_copies":4824,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1608,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13058,"num_write_kb":469312,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.3","version":"69'19","reported_seq":228,"reported_epoch":149,"state":"active+clean","last_fresh":"2024-09-18T08:07:08.226791+0000","last_change":"2024-09-18T08:04:07.231406+0000","last_active":"2024-09-18T08:07:08.226791+0000","last_peered":"2024-09-18T08:07:08.226791+0000","last_clean":"2024-09-18T08:07:08.226791+0000","last_became_active":"2024-09-18T08:02:49.890318+0000","last_became_peered":"2024-09-18T08:02:49.890318+0000","last_unstale":"2024-09-18T08:07:08.226791+0000","last_undegraded":"2024-09-18T08:07:08.226791+0000","last_fullsized":"2024-09-18T08:07:08.226791+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'19","last_scrub_stamp":"2024-09-18T08:04:07.229846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_clean_scrub_stamp":"2024-09-18T08:04:07.229846+0000","objects_scrubbed":8,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:07:33.520229+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":39,"num_read_kb":25,"num_write":22,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.4","version":"62'1","reported_seq":202,"reported_epoch":149,"state":"active+clean","last_fresh":"2024-09-18T08:07:08.240506+0000","last_change":"2024-09-18T08:04:28.827545+0000","last_active":"2024-09-18T08:07:08.240506+0000","last_peered":"2024-09-18T08:07:08.240506+0000","last_clean":"2024-09-18T08:07:08.240506+0000","last_became_active":"2024-09-18T08:02:47.874764+0000","last_became_peered":"2024-09-18T08:02:47.874764+0000","last_unstale":"2024-09-18T08:07:08.240506+0000","last_undegraded":"2024-09-18T08:07:08.240506+0000","last_fullsized":"2024-09-18T08:07:08.240506+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"62'1","last_scrub_stamp":"2024-09-18T08:04:28.827464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:46.860092+0000","last_clean_scrub_stamp":"2024-09-18T08:04:28.827464+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:07:54.534873+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":436,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":122,"reported_epoch":149,"state":"active+clean","last_fresh":"2024-09-18T08:07:08.226184+0000","last_change":"2024-09-18T08:06:24.133954+0000","last_active":"2024-09-18T08:07:08.226184+0000","last_peered":"2024-09-18T08:07:08.226184+0000","last_clean":"2024-09-18T08:07:08.226184+0000","last_became_active":"2024-09-18T08:06:24.133262+0000","last_became_peered":"2024-09-18T08:06:24.133262+0000","last_unstale":"2024-09-18T08:07:08.226184+0000","last_undegraded":"2024-09-18T08:07:08.226184+0000","last_fullsized":"2024-09-18T08:07:08.226184+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:04:13.377561+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:50.886830+0000","last_clean_scrub_stamp":"2024-09-18T08:04:13.377561+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:07:53.735285+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":206,"reported_epoch":149,"state":"active+clean","last_fresh":"2024-09-18T08:07:08.232558+0000","last_change":"2024-09-18T08:06:24.139017+0000","last_active":"2024-09-18T08:07:08.232558+0000","last_peered":"2024-09-18T08:07:08.232558+0000","last_clean":"2024-09-18T08:07:08.232558+0000","last_became_active":"2024-09-18T08:06:24.138652+0000","last_became_peered":"2024-09-18T08:06:24.138652+0000","last_unstale":"2024-09-18T08:07:08.232558+0000","last_undegraded":"2024-09-18T08:07:08.232558+0000","last_fullsized":"2024-09-18T08:07:08.232558+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:06:10.507191+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_clean_scrub_stamp":"2024-09-18T08:06:10.507191+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:07:38.239809+0000","scrub_duration":25,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.7","version":"149'13273","reported_seq":15065,"reported_epoch":149,"state":"active+clean","last_fresh":"2024-09-18T08:07:13.568583+0000","last_change":"2024-09-18T08:06:25.150134+0000","last_active":"2024-09-18T08:07:13.568583+0000","last_peered":"2024-09-18T08:07:13.568583+0000","last_clean":"2024-09-18T08:07:13.568583+0000","last_became_active":"2024-09-18T08:06:25.123728+0000","last_became_peered":"2024-09-18T08:06:25.123728+0000","last_unstale":"2024-09-18T08:07:13.568583+0000","last_undegraded":"2024-09-18T08:07:13.568583+0000","last_fullsized":"2024-09-18T08:07:13.568583+0000","mapping_epoch":137,"log_start":"120'7666","ondisk_log_start":"120'7666","created":83,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:03:56.449957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:03:56.449957+0000","last_clean_scrub_stamp":"2024-09-18T08:03:56.449957+0000","objects_scrubbed":4174,"log_size":5607,"log_dups_size":0,"ondisk_log_size":5607,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:07:31.458495+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":105054208,"num_objects":1603,"num_object_clones":0,"num_object_copies":4809,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1603,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13273,"num_write_kb":476032,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":787,"num_bytes_recovered":51511296,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,3],"acting":[0,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.4","version":"69'28","reported_seq":294,"reported_epoch":149,"state":"active+clean","last_fresh":"2024-09-18T08:07:08.240405+0000","last_change":"2024-09-18T08:04:12.737618+0000","last_active":"2024-09-18T08:07:08.240405+0000","last_peered":"2024-09-18T08:07:08.240405+0000","last_clean":"2024-09-18T08:07:08.240405+0000","last_became_active":"2024-09-18T08:02:49.897221+0000","last_became_peered":"2024-09-18T08:02:49.897221+0000","last_unstale":"2024-09-18T08:07:08.240405+0000","last_undegraded":"2024-09-18T08:07:08.240405+0000","last_fullsized":"2024-09-18T08:07:08.240405+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'28","last_scrub_stamp":"2024-09-18T08:04:12.737528+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_clean_scrub_stamp":"2024-09-18T08:04:12.737528+0000","objects_scrubbed":10,"log_size":28,"log_dups_size":0,"ondisk_log_size":28,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:07:53.542712+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":48,"num_read_kb":33,"num_write":26,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":210,"reported_epoch":149,"state":"active+clean","last_fresh":"2024-09-18T08:07:08.216299+0000","last_change":"2024-09-18T08:06:24.143321+0000","last_active":"2024-09-18T08:07:08.216299+0000","last_peered":"2024-09-18T08:07:08.216299+0000","last_clean":"2024-09-18T08:07:08.216299+0000","last_became_active":"2024-09-18T08:06:24.143019+0000","last_became_peered":"2024-09-18T08:06:24.143019+0000","last_unstale":"2024-09-18T08:07:08.216299+0000","last_undegraded":"2024-09-18T08:07:08.216299+0000","last_fullsized":"2024-09-18T08:07:08.216299+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:05:39.490141+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:46.860092+0000","last_clean_scrub_stamp":"2024-09-18T08:05:39.490141+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"64'2","reported_seq":236,"reported_epoch":149,"state":"active+clean","last_fresh":"2024-09-18T08:07:08.229712+0000","last_change":"2024-09-18T08:06:24.165227+0000","last_active":"2024-09-18T08:07:08.229712+0000","last_peered":"2024-09-18T08:07:08.229712+0000","last_clean":"2024-09-18T08:07:08.229712+0000","last_became_active":"2024-09-18T08:06:24.136080+0000","last_became_peered":"2024-09-18T08:06:24.136080+0000","last_unstale":"2024-09-18T08:07:08.229712+0000","last_undegraded":"2024-09-18T08:07:08.229712+0000","last_fullsized":"2024-09-18T08:07:08.229712+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"64'2","last_scrub_stamp":"2024-09-18T08:05:44.802942+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:45.855979+0000","last_clean_scrub_stamp":"2024-09-18T08:05:44.802942+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:07:59.917939+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.001348349,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":38,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":180,"reported_epoch":149,"state":"active+clean","last_fresh":"2024-09-18T08:07:08.240962+0000","last_change":"2024-09-18T08:02:51.905694+0000","last_active":"2024-09-18T08:07:08.240962+0000","last_peered":"2024-09-18T08:07:08.240962+0000","last_clean":"2024-09-18T08:07:08.240962+0000","last_became_active":"2024-09-18T08:02:51.905357+0000","last_became_peered":"2024-09-18T08:02:51.905357+0000","last_unstale":"2024-09-18T08:07:08.240962+0000","last_undegraded":"2024-09-18T08:07:08.240962+0000","last_fullsized":"2024-09-18T08:07:08.240962+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:02:50.886830+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:50.886830+0000","last_clean_scrub_stamp":"2024-09-18T08:02:50.886830+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:07:52.727169+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":193,"reported_epoch":149,"state":"active+clean","last_fresh":"2024-09-18T08:07:08.245209+0000","last_change":"2024-09-18T08:07:08.245134+0000","last_active":"2024-09-18T08:07:08.245209+0000","last_peered":"2024-09-18T08:07:08.245209+0000","last_clean":"2024-09-18T08:07:08.245209+0000","last_became_active":"2024-09-18T08:02:53.911036+0000","last_became_peered":"2024-09-18T08:02:53.911036+0000","last_unstale":"2024-09-18T08:07:08.245209+0000","last_undegraded":"2024-09-18T08:07:08.245209+0000","last_fullsized":"2024-09-18T08:07:08.245209+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:07:08.245063+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_clean_scrub_stamp":"2024-09-18T08:07:08.245063+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:08:34.264472+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.4","version":"149'13288","reported_seq":13471,"reported_epoch":149,"state":"active+clean","last_fresh":"2024-09-18T08:07:13.553401+0000","last_change":"2024-09-18T08:06:50.953755+0000","last_active":"2024-09-18T08:07:13.553401+0000","last_peered":"2024-09-18T08:07:13.553401+0000","last_clean":"2024-09-18T08:07:13.553401+0000","last_became_active":"2024-09-18T08:03:57.466232+0000","last_became_peered":"2024-09-18T08:03:57.466232+0000","last_unstale":"2024-09-18T08:07:13.553401+0000","last_undegraded":"2024-09-18T08:07:13.553401+0000","last_fullsized":"2024-09-18T08:07:13.553401+0000","mapping_epoch":83,"log_start":"120'7685","ondisk_log_start":"120'7685","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"144'11890","last_scrub_stamp":"2024-09-18T08:06:50.951226+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:03:56.449957+0000","last_clean_scrub_stamp":"2024-09-18T08:06:50.951226+0000","objects_scrubbed":3015,"log_size":5603,"log_dups_size":0,"ondisk_log_size":5603,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:07:55.636387+0000","scrub_duration":1266,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":103284736,"num_objects":1576,"num_object_clones":0,"num_object_copies":4728,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1576,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13288,"num_write_kb":475648,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,1],"acting":[0,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.7","version":"69'13","reported_seq":137,"reported_epoch":149,"state":"active+clean","last_fresh":"2024-09-18T08:07:08.222787+0000","last_change":"2024-09-18T08:04:18.164662+0000","last_active":"2024-09-18T08:07:08.222787+0000","last_peered":"2024-09-18T08:07:08.222787+0000","last_clean":"2024-09-18T08:07:08.222787+0000","last_became_active":"2024-09-18T08:04:18.144925+0000","last_became_peered":"2024-09-18T08:04:18.144925+0000","last_unstale":"2024-09-18T08:07:08.222787+0000","last_undegraded":"2024-09-18T08:07:08.222787+0000","last_fullsized":"2024-09-18T08:07:08.222787+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'13","last_scrub_stamp":"2024-09-18T08:03:50.700818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_clean_scrub_stamp":"2024-09-18T08:03:50.700818+0000","objects_scrubbed":0,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:07:29.027985+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":29,"num_read_kb":18,"num_write":16,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,0],"acting":[2,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":210,"reported_epoch":149,"state":"active+clean","last_fresh":"2024-09-18T08:07:08.233473+0000","last_change":"2024-09-18T08:06:24.130858+0000","last_active":"2024-09-18T08:07:08.233473+0000","last_peered":"2024-09-18T08:07:08.233473+0000","last_clean":"2024-09-18T08:07:08.233473+0000","last_became_active":"2024-09-18T08:06:24.126334+0000","last_became_peered":"2024-09-18T08:06:24.126334+0000","last_unstale":"2024-09-18T08:07:08.233473+0000","last_undegraded":"2024-09-18T08:07:08.233473+0000","last_fullsized":"2024-09-18T08:07:08.233473+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:03:49.704004+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:46.860092+0000","last_clean_scrub_stamp":"2024-09-18T08:03:49.704004+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:07:31.622951+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"62'1","reported_seq":205,"reported_epoch":149,"state":"active+clean","last_fresh":"2024-09-18T08:07:08.221596+0000","last_change":"2024-09-18T08:04:09.732724+0000","last_active":"2024-09-18T08:07:08.221596+0000","last_peered":"2024-09-18T08:07:08.221596+0000","last_clean":"2024-09-18T08:07:08.221596+0000","last_became_active":"2024-09-18T08:02:46.868175+0000","last_became_peered":"2024-09-18T08:02:46.868175+0000","last_unstale":"2024-09-18T08:07:08.221596+0000","last_undegraded":"2024-09-18T08:07:08.221596+0000","last_fullsized":"2024-09-18T08:07:08.221596+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"62'1","last_scrub_stamp":"2024-09-18T08:04:09.732645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:45.855979+0000","last_clean_scrub_stamp":"2024-09-18T08:04:09.732645+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:07:52.854217+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0.00066785499999999997,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":204,"reported_epoch":149,"state":"active+clean","last_fresh":"2024-09-18T08:07:08.808087+0000","last_change":"2024-09-18T08:07:08.808087+0000","last_active":"2024-09-18T08:07:08.808087+0000","last_peered":"2024-09-18T08:07:08.808087+0000","last_clean":"2024-09-18T08:07:08.808087+0000","last_became_active":"2024-09-18T08:02:51.908156+0000","last_became_peered":"2024-09-18T08:02:51.908156+0000","last_unstale":"2024-09-18T08:07:08.808087+0000","last_undegraded":"2024-09-18T08:07:08.808087+0000","last_fullsized":"2024-09-18T08:07:08.808087+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:07:08.806748+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:50.886830+0000","last_clean_scrub_stamp":"2024-09-18T08:07:08.806748+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:08:09.365103+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":207,"reported_epoch":149,"state":"active+clean","last_fresh":"2024-09-18T08:07:10.837850+0000","last_change":"2024-09-18T08:07:10.837850+0000","last_active":"2024-09-18T08:07:10.837850+0000","last_peered":"2024-09-18T08:07:10.837850+0000","last_clean":"2024-09-18T08:07:10.837850+0000","last_became_active":"2024-09-18T08:06:24.145382+0000","last_became_peered":"2024-09-18T08:06:24.145382+0000","last_unstale":"2024-09-18T08:07:10.837850+0000","last_undegraded":"2024-09-18T08:07:10.837850+0000","last_fullsized":"2024-09-18T08:07:10.837850+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:07:10.837780+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_clean_scrub_stamp":"2024-09-18T08:07:10.837780+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:08:18.424418+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.5","version":"149'13165","reported_seq":13356,"reported_epoch":149,"state":"active+clean","last_fresh":"2024-09-18T08:07:13.319952+0000","last_change":"2024-09-18T08:06:48.205596+0000","last_active":"2024-09-18T08:07:13.319952+0000","last_peered":"2024-09-18T08:07:13.319952+0000","last_clean":"2024-09-18T08:07:13.319952+0000","last_became_active":"2024-09-18T08:06:46.140892+0000","last_became_peered":"2024-09-18T08:06:46.140892+0000","last_unstale":"2024-09-18T08:07:13.319952+0000","last_undegraded":"2024-09-18T08:07:13.319952+0000","last_fullsized":"2024-09-18T08:07:13.319952+0000","mapping_epoch":143,"log_start":"112'6323","ondisk_log_start":"112'6323","created":83,"last_epoch_clean":144,"parent":"0.0","parent_split_bits":0,"last_scrub":"144'11604","last_scrub_stamp":"2024-09-18T08:06:48.205349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:03:56.449957+0000","last_clean_scrub_stamp":"2024-09-18T08:06:48.205349+0000","objects_scrubbed":3183,"log_size":6842,"log_dups_size":0,"ondisk_log_size":6842,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:08:17.112045+0000","scrub_duration":1267,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":103743488,"num_objects":1583,"num_object_clones":0,"num_object_copies":4749,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1583,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13165,"num_write_kb":471936,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,0],"acting":[2,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.6","version":"69'12","reported_seq":210,"reported_epoch":149,"state":"active+clean","last_fresh":"2024-09-18T08:07:08.234100+0000","last_change":"2024-09-18T08:05:38.471012+0000","last_active":"2024-09-18T08:07:08.234100+0000","last_peered":"2024-09-18T08:07:08.234100+0000","last_clean":"2024-09-18T08:07:08.234100+0000","last_became_active":"2024-09-18T08:02:49.898079+0000","last_became_peered":"2024-09-18T08:02:49.898079+0000","last_unstale":"2024-09-18T08:07:08.234100+0000","last_undegraded":"2024-09-18T08:07:08.234100+0000","last_fullsized":"2024-09-18T08:07:08.234100+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'12","last_scrub_stamp":"2024-09-18T08:05:38.470713+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_clean_scrub_stamp":"2024-09-18T08:05:38.470713+0000","objects_scrubbed":6,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":28,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":18,"num_read_kb":12,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":204,"reported_epoch":149,"state":"active+clean","last_fresh":"2024-09-18T08:07:10.782142+0000","last_change":"2024-09-18T08:07:10.782142+0000","last_active":"2024-09-18T08:07:10.782142+0000","last_peered":"2024-09-18T08:07:10.782142+0000","last_clean":"2024-09-18T08:07:10.782142+0000","last_became_active":"2024-09-18T08:02:47.873809+0000","last_became_peered":"2024-09-18T08:02:47.873809+0000","last_unstale":"2024-09-18T08:07:10.782142+0000","last_undegraded":"2024-09-18T08:07:10.782142+0000","last_fullsized":"2024-09-18T08:07:10.782142+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:07:10.781678+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:46.860092+0000","last_clean_scrub_stamp":"2024-09-18T08:07:10.781678+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:08:17.733635+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"70'5","reported_seq":102,"reported_epoch":149,"state":"active+clean","last_fresh":"2024-09-18T08:07:12.210781+0000","last_change":"2024-09-18T08:06:51.522688+0000","last_active":"2024-09-18T08:07:12.210781+0000","last_peered":"2024-09-18T08:07:12.210781+0000","last_clean":"2024-09-18T08:07:12.210781+0000","last_became_active":"2024-09-18T08:06:46.141667+0000","last_became_peered":"2024-09-18T08:06:46.141667+0000","last_unstale":"2024-09-18T08:07:12.210781+0000","last_undegraded":"2024-09-18T08:07:12.210781+0000","last_fullsized":"2024-09-18T08:07:12.210781+0000","mapping_epoch":143,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":144,"parent":"0.0","parent_split_bits":0,"last_scrub":"70'5","last_scrub_stamp":"2024-09-18T08:06:51.522442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:45.855979+0000","last_clean_scrub_stamp":"2024-09-18T08:06:51.522442+0000","objects_scrubbed":1,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:07:58.661843+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0.000504055,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":34,"num_read_kb":29,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":196,"reported_epoch":149,"state":"active+clean","last_fresh":"2024-09-18T08:07:08.230230+0000","last_change":"2024-09-18T08:05:41.750515+0000","last_active":"2024-09-18T08:07:08.230230+0000","last_peered":"2024-09-18T08:07:08.230230+0000","last_clean":"2024-09-18T08:07:08.230230+0000","last_became_active":"2024-09-18T08:02:51.913112+0000","last_became_peered":"2024-09-18T08:02:51.913112+0000","last_unstale":"2024-09-18T08:07:08.230230+0000","last_undegraded":"2024-09-18T08:07:08.230230+0000","last_fullsized":"2024-09-18T08:07:08.230230+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:05:41.750446+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:50.886830+0000","last_clean_scrub_stamp":"2024-09-18T08:05:41.750446+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":177,"reported_epoch":149,"state":"active+clean","last_fresh":"2024-09-18T08:07:08.232691+0000","last_change":"2024-09-18T08:02:53.907689+0000","last_active":"2024-09-18T08:07:08.232691+0000","last_peered":"2024-09-18T08:07:08.232691+0000","last_clean":"2024-09-18T08:07:08.232691+0000","last_became_active":"2024-09-18T08:02:53.907404+0000","last_became_peered":"2024-09-18T08:02:53.907404+0000","last_unstale":"2024-09-18T08:07:08.232691+0000","last_undegraded":"2024-09-18T08:07:08.232691+0000","last_fullsized":"2024-09-18T08:07:08.232691+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_clean_scrub_stamp":"2024-09-18T08:02:52.890475+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:07:25.591758+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.6","version":"149'13153","reported_seq":13366,"reported_epoch":149,"state":"active+clean","last_fresh":"2024-09-18T08:07:13.527155+0000","last_change":"2024-09-18T08:07:09.697275+0000","last_active":"2024-09-18T08:07:13.527155+0000","last_peered":"2024-09-18T08:07:13.527155+0000","last_clean":"2024-09-18T08:07:13.527155+0000","last_became_active":"2024-09-18T08:03:57.466391+0000","last_became_peered":"2024-09-18T08:03:57.466391+0000","last_unstale":"2024-09-18T08:07:13.527155+0000","last_undegraded":"2024-09-18T08:07:13.527155+0000","last_fullsized":"2024-09-18T08:07:13.527155+0000","mapping_epoch":83,"log_start":"120'7527","ondisk_log_start":"120'7527","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"149'12922","last_scrub_stamp":"2024-09-18T08:07:09.696986+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:03:56.449957+0000","last_clean_scrub_stamp":"2024-09-18T08:07:09.696986+0000","objects_scrubbed":1879,"log_size":5626,"log_dups_size":0,"ondisk_log_size":5626,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:08:34.043189+0000","scrub_duration":889,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":106233856,"num_objects":1621,"num_object_clones":0,"num_object_copies":4863,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1621,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13153,"num_write_kb":472768,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.5","version":"138'19","reported_seq":242,"reported_epoch":149,"state":"active+clean","last_fresh":"2024-09-18T08:07:08.215345+0000","last_change":"2024-09-18T08:06:24.191102+0000","last_active":"2024-09-18T08:07:08.215345+0000","last_peered":"2024-09-18T08:07:08.215345+0000","last_clean":"2024-09-18T08:07:08.215345+0000","last_became_active":"2024-09-18T08:06:24.146117+0000","last_became_peered":"2024-09-18T08:06:24.146117+0000","last_unstale":"2024-09-18T08:07:08.215345+0000","last_undegraded":"2024-09-18T08:07:08.215345+0000","last_fullsized":"2024-09-18T08:07:08.215345+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"130'18","last_scrub_stamp":"2024-09-18T08:06:15.455503+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_clean_scrub_stamp":"2024-09-18T08:06:15.455503+0000","objects_scrubbed":8,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:07:41.325892+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":24,"num_read_kb":14,"num_write":13,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":308,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,7],"acting":[4,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":210,"reported_epoch":149,"state":"active+clean","last_fresh":"2024-09-18T08:07:08.227145+0000","last_change":"2024-09-18T08:06:24.134043+0000","last_active":"2024-09-18T08:07:08.227145+0000","last_peered":"2024-09-18T08:07:08.227145+0000","last_clean":"2024-09-18T08:07:08.227145+0000","last_became_active":"2024-09-18T08:06:24.133425+0000","last_became_peered":"2024-09-18T08:06:24.133425+0000","last_unstale":"2024-09-18T08:07:08.227145+0000","last_undegraded":"2024-09-18T08:07:08.227145+0000","last_fullsized":"2024-09-18T08:07:08.227145+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:05:38.080993+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:46.860092+0000","last_clean_scrub_stamp":"2024-09-18T08:05:38.080993+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"74'79","reported_seq":270,"reported_epoch":149,"state":"active+clean","last_fresh":"2024-09-18T08:07:08.234595+0000","last_change":"2024-09-18T08:06:24.167946+0000","last_active":"2024-09-18T08:07:08.234595+0000","last_peered":"2024-09-18T08:07:08.234595+0000","last_clean":"2024-09-18T08:07:08.234595+0000","last_became_active":"2024-09-18T08:06:24.146679+0000","last_became_peered":"2024-09-18T08:06:24.146679+0000","last_unstale":"2024-09-18T08:07:08.234595+0000","last_undegraded":"2024-09-18T08:07:08.234595+0000","last_fullsized":"2024-09-18T08:07:08.234595+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'79","last_scrub_stamp":"2024-09-18T08:04:33.791841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:00:11.893742+0000","last_clean_scrub_stamp":"2024-09-18T08:04:33.791841+0000","objects_scrubbed":2,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:07:58.836812+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":231,"num_write":113,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1771104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":31,"reported_epoch":149,"state":"active+clean","last_fresh":"2024-09-18T08:07:09.744206+0000","last_change":"2024-09-18T08:07:09.744206+0000","last_active":"2024-09-18T08:07:09.744206+0000","last_peered":"2024-09-18T08:07:09.744206+0000","last_clean":"2024-09-18T08:07:09.744206+0000","last_became_active":"2024-09-18T08:06:46.141046+0000","last_became_peered":"2024-09-18T08:06:46.141046+0000","last_unstale":"2024-09-18T08:07:09.744206+0000","last_undegraded":"2024-09-18T08:07:09.744206+0000","last_fullsized":"2024-09-18T08:07:09.744206+0000","mapping_epoch":143,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":144,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:07:09.744137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:50.886830+0000","last_clean_scrub_stamp":"2024-09-18T08:07:09.744137+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:08:16.138578+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":192,"reported_epoch":149,"state":"active+clean","last_fresh":"2024-09-18T08:07:08.230100+0000","last_change":"2024-09-18T08:06:50.710637+0000","last_active":"2024-09-18T08:07:08.230100+0000","last_peered":"2024-09-18T08:07:08.230100+0000","last_clean":"2024-09-18T08:07:08.230100+0000","last_became_active":"2024-09-18T08:02:53.908668+0000","last_became_peered":"2024-09-18T08:02:53.908668+0000","last_unstale":"2024-09-18T08:07:08.230100+0000","last_undegraded":"2024-09-18T08:07:08.230100+0000","last_fullsized":"2024-09-18T08:07:08.230100+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:06:50.710565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_clean_scrub_stamp":"2024-09-18T08:06:50.710565+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:08:16.618092+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.d","version":"149'13182","reported_seq":13442,"reported_epoch":149,"state":"active+clean+scrubbing","last_fresh":"2024-09-18T08:07:13.308211+0000","last_change":"2024-09-18T08:07:12.696588+0000","last_active":"2024-09-18T08:07:13.308211+0000","last_peered":"2024-09-18T08:07:13.308211+0000","last_clean":"2024-09-18T08:07:13.308211+0000","last_became_active":"2024-09-18T08:06:25.123627+0000","last_became_peered":"2024-09-18T08:06:25.123627+0000","last_unstale":"2024-09-18T08:07:13.308211+0000","last_undegraded":"2024-09-18T08:07:13.308211+0000","last_fullsized":"2024-09-18T08:07:13.308211+0000","mapping_epoch":137,"log_start":"124'8137","ondisk_log_start":"124'8137","created":83,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"123'7982","last_scrub_stamp":"2024-09-18T08:05:44.590884+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:03:56.449957+0000","last_clean_scrub_stamp":"2024-09-18T08:05:44.590884+0000","objects_scrubbed":1609,"log_size":5045,"log_dups_size":0,"ondisk_log_size":5045,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"scrubbing for 0s","scrub_duration":2853,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":104005656,"num_objects":1588,"num_object_clones":0,"num_object_copies":4764,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1588,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":13182,"num_write_kb":472577,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,0],"acting":[1,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.e","version":"69'11","reported_seq":248,"reported_epoch":149,"state":"active+clean","last_fresh":"2024-09-18T08:07:08.215861+0000","last_change":"2024-09-18T08:06:24.173542+0000","last_active":"2024-09-18T08:07:08.215861+0000","last_peered":"2024-09-18T08:07:08.215861+0000","last_clean":"2024-09-18T08:07:08.215861+0000","last_became_active":"2024-09-18T08:06:24.143839+0000","last_became_peered":"2024-09-18T08:06:24.143839+0000","last_unstale":"2024-09-18T08:07:08.215861+0000","last_undegraded":"2024-09-18T08:07:08.215861+0000","last_fullsized":"2024-09-18T08:07:08.215861+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'11","last_scrub_stamp":"2024-09-18T08:06:14.472168+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_clean_scrub_stamp":"2024-09-18T08:06:14.472168+0000","objects_scrubbed":5,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:07:20.674211+0000","scrub_duration":23,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":188,"reported_epoch":149,"state":"active+clean","last_fresh":"2024-09-18T08:07:08.215543+0000","last_change":"2024-09-18T08:02:47.880265+0000","last_active":"2024-09-18T08:07:08.215543+0000","last_peered":"2024-09-18T08:07:08.215543+0000","last_clean":"2024-09-18T08:07:08.215543+0000","last_became_active":"2024-09-18T08:02:47.879764+0000","last_became_peered":"2024-09-18T08:02:47.879764+0000","last_unstale":"2024-09-18T08:07:08.215543+0000","last_undegraded":"2024-09-18T08:07:08.215543+0000","last_fullsized":"2024-09-18T08:07:08.215543+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:02:46.860092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:46.860092+0000","last_clean_scrub_stamp":"2024-09-18T08:02:46.860092+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:07:56.537498+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":201,"reported_epoch":149,"state":"active+clean","last_fresh":"2024-09-18T08:07:08.230004+0000","last_change":"2024-09-18T08:06:24.132512+0000","last_active":"2024-09-18T08:07:08.230004+0000","last_peered":"2024-09-18T08:07:08.230004+0000","last_clean":"2024-09-18T08:07:08.230004+0000","last_became_active":"2024-09-18T08:06:24.132140+0000","last_became_peered":"2024-09-18T08:06:24.132140+0000","last_unstale":"2024-09-18T08:07:08.230004+0000","last_undegraded":"2024-09-18T08:07:08.230004+0000","last_fullsized":"2024-09-18T08:07:08.230004+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:05:08.839644+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:50.886830+0000","last_clean_scrub_stamp":"2024-09-18T08:05:08.839644+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:07:35.376189+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":205,"reported_epoch":149,"state":"active+clean","last_fresh":"2024-09-18T08:07:09.197912+0000","last_change":"2024-09-18T08:07:09.197912+0000","last_active":"2024-09-18T08:07:09.197912+0000","last_peered":"2024-09-18T08:07:09.197912+0000","last_clean":"2024-09-18T08:07:09.197912+0000","last_became_active":"2024-09-18T08:06:24.130266+0000","last_became_peered":"2024-09-18T08:06:24.130266+0000","last_unstale":"2024-09-18T08:07:09.197912+0000","last_undegraded":"2024-09-18T08:07:09.197912+0000","last_fullsized":"2024-09-18T08:07:09.197912+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:07:09.197841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_clean_scrub_stamp":"2024-09-18T08:07:09.197841+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:08:21.836105+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.e","version":"149'13200","reported_seq":13462,"reported_epoch":149,"state":"active+clean","last_fresh":"2024-09-18T08:07:13.295320+0000","last_change":"2024-09-18T08:05:47.424879+0000","last_active":"2024-09-18T08:07:13.295320+0000","last_peered":"2024-09-18T08:07:13.295320+0000","last_clean":"2024-09-18T08:07:13.295320+0000","last_became_active":"2024-09-18T08:03:57.464311+0000","last_became_peered":"2024-09-18T08:03:57.464311+0000","last_unstale":"2024-09-18T08:07:13.295320+0000","last_undegraded":"2024-09-18T08:07:13.295320+0000","last_fullsized":"2024-09-18T08:07:13.295320+0000","mapping_epoch":83,"log_start":"111'6339","ondisk_log_start":"111'6339","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:03:56.449957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:03:56.449957+0000","last_clean_scrub_stamp":"2024-09-18T08:03:56.449957+0000","objects_scrubbed":5849,"log_size":6861,"log_dups_size":0,"ondisk_log_size":6861,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:07:37.027598+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":103153664,"num_objects":1574,"num_object_clones":0,"num_object_copies":4722,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1574,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13200,"num_write_kb":472768,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,0],"acting":[2,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.d","version":"69'17","reported_seq":211,"reported_epoch":149,"state":"active+clean","last_fresh":"2024-09-18T08:07:08.215445+0000","last_change":"2024-09-18T08:02:49.895971+0000","last_active":"2024-09-18T08:07:08.215445+0000","last_peered":"2024-09-18T08:07:08.215445+0000","last_clean":"2024-09-18T08:07:08.215445+0000","last_became_active":"2024-09-18T08:02:49.895578+0000","last_became_peered":"2024-09-18T08:02:49.895578+0000","last_unstale":"2024-09-18T08:07:08.215445+0000","last_undegraded":"2024-09-18T08:07:08.215445+0000","last_fullsized":"2024-09-18T08:07:08.215445+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_clean_scrub_stamp":"2024-09-18T08:02:48.870718+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:08:06.433938+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":29,"num_read_kb":19,"num_write":18,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,1],"acting":[4,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":130,"reported_epoch":149,"state":"active+clean","last_fresh":"2024-09-18T08:07:08.214560+0000","last_change":"2024-09-18T08:06:24.144485+0000","last_active":"2024-09-18T08:07:08.214560+0000","last_peered":"2024-09-18T08:07:08.214560+0000","last_clean":"2024-09-18T08:07:08.214560+0000","last_became_active":"2024-09-18T08:06:24.143152+0000","last_became_peered":"2024-09-18T08:06:24.143152+0000","last_unstale":"2024-09-18T08:07:08.214560+0000","last_undegraded":"2024-09-18T08:07:08.214560+0000","last_fullsized":"2024-09-18T08:07:08.214560+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:05:12.622901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:46.860092+0000","last_clean_scrub_stamp":"2024-09-18T08:05:12.622901+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:08:00.591147+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":195,"reported_epoch":149,"state":"active+clean","last_fresh":"2024-09-18T08:07:08.232736+0000","last_change":"2024-09-18T08:05:31.851989+0000","last_active":"2024-09-18T08:07:08.232736+0000","last_peered":"2024-09-18T08:07:08.232736+0000","last_clean":"2024-09-18T08:07:08.232736+0000","last_became_active":"2024-09-18T08:02:51.907196+0000","last_became_peered":"2024-09-18T08:02:51.907196+0000","last_unstale":"2024-09-18T08:07:08.232736+0000","last_undegraded":"2024-09-18T08:07:08.232736+0000","last_fullsized":"2024-09-18T08:07:08.232736+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:05:31.851797+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:50.886830+0000","last_clean_scrub_stamp":"2024-09-18T08:05:31.851797+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:08:02.574997+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":184,"reported_epoch":149,"state":"active+clean","last_fresh":"2024-09-18T08:07:08.222962+0000","last_change":"2024-09-18T08:04:06.709703+0000","last_active":"2024-09-18T08:07:08.222962+0000","last_peered":"2024-09-18T08:07:08.222962+0000","last_clean":"2024-09-18T08:07:08.222962+0000","last_became_active":"2024-09-18T08:02:53.904144+0000","last_became_peered":"2024-09-18T08:02:53.904144+0000","last_unstale":"2024-09-18T08:07:08.222962+0000","last_undegraded":"2024-09-18T08:07:08.222962+0000","last_fullsized":"2024-09-18T08:07:08.222962+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:04:06.709631+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_clean_scrub_stamp":"2024-09-18T08:04:06.709631+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:07:24.993615+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.f","version":"149'13135","reported_seq":13379,"reported_epoch":149,"state":"active+clean","last_fresh":"2024-09-18T08:07:13.572102+0000","last_change":"2024-09-18T08:06:49.956287+0000","last_active":"2024-09-18T08:07:13.572102+0000","last_peered":"2024-09-18T08:07:13.572102+0000","last_clean":"2024-09-18T08:07:13.572102+0000","last_became_active":"2024-09-18T08:03:57.467980+0000","last_became_peered":"2024-09-18T08:03:57.467980+0000","last_unstale":"2024-09-18T08:07:13.572102+0000","last_undegraded":"2024-09-18T08:07:13.572102+0000","last_fullsized":"2024-09-18T08:07:13.572102+0000","mapping_epoch":83,"log_start":"120'7563","ondisk_log_start":"120'7563","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"144'11682","last_scrub_stamp":"2024-09-18T08:06:49.956139+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:03:56.449957+0000","last_clean_scrub_stamp":"2024-09-18T08:06:49.956139+0000","objects_scrubbed":3019,"log_size":5572,"log_dups_size":0,"ondisk_log_size":5572,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:08:15.903850+0000","scrub_duration":1309,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":100597760,"num_objects":1535,"num_object_clones":0,"num_object_copies":4605,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1535,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13135,"num_write_kb":469440,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.c","version":"69'10","reported_seq":244,"reported_epoch":149,"state":"active+clean","last_fresh":"2024-09-18T08:07:08.215125+0000","last_change":"2024-09-18T08:06:24.180438+0000","last_active":"2024-09-18T08:07:08.215125+0000","last_peered":"2024-09-18T08:07:08.215125+0000","last_clean":"2024-09-18T08:07:08.215125+0000","last_became_active":"2024-09-18T08:06:24.145005+0000","last_became_peered":"2024-09-18T08:06:24.145005+0000","last_unstale":"2024-09-18T08:07:08.215125+0000","last_undegraded":"2024-09-18T08:07:08.215125+0000","last_fullsized":"2024-09-18T08:07:08.215125+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'10","last_scrub_stamp":"2024-09-18T08:05:41.474813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_clean_scrub_stamp":"2024-09-18T08:05:41.474813+0000","objects_scrubbed":5,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":203,"reported_epoch":149,"state":"active+clean","last_fresh":"2024-09-18T08:07:08.224560+0000","last_change":"2024-09-18T08:06:46.339604+0000","last_active":"2024-09-18T08:07:08.224560+0000","last_peered":"2024-09-18T08:07:08.224560+0000","last_clean":"2024-09-18T08:07:08.224560+0000","last_became_active":"2024-09-18T08:02:47.881512+0000","last_became_peered":"2024-09-18T08:02:47.881512+0000","last_unstale":"2024-09-18T08:07:08.224560+0000","last_undegraded":"2024-09-18T08:07:08.224560+0000","last_fullsized":"2024-09-18T08:07:08.224560+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:06:46.339428+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:46.860092+0000","last_clean_scrub_stamp":"2024-09-18T08:06:46.339428+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:07:54.370307+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.d","version":"69'8","reported_seq":257,"reported_epoch":149,"state":"active+clean","last_fresh":"2024-09-18T08:07:08.232802+0000","last_change":"2024-09-18T08:05:33.027988+0000","last_active":"2024-09-18T08:07:08.232802+0000","last_peered":"2024-09-18T08:07:08.232802+0000","last_clean":"2024-09-18T08:07:08.232802+0000","last_became_active":"2024-09-18T08:02:51.909011+0000","last_became_peered":"2024-09-18T08:02:51.909011+0000","last_unstale":"2024-09-18T08:07:08.232802+0000","last_undegraded":"2024-09-18T08:07:08.232802+0000","last_fullsized":"2024-09-18T08:07:08.232802+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'8","last_scrub_stamp":"2024-09-18T08:05:33.027783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:50.886830+0000","last_clean_scrub_stamp":"2024-09-18T08:05:33.027783+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:08:06.774918+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":184,"reported_epoch":149,"state":"active+clean","last_fresh":"2024-09-18T08:07:08.215596+0000","last_change":"2024-09-18T08:04:32.377211+0000","last_active":"2024-09-18T08:07:08.215596+0000","last_peered":"2024-09-18T08:07:08.215596+0000","last_clean":"2024-09-18T08:07:08.215596+0000","last_became_active":"2024-09-18T08:02:53.901858+0000","last_became_peered":"2024-09-18T08:02:53.901858+0000","last_unstale":"2024-09-18T08:07:08.215596+0000","last_undegraded":"2024-09-18T08:07:08.215596+0000","last_fullsized":"2024-09-18T08:07:08.215596+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:04:32.377157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_clean_scrub_stamp":"2024-09-18T08:04:32.377157+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.8","version":"149'13187","reported_seq":13375,"reported_epoch":149,"state":"active+clean","last_fresh":"2024-09-18T08:07:13.291716+0000","last_change":"2024-09-18T08:06:43.340032+0000","last_active":"2024-09-18T08:07:13.291716+0000","last_peered":"2024-09-18T08:07:13.291716+0000","last_clean":"2024-09-18T08:07:13.291716+0000","last_became_active":"2024-09-18T08:03:57.464738+0000","last_became_peered":"2024-09-18T08:03:57.464738+0000","last_unstale":"2024-09-18T08:07:13.291716+0000","last_undegraded":"2024-09-18T08:07:13.291716+0000","last_fullsized":"2024-09-18T08:07:13.291716+0000","mapping_epoch":83,"log_start":"111'6327","ondisk_log_start":"111'6327","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"142'11403","last_scrub_stamp":"2024-09-18T08:06:43.339730+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:03:56.449957+0000","last_clean_scrub_stamp":"2024-09-18T08:06:43.339730+0000","objects_scrubbed":3493,"log_size":6860,"log_dups_size":0,"ondisk_log_size":6860,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:07:53.964339+0000","scrub_duration":1312,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":108986368,"num_objects":1663,"num_object_clones":0,"num_object_copies":4989,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1663,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13187,"num_write_kb":475200,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,4],"acting":[2,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.b","version":"69'9","reported_seq":207,"reported_epoch":149,"state":"active+clean","last_fresh":"2024-09-18T08:07:08.226672+0000","last_change":"2024-09-18T08:05:05.294031+0000","last_active":"2024-09-18T08:07:08.226672+0000","last_peered":"2024-09-18T08:07:08.226672+0000","last_clean":"2024-09-18T08:07:08.226672+0000","last_became_active":"2024-09-18T08:02:49.897998+0000","last_became_peered":"2024-09-18T08:02:49.897998+0000","last_unstale":"2024-09-18T08:07:08.226672+0000","last_undegraded":"2024-09-18T08:07:08.226672+0000","last_fullsized":"2024-09-18T08:07:08.226672+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'9","last_scrub_stamp":"2024-09-18T08:05:05.293936+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_clean_scrub_stamp":"2024-09-18T08:05:05.293936+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:08:03.837067+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":209,"reported_epoch":149,"state":"active+clean","last_fresh":"2024-09-18T08:07:08.230081+0000","last_change":"2024-09-18T08:06:24.135338+0000","last_active":"2024-09-18T08:07:08.230081+0000","last_peered":"2024-09-18T08:07:08.230081+0000","last_clean":"2024-09-18T08:07:08.230081+0000","last_became_active":"2024-09-18T08:06:24.135005+0000","last_became_peered":"2024-09-18T08:06:24.135005+0000","last_unstale":"2024-09-18T08:07:08.230081+0000","last_undegraded":"2024-09-18T08:07:08.230081+0000","last_fullsized":"2024-09-18T08:07:08.230081+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:04:59.080115+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:46.860092+0000","last_clean_scrub_stamp":"2024-09-18T08:04:59.080115+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:07:30.323111+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":204,"reported_epoch":149,"state":"active+clean","last_fresh":"2024-09-18T08:07:10.728812+0000","last_change":"2024-09-18T08:07:10.728745+0000","last_active":"2024-09-18T08:07:10.728812+0000","last_peered":"2024-09-18T08:07:10.728812+0000","last_clean":"2024-09-18T08:07:10.728812+0000","last_became_active":"2024-09-18T08:02:51.901330+0000","last_became_peered":"2024-09-18T08:02:51.901330+0000","last_unstale":"2024-09-18T08:07:10.728812+0000","last_undegraded":"2024-09-18T08:07:10.728812+0000","last_fullsized":"2024-09-18T08:07:10.728812+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:07:10.728679+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:50.886830+0000","last_clean_scrub_stamp":"2024-09-18T08:07:10.728679+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:08:31.598375+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,3],"acting":[2,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":192,"reported_epoch":149,"state":"active+clean","last_fresh":"2024-09-18T08:07:11.804906+0000","last_change":"2024-09-18T08:07:11.804906+0000","last_active":"2024-09-18T08:07:11.804906+0000","last_peered":"2024-09-18T08:07:11.804906+0000","last_clean":"2024-09-18T08:07:11.804906+0000","last_became_active":"2024-09-18T08:02:53.911068+0000","last_became_peered":"2024-09-18T08:02:53.911068+0000","last_unstale":"2024-09-18T08:07:11.804906+0000","last_undegraded":"2024-09-18T08:07:11.804906+0000","last_fullsized":"2024-09-18T08:07:11.804906+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:07:11.804821+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_clean_scrub_stamp":"2024-09-18T08:07:11.804821+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:08:17.666214+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.9","version":"149'13170","reported_seq":13357,"reported_epoch":149,"state":"active+clean","last_fresh":"2024-09-18T08:07:13.531624+0000","last_change":"2024-09-18T08:06:43.822074+0000","last_active":"2024-09-18T08:07:13.531624+0000","last_peered":"2024-09-18T08:07:13.531624+0000","last_clean":"2024-09-18T08:07:13.531624+0000","last_became_active":"2024-09-18T08:03:57.467065+0000","last_became_peered":"2024-09-18T08:03:57.467065+0000","last_unstale":"2024-09-18T08:07:13.531624+0000","last_undegraded":"2024-09-18T08:07:13.531624+0000","last_fullsized":"2024-09-18T08:07:13.531624+0000","mapping_epoch":83,"log_start":"120'7580","ondisk_log_start":"120'7580","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"142'11396","last_scrub_stamp":"2024-09-18T08:06:43.821646+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:03:56.449957+0000","last_clean_scrub_stamp":"2024-09-18T08:06:43.821646+0000","objects_scrubbed":3394,"log_size":5590,"log_dups_size":0,"ondisk_log_size":5590,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:07:54.095340+0000","scrub_duration":1188,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":103546880,"num_objects":1580,"num_object_clones":0,"num_object_copies":4740,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1580,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13170,"num_write_kb":472000,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.a","version":"69'19","reported_seq":213,"reported_epoch":149,"state":"active+clean","last_fresh":"2024-09-18T08:07:08.233646+0000","last_change":"2024-09-18T08:06:24.162464+0000","last_active":"2024-09-18T08:07:08.233646+0000","last_peered":"2024-09-18T08:07:08.233646+0000","last_clean":"2024-09-18T08:07:08.233646+0000","last_became_active":"2024-09-18T08:06:24.139909+0000","last_became_peered":"2024-09-18T08:06:24.139909+0000","last_unstale":"2024-09-18T08:07:08.233646+0000","last_undegraded":"2024-09-18T08:07:08.233646+0000","last_fullsized":"2024-09-18T08:07:08.233646+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'19","last_scrub_stamp":"2024-09-18T08:04:20.391859+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_clean_scrub_stamp":"2024-09-18T08:04:20.391859+0000","objects_scrubbed":0,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:07:32.660944+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":31,"num_read_kb":20,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,7],"acting":[1,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":206,"reported_epoch":149,"state":"active+clean","last_fresh":"2024-09-18T08:07:08.229933+0000","last_change":"2024-09-18T08:06:47.638974+0000","last_active":"2024-09-18T08:07:08.229933+0000","last_peered":"2024-09-18T08:07:08.229933+0000","last_clean":"2024-09-18T08:07:08.229933+0000","last_became_active":"2024-09-18T08:06:46.171298+0000","last_became_peered":"2024-09-18T08:06:46.171298+0000","last_unstale":"2024-09-18T08:07:08.229933+0000","last_undegraded":"2024-09-18T08:07:08.229933+0000","last_fullsized":"2024-09-18T08:07:08.229933+0000","mapping_epoch":143,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":144,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:06:47.638911+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:46.860092+0000","last_clean_scrub_stamp":"2024-09-18T08:06:47.638911+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:07:53.269627+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":187,"reported_epoch":149,"state":"active+clean","last_fresh":"2024-09-18T08:07:08.221931+0000","last_change":"2024-09-18T08:04:56.004261+0000","last_active":"2024-09-18T08:07:08.221931+0000","last_peered":"2024-09-18T08:07:08.221931+0000","last_clean":"2024-09-18T08:07:08.221931+0000","last_became_active":"2024-09-18T08:02:51.903439+0000","last_became_peered":"2024-09-18T08:02:51.903439+0000","last_unstale":"2024-09-18T08:07:08.221931+0000","last_undegraded":"2024-09-18T08:07:08.221931+0000","last_fullsized":"2024-09-18T08:07:08.221931+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:04:56.004150+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:50.886830+0000","last_clean_scrub_stamp":"2024-09-18T08:04:56.004150+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:08:02.816710+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":31,"reported_epoch":149,"state":"active+clean","last_fresh":"2024-09-18T08:07:08.223051+0000","last_change":"2024-09-18T08:06:50.902917+0000","last_active":"2024-09-18T08:07:08.223051+0000","last_peered":"2024-09-18T08:07:08.223051+0000","last_clean":"2024-09-18T08:07:08.223051+0000","last_became_active":"2024-09-18T08:06:46.137872+0000","last_became_peered":"2024-09-18T08:06:46.137872+0000","last_unstale":"2024-09-18T08:07:08.223051+0000","last_undegraded":"2024-09-18T08:07:08.223051+0000","last_fullsized":"2024-09-18T08:07:08.223051+0000","mapping_epoch":143,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":144,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:06:50.902835+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_clean_scrub_stamp":"2024-09-18T08:06:50.902835+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:07:54.316655+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,3],"acting":[2,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.a","version":"149'13290","reported_seq":13434,"reported_epoch":149,"state":"active+clean","last_fresh":"2024-09-18T08:07:13.499085+0000","last_change":"2024-09-18T08:03:57.466483+0000","last_active":"2024-09-18T08:07:13.499085+0000","last_peered":"2024-09-18T08:07:13.499085+0000","last_clean":"2024-09-18T08:07:13.499085+0000","last_became_active":"2024-09-18T08:03:57.466168+0000","last_became_peered":"2024-09-18T08:03:57.466168+0000","last_unstale":"2024-09-18T08:07:13.499085+0000","last_undegraded":"2024-09-18T08:07:13.499085+0000","last_fullsized":"2024-09-18T08:07:13.499085+0000","mapping_epoch":83,"log_start":"126'8606","ondisk_log_start":"126'8606","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:03:56.449957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:03:56.449957+0000","last_clean_scrub_stamp":"2024-09-18T08:03:56.449957+0000","objects_scrubbed":0,"log_size":4684,"log_dups_size":0,"ondisk_log_size":4684,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:07:32.247056+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":106692608,"num_objects":1628,"num_object_clones":0,"num_object_copies":4884,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1628,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13290,"num_write_kb":477376,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.9","version":"69'12","reported_seq":223,"reported_epoch":149,"state":"active+clean","last_fresh":"2024-09-18T08:07:08.214807+0000","last_change":"2024-09-18T08:05:33.544197+0000","last_active":"2024-09-18T08:07:08.214807+0000","last_peered":"2024-09-18T08:07:08.214807+0000","last_clean":"2024-09-18T08:07:08.214807+0000","last_became_active":"2024-09-18T08:02:49.894762+0000","last_became_peered":"2024-09-18T08:02:49.894762+0000","last_unstale":"2024-09-18T08:07:08.214807+0000","last_undegraded":"2024-09-18T08:07:08.214807+0000","last_fullsized":"2024-09-18T08:07:08.214807+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'12","last_scrub_stamp":"2024-09-18T08:05:33.544135+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_clean_scrub_stamp":"2024-09-18T08:05:33.544135+0000","objects_scrubbed":5,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":25,"num_read_kb":16,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":203,"reported_epoch":149,"state":"active+clean","last_fresh":"2024-09-18T08:07:08.235548+0000","last_change":"2024-09-18T08:05:39.906123+0000","last_active":"2024-09-18T08:07:08.235548+0000","last_peered":"2024-09-18T08:07:08.235548+0000","last_clean":"2024-09-18T08:07:08.235548+0000","last_became_active":"2024-09-18T08:02:47.884052+0000","last_became_peered":"2024-09-18T08:02:47.884052+0000","last_unstale":"2024-09-18T08:07:08.235548+0000","last_undegraded":"2024-09-18T08:07:08.235548+0000","last_fullsized":"2024-09-18T08:07:08.235548+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:05:39.906062+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:46.860092+0000","last_clean_scrub_stamp":"2024-09-18T08:05:39.906062+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:08:03.739004+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":188,"reported_epoch":149,"state":"active+clean","last_fresh":"2024-09-18T08:07:08.223215+0000","last_change":"2024-09-18T08:05:06.828638+0000","last_active":"2024-09-18T08:07:08.223215+0000","last_peered":"2024-09-18T08:07:08.223215+0000","last_clean":"2024-09-18T08:07:08.223215+0000","last_became_active":"2024-09-18T08:02:51.902163+0000","last_became_peered":"2024-09-18T08:02:51.902163+0000","last_unstale":"2024-09-18T08:07:08.223215+0000","last_undegraded":"2024-09-18T08:07:08.223215+0000","last_fullsized":"2024-09-18T08:07:08.223215+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:05:06.828408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:50.886830+0000","last_clean_scrub_stamp":"2024-09-18T08:05:06.828408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:07:53.834534+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":184,"reported_epoch":149,"state":"active+clean","last_fresh":"2024-09-18T08:07:08.224200+0000","last_change":"2024-09-18T08:05:44.074994+0000","last_active":"2024-09-18T08:07:08.224200+0000","last_peered":"2024-09-18T08:07:08.224200+0000","last_clean":"2024-09-18T08:07:08.224200+0000","last_became_active":"2024-09-18T08:02:53.907412+0000","last_became_peered":"2024-09-18T08:02:53.907412+0000","last_unstale":"2024-09-18T08:07:08.224200+0000","last_undegraded":"2024-09-18T08:07:08.224200+0000","last_fullsized":"2024-09-18T08:07:08.224200+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:05:44.074555+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_clean_scrub_stamp":"2024-09-18T08:05:44.074555+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:07:10.508813+0000","scrub_duration":23,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.b","version":"149'13039","reported_seq":15119,"reported_epoch":149,"state":"active+clean","last_fresh":"2024-09-18T08:07:13.285491+0000","last_change":"2024-09-18T08:06:42.616279+0000","last_active":"2024-09-18T08:07:13.285491+0000","last_peered":"2024-09-18T08:07:13.285491+0000","last_clean":"2024-09-18T08:07:13.285491+0000","last_became_active":"2024-09-18T08:06:24.157344+0000","last_became_peered":"2024-09-18T08:06:24.157344+0000","last_unstale":"2024-09-18T08:07:13.285491+0000","last_undegraded":"2024-09-18T08:07:13.285491+0000","last_fullsized":"2024-09-18T08:07:13.285491+0000","mapping_epoch":136,"log_start":"124'8026","ondisk_log_start":"124'8026","created":83,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"142'11259","last_scrub_stamp":"2024-09-18T08:06:42.615670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:03:56.449957+0000","last_clean_scrub_stamp":"2024-09-18T08:06:42.615670+0000","objects_scrubbed":3328,"log_size":5013,"log_dups_size":0,"ondisk_log_size":5013,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:08:03.658234+0000","scrub_duration":1074,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":99942400,"num_objects":1525,"num_object_clones":0,"num_object_copies":4575,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1525,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13039,"num_write_kb":466048,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":902,"num_bytes_recovered":58982400,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6],"acting":[1,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.8","version":"69'15","reported_seq":263,"reported_epoch":149,"state":"active+clean","last_fresh":"2024-09-18T08:07:08.229462+0000","last_change":"2024-09-18T08:06:24.162625+0000","last_active":"2024-09-18T08:07:08.229462+0000","last_peered":"2024-09-18T08:07:08.229462+0000","last_clean":"2024-09-18T08:07:08.229462+0000","last_became_active":"2024-09-18T08:06:24.138259+0000","last_became_peered":"2024-09-18T08:06:24.138259+0000","last_unstale":"2024-09-18T08:07:08.229462+0000","last_undegraded":"2024-09-18T08:07:08.229462+0000","last_fullsized":"2024-09-18T08:07:08.229462+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'15","last_scrub_stamp":"2024-09-18T08:04:26.859044+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_clean_scrub_stamp":"2024-09-18T08:04:26.859044+0000","objects_scrubbed":7,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:07:26.277667+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.f","version":"62'2","reported_seq":215,"reported_epoch":149,"state":"active+clean","last_fresh":"2024-09-18T08:07:08.235800+0000","last_change":"2024-09-18T08:05:11.906358+0000","last_active":"2024-09-18T08:07:08.235800+0000","last_peered":"2024-09-18T08:07:08.235800+0000","last_clean":"2024-09-18T08:07:08.235800+0000","last_became_active":"2024-09-18T08:02:47.881165+0000","last_became_peered":"2024-09-18T08:02:47.881165+0000","last_unstale":"2024-09-18T08:07:08.235800+0000","last_undegraded":"2024-09-18T08:07:08.235800+0000","last_fullsized":"2024-09-18T08:07:08.235800+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"62'2","last_scrub_stamp":"2024-09-18T08:05:11.906308+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:46.860092+0000","last_clean_scrub_stamp":"2024-09-18T08:05:11.906308+0000","objects_scrubbed":2,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:07:39.906870+0000","scrub_duration":24,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":4,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.9","version":"138'10","reported_seq":284,"reported_epoch":149,"state":"active+clean","last_fresh":"2024-09-18T08:07:12.982798+0000","last_change":"2024-09-18T08:06:24.176322+0000","last_active":"2024-09-18T08:07:12.982798+0000","last_peered":"2024-09-18T08:07:12.982798+0000","last_clean":"2024-09-18T08:07:12.982798+0000","last_became_active":"2024-09-18T08:06:24.170165+0000","last_became_peered":"2024-09-18T08:06:24.170165+0000","last_unstale":"2024-09-18T08:07:12.982798+0000","last_undegraded":"2024-09-18T08:07:12.982798+0000","last_fullsized":"2024-09-18T08:07:12.982798+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"130'9","last_scrub_stamp":"2024-09-18T08:06:14.178538+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:50.886830+0000","last_clean_scrub_stamp":"2024-09-18T08:06:14.178538+0000","objects_scrubbed":1,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:07:19.547201+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":197,"reported_epoch":149,"state":"active+clean","last_fresh":"2024-09-18T08:07:08.229489+0000","last_change":"2024-09-18T08:06:24.135990+0000","last_active":"2024-09-18T08:07:08.229489+0000","last_peered":"2024-09-18T08:07:08.229489+0000","last_clean":"2024-09-18T08:07:08.229489+0000","last_became_active":"2024-09-18T08:06:24.133119+0000","last_became_peered":"2024-09-18T08:06:24.133119+0000","last_unstale":"2024-09-18T08:07:08.229489+0000","last_undegraded":"2024-09-18T08:07:08.229489+0000","last_fullsized":"2024-09-18T08:07:08.229489+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:05:07.879501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_clean_scrub_stamp":"2024-09-18T08:05:07.879501+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:07:32.342623+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":122,"reported_epoch":149,"state":"active+clean","last_fresh":"2024-09-18T08:07:08.225400+0000","last_change":"2024-09-18T08:06:24.133955+0000","last_active":"2024-09-18T08:07:08.225400+0000","last_peered":"2024-09-18T08:07:08.225400+0000","last_clean":"2024-09-18T08:07:08.225400+0000","last_became_active":"2024-09-18T08:06:24.133644+0000","last_became_peered":"2024-09-18T08:06:24.133644+0000","last_unstale":"2024-09-18T08:07:08.225400+0000","last_undegraded":"2024-09-18T08:07:08.225400+0000","last_fullsized":"2024-09-18T08:07:08.225400+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:04:16.378205+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:46.860092+0000","last_clean_scrub_stamp":"2024-09-18T08:04:16.378205+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:07:35.575551+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.17","version":"69'6","reported_seq":208,"reported_epoch":149,"state":"active+clean","last_fresh":"2024-09-18T08:07:08.222563+0000","last_change":"2024-09-18T08:05:26.228344+0000","last_active":"2024-09-18T08:07:08.222563+0000","last_peered":"2024-09-18T08:07:08.222563+0000","last_clean":"2024-09-18T08:07:08.222563+0000","last_became_active":"2024-09-18T08:02:49.895067+0000","last_became_peered":"2024-09-18T08:02:49.895067+0000","last_unstale":"2024-09-18T08:07:08.222563+0000","last_undegraded":"2024-09-18T08:07:08.222563+0000","last_fullsized":"2024-09-18T08:07:08.222563+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'6","last_scrub_stamp":"2024-09-18T08:05:26.227814+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_clean_scrub_stamp":"2024-09-18T08:05:26.227814+0000","objects_scrubbed":3,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":18,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":9,"num_read_kb":6,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":195,"reported_epoch":149,"state":"active+clean","last_fresh":"2024-09-18T08:07:08.231326+0000","last_change":"2024-09-18T08:05:45.812436+0000","last_active":"2024-09-18T08:07:08.231326+0000","last_peered":"2024-09-18T08:07:08.231326+0000","last_clean":"2024-09-18T08:07:08.231326+0000","last_became_active":"2024-09-18T08:02:51.912557+0000","last_became_peered":"2024-09-18T08:02:51.912557+0000","last_unstale":"2024-09-18T08:07:08.231326+0000","last_undegraded":"2024-09-18T08:07:08.231326+0000","last_fullsized":"2024-09-18T08:07:08.231326+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:05:45.812389+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:50.886830+0000","last_clean_scrub_stamp":"2024-09-18T08:05:45.812389+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,1],"acting":[5,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":206,"reported_epoch":149,"state":"active+clean","last_fresh":"2024-09-18T08:07:08.237328+0000","last_change":"2024-09-18T08:06:43.808497+0000","last_active":"2024-09-18T08:07:08.237328+0000","last_peered":"2024-09-18T08:07:08.237328+0000","last_clean":"2024-09-18T08:07:08.237328+0000","last_became_active":"2024-09-18T08:06:24.145909+0000","last_became_peered":"2024-09-18T08:06:24.145909+0000","last_unstale":"2024-09-18T08:07:08.237328+0000","last_undegraded":"2024-09-18T08:07:08.237328+0000","last_fullsized":"2024-09-18T08:07:08.237328+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:06:43.808433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_clean_scrub_stamp":"2024-09-18T08:06:43.808433+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:07:52.050872+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"69'9","reported_seq":207,"reported_epoch":149,"state":"active+clean","last_fresh":"2024-09-18T08:07:08.234152+0000","last_change":"2024-09-18T08:05:00.265939+0000","last_active":"2024-09-18T08:07:08.234152+0000","last_peered":"2024-09-18T08:07:08.234152+0000","last_clean":"2024-09-18T08:07:08.234152+0000","last_became_active":"2024-09-18T08:02:49.891567+0000","last_became_peered":"2024-09-18T08:02:49.891567+0000","last_unstale":"2024-09-18T08:07:08.234152+0000","last_undegraded":"2024-09-18T08:07:08.234152+0000","last_fullsized":"2024-09-18T08:07:08.234152+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'9","last_scrub_stamp":"2024-09-18T08:05:00.265738+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_clean_scrub_stamp":"2024-09-18T08:05:00.265738+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":17,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":206,"reported_epoch":149,"state":"active+clean","last_fresh":"2024-09-18T08:07:08.214693+0000","last_change":"2024-09-18T08:06:46.171693+0000","last_active":"2024-09-18T08:07:08.214693+0000","last_peered":"2024-09-18T08:07:08.214693+0000","last_clean":"2024-09-18T08:07:08.214693+0000","last_became_active":"2024-09-18T08:06:46.171291+0000","last_became_peered":"2024-09-18T08:06:46.171291+0000","last_unstale":"2024-09-18T08:07:08.214693+0000","last_undegraded":"2024-09-18T08:07:08.214693+0000","last_fullsized":"2024-09-18T08:07:08.214693+0000","mapping_epoch":143,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":144,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:06:13.214705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:46.860092+0000","last_clean_scrub_stamp":"2024-09-18T08:06:13.214705+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:07:42.434590+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":187,"reported_epoch":149,"state":"active+clean","last_fresh":"2024-09-18T08:07:08.222235+0000","last_change":"2024-09-18T08:05:42.061649+0000","last_active":"2024-09-18T08:07:08.222235+0000","last_peered":"2024-09-18T08:07:08.222235+0000","last_clean":"2024-09-18T08:07:08.222235+0000","last_became_active":"2024-09-18T08:02:51.907361+0000","last_became_peered":"2024-09-18T08:02:51.907361+0000","last_unstale":"2024-09-18T08:07:08.222235+0000","last_undegraded":"2024-09-18T08:07:08.222235+0000","last_fullsized":"2024-09-18T08:07:08.222235+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:05:42.061452+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:50.886830+0000","last_clean_scrub_stamp":"2024-09-18T08:05:42.061452+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":184,"reported_epoch":149,"state":"active+clean","last_fresh":"2024-09-18T08:07:08.221774+0000","last_change":"2024-09-18T08:05:07.780899+0000","last_active":"2024-09-18T08:07:08.221774+0000","last_peered":"2024-09-18T08:07:08.221774+0000","last_clean":"2024-09-18T08:07:08.221774+0000","last_became_active":"2024-09-18T08:02:53.908917+0000","last_became_peered":"2024-09-18T08:02:53.908917+0000","last_unstale":"2024-09-18T08:07:08.221774+0000","last_undegraded":"2024-09-18T08:07:08.221774+0000","last_fullsized":"2024-09-18T08:07:08.221774+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:05:07.780816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_clean_scrub_stamp":"2024-09-18T08:05:07.780816+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:07:56.898926+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.15","version":"69'9","reported_seq":215,"reported_epoch":149,"state":"active+clean","last_fresh":"2024-09-18T08:07:08.229940+0000","last_change":"2024-09-18T08:05:36.896299+0000","last_active":"2024-09-18T08:07:08.229940+0000","last_peered":"2024-09-18T08:07:08.229940+0000","last_clean":"2024-09-18T08:07:08.229940+0000","last_became_active":"2024-09-18T08:02:49.890497+0000","last_became_peered":"2024-09-18T08:02:49.890497+0000","last_unstale":"2024-09-18T08:07:08.229940+0000","last_undegraded":"2024-09-18T08:07:08.229940+0000","last_fullsized":"2024-09-18T08:07:08.229940+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'9","last_scrub_stamp":"2024-09-18T08:05:36.896238+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_clean_scrub_stamp":"2024-09-18T08:05:36.896238+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:08:07.806082+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":203,"reported_epoch":149,"state":"active+clean","last_fresh":"2024-09-18T08:07:08.226331+0000","last_change":"2024-09-18T08:06:12.437630+0000","last_active":"2024-09-18T08:07:08.226331+0000","last_peered":"2024-09-18T08:07:08.226331+0000","last_clean":"2024-09-18T08:07:08.226331+0000","last_became_active":"2024-09-18T08:02:47.875983+0000","last_became_peered":"2024-09-18T08:02:47.875983+0000","last_unstale":"2024-09-18T08:07:08.226331+0000","last_undegraded":"2024-09-18T08:07:08.226331+0000","last_fullsized":"2024-09-18T08:07:08.226331+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:06:12.437584+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:46.860092+0000","last_clean_scrub_stamp":"2024-09-18T08:06:12.437584+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:07:30.735772+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.14","version":"69'8","reported_seq":249,"reported_epoch":149,"state":"active+clean","last_fresh":"2024-09-18T08:07:08.226422+0000","last_change":"2024-09-18T08:05:01.083803+0000","last_active":"2024-09-18T08:07:08.226422+0000","last_peered":"2024-09-18T08:07:08.226422+0000","last_clean":"2024-09-18T08:07:08.226422+0000","last_became_active":"2024-09-18T08:02:51.909513+0000","last_became_peered":"2024-09-18T08:02:51.909513+0000","last_unstale":"2024-09-18T08:07:08.226422+0000","last_undegraded":"2024-09-18T08:07:08.226422+0000","last_fullsized":"2024-09-18T08:07:08.226422+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'8","last_scrub_stamp":"2024-09-18T08:05:01.082570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:50.886830+0000","last_clean_scrub_stamp":"2024-09-18T08:05:01.082570+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":17,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":192,"reported_epoch":149,"state":"active+clean","last_fresh":"2024-09-18T08:07:08.216098+0000","last_change":"2024-09-18T08:06:43.510575+0000","last_active":"2024-09-18T08:07:08.216098+0000","last_peered":"2024-09-18T08:07:08.216098+0000","last_clean":"2024-09-18T08:07:08.216098+0000","last_became_active":"2024-09-18T08:02:53.907428+0000","last_became_peered":"2024-09-18T08:02:53.907428+0000","last_unstale":"2024-09-18T08:07:08.216098+0000","last_undegraded":"2024-09-18T08:07:08.216098+0000","last_fullsized":"2024-09-18T08:07:08.216098+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:06:43.510513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_clean_scrub_stamp":"2024-09-18T08:06:43.510513+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:08:09.391404+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.14","version":"69'10","reported_seq":225,"reported_epoch":149,"state":"active+clean","last_fresh":"2024-09-18T08:07:08.231557+0000","last_change":"2024-09-18T08:04:18.172364+0000","last_active":"2024-09-18T08:07:08.231557+0000","last_peered":"2024-09-18T08:07:08.231557+0000","last_clean":"2024-09-18T08:07:08.231557+0000","last_became_active":"2024-09-18T08:04:18.150968+0000","last_became_peered":"2024-09-18T08:04:18.150968+0000","last_unstale":"2024-09-18T08:07:08.231557+0000","last_undegraded":"2024-09-18T08:07:08.231557+0000","last_fullsized":"2024-09-18T08:07:08.231557+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'10","last_scrub_stamp":"2024-09-18T08:04:16.940778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_clean_scrub_stamp":"2024-09-18T08:04:16.940778+0000","objects_scrubbed":5,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:07:57.221037+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,7],"acting":[3,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":203,"reported_epoch":149,"state":"active+clean","last_fresh":"2024-09-18T08:07:08.235015+0000","last_change":"2024-09-18T08:06:46.782184+0000","last_active":"2024-09-18T08:07:08.235015+0000","last_peered":"2024-09-18T08:07:08.235015+0000","last_clean":"2024-09-18T08:07:08.235015+0000","last_became_active":"2024-09-18T08:02:47.882065+0000","last_became_peered":"2024-09-18T08:02:47.882065+0000","last_unstale":"2024-09-18T08:07:08.235015+0000","last_undegraded":"2024-09-18T08:07:08.235015+0000","last_fullsized":"2024-09-18T08:07:08.235015+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:06:46.782109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:46.860092+0000","last_clean_scrub_stamp":"2024-09-18T08:06:46.782109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:08:11.695710+0000","scrub_duration":21,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.15","version":"69'8","reported_seq":265,"reported_epoch":149,"state":"active+clean","last_fresh":"2024-09-18T08:07:08.229858+0000","last_change":"2024-09-18T08:06:45.555513+0000","last_active":"2024-09-18T08:07:08.229858+0000","last_peered":"2024-09-18T08:07:08.229858+0000","last_clean":"2024-09-18T08:07:08.229858+0000","last_became_active":"2024-09-18T08:02:51.914135+0000","last_became_peered":"2024-09-18T08:02:51.914135+0000","last_unstale":"2024-09-18T08:07:08.229858+0000","last_undegraded":"2024-09-18T08:07:08.229858+0000","last_fullsized":"2024-09-18T08:07:08.229858+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'8","last_scrub_stamp":"2024-09-18T08:06:45.555459+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:50.886830+0000","last_clean_scrub_stamp":"2024-09-18T08:06:45.555459+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:07:48.944224+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":184,"reported_epoch":149,"state":"active+clean","last_fresh":"2024-09-18T08:07:08.234424+0000","last_change":"2024-09-18T08:05:03.302755+0000","last_active":"2024-09-18T08:07:08.234424+0000","last_peered":"2024-09-18T08:07:08.234424+0000","last_clean":"2024-09-18T08:07:08.234424+0000","last_became_active":"2024-09-18T08:02:53.912112+0000","last_became_peered":"2024-09-18T08:02:53.912112+0000","last_unstale":"2024-09-18T08:07:08.234424+0000","last_undegraded":"2024-09-18T08:07:08.234424+0000","last_fullsized":"2024-09-18T08:07:08.234424+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:05:03.302444+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_clean_scrub_stamp":"2024-09-18T08:05:03.302444+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":16,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.13","version":"69'11","reported_seq":248,"reported_epoch":149,"state":"active+clean","last_fresh":"2024-09-18T08:07:08.214912+0000","last_change":"2024-09-18T08:06:24.189911+0000","last_active":"2024-09-18T08:07:08.214912+0000","last_peered":"2024-09-18T08:07:08.214912+0000","last_clean":"2024-09-18T08:07:08.214912+0000","last_became_active":"2024-09-18T08:06:24.150800+0000","last_became_peered":"2024-09-18T08:06:24.150800+0000","last_unstale":"2024-09-18T08:07:08.214912+0000","last_undegraded":"2024-09-18T08:07:08.214912+0000","last_fullsized":"2024-09-18T08:07:08.214912+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'11","last_scrub_stamp":"2024-09-18T08:04:34.451379+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_clean_scrub_stamp":"2024-09-18T08:04:34.451379+0000","objects_scrubbed":5,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:07:35.515469+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":217,"reported_epoch":149,"state":"active+clean","last_fresh":"2024-09-18T08:07:08.215346+0000","last_change":"2024-09-18T08:06:46.502473+0000","last_active":"2024-09-18T08:07:08.215346+0000","last_peered":"2024-09-18T08:07:08.215346+0000","last_clean":"2024-09-18T08:07:08.215346+0000","last_became_active":"2024-09-18T08:06:24.145458+0000","last_became_peered":"2024-09-18T08:06:24.145458+0000","last_unstale":"2024-09-18T08:07:08.215346+0000","last_undegraded":"2024-09-18T08:07:08.215346+0000","last_fullsized":"2024-09-18T08:07:08.215346+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:06:46.502422+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:46.860092+0000","last_clean_scrub_stamp":"2024-09-18T08:06:46.502422+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:08:03.119365+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":195,"reported_epoch":149,"state":"active+clean","last_fresh":"2024-09-18T08:07:08.232577+0000","last_change":"2024-09-18T08:05:29.896334+0000","last_active":"2024-09-18T08:07:08.232577+0000","last_peered":"2024-09-18T08:07:08.232577+0000","last_clean":"2024-09-18T08:07:08.232577+0000","last_became_active":"2024-09-18T08:02:51.910752+0000","last_became_peered":"2024-09-18T08:02:51.910752+0000","last_unstale":"2024-09-18T08:07:08.232577+0000","last_undegraded":"2024-09-18T08:07:08.232577+0000","last_fullsized":"2024-09-18T08:07:08.232577+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:05:29.888927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:50.886830+0000","last_clean_scrub_stamp":"2024-09-18T08:05:29.888927+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:07:57.612901+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":192,"reported_epoch":149,"state":"active+clean","last_fresh":"2024-09-18T08:07:08.222515+0000","last_change":"2024-09-18T08:06:11.249403+0000","last_active":"2024-09-18T08:07:08.222515+0000","last_peered":"2024-09-18T08:07:08.222515+0000","last_clean":"2024-09-18T08:07:08.222515+0000","last_became_active":"2024-09-18T08:02:53.911429+0000","last_became_peered":"2024-09-18T08:02:53.911429+0000","last_unstale":"2024-09-18T08:07:08.222515+0000","last_undegraded":"2024-09-18T08:07:08.222515+0000","last_fullsized":"2024-09-18T08:07:08.222515+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:06:11.249317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_clean_scrub_stamp":"2024-09-18T08:06:11.249317+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:07:33.181678+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.12","version":"69'9","reported_seq":233,"reported_epoch":149,"state":"active+clean","last_fresh":"2024-09-18T08:07:08.233916+0000","last_change":"2024-09-18T08:06:24.152803+0000","last_active":"2024-09-18T08:07:08.233916+0000","last_peered":"2024-09-18T08:07:08.233916+0000","last_clean":"2024-09-18T08:07:08.233916+0000","last_became_active":"2024-09-18T08:06:24.132270+0000","last_became_peered":"2024-09-18T08:06:24.132270+0000","last_unstale":"2024-09-18T08:07:08.233916+0000","last_undegraded":"2024-09-18T08:07:08.233916+0000","last_fullsized":"2024-09-18T08:07:08.233916+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_clean_scrub_stamp":"2024-09-18T08:02:48.870718+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:07:28.580662+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":203,"reported_epoch":149,"state":"active+clean","last_fresh":"2024-09-18T08:07:08.237444+0000","last_change":"2024-09-18T08:05:30.820244+0000","last_active":"2024-09-18T08:07:08.237444+0000","last_peered":"2024-09-18T08:07:08.237444+0000","last_clean":"2024-09-18T08:07:08.237444+0000","last_became_active":"2024-09-18T08:02:47.882096+0000","last_became_peered":"2024-09-18T08:02:47.882096+0000","last_unstale":"2024-09-18T08:07:08.237444+0000","last_undegraded":"2024-09-18T08:07:08.237444+0000","last_fullsized":"2024-09-18T08:07:08.237444+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:05:30.820200+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:46.860092+0000","last_clean_scrub_stamp":"2024-09-18T08:05:30.820200+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:07:54.767056+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":187,"reported_epoch":149,"state":"active+clean","last_fresh":"2024-09-18T08:07:08.224301+0000","last_change":"2024-09-18T08:05:43.165522+0000","last_active":"2024-09-18T08:07:08.224301+0000","last_peered":"2024-09-18T08:07:08.224301+0000","last_clean":"2024-09-18T08:07:08.224301+0000","last_became_active":"2024-09-18T08:02:51.906277+0000","last_became_peered":"2024-09-18T08:02:51.906277+0000","last_unstale":"2024-09-18T08:07:08.224301+0000","last_undegraded":"2024-09-18T08:07:08.224301+0000","last_fullsized":"2024-09-18T08:07:08.224301+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:05:43.165258+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:50.886830+0000","last_clean_scrub_stamp":"2024-09-18T08:05:43.165258+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:07:08.715609+0000","scrub_duration":89,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.10","version":"69'1","reported_seq":180,"reported_epoch":149,"state":"active+clean","last_fresh":"2024-09-18T08:07:08.224839+0000","last_change":"2024-09-18T08:05:51.429631+0000","last_active":"2024-09-18T08:07:08.224839+0000","last_peered":"2024-09-18T08:07:08.224839+0000","last_clean":"2024-09-18T08:07:08.224839+0000","last_became_active":"2024-09-18T08:02:53.912699+0000","last_became_peered":"2024-09-18T08:02:53.912699+0000","last_unstale":"2024-09-18T08:07:08.224839+0000","last_undegraded":"2024-09-18T08:07:08.224839+0000","last_fullsized":"2024-09-18T08:07:08.224839+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_clean_scrub_stamp":"2024-09-18T08:02:52.890475+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:07:25.467898+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.11","version":"69'11","reported_seq":256,"reported_epoch":149,"state":"active+clean","last_fresh":"2024-09-18T08:07:08.224996+0000","last_change":"2024-09-18T08:06:24.154365+0000","last_active":"2024-09-18T08:07:08.224996+0000","last_peered":"2024-09-18T08:07:08.224996+0000","last_clean":"2024-09-18T08:07:08.224996+0000","last_became_active":"2024-09-18T08:06:24.132416+0000","last_became_peered":"2024-09-18T08:06:24.132416+0000","last_unstale":"2024-09-18T08:07:08.224996+0000","last_undegraded":"2024-09-18T08:07:08.224996+0000","last_fullsized":"2024-09-18T08:07:08.224996+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'11","last_scrub_stamp":"2024-09-18T08:05:13.037615+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_clean_scrub_stamp":"2024-09-18T08:05:13.037615+0000","objects_scrubbed":5,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":203,"reported_epoch":149,"state":"active+clean","last_fresh":"2024-09-18T08:07:08.231356+0000","last_change":"2024-09-18T08:06:18.168254+0000","last_active":"2024-09-18T08:07:08.231356+0000","last_peered":"2024-09-18T08:07:08.231356+0000","last_clean":"2024-09-18T08:07:08.231356+0000","last_became_active":"2024-09-18T08:02:47.878886+0000","last_became_peered":"2024-09-18T08:02:47.878886+0000","last_unstale":"2024-09-18T08:07:08.231356+0000","last_undegraded":"2024-09-18T08:07:08.231356+0000","last_fullsized":"2024-09-18T08:07:08.231356+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:06:18.168206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:46.860092+0000","last_clean_scrub_stamp":"2024-09-18T08:06:18.168206+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:07:28.641116+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":209,"reported_epoch":149,"state":"active+clean","last_fresh":"2024-09-18T08:07:08.233959+0000","last_change":"2024-09-18T08:06:24.148922+0000","last_active":"2024-09-18T08:07:08.233959+0000","last_peered":"2024-09-18T08:07:08.233959+0000","last_clean":"2024-09-18T08:07:08.233959+0000","last_became_active":"2024-09-18T08:06:24.148302+0000","last_became_peered":"2024-09-18T08:06:24.148302+0000","last_unstale":"2024-09-18T08:07:08.233959+0000","last_undegraded":"2024-09-18T08:07:08.233959+0000","last_fullsized":"2024-09-18T08:07:08.233959+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:06:15.177302+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:50.886830+0000","last_clean_scrub_stamp":"2024-09-18T08:06:15.177302+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:07:19.110873+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":197,"reported_epoch":149,"state":"active+clean","last_fresh":"2024-09-18T08:07:08.224460+0000","last_change":"2024-09-18T08:06:24.128970+0000","last_active":"2024-09-18T08:07:08.224460+0000","last_peered":"2024-09-18T08:07:08.224460+0000","last_clean":"2024-09-18T08:07:08.224460+0000","last_became_active":"2024-09-18T08:06:24.128582+0000","last_became_peered":"2024-09-18T08:06:24.128582+0000","last_unstale":"2024-09-18T08:07:08.224460+0000","last_undegraded":"2024-09-18T08:07:08.224460+0000","last_fullsized":"2024-09-18T08:07:08.224460+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:06:13.241940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_clean_scrub_stamp":"2024-09-18T08:06:13.241940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:07:27.180968+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.10","version":"69'4","reported_seq":223,"reported_epoch":149,"state":"active+clean","last_fresh":"2024-09-18T08:07:08.223652+0000","last_change":"2024-09-18T08:06:24.151823+0000","last_active":"2024-09-18T08:07:08.223652+0000","last_peered":"2024-09-18T08:07:08.223652+0000","last_clean":"2024-09-18T08:07:08.223652+0000","last_became_active":"2024-09-18T08:06:24.131309+0000","last_became_peered":"2024-09-18T08:06:24.131309+0000","last_unstale":"2024-09-18T08:07:08.223652+0000","last_undegraded":"2024-09-18T08:07:08.223652+0000","last_fullsized":"2024-09-18T08:07:08.223652+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'4","last_scrub_stamp":"2024-09-18T08:04:12.999978+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_clean_scrub_stamp":"2024-09-18T08:04:12.999978+0000","objects_scrubbed":2,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:07:35.247320+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":6,"num_read_kb":4,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":203,"reported_epoch":149,"state":"active+clean","last_fresh":"2024-09-18T08:07:08.229233+0000","last_change":"2024-09-18T08:06:10.457663+0000","last_active":"2024-09-18T08:07:08.229233+0000","last_peered":"2024-09-18T08:07:08.229233+0000","last_clean":"2024-09-18T08:07:08.229233+0000","last_became_active":"2024-09-18T08:02:47.877616+0000","last_became_peered":"2024-09-18T08:02:47.877616+0000","last_unstale":"2024-09-18T08:07:08.229233+0000","last_undegraded":"2024-09-18T08:07:08.229233+0000","last_fullsized":"2024-09-18T08:07:08.229233+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:06:10.457502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:46.860092+0000","last_clean_scrub_stamp":"2024-09-18T08:06:10.457502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:07:37.803172+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":130,"reported_epoch":149,"state":"active+clean","last_fresh":"2024-09-18T08:07:08.214407+0000","last_change":"2024-09-18T08:06:24.168175+0000","last_active":"2024-09-18T08:07:08.214407+0000","last_peered":"2024-09-18T08:07:08.214407+0000","last_clean":"2024-09-18T08:07:08.214407+0000","last_became_active":"2024-09-18T08:06:24.166698+0000","last_became_peered":"2024-09-18T08:06:24.166698+0000","last_unstale":"2024-09-18T08:07:08.214407+0000","last_undegraded":"2024-09-18T08:07:08.214407+0000","last_fullsized":"2024-09-18T08:07:08.214407+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:05:11.591275+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:50.886830+0000","last_clean_scrub_stamp":"2024-09-18T08:05:11.591275+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,7],"acting":[4,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":200,"reported_epoch":149,"state":"active+clean","last_fresh":"2024-09-18T08:07:08.234229+0000","last_change":"2024-09-18T08:06:47.791172+0000","last_active":"2024-09-18T08:07:08.234229+0000","last_peered":"2024-09-18T08:07:08.234229+0000","last_clean":"2024-09-18T08:07:08.234229+0000","last_became_active":"2024-09-18T08:02:53.909146+0000","last_became_peered":"2024-09-18T08:02:53.909146+0000","last_unstale":"2024-09-18T08:07:08.234229+0000","last_undegraded":"2024-09-18T08:07:08.234229+0000","last_fullsized":"2024-09-18T08:07:08.234229+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:06:47.791091+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_clean_scrub_stamp":"2024-09-18T08:06:47.791091+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:08:13.139318+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,4],"acting":[7,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":176,"reported_epoch":149,"state":"active+clean","last_fresh":"2024-09-18T08:07:08.237224+0000","last_change":"2024-09-18T08:02:53.907055+0000","last_active":"2024-09-18T08:07:08.237224+0000","last_peered":"2024-09-18T08:07:08.237224+0000","last_clean":"2024-09-18T08:07:08.237224+0000","last_became_active":"2024-09-18T08:02:53.906726+0000","last_became_peered":"2024-09-18T08:02:53.906726+0000","last_unstale":"2024-09-18T08:07:08.237224+0000","last_undegraded":"2024-09-18T08:07:08.237224+0000","last_fullsized":"2024-09-18T08:07:08.237224+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_clean_scrub_stamp":"2024-09-18T08:02:52.890475+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:07:27.599829+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":195,"reported_epoch":149,"state":"active+clean","last_fresh":"2024-09-18T08:07:08.223938+0000","last_change":"2024-09-18T08:05:02.063591+0000","last_active":"2024-09-18T08:07:08.223938+0000","last_peered":"2024-09-18T08:07:08.223938+0000","last_clean":"2024-09-18T08:07:08.223938+0000","last_became_active":"2024-09-18T08:02:47.878922+0000","last_became_peered":"2024-09-18T08:02:47.878922+0000","last_unstale":"2024-09-18T08:07:08.223938+0000","last_undegraded":"2024-09-18T08:07:08.223938+0000","last_fullsized":"2024-09-18T08:07:08.223938+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:05:02.063478+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:46.860092+0000","last_clean_scrub_stamp":"2024-09-18T08:05:02.063478+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"69'11","reported_seq":193,"reported_epoch":149,"state":"active+clean","last_fresh":"2024-09-18T08:07:08.230104+0000","last_change":"2024-09-18T08:06:24.164086+0000","last_active":"2024-09-18T08:07:08.230104+0000","last_peered":"2024-09-18T08:07:08.230104+0000","last_clean":"2024-09-18T08:07:08.230104+0000","last_became_active":"2024-09-18T08:06:24.135232+0000","last_became_peered":"2024-09-18T08:06:24.135232+0000","last_unstale":"2024-09-18T08:07:08.230104+0000","last_undegraded":"2024-09-18T08:07:08.230104+0000","last_fullsized":"2024-09-18T08:07:08.230104+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'11","last_scrub_stamp":"2024-09-18T08:06:12.967462+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_clean_scrub_stamp":"2024-09-18T08:06:12.967462+0000","objects_scrubbed":5,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:07:25.928591+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":19,"num_read_kb":12,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,1],"acting":[5,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":188,"reported_epoch":149,"state":"active+clean","last_fresh":"2024-09-18T08:07:08.233898+0000","last_change":"2024-09-18T08:04:26.117117+0000","last_active":"2024-09-18T08:07:08.233898+0000","last_peered":"2024-09-18T08:07:08.233898+0000","last_clean":"2024-09-18T08:07:08.233898+0000","last_became_active":"2024-09-18T08:02:51.900300+0000","last_became_peered":"2024-09-18T08:02:51.900300+0000","last_unstale":"2024-09-18T08:07:08.233898+0000","last_undegraded":"2024-09-18T08:07:08.233898+0000","last_fullsized":"2024-09-18T08:07:08.233898+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:04:26.117051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:50.886830+0000","last_clean_scrub_stamp":"2024-09-18T08:04:26.117051+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:07:55.738092+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":7,"num_pg":16,"stat_sum":{"num_bytes":1661599768,"num_objects":25355,"num_object_clones":0,"num_object_copies":76065,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":25355,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":211223,"num_write_kb":7570433,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2629,"num_bytes_recovered":172097536,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":5005717504,"data_stored":5005705288,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":87872,"ondisk_log_size":87872,"up":48,"acting":48,"num_store_stats":8},{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":416,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":71,"ondisk_log_size":71,"up":96,"acting":96,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":178,"num_object_clones":0,"num_object_copies":534,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":178,"num_whiteouts":0,"num_read":694,"num_read_kb":451,"num_write":417,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":149,"num_bytes_recovered":2508,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":417792,"data_stored":11106,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":396,"ondisk_log_size":396,"up":96,"acting":96,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":1613,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":24,"num_read_kb":24,"num_write":10,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":4839,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":6,"ondisk_log_size":6,"up":96,"acting":96,"num_store_stats":8},{"poolid":2,"num_pg":3,"stat_sum":{"num_bytes":408,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":34,"num_read_kb":29,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":38,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":8,"ondisk_log_size":8,"up":9,"acting":9,"num_store_stats":7},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":231,"num_write":113,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1771104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":2375680,"data_stored":2361472,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":79,"ondisk_log_size":79,"up":3,"acting":3,"num_store_stats":4}],"osd_stats":[{"osd":7,"up_from":56,"seq":240518168633,"num_pgs":58,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":622600,"kb_used_data":523384,"kb_used_omap":1,"kb_used_meta":99198,"kb_avail":93122552,"statfs":{"total":95995035648,"available":95357493248,"internally_reserved":0,"allocated":535945216,"data_stored":535597334,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":101579210},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":47,"seq":201863462975,"num_pgs":48,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":616312,"kb_used_data":517096,"kb_used_omap":1,"kb_used_meta":99198,"kb_avail":93128840,"statfs":{"total":95995035648,"available":95363932160,"internally_reserved":0,"allocated":529506304,"data_stored":529173745,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":101579206},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":39,"seq":167503724613,"num_pgs":52,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":495488,"kb_used_data":415728,"kb_used_omap":1,"kb_used_meta":79742,"kb_avail":93249664,"statfs":{"total":95995035648,"available":95487655936,"internally_reserved":0,"allocated":425705472,"data_stored":425365071,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":81656262},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,1,2],"upper_bound":64},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":32,"seq":137438953546,"num_pgs":63,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":622224,"kb_used_data":523008,"kb_used_omap":1,"kb_used_meta":99198,"kb_avail":93122928,"statfs":{"total":95995035648,"available":95357878272,"internally_reserved":0,"allocated":535560192,"data_stored":535204087,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":101579205},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":26,"seq":111669149777,"num_pgs":65,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1088340,"kb_used_data":922500,"kb_used_omap":1,"kb_used_meta":165822,"kb_avail":92656812,"statfs":{"total":95995035648,"available":94880575488,"internally_reserved":0,"allocated":944640000,"data_stored":944279092,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":169802184},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411415,"num_pgs":44,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":746556,"kb_used_data":629292,"kb_used_omap":1,"kb_used_meta":117246,"kb_avail":92998596,"statfs":{"total":95995035648,"available":95230562304,"internally_reserved":0,"allocated":644395008,"data_stored":644058699,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":120060357},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,5,0,0,2],"upper_bound":32},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574939,"num_pgs":60,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":751792,"kb_used_data":626208,"kb_used_omap":1,"kb_used_meta":125566,"kb_avail":92993360,"statfs":{"total":95995035648,"available":95225200640,"internally_reserved":0,"allocated":641236992,"data_stored":640849332,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":128580037},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,1,1],"upper_bound":8},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738466,"num_pgs":54,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":964952,"kb_used_data":835144,"kb_used_omap":1,"kb_used_meta":129726,"kb_avail":92780200,"statfs":{"total":95995035648,"available":95006924800,"internally_reserved":0,"allocated":855187456,"data_stored":854823929,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":132839877},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1131,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1521,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":436,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":46,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1177,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":436,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":1898,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1128,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":1898,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1474,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":1100,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1034,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1254,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":403,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":416,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":403,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":825233408,"data_stored":825229336,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":624758784,"data_stored":624754712,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":629800960,"data_stored":629800960,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":932773888,"data_stored":932773888,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":524156928,"data_stored":524156928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":418119680,"data_stored":418119680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":520556544,"data_stored":520552472,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":530317312,"data_stored":530317312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T08:07:16.983 INFO:tasks.cephadm.ceph_manager.ceph:increase pool size by 10 2024-09-18T08:07:16.983 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid 779349ca-7593-11ef-bceb-c7b262605968 -- ceph osd pool set .mgr pg_num 11 2024-09-18T08:07:16.983 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0 2024-09-18T08:07:17.025 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:07:17.025 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:07:17.025 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0' 2024-09-18T08:07:17.126 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6 2024-09-18T08:07:17.151 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:07:17.152 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:07:17.152 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6' 2024-09-18T08:07:17.253 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5 2024-09-18T08:07:17.279 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:07:17.280 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:07:17.280 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5' 2024-09-18T08:07:17.381 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7 2024-09-18T08:07:17.406 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:07:17 smithi050 ceph-mon[32371]: from='client.15108 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-18T08:07:17.406 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:07:17 smithi050 ceph-mon[32371]: 5.14 scrub starts 2024-09-18T08:07:17.406 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:07:17 smithi050 ceph-mon[32371]: 5.14 scrub ok 2024-09-18T08:07:17.406 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:07:17 smithi050 ceph-mon[32371]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-18T08:07:17.406 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:07:17 smithi050 ceph-mon[32371]: Cluster is now healthy 2024-09-18T08:07:17.406 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:07:17 smithi050 ceph-mon[32371]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-18T08:07:17.406 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:07:17 smithi050 ceph-mon[32371]: osdmap e150: 8 total, 8 up, 8 in 2024-09-18T08:07:17.410 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:07:17.411 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:07:17.411 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7' 2024-09-18T08:07:17.512 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2 2024-09-18T08:07:17.554 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:07:17.555 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:07:17.555 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2' 2024-09-18T08:07:17.656 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3 2024-09-18T08:07:17.703 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:07:17.703 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:07:17.704 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3' 2024-09-18T08:07:17.731 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:07:17 smithi007 ceph-mon[29459]: from='client.15108 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-18T08:07:17.731 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:07:17 smithi007 ceph-mon[29459]: 5.14 scrub starts 2024-09-18T08:07:17.731 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:07:17 smithi007 ceph-mon[29459]: 5.14 scrub ok 2024-09-18T08:07:17.731 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:07:17 smithi007 ceph-mon[29459]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-18T08:07:17.731 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:07:17 smithi007 ceph-mon[29459]: Cluster is now healthy 2024-09-18T08:07:17.731 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:07:17 smithi007 ceph-mon[29459]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-18T08:07:17.731 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:07:17 smithi007 ceph-mon[29459]: osdmap e150: 8 total, 8 up, 8 in 2024-09-18T08:07:17.732 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:07:17 smithi007 ceph-mon[39249]: from='client.15108 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-18T08:07:17.732 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:07:17 smithi007 ceph-mon[39249]: 5.14 scrub starts 2024-09-18T08:07:17.732 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:07:17 smithi007 ceph-mon[39249]: 5.14 scrub ok 2024-09-18T08:07:17.732 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:07:17 smithi007 ceph-mon[39249]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-18T08:07:17.732 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:07:17 smithi007 ceph-mon[39249]: Cluster is now healthy 2024-09-18T08:07:17.732 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:07:17 smithi007 ceph-mon[39249]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-18T08:07:17.732 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:07:17 smithi007 ceph-mon[39249]: osdmap e150: 8 total, 8 up, 8 in 2024-09-18T08:07:17.805 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2 2024-09-18T08:07:17.843 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:07:17.880 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:07:17.880 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2' 2024-09-18T08:07:17.981 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4 2024-09-18T08:07:18.009 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:07:18.009 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:07:18.010 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4' 2024-09-18T08:07:18.112 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4 2024-09-18T08:07:18.142 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:07:18.143 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:07:18.143 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4' 2024-09-18T08:07:18.244 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5 2024-09-18T08:07:18.270 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:07:18.271 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:07:18.272 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5' 2024-09-18T08:07:18.372 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5 2024-09-18T08:07:18.400 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:07:18.400 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:07:18.401 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5' 2024-09-18T08:07:18.501 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4 2024-09-18T08:07:18.526 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:07:18.527 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:07:18.527 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4' 2024-09-18T08:07:18.553 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:07:18 smithi007 ceph-mon[29459]: 6.e scrub starts 2024-09-18T08:07:18.553 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:07:18 smithi007 ceph-mon[29459]: 6.e scrub ok 2024-09-18T08:07:18.553 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:07:18 smithi007 ceph-mon[29459]: pgmap v200: 148 pgs: 1 active+clean+scrubbing, 147 active+clean; 1.5 GiB data, 5.6 GiB used, 710 GiB / 715 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 1.05k op/s 2024-09-18T08:07:18.553 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:07:18 smithi007 ceph-mon[29459]: 3.18 scrub starts 2024-09-18T08:07:18.553 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:07:18 smithi007 ceph-mon[29459]: 3.18 scrub ok 2024-09-18T08:07:18.553 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:07:18 smithi007 ceph-mon[29459]: from='mgr.24470 172.21.15.7:0/2594390261' entity='mgr.y' 2024-09-18T08:07:18.553 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:07:18 smithi007 ceph-mon[29459]: from='mgr.24470 172.21.15.7:0/2594390261' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-18T08:07:18.554 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:07:18 smithi007 ceph-mon[39249]: 6.e scrub starts 2024-09-18T08:07:18.554 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:07:18 smithi007 ceph-mon[39249]: 6.e scrub ok 2024-09-18T08:07:18.554 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:07:18 smithi007 ceph-mon[39249]: pgmap v200: 148 pgs: 1 active+clean+scrubbing, 147 active+clean; 1.5 GiB data, 5.6 GiB used, 710 GiB / 715 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 1.05k op/s 2024-09-18T08:07:18.554 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:07:18 smithi007 ceph-mon[39249]: 3.18 scrub starts 2024-09-18T08:07:18.554 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:07:18 smithi007 ceph-mon[39249]: 3.18 scrub ok 2024-09-18T08:07:18.554 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:07:18 smithi007 ceph-mon[39249]: from='mgr.24470 172.21.15.7:0/2594390261' entity='mgr.y' 2024-09-18T08:07:18.554 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:07:18 smithi007 ceph-mon[39249]: from='mgr.24470 172.21.15.7:0/2594390261' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-18T08:07:18.628 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6 2024-09-18T08:07:18.657 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:07:18.657 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:07:18.657 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6' 2024-09-18T08:07:18.658 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:07:18 smithi050 ceph-mon[32371]: 6.e scrub starts 2024-09-18T08:07:18.658 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:07:18 smithi050 ceph-mon[32371]: 6.e scrub ok 2024-09-18T08:07:18.658 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:07:18 smithi050 ceph-mon[32371]: pgmap v200: 148 pgs: 1 active+clean+scrubbing, 147 active+clean; 1.5 GiB data, 5.6 GiB used, 710 GiB / 715 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 1.05k op/s 2024-09-18T08:07:18.658 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:07:18 smithi050 ceph-mon[32371]: 3.18 scrub starts 2024-09-18T08:07:18.658 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:07:18 smithi050 ceph-mon[32371]: 3.18 scrub ok 2024-09-18T08:07:18.658 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:07:18 smithi050 ceph-mon[32371]: from='mgr.24470 172.21.15.7:0/2594390261' entity='mgr.y' 2024-09-18T08:07:18.658 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:07:18 smithi050 ceph-mon[32371]: from='mgr.24470 172.21.15.7:0/2594390261' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-18T08:07:18.722 INFO:teuthology.orchestra.run.smithi007.stdout:ERROR: (22) Invalid argument 2024-09-18T08:07:18.722 INFO:teuthology.orchestra.run.smithi007.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-18T08:07:18.758 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7 2024-09-18T08:07:18.786 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:07:18.787 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:07:18.787 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7' 2024-09-18T08:07:18.888 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1 2024-09-18T08:07:18.936 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:07:18.937 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:07:18.937 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1' 2024-09-18T08:07:19.038 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2 2024-09-18T08:07:19.078 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:07:19.079 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:07:19.079 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2' 2024-09-18T08:07:19.180 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7 2024-09-18T08:07:19.206 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:07:19.207 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:07:19.207 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7' 2024-09-18T08:07:19.302 INFO:journalctl@ceph.iscsi.iscsi.a.smithi050.stdout:Sep 18 08:07:18 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-iscsi-iscsi-a[57782]: debug there is no tcmu-runner data available 2024-09-18T08:07:19.308 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1 2024-09-18T08:07:19.348 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:07:19.349 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:07:19.350 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1' 2024-09-18T08:07:19.403 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:07:19 smithi007 ceph-mon[29459]: 5.11 scrub starts 2024-09-18T08:07:19.404 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:07:19 smithi007 ceph-mon[29459]: 5.11 scrub ok 2024-09-18T08:07:19.404 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:07:19 smithi007 ceph-mon[29459]: 4.11 scrub starts 2024-09-18T08:07:19.404 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:07:19 smithi007 ceph-mon[29459]: 4.11 scrub ok 2024-09-18T08:07:19.404 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:07:19 smithi007 ceph-mon[29459]: 5.19 scrub starts 2024-09-18T08:07:19.404 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:07:19 smithi007 ceph-mon[29459]: 5.19 scrub ok 2024-09-18T08:07:19.404 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:07:19 smithi007 ceph-mon[39249]: 5.11 scrub starts 2024-09-18T08:07:19.404 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:07:19 smithi007 ceph-mon[39249]: 5.11 scrub ok 2024-09-18T08:07:19.404 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:07:19 smithi007 ceph-mon[39249]: 4.11 scrub starts 2024-09-18T08:07:19.405 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:07:19 smithi007 ceph-mon[39249]: 4.11 scrub ok 2024-09-18T08:07:19.405 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:07:19 smithi007 ceph-mon[39249]: 5.19 scrub starts 2024-09-18T08:07:19.405 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:07:19 smithi007 ceph-mon[39249]: 5.19 scrub ok 2024-09-18T08:07:19.413 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/779349ca-7593-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T08:07:19.451 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0 2024-09-18T08:07:19.452 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/779349ca-7593-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T08:07:19.460 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid 779349ca-7593-11ef-bceb-c7b262605968 -- ceph daemon osd.1 dump_historic_ops 2024-09-18T08:07:19.502 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:07:19.503 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:07:19.503 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0' 2024-09-18T08:07:19.604 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5 2024-09-18T08:07:19.633 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:07:19.633 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:07:19.634 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5' 2024-09-18T08:07:19.656 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:07:19 smithi050 ceph-mon[32371]: 5.11 scrub starts 2024-09-18T08:07:19.656 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:07:19 smithi050 ceph-mon[32371]: 5.11 scrub ok 2024-09-18T08:07:19.656 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:07:19 smithi050 ceph-mon[32371]: 4.11 scrub starts 2024-09-18T08:07:19.656 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:07:19 smithi050 ceph-mon[32371]: 4.11 scrub ok 2024-09-18T08:07:19.656 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:07:19 smithi050 ceph-mon[32371]: 5.19 scrub starts 2024-09-18T08:07:19.656 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:07:19 smithi050 ceph-mon[32371]: 5.19 scrub ok 2024-09-18T08:07:19.735 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2 2024-09-18T08:07:19.807 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:07:19.808 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:07:19.808 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2' 2024-09-18T08:07:19.909 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3 2024-09-18T08:07:19.950 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:07:19.951 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:07:19.951 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3' 2024-09-18T08:07:20.052 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7 2024-09-18T08:07:20.074 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:07:20.075 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:07:20.075 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7' 2024-09-18T08:07:20.176 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6 2024-09-18T08:07:20.204 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:07:20.204 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:07:20.205 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6' 2024-09-18T08:07:20.306 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1 2024-09-18T08:07:20.353 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:07:20.353 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:07:20.370 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1' 2024-09-18T08:07:20.427 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid 779349ca-7593-11ef-bceb-c7b262605968 -- ceph osd set noscrub 2024-09-18T08:07:20.465 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:07:20 smithi007 ceph-mon[39249]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-18T08:07:20.465 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:07:20 smithi007 ceph-mon[39249]: pgmap v201: 148 pgs: 148 active+clean; 1.3 GiB data, 4.8 GiB used, 710 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 971 op/s 2024-09-18T08:07:20.465 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:07:20 smithi007 ceph-mon[39249]: 4.17 scrub starts 2024-09-18T08:07:20.466 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:07:20 smithi007 ceph-mon[39249]: 4.17 scrub ok 2024-09-18T08:07:20.466 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:07:20 smithi007 ceph-mon[29459]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-18T08:07:20.466 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:07:20 smithi007 ceph-mon[29459]: pgmap v201: 148 pgs: 148 active+clean; 1.3 GiB data, 4.8 GiB used, 710 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 971 op/s 2024-09-18T08:07:20.466 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:07:20 smithi007 ceph-mon[29459]: 4.17 scrub starts 2024-09-18T08:07:20.466 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:07:20 smithi007 ceph-mon[29459]: 4.17 scrub ok 2024-09-18T08:07:20.471 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1 2024-09-18T08:07:20.526 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:07:20.526 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:07:20.527 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1' 2024-09-18T08:07:20.628 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5 2024-09-18T08:07:20.652 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:07:20.653 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:07:20.653 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5' 2024-09-18T08:07:20.655 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:07:20 smithi050 ceph-mon[32371]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-18T08:07:20.656 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:07:20 smithi050 ceph-mon[32371]: pgmap v201: 148 pgs: 148 active+clean; 1.3 GiB data, 4.8 GiB used, 710 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 971 op/s 2024-09-18T08:07:20.656 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:07:20 smithi050 ceph-mon[32371]: 4.17 scrub starts 2024-09-18T08:07:20.656 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:07:20 smithi050 ceph-mon[32371]: 4.17 scrub ok 2024-09-18T08:07:20.745 INFO:teuthology.orchestra.run.smithi007.stdout:osd.0: {} 2024-09-18T08:07:20.745 INFO:teuthology.orchestra.run.smithi007.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-18T08:07:20.754 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0 2024-09-18T08:07:20.784 INFO:teuthology.orchestra.run.smithi007.stdout:osd.1: {} 2024-09-18T08:07:20.785 INFO:teuthology.orchestra.run.smithi007.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-18T08:07:20.820 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:07:20.820 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:07:20.821 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0' 2024-09-18T08:07:20.833 INFO:teuthology.orchestra.run.smithi007.stdout:osd.2: {} 2024-09-18T08:07:20.833 INFO:teuthology.orchestra.run.smithi007.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-18T08:07:20.877 INFO:teuthology.orchestra.run.smithi007.stdout:osd.3: {} 2024-09-18T08:07:20.877 INFO:teuthology.orchestra.run.smithi007.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-18T08:07:20.922 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1 2024-09-18T08:07:20.922 INFO:teuthology.orchestra.run.smithi007.stdout:osd.4: {} 2024-09-18T08:07:20.923 INFO:teuthology.orchestra.run.smithi007.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-18T08:07:20.954 INFO:teuthology.orchestra.run.smithi007.stdout:osd.5: {} 2024-09-18T08:07:20.954 INFO:teuthology.orchestra.run.smithi007.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-18T08:07:20.974 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:07:20.975 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:07:20.976 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1' 2024-09-18T08:07:20.990 INFO:teuthology.orchestra.run.smithi007.stdout:osd.6: {} 2024-09-18T08:07:20.990 INFO:teuthology.orchestra.run.smithi007.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-18T08:07:20.994 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/779349ca-7593-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T08:07:21.000 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/779349ca-7593-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T08:07:21.036 INFO:teuthology.orchestra.run.smithi007.stdout:osd.7: {} 2024-09-18T08:07:21.036 INFO:teuthology.orchestra.run.smithi007.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-18T08:07:21.076 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0 2024-09-18T08:07:21.126 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:07:21.127 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:07:21.127 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0' 2024-09-18T08:07:21.228 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4 2024-09-18T08:07:21.254 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:07:21.254 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:07:21.255 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4' 2024-09-18T08:07:21.355 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1 2024-09-18T08:07:21.396 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:07:21.396 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:07:21.397 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1' 2024-09-18T08:07:21.498 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0 2024-09-18T08:07:21.537 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:07:21.538 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:07:21.538 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0' 2024-09-18T08:07:21.570 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:07:21 smithi007 ceph-mon[39249]: 6.d scrub starts 2024-09-18T08:07:21.571 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:07:21 smithi007 ceph-mon[39249]: 6.d scrub ok 2024-09-18T08:07:21.571 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:07:21 smithi007 ceph-mon[39249]: 4.1d scrub starts 2024-09-18T08:07:21.571 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:07:21 smithi007 ceph-mon[39249]: 4.1d scrub ok 2024-09-18T08:07:21.571 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:07:21 smithi007 ceph-mon[39249]: from='client.? 172.21.15.7:0/3999874317' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-18T08:07:21.571 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:07:21 smithi007 ceph-mon[29459]: 6.d scrub starts 2024-09-18T08:07:21.571 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:07:21 smithi007 ceph-mon[29459]: 6.d scrub ok 2024-09-18T08:07:21.571 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:07:21 smithi007 ceph-mon[29459]: 4.1d scrub starts 2024-09-18T08:07:21.571 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:07:21 smithi007 ceph-mon[29459]: 4.1d scrub ok 2024-09-18T08:07:21.571 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:07:21 smithi007 ceph-mon[29459]: from='client.? 172.21.15.7:0/3999874317' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-18T08:07:21.640 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5 2024-09-18T08:07:21.655 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:07:21 smithi050 ceph-mon[32371]: 6.d scrub starts 2024-09-18T08:07:21.656 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:07:21 smithi050 ceph-mon[32371]: 6.d scrub ok 2024-09-18T08:07:21.656 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:07:21 smithi050 ceph-mon[32371]: 4.1d scrub starts 2024-09-18T08:07:21.656 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:07:21 smithi050 ceph-mon[32371]: 4.1d scrub ok 2024-09-18T08:07:21.656 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:07:21 smithi050 ceph-mon[32371]: from='client.? 172.21.15.7:0/3999874317' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-18T08:07:21.665 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:07:21.666 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:07:21.666 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5' 2024-09-18T08:07:21.767 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6 2024-09-18T08:07:21.792 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:07:21.793 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:07:21.794 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6' 2024-09-18T08:07:21.895 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1 2024-09-18T08:07:21.936 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:07:21.936 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:07:21.936 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1' 2024-09-18T08:07:22.037 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3 2024-09-18T08:07:22.078 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:07:22.078 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:07:22.079 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3' 2024-09-18T08:07:22.180 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3 2024-09-18T08:07:22.222 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:07:22.223 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:07:22.223 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3' 2024-09-18T08:07:22.324 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0 2024-09-18T08:07:22.343 INFO:teuthology.orchestra.run.smithi007.stderr:set pool 1 pg_num to 11 2024-09-18T08:07:22.365 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:07:22.366 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:07:22.366 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0' 2024-09-18T08:07:22.395 INFO:teuthology.orchestra.run.smithi007.stdout:{ 2024-09-18T08:07:22.395 INFO:teuthology.orchestra.run.smithi007.stdout: "size": 20, 2024-09-18T08:07:22.395 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 600, 2024-09-18T08:07:22.395 INFO:teuthology.orchestra.run.smithi007.stdout: "ops": [ 2024-09-18T08:07:22.396 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:07:22.396 INFO:teuthology.orchestra.run.smithi007.stdout: "description": "osd_op(client.14805.0:65051 7.d 7:bcc5331c:::benchmark_data_smithi007_73155_object65050:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e101)", 2024-09-18T08:07:22.396 INFO:teuthology.orchestra.run.smithi007.stdout: "initiated_at": "2024-09-18T08:04:42.307415+0000", 2024-09-18T08:07:22.396 INFO:teuthology.orchestra.run.smithi007.stdout: "age": 160.02648623799999, 2024-09-18T08:07:22.396 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.99828821199999995, 2024-09-18T08:07:22.396 INFO:teuthology.orchestra.run.smithi007.stdout: "continuous": false, 2024-09-18T08:07:22.396 INFO:teuthology.orchestra.run.smithi007.stdout: "type_data": { 2024-09-18T08:07:22.396 INFO:teuthology.orchestra.run.smithi007.stdout: "flag_point": "reached pg", 2024-09-18T08:07:22.397 INFO:teuthology.orchestra.run.smithi007.stdout: "client_info": { 2024-09-18T08:07:22.397 INFO:teuthology.orchestra.run.smithi007.stdout: "client": "client.14805", 2024-09-18T08:07:22.397 INFO:teuthology.orchestra.run.smithi007.stdout: "client_addr": "172.21.15.7:0/640320436", 2024-09-18T08:07:22.397 INFO:teuthology.orchestra.run.smithi007.stdout: "tid": 65051 2024-09-18T08:07:22.397 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:07:22.397 INFO:teuthology.orchestra.run.smithi007.stdout: "events": [ 2024-09-18T08:07:22.397 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:07:22.397 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "initiated", 2024-09-18T08:07:22.398 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:04:42.307415+0000", 2024-09-18T08:07:22.398 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-09-18T08:07:22.398 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:07:22.398 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:07:22.398 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "header_read", 2024-09-18T08:07:22.398 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:04:42.307415+0000", 2024-09-18T08:07:22.399 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-09-18T08:07:22.399 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:07:22.399 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:07:22.399 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "throttled", 2024-09-18T08:07:22.400 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:04:42.307417+0000", 2024-09-18T08:07:22.400 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.55e-06 2024-09-18T08:07:22.400 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:07:22.400 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:07:22.400 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "all_read", 2024-09-18T08:07:22.400 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:04:42.307438+0000", 2024-09-18T08:07:22.400 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 2.1478999999999999e-05 2024-09-18T08:07:22.401 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:07:22.401 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:07:22.401 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "dispatched", 2024-09-18T08:07:22.401 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:04:42.307454+0000", 2024-09-18T08:07:22.401 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.6413999999999999e-05 2024-09-18T08:07:22.401 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:07:22.401 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:07:22.401 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "queued_for_pg", 2024-09-18T08:07:22.401 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:04:42.307463+0000", 2024-09-18T08:07:22.402 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 8.1499999999999999e-06 2024-09-18T08:07:22.402 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:07:22.402 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:07:22.402 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-09-18T08:07:22.402 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:04:42.307632+0000", 2024-09-18T08:07:22.402 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.000169037 2024-09-18T08:07:22.402 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:07:22.402 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:07:22.402 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "waiting for peered", 2024-09-18T08:07:22.403 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:04:42.307641+0000", 2024-09-18T08:07:22.403 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 9.1269999999999992e-06 2024-09-18T08:07:22.403 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:07:22.403 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:07:22.403 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-09-18T08:07:22.403 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:04:43.304675+0000", 2024-09-18T08:07:22.403 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.997034541 2024-09-18T08:07:22.403 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:07:22.404 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:07:22.404 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "done", 2024-09-18T08:07:22.404 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:04:43.305703+0000", 2024-09-18T08:07:22.404 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.001027914 2024-09-18T08:07:22.404 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-09-18T08:07:22.405 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-09-18T08:07:22.405 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-09-18T08:07:22.405 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:07:22.405 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:07:22.406 INFO:teuthology.orchestra.run.smithi007.stdout: "description": "osd_op(client.14805.0:65057 7.d 7:bffbe46c:::benchmark_data_smithi007_73155_object65056:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e101)", 2024-09-18T08:07:22.406 INFO:teuthology.orchestra.run.smithi007.stdout: "initiated_at": "2024-09-18T08:04:42.307482+0000", 2024-09-18T08:07:22.406 INFO:teuthology.orchestra.run.smithi007.stdout: "age": 160.02641957, 2024-09-18T08:07:22.406 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.99879061499999999, 2024-09-18T08:07:22.406 INFO:teuthology.orchestra.run.smithi007.stdout: "continuous": false, 2024-09-18T08:07:22.407 INFO:teuthology.orchestra.run.smithi007.stdout: "type_data": { 2024-09-18T08:07:22.407 INFO:teuthology.orchestra.run.smithi007.stdout: "flag_point": "reached pg", 2024-09-18T08:07:22.407 INFO:teuthology.orchestra.run.smithi007.stdout: "client_info": { 2024-09-18T08:07:22.407 INFO:teuthology.orchestra.run.smithi007.stdout: "client": "client.14805", 2024-09-18T08:07:22.407 INFO:teuthology.orchestra.run.smithi007.stdout: "client_addr": "172.21.15.7:0/640320436", 2024-09-18T08:07:22.407 INFO:teuthology.orchestra.run.smithi007.stdout: "tid": 65057 2024-09-18T08:07:22.408 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:07:22.408 INFO:teuthology.orchestra.run.smithi007.stdout: "events": [ 2024-09-18T08:07:22.408 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:07:22.408 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "initiated", 2024-09-18T08:07:22.409 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:04:42.307482+0000", 2024-09-18T08:07:22.409 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-09-18T08:07:22.409 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:07:22.409 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:07:22.409 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "header_read", 2024-09-18T08:07:22.409 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:04:42.307482+0000", 2024-09-18T08:07:22.410 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-09-18T08:07:22.410 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:07:22.410 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:07:22.410 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "throttled", 2024-09-18T08:07:22.410 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:04:42.307482+0000", 2024-09-18T08:07:22.410 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 5.7400000000000003e-07 2024-09-18T08:07:22.411 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:07:22.411 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:07:22.411 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "all_read", 2024-09-18T08:07:22.411 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:04:42.307501+0000", 2024-09-18T08:07:22.411 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.9202999999999999e-05 2024-09-18T08:07:22.412 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:07:22.412 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:07:22.412 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "dispatched", 2024-09-18T08:07:22.412 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:04:42.307508+0000", 2024-09-18T08:07:22.412 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 6.7179999999999999e-06 2024-09-18T08:07:22.412 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:07:22.413 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:07:22.416 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "queued_for_pg", 2024-09-18T08:07:22.416 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:04:42.307514+0000", 2024-09-18T08:07:22.416 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 5.8089999999999998e-06 2024-09-18T08:07:22.416 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:07:22.416 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:07:22.416 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-09-18T08:07:22.417 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:04:42.307661+0000", 2024-09-18T08:07:22.417 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.00014706699999999999 2024-09-18T08:07:22.417 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:07:22.417 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:07:22.417 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "waiting for peered", 2024-09-18T08:07:22.417 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:04:42.307667+0000", 2024-09-18T08:07:22.417 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 5.9329999999999998e-06 2024-09-18T08:07:22.417 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:07:22.417 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:07:22.417 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-09-18T08:07:22.417 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:04:43.306196+0000", 2024-09-18T08:07:22.418 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.99852927199999997 2024-09-18T08:07:22.418 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:07:22.418 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:07:22.418 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "done", 2024-09-18T08:07:22.418 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:04:43.306272+0000", 2024-09-18T08:07:22.418 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 7.6038999999999995e-05 2024-09-18T08:07:22.418 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-09-18T08:07:22.418 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-09-18T08:07:22.418 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-09-18T08:07:22.418 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:07:22.418 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:07:22.419 INFO:teuthology.orchestra.run.smithi007.stdout: "description": "osd_op(client.14805.0:65062 7.d 7:b1e686a2:::benchmark_data_smithi007_73155_object65061:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e101)", 2024-09-18T08:07:22.419 INFO:teuthology.orchestra.run.smithi007.stdout: "initiated_at": "2024-09-18T08:04:42.307522+0000", 2024-09-18T08:07:22.419 INFO:teuthology.orchestra.run.smithi007.stdout: "age": 160.02637965700001, 2024-09-18T08:07:22.419 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.99897097700000004, 2024-09-18T08:07:22.419 INFO:teuthology.orchestra.run.smithi007.stdout: "continuous": false, 2024-09-18T08:07:22.419 INFO:teuthology.orchestra.run.smithi007.stdout: "type_data": { 2024-09-18T08:07:22.419 INFO:teuthology.orchestra.run.smithi007.stdout: "flag_point": "reached pg", 2024-09-18T08:07:22.419 INFO:teuthology.orchestra.run.smithi007.stdout: "client_info": { 2024-09-18T08:07:22.419 INFO:teuthology.orchestra.run.smithi007.stdout: "client": "client.14805", 2024-09-18T08:07:22.419 INFO:teuthology.orchestra.run.smithi007.stdout: "client_addr": "172.21.15.7:0/640320436", 2024-09-18T08:07:22.419 INFO:teuthology.orchestra.run.smithi007.stdout: "tid": 65062 2024-09-18T08:07:22.419 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:07:22.420 INFO:teuthology.orchestra.run.smithi007.stdout: "events": [ 2024-09-18T08:07:22.420 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:07:22.420 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "initiated", 2024-09-18T08:07:22.420 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:04:42.307522+0000", 2024-09-18T08:07:22.420 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-09-18T08:07:22.420 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:07:22.420 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:07:22.420 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "header_read", 2024-09-18T08:07:22.420 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:04:42.307522+0000", 2024-09-18T08:07:22.420 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-09-18T08:07:22.420 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:07:22.420 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:07:22.420 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "throttled", 2024-09-18T08:07:22.421 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:04:42.307522+0000", 2024-09-18T08:07:22.421 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 4.6600000000000002e-07 2024-09-18T08:07:22.421 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:07:22.421 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:07:22.421 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "all_read", 2024-09-18T08:07:22.421 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:04:42.307540+0000", 2024-09-18T08:07:22.421 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.8094999999999999e-05 2024-09-18T08:07:22.421 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:07:22.421 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:07:22.421 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "dispatched", 2024-09-18T08:07:22.421 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:04:42.307546+0000", 2024-09-18T08:07:22.421 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 5.772e-06 2024-09-18T08:07:22.421 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:07:22.421 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:07:22.422 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "queued_for_pg", 2024-09-18T08:07:22.422 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:04:42.307551+0000", 2024-09-18T08:07:22.422 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 4.7260000000000002e-06 2024-09-18T08:07:22.422 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:07:22.422 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:07:22.422 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-09-18T08:07:22.422 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:04:42.307685+0000", 2024-09-18T08:07:22.422 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.00013448900000000001 2024-09-18T08:07:22.422 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:07:22.422 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:07:22.422 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "waiting for peered", 2024-09-18T08:07:22.423 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:04:42.307691+0000", 2024-09-18T08:07:22.423 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 5.7830000000000004e-06 2024-09-18T08:07:22.423 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:07:22.423 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:07:22.423 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-09-18T08:07:22.423 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:04:43.306322+0000", 2024-09-18T08:07:22.423 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.99863141 2024-09-18T08:07:22.423 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:07:22.423 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:07:22.423 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "done", 2024-09-18T08:07:22.423 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:04:43.306493+0000", 2024-09-18T08:07:22.423 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.00017023599999999999 2024-09-18T08:07:22.424 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-09-18T08:07:22.424 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-09-18T08:07:22.424 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-09-18T08:07:22.424 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:07:22.424 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:07:22.424 INFO:teuthology.orchestra.run.smithi007.stdout: "description": "osd_op(client.14805.0:65063 7.d 7:b037f784:::benchmark_data_smithi007_73155_object65062:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e101)", 2024-09-18T08:07:22.424 INFO:teuthology.orchestra.run.smithi007.stdout: "initiated_at": "2024-09-18T08:04:42.307557+0000", 2024-09-18T08:07:22.424 INFO:teuthology.orchestra.run.smithi007.stdout: "age": 160.02634378900001, 2024-09-18T08:07:22.424 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.99907621800000002, 2024-09-18T08:07:22.424 INFO:teuthology.orchestra.run.smithi007.stdout: "continuous": false, 2024-09-18T08:07:22.424 INFO:teuthology.orchestra.run.smithi007.stdout: "type_data": { 2024-09-18T08:07:22.424 INFO:teuthology.orchestra.run.smithi007.stdout: "flag_point": "reached pg", 2024-09-18T08:07:22.425 INFO:teuthology.orchestra.run.smithi007.stdout: "client_info": { 2024-09-18T08:07:22.425 INFO:teuthology.orchestra.run.smithi007.stdout: "client": "client.14805", 2024-09-18T08:07:22.425 INFO:teuthology.orchestra.run.smithi007.stdout: "client_addr": "172.21.15.7:0/640320436", 2024-09-18T08:07:22.425 INFO:teuthology.orchestra.run.smithi007.stdout: "tid": 65063 2024-09-18T08:07:22.425 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:07:22.425 INFO:teuthology.orchestra.run.smithi007.stdout: "events": [ 2024-09-18T08:07:22.425 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:07:22.425 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "initiated", 2024-09-18T08:07:22.425 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:04:42.307557+0000", 2024-09-18T08:07:22.425 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-09-18T08:07:22.425 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:07:22.425 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:07:22.426 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "header_read", 2024-09-18T08:07:22.426 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:04:42.307557+0000", 2024-09-18T08:07:22.426 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-09-18T08:07:22.426 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:07:22.426 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:07:22.426 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "throttled", 2024-09-18T08:07:22.426 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:04:42.307558+0000", 2024-09-18T08:07:22.426 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 4.2899999999999999e-07 2024-09-18T08:07:22.426 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:07:22.426 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:07:22.426 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "all_read", 2024-09-18T08:07:22.426 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:04:42.307576+0000", 2024-09-18T08:07:22.426 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.8413000000000001e-05 2024-09-18T08:07:22.426 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:07:22.427 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:07:22.427 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "dispatched", 2024-09-18T08:07:22.427 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:04:42.307582+0000", 2024-09-18T08:07:22.427 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 5.4070000000000001e-06 2024-09-18T08:07:22.427 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:07:22.427 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:07:22.427 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "queued_for_pg", 2024-09-18T08:07:22.427 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:04:42.307586+0000", 2024-09-18T08:07:22.427 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 4.7119999999999999e-06 2024-09-18T08:07:22.427 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:07:22.427 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:07:22.427 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-09-18T08:07:22.427 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:04:42.307709+0000", 2024-09-18T08:07:22.428 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.00012292 2024-09-18T08:07:22.428 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:07:22.428 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:07:22.428 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "waiting for peered", 2024-09-18T08:07:22.428 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:04:42.307715+0000", 2024-09-18T08:07:22.428 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 5.6679999999999998e-06 2024-09-18T08:07:22.428 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:07:22.428 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:07:22.428 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-09-18T08:07:22.428 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:04:43.306547+0000", 2024-09-18T08:07:22.428 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.99883231400000005 2024-09-18T08:07:22.428 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:07:22.429 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:07:22.429 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "done", 2024-09-18T08:07:22.429 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:04:43.306634+0000", 2024-09-18T08:07:22.429 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 8.6354999999999996e-05 2024-09-18T08:07:22.429 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-09-18T08:07:22.429 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-09-18T08:07:22.429 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-09-18T08:07:22.429 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:07:22.429 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:07:22.429 INFO:teuthology.orchestra.run.smithi007.stdout: "description": "osd_op(client.14805.0:65074 7.d 7:b74c2f3b:::benchmark_data_smithi007_73155_object65073:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e101)", 2024-09-18T08:07:22.429 INFO:teuthology.orchestra.run.smithi007.stdout: "initiated_at": "2024-09-18T08:04:42.316968+0000", 2024-09-18T08:07:22.429 INFO:teuthology.orchestra.run.smithi007.stdout: "age": 160.01693361400001, 2024-09-18T08:07:22.429 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.99769510299999997, 2024-09-18T08:07:22.430 INFO:teuthology.orchestra.run.smithi007.stdout: "continuous": false, 2024-09-18T08:07:22.430 INFO:teuthology.orchestra.run.smithi007.stdout: "type_data": { 2024-09-18T08:07:22.430 INFO:teuthology.orchestra.run.smithi007.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T08:07:22.430 INFO:teuthology.orchestra.run.smithi007.stdout: "client_info": { 2024-09-18T08:07:22.430 INFO:teuthology.orchestra.run.smithi007.stdout: "client": "client.14805", 2024-09-18T08:07:22.430 INFO:teuthology.orchestra.run.smithi007.stdout: "client_addr": "172.21.15.7:0/640320436", 2024-09-18T08:07:22.430 INFO:teuthology.orchestra.run.smithi007.stdout: "tid": 65074 2024-09-18T08:07:22.430 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:07:22.430 INFO:teuthology.orchestra.run.smithi007.stdout: "events": [ 2024-09-18T08:07:22.430 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:07:22.430 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "initiated", 2024-09-18T08:07:22.430 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:04:42.316968+0000", 2024-09-18T08:07:22.430 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-09-18T08:07:22.430 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:07:22.431 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:07:22.431 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "header_read", 2024-09-18T08:07:22.431 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:04:42.316968+0000", 2024-09-18T08:07:22.431 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-09-18T08:07:22.431 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:07:22.431 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:07:22.431 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "throttled", 2024-09-18T08:07:22.431 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:04:42.316969+0000", 2024-09-18T08:07:22.431 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.6649999999999999e-06 2024-09-18T08:07:22.431 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:07:22.431 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:07:22.432 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "all_read", 2024-09-18T08:07:22.432 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:04:42.316999+0000", 2024-09-18T08:07:22.432 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 2.9268000000000002e-05 2024-09-18T08:07:22.432 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:07:22.432 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:07:22.432 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "dispatched", 2024-09-18T08:07:22.432 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:04:42.317017+0000", 2024-09-18T08:07:22.432 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.8785999999999999e-05 2024-09-18T08:07:22.432 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:07:22.432 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:07:22.432 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "queued_for_pg", 2024-09-18T08:07:22.432 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:04:42.317027+0000", 2024-09-18T08:07:22.432 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 9.6930000000000002e-06 2024-09-18T08:07:22.432 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:07:22.433 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:07:22.433 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-09-18T08:07:22.433 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:04:42.317759+0000", 2024-09-18T08:07:22.433 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.00073177599999999995 2024-09-18T08:07:22.433 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:07:22.433 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:07:22.433 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "waiting for peered", 2024-09-18T08:07:22.433 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:04:42.317768+0000", 2024-09-18T08:07:22.433 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 9.6709999999999994e-06 2024-09-18T08:07:22.433 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:07:22.433 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:07:22.434 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-09-18T08:07:22.434 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:04:43.306668+0000", 2024-09-18T08:07:22.434 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.98889915699999997 2024-09-18T08:07:22.434 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:07:22.434 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:07:22.434 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "started", 2024-09-18T08:07:22.434 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:04:43.307139+0000", 2024-09-18T08:07:22.434 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.00047175500000000003 2024-09-18T08:07:22.434 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:07:22.434 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:07:22.434 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "waiting for subops from 0,6", 2024-09-18T08:07:22.434 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:04:43.307270+0000", 2024-09-18T08:07:22.434 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.00013037500000000001 2024-09-18T08:07:22.435 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:07:22.435 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:07:22.435 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:07:22.435 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:04:43.314522+0000", 2024-09-18T08:07:22.435 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.0072521779999999997 2024-09-18T08:07:22.435 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:07:22.435 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:07:22.435 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:07:22.435 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:04:43.314561+0000", 2024-09-18T08:07:22.435 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 3.9164e-05 2024-09-18T08:07:22.435 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:07:22.435 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:07:22.435 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "op_commit", 2024-09-18T08:07:22.435 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:04:43.314571+0000", 2024-09-18T08:07:22.436 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.0166000000000001e-05 2024-09-18T08:07:22.436 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:07:22.436 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:07:22.436 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "commit_sent", 2024-09-18T08:07:22.436 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:04:43.314621+0000", 2024-09-18T08:07:22.436 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 4.9388000000000002e-05 2024-09-18T08:07:22.436 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:07:22.436 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:07:22.436 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "done", 2024-09-18T08:07:22.436 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:04:43.314663+0000", 2024-09-18T08:07:22.436 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 4.2061000000000003e-05 2024-09-18T08:07:22.436 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-09-18T08:07:22.436 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-09-18T08:07:22.437 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-09-18T08:07:22.437 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:07:22.437 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:07:22.437 INFO:teuthology.orchestra.run.smithi007.stdout: "description": "osd_op(client.14805.0:65107 7.d 7:b155ef47:::benchmark_data_smithi007_73155_object65106:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e101)", 2024-09-18T08:07:22.437 INFO:teuthology.orchestra.run.smithi007.stdout: "initiated_at": "2024-09-18T08:04:42.342022+0000", 2024-09-18T08:07:22.437 INFO:teuthology.orchestra.run.smithi007.stdout: "age": 159.99187920099999, 2024-09-18T08:07:22.437 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.97630728899999997, 2024-09-18T08:07:22.437 INFO:teuthology.orchestra.run.smithi007.stdout: "continuous": false, 2024-09-18T08:07:22.437 INFO:teuthology.orchestra.run.smithi007.stdout: "type_data": { 2024-09-18T08:07:22.437 INFO:teuthology.orchestra.run.smithi007.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T08:07:22.437 INFO:teuthology.orchestra.run.smithi007.stdout: "client_info": { 2024-09-18T08:07:22.437 INFO:teuthology.orchestra.run.smithi007.stdout: "client": "client.14805", 2024-09-18T08:07:22.438 INFO:teuthology.orchestra.run.smithi007.stdout: "client_addr": "172.21.15.7:0/640320436", 2024-09-18T08:07:22.438 INFO:teuthology.orchestra.run.smithi007.stdout: "tid": 65107 2024-09-18T08:07:22.438 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:07:22.438 INFO:teuthology.orchestra.run.smithi007.stdout: "events": [ 2024-09-18T08:07:22.438 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:07:22.438 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "initiated", 2024-09-18T08:07:22.438 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:04:42.342022+0000", 2024-09-18T08:07:22.438 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-09-18T08:07:22.438 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:07:22.438 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:07:22.438 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "header_read", 2024-09-18T08:07:22.438 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:04:42.342022+0000", 2024-09-18T08:07:22.438 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-09-18T08:07:22.439 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:07:22.439 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:07:22.439 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "throttled", 2024-09-18T08:07:22.439 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:04:42.342024+0000", 2024-09-18T08:07:22.439 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.649e-06 2024-09-18T08:07:22.439 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:07:22.439 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:07:22.439 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "all_read", 2024-09-18T08:07:22.439 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:04:42.342045+0000", 2024-09-18T08:07:22.439 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 2.1177000000000001e-05 2024-09-18T08:07:22.439 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:07:22.439 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:07:22.439 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "dispatched", 2024-09-18T08:07:22.440 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:04:42.342062+0000", 2024-09-18T08:07:22.440 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.7482999999999998e-05 2024-09-18T08:07:22.440 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:07:22.440 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:07:22.440 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "queued_for_pg", 2024-09-18T08:07:22.440 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:04:42.342071+0000", 2024-09-18T08:07:22.440 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 8.5399999999999996e-06 2024-09-18T08:07:22.440 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:07:22.440 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:07:22.440 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-09-18T08:07:22.440 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:04:42.342129+0000", 2024-09-18T08:07:22.440 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 5.8038000000000002e-05 2024-09-18T08:07:22.440 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:07:22.441 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:07:22.441 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "waiting for peered", 2024-09-18T08:07:22.441 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:04:42.342138+0000", 2024-09-18T08:07:22.441 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 9.4210000000000001e-06 2024-09-18T08:07:22.441 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:07:22.441 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:07:22.441 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-09-18T08:07:22.441 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:04:43.309666+0000", 2024-09-18T08:07:22.441 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.96752762000000003 2024-09-18T08:07:22.441 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:07:22.441 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:07:22.441 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "started", 2024-09-18T08:07:22.441 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:04:43.309777+0000", 2024-09-18T08:07:22.442 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.000110593 2024-09-18T08:07:22.442 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:07:22.442 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:07:22.442 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "waiting for subops from 0,6", 2024-09-18T08:07:22.442 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:04:43.309916+0000", 2024-09-18T08:07:22.442 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.00013957800000000001 2024-09-18T08:07:22.442 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:07:22.442 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:07:22.442 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "op_commit", 2024-09-18T08:07:22.442 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:04:43.314670+0000", 2024-09-18T08:07:22.442 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.0047540890000000004 2024-09-18T08:07:22.442 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:07:22.442 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:07:22.442 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:07:22.443 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:04:43.314703+0000", 2024-09-18T08:07:22.443 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 3.3111999999999997e-05 2024-09-18T08:07:22.443 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:07:22.443 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:07:22.443 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:07:22.443 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:04:43.318171+0000", 2024-09-18T08:07:22.443 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.0034679630000000001 2024-09-18T08:07:22.443 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:07:22.443 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:07:22.443 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "commit_sent", 2024-09-18T08:07:22.443 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:04:43.318285+0000", 2024-09-18T08:07:22.443 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.000113361 2024-09-18T08:07:22.443 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:07:22.443 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:07:22.444 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "done", 2024-09-18T08:07:22.444 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:04:43.318329+0000", 2024-09-18T08:07:22.444 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 4.4665e-05 2024-09-18T08:07:22.444 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-09-18T08:07:22.444 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-09-18T08:07:22.444 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-09-18T08:07:22.444 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:07:22.444 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:07:22.444 INFO:teuthology.orchestra.run.smithi007.stdout: "description": "osd_op(client.14805.0:65122 7.d 7:b67397cb:::benchmark_data_smithi007_73155_object65121:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e101)", 2024-09-18T08:07:22.444 INFO:teuthology.orchestra.run.smithi007.stdout: "initiated_at": "2024-09-18T08:04:42.361528+0000", 2024-09-18T08:07:22.444 INFO:teuthology.orchestra.run.smithi007.stdout: "age": 159.97237278599999, 2024-09-18T08:07:22.444 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.95716828700000001, 2024-09-18T08:07:22.444 INFO:teuthology.orchestra.run.smithi007.stdout: "continuous": false, 2024-09-18T08:07:22.445 INFO:teuthology.orchestra.run.smithi007.stdout: "type_data": { 2024-09-18T08:07:22.445 INFO:teuthology.orchestra.run.smithi007.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T08:07:22.445 INFO:teuthology.orchestra.run.smithi007.stdout: "client_info": { 2024-09-18T08:07:22.445 INFO:teuthology.orchestra.run.smithi007.stdout: "client": "client.14805", 2024-09-18T08:07:22.445 INFO:teuthology.orchestra.run.smithi007.stdout: "client_addr": "172.21.15.7:0/640320436", 2024-09-18T08:07:22.445 INFO:teuthology.orchestra.run.smithi007.stdout: "tid": 65122 2024-09-18T08:07:22.445 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:07:22.445 INFO:teuthology.orchestra.run.smithi007.stdout: "events": [ 2024-09-18T08:07:22.445 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:07:22.445 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "initiated", 2024-09-18T08:07:22.445 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:04:42.361528+0000", 2024-09-18T08:07:22.445 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-09-18T08:07:22.446 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:07:22.446 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:07:22.446 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "header_read", 2024-09-18T08:07:22.446 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:04:42.361528+0000", 2024-09-18T08:07:22.446 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-09-18T08:07:22.446 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:07:22.446 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:07:22.446 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "throttled", 2024-09-18T08:07:22.446 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:04:42.361530+0000", 2024-09-18T08:07:22.446 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.9199999999999998e-06 2024-09-18T08:07:22.446 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:07:22.446 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:07:22.446 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "all_read", 2024-09-18T08:07:22.447 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:04:42.361556+0000", 2024-09-18T08:07:22.447 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 2.5559999999999999e-05 2024-09-18T08:07:22.447 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:07:22.447 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:07:22.447 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "dispatched", 2024-09-18T08:07:22.447 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:04:42.361573+0000", 2024-09-18T08:07:22.447 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.732e-05 2024-09-18T08:07:22.447 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:07:22.447 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:07:22.447 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "queued_for_pg", 2024-09-18T08:07:22.447 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:04:42.361583+0000", 2024-09-18T08:07:22.447 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 9.9590000000000007e-06 2024-09-18T08:07:22.447 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:07:22.447 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:07:22.448 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-09-18T08:07:22.448 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:04:42.361644+0000", 2024-09-18T08:07:22.448 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 6.0309000000000001e-05 2024-09-18T08:07:22.448 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:07:22.448 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:07:22.448 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "waiting for peered", 2024-09-18T08:07:22.448 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:04:42.361651+0000", 2024-09-18T08:07:22.448 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 7.3289999999999996e-06 2024-09-18T08:07:22.448 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:07:22.448 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:07:22.448 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-09-18T08:07:22.448 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:04:43.312147+0000", 2024-09-18T08:07:22.448 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.95049641600000001 2024-09-18T08:07:22.449 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:07:22.449 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:07:22.449 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "started", 2024-09-18T08:07:22.449 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:04:43.312263+0000", 2024-09-18T08:07:22.449 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.000115937 2024-09-18T08:07:22.449 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:07:22.449 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:07:22.449 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "waiting for subops from 0,6", 2024-09-18T08:07:22.449 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:04:43.312402+0000", 2024-09-18T08:07:22.449 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.00013872299999999999 2024-09-18T08:07:22.449 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:07:22.449 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:07:22.449 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "op_commit", 2024-09-18T08:07:22.449 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:04:43.314713+0000", 2024-09-18T08:07:22.450 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.0023107539999999999 2024-09-18T08:07:22.450 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:07:22.450 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:07:22.450 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:07:22.450 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:04:43.315175+0000", 2024-09-18T08:07:22.450 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.00046251199999999999 2024-09-18T08:07:22.450 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:07:22.450 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:07:22.450 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:07:22.450 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:04:43.318571+0000", 2024-09-18T08:07:22.450 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.0033958410000000001 2024-09-18T08:07:22.450 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:07:22.450 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:07:22.451 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "commit_sent", 2024-09-18T08:07:22.451 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:04:43.318658+0000", 2024-09-18T08:07:22.451 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 8.7299000000000006e-05 2024-09-18T08:07:22.451 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:07:22.451 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:07:22.451 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "done", 2024-09-18T08:07:22.451 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:04:43.318697+0000", 2024-09-18T08:07:22.451 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 3.8408000000000002e-05 2024-09-18T08:07:22.451 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-09-18T08:07:22.451 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-09-18T08:07:22.451 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-09-18T08:07:22.451 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:07:22.451 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:07:22.452 INFO:teuthology.orchestra.run.smithi007.stdout: "description": "osd_op(client.14805.0:66035 7.d 7:bb688bf3:::benchmark_data_smithi007_73155_object66034:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e102)", 2024-09-18T08:07:22.452 INFO:teuthology.orchestra.run.smithi007.stdout: "initiated_at": "2024-09-18T08:04:43.964941+0000", 2024-09-18T08:07:22.452 INFO:teuthology.orchestra.run.smithi007.stdout: "age": 158.36895977099999, 2024-09-18T08:07:22.452 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.084580944000000005, 2024-09-18T08:07:22.452 INFO:teuthology.orchestra.run.smithi007.stdout: "continuous": false, 2024-09-18T08:07:22.452 INFO:teuthology.orchestra.run.smithi007.stdout: "type_data": { 2024-09-18T08:07:22.452 INFO:teuthology.orchestra.run.smithi007.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T08:07:22.452 INFO:teuthology.orchestra.run.smithi007.stdout: "client_info": { 2024-09-18T08:07:22.452 INFO:teuthology.orchestra.run.smithi007.stdout: "client": "client.14805", 2024-09-18T08:07:22.452 INFO:teuthology.orchestra.run.smithi007.stdout: "client_addr": "172.21.15.7:0/640320436", 2024-09-18T08:07:22.452 INFO:teuthology.orchestra.run.smithi007.stdout: "tid": 66035 2024-09-18T08:07:22.452 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:07:22.452 INFO:teuthology.orchestra.run.smithi007.stdout: "events": [ 2024-09-18T08:07:22.453 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:07:22.453 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "initiated", 2024-09-18T08:07:22.453 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:04:43.964941+0000", 2024-09-18T08:07:22.453 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-09-18T08:07:22.453 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:07:22.453 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:07:22.453 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "header_read", 2024-09-18T08:07:22.453 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:04:43.964941+0000", 2024-09-18T08:07:22.453 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-09-18T08:07:22.453 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:07:22.453 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:07:22.453 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "throttled", 2024-09-18T08:07:22.453 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:04:43.964943+0000", 2024-09-18T08:07:22.454 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.7600000000000001e-06 2024-09-18T08:07:22.454 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:07:22.454 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:07:22.454 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "all_read", 2024-09-18T08:07:22.454 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:04:43.964969+0000", 2024-09-18T08:07:22.454 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 2.6012000000000001e-05 2024-09-18T08:07:22.454 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:07:22.454 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:07:22.454 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "dispatched", 2024-09-18T08:07:22.454 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:04:43.964986+0000", 2024-09-18T08:07:22.454 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.7252e-05 2024-09-18T08:07:22.454 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:07:22.454 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:07:22.455 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "queued_for_pg", 2024-09-18T08:07:22.455 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:04:43.964995+0000", 2024-09-18T08:07:22.455 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 8.4519999999999997e-06 2024-09-18T08:07:22.455 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:07:22.455 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:07:22.455 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-09-18T08:07:22.455 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:04:43.965180+0000", 2024-09-18T08:07:22.455 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.000184721 2024-09-18T08:07:22.455 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:07:22.455 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:07:22.455 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "started", 2024-09-18T08:07:22.455 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:04:43.965340+0000", 2024-09-18T08:07:22.455 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.00016080399999999999 2024-09-18T08:07:22.455 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:07:22.456 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:07:22.456 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "waiting for subops from 0,6", 2024-09-18T08:07:22.456 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:04:43.965468+0000", 2024-09-18T08:07:22.456 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.000127737 2024-09-18T08:07:22.456 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:07:22.456 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:07:22.456 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:07:22.456 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:04:43.968337+0000", 2024-09-18T08:07:22.456 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.0028687360000000002 2024-09-18T08:07:22.456 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:07:22.456 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:07:22.456 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "op_commit", 2024-09-18T08:07:22.457 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:04:43.968435+0000", 2024-09-18T08:07:22.457 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 9.8481999999999997e-05 2024-09-18T08:07:22.457 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:07:22.457 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:07:22.457 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:07:22.457 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:04:44.049428+0000", 2024-09-18T08:07:22.457 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.080992475999999994 2024-09-18T08:07:22.457 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:07:22.457 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:07:22.457 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "commit_sent", 2024-09-18T08:07:22.457 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:04:44.049482+0000", 2024-09-18T08:07:22.457 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 5.4465000000000001e-05 2024-09-18T08:07:22.457 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:07:22.458 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:07:22.458 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "done", 2024-09-18T08:07:22.458 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:04:44.049522+0000", 2024-09-18T08:07:22.458 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 4.0046999999999998e-05 2024-09-18T08:07:22.458 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-09-18T08:07:22.458 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-09-18T08:07:22.458 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-09-18T08:07:22.458 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:07:22.458 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:07:22.458 INFO:teuthology.orchestra.run.smithi007.stdout: "description": "osd_op(client.14805.0:94192 7.d 7:bbf6f9c5:::benchmark_data_smithi007_73155_object94191:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e108)", 2024-09-18T08:07:22.458 INFO:teuthology.orchestra.run.smithi007.stdout: "initiated_at": "2024-09-18T08:05:09.973790+0000", 2024-09-18T08:07:22.458 INFO:teuthology.orchestra.run.smithi007.stdout: "age": 132.36011154299999, 2024-09-18T08:07:22.458 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.098483875999999998, 2024-09-18T08:07:22.459 INFO:teuthology.orchestra.run.smithi007.stdout: "continuous": false, 2024-09-18T08:07:22.459 INFO:teuthology.orchestra.run.smithi007.stdout: "type_data": { 2024-09-18T08:07:22.459 INFO:teuthology.orchestra.run.smithi007.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T08:07:22.459 INFO:teuthology.orchestra.run.smithi007.stdout: "client_info": { 2024-09-18T08:07:22.459 INFO:teuthology.orchestra.run.smithi007.stdout: "client": "client.14805", 2024-09-18T08:07:22.459 INFO:teuthology.orchestra.run.smithi007.stdout: "client_addr": "172.21.15.7:0/640320436", 2024-09-18T08:07:22.459 INFO:teuthology.orchestra.run.smithi007.stdout: "tid": 94192 2024-09-18T08:07:22.459 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:07:22.459 INFO:teuthology.orchestra.run.smithi007.stdout: "events": [ 2024-09-18T08:07:22.459 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:07:22.459 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "initiated", 2024-09-18T08:07:22.459 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:05:09.973790+0000", 2024-09-18T08:07:22.459 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-09-18T08:07:22.459 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:07:22.459 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:07:22.460 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "header_read", 2024-09-18T08:07:22.460 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:05:09.973790+0000", 2024-09-18T08:07:22.460 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-09-18T08:07:22.460 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:07:22.460 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:07:22.460 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "throttled", 2024-09-18T08:07:22.460 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:05:09.973791+0000", 2024-09-18T08:07:22.460 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.266e-06 2024-09-18T08:07:22.460 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:07:22.460 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:07:22.460 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "all_read", 2024-09-18T08:07:22.460 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:05:09.973810+0000", 2024-09-18T08:07:22.460 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.8915000000000001e-05 2024-09-18T08:07:22.461 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:07:22.461 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:07:22.461 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "dispatched", 2024-09-18T08:07:22.461 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:05:09.973818+0000", 2024-09-18T08:07:22.461 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 8.3100000000000001e-06 2024-09-18T08:07:22.461 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:07:22.461 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:07:22.461 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "queued_for_pg", 2024-09-18T08:07:22.461 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:05:09.973823+0000", 2024-09-18T08:07:22.461 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 5.0309999999999998e-06 2024-09-18T08:07:22.461 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:07:22.461 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:07:22.461 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-09-18T08:07:22.461 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:05:09.973870+0000", 2024-09-18T08:07:22.462 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 4.6934000000000002e-05 2024-09-18T08:07:22.462 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:07:22.462 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:07:22.462 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "started", 2024-09-18T08:07:22.462 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:05:09.973950+0000", 2024-09-18T08:07:22.462 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 7.9812000000000001e-05 2024-09-18T08:07:22.462 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:07:22.462 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:07:22.462 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "waiting for subops from 0,6", 2024-09-18T08:07:22.462 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:05:09.974036+0000", 2024-09-18T08:07:22.462 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 8.5747000000000004e-05 2024-09-18T08:07:22.462 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:07:22.462 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:07:22.462 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "op_commit", 2024-09-18T08:07:22.463 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:05:09.975856+0000", 2024-09-18T08:07:22.463 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.0018199069999999999 2024-09-18T08:07:22.463 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:07:22.463 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:07:22.463 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:07:22.463 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:05:09.976276+0000", 2024-09-18T08:07:22.463 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.00042034899999999998 2024-09-18T08:07:22.463 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:07:22.463 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:07:22.463 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:07:22.463 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:05:10.071890+0000", 2024-09-18T08:07:22.463 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.095614499000000006 2024-09-18T08:07:22.463 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:07:22.463 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:07:22.464 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "commit_sent", 2024-09-18T08:07:22.464 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:05:10.072014+0000", 2024-09-18T08:07:22.464 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.000123103 2024-09-18T08:07:22.464 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:07:22.464 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:07:22.464 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "done", 2024-09-18T08:07:22.464 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:05:10.072274+0000", 2024-09-18T08:07:22.464 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.00026000300000000002 2024-09-18T08:07:22.464 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-09-18T08:07:22.464 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-09-18T08:07:22.464 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-09-18T08:07:22.464 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:07:22.464 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:07:22.464 INFO:teuthology.orchestra.run.smithi007.stdout: "description": "osd_op(client.14805.0:110380 7.b 7:dd836458:::benchmark_data_smithi007_73155_object110379:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e118)", 2024-09-18T08:07:22.465 INFO:teuthology.orchestra.run.smithi007.stdout: "initiated_at": "2024-09-18T08:05:26.996232+0000", 2024-09-18T08:07:22.465 INFO:teuthology.orchestra.run.smithi007.stdout: "age": 115.337669441, 2024-09-18T08:07:22.465 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.10527898099999999, 2024-09-18T08:07:22.465 INFO:teuthology.orchestra.run.smithi007.stdout: "continuous": false, 2024-09-18T08:07:22.465 INFO:teuthology.orchestra.run.smithi007.stdout: "type_data": { 2024-09-18T08:07:22.465 INFO:teuthology.orchestra.run.smithi007.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T08:07:22.465 INFO:teuthology.orchestra.run.smithi007.stdout: "client_info": { 2024-09-18T08:07:22.465 INFO:teuthology.orchestra.run.smithi007.stdout: "client": "client.14805", 2024-09-18T08:07:22.465 INFO:teuthology.orchestra.run.smithi007.stdout: "client_addr": "172.21.15.7:0/640320436", 2024-09-18T08:07:22.465 INFO:teuthology.orchestra.run.smithi007.stdout: "tid": 110380 2024-09-18T08:07:22.465 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:07:22.465 INFO:teuthology.orchestra.run.smithi007.stdout: "events": [ 2024-09-18T08:07:22.465 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:07:22.466 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "initiated", 2024-09-18T08:07:22.466 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:05:26.996232+0000", 2024-09-18T08:07:22.466 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-09-18T08:07:22.466 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:07:22.466 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:07:22.466 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "header_read", 2024-09-18T08:07:22.466 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:05:26.996232+0000", 2024-09-18T08:07:22.466 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-09-18T08:07:22.466 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:07:22.466 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:07:22.466 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "throttled", 2024-09-18T08:07:22.466 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:05:26.996234+0000", 2024-09-18T08:07:22.466 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 2.1500000000000002e-06 2024-09-18T08:07:22.466 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:07:22.467 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:07:22.467 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "all_read", 2024-09-18T08:07:22.467 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:05:26.996262+0000", 2024-09-18T08:07:22.467 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 2.8328999999999999e-05 2024-09-18T08:07:22.467 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:07:22.467 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:07:22.467 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "dispatched", 2024-09-18T08:07:22.467 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:05:26.996283+0000", 2024-09-18T08:07:22.467 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 2.1112e-05 2024-09-18T08:07:22.467 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:07:22.467 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:07:22.467 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "queued_for_pg", 2024-09-18T08:07:22.467 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:05:26.996293+0000", 2024-09-18T08:07:22.468 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 9.3680000000000008e-06 2024-09-18T08:07:22.468 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:07:22.468 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:07:22.468 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-09-18T08:07:22.468 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:05:26.996358+0000", 2024-09-18T08:07:22.468 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 6.5665000000000002e-05 2024-09-18T08:07:22.468 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:07:22.468 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:07:22.468 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "started", 2024-09-18T08:07:22.468 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:05:26.996488+0000", 2024-09-18T08:07:22.468 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.00012954399999999999 2024-09-18T08:07:22.468 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:07:22.468 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:07:22.469 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "waiting for subops from 5,6", 2024-09-18T08:07:22.469 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:05:26.996634+0000", 2024-09-18T08:07:22.469 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.00014654300000000001 2024-09-18T08:07:22.469 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:07:22.469 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:07:22.469 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:07:22.469 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:05:27.000732+0000", 2024-09-18T08:07:22.469 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.0040972229999999997 2024-09-18T08:07:22.469 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:07:22.469 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:07:22.469 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:07:22.469 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:05:27.000772+0000", 2024-09-18T08:07:22.469 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 4.0679000000000001e-05 2024-09-18T08:07:22.469 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:07:22.470 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:07:22.470 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "op_commit", 2024-09-18T08:07:22.470 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:05:27.101419+0000", 2024-09-18T08:07:22.470 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.100646395 2024-09-18T08:07:22.470 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:07:22.470 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:07:22.470 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "commit_sent", 2024-09-18T08:07:22.470 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:05:27.101459+0000", 2024-09-18T08:07:22.470 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 4.0256000000000002e-05 2024-09-18T08:07:22.470 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:07:22.470 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:07:22.470 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "done", 2024-09-18T08:07:22.470 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:05:27.101511+0000", 2024-09-18T08:07:22.470 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 5.1717e-05 2024-09-18T08:07:22.471 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-09-18T08:07:22.471 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-09-18T08:07:22.471 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-09-18T08:07:22.471 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:07:22.471 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:07:22.471 INFO:teuthology.orchestra.run.smithi007.stdout: "description": "osd_op(client.14805.0:110390 7.d 7:b2cd4d32:::benchmark_data_smithi007_73155_object110389:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e118)", 2024-09-18T08:07:22.471 INFO:teuthology.orchestra.run.smithi007.stdout: "initiated_at": "2024-09-18T08:05:27.004463+0000", 2024-09-18T08:07:22.471 INFO:teuthology.orchestra.run.smithi007.stdout: "age": 115.329438185, 2024-09-18T08:07:22.471 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.092401360000000002, 2024-09-18T08:07:22.471 INFO:teuthology.orchestra.run.smithi007.stdout: "continuous": false, 2024-09-18T08:07:22.471 INFO:teuthology.orchestra.run.smithi007.stdout: "type_data": { 2024-09-18T08:07:22.471 INFO:teuthology.orchestra.run.smithi007.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T08:07:22.471 INFO:teuthology.orchestra.run.smithi007.stdout: "client_info": { 2024-09-18T08:07:22.472 INFO:teuthology.orchestra.run.smithi007.stdout: "client": "client.14805", 2024-09-18T08:07:22.472 INFO:teuthology.orchestra.run.smithi007.stdout: "client_addr": "172.21.15.7:0/640320436", 2024-09-18T08:07:22.472 INFO:teuthology.orchestra.run.smithi007.stdout: "tid": 110390 2024-09-18T08:07:22.472 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:07:22.472 INFO:teuthology.orchestra.run.smithi007.stdout: "events": [ 2024-09-18T08:07:22.472 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:07:22.472 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "initiated", 2024-09-18T08:07:22.472 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:05:27.004463+0000", 2024-09-18T08:07:22.472 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-09-18T08:07:22.473 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:07:22.473 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:07:22.473 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "header_read", 2024-09-18T08:07:22.473 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:05:27.004463+0000", 2024-09-18T08:07:22.473 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-09-18T08:07:22.473 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:07:22.473 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:07:22.473 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "throttled", 2024-09-18T08:07:22.473 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:05:27.004465+0000", 2024-09-18T08:07:22.473 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.77e-06 2024-09-18T08:07:22.473 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:07:22.473 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:07:22.473 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "all_read", 2024-09-18T08:07:22.474 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:05:27.004488+0000", 2024-09-18T08:07:22.474 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 2.3674999999999999e-05 2024-09-18T08:07:22.474 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:07:22.474 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:07:22.474 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "dispatched", 2024-09-18T08:07:22.474 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:05:27.004500+0000", 2024-09-18T08:07:22.474 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.1851000000000001e-05 2024-09-18T08:07:22.474 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:07:22.474 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:07:22.474 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "queued_for_pg", 2024-09-18T08:07:22.474 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:05:27.004507+0000", 2024-09-18T08:07:22.474 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 6.8759999999999999e-06 2024-09-18T08:07:22.474 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:07:22.474 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:07:22.475 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-09-18T08:07:22.475 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:05:27.007030+0000", 2024-09-18T08:07:22.475 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.0025223310000000001 2024-09-18T08:07:22.475 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:07:22.475 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:07:22.475 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "started", 2024-09-18T08:07:22.475 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:05:27.007148+0000", 2024-09-18T08:07:22.475 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.000118047 2024-09-18T08:07:22.476 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:07:22.476 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:07:22.476 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "waiting for subops from 0,6", 2024-09-18T08:07:22.476 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:05:27.007288+0000", 2024-09-18T08:07:22.476 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.000140016 2024-09-18T08:07:22.476 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:07:22.476 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:07:22.476 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "op_commit", 2024-09-18T08:07:22.476 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:05:27.089508+0000", 2024-09-18T08:07:22.476 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.082220286000000004 2024-09-18T08:07:22.476 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:07:22.476 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:07:22.476 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:07:22.477 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:05:27.092302+0000", 2024-09-18T08:07:22.477 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.0027944760000000002 2024-09-18T08:07:22.477 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:07:22.477 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:07:22.477 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:07:22.477 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:05:27.096741+0000", 2024-09-18T08:07:22.477 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.0044385020000000004 2024-09-18T08:07:22.477 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:07:22.477 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:07:22.477 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "commit_sent", 2024-09-18T08:07:22.478 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:05:27.096807+0000", 2024-09-18T08:07:22.478 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 6.6112000000000007e-05 2024-09-18T08:07:22.478 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:07:22.478 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:07:22.478 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "done", 2024-09-18T08:07:22.478 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:05:27.096864+0000", 2024-09-18T08:07:22.478 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 5.7417999999999998e-05 2024-09-18T08:07:22.478 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-09-18T08:07:22.478 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-09-18T08:07:22.478 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-09-18T08:07:22.478 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:07:22.478 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:07:22.478 INFO:teuthology.orchestra.run.smithi007.stdout: "description": "osd_op(client.14805.0:110393 7.b 7:d22e8b52:::benchmark_data_smithi007_73155_object110392:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e118)", 2024-09-18T08:07:22.478 INFO:teuthology.orchestra.run.smithi007.stdout: "initiated_at": "2024-09-18T08:05:27.004525+0000", 2024-09-18T08:07:22.479 INFO:teuthology.orchestra.run.smithi007.stdout: "age": 115.32937636, 2024-09-18T08:07:22.479 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.097118418999999997, 2024-09-18T08:07:22.479 INFO:teuthology.orchestra.run.smithi007.stdout: "continuous": false, 2024-09-18T08:07:22.479 INFO:teuthology.orchestra.run.smithi007.stdout: "type_data": { 2024-09-18T08:07:22.479 INFO:teuthology.orchestra.run.smithi007.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T08:07:22.479 INFO:teuthology.orchestra.run.smithi007.stdout: "client_info": { 2024-09-18T08:07:22.479 INFO:teuthology.orchestra.run.smithi007.stdout: "client": "client.14805", 2024-09-18T08:07:22.479 INFO:teuthology.orchestra.run.smithi007.stdout: "client_addr": "172.21.15.7:0/640320436", 2024-09-18T08:07:22.480 INFO:teuthology.orchestra.run.smithi007.stdout: "tid": 110393 2024-09-18T08:07:22.480 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:07:22.480 INFO:teuthology.orchestra.run.smithi007.stdout: "events": [ 2024-09-18T08:07:22.480 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:07:22.480 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "initiated", 2024-09-18T08:07:22.480 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:05:27.004525+0000", 2024-09-18T08:07:22.480 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-09-18T08:07:22.480 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:07:22.480 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:07:22.480 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "header_read", 2024-09-18T08:07:22.480 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:05:27.004525+0000", 2024-09-18T08:07:22.480 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-09-18T08:07:22.480 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:07:22.481 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:07:22.481 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "throttled", 2024-09-18T08:07:22.481 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:05:27.004525+0000", 2024-09-18T08:07:22.481 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 5.4600000000000005e-07 2024-09-18T08:07:22.481 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:07:22.481 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:07:22.481 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "all_read", 2024-09-18T08:07:22.481 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:05:27.004544+0000", 2024-09-18T08:07:22.481 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.9018e-05 2024-09-18T08:07:22.481 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:07:22.481 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:07:22.481 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "dispatched", 2024-09-18T08:07:22.481 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:05:27.004551+0000", 2024-09-18T08:07:22.481 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 6.7020000000000004e-06 2024-09-18T08:07:22.482 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:07:22.482 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:07:22.482 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "queued_for_pg", 2024-09-18T08:07:22.482 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:05:27.004559+0000", 2024-09-18T08:07:22.482 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 7.5549999999999997e-06 2024-09-18T08:07:22.482 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:07:22.482 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:07:22.482 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-09-18T08:07:22.482 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:05:27.096081+0000", 2024-09-18T08:07:22.482 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.091522084000000004 2024-09-18T08:07:22.482 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:07:22.482 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:07:22.482 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "started", 2024-09-18T08:07:22.483 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:05:27.096214+0000", 2024-09-18T08:07:22.483 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.000133564 2024-09-18T08:07:22.483 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:07:22.483 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:07:22.483 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "waiting for subops from 5,6", 2024-09-18T08:07:22.483 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:05:27.096375+0000", 2024-09-18T08:07:22.483 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.00016054099999999999 2024-09-18T08:07:22.483 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:07:22.483 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:07:22.483 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:07:22.483 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:05:27.101327+0000", 2024-09-18T08:07:22.483 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.004952606 2024-09-18T08:07:22.483 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:07:22.483 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:07:22.484 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:07:22.484 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:05:27.101360+0000", 2024-09-18T08:07:22.484 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 3.2870999999999997e-05 2024-09-18T08:07:22.484 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:07:22.484 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:07:22.484 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "op_commit", 2024-09-18T08:07:22.484 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:05:27.101551+0000", 2024-09-18T08:07:22.484 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.00019097100000000001 2024-09-18T08:07:22.484 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:07:22.484 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:07:22.484 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "commit_sent", 2024-09-18T08:07:22.484 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:05:27.101596+0000", 2024-09-18T08:07:22.484 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 4.4776e-05 2024-09-18T08:07:22.484 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:07:22.485 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:07:22.485 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "done", 2024-09-18T08:07:22.485 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:05:27.101643+0000", 2024-09-18T08:07:22.485 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 4.7185000000000003e-05 2024-09-18T08:07:22.485 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-09-18T08:07:22.485 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-09-18T08:07:22.485 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-09-18T08:07:22.485 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:07:22.485 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:07:22.485 INFO:teuthology.orchestra.run.smithi007.stdout: "description": "osd_op(client.14805.0:110394 7.b 7:d9bd6b02:::benchmark_data_smithi007_73155_object110393:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e118)", 2024-09-18T08:07:22.485 INFO:teuthology.orchestra.run.smithi007.stdout: "initiated_at": "2024-09-18T08:05:27.006321+0000", 2024-09-18T08:07:22.485 INFO:teuthology.orchestra.run.smithi007.stdout: "age": 115.327579865, 2024-09-18T08:07:22.485 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.096031278999999997, 2024-09-18T08:07:22.485 INFO:teuthology.orchestra.run.smithi007.stdout: "continuous": false, 2024-09-18T08:07:22.486 INFO:teuthology.orchestra.run.smithi007.stdout: "type_data": { 2024-09-18T08:07:22.486 INFO:teuthology.orchestra.run.smithi007.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T08:07:22.486 INFO:teuthology.orchestra.run.smithi007.stdout: "client_info": { 2024-09-18T08:07:22.486 INFO:teuthology.orchestra.run.smithi007.stdout: "client": "client.14805", 2024-09-18T08:07:22.486 INFO:teuthology.orchestra.run.smithi007.stdout: "client_addr": "172.21.15.7:0/640320436", 2024-09-18T08:07:22.486 INFO:teuthology.orchestra.run.smithi007.stdout: "tid": 110394 2024-09-18T08:07:22.486 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:07:22.486 INFO:teuthology.orchestra.run.smithi007.stdout: "events": [ 2024-09-18T08:07:22.486 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:07:22.486 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "initiated", 2024-09-18T08:07:22.486 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:05:27.006321+0000", 2024-09-18T08:07:22.486 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-09-18T08:07:22.486 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:07:22.487 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:07:22.487 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "header_read", 2024-09-18T08:07:22.487 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:05:27.006321+0000", 2024-09-18T08:07:22.487 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-09-18T08:07:22.487 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:07:22.487 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:07:22.487 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "throttled", 2024-09-18T08:07:22.487 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:05:27.006323+0000", 2024-09-18T08:07:22.487 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.4869999999999999e-06 2024-09-18T08:07:22.487 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:07:22.487 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:07:22.487 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "all_read", 2024-09-18T08:07:22.487 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:05:27.006370+0000", 2024-09-18T08:07:22.487 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 4.6788e-05 2024-09-18T08:07:22.488 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:07:22.488 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:07:22.488 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "dispatched", 2024-09-18T08:07:22.488 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:05:27.006387+0000", 2024-09-18T08:07:22.488 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.751e-05 2024-09-18T08:07:22.488 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:07:22.488 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:07:22.488 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "queued_for_pg", 2024-09-18T08:07:22.488 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:05:27.006395+0000", 2024-09-18T08:07:22.488 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 7.6939999999999995e-06 2024-09-18T08:07:22.488 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:07:22.488 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:07:22.488 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-09-18T08:07:22.489 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:05:27.098909+0000", 2024-09-18T08:07:22.489 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.092514425999999997 2024-09-18T08:07:22.489 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:07:22.489 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:07:22.489 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "started", 2024-09-18T08:07:22.489 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:05:27.099029+0000", 2024-09-18T08:07:22.489 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.00011928400000000001 2024-09-18T08:07:22.489 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:07:22.489 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:07:22.489 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "waiting for subops from 5,6", 2024-09-18T08:07:22.489 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:05:27.099249+0000", 2024-09-18T08:07:22.489 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.000220936 2024-09-18T08:07:22.489 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:07:22.490 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:07:22.490 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "op_commit", 2024-09-18T08:07:22.490 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:05:27.101660+0000", 2024-09-18T08:07:22.490 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.0024101529999999999 2024-09-18T08:07:22.490 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:07:22.490 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:07:22.490 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:07:22.490 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:05:27.102184+0000", 2024-09-18T08:07:22.490 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.00052446000000000003 2024-09-18T08:07:22.490 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:07:22.490 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:07:22.490 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:07:22.490 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:05:27.102240+0000", 2024-09-18T08:07:22.490 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 5.5714999999999998e-05 2024-09-18T08:07:22.491 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:07:22.491 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:07:22.491 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "commit_sent", 2024-09-18T08:07:22.491 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:05:27.102276+0000", 2024-09-18T08:07:22.491 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 3.5737e-05 2024-09-18T08:07:22.491 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:07:22.491 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:07:22.491 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "done", 2024-09-18T08:07:22.491 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:05:27.102353+0000", 2024-09-18T08:07:22.491 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 7.7088999999999993e-05 2024-09-18T08:07:22.491 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-09-18T08:07:22.491 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-09-18T08:07:22.491 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-09-18T08:07:22.492 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:07:22.492 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:07:22.492 INFO:teuthology.orchestra.run.smithi007.stdout: "description": "osd_repop(client.14805.0:110395 7.9 e118/83 7:9bdc7d38:::benchmark_data_smithi007_73155_object110394:head v 118'6848, mlcod=118'6846)", 2024-09-18T08:07:22.492 INFO:teuthology.orchestra.run.smithi007.stdout: "initiated_at": "2024-09-18T08:05:27.009405+0000", 2024-09-18T08:07:22.492 INFO:teuthology.orchestra.run.smithi007.stdout: "age": 115.324495766, 2024-09-18T08:07:22.492 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.088193234999999995, 2024-09-18T08:07:22.492 INFO:teuthology.orchestra.run.smithi007.stdout: "continuous": false, 2024-09-18T08:07:22.492 INFO:teuthology.orchestra.run.smithi007.stdout: "type_data": { 2024-09-18T08:07:22.493 INFO:teuthology.orchestra.run.smithi007.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T08:07:22.493 INFO:teuthology.orchestra.run.smithi007.stdout: "events": [ 2024-09-18T08:07:22.493 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:07:22.493 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "initiated", 2024-09-18T08:07:22.493 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:05:27.009405+0000", 2024-09-18T08:07:22.493 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-09-18T08:07:22.493 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:07:22.493 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:07:22.493 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "header_read", 2024-09-18T08:07:22.493 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:05:27.009405+0000", 2024-09-18T08:07:22.493 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-09-18T08:07:22.493 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:07:22.493 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:07:22.493 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "throttled", 2024-09-18T08:07:22.494 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:05:27.009407+0000", 2024-09-18T08:07:22.494 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.285e-06 2024-09-18T08:07:22.494 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:07:22.494 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:07:22.494 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "all_read", 2024-09-18T08:07:22.494 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:05:27.009433+0000", 2024-09-18T08:07:22.494 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 2.6514000000000001e-05 2024-09-18T08:07:22.494 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:07:22.494 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:07:22.494 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "dispatched", 2024-09-18T08:07:22.494 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:05:27.089458+0000", 2024-09-18T08:07:22.494 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.080025154000000001 2024-09-18T08:07:22.494 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:07:22.494 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:07:22.495 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "queued_for_pg", 2024-09-18T08:07:22.495 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:05:27.089468+0000", 2024-09-18T08:07:22.495 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 9.5109999999999992e-06 2024-09-18T08:07:22.495 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:07:22.495 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:07:22.495 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-09-18T08:07:22.495 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:05:27.089879+0000", 2024-09-18T08:07:22.495 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.00041120600000000001 2024-09-18T08:07:22.495 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:07:22.495 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:07:22.496 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2 2024-09-18T08:07:22.497 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "started", 2024-09-18T08:07:22.497 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:05:27.089899+0000", 2024-09-18T08:07:22.497 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 2.0327999999999999e-05 2024-09-18T08:07:22.497 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:07:22.497 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:07:22.497 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "commit_sent", 2024-09-18T08:07:22.497 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:05:27.097552+0000", 2024-09-18T08:07:22.497 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.0076520939999999999 2024-09-18T08:07:22.497 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:07:22.497 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:07:22.498 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "done", 2024-09-18T08:07:22.498 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:05:27.097599+0000", 2024-09-18T08:07:22.498 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 4.7142999999999999e-05 2024-09-18T08:07:22.498 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-09-18T08:07:22.498 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-09-18T08:07:22.498 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-09-18T08:07:22.498 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:07:22.498 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:07:22.498 INFO:teuthology.orchestra.run.smithi007.stdout: "description": "osd_op(client.14805.0:114974 7.d 7:b1c89472:::benchmark_data_smithi007_73155_object114973:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e119)", 2024-09-18T08:07:22.498 INFO:teuthology.orchestra.run.smithi007.stdout: "initiated_at": "2024-09-18T08:05:32.156013+0000", 2024-09-18T08:07:22.498 INFO:teuthology.orchestra.run.smithi007.stdout: "age": 110.17788841700001, 2024-09-18T08:07:22.498 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.081817602000000003, 2024-09-18T08:07:22.498 INFO:teuthology.orchestra.run.smithi007.stdout: "continuous": false, 2024-09-18T08:07:22.499 INFO:teuthology.orchestra.run.smithi007.stdout: "type_data": { 2024-09-18T08:07:22.499 INFO:teuthology.orchestra.run.smithi007.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T08:07:22.499 INFO:teuthology.orchestra.run.smithi007.stdout: "client_info": { 2024-09-18T08:07:22.499 INFO:teuthology.orchestra.run.smithi007.stdout: "client": "client.14805", 2024-09-18T08:07:22.499 INFO:teuthology.orchestra.run.smithi007.stdout: "client_addr": "172.21.15.7:0/640320436", 2024-09-18T08:07:22.499 INFO:teuthology.orchestra.run.smithi007.stdout: "tid": 114974 2024-09-18T08:07:22.499 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:07:22.499 INFO:teuthology.orchestra.run.smithi007.stdout: "events": [ 2024-09-18T08:07:22.499 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:07:22.499 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "initiated", 2024-09-18T08:07:22.499 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:05:32.156013+0000", 2024-09-18T08:07:22.499 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-09-18T08:07:22.499 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:07:22.500 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:07:22.500 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "header_read", 2024-09-18T08:07:22.500 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:05:32.156013+0000", 2024-09-18T08:07:22.500 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-09-18T08:07:22.500 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:07:22.500 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:07:22.500 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "throttled", 2024-09-18T08:07:22.500 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:05:32.156014+0000", 2024-09-18T08:07:22.500 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.649e-06 2024-09-18T08:07:22.500 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:07:22.500 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:07:22.500 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "all_read", 2024-09-18T08:07:22.500 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:05:32.156038+0000", 2024-09-18T08:07:22.500 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 2.3493999999999999e-05 2024-09-18T08:07:22.501 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:07:22.501 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:07:22.501 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "dispatched", 2024-09-18T08:07:22.501 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:05:32.156049+0000", 2024-09-18T08:07:22.501 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.1518999999999999e-05 2024-09-18T08:07:22.501 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:07:22.501 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:07:22.501 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "queued_for_pg", 2024-09-18T08:07:22.501 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:05:32.156057+0000", 2024-09-18T08:07:22.501 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 7.498e-06 2024-09-18T08:07:22.501 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:07:22.502 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:07:22.502 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-09-18T08:07:22.502 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:05:32.156273+0000", 2024-09-18T08:07:22.502 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.000216457 2024-09-18T08:07:22.502 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:07:22.502 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:07:22.502 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "started", 2024-09-18T08:07:22.502 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:05:32.156449+0000", 2024-09-18T08:07:22.502 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.000175205 2024-09-18T08:07:22.502 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:07:22.502 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:07:22.502 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "waiting for subops from 0,6", 2024-09-18T08:07:22.502 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:05:32.156603+0000", 2024-09-18T08:07:22.502 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.00015435899999999999 2024-09-18T08:07:22.503 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:07:22.503 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:07:22.503 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:07:22.503 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:05:32.159984+0000", 2024-09-18T08:07:22.503 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.0033807059999999998 2024-09-18T08:07:22.503 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:07:22.503 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:07:22.503 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "op_commit", 2024-09-18T08:07:22.503 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:05:32.164345+0000", 2024-09-18T08:07:22.503 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.0043609590000000002 2024-09-18T08:07:22.503 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:07:22.503 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:07:22.503 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:07:22.503 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:05:32.237524+0000", 2024-09-18T08:07:22.504 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.073178875000000004 2024-09-18T08:07:22.504 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:07:22.504 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:07:22.504 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "commit_sent", 2024-09-18T08:07:22.504 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:05:32.237600+0000", 2024-09-18T08:07:22.504 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 7.6763999999999999e-05 2024-09-18T08:07:22.504 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:07:22.504 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:07:22.504 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "done", 2024-09-18T08:07:22.504 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:05:32.237830+0000", 2024-09-18T08:07:22.504 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.00023011700000000001 2024-09-18T08:07:22.504 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-09-18T08:07:22.505 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-09-18T08:07:22.505 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-09-18T08:07:22.505 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:07:22.505 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:07:22.505 INFO:teuthology.orchestra.run.smithi007.stdout: "description": "osd_op(client.14805.0:141889 7.d 7:b4e5db3b:::benchmark_data_smithi007_73155_object23598:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e127)", 2024-09-18T08:07:22.505 INFO:teuthology.orchestra.run.smithi007.stdout: "initiated_at": "2024-09-18T08:05:59.091118+0000", 2024-09-18T08:07:22.505 INFO:teuthology.orchestra.run.smithi007.stdout: "age": 83.242782750999993, 2024-09-18T08:07:22.505 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.10400169099999999, 2024-09-18T08:07:22.505 INFO:teuthology.orchestra.run.smithi007.stdout: "continuous": false, 2024-09-18T08:07:22.505 INFO:teuthology.orchestra.run.smithi007.stdout: "type_data": { 2024-09-18T08:07:22.505 INFO:teuthology.orchestra.run.smithi007.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T08:07:22.505 INFO:teuthology.orchestra.run.smithi007.stdout: "client_info": { 2024-09-18T08:07:22.505 INFO:teuthology.orchestra.run.smithi007.stdout: "client": "client.14805", 2024-09-18T08:07:22.506 INFO:teuthology.orchestra.run.smithi007.stdout: "client_addr": "172.21.15.7:0/640320436", 2024-09-18T08:07:22.506 INFO:teuthology.orchestra.run.smithi007.stdout: "tid": 141889 2024-09-18T08:07:22.506 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:07:22.506 INFO:teuthology.orchestra.run.smithi007.stdout: "events": [ 2024-09-18T08:07:22.506 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:07:22.506 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "initiated", 2024-09-18T08:07:22.506 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:05:59.091118+0000", 2024-09-18T08:07:22.506 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-09-18T08:07:22.506 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:07:22.506 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:07:22.506 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "header_read", 2024-09-18T08:07:22.506 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:05:59.091118+0000", 2024-09-18T08:07:22.506 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-09-18T08:07:22.507 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:07:22.507 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:07:22.507 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "throttled", 2024-09-18T08:07:22.507 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:05:59.091120+0000", 2024-09-18T08:07:22.507 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.3400000000000001e-06 2024-09-18T08:07:22.507 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:07:22.507 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:07:22.507 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "all_read", 2024-09-18T08:07:22.507 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:05:59.091126+0000", 2024-09-18T08:07:22.507 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 5.9599999999999997e-06 2024-09-18T08:07:22.507 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:07:22.507 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:07:22.507 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "dispatched", 2024-09-18T08:07:22.508 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:05:59.091136+0000", 2024-09-18T08:07:22.508 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.0417e-05 2024-09-18T08:07:22.508 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:07:22.508 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:07:22.508 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "queued_for_pg", 2024-09-18T08:07:22.508 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:05:59.091143+0000", 2024-09-18T08:07:22.508 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 7.2690000000000002e-06 2024-09-18T08:07:22.508 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:07:22.508 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:07:22.508 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-09-18T08:07:22.508 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:05:59.091188+0000", 2024-09-18T08:07:22.508 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 4.4150000000000003e-05 2024-09-18T08:07:22.508 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:07:22.509 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:07:22.509 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "started", 2024-09-18T08:07:22.509 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:05:59.091292+0000", 2024-09-18T08:07:22.509 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.000104189 2024-09-18T08:07:22.509 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:07:22.509 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:07:22.509 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "waiting for subops from 0,6", 2024-09-18T08:07:22.509 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:05:59.091395+0000", 2024-09-18T08:07:22.509 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.000103077 2024-09-18T08:07:22.509 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:07:22.509 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:07:22.509 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "op_commit", 2024-09-18T08:07:22.509 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:05:59.093785+0000", 2024-09-18T08:07:22.509 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.0023899429999999998 2024-09-18T08:07:22.510 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:07:22.510 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:07:22.510 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:07:22.510 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:05:59.093906+0000", 2024-09-18T08:07:22.510 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.000121598 2024-09-18T08:07:22.510 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:07:22.510 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:07:22.510 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:07:22.510 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:05:59.195012+0000", 2024-09-18T08:07:22.510 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.101105768 2024-09-18T08:07:22.510 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:07:22.510 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:07:22.510 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "commit_sent", 2024-09-18T08:07:22.510 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:05:59.195073+0000", 2024-09-18T08:07:22.511 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 6.0358e-05 2024-09-18T08:07:22.511 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:07:22.511 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:07:22.511 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "done", 2024-09-18T08:07:22.511 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:05:59.195120+0000", 2024-09-18T08:07:22.511 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 4.7621999999999999e-05 2024-09-18T08:07:22.511 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-09-18T08:07:22.511 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-09-18T08:07:22.511 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-09-18T08:07:22.511 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:07:22.511 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:07:22.511 INFO:teuthology.orchestra.run.smithi007.stdout: "description": "osd_op(client.14805.0:148360 7.d 7:bc501593:::benchmark_data_smithi007_73155_object30069:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e127)", 2024-09-18T08:07:22.511 INFO:teuthology.orchestra.run.smithi007.stdout: "initiated_at": "2024-09-18T08:06:05.266586+0000", 2024-09-18T08:07:22.511 INFO:teuthology.orchestra.run.smithi007.stdout: "age": 77.067315504000007, 2024-09-18T08:07:22.512 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.083293002000000005, 2024-09-18T08:07:22.512 INFO:teuthology.orchestra.run.smithi007.stdout: "continuous": false, 2024-09-18T08:07:22.512 INFO:teuthology.orchestra.run.smithi007.stdout: "type_data": { 2024-09-18T08:07:22.512 INFO:teuthology.orchestra.run.smithi007.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T08:07:22.512 INFO:teuthology.orchestra.run.smithi007.stdout: "client_info": { 2024-09-18T08:07:22.512 INFO:teuthology.orchestra.run.smithi007.stdout: "client": "client.14805", 2024-09-18T08:07:22.512 INFO:teuthology.orchestra.run.smithi007.stdout: "client_addr": "172.21.15.7:0/640320436", 2024-09-18T08:07:22.512 INFO:teuthology.orchestra.run.smithi007.stdout: "tid": 148360 2024-09-18T08:07:22.512 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:07:22.512 INFO:teuthology.orchestra.run.smithi007.stdout: "events": [ 2024-09-18T08:07:22.512 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:07:22.512 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "initiated", 2024-09-18T08:07:22.513 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:06:05.266586+0000", 2024-09-18T08:07:22.513 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-09-18T08:07:22.513 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:07:22.513 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:07:22.513 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "header_read", 2024-09-18T08:07:22.513 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:06:05.266586+0000", 2024-09-18T08:07:22.513 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-09-18T08:07:22.513 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:07:22.513 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:07:22.513 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "throttled", 2024-09-18T08:07:22.513 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:06:05.266587+0000", 2024-09-18T08:07:22.513 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.7010000000000001e-06 2024-09-18T08:07:22.513 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:07:22.514 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:07:22.514 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "all_read", 2024-09-18T08:07:22.514 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:06:05.266593+0000", 2024-09-18T08:07:22.514 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 5.519e-06 2024-09-18T08:07:22.514 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:07:22.514 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:07:22.514 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "dispatched", 2024-09-18T08:07:22.514 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:06:05.266604+0000", 2024-09-18T08:07:22.514 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.1258e-05 2024-09-18T08:07:22.514 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:07:22.514 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:07:22.514 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "queued_for_pg", 2024-09-18T08:07:22.514 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:06:05.266611+0000", 2024-09-18T08:07:22.514 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 6.7959999999999998e-06 2024-09-18T08:07:22.515 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:07:22.515 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:07:22.515 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-09-18T08:07:22.515 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:06:05.266664+0000", 2024-09-18T08:07:22.515 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 5.2855999999999997e-05 2024-09-18T08:07:22.515 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:07:22.515 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:07:22.515 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "started", 2024-09-18T08:07:22.515 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:06:05.266770+0000", 2024-09-18T08:07:22.515 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.000106007 2024-09-18T08:07:22.515 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:07:22.515 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:07:22.515 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "waiting for subops from 0,6", 2024-09-18T08:07:22.515 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:06:05.266888+0000", 2024-09-18T08:07:22.516 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.000117657 2024-09-18T08:07:22.516 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:07:22.516 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:07:22.516 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "op_commit", 2024-09-18T08:07:22.516 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:06:05.269100+0000", 2024-09-18T08:07:22.516 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.002212739 2024-09-18T08:07:22.516 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:07:22.516 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:07:22.516 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:07:22.516 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:06:05.269290+0000", 2024-09-18T08:07:22.516 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.00018953199999999999 2024-09-18T08:07:22.516 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:07:22.516 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:07:22.517 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:07:22.517 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:06:05.349771+0000", 2024-09-18T08:07:22.517 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.080481358000000003 2024-09-18T08:07:22.517 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:07:22.517 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:07:22.517 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "commit_sent", 2024-09-18T08:07:22.517 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:06:05.349824+0000", 2024-09-18T08:07:22.517 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 5.2465e-05 2024-09-18T08:07:22.517 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:07:22.517 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:07:22.517 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "done", 2024-09-18T08:07:22.517 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:06:05.349879+0000", 2024-09-18T08:07:22.517 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 5.5114000000000001e-05 2024-09-18T08:07:22.518 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-09-18T08:07:22.518 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-09-18T08:07:22.518 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-09-18T08:07:22.518 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:07:22.518 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:07:22.518 INFO:teuthology.orchestra.run.smithi007.stdout: "description": "osd_op(client.14805.0:177698 7.d 7:bb18c981:::benchmark_data_smithi007_73155_object59407:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e139)", 2024-09-18T08:07:22.518 INFO:teuthology.orchestra.run.smithi007.stdout: "initiated_at": "2024-09-18T08:06:38.564955+0000", 2024-09-18T08:07:22.518 INFO:teuthology.orchestra.run.smithi007.stdout: "age": 43.768946198000002, 2024-09-18T08:07:22.518 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.10433407, 2024-09-18T08:07:22.518 INFO:teuthology.orchestra.run.smithi007.stdout: "continuous": false, 2024-09-18T08:07:22.518 INFO:teuthology.orchestra.run.smithi007.stdout: "type_data": { 2024-09-18T08:07:22.518 INFO:teuthology.orchestra.run.smithi007.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T08:07:22.518 INFO:teuthology.orchestra.run.smithi007.stdout: "client_info": { 2024-09-18T08:07:22.518 INFO:teuthology.orchestra.run.smithi007.stdout: "client": "client.14805", 2024-09-18T08:07:22.519 INFO:teuthology.orchestra.run.smithi007.stdout: "client_addr": "172.21.15.7:0/640320436", 2024-09-18T08:07:22.519 INFO:teuthology.orchestra.run.smithi007.stdout: "tid": 177698 2024-09-18T08:07:22.519 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:07:22.519 INFO:teuthology.orchestra.run.smithi007.stdout: "events": [ 2024-09-18T08:07:22.519 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:07:22.519 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "initiated", 2024-09-18T08:07:22.519 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:06:38.564955+0000", 2024-09-18T08:07:22.519 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-09-18T08:07:22.519 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:07:22.519 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:07:22.519 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "header_read", 2024-09-18T08:07:22.519 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:06:38.564955+0000", 2024-09-18T08:07:22.519 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-09-18T08:07:22.519 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:07:22.520 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:07:22.520 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "throttled", 2024-09-18T08:07:22.520 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:06:38.564957+0000", 2024-09-18T08:07:22.520 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.6360000000000001e-06 2024-09-18T08:07:22.520 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:07:22.520 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:07:22.520 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "all_read", 2024-09-18T08:07:22.520 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:06:38.564962+0000", 2024-09-18T08:07:22.520 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 5.2560000000000002e-06 2024-09-18T08:07:22.520 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:07:22.520 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:07:22.520 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "dispatched", 2024-09-18T08:07:22.520 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:06:38.564973+0000", 2024-09-18T08:07:22.521 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.0767999999999999e-05 2024-09-18T08:07:22.521 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:07:22.521 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:07:22.521 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "queued_for_pg", 2024-09-18T08:07:22.521 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:06:38.564980+0000", 2024-09-18T08:07:22.521 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 7.0099999999999998e-06 2024-09-18T08:07:22.521 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:07:22.521 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:07:22.521 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-09-18T08:07:22.521 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:06:38.572527+0000", 2024-09-18T08:07:22.521 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.007547064 2024-09-18T08:07:22.521 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:07:22.521 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:07:22.522 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "started", 2024-09-18T08:07:22.522 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:06:38.572671+0000", 2024-09-18T08:07:22.522 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.000143988 2024-09-18T08:07:22.522 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:07:22.522 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:07:22.522 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "waiting for subops from 0,6", 2024-09-18T08:07:22.522 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:06:38.572778+0000", 2024-09-18T08:07:22.522 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.000106841 2024-09-18T08:07:22.522 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:07:22.522 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:07:22.522 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "op_commit", 2024-09-18T08:07:22.522 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:06:38.577567+0000", 2024-09-18T08:07:22.522 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.0047890900000000002 2024-09-18T08:07:22.522 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:07:22.523 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:07:22.523 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:07:22.523 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:06:38.577766+0000", 2024-09-18T08:07:22.523 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.00019934999999999999 2024-09-18T08:07:22.523 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:07:22.523 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:07:22.523 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:07:22.523 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:06:38.669184+0000", 2024-09-18T08:07:22.523 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.091417562999999993 2024-09-18T08:07:22.523 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:07:22.523 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:07:22.523 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "commit_sent", 2024-09-18T08:07:22.523 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:06:38.669241+0000", 2024-09-18T08:07:22.524 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 5.7129999999999997e-05 2024-09-18T08:07:22.524 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:07:22.524 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:07:22.524 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "done", 2024-09-18T08:07:22.524 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:06:38.669289+0000", 2024-09-18T08:07:22.524 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 4.8374000000000001e-05 2024-09-18T08:07:22.524 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-09-18T08:07:22.524 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-09-18T08:07:22.524 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-09-18T08:07:22.524 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:07:22.524 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:07:22.524 INFO:teuthology.orchestra.run.smithi007.stdout: "description": "osd_op(client.14805.0:177701 7.d 7:bd37e3ec:::benchmark_data_smithi007_73155_object59410:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e139)", 2024-09-18T08:07:22.524 INFO:teuthology.orchestra.run.smithi007.stdout: "initiated_at": "2024-09-18T08:06:38.566207+0000", 2024-09-18T08:07:22.524 INFO:teuthology.orchestra.run.smithi007.stdout: "age": 43.767694005000003, 2024-09-18T08:07:22.525 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.11481989500000001, 2024-09-18T08:07:22.525 INFO:teuthology.orchestra.run.smithi007.stdout: "continuous": false, 2024-09-18T08:07:22.525 INFO:teuthology.orchestra.run.smithi007.stdout: "type_data": { 2024-09-18T08:07:22.525 INFO:teuthology.orchestra.run.smithi007.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T08:07:22.525 INFO:teuthology.orchestra.run.smithi007.stdout: "client_info": { 2024-09-18T08:07:22.525 INFO:teuthology.orchestra.run.smithi007.stdout: "client": "client.14805", 2024-09-18T08:07:22.525 INFO:teuthology.orchestra.run.smithi007.stdout: "client_addr": "172.21.15.7:0/640320436", 2024-09-18T08:07:22.525 INFO:teuthology.orchestra.run.smithi007.stdout: "tid": 177701 2024-09-18T08:07:22.525 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:07:22.525 INFO:teuthology.orchestra.run.smithi007.stdout: "events": [ 2024-09-18T08:07:22.525 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:07:22.525 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "initiated", 2024-09-18T08:07:22.526 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:07:22 smithi007 ceph-mon[29459]: 5.7 scrub starts 2024-09-18T08:07:22.526 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:07:22 smithi007 ceph-mon[29459]: 5.7 scrub ok 2024-09-18T08:07:22.526 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:07:22 smithi007 ceph-mon[29459]: pgmap v202: 148 pgs: 148 active+clean; 1.3 GiB data, 4.8 GiB used, 710 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 971 op/s 2024-09-18T08:07:22.526 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:07:22 smithi007 ceph-mon[29459]: 3.7 scrub starts 2024-09-18T08:07:22.526 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:07:22 smithi007 ceph-mon[29459]: 3.7 scrub ok 2024-09-18T08:07:22.526 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:07:22 smithi007 ceph-mon[29459]: from='client.? 172.21.15.7:0/3854885354' entity='client.admin' cmd={"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "11"} : dispatch 2024-09-18T08:07:22.526 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:07:22 smithi007 ceph-mon[29459]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "11"} : dispatch 2024-09-18T08:07:22.527 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:07:22 smithi007 ceph-mon[39249]: 5.7 scrub starts 2024-09-18T08:07:22.527 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:07:22 smithi007 ceph-mon[39249]: 5.7 scrub ok 2024-09-18T08:07:22.527 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:07:22 smithi007 ceph-mon[39249]: pgmap v202: 148 pgs: 148 active+clean; 1.3 GiB data, 4.8 GiB used, 710 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 971 op/s 2024-09-18T08:07:22.527 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:07:22 smithi007 ceph-mon[39249]: 3.7 scrub starts 2024-09-18T08:07:22.527 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:07:22 smithi007 ceph-mon[39249]: 3.7 scrub ok 2024-09-18T08:07:22.527 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:07:22 smithi007 ceph-mon[39249]: from='client.? 172.21.15.7:0/3854885354' entity='client.admin' cmd={"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "11"} : dispatch 2024-09-18T08:07:22.527 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:07:22 smithi007 ceph-mon[39249]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "11"} : dispatch 2024-09-18T08:07:22.527 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:06:38.566207+0000", 2024-09-18T08:07:22.527 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-09-18T08:07:22.528 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:07:22.528 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:07:22.528 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "header_read", 2024-09-18T08:07:22.528 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:06:38.566207+0000", 2024-09-18T08:07:22.528 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-09-18T08:07:22.528 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:07:22.528 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:07:22.528 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "throttled", 2024-09-18T08:07:22.528 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:06:38.566208+0000", 2024-09-18T08:07:22.528 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.1459999999999999e-06 2024-09-18T08:07:22.528 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:07:22.528 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:07:22.529 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "all_read", 2024-09-18T08:07:22.529 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:06:38.566213+0000", 2024-09-18T08:07:22.529 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 4.8840000000000002e-06 2024-09-18T08:07:22.529 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:07:22.529 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:07:22.529 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "dispatched", 2024-09-18T08:07:22.529 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:06:38.566224+0000", 2024-09-18T08:07:22.529 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.0446e-05 2024-09-18T08:07:22.529 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:07:22.529 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:07:22.529 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "queued_for_pg", 2024-09-18T08:07:22.529 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:06:38.566231+0000", 2024-09-18T08:07:22.529 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 7.1239999999999999e-06 2024-09-18T08:07:22.530 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:07:22.530 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:07:22.530 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-09-18T08:07:22.530 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:06:38.577995+0000", 2024-09-18T08:07:22.530 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.011764383 2024-09-18T08:07:22.530 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:07:22.530 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:07:22.530 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "started", 2024-09-18T08:07:22.530 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:06:38.578066+0000", 2024-09-18T08:07:22.530 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 7.0649999999999996e-05 2024-09-18T08:07:22.530 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:07:22.530 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:07:22.530 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "waiting for subops from 0,6", 2024-09-18T08:07:22.531 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:06:38.578131+0000", 2024-09-18T08:07:22.531 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 6.5443000000000003e-05 2024-09-18T08:07:22.531 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:07:22.531 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:07:22.531 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "op_commit", 2024-09-18T08:07:22.531 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:06:38.580460+0000", 2024-09-18T08:07:22.531 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.0023290759999999998 2024-09-18T08:07:22.531 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:07:22.531 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:07:22.531 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:07:22.531 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:06:38.581450+0000", 2024-09-18T08:07:22.531 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.00099010000000000005 2024-09-18T08:07:22.532 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:07:22.532 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:07:22.532 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:07:22.532 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:06:38.680937+0000", 2024-09-18T08:07:22.532 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.099487022999999994 2024-09-18T08:07:22.532 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:07:22.532 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:07:22.532 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "commit_sent", 2024-09-18T08:07:22.532 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:06:38.680989+0000", 2024-09-18T08:07:22.532 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 5.1184999999999999e-05 2024-09-18T08:07:22.533 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:07:22.533 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:07:22.533 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "done", 2024-09-18T08:07:22.533 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:06:38.681027+0000", 2024-09-18T08:07:22.533 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 3.8435e-05 2024-09-18T08:07:22.533 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-09-18T08:07:22.533 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-09-18T08:07:22.533 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-09-18T08:07:22.533 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:07:22.533 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:07:22.533 INFO:teuthology.orchestra.run.smithi007.stdout: "description": "osd_op(client.14805.0:177710 7.d 7:be72ff22:::benchmark_data_smithi007_73155_object59419:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e139)", 2024-09-18T08:07:22.533 INFO:teuthology.orchestra.run.smithi007.stdout: "initiated_at": "2024-09-18T08:06:38.569886+0000", 2024-09-18T08:07:22.533 INFO:teuthology.orchestra.run.smithi007.stdout: "age": 43.764015497000003, 2024-09-18T08:07:22.533 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.11125222, 2024-09-18T08:07:22.534 INFO:teuthology.orchestra.run.smithi007.stdout: "continuous": false, 2024-09-18T08:07:22.534 INFO:teuthology.orchestra.run.smithi007.stdout: "type_data": { 2024-09-18T08:07:22.534 INFO:teuthology.orchestra.run.smithi007.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T08:07:22.534 INFO:teuthology.orchestra.run.smithi007.stdout: "client_info": { 2024-09-18T08:07:22.534 INFO:teuthology.orchestra.run.smithi007.stdout: "client": "client.14805", 2024-09-18T08:07:22.534 INFO:teuthology.orchestra.run.smithi007.stdout: "client_addr": "172.21.15.7:0/640320436", 2024-09-18T08:07:22.534 INFO:teuthology.orchestra.run.smithi007.stdout: "tid": 177710 2024-09-18T08:07:22.534 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:07:22.534 INFO:teuthology.orchestra.run.smithi007.stdout: "events": [ 2024-09-18T08:07:22.534 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:07:22.534 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "initiated", 2024-09-18T08:07:22.534 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:06:38.569886+0000", 2024-09-18T08:07:22.534 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-09-18T08:07:22.534 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:07:22.535 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:07:22.535 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "header_read", 2024-09-18T08:07:22.535 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:06:38.569886+0000", 2024-09-18T08:07:22.535 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-09-18T08:07:22.535 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:07:22.535 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:07:22.535 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "throttled", 2024-09-18T08:07:22.535 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:06:38.569887+0000", 2024-09-18T08:07:22.535 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.37e-06 2024-09-18T08:07:22.535 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:07:22.535 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:07:22.535 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "all_read", 2024-09-18T08:07:22.535 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:06:38.569892+0000", 2024-09-18T08:07:22.536 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 5.361e-06 2024-09-18T08:07:22.536 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:07:22.536 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:07:22.536 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "dispatched", 2024-09-18T08:07:22.536 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:06:38.569903+0000", 2024-09-18T08:07:22.536 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.0920999999999999e-05 2024-09-18T08:07:22.536 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:07:22.536 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:07:22.536 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "queued_for_pg", 2024-09-18T08:07:22.536 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:06:38.569910+0000", 2024-09-18T08:07:22.536 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 6.9890000000000003e-06 2024-09-18T08:07:22.536 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:07:22.536 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:07:22.536 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-09-18T08:07:22.537 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:06:38.579438+0000", 2024-09-18T08:07:22.537 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.0095277190000000005 2024-09-18T08:07:22.537 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:07:22.537 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:07:22.537 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "started", 2024-09-18T08:07:22.537 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:06:38.579499+0000", 2024-09-18T08:07:22.537 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 6.1019e-05 2024-09-18T08:07:22.537 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:07:22.537 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:07:22.537 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "waiting for subops from 0,6", 2024-09-18T08:07:22.537 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:06:38.579559+0000", 2024-09-18T08:07:22.537 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 5.978e-05 2024-09-18T08:07:22.537 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:07:22.538 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:07:22.538 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "op_commit", 2024-09-18T08:07:22.538 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:06:38.580562+0000", 2024-09-18T08:07:22.538 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.0010029870000000001 2024-09-18T08:07:22.538 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:07:22.538 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:07:22.538 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:07:22.538 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:06:38.581518+0000", 2024-09-18T08:07:22.538 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.00095614099999999998 2024-09-18T08:07:22.538 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:07:22.538 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:07:22.538 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:07:22.538 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:06:38.681061+0000", 2024-09-18T08:07:22.539 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.099542862999999995 2024-09-18T08:07:22.539 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:07:22.539 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:07:22.539 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "commit_sent", 2024-09-18T08:07:22.539 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:06:38.681105+0000", 2024-09-18T08:07:22.539 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 4.4598000000000002e-05 2024-09-18T08:07:22.539 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:07:22.539 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:07:22.539 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "done", 2024-09-18T08:07:22.539 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:06:38.681138+0000", 2024-09-18T08:07:22.539 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 3.2471999999999997e-05 2024-09-18T08:07:22.539 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-09-18T08:07:22.539 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-09-18T08:07:22.540 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-09-18T08:07:22.540 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-09-18T08:07:22.540 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-09-18T08:07:22.540 INFO:teuthology.orchestra.run.smithi007.stdout:} 2024-09-18T08:07:22.582 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:07:22.583 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:07:22.583 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2' 2024-09-18T08:07:22.656 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:07:22 smithi050 ceph-mon[32371]: 5.7 scrub starts 2024-09-18T08:07:22.656 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:07:22 smithi050 ceph-mon[32371]: 5.7 scrub ok 2024-09-18T08:07:22.656 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:07:22 smithi050 ceph-mon[32371]: pgmap v202: 148 pgs: 148 active+clean; 1.3 GiB data, 4.8 GiB used, 710 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 971 op/s 2024-09-18T08:07:22.656 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:07:22 smithi050 ceph-mon[32371]: 3.7 scrub starts 2024-09-18T08:07:22.656 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:07:22 smithi050 ceph-mon[32371]: 3.7 scrub ok 2024-09-18T08:07:22.656 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:07:22 smithi050 ceph-mon[32371]: from='client.? 172.21.15.7:0/3854885354' entity='client.admin' cmd={"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "11"} : dispatch 2024-09-18T08:07:22.656 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:07:22 smithi050 ceph-mon[32371]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "11"} : dispatch 2024-09-18T08:07:22.684 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1 2024-09-18T08:07:22.726 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:07:22.726 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:07:22.727 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1' 2024-09-18T08:07:22.827 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5 2024-09-18T08:07:22.854 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:07:22.855 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:07:22.855 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5' 2024-09-18T08:07:22.956 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6 2024-09-18T08:07:22.984 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:07:22.984 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:07:22.985 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6' 2024-09-18T08:07:23.086 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6 2024-09-18T08:07:23.111 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:07:23.111 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:07:23.111 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6' 2024-09-18T08:07:23.212 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1 2024-09-18T08:07:23.251 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:07:23.251 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:07:23.252 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1' 2024-09-18T08:07:23.353 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0 2024-09-18T08:07:23.397 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:07:23.398 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:07:23.398 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0' 2024-09-18T08:07:23.499 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5 2024-09-18T08:07:23.528 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:07:23.529 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:07:23.529 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5' 2024-09-18T08:07:23.630 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0 2024-09-18T08:07:23.656 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:07:23 smithi050 ceph-mon[32371]: 5.10 scrub starts 2024-09-18T08:07:23.656 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:07:23 smithi050 ceph-mon[32371]: 5.10 scrub ok 2024-09-18T08:07:23.656 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:07:23 smithi050 ceph-mon[32371]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "11"}]': finished 2024-09-18T08:07:23.656 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:07:23 smithi050 ceph-mon[32371]: osdmap e151: 8 total, 8 up, 8 in 2024-09-18T08:07:23.656 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:07:23 smithi050 ceph-mon[32371]: 4.2 scrub starts 2024-09-18T08:07:23.656 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:07:23 smithi050 ceph-mon[32371]: 4.2 scrub ok 2024-09-18T08:07:23.657 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:07:23 smithi050 ceph-mon[32371]: from='mgr.24470 172.21.15.7:0/2594390261' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "11"} : dispatch 2024-09-18T08:07:23.671 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:07:23.672 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:07:23.672 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0' 2024-09-18T08:07:23.714 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:07:23 smithi007 ceph-mon[29459]: 5.10 scrub starts 2024-09-18T08:07:23.714 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:07:23 smithi007 ceph-mon[29459]: 5.10 scrub ok 2024-09-18T08:07:23.715 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:07:23 smithi007 ceph-mon[29459]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "11"}]': finished 2024-09-18T08:07:23.715 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:07:23 smithi007 ceph-mon[29459]: osdmap e151: 8 total, 8 up, 8 in 2024-09-18T08:07:23.715 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:07:23 smithi007 ceph-mon[29459]: 4.2 scrub starts 2024-09-18T08:07:23.715 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:07:23 smithi007 ceph-mon[29459]: 4.2 scrub ok 2024-09-18T08:07:23.715 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:07:23 smithi007 ceph-mon[29459]: from='mgr.24470 172.21.15.7:0/2594390261' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "11"} : dispatch 2024-09-18T08:07:23.715 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:07:23 smithi007 ceph-mon[39249]: 5.10 scrub starts 2024-09-18T08:07:23.715 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:07:23 smithi007 ceph-mon[39249]: 5.10 scrub ok 2024-09-18T08:07:23.715 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:07:23 smithi007 ceph-mon[39249]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "11"}]': finished 2024-09-18T08:07:23.715 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:07:23 smithi007 ceph-mon[39249]: osdmap e151: 8 total, 8 up, 8 in 2024-09-18T08:07:23.715 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:07:23 smithi007 ceph-mon[39249]: 4.2 scrub starts 2024-09-18T08:07:23.715 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:07:23 smithi007 ceph-mon[39249]: 4.2 scrub ok 2024-09-18T08:07:23.715 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:07:23 smithi007 ceph-mon[39249]: from='mgr.24470 172.21.15.7:0/2594390261' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "11"} : dispatch 2024-09-18T08:07:23.773 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0 2024-09-18T08:07:23.809 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:07:23.810 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:07:23.810 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0' 2024-09-18T08:07:23.911 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0 2024-09-18T08:07:23.948 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:07:23.949 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:07:23.949 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0' 2024-09-18T08:07:24.050 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7 2024-09-18T08:07:24.076 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:07:24.077 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:07:24.077 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7' 2024-09-18T08:07:24.178 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0 2024-09-18T08:07:24.220 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:07:24.221 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:07:24.221 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0' 2024-09-18T08:07:24.322 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6 2024-09-18T08:07:24.350 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:07:24.351 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:07:24.352 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6' 2024-09-18T08:07:24.352 INFO:teuthology.orchestra.run.smithi007.stderr:noscrub is set 2024-09-18T08:07:24.417 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:07:24 smithi007 ceph-mon[29459]: pgmap v204: 148 pgs: 148 active+clean; 1.2 GiB data, 4.4 GiB used, 711 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 679 op/s 2024-09-18T08:07:24.417 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:07:24 smithi007 ceph-mon[29459]: from='mgr.24470 172.21.15.7:0/2594390261' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "11"}]': finished 2024-09-18T08:07:24.417 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:07:24 smithi007 ceph-mon[29459]: osdmap e152: 8 total, 8 up, 8 in 2024-09-18T08:07:24.417 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:07:24 smithi007 ceph-mon[29459]: from='client.? 172.21.15.7:0/175470526' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-18T08:07:24.417 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:07:24 smithi007 ceph-mon[29459]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-18T08:07:24.418 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:07:24 smithi007 ceph-mon[39249]: pgmap v204: 148 pgs: 148 active+clean; 1.2 GiB data, 4.4 GiB used, 711 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 679 op/s 2024-09-18T08:07:24.418 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:07:24 smithi007 ceph-mon[39249]: from='mgr.24470 172.21.15.7:0/2594390261' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "11"}]': finished 2024-09-18T08:07:24.418 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:07:24 smithi007 ceph-mon[39249]: osdmap e152: 8 total, 8 up, 8 in 2024-09-18T08:07:24.418 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:07:24 smithi007 ceph-mon[39249]: from='client.? 172.21.15.7:0/175470526' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-18T08:07:24.418 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:07:24 smithi007 ceph-mon[39249]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-18T08:07:24.452 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2 2024-09-18T08:07:24.492 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:07:24.493 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:07:24.493 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2' 2024-09-18T08:07:24.594 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7 2024-09-18T08:07:24.623 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:07:24.624 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:07:24.624 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7' 2024-09-18T08:07:24.656 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:07:24 smithi050 ceph-mon[32371]: pgmap v204: 148 pgs: 148 active+clean; 1.2 GiB data, 4.4 GiB used, 711 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 679 op/s 2024-09-18T08:07:24.656 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:07:24 smithi050 ceph-mon[32371]: from='mgr.24470 172.21.15.7:0/2594390261' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "11"}]': finished 2024-09-18T08:07:24.656 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:07:24 smithi050 ceph-mon[32371]: osdmap e152: 8 total, 8 up, 8 in 2024-09-18T08:07:24.656 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:07:24 smithi050 ceph-mon[32371]: from='client.? 172.21.15.7:0/175470526' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-18T08:07:24.656 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:07:24 smithi050 ceph-mon[32371]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-18T08:07:24.726 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4 2024-09-18T08:07:24.752 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:07:24.752 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:07:24.753 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4' 2024-09-18T08:07:24.854 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3 2024-09-18T08:07:24.883 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid 779349ca-7593-11ef-bceb-c7b262605968 -- ceph daemon osd.2 dump_ops_in_flight 2024-09-18T08:07:24.900 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:07:24.901 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:07:24.901 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3' 2024-09-18T08:07:25.002 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2 2024-09-18T08:07:25.044 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:07:25.045 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:07:25.045 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2' 2024-09-18T08:07:25.146 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7 2024-09-18T08:07:25.171 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:07:25.172 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:07:25.172 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7' 2024-09-18T08:07:25.273 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6 2024-09-18T08:07:25.303 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:07:25.303 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:07:25.304 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6' 2024-09-18T08:07:25.404 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3 2024-09-18T08:07:25.448 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:07:25.449 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:07:25.449 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3' 2024-09-18T08:07:25.480 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:07:25 smithi007 ceph-mon[29459]: 4.16 scrub starts 2024-09-18T08:07:25.481 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:07:25 smithi007 ceph-mon[29459]: 4.16 scrub ok 2024-09-18T08:07:25.481 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:07:25 smithi007 ceph-mon[29459]: 5.9 scrub starts 2024-09-18T08:07:25.481 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:07:25 smithi007 ceph-mon[29459]: 5.9 scrub ok 2024-09-18T08:07:25.481 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:07:25 smithi007 ceph-mon[29459]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-18T08:07:25.481 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:07:25 smithi007 ceph-mon[29459]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-18T08:07:25.481 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:07:25 smithi007 ceph-mon[29459]: osdmap e153: 8 total, 8 up, 8 in 2024-09-18T08:07:25.481 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:07:25 smithi007 ceph-mon[29459]: from='mgr.24470 172.21.15.7:0/2594390261' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num_actual", "val": "11"} : dispatch 2024-09-18T08:07:25.481 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:07:25 smithi007 ceph-mon[39249]: 4.16 scrub starts 2024-09-18T08:07:25.481 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:07:25 smithi007 ceph-mon[39249]: 4.16 scrub ok 2024-09-18T08:07:25.484 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:07:25 smithi007 ceph-mon[39249]: 5.9 scrub starts 2024-09-18T08:07:25.484 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:07:25 smithi007 ceph-mon[39249]: 5.9 scrub ok 2024-09-18T08:07:25.484 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:07:25 smithi007 ceph-mon[39249]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-18T08:07:25.484 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:07:25 smithi007 ceph-mon[39249]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-18T08:07:25.484 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:07:25 smithi007 ceph-mon[39249]: osdmap e153: 8 total, 8 up, 8 in 2024-09-18T08:07:25.484 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:07:25 smithi007 ceph-mon[39249]: from='mgr.24470 172.21.15.7:0/2594390261' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num_actual", "val": "11"} : dispatch 2024-09-18T08:07:25.550 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4 2024-09-18T08:07:25.574 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:07:25.575 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:07:25.575 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4' 2024-09-18T08:07:25.656 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:07:25 smithi050 ceph-mon[32371]: 4.16 scrub starts 2024-09-18T08:07:25.656 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:07:25 smithi050 ceph-mon[32371]: 4.16 scrub ok 2024-09-18T08:07:25.656 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:07:25 smithi050 ceph-mon[32371]: 5.9 scrub starts 2024-09-18T08:07:25.656 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:07:25 smithi050 ceph-mon[32371]: 5.9 scrub ok 2024-09-18T08:07:25.656 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:07:25 smithi050 ceph-mon[32371]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-18T08:07:25.656 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:07:25 smithi050 ceph-mon[32371]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-18T08:07:25.656 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:07:25 smithi050 ceph-mon[32371]: osdmap e153: 8 total, 8 up, 8 in 2024-09-18T08:07:25.656 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:07:25 smithi050 ceph-mon[32371]: from='mgr.24470 172.21.15.7:0/2594390261' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num_actual", "val": "11"} : dispatch 2024-09-18T08:07:25.676 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2 2024-09-18T08:07:25.720 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:07:25.721 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:07:25.721 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2' 2024-09-18T08:07:25.763 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid 779349ca-7593-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-18T08:07:25.822 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7 2024-09-18T08:07:25.849 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:07:25.849 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:07:25.850 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7' 2024-09-18T08:07:25.950 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3 2024-09-18T08:07:25.995 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:07:25.996 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:07:25.996 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3' 2024-09-18T08:07:26.097 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6 2024-09-18T08:07:26.123 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:07:26.124 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:07:26.124 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6' 2024-09-18T08:07:26.225 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5 2024-09-18T08:07:26.248 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:07:26.248 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:07:26.249 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5' 2024-09-18T08:07:26.349 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4 2024-09-18T08:07:26.378 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:07:26.378 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:07:26.379 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4' 2024-09-18T08:07:26.431 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:07:26 smithi007 ceph-mon[29459]: 1.0 scrub starts 2024-09-18T08:07:26.431 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:07:26 smithi007 ceph-mon[29459]: pgmap v207: 158 pgs: 10 unknown, 148 active+clean; 978 MiB data, 3.9 GiB used, 711 GiB / 715 GiB avail; 853 B/s rd, 0 B/s wr, 818 op/s 2024-09-18T08:07:26.431 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:07:26 smithi007 ceph-mon[29459]: from='mgr.24470 172.21.15.7:0/2594390261' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num_actual", "val": "11"}]': finished 2024-09-18T08:07:26.431 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:07:26 smithi007 ceph-mon[29459]: osdmap e154: 8 total, 8 up, 8 in 2024-09-18T08:07:26.431 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:07:26 smithi007 ceph-mon[29459]: purged_snaps scrub starts 2024-09-18T08:07:26.431 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:07:26 smithi007 ceph-mon[29459]: purged_snaps scrub ok 2024-09-18T08:07:26.432 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:07:26 smithi007 ceph-mon[39249]: 1.0 scrub starts 2024-09-18T08:07:26.432 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:07:26 smithi007 ceph-mon[39249]: pgmap v207: 158 pgs: 10 unknown, 148 active+clean; 978 MiB data, 3.9 GiB used, 711 GiB / 715 GiB avail; 853 B/s rd, 0 B/s wr, 818 op/s 2024-09-18T08:07:26.432 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:07:26 smithi007 ceph-mon[39249]: from='mgr.24470 172.21.15.7:0/2594390261' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num_actual", "val": "11"}]': finished 2024-09-18T08:07:26.432 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:07:26 smithi007 ceph-mon[39249]: osdmap e154: 8 total, 8 up, 8 in 2024-09-18T08:07:26.432 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:07:26 smithi007 ceph-mon[39249]: purged_snaps scrub starts 2024-09-18T08:07:26.432 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:07:26 smithi007 ceph-mon[39249]: purged_snaps scrub ok 2024-09-18T08:07:26.480 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2 2024-09-18T08:07:26.537 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:07:26.537 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:07:26.537 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2' 2024-09-18T08:07:26.561 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/779349ca-7593-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T08:07:26.618 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/779349ca-7593-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T08:07:26.638 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6 2024-09-18T08:07:26.656 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:07:26 smithi050 ceph-mon[32371]: 1.0 scrub starts 2024-09-18T08:07:26.656 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:07:26 smithi050 ceph-mon[32371]: pgmap v207: 158 pgs: 10 unknown, 148 active+clean; 978 MiB data, 3.9 GiB used, 711 GiB / 715 GiB avail; 853 B/s rd, 0 B/s wr, 818 op/s 2024-09-18T08:07:26.656 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:07:26 smithi050 ceph-mon[32371]: from='mgr.24470 172.21.15.7:0/2594390261' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num_actual", "val": "11"}]': finished 2024-09-18T08:07:26.656 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:07:26 smithi050 ceph-mon[32371]: osdmap e154: 8 total, 8 up, 8 in 2024-09-18T08:07:26.656 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:07:26 smithi050 ceph-mon[32371]: purged_snaps scrub starts 2024-09-18T08:07:26.656 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:07:26 smithi050 ceph-mon[32371]: purged_snaps scrub ok 2024-09-18T08:07:26.667 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:07:26.668 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:07:26.669 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6' 2024-09-18T08:07:26.769 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0 2024-09-18T08:07:26.818 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:07:26.819 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:07:26.819 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0' 2024-09-18T08:07:26.920 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6 2024-09-18T08:07:26.946 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:07:26.947 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:07:26.947 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6' 2024-09-18T08:07:26.981 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Sep 18 08:07:26 smithi007 ceph-779349ca-7593-11ef-bceb-c7b262605968-mgr-y[29688]: ::ffff:172.21.15.50 - - [18/Sep/2024:08:07:26] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-09-18T08:07:27.048 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6 2024-09-18T08:07:27.074 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:07:27.075 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:07:27.075 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6' 2024-09-18T08:07:27.176 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1 2024-09-18T08:07:27.212 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:07:27.213 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:07:27.213 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1' 2024-09-18T08:07:27.314 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0 2024-09-18T08:07:27.356 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:07:27.357 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:07:27.357 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0' 2024-09-18T08:07:27.458 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1 2024-09-18T08:07:27.489 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:07:27.490 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:07:27.490 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1' 2024-09-18T08:07:27.591 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2 2024-09-18T08:07:27.634 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:07:27.635 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:07:27.635 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2' 2024-09-18T08:07:27.651 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:07:27 smithi007 ceph-mon[39249]: 1.0 scrub starts 2024-09-18T08:07:27.651 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:07:27 smithi007 ceph-mon[39249]: osdmap e155: 8 total, 8 up, 8 in 2024-09-18T08:07:27.651 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:07:27 smithi007 ceph-mon[39249]: 1.2 scrub starts 2024-09-18T08:07:27.651 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:07:27 smithi007 ceph-mon[39249]: 1.5 scrub starts 2024-09-18T08:07:27.652 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:07:27 smithi007 ceph-mon[39249]: 1.a scrub starts 2024-09-18T08:07:27.652 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:07:27 smithi007 ceph-mon[39249]: 1.3 scrub starts 2024-09-18T08:07:27.652 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:07:27 smithi007 ceph-mon[39249]: 1.7 scrub starts 2024-09-18T08:07:27.652 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:07:27 smithi007 ceph-mon[29459]: 1.0 scrub starts 2024-09-18T08:07:27.652 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:07:27 smithi007 ceph-mon[29459]: osdmap e155: 8 total, 8 up, 8 in 2024-09-18T08:07:27.652 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:07:27 smithi007 ceph-mon[29459]: 1.2 scrub starts 2024-09-18T08:07:27.652 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:07:27 smithi007 ceph-mon[29459]: 1.5 scrub starts 2024-09-18T08:07:27.652 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:07:27 smithi007 ceph-mon[29459]: 1.a scrub starts 2024-09-18T08:07:27.652 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:07:27 smithi007 ceph-mon[29459]: 1.3 scrub starts 2024-09-18T08:07:27.652 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:07:27 smithi007 ceph-mon[29459]: 1.7 scrub starts 2024-09-18T08:07:27.655 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:07:27 smithi050 ceph-mon[32371]: 1.0 scrub starts 2024-09-18T08:07:27.656 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:07:27 smithi050 ceph-mon[32371]: osdmap e155: 8 total, 8 up, 8 in 2024-09-18T08:07:27.656 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:07:27 smithi050 ceph-mon[32371]: 1.2 scrub starts 2024-09-18T08:07:27.656 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:07:27 smithi050 ceph-mon[32371]: 1.5 scrub starts 2024-09-18T08:07:27.656 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:07:27 smithi050 ceph-mon[32371]: 1.a scrub starts 2024-09-18T08:07:27.656 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:07:27 smithi050 ceph-mon[32371]: 1.3 scrub starts 2024-09-18T08:07:27.656 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:07:27 smithi050 ceph-mon[32371]: 1.7 scrub starts 2024-09-18T08:07:27.736 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7 2024-09-18T08:07:27.765 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:07:27.765 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:07:27.766 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7' 2024-09-18T08:07:27.867 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6 2024-09-18T08:07:27.891 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:07:27.892 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:07:27.894 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6' 2024-09-18T08:07:27.988 INFO:teuthology.orchestra.run.smithi007.stdout:osd.0: {} 2024-09-18T08:07:27.989 INFO:teuthology.orchestra.run.smithi007.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-18T08:07:27.994 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1 2024-09-18T08:07:28.026 INFO:teuthology.orchestra.run.smithi007.stdout:osd.1: {} 2024-09-18T08:07:28.027 INFO:teuthology.orchestra.run.smithi007.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-18T08:07:28.033 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:07:28.033 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:07:28.034 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1' 2024-09-18T08:07:28.074 INFO:teuthology.orchestra.run.smithi007.stdout:osd.2: {} 2024-09-18T08:07:28.124 INFO:teuthology.orchestra.run.smithi007.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-18T08:07:28.124 INFO:teuthology.orchestra.run.smithi007.stdout:osd.3: {} 2024-09-18T08:07:28.125 INFO:teuthology.orchestra.run.smithi007.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-18T08:07:28.135 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6 2024-09-18T08:07:28.160 INFO:teuthology.orchestra.run.smithi007.stdout:osd.4: {} 2024-09-18T08:07:28.160 INFO:teuthology.orchestra.run.smithi007.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-18T08:07:28.163 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:07:28.163 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:07:28.164 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6' 2024-09-18T08:07:28.193 INFO:teuthology.orchestra.run.smithi007.stdout:ERROR: (22) Invalid argument 2024-09-18T08:07:28.194 INFO:teuthology.orchestra.run.smithi007.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-18T08:07:28.203 INFO:teuthology.orchestra.run.smithi007.stdout:osd.5: {} 2024-09-18T08:07:28.203 INFO:teuthology.orchestra.run.smithi007.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-18T08:07:28.245 INFO:teuthology.orchestra.run.smithi007.stdout:osd.6: {} 2024-09-18T08:07:28.246 INFO:teuthology.orchestra.run.smithi007.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-18T08:07:28.264 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1 2024-09-18T08:07:28.287 INFO:teuthology.orchestra.run.smithi007.stdout:osd.7: {} 2024-09-18T08:07:28.287 INFO:teuthology.orchestra.run.smithi007.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-18T08:07:28.310 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:07:28.310 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:07:28.311 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1' 2024-09-18T08:07:28.412 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6 2024-09-18T08:07:28.434 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid 779349ca-7593-11ef-bceb-c7b262605968 -- ceph osd set nodeep-scrub 2024-09-18T08:07:28.439 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:07:28.440 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:07:28.440 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6' 2024-09-18T08:07:28.541 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3 2024-09-18T08:07:28.584 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:07:28.585 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:07:28.585 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3' 2024-09-18T08:07:28.656 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:07:28 smithi050 ceph-mon[32371]: 1.9 scrub starts 2024-09-18T08:07:28.656 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:07:28 smithi050 ceph-mon[32371]: pgmap v210: 158 pgs: 10 unknown, 148 active+clean; 978 MiB data, 3.9 GiB used, 711 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.07k op/s 2024-09-18T08:07:28.656 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:07:28 smithi050 ceph-mon[32371]: 1.6 scrub starts 2024-09-18T08:07:28.656 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:07:28 smithi050 ceph-mon[32371]: 1.1 scrub starts 2024-09-18T08:07:28.656 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:07:28 smithi050 ceph-mon[32371]: from='client.? 172.21.15.7:0/4022089863' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-18T08:07:28.656 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:07:28 smithi050 ceph-mon[32371]: 1.4 scrub starts 2024-09-18T08:07:28.686 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1 2024-09-18T08:07:28.730 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:07:28 smithi007 ceph-mon[29459]: 1.9 scrub starts 2024-09-18T08:07:28.731 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:07:28 smithi007 ceph-mon[29459]: pgmap v210: 158 pgs: 10 unknown, 148 active+clean; 978 MiB data, 3.9 GiB used, 711 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.07k op/s 2024-09-18T08:07:28.731 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:07:28 smithi007 ceph-mon[29459]: 1.6 scrub starts 2024-09-18T08:07:28.731 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:07:28 smithi007 ceph-mon[29459]: 1.1 scrub starts 2024-09-18T08:07:28.731 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:07:28 smithi007 ceph-mon[29459]: from='client.? 172.21.15.7:0/4022089863' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-18T08:07:28.731 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:07:28 smithi007 ceph-mon[29459]: 1.4 scrub starts 2024-09-18T08:07:28.731 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:07:28 smithi007 ceph-mon[39249]: 1.9 scrub starts 2024-09-18T08:07:28.731 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:07:28 smithi007 ceph-mon[39249]: pgmap v210: 158 pgs: 10 unknown, 148 active+clean; 978 MiB data, 3.9 GiB used, 711 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.07k op/s 2024-09-18T08:07:28.731 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:07:28 smithi007 ceph-mon[39249]: 1.6 scrub starts 2024-09-18T08:07:28.731 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:07:28 smithi007 ceph-mon[39249]: 1.1 scrub starts 2024-09-18T08:07:28.731 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:07:28 smithi007 ceph-mon[39249]: from='client.? 172.21.15.7:0/4022089863' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-18T08:07:28.732 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:07:28 smithi007 ceph-mon[39249]: 1.4 scrub starts 2024-09-18T08:07:28.742 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:07:28.742 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:07:28.742 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1' 2024-09-18T08:07:28.843 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2 2024-09-18T08:07:28.893 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:07:28.894 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:07:28.894 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2' 2024-09-18T08:07:28.995 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2 2024-09-18T08:07:29.033 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:07:29.034 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:07:29.034 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2' 2024-09-18T08:07:29.135 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2 2024-09-18T08:07:29.174 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:07:29.174 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:07:29.175 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2' 2024-09-18T08:07:29.196 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 7, 6] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-09-18T08:07:29.196 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-18T08:07:29.197 INFO:tasks.thrashosds.thrasher:thrash_pg_upmap_items 2024-09-18T08:07:29.197 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid 779349ca-7593-11ef-bceb-c7b262605968 -- ceph osd dump -f json-pretty 2024-09-18T08:07:29.275 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5 2024-09-18T08:07:29.302 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:07:29.303 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:07:29.303 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5' 2024-09-18T08:07:29.404 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5 2024-09-18T08:07:29.406 INFO:journalctl@ceph.iscsi.iscsi.a.smithi050.stdout:Sep 18 08:07:28 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-iscsi-iscsi-a[57782]: debug there is no tcmu-runner data available 2024-09-18T08:07:29.427 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:07:29.428 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:07:29.429 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5' 2024-09-18T08:07:29.529 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3 2024-09-18T08:07:29.572 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:07:29.573 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:07:29.573 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3' 2024-09-18T08:07:29.674 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3 2024-09-18T08:07:29.717 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:07:29.717 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:07:29.717 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3' 2024-09-18T08:07:29.730 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:07:29 smithi007 ceph-mon[29459]: purged_snaps scrub starts 2024-09-18T08:07:29.730 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:07:29 smithi007 ceph-mon[29459]: purged_snaps scrub ok 2024-09-18T08:07:29.730 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:07:29 smithi007 ceph-mon[29459]: osdmap e156: 8 total, 8 up, 8 in 2024-09-18T08:07:29.731 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:07:29 smithi007 ceph-mon[39249]: purged_snaps scrub starts 2024-09-18T08:07:29.731 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:07:29 smithi007 ceph-mon[39249]: purged_snaps scrub ok 2024-09-18T08:07:29.731 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:07:29 smithi007 ceph-mon[39249]: osdmap e156: 8 total, 8 up, 8 in 2024-09-18T08:07:29.818 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0 2024-09-18T08:07:29.853 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:07:29.854 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:07:29.855 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0' 2024-09-18T08:07:29.906 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:07:29 smithi050 ceph-mon[32371]: purged_snaps scrub starts 2024-09-18T08:07:29.906 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:07:29 smithi050 ceph-mon[32371]: purged_snaps scrub ok 2024-09-18T08:07:29.906 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:07:29 smithi050 ceph-mon[32371]: osdmap e156: 8 total, 8 up, 8 in 2024-09-18T08:07:29.932 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid 779349ca-7593-11ef-bceb-c7b262605968 -- ceph daemon osd.2 dump_blocked_ops 2024-09-18T08:07:29.956 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3 2024-09-18T08:07:29.989 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/779349ca-7593-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T08:07:30.046 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/779349ca-7593-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T08:07:30.054 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:07:30.055 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:07:30.055 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3' 2024-09-18T08:07:30.156 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2 2024-09-18T08:07:30.208 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:07:30.208 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:07:30.209 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2' 2024-09-18T08:07:30.309 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1 2024-09-18T08:07:30.359 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:07:30.359 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:07:30.360 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1' 2024-09-18T08:07:30.436 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:07:30 smithi007 ceph-mon[39249]: 1.8 scrub starts 2024-09-18T08:07:30.436 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:07:30 smithi007 ceph-mon[39249]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-18T08:07:30.436 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:07:30 smithi007 ceph-mon[39249]: pgmap v212: 158 pgs: 158 active+clean; 653 MiB data, 2.9 GiB used, 712 GiB / 715 GiB avail; 0 B/s wr, 1.08k op/s 2024-09-18T08:07:30.460 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3 2024-09-18T08:07:30.511 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:07:30.512 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:07:30.512 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3' 2024-09-18T08:07:30.613 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3 2024-09-18T08:07:30.657 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:07:30.657 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:07:30.658 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3' 2024-09-18T08:07:30.715 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:07:30 smithi007 ceph-mon[29459]: 1.8 scrub starts 2024-09-18T08:07:30.715 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:07:30 smithi007 ceph-mon[29459]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-18T08:07:30.715 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:07:30 smithi007 ceph-mon[29459]: pgmap v212: 158 pgs: 158 active+clean; 653 MiB data, 2.9 GiB used, 712 GiB / 715 GiB avail; 0 B/s wr, 1.08k op/s 2024-09-18T08:07:30.759 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4 2024-09-18T08:07:30.783 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:07:30.784 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:07:30.784 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4' 2024-09-18T08:07:30.885 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2 2024-09-18T08:07:30.905 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:07:30 smithi050 ceph-mon[32371]: 1.8 scrub starts 2024-09-18T08:07:30.906 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:07:30 smithi050 ceph-mon[32371]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-18T08:07:30.906 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:07:30 smithi050 ceph-mon[32371]: pgmap v212: 158 pgs: 158 active+clean; 653 MiB data, 2.9 GiB used, 712 GiB / 715 GiB avail; 0 B/s wr, 1.08k op/s 2024-09-18T08:07:30.938 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:07:30.938 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:07:30.939 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2' 2024-09-18T08:07:31.040 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0 2024-09-18T08:07:31.079 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:07:31.080 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:07:31.080 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0' 2024-09-18T08:07:31.172 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/779349ca-7593-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T08:07:31.180 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4 2024-09-18T08:07:31.206 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:07:31.206 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:07:31.207 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4' 2024-09-18T08:07:31.308 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0 2024-09-18T08:07:31.350 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:07:31.351 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:07:31.351 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0' 2024-09-18T08:07:31.417 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid 779349ca-7593-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-18T08:07:31.451 INFO:teuthology.orchestra.run.smithi007.stderr:nodeep-scrub is set 2024-09-18T08:07:31.452 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0 2024-09-18T08:07:31.503 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:07:31.504 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:07:31.504 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0' 2024-09-18T08:07:31.511 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-09-18T08:07:31.511 INFO:teuthology.orchestra.run.smithi007.stdout:{ 2024-09-18T08:07:31.511 INFO:teuthology.orchestra.run.smithi007.stdout: "epoch": 157, 2024-09-18T08:07:31.511 INFO:teuthology.orchestra.run.smithi007.stdout: "fsid": "779349ca-7593-11ef-bceb-c7b262605968", 2024-09-18T08:07:31.511 INFO:teuthology.orchestra.run.smithi007.stdout: "created": "2024-09-18T07:57:09.295710+0000", 2024-09-18T08:07:31.512 INFO:teuthology.orchestra.run.smithi007.stdout: "modified": "2024-09-18T08:07:31.435674+0000", 2024-09-18T08:07:31.512 INFO:teuthology.orchestra.run.smithi007.stdout: "last_up_change": "2024-09-18T08:02:36.989876+0000", 2024-09-18T08:07:31.512 INFO:teuthology.orchestra.run.smithi007.stdout: "last_in_change": "2024-09-18T08:06:23.499918+0000", 2024-09-18T08:07:31.512 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": "noscrub,nodeep-scrub,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit", 2024-09-18T08:07:31.512 INFO:teuthology.orchestra.run.smithi007.stdout: "flags_num": 5806080, 2024-09-18T08:07:31.512 INFO:teuthology.orchestra.run.smithi007.stdout: "flags_set": [ 2024-09-18T08:07:31.512 INFO:teuthology.orchestra.run.smithi007.stdout: "nodeep-scrub", 2024-09-18T08:07:31.512 INFO:teuthology.orchestra.run.smithi007.stdout: "noscrub", 2024-09-18T08:07:31.512 INFO:teuthology.orchestra.run.smithi007.stdout: "pglog_hardlimit", 2024-09-18T08:07:31.512 INFO:teuthology.orchestra.run.smithi007.stdout: "purged_snapdirs", 2024-09-18T08:07:31.512 INFO:teuthology.orchestra.run.smithi007.stdout: "recovery_deletes", 2024-09-18T08:07:31.512 INFO:teuthology.orchestra.run.smithi007.stdout: "sortbitwise" 2024-09-18T08:07:31.512 INFO:teuthology.orchestra.run.smithi007.stdout: ], 2024-09-18T08:07:31.513 INFO:teuthology.orchestra.run.smithi007.stdout: "crush_version": 18, 2024-09-18T08:07:31.513 INFO:teuthology.orchestra.run.smithi007.stdout: "full_ratio": 0.94999998807907104, 2024-09-18T08:07:31.513 INFO:teuthology.orchestra.run.smithi007.stdout: "backfillfull_ratio": 0.89999997615814209, 2024-09-18T08:07:31.513 INFO:teuthology.orchestra.run.smithi007.stdout: "nearfull_ratio": 0.85000002384185791, 2024-09-18T08:07:31.513 INFO:teuthology.orchestra.run.smithi007.stdout: "cluster_snapshot": "", 2024-09-18T08:07:31.513 INFO:teuthology.orchestra.run.smithi007.stdout: "pool_max": 7, 2024-09-18T08:07:31.513 INFO:teuthology.orchestra.run.smithi007.stdout: "max_osd": 8, 2024-09-18T08:07:31.513 INFO:teuthology.orchestra.run.smithi007.stdout: "require_min_compat_client": "luminous", 2024-09-18T08:07:31.513 INFO:teuthology.orchestra.run.smithi007.stdout: "min_compat_client": "luminous", 2024-09-18T08:07:31.513 INFO:teuthology.orchestra.run.smithi007.stdout: "require_osd_release": "squid", 2024-09-18T08:07:31.513 INFO:teuthology.orchestra.run.smithi007.stdout: "allow_crimson": false, 2024-09-18T08:07:31.514 INFO:teuthology.orchestra.run.smithi007.stdout: "pools": [ 2024-09-18T08:07:31.514 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:07:31.514 INFO:teuthology.orchestra.run.smithi007.stdout: "pool": 1, 2024-09-18T08:07:31.514 INFO:teuthology.orchestra.run.smithi007.stdout: "pool_name": ".mgr", 2024-09-18T08:07:31.514 INFO:teuthology.orchestra.run.smithi007.stdout: "create_time": "2024-09-18T08:00:11.802627+0000", 2024-09-18T08:07:31.514 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 1, 2024-09-18T08:07:31.514 INFO:teuthology.orchestra.run.smithi007.stdout: "flags_names": "hashpspool", 2024-09-18T08:07:31.514 INFO:teuthology.orchestra.run.smithi007.stdout: "type": 1, 2024-09-18T08:07:31.514 INFO:teuthology.orchestra.run.smithi007.stdout: "size": 3, 2024-09-18T08:07:31.514 INFO:teuthology.orchestra.run.smithi007.stdout: "min_size": 2, 2024-09-18T08:07:31.514 INFO:teuthology.orchestra.run.smithi007.stdout: "crush_rule": 0, 2024-09-18T08:07:31.514 INFO:teuthology.orchestra.run.smithi007.stdout: "peering_crush_bucket_count": 0, 2024-09-18T08:07:31.514 INFO:teuthology.orchestra.run.smithi007.stdout: "peering_crush_bucket_target": 0, 2024-09-18T08:07:31.514 INFO:teuthology.orchestra.run.smithi007.stdout: "peering_crush_bucket_barrier": 0, 2024-09-18T08:07:31.515 INFO:teuthology.orchestra.run.smithi007.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-09-18T08:07:31.515 INFO:teuthology.orchestra.run.smithi007.stdout: "is_stretch_pool": false, 2024-09-18T08:07:31.515 INFO:teuthology.orchestra.run.smithi007.stdout: "object_hash": 2, 2024-09-18T08:07:31.515 INFO:teuthology.orchestra.run.smithi007.stdout: "pg_autoscale_mode": "off", 2024-09-18T08:07:31.515 INFO:teuthology.orchestra.run.smithi007.stdout: "pg_num": 11, 2024-09-18T08:07:31.515 INFO:teuthology.orchestra.run.smithi007.stdout: "pg_placement_num": 11, 2024-09-18T08:07:31.515 INFO:teuthology.orchestra.run.smithi007.stdout: "pg_placement_num_target": 11, 2024-09-18T08:07:31.515 INFO:teuthology.orchestra.run.smithi007.stdout: "pg_num_target": 11, 2024-09-18T08:07:31.515 INFO:teuthology.orchestra.run.smithi007.stdout: "pg_num_pending": 11, 2024-09-18T08:07:31.515 INFO:teuthology.orchestra.run.smithi007.stdout: "last_pg_merge_meta": { 2024-09-18T08:07:31.515 INFO:teuthology.orchestra.run.smithi007.stdout: "source_pgid": "0.0", 2024-09-18T08:07:31.515 INFO:teuthology.orchestra.run.smithi007.stdout: "ready_epoch": 0, 2024-09-18T08:07:31.515 INFO:teuthology.orchestra.run.smithi007.stdout: "last_epoch_started": 0, 2024-09-18T08:07:31.516 INFO:teuthology.orchestra.run.smithi007.stdout: "last_epoch_clean": 0, 2024-09-18T08:07:31.516 INFO:teuthology.orchestra.run.smithi007.stdout: "source_version": "0'0", 2024-09-18T08:07:31.516 INFO:teuthology.orchestra.run.smithi007.stdout: "target_version": "0'0" 2024-09-18T08:07:31.516 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:07:31.516 INFO:teuthology.orchestra.run.smithi007.stdout: "last_change": "154", 2024-09-18T08:07:31.516 INFO:teuthology.orchestra.run.smithi007.stdout: "last_force_op_resend": "0", 2024-09-18T08:07:31.516 INFO:teuthology.orchestra.run.smithi007.stdout: "last_force_op_resend_prenautilus": "0", 2024-09-18T08:07:31.516 INFO:teuthology.orchestra.run.smithi007.stdout: "last_force_op_resend_preluminous": "152", 2024-09-18T08:07:31.516 INFO:teuthology.orchestra.run.smithi007.stdout: "auid": 0, 2024-09-18T08:07:31.516 INFO:teuthology.orchestra.run.smithi007.stdout: "snap_mode": "selfmanaged", 2024-09-18T08:07:31.516 INFO:teuthology.orchestra.run.smithi007.stdout: "snap_seq": 0, 2024-09-18T08:07:31.516 INFO:teuthology.orchestra.run.smithi007.stdout: "snap_epoch": 0, 2024-09-18T08:07:31.516 INFO:teuthology.orchestra.run.smithi007.stdout: "pool_snaps": [], 2024-09-18T08:07:31.516 INFO:teuthology.orchestra.run.smithi007.stdout: "removed_snaps": "[]", 2024-09-18T08:07:31.517 INFO:teuthology.orchestra.run.smithi007.stdout: "quota_max_bytes": 0, 2024-09-18T08:07:31.517 INFO:teuthology.orchestra.run.smithi007.stdout: "quota_max_objects": 0, 2024-09-18T08:07:31.517 INFO:teuthology.orchestra.run.smithi007.stdout: "tiers": [], 2024-09-18T08:07:31.517 INFO:teuthology.orchestra.run.smithi007.stdout: "tier_of": -1, 2024-09-18T08:07:31.517 INFO:teuthology.orchestra.run.smithi007.stdout: "read_tier": -1, 2024-09-18T08:07:31.517 INFO:teuthology.orchestra.run.smithi007.stdout: "write_tier": -1, 2024-09-18T08:07:31.517 INFO:teuthology.orchestra.run.smithi007.stdout: "cache_mode": "none", 2024-09-18T08:07:31.517 INFO:teuthology.orchestra.run.smithi007.stdout: "target_max_bytes": 0, 2024-09-18T08:07:31.517 INFO:teuthology.orchestra.run.smithi007.stdout: "target_max_objects": 0, 2024-09-18T08:07:31.517 INFO:teuthology.orchestra.run.smithi007.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-09-18T08:07:31.517 INFO:teuthology.orchestra.run.smithi007.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-09-18T08:07:31.518 INFO:teuthology.orchestra.run.smithi007.stdout: "cache_target_full_ratio_micro": 800000, 2024-09-18T08:07:31.518 INFO:teuthology.orchestra.run.smithi007.stdout: "cache_min_flush_age": 0, 2024-09-18T08:07:31.518 INFO:teuthology.orchestra.run.smithi007.stdout: "cache_min_evict_age": 0, 2024-09-18T08:07:31.518 INFO:teuthology.orchestra.run.smithi007.stdout: "erasure_code_profile": "", 2024-09-18T08:07:31.518 INFO:teuthology.orchestra.run.smithi007.stdout: "hit_set_params": { 2024-09-18T08:07:31.518 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "none" 2024-09-18T08:07:31.518 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:07:31.518 INFO:teuthology.orchestra.run.smithi007.stdout: "hit_set_period": 0, 2024-09-18T08:07:31.518 INFO:teuthology.orchestra.run.smithi007.stdout: "hit_set_count": 0, 2024-09-18T08:07:31.518 INFO:teuthology.orchestra.run.smithi007.stdout: "use_gmt_hitset": true, 2024-09-18T08:07:31.518 INFO:teuthology.orchestra.run.smithi007.stdout: "min_read_recency_for_promote": 0, 2024-09-18T08:07:31.518 INFO:teuthology.orchestra.run.smithi007.stdout: "min_write_recency_for_promote": 0, 2024-09-18T08:07:31.518 INFO:teuthology.orchestra.run.smithi007.stdout: "hit_set_grade_decay_rate": 0, 2024-09-18T08:07:31.518 INFO:teuthology.orchestra.run.smithi007.stdout: "hit_set_search_last_n": 0, 2024-09-18T08:07:31.519 INFO:teuthology.orchestra.run.smithi007.stdout: "grade_table": [], 2024-09-18T08:07:31.519 INFO:teuthology.orchestra.run.smithi007.stdout: "stripe_width": 0, 2024-09-18T08:07:31.519 INFO:teuthology.orchestra.run.smithi007.stdout: "expected_num_objects": 0, 2024-09-18T08:07:31.519 INFO:teuthology.orchestra.run.smithi007.stdout: "fast_read": false, 2024-09-18T08:07:31.519 INFO:teuthology.orchestra.run.smithi007.stdout: "options": { 2024-09-18T08:07:31.519 INFO:teuthology.orchestra.run.smithi007.stdout: "pg_num_max": 32, 2024-09-18T08:07:31.519 INFO:teuthology.orchestra.run.smithi007.stdout: "pg_num_min": 1 2024-09-18T08:07:31.519 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:07:31.519 INFO:teuthology.orchestra.run.smithi007.stdout: "application_metadata": { 2024-09-18T08:07:31.519 INFO:teuthology.orchestra.run.smithi007.stdout: "mgr": {} 2024-09-18T08:07:31.519 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:07:31.519 INFO:teuthology.orchestra.run.smithi007.stdout: "read_balance": { 2024-09-18T08:07:31.519 INFO:teuthology.orchestra.run.smithi007.stdout: "score_type": "Fair distribution", 2024-09-18T08:07:31.519 INFO:teuthology.orchestra.run.smithi007.stdout: "score_acting": 1.9099999666213989, 2024-09-18T08:07:31.520 INFO:teuthology.orchestra.run.smithi007.stdout: "score_stable": 1.9099999666213989, 2024-09-18T08:07:31.520 INFO:teuthology.orchestra.run.smithi007.stdout: "optimal_score": 1.1399999856948853, 2024-09-18T08:07:31.520 INFO:teuthology.orchestra.run.smithi007.stdout: "raw_score_acting": 2.1800000667572021, 2024-09-18T08:07:31.520 INFO:teuthology.orchestra.run.smithi007.stdout: "raw_score_stable": 2.1800000667572021, 2024-09-18T08:07:31.520 INFO:teuthology.orchestra.run.smithi007.stdout: "primary_affinity_weighted": 0.84128189086914062, 2024-09-18T08:07:31.520 INFO:teuthology.orchestra.run.smithi007.stdout: "average_primary_affinity": 0.95999997854232788, 2024-09-18T08:07:31.520 INFO:teuthology.orchestra.run.smithi007.stdout: "average_primary_affinity_weighted": 0.8399999737739563 2024-09-18T08:07:31.520 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-09-18T08:07:31.520 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:07:31.520 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:07:31.520 INFO:teuthology.orchestra.run.smithi007.stdout: "pool": 2, 2024-09-18T08:07:31.520 INFO:teuthology.orchestra.run.smithi007.stdout: "pool_name": "datapool", 2024-09-18T08:07:31.520 INFO:teuthology.orchestra.run.smithi007.stdout: "create_time": "2024-09-18T08:02:45.374099+0000", 2024-09-18T08:07:31.520 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 8193, 2024-09-18T08:07:31.521 INFO:teuthology.orchestra.run.smithi007.stdout: "flags_names": "hashpspool,selfmanaged_snaps", 2024-09-18T08:07:31.521 INFO:teuthology.orchestra.run.smithi007.stdout: "type": 1, 2024-09-18T08:07:31.521 INFO:teuthology.orchestra.run.smithi007.stdout: "size": 3, 2024-09-18T08:07:31.521 INFO:teuthology.orchestra.run.smithi007.stdout: "min_size": 2, 2024-09-18T08:07:31.521 INFO:teuthology.orchestra.run.smithi007.stdout: "crush_rule": 0, 2024-09-18T08:07:31.521 INFO:teuthology.orchestra.run.smithi007.stdout: "peering_crush_bucket_count": 0, 2024-09-18T08:07:31.521 INFO:teuthology.orchestra.run.smithi007.stdout: "peering_crush_bucket_target": 0, 2024-09-18T08:07:31.521 INFO:teuthology.orchestra.run.smithi007.stdout: "peering_crush_bucket_barrier": 0, 2024-09-18T08:07:31.521 INFO:teuthology.orchestra.run.smithi007.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-09-18T08:07:31.521 INFO:teuthology.orchestra.run.smithi007.stdout: "is_stretch_pool": false, 2024-09-18T08:07:31.521 INFO:teuthology.orchestra.run.smithi007.stdout: "object_hash": 2, 2024-09-18T08:07:31.521 INFO:teuthology.orchestra.run.smithi007.stdout: "pg_autoscale_mode": "off", 2024-09-18T08:07:31.521 INFO:teuthology.orchestra.run.smithi007.stdout: "pg_num": 3, 2024-09-18T08:07:31.521 INFO:teuthology.orchestra.run.smithi007.stdout: "pg_placement_num": 3, 2024-09-18T08:07:31.522 INFO:teuthology.orchestra.run.smithi007.stdout: "pg_placement_num_target": 3, 2024-09-18T08:07:31.522 INFO:teuthology.orchestra.run.smithi007.stdout: "pg_num_target": 3, 2024-09-18T08:07:31.522 INFO:teuthology.orchestra.run.smithi007.stdout: "pg_num_pending": 3, 2024-09-18T08:07:31.522 INFO:teuthology.orchestra.run.smithi007.stdout: "last_pg_merge_meta": { 2024-09-18T08:07:31.522 INFO:teuthology.orchestra.run.smithi007.stdout: "source_pgid": "0.0", 2024-09-18T08:07:31.522 INFO:teuthology.orchestra.run.smithi007.stdout: "ready_epoch": 0, 2024-09-18T08:07:31.522 INFO:teuthology.orchestra.run.smithi007.stdout: "last_epoch_started": 0, 2024-09-18T08:07:31.522 INFO:teuthology.orchestra.run.smithi007.stdout: "last_epoch_clean": 0, 2024-09-18T08:07:31.522 INFO:teuthology.orchestra.run.smithi007.stdout: "source_version": "0'0", 2024-09-18T08:07:31.522 INFO:teuthology.orchestra.run.smithi007.stdout: "target_version": "0'0" 2024-09-18T08:07:31.522 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:07:31.522 INFO:teuthology.orchestra.run.smithi007.stdout: "last_change": "64", 2024-09-18T08:07:31.522 INFO:teuthology.orchestra.run.smithi007.stdout: "last_force_op_resend": "0", 2024-09-18T08:07:31.523 INFO:teuthology.orchestra.run.smithi007.stdout: "last_force_op_resend_prenautilus": "0", 2024-09-18T08:07:31.523 INFO:teuthology.orchestra.run.smithi007.stdout: "last_force_op_resend_preluminous": "0", 2024-09-18T08:07:31.523 INFO:teuthology.orchestra.run.smithi007.stdout: "auid": 0, 2024-09-18T08:07:31.523 INFO:teuthology.orchestra.run.smithi007.stdout: "snap_mode": "selfmanaged", 2024-09-18T08:07:31.523 INFO:teuthology.orchestra.run.smithi007.stdout: "snap_seq": 2, 2024-09-18T08:07:31.523 INFO:teuthology.orchestra.run.smithi007.stdout: "snap_epoch": 64, 2024-09-18T08:07:31.523 INFO:teuthology.orchestra.run.smithi007.stdout: "pool_snaps": [], 2024-09-18T08:07:31.523 INFO:teuthology.orchestra.run.smithi007.stdout: "removed_snaps": "[]", 2024-09-18T08:07:31.523 INFO:teuthology.orchestra.run.smithi007.stdout: "quota_max_bytes": 0, 2024-09-18T08:07:31.523 INFO:teuthology.orchestra.run.smithi007.stdout: "quota_max_objects": 0, 2024-09-18T08:07:31.523 INFO:teuthology.orchestra.run.smithi007.stdout: "tiers": [], 2024-09-18T08:07:31.523 INFO:teuthology.orchestra.run.smithi007.stdout: "tier_of": -1, 2024-09-18T08:07:31.523 INFO:teuthology.orchestra.run.smithi007.stdout: "read_tier": -1, 2024-09-18T08:07:31.524 INFO:teuthology.orchestra.run.smithi007.stdout: "write_tier": -1, 2024-09-18T08:07:31.524 INFO:teuthology.orchestra.run.smithi007.stdout: "cache_mode": "none", 2024-09-18T08:07:31.524 INFO:teuthology.orchestra.run.smithi007.stdout: "target_max_bytes": 0, 2024-09-18T08:07:31.524 INFO:teuthology.orchestra.run.smithi007.stdout: "target_max_objects": 0, 2024-09-18T08:07:31.524 INFO:teuthology.orchestra.run.smithi007.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-09-18T08:07:31.524 INFO:teuthology.orchestra.run.smithi007.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-09-18T08:07:31.524 INFO:teuthology.orchestra.run.smithi007.stdout: "cache_target_full_ratio_micro": 800000, 2024-09-18T08:07:31.524 INFO:teuthology.orchestra.run.smithi007.stdout: "cache_min_flush_age": 0, 2024-09-18T08:07:31.524 INFO:teuthology.orchestra.run.smithi007.stdout: "cache_min_evict_age": 0, 2024-09-18T08:07:31.524 INFO:teuthology.orchestra.run.smithi007.stdout: "erasure_code_profile": "", 2024-09-18T08:07:31.524 INFO:teuthology.orchestra.run.smithi007.stdout: "hit_set_params": { 2024-09-18T08:07:31.524 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "none" 2024-09-18T08:07:31.525 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:07:31.525 INFO:teuthology.orchestra.run.smithi007.stdout: "hit_set_period": 0, 2024-09-18T08:07:31.525 INFO:teuthology.orchestra.run.smithi007.stdout: "hit_set_count": 0, 2024-09-18T08:07:31.525 INFO:teuthology.orchestra.run.smithi007.stdout: "use_gmt_hitset": true, 2024-09-18T08:07:31.525 INFO:teuthology.orchestra.run.smithi007.stdout: "min_read_recency_for_promote": 0, 2024-09-18T08:07:31.525 INFO:teuthology.orchestra.run.smithi007.stdout: "min_write_recency_for_promote": 0, 2024-09-18T08:07:31.525 INFO:teuthology.orchestra.run.smithi007.stdout: "hit_set_grade_decay_rate": 0, 2024-09-18T08:07:31.525 INFO:teuthology.orchestra.run.smithi007.stdout: "hit_set_search_last_n": 0, 2024-09-18T08:07:31.525 INFO:teuthology.orchestra.run.smithi007.stdout: "grade_table": [], 2024-09-18T08:07:31.525 INFO:teuthology.orchestra.run.smithi007.stdout: "stripe_width": 0, 2024-09-18T08:07:31.525 INFO:teuthology.orchestra.run.smithi007.stdout: "expected_num_objects": 0, 2024-09-18T08:07:31.525 INFO:teuthology.orchestra.run.smithi007.stdout: "fast_read": false, 2024-09-18T08:07:31.525 INFO:teuthology.orchestra.run.smithi007.stdout: "options": {}, 2024-09-18T08:07:31.526 INFO:teuthology.orchestra.run.smithi007.stdout: "application_metadata": { 2024-09-18T08:07:31.526 INFO:teuthology.orchestra.run.smithi007.stdout: "rbd": {} 2024-09-18T08:07:31.526 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:07:31.526 INFO:teuthology.orchestra.run.smithi007.stdout: "read_balance": { 2024-09-18T08:07:31.526 INFO:teuthology.orchestra.run.smithi007.stdout: "score_type": "Fair distribution", 2024-09-18T08:07:31.526 INFO:teuthology.orchestra.run.smithi007.stdout: "score_acting": 2.3299999237060547, 2024-09-18T08:07:31.526 INFO:teuthology.orchestra.run.smithi007.stdout: "score_stable": 2.3299999237060547, 2024-09-18T08:07:31.526 INFO:teuthology.orchestra.run.smithi007.stdout: "optimal_score": 1, 2024-09-18T08:07:31.526 INFO:teuthology.orchestra.run.smithi007.stdout: "raw_score_acting": 2.3299999237060547, 2024-09-18T08:07:31.527 INFO:teuthology.orchestra.run.smithi007.stdout: "raw_score_stable": 2.3299999237060547, 2024-09-18T08:07:31.527 INFO:teuthology.orchestra.run.smithi007.stdout: "primary_affinity_weighted": 0.84128189086914062, 2024-09-18T08:07:31.527 INFO:teuthology.orchestra.run.smithi007.stdout: "average_primary_affinity": 0.95999997854232788, 2024-09-18T08:07:31.527 INFO:teuthology.orchestra.run.smithi007.stdout: "average_primary_affinity_weighted": 0.8399999737739563 2024-09-18T08:07:31.527 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-09-18T08:07:31.527 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:07:31.527 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:07:31.527 INFO:teuthology.orchestra.run.smithi007.stdout: "pool": 3, 2024-09-18T08:07:31.527 INFO:teuthology.orchestra.run.smithi007.stdout: "pool_name": ".rgw.root", 2024-09-18T08:07:31.527 INFO:teuthology.orchestra.run.smithi007.stdout: "create_time": "2024-09-18T08:02:46.319947+0000", 2024-09-18T08:07:31.527 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 1, 2024-09-18T08:07:31.527 INFO:teuthology.orchestra.run.smithi007.stdout: "flags_names": "hashpspool", 2024-09-18T08:07:31.528 INFO:teuthology.orchestra.run.smithi007.stdout: "type": 1, 2024-09-18T08:07:31.528 INFO:teuthology.orchestra.run.smithi007.stdout: "size": 3, 2024-09-18T08:07:31.528 INFO:teuthology.orchestra.run.smithi007.stdout: "min_size": 2, 2024-09-18T08:07:31.528 INFO:teuthology.orchestra.run.smithi007.stdout: "crush_rule": 0, 2024-09-18T08:07:31.528 INFO:teuthology.orchestra.run.smithi007.stdout: "peering_crush_bucket_count": 0, 2024-09-18T08:07:31.528 INFO:teuthology.orchestra.run.smithi007.stdout: "peering_crush_bucket_target": 0, 2024-09-18T08:07:31.528 INFO:teuthology.orchestra.run.smithi007.stdout: "peering_crush_bucket_barrier": 0, 2024-09-18T08:07:31.528 INFO:teuthology.orchestra.run.smithi007.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-09-18T08:07:31.528 INFO:teuthology.orchestra.run.smithi007.stdout: "is_stretch_pool": false, 2024-09-18T08:07:31.528 INFO:teuthology.orchestra.run.smithi007.stdout: "object_hash": 2, 2024-09-18T08:07:31.528 INFO:teuthology.orchestra.run.smithi007.stdout: "pg_autoscale_mode": "off", 2024-09-18T08:07:31.528 INFO:teuthology.orchestra.run.smithi007.stdout: "pg_num": 32, 2024-09-18T08:07:31.528 INFO:teuthology.orchestra.run.smithi007.stdout: "pg_placement_num": 32, 2024-09-18T08:07:31.529 INFO:teuthology.orchestra.run.smithi007.stdout: "pg_placement_num_target": 32, 2024-09-18T08:07:31.529 INFO:teuthology.orchestra.run.smithi007.stdout: "pg_num_target": 32, 2024-09-18T08:07:31.529 INFO:teuthology.orchestra.run.smithi007.stdout: "pg_num_pending": 32, 2024-09-18T08:07:31.529 INFO:teuthology.orchestra.run.smithi007.stdout: "last_pg_merge_meta": { 2024-09-18T08:07:31.529 INFO:teuthology.orchestra.run.smithi007.stdout: "source_pgid": "0.0", 2024-09-18T08:07:31.529 INFO:teuthology.orchestra.run.smithi007.stdout: "ready_epoch": 0, 2024-09-18T08:07:31.530 INFO:teuthology.orchestra.run.smithi007.stdout: "last_epoch_started": 0, 2024-09-18T08:07:31.530 INFO:teuthology.orchestra.run.smithi007.stdout: "last_epoch_clean": 0, 2024-09-18T08:07:31.530 INFO:teuthology.orchestra.run.smithi007.stdout: "source_version": "0'0", 2024-09-18T08:07:31.530 INFO:teuthology.orchestra.run.smithi007.stdout: "target_version": "0'0" 2024-09-18T08:07:31.530 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:07:31.530 INFO:teuthology.orchestra.run.smithi007.stdout: "last_change": "63", 2024-09-18T08:07:31.530 INFO:teuthology.orchestra.run.smithi007.stdout: "last_force_op_resend": "0", 2024-09-18T08:07:31.530 INFO:teuthology.orchestra.run.smithi007.stdout: "last_force_op_resend_prenautilus": "0", 2024-09-18T08:07:31.530 INFO:teuthology.orchestra.run.smithi007.stdout: "last_force_op_resend_preluminous": "0", 2024-09-18T08:07:31.530 INFO:teuthology.orchestra.run.smithi007.stdout: "auid": 0, 2024-09-18T08:07:31.530 INFO:teuthology.orchestra.run.smithi007.stdout: "snap_mode": "selfmanaged", 2024-09-18T08:07:31.530 INFO:teuthology.orchestra.run.smithi007.stdout: "snap_seq": 0, 2024-09-18T08:07:31.531 INFO:teuthology.orchestra.run.smithi007.stdout: "snap_epoch": 0, 2024-09-18T08:07:31.531 INFO:teuthology.orchestra.run.smithi007.stdout: "pool_snaps": [], 2024-09-18T08:07:31.531 INFO:teuthology.orchestra.run.smithi007.stdout: "removed_snaps": "[]", 2024-09-18T08:07:31.531 INFO:teuthology.orchestra.run.smithi007.stdout: "quota_max_bytes": 0, 2024-09-18T08:07:31.531 INFO:teuthology.orchestra.run.smithi007.stdout: "quota_max_objects": 0, 2024-09-18T08:07:31.531 INFO:teuthology.orchestra.run.smithi007.stdout: "tiers": [], 2024-09-18T08:07:31.531 INFO:teuthology.orchestra.run.smithi007.stdout: "tier_of": -1, 2024-09-18T08:07:31.531 INFO:teuthology.orchestra.run.smithi007.stdout: "read_tier": -1, 2024-09-18T08:07:31.531 INFO:teuthology.orchestra.run.smithi007.stdout: "write_tier": -1, 2024-09-18T08:07:31.531 INFO:teuthology.orchestra.run.smithi007.stdout: "cache_mode": "none", 2024-09-18T08:07:31.531 INFO:teuthology.orchestra.run.smithi007.stdout: "target_max_bytes": 0, 2024-09-18T08:07:31.531 INFO:teuthology.orchestra.run.smithi007.stdout: "target_max_objects": 0, 2024-09-18T08:07:31.532 INFO:teuthology.orchestra.run.smithi007.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-09-18T08:07:31.532 INFO:teuthology.orchestra.run.smithi007.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-09-18T08:07:31.532 INFO:teuthology.orchestra.run.smithi007.stdout: "cache_target_full_ratio_micro": 800000, 2024-09-18T08:07:31.532 INFO:teuthology.orchestra.run.smithi007.stdout: "cache_min_flush_age": 0, 2024-09-18T08:07:31.532 INFO:teuthology.orchestra.run.smithi007.stdout: "cache_min_evict_age": 0, 2024-09-18T08:07:31.532 INFO:teuthology.orchestra.run.smithi007.stdout: "erasure_code_profile": "", 2024-09-18T08:07:31.532 INFO:teuthology.orchestra.run.smithi007.stdout: "hit_set_params": { 2024-09-18T08:07:31.532 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "none" 2024-09-18T08:07:31.532 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:07:31.532 INFO:teuthology.orchestra.run.smithi007.stdout: "hit_set_period": 0, 2024-09-18T08:07:31.532 INFO:teuthology.orchestra.run.smithi007.stdout: "hit_set_count": 0, 2024-09-18T08:07:31.532 INFO:teuthology.orchestra.run.smithi007.stdout: "use_gmt_hitset": true, 2024-09-18T08:07:31.533 INFO:teuthology.orchestra.run.smithi007.stdout: "min_read_recency_for_promote": 0, 2024-09-18T08:07:31.533 INFO:teuthology.orchestra.run.smithi007.stdout: "min_write_recency_for_promote": 0, 2024-09-18T08:07:31.533 INFO:teuthology.orchestra.run.smithi007.stdout: "hit_set_grade_decay_rate": 0, 2024-09-18T08:07:31.533 INFO:teuthology.orchestra.run.smithi007.stdout: "hit_set_search_last_n": 0, 2024-09-18T08:07:31.533 INFO:teuthology.orchestra.run.smithi007.stdout: "grade_table": [], 2024-09-18T08:07:31.533 INFO:teuthology.orchestra.run.smithi007.stdout: "stripe_width": 0, 2024-09-18T08:07:31.533 INFO:teuthology.orchestra.run.smithi007.stdout: "expected_num_objects": 0, 2024-09-18T08:07:31.533 INFO:teuthology.orchestra.run.smithi007.stdout: "fast_read": false, 2024-09-18T08:07:31.533 INFO:teuthology.orchestra.run.smithi007.stdout: "options": {}, 2024-09-18T08:07:31.533 INFO:teuthology.orchestra.run.smithi007.stdout: "application_metadata": { 2024-09-18T08:07:31.533 INFO:teuthology.orchestra.run.smithi007.stdout: "rgw": {} 2024-09-18T08:07:31.533 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:07:31.534 INFO:teuthology.orchestra.run.smithi007.stdout: "read_balance": { 2024-09-18T08:07:31.534 INFO:teuthology.orchestra.run.smithi007.stdout: "score_type": "Fair distribution", 2024-09-18T08:07:31.534 INFO:teuthology.orchestra.run.smithi007.stdout: "score_acting": 1.5399999618530273, 2024-09-18T08:07:31.534 INFO:teuthology.orchestra.run.smithi007.stdout: "score_stable": 1.5399999618530273, 2024-09-18T08:07:31.534 INFO:teuthology.orchestra.run.smithi007.stdout: "optimal_score": 1.1399999856948853, 2024-09-18T08:07:31.534 INFO:teuthology.orchestra.run.smithi007.stdout: "raw_score_acting": 1.75, 2024-09-18T08:07:31.534 INFO:teuthology.orchestra.run.smithi007.stdout: "raw_score_stable": 1.75, 2024-09-18T08:07:31.534 INFO:teuthology.orchestra.run.smithi007.stdout: "primary_affinity_weighted": 0.84128189086914062, 2024-09-18T08:07:31.534 INFO:teuthology.orchestra.run.smithi007.stdout: "average_primary_affinity": 0.95999997854232788, 2024-09-18T08:07:31.534 INFO:teuthology.orchestra.run.smithi007.stdout: "average_primary_affinity_weighted": 0.8399999737739563 2024-09-18T08:07:31.534 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-09-18T08:07:31.534 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:07:31.534 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:07:31.535 INFO:teuthology.orchestra.run.smithi007.stdout: "pool": 4, 2024-09-18T08:07:31.535 INFO:teuthology.orchestra.run.smithi007.stdout: "pool_name": "default.rgw.log", 2024-09-18T08:07:31.535 INFO:teuthology.orchestra.run.smithi007.stdout: "create_time": "2024-09-18T08:02:47.948552+0000", 2024-09-18T08:07:31.535 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 1, 2024-09-18T08:07:31.535 INFO:teuthology.orchestra.run.smithi007.stdout: "flags_names": "hashpspool", 2024-09-18T08:07:31.535 INFO:teuthology.orchestra.run.smithi007.stdout: "type": 1, 2024-09-18T08:07:31.535 INFO:teuthology.orchestra.run.smithi007.stdout: "size": 3, 2024-09-18T08:07:31.535 INFO:teuthology.orchestra.run.smithi007.stdout: "min_size": 2, 2024-09-18T08:07:31.535 INFO:teuthology.orchestra.run.smithi007.stdout: "crush_rule": 0, 2024-09-18T08:07:31.535 INFO:teuthology.orchestra.run.smithi007.stdout: "peering_crush_bucket_count": 0, 2024-09-18T08:07:31.535 INFO:teuthology.orchestra.run.smithi007.stdout: "peering_crush_bucket_target": 0, 2024-09-18T08:07:31.535 INFO:teuthology.orchestra.run.smithi007.stdout: "peering_crush_bucket_barrier": 0, 2024-09-18T08:07:31.535 INFO:teuthology.orchestra.run.smithi007.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-09-18T08:07:31.536 INFO:teuthology.orchestra.run.smithi007.stdout: "is_stretch_pool": false, 2024-09-18T08:07:31.536 INFO:teuthology.orchestra.run.smithi007.stdout: "object_hash": 2, 2024-09-18T08:07:31.536 INFO:teuthology.orchestra.run.smithi007.stdout: "pg_autoscale_mode": "off", 2024-09-18T08:07:31.536 INFO:teuthology.orchestra.run.smithi007.stdout: "pg_num": 32, 2024-09-18T08:07:31.536 INFO:teuthology.orchestra.run.smithi007.stdout: "pg_placement_num": 32, 2024-09-18T08:07:31.536 INFO:teuthology.orchestra.run.smithi007.stdout: "pg_placement_num_target": 32, 2024-09-18T08:07:31.536 INFO:teuthology.orchestra.run.smithi007.stdout: "pg_num_target": 32, 2024-09-18T08:07:31.536 INFO:teuthology.orchestra.run.smithi007.stdout: "pg_num_pending": 32, 2024-09-18T08:07:31.536 INFO:teuthology.orchestra.run.smithi007.stdout: "last_pg_merge_meta": { 2024-09-18T08:07:31.536 INFO:teuthology.orchestra.run.smithi007.stdout: "source_pgid": "0.0", 2024-09-18T08:07:31.536 INFO:teuthology.orchestra.run.smithi007.stdout: "ready_epoch": 0, 2024-09-18T08:07:31.536 INFO:teuthology.orchestra.run.smithi007.stdout: "last_epoch_started": 0, 2024-09-18T08:07:31.537 INFO:teuthology.orchestra.run.smithi007.stdout: "last_epoch_clean": 0, 2024-09-18T08:07:31.537 INFO:teuthology.orchestra.run.smithi007.stdout: "source_version": "0'0", 2024-09-18T08:07:31.537 INFO:teuthology.orchestra.run.smithi007.stdout: "target_version": "0'0" 2024-09-18T08:07:31.537 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:07:31.537 INFO:teuthology.orchestra.run.smithi007.stdout: "last_change": "65", 2024-09-18T08:07:31.537 INFO:teuthology.orchestra.run.smithi007.stdout: "last_force_op_resend": "0", 2024-09-18T08:07:31.537 INFO:teuthology.orchestra.run.smithi007.stdout: "last_force_op_resend_prenautilus": "0", 2024-09-18T08:07:31.537 INFO:teuthology.orchestra.run.smithi007.stdout: "last_force_op_resend_preluminous": "0", 2024-09-18T08:07:31.537 INFO:teuthology.orchestra.run.smithi007.stdout: "auid": 0, 2024-09-18T08:07:31.537 INFO:teuthology.orchestra.run.smithi007.stdout: "snap_mode": "selfmanaged", 2024-09-18T08:07:31.537 INFO:teuthology.orchestra.run.smithi007.stdout: "snap_seq": 0, 2024-09-18T08:07:31.537 INFO:teuthology.orchestra.run.smithi007.stdout: "snap_epoch": 0, 2024-09-18T08:07:31.538 INFO:teuthology.orchestra.run.smithi007.stdout: "pool_snaps": [], 2024-09-18T08:07:31.538 INFO:teuthology.orchestra.run.smithi007.stdout: "removed_snaps": "[]", 2024-09-18T08:07:31.538 INFO:teuthology.orchestra.run.smithi007.stdout: "quota_max_bytes": 0, 2024-09-18T08:07:31.538 INFO:teuthology.orchestra.run.smithi007.stdout: "quota_max_objects": 0, 2024-09-18T08:07:31.538 INFO:teuthology.orchestra.run.smithi007.stdout: "tiers": [], 2024-09-18T08:07:31.538 INFO:teuthology.orchestra.run.smithi007.stdout: "tier_of": -1, 2024-09-18T08:07:31.538 INFO:teuthology.orchestra.run.smithi007.stdout: "read_tier": -1, 2024-09-18T08:07:31.538 INFO:teuthology.orchestra.run.smithi007.stdout: "write_tier": -1, 2024-09-18T08:07:31.538 INFO:teuthology.orchestra.run.smithi007.stdout: "cache_mode": "none", 2024-09-18T08:07:31.538 INFO:teuthology.orchestra.run.smithi007.stdout: "target_max_bytes": 0, 2024-09-18T08:07:31.538 INFO:teuthology.orchestra.run.smithi007.stdout: "target_max_objects": 0, 2024-09-18T08:07:31.538 INFO:teuthology.orchestra.run.smithi007.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-09-18T08:07:31.538 INFO:teuthology.orchestra.run.smithi007.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-09-18T08:07:31.538 INFO:teuthology.orchestra.run.smithi007.stdout: "cache_target_full_ratio_micro": 800000, 2024-09-18T08:07:31.539 INFO:teuthology.orchestra.run.smithi007.stdout: "cache_min_flush_age": 0, 2024-09-18T08:07:31.539 INFO:teuthology.orchestra.run.smithi007.stdout: "cache_min_evict_age": 0, 2024-09-18T08:07:31.539 INFO:teuthology.orchestra.run.smithi007.stdout: "erasure_code_profile": "", 2024-09-18T08:07:31.539 INFO:teuthology.orchestra.run.smithi007.stdout: "hit_set_params": { 2024-09-18T08:07:31.539 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "none" 2024-09-18T08:07:31.539 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:07:31.539 INFO:teuthology.orchestra.run.smithi007.stdout: "hit_set_period": 0, 2024-09-18T08:07:31.539 INFO:teuthology.orchestra.run.smithi007.stdout: "hit_set_count": 0, 2024-09-18T08:07:31.539 INFO:teuthology.orchestra.run.smithi007.stdout: "use_gmt_hitset": true, 2024-09-18T08:07:31.540 INFO:teuthology.orchestra.run.smithi007.stdout: "min_read_recency_for_promote": 0, 2024-09-18T08:07:31.540 INFO:teuthology.orchestra.run.smithi007.stdout: "min_write_recency_for_promote": 0, 2024-09-18T08:07:31.540 INFO:teuthology.orchestra.run.smithi007.stdout: "hit_set_grade_decay_rate": 0, 2024-09-18T08:07:31.540 INFO:teuthology.orchestra.run.smithi007.stdout: "hit_set_search_last_n": 0, 2024-09-18T08:07:31.540 INFO:teuthology.orchestra.run.smithi007.stdout: "grade_table": [], 2024-09-18T08:07:31.540 INFO:teuthology.orchestra.run.smithi007.stdout: "stripe_width": 0, 2024-09-18T08:07:31.540 INFO:teuthology.orchestra.run.smithi007.stdout: "expected_num_objects": 0, 2024-09-18T08:07:31.540 INFO:teuthology.orchestra.run.smithi007.stdout: "fast_read": false, 2024-09-18T08:07:31.540 INFO:teuthology.orchestra.run.smithi007.stdout: "options": {}, 2024-09-18T08:07:31.540 INFO:teuthology.orchestra.run.smithi007.stdout: "application_metadata": { 2024-09-18T08:07:31.540 INFO:teuthology.orchestra.run.smithi007.stdout: "rgw": {} 2024-09-18T08:07:31.541 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:07:31.541 INFO:teuthology.orchestra.run.smithi007.stdout: "read_balance": { 2024-09-18T08:07:31.541 INFO:teuthology.orchestra.run.smithi007.stdout: "score_type": "Fair distribution", 2024-09-18T08:07:31.541 INFO:teuthology.orchestra.run.smithi007.stdout: "score_acting": 2.190000057220459, 2024-09-18T08:07:31.541 INFO:teuthology.orchestra.run.smithi007.stdout: "score_stable": 2.190000057220459, 2024-09-18T08:07:31.541 INFO:teuthology.orchestra.run.smithi007.stdout: "optimal_score": 1.1399999856948853, 2024-09-18T08:07:31.541 INFO:teuthology.orchestra.run.smithi007.stdout: "raw_score_acting": 2.5, 2024-09-18T08:07:31.541 INFO:teuthology.orchestra.run.smithi007.stdout: "raw_score_stable": 2.5, 2024-09-18T08:07:31.541 INFO:teuthology.orchestra.run.smithi007.stdout: "primary_affinity_weighted": 0.84128189086914062, 2024-09-18T08:07:31.541 INFO:teuthology.orchestra.run.smithi007.stdout: "average_primary_affinity": 0.95999997854232788, 2024-09-18T08:07:31.541 INFO:teuthology.orchestra.run.smithi007.stdout: "average_primary_affinity_weighted": 0.8399999737739563 2024-09-18T08:07:31.541 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-09-18T08:07:31.541 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:07:31.541 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:07:31.542 INFO:teuthology.orchestra.run.smithi007.stdout: "pool": 5, 2024-09-18T08:07:31.542 INFO:teuthology.orchestra.run.smithi007.stdout: "pool_name": "default.rgw.control", 2024-09-18T08:07:31.542 INFO:teuthology.orchestra.run.smithi007.stdout: "create_time": "2024-09-18T08:02:49.904302+0000", 2024-09-18T08:07:31.542 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 1, 2024-09-18T08:07:31.542 INFO:teuthology.orchestra.run.smithi007.stdout: "flags_names": "hashpspool", 2024-09-18T08:07:31.542 INFO:teuthology.orchestra.run.smithi007.stdout: "type": 1, 2024-09-18T08:07:31.542 INFO:teuthology.orchestra.run.smithi007.stdout: "size": 3, 2024-09-18T08:07:31.542 INFO:teuthology.orchestra.run.smithi007.stdout: "min_size": 2, 2024-09-18T08:07:31.542 INFO:teuthology.orchestra.run.smithi007.stdout: "crush_rule": 0, 2024-09-18T08:07:31.542 INFO:teuthology.orchestra.run.smithi007.stdout: "peering_crush_bucket_count": 0, 2024-09-18T08:07:31.542 INFO:teuthology.orchestra.run.smithi007.stdout: "peering_crush_bucket_target": 0, 2024-09-18T08:07:31.542 INFO:teuthology.orchestra.run.smithi007.stdout: "peering_crush_bucket_barrier": 0, 2024-09-18T08:07:31.542 INFO:teuthology.orchestra.run.smithi007.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-09-18T08:07:31.543 INFO:teuthology.orchestra.run.smithi007.stdout: "is_stretch_pool": false, 2024-09-18T08:07:31.543 INFO:teuthology.orchestra.run.smithi007.stdout: "object_hash": 2, 2024-09-18T08:07:31.543 INFO:teuthology.orchestra.run.smithi007.stdout: "pg_autoscale_mode": "off", 2024-09-18T08:07:31.543 INFO:teuthology.orchestra.run.smithi007.stdout: "pg_num": 32, 2024-09-18T08:07:31.543 INFO:teuthology.orchestra.run.smithi007.stdout: "pg_placement_num": 32, 2024-09-18T08:07:31.543 INFO:teuthology.orchestra.run.smithi007.stdout: "pg_placement_num_target": 32, 2024-09-18T08:07:31.543 INFO:teuthology.orchestra.run.smithi007.stdout: "pg_num_target": 32, 2024-09-18T08:07:31.543 INFO:teuthology.orchestra.run.smithi007.stdout: "pg_num_pending": 32, 2024-09-18T08:07:31.543 INFO:teuthology.orchestra.run.smithi007.stdout: "last_pg_merge_meta": { 2024-09-18T08:07:31.543 INFO:teuthology.orchestra.run.smithi007.stdout: "source_pgid": "0.0", 2024-09-18T08:07:31.543 INFO:teuthology.orchestra.run.smithi007.stdout: "ready_epoch": 0, 2024-09-18T08:07:31.543 INFO:teuthology.orchestra.run.smithi007.stdout: "last_epoch_started": 0, 2024-09-18T08:07:31.543 INFO:teuthology.orchestra.run.smithi007.stdout: "last_epoch_clean": 0, 2024-09-18T08:07:31.544 INFO:teuthology.orchestra.run.smithi007.stdout: "source_version": "0'0", 2024-09-18T08:07:31.544 INFO:teuthology.orchestra.run.smithi007.stdout: "target_version": "0'0" 2024-09-18T08:07:31.544 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:07:31.544 INFO:teuthology.orchestra.run.smithi007.stdout: "last_change": "67", 2024-09-18T08:07:31.544 INFO:teuthology.orchestra.run.smithi007.stdout: "last_force_op_resend": "0", 2024-09-18T08:07:31.544 INFO:teuthology.orchestra.run.smithi007.stdout: "last_force_op_resend_prenautilus": "0", 2024-09-18T08:07:31.544 INFO:teuthology.orchestra.run.smithi007.stdout: "last_force_op_resend_preluminous": "0", 2024-09-18T08:07:31.544 INFO:teuthology.orchestra.run.smithi007.stdout: "auid": 0, 2024-09-18T08:07:31.544 INFO:teuthology.orchestra.run.smithi007.stdout: "snap_mode": "selfmanaged", 2024-09-18T08:07:31.544 INFO:teuthology.orchestra.run.smithi007.stdout: "snap_seq": 0, 2024-09-18T08:07:31.544 INFO:teuthology.orchestra.run.smithi007.stdout: "snap_epoch": 0, 2024-09-18T08:07:31.544 INFO:teuthology.orchestra.run.smithi007.stdout: "pool_snaps": [], 2024-09-18T08:07:31.545 INFO:teuthology.orchestra.run.smithi007.stdout: "removed_snaps": "[]", 2024-09-18T08:07:31.545 INFO:teuthology.orchestra.run.smithi007.stdout: "quota_max_bytes": 0, 2024-09-18T08:07:31.545 INFO:teuthology.orchestra.run.smithi007.stdout: "quota_max_objects": 0, 2024-09-18T08:07:31.545 INFO:teuthology.orchestra.run.smithi007.stdout: "tiers": [], 2024-09-18T08:07:31.545 INFO:teuthology.orchestra.run.smithi007.stdout: "tier_of": -1, 2024-09-18T08:07:31.545 INFO:teuthology.orchestra.run.smithi007.stdout: "read_tier": -1, 2024-09-18T08:07:31.545 INFO:teuthology.orchestra.run.smithi007.stdout: "write_tier": -1, 2024-09-18T08:07:31.545 INFO:teuthology.orchestra.run.smithi007.stdout: "cache_mode": "none", 2024-09-18T08:07:31.545 INFO:teuthology.orchestra.run.smithi007.stdout: "target_max_bytes": 0, 2024-09-18T08:07:31.545 INFO:teuthology.orchestra.run.smithi007.stdout: "target_max_objects": 0, 2024-09-18T08:07:31.545 INFO:teuthology.orchestra.run.smithi007.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-09-18T08:07:31.545 INFO:teuthology.orchestra.run.smithi007.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-09-18T08:07:31.545 INFO:teuthology.orchestra.run.smithi007.stdout: "cache_target_full_ratio_micro": 800000, 2024-09-18T08:07:31.546 INFO:teuthology.orchestra.run.smithi007.stdout: "cache_min_flush_age": 0, 2024-09-18T08:07:31.546 INFO:teuthology.orchestra.run.smithi007.stdout: "cache_min_evict_age": 0, 2024-09-18T08:07:31.546 INFO:teuthology.orchestra.run.smithi007.stdout: "erasure_code_profile": "", 2024-09-18T08:07:31.546 INFO:teuthology.orchestra.run.smithi007.stdout: "hit_set_params": { 2024-09-18T08:07:31.546 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "none" 2024-09-18T08:07:31.546 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:07:31.546 INFO:teuthology.orchestra.run.smithi007.stdout: "hit_set_period": 0, 2024-09-18T08:07:31.546 INFO:teuthology.orchestra.run.smithi007.stdout: "hit_set_count": 0, 2024-09-18T08:07:31.546 INFO:teuthology.orchestra.run.smithi007.stdout: "use_gmt_hitset": true, 2024-09-18T08:07:31.546 INFO:teuthology.orchestra.run.smithi007.stdout: "min_read_recency_for_promote": 0, 2024-09-18T08:07:31.546 INFO:teuthology.orchestra.run.smithi007.stdout: "min_write_recency_for_promote": 0, 2024-09-18T08:07:31.546 INFO:teuthology.orchestra.run.smithi007.stdout: "hit_set_grade_decay_rate": 0, 2024-09-18T08:07:31.547 INFO:teuthology.orchestra.run.smithi007.stdout: "hit_set_search_last_n": 0, 2024-09-18T08:07:31.547 INFO:teuthology.orchestra.run.smithi007.stdout: "grade_table": [], 2024-09-18T08:07:31.547 INFO:teuthology.orchestra.run.smithi007.stdout: "stripe_width": 0, 2024-09-18T08:07:31.547 INFO:teuthology.orchestra.run.smithi007.stdout: "expected_num_objects": 0, 2024-09-18T08:07:31.547 INFO:teuthology.orchestra.run.smithi007.stdout: "fast_read": false, 2024-09-18T08:07:31.547 INFO:teuthology.orchestra.run.smithi007.stdout: "options": {}, 2024-09-18T08:07:31.547 INFO:teuthology.orchestra.run.smithi007.stdout: "application_metadata": { 2024-09-18T08:07:31.547 INFO:teuthology.orchestra.run.smithi007.stdout: "rgw": {} 2024-09-18T08:07:31.547 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:07:31.547 INFO:teuthology.orchestra.run.smithi007.stdout: "read_balance": { 2024-09-18T08:07:31.547 INFO:teuthology.orchestra.run.smithi007.stdout: "score_type": "Fair distribution", 2024-09-18T08:07:31.547 INFO:teuthology.orchestra.run.smithi007.stdout: "score_acting": 1.3200000524520874, 2024-09-18T08:07:31.547 INFO:teuthology.orchestra.run.smithi007.stdout: "score_stable": 1.3200000524520874, 2024-09-18T08:07:31.548 INFO:teuthology.orchestra.run.smithi007.stdout: "optimal_score": 1.1399999856948853, 2024-09-18T08:07:31.548 INFO:teuthology.orchestra.run.smithi007.stdout: "raw_score_acting": 1.5, 2024-09-18T08:07:31.548 INFO:teuthology.orchestra.run.smithi007.stdout: "raw_score_stable": 1.5, 2024-09-18T08:07:31.548 INFO:teuthology.orchestra.run.smithi007.stdout: "primary_affinity_weighted": 0.84128189086914062, 2024-09-18T08:07:31.548 INFO:teuthology.orchestra.run.smithi007.stdout: "average_primary_affinity": 0.95999997854232788, 2024-09-18T08:07:31.548 INFO:teuthology.orchestra.run.smithi007.stdout: "average_primary_affinity_weighted": 0.8399999737739563 2024-09-18T08:07:31.548 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-09-18T08:07:31.548 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:07:31.548 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:07:31.548 INFO:teuthology.orchestra.run.smithi007.stdout: "pool": 6, 2024-09-18T08:07:31.548 INFO:teuthology.orchestra.run.smithi007.stdout: "pool_name": "default.rgw.meta", 2024-09-18T08:07:31.548 INFO:teuthology.orchestra.run.smithi007.stdout: "create_time": "2024-09-18T08:02:51.971465+0000", 2024-09-18T08:07:31.548 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 1, 2024-09-18T08:07:31.548 INFO:teuthology.orchestra.run.smithi007.stdout: "flags_names": "hashpspool", 2024-09-18T08:07:31.549 INFO:teuthology.orchestra.run.smithi007.stdout: "type": 1, 2024-09-18T08:07:31.549 INFO:teuthology.orchestra.run.smithi007.stdout: "size": 3, 2024-09-18T08:07:31.549 INFO:teuthology.orchestra.run.smithi007.stdout: "min_size": 2, 2024-09-18T08:07:31.549 INFO:teuthology.orchestra.run.smithi007.stdout: "crush_rule": 0, 2024-09-18T08:07:31.549 INFO:teuthology.orchestra.run.smithi007.stdout: "peering_crush_bucket_count": 0, 2024-09-18T08:07:31.549 INFO:teuthology.orchestra.run.smithi007.stdout: "peering_crush_bucket_target": 0, 2024-09-18T08:07:31.549 INFO:teuthology.orchestra.run.smithi007.stdout: "peering_crush_bucket_barrier": 0, 2024-09-18T08:07:31.549 INFO:teuthology.orchestra.run.smithi007.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-09-18T08:07:31.549 INFO:teuthology.orchestra.run.smithi007.stdout: "is_stretch_pool": false, 2024-09-18T08:07:31.549 INFO:teuthology.orchestra.run.smithi007.stdout: "object_hash": 2, 2024-09-18T08:07:31.549 INFO:teuthology.orchestra.run.smithi007.stdout: "pg_autoscale_mode": "off", 2024-09-18T08:07:31.549 INFO:teuthology.orchestra.run.smithi007.stdout: "pg_num": 32, 2024-09-18T08:07:31.550 INFO:teuthology.orchestra.run.smithi007.stdout: "pg_placement_num": 32, 2024-09-18T08:07:31.550 INFO:teuthology.orchestra.run.smithi007.stdout: "pg_placement_num_target": 32, 2024-09-18T08:07:31.550 INFO:teuthology.orchestra.run.smithi007.stdout: "pg_num_target": 32, 2024-09-18T08:07:31.550 INFO:teuthology.orchestra.run.smithi007.stdout: "pg_num_pending": 32, 2024-09-18T08:07:31.550 INFO:teuthology.orchestra.run.smithi007.stdout: "last_pg_merge_meta": { 2024-09-18T08:07:31.550 INFO:teuthology.orchestra.run.smithi007.stdout: "source_pgid": "0.0", 2024-09-18T08:07:31.550 INFO:teuthology.orchestra.run.smithi007.stdout: "ready_epoch": 0, 2024-09-18T08:07:31.550 INFO:teuthology.orchestra.run.smithi007.stdout: "last_epoch_started": 0, 2024-09-18T08:07:31.550 INFO:teuthology.orchestra.run.smithi007.stdout: "last_epoch_clean": 0, 2024-09-18T08:07:31.550 INFO:teuthology.orchestra.run.smithi007.stdout: "source_version": "0'0", 2024-09-18T08:07:31.550 INFO:teuthology.orchestra.run.smithi007.stdout: "target_version": "0'0" 2024-09-18T08:07:31.550 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:07:31.550 INFO:teuthology.orchestra.run.smithi007.stdout: "last_change": "70", 2024-09-18T08:07:31.551 INFO:teuthology.orchestra.run.smithi007.stdout: "last_force_op_resend": "0", 2024-09-18T08:07:31.551 INFO:teuthology.orchestra.run.smithi007.stdout: "last_force_op_resend_prenautilus": "0", 2024-09-18T08:07:31.551 INFO:teuthology.orchestra.run.smithi007.stdout: "last_force_op_resend_preluminous": "0", 2024-09-18T08:07:31.551 INFO:teuthology.orchestra.run.smithi007.stdout: "auid": 0, 2024-09-18T08:07:31.551 INFO:teuthology.orchestra.run.smithi007.stdout: "snap_mode": "selfmanaged", 2024-09-18T08:07:31.551 INFO:teuthology.orchestra.run.smithi007.stdout: "snap_seq": 0, 2024-09-18T08:07:31.551 INFO:teuthology.orchestra.run.smithi007.stdout: "snap_epoch": 0, 2024-09-18T08:07:31.551 INFO:teuthology.orchestra.run.smithi007.stdout: "pool_snaps": [], 2024-09-18T08:07:31.551 INFO:teuthology.orchestra.run.smithi007.stdout: "removed_snaps": "[]", 2024-09-18T08:07:31.551 INFO:teuthology.orchestra.run.smithi007.stdout: "quota_max_bytes": 0, 2024-09-18T08:07:31.551 INFO:teuthology.orchestra.run.smithi007.stdout: "quota_max_objects": 0, 2024-09-18T08:07:31.551 INFO:teuthology.orchestra.run.smithi007.stdout: "tiers": [], 2024-09-18T08:07:31.551 INFO:teuthology.orchestra.run.smithi007.stdout: "tier_of": -1, 2024-09-18T08:07:31.551 INFO:teuthology.orchestra.run.smithi007.stdout: "read_tier": -1, 2024-09-18T08:07:31.552 INFO:teuthology.orchestra.run.smithi007.stdout: "write_tier": -1, 2024-09-18T08:07:31.552 INFO:teuthology.orchestra.run.smithi007.stdout: "cache_mode": "none", 2024-09-18T08:07:31.552 INFO:teuthology.orchestra.run.smithi007.stdout: "target_max_bytes": 0, 2024-09-18T08:07:31.552 INFO:teuthology.orchestra.run.smithi007.stdout: "target_max_objects": 0, 2024-09-18T08:07:31.552 INFO:teuthology.orchestra.run.smithi007.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-09-18T08:07:31.552 INFO:teuthology.orchestra.run.smithi007.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-09-18T08:07:31.552 INFO:teuthology.orchestra.run.smithi007.stdout: "cache_target_full_ratio_micro": 800000, 2024-09-18T08:07:31.552 INFO:teuthology.orchestra.run.smithi007.stdout: "cache_min_flush_age": 0, 2024-09-18T08:07:31.552 INFO:teuthology.orchestra.run.smithi007.stdout: "cache_min_evict_age": 0, 2024-09-18T08:07:31.552 INFO:teuthology.orchestra.run.smithi007.stdout: "erasure_code_profile": "", 2024-09-18T08:07:31.552 INFO:teuthology.orchestra.run.smithi007.stdout: "hit_set_params": { 2024-09-18T08:07:31.552 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "none" 2024-09-18T08:07:31.553 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:07:31.553 INFO:teuthology.orchestra.run.smithi007.stdout: "hit_set_period": 0, 2024-09-18T08:07:31.553 INFO:teuthology.orchestra.run.smithi007.stdout: "hit_set_count": 0, 2024-09-18T08:07:31.553 INFO:teuthology.orchestra.run.smithi007.stdout: "use_gmt_hitset": true, 2024-09-18T08:07:31.553 INFO:teuthology.orchestra.run.smithi007.stdout: "min_read_recency_for_promote": 0, 2024-09-18T08:07:31.553 INFO:teuthology.orchestra.run.smithi007.stdout: "min_write_recency_for_promote": 0, 2024-09-18T08:07:31.553 INFO:teuthology.orchestra.run.smithi007.stdout: "hit_set_grade_decay_rate": 0, 2024-09-18T08:07:31.553 INFO:teuthology.orchestra.run.smithi007.stdout: "hit_set_search_last_n": 0, 2024-09-18T08:07:31.553 INFO:teuthology.orchestra.run.smithi007.stdout: "grade_table": [], 2024-09-18T08:07:31.553 INFO:teuthology.orchestra.run.smithi007.stdout: "stripe_width": 0, 2024-09-18T08:07:31.553 INFO:teuthology.orchestra.run.smithi007.stdout: "expected_num_objects": 0, 2024-09-18T08:07:31.553 INFO:teuthology.orchestra.run.smithi007.stdout: "fast_read": false, 2024-09-18T08:07:31.553 INFO:teuthology.orchestra.run.smithi007.stdout: "options": { 2024-09-18T08:07:31.554 INFO:teuthology.orchestra.run.smithi007.stdout: "pg_autoscale_bias": 4 2024-09-18T08:07:31.554 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:07:31.554 INFO:teuthology.orchestra.run.smithi007.stdout: "application_metadata": { 2024-09-18T08:07:31.554 INFO:teuthology.orchestra.run.smithi007.stdout: "rgw": {} 2024-09-18T08:07:31.554 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:07:31.554 INFO:teuthology.orchestra.run.smithi007.stdout: "read_balance": { 2024-09-18T08:07:31.554 INFO:teuthology.orchestra.run.smithi007.stdout: "score_type": "Fair distribution", 2024-09-18T08:07:31.554 INFO:teuthology.orchestra.run.smithi007.stdout: "score_acting": 1.5399999618530273, 2024-09-18T08:07:31.554 INFO:teuthology.orchestra.run.smithi007.stdout: "score_stable": 1.5399999618530273, 2024-09-18T08:07:31.554 INFO:teuthology.orchestra.run.smithi007.stdout: "optimal_score": 1.1399999856948853, 2024-09-18T08:07:31.554 INFO:teuthology.orchestra.run.smithi007.stdout: "raw_score_acting": 1.75, 2024-09-18T08:07:31.554 INFO:teuthology.orchestra.run.smithi007.stdout: "raw_score_stable": 1.75, 2024-09-18T08:07:31.555 INFO:teuthology.orchestra.run.smithi007.stdout: "primary_affinity_weighted": 0.84128189086914062, 2024-09-18T08:07:31.555 INFO:teuthology.orchestra.run.smithi007.stdout: "average_primary_affinity": 0.95999997854232788, 2024-09-18T08:07:31.555 INFO:teuthology.orchestra.run.smithi007.stdout: "average_primary_affinity_weighted": 0.8399999737739563 2024-09-18T08:07:31.555 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-09-18T08:07:31.555 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:07:31.555 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:07:31.555 INFO:teuthology.orchestra.run.smithi007.stdout: "pool": 7, 2024-09-18T08:07:31.555 INFO:teuthology.orchestra.run.smithi007.stdout: "pool_name": "unique_pool_0", 2024-09-18T08:07:31.555 INFO:teuthology.orchestra.run.smithi007.stdout: "create_time": "2024-09-18T08:03:55.823788+0000", 2024-09-18T08:07:31.555 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 1, 2024-09-18T08:07:31.555 INFO:teuthology.orchestra.run.smithi007.stdout: "flags_names": "hashpspool", 2024-09-18T08:07:31.555 INFO:teuthology.orchestra.run.smithi007.stdout: "type": 1, 2024-09-18T08:07:31.555 INFO:teuthology.orchestra.run.smithi007.stdout: "size": 3, 2024-09-18T08:07:31.556 INFO:teuthology.orchestra.run.smithi007.stdout: "min_size": 2, 2024-09-18T08:07:31.556 INFO:teuthology.orchestra.run.smithi007.stdout: "crush_rule": 0, 2024-09-18T08:07:31.556 INFO:teuthology.orchestra.run.smithi007.stdout: "peering_crush_bucket_count": 0, 2024-09-18T08:07:31.556 INFO:teuthology.orchestra.run.smithi007.stdout: "peering_crush_bucket_target": 0, 2024-09-18T08:07:31.556 INFO:teuthology.orchestra.run.smithi007.stdout: "peering_crush_bucket_barrier": 0, 2024-09-18T08:07:31.556 INFO:teuthology.orchestra.run.smithi007.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-09-18T08:07:31.556 INFO:teuthology.orchestra.run.smithi007.stdout: "is_stretch_pool": false, 2024-09-18T08:07:31.556 INFO:teuthology.orchestra.run.smithi007.stdout: "object_hash": 2, 2024-09-18T08:07:31.556 INFO:teuthology.orchestra.run.smithi007.stdout: "pg_autoscale_mode": "off", 2024-09-18T08:07:31.556 INFO:teuthology.orchestra.run.smithi007.stdout: "pg_num": 16, 2024-09-18T08:07:31.556 INFO:teuthology.orchestra.run.smithi007.stdout: "pg_placement_num": 16, 2024-09-18T08:07:31.556 INFO:teuthology.orchestra.run.smithi007.stdout: "pg_placement_num_target": 16, 2024-09-18T08:07:31.557 INFO:teuthology.orchestra.run.smithi007.stdout: "pg_num_target": 16, 2024-09-18T08:07:31.557 INFO:teuthology.orchestra.run.smithi007.stdout: "pg_num_pending": 16, 2024-09-18T08:07:31.557 INFO:teuthology.orchestra.run.smithi007.stdout: "last_pg_merge_meta": { 2024-09-18T08:07:31.557 INFO:teuthology.orchestra.run.smithi007.stdout: "source_pgid": "0.0", 2024-09-18T08:07:31.557 INFO:teuthology.orchestra.run.smithi007.stdout: "ready_epoch": 0, 2024-09-18T08:07:31.557 INFO:teuthology.orchestra.run.smithi007.stdout: "last_epoch_started": 0, 2024-09-18T08:07:31.557 INFO:teuthology.orchestra.run.smithi007.stdout: "last_epoch_clean": 0, 2024-09-18T08:07:31.557 INFO:teuthology.orchestra.run.smithi007.stdout: "source_version": "0'0", 2024-09-18T08:07:31.557 INFO:teuthology.orchestra.run.smithi007.stdout: "target_version": "0'0" 2024-09-18T08:07:31.557 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:07:31.557 INFO:teuthology.orchestra.run.smithi007.stdout: "last_change": "88", 2024-09-18T08:07:31.557 INFO:teuthology.orchestra.run.smithi007.stdout: "last_force_op_resend": "0", 2024-09-18T08:07:31.558 INFO:teuthology.orchestra.run.smithi007.stdout: "last_force_op_resend_prenautilus": "0", 2024-09-18T08:07:31.558 INFO:teuthology.orchestra.run.smithi007.stdout: "last_force_op_resend_preluminous": "0", 2024-09-18T08:07:31.558 INFO:teuthology.orchestra.run.smithi007.stdout: "auid": 0, 2024-09-18T08:07:31.558 INFO:teuthology.orchestra.run.smithi007.stdout: "snap_mode": "selfmanaged", 2024-09-18T08:07:31.558 INFO:teuthology.orchestra.run.smithi007.stdout: "snap_seq": 0, 2024-09-18T08:07:31.558 INFO:teuthology.orchestra.run.smithi007.stdout: "snap_epoch": 0, 2024-09-18T08:07:31.558 INFO:teuthology.orchestra.run.smithi007.stdout: "pool_snaps": [], 2024-09-18T08:07:31.558 INFO:teuthology.orchestra.run.smithi007.stdout: "removed_snaps": "[]", 2024-09-18T08:07:31.558 INFO:teuthology.orchestra.run.smithi007.stdout: "quota_max_bytes": 0, 2024-09-18T08:07:31.558 INFO:teuthology.orchestra.run.smithi007.stdout: "quota_max_objects": 0, 2024-09-18T08:07:31.558 INFO:teuthology.orchestra.run.smithi007.stdout: "tiers": [], 2024-09-18T08:07:31.558 INFO:teuthology.orchestra.run.smithi007.stdout: "tier_of": -1, 2024-09-18T08:07:31.559 INFO:teuthology.orchestra.run.smithi007.stdout: "read_tier": -1, 2024-09-18T08:07:31.559 INFO:teuthology.orchestra.run.smithi007.stdout: "write_tier": -1, 2024-09-18T08:07:31.559 INFO:teuthology.orchestra.run.smithi007.stdout: "cache_mode": "none", 2024-09-18T08:07:31.559 INFO:teuthology.orchestra.run.smithi007.stdout: "target_max_bytes": 0, 2024-09-18T08:07:31.559 INFO:teuthology.orchestra.run.smithi007.stdout: "target_max_objects": 0, 2024-09-18T08:07:31.559 INFO:teuthology.orchestra.run.smithi007.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-09-18T08:07:31.559 INFO:teuthology.orchestra.run.smithi007.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-09-18T08:07:31.559 INFO:teuthology.orchestra.run.smithi007.stdout: "cache_target_full_ratio_micro": 800000, 2024-09-18T08:07:31.559 INFO:teuthology.orchestra.run.smithi007.stdout: "cache_min_flush_age": 0, 2024-09-18T08:07:31.559 INFO:teuthology.orchestra.run.smithi007.stdout: "cache_min_evict_age": 0, 2024-09-18T08:07:31.559 INFO:teuthology.orchestra.run.smithi007.stdout: "erasure_code_profile": "", 2024-09-18T08:07:31.559 INFO:teuthology.orchestra.run.smithi007.stdout: "hit_set_params": { 2024-09-18T08:07:31.559 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "none" 2024-09-18T08:07:31.560 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:07:31.560 INFO:teuthology.orchestra.run.smithi007.stdout: "hit_set_period": 0, 2024-09-18T08:07:31.560 INFO:teuthology.orchestra.run.smithi007.stdout: "hit_set_count": 0, 2024-09-18T08:07:31.560 INFO:teuthology.orchestra.run.smithi007.stdout: "use_gmt_hitset": true, 2024-09-18T08:07:31.560 INFO:teuthology.orchestra.run.smithi007.stdout: "min_read_recency_for_promote": 0, 2024-09-18T08:07:31.560 INFO:teuthology.orchestra.run.smithi007.stdout: "min_write_recency_for_promote": 0, 2024-09-18T08:07:31.560 INFO:teuthology.orchestra.run.smithi007.stdout: "hit_set_grade_decay_rate": 0, 2024-09-18T08:07:31.560 INFO:teuthology.orchestra.run.smithi007.stdout: "hit_set_search_last_n": 0, 2024-09-18T08:07:31.560 INFO:teuthology.orchestra.run.smithi007.stdout: "grade_table": [], 2024-09-18T08:07:31.560 INFO:teuthology.orchestra.run.smithi007.stdout: "stripe_width": 0, 2024-09-18T08:07:31.560 INFO:teuthology.orchestra.run.smithi007.stdout: "expected_num_objects": 0, 2024-09-18T08:07:31.560 INFO:teuthology.orchestra.run.smithi007.stdout: "fast_read": false, 2024-09-18T08:07:31.561 INFO:teuthology.orchestra.run.smithi007.stdout: "options": {}, 2024-09-18T08:07:31.561 INFO:teuthology.orchestra.run.smithi007.stdout: "application_metadata": { 2024-09-18T08:07:31.561 INFO:teuthology.orchestra.run.smithi007.stdout: "rados": {} 2024-09-18T08:07:31.561 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:07:31.561 INFO:teuthology.orchestra.run.smithi007.stdout: "read_balance": { 2024-09-18T08:07:31.561 INFO:teuthology.orchestra.run.smithi007.stdout: "score_type": "Fair distribution", 2024-09-18T08:07:31.561 INFO:teuthology.orchestra.run.smithi007.stdout: "score_acting": 2.190000057220459, 2024-09-18T08:07:31.561 INFO:teuthology.orchestra.run.smithi007.stdout: "score_stable": 2.190000057220459, 2024-09-18T08:07:31.561 INFO:teuthology.orchestra.run.smithi007.stdout: "optimal_score": 1.1399999856948853, 2024-09-18T08:07:31.561 INFO:teuthology.orchestra.run.smithi007.stdout: "raw_score_acting": 2.5, 2024-09-18T08:07:31.561 INFO:teuthology.orchestra.run.smithi007.stdout: "raw_score_stable": 2.5, 2024-09-18T08:07:31.561 INFO:teuthology.orchestra.run.smithi007.stdout: "primary_affinity_weighted": 0.84128189086914062, 2024-09-18T08:07:31.561 INFO:teuthology.orchestra.run.smithi007.stdout: "average_primary_affinity": 0.95999997854232788, 2024-09-18T08:07:31.562 INFO:teuthology.orchestra.run.smithi007.stdout: "average_primary_affinity_weighted": 0.8399999737739563 2024-09-18T08:07:31.562 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-09-18T08:07:31.562 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-09-18T08:07:31.562 INFO:teuthology.orchestra.run.smithi007.stdout: ], 2024-09-18T08:07:31.562 INFO:teuthology.orchestra.run.smithi007.stdout: "osds": [ 2024-09-18T08:07:31.562 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:07:31.562 INFO:teuthology.orchestra.run.smithi007.stdout: "osd": 0, 2024-09-18T08:07:31.562 INFO:teuthology.orchestra.run.smithi007.stdout: "uuid": "ae8f60df-72ac-46fb-91a1-f132b851c876", 2024-09-18T08:07:31.562 INFO:teuthology.orchestra.run.smithi007.stdout: "up": 1, 2024-09-18T08:07:31.562 INFO:teuthology.orchestra.run.smithi007.stdout: "in": 1, 2024-09-18T08:07:31.563 INFO:teuthology.orchestra.run.smithi007.stdout: "weight": 1, 2024-09-18T08:07:31.563 INFO:teuthology.orchestra.run.smithi007.stdout: "primary_affinity": 1, 2024-09-18T08:07:31.563 INFO:teuthology.orchestra.run.smithi007.stdout: "last_clean_begin": 0, 2024-09-18T08:07:31.563 INFO:teuthology.orchestra.run.smithi007.stdout: "last_clean_end": 0, 2024-09-18T08:07:31.563 INFO:teuthology.orchestra.run.smithi007.stdout: "up_from": 8, 2024-09-18T08:07:31.563 INFO:teuthology.orchestra.run.smithi007.stdout: "up_thru": 154, 2024-09-18T08:07:31.563 INFO:teuthology.orchestra.run.smithi007.stdout: "down_at": 0, 2024-09-18T08:07:31.563 INFO:teuthology.orchestra.run.smithi007.stdout: "lost_at": 0, 2024-09-18T08:07:31.563 INFO:teuthology.orchestra.run.smithi007.stdout: "public_addrs": { 2024-09-18T08:07:31.563 INFO:teuthology.orchestra.run.smithi007.stdout: "addrvec": [ 2024-09-18T08:07:31.563 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:07:31.563 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "v2", 2024-09-18T08:07:31.563 INFO:teuthology.orchestra.run.smithi007.stdout: "addr": "172.21.15.7:6801", 2024-09-18T08:07:31.564 INFO:teuthology.orchestra.run.smithi007.stdout: "nonce": 2896425794 2024-09-18T08:07:31.564 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-09-18T08:07:31.564 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-09-18T08:07:31.564 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:07:31.564 INFO:teuthology.orchestra.run.smithi007.stdout: "cluster_addrs": { 2024-09-18T08:07:31.564 INFO:teuthology.orchestra.run.smithi007.stdout: "addrvec": [ 2024-09-18T08:07:31.564 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:07:31.564 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "v2", 2024-09-18T08:07:31.564 INFO:teuthology.orchestra.run.smithi007.stdout: "addr": "172.21.15.7:6802", 2024-09-18T08:07:31.564 INFO:teuthology.orchestra.run.smithi007.stdout: "nonce": 2896425794 2024-09-18T08:07:31.564 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-09-18T08:07:31.564 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-09-18T08:07:31.564 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:07:31.565 INFO:teuthology.orchestra.run.smithi007.stdout: "heartbeat_back_addrs": { 2024-09-18T08:07:31.565 INFO:teuthology.orchestra.run.smithi007.stdout: "addrvec": [ 2024-09-18T08:07:31.565 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:07:31.565 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "v2", 2024-09-18T08:07:31.565 INFO:teuthology.orchestra.run.smithi007.stdout: "addr": "172.21.15.7:6804", 2024-09-18T08:07:31.565 INFO:teuthology.orchestra.run.smithi007.stdout: "nonce": 2896425794 2024-09-18T08:07:31.565 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-09-18T08:07:31.565 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-09-18T08:07:31.565 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:07:31.565 INFO:teuthology.orchestra.run.smithi007.stdout: "heartbeat_front_addrs": { 2024-09-18T08:07:31.565 INFO:teuthology.orchestra.run.smithi007.stdout: "addrvec": [ 2024-09-18T08:07:31.565 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:07:31.565 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "v2", 2024-09-18T08:07:31.566 INFO:teuthology.orchestra.run.smithi007.stdout: "addr": "172.21.15.7:6803", 2024-09-18T08:07:31.566 INFO:teuthology.orchestra.run.smithi007.stdout: "nonce": 2896425794 2024-09-18T08:07:31.566 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-09-18T08:07:31.566 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-09-18T08:07:31.566 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:07:31.566 INFO:teuthology.orchestra.run.smithi007.stdout: "public_addr": "172.21.15.7:6801/2896425794", 2024-09-18T08:07:31.567 INFO:teuthology.orchestra.run.smithi007.stdout: "cluster_addr": "172.21.15.7:6802/2896425794", 2024-09-18T08:07:31.567 INFO:teuthology.orchestra.run.smithi007.stdout: "heartbeat_back_addr": "172.21.15.7:6804/2896425794", 2024-09-18T08:07:31.567 INFO:teuthology.orchestra.run.smithi007.stdout: "heartbeat_front_addr": "172.21.15.7:6803/2896425794", 2024-09-18T08:07:31.567 INFO:teuthology.orchestra.run.smithi007.stdout: "state": [ 2024-09-18T08:07:31.567 INFO:teuthology.orchestra.run.smithi007.stdout: "exists", 2024-09-18T08:07:31.567 INFO:teuthology.orchestra.run.smithi007.stdout: "up" 2024-09-18T08:07:31.567 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-09-18T08:07:31.567 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:07:31.567 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:07:31.567 INFO:teuthology.orchestra.run.smithi007.stdout: "osd": 1, 2024-09-18T08:07:31.567 INFO:teuthology.orchestra.run.smithi007.stdout: "uuid": "be28b70c-1f73-494f-ad4b-cb7ec5311999", 2024-09-18T08:07:31.567 INFO:teuthology.orchestra.run.smithi007.stdout: "up": 1, 2024-09-18T08:07:31.568 INFO:teuthology.orchestra.run.smithi007.stdout: "in": 1, 2024-09-18T08:07:31.568 INFO:teuthology.orchestra.run.smithi007.stdout: "weight": 1, 2024-09-18T08:07:31.568 INFO:teuthology.orchestra.run.smithi007.stdout: "primary_affinity": 1, 2024-09-18T08:07:31.568 INFO:teuthology.orchestra.run.smithi007.stdout: "last_clean_begin": 0, 2024-09-18T08:07:31.568 INFO:teuthology.orchestra.run.smithi007.stdout: "last_clean_end": 0, 2024-09-18T08:07:31.568 INFO:teuthology.orchestra.run.smithi007.stdout: "up_from": 13, 2024-09-18T08:07:31.568 INFO:teuthology.orchestra.run.smithi007.stdout: "up_thru": 154, 2024-09-18T08:07:31.568 INFO:teuthology.orchestra.run.smithi007.stdout: "down_at": 0, 2024-09-18T08:07:31.568 INFO:teuthology.orchestra.run.smithi007.stdout: "lost_at": 0, 2024-09-18T08:07:31.568 INFO:teuthology.orchestra.run.smithi007.stdout: "public_addrs": { 2024-09-18T08:07:31.568 INFO:teuthology.orchestra.run.smithi007.stdout: "addrvec": [ 2024-09-18T08:07:31.568 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:07:31.568 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "v2", 2024-09-18T08:07:31.569 INFO:teuthology.orchestra.run.smithi007.stdout: "addr": "172.21.15.7:6805", 2024-09-18T08:07:31.569 INFO:teuthology.orchestra.run.smithi007.stdout: "nonce": 1971397549 2024-09-18T08:07:31.569 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-09-18T08:07:31.569 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-09-18T08:07:31.569 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:07:31.569 INFO:teuthology.orchestra.run.smithi007.stdout: "cluster_addrs": { 2024-09-18T08:07:31.569 INFO:teuthology.orchestra.run.smithi007.stdout: "addrvec": [ 2024-09-18T08:07:31.569 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:07:31.569 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "v2", 2024-09-18T08:07:31.569 INFO:teuthology.orchestra.run.smithi007.stdout: "addr": "172.21.15.7:6806", 2024-09-18T08:07:31.569 INFO:teuthology.orchestra.run.smithi007.stdout: "nonce": 1971397549 2024-09-18T08:07:31.569 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-09-18T08:07:31.570 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-09-18T08:07:31.570 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:07:31.570 INFO:teuthology.orchestra.run.smithi007.stdout: "heartbeat_back_addrs": { 2024-09-18T08:07:31.570 INFO:teuthology.orchestra.run.smithi007.stdout: "addrvec": [ 2024-09-18T08:07:31.570 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:07:31.570 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "v2", 2024-09-18T08:07:31.570 INFO:teuthology.orchestra.run.smithi007.stdout: "addr": "172.21.15.7:6808", 2024-09-18T08:07:31.570 INFO:teuthology.orchestra.run.smithi007.stdout: "nonce": 1971397549 2024-09-18T08:07:31.570 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-09-18T08:07:31.570 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-09-18T08:07:31.570 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:07:31.570 INFO:teuthology.orchestra.run.smithi007.stdout: "heartbeat_front_addrs": { 2024-09-18T08:07:31.571 INFO:teuthology.orchestra.run.smithi007.stdout: "addrvec": [ 2024-09-18T08:07:31.571 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:07:31.571 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "v2", 2024-09-18T08:07:31.571 INFO:teuthology.orchestra.run.smithi007.stdout: "addr": "172.21.15.7:6807", 2024-09-18T08:07:31.571 INFO:teuthology.orchestra.run.smithi007.stdout: "nonce": 1971397549 2024-09-18T08:07:31.571 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-09-18T08:07:31.571 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-09-18T08:07:31.571 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:07:31.571 INFO:teuthology.orchestra.run.smithi007.stdout: "public_addr": "172.21.15.7:6805/1971397549", 2024-09-18T08:07:31.571 INFO:teuthology.orchestra.run.smithi007.stdout: "cluster_addr": "172.21.15.7:6806/1971397549", 2024-09-18T08:07:31.571 INFO:teuthology.orchestra.run.smithi007.stdout: "heartbeat_back_addr": "172.21.15.7:6808/1971397549", 2024-09-18T08:07:31.571 INFO:teuthology.orchestra.run.smithi007.stdout: "heartbeat_front_addr": "172.21.15.7:6807/1971397549", 2024-09-18T08:07:31.571 INFO:teuthology.orchestra.run.smithi007.stdout: "state": [ 2024-09-18T08:07:31.572 INFO:teuthology.orchestra.run.smithi007.stdout: "exists", 2024-09-18T08:07:31.572 INFO:teuthology.orchestra.run.smithi007.stdout: "up" 2024-09-18T08:07:31.572 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-09-18T08:07:31.572 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:07:31.572 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:07:31.572 INFO:teuthology.orchestra.run.smithi007.stdout: "osd": 2, 2024-09-18T08:07:31.572 INFO:teuthology.orchestra.run.smithi007.stdout: "uuid": "cfa4f083-c7fd-4743-be65-84a3040fe6dc", 2024-09-18T08:07:31.572 INFO:teuthology.orchestra.run.smithi007.stdout: "up": 1, 2024-09-18T08:07:31.572 INFO:teuthology.orchestra.run.smithi007.stdout: "in": 1, 2024-09-18T08:07:31.572 INFO:teuthology.orchestra.run.smithi007.stdout: "weight": 1, 2024-09-18T08:07:31.572 INFO:teuthology.orchestra.run.smithi007.stdout: "primary_affinity": 1, 2024-09-18T08:07:31.572 INFO:teuthology.orchestra.run.smithi007.stdout: "last_clean_begin": 0, 2024-09-18T08:07:31.572 INFO:teuthology.orchestra.run.smithi007.stdout: "last_clean_end": 0, 2024-09-18T08:07:31.572 INFO:teuthology.orchestra.run.smithi007.stdout: "up_from": 18, 2024-09-18T08:07:31.573 INFO:teuthology.orchestra.run.smithi007.stdout: "up_thru": 143, 2024-09-18T08:07:31.573 INFO:teuthology.orchestra.run.smithi007.stdout: "down_at": 0, 2024-09-18T08:07:31.573 INFO:teuthology.orchestra.run.smithi007.stdout: "lost_at": 0, 2024-09-18T08:07:31.573 INFO:teuthology.orchestra.run.smithi007.stdout: "public_addrs": { 2024-09-18T08:07:31.573 INFO:teuthology.orchestra.run.smithi007.stdout: "addrvec": [ 2024-09-18T08:07:31.573 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:07:31.573 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "v2", 2024-09-18T08:07:31.573 INFO:teuthology.orchestra.run.smithi007.stdout: "addr": "172.21.15.7:6809", 2024-09-18T08:07:31.573 INFO:teuthology.orchestra.run.smithi007.stdout: "nonce": 556142559 2024-09-18T08:07:31.573 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-09-18T08:07:31.573 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-09-18T08:07:31.573 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:07:31.573 INFO:teuthology.orchestra.run.smithi007.stdout: "cluster_addrs": { 2024-09-18T08:07:31.574 INFO:teuthology.orchestra.run.smithi007.stdout: "addrvec": [ 2024-09-18T08:07:31.574 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:07:31.574 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "v2", 2024-09-18T08:07:31.574 INFO:teuthology.orchestra.run.smithi007.stdout: "addr": "172.21.15.7:6810", 2024-09-18T08:07:31.574 INFO:teuthology.orchestra.run.smithi007.stdout: "nonce": 556142559 2024-09-18T08:07:31.574 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-09-18T08:07:31.574 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-09-18T08:07:31.574 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:07:31.574 INFO:teuthology.orchestra.run.smithi007.stdout: "heartbeat_back_addrs": { 2024-09-18T08:07:31.574 INFO:teuthology.orchestra.run.smithi007.stdout: "addrvec": [ 2024-09-18T08:07:31.574 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:07:31.574 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "v2", 2024-09-18T08:07:31.575 INFO:teuthology.orchestra.run.smithi007.stdout: "addr": "172.21.15.7:6812", 2024-09-18T08:07:31.575 INFO:teuthology.orchestra.run.smithi007.stdout: "nonce": 556142559 2024-09-18T08:07:31.575 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-09-18T08:07:31.575 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-09-18T08:07:31.575 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:07:31.575 INFO:teuthology.orchestra.run.smithi007.stdout: "heartbeat_front_addrs": { 2024-09-18T08:07:31.575 INFO:teuthology.orchestra.run.smithi007.stdout: "addrvec": [ 2024-09-18T08:07:31.575 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:07:31.575 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "v2", 2024-09-18T08:07:31.575 INFO:teuthology.orchestra.run.smithi007.stdout: "addr": "172.21.15.7:6811", 2024-09-18T08:07:31.575 INFO:teuthology.orchestra.run.smithi007.stdout: "nonce": 556142559 2024-09-18T08:07:31.575 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-09-18T08:07:31.575 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-09-18T08:07:31.576 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:07:31.576 INFO:teuthology.orchestra.run.smithi007.stdout: "public_addr": "172.21.15.7:6809/556142559", 2024-09-18T08:07:31.576 INFO:teuthology.orchestra.run.smithi007.stdout: "cluster_addr": "172.21.15.7:6810/556142559", 2024-09-18T08:07:31.576 INFO:teuthology.orchestra.run.smithi007.stdout: "heartbeat_back_addr": "172.21.15.7:6812/556142559", 2024-09-18T08:07:31.576 INFO:teuthology.orchestra.run.smithi007.stdout: "heartbeat_front_addr": "172.21.15.7:6811/556142559", 2024-09-18T08:07:31.576 INFO:teuthology.orchestra.run.smithi007.stdout: "state": [ 2024-09-18T08:07:31.576 INFO:teuthology.orchestra.run.smithi007.stdout: "exists", 2024-09-18T08:07:31.576 INFO:teuthology.orchestra.run.smithi007.stdout: "up" 2024-09-18T08:07:31.576 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-09-18T08:07:31.576 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:07:31.576 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:07:31.576 INFO:teuthology.orchestra.run.smithi007.stdout: "osd": 3, 2024-09-18T08:07:31.576 INFO:teuthology.orchestra.run.smithi007.stdout: "uuid": "11ecd7e1-a751-4b75-95f5-00b32d2a5bc4", 2024-09-18T08:07:31.576 INFO:teuthology.orchestra.run.smithi007.stdout: "up": 1, 2024-09-18T08:07:31.576 INFO:teuthology.orchestra.run.smithi007.stdout: "in": 1, 2024-09-18T08:07:31.577 INFO:teuthology.orchestra.run.smithi007.stdout: "weight": 1, 2024-09-18T08:07:31.577 INFO:teuthology.orchestra.run.smithi007.stdout: "primary_affinity": 1, 2024-09-18T08:07:31.577 INFO:teuthology.orchestra.run.smithi007.stdout: "last_clean_begin": 0, 2024-09-18T08:07:31.577 INFO:teuthology.orchestra.run.smithi007.stdout: "last_clean_end": 0, 2024-09-18T08:07:31.577 INFO:teuthology.orchestra.run.smithi007.stdout: "up_from": 26, 2024-09-18T08:07:31.577 INFO:teuthology.orchestra.run.smithi007.stdout: "up_thru": 154, 2024-09-18T08:07:31.577 INFO:teuthology.orchestra.run.smithi007.stdout: "down_at": 0, 2024-09-18T08:07:31.577 INFO:teuthology.orchestra.run.smithi007.stdout: "lost_at": 0, 2024-09-18T08:07:31.577 INFO:teuthology.orchestra.run.smithi007.stdout: "public_addrs": { 2024-09-18T08:07:31.577 INFO:teuthology.orchestra.run.smithi007.stdout: "addrvec": [ 2024-09-18T08:07:31.577 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:07:31.577 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "v2", 2024-09-18T08:07:31.577 INFO:teuthology.orchestra.run.smithi007.stdout: "addr": "172.21.15.7:6813", 2024-09-18T08:07:31.578 INFO:teuthology.orchestra.run.smithi007.stdout: "nonce": 163587772 2024-09-18T08:07:31.578 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-09-18T08:07:31.578 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-09-18T08:07:31.578 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:07:31.578 INFO:teuthology.orchestra.run.smithi007.stdout: "cluster_addrs": { 2024-09-18T08:07:31.578 INFO:teuthology.orchestra.run.smithi007.stdout: "addrvec": [ 2024-09-18T08:07:31.578 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:07:31.578 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "v2", 2024-09-18T08:07:31.578 INFO:teuthology.orchestra.run.smithi007.stdout: "addr": "172.21.15.7:6814", 2024-09-18T08:07:31.578 INFO:teuthology.orchestra.run.smithi007.stdout: "nonce": 163587772 2024-09-18T08:07:31.578 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-09-18T08:07:31.578 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-09-18T08:07:31.578 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:07:31.578 INFO:teuthology.orchestra.run.smithi007.stdout: "heartbeat_back_addrs": { 2024-09-18T08:07:31.579 INFO:teuthology.orchestra.run.smithi007.stdout: "addrvec": [ 2024-09-18T08:07:31.579 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:07:31.579 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "v2", 2024-09-18T08:07:31.579 INFO:teuthology.orchestra.run.smithi007.stdout: "addr": "172.21.15.7:6816", 2024-09-18T08:07:31.579 INFO:teuthology.orchestra.run.smithi007.stdout: "nonce": 163587772 2024-09-18T08:07:31.579 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-09-18T08:07:31.579 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-09-18T08:07:31.579 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:07:31.579 INFO:teuthology.orchestra.run.smithi007.stdout: "heartbeat_front_addrs": { 2024-09-18T08:07:31.579 INFO:teuthology.orchestra.run.smithi007.stdout: "addrvec": [ 2024-09-18T08:07:31.579 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:07:31.579 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "v2", 2024-09-18T08:07:31.579 INFO:teuthology.orchestra.run.smithi007.stdout: "addr": "172.21.15.7:6815", 2024-09-18T08:07:31.579 INFO:teuthology.orchestra.run.smithi007.stdout: "nonce": 163587772 2024-09-18T08:07:31.580 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-09-18T08:07:31.580 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-09-18T08:07:31.580 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:07:31.580 INFO:teuthology.orchestra.run.smithi007.stdout: "public_addr": "172.21.15.7:6813/163587772", 2024-09-18T08:07:31.580 INFO:teuthology.orchestra.run.smithi007.stdout: "cluster_addr": "172.21.15.7:6814/163587772", 2024-09-18T08:07:31.580 INFO:teuthology.orchestra.run.smithi007.stdout: "heartbeat_back_addr": "172.21.15.7:6816/163587772", 2024-09-18T08:07:31.580 INFO:teuthology.orchestra.run.smithi007.stdout: "heartbeat_front_addr": "172.21.15.7:6815/163587772", 2024-09-18T08:07:31.580 INFO:teuthology.orchestra.run.smithi007.stdout: "state": [ 2024-09-18T08:07:31.580 INFO:teuthology.orchestra.run.smithi007.stdout: "exists", 2024-09-18T08:07:31.580 INFO:teuthology.orchestra.run.smithi007.stdout: "up" 2024-09-18T08:07:31.580 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-09-18T08:07:31.580 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:07:31.581 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:07:31.581 INFO:teuthology.orchestra.run.smithi007.stdout: "osd": 4, 2024-09-18T08:07:31.581 INFO:teuthology.orchestra.run.smithi007.stdout: "uuid": "98992b34-cfa1-45bf-97a1-7d557739252f", 2024-09-18T08:07:31.581 INFO:teuthology.orchestra.run.smithi007.stdout: "up": 1, 2024-09-18T08:07:31.581 INFO:teuthology.orchestra.run.smithi007.stdout: "in": 1, 2024-09-18T08:07:31.581 INFO:teuthology.orchestra.run.smithi007.stdout: "weight": 1, 2024-09-18T08:07:31.581 INFO:teuthology.orchestra.run.smithi007.stdout: "primary_affinity": 1, 2024-09-18T08:07:31.581 INFO:teuthology.orchestra.run.smithi007.stdout: "last_clean_begin": 0, 2024-09-18T08:07:31.581 INFO:teuthology.orchestra.run.smithi007.stdout: "last_clean_end": 0, 2024-09-18T08:07:31.581 INFO:teuthology.orchestra.run.smithi007.stdout: "up_from": 32, 2024-09-18T08:07:31.581 INFO:teuthology.orchestra.run.smithi007.stdout: "up_thru": 154, 2024-09-18T08:07:31.581 INFO:teuthology.orchestra.run.smithi007.stdout: "down_at": 0, 2024-09-18T08:07:31.582 INFO:teuthology.orchestra.run.smithi007.stdout: "lost_at": 0, 2024-09-18T08:07:31.582 INFO:teuthology.orchestra.run.smithi007.stdout: "public_addrs": { 2024-09-18T08:07:31.582 INFO:teuthology.orchestra.run.smithi007.stdout: "addrvec": [ 2024-09-18T08:07:31.582 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:07:31.582 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "v2", 2024-09-18T08:07:31.582 INFO:teuthology.orchestra.run.smithi007.stdout: "addr": "172.21.15.50:6800", 2024-09-18T08:07:31.582 INFO:teuthology.orchestra.run.smithi007.stdout: "nonce": 3662119805 2024-09-18T08:07:31.582 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-09-18T08:07:31.582 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-09-18T08:07:31.582 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:07:31.582 INFO:teuthology.orchestra.run.smithi007.stdout: "cluster_addrs": { 2024-09-18T08:07:31.582 INFO:teuthology.orchestra.run.smithi007.stdout: "addrvec": [ 2024-09-18T08:07:31.582 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:07:31.583 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "v2", 2024-09-18T08:07:31.583 INFO:teuthology.orchestra.run.smithi007.stdout: "addr": "172.21.15.50:6801", 2024-09-18T08:07:31.583 INFO:teuthology.orchestra.run.smithi007.stdout: "nonce": 3662119805 2024-09-18T08:07:31.583 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-09-18T08:07:31.583 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-09-18T08:07:31.583 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:07:31.583 INFO:teuthology.orchestra.run.smithi007.stdout: "heartbeat_back_addrs": { 2024-09-18T08:07:31.583 INFO:teuthology.orchestra.run.smithi007.stdout: "addrvec": [ 2024-09-18T08:07:31.583 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:07:31.583 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "v2", 2024-09-18T08:07:31.583 INFO:teuthology.orchestra.run.smithi007.stdout: "addr": "172.21.15.50:6803", 2024-09-18T08:07:31.583 INFO:teuthology.orchestra.run.smithi007.stdout: "nonce": 3662119805 2024-09-18T08:07:31.583 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-09-18T08:07:31.583 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-09-18T08:07:31.584 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:07:31.584 INFO:teuthology.orchestra.run.smithi007.stdout: "heartbeat_front_addrs": { 2024-09-18T08:07:31.584 INFO:teuthology.orchestra.run.smithi007.stdout: "addrvec": [ 2024-09-18T08:07:31.584 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:07:31.584 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "v2", 2024-09-18T08:07:31.584 INFO:teuthology.orchestra.run.smithi007.stdout: "addr": "172.21.15.50:6802", 2024-09-18T08:07:31.584 INFO:teuthology.orchestra.run.smithi007.stdout: "nonce": 3662119805 2024-09-18T08:07:31.584 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-09-18T08:07:31.584 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-09-18T08:07:31.584 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:07:31.584 INFO:teuthology.orchestra.run.smithi007.stdout: "public_addr": "172.21.15.50:6800/3662119805", 2024-09-18T08:07:31.584 INFO:teuthology.orchestra.run.smithi007.stdout: "cluster_addr": "172.21.15.50:6801/3662119805", 2024-09-18T08:07:31.584 INFO:teuthology.orchestra.run.smithi007.stdout: "heartbeat_back_addr": "172.21.15.50:6803/3662119805", 2024-09-18T08:07:31.584 INFO:teuthology.orchestra.run.smithi007.stdout: "heartbeat_front_addr": "172.21.15.50:6802/3662119805", 2024-09-18T08:07:31.585 INFO:teuthology.orchestra.run.smithi007.stdout: "state": [ 2024-09-18T08:07:31.585 INFO:teuthology.orchestra.run.smithi007.stdout: "exists", 2024-09-18T08:07:31.585 INFO:teuthology.orchestra.run.smithi007.stdout: "up" 2024-09-18T08:07:31.585 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-09-18T08:07:31.585 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:07:31.585 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:07:31.585 INFO:teuthology.orchestra.run.smithi007.stdout: "osd": 5, 2024-09-18T08:07:31.585 INFO:teuthology.orchestra.run.smithi007.stdout: "uuid": "7f290005-ebca-467b-87e4-02ead1063610", 2024-09-18T08:07:31.585 INFO:teuthology.orchestra.run.smithi007.stdout: "up": 1, 2024-09-18T08:07:31.585 INFO:teuthology.orchestra.run.smithi007.stdout: "in": 1, 2024-09-18T08:07:31.585 INFO:teuthology.orchestra.run.smithi007.stdout: "weight": 1, 2024-09-18T08:07:31.585 INFO:teuthology.orchestra.run.smithi007.stdout: "primary_affinity": 1, 2024-09-18T08:07:31.585 INFO:teuthology.orchestra.run.smithi007.stdout: "last_clean_begin": 0, 2024-09-18T08:07:31.586 INFO:teuthology.orchestra.run.smithi007.stdout: "last_clean_end": 0, 2024-09-18T08:07:31.586 INFO:teuthology.orchestra.run.smithi007.stdout: "up_from": 39, 2024-09-18T08:07:31.586 INFO:teuthology.orchestra.run.smithi007.stdout: "up_thru": 154, 2024-09-18T08:07:31.586 INFO:teuthology.orchestra.run.smithi007.stdout: "down_at": 0, 2024-09-18T08:07:31.586 INFO:teuthology.orchestra.run.smithi007.stdout: "lost_at": 0, 2024-09-18T08:07:31.586 INFO:teuthology.orchestra.run.smithi007.stdout: "public_addrs": { 2024-09-18T08:07:31.586 INFO:teuthology.orchestra.run.smithi007.stdout: "addrvec": [ 2024-09-18T08:07:31.586 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:07:31.586 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "v2", 2024-09-18T08:07:31.586 INFO:teuthology.orchestra.run.smithi007.stdout: "addr": "172.21.15.50:6804", 2024-09-18T08:07:31.586 INFO:teuthology.orchestra.run.smithi007.stdout: "nonce": 1199797954 2024-09-18T08:07:31.586 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-09-18T08:07:31.586 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-09-18T08:07:31.587 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:07:31.587 INFO:teuthology.orchestra.run.smithi007.stdout: "cluster_addrs": { 2024-09-18T08:07:31.587 INFO:teuthology.orchestra.run.smithi007.stdout: "addrvec": [ 2024-09-18T08:07:31.587 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:07:31.587 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "v2", 2024-09-18T08:07:31.587 INFO:teuthology.orchestra.run.smithi007.stdout: "addr": "172.21.15.50:6805", 2024-09-18T08:07:31.587 INFO:teuthology.orchestra.run.smithi007.stdout: "nonce": 1199797954 2024-09-18T08:07:31.587 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-09-18T08:07:31.587 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-09-18T08:07:31.587 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:07:31.587 INFO:teuthology.orchestra.run.smithi007.stdout: "heartbeat_back_addrs": { 2024-09-18T08:07:31.587 INFO:teuthology.orchestra.run.smithi007.stdout: "addrvec": [ 2024-09-18T08:07:31.588 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:07:31.588 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "v2", 2024-09-18T08:07:31.588 INFO:teuthology.orchestra.run.smithi007.stdout: "addr": "172.21.15.50:6807", 2024-09-18T08:07:31.588 INFO:teuthology.orchestra.run.smithi007.stdout: "nonce": 1199797954 2024-09-18T08:07:31.588 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-09-18T08:07:31.588 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-09-18T08:07:31.588 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:07:31.588 INFO:teuthology.orchestra.run.smithi007.stdout: "heartbeat_front_addrs": { 2024-09-18T08:07:31.588 INFO:teuthology.orchestra.run.smithi007.stdout: "addrvec": [ 2024-09-18T08:07:31.588 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:07:31.588 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "v2", 2024-09-18T08:07:31.588 INFO:teuthology.orchestra.run.smithi007.stdout: "addr": "172.21.15.50:6806", 2024-09-18T08:07:31.588 INFO:teuthology.orchestra.run.smithi007.stdout: "nonce": 1199797954 2024-09-18T08:07:31.589 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-09-18T08:07:31.589 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-09-18T08:07:31.589 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:07:31.589 INFO:teuthology.orchestra.run.smithi007.stdout: "public_addr": "172.21.15.50:6804/1199797954", 2024-09-18T08:07:31.589 INFO:teuthology.orchestra.run.smithi007.stdout: "cluster_addr": "172.21.15.50:6805/1199797954", 2024-09-18T08:07:31.589 INFO:teuthology.orchestra.run.smithi007.stdout: "heartbeat_back_addr": "172.21.15.50:6807/1199797954", 2024-09-18T08:07:31.589 INFO:teuthology.orchestra.run.smithi007.stdout: "heartbeat_front_addr": "172.21.15.50:6806/1199797954", 2024-09-18T08:07:31.589 INFO:teuthology.orchestra.run.smithi007.stdout: "state": [ 2024-09-18T08:07:31.589 INFO:teuthology.orchestra.run.smithi007.stdout: "exists", 2024-09-18T08:07:31.589 INFO:teuthology.orchestra.run.smithi007.stdout: "up" 2024-09-18T08:07:31.589 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-09-18T08:07:31.589 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:07:31.590 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:07:31.590 INFO:teuthology.orchestra.run.smithi007.stdout: "osd": 6, 2024-09-18T08:07:31.590 INFO:teuthology.orchestra.run.smithi007.stdout: "uuid": "e07d846c-ae42-4fea-aa5c-3a25835d18a6", 2024-09-18T08:07:31.590 INFO:teuthology.orchestra.run.smithi007.stdout: "up": 1, 2024-09-18T08:07:31.590 INFO:teuthology.orchestra.run.smithi007.stdout: "in": 1, 2024-09-18T08:07:31.590 INFO:teuthology.orchestra.run.smithi007.stdout: "weight": 1, 2024-09-18T08:07:31.590 INFO:teuthology.orchestra.run.smithi007.stdout: "primary_affinity": 0, 2024-09-18T08:07:31.590 INFO:teuthology.orchestra.run.smithi007.stdout: "last_clean_begin": 0, 2024-09-18T08:07:31.590 INFO:teuthology.orchestra.run.smithi007.stdout: "last_clean_end": 0, 2024-09-18T08:07:31.590 INFO:teuthology.orchestra.run.smithi007.stdout: "up_from": 47, 2024-09-18T08:07:31.590 INFO:teuthology.orchestra.run.smithi007.stdout: "up_thru": 83, 2024-09-18T08:07:31.590 INFO:teuthology.orchestra.run.smithi007.stdout: "down_at": 0, 2024-09-18T08:07:31.590 INFO:teuthology.orchestra.run.smithi007.stdout: "lost_at": 0, 2024-09-18T08:07:31.590 INFO:teuthology.orchestra.run.smithi007.stdout: "public_addrs": { 2024-09-18T08:07:31.590 INFO:teuthology.orchestra.run.smithi007.stdout: "addrvec": [ 2024-09-18T08:07:31.591 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:07:31.591 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "v2", 2024-09-18T08:07:31.591 INFO:teuthology.orchestra.run.smithi007.stdout: "addr": "172.21.15.50:6808", 2024-09-18T08:07:31.591 INFO:teuthology.orchestra.run.smithi007.stdout: "nonce": 255960422 2024-09-18T08:07:31.591 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-09-18T08:07:31.591 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-09-18T08:07:31.591 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:07:31.591 INFO:teuthology.orchestra.run.smithi007.stdout: "cluster_addrs": { 2024-09-18T08:07:31.591 INFO:teuthology.orchestra.run.smithi007.stdout: "addrvec": [ 2024-09-18T08:07:31.591 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:07:31.591 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "v2", 2024-09-18T08:07:31.591 INFO:teuthology.orchestra.run.smithi007.stdout: "addr": "172.21.15.50:6809", 2024-09-18T08:07:31.591 INFO:teuthology.orchestra.run.smithi007.stdout: "nonce": 255960422 2024-09-18T08:07:31.591 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-09-18T08:07:31.592 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-09-18T08:07:31.592 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:07:31.592 INFO:teuthology.orchestra.run.smithi007.stdout: "heartbeat_back_addrs": { 2024-09-18T08:07:31.592 INFO:teuthology.orchestra.run.smithi007.stdout: "addrvec": [ 2024-09-18T08:07:31.592 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:07:31.592 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "v2", 2024-09-18T08:07:31.592 INFO:teuthology.orchestra.run.smithi007.stdout: "addr": "172.21.15.50:6811", 2024-09-18T08:07:31.592 INFO:teuthology.orchestra.run.smithi007.stdout: "nonce": 255960422 2024-09-18T08:07:31.592 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-09-18T08:07:31.592 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-09-18T08:07:31.592 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:07:31.592 INFO:teuthology.orchestra.run.smithi007.stdout: "heartbeat_front_addrs": { 2024-09-18T08:07:31.592 INFO:teuthology.orchestra.run.smithi007.stdout: "addrvec": [ 2024-09-18T08:07:31.592 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:07:31.592 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "v2", 2024-09-18T08:07:31.593 INFO:teuthology.orchestra.run.smithi007.stdout: "addr": "172.21.15.50:6810", 2024-09-18T08:07:31.593 INFO:teuthology.orchestra.run.smithi007.stdout: "nonce": 255960422 2024-09-18T08:07:31.593 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-09-18T08:07:31.593 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-09-18T08:07:31.593 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:07:31.593 INFO:teuthology.orchestra.run.smithi007.stdout: "public_addr": "172.21.15.50:6808/255960422", 2024-09-18T08:07:31.593 INFO:teuthology.orchestra.run.smithi007.stdout: "cluster_addr": "172.21.15.50:6809/255960422", 2024-09-18T08:07:31.593 INFO:teuthology.orchestra.run.smithi007.stdout: "heartbeat_back_addr": "172.21.15.50:6811/255960422", 2024-09-18T08:07:31.593 INFO:teuthology.orchestra.run.smithi007.stdout: "heartbeat_front_addr": "172.21.15.50:6810/255960422", 2024-09-18T08:07:31.593 INFO:teuthology.orchestra.run.smithi007.stdout: "state": [ 2024-09-18T08:07:31.593 INFO:teuthology.orchestra.run.smithi007.stdout: "exists", 2024-09-18T08:07:31.593 INFO:teuthology.orchestra.run.smithi007.stdout: "up" 2024-09-18T08:07:31.593 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-09-18T08:07:31.593 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:07:31.594 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:07:31.594 INFO:teuthology.orchestra.run.smithi007.stdout: "osd": 7, 2024-09-18T08:07:31.594 INFO:teuthology.orchestra.run.smithi007.stdout: "uuid": "c9dac3b7-d651-489d-b96e-5b10fa32d9af", 2024-09-18T08:07:31.594 INFO:teuthology.orchestra.run.smithi007.stdout: "up": 1, 2024-09-18T08:07:31.594 INFO:teuthology.orchestra.run.smithi007.stdout: "in": 1, 2024-09-18T08:07:31.594 INFO:teuthology.orchestra.run.smithi007.stdout: "weight": 1, 2024-09-18T08:07:31.594 INFO:teuthology.orchestra.run.smithi007.stdout: "primary_affinity": 0.730255126953125, 2024-09-18T08:07:31.594 INFO:teuthology.orchestra.run.smithi007.stdout: "last_clean_begin": 0, 2024-09-18T08:07:31.594 INFO:teuthology.orchestra.run.smithi007.stdout: "last_clean_end": 0, 2024-09-18T08:07:31.594 INFO:teuthology.orchestra.run.smithi007.stdout: "up_from": 56, 2024-09-18T08:07:31.594 INFO:teuthology.orchestra.run.smithi007.stdout: "up_thru": 154, 2024-09-18T08:07:31.594 INFO:teuthology.orchestra.run.smithi007.stdout: "down_at": 0, 2024-09-18T08:07:31.594 INFO:teuthology.orchestra.run.smithi007.stdout: "lost_at": 0, 2024-09-18T08:07:31.594 INFO:teuthology.orchestra.run.smithi007.stdout: "public_addrs": { 2024-09-18T08:07:31.594 INFO:teuthology.orchestra.run.smithi007.stdout: "addrvec": [ 2024-09-18T08:07:31.595 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:07:31.595 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "v2", 2024-09-18T08:07:31.595 INFO:teuthology.orchestra.run.smithi007.stdout: "addr": "172.21.15.50:6812", 2024-09-18T08:07:31.595 INFO:teuthology.orchestra.run.smithi007.stdout: "nonce": 874923904 2024-09-18T08:07:31.595 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-09-18T08:07:31.595 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-09-18T08:07:31.595 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:07:31.595 INFO:teuthology.orchestra.run.smithi007.stdout: "cluster_addrs": { 2024-09-18T08:07:31.595 INFO:teuthology.orchestra.run.smithi007.stdout: "addrvec": [ 2024-09-18T08:07:31.595 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:07:31.595 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "v2", 2024-09-18T08:07:31.595 INFO:teuthology.orchestra.run.smithi007.stdout: "addr": "172.21.15.50:6813", 2024-09-18T08:07:31.595 INFO:teuthology.orchestra.run.smithi007.stdout: "nonce": 874923904 2024-09-18T08:07:31.595 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-09-18T08:07:31.595 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-09-18T08:07:31.596 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:07:31.596 INFO:teuthology.orchestra.run.smithi007.stdout: "heartbeat_back_addrs": { 2024-09-18T08:07:31.596 INFO:teuthology.orchestra.run.smithi007.stdout: "addrvec": [ 2024-09-18T08:07:31.596 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:07:31.596 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "v2", 2024-09-18T08:07:31.596 INFO:teuthology.orchestra.run.smithi007.stdout: "addr": "172.21.15.50:6815", 2024-09-18T08:07:31.596 INFO:teuthology.orchestra.run.smithi007.stdout: "nonce": 874923904 2024-09-18T08:07:31.596 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-09-18T08:07:31.596 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-09-18T08:07:31.596 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:07:31.596 INFO:teuthology.orchestra.run.smithi007.stdout: "heartbeat_front_addrs": { 2024-09-18T08:07:31.596 INFO:teuthology.orchestra.run.smithi007.stdout: "addrvec": [ 2024-09-18T08:07:31.596 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:07:31.597 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "v2", 2024-09-18T08:07:31.597 INFO:teuthology.orchestra.run.smithi007.stdout: "addr": "172.21.15.50:6814", 2024-09-18T08:07:31.597 INFO:teuthology.orchestra.run.smithi007.stdout: "nonce": 874923904 2024-09-18T08:07:31.597 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-09-18T08:07:31.597 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-09-18T08:07:31.597 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:07:31.597 INFO:teuthology.orchestra.run.smithi007.stdout: "public_addr": "172.21.15.50:6812/874923904", 2024-09-18T08:07:31.597 INFO:teuthology.orchestra.run.smithi007.stdout: "cluster_addr": "172.21.15.50:6813/874923904", 2024-09-18T08:07:31.597 INFO:teuthology.orchestra.run.smithi007.stdout: "heartbeat_back_addr": "172.21.15.50:6815/874923904", 2024-09-18T08:07:31.597 INFO:teuthology.orchestra.run.smithi007.stdout: "heartbeat_front_addr": "172.21.15.50:6814/874923904", 2024-09-18T08:07:31.597 INFO:teuthology.orchestra.run.smithi007.stdout: "state": [ 2024-09-18T08:07:31.597 INFO:teuthology.orchestra.run.smithi007.stdout: "exists", 2024-09-18T08:07:31.597 INFO:teuthology.orchestra.run.smithi007.stdout: "up" 2024-09-18T08:07:31.598 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-09-18T08:07:31.598 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-09-18T08:07:31.598 INFO:teuthology.orchestra.run.smithi007.stdout: ], 2024-09-18T08:07:31.598 INFO:teuthology.orchestra.run.smithi007.stdout: "osd_xinfo": [ 2024-09-18T08:07:31.598 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:07:31.598 INFO:teuthology.orchestra.run.smithi007.stdout: "osd": 0, 2024-09-18T08:07:31.598 INFO:teuthology.orchestra.run.smithi007.stdout: "down_stamp": "0.000000", 2024-09-18T08:07:31.598 INFO:teuthology.orchestra.run.smithi007.stdout: "laggy_probability": 0, 2024-09-18T08:07:31.598 INFO:teuthology.orchestra.run.smithi007.stdout: "laggy_interval": 0, 2024-09-18T08:07:31.598 INFO:teuthology.orchestra.run.smithi007.stdout: "features": 4540719139924082687, 2024-09-18T08:07:31.598 INFO:teuthology.orchestra.run.smithi007.stdout: "old_weight": 0, 2024-09-18T08:07:31.598 INFO:teuthology.orchestra.run.smithi007.stdout: "last_purged_snaps_scrub": "2024-09-18T08:06:14.699199+0000", 2024-09-18T08:07:31.598 INFO:teuthology.orchestra.run.smithi007.stdout: "dead_epoch": 0 2024-09-18T08:07:31.598 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:07:31.599 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:07:31.599 INFO:teuthology.orchestra.run.smithi007.stdout: "osd": 1, 2024-09-18T08:07:31.599 INFO:teuthology.orchestra.run.smithi007.stdout: "down_stamp": "0.000000", 2024-09-18T08:07:31.599 INFO:teuthology.orchestra.run.smithi007.stdout: "laggy_probability": 0, 2024-09-18T08:07:31.599 INFO:teuthology.orchestra.run.smithi007.stdout: "laggy_interval": 0, 2024-09-18T08:07:31.599 INFO:teuthology.orchestra.run.smithi007.stdout: "features": 4540719139924082687, 2024-09-18T08:07:31.599 INFO:teuthology.orchestra.run.smithi007.stdout: "old_weight": 0, 2024-09-18T08:07:31.599 INFO:teuthology.orchestra.run.smithi007.stdout: "last_purged_snaps_scrub": "2024-09-18T08:06:40.626049+0000", 2024-09-18T08:07:31.599 INFO:teuthology.orchestra.run.smithi007.stdout: "dead_epoch": 0 2024-09-18T08:07:31.599 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:07:31.599 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:07:31.599 INFO:teuthology.orchestra.run.smithi007.stdout: "osd": 2, 2024-09-18T08:07:31.599 INFO:teuthology.orchestra.run.smithi007.stdout: "down_stamp": "0.000000", 2024-09-18T08:07:31.599 INFO:teuthology.orchestra.run.smithi007.stdout: "laggy_probability": 0, 2024-09-18T08:07:31.600 INFO:teuthology.orchestra.run.smithi007.stdout: "laggy_interval": 0, 2024-09-18T08:07:31.600 INFO:teuthology.orchestra.run.smithi007.stdout: "features": 4540719139924082687, 2024-09-18T08:07:31.600 INFO:teuthology.orchestra.run.smithi007.stdout: "old_weight": 0, 2024-09-18T08:07:31.600 INFO:teuthology.orchestra.run.smithi007.stdout: "last_purged_snaps_scrub": "2024-09-18T08:06:12.916079+0000", 2024-09-18T08:07:31.600 INFO:teuthology.orchestra.run.smithi007.stdout: "dead_epoch": 0 2024-09-18T08:07:31.600 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:07:31.600 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:07:31.600 INFO:teuthology.orchestra.run.smithi007.stdout: "osd": 3, 2024-09-18T08:07:31.600 INFO:teuthology.orchestra.run.smithi007.stdout: "down_stamp": "0.000000", 2024-09-18T08:07:31.600 INFO:teuthology.orchestra.run.smithi007.stdout: "laggy_probability": 0, 2024-09-18T08:07:31.600 INFO:teuthology.orchestra.run.smithi007.stdout: "laggy_interval": 0, 2024-09-18T08:07:31.600 INFO:teuthology.orchestra.run.smithi007.stdout: "features": 4540719139924082687, 2024-09-18T08:07:31.600 INFO:teuthology.orchestra.run.smithi007.stdout: "old_weight": 0, 2024-09-18T08:07:31.600 INFO:teuthology.orchestra.run.smithi007.stdout: "last_purged_snaps_scrub": "2024-09-18T08:07:25.802428+0000", 2024-09-18T08:07:31.601 INFO:teuthology.orchestra.run.smithi007.stdout: "dead_epoch": 0 2024-09-18T08:07:31.601 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:07:31.601 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:07:31.601 INFO:teuthology.orchestra.run.smithi007.stdout: "osd": 4, 2024-09-18T08:07:31.601 INFO:teuthology.orchestra.run.smithi007.stdout: "down_stamp": "0.000000", 2024-09-18T08:07:31.601 INFO:teuthology.orchestra.run.smithi007.stdout: "laggy_probability": 0, 2024-09-18T08:07:31.601 INFO:teuthology.orchestra.run.smithi007.stdout: "laggy_interval": 0, 2024-09-18T08:07:31.601 INFO:teuthology.orchestra.run.smithi007.stdout: "features": 4540719139924082687, 2024-09-18T08:07:31.601 INFO:teuthology.orchestra.run.smithi007.stdout: "old_weight": 0, 2024-09-18T08:07:31.601 INFO:teuthology.orchestra.run.smithi007.stdout: "last_purged_snaps_scrub": "2024-09-18T08:06:56.054937+0000", 2024-09-18T08:07:31.601 INFO:teuthology.orchestra.run.smithi007.stdout: "dead_epoch": 0 2024-09-18T08:07:31.601 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:07:31.601 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:07:31.602 INFO:teuthology.orchestra.run.smithi007.stdout: "osd": 5, 2024-09-18T08:07:31.602 INFO:teuthology.orchestra.run.smithi007.stdout: "down_stamp": "0.000000", 2024-09-18T08:07:31.602 INFO:teuthology.orchestra.run.smithi007.stdout: "laggy_probability": 0, 2024-09-18T08:07:31.602 INFO:teuthology.orchestra.run.smithi007.stdout: "laggy_interval": 0, 2024-09-18T08:07:31.602 INFO:teuthology.orchestra.run.smithi007.stdout: "features": 4540719139924082687, 2024-09-18T08:07:31.602 INFO:teuthology.orchestra.run.smithi007.stdout: "old_weight": 0, 2024-09-18T08:07:31.602 INFO:teuthology.orchestra.run.smithi007.stdout: "last_purged_snaps_scrub": "2024-09-18T08:07:28.333379+0000", 2024-09-18T08:07:31.602 INFO:teuthology.orchestra.run.smithi007.stdout: "dead_epoch": 0 2024-09-18T08:07:31.602 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:07:31.602 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:07:31.602 INFO:teuthology.orchestra.run.smithi007.stdout: "osd": 6, 2024-09-18T08:07:31.602 INFO:teuthology.orchestra.run.smithi007.stdout: "down_stamp": "0.000000", 2024-09-18T08:07:31.602 INFO:teuthology.orchestra.run.smithi007.stdout: "laggy_probability": 0, 2024-09-18T08:07:31.603 INFO:teuthology.orchestra.run.smithi007.stdout: "laggy_interval": 0, 2024-09-18T08:07:31.603 INFO:teuthology.orchestra.run.smithi007.stdout: "features": 4540719139924082687, 2024-09-18T08:07:31.603 INFO:teuthology.orchestra.run.smithi007.stdout: "old_weight": 0, 2024-09-18T08:07:31.603 INFO:teuthology.orchestra.run.smithi007.stdout: "last_purged_snaps_scrub": "2024-09-18T08:06:59.725318+0000", 2024-09-18T08:07:31.603 INFO:teuthology.orchestra.run.smithi007.stdout: "dead_epoch": 0 2024-09-18T08:07:31.603 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:07:31.603 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:07:31.603 INFO:teuthology.orchestra.run.smithi007.stdout: "osd": 7, 2024-09-18T08:07:31.603 INFO:teuthology.orchestra.run.smithi007.stdout: "down_stamp": "0.000000", 2024-09-18T08:07:31.603 INFO:teuthology.orchestra.run.smithi007.stdout: "laggy_probability": 0, 2024-09-18T08:07:31.603 INFO:teuthology.orchestra.run.smithi007.stdout: "laggy_interval": 0, 2024-09-18T08:07:31.603 INFO:teuthology.orchestra.run.smithi007.stdout: "features": 4540719139924082687, 2024-09-18T08:07:31.603 INFO:teuthology.orchestra.run.smithi007.stdout: "old_weight": 0, 2024-09-18T08:07:31.603 INFO:teuthology.orchestra.run.smithi007.stdout: "last_purged_snaps_scrub": "2024-09-18T08:06:42.410072+0000", 2024-09-18T08:07:31.604 INFO:teuthology.orchestra.run.smithi007.stdout: "dead_epoch": 0 2024-09-18T08:07:31.604 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-09-18T08:07:31.604 INFO:teuthology.orchestra.run.smithi007.stdout: ], 2024-09-18T08:07:31.604 INFO:teuthology.orchestra.run.smithi007.stdout: "pg_upmap": [], 2024-09-18T08:07:31.604 INFO:teuthology.orchestra.run.smithi007.stdout: "pg_upmap_items": [ 2024-09-18T08:07:31.604 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:07:31.604 INFO:teuthology.orchestra.run.smithi007.stdout: "pgid": "4.7", 2024-09-18T08:07:31.604 INFO:teuthology.orchestra.run.smithi007.stdout: "mappings": [ 2024-09-18T08:07:31.604 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:07:31.604 INFO:teuthology.orchestra.run.smithi007.stdout: "from": 1, 2024-09-18T08:07:31.604 INFO:teuthology.orchestra.run.smithi007.stdout: "to": 2 2024-09-18T08:07:31.604 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-09-18T08:07:31.604 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-09-18T08:07:31.604 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:07:31.605 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:07:31.605 INFO:teuthology.orchestra.run.smithi007.stdout: "pgid": "4.14", 2024-09-18T08:07:31.605 INFO:teuthology.orchestra.run.smithi007.stdout: "mappings": [ 2024-09-18T08:07:31.605 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:07:31.605 INFO:teuthology.orchestra.run.smithi007.stdout: "from": 1, 2024-09-18T08:07:31.605 INFO:teuthology.orchestra.run.smithi007.stdout: "to": 2 2024-09-18T08:07:31.605 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-09-18T08:07:31.605 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-09-18T08:07:31.605 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:07:31.605 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:07:31.605 INFO:teuthology.orchestra.run.smithi007.stdout: "pgid": "4.1c", 2024-09-18T08:07:31.605 INFO:teuthology.orchestra.run.smithi007.stdout: "mappings": [ 2024-09-18T08:07:31.606 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:07:31.606 INFO:teuthology.orchestra.run.smithi007.stdout: "from": 1, 2024-09-18T08:07:31.606 INFO:teuthology.orchestra.run.smithi007.stdout: "to": 5 2024-09-18T08:07:31.606 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-09-18T08:07:31.606 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-09-18T08:07:31.606 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-09-18T08:07:31.606 INFO:teuthology.orchestra.run.smithi007.stdout: ], 2024-09-18T08:07:31.606 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2 2024-09-18T08:07:31.607 INFO:teuthology.orchestra.run.smithi007.stdout: "pg_upmap_primaries": [], 2024-09-18T08:07:31.607 INFO:teuthology.orchestra.run.smithi007.stdout: "pg_temp": [], 2024-09-18T08:07:31.607 INFO:teuthology.orchestra.run.smithi007.stdout: "primary_temp": [], 2024-09-18T08:07:31.607 INFO:teuthology.orchestra.run.smithi007.stdout: "blocklist": { 2024-09-18T08:07:31.607 INFO:teuthology.orchestra.run.smithi007.stdout: "172.21.15.7:0/398367658": "2024-09-19T08:03:17.052903+0000", 2024-09-18T08:07:31.607 INFO:teuthology.orchestra.run.smithi007.stdout: "172.21.15.7:6800/2918496291": "2024-09-19T07:57:31.015269+0000", 2024-09-18T08:07:31.607 INFO:teuthology.orchestra.run.smithi007.stdout: "172.21.15.7:0/2514354773": "2024-09-19T07:57:31.015269+0000", 2024-09-18T08:07:31.607 INFO:teuthology.orchestra.run.smithi007.stdout: "172.21.15.7:0/3924911335": "2024-09-19T07:57:31.015269+0000", 2024-09-18T08:07:31.608 INFO:teuthology.orchestra.run.smithi007.stdout: "172.21.15.7:0/1668598084": "2024-09-19T07:57:57.346149+0000", 2024-09-18T08:07:31.608 INFO:teuthology.orchestra.run.smithi007.stdout: "172.21.15.7:0/4272027292": "2024-09-19T07:57:31.015269+0000", 2024-09-18T08:07:31.608 INFO:teuthology.orchestra.run.smithi007.stdout: "172.21.15.7:6800/1013061071": "2024-09-19T07:57:57.346149+0000", 2024-09-18T08:07:31.608 INFO:teuthology.orchestra.run.smithi007.stdout: "172.21.15.7:6800/1491061076": "2024-09-19T08:03:17.052903+0000", 2024-09-18T08:07:31.608 INFO:teuthology.orchestra.run.smithi007.stdout: "172.21.15.7:0/533727535": "2024-09-19T07:57:57.346149+0000", 2024-09-18T08:07:31.608 INFO:teuthology.orchestra.run.smithi007.stdout: "172.21.15.7:0/1605217610": "2024-09-19T08:03:17.052903+0000", 2024-09-18T08:07:31.608 INFO:teuthology.orchestra.run.smithi007.stdout: "172.21.15.7:0/1722968065": "2024-09-19T08:03:17.052903+0000", 2024-09-18T08:07:31.608 INFO:teuthology.orchestra.run.smithi007.stdout: "172.21.15.7:0/1277549846": "2024-09-19T07:57:57.346149+0000", 2024-09-18T08:07:31.608 INFO:teuthology.orchestra.run.smithi007.stdout: "172.21.15.7:0/232452890": "2024-09-19T08:03:17.052903+0000", 2024-09-18T08:07:31.608 INFO:teuthology.orchestra.run.smithi007.stdout: "172.21.15.7:0/3881670470": "2024-09-19T08:03:17.052903+0000" 2024-09-18T08:07:31.608 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:07:31.608 INFO:teuthology.orchestra.run.smithi007.stdout: "range_blocklist": {}, 2024-09-18T08:07:31.609 INFO:teuthology.orchestra.run.smithi007.stdout: "erasure_code_profiles": { 2024-09-18T08:07:31.609 INFO:teuthology.orchestra.run.smithi007.stdout: "default": { 2024-09-18T08:07:31.609 INFO:teuthology.orchestra.run.smithi007.stdout: "crush-failure-domain": "osd", 2024-09-18T08:07:31.609 INFO:teuthology.orchestra.run.smithi007.stdout: "k": "2", 2024-09-18T08:07:31.609 INFO:teuthology.orchestra.run.smithi007.stdout: "m": "1", 2024-09-18T08:07:31.609 INFO:teuthology.orchestra.run.smithi007.stdout: "plugin": "jerasure", 2024-09-18T08:07:31.609 INFO:teuthology.orchestra.run.smithi007.stdout: "technique": "reed_sol_van" 2024-09-18T08:07:31.609 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-09-18T08:07:31.609 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:07:31.609 INFO:teuthology.orchestra.run.smithi007.stdout: "removed_snaps_queue": [], 2024-09-18T08:07:31.609 INFO:teuthology.orchestra.run.smithi007.stdout: "new_removed_snaps": [], 2024-09-18T08:07:31.609 INFO:teuthology.orchestra.run.smithi007.stdout: "new_purged_snaps": [], 2024-09-18T08:07:31.610 INFO:teuthology.orchestra.run.smithi007.stdout: "crush_node_flags": {}, 2024-09-18T08:07:31.610 INFO:teuthology.orchestra.run.smithi007.stdout: "device_class_flags": {}, 2024-09-18T08:07:31.610 INFO:teuthology.orchestra.run.smithi007.stdout: "stretch_mode": { 2024-09-18T08:07:31.610 INFO:teuthology.orchestra.run.smithi007.stdout: "stretch_mode_enabled": false, 2024-09-18T08:07:31.610 INFO:teuthology.orchestra.run.smithi007.stdout: "stretch_bucket_count": 0, 2024-09-18T08:07:31.610 INFO:teuthology.orchestra.run.smithi007.stdout: "degraded_stretch_mode": 0, 2024-09-18T08:07:31.610 INFO:teuthology.orchestra.run.smithi007.stdout: "recovering_stretch_mode": 0, 2024-09-18T08:07:31.610 INFO:teuthology.orchestra.run.smithi007.stdout: "stretch_mode_bucket": 0 2024-09-18T08:07:31.610 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-09-18T08:07:31.610 INFO:teuthology.orchestra.run.smithi007.stdout:} 2024-09-18T08:07:31.663 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:07:31.663 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:07:31.664 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2' 2024-09-18T08:07:31.733 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:07:31 smithi007 ceph-mon[29459]: from='client.? 172.21.15.7:0/1427664164' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-18T08:07:31.733 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:07:31 smithi007 ceph-mon[39249]: from='client.? 172.21.15.7:0/1427664164' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-18T08:07:31.765 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5 2024-09-18T08:07:31.790 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:07:31.791 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:07:31.791 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5' 2024-09-18T08:07:31.892 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4 2024-09-18T08:07:31.905 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:07:31 smithi050 ceph-mon[32371]: from='client.? 172.21.15.7:0/1427664164' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-18T08:07:31.918 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:07:31.918 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:07:31.919 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4' 2024-09-18T08:07:32.019 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6 2024-09-18T08:07:32.049 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:07:32.050 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:07:32.050 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6' 2024-09-18T08:07:32.151 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0 2024-09-18T08:07:32.194 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:07:32.195 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:07:32.195 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0' 2024-09-18T08:07:32.296 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4 2024-09-18T08:07:32.323 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:07:32.324 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:07:32.325 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4' 2024-09-18T08:07:32.425 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1 2024-09-18T08:07:32.470 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:07:32.470 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:07:32.471 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1' 2024-09-18T08:07:32.571 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1 2024-09-18T08:07:32.605 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:07:32.606 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:07:32.606 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1' 2024-09-18T08:07:32.707 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6 2024-09-18T08:07:32.730 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:07:32 smithi007 ceph-mon[29459]: pgmap v213: 158 pgs: 158 active+clean; 653 MiB data, 2.9 GiB used, 712 GiB / 715 GiB avail; 0 B/s wr, 867 op/s 2024-09-18T08:07:32.730 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:07:32 smithi007 ceph-mon[29459]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-18T08:07:32.731 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:07:32 smithi007 ceph-mon[29459]: from='client.? 172.21.15.7:0/1427664164' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-18T08:07:32.731 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:07:32 smithi007 ceph-mon[29459]: osdmap e157: 8 total, 8 up, 8 in 2024-09-18T08:07:32.731 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:07:32 smithi007 ceph-mon[29459]: from='client.? 172.21.15.7:0/625853422' entity='client.admin' cmd={"prefix": "osd dump", "format": "json-pretty"} : dispatch 2024-09-18T08:07:32.731 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:07:32 smithi007 ceph-mon[39249]: pgmap v213: 158 pgs: 158 active+clean; 653 MiB data, 2.9 GiB used, 712 GiB / 715 GiB avail; 0 B/s wr, 867 op/s 2024-09-18T08:07:32.731 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:07:32 smithi007 ceph-mon[39249]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-18T08:07:32.731 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:07:32 smithi007 ceph-mon[39249]: from='client.? 172.21.15.7:0/1427664164' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-18T08:07:32.731 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:07:32 smithi007 ceph-mon[39249]: osdmap e157: 8 total, 8 up, 8 in 2024-09-18T08:07:32.731 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:07:32 smithi007 ceph-mon[39249]: from='client.? 172.21.15.7:0/625853422' entity='client.admin' cmd={"prefix": "osd dump", "format": "json-pretty"} : dispatch 2024-09-18T08:07:32.733 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:07:32.734 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:07:32.734 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6' 2024-09-18T08:07:32.791 INFO:teuthology.orchestra.run.smithi007.stdout:ERROR: (22) Invalid argument 2024-09-18T08:07:32.791 INFO:teuthology.orchestra.run.smithi007.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-18T08:07:32.835 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6 2024-09-18T08:07:32.862 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:07:32.863 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:07:32.864 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6' 2024-09-18T08:07:32.906 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:07:32 smithi050 ceph-mon[32371]: pgmap v213: 158 pgs: 158 active+clean; 653 MiB data, 2.9 GiB used, 712 GiB / 715 GiB avail; 0 B/s wr, 867 op/s 2024-09-18T08:07:32.906 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:07:32 smithi050 ceph-mon[32371]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-18T08:07:32.906 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:07:32 smithi050 ceph-mon[32371]: from='client.? 172.21.15.7:0/1427664164' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-18T08:07:32.906 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:07:32 smithi050 ceph-mon[32371]: osdmap e157: 8 total, 8 up, 8 in 2024-09-18T08:07:32.906 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:07:32 smithi050 ceph-mon[32371]: from='client.? 172.21.15.7:0/625853422' entity='client.admin' cmd={"prefix": "osd dump", "format": "json-pretty"} : dispatch 2024-09-18T08:07:32.965 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0 2024-09-18T08:07:33.004 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:07:33.005 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:07:33.005 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0' 2024-09-18T08:07:33.106 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5 2024-09-18T08:07:33.134 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:07:33.134 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:07:33.135 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5' 2024-09-18T08:07:33.236 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2 2024-09-18T08:07:33.273 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:07:33.273 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:07:33.273 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2' 2024-09-18T08:07:33.374 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0 2024-09-18T08:07:33.414 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:07:33.415 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:07:33.415 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0' 2024-09-18T08:07:33.516 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4 2024-09-18T08:07:33.543 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:07:33.544 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:07:33.544 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4' 2024-09-18T08:07:33.645 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5 2024-09-18T08:07:33.676 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:07:33.677 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:07:33.678 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5' 2024-09-18T08:07:33.778 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5 2024-09-18T08:07:33.806 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:07:33.806 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:07:33.807 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5' 2024-09-18T08:07:33.907 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4 2024-09-18T08:07:33.936 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:07:33.937 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:07:33.937 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4' 2024-09-18T08:07:34.038 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5 2024-09-18T08:07:34.044 INFO:tasks.thrashosds.thrasher:j is {'epoch': 157, 'fsid': '779349ca-7593-11ef-bceb-c7b262605968', 'created': '2024-09-18T07:57:09.295710+0000', 'modified': '2024-09-18T08:07:31.435674+0000', 'last_up_change': '2024-09-18T08:02:36.989876+0000', 'last_in_change': '2024-09-18T08:06:23.499918+0000', 'flags': 'noscrub,nodeep-scrub,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit', 'flags_num': 5806080, 'flags_set': ['nodeep-scrub', 'noscrub', 'pglog_hardlimit', 'purged_snapdirs', 'recovery_deletes', 'sortbitwise'], 'crush_version': 18, 'full_ratio': 0.949999988079071, 'backfillfull_ratio': 0.8999999761581421, 'nearfull_ratio': 0.8500000238418579, 'cluster_snapshot': '', 'pool_max': 7, 'max_osd': 8, 'require_min_compat_client': 'luminous', 'min_compat_client': 'luminous', 'require_osd_release': 'squid', 'allow_crimson': False, 'pools': [{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-09-18T08:00:11.802627+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'is_stretch_pool': False, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 11, 'pg_placement_num': 11, 'pg_placement_num_target': 11, 'pg_num_target': 11, 'pg_num_pending': 11, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '154', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '152', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.909999966621399, 'score_stable': 1.909999966621399, 'optimal_score': 1.1399999856948853, 'raw_score_acting': 2.180000066757202, 'raw_score_stable': 2.180000066757202, 'primary_affinity_weighted': 0.8412818908691406, 'average_primary_affinity': 0.9599999785423279, 'average_primary_affinity_weighted': 0.8399999737739563}}, {'pool': 2, 'pool_name': 'datapool', 'create_time': '2024-09-18T08:02:45.374099+0000', 'flags': 8193, 'flags_names': 'hashpspool,selfmanaged_snaps', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'is_stretch_pool': False, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 3, 'pg_placement_num': 3, 'pg_placement_num_target': 3, 'pg_num_target': 3, 'pg_num_pending': 3, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '64', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 2, 'snap_epoch': 64, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.3299999237060547, 'score_stable': 2.3299999237060547, 'optimal_score': 1, 'raw_score_acting': 2.3299999237060547, 'raw_score_stable': 2.3299999237060547, 'primary_affinity_weighted': 0.8412818908691406, 'average_primary_affinity': 0.9599999785423279, 'average_primary_affinity_weighted': 0.8399999737739563}}, {'pool': 3, 'pool_name': '.rgw.root', 'create_time': '2024-09-18T08:02:46.319947+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'is_stretch_pool': False, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '63', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rgw': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5399999618530273, 'score_stable': 1.5399999618530273, 'optimal_score': 1.1399999856948853, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 0.8412818908691406, 'average_primary_affinity': 0.9599999785423279, 'average_primary_affinity_weighted': 0.8399999737739563}}, {'pool': 4, 'pool_name': 'default.rgw.log', 'create_time': '2024-09-18T08:02:47.948552+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'is_stretch_pool': False, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '65', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rgw': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.190000057220459, 'score_stable': 2.190000057220459, 'optimal_score': 1.1399999856948853, 'raw_score_acting': 2.5, 'raw_score_stable': 2.5, 'primary_affinity_weighted': 0.8412818908691406, 'average_primary_affinity': 0.9599999785423279, 'average_primary_affinity_weighted': 0.8399999737739563}}, {'pool': 5, 'pool_name': 'default.rgw.control', 'create_time': '2024-09-18T08:02:49.904302+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'is_stretch_pool': False, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '67', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rgw': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.3200000524520874, 'score_stable': 1.3200000524520874, 'optimal_score': 1.1399999856948853, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.8412818908691406, 'average_primary_affinity': 0.9599999785423279, 'average_primary_affinity_weighted': 0.8399999737739563}}, {'pool': 6, 'pool_name': 'default.rgw.meta', 'create_time': '2024-09-18T08:02:51.971465+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'is_stretch_pool': False, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '70', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4}, 'application_metadata': {'rgw': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5399999618530273, 'score_stable': 1.5399999618530273, 'optimal_score': 1.1399999856948853, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 0.8412818908691406, 'average_primary_affinity': 0.9599999785423279, 'average_primary_affinity_weighted': 0.8399999737739563}}, {'pool': 7, 'pool_name': 'unique_pool_0', 'create_time': '2024-09-18T08:03:55.823788+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'is_stretch_pool': False, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 16, 'pg_placement_num': 16, 'pg_placement_num_target': 16, 'pg_num_target': 16, 'pg_num_pending': 16, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '88', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rados': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.190000057220459, 'score_stable': 2.190000057220459, 'optimal_score': 1.1399999856948853, 'raw_score_acting': 2.5, 'raw_score_stable': 2.5, 'primary_affinity_weighted': 0.8412818908691406, 'average_primary_affinity': 0.9599999785423279, 'average_primary_affinity_weighted': 0.8399999737739563}}], 'osds': [{'osd': 0, 'uuid': 'ae8f60df-72ac-46fb-91a1-f132b851c876', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 8, 'up_thru': 154, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.7:6801', 'nonce': 2896425794}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.7:6802', 'nonce': 2896425794}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.7:6804', 'nonce': 2896425794}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.7:6803', 'nonce': 2896425794}]}, 'public_addr': '172.21.15.7:6801/2896425794', 'cluster_addr': '172.21.15.7:6802/2896425794', 'heartbeat_back_addr': '172.21.15.7:6804/2896425794', 'heartbeat_front_addr': '172.21.15.7:6803/2896425794', 'state': ['exists', 'up']}, {'osd': 1, 'uuid': 'be28b70c-1f73-494f-ad4b-cb7ec5311999', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 13, 'up_thru': 154, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.7:6805', 'nonce': 1971397549}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.7:6806', 'nonce': 1971397549}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.7:6808', 'nonce': 1971397549}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.7:6807', 'nonce': 1971397549}]}, 'public_addr': '172.21.15.7:6805/1971397549', 'cluster_addr': '172.21.15.7:6806/1971397549', 'heartbeat_back_addr': '172.21.15.7:6808/1971397549', 'heartbeat_front_addr': '172.21.15.7:6807/1971397549', 'state': ['exists', 'up']}, {'osd': 2, 'uuid': 'cfa4f083-c7fd-4743-be65-84a3040fe6dc', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 18, 'up_thru': 143, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.7:6809', 'nonce': 556142559}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.7:6810', 'nonce': 556142559}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.7:6812', 'nonce': 556142559}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.7:6811', 'nonce': 556142559}]}, 'public_addr': '172.21.15.7:6809/556142559', 'cluster_addr': '172.21.15.7:6810/556142559', 'heartbeat_back_addr': '172.21.15.7:6812/556142559', 'heartbeat_front_addr': '172.21.15.7:6811/556142559', 'state': ['exists', 'up']}, {'osd': 3, 'uuid': '11ecd7e1-a751-4b75-95f5-00b32d2a5bc4', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 26, 'up_thru': 154, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.7:6813', 'nonce': 163587772}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.7:6814', 'nonce': 163587772}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.7:6816', 'nonce': 163587772}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.7:6815', 'nonce': 163587772}]}, 'public_addr': '172.21.15.7:6813/163587772', 'cluster_addr': '172.21.15.7:6814/163587772', 'heartbeat_back_addr': '172.21.15.7:6816/163587772', 'heartbeat_front_addr': '172.21.15.7:6815/163587772', 'state': ['exists', 'up']}, {'osd': 4, 'uuid': '98992b34-cfa1-45bf-97a1-7d557739252f', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 32, 'up_thru': 154, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.50:6800', 'nonce': 3662119805}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.50:6801', 'nonce': 3662119805}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.50:6803', 'nonce': 3662119805}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.50:6802', 'nonce': 3662119805}]}, 'public_addr': '172.21.15.50:6800/3662119805', 'cluster_addr': '172.21.15.50:6801/3662119805', 'heartbeat_back_addr': '172.21.15.50:6803/3662119805', 'heartbeat_front_addr': '172.21.15.50:6802/3662119805', 'state': ['exists', 'up']}, {'osd': 5, 'uuid': '7f290005-ebca-467b-87e4-02ead1063610', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 39, 'up_thru': 154, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.50:6804', 'nonce': 1199797954}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.50:6805', 'nonce': 1199797954}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.50:6807', 'nonce': 1199797954}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.50:6806', 'nonce': 1199797954}]}, 'public_addr': '172.21.15.50:6804/1199797954', 'cluster_addr': '172.21.15.50:6805/1199797954', 'heartbeat_back_addr': '172.21.15.50:6807/1199797954', 'heartbeat_front_addr': '172.21.15.50:6806/1199797954', 'state': ['exists', 'up']}, {'osd': 6, 'uuid': 'e07d846c-ae42-4fea-aa5c-3a25835d18a6', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 0, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 47, 'up_thru': 83, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.50:6808', 'nonce': 255960422}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.50:6809', 'nonce': 255960422}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.50:6811', 'nonce': 255960422}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.50:6810', 'nonce': 255960422}]}, 'public_addr': '172.21.15.50:6808/255960422', 'cluster_addr': '172.21.15.50:6809/255960422', 'heartbeat_back_addr': '172.21.15.50:6811/255960422', 'heartbeat_front_addr': '172.21.15.50:6810/255960422', 'state': ['exists', 'up']}, {'osd': 7, 'uuid': 'c9dac3b7-d651-489d-b96e-5b10fa32d9af', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 0.730255126953125, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 56, 'up_thru': 154, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.50:6812', 'nonce': 874923904}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.50:6813', 'nonce': 874923904}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.50:6815', 'nonce': 874923904}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.50:6814', 'nonce': 874923904}]}, 'public_addr': '172.21.15.50:6812/874923904', 'cluster_addr': '172.21.15.50:6813/874923904', 'heartbeat_back_addr': '172.21.15.50:6815/874923904', 'heartbeat_front_addr': '172.21.15.50:6814/874923904', 'state': ['exists', 'up']}], 'osd_xinfo': [{'osd': 0, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540719139924082687, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-18T08:06:14.699199+0000', 'dead_epoch': 0}, {'osd': 1, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540719139924082687, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-18T08:06:40.626049+0000', 'dead_epoch': 0}, {'osd': 2, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540719139924082687, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-18T08:06:12.916079+0000', 'dead_epoch': 0}, {'osd': 3, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540719139924082687, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-18T08:07:25.802428+0000', 'dead_epoch': 0}, {'osd': 4, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540719139924082687, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-18T08:06:56.054937+0000', 'dead_epoch': 0}, {'osd': 5, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540719139924082687, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-18T08:07:28.333379+0000', 'dead_epoch': 0}, {'osd': 6, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540719139924082687, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-18T08:06:59.725318+0000', 'dead_epoch': 0}, {'osd': 7, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540719139924082687, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-18T08:06:42.410072+0000', 'dead_epoch': 0}], 'pg_upmap': [], 'pg_upmap_items': [{'pgid': '4.7', 'mappings': [{'from': 1, 'to': 2}]}, {'pgid': '4.14', 'mappings': [{'from': 1, 'to': 2}]}, {'pgid': '4.1c', 'mappings': [{'from': 1, 'to': 5}]}], 'pg_upmap_primaries': [], 'pg_temp': [], 'primary_temp': [], 'blocklist': {'172.21.15.7:0/398367658': '2024-09-19T08:03:17.052903+0000', '172.21.15.7:6800/2918496291': '2024-09-19T07:57:31.015269+0000', '172.21.15.7:0/2514354773': '2024-09-19T07:57:31.015269+0000', '172.21.15.7:0/3924911335': '2024-09-19T07:57:31.015269+0000', '172.21.15.7:0/1668598084': '2024-09-19T07:57:57.346149+0000', '172.21.15.7:0/4272027292': '2024-09-19T07:57:31.015269+0000', '172.21.15.7:6800/1013061071': '2024-09-19T07:57:57.346149+0000', '172.21.15.7:6800/1491061076': '2024-09-19T08:03:17.052903+0000', '172.21.15.7:0/533727535': '2024-09-19T07:57:57.346149+0000', '172.21.15.7:0/1605217610': '2024-09-19T08:03:17.052903+0000', '172.21.15.7:0/1722968065': '2024-09-19T08:03:17.052903+0000', '172.21.15.7:0/1277549846': '2024-09-19T07:57:57.346149+0000', '172.21.15.7:0/232452890': '2024-09-19T08:03:17.052903+0000', '172.21.15.7:0/3881670470': '2024-09-19T08:03:17.052903+0000'}, 'range_blocklist': {}, 'erasure_code_profiles': {'default': {'crush-failure-domain': 'osd', 'k': '2', 'm': '1', 'plugin': 'jerasure', 'technique': 'reed_sol_van'}}, 'removed_snaps_queue': [], 'new_removed_snaps': [], 'new_purged_snaps': [], 'crush_node_flags': {}, 'device_class_flags': {}, 'stretch_mode': {'stretch_mode_enabled': False, 'stretch_bucket_count': 0, 'degraded_stretch_mode': 0, 'recovering_stretch_mode': 0, 'stretch_mode_bucket': 0}} 2024-09-18T08:07:34.044 INFO:tasks.thrashosds.thrasher:Clearing pg_upmap on 4.1c 2024-09-18T08:07:34.044 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid 779349ca-7593-11ef-bceb-c7b262605968 -- ceph osd rm-pg-upmap-items 4.1c 2024-09-18T08:07:34.049 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/779349ca-7593-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T08:07:34.062 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:07:34.063 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:07:34.063 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5' 2024-09-18T08:07:34.164 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6 2024-09-18T08:07:34.188 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:07:34.189 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:07:34.190 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6' 2024-09-18T08:07:34.230 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:07:33 smithi007 ceph-mon[29459]: from='mgr.24470 172.21.15.7:0/2594390261' entity='mgr.y' 2024-09-18T08:07:34.231 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:07:33 smithi007 ceph-mon[29459]: from='mgr.24470 172.21.15.7:0/2594390261' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-18T08:07:34.231 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:07:33 smithi007 ceph-mon[29459]: pgmap v215: 158 pgs: 158 active+clean; 613 MiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 0 B/s wr, 862 op/s 2024-09-18T08:07:34.231 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:07:33 smithi007 ceph-mon[39249]: from='mgr.24470 172.21.15.7:0/2594390261' entity='mgr.y' 2024-09-18T08:07:34.231 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:07:33 smithi007 ceph-mon[39249]: from='mgr.24470 172.21.15.7:0/2594390261' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-18T08:07:34.231 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:07:33 smithi007 ceph-mon[39249]: pgmap v215: 158 pgs: 158 active+clean; 613 MiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 0 B/s wr, 862 op/s 2024-09-18T08:07:34.291 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1 2024-09-18T08:07:34.340 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:07:34.341 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:07:34.341 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1' 2024-09-18T08:07:34.406 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:07:33 smithi050 ceph-mon[32371]: from='mgr.24470 172.21.15.7:0/2594390261' entity='mgr.y' 2024-09-18T08:07:34.406 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:07:33 smithi050 ceph-mon[32371]: from='mgr.24470 172.21.15.7:0/2594390261' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-18T08:07:34.406 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:07:33 smithi050 ceph-mon[32371]: pgmap v215: 158 pgs: 158 active+clean; 613 MiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 0 B/s wr, 862 op/s 2024-09-18T08:07:34.441 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0 2024-09-18T08:07:34.465 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:07:34.465 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:07:34.466 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0' 2024-09-18T08:07:34.492 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid 779349ca-7593-11ef-bceb-c7b262605968 -- ceph osd unset noscrub 2024-09-18T08:07:34.567 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6 2024-09-18T08:07:34.595 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:07:34.596 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:07:34.596 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6' 2024-09-18T08:07:34.697 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1 2024-09-18T08:07:34.744 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:07:34.744 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:07:34.745 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1' 2024-09-18T08:07:34.845 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2 2024-09-18T08:07:34.894 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:07:34.894 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:07:34.894 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2' 2024-09-18T08:07:34.995 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6 2024-09-18T08:07:35.019 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:07:35.020 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:07:35.020 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6' 2024-09-18T08:07:35.121 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0 2024-09-18T08:07:35.156 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:07:35.157 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:07:35.158 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0' 2024-09-18T08:07:35.259 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0 2024-09-18T08:07:35.301 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:07:35.302 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:07:35.303 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0' 2024-09-18T08:07:35.403 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2 2024-09-18T08:07:35.448 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:07:35.449 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:07:35.449 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2' 2024-09-18T08:07:35.472 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid 779349ca-7593-11ef-bceb-c7b262605968 -- ceph daemon osd.2 dump_historic_ops 2024-09-18T08:07:35.550 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3 2024-09-18T08:07:35.609 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:07:35.610 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:07:35.610 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3' 2024-09-18T08:07:35.711 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2 2024-09-18T08:07:35.766 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:07:35.766 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:07:35.767 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2' 2024-09-18T08:07:35.867 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2 2024-09-18T08:07:35.913 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:07:35.914 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:07:35.914 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2' 2024-09-18T08:07:36.015 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4 2024-09-18T08:07:36.040 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:07:36.041 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:07:36.041 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4' 2024-09-18T08:07:36.111 INFO:teuthology.orchestra.run.smithi007.stdout:osd.0: {} 2024-09-18T08:07:36.112 INFO:teuthology.orchestra.run.smithi007.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-18T08:07:36.131 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/779349ca-7593-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T08:07:36.138 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/779349ca-7593-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T08:07:36.142 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2 2024-09-18T08:07:36.166 INFO:teuthology.orchestra.run.smithi007.stdout:osd.1: {} 2024-09-18T08:07:36.166 INFO:teuthology.orchestra.run.smithi007.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-18T08:07:36.196 INFO:teuthology.orchestra.run.smithi007.stdout:osd.2: {} 2024-09-18T08:07:36.197 INFO:teuthology.orchestra.run.smithi007.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-18T08:07:36.209 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:07:36.209 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:07:36.209 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2' 2024-09-18T08:07:36.241 INFO:teuthology.orchestra.run.smithi007.stdout:osd.3: {} 2024-09-18T08:07:36.242 INFO:teuthology.orchestra.run.smithi007.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-18T08:07:36.273 INFO:teuthology.orchestra.run.smithi007.stdout:osd.4: {} 2024-09-18T08:07:36.273 INFO:teuthology.orchestra.run.smithi007.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-18T08:07:36.273 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:07:36 smithi007 ceph-mon[39249]: pgmap v216: 158 pgs: 158 active+clean; 323 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 0 B/s wr, 1.31k op/s 2024-09-18T08:07:36.273 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:07:36 smithi007 ceph-mon[39249]: from='client.? 172.21.15.7:0/1384633505' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-18T08:07:36.274 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:07:36 smithi007 ceph-mon[29459]: pgmap v216: 158 pgs: 158 active+clean; 323 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 0 B/s wr, 1.31k op/s 2024-09-18T08:07:36.274 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:07:36 smithi007 ceph-mon[29459]: from='client.? 172.21.15.7:0/1384633505' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-18T08:07:36.310 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2 2024-09-18T08:07:36.312 INFO:teuthology.orchestra.run.smithi007.stdout:osd.5: {} 2024-09-18T08:07:36.312 INFO:teuthology.orchestra.run.smithi007.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-18T08:07:36.342 INFO:teuthology.orchestra.run.smithi007.stdout:osd.6: {} 2024-09-18T08:07:36.342 INFO:teuthology.orchestra.run.smithi007.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-18T08:07:36.369 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:07:36.370 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:07:36.371 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2' 2024-09-18T08:07:36.377 INFO:teuthology.orchestra.run.smithi007.stdout:osd.7: {} 2024-09-18T08:07:36.377 INFO:teuthology.orchestra.run.smithi007.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-18T08:07:36.472 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1 2024-09-18T08:07:36.524 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:07:36.524 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:07:36.525 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1' 2024-09-18T08:07:36.626 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5 2024-09-18T08:07:36.653 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:07:36.654 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:07:36.654 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5' 2024-09-18T08:07:36.655 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:07:36 smithi050 ceph-mon[32371]: pgmap v216: 158 pgs: 158 active+clean; 323 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 0 B/s wr, 1.31k op/s 2024-09-18T08:07:36.655 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:07:36 smithi050 ceph-mon[32371]: from='client.? 172.21.15.7:0/1384633505' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-18T08:07:36.755 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5 2024-09-18T08:07:36.782 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:07:36.783 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:07:36.783 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5' 2024-09-18T08:07:36.884 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4 2024-09-18T08:07:36.910 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:07:36.911 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:07:36.911 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4' 2024-09-18T08:07:36.980 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Sep 18 08:07:36 smithi007 ceph-779349ca-7593-11ef-bceb-c7b262605968-mgr-y[29688]: ::ffff:172.21.15.50 - - [18/Sep/2024:08:07:36] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-09-18T08:07:37.012 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7 2024-09-18T08:07:37.037 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:07:37.038 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:07:37.038 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7' 2024-09-18T08:07:37.139 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3 2024-09-18T08:07:37.177 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:07:37.177 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:07:37.177 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3' 2024-09-18T08:07:37.278 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7 2024-09-18T08:07:37.304 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:07:37.305 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:07:37.305 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7' 2024-09-18T08:07:37.406 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4 2024-09-18T08:07:37.438 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:07:37.439 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:07:37.439 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4' 2024-09-18T08:07:37.480 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:07:37 smithi007 ceph-mon[29459]: from='mgr.24470 172.21.15.7:0/2594390261' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-18T08:07:37.481 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:07:37 smithi007 ceph-mon[39249]: from='mgr.24470 172.21.15.7:0/2594390261' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-18T08:07:37.540 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1 2024-09-18T08:07:37.604 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:07:37.605 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:07:37.605 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1' 2024-09-18T08:07:37.655 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:07:37 smithi050 ceph-mon[32371]: from='mgr.24470 172.21.15.7:0/2594390261' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-18T08:07:37.678 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/779349ca-7593-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T08:07:37.706 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2 2024-09-18T08:07:37.751 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:07:37.752 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:07:37.752 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2' 2024-09-18T08:07:37.853 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0 2024-09-18T08:07:37.909 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:07:37.910 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:07:37.910 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0' 2024-09-18T08:07:38.011 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2 2024-09-18T08:07:38.050 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:07:38.051 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:07:38.051 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2' 2024-09-18T08:07:38.152 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6 2024-09-18T08:07:38.177 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:07:38.178 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:07:38.179 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6' 2024-09-18T08:07:38.263 INFO:teuthology.orchestra.run.smithi007.stderr:clear 4.1c pg_upmap_items mapping 2024-09-18T08:07:38.279 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3 2024-09-18T08:07:38.353 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:07:38.353 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:07:38.354 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3' 2024-09-18T08:07:38.368 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:07:38 smithi007 ceph-mon[29459]: purged_snaps scrub starts 2024-09-18T08:07:38.369 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:07:38 smithi007 ceph-mon[29459]: purged_snaps scrub ok 2024-09-18T08:07:38.369 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:07:38 smithi007 ceph-mon[29459]: pgmap v217: 158 pgs: 158 active+clean; 323 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 9.7 KiB/s rd, 68 KiB/s wr, 949 op/s; 60 KiB/s, 0 objects/s recovering 2024-09-18T08:07:38.369 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:07:38 smithi007 ceph-mon[29459]: osdmap e158: 8 total, 8 up, 8 in 2024-09-18T08:07:38.369 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:07:38 smithi007 ceph-mon[29459]: from='client.? 172.21.15.7:0/4186386333' entity='client.admin' cmd={"prefix": "osd rm-pg-upmap-items", "pgid": "4.1c"} : dispatch 2024-09-18T08:07:38.369 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:07:38 smithi007 ceph-mon[29459]: from='mgr.24470 172.21.15.7:0/2594390261' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-18T08:07:38.369 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:07:38 smithi007 ceph-mon[29459]: from='mgr.24470 172.21.15.7:0/2594390261' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-18T08:07:38.369 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:07:38 smithi007 ceph-mon[29459]: from='mgr.24470 172.21.15.7:0/2594390261' entity='mgr.y' 2024-09-18T08:07:38.370 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:07:38 smithi007 ceph-mon[39249]: purged_snaps scrub starts 2024-09-18T08:07:38.370 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:07:38 smithi007 ceph-mon[39249]: purged_snaps scrub ok 2024-09-18T08:07:38.370 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:07:38 smithi007 ceph-mon[39249]: pgmap v217: 158 pgs: 158 active+clean; 323 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 9.7 KiB/s rd, 68 KiB/s wr, 949 op/s; 60 KiB/s, 0 objects/s recovering 2024-09-18T08:07:38.370 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:07:38 smithi007 ceph-mon[39249]: osdmap e158: 8 total, 8 up, 8 in 2024-09-18T08:07:38.370 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:07:38 smithi007 ceph-mon[39249]: from='client.? 172.21.15.7:0/4186386333' entity='client.admin' cmd={"prefix": "osd rm-pg-upmap-items", "pgid": "4.1c"} : dispatch 2024-09-18T08:07:38.370 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:07:38 smithi007 ceph-mon[39249]: from='mgr.24470 172.21.15.7:0/2594390261' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-18T08:07:38.370 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:07:38 smithi007 ceph-mon[39249]: from='mgr.24470 172.21.15.7:0/2594390261' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-18T08:07:38.370 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:07:38 smithi007 ceph-mon[39249]: from='mgr.24470 172.21.15.7:0/2594390261' entity='mgr.y' 2024-09-18T08:07:38.455 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1 2024-09-18T08:07:38.499 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:07:38.500 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:07:38.500 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1' 2024-09-18T08:07:38.601 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7 2024-09-18T08:07:38.627 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:07:38.628 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:07:38.628 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7' 2024-09-18T08:07:38.656 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:07:38 smithi050 ceph-mon[32371]: purged_snaps scrub starts 2024-09-18T08:07:38.656 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:07:38 smithi050 ceph-mon[32371]: purged_snaps scrub ok 2024-09-18T08:07:38.656 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:07:38 smithi050 ceph-mon[32371]: pgmap v217: 158 pgs: 158 active+clean; 323 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 9.7 KiB/s rd, 68 KiB/s wr, 949 op/s; 60 KiB/s, 0 objects/s recovering 2024-09-18T08:07:38.656 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:07:38 smithi050 ceph-mon[32371]: osdmap e158: 8 total, 8 up, 8 in 2024-09-18T08:07:38.656 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:07:38 smithi050 ceph-mon[32371]: from='client.? 172.21.15.7:0/4186386333' entity='client.admin' cmd={"prefix": "osd rm-pg-upmap-items", "pgid": "4.1c"} : dispatch 2024-09-18T08:07:38.656 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:07:38 smithi050 ceph-mon[32371]: from='mgr.24470 172.21.15.7:0/2594390261' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-18T08:07:38.656 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:07:38 smithi050 ceph-mon[32371]: from='mgr.24470 172.21.15.7:0/2594390261' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-18T08:07:38.656 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:07:38 smithi050 ceph-mon[32371]: from='mgr.24470 172.21.15.7:0/2594390261' entity='mgr.y' 2024-09-18T08:07:38.682 INFO:tasks.radosbench.radosbench.0.smithi007.stdout:Removed 118288 objects 2024-09-18T08:07:38.682 INFO:tasks.radosbench.radosbench.0.smithi007.stdout:Clean up completed and total clean up time :123.085 2024-09-18T08:07:38.729 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3 2024-09-18T08:07:38.751 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:07:38.752 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:07:38.752 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3' 2024-09-18T08:07:38.853 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4 2024-09-18T08:07:38.876 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:07:38.876 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:07:38.876 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4' 2024-09-18T08:07:38.977 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0 2024-09-18T08:07:39.001 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:07:39.001 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:07:39.002 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0' 2024-09-18T08:07:39.031 INFO:tasks.cephadm.ceph_manager.ceph:removing pool_name unique_pool_0 2024-09-18T08:07:39.032 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid 779349ca-7593-11ef-bceb-c7b262605968 -- ceph osd pool rm unique_pool_0 unique_pool_0 --yes-i-really-really-mean-it 2024-09-18T08:07:39.102 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7 2024-09-18T08:07:39.127 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:07:39.128 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:07:39.128 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7' 2024-09-18T08:07:39.229 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0 2024-09-18T08:07:39.252 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:07:39.253 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:07:39.253 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0' 2024-09-18T08:07:39.265 INFO:journalctl@ceph.iscsi.iscsi.a.smithi050.stdout:Sep 18 08:07:38 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-iscsi-iscsi-a[57782]: debug there is no tcmu-runner data available 2024-09-18T08:07:39.269 INFO:teuthology.orchestra.run.smithi007.stderr:noscrub is unset 2024-09-18T08:07:39.354 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0 2024-09-18T08:07:39.378 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:07:39.378 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:07:39.378 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0' 2024-09-18T08:07:39.479 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7 2024-09-18T08:07:39.480 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:07:39 smithi007 ceph-mon[39249]: from='client.? 172.21.15.7:0/4186386333' entity='client.admin' cmd='[{"prefix": "osd rm-pg-upmap-items", "pgid": "4.1c"}]': finished 2024-09-18T08:07:39.481 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:07:39 smithi007 ceph-mon[39249]: osdmap e159: 8 total, 8 up, 8 in 2024-09-18T08:07:39.481 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:07:39 smithi007 ceph-mon[39249]: from='client.? 172.21.15.7:0/1473165803' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-18T08:07:39.481 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:07:39 smithi007 ceph-mon[39249]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-18T08:07:39.481 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:07:39 smithi007 ceph-mon[29459]: from='client.? 172.21.15.7:0/4186386333' entity='client.admin' cmd='[{"prefix": "osd rm-pg-upmap-items", "pgid": "4.1c"}]': finished 2024-09-18T08:07:39.482 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:07:39 smithi007 ceph-mon[29459]: osdmap e159: 8 total, 8 up, 8 in 2024-09-18T08:07:39.482 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:07:39 smithi007 ceph-mon[29459]: from='client.? 172.21.15.7:0/1473165803' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-18T08:07:39.482 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:07:39 smithi007 ceph-mon[29459]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-18T08:07:39.508 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:07:39.509 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:07:39.509 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7' 2024-09-18T08:07:39.590 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid 779349ca-7593-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-18T08:07:39.611 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5 2024-09-18T08:07:39.637 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:07:39.638 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:07:39.639 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5' 2024-09-18T08:07:39.656 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:07:39 smithi050 ceph-mon[32371]: from='client.? 172.21.15.7:0/4186386333' entity='client.admin' cmd='[{"prefix": "osd rm-pg-upmap-items", "pgid": "4.1c"}]': finished 2024-09-18T08:07:39.656 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:07:39 smithi050 ceph-mon[32371]: osdmap e159: 8 total, 8 up, 8 in 2024-09-18T08:07:39.656 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:07:39 smithi050 ceph-mon[32371]: from='client.? 172.21.15.7:0/1473165803' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-18T08:07:39.656 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:07:39 smithi050 ceph-mon[32371]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-18T08:07:39.739 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3 2024-09-18T08:07:39.761 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:07:39.762 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:07:39.762 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3' 2024-09-18T08:07:39.863 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0 2024-09-18T08:07:39.885 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:07:39.885 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:07:39.886 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0' 2024-09-18T08:07:39.987 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3 2024-09-18T08:07:40.009 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:07:40.009 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:07:40.010 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3' 2024-09-18T08:07:40.111 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0 2024-09-18T08:07:40.133 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:07:40.134 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:07:40.134 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0' 2024-09-18T08:07:40.235 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1 2024-09-18T08:07:40.258 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:07:40.259 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:07:40.259 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1' 2024-09-18T08:07:40.360 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2 2024-09-18T08:07:40.386 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:07:40.387 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:07:40.387 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2' 2024-09-18T08:07:40.479 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:07:40 smithi007 ceph-mon[29459]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-18T08:07:40.479 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:07:40 smithi007 ceph-mon[29459]: pgmap v220: 158 pgs: 1 peering, 157 active+clean; 11 MiB data, 1.0 GiB used, 714 GiB / 715 GiB avail; 1.3 KiB/s rd, 0 B/s wr, 1.33k op/s 2024-09-18T08:07:40.479 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:07:40 smithi007 ceph-mon[29459]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-18T08:07:40.480 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:07:40 smithi007 ceph-mon[29459]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-18T08:07:40.480 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:07:40 smithi007 ceph-mon[29459]: osdmap e160: 8 total, 8 up, 8 in 2024-09-18T08:07:40.480 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:07:40 smithi007 ceph-mon[29459]: 4.1c scrub starts 2024-09-18T08:07:40.480 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:07:40 smithi007 ceph-mon[29459]: 4.1c scrub ok 2024-09-18T08:07:40.480 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:07:40 smithi007 ceph-mon[29459]: 4.6 scrub starts 2024-09-18T08:07:40.480 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:07:40 smithi007 ceph-mon[29459]: 4.6 scrub ok 2024-09-18T08:07:40.481 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:07:40 smithi007 ceph-mon[39249]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-18T08:07:40.481 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:07:40 smithi007 ceph-mon[39249]: pgmap v220: 158 pgs: 1 peering, 157 active+clean; 11 MiB data, 1.0 GiB used, 714 GiB / 715 GiB avail; 1.3 KiB/s rd, 0 B/s wr, 1.33k op/s 2024-09-18T08:07:40.481 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:07:40 smithi007 ceph-mon[39249]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-18T08:07:40.481 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:07:40 smithi007 ceph-mon[39249]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-18T08:07:40.482 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:07:40 smithi007 ceph-mon[39249]: osdmap e160: 8 total, 8 up, 8 in 2024-09-18T08:07:40.482 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:07:40 smithi007 ceph-mon[39249]: 4.1c scrub starts 2024-09-18T08:07:40.482 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:07:40 smithi007 ceph-mon[39249]: 4.1c scrub ok 2024-09-18T08:07:40.482 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:07:40 smithi007 ceph-mon[39249]: 4.6 scrub starts 2024-09-18T08:07:40.482 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:07:40 smithi007 ceph-mon[39249]: 4.6 scrub ok 2024-09-18T08:07:40.489 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0 2024-09-18T08:07:40.511 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:07:40.511 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:07:40.512 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0' 2024-09-18T08:07:40.588 INFO:teuthology.orchestra.run.smithi007.stdout:{ 2024-09-18T08:07:40.588 INFO:teuthology.orchestra.run.smithi007.stdout: "size": 20, 2024-09-18T08:07:40.588 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 600, 2024-09-18T08:07:40.588 INFO:teuthology.orchestra.run.smithi007.stdout: "ops": [ 2024-09-18T08:07:40.588 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:07:40.588 INFO:teuthology.orchestra.run.smithi007.stdout: "description": "osd_op(client.14805.0:66027 7.e 7:717f12e1:::benchmark_data_smithi007_73155_object66026:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e102)", 2024-09-18T08:07:40.588 INFO:teuthology.orchestra.run.smithi007.stdout: "initiated_at": "2024-09-18T08:04:43.957914+0000", 2024-09-18T08:07:40.588 INFO:teuthology.orchestra.run.smithi007.stdout: "age": 176.62180358000001, 2024-09-18T08:07:40.588 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.087895244999999997, 2024-09-18T08:07:40.588 INFO:teuthology.orchestra.run.smithi007.stdout: "continuous": false, 2024-09-18T08:07:40.589 INFO:teuthology.orchestra.run.smithi007.stdout: "type_data": { 2024-09-18T08:07:40.589 INFO:teuthology.orchestra.run.smithi007.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T08:07:40.589 INFO:teuthology.orchestra.run.smithi007.stdout: "client_info": { 2024-09-18T08:07:40.589 INFO:teuthology.orchestra.run.smithi007.stdout: "client": "client.14805", 2024-09-18T08:07:40.589 INFO:teuthology.orchestra.run.smithi007.stdout: "client_addr": "172.21.15.7:0/640320436", 2024-09-18T08:07:40.589 INFO:teuthology.orchestra.run.smithi007.stdout: "tid": 66027 2024-09-18T08:07:40.589 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:07:40.589 INFO:teuthology.orchestra.run.smithi007.stdout: "events": [ 2024-09-18T08:07:40.589 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:07:40.589 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "initiated", 2024-09-18T08:07:40.590 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:04:43.957914+0000", 2024-09-18T08:07:40.590 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-09-18T08:07:40.590 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:07:40.590 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:07:40.590 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "header_read", 2024-09-18T08:07:40.590 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:04:43.957914+0000", 2024-09-18T08:07:40.590 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-09-18T08:07:40.590 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:07:40.590 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:07:40.590 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "throttled", 2024-09-18T08:07:40.590 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:04:43.957915+0000", 2024-09-18T08:07:40.590 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 5.37e-07 2024-09-18T08:07:40.591 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:07:40.591 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:07:40.591 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "all_read", 2024-09-18T08:07:40.591 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:04:43.957936+0000", 2024-09-18T08:07:40.591 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 2.0885000000000001e-05 2024-09-18T08:07:40.591 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:07:40.591 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:07:40.591 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "dispatched", 2024-09-18T08:07:40.591 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:04:43.957943+0000", 2024-09-18T08:07:40.591 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 6.9190000000000002e-06 2024-09-18T08:07:40.591 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:07:40.591 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:07:40.591 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "queued_for_pg", 2024-09-18T08:07:40.592 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:04:43.957948+0000", 2024-09-18T08:07:40.592 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 5.3539999999999999e-06 2024-09-18T08:07:40.592 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:07:40.592 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:07:40.592 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-09-18T08:07:40.592 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:04:43.960745+0000", 2024-09-18T08:07:40.592 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.002796672 2024-09-18T08:07:40.592 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:07:40.592 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:07:40.592 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "started", 2024-09-18T08:07:40.592 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:04:43.960952+0000", 2024-09-18T08:07:40.592 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.00020684300000000001 2024-09-18T08:07:40.593 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:07:40.593 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:07:40.593 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "waiting for subops from 0,5", 2024-09-18T08:07:40.593 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:04:43.961089+0000", 2024-09-18T08:07:40.593 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.000137155 2024-09-18T08:07:40.593 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:07:40.593 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:07:40.593 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:07:40.593 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:04:43.970514+0000", 2024-09-18T08:07:40.593 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.0094257409999999996 2024-09-18T08:07:40.593 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:07:40.593 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:07:40.593 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "op_commit", 2024-09-18T08:07:40.594 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:04:43.970575+0000", 2024-09-18T08:07:40.594 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 6.0489999999999999e-05 2024-09-18T08:07:40.594 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:07:40.594 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:07:40.594 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:07:40.594 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:04:44.045722+0000", 2024-09-18T08:07:40.594 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.075147275999999999 2024-09-18T08:07:40.594 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:07:40.594 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:07:40.594 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "commit_sent", 2024-09-18T08:07:40.594 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:04:44.045770+0000", 2024-09-18T08:07:40.594 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 4.7539e-05 2024-09-18T08:07:40.594 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:07:40.595 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:07:40.595 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "done", 2024-09-18T08:07:40.595 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:04:44.045810+0000", 2024-09-18T08:07:40.595 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 3.9833999999999998e-05 2024-09-18T08:07:40.595 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-09-18T08:07:40.596 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-09-18T08:07:40.596 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-09-18T08:07:40.596 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:07:40.596 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:07:40.596 INFO:teuthology.orchestra.run.smithi007.stdout: "description": "osd_op(client.14805.0:66030 7.e 7:7b16ec01:::benchmark_data_smithi007_73155_object66029:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e102)", 2024-09-18T08:07:40.596 INFO:teuthology.orchestra.run.smithi007.stdout: "initiated_at": "2024-09-18T08:04:43.960797+0000", 2024-09-18T08:07:40.596 INFO:teuthology.orchestra.run.smithi007.stdout: "age": 176.61892117900001, 2024-09-18T08:07:40.596 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.089093715000000004, 2024-09-18T08:07:40.596 INFO:teuthology.orchestra.run.smithi007.stdout: "continuous": false, 2024-09-18T08:07:40.596 INFO:teuthology.orchestra.run.smithi007.stdout: "type_data": { 2024-09-18T08:07:40.596 INFO:teuthology.orchestra.run.smithi007.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T08:07:40.596 INFO:teuthology.orchestra.run.smithi007.stdout: "client_info": { 2024-09-18T08:07:40.597 INFO:teuthology.orchestra.run.smithi007.stdout: "client": "client.14805", 2024-09-18T08:07:40.597 INFO:teuthology.orchestra.run.smithi007.stdout: "client_addr": "172.21.15.7:0/640320436", 2024-09-18T08:07:40.597 INFO:teuthology.orchestra.run.smithi007.stdout: "tid": 66030 2024-09-18T08:07:40.597 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:07:40.597 INFO:teuthology.orchestra.run.smithi007.stdout: "events": [ 2024-09-18T08:07:40.597 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:07:40.597 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "initiated", 2024-09-18T08:07:40.597 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:04:43.960797+0000", 2024-09-18T08:07:40.597 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-09-18T08:07:40.597 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:07:40.597 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:07:40.597 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "header_read", 2024-09-18T08:07:40.598 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:04:43.960797+0000", 2024-09-18T08:07:40.598 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-09-18T08:07:40.598 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:07:40.598 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:07:40.598 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "throttled", 2024-09-18T08:07:40.598 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:04:43.960798+0000", 2024-09-18T08:07:40.598 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.1909999999999999e-06 2024-09-18T08:07:40.598 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:07:40.598 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:07:40.598 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "all_read", 2024-09-18T08:07:40.598 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:04:43.960822+0000", 2024-09-18T08:07:40.598 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 2.3785999999999999e-05 2024-09-18T08:07:40.598 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:07:40.598 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:07:40.599 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "dispatched", 2024-09-18T08:07:40.599 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:04:43.960830+0000", 2024-09-18T08:07:40.599 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 8.2400000000000007e-06 2024-09-18T08:07:40.599 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:07:40.599 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:07:40.599 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "queued_for_pg", 2024-09-18T08:07:40.599 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:04:43.960837+0000", 2024-09-18T08:07:40.599 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 6.7109999999999998e-06 2024-09-18T08:07:40.599 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:07:40.599 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:07:40.599 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-09-18T08:07:40.599 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:04:43.964072+0000", 2024-09-18T08:07:40.599 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.0032352380000000001 2024-09-18T08:07:40.600 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:07:40.600 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:07:40.600 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "started", 2024-09-18T08:07:40.600 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:04:43.964179+0000", 2024-09-18T08:07:40.600 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.000107524 2024-09-18T08:07:40.600 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:07:40.600 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:07:40.600 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "waiting for subops from 0,5", 2024-09-18T08:07:40.600 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:04:43.964697+0000", 2024-09-18T08:07:40.600 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.000517097 2024-09-18T08:07:40.601 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:07:40.601 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:07:40.601 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:07:40.601 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:04:43.970563+0000", 2024-09-18T08:07:40.601 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.0058661649999999996 2024-09-18T08:07:40.601 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:07:40.601 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:07:40.601 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "op_commit", 2024-09-18T08:07:40.601 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:04:43.970665+0000", 2024-09-18T08:07:40.601 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.000101888 2024-09-18T08:07:40.601 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:07:40.601 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:07:40.601 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:07:40.601 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:04:44.049154+0000", 2024-09-18T08:07:40.602 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.078489797 2024-09-18T08:07:40.602 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:07:40.602 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:07:40.602 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "commit_sent", 2024-09-18T08:07:40.602 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:04:44.049385+0000", 2024-09-18T08:07:40.602 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.00023076499999999999 2024-09-18T08:07:40.602 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:07:40.602 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:07:40.602 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "done", 2024-09-18T08:07:40.603 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:04:44.049890+0000", 2024-09-18T08:07:40.603 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.00050531299999999996 2024-09-18T08:07:40.603 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-09-18T08:07:40.603 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-09-18T08:07:40.603 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-09-18T08:07:40.603 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:07:40.603 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:07:40.603 INFO:teuthology.orchestra.run.smithi007.stdout: "description": "osd_op(client.14805.0:94181 7.e 7:72880d48:::benchmark_data_smithi007_73155_object94180:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e108)", 2024-09-18T08:07:40.603 INFO:teuthology.orchestra.run.smithi007.stdout: "initiated_at": "2024-09-18T08:05:09.961032+0000", 2024-09-18T08:07:40.603 INFO:teuthology.orchestra.run.smithi007.stdout: "age": 150.61868568200001, 2024-09-18T08:07:40.604 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.103403914, 2024-09-18T08:07:40.604 INFO:teuthology.orchestra.run.smithi007.stdout: "continuous": false, 2024-09-18T08:07:40.604 INFO:teuthology.orchestra.run.smithi007.stdout: "type_data": { 2024-09-18T08:07:40.604 INFO:teuthology.orchestra.run.smithi007.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T08:07:40.604 INFO:teuthology.orchestra.run.smithi007.stdout: "client_info": { 2024-09-18T08:07:40.604 INFO:teuthology.orchestra.run.smithi007.stdout: "client": "client.14805", 2024-09-18T08:07:40.604 INFO:teuthology.orchestra.run.smithi007.stdout: "client_addr": "172.21.15.7:0/640320436", 2024-09-18T08:07:40.604 INFO:teuthology.orchestra.run.smithi007.stdout: "tid": 94181 2024-09-18T08:07:40.604 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:07:40.605 INFO:teuthology.orchestra.run.smithi007.stdout: "events": [ 2024-09-18T08:07:40.605 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:07:40.605 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "initiated", 2024-09-18T08:07:40.605 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:05:09.961032+0000", 2024-09-18T08:07:40.605 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-09-18T08:07:40.605 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:07:40.605 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:07:40.605 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "header_read", 2024-09-18T08:07:40.605 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:05:09.961032+0000", 2024-09-18T08:07:40.605 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-09-18T08:07:40.605 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:07:40.605 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:07:40.605 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "throttled", 2024-09-18T08:07:40.606 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:05:09.961034+0000", 2024-09-18T08:07:40.606 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.342e-06 2024-09-18T08:07:40.606 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:07:40.606 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:07:40.606 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "all_read", 2024-09-18T08:07:40.606 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:05:09.961055+0000", 2024-09-18T08:07:40.606 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 2.1114000000000001e-05 2024-09-18T08:07:40.606 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:07:40.606 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:07:40.606 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "dispatched", 2024-09-18T08:07:40.606 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:05:09.961066+0000", 2024-09-18T08:07:40.606 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.1185999999999999e-05 2024-09-18T08:07:40.606 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:07:40.606 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:07:40.607 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "queued_for_pg", 2024-09-18T08:07:40.607 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:05:09.961073+0000", 2024-09-18T08:07:40.607 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 6.9630000000000001e-06 2024-09-18T08:07:40.607 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:07:40.607 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:07:40.607 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-09-18T08:07:40.607 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:05:09.961131+0000", 2024-09-18T08:07:40.607 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 5.7666e-05 2024-09-18T08:07:40.607 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:07:40.607 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:07:40.608 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "started", 2024-09-18T08:07:40.608 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:05:09.961241+0000", 2024-09-18T08:07:40.608 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.000110321 2024-09-18T08:07:40.608 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:07:40.608 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:07:40.608 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "waiting for subops from 0,5", 2024-09-18T08:07:40.608 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:05:09.961364+0000", 2024-09-18T08:07:40.608 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.00012286400000000001 2024-09-18T08:07:40.608 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:07:40.608 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:07:40.608 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:07:40.608 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:05:09.965873+0000", 2024-09-18T08:07:40.608 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.0045095930000000001 2024-09-18T08:07:40.609 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:07:40.609 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:07:40.609 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "op_commit", 2024-09-18T08:07:40.609 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:05:09.965897+0000", 2024-09-18T08:07:40.609 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 2.4077999999999999e-05 2024-09-18T08:07:40.609 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:07:40.609 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:07:40.609 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:07:40.609 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:05:10.064128+0000", 2024-09-18T08:07:40.609 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.098230856000000005 2024-09-18T08:07:40.609 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:07:40.609 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:07:40.609 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "commit_sent", 2024-09-18T08:07:40.610 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:05:10.064388+0000", 2024-09-18T08:07:40.610 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.00025948099999999999 2024-09-18T08:07:40.610 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:07:40.610 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:07:40.610 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "done", 2024-09-18T08:07:40.610 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:05:10.064436+0000", 2024-09-18T08:07:40.610 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 4.8449999999999999e-05 2024-09-18T08:07:40.610 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-09-18T08:07:40.611 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-09-18T08:07:40.611 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-09-18T08:07:40.611 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:07:40.611 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:07:40.611 INFO:teuthology.orchestra.run.smithi007.stdout: "description": "osd_op(client.14805.0:95663 7.e 7:70abb2e4:::benchmark_data_smithi007_73155_object95662:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e108)", 2024-09-18T08:07:40.611 INFO:teuthology.orchestra.run.smithi007.stdout: "initiated_at": "2024-09-18T08:05:11.692241+0000", 2024-09-18T08:07:40.611 INFO:teuthology.orchestra.run.smithi007.stdout: "age": 148.88747644200001, 2024-09-18T08:07:40.611 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.086321836999999998, 2024-09-18T08:07:40.611 INFO:teuthology.orchestra.run.smithi007.stdout: "continuous": false, 2024-09-18T08:07:40.611 INFO:teuthology.orchestra.run.smithi007.stdout: "type_data": { 2024-09-18T08:07:40.611 INFO:teuthology.orchestra.run.smithi007.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T08:07:40.611 INFO:teuthology.orchestra.run.smithi007.stdout: "client_info": { 2024-09-18T08:07:40.611 INFO:teuthology.orchestra.run.smithi007.stdout: "client": "client.14805", 2024-09-18T08:07:40.611 INFO:teuthology.orchestra.run.smithi007.stdout: "client_addr": "172.21.15.7:0/640320436", 2024-09-18T08:07:40.612 INFO:teuthology.orchestra.run.smithi007.stdout: "tid": 95663 2024-09-18T08:07:40.612 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:07:40.612 INFO:teuthology.orchestra.run.smithi007.stdout: "events": [ 2024-09-18T08:07:40.612 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:07:40.612 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "initiated", 2024-09-18T08:07:40.612 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:05:11.692241+0000", 2024-09-18T08:07:40.612 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-09-18T08:07:40.612 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:07:40.612 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:07:40.612 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "header_read", 2024-09-18T08:07:40.612 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:05:11.692241+0000", 2024-09-18T08:07:40.612 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-09-18T08:07:40.612 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:07:40.613 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:07:40.613 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "throttled", 2024-09-18T08:07:40.613 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:05:11.692243+0000", 2024-09-18T08:07:40.613 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.3340000000000001e-06 2024-09-18T08:07:40.613 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:07:40.613 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:07:40.613 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "all_read", 2024-09-18T08:07:40.613 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:05:11.692262+0000", 2024-09-18T08:07:40.613 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.9420999999999999e-05 2024-09-18T08:07:40.613 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:07:40.613 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:07:40.613 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "dispatched", 2024-09-18T08:07:40.613 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:05:11.692273+0000", 2024-09-18T08:07:40.613 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.1049e-05 2024-09-18T08:07:40.614 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:07:40.614 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:07:40.614 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "queued_for_pg", 2024-09-18T08:07:40.614 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:05:11.692282+0000", 2024-09-18T08:07:40.614 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 8.3299999999999999e-06 2024-09-18T08:07:40.614 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:07:40.614 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:07:40.614 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-09-18T08:07:40.614 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:05:11.706111+0000", 2024-09-18T08:07:40.614 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.013829077 2024-09-18T08:07:40.614 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:07:40.614 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:07:40.614 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "started", 2024-09-18T08:07:40.614 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:05:11.706251+0000", 2024-09-18T08:07:40.615 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.00014055400000000001 2024-09-18T08:07:40.615 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:07:40.615 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:07:40.615 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "waiting for subops from 0,5", 2024-09-18T08:07:40.615 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:05:11.706408+0000", 2024-09-18T08:07:40.615 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.00015712199999999999 2024-09-18T08:07:40.615 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:07:40.615 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:07:40.615 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "op_commit", 2024-09-18T08:07:40.615 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:05:11.711917+0000", 2024-09-18T08:07:40.615 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.0055082100000000004 2024-09-18T08:07:40.615 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:07:40.616 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:07:40.616 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:07:40.616 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:05:11.714609+0000", 2024-09-18T08:07:40.616 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.0026926939999999998 2024-09-18T08:07:40.616 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:07:40.616 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:07:40.616 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:07:40.616 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:05:11.778458+0000", 2024-09-18T08:07:40.616 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.063848461999999995 2024-09-18T08:07:40.616 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:07:40.616 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:07:40.616 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "commit_sent", 2024-09-18T08:07:40.617 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:05:11.778509+0000", 2024-09-18T08:07:40.617 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 5.1638999999999998e-05 2024-09-18T08:07:40.617 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:07:40.617 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:07:40.617 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "done", 2024-09-18T08:07:40.617 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:05:11.778563+0000", 2024-09-18T08:07:40.617 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 5.3944999999999999e-05 2024-09-18T08:07:40.617 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-09-18T08:07:40.617 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-09-18T08:07:40.617 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-09-18T08:07:40.617 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:07:40.617 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:07:40.617 INFO:teuthology.orchestra.run.smithi007.stdout: "description": "osd_op(client.14805.0:96235 7.e 7:70c09d0d:::benchmark_data_smithi007_73155_object96234:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e109)", 2024-09-18T08:07:40.618 INFO:teuthology.orchestra.run.smithi007.stdout: "initiated_at": "2024-09-18T08:05:12.494260+0000", 2024-09-18T08:07:40.618 INFO:teuthology.orchestra.run.smithi007.stdout: "age": 148.08545819899999, 2024-09-18T08:07:40.618 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.089927492999999997, 2024-09-18T08:07:40.618 INFO:teuthology.orchestra.run.smithi007.stdout: "continuous": false, 2024-09-18T08:07:40.618 INFO:teuthology.orchestra.run.smithi007.stdout: "type_data": { 2024-09-18T08:07:40.618 INFO:teuthology.orchestra.run.smithi007.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T08:07:40.618 INFO:teuthology.orchestra.run.smithi007.stdout: "client_info": { 2024-09-18T08:07:40.618 INFO:teuthology.orchestra.run.smithi007.stdout: "client": "client.14805", 2024-09-18T08:07:40.618 INFO:teuthology.orchestra.run.smithi007.stdout: "client_addr": "172.21.15.7:0/640320436", 2024-09-18T08:07:40.618 INFO:teuthology.orchestra.run.smithi007.stdout: "tid": 96235 2024-09-18T08:07:40.618 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:07:40.618 INFO:teuthology.orchestra.run.smithi007.stdout: "events": [ 2024-09-18T08:07:40.619 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:07:40.619 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "initiated", 2024-09-18T08:07:40.619 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:05:12.494260+0000", 2024-09-18T08:07:40.619 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-09-18T08:07:40.619 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:07:40.619 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:07:40.620 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "header_read", 2024-09-18T08:07:40.620 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:05:12.494260+0000", 2024-09-18T08:07:40.620 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-09-18T08:07:40.620 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:07:40.621 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:07:40.621 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "throttled", 2024-09-18T08:07:40.621 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:05:12.494351+0000", 2024-09-18T08:07:40.621 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 9.0839000000000002e-05 2024-09-18T08:07:40.621 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:07:40.621 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:07:40.622 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "all_read", 2024-09-18T08:07:40.622 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:05:12.494375+0000", 2024-09-18T08:07:40.622 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 2.4921000000000001e-05 2024-09-18T08:07:40.622 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:07:40.622 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:07:40.622 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "dispatched", 2024-09-18T08:07:40.623 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:05:12.494388+0000", 2024-09-18T08:07:40.623 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.2415000000000001e-05 2024-09-18T08:07:40.623 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:07:40.623 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:07:40.623 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "queued_for_pg", 2024-09-18T08:07:40.623 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:05:12.494396+0000", 2024-09-18T08:07:40.623 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 8.0770000000000008e-06 2024-09-18T08:07:40.623 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:07:40.624 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:07:40.624 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-09-18T08:07:40.624 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:05:12.500910+0000", 2024-09-18T08:07:40.624 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.0065145089999999999 2024-09-18T08:07:40.624 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:07:40.625 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:07:40.626 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "started", 2024-09-18T08:07:40.626 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:05:12.501024+0000", 2024-09-18T08:07:40.626 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.000113992 2024-09-18T08:07:40.626 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:07:40.626 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:07:40.626 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "waiting for subops from 0,5", 2024-09-18T08:07:40.626 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:05:12.501165+0000", 2024-09-18T08:07:40.626 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.000140347 2024-09-18T08:07:40.626 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:07:40.626 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:07:40.626 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:07:40.627 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:05:12.531672+0000", 2024-09-18T08:07:40.627 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.030506945000000001 2024-09-18T08:07:40.627 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:07:40.627 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:07:40.627 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "op_commit", 2024-09-18T08:07:40.627 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:05:12.531683+0000", 2024-09-18T08:07:40.627 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.1092e-05 2024-09-18T08:07:40.627 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:07:40.627 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:07:40.627 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:07:40.627 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:05:12.583879+0000", 2024-09-18T08:07:40.628 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.052195966000000003 2024-09-18T08:07:40.628 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:07:40.628 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:07:40.628 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "commit_sent", 2024-09-18T08:07:40.628 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:05:12.584048+0000", 2024-09-18T08:07:40.629 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.00016902 2024-09-18T08:07:40.629 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:07:40.629 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:07:40.629 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "done", 2024-09-18T08:07:40.629 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:05:12.584187+0000", 2024-09-18T08:07:40.630 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.00013936999999999999 2024-09-18T08:07:40.630 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-09-18T08:07:40.630 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-09-18T08:07:40.630 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-09-18T08:07:40.630 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:07:40.630 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:07:40.630 INFO:teuthology.orchestra.run.smithi007.stdout: "description": "osd_op(client.14805.0:96694 7.e 7:77dc6698:::benchmark_data_smithi007_73155_object96693:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e109)", 2024-09-18T08:07:40.631 INFO:teuthology.orchestra.run.smithi007.stdout: "initiated_at": "2024-09-18T08:05:13.227222+0000", 2024-09-18T08:07:40.631 INFO:teuthology.orchestra.run.smithi007.stdout: "age": 147.35249609900001, 2024-09-18T08:07:40.631 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.092647683999999994, 2024-09-18T08:07:40.631 INFO:teuthology.orchestra.run.smithi007.stdout: "continuous": false, 2024-09-18T08:07:40.631 INFO:teuthology.orchestra.run.smithi007.stdout: "type_data": { 2024-09-18T08:07:40.631 INFO:teuthology.orchestra.run.smithi007.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T08:07:40.631 INFO:teuthology.orchestra.run.smithi007.stdout: "client_info": { 2024-09-18T08:07:40.632 INFO:teuthology.orchestra.run.smithi007.stdout: "client": "client.14805", 2024-09-18T08:07:40.632 INFO:teuthology.orchestra.run.smithi007.stdout: "client_addr": "172.21.15.7:0/640320436", 2024-09-18T08:07:40.632 INFO:teuthology.orchestra.run.smithi007.stdout: "tid": 96694 2024-09-18T08:07:40.632 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:07:40.632 INFO:teuthology.orchestra.run.smithi007.stdout: "events": [ 2024-09-18T08:07:40.632 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:07:40.632 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "initiated", 2024-09-18T08:07:40.633 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:05:13.227222+0000", 2024-09-18T08:07:40.633 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-09-18T08:07:40.633 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:07:40.633 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:07:40.633 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "header_read", 2024-09-18T08:07:40.634 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:05:13.227222+0000", 2024-09-18T08:07:40.634 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-09-18T08:07:40.634 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:07:40.634 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:07:40.635 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "throttled", 2024-09-18T08:07:40.635 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:05:13.227224+0000", 2024-09-18T08:07:40.635 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 2.243e-06 2024-09-18T08:07:40.635 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:07:40.635 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:07:40.636 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "all_read", 2024-09-18T08:07:40.636 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:05:13.227249+0000", 2024-09-18T08:07:40.636 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 2.4820999999999998e-05 2024-09-18T08:07:40.636 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:07:40.636 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:07:40.636 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "dispatched", 2024-09-18T08:07:40.636 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:05:13.227262+0000", 2024-09-18T08:07:40.636 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.2918999999999999e-05 2024-09-18T08:07:40.636 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:07:40.636 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:07:40.636 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "queued_for_pg", 2024-09-18T08:07:40.637 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:05:13.227275+0000", 2024-09-18T08:07:40.637 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.3607e-05 2024-09-18T08:07:40.637 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:07:40.637 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:07:40.637 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-09-18T08:07:40.637 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:05:13.239760+0000", 2024-09-18T08:07:40.637 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.012484419 2024-09-18T08:07:40.637 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:07:40.640 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:07:40.641 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "started", 2024-09-18T08:07:40.641 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:05:13.239892+0000", 2024-09-18T08:07:40.641 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.00013202900000000001 2024-09-18T08:07:40.641 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:07:40.641 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:07:40.641 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "waiting for subops from 0,5", 2024-09-18T08:07:40.641 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:05:13.240049+0000", 2024-09-18T08:07:40.641 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.00015725599999999999 2024-09-18T08:07:40.641 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:07:40.641 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:07:40.641 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "op_commit", 2024-09-18T08:07:40.642 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:05:13.247290+0000", 2024-09-18T08:07:40.642 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.00724072 2024-09-18T08:07:40.642 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:07:40.642 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:07:40.642 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:07:40.642 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:05:13.247559+0000", 2024-09-18T08:07:40.642 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.00026900699999999998 2024-09-18T08:07:40.642 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:07:40.642 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:07:40.642 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:07:40.642 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:05:13.319747+0000", 2024-09-18T08:07:40.642 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.072187789000000002 2024-09-18T08:07:40.643 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:07:40.643 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:07:40.643 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "commit_sent", 2024-09-18T08:07:40.643 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:05:13.319810+0000", 2024-09-18T08:07:40.643 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 6.3139000000000007e-05 2024-09-18T08:07:40.643 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:07:40.643 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:07:40.644 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "done", 2024-09-18T08:07:40.644 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:05:13.319869+0000", 2024-09-18T08:07:40.644 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 5.9735000000000003e-05 2024-09-18T08:07:40.644 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-09-18T08:07:40.644 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-09-18T08:07:40.644 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-09-18T08:07:40.644 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:07:40.644 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:07:40.644 INFO:teuthology.orchestra.run.smithi007.stdout: "description": "osd_op(client.14805.0:96719 7.e 7:7f123a7e:::benchmark_data_smithi007_73155_object96718:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e109)", 2024-09-18T08:07:40.644 INFO:teuthology.orchestra.run.smithi007.stdout: "initiated_at": "2024-09-18T08:05:13.293471+0000", 2024-09-18T08:07:40.644 INFO:teuthology.orchestra.run.smithi007.stdout: "age": 147.286247312, 2024-09-18T08:07:40.645 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.091559383999999994, 2024-09-18T08:07:40.645 INFO:teuthology.orchestra.run.smithi007.stdout: "continuous": false, 2024-09-18T08:07:40.645 INFO:teuthology.orchestra.run.smithi007.stdout: "type_data": { 2024-09-18T08:07:40.645 INFO:teuthology.orchestra.run.smithi007.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T08:07:40.645 INFO:teuthology.orchestra.run.smithi007.stdout: "client_info": { 2024-09-18T08:07:40.645 INFO:teuthology.orchestra.run.smithi007.stdout: "client": "client.14805", 2024-09-18T08:07:40.645 INFO:teuthology.orchestra.run.smithi007.stdout: "client_addr": "172.21.15.7:0/640320436", 2024-09-18T08:07:40.645 INFO:teuthology.orchestra.run.smithi007.stdout: "tid": 96719 2024-09-18T08:07:40.645 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:07:40.645 INFO:teuthology.orchestra.run.smithi007.stdout: "events": [ 2024-09-18T08:07:40.645 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:07:40.645 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "initiated", 2024-09-18T08:07:40.646 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:05:13.293471+0000", 2024-09-18T08:07:40.646 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-09-18T08:07:40.646 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:07:40.646 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:07:40.646 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "header_read", 2024-09-18T08:07:40.646 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:05:13.293471+0000", 2024-09-18T08:07:40.646 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-09-18T08:07:40.646 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:07:40.646 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:07:40.646 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "throttled", 2024-09-18T08:07:40.646 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:05:13.293472+0000", 2024-09-18T08:07:40.646 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.482e-06 2024-09-18T08:07:40.647 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:07:40.647 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:07:40.647 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "all_read", 2024-09-18T08:07:40.647 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:05:13.293496+0000", 2024-09-18T08:07:40.647 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 2.3708000000000001e-05 2024-09-18T08:07:40.647 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:07:40.647 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:07:40.647 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "dispatched", 2024-09-18T08:07:40.647 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:05:13.293506+0000", 2024-09-18T08:07:40.647 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.0706e-05 2024-09-18T08:07:40.647 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:07:40.648 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:07:40.648 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1 2024-09-18T08:07:40.650 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "queued_for_pg", 2024-09-18T08:07:40.650 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:05:13.293514+0000", 2024-09-18T08:07:40.650 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 7.4449999999999999e-06 2024-09-18T08:07:40.650 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:07:40.651 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:07:40.651 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-09-18T08:07:40.651 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:05:13.320155+0000", 2024-09-18T08:07:40.651 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.026641436000000001 2024-09-18T08:07:40.651 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:07:40.651 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:07:40.651 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "started", 2024-09-18T08:07:40.652 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:05:13.320303+0000", 2024-09-18T08:07:40.652 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.00014768800000000001 2024-09-18T08:07:40.652 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:07:40.652 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:07:40.652 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "waiting for subops from 0,5", 2024-09-18T08:07:40.652 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:05:13.320454+0000", 2024-09-18T08:07:40.652 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.000151328 2024-09-18T08:07:40.652 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:07:40.652 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:07:40.652 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:07:40.652 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:05:13.326669+0000", 2024-09-18T08:07:40.652 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.0062146090000000003 2024-09-18T08:07:40.653 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:07:40.653 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:07:40.653 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "op_commit", 2024-09-18T08:07:40.653 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:05:13.326681+0000", 2024-09-18T08:07:40.653 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.2160999999999999e-05 2024-09-18T08:07:40.653 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:07:40.653 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:07:40.653 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:07:40.653 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:05:13.384909+0000", 2024-09-18T08:07:40.653 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.058227999000000003 2024-09-18T08:07:40.653 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:07:40.653 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:07:40.654 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "commit_sent", 2024-09-18T08:07:40.654 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:05:13.384968+0000", 2024-09-18T08:07:40.654 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 5.8479e-05 2024-09-18T08:07:40.654 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:07:40.654 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:07:40.654 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "done", 2024-09-18T08:07:40.654 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:05:13.385030+0000", 2024-09-18T08:07:40.654 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 6.2342999999999996e-05 2024-09-18T08:07:40.654 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-09-18T08:07:40.654 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-09-18T08:07:40.654 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-09-18T08:07:40.654 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:07:40.654 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:07:40.655 INFO:teuthology.orchestra.run.smithi007.stdout: "description": "osd_op(client.14805.0:141891 7.e 7:7d0cb9e1:::benchmark_data_smithi007_73155_object23600:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e127)", 2024-09-18T08:07:40.655 INFO:teuthology.orchestra.run.smithi007.stdout: "initiated_at": "2024-09-18T08:05:59.091427+0000", 2024-09-18T08:07:40.655 INFO:teuthology.orchestra.run.smithi007.stdout: "age": 101.488290668, 2024-09-18T08:07:40.655 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.090109581999999994, 2024-09-18T08:07:40.655 INFO:teuthology.orchestra.run.smithi007.stdout: "continuous": false, 2024-09-18T08:07:40.655 INFO:teuthology.orchestra.run.smithi007.stdout: "type_data": { 2024-09-18T08:07:40.655 INFO:teuthology.orchestra.run.smithi007.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T08:07:40.655 INFO:teuthology.orchestra.run.smithi007.stdout: "client_info": { 2024-09-18T08:07:40.655 INFO:teuthology.orchestra.run.smithi007.stdout: "client": "client.14805", 2024-09-18T08:07:40.656 INFO:teuthology.orchestra.run.smithi007.stdout: "client_addr": "172.21.15.7:0/640320436", 2024-09-18T08:07:40.656 INFO:teuthology.orchestra.run.smithi007.stdout: "tid": 141891 2024-09-18T08:07:40.656 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:07:40.656 INFO:teuthology.orchestra.run.smithi007.stdout: "events": [ 2024-09-18T08:07:40.656 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:07:40.656 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "initiated", 2024-09-18T08:07:40.656 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:05:59.091427+0000", 2024-09-18T08:07:40.656 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-09-18T08:07:40.657 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:07:40.657 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:07:40.657 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "header_read", 2024-09-18T08:07:40.657 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:05:59.091427+0000", 2024-09-18T08:07:40.657 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-09-18T08:07:40.657 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:07:40.657 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:07:40.657 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "throttled", 2024-09-18T08:07:40.657 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:05:59.091429+0000", 2024-09-18T08:07:40.657 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.719e-06 2024-09-18T08:07:40.657 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:07:40.658 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:07:40.658 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "all_read", 2024-09-18T08:07:40.658 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:05:59.091435+0000", 2024-09-18T08:07:40.658 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 5.8850000000000004e-06 2024-09-18T08:07:40.658 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:07:40.658 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:07:40.658 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "dispatched", 2024-09-18T08:07:40.658 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:05:59.091447+0000", 2024-09-18T08:07:40.658 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.2007e-05 2024-09-18T08:07:40.658 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:07:40.658 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:07:40.658 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "queued_for_pg", 2024-09-18T08:07:40.658 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:05:59.091455+0000", 2024-09-18T08:07:40.659 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 8.5490000000000006e-06 2024-09-18T08:07:40.659 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:07:40.659 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:07:40.659 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-09-18T08:07:40.659 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:05:59.091549+0000", 2024-09-18T08:07:40.659 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 9.3411000000000004e-05 2024-09-18T08:07:40.659 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:07:40.659 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:07:40.659 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "started", 2024-09-18T08:07:40.659 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:05:59.091625+0000", 2024-09-18T08:07:40.659 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 7.5745000000000001e-05 2024-09-18T08:07:40.659 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:07:40.659 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:07:40.659 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "waiting for subops from 0,5", 2024-09-18T08:07:40.660 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:05:59.091687+0000", 2024-09-18T08:07:40.660 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 6.2831999999999997e-05 2024-09-18T08:07:40.660 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:07:40.660 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:07:40.660 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "op_commit", 2024-09-18T08:07:40.660 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:05:59.094073+0000", 2024-09-18T08:07:40.660 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.0023852959999999999 2024-09-18T08:07:40.660 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:07:40.660 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:07:40.660 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:07:40.660 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:05:59.094439+0000", 2024-09-18T08:07:40.660 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.00036632800000000002 2024-09-18T08:07:40.660 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:07:40.660 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:07:40.661 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:07:40.661 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:05:59.181431+0000", 2024-09-18T08:07:40.661 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.086991857000000006 2024-09-18T08:07:40.661 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:07:40.661 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:07:40.661 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "commit_sent", 2024-09-18T08:07:40.661 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:05:59.181489+0000", 2024-09-18T08:07:40.661 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 5.8477999999999998e-05 2024-09-18T08:07:40.661 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:07:40.661 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:07:40.661 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "done", 2024-09-18T08:07:40.661 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:05:59.181537+0000", 2024-09-18T08:07:40.662 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 4.7475000000000002e-05 2024-09-18T08:07:40.662 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-09-18T08:07:40.662 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-09-18T08:07:40.662 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-09-18T08:07:40.662 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:07:40.662 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:07:40.662 INFO:teuthology.orchestra.run.smithi007.stdout: "description": "osd_op(client.14805.0:147830 7.e 7:7137fe31:::benchmark_data_smithi007_73155_object29539:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e127)", 2024-09-18T08:07:40.662 INFO:teuthology.orchestra.run.smithi007.stdout: "initiated_at": "2024-09-18T08:06:04.696707+0000", 2024-09-18T08:07:40.662 INFO:teuthology.orchestra.run.smithi007.stdout: "age": 95.883010744000003, 2024-09-18T08:07:40.662 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.093481047999999997, 2024-09-18T08:07:40.663 INFO:teuthology.orchestra.run.smithi007.stdout: "continuous": false, 2024-09-18T08:07:40.663 INFO:teuthology.orchestra.run.smithi007.stdout: "type_data": { 2024-09-18T08:07:40.663 INFO:teuthology.orchestra.run.smithi007.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T08:07:40.663 INFO:teuthology.orchestra.run.smithi007.stdout: "client_info": { 2024-09-18T08:07:40.663 INFO:teuthology.orchestra.run.smithi007.stdout: "client": "client.14805", 2024-09-18T08:07:40.663 INFO:teuthology.orchestra.run.smithi007.stdout: "client_addr": "172.21.15.7:0/640320436", 2024-09-18T08:07:40.663 INFO:teuthology.orchestra.run.smithi007.stdout: "tid": 147830 2024-09-18T08:07:40.663 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:07:40.663 INFO:teuthology.orchestra.run.smithi007.stdout: "events": [ 2024-09-18T08:07:40.663 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:07:40.663 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "initiated", 2024-09-18T08:07:40.663 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:06:04.696707+0000", 2024-09-18T08:07:40.664 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-09-18T08:07:40.664 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:07:40.664 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:07:40.664 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "header_read", 2024-09-18T08:07:40.664 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:06:04.696707+0000", 2024-09-18T08:07:40.664 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-09-18T08:07:40.664 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:07:40.664 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:07:40.664 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "throttled", 2024-09-18T08:07:40.664 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:06:04.696709+0000", 2024-09-18T08:07:40.664 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.4610000000000001e-06 2024-09-18T08:07:40.665 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:07:40.665 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:07:40.665 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "all_read", 2024-09-18T08:07:40.665 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:06:04.696714+0000", 2024-09-18T08:07:40.665 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 5.4929999999999998e-06 2024-09-18T08:07:40.665 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:07:40.665 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:07:40.665 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "dispatched", 2024-09-18T08:07:40.665 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:06:04.696726+0000", 2024-09-18T08:07:40.665 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.1405e-05 2024-09-18T08:07:40.665 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:07:40.665 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:07:40.665 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "queued_for_pg", 2024-09-18T08:07:40.666 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:06:04.696733+0000", 2024-09-18T08:07:40.666 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 7.5440000000000001e-06 2024-09-18T08:07:40.666 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:07:40.666 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:07:40.666 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-09-18T08:07:40.666 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:06:04.696834+0000", 2024-09-18T08:07:40.666 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.00010042799999999999 2024-09-18T08:07:40.666 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:07:40.666 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:07:40.666 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "started", 2024-09-18T08:07:40.666 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:06:04.696964+0000", 2024-09-18T08:07:40.666 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.00013050599999999999 2024-09-18T08:07:40.667 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:07:40.667 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:07:40.667 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "waiting for subops from 0,5", 2024-09-18T08:07:40.667 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:06:04.697079+0000", 2024-09-18T08:07:40.667 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.000114769 2024-09-18T08:07:40.667 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:07:40.667 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:07:40.667 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:07:40.667 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:06:04.703144+0000", 2024-09-18T08:07:40.667 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.0060647729999999999 2024-09-18T08:07:40.667 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:07:40.667 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:07:40.668 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "op_commit", 2024-09-18T08:07:40.668 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:06:04.703157+0000", 2024-09-18T08:07:40.668 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.3434e-05 2024-09-18T08:07:40.668 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:07:40.668 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:07:40.668 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:07:40.668 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:06:04.789746+0000", 2024-09-18T08:07:40.668 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.086589049000000001 2024-09-18T08:07:40.668 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:07:40.668 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:07:40.668 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "commit_sent", 2024-09-18T08:07:40.668 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:06:04.789946+0000", 2024-09-18T08:07:40.668 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.000199996 2024-09-18T08:07:40.669 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:07:40.669 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:07:40.669 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "done", 2024-09-18T08:07:40.669 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:06:04.790188+0000", 2024-09-18T08:07:40.669 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.00024219000000000001 2024-09-18T08:07:40.669 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-09-18T08:07:40.669 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-09-18T08:07:40.669 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-09-18T08:07:40.669 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:07:40.669 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:07:40.669 INFO:teuthology.orchestra.run.smithi007.stdout: "description": "osd_op(client.14805.0:147846 7.e 7:77e37675:::benchmark_data_smithi007_73155_object29555:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e127)", 2024-09-18T08:07:40.669 INFO:teuthology.orchestra.run.smithi007.stdout: "initiated_at": "2024-09-18T08:06:04.708478+0000", 2024-09-18T08:07:40.670 INFO:teuthology.orchestra.run.smithi007.stdout: "age": 95.871239559000003, 2024-09-18T08:07:40.670 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.089318937000000001, 2024-09-18T08:07:40.670 INFO:teuthology.orchestra.run.smithi007.stdout: "continuous": false, 2024-09-18T08:07:40.670 INFO:teuthology.orchestra.run.smithi007.stdout: "type_data": { 2024-09-18T08:07:40.670 INFO:teuthology.orchestra.run.smithi007.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T08:07:40.670 INFO:teuthology.orchestra.run.smithi007.stdout: "client_info": { 2024-09-18T08:07:40.670 INFO:teuthology.orchestra.run.smithi007.stdout: "client": "client.14805", 2024-09-18T08:07:40.670 INFO:teuthology.orchestra.run.smithi007.stdout: "client_addr": "172.21.15.7:0/640320436", 2024-09-18T08:07:40.670 INFO:teuthology.orchestra.run.smithi007.stdout: "tid": 147846 2024-09-18T08:07:40.670 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:07:40.670 INFO:teuthology.orchestra.run.smithi007.stdout: "events": [ 2024-09-18T08:07:40.670 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:07:40.671 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "initiated", 2024-09-18T08:07:40.671 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:06:04.708478+0000", 2024-09-18T08:07:40.671 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-09-18T08:07:40.671 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:07:40.671 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:07:40.671 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "header_read", 2024-09-18T08:07:40.671 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:06:04.708478+0000", 2024-09-18T08:07:40.671 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-09-18T08:07:40.671 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:07:40.671 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:07:40.671 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "throttled", 2024-09-18T08:07:40.671 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:06:04.708480+0000", 2024-09-18T08:07:40.671 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.3799999999999999e-06 2024-09-18T08:07:40.671 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:07:40.672 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:07:40.672 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "all_read", 2024-09-18T08:07:40.672 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:06:04.708485+0000", 2024-09-18T08:07:40.672 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 5.5140000000000001e-06 2024-09-18T08:07:40.672 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:07:40.672 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:07:40.672 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "dispatched", 2024-09-18T08:07:40.672 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:06:04.708496+0000", 2024-09-18T08:07:40.672 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.0468e-05 2024-09-18T08:07:40.672 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:07:40.672 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:07:40.673 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "queued_for_pg", 2024-09-18T08:07:40.673 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:06:04.708503+0000", 2024-09-18T08:07:40.673 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 6.9709999999999998e-06 2024-09-18T08:07:40.673 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:07:40.673 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:07:40.674 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-09-18T08:07:40.674 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:06:04.708559+0000", 2024-09-18T08:07:40.674 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 5.5967999999999997e-05 2024-09-18T08:07:40.674 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:07:40.674 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:07:40.674 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "started", 2024-09-18T08:07:40.674 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:06:04.708664+0000", 2024-09-18T08:07:40.674 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.000104956 2024-09-18T08:07:40.674 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:07:40.675 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:07:40.675 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "waiting for subops from 0,5", 2024-09-18T08:07:40.675 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:06:04.708767+0000", 2024-09-18T08:07:40.675 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.000102889 2024-09-18T08:07:40.675 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:07:40.675 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:07:40.675 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:07:40.675 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:06:04.781177+0000", 2024-09-18T08:07:40.676 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.072410522000000005 2024-09-18T08:07:40.676 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:07:40.676 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:07:40.676 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "op_commit", 2024-09-18T08:07:40.676 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:06:04.782476+0000", 2024-09-18T08:07:40.676 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.0012993150000000001 2024-09-18T08:07:40.676 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:07:40.676 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:07:40.676 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:07:40.676 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:06:04.797423+0000", 2024-09-18T08:07:40.676 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.014946213999999999 2024-09-18T08:07:40.677 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:07:40.677 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:07:40.677 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "commit_sent", 2024-09-18T08:07:40.677 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:06:04.797637+0000", 2024-09-18T08:07:40.677 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.000214353 2024-09-18T08:07:40.677 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:07:40.677 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:07:40.677 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "done", 2024-09-18T08:07:40.677 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:06:04.797797+0000", 2024-09-18T08:07:40.677 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.00016038700000000001 2024-09-18T08:07:40.677 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-09-18T08:07:40.677 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-09-18T08:07:40.677 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-09-18T08:07:40.678 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:07:40.678 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:07:40.678 INFO:teuthology.orchestra.run.smithi007.stdout: "description": "osd_op(client.14805.0:147849 7.e 7:7e6f39e7:::benchmark_data_smithi007_73155_object29558:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e127)", 2024-09-18T08:07:40.678 INFO:teuthology.orchestra.run.smithi007.stdout: "initiated_at": "2024-09-18T08:06:04.708908+0000", 2024-09-18T08:07:40.678 INFO:teuthology.orchestra.run.smithi007.stdout: "age": 95.870810274999997, 2024-09-18T08:07:40.678 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.089203497000000007, 2024-09-18T08:07:40.678 INFO:teuthology.orchestra.run.smithi007.stdout: "continuous": false, 2024-09-18T08:07:40.678 INFO:teuthology.orchestra.run.smithi007.stdout: "type_data": { 2024-09-18T08:07:40.678 INFO:teuthology.orchestra.run.smithi007.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T08:07:40.678 INFO:teuthology.orchestra.run.smithi007.stdout: "client_info": { 2024-09-18T08:07:40.678 INFO:teuthology.orchestra.run.smithi007.stdout: "client": "client.14805", 2024-09-18T08:07:40.678 INFO:teuthology.orchestra.run.smithi007.stdout: "client_addr": "172.21.15.7:0/640320436", 2024-09-18T08:07:40.678 INFO:teuthology.orchestra.run.smithi007.stdout: "tid": 147849 2024-09-18T08:07:40.678 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:07:40.679 INFO:teuthology.orchestra.run.smithi007.stdout: "events": [ 2024-09-18T08:07:40.679 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:07:40.679 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "initiated", 2024-09-18T08:07:40.679 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:06:04.708908+0000", 2024-09-18T08:07:40.679 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-09-18T08:07:40.679 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:07:40.679 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:07:40.679 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "header_read", 2024-09-18T08:07:40.679 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:06:04.708908+0000", 2024-09-18T08:07:40.679 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-09-18T08:07:40.679 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:07:40.679 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:07:40.679 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "throttled", 2024-09-18T08:07:40.680 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:06:04.708908+0000", 2024-09-18T08:07:40.680 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 7.6199999999999997e-07 2024-09-18T08:07:40.680 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:07:40.680 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:07:40.680 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "all_read", 2024-09-18T08:07:40.680 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:06:04.708913+0000", 2024-09-18T08:07:40.680 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 4.1049999999999997e-06 2024-09-18T08:07:40.680 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:07:40.680 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:07:40.680 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "dispatched", 2024-09-18T08:07:40.681 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:06:04.708922+0000", 2024-09-18T08:07:40.681 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 9.2930000000000006e-06 2024-09-18T08:07:40.681 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:07:40.681 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:07:40.681 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "queued_for_pg", 2024-09-18T08:07:40.681 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:06:04.708928+0000", 2024-09-18T08:07:40.681 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 6.4919999999999998e-06 2024-09-18T08:07:40.681 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:07:40.681 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:07:40.681 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-09-18T08:07:40.681 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:06:04.712299+0000", 2024-09-18T08:07:40.681 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.0033709510000000001 2024-09-18T08:07:40.681 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:07:40.681 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:07:40.682 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "started", 2024-09-18T08:07:40.682 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:06:04.712407+0000", 2024-09-18T08:07:40.682 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.000108229 2024-09-18T08:07:40.682 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:07:40.682 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:07:40.682 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "waiting for subops from 0,5", 2024-09-18T08:07:40.682 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:06:04.712519+0000", 2024-09-18T08:07:40.682 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.00011200399999999999 2024-09-18T08:07:40.682 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:07:40.682 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:07:40.682 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:07:40.682 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:06:04.782460+0000", 2024-09-18T08:07:40.682 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.069940536999999997 2024-09-18T08:07:40.682 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:07:40.683 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:07:40.683 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "op_commit", 2024-09-18T08:07:40.683 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:06:04.782500+0000", 2024-09-18T08:07:40.683 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 4.0463000000000002e-05 2024-09-18T08:07:40.683 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:07:40.683 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:07:40.683 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:07:40.683 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:06:04.797935+0000", 2024-09-18T08:07:40.683 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.015434188 2024-09-18T08:07:40.683 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:07:40.683 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:07:40.683 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "commit_sent", 2024-09-18T08:07:40.683 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:06:04.798073+0000", 2024-09-18T08:07:40.684 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.00013849000000000001 2024-09-18T08:07:40.684 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:07:40.684 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:07:40.684 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "done", 2024-09-18T08:07:40.684 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:06:04.798111+0000", 2024-09-18T08:07:40.684 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 3.7982999999999999e-05 2024-09-18T08:07:40.684 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-09-18T08:07:40.684 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-09-18T08:07:40.684 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-09-18T08:07:40.684 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:07:40.684 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:07:40.684 INFO:teuthology.orchestra.run.smithi007.stdout: "description": "osd_op(client.14805.0:148158 7.e 7:7447257d:::benchmark_data_smithi007_73155_object29867:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e127)", 2024-09-18T08:07:40.684 INFO:teuthology.orchestra.run.smithi007.stdout: "initiated_at": "2024-09-18T08:06:05.033287+0000", 2024-09-18T08:07:40.684 INFO:teuthology.orchestra.run.smithi007.stdout: "age": 95.546431295000005, 2024-09-18T08:07:40.685 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.085660061999999995, 2024-09-18T08:07:40.685 INFO:teuthology.orchestra.run.smithi007.stdout: "continuous": false, 2024-09-18T08:07:40.685 INFO:teuthology.orchestra.run.smithi007.stdout: "type_data": { 2024-09-18T08:07:40.685 INFO:teuthology.orchestra.run.smithi007.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T08:07:40.685 INFO:teuthology.orchestra.run.smithi007.stdout: "client_info": { 2024-09-18T08:07:40.685 INFO:teuthology.orchestra.run.smithi007.stdout: "client": "client.14805", 2024-09-18T08:07:40.685 INFO:teuthology.orchestra.run.smithi007.stdout: "client_addr": "172.21.15.7:0/640320436", 2024-09-18T08:07:40.685 INFO:teuthology.orchestra.run.smithi007.stdout: "tid": 148158 2024-09-18T08:07:40.685 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:07:40.685 INFO:teuthology.orchestra.run.smithi007.stdout: "events": [ 2024-09-18T08:07:40.685 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:07:40.685 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "initiated", 2024-09-18T08:07:40.685 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:06:05.033287+0000", 2024-09-18T08:07:40.686 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-09-18T08:07:40.686 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:07:40.686 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:07:40.686 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "header_read", 2024-09-18T08:07:40.686 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:06:05.033287+0000", 2024-09-18T08:07:40.686 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-09-18T08:07:40.686 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:07:40.686 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:07:40.686 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "throttled", 2024-09-18T08:07:40.686 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:06:05.033288+0000", 2024-09-18T08:07:40.686 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 9.1500000000000003e-07 2024-09-18T08:07:40.686 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:07:40.687 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:07:40.687 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "all_read", 2024-09-18T08:07:40.687 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:06:05.033292+0000", 2024-09-18T08:07:40.687 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 4.6129999999999997e-06 2024-09-18T08:07:40.687 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:07:40.687 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:07:40.687 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "dispatched", 2024-09-18T08:07:40.687 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:06:05.033302+0000", 2024-09-18T08:07:40.687 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 9.4630000000000007e-06 2024-09-18T08:07:40.687 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:07:40.687 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:07:40.687 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "queued_for_pg", 2024-09-18T08:07:40.687 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:06:05.033309+0000", 2024-09-18T08:07:40.688 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 7.6550000000000004e-06 2024-09-18T08:07:40.688 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:07:40.688 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:07:40.688 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-09-18T08:07:40.688 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:06:05.036668+0000", 2024-09-18T08:07:40.688 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.0033588480000000002 2024-09-18T08:07:40.688 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:07:40.688 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:07:40.688 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "started", 2024-09-18T08:07:40.688 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:06:05.036775+0000", 2024-09-18T08:07:40.688 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.000107117 2024-09-18T08:07:40.688 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:07:40.688 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:07:40.689 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "waiting for subops from 0,5", 2024-09-18T08:07:40.689 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:06:05.036888+0000", 2024-09-18T08:07:40.689 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.00011260999999999999 2024-09-18T08:07:40.689 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:07:40.689 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:07:40.689 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:07:40.689 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:06:05.040643+0000", 2024-09-18T08:07:40.689 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.0037552969999999999 2024-09-18T08:07:40.689 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:07:40.689 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:07:40.689 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "op_commit", 2024-09-18T08:07:40.689 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:06:05.040912+0000", 2024-09-18T08:07:40.689 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.00026901699999999998 2024-09-18T08:07:40.690 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:07:40.690 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:07:40.690 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:07:40.690 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:06:05.118826+0000", 2024-09-18T08:07:40.690 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.077913498999999997 2024-09-18T08:07:40.690 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:07:40.690 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:07:40.690 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "commit_sent", 2024-09-18T08:07:40.690 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:06:05.118894+0000", 2024-09-18T08:07:40.690 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 6.8590000000000006e-05 2024-09-18T08:07:40.690 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:07:40.690 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:07:40.690 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "done", 2024-09-18T08:07:40.691 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:06:05.118947+0000", 2024-09-18T08:07:40.691 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 5.2438000000000002e-05 2024-09-18T08:07:40.691 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-09-18T08:07:40.691 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-09-18T08:07:40.691 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-09-18T08:07:40.691 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:07:40.691 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:07:40.691 INFO:teuthology.orchestra.run.smithi007.stdout: "description": "osd_op(client.14805.0:177709 7.e 7:780b878d:::benchmark_data_smithi007_73155_object59418:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e139)", 2024-09-18T08:07:40.691 INFO:teuthology.orchestra.run.smithi007.stdout: "initiated_at": "2024-09-18T08:06:38.569829+0000", 2024-09-18T08:07:40.691 INFO:teuthology.orchestra.run.smithi007.stdout: "age": 62.009888728999996, 2024-09-18T08:07:40.691 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.111834402, 2024-09-18T08:07:40.691 INFO:teuthology.orchestra.run.smithi007.stdout: "continuous": false, 2024-09-18T08:07:40.691 INFO:teuthology.orchestra.run.smithi007.stdout: "type_data": { 2024-09-18T08:07:40.691 INFO:teuthology.orchestra.run.smithi007.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T08:07:40.692 INFO:teuthology.orchestra.run.smithi007.stdout: "client_info": { 2024-09-18T08:07:40.692 INFO:teuthology.orchestra.run.smithi007.stdout: "client": "client.14805", 2024-09-18T08:07:40.692 INFO:teuthology.orchestra.run.smithi007.stdout: "client_addr": "172.21.15.7:0/640320436", 2024-09-18T08:07:40.692 INFO:teuthology.orchestra.run.smithi007.stdout: "tid": 177709 2024-09-18T08:07:40.692 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:07:40.692 INFO:teuthology.orchestra.run.smithi007.stdout: "events": [ 2024-09-18T08:07:40.692 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:07:40.692 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "initiated", 2024-09-18T08:07:40.692 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:06:38.569829+0000", 2024-09-18T08:07:40.692 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-09-18T08:07:40.692 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:07:40.692 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:07:40.692 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "header_read", 2024-09-18T08:07:40.693 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:06:38.569829+0000", 2024-09-18T08:07:40.693 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-09-18T08:07:40.693 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:07:40.693 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:07:40.693 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "throttled", 2024-09-18T08:07:40.693 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:06:38.569831+0000", 2024-09-18T08:07:40.693 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.451e-06 2024-09-18T08:07:40.693 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:07:40.693 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:07:40.694 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "all_read", 2024-09-18T08:07:40.694 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:06:38.569836+0000", 2024-09-18T08:07:40.694 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 5.4269999999999999e-06 2024-09-18T08:07:40.694 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:07:40.694 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:07:40.694 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "dispatched", 2024-09-18T08:07:40.694 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:06:38.569852+0000", 2024-09-18T08:07:40.694 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.6375000000000002e-05 2024-09-18T08:07:40.694 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:07:40.694 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:07:40.694 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "queued_for_pg", 2024-09-18T08:07:40.694 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:06:38.569861+0000", 2024-09-18T08:07:40.694 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 8.2749999999999995e-06 2024-09-18T08:07:40.694 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:07:40.695 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:07:40.695 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-09-18T08:07:40.695 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:06:38.571353+0000", 2024-09-18T08:07:40.695 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.0014927650000000001 2024-09-18T08:07:40.695 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:07:40.695 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:07:40.695 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "started", 2024-09-18T08:07:40.695 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:06:38.571454+0000", 2024-09-18T08:07:40.695 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.000100065 2024-09-18T08:07:40.695 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:07:40.696 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:07:40.696 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "waiting for subops from 0,5", 2024-09-18T08:07:40.696 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:06:38.571551+0000", 2024-09-18T08:07:40.696 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 9.7884e-05 2024-09-18T08:07:40.696 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:07:40.696 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:07:40.696 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:07:40.696 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:06:38.576046+0000", 2024-09-18T08:07:40.696 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.0044943379999999996 2024-09-18T08:07:40.696 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:07:40.696 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:07:40.696 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "op_commit", 2024-09-18T08:07:40.697 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:06:38.576188+0000", 2024-09-18T08:07:40.697 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.00014268500000000001 2024-09-18T08:07:40.697 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:07:40.697 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:07:40.697 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:07:40.697 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:06:38.681566+0000", 2024-09-18T08:07:40.697 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.10537708 2024-09-18T08:07:40.697 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:07:40.697 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:07:40.697 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "commit_sent", 2024-09-18T08:07:40.697 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:06:38.681619+0000", 2024-09-18T08:07:40.697 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 5.3919999999999999e-05 2024-09-18T08:07:40.697 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:07:40.697 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:07:40.698 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "done", 2024-09-18T08:07:40.698 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:06:38.681664+0000", 2024-09-18T08:07:40.698 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 4.4137000000000001e-05 2024-09-18T08:07:40.698 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-09-18T08:07:40.698 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-09-18T08:07:40.698 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-09-18T08:07:40.698 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:07:40.698 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:07:40.698 INFO:teuthology.orchestra.run.smithi007.stdout: "description": "osd_op(client.14805.0:204554 7.e 7:75af683c:::benchmark_data_smithi007_73155_object86263:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e148)", 2024-09-18T08:07:40.698 INFO:teuthology.orchestra.run.smithi007.stdout: "initiated_at": "2024-09-18T08:07:06.550254+0000", 2024-09-18T08:07:40.698 INFO:teuthology.orchestra.run.smithi007.stdout: "age": 34.029463702000001, 2024-09-18T08:07:40.698 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.090495268000000004, 2024-09-18T08:07:40.698 INFO:teuthology.orchestra.run.smithi007.stdout: "continuous": false, 2024-09-18T08:07:40.698 INFO:teuthology.orchestra.run.smithi007.stdout: "type_data": { 2024-09-18T08:07:40.699 INFO:teuthology.orchestra.run.smithi007.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T08:07:40.699 INFO:teuthology.orchestra.run.smithi007.stdout: "client_info": { 2024-09-18T08:07:40.699 INFO:teuthology.orchestra.run.smithi007.stdout: "client": "client.14805", 2024-09-18T08:07:40.699 INFO:teuthology.orchestra.run.smithi007.stdout: "client_addr": "172.21.15.7:0/640320436", 2024-09-18T08:07:40.699 INFO:teuthology.orchestra.run.smithi007.stdout: "tid": 204554 2024-09-18T08:07:40.699 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:07:40.699 INFO:teuthology.orchestra.run.smithi007.stdout: "events": [ 2024-09-18T08:07:40.699 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:07:40.699 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "initiated", 2024-09-18T08:07:40.699 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:07:06.550254+0000", 2024-09-18T08:07:40.699 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-09-18T08:07:40.699 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:07:40.699 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:07:40.700 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "header_read", 2024-09-18T08:07:40.700 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:07:06.550254+0000", 2024-09-18T08:07:40.700 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-09-18T08:07:40.700 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:07:40.700 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:07:40.700 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "throttled", 2024-09-18T08:07:40.700 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:07:06.550255+0000", 2024-09-18T08:07:40.700 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.04e-06 2024-09-18T08:07:40.700 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:07:40.700 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:07:40.700 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "all_read", 2024-09-18T08:07:40.701 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:07:06.550260+0000", 2024-09-18T08:07:40.701 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 4.4939999999999997e-06 2024-09-18T08:07:40.701 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:07:40.701 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:07:40.701 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "dispatched", 2024-09-18T08:07:40.701 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:07:06.550272+0000", 2024-09-18T08:07:40.701 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.2672e-05 2024-09-18T08:07:40.701 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:07:40.701 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:07:40.701 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "queued_for_pg", 2024-09-18T08:07:40.701 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:07:06.550279+0000", 2024-09-18T08:07:40.701 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 7.0099999999999998e-06 2024-09-18T08:07:40.701 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:07:40.702 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:07:40.702 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-09-18T08:07:40.702 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:07:06.553248+0000", 2024-09-18T08:07:40.702 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.0029683999999999999 2024-09-18T08:07:40.702 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:07:40.702 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:07:40.702 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "started", 2024-09-18T08:07:40.702 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:07:06.553346+0000", 2024-09-18T08:07:40.702 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 9.8618999999999999e-05 2024-09-18T08:07:40.702 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:07:40.702 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:07:40.702 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "waiting for subops from 0,5", 2024-09-18T08:07:40.702 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:07:06.553450+0000", 2024-09-18T08:07:40.702 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.00010311599999999999 2024-09-18T08:07:40.703 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:07:40.703 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:07:40.703 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "op_commit", 2024-09-18T08:07:40.703 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:07:06.560605+0000", 2024-09-18T08:07:40.703 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.0071556229999999998 2024-09-18T08:07:40.703 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:07:40.703 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:07:40.703 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:07:40.703 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:07:06.565908+0000", 2024-09-18T08:07:40.703 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.0053023480000000001 2024-09-18T08:07:40.703 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:07:40.703 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:07:40.704 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:07:40.704 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:07:06.640602+0000", 2024-09-18T08:07:40.704 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.074694521 2024-09-18T08:07:40.704 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:07:40.704 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:07:40.704 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "commit_sent", 2024-09-18T08:07:40.704 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:07:06.640706+0000", 2024-09-18T08:07:40.704 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.00010401800000000001 2024-09-18T08:07:40.704 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:07:40.704 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:07:40.704 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "done", 2024-09-18T08:07:40.704 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:07:06.640749+0000", 2024-09-18T08:07:40.704 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 4.3407e-05 2024-09-18T08:07:40.705 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-09-18T08:07:40.705 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-09-18T08:07:40.705 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-09-18T08:07:40.705 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:07:40.705 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:07:40.705 INFO:teuthology.orchestra.run.smithi007.stdout: "description": "osd_op(client.14805.0:204556 7.e 7:775e8c17:::benchmark_data_smithi007_73155_object86265:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e148)", 2024-09-18T08:07:40.705 INFO:teuthology.orchestra.run.smithi007.stdout: "initiated_at": "2024-09-18T08:07:06.556408+0000", 2024-09-18T08:07:40.705 INFO:teuthology.orchestra.run.smithi007.stdout: "age": 34.023309781999998, 2024-09-18T08:07:40.705 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.084827296999999996, 2024-09-18T08:07:40.705 INFO:teuthology.orchestra.run.smithi007.stdout: "continuous": false, 2024-09-18T08:07:40.705 INFO:teuthology.orchestra.run.smithi007.stdout: "type_data": { 2024-09-18T08:07:40.705 INFO:teuthology.orchestra.run.smithi007.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T08:07:40.706 INFO:teuthology.orchestra.run.smithi007.stdout: "client_info": { 2024-09-18T08:07:40.706 INFO:teuthology.orchestra.run.smithi007.stdout: "client": "client.14805", 2024-09-18T08:07:40.706 INFO:teuthology.orchestra.run.smithi007.stdout: "client_addr": "172.21.15.7:0/640320436", 2024-09-18T08:07:40.706 INFO:teuthology.orchestra.run.smithi007.stdout: "tid": 204556 2024-09-18T08:07:40.706 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:07:40.706 INFO:teuthology.orchestra.run.smithi007.stdout: "events": [ 2024-09-18T08:07:40.706 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:07:40.706 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "initiated", 2024-09-18T08:07:40.706 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:07:06.556408+0000", 2024-09-18T08:07:40.706 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-09-18T08:07:40.706 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:07:40.706 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:07:40.706 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "header_read", 2024-09-18T08:07:40.707 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:07:06.556408+0000", 2024-09-18T08:07:40.707 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-09-18T08:07:40.707 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:07:40.707 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:07:40.707 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "throttled", 2024-09-18T08:07:40.707 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:07:06.556409+0000", 2024-09-18T08:07:40.707 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 4.75e-07 2024-09-18T08:07:40.707 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:07:40.707 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:07:40.707 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "all_read", 2024-09-18T08:07:40.707 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:07:06.556411+0000", 2024-09-18T08:07:40.708 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 2.385e-06 2024-09-18T08:07:40.708 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:07:40.708 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:07:40.708 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "dispatched", 2024-09-18T08:07:40.708 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:07:06.556417+0000", 2024-09-18T08:07:40.708 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 5.6899999999999997e-06 2024-09-18T08:07:40.708 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:07:40.708 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:07:40.708 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "queued_for_pg", 2024-09-18T08:07:40.708 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:07:06.556422+0000", 2024-09-18T08:07:40.708 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 5.4759999999999997e-06 2024-09-18T08:07:40.708 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:07:40.708 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:07:40.709 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-09-18T08:07:40.709 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:07:06.556816+0000", 2024-09-18T08:07:40.709 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.000393464 2024-09-18T08:07:40.709 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:07:40.709 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:07:40.709 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "started", 2024-09-18T08:07:40.709 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:07:06.556918+0000", 2024-09-18T08:07:40.709 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.000101906 2024-09-18T08:07:40.709 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:07:40.709 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:07:40.709 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "waiting for subops from 0,5", 2024-09-18T08:07:40.709 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:07:06.557041+0000", 2024-09-18T08:07:40.709 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.00012318999999999999 2024-09-18T08:07:40.710 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:07:40.710 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:07:40.710 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "op_commit", 2024-09-18T08:07:40.710 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:07:06.560731+0000", 2024-09-18T08:07:40.710 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.0036905409999999999 2024-09-18T08:07:40.710 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:07:40.710 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:07:40.710 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:07:40.710 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:07:06.565945+0000", 2024-09-18T08:07:40.710 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.0052139070000000003 2024-09-18T08:07:40.710 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:07:40.711 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:07:40.711 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:07:40.711 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:07:06.640790+0000", 2024-09-18T08:07:40.711 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.074844847000000006 2024-09-18T08:07:40.711 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:07:40.711 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:07:40.711 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "commit_sent", 2024-09-18T08:07:40.711 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:07:06.640919+0000", 2024-09-18T08:07:40.711 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.00012856099999999999 2024-09-18T08:07:40.711 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:07:40.711 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:07:40.712 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "done", 2024-09-18T08:07:40.712 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:07:06.641235+0000", 2024-09-18T08:07:40.712 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.000316855 2024-09-18T08:07:40.712 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-09-18T08:07:40.712 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-09-18T08:07:40.712 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-09-18T08:07:40.712 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:07:40.712 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:07:40.712 INFO:teuthology.orchestra.run.smithi007.stdout: "description": "osd_op(client.14805.0:204569 7.5 7:a57af9cd:::benchmark_data_smithi007_73155_object86278:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e148)", 2024-09-18T08:07:40.712 INFO:teuthology.orchestra.run.smithi007.stdout: "initiated_at": "2024-09-18T08:07:06.563553+0000", 2024-09-18T08:07:40.712 INFO:teuthology.orchestra.run.smithi007.stdout: "age": 34.016164619000001, 2024-09-18T08:07:40.712 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.089587238, 2024-09-18T08:07:40.713 INFO:teuthology.orchestra.run.smithi007.stdout: "continuous": false, 2024-09-18T08:07:40.713 INFO:teuthology.orchestra.run.smithi007.stdout: "type_data": { 2024-09-18T08:07:40.713 INFO:teuthology.orchestra.run.smithi007.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T08:07:40.713 INFO:teuthology.orchestra.run.smithi007.stdout: "client_info": { 2024-09-18T08:07:40.713 INFO:teuthology.orchestra.run.smithi007.stdout: "client": "client.14805", 2024-09-18T08:07:40.713 INFO:teuthology.orchestra.run.smithi007.stdout: "client_addr": "172.21.15.7:0/640320436", 2024-09-18T08:07:40.713 INFO:teuthology.orchestra.run.smithi007.stdout: "tid": 204569 2024-09-18T08:07:40.713 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:07:40.713 INFO:teuthology.orchestra.run.smithi007.stdout: "events": [ 2024-09-18T08:07:40.713 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:07:40.713 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "initiated", 2024-09-18T08:07:40.713 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:07:06.563553+0000", 2024-09-18T08:07:40.713 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-09-18T08:07:40.714 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:07:40.714 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:07:40.714 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "header_read", 2024-09-18T08:07:40.714 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:07:06.563553+0000", 2024-09-18T08:07:40.714 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-09-18T08:07:40.714 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:07:40.714 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:07:40.714 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "throttled", 2024-09-18T08:07:40.714 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:07:06.563554+0000", 2024-09-18T08:07:40.714 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 4.8100000000000003e-07 2024-09-18T08:07:40.715 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:07:40.715 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:07:40.715 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "all_read", 2024-09-18T08:07:40.715 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:07:06.563557+0000", 2024-09-18T08:07:40.715 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 2.7839999999999999e-06 2024-09-18T08:07:40.715 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:07:40.715 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:07:40.715 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "dispatched", 2024-09-18T08:07:40.715 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:07:06.563562+0000", 2024-09-18T08:07:40.715 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 5.7289999999999997e-06 2024-09-18T08:07:40.715 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:07:40.715 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:07:40.715 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "queued_for_pg", 2024-09-18T08:07:40.716 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:07:06.563567+0000", 2024-09-18T08:07:40.716 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 4.9949999999999996e-06 2024-09-18T08:07:40.716 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:07:40.716 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:07:40.716 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-09-18T08:07:40.716 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:07:06.563655+0000", 2024-09-18T08:07:40.716 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 8.7689999999999996e-05 2024-09-18T08:07:40.716 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:07:40.716 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:07:40.716 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "started", 2024-09-18T08:07:40.716 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:07:06.563758+0000", 2024-09-18T08:07:40.716 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.000102538 2024-09-18T08:07:40.717 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:07:40.717 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:07:40.717 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "waiting for subops from 0,7", 2024-09-18T08:07:40.717 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:07:06.563861+0000", 2024-09-18T08:07:40.717 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.00010385799999999999 2024-09-18T08:07:40.717 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:07:40.717 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:07:40.717 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:07:40.717 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:07:06.570416+0000", 2024-09-18T08:07:40.717 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.0065544449999999999 2024-09-18T08:07:40.717 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:07:40.717 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:07:40.718 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "op_commit", 2024-09-18T08:07:40.718 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:07:06.570428+0000", 2024-09-18T08:07:40.718 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.2537999999999999e-05 2024-09-18T08:07:40.718 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:07:40.718 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:07:40.718 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:07:40.718 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:07:06.649777+0000", 2024-09-18T08:07:40.718 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.079348518000000007 2024-09-18T08:07:40.718 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:07:40.718 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:07:40.718 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "commit_sent", 2024-09-18T08:07:40.718 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:07:06.653092+0000", 2024-09-18T08:07:40.718 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.0033149939999999999 2024-09-18T08:07:40.719 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:07:40.719 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:07:40.719 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "done", 2024-09-18T08:07:40.719 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:07:06.653141+0000", 2024-09-18T08:07:40.719 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 4.8668000000000002e-05 2024-09-18T08:07:40.719 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-09-18T08:07:40.719 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-09-18T08:07:40.719 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-09-18T08:07:40.719 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:07:40.719 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:07:40.719 INFO:teuthology.orchestra.run.smithi007.stdout: "description": "osd_op(client.14805.0:204570 7.e 7:7cca7ac7:::benchmark_data_smithi007_73155_object86279:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e148)", 2024-09-18T08:07:40.720 INFO:teuthology.orchestra.run.smithi007.stdout: "initiated_at": "2024-09-18T08:07:06.564546+0000", 2024-09-18T08:07:40.720 INFO:teuthology.orchestra.run.smithi007.stdout: "age": 34.015172024999998, 2024-09-18T08:07:40.720 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.083745940000000005, 2024-09-18T08:07:40.720 INFO:teuthology.orchestra.run.smithi007.stdout: "continuous": false, 2024-09-18T08:07:40.720 INFO:teuthology.orchestra.run.smithi007.stdout: "type_data": { 2024-09-18T08:07:40.720 INFO:teuthology.orchestra.run.smithi007.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T08:07:40.720 INFO:teuthology.orchestra.run.smithi007.stdout: "client_info": { 2024-09-18T08:07:40.720 INFO:teuthology.orchestra.run.smithi007.stdout: "client": "client.14805", 2024-09-18T08:07:40.720 INFO:teuthology.orchestra.run.smithi007.stdout: "client_addr": "172.21.15.7:0/640320436", 2024-09-18T08:07:40.720 INFO:teuthology.orchestra.run.smithi007.stdout: "tid": 204570 2024-09-18T08:07:40.720 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:07:40.720 INFO:teuthology.orchestra.run.smithi007.stdout: "events": [ 2024-09-18T08:07:40.720 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:07:40.721 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "initiated", 2024-09-18T08:07:40.721 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:07:06.564546+0000", 2024-09-18T08:07:40.721 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-09-18T08:07:40.721 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:07:40.721 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:07:40.721 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "header_read", 2024-09-18T08:07:40.721 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:07:06.564546+0000", 2024-09-18T08:07:40.721 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-09-18T08:07:40.721 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:07:40.721 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:07:40.721 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "throttled", 2024-09-18T08:07:40.721 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:07:06.564547+0000", 2024-09-18T08:07:40.722 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 8.8299999999999995e-07 2024-09-18T08:07:40.722 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:07:40.722 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:07:40.722 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "all_read", 2024-09-18T08:07:40.722 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:07:06.564551+0000", 2024-09-18T08:07:40.722 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 4.2930000000000002e-06 2024-09-18T08:07:40.722 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:07:40.722 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:07:40.722 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "dispatched", 2024-09-18T08:07:40.722 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:07:06.564561+0000", 2024-09-18T08:07:40.722 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 9.9590000000000007e-06 2024-09-18T08:07:40.722 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:07:40.722 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:07:40.722 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "queued_for_pg", 2024-09-18T08:07:40.723 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:07:06.564568+0000", 2024-09-18T08:07:40.723 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 7.0119999999999999e-06 2024-09-18T08:07:40.723 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:07:40.723 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:07:40.723 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-09-18T08:07:40.723 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:07:06.565973+0000", 2024-09-18T08:07:40.723 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.001405199 2024-09-18T08:07:40.723 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:07:40.723 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:07:40.723 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "started", 2024-09-18T08:07:40.723 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:07:06.566089+0000", 2024-09-18T08:07:40.723 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.000116174 2024-09-18T08:07:40.723 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:07:40.724 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:07:40.724 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "waiting for subops from 0,5", 2024-09-18T08:07:40.724 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:07:06.566207+0000", 2024-09-18T08:07:40.724 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.000117908 2024-09-18T08:07:40.724 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:07:40.724 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:07:40.724 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:07:40.724 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:07:06.569896+0000", 2024-09-18T08:07:40.724 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.003688794 2024-09-18T08:07:40.724 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:07:40.724 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:07:40.724 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "op_commit", 2024-09-18T08:07:40.724 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:07:06.569907+0000", 2024-09-18T08:07:40.724 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.1331e-05 2024-09-18T08:07:40.725 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:07:40.725 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:07:40.725 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:07:40.725 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:07:06.647821+0000", 2024-09-18T08:07:40.725 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.077913123000000001 2024-09-18T08:07:40.725 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:07:40.725 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:07:40.725 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "commit_sent", 2024-09-18T08:07:40.725 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:07:06.648242+0000", 2024-09-18T08:07:40.725 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.00042181199999999998 2024-09-18T08:07:40.725 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:07:40.725 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:07:40.725 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "done", 2024-09-18T08:07:40.726 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:07:06.648292+0000", 2024-09-18T08:07:40.726 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 4.9452e-05 2024-09-18T08:07:40.726 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-09-18T08:07:40.726 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-09-18T08:07:40.726 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-09-18T08:07:40.726 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:07:40.726 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:07:40.726 INFO:teuthology.orchestra.run.smithi007.stdout: "description": "osd_op(client.14805.0:204572 7.5 7:a6f477fa:::benchmark_data_smithi007_73155_object86281:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e148)", 2024-09-18T08:07:40.726 INFO:teuthology.orchestra.run.smithi007.stdout: "initiated_at": "2024-09-18T08:07:06.565647+0000", 2024-09-18T08:07:40.726 INFO:teuthology.orchestra.run.smithi007.stdout: "age": 34.014071348000002, 2024-09-18T08:07:40.726 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.087897958999999998, 2024-09-18T08:07:40.726 INFO:teuthology.orchestra.run.smithi007.stdout: "continuous": false, 2024-09-18T08:07:40.726 INFO:teuthology.orchestra.run.smithi007.stdout: "type_data": { 2024-09-18T08:07:40.726 INFO:teuthology.orchestra.run.smithi007.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T08:07:40.726 INFO:teuthology.orchestra.run.smithi007.stdout: "client_info": { 2024-09-18T08:07:40.727 INFO:teuthology.orchestra.run.smithi007.stdout: "client": "client.14805", 2024-09-18T08:07:40.727 INFO:teuthology.orchestra.run.smithi007.stdout: "client_addr": "172.21.15.7:0/640320436", 2024-09-18T08:07:40.727 INFO:teuthology.orchestra.run.smithi007.stdout: "tid": 204572 2024-09-18T08:07:40.727 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:07:40.727 INFO:teuthology.orchestra.run.smithi007.stdout: "events": [ 2024-09-18T08:07:40.727 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:07:40.727 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "initiated", 2024-09-18T08:07:40.727 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:07:06.565647+0000", 2024-09-18T08:07:40.727 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-09-18T08:07:40.727 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:07:40.727 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:07:40.727 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "header_read", 2024-09-18T08:07:40.727 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:07:06.565647+0000", 2024-09-18T08:07:40.728 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-09-18T08:07:40.728 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:07:40.728 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:07:40.728 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "throttled", 2024-09-18T08:07:40.728 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:07:06.565647+0000", 2024-09-18T08:07:40.728 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 7.0800000000000004e-07 2024-09-18T08:07:40.728 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:07:40.728 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:07:40.728 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "all_read", 2024-09-18T08:07:40.728 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:07:06.565651+0000", 2024-09-18T08:07:40.728 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 3.3299999999999999e-06 2024-09-18T08:07:40.728 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:07:40.728 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:07:40.728 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "dispatched", 2024-09-18T08:07:40.729 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:07:06.565659+0000", 2024-09-18T08:07:40.729 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 8.7250000000000003e-06 2024-09-18T08:07:40.729 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:07:40.729 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:07:40.729 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "queued_for_pg", 2024-09-18T08:07:40.729 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:07:06.565665+0000", 2024-09-18T08:07:40.729 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 5.7259999999999999e-06 2024-09-18T08:07:40.729 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:07:40.729 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:07:40.729 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-09-18T08:07:40.729 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:07:06.567127+0000", 2024-09-18T08:07:40.729 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.001461604 2024-09-18T08:07:40.730 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:07:40.730 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:07:40.730 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "started", 2024-09-18T08:07:40.730 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:07:06.567226+0000", 2024-09-18T08:07:40.730 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 9.9807000000000002e-05 2024-09-18T08:07:40.730 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:07:40.730 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:07:40.730 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "waiting for subops from 0,7", 2024-09-18T08:07:40.730 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:07:06.567324+0000", 2024-09-18T08:07:40.730 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 9.7994999999999999e-05 2024-09-18T08:07:40.730 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:07:40.730 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:07:40.730 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:07:40.731 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:07:06.570467+0000", 2024-09-18T08:07:40.731 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.0031425300000000001 2024-09-18T08:07:40.731 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:07:40.731 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:07:40.731 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "op_commit", 2024-09-18T08:07:40.731 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:07:06.570486+0000", 2024-09-18T08:07:40.731 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.906e-05 2024-09-18T08:07:40.732 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:07:40.732 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:07:40.732 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:07:40.732 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:07:06.653186+0000", 2024-09-18T08:07:40.732 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.082700138000000006 2024-09-18T08:07:40.732 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:07:40.733 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:07:40.733 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "commit_sent", 2024-09-18T08:07:40.733 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:07:06.653506+0000", 2024-09-18T08:07:40.733 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.00031962000000000001 2024-09-18T08:07:40.733 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:07:40.734 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:07:40.734 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "done", 2024-09-18T08:07:40.734 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:07:06.653545+0000", 2024-09-18T08:07:40.734 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 3.8716e-05 2024-09-18T08:07:40.734 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-09-18T08:07:40.734 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-09-18T08:07:40.735 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-09-18T08:07:40.735 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:07:40.735 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:07:40.735 INFO:teuthology.orchestra.run.smithi007.stdout: "description": "osd_op(client.14805.0:223793 7.5 7:a37824ff:::benchmark_data_smithi007_73155_object105502:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e154)", 2024-09-18T08:07:40.735 INFO:teuthology.orchestra.run.smithi007.stdout: "initiated_at": "2024-09-18T08:07:26.183612+0000", 2024-09-18T08:07:40.736 INFO:teuthology.orchestra.run.smithi007.stdout: "age": 14.396106143000001, 2024-09-18T08:07:40.736 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.104886888, 2024-09-18T08:07:40.736 INFO:teuthology.orchestra.run.smithi007.stdout: "continuous": false, 2024-09-18T08:07:40.736 INFO:teuthology.orchestra.run.smithi007.stdout: "type_data": { 2024-09-18T08:07:40.736 INFO:teuthology.orchestra.run.smithi007.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T08:07:40.736 INFO:teuthology.orchestra.run.smithi007.stdout: "client_info": { 2024-09-18T08:07:40.737 INFO:teuthology.orchestra.run.smithi007.stdout: "client": "client.14805", 2024-09-18T08:07:40.737 INFO:teuthology.orchestra.run.smithi007.stdout: "client_addr": "172.21.15.7:0/640320436", 2024-09-18T08:07:40.737 INFO:teuthology.orchestra.run.smithi007.stdout: "tid": 223793 2024-09-18T08:07:40.737 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:07:40.737 INFO:teuthology.orchestra.run.smithi007.stdout: "events": [ 2024-09-18T08:07:40.738 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:07:40.738 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "initiated", 2024-09-18T08:07:40.738 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:07:26.183612+0000", 2024-09-18T08:07:40.738 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-09-18T08:07:40.738 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:07:40.738 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:07:40.739 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "header_read", 2024-09-18T08:07:40.739 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:07:26.183612+0000", 2024-09-18T08:07:40.739 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-09-18T08:07:40.739 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:07:40.739 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:07:40.739 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "throttled", 2024-09-18T08:07:40.740 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:07:26.183613+0000", 2024-09-18T08:07:40.740 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.3030000000000001e-06 2024-09-18T08:07:40.740 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:07:40.740 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:07:40.740 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "all_read", 2024-09-18T08:07:40.740 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:07:26.183618+0000", 2024-09-18T08:07:40.741 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 4.7870000000000001e-06 2024-09-18T08:07:40.741 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:07:40.741 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:07:40.741 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "dispatched", 2024-09-18T08:07:40.741 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:07:26.183629+0000", 2024-09-18T08:07:40.741 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.1100999999999999e-05 2024-09-18T08:07:40.742 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:07:40.742 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:07:40.742 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "queued_for_pg", 2024-09-18T08:07:40.742 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:07:26.183636+0000", 2024-09-18T08:07:40.742 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 7.3980000000000002e-06 2024-09-18T08:07:40.742 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:07:40.743 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:07:40.743 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-09-18T08:07:40.743 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:07:26.183957+0000", 2024-09-18T08:07:40.743 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.00032060999999999998 2024-09-18T08:07:40.743 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:07:40.743 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:07:40.743 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "started", 2024-09-18T08:07:40.743 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:07:26.184232+0000", 2024-09-18T08:07:40.744 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.00027534499999999997 2024-09-18T08:07:40.744 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:07:40.744 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:07:40.744 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "waiting for subops from 0,7", 2024-09-18T08:07:40.744 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:07:26.184350+0000", 2024-09-18T08:07:40.744 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.00011803600000000001 2024-09-18T08:07:40.744 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:07:40.745 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:07:40.745 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:07:40.745 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:07:26.235645+0000", 2024-09-18T08:07:40.745 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.051294868 2024-09-18T08:07:40.745 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:07:40.745 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:07:40.745 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "op_commit", 2024-09-18T08:07:40.745 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:07:26.235664+0000", 2024-09-18T08:07:40.746 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.9236e-05 2024-09-18T08:07:40.746 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:07:40.746 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:07:40.746 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:07:40.746 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:07:26.288261+0000", 2024-09-18T08:07:40.746 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.052596716000000002 2024-09-18T08:07:40.746 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:07:40.747 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:07:40.747 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "commit_sent", 2024-09-18T08:07:40.747 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:07:26.288452+0000", 2024-09-18T08:07:40.747 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.00019043700000000001 2024-09-18T08:07:40.747 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:07:40.748 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:07:40.748 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "done", 2024-09-18T08:07:40.748 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:07:26.288499+0000", 2024-09-18T08:07:40.748 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 4.7051000000000001e-05 2024-09-18T08:07:40.748 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-09-18T08:07:40.748 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-09-18T08:07:40.749 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-09-18T08:07:40.749 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:07:40.749 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:07:40.749 INFO:teuthology.orchestra.run.smithi007.stdout: "description": "osd_op(client.14805.0:223799 7.e 7:7f447ed9:::benchmark_data_smithi007_73155_object105508:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e154)", 2024-09-18T08:07:40.749 INFO:teuthology.orchestra.run.smithi007.stdout: "initiated_at": "2024-09-18T08:07:26.184566+0000", 2024-09-18T08:07:40.749 INFO:teuthology.orchestra.run.smithi007.stdout: "age": 14.395152177, 2024-09-18T08:07:40.749 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.114658156, 2024-09-18T08:07:40.749 INFO:teuthology.orchestra.run.smithi007.stdout: "continuous": false, 2024-09-18T08:07:40.749 INFO:teuthology.orchestra.run.smithi007.stdout: "type_data": { 2024-09-18T08:07:40.749 INFO:teuthology.orchestra.run.smithi007.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T08:07:40.749 INFO:teuthology.orchestra.run.smithi007.stdout: "client_info": { 2024-09-18T08:07:40.749 INFO:teuthology.orchestra.run.smithi007.stdout: "client": "client.14805", 2024-09-18T08:07:40.750 INFO:teuthology.orchestra.run.smithi007.stdout: "client_addr": "172.21.15.7:0/640320436", 2024-09-18T08:07:40.750 INFO:teuthology.orchestra.run.smithi007.stdout: "tid": 223799 2024-09-18T08:07:40.750 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:07:40.750 INFO:teuthology.orchestra.run.smithi007.stdout: "events": [ 2024-09-18T08:07:40.750 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:07:40.750 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "initiated", 2024-09-18T08:07:40.750 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:07:26.184566+0000", 2024-09-18T08:07:40.750 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-09-18T08:07:40.751 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:07:40.751 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:07:40.751 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "header_read", 2024-09-18T08:07:40.751 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:07:26.184566+0000", 2024-09-18T08:07:40.751 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-09-18T08:07:40.751 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:07:40.751 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:07:40.751 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "throttled", 2024-09-18T08:07:40.751 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:07:26.184593+0000", 2024-09-18T08:07:40.751 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 2.6831000000000001e-05 2024-09-18T08:07:40.751 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:07:40.752 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:07:40.752 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "all_read", 2024-09-18T08:07:40.752 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:07:26.184597+0000", 2024-09-18T08:07:40.752 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 4.8119999999999998e-06 2024-09-18T08:07:40.752 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:07:40.752 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:07:40.752 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "dispatched", 2024-09-18T08:07:40.752 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:07:26.184607+0000", 2024-09-18T08:07:40.752 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 9.9559999999999992e-06 2024-09-18T08:07:40.752 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:07:40.752 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:07:40.752 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "queued_for_pg", 2024-09-18T08:07:40.753 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:07:26.184614+0000", 2024-09-18T08:07:40.753 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 6.7730000000000002e-06 2024-09-18T08:07:40.753 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:07:40.753 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:07:40.753 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-09-18T08:07:40.753 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:07:26.184669+0000", 2024-09-18T08:07:40.753 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 5.4719000000000002e-05 2024-09-18T08:07:40.753 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:07:40.753 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:07:40.753 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "started", 2024-09-18T08:07:40.753 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:07:26.184764+0000", 2024-09-18T08:07:40.753 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 9.5529e-05 2024-09-18T08:07:40.754 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:07:40.754 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:07:40.754 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "waiting for subops from 0,5", 2024-09-18T08:07:40.754 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:07:26.184872+0000", 2024-09-18T08:07:40.754 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.000108115 2024-09-18T08:07:40.754 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:07:40.754 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:07:40.754 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:07:40.754 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:07:26.235645+0000", 2024-09-18T08:07:40.754 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.050772920999999999 2024-09-18T08:07:40.754 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:07:40.754 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:07:40.754 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "op_commit", 2024-09-18T08:07:40.755 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:07:26.235665+0000", 2024-09-18T08:07:40.755 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.9585999999999998e-05 2024-09-18T08:07:40.755 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:07:40.755 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:07:40.755 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:07:40.755 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:07:26.298883+0000", 2024-09-18T08:07:40.755 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.063217624 2024-09-18T08:07:40.755 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:07:40.755 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:07:40.755 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "commit_sent", 2024-09-18T08:07:40.755 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:07:26.299173+0000", 2024-09-18T08:07:40.755 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.00029084500000000002 2024-09-18T08:07:40.755 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:07:40.755 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:07:40.756 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "done", 2024-09-18T08:07:40.756 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:07:26.299224+0000", 2024-09-18T08:07:40.756 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 5.0445000000000002e-05 2024-09-18T08:07:40.756 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-09-18T08:07:40.756 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-09-18T08:07:40.756 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-09-18T08:07:40.756 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-09-18T08:07:40.756 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-09-18T08:07:40.756 INFO:teuthology.orchestra.run.smithi007.stdout:} 2024-09-18T08:07:40.757 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/779349ca-7593-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T08:07:40.757 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:07:40 smithi050 ceph-mon[32371]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-18T08:07:40.757 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:07:40 smithi050 ceph-mon[32371]: pgmap v220: 158 pgs: 1 peering, 157 active+clean; 11 MiB data, 1.0 GiB used, 714 GiB / 715 GiB avail; 1.3 KiB/s rd, 0 B/s wr, 1.33k op/s 2024-09-18T08:07:40.758 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:07:40 smithi050 ceph-mon[32371]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-18T08:07:40.758 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:07:40 smithi050 ceph-mon[32371]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-18T08:07:40.758 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:07:40 smithi050 ceph-mon[32371]: osdmap e160: 8 total, 8 up, 8 in 2024-09-18T08:07:40.758 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:07:40 smithi050 ceph-mon[32371]: 4.1c scrub starts 2024-09-18T08:07:40.758 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:07:40 smithi050 ceph-mon[32371]: 4.1c scrub ok 2024-09-18T08:07:40.758 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:07:40 smithi050 ceph-mon[32371]: 4.6 scrub starts 2024-09-18T08:07:40.758 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:07:40 smithi050 ceph-mon[32371]: 4.6 scrub ok 2024-09-18T08:07:40.779 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:07:40.780 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:07:40.780 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1' 2024-09-18T08:07:40.881 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0 2024-09-18T08:07:40.904 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:07:40.904 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:07:40.905 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0' 2024-09-18T08:07:41.006 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2 2024-09-18T08:07:41.028 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:07:41.029 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:07:41.029 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2' 2024-09-18T08:07:41.130 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7 2024-09-18T08:07:41.157 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:07:41.157 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:07:41.158 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7' 2024-09-18T08:07:41.259 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1 2024-09-18T08:07:41.282 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:07:41.283 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:07:41.284 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1' 2024-09-18T08:07:41.384 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7 2024-09-18T08:07:41.411 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:07:41.412 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:07:41.412 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7' 2024-09-18T08:07:41.513 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7 2024-09-18T08:07:41.540 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:07:41.541 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:07:41.542 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7' 2024-09-18T08:07:41.629 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:07:41 smithi007 ceph-mon[29459]: purged_snaps scrub starts 2024-09-18T08:07:41.630 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:07:41 smithi007 ceph-mon[29459]: purged_snaps scrub ok 2024-09-18T08:07:41.630 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:07:41 smithi007 ceph-mon[29459]: osdmap e161: 8 total, 8 up, 8 in 2024-09-18T08:07:41.630 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:07:41 smithi007 ceph-mon[39249]: purged_snaps scrub starts 2024-09-18T08:07:41.630 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:07:41 smithi007 ceph-mon[39249]: purged_snaps scrub ok 2024-09-18T08:07:41.630 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:07:41 smithi007 ceph-mon[39249]: osdmap e161: 8 total, 8 up, 8 in 2024-09-18T08:07:41.643 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3 2024-09-18T08:07:41.655 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:07:41 smithi050 ceph-mon[32371]: purged_snaps scrub starts 2024-09-18T08:07:41.656 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:07:41 smithi050 ceph-mon[32371]: purged_snaps scrub ok 2024-09-18T08:07:41.656 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:07:41 smithi050 ceph-mon[32371]: osdmap e161: 8 total, 8 up, 8 in 2024-09-18T08:07:41.667 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid 779349ca-7593-11ef-bceb-c7b262605968 -- ceph daemon osd.3 dump_ops_in_flight 2024-09-18T08:07:41.669 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:07:41.670 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:07:41.670 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3' 2024-09-18T08:07:41.771 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3 2024-09-18T08:07:41.793 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:07:41.794 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:07:41.794 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3' 2024-09-18T08:07:41.895 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3 2024-09-18T08:07:41.918 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:07:41.918 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:07:41.918 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3' 2024-09-18T08:07:42.019 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6 2024-09-18T08:07:42.046 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:07:42.047 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:07:42.047 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6' 2024-09-18T08:07:42.113 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/779349ca-7593-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T08:07:42.119 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/779349ca-7593-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T08:07:42.148 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1 2024-09-18T08:07:42.171 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:07:42.171 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:07:42.171 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1' 2024-09-18T08:07:42.272 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0 2024-09-18T08:07:42.295 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:07:42.295 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:07:42.296 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0' 2024-09-18T08:07:42.357 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:07:42 smithi007 ceph-mon[29459]: 3.c scrub starts 2024-09-18T08:07:42.357 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:07:42 smithi007 ceph-mon[29459]: 3.c scrub ok 2024-09-18T08:07:42.358 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:07:42 smithi007 ceph-mon[29459]: 4.a scrub starts 2024-09-18T08:07:42.358 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:07:42 smithi007 ceph-mon[29459]: 4.a scrub ok 2024-09-18T08:07:42.358 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:07:42 smithi007 ceph-mon[29459]: pgmap v223: 158 pgs: 1 peering, 157 active+clean; 11 MiB data, 1.0 GiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.25k op/s 2024-09-18T08:07:42.358 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:07:42 smithi007 ceph-mon[29459]: 4.10 scrub starts 2024-09-18T08:07:42.358 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:07:42 smithi007 ceph-mon[29459]: 4.10 scrub ok 2024-09-18T08:07:42.358 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:07:42 smithi007 ceph-mon[39249]: 3.c scrub starts 2024-09-18T08:07:42.358 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:07:42 smithi007 ceph-mon[39249]: 3.c scrub ok 2024-09-18T08:07:42.358 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:07:42 smithi007 ceph-mon[39249]: 4.a scrub starts 2024-09-18T08:07:42.358 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:07:42 smithi007 ceph-mon[39249]: 4.a scrub ok 2024-09-18T08:07:42.358 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:07:42 smithi007 ceph-mon[39249]: pgmap v223: 158 pgs: 1 peering, 157 active+clean; 11 MiB data, 1.0 GiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.25k op/s 2024-09-18T08:07:42.359 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:07:42 smithi007 ceph-mon[39249]: 4.10 scrub starts 2024-09-18T08:07:42.359 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:07:42 smithi007 ceph-mon[39249]: 4.10 scrub ok 2024-09-18T08:07:42.397 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7 2024-09-18T08:07:42.424 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:07:42.425 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:07:42.426 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7' 2024-09-18T08:07:42.526 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1 2024-09-18T08:07:42.548 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:07:42.549 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:07:42.549 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1' 2024-09-18T08:07:42.557 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:07:42 smithi050 ceph-mon[32371]: 3.c scrub starts 2024-09-18T08:07:42.557 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:07:42 smithi050 ceph-mon[32371]: 3.c scrub ok 2024-09-18T08:07:42.557 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:07:42 smithi050 ceph-mon[32371]: 4.a scrub starts 2024-09-18T08:07:42.557 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:07:42 smithi050 ceph-mon[32371]: 4.a scrub ok 2024-09-18T08:07:42.557 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:07:42 smithi050 ceph-mon[32371]: pgmap v223: 158 pgs: 1 peering, 157 active+clean; 11 MiB data, 1.0 GiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.25k op/s 2024-09-18T08:07:42.557 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:07:42 smithi050 ceph-mon[32371]: 4.10 scrub starts 2024-09-18T08:07:42.558 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:07:42 smithi050 ceph-mon[32371]: 4.10 scrub ok 2024-09-18T08:07:42.650 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2 2024-09-18T08:07:42.672 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:07:42.673 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:07:42.673 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2' 2024-09-18T08:07:42.774 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3 2024-09-18T08:07:42.782 INFO:teuthology.orchestra.run.smithi007.stdout:osd.0: {} 2024-09-18T08:07:42.783 INFO:teuthology.orchestra.run.smithi007.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-18T08:07:42.796 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:07:42.796 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:07:42.797 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3' 2024-09-18T08:07:42.800 INFO:teuthology.orchestra.run.smithi007.stdout:osd.1: {} 2024-09-18T08:07:42.800 INFO:teuthology.orchestra.run.smithi007.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-18T08:07:42.817 INFO:teuthology.orchestra.run.smithi007.stdout:osd.2: {} 2024-09-18T08:07:42.817 INFO:teuthology.orchestra.run.smithi007.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-18T08:07:42.833 INFO:teuthology.orchestra.run.smithi007.stdout:osd.3: {} 2024-09-18T08:07:42.833 INFO:teuthology.orchestra.run.smithi007.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-18T08:07:42.851 INFO:teuthology.orchestra.run.smithi007.stdout:osd.4: {} 2024-09-18T08:07:42.851 INFO:teuthology.orchestra.run.smithi007.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-18T08:07:42.868 INFO:teuthology.orchestra.run.smithi007.stdout:osd.5: {} 2024-09-18T08:07:42.868 INFO:teuthology.orchestra.run.smithi007.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-18T08:07:42.885 INFO:teuthology.orchestra.run.smithi007.stdout:osd.6: {} 2024-09-18T08:07:42.885 INFO:teuthology.orchestra.run.smithi007.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-18T08:07:42.898 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1 2024-09-18T08:07:42.902 INFO:teuthology.orchestra.run.smithi007.stdout:osd.7: {} 2024-09-18T08:07:42.903 INFO:teuthology.orchestra.run.smithi007.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-18T08:07:42.922 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:07:42.922 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:07:42.923 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1' 2024-09-18T08:07:43.025 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0 2024-09-18T08:07:43.048 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:07:43.048 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:07:43.049 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0' 2024-09-18T08:07:43.150 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2 2024-09-18T08:07:43.172 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:07:43.172 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:07:43.173 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2' 2024-09-18T08:07:43.251 INFO:teuthology.orchestra.run.smithi007.stdout:ERROR: (22) Invalid argument 2024-09-18T08:07:43.251 INFO:teuthology.orchestra.run.smithi007.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-18T08:07:43.273 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3 2024-09-18T08:07:43.282 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:07:43 smithi007 ceph-mon[39249]: 5.13 scrub starts 2024-09-18T08:07:43.282 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:07:43 smithi007 ceph-mon[39249]: 5.13 scrub ok 2024-09-18T08:07:43.282 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:07:43 smithi007 ceph-mon[39249]: from='client.? 172.21.15.7:0/1923956075' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-18T08:07:43.296 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:07:43.297 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:07:43.297 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3' 2024-09-18T08:07:43.398 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4 2024-09-18T08:07:43.425 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:07:43.426 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:07:43.426 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4' 2024-09-18T08:07:43.527 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6 2024-09-18T08:07:43.555 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:07:43.555 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:07:43.555 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6' 2024-09-18T08:07:43.656 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:07:43 smithi050 ceph-mon[32371]: 5.13 scrub starts 2024-09-18T08:07:43.656 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:07:43 smithi050 ceph-mon[32371]: 5.13 scrub ok 2024-09-18T08:07:43.656 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:07:43 smithi050 ceph-mon[32371]: from='client.? 172.21.15.7:0/1923956075' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-18T08:07:43.656 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3 2024-09-18T08:07:43.681 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:07:43.682 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:07:43.682 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3' 2024-09-18T08:07:43.730 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:07:43 smithi007 ceph-mon[29459]: 5.13 scrub starts 2024-09-18T08:07:43.730 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:07:43 smithi007 ceph-mon[29459]: 5.13 scrub ok 2024-09-18T08:07:43.730 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:07:43 smithi007 ceph-mon[29459]: from='client.? 172.21.15.7:0/1923956075' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-18T08:07:43.783 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5 2024-09-18T08:07:43.810 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:07:43.811 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:07:43.811 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5' 2024-09-18T08:07:43.912 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2 2024-09-18T08:07:43.939 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:07:43.940 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:07:43.940 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2' 2024-09-18T08:07:44.041 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4 2024-09-18T08:07:44.068 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:07:44.068 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:07:44.069 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4' 2024-09-18T08:07:44.075 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid 779349ca-7593-11ef-bceb-c7b262605968 -- ceph osd unset nodeep-scrub 2024-09-18T08:07:44.170 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3 2024-09-18T08:07:44.193 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:07:44.194 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:07:44.194 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3' 2024-09-18T08:07:44.295 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1 2024-09-18T08:07:44.317 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:07:44.318 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:07:44.318 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1' 2024-09-18T08:07:44.419 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7 2024-09-18T08:07:44.446 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:07:44.447 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:07:44.447 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7' 2024-09-18T08:07:44.480 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:07:44 smithi007 ceph-mon[29459]: 3.19 scrub starts 2024-09-18T08:07:44.481 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:07:44 smithi007 ceph-mon[29459]: 3.19 scrub ok 2024-09-18T08:07:44.481 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:07:44 smithi007 ceph-mon[29459]: pgmap v224: 158 pgs: 1 peering, 157 active+clean; 7.3 MiB data, 980 MiB used, 714 GiB / 715 GiB avail; 861 B/s rd, 0 B/s wr, 850 op/s 2024-09-18T08:07:44.481 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:07:44 smithi007 ceph-mon[39249]: 3.19 scrub starts 2024-09-18T08:07:44.481 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:07:44 smithi007 ceph-mon[39249]: 3.19 scrub ok 2024-09-18T08:07:44.481 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:07:44 smithi007 ceph-mon[39249]: pgmap v224: 158 pgs: 1 peering, 157 active+clean; 7.3 MiB data, 980 MiB used, 714 GiB / 715 GiB avail; 861 B/s rd, 0 B/s wr, 850 op/s 2024-09-18T08:07:44.548 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4 2024-09-18T08:07:44.573 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:07:44.574 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:07:44.575 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4' 2024-09-18T08:07:44.656 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:07:44 smithi050 ceph-mon[32371]: 3.19 scrub starts 2024-09-18T08:07:44.656 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:07:44 smithi050 ceph-mon[32371]: 3.19 scrub ok 2024-09-18T08:07:44.656 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:07:44 smithi050 ceph-mon[32371]: pgmap v224: 158 pgs: 1 peering, 157 active+clean; 7.3 MiB data, 980 MiB used, 714 GiB / 715 GiB avail; 861 B/s rd, 0 B/s wr, 850 op/s 2024-09-18T08:07:44.676 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2 2024-09-18T08:07:44.700 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:07:44.701 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:07:44.701 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2' 2024-09-18T08:07:44.721 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 7, 6] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-09-18T08:07:44.722 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-18T08:07:44.722 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid 779349ca-7593-11ef-bceb-c7b262605968 -- ceph pg dump --format=json 2024-09-18T08:07:44.802 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2 2024-09-18T08:07:44.824 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:07:44.825 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:07:44.825 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2' 2024-09-18T08:07:44.926 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3 2024-09-18T08:07:44.933 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid 779349ca-7593-11ef-bceb-c7b262605968 -- ceph daemon osd.3 dump_blocked_ops 2024-09-18T08:07:44.952 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:07:44.953 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:07:44.953 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3' 2024-09-18T08:07:45.054 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7 2024-09-18T08:07:45.082 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:07:45.082 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:07:45.083 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7' 2024-09-18T08:07:45.183 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4 2024-09-18T08:07:45.210 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:07:45.211 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:07:45.211 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4' 2024-09-18T08:07:45.312 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1 2024-09-18T08:07:45.335 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:07:45.336 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:07:45.336 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1' 2024-09-18T08:07:45.385 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:07:45 smithi007 ceph-mon[29459]: 3.f scrub starts 2024-09-18T08:07:45.385 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:07:45 smithi007 ceph-mon[29459]: 3.f scrub ok 2024-09-18T08:07:45.386 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:07:45 smithi007 ceph-mon[29459]: 6.b scrub starts 2024-09-18T08:07:45.386 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:07:45 smithi007 ceph-mon[29459]: 6.b scrub ok 2024-09-18T08:07:45.386 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:07:45 smithi007 ceph-mon[39249]: 3.f scrub starts 2024-09-18T08:07:45.386 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:07:45 smithi007 ceph-mon[39249]: 3.f scrub ok 2024-09-18T08:07:45.386 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:07:45 smithi007 ceph-mon[39249]: 6.b scrub starts 2024-09-18T08:07:45.386 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:07:45 smithi007 ceph-mon[39249]: 6.b scrub ok 2024-09-18T08:07:45.437 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7 2024-09-18T08:07:45.464 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:07:45.465 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:07:45.465 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7' 2024-09-18T08:07:45.566 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2 2024-09-18T08:07:45.589 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:07:45.590 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:07:45.590 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2' 2024-09-18T08:07:45.656 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:07:45 smithi050 ceph-mon[32371]: 3.f scrub starts 2024-09-18T08:07:45.656 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:07:45 smithi050 ceph-mon[32371]: 3.f scrub ok 2024-09-18T08:07:45.656 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:07:45 smithi050 ceph-mon[32371]: 6.b scrub starts 2024-09-18T08:07:45.656 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:07:45 smithi050 ceph-mon[32371]: 6.b scrub ok 2024-09-18T08:07:45.690 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7 2024-09-18T08:07:45.718 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:07:45.718 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:07:45.719 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7' 2024-09-18T08:07:45.819 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0 2024-09-18T08:07:45.846 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:07:45.847 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:07:45.847 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0' 2024-09-18T08:07:45.893 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/779349ca-7593-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T08:07:45.894 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/779349ca-7593-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T08:07:45.894 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/779349ca-7593-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T08:07:45.948 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5 2024-09-18T08:07:45.975 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:07:45.975 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:07:45.976 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5' 2024-09-18T08:07:46.077 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0 2024-09-18T08:07:46.099 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:07:46.099 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:07:46.100 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0' 2024-09-18T08:07:46.201 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1 2024-09-18T08:07:46.223 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:07:46.224 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:07:46.224 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1' 2024-09-18T08:07:46.291 INFO:teuthology.orchestra.run.smithi007.stderr:pool 'unique_pool_0' removed 2024-09-18T08:07:46.325 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6 2024-09-18T08:07:46.350 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:07:46.351 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:07:46.351 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6' 2024-09-18T08:07:46.452 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1 2024-09-18T08:07:46.474 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:07:46.475 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:07:46.475 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1' 2024-09-18T08:07:46.545 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:07:46 smithi007 ceph-mon[29459]: 6.0 scrub starts 2024-09-18T08:07:46.545 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:07:46 smithi007 ceph-mon[29459]: 6.0 scrub ok 2024-09-18T08:07:46.545 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:07:46 smithi007 ceph-mon[29459]: pgmap v225: 158 pgs: 158 active+clean; 583 KiB data, 934 MiB used, 714 GiB / 715 GiB avail; 1.4 KiB/s rd, 0 B/s wr, 658 op/s; 15 B/s, 1 objects/s recovering 2024-09-18T08:07:46.545 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:07:46 smithi007 ceph-mon[29459]: from='client.? 172.21.15.7:0/3719279647' entity='client.admin' cmd={"prefix": "osd pool rm", "pool": "unique_pool_0", "pool2": "unique_pool_0", "yes_i_really_really_mean_it": true} : dispatch 2024-09-18T08:07:46.546 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:07:46 smithi007 ceph-mon[39249]: 6.0 scrub starts 2024-09-18T08:07:46.546 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:07:46 smithi007 ceph-mon[39249]: 6.0 scrub ok 2024-09-18T08:07:46.546 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:07:46 smithi007 ceph-mon[39249]: pgmap v225: 158 pgs: 158 active+clean; 583 KiB data, 934 MiB used, 714 GiB / 715 GiB avail; 1.4 KiB/s rd, 0 B/s wr, 658 op/s; 15 B/s, 1 objects/s recovering 2024-09-18T08:07:46.546 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:07:46 smithi007 ceph-mon[39249]: from='client.? 172.21.15.7:0/3719279647' entity='client.admin' cmd={"prefix": "osd pool rm", "pool": "unique_pool_0", "pool2": "unique_pool_0", "yes_i_really_really_mean_it": true} : dispatch 2024-09-18T08:07:46.577 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0 2024-09-18T08:07:46.599 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:07:46.599 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:07:46.600 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0' 2024-09-18T08:07:46.656 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:07:46 smithi050 ceph-mon[32371]: 6.0 scrub starts 2024-09-18T08:07:46.656 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:07:46 smithi050 ceph-mon[32371]: 6.0 scrub ok 2024-09-18T08:07:46.656 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:07:46 smithi050 ceph-mon[32371]: pgmap v225: 158 pgs: 158 active+clean; 583 KiB data, 934 MiB used, 714 GiB / 715 GiB avail; 1.4 KiB/s rd, 0 B/s wr, 658 op/s; 15 B/s, 1 objects/s recovering 2024-09-18T08:07:46.656 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:07:46 smithi050 ceph-mon[32371]: from='client.? 172.21.15.7:0/3719279647' entity='client.admin' cmd={"prefix": "osd pool rm", "pool": "unique_pool_0", "pool2": "unique_pool_0", "yes_i_really_really_mean_it": true} : dispatch 2024-09-18T08:07:46.700 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3 2024-09-18T08:07:46.724 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:07:46.725 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:07:46.725 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3' 2024-09-18T08:07:46.796 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Sep 18 08:07:46 smithi007 ceph-779349ca-7593-11ef-bceb-c7b262605968-mgr-y[29688]: ::ffff:172.21.15.50 - - [18/Sep/2024:08:07:46] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-09-18T08:07:46.826 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1 2024-09-18T08:07:46.848 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:07:46.849 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:07:46.849 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1' 2024-09-18T08:07:46.905 INFO:teuthology.orchestra.run.smithi007.stdout:ERROR: (22) Invalid argument 2024-09-18T08:07:46.905 INFO:teuthology.orchestra.run.smithi007.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-18T08:07:46.950 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3 2024-09-18T08:07:46.972 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:07:46.973 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:07:46.973 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3' 2024-09-18T08:07:47.074 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6 2024-09-18T08:07:47.102 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:07:47.102 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:07:47.103 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6' 2024-09-18T08:07:47.203 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1 2024-09-18T08:07:47.226 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:07:47.226 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:07:47.227 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1' 2024-09-18T08:07:47.307 INFO:teuthology.orchestra.run.smithi007.stderr:nodeep-scrub is unset 2024-09-18T08:07:47.327 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7 2024-09-18T08:07:47.355 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:07:47.355 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:07:47.356 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7' 2024-09-18T08:07:47.457 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4 2024-09-18T08:07:47.480 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:07:47 smithi007 ceph-mon[29459]: 6.a scrub starts 2024-09-18T08:07:47.480 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:07:47 smithi007 ceph-mon[29459]: 6.a scrub ok 2024-09-18T08:07:47.480 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:07:47 smithi007 ceph-mon[29459]: from='client.? 172.21.15.7:0/3719279647' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "unique_pool_0", "pool2": "unique_pool_0", "yes_i_really_really_mean_it": true}]': finished 2024-09-18T08:07:47.480 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:07:47 smithi007 ceph-mon[29459]: osdmap e162: 8 total, 8 up, 8 in 2024-09-18T08:07:47.481 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:07:47 smithi007 ceph-mon[29459]: from='client.? 172.21.15.7:0/1632910072' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-18T08:07:47.481 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:07:47 smithi007 ceph-mon[29459]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-18T08:07:47.481 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:07:47 smithi007 ceph-mon[39249]: 6.a scrub starts 2024-09-18T08:07:47.481 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:07:47 smithi007 ceph-mon[39249]: 6.a scrub ok 2024-09-18T08:07:47.481 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:07:47 smithi007 ceph-mon[39249]: from='client.? 172.21.15.7:0/3719279647' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "unique_pool_0", "pool2": "unique_pool_0", "yes_i_really_really_mean_it": true}]': finished 2024-09-18T08:07:47.481 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:07:47 smithi007 ceph-mon[39249]: osdmap e162: 8 total, 8 up, 8 in 2024-09-18T08:07:47.481 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:07:47 smithi007 ceph-mon[39249]: from='client.? 172.21.15.7:0/1632910072' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-18T08:07:47.481 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:07:47 smithi007 ceph-mon[39249]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-18T08:07:47.483 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:07:47.484 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:07:47.484 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4' 2024-09-18T08:07:47.573 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid 779349ca-7593-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-18T08:07:47.585 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0 2024-09-18T08:07:47.640 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:07:47.640 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:07:47.640 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0' 2024-09-18T08:07:47.656 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:07:47 smithi050 ceph-mon[32371]: 6.a scrub starts 2024-09-18T08:07:47.656 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:07:47 smithi050 ceph-mon[32371]: 6.a scrub ok 2024-09-18T08:07:47.656 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:07:47 smithi050 ceph-mon[32371]: from='client.? 172.21.15.7:0/3719279647' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "unique_pool_0", "pool2": "unique_pool_0", "yes_i_really_really_mean_it": true}]': finished 2024-09-18T08:07:47.656 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:07:47 smithi050 ceph-mon[32371]: osdmap e162: 8 total, 8 up, 8 in 2024-09-18T08:07:47.656 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:07:47 smithi050 ceph-mon[32371]: from='client.? 172.21.15.7:0/1632910072' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-18T08:07:47.656 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:07:47 smithi050 ceph-mon[32371]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-18T08:07:47.741 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2 2024-09-18T08:07:47.763 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:07:47.764 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:07:47.764 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2' 2024-09-18T08:07:47.865 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7 2024-09-18T08:07:47.893 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:07:47.894 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:07:47.894 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7' 2024-09-18T08:07:47.995 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7 2024-09-18T08:07:48.021 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:07:48.021 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:07:48.022 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7' 2024-09-18T08:07:48.123 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7 2024-09-18T08:07:48.168 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:07:48.169 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:07:48.169 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7' 2024-09-18T08:07:48.270 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3 2024-09-18T08:07:48.292 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:07:48.293 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:07:48.294 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3' 2024-09-18T08:07:48.395 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0 2024-09-18T08:07:48.417 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:07:48.418 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:07:48.418 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0' 2024-09-18T08:07:48.519 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2 2024-09-18T08:07:48.541 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:07:48.590 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:07:48.591 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2' 2024-09-18T08:07:48.597 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:07:48 smithi007 ceph-mon[39249]: 5.3 scrub starts 2024-09-18T08:07:48.597 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:07:48 smithi007 ceph-mon[39249]: 5.3 scrub ok 2024-09-18T08:07:48.598 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:07:48 smithi007 ceph-mon[39249]: pgmap v227: 142 pgs: 142 active+clean; 583 KiB data, 934 MiB used, 714 GiB / 715 GiB avail; 518 B/s rd, 0 op/s 2024-09-18T08:07:48.598 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:07:48 smithi007 ceph-mon[39249]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-18T08:07:48.598 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:07:48 smithi007 ceph-mon[39249]: Cluster is now healthy 2024-09-18T08:07:48.598 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:07:48 smithi007 ceph-mon[39249]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-18T08:07:48.598 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:07:48 smithi007 ceph-mon[39249]: osdmap e163: 8 total, 8 up, 8 in 2024-09-18T08:07:48.599 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:07:48 smithi007 ceph-mon[39249]: from='mgr.24470 172.21.15.7:0/2594390261' entity='mgr.y' 2024-09-18T08:07:48.599 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:07:48 smithi007 ceph-mon[39249]: from='mgr.24470 172.21.15.7:0/2594390261' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-18T08:07:48.599 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:07:48 smithi007 ceph-mon[29459]: 5.3 scrub starts 2024-09-18T08:07:48.599 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:07:48 smithi007 ceph-mon[29459]: 5.3 scrub ok 2024-09-18T08:07:48.599 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:07:48 smithi007 ceph-mon[29459]: pgmap v227: 142 pgs: 142 active+clean; 583 KiB data, 934 MiB used, 714 GiB / 715 GiB avail; 518 B/s rd, 0 op/s 2024-09-18T08:07:48.599 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:07:48 smithi007 ceph-mon[29459]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-18T08:07:48.599 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:07:48 smithi007 ceph-mon[29459]: Cluster is now healthy 2024-09-18T08:07:48.600 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:07:48 smithi007 ceph-mon[29459]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-18T08:07:48.600 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:07:48 smithi007 ceph-mon[29459]: osdmap e163: 8 total, 8 up, 8 in 2024-09-18T08:07:48.600 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:07:48 smithi007 ceph-mon[29459]: from='mgr.24470 172.21.15.7:0/2594390261' entity='mgr.y' 2024-09-18T08:07:48.600 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:07:48 smithi007 ceph-mon[29459]: from='mgr.24470 172.21.15.7:0/2594390261' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-18T08:07:48.627 INFO:teuthology.task.full_sequential:In full_sequential, running task radosbench... 2024-09-18T08:07:48.628 INFO:tasks.radosbench:Beginning radosbench... 2024-09-18T08:07:48.629 INFO:tasks.cephadm.ceph_manager.ceph:creating pool_name unique_pool_1 2024-09-18T08:07:48.629 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid 779349ca-7593-11ef-bceb-c7b262605968 -- ceph osd pool create unique_pool_1 16 2024-09-18T08:07:48.656 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:07:48 smithi050 ceph-mon[32371]: 5.3 scrub starts 2024-09-18T08:07:48.656 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:07:48 smithi050 ceph-mon[32371]: 5.3 scrub ok 2024-09-18T08:07:48.656 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:07:48 smithi050 ceph-mon[32371]: pgmap v227: 142 pgs: 142 active+clean; 583 KiB data, 934 MiB used, 714 GiB / 715 GiB avail; 518 B/s rd, 0 op/s 2024-09-18T08:07:48.656 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:07:48 smithi050 ceph-mon[32371]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-18T08:07:48.656 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:07:48 smithi050 ceph-mon[32371]: Cluster is now healthy 2024-09-18T08:07:48.656 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:07:48 smithi050 ceph-mon[32371]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-18T08:07:48.656 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:07:48 smithi050 ceph-mon[32371]: osdmap e163: 8 total, 8 up, 8 in 2024-09-18T08:07:48.656 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:07:48 smithi050 ceph-mon[32371]: from='mgr.24470 172.21.15.7:0/2594390261' entity='mgr.y' 2024-09-18T08:07:48.657 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:07:48 smithi050 ceph-mon[32371]: from='mgr.24470 172.21.15.7:0/2594390261' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-18T08:07:48.693 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2 2024-09-18T08:07:48.735 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:07:48.735 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:07:48.736 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2' 2024-09-18T08:07:48.836 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6 2024-09-18T08:07:48.863 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:07:48.865 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:07:48.865 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6' 2024-09-18T08:07:48.966 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0 2024-09-18T08:07:48.989 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:07:48.989 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:07:48.990 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0' 2024-09-18T08:07:49.090 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1 2024-09-18T08:07:49.113 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:07:49.114 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:07:49.114 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1' 2024-09-18T08:07:49.123 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid 779349ca-7593-11ef-bceb-c7b262605968 -- ceph daemon osd.3 dump_historic_ops 2024-09-18T08:07:49.215 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6 2024-09-18T08:07:49.243 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:07:49.244 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:07:49.244 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6' 2024-09-18T08:07:49.303 INFO:journalctl@ceph.iscsi.iscsi.a.smithi050.stdout:Sep 18 08:07:48 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-iscsi-iscsi-a[57782]: debug there is no tcmu-runner data available 2024-09-18T08:07:49.346 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2 2024-09-18T08:07:49.369 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:07:49.370 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:07:49.371 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2' 2024-09-18T08:07:49.410 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:07:49 smithi007 ceph-mon[39249]: 6.13 scrub starts 2024-09-18T08:07:49.410 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:07:49 smithi007 ceph-mon[39249]: 6.13 scrub ok 2024-09-18T08:07:49.410 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:07:49 smithi007 ceph-mon[39249]: osdmap e164: 8 total, 8 up, 8 in 2024-09-18T08:07:49.411 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:07:49 smithi007 ceph-mon[29459]: 6.13 scrub starts 2024-09-18T08:07:49.411 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:07:49 smithi007 ceph-mon[29459]: 6.13 scrub ok 2024-09-18T08:07:49.411 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:07:49 smithi007 ceph-mon[29459]: osdmap e164: 8 total, 8 up, 8 in 2024-09-18T08:07:49.472 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2 2024-09-18T08:07:49.499 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:07:49.499 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:07:49.500 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2' 2024-09-18T08:07:49.601 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2 2024-09-18T08:07:49.608 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-09-18T08:07:49.610 INFO:teuthology.orchestra.run.smithi007.stderr:dumped all 2024-09-18T08:07:49.625 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:07:49.626 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:07:49.626 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2' 2024-09-18T08:07:49.656 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:07:49 smithi050 ceph-mon[32371]: 6.13 scrub starts 2024-09-18T08:07:49.656 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:07:49 smithi050 ceph-mon[32371]: 6.13 scrub ok 2024-09-18T08:07:49.656 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:07:49 smithi050 ceph-mon[32371]: osdmap e164: 8 total, 8 up, 8 in 2024-09-18T08:07:49.728 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5 2024-09-18T08:07:49.755 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:07:49.756 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:07:49.756 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5' 2024-09-18T08:07:49.857 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0 2024-09-18T08:07:49.880 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:07:49.881 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:07:49.881 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0' 2024-09-18T08:07:49.981 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7 2024-09-18T08:07:50.008 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:07:50.009 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:07:50.009 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7' 2024-09-18T08:07:50.110 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5 2024-09-18T08:07:50.136 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:07:50.137 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:07:50.137 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5' 2024-09-18T08:07:50.169 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/779349ca-7593-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T08:07:50.176 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/779349ca-7593-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T08:07:50.238 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4 2024-09-18T08:07:50.263 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:07:50.264 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:07:50.264 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4' 2024-09-18T08:07:50.365 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1 2024-09-18T08:07:50.388 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:07:50.389 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:07:50.389 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1' 2024-09-18T08:07:50.490 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7 2024-09-18T08:07:50.514 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:07:50.515 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:07:50.515 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7' 2024-09-18T08:07:50.616 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4 2024-09-18T08:07:50.641 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:07:50.641 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:07:50.642 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4' 2024-09-18T08:07:50.656 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:07:50 smithi050 ceph-mon[32371]: 3.12 scrub starts 2024-09-18T08:07:50.656 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:07:50 smithi050 ceph-mon[32371]: 3.12 scrub ok 2024-09-18T08:07:50.656 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:07:50 smithi050 ceph-mon[32371]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-18T08:07:50.656 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:07:50 smithi050 ceph-mon[32371]: 6.1 scrub starts 2024-09-18T08:07:50.656 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:07:50 smithi050 ceph-mon[32371]: 6.1 scrub ok 2024-09-18T08:07:50.656 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:07:50 smithi050 ceph-mon[32371]: pgmap v230: 142 pgs: 142 active+clean; 583 KiB data, 952 MiB used, 714 GiB / 715 GiB avail; 1.5 KiB/s rd, 1 op/s 2024-09-18T08:07:50.660 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:07:50 smithi007 ceph-mon[29459]: 3.12 scrub starts 2024-09-18T08:07:50.660 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:07:50 smithi007 ceph-mon[29459]: 3.12 scrub ok 2024-09-18T08:07:50.660 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:07:50 smithi007 ceph-mon[29459]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-18T08:07:50.660 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:07:50 smithi007 ceph-mon[29459]: 6.1 scrub starts 2024-09-18T08:07:50.660 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:07:50 smithi007 ceph-mon[29459]: 6.1 scrub ok 2024-09-18T08:07:50.660 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:07:50 smithi007 ceph-mon[29459]: pgmap v230: 142 pgs: 142 active+clean; 583 KiB data, 952 MiB used, 714 GiB / 715 GiB avail; 1.5 KiB/s rd, 1 op/s 2024-09-18T08:07:50.660 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:07:50 smithi007 ceph-mon[39249]: 3.12 scrub starts 2024-09-18T08:07:50.660 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:07:50 smithi007 ceph-mon[39249]: 3.12 scrub ok 2024-09-18T08:07:50.661 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:07:50 smithi007 ceph-mon[39249]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-18T08:07:50.661 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:07:50 smithi007 ceph-mon[39249]: 6.1 scrub starts 2024-09-18T08:07:50.661 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:07:50 smithi007 ceph-mon[39249]: 6.1 scrub ok 2024-09-18T08:07:50.661 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:07:50 smithi007 ceph-mon[39249]: pgmap v230: 142 pgs: 142 active+clean; 583 KiB data, 952 MiB used, 714 GiB / 715 GiB avail; 1.5 KiB/s rd, 1 op/s 2024-09-18T08:07:50.667 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/779349ca-7593-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T08:07:50.693 INFO:teuthology.orchestra.run.smithi007.stdout:{"pg_ready":true,"pg_map":{"version":230,"stamp":"2024-09-18T08:07:49.167092+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":596507,"num_objects":199,"num_object_clones":0,"num_object_copies":597,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":199,"num_whiteouts":0,"num_read":901,"num_read_kb":761,"num_write":549,"num_write_kb":1673,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":168,"num_bytes_recovered":1398071,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1352,"ondisk_log_size":1352,"up":426,"acting":426,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":462,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":974440,"kb_used_data":8588,"kb_used_omap":12,"kb_used_meta":965683,"kb_avail":748986776,"statfs":{"total":767960285184,"available":766962458624,"internally_reserved":0,"allocated":8794112,"data_stored":5408501,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12750,"internal_metadata":988859954},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":76,"apply_latency_ms":76,"commit_latency_ns":76000000,"apply_latency_ns":76000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-7077912,"num_objects":-109,"num_object_clones":0,"num_object_copies":-327,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-109,"num_whiteouts":0,"num_read":9,"num_read_kb":9,"num_write":-236469,"num_write_kb":-7570433,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":-2622,"num_bytes_recovered":-172097426,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.001626"},"pg_stats":[{"pgid":"3.1f","version":"0'0","reported_seq":223,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-09-18T08:07:48.065840+0000","last_change":"2024-09-18T08:04:34.226236+0000","last_active":"2024-09-18T08:07:48.065840+0000","last_peered":"2024-09-18T08:07:48.065840+0000","last_clean":"2024-09-18T08:07:48.065840+0000","last_became_active":"2024-09-18T08:02:47.877698+0000","last_became_peered":"2024-09-18T08:02:47.877698+0000","last_unstale":"2024-09-18T08:07:48.065840+0000","last_undegraded":"2024-09-18T08:07:48.065840+0000","last_fullsized":"2024-09-18T08:07:48.065840+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:04:34.226054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:46.860092+0000","last_clean_scrub_stamp":"2024-09-18T08:04:34.226054+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:07:54.705201+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"69'9","reported_seq":270,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-09-18T08:07:47.300524+0000","last_change":"2024-09-18T08:06:24.171122+0000","last_active":"2024-09-18T08:07:47.300524+0000","last_peered":"2024-09-18T08:07:47.300524+0000","last_clean":"2024-09-18T08:07:47.300524+0000","last_became_active":"2024-09-18T08:06:24.140103+0000","last_became_peered":"2024-09-18T08:06:24.140103+0000","last_unstale":"2024-09-18T08:07:47.300524+0000","last_undegraded":"2024-09-18T08:07:47.300524+0000","last_fullsized":"2024-09-18T08:07:47.300524+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'9","last_scrub_stamp":"2024-09-18T08:05:08.696796+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_clean_scrub_stamp":"2024-09-18T08:05:08.696796+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:08:01.568014+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":231,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-09-18T08:07:47.300707+0000","last_change":"2024-09-18T08:07:18.737364+0000","last_active":"2024-09-18T08:07:47.300707+0000","last_peered":"2024-09-18T08:07:47.300707+0000","last_clean":"2024-09-18T08:07:47.300707+0000","last_became_active":"2024-09-18T08:02:51.907140+0000","last_became_peered":"2024-09-18T08:02:51.907140+0000","last_unstale":"2024-09-18T08:07:47.300707+0000","last_undegraded":"2024-09-18T08:07:47.300707+0000","last_fullsized":"2024-09-18T08:07:47.300707+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:07:18.737122+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:50.886830+0000","last_clean_scrub_stamp":"2024-09-18T08:07:18.737122+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:08:28.692935+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":212,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-09-18T08:07:47.300722+0000","last_change":"2024-09-18T08:07:08.451937+0000","last_active":"2024-09-18T08:07:47.300722+0000","last_peered":"2024-09-18T08:07:47.300722+0000","last_clean":"2024-09-18T08:07:47.300722+0000","last_became_active":"2024-09-18T08:02:53.906622+0000","last_became_peered":"2024-09-18T08:02:53.906622+0000","last_unstale":"2024-09-18T08:07:47.300722+0000","last_undegraded":"2024-09-18T08:07:47.300722+0000","last_fullsized":"2024-09-18T08:07:47.300722+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:07:08.451865+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_clean_scrub_stamp":"2024-09-18T08:07:08.451865+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:08:20.709312+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":233,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-09-18T08:07:48.103235+0000","last_change":"2024-09-18T08:07:13.244647+0000","last_active":"2024-09-18T08:07:48.103235+0000","last_peered":"2024-09-18T08:07:48.103235+0000","last_clean":"2024-09-18T08:07:48.103235+0000","last_became_active":"2024-09-18T08:06:24.128511+0000","last_became_peered":"2024-09-18T08:06:24.128511+0000","last_unstale":"2024-09-18T08:07:48.103235+0000","last_undegraded":"2024-09-18T08:07:48.103235+0000","last_fullsized":"2024-09-18T08:07:48.103235+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:07:13.244555+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_clean_scrub_stamp":"2024-09-18T08:07:13.244555+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:08:18.290204+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":246,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-09-18T08:07:48.103465+0000","last_change":"2024-09-18T08:07:10.257236+0000","last_active":"2024-09-18T08:07:48.103465+0000","last_peered":"2024-09-18T08:07:48.103465+0000","last_clean":"2024-09-18T08:07:48.103465+0000","last_became_active":"2024-09-18T08:06:24.126194+0000","last_became_peered":"2024-09-18T08:06:24.126194+0000","last_unstale":"2024-09-18T08:07:48.103465+0000","last_undegraded":"2024-09-18T08:07:48.103465+0000","last_fullsized":"2024-09-18T08:07:48.103465+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:07:10.257170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:46.860092+0000","last_clean_scrub_stamp":"2024-09-18T08:07:10.257170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:08:28.333021+0000","scrub_duration":37,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"69'15","reported_seq":237,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-09-18T08:07:48.102949+0000","last_change":"2024-09-18T08:02:49.887111+0000","last_active":"2024-09-18T08:07:48.102949+0000","last_peered":"2024-09-18T08:07:48.102949+0000","last_clean":"2024-09-18T08:07:48.102949+0000","last_became_active":"2024-09-18T08:02:49.886106+0000","last_became_peered":"2024-09-18T08:02:49.886106+0000","last_unstale":"2024-09-18T08:07:48.102949+0000","last_undegraded":"2024-09-18T08:07:48.102949+0000","last_fullsized":"2024-09-18T08:07:48.102949+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_clean_scrub_stamp":"2024-09-18T08:02:48.870718+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:07:55.263331+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":230,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-09-18T08:07:47.300521+0000","last_change":"2024-09-18T08:06:24.146536+0000","last_active":"2024-09-18T08:07:47.300521+0000","last_peered":"2024-09-18T08:07:47.300521+0000","last_clean":"2024-09-18T08:07:47.300521+0000","last_became_active":"2024-09-18T08:06:24.144841+0000","last_became_peered":"2024-09-18T08:06:24.144841+0000","last_unstale":"2024-09-18T08:07:47.300521+0000","last_undegraded":"2024-09-18T08:07:47.300521+0000","last_fullsized":"2024-09-18T08:07:47.300521+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:04:29.435669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:50.886830+0000","last_clean_scrub_stamp":"2024-09-18T08:04:29.435669+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:08:37.651634+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":245,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-09-18T08:07:48.114419+0000","last_change":"2024-09-18T08:06:24.132542+0000","last_active":"2024-09-18T08:07:48.114419+0000","last_peered":"2024-09-18T08:07:48.114419+0000","last_clean":"2024-09-18T08:07:48.114419+0000","last_became_active":"2024-09-18T08:06:24.132206+0000","last_became_peered":"2024-09-18T08:06:24.132206+0000","last_unstale":"2024-09-18T08:07:48.114419+0000","last_undegraded":"2024-09-18T08:07:48.114419+0000","last_fullsized":"2024-09-18T08:07:48.114419+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:06:14.027077+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:46.860092+0000","last_clean_scrub_stamp":"2024-09-18T08:06:14.027077+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"69'9","reported_seq":227,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-09-18T08:07:47.300921+0000","last_change":"2024-09-18T08:02:49.887589+0000","last_active":"2024-09-18T08:07:47.300921+0000","last_peered":"2024-09-18T08:07:47.300921+0000","last_clean":"2024-09-18T08:07:47.300921+0000","last_became_active":"2024-09-18T08:02:49.887161+0000","last_became_peered":"2024-09-18T08:02:49.887161+0000","last_unstale":"2024-09-18T08:07:47.300921+0000","last_undegraded":"2024-09-18T08:07:47.300921+0000","last_fullsized":"2024-09-18T08:07:47.300921+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_clean_scrub_stamp":"2024-09-18T08:02:48.870718+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:07:59.552425+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":222,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-09-18T08:07:48.105251+0000","last_change":"2024-09-18T08:05:33.935844+0000","last_active":"2024-09-18T08:07:48.105251+0000","last_peered":"2024-09-18T08:07:48.105251+0000","last_clean":"2024-09-18T08:07:48.105251+0000","last_became_active":"2024-09-18T08:02:51.907296+0000","last_became_peered":"2024-09-18T08:02:51.907296+0000","last_unstale":"2024-09-18T08:07:48.105251+0000","last_undegraded":"2024-09-18T08:07:48.105251+0000","last_fullsized":"2024-09-18T08:07:48.105251+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:05:33.935798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:50.886830+0000","last_clean_scrub_stamp":"2024-09-18T08:05:33.935798+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:07:55.907350+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":212,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-09-18T08:07:48.060147+0000","last_change":"2024-09-18T08:05:01.314402+0000","last_active":"2024-09-18T08:07:48.060147+0000","last_peered":"2024-09-18T08:07:48.060147+0000","last_clean":"2024-09-18T08:07:48.060147+0000","last_became_active":"2024-09-18T08:02:53.913519+0000","last_became_peered":"2024-09-18T08:02:53.913519+0000","last_unstale":"2024-09-18T08:07:48.060147+0000","last_undegraded":"2024-09-18T08:07:48.060147+0000","last_fullsized":"2024-09-18T08:07:48.060147+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:05:01.314330+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_clean_scrub_stamp":"2024-09-18T08:05:01.314330+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:07:56.780219+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":223,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-09-18T08:07:48.104978+0000","last_change":"2024-09-18T08:04:32.849176+0000","last_active":"2024-09-18T08:07:48.104978+0000","last_peered":"2024-09-18T08:07:48.104978+0000","last_clean":"2024-09-18T08:07:48.104978+0000","last_became_active":"2024-09-18T08:02:47.875042+0000","last_became_peered":"2024-09-18T08:02:47.875042+0000","last_unstale":"2024-09-18T08:07:48.104978+0000","last_undegraded":"2024-09-18T08:07:48.104978+0000","last_fullsized":"2024-09-18T08:07:48.104978+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:04:32.849119+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:46.860092+0000","last_clean_scrub_stamp":"2024-09-18T08:04:32.849119+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:08:06.853060+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1b","version":"69'5","reported_seq":237,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-09-18T08:07:47.301003+0000","last_change":"2024-09-18T08:06:09.466075+0000","last_active":"2024-09-18T08:07:47.301003+0000","last_peered":"2024-09-18T08:07:47.301003+0000","last_clean":"2024-09-18T08:07:47.301003+0000","last_became_active":"2024-09-18T08:02:49.895344+0000","last_became_peered":"2024-09-18T08:02:49.895344+0000","last_unstale":"2024-09-18T08:07:47.301003+0000","last_undegraded":"2024-09-18T08:07:47.301003+0000","last_fullsized":"2024-09-18T08:07:47.301003+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'5","last_scrub_stamp":"2024-09-18T08:06:09.465979+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_clean_scrub_stamp":"2024-09-18T08:06:09.465979+0000","objects_scrubbed":2,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":11,"num_read_kb":7,"num_write":6,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":223,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-09-18T08:07:48.103955+0000","last_change":"2024-09-18T08:05:38.866695+0000","last_active":"2024-09-18T08:07:48.103955+0000","last_peered":"2024-09-18T08:07:48.103955+0000","last_clean":"2024-09-18T08:07:48.103955+0000","last_became_active":"2024-09-18T08:02:51.905776+0000","last_became_peered":"2024-09-18T08:02:51.905776+0000","last_unstale":"2024-09-18T08:07:48.103955+0000","last_undegraded":"2024-09-18T08:07:48.103955+0000","last_fullsized":"2024-09-18T08:07:48.103955+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:05:38.866651+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:50.886830+0000","last_clean_scrub_stamp":"2024-09-18T08:05:38.866651+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:07:53.746353+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":220,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-09-18T08:07:48.102382+0000","last_change":"2024-09-18T08:06:44.518882+0000","last_active":"2024-09-18T08:07:48.102382+0000","last_peered":"2024-09-18T08:07:48.102382+0000","last_clean":"2024-09-18T08:07:48.102382+0000","last_became_active":"2024-09-18T08:02:53.911648+0000","last_became_peered":"2024-09-18T08:02:53.911648+0000","last_unstale":"2024-09-18T08:07:48.102382+0000","last_undegraded":"2024-09-18T08:07:48.102382+0000","last_fullsized":"2024-09-18T08:07:48.102382+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:06:44.518835+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_clean_scrub_stamp":"2024-09-18T08:06:44.518835+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:08:04.920034+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":226,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-09-18T08:07:47.301298+0000","last_change":"2024-09-18T08:06:24.132553+0000","last_active":"2024-09-18T08:07:47.301298+0000","last_peered":"2024-09-18T08:07:47.301298+0000","last_clean":"2024-09-18T08:07:47.301298+0000","last_became_active":"2024-09-18T08:06:24.132228+0000","last_became_peered":"2024-09-18T08:06:24.132228+0000","last_unstale":"2024-09-18T08:07:47.301298+0000","last_undegraded":"2024-09-18T08:07:47.301298+0000","last_fullsized":"2024-09-18T08:07:47.301298+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:05:40.477965+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_clean_scrub_stamp":"2024-09-18T08:05:40.477965+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":231,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-09-18T08:07:48.060967+0000","last_change":"2024-09-18T08:05:27.476211+0000","last_active":"2024-09-18T08:07:48.060967+0000","last_peered":"2024-09-18T08:07:48.060967+0000","last_clean":"2024-09-18T08:07:48.060967+0000","last_became_active":"2024-09-18T08:02:47.875963+0000","last_became_peered":"2024-09-18T08:02:47.875963+0000","last_unstale":"2024-09-18T08:07:48.060967+0000","last_undegraded":"2024-09-18T08:07:48.060967+0000","last_fullsized":"2024-09-18T08:07:48.060967+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:05:27.476144+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:46.860092+0000","last_clean_scrub_stamp":"2024-09-18T08:05:27.476144+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:08:32.614667+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1c","version":"69'15","reported_seq":306,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-09-18T08:07:47.306376+0000","last_change":"2024-09-18T08:07:39.530425+0000","last_active":"2024-09-18T08:07:47.306376+0000","last_peered":"2024-09-18T08:07:47.306376+0000","last_clean":"2024-09-18T08:07:47.306376+0000","last_became_active":"2024-09-18T08:07:39.391999+0000","last_became_peered":"2024-09-18T08:07:39.391999+0000","last_unstale":"2024-09-18T08:07:47.306376+0000","last_undegraded":"2024-09-18T08:07:47.306376+0000","last_fullsized":"2024-09-18T08:07:47.306376+0000","mapping_epoch":159,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'15","last_scrub_stamp":"2024-09-18T08:07:39.530340+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_clean_scrub_stamp":"2024-09-18T08:07:39.530340+0000","objects_scrubbed":7,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:09:06.869426+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":230,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-09-18T08:07:47.303595+0000","last_change":"2024-09-18T08:07:08.743521+0000","last_active":"2024-09-18T08:07:47.303595+0000","last_peered":"2024-09-18T08:07:47.303595+0000","last_clean":"2024-09-18T08:07:47.303595+0000","last_became_active":"2024-09-18T08:02:51.908430+0000","last_became_peered":"2024-09-18T08:02:51.908430+0000","last_unstale":"2024-09-18T08:07:47.303595+0000","last_undegraded":"2024-09-18T08:07:47.303595+0000","last_fullsized":"2024-09-18T08:07:47.303595+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:07:08.743455+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:50.886830+0000","last_clean_scrub_stamp":"2024-09-18T08:07:08.743455+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:08:19.243807+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":226,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-09-18T08:07:48.103793+0000","last_change":"2024-09-18T08:06:24.129031+0000","last_active":"2024-09-18T08:07:48.103793+0000","last_peered":"2024-09-18T08:07:48.103793+0000","last_clean":"2024-09-18T08:07:48.103793+0000","last_became_active":"2024-09-18T08:06:24.128659+0000","last_became_peered":"2024-09-18T08:06:24.128659+0000","last_unstale":"2024-09-18T08:07:48.103793+0000","last_undegraded":"2024-09-18T08:07:48.103793+0000","last_fullsized":"2024-09-18T08:07:48.103793+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:04:29.955822+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_clean_scrub_stamp":"2024-09-18T08:04:29.955822+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:08:05.214492+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":232,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-09-18T08:07:47.301103+0000","last_change":"2024-09-18T08:07:13.543787+0000","last_active":"2024-09-18T08:07:47.301103+0000","last_peered":"2024-09-18T08:07:47.301103+0000","last_clean":"2024-09-18T08:07:47.301103+0000","last_became_active":"2024-09-18T08:02:47.871523+0000","last_became_peered":"2024-09-18T08:02:47.871523+0000","last_unstale":"2024-09-18T08:07:47.301103+0000","last_undegraded":"2024-09-18T08:07:47.301103+0000","last_fullsized":"2024-09-18T08:07:47.301103+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:07:13.543722+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:46.860092+0000","last_clean_scrub_stamp":"2024-09-18T08:07:13.543722+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:08:32.834510+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"69'12","reported_seq":250,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-09-18T08:07:48.103548+0000","last_change":"2024-09-18T08:07:20.251611+0000","last_active":"2024-09-18T08:07:48.103548+0000","last_peered":"2024-09-18T08:07:48.103548+0000","last_clean":"2024-09-18T08:07:48.103548+0000","last_became_active":"2024-09-18T08:02:49.888831+0000","last_became_peered":"2024-09-18T08:02:49.888831+0000","last_unstale":"2024-09-18T08:07:48.103548+0000","last_undegraded":"2024-09-18T08:07:48.103548+0000","last_fullsized":"2024-09-18T08:07:48.103548+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'12","last_scrub_stamp":"2024-09-18T08:07:20.251529+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_clean_scrub_stamp":"2024-09-18T08:07:20.251529+0000","objects_scrubbed":5,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:08:42.250683+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":25,"num_read_kb":16,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":223,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-09-18T08:07:47.301298+0000","last_change":"2024-09-18T08:06:12.434234+0000","last_active":"2024-09-18T08:07:47.301298+0000","last_peered":"2024-09-18T08:07:47.301298+0000","last_clean":"2024-09-18T08:07:47.301298+0000","last_became_active":"2024-09-18T08:02:51.910316+0000","last_became_peered":"2024-09-18T08:02:51.910316+0000","last_unstale":"2024-09-18T08:07:47.301298+0000","last_undegraded":"2024-09-18T08:07:47.301298+0000","last_fullsized":"2024-09-18T08:07:47.301298+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:06:12.434179+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:50.886830+0000","last_clean_scrub_stamp":"2024-09-18T08:06:12.434179+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1c","version":"69'1","reported_seq":62,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-09-18T08:07:48.104724+0000","last_change":"2024-09-18T08:06:46.142143+0000","last_active":"2024-09-18T08:07:48.104724+0000","last_peered":"2024-09-18T08:07:48.104724+0000","last_clean":"2024-09-18T08:07:48.104724+0000","last_became_active":"2024-09-18T08:06:46.141661+0000","last_became_peered":"2024-09-18T08:06:46.141661+0000","last_unstale":"2024-09-18T08:07:48.104724+0000","last_undegraded":"2024-09-18T08:07:48.104724+0000","last_fullsized":"2024-09-18T08:07:48.104724+0000","mapping_epoch":143,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":144,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'1","last_scrub_stamp":"2024-09-18T08:05:36.776369+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_clean_scrub_stamp":"2024-09-18T08:05:36.776369+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":403,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,2],"acting":[5,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"62'1","reported_seq":232,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-09-18T08:07:47.300965+0000","last_change":"2024-09-18T08:07:43.066011+0000","last_active":"2024-09-18T08:07:47.300965+0000","last_peered":"2024-09-18T08:07:47.300965+0000","last_clean":"2024-09-18T08:07:47.300965+0000","last_became_active":"2024-09-18T08:02:47.880998+0000","last_became_peered":"2024-09-18T08:02:47.880998+0000","last_unstale":"2024-09-18T08:07:47.300965+0000","last_undegraded":"2024-09-18T08:07:47.300965+0000","last_fullsized":"2024-09-18T08:07:47.300965+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"62'1","last_scrub_stamp":"2024-09-18T08:07:43.065962+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:46.860092+0000","last_clean_scrub_stamp":"2024-09-18T08:07:43.065962+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:08:50.443599+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1e","version":"69'10","reported_seq":226,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-09-18T08:07:48.063553+0000","last_change":"2024-09-18T08:02:49.892755+0000","last_active":"2024-09-18T08:07:48.063553+0000","last_peered":"2024-09-18T08:07:48.063553+0000","last_clean":"2024-09-18T08:07:48.063553+0000","last_became_active":"2024-09-18T08:02:49.892024+0000","last_became_peered":"2024-09-18T08:02:49.892024+0000","last_unstale":"2024-09-18T08:07:48.063553+0000","last_undegraded":"2024-09-18T08:07:48.063553+0000","last_fullsized":"2024-09-18T08:07:48.063553+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_clean_scrub_stamp":"2024-09-18T08:02:48.870718+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:08:01.836462+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1f","version":"138'11","reported_seq":234,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-09-18T08:07:47.301926+0000","last_change":"2024-09-18T08:06:24.180060+0000","last_active":"2024-09-18T08:07:47.301926+0000","last_peered":"2024-09-18T08:07:47.301926+0000","last_clean":"2024-09-18T08:07:47.301926+0000","last_became_active":"2024-09-18T08:06:24.145360+0000","last_became_peered":"2024-09-18T08:06:24.145360+0000","last_unstale":"2024-09-18T08:07:47.301926+0000","last_undegraded":"2024-09-18T08:07:47.301926+0000","last_fullsized":"2024-09-18T08:07:47.301926+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"102'9","last_scrub_stamp":"2024-09-18T08:05:44.463780+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:50.886830+0000","last_clean_scrub_stamp":"2024-09-18T08:05:44.463780+0000","objects_scrubbed":1,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,7],"acting":[4,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.a","version":"74'79","reported_seq":30,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-09-18T08:07:48.104128+0000","last_change":"2024-09-18T08:07:28.117214+0000","last_active":"2024-09-18T08:07:48.104128+0000","last_peered":"2024-09-18T08:07:48.104128+0000","last_clean":"2024-09-18T08:07:48.104128+0000","last_became_active":"2024-09-18T08:07:26.394584+0000","last_became_peered":"2024-09-18T08:07:26.394584+0000","last_unstale":"2024-09-18T08:07:48.104128+0000","last_undegraded":"2024-09-18T08:07:48.104128+0000","last_fullsized":"2024-09-18T08:07:48.104128+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":152,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":4,"last_scrub":"74'79","last_scrub_stamp":"2024-09-18T08:04:33.791841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:00:11.893742+0000","last_clean_scrub_stamp":"2024-09-18T08:04:33.791841+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"user requested scrub scheduled @ 2024-09-18T08:07:56.655105+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":53669,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":10,"num_read_kb":20,"num_write":10,"num_write_kb":148,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":53669,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,6],"acting":[5,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.f","version":"69'15","reported_seq":244,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-09-18T08:07:47.301054+0000","last_change":"2024-09-18T08:04:22.741342+0000","last_active":"2024-09-18T08:07:47.301054+0000","last_peered":"2024-09-18T08:07:47.301054+0000","last_clean":"2024-09-18T08:07:47.301054+0000","last_became_active":"2024-09-18T08:02:49.899083+0000","last_became_peered":"2024-09-18T08:02:49.899083+0000","last_unstale":"2024-09-18T08:07:47.301054+0000","last_undegraded":"2024-09-18T08:07:47.301054+0000","last_fullsized":"2024-09-18T08:07:47.301054+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'15","last_scrub_stamp":"2024-09-18T08:04:22.741261+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_clean_scrub_stamp":"2024-09-18T08:04:22.741261+0000","objects_scrubbed":7,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:07:56.574487+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":224,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-09-18T08:07:48.102455+0000","last_change":"2024-09-18T08:04:08.982535+0000","last_active":"2024-09-18T08:07:48.102455+0000","last_peered":"2024-09-18T08:07:48.102455+0000","last_clean":"2024-09-18T08:07:48.102455+0000","last_became_active":"2024-09-18T08:02:47.879328+0000","last_became_peered":"2024-09-18T08:02:47.879328+0000","last_unstale":"2024-09-18T08:07:48.102455+0000","last_undegraded":"2024-09-18T08:07:48.102455+0000","last_fullsized":"2024-09-18T08:07:48.102455+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:04:08.982469+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:46.860092+0000","last_clean_scrub_stamp":"2024-09-18T08:04:08.982469+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:07:59.169101+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.e","version":"69'8","reported_seq":293,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-09-18T08:07:47.301520+0000","last_change":"2024-09-18T08:06:19.459452+0000","last_active":"2024-09-18T08:07:47.301520+0000","last_peered":"2024-09-18T08:07:47.301520+0000","last_clean":"2024-09-18T08:07:47.301520+0000","last_became_active":"2024-09-18T08:02:51.903369+0000","last_became_peered":"2024-09-18T08:02:51.903369+0000","last_unstale":"2024-09-18T08:07:47.301520+0000","last_undegraded":"2024-09-18T08:07:47.301520+0000","last_fullsized":"2024-09-18T08:07:47.301520+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'8","last_scrub_stamp":"2024-09-18T08:06:19.459375+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:50.886830+0000","last_clean_scrub_stamp":"2024-09-18T08:06:19.459375+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":228,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-09-18T08:07:48.104577+0000","last_change":"2024-09-18T08:07:19.714477+0000","last_active":"2024-09-18T08:07:48.104577+0000","last_peered":"2024-09-18T08:07:48.104577+0000","last_clean":"2024-09-18T08:07:48.104577+0000","last_became_active":"2024-09-18T08:02:53.909834+0000","last_became_peered":"2024-09-18T08:02:53.909834+0000","last_unstale":"2024-09-18T08:07:48.104577+0000","last_undegraded":"2024-09-18T08:07:48.104577+0000","last_fullsized":"2024-09-18T08:07:48.104577+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:07:19.714430+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_clean_scrub_stamp":"2024-09-18T08:07:19.714430+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:08:34.940835+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.5","version":"74'79","reported_seq":31,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-09-18T08:07:47.301509+0000","last_change":"2024-09-18T08:07:28.133248+0000","last_active":"2024-09-18T08:07:47.301509+0000","last_peered":"2024-09-18T08:07:47.301509+0000","last_clean":"2024-09-18T08:07:47.301509+0000","last_became_active":"2024-09-18T08:07:26.394760+0000","last_became_peered":"2024-09-18T08:07:26.394760+0000","last_unstale":"2024-09-18T08:07:47.301509+0000","last_undegraded":"2024-09-18T08:07:47.301509+0000","last_fullsized":"2024-09-18T08:07:47.301509+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":152,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":3,"last_scrub":"74'79","last_scrub_stamp":"2024-09-18T08:04:33.791841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:00:11.893742+0000","last_clean_scrub_stamp":"2024-09-18T08:04:33.791841+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"user requested scrub scheduled @ 2024-09-18T08:07:56.586989+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":53670,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":10,"num_read_kb":20,"num_write":10,"num_write_kb":148,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":53670,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,6],"acting":[4,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.0","version":"69'18","reported_seq":260,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-09-18T08:07:48.102252+0000","last_change":"2024-09-18T08:06:44.301130+0000","last_active":"2024-09-18T08:07:48.102252+0000","last_peered":"2024-09-18T08:07:48.102252+0000","last_clean":"2024-09-18T08:07:48.102252+0000","last_became_active":"2024-09-18T08:02:49.886818+0000","last_became_peered":"2024-09-18T08:02:49.886818+0000","last_unstale":"2024-09-18T08:07:48.102252+0000","last_undegraded":"2024-09-18T08:07:48.102252+0000","last_fullsized":"2024-09-18T08:07:48.102252+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'18","last_scrub_stamp":"2024-09-18T08:06:44.300772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_clean_scrub_stamp":"2024-09-18T08:06:44.300772+0000","objects_scrubbed":8,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:07:52.014378+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":34,"num_read_kb":22,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":240,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-09-18T08:07:48.104091+0000","last_change":"2024-09-18T08:07:21.242587+0000","last_active":"2024-09-18T08:07:48.104091+0000","last_peered":"2024-09-18T08:07:48.104091+0000","last_clean":"2024-09-18T08:07:48.104091+0000","last_became_active":"2024-09-18T08:02:47.880225+0000","last_became_peered":"2024-09-18T08:02:47.880225+0000","last_unstale":"2024-09-18T08:07:48.104091+0000","last_undegraded":"2024-09-18T08:07:48.104091+0000","last_fullsized":"2024-09-18T08:07:48.104091+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:07:21.242494+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:46.860092+0000","last_clean_scrub_stamp":"2024-09-18T08:07:21.242494+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:08:29.416879+0000","scrub_duration":18,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":224,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-09-18T08:07:47.300546+0000","last_change":"2024-09-18T08:06:10.468172+0000","last_active":"2024-09-18T08:07:47.300546+0000","last_peered":"2024-09-18T08:07:47.300546+0000","last_clean":"2024-09-18T08:07:47.300546+0000","last_became_active":"2024-09-18T08:02:51.899594+0000","last_became_peered":"2024-09-18T08:02:51.899594+0000","last_unstale":"2024-09-18T08:07:47.300546+0000","last_undegraded":"2024-09-18T08:07:47.300546+0000","last_fullsized":"2024-09-18T08:07:47.300546+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:06:10.468119+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:50.886830+0000","last_clean_scrub_stamp":"2024-09-18T08:06:10.468119+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":213,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-09-18T08:07:47.300872+0000","last_change":"2024-09-18T08:04:16.427470+0000","last_active":"2024-09-18T08:07:47.300872+0000","last_peered":"2024-09-18T08:07:47.300872+0000","last_clean":"2024-09-18T08:07:47.300872+0000","last_became_active":"2024-09-18T08:02:53.903957+0000","last_became_peered":"2024-09-18T08:02:53.903957+0000","last_unstale":"2024-09-18T08:07:47.300872+0000","last_undegraded":"2024-09-18T08:07:47.300872+0000","last_fullsized":"2024-09-18T08:07:47.300872+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:04:16.427390+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_clean_scrub_stamp":"2024-09-18T08:04:16.427390+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:08:27.625416+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.4","version":"74'79","reported_seq":36,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-09-18T08:07:48.103482+0000","last_change":"2024-09-18T08:07:28.425814+0000","last_active":"2024-09-18T08:07:48.103482+0000","last_peered":"2024-09-18T08:07:48.103482+0000","last_clean":"2024-09-18T08:07:48.103482+0000","last_became_active":"2024-09-18T08:07:26.398930+0000","last_became_peered":"2024-09-18T08:07:26.398930+0000","last_unstale":"2024-09-18T08:07:48.103482+0000","last_undegraded":"2024-09-18T08:07:48.103482+0000","last_fullsized":"2024-09-18T08:07:48.103482+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":152,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":3,"last_scrub":"74'79","last_scrub_stamp":"2024-09-18T08:04:33.791841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:00:11.893742+0000","last_clean_scrub_stamp":"2024-09-18T08:04:33.791841+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"user requested scrub scheduled @ 2024-09-18T08:07:58.295459+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":53670,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":10,"num_read_kb":20,"num_write":10,"num_write_kb":148,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":381350,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,7],"acting":[3,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"69'14","reported_seq":295,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-09-18T08:07:47.300672+0000","last_change":"2024-09-18T08:06:24.184801+0000","last_active":"2024-09-18T08:07:47.300672+0000","last_peered":"2024-09-18T08:07:47.300672+0000","last_clean":"2024-09-18T08:07:47.300672+0000","last_became_active":"2024-09-18T08:06:24.133949+0000","last_became_peered":"2024-09-18T08:06:24.133949+0000","last_unstale":"2024-09-18T08:07:47.300672+0000","last_undegraded":"2024-09-18T08:07:47.300672+0000","last_fullsized":"2024-09-18T08:07:47.300672+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'14","last_scrub_stamp":"2024-09-18T08:04:56.545262+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_clean_scrub_stamp":"2024-09-18T08:04:56.545262+0000","objects_scrubbed":7,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":21,"num_read_kb":14,"num_write":14,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"62'2","reported_seq":238,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-09-18T08:07:48.063982+0000","last_change":"2024-09-18T08:06:43.607646+0000","last_active":"2024-09-18T08:07:48.063982+0000","last_peered":"2024-09-18T08:07:48.063982+0000","last_clean":"2024-09-18T08:07:48.063982+0000","last_became_active":"2024-09-18T08:02:47.872857+0000","last_became_peered":"2024-09-18T08:02:47.872857+0000","last_unstale":"2024-09-18T08:07:48.063982+0000","last_undegraded":"2024-09-18T08:07:48.063982+0000","last_fullsized":"2024-09-18T08:07:48.063982+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"62'2","last_scrub_stamp":"2024-09-18T08:06:43.607152+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:46.860092+0000","last_clean_scrub_stamp":"2024-09-18T08:06:43.607152+0000","objects_scrubbed":2,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:07:53.027436+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1039,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.0","version":"69'8","reported_seq":285,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-09-18T08:07:48.101663+0000","last_change":"2024-09-18T08:04:23.863573+0000","last_active":"2024-09-18T08:07:48.101663+0000","last_peered":"2024-09-18T08:07:48.101663+0000","last_clean":"2024-09-18T08:07:48.101663+0000","last_became_active":"2024-09-18T08:02:51.909680+0000","last_became_peered":"2024-09-18T08:02:51.909680+0000","last_unstale":"2024-09-18T08:07:48.101663+0000","last_undegraded":"2024-09-18T08:07:48.101663+0000","last_fullsized":"2024-09-18T08:07:48.101663+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'8","last_scrub_stamp":"2024-09-18T08:04:23.863330+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:50.886830+0000","last_clean_scrub_stamp":"2024-09-18T08:04:23.863330+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:08:04.168099+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":234,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-09-18T08:07:48.102767+0000","last_change":"2024-09-18T08:06:44.781751+0000","last_active":"2024-09-18T08:07:48.102767+0000","last_peered":"2024-09-18T08:07:48.102767+0000","last_clean":"2024-09-18T08:07:48.102767+0000","last_became_active":"2024-09-18T08:06:24.140303+0000","last_became_peered":"2024-09-18T08:06:24.140303+0000","last_unstale":"2024-09-18T08:07:48.102767+0000","last_undegraded":"2024-09-18T08:07:48.102767+0000","last_fullsized":"2024-09-18T08:07:48.102767+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:06:44.781685+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_clean_scrub_stamp":"2024-09-18T08:06:44.781685+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:07:48.361147+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.7","version":"74'79","reported_seq":31,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-09-18T08:07:48.103976+0000","last_change":"2024-09-18T08:07:28.430282+0000","last_active":"2024-09-18T08:07:48.103976+0000","last_peered":"2024-09-18T08:07:48.103976+0000","last_clean":"2024-09-18T08:07:48.103976+0000","last_became_active":"2024-09-18T08:07:26.398289+0000","last_became_peered":"2024-09-18T08:07:26.398289+0000","last_unstale":"2024-09-18T08:07:48.103976+0000","last_undegraded":"2024-09-18T08:07:48.103976+0000","last_fullsized":"2024-09-18T08:07:48.103976+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":152,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":3,"last_scrub":"74'79","last_scrub_stamp":"2024-09-18T08:04:33.791841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:00:11.893742+0000","last_clean_scrub_stamp":"2024-09-18T08:04:33.791841+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"user requested scrub scheduled @ 2024-09-18T08:07:57.298170+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":53670,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":10,"num_read_kb":20,"num_write":10,"num_write_kb":148,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":53670,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,7],"acting":[3,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.2","version":"69'10","reported_seq":250,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-09-18T08:07:47.301406+0000","last_change":"2024-09-18T08:07:22.666123+0000","last_active":"2024-09-18T08:07:47.301406+0000","last_peered":"2024-09-18T08:07:47.301406+0000","last_clean":"2024-09-18T08:07:47.301406+0000","last_became_active":"2024-09-18T08:02:49.895265+0000","last_became_peered":"2024-09-18T08:02:49.895265+0000","last_unstale":"2024-09-18T08:07:47.301406+0000","last_undegraded":"2024-09-18T08:07:47.301406+0000","last_fullsized":"2024-09-18T08:07:47.301406+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'10","last_scrub_stamp":"2024-09-18T08:07:22.666042+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_clean_scrub_stamp":"2024-09-18T08:07:22.666042+0000","objects_scrubbed":5,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:08:36.275745+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":223,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-09-18T08:07:48.114493+0000","last_change":"2024-09-18T08:03:48.009689+0000","last_active":"2024-09-18T08:07:48.114493+0000","last_peered":"2024-09-18T08:07:48.114493+0000","last_clean":"2024-09-18T08:07:48.114493+0000","last_became_active":"2024-09-18T08:02:47.874727+0000","last_became_peered":"2024-09-18T08:02:47.874727+0000","last_unstale":"2024-09-18T08:07:48.114493+0000","last_undegraded":"2024-09-18T08:07:48.114493+0000","last_fullsized":"2024-09-18T08:07:48.114493+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:03:48.009611+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:46.860092+0000","last_clean_scrub_stamp":"2024-09-18T08:03:48.009611+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:08:35.608502+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.3","version":"138'10","reported_seq":315,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-09-18T08:07:48.060349+0000","last_change":"2024-09-18T08:07:46.695714+0000","last_active":"2024-09-18T08:07:48.060349+0000","last_peered":"2024-09-18T08:07:48.060349+0000","last_clean":"2024-09-18T08:07:48.060349+0000","last_became_active":"2024-09-18T08:06:24.134194+0000","last_became_peered":"2024-09-18T08:06:24.134194+0000","last_unstale":"2024-09-18T08:07:48.060349+0000","last_undegraded":"2024-09-18T08:07:48.060349+0000","last_fullsized":"2024-09-18T08:07:48.060349+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"138'10","last_scrub_stamp":"2024-09-18T08:07:46.695656+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:50.886830+0000","last_clean_scrub_stamp":"2024-09-18T08:07:46.695656+0000","objects_scrubbed":1,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:09:10.517915+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":228,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-09-18T08:07:48.060257+0000","last_change":"2024-09-18T08:07:44.739629+0000","last_active":"2024-09-18T08:07:48.060257+0000","last_peered":"2024-09-18T08:07:48.060257+0000","last_clean":"2024-09-18T08:07:48.060257+0000","last_became_active":"2024-09-18T08:02:53.909583+0000","last_became_peered":"2024-09-18T08:02:53.909583+0000","last_unstale":"2024-09-18T08:07:48.060257+0000","last_undegraded":"2024-09-18T08:07:48.060257+0000","last_fullsized":"2024-09-18T08:07:48.060257+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:07:44.739578+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_clean_scrub_stamp":"2024-09-18T08:07:44.739578+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:09:01.654165+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.6","version":"74'79","reported_seq":111,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-09-18T08:07:48.061878+0000","last_change":"2024-09-18T08:07:28.414757+0000","last_active":"2024-09-18T08:07:48.061878+0000","last_peered":"2024-09-18T08:07:48.061878+0000","last_clean":"2024-09-18T08:07:48.061878+0000","last_became_active":"2024-09-18T08:07:26.426532+0000","last_became_peered":"2024-09-18T08:07:26.426532+0000","last_unstale":"2024-09-18T08:07:48.061878+0000","last_undegraded":"2024-09-18T08:07:48.061878+0000","last_fullsized":"2024-09-18T08:07:48.061878+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":152,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":3,"last_scrub":"74'79","last_scrub_stamp":"2024-09-18T08:04:33.791841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:00:11.893742+0000","last_clean_scrub_stamp":"2024-09-18T08:04:33.791841+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"user requested scrub scheduled @ 2024-09-18T08:07:57.597721+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":53670,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":10,"num_read_kb":20,"num_write":10,"num_write_kb":148,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":53670,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,3],"acting":[0,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.3","version":"69'19","reported_seq":255,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-09-18T08:07:48.060633+0000","last_change":"2024-09-18T08:04:07.231406+0000","last_active":"2024-09-18T08:07:48.060633+0000","last_peered":"2024-09-18T08:07:48.060633+0000","last_clean":"2024-09-18T08:07:48.060633+0000","last_became_active":"2024-09-18T08:02:49.890318+0000","last_became_peered":"2024-09-18T08:02:49.890318+0000","last_unstale":"2024-09-18T08:07:48.060633+0000","last_undegraded":"2024-09-18T08:07:48.060633+0000","last_fullsized":"2024-09-18T08:07:48.060633+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'19","last_scrub_stamp":"2024-09-18T08:04:07.229846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_clean_scrub_stamp":"2024-09-18T08:04:07.229846+0000","objects_scrubbed":8,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:08:33.599543+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":39,"num_read_kb":25,"num_write":22,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.4","version":"62'1","reported_seq":229,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-09-18T08:07:47.303220+0000","last_change":"2024-09-18T08:04:28.827545+0000","last_active":"2024-09-18T08:07:47.303220+0000","last_peered":"2024-09-18T08:07:47.303220+0000","last_clean":"2024-09-18T08:07:47.303220+0000","last_became_active":"2024-09-18T08:02:47.874764+0000","last_became_peered":"2024-09-18T08:02:47.874764+0000","last_unstale":"2024-09-18T08:07:47.303220+0000","last_undegraded":"2024-09-18T08:07:47.303220+0000","last_fullsized":"2024-09-18T08:07:47.303220+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"62'1","last_scrub_stamp":"2024-09-18T08:04:28.827464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:46.860092+0000","last_clean_scrub_stamp":"2024-09-18T08:04:28.827464+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:07:54.534873+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":436,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":149,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-09-18T08:07:48.060701+0000","last_change":"2024-09-18T08:06:24.133954+0000","last_active":"2024-09-18T08:07:48.060701+0000","last_peered":"2024-09-18T08:07:48.060701+0000","last_clean":"2024-09-18T08:07:48.060701+0000","last_became_active":"2024-09-18T08:06:24.133262+0000","last_became_peered":"2024-09-18T08:06:24.133262+0000","last_unstale":"2024-09-18T08:07:48.060701+0000","last_undegraded":"2024-09-18T08:07:48.060701+0000","last_fullsized":"2024-09-18T08:07:48.060701+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:04:13.377561+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:50.886830+0000","last_clean_scrub_stamp":"2024-09-18T08:04:13.377561+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:07:53.735285+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":233,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-09-18T08:07:47.300578+0000","last_change":"2024-09-18T08:06:24.139017+0000","last_active":"2024-09-18T08:07:47.300578+0000","last_peered":"2024-09-18T08:07:47.300578+0000","last_clean":"2024-09-18T08:07:47.300578+0000","last_became_active":"2024-09-18T08:06:24.138652+0000","last_became_peered":"2024-09-18T08:06:24.138652+0000","last_unstale":"2024-09-18T08:07:47.300578+0000","last_undegraded":"2024-09-18T08:07:47.300578+0000","last_fullsized":"2024-09-18T08:07:47.300578+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:06:10.507191+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_clean_scrub_stamp":"2024-09-18T08:06:10.507191+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":25,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"74'79","reported_seq":30,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-09-18T08:07:48.102142+0000","last_change":"2024-09-18T08:07:28.117202+0000","last_active":"2024-09-18T08:07:48.102142+0000","last_peered":"2024-09-18T08:07:48.102142+0000","last_clean":"2024-09-18T08:07:48.102142+0000","last_became_active":"2024-09-18T08:07:26.400643+0000","last_became_peered":"2024-09-18T08:07:26.400643+0000","last_unstale":"2024-09-18T08:07:48.102142+0000","last_undegraded":"2024-09-18T08:07:48.102142+0000","last_fullsized":"2024-09-18T08:07:48.102142+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":152,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":4,"last_scrub":"74'79","last_scrub_stamp":"2024-09-18T08:04:33.791841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:00:11.893742+0000","last_clean_scrub_stamp":"2024-09-18T08:04:33.791841+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"user requested scrub scheduled @ 2024-09-18T08:07:57.694967+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":53670,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":11,"num_read_kb":20,"num_write":11,"num_write_kb":148,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":53670,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,2],"acting":[5,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.4","version":"69'28","reported_seq":328,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-09-18T08:07:47.303055+0000","last_change":"2024-09-18T08:04:12.737618+0000","last_active":"2024-09-18T08:07:47.303055+0000","last_peered":"2024-09-18T08:07:47.303055+0000","last_clean":"2024-09-18T08:07:47.303055+0000","last_became_active":"2024-09-18T08:02:49.897221+0000","last_became_peered":"2024-09-18T08:02:49.897221+0000","last_unstale":"2024-09-18T08:07:47.303055+0000","last_undegraded":"2024-09-18T08:07:47.303055+0000","last_fullsized":"2024-09-18T08:07:47.303055+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'28","last_scrub_stamp":"2024-09-18T08:04:12.737528+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_clean_scrub_stamp":"2024-09-18T08:04:12.737528+0000","objects_scrubbed":10,"log_size":28,"log_dups_size":0,"ondisk_log_size":28,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:07:53.542712+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":48,"num_read_kb":33,"num_write":26,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":238,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-09-18T08:07:47.301558+0000","last_change":"2024-09-18T08:06:24.143321+0000","last_active":"2024-09-18T08:07:47.301558+0000","last_peered":"2024-09-18T08:07:47.301558+0000","last_clean":"2024-09-18T08:07:47.301558+0000","last_became_active":"2024-09-18T08:06:24.143019+0000","last_became_peered":"2024-09-18T08:06:24.143019+0000","last_unstale":"2024-09-18T08:07:47.301558+0000","last_undegraded":"2024-09-18T08:07:47.301558+0000","last_fullsized":"2024-09-18T08:07:47.301558+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:05:39.490141+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:46.860092+0000","last_clean_scrub_stamp":"2024-09-18T08:05:39.490141+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:08:25.597232+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"64'2","reported_seq":263,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-09-18T08:07:48.104918+0000","last_change":"2024-09-18T08:06:24.165227+0000","last_active":"2024-09-18T08:07:48.104918+0000","last_peered":"2024-09-18T08:07:48.104918+0000","last_clean":"2024-09-18T08:07:48.104918+0000","last_became_active":"2024-09-18T08:06:24.136080+0000","last_became_peered":"2024-09-18T08:06:24.136080+0000","last_unstale":"2024-09-18T08:07:48.104918+0000","last_undegraded":"2024-09-18T08:07:48.104918+0000","last_fullsized":"2024-09-18T08:07:48.104918+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"64'2","last_scrub_stamp":"2024-09-18T08:05:44.802942+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:45.855979+0000","last_clean_scrub_stamp":"2024-09-18T08:05:44.802942+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:07:59.917939+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.001348349,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":38,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":207,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-09-18T08:07:48.059665+0000","last_change":"2024-09-18T08:02:51.905694+0000","last_active":"2024-09-18T08:07:48.059665+0000","last_peered":"2024-09-18T08:07:48.059665+0000","last_clean":"2024-09-18T08:07:48.059665+0000","last_became_active":"2024-09-18T08:02:51.905357+0000","last_became_peered":"2024-09-18T08:02:51.905357+0000","last_unstale":"2024-09-18T08:07:48.059665+0000","last_undegraded":"2024-09-18T08:07:48.059665+0000","last_fullsized":"2024-09-18T08:07:48.059665+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:02:50.886830+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:50.886830+0000","last_clean_scrub_stamp":"2024-09-18T08:02:50.886830+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:07:52.727169+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":221,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-09-18T08:07:48.103189+0000","last_change":"2024-09-18T08:07:08.245134+0000","last_active":"2024-09-18T08:07:48.103189+0000","last_peered":"2024-09-18T08:07:48.103189+0000","last_clean":"2024-09-18T08:07:48.103189+0000","last_became_active":"2024-09-18T08:02:53.911036+0000","last_became_peered":"2024-09-18T08:02:53.911036+0000","last_unstale":"2024-09-18T08:07:48.103189+0000","last_undegraded":"2024-09-18T08:07:48.103189+0000","last_fullsized":"2024-09-18T08:07:48.103189+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:07:08.245063+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_clean_scrub_stamp":"2024-09-18T08:07:08.245063+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:08:34.264472+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"74'79","reported_seq":111,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-09-18T08:07:48.060054+0000","last_change":"2024-09-18T08:07:28.414271+0000","last_active":"2024-09-18T08:07:48.060054+0000","last_peered":"2024-09-18T08:07:48.060054+0000","last_clean":"2024-09-18T08:07:48.060054+0000","last_became_active":"2024-09-18T08:07:26.407182+0000","last_became_peered":"2024-09-18T08:07:26.407182+0000","last_unstale":"2024-09-18T08:07:48.060054+0000","last_undegraded":"2024-09-18T08:07:48.060054+0000","last_fullsized":"2024-09-18T08:07:48.060054+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":152,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":4,"last_scrub":"74'79","last_scrub_stamp":"2024-09-18T08:04:33.791841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:00:11.893742+0000","last_clean_scrub_stamp":"2024-09-18T08:04:33.791841+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"user requested scrub scheduled @ 2024-09-18T08:07:56.582469+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":53670,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":11,"num_read_kb":20,"num_write":11,"num_write_kb":148,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":53670,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,4],"acting":[0,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.7","version":"69'13","reported_seq":165,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-09-18T08:07:47.306745+0000","last_change":"2024-09-18T08:04:18.164662+0000","last_active":"2024-09-18T08:07:47.306745+0000","last_peered":"2024-09-18T08:07:47.306745+0000","last_clean":"2024-09-18T08:07:47.306745+0000","last_became_active":"2024-09-18T08:04:18.144925+0000","last_became_peered":"2024-09-18T08:04:18.144925+0000","last_unstale":"2024-09-18T08:07:47.306745+0000","last_undegraded":"2024-09-18T08:07:47.306745+0000","last_fullsized":"2024-09-18T08:07:47.306745+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'13","last_scrub_stamp":"2024-09-18T08:03:50.700818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_clean_scrub_stamp":"2024-09-18T08:03:50.700818+0000","objects_scrubbed":0,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:08:32.584716+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":29,"num_read_kb":18,"num_write":16,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,0],"acting":[2,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":237,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-09-18T08:07:47.301920+0000","last_change":"2024-09-18T08:06:24.130858+0000","last_active":"2024-09-18T08:07:47.301920+0000","last_peered":"2024-09-18T08:07:47.301920+0000","last_clean":"2024-09-18T08:07:47.301920+0000","last_became_active":"2024-09-18T08:06:24.126334+0000","last_became_peered":"2024-09-18T08:06:24.126334+0000","last_unstale":"2024-09-18T08:07:47.301920+0000","last_undegraded":"2024-09-18T08:07:47.301920+0000","last_fullsized":"2024-09-18T08:07:47.301920+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:03:49.704004+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:46.860092+0000","last_clean_scrub_stamp":"2024-09-18T08:03:49.704004+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:08:37.938363+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"62'1","reported_seq":233,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-09-18T08:07:47.305907+0000","last_change":"2024-09-18T08:04:09.732724+0000","last_active":"2024-09-18T08:07:47.305907+0000","last_peered":"2024-09-18T08:07:47.305907+0000","last_clean":"2024-09-18T08:07:47.305907+0000","last_became_active":"2024-09-18T08:02:46.868175+0000","last_became_peered":"2024-09-18T08:02:46.868175+0000","last_unstale":"2024-09-18T08:07:47.305907+0000","last_undegraded":"2024-09-18T08:07:47.305907+0000","last_fullsized":"2024-09-18T08:07:47.305907+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"62'1","last_scrub_stamp":"2024-09-18T08:04:09.732645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:45.855979+0000","last_clean_scrub_stamp":"2024-09-18T08:04:09.732645+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:07:52.854217+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0.00066785499999999997,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":232,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-09-18T08:07:47.306915+0000","last_change":"2024-09-18T08:07:08.808087+0000","last_active":"2024-09-18T08:07:47.306915+0000","last_peered":"2024-09-18T08:07:47.306915+0000","last_clean":"2024-09-18T08:07:47.306915+0000","last_became_active":"2024-09-18T08:02:51.908156+0000","last_became_peered":"2024-09-18T08:02:51.908156+0000","last_unstale":"2024-09-18T08:07:47.306915+0000","last_undegraded":"2024-09-18T08:07:47.306915+0000","last_fullsized":"2024-09-18T08:07:47.306915+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:07:08.806748+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:50.886830+0000","last_clean_scrub_stamp":"2024-09-18T08:07:08.806748+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:08:09.365103+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":234,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-09-18T08:07:48.107362+0000","last_change":"2024-09-18T08:07:10.837850+0000","last_active":"2024-09-18T08:07:48.107362+0000","last_peered":"2024-09-18T08:07:48.107362+0000","last_clean":"2024-09-18T08:07:48.107362+0000","last_became_active":"2024-09-18T08:06:24.145382+0000","last_became_peered":"2024-09-18T08:06:24.145382+0000","last_unstale":"2024-09-18T08:07:48.107362+0000","last_undegraded":"2024-09-18T08:07:48.107362+0000","last_fullsized":"2024-09-18T08:07:48.107362+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:07:10.837780+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_clean_scrub_stamp":"2024-09-18T08:07:10.837780+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:08:18.424418+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.3","version":"74'79","reported_seq":30,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-09-18T08:07:47.301241+0000","last_change":"2024-09-18T08:07:28.330501+0000","last_active":"2024-09-18T08:07:47.301241+0000","last_peered":"2024-09-18T08:07:47.301241+0000","last_clean":"2024-09-18T08:07:47.301241+0000","last_became_active":"2024-09-18T08:07:26.404812+0000","last_became_peered":"2024-09-18T08:07:26.404812+0000","last_unstale":"2024-09-18T08:07:47.301241+0000","last_undegraded":"2024-09-18T08:07:47.301241+0000","last_fullsized":"2024-09-18T08:07:47.301241+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":152,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":3,"last_scrub":"74'79","last_scrub_stamp":"2024-09-18T08:04:33.791841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:00:11.893742+0000","last_clean_scrub_stamp":"2024-09-18T08:04:33.791841+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"user requested scrub scheduled @ 2024-09-18T08:07:56.735564+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":53670,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":10,"num_read_kb":20,"num_write":11,"num_write_kb":148,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":53670,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.6","version":"69'12","reported_seq":245,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-09-18T08:07:48.063692+0000","last_change":"2024-09-18T08:07:39.603625+0000","last_active":"2024-09-18T08:07:48.063692+0000","last_peered":"2024-09-18T08:07:48.063692+0000","last_clean":"2024-09-18T08:07:48.063692+0000","last_became_active":"2024-09-18T08:02:49.898079+0000","last_became_peered":"2024-09-18T08:02:49.898079+0000","last_unstale":"2024-09-18T08:07:48.063692+0000","last_undegraded":"2024-09-18T08:07:48.063692+0000","last_fullsized":"2024-09-18T08:07:48.063692+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'12","last_scrub_stamp":"2024-09-18T08:07:39.603562+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_clean_scrub_stamp":"2024-09-18T08:07:39.603562+0000","objects_scrubbed":6,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:09:06.884765+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":18,"num_read_kb":12,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":231,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-09-18T08:07:48.061978+0000","last_change":"2024-09-18T08:07:10.782142+0000","last_active":"2024-09-18T08:07:48.061978+0000","last_peered":"2024-09-18T08:07:48.061978+0000","last_clean":"2024-09-18T08:07:48.061978+0000","last_became_active":"2024-09-18T08:02:47.873809+0000","last_became_peered":"2024-09-18T08:02:47.873809+0000","last_unstale":"2024-09-18T08:07:48.061978+0000","last_undegraded":"2024-09-18T08:07:48.061978+0000","last_fullsized":"2024-09-18T08:07:48.061978+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:07:10.781678+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:46.860092+0000","last_clean_scrub_stamp":"2024-09-18T08:07:10.781678+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:08:17.733635+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"70'5","reported_seq":164,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-09-18T08:07:47.314967+0000","last_change":"2024-09-18T08:06:51.522688+0000","last_active":"2024-09-18T08:07:47.314967+0000","last_peered":"2024-09-18T08:07:47.314967+0000","last_clean":"2024-09-18T08:07:47.314967+0000","last_became_active":"2024-09-18T08:06:46.141667+0000","last_became_peered":"2024-09-18T08:06:46.141667+0000","last_unstale":"2024-09-18T08:07:47.314967+0000","last_undegraded":"2024-09-18T08:07:47.314967+0000","last_fullsized":"2024-09-18T08:07:47.314967+0000","mapping_epoch":143,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":144,"parent":"0.0","parent_split_bits":0,"last_scrub":"70'5","last_scrub_stamp":"2024-09-18T08:06:51.522442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:45.855979+0000","last_clean_scrub_stamp":"2024-09-18T08:06:51.522442+0000","objects_scrubbed":1,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:07:58.661843+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0.000504055,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":69,"num_read_kb":64,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":231,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-09-18T08:07:48.101797+0000","last_change":"2024-09-18T08:07:20.767916+0000","last_active":"2024-09-18T08:07:48.101797+0000","last_peered":"2024-09-18T08:07:48.101797+0000","last_clean":"2024-09-18T08:07:48.101797+0000","last_became_active":"2024-09-18T08:02:51.913112+0000","last_became_peered":"2024-09-18T08:02:51.913112+0000","last_unstale":"2024-09-18T08:07:48.101797+0000","last_undegraded":"2024-09-18T08:07:48.101797+0000","last_fullsized":"2024-09-18T08:07:48.101797+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:07:20.767841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:50.886830+0000","last_clean_scrub_stamp":"2024-09-18T08:07:20.767841+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:08:41.680214+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":204,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-09-18T08:07:47.302342+0000","last_change":"2024-09-18T08:02:53.907689+0000","last_active":"2024-09-18T08:07:47.302342+0000","last_peered":"2024-09-18T08:07:47.302342+0000","last_clean":"2024-09-18T08:07:47.302342+0000","last_became_active":"2024-09-18T08:02:53.907404+0000","last_became_peered":"2024-09-18T08:02:53.907404+0000","last_unstale":"2024-09-18T08:07:47.302342+0000","last_undegraded":"2024-09-18T08:07:47.302342+0000","last_fullsized":"2024-09-18T08:07:47.302342+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_clean_scrub_stamp":"2024-09-18T08:02:52.890475+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:08:27.718155+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.5","version":"138'19","reported_seq":277,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-09-18T08:07:47.301357+0000","last_change":"2024-09-18T08:06:24.191102+0000","last_active":"2024-09-18T08:07:47.301357+0000","last_peered":"2024-09-18T08:07:47.301357+0000","last_clean":"2024-09-18T08:07:47.301357+0000","last_became_active":"2024-09-18T08:06:24.146117+0000","last_became_peered":"2024-09-18T08:06:24.146117+0000","last_unstale":"2024-09-18T08:07:47.301357+0000","last_undegraded":"2024-09-18T08:07:47.301357+0000","last_fullsized":"2024-09-18T08:07:47.301357+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"130'18","last_scrub_stamp":"2024-09-18T08:06:15.455503+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_clean_scrub_stamp":"2024-09-18T08:06:15.455503+0000","objects_scrubbed":8,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":24,"num_read_kb":14,"num_write":13,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":308,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,7],"acting":[4,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":238,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-09-18T08:07:48.103187+0000","last_change":"2024-09-18T08:06:24.134043+0000","last_active":"2024-09-18T08:07:48.103187+0000","last_peered":"2024-09-18T08:07:48.103187+0000","last_clean":"2024-09-18T08:07:48.103187+0000","last_became_active":"2024-09-18T08:06:24.133425+0000","last_became_peered":"2024-09-18T08:06:24.133425+0000","last_unstale":"2024-09-18T08:07:48.103187+0000","last_undegraded":"2024-09-18T08:07:48.103187+0000","last_fullsized":"2024-09-18T08:07:48.103187+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:05:38.080993+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:46.860092+0000","last_clean_scrub_stamp":"2024-09-18T08:05:38.080993+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:08:26.282275+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"74'79","reported_seq":313,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-09-18T08:07:48.107343+0000","last_change":"2024-09-18T08:07:26.389589+0000","last_active":"2024-09-18T08:07:48.107343+0000","last_peered":"2024-09-18T08:07:48.107343+0000","last_clean":"2024-09-18T08:07:48.107343+0000","last_became_active":"2024-09-18T08:07:24.376931+0000","last_became_peered":"2024-09-18T08:07:24.376931+0000","last_unstale":"2024-09-18T08:07:48.107343+0000","last_undegraded":"2024-09-18T08:07:48.107343+0000","last_fullsized":"2024-09-18T08:07:48.107343+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":153,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'79","last_scrub_stamp":"2024-09-18T08:04:33.791841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:00:11.893742+0000","last_clean_scrub_stamp":"2024-09-18T08:04:33.791841+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"user requested scrub scheduled @ 2024-09-18T08:07:55.672750+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":53669,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":11,"num_read_kb":21,"num_write":10,"num_write_kb":148,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":161009,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":59,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-09-18T08:07:47.306166+0000","last_change":"2024-09-18T08:07:09.744206+0000","last_active":"2024-09-18T08:07:47.306166+0000","last_peered":"2024-09-18T08:07:47.306166+0000","last_clean":"2024-09-18T08:07:47.306166+0000","last_became_active":"2024-09-18T08:06:46.141046+0000","last_became_peered":"2024-09-18T08:06:46.141046+0000","last_unstale":"2024-09-18T08:07:47.306166+0000","last_undegraded":"2024-09-18T08:07:47.306166+0000","last_fullsized":"2024-09-18T08:07:47.306166+0000","mapping_epoch":143,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":144,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:07:09.744137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:50.886830+0000","last_clean_scrub_stamp":"2024-09-18T08:07:09.744137+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:08:16.138578+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":220,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-09-18T08:07:48.113846+0000","last_change":"2024-09-18T08:06:50.710637+0000","last_active":"2024-09-18T08:07:48.113846+0000","last_peered":"2024-09-18T08:07:48.113846+0000","last_clean":"2024-09-18T08:07:48.113846+0000","last_became_active":"2024-09-18T08:02:53.908668+0000","last_became_peered":"2024-09-18T08:02:53.908668+0000","last_unstale":"2024-09-18T08:07:48.113846+0000","last_undegraded":"2024-09-18T08:07:48.113846+0000","last_fullsized":"2024-09-18T08:07:48.113846+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:06:50.710565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_clean_scrub_stamp":"2024-09-18T08:06:50.710565+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:08:16.618092+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.e","version":"69'11","reported_seq":277,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-09-18T08:07:47.301652+0000","last_change":"2024-09-18T08:06:24.173542+0000","last_active":"2024-09-18T08:07:47.301652+0000","last_peered":"2024-09-18T08:07:47.301652+0000","last_clean":"2024-09-18T08:07:47.301652+0000","last_became_active":"2024-09-18T08:06:24.143839+0000","last_became_peered":"2024-09-18T08:06:24.143839+0000","last_unstale":"2024-09-18T08:07:47.301652+0000","last_undegraded":"2024-09-18T08:07:47.301652+0000","last_fullsized":"2024-09-18T08:07:47.301652+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'11","last_scrub_stamp":"2024-09-18T08:06:14.472168+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_clean_scrub_stamp":"2024-09-18T08:06:14.472168+0000","objects_scrubbed":5,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":23,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":216,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-09-18T08:07:47.300960+0000","last_change":"2024-09-18T08:02:47.880265+0000","last_active":"2024-09-18T08:07:47.300960+0000","last_peered":"2024-09-18T08:07:47.300960+0000","last_clean":"2024-09-18T08:07:47.300960+0000","last_became_active":"2024-09-18T08:02:47.879764+0000","last_became_peered":"2024-09-18T08:02:47.879764+0000","last_unstale":"2024-09-18T08:07:47.300960+0000","last_undegraded":"2024-09-18T08:07:47.300960+0000","last_fullsized":"2024-09-18T08:07:47.300960+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:02:46.860092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:46.860092+0000","last_clean_scrub_stamp":"2024-09-18T08:02:46.860092+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:07:56.537498+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":229,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-09-18T08:07:48.113662+0000","last_change":"2024-09-18T08:06:24.132512+0000","last_active":"2024-09-18T08:07:48.113662+0000","last_peered":"2024-09-18T08:07:48.113662+0000","last_clean":"2024-09-18T08:07:48.113662+0000","last_became_active":"2024-09-18T08:06:24.132140+0000","last_became_peered":"2024-09-18T08:06:24.132140+0000","last_unstale":"2024-09-18T08:07:48.113662+0000","last_undegraded":"2024-09-18T08:07:48.113662+0000","last_fullsized":"2024-09-18T08:07:48.113662+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:05:08.839644+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:50.886830+0000","last_clean_scrub_stamp":"2024-09-18T08:05:08.839644+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":234,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-09-18T08:07:48.103568+0000","last_change":"2024-09-18T08:07:09.197912+0000","last_active":"2024-09-18T08:07:48.103568+0000","last_peered":"2024-09-18T08:07:48.103568+0000","last_clean":"2024-09-18T08:07:48.103568+0000","last_became_active":"2024-09-18T08:06:24.130266+0000","last_became_peered":"2024-09-18T08:06:24.130266+0000","last_unstale":"2024-09-18T08:07:48.103568+0000","last_undegraded":"2024-09-18T08:07:48.103568+0000","last_fullsized":"2024-09-18T08:07:48.103568+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:07:09.197841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_clean_scrub_stamp":"2024-09-18T08:07:09.197841+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:08:21.836105+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"74'79","reported_seq":37,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-09-18T08:07:48.103307+0000","last_change":"2024-09-18T08:07:31.447652+0000","last_active":"2024-09-18T08:07:48.103307+0000","last_peered":"2024-09-18T08:07:48.103307+0000","last_clean":"2024-09-18T08:07:48.103307+0000","last_became_active":"2024-09-18T08:07:26.407298+0000","last_became_peered":"2024-09-18T08:07:26.407298+0000","last_unstale":"2024-09-18T08:07:48.103307+0000","last_undegraded":"2024-09-18T08:07:48.103307+0000","last_fullsized":"2024-09-18T08:07:48.103307+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":152,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":4,"last_scrub":"74'79","last_scrub_stamp":"2024-09-18T08:04:33.791841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:00:11.893742+0000","last_clean_scrub_stamp":"2024-09-18T08:04:33.791841+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"user requested scrub scheduled @ 2024-09-18T08:07:58.706072+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":53670,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":10,"num_read_kb":20,"num_write":10,"num_write_kb":148,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":316358,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,3],"acting":[5,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.d","version":"69'17","reported_seq":240,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-09-18T08:07:47.301410+0000","last_change":"2024-09-18T08:02:49.895971+0000","last_active":"2024-09-18T08:07:47.301410+0000","last_peered":"2024-09-18T08:07:47.301410+0000","last_clean":"2024-09-18T08:07:47.301410+0000","last_became_active":"2024-09-18T08:02:49.895578+0000","last_became_peered":"2024-09-18T08:02:49.895578+0000","last_unstale":"2024-09-18T08:07:47.301410+0000","last_undegraded":"2024-09-18T08:07:47.301410+0000","last_fullsized":"2024-09-18T08:07:47.301410+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_clean_scrub_stamp":"2024-09-18T08:02:48.870718+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:08:06.433938+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":29,"num_read_kb":19,"num_write":18,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,1],"acting":[4,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":158,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-09-18T08:07:47.301032+0000","last_change":"2024-09-18T08:06:24.144485+0000","last_active":"2024-09-18T08:07:47.301032+0000","last_peered":"2024-09-18T08:07:47.301032+0000","last_clean":"2024-09-18T08:07:47.301032+0000","last_became_active":"2024-09-18T08:06:24.143152+0000","last_became_peered":"2024-09-18T08:06:24.143152+0000","last_unstale":"2024-09-18T08:07:47.301032+0000","last_undegraded":"2024-09-18T08:07:47.301032+0000","last_fullsized":"2024-09-18T08:07:47.301032+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:05:12.622901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:46.860092+0000","last_clean_scrub_stamp":"2024-09-18T08:05:12.622901+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:08:00.591147+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":223,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-09-18T08:07:47.302477+0000","last_change":"2024-09-18T08:05:31.851989+0000","last_active":"2024-09-18T08:07:47.302477+0000","last_peered":"2024-09-18T08:07:47.302477+0000","last_clean":"2024-09-18T08:07:47.302477+0000","last_became_active":"2024-09-18T08:02:51.907196+0000","last_became_peered":"2024-09-18T08:02:51.907196+0000","last_unstale":"2024-09-18T08:07:47.302477+0000","last_undegraded":"2024-09-18T08:07:47.302477+0000","last_fullsized":"2024-09-18T08:07:47.302477+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:05:31.851797+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:50.886830+0000","last_clean_scrub_stamp":"2024-09-18T08:05:31.851797+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:08:02.574997+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":213,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-09-18T08:07:47.306869+0000","last_change":"2024-09-18T08:04:06.709703+0000","last_active":"2024-09-18T08:07:47.306869+0000","last_peered":"2024-09-18T08:07:47.306869+0000","last_clean":"2024-09-18T08:07:47.306869+0000","last_became_active":"2024-09-18T08:02:53.904144+0000","last_became_peered":"2024-09-18T08:02:53.904144+0000","last_unstale":"2024-09-18T08:07:47.306869+0000","last_undegraded":"2024-09-18T08:07:47.306869+0000","last_fullsized":"2024-09-18T08:07:47.306869+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:04:06.709631+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_clean_scrub_stamp":"2024-09-18T08:04:06.709631+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:08:25.662878+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"74'79","reported_seq":316,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-09-18T08:07:48.103392+0000","last_change":"2024-09-18T08:07:28.132513+0000","last_active":"2024-09-18T08:07:48.103392+0000","last_peered":"2024-09-18T08:07:48.103392+0000","last_clean":"2024-09-18T08:07:48.103392+0000","last_became_active":"2024-09-18T08:07:26.398853+0000","last_became_peered":"2024-09-18T08:07:26.398853+0000","last_unstale":"2024-09-18T08:07:48.103392+0000","last_undegraded":"2024-09-18T08:07:48.103392+0000","last_fullsized":"2024-09-18T08:07:48.103392+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":152,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":4,"last_scrub":"74'79","last_scrub_stamp":"2024-09-18T08:04:33.791841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:00:11.893742+0000","last_clean_scrub_stamp":"2024-09-18T08:04:33.791841+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"user requested scrub scheduled @ 2024-09-18T08:07:56.641788+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":53670,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":11,"num_read_kb":21,"num_write":10,"num_write_kb":148,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":161009,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,2],"acting":[7,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.c","version":"69'10","reported_seq":273,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-09-18T08:07:47.301167+0000","last_change":"2024-09-18T08:06:24.180438+0000","last_active":"2024-09-18T08:07:47.301167+0000","last_peered":"2024-09-18T08:07:47.301167+0000","last_clean":"2024-09-18T08:07:47.301167+0000","last_became_active":"2024-09-18T08:06:24.145005+0000","last_became_peered":"2024-09-18T08:06:24.145005+0000","last_unstale":"2024-09-18T08:07:47.301167+0000","last_undegraded":"2024-09-18T08:07:47.301167+0000","last_fullsized":"2024-09-18T08:07:47.301167+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'10","last_scrub_stamp":"2024-09-18T08:05:41.474813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_clean_scrub_stamp":"2024-09-18T08:05:41.474813+0000","objects_scrubbed":5,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":232,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-09-18T08:07:48.103408+0000","last_change":"2024-09-18T08:06:46.339604+0000","last_active":"2024-09-18T08:07:48.103408+0000","last_peered":"2024-09-18T08:07:48.103408+0000","last_clean":"2024-09-18T08:07:48.103408+0000","last_became_active":"2024-09-18T08:02:47.881512+0000","last_became_peered":"2024-09-18T08:02:47.881512+0000","last_unstale":"2024-09-18T08:07:48.103408+0000","last_undegraded":"2024-09-18T08:07:48.103408+0000","last_fullsized":"2024-09-18T08:07:48.103408+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:06:46.339428+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:46.860092+0000","last_clean_scrub_stamp":"2024-09-18T08:06:46.339428+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:07:54.370307+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.d","version":"69'8","reported_seq":293,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-09-18T08:07:47.306606+0000","last_change":"2024-09-18T08:05:33.027988+0000","last_active":"2024-09-18T08:07:47.306606+0000","last_peered":"2024-09-18T08:07:47.306606+0000","last_clean":"2024-09-18T08:07:47.306606+0000","last_became_active":"2024-09-18T08:02:51.909011+0000","last_became_peered":"2024-09-18T08:02:51.909011+0000","last_unstale":"2024-09-18T08:07:47.306606+0000","last_undegraded":"2024-09-18T08:07:47.306606+0000","last_fullsized":"2024-09-18T08:07:47.306606+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'8","last_scrub_stamp":"2024-09-18T08:05:33.027783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:50.886830+0000","last_clean_scrub_stamp":"2024-09-18T08:05:33.027783+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:08:06.774918+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":221,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-09-18T08:07:47.301223+0000","last_change":"2024-09-18T08:07:16.559653+0000","last_active":"2024-09-18T08:07:47.301223+0000","last_peered":"2024-09-18T08:07:47.301223+0000","last_clean":"2024-09-18T08:07:47.301223+0000","last_became_active":"2024-09-18T08:02:53.901858+0000","last_became_peered":"2024-09-18T08:02:53.901858+0000","last_unstale":"2024-09-18T08:07:47.301223+0000","last_undegraded":"2024-09-18T08:07:47.301223+0000","last_fullsized":"2024-09-18T08:07:47.301223+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:07:16.559597+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_clean_scrub_stamp":"2024-09-18T08:07:16.559597+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:08:30.001882+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.b","version":"69'9","reported_seq":235,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-09-18T08:07:48.060478+0000","last_change":"2024-09-18T08:05:05.294031+0000","last_active":"2024-09-18T08:07:48.060478+0000","last_peered":"2024-09-18T08:07:48.060478+0000","last_clean":"2024-09-18T08:07:48.060478+0000","last_became_active":"2024-09-18T08:02:49.897998+0000","last_became_peered":"2024-09-18T08:02:49.897998+0000","last_unstale":"2024-09-18T08:07:48.060478+0000","last_undegraded":"2024-09-18T08:07:48.060478+0000","last_fullsized":"2024-09-18T08:07:48.060478+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'9","last_scrub_stamp":"2024-09-18T08:05:05.293936+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_clean_scrub_stamp":"2024-09-18T08:05:05.293936+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:08:03.837067+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":245,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-09-18T08:07:48.105052+0000","last_change":"2024-09-18T08:07:40.708363+0000","last_active":"2024-09-18T08:07:48.105052+0000","last_peered":"2024-09-18T08:07:48.105052+0000","last_clean":"2024-09-18T08:07:48.105052+0000","last_became_active":"2024-09-18T08:06:24.135005+0000","last_became_peered":"2024-09-18T08:06:24.135005+0000","last_unstale":"2024-09-18T08:07:48.105052+0000","last_undegraded":"2024-09-18T08:07:48.105052+0000","last_fullsized":"2024-09-18T08:07:48.105052+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:07:40.708307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:46.860092+0000","last_clean_scrub_stamp":"2024-09-18T08:07:40.708307+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:08:52.746885+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":232,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-09-18T08:07:47.306158+0000","last_change":"2024-09-18T08:07:10.728745+0000","last_active":"2024-09-18T08:07:47.306158+0000","last_peered":"2024-09-18T08:07:47.306158+0000","last_clean":"2024-09-18T08:07:47.306158+0000","last_became_active":"2024-09-18T08:02:51.901330+0000","last_became_peered":"2024-09-18T08:02:51.901330+0000","last_unstale":"2024-09-18T08:07:47.306158+0000","last_undegraded":"2024-09-18T08:07:47.306158+0000","last_fullsized":"2024-09-18T08:07:47.306158+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:07:10.728679+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:50.886830+0000","last_clean_scrub_stamp":"2024-09-18T08:07:10.728679+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:08:31.598375+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,3],"acting":[2,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":220,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-09-18T08:07:48.061840+0000","last_change":"2024-09-18T08:07:11.804906+0000","last_active":"2024-09-18T08:07:48.061840+0000","last_peered":"2024-09-18T08:07:48.061840+0000","last_clean":"2024-09-18T08:07:48.061840+0000","last_became_active":"2024-09-18T08:02:53.911068+0000","last_became_peered":"2024-09-18T08:02:53.911068+0000","last_unstale":"2024-09-18T08:07:48.061840+0000","last_undegraded":"2024-09-18T08:07:48.061840+0000","last_fullsized":"2024-09-18T08:07:48.061840+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:07:11.804821+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_clean_scrub_stamp":"2024-09-18T08:07:11.804821+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:08:17.666214+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.a","version":"69'19","reported_seq":248,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-09-18T08:07:47.301242+0000","last_change":"2024-09-18T08:07:41.000210+0000","last_active":"2024-09-18T08:07:47.301242+0000","last_peered":"2024-09-18T08:07:47.301242+0000","last_clean":"2024-09-18T08:07:47.301242+0000","last_became_active":"2024-09-18T08:06:24.139909+0000","last_became_peered":"2024-09-18T08:06:24.139909+0000","last_unstale":"2024-09-18T08:07:47.301242+0000","last_undegraded":"2024-09-18T08:07:47.301242+0000","last_fullsized":"2024-09-18T08:07:47.301242+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'19","last_scrub_stamp":"2024-09-18T08:07:41.000146+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_clean_scrub_stamp":"2024-09-18T08:07:41.000146+0000","objects_scrubbed":9,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:09:01.107938+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":31,"num_read_kb":20,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,7],"acting":[1,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":233,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-09-18T08:07:48.105595+0000","last_change":"2024-09-18T08:06:47.638974+0000","last_active":"2024-09-18T08:07:48.105595+0000","last_peered":"2024-09-18T08:07:48.105595+0000","last_clean":"2024-09-18T08:07:48.105595+0000","last_became_active":"2024-09-18T08:06:46.171298+0000","last_became_peered":"2024-09-18T08:06:46.171298+0000","last_unstale":"2024-09-18T08:07:48.105595+0000","last_undegraded":"2024-09-18T08:07:48.105595+0000","last_fullsized":"2024-09-18T08:07:48.105595+0000","mapping_epoch":143,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":144,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:06:47.638911+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:46.860092+0000","last_clean_scrub_stamp":"2024-09-18T08:06:47.638911+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:07:53.269627+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":216,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-09-18T08:07:47.307143+0000","last_change":"2024-09-18T08:04:56.004261+0000","last_active":"2024-09-18T08:07:47.307143+0000","last_peered":"2024-09-18T08:07:47.307143+0000","last_clean":"2024-09-18T08:07:47.307143+0000","last_became_active":"2024-09-18T08:02:51.903439+0000","last_became_peered":"2024-09-18T08:02:51.903439+0000","last_unstale":"2024-09-18T08:07:47.307143+0000","last_undegraded":"2024-09-18T08:07:47.307143+0000","last_fullsized":"2024-09-18T08:07:47.307143+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:04:56.004150+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:50.886830+0000","last_clean_scrub_stamp":"2024-09-18T08:04:56.004150+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:08:02.816710+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":59,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-09-18T08:07:47.307750+0000","last_change":"2024-09-18T08:06:50.902917+0000","last_active":"2024-09-18T08:07:47.307750+0000","last_peered":"2024-09-18T08:07:47.307750+0000","last_clean":"2024-09-18T08:07:47.307750+0000","last_became_active":"2024-09-18T08:06:46.137872+0000","last_became_peered":"2024-09-18T08:06:46.137872+0000","last_unstale":"2024-09-18T08:07:47.307750+0000","last_undegraded":"2024-09-18T08:07:47.307750+0000","last_fullsized":"2024-09-18T08:07:47.307750+0000","mapping_epoch":143,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":144,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:06:50.902835+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_clean_scrub_stamp":"2024-09-18T08:06:50.902835+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:07:54.316655+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,3],"acting":[2,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.9","version":"69'12","reported_seq":251,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-09-18T08:07:47.300826+0000","last_change":"2024-09-18T08:05:33.544197+0000","last_active":"2024-09-18T08:07:47.300826+0000","last_peered":"2024-09-18T08:07:47.300826+0000","last_clean":"2024-09-18T08:07:47.300826+0000","last_became_active":"2024-09-18T08:02:49.894762+0000","last_became_peered":"2024-09-18T08:02:49.894762+0000","last_unstale":"2024-09-18T08:07:47.300826+0000","last_undegraded":"2024-09-18T08:07:47.300826+0000","last_fullsized":"2024-09-18T08:07:47.300826+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'12","last_scrub_stamp":"2024-09-18T08:05:33.544135+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_clean_scrub_stamp":"2024-09-18T08:05:33.544135+0000","objects_scrubbed":5,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":25,"num_read_kb":16,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":231,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-09-18T08:07:48.108625+0000","last_change":"2024-09-18T08:05:39.906123+0000","last_active":"2024-09-18T08:07:48.108625+0000","last_peered":"2024-09-18T08:07:48.108625+0000","last_clean":"2024-09-18T08:07:48.108625+0000","last_became_active":"2024-09-18T08:02:47.884052+0000","last_became_peered":"2024-09-18T08:02:47.884052+0000","last_unstale":"2024-09-18T08:07:48.108625+0000","last_undegraded":"2024-09-18T08:07:48.108625+0000","last_fullsized":"2024-09-18T08:07:48.108625+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:05:39.906062+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:46.860092+0000","last_clean_scrub_stamp":"2024-09-18T08:05:39.906062+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:08:03.739004+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":216,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-09-18T08:07:47.307443+0000","last_change":"2024-09-18T08:05:06.828638+0000","last_active":"2024-09-18T08:07:47.307443+0000","last_peered":"2024-09-18T08:07:47.307443+0000","last_clean":"2024-09-18T08:07:47.307443+0000","last_became_active":"2024-09-18T08:02:51.902163+0000","last_became_peered":"2024-09-18T08:02:51.902163+0000","last_unstale":"2024-09-18T08:07:47.307443+0000","last_undegraded":"2024-09-18T08:07:47.307443+0000","last_fullsized":"2024-09-18T08:07:47.307443+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:05:06.828408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:50.886830+0000","last_clean_scrub_stamp":"2024-09-18T08:05:06.828408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:07:53.834534+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":221,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-09-18T08:07:48.103290+0000","last_change":"2024-09-18T08:07:44.113695+0000","last_active":"2024-09-18T08:07:48.103290+0000","last_peered":"2024-09-18T08:07:48.103290+0000","last_clean":"2024-09-18T08:07:48.103290+0000","last_became_active":"2024-09-18T08:02:53.907412+0000","last_became_peered":"2024-09-18T08:02:53.907412+0000","last_unstale":"2024-09-18T08:07:48.103290+0000","last_undegraded":"2024-09-18T08:07:48.103290+0000","last_fullsized":"2024-09-18T08:07:48.103290+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:07:44.113639+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_clean_scrub_stamp":"2024-09-18T08:07:44.113639+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:08:56.711179+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.8","version":"69'15","reported_seq":290,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-09-18T08:07:48.103585+0000","last_change":"2024-09-18T08:06:24.162625+0000","last_active":"2024-09-18T08:07:48.103585+0000","last_peered":"2024-09-18T08:07:48.103585+0000","last_clean":"2024-09-18T08:07:48.103585+0000","last_became_active":"2024-09-18T08:06:24.138259+0000","last_became_peered":"2024-09-18T08:06:24.138259+0000","last_unstale":"2024-09-18T08:07:48.103585+0000","last_undegraded":"2024-09-18T08:07:48.103585+0000","last_fullsized":"2024-09-18T08:07:48.103585+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'15","last_scrub_stamp":"2024-09-18T08:04:26.859044+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_clean_scrub_stamp":"2024-09-18T08:04:26.859044+0000","objects_scrubbed":7,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:08:36.637538+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.f","version":"62'2","reported_seq":251,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-09-18T08:07:48.109879+0000","last_change":"2024-09-18T08:07:43.540290+0000","last_active":"2024-09-18T08:07:48.109879+0000","last_peered":"2024-09-18T08:07:48.109879+0000","last_clean":"2024-09-18T08:07:48.109879+0000","last_became_active":"2024-09-18T08:02:47.881165+0000","last_became_peered":"2024-09-18T08:02:47.881165+0000","last_unstale":"2024-09-18T08:07:48.109879+0000","last_undegraded":"2024-09-18T08:07:48.109879+0000","last_fullsized":"2024-09-18T08:07:48.109879+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"62'2","last_scrub_stamp":"2024-09-18T08:07:43.540222+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:46.860092+0000","last_clean_scrub_stamp":"2024-09-18T08:07:43.540222+0000","objects_scrubbed":2,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:08:59.901037+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":4,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.9","version":"138'10","reported_seq":326,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-09-18T08:07:48.103236+0000","last_change":"2024-09-18T08:07:23.729774+0000","last_active":"2024-09-18T08:07:48.103236+0000","last_peered":"2024-09-18T08:07:48.103236+0000","last_clean":"2024-09-18T08:07:48.103236+0000","last_became_active":"2024-09-18T08:06:24.170165+0000","last_became_peered":"2024-09-18T08:06:24.170165+0000","last_unstale":"2024-09-18T08:07:48.103236+0000","last_undegraded":"2024-09-18T08:07:48.103236+0000","last_fullsized":"2024-09-18T08:07:48.103236+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"138'10","last_scrub_stamp":"2024-09-18T08:07:23.729715+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:50.886830+0000","last_clean_scrub_stamp":"2024-09-18T08:07:23.729715+0000","objects_scrubbed":1,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:08:49.848841+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":233,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-09-18T08:07:48.104206+0000","last_change":"2024-09-18T08:07:45.707631+0000","last_active":"2024-09-18T08:07:48.104206+0000","last_peered":"2024-09-18T08:07:48.104206+0000","last_clean":"2024-09-18T08:07:48.104206+0000","last_became_active":"2024-09-18T08:06:24.133119+0000","last_became_peered":"2024-09-18T08:06:24.133119+0000","last_unstale":"2024-09-18T08:07:48.104206+0000","last_undegraded":"2024-09-18T08:07:48.104206+0000","last_fullsized":"2024-09-18T08:07:48.104206+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:07:45.707562+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_clean_scrub_stamp":"2024-09-18T08:07:45.707562+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:08:53.653666+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":149,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-09-18T08:07:48.060374+0000","last_change":"2024-09-18T08:06:24.133955+0000","last_active":"2024-09-18T08:07:48.060374+0000","last_peered":"2024-09-18T08:07:48.060374+0000","last_clean":"2024-09-18T08:07:48.060374+0000","last_became_active":"2024-09-18T08:06:24.133644+0000","last_became_peered":"2024-09-18T08:06:24.133644+0000","last_unstale":"2024-09-18T08:07:48.060374+0000","last_undegraded":"2024-09-18T08:07:48.060374+0000","last_fullsized":"2024-09-18T08:07:48.060374+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:04:16.378205+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:46.860092+0000","last_clean_scrub_stamp":"2024-09-18T08:04:16.378205+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:08:37.572063+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.17","version":"69'6","reported_seq":245,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-09-18T08:07:48.103995+0000","last_change":"2024-09-18T08:07:19.309941+0000","last_active":"2024-09-18T08:07:48.103995+0000","last_peered":"2024-09-18T08:07:48.103995+0000","last_clean":"2024-09-18T08:07:48.103995+0000","last_became_active":"2024-09-18T08:02:49.895067+0000","last_became_peered":"2024-09-18T08:02:49.895067+0000","last_unstale":"2024-09-18T08:07:48.103995+0000","last_undegraded":"2024-09-18T08:07:48.103995+0000","last_fullsized":"2024-09-18T08:07:48.103995+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'6","last_scrub_stamp":"2024-09-18T08:07:19.309844+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_clean_scrub_stamp":"2024-09-18T08:07:19.309844+0000","objects_scrubbed":3,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:08:43.071787+0000","scrub_duration":26,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":9,"num_read_kb":6,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":223,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-09-18T08:07:48.101873+0000","last_change":"2024-09-18T08:05:45.812436+0000","last_active":"2024-09-18T08:07:48.101873+0000","last_peered":"2024-09-18T08:07:48.101873+0000","last_clean":"2024-09-18T08:07:48.101873+0000","last_became_active":"2024-09-18T08:02:51.912557+0000","last_became_peered":"2024-09-18T08:02:51.912557+0000","last_unstale":"2024-09-18T08:07:48.101873+0000","last_undegraded":"2024-09-18T08:07:48.101873+0000","last_fullsized":"2024-09-18T08:07:48.101873+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:05:45.812389+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:50.886830+0000","last_clean_scrub_stamp":"2024-09-18T08:05:45.812389+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,1],"acting":[5,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":234,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-09-18T08:07:48.107243+0000","last_change":"2024-09-18T08:06:43.808497+0000","last_active":"2024-09-18T08:07:48.107243+0000","last_peered":"2024-09-18T08:07:48.107243+0000","last_clean":"2024-09-18T08:07:48.107243+0000","last_became_active":"2024-09-18T08:06:24.145909+0000","last_became_peered":"2024-09-18T08:06:24.145909+0000","last_unstale":"2024-09-18T08:07:48.107243+0000","last_undegraded":"2024-09-18T08:07:48.107243+0000","last_fullsized":"2024-09-18T08:07:48.107243+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:06:43.808433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_clean_scrub_stamp":"2024-09-18T08:06:43.808433+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:07:52.050872+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"69'9","reported_seq":243,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-09-18T08:07:48.063826+0000","last_change":"2024-09-18T08:07:23.638373+0000","last_active":"2024-09-18T08:07:48.063826+0000","last_peered":"2024-09-18T08:07:48.063826+0000","last_clean":"2024-09-18T08:07:48.063826+0000","last_became_active":"2024-09-18T08:02:49.891567+0000","last_became_peered":"2024-09-18T08:02:49.891567+0000","last_unstale":"2024-09-18T08:07:48.063826+0000","last_undegraded":"2024-09-18T08:07:48.063826+0000","last_fullsized":"2024-09-18T08:07:48.063826+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'9","last_scrub_stamp":"2024-09-18T08:07:23.638294+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_clean_scrub_stamp":"2024-09-18T08:07:23.638294+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:08:43.060405+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":234,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-09-18T08:07:47.300677+0000","last_change":"2024-09-18T08:06:46.171693+0000","last_active":"2024-09-18T08:07:47.300677+0000","last_peered":"2024-09-18T08:07:47.300677+0000","last_clean":"2024-09-18T08:07:47.300677+0000","last_became_active":"2024-09-18T08:06:46.171291+0000","last_became_peered":"2024-09-18T08:06:46.171291+0000","last_unstale":"2024-09-18T08:07:47.300677+0000","last_undegraded":"2024-09-18T08:07:47.300677+0000","last_fullsized":"2024-09-18T08:07:47.300677+0000","mapping_epoch":143,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":144,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:06:13.214705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:46.860092+0000","last_clean_scrub_stamp":"2024-09-18T08:06:13.214705+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":216,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-09-18T08:07:48.102319+0000","last_change":"2024-09-18T08:05:42.061649+0000","last_active":"2024-09-18T08:07:48.102319+0000","last_peered":"2024-09-18T08:07:48.102319+0000","last_clean":"2024-09-18T08:07:48.102319+0000","last_became_active":"2024-09-18T08:02:51.907361+0000","last_became_peered":"2024-09-18T08:02:51.907361+0000","last_unstale":"2024-09-18T08:07:48.102319+0000","last_undegraded":"2024-09-18T08:07:48.102319+0000","last_fullsized":"2024-09-18T08:07:48.102319+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:05:42.061452+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:50.886830+0000","last_clean_scrub_stamp":"2024-09-18T08:05:42.061452+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:08:29.335464+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":213,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-09-18T08:07:47.306029+0000","last_change":"2024-09-18T08:05:07.780899+0000","last_active":"2024-09-18T08:07:47.306029+0000","last_peered":"2024-09-18T08:07:47.306029+0000","last_clean":"2024-09-18T08:07:47.306029+0000","last_became_active":"2024-09-18T08:02:53.908917+0000","last_became_peered":"2024-09-18T08:02:53.908917+0000","last_unstale":"2024-09-18T08:07:47.306029+0000","last_undegraded":"2024-09-18T08:07:47.306029+0000","last_fullsized":"2024-09-18T08:07:47.306029+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:05:07.780816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_clean_scrub_stamp":"2024-09-18T08:05:07.780816+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:07:56.898926+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.15","version":"69'9","reported_seq":243,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-09-18T08:07:48.114274+0000","last_change":"2024-09-18T08:05:36.896299+0000","last_active":"2024-09-18T08:07:48.114274+0000","last_peered":"2024-09-18T08:07:48.114274+0000","last_clean":"2024-09-18T08:07:48.114274+0000","last_became_active":"2024-09-18T08:02:49.890497+0000","last_became_peered":"2024-09-18T08:02:49.890497+0000","last_unstale":"2024-09-18T08:07:48.114274+0000","last_undegraded":"2024-09-18T08:07:48.114274+0000","last_fullsized":"2024-09-18T08:07:48.114274+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'9","last_scrub_stamp":"2024-09-18T08:05:36.896238+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_clean_scrub_stamp":"2024-09-18T08:05:36.896238+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:08:07.806082+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":231,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-09-18T08:07:48.060503+0000","last_change":"2024-09-18T08:06:12.437630+0000","last_active":"2024-09-18T08:07:48.060503+0000","last_peered":"2024-09-18T08:07:48.060503+0000","last_clean":"2024-09-18T08:07:48.060503+0000","last_became_active":"2024-09-18T08:02:47.875983+0000","last_became_peered":"2024-09-18T08:02:47.875983+0000","last_unstale":"2024-09-18T08:07:48.060503+0000","last_undegraded":"2024-09-18T08:07:48.060503+0000","last_fullsized":"2024-09-18T08:07:48.060503+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:06:12.437584+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:46.860092+0000","last_clean_scrub_stamp":"2024-09-18T08:06:12.437584+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.14","version":"69'8","reported_seq":293,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-09-18T08:07:48.102187+0000","last_change":"2024-09-18T08:07:16.241808+0000","last_active":"2024-09-18T08:07:48.102187+0000","last_peered":"2024-09-18T08:07:48.102187+0000","last_clean":"2024-09-18T08:07:48.102187+0000","last_became_active":"2024-09-18T08:02:51.909513+0000","last_became_peered":"2024-09-18T08:02:51.909513+0000","last_unstale":"2024-09-18T08:07:48.102187+0000","last_undegraded":"2024-09-18T08:07:48.102187+0000","last_fullsized":"2024-09-18T08:07:48.102187+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'8","last_scrub_stamp":"2024-09-18T08:07:16.241607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:50.886830+0000","last_clean_scrub_stamp":"2024-09-18T08:07:16.241607+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:08:24.745948+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":221,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-09-18T08:07:47.301684+0000","last_change":"2024-09-18T08:06:43.510575+0000","last_active":"2024-09-18T08:07:47.301684+0000","last_peered":"2024-09-18T08:07:47.301684+0000","last_clean":"2024-09-18T08:07:47.301684+0000","last_became_active":"2024-09-18T08:02:53.907428+0000","last_became_peered":"2024-09-18T08:02:53.907428+0000","last_unstale":"2024-09-18T08:07:47.301684+0000","last_undegraded":"2024-09-18T08:07:47.301684+0000","last_fullsized":"2024-09-18T08:07:47.301684+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:06:43.510513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_clean_scrub_stamp":"2024-09-18T08:06:43.510513+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:08:09.391404+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.14","version":"69'10","reported_seq":254,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-09-18T08:07:48.103770+0000","last_change":"2024-09-18T08:04:18.172364+0000","last_active":"2024-09-18T08:07:48.103770+0000","last_peered":"2024-09-18T08:07:48.103770+0000","last_clean":"2024-09-18T08:07:48.103770+0000","last_became_active":"2024-09-18T08:04:18.150968+0000","last_became_peered":"2024-09-18T08:04:18.150968+0000","last_unstale":"2024-09-18T08:07:48.103770+0000","last_undegraded":"2024-09-18T08:07:48.103770+0000","last_fullsized":"2024-09-18T08:07:48.103770+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'10","last_scrub_stamp":"2024-09-18T08:04:16.940778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_clean_scrub_stamp":"2024-09-18T08:04:16.940778+0000","objects_scrubbed":5,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:07:57.221037+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,7],"acting":[3,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":231,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-09-18T08:07:48.103300+0000","last_change":"2024-09-18T08:06:46.782184+0000","last_active":"2024-09-18T08:07:48.103300+0000","last_peered":"2024-09-18T08:07:48.103300+0000","last_clean":"2024-09-18T08:07:48.103300+0000","last_became_active":"2024-09-18T08:02:47.882065+0000","last_became_peered":"2024-09-18T08:02:47.882065+0000","last_unstale":"2024-09-18T08:07:48.103300+0000","last_undegraded":"2024-09-18T08:07:48.103300+0000","last_fullsized":"2024-09-18T08:07:48.103300+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:06:46.782109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:46.860092+0000","last_clean_scrub_stamp":"2024-09-18T08:06:46.782109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:08:11.695710+0000","scrub_duration":21,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.15","version":"69'8","reported_seq":300,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-09-18T08:07:48.101821+0000","last_change":"2024-09-18T08:06:45.555513+0000","last_active":"2024-09-18T08:07:48.101821+0000","last_peered":"2024-09-18T08:07:48.101821+0000","last_clean":"2024-09-18T08:07:48.101821+0000","last_became_active":"2024-09-18T08:02:51.914135+0000","last_became_peered":"2024-09-18T08:02:51.914135+0000","last_unstale":"2024-09-18T08:07:48.101821+0000","last_undegraded":"2024-09-18T08:07:48.101821+0000","last_fullsized":"2024-09-18T08:07:48.101821+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'8","last_scrub_stamp":"2024-09-18T08:06:45.555459+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:50.886830+0000","last_clean_scrub_stamp":"2024-09-18T08:06:45.555459+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:07:48.944224+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":220,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-09-18T08:07:48.062407+0000","last_change":"2024-09-18T08:07:14.805012+0000","last_active":"2024-09-18T08:07:48.062407+0000","last_peered":"2024-09-18T08:07:48.062407+0000","last_clean":"2024-09-18T08:07:48.062407+0000","last_became_active":"2024-09-18T08:02:53.912112+0000","last_became_peered":"2024-09-18T08:02:53.912112+0000","last_unstale":"2024-09-18T08:07:48.062407+0000","last_undegraded":"2024-09-18T08:07:48.062407+0000","last_fullsized":"2024-09-18T08:07:48.062407+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:07:14.804953+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_clean_scrub_stamp":"2024-09-18T08:07:14.804953+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:08:42.074755+0000","scrub_duration":46,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.13","version":"69'11","reported_seq":277,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-09-18T08:07:47.301128+0000","last_change":"2024-09-18T08:06:24.189911+0000","last_active":"2024-09-18T08:07:47.301128+0000","last_peered":"2024-09-18T08:07:47.301128+0000","last_clean":"2024-09-18T08:07:47.301128+0000","last_became_active":"2024-09-18T08:06:24.150800+0000","last_became_peered":"2024-09-18T08:06:24.150800+0000","last_unstale":"2024-09-18T08:07:47.301128+0000","last_undegraded":"2024-09-18T08:07:47.301128+0000","last_fullsized":"2024-09-18T08:07:47.301128+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'11","last_scrub_stamp":"2024-09-18T08:04:34.451379+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_clean_scrub_stamp":"2024-09-18T08:04:34.451379+0000","objects_scrubbed":5,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":246,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-09-18T08:07:47.301702+0000","last_change":"2024-09-18T08:06:46.502473+0000","last_active":"2024-09-18T08:07:47.301702+0000","last_peered":"2024-09-18T08:07:47.301702+0000","last_clean":"2024-09-18T08:07:47.301702+0000","last_became_active":"2024-09-18T08:06:24.145458+0000","last_became_peered":"2024-09-18T08:06:24.145458+0000","last_unstale":"2024-09-18T08:07:47.301702+0000","last_undegraded":"2024-09-18T08:07:47.301702+0000","last_fullsized":"2024-09-18T08:07:47.301702+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:06:46.502422+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:46.860092+0000","last_clean_scrub_stamp":"2024-09-18T08:06:46.502422+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:08:03.119365+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":223,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-09-18T08:07:47.301101+0000","last_change":"2024-09-18T08:05:29.896334+0000","last_active":"2024-09-18T08:07:47.301101+0000","last_peered":"2024-09-18T08:07:47.301101+0000","last_clean":"2024-09-18T08:07:47.301101+0000","last_became_active":"2024-09-18T08:02:51.910752+0000","last_became_peered":"2024-09-18T08:02:51.910752+0000","last_unstale":"2024-09-18T08:07:47.301101+0000","last_undegraded":"2024-09-18T08:07:47.301101+0000","last_fullsized":"2024-09-18T08:07:47.301101+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:05:29.888927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:50.886830+0000","last_clean_scrub_stamp":"2024-09-18T08:05:29.888927+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:07:57.612901+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":221,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-09-18T08:07:48.102673+0000","last_change":"2024-09-18T08:06:11.249403+0000","last_active":"2024-09-18T08:07:48.102673+0000","last_peered":"2024-09-18T08:07:48.102673+0000","last_clean":"2024-09-18T08:07:48.102673+0000","last_became_active":"2024-09-18T08:02:53.911429+0000","last_became_peered":"2024-09-18T08:02:53.911429+0000","last_unstale":"2024-09-18T08:07:48.102673+0000","last_undegraded":"2024-09-18T08:07:48.102673+0000","last_fullsized":"2024-09-18T08:07:48.102673+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:06:11.249317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_clean_scrub_stamp":"2024-09-18T08:06:11.249317+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.12","version":"69'9","reported_seq":261,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-09-18T08:07:47.301536+0000","last_change":"2024-09-18T08:06:24.152803+0000","last_active":"2024-09-18T08:07:47.301536+0000","last_peered":"2024-09-18T08:07:47.301536+0000","last_clean":"2024-09-18T08:07:47.301536+0000","last_became_active":"2024-09-18T08:06:24.132270+0000","last_became_peered":"2024-09-18T08:06:24.132270+0000","last_unstale":"2024-09-18T08:07:47.301536+0000","last_undegraded":"2024-09-18T08:07:47.301536+0000","last_fullsized":"2024-09-18T08:07:47.301536+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_clean_scrub_stamp":"2024-09-18T08:02:48.870718+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:08:30.809078+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":231,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-09-18T08:07:48.107564+0000","last_change":"2024-09-18T08:05:30.820244+0000","last_active":"2024-09-18T08:07:48.107564+0000","last_peered":"2024-09-18T08:07:48.107564+0000","last_clean":"2024-09-18T08:07:48.107564+0000","last_became_active":"2024-09-18T08:02:47.882096+0000","last_became_peered":"2024-09-18T08:02:47.882096+0000","last_unstale":"2024-09-18T08:07:48.107564+0000","last_undegraded":"2024-09-18T08:07:48.107564+0000","last_fullsized":"2024-09-18T08:07:48.107564+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:05:30.820200+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:46.860092+0000","last_clean_scrub_stamp":"2024-09-18T08:05:30.820200+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:07:54.767056+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":224,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-09-18T08:07:48.101804+0000","last_change":"2024-09-18T08:07:42.165434+0000","last_active":"2024-09-18T08:07:48.101804+0000","last_peered":"2024-09-18T08:07:48.101804+0000","last_clean":"2024-09-18T08:07:48.101804+0000","last_became_active":"2024-09-18T08:02:51.906277+0000","last_became_peered":"2024-09-18T08:02:51.906277+0000","last_unstale":"2024-09-18T08:07:48.101804+0000","last_undegraded":"2024-09-18T08:07:48.101804+0000","last_fullsized":"2024-09-18T08:07:48.101804+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:07:42.165388+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:50.886830+0000","last_clean_scrub_stamp":"2024-09-18T08:07:42.165388+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:09:11.035445+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.10","version":"69'1","reported_seq":208,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-09-18T08:07:48.060093+0000","last_change":"2024-09-18T08:05:51.429631+0000","last_active":"2024-09-18T08:07:48.060093+0000","last_peered":"2024-09-18T08:07:48.060093+0000","last_clean":"2024-09-18T08:07:48.060093+0000","last_became_active":"2024-09-18T08:02:53.912699+0000","last_became_peered":"2024-09-18T08:02:53.912699+0000","last_unstale":"2024-09-18T08:07:48.060093+0000","last_undegraded":"2024-09-18T08:07:48.060093+0000","last_fullsized":"2024-09-18T08:07:48.060093+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_clean_scrub_stamp":"2024-09-18T08:02:52.890475+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:08:28.573050+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.11","version":"69'11","reported_seq":293,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-09-18T08:07:48.102770+0000","last_change":"2024-09-18T08:07:18.282247+0000","last_active":"2024-09-18T08:07:48.102770+0000","last_peered":"2024-09-18T08:07:48.102770+0000","last_clean":"2024-09-18T08:07:48.102770+0000","last_became_active":"2024-09-18T08:06:24.132416+0000","last_became_peered":"2024-09-18T08:06:24.132416+0000","last_unstale":"2024-09-18T08:07:48.102770+0000","last_undegraded":"2024-09-18T08:07:48.102770+0000","last_fullsized":"2024-09-18T08:07:48.102770+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'11","last_scrub_stamp":"2024-09-18T08:07:18.282155+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_clean_scrub_stamp":"2024-09-18T08:07:18.282155+0000","objects_scrubbed":5,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:08:45.161997+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":231,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-09-18T08:07:48.124925+0000","last_change":"2024-09-18T08:06:18.168254+0000","last_active":"2024-09-18T08:07:48.124925+0000","last_peered":"2024-09-18T08:07:48.124925+0000","last_clean":"2024-09-18T08:07:48.124925+0000","last_became_active":"2024-09-18T08:02:47.878886+0000","last_became_peered":"2024-09-18T08:02:47.878886+0000","last_unstale":"2024-09-18T08:07:48.124925+0000","last_undegraded":"2024-09-18T08:07:48.124925+0000","last_fullsized":"2024-09-18T08:07:48.124925+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:06:18.168206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:46.860092+0000","last_clean_scrub_stamp":"2024-09-18T08:06:18.168206+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":245,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-09-18T08:07:48.106006+0000","last_change":"2024-09-18T08:07:21.724590+0000","last_active":"2024-09-18T08:07:48.106006+0000","last_peered":"2024-09-18T08:07:48.106006+0000","last_clean":"2024-09-18T08:07:48.106006+0000","last_became_active":"2024-09-18T08:06:24.148302+0000","last_became_peered":"2024-09-18T08:06:24.148302+0000","last_unstale":"2024-09-18T08:07:48.106006+0000","last_undegraded":"2024-09-18T08:07:48.106006+0000","last_fullsized":"2024-09-18T08:07:48.106006+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:07:21.724545+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:50.886830+0000","last_clean_scrub_stamp":"2024-09-18T08:07:21.724545+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:08:40.457830+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":234,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-09-18T08:07:48.218010+0000","last_change":"2024-09-18T08:07:48.218010+0000","last_active":"2024-09-18T08:07:48.218010+0000","last_peered":"2024-09-18T08:07:48.218010+0000","last_clean":"2024-09-18T08:07:48.218010+0000","last_became_active":"2024-09-18T08:06:24.128582+0000","last_became_peered":"2024-09-18T08:06:24.128582+0000","last_unstale":"2024-09-18T08:07:48.218010+0000","last_undegraded":"2024-09-18T08:07:48.218010+0000","last_fullsized":"2024-09-18T08:07:48.218010+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:07:48.217964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_clean_scrub_stamp":"2024-09-18T08:07:48.217964+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:09:07.458601+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.10","version":"69'4","reported_seq":260,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-09-18T08:07:48.102166+0000","last_change":"2024-09-18T08:07:41.210418+0000","last_active":"2024-09-18T08:07:48.102166+0000","last_peered":"2024-09-18T08:07:48.102166+0000","last_clean":"2024-09-18T08:07:48.102166+0000","last_became_active":"2024-09-18T08:06:24.131309+0000","last_became_peered":"2024-09-18T08:06:24.131309+0000","last_unstale":"2024-09-18T08:07:48.102166+0000","last_undegraded":"2024-09-18T08:07:48.102166+0000","last_fullsized":"2024-09-18T08:07:48.102166+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'4","last_scrub_stamp":"2024-09-18T08:07:41.210347+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_clean_scrub_stamp":"2024-09-18T08:07:41.210347+0000","objects_scrubbed":2,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:08:57.793182+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":6,"num_read_kb":4,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":231,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-09-18T08:07:48.065943+0000","last_change":"2024-09-18T08:06:10.457663+0000","last_active":"2024-09-18T08:07:48.065943+0000","last_peered":"2024-09-18T08:07:48.065943+0000","last_clean":"2024-09-18T08:07:48.065943+0000","last_became_active":"2024-09-18T08:02:47.877616+0000","last_became_peered":"2024-09-18T08:02:47.877616+0000","last_unstale":"2024-09-18T08:07:48.065943+0000","last_undegraded":"2024-09-18T08:07:48.065943+0000","last_fullsized":"2024-09-18T08:07:48.065943+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:06:10.457502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:46.860092+0000","last_clean_scrub_stamp":"2024-09-18T08:06:10.457502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":166,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-09-18T08:07:47.300468+0000","last_change":"2024-09-18T08:07:17.605626+0000","last_active":"2024-09-18T08:07:47.300468+0000","last_peered":"2024-09-18T08:07:47.300468+0000","last_clean":"2024-09-18T08:07:47.300468+0000","last_became_active":"2024-09-18T08:06:24.166698+0000","last_became_peered":"2024-09-18T08:06:24.166698+0000","last_unstale":"2024-09-18T08:07:47.300468+0000","last_undegraded":"2024-09-18T08:07:47.300468+0000","last_fullsized":"2024-09-18T08:07:47.300468+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:07:17.605562+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:50.886830+0000","last_clean_scrub_stamp":"2024-09-18T08:07:17.605562+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:08:24.477286+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,7],"acting":[4,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":228,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-09-18T08:07:48.104253+0000","last_change":"2024-09-18T08:06:47.791172+0000","last_active":"2024-09-18T08:07:48.104253+0000","last_peered":"2024-09-18T08:07:48.104253+0000","last_clean":"2024-09-18T08:07:48.104253+0000","last_became_active":"2024-09-18T08:02:53.909146+0000","last_became_peered":"2024-09-18T08:02:53.909146+0000","last_unstale":"2024-09-18T08:07:48.104253+0000","last_undegraded":"2024-09-18T08:07:48.104253+0000","last_fullsized":"2024-09-18T08:07:48.104253+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:06:47.791091+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_clean_scrub_stamp":"2024-09-18T08:06:47.791091+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:08:13.139318+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,4],"acting":[7,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":203,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-09-18T08:07:47.303354+0000","last_change":"2024-09-18T08:02:53.907055+0000","last_active":"2024-09-18T08:07:47.303354+0000","last_peered":"2024-09-18T08:07:47.303354+0000","last_clean":"2024-09-18T08:07:47.303354+0000","last_became_active":"2024-09-18T08:02:53.906726+0000","last_became_peered":"2024-09-18T08:02:53.906726+0000","last_unstale":"2024-09-18T08:07:47.303354+0000","last_undegraded":"2024-09-18T08:07:47.303354+0000","last_fullsized":"2024-09-18T08:07:47.303354+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_clean_scrub_stamp":"2024-09-18T08:02:52.890475+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:08:29.803008+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":232,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-09-18T08:07:48.102381+0000","last_change":"2024-09-18T08:07:17.277964+0000","last_active":"2024-09-18T08:07:48.102381+0000","last_peered":"2024-09-18T08:07:48.102381+0000","last_clean":"2024-09-18T08:07:48.102381+0000","last_became_active":"2024-09-18T08:02:47.878922+0000","last_became_peered":"2024-09-18T08:02:47.878922+0000","last_unstale":"2024-09-18T08:07:48.102381+0000","last_undegraded":"2024-09-18T08:07:48.102381+0000","last_fullsized":"2024-09-18T08:07:48.102381+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:07:17.277894+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:46.860092+0000","last_clean_scrub_stamp":"2024-09-18T08:07:17.277894+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:08:24.233327+0000","scrub_duration":16,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"69'11","reported_seq":220,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-09-18T08:07:48.113974+0000","last_change":"2024-09-18T08:06:24.164086+0000","last_active":"2024-09-18T08:07:48.113974+0000","last_peered":"2024-09-18T08:07:48.113974+0000","last_clean":"2024-09-18T08:07:48.113974+0000","last_became_active":"2024-09-18T08:06:24.135232+0000","last_became_peered":"2024-09-18T08:06:24.135232+0000","last_unstale":"2024-09-18T08:07:48.113974+0000","last_undegraded":"2024-09-18T08:07:48.113974+0000","last_fullsized":"2024-09-18T08:07:48.113974+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'11","last_scrub_stamp":"2024-09-18T08:06:12.967462+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_clean_scrub_stamp":"2024-09-18T08:06:12.967462+0000","objects_scrubbed":5,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":19,"num_read_kb":12,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,1],"acting":[5,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":215,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-09-18T08:07:48.059673+0000","last_change":"2024-09-18T08:04:26.117117+0000","last_active":"2024-09-18T08:07:48.059673+0000","last_peered":"2024-09-18T08:07:48.059673+0000","last_clean":"2024-09-18T08:07:48.059673+0000","last_became_active":"2024-09-18T08:02:51.900300+0000","last_became_peered":"2024-09-18T08:02:51.900300+0000","last_unstale":"2024-09-18T08:07:48.059673+0000","last_undegraded":"2024-09-18T08:07:48.059673+0000","last_fullsized":"2024-09-18T08:07:48.059673+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:04:26.117051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:50.886830+0000","last_clean_scrub_stamp":"2024-09-18T08:04:26.117051+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:07:55.738092+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":416,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":71,"ondisk_log_size":71,"up":96,"acting":96,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":178,"num_object_clones":0,"num_object_copies":534,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":178,"num_whiteouts":0,"num_read":694,"num_read_kb":451,"num_write":417,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":156,"num_bytes_recovered":2618,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":417792,"data_stored":11106,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":396,"ondisk_log_size":396,"up":96,"acting":96,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":1613,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":24,"num_read_kb":24,"num_write":10,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":4839,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":6,"ondisk_log_size":6,"up":96,"acting":96,"num_store_stats":8},{"poolid":2,"num_pg":3,"stat_sum":{"num_bytes":408,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":69,"num_read_kb":64,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":38,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":8,"ondisk_log_size":8,"up":9,"acting":9,"num_store_stats":7},{"poolid":1,"num_pg":11,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":114,"num_read_kb":222,"num_write":113,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1395415,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":869,"ondisk_log_size":869,"up":33,"acting":33,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":56,"seq":240518168640,"num_pgs":62,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":100372,"kb_used_data":1156,"kb_used_omap":1,"kb_used_meta":99198,"kb_avail":93644780,"statfs":{"total":95995035648,"available":95892254720,"internally_reserved":0,"allocated":1183744,"data_stored":775542,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":101579210},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[]},{"osd":6,"up_from":47,"seq":201863462982,"num_pgs":53,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":100032,"kb_used_data":816,"kb_used_omap":1,"kb_used_meta":99198,"kb_avail":93645120,"statfs":{"total":95995035648,"available":95892602880,"internally_reserved":0,"allocated":835584,"data_stored":446777,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":101579206},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":12,"apply_latency_ms":12,"commit_latency_ns":12000000,"apply_latency_ns":12000000},"alerts":[]},{"osd":5,"up_from":39,"seq":167503724620,"num_pgs":55,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":90896,"kb_used_data":1408,"kb_used_omap":1,"kb_used_meta":89470,"kb_avail":93654256,"statfs":{"total":95995035648,"available":95901958144,"internally_reserved":0,"allocated":1441792,"data_stored":1037953,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":91617734},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":4,"up_from":32,"seq":137438953553,"num_pgs":66,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":100100,"kb_used_data":876,"kb_used_omap":1,"kb_used_meta":99198,"kb_avail":93645052,"statfs":{"total":95995035648,"available":95892533248,"internally_reserved":0,"allocated":897024,"data_stored":461672,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":101579205},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":3,"up_from":26,"seq":111669149784,"num_pgs":70,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":185356,"kb_used_data":1460,"kb_used_omap":1,"kb_used_meta":183870,"kb_avail":93559796,"statfs":{"total":95995035648,"available":95805231104,"internally_reserved":0,"allocated":1495040,"data_stored":1051333,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":188283336},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":13,"apply_latency_ms":13,"commit_latency_ns":13000000,"apply_latency_ns":13000000},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411422,"num_pgs":43,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":122548,"kb_used_data":1116,"kb_used_omap":1,"kb_used_meta":121406,"kb_avail":93622604,"statfs":{"total":95995035648,"available":95869546496,"internally_reserved":0,"allocated":1142784,"data_stored":723676,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":124320197},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":14,"apply_latency_ms":14,"commit_latency_ns":14000000,"apply_latency_ns":14000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574946,"num_pgs":56,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":130656,"kb_used_data":908,"kb_used_omap":1,"kb_used_meta":129726,"kb_avail":93614496,"statfs":{"total":95995035648,"available":95861243904,"internally_reserved":0,"allocated":929792,"data_stored":462971,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":132839877},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738473,"num_pgs":57,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":144480,"kb_used_data":848,"kb_used_omap":1,"kb_used_meta":143614,"kb_avail":93600672,"statfs":{"total":95995035648,"available":95847088128,"internally_reserved":0,"allocated":868352,"data_stored":448577,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":147061189},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":266240,"data_stored":262688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1131,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1521,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":436,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":46,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1177,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":436,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":77824,"data_stored":2008,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1128,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":1898,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1474,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":990,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1034,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1254,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":403,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":416,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":403,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T08:07:50.743 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6 2024-09-18T08:07:50.769 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:07:50.769 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:07:50.770 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6' 2024-09-18T08:07:50.871 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1 2024-09-18T08:07:50.894 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:07:50.895 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:07:50.895 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1' 2024-09-18T08:07:50.996 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7 2024-09-18T08:07:51.021 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:07:51.022 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:07:51.022 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7' 2024-09-18T08:07:51.123 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2 2024-09-18T08:07:51.146 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:07:51.147 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:07:51.147 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2' 2024-09-18T08:07:51.248 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6 2024-09-18T08:07:51.273 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:07:51.274 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:07:51.274 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6' 2024-09-18T08:07:51.375 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2 2024-09-18T08:07:51.398 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:07:51.398 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:07:51.399 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2' 2024-09-18T08:07:51.480 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:07:51 smithi007 ceph-mon[29459]: from='client.15171 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-18T08:07:51.480 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:07:51 smithi007 ceph-mon[29459]: 6.3 scrub starts 2024-09-18T08:07:51.480 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:07:51 smithi007 ceph-mon[29459]: 6.3 scrub ok 2024-09-18T08:07:51.481 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:07:51 smithi007 ceph-mon[39249]: from='client.15171 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-18T08:07:51.481 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:07:51 smithi007 ceph-mon[39249]: 6.3 scrub starts 2024-09-18T08:07:51.481 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:07:51 smithi007 ceph-mon[39249]: 6.3 scrub ok 2024-09-18T08:07:51.500 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3 2024-09-18T08:07:51.523 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:07:51.524 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:07:51.524 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3' 2024-09-18T08:07:51.625 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2 2024-09-18T08:07:51.648 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:07:51.649 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:07:51.650 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2' 2024-09-18T08:07:51.656 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:07:51 smithi050 ceph-mon[32371]: from='client.15171 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-18T08:07:51.656 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:07:51 smithi050 ceph-mon[32371]: 6.3 scrub starts 2024-09-18T08:07:51.656 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:07:51 smithi050 ceph-mon[32371]: 6.3 scrub ok 2024-09-18T08:07:51.696 INFO:teuthology.orchestra.run.smithi007.stdout:ERROR: (22) Invalid argument 2024-09-18T08:07:51.696 INFO:teuthology.orchestra.run.smithi007.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-18T08:07:51.751 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6 2024-09-18T08:07:51.777 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:07:51.778 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:07:51.778 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6' 2024-09-18T08:07:51.879 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5 2024-09-18T08:07:51.906 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:07:51.906 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:07:51.907 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5' 2024-09-18T08:07:51.933 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid 779349ca-7593-11ef-bceb-c7b262605968 -- ceph osd set noscrub 2024-09-18T08:07:52.008 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2 2024-09-18T08:07:52.030 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:07:52.030 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:07:52.031 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2' 2024-09-18T08:07:52.132 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1 2024-09-18T08:07:52.155 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:07:52.155 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:07:52.156 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1' 2024-09-18T08:07:52.257 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0 2024-09-18T08:07:52.279 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:07:52.280 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:07:52.280 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0' 2024-09-18T08:07:52.320 INFO:teuthology.orchestra.run.smithi007.stderr:pool 'unique_pool_1' created 2024-09-18T08:07:52.381 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3 2024-09-18T08:07:52.403 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:07:52.404 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:07:52.404 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3' 2024-09-18T08:07:52.480 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:07:52 smithi007 ceph-mon[29459]: pgmap v231: 142 pgs: 142 active+clean; 583 KiB data, 952 MiB used, 714 GiB / 715 GiB avail; 682 B/s rd, 0 op/s 2024-09-18T08:07:52.480 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:07:52 smithi007 ceph-mon[29459]: from='client.? 172.21.15.7:0/2582181115' entity='client.admin' cmd={"prefix": "osd pool create", "pool": "unique_pool_1", "pg_num": 16} : dispatch 2024-09-18T08:07:52.480 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:07:52 smithi007 ceph-mon[29459]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool create", "pool": "unique_pool_1", "pg_num": 16} : dispatch 2024-09-18T08:07:52.481 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:07:52 smithi007 ceph-mon[39249]: pgmap v231: 142 pgs: 142 active+clean; 583 KiB data, 952 MiB used, 714 GiB / 715 GiB avail; 682 B/s rd, 0 op/s 2024-09-18T08:07:52.481 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:07:52 smithi007 ceph-mon[39249]: from='client.? 172.21.15.7:0/2582181115' entity='client.admin' cmd={"prefix": "osd pool create", "pool": "unique_pool_1", "pg_num": 16} : dispatch 2024-09-18T08:07:52.481 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:07:52 smithi007 ceph-mon[39249]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool create", "pool": "unique_pool_1", "pg_num": 16} : dispatch 2024-09-18T08:07:52.505 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4 2024-09-18T08:07:52.533 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:07:52.534 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:07:52.534 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4' 2024-09-18T08:07:52.553 DEBUG:teuthology.orchestra.run.smithi050:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid 779349ca-7593-11ef-bceb-c7b262605968 -- ceph daemon osd.4 dump_ops_in_flight 2024-09-18T08:07:52.599 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:07:52 smithi050 ceph-mon[32371]: pgmap v231: 142 pgs: 142 active+clean; 583 KiB data, 952 MiB used, 714 GiB / 715 GiB avail; 682 B/s rd, 0 op/s 2024-09-18T08:07:52.599 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:07:52 smithi050 ceph-mon[32371]: from='client.? 172.21.15.7:0/2582181115' entity='client.admin' cmd={"prefix": "osd pool create", "pool": "unique_pool_1", "pg_num": 16} : dispatch 2024-09-18T08:07:52.600 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:07:52 smithi050 ceph-mon[32371]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool create", "pool": "unique_pool_1", "pg_num": 16} : dispatch 2024-09-18T08:07:52.635 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2 2024-09-18T08:07:52.657 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:07:52.657 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:07:52.658 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2' 2024-09-18T08:07:52.759 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0 2024-09-18T08:07:52.780 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:07:52.781 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:07:52.781 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0' 2024-09-18T08:07:52.840 INFO:teuthology.orchestra.run.smithi050.stderr:Inferring config /var/lib/ceph/779349ca-7593-11ef-bceb-c7b262605968/mon.b/config 2024-09-18T08:07:52.882 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7 2024-09-18T08:07:52.904 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:07:52.905 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:07:52.906 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7' 2024-09-18T08:07:53.006 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3 2024-09-18T08:07:53.029 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:07:53.029 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:07:53.030 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3' 2024-09-18T08:07:53.131 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5 2024-09-18T08:07:53.153 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:07:53.154 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:07:53.154 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5' 2024-09-18T08:07:53.176 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid 779349ca-7593-11ef-bceb-c7b262605968 -- ceph osd pool application enable unique_pool_1 rados --yes-i-really-mean-it || true 2024-09-18T08:07:53.254 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6 2024-09-18T08:07:53.278 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:07:53.279 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:07:53.279 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6' 2024-09-18T08:07:53.374 INFO:teuthology.orchestra.run.smithi050.stdout:ERROR: (22) Invalid argument 2024-09-18T08:07:53.374 INFO:teuthology.orchestra.run.smithi050.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-18T08:07:53.380 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2 2024-09-18T08:07:53.402 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:07:53.403 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:07:53.403 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2' 2024-09-18T08:07:53.480 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:07:53 smithi007 ceph-mon[29459]: 5.f scrub starts 2024-09-18T08:07:53.480 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:07:53 smithi007 ceph-mon[29459]: 5.f scrub ok 2024-09-18T08:07:53.480 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:07:53 smithi007 ceph-mon[29459]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "unique_pool_1", "pg_num": 16}]': finished 2024-09-18T08:07:53.480 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:07:53 smithi007 ceph-mon[29459]: osdmap e165: 8 total, 8 up, 8 in 2024-09-18T08:07:53.482 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:07:53 smithi007 ceph-mon[29459]: 3.17 scrub starts 2024-09-18T08:07:53.482 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:07:53 smithi007 ceph-mon[29459]: 3.17 scrub ok 2024-09-18T08:07:53.482 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:07:53 smithi007 ceph-mon[39249]: 5.f scrub starts 2024-09-18T08:07:53.482 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:07:53 smithi007 ceph-mon[39249]: 5.f scrub ok 2024-09-18T08:07:53.482 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:07:53 smithi007 ceph-mon[39249]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "unique_pool_1", "pg_num": 16}]': finished 2024-09-18T08:07:53.482 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:07:53 smithi007 ceph-mon[39249]: osdmap e165: 8 total, 8 up, 8 in 2024-09-18T08:07:53.482 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:07:53 smithi007 ceph-mon[39249]: 3.17 scrub starts 2024-09-18T08:07:53.482 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:07:53 smithi007 ceph-mon[39249]: 3.17 scrub ok 2024-09-18T08:07:53.504 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4 2024-09-18T08:07:53.526 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:07:53.527 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:07:53.527 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4' 2024-09-18T08:07:53.537 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:07:53 smithi050 ceph-mon[32371]: 5.f scrub starts 2024-09-18T08:07:53.537 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:07:53 smithi050 ceph-mon[32371]: 5.f scrub ok 2024-09-18T08:07:53.537 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:07:53 smithi050 ceph-mon[32371]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "unique_pool_1", "pg_num": 16}]': finished 2024-09-18T08:07:53.537 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:07:53 smithi050 ceph-mon[32371]: osdmap e165: 8 total, 8 up, 8 in 2024-09-18T08:07:53.537 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:07:53 smithi050 ceph-mon[32371]: 3.17 scrub starts 2024-09-18T08:07:53.537 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:07:53 smithi050 ceph-mon[32371]: 3.17 scrub ok 2024-09-18T08:07:53.628 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7 2024-09-18T08:07:53.650 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:07:53.650 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:07:53.651 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7' 2024-09-18T08:07:53.751 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6 2024-09-18T08:07:53.773 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:07:53.774 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:07:53.774 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6' 2024-09-18T08:07:53.875 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7 2024-09-18T08:07:53.897 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:07:53.897 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:07:53.898 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7' 2024-09-18T08:07:53.975 INFO:teuthology.orchestra.run.smithi007.stdout:osd.0: {} 2024-09-18T08:07:53.976 INFO:teuthology.orchestra.run.smithi007.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-18T08:07:53.991 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/779349ca-7593-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T08:07:53.995 INFO:teuthology.orchestra.run.smithi007.stdout:osd.1: {} 2024-09-18T08:07:53.995 INFO:teuthology.orchestra.run.smithi007.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-18T08:07:53.998 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1 2024-09-18T08:07:54.016 INFO:teuthology.orchestra.run.smithi007.stdout:osd.2: {} 2024-09-18T08:07:54.016 INFO:teuthology.orchestra.run.smithi007.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-18T08:07:54.023 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:07:54.024 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:07:54.024 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1' 2024-09-18T08:07:54.038 INFO:teuthology.orchestra.run.smithi007.stdout:osd.3: {} 2024-09-18T08:07:54.038 INFO:teuthology.orchestra.run.smithi007.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-18T08:07:54.055 INFO:teuthology.orchestra.run.smithi007.stdout:osd.4: {} 2024-09-18T08:07:54.055 INFO:teuthology.orchestra.run.smithi007.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-18T08:07:54.071 INFO:teuthology.orchestra.run.smithi007.stdout:osd.5: {} 2024-09-18T08:07:54.072 INFO:teuthology.orchestra.run.smithi007.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-18T08:07:54.088 INFO:teuthology.orchestra.run.smithi007.stdout:osd.6: {} 2024-09-18T08:07:54.088 INFO:teuthology.orchestra.run.smithi007.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-18T08:07:54.104 INFO:teuthology.orchestra.run.smithi007.stdout:osd.7: {} 2024-09-18T08:07:54.105 INFO:teuthology.orchestra.run.smithi007.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-18T08:07:54.125 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2 2024-09-18T08:07:54.149 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:07:54.150 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:07:54.150 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2' 2024-09-18T08:07:54.177 DEBUG:teuthology.orchestra.run.smithi050:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid 779349ca-7593-11ef-bceb-c7b262605968 -- ceph daemon osd.4 dump_blocked_ops 2024-09-18T08:07:54.251 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1 2024-09-18T08:07:54.275 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:07:54.276 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:07:54.276 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1' 2024-09-18T08:07:54.377 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0 2024-09-18T08:07:54.399 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:07:54.399 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:07:54.400 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0' 2024-09-18T08:07:54.440 INFO:teuthology.orchestra.run.smithi050.stderr:Inferring config /var/lib/ceph/779349ca-7593-11ef-bceb-c7b262605968/mon.b/config 2024-09-18T08:07:54.481 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:07:54 smithi007 ceph-mon[29459]: pgmap v233: 158 pgs: 6 creating+peering, 10 unknown, 142 active+clean; 583 KiB data, 952 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-18T08:07:54.481 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:07:54 smithi007 ceph-mon[29459]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-18T08:07:54.481 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:07:54 smithi007 ceph-mon[29459]: osdmap e166: 8 total, 8 up, 8 in 2024-09-18T08:07:54.481 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:07:54 smithi007 ceph-mon[29459]: 5.5 scrub starts 2024-09-18T08:07:54.481 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:07:54 smithi007 ceph-mon[29459]: 5.5 scrub ok 2024-09-18T08:07:54.481 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:07:54 smithi007 ceph-mon[29459]: 4.13 scrub starts 2024-09-18T08:07:54.481 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:07:54 smithi007 ceph-mon[29459]: 4.13 scrub ok 2024-09-18T08:07:54.481 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:07:54 smithi007 ceph-mon[29459]: from='client.? 172.21.15.7:0/3248998911' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-18T08:07:54.481 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:07:54 smithi007 ceph-mon[29459]: osdmap e167: 8 total, 8 up, 8 in 2024-09-18T08:07:54.482 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:07:54 smithi007 ceph-mon[39249]: pgmap v233: 158 pgs: 6 creating+peering, 10 unknown, 142 active+clean; 583 KiB data, 952 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-18T08:07:54.482 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:07:54 smithi007 ceph-mon[39249]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-18T08:07:54.482 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:07:54 smithi007 ceph-mon[39249]: osdmap e166: 8 total, 8 up, 8 in 2024-09-18T08:07:54.482 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:07:54 smithi007 ceph-mon[39249]: 5.5 scrub starts 2024-09-18T08:07:54.482 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:07:54 smithi007 ceph-mon[39249]: 5.5 scrub ok 2024-09-18T08:07:54.482 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:07:54 smithi007 ceph-mon[39249]: 4.13 scrub starts 2024-09-18T08:07:54.482 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:07:54 smithi007 ceph-mon[39249]: 4.13 scrub ok 2024-09-18T08:07:54.482 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:07:54 smithi007 ceph-mon[39249]: from='client.? 172.21.15.7:0/3248998911' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-18T08:07:54.482 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:07:54 smithi007 ceph-mon[39249]: osdmap e167: 8 total, 8 up, 8 in 2024-09-18T08:07:54.501 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0 2024-09-18T08:07:54.523 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:07:54.524 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:07:54.524 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0' 2024-09-18T08:07:54.591 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:07:54 smithi050 ceph-mon[32371]: pgmap v233: 158 pgs: 6 creating+peering, 10 unknown, 142 active+clean; 583 KiB data, 952 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-18T08:07:54.591 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:07:54 smithi050 ceph-mon[32371]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-18T08:07:54.591 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:07:54 smithi050 ceph-mon[32371]: osdmap e166: 8 total, 8 up, 8 in 2024-09-18T08:07:54.591 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:07:54 smithi050 ceph-mon[32371]: 5.5 scrub starts 2024-09-18T08:07:54.591 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:07:54 smithi050 ceph-mon[32371]: 5.5 scrub ok 2024-09-18T08:07:54.592 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:07:54 smithi050 ceph-mon[32371]: 4.13 scrub starts 2024-09-18T08:07:54.592 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:07:54 smithi050 ceph-mon[32371]: 4.13 scrub ok 2024-09-18T08:07:54.592 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:07:54 smithi050 ceph-mon[32371]: from='client.? 172.21.15.7:0/3248998911' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-18T08:07:54.592 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:07:54 smithi050 ceph-mon[32371]: osdmap e167: 8 total, 8 up, 8 in 2024-09-18T08:07:54.624 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4 2024-09-18T08:07:54.647 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:07:54.648 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:07:54.648 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4' 2024-09-18T08:07:54.749 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3 2024-09-18T08:07:54.772 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:07:54.773 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:07:54.773 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3' 2024-09-18T08:07:54.874 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7 2024-09-18T08:07:54.897 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:07:54.898 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:07:54.898 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7' 2024-09-18T08:07:54.999 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2 2024-09-18T08:07:55.021 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:07:55.021 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:07:55.022 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2' 2024-09-18T08:07:55.069 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2024-09-18T08:07:55.069 INFO:teuthology.orchestra.run.smithi050.stdout: "ops": [], 2024-09-18T08:07:55.069 INFO:teuthology.orchestra.run.smithi050.stdout: "complaint_time": 30, 2024-09-18T08:07:55.069 INFO:teuthology.orchestra.run.smithi050.stdout: "num_blocked_ops": 0 2024-09-18T08:07:55.069 INFO:teuthology.orchestra.run.smithi050.stdout:} 2024-09-18T08:07:55.122 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2 2024-09-18T08:07:55.145 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:07:55.145 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:07:55.146 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2' 2024-09-18T08:07:55.184 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/779349ca-7593-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T08:07:55.247 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5 2024-09-18T08:07:55.270 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:07:55.271 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:07:55.271 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5' 2024-09-18T08:07:55.326 INFO:teuthology.orchestra.run.smithi007.stderr:noscrub is set 2024-09-18T08:07:55.373 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6 2024-09-18T08:07:55.396 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:07:55.396 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:07:55.397 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6' 2024-09-18T08:07:55.419 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:07:55 smithi007 ceph-mon[29459]: 6.15 scrub starts 2024-09-18T08:07:55.419 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:07:55 smithi007 ceph-mon[29459]: 6.15 scrub ok 2024-09-18T08:07:55.419 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:07:55 smithi007 ceph-mon[29459]: 6.11 scrub starts 2024-09-18T08:07:55.419 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:07:55 smithi007 ceph-mon[29459]: 6.11 scrub ok 2024-09-18T08:07:55.419 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:07:55 smithi007 ceph-mon[29459]: from='client.? 172.21.15.7:0/1219349348' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-18T08:07:55.419 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:07:55 smithi007 ceph-mon[39249]: 6.15 scrub starts 2024-09-18T08:07:55.419 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:07:55 smithi007 ceph-mon[39249]: 6.15 scrub ok 2024-09-18T08:07:55.419 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:07:55 smithi007 ceph-mon[39249]: 6.11 scrub starts 2024-09-18T08:07:55.419 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:07:55 smithi007 ceph-mon[39249]: 6.11 scrub ok 2024-09-18T08:07:55.420 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:07:55 smithi007 ceph-mon[39249]: from='client.? 172.21.15.7:0/1219349348' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-18T08:07:55.497 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2 2024-09-18T08:07:55.520 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:07:55.520 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:07:55.521 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2' 2024-09-18T08:07:55.622 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2 2024-09-18T08:07:55.643 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:07:55.644 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:07:55.644 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2' 2024-09-18T08:07:55.656 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:07:55 smithi050 ceph-mon[32371]: 6.15 scrub starts 2024-09-18T08:07:55.656 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:07:55 smithi050 ceph-mon[32371]: 6.15 scrub ok 2024-09-18T08:07:55.656 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:07:55 smithi050 ceph-mon[32371]: 6.11 scrub starts 2024-09-18T08:07:55.656 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:07:55 smithi050 ceph-mon[32371]: 6.11 scrub ok 2024-09-18T08:07:55.656 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:07:55 smithi050 ceph-mon[32371]: from='client.? 172.21.15.7:0/1219349348' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-18T08:07:55.700 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 7, 6] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-09-18T08:07:55.700 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-18T08:07:55.701 INFO:tasks.thrashosds.thrasher:Removing osd 4, in_osds are: [0, 1, 2, 3, 4, 5, 7, 6] 2024-09-18T08:07:55.701 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid 779349ca-7593-11ef-bceb-c7b262605968 -- ceph osd out 4 2024-09-18T08:07:55.721 DEBUG:teuthology.orchestra.run.smithi050:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid 779349ca-7593-11ef-bceb-c7b262605968 -- ceph daemon osd.4 dump_historic_ops 2024-09-18T08:07:55.745 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0 2024-09-18T08:07:55.767 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:07:55.767 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:07:55.768 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0' 2024-09-18T08:07:55.868 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3 2024-09-18T08:07:55.892 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:07:55.893 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:07:55.893 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3' 2024-09-18T08:07:55.976 INFO:teuthology.orchestra.run.smithi050.stderr:Inferring config /var/lib/ceph/779349ca-7593-11ef-bceb-c7b262605968/mon.b/config 2024-09-18T08:07:55.994 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4 2024-09-18T08:07:56.017 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:07:56.017 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:07:56.018 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4' 2024-09-18T08:07:56.064 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/779349ca-7593-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T08:07:56.119 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2 2024-09-18T08:07:56.140 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:07:56.142 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:07:56.142 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2' 2024-09-18T08:07:56.243 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7 2024-09-18T08:07:56.266 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:07:56.267 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:07:56.267 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7' 2024-09-18T08:07:56.332 INFO:teuthology.orchestra.run.smithi007.stderr:enabled application 'rados' on pool 'unique_pool_1' 2024-09-18T08:07:56.368 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7 2024-09-18T08:07:56.382 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:07:56 smithi050 ceph-mon[32371]: pgmap v236: 158 pgs: 4 creating+activating, 6 creating+peering, 148 active+clean; 583 KiB data, 952 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-18T08:07:56.382 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:07:56 smithi050 ceph-mon[32371]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-18T08:07:56.382 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:07:56 smithi050 ceph-mon[32371]: from='client.? 172.21.15.7:0/1219349348' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-18T08:07:56.382 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:07:56 smithi050 ceph-mon[32371]: osdmap e168: 8 total, 8 up, 8 in 2024-09-18T08:07:56.382 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:07:56 smithi050 ceph-mon[32371]: from='client.? 172.21.15.7:0/3159325111' entity='client.admin' cmd={"prefix": "osd pool application enable", "pool": "unique_pool_1", "app": "rados", "yes_i_really_mean_it": true} : dispatch 2024-09-18T08:07:56.382 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:07:56 smithi050 ceph-mon[32371]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool application enable", "pool": "unique_pool_1", "app": "rados", "yes_i_really_mean_it": true} : dispatch 2024-09-18T08:07:56.391 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:07:56.392 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:07:56.392 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7' 2024-09-18T08:07:56.490 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2024-09-18T08:07:56.490 INFO:teuthology.orchestra.run.smithi050.stdout: "size": 20, 2024-09-18T08:07:56.491 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 600, 2024-09-18T08:07:56.491 INFO:teuthology.orchestra.run.smithi050.stdout: "ops": [ 2024-09-18T08:07:56.491 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:07:56.491 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.24356.0:1183 5.1f 5:f95f44c2:::notify.0:head [watch reconnect cookie 94700888679552 gen 1] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e101)", 2024-09-18T08:07:56.491 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-09-18T08:04:42.950623+0000", 2024-09-18T08:07:56.491 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 193.53334400099999, 2024-09-18T08:07:56.491 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.35355955700000002, 2024-09-18T08:07:56.491 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-09-18T08:07:56.491 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-09-18T08:07:56.491 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "started", 2024-09-18T08:07:56.491 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2024-09-18T08:07:56.491 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.24356", 2024-09-18T08:07:56.491 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "172.21.15.7:0/219168083", 2024-09-18T08:07:56.491 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 1183 2024-09-18T08:07:56.491 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:07:56.492 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-09-18T08:07:56.492 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:07:56.492 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-09-18T08:07:56.492 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:42.950623+0000", 2024-09-18T08:07:56.492 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-18T08:07:56.492 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:07:56.492 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:07:56.492 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-09-18T08:07:56.492 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:42.950623+0000", 2024-09-18T08:07:56.492 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-18T08:07:56.492 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:07:56.492 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:07:56.492 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-09-18T08:07:56.493 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:42.950624+0000", 2024-09-18T08:07:56.493 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 8.6499999999999998e-07 2024-09-18T08:07:56.493 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:07:56.493 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:07:56.493 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-09-18T08:07:56.493 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:42.950628+0000", 2024-09-18T08:07:56.493 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4.0879999999999997e-06 2024-09-18T08:07:56.493 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:07:56.493 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:07:56.493 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-09-18T08:07:56.493 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:42.950644+0000", 2024-09-18T08:07:56.493 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.6507e-05 2024-09-18T08:07:56.493 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:07:56.493 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:07:56.494 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-09-18T08:07:56.494 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:42.950651+0000", 2024-09-18T08:07:56.494 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 6.5599999999999999e-06 2024-09-18T08:07:56.494 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:07:56.494 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:07:56.494 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-18T08:07:56.494 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:42.950713+0000", 2024-09-18T08:07:56.494 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 6.2360000000000006e-05 2024-09-18T08:07:56.494 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:07:56.494 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:07:56.494 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "waiting for peered", 2024-09-18T08:07:56.494 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:42.950724+0000", 2024-09-18T08:07:56.494 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.0329999999999999e-05 2024-09-18T08:07:56.495 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:07:56.495 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:07:56.495 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-18T08:07:56.495 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:43.301407+0000", 2024-09-18T08:07:56.495 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.35068303499999998 2024-09-18T08:07:56.495 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:07:56.495 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:07:56.495 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-09-18T08:07:56.495 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:43.301697+0000", 2024-09-18T08:07:56.495 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000289938 2024-09-18T08:07:56.495 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:07:56.495 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:07:56.495 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-09-18T08:07:56.495 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:43.304183+0000", 2024-09-18T08:07:56.495 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0024858739999999999 2024-09-18T08:07:56.496 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-18T08:07:56.496 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-09-18T08:07:56.496 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-18T08:07:56.496 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:07:56.496 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:07:56.496 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.24356.0:1184 4.5 4:a60f9269:lc::lua_package_allowlist:head [watch reconnect cookie 94700889442560 gen 1] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e101)", 2024-09-18T08:07:56.496 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-09-18T08:04:42.950665+0000", 2024-09-18T08:07:56.496 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 193.53330162200001, 2024-09-18T08:07:56.496 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.35372086800000002, 2024-09-18T08:07:56.496 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-09-18T08:07:56.496 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-09-18T08:07:56.496 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "started", 2024-09-18T08:07:56.496 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2024-09-18T08:07:56.496 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.24356", 2024-09-18T08:07:56.497 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "172.21.15.7:0/219168083", 2024-09-18T08:07:56.497 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 1184 2024-09-18T08:07:56.497 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:07:56.497 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-09-18T08:07:56.497 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:07:56.497 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-09-18T08:07:56.497 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:42.950665+0000", 2024-09-18T08:07:56.497 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-18T08:07:56.497 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:07:56.497 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:07:56.497 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-09-18T08:07:56.498 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:42.950665+0000", 2024-09-18T08:07:56.498 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-18T08:07:56.498 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:07:56.498 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:07:56.498 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-09-18T08:07:56.498 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:42.950666+0000", 2024-09-18T08:07:56.498 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 6.1099999999999995e-07 2024-09-18T08:07:56.498 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:07:56.498 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:07:56.498 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-09-18T08:07:56.498 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:42.950668+0000", 2024-09-18T08:07:56.498 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 2.514e-06 2024-09-18T08:07:56.498 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:07:56.499 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:07:56.499 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-09-18T08:07:56.499 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:42.950683+0000", 2024-09-18T08:07:56.499 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.4827000000000001e-05 2024-09-18T08:07:56.499 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:07:56.499 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:07:56.499 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-09-18T08:07:56.499 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:42.950689+0000", 2024-09-18T08:07:56.499 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 6.0789999999999997e-06 2024-09-18T08:07:56.499 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:07:56.499 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:07:56.499 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-18T08:07:56.499 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:42.950753+0000", 2024-09-18T08:07:56.499 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 6.3466999999999994e-05 2024-09-18T08:07:56.500 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:07:56.500 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:07:56.500 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "waiting for peered", 2024-09-18T08:07:56.500 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:42.950763+0000", 2024-09-18T08:07:56.500 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.0193e-05 2024-09-18T08:07:56.500 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:07:56.500 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:07:56.500 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-18T08:07:56.500 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:43.301959+0000", 2024-09-18T08:07:56.500 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.35119641299999999 2024-09-18T08:07:56.500 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:07:56.500 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:07:56.500 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-09-18T08:07:56.500 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:43.302091+0000", 2024-09-18T08:07:56.501 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00013126500000000001 2024-09-18T08:07:56.501 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:07:56.501 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:07:56.501 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-09-18T08:07:56.501 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:43.304386+0000", 2024-09-18T08:07:56.501 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0022954989999999999 2024-09-18T08:07:56.501 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-18T08:07:56.501 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-09-18T08:07:56.501 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-18T08:07:56.501 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:07:56.501 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:07:56.501 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.14805.0:84652 7.1 7:8d4001d4:::benchmark_data_smithi007_73155_object84651:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e107)", 2024-09-18T08:07:56.501 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-09-18T08:05:00.706304+0000", 2024-09-18T08:07:56.502 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 175.77766343799999, 2024-09-18T08:07:56.502 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.066859704000000006, 2024-09-18T08:07:56.502 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-09-18T08:07:56.502 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-09-18T08:07:56.502 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T08:07:56.502 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2024-09-18T08:07:56.502 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.14805", 2024-09-18T08:07:56.502 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "172.21.15.7:0/640320436", 2024-09-18T08:07:56.502 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 84652 2024-09-18T08:07:56.502 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:07:56.502 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-09-18T08:07:56.502 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:07:56.502 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-09-18T08:07:56.502 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:05:00.706304+0000", 2024-09-18T08:07:56.502 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-18T08:07:56.503 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:07:56.503 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:07:56.503 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-09-18T08:07:56.503 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:05:00.706304+0000", 2024-09-18T08:07:56.503 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-18T08:07:56.503 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:07:56.503 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:07:56.503 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-09-18T08:07:56.503 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:05:00.706305+0000", 2024-09-18T08:07:56.503 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.116e-06 2024-09-18T08:07:56.504 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:07:56.504 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:07:56.504 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-09-18T08:07:56.504 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:05:00.706355+0000", 2024-09-18T08:07:56.504 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 5.0302000000000001e-05 2024-09-18T08:07:56.504 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:07:56.504 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:07:56.504 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-09-18T08:07:56.504 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:05:00.706364+0000", 2024-09-18T08:07:56.504 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 9.4609999999999997e-06 2024-09-18T08:07:56.504 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:07:56.504 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:07:56.504 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-09-18T08:07:56.504 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:05:00.706369+0000", 2024-09-18T08:07:56.505 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 5.0889999999999999e-06 2024-09-18T08:07:56.505 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:07:56.505 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:07:56.505 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-18T08:07:56.505 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:05:00.706412+0000", 2024-09-18T08:07:56.505 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4.2211e-05 2024-09-18T08:07:56.505 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:07:56.505 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:07:56.505 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-09-18T08:07:56.505 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:05:00.706513+0000", 2024-09-18T08:07:56.505 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000100887 2024-09-18T08:07:56.505 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:07:56.505 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:07:56.506 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "waiting for subops from 3,6", 2024-09-18T08:07:56.506 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:05:00.706613+0000", 2024-09-18T08:07:56.506 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 9.9984999999999999e-05 2024-09-18T08:07:56.506 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:07:56.506 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:07:56.506 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "op_commit", 2024-09-18T08:07:56.506 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:05:00.708644+0000", 2024-09-18T08:07:56.506 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0020316119999999999 2024-09-18T08:07:56.506 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:07:56.506 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:07:56.506 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:07:56.506 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:05:00.708857+0000", 2024-09-18T08:07:56.506 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000212494 2024-09-18T08:07:56.506 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:07:56.506 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:07:56.507 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:07:56.507 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:05:00.773039+0000", 2024-09-18T08:07:56.507 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.064182634000000002 2024-09-18T08:07:56.507 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:07:56.507 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:07:56.507 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "commit_sent", 2024-09-18T08:07:56.507 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:05:00.773110+0000", 2024-09-18T08:07:56.507 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 7.0203999999999994e-05 2024-09-18T08:07:56.507 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:07:56.507 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:07:56.507 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-09-18T08:07:56.507 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:05:00.773163+0000", 2024-09-18T08:07:56.507 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 5.3708999999999997e-05 2024-09-18T08:07:56.507 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-18T08:07:56.508 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-09-18T08:07:56.508 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-18T08:07:56.508 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:07:56.508 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:07:56.508 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.14805.0:84658 7.1 7:893b354a:::benchmark_data_smithi007_73155_object84657:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e107)", 2024-09-18T08:07:56.508 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-09-18T08:05:00.713288+0000", 2024-09-18T08:07:56.508 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 175.77067873799999, 2024-09-18T08:07:56.508 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.060005970999999998, 2024-09-18T08:07:56.508 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-09-18T08:07:56.508 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-09-18T08:07:56.508 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T08:07:56.508 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2024-09-18T08:07:56.508 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.14805", 2024-09-18T08:07:56.508 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "172.21.15.7:0/640320436", 2024-09-18T08:07:56.509 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 84658 2024-09-18T08:07:56.509 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:07:56.509 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-09-18T08:07:56.509 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:07:56.509 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-09-18T08:07:56.509 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:05:00.713288+0000", 2024-09-18T08:07:56.509 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-18T08:07:56.509 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:07:56.509 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:07:56.509 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-09-18T08:07:56.509 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:05:00.713288+0000", 2024-09-18T08:07:56.509 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-18T08:07:56.510 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:07:56.510 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:07:56.510 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-09-18T08:07:56.510 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:05:00.713289+0000", 2024-09-18T08:07:56.510 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.2219999999999999e-06 2024-09-18T08:07:56.510 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:07:56.510 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:07:56.510 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-09-18T08:07:56.510 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:05:00.713317+0000", 2024-09-18T08:07:56.510 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 2.7093e-05 2024-09-18T08:07:56.510 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:07:56.510 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:07:56.510 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-09-18T08:07:56.510 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:05:00.713328+0000", 2024-09-18T08:07:56.511 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.1313e-05 2024-09-18T08:07:56.511 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:07:56.511 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:07:56.511 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-09-18T08:07:56.512 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:05:00.713333+0000", 2024-09-18T08:07:56.512 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 5.5300000000000004e-06 2024-09-18T08:07:56.512 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:07:56.512 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:07:56.512 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-18T08:07:56.512 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:05:00.713374+0000", 2024-09-18T08:07:56.512 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4.0632e-05 2024-09-18T08:07:56.512 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:07:56.512 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:07:56.512 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-09-18T08:07:56.512 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:05:00.713460+0000", 2024-09-18T08:07:56.512 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 8.6366000000000006e-05 2024-09-18T08:07:56.512 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:07:56.513 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:07:56.513 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "waiting for subops from 3,6", 2024-09-18T08:07:56.513 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:05:00.713557+0000", 2024-09-18T08:07:56.513 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 9.6329000000000006e-05 2024-09-18T08:07:56.513 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:07:56.513 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:07:56.513 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "op_commit", 2024-09-18T08:07:56.513 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:05:00.715531+0000", 2024-09-18T08:07:56.513 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.001974066 2024-09-18T08:07:56.513 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:07:56.514 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:07:56.514 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:07:56.514 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:05:00.715621+0000", 2024-09-18T08:07:56.514 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 9.0563999999999996e-05 2024-09-18T08:07:56.514 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:07:56.514 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:07:56.514 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:07:56.514 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:05:00.773245+0000", 2024-09-18T08:07:56.514 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.057623665999999997 2024-09-18T08:07:56.514 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:07:56.514 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:07:56.514 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "commit_sent", 2024-09-18T08:07:56.514 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:05:00.773273+0000", 2024-09-18T08:07:56.514 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 2.8470999999999999e-05 2024-09-18T08:07:56.515 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:07:56.515 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:07:56.515 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-09-18T08:07:56.515 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:05:00.773294+0000", 2024-09-18T08:07:56.515 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 2.0718999999999999e-05 2024-09-18T08:07:56.515 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-18T08:07:56.515 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-09-18T08:07:56.515 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-18T08:07:56.515 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:07:56.515 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:07:56.515 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.14805.0:113361 7.1 7:807e3ad4:::benchmark_data_smithi007_73155_object113360:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e119)", 2024-09-18T08:07:56.516 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-09-18T08:05:30.525448+0000", 2024-09-18T08:07:56.516 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 145.95851904200001, 2024-09-18T08:07:56.516 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.069507528999999998, 2024-09-18T08:07:56.516 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-09-18T08:07:56.516 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-09-18T08:07:56.516 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T08:07:56.516 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2024-09-18T08:07:56.516 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.14805", 2024-09-18T08:07:56.516 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "172.21.15.7:0/640320436", 2024-09-18T08:07:56.516 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 113361 2024-09-18T08:07:56.516 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:07:56.516 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-09-18T08:07:56.516 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:07:56.517 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-09-18T08:07:56.517 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:05:30.525448+0000", 2024-09-18T08:07:56.517 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-18T08:07:56.517 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:07:56.517 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:07:56.517 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-09-18T08:07:56.517 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:05:30.525448+0000", 2024-09-18T08:07:56.517 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-18T08:07:56.517 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:07:56.517 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:07:56.517 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-09-18T08:07:56.517 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:05:30.525450+0000", 2024-09-18T08:07:56.517 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.8589999999999999e-06 2024-09-18T08:07:56.518 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:07:56.518 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:07:56.518 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-09-18T08:07:56.518 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:05:30.525500+0000", 2024-09-18T08:07:56.518 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4.9812999999999999e-05 2024-09-18T08:07:56.518 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:07:56.518 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:07:56.518 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-09-18T08:07:56.518 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:05:30.525513+0000", 2024-09-18T08:07:56.518 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.3337e-05 2024-09-18T08:07:56.518 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:07:56.518 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:07:56.519 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-09-18T08:07:56.519 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:05:30.525521+0000", 2024-09-18T08:07:56.519 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 7.7100000000000007e-06 2024-09-18T08:07:56.519 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:07:56.519 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:07:56.519 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-18T08:07:56.519 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:05:30.525586+0000", 2024-09-18T08:07:56.519 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 6.5506000000000006e-05 2024-09-18T08:07:56.519 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:07:56.519 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:07:56.519 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-09-18T08:07:56.519 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:05:30.525724+0000", 2024-09-18T08:07:56.519 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00013775699999999999 2024-09-18T08:07:56.519 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:07:56.519 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:07:56.520 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "waiting for subops from 3,6", 2024-09-18T08:07:56.520 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:05:30.525875+0000", 2024-09-18T08:07:56.520 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00015107799999999999 2024-09-18T08:07:56.520 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:07:56.520 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:07:56.520 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:07:56.520 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:05:30.528390+0000", 2024-09-18T08:07:56.520 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0025149479999999999 2024-09-18T08:07:56.520 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:07:56.520 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:07:56.520 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "op_commit", 2024-09-18T08:07:56.520 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:05:30.528441+0000", 2024-09-18T08:07:56.520 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 5.0591999999999999e-05 2024-09-18T08:07:56.521 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:07:56.521 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:07:56.521 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:07:56.521 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:05:30.594832+0000", 2024-09-18T08:07:56.521 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.066391475000000005 2024-09-18T08:07:56.521 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:07:56.521 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:07:56.521 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "commit_sent", 2024-09-18T08:07:56.521 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:05:30.594907+0000", 2024-09-18T08:07:56.521 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 7.4995999999999998e-05 2024-09-18T08:07:56.521 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:07:56.521 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:07:56.521 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-09-18T08:07:56.521 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:05:30.594955+0000", 2024-09-18T08:07:56.521 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4.8458000000000003e-05 2024-09-18T08:07:56.522 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-18T08:07:56.522 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-09-18T08:07:56.522 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-18T08:07:56.522 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:07:56.522 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:07:56.522 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.14805.0:113528 7.1 7:8ab97e16:::benchmark_data_smithi007_73155_object113527:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e119)", 2024-09-18T08:07:56.523 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-09-18T08:05:30.734120+0000", 2024-09-18T08:07:56.523 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 145.74984690900001, 2024-09-18T08:07:56.523 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.056096998000000002, 2024-09-18T08:07:56.523 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-09-18T08:07:56.523 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-09-18T08:07:56.523 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T08:07:56.524 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2024-09-18T08:07:56.524 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.14805", 2024-09-18T08:07:56.524 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "172.21.15.7:0/640320436", 2024-09-18T08:07:56.524 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 113528 2024-09-18T08:07:56.524 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:07:56.524 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-09-18T08:07:56.524 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:07:56.524 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-09-18T08:07:56.524 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:05:30.734120+0000", 2024-09-18T08:07:56.525 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-18T08:07:56.525 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:07:56.525 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:07:56.525 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-09-18T08:07:56.525 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:05:30.734120+0000", 2024-09-18T08:07:56.525 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-18T08:07:56.525 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:07:56.525 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:07:56.525 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-09-18T08:07:56.525 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:05:30.734121+0000", 2024-09-18T08:07:56.526 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 7.6700000000000003e-07 2024-09-18T08:07:56.526 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:07:56.526 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:07:56.526 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-09-18T08:07:56.526 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:05:30.734198+0000", 2024-09-18T08:07:56.526 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 7.6809000000000003e-05 2024-09-18T08:07:56.526 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:07:56.526 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:07:56.526 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-09-18T08:07:56.526 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:05:30.734208+0000", 2024-09-18T08:07:56.526 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.0485999999999999e-05 2024-09-18T08:07:56.526 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:07:56.526 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:07:56.526 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-09-18T08:07:56.527 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:05:30.734214+0000", 2024-09-18T08:07:56.527 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 5.626e-06 2024-09-18T08:07:56.527 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:07:56.527 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:07:56.527 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-18T08:07:56.527 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:05:30.735773+0000", 2024-09-18T08:07:56.527 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.001559178 2024-09-18T08:07:56.527 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:07:56.527 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:07:56.527 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-09-18T08:07:56.527 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:05:30.735914+0000", 2024-09-18T08:07:56.527 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00014068699999999999 2024-09-18T08:07:56.528 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:07:56.528 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:07:56.528 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "waiting for subops from 3,6", 2024-09-18T08:07:56.528 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:05:30.736006+0000", 2024-09-18T08:07:56.528 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 9.2829000000000002e-05 2024-09-18T08:07:56.528 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:07:56.528 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:07:56.528 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:07:56.528 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:05:30.738536+0000", 2024-09-18T08:07:56.528 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0025297399999999999 2024-09-18T08:07:56.528 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:07:56.528 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:07:56.529 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "op_commit", 2024-09-18T08:07:56.529 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:05:30.738550+0000", 2024-09-18T08:07:56.529 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.3509999999999999e-05 2024-09-18T08:07:56.529 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:07:56.529 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:07:56.529 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:07:56.529 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:05:30.790151+0000", 2024-09-18T08:07:56.529 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.051601683000000002 2024-09-18T08:07:56.529 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:07:56.529 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:07:56.529 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "commit_sent", 2024-09-18T08:07:56.530 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:05:30.790188+0000", 2024-09-18T08:07:56.530 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 3.6371999999999997e-05 2024-09-18T08:07:56.530 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:07:56.530 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:07:56.530 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-09-18T08:07:56.530 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:05:30.790217+0000", 2024-09-18T08:07:56.530 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 2.9311000000000001e-05 2024-09-18T08:07:56.530 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-18T08:07:56.530 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-09-18T08:07:56.530 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-18T08:07:56.530 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:07:56.530 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:07:56.530 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.14805.0:141888 7.1 7:82d8ae08:::benchmark_data_smithi007_73155_object23597:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e127)", 2024-09-18T08:07:56.531 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-09-18T08:05:59.090389+0000", 2024-09-18T08:07:56.531 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 117.393577478, 2024-09-18T08:07:56.531 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.070629848999999995, 2024-09-18T08:07:56.531 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-09-18T08:07:56.531 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-09-18T08:07:56.531 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T08:07:56.531 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2024-09-18T08:07:56.531 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.14805", 2024-09-18T08:07:56.531 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "172.21.15.7:0/640320436", 2024-09-18T08:07:56.531 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 141888 2024-09-18T08:07:56.531 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:07:56.531 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-09-18T08:07:56.531 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:07:56.532 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-09-18T08:07:56.532 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:05:59.090389+0000", 2024-09-18T08:07:56.532 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-18T08:07:56.532 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:07:56.532 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:07:56.532 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-09-18T08:07:56.532 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:05:59.090389+0000", 2024-09-18T08:07:56.532 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-18T08:07:56.532 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:07:56.532 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:07:56.532 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-09-18T08:07:56.532 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:05:59.090391+0000", 2024-09-18T08:07:56.532 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.1689999999999999e-06 2024-09-18T08:07:56.533 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:07:56.533 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:07:56.533 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-09-18T08:07:56.533 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:05:59.090395+0000", 2024-09-18T08:07:56.533 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4.5650000000000003e-06 2024-09-18T08:07:56.533 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:07:56.533 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:07:56.533 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-09-18T08:07:56.533 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:05:59.090406+0000", 2024-09-18T08:07:56.533 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.0407999999999999e-05 2024-09-18T08:07:56.533 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:07:56.533 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:07:56.533 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-09-18T08:07:56.534 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:05:59.090413+0000", 2024-09-18T08:07:56.534 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 7.7510000000000008e-06 2024-09-18T08:07:56.534 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:07:56.534 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:07:56.534 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-18T08:07:56.534 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:05:59.090468+0000", 2024-09-18T08:07:56.534 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 5.4684e-05 2024-09-18T08:07:56.535 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:07:56.535 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:07:56.535 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-09-18T08:07:56.535 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:05:59.090560+0000", 2024-09-18T08:07:56.535 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 9.1528999999999997e-05 2024-09-18T08:07:56.535 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:07:56.535 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:07:56.535 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "waiting for subops from 3,6", 2024-09-18T08:07:56.535 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:05:59.090626+0000", 2024-09-18T08:07:56.535 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 6.6403999999999996e-05 2024-09-18T08:07:56.536 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:07:56.536 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:07:56.536 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "op_commit", 2024-09-18T08:07:56.536 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:05:59.092907+0000", 2024-09-18T08:07:56.536 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.002280528 2024-09-18T08:07:56.536 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:07:56.536 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:07:56.536 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:07:56.536 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:05:59.093035+0000", 2024-09-18T08:07:56.536 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00012871900000000001 2024-09-18T08:07:56.536 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:07:56.536 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:07:56.537 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:07:56.537 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:05:59.160952+0000", 2024-09-18T08:07:56.537 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4 2024-09-18T08:07:56.539 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.067916502000000004 2024-09-18T08:07:56.539 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:07:56.539 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:07:56.539 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "commit_sent", 2024-09-18T08:07:56.540 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:05:59.160987+0000", 2024-09-18T08:07:56.540 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 3.5267e-05 2024-09-18T08:07:56.540 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:07:56.540 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:07:56.540 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-09-18T08:07:56.540 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:05:59.161019+0000", 2024-09-18T08:07:56.540 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 3.2323000000000002e-05 2024-09-18T08:07:56.540 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-18T08:07:56.540 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-09-18T08:07:56.540 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-18T08:07:56.540 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:07:56.540 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:07:56.541 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.14805.0:142145 7.1 7:85ef7aca:::benchmark_data_smithi007_73155_object23854:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e127)", 2024-09-18T08:07:56.541 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-09-18T08:05:59.413428+0000", 2024-09-18T08:07:56.541 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 117.07053909, 2024-09-18T08:07:56.541 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.061856467999999998, 2024-09-18T08:07:56.541 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-09-18T08:07:56.541 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-09-18T08:07:56.541 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T08:07:56.541 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2024-09-18T08:07:56.541 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.14805", 2024-09-18T08:07:56.541 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "172.21.15.7:0/640320436", 2024-09-18T08:07:56.541 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 142145 2024-09-18T08:07:56.541 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:07:56.542 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-09-18T08:07:56.542 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:07:56.542 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-09-18T08:07:56.542 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:05:59.413428+0000", 2024-09-18T08:07:56.542 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-18T08:07:56.542 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:07:56.542 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:07:56.542 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-09-18T08:07:56.542 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:05:59.413428+0000", 2024-09-18T08:07:56.542 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-18T08:07:56.542 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:07:56.542 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:07:56.542 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-09-18T08:07:56.543 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:05:59.413429+0000", 2024-09-18T08:07:56.543 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.4759999999999999e-06 2024-09-18T08:07:56.543 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:07:56.543 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:07:56.543 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-09-18T08:07:56.543 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:05:59.413435+0000", 2024-09-18T08:07:56.543 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 6.0349999999999998e-06 2024-09-18T08:07:56.543 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:07:56.543 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:07:56.543 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-09-18T08:07:56.543 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:05:59.413449+0000", 2024-09-18T08:07:56.544 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.397e-05 2024-09-18T08:07:56.544 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:07:56.544 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:07:56.544 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-09-18T08:07:56.544 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:05:59.413458+0000", 2024-09-18T08:07:56.544 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 8.2779999999999993e-06 2024-09-18T08:07:56.544 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:07:56.545 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:07:56.545 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-18T08:07:56.545 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:05:59.413520+0000", 2024-09-18T08:07:56.545 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 6.2385000000000007e-05 2024-09-18T08:07:56.545 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:07:56.545 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:07:56.545 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-09-18T08:07:56.545 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:05:59.413650+0000", 2024-09-18T08:07:56.545 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00012997599999999999 2024-09-18T08:07:56.545 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:07:56.546 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:07:56.546 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "waiting for subops from 3,6", 2024-09-18T08:07:56.546 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:05:59.413777+0000", 2024-09-18T08:07:56.546 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000127133 2024-09-18T08:07:56.546 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:07:56.546 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:07:56.546 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "op_commit", 2024-09-18T08:07:56.546 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:05:59.416223+0000", 2024-09-18T08:07:56.546 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.002445773 2024-09-18T08:07:56.546 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:07:56.547 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:07:56.547 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:07:56.547 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:05:59.416586+0000", 2024-09-18T08:07:56.547 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00036319999999999999 2024-09-18T08:07:56.547 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:07:56.547 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:07:56.547 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:07:56.547 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:05:59.475199+0000", 2024-09-18T08:07:56.547 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.058613382999999998 2024-09-18T08:07:56.547 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:07:56.547 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:07:56.547 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "commit_sent", 2024-09-18T08:07:56.547 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:05:59.475239+0000", 2024-09-18T08:07:56.547 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 3.9280999999999999e-05 2024-09-18T08:07:56.548 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:07:56.548 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:07:56.548 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-09-18T08:07:56.548 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:05:59.475284+0000", 2024-09-18T08:07:56.548 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4.5578000000000003e-05 2024-09-18T08:07:56.548 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-18T08:07:56.548 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-09-18T08:07:56.548 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-18T08:07:56.548 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:07:56.548 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:07:56.548 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.14805.0:147845 7.1 7:8a0a44bd:::benchmark_data_smithi007_73155_object29554:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e127)", 2024-09-18T08:07:56.548 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-09-18T08:06:04.708313+0000", 2024-09-18T08:07:56.548 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 111.775654192, 2024-09-18T08:07:56.548 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.073618763000000004, 2024-09-18T08:07:56.549 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-09-18T08:07:56.549 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-09-18T08:07:56.549 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T08:07:56.549 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2024-09-18T08:07:56.549 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.14805", 2024-09-18T08:07:56.549 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "172.21.15.7:0/640320436", 2024-09-18T08:07:56.549 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 147845 2024-09-18T08:07:56.549 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:07:56.549 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-09-18T08:07:56.549 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:07:56.549 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-09-18T08:07:56.549 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:06:04.708313+0000", 2024-09-18T08:07:56.549 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-18T08:07:56.550 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:07:56.550 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:07:56.550 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-09-18T08:07:56.550 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:06:04.708313+0000", 2024-09-18T08:07:56.550 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-18T08:07:56.550 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:07:56.550 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:07:56.550 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-09-18T08:07:56.550 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:06:04.708313+0000", 2024-09-18T08:07:56.551 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 6.9999999999999997e-07 2024-09-18T08:07:56.551 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:07:56.551 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:07:56.551 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-09-18T08:07:56.551 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:06:04.708317+0000", 2024-09-18T08:07:56.551 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 3.36e-06 2024-09-18T08:07:56.551 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:07:56.551 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:07:56.551 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-09-18T08:07:56.551 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:06:04.708324+0000", 2024-09-18T08:07:56.551 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 7.5739999999999998e-06 2024-09-18T08:07:56.551 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:07:56.551 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:07:56.551 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-09-18T08:07:56.552 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:06:04.708329+0000", 2024-09-18T08:07:56.552 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4.4209999999999997e-06 2024-09-18T08:07:56.552 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:07:56.552 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:07:56.552 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-18T08:07:56.552 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:06:04.709653+0000", 2024-09-18T08:07:56.553 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0013239090000000001 2024-09-18T08:07:56.553 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:07:56.553 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:07:56.553 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-09-18T08:07:56.553 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:06:04.710395+0000", 2024-09-18T08:07:56.553 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00074240000000000005 2024-09-18T08:07:56.554 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:07:56.554 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:07:56.554 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "waiting for subops from 3,6", 2024-09-18T08:07:56.554 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:06:04.710508+0000", 2024-09-18T08:07:56.554 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000113248 2024-09-18T08:07:56.554 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:07:56.555 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:07:56.555 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "op_commit", 2024-09-18T08:07:56.555 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:06:04.712970+0000", 2024-09-18T08:07:56.555 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0024617160000000001 2024-09-18T08:07:56.555 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:07:56.555 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:07:56.556 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:07:56.556 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:06:04.714318+0000", 2024-09-18T08:07:56.556 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0013482629999999999 2024-09-18T08:07:56.556 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:07:56.556 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:07:56.556 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:07:56.557 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:06:04.781826+0000", 2024-09-18T08:07:56.557 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.067507519000000002 2024-09-18T08:07:56.557 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:07:56.557 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:07:56.557 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "commit_sent", 2024-09-18T08:07:56.557 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:06:04.781893+0000", 2024-09-18T08:07:56.557 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 6.7374999999999998e-05 2024-09-18T08:07:56.558 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:07:56.558 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:07:56.558 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-09-18T08:07:56.558 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:06:04.781932+0000", 2024-09-18T08:07:56.558 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 3.8278000000000002e-05 2024-09-18T08:07:56.558 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-18T08:07:56.559 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-09-18T08:07:56.559 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-18T08:07:56.559 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:07:56.559 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:07:56.559 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.14805.0:148169 7.1 7:8d71082f:::benchmark_data_smithi007_73155_object29878:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e127)", 2024-09-18T08:07:56.560 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-09-18T08:06:05.041000+0000", 2024-09-18T08:07:56.560 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 111.442967086, 2024-09-18T08:07:56.560 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.062830021, 2024-09-18T08:07:56.560 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-09-18T08:07:56.560 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-09-18T08:07:56.560 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T08:07:56.561 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2024-09-18T08:07:56.561 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.14805", 2024-09-18T08:07:56.561 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "172.21.15.7:0/640320436", 2024-09-18T08:07:56.561 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 148169 2024-09-18T08:07:56.561 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:07:56.561 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-09-18T08:07:56.562 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:07:56.562 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-09-18T08:07:56.562 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:06:05.041000+0000", 2024-09-18T08:07:56.562 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-18T08:07:56.562 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:07:56.562 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:07:56.563 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-09-18T08:07:56.563 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:06:05.041000+0000", 2024-09-18T08:07:56.563 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-18T08:07:56.563 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:07:56.564 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:07:56.564 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-09-18T08:07:56.564 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:06:05.041001+0000", 2024-09-18T08:07:56.564 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.575e-06 2024-09-18T08:07:56.564 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:07:56.564 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:07:56.565 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-09-18T08:07:56.565 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:06:05.041007+0000", 2024-09-18T08:07:56.565 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 5.0839999999999999e-06 2024-09-18T08:07:56.565 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:07:56.565 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:07:56.565 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-09-18T08:07:56.566 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:06:05.041019+0000", 2024-09-18T08:07:56.566 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.2668e-05 2024-09-18T08:07:56.566 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:07:56.566 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:07:56.566 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-09-18T08:07:56.567 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:06:05.041026+0000", 2024-09-18T08:07:56.567 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 7.0910000000000004e-06 2024-09-18T08:07:56.567 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:07:56.567 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:07:56.567 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-18T08:07:56.567 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:06:05.041070+0000", 2024-09-18T08:07:56.568 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4.4030999999999999e-05 2024-09-18T08:07:56.568 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:07:56.568 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:07:56.568 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-09-18T08:07:56.568 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:06:05.041178+0000", 2024-09-18T08:07:56.569 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00010746700000000001 2024-09-18T08:07:56.569 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:07:56.569 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:07:56.569 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "waiting for subops from 3,6", 2024-09-18T08:07:56.569 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:06:05.041302+0000", 2024-09-18T08:07:56.570 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000123788 2024-09-18T08:07:56.570 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:07:56.570 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:07:56.570 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "op_commit", 2024-09-18T08:07:56.570 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:06:05.044159+0000", 2024-09-18T08:07:56.570 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.002857264 2024-09-18T08:07:56.571 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:07:56.571 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:07:56.571 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:07:56.571 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:06:05.044286+0000", 2024-09-18T08:07:56.571 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00012713500000000001 2024-09-18T08:07:56.571 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:07:56.572 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:07:56.572 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:07:56.572 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:06:05.103732+0000", 2024-09-18T08:07:56.572 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.059446163000000003 2024-09-18T08:07:56.572 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:07:56.573 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:07:56.573 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "commit_sent", 2024-09-18T08:07:56.573 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:06:05.103772+0000", 2024-09-18T08:07:56.573 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 3.9966000000000003e-05 2024-09-18T08:07:56.573 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:07:56.573 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:07:56.574 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-09-18T08:07:56.574 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:06:05.103830+0000", 2024-09-18T08:07:56.574 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 5.7788999999999998e-05 2024-09-18T08:07:56.574 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-18T08:07:56.574 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-09-18T08:07:56.575 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-18T08:07:56.575 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:07:56.575 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:07:56.575 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.14805.0:148357 7.1 7:8ec47016:::benchmark_data_smithi007_73155_object30066:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e127)", 2024-09-18T08:07:56.575 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-09-18T08:06:05.261852+0000", 2024-09-18T08:07:56.575 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 111.22211466500001, 2024-09-18T08:07:56.576 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.058562336999999999, 2024-09-18T08:07:56.576 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-09-18T08:07:56.576 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-09-18T08:07:56.576 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T08:07:56.576 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2024-09-18T08:07:56.577 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.14805", 2024-09-18T08:07:56.577 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "172.21.15.7:0/640320436", 2024-09-18T08:07:56.577 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 148357 2024-09-18T08:07:56.577 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:07:56.577 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-09-18T08:07:56.577 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:07:56.578 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-09-18T08:07:56.578 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:06:05.261852+0000", 2024-09-18T08:07:56.578 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-18T08:07:56.578 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:07:56.578 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:07:56.579 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-09-18T08:07:56.579 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:06:05.261852+0000", 2024-09-18T08:07:56.579 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-18T08:07:56.579 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:07:56.579 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:07:56.579 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-09-18T08:07:56.580 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:06:05.261854+0000", 2024-09-18T08:07:56.580 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.4789999999999999e-06 2024-09-18T08:07:56.580 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:07:56.580 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:07:56.580 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-09-18T08:07:56.580 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:06:05.261860+0000", 2024-09-18T08:07:56.581 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 5.993e-06 2024-09-18T08:07:56.581 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:07:56.581 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:07:56.581 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-09-18T08:07:56.581 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:06:05.261874+0000", 2024-09-18T08:07:56.581 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.4637000000000001e-05 2024-09-18T08:07:56.582 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:07:56.582 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:07:56.582 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-09-18T08:07:56.582 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:06:05.261882+0000", 2024-09-18T08:07:56.582 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 7.9330000000000001e-06 2024-09-18T08:07:56.582 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:07:56.583 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:07:56.583 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-18T08:07:56.583 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:06:05.261954+0000", 2024-09-18T08:07:56.583 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 7.2095999999999995e-05 2024-09-18T08:07:56.583 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:07:56.583 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:07:56.584 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-09-18T08:07:56.584 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:06:05.262078+0000", 2024-09-18T08:07:56.584 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00012376600000000001 2024-09-18T08:07:56.584 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:07:56.584 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:07:56.584 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "waiting for subops from 3,6", 2024-09-18T08:07:56.584 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:06:05.262201+0000", 2024-09-18T08:07:56.584 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00012270299999999999 2024-09-18T08:07:56.585 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:07:56.585 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:07:56.585 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:07:56.585 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:06:05.264434+0000", 2024-09-18T08:07:56.585 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0022327789999999998 2024-09-18T08:07:56.585 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:07:56.585 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:07:56.585 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "op_commit", 2024-09-18T08:07:56.585 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:06:05.266518+0000", 2024-09-18T08:07:56.585 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0020840049999999999 2024-09-18T08:07:56.585 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:07:56.585 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:07:56.585 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:07:56.585 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:06:05.320298+0000", 2024-09-18T08:07:56.586 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.053779929999999997 2024-09-18T08:07:56.586 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:07:56.586 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:07:56.586 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "commit_sent", 2024-09-18T08:07:56.586 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:06:05.320362+0000", 2024-09-18T08:07:56.586 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 6.4121000000000005e-05 2024-09-18T08:07:56.586 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:07:56.586 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:07:56.586 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-09-18T08:07:56.586 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:06:05.320415+0000", 2024-09-18T08:07:56.586 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 5.2895000000000001e-05 2024-09-18T08:07:56.586 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-18T08:07:56.586 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-09-18T08:07:56.586 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-18T08:07:56.587 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:07:56.587 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:07:56.587 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.14805.0:151467 7.1 7:87478555:::benchmark_data_smithi007_73155_object33176:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e127)", 2024-09-18T08:07:56.587 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-09-18T08:06:08.235608+0000", 2024-09-18T08:07:56.587 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 108.24835931299999, 2024-09-18T08:07:56.587 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.058523480000000003, 2024-09-18T08:07:56.587 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-09-18T08:07:56.587 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-09-18T08:07:56.587 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T08:07:56.587 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2024-09-18T08:07:56.587 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.14805", 2024-09-18T08:07:56.587 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "172.21.15.7:0/640320436", 2024-09-18T08:07:56.587 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 151467 2024-09-18T08:07:56.587 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:07:56.587 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-09-18T08:07:56.588 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:07:56.588 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-09-18T08:07:56.588 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:06:08.235608+0000", 2024-09-18T08:07:56.588 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-18T08:07:56.588 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:07:56.588 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:07:56.588 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-09-18T08:07:56.588 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:06:08.235608+0000", 2024-09-18T08:07:56.588 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-18T08:07:56.588 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:07:56.588 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:07:56.588 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-09-18T08:07:56.588 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:06:08.235609+0000", 2024-09-18T08:07:56.589 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.514e-06 2024-09-18T08:07:56.589 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:07:56.589 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:07:56.589 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-09-18T08:07:56.589 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:06:08.235615+0000", 2024-09-18T08:07:56.589 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 5.6080000000000003e-06 2024-09-18T08:07:56.589 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:07:56.589 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:07:56.589 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-09-18T08:07:56.589 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:06:08.235629+0000", 2024-09-18T08:07:56.589 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.4343e-05 2024-09-18T08:07:56.589 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:07:56.589 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:07:56.589 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-09-18T08:07:56.590 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:06:08.235637+0000", 2024-09-18T08:07:56.590 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 8.0739999999999993e-06 2024-09-18T08:07:56.590 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:07:56.590 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:07:56.590 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-18T08:07:56.590 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:06:08.235692+0000", 2024-09-18T08:07:56.590 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 5.4851000000000001e-05 2024-09-18T08:07:56.590 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:07:56.590 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:07:56.590 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-09-18T08:07:56.590 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:06:08.235809+0000", 2024-09-18T08:07:56.590 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00011712799999999999 2024-09-18T08:07:56.590 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:07:56.590 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:07:56.591 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "waiting for subops from 3,6", 2024-09-18T08:07:56.591 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:06:08.236204+0000", 2024-09-18T08:07:56.591 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000395103 2024-09-18T08:07:56.591 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:07:56.591 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:07:56.591 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:07:56.591 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:06:08.239329+0000", 2024-09-18T08:07:56.591 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.003124319 2024-09-18T08:07:56.591 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:07:56.591 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:07:56.591 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "op_commit", 2024-09-18T08:07:56.591 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:06:08.240293+0000", 2024-09-18T08:07:56.591 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00096459200000000001 2024-09-18T08:07:56.592 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:07:56.592 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:07:56.592 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:07:56.592 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:06:08.294031+0000", 2024-09-18T08:07:56.592 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.053738068 2024-09-18T08:07:56.592 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:07:56.592 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:07:56.592 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "commit_sent", 2024-09-18T08:07:56.592 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:06:08.294084+0000", 2024-09-18T08:07:56.592 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 5.2902000000000003e-05 2024-09-18T08:07:56.592 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:07:56.592 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:07:56.592 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-09-18T08:07:56.593 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:06:08.294131+0000", 2024-09-18T08:07:56.593 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4.6978000000000003e-05 2024-09-18T08:07:56.593 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-18T08:07:56.593 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-09-18T08:07:56.593 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-18T08:07:56.593 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:07:56.593 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:07:56.593 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.14805.0:151470 7.1 7:8a658e0b:::benchmark_data_smithi007_73155_object33179:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e127)", 2024-09-18T08:07:56.593 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-09-18T08:06:08.241975+0000", 2024-09-18T08:07:56.593 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 108.241992322, 2024-09-18T08:07:56.593 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.062786429000000005, 2024-09-18T08:07:56.593 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-09-18T08:07:56.593 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-09-18T08:07:56.594 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T08:07:56.594 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2024-09-18T08:07:56.594 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.14805", 2024-09-18T08:07:56.594 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "172.21.15.7:0/640320436", 2024-09-18T08:07:56.594 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 151470 2024-09-18T08:07:56.594 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:07:56.594 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-09-18T08:07:56.594 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:07:56.594 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-09-18T08:07:56.594 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:06:08.241975+0000", 2024-09-18T08:07:56.594 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-18T08:07:56.594 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:07:56.594 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:07:56.595 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-09-18T08:07:56.595 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:06:08.241975+0000", 2024-09-18T08:07:56.595 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-18T08:07:56.595 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:07:56.595 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:07:56.595 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-09-18T08:07:56.595 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:06:08.241977+0000", 2024-09-18T08:07:56.595 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 2.3970000000000001e-06 2024-09-18T08:07:56.595 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:07:56.595 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:07:56.595 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-09-18T08:07:56.595 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:06:08.241986+0000", 2024-09-18T08:07:56.595 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 8.7809999999999994e-06 2024-09-18T08:07:56.595 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:07:56.596 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:07:56.596 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-09-18T08:07:56.596 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:06:08.242005+0000", 2024-09-18T08:07:56.596 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.9017000000000001e-05 2024-09-18T08:07:56.596 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:07:56.596 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:07:56.596 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-09-18T08:07:56.596 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:06:08.242013+0000", 2024-09-18T08:07:56.596 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 8.5010000000000004e-06 2024-09-18T08:07:56.596 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:07:56.596 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:07:56.596 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-18T08:07:56.596 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:06:08.242078+0000", 2024-09-18T08:07:56.596 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 6.4772999999999998e-05 2024-09-18T08:07:56.597 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:07:56.597 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:07:56.597 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-09-18T08:07:56.597 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:06:08.242207+0000", 2024-09-18T08:07:56.597 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00012895199999999999 2024-09-18T08:07:56.597 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:07:56.597 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:07:56.597 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "waiting for subops from 3,6", 2024-09-18T08:07:56.597 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:06:08.242281+0000", 2024-09-18T08:07:56.597 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 7.3670000000000004e-05 2024-09-18T08:07:56.597 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:07:56.597 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:07:56.597 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "op_commit", 2024-09-18T08:07:56.597 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:06:08.244516+0000", 2024-09-18T08:07:56.598 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0022348390000000002 2024-09-18T08:07:56.598 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:07:56.598 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:07:56.598 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:07:56.598 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:06:08.244704+0000", 2024-09-18T08:07:56.598 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000188884 2024-09-18T08:07:56.598 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:07:56.598 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:07:56.598 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:07:56.598 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:06:08.304618+0000", 2024-09-18T08:07:56.598 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.059913392000000003 2024-09-18T08:07:56.598 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:07:56.598 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:07:56.599 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "commit_sent", 2024-09-18T08:07:56.599 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:06:08.304704+0000", 2024-09-18T08:07:56.599 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 8.6261000000000006e-05 2024-09-18T08:07:56.599 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:07:56.599 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:07:56.599 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-09-18T08:07:56.599 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:06:08.304761+0000", 2024-09-18T08:07:56.599 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 5.6962000000000001e-05 2024-09-18T08:07:56.599 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-18T08:07:56.599 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-09-18T08:07:56.599 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-18T08:07:56.599 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:07:56.599 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:07:56.599 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.14805.0:151646 7.1 7:8ab88cf8:::benchmark_data_smithi007_73155_object33355:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e127)", 2024-09-18T08:07:56.600 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-09-18T08:06:08.437973+0000", 2024-09-18T08:07:56.600 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 108.0459944, 2024-09-18T08:07:56.600 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.063089381, 2024-09-18T08:07:56.600 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-09-18T08:07:56.600 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-09-18T08:07:56.600 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T08:07:56.600 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2024-09-18T08:07:56.600 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.14805", 2024-09-18T08:07:56.600 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "172.21.15.7:0/640320436", 2024-09-18T08:07:56.600 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 151646 2024-09-18T08:07:56.600 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:07:56.600 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-09-18T08:07:56.600 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:07:56.601 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-09-18T08:07:56.601 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:06:08.437973+0000", 2024-09-18T08:07:56.601 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-18T08:07:56.601 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:07:56.601 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:07:56.601 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-09-18T08:07:56.601 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:06:08.437973+0000", 2024-09-18T08:07:56.601 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-18T08:07:56.601 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:07:56.601 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:07:56.601 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-09-18T08:07:56.601 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:06:08.437974+0000", 2024-09-18T08:07:56.601 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.1939999999999999e-06 2024-09-18T08:07:56.602 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:07:56.602 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:07:56.602 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-09-18T08:07:56.602 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:06:08.437978+0000", 2024-09-18T08:07:56.602 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4.5290000000000002e-06 2024-09-18T08:07:56.602 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:07:56.602 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:07:56.602 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-09-18T08:07:56.602 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:06:08.437990+0000", 2024-09-18T08:07:56.602 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.1888e-05 2024-09-18T08:07:56.602 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:07:56.602 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:07:56.602 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-09-18T08:07:56.602 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:06:08.437997+0000", 2024-09-18T08:07:56.603 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 6.9530000000000002e-06 2024-09-18T08:07:56.603 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:07:56.603 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:07:56.603 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-18T08:07:56.603 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:06:08.438043+0000", 2024-09-18T08:07:56.603 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4.5904e-05 2024-09-18T08:07:56.603 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:07:56.603 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:07:56.603 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-09-18T08:07:56.603 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:06:08.438120+0000", 2024-09-18T08:07:56.603 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 7.6612000000000004e-05 2024-09-18T08:07:56.603 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:07:56.603 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:07:56.604 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "waiting for subops from 3,6", 2024-09-18T08:07:56.604 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:06:08.438188+0000", 2024-09-18T08:07:56.604 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 6.8455999999999996e-05 2024-09-18T08:07:56.604 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:07:56.604 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:07:56.604 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "op_commit", 2024-09-18T08:07:56.604 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:06:08.445164+0000", 2024-09-18T08:07:56.604 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.006976192 2024-09-18T08:07:56.604 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:07:56.604 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:07:56.604 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:07:56.604 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:06:08.445222+0000", 2024-09-18T08:07:56.604 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 5.7447000000000001e-05 2024-09-18T08:07:56.605 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:07:56.605 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:07:56.605 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:07:56.605 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:06:08.500981+0000", 2024-09-18T08:07:56.605 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.055759727000000002 2024-09-18T08:07:56.605 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:07:56.605 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:07:56.605 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "commit_sent", 2024-09-18T08:07:56.605 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:06:08.501021+0000", 2024-09-18T08:07:56.605 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 3.9851000000000002e-05 2024-09-18T08:07:56.605 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:07:56.605 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:07:56.605 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-09-18T08:07:56.605 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:06:08.501062+0000", 2024-09-18T08:07:56.606 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4.0627999999999998e-05 2024-09-18T08:07:56.606 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-18T08:07:56.606 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-09-18T08:07:56.606 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-18T08:07:56.606 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:07:56.606 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:07:56.606 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.14805.0:151648 7.1 7:834e324e:::benchmark_data_smithi007_73155_object33357:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e127)", 2024-09-18T08:07:56.606 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-09-18T08:06:08.439698+0000", 2024-09-18T08:07:56.606 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 108.04426854899999, 2024-09-18T08:07:56.606 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.061456726000000003, 2024-09-18T08:07:56.606 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-09-18T08:07:56.606 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-09-18T08:07:56.606 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T08:07:56.606 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2024-09-18T08:07:56.607 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.14805", 2024-09-18T08:07:56.607 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "172.21.15.7:0/640320436", 2024-09-18T08:07:56.607 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 151648 2024-09-18T08:07:56.607 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:07:56.607 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-09-18T08:07:56.607 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:07:56.607 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-09-18T08:07:56.607 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:06:08.439698+0000", 2024-09-18T08:07:56.607 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-18T08:07:56.607 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:07:56.607 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:07:56.607 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-09-18T08:07:56.607 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:06:08.439698+0000", 2024-09-18T08:07:56.608 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-18T08:07:56.608 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:07:56.608 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:07:56.608 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-09-18T08:07:56.608 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:06:08.439699+0000", 2024-09-18T08:07:56.608 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.057e-06 2024-09-18T08:07:56.608 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:07:56.608 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:07:56.608 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-09-18T08:07:56.608 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:06:08.439707+0000", 2024-09-18T08:07:56.608 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 7.1450000000000002e-06 2024-09-18T08:07:56.608 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:07:56.608 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:07:56.608 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-09-18T08:07:56.609 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:06:08.439719+0000", 2024-09-18T08:07:56.609 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.1946e-05 2024-09-18T08:07:56.609 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:07:56.609 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:07:56.609 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-09-18T08:07:56.609 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:06:08.439725+0000", 2024-09-18T08:07:56.609 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 6.5540000000000003e-06 2024-09-18T08:07:56.609 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:07:56.609 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:07:56.609 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-18T08:07:56.609 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:06:08.439955+0000", 2024-09-18T08:07:56.609 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000229662 2024-09-18T08:07:56.609 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:07:56.610 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:07:56.610 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-09-18T08:07:56.610 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:06:08.440063+0000", 2024-09-18T08:07:56.610 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000108257 2024-09-18T08:07:56.610 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:07:56.610 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:07:56.610 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "waiting for subops from 3,6", 2024-09-18T08:07:56.610 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:06:08.440176+0000", 2024-09-18T08:07:56.610 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000113028 2024-09-18T08:07:56.610 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:07:56.610 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:07:56.610 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "op_commit", 2024-09-18T08:07:56.610 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:06:08.445175+0000", 2024-09-18T08:07:56.611 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0049987900000000004 2024-09-18T08:07:56.611 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:07:56.611 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:07:56.611 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:07:56.611 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:06:08.445415+0000", 2024-09-18T08:07:56.611 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00024047200000000001 2024-09-18T08:07:56.611 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:07:56.611 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:07:56.611 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:07:56.611 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:06:08.501099+0000", 2024-09-18T08:07:56.611 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.055683208999999997 2024-09-18T08:07:56.611 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:07:56.611 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:07:56.611 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "commit_sent", 2024-09-18T08:07:56.612 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:06:08.501127+0000", 2024-09-18T08:07:56.612 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 2.8503000000000001e-05 2024-09-18T08:07:56.612 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:07:56.612 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:07:56.612 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-09-18T08:07:56.612 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:06:08.501155+0000", 2024-09-18T08:07:56.612 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 2.8102999999999998e-05 2024-09-18T08:07:56.612 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-18T08:07:56.612 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-09-18T08:07:56.613 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-18T08:07:56.613 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:07:56.613 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:07:56.613 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.14805.0:151649 7.1 7:87d4917c:::benchmark_data_smithi007_73155_object33358:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e127)", 2024-09-18T08:07:56.613 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-09-18T08:06:08.439739+0000", 2024-09-18T08:07:56.613 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 108.044228362, 2024-09-18T08:07:56.613 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.062102242000000002, 2024-09-18T08:07:56.613 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-09-18T08:07:56.613 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-09-18T08:07:56.613 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T08:07:56.613 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2024-09-18T08:07:56.613 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.14805", 2024-09-18T08:07:56.613 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "172.21.15.7:0/640320436", 2024-09-18T08:07:56.613 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 151649 2024-09-18T08:07:56.614 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:07:56.614 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-09-18T08:07:56.614 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:07:56.614 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-09-18T08:07:56.614 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:06:08.439739+0000", 2024-09-18T08:07:56.614 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-18T08:07:56.614 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:07:56.614 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:07:56.614 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-09-18T08:07:56.614 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:06:08.439739+0000", 2024-09-18T08:07:56.614 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-18T08:07:56.614 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:07:56.614 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:07:56.614 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-09-18T08:07:56.614 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:06:08.439739+0000", 2024-09-18T08:07:56.615 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4.63e-07 2024-09-18T08:07:56.615 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:07:56.615 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:07:56.615 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-09-18T08:07:56.615 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:06:08.439741+0000", 2024-09-18T08:07:56.615 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 2.3360000000000002e-06 2024-09-18T08:07:56.615 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:07:56.615 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:07:56.615 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-09-18T08:07:56.615 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:06:08.439748+0000", 2024-09-18T08:07:56.615 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 6.6900000000000003e-06 2024-09-18T08:07:56.615 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:07:56.615 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:07:56.615 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-09-18T08:07:56.616 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:06:08.439753+0000", 2024-09-18T08:07:56.616 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4.5249999999999999e-06 2024-09-18T08:07:56.616 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:07:56.616 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:07:56.616 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-18T08:07:56.616 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:06:08.442205+0000", 2024-09-18T08:07:56.616 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.002451997 2024-09-18T08:07:56.616 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:07:56.616 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:07:56.616 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-09-18T08:07:56.616 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:06:08.442326+0000", 2024-09-18T08:07:56.616 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000121725 2024-09-18T08:07:56.616 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:07:56.616 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:07:56.617 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "waiting for subops from 3,6", 2024-09-18T08:07:56.617 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:06:08.442444+0000", 2024-09-18T08:07:56.617 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000117799 2024-09-18T08:07:56.617 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:07:56.617 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:07:56.617 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "op_commit", 2024-09-18T08:07:56.617 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:06:08.445294+0000", 2024-09-18T08:07:56.617 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.002849659 2024-09-18T08:07:56.617 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:07:56.617 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:07:56.617 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:07:56.617 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:06:08.445503+0000", 2024-09-18T08:07:56.617 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000209621 2024-09-18T08:07:56.618 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:07:56.618 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:07:56.618 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:07:56.618 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:06:08.501752+0000", 2024-09-18T08:07:56.618 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.056248116000000001 2024-09-18T08:07:56.618 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:07:56.618 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:07:56.618 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "commit_sent", 2024-09-18T08:07:56.618 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:06:08.501803+0000", 2024-09-18T08:07:56.618 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 5.1403000000000002e-05 2024-09-18T08:07:56.618 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:07:56.618 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:07:56.618 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-09-18T08:07:56.619 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:06:08.501841+0000", 2024-09-18T08:07:56.619 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 3.7907999999999997e-05 2024-09-18T08:07:56.619 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-18T08:07:56.619 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-09-18T08:07:56.619 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-18T08:07:56.619 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:07:56.619 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:07:56.619 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.14805.0:161175 7.1 7:8e9a4b6b:::benchmark_data_smithi007_73155_object42884:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e134)", 2024-09-18T08:07:56.619 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-09-18T08:06:21.016529+0000", 2024-09-18T08:07:56.619 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 95.467437797000002, 2024-09-18T08:07:56.619 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.060141834999999998, 2024-09-18T08:07:56.619 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-09-18T08:07:56.619 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-09-18T08:07:56.619 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T08:07:56.620 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2024-09-18T08:07:56.620 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.14805", 2024-09-18T08:07:56.620 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "172.21.15.7:0/640320436", 2024-09-18T08:07:56.620 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 161175 2024-09-18T08:07:56.620 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:07:56.620 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-09-18T08:07:56.620 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:07:56.620 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-09-18T08:07:56.620 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:06:21.016529+0000", 2024-09-18T08:07:56.620 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-18T08:07:56.620 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:07:56.620 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:07:56.620 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-09-18T08:07:56.621 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:06:21.016529+0000", 2024-09-18T08:07:56.621 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-18T08:07:56.621 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:07:56.621 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:07:56.621 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-09-18T08:07:56.621 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:06:21.016530+0000", 2024-09-18T08:07:56.621 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 8.9199999999999999e-07 2024-09-18T08:07:56.621 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:07:56.621 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:07:56.621 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-09-18T08:07:56.621 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:06:21.016534+0000", 2024-09-18T08:07:56.621 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4.2699999999999998e-06 2024-09-18T08:07:56.621 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:07:56.621 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:07:56.622 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-09-18T08:07:56.622 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:06:21.016543+0000", 2024-09-18T08:07:56.622 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 8.5629999999999992e-06 2024-09-18T08:07:56.622 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:07:56.622 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:07:56.622 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-09-18T08:07:56.622 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:06:21.016548+0000", 2024-09-18T08:07:56.622 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4.6530000000000002e-06 2024-09-18T08:07:56.622 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:07:56.622 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:07:56.622 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-18T08:07:56.622 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:06:21.016595+0000", 2024-09-18T08:07:56.622 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4.7082000000000001e-05 2024-09-18T08:07:56.622 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:07:56.623 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:07:56.623 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-09-18T08:07:56.623 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:06:21.016726+0000", 2024-09-18T08:07:56.623 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00013156999999999999 2024-09-18T08:07:56.623 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:07:56.623 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:07:56.623 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "waiting for subops from 1,3,6", 2024-09-18T08:07:56.623 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:06:21.016873+0000", 2024-09-18T08:07:56.623 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00014691899999999999 2024-09-18T08:07:56.623 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:07:56.623 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:07:56.623 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:07:56.623 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:06:21.019088+0000", 2024-09-18T08:07:56.623 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0022153099999999998 2024-09-18T08:07:56.624 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:07:56.624 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:07:56.624 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "op_commit", 2024-09-18T08:07:56.624 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:06:21.019167+0000", 2024-09-18T08:07:56.624 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 7.8498999999999995e-05 2024-09-18T08:07:56.624 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:07:56.624 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:07:56.624 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:07:56.624 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:06:21.068944+0000", 2024-09-18T08:07:56.624 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.049777228999999999 2024-09-18T08:07:56.624 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:07:56.624 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:07:56.624 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:07:56.624 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:06:21.076480+0000", 2024-09-18T08:07:56.625 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.007535623 2024-09-18T08:07:56.625 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:07:56.625 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:07:56.625 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "commit_sent", 2024-09-18T08:07:56.625 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:06:21.076556+0000", 2024-09-18T08:07:56.625 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 7.5822999999999996e-05 2024-09-18T08:07:56.625 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:07:56.625 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:07:56.625 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-09-18T08:07:56.625 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:06:21.076671+0000", 2024-09-18T08:07:56.625 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000115402 2024-09-18T08:07:56.625 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-18T08:07:56.625 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-09-18T08:07:56.625 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-18T08:07:56.626 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:07:56.626 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:07:56.626 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.14805.0:177922 7.1 7:84c9dd1e:::benchmark_data_smithi007_73155_object59631:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e139)", 2024-09-18T08:07:56.626 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-09-18T08:06:38.827370+0000", 2024-09-18T08:07:56.626 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 77.656597332999993, 2024-09-18T08:07:56.626 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.055124491999999997, 2024-09-18T08:07:56.626 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-09-18T08:07:56.626 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-09-18T08:07:56.626 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T08:07:56.626 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2024-09-18T08:07:56.626 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.14805", 2024-09-18T08:07:56.626 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "172.21.15.7:0/640320436", 2024-09-18T08:07:56.626 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 177922 2024-09-18T08:07:56.626 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:07:56.627 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-09-18T08:07:56.627 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:07:56.627 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-09-18T08:07:56.627 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:06:38.827370+0000", 2024-09-18T08:07:56.627 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-18T08:07:56.627 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:07:56.627 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:07:56.627 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-09-18T08:07:56.627 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:06:38.827370+0000", 2024-09-18T08:07:56.627 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-18T08:07:56.627 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:07:56.627 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:07:56.627 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-09-18T08:07:56.627 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:06:38.827371+0000", 2024-09-18T08:07:56.628 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.4249999999999999e-06 2024-09-18T08:07:56.628 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:07:56.628 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:07:56.628 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-09-18T08:07:56.628 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:06:38.827377+0000", 2024-09-18T08:07:56.628 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 6.1330000000000004e-06 2024-09-18T08:07:56.628 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:07:56.628 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:07:56.628 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-09-18T08:07:56.628 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:06:38.827392+0000", 2024-09-18T08:07:56.628 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.4439e-05 2024-09-18T08:07:56.628 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:07:56.628 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:07:56.628 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-09-18T08:07:56.629 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:06:38.827398+0000", 2024-09-18T08:07:56.629 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 6.6599999999999998e-06 2024-09-18T08:07:56.629 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:07:56.629 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:07:56.629 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-18T08:07:56.629 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:06:38.827459+0000", 2024-09-18T08:07:56.629 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 6.1097000000000001e-05 2024-09-18T08:07:56.629 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:07:56.629 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:07:56.629 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-09-18T08:07:56.629 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:06:38.827576+0000", 2024-09-18T08:07:56.629 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000117009 2024-09-18T08:07:56.629 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:07:56.630 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:07:56.630 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "waiting for subops from 3,6", 2024-09-18T08:07:56.630 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:06:38.827660+0000", 2024-09-18T08:07:56.630 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 8.4037999999999997e-05 2024-09-18T08:07:56.630 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:07:56.630 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:07:56.630 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "op_commit", 2024-09-18T08:07:56.630 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:06:38.830122+0000", 2024-09-18T08:07:56.630 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0024618690000000002 2024-09-18T08:07:56.630 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:07:56.630 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:07:56.630 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:07:56.630 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:06:38.830436+0000", 2024-09-18T08:07:56.631 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00031349399999999999 2024-09-18T08:07:56.631 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:07:56.631 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:07:56.631 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:07:56.631 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:06:38.882370+0000", 2024-09-18T08:07:56.631 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.051934655000000003 2024-09-18T08:07:56.631 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:07:56.631 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:07:56.631 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "commit_sent", 2024-09-18T08:07:56.631 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:06:38.882439+0000", 2024-09-18T08:07:56.631 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 6.8448000000000006e-05 2024-09-18T08:07:56.631 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:07:56.631 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:07:56.631 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-09-18T08:07:56.632 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:06:38.882494+0000", 2024-09-18T08:07:56.632 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 5.5225000000000001e-05 2024-09-18T08:07:56.632 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-18T08:07:56.632 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-09-18T08:07:56.632 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-18T08:07:56.632 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:07:56.632 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:07:56.632 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.14805.0:223167 7.1 7:8e8c717b:::benchmark_data_smithi007_73155_object104876:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e154)", 2024-09-18T08:07:56.632 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-09-18T08:07:25.501787+0000", 2024-09-18T08:07:56.632 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 30.982180355000001, 2024-09-18T08:07:56.632 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.063531656000000006, 2024-09-18T08:07:56.632 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-09-18T08:07:56.632 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-09-18T08:07:56.633 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T08:07:56.633 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2024-09-18T08:07:56.633 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.14805", 2024-09-18T08:07:56.633 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "172.21.15.7:0/640320436", 2024-09-18T08:07:56.633 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 223167 2024-09-18T08:07:56.633 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:07:56.633 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-09-18T08:07:56.633 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:07:56.633 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-09-18T08:07:56.633 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:07:25.501787+0000", 2024-09-18T08:07:56.633 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-18T08:07:56.633 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:07:56.633 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:07:56.634 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-09-18T08:07:56.634 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:07:25.501787+0000", 2024-09-18T08:07:56.634 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-18T08:07:56.634 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:07:56.634 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:07:56.634 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-09-18T08:07:56.634 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:07:25.501788+0000", 2024-09-18T08:07:56.634 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.4190000000000001e-06 2024-09-18T08:07:56.634 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:07:56.634 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:07:56.634 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-09-18T08:07:56.634 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:07:25.501794+0000", 2024-09-18T08:07:56.634 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 5.7379999999999999e-06 2024-09-18T08:07:56.635 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:07:56.635 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:07:56.635 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-09-18T08:07:56.635 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:07:25.501806+0000", 2024-09-18T08:07:56.635 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.2417e-05 2024-09-18T08:07:56.635 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:07:56.635 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:07:56.635 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-09-18T08:07:56.635 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:07:25.501814+0000", 2024-09-18T08:07:56.635 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 7.6410000000000002e-06 2024-09-18T08:07:56.635 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:07:56.635 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:07:56.635 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-18T08:07:56.635 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:07:25.501870+0000", 2024-09-18T08:07:56.636 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 5.6237999999999999e-05 2024-09-18T08:07:56.636 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:07:56.636 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:07:56.636 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-09-18T08:07:56.636 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:07:25.502028+0000", 2024-09-18T08:07:56.636 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000158185 2024-09-18T08:07:56.636 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:07:56.636 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:07:56.636 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "waiting for subops from 3,6", 2024-09-18T08:07:56.636 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:07:25.502179+0000", 2024-09-18T08:07:56.636 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000150543 2024-09-18T08:07:56.636 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:07:56.636 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:07:56.636 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "op_commit", 2024-09-18T08:07:56.637 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:07:25.506959+0000", 2024-09-18T08:07:56.637 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0047797810000000003 2024-09-18T08:07:56.637 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:07:56.637 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:07:56.637 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:07:56.637 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:07:25.506998+0000", 2024-09-18T08:07:56.637 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 3.9010000000000001e-05 2024-09-18T08:07:56.637 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:07:56.637 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:07:56.637 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:07:56.637 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:07:25.565196+0000", 2024-09-18T08:07:56.637 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.058198666000000003 2024-09-18T08:07:56.637 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:07:56.638 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:07:56.638 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "commit_sent", 2024-09-18T08:07:56.638 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:07:25.565259+0000", 2024-09-18T08:07:56.638 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 6.2801999999999999e-05 2024-09-18T08:07:56.638 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:07:56.638 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:07:56.638 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-09-18T08:07:56.638 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:07:25.565318+0000", 2024-09-18T08:07:56.638 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 5.9215999999999997e-05 2024-09-18T08:07:56.638 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-18T08:07:56.638 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-09-18T08:07:56.639 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-18T08:07:56.639 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:07:56.639 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:07:56.639 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.14805.0:223171 7.1 7:8a32bd1d:::benchmark_data_smithi007_73155_object104880:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e154)", 2024-09-18T08:07:56.639 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-09-18T08:07:25.503541+0000", 2024-09-18T08:07:56.639 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 30.980426040000001, 2024-09-18T08:07:56.639 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.06216, 2024-09-18T08:07:56.639 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-09-18T08:07:56.639 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-09-18T08:07:56.639 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T08:07:56.639 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2024-09-18T08:07:56.639 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.14805", 2024-09-18T08:07:56.639 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "172.21.15.7:0/640320436", 2024-09-18T08:07:56.639 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 223171 2024-09-18T08:07:56.640 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:07:56.640 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-09-18T08:07:56.640 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:07:56.640 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-09-18T08:07:56.640 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:07:25.503541+0000", 2024-09-18T08:07:56.640 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-18T08:07:56.640 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:07:56.640 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:07:56.640 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-09-18T08:07:56.640 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:07:25.503541+0000", 2024-09-18T08:07:56.640 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-18T08:07:56.640 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:07:56.640 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:07:56.640 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-09-18T08:07:56.641 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:07:25.503542+0000", 2024-09-18T08:07:56.641 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.186e-06 2024-09-18T08:07:56.641 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:07:56.641 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:07:56.641 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-09-18T08:07:56.641 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:07:25.503547+0000", 2024-09-18T08:07:56.641 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 5.2229999999999998e-06 2024-09-18T08:07:56.641 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:07:56.641 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:07:56.641 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-09-18T08:07:56.641 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:07:25.503559+0000", 2024-09-18T08:07:56.641 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.1901e-05 2024-09-18T08:07:56.641 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:07:56.642 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:07:56.642 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-09-18T08:07:56.642 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:07:25.503566+0000", 2024-09-18T08:07:56.642 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 6.737e-06 2024-09-18T08:07:56.642 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:07:56.642 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:07:56.642 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-18T08:07:56.642 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:07:25.504462+0000", 2024-09-18T08:07:56.642 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00089583100000000004 2024-09-18T08:07:56.642 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:07:56.642 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:07:56.642 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-09-18T08:07:56.642 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:07:25.504589+0000", 2024-09-18T08:07:56.642 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000126716 2024-09-18T08:07:56.643 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:07:56.643 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:07:56.643 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "waiting for subops from 3,6", 2024-09-18T08:07:56.643 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:07:25.504723+0000", 2024-09-18T08:07:56.643 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00013468700000000001 2024-09-18T08:07:56.643 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:07:56.643 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:07:56.643 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "op_commit", 2024-09-18T08:07:56.643 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:07:25.507049+0000", 2024-09-18T08:07:56.643 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0023254310000000002 2024-09-18T08:07:56.643 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:07:56.643 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:07:56.643 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:07:56.643 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:07:25.507220+0000", 2024-09-18T08:07:56.644 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00017161499999999999 2024-09-18T08:07:56.644 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:07:56.644 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:07:56.644 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:07:56.644 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:07:25.565610+0000", 2024-09-18T08:07:56.644 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.058389420999999997 2024-09-18T08:07:56.644 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:07:56.644 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:07:56.644 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "commit_sent", 2024-09-18T08:07:56.644 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:07:25.565664+0000", 2024-09-18T08:07:56.644 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 5.3933e-05 2024-09-18T08:07:56.644 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:07:56.644 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:07:56.645 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-09-18T08:07:56.645 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:07:25.565701+0000", 2024-09-18T08:07:56.645 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 3.7319e-05 2024-09-18T08:07:56.645 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-18T08:07:56.645 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-09-18T08:07:56.645 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-18T08:07:56.645 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-18T08:07:56.645 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-09-18T08:07:56.645 INFO:teuthology.orchestra.run.smithi050.stdout:} 2024-09-18T08:07:56.667 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:07:56.667 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:07:56.668 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4' 2024-09-18T08:07:56.697 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:07:56 smithi007 ceph-mon[29459]: pgmap v236: 158 pgs: 4 creating+activating, 6 creating+peering, 148 active+clean; 583 KiB data, 952 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-18T08:07:56.697 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:07:56 smithi007 ceph-mon[29459]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-18T08:07:56.697 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:07:56 smithi007 ceph-mon[29459]: from='client.? 172.21.15.7:0/1219349348' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-18T08:07:56.697 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:07:56 smithi007 ceph-mon[29459]: osdmap e168: 8 total, 8 up, 8 in 2024-09-18T08:07:56.697 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:07:56 smithi007 ceph-mon[29459]: from='client.? 172.21.15.7:0/3159325111' entity='client.admin' cmd={"prefix": "osd pool application enable", "pool": "unique_pool_1", "app": "rados", "yes_i_really_mean_it": true} : dispatch 2024-09-18T08:07:56.697 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:07:56 smithi007 ceph-mon[29459]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool application enable", "pool": "unique_pool_1", "app": "rados", "yes_i_really_mean_it": true} : dispatch 2024-09-18T08:07:56.698 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:07:56 smithi007 ceph-mon[39249]: pgmap v236: 158 pgs: 4 creating+activating, 6 creating+peering, 148 active+clean; 583 KiB data, 952 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-18T08:07:56.698 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:07:56 smithi007 ceph-mon[39249]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-18T08:07:56.698 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:07:56 smithi007 ceph-mon[39249]: from='client.? 172.21.15.7:0/1219349348' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-18T08:07:56.698 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:07:56 smithi007 ceph-mon[39249]: osdmap e168: 8 total, 8 up, 8 in 2024-09-18T08:07:56.698 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:07:56 smithi007 ceph-mon[39249]: from='client.? 172.21.15.7:0/3159325111' entity='client.admin' cmd={"prefix": "osd pool application enable", "pool": "unique_pool_1", "app": "rados", "yes_i_really_mean_it": true} : dispatch 2024-09-18T08:07:56.698 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:07:56 smithi007 ceph-mon[39249]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool application enable", "pool": "unique_pool_1", "app": "rados", "yes_i_really_mean_it": true} : dispatch 2024-09-18T08:07:56.768 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1 2024-09-18T08:07:56.791 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:07:56.792 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:07:56.792 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1' 2024-09-18T08:07:56.893 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3 2024-09-18T08:07:56.915 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:07:56.916 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:07:56.916 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3' 2024-09-18T08:07:56.980 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Sep 18 08:07:56 smithi007 ceph-779349ca-7593-11ef-bceb-c7b262605968-mgr-y[29688]: ::ffff:172.21.15.50 - - [18/Sep/2024:08:07:56] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-09-18T08:07:57.017 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1 2024-09-18T08:07:57.040 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:07:57.040 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:07:57.041 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1' 2024-09-18T08:07:57.142 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6 2024-09-18T08:07:57.165 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:07:57.166 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:07:57.166 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6' 2024-09-18T08:07:57.181 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid 779349ca-7593-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-18T08:07:57.267 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2 2024-09-18T08:07:57.289 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:07:57.290 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:07:57.290 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2' 2024-09-18T08:07:57.391 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5 2024-09-18T08:07:57.413 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:07:57.414 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:07:57.414 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5' 2024-09-18T08:07:57.515 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1 2024-09-18T08:07:57.537 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:07:57.538 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:07:57.538 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1' 2024-09-18T08:07:57.550 DEBUG:teuthology.orchestra.run.smithi050:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid 779349ca-7593-11ef-bceb-c7b262605968 -- ceph daemon osd.5 dump_ops_in_flight 2024-09-18T08:07:57.638 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3 2024-09-18T08:07:57.639 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/779349ca-7593-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T08:07:57.656 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:07:57 smithi050 ceph-mon[32371]: 5.1a scrub starts 2024-09-18T08:07:57.656 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:07:57 smithi050 ceph-mon[32371]: 5.1a scrub ok 2024-09-18T08:07:57.656 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:07:57 smithi050 ceph-mon[32371]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "unique_pool_1", "app": "rados", "yes_i_really_mean_it": true}]': finished 2024-09-18T08:07:57.656 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:07:57 smithi050 ceph-mon[32371]: osdmap e169: 8 total, 8 up, 8 in 2024-09-18T08:07:57.656 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:07:57 smithi050 ceph-mon[32371]: 1.a scrub starts 2024-09-18T08:07:57.656 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:07:57 smithi050 ceph-mon[32371]: 1.5 scrub starts 2024-09-18T08:07:57.663 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:07:57.663 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:07:57.664 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3' 2024-09-18T08:07:57.730 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:07:57 smithi007 ceph-mon[29459]: 5.1a scrub starts 2024-09-18T08:07:57.730 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:07:57 smithi007 ceph-mon[29459]: 5.1a scrub ok 2024-09-18T08:07:57.731 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:07:57 smithi007 ceph-mon[29459]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "unique_pool_1", "app": "rados", "yes_i_really_mean_it": true}]': finished 2024-09-18T08:07:57.731 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:07:57 smithi007 ceph-mon[29459]: osdmap e169: 8 total, 8 up, 8 in 2024-09-18T08:07:57.731 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:07:57 smithi007 ceph-mon[29459]: 1.a scrub starts 2024-09-18T08:07:57.731 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:07:57 smithi007 ceph-mon[29459]: 1.5 scrub starts 2024-09-18T08:07:57.731 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:07:57 smithi007 ceph-mon[39249]: 5.1a scrub starts 2024-09-18T08:07:57.731 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:07:57 smithi007 ceph-mon[39249]: 5.1a scrub ok 2024-09-18T08:07:57.731 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:07:57 smithi007 ceph-mon[39249]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "unique_pool_1", "app": "rados", "yes_i_really_mean_it": true}]': finished 2024-09-18T08:07:57.731 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:07:57 smithi007 ceph-mon[39249]: osdmap e169: 8 total, 8 up, 8 in 2024-09-18T08:07:57.731 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:07:57 smithi007 ceph-mon[39249]: 1.a scrub starts 2024-09-18T08:07:57.731 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:07:57 smithi007 ceph-mon[39249]: 1.5 scrub starts 2024-09-18T08:07:57.765 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0 2024-09-18T08:07:57.787 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:07:57.788 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:07:57.788 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0' 2024-09-18T08:07:57.807 INFO:teuthology.orchestra.run.smithi050.stderr:Inferring config /var/lib/ceph/779349ca-7593-11ef-bceb-c7b262605968/mon.b/config 2024-09-18T08:07:57.889 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3 2024-09-18T08:07:57.911 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:07:57.912 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:07:57.912 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3' 2024-09-18T08:07:58.013 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2 2024-09-18T08:07:58.037 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:07:58.037 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:07:58.038 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2' 2024-09-18T08:07:58.075 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:07:57 smithi007 ceph-779349ca-7593-11ef-bceb-c7b262605968-mon-a[29455]: 2024-09-18T08:07:57.782+0000 7fcd2b595640 -1 mon.a@0(leader).osd e170 definitely_dead 0 2024-09-18T08:07:58.095 INFO:teuthology.orchestra.run.smithi007.stdout:osd.0: {} 2024-09-18T08:07:58.095 INFO:teuthology.orchestra.run.smithi007.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-18T08:07:58.095 DEBUG:teuthology.orchestra.run.smithi007:> /bin/sh -c 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados --no-log-to-stderr --name client.0 -b 65536 --object-size 65536 -p unique_pool_1 bench 90 write' 2024-09-18T08:07:58.097 INFO:tasks.radosbench:joining radosbench (timing out after 3000s) 2024-09-18T08:07:58.097 INFO:teuthology.orchestra.run:waiting for 3000 2024-09-18T08:07:58.112 INFO:teuthology.orchestra.run.smithi007.stdout:osd.1: {} 2024-09-18T08:07:58.113 INFO:teuthology.orchestra.run.smithi007.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-18T08:07:58.129 INFO:teuthology.orchestra.run.smithi007.stdout:osd.2: {} 2024-09-18T08:07:58.129 INFO:teuthology.orchestra.run.smithi007.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-18T08:07:58.139 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3 2024-09-18T08:07:58.147 INFO:teuthology.orchestra.run.smithi007.stdout:osd.3: {} 2024-09-18T08:07:58.147 INFO:teuthology.orchestra.run.smithi007.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-18T08:07:58.152 INFO:tasks.radosbench.radosbench.0.smithi007.stdout:hints = 1 2024-09-18T08:07:58.152 INFO:tasks.radosbench.radosbench.0.smithi007.stdout:Maintaining 16 concurrent writes of 65536 bytes to objects of size 65536 for up to 90 seconds or 0 objects 2024-09-18T08:07:58.152 INFO:tasks.radosbench.radosbench.0.smithi007.stdout:Object prefix: benchmark_data_smithi007_101583 2024-09-18T08:07:58.152 INFO:tasks.radosbench.radosbench.0.smithi007.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-09-18T08:07:58.152 INFO:tasks.radosbench.radosbench.0.smithi007.stdout: 0 0 0 0 0 0 - 0 2024-09-18T08:07:58.172 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:07:58.173 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:07:58.174 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3' 2024-09-18T08:07:58.179 INFO:teuthology.orchestra.run.smithi007.stdout:osd.4: {} 2024-09-18T08:07:58.179 INFO:teuthology.orchestra.run.smithi007.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-18T08:07:58.217 INFO:teuthology.orchestra.run.smithi007.stdout:osd.5: {} 2024-09-18T08:07:58.217 INFO:teuthology.orchestra.run.smithi007.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-18T08:07:58.251 INFO:teuthology.orchestra.run.smithi007.stdout:osd.6: {} 2024-09-18T08:07:58.252 INFO:teuthology.orchestra.run.smithi007.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-18T08:07:58.275 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3 2024-09-18T08:07:58.289 INFO:teuthology.orchestra.run.smithi007.stdout:osd.7: {} 2024-09-18T08:07:58.289 INFO:teuthology.orchestra.run.smithi007.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-18T08:07:58.315 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:07:58.316 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:07:58.316 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3' 2024-09-18T08:07:58.339 INFO:teuthology.orchestra.run.smithi050.stdout:ERROR: (22) Invalid argument 2024-09-18T08:07:58.339 INFO:teuthology.orchestra.run.smithi050.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-18T08:07:58.349 INFO:teuthology.orchestra.run.smithi007.stderr:marked out osd.4. 2024-09-18T08:07:58.417 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2 2024-09-18T08:07:58.439 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:07:58.440 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:07:58.440 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2' 2024-09-18T08:07:58.540 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid 779349ca-7593-11ef-bceb-c7b262605968 -- ceph osd set nodeep-scrub 2024-09-18T08:07:58.541 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5 2024-09-18T08:07:58.564 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:07:58.565 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:07:58.565 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5' 2024-09-18T08:07:58.656 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:07:58 smithi050 ceph-mon[32371]: 1.0 scrub starts 2024-09-18T08:07:58.656 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:07:58 smithi050 ceph-mon[32371]: 1.2 scrub starts 2024-09-18T08:07:58.656 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:07:58 smithi050 ceph-mon[32371]: purged_snaps scrub starts 2024-09-18T08:07:58.656 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:07:58 smithi050 ceph-mon[32371]: purged_snaps scrub ok 2024-09-18T08:07:58.656 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:07:58 smithi050 ceph-mon[32371]: 1.3 scrub starts 2024-09-18T08:07:58.656 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:07:58 smithi050 ceph-mon[32371]: pgmap v239: 158 pgs: 4 creating+activating, 6 creating+peering, 148 active+clean; 583 KiB data, 952 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-18T08:07:58.656 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:07:58 smithi050 ceph-mon[32371]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-09-18T08:07:58.657 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:07:58 smithi050 ceph-mon[32371]: osdmap e170: 8 total, 8 up, 8 in 2024-09-18T08:07:58.657 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:07:58 smithi050 ceph-mon[32371]: 1.7 scrub starts 2024-09-18T08:07:58.657 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:07:58 smithi050 ceph-mon[32371]: 1.6 scrub starts 2024-09-18T08:07:58.657 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:07:58 smithi050 ceph-mon[32371]: from='client.? 172.21.15.7:0/608921094' entity='client.admin' cmd={"prefix": "osd out", "ids": ["4"]} : dispatch 2024-09-18T08:07:58.657 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:07:58 smithi050 ceph-mon[32371]: Client client.admin marked osd.4 out, while it was still marked up 2024-09-18T08:07:58.657 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:07:58 smithi050 ceph-mon[32371]: from='client.? 172.21.15.7:0/3709609918' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-18T08:07:58.666 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0 2024-09-18T08:07:58.689 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:07:58.689 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:07:58.690 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0' 2024-09-18T08:07:58.731 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:07:58 smithi007 ceph-mon[39249]: 1.0 scrub starts 2024-09-18T08:07:58.731 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:07:58 smithi007 ceph-mon[39249]: 1.2 scrub starts 2024-09-18T08:07:58.731 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:07:58 smithi007 ceph-mon[39249]: purged_snaps scrub starts 2024-09-18T08:07:58.731 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:07:58 smithi007 ceph-mon[39249]: purged_snaps scrub ok 2024-09-18T08:07:58.731 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:07:58 smithi007 ceph-mon[39249]: 1.3 scrub starts 2024-09-18T08:07:58.731 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:07:58 smithi007 ceph-mon[39249]: pgmap v239: 158 pgs: 4 creating+activating, 6 creating+peering, 148 active+clean; 583 KiB data, 952 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-18T08:07:58.731 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:07:58 smithi007 ceph-mon[39249]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-09-18T08:07:58.731 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:07:58 smithi007 ceph-mon[39249]: osdmap e170: 8 total, 8 up, 8 in 2024-09-18T08:07:58.731 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:07:58 smithi007 ceph-mon[39249]: 1.7 scrub starts 2024-09-18T08:07:58.731 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:07:58 smithi007 ceph-mon[39249]: 1.6 scrub starts 2024-09-18T08:07:58.731 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:07:58 smithi007 ceph-mon[39249]: from='client.? 172.21.15.7:0/608921094' entity='client.admin' cmd={"prefix": "osd out", "ids": ["4"]} : dispatch 2024-09-18T08:07:58.732 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:07:58 smithi007 ceph-mon[39249]: Client client.admin marked osd.4 out, while it was still marked up 2024-09-18T08:07:58.732 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:07:58 smithi007 ceph-mon[39249]: from='client.? 172.21.15.7:0/3709609918' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-18T08:07:58.732 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:07:58 smithi007 ceph-mon[29459]: 1.0 scrub starts 2024-09-18T08:07:58.732 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:07:58 smithi007 ceph-mon[29459]: 1.2 scrub starts 2024-09-18T08:07:58.732 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:07:58 smithi007 ceph-mon[29459]: purged_snaps scrub starts 2024-09-18T08:07:58.732 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:07:58 smithi007 ceph-mon[29459]: purged_snaps scrub ok 2024-09-18T08:07:58.732 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:07:58 smithi007 ceph-mon[29459]: 1.3 scrub starts 2024-09-18T08:07:58.732 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:07:58 smithi007 ceph-mon[29459]: pgmap v239: 158 pgs: 4 creating+activating, 6 creating+peering, 148 active+clean; 583 KiB data, 952 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-18T08:07:58.732 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:07:58 smithi007 ceph-mon[29459]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-09-18T08:07:58.732 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:07:58 smithi007 ceph-mon[29459]: osdmap e170: 8 total, 8 up, 8 in 2024-09-18T08:07:58.733 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:07:58 smithi007 ceph-mon[29459]: 1.7 scrub starts 2024-09-18T08:07:58.733 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:07:58 smithi007 ceph-mon[29459]: 1.6 scrub starts 2024-09-18T08:07:58.733 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:07:58 smithi007 ceph-mon[29459]: from='client.? 172.21.15.7:0/608921094' entity='client.admin' cmd={"prefix": "osd out", "ids": ["4"]} : dispatch 2024-09-18T08:07:58.733 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:07:58 smithi007 ceph-mon[29459]: Client client.admin marked osd.4 out, while it was still marked up 2024-09-18T08:07:58.733 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:07:58 smithi007 ceph-mon[29459]: from='client.? 172.21.15.7:0/3709609918' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-18T08:07:58.791 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7 2024-09-18T08:07:58.814 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:07:58.815 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:07:58.815 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7' 2024-09-18T08:07:58.916 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2 2024-09-18T08:07:58.936 DEBUG:teuthology.orchestra.run.smithi050:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid 779349ca-7593-11ef-bceb-c7b262605968 -- ceph daemon osd.5 dump_blocked_ops 2024-09-18T08:07:58.938 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:07:58.939 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:07:58.940 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2' 2024-09-18T08:07:59.041 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4 2024-09-18T08:07:59.062 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:07:59.063 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:07:59.064 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4' 2024-09-18T08:07:59.152 INFO:tasks.radosbench.radosbench.0.smithi007.stdout: 1 16 677 661 41.3081 41.3125 0.0018854 0.00523591 2024-09-18T08:07:59.164 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7 2024-09-18T08:07:59.187 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:07:59.187 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:07:59.188 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7' 2024-09-18T08:07:59.192 INFO:teuthology.orchestra.run.smithi050.stderr:Inferring config /var/lib/ceph/779349ca-7593-11ef-bceb-c7b262605968/mon.b/config 2024-09-18T08:07:59.286 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/779349ca-7593-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T08:07:59.288 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4 2024-09-18T08:07:59.311 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:07:59.312 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:07:59.312 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4' 2024-09-18T08:07:59.313 INFO:journalctl@ceph.iscsi.iscsi.a.smithi050.stdout:Sep 18 08:07:58 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-iscsi-iscsi-a[57782]: debug there is no tcmu-runner data available 2024-09-18T08:07:59.412 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6 2024-09-18T08:07:59.449 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:07:59.449 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:07:59.450 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6' 2024-09-18T08:07:59.480 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:07:59 smithi007 ceph-mon[29459]: 1.9 scrub starts 2024-09-18T08:07:59.480 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:07:59 smithi007 ceph-mon[29459]: from='client.? 172.21.15.7:0/608921094' entity='client.admin' cmd='[{"prefix": "osd out", "ids": ["4"]}]': finished 2024-09-18T08:07:59.480 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:07:59 smithi007 ceph-mon[29459]: osdmap e171: 8 total, 8 up, 7 in 2024-09-18T08:07:59.480 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:07:59 smithi007 ceph-mon[29459]: 1.4 scrub starts 2024-09-18T08:07:59.481 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:07:59 smithi007 ceph-mon[29459]: 1.1 scrub starts 2024-09-18T08:07:59.481 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:07:59 smithi007 ceph-mon[39249]: 1.9 scrub starts 2024-09-18T08:07:59.481 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:07:59 smithi007 ceph-mon[39249]: from='client.? 172.21.15.7:0/608921094' entity='client.admin' cmd='[{"prefix": "osd out", "ids": ["4"]}]': finished 2024-09-18T08:07:59.481 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:07:59 smithi007 ceph-mon[39249]: osdmap e171: 8 total, 8 up, 7 in 2024-09-18T08:07:59.481 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:07:59 smithi007 ceph-mon[39249]: 1.4 scrub starts 2024-09-18T08:07:59.481 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:07:59 smithi007 ceph-mon[39249]: 1.1 scrub starts 2024-09-18T08:07:59.551 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5 2024-09-18T08:07:59.583 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:07:59.584 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:07:59.584 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5' 2024-09-18T08:07:59.603 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:07:59 smithi050 ceph-mon[32371]: 1.9 scrub starts 2024-09-18T08:07:59.603 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:07:59 smithi050 ceph-mon[32371]: from='client.? 172.21.15.7:0/608921094' entity='client.admin' cmd='[{"prefix": "osd out", "ids": ["4"]}]': finished 2024-09-18T08:07:59.603 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:07:59 smithi050 ceph-mon[32371]: osdmap e171: 8 total, 8 up, 7 in 2024-09-18T08:07:59.603 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:07:59 smithi050 ceph-mon[32371]: 1.4 scrub starts 2024-09-18T08:07:59.603 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:07:59 smithi050 ceph-mon[32371]: 1.1 scrub starts 2024-09-18T08:07:59.685 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1 2024-09-18T08:07:59.729 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:07:59.730 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:07:59.730 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1' 2024-09-18T08:07:59.759 INFO:teuthology.orchestra.run.smithi050.stdout:ERROR: (22) Invalid argument 2024-09-18T08:07:59.759 INFO:teuthology.orchestra.run.smithi050.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-18T08:07:59.830 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2 2024-09-18T08:07:59.871 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:07:59.872 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:07:59.872 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2' 2024-09-18T08:07:59.973 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2 2024-09-18T08:08:00.013 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:08:00.014 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:08:00.015 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2' 2024-09-18T08:08:00.115 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2 2024-09-18T08:08:00.152 INFO:tasks.radosbench.radosbench.0.smithi007.stdout: 2 16 2847 2831 88.4598 135.625 0.00530836 0.0112798 2024-09-18T08:08:00.157 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:08:00.158 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:08:00.158 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2' 2024-09-18T08:08:00.259 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1 2024-09-18T08:08:00.297 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:08:00.298 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:08:00.298 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1' 2024-09-18T08:08:00.399 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2 2024-09-18T08:08:00.431 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:08:00.433 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:08:00.433 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2' 2024-09-18T08:08:00.534 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2 2024-09-18T08:08:00.574 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:08:00.575 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:08:00.575 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2' 2024-09-18T08:08:00.656 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:08:00 smithi050 ceph-mon[32371]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-18T08:08:00.656 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:08:00 smithi050 ceph-mon[32371]: pgmap v242: 158 pgs: 43 peering, 2 active+clean+scrubbing, 113 active+clean; 29 MiB data, 870 MiB used, 625 GiB / 626 GiB avail; 7.1 MiB/s wr, 114 op/s 2024-09-18T08:08:00.656 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:08:00 smithi050 ceph-mon[32371]: Health check failed: Reduced data availability: 10 pgs inactive, 19 pgs peering (PG_AVAILABILITY) 2024-09-18T08:08:00.656 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:08:00 smithi050 ceph-mon[32371]: osdmap e172: 8 total, 8 up, 7 in 2024-09-18T08:08:00.656 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:08:00 smithi050 ceph-mon[32371]: 1.5 scrub starts 2024-09-18T08:08:00.656 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:08:00 smithi050 ceph-mon[32371]: 1.8 scrub starts 2024-09-18T08:08:00.676 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4 2024-09-18T08:08:00.681 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:08:00 smithi007 ceph-mon[29459]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-18T08:08:00.682 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:08:00 smithi007 ceph-mon[29459]: pgmap v242: 158 pgs: 43 peering, 2 active+clean+scrubbing, 113 active+clean; 29 MiB data, 870 MiB used, 625 GiB / 626 GiB avail; 7.1 MiB/s wr, 114 op/s 2024-09-18T08:08:00.682 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:08:00 smithi007 ceph-mon[29459]: Health check failed: Reduced data availability: 10 pgs inactive, 19 pgs peering (PG_AVAILABILITY) 2024-09-18T08:08:00.682 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:08:00 smithi007 ceph-mon[29459]: osdmap e172: 8 total, 8 up, 7 in 2024-09-18T08:08:00.682 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:08:00 smithi007 ceph-mon[29459]: 1.5 scrub starts 2024-09-18T08:08:00.682 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:08:00 smithi007 ceph-mon[29459]: 1.8 scrub starts 2024-09-18T08:08:00.682 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:08:00 smithi007 ceph-mon[39249]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-18T08:08:00.682 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:08:00 smithi007 ceph-mon[39249]: pgmap v242: 158 pgs: 43 peering, 2 active+clean+scrubbing, 113 active+clean; 29 MiB data, 870 MiB used, 625 GiB / 626 GiB avail; 7.1 MiB/s wr, 114 op/s 2024-09-18T08:08:00.682 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:08:00 smithi007 ceph-mon[39249]: Health check failed: Reduced data availability: 10 pgs inactive, 19 pgs peering (PG_AVAILABILITY) 2024-09-18T08:08:00.682 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:08:00 smithi007 ceph-mon[39249]: osdmap e172: 8 total, 8 up, 7 in 2024-09-18T08:08:00.682 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:08:00 smithi007 ceph-mon[39249]: 1.5 scrub starts 2024-09-18T08:08:00.683 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:08:00 smithi007 ceph-mon[39249]: 1.8 scrub starts 2024-09-18T08:08:00.704 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:08:00.705 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:08:00.705 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4' 2024-09-18T08:08:00.806 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3 2024-09-18T08:08:00.843 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:08:00.844 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:08:00.845 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3' 2024-09-18T08:08:00.940 DEBUG:teuthology.orchestra.run.smithi050:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid 779349ca-7593-11ef-bceb-c7b262605968 -- ceph daemon osd.5 dump_historic_ops 2024-09-18T08:08:00.946 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4 2024-09-18T08:08:01.018 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:08:01.018 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:08:01.019 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4' 2024-09-18T08:08:01.120 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2 2024-09-18T08:08:01.153 INFO:tasks.radosbench.radosbench.0.smithi007.stdout: 3 16 5869 5853 121.926 188.875 0.00367577 0.00818893 2024-09-18T08:08:01.161 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:08:01.162 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:08:01.162 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2' 2024-09-18T08:08:01.263 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4 2024-09-18T08:08:01.298 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:08:01.299 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:08:01.299 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4' 2024-09-18T08:08:01.331 INFO:teuthology.orchestra.run.smithi050.stderr:Inferring config /var/lib/ceph/779349ca-7593-11ef-bceb-c7b262605968/mon.b/config 2024-09-18T08:08:01.365 INFO:teuthology.orchestra.run.smithi007.stderr:nodeep-scrub is set 2024-09-18T08:08:01.400 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2 2024-09-18T08:08:01.442 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:08:01.443 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:08:01.443 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2' 2024-09-18T08:08:01.544 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2 2024-09-18T08:08:01.586 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:08:01.587 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:08:01.587 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2' 2024-09-18T08:08:01.656 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:08:01 smithi050 ceph-mon[32371]: 1.2 scrub starts 2024-09-18T08:08:01.656 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:08:01 smithi050 ceph-mon[32371]: 1.3 scrub starts 2024-09-18T08:08:01.656 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:08:01 smithi050 ceph-mon[32371]: purged_snaps scrub starts 2024-09-18T08:08:01.656 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:08:01 smithi050 ceph-mon[32371]: purged_snaps scrub ok 2024-09-18T08:08:01.656 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:08:01 smithi050 ceph-mon[32371]: from='client.? 172.21.15.7:0/3727317036' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-18T08:08:01.688 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4 2024-09-18T08:08:01.716 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:08:01.717 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:08:01.717 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4' 2024-09-18T08:08:01.719 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:08:01 smithi007 ceph-mon[29459]: 1.2 scrub starts 2024-09-18T08:08:01.719 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:08:01 smithi007 ceph-mon[29459]: 1.3 scrub starts 2024-09-18T08:08:01.719 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:08:01 smithi007 ceph-mon[29459]: purged_snaps scrub starts 2024-09-18T08:08:01.719 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:08:01 smithi007 ceph-mon[29459]: purged_snaps scrub ok 2024-09-18T08:08:01.720 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:08:01 smithi007 ceph-mon[29459]: from='client.? 172.21.15.7:0/3727317036' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-18T08:08:01.720 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:08:01 smithi007 ceph-mon[39249]: 1.2 scrub starts 2024-09-18T08:08:01.720 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:08:01 smithi007 ceph-mon[39249]: 1.3 scrub starts 2024-09-18T08:08:01.720 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:08:01 smithi007 ceph-mon[39249]: purged_snaps scrub starts 2024-09-18T08:08:01.720 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:08:01 smithi007 ceph-mon[39249]: purged_snaps scrub ok 2024-09-18T08:08:01.720 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:08:01 smithi007 ceph-mon[39249]: from='client.? 172.21.15.7:0/3727317036' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-18T08:08:01.765 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid 779349ca-7593-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-18T08:08:01.818 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2 2024-09-18T08:08:01.858 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:08:01.859 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:08:01.859 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2' 2024-09-18T08:08:01.960 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1 2024-09-18T08:08:02.007 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:08:02.007 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:08:02.008 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1' 2024-09-18T08:08:02.051 INFO:teuthology.orchestra.run.smithi050.stdout:ERROR: (22) Invalid argument 2024-09-18T08:08:02.051 INFO:teuthology.orchestra.run.smithi050.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-18T08:08:02.109 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6 2024-09-18T08:08:02.140 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:08:02.141 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:08:02.141 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6' 2024-09-18T08:08:02.153 INFO:tasks.radosbench.radosbench.0.smithi007.stdout: 4 15 8652 8637 134.929 174 0.0094089 0.00740212 2024-09-18T08:08:02.242 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3 2024-09-18T08:08:02.269 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/779349ca-7593-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T08:08:02.288 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:08:02.289 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:08:02.289 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3' 2024-09-18T08:08:02.390 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0 2024-09-18T08:08:02.406 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:08:02 smithi007 ceph-mon[39249]: pgmap v244: 158 pgs: 43 peering, 2 active+clean+scrubbing, 113 active+clean; 29 MiB data, 870 MiB used, 625 GiB / 626 GiB avail; 5.9 MiB/s wr, 94 op/s 2024-09-18T08:08:02.406 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:08:02 smithi007 ceph-mon[39249]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-18T08:08:02.406 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:08:02 smithi007 ceph-mon[39249]: from='client.? 172.21.15.7:0/3727317036' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-18T08:08:02.406 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:08:02 smithi007 ceph-mon[39249]: osdmap e173: 8 total, 8 up, 7 in 2024-09-18T08:08:02.407 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:08:02 smithi007 ceph-mon[29459]: pgmap v244: 158 pgs: 43 peering, 2 active+clean+scrubbing, 113 active+clean; 29 MiB data, 870 MiB used, 625 GiB / 626 GiB avail; 5.9 MiB/s wr, 94 op/s 2024-09-18T08:08:02.407 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:08:02 smithi007 ceph-mon[29459]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-18T08:08:02.407 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:08:02 smithi007 ceph-mon[29459]: from='client.? 172.21.15.7:0/3727317036' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-18T08:08:02.407 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:08:02 smithi007 ceph-mon[29459]: osdmap e173: 8 total, 8 up, 7 in 2024-09-18T08:08:02.429 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:08:02.430 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:08:02.430 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0' 2024-09-18T08:08:02.531 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6 2024-09-18T08:08:02.564 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:08:02.565 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:08:02.565 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6' 2024-09-18T08:08:02.656 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:08:02 smithi050 ceph-mon[32371]: pgmap v244: 158 pgs: 43 peering, 2 active+clean+scrubbing, 113 active+clean; 29 MiB data, 870 MiB used, 625 GiB / 626 GiB avail; 5.9 MiB/s wr, 94 op/s 2024-09-18T08:08:02.656 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:08:02 smithi050 ceph-mon[32371]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-18T08:08:02.656 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:08:02 smithi050 ceph-mon[32371]: from='client.? 172.21.15.7:0/3727317036' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-18T08:08:02.657 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:08:02 smithi050 ceph-mon[32371]: osdmap e173: 8 total, 8 up, 7 in 2024-09-18T08:08:02.666 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1 2024-09-18T08:08:02.705 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:08:02.706 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:08:02.706 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1' 2024-09-18T08:08:02.807 DEBUG:teuthology.orchestra.run.smithi050:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid 779349ca-7593-11ef-bceb-c7b262605968 -- ceph daemon osd.6 dump_ops_in_flight 2024-09-18T08:08:02.807 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4 2024-09-18T08:08:02.838 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:08:02.839 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:08:02.839 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4' 2024-09-18T08:08:02.940 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1 2024-09-18T08:08:02.950 INFO:teuthology.orchestra.run.smithi007.stdout:osd.0: {} 2024-09-18T08:08:02.950 INFO:teuthology.orchestra.run.smithi007.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-18T08:08:02.981 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:08:02.982 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:08:02.982 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1' 2024-09-18T08:08:02.991 INFO:teuthology.orchestra.run.smithi007.stdout:osd.1: {} 2024-09-18T08:08:02.991 INFO:teuthology.orchestra.run.smithi007.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-18T08:08:03.031 INFO:teuthology.orchestra.run.smithi007.stdout:osd.2: {} 2024-09-18T08:08:03.031 INFO:teuthology.orchestra.run.smithi007.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-18T08:08:03.068 INFO:teuthology.orchestra.run.smithi007.stdout:osd.3: {} 2024-09-18T08:08:03.068 INFO:teuthology.orchestra.run.smithi007.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-18T08:08:03.083 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4 2024-09-18T08:08:03.105 INFO:teuthology.orchestra.run.smithi007.stdout:osd.4: {} 2024-09-18T08:08:03.105 INFO:teuthology.orchestra.run.smithi007.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-18T08:08:03.116 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:08:03.116 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:08:03.117 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4' 2024-09-18T08:08:03.141 INFO:teuthology.orchestra.run.smithi007.stdout:osd.5: {} 2024-09-18T08:08:03.141 INFO:teuthology.orchestra.run.smithi007.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-18T08:08:03.153 INFO:tasks.radosbench.radosbench.0.smithi007.stdout: 5 16 11299 11283 141.015 165.375 0.00693776 0.00708154 2024-09-18T08:08:03.176 INFO:teuthology.orchestra.run.smithi007.stdout:osd.6: {} 2024-09-18T08:08:03.176 INFO:teuthology.orchestra.run.smithi007.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-18T08:08:03.187 INFO:teuthology.orchestra.run.smithi050.stderr:Inferring config /var/lib/ceph/779349ca-7593-11ef-bceb-c7b262605968/mon.b/config 2024-09-18T08:08:03.215 INFO:teuthology.orchestra.run.smithi007.stdout:osd.7: {} 2024-09-18T08:08:03.215 INFO:teuthology.orchestra.run.smithi007.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-18T08:08:03.218 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5 2024-09-18T08:08:03.249 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:08:03.250 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:08:03.250 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5' 2024-09-18T08:08:03.351 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5 2024-09-18T08:08:03.382 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:08:03.383 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:08:03.383 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5' 2024-09-18T08:08:03.484 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1 2024-09-18T08:08:03.519 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:08:03.520 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:08:03.520 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1' 2024-09-18T08:08:03.611 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:08:03 smithi050 ceph-mon[32371]: from='client.? 172.21.15.7:0/515222465' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-18T08:08:03.611 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:08:03 smithi050 ceph-mon[32371]: from='mgr.24470 172.21.15.7:0/2594390261' entity='mgr.y' 2024-09-18T08:08:03.612 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:08:03 smithi050 ceph-mon[32371]: from='mgr.24470 172.21.15.7:0/2594390261' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-18T08:08:03.621 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2 2024-09-18T08:08:03.661 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:08:03.662 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:08:03.662 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2' 2024-09-18T08:08:03.730 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:08:03 smithi007 ceph-mon[39249]: from='client.? 172.21.15.7:0/515222465' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-18T08:08:03.730 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:08:03 smithi007 ceph-mon[39249]: from='mgr.24470 172.21.15.7:0/2594390261' entity='mgr.y' 2024-09-18T08:08:03.730 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:08:03 smithi007 ceph-mon[39249]: from='mgr.24470 172.21.15.7:0/2594390261' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-18T08:08:03.731 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:08:03 smithi007 ceph-mon[29459]: from='client.? 172.21.15.7:0/515222465' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-18T08:08:03.731 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:08:03 smithi007 ceph-mon[29459]: from='mgr.24470 172.21.15.7:0/2594390261' entity='mgr.y' 2024-09-18T08:08:03.731 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:08:03 smithi007 ceph-mon[29459]: from='mgr.24470 172.21.15.7:0/2594390261' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-18T08:08:03.762 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2024-09-18T08:08:03.763 INFO:teuthology.orchestra.run.smithi050.stdout: "ops": [ 2024-09-18T08:08:03.763 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:08:03.763 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_repop(client.15216.0:12888 8.5 e173/171)", 2024-09-18T08:08:03.763 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-09-18T08:08:03.759715+0000", 2024-09-18T08:08:03.763 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 4.2867999999999997e-05, 2024-09-18T08:08:03.763 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 6.4029999999999995e-05, 2024-09-18T08:08:03.763 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-09-18T08:08:03.763 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-09-18T08:08:03.763 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "queued for pg", 2024-09-18T08:08:03.763 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-09-18T08:08:03.764 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:08:03.764 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-09-18T08:08:03.764 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:08:03.759715+0000", 2024-09-18T08:08:03.764 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-18T08:08:03.764 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:08:03.764 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:08:03.764 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-09-18T08:08:03.764 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:08:03.759715+0000", 2024-09-18T08:08:03.764 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-18T08:08:03.764 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:08:03.764 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:08:03.764 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-09-18T08:08:03.764 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:08:03.759715+0000", 2024-09-18T08:08:03.765 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 3.89e-07 2024-09-18T08:08:03.765 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:08:03.765 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:08:03.765 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-09-18T08:08:03.765 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:08:03.759734+0000", 2024-09-18T08:08:03.765 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.9352999999999999e-05 2024-09-18T08:08:03.765 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:08:03.765 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:08:03.765 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-09-18T08:08:03.765 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:08:03.759743+0000", 2024-09-18T08:08:03.765 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 8.7169999999999997e-06 2024-09-18T08:08:03.765 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:08:03.765 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:08:03.766 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-09-18T08:08:03.766 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:08:03.759748+0000", 2024-09-18T08:08:03.766 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 5.2700000000000004e-06 2024-09-18T08:08:03.766 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-18T08:08:03.766 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-09-18T08:08:03.766 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-18T08:08:03.766 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:08:03.766 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:08:03.766 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_repop(client.15216.0:12889 8.7 e173/165 8:e207700e:::benchmark_data_smithi007_101583_object12888:head v 173'834, mlcod=173'832)", 2024-09-18T08:08:03.766 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-09-18T08:08:03.759604+0000", 2024-09-18T08:08:03.766 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 0.00015324699999999999, 2024-09-18T08:08:03.766 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00021367099999999999, 2024-09-18T08:08:03.766 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-09-18T08:08:03.767 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-09-18T08:08:03.767 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "started", 2024-09-18T08:08:03.767 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-09-18T08:08:03.767 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:08:03.767 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-09-18T08:08:03.767 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:08:03.759604+0000", 2024-09-18T08:08:03.767 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-18T08:08:03.767 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:08:03.767 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:08:03.767 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-09-18T08:08:03.767 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:08:03.759604+0000", 2024-09-18T08:08:03.767 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-18T08:08:03.767 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:08:03.768 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:08:03.768 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-09-18T08:08:03.768 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:08:03.759605+0000", 2024-09-18T08:08:03.768 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 6.1600000000000001e-07 2024-09-18T08:08:03.768 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:08:03.768 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:08:03.768 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-09-18T08:08:03.768 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:08:03.759658+0000", 2024-09-18T08:08:03.768 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 5.2976999999999998e-05 2024-09-18T08:08:03.768 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:08:03.768 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:08:03.768 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-09-18T08:08:03.768 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:08:03.759668+0000", 2024-09-18T08:08:03.769 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.0407e-05 2024-09-18T08:08:03.769 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:08:03.769 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:08:03.769 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-09-18T08:08:03.769 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:08:03.759674+0000", 2024-09-18T08:08:03.769 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 6.1510000000000001e-06 2024-09-18T08:08:03.769 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:08:03.769 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:08:03.769 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-18T08:08:03.769 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:08:03.759733+0000", 2024-09-18T08:08:03.769 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 5.8826999999999998e-05 2024-09-18T08:08:03.769 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:08:03.770 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:08:03.770 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-09-18T08:08:03.770 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:08:03.759751+0000", 2024-09-18T08:08:03.770 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.736e-05 2024-09-18T08:08:03.770 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-18T08:08:03.770 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-09-18T08:08:03.770 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-18T08:08:03.770 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-18T08:08:03.770 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2024-09-18T08:08:03.770 INFO:teuthology.orchestra.run.smithi050.stdout: "num_ops": 2 2024-09-18T08:08:03.770 INFO:teuthology.orchestra.run.smithi050.stdout:} 2024-09-18T08:08:03.771 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0 2024-09-18T08:08:03.806 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:08:03.807 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:08:03.807 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0' 2024-09-18T08:08:03.908 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4 2024-09-18T08:08:03.942 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:08:03.943 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:08:03.943 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4' 2024-09-18T08:08:04.044 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4 2024-09-18T08:08:04.080 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:08:04.081 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:08:04.081 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4' 2024-09-18T08:08:04.153 INFO:tasks.radosbench.radosbench.0.smithi007.stdout: 6 16 13963 13947 145.26 166.5 0.00234443 0.00687399 2024-09-18T08:08:04.182 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5 2024-09-18T08:08:04.198 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid 779349ca-7593-11ef-bceb-c7b262605968 -- ceph osd unset noscrub 2024-09-18T08:08:04.211 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:08:04.212 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:08:04.212 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5' 2024-09-18T08:08:04.219 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 5, 7, 6] out_osds: [4] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-09-18T08:08:04.219 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-18T08:08:04.219 INFO:tasks.thrashosds.thrasher:Reweighting osd 2 to 0.6778833996177698 2024-09-18T08:08:04.219 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid 779349ca-7593-11ef-bceb-c7b262605968 -- ceph osd reweight 2 0.6778833996177698 2024-09-18T08:08:04.313 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1 2024-09-18T08:08:04.374 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:08:04.375 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:08:04.375 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1' 2024-09-18T08:08:04.476 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2 2024-09-18T08:08:04.537 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:08:04.538 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:08:04.538 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2' 2024-09-18T08:08:04.594 DEBUG:teuthology.orchestra.run.smithi050:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid 779349ca-7593-11ef-bceb-c7b262605968 -- ceph daemon osd.6 dump_blocked_ops 2024-09-18T08:08:04.626 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:08:04 smithi050 ceph-mon[32371]: pgmap v246: 158 pgs: 43 peering, 2 active+clean+scrubbing, 113 active+clean; 202 MiB data, 1.2 GiB used, 625 GiB / 626 GiB avail; 34 MiB/s wr, 551 op/s; 362 KiB/s, 0 objects/s recovering 2024-09-18T08:08:04.639 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4 2024-09-18T08:08:04.673 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:08:04.673 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:08:04.674 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4' 2024-09-18T08:08:04.701 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/779349ca-7593-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T08:08:04.730 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:08:04 smithi007 ceph-mon[39249]: pgmap v246: 158 pgs: 43 peering, 2 active+clean+scrubbing, 113 active+clean; 202 MiB data, 1.2 GiB used, 625 GiB / 626 GiB avail; 34 MiB/s wr, 551 op/s; 362 KiB/s, 0 objects/s recovering 2024-09-18T08:08:04.733 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:08:04 smithi007 ceph-mon[29459]: pgmap v246: 158 pgs: 43 peering, 2 active+clean+scrubbing, 113 active+clean; 202 MiB data, 1.2 GiB used, 625 GiB / 626 GiB avail; 34 MiB/s wr, 551 op/s; 362 KiB/s, 0 objects/s recovering 2024-09-18T08:08:04.761 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/779349ca-7593-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T08:08:04.775 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2 2024-09-18T08:08:04.820 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:08:04.821 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:08:04.822 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2' 2024-09-18T08:08:04.922 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6 2024-09-18T08:08:04.959 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:08:04.960 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:08:04.961 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6' 2024-09-18T08:08:04.982 INFO:teuthology.orchestra.run.smithi050.stderr:Inferring config /var/lib/ceph/779349ca-7593-11ef-bceb-c7b262605968/mon.b/config 2024-09-18T08:08:05.061 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3 2024-09-18T08:08:05.099 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:08:05.101 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:08:05.101 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3' 2024-09-18T08:08:05.153 INFO:tasks.radosbench.radosbench.0.smithi007.stdout: 7 16 16272 16256 145.122 144.312 0.00289237 0.0068831 2024-09-18T08:08:05.202 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3 2024-09-18T08:08:05.243 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:08:05.243 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:08:05.244 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3' 2024-09-18T08:08:05.344 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6 2024-09-18T08:08:05.376 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:08:05.377 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:08:05.377 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6' 2024-09-18T08:08:05.478 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6 2024-09-18T08:08:05.509 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:08:05.509 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:08:05.510 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6' 2024-09-18T08:08:05.611 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2 2024-09-18T08:08:05.648 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:08:05 smithi050 ceph-mon[32371]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 10 pgs inactive, 19 pgs peering) 2024-09-18T08:08:05.663 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:08:05.664 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:08:05.664 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2' 2024-09-18T08:08:05.686 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2024-09-18T08:08:05.687 INFO:teuthology.orchestra.run.smithi050.stdout: "ops": [], 2024-09-18T08:08:05.687 INFO:teuthology.orchestra.run.smithi050.stdout: "complaint_time": 30, 2024-09-18T08:08:05.687 INFO:teuthology.orchestra.run.smithi050.stdout: "num_blocked_ops": 0 2024-09-18T08:08:05.687 INFO:teuthology.orchestra.run.smithi050.stdout:} 2024-09-18T08:08:05.727 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:08:05 smithi007 ceph-mon[29459]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 10 pgs inactive, 19 pgs peering) 2024-09-18T08:08:05.728 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:08:05 smithi007 ceph-mon[39249]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 10 pgs inactive, 19 pgs peering) 2024-09-18T08:08:05.765 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2 2024-09-18T08:08:05.809 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:08:05.810 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:08:05.810 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2' 2024-09-18T08:08:05.911 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0 2024-09-18T08:08:05.933 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:08:05.933 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:08:05.934 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0' 2024-09-18T08:08:05.991 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid 779349ca-7593-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-18T08:08:06.035 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5 2024-09-18T08:08:06.066 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:08:06.066 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:08:06.067 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5' 2024-09-18T08:08:06.153 INFO:tasks.radosbench.radosbench.0.smithi007.stdout: 8 16 18338 18322 143.121 129.125 0.00995228 0.00697974 2024-09-18T08:08:06.167 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3 2024-09-18T08:08:06.228 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:08:06.228 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:08:06.229 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3' 2024-09-18T08:08:06.330 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6 2024-09-18T08:08:06.359 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:08:06.360 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:08:06.360 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6' 2024-09-18T08:08:06.429 INFO:teuthology.orchestra.run.smithi007.stderr:noscrub is unset 2024-09-18T08:08:06.461 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4 2024-09-18T08:08:06.492 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:08:06.493 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:08:06.494 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4' 2024-09-18T08:08:06.594 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5 2024-09-18T08:08:06.616 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:08:06.617 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:08:06.617 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5' 2024-09-18T08:08:06.699 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:08:06 smithi007 ceph-mon[29459]: pgmap v247: 158 pgs: 158 active+clean; 749 MiB data, 2.8 GiB used, 623 GiB / 626 GiB avail; 150 B/s rd, 110 MiB/s wr, 1.75k op/s; 1.9 MiB/s, 32 objects/s recovering 2024-09-18T08:08:06.699 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:08:06 smithi007 ceph-mon[29459]: purged_snaps scrub starts 2024-09-18T08:08:06.699 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:08:06 smithi007 ceph-mon[29459]: purged_snaps scrub ok 2024-09-18T08:08:06.700 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:08:06 smithi007 ceph-mon[29459]: from='client.? 172.21.15.7:0/2700799576' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-18T08:08:06.700 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:08:06 smithi007 ceph-mon[29459]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-18T08:08:06.700 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:08:06 smithi007 ceph-mon[39249]: pgmap v247: 158 pgs: 158 active+clean; 749 MiB data, 2.8 GiB used, 623 GiB / 626 GiB avail; 150 B/s rd, 110 MiB/s wr, 1.75k op/s; 1.9 MiB/s, 32 objects/s recovering 2024-09-18T08:08:06.700 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:08:06 smithi007 ceph-mon[39249]: purged_snaps scrub starts 2024-09-18T08:08:06.700 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:08:06 smithi007 ceph-mon[39249]: purged_snaps scrub ok 2024-09-18T08:08:06.700 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:08:06 smithi007 ceph-mon[39249]: from='client.? 172.21.15.7:0/2700799576' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-18T08:08:06.700 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:08:06 smithi007 ceph-mon[39249]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-18T08:08:06.718 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3 2024-09-18T08:08:06.760 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:08:06.761 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:08:06.761 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3' 2024-09-18T08:08:06.862 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0 2024-09-18T08:08:06.884 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:08:06.884 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:08:06.885 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0' 2024-09-18T08:08:06.906 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:08:06 smithi050 ceph-mon[32371]: pgmap v247: 158 pgs: 158 active+clean; 749 MiB data, 2.8 GiB used, 623 GiB / 626 GiB avail; 150 B/s rd, 110 MiB/s wr, 1.75k op/s; 1.9 MiB/s, 32 objects/s recovering 2024-09-18T08:08:06.906 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:08:06 smithi050 ceph-mon[32371]: purged_snaps scrub starts 2024-09-18T08:08:06.906 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:08:06 smithi050 ceph-mon[32371]: purged_snaps scrub ok 2024-09-18T08:08:06.906 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:08:06 smithi050 ceph-mon[32371]: from='client.? 172.21.15.7:0/2700799576' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-18T08:08:06.906 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:08:06 smithi050 ceph-mon[32371]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-18T08:08:06.980 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Sep 18 08:08:06 smithi007 ceph-779349ca-7593-11ef-bceb-c7b262605968-mgr-y[29688]: ::ffff:172.21.15.50 - - [18/Sep/2024:08:08:06] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-09-18T08:08:06.986 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2 2024-09-18T08:08:07.029 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:08:07.029 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:08:07.030 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2' 2024-09-18T08:08:07.131 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3 2024-09-18T08:08:07.153 INFO:tasks.radosbench.radosbench.0.smithi007.stdout: 9 16 20125 20109 139.628 111.688 0.00820816 0.00715412 2024-09-18T08:08:07.172 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:08:07.173 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:08:07.173 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3' 2024-09-18T08:08:07.274 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5 2024-09-18T08:08:07.297 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:08:07.297 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:08:07.297 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5' 2024-09-18T08:08:07.398 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7 2024-09-18T08:08:07.429 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:08:07.429 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:08:07.430 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7' 2024-09-18T08:08:07.530 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0 2024-09-18T08:08:07.549 DEBUG:teuthology.orchestra.run.smithi050:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid 779349ca-7593-11ef-bceb-c7b262605968 -- ceph daemon osd.6 dump_historic_ops 2024-09-18T08:08:07.569 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:08:07.569 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:08:07.570 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0' 2024-09-18T08:08:07.671 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7 2024-09-18T08:08:07.702 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:08:07.704 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:08:07.705 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7' 2024-09-18T08:08:07.730 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:08:07 smithi007 ceph-mon[29459]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-18T08:08:07.730 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:08:07 smithi007 ceph-mon[29459]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-18T08:08:07.731 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:08:07 smithi007 ceph-mon[29459]: osdmap e174: 8 total, 8 up, 7 in 2024-09-18T08:08:07.731 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:08:07 smithi007 ceph-mon[39249]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-18T08:08:07.731 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:08:07 smithi007 ceph-mon[39249]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-18T08:08:07.731 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:08:07 smithi007 ceph-mon[39249]: osdmap e174: 8 total, 8 up, 7 in 2024-09-18T08:08:07.805 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2 2024-09-18T08:08:07.846 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:08:07.847 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:08:07.847 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2' 2024-09-18T08:08:07.906 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:08:07 smithi050 ceph-mon[32371]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-18T08:08:07.906 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:08:07 smithi050 ceph-mon[32371]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-18T08:08:07.906 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:08:07 smithi050 ceph-mon[32371]: osdmap e174: 8 total, 8 up, 7 in 2024-09-18T08:08:07.917 INFO:teuthology.orchestra.run.smithi050.stderr:Inferring config /var/lib/ceph/779349ca-7593-11ef-bceb-c7b262605968/mon.b/config 2024-09-18T08:08:07.948 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5 2024-09-18T08:08:07.978 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:08:07.978 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:08:07.978 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5' 2024-09-18T08:08:08.079 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7 2024-09-18T08:08:08.111 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:08:08.112 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:08:08.112 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7' 2024-09-18T08:08:08.153 INFO:tasks.radosbench.radosbench.0.smithi007.stdout: 10 16 22015 21999 137.477 118.125 0.00694748 0.00726781 2024-09-18T08:08:08.213 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5 2024-09-18T08:08:08.238 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:08:08.240 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:08:08.240 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5' 2024-09-18T08:08:08.341 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3 2024-09-18T08:08:08.382 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:08:08.382 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:08:08.383 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3' 2024-09-18T08:08:08.444 INFO:teuthology.orchestra.run.smithi007.stderr:reweighted osd.2 to 0.677883 (ad89) 2024-09-18T08:08:08.484 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5 2024-09-18T08:08:08.490 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2024-09-18T08:08:08.491 INFO:teuthology.orchestra.run.smithi050.stdout: "size": 20, 2024-09-18T08:08:08.491 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 600, 2024-09-18T08:08:08.491 INFO:teuthology.orchestra.run.smithi050.stdout: "ops": [ 2024-09-18T08:08:08.491 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:08:08.491 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.14805.0:38000 7.7 7:e509d896:::benchmark_data_smithi007_73155_object37999:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e94)", 2024-09-18T08:08:08.491 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-09-18T08:04:23.049009+0000", 2024-09-18T08:08:08.491 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 225.43625996200001, 2024-09-18T08:08:08.491 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.027565702000000001, 2024-09-18T08:08:08.491 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-09-18T08:08:08.491 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-09-18T08:08:08.492 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T08:08:08.492 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2024-09-18T08:08:08.492 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.14805", 2024-09-18T08:08:08.492 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "172.21.15.7:0/640320436", 2024-09-18T08:08:08.492 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 38000 2024-09-18T08:08:08.492 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:08:08.492 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-09-18T08:08:08.492 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:08:08.492 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-09-18T08:08:08.492 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:23.049009+0000", 2024-09-18T08:08:08.492 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-18T08:08:08.493 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:08:08.493 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:08:08.493 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-09-18T08:08:08.493 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:23.049009+0000", 2024-09-18T08:08:08.493 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-18T08:08:08.493 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:08:08.493 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:08:08.493 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-09-18T08:08:08.493 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:23.049010+0000", 2024-09-18T08:08:08.493 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.11e-06 2024-09-18T08:08:08.493 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:08:08.493 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:08:08.494 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-09-18T08:08:08.494 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:23.068284+0000", 2024-09-18T08:08:08.494 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.019274264999999999 2024-09-18T08:08:08.494 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:08:08.494 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:08:08.494 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-09-18T08:08:08.494 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:23.068323+0000", 2024-09-18T08:08:08.494 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 3.8167999999999998e-05 2024-09-18T08:08:08.494 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:08:08.494 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:08:08.494 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-09-18T08:08:08.494 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:23.068329+0000", 2024-09-18T08:08:08.494 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 6.331e-06 2024-09-18T08:08:08.495 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:08:08.495 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:08:08.495 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-18T08:08:08.495 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:23.068399+0000", 2024-09-18T08:08:08.495 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 7.0055999999999995e-05 2024-09-18T08:08:08.495 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:08:08.495 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:08:08.495 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-09-18T08:08:08.495 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:23.068503+0000", 2024-09-18T08:08:08.495 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000104206 2024-09-18T08:08:08.495 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:08:08.495 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:08:08.496 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "waiting for subops from 0,3", 2024-09-18T08:08:08.496 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:23.068640+0000", 2024-09-18T08:08:08.496 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00013645000000000001 2024-09-18T08:08:08.496 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:08:08.496 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:08:08.496 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "op_commit", 2024-09-18T08:08:08.496 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:23.073270+0000", 2024-09-18T08:08:08.496 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0046298950000000002 2024-09-18T08:08:08.496 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:08:08.496 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:08:08.496 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:08:08.496 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:23.073311+0000", 2024-09-18T08:08:08.496 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4.1406999999999998e-05 2024-09-18T08:08:08.496 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:08:08.497 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:08:08.497 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:08:08.497 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:23.076521+0000", 2024-09-18T08:08:08.497 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.003209862 2024-09-18T08:08:08.497 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:08:08.497 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:08:08.497 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "commit_sent", 2024-09-18T08:08:08.497 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:23.076552+0000", 2024-09-18T08:08:08.497 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 3.1698e-05 2024-09-18T08:08:08.498 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:08:08.498 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:08:08.498 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-09-18T08:08:08.498 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:23.076575+0000", 2024-09-18T08:08:08.498 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 2.2254000000000001e-05 2024-09-18T08:08:08.498 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-18T08:08:08.498 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-09-18T08:08:08.498 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-18T08:08:08.498 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:08:08.498 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:08:08.498 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.14805.0:39064 7.7 7:ef833039:::benchmark_data_smithi007_73155_object39063:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e95)", 2024-09-18T08:08:08.498 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-09-18T08:04:23.658960+0000", 2024-09-18T08:08:08.498 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 224.82630868999999, 2024-09-18T08:08:08.499 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.029075624000000001, 2024-09-18T08:08:08.499 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-09-18T08:08:08.499 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-09-18T08:08:08.499 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T08:08:08.499 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2024-09-18T08:08:08.499 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.14805", 2024-09-18T08:08:08.499 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "172.21.15.7:0/640320436", 2024-09-18T08:08:08.499 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 39064 2024-09-18T08:08:08.499 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:08:08.499 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-09-18T08:08:08.499 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:08:08.500 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-09-18T08:08:08.500 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:23.658960+0000", 2024-09-18T08:08:08.500 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-18T08:08:08.500 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:08:08.500 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:08:08.500 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-09-18T08:08:08.500 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:23.658960+0000", 2024-09-18T08:08:08.500 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-18T08:08:08.500 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:08:08.500 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:08:08.500 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-09-18T08:08:08.500 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:23.658961+0000", 2024-09-18T08:08:08.500 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 6.7899999999999998e-07 2024-09-18T08:08:08.501 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:08:08.501 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:08:08.501 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-09-18T08:08:08.501 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:23.658991+0000", 2024-09-18T08:08:08.501 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 3.0168999999999999e-05 2024-09-18T08:08:08.501 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:08:08.501 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:08:08.501 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-09-18T08:08:08.501 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:23.659001+0000", 2024-09-18T08:08:08.501 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 9.6220000000000004e-06 2024-09-18T08:08:08.502 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:08:08.502 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:08:08.502 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-09-18T08:08:08.502 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:23.659010+0000", 2024-09-18T08:08:08.502 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 8.9160000000000007e-06 2024-09-18T08:08:08.502 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:08:08.502 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:08:08.502 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-18T08:08:08.503 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:23.661923+0000", 2024-09-18T08:08:08.503 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.002913279 2024-09-18T08:08:08.503 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:08:08.503 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:08:08.503 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-09-18T08:08:08.503 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:23.662030+0000", 2024-09-18T08:08:08.503 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00010665700000000001 2024-09-18T08:08:08.503 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:08:08.504 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:08:08.504 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "waiting for subops from 0,3", 2024-09-18T08:08:08.504 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:23.662167+0000", 2024-09-18T08:08:08.504 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00013728300000000001 2024-09-18T08:08:08.504 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:08:08.504 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:08:08.504 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "op_commit", 2024-09-18T08:08:08.504 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:23.663255+0000", 2024-09-18T08:08:08.505 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0010883900000000001 2024-09-18T08:08:08.505 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:08:08.505 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:08:08.505 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:08:08.505 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:23.671279+0000", 2024-09-18T08:08:08.505 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0080233749999999993 2024-09-18T08:08:08.505 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:08:08.505 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:08:08.505 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:08:08.506 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:23.687965+0000", 2024-09-18T08:08:08.506 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.016686590000000001 2024-09-18T08:08:08.506 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:08:08.506 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:08:08.506 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "commit_sent", 2024-09-18T08:08:08.506 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:23.688005+0000", 2024-09-18T08:08:08.506 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 3.9604000000000002e-05 2024-09-18T08:08:08.506 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:08:08.507 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:08:08.507 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-09-18T08:08:08.507 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:23.688036+0000", 2024-09-18T08:08:08.507 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 3.1059999999999997e-05 2024-09-18T08:08:08.507 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-18T08:08:08.507 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-09-18T08:08:08.507 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-18T08:08:08.507 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:08:08.508 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:08:08.508 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.14805.0:39805 7.7 7:e9cd36b3:::benchmark_data_smithi007_73155_object39804:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e95)", 2024-09-18T08:08:08.508 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-09-18T08:04:24.061655+0000", 2024-09-18T08:08:08.508 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 224.423614009, 2024-09-18T08:08:08.508 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.036883131, 2024-09-18T08:08:08.508 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-09-18T08:08:08.508 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-09-18T08:08:08.508 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T08:08:08.509 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2024-09-18T08:08:08.509 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.14805", 2024-09-18T08:08:08.509 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "172.21.15.7:0/640320436", 2024-09-18T08:08:08.509 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 39805 2024-09-18T08:08:08.509 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:08:08.509 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-09-18T08:08:08.509 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:08:08.509 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-09-18T08:08:08.510 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:24.061655+0000", 2024-09-18T08:08:08.510 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-18T08:08:08.510 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:08:08.510 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:08:08.510 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-09-18T08:08:08.510 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:24.061655+0000", 2024-09-18T08:08:08.510 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-18T08:08:08.510 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:08:08.510 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:08:08.511 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-09-18T08:08:08.511 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:24.061656+0000", 2024-09-18T08:08:08.511 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.4449999999999999e-06 2024-09-18T08:08:08.511 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:08:08.511 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:08:08.511 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-09-18T08:08:08.511 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:24.061686+0000", 2024-09-18T08:08:08.511 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 2.9984e-05 2024-09-18T08:08:08.511 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:08:08.512 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:08:08.512 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-09-18T08:08:08.512 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:24.061699+0000", 2024-09-18T08:08:08.512 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.2850000000000001e-05 2024-09-18T08:08:08.512 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:08:08.512 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:08:08.512 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-09-18T08:08:08.512 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:24.061706+0000", 2024-09-18T08:08:08.513 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 6.8560000000000001e-06 2024-09-18T08:08:08.513 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:08:08.513 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:08:08.513 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-18T08:08:08.513 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:24.061755+0000", 2024-09-18T08:08:08.513 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4.9001000000000001e-05 2024-09-18T08:08:08.513 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:08:08.514 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:08:08.514 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-09-18T08:08:08.514 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:24.061896+0000", 2024-09-18T08:08:08.514 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00014040499999999999 2024-09-18T08:08:08.514 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:08:08.514 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:08:08.514 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "waiting for subops from 0,3", 2024-09-18T08:08:08.514 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:24.062006+0000", 2024-09-18T08:08:08.515 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000110813 2024-09-18T08:08:08.515 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:08:08.515 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:08:08.515 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:08:08.515 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:24.065677+0000", 2024-09-18T08:08:08.515 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0036702800000000002 2024-09-18T08:08:08.515 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:08:08.516 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:08:08.516 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "op_commit", 2024-09-18T08:08:08.516 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:24.065694+0000", 2024-09-18T08:08:08.516 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.7876e-05 2024-09-18T08:08:08.516 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:08:08.516 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:08:08.516 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:08:08.516 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:24.098357+0000", 2024-09-18T08:08:08.516 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.032662140999999999 2024-09-18T08:08:08.516 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:08:08.516 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:08:08.516 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "commit_sent", 2024-09-18T08:08:08.516 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:24.098447+0000", 2024-09-18T08:08:08.517 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 9.0724999999999996e-05 2024-09-18T08:08:08.517 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:08:08.517 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:08:08.517 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-09-18T08:08:08.517 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:24.098538+0000", 2024-09-18T08:08:08.517 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 9.0754999999999994e-05 2024-09-18T08:08:08.517 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-18T08:08:08.517 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-09-18T08:08:08.520 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-18T08:08:08.520 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:08:08.520 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:08:08.520 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.14805.0:39807 7.d 7:bf1e27dc:::benchmark_data_smithi007_73155_object39806:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e95)", 2024-09-18T08:08:08.520 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-09-18T08:04:24.061827+0000", 2024-09-18T08:08:08.520 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 224.42344171400001, 2024-09-18T08:08:08.520 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.037358672000000002, 2024-09-18T08:08:08.520 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-09-18T08:08:08.521 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-09-18T08:08:08.521 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T08:08:08.521 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2024-09-18T08:08:08.521 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.14805", 2024-09-18T08:08:08.521 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "172.21.15.7:0/640320436", 2024-09-18T08:08:08.521 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 39807 2024-09-18T08:08:08.521 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:08:08.521 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-09-18T08:08:08.521 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:08:08.521 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-09-18T08:08:08.521 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:24.061827+0000", 2024-09-18T08:08:08.521 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-18T08:08:08.521 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:08:08.522 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:08:08.522 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-09-18T08:08:08.522 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:24.061827+0000", 2024-09-18T08:08:08.522 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-18T08:08:08.522 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:08:08.522 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:08:08.522 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-09-18T08:08:08.522 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:24.061828+0000", 2024-09-18T08:08:08.522 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 7.9899999999999999e-07 2024-09-18T08:08:08.522 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:08:08.522 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:08:08.522 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-09-18T08:08:08.523 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:24.061888+0000", 2024-09-18T08:08:08.523 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 5.9497000000000003e-05 2024-09-18T08:08:08.523 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:08:08.523 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:08:08.523 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-09-18T08:08:08.523 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:24.061904+0000", 2024-09-18T08:08:08.523 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.6393999999999999e-05 2024-09-18T08:08:08.523 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:08:08.523 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:08:08.523 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-09-18T08:08:08.523 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:24.061912+0000", 2024-09-18T08:08:08.523 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 8.0909999999999993e-06 2024-09-18T08:08:08.523 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:08:08.524 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:08:08.524 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-18T08:08:08.524 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:24.061968+0000", 2024-09-18T08:08:08.524 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 5.5543e-05 2024-09-18T08:08:08.524 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:08:08.524 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:08:08.524 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-09-18T08:08:08.524 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:24.062061+0000", 2024-09-18T08:08:08.524 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 9.3558000000000001e-05 2024-09-18T08:08:08.524 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:08:08.524 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:08:08.524 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "waiting for subops from 0,1", 2024-09-18T08:08:08.524 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:24.062163+0000", 2024-09-18T08:08:08.524 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000101903 2024-09-18T08:08:08.525 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:08:08.525 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:08:08.525 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "op_commit", 2024-09-18T08:08:08.525 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:24.064018+0000", 2024-09-18T08:08:08.525 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.001855071 2024-09-18T08:08:08.525 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:08:08.525 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:08:08.525 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:08:08.525 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:24.064283+0000", 2024-09-18T08:08:08.525 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000265173 2024-09-18T08:08:08.525 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:08:08.525 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:08:08.525 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:08:08.526 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:24.099043+0000", 2024-09-18T08:08:08.526 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.034760159999999998 2024-09-18T08:08:08.526 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:08:08.526 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:08:08.526 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "commit_sent", 2024-09-18T08:08:08.526 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:24.099100+0000", 2024-09-18T08:08:08.526 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 5.6147000000000003e-05 2024-09-18T08:08:08.526 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:08:08.526 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:08:08.526 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-09-18T08:08:08.526 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:24.099186+0000", 2024-09-18T08:08:08.526 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 8.6335999999999995e-05 2024-09-18T08:08:08.526 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-18T08:08:08.527 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-09-18T08:08:08.527 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-18T08:08:08.527 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:08:08.527 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:08:08.527 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.14805.0:39808 7.d 7:b98cab8f:::benchmark_data_smithi007_73155_object39807:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e95)", 2024-09-18T08:08:08.527 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-09-18T08:04:24.067964+0000", 2024-09-18T08:08:08.527 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 224.417305277, 2024-09-18T08:08:08.527 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.034258180999999999, 2024-09-18T08:08:08.527 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-09-18T08:08:08.527 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-09-18T08:08:08.527 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T08:08:08.527 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2024-09-18T08:08:08.527 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.14805", 2024-09-18T08:08:08.527 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "172.21.15.7:0/640320436", 2024-09-18T08:08:08.528 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 39808 2024-09-18T08:08:08.528 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:08:08.528 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-09-18T08:08:08.528 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:08:08.528 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-09-18T08:08:08.528 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:24.067964+0000", 2024-09-18T08:08:08.528 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-18T08:08:08.528 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:08:08.528 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:08:08.528 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-09-18T08:08:08.528 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:24.067964+0000", 2024-09-18T08:08:08.528 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-18T08:08:08.528 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:08:08.529 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:08:08.529 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-09-18T08:08:08.530 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:24.067965+0000", 2024-09-18T08:08:08.530 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.4750000000000001e-06 2024-09-18T08:08:08.530 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:08:08.530 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:08:08.530 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-09-18T08:08:08.530 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:24.067992+0000", 2024-09-18T08:08:08.530 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 2.6760000000000001e-05 2024-09-18T08:08:08.530 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:08:08.530 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:08:08.530 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-09-18T08:08:08.530 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:24.068004+0000", 2024-09-18T08:08:08.530 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.2456000000000001e-05 2024-09-18T08:08:08.531 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:08:08.531 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:08:08.531 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-09-18T08:08:08.531 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:24.068012+0000", 2024-09-18T08:08:08.531 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 8.0779999999999996e-06 2024-09-18T08:08:08.531 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:08:08.531 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:08:08.531 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-18T08:08:08.531 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:24.068072+0000", 2024-09-18T08:08:08.531 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 5.9299000000000003e-05 2024-09-18T08:08:08.531 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:08:08.531 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:08:08.531 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-09-18T08:08:08.532 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:24.068164+0000", 2024-09-18T08:08:08.532 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 9.2053999999999997e-05 2024-09-18T08:08:08.532 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:08:08.532 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:08:08.532 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "waiting for subops from 0,1", 2024-09-18T08:08:08.532 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:24.096945+0000", 2024-09-18T08:08:08.532 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.028781128 2024-09-18T08:08:08.532 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:08:08.532 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:08:08.532 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "op_commit", 2024-09-18T08:08:08.532 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:24.099254+0000", 2024-09-18T08:08:08.532 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.002309218 2024-09-18T08:08:08.533 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:08:08.533 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:08:08.533 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:08:08.533 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:24.100291+0000", 2024-09-18T08:08:08.533 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0010365280000000001 2024-09-18T08:08:08.533 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:08:08.533 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:08:08.533 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:08:08.533 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:24.102125+0000", 2024-09-18T08:08:08.533 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0018346599999999999 2024-09-18T08:08:08.533 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:08:08.533 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:08:08.533 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "commit_sent", 2024-09-18T08:08:08.534 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:24.102184+0000", 2024-09-18T08:08:08.534 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 5.8965000000000002e-05 2024-09-18T08:08:08.534 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:08:08.534 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:08:08.534 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-09-18T08:08:08.534 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:24.102222+0000", 2024-09-18T08:08:08.534 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 3.756e-05 2024-09-18T08:08:08.534 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-18T08:08:08.534 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-09-18T08:08:08.534 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-18T08:08:08.534 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:08:08.535 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:08:08.535 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_repop_reply(client.14805.0:39807 7.d e95/83 ondisk, result = 0)", 2024-09-18T08:08:08.535 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-09-18T08:04:24.069321+0000", 2024-09-18T08:08:08.535 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 224.415947913, 2024-09-18T08:08:08.535 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.029872724999999999, 2024-09-18T08:08:08.535 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-09-18T08:08:08.535 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-09-18T08:08:08.535 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "started", 2024-09-18T08:08:08.535 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-09-18T08:08:08.535 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:08:08.535 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-09-18T08:08:08.535 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:24.069321+0000", 2024-09-18T08:08:08.535 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-18T08:08:08.536 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:08:08.536 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:08:08.536 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-09-18T08:08:08.536 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:24.069321+0000", 2024-09-18T08:08:08.536 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-18T08:08:08.536 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:08:08.536 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:08:08.536 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-09-18T08:08:08.536 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:24.069322+0000", 2024-09-18T08:08:08.536 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 8.3799999999999996e-07 2024-09-18T08:08:08.536 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:08:08.536 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:08:08.537 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-09-18T08:08:08.537 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:24.069327+0000", 2024-09-18T08:08:08.537 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4.8679999999999998e-06 2024-09-18T08:08:08.537 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:08:08.537 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:08:08.537 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-09-18T08:08:08.537 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:24.098180+0000", 2024-09-18T08:08:08.537 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.028853407000000001 2024-09-18T08:08:08.537 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:08:08.537 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:08:08.537 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-09-18T08:08:08.537 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:24.098189+0000", 2024-09-18T08:08:08.537 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 8.8850000000000005e-06 2024-09-18T08:08:08.538 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:08:08.538 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:08:08.538 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-18T08:08:08.538 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:24.099029+0000", 2024-09-18T08:08:08.538 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00084036500000000004 2024-09-18T08:08:08.538 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:08:08.538 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:08:08.538 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-09-18T08:08:08.538 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:24.099037+0000", 2024-09-18T08:08:08.538 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 8.0150000000000003e-06 2024-09-18T08:08:08.538 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:08:08.538 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:08:08.538 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-09-18T08:08:08.539 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:24.099194+0000", 2024-09-18T08:08:08.539 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00015634700000000001 2024-09-18T08:08:08.539 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-18T08:08:08.539 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-09-18T08:08:08.539 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-18T08:08:08.539 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:08:08.539 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:08:08.539 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.14805.0:39817 7.c 7:3826d3a8:::benchmark_data_smithi007_73155_object39816:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e96)", 2024-09-18T08:08:08.539 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-09-18T08:04:24.077849+0000", 2024-09-18T08:08:08.539 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 224.407419965, 2024-09-18T08:08:08.539 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.028025705000000001, 2024-09-18T08:08:08.539 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-09-18T08:08:08.539 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-09-18T08:08:08.540 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T08:08:08.540 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2024-09-18T08:08:08.540 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.14805", 2024-09-18T08:08:08.540 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "172.21.15.7:0/640320436", 2024-09-18T08:08:08.540 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 39817 2024-09-18T08:08:08.540 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:08:08.540 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-09-18T08:08:08.540 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:08:08.540 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-09-18T08:08:08.540 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:24.077849+0000", 2024-09-18T08:08:08.540 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-18T08:08:08.540 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:08:08.540 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:08:08.541 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-09-18T08:08:08.541 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:24.077849+0000", 2024-09-18T08:08:08.541 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-18T08:08:08.541 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:08:08.541 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:08:08.541 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-09-18T08:08:08.541 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:24.077851+0000", 2024-09-18T08:08:08.541 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 2.322e-06 2024-09-18T08:08:08.541 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:08:08.541 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:08:08.541 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-09-18T08:08:08.541 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:24.077879+0000", 2024-09-18T08:08:08.541 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 2.7596000000000001e-05 2024-09-18T08:08:08.541 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:08:08.542 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:08:08.542 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-09-18T08:08:08.542 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:24.097805+0000", 2024-09-18T08:08:08.542 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.019925552999999999 2024-09-18T08:08:08.542 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:08:08.542 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:08:08.542 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-09-18T08:08:08.542 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:24.097820+0000", 2024-09-18T08:08:08.542 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.5705999999999998e-05 2024-09-18T08:08:08.542 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:08:08.542 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:08:08.542 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-18T08:08:08.542 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:24.097984+0000", 2024-09-18T08:08:08.542 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00016333499999999999 2024-09-18T08:08:08.543 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:08:08.543 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:08:08.543 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "op must wait for map", 2024-09-18T08:08:08.543 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:24.097990+0000", 2024-09-18T08:08:08.543 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 6.81e-06 2024-09-18T08:08:08.543 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:08:08.543 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:08:08.543 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-18T08:08:08.543 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:24.102003+0000", 2024-09-18T08:08:08.543 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0040126160000000001 2024-09-18T08:08:08.543 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:08:08.544 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:08:08.544 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-09-18T08:08:08.544 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:24.102155+0000", 2024-09-18T08:08:08.544 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00015247500000000001 2024-09-18T08:08:08.544 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:08:08.544 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:08:08.544 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "waiting for subops from 3,7", 2024-09-18T08:08:08.544 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:24.102305+0000", 2024-09-18T08:08:08.544 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000149872 2024-09-18T08:08:08.544 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:08:08.544 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:08:08.544 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "op_commit", 2024-09-18T08:08:08.544 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:24.105200+0000", 2024-09-18T08:08:08.545 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0028950730000000002 2024-09-18T08:08:08.545 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:08:08.545 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:08:08.545 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:08:08.545 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:24.105248+0000", 2024-09-18T08:08:08.545 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4.7787999999999997e-05 2024-09-18T08:08:08.545 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:08:08.545 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:08:08.545 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:08:08.545 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:24.105772+0000", 2024-09-18T08:08:08.545 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00052425699999999998 2024-09-18T08:08:08.545 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:08:08.545 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:08:08.546 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "commit_sent", 2024-09-18T08:08:08.546 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:24.105826+0000", 2024-09-18T08:08:08.546 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 5.393e-05 2024-09-18T08:08:08.546 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:08:08.546 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:08:08.546 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-09-18T08:08:08.546 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:24.105875+0000", 2024-09-18T08:08:08.546 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4.8371999999999997e-05 2024-09-18T08:08:08.546 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-18T08:08:08.546 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-09-18T08:08:08.546 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-18T08:08:08.546 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:08:08.546 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:08:08.547 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.14805.0:43273 7.c 7:38821c7c:::benchmark_data_smithi007_73155_object43272:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e96)", 2024-09-18T08:08:08.547 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-09-18T08:04:26.167716+0000", 2024-09-18T08:08:08.547 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 222.31755303599999, 2024-09-18T08:08:08.547 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.027026471999999999, 2024-09-18T08:08:08.547 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-09-18T08:08:08.547 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-09-18T08:08:08.547 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T08:08:08.547 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2024-09-18T08:08:08.547 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.14805", 2024-09-18T08:08:08.547 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "172.21.15.7:0/640320436", 2024-09-18T08:08:08.547 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 43273 2024-09-18T08:08:08.547 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:08:08.547 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-09-18T08:08:08.547 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:08:08.548 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-09-18T08:08:08.548 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:26.167716+0000", 2024-09-18T08:08:08.548 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-18T08:08:08.548 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:08:08.548 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:08:08.548 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-09-18T08:08:08.548 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:26.167716+0000", 2024-09-18T08:08:08.548 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-18T08:08:08.548 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:08:08.548 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:08:08.548 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-09-18T08:08:08.548 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:26.167717+0000", 2024-09-18T08:08:08.548 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.3629999999999999e-06 2024-09-18T08:08:08.549 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:08:08.549 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:08:08.549 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-09-18T08:08:08.549 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:26.187242+0000", 2024-09-18T08:08:08.549 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.019525094999999999 2024-09-18T08:08:08.549 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:08:08.549 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:08:08.549 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-09-18T08:08:08.549 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:26.187255+0000", 2024-09-18T08:08:08.549 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.2669e-05 2024-09-18T08:08:08.549 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:08:08.549 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:08:08.549 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-09-18T08:08:08.550 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:26.187264+0000", 2024-09-18T08:08:08.550 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 9.3770000000000002e-06 2024-09-18T08:08:08.550 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:08:08.550 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:08:08.550 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-18T08:08:08.550 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:26.187321+0000", 2024-09-18T08:08:08.550 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 5.6379999999999999e-05 2024-09-18T08:08:08.550 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:08:08.550 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:08:08.550 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-09-18T08:08:08.550 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:26.187444+0000", 2024-09-18T08:08:08.550 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00012314699999999999 2024-09-18T08:08:08.550 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:08:08.551 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:08:08.551 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "waiting for subops from 3,7", 2024-09-18T08:08:08.551 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:26.187575+0000", 2024-09-18T08:08:08.551 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00013138700000000001 2024-09-18T08:08:08.551 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:08:08.551 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:08:08.551 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "op_commit", 2024-09-18T08:08:08.551 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:26.189195+0000", 2024-09-18T08:08:08.551 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0016195770000000001 2024-09-18T08:08:08.551 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:08:08.552 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:08:08.552 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:08:08.552 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:26.189327+0000", 2024-09-18T08:08:08.552 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00013181099999999999 2024-09-18T08:08:08.552 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:08:08.552 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:08:08.552 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:08:08.552 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:26.194627+0000", 2024-09-18T08:08:08.552 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0052998799999999999 2024-09-18T08:08:08.552 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:08:08.552 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:08:08.552 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "commit_sent", 2024-09-18T08:08:08.552 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:26.194693+0000", 2024-09-18T08:08:08.553 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 6.6748999999999994e-05 2024-09-18T08:08:08.553 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:08:08.553 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:08:08.553 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-09-18T08:08:08.553 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:26.194742+0000", 2024-09-18T08:08:08.553 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4.9036999999999998e-05 2024-09-18T08:08:08.553 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-18T08:08:08.553 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-09-18T08:08:08.553 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-18T08:08:08.553 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:08:08.553 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:08:08.554 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.14805.0:43281 7.c 7:35a814bc:::benchmark_data_smithi007_73155_object43280:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e96)", 2024-09-18T08:08:08.554 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-09-18T08:04:26.187285+0000", 2024-09-18T08:08:08.554 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 222.297983848, 2024-09-18T08:08:08.554 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.026194973999999999, 2024-09-18T08:08:08.554 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-09-18T08:08:08.554 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-09-18T08:08:08.554 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T08:08:08.554 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2024-09-18T08:08:08.554 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.14805", 2024-09-18T08:08:08.554 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "172.21.15.7:0/640320436", 2024-09-18T08:08:08.554 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 43281 2024-09-18T08:08:08.555 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:08:08.555 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-09-18T08:08:08.555 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:08:08.555 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-09-18T08:08:08.555 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:26.187285+0000", 2024-09-18T08:08:08.555 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-18T08:08:08.555 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:08:08.555 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:08:08.555 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-09-18T08:08:08.555 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:26.187285+0000", 2024-09-18T08:08:08.556 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-18T08:08:08.556 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:08:08.556 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:08:08.556 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-09-18T08:08:08.556 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:26.187286+0000", 2024-09-18T08:08:08.556 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.2190000000000001e-06 2024-09-18T08:08:08.556 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:08:08.556 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:08:08.556 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-09-18T08:08:08.556 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:26.206523+0000", 2024-09-18T08:08:08.557 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.019236797999999999 2024-09-18T08:08:08.557 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:08:08.557 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:08:08.557 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-09-18T08:08:08.557 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:26.206550+0000", 2024-09-18T08:08:08.557 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 2.6509999999999999e-05 2024-09-18T08:08:08.557 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:08:08.557 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:08:08.557 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-09-18T08:08:08.557 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:26.206567+0000", 2024-09-18T08:08:08.558 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.7401000000000002e-05 2024-09-18T08:08:08.558 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:08:08.558 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:08:08.558 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-18T08:08:08.558 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:26.206657+0000", 2024-09-18T08:08:08.558 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 8.9485000000000001e-05 2024-09-18T08:08:08.558 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:08:08.558 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:08:08.558 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-09-18T08:08:08.558 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:26.206738+0000", 2024-09-18T08:08:08.558 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 8.1077999999999999e-05 2024-09-18T08:08:08.559 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:08:08.559 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:08:08.559 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "waiting for subops from 3,7", 2024-09-18T08:08:08.559 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:26.206832+0000", 2024-09-18T08:08:08.559 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 9.4160000000000006e-05 2024-09-18T08:08:08.559 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:08:08.559 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:08:08.559 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:08:08.559 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:26.213326+0000", 2024-09-18T08:08:08.559 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0064943099999999997 2024-09-18T08:08:08.559 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:08:08.559 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:08:08.559 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "op_commit", 2024-09-18T08:08:08.560 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:26.213342+0000", 2024-09-18T08:08:08.560 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.5415000000000001e-05 2024-09-18T08:08:08.560 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:08:08.560 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:08:08.560 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:08:08.560 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:26.213393+0000", 2024-09-18T08:08:08.560 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 5.1038999999999997e-05 2024-09-18T08:08:08.560 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:08:08.560 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:08:08.560 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "commit_sent", 2024-09-18T08:08:08.560 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:26.213443+0000", 2024-09-18T08:08:08.560 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4.9938999999999997e-05 2024-09-18T08:08:08.560 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:08:08.561 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:08:08.561 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-09-18T08:08:08.561 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:26.213480+0000", 2024-09-18T08:08:08.561 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 3.7620000000000002e-05 2024-09-18T08:08:08.561 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-18T08:08:08.561 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-09-18T08:08:08.561 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-18T08:08:08.561 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:08:08.561 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:08:08.561 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.14805.0:43731 7.d 7:b3fc8faf:::benchmark_data_smithi007_73155_object43730:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e97)", 2024-09-18T08:08:08.561 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-09-18T08:04:26.432079+0000", 2024-09-18T08:08:08.561 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 222.053189691, 2024-09-18T08:08:08.562 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.027497293999999999, 2024-09-18T08:08:08.562 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-09-18T08:08:08.562 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-09-18T08:08:08.562 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T08:08:08.562 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2024-09-18T08:08:08.562 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.14805", 2024-09-18T08:08:08.562 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "172.21.15.7:0/640320436", 2024-09-18T08:08:08.562 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 43731 2024-09-18T08:08:08.562 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:08:08.562 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-09-18T08:08:08.562 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:08:08.562 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-09-18T08:08:08.562 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:26.432079+0000", 2024-09-18T08:08:08.563 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-18T08:08:08.563 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:08:08.563 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:08:08.563 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-09-18T08:08:08.563 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:26.432079+0000", 2024-09-18T08:08:08.563 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-18T08:08:08.563 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:08:08.563 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:08:08.563 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-09-18T08:08:08.563 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:26.432080+0000", 2024-09-18T08:08:08.563 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.128e-06 2024-09-18T08:08:08.563 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:08:08.563 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:08:08.563 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-09-18T08:08:08.564 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:26.432153+0000", 2024-09-18T08:08:08.564 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 7.2521999999999994e-05 2024-09-18T08:08:08.564 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:08:08.564 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:08:08.564 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-09-18T08:08:08.564 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:26.432163+0000", 2024-09-18T08:08:08.564 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.0115999999999999e-05 2024-09-18T08:08:08.564 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:08:08.564 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:08:08.564 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-09-18T08:08:08.564 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:26.432170+0000", 2024-09-18T08:08:08.564 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 7.0729999999999999e-06 2024-09-18T08:08:08.565 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:08:08.565 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:08:08.565 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-18T08:08:08.565 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:26.432225+0000", 2024-09-18T08:08:08.565 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 5.4463999999999999e-05 2024-09-18T08:08:08.565 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:08:08.565 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:08:08.565 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-09-18T08:08:08.565 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:26.432342+0000", 2024-09-18T08:08:08.565 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000117118 2024-09-18T08:08:08.565 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:08:08.565 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:08:08.565 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "waiting for subops from 0,1", 2024-09-18T08:08:08.566 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:26.432467+0000", 2024-09-18T08:08:08.566 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00012505700000000001 2024-09-18T08:08:08.566 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:08:08.566 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:08:08.566 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "op_commit", 2024-09-18T08:08:08.566 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:26.433881+0000", 2024-09-18T08:08:08.566 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.001413812 2024-09-18T08:08:08.566 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:08:08.566 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:08:08.566 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:08:08.567 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:26.434589+0000", 2024-09-18T08:08:08.567 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00070883600000000001 2024-09-18T08:08:08.567 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:08:08.567 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:08:08.567 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:08:08.567 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:26.459469+0000", 2024-09-18T08:08:08.567 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.024879951000000001 2024-09-18T08:08:08.567 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:08:08.567 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:08:08.567 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "commit_sent", 2024-09-18T08:08:08.567 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:26.459527+0000", 2024-09-18T08:08:08.567 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 5.7126999999999997e-05 2024-09-18T08:08:08.568 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:08:08.568 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:08:08.568 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-09-18T08:08:08.568 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:26.459577+0000", 2024-09-18T08:08:08.568 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 5.0090000000000003e-05 2024-09-18T08:08:08.568 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-18T08:08:08.568 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-09-18T08:08:08.568 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-18T08:08:08.568 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:08:08.568 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:08:08.568 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.14805.0:63861 7.d 7:bf91474b:::benchmark_data_smithi007_73155_object63860:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e100)", 2024-09-18T08:08:08.569 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-09-18T08:04:41.125989+0000", 2024-09-18T08:08:08.569 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 207.359280399, 2024-09-18T08:08:08.569 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.031566575999999999, 2024-09-18T08:08:08.569 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-09-18T08:08:08.569 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-09-18T08:08:08.569 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T08:08:08.569 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2024-09-18T08:08:08.569 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.14805", 2024-09-18T08:08:08.569 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "172.21.15.7:0/640320436", 2024-09-18T08:08:08.569 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 63861 2024-09-18T08:08:08.569 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:08:08.570 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-09-18T08:08:08.570 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:08:08.570 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-09-18T08:08:08.570 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:41.125989+0000", 2024-09-18T08:08:08.570 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-18T08:08:08.570 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:08:08.570 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:08:08.570 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-09-18T08:08:08.570 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:41.125989+0000", 2024-09-18T08:08:08.570 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-18T08:08:08.570 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:08:08.570 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:08:08.570 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-09-18T08:08:08.570 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:41.125990+0000", 2024-09-18T08:08:08.571 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.6109999999999999e-06 2024-09-18T08:08:08.571 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:08:08.571 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:08:08.571 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-09-18T08:08:08.571 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:41.126017+0000", 2024-09-18T08:08:08.571 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 2.6594999999999999e-05 2024-09-18T08:08:08.571 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:08:08.571 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:08:08.571 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-09-18T08:08:08.572 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:41.126035+0000", 2024-09-18T08:08:08.572 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.7842999999999998e-05 2024-09-18T08:08:08.572 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:08:08.572 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:08:08.572 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-09-18T08:08:08.572 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:41.126043+0000", 2024-09-18T08:08:08.572 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 8.8489999999999995e-06 2024-09-18T08:08:08.572 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:08:08.572 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:08:08.572 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-18T08:08:08.572 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:41.126104+0000", 2024-09-18T08:08:08.573 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 6.0448999999999997e-05 2024-09-18T08:08:08.573 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:08:08.573 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:08:08.573 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-09-18T08:08:08.573 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:41.126225+0000", 2024-09-18T08:08:08.573 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000121101 2024-09-18T08:08:08.573 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:08:08.573 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:08:08.573 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "waiting for subops from 0,1", 2024-09-18T08:08:08.573 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:41.126366+0000", 2024-09-18T08:08:08.573 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00014059899999999999 2024-09-18T08:08:08.573 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:08:08.574 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:08:08.574 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "op_commit", 2024-09-18T08:08:08.574 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:41.129750+0000", 2024-09-18T08:08:08.574 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.003383863 2024-09-18T08:08:08.574 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:08:08.574 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:08:08.574 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:08:08.574 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:41.140266+0000", 2024-09-18T08:08:08.574 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.010516266999999999 2024-09-18T08:08:08.574 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:08:08.574 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:08:08.574 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:08:08.574 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:41.157441+0000", 2024-09-18T08:08:08.574 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.017174891000000001 2024-09-18T08:08:08.575 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:08:08.575 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:08:08.575 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "commit_sent", 2024-09-18T08:08:08.575 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:41.157509+0000", 2024-09-18T08:08:08.575 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 6.8627999999999994e-05 2024-09-18T08:08:08.575 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:08:08.575 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:08:08.575 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-09-18T08:08:08.575 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:41.157555+0000", 2024-09-18T08:08:08.576 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4.5880000000000001e-05 2024-09-18T08:08:08.576 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-18T08:08:08.576 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-09-18T08:08:08.576 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-18T08:08:08.576 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:08:08.576 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:08:08.577 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:08:08.577 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:08:08.578 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5' 2024-09-18T08:08:08.578 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.14805.0:63862 7.d 7:b1f318bc:::benchmark_data_smithi007_73155_object63861:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e100)", 2024-09-18T08:08:08.578 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-09-18T08:04:41.126934+0000", 2024-09-18T08:08:08.578 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 207.358335262, 2024-09-18T08:08:08.579 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.030729600999999999, 2024-09-18T08:08:08.579 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-09-18T08:08:08.579 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-09-18T08:08:08.579 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T08:08:08.579 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2024-09-18T08:08:08.579 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.14805", 2024-09-18T08:08:08.579 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "172.21.15.7:0/640320436", 2024-09-18T08:08:08.579 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 63862 2024-09-18T08:08:08.579 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:08:08.579 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-09-18T08:08:08.579 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:08:08.579 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-09-18T08:08:08.580 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:41.126934+0000", 2024-09-18T08:08:08.580 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-18T08:08:08.580 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:08:08.580 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:08:08.580 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-09-18T08:08:08.580 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:41.126934+0000", 2024-09-18T08:08:08.580 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-18T08:08:08.580 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:08:08.580 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:08:08.580 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-09-18T08:08:08.580 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:41.126935+0000", 2024-09-18T08:08:08.580 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 7.9599999999999998e-07 2024-09-18T08:08:08.580 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:08:08.581 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:08:08.581 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-09-18T08:08:08.581 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:41.126989+0000", 2024-09-18T08:08:08.581 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 5.4783e-05 2024-09-18T08:08:08.581 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:08:08.581 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:08:08.581 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-09-18T08:08:08.581 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:41.126999+0000", 2024-09-18T08:08:08.581 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 9.8889999999999997e-06 2024-09-18T08:08:08.581 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:08:08.581 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:08:08.581 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-09-18T08:08:08.581 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:41.127007+0000", 2024-09-18T08:08:08.581 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 7.4399999999999999e-06 2024-09-18T08:08:08.582 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:08:08.582 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:08:08.582 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-18T08:08:08.582 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:41.128171+0000", 2024-09-18T08:08:08.582 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0011642600000000001 2024-09-18T08:08:08.582 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:08:08.582 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:08:08.582 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-09-18T08:08:08.582 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:41.128287+0000", 2024-09-18T08:08:08.582 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000116508 2024-09-18T08:08:08.582 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:08:08.582 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:08:08.583 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "waiting for subops from 0,1", 2024-09-18T08:08:08.583 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:41.128375+0000", 2024-09-18T08:08:08.583 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 8.7300999999999996e-05 2024-09-18T08:08:08.583 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:08:08.583 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:08:08.583 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "op_commit", 2024-09-18T08:08:08.583 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:41.130919+0000", 2024-09-18T08:08:08.583 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0025447450000000002 2024-09-18T08:08:08.583 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:08:08.583 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:08:08.583 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:08:08.583 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:41.140334+0000", 2024-09-18T08:08:08.583 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0094146549999999992 2024-09-18T08:08:08.584 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:08:08.584 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:08:08.584 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:08:08.584 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:41.157602+0000", 2024-09-18T08:08:08.584 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.017267966999999999 2024-09-18T08:08:08.584 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:08:08.584 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:08:08.584 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "commit_sent", 2024-09-18T08:08:08.584 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:41.157636+0000", 2024-09-18T08:08:08.584 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 3.4214000000000001e-05 2024-09-18T08:08:08.585 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:08:08.585 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:08:08.585 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-09-18T08:08:08.585 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:41.157663+0000", 2024-09-18T08:08:08.585 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 2.7042999999999998e-05 2024-09-18T08:08:08.585 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-18T08:08:08.585 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-09-18T08:08:08.585 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-18T08:08:08.585 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:08:08.585 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:08:08.585 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.14805.0:63864 7.d 7:bb7b1761:::benchmark_data_smithi007_73155_object63863:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e100)", 2024-09-18T08:08:08.585 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-09-18T08:04:41.127581+0000", 2024-09-18T08:08:08.585 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 207.35768779899999, 2024-09-18T08:08:08.586 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.030231638000000002, 2024-09-18T08:08:08.586 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-09-18T08:08:08.586 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-09-18T08:08:08.586 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T08:08:08.586 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2024-09-18T08:08:08.586 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.14805", 2024-09-18T08:08:08.586 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "172.21.15.7:0/640320436", 2024-09-18T08:08:08.586 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 63864 2024-09-18T08:08:08.586 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:08:08.586 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-09-18T08:08:08.586 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:08:08.586 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-09-18T08:08:08.586 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:41.127581+0000", 2024-09-18T08:08:08.587 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-18T08:08:08.587 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:08:08.587 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:08:08.587 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-09-18T08:08:08.587 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:41.127581+0000", 2024-09-18T08:08:08.587 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-18T08:08:08.587 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:08:08.587 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:08:08.587 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-09-18T08:08:08.587 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:41.127582+0000", 2024-09-18T08:08:08.587 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 7.8999999999999995e-07 2024-09-18T08:08:08.588 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:08:08.588 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:08:08.588 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-09-18T08:08:08.588 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:41.127648+0000", 2024-09-18T08:08:08.588 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 6.5618999999999996e-05 2024-09-18T08:08:08.588 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:08:08.588 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:08:08.588 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-09-18T08:08:08.588 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:41.127658+0000", 2024-09-18T08:08:08.588 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.0192e-05 2024-09-18T08:08:08.588 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:08:08.588 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:08:08.588 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-09-18T08:08:08.589 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:41.127663+0000", 2024-09-18T08:08:08.589 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 5.2800000000000003e-06 2024-09-18T08:08:08.589 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:08:08.589 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:08:08.589 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-18T08:08:08.589 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:41.129783+0000", 2024-09-18T08:08:08.589 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0021199740000000002 2024-09-18T08:08:08.589 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:08:08.589 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:08:08.589 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-09-18T08:08:08.589 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:41.129885+0000", 2024-09-18T08:08:08.589 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000101987 2024-09-18T08:08:08.589 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:08:08.590 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:08:08.590 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "waiting for subops from 0,1", 2024-09-18T08:08:08.590 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:41.129980+0000", 2024-09-18T08:08:08.590 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 9.4480000000000003e-05 2024-09-18T08:08:08.590 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:08:08.590 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:08:08.590 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "op_commit", 2024-09-18T08:08:08.590 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:41.131035+0000", 2024-09-18T08:08:08.590 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.001055884 2024-09-18T08:08:08.590 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:08:08.590 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:08:08.590 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:08:08.590 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:41.140376+0000", 2024-09-18T08:08:08.591 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0093401439999999999 2024-09-18T08:08:08.591 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:08:08.591 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:08:08.591 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:08:08.591 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:41.157713+0000", 2024-09-18T08:08:08.591 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.017337676999999999 2024-09-18T08:08:08.591 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:08:08.591 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:08:08.591 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "commit_sent", 2024-09-18T08:08:08.591 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:41.157771+0000", 2024-09-18T08:08:08.591 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 5.7674999999999999e-05 2024-09-18T08:08:08.591 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:08:08.591 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:08:08.592 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-09-18T08:08:08.592 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:41.157813+0000", 2024-09-18T08:08:08.592 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4.1936e-05 2024-09-18T08:08:08.592 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-18T08:08:08.592 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-09-18T08:08:08.592 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-18T08:08:08.592 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:08:08.592 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:08:08.592 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.14805.0:63883 7.7 7:ebafa8b8:::benchmark_data_smithi007_73155_object63882:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e100)", 2024-09-18T08:08:08.592 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-09-18T08:04:41.154209+0000", 2024-09-18T08:08:08.592 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 207.33105960699999, 2024-09-18T08:08:08.592 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.042338318999999999, 2024-09-18T08:08:08.592 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-09-18T08:08:08.593 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-09-18T08:08:08.593 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T08:08:08.593 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2024-09-18T08:08:08.593 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.14805", 2024-09-18T08:08:08.593 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "172.21.15.7:0/640320436", 2024-09-18T08:08:08.593 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 63883 2024-09-18T08:08:08.593 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:08:08.593 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-09-18T08:08:08.593 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:08:08.593 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-09-18T08:08:08.593 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:41.154209+0000", 2024-09-18T08:08:08.594 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-18T08:08:08.594 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:08:08.594 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:08:08.594 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-09-18T08:08:08.594 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:41.154209+0000", 2024-09-18T08:08:08.594 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-18T08:08:08.595 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:08:08.595 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:08:08.595 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-09-18T08:08:08.595 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:41.154211+0000", 2024-09-18T08:08:08.595 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.2109999999999999e-06 2024-09-18T08:08:08.595 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:08:08.595 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:08:08.595 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-09-18T08:08:08.596 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:41.154259+0000", 2024-09-18T08:08:08.596 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4.8412000000000003e-05 2024-09-18T08:08:08.596 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:08:08.596 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:08:08.596 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-09-18T08:08:08.596 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:41.154270+0000", 2024-09-18T08:08:08.596 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.0515e-05 2024-09-18T08:08:08.596 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:08:08.597 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:08:08.597 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-09-18T08:08:08.597 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:41.154275+0000", 2024-09-18T08:08:08.597 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 5.4469999999999997e-06 2024-09-18T08:08:08.597 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:08:08.597 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:08:08.597 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-18T08:08:08.598 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:41.154323+0000", 2024-09-18T08:08:08.598 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4.8297999999999997e-05 2024-09-18T08:08:08.598 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:08:08.598 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:08:08.598 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-09-18T08:08:08.598 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:41.154456+0000", 2024-09-18T08:08:08.598 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000132593 2024-09-18T08:08:08.598 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:08:08.598 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:08:08.599 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "waiting for subops from 0,3", 2024-09-18T08:08:08.599 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:41.154615+0000", 2024-09-18T08:08:08.599 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000159049 2024-09-18T08:08:08.599 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:08:08.599 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:08:08.599 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "op_commit", 2024-09-18T08:08:08.599 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:41.156649+0000", 2024-09-18T08:08:08.600 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0020336709999999999 2024-09-18T08:08:08.600 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:08:08.600 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:08:08.600 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:08:08.600 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:41.165163+0000", 2024-09-18T08:08:08.600 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0085141100000000001 2024-09-18T08:08:08.600 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:08:08.600 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:08:08.601 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:08:08.601 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:41.196483+0000", 2024-09-18T08:08:08.601 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.031320300000000002 2024-09-18T08:08:08.601 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:08:08.601 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:08:08.601 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "commit_sent", 2024-09-18T08:08:08.601 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:41.196519+0000", 2024-09-18T08:08:08.602 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 3.6109999999999998e-05 2024-09-18T08:08:08.602 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:08:08.602 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:08:08.602 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-09-18T08:08:08.602 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:41.196548+0000", 2024-09-18T08:08:08.602 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 2.8603e-05 2024-09-18T08:08:08.602 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-18T08:08:08.602 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-09-18T08:08:08.603 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-18T08:08:08.603 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:08:08.603 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:08:08.603 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.14805.0:63908 7.d 7:bd06bac1:::benchmark_data_smithi007_73155_object63907:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e100)", 2024-09-18T08:08:08.603 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-09-18T08:04:41.183375+0000", 2024-09-18T08:08:08.603 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 207.301893845, 2024-09-18T08:08:08.603 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.033452764000000003, 2024-09-18T08:08:08.603 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-09-18T08:08:08.603 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-09-18T08:08:08.604 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T08:08:08.604 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2024-09-18T08:08:08.604 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.14805", 2024-09-18T08:08:08.604 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "172.21.15.7:0/640320436", 2024-09-18T08:08:08.604 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 63908 2024-09-18T08:08:08.604 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:08:08.604 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-09-18T08:08:08.604 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:08:08.605 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-09-18T08:08:08.605 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:41.183375+0000", 2024-09-18T08:08:08.605 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-18T08:08:08.605 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:08:08.605 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:08:08.605 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-09-18T08:08:08.605 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:41.183375+0000", 2024-09-18T08:08:08.605 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-18T08:08:08.606 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:08:08.606 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:08:08.606 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-09-18T08:08:08.606 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:41.183376+0000", 2024-09-18T08:08:08.606 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.226e-06 2024-09-18T08:08:08.606 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:08:08.606 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:08:08.606 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-09-18T08:08:08.607 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:41.183403+0000", 2024-09-18T08:08:08.607 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 2.6404e-05 2024-09-18T08:08:08.607 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:08:08.607 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:08:08.607 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-09-18T08:08:08.607 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:41.183417+0000", 2024-09-18T08:08:08.607 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.4399000000000001e-05 2024-09-18T08:08:08.607 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:08:08.607 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:08:08.608 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-09-18T08:08:08.608 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:41.183425+0000", 2024-09-18T08:08:08.608 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 8.1459999999999996e-06 2024-09-18T08:08:08.608 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:08:08.608 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:08:08.608 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-18T08:08:08.608 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:41.183485+0000", 2024-09-18T08:08:08.608 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 5.9673999999999998e-05 2024-09-18T08:08:08.608 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:08:08.608 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:08:08.608 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-09-18T08:08:08.608 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:41.183587+0000", 2024-09-18T08:08:08.608 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000101611 2024-09-18T08:08:08.609 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:08:08.609 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:08:08.609 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "waiting for subops from 0,1", 2024-09-18T08:08:08.609 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:41.183686+0000", 2024-09-18T08:08:08.609 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 9.9823999999999998e-05 2024-09-18T08:08:08.609 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:08:08.609 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:08:08.609 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "op_commit", 2024-09-18T08:08:08.609 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:41.185240+0000", 2024-09-18T08:08:08.609 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0015539340000000001 2024-09-18T08:08:08.609 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:08:08.609 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:08:08.609 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:08:08.610 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:41.191524+0000", 2024-09-18T08:08:08.610 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0062835690000000001 2024-09-18T08:08:08.610 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:08:08.610 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:08:08.610 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:08:08.610 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:41.216746+0000", 2024-09-18T08:08:08.610 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.025222364000000001 2024-09-18T08:08:08.610 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:08:08.610 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:08:08.610 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "commit_sent", 2024-09-18T08:08:08.610 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:41.216782+0000", 2024-09-18T08:08:08.610 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 3.6069999999999999e-05 2024-09-18T08:08:08.610 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:08:08.611 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:08:08.611 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-09-18T08:08:08.611 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:41.216828+0000", 2024-09-18T08:08:08.611 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4.5543000000000001e-05 2024-09-18T08:08:08.611 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-18T08:08:08.611 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-09-18T08:08:08.611 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-18T08:08:08.611 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:08:08.611 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:08:08.611 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.14805.0:63960 7.c 7:30cc2060:::benchmark_data_smithi007_73155_object63959:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e100)", 2024-09-18T08:08:08.611 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-09-18T08:04:41.252452+0000", 2024-09-18T08:08:08.611 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 207.232817119, 2024-09-18T08:08:08.611 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.027634619999999999, 2024-09-18T08:08:08.611 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-09-18T08:08:08.612 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-09-18T08:08:08.612 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T08:08:08.612 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2024-09-18T08:08:08.612 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.14805", 2024-09-18T08:08:08.612 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "172.21.15.7:0/640320436", 2024-09-18T08:08:08.612 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 63960 2024-09-18T08:08:08.612 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:08:08.612 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-09-18T08:08:08.612 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:08:08.612 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-09-18T08:08:08.612 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:41.252452+0000", 2024-09-18T08:08:08.612 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-18T08:08:08.612 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:08:08.613 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:08:08.613 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-09-18T08:08:08.613 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:41.252452+0000", 2024-09-18T08:08:08.613 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-18T08:08:08.613 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:08:08.613 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:08:08.613 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-09-18T08:08:08.613 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:41.252453+0000", 2024-09-18T08:08:08.613 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.063e-06 2024-09-18T08:08:08.613 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:08:08.613 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:08:08.613 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-09-18T08:08:08.613 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:41.252503+0000", 2024-09-18T08:08:08.613 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 5.024e-05 2024-09-18T08:08:08.614 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:08:08.614 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:08:08.614 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-09-18T08:08:08.614 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:41.252513+0000", 2024-09-18T08:08:08.614 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.0137e-05 2024-09-18T08:08:08.614 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:08:08.614 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:08:08.614 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-09-18T08:08:08.614 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:41.252519+0000", 2024-09-18T08:08:08.614 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 5.2630000000000003e-06 2024-09-18T08:08:08.614 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:08:08.614 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:08:08.614 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-18T08:08:08.615 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:41.252564+0000", 2024-09-18T08:08:08.615 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4.5528999999999997e-05 2024-09-18T08:08:08.615 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:08:08.615 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:08:08.615 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-09-18T08:08:08.615 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:41.252645+0000", 2024-09-18T08:08:08.615 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 8.1118000000000005e-05 2024-09-18T08:08:08.615 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:08:08.615 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:08:08.615 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "waiting for subops from 3,7", 2024-09-18T08:08:08.615 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:41.252726+0000", 2024-09-18T08:08:08.616 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 8.0912000000000001e-05 2024-09-18T08:08:08.616 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:08:08.616 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:08:08.616 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "op_commit", 2024-09-18T08:08:08.616 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:41.254200+0000", 2024-09-18T08:08:08.616 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0014738010000000001 2024-09-18T08:08:08.616 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:08:08.616 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:08:08.616 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:08:08.616 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:41.254525+0000", 2024-09-18T08:08:08.616 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00032529599999999998 2024-09-18T08:08:08.616 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:08:08.616 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:08:08.617 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:08:08.617 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:41.280019+0000", 2024-09-18T08:08:08.617 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.025493512999999999 2024-09-18T08:08:08.617 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:08:08.617 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:08:08.617 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "commit_sent", 2024-09-18T08:08:08.617 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:41.280057+0000", 2024-09-18T08:08:08.617 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 3.8287000000000001e-05 2024-09-18T08:08:08.617 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:08:08.617 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:08:08.617 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-09-18T08:08:08.617 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:41.280086+0000", 2024-09-18T08:08:08.617 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 2.9461000000000001e-05 2024-09-18T08:08:08.618 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-18T08:08:08.618 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-09-18T08:08:08.618 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-18T08:08:08.618 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:08:08.618 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:08:08.618 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.14805.0:65051 7.d 7:bcc5331c:::benchmark_data_smithi007_73155_object65050:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e100)", 2024-09-18T08:08:08.618 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-09-18T08:04:42.227551+0000", 2024-09-18T08:08:08.618 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 206.25771835800001, 2024-09-18T08:08:08.618 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.057084449000000002, 2024-09-18T08:08:08.618 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-09-18T08:08:08.618 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-09-18T08:08:08.618 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "waiting for sub ops", 2024-09-18T08:08:08.618 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2024-09-18T08:08:08.619 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.14805", 2024-09-18T08:08:08.619 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "172.21.15.7:0/640320436", 2024-09-18T08:08:08.619 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 65051 2024-09-18T08:08:08.619 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:08:08.619 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-09-18T08:08:08.619 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:08:08.619 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-09-18T08:08:08.619 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:42.227551+0000", 2024-09-18T08:08:08.619 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-18T08:08:08.619 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:08:08.619 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:08:08.619 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-09-18T08:08:08.620 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:42.227551+0000", 2024-09-18T08:08:08.620 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-18T08:08:08.620 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:08:08.620 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:08:08.620 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-09-18T08:08:08.620 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:42.227552+0000", 2024-09-18T08:08:08.620 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.5319999999999999e-06 2024-09-18T08:08:08.620 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:08:08.620 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:08:08.620 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-09-18T08:08:08.620 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:42.227606+0000", 2024-09-18T08:08:08.620 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 5.4290999999999999e-05 2024-09-18T08:08:08.620 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:08:08.621 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:08:08.621 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-09-18T08:08:08.621 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:42.227619+0000", 2024-09-18T08:08:08.621 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.2937e-05 2024-09-18T08:08:08.621 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:08:08.621 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:08:08.621 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-09-18T08:08:08.621 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:42.227627+0000", 2024-09-18T08:08:08.621 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 7.9340000000000006e-06 2024-09-18T08:08:08.621 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:08:08.621 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:08:08.621 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-18T08:08:08.622 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:42.227681+0000", 2024-09-18T08:08:08.622 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 5.3771999999999999e-05 2024-09-18T08:08:08.622 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:08:08.622 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:08:08.622 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-09-18T08:08:08.622 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:42.227809+0000", 2024-09-18T08:08:08.622 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00012750399999999999 2024-09-18T08:08:08.622 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:08:08.622 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:08:08.622 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "waiting for subops from 0,1", 2024-09-18T08:08:08.623 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:42.227961+0000", 2024-09-18T08:08:08.623 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00015280599999999999 2024-09-18T08:08:08.623 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:08:08.623 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:08:08.623 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "op_commit", 2024-09-18T08:08:08.623 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:42.229971+0000", 2024-09-18T08:08:08.623 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0020099179999999999 2024-09-18T08:08:08.623 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:08:08.623 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:08:08.623 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:08:08.623 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:42.232646+0000", 2024-09-18T08:08:08.623 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0026748359999999999 2024-09-18T08:08:08.623 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:08:08.623 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:08:08.624 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-09-18T08:08:08.624 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:42.284635+0000", 2024-09-18T08:08:08.624 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.051988919000000001 2024-09-18T08:08:08.624 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-18T08:08:08.624 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-09-18T08:08:08.624 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-18T08:08:08.624 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:08:08.624 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:08:08.624 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.14805.0:65057 7.d 7:bffbe46c:::benchmark_data_smithi007_73155_object65056:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e100)", 2024-09-18T08:08:08.624 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-09-18T08:04:42.229929+0000", 2024-09-18T08:08:08.624 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 206.25533966200001, 2024-09-18T08:08:08.624 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.054704183000000003, 2024-09-18T08:08:08.624 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-09-18T08:08:08.625 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-09-18T08:08:08.625 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "waiting for sub ops", 2024-09-18T08:08:08.625 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2024-09-18T08:08:08.625 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.14805", 2024-09-18T08:08:08.625 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "172.21.15.7:0/640320436", 2024-09-18T08:08:08.625 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 65057 2024-09-18T08:08:08.625 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:08:08.625 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-09-18T08:08:08.625 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:08:08.625 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-09-18T08:08:08.625 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:42.229929+0000", 2024-09-18T08:08:08.625 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-18T08:08:08.626 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:08:08.626 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:08:08.626 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-09-18T08:08:08.626 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:42.229929+0000", 2024-09-18T08:08:08.626 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-18T08:08:08.626 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:08:08.626 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:08:08.626 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-09-18T08:08:08.626 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:42.229930+0000", 2024-09-18T08:08:08.626 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.0219999999999999e-06 2024-09-18T08:08:08.626 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:08:08.626 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:08:08.627 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-09-18T08:08:08.627 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:42.229973+0000", 2024-09-18T08:08:08.627 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4.2904000000000001e-05 2024-09-18T08:08:08.627 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:08:08.627 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:08:08.627 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-09-18T08:08:08.627 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:42.229987+0000", 2024-09-18T08:08:08.627 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.3405e-05 2024-09-18T08:08:08.627 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:08:08.627 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:08:08.627 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-09-18T08:08:08.628 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:42.229996+0000", 2024-09-18T08:08:08.628 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 9.6299999999999993e-06 2024-09-18T08:08:08.628 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:08:08.628 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:08:08.628 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-18T08:08:08.628 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:42.230048+0000", 2024-09-18T08:08:08.628 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 5.1839000000000003e-05 2024-09-18T08:08:08.628 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:08:08.628 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:08:08.628 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-09-18T08:08:08.628 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:42.230146+0000", 2024-09-18T08:08:08.628 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 9.8101000000000001e-05 2024-09-18T08:08:08.628 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:08:08.629 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:08:08.629 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "waiting for subops from 0,1", 2024-09-18T08:08:08.629 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:42.230237+0000", 2024-09-18T08:08:08.629 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 9.0634e-05 2024-09-18T08:08:08.629 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:08:08.629 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:08:08.629 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "op_commit", 2024-09-18T08:08:08.629 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:42.231752+0000", 2024-09-18T08:08:08.629 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.001515548 2024-09-18T08:08:08.629 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:08:08.629 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:08:08.629 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:08:08.630 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:42.232681+0000", 2024-09-18T08:08:08.630 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00092884 2024-09-18T08:08:08.630 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:08:08.630 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:08:08.630 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-09-18T08:08:08.630 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:42.284634+0000", 2024-09-18T08:08:08.630 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.05195226 2024-09-18T08:08:08.630 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-18T08:08:08.630 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-09-18T08:08:08.630 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-18T08:08:08.630 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:08:08.630 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:08:08.630 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.14805.0:65062 7.d 7:b1e686a2:::benchmark_data_smithi007_73155_object65061:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e100)", 2024-09-18T08:08:08.630 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-09-18T08:04:42.236979+0000", 2024-09-18T08:08:08.631 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 206.24829034699999, 2024-09-18T08:08:08.631 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.047653305999999999, 2024-09-18T08:08:08.631 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-09-18T08:08:08.631 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-09-18T08:08:08.631 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "waiting for sub ops", 2024-09-18T08:08:08.631 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2024-09-18T08:08:08.631 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.14805", 2024-09-18T08:08:08.631 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "172.21.15.7:0/640320436", 2024-09-18T08:08:08.631 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 65062 2024-09-18T08:08:08.631 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:08:08.631 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-09-18T08:08:08.631 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:08:08.631 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-09-18T08:08:08.631 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:42.236979+0000", 2024-09-18T08:08:08.632 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-18T08:08:08.632 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:08:08.632 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:08:08.632 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-09-18T08:08:08.632 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:42.236979+0000", 2024-09-18T08:08:08.632 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-18T08:08:08.632 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:08:08.632 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:08:08.632 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-09-18T08:08:08.632 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:42.236980+0000", 2024-09-18T08:08:08.632 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.124e-06 2024-09-18T08:08:08.632 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:08:08.632 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:08:08.633 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-09-18T08:08:08.633 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:42.237029+0000", 2024-09-18T08:08:08.633 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4.9450999999999998e-05 2024-09-18T08:08:08.633 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:08:08.633 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:08:08.633 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-09-18T08:08:08.633 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:42.237041+0000", 2024-09-18T08:08:08.633 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.1418e-05 2024-09-18T08:08:08.633 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:08:08.633 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:08:08.633 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-09-18T08:08:08.633 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:42.237049+0000", 2024-09-18T08:08:08.633 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 8.0059999999999992e-06 2024-09-18T08:08:08.634 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:08:08.634 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:08:08.634 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-18T08:08:08.634 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:42.237104+0000", 2024-09-18T08:08:08.634 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 5.4870999999999997e-05 2024-09-18T08:08:08.634 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:08:08.634 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:08:08.634 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-09-18T08:08:08.634 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:42.237258+0000", 2024-09-18T08:08:08.634 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000154857 2024-09-18T08:08:08.635 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:08:08.635 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:08:08.635 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "waiting for subops from 0,1", 2024-09-18T08:08:08.635 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:42.237413+0000", 2024-09-18T08:08:08.635 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000154764 2024-09-18T08:08:08.635 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:08:08.636 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:08:08.636 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "op_commit", 2024-09-18T08:08:08.636 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:42.240054+0000", 2024-09-18T08:08:08.636 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0026404549999999999 2024-09-18T08:08:08.636 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:08:08.636 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:08:08.637 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-09-18T08:08:08.637 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:42.284632+0000", 2024-09-18T08:08:08.637 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.044578359999999997 2024-09-18T08:08:08.637 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-18T08:08:08.637 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-09-18T08:08:08.637 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-18T08:08:08.638 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:08:08.638 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:08:08.638 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.14805.0:65063 7.d 7:b037f784:::benchmark_data_smithi007_73155_object65062:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e100)", 2024-09-18T08:08:08.638 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-09-18T08:04:42.237116+0000", 2024-09-18T08:08:08.638 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 206.24815288100001, 2024-09-18T08:08:08.638 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.047512190000000003, 2024-09-18T08:08:08.639 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-09-18T08:08:08.639 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-09-18T08:08:08.639 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "waiting for sub ops", 2024-09-18T08:08:08.639 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2024-09-18T08:08:08.639 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.14805", 2024-09-18T08:08:08.639 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "172.21.15.7:0/640320436", 2024-09-18T08:08:08.639 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 65063 2024-09-18T08:08:08.639 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:08:08.639 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-09-18T08:08:08.640 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:08:08.640 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-09-18T08:08:08.640 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:42.237116+0000", 2024-09-18T08:08:08.640 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-18T08:08:08.640 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:08:08.640 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:08:08.640 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-09-18T08:08:08.641 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:42.237116+0000", 2024-09-18T08:08:08.641 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-18T08:08:08.641 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:08:08.641 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:08:08.641 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-09-18T08:08:08.641 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:42.237117+0000", 2024-09-18T08:08:08.642 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 6.7000000000000004e-07 2024-09-18T08:08:08.642 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:08:08.642 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:08:08.642 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-09-18T08:08:08.642 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:42.237190+0000", 2024-09-18T08:08:08.642 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 7.2972000000000005e-05 2024-09-18T08:08:08.642 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:08:08.642 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:08:08.642 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-09-18T08:08:08.642 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:42.237197+0000", 2024-09-18T08:08:08.642 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 7.2729999999999997e-06 2024-09-18T08:08:08.643 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:08:08.643 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:08:08.643 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-09-18T08:08:08.643 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:42.237203+0000", 2024-09-18T08:08:08.643 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 5.5289999999999999e-06 2024-09-18T08:08:08.643 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:08:08.643 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:08:08.643 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-18T08:08:08.643 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:42.238960+0000", 2024-09-18T08:08:08.643 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.001757759 2024-09-18T08:08:08.644 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:08:08.644 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:08:08.644 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-09-18T08:08:08.644 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:42.239038+0000", 2024-09-18T08:08:08.644 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 7.8054999999999997e-05 2024-09-18T08:08:08.644 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:08:08.644 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:08:08.644 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "waiting for subops from 0,1", 2024-09-18T08:08:08.644 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:42.239122+0000", 2024-09-18T08:08:08.644 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 8.3963999999999998e-05 2024-09-18T08:08:08.644 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:08:08.645 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:08:08.645 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "op_commit", 2024-09-18T08:08:08.645 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:42.240157+0000", 2024-09-18T08:08:08.645 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.001034372 2024-09-18T08:08:08.645 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:08:08.645 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:08:08.645 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-09-18T08:08:08.645 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:42.284628+0000", 2024-09-18T08:08:08.645 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.044471596000000002 2024-09-18T08:08:08.645 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-18T08:08:08.646 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-09-18T08:08:08.646 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-18T08:08:08.646 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-18T08:08:08.646 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-09-18T08:08:08.646 INFO:teuthology.orchestra.run.smithi050.stdout:} 2024-09-18T08:08:08.655 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:08:08 smithi050 ceph-mon[32371]: 3.1f scrub starts 2024-09-18T08:08:08.656 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:08:08 smithi050 ceph-mon[32371]: 3.1f scrub ok 2024-09-18T08:08:08.656 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:08:08 smithi050 ceph-mon[32371]: pgmap v249: 158 pgs: 158 active+clean; 749 MiB data, 2.8 GiB used, 623 GiB / 626 GiB avail; 523 B/s rd, 92 MiB/s wr, 1.47k op/s; 1.7 MiB/s, 33 objects/s recovering 2024-09-18T08:08:08.656 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:08:08 smithi050 ceph-mon[32371]: from='client.? 172.21.15.7:0/412935843' entity='client.admin' cmd={"prefix": "osd reweight", "id": 2, "weight": 0.6778833996177698} : dispatch 2024-09-18T08:08:08.656 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:08:08 smithi050 ceph-mon[32371]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 2, "weight": 0.6778833996177698} : dispatch 2024-09-18T08:08:08.667 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:08:08 smithi007 ceph-mon[29459]: 3.1f scrub starts 2024-09-18T08:08:08.668 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:08:08 smithi007 ceph-mon[29459]: 3.1f scrub ok 2024-09-18T08:08:08.668 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:08:08 smithi007 ceph-mon[29459]: pgmap v249: 158 pgs: 158 active+clean; 749 MiB data, 2.8 GiB used, 623 GiB / 626 GiB avail; 523 B/s rd, 92 MiB/s wr, 1.47k op/s; 1.7 MiB/s, 33 objects/s recovering 2024-09-18T08:08:08.668 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:08:08 smithi007 ceph-mon[29459]: from='client.? 172.21.15.7:0/412935843' entity='client.admin' cmd={"prefix": "osd reweight", "id": 2, "weight": 0.6778833996177698} : dispatch 2024-09-18T08:08:08.668 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:08:08 smithi007 ceph-mon[29459]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 2, "weight": 0.6778833996177698} : dispatch 2024-09-18T08:08:08.668 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:08:08 smithi007 ceph-mon[39249]: 3.1f scrub starts 2024-09-18T08:08:08.668 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:08:08 smithi007 ceph-mon[39249]: 3.1f scrub ok 2024-09-18T08:08:08.668 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:08:08 smithi007 ceph-mon[39249]: pgmap v249: 158 pgs: 158 active+clean; 749 MiB data, 2.8 GiB used, 623 GiB / 626 GiB avail; 523 B/s rd, 92 MiB/s wr, 1.47k op/s; 1.7 MiB/s, 33 objects/s recovering 2024-09-18T08:08:08.668 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:08:08 smithi007 ceph-mon[39249]: from='client.? 172.21.15.7:0/412935843' entity='client.admin' cmd={"prefix": "osd reweight", "id": 2, "weight": 0.6778833996177698} : dispatch 2024-09-18T08:08:08.668 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:08:08 smithi007 ceph-mon[39249]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 2, "weight": 0.6778833996177698} : dispatch 2024-09-18T08:08:08.679 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4 2024-09-18T08:08:08.701 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:08:08.702 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:08:08.702 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4' 2024-09-18T08:08:08.803 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0 2024-09-18T08:08:08.825 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:08:08.826 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:08:08.827 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0' 2024-09-18T08:08:08.928 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5 2024-09-18T08:08:08.950 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:08:08.951 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:08:08.951 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5' 2024-09-18T08:08:09.052 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3 2024-09-18T08:08:09.074 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:08:09.075 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:08:09.076 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3' 2024-09-18T08:08:09.154 INFO:tasks.radosbench.radosbench.0.smithi007.stdout: 11 16 22694 22678 128.836 42.4375 0.00174156 0.00727906 2024-09-18T08:08:09.177 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2 2024-09-18T08:08:09.200 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:08:09.200 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:08:09.201 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2' 2024-09-18T08:08:09.302 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7 2024-09-18T08:08:09.325 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:08:09.325 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:08:09.326 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7' 2024-09-18T08:08:09.372 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/779349ca-7593-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T08:08:09.406 INFO:journalctl@ceph.iscsi.iscsi.a.smithi050.stdout:Sep 18 08:08:08 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-iscsi-iscsi-a[57782]: debug there is no tcmu-runner data available 2024-09-18T08:08:09.426 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0 2024-09-18T08:08:09.449 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:08:09.450 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:08:09.450 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0' 2024-09-18T08:08:09.551 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1 2024-09-18T08:08:09.567 DEBUG:teuthology.orchestra.run.smithi050:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid 779349ca-7593-11ef-bceb-c7b262605968 -- ceph daemon osd.7 dump_ops_in_flight 2024-09-18T08:08:09.573 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:08:09.573 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:08:09.574 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1' 2024-09-18T08:08:09.675 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3 2024-09-18T08:08:09.696 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:08:09.697 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:08:09.698 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3' 2024-09-18T08:08:09.730 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:08:09 smithi007 ceph-mon[29459]: 5.1f scrub starts 2024-09-18T08:08:09.731 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:08:09 smithi007 ceph-mon[29459]: 5.1f scrub ok 2024-09-18T08:08:09.731 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:08:09 smithi007 ceph-mon[29459]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 2, "weight": 0.6778833996177698}]': finished 2024-09-18T08:08:09.731 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:08:09 smithi007 ceph-mon[29459]: osdmap e175: 8 total, 8 up, 7 in 2024-09-18T08:08:09.731 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:08:09 smithi007 ceph-mon[29459]: 4.14 scrub starts 2024-09-18T08:08:09.731 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:08:09 smithi007 ceph-mon[29459]: 4.14 scrub ok 2024-09-18T08:08:09.732 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:08:09 smithi007 ceph-mon[29459]: 4.1 scrub starts 2024-09-18T08:08:09.732 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:08:09 smithi007 ceph-mon[29459]: 4.1 scrub ok 2024-09-18T08:08:09.732 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:08:09 smithi007 ceph-mon[29459]: osdmap e176: 8 total, 8 up, 7 in 2024-09-18T08:08:09.732 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:08:09 smithi007 ceph-mon[39249]: 5.1f scrub starts 2024-09-18T08:08:09.732 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:08:09 smithi007 ceph-mon[39249]: 5.1f scrub ok 2024-09-18T08:08:09.732 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:08:09 smithi007 ceph-mon[39249]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 2, "weight": 0.6778833996177698}]': finished 2024-09-18T08:08:09.733 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:08:09 smithi007 ceph-mon[39249]: osdmap e175: 8 total, 8 up, 7 in 2024-09-18T08:08:09.733 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:08:09 smithi007 ceph-mon[39249]: 4.14 scrub starts 2024-09-18T08:08:09.733 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:08:09 smithi007 ceph-mon[39249]: 4.14 scrub ok 2024-09-18T08:08:09.733 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:08:09 smithi007 ceph-mon[39249]: 4.1 scrub starts 2024-09-18T08:08:09.733 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:08:09 smithi007 ceph-mon[39249]: 4.1 scrub ok 2024-09-18T08:08:09.733 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:08:09 smithi007 ceph-mon[39249]: osdmap e176: 8 total, 8 up, 7 in 2024-09-18T08:08:09.799 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0 2024-09-18T08:08:09.821 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:08:09.821 INFO:teuthology.orchestra.run.smithi050.stderr:Inferring config /var/lib/ceph/779349ca-7593-11ef-bceb-c7b262605968/mon.b/config 2024-09-18T08:08:09.822 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:08:09.822 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0' 2024-09-18T08:08:09.906 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:08:09 smithi050 ceph-mon[32371]: 5.1f scrub starts 2024-09-18T08:08:09.906 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:08:09 smithi050 ceph-mon[32371]: 5.1f scrub ok 2024-09-18T08:08:09.906 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:08:09 smithi050 ceph-mon[32371]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 2, "weight": 0.6778833996177698}]': finished 2024-09-18T08:08:09.906 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:08:09 smithi050 ceph-mon[32371]: osdmap e175: 8 total, 8 up, 7 in 2024-09-18T08:08:09.906 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:08:09 smithi050 ceph-mon[32371]: 4.14 scrub starts 2024-09-18T08:08:09.906 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:08:09 smithi050 ceph-mon[32371]: 4.14 scrub ok 2024-09-18T08:08:09.906 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:08:09 smithi050 ceph-mon[32371]: 4.1 scrub starts 2024-09-18T08:08:09.906 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:08:09 smithi050 ceph-mon[32371]: 4.1 scrub ok 2024-09-18T08:08:09.906 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:08:09 smithi050 ceph-mon[32371]: osdmap e176: 8 total, 8 up, 7 in 2024-09-18T08:08:09.923 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1 2024-09-18T08:08:09.945 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:08:09.946 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:08:09.946 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1' 2024-09-18T08:08:10.047 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6 2024-09-18T08:08:10.071 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:08:10.072 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:08:10.072 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6' 2024-09-18T08:08:10.156 INFO:tasks.radosbench.radosbench.0.smithi007.stdout: 12 16 22694 22678 118.1 0 - 0.00727906 2024-09-18T08:08:10.173 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5 2024-09-18T08:08:10.196 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:08:10.197 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:08:10.197 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5' 2024-09-18T08:08:10.298 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4 2024-09-18T08:08:10.330 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2024-09-18T08:08:10.330 INFO:teuthology.orchestra.run.smithi050.stdout: "ops": [ 2024-09-18T08:08:10.330 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:08:10.330 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "MOSDPGPush(8.b 177/176 [PushOp(8:d6f4f19d:::benchmark_data_smithi007_101583_object5:head, version: 170'2, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(8:d6f4f19d:::benchmark_data_smithi007_101583_object5:head@170'2, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-09-18T08:08:10.330 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-09-18T08:08:10.326121+0000", 2024-09-18T08:08:10.330 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 0.0013666679999999999, 2024-09-18T08:08:10.330 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.001423267, 2024-09-18T08:08:10.330 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-09-18T08:08:10.330 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-09-18T08:08:10.331 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "queued for pg", 2024-09-18T08:08:10.331 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-09-18T08:08:10.331 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:08:10.331 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-09-18T08:08:10.331 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:08:10.326121+0000", 2024-09-18T08:08:10.331 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-18T08:08:10.331 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:08:10.331 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:08:10.331 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-09-18T08:08:10.331 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:08:10.326121+0000", 2024-09-18T08:08:10.331 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-18T08:08:10.332 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:08:10.332 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:08:10.332 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-09-18T08:08:10.332 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:08:10.326121+0000", 2024-09-18T08:08:10.332 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 3.4700000000000002e-07 2024-09-18T08:08:10.332 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:08:10.332 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:08:10.332 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-09-18T08:08:10.332 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:08:10.326160+0000", 2024-09-18T08:08:10.332 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 3.8940999999999999e-05 2024-09-18T08:08:10.332 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:08:10.332 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:08:10.333 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-09-18T08:08:10.333 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:08:10.326177+0000", 2024-09-18T08:08:10.333 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.6701000000000002e-05 2024-09-18T08:08:10.333 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:08:10.333 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:08:10.333 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-09-18T08:08:10.333 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:08:10.326192+0000", 2024-09-18T08:08:10.333 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.465e-05 2024-09-18T08:08:10.333 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-18T08:08:10.333 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-09-18T08:08:10.333 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-18T08:08:10.333 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:08:10.333 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:08:10.333 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_repop(client.15216.0:22848 8.b e177/176 8:ddeba846:::benchmark_data_smithi007_101583_object22847:head v 177'1420, mlcod=170'1)", 2024-09-18T08:08:10.333 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-09-18T08:08:10.326017+0000", 2024-09-18T08:08:10.334 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 0.0014702879999999999, 2024-09-18T08:08:10.334 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.001574478, 2024-09-18T08:08:10.334 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-09-18T08:08:10.334 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-09-18T08:08:10.334 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "started", 2024-09-18T08:08:10.334 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-09-18T08:08:10.334 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:08:10.334 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-09-18T08:08:10.334 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:08:10.326017+0000", 2024-09-18T08:08:10.334 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-18T08:08:10.334 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:08:10.334 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:08:10.334 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-09-18T08:08:10.335 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:08:10.326017+0000", 2024-09-18T08:08:10.335 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-18T08:08:10.335 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:08:10.335 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:08:10.335 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-09-18T08:08:10.335 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:08:10.326018+0000", 2024-09-18T08:08:10.335 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 8.2099999999999995e-07 2024-09-18T08:08:10.335 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:08:10.335 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:08:10.335 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-09-18T08:08:10.335 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:08:10.326083+0000", 2024-09-18T08:08:10.335 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 6.4591000000000006e-05 2024-09-18T08:08:10.335 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:08:10.335 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:08:10.336 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-09-18T08:08:10.336 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:08:10.326097+0000", 2024-09-18T08:08:10.336 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.3976e-05 2024-09-18T08:08:10.336 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:08:10.336 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:08:10.336 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-09-18T08:08:10.336 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:08:10.326104+0000", 2024-09-18T08:08:10.336 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 7.0010000000000004e-06 2024-09-18T08:08:10.336 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:08:10.336 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:08:10.336 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-18T08:08:10.336 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:08:10.326187+0000", 2024-09-18T08:08:10.337 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 8.3269999999999994e-05 2024-09-18T08:08:10.337 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:08:10.337 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:08:10.337 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-09-18T08:08:10.337 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:08:10.326212+0000", 2024-09-18T08:08:10.337 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 2.5068000000000001e-05 2024-09-18T08:08:10.337 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-18T08:08:10.337 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-09-18T08:08:10.338 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-18T08:08:10.338 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-18T08:08:10.338 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2024-09-18T08:08:10.338 INFO:teuthology.orchestra.run.smithi050.stdout: "num_ops": 2 2024-09-18T08:08:10.338 INFO:teuthology.orchestra.run.smithi050.stdout:} 2024-09-18T08:08:10.338 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:08:10.339 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:08:10.339 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4' 2024-09-18T08:08:10.343 INFO:teuthology.orchestra.run.smithi007.stdout:osd.0: {} 2024-09-18T08:08:10.343 INFO:teuthology.orchestra.run.smithi007.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-18T08:08:10.380 INFO:teuthology.orchestra.run.smithi007.stdout:osd.1: {} 2024-09-18T08:08:10.380 INFO:teuthology.orchestra.run.smithi007.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-18T08:08:10.417 INFO:teuthology.orchestra.run.smithi007.stdout:osd.2: {} 2024-09-18T08:08:10.418 INFO:teuthology.orchestra.run.smithi007.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-18T08:08:10.441 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6 2024-09-18T08:08:10.456 INFO:teuthology.orchestra.run.smithi007.stdout:osd.3: {} 2024-09-18T08:08:10.456 INFO:teuthology.orchestra.run.smithi007.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-18T08:08:10.472 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:08:10.472 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:08:10.473 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6' 2024-09-18T08:08:10.491 INFO:teuthology.orchestra.run.smithi007.stdout:osd.4: {} 2024-09-18T08:08:10.491 INFO:teuthology.orchestra.run.smithi007.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-18T08:08:10.524 INFO:teuthology.orchestra.run.smithi007.stdout:osd.5: {} 2024-09-18T08:08:10.525 INFO:teuthology.orchestra.run.smithi007.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-18T08:08:10.561 INFO:teuthology.orchestra.run.smithi007.stdout:osd.6: {} 2024-09-18T08:08:10.561 INFO:teuthology.orchestra.run.smithi007.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-18T08:08:10.574 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1 2024-09-18T08:08:10.596 INFO:teuthology.orchestra.run.smithi007.stdout:osd.7: {} 2024-09-18T08:08:10.596 INFO:teuthology.orchestra.run.smithi007.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-18T08:08:10.614 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:08:10.615 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:08:10.615 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1' 2024-09-18T08:08:10.656 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:08:10 smithi050 ceph-mon[32371]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-18T08:08:10.656 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:08:10 smithi050 ceph-mon[32371]: 5.18 scrub starts 2024-09-18T08:08:10.656 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:08:10 smithi050 ceph-mon[32371]: 5.18 scrub ok 2024-09-18T08:08:10.656 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:08:10 smithi050 ceph-mon[32371]: pgmap v252: 158 pgs: 13 peering, 145 active+clean; 1.4 GiB data, 4.9 GiB used, 621 GiB / 626 GiB avail; 1.7 KiB/s rd, 200 MiB/s wr, 3.20k op/s; 1.9 MiB/s, 39 objects/s recovering 2024-09-18T08:08:10.656 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:08:10 smithi050 ceph-mon[32371]: 1.2 scrub starts 2024-09-18T08:08:10.657 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:08:10 smithi050 ceph-mon[32371]: 1.2 scrub ok 2024-09-18T08:08:10.657 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:08:10 smithi050 ceph-mon[32371]: 1.5 scrub starts 2024-09-18T08:08:10.657 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:08:10 smithi050 ceph-mon[32371]: 1.5 scrub ok 2024-09-18T08:08:10.657 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:08:10 smithi050 ceph-mon[32371]: 4.d scrub starts 2024-09-18T08:08:10.657 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:08:10 smithi050 ceph-mon[32371]: 4.d scrub ok 2024-09-18T08:08:10.657 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:08:10 smithi050 ceph-mon[32371]: Health check failed: Reduced data availability: 1 pg inactive, 1 pg peering (PG_AVAILABILITY) 2024-09-18T08:08:10.657 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:08:10 smithi050 ceph-mon[32371]: osdmap e177: 8 total, 8 up, 7 in 2024-09-18T08:08:10.657 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:08:10 smithi050 ceph-mon[32371]: from='client.? 172.21.15.7:0/2749465426' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-18T08:08:10.697 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid 779349ca-7593-11ef-bceb-c7b262605968 -- ceph osd unset nodeep-scrub 2024-09-18T08:08:10.716 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7 2024-09-18T08:08:10.731 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:08:10 smithi007 ceph-mon[29459]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-18T08:08:10.731 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:08:10 smithi007 ceph-mon[29459]: 5.18 scrub starts 2024-09-18T08:08:10.731 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:08:10 smithi007 ceph-mon[29459]: 5.18 scrub ok 2024-09-18T08:08:10.731 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:08:10 smithi007 ceph-mon[29459]: pgmap v252: 158 pgs: 13 peering, 145 active+clean; 1.4 GiB data, 4.9 GiB used, 621 GiB / 626 GiB avail; 1.7 KiB/s rd, 200 MiB/s wr, 3.20k op/s; 1.9 MiB/s, 39 objects/s recovering 2024-09-18T08:08:10.731 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:08:10 smithi007 ceph-mon[29459]: 1.2 scrub starts 2024-09-18T08:08:10.732 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:08:10 smithi007 ceph-mon[29459]: 1.2 scrub ok 2024-09-18T08:08:10.732 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:08:10 smithi007 ceph-mon[29459]: 1.5 scrub starts 2024-09-18T08:08:10.732 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:08:10 smithi007 ceph-mon[29459]: 1.5 scrub ok 2024-09-18T08:08:10.732 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:08:10 smithi007 ceph-mon[29459]: 4.d scrub starts 2024-09-18T08:08:10.732 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:08:10 smithi007 ceph-mon[29459]: 4.d scrub ok 2024-09-18T08:08:10.732 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:08:10 smithi007 ceph-mon[29459]: Health check failed: Reduced data availability: 1 pg inactive, 1 pg peering (PG_AVAILABILITY) 2024-09-18T08:08:10.732 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:08:10 smithi007 ceph-mon[29459]: osdmap e177: 8 total, 8 up, 7 in 2024-09-18T08:08:10.733 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:08:10 smithi007 ceph-mon[29459]: from='client.? 172.21.15.7:0/2749465426' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-18T08:08:10.733 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:08:10 smithi007 ceph-mon[39249]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-18T08:08:10.733 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:08:10 smithi007 ceph-mon[39249]: 5.18 scrub starts 2024-09-18T08:08:10.733 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:08:10 smithi007 ceph-mon[39249]: 5.18 scrub ok 2024-09-18T08:08:10.733 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:08:10 smithi007 ceph-mon[39249]: pgmap v252: 158 pgs: 13 peering, 145 active+clean; 1.4 GiB data, 4.9 GiB used, 621 GiB / 626 GiB avail; 1.7 KiB/s rd, 200 MiB/s wr, 3.20k op/s; 1.9 MiB/s, 39 objects/s recovering 2024-09-18T08:08:10.734 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:08:10 smithi007 ceph-mon[39249]: 1.2 scrub starts 2024-09-18T08:08:10.734 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:08:10 smithi007 ceph-mon[39249]: 1.2 scrub ok 2024-09-18T08:08:10.734 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:08:10 smithi007 ceph-mon[39249]: 1.5 scrub starts 2024-09-18T08:08:10.734 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:08:10 smithi007 ceph-mon[39249]: 1.5 scrub ok 2024-09-18T08:08:10.734 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:08:10 smithi007 ceph-mon[39249]: 4.d scrub starts 2024-09-18T08:08:10.734 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:08:10 smithi007 ceph-mon[39249]: 4.d scrub ok 2024-09-18T08:08:10.735 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:08:10 smithi007 ceph-mon[39249]: Health check failed: Reduced data availability: 1 pg inactive, 1 pg peering (PG_AVAILABILITY) 2024-09-18T08:08:10.735 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:08:10 smithi007 ceph-mon[39249]: osdmap e177: 8 total, 8 up, 7 in 2024-09-18T08:08:10.735 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:08:10 smithi007 ceph-mon[39249]: from='client.? 172.21.15.7:0/2749465426' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-18T08:08:10.746 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:08:10.747 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:08:10.748 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7' 2024-09-18T08:08:10.849 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7 2024-09-18T08:08:10.878 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:08:10.879 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:08:10.879 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7' 2024-09-18T08:08:10.980 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1 2024-09-18T08:08:11.012 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:08:11.013 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:08:11.013 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1' 2024-09-18T08:08:11.114 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4 2024-09-18T08:08:11.152 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:08:11.152 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:08:11.153 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4' 2024-09-18T08:08:11.154 INFO:tasks.radosbench.radosbench.0.smithi007.stdout: 13 16 24219 24203 116.346 47.6562 0.00985833 0.00858717 2024-09-18T08:08:11.188 DEBUG:teuthology.orchestra.run.smithi050:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid 779349ca-7593-11ef-bceb-c7b262605968 -- ceph daemon osd.7 dump_blocked_ops 2024-09-18T08:08:11.254 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2 2024-09-18T08:08:11.294 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:08:11.294 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:08:11.294 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2' 2024-09-18T08:08:11.395 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0 2024-09-18T08:08:11.435 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:08:11.436 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:08:11.437 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0' 2024-09-18T08:08:11.538 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5 2024-09-18T08:08:11.576 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:08:11.577 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:08:11.577 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5' 2024-09-18T08:08:11.643 INFO:teuthology.orchestra.run.smithi050.stderr:Inferring config /var/lib/ceph/779349ca-7593-11ef-bceb-c7b262605968/mon.b/config 2024-09-18T08:08:11.678 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2 2024-09-18T08:08:11.717 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:08:11.718 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:08:11.718 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2' 2024-09-18T08:08:11.730 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:08:11 smithi007 ceph-mon[29459]: 1.9 scrub starts 2024-09-18T08:08:11.730 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:08:11 smithi007 ceph-mon[29459]: 1.9 scrub ok 2024-09-18T08:08:11.730 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:08:11 smithi007 ceph-mon[29459]: 6.14 scrub starts 2024-09-18T08:08:11.730 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:08:11 smithi007 ceph-mon[29459]: 6.14 scrub ok 2024-09-18T08:08:11.731 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:08:11 smithi007 ceph-mon[39249]: 1.9 scrub starts 2024-09-18T08:08:11.731 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:08:11 smithi007 ceph-mon[39249]: 1.9 scrub ok 2024-09-18T08:08:11.731 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:08:11 smithi007 ceph-mon[39249]: 6.14 scrub starts 2024-09-18T08:08:11.731 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:08:11 smithi007 ceph-mon[39249]: 6.14 scrub ok 2024-09-18T08:08:11.774 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:08:11 smithi050 ceph-mon[32371]: 1.9 scrub starts 2024-09-18T08:08:11.775 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:08:11 smithi050 ceph-mon[32371]: 1.9 scrub ok 2024-09-18T08:08:11.775 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:08:11 smithi050 ceph-mon[32371]: 6.14 scrub starts 2024-09-18T08:08:11.775 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:08:11 smithi050 ceph-mon[32371]: 6.14 scrub ok 2024-09-18T08:08:11.819 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1 2024-09-18T08:08:11.858 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:08:11.859 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:08:11.859 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1' 2024-09-18T08:08:11.960 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3 2024-09-18T08:08:11.987 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:08:11.987 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:08:11.988 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3' 2024-09-18T08:08:12.089 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2 2024-09-18T08:08:12.119 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:08:12.120 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:08:12.120 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2' 2024-09-18T08:08:12.154 INFO:tasks.radosbench.radosbench.0.smithi007.stdout: 14 16 26010 25994 116.031 111.938 0.00582754 0.00858403 2024-09-18T08:08:12.221 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6 2024-09-18T08:08:12.255 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:08:12.256 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:08:12.256 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6' 2024-09-18T08:08:12.357 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2 2024-09-18T08:08:12.400 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:08:12.400 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:08:12.401 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2' 2024-09-18T08:08:12.502 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2 2024-09-18T08:08:12.542 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:08:12.543 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:08:12.543 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2' 2024-09-18T08:08:12.644 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2 2024-09-18T08:08:12.682 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:08:12.683 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:08:12.683 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2' 2024-09-18T08:08:12.730 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:08:12 smithi007 ceph-mon[29459]: pgmap v254: 158 pgs: 13 peering, 145 active+clean; 1.4 GiB data, 4.9 GiB used, 621 GiB / 626 GiB avail; 1.1 KiB/s rd, 137 MiB/s wr, 2.20k op/s 2024-09-18T08:08:12.731 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:08:12 smithi007 ceph-mon[39249]: pgmap v254: 158 pgs: 13 peering, 145 active+clean; 1.4 GiB data, 4.9 GiB used, 621 GiB / 626 GiB avail; 1.1 KiB/s rd, 137 MiB/s wr, 2.20k op/s 2024-09-18T08:08:12.769 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:08:12 smithi050 ceph-mon[32371]: pgmap v254: 158 pgs: 13 peering, 145 active+clean; 1.4 GiB data, 4.9 GiB used, 621 GiB / 626 GiB avail; 1.1 KiB/s rd, 137 MiB/s wr, 2.20k op/s 2024-09-18T08:08:12.784 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2 2024-09-18T08:08:12.814 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:08:12.814 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:08:12.815 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2' 2024-09-18T08:08:12.916 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1 2024-09-18T08:08:12.944 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:08:12.944 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:08:12.945 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1' 2024-09-18T08:08:13.046 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4 2024-09-18T08:08:13.085 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:08:13.086 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:08:13.087 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4' 2024-09-18T08:08:13.154 INFO:tasks.radosbench.radosbench.0.smithi007.stdout: 15 16 27511 27495 114.549 93.8125 0.0139259 0.00872351 2024-09-18T08:08:13.188 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1 2024-09-18T08:08:13.229 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:08:13.230 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:08:13.230 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1' 2024-09-18T08:08:13.307 INFO:teuthology.orchestra.run.smithi050.stdout:ERROR: (22) Invalid argument 2024-09-18T08:08:13.307 INFO:teuthology.orchestra.run.smithi050.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-18T08:08:13.331 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7 2024-09-18T08:08:13.363 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:08:13.364 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:08:13.364 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7' 2024-09-18T08:08:13.465 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4 2024-09-18T08:08:13.478 INFO:journalctl@ceph.rgw.foo.a.smithi007.stdout:Sep 18 08:08:13 smithi007 ceph-779349ca-7593-11ef-bceb-c7b262605968-rgw-foo-a[63271]: 2024-09-18T08:08:12.994+0000 7fdec12d6640 -1 rgw watcher librados: RGWWatcher::handle_error cookie 94700888683008 err (110) Connection timed out 2024-09-18T08:08:13.478 INFO:journalctl@ceph.rgw.foo.a.smithi007.stdout:Sep 18 08:08:13 smithi007 ceph-779349ca-7593-11ef-bceb-c7b262605968-rgw-foo-a[63271]: 2024-09-18T08:08:12.995+0000 7fdec1ad7640 -1 rgw watcher librados: RGWWatcher::handle_error cookie 94700888686464 err (110) Connection timed out 2024-09-18T08:08:13.499 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:08:13.499 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:08:13.500 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4' 2024-09-18T08:08:13.600 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2 2024-09-18T08:08:13.645 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:08:13.646 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:08:13.646 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2' 2024-09-18T08:08:13.708 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/779349ca-7593-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T08:08:13.730 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:08:13 smithi007 ceph-mon[29459]: purged_snaps scrub starts 2024-09-18T08:08:13.730 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:08:13 smithi007 ceph-mon[29459]: purged_snaps scrub ok 2024-09-18T08:08:13.730 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:08:13 smithi007 ceph-mon[29459]: osdmap e178: 8 total, 8 up, 7 in 2024-09-18T08:08:13.731 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:08:13 smithi007 ceph-mon[39249]: purged_snaps scrub starts 2024-09-18T08:08:13.731 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:08:13 smithi007 ceph-mon[39249]: purged_snaps scrub ok 2024-09-18T08:08:13.731 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:08:13 smithi007 ceph-mon[39249]: osdmap e178: 8 total, 8 up, 7 in 2024-09-18T08:08:13.747 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3 2024-09-18T08:08:13.792 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:08:13.792 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:08:13.793 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3' 2024-09-18T08:08:13.894 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1 2024-09-18T08:08:13.906 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:08:13 smithi050 ceph-mon[32371]: purged_snaps scrub starts 2024-09-18T08:08:13.906 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:08:13 smithi050 ceph-mon[32371]: purged_snaps scrub ok 2024-09-18T08:08:13.906 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:08:13 smithi050 ceph-mon[32371]: osdmap e178: 8 total, 8 up, 7 in 2024-09-18T08:08:13.933 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:08:13.933 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:08:13.934 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1' 2024-09-18T08:08:14.035 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1 2024-09-18T08:08:14.071 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:08:14.072 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:08:14.072 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1' 2024-09-18T08:08:14.154 INFO:tasks.radosbench.radosbench.0.smithi007.stdout: 16 16 28954 28938 113.026 90.1875 0.0395612 0.00883731 2024-09-18T08:08:14.173 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7 2024-09-18T08:08:14.200 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:08:14.201 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:08:14.201 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7' 2024-09-18T08:08:14.302 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7 2024-09-18T08:08:14.323 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 5, 7, 6] out_osds: [4] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-09-18T08:08:14.323 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-18T08:08:14.323 INFO:tasks.thrashosds.thrasher:Adding osd 4 2024-09-18T08:08:14.324 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid 779349ca-7593-11ef-bceb-c7b262605968 -- ceph osd in 4 2024-09-18T08:08:14.333 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:08:14.334 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:08:14.334 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7' 2024-09-18T08:08:14.435 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1 2024-09-18T08:08:14.492 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:08:14.493 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:08:14.494 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1' 2024-09-18T08:08:14.594 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4 2024-09-18T08:08:14.622 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:08:14.622 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:08:14.623 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4' 2024-09-18T08:08:14.723 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5 2024-09-18T08:08:14.752 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:08:14.753 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:08:14.753 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5' 2024-09-18T08:08:14.804 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:08:14 smithi050 ceph-mon[32371]: 5.1e scrub starts 2024-09-18T08:08:14.805 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:08:14 smithi050 ceph-mon[32371]: 5.1e scrub ok 2024-09-18T08:08:14.805 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:08:14 smithi050 ceph-mon[32371]: pgmap v256: 158 pgs: 1 active+recovering+undersized+remapped, 13 peering, 144 active+clean; 1.5 GiB data, 5.1 GiB used, 621 GiB / 626 GiB avail; 77 MiB/s wr, 1.23k op/s; 1218/72114 objects misplaced (1.689%); 2.8 MiB/s, 44 objects/s recovering 2024-09-18T08:08:14.805 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:08:14 smithi050 ceph-mon[32371]: osdmap e179: 8 total, 8 up, 7 in 2024-09-18T08:08:14.818 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/779349ca-7593-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T08:08:14.852 DEBUG:teuthology.orchestra.run.smithi050:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid 779349ca-7593-11ef-bceb-c7b262605968 -- ceph daemon osd.7 dump_historic_ops 2024-09-18T08:08:14.854 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2 2024-09-18T08:08:14.905 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:08:14.905 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:08:14.906 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2' 2024-09-18T08:08:14.956 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:08:14 smithi007 ceph-mon[29459]: 5.1e scrub starts 2024-09-18T08:08:14.956 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:08:14 smithi007 ceph-mon[29459]: 5.1e scrub ok 2024-09-18T08:08:14.956 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:08:14 smithi007 ceph-mon[29459]: pgmap v256: 158 pgs: 1 active+recovering+undersized+remapped, 13 peering, 144 active+clean; 1.5 GiB data, 5.1 GiB used, 621 GiB / 626 GiB avail; 77 MiB/s wr, 1.23k op/s; 1218/72114 objects misplaced (1.689%); 2.8 MiB/s, 44 objects/s recovering 2024-09-18T08:08:14.956 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:08:14 smithi007 ceph-mon[29459]: osdmap e179: 8 total, 8 up, 7 in 2024-09-18T08:08:14.957 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:08:14 smithi007 ceph-mon[39249]: 5.1e scrub starts 2024-09-18T08:08:14.957 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:08:14 smithi007 ceph-mon[39249]: 5.1e scrub ok 2024-09-18T08:08:14.957 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:08:14 smithi007 ceph-mon[39249]: pgmap v256: 158 pgs: 1 active+recovering+undersized+remapped, 13 peering, 144 active+clean; 1.5 GiB data, 5.1 GiB used, 621 GiB / 626 GiB avail; 77 MiB/s wr, 1.23k op/s; 1218/72114 objects misplaced (1.689%); 2.8 MiB/s, 44 objects/s recovering 2024-09-18T08:08:14.957 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:08:14 smithi007 ceph-mon[39249]: osdmap e179: 8 total, 8 up, 7 in 2024-09-18T08:08:15.007 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7 2024-09-18T08:08:15.041 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:08:15.042 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:08:15.042 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7' 2024-09-18T08:08:15.143 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5 2024-09-18T08:08:15.154 INFO:tasks.radosbench.radosbench.0.smithi007.stdout: 17 16 30501 30485 112.064 96.6875 0.0134021 0.00891737 2024-09-18T08:08:15.182 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:08:15.183 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:08:15.183 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5' 2024-09-18T08:08:15.247 INFO:teuthology.orchestra.run.smithi050.stderr:Inferring config /var/lib/ceph/779349ca-7593-11ef-bceb-c7b262605968/mon.b/config 2024-09-18T08:08:15.284 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6 2024-09-18T08:08:15.318 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:08:15.319 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:08:15.319 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6' 2024-09-18T08:08:15.420 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4 2024-09-18T08:08:15.451 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:08:15.452 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:08:15.452 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4' 2024-09-18T08:08:15.505 INFO:teuthology.orchestra.run.smithi007.stderr:nodeep-scrub is unset 2024-09-18T08:08:15.554 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7 2024-09-18T08:08:15.611 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:08:15.612 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:08:15.612 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7' 2024-09-18T08:08:15.619 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid 779349ca-7593-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-18T08:08:15.656 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:08:15 smithi050 ceph-mon[32371]: from='client.? 172.21.15.7:0/3693188888' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-18T08:08:15.713 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2 2024-09-18T08:08:15.730 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:08:15 smithi007 ceph-mon[29459]: from='client.? 172.21.15.7:0/3693188888' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-18T08:08:15.730 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:08:15 smithi007 ceph-mon[39249]: from='client.? 172.21.15.7:0/3693188888' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-18T08:08:15.745 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:08:15.745 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:08:15.746 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2' 2024-09-18T08:08:15.810 INFO:teuthology.orchestra.run.smithi050.stdout:ERROR: (22) Invalid argument 2024-09-18T08:08:15.810 INFO:teuthology.orchestra.run.smithi050.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-18T08:08:15.847 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4 2024-09-18T08:08:15.874 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:08:15.875 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:08:15.875 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4' 2024-09-18T08:08:15.976 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7 2024-09-18T08:08:16.008 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:08:16.009 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:08:16.009 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7' 2024-09-18T08:08:16.110 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5 2024-09-18T08:08:16.140 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:08:16.141 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:08:16.141 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5' 2024-09-18T08:08:16.154 INFO:tasks.radosbench.radosbench.0.smithi007.stdout: 18 16 31935 31919 110.817 89.625 0.00538898 0.00901821 2024-09-18T08:08:16.242 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6 2024-09-18T08:08:16.274 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:08:16.275 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:08:16.276 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6' 2024-09-18T08:08:16.376 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5 2024-09-18T08:08:16.410 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:08:16.411 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:08:16.411 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5' 2024-09-18T08:08:16.513 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6 2024-09-18T08:08:16.551 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:08:16.552 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:08:16.552 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6' 2024-09-18T08:08:16.568 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid 779349ca-7593-11ef-bceb-c7b262605968 -- ceph daemon osd.0 dump_ops_in_flight 2024-09-18T08:08:16.653 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5 2024-09-18T08:08:16.656 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:08:16 smithi050 ceph-mon[32371]: 6.18 scrub starts 2024-09-18T08:08:16.656 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:08:16 smithi050 ceph-mon[32371]: 6.18 scrub ok 2024-09-18T08:08:16.656 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:08:16 smithi050 ceph-mon[32371]: pgmap v258: 158 pgs: 2 active+recovering+undersized+degraded+remapped, 1 active+recovering+undersized+remapped, 155 active+clean; 1.7 GiB data, 6.0 GiB used, 620 GiB / 626 GiB avail; 57 MiB/s wr, 913 op/s; 2/84381 objects degraded (0.002%); 3554/84381 objects misplaced (4.212%); 7.3 MiB/s, 120 objects/s recovering 2024-09-18T08:08:16.657 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:08:16 smithi050 ceph-mon[32371]: Health check failed: Degraded data redundancy: 2/84381 objects degraded (0.002%), 2 pgs degraded (PG_DEGRADED) 2024-09-18T08:08:16.657 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:08:16 smithi050 ceph-mon[32371]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg inactive, 1 pg peering) 2024-09-18T08:08:16.657 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:08:16 smithi050 ceph-mon[32371]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-18T08:08:16.657 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:08:16 smithi050 ceph-mon[32371]: from='client.? 172.21.15.7:0/3693188888' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-18T08:08:16.657 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:08:16 smithi050 ceph-mon[32371]: osdmap e180: 8 total, 8 up, 7 in 2024-09-18T08:08:16.657 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:08:16 smithi050 ceph-mon[32371]: 5.8 scrub starts 2024-09-18T08:08:16.657 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:08:16 smithi050 ceph-mon[32371]: 5.8 scrub ok 2024-09-18T08:08:16.685 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:08:16.686 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:08:16.686 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5' 2024-09-18T08:08:16.787 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0 2024-09-18T08:08:16.834 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:08:16.835 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:08:16.835 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0' 2024-09-18T08:08:16.936 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6 2024-09-18T08:08:16.969 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:08:16.969 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:08:16.970 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6' 2024-09-18T08:08:16.980 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Sep 18 08:08:16 smithi007 ceph-779349ca-7593-11ef-bceb-c7b262605968-mgr-y[29688]: ::ffff:172.21.15.50 - - [18/Sep/2024:08:08:16] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-09-18T08:08:16.981 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:08:16 smithi007 ceph-mon[29459]: 6.18 scrub starts 2024-09-18T08:08:16.981 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:08:16 smithi007 ceph-mon[29459]: 6.18 scrub ok 2024-09-18T08:08:16.981 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:08:16 smithi007 ceph-mon[29459]: pgmap v258: 158 pgs: 2 active+recovering+undersized+degraded+remapped, 1 active+recovering+undersized+remapped, 155 active+clean; 1.7 GiB data, 6.0 GiB used, 620 GiB / 626 GiB avail; 57 MiB/s wr, 913 op/s; 2/84381 objects degraded (0.002%); 3554/84381 objects misplaced (4.212%); 7.3 MiB/s, 120 objects/s recovering 2024-09-18T08:08:16.981 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:08:16 smithi007 ceph-mon[29459]: Health check failed: Degraded data redundancy: 2/84381 objects degraded (0.002%), 2 pgs degraded (PG_DEGRADED) 2024-09-18T08:08:16.981 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:08:16 smithi007 ceph-mon[29459]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg inactive, 1 pg peering) 2024-09-18T08:08:16.981 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:08:16 smithi007 ceph-mon[29459]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-18T08:08:16.981 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:08:16 smithi007 ceph-mon[29459]: from='client.? 172.21.15.7:0/3693188888' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-18T08:08:16.981 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:08:16 smithi007 ceph-mon[29459]: osdmap e180: 8 total, 8 up, 7 in 2024-09-18T08:08:16.981 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:08:16 smithi007 ceph-mon[29459]: 5.8 scrub starts 2024-09-18T08:08:16.981 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:08:16 smithi007 ceph-mon[29459]: 5.8 scrub ok 2024-09-18T08:08:16.982 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:08:16 smithi007 ceph-mon[39249]: 6.18 scrub starts 2024-09-18T08:08:16.982 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:08:16 smithi007 ceph-mon[39249]: 6.18 scrub ok 2024-09-18T08:08:16.982 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:08:16 smithi007 ceph-mon[39249]: pgmap v258: 158 pgs: 2 active+recovering+undersized+degraded+remapped, 1 active+recovering+undersized+remapped, 155 active+clean; 1.7 GiB data, 6.0 GiB used, 620 GiB / 626 GiB avail; 57 MiB/s wr, 913 op/s; 2/84381 objects degraded (0.002%); 3554/84381 objects misplaced (4.212%); 7.3 MiB/s, 120 objects/s recovering 2024-09-18T08:08:16.982 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:08:16 smithi007 ceph-mon[39249]: Health check failed: Degraded data redundancy: 2/84381 objects degraded (0.002%), 2 pgs degraded (PG_DEGRADED) 2024-09-18T08:08:16.982 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:08:16 smithi007 ceph-mon[39249]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg inactive, 1 pg peering) 2024-09-18T08:08:16.982 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:08:16 smithi007 ceph-mon[39249]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-18T08:08:16.982 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:08:16 smithi007 ceph-mon[39249]: from='client.? 172.21.15.7:0/3693188888' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-18T08:08:16.982 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:08:16 smithi007 ceph-mon[39249]: osdmap e180: 8 total, 8 up, 7 in 2024-09-18T08:08:16.982 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:08:16 smithi007 ceph-mon[39249]: 5.8 scrub starts 2024-09-18T08:08:16.982 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:08:16 smithi007 ceph-mon[39249]: 5.8 scrub ok 2024-09-18T08:08:17.071 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5 2024-09-18T08:08:17.101 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:08:17.102 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:08:17.102 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5' 2024-09-18T08:08:17.154 INFO:tasks.radosbench.radosbench.0.smithi007.stdout: 19 16 33440 33424 109.935 94.0625 0.00590403 0.00909011 2024-09-18T08:08:17.203 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4 2024-09-18T08:08:17.234 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:08:17.235 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:08:17.236 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4' 2024-09-18T08:08:17.338 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3 2024-09-18T08:08:17.383 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:08:17.384 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:08:17.384 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3' 2024-09-18T08:08:17.480 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:08:17 smithi007 ceph-779349ca-7593-11ef-bceb-c7b262605968-mon-a[29455]: 2024-09-18T08:08:17.191+0000 7fcd2b595640 -1 mon.a@0(leader).osd e180 definitely_dead 0 2024-09-18T08:08:17.485 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0 2024-09-18T08:08:17.506 INFO:teuthology.orchestra.run.smithi007.stderr:marked in osd.4. 2024-09-18T08:08:17.535 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:08:17.536 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:08:17.536 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0' 2024-09-18T08:08:17.637 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0 2024-09-18T08:08:17.659 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:08:17.660 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:08:17.660 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0' 2024-09-18T08:08:17.761 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4 2024-09-18T08:08:17.785 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:08:17.785 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:08:17.786 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4' 2024-09-18T08:08:17.886 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1 2024-09-18T08:08:17.906 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:08:17 smithi050 ceph-mon[32371]: from='client.? 172.21.15.7:0/1396190963' entity='client.admin' cmd={"prefix": "osd in", "ids": ["4"]} : dispatch 2024-09-18T08:08:17.906 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:08:17 smithi050 ceph-mon[32371]: from='client.? ' entity='client.admin' cmd={"prefix": "osd in", "ids": ["4"]} : dispatch 2024-09-18T08:08:17.910 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:08:17.911 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:08:17.911 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1' 2024-09-18T08:08:17.980 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:08:17 smithi007 ceph-mon[29459]: from='client.? 172.21.15.7:0/1396190963' entity='client.admin' cmd={"prefix": "osd in", "ids": ["4"]} : dispatch 2024-09-18T08:08:17.980 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:08:17 smithi007 ceph-mon[29459]: from='client.? ' entity='client.admin' cmd={"prefix": "osd in", "ids": ["4"]} : dispatch 2024-09-18T08:08:17.981 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:08:17 smithi007 ceph-mon[39249]: from='client.? 172.21.15.7:0/1396190963' entity='client.admin' cmd={"prefix": "osd in", "ids": ["4"]} : dispatch 2024-09-18T08:08:17.981 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:08:17 smithi007 ceph-mon[39249]: from='client.? ' entity='client.admin' cmd={"prefix": "osd in", "ids": ["4"]} : dispatch 2024-09-18T08:08:18.012 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3 2024-09-18T08:08:18.036 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:08:18.037 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:08:18.037 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3' 2024-09-18T08:08:18.138 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6 2024-09-18T08:08:18.155 INFO:tasks.radosbench.radosbench.0.smithi007.stdout:2024-09-18T08:08:18.153801+0000 min lat: 0.00120772 max lat: 1.82134 avg lat: 0.00910396 2024-09-18T08:08:18.155 INFO:tasks.radosbench.radosbench.0.smithi007.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-09-18T08:08:18.155 INFO:tasks.radosbench.radosbench.0.smithi007.stdout: 20 16 34067 34051 106.398 39.1875 0.00213613 0.00910396 2024-09-18T08:08:18.165 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:08:18.165 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:08:18.166 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6' 2024-09-18T08:08:18.266 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5 2024-09-18T08:08:18.293 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:08:18.294 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:08:18.295 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5' 2024-09-18T08:08:18.396 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1 2024-09-18T08:08:18.423 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:08:18.423 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:08:18.424 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1' 2024-09-18T08:08:18.525 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4 2024-09-18T08:08:18.568 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:08:18.569 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:08:18.569 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4' 2024-09-18T08:08:18.670 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3 2024-09-18T08:08:18.697 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:08:18.698 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:08:18.698 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3' 2024-09-18T08:08:18.799 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4 2024-09-18T08:08:18.824 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:08:18.825 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:08:18.825 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4' 2024-09-18T08:08:18.906 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:08:18 smithi050 ceph-mon[32371]: 5.b scrub starts 2024-09-18T08:08:18.906 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:08:18 smithi050 ceph-mon[32371]: 5.b scrub ok 2024-09-18T08:08:18.906 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:08:18 smithi050 ceph-mon[32371]: pgmap v260: 158 pgs: 2 active+recovering+undersized+degraded+remapped, 1 active+recovering+undersized+remapped, 155 active+clean; 1.7 GiB data, 6.0 GiB used, 620 GiB / 626 GiB avail; 57 MiB/s wr, 913 op/s; 2/84381 objects degraded (0.002%); 3554/84381 objects misplaced (4.212%); 7.3 MiB/s, 120 objects/s recovering 2024-09-18T08:08:18.906 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:08:18 smithi050 ceph-mon[32371]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd in", "ids": ["4"]}]': finished 2024-09-18T08:08:18.906 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:08:18 smithi050 ceph-mon[32371]: osdmap e181: 8 total, 8 up, 8 in 2024-09-18T08:08:18.906 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:08:18 smithi050 ceph-mon[32371]: from='mgr.24470 172.21.15.7:0/2594390261' entity='mgr.y' 2024-09-18T08:08:18.906 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:08:18 smithi050 ceph-mon[32371]: from='mgr.24470 172.21.15.7:0/2594390261' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-18T08:08:18.906 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:08:18 smithi050 ceph-mon[32371]: 3.8 scrub starts 2024-09-18T08:08:18.906 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:08:18 smithi050 ceph-mon[32371]: 3.8 scrub ok 2024-09-18T08:08:18.927 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5 2024-09-18T08:08:18.951 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:08:18.951 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:08:18.952 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5' 2024-09-18T08:08:18.980 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:08:18 smithi007 ceph-mon[29459]: 5.b scrub starts 2024-09-18T08:08:18.981 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:08:18 smithi007 ceph-mon[29459]: 5.b scrub ok 2024-09-18T08:08:18.981 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:08:18 smithi007 ceph-mon[29459]: pgmap v260: 158 pgs: 2 active+recovering+undersized+degraded+remapped, 1 active+recovering+undersized+remapped, 155 active+clean; 1.7 GiB data, 6.0 GiB used, 620 GiB / 626 GiB avail; 57 MiB/s wr, 913 op/s; 2/84381 objects degraded (0.002%); 3554/84381 objects misplaced (4.212%); 7.3 MiB/s, 120 objects/s recovering 2024-09-18T08:08:18.981 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:08:18 smithi007 ceph-mon[29459]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd in", "ids": ["4"]}]': finished 2024-09-18T08:08:18.985 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:08:18 smithi007 ceph-mon[29459]: osdmap e181: 8 total, 8 up, 8 in 2024-09-18T08:08:18.985 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:08:18 smithi007 ceph-mon[29459]: from='mgr.24470 172.21.15.7:0/2594390261' entity='mgr.y' 2024-09-18T08:08:18.985 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:08:18 smithi007 ceph-mon[29459]: from='mgr.24470 172.21.15.7:0/2594390261' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-18T08:08:18.985 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:08:18 smithi007 ceph-mon[29459]: 3.8 scrub starts 2024-09-18T08:08:18.985 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:08:18 smithi007 ceph-mon[29459]: 3.8 scrub ok 2024-09-18T08:08:18.985 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:08:18 smithi007 ceph-mon[39249]: 5.b scrub starts 2024-09-18T08:08:18.985 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:08:18 smithi007 ceph-mon[39249]: 5.b scrub ok 2024-09-18T08:08:18.985 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:08:18 smithi007 ceph-mon[39249]: pgmap v260: 158 pgs: 2 active+recovering+undersized+degraded+remapped, 1 active+recovering+undersized+remapped, 155 active+clean; 1.7 GiB data, 6.0 GiB used, 620 GiB / 626 GiB avail; 57 MiB/s wr, 913 op/s; 2/84381 objects degraded (0.002%); 3554/84381 objects misplaced (4.212%); 7.3 MiB/s, 120 objects/s recovering 2024-09-18T08:08:18.985 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:08:18 smithi007 ceph-mon[39249]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd in", "ids": ["4"]}]': finished 2024-09-18T08:08:18.985 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:08:18 smithi007 ceph-mon[39249]: osdmap e181: 8 total, 8 up, 8 in 2024-09-18T08:08:18.986 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:08:18 smithi007 ceph-mon[39249]: from='mgr.24470 172.21.15.7:0/2594390261' entity='mgr.y' 2024-09-18T08:08:18.986 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:08:18 smithi007 ceph-mon[39249]: from='mgr.24470 172.21.15.7:0/2594390261' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-18T08:08:18.986 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:08:18 smithi007 ceph-mon[39249]: 3.8 scrub starts 2024-09-18T08:08:18.986 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:08:18 smithi007 ceph-mon[39249]: 3.8 scrub ok 2024-09-18T08:08:19.052 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3 2024-09-18T08:08:19.078 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:08:19.079 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:08:19.079 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3' 2024-09-18T08:08:19.155 INFO:tasks.radosbench.radosbench.0.smithi007.stdout: 21 16 34116 34100 101.477 3.0625 0.00240316 0.00932391 2024-09-18T08:08:19.180 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3 2024-09-18T08:08:19.205 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:08:19.206 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:08:19.206 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3' 2024-09-18T08:08:19.307 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0 2024-09-18T08:08:19.353 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:08:19.354 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:08:19.354 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0' 2024-09-18T08:08:19.406 INFO:journalctl@ceph.iscsi.iscsi.a.smithi050.stdout:Sep 18 08:08:18 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-iscsi-iscsi-a[57782]: debug there is no tcmu-runner data available 2024-09-18T08:08:19.455 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6 2024-09-18T08:08:19.484 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:08:19.485 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:08:19.485 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6' 2024-09-18T08:08:19.586 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1 2024-09-18T08:08:19.608 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:08:19.609 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:08:19.609 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1' 2024-09-18T08:08:19.710 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6 2024-09-18T08:08:19.743 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:08:19.744 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:08:19.744 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6' 2024-09-18T08:08:19.846 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2 2024-09-18T08:08:19.886 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:08:19.886 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:08:19.886 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2' 2024-09-18T08:08:19.906 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:08:19 smithi050 ceph-mon[32371]: osdmap e182: 8 total, 8 up, 8 in 2024-09-18T08:08:19.906 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:08:19 smithi050 ceph-mon[32371]: 1.5 scrub starts 2024-09-18T08:08:19.906 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:08:19 smithi050 ceph-mon[32371]: 1.5 scrub ok 2024-09-18T08:08:19.906 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:08:19 smithi050 ceph-mon[32371]: osdmap e183: 8 total, 8 up, 8 in 2024-09-18T08:08:19.980 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:08:19 smithi007 ceph-mon[29459]: osdmap e182: 8 total, 8 up, 8 in 2024-09-18T08:08:19.980 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:08:19 smithi007 ceph-mon[29459]: 1.5 scrub starts 2024-09-18T08:08:19.981 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:08:19 smithi007 ceph-mon[29459]: 1.5 scrub ok 2024-09-18T08:08:19.981 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:08:19 smithi007 ceph-mon[29459]: osdmap e183: 8 total, 8 up, 8 in 2024-09-18T08:08:19.981 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:08:19 smithi007 ceph-mon[39249]: osdmap e182: 8 total, 8 up, 8 in 2024-09-18T08:08:19.981 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:08:19 smithi007 ceph-mon[39249]: 1.5 scrub starts 2024-09-18T08:08:19.981 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:08:19 smithi007 ceph-mon[39249]: 1.5 scrub ok 2024-09-18T08:08:19.981 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:08:19 smithi007 ceph-mon[39249]: osdmap e183: 8 total, 8 up, 8 in 2024-09-18T08:08:19.987 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4 2024-09-18T08:08:20.026 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:08:20.026 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:08:20.027 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4' 2024-09-18T08:08:20.128 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4 2024-09-18T08:08:20.155 INFO:tasks.radosbench.radosbench.0.smithi007.stdout: 22 16 35349 35333 100.367 77.0625 0.00534067 0.00995613 2024-09-18T08:08:20.165 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:08:20.166 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:08:20.166 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4' 2024-09-18T08:08:20.267 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3 2024-09-18T08:08:20.303 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:08:20.304 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:08:20.304 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3' 2024-09-18T08:08:20.405 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3 2024-09-18T08:08:20.429 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:08:20.429 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:08:20.429 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3' 2024-09-18T08:08:20.530 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2 2024-09-18T08:08:20.568 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:08:20.569 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:08:20.569 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2' 2024-09-18T08:08:20.670 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7 2024-09-18T08:08:20.697 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:08:20.697 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:08:20.698 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7' 2024-09-18T08:08:20.799 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6 2024-09-18T08:08:20.835 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:08:20.835 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:08:20.835 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6' 2024-09-18T08:08:20.863 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:08:20 smithi007 ceph-mon[29459]: 4.b scrub starts 2024-09-18T08:08:20.863 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:08:20 smithi007 ceph-mon[29459]: 4.b scrub ok 2024-09-18T08:08:20.863 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:08:20 smithi007 ceph-mon[29459]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-18T08:08:20.863 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:08:20 smithi007 ceph-mon[29459]: pgmap v264: 158 pgs: 2 activating+undersized+degraded+remapped, 3 remapped+peering, 1 unknown, 51 peering, 101 active+clean; 2.0 GiB data, 7.5 GiB used, 708 GiB / 715 GiB avail; 6.2 KiB/s rd, 63 MiB/s wr, 1.02k op/s; 4303/96525 objects degraded (4.458%); 13 MiB/s, 206 objects/s recovering 2024-09-18T08:08:20.863 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:08:20 smithi007 ceph-mon[29459]: 1.3 scrub starts 2024-09-18T08:08:20.864 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:08:20 smithi007 ceph-mon[29459]: 1.3 scrub ok 2024-09-18T08:08:20.864 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:08:20 smithi007 ceph-mon[29459]: Health check failed: Reduced data availability: 10 pgs inactive, 18 pgs peering (PG_AVAILABILITY) 2024-09-18T08:08:20.865 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:08:20 smithi007 ceph-mon[39249]: 4.b scrub starts 2024-09-18T08:08:20.865 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:08:20 smithi007 ceph-mon[39249]: 4.b scrub ok 2024-09-18T08:08:20.865 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:08:20 smithi007 ceph-mon[39249]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-18T08:08:20.866 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:08:20 smithi007 ceph-mon[39249]: pgmap v264: 158 pgs: 2 activating+undersized+degraded+remapped, 3 remapped+peering, 1 unknown, 51 peering, 101 active+clean; 2.0 GiB data, 7.5 GiB used, 708 GiB / 715 GiB avail; 6.2 KiB/s rd, 63 MiB/s wr, 1.02k op/s; 4303/96525 objects degraded (4.458%); 13 MiB/s, 206 objects/s recovering 2024-09-18T08:08:20.866 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:08:20 smithi007 ceph-mon[39249]: 1.3 scrub starts 2024-09-18T08:08:20.866 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:08:20 smithi007 ceph-mon[39249]: 1.3 scrub ok 2024-09-18T08:08:20.866 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:08:20 smithi007 ceph-mon[39249]: Health check failed: Reduced data availability: 10 pgs inactive, 18 pgs peering (PG_AVAILABILITY) 2024-09-18T08:08:20.906 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:08:20 smithi050 ceph-mon[32371]: 4.b scrub starts 2024-09-18T08:08:20.906 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:08:20 smithi050 ceph-mon[32371]: 4.b scrub ok 2024-09-18T08:08:20.906 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:08:20 smithi050 ceph-mon[32371]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-18T08:08:20.906 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:08:20 smithi050 ceph-mon[32371]: pgmap v264: 158 pgs: 2 activating+undersized+degraded+remapped, 3 remapped+peering, 1 unknown, 51 peering, 101 active+clean; 2.0 GiB data, 7.5 GiB used, 708 GiB / 715 GiB avail; 6.2 KiB/s rd, 63 MiB/s wr, 1.02k op/s; 4303/96525 objects degraded (4.458%); 13 MiB/s, 206 objects/s recovering 2024-09-18T08:08:20.906 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:08:20 smithi050 ceph-mon[32371]: 1.3 scrub starts 2024-09-18T08:08:20.906 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:08:20 smithi050 ceph-mon[32371]: 1.3 scrub ok 2024-09-18T08:08:20.906 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:08:20 smithi050 ceph-mon[32371]: Health check failed: Reduced data availability: 10 pgs inactive, 18 pgs peering (PG_AVAILABILITY) 2024-09-18T08:08:20.936 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0 2024-09-18T08:08:20.976 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:08:20.977 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:08:20.977 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0' 2024-09-18T08:08:21.078 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6 2024-09-18T08:08:21.113 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:08:21.114 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:08:21.114 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6' 2024-09-18T08:08:21.155 INFO:tasks.radosbench.radosbench.0.smithi007.stdout: 23 16 36628 36612 99.478 79.9375 0.0172217 0.0100461 2024-09-18T08:08:21.215 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2 2024-09-18T08:08:21.253 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:08:21.254 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:08:21.254 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2' 2024-09-18T08:08:21.355 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6 2024-09-18T08:08:21.393 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:08:21.394 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:08:21.394 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6' 2024-09-18T08:08:21.495 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6 2024-09-18T08:08:21.528 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:08:21.528 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:08:21.529 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6' 2024-09-18T08:08:21.630 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4 2024-09-18T08:08:21.665 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:08:21.666 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:08:21.666 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4' 2024-09-18T08:08:21.686 INFO:tasks.thrashosds.thrasher:Added osd 4 2024-09-18T08:08:21.738 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/779349ca-7593-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T08:08:21.767 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4 2024-09-18T08:08:21.778 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/779349ca-7593-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T08:08:21.801 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:08:21.802 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:08:21.802 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4' 2024-09-18T08:08:21.871 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:08:21 smithi007 ceph-mon[29459]: 3.15 scrub starts 2024-09-18T08:08:21.871 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:08:21 smithi007 ceph-mon[29459]: 3.15 scrub ok 2024-09-18T08:08:21.873 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:08:21 smithi007 ceph-mon[39249]: 3.15 scrub starts 2024-09-18T08:08:21.873 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:08:21 smithi007 ceph-mon[39249]: 3.15 scrub ok 2024-09-18T08:08:21.903 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2 2024-09-18T08:08:21.906 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:08:21 smithi050 ceph-mon[32371]: 3.15 scrub starts 2024-09-18T08:08:21.906 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:08:21 smithi050 ceph-mon[32371]: 3.15 scrub ok 2024-09-18T08:08:21.951 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:08:21.951 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:08:21.951 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2' 2024-09-18T08:08:22.052 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2 2024-09-18T08:08:22.102 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:08:22.103 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:08:22.103 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2' 2024-09-18T08:08:22.155 INFO:tasks.radosbench.radosbench.0.smithi007.stdout: 24 15 38062 38047 99.0697 89.6875 0.00435987 0.0100879 2024-09-18T08:08:22.205 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7 2024-09-18T08:08:22.241 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:08:22.242 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:08:22.242 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7' 2024-09-18T08:08:22.343 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7 2024-09-18T08:08:22.370 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:08:22.371 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:08:22.371 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7' 2024-09-18T08:08:22.472 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2 2024-09-18T08:08:22.510 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:08:22.510 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:08:22.511 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2' 2024-09-18T08:08:22.611 INFO:teuthology.orchestra.run.smithi007.stdout:ERROR: (22) Invalid argument 2024-09-18T08:08:22.611 INFO:teuthology.orchestra.run.smithi007.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-18T08:08:22.612 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1 2024-09-18T08:08:22.658 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:08:22.659 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:08:22.659 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1' 2024-09-18T08:08:22.730 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:08:22 smithi007 ceph-mon[29459]: pgmap v265: 158 pgs: 2 activating+undersized+degraded+remapped, 3 remapped+peering, 1 unknown, 51 peering, 101 active+clean; 2.0 GiB data, 7.5 GiB used, 708 GiB / 715 GiB avail; 4.4 KiB/s rd, 45 MiB/s wr, 717 op/s; 4303/96525 objects degraded (4.458%); 9.4 MiB/s, 145 objects/s recovering 2024-09-18T08:08:22.731 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:08:22 smithi007 ceph-mon[39249]: pgmap v265: 158 pgs: 2 activating+undersized+degraded+remapped, 3 remapped+peering, 1 unknown, 51 peering, 101 active+clean; 2.0 GiB data, 7.5 GiB used, 708 GiB / 715 GiB avail; 4.4 KiB/s rd, 45 MiB/s wr, 717 op/s; 4303/96525 objects degraded (4.458%); 9.4 MiB/s, 145 objects/s recovering 2024-09-18T08:08:22.760 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3 2024-09-18T08:08:22.798 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:08:22.798 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:08:22.799 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3' 2024-09-18T08:08:22.833 INFO:teuthology.orchestra.run.smithi007.stdout:osd.0: {} 2024-09-18T08:08:22.833 INFO:teuthology.orchestra.run.smithi007.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-18T08:08:22.882 INFO:teuthology.orchestra.run.smithi007.stdout:osd.1: {} 2024-09-18T08:08:22.882 INFO:teuthology.orchestra.run.smithi007.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-18T08:08:22.900 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2 2024-09-18T08:08:22.906 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:08:22 smithi050 ceph-mon[32371]: pgmap v265: 158 pgs: 2 activating+undersized+degraded+remapped, 3 remapped+peering, 1 unknown, 51 peering, 101 active+clean; 2.0 GiB data, 7.5 GiB used, 708 GiB / 715 GiB avail; 4.4 KiB/s rd, 45 MiB/s wr, 717 op/s; 4303/96525 objects degraded (4.458%); 9.4 MiB/s, 145 objects/s recovering 2024-09-18T08:08:22.913 INFO:teuthology.orchestra.run.smithi007.stdout:osd.2: {} 2024-09-18T08:08:22.913 INFO:teuthology.orchestra.run.smithi007.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-18T08:08:22.924 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid 779349ca-7593-11ef-bceb-c7b262605968 -- ceph osd set noscrub 2024-09-18T08:08:22.951 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:08:22.951 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:08:22.952 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2' 2024-09-18T08:08:22.952 INFO:teuthology.orchestra.run.smithi007.stdout:osd.3: {} 2024-09-18T08:08:22.952 INFO:teuthology.orchestra.run.smithi007.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-18T08:08:22.989 INFO:teuthology.orchestra.run.smithi007.stdout:osd.4: {} 2024-09-18T08:08:22.989 INFO:teuthology.orchestra.run.smithi007.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-18T08:08:23.026 INFO:teuthology.orchestra.run.smithi007.stdout:osd.5: {} 2024-09-18T08:08:23.026 INFO:teuthology.orchestra.run.smithi007.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-18T08:08:23.053 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2 2024-09-18T08:08:23.059 INFO:teuthology.orchestra.run.smithi007.stdout:osd.6: {} 2024-09-18T08:08:23.059 INFO:teuthology.orchestra.run.smithi007.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-18T08:08:23.100 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:08:23.100 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:08:23.101 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2' 2024-09-18T08:08:23.101 INFO:teuthology.orchestra.run.smithi007.stdout:osd.7: {} 2024-09-18T08:08:23.101 INFO:teuthology.orchestra.run.smithi007.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-18T08:08:23.155 INFO:tasks.radosbench.radosbench.0.smithi007.stdout: 25 16 39351 39335 98.3267 80.5 0.00741661 0.0101634 2024-09-18T08:08:23.202 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6 2024-09-18T08:08:23.234 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:08:23.234 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:08:23.235 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6' 2024-09-18T08:08:23.336 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4 2024-09-18T08:08:23.378 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:08:23.378 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:08:23.379 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4' 2024-09-18T08:08:23.479 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7 2024-09-18T08:08:23.516 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:08:23.516 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:08:23.517 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7' 2024-09-18T08:08:23.617 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4 2024-09-18T08:08:23.656 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:08:23 smithi050 ceph-mon[32371]: from='client.? 172.21.15.7:0/3022390309' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-18T08:08:23.657 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:08:23.657 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:08:23.657 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4' 2024-09-18T08:08:23.758 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6 2024-09-18T08:08:23.781 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid 779349ca-7593-11ef-bceb-c7b262605968 -- ceph daemon osd.0 dump_blocked_ops 2024-09-18T08:08:23.793 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:08:23.794 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:08:23.795 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6' 2024-09-18T08:08:23.818 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:08:23 smithi007 ceph-mon[29459]: from='client.? 172.21.15.7:0/3022390309' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-18T08:08:23.818 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:08:23 smithi007 ceph-mon[39249]: from='client.? 172.21.15.7:0/3022390309' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-18T08:08:23.895 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5 2024-09-18T08:08:23.925 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:08:23.926 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:08:23.926 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5' 2024-09-18T08:08:24.027 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5 2024-09-18T08:08:24.061 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:08:24.061 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:08:24.062 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5' 2024-09-18T08:08:24.155 INFO:tasks.radosbench.radosbench.0.smithi007.stdout: 26 16 40747 40731 97.8987 87.25 0.00730138 0.0102078 2024-09-18T08:08:24.163 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7 2024-09-18T08:08:24.201 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:08:24.201 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:08:24.202 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7' 2024-09-18T08:08:24.302 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6 2024-09-18T08:08:24.336 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:08:24.337 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:08:24.337 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6' 2024-09-18T08:08:24.438 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1 2024-09-18T08:08:24.519 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:08:24.520 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:08:24.520 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1' 2024-09-18T08:08:24.580 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/779349ca-7593-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T08:08:24.582 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/779349ca-7593-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T08:08:24.621 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3 2024-09-18T08:08:24.674 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:08:24.674 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:08:24.675 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3' 2024-09-18T08:08:24.725 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:08:24 smithi007 ceph-mon[29459]: pgmap v266: 158 pgs: 1 active+recovering+undersized+remapped, 2 activating+undersized+degraded+remapped, 2 remapped+peering, 1 unknown, 18 peering, 134 active+clean; 2.0 GiB data, 7.9 GiB used, 707 GiB / 715 GiB avail; 853 B/s rd, 56 MiB/s wr, 896 op/s; 4303/100503 objects degraded (4.281%); 1699/100503 objects misplaced (1.690%); 14 MiB/s, 222 objects/s recovering 2024-09-18T08:08:24.725 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:08:24 smithi007 ceph-mon[29459]: Health check update: Degraded data redundancy: 4303/100503 objects degraded (4.281%), 2 pgs degraded (PG_DEGRADED) 2024-09-18T08:08:24.725 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:08:24 smithi007 ceph-mon[29459]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 10 pgs inactive, 18 pgs peering) 2024-09-18T08:08:24.725 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:08:24 smithi007 ceph-mon[39249]: pgmap v266: 158 pgs: 1 active+recovering+undersized+remapped, 2 activating+undersized+degraded+remapped, 2 remapped+peering, 1 unknown, 18 peering, 134 active+clean; 2.0 GiB data, 7.9 GiB used, 707 GiB / 715 GiB avail; 853 B/s rd, 56 MiB/s wr, 896 op/s; 4303/100503 objects degraded (4.281%); 1699/100503 objects misplaced (1.690%); 14 MiB/s, 222 objects/s recovering 2024-09-18T08:08:24.726 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:08:24 smithi007 ceph-mon[39249]: Health check update: Degraded data redundancy: 4303/100503 objects degraded (4.281%), 2 pgs degraded (PG_DEGRADED) 2024-09-18T08:08:24.726 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:08:24 smithi007 ceph-mon[39249]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 10 pgs inactive, 18 pgs peering) 2024-09-18T08:08:24.775 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0 2024-09-18T08:08:24.811 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:08:24.812 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:08:24.812 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0' 2024-09-18T08:08:24.906 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:08:24 smithi050 ceph-mon[32371]: pgmap v266: 158 pgs: 1 active+recovering+undersized+remapped, 2 activating+undersized+degraded+remapped, 2 remapped+peering, 1 unknown, 18 peering, 134 active+clean; 2.0 GiB data, 7.9 GiB used, 707 GiB / 715 GiB avail; 853 B/s rd, 56 MiB/s wr, 896 op/s; 4303/100503 objects degraded (4.281%); 1699/100503 objects misplaced (1.690%); 14 MiB/s, 222 objects/s recovering 2024-09-18T08:08:24.906 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:08:24 smithi050 ceph-mon[32371]: Health check update: Degraded data redundancy: 4303/100503 objects degraded (4.281%), 2 pgs degraded (PG_DEGRADED) 2024-09-18T08:08:24.906 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:08:24 smithi050 ceph-mon[32371]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 10 pgs inactive, 18 pgs peering) 2024-09-18T08:08:24.912 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2 2024-09-18T08:08:24.949 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:08:24.950 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:08:24.950 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2' 2024-09-18T08:08:25.051 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6 2024-09-18T08:08:25.085 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:08:25.086 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:08:25.086 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6' 2024-09-18T08:08:25.156 INFO:tasks.radosbench.radosbench.0.smithi007.stdout: 27 15 42094 42079 97.3929 84.25 0.0217688 0.0102617 2024-09-18T08:08:25.188 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6 2024-09-18T08:08:25.211 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:08:25.211 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:08:25.212 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6' 2024-09-18T08:08:25.313 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6 2024-09-18T08:08:25.350 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:08:25.351 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:08:25.351 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6' 2024-09-18T08:08:25.452 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5 2024-09-18T08:08:25.486 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:08:25.486 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:08:25.487 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5' 2024-09-18T08:08:25.588 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3 2024-09-18T08:08:25.630 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:08:25.631 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:08:25.631 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3' 2024-09-18T08:08:25.732 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6 2024-09-18T08:08:25.764 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:08:25.764 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:08:25.765 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6' 2024-09-18T08:08:25.866 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6 2024-09-18T08:08:25.898 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:08:25.899 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:08:25.899 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6' 2024-09-18T08:08:26.000 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2 2024-09-18T08:08:26.052 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:08:26.053 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:08:26.053 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2' 2024-09-18T08:08:26.153 INFO:teuthology.orchestra.run.smithi007.stdout:{ 2024-09-18T08:08:26.153 INFO:teuthology.orchestra.run.smithi007.stdout: "ops": [], 2024-09-18T08:08:26.153 INFO:teuthology.orchestra.run.smithi007.stdout: "complaint_time": 30, 2024-09-18T08:08:26.153 INFO:teuthology.orchestra.run.smithi007.stdout: "num_blocked_ops": 0 2024-09-18T08:08:26.153 INFO:teuthology.orchestra.run.smithi007.stdout:} 2024-09-18T08:08:26.153 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0 2024-09-18T08:08:26.157 INFO:tasks.radosbench.radosbench.0.smithi007.stdout: 28 16 43331 43315 96.6733 77.25 0.00461368 0.010335 2024-09-18T08:08:26.196 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:08:26.196 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:08:26.197 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0' 2024-09-18T08:08:26.297 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2 2024-09-18T08:08:26.336 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:08:26.337 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:08:26.337 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2' 2024-09-18T08:08:26.438 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2 2024-09-18T08:08:26.467 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid 779349ca-7593-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-18T08:08:26.477 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:08:26.477 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:08:26.477 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2' 2024-09-18T08:08:26.578 INFO:teuthology.orchestra.run.smithi007.stderr:noscrub is set 2024-09-18T08:08:26.578 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2 2024-09-18T08:08:26.653 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:08:26.653 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:08:26.654 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2' 2024-09-18T08:08:26.686 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 5, 7, 6, 4] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-09-18T08:08:26.686 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid 779349ca-7593-11ef-bceb-c7b262605968 -- ceph osd reweight 0 1 2024-09-18T08:08:26.755 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7 2024-09-18T08:08:26.791 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:08:26.792 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:08:26.792 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7' 2024-09-18T08:08:26.893 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4 2024-09-18T08:08:26.905 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:08:26 smithi050 ceph-mon[32371]: pgmap v267: 158 pgs: 2 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+remapped, 152 active+clean; 2.4 GiB data, 8.6 GiB used, 707 GiB / 715 GiB avail; 97 MiB/s wr, 1.55k op/s; 4146/120060 objects degraded (3.453%); 7494/120060 objects misplaced (6.242%); 16 MiB/s, 264 objects/s recovering 2024-09-18T08:08:26.906 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:08:26 smithi050 ceph-mon[32371]: from='client.? 172.21.15.7:0/1772992285' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-18T08:08:26.928 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:08:26.928 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:08:26.929 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4' 2024-09-18T08:08:26.980 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:08:26 smithi007 ceph-mon[29459]: pgmap v267: 158 pgs: 2 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+remapped, 152 active+clean; 2.4 GiB data, 8.6 GiB used, 707 GiB / 715 GiB avail; 97 MiB/s wr, 1.55k op/s; 4146/120060 objects degraded (3.453%); 7494/120060 objects misplaced (6.242%); 16 MiB/s, 264 objects/s recovering 2024-09-18T08:08:26.981 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:08:26 smithi007 ceph-mon[29459]: from='client.? 172.21.15.7:0/1772992285' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-18T08:08:26.981 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Sep 18 08:08:26 smithi007 ceph-779349ca-7593-11ef-bceb-c7b262605968-mgr-y[29688]: ::ffff:172.21.15.50 - - [18/Sep/2024:08:08:26] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-09-18T08:08:26.981 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:08:26 smithi007 ceph-mon[39249]: pgmap v267: 158 pgs: 2 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+remapped, 152 active+clean; 2.4 GiB data, 8.6 GiB used, 707 GiB / 715 GiB avail; 97 MiB/s wr, 1.55k op/s; 4146/120060 objects degraded (3.453%); 7494/120060 objects misplaced (6.242%); 16 MiB/s, 264 objects/s recovering 2024-09-18T08:08:26.981 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:08:26 smithi007 ceph-mon[39249]: from='client.? 172.21.15.7:0/1772992285' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-18T08:08:27.030 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7 2024-09-18T08:08:27.048 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid 779349ca-7593-11ef-bceb-c7b262605968 -- ceph daemon osd.0 dump_historic_ops 2024-09-18T08:08:27.060 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:08:27.061 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:08:27.061 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7' 2024-09-18T08:08:27.156 INFO:tasks.radosbench.radosbench.0.smithi007.stdout: 29 16 44584 44568 96.0386 78.3125 0.0153697 0.010406 2024-09-18T08:08:27.162 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0 2024-09-18T08:08:27.206 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:08:27.207 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:08:27.207 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0' 2024-09-18T08:08:27.308 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5 2024-09-18T08:08:27.337 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:08:27.338 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:08:27.338 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5' 2024-09-18T08:08:27.439 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6 2024-09-18T08:08:27.478 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:08:27.479 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:08:27.479 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6' 2024-09-18T08:08:27.580 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6 2024-09-18T08:08:27.615 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:08:27.616 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:08:27.616 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6' 2024-09-18T08:08:27.717 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3 2024-09-18T08:08:27.755 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:08:27.756 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:08:27.757 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3' 2024-09-18T08:08:27.857 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0 2024-09-18T08:08:27.897 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:08:27.898 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:08:27.898 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0' 2024-09-18T08:08:27.906 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:08:27 smithi050 ceph-mon[32371]: 3.e scrub starts 2024-09-18T08:08:27.906 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:08:27 smithi050 ceph-mon[32371]: 3.e scrub ok 2024-09-18T08:08:27.906 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:08:27 smithi050 ceph-mon[32371]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-18T08:08:27.906 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:08:27 smithi050 ceph-mon[32371]: from='client.? 172.21.15.7:0/1772992285' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-18T08:08:27.906 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:08:27 smithi050 ceph-mon[32371]: osdmap e184: 8 total, 8 up, 8 in 2024-09-18T08:08:27.906 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:08:27 smithi050 ceph-mon[32371]: 1.7 scrub starts 2024-09-18T08:08:27.980 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:08:27 smithi007 ceph-mon[39249]: 3.e scrub starts 2024-09-18T08:08:27.980 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:08:27 smithi007 ceph-mon[39249]: 3.e scrub ok 2024-09-18T08:08:27.980 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:08:27 smithi007 ceph-mon[39249]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-18T08:08:27.981 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:08:27 smithi007 ceph-mon[39249]: from='client.? 172.21.15.7:0/1772992285' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-18T08:08:27.981 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:08:27 smithi007 ceph-mon[39249]: osdmap e184: 8 total, 8 up, 8 in 2024-09-18T08:08:27.981 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:08:27 smithi007 ceph-mon[39249]: 1.7 scrub starts 2024-09-18T08:08:27.981 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:08:27 smithi007 ceph-mon[29459]: 3.e scrub starts 2024-09-18T08:08:27.981 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:08:27 smithi007 ceph-mon[29459]: 3.e scrub ok 2024-09-18T08:08:27.981 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:08:27 smithi007 ceph-mon[29459]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-18T08:08:27.981 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:08:27 smithi007 ceph-mon[29459]: from='client.? 172.21.15.7:0/1772992285' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-18T08:08:27.981 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:08:27 smithi007 ceph-mon[29459]: osdmap e184: 8 total, 8 up, 8 in 2024-09-18T08:08:27.981 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:08:27 smithi007 ceph-mon[29459]: 1.7 scrub starts 2024-09-18T08:08:27.999 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2 2024-09-18T08:08:28.033 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:08:28.034 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:08:28.034 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2' 2024-09-18T08:08:28.134 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3 2024-09-18T08:08:28.156 INFO:tasks.radosbench.radosbench.0.smithi007.stdout: 30 16 45890 45874 95.5577 81.625 0.00901399 0.0104587 2024-09-18T08:08:28.168 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:08:28.169 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:08:28.169 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3' 2024-09-18T08:08:28.270 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4 2024-09-18T08:08:28.307 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:08:28.307 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:08:28.308 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4' 2024-09-18T08:08:28.409 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3 2024-09-18T08:08:28.444 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:08:28.445 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:08:28.445 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3' 2024-09-18T08:08:28.546 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1 2024-09-18T08:08:28.583 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:08:28.584 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:08:28.584 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1' 2024-09-18T08:08:28.685 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5 2024-09-18T08:08:28.721 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:08:28.722 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:08:28.722 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5' 2024-09-18T08:08:28.823 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2 2024-09-18T08:08:28.860 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:08:28.861 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:08:28.861 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2' 2024-09-18T08:08:28.906 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:08:28 smithi050 ceph-mon[32371]: pgmap v269: 158 pgs: 2 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+remapped, 152 active+clean; 2.4 GiB data, 8.6 GiB used, 707 GiB / 715 GiB avail; 61 MiB/s wr, 974 op/s; 4146/120060 objects degraded (3.453%); 7494/120060 objects misplaced (6.242%); 8.8 MiB/s, 149 objects/s recovering 2024-09-18T08:08:28.906 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:08:28 smithi050 ceph-mon[32371]: 1.a scrub starts 2024-09-18T08:08:28.906 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:08:28 smithi050 ceph-mon[32371]: 1.0 scrub starts 2024-09-18T08:08:28.906 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:08:28 smithi050 ceph-mon[32371]: 1.6 scrub starts 2024-09-18T08:08:28.906 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:08:28 smithi050 ceph-mon[32371]: 1.4 scrub starts 2024-09-18T08:08:28.962 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2 2024-09-18T08:08:28.982 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:08:28 smithi007 ceph-mon[29459]: pgmap v269: 158 pgs: 2 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+remapped, 152 active+clean; 2.4 GiB data, 8.6 GiB used, 707 GiB / 715 GiB avail; 61 MiB/s wr, 974 op/s; 4146/120060 objects degraded (3.453%); 7494/120060 objects misplaced (6.242%); 8.8 MiB/s, 149 objects/s recovering 2024-09-18T08:08:28.982 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:08:28 smithi007 ceph-mon[29459]: 1.a scrub starts 2024-09-18T08:08:28.983 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:08:28 smithi007 ceph-mon[29459]: 1.0 scrub starts 2024-09-18T08:08:28.983 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:08:28 smithi007 ceph-mon[29459]: 1.6 scrub starts 2024-09-18T08:08:28.983 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:08:28 smithi007 ceph-mon[29459]: 1.4 scrub starts 2024-09-18T08:08:28.983 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:08:28 smithi007 ceph-mon[39249]: pgmap v269: 158 pgs: 2 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+remapped, 152 active+clean; 2.4 GiB data, 8.6 GiB used, 707 GiB / 715 GiB avail; 61 MiB/s wr, 974 op/s; 4146/120060 objects degraded (3.453%); 7494/120060 objects misplaced (6.242%); 8.8 MiB/s, 149 objects/s recovering 2024-09-18T08:08:28.983 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:08:28 smithi007 ceph-mon[39249]: 1.a scrub starts 2024-09-18T08:08:28.984 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:08:28 smithi007 ceph-mon[39249]: 1.0 scrub starts 2024-09-18T08:08:28.984 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:08:28 smithi007 ceph-mon[39249]: 1.6 scrub starts 2024-09-18T08:08:28.984 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:08:28 smithi007 ceph-mon[39249]: 1.4 scrub starts 2024-09-18T08:08:29.004 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:08:29.008 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/779349ca-7593-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T08:08:29.008 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:08:29.008 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2' 2024-09-18T08:08:29.109 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3 2024-09-18T08:08:29.157 INFO:tasks.radosbench.radosbench.0.smithi007.stdout: 31 16 47199 47183 95.1128 81.8125 0.0167417 0.0105075 2024-09-18T08:08:29.158 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:08:29.159 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:08:29.159 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3' 2024-09-18T08:08:29.194 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/779349ca-7593-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T08:08:29.261 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4 2024-09-18T08:08:29.296 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:08:29.296 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:08:29.297 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4' 2024-09-18T08:08:29.397 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2 2024-09-18T08:08:29.405 INFO:journalctl@ceph.iscsi.iscsi.a.smithi050.stdout:Sep 18 08:08:29 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-iscsi-iscsi-a[57782]: debug there is no tcmu-runner data available 2024-09-18T08:08:29.443 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:08:29.444 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:08:29.444 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2' 2024-09-18T08:08:29.545 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0 2024-09-18T08:08:29.588 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:08:29.589 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:08:29.590 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0' 2024-09-18T08:08:29.691 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3 2024-09-18T08:08:29.728 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:08:29.728 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:08:29.729 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3' 2024-09-18T08:08:29.824 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:08:29 smithi007 ceph-mon[29459]: Health check update: Degraded data redundancy: 4146/120060 objects degraded (3.453%), 4 pgs degraded (PG_DEGRADED) 2024-09-18T08:08:29.825 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:08:29 smithi007 ceph-mon[29459]: osdmap e185: 8 total, 8 up, 8 in 2024-09-18T08:08:29.825 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:08:29 smithi007 ceph-mon[39249]: Health check update: Degraded data redundancy: 4146/120060 objects degraded (3.453%), 4 pgs degraded (PG_DEGRADED) 2024-09-18T08:08:29.825 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:08:29 smithi007 ceph-mon[39249]: osdmap e185: 8 total, 8 up, 8 in 2024-09-18T08:08:29.830 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7 2024-09-18T08:08:29.863 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:08:29.863 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:08:29.864 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7' 2024-09-18T08:08:29.906 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:08:29 smithi050 ceph-mon[32371]: Health check update: Degraded data redundancy: 4146/120060 objects degraded (3.453%), 4 pgs degraded (PG_DEGRADED) 2024-09-18T08:08:29.906 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:08:29 smithi050 ceph-mon[32371]: osdmap e185: 8 total, 8 up, 8 in 2024-09-18T08:08:29.965 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4 2024-09-18T08:08:29.998 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:08:29.999 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:08:30.000 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4' 2024-09-18T08:08:30.051 INFO:teuthology.orchestra.run.smithi007.stdout:osd.0: {} 2024-09-18T08:08:30.051 INFO:teuthology.orchestra.run.smithi007.stdout:osd.1: {} 2024-09-18T08:08:30.051 INFO:teuthology.orchestra.run.smithi007.stdout:osd.2: {} 2024-09-18T08:08:30.051 INFO:teuthology.orchestra.run.smithi007.stdout:osd.3: {} 2024-09-18T08:08:30.051 INFO:teuthology.orchestra.run.smithi007.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-18T08:08:30.052 INFO:teuthology.orchestra.run.smithi007.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-18T08:08:30.052 INFO:teuthology.orchestra.run.smithi007.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-18T08:08:30.052 INFO:teuthology.orchestra.run.smithi007.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-18T08:08:30.088 INFO:teuthology.orchestra.run.smithi007.stdout:osd.4: {} 2024-09-18T08:08:30.088 INFO:teuthology.orchestra.run.smithi007.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-18T08:08:30.100 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1 2024-09-18T08:08:30.129 INFO:teuthology.orchestra.run.smithi007.stdout:osd.5: {} 2024-09-18T08:08:30.129 INFO:teuthology.orchestra.run.smithi007.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-18T08:08:30.142 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/779349ca-7593-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T08:08:30.148 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:08:30.148 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:08:30.149 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1' 2024-09-18T08:08:30.162 INFO:tasks.radosbench.radosbench.0.smithi007.stdout: 32 16 48416 48400 94.5038 76.0625 0.00825246 0.0105738 2024-09-18T08:08:30.178 INFO:teuthology.orchestra.run.smithi007.stdout:osd.6: {} 2024-09-18T08:08:30.179 INFO:teuthology.orchestra.run.smithi007.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-18T08:08:30.228 INFO:teuthology.orchestra.run.smithi007.stdout:osd.7: {} 2024-09-18T08:08:30.228 INFO:teuthology.orchestra.run.smithi007.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-18T08:08:30.249 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7 2024-09-18T08:08:30.284 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:08:30.284 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:08:30.285 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7' 2024-09-18T08:08:30.386 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6 2024-09-18T08:08:30.422 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:08:30.423 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:08:30.423 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6' 2024-09-18T08:08:30.524 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5 2024-09-18T08:08:30.559 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:08:30.560 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:08:30.560 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5' 2024-09-18T08:08:30.661 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7 2024-09-18T08:08:30.692 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:08:30.692 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:08:30.693 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7' 2024-09-18T08:08:30.794 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5 2024-09-18T08:08:30.823 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:08:30.824 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:08:30.824 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5' 2024-09-18T08:08:30.906 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:08:30 smithi050 ceph-mon[32371]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-18T08:08:30.906 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:08:30 smithi050 ceph-mon[32371]: pgmap v271: 158 pgs: 1 active+recovery_wait+remapped, 1 active+recovering+undersized+remapped, 2 active+recovering+undersized+degraded+remapped, 5 active+clean+scrubbing, 2 active+recovery_wait+undersized+degraded+remapped, 147 active+clean; 2.8 GiB data, 9.9 GiB used, 705 GiB / 715 GiB avail; 112 MiB/s wr, 1.78k op/s; 4146/139419 objects degraded (2.974%); 6454/139419 objects misplaced (4.629%); 17 MiB/s, 279 objects/s recovering 2024-09-18T08:08:30.906 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:08:30 smithi050 ceph-mon[32371]: 1.1 scrub starts 2024-09-18T08:08:30.907 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:08:30 smithi050 ceph-mon[32371]: from='client.? 172.21.15.7:0/3213341493' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-18T08:08:30.907 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:08:30 smithi050 ceph-mon[32371]: from='client.? 172.21.15.7:0/890696257' entity='client.admin' cmd={"prefix": "osd reweight", "id": 0, "weight": 1.0} : dispatch 2024-09-18T08:08:30.925 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0 2024-09-18T08:08:30.965 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:08:30.965 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:08:30.966 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0' 2024-09-18T08:08:30.980 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:08:30 smithi007 ceph-mon[39249]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-18T08:08:30.980 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:08:30 smithi007 ceph-mon[39249]: pgmap v271: 158 pgs: 1 active+recovery_wait+remapped, 1 active+recovering+undersized+remapped, 2 active+recovering+undersized+degraded+remapped, 5 active+clean+scrubbing, 2 active+recovery_wait+undersized+degraded+remapped, 147 active+clean; 2.8 GiB data, 9.9 GiB used, 705 GiB / 715 GiB avail; 112 MiB/s wr, 1.78k op/s; 4146/139419 objects degraded (2.974%); 6454/139419 objects misplaced (4.629%); 17 MiB/s, 279 objects/s recovering 2024-09-18T08:08:30.980 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:08:30 smithi007 ceph-mon[39249]: 1.1 scrub starts 2024-09-18T08:08:30.981 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:08:30 smithi007 ceph-mon[39249]: from='client.? 172.21.15.7:0/3213341493' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-18T08:08:30.981 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:08:30 smithi007 ceph-mon[39249]: from='client.? 172.21.15.7:0/890696257' entity='client.admin' cmd={"prefix": "osd reweight", "id": 0, "weight": 1.0} : dispatch 2024-09-18T08:08:30.981 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:08:30 smithi007 ceph-mon[29459]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-18T08:08:30.981 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:08:30 smithi007 ceph-mon[29459]: pgmap v271: 158 pgs: 1 active+recovery_wait+remapped, 1 active+recovering+undersized+remapped, 2 active+recovering+undersized+degraded+remapped, 5 active+clean+scrubbing, 2 active+recovery_wait+undersized+degraded+remapped, 147 active+clean; 2.8 GiB data, 9.9 GiB used, 705 GiB / 715 GiB avail; 112 MiB/s wr, 1.78k op/s; 4146/139419 objects degraded (2.974%); 6454/139419 objects misplaced (4.629%); 17 MiB/s, 279 objects/s recovering 2024-09-18T08:08:30.981 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:08:30 smithi007 ceph-mon[29459]: 1.1 scrub starts 2024-09-18T08:08:30.982 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:08:30 smithi007 ceph-mon[29459]: from='client.? 172.21.15.7:0/3213341493' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-18T08:08:30.982 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:08:30 smithi007 ceph-mon[29459]: from='client.? 172.21.15.7:0/890696257' entity='client.admin' cmd={"prefix": "osd reweight", "id": 0, "weight": 1.0} : dispatch 2024-09-18T08:08:31.066 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7 2024-09-18T08:08:31.070 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid 779349ca-7593-11ef-bceb-c7b262605968 -- ceph osd set nodeep-scrub 2024-09-18T08:08:31.104 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:08:31.105 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:08:31.105 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7' 2024-09-18T08:08:31.162 INFO:tasks.radosbench.radosbench.0.smithi007.stdout: 33 16 49556 49540 93.7986 71.25 0.00676682 0.0106531 2024-09-18T08:08:31.163 INFO:teuthology.orchestra.run.smithi007.stderr:reweighted osd.0 to 1 (10000) 2024-09-18T08:08:31.205 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0 2024-09-18T08:08:31.256 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:08:31.257 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:08:31.257 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0' 2024-09-18T08:08:31.358 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2 2024-09-18T08:08:31.411 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:08:31.411 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:08:31.412 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2' 2024-09-18T08:08:31.513 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3 2024-09-18T08:08:31.558 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:08:31.559 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:08:31.559 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3' 2024-09-18T08:08:31.660 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3 2024-09-18T08:08:31.698 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:08:31.699 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:08:31.699 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3' 2024-09-18T08:08:31.800 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0 2024-09-18T08:08:31.840 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:08:31.841 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:08:31.841 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0' 2024-09-18T08:08:31.906 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:08:31 smithi050 ceph-mon[32371]: purged_snaps scrub starts 2024-09-18T08:08:31.906 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:08:31 smithi050 ceph-mon[32371]: purged_snaps scrub ok 2024-09-18T08:08:31.906 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:08:31 smithi050 ceph-mon[32371]: 1.8 scrub starts 2024-09-18T08:08:31.906 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:08:31 smithi050 ceph-mon[32371]: from='client.? 172.21.15.7:0/890696257' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]': finished 2024-09-18T08:08:31.906 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:08:31 smithi050 ceph-mon[32371]: osdmap e186: 8 total, 8 up, 8 in 2024-09-18T08:08:31.942 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0 2024-09-18T08:08:31.946 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:08:31 smithi007 ceph-mon[29459]: purged_snaps scrub starts 2024-09-18T08:08:31.946 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:08:31 smithi007 ceph-mon[29459]: purged_snaps scrub ok 2024-09-18T08:08:31.946 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:08:31 smithi007 ceph-mon[29459]: 1.8 scrub starts 2024-09-18T08:08:31.946 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:08:31 smithi007 ceph-mon[29459]: from='client.? 172.21.15.7:0/890696257' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]': finished 2024-09-18T08:08:31.946 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:08:31 smithi007 ceph-mon[29459]: osdmap e186: 8 total, 8 up, 8 in 2024-09-18T08:08:31.946 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:08:31 smithi007 ceph-mon[39249]: purged_snaps scrub starts 2024-09-18T08:08:31.946 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:08:31 smithi007 ceph-mon[39249]: purged_snaps scrub ok 2024-09-18T08:08:31.947 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:08:31 smithi007 ceph-mon[39249]: 1.8 scrub starts 2024-09-18T08:08:31.947 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:08:31 smithi007 ceph-mon[39249]: from='client.? 172.21.15.7:0/890696257' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]': finished 2024-09-18T08:08:31.947 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:08:31 smithi007 ceph-mon[39249]: osdmap e186: 8 total, 8 up, 8 in 2024-09-18T08:08:31.999 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:08:32.000 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:08:32.000 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0' 2024-09-18T08:08:32.101 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2 2024-09-18T08:08:32.142 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:08:32.143 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:08:32.143 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2' 2024-09-18T08:08:32.162 INFO:tasks.radosbench.radosbench.0.smithi007.stdout: 34 15 50864 50849 93.4459 81.8125 0.0166493 0.0106964 2024-09-18T08:08:32.244 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7 2024-09-18T08:08:32.279 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:08:32.279 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:08:32.280 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7' 2024-09-18T08:08:32.381 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0 2024-09-18T08:08:32.421 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:08:32.421 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:08:32.421 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0' 2024-09-18T08:08:32.503 INFO:teuthology.orchestra.run.smithi007.stdout:ERROR: (22) Invalid argument 2024-09-18T08:08:32.504 INFO:teuthology.orchestra.run.smithi007.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-18T08:08:32.524 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4 2024-09-18T08:08:32.557 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:08:32.558 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:08:32.558 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4' 2024-09-18T08:08:32.660 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0 2024-09-18T08:08:32.697 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:08:32.698 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:08:32.698 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0' 2024-09-18T08:08:32.799 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0 2024-09-18T08:08:32.840 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:08:32.841 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:08:32.841 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0' 2024-09-18T08:08:32.906 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:08:32 smithi050 ceph-mon[32371]: pgmap v273: 158 pgs: 1 active+recovery_wait+remapped, 1 active+recovering+undersized+remapped, 2 active+recovering+undersized+degraded+remapped, 5 active+clean+scrubbing, 2 active+recovery_wait+undersized+degraded+remapped, 147 active+clean; 2.8 GiB data, 9.9 GiB used, 705 GiB / 715 GiB avail; 853 B/s rd, 67 MiB/s wr, 1.08k op/s; 4146/139419 objects degraded (2.974%); 6454/139419 objects misplaced (4.629%); 11 MiB/s, 173 objects/s recovering 2024-09-18T08:08:32.942 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0 2024-09-18T08:08:32.980 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:08:32 smithi007 ceph-mon[29459]: pgmap v273: 158 pgs: 1 active+recovery_wait+remapped, 1 active+recovering+undersized+remapped, 2 active+recovering+undersized+degraded+remapped, 5 active+clean+scrubbing, 2 active+recovery_wait+undersized+degraded+remapped, 147 active+clean; 2.8 GiB data, 9.9 GiB used, 705 GiB / 715 GiB avail; 853 B/s rd, 67 MiB/s wr, 1.08k op/s; 4146/139419 objects degraded (2.974%); 6454/139419 objects misplaced (4.629%); 11 MiB/s, 173 objects/s recovering 2024-09-18T08:08:32.981 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:08:32 smithi007 ceph-mon[39249]: pgmap v273: 158 pgs: 1 active+recovery_wait+remapped, 1 active+recovering+undersized+remapped, 2 active+recovering+undersized+degraded+remapped, 5 active+clean+scrubbing, 2 active+recovery_wait+undersized+degraded+remapped, 147 active+clean; 2.8 GiB data, 9.9 GiB used, 705 GiB / 715 GiB avail; 853 B/s rd, 67 MiB/s wr, 1.08k op/s; 4146/139419 objects degraded (2.974%); 6454/139419 objects misplaced (4.629%); 11 MiB/s, 173 objects/s recovering 2024-09-18T08:08:32.983 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:08:32.984 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:08:32.984 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0' 2024-09-18T08:08:33.060 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid 779349ca-7593-11ef-bceb-c7b262605968 -- ceph osd reweight 1 1 2024-09-18T08:08:33.085 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6 2024-09-18T08:08:33.120 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:08:33.121 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:08:33.121 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6' 2024-09-18T08:08:33.163 INFO:tasks.radosbench.radosbench.0.smithi007.stdout: 35 16 52130 52114 93.0327 79.0625 0.0245336 0.0107423 2024-09-18T08:08:33.222 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3 2024-09-18T08:08:33.276 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:08:33.277 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:08:33.277 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3' 2024-09-18T08:08:33.378 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6 2024-09-18T08:08:33.417 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:08:33.418 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:08:33.418 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6' 2024-09-18T08:08:33.519 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1 2024-09-18T08:08:33.560 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:08:33.561 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:08:33.562 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1' 2024-09-18T08:08:33.662 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7 2024-09-18T08:08:33.700 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:08:33.701 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:08:33.701 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7' 2024-09-18T08:08:33.769 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/779349ca-7593-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T08:08:33.773 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid 779349ca-7593-11ef-bceb-c7b262605968 -- ceph daemon osd.1 dump_ops_in_flight 2024-09-18T08:08:33.802 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6 2024-09-18T08:08:33.837 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:08:33.837 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:08:33.837 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6' 2024-09-18T08:08:33.938 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2 2024-09-18T08:08:33.998 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:08:33.999 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:08:33.999 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2' 2024-09-18T08:08:34.003 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid 779349ca-7593-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-18T08:08:34.101 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7 2024-09-18T08:08:34.131 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:08:34.132 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:08:34.132 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7' 2024-09-18T08:08:34.167 INFO:tasks.radosbench.radosbench.0.smithi007.stdout: 36 16 53320 53304 92.5076 74.375 0.0177459 0.0108002 2024-09-18T08:08:34.231 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:08:34 smithi007 ceph-mon[29459]: from='mgr.24470 172.21.15.7:0/2594390261' entity='mgr.y' 2024-09-18T08:08:34.231 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:08:34 smithi007 ceph-mon[29459]: from='mgr.24470 172.21.15.7:0/2594390261' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-18T08:08:34.231 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:08:34 smithi007 ceph-mon[29459]: pgmap v274: 158 pgs: 1 active+recovery_wait+remapped, 1 active+recovering+undersized+remapped, 2 active+recovering+undersized+degraded+remapped, 5 active+clean+scrubbing, 2 active+recovery_wait+undersized+degraded+remapped, 147 active+clean; 2.8 GiB data, 10 GiB used, 705 GiB / 715 GiB avail; 775 B/s rd, 61 MiB/s wr, 978 op/s; 4146/139419 objects degraded (2.974%); 6454/139419 objects misplaced (4.629%); 9.8 MiB/s, 157 objects/s recovering 2024-09-18T08:08:34.231 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:08:34 smithi007 ceph-mon[39249]: from='mgr.24470 172.21.15.7:0/2594390261' entity='mgr.y' 2024-09-18T08:08:34.231 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:08:34 smithi007 ceph-mon[39249]: from='mgr.24470 172.21.15.7:0/2594390261' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-18T08:08:34.231 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:08:34 smithi007 ceph-mon[39249]: pgmap v274: 158 pgs: 1 active+recovery_wait+remapped, 1 active+recovering+undersized+remapped, 2 active+recovering+undersized+degraded+remapped, 5 active+clean+scrubbing, 2 active+recovery_wait+undersized+degraded+remapped, 147 active+clean; 2.8 GiB data, 10 GiB used, 705 GiB / 715 GiB avail; 775 B/s rd, 61 MiB/s wr, 978 op/s; 4146/139419 objects degraded (2.974%); 6454/139419 objects misplaced (4.629%); 9.8 MiB/s, 157 objects/s recovering 2024-09-18T08:08:34.233 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4 2024-09-18T08:08:34.266 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:08:34.267 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:08:34.268 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4' 2024-09-18T08:08:34.368 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6 2024-09-18T08:08:34.403 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/779349ca-7593-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T08:08:34.404 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:08:34.405 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:08:34.405 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6' 2024-09-18T08:08:34.406 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:08:34 smithi050 ceph-mon[32371]: from='mgr.24470 172.21.15.7:0/2594390261' entity='mgr.y' 2024-09-18T08:08:34.406 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:08:34 smithi050 ceph-mon[32371]: from='mgr.24470 172.21.15.7:0/2594390261' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-18T08:08:34.406 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:08:34 smithi050 ceph-mon[32371]: pgmap v274: 158 pgs: 1 active+recovery_wait+remapped, 1 active+recovering+undersized+remapped, 2 active+recovering+undersized+degraded+remapped, 5 active+clean+scrubbing, 2 active+recovery_wait+undersized+degraded+remapped, 147 active+clean; 2.8 GiB data, 10 GiB used, 705 GiB / 715 GiB avail; 775 B/s rd, 61 MiB/s wr, 978 op/s; 4146/139419 objects degraded (2.974%); 6454/139419 objects misplaced (4.629%); 9.8 MiB/s, 157 objects/s recovering 2024-09-18T08:08:34.507 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0 2024-09-18T08:08:34.565 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:08:34.565 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:08:34.566 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0' 2024-09-18T08:08:34.666 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1 2024-09-18T08:08:34.711 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:08:34.712 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:08:34.712 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1' 2024-09-18T08:08:34.813 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4 2024-09-18T08:08:34.846 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:08:34.847 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:08:34.847 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4' 2024-09-18T08:08:34.948 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7 2024-09-18T08:08:34.982 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:08:34.983 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:08:34.984 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7' 2024-09-18T08:08:35.024 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/779349ca-7593-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T08:08:35.081 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/779349ca-7593-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T08:08:35.084 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1 2024-09-18T08:08:35.135 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:08:35.136 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:08:35.137 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1' 2024-09-18T08:08:35.169 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:08:35 smithi007 ceph-mon[29459]: Health check update: Degraded data redundancy: 4146/139419 objects degraded (2.974%), 4 pgs degraded (PG_DEGRADED) 2024-09-18T08:08:35.169 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:08:35 smithi007 ceph-mon[29459]: osdmap e187: 8 total, 8 up, 8 in 2024-09-18T08:08:35.169 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:08:35 smithi007 ceph-mon[29459]: from='client.? 172.21.15.7:0/1549991745' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-18T08:08:35.169 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:08:35 smithi007 ceph-mon[29459]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-18T08:08:35.170 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:08:35 smithi007 ceph-mon[39249]: Health check update: Degraded data redundancy: 4146/139419 objects degraded (2.974%), 4 pgs degraded (PG_DEGRADED) 2024-09-18T08:08:35.170 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:08:35 smithi007 ceph-mon[39249]: osdmap e187: 8 total, 8 up, 8 in 2024-09-18T08:08:35.170 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:08:35 smithi007 ceph-mon[39249]: from='client.? 172.21.15.7:0/1549991745' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-18T08:08:35.170 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:08:35 smithi007 ceph-mon[39249]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-18T08:08:35.170 INFO:teuthology.orchestra.run.smithi007.stderr:nodeep-scrub is set 2024-09-18T08:08:35.170 INFO:tasks.radosbench.radosbench.0.smithi007.stdout: 37 16 54413 54397 91.8491 68.3125 0.0139137 0.0108801 2024-09-18T08:08:35.237 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7 2024-09-18T08:08:35.269 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:08:35.270 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:08:35.271 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7' 2024-09-18T08:08:35.371 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5 2024-09-18T08:08:35.404 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:08:35.404 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:08:35.405 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5' 2024-09-18T08:08:35.405 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:08:35 smithi050 ceph-mon[32371]: Health check update: Degraded data redundancy: 4146/139419 objects degraded (2.974%), 4 pgs degraded (PG_DEGRADED) 2024-09-18T08:08:35.406 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:08:35 smithi050 ceph-mon[32371]: osdmap e187: 8 total, 8 up, 8 in 2024-09-18T08:08:35.406 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:08:35 smithi050 ceph-mon[32371]: from='client.? 172.21.15.7:0/1549991745' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-18T08:08:35.406 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:08:35 smithi050 ceph-mon[32371]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-18T08:08:35.506 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6 2024-09-18T08:08:35.541 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:08:35.541 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:08:35.542 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6' 2024-09-18T08:08:35.643 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7 2024-09-18T08:08:35.679 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:08:35.679 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:08:35.679 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7' 2024-09-18T08:08:35.780 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4 2024-09-18T08:08:35.818 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:08:35.819 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:08:35.819 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4' 2024-09-18T08:08:35.920 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1 2024-09-18T08:08:35.963 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:08:35.964 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:08:35.964 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1' 2024-09-18T08:08:36.065 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2 2024-09-18T08:08:36.106 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:08:36.107 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:08:36.107 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2' 2024-09-18T08:08:36.167 INFO:tasks.radosbench.radosbench.0.smithi007.stdout: 38 16 55648 55632 91.4632 77.1875 0.00609038 0.0109264 2024-09-18T08:08:36.171 INFO:teuthology.orchestra.run.smithi007.stderr:reweighted osd.1 to 1 (10000) 2024-09-18T08:08:36.208 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4 2024-09-18T08:08:36.230 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:08:36 smithi007 ceph-mon[29459]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-18T08:08:36.231 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:08:36 smithi007 ceph-mon[29459]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-18T08:08:36.231 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:08:36 smithi007 ceph-mon[29459]: osdmap e188: 8 total, 8 up, 8 in 2024-09-18T08:08:36.231 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:08:36 smithi007 ceph-mon[29459]: pgmap v277: 158 pgs: 1 active+recovery_wait+remapped, 1 active+recovering+undersized+remapped, 2 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 152 active+clean; 3.2 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 852 B/s rd, 65 MiB/s wr, 1.04k op/s; 4146/158130 objects degraded (2.622%); 5179/158130 objects misplaced (3.275%); 13 MiB/s, 212 objects/s recovering 2024-09-18T08:08:36.231 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:08:36 smithi007 ceph-mon[29459]: from='client.? 172.21.15.7:0/2705137002' entity='client.admin' cmd={"prefix": "osd reweight", "id": 1, "weight": 1.0} : dispatch 2024-09-18T08:08:36.231 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:08:36 smithi007 ceph-mon[29459]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 1, "weight": 1.0} : dispatch 2024-09-18T08:08:36.231 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:08:36 smithi007 ceph-mon[39249]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-18T08:08:36.231 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:08:36 smithi007 ceph-mon[39249]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-18T08:08:36.232 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:08:36 smithi007 ceph-mon[39249]: osdmap e188: 8 total, 8 up, 8 in 2024-09-18T08:08:36.232 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:08:36 smithi007 ceph-mon[39249]: pgmap v277: 158 pgs: 1 active+recovery_wait+remapped, 1 active+recovering+undersized+remapped, 2 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 152 active+clean; 3.2 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 852 B/s rd, 65 MiB/s wr, 1.04k op/s; 4146/158130 objects degraded (2.622%); 5179/158130 objects misplaced (3.275%); 13 MiB/s, 212 objects/s recovering 2024-09-18T08:08:36.232 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:08:36 smithi007 ceph-mon[39249]: from='client.? 172.21.15.7:0/2705137002' entity='client.admin' cmd={"prefix": "osd reweight", "id": 1, "weight": 1.0} : dispatch 2024-09-18T08:08:36.232 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:08:36 smithi007 ceph-mon[39249]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 1, "weight": 1.0} : dispatch 2024-09-18T08:08:36.240 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:08:36.240 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:08:36.241 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4' 2024-09-18T08:08:36.342 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7 2024-09-18T08:08:36.365 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:08:36.366 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:08:36.366 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7' 2024-09-18T08:08:36.406 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:08:36 smithi050 ceph-mon[32371]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-18T08:08:36.406 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:08:36 smithi050 ceph-mon[32371]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-18T08:08:36.406 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:08:36 smithi050 ceph-mon[32371]: osdmap e188: 8 total, 8 up, 8 in 2024-09-18T08:08:36.406 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:08:36 smithi050 ceph-mon[32371]: pgmap v277: 158 pgs: 1 active+recovery_wait+remapped, 1 active+recovering+undersized+remapped, 2 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 152 active+clean; 3.2 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 852 B/s rd, 65 MiB/s wr, 1.04k op/s; 4146/158130 objects degraded (2.622%); 5179/158130 objects misplaced (3.275%); 13 MiB/s, 212 objects/s recovering 2024-09-18T08:08:36.406 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:08:36 smithi050 ceph-mon[32371]: from='client.? 172.21.15.7:0/2705137002' entity='client.admin' cmd={"prefix": "osd reweight", "id": 1, "weight": 1.0} : dispatch 2024-09-18T08:08:36.406 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:08:36 smithi050 ceph-mon[32371]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 1, "weight": 1.0} : dispatch 2024-09-18T08:08:36.467 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5 2024-09-18T08:08:36.490 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:08:36.491 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:08:36.491 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5' 2024-09-18T08:08:36.592 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5 2024-09-18T08:08:36.614 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:08:36.615 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:08:36.616 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5' 2024-09-18T08:08:36.716 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4 2024-09-18T08:08:36.740 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:08:36.740 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:08:36.741 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4' 2024-09-18T08:08:36.841 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3 2024-09-18T08:08:36.868 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:08:36.869 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:08:36.870 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3' 2024-09-18T08:08:36.970 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7 2024-09-18T08:08:36.980 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Sep 18 08:08:36 smithi007 ceph-779349ca-7593-11ef-bceb-c7b262605968-mgr-y[29688]: ::ffff:172.21.15.50 - - [18/Sep/2024:08:08:36] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-09-18T08:08:36.994 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:08:36.994 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:08:36.995 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7' 2024-09-18T08:08:37.095 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0 2024-09-18T08:08:37.122 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:08:37.123 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:08:37.123 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0' 2024-09-18T08:08:37.168 INFO:tasks.radosbench.radosbench.0.smithi007.stdout: 39 16 55790 55774 89.3462 8.875 0.00295943 0.0109286 2024-09-18T08:08:37.224 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7 2024-09-18T08:08:37.260 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:08:37.260 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:08:37.261 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7' 2024-09-18T08:08:37.362 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0 2024-09-18T08:08:37.400 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:08:37.401 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:08:37.402 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0' 2024-09-18T08:08:37.480 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:08:37 smithi007 ceph-mon[29459]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]': finished 2024-09-18T08:08:37.480 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:08:37 smithi007 ceph-mon[29459]: osdmap e189: 8 total, 8 up, 8 in 2024-09-18T08:08:37.481 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:08:37 smithi007 ceph-mon[39249]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]': finished 2024-09-18T08:08:37.481 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:08:37 smithi007 ceph-mon[39249]: osdmap e189: 8 total, 8 up, 8 in 2024-09-18T08:08:37.503 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7 2024-09-18T08:08:37.536 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:08:37.537 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:08:37.537 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7' 2024-09-18T08:08:37.638 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1 2024-09-18T08:08:37.656 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:08:37 smithi050 ceph-mon[32371]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]': finished 2024-09-18T08:08:37.656 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:08:37 smithi050 ceph-mon[32371]: osdmap e189: 8 total, 8 up, 8 in 2024-09-18T08:08:37.687 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:08:37.688 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:08:37.688 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1' 2024-09-18T08:08:37.789 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1 2024-09-18T08:08:37.830 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:08:37.830 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:08:37.831 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1' 2024-09-18T08:08:37.931 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2 2024-09-18T08:08:37.954 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:08:37.955 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:08:37.955 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2' 2024-09-18T08:08:38.056 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5 2024-09-18T08:08:38.088 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:08:38.089 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:08:38.089 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5' 2024-09-18T08:08:38.168 INFO:tasks.radosbench.radosbench.0.smithi007.stdout:2024-09-18T08:08:38.167051+0000 min lat: 0.00120772 max lat: 1.82134 avg lat: 0.0112388 2024-09-18T08:08:38.168 INFO:tasks.radosbench.radosbench.0.smithi007.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-09-18T08:08:38.168 INFO:tasks.radosbench.radosbench.0.smithi007.stdout: 40 16 56952 56936 88.9281 72.625 0.0128292 0.0112388 2024-09-18T08:08:38.190 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4 2024-09-18T08:08:38.224 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:08:38.225 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:08:38.226 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4' 2024-09-18T08:08:38.230 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:08:38 smithi007 ceph-mon[29459]: osdmap e190: 8 total, 8 up, 8 in 2024-09-18T08:08:38.230 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:08:38 smithi007 ceph-mon[29459]: pgmap v280: 158 pgs: 1 active+recovery_wait+remapped, 1 active+recovering+undersized+remapped, 2 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 152 active+clean; 3.2 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 1.2 KiB/s rd, 97 MiB/s wr, 1.56k op/s; 4146/158130 objects degraded (2.622%); 5179/158130 objects misplaced (3.275%); 20 MiB/s, 318 objects/s recovering 2024-09-18T08:08:38.230 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:08:38 smithi007 ceph-mon[29459]: from='client.? 172.21.15.7:0/421373222' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-18T08:08:38.230 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:08:38 smithi007 ceph-mon[39249]: osdmap e190: 8 total, 8 up, 8 in 2024-09-18T08:08:38.231 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:08:38 smithi007 ceph-mon[39249]: pgmap v280: 158 pgs: 1 active+recovery_wait+remapped, 1 active+recovering+undersized+remapped, 2 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 152 active+clean; 3.2 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 1.2 KiB/s rd, 97 MiB/s wr, 1.56k op/s; 4146/158130 objects degraded (2.622%); 5179/158130 objects misplaced (3.275%); 20 MiB/s, 318 objects/s recovering 2024-09-18T08:08:38.231 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:08:38 smithi007 ceph-mon[39249]: from='client.? 172.21.15.7:0/421373222' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-18T08:08:38.326 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0 2024-09-18T08:08:38.368 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:08:38.368 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:08:38.369 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0' 2024-09-18T08:08:38.470 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7 2024-09-18T08:08:38.504 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:08:38.504 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:08:38.505 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7' 2024-09-18T08:08:38.576 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid 779349ca-7593-11ef-bceb-c7b262605968 -- ceph osd unset noscrub 2024-09-18T08:08:38.606 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6 2024-09-18T08:08:38.642 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:08:38.643 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:08:38.643 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6' 2024-09-18T08:08:38.655 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:08:38 smithi050 ceph-mon[32371]: osdmap e190: 8 total, 8 up, 8 in 2024-09-18T08:08:38.655 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:08:38 smithi050 ceph-mon[32371]: pgmap v280: 158 pgs: 1 active+recovery_wait+remapped, 1 active+recovering+undersized+remapped, 2 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 152 active+clean; 3.2 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 1.2 KiB/s rd, 97 MiB/s wr, 1.56k op/s; 4146/158130 objects degraded (2.622%); 5179/158130 objects misplaced (3.275%); 20 MiB/s, 318 objects/s recovering 2024-09-18T08:08:38.656 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:08:38 smithi050 ceph-mon[32371]: from='client.? 172.21.15.7:0/421373222' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-18T08:08:38.744 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3 2024-09-18T08:08:38.790 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:08:38.791 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:08:38.791 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3' 2024-09-18T08:08:38.843 INFO:teuthology.orchestra.run.smithi007.stdout:osd.0: {} 2024-09-18T08:08:38.843 INFO:teuthology.orchestra.run.smithi007.stdout:osd.1: {} 2024-09-18T08:08:38.843 INFO:teuthology.orchestra.run.smithi007.stdout:osd.2: {} 2024-09-18T08:08:38.843 INFO:teuthology.orchestra.run.smithi007.stdout:osd.3: {} 2024-09-18T08:08:38.843 INFO:teuthology.orchestra.run.smithi007.stdout:osd.4: {} 2024-09-18T08:08:38.843 INFO:teuthology.orchestra.run.smithi007.stdout:osd.5: {} 2024-09-18T08:08:38.843 INFO:teuthology.orchestra.run.smithi007.stdout:osd.6: {} 2024-09-18T08:08:38.843 INFO:teuthology.orchestra.run.smithi007.stdout:osd.7: {} 2024-09-18T08:08:38.844 INFO:teuthology.orchestra.run.smithi007.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-18T08:08:38.844 INFO:teuthology.orchestra.run.smithi007.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-18T08:08:38.844 INFO:teuthology.orchestra.run.smithi007.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-18T08:08:38.844 INFO:teuthology.orchestra.run.smithi007.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-18T08:08:38.844 INFO:teuthology.orchestra.run.smithi007.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-18T08:08:38.844 INFO:teuthology.orchestra.run.smithi007.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-18T08:08:38.844 INFO:teuthology.orchestra.run.smithi007.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-18T08:08:38.844 INFO:teuthology.orchestra.run.smithi007.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-18T08:08:38.892 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5 2024-09-18T08:08:38.930 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:08:38.931 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:08:38.931 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5' 2024-09-18T08:08:39.032 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4 2024-09-18T08:08:39.069 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:08:39.070 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:08:39.070 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4' 2024-09-18T08:08:39.168 INFO:tasks.radosbench.radosbench.0.smithi007.stdout: 41 16 58017 58001 88.3826 66.5625 0.0092892 0.0113082 2024-09-18T08:08:39.170 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4 2024-09-18T08:08:39.210 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:08:39.211 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:08:39.211 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4' 2024-09-18T08:08:39.312 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4 2024-09-18T08:08:39.334 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:08:39.335 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:08:39.336 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4' 2024-09-18T08:08:39.409 INFO:journalctl@ceph.iscsi.iscsi.a.smithi050.stdout:Sep 18 08:08:39 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-iscsi-iscsi-a[57782]: debug there is no tcmu-runner data available 2024-09-18T08:08:39.409 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:08:39 smithi050 ceph-mon[32371]: from='mgr.24470 172.21.15.7:0/2594390261' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-18T08:08:39.410 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:08:39 smithi050 ceph-mon[32371]: from='mgr.24470 172.21.15.7:0/2594390261' entity='mgr.y' 2024-09-18T08:08:39.410 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:08:39 smithi050 ceph-mon[32371]: from='mgr.24470 172.21.15.7:0/2594390261' entity='mgr.y' 2024-09-18T08:08:39.410 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:08:39 smithi050 ceph-mon[32371]: Health check update: Degraded data redundancy: 4146/158130 objects degraded (2.622%), 4 pgs degraded (PG_DEGRADED) 2024-09-18T08:08:39.410 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:08:39 smithi050 ceph-mon[32371]: osdmap e191: 8 total, 8 up, 8 in 2024-09-18T08:08:39.436 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0 2024-09-18T08:08:39.479 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:08:39.480 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:08:39 smithi007 ceph-mon[29459]: from='mgr.24470 172.21.15.7:0/2594390261' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-18T08:08:39.480 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:08:39 smithi007 ceph-mon[29459]: from='mgr.24470 172.21.15.7:0/2594390261' entity='mgr.y' 2024-09-18T08:08:39.480 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:08:39 smithi007 ceph-mon[29459]: from='mgr.24470 172.21.15.7:0/2594390261' entity='mgr.y' 2024-09-18T08:08:39.481 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:08:39 smithi007 ceph-mon[29459]: Health check update: Degraded data redundancy: 4146/158130 objects degraded (2.622%), 4 pgs degraded (PG_DEGRADED) 2024-09-18T08:08:39.481 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:08:39 smithi007 ceph-mon[29459]: osdmap e191: 8 total, 8 up, 8 in 2024-09-18T08:08:39.481 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:08:39.481 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0' 2024-09-18T08:08:39.482 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:08:39 smithi007 ceph-mon[39249]: from='mgr.24470 172.21.15.7:0/2594390261' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-18T08:08:39.482 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:08:39 smithi007 ceph-mon[39249]: from='mgr.24470 172.21.15.7:0/2594390261' entity='mgr.y' 2024-09-18T08:08:39.482 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:08:39 smithi007 ceph-mon[39249]: from='mgr.24470 172.21.15.7:0/2594390261' entity='mgr.y' 2024-09-18T08:08:39.482 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:08:39 smithi007 ceph-mon[39249]: Health check update: Degraded data redundancy: 4146/158130 objects degraded (2.622%), 4 pgs degraded (PG_DEGRADED) 2024-09-18T08:08:39.482 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:08:39 smithi007 ceph-mon[39249]: osdmap e191: 8 total, 8 up, 8 in 2024-09-18T08:08:39.583 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5 2024-09-18T08:08:39.609 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:08:39.609 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:08:39.610 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5' 2024-09-18T08:08:39.710 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1 2024-09-18T08:08:39.738 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:08:39.739 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:08:39.739 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1' 2024-09-18T08:08:39.840 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0 2024-09-18T08:08:39.884 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:08:39.885 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:08:39.885 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0' 2024-09-18T08:08:39.986 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3 2024-09-18T08:08:40.008 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:08:40.009 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:08:40.009 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3' 2024-09-18T08:08:40.110 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7 2024-09-18T08:08:40.146 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:08:40.147 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:08:40.147 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7' 2024-09-18T08:08:40.168 INFO:tasks.radosbench.radosbench.0.smithi007.stdout: 42 15 58757 58742 87.3811 46.3125 0.0156249 0.0114392 2024-09-18T08:08:40.247 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4 2024-09-18T08:08:40.283 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:08:40.283 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:08:40.284 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4' 2024-09-18T08:08:40.385 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1 2024-09-18T08:08:40.424 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:08:40.425 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:08:40.425 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1' 2024-09-18T08:08:40.480 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:08:40 smithi007 ceph-mon[29459]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-18T08:08:40.480 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:08:40 smithi007 ceph-mon[29459]: pgmap v282: 158 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovery_wait+remapped, 2 active+recovering+undersized+remapped, 153 active+clean; 3.5 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 1.2 KiB/s rd, 76 MiB/s wr, 1.21k op/s; 2068/172713 objects degraded (1.197%); 5545/172713 objects misplaced (3.211%); 27 MiB/s, 425 objects/s recovering 2024-09-18T08:08:40.480 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:08:40 smithi007 ceph-mon[39249]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-18T08:08:40.480 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:08:40 smithi007 ceph-mon[39249]: pgmap v282: 158 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovery_wait+remapped, 2 active+recovering+undersized+remapped, 153 active+clean; 3.5 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 1.2 KiB/s rd, 76 MiB/s wr, 1.21k op/s; 2068/172713 objects degraded (1.197%); 5545/172713 objects misplaced (3.211%); 27 MiB/s, 425 objects/s recovering 2024-09-18T08:08:40.526 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5 2024-09-18T08:08:40.558 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:08:40.558 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:08:40.559 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5' 2024-09-18T08:08:40.656 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:08:40 smithi050 ceph-mon[32371]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-18T08:08:40.656 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:08:40 smithi050 ceph-mon[32371]: pgmap v282: 158 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovery_wait+remapped, 2 active+recovering+undersized+remapped, 153 active+clean; 3.5 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 1.2 KiB/s rd, 76 MiB/s wr, 1.21k op/s; 2068/172713 objects degraded (1.197%); 5545/172713 objects misplaced (3.211%); 27 MiB/s, 425 objects/s recovering 2024-09-18T08:08:40.659 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7 2024-09-18T08:08:40.681 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:08:40.682 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:08:40.682 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7' 2024-09-18T08:08:40.783 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3 2024-09-18T08:08:40.820 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:08:40.821 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:08:40.821 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3' 2024-09-18T08:08:40.922 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0 2024-09-18T08:08:40.961 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:08:40.961 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:08:40.962 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0' 2024-09-18T08:08:41.062 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3 2024-09-18T08:08:41.105 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:08:41.106 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:08:41.106 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3' 2024-09-18T08:08:41.168 INFO:tasks.radosbench.radosbench.0.smithi007.stdout: 43 16 59946 59930 87.0757 74.25 0.00754098 0.0114787 2024-09-18T08:08:41.207 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1 2024-09-18T08:08:41.235 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:08:41.235 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:08:41.235 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1' 2024-09-18T08:08:41.336 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1 2024-09-18T08:08:41.378 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:08:41.378 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:08:41.379 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1' 2024-09-18T08:08:41.479 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4 2024-09-18T08:08:41.503 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:08:41.504 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:08:41.504 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4' 2024-09-18T08:08:41.605 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2 2024-09-18T08:08:41.645 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:08:41.645 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:08:41.646 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2' 2024-09-18T08:08:41.746 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7 2024-09-18T08:08:41.770 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:08:41.771 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:08:41.771 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7' 2024-09-18T08:08:41.872 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4 2024-09-18T08:08:41.903 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:08:41.904 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:08:41.905 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4' 2024-09-18T08:08:42.006 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4 2024-09-18T08:08:42.041 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:08:42.042 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:08:42.042 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4' 2024-09-18T08:08:42.144 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5 2024-09-18T08:08:42.168 INFO:tasks.radosbench.radosbench.0.smithi007.stdout: 44 16 60906 60890 86.4604 60 0.0109165 0.0115608 2024-09-18T08:08:42.180 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:08:42.180 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:08:42.181 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5' 2024-09-18T08:08:42.282 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2 2024-09-18T08:08:42.326 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:08:42.326 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:08:42.327 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2' 2024-09-18T08:08:42.428 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7 2024-09-18T08:08:42.456 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:08:42.457 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:08:42.457 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7' 2024-09-18T08:08:42.457 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:08:42 smithi050 ceph-mon[32371]: pgmap v283: 158 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovery_wait+remapped, 2 active+recovering+undersized+remapped, 153 active+clean; 3.5 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 853 B/s rd, 51 MiB/s wr, 810 op/s; 2068/172713 objects degraded (1.197%); 5545/172713 objects misplaced (3.211%); 18 MiB/s, 285 objects/s recovering 2024-09-18T08:08:42.480 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:08:42 smithi007 ceph-mon[29459]: pgmap v283: 158 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovery_wait+remapped, 2 active+recovering+undersized+remapped, 153 active+clean; 3.5 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 853 B/s rd, 51 MiB/s wr, 810 op/s; 2068/172713 objects degraded (1.197%); 5545/172713 objects misplaced (3.211%); 18 MiB/s, 285 objects/s recovering 2024-09-18T08:08:42.481 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:08:42 smithi007 ceph-mon[39249]: pgmap v283: 158 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovery_wait+remapped, 2 active+recovering+undersized+remapped, 153 active+clean; 3.5 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 853 B/s rd, 51 MiB/s wr, 810 op/s; 2068/172713 objects degraded (1.197%); 5545/172713 objects misplaced (3.211%); 18 MiB/s, 285 objects/s recovering 2024-09-18T08:08:42.558 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1 2024-09-18T08:08:42.596 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:08:42.597 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:08:42.597 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1' 2024-09-18T08:08:42.698 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2 2024-09-18T08:08:42.737 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:08:42.738 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:08:42.738 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2' 2024-09-18T08:08:42.839 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5 2024-09-18T08:08:42.876 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:08:42.877 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:08:42.877 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5' 2024-09-18T08:08:42.978 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2 2024-09-18T08:08:43.018 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:08:43.019 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:08:43.019 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2' 2024-09-18T08:08:43.120 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2 2024-09-18T08:08:43.161 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:08:43.162 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:08:43.162 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2' 2024-09-18T08:08:43.168 INFO:tasks.radosbench.radosbench.0.smithi007.stdout: 45 16 62042 62026 86.1167 71 0.00532893 0.0116066 2024-09-18T08:08:43.263 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0 2024-09-18T08:08:43.308 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:08:43.309 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:08:43.309 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0' 2024-09-18T08:08:43.410 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5 2024-09-18T08:08:43.448 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:08:43.449 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:08:43.449 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5' 2024-09-18T08:08:43.550 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7 2024-09-18T08:08:43.555 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid 779349ca-7593-11ef-bceb-c7b262605968 -- ceph osd reweight 2 1 2024-09-18T08:08:43.590 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:08:43.591 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:08:43.591 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7' 2024-09-18T08:08:43.692 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4 2024-09-18T08:08:43.724 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:08:43.724 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:08:43.725 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4' 2024-09-18T08:08:43.825 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3 2024-09-18T08:08:43.937 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:08:43.938 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:08:43.938 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3' 2024-09-18T08:08:44.039 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4 2024-09-18T08:08:44.071 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:08:44.071 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:08:44.072 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4' 2024-09-18T08:08:44.095 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/779349ca-7593-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T08:08:44.166 INFO:teuthology.orchestra.run.smithi007.stdout:{ 2024-09-18T08:08:44.166 INFO:teuthology.orchestra.run.smithi007.stdout: "ops": [ 2024-09-18T08:08:44.166 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:08:44.166 INFO:teuthology.orchestra.run.smithi007.stdout: "description": "osd_op(client.15216.0:63034 8.a 8:5985c87f:::benchmark_data_smithi007_101583_object63033:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536 in=65536b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e191)", 2024-09-18T08:08:44.166 INFO:teuthology.orchestra.run.smithi007.stdout: "initiated_at": "2024-09-18T08:08:44.150255+0000", 2024-09-18T08:08:44.166 INFO:teuthology.orchestra.run.smithi007.stdout: "age": 0.0029904480000000002, 2024-09-18T08:08:44.167 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.0030262449999999999, 2024-09-18T08:08:44.167 INFO:teuthology.orchestra.run.smithi007.stdout: "continuous": false, 2024-09-18T08:08:44.167 INFO:teuthology.orchestra.run.smithi007.stdout: "type_data": { 2024-09-18T08:08:44.167 INFO:teuthology.orchestra.run.smithi007.stdout: "flag_point": "waiting for sub ops", 2024-09-18T08:08:44.167 INFO:teuthology.orchestra.run.smithi007.stdout: "client_info": { 2024-09-18T08:08:44.167 INFO:teuthology.orchestra.run.smithi007.stdout: "client": "client.15216", 2024-09-18T08:08:44.167 INFO:teuthology.orchestra.run.smithi007.stdout: "client_addr": "172.21.15.7:0/2596041177", 2024-09-18T08:08:44.167 INFO:teuthology.orchestra.run.smithi007.stdout: "tid": 63034 2024-09-18T08:08:44.167 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:08:44.167 INFO:teuthology.orchestra.run.smithi007.stdout: "events": [ 2024-09-18T08:08:44.167 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:08:44.167 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "initiated", 2024-09-18T08:08:44.168 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:44.150255+0000", 2024-09-18T08:08:44.168 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-09-18T08:08:44.168 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:08:44.168 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:08:44.168 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "header_read", 2024-09-18T08:08:44.168 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:44.150255+0000", 2024-09-18T08:08:44.168 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-09-18T08:08:44.168 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:08:44.168 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:08:44.168 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "throttled", 2024-09-18T08:08:44.168 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:44.150257+0000", 2024-09-18T08:08:44.168 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.4750000000000001e-06 2024-09-18T08:08:44.168 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:08:44.169 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:08:44.169 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "all_read", 2024-09-18T08:08:44.169 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:44.150278+0000", 2024-09-18T08:08:44.169 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 2.1041000000000001e-05 2024-09-18T08:08:44.169 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:08:44.169 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:08:44.169 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "dispatched", 2024-09-18T08:08:44.169 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:44.150288+0000", 2024-09-18T08:08:44.169 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.0524e-05 2024-09-18T08:08:44.169 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:08:44.169 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:08:44.169 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "queued_for_pg", 2024-09-18T08:08:44.169 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:44.150296+0000", 2024-09-18T08:08:44.170 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 7.1049999999999997e-06 2024-09-18T08:08:44.170 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:08:44.170 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:08:44.170 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-09-18T08:08:44.170 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:44.150402+0000", 2024-09-18T08:08:44.170 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.00010642899999999999 2024-09-18T08:08:44.170 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:08:44.170 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:08:44.170 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "started", 2024-09-18T08:08:44.170 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:44.150514+0000", 2024-09-18T08:08:44.170 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.000111654 2024-09-18T08:08:44.170 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:08:44.171 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:08:44.171 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "waiting for subops from 2,5", 2024-09-18T08:08:44.171 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:44.150634+0000", 2024-09-18T08:08:44.171 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.000119906 2024-09-18T08:08:44.171 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-09-18T08:08:44.171 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-09-18T08:08:44.171 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-09-18T08:08:44.171 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:08:44.171 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:08:44.171 INFO:teuthology.orchestra.run.smithi007.stdout: "description": "osd_op(client.15216.0:63036 8.a 8.92d5821a (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e191)", 2024-09-18T08:08:44.171 INFO:teuthology.orchestra.run.smithi007.stdout: "initiated_at": "2024-09-18T08:08:44.150317+0000", 2024-09-18T08:08:44.171 INFO:teuthology.orchestra.run.smithi007.stdout: "age": 0.0029291120000000002, 2024-09-18T08:08:44.171 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.0030453110000000002, 2024-09-18T08:08:44.171 INFO:teuthology.orchestra.run.smithi007.stdout: "continuous": false, 2024-09-18T08:08:44.172 INFO:teuthology.orchestra.run.smithi007.stdout: "type_data": { 2024-09-18T08:08:44.172 INFO:teuthology.orchestra.run.smithi007.stdout: "flag_point": "queued for pg", 2024-09-18T08:08:44.172 INFO:teuthology.orchestra.run.smithi007.stdout: "client_info": { 2024-09-18T08:08:44.172 INFO:teuthology.orchestra.run.smithi007.stdout: "client": "client.15216", 2024-09-18T08:08:44.172 INFO:teuthology.orchestra.run.smithi007.stdout: "client_addr": "172.21.15.7:0/2596041177", 2024-09-18T08:08:44.172 INFO:teuthology.orchestra.run.smithi007.stdout: "tid": 63036 2024-09-18T08:08:44.172 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:08:44.172 INFO:teuthology.orchestra.run.smithi007.stdout: "events": [ 2024-09-18T08:08:44.172 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:08:44.172 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "initiated", 2024-09-18T08:08:44.172 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:44.150317+0000", 2024-09-18T08:08:44.172 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-09-18T08:08:44.172 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:08:44.173 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:08:44.173 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "header_read", 2024-09-18T08:08:44.173 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:44.150317+0000", 2024-09-18T08:08:44.173 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-09-18T08:08:44.173 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:08:44.173 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:08:44.173 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "throttled", 2024-09-18T08:08:44.173 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:44.150317+0000", 2024-09-18T08:08:44.173 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 5.7000000000000005e-07 2024-09-18T08:08:44.173 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:08:44.173 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:08:44.173 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "all_read", 2024-09-18T08:08:44.173 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:44.150338+0000", 2024-09-18T08:08:44.173 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 2.0265e-05 2024-09-18T08:08:44.174 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:08:44.174 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:08:44.174 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "dispatched", 2024-09-18T08:08:44.174 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:44.150349+0000", 2024-09-18T08:08:44.174 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.1525999999999999e-05 2024-09-18T08:08:44.174 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:08:44.174 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:08:44.174 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "queued_for_pg", 2024-09-18T08:08:44.174 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:44.150355+0000", 2024-09-18T08:08:44.174 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 6.229e-06 2024-09-18T08:08:44.174 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-09-18T08:08:44.174 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-09-18T08:08:44.174 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-09-18T08:08:44.174 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:08:44.175 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:08:44.175 INFO:teuthology.orchestra.run.smithi007.stdout: "description": "osd_repop(client.15216.0:63029 8.f e191/181 8:fa2e0267:::benchmark_data_smithi007_101583_object63028:head v 191'4021, mlcod=177'1608)", 2024-09-18T08:08:44.175 INFO:teuthology.orchestra.run.smithi007.stdout: "initiated_at": "2024-09-18T08:08:44.151631+0000", 2024-09-18T08:08:44.175 INFO:teuthology.orchestra.run.smithi007.stdout: "age": 0.001614648, 2024-09-18T08:08:44.175 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.0017891039999999999, 2024-09-18T08:08:44.175 INFO:teuthology.orchestra.run.smithi007.stdout: "continuous": false, 2024-09-18T08:08:44.175 INFO:teuthology.orchestra.run.smithi007.stdout: "type_data": { 2024-09-18T08:08:44.175 INFO:teuthology.orchestra.run.smithi007.stdout: "flag_point": "started", 2024-09-18T08:08:44.175 INFO:teuthology.orchestra.run.smithi007.stdout: "events": [ 2024-09-18T08:08:44.175 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:08:44.176 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "initiated", 2024-09-18T08:08:44.176 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:44.151631+0000", 2024-09-18T08:08:44.176 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-09-18T08:08:44.176 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:08:44.176 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:08:44.176 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "header_read", 2024-09-18T08:08:44.176 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:44.151631+0000", 2024-09-18T08:08:44.176 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-09-18T08:08:44.176 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:08:44.176 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:08:44.176 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "throttled", 2024-09-18T08:08:44.176 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:44.151632+0000", 2024-09-18T08:08:44.177 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 8.8100000000000001e-07 2024-09-18T08:08:44.177 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:08:44.177 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:08:44.177 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "all_read", 2024-09-18T08:08:44.177 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:44.151657+0000", 2024-09-18T08:08:44.177 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 2.4406e-05 2024-09-18T08:08:44.177 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:08:44.177 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:08:44.177 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "dispatched", 2024-09-18T08:08:44.177 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:44.151670+0000", 2024-09-18T08:08:44.177 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.3464e-05 2024-09-18T08:08:44.177 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:08:44.178 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:08:44.178 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "queued_for_pg", 2024-09-18T08:08:44.178 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:44.151677+0000", 2024-09-18T08:08:44.178 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 6.7050000000000002e-06 2024-09-18T08:08:44.178 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:08:44.178 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:08:44.178 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-09-18T08:08:44.178 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:44.151892+0000", 2024-09-18T08:08:44.178 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.00021484 2024-09-18T08:08:44.178 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:08:44.178 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:08:44.178 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "started", 2024-09-18T08:08:44.178 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:44.151911+0000", 2024-09-18T08:08:44.178 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.9562e-05 2024-09-18T08:08:44.179 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-09-18T08:08:44.179 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-09-18T08:08:44.179 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-09-18T08:08:44.179 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-09-18T08:08:44.179 INFO:teuthology.orchestra.run.smithi007.stdout: ], 2024-09-18T08:08:44.179 INFO:teuthology.orchestra.run.smithi007.stdout: "num_ops": 3 2024-09-18T08:08:44.179 INFO:teuthology.orchestra.run.smithi007.stdout:} 2024-09-18T08:08:44.179 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2 2024-09-18T08:08:44.180 INFO:tasks.radosbench.radosbench.0.smithi007.stdout: 46 16 63047 63031 85.6101 62.8125 0.0195097 0.0116736 2024-09-18T08:08:44.245 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:08:44.246 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:08:44.246 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2' 2024-09-18T08:08:44.347 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6 2024-09-18T08:08:44.380 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:08:44.381 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:08:44.381 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6' 2024-09-18T08:08:44.480 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:08:44 smithi007 ceph-mon[29459]: pgmap v284: 158 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovery_wait+remapped, 2 active+recovering+undersized+remapped, 153 active+clean; 3.5 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 729 B/s rd, 43 MiB/s wr, 693 op/s; 2068/172713 objects degraded (1.197%); 5545/172713 objects misplaced (3.211%); 15 MiB/s, 243 objects/s recovering 2024-09-18T08:08:44.481 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:08:44 smithi007 ceph-mon[29459]: from='mgr.24470 172.21.15.7:0/2594390261' entity='mgr.y' 2024-09-18T08:08:44.481 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:08:44 smithi007 ceph-mon[29459]: from='mgr.24470 172.21.15.7:0/2594390261' entity='mgr.y' 2024-09-18T08:08:44.481 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:08:44 smithi007 ceph-mon[29459]: purged_snaps scrub starts 2024-09-18T08:08:44.481 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:08:44 smithi007 ceph-mon[29459]: purged_snaps scrub ok 2024-09-18T08:08:44.481 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:08:44 smithi007 ceph-mon[29459]: Health check update: Degraded data redundancy: 2068/172713 objects degraded (1.197%), 2 pgs degraded (PG_DEGRADED) 2024-09-18T08:08:44.481 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:08:44 smithi007 ceph-mon[39249]: pgmap v284: 158 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovery_wait+remapped, 2 active+recovering+undersized+remapped, 153 active+clean; 3.5 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 729 B/s rd, 43 MiB/s wr, 693 op/s; 2068/172713 objects degraded (1.197%); 5545/172713 objects misplaced (3.211%); 15 MiB/s, 243 objects/s recovering 2024-09-18T08:08:44.481 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:08:44 smithi007 ceph-mon[39249]: from='mgr.24470 172.21.15.7:0/2594390261' entity='mgr.y' 2024-09-18T08:08:44.481 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:08:44 smithi007 ceph-mon[39249]: from='mgr.24470 172.21.15.7:0/2594390261' entity='mgr.y' 2024-09-18T08:08:44.481 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:08:44 smithi007 ceph-mon[39249]: purged_snaps scrub starts 2024-09-18T08:08:44.481 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:08:44 smithi007 ceph-mon[39249]: purged_snaps scrub ok 2024-09-18T08:08:44.482 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:08:44 smithi007 ceph-mon[39249]: Health check update: Degraded data redundancy: 2068/172713 objects degraded (1.197%), 2 pgs degraded (PG_DEGRADED) 2024-09-18T08:08:44.482 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7 2024-09-18T08:08:44.511 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:08:44.512 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:08:44.512 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7' 2024-09-18T08:08:44.613 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1 2024-09-18T08:08:44.650 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:08:44.651 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:08:44.651 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1' 2024-09-18T08:08:44.656 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:08:44 smithi050 ceph-mon[32371]: pgmap v284: 158 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovery_wait+remapped, 2 active+recovering+undersized+remapped, 153 active+clean; 3.5 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 729 B/s rd, 43 MiB/s wr, 693 op/s; 2068/172713 objects degraded (1.197%); 5545/172713 objects misplaced (3.211%); 15 MiB/s, 243 objects/s recovering 2024-09-18T08:08:44.656 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:08:44 smithi050 ceph-mon[32371]: from='mgr.24470 172.21.15.7:0/2594390261' entity='mgr.y' 2024-09-18T08:08:44.656 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:08:44 smithi050 ceph-mon[32371]: from='mgr.24470 172.21.15.7:0/2594390261' entity='mgr.y' 2024-09-18T08:08:44.656 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:08:44 smithi050 ceph-mon[32371]: purged_snaps scrub starts 2024-09-18T08:08:44.656 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:08:44 smithi050 ceph-mon[32371]: purged_snaps scrub ok 2024-09-18T08:08:44.656 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:08:44 smithi050 ceph-mon[32371]: Health check update: Degraded data redundancy: 2068/172713 objects degraded (1.197%), 2 pgs degraded (PG_DEGRADED) 2024-09-18T08:08:44.752 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7 2024-09-18T08:08:44.789 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:08:44.789 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:08:44.790 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7' 2024-09-18T08:08:44.891 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3 2024-09-18T08:08:44.938 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:08:44.939 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:08:44.939 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3' 2024-09-18T08:08:45.040 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4 2024-09-18T08:08:45.069 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:08:45.069 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:08:45.070 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4' 2024-09-18T08:08:45.168 INFO:tasks.radosbench.radosbench.0.smithi007.stdout: 47 15 64099 64084 85.1889 65.8125 0.00786523 0.0117335 2024-09-18T08:08:45.170 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6 2024-09-18T08:08:45.201 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:08:45.202 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:08:45.202 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6' 2024-09-18T08:08:45.303 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3 2024-09-18T08:08:45.343 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:08:45.343 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:08:45.344 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3' 2024-09-18T08:08:45.444 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1 2024-09-18T08:08:45.480 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:08:45 smithi007 ceph-mon[29459]: from='mgr.24470 172.21.15.7:0/2594390261' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-18T08:08:45.480 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:08:45 smithi007 ceph-mon[29459]: from='mgr.24470 172.21.15.7:0/2594390261' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-18T08:08:45.480 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:08:45 smithi007 ceph-mon[29459]: from='mgr.24470 172.21.15.7:0/2594390261' entity='mgr.y' 2024-09-18T08:08:45.480 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:08:45 smithi007 ceph-mon[29459]: osdmap e192: 8 total, 8 up, 8 in 2024-09-18T08:08:45.481 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:08:45 smithi007 ceph-mon[39249]: from='mgr.24470 172.21.15.7:0/2594390261' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-18T08:08:45.481 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:08:45 smithi007 ceph-mon[39249]: from='mgr.24470 172.21.15.7:0/2594390261' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-18T08:08:45.481 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:08:45 smithi007 ceph-mon[39249]: from='mgr.24470 172.21.15.7:0/2594390261' entity='mgr.y' 2024-09-18T08:08:45.481 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:08:45 smithi007 ceph-mon[39249]: osdmap e192: 8 total, 8 up, 8 in 2024-09-18T08:08:45.482 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:08:45.482 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:08:45.483 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1' 2024-09-18T08:08:45.584 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1 2024-09-18T08:08:45.626 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:08:45.626 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:08:45.627 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1' 2024-09-18T08:08:45.656 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:08:45 smithi050 ceph-mon[32371]: from='mgr.24470 172.21.15.7:0/2594390261' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-18T08:08:45.656 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:08:45 smithi050 ceph-mon[32371]: from='mgr.24470 172.21.15.7:0/2594390261' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-18T08:08:45.656 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:08:45 smithi050 ceph-mon[32371]: from='mgr.24470 172.21.15.7:0/2594390261' entity='mgr.y' 2024-09-18T08:08:45.656 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:08:45 smithi050 ceph-mon[32371]: osdmap e192: 8 total, 8 up, 8 in 2024-09-18T08:08:45.727 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7 2024-09-18T08:08:45.762 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:08:45.763 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:08:45.763 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7' 2024-09-18T08:08:45.864 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4 2024-09-18T08:08:45.900 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:08:45.901 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:08:45.901 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4' 2024-09-18T08:08:46.002 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1 2024-09-18T08:08:46.048 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:08:46.049 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:08:46.049 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1' 2024-09-18T08:08:46.150 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2 2024-09-18T08:08:46.168 INFO:tasks.radosbench.radosbench.0.smithi007.stdout: 48 16 65318 65302 85 76.125 0.0162661 0.0117592 2024-09-18T08:08:46.193 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:08:46.194 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:08:46.194 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2' 2024-09-18T08:08:46.295 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7 2024-09-18T08:08:46.317 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:08:46.317 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:08:46.317 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7' 2024-09-18T08:08:46.418 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3 2024-09-18T08:08:46.453 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:08:46.454 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:08:46.454 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3' 2024-09-18T08:08:46.480 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:08:46 smithi007 ceph-mon[29459]: pgmap v286: 158 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovery_wait+remapped, 3 active+recovering+undersized+remapped, 153 active+clean; 3.8 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 1.2 KiB/s rd, 77 MiB/s wr, 1.24k op/s; 2067/187878 objects degraded (1.100%); 4637/187878 objects misplaced (2.468%); 20 MiB/s, 327 objects/s recovering 2024-09-18T08:08:46.481 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:08:46 smithi007 ceph-mon[39249]: pgmap v286: 158 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovery_wait+remapped, 3 active+recovering+undersized+remapped, 153 active+clean; 3.8 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 1.2 KiB/s rd, 77 MiB/s wr, 1.24k op/s; 2067/187878 objects degraded (1.100%); 4637/187878 objects misplaced (2.468%); 20 MiB/s, 327 objects/s recovering 2024-09-18T08:08:46.554 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5 2024-09-18T08:08:46.584 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:08:46.585 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:08:46.586 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5' 2024-09-18T08:08:46.590 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid 779349ca-7593-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-18T08:08:46.656 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:08:46 smithi050 ceph-mon[32371]: pgmap v286: 158 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovery_wait+remapped, 3 active+recovering+undersized+remapped, 153 active+clean; 3.8 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 1.2 KiB/s rd, 77 MiB/s wr, 1.24k op/s; 2067/187878 objects degraded (1.100%); 4637/187878 objects misplaced (2.468%); 20 MiB/s, 327 objects/s recovering 2024-09-18T08:08:46.686 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4 2024-09-18T08:08:46.708 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:08:46.709 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:08:46.709 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4' 2024-09-18T08:08:46.810 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0 2024-09-18T08:08:46.859 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:08:46.860 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:08:46.860 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0' 2024-09-18T08:08:46.961 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2 2024-09-18T08:08:46.980 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Sep 18 08:08:46 smithi007 ceph-779349ca-7593-11ef-bceb-c7b262605968-mgr-y[29688]: ::ffff:172.21.15.50 - - [18/Sep/2024:08:08:46] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-09-18T08:08:46.988 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:08:46.989 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:08:46.989 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2' 2024-09-18T08:08:47.090 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1 2024-09-18T08:08:47.135 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:08:47.136 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:08:47.137 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1' 2024-09-18T08:08:47.168 INFO:tasks.radosbench.radosbench.0.smithi007.stdout: 49 16 66157 66141 84.3355 52.4375 0.0188269 0.0118525 2024-09-18T08:08:47.237 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2 2024-09-18T08:08:47.261 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:08:47.262 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:08:47.262 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2' 2024-09-18T08:08:47.363 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3 2024-09-18T08:08:47.406 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:08:47.406 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:08:47.407 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3' 2024-09-18T08:08:47.477 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:08:47 smithi007 ceph-mon[29459]: purged_snaps scrub starts 2024-09-18T08:08:47.477 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:08:47 smithi007 ceph-mon[29459]: purged_snaps scrub ok 2024-09-18T08:08:47.477 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:08:47 smithi007 ceph-mon[29459]: osdmap e193: 8 total, 8 up, 8 in 2024-09-18T08:08:47.477 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:08:47 smithi007 ceph-mon[39249]: purged_snaps scrub starts 2024-09-18T08:08:47.478 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:08:47 smithi007 ceph-mon[39249]: purged_snaps scrub ok 2024-09-18T08:08:47.478 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:08:47 smithi007 ceph-mon[39249]: osdmap e193: 8 total, 8 up, 8 in 2024-09-18T08:08:47.479 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid 779349ca-7593-11ef-bceb-c7b262605968 -- ceph daemon osd.1 dump_blocked_ops 2024-09-18T08:08:47.507 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1 2024-09-18T08:08:47.575 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:08:47.576 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:08:47.576 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1' 2024-09-18T08:08:47.656 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:08:47 smithi050 ceph-mon[32371]: purged_snaps scrub starts 2024-09-18T08:08:47.656 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:08:47 smithi050 ceph-mon[32371]: purged_snaps scrub ok 2024-09-18T08:08:47.656 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:08:47 smithi050 ceph-mon[32371]: osdmap e193: 8 total, 8 up, 8 in 2024-09-18T08:08:47.677 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6 2024-09-18T08:08:47.712 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:08:47.713 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:08:47.713 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6' 2024-09-18T08:08:47.814 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7 2024-09-18T08:08:47.836 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:08:47.837 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:08:47.837 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7' 2024-09-18T08:08:47.938 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0 2024-09-18T08:08:47.965 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:08:47.966 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:08:47.966 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0' 2024-09-18T08:08:48.067 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5 2024-09-18T08:08:48.092 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:08:48.092 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:08:48.093 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5' 2024-09-18T08:08:48.169 INFO:tasks.radosbench.radosbench.0.smithi007.stdout: 50 16 66875 66859 83.5464 44.875 0.00308728 0.0118836 2024-09-18T08:08:48.193 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4 2024-09-18T08:08:48.218 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:08:48.218 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:08:48.219 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4' 2024-09-18T08:08:48.319 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4 2024-09-18T08:08:48.343 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:08:48.344 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:08:48.345 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4' 2024-09-18T08:08:48.445 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2 2024-09-18T08:08:48.471 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:08:48.472 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:08:48.472 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2' 2024-09-18T08:08:48.480 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:08:48 smithi007 ceph-mon[29459]: purged_snaps scrub starts 2024-09-18T08:08:48.480 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:08:48 smithi007 ceph-mon[29459]: purged_snaps scrub ok 2024-09-18T08:08:48.480 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:08:48 smithi007 ceph-mon[29459]: pgmap v288: 158 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovery_wait+remapped, 3 active+recovering+undersized+remapped, 153 active+clean; 3.8 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 639 B/s rd, 39 MiB/s wr, 632 op/s; 2067/187878 objects degraded (1.100%); 4637/187878 objects misplaced (2.468%); 7.1 MiB/s, 113 objects/s recovering 2024-09-18T08:08:48.480 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:08:48 smithi007 ceph-mon[29459]: osdmap e194: 8 total, 8 up, 8 in 2024-09-18T08:08:48.480 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:08:48 smithi007 ceph-mon[29459]: from='mgr.24470 172.21.15.7:0/2594390261' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-18T08:08:48.481 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:08:48 smithi007 ceph-mon[39249]: purged_snaps scrub starts 2024-09-18T08:08:48.481 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:08:48 smithi007 ceph-mon[39249]: purged_snaps scrub ok 2024-09-18T08:08:48.481 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:08:48 smithi007 ceph-mon[39249]: pgmap v288: 158 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovery_wait+remapped, 3 active+recovering+undersized+remapped, 153 active+clean; 3.8 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 639 B/s rd, 39 MiB/s wr, 632 op/s; 2067/187878 objects degraded (1.100%); 4637/187878 objects misplaced (2.468%); 7.1 MiB/s, 113 objects/s recovering 2024-09-18T08:08:48.481 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:08:48 smithi007 ceph-mon[39249]: osdmap e194: 8 total, 8 up, 8 in 2024-09-18T08:08:48.481 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:08:48 smithi007 ceph-mon[39249]: from='mgr.24470 172.21.15.7:0/2594390261' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-18T08:08:48.573 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3 2024-09-18T08:08:48.600 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:08:48.601 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:08:48.601 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3' 2024-09-18T08:08:48.656 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:08:48 smithi050 ceph-mon[32371]: purged_snaps scrub starts 2024-09-18T08:08:48.656 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:08:48 smithi050 ceph-mon[32371]: purged_snaps scrub ok 2024-09-18T08:08:48.656 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:08:48 smithi050 ceph-mon[32371]: pgmap v288: 158 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovery_wait+remapped, 3 active+recovering+undersized+remapped, 153 active+clean; 3.8 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 639 B/s rd, 39 MiB/s wr, 632 op/s; 2067/187878 objects degraded (1.100%); 4637/187878 objects misplaced (2.468%); 7.1 MiB/s, 113 objects/s recovering 2024-09-18T08:08:48.656 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:08:48 smithi050 ceph-mon[32371]: osdmap e194: 8 total, 8 up, 8 in 2024-09-18T08:08:48.656 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:08:48 smithi050 ceph-mon[32371]: from='mgr.24470 172.21.15.7:0/2594390261' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-18T08:08:48.704 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7 2024-09-18T08:08:48.735 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:08:48.735 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:08:48.736 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7' 2024-09-18T08:08:48.837 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5 2024-09-18T08:08:48.861 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:08:48.862 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:08:48.862 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5' 2024-09-18T08:08:48.963 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2 2024-09-18T08:08:48.989 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:08:48.990 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:08:48.990 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2' 2024-09-18T08:08:49.091 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4 2024-09-18T08:08:49.113 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:08:49.114 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:08:49.115 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4' 2024-09-18T08:08:49.169 INFO:tasks.radosbench.radosbench.0.smithi007.stdout: 51 16 67090 67074 82.172 13.4375 0.0033525 0.0120722 2024-09-18T08:08:49.215 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2 2024-09-18T08:08:49.241 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:08:49.241 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:08:49.242 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2' 2024-09-18T08:08:49.342 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0 2024-09-18T08:08:49.367 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:08:49.368 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:08:49.368 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0' 2024-09-18T08:08:49.406 INFO:journalctl@ceph.iscsi.iscsi.a.smithi050.stdout:Sep 18 08:08:49 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-iscsi-iscsi-a[57782]: debug there is no tcmu-runner data available 2024-09-18T08:08:49.469 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3 2024-09-18T08:08:49.493 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:08:49.494 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:08:49.494 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3' 2024-09-18T08:08:49.595 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5 2024-09-18T08:08:49.618 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:08:49.619 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:08:49.619 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5' 2024-09-18T08:08:49.720 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1 2024-09-18T08:08:49.765 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:08:49.766 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:08:49.766 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1' 2024-09-18T08:08:49.867 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7 2024-09-18T08:08:49.896 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:08:49.897 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:08:49.897 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7' 2024-09-18T08:08:49.980 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:08:49 smithi007 ceph-mon[29459]: osdmap e195: 8 total, 8 up, 8 in 2024-09-18T08:08:49.980 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:08:49 smithi007 ceph-mon[29459]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-18T08:08:49.981 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:08:49 smithi007 ceph-mon[29459]: Health check update: Degraded data redundancy: 2067/187878 objects degraded (1.100%), 1 pg degraded (PG_DEGRADED) 2024-09-18T08:08:49.981 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:08:49 smithi007 ceph-mon[29459]: pgmap v291: 158 pgs: 1 peering, 1 active+recovering+remapped, 2 active+recovering+undersized+remapped, 154 active+clean; 4.1 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 1.1 KiB/s rd, 62 MiB/s wr, 985 op/s; 5232/201315 objects misplaced (2.599%); 20 MiB/s, 323 objects/s recovering 2024-09-18T08:08:49.981 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:08:49 smithi007 ceph-mon[39249]: osdmap e195: 8 total, 8 up, 8 in 2024-09-18T08:08:50.018 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:08:49 smithi007 ceph-mon[39249]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-18T08:08:50.018 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:08:49 smithi007 ceph-mon[39249]: Health check update: Degraded data redundancy: 2067/187878 objects degraded (1.100%), 1 pg degraded (PG_DEGRADED) 2024-09-18T08:08:50.018 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:08:49 smithi007 ceph-mon[39249]: pgmap v291: 158 pgs: 1 peering, 1 active+recovering+remapped, 2 active+recovering+undersized+remapped, 154 active+clean; 4.1 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 1.1 KiB/s rd, 62 MiB/s wr, 985 op/s; 5232/201315 objects misplaced (2.599%); 20 MiB/s, 323 objects/s recovering 2024-09-18T08:08:50.019 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7 2024-09-18T08:08:50.041 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:08:50.041 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:08:50.042 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7' 2024-09-18T08:08:50.143 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0 2024-09-18T08:08:50.155 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:08:49 smithi050 ceph-mon[32371]: osdmap e195: 8 total, 8 up, 8 in 2024-09-18T08:08:50.156 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:08:49 smithi050 ceph-mon[32371]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-18T08:08:50.156 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:08:49 smithi050 ceph-mon[32371]: Health check update: Degraded data redundancy: 2067/187878 objects degraded (1.100%), 1 pg degraded (PG_DEGRADED) 2024-09-18T08:08:50.156 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:08:49 smithi050 ceph-mon[32371]: pgmap v291: 158 pgs: 1 peering, 1 active+recovering+remapped, 2 active+recovering+undersized+remapped, 154 active+clean; 4.1 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 1.1 KiB/s rd, 62 MiB/s wr, 985 op/s; 5232/201315 objects misplaced (2.599%); 20 MiB/s, 323 objects/s recovering 2024-09-18T08:08:50.169 INFO:tasks.radosbench.radosbench.0.smithi007.stdout: 52 16 67542 67526 81.1352 28.25 0.0326295 0.0123182 2024-09-18T08:08:50.183 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:08:50.184 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:08:50.184 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0' 2024-09-18T08:08:50.231 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/779349ca-7593-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T08:08:50.285 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3 2024-09-18T08:08:50.302 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/779349ca-7593-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T08:08:50.352 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:08:50.353 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:08:50.353 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3' 2024-09-18T08:08:50.454 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4 2024-09-18T08:08:50.479 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:08:50.479 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:08:50.480 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4' 2024-09-18T08:08:50.580 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5 2024-09-18T08:08:50.615 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:08:50.616 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:08:50.616 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5' 2024-09-18T08:08:50.680 INFO:teuthology.orchestra.run.smithi007.stderr:noscrub is unset 2024-09-18T08:08:50.717 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5 2024-09-18T08:08:50.749 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:08:50.750 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:08:50.750 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5' 2024-09-18T08:08:50.851 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0 2024-09-18T08:08:50.897 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:08:50.897 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:08:50.897 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0' 2024-09-18T08:08:50.980 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:08:50 smithi007 ceph-mon[29459]: osdmap e196: 8 total, 8 up, 8 in 2024-09-18T08:08:50.980 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:08:50 smithi007 ceph-mon[29459]: from='client.? 172.21.15.7:0/409959367' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-18T08:08:50.981 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:08:50 smithi007 ceph-mon[29459]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-18T08:08:50.981 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:08:50 smithi007 ceph-mon[29459]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 2067/187878 objects degraded (1.100%), 1 pg degraded) 2024-09-18T08:08:50.981 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:08:50 smithi007 ceph-mon[39249]: osdmap e196: 8 total, 8 up, 8 in 2024-09-18T08:08:50.981 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:08:50 smithi007 ceph-mon[39249]: from='client.? 172.21.15.7:0/409959367' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-18T08:08:50.981 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:08:50 smithi007 ceph-mon[39249]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-18T08:08:50.981 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:08:50 smithi007 ceph-mon[39249]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 2067/187878 objects degraded (1.100%), 1 pg degraded) 2024-09-18T08:08:50.999 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3 2024-09-18T08:08:51.039 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:08:51.039 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:08:51.040 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3' 2024-09-18T08:08:51.140 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0 2024-09-18T08:08:51.155 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:08:50 smithi050 ceph-mon[32371]: osdmap e196: 8 total, 8 up, 8 in 2024-09-18T08:08:51.156 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:08:50 smithi050 ceph-mon[32371]: from='client.? 172.21.15.7:0/409959367' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-18T08:08:51.156 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:08:50 smithi050 ceph-mon[32371]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-18T08:08:51.156 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:08:50 smithi050 ceph-mon[32371]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 2067/187878 objects degraded (1.100%), 1 pg degraded) 2024-09-18T08:08:51.172 INFO:tasks.radosbench.radosbench.0.smithi007.stdout: 53 16 68636 68620 80.8899 68.375 0.015934 0.0123558 2024-09-18T08:08:51.179 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:08:51.180 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:08:51.181 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0' 2024-09-18T08:08:51.282 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4 2024-09-18T08:08:51.311 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:08:51.312 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:08:51.312 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4' 2024-09-18T08:08:51.413 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2 2024-09-18T08:08:51.453 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:08:51.453 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:08:51.454 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2' 2024-09-18T08:08:51.554 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3 2024-09-18T08:08:51.596 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:08:51.597 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:08:51.597 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3' 2024-09-18T08:08:51.683 INFO:teuthology.orchestra.run.smithi007.stderr:reweighted osd.2 to 1 (10000) 2024-09-18T08:08:51.698 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5 2024-09-18T08:08:51.727 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:08:51.728 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:08:51.728 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5' 2024-09-18T08:08:51.829 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2 2024-09-18T08:08:51.863 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:08:51.864 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:08:51.864 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2' 2024-09-18T08:08:51.965 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1 2024-09-18T08:08:51.969 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:08:51 smithi007 ceph-mon[29459]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-18T08:08:51.969 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:08:51 smithi007 ceph-mon[29459]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-18T08:08:51.969 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:08:51 smithi007 ceph-mon[29459]: osdmap e197: 8 total, 8 up, 8 in 2024-09-18T08:08:51.969 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:08:51 smithi007 ceph-mon[29459]: pgmap v294: 158 pgs: 1 peering, 1 active+recovering+remapped, 2 active+recovering+undersized+remapped, 154 active+clean; 4.1 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 1.2 KiB/s rd, 70 MiB/s wr, 1.12k op/s; 5232/201315 objects misplaced (2.599%); 23 MiB/s, 367 objects/s recovering 2024-09-18T08:08:51.969 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:08:51 smithi007 ceph-mon[29459]: from='client.? 172.21.15.7:0/3014542625' entity='client.admin' cmd={"prefix": "osd reweight", "id": 2, "weight": 1.0} : dispatch 2024-09-18T08:08:51.969 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:08:51 smithi007 ceph-mon[29459]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 2, "weight": 1.0} : dispatch 2024-09-18T08:08:51.970 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:08:51 smithi007 ceph-mon[39249]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-18T08:08:51.970 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:08:51 smithi007 ceph-mon[39249]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-18T08:08:51.970 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:08:51 smithi007 ceph-mon[39249]: osdmap e197: 8 total, 8 up, 8 in 2024-09-18T08:08:51.970 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:08:51 smithi007 ceph-mon[39249]: pgmap v294: 158 pgs: 1 peering, 1 active+recovering+remapped, 2 active+recovering+undersized+remapped, 154 active+clean; 4.1 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 1.2 KiB/s rd, 70 MiB/s wr, 1.12k op/s; 5232/201315 objects misplaced (2.599%); 23 MiB/s, 367 objects/s recovering 2024-09-18T08:08:51.970 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:08:51 smithi007 ceph-mon[39249]: from='client.? 172.21.15.7:0/3014542625' entity='client.admin' cmd={"prefix": "osd reweight", "id": 2, "weight": 1.0} : dispatch 2024-09-18T08:08:51.970 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:08:51 smithi007 ceph-mon[39249]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 2, "weight": 1.0} : dispatch 2024-09-18T08:08:51.997 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:08:51.998 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:08:51.998 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1' 2024-09-18T08:08:52.099 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2 2024-09-18T08:08:52.125 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:08:52.126 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:08:52.126 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2' 2024-09-18T08:08:52.156 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:08:51 smithi050 ceph-mon[32371]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-18T08:08:52.156 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:08:51 smithi050 ceph-mon[32371]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-18T08:08:52.156 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:08:51 smithi050 ceph-mon[32371]: osdmap e197: 8 total, 8 up, 8 in 2024-09-18T08:08:52.156 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:08:51 smithi050 ceph-mon[32371]: pgmap v294: 158 pgs: 1 peering, 1 active+recovering+remapped, 2 active+recovering+undersized+remapped, 154 active+clean; 4.1 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 1.2 KiB/s rd, 70 MiB/s wr, 1.12k op/s; 5232/201315 objects misplaced (2.599%); 23 MiB/s, 367 objects/s recovering 2024-09-18T08:08:52.156 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:08:51 smithi050 ceph-mon[32371]: from='client.? 172.21.15.7:0/3014542625' entity='client.admin' cmd={"prefix": "osd reweight", "id": 2, "weight": 1.0} : dispatch 2024-09-18T08:08:52.156 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:08:51 smithi050 ceph-mon[32371]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 2, "weight": 1.0} : dispatch 2024-09-18T08:08:52.172 INFO:tasks.radosbench.radosbench.0.smithi007.stdout: 54 16 69360 69344 80.23 45.25 0.00531888 0.0123704 2024-09-18T08:08:52.227 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3 2024-09-18T08:08:52.239 INFO:teuthology.orchestra.run.smithi007.stdout:{ 2024-09-18T08:08:52.239 INFO:teuthology.orchestra.run.smithi007.stdout: "ops": [], 2024-09-18T08:08:52.239 INFO:teuthology.orchestra.run.smithi007.stdout: "complaint_time": 30, 2024-09-18T08:08:52.239 INFO:teuthology.orchestra.run.smithi007.stdout: "num_blocked_ops": 0 2024-09-18T08:08:52.239 INFO:teuthology.orchestra.run.smithi007.stdout:} 2024-09-18T08:08:52.256 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:08:52.257 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:08:52.257 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3' 2024-09-18T08:08:52.358 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5 2024-09-18T08:08:52.381 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:08:52.382 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:08:52.382 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5' 2024-09-18T08:08:52.483 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1 2024-09-18T08:08:52.508 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:08:52.509 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:08:52.509 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1' 2024-09-18T08:08:52.610 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1 2024-09-18T08:08:52.634 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:08:52.635 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:08:52.636 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1' 2024-09-18T08:08:52.736 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4 2024-09-18T08:08:52.760 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:08:52.761 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:08:52.761 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4' 2024-09-18T08:08:52.862 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7 2024-09-18T08:08:52.885 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:08:52.886 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:08:52.886 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7' 2024-09-18T08:08:52.943 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:08:52 smithi007 ceph-mon[29459]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]': finished 2024-09-18T08:08:52.943 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:08:52 smithi007 ceph-mon[29459]: osdmap e198: 8 total, 8 up, 8 in 2024-09-18T08:08:52.943 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:08:52 smithi007 ceph-mon[39249]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]': finished 2024-09-18T08:08:52.944 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:08:52 smithi007 ceph-mon[39249]: osdmap e198: 8 total, 8 up, 8 in 2024-09-18T08:08:52.944 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid 779349ca-7593-11ef-bceb-c7b262605968 -- ceph osd reweight 3 1 2024-09-18T08:08:52.986 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2 2024-09-18T08:08:53.013 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:08:53.014 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:08:53.015 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2' 2024-09-18T08:08:53.116 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5 2024-09-18T08:08:53.138 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:08:53.139 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:08:53.139 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5' 2024-09-18T08:08:53.155 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:08:52 smithi050 ceph-mon[32371]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]': finished 2024-09-18T08:08:53.156 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:08:52 smithi050 ceph-mon[32371]: osdmap e198: 8 total, 8 up, 8 in 2024-09-18T08:08:53.172 INFO:tasks.radosbench.radosbench.0.smithi007.stdout: 55 16 69360 69344 78.7716 0 - 0.0123704 2024-09-18T08:08:53.241 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2 2024-09-18T08:08:53.264 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:08:53.264 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:08:53.265 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2' 2024-09-18T08:08:53.365 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7 2024-09-18T08:08:53.389 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:08:53.390 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:08:53.390 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7' 2024-09-18T08:08:53.491 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5 2024-09-18T08:08:53.514 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:08:53.515 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:08:53.515 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5' 2024-09-18T08:08:53.616 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3 2024-09-18T08:08:53.640 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:08:53.640 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:08:53.641 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3' 2024-09-18T08:08:53.741 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6 2024-09-18T08:08:53.764 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:08:53.765 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:08:53.765 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6' 2024-09-18T08:08:53.866 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0 2024-09-18T08:08:53.904 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:08:53.905 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:08:53.905 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0' 2024-09-18T08:08:53.906 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:08:53 smithi050 ceph-mon[32371]: 4.12 scrub starts 2024-09-18T08:08:53.906 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:08:53 smithi050 ceph-mon[32371]: 4.12 scrub ok 2024-09-18T08:08:53.906 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:08:53 smithi050 ceph-mon[32371]: osdmap e199: 8 total, 8 up, 8 in 2024-09-18T08:08:53.906 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:08:53 smithi050 ceph-mon[32371]: pgmap v297: 158 pgs: 1 peering, 1 active+recovering+remapped, 2 active+recovering+undersized+remapped, 154 active+clean; 4.1 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 5232/201315 objects misplaced (2.599%) 2024-09-18T08:08:53.906 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:08:53 smithi050 ceph-mon[32371]: osdmap e200: 8 total, 8 up, 8 in 2024-09-18T08:08:53.981 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:08:53 smithi007 ceph-mon[29459]: 4.12 scrub starts 2024-09-18T08:08:53.981 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:08:53 smithi007 ceph-mon[29459]: 4.12 scrub ok 2024-09-18T08:08:53.981 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:08:53 smithi007 ceph-mon[29459]: osdmap e199: 8 total, 8 up, 8 in 2024-09-18T08:08:53.981 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:08:53 smithi007 ceph-mon[29459]: pgmap v297: 158 pgs: 1 peering, 1 active+recovering+remapped, 2 active+recovering+undersized+remapped, 154 active+clean; 4.1 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 5232/201315 objects misplaced (2.599%) 2024-09-18T08:08:53.981 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:08:53 smithi007 ceph-mon[29459]: osdmap e200: 8 total, 8 up, 8 in 2024-09-18T08:08:53.982 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:08:53 smithi007 ceph-mon[39249]: 4.12 scrub starts 2024-09-18T08:08:53.982 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:08:53 smithi007 ceph-mon[39249]: 4.12 scrub ok 2024-09-18T08:08:53.983 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:08:53 smithi007 ceph-mon[39249]: osdmap e199: 8 total, 8 up, 8 in 2024-09-18T08:08:53.983 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:08:53 smithi007 ceph-mon[39249]: pgmap v297: 158 pgs: 1 peering, 1 active+recovering+remapped, 2 active+recovering+undersized+remapped, 154 active+clean; 4.1 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 5232/201315 objects misplaced (2.599%) 2024-09-18T08:08:53.983 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:08:53 smithi007 ceph-mon[39249]: osdmap e200: 8 total, 8 up, 8 in 2024-09-18T08:08:54.006 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1 2024-09-18T08:08:54.013 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid 779349ca-7593-11ef-bceb-c7b262605968 -- ceph osd unset nodeep-scrub 2024-09-18T08:08:54.049 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:08:54.050 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:08:54.050 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1' 2024-09-18T08:08:54.151 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5 2024-09-18T08:08:54.172 INFO:tasks.radosbench.radosbench.0.smithi007.stdout: 56 15 69686 69671 77.7302 10.2188 0.0115271 0.0128585 2024-09-18T08:08:54.180 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:08:54.181 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:08:54.181 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5' 2024-09-18T08:08:54.282 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1 2024-09-18T08:08:54.368 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:08:54.369 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:08:54.369 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1' 2024-09-18T08:08:54.470 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2 2024-09-18T08:08:54.524 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:08:54.525 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:08:54.525 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2' 2024-09-18T08:08:54.626 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3 2024-09-18T08:08:54.661 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:08:54.661 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:08:54.662 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3' 2024-09-18T08:08:54.762 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0 2024-09-18T08:08:54.805 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:08:54.805 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:08:54.806 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0' 2024-09-18T08:08:54.897 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/779349ca-7593-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T08:08:54.907 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5 2024-09-18T08:08:54.936 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:08:54.937 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:08:54.937 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5' 2024-09-18T08:08:54.982 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:08:54 smithi007 ceph-mon[29459]: 3.0 scrub starts 2024-09-18T08:08:54.982 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:08:54 smithi007 ceph-mon[29459]: 3.0 scrub ok 2024-09-18T08:08:54.982 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:08:54 smithi007 ceph-mon[29459]: 8.f starting backfill to osd.2 from (0'0,0'0] MAX to 197'4403 2024-09-18T08:08:54.982 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:08:54 smithi007 ceph-mon[39249]: 3.0 scrub starts 2024-09-18T08:08:54.982 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:08:54 smithi007 ceph-mon[39249]: 3.0 scrub ok 2024-09-18T08:08:54.982 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:08:54 smithi007 ceph-mon[39249]: 8.f starting backfill to osd.2 from (0'0,0'0] MAX to 197'4403 2024-09-18T08:08:55.038 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5 2024-09-18T08:08:55.075 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:08:55.076 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:08:55.077 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5' 2024-09-18T08:08:55.156 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:08:54 smithi050 ceph-mon[32371]: 3.0 scrub starts 2024-09-18T08:08:55.156 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:08:54 smithi050 ceph-mon[32371]: 3.0 scrub ok 2024-09-18T08:08:55.156 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:08:54 smithi050 ceph-mon[32371]: 8.f starting backfill to osd.2 from (0'0,0'0] MAX to 197'4403 2024-09-18T08:08:55.173 INFO:tasks.radosbench.radosbench.0.smithi007.stdout: 57 16 70669 70653 77.4421 61.375 0.00963076 0.0129068 2024-09-18T08:08:55.178 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1 2024-09-18T08:08:55.222 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:08:55.222 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:08:55.223 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1' 2024-09-18T08:08:55.324 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7 2024-09-18T08:08:55.359 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:08:55.360 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:08:55.360 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7' 2024-09-18T08:08:55.461 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5 2024-09-18T08:08:55.488 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:08:55.488 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:08:55.489 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5' 2024-09-18T08:08:55.524 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid 779349ca-7593-11ef-bceb-c7b262605968 -- ceph daemon osd.1 dump_historic_ops 2024-09-18T08:08:55.589 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1 2024-09-18T08:08:55.646 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:08:55.647 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:08:55.647 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1' 2024-09-18T08:08:55.730 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:08:55 smithi007 ceph-mon[29459]: pgmap v299: 158 pgs: 1 active+recovering+remapped, 1 active+recovery_wait+degraded, 1 activating+remapped, 1 remapped+peering, 2 active+recovering+undersized+remapped, 152 active+clean; 4.2 GiB data, 15 GiB used, 701 GiB / 715 GiB avail; 1.1 KiB/s rd, 34 MiB/s wr, 541 op/s; 6/208626 objects degraded (0.003%); 7639/208626 objects misplaced (3.662%); 28 MiB/s, 442 objects/s recovering 2024-09-18T08:08:55.731 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:08:55 smithi007 ceph-mon[39249]: pgmap v299: 158 pgs: 1 active+recovering+remapped, 1 active+recovery_wait+degraded, 1 activating+remapped, 1 remapped+peering, 2 active+recovering+undersized+remapped, 152 active+clean; 4.2 GiB data, 15 GiB used, 701 GiB / 715 GiB avail; 1.1 KiB/s rd, 34 MiB/s wr, 541 op/s; 6/208626 objects degraded (0.003%); 7639/208626 objects misplaced (3.662%); 28 MiB/s, 442 objects/s recovering 2024-09-18T08:08:55.749 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0 2024-09-18T08:08:55.823 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:08:55.823 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:08:55.824 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0' 2024-09-18T08:08:55.924 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7 2024-09-18T08:08:55.959 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:08:55.959 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:08:55.960 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7' 2024-09-18T08:08:56.061 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4 2024-09-18T08:08:56.088 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/779349ca-7593-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T08:08:56.089 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:08:56.089 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:08:56.089 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4' 2024-09-18T08:08:56.156 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:08:55 smithi050 ceph-mon[32371]: pgmap v299: 158 pgs: 1 active+recovering+remapped, 1 active+recovery_wait+degraded, 1 activating+remapped, 1 remapped+peering, 2 active+recovering+undersized+remapped, 152 active+clean; 4.2 GiB data, 15 GiB used, 701 GiB / 715 GiB avail; 1.1 KiB/s rd, 34 MiB/s wr, 541 op/s; 6/208626 objects degraded (0.003%); 7639/208626 objects misplaced (3.662%); 28 MiB/s, 442 objects/s recovering 2024-09-18T08:08:56.172 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/779349ca-7593-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T08:08:56.173 INFO:tasks.radosbench.radosbench.0.smithi007.stdout: 58 16 71582 71566 77.0904 57.0625 0.0213042 0.0129654 2024-09-18T08:08:56.175 INFO:teuthology.orchestra.run.smithi007.stdout:osd.0: {} 2024-09-18T08:08:56.175 INFO:teuthology.orchestra.run.smithi007.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-18T08:08:56.190 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6 2024-09-18T08:08:56.217 INFO:teuthology.orchestra.run.smithi007.stdout:osd.1: {} 2024-09-18T08:08:56.217 INFO:teuthology.orchestra.run.smithi007.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-18T08:08:56.222 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:08:56.222 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:08:56.223 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6' 2024-09-18T08:08:56.249 INFO:teuthology.orchestra.run.smithi007.stdout:osd.2: {} 2024-09-18T08:08:56.249 INFO:teuthology.orchestra.run.smithi007.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-18T08:08:56.282 INFO:teuthology.orchestra.run.smithi007.stdout:osd.3: {} 2024-09-18T08:08:56.282 INFO:teuthology.orchestra.run.smithi007.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-18T08:08:56.318 INFO:teuthology.orchestra.run.smithi007.stdout:osd.4: {} 2024-09-18T08:08:56.318 INFO:teuthology.orchestra.run.smithi007.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-18T08:08:56.324 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0 2024-09-18T08:08:56.357 INFO:teuthology.orchestra.run.smithi007.stdout:osd.5: {} 2024-09-18T08:08:56.357 INFO:teuthology.orchestra.run.smithi007.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-18T08:08:56.368 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:08:56.369 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:08:56.369 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0' 2024-09-18T08:08:56.392 INFO:teuthology.orchestra.run.smithi007.stdout:osd.6: {} 2024-09-18T08:08:56.392 INFO:teuthology.orchestra.run.smithi007.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-18T08:08:56.430 INFO:teuthology.orchestra.run.smithi007.stdout:osd.7: {} 2024-09-18T08:08:56.431 INFO:teuthology.orchestra.run.smithi007.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-18T08:08:56.470 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4 2024-09-18T08:08:56.502 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:08:56.503 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:08:56.503 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4' 2024-09-18T08:08:56.604 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4 2024-09-18T08:08:56.636 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:08:56.637 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:08:56.638 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4' 2024-09-18T08:08:56.739 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2 2024-09-18T08:08:56.784 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:08:56.784 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:08:56.785 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2' 2024-09-18T08:08:56.885 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0 2024-09-18T08:08:56.932 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:08:56.932 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:08:56.933 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0' 2024-09-18T08:08:56.980 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:08:56 smithi007 ceph-mon[29459]: Health check failed: Degraded data redundancy: 6/208626 objects degraded (0.003%), 1 pg degraded (PG_DEGRADED) 2024-09-18T08:08:56.980 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:08:56 smithi007 ceph-mon[29459]: from='client.? 172.21.15.7:0/1521840577' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-18T08:08:56.981 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Sep 18 08:08:56 smithi007 ceph-779349ca-7593-11ef-bceb-c7b262605968-mgr-y[29688]: ::ffff:172.21.15.50 - - [18/Sep/2024:08:08:56] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-09-18T08:08:56.981 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:08:56 smithi007 ceph-mon[39249]: Health check failed: Degraded data redundancy: 6/208626 objects degraded (0.003%), 1 pg degraded (PG_DEGRADED) 2024-09-18T08:08:56.981 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:08:56 smithi007 ceph-mon[39249]: from='client.? 172.21.15.7:0/1521840577' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-18T08:08:57.034 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6 2024-09-18T08:08:57.067 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:08:57.068 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:08:57.069 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6' 2024-09-18T08:08:57.156 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:08:56 smithi050 ceph-mon[32371]: Health check failed: Degraded data redundancy: 6/208626 objects degraded (0.003%), 1 pg degraded (PG_DEGRADED) 2024-09-18T08:08:57.156 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:08:56 smithi050 ceph-mon[32371]: from='client.? 172.21.15.7:0/1521840577' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-18T08:08:57.169 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1 2024-09-18T08:08:57.174 INFO:tasks.radosbench.radosbench.0.smithi007.stdout: 59 16 72528 72512 76.7855 59.125 0.0206124 0.013017 2024-09-18T08:08:57.209 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:08:57.210 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:08:57.210 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1' 2024-09-18T08:08:57.311 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6 2024-09-18T08:08:57.342 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:08:57.342 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:08:57.343 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6' 2024-09-18T08:08:57.444 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5 2024-09-18T08:08:57.473 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:08:57.474 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:08:57.474 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5' 2024-09-18T08:08:57.575 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0 2024-09-18T08:08:57.622 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:08:57.622 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:08:57.622 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0' 2024-09-18T08:08:57.724 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1 2024-09-18T08:08:57.733 INFO:teuthology.orchestra.run.smithi007.stderr:reweighted osd.3 to 1 (10000) 2024-09-18T08:08:57.775 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:08:57.775 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:08:57.776 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1' 2024-09-18T08:08:57.877 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0 2024-09-18T08:08:57.915 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:08:57.916 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:08:57.916 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0' 2024-09-18T08:08:57.968 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:08:57 smithi007 ceph-mon[29459]: 4.f scrub starts 2024-09-18T08:08:57.969 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:08:57 smithi007 ceph-mon[29459]: 4.f scrub ok 2024-09-18T08:08:57.969 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:08:57 smithi007 ceph-mon[29459]: pgmap v300: 158 pgs: 1 active+recovering+remapped, 1 active+recovery_wait+degraded, 1 activating+remapped, 1 remapped+peering, 2 active+recovering+undersized+remapped, 152 active+clean; 4.2 GiB data, 15 GiB used, 701 GiB / 715 GiB avail; 853 B/s rd, 25 MiB/s wr, 406 op/s; 6/208626 objects degraded (0.003%); 7639/208626 objects misplaced (3.662%); 21 MiB/s, 332 objects/s recovering 2024-09-18T08:08:57.969 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:08:57 smithi007 ceph-mon[29459]: from='client.? 172.21.15.7:0/1241729761' entity='client.admin' cmd={"prefix": "osd reweight", "id": 3, "weight": 1.0} : dispatch 2024-09-18T08:08:57.969 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:08:57 smithi007 ceph-mon[29459]: 1.7 scrub starts 2024-09-18T08:08:57.969 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:08:57 smithi007 ceph-mon[29459]: 1.7 scrub ok 2024-09-18T08:08:57.969 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:08:57 smithi007 ceph-mon[39249]: 4.f scrub starts 2024-09-18T08:08:57.969 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:08:57 smithi007 ceph-mon[39249]: 4.f scrub ok 2024-09-18T08:08:57.969 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:08:57 smithi007 ceph-mon[39249]: pgmap v300: 158 pgs: 1 active+recovering+remapped, 1 active+recovery_wait+degraded, 1 activating+remapped, 1 remapped+peering, 2 active+recovering+undersized+remapped, 152 active+clean; 4.2 GiB data, 15 GiB used, 701 GiB / 715 GiB avail; 853 B/s rd, 25 MiB/s wr, 406 op/s; 6/208626 objects degraded (0.003%); 7639/208626 objects misplaced (3.662%); 21 MiB/s, 332 objects/s recovering 2024-09-18T08:08:57.969 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:08:57 smithi007 ceph-mon[39249]: from='client.? 172.21.15.7:0/1241729761' entity='client.admin' cmd={"prefix": "osd reweight", "id": 3, "weight": 1.0} : dispatch 2024-09-18T08:08:57.969 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:08:57 smithi007 ceph-mon[39249]: 1.7 scrub starts 2024-09-18T08:08:57.970 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:08:57 smithi007 ceph-mon[39249]: 1.7 scrub ok 2024-09-18T08:08:58.017 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1 2024-09-18T08:08:58.050 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:08:58.051 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:08:58.051 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1' 2024-09-18T08:08:58.152 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3 2024-09-18T08:08:58.156 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:08:57 smithi050 ceph-mon[32371]: 4.f scrub starts 2024-09-18T08:08:58.156 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:08:57 smithi050 ceph-mon[32371]: 4.f scrub ok 2024-09-18T08:08:58.156 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:08:57 smithi050 ceph-mon[32371]: pgmap v300: 158 pgs: 1 active+recovering+remapped, 1 active+recovery_wait+degraded, 1 activating+remapped, 1 remapped+peering, 2 active+recovering+undersized+remapped, 152 active+clean; 4.2 GiB data, 15 GiB used, 701 GiB / 715 GiB avail; 853 B/s rd, 25 MiB/s wr, 406 op/s; 6/208626 objects degraded (0.003%); 7639/208626 objects misplaced (3.662%); 21 MiB/s, 332 objects/s recovering 2024-09-18T08:08:58.156 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:08:57 smithi050 ceph-mon[32371]: from='client.? 172.21.15.7:0/1241729761' entity='client.admin' cmd={"prefix": "osd reweight", "id": 3, "weight": 1.0} : dispatch 2024-09-18T08:08:58.156 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:08:57 smithi050 ceph-mon[32371]: 1.7 scrub starts 2024-09-18T08:08:58.156 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:08:57 smithi050 ceph-mon[32371]: 1.7 scrub ok 2024-09-18T08:08:58.174 INFO:tasks.radosbench.radosbench.0.smithi007.stdout:2024-09-18T08:08:58.173235+0000 min lat: 0.00120772 max lat: 2.15499 avg lat: 0.0130425 2024-09-18T08:08:58.174 INFO:tasks.radosbench.radosbench.0.smithi007.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-09-18T08:08:58.174 INFO:tasks.radosbench.radosbench.0.smithi007.stdout: 60 16 73294 73278 76.3037 47.875 0.00321289 0.0130425 2024-09-18T08:08:58.179 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:08:58.179 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:08:58.180 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3' 2024-09-18T08:08:58.281 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4 2024-09-18T08:08:58.305 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:08:58.305 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:08:58.306 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4' 2024-09-18T08:08:58.337 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/779349ca-7593-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T08:08:58.406 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7 2024-09-18T08:08:58.430 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:08:58.431 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:08:58.431 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7' 2024-09-18T08:08:58.532 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2 2024-09-18T08:08:58.561 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:08:58.562 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:08:58.562 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2' 2024-09-18T08:08:58.663 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3 2024-09-18T08:08:58.692 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:08:58.692 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:08:58.693 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3' 2024-09-18T08:08:58.735 INFO:teuthology.orchestra.run.smithi007.stderr:nodeep-scrub is unset 2024-09-18T08:08:58.793 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7 2024-09-18T08:08:58.821 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:08:58.824 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:08:58.824 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7' 2024-09-18T08:08:58.835 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid 779349ca-7593-11ef-bceb-c7b262605968 -- ceph osd reweight 5 1 2024-09-18T08:08:58.925 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3 2024-09-18T08:08:58.970 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:08:58.970 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:08:58.971 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3' 2024-09-18T08:08:58.980 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:08:58 smithi007 ceph-mon[29459]: 1.a scrub starts 2024-09-18T08:08:58.981 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:08:58 smithi007 ceph-mon[29459]: 1.a scrub ok 2024-09-18T08:08:58.981 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:08:58 smithi007 ceph-mon[29459]: 4.4 scrub starts 2024-09-18T08:08:58.981 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:08:58 smithi007 ceph-mon[29459]: 4.4 scrub ok 2024-09-18T08:08:58.981 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:08:58 smithi007 ceph-mon[29459]: from='client.? 172.21.15.7:0/1241729761' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]': finished 2024-09-18T08:08:58.981 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:08:58 smithi007 ceph-mon[29459]: 1.6 scrub starts 2024-09-18T08:08:58.981 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:08:58 smithi007 ceph-mon[29459]: 1.6 scrub ok 2024-09-18T08:08:58.981 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:08:58 smithi007 ceph-mon[29459]: osdmap e201: 8 total, 8 up, 8 in 2024-09-18T08:08:58.981 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:08:58 smithi007 ceph-mon[29459]: 1.0 scrub starts 2024-09-18T08:08:58.981 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:08:58 smithi007 ceph-mon[29459]: 1.0 scrub ok 2024-09-18T08:08:58.981 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:08:58 smithi007 ceph-mon[29459]: from='client.? 172.21.15.7:0/76675673' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-18T08:08:58.981 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:08:58 smithi007 ceph-mon[29459]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-18T08:08:58.982 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:08:58 smithi007 ceph-mon[39249]: 1.a scrub starts 2024-09-18T08:08:58.982 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:08:58 smithi007 ceph-mon[39249]: 1.a scrub ok 2024-09-18T08:08:58.982 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:08:58 smithi007 ceph-mon[39249]: 4.4 scrub starts 2024-09-18T08:08:58.982 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:08:58 smithi007 ceph-mon[39249]: 4.4 scrub ok 2024-09-18T08:08:58.982 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:08:58 smithi007 ceph-mon[39249]: from='client.? 172.21.15.7:0/1241729761' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]': finished 2024-09-18T08:08:58.982 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:08:58 smithi007 ceph-mon[39249]: 1.6 scrub starts 2024-09-18T08:08:58.982 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:08:58 smithi007 ceph-mon[39249]: 1.6 scrub ok 2024-09-18T08:08:58.982 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:08:58 smithi007 ceph-mon[39249]: osdmap e201: 8 total, 8 up, 8 in 2024-09-18T08:08:58.982 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:08:58 smithi007 ceph-mon[39249]: 1.0 scrub starts 2024-09-18T08:08:58.982 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:08:58 smithi007 ceph-mon[39249]: 1.0 scrub ok 2024-09-18T08:08:58.983 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:08:58 smithi007 ceph-mon[39249]: from='client.? 172.21.15.7:0/76675673' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-18T08:08:58.983 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:08:58 smithi007 ceph-mon[39249]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-18T08:08:59.033 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:08:58 smithi050 ceph-mon[32371]: 1.a scrub starts 2024-09-18T08:08:59.034 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:08:58 smithi050 ceph-mon[32371]: 1.a scrub ok 2024-09-18T08:08:59.034 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:08:58 smithi050 ceph-mon[32371]: 4.4 scrub starts 2024-09-18T08:08:59.034 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:08:58 smithi050 ceph-mon[32371]: 4.4 scrub ok 2024-09-18T08:08:59.034 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:08:58 smithi050 ceph-mon[32371]: from='client.? 172.21.15.7:0/1241729761' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]': finished 2024-09-18T08:08:59.034 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:08:58 smithi050 ceph-mon[32371]: 1.6 scrub starts 2024-09-18T08:08:59.034 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:08:58 smithi050 ceph-mon[32371]: 1.6 scrub ok 2024-09-18T08:08:59.034 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:08:58 smithi050 ceph-mon[32371]: osdmap e201: 8 total, 8 up, 8 in 2024-09-18T08:08:59.034 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:08:58 smithi050 ceph-mon[32371]: 1.0 scrub starts 2024-09-18T08:08:59.034 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:08:58 smithi050 ceph-mon[32371]: 1.0 scrub ok 2024-09-18T08:08:59.034 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:08:58 smithi050 ceph-mon[32371]: from='client.? 172.21.15.7:0/76675673' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-18T08:08:59.034 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:08:58 smithi050 ceph-mon[32371]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-18T08:08:59.072 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3 2024-09-18T08:08:59.115 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:08:59.115 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:08:59.116 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3' 2024-09-18T08:08:59.176 INFO:tasks.radosbench.radosbench.0.smithi007.stdout: 61 16 73685 73669 75.4536 24.4375 0.0293907 0.0132471 2024-09-18T08:08:59.216 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2 2024-09-18T08:08:59.261 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:08:59.262 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:08:59.262 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2' 2024-09-18T08:08:59.363 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1 2024-09-18T08:08:59.405 INFO:journalctl@ceph.iscsi.iscsi.a.smithi050.stdout:Sep 18 08:08:59 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-iscsi-iscsi-a[57782]: debug there is no tcmu-runner data available 2024-09-18T08:08:59.419 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:08:59.419 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:08:59.419 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1' 2024-09-18T08:08:59.520 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7 2024-09-18T08:08:59.555 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:08:59.555 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:08:59.556 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7' 2024-09-18T08:08:59.656 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4 2024-09-18T08:08:59.688 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:08:59.689 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:08:59.690 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4' 2024-09-18T08:08:59.790 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5 2024-09-18T08:08:59.822 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:08:59.823 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:08:59.824 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5' 2024-09-18T08:08:59.924 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7 2024-09-18T08:08:59.954 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:08:59.955 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:08:59.956 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7' 2024-09-18T08:08:59.981 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:08:59 smithi007 ceph-mon[29459]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-18T08:08:59.981 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:08:59 smithi007 ceph-mon[29459]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-18T08:08:59.981 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:08:59 smithi007 ceph-mon[29459]: osdmap e202: 8 total, 8 up, 8 in 2024-09-18T08:08:59.981 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:08:59 smithi007 ceph-mon[29459]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-18T08:08:59.981 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:08:59 smithi007 ceph-mon[29459]: pgmap v303: 158 pgs: 1 active+remapped+backfilling, 1 active+recovering+remapped, 1 creating+peering, 2 active+recovering+undersized+remapped, 153 active+clean; 4.5 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 66 MiB/s wr, 1.05k op/s; 10160/220428 objects misplaced (4.609%); 43 MiB/s, 688 objects/s recovering 2024-09-18T08:08:59.981 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:08:59 smithi007 ceph-mon[29459]: 1.4 scrub starts 2024-09-18T08:08:59.981 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:08:59 smithi007 ceph-mon[29459]: 1.4 scrub ok 2024-09-18T08:08:59.982 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:08:59 smithi007 ceph-mon[39249]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-18T08:08:59.982 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:08:59 smithi007 ceph-mon[39249]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-18T08:08:59.982 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:08:59 smithi007 ceph-mon[39249]: osdmap e202: 8 total, 8 up, 8 in 2024-09-18T08:08:59.982 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:08:59 smithi007 ceph-mon[39249]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-18T08:08:59.982 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:08:59 smithi007 ceph-mon[39249]: pgmap v303: 158 pgs: 1 active+remapped+backfilling, 1 active+recovering+remapped, 1 creating+peering, 2 active+recovering+undersized+remapped, 153 active+clean; 4.5 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 66 MiB/s wr, 1.05k op/s; 10160/220428 objects misplaced (4.609%); 43 MiB/s, 688 objects/s recovering 2024-09-18T08:08:59.982 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:08:59 smithi007 ceph-mon[39249]: 1.4 scrub starts 2024-09-18T08:08:59.982 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:08:59 smithi007 ceph-mon[39249]: 1.4 scrub ok 2024-09-18T08:09:00.057 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6 2024-09-18T08:09:00.091 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:09:00.091 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:09:00.092 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6' 2024-09-18T08:09:00.156 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:08:59 smithi050 ceph-mon[32371]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-18T08:09:00.156 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:08:59 smithi050 ceph-mon[32371]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-18T08:09:00.156 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:08:59 smithi050 ceph-mon[32371]: osdmap e202: 8 total, 8 up, 8 in 2024-09-18T08:09:00.156 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:08:59 smithi050 ceph-mon[32371]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-18T08:09:00.156 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:08:59 smithi050 ceph-mon[32371]: pgmap v303: 158 pgs: 1 active+remapped+backfilling, 1 active+recovering+remapped, 1 creating+peering, 2 active+recovering+undersized+remapped, 153 active+clean; 4.5 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 66 MiB/s wr, 1.05k op/s; 10160/220428 objects misplaced (4.609%); 43 MiB/s, 688 objects/s recovering 2024-09-18T08:09:00.156 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:08:59 smithi050 ceph-mon[32371]: 1.4 scrub starts 2024-09-18T08:09:00.156 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:08:59 smithi050 ceph-mon[32371]: 1.4 scrub ok 2024-09-18T08:09:00.174 INFO:tasks.radosbench.radosbench.0.smithi007.stdout: 62 16 74722 74706 75.2819 64.8125 0.0116703 0.0132775 2024-09-18T08:09:00.193 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1 2024-09-18T08:09:00.209 INFO:teuthology.orchestra.run.smithi007.stdout:ERROR: (22) Invalid argument 2024-09-18T08:09:00.209 INFO:teuthology.orchestra.run.smithi007.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-18T08:09:00.248 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:09:00.248 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:09:00.248 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1' 2024-09-18T08:09:00.349 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1 2024-09-18T08:09:00.377 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid 779349ca-7593-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-18T08:09:00.389 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:09:00.390 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:09:00.390 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1' 2024-09-18T08:09:00.491 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7 2024-09-18T08:09:00.520 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:09:00.521 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:09:00.522 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7' 2024-09-18T08:09:00.622 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3 2024-09-18T08:09:00.666 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:09:00.666 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:09:00.666 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3' 2024-09-18T08:09:00.767 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3 2024-09-18T08:09:00.790 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:09:00.790 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:09:00.791 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3' 2024-09-18T08:09:00.892 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6 2024-09-18T08:09:00.923 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:09:00.925 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:09:00.925 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6' 2024-09-18T08:09:00.980 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:09:00 smithi007 ceph-mon[29459]: 1.1 scrub starts 2024-09-18T08:09:00.981 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:09:00 smithi007 ceph-mon[29459]: 1.1 scrub ok 2024-09-18T08:09:00.981 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:09:00 smithi007 ceph-mon[29459]: Health check failed: Reduced data availability: 1 pg peering (PG_AVAILABILITY) 2024-09-18T08:09:00.981 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:09:00 smithi007 ceph-mon[29459]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 6/208626 objects degraded (0.003%), 1 pg degraded) 2024-09-18T08:09:00.981 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:09:00 smithi007 ceph-mon[39249]: 1.1 scrub starts 2024-09-18T08:09:00.981 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:09:00 smithi007 ceph-mon[39249]: 1.1 scrub ok 2024-09-18T08:09:00.981 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:09:00 smithi007 ceph-mon[39249]: Health check failed: Reduced data availability: 1 pg peering (PG_AVAILABILITY) 2024-09-18T08:09:00.981 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:09:00 smithi007 ceph-mon[39249]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 6/208626 objects degraded (0.003%), 1 pg degraded) 2024-09-18T08:09:01.026 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0 2024-09-18T08:09:01.073 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:09:01.074 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:09:01.074 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0' 2024-09-18T08:09:01.156 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:09:00 smithi050 ceph-mon[32371]: 1.1 scrub starts 2024-09-18T08:09:01.156 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:09:00 smithi050 ceph-mon[32371]: 1.1 scrub ok 2024-09-18T08:09:01.156 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:09:00 smithi050 ceph-mon[32371]: Health check failed: Reduced data availability: 1 pg peering (PG_AVAILABILITY) 2024-09-18T08:09:01.156 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:09:00 smithi050 ceph-mon[32371]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 6/208626 objects degraded (0.003%), 1 pg degraded) 2024-09-18T08:09:01.174 INFO:tasks.radosbench.radosbench.0.smithi007.stdout: 63 16 75600 75584 74.958 54.875 0.0139716 0.013335 2024-09-18T08:09:01.176 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5 2024-09-18T08:09:01.206 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:09:01.206 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:09:01.206 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5' 2024-09-18T08:09:01.307 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4 2024-09-18T08:09:01.335 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:09:01.336 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:09:01.336 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4' 2024-09-18T08:09:01.437 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3 2024-09-18T08:09:01.474 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:09:01.475 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:09:01.476 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3' 2024-09-18T08:09:01.577 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4 2024-09-18T08:09:01.611 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:09:01.612 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:09:01.612 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4' 2024-09-18T08:09:01.713 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2 2024-09-18T08:09:01.751 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:09:01.752 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:09:01.752 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2' 2024-09-18T08:09:01.853 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7 2024-09-18T08:09:01.886 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:09:01.887 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:09:01.887 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7' 2024-09-18T08:09:01.944 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid 779349ca-7593-11ef-bceb-c7b262605968 -- ceph osd set noscrub 2024-09-18T08:09:01.977 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:09:01 smithi007 ceph-mon[39249]: 1.8 scrub starts 2024-09-18T08:09:01.977 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:09:01 smithi007 ceph-mon[39249]: 1.8 scrub ok 2024-09-18T08:09:01.977 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:09:01 smithi007 ceph-mon[39249]: pgmap v304: 158 pgs: 1 active+remapped+backfilling, 1 active+recovering+remapped, 1 creating+peering, 2 active+recovering+undersized+remapped, 153 active+clean; 4.5 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 43 MiB/s wr, 686 op/s; 10160/220428 objects misplaced (4.609%); 34 MiB/s, 549 objects/s recovering 2024-09-18T08:09:01.977 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:09:01 smithi007 ceph-mon[29459]: 1.8 scrub starts 2024-09-18T08:09:01.978 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:09:01 smithi007 ceph-mon[29459]: 1.8 scrub ok 2024-09-18T08:09:01.978 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:09:01 smithi007 ceph-mon[29459]: pgmap v304: 158 pgs: 1 active+remapped+backfilling, 1 active+recovering+remapped, 1 creating+peering, 2 active+recovering+undersized+remapped, 153 active+clean; 4.5 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 43 MiB/s wr, 686 op/s; 10160/220428 objects misplaced (4.609%); 34 MiB/s, 549 objects/s recovering 2024-09-18T08:09:01.988 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0 2024-09-18T08:09:02.014 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:09:02.014 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:09:02.015 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0' 2024-09-18T08:09:02.115 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3 2024-09-18T08:09:02.156 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:09:01 smithi050 ceph-mon[32371]: 1.8 scrub starts 2024-09-18T08:09:02.156 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:09:01 smithi050 ceph-mon[32371]: 1.8 scrub ok 2024-09-18T08:09:02.156 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:09:01 smithi050 ceph-mon[32371]: pgmap v304: 158 pgs: 1 active+remapped+backfilling, 1 active+recovering+remapped, 1 creating+peering, 2 active+recovering+undersized+remapped, 153 active+clean; 4.5 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 43 MiB/s wr, 686 op/s; 10160/220428 objects misplaced (4.609%); 34 MiB/s, 549 objects/s recovering 2024-09-18T08:09:02.175 INFO:tasks.radosbench.radosbench.0.smithi007.stdout: 64 16 76485 76469 74.651 55.3125 0.0112924 0.0133901 2024-09-18T08:09:02.181 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:09:02.182 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:09:02.183 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3' 2024-09-18T08:09:02.283 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0 2024-09-18T08:09:02.322 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:09:02.324 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:09:02.324 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0' 2024-09-18T08:09:02.425 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2 2024-09-18T08:09:02.467 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:09:02.468 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:09:02.468 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2' 2024-09-18T08:09:02.569 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7 2024-09-18T08:09:02.601 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:09:02.602 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:09:02.602 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7' 2024-09-18T08:09:02.703 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2 2024-09-18T08:09:02.763 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:09:02.764 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:09:02.764 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2' 2024-09-18T08:09:02.865 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6 2024-09-18T08:09:02.899 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:09:02.899 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:09:02.900 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6' 2024-09-18T08:09:03.001 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3 2024-09-18T08:09:03.023 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:09:03.024 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:09:03.024 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3' 2024-09-18T08:09:03.125 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7 2024-09-18T08:09:03.165 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:09:03.165 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:09:03.165 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7' 2024-09-18T08:09:03.174 INFO:tasks.radosbench.radosbench.0.smithi007.stdout: 65 16 77451 77435 74.4314 60.375 0.00990485 0.013429 2024-09-18T08:09:03.230 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:09:02 smithi007 ceph-mon[39249]: 3.4 scrub starts 2024-09-18T08:09:03.230 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:09:03 smithi007 ceph-mon[39249]: 3.4 scrub ok 2024-09-18T08:09:03.230 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:09:02 smithi007 ceph-mon[29459]: 3.4 scrub starts 2024-09-18T08:09:03.230 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:09:03 smithi007 ceph-mon[29459]: 3.4 scrub ok 2024-09-18T08:09:03.266 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3 2024-09-18T08:09:03.299 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:09:03.300 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:09:03.300 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3' 2024-09-18T08:09:03.401 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3 2024-09-18T08:09:03.405 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:09:02 smithi050 ceph-mon[32371]: 3.4 scrub starts 2024-09-18T08:09:03.406 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:09:02 smithi050 ceph-mon[32371]: 3.4 scrub ok 2024-09-18T08:09:03.450 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:09:03.451 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:09:03.451 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3' 2024-09-18T08:09:03.552 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3 2024-09-18T08:09:03.591 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:09:03.592 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:09:03.592 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3' 2024-09-18T08:09:03.693 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6 2024-09-18T08:09:03.729 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:09:03.729 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:09:03.730 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6' 2024-09-18T08:09:03.831 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5 2024-09-18T08:09:03.865 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:09:03.865 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:09:03.866 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5' 2024-09-18T08:09:03.966 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0 2024-09-18T08:09:03.989 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:09:03.990 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:09:03.990 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0' 2024-09-18T08:09:04.091 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2 2024-09-18T08:09:04.131 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:09:04.131 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:09:04.132 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2' 2024-09-18T08:09:04.174 INFO:tasks.radosbench.radosbench.0.smithi007.stdout: 66 16 78510 78494 74.3064 66.1875 0.0128002 0.0134518 2024-09-18T08:09:04.233 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2 2024-09-18T08:09:04.277 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:09:04.278 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:09:04.279 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2' 2024-09-18T08:09:04.380 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0 2024-09-18T08:09:04.406 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:09:04 smithi050 ceph-mon[32371]: from='mgr.24470 172.21.15.7:0/2594390261' entity='mgr.y' 2024-09-18T08:09:04.406 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:09:04 smithi050 ceph-mon[32371]: from='mgr.24470 172.21.15.7:0/2594390261' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-18T08:09:04.406 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:09:04 smithi050 ceph-mon[32371]: pgmap v305: 158 pgs: 1 active+remapped+backfilling, 1 active+recovering+remapped, 2 active+recovering+undersized+remapped, 154 active+clean; 4.5 GiB data, 16 GiB used, 700 GiB / 715 GiB avail; 33 MiB/s wr, 522 op/s; 10160/221289 objects misplaced (4.591%); 17 MiB/s, 266 objects/s recovering 2024-09-18T08:09:04.406 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:09:04 smithi050 ceph-mon[32371]: 5.12 scrub starts 2024-09-18T08:09:04.406 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:09:04 smithi050 ceph-mon[32371]: 5.12 scrub ok 2024-09-18T08:09:04.420 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:09:04.420 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:09:04.421 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0' 2024-09-18T08:09:04.481 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:09:04 smithi007 ceph-mon[29459]: from='mgr.24470 172.21.15.7:0/2594390261' entity='mgr.y' 2024-09-18T08:09:04.481 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:09:04 smithi007 ceph-mon[29459]: from='mgr.24470 172.21.15.7:0/2594390261' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-18T08:09:04.481 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:09:04 smithi007 ceph-mon[29459]: pgmap v305: 158 pgs: 1 active+remapped+backfilling, 1 active+recovering+remapped, 2 active+recovering+undersized+remapped, 154 active+clean; 4.5 GiB data, 16 GiB used, 700 GiB / 715 GiB avail; 33 MiB/s wr, 522 op/s; 10160/221289 objects misplaced (4.591%); 17 MiB/s, 266 objects/s recovering 2024-09-18T08:09:04.481 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:09:04 smithi007 ceph-mon[29459]: 5.12 scrub starts 2024-09-18T08:09:04.481 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:09:04 smithi007 ceph-mon[29459]: 5.12 scrub ok 2024-09-18T08:09:04.481 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:09:04 smithi007 ceph-mon[39249]: from='mgr.24470 172.21.15.7:0/2594390261' entity='mgr.y' 2024-09-18T08:09:04.481 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:09:04 smithi007 ceph-mon[39249]: from='mgr.24470 172.21.15.7:0/2594390261' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-18T08:09:04.481 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:09:04 smithi007 ceph-mon[39249]: pgmap v305: 158 pgs: 1 active+remapped+backfilling, 1 active+recovering+remapped, 2 active+recovering+undersized+remapped, 154 active+clean; 4.5 GiB data, 16 GiB used, 700 GiB / 715 GiB avail; 33 MiB/s wr, 522 op/s; 10160/221289 objects misplaced (4.591%); 17 MiB/s, 266 objects/s recovering 2024-09-18T08:09:04.482 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:09:04 smithi007 ceph-mon[39249]: 5.12 scrub starts 2024-09-18T08:09:04.482 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:09:04 smithi007 ceph-mon[39249]: 5.12 scrub ok 2024-09-18T08:09:04.522 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2 2024-09-18T08:09:04.570 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:09:04.571 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:09:04.571 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2' 2024-09-18T08:09:04.673 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5 2024-09-18T08:09:04.704 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:09:04.705 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:09:04.705 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5' 2024-09-18T08:09:04.806 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0 2024-09-18T08:09:04.848 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:09:04.848 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:09:04.849 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0' 2024-09-18T08:09:04.949 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1 2024-09-18T08:09:04.987 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:09:04.988 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:09:04.988 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1' 2024-09-18T08:09:05.089 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6 2024-09-18T08:09:05.116 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:09:05.117 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:09:05.118 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6' 2024-09-18T08:09:05.175 INFO:tasks.radosbench.radosbench.0.smithi007.stdout: 67 15 79464 79449 74.0882 59.6875 0.0979399 0.0134842 2024-09-18T08:09:05.219 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4 2024-09-18T08:09:05.250 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:09:05.251 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:09:05.251 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4' 2024-09-18T08:09:05.352 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7 2024-09-18T08:09:05.376 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:09:05.376 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:09:05.376 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7' 2024-09-18T08:09:05.406 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:09:05 smithi050 ceph-mon[32371]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg peering) 2024-09-18T08:09:05.406 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:09:05 smithi050 ceph-mon[32371]: Cluster is now healthy 2024-09-18T08:09:05.477 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0 2024-09-18T08:09:05.481 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:09:05 smithi007 ceph-mon[29459]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg peering) 2024-09-18T08:09:05.481 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:09:05 smithi007 ceph-mon[29459]: Cluster is now healthy 2024-09-18T08:09:05.482 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:09:05 smithi007 ceph-mon[39249]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg peering) 2024-09-18T08:09:05.482 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:09:05 smithi007 ceph-mon[39249]: Cluster is now healthy 2024-09-18T08:09:05.527 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:09:05.527 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:09:05.528 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0' 2024-09-18T08:09:05.628 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1 2024-09-18T08:09:05.669 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:09:05.670 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:09:05.670 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1' 2024-09-18T08:09:05.771 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3 2024-09-18T08:09:05.816 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:09:05.817 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:09:05.817 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3' 2024-09-18T08:09:05.918 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3 2024-09-18T08:09:05.958 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:09:05.959 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:09:05.959 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3' 2024-09-18T08:09:06.060 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4 2024-09-18T08:09:06.096 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:09:06.097 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:09:06.097 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4' 2024-09-18T08:09:06.175 INFO:tasks.radosbench.radosbench.0.smithi007.stdout: 68 16 80385 80369 73.8443 57.5 0.01377 0.0135363 2024-09-18T08:09:06.198 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5 2024-09-18T08:09:06.235 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:09:06.235 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:09:06.236 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5' 2024-09-18T08:09:06.336 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2 2024-09-18T08:09:06.359 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:09:06.359 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:09:06.360 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2' 2024-09-18T08:09:06.406 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:09:06 smithi050 ceph-mon[32371]: pgmap v306: 158 pgs: 1 active+remapped+backfilling, 1 active+recovering+remapped, 2 active+recovering+undersized+remapped, 154 active+clean; 4.7 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 66 MiB/s wr, 1.05k op/s; 8966/234024 objects misplaced (3.831%); 28 MiB/s, 442 objects/s recovering 2024-09-18T08:09:06.460 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5 2024-09-18T08:09:06.480 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:09:06 smithi007 ceph-mon[29459]: pgmap v306: 158 pgs: 1 active+remapped+backfilling, 1 active+recovering+remapped, 2 active+recovering+undersized+remapped, 154 active+clean; 4.7 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 66 MiB/s wr, 1.05k op/s; 8966/234024 objects misplaced (3.831%); 28 MiB/s, 442 objects/s recovering 2024-09-18T08:09:06.480 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:09:06 smithi007 ceph-mon[39249]: pgmap v306: 158 pgs: 1 active+remapped+backfilling, 1 active+recovering+remapped, 2 active+recovering+undersized+remapped, 154 active+clean; 4.7 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 66 MiB/s wr, 1.05k op/s; 8966/234024 objects misplaced (3.831%); 28 MiB/s, 442 objects/s recovering 2024-09-18T08:09:06.496 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:09:06.497 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:09:06.497 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5' 2024-09-18T08:09:06.598 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1 2024-09-18T08:09:06.641 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:09:06.641 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:09:06.642 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1' 2024-09-18T08:09:06.742 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6 2024-09-18T08:09:06.780 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:09:06.781 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:09:06.781 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6' 2024-09-18T08:09:06.882 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5 2024-09-18T08:09:06.924 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:09:06.925 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:09:06.925 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5' 2024-09-18T08:09:06.980 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Sep 18 08:09:06 smithi007 ceph-779349ca-7593-11ef-bceb-c7b262605968-mgr-y[29688]: ::ffff:172.21.15.50 - - [18/Sep/2024:08:09:06] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-09-18T08:09:07.026 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4 2024-09-18T08:09:07.061 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:09:07.062 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:09:07.062 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4' 2024-09-18T08:09:07.163 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7 2024-09-18T08:09:07.174 INFO:tasks.radosbench.radosbench.0.smithi007.stdout: 69 16 81410 81394 73.7024 64.0625 0.0109831 0.0135538 2024-09-18T08:09:07.186 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:09:07.186 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:09:07.187 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7' 2024-09-18T08:09:07.288 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6 2024-09-18T08:09:07.319 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:09:07.320 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:09:07.320 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6' 2024-09-18T08:09:07.406 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:09:07 smithi050 ceph-mon[32371]: 5.c scrub starts 2024-09-18T08:09:07.406 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:09:07 smithi050 ceph-mon[32371]: 5.c scrub ok 2024-09-18T08:09:07.421 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5 2024-09-18T08:09:07.454 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:09:07.455 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:09:07.455 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5' 2024-09-18T08:09:07.480 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:09:07 smithi007 ceph-mon[29459]: 5.c scrub starts 2024-09-18T08:09:07.480 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:09:07 smithi007 ceph-mon[29459]: 5.c scrub ok 2024-09-18T08:09:07.481 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:09:07 smithi007 ceph-mon[39249]: 5.c scrub starts 2024-09-18T08:09:07.481 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:09:07 smithi007 ceph-mon[39249]: 5.c scrub ok 2024-09-18T08:09:07.556 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0 2024-09-18T08:09:07.594 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:09:07.594 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:09:07.595 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0' 2024-09-18T08:09:07.695 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1 2024-09-18T08:09:07.732 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:09:07.733 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:09:07.733 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1' 2024-09-18T08:09:07.834 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6 2024-09-18T08:09:07.858 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:09:07.859 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:09:07.859 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6' 2024-09-18T08:09:07.960 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3 2024-09-18T08:09:08.002 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:09:08.002 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:09:08.003 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3' 2024-09-18T08:09:08.077 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid 779349ca-7593-11ef-bceb-c7b262605968 -- ceph daemon osd.2 dump_ops_in_flight 2024-09-18T08:09:08.104 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0 2024-09-18T08:09:08.154 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/779349ca-7593-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T08:09:08.166 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/779349ca-7593-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T08:09:08.170 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/779349ca-7593-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T08:09:08.177 INFO:tasks.radosbench.radosbench.0.smithi007.stdout: 70 16 82296 82280 73.4378 55.375 0.0280912 0.0136106 2024-09-18T08:09:08.197 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:09:08.198 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:09:08.198 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0' 2024-09-18T08:09:08.299 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5 2024-09-18T08:09:08.317 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:09:08 smithi007 ceph-mon[29459]: pgmap v307: 158 pgs: 1 active+remapped+backfilling, 1 active+recovering+remapped, 2 active+recovering+undersized+remapped, 154 active+clean; 4.7 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 56 MiB/s wr, 886 op/s; 8966/234024 objects misplaced (3.831%); 23 MiB/s, 374 objects/s recovering 2024-09-18T08:09:08.317 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:09:08 smithi007 ceph-mon[29459]: purged_snaps scrub starts 2024-09-18T08:09:08.317 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:09:08 smithi007 ceph-mon[29459]: purged_snaps scrub ok 2024-09-18T08:09:08.318 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:09:08 smithi007 ceph-mon[39249]: pgmap v307: 158 pgs: 1 active+remapped+backfilling, 1 active+recovering+remapped, 2 active+recovering+undersized+remapped, 154 active+clean; 4.7 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 56 MiB/s wr, 886 op/s; 8966/234024 objects misplaced (3.831%); 23 MiB/s, 374 objects/s recovering 2024-09-18T08:09:08.318 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:09:08 smithi007 ceph-mon[39249]: purged_snaps scrub starts 2024-09-18T08:09:08.318 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:09:08 smithi007 ceph-mon[39249]: purged_snaps scrub ok 2024-09-18T08:09:08.336 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:09:08.337 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:09:08.337 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5' 2024-09-18T08:09:08.406 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:09:08 smithi050 ceph-mon[32371]: pgmap v307: 158 pgs: 1 active+remapped+backfilling, 1 active+recovering+remapped, 2 active+recovering+undersized+remapped, 154 active+clean; 4.7 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 56 MiB/s wr, 886 op/s; 8966/234024 objects misplaced (3.831%); 23 MiB/s, 374 objects/s recovering 2024-09-18T08:09:08.406 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:09:08 smithi050 ceph-mon[32371]: purged_snaps scrub starts 2024-09-18T08:09:08.406 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:09:08 smithi050 ceph-mon[32371]: purged_snaps scrub ok 2024-09-18T08:09:08.438 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1 2024-09-18T08:09:08.513 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:09:08.513 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:09:08.514 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1' 2024-09-18T08:09:08.614 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7 2024-09-18T08:09:08.645 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:09:08.645 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:09:08.646 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7' 2024-09-18T08:09:08.746 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5 2024-09-18T08:09:08.779 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:09:08.779 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:09:08.780 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5' 2024-09-18T08:09:08.881 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2 2024-09-18T08:09:08.922 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:09:08.922 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:09:08.923 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2' 2024-09-18T08:09:09.024 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3 2024-09-18T08:09:09.063 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:09:09.063 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:09:09.064 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3' 2024-09-18T08:09:09.164 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0 2024-09-18T08:09:09.179 INFO:tasks.radosbench.radosbench.0.smithi007.stdout: 71 16 83256 83240 73.2469 60 0.0180447 0.0136468 2024-09-18T08:09:09.182 INFO:teuthology.orchestra.run.smithi007.stdout:osd.0: {} 2024-09-18T08:09:09.182 INFO:teuthology.orchestra.run.smithi007.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-18T08:09:09.217 INFO:teuthology.orchestra.run.smithi007.stdout:osd.1: {} 2024-09-18T08:09:09.218 INFO:teuthology.orchestra.run.smithi007.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-18T08:09:09.230 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:09:09.230 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:09:09.230 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0' 2024-09-18T08:09:09.231 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:09:09 smithi007 ceph-mon[29459]: osdmap e203: 8 total, 8 up, 8 in 2024-09-18T08:09:09.231 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:09:09 smithi007 ceph-mon[39249]: osdmap e203: 8 total, 8 up, 8 in 2024-09-18T08:09:09.262 INFO:teuthology.orchestra.run.smithi007.stdout:osd.2: {} 2024-09-18T08:09:09.262 INFO:teuthology.orchestra.run.smithi007.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-18T08:09:09.320 INFO:teuthology.orchestra.run.smithi007.stdout:osd.3: {} 2024-09-18T08:09:09.320 INFO:teuthology.orchestra.run.smithi007.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-18T08:09:09.331 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2 2024-09-18T08:09:09.356 INFO:teuthology.orchestra.run.smithi007.stdout:osd.4: {} 2024-09-18T08:09:09.356 INFO:teuthology.orchestra.run.smithi007.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-18T08:09:09.374 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:09:09.375 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:09:09.375 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2' 2024-09-18T08:09:09.405 INFO:teuthology.orchestra.run.smithi007.stdout:osd.5: {} 2024-09-18T08:09:09.405 INFO:teuthology.orchestra.run.smithi007.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-18T08:09:09.406 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:09:09 smithi050 ceph-mon[32371]: osdmap e203: 8 total, 8 up, 8 in 2024-09-18T08:09:09.406 INFO:journalctl@ceph.iscsi.iscsi.a.smithi050.stdout:Sep 18 08:09:09 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-iscsi-iscsi-a[57782]: debug there is no tcmu-runner data available 2024-09-18T08:09:09.429 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/779349ca-7593-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T08:09:09.444 INFO:teuthology.orchestra.run.smithi007.stdout:osd.6: {} 2024-09-18T08:09:09.444 INFO:teuthology.orchestra.run.smithi007.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-18T08:09:09.476 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3 2024-09-18T08:09:09.481 INFO:teuthology.orchestra.run.smithi007.stdout:osd.7: {} 2024-09-18T08:09:09.481 INFO:teuthology.orchestra.run.smithi007.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-18T08:09:09.524 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:09:09.524 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:09:09.525 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3' 2024-09-18T08:09:09.626 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1 2024-09-18T08:09:09.671 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:09:09.672 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:09:09.672 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1' 2024-09-18T08:09:09.773 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3 2024-09-18T08:09:09.810 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:09:09.811 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:09:09.811 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3' 2024-09-18T08:09:09.912 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6 2024-09-18T08:09:09.942 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:09:09.943 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:09:09.943 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6' 2024-09-18T08:09:10.044 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5 2024-09-18T08:09:10.077 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:09:10.077 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:09:10.078 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5' 2024-09-18T08:09:10.133 INFO:teuthology.orchestra.run.smithi007.stderr:reweighted osd.5 to 1 (10000) 2024-09-18T08:09:10.179 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0 2024-09-18T08:09:10.180 INFO:tasks.radosbench.radosbench.0.smithi007.stdout: 72 16 84160 84144 73.0143 56.5 0.00697372 0.0136906 2024-09-18T08:09:10.218 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:09:10.219 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:09:10.219 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0' 2024-09-18T08:09:10.320 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2 2024-09-18T08:09:10.359 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:09:10.360 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:09:10.361 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2' 2024-09-18T08:09:10.406 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:09:10 smithi050 ceph-mon[32371]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-18T08:09:10.406 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:09:10 smithi050 ceph-mon[32371]: from='client.? 172.21.15.7:0/3910150884' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-18T08:09:10.406 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:09:10 smithi050 ceph-mon[32371]: pgmap v309: 158 pgs: 1 active+remapped+backfilling, 1 active+recovering+remapped, 2 active+recovering+undersized+remapped, 154 active+clean; 5.0 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 58 MiB/s wr, 929 op/s; 7678/248421 objects misplaced (3.091%); 18 MiB/s, 287 objects/s recovering 2024-09-18T08:09:10.406 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:09:10 smithi050 ceph-mon[32371]: 2.0 scrub starts 2024-09-18T08:09:10.406 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:09:10 smithi050 ceph-mon[32371]: 2.0 scrub ok 2024-09-18T08:09:10.406 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:09:10 smithi050 ceph-mon[32371]: from='client.? 172.21.15.7:0/986718939' entity='client.admin' cmd={"prefix": "osd reweight", "id": 5, "weight": 1.0} : dispatch 2024-09-18T08:09:10.406 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:09:10 smithi050 ceph-mon[32371]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 5, "weight": 1.0} : dispatch 2024-09-18T08:09:10.462 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6 2024-09-18T08:09:10.481 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:09:10 smithi007 ceph-mon[29459]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-18T08:09:10.481 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:09:10 smithi007 ceph-mon[29459]: from='client.? 172.21.15.7:0/3910150884' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-18T08:09:10.481 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:09:10 smithi007 ceph-mon[29459]: pgmap v309: 158 pgs: 1 active+remapped+backfilling, 1 active+recovering+remapped, 2 active+recovering+undersized+remapped, 154 active+clean; 5.0 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 58 MiB/s wr, 929 op/s; 7678/248421 objects misplaced (3.091%); 18 MiB/s, 287 objects/s recovering 2024-09-18T08:09:10.481 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:09:10 smithi007 ceph-mon[29459]: 2.0 scrub starts 2024-09-18T08:09:10.481 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:09:10 smithi007 ceph-mon[29459]: 2.0 scrub ok 2024-09-18T08:09:10.481 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:09:10 smithi007 ceph-mon[29459]: from='client.? 172.21.15.7:0/986718939' entity='client.admin' cmd={"prefix": "osd reweight", "id": 5, "weight": 1.0} : dispatch 2024-09-18T08:09:10.481 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:09:10 smithi007 ceph-mon[29459]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 5, "weight": 1.0} : dispatch 2024-09-18T08:09:10.481 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:09:10 smithi007 ceph-mon[39249]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-18T08:09:10.481 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:09:10 smithi007 ceph-mon[39249]: from='client.? 172.21.15.7:0/3910150884' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-18T08:09:10.482 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:09:10 smithi007 ceph-mon[39249]: pgmap v309: 158 pgs: 1 active+remapped+backfilling, 1 active+recovering+remapped, 2 active+recovering+undersized+remapped, 154 active+clean; 5.0 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 58 MiB/s wr, 929 op/s; 7678/248421 objects misplaced (3.091%); 18 MiB/s, 287 objects/s recovering 2024-09-18T08:09:10.482 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:09:10 smithi007 ceph-mon[39249]: 2.0 scrub starts 2024-09-18T08:09:10.482 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:09:10 smithi007 ceph-mon[39249]: 2.0 scrub ok 2024-09-18T08:09:10.482 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:09:10 smithi007 ceph-mon[39249]: from='client.? 172.21.15.7:0/986718939' entity='client.admin' cmd={"prefix": "osd reweight", "id": 5, "weight": 1.0} : dispatch 2024-09-18T08:09:10.482 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:09:10 smithi007 ceph-mon[39249]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 5, "weight": 1.0} : dispatch 2024-09-18T08:09:10.501 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:09:10.502 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:09:10.502 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6' 2024-09-18T08:09:10.603 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2 2024-09-18T08:09:10.644 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:09:10.644 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:09:10.645 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2' 2024-09-18T08:09:10.746 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6 2024-09-18T08:09:10.777 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:09:10.778 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:09:10.778 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6' 2024-09-18T08:09:10.879 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4 2024-09-18T08:09:10.911 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:09:10.912 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:09:10.912 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4' 2024-09-18T08:09:11.013 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5 2024-09-18T08:09:11.053 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:09:11.054 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:09:11.054 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5' 2024-09-18T08:09:11.155 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7 2024-09-18T08:09:11.179 INFO:tasks.radosbench.radosbench.0.smithi007.stdout: 73 16 85214 85198 72.9165 65.875 0.0116857 0.013709 2024-09-18T08:09:11.194 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:09:11.195 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:09:11.195 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7' 2024-09-18T08:09:11.296 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6 2024-09-18T08:09:11.327 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:09:11.328 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:09:11.328 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6' 2024-09-18T08:09:11.406 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:09:11 smithi050 ceph-mon[32371]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]': finished 2024-09-18T08:09:11.406 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:09:11 smithi050 ceph-mon[32371]: osdmap e204: 8 total, 8 up, 8 in 2024-09-18T08:09:11.417 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:09:11 smithi007 ceph-mon[29459]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]': finished 2024-09-18T08:09:11.417 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:09:11 smithi007 ceph-mon[29459]: osdmap e204: 8 total, 8 up, 8 in 2024-09-18T08:09:11.417 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:09:11 smithi007 ceph-mon[39249]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]': finished 2024-09-18T08:09:11.418 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:09:11 smithi007 ceph-mon[39249]: osdmap e204: 8 total, 8 up, 8 in 2024-09-18T08:09:11.429 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6 2024-09-18T08:09:11.467 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:09:11.468 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:09:11.468 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6' 2024-09-18T08:09:11.569 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0 2024-09-18T08:09:11.609 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:09:11.610 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:09:11.611 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0' 2024-09-18T08:09:11.712 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5 2024-09-18T08:09:11.746 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:09:11.747 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:09:11.747 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5' 2024-09-18T08:09:11.831 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid 779349ca-7593-11ef-bceb-c7b262605968 -- ceph osd reweight 7 1 2024-09-18T08:09:11.848 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1 2024-09-18T08:09:11.922 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:09:11.923 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:09:11.923 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1' 2024-09-18T08:09:12.024 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5 2024-09-18T08:09:12.052 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:09:12.053 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:09:12.053 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5' 2024-09-18T08:09:12.154 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3 2024-09-18T08:09:12.180 INFO:tasks.radosbench.radosbench.0.smithi007.stdout: 74 16 86240 86224 72.7965 64.125 0.0284946 0.0137294 2024-09-18T08:09:12.213 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:09:12.214 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:09:12.214 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3' 2024-09-18T08:09:12.315 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7 2024-09-18T08:09:12.340 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:09:12 smithi007 ceph-mon[29459]: pgmap v311: 158 pgs: 1 active+remapped+backfilling, 1 active+recovering+remapped, 2 active+recovering+undersized+remapped, 154 active+clean; 5.0 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 70 MiB/s wr, 1.12k op/s; 7678/248421 objects misplaced (3.091%); 22 MiB/s, 359 objects/s recovering 2024-09-18T08:09:12.340 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:09:12 smithi007 ceph-mon[29459]: 5.1d scrub starts 2024-09-18T08:09:12.341 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:09:12 smithi007 ceph-mon[29459]: 5.1d scrub ok 2024-09-18T08:09:12.341 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:09:12 smithi007 ceph-mon[39249]: pgmap v311: 158 pgs: 1 active+remapped+backfilling, 1 active+recovering+remapped, 2 active+recovering+undersized+remapped, 154 active+clean; 5.0 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 70 MiB/s wr, 1.12k op/s; 7678/248421 objects misplaced (3.091%); 22 MiB/s, 359 objects/s recovering 2024-09-18T08:09:12.341 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:09:12 smithi007 ceph-mon[39249]: 5.1d scrub starts 2024-09-18T08:09:12.341 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:09:12 smithi007 ceph-mon[39249]: 5.1d scrub ok 2024-09-18T08:09:12.346 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:09:12.347 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:09:12.347 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7' 2024-09-18T08:09:12.448 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1 2024-09-18T08:09:12.486 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:09:12.486 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:09:12.487 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1' 2024-09-18T08:09:12.506 INFO:teuthology.orchestra.run.smithi007.stdout:{ 2024-09-18T08:09:12.506 INFO:teuthology.orchestra.run.smithi007.stdout: "ops": [], 2024-09-18T08:09:12.506 INFO:teuthology.orchestra.run.smithi007.stdout: "num_ops": 0 2024-09-18T08:09:12.506 INFO:teuthology.orchestra.run.smithi007.stdout:} 2024-09-18T08:09:12.588 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5 2024-09-18T08:09:12.613 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:09:12.614 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:09:12.615 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5' 2024-09-18T08:09:12.655 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:09:12 smithi050 ceph-mon[32371]: pgmap v311: 158 pgs: 1 active+remapped+backfilling, 1 active+recovering+remapped, 2 active+recovering+undersized+remapped, 154 active+clean; 5.0 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 70 MiB/s wr, 1.12k op/s; 7678/248421 objects misplaced (3.091%); 22 MiB/s, 359 objects/s recovering 2024-09-18T08:09:12.655 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:09:12 smithi050 ceph-mon[32371]: 5.1d scrub starts 2024-09-18T08:09:12.655 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:09:12 smithi050 ceph-mon[32371]: 5.1d scrub ok 2024-09-18T08:09:12.716 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5 2024-09-18T08:09:12.739 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:09:12.740 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:09:12.741 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5' 2024-09-18T08:09:12.841 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6 2024-09-18T08:09:12.866 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:09:12.867 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:09:12.867 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6' 2024-09-18T08:09:12.967 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4 2024-09-18T08:09:12.990 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:09:12.991 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:09:12.992 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4' 2024-09-18T08:09:13.093 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7 2024-09-18T08:09:13.117 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:09:13.117 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:09:13.118 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7' 2024-09-18T08:09:13.161 INFO:teuthology.orchestra.run.smithi007.stderr:noscrub is set 2024-09-18T08:09:13.180 INFO:tasks.radosbench.radosbench.0.smithi007.stdout: 75 16 86500 86484 72.0427 16.25 1.02168 0.0137343 2024-09-18T08:09:13.219 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0 2024-09-18T08:09:13.257 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:09:13.258 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:09:13.259 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0' 2024-09-18T08:09:13.359 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4 2024-09-18T08:09:13.398 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:09:13.399 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:09:13.399 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4' 2024-09-18T08:09:13.480 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:09:13 smithi007 ceph-mon[29459]: osdmap e205: 8 total, 8 up, 8 in 2024-09-18T08:09:13.480 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:09:13 smithi007 ceph-mon[29459]: from='client.? 172.21.15.7:0/523412509' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-18T08:09:13.481 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:09:13 smithi007 ceph-mon[29459]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-18T08:09:13.481 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:09:13 smithi007 ceph-mon[39249]: osdmap e205: 8 total, 8 up, 8 in 2024-09-18T08:09:13.481 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:09:13 smithi007 ceph-mon[39249]: from='client.? 172.21.15.7:0/523412509' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-18T08:09:13.481 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:09:13 smithi007 ceph-mon[39249]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-18T08:09:13.500 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6 2024-09-18T08:09:13.538 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:09:13.539 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:09:13.539 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6' 2024-09-18T08:09:13.640 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0 2024-09-18T08:09:13.656 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:09:13 smithi050 ceph-mon[32371]: osdmap e205: 8 total, 8 up, 8 in 2024-09-18T08:09:13.656 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:09:13 smithi050 ceph-mon[32371]: from='client.? 172.21.15.7:0/523412509' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-18T08:09:13.656 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:09:13 smithi050 ceph-mon[32371]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-18T08:09:13.681 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:09:13.682 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:09:13.682 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0' 2024-09-18T08:09:13.783 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5 2024-09-18T08:09:13.822 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:09:13.823 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:09:13.823 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5' 2024-09-18T08:09:13.924 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2 2024-09-18T08:09:13.964 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:09:13.964 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:09:13.965 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2' 2024-09-18T08:09:14.066 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4 2024-09-18T08:09:14.097 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:09:14.097 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:09:14.098 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4' 2024-09-18T08:09:14.181 INFO:tasks.radosbench.radosbench.0.smithi007.stdout: 76 16 87516 87500 71.9303 63.5 0.0119128 0.0138958 2024-09-18T08:09:14.198 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2 2024-09-18T08:09:14.250 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:09:14.250 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:09:14.251 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2' 2024-09-18T08:09:14.313 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid 779349ca-7593-11ef-bceb-c7b262605968 -- ceph daemon osd.2 dump_blocked_ops 2024-09-18T08:09:14.352 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0 2024-09-18T08:09:14.398 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:09:14.399 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:09:14.399 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0' 2024-09-18T08:09:14.406 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:09:14 smithi050 ceph-mon[32371]: 5.1a scrub starts 2024-09-18T08:09:14.406 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:09:14 smithi050 ceph-mon[32371]: 5.1a scrub ok 2024-09-18T08:09:14.406 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:09:14 smithi050 ceph-mon[32371]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-18T08:09:14.406 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:09:14 smithi050 ceph-mon[32371]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-18T08:09:14.406 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:09:14 smithi050 ceph-mon[32371]: osdmap e206: 8 total, 8 up, 8 in 2024-09-18T08:09:14.406 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:09:14 smithi050 ceph-mon[32371]: pgmap v314: 158 pgs: 1 creating+activating, 1 active+remapped+backfilling, 2 active+recovering+undersized+remapped, 154 active+clean; 5.1 GiB data, 18 GiB used, 698 GiB / 715 GiB avail; 1005 B/s rd, 65 MiB/s wr, 1.04k op/s; 7427/249843 objects misplaced (2.973%); 21 MiB/s, 337 objects/s recovering 2024-09-18T08:09:14.480 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:09:14 smithi007 ceph-mon[29459]: 5.1a scrub starts 2024-09-18T08:09:14.480 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:09:14 smithi007 ceph-mon[29459]: 5.1a scrub ok 2024-09-18T08:09:14.481 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:09:14 smithi007 ceph-mon[29459]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-18T08:09:14.481 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:09:14 smithi007 ceph-mon[29459]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-18T08:09:14.481 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:09:14 smithi007 ceph-mon[29459]: osdmap e206: 8 total, 8 up, 8 in 2024-09-18T08:09:14.481 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:09:14 smithi007 ceph-mon[29459]: pgmap v314: 158 pgs: 1 creating+activating, 1 active+remapped+backfilling, 2 active+recovering+undersized+remapped, 154 active+clean; 5.1 GiB data, 18 GiB used, 698 GiB / 715 GiB avail; 1005 B/s rd, 65 MiB/s wr, 1.04k op/s; 7427/249843 objects misplaced (2.973%); 21 MiB/s, 337 objects/s recovering 2024-09-18T08:09:14.481 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:09:14 smithi007 ceph-mon[39249]: 5.1a scrub starts 2024-09-18T08:09:14.481 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:09:14 smithi007 ceph-mon[39249]: 5.1a scrub ok 2024-09-18T08:09:14.481 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:09:14 smithi007 ceph-mon[39249]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-18T08:09:14.481 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:09:14 smithi007 ceph-mon[39249]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-18T08:09:14.481 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:09:14 smithi007 ceph-mon[39249]: osdmap e206: 8 total, 8 up, 8 in 2024-09-18T08:09:14.481 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:09:14 smithi007 ceph-mon[39249]: pgmap v314: 158 pgs: 1 creating+activating, 1 active+remapped+backfilling, 2 active+recovering+undersized+remapped, 154 active+clean; 5.1 GiB data, 18 GiB used, 698 GiB / 715 GiB avail; 1005 B/s rd, 65 MiB/s wr, 1.04k op/s; 7427/249843 objects misplaced (2.973%); 21 MiB/s, 337 objects/s recovering 2024-09-18T08:09:14.500 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0 2024-09-18T08:09:14.556 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:09:14.556 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:09:14.556 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0' 2024-09-18T08:09:14.657 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7 2024-09-18T08:09:14.693 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:09:14.694 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:09:14.694 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7' 2024-09-18T08:09:14.795 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4 2024-09-18T08:09:14.837 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:09:14.837 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:09:14.838 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4' 2024-09-18T08:09:14.906 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid 779349ca-7593-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-18T08:09:14.939 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5 2024-09-18T08:09:14.970 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:09:14.971 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:09:14.971 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5' 2024-09-18T08:09:15.072 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2 2024-09-18T08:09:15.158 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:09:15.159 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:09:15.159 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2' 2024-09-18T08:09:15.183 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/779349ca-7593-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T08:09:15.183 INFO:tasks.radosbench.radosbench.0.smithi007.stdout: 77 16 88545 88529 71.8308 64.3125 0.0104582 0.0139146 2024-09-18T08:09:15.260 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1 2024-09-18T08:09:15.321 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:09:15.322 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:09:15.322 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1' 2024-09-18T08:09:15.423 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1 2024-09-18T08:09:15.462 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:09:15.463 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:09:15.463 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1' 2024-09-18T08:09:15.564 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0 2024-09-18T08:09:15.616 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:09:15.649 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:09:15.649 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0' 2024-09-18T08:09:15.750 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5 2024-09-18T08:09:15.779 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:09:15.780 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:09:15.780 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5' 2024-09-18T08:09:15.868 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/779349ca-7593-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T08:09:15.880 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1 2024-09-18T08:09:15.949 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:09:15.950 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:09:15.950 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1' 2024-09-18T08:09:16.051 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5 2024-09-18T08:09:16.081 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:09:16.082 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:09:16.082 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5' 2024-09-18T08:09:16.182 INFO:tasks.radosbench.radosbench.0.smithi007.stdout: 78 16 89589 89573 71.7463 65.25 0.027771 0.0139325 2024-09-18T08:09:16.183 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4 2024-09-18T08:09:16.197 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/779349ca-7593-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T08:09:16.211 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:09:16.212 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:09:16.212 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4' 2024-09-18T08:09:16.264 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:09:16 smithi007 ceph-mon[29459]: pgmap v315: 158 pgs: 1 active+remapped+backfilling, 1 creating+activating, 2 active+recovering+undersized+remapped, 154 active+clean; 5.3 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 852 B/s rd, 43 MiB/s wr, 695 op/s; 5799/260937 objects misplaced (2.222%); 21 MiB/s, 333 objects/s recovering 2024-09-18T08:09:16.265 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:09:16 smithi007 ceph-mon[39249]: pgmap v315: 158 pgs: 1 active+remapped+backfilling, 1 creating+activating, 2 active+recovering+undersized+remapped, 154 active+clean; 5.3 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 852 B/s rd, 43 MiB/s wr, 695 op/s; 5799/260937 objects misplaced (2.222%); 21 MiB/s, 333 objects/s recovering 2024-09-18T08:09:16.313 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1 2024-09-18T08:09:16.362 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:09:16.363 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:09:16.363 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1' 2024-09-18T08:09:16.463 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1 2024-09-18T08:09:16.497 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:09:16.497 INFO:teuthology.orchestra.run.smithi007.stdout:{ 2024-09-18T08:09:16.497 INFO:teuthology.orchestra.run.smithi007.stdout: "ops": [], 2024-09-18T08:09:16.497 INFO:teuthology.orchestra.run.smithi007.stdout: "complaint_time": 30, 2024-09-18T08:09:16.498 INFO:teuthology.orchestra.run.smithi007.stdout: "num_blocked_ops": 0 2024-09-18T08:09:16.498 INFO:teuthology.orchestra.run.smithi007.stdout:} 2024-09-18T08:09:16.498 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:09:16.499 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1' 2024-09-18T08:09:16.599 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7 2024-09-18T08:09:16.622 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:09:16.623 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:09:16.623 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7' 2024-09-18T08:09:16.656 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:09:16 smithi050 ceph-mon[32371]: pgmap v315: 158 pgs: 1 active+remapped+backfilling, 1 creating+activating, 2 active+recovering+undersized+remapped, 154 active+clean; 5.3 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 852 B/s rd, 43 MiB/s wr, 695 op/s; 5799/260937 objects misplaced (2.222%); 21 MiB/s, 333 objects/s recovering 2024-09-18T08:09:16.723 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7 2024-09-18T08:09:16.747 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:09:16.747 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:09:16.748 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7' 2024-09-18T08:09:16.849 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7 2024-09-18T08:09:16.871 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:09:16.872 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:09:16.873 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7' 2024-09-18T08:09:16.973 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2 2024-09-18T08:09:16.980 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Sep 18 08:09:16 smithi007 ceph-779349ca-7593-11ef-bceb-c7b262605968-mgr-y[29688]: ::ffff:172.21.15.50 - - [18/Sep/2024:08:09:16] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-09-18T08:09:17.000 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:09:17.001 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:09:17.001 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2' 2024-09-18T08:09:17.102 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1 2024-09-18T08:09:17.129 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:09:17.130 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:09:17.130 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1' 2024-09-18T08:09:17.168 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid 779349ca-7593-11ef-bceb-c7b262605968 -- ceph osd set nodeep-scrub 2024-09-18T08:09:17.181 INFO:tasks.radosbench.radosbench.0.smithi007.stdout: 79 16 89909 89893 71.0915 20 0.00340103 0.0139298 2024-09-18T08:09:17.231 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2 2024-09-18T08:09:17.246 INFO:teuthology.orchestra.run.smithi007.stdout:osd.0: {} 2024-09-18T08:09:17.246 INFO:teuthology.orchestra.run.smithi007.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-18T08:09:17.258 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:09:17.259 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:09:17.259 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2' 2024-09-18T08:09:17.280 INFO:teuthology.orchestra.run.smithi007.stderr:reweighted osd.7 to 1 (10000) 2024-09-18T08:09:17.283 INFO:teuthology.orchestra.run.smithi007.stdout:osd.1: {} 2024-09-18T08:09:17.283 INFO:teuthology.orchestra.run.smithi007.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-18T08:09:17.323 INFO:teuthology.orchestra.run.smithi007.stdout:osd.2: {} 2024-09-18T08:09:17.324 INFO:teuthology.orchestra.run.smithi007.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-18T08:09:17.360 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7 2024-09-18T08:09:17.365 INFO:teuthology.orchestra.run.smithi007.stdout:osd.3: {} 2024-09-18T08:09:17.365 INFO:teuthology.orchestra.run.smithi007.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-18T08:09:17.365 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:09:17 smithi007 ceph-mon[29459]: osdmap e207: 8 total, 8 up, 8 in 2024-09-18T08:09:17.365 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:09:17 smithi007 ceph-mon[29459]: from='client.? 172.21.15.7:0/3234219379' entity='client.admin' cmd={"prefix": "osd reweight", "id": 7, "weight": 1.0} : dispatch 2024-09-18T08:09:17.366 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:09:17 smithi007 ceph-mon[29459]: from='client.? 172.21.15.7:0/2090430791' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-18T08:09:17.366 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:09:17 smithi007 ceph-mon[39249]: osdmap e207: 8 total, 8 up, 8 in 2024-09-18T08:09:17.366 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:09:17 smithi007 ceph-mon[39249]: from='client.? 172.21.15.7:0/3234219379' entity='client.admin' cmd={"prefix": "osd reweight", "id": 7, "weight": 1.0} : dispatch 2024-09-18T08:09:17.366 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:09:17 smithi007 ceph-mon[39249]: from='client.? 172.21.15.7:0/2090430791' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-18T08:09:17.389 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:09:17.390 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:09:17.391 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7' 2024-09-18T08:09:17.404 INFO:teuthology.orchestra.run.smithi007.stdout:osd.4: {} 2024-09-18T08:09:17.404 INFO:teuthology.orchestra.run.smithi007.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-18T08:09:17.453 INFO:teuthology.orchestra.run.smithi007.stdout:osd.5: {} 2024-09-18T08:09:17.454 INFO:teuthology.orchestra.run.smithi007.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-18T08:09:17.488 INFO:teuthology.orchestra.run.smithi007.stdout:osd.6: {} 2024-09-18T08:09:17.488 INFO:teuthology.orchestra.run.smithi007.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-18T08:09:17.492 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7 2024-09-18T08:09:17.522 INFO:teuthology.orchestra.run.smithi007.stdout:osd.7: {} 2024-09-18T08:09:17.522 INFO:teuthology.orchestra.run.smithi007.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-18T08:09:17.526 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:09:17.527 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:09:17.527 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7' 2024-09-18T08:09:17.628 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2 2024-09-18T08:09:17.656 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:09:17 smithi050 ceph-mon[32371]: osdmap e207: 8 total, 8 up, 8 in 2024-09-18T08:09:17.656 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:09:17 smithi050 ceph-mon[32371]: from='client.? 172.21.15.7:0/3234219379' entity='client.admin' cmd={"prefix": "osd reweight", "id": 7, "weight": 1.0} : dispatch 2024-09-18T08:09:17.656 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:09:17 smithi050 ceph-mon[32371]: from='client.? 172.21.15.7:0/2090430791' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-18T08:09:17.665 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:09:17.666 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:09:17.666 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2' 2024-09-18T08:09:17.767 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5 2024-09-18T08:09:17.800 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:09:17.801 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:09:17.801 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5' 2024-09-18T08:09:17.803 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid 779349ca-7593-11ef-bceb-c7b262605968 -- ceph daemon osd.2 dump_historic_ops 2024-09-18T08:09:17.902 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7 2024-09-18T08:09:17.934 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:09:17.935 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:09:17.935 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7' 2024-09-18T08:09:18.036 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7 2024-09-18T08:09:18.072 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:09:18.073 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:09:18.073 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7' 2024-09-18T08:09:18.174 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2 2024-09-18T08:09:18.183 INFO:tasks.radosbench.radosbench.0.smithi007.stdout:2024-09-18T08:09:18.181059+0000 min lat: 0.00120772 max lat: 2.15499 avg lat: 0.0140948 2024-09-18T08:09:18.183 INFO:tasks.radosbench.radosbench.0.smithi007.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-09-18T08:09:18.183 INFO:tasks.radosbench.radosbench.0.smithi007.stdout: 80 16 90824 90808 70.9176 57.1875 0.0140642 0.0140948 2024-09-18T08:09:18.222 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:09:18.223 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:09:18.223 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2' 2024-09-18T08:09:18.324 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1 2024-09-18T08:09:18.370 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:09:18.370 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:09:18.371 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1' 2024-09-18T08:09:18.472 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4 2024-09-18T08:09:18.506 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:09:18.507 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:09:18.507 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4' 2024-09-18T08:09:18.603 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:09:18 smithi007 ceph-mon[29459]: purged_snaps scrub starts 2024-09-18T08:09:18.604 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:09:18 smithi007 ceph-mon[29459]: purged_snaps scrub ok 2024-09-18T08:09:18.604 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:09:18 smithi007 ceph-mon[29459]: pgmap v317: 158 pgs: 1 active+remapped+backfilling, 1 creating+activating, 2 active+recovering+undersized+remapped, 154 active+clean; 5.3 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 853 B/s rd, 43 MiB/s wr, 695 op/s; 5799/260937 objects misplaced (2.222%); 21 MiB/s, 333 objects/s recovering 2024-09-18T08:09:18.604 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:09:18 smithi007 ceph-mon[29459]: from='client.? 172.21.15.7:0/3234219379' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]': finished 2024-09-18T08:09:18.604 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:09:18 smithi007 ceph-mon[29459]: osdmap e208: 8 total, 8 up, 8 in 2024-09-18T08:09:18.604 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:09:18 smithi007 ceph-mon[29459]: from='mgr.24470 172.21.15.7:0/2594390261' entity='mgr.y' 2024-09-18T08:09:18.604 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:09:18 smithi007 ceph-mon[29459]: from='mgr.24470 172.21.15.7:0/2594390261' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-18T08:09:18.604 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:09:18 smithi007 ceph-mon[39249]: purged_snaps scrub starts 2024-09-18T08:09:18.604 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:09:18 smithi007 ceph-mon[39249]: purged_snaps scrub ok 2024-09-18T08:09:18.604 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:09:18 smithi007 ceph-mon[39249]: pgmap v317: 158 pgs: 1 active+remapped+backfilling, 1 creating+activating, 2 active+recovering+undersized+remapped, 154 active+clean; 5.3 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 853 B/s rd, 43 MiB/s wr, 695 op/s; 5799/260937 objects misplaced (2.222%); 21 MiB/s, 333 objects/s recovering 2024-09-18T08:09:18.605 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:09:18 smithi007 ceph-mon[39249]: from='client.? 172.21.15.7:0/3234219379' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]': finished 2024-09-18T08:09:18.605 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:09:18 smithi007 ceph-mon[39249]: osdmap e208: 8 total, 8 up, 8 in 2024-09-18T08:09:18.605 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:09:18 smithi007 ceph-mon[39249]: from='mgr.24470 172.21.15.7:0/2594390261' entity='mgr.y' 2024-09-18T08:09:18.605 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:09:18 smithi007 ceph-mon[39249]: from='mgr.24470 172.21.15.7:0/2594390261' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-18T08:09:18.608 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3 2024-09-18T08:09:18.656 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:09:18 smithi050 ceph-mon[32371]: purged_snaps scrub starts 2024-09-18T08:09:18.656 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:09:18 smithi050 ceph-mon[32371]: purged_snaps scrub ok 2024-09-18T08:09:18.656 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:09:18 smithi050 ceph-mon[32371]: pgmap v317: 158 pgs: 1 active+remapped+backfilling, 1 creating+activating, 2 active+recovering+undersized+remapped, 154 active+clean; 5.3 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 853 B/s rd, 43 MiB/s wr, 695 op/s; 5799/260937 objects misplaced (2.222%); 21 MiB/s, 333 objects/s recovering 2024-09-18T08:09:18.656 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:09:18 smithi050 ceph-mon[32371]: from='client.? 172.21.15.7:0/3234219379' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]': finished 2024-09-18T08:09:18.656 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:09:18 smithi050 ceph-mon[32371]: osdmap e208: 8 total, 8 up, 8 in 2024-09-18T08:09:18.656 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:09:18 smithi050 ceph-mon[32371]: from='mgr.24470 172.21.15.7:0/2594390261' entity='mgr.y' 2024-09-18T08:09:18.656 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:09:18 smithi050 ceph-mon[32371]: from='mgr.24470 172.21.15.7:0/2594390261' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-18T08:09:18.660 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid 779349ca-7593-11ef-bceb-c7b262605968 -- ceph osd reweight 6 1 2024-09-18T08:09:18.670 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:09:18.671 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:09:18.671 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3' 2024-09-18T08:09:18.772 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4 2024-09-18T08:09:18.808 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:09:18.809 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:09:18.809 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4' 2024-09-18T08:09:18.910 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1 2024-09-18T08:09:18.964 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:09:18.965 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:09:18.965 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1' 2024-09-18T08:09:19.067 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2 2024-09-18T08:09:19.114 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:09:19.115 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:09:19.115 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2' 2024-09-18T08:09:19.182 INFO:tasks.radosbench.radosbench.0.smithi007.stdout: 81 16 91899 91883 70.8709 67.1875 0.0101132 0.0141048 2024-09-18T08:09:19.196 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/779349ca-7593-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T08:09:19.216 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1 2024-09-18T08:09:19.247 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/779349ca-7593-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T08:09:19.296 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:09:19.297 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:09:19.297 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1' 2024-09-18T08:09:19.398 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6 2024-09-18T08:09:19.405 INFO:journalctl@ceph.iscsi.iscsi.a.smithi050.stdout:Sep 18 08:09:19 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-iscsi-iscsi-a[57782]: debug there is no tcmu-runner data available 2024-09-18T08:09:19.429 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:09:19.430 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:09:19.430 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6' 2024-09-18T08:09:19.531 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0 2024-09-18T08:09:19.575 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:09:19.576 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:09:19.577 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0' 2024-09-18T08:09:19.678 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3 2024-09-18T08:09:19.712 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:09:19.713 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:09:19.713 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3' 2024-09-18T08:09:19.814 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5 2024-09-18T08:09:19.838 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:09:19.838 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:09:19.839 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5' 2024-09-18T08:09:19.939 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3 2024-09-18T08:09:19.982 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:09:19.983 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:09:19.983 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3' 2024-09-18T08:09:20.054 INFO:teuthology.orchestra.run.smithi007.stdout:ERROR: (22) Invalid argument 2024-09-18T08:09:20.055 INFO:teuthology.orchestra.run.smithi007.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-18T08:09:20.084 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6 2024-09-18T08:09:20.113 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:09:20.114 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:09:20.114 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6' 2024-09-18T08:09:20.182 INFO:tasks.radosbench.radosbench.0.smithi007.stdout: 82 16 92937 92921 70.7977 64.875 0.0133301 0.0141193 2024-09-18T08:09:20.215 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3 2024-09-18T08:09:20.257 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:09:20.257 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:09:20.258 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3' 2024-09-18T08:09:20.359 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5 2024-09-18T08:09:20.392 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:09:20.392 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:09:20.393 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5' 2024-09-18T08:09:20.493 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/779349ca-7593-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T08:09:20.493 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2 2024-09-18T08:09:20.547 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:09:20.548 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:09:20.548 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2' 2024-09-18T08:09:20.649 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4 2024-09-18T08:09:20.651 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:09:20 smithi007 ceph-mon[29459]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-18T08:09:20.651 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:09:20 smithi007 ceph-mon[29459]: pgmap v319: 158 pgs: 1 active+remapped+backfilling, 1 active+recovering+undersized+remapped, 156 active+clean; 5.6 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 1.7 KiB/s rd, 89 MiB/s wr, 1.42k op/s; 4326/274128 objects misplaced (1.578%); 37 MiB/s, 587 objects/s recovering 2024-09-18T08:09:20.652 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:09:20 smithi007 ceph-mon[39249]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-18T08:09:20.652 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:09:20 smithi007 ceph-mon[39249]: pgmap v319: 158 pgs: 1 active+remapped+backfilling, 1 active+recovering+undersized+remapped, 156 active+clean; 5.6 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 1.7 KiB/s rd, 89 MiB/s wr, 1.42k op/s; 4326/274128 objects misplaced (1.578%); 37 MiB/s, 587 objects/s recovering 2024-09-18T08:09:20.655 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:09:20 smithi050 ceph-mon[32371]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-18T08:09:20.656 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:09:20 smithi050 ceph-mon[32371]: pgmap v319: 158 pgs: 1 active+remapped+backfilling, 1 active+recovering+undersized+remapped, 156 active+clean; 5.6 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 1.7 KiB/s rd, 89 MiB/s wr, 1.42k op/s; 4326/274128 objects misplaced (1.578%); 37 MiB/s, 587 objects/s recovering 2024-09-18T08:09:20.681 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:09:20.682 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:09:20.682 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4' 2024-09-18T08:09:20.783 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6 2024-09-18T08:09:20.828 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:09:20.829 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:09:20.829 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6' 2024-09-18T08:09:20.930 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0 2024-09-18T08:09:20.981 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:09:20.982 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:09:20.983 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0' 2024-09-18T08:09:20.993 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid 779349ca-7593-11ef-bceb-c7b262605968 -- ceph daemon osd.3 dump_ops_in_flight 2024-09-18T08:09:21.083 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4 2024-09-18T08:09:21.115 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:09:21.115 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:09:21.115 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4' 2024-09-18T08:09:21.173 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid 779349ca-7593-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-18T08:09:21.182 INFO:tasks.radosbench.radosbench.0.smithi007.stdout: 83 16 94039 94023 70.7744 68.875 0.0122946 0.0141238 2024-09-18T08:09:21.216 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1 2024-09-18T08:09:21.322 INFO:teuthology.orchestra.run.smithi007.stderr:nodeep-scrub is set 2024-09-18T08:09:21.329 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:09:21.330 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:09:21.330 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1' 2024-09-18T08:09:21.431 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4 2024-09-18T08:09:21.457 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:09:21.457 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:09:21.458 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4' 2024-09-18T08:09:21.481 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:09:21 smithi007 ceph-mon[29459]: from='client.? 172.21.15.7:0/3056462288' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-18T08:09:21.481 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:09:21 smithi007 ceph-mon[29459]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-18T08:09:21.481 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:09:21 smithi007 ceph-mon[39249]: from='client.? 172.21.15.7:0/3056462288' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-18T08:09:21.481 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:09:21 smithi007 ceph-mon[39249]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-18T08:09:21.559 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3 2024-09-18T08:09:21.608 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:09:21.608 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:09:21.609 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3' 2024-09-18T08:09:21.656 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:09:21 smithi050 ceph-mon[32371]: from='client.? 172.21.15.7:0/3056462288' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-18T08:09:21.656 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:09:21 smithi050 ceph-mon[32371]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-18T08:09:21.709 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4 2024-09-18T08:09:21.742 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:09:21.743 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:09:21.743 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4' 2024-09-18T08:09:21.844 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6 2024-09-18T08:09:21.875 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:09:21.875 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:09:21.876 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6' 2024-09-18T08:09:21.976 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3 2024-09-18T08:09:22.014 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:09:22.014 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:09:22.015 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3' 2024-09-18T08:09:22.115 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4 2024-09-18T08:09:22.152 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:09:22.153 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:09:22.153 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4' 2024-09-18T08:09:22.183 INFO:tasks.radosbench.radosbench.0.smithi007.stdout: 84 16 95027 95011 70.667 61.75 0.0135284 0.0141454 2024-09-18T08:09:22.254 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6 2024-09-18T08:09:22.285 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:09:22.286 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:09:22.286 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6' 2024-09-18T08:09:22.325 INFO:teuthology.orchestra.run.smithi007.stderr:reweighted osd.6 to 1 (10000) 2024-09-18T08:09:22.387 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0 2024-09-18T08:09:22.437 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:09:22.438 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:09:22.438 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0' 2024-09-18T08:09:22.539 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6 2024-09-18T08:09:22.578 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:09:22.578 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:09:22.578 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6' 2024-09-18T08:09:22.656 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:09:22 smithi050 ceph-mon[32371]: pgmap v320: 158 pgs: 1 active+remapped+backfilling, 1 active+recovering+undersized+remapped, 156 active+clean; 5.6 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 1.2 KiB/s rd, 63 MiB/s wr, 1.01k op/s; 4326/274128 objects misplaced (1.578%); 26 MiB/s, 412 objects/s recovering 2024-09-18T08:09:22.656 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:09:22 smithi050 ceph-mon[32371]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-18T08:09:22.656 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:09:22 smithi050 ceph-mon[32371]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-18T08:09:22.656 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:09:22 smithi050 ceph-mon[32371]: osdmap e209: 8 total, 8 up, 8 in 2024-09-18T08:09:22.656 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:09:22 smithi050 ceph-mon[32371]: from='client.? 172.21.15.7:0/1021840711' entity='client.admin' cmd={"prefix": "osd reweight", "id": 6, "weight": 1.0} : dispatch 2024-09-18T08:09:22.656 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:09:22 smithi050 ceph-mon[32371]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 6, "weight": 1.0} : dispatch 2024-09-18T08:09:22.680 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3 2024-09-18T08:09:22.705 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:09:22 smithi007 ceph-mon[29459]: pgmap v320: 158 pgs: 1 active+remapped+backfilling, 1 active+recovering+undersized+remapped, 156 active+clean; 5.6 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 1.2 KiB/s rd, 63 MiB/s wr, 1.01k op/s; 4326/274128 objects misplaced (1.578%); 26 MiB/s, 412 objects/s recovering 2024-09-18T08:09:22.705 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:09:22 smithi007 ceph-mon[29459]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-18T08:09:22.705 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:09:22 smithi007 ceph-mon[29459]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-18T08:09:22.705 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:09:22 smithi007 ceph-mon[29459]: osdmap e209: 8 total, 8 up, 8 in 2024-09-18T08:09:22.705 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:09:22 smithi007 ceph-mon[29459]: from='client.? 172.21.15.7:0/1021840711' entity='client.admin' cmd={"prefix": "osd reweight", "id": 6, "weight": 1.0} : dispatch 2024-09-18T08:09:22.705 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:09:22 smithi007 ceph-mon[29459]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 6, "weight": 1.0} : dispatch 2024-09-18T08:09:22.705 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:09:22 smithi007 ceph-mon[39249]: pgmap v320: 158 pgs: 1 active+remapped+backfilling, 1 active+recovering+undersized+remapped, 156 active+clean; 5.6 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 1.2 KiB/s rd, 63 MiB/s wr, 1.01k op/s; 4326/274128 objects misplaced (1.578%); 26 MiB/s, 412 objects/s recovering 2024-09-18T08:09:22.705 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:09:22 smithi007 ceph-mon[39249]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-18T08:09:22.706 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:09:22 smithi007 ceph-mon[39249]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-18T08:09:22.706 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:09:22 smithi007 ceph-mon[39249]: osdmap e209: 8 total, 8 up, 8 in 2024-09-18T08:09:22.706 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:09:22 smithi007 ceph-mon[39249]: from='client.? 172.21.15.7:0/1021840711' entity='client.admin' cmd={"prefix": "osd reweight", "id": 6, "weight": 1.0} : dispatch 2024-09-18T08:09:22.706 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:09:22 smithi007 ceph-mon[39249]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 6, "weight": 1.0} : dispatch 2024-09-18T08:09:22.725 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:09:22.726 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:09:22.727 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3' 2024-09-18T08:09:22.827 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6 2024-09-18T08:09:22.861 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:09:22.861 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:09:22.862 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6' 2024-09-18T08:09:22.963 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4 2024-09-18T08:09:22.997 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:09:22.998 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:09:22.998 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4' 2024-09-18T08:09:23.043 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/779349ca-7593-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T08:09:23.089 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid 779349ca-7593-11ef-bceb-c7b262605968 -- ceph osd reweight 4 1 2024-09-18T08:09:23.099 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4 2024-09-18T08:09:23.123 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/779349ca-7593-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T08:09:23.130 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:09:23.131 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:09:23.131 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4' 2024-09-18T08:09:23.183 INFO:tasks.radosbench.radosbench.0.smithi007.stdout: 85 16 96148 96132 70.6598 70.0625 0.00971188 0.0141469 2024-09-18T08:09:23.231 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6 2024-09-18T08:09:23.263 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:09:23.264 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:09:23.265 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6' 2024-09-18T08:09:23.335 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:09:23 smithi007 ceph-mon[39249]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]': finished 2024-09-18T08:09:23.335 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:09:23 smithi007 ceph-mon[39249]: osdmap e210: 8 total, 8 up, 8 in 2024-09-18T08:09:23.366 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4 2024-09-18T08:09:23.393 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:09:23.394 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:09:23.394 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4' 2024-09-18T08:09:23.495 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4 2024-09-18T08:09:23.530 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:09:23.530 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:09:23.531 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4' 2024-09-18T08:09:23.632 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2 2024-09-18T08:09:23.656 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:09:23 smithi050 ceph-mon[32371]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]': finished 2024-09-18T08:09:23.656 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:09:23 smithi050 ceph-mon[32371]: osdmap e210: 8 total, 8 up, 8 in 2024-09-18T08:09:23.672 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:09:23 smithi007 ceph-mon[29459]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]': finished 2024-09-18T08:09:23.672 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:09:23 smithi007 ceph-mon[29459]: osdmap e210: 8 total, 8 up, 8 in 2024-09-18T08:09:23.673 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:09:23.673 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:09:23.674 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2' 2024-09-18T08:09:23.775 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7 2024-09-18T08:09:23.806 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:09:23.807 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:09:23.807 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7' 2024-09-18T08:09:23.908 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4 2024-09-18T08:09:23.941 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:09:23.941 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:09:23.942 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4' 2024-09-18T08:09:24.019 INFO:teuthology.orchestra.run.smithi007.stdout:ERROR: (22) Invalid argument 2024-09-18T08:09:24.019 INFO:teuthology.orchestra.run.smithi007.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-18T08:09:24.042 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1 2024-09-18T08:09:24.091 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:09:24.091 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:09:24.092 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1' 2024-09-18T08:09:24.183 INFO:tasks.radosbench.radosbench.0.smithi007.stdout: 86 16 97211 97195 70.6106 66.4375 0.0206655 0.0141569 2024-09-18T08:09:24.194 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3 2024-09-18T08:09:24.233 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:09:24.234 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:09:24.234 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3' 2024-09-18T08:09:24.335 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2 2024-09-18T08:09:24.341 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid 779349ca-7593-11ef-bceb-c7b262605968 -- ceph osd unset noscrub 2024-09-18T08:09:24.377 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:09:24.378 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:09:24.379 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2' 2024-09-18T08:09:24.406 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:09:24 smithi050 ceph-mon[32371]: pgmap v323: 158 pgs: 1 active+remapped+backfilling, 1 active+recovering+undersized+remapped, 156 active+clean; 5.6 GiB data, 20 GiB used, 696 GiB / 715 GiB avail; 852 B/s rd, 53 MiB/s wr, 847 op/s; 4326/276189 objects misplaced (1.566%); 16 MiB/s, 258 objects/s recovering 2024-09-18T08:09:24.479 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1 2024-09-18T08:09:24.524 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:09:24.525 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:09:24.525 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1' 2024-09-18T08:09:24.626 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4 2024-09-18T08:09:24.663 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:09:24.664 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:09:24.664 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4' 2024-09-18T08:09:24.730 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:09:24 smithi007 ceph-mon[39249]: pgmap v323: 158 pgs: 1 active+remapped+backfilling, 1 active+recovering+undersized+remapped, 156 active+clean; 5.6 GiB data, 20 GiB used, 696 GiB / 715 GiB avail; 852 B/s rd, 53 MiB/s wr, 847 op/s; 4326/276189 objects misplaced (1.566%); 16 MiB/s, 258 objects/s recovering 2024-09-18T08:09:24.731 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:09:24 smithi007 ceph-mon[29459]: pgmap v323: 158 pgs: 1 active+remapped+backfilling, 1 active+recovering+undersized+remapped, 156 active+clean; 5.6 GiB data, 20 GiB used, 696 GiB / 715 GiB avail; 852 B/s rd, 53 MiB/s wr, 847 op/s; 4326/276189 objects misplaced (1.566%); 16 MiB/s, 258 objects/s recovering 2024-09-18T08:09:24.764 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2 2024-09-18T08:09:24.812 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:09:24.814 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:09:24.815 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2' 2024-09-18T08:09:24.916 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3 2024-09-18T08:09:24.959 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:09:24.960 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:09:24.960 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3' 2024-09-18T08:09:25.020 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid 779349ca-7593-11ef-bceb-c7b262605968 -- ceph daemon osd.3 dump_blocked_ops 2024-09-18T08:09:25.061 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1 2024-09-18T08:09:25.114 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:09:25.115 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:09:25.115 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1' 2024-09-18T08:09:25.183 INFO:tasks.radosbench.radosbench.0.smithi007.stdout: 87 16 98279 98263 70.5661 66.75 0.0219975 0.0141665 2024-09-18T08:09:25.216 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5 2024-09-18T08:09:25.249 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:09:25.250 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:09:25.250 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5' 2024-09-18T08:09:25.351 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6 2024-09-18T08:09:25.388 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:09:25.389 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:09:25.389 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6' 2024-09-18T08:09:25.433 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/779349ca-7593-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T08:09:25.490 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3 2024-09-18T08:09:25.520 INFO:teuthology.orchestra.run.smithi007.stdout:osd.0: {} 2024-09-18T08:09:25.521 INFO:teuthology.orchestra.run.smithi007.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-18T08:09:25.561 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:09:25.562 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:09:25.562 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3' 2024-09-18T08:09:25.565 INFO:teuthology.orchestra.run.smithi007.stdout:osd.1: {} 2024-09-18T08:09:25.565 INFO:teuthology.orchestra.run.smithi007.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-18T08:09:25.606 INFO:teuthology.orchestra.run.smithi007.stdout:osd.2: {} 2024-09-18T08:09:25.606 INFO:teuthology.orchestra.run.smithi007.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-18T08:09:25.641 INFO:teuthology.orchestra.run.smithi007.stdout:osd.3: {} 2024-09-18T08:09:25.641 INFO:teuthology.orchestra.run.smithi007.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-18T08:09:25.664 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1 2024-09-18T08:09:25.682 INFO:teuthology.orchestra.run.smithi007.stdout:osd.4: {} 2024-09-18T08:09:25.682 INFO:teuthology.orchestra.run.smithi007.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-18T08:09:25.716 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:09:25.716 INFO:teuthology.orchestra.run.smithi007.stdout:osd.5: {} 2024-09-18T08:09:25.716 INFO:teuthology.orchestra.run.smithi007.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-18T08:09:25.717 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:09:25.717 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1' 2024-09-18T08:09:25.750 INFO:teuthology.orchestra.run.smithi007.stdout:osd.6: {} 2024-09-18T08:09:25.750 INFO:teuthology.orchestra.run.smithi007.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-18T08:09:25.782 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/779349ca-7593-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T08:09:25.789 INFO:teuthology.orchestra.run.smithi007.stdout:osd.7: {} 2024-09-18T08:09:25.790 INFO:teuthology.orchestra.run.smithi007.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-18T08:09:25.818 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0 2024-09-18T08:09:25.876 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:09:25.876 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:09:25.877 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0' 2024-09-18T08:09:25.977 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6 2024-09-18T08:09:26.010 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:09:26.011 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:09:26.011 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6' 2024-09-18T08:09:26.113 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4 2024-09-18T08:09:26.150 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:09:26.151 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:09:26.151 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4' 2024-09-18T08:09:26.189 INFO:tasks.radosbench.radosbench.0.smithi007.stdout: 88 16 99295 99279 70.4832 63.5 0.0163124 0.0141813 2024-09-18T08:09:26.252 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0 2024-09-18T08:09:26.299 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:09:26.300 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:09:26.300 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0' 2024-09-18T08:09:26.401 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7 2024-09-18T08:09:26.436 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:09:26.437 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:09:26.437 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7' 2024-09-18T08:09:26.468 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:09:26 smithi007 ceph-mon[29459]: pgmap v324: 158 pgs: 1 active+remapped+backfilling, 1 active+recovering+undersized+remapped, 156 active+clean; 5.9 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 1.3 KiB/s rd, 76 MiB/s wr, 1.22k op/s; 3593/290007 objects misplaced (1.239%); 18 MiB/s, 293 objects/s recovering 2024-09-18T08:09:26.468 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:09:26 smithi007 ceph-mon[29459]: from='client.? 172.21.15.7:0/897447203' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-18T08:09:26.470 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:09:26 smithi007 ceph-mon[39249]: pgmap v324: 158 pgs: 1 active+remapped+backfilling, 1 active+recovering+undersized+remapped, 156 active+clean; 5.9 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 1.3 KiB/s rd, 76 MiB/s wr, 1.22k op/s; 3593/290007 objects misplaced (1.239%); 18 MiB/s, 293 objects/s recovering 2024-09-18T08:09:26.471 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:09:26 smithi007 ceph-mon[39249]: from='client.? 172.21.15.7:0/897447203' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-18T08:09:26.538 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6 2024-09-18T08:09:26.566 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:09:26.567 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:09:26.567 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6' 2024-09-18T08:09:26.656 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:09:26 smithi050 ceph-mon[32371]: pgmap v324: 158 pgs: 1 active+remapped+backfilling, 1 active+recovering+undersized+remapped, 156 active+clean; 5.9 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 1.3 KiB/s rd, 76 MiB/s wr, 1.22k op/s; 3593/290007 objects misplaced (1.239%); 18 MiB/s, 293 objects/s recovering 2024-09-18T08:09:26.656 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:09:26 smithi050 ceph-mon[32371]: from='client.? 172.21.15.7:0/897447203' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-18T08:09:26.668 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4 2024-09-18T08:09:26.697 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:09:26.698 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:09:26.699 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4' 2024-09-18T08:09:26.730 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Sep 18 08:09:26 smithi007 ceph-779349ca-7593-11ef-bceb-c7b262605968-mgr-y[29688]: ::ffff:172.21.15.50 - - [18/Sep/2024:08:09:26] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-09-18T08:09:26.800 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7 2024-09-18T08:09:26.832 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:09:26.833 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:09:26.833 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7' 2024-09-18T08:09:26.934 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3 2024-09-18T08:09:26.977 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:09:26.977 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:09:26.978 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3' 2024-09-18T08:09:27.078 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6 2024-09-18T08:09:27.112 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:09:27.113 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:09:27.113 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6' 2024-09-18T08:09:27.130 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/779349ca-7593-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T08:09:27.188 INFO:tasks.radosbench.radosbench.0.smithi007.stdout: 89 16 100287 100271 70.3864 62 0.0272049 0.014202 2024-09-18T08:09:27.214 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5 2024-09-18T08:09:27.253 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:09:27.254 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:09:27.254 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5' 2024-09-18T08:09:27.345 INFO:teuthology.orchestra.run.smithi007.stderr:noscrub is unset 2024-09-18T08:09:27.349 INFO:teuthology.orchestra.run.smithi007.stderr:reweighted osd.4 to 1 (10000) 2024-09-18T08:09:27.355 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0 2024-09-18T08:09:27.368 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:09:27 smithi007 ceph-mon[29459]: from='client.? 172.21.15.7:0/3784966783' entity='client.admin' cmd={"prefix": "osd reweight", "id": 4, "weight": 1.0} : dispatch 2024-09-18T08:09:27.369 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:09:27 smithi007 ceph-mon[29459]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 4, "weight": 1.0} : dispatch 2024-09-18T08:09:27.369 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:09:27 smithi007 ceph-mon[29459]: from='client.? 172.21.15.7:0/3963592295' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-18T08:09:27.370 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:09:27 smithi007 ceph-mon[39249]: from='client.? 172.21.15.7:0/3784966783' entity='client.admin' cmd={"prefix": "osd reweight", "id": 4, "weight": 1.0} : dispatch 2024-09-18T08:09:27.370 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:09:27 smithi007 ceph-mon[39249]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 4, "weight": 1.0} : dispatch 2024-09-18T08:09:27.370 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:09:27 smithi007 ceph-mon[39249]: from='client.? 172.21.15.7:0/3963592295' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-18T08:09:27.396 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:09:27.397 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:09:27.398 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0' 2024-09-18T08:09:27.499 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2 2024-09-18T08:09:27.529 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:09:27.530 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:09:27.530 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2' 2024-09-18T08:09:27.631 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7 2024-09-18T08:09:27.655 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:09:27 smithi050 ceph-mon[32371]: from='client.? 172.21.15.7:0/3784966783' entity='client.admin' cmd={"prefix": "osd reweight", "id": 4, "weight": 1.0} : dispatch 2024-09-18T08:09:27.656 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:09:27 smithi050 ceph-mon[32371]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 4, "weight": 1.0} : dispatch 2024-09-18T08:09:27.656 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:09:27 smithi050 ceph-mon[32371]: from='client.? 172.21.15.7:0/3963592295' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-18T08:09:27.658 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:09:27.658 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:09:27.659 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7' 2024-09-18T08:09:27.759 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5 2024-09-18T08:09:27.783 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:09:27.784 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:09:27.784 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5' 2024-09-18T08:09:27.885 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6 2024-09-18T08:09:27.909 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:09:27.910 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:09:27.910 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6' 2024-09-18T08:09:28.011 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5 2024-09-18T08:09:28.035 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:09:28.035 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:09:28.036 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5' 2024-09-18T08:09:28.136 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0 2024-09-18T08:09:28.160 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:09:28.161 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:09:28.161 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0' 2024-09-18T08:09:28.188 INFO:tasks.radosbench.radosbench.0.smithi007.stdout: 90 16 100749 100733 69.9252 28.875 0.00402632 0.0141843 2024-09-18T08:09:28.262 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1 2024-09-18T08:09:28.284 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:09:28.285 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:09:28.285 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1' 2024-09-18T08:09:28.386 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5 2024-09-18T08:09:28.408 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:09:28.409 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:09:28.409 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5' 2024-09-18T08:09:28.510 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3 2024-09-18T08:09:28.533 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:09:28.534 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:09:28.534 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3' 2024-09-18T08:09:28.635 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7 2024-09-18T08:09:28.656 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:09:28 smithi050 ceph-mon[32371]: pgmap v325: 158 pgs: 1 active+remapped+backfilling, 1 active+recovering+undersized+remapped, 156 active+clean; 5.9 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 639 B/s rd, 41 MiB/s wr, 661 op/s; 3593/290007 objects misplaced (1.239%); 6.1 MiB/s, 97 objects/s recovering 2024-09-18T08:09:28.656 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:09:28 smithi050 ceph-mon[32371]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-18T08:09:28.656 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:09:28 smithi050 ceph-mon[32371]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]': finished 2024-09-18T08:09:28.656 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:09:28 smithi050 ceph-mon[32371]: from='client.? 172.21.15.7:0/3963592295' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-18T08:09:28.656 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:09:28 smithi050 ceph-mon[32371]: osdmap e211: 8 total, 8 up, 8 in 2024-09-18T08:09:28.656 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:09:28 smithi050 ceph-mon[32371]: 8.3 scrub starts 2024-09-18T08:09:28.656 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:09:28 smithi050 ceph-mon[32371]: purged_snaps scrub starts 2024-09-18T08:09:28.656 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:09:28 smithi050 ceph-mon[32371]: purged_snaps scrub ok 2024-09-18T08:09:28.657 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:09:28.658 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:09:28.658 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7' 2024-09-18T08:09:28.703 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:09:28 smithi007 ceph-mon[29459]: pgmap v325: 158 pgs: 1 active+remapped+backfilling, 1 active+recovering+undersized+remapped, 156 active+clean; 5.9 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 639 B/s rd, 41 MiB/s wr, 661 op/s; 3593/290007 objects misplaced (1.239%); 6.1 MiB/s, 97 objects/s recovering 2024-09-18T08:09:28.703 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:09:28 smithi007 ceph-mon[29459]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-18T08:09:28.703 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:09:28 smithi007 ceph-mon[29459]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]': finished 2024-09-18T08:09:28.703 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:09:28 smithi007 ceph-mon[29459]: from='client.? 172.21.15.7:0/3963592295' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-18T08:09:28.703 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:09:28 smithi007 ceph-mon[29459]: osdmap e211: 8 total, 8 up, 8 in 2024-09-18T08:09:28.704 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:09:28 smithi007 ceph-mon[29459]: 8.3 scrub starts 2024-09-18T08:09:28.704 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:09:28 smithi007 ceph-mon[29459]: purged_snaps scrub starts 2024-09-18T08:09:28.704 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:09:28 smithi007 ceph-mon[29459]: purged_snaps scrub ok 2024-09-18T08:09:28.704 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:09:28 smithi007 ceph-mon[39249]: pgmap v325: 158 pgs: 1 active+remapped+backfilling, 1 active+recovering+undersized+remapped, 156 active+clean; 5.9 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 639 B/s rd, 41 MiB/s wr, 661 op/s; 3593/290007 objects misplaced (1.239%); 6.1 MiB/s, 97 objects/s recovering 2024-09-18T08:09:28.704 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:09:28 smithi007 ceph-mon[39249]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-18T08:09:28.704 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:09:28 smithi007 ceph-mon[39249]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]': finished 2024-09-18T08:09:28.704 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:09:28 smithi007 ceph-mon[39249]: from='client.? 172.21.15.7:0/3963592295' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-18T08:09:28.704 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:09:28 smithi007 ceph-mon[39249]: osdmap e211: 8 total, 8 up, 8 in 2024-09-18T08:09:28.704 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:09:28 smithi007 ceph-mon[39249]: 8.3 scrub starts 2024-09-18T08:09:28.704 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:09:28 smithi007 ceph-mon[39249]: purged_snaps scrub starts 2024-09-18T08:09:28.705 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:09:28 smithi007 ceph-mon[39249]: purged_snaps scrub ok 2024-09-18T08:09:28.759 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1 2024-09-18T08:09:28.782 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:09:28.783 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:09:28.783 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1' 2024-09-18T08:09:28.884 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4 2024-09-18T08:09:28.895 INFO:teuthology.orchestra.run.smithi007.stdout:{ 2024-09-18T08:09:28.895 INFO:teuthology.orchestra.run.smithi007.stdout: "ops": [], 2024-09-18T08:09:28.895 INFO:teuthology.orchestra.run.smithi007.stdout: "complaint_time": 30, 2024-09-18T08:09:28.895 INFO:teuthology.orchestra.run.smithi007.stdout: "num_blocked_ops": 0 2024-09-18T08:09:28.895 INFO:teuthology.orchestra.run.smithi007.stdout:} 2024-09-18T08:09:28.906 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:09:28.907 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:09:28.907 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4' 2024-09-18T08:09:29.008 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6 2024-09-18T08:09:29.030 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:09:29.031 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:09:29.031 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6' 2024-09-18T08:09:29.132 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4 2024-09-18T08:09:29.155 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:09:29.155 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:09:29.156 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4' 2024-09-18T08:09:29.189 INFO:tasks.radosbench.radosbench.0.smithi007.stdout:Total time run: 90.2472 2024-09-18T08:09:29.189 INFO:tasks.radosbench.radosbench.0.smithi007.stdout:Total writes made: 100749 2024-09-18T08:09:29.189 INFO:tasks.radosbench.radosbench.0.smithi007.stdout:Write size: 65536 2024-09-18T08:09:29.189 INFO:tasks.radosbench.radosbench.0.smithi007.stdout:Object size: 65536 2024-09-18T08:09:29.189 INFO:tasks.radosbench.radosbench.0.smithi007.stdout:Bandwidth (MB/sec): 69.773 2024-09-18T08:09:29.189 INFO:tasks.radosbench.radosbench.0.smithi007.stdout:Stddev Bandwidth: 35.2882 2024-09-18T08:09:29.189 INFO:tasks.radosbench.radosbench.0.smithi007.stdout:Max bandwidth (MB/sec): 188.875 2024-09-18T08:09:29.189 INFO:tasks.radosbench.radosbench.0.smithi007.stdout:Min bandwidth (MB/sec): 0 2024-09-18T08:09:29.189 INFO:tasks.radosbench.radosbench.0.smithi007.stdout:Average IOPS: 1116 2024-09-18T08:09:29.190 INFO:tasks.radosbench.radosbench.0.smithi007.stdout:Stddev IOPS: 564.623 2024-09-18T08:09:29.190 INFO:tasks.radosbench.radosbench.0.smithi007.stdout:Max IOPS: 3022 2024-09-18T08:09:29.190 INFO:tasks.radosbench.radosbench.0.smithi007.stdout:Min IOPS: 0 2024-09-18T08:09:29.190 INFO:tasks.radosbench.radosbench.0.smithi007.stdout:Average Latency(s): 0.0143262 2024-09-18T08:09:29.190 INFO:tasks.radosbench.radosbench.0.smithi007.stdout:Stddev Latency(s): 0.0519018 2024-09-18T08:09:29.190 INFO:tasks.radosbench.radosbench.0.smithi007.stdout:Max latency(s): 2.15499 2024-09-18T08:09:29.190 INFO:tasks.radosbench.radosbench.0.smithi007.stdout:Min latency(s): 0.00120772 2024-09-18T08:09:29.193 INFO:tasks.radosbench.radosbench.0.smithi007.stdout:Cleaning up (deleting benchmark objects) 2024-09-18T08:09:29.257 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4 2024-09-18T08:09:29.286 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:09:29.287 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:09:29.287 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4' 2024-09-18T08:09:29.366 INFO:journalctl@ceph.iscsi.iscsi.a.smithi050.stdout:Sep 18 08:09:29 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-iscsi-iscsi-a[57782]: debug there is no tcmu-runner data available 2024-09-18T08:09:29.388 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3 2024-09-18T08:09:29.428 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:09:29.428 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:09:29.429 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3' 2024-09-18T08:09:29.530 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7 2024-09-18T08:09:29.567 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:09:29.567 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:09:29.568 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7' 2024-09-18T08:09:29.656 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:09:29 smithi050 ceph-mon[32371]: 1.9 scrub starts 2024-09-18T08:09:29.656 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:09:29 smithi050 ceph-mon[32371]: 1.9 scrub ok 2024-09-18T08:09:29.656 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:09:29 smithi050 ceph-mon[32371]: 6.9 scrub starts 2024-09-18T08:09:29.656 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:09:29 smithi050 ceph-mon[32371]: 6.9 scrub ok 2024-09-18T08:09:29.656 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:09:29 smithi050 ceph-mon[32371]: 8.3 scrub ok 2024-09-18T08:09:29.656 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:09:29 smithi050 ceph-mon[32371]: osdmap e212: 8 total, 8 up, 8 in 2024-09-18T08:09:29.669 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3 2024-09-18T08:09:29.707 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:09:29.707 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:09:29.708 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3' 2024-09-18T08:09:29.730 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:09:29 smithi007 ceph-mon[29459]: 1.9 scrub starts 2024-09-18T08:09:29.730 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:09:29 smithi007 ceph-mon[29459]: 1.9 scrub ok 2024-09-18T08:09:29.730 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:09:29 smithi007 ceph-mon[29459]: 6.9 scrub starts 2024-09-18T08:09:29.730 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:09:29 smithi007 ceph-mon[29459]: 6.9 scrub ok 2024-09-18T08:09:29.731 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:09:29 smithi007 ceph-mon[29459]: 8.3 scrub ok 2024-09-18T08:09:29.731 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:09:29 smithi007 ceph-mon[29459]: osdmap e212: 8 total, 8 up, 8 in 2024-09-18T08:09:29.731 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:09:29 smithi007 ceph-mon[39249]: 1.9 scrub starts 2024-09-18T08:09:29.731 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:09:29 smithi007 ceph-mon[39249]: 1.9 scrub ok 2024-09-18T08:09:29.731 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:09:29 smithi007 ceph-mon[39249]: 6.9 scrub starts 2024-09-18T08:09:29.731 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:09:29 smithi007 ceph-mon[39249]: 6.9 scrub ok 2024-09-18T08:09:29.731 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:09:29 smithi007 ceph-mon[39249]: 8.3 scrub ok 2024-09-18T08:09:29.731 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:09:29 smithi007 ceph-mon[39249]: osdmap e212: 8 total, 8 up, 8 in 2024-09-18T08:09:29.809 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6 2024-09-18T08:09:29.844 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:09:29.844 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:09:29.845 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6' 2024-09-18T08:09:29.946 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1 2024-09-18T08:09:29.984 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:09:29.985 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:09:29.985 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1' 2024-09-18T08:09:30.078 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid 779349ca-7593-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-18T08:09:30.086 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6 2024-09-18T08:09:30.120 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:09:30.120 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:09:30.121 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6' 2024-09-18T08:09:30.222 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7 2024-09-18T08:09:30.255 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:09:30.256 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:09:30.256 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7' 2024-09-18T08:09:30.357 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5 2024-09-18T08:09:30.395 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:09:30.396 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:09:30.396 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5' 2024-09-18T08:09:30.480 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:09:30 smithi007 ceph-mon[39249]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-18T08:09:30.481 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:09:30 smithi007 ceph-mon[39249]: pgmap v328: 158 pgs: 1 active+recovering+undersized+remapped, 157 active+clean; 6.1 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 1.5 KiB/s rd, 87 MiB/s wr, 1.39k op/s; 2908/302844 objects misplaced (0.960%); 13 MiB/s, 212 objects/s recovering 2024-09-18T08:09:30.481 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:09:30 smithi007 ceph-mon[39249]: 3.19 scrub starts 2024-09-18T08:09:30.481 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:09:30 smithi007 ceph-mon[39249]: 3.19 scrub ok 2024-09-18T08:09:30.481 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:09:30 smithi007 ceph-mon[29459]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-18T08:09:30.481 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:09:30 smithi007 ceph-mon[29459]: pgmap v328: 158 pgs: 1 active+recovering+undersized+remapped, 157 active+clean; 6.1 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 1.5 KiB/s rd, 87 MiB/s wr, 1.39k op/s; 2908/302844 objects misplaced (0.960%); 13 MiB/s, 212 objects/s recovering 2024-09-18T08:09:30.481 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:09:30 smithi007 ceph-mon[29459]: 3.19 scrub starts 2024-09-18T08:09:30.481 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:09:30 smithi007 ceph-mon[29459]: 3.19 scrub ok 2024-09-18T08:09:30.497 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1 2024-09-18T08:09:30.541 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:09:30.541 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:09:30.542 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1' 2024-09-18T08:09:30.643 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3 2024-09-18T08:09:30.655 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:09:30 smithi050 ceph-mon[32371]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-18T08:09:30.655 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:09:30 smithi050 ceph-mon[32371]: pgmap v328: 158 pgs: 1 active+recovering+undersized+remapped, 157 active+clean; 6.1 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 1.5 KiB/s rd, 87 MiB/s wr, 1.39k op/s; 2908/302844 objects misplaced (0.960%); 13 MiB/s, 212 objects/s recovering 2024-09-18T08:09:30.656 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:09:30 smithi050 ceph-mon[32371]: 3.19 scrub starts 2024-09-18T08:09:30.656 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:09:30 smithi050 ceph-mon[32371]: 3.19 scrub ok 2024-09-18T08:09:30.679 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid 779349ca-7593-11ef-bceb-c7b262605968 -- ceph daemon osd.3 dump_historic_ops 2024-09-18T08:09:30.683 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:09:30.683 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:09:30.684 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3' 2024-09-18T08:09:30.785 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2 2024-09-18T08:09:30.832 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:09:30.833 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:09:30.833 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2' 2024-09-18T08:09:30.934 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6 2024-09-18T08:09:30.968 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:09:30.969 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:09:30.969 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6' 2024-09-18T08:09:31.070 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6 2024-09-18T08:09:31.104 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:09:31.104 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:09:31.105 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6' 2024-09-18T08:09:31.205 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6 2024-09-18T08:09:31.240 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:09:31.241 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:09:31.241 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6' 2024-09-18T08:09:31.342 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2 2024-09-18T08:09:31.384 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:09:31.385 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:09:31.385 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2' 2024-09-18T08:09:31.486 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7 2024-09-18T08:09:31.524 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:09:31.525 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:09:31.525 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7' 2024-09-18T08:09:31.626 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7 2024-09-18T08:09:31.660 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:09:31.660 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:09:31.661 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7' 2024-09-18T08:09:31.762 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2 2024-09-18T08:09:31.803 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:09:31.803 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:09:31.803 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2' 2024-09-18T08:09:31.824 INFO:tasks.cephadm.ceph_manager.ceph:waiting for recovery to complete 2024-09-18T08:09:31.824 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid 779349ca-7593-11ef-bceb-c7b262605968 -- ceph pg dump --format=json 2024-09-18T08:09:31.904 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0 2024-09-18T08:09:31.948 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/779349ca-7593-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T08:09:31.959 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/779349ca-7593-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T08:09:31.970 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:09:31.971 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:09:31.971 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0' 2024-09-18T08:09:32.072 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3 2024-09-18T08:09:32.126 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:09:32.127 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:09:32.127 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3' 2024-09-18T08:09:32.228 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6 2024-09-18T08:09:32.259 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:09:32.260 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:09:32.260 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6' 2024-09-18T08:09:32.361 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1 2024-09-18T08:09:32.409 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:09:32.410 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:09:32.410 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1' 2024-09-18T08:09:32.511 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2 2024-09-18T08:09:32.546 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:09:32.547 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:09:32.547 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2' 2024-09-18T08:09:32.648 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0 2024-09-18T08:09:32.655 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:09:32 smithi050 ceph-mon[32371]: pgmap v329: 158 pgs: 1 active+recovering+undersized+remapped, 157 active+clean; 6.1 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 1.2 KiB/s rd, 69 MiB/s wr, 1.11k op/s; 2908/302844 objects misplaced (0.960%); 11 MiB/s, 183 objects/s recovering 2024-09-18T08:09:32.691 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:09:32.692 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:09:32.693 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0' 2024-09-18T08:09:32.730 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:09:32 smithi007 ceph-mon[29459]: pgmap v329: 158 pgs: 1 active+recovering+undersized+remapped, 157 active+clean; 6.1 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 1.2 KiB/s rd, 69 MiB/s wr, 1.11k op/s; 2908/302844 objects misplaced (0.960%); 11 MiB/s, 183 objects/s recovering 2024-09-18T08:09:32.731 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:09:32 smithi007 ceph-mon[39249]: pgmap v329: 158 pgs: 1 active+recovering+undersized+remapped, 157 active+clean; 6.1 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 1.2 KiB/s rd, 69 MiB/s wr, 1.11k op/s; 2908/302844 objects misplaced (0.960%); 11 MiB/s, 183 objects/s recovering 2024-09-18T08:09:32.794 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5 2024-09-18T08:09:32.827 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:09:32.827 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:09:32.828 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5' 2024-09-18T08:09:32.928 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4 2024-09-18T08:09:32.965 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:09:32.966 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:09:32.966 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4' 2024-09-18T08:09:33.067 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6 2024-09-18T08:09:33.102 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:09:33.103 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:09:33.104 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6' 2024-09-18T08:09:33.156 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid 779349ca-7593-11ef-bceb-c7b262605968 -- ceph osd unset nodeep-scrub 2024-09-18T08:09:33.204 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5 2024-09-18T08:09:33.233 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:09:33.233 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:09:33.234 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5' 2024-09-18T08:09:33.335 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4 2024-09-18T08:09:33.362 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:09:33.363 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:09:33.363 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4' 2024-09-18T08:09:33.464 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3 2024-09-18T08:09:33.480 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:09:33 smithi007 ceph-mon[29459]: 3.13 scrub starts 2024-09-18T08:09:33.480 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:09:33 smithi007 ceph-mon[29459]: 3.13 scrub ok 2024-09-18T08:09:33.480 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:09:33 smithi007 ceph-mon[29459]: 4.19 scrub starts 2024-09-18T08:09:33.480 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:09:33 smithi007 ceph-mon[29459]: 4.19 scrub ok 2024-09-18T08:09:33.480 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:09:33 smithi007 ceph-mon[29459]: from='mgr.24470 172.21.15.7:0/2594390261' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-18T08:09:33.481 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:09:33 smithi007 ceph-mon[39249]: 3.13 scrub starts 2024-09-18T08:09:33.481 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:09:33 smithi007 ceph-mon[39249]: 3.13 scrub ok 2024-09-18T08:09:33.481 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:09:33 smithi007 ceph-mon[39249]: 4.19 scrub starts 2024-09-18T08:09:33.481 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:09:33 smithi007 ceph-mon[39249]: 4.19 scrub ok 2024-09-18T08:09:33.481 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:09:33 smithi007 ceph-mon[39249]: from='mgr.24470 172.21.15.7:0/2594390261' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-18T08:09:33.520 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:09:33.520 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:09:33.521 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3' 2024-09-18T08:09:33.622 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0 2024-09-18T08:09:33.650 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:09:33.651 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:09:33.651 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0' 2024-09-18T08:09:33.655 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:09:33 smithi050 ceph-mon[32371]: 3.13 scrub starts 2024-09-18T08:09:33.656 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:09:33 smithi050 ceph-mon[32371]: 3.13 scrub ok 2024-09-18T08:09:33.656 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:09:33 smithi050 ceph-mon[32371]: 4.19 scrub starts 2024-09-18T08:09:33.656 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:09:33 smithi050 ceph-mon[32371]: 4.19 scrub ok 2024-09-18T08:09:33.656 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:09:33 smithi050 ceph-mon[32371]: from='mgr.24470 172.21.15.7:0/2594390261' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-18T08:09:33.752 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4 2024-09-18T08:09:33.783 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:09:33.784 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:09:33.784 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4' 2024-09-18T08:09:33.885 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0 2024-09-18T08:09:33.925 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:09:33.925 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:09:33.925 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0' 2024-09-18T08:09:34.026 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2 2024-09-18T08:09:34.049 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:09:34.050 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:09:34.050 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2' 2024-09-18T08:09:34.151 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5 2024-09-18T08:09:34.185 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:09:34.186 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:09:34.186 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5' 2024-09-18T08:09:34.287 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2 2024-09-18T08:09:34.322 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:09:34.323 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:09:34.323 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2' 2024-09-18T08:09:34.424 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2 2024-09-18T08:09:34.464 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:09:34.465 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:09:34.465 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2' 2024-09-18T08:09:34.566 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3 2024-09-18T08:09:34.606 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:09:34.606 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:09:34.607 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3' 2024-09-18T08:09:34.656 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:09:34 smithi050 ceph-mon[32371]: pgmap v330: 158 pgs: 1 active+recovering+undersized+remapped, 157 active+clean; 6.1 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 639 B/s rd, 33 MiB/s wr, 616 op/s; 2908/300906 objects misplaced (0.966%); 5.4 MiB/s, 86 objects/s recovering 2024-09-18T08:09:34.656 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:09:34 smithi050 ceph-mon[32371]: purged_snaps scrub starts 2024-09-18T08:09:34.656 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:09:34 smithi050 ceph-mon[32371]: purged_snaps scrub ok 2024-09-18T08:09:34.656 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:09:34 smithi050 ceph-mon[32371]: 8.7 scrub starts 2024-09-18T08:09:34.707 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6 2024-09-18T08:09:34.730 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:09:34 smithi007 ceph-mon[29459]: pgmap v330: 158 pgs: 1 active+recovering+undersized+remapped, 157 active+clean; 6.1 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 639 B/s rd, 33 MiB/s wr, 616 op/s; 2908/300906 objects misplaced (0.966%); 5.4 MiB/s, 86 objects/s recovering 2024-09-18T08:09:34.730 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:09:34 smithi007 ceph-mon[29459]: purged_snaps scrub starts 2024-09-18T08:09:34.730 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:09:34 smithi007 ceph-mon[29459]: purged_snaps scrub ok 2024-09-18T08:09:34.731 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:09:34 smithi007 ceph-mon[29459]: 8.7 scrub starts 2024-09-18T08:09:34.731 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:09:34 smithi007 ceph-mon[39249]: pgmap v330: 158 pgs: 1 active+recovering+undersized+remapped, 157 active+clean; 6.1 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 639 B/s rd, 33 MiB/s wr, 616 op/s; 2908/300906 objects misplaced (0.966%); 5.4 MiB/s, 86 objects/s recovering 2024-09-18T08:09:34.731 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:09:34 smithi007 ceph-mon[39249]: purged_snaps scrub starts 2024-09-18T08:09:34.731 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:09:34 smithi007 ceph-mon[39249]: purged_snaps scrub ok 2024-09-18T08:09:34.731 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:09:34 smithi007 ceph-mon[39249]: 8.7 scrub starts 2024-09-18T08:09:34.735 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:09:34.737 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:09:34.737 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6' 2024-09-18T08:09:34.838 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1 2024-09-18T08:09:34.880 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:09:34.881 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:09:34.881 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1' 2024-09-18T08:09:34.982 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0 2024-09-18T08:09:35.028 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:09:35.028 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:09:35.029 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0' 2024-09-18T08:09:35.130 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6 2024-09-18T08:09:35.152 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:09:35.152 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:09:35.153 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6' 2024-09-18T08:09:35.254 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2 2024-09-18T08:09:35.291 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:09:35.292 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:09:35.292 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2' 2024-09-18T08:09:35.393 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6 2024-09-18T08:09:35.420 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:09:35.421 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:09:35.421 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6' 2024-09-18T08:09:35.522 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4 2024-09-18T08:09:35.556 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:09:35.557 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:09:35.557 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4' 2024-09-18T08:09:35.655 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:09:35 smithi050 ceph-mon[32371]: 3.1 scrub starts 2024-09-18T08:09:35.656 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:09:35 smithi050 ceph-mon[32371]: 3.1 scrub ok 2024-09-18T08:09:35.656 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:09:35 smithi050 ceph-mon[32371]: osdmap e213: 8 total, 8 up, 8 in 2024-09-18T08:09:35.658 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5 2024-09-18T08:09:35.690 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:09:35.691 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:09:35.691 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5' 2024-09-18T08:09:35.730 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:09:35 smithi007 ceph-mon[29459]: 3.1 scrub starts 2024-09-18T08:09:35.731 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:09:35 smithi007 ceph-mon[29459]: 3.1 scrub ok 2024-09-18T08:09:35.731 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:09:35 smithi007 ceph-mon[29459]: osdmap e213: 8 total, 8 up, 8 in 2024-09-18T08:09:35.731 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:09:35 smithi007 ceph-mon[39249]: 3.1 scrub starts 2024-09-18T08:09:35.731 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:09:35 smithi007 ceph-mon[39249]: 3.1 scrub ok 2024-09-18T08:09:35.731 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:09:35 smithi007 ceph-mon[39249]: osdmap e213: 8 total, 8 up, 8 in 2024-09-18T08:09:35.792 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5 2024-09-18T08:09:35.816 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:09:35.817 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:09:35.817 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5' 2024-09-18T08:09:35.918 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2 2024-09-18T08:09:35.957 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:09:35.957 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:09:35.958 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2' 2024-09-18T08:09:36.052 INFO:teuthology.orchestra.run.smithi007.stdout:{ 2024-09-18T08:09:36.052 INFO:teuthology.orchestra.run.smithi007.stdout: "size": 20, 2024-09-18T08:09:36.052 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 600, 2024-09-18T08:09:36.052 INFO:teuthology.orchestra.run.smithi007.stdout: "ops": [ 2024-09-18T08:09:36.052 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:09:36.052 INFO:teuthology.orchestra.run.smithi007.stdout: "description": "osd_op(client.15216.0:22606 8.f 8:f10856c4:::benchmark_data_smithi007_101583_object22605:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e174)", 2024-09-18T08:09:36.052 INFO:teuthology.orchestra.run.smithi007.stdout: "initiated_at": "2024-09-18T08:08:08.436554+0000", 2024-09-18T08:09:36.052 INFO:teuthology.orchestra.run.smithi007.stdout: "age": 86.515259649000001, 2024-09-18T08:09:36.052 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.768179902, 2024-09-18T08:09:36.052 INFO:teuthology.orchestra.run.smithi007.stdout: "continuous": false, 2024-09-18T08:09:36.053 INFO:teuthology.orchestra.run.smithi007.stdout: "type_data": { 2024-09-18T08:09:36.053 INFO:teuthology.orchestra.run.smithi007.stdout: "flag_point": "reached pg", 2024-09-18T08:09:36.053 INFO:teuthology.orchestra.run.smithi007.stdout: "client_info": { 2024-09-18T08:09:36.053 INFO:teuthology.orchestra.run.smithi007.stdout: "client": "client.15216", 2024-09-18T08:09:36.053 INFO:teuthology.orchestra.run.smithi007.stdout: "client_addr": "172.21.15.7:0/2596041177", 2024-09-18T08:09:36.053 INFO:teuthology.orchestra.run.smithi007.stdout: "tid": 22606 2024-09-18T08:09:36.053 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:09:36.053 INFO:teuthology.orchestra.run.smithi007.stdout: "events": [ 2024-09-18T08:09:36.053 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:09:36.054 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "initiated", 2024-09-18T08:09:36.054 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:08.436554+0000", 2024-09-18T08:09:36.054 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-09-18T08:09:36.054 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:09:36.054 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:09:36.054 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "header_read", 2024-09-18T08:09:36.054 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:08.436554+0000", 2024-09-18T08:09:36.054 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-09-18T08:09:36.054 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:09:36.055 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:09:36.055 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "throttled", 2024-09-18T08:09:36.055 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:08.436555+0000", 2024-09-18T08:09:36.055 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 7.8299999999999996e-07 2024-09-18T08:09:36.055 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:09:36.055 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:09:36.055 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "all_read", 2024-09-18T08:09:36.055 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:08.436576+0000", 2024-09-18T08:09:36.055 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 2.1214999999999999e-05 2024-09-18T08:09:36.055 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:09:36.055 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:09:36.055 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "dispatched", 2024-09-18T08:09:36.055 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:08.436586+0000", 2024-09-18T08:09:36.056 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.0088999999999999e-05 2024-09-18T08:09:36.056 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:09:36.056 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:09:36.056 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "queued_for_pg", 2024-09-18T08:09:36.056 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:08.436593+0000", 2024-09-18T08:09:36.056 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 7.3379999999999999e-06 2024-09-18T08:09:36.056 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:09:36.056 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:09:36.056 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-09-18T08:09:36.056 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:08.436642+0000", 2024-09-18T08:09:36.056 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 4.8928000000000003e-05 2024-09-18T08:09:36.056 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:09:36.057 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:09:36.057 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "started", 2024-09-18T08:09:36.057 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:08.436744+0000", 2024-09-18T08:09:36.057 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.000101778 2024-09-18T08:09:36.057 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:09:36.057 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:09:36.057 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "waiting for subops from 1,2", 2024-09-18T08:09:36.057 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:08.436890+0000", 2024-09-18T08:09:36.057 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.00014579099999999999 2024-09-18T08:09:36.057 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:09:36.058 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:09:36.058 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "op_commit", 2024-09-18T08:09:36.058 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:08.449655+0000", 2024-09-18T08:09:36.058 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.012765334 2024-09-18T08:09:36.058 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:09:36.058 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:09:36.058 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-09-18T08:09:36.058 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:08.460251+0000", 2024-09-18T08:09:36.058 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.010595635000000001 2024-09-18T08:09:36.058 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:09:36.058 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:09:36.059 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "waiting for peered", 2024-09-18T08:09:36.059 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:08.460264+0000", 2024-09-18T08:09:36.059 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.2921e-05 2024-09-18T08:09:36.059 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:09:36.059 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:09:36.059 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-09-18T08:09:36.059 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:09.152996+0000", 2024-09-18T08:09:36.059 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.69273260999999997 2024-09-18T08:09:36.059 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:09:36.059 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:09:36.060 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "waiting for peered", 2024-09-18T08:09:36.060 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:09.153008+0000", 2024-09-18T08:09:36.060 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.1459e-05 2024-09-18T08:09:36.060 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:09:36.060 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:09:36.060 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-09-18T08:09:36.060 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:10.204625+0000", 2024-09-18T08:09:36.060 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.0516172269999999 2024-09-18T08:09:36.061 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:09:36.061 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:09:36.061 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "done", 2024-09-18T08:09:36.061 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:10.204734+0000", 2024-09-18T08:09:36.061 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.000108794 2024-09-18T08:09:36.061 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-09-18T08:09:36.061 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-09-18T08:09:36.061 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-09-18T08:09:36.061 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:09:36.061 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:09:36.061 INFO:teuthology.orchestra.run.smithi007.stdout: "description": "osd_op(client.15216.0:22608 8.b 8:d92bd5a2:::benchmark_data_smithi007_101583_object22607:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e174)", 2024-09-18T08:09:36.061 INFO:teuthology.orchestra.run.smithi007.stdout: "initiated_at": "2024-09-18T08:08:08.436747+0000", 2024-09-18T08:09:36.061 INFO:teuthology.orchestra.run.smithi007.stdout: "age": 86.515066017999999, 2024-09-18T08:09:36.062 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.8187816670000001, 2024-09-18T08:09:36.062 INFO:teuthology.orchestra.run.smithi007.stdout: "continuous": false, 2024-09-18T08:09:36.062 INFO:teuthology.orchestra.run.smithi007.stdout: "type_data": { 2024-09-18T08:09:36.062 INFO:teuthology.orchestra.run.smithi007.stdout: "flag_point": "reached pg", 2024-09-18T08:09:36.062 INFO:teuthology.orchestra.run.smithi007.stdout: "client_info": { 2024-09-18T08:09:36.062 INFO:teuthology.orchestra.run.smithi007.stdout: "client": "client.15216", 2024-09-18T08:09:36.062 INFO:teuthology.orchestra.run.smithi007.stdout: "client_addr": "172.21.15.7:0/2596041177", 2024-09-18T08:09:36.062 INFO:teuthology.orchestra.run.smithi007.stdout: "tid": 22608 2024-09-18T08:09:36.062 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:09:36.063 INFO:teuthology.orchestra.run.smithi007.stdout: "events": [ 2024-09-18T08:09:36.063 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:09:36.063 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "initiated", 2024-09-18T08:09:36.063 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:08.436747+0000", 2024-09-18T08:09:36.063 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-09-18T08:09:36.063 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:09:36.063 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:09:36.063 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "header_read", 2024-09-18T08:09:36.063 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:08.436747+0000", 2024-09-18T08:09:36.063 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-09-18T08:09:36.063 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:09:36.063 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:09:36.064 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "throttled", 2024-09-18T08:09:36.064 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:08.436748+0000", 2024-09-18T08:09:36.064 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 9.2299999999999999e-07 2024-09-18T08:09:36.064 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:09:36.064 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:09:36.064 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "all_read", 2024-09-18T08:09:36.064 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:08.436769+0000", 2024-09-18T08:09:36.064 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 2.0392999999999999e-05 2024-09-18T08:09:36.064 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:09:36.064 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:09:36.064 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "dispatched", 2024-09-18T08:09:36.065 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:08.436778+0000", 2024-09-18T08:09:36.065 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 8.8689999999999993e-06 2024-09-18T08:09:36.065 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:09:36.065 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:09:36.065 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "queued_for_pg", 2024-09-18T08:09:36.065 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:08.436785+0000", 2024-09-18T08:09:36.065 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 7.199e-06 2024-09-18T08:09:36.065 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:09:36.065 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:09:36.065 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-09-18T08:09:36.065 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:08.437535+0000", 2024-09-18T08:09:36.065 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.00075027000000000004 2024-09-18T08:09:36.065 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:09:36.066 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:09:36.066 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "started", 2024-09-18T08:09:36.066 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:08.438489+0000", 2024-09-18T08:09:36.066 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.00095404400000000003 2024-09-18T08:09:36.066 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:09:36.066 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:09:36.066 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "waiting for subops from 1,2", 2024-09-18T08:09:36.066 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:08.438742+0000", 2024-09-18T08:09:36.066 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.00025305599999999998 2024-09-18T08:09:36.066 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:09:36.066 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:09:36.066 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "op_commit", 2024-09-18T08:09:36.066 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:08.440394+0000", 2024-09-18T08:09:36.066 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.001652096 2024-09-18T08:09:36.067 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:09:36.067 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:09:36.067 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:09:36.067 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:08.449571+0000", 2024-09-18T08:09:36.067 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.0091767580000000001 2024-09-18T08:09:36.067 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:09:36.067 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:09:36.067 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-09-18T08:09:36.067 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:08.455097+0000", 2024-09-18T08:09:36.067 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.0055257730000000003 2024-09-18T08:09:36.067 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:09:36.068 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:09:36.068 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "waiting for peered", 2024-09-18T08:09:36.068 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:08.455115+0000", 2024-09-18T08:09:36.068 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.8501000000000001e-05 2024-09-18T08:09:36.068 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:09:36.068 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:09:36.068 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-09-18T08:09:36.068 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:09.152107+0000", 2024-09-18T08:09:36.068 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.69699121100000005 2024-09-18T08:09:36.068 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:09:36.068 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:09:36.068 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "waiting for peered", 2024-09-18T08:09:36.069 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:09.152128+0000", 2024-09-18T08:09:36.069 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 2.1557e-05 2024-09-18T08:09:36.069 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:09:36.069 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:09:36.069 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-09-18T08:09:36.069 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:10.255451+0000", 2024-09-18T08:09:36.069 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.1033232070000001 2024-09-18T08:09:36.069 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:09:36.069 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:09:36.069 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "done", 2024-09-18T08:09:36.069 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:10.255529+0000", 2024-09-18T08:09:36.069 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 7.7810000000000002e-05 2024-09-18T08:09:36.069 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-09-18T08:09:36.070 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-09-18T08:09:36.070 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-09-18T08:09:36.070 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:09:36.070 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:09:36.070 INFO:teuthology.orchestra.run.smithi007.stdout: "description": "osd_op(client.15216.0:22614 8.f 8:f07b5883:::benchmark_data_smithi007_101583_object22613:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e174)", 2024-09-18T08:09:36.070 INFO:teuthology.orchestra.run.smithi007.stdout: "initiated_at": "2024-09-18T08:08:08.439788+0000", 2024-09-18T08:09:36.070 INFO:teuthology.orchestra.run.smithi007.stdout: "age": 86.512025954999999, 2024-09-18T08:09:36.070 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.7882368309999999, 2024-09-18T08:09:36.070 INFO:teuthology.orchestra.run.smithi007.stdout: "continuous": false, 2024-09-18T08:09:36.070 INFO:teuthology.orchestra.run.smithi007.stdout: "type_data": { 2024-09-18T08:09:36.071 INFO:teuthology.orchestra.run.smithi007.stdout: "flag_point": "reached pg", 2024-09-18T08:09:36.071 INFO:teuthology.orchestra.run.smithi007.stdout: "client_info": { 2024-09-18T08:09:36.071 INFO:teuthology.orchestra.run.smithi007.stdout: "client": "client.15216", 2024-09-18T08:09:36.071 INFO:teuthology.orchestra.run.smithi007.stdout: "client_addr": "172.21.15.7:0/2596041177", 2024-09-18T08:09:36.071 INFO:teuthology.orchestra.run.smithi007.stdout: "tid": 22614 2024-09-18T08:09:36.071 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:09:36.071 INFO:teuthology.orchestra.run.smithi007.stdout: "events": [ 2024-09-18T08:09:36.071 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:09:36.071 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "initiated", 2024-09-18T08:09:36.071 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:08.439788+0000", 2024-09-18T08:09:36.071 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-09-18T08:09:36.071 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:09:36.071 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:09:36.072 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "header_read", 2024-09-18T08:09:36.072 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:08.439788+0000", 2024-09-18T08:09:36.072 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-09-18T08:09:36.072 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:09:36.072 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:09:36.072 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "throttled", 2024-09-18T08:09:36.072 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:08.439789+0000", 2024-09-18T08:09:36.072 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.5009999999999999e-06 2024-09-18T08:09:36.072 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:09:36.072 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:09:36.072 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "all_read", 2024-09-18T08:09:36.072 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:08.439813+0000", 2024-09-18T08:09:36.072 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 2.442e-05 2024-09-18T08:09:36.073 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:09:36.073 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:09:36.073 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "dispatched", 2024-09-18T08:09:36.073 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:08.439827+0000", 2024-09-18T08:09:36.073 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.3540000000000001e-05 2024-09-18T08:09:36.073 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:09:36.073 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:09:36.073 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "queued_for_pg", 2024-09-18T08:09:36.073 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:08.439836+0000", 2024-09-18T08:09:36.073 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 8.7630000000000006e-06 2024-09-18T08:09:36.073 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:09:36.073 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:09:36.073 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-09-18T08:09:36.074 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:08.445915+0000", 2024-09-18T08:09:36.074 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.0060795210000000001 2024-09-18T08:09:36.074 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:09:36.074 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:09:36.074 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "started", 2024-09-18T08:09:36.074 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:08.446054+0000", 2024-09-18T08:09:36.074 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.00013859699999999999 2024-09-18T08:09:36.074 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:09:36.074 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:09:36.074 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "waiting for subops from 1,2", 2024-09-18T08:09:36.074 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:08.448436+0000", 2024-09-18T08:09:36.074 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.0023821939999999998 2024-09-18T08:09:36.074 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:09:36.074 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:09:36.075 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "op_commit", 2024-09-18T08:09:36.075 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:08.449664+0000", 2024-09-18T08:09:36.075 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.0012275000000000001 2024-09-18T08:09:36.075 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:09:36.075 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:09:36.075 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-09-18T08:09:36.075 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:08.460300+0000", 2024-09-18T08:09:36.075 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.010636487999999999 2024-09-18T08:09:36.075 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:09:36.075 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:09:36.075 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "waiting for peered", 2024-09-18T08:09:36.075 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:08.460310+0000", 2024-09-18T08:09:36.075 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 9.5240000000000006e-06 2024-09-18T08:09:36.076 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:09:36.076 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:09:36.076 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-09-18T08:09:36.076 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:09.153052+0000", 2024-09-18T08:09:36.076 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.69274252300000005 2024-09-18T08:09:36.076 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:09:36.076 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:09:36.076 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "waiting for peered", 2024-09-18T08:09:36.076 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:09.153062+0000", 2024-09-18T08:09:36.076 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 9.7569999999999999e-06 2024-09-18T08:09:36.076 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:09:36.077 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:09:36.077 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-09-18T08:09:36.077 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:10.220005+0000", 2024-09-18T08:09:36.077 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.0669431330000001 2024-09-18T08:09:36.077 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:09:36.077 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:09:36.077 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "waiting for degraded object", 2024-09-18T08:09:36.077 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:10.220517+0000", 2024-09-18T08:09:36.077 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.00051243100000000004 2024-09-18T08:09:36.077 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:09:36.077 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:09:36.077 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-09-18T08:09:36.078 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:10.227921+0000", 2024-09-18T08:09:36.078 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.0074033850000000002 2024-09-18T08:09:36.078 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:09:36.078 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:09:36.078 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "done", 2024-09-18T08:09:36.078 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:10.228024+0000", 2024-09-18T08:09:36.078 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.000103554 2024-09-18T08:09:36.078 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-09-18T08:09:36.078 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-09-18T08:09:36.078 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-09-18T08:09:36.078 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:09:36.078 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:09:36.078 INFO:teuthology.orchestra.run.smithi007.stdout: "description": "osd_op(client.15216.0:22617 8.b 8:dab1c1d5:::benchmark_data_smithi007_101583_object22616:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e174)", 2024-09-18T08:09:36.079 INFO:teuthology.orchestra.run.smithi007.stdout: "initiated_at": "2024-09-18T08:08:08.447023+0000", 2024-09-18T08:09:36.079 INFO:teuthology.orchestra.run.smithi007.stdout: "age": 86.504790142999994, 2024-09-18T08:09:36.079 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.8206128700000002, 2024-09-18T08:09:36.079 INFO:teuthology.orchestra.run.smithi007.stdout: "continuous": false, 2024-09-18T08:09:36.079 INFO:teuthology.orchestra.run.smithi007.stdout: "type_data": { 2024-09-18T08:09:36.079 INFO:teuthology.orchestra.run.smithi007.stdout: "flag_point": "reached pg", 2024-09-18T08:09:36.079 INFO:teuthology.orchestra.run.smithi007.stdout: "client_info": { 2024-09-18T08:09:36.079 INFO:teuthology.orchestra.run.smithi007.stdout: "client": "client.15216", 2024-09-18T08:09:36.079 INFO:teuthology.orchestra.run.smithi007.stdout: "client_addr": "172.21.15.7:0/2596041177", 2024-09-18T08:09:36.079 INFO:teuthology.orchestra.run.smithi007.stdout: "tid": 22617 2024-09-18T08:09:36.079 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:09:36.080 INFO:teuthology.orchestra.run.smithi007.stdout: "events": [ 2024-09-18T08:09:36.080 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:09:36.080 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "initiated", 2024-09-18T08:09:36.080 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:08.447023+0000", 2024-09-18T08:09:36.080 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-09-18T08:09:36.080 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:09:36.080 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:09:36.080 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "header_read", 2024-09-18T08:09:36.080 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:08.447023+0000", 2024-09-18T08:09:36.080 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-09-18T08:09:36.080 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:09:36.081 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:09:36.081 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "throttled", 2024-09-18T08:09:36.081 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:08.447025+0000", 2024-09-18T08:09:36.081 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.685e-06 2024-09-18T08:09:36.081 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:09:36.081 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:09:36.081 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "all_read", 2024-09-18T08:09:36.081 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:08.447053+0000", 2024-09-18T08:09:36.081 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 2.7953000000000002e-05 2024-09-18T08:09:36.081 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:09:36.081 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:09:36.081 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "dispatched", 2024-09-18T08:09:36.081 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:08.447070+0000", 2024-09-18T08:09:36.081 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.6650000000000002e-05 2024-09-18T08:09:36.082 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:09:36.082 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:09:36.082 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "queued_for_pg", 2024-09-18T08:09:36.082 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:08.447080+0000", 2024-09-18T08:09:36.082 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.0148e-05 2024-09-18T08:09:36.082 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:09:36.082 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:09:36.082 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-09-18T08:09:36.082 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:08.449604+0000", 2024-09-18T08:09:36.082 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.0025244640000000001 2024-09-18T08:09:36.082 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:09:36.082 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:09:36.082 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "started", 2024-09-18T08:09:36.082 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:08.449753+0000", 2024-09-18T08:09:36.083 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.00014878600000000001 2024-09-18T08:09:36.083 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:09:36.083 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:09:36.083 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "waiting for subops from 1,2", 2024-09-18T08:09:36.083 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:08.449904+0000", 2024-09-18T08:09:36.083 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.00015076299999999999 2024-09-18T08:09:36.083 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:09:36.083 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:09:36.083 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-09-18T08:09:36.083 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:08.455203+0000", 2024-09-18T08:09:36.083 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.0052987950000000002 2024-09-18T08:09:36.083 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:09:36.083 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:09:36.083 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "waiting for peered", 2024-09-18T08:09:36.083 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:08.455211+0000", 2024-09-18T08:09:36.084 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 8.8179999999999993e-06 2024-09-18T08:09:36.084 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:09:36.084 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:09:36.084 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-09-18T08:09:36.084 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:09.152202+0000", 2024-09-18T08:09:36.084 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.69699049000000002 2024-09-18T08:09:36.084 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:09:36.084 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:09:36.084 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "waiting for peered", 2024-09-18T08:09:36.084 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:09.152212+0000", 2024-09-18T08:09:36.084 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.0251000000000001e-05 2024-09-18T08:09:36.084 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:09:36.084 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:09:36.085 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-09-18T08:09:36.085 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:10.256406+0000", 2024-09-18T08:09:36.085 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.104194101 2024-09-18T08:09:36.085 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:09:36.085 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:09:36.085 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "waiting for degraded object", 2024-09-18T08:09:36.085 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:10.256961+0000", 2024-09-18T08:09:36.085 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.00055520899999999998 2024-09-18T08:09:36.085 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:09:36.085 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:09:36.086 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-09-18T08:09:36.086 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:10.267550+0000", 2024-09-18T08:09:36.086 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.010588640999999999 2024-09-18T08:09:36.086 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:09:36.086 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:09:36.086 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "done", 2024-09-18T08:09:36.086 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:10.267636+0000", 2024-09-18T08:09:36.086 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 8.6116e-05 2024-09-18T08:09:36.086 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-09-18T08:09:36.086 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-09-18T08:09:36.086 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-09-18T08:09:36.086 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:09:36.086 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:09:36.087 INFO:teuthology.orchestra.run.smithi007.stdout: "description": "osd_op(client.15216.0:22630 8.b 8:da804186:::benchmark_data_smithi007_101583_object22629:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e175)", 2024-09-18T08:09:36.087 INFO:teuthology.orchestra.run.smithi007.stdout: "initiated_at": "2024-09-18T08:08:08.467388+0000", 2024-09-18T08:09:36.087 INFO:teuthology.orchestra.run.smithi007.stdout: "age": 86.484425396000006, 2024-09-18T08:09:36.087 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.8127456770000001, 2024-09-18T08:09:36.087 INFO:teuthology.orchestra.run.smithi007.stdout: "continuous": false, 2024-09-18T08:09:36.087 INFO:teuthology.orchestra.run.smithi007.stdout: "type_data": { 2024-09-18T08:09:36.087 INFO:teuthology.orchestra.run.smithi007.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T08:09:36.087 INFO:teuthology.orchestra.run.smithi007.stdout: "client_info": { 2024-09-18T08:09:36.087 INFO:teuthology.orchestra.run.smithi007.stdout: "client": "client.15216", 2024-09-18T08:09:36.087 INFO:teuthology.orchestra.run.smithi007.stdout: "client_addr": "172.21.15.7:0/2596041177", 2024-09-18T08:09:36.087 INFO:teuthology.orchestra.run.smithi007.stdout: "tid": 22630 2024-09-18T08:09:36.087 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:09:36.087 INFO:teuthology.orchestra.run.smithi007.stdout: "events": [ 2024-09-18T08:09:36.088 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:09:36.088 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "initiated", 2024-09-18T08:09:36.088 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:08.467388+0000", 2024-09-18T08:09:36.088 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-09-18T08:09:36.088 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:09:36.088 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:09:36.088 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "header_read", 2024-09-18T08:09:36.088 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:08.467388+0000", 2024-09-18T08:09:36.088 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-09-18T08:09:36.088 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:09:36.089 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:09:36.089 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "throttled", 2024-09-18T08:09:36.089 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:08.467389+0000", 2024-09-18T08:09:36.089 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.373e-06 2024-09-18T08:09:36.089 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:09:36.089 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:09:36.089 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "all_read", 2024-09-18T08:09:36.089 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:08.467447+0000", 2024-09-18T08:09:36.089 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 5.7238999999999999e-05 2024-09-18T08:09:36.089 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:09:36.089 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:09:36.089 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "dispatched", 2024-09-18T08:09:36.090 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:08.467493+0000", 2024-09-18T08:09:36.090 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 4.6755000000000002e-05 2024-09-18T08:09:36.090 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:09:36.090 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:09:36.090 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "queued_for_pg", 2024-09-18T08:09:36.090 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:08.467513+0000", 2024-09-18T08:09:36.090 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.9873000000000001e-05 2024-09-18T08:09:36.090 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:09:36.090 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:09:36.090 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-09-18T08:09:36.090 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:08.467970+0000", 2024-09-18T08:09:36.090 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.00045696899999999999 2024-09-18T08:09:36.090 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:09:36.091 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:09:36.091 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "waiting for peered", 2024-09-18T08:09:36.091 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:08.467979+0000", 2024-09-18T08:09:36.091 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 8.6440000000000006e-06 2024-09-18T08:09:36.091 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:09:36.091 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:09:36.091 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-09-18T08:09:36.091 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:09.152276+0000", 2024-09-18T08:09:36.091 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.68429694500000005 2024-09-18T08:09:36.092 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:09:36.092 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:09:36.092 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "waiting for peered", 2024-09-18T08:09:36.092 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:09.152284+0000", 2024-09-18T08:09:36.092 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 7.7049999999999999e-06 2024-09-18T08:09:36.092 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:09:36.092 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:09:36.092 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-09-18T08:09:36.092 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:10.257004+0000", 2024-09-18T08:09:36.092 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.104719961 2024-09-18T08:09:36.092 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:09:36.092 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:09:36.093 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "started", 2024-09-18T08:09:36.093 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:10.258354+0000", 2024-09-18T08:09:36.093 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.001350097 2024-09-18T08:09:36.093 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:09:36.093 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:09:36.093 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "waiting for subops from 1,7", 2024-09-18T08:09:36.093 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:10.258951+0000", 2024-09-18T08:09:36.093 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.00059747000000000001 2024-09-18T08:09:36.093 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:09:36.093 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:09:36.093 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:09:36.093 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:10.267720+0000", 2024-09-18T08:09:36.094 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.0087689169999999993 2024-09-18T08:09:36.094 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:09:36.094 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:09:36.094 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:09:36.094 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:10.267767+0000", 2024-09-18T08:09:36.094 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 4.6616000000000002e-05 2024-09-18T08:09:36.094 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:09:36.094 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:09:36.094 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "op_commit", 2024-09-18T08:09:36.094 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:10.278730+0000", 2024-09-18T08:09:36.094 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.010963818 2024-09-18T08:09:36.094 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:09:36.094 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:09:36.094 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "commit_sent", 2024-09-18T08:09:36.095 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:10.278929+0000", 2024-09-18T08:09:36.095 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.000198638 2024-09-18T08:09:36.095 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:09:36.095 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:09:36.095 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "done", 2024-09-18T08:09:36.095 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:10.280134+0000", 2024-09-18T08:09:36.095 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.001204657 2024-09-18T08:09:36.095 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-09-18T08:09:36.095 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-09-18T08:09:36.095 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-09-18T08:09:36.095 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:09:36.095 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:09:36.095 INFO:teuthology.orchestra.run.smithi007.stdout: "description": "osd_op(client.15216.0:22665 8.f 8:f8e596fa:::benchmark_data_smithi007_101583_object22664:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e175)", 2024-09-18T08:09:36.095 INFO:teuthology.orchestra.run.smithi007.stdout: "initiated_at": "2024-09-18T08:08:08.484581+0000", 2024-09-18T08:09:36.096 INFO:teuthology.orchestra.run.smithi007.stdout: "age": 86.467232476000007, 2024-09-18T08:09:36.096 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.7443272359999999, 2024-09-18T08:09:36.096 INFO:teuthology.orchestra.run.smithi007.stdout: "continuous": false, 2024-09-18T08:09:36.096 INFO:teuthology.orchestra.run.smithi007.stdout: "type_data": { 2024-09-18T08:09:36.096 INFO:teuthology.orchestra.run.smithi007.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T08:09:36.096 INFO:teuthology.orchestra.run.smithi007.stdout: "client_info": { 2024-09-18T08:09:36.096 INFO:teuthology.orchestra.run.smithi007.stdout: "client": "client.15216", 2024-09-18T08:09:36.096 INFO:teuthology.orchestra.run.smithi007.stdout: "client_addr": "172.21.15.7:0/2596041177", 2024-09-18T08:09:36.096 INFO:teuthology.orchestra.run.smithi007.stdout: "tid": 22665 2024-09-18T08:09:36.096 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:09:36.097 INFO:teuthology.orchestra.run.smithi007.stdout: "events": [ 2024-09-18T08:09:36.097 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:09:36.097 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "initiated", 2024-09-18T08:09:36.097 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:08.484581+0000", 2024-09-18T08:09:36.097 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-09-18T08:09:36.097 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:09:36.097 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:09:36.097 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "header_read", 2024-09-18T08:09:36.097 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:08.484581+0000", 2024-09-18T08:09:36.097 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-09-18T08:09:36.097 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:09:36.097 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:09:36.098 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "throttled", 2024-09-18T08:09:36.098 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:08.484583+0000", 2024-09-18T08:09:36.098 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.9250000000000002e-06 2024-09-18T08:09:36.098 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:09:36.098 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:09:36.098 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "all_read", 2024-09-18T08:09:36.098 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:08.484606+0000", 2024-09-18T08:09:36.098 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 2.2965000000000001e-05 2024-09-18T08:09:36.098 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:09:36.098 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:09:36.098 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "dispatched", 2024-09-18T08:09:36.098 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:08.484617+0000", 2024-09-18T08:09:36.098 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.1240000000000001e-05 2024-09-18T08:09:36.099 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:09:36.099 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:09:36.099 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "queued_for_pg", 2024-09-18T08:09:36.099 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:08.484625+0000", 2024-09-18T08:09:36.099 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 7.5020000000000003e-06 2024-09-18T08:09:36.099 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:09:36.099 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:09:36.099 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-09-18T08:09:36.099 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:08.484862+0000", 2024-09-18T08:09:36.099 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.00023732500000000001 2024-09-18T08:09:36.099 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:09:36.099 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:09:36.099 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "waiting for peered", 2024-09-18T08:09:36.100 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:08.484872+0000", 2024-09-18T08:09:36.100 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.0231000000000001e-05 2024-09-18T08:09:36.100 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:09:36.100 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:09:36.100 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-09-18T08:09:36.100 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:09.153091+0000", 2024-09-18T08:09:36.100 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.66821881900000002 2024-09-18T08:09:36.100 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:09:36.100 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:09:36.100 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "waiting for peered", 2024-09-18T08:09:36.100 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:09.153100+0000", 2024-09-18T08:09:36.100 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 8.7560000000000005e-06 2024-09-18T08:09:36.100 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:09:36.100 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:09:36.101 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-09-18T08:09:36.101 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:10.220552+0000", 2024-09-18T08:09:36.101 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.0674517699999999 2024-09-18T08:09:36.101 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:09:36.101 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:09:36.101 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "started", 2024-09-18T08:09:36.101 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:10.220667+0000", 2024-09-18T08:09:36.101 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.000115671 2024-09-18T08:09:36.101 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:09:36.102 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:09:36.102 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "waiting for subops from 1,6", 2024-09-18T08:09:36.102 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:10.220826+0000", 2024-09-18T08:09:36.102 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.00015917700000000001 2024-09-18T08:09:36.102 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:09:36.102 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:09:36.102 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "op_commit", 2024-09-18T08:09:36.102 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:10.224779+0000", 2024-09-18T08:09:36.102 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.0039521119999999998 2024-09-18T08:09:36.102 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:09:36.102 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:09:36.103 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:09:36.103 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:10.228102+0000", 2024-09-18T08:09:36.103 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.0033237230000000002 2024-09-18T08:09:36.103 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:09:36.103 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:09:36.103 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:09:36.103 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:10.228153+0000", 2024-09-18T08:09:36.103 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 5.0705000000000003e-05 2024-09-18T08:09:36.103 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:09:36.103 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:09:36.103 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "commit_sent", 2024-09-18T08:09:36.104 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:10.228212+0000", 2024-09-18T08:09:36.104 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 5.9404999999999998e-05 2024-09-18T08:09:36.104 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:09:36.104 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:09:36.104 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "done", 2024-09-18T08:09:36.104 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:10.228908+0000", 2024-09-18T08:09:36.104 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.00069591 2024-09-18T08:09:36.104 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-09-18T08:09:36.104 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-09-18T08:09:36.104 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-09-18T08:09:36.104 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:09:36.104 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:09:36.104 INFO:teuthology.orchestra.run.smithi007.stdout: "description": "osd_op(client.15216.0:22668 8.f 8:f26239dc:::benchmark_data_smithi007_101583_object22667:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e175)", 2024-09-18T08:09:36.105 INFO:teuthology.orchestra.run.smithi007.stdout: "initiated_at": "2024-09-18T08:08:08.486691+0000", 2024-09-18T08:09:36.105 INFO:teuthology.orchestra.run.smithi007.stdout: "age": 86.465122417999993, 2024-09-18T08:09:36.105 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.7435772489999999, 2024-09-18T08:09:36.105 INFO:teuthology.orchestra.run.smithi007.stdout: "continuous": false, 2024-09-18T08:09:36.105 INFO:teuthology.orchestra.run.smithi007.stdout: "type_data": { 2024-09-18T08:09:36.105 INFO:teuthology.orchestra.run.smithi007.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T08:09:36.105 INFO:teuthology.orchestra.run.smithi007.stdout: "client_info": { 2024-09-18T08:09:36.105 INFO:teuthology.orchestra.run.smithi007.stdout: "client": "client.15216", 2024-09-18T08:09:36.105 INFO:teuthology.orchestra.run.smithi007.stdout: "client_addr": "172.21.15.7:0/2596041177", 2024-09-18T08:09:36.105 INFO:teuthology.orchestra.run.smithi007.stdout: "tid": 22668 2024-09-18T08:09:36.105 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:09:36.105 INFO:teuthology.orchestra.run.smithi007.stdout: "events": [ 2024-09-18T08:09:36.105 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:09:36.106 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "initiated", 2024-09-18T08:09:36.106 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:08.486691+0000", 2024-09-18T08:09:36.106 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-09-18T08:09:36.106 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:09:36.106 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:09:36.106 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "header_read", 2024-09-18T08:09:36.106 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:08.486691+0000", 2024-09-18T08:09:36.106 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-09-18T08:09:36.106 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:09:36.106 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:09:36.106 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "throttled", 2024-09-18T08:09:36.106 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:08.486692+0000", 2024-09-18T08:09:36.106 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.3030000000000001e-06 2024-09-18T08:09:36.107 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:09:36.107 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:09:36.107 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "all_read", 2024-09-18T08:09:36.107 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:08.486716+0000", 2024-09-18T08:09:36.107 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 2.3164999999999999e-05 2024-09-18T08:09:36.107 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:09:36.107 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:09:36.107 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "dispatched", 2024-09-18T08:09:36.107 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:08.486728+0000", 2024-09-18T08:09:36.107 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.2648e-05 2024-09-18T08:09:36.107 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:09:36.107 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:09:36.107 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "queued_for_pg", 2024-09-18T08:09:36.107 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:08.486736+0000", 2024-09-18T08:09:36.108 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 7.8779999999999998e-06 2024-09-18T08:09:36.108 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:09:36.108 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:09:36.108 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-09-18T08:09:36.108 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:08.486921+0000", 2024-09-18T08:09:36.108 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.00018532100000000001 2024-09-18T08:09:36.108 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:09:36.108 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:09:36.108 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "waiting for peered", 2024-09-18T08:09:36.108 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:08.486930+0000", 2024-09-18T08:09:36.108 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 8.6300000000000004e-06 2024-09-18T08:09:36.108 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:09:36.108 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:09:36.109 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-09-18T08:09:36.109 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:09.153126+0000", 2024-09-18T08:09:36.109 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.66619636299999996 2024-09-18T08:09:36.109 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:09:36.109 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:09:36.109 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "waiting for peered", 2024-09-18T08:09:36.109 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:09.153280+0000", 2024-09-18T08:09:36.109 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.00015323800000000001 2024-09-18T08:09:36.109 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:09:36.109 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:09:36.109 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-09-18T08:09:36.109 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:10.223256+0000", 2024-09-18T08:09:36.109 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.069976313 2024-09-18T08:09:36.109 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:09:36.110 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:09:36.110 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "started", 2024-09-18T08:09:36.110 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:10.223428+0000", 2024-09-18T08:09:36.110 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.00017207299999999999 2024-09-18T08:09:36.110 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:09:36.110 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:09:36.110 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "waiting for subops from 1,6", 2024-09-18T08:09:36.110 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:10.223632+0000", 2024-09-18T08:09:36.110 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.000204289 2024-09-18T08:09:36.110 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:09:36.110 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:09:36.110 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "op_commit", 2024-09-18T08:09:36.110 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:10.228023+0000", 2024-09-18T08:09:36.110 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.0043910140000000004 2024-09-18T08:09:36.111 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:09:36.111 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:09:36.111 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:09:36.111 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:10.228966+0000", 2024-09-18T08:09:36.111 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.00094256400000000001 2024-09-18T08:09:36.111 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:09:36.111 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:09:36.111 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:09:36.111 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:10.229018+0000", 2024-09-18T08:09:36.111 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 5.2089000000000002e-05 2024-09-18T08:09:36.112 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:09:36.112 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:09:36.112 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "commit_sent", 2024-09-18T08:09:36.112 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:10.229075+0000", 2024-09-18T08:09:36.112 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 5.7079e-05 2024-09-18T08:09:36.112 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:09:36.112 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:09:36.112 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "done", 2024-09-18T08:09:36.113 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:10.230268+0000", 2024-09-18T08:09:36.113 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.0011932819999999999 2024-09-18T08:09:36.113 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-09-18T08:09:36.113 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-09-18T08:09:36.113 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-09-18T08:09:36.113 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:09:36.113 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:09:36.113 INFO:teuthology.orchestra.run.smithi007.stdout: "description": "osd_op(client.15216.0:22675 8.f 8:f41edfb7:::benchmark_data_smithi007_101583_object22674:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e175)", 2024-09-18T08:09:36.113 INFO:teuthology.orchestra.run.smithi007.stdout: "initiated_at": "2024-09-18T08:08:08.490781+0000", 2024-09-18T08:09:36.113 INFO:teuthology.orchestra.run.smithi007.stdout: "age": 86.461032605, 2024-09-18T08:09:36.113 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.74874595, 2024-09-18T08:09:36.113 INFO:teuthology.orchestra.run.smithi007.stdout: "continuous": false, 2024-09-18T08:09:36.113 INFO:teuthology.orchestra.run.smithi007.stdout: "type_data": { 2024-09-18T08:09:36.114 INFO:teuthology.orchestra.run.smithi007.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T08:09:36.114 INFO:teuthology.orchestra.run.smithi007.stdout: "client_info": { 2024-09-18T08:09:36.114 INFO:teuthology.orchestra.run.smithi007.stdout: "client": "client.15216", 2024-09-18T08:09:36.114 INFO:teuthology.orchestra.run.smithi007.stdout: "client_addr": "172.21.15.7:0/2596041177", 2024-09-18T08:09:36.114 INFO:teuthology.orchestra.run.smithi007.stdout: "tid": 22675 2024-09-18T08:09:36.114 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:09:36.114 INFO:teuthology.orchestra.run.smithi007.stdout: "events": [ 2024-09-18T08:09:36.114 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:09:36.114 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "initiated", 2024-09-18T08:09:36.114 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:08.490781+0000", 2024-09-18T08:09:36.114 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-09-18T08:09:36.114 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:09:36.115 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:09:36.115 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "header_read", 2024-09-18T08:09:36.115 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:08.490781+0000", 2024-09-18T08:09:36.115 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-09-18T08:09:36.115 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:09:36.115 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:09:36.115 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "throttled", 2024-09-18T08:09:36.115 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:08.490783+0000", 2024-09-18T08:09:36.115 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.708e-06 2024-09-18T08:09:36.115 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:09:36.116 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:09:36.116 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "all_read", 2024-09-18T08:09:36.116 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:08.490807+0000", 2024-09-18T08:09:36.116 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 2.4300000000000001e-05 2024-09-18T08:09:36.116 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:09:36.116 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:09:36.116 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "dispatched", 2024-09-18T08:09:36.116 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:08.490819+0000", 2024-09-18T08:09:36.116 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.2468e-05 2024-09-18T08:09:36.116 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:09:36.116 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:09:36.116 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "queued_for_pg", 2024-09-18T08:09:36.117 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:08.490829+0000", 2024-09-18T08:09:36.117 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 9.2320000000000007e-06 2024-09-18T08:09:36.117 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:09:36.117 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:09:36.117 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-09-18T08:09:36.117 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:08.490917+0000", 2024-09-18T08:09:36.117 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 8.8350000000000006e-05 2024-09-18T08:09:36.117 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:09:36.117 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:09:36.117 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "waiting for peered", 2024-09-18T08:09:36.117 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:08.490928+0000", 2024-09-18T08:09:36.117 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.0848e-05 2024-09-18T08:09:36.117 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:09:36.118 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:09:36.118 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-09-18T08:09:36.118 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:09.153316+0000", 2024-09-18T08:09:36.118 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.66238866600000001 2024-09-18T08:09:36.118 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:09:36.118 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:09:36.118 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "waiting for peered", 2024-09-18T08:09:36.118 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:09.153325+0000", 2024-09-18T08:09:36.118 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 8.1089999999999998e-06 2024-09-18T08:09:36.118 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:09:36.118 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:09:36.118 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-09-18T08:09:36.119 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:10.230312+0000", 2024-09-18T08:09:36.119 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.0769875980000001 2024-09-18T08:09:36.119 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:09:36.119 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:09:36.119 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "started", 2024-09-18T08:09:36.119 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:10.230436+0000", 2024-09-18T08:09:36.119 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.000123824 2024-09-18T08:09:36.119 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:09:36.119 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:09:36.119 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "waiting for subops from 1,6", 2024-09-18T08:09:36.120 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:10.230594+0000", 2024-09-18T08:09:36.120 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.00015772099999999999 2024-09-18T08:09:36.120 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:09:36.120 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:09:36.120 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "op_commit", 2024-09-18T08:09:36.120 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:10.234562+0000", 2024-09-18T08:09:36.120 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.0039686460000000002 2024-09-18T08:09:36.120 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:09:36.120 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:09:36.120 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:09:36.120 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:10.234625+0000", 2024-09-18T08:09:36.120 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 6.2675999999999994e-05 2024-09-18T08:09:36.120 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:09:36.121 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:09:36.121 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:09:36.121 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:10.238138+0000", 2024-09-18T08:09:36.121 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.0035124710000000001 2024-09-18T08:09:36.121 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:09:36.121 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:09:36.121 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "commit_sent", 2024-09-18T08:09:36.121 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:10.238230+0000", 2024-09-18T08:09:36.121 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 9.2304000000000003e-05 2024-09-18T08:09:36.121 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:09:36.121 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:09:36.121 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "done", 2024-09-18T08:09:36.121 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:10.239527+0000", 2024-09-18T08:09:36.122 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.0012970290000000001 2024-09-18T08:09:36.122 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-09-18T08:09:36.122 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-09-18T08:09:36.122 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-09-18T08:09:36.122 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:09:36.123 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:09:36.123 INFO:teuthology.orchestra.run.smithi007.stdout: "description": "osd_op(client.15216.0:22685 8.b 8:d2c6b3c8:::benchmark_data_smithi007_101583_object22684:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e175)", 2024-09-18T08:09:36.123 INFO:teuthology.orchestra.run.smithi007.stdout: "initiated_at": "2024-09-18T08:08:08.499266+0000", 2024-09-18T08:09:36.123 INFO:teuthology.orchestra.run.smithi007.stdout: "age": 86.452547631000002, 2024-09-18T08:09:36.123 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.781783984, 2024-09-18T08:09:36.123 INFO:teuthology.orchestra.run.smithi007.stdout: "continuous": false, 2024-09-18T08:09:36.123 INFO:teuthology.orchestra.run.smithi007.stdout: "type_data": { 2024-09-18T08:09:36.123 INFO:teuthology.orchestra.run.smithi007.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T08:09:36.123 INFO:teuthology.orchestra.run.smithi007.stdout: "client_info": { 2024-09-18T08:09:36.123 INFO:teuthology.orchestra.run.smithi007.stdout: "client": "client.15216", 2024-09-18T08:09:36.123 INFO:teuthology.orchestra.run.smithi007.stdout: "client_addr": "172.21.15.7:0/2596041177", 2024-09-18T08:09:36.123 INFO:teuthology.orchestra.run.smithi007.stdout: "tid": 22685 2024-09-18T08:09:36.124 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:09:36.124 INFO:teuthology.orchestra.run.smithi007.stdout: "events": [ 2024-09-18T08:09:36.124 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:09:36.124 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "initiated", 2024-09-18T08:09:36.124 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:08.499266+0000", 2024-09-18T08:09:36.124 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-09-18T08:09:36.124 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:09:36.124 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:09:36.124 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "header_read", 2024-09-18T08:09:36.124 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:08.499266+0000", 2024-09-18T08:09:36.124 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-09-18T08:09:36.124 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:09:36.124 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:09:36.125 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "throttled", 2024-09-18T08:09:36.125 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:08.499268+0000", 2024-09-18T08:09:36.125 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.7060000000000001e-06 2024-09-18T08:09:36.125 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:09:36.125 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:09:36.125 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "all_read", 2024-09-18T08:09:36.125 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:08.499297+0000", 2024-09-18T08:09:36.125 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 2.9298e-05 2024-09-18T08:09:36.125 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:09:36.125 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:09:36.125 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "dispatched", 2024-09-18T08:09:36.125 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:08.499318+0000", 2024-09-18T08:09:36.125 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 2.0758999999999999e-05 2024-09-18T08:09:36.125 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:09:36.126 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:09:36.126 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "queued_for_pg", 2024-09-18T08:09:36.126 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:08.499325+0000", 2024-09-18T08:09:36.126 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 7.8599999999999993e-06 2024-09-18T08:09:36.126 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:09:36.126 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:09:36.126 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-09-18T08:09:36.126 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:08.499379+0000", 2024-09-18T08:09:36.126 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 5.3257999999999997e-05 2024-09-18T08:09:36.126 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:09:36.126 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:09:36.126 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "waiting for peered", 2024-09-18T08:09:36.126 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:08.499391+0000", 2024-09-18T08:09:36.126 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.2483000000000001e-05 2024-09-18T08:09:36.126 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:09:36.127 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:09:36.127 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-09-18T08:09:36.127 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:09.152309+0000", 2024-09-18T08:09:36.127 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.65291789700000002 2024-09-18T08:09:36.127 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:09:36.127 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:09:36.127 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "waiting for peered", 2024-09-18T08:09:36.127 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:09.152317+0000", 2024-09-18T08:09:36.127 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 7.4830000000000002e-06 2024-09-18T08:09:36.127 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:09:36.127 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:09:36.127 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-09-18T08:09:36.127 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:10.260397+0000", 2024-09-18T08:09:36.127 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.108080043 2024-09-18T08:09:36.128 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:09:36.128 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:09:36.128 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "started", 2024-09-18T08:09:36.128 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:10.260553+0000", 2024-09-18T08:09:36.128 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.000156594 2024-09-18T08:09:36.128 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:09:36.128 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:09:36.128 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "waiting for subops from 1,7", 2024-09-18T08:09:36.128 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:10.260761+0000", 2024-09-18T08:09:36.128 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.00020734200000000001 2024-09-18T08:09:36.128 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:09:36.128 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:09:36.128 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:09:36.128 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:10.267815+0000", 2024-09-18T08:09:36.129 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.007053948 2024-09-18T08:09:36.129 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:09:36.129 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:09:36.129 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:09:36.129 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:10.267878+0000", 2024-09-18T08:09:36.129 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 6.3346e-05 2024-09-18T08:09:36.129 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:09:36.129 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:09:36.129 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "op_commit", 2024-09-18T08:09:36.129 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:10.280193+0000", 2024-09-18T08:09:36.129 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.012314782999999999 2024-09-18T08:09:36.129 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:09:36.129 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:09:36.129 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "commit_sent", 2024-09-18T08:09:36.130 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:10.280222+0000", 2024-09-18T08:09:36.130 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 2.9224e-05 2024-09-18T08:09:36.130 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:09:36.130 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:09:36.130 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "done", 2024-09-18T08:09:36.130 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:10.281050+0000", 2024-09-18T08:09:36.130 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.00082795999999999998 2024-09-18T08:09:36.130 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-09-18T08:09:36.130 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-09-18T08:09:36.130 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-09-18T08:09:36.130 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:09:36.130 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:09:36.130 INFO:teuthology.orchestra.run.smithi007.stdout: "description": "osd_op(client.15216.0:22689 8.f 8:f532acf4:::benchmark_data_smithi007_101583_object22688:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e175)", 2024-09-18T08:09:36.130 INFO:teuthology.orchestra.run.smithi007.stdout: "initiated_at": "2024-09-18T08:08:08.501798+0000", 2024-09-18T08:09:36.131 INFO:teuthology.orchestra.run.smithi007.stdout: "age": 86.450015616000002, 2024-09-18T08:09:36.131 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.7388306359999999, 2024-09-18T08:09:36.131 INFO:teuthology.orchestra.run.smithi007.stdout: "continuous": false, 2024-09-18T08:09:36.131 INFO:teuthology.orchestra.run.smithi007.stdout: "type_data": { 2024-09-18T08:09:36.131 INFO:teuthology.orchestra.run.smithi007.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T08:09:36.131 INFO:teuthology.orchestra.run.smithi007.stdout: "client_info": { 2024-09-18T08:09:36.131 INFO:teuthology.orchestra.run.smithi007.stdout: "client": "client.15216", 2024-09-18T08:09:36.131 INFO:teuthology.orchestra.run.smithi007.stdout: "client_addr": "172.21.15.7:0/2596041177", 2024-09-18T08:09:36.131 INFO:teuthology.orchestra.run.smithi007.stdout: "tid": 22689 2024-09-18T08:09:36.131 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:09:36.131 INFO:teuthology.orchestra.run.smithi007.stdout: "events": [ 2024-09-18T08:09:36.131 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:09:36.131 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "initiated", 2024-09-18T08:09:36.132 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:08.501798+0000", 2024-09-18T08:09:36.132 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-09-18T08:09:36.132 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:09:36.132 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:09:36.132 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "header_read", 2024-09-18T08:09:36.132 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:08.501798+0000", 2024-09-18T08:09:36.132 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-09-18T08:09:36.132 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:09:36.132 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:09:36.132 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "throttled", 2024-09-18T08:09:36.132 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:08.501798+0000", 2024-09-18T08:09:36.132 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 5.9400000000000005e-07 2024-09-18T08:09:36.132 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:09:36.133 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:09:36.133 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "all_read", 2024-09-18T08:09:36.133 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:08.501835+0000", 2024-09-18T08:09:36.133 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 3.6974000000000002e-05 2024-09-18T08:09:36.133 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:09:36.133 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:09:36.133 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "dispatched", 2024-09-18T08:09:36.133 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:08.501841+0000", 2024-09-18T08:09:36.133 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 5.2909999999999998e-06 2024-09-18T08:09:36.134 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:09:36.134 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:09:36.134 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "queued_for_pg", 2024-09-18T08:09:36.134 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:08.501844+0000", 2024-09-18T08:09:36.134 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 3.7330000000000001e-06 2024-09-18T08:09:36.134 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:09:36.134 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:09:36.134 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-09-18T08:09:36.134 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:08.501904+0000", 2024-09-18T08:09:36.134 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 5.9861000000000002e-05 2024-09-18T08:09:36.134 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:09:36.134 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:09:36.134 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "waiting for peered", 2024-09-18T08:09:36.135 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:08.501917+0000", 2024-09-18T08:09:36.135 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.2181000000000001e-05 2024-09-18T08:09:36.135 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:09:36.135 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:09:36.135 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-09-18T08:09:36.135 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:09.153352+0000", 2024-09-18T08:09:36.135 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.65143504100000005 2024-09-18T08:09:36.135 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:09:36.135 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:09:36.135 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "waiting for peered", 2024-09-18T08:09:36.135 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:09.153359+0000", 2024-09-18T08:09:36.135 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 7.4760000000000001e-06 2024-09-18T08:09:36.135 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:09:36.136 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:09:36.136 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-09-18T08:09:36.136 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:10.232890+0000", 2024-09-18T08:09:36.136 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.0795311249999999 2024-09-18T08:09:36.136 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:09:36.136 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:09:36.136 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "started", 2024-09-18T08:09:36.136 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:10.233035+0000", 2024-09-18T08:09:36.136 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.000145282 2024-09-18T08:09:36.136 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:09:36.136 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:09:36.136 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "waiting for subops from 1,6", 2024-09-18T08:09:36.136 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:10.233242+0000", 2024-09-18T08:09:36.136 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.000206181 2024-09-18T08:09:36.137 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:09:36.137 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:09:36.137 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:09:36.137 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:10.238074+0000", 2024-09-18T08:09:36.137 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.0048323649999999999 2024-09-18T08:09:36.137 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:09:36.137 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:09:36.137 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "op_commit", 2024-09-18T08:09:36.137 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:10.238090+0000", 2024-09-18T08:09:36.137 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.5934e-05 2024-09-18T08:09:36.137 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:09:36.137 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:09:36.137 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:09:36.137 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:10.239669+0000", 2024-09-18T08:09:36.138 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.0015793039999999999 2024-09-18T08:09:36.138 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:09:36.138 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:09:36.138 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "commit_sent", 2024-09-18T08:09:36.138 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:10.239733+0000", 2024-09-18T08:09:36.138 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 6.3404999999999993e-05 2024-09-18T08:09:36.138 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:09:36.138 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:09:36.138 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "done", 2024-09-18T08:09:36.138 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:10.240629+0000", 2024-09-18T08:09:36.138 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.00089588899999999997 2024-09-18T08:09:36.138 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-09-18T08:09:36.138 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-09-18T08:09:36.138 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-09-18T08:09:36.139 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:09:36.139 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:09:36.139 INFO:teuthology.orchestra.run.smithi007.stdout: "description": "osd_op(client.15216.0:69187 8.f 8:f51333c2:::benchmark_data_smithi007_101583_object69186:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e198)", 2024-09-18T08:09:36.139 INFO:teuthology.orchestra.run.smithi007.stdout: "initiated_at": "2024-09-18T08:08:51.699105+0000", 2024-09-18T08:09:36.139 INFO:teuthology.orchestra.run.smithi007.stdout: "age": 43.252708120000001, 2024-09-18T08:09:36.139 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 2.1312144279999998, 2024-09-18T08:09:36.139 INFO:teuthology.orchestra.run.smithi007.stdout: "continuous": false, 2024-09-18T08:09:36.139 INFO:teuthology.orchestra.run.smithi007.stdout: "type_data": { 2024-09-18T08:09:36.139 INFO:teuthology.orchestra.run.smithi007.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T08:09:36.139 INFO:teuthology.orchestra.run.smithi007.stdout: "client_info": { 2024-09-18T08:09:36.139 INFO:teuthology.orchestra.run.smithi007.stdout: "client": "client.15216", 2024-09-18T08:09:36.139 INFO:teuthology.orchestra.run.smithi007.stdout: "client_addr": "172.21.15.7:0/2596041177", 2024-09-18T08:09:36.139 INFO:teuthology.orchestra.run.smithi007.stdout: "tid": 69187 2024-09-18T08:09:36.140 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:09:36.140 INFO:teuthology.orchestra.run.smithi007.stdout: "events": [ 2024-09-18T08:09:36.140 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:09:36.140 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "initiated", 2024-09-18T08:09:36.140 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:51.699105+0000", 2024-09-18T08:09:36.140 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-09-18T08:09:36.140 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:09:36.140 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:09:36.140 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "header_read", 2024-09-18T08:09:36.140 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:51.699105+0000", 2024-09-18T08:09:36.140 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-09-18T08:09:36.140 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:09:36.140 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:09:36.140 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "throttled", 2024-09-18T08:09:36.141 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:51.699107+0000", 2024-09-18T08:09:36.141 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.5149999999999999e-06 2024-09-18T08:09:36.141 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:09:36.141 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:09:36.141 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "all_read", 2024-09-18T08:09:36.141 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:51.699133+0000", 2024-09-18T08:09:36.141 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 2.5752999999999999e-05 2024-09-18T08:09:36.141 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:09:36.141 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:09:36.141 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "dispatched", 2024-09-18T08:09:36.141 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:51.699141+0000", 2024-09-18T08:09:36.142 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 8.2709999999999992e-06 2024-09-18T08:09:36.142 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:09:36.142 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:09:36.142 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "queued_for_pg", 2024-09-18T08:09:36.142 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:51.699163+0000", 2024-09-18T08:09:36.142 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 2.2432000000000001e-05 2024-09-18T08:09:36.142 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:09:36.142 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:09:36.142 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-09-18T08:09:36.142 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:51.699402+0000", 2024-09-18T08:09:36.142 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.000239046 2024-09-18T08:09:36.142 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:09:36.143 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:09:36.143 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "waiting for peered", 2024-09-18T08:09:36.143 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:51.699413+0000", 2024-09-18T08:09:36.143 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.0387000000000001e-05 2024-09-18T08:09:36.143 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:09:36.143 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:09:36.143 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-09-18T08:09:36.143 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:52.696975+0000", 2024-09-18T08:09:36.143 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.99756218100000005 2024-09-18T08:09:36.143 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:09:36.144 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:09:36.144 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "waiting for peered", 2024-09-18T08:09:36.144 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:52.697002+0000", 2024-09-18T08:09:36.144 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 2.7464e-05 2024-09-18T08:09:36.144 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:09:36.144 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:09:36.144 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-09-18T08:09:36.144 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:53.824414+0000", 2024-09-18T08:09:36.144 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.1274112789999999 2024-09-18T08:09:36.144 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:09:36.144 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:09:36.144 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "started", 2024-09-18T08:09:36.144 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:53.824573+0000", 2024-09-18T08:09:36.144 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.00015908400000000001 2024-09-18T08:09:36.145 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:09:36.145 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:09:36.145 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "waiting for subops from 1,2,4", 2024-09-18T08:09:36.145 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:53.824756+0000", 2024-09-18T08:09:36.145 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.00018345199999999999 2024-09-18T08:09:36.145 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:09:36.145 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:09:36.145 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:09:36.145 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:53.827276+0000", 2024-09-18T08:09:36.145 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.002519444 2024-09-18T08:09:36.145 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:09:36.145 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:09:36.145 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:09:36.146 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:53.827356+0000", 2024-09-18T08:09:36.146 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 8.0029000000000002e-05 2024-09-18T08:09:36.146 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:09:36.146 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:09:36.146 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "op_commit", 2024-09-18T08:09:36.146 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:53.827376+0000", 2024-09-18T08:09:36.147 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 2.0250000000000001e-05 2024-09-18T08:09:36.147 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:09:36.147 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:09:36.147 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:09:36.147 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:53.830181+0000", 2024-09-18T08:09:36.147 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.0028047469999999998 2024-09-18T08:09:36.148 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:09:36.148 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:09:36.148 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "commit_sent", 2024-09-18T08:09:36.148 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:53.830246+0000", 2024-09-18T08:09:36.148 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 6.5140000000000003e-05 2024-09-18T08:09:36.148 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:09:36.149 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:09:36.149 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "done", 2024-09-18T08:09:36.149 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:53.830320+0000", 2024-09-18T08:09:36.149 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 7.3954000000000004e-05 2024-09-18T08:09:36.149 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-09-18T08:09:36.149 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-09-18T08:09:36.149 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-09-18T08:09:36.149 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:09:36.149 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:09:36.150 INFO:teuthology.orchestra.run.smithi007.stdout: "description": "osd_op(client.15216.0:69195 8.f 8:fafe41e6:::benchmark_data_smithi007_101583_object69194:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e198)", 2024-09-18T08:09:36.150 INFO:teuthology.orchestra.run.smithi007.stdout: "initiated_at": "2024-09-18T08:08:51.711966+0000", 2024-09-18T08:09:36.150 INFO:teuthology.orchestra.run.smithi007.stdout: "age": 43.239847357999999, 2024-09-18T08:09:36.150 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 2.1187014670000002, 2024-09-18T08:09:36.150 INFO:teuthology.orchestra.run.smithi007.stdout: "continuous": false, 2024-09-18T08:09:36.150 INFO:teuthology.orchestra.run.smithi007.stdout: "type_data": { 2024-09-18T08:09:36.150 INFO:teuthology.orchestra.run.smithi007.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T08:09:36.150 INFO:teuthology.orchestra.run.smithi007.stdout: "client_info": { 2024-09-18T08:09:36.150 INFO:teuthology.orchestra.run.smithi007.stdout: "client": "client.15216", 2024-09-18T08:09:36.150 INFO:teuthology.orchestra.run.smithi007.stdout: "client_addr": "172.21.15.7:0/2596041177", 2024-09-18T08:09:36.150 INFO:teuthology.orchestra.run.smithi007.stdout: "tid": 69195 2024-09-18T08:09:36.150 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:09:36.150 INFO:teuthology.orchestra.run.smithi007.stdout: "events": [ 2024-09-18T08:09:36.151 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:09:36.151 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "initiated", 2024-09-18T08:09:36.151 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:51.711966+0000", 2024-09-18T08:09:36.151 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-09-18T08:09:36.151 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:09:36.151 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:09:36.152 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "header_read", 2024-09-18T08:09:36.152 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:51.711966+0000", 2024-09-18T08:09:36.152 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-09-18T08:09:36.152 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:09:36.152 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:09:36.152 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "throttled", 2024-09-18T08:09:36.152 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:51.711968+0000", 2024-09-18T08:09:36.152 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 2.013e-06 2024-09-18T08:09:36.153 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:09:36.153 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:09:36.153 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "all_read", 2024-09-18T08:09:36.153 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:51.711997+0000", 2024-09-18T08:09:36.153 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 2.8561999999999999e-05 2024-09-18T08:09:36.153 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:09:36.153 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:09:36.153 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "dispatched", 2024-09-18T08:09:36.153 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:51.712012+0000", 2024-09-18T08:09:36.153 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.5184000000000001e-05 2024-09-18T08:09:36.153 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:09:36.154 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:09:36.154 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "queued_for_pg", 2024-09-18T08:09:36.154 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:51.712022+0000", 2024-09-18T08:09:36.154 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 9.6530000000000006e-06 2024-09-18T08:09:36.154 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:09:36.154 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:09:36.154 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-09-18T08:09:36.154 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:51.712088+0000", 2024-09-18T08:09:36.155 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 6.6921000000000005e-05 2024-09-18T08:09:36.155 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:09:36.155 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:09:36.155 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "waiting for peered", 2024-09-18T08:09:36.155 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:51.712101+0000", 2024-09-18T08:09:36.155 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.2768e-05 2024-09-18T08:09:36.155 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:09:36.155 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:09:36.155 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-09-18T08:09:36.155 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:52.697039+0000", 2024-09-18T08:09:36.155 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.98493815299999998 2024-09-18T08:09:36.155 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:09:36.156 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:09:36.156 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "waiting for peered", 2024-09-18T08:09:36.156 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:52.697052+0000", 2024-09-18T08:09:36.156 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.2475e-05 2024-09-18T08:09:36.156 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:09:36.156 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:09:36.156 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-09-18T08:09:36.156 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:53.827400+0000", 2024-09-18T08:09:36.156 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.1303482419999999 2024-09-18T08:09:36.157 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:09:36.157 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:09:36.157 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "started", 2024-09-18T08:09:36.157 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:53.827528+0000", 2024-09-18T08:09:36.157 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.000128009 2024-09-18T08:09:36.157 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:09:36.157 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:09:36.157 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "waiting for subops from 1,2,4", 2024-09-18T08:09:36.157 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:53.827701+0000", 2024-09-18T08:09:36.157 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.00017307100000000001 2024-09-18T08:09:36.158 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:09:36.158 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:09:36.158 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:09:36.158 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:53.830376+0000", 2024-09-18T08:09:36.158 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.002675144 2024-09-18T08:09:36.158 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:09:36.158 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:09:36.158 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:09:36.158 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:53.830419+0000", 2024-09-18T08:09:36.158 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 4.3087000000000003e-05 2024-09-18T08:09:36.158 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:09:36.158 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:09:36.158 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "op_commit", 2024-09-18T08:09:36.159 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:53.830444+0000", 2024-09-18T08:09:36.159 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 2.4579e-05 2024-09-18T08:09:36.159 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:09:36.159 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:09:36.159 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:09:36.159 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:53.830485+0000", 2024-09-18T08:09:36.159 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 4.1236999999999998e-05 2024-09-18T08:09:36.159 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:09:36.159 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:09:36.159 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "commit_sent", 2024-09-18T08:09:36.159 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:53.830541+0000", 2024-09-18T08:09:36.160 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 5.6075000000000001e-05 2024-09-18T08:09:36.160 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:09:36.160 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:09:36.160 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "done", 2024-09-18T08:09:36.160 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:53.830668+0000", 2024-09-18T08:09:36.160 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.00012629399999999999 2024-09-18T08:09:36.160 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-09-18T08:09:36.160 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-09-18T08:09:36.160 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-09-18T08:09:36.160 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:09:36.160 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:09:36.160 INFO:teuthology.orchestra.run.smithi007.stdout: "description": "osd_op(client.15216.0:69207 8.f 8:f63411d3:::benchmark_data_smithi007_101583_object69206:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e198)", 2024-09-18T08:09:36.161 INFO:teuthology.orchestra.run.smithi007.stdout: "initiated_at": "2024-09-18T08:08:51.719518+0000", 2024-09-18T08:09:36.161 INFO:teuthology.orchestra.run.smithi007.stdout: "age": 43.232294998, 2024-09-18T08:09:36.161 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 2.1171781319999998, 2024-09-18T08:09:36.161 INFO:teuthology.orchestra.run.smithi007.stdout: "continuous": false, 2024-09-18T08:09:36.161 INFO:teuthology.orchestra.run.smithi007.stdout: "type_data": { 2024-09-18T08:09:36.161 INFO:teuthology.orchestra.run.smithi007.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T08:09:36.161 INFO:teuthology.orchestra.run.smithi007.stdout: "client_info": { 2024-09-18T08:09:36.161 INFO:teuthology.orchestra.run.smithi007.stdout: "client": "client.15216", 2024-09-18T08:09:36.161 INFO:teuthology.orchestra.run.smithi007.stdout: "client_addr": "172.21.15.7:0/2596041177", 2024-09-18T08:09:36.161 INFO:teuthology.orchestra.run.smithi007.stdout: "tid": 69207 2024-09-18T08:09:36.161 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:09:36.162 INFO:teuthology.orchestra.run.smithi007.stdout: "events": [ 2024-09-18T08:09:36.162 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:09:36.162 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "initiated", 2024-09-18T08:09:36.162 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:51.719518+0000", 2024-09-18T08:09:36.162 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-09-18T08:09:36.162 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:09:36.162 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:09:36.162 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "header_read", 2024-09-18T08:09:36.163 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:51.719518+0000", 2024-09-18T08:09:36.163 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-09-18T08:09:36.163 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:09:36.163 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:09:36.163 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "throttled", 2024-09-18T08:09:36.163 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:51.719519+0000", 2024-09-18T08:09:36.163 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 5.7999999999999995e-07 2024-09-18T08:09:36.164 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:09:36.164 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:09:36.164 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "all_read", 2024-09-18T08:09:36.164 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:51.719540+0000", 2024-09-18T08:09:36.164 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 2.1097000000000001e-05 2024-09-18T08:09:36.164 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:09:36.164 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:09:36.164 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "dispatched", 2024-09-18T08:09:36.165 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:51.719547+0000", 2024-09-18T08:09:36.165 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 6.4810000000000003e-06 2024-09-18T08:09:36.165 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:09:36.165 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:09:36.165 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "queued_for_pg", 2024-09-18T08:09:36.165 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:51.719552+0000", 2024-09-18T08:09:36.165 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 5.2100000000000001e-06 2024-09-18T08:09:36.165 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:09:36.165 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:09:36.165 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-09-18T08:09:36.165 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:51.719676+0000", 2024-09-18T08:09:36.165 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.00012415899999999999 2024-09-18T08:09:36.166 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:09:36.166 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:09:36.166 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "waiting for peered", 2024-09-18T08:09:36.166 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:51.719686+0000", 2024-09-18T08:09:36.166 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 9.5000000000000005e-06 2024-09-18T08:09:36.166 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:09:36.167 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:09:36.167 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-09-18T08:09:36.167 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:52.697757+0000", 2024-09-18T08:09:36.167 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.97807110100000005 2024-09-18T08:09:36.167 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:09:36.167 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:09:36.168 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "waiting for peered", 2024-09-18T08:09:36.168 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:52.697767+0000", 2024-09-18T08:09:36.168 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 9.8810000000000008e-06 2024-09-18T08:09:36.168 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:09:36.168 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:09:36.168 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-09-18T08:09:36.168 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:53.830718+0000", 2024-09-18T08:09:36.168 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.1329519370000001 2024-09-18T08:09:36.169 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:09:36.169 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:09:36.169 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "started", 2024-09-18T08:09:36.169 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:53.830870+0000", 2024-09-18T08:09:36.169 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.00015120300000000001 2024-09-18T08:09:36.169 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:09:36.169 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:09:36.169 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "waiting for subops from 1,2,4", 2024-09-18T08:09:36.169 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:53.831043+0000", 2024-09-18T08:09:36.169 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.00017385499999999999 2024-09-18T08:09:36.170 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:09:36.170 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:09:36.170 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "op_commit", 2024-09-18T08:09:36.170 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:53.836372+0000", 2024-09-18T08:09:36.170 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.0053287689999999997 2024-09-18T08:09:36.171 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:09:36.171 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:09:36.171 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:09:36.171 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:53.836420+0000", 2024-09-18T08:09:36.171 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 4.7490000000000001e-05 2024-09-18T08:09:36.171 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:09:36.171 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:09:36.171 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:09:36.171 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:53.836464+0000", 2024-09-18T08:09:36.171 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 4.4254e-05 2024-09-18T08:09:36.171 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:09:36.171 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:09:36.171 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:09:36.171 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:53.836505+0000", 2024-09-18T08:09:36.172 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 4.1414e-05 2024-09-18T08:09:36.172 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:09:36.172 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:09:36.172 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "commit_sent", 2024-09-18T08:09:36.172 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:53.836566+0000", 2024-09-18T08:09:36.172 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 6.0538999999999998e-05 2024-09-18T08:09:36.172 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:09:36.172 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:09:36.172 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "done", 2024-09-18T08:09:36.172 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:53.836697+0000", 2024-09-18T08:09:36.172 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.000130662 2024-09-18T08:09:36.172 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-09-18T08:09:36.172 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-09-18T08:09:36.173 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-09-18T08:09:36.173 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:09:36.173 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:09:36.173 INFO:teuthology.orchestra.run.smithi007.stdout: "description": "osd_op(client.15216.0:69218 8.f 8:fd5c3396:::benchmark_data_smithi007_101583_object69217:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e198)", 2024-09-18T08:09:36.173 INFO:teuthology.orchestra.run.smithi007.stdout: "initiated_at": "2024-09-18T08:08:51.731702+0000", 2024-09-18T08:09:36.173 INFO:teuthology.orchestra.run.smithi007.stdout: "age": 43.220111742, 2024-09-18T08:09:36.173 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 2.1112667140000001, 2024-09-18T08:09:36.173 INFO:teuthology.orchestra.run.smithi007.stdout: "continuous": false, 2024-09-18T08:09:36.173 INFO:teuthology.orchestra.run.smithi007.stdout: "type_data": { 2024-09-18T08:09:36.173 INFO:teuthology.orchestra.run.smithi007.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T08:09:36.173 INFO:teuthology.orchestra.run.smithi007.stdout: "client_info": { 2024-09-18T08:09:36.173 INFO:teuthology.orchestra.run.smithi007.stdout: "client": "client.15216", 2024-09-18T08:09:36.173 INFO:teuthology.orchestra.run.smithi007.stdout: "client_addr": "172.21.15.7:0/2596041177", 2024-09-18T08:09:36.174 INFO:teuthology.orchestra.run.smithi007.stdout: "tid": 69218 2024-09-18T08:09:36.174 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:09:36.174 INFO:teuthology.orchestra.run.smithi007.stdout: "events": [ 2024-09-18T08:09:36.174 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:09:36.174 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "initiated", 2024-09-18T08:09:36.174 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:51.731702+0000", 2024-09-18T08:09:36.174 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-09-18T08:09:36.174 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:09:36.174 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:09:36.174 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "header_read", 2024-09-18T08:09:36.174 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:51.731702+0000", 2024-09-18T08:09:36.174 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-09-18T08:09:36.174 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:09:36.174 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:09:36.175 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "throttled", 2024-09-18T08:09:36.175 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:51.731703+0000", 2024-09-18T08:09:36.175 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.265e-06 2024-09-18T08:09:36.175 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:09:36.175 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:09:36.175 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "all_read", 2024-09-18T08:09:36.175 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:51.731771+0000", 2024-09-18T08:09:36.175 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 6.7945000000000001e-05 2024-09-18T08:09:36.175 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:09:36.175 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:09:36.175 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "dispatched", 2024-09-18T08:09:36.175 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:51.731781+0000", 2024-09-18T08:09:36.175 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 9.8030000000000008e-06 2024-09-18T08:09:36.175 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:09:36.176 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:09:36.176 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "queued_for_pg", 2024-09-18T08:09:36.176 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:51.731788+0000", 2024-09-18T08:09:36.176 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 7.1620000000000002e-06 2024-09-18T08:09:36.176 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:09:36.176 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:09:36.176 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-09-18T08:09:36.176 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:51.731861+0000", 2024-09-18T08:09:36.176 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 7.2669999999999994e-05 2024-09-18T08:09:36.176 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:09:36.176 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:09:36.176 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "waiting for peered", 2024-09-18T08:09:36.176 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:51.731875+0000", 2024-09-18T08:09:36.177 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.4209999999999999e-05 2024-09-18T08:09:36.177 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:09:36.177 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:09:36.177 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-09-18T08:09:36.177 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:52.697807+0000", 2024-09-18T08:09:36.177 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.96593184200000004 2024-09-18T08:09:36.177 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:09:36.177 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:09:36.177 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "waiting for peered", 2024-09-18T08:09:36.177 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:52.697815+0000", 2024-09-18T08:09:36.177 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 8.4239999999999993e-06 2024-09-18T08:09:36.177 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:09:36.177 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:09:36.178 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-09-18T08:09:36.178 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:53.833616+0000", 2024-09-18T08:09:36.178 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.1358008079999999 2024-09-18T08:09:36.178 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:09:36.178 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:09:36.178 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "started", 2024-09-18T08:09:36.178 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:53.833763+0000", 2024-09-18T08:09:36.178 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.000146663 2024-09-18T08:09:36.178 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:09:36.178 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:09:36.178 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "waiting for subops from 1,2,4", 2024-09-18T08:09:36.178 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:53.833911+0000", 2024-09-18T08:09:36.179 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.00014877900000000001 2024-09-18T08:09:36.179 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:09:36.179 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:09:36.179 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:09:36.179 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:53.836740+0000", 2024-09-18T08:09:36.179 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.002828882 2024-09-18T08:09:36.179 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:09:36.179 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:09:36.179 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "op_commit", 2024-09-18T08:09:36.179 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:53.836756+0000", 2024-09-18T08:09:36.179 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.6195999999999999e-05 2024-09-18T08:09:36.179 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:09:36.179 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:09:36.179 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:09:36.180 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:53.842695+0000", 2024-09-18T08:09:36.180 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.0059383209999999999 2024-09-18T08:09:36.180 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:09:36.180 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:09:36.180 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:09:36.180 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:53.842746+0000", 2024-09-18T08:09:36.180 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 5.0966e-05 2024-09-18T08:09:36.180 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:09:36.180 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:09:36.180 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "commit_sent", 2024-09-18T08:09:36.180 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:53.842817+0000", 2024-09-18T08:09:36.180 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 7.1437000000000001e-05 2024-09-18T08:09:36.180 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:09:36.181 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:09:36.181 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "done", 2024-09-18T08:09:36.181 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:53.842968+0000", 2024-09-18T08:09:36.181 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.00015134100000000001 2024-09-18T08:09:36.181 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-09-18T08:09:36.181 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-09-18T08:09:36.181 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-09-18T08:09:36.181 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:09:36.181 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:09:36.181 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6 2024-09-18T08:09:36.183 INFO:teuthology.orchestra.run.smithi007.stdout: "description": "osd_op(client.15216.0:69239 8.f 8:f2d8399e:::benchmark_data_smithi007_101583_object69238:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e198)", 2024-09-18T08:09:36.183 INFO:teuthology.orchestra.run.smithi007.stdout: "initiated_at": "2024-09-18T08:08:51.752739+0000", 2024-09-18T08:09:36.183 INFO:teuthology.orchestra.run.smithi007.stdout: "age": 43.199074211000003, 2024-09-18T08:09:36.183 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 2.0962772900000002, 2024-09-18T08:09:36.183 INFO:teuthology.orchestra.run.smithi007.stdout: "continuous": false, 2024-09-18T08:09:36.183 INFO:teuthology.orchestra.run.smithi007.stdout: "type_data": { 2024-09-18T08:09:36.183 INFO:teuthology.orchestra.run.smithi007.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T08:09:36.183 INFO:teuthology.orchestra.run.smithi007.stdout: "client_info": { 2024-09-18T08:09:36.183 INFO:teuthology.orchestra.run.smithi007.stdout: "client": "client.15216", 2024-09-18T08:09:36.183 INFO:teuthology.orchestra.run.smithi007.stdout: "client_addr": "172.21.15.7:0/2596041177", 2024-09-18T08:09:36.183 INFO:teuthology.orchestra.run.smithi007.stdout: "tid": 69239 2024-09-18T08:09:36.184 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:09:36.184 INFO:teuthology.orchestra.run.smithi007.stdout: "events": [ 2024-09-18T08:09:36.184 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:09:36.184 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "initiated", 2024-09-18T08:09:36.184 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:51.752739+0000", 2024-09-18T08:09:36.184 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-09-18T08:09:36.184 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:09:36.184 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:09:36.184 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "header_read", 2024-09-18T08:09:36.184 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:51.752739+0000", 2024-09-18T08:09:36.184 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-09-18T08:09:36.184 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:09:36.184 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:09:36.185 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "throttled", 2024-09-18T08:09:36.185 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:51.752741+0000", 2024-09-18T08:09:36.185 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.829e-06 2024-09-18T08:09:36.185 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:09:36.185 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:09:36.185 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "all_read", 2024-09-18T08:09:36.185 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:51.752766+0000", 2024-09-18T08:09:36.185 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 2.4842e-05 2024-09-18T08:09:36.185 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:09:36.185 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:09:36.185 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "dispatched", 2024-09-18T08:09:36.185 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:51.752777+0000", 2024-09-18T08:09:36.185 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.1395e-05 2024-09-18T08:09:36.186 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:09:36.186 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:09:36.186 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "queued_for_pg", 2024-09-18T08:09:36.186 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:51.752785+0000", 2024-09-18T08:09:36.186 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 7.401e-06 2024-09-18T08:09:36.186 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:09:36.186 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:09:36.186 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-09-18T08:09:36.186 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:51.753101+0000", 2024-09-18T08:09:36.186 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.00031663600000000002 2024-09-18T08:09:36.186 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:09:36.187 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:09:36.187 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "waiting for peered", 2024-09-18T08:09:36.187 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:51.753112+0000", 2024-09-18T08:09:36.187 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.0825000000000001e-05 2024-09-18T08:09:36.187 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:09:36.187 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:09:36.187 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-09-18T08:09:36.187 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:52.697883+0000", 2024-09-18T08:09:36.187 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.94477077700000001 2024-09-18T08:09:36.187 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:09:36.187 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:09:36.187 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "waiting for peered", 2024-09-18T08:09:36.187 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:52.697893+0000", 2024-09-18T08:09:36.188 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 9.5820000000000008e-06 2024-09-18T08:09:36.188 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:09:36.188 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:09:36.188 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-09-18T08:09:36.188 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:53.836781+0000", 2024-09-18T08:09:36.188 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.1388879329999999 2024-09-18T08:09:36.188 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:09:36.188 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:09:36.188 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "started", 2024-09-18T08:09:36.188 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:53.836912+0000", 2024-09-18T08:09:36.188 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.00013171799999999999 2024-09-18T08:09:36.188 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:09:36.188 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:09:36.189 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "waiting for subops from 1,2,4", 2024-09-18T08:09:36.189 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:53.837063+0000", 2024-09-18T08:09:36.189 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.00015040799999999999 2024-09-18T08:09:36.189 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:09:36.189 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:09:36.189 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:09:36.189 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:53.843015+0000", 2024-09-18T08:09:36.189 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.0059519250000000003 2024-09-18T08:09:36.190 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:09:36.190 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:09:36.190 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:09:36.190 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:53.843063+0000", 2024-09-18T08:09:36.190 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 4.8199999999999999e-05 2024-09-18T08:09:36.190 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:09:36.190 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:09:36.190 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "op_commit", 2024-09-18T08:09:36.190 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:53.848550+0000", 2024-09-18T08:09:36.190 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.0054868929999999996 2024-09-18T08:09:36.190 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:09:36.190 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:09:36.190 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:09:36.190 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:53.848663+0000", 2024-09-18T08:09:36.190 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.000113195 2024-09-18T08:09:36.191 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:09:36.191 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:09:36.191 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "commit_sent", 2024-09-18T08:09:36.191 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:53.848860+0000", 2024-09-18T08:09:36.191 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.00019676000000000001 2024-09-18T08:09:36.191 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:09:36.191 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:09:36.191 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "done", 2024-09-18T08:09:36.191 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:53.849017+0000", 2024-09-18T08:09:36.191 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.000156971 2024-09-18T08:09:36.191 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-09-18T08:09:36.191 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-09-18T08:09:36.191 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-09-18T08:09:36.191 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:09:36.192 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:09:36.192 INFO:teuthology.orchestra.run.smithi007.stdout: "description": "osd_op(client.15216.0:69262 8.f 8:f2d117f8:::benchmark_data_smithi007_101583_object69261:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e198)", 2024-09-18T08:09:36.192 INFO:teuthology.orchestra.run.smithi007.stdout: "initiated_at": "2024-09-18T08:08:51.776735+0000", 2024-09-18T08:09:36.192 INFO:teuthology.orchestra.run.smithi007.stdout: "age": 43.175078571, 2024-09-18T08:09:36.192 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 2.0727967970000001, 2024-09-18T08:09:36.192 INFO:teuthology.orchestra.run.smithi007.stdout: "continuous": false, 2024-09-18T08:09:36.192 INFO:teuthology.orchestra.run.smithi007.stdout: "type_data": { 2024-09-18T08:09:36.192 INFO:teuthology.orchestra.run.smithi007.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T08:09:36.192 INFO:teuthology.orchestra.run.smithi007.stdout: "client_info": { 2024-09-18T08:09:36.192 INFO:teuthology.orchestra.run.smithi007.stdout: "client": "client.15216", 2024-09-18T08:09:36.192 INFO:teuthology.orchestra.run.smithi007.stdout: "client_addr": "172.21.15.7:0/2596041177", 2024-09-18T08:09:36.193 INFO:teuthology.orchestra.run.smithi007.stdout: "tid": 69262 2024-09-18T08:09:36.193 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:09:36.193 INFO:teuthology.orchestra.run.smithi007.stdout: "events": [ 2024-09-18T08:09:36.193 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:09:36.193 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "initiated", 2024-09-18T08:09:36.193 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:51.776735+0000", 2024-09-18T08:09:36.193 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-09-18T08:09:36.193 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:09:36.193 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:09:36.193 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "header_read", 2024-09-18T08:09:36.193 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:51.776735+0000", 2024-09-18T08:09:36.193 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-09-18T08:09:36.194 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:09:36.194 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:09:36.194 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "throttled", 2024-09-18T08:09:36.194 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:51.776737+0000", 2024-09-18T08:09:36.194 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 2.0990000000000001e-06 2024-09-18T08:09:36.194 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:09:36.194 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:09:36.194 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "all_read", 2024-09-18T08:09:36.194 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:51.776765+0000", 2024-09-18T08:09:36.194 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 2.7562000000000001e-05 2024-09-18T08:09:36.194 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:09:36.194 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:09:36.194 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "dispatched", 2024-09-18T08:09:36.195 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:51.776785+0000", 2024-09-18T08:09:36.195 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 2.0136000000000002e-05 2024-09-18T08:09:36.195 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:09:36.195 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:09:36.195 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "queued_for_pg", 2024-09-18T08:09:36.195 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:51.776794+0000", 2024-09-18T08:09:36.195 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 9.3109999999999995e-06 2024-09-18T08:09:36.195 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:09:36.195 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:09:36.195 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-09-18T08:09:36.195 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:51.776887+0000", 2024-09-18T08:09:36.195 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 9.3382999999999997e-05 2024-09-18T08:09:36.195 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:09:36.196 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:09:36.196 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "waiting for peered", 2024-09-18T08:09:36.196 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:51.776898+0000", 2024-09-18T08:09:36.196 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.0278999999999999e-05 2024-09-18T08:09:36.196 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:09:36.196 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:09:36.196 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-09-18T08:09:36.196 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:52.697921+0000", 2024-09-18T08:09:36.196 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.92102297700000002 2024-09-18T08:09:36.196 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:09:36.196 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:09:36.196 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "waiting for peered", 2024-09-18T08:09:36.196 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:52.697929+0000", 2024-09-18T08:09:36.196 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 8.0709999999999995e-06 2024-09-18T08:09:36.197 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:09:36.197 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:09:36.197 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-09-18T08:09:36.197 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:53.839602+0000", 2024-09-18T08:09:36.197 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.1416730690000001 2024-09-18T08:09:36.197 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:09:36.197 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:09:36.197 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "started", 2024-09-18T08:09:36.197 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:53.839748+0000", 2024-09-18T08:09:36.197 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.00014631299999999999 2024-09-18T08:09:36.197 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:09:36.197 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:09:36.197 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "waiting for subops from 1,2,4", 2024-09-18T08:09:36.197 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:53.839927+0000", 2024-09-18T08:09:36.198 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.00017868800000000001 2024-09-18T08:09:36.198 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:09:36.198 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:09:36.198 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:09:36.198 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:53.843113+0000", 2024-09-18T08:09:36.198 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.003186595 2024-09-18T08:09:36.198 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:09:36.198 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:09:36.198 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:09:36.198 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:53.848599+0000", 2024-09-18T08:09:36.198 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.0054851049999999997 2024-09-18T08:09:36.198 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:09:36.198 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:09:36.198 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "op_commit", 2024-09-18T08:09:36.199 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:53.848616+0000", 2024-09-18T08:09:36.199 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.785e-05 2024-09-18T08:09:36.199 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:09:36.199 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:09:36.199 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:09:36.199 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:53.849125+0000", 2024-09-18T08:09:36.199 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.000508579 2024-09-18T08:09:36.199 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:09:36.199 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:09:36.199 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "commit_sent", 2024-09-18T08:09:36.199 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:53.849311+0000", 2024-09-18T08:09:36.199 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.00018621299999999999 2024-09-18T08:09:36.199 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:09:36.199 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:09:36.200 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "done", 2024-09-18T08:09:36.200 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:53.849532+0000", 2024-09-18T08:09:36.200 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.00022056699999999999 2024-09-18T08:09:36.200 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-09-18T08:09:36.200 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-09-18T08:09:36.200 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-09-18T08:09:36.200 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:09:36.200 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:09:36.200 INFO:teuthology.orchestra.run.smithi007.stdout: "description": "osd_op(client.15216.0:69340 8.f 8:ffd37843:::benchmark_data_smithi007_101583_object69339:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e198)", 2024-09-18T08:09:36.200 INFO:teuthology.orchestra.run.smithi007.stdout: "initiated_at": "2024-09-18T08:08:51.891119+0000", 2024-09-18T08:09:36.200 INFO:teuthology.orchestra.run.smithi007.stdout: "age": 43.060694845, 2024-09-18T08:09:36.200 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.9666553580000001, 2024-09-18T08:09:36.201 INFO:teuthology.orchestra.run.smithi007.stdout: "continuous": false, 2024-09-18T08:09:36.201 INFO:teuthology.orchestra.run.smithi007.stdout: "type_data": { 2024-09-18T08:09:36.201 INFO:teuthology.orchestra.run.smithi007.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T08:09:36.201 INFO:teuthology.orchestra.run.smithi007.stdout: "client_info": { 2024-09-18T08:09:36.201 INFO:teuthology.orchestra.run.smithi007.stdout: "client": "client.15216", 2024-09-18T08:09:36.201 INFO:teuthology.orchestra.run.smithi007.stdout: "client_addr": "172.21.15.7:0/2596041177", 2024-09-18T08:09:36.201 INFO:teuthology.orchestra.run.smithi007.stdout: "tid": 69340 2024-09-18T08:09:36.201 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:09:36.201 INFO:teuthology.orchestra.run.smithi007.stdout: "events": [ 2024-09-18T08:09:36.201 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:09:36.201 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "initiated", 2024-09-18T08:09:36.201 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:51.891119+0000", 2024-09-18T08:09:36.201 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-09-18T08:09:36.201 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:09:36.202 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:09:36.202 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "header_read", 2024-09-18T08:09:36.202 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:51.891119+0000", 2024-09-18T08:09:36.202 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-09-18T08:09:36.202 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:09:36.202 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:09:36.202 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "throttled", 2024-09-18T08:09:36.202 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:51.891120+0000", 2024-09-18T08:09:36.202 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.6649999999999999e-06 2024-09-18T08:09:36.202 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:09:36.202 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:09:36.202 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "all_read", 2024-09-18T08:09:36.202 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:51.891145+0000", 2024-09-18T08:09:36.203 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 2.4229000000000001e-05 2024-09-18T08:09:36.203 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:09:36.203 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:09:36.203 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "dispatched", 2024-09-18T08:09:36.203 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:51.891156+0000", 2024-09-18T08:09:36.203 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.1479e-05 2024-09-18T08:09:36.203 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:09:36.203 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:09:36.203 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "queued_for_pg", 2024-09-18T08:09:36.203 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:51.891163+0000", 2024-09-18T08:09:36.203 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 6.4989999999999999e-06 2024-09-18T08:09:36.204 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:09:36.204 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:09:36.204 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-09-18T08:09:36.204 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:51.891224+0000", 2024-09-18T08:09:36.204 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 6.1450000000000003e-05 2024-09-18T08:09:36.204 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:09:36.204 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:09:36.204 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "waiting for peered", 2024-09-18T08:09:36.204 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:51.891236+0000", 2024-09-18T08:09:36.204 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.205e-05 2024-09-18T08:09:36.204 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:09:36.204 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:09:36.204 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-09-18T08:09:36.205 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:52.697963+0000", 2024-09-18T08:09:36.205 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.80672717599999999 2024-09-18T08:09:36.205 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:09:36.205 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:09:36.205 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "waiting for peered", 2024-09-18T08:09:36.205 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:52.697971+0000", 2024-09-18T08:09:36.205 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 7.9000000000000006e-06 2024-09-18T08:09:36.205 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:09:36.205 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:09:36.205 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-09-18T08:09:36.205 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:53.843149+0000", 2024-09-18T08:09:36.205 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.1451775449999999 2024-09-18T08:09:36.205 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:09:36.206 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:09:36.206 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "started", 2024-09-18T08:09:36.206 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:53.843325+0000", 2024-09-18T08:09:36.206 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.000176059 2024-09-18T08:09:36.206 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:09:36.206 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:09:36.206 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "waiting for subops from 1,2,4", 2024-09-18T08:09:36.206 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:53.843462+0000", 2024-09-18T08:09:36.206 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.000137446 2024-09-18T08:09:36.206 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:09:36.206 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:09:36.206 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "op_commit", 2024-09-18T08:09:36.206 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:53.848623+0000", 2024-09-18T08:09:36.207 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.0051606589999999997 2024-09-18T08:09:36.207 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:09:36.207 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:09:36.207 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:09:36.207 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:53.849075+0000", 2024-09-18T08:09:36.207 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.00045226899999999999 2024-09-18T08:09:36.207 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:09:36.207 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:09:36.207 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:09:36.207 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:53.849640+0000", 2024-09-18T08:09:36.207 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.00056498900000000001 2024-09-18T08:09:36.207 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:09:36.207 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:09:36.208 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:09:36.208 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:53.857586+0000", 2024-09-18T08:09:36.208 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.0079462319999999993 2024-09-18T08:09:36.208 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:09:36.208 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:09:36.208 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "commit_sent", 2024-09-18T08:09:36.208 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:53.857652+0000", 2024-09-18T08:09:36.208 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 6.5831e-05 2024-09-18T08:09:36.208 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:09:36.208 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:09:36.208 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "done", 2024-09-18T08:09:36.208 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:53.857774+0000", 2024-09-18T08:09:36.208 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.00012188000000000001 2024-09-18T08:09:36.208 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-09-18T08:09:36.209 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-09-18T08:09:36.209 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-09-18T08:09:36.209 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:09:36.209 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:09:36.209 INFO:teuthology.orchestra.run.smithi007.stdout: "description": "osd_op(client.15216.0:69360 8.f 8:fee0f726:::benchmark_data_smithi007_101583_object69359:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e198)", 2024-09-18T08:09:36.209 INFO:teuthology.orchestra.run.smithi007.stdout: "initiated_at": "2024-09-18T08:08:51.959160+0000", 2024-09-18T08:09:36.209 INFO:teuthology.orchestra.run.smithi007.stdout: "age": 42.992653830999998, 2024-09-18T08:09:36.209 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.8990693009999999, 2024-09-18T08:09:36.209 INFO:teuthology.orchestra.run.smithi007.stdout: "continuous": false, 2024-09-18T08:09:36.209 INFO:teuthology.orchestra.run.smithi007.stdout: "type_data": { 2024-09-18T08:09:36.209 INFO:teuthology.orchestra.run.smithi007.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T08:09:36.209 INFO:teuthology.orchestra.run.smithi007.stdout: "client_info": { 2024-09-18T08:09:36.209 INFO:teuthology.orchestra.run.smithi007.stdout: "client": "client.15216", 2024-09-18T08:09:36.210 INFO:teuthology.orchestra.run.smithi007.stdout: "client_addr": "172.21.15.7:0/2596041177", 2024-09-18T08:09:36.210 INFO:teuthology.orchestra.run.smithi007.stdout: "tid": 69360 2024-09-18T08:09:36.210 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:09:36.210 INFO:teuthology.orchestra.run.smithi007.stdout: "events": [ 2024-09-18T08:09:36.210 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:09:36.210 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "initiated", 2024-09-18T08:09:36.210 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:51.959160+0000", 2024-09-18T08:09:36.210 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-09-18T08:09:36.210 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:09:36.210 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:09:36.210 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "header_read", 2024-09-18T08:09:36.210 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:51.959160+0000", 2024-09-18T08:09:36.210 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-09-18T08:09:36.211 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:09:36.211 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:09:36.211 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "throttled", 2024-09-18T08:09:36.211 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:51.959162+0000", 2024-09-18T08:09:36.211 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 2.1069999999999999e-06 2024-09-18T08:09:36.211 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:09:36.211 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:09:36.211 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "all_read", 2024-09-18T08:09:36.211 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:51.959187+0000", 2024-09-18T08:09:36.211 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 2.4902e-05 2024-09-18T08:09:36.211 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:09:36.211 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:09:36.212 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "dispatched", 2024-09-18T08:09:36.212 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:51.959207+0000", 2024-09-18T08:09:36.212 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 2.0051000000000002e-05 2024-09-18T08:09:36.212 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:09:36.212 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:09:36.212 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "queued_for_pg", 2024-09-18T08:09:36.212 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:51.959215+0000", 2024-09-18T08:09:36.212 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 8.7679999999999997e-06 2024-09-18T08:09:36.212 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:09:36.213 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:09:36.213 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-09-18T08:09:36.213 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:51.959299+0000", 2024-09-18T08:09:36.213 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 8.3129999999999999e-05 2024-09-18T08:09:36.213 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:09:36.213 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:09:36.213 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "waiting for peered", 2024-09-18T08:09:36.213 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:51.959313+0000", 2024-09-18T08:09:36.213 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.3967e-05 2024-09-18T08:09:36.213 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:09:36.213 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:09:36.213 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-09-18T08:09:36.213 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:52.698965+0000", 2024-09-18T08:09:36.213 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.73965287599999996 2024-09-18T08:09:36.214 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:09:36.214 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:09:36.214 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "waiting for peered", 2024-09-18T08:09:36.214 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:52.698974+0000", 2024-09-18T08:09:36.214 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 8.1929999999999993e-06 2024-09-18T08:09:36.214 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:09:36.214 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:09:36.214 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-09-18T08:09:36.214 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:53.845783+0000", 2024-09-18T08:09:36.214 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.1468095030000001 2024-09-18T08:09:36.214 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:09:36.214 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:09:36.214 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "started", 2024-09-18T08:09:36.214 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:53.845945+0000", 2024-09-18T08:09:36.215 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.00016167500000000001 2024-09-18T08:09:36.215 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:09:36.215 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:09:36.215 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "waiting for subops from 1,2,4", 2024-09-18T08:09:36.215 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:53.846104+0000", 2024-09-18T08:09:36.215 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.00015877600000000001 2024-09-18T08:09:36.215 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:09:36.215 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:09:36.215 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "op_commit", 2024-09-18T08:09:36.215 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:53.849087+0000", 2024-09-18T08:09:36.215 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.0029835420000000001 2024-09-18T08:09:36.216 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:09:36.216 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:09:36.216 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:09:36.216 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:53.849594+0000", 2024-09-18T08:09:36.216 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.00050686399999999999 2024-09-18T08:09:36.216 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:09:36.216 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:09:36.216 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:09:36.216 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:53.857524+0000", 2024-09-18T08:09:36.216 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.0079296000000000002 2024-09-18T08:09:36.216 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:09:36.217 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:09:36.217 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:09:36.217 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:53.857905+0000", 2024-09-18T08:09:36.217 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.000381107 2024-09-18T08:09:36.217 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:09:36.217 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:09:36.217 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "commit_sent", 2024-09-18T08:09:36.217 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:53.858073+0000", 2024-09-18T08:09:36.217 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.00016861900000000001 2024-09-18T08:09:36.217 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:09:36.217 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:09:36.217 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "done", 2024-09-18T08:09:36.218 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:53.858229+0000", 2024-09-18T08:09:36.218 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.00015562099999999999 2024-09-18T08:09:36.218 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-09-18T08:09:36.218 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-09-18T08:09:36.218 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-09-18T08:09:36.218 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:09:36.218 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:09:36.218 INFO:teuthology.orchestra.run.smithi007.stdout: "description": "osd_op(client.15216.0:100478 8.f 8:fbdbf1d3:::benchmark_data_smithi007_101583_object100477:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e210)", 2024-09-18T08:09:36.218 INFO:teuthology.orchestra.run.smithi007.stdout: "initiated_at": "2024-09-18T08:09:27.339441+0000", 2024-09-18T08:09:36.218 INFO:teuthology.orchestra.run.smithi007.stdout: "age": 7.612372336, 2024-09-18T08:09:36.218 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.0272729549999999, 2024-09-18T08:09:36.218 INFO:teuthology.orchestra.run.smithi007.stdout: "continuous": false, 2024-09-18T08:09:36.219 INFO:teuthology.orchestra.run.smithi007.stdout: "type_data": { 2024-09-18T08:09:36.219 INFO:teuthology.orchestra.run.smithi007.stdout: "flag_point": "reached pg", 2024-09-18T08:09:36.219 INFO:teuthology.orchestra.run.smithi007.stdout: "client_info": { 2024-09-18T08:09:36.219 INFO:teuthology.orchestra.run.smithi007.stdout: "client": "client.15216", 2024-09-18T08:09:36.219 INFO:teuthology.orchestra.run.smithi007.stdout: "client_addr": "172.21.15.7:0/2596041177", 2024-09-18T08:09:36.219 INFO:teuthology.orchestra.run.smithi007.stdout: "tid": 100478 2024-09-18T08:09:36.219 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:09:36.219 INFO:teuthology.orchestra.run.smithi007.stdout: "events": [ 2024-09-18T08:09:36.219 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:09:36.219 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "initiated", 2024-09-18T08:09:36.220 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:09:27.339441+0000", 2024-09-18T08:09:36.220 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-09-18T08:09:36.220 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:09:36.220 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:09:36.220 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "header_read", 2024-09-18T08:09:36.220 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:09:27.339441+0000", 2024-09-18T08:09:36.220 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-09-18T08:09:36.220 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:09:36.220 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:09:36.221 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "throttled", 2024-09-18T08:09:36.221 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:09:27.339442+0000", 2024-09-18T08:09:36.221 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 8.9800000000000002e-07 2024-09-18T08:09:36.221 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:09:36.221 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:09:36.221 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "all_read", 2024-09-18T08:09:36.221 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:09:27.339463+0000", 2024-09-18T08:09:36.221 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 2.0744999999999999e-05 2024-09-18T08:09:36.221 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:09:36.221 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:09:36.221 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "dispatched", 2024-09-18T08:09:36.221 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:09:27.339477+0000", 2024-09-18T08:09:36.222 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.4656e-05 2024-09-18T08:09:36.222 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:09:36.222 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:09:36.222 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "queued_for_pg", 2024-09-18T08:09:36.222 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:09:27.339485+0000", 2024-09-18T08:09:36.222 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 7.2699999999999999e-06 2024-09-18T08:09:36.222 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:09:36.222 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:09:36.223 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-09-18T08:09:36.223 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:09:27.339537+0000", 2024-09-18T08:09:36.223 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 5.2635e-05 2024-09-18T08:09:36.223 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:09:36.223 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:09:36.223 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "started", 2024-09-18T08:09:36.223 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:09:27.339659+0000", 2024-09-18T08:09:36.223 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.000121951 2024-09-18T08:09:36.223 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:09:36.223 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:09:36.223 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "waiting for subops from 1,2,4", 2024-09-18T08:09:36.223 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:09:27.339805+0000", 2024-09-18T08:09:36.224 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.00014521799999999999 2024-09-18T08:09:36.224 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:09:36.224 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:09:36.224 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "op_commit", 2024-09-18T08:09:36.224 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:09:27.343945+0000", 2024-09-18T08:09:36.224 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.0041406120000000001 2024-09-18T08:09:36.224 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:09:36.224 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:09:36.224 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:09:36.224 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:09:27.345369+0000", 2024-09-18T08:09:36.224 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.0014239299999999999 2024-09-18T08:09:36.224 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:09:36.224 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:09:36.224 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:09:36.225 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:09:27.345414+0000", 2024-09-18T08:09:36.225 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 4.4471999999999997e-05 2024-09-18T08:09:36.225 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:09:36.225 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:09:36.225 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-09-18T08:09:36.225 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:09:27.362113+0000", 2024-09-18T08:09:36.225 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.016699421999999998 2024-09-18T08:09:36.225 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:09:36.225 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:09:36.225 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "waiting for peered", 2024-09-18T08:09:36.225 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:09:27.362124+0000", 2024-09-18T08:09:36.225 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.1268e-05 2024-09-18T08:09:36.226 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:09:36.226 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:09:36.226 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-09-18T08:09:36.226 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:09:28.366617+0000", 2024-09-18T08:09:36.226 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.0044926059999999 2024-09-18T08:09:36.226 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:09:36.226 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:09:36.226 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "done", 2024-09-18T08:09:36.226 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:09:28.366714+0000", 2024-09-18T08:09:36.226 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 9.7272e-05 2024-09-18T08:09:36.226 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-09-18T08:09:36.226 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-09-18T08:09:36.226 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-09-18T08:09:36.227 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:09:36.227 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:09:36.227 INFO:teuthology.orchestra.run.smithi007.stdout: "description": "osd_op(client.15216.0:100489 8.f 8:f94e46ee:::benchmark_data_smithi007_101583_object100488:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e211)", 2024-09-18T08:09:36.227 INFO:teuthology.orchestra.run.smithi007.stdout: "initiated_at": "2024-09-18T08:09:27.351893+0000", 2024-09-18T08:09:36.227 INFO:teuthology.orchestra.run.smithi007.stdout: "age": 7.5999200220000001, 2024-09-18T08:09:36.227 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.0213833750000001, 2024-09-18T08:09:36.227 INFO:teuthology.orchestra.run.smithi007.stdout: "continuous": false, 2024-09-18T08:09:36.227 INFO:teuthology.orchestra.run.smithi007.stdout: "type_data": { 2024-09-18T08:09:36.227 INFO:teuthology.orchestra.run.smithi007.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T08:09:36.227 INFO:teuthology.orchestra.run.smithi007.stdout: "client_info": { 2024-09-18T08:09:36.227 INFO:teuthology.orchestra.run.smithi007.stdout: "client": "client.15216", 2024-09-18T08:09:36.228 INFO:teuthology.orchestra.run.smithi007.stdout: "client_addr": "172.21.15.7:0/2596041177", 2024-09-18T08:09:36.228 INFO:teuthology.orchestra.run.smithi007.stdout: "tid": 100489 2024-09-18T08:09:36.228 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:09:36.228 INFO:teuthology.orchestra.run.smithi007.stdout: "events": [ 2024-09-18T08:09:36.228 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:09:36.228 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "initiated", 2024-09-18T08:09:36.228 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:09:27.351893+0000", 2024-09-18T08:09:36.228 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-09-18T08:09:36.228 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:09:36.228 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:09:36.228 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "header_read", 2024-09-18T08:09:36.228 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:09:27.351893+0000", 2024-09-18T08:09:36.229 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-09-18T08:09:36.229 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:09:36.229 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:09:36.229 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "throttled", 2024-09-18T08:09:36.229 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:09:27.351894+0000", 2024-09-18T08:09:36.229 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 5.5300000000000004e-07 2024-09-18T08:09:36.229 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:09:36.229 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:09:36.234 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "all_read", 2024-09-18T08:09:36.234 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:09:27.351919+0000", 2024-09-18T08:09:36.234 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 2.4837999999999998e-05 2024-09-18T08:09:36.235 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:09:36.235 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:09:36.235 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "dispatched", 2024-09-18T08:09:36.235 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:09:27.351926+0000", 2024-09-18T08:09:36.235 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 7.277e-06 2024-09-18T08:09:36.235 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:09:36.236 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:09:36.236 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "queued_for_pg", 2024-09-18T08:09:36.236 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:09:27.351933+0000", 2024-09-18T08:09:36.236 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 6.4810000000000003e-06 2024-09-18T08:09:36.236 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:09:36.236 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:09:36.237 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-09-18T08:09:36.237 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:09:27.352062+0000", 2024-09-18T08:09:36.237 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.00012982900000000001 2024-09-18T08:09:36.237 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:09:36.237 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:09:36.237 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "op must wait for map", 2024-09-18T08:09:36.237 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:09:27.352069+0000", 2024-09-18T08:09:36.237 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 6.8739999999999997e-06 2024-09-18T08:09:36.237 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:09:36.237 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:09:36.237 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-09-18T08:09:36.237 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:09:27.368007+0000", 2024-09-18T08:09:36.238 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.015938009999999999 2024-09-18T08:09:36.238 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:09:36.238 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:09:36.238 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "waiting for peered", 2024-09-18T08:09:36.238 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:09:27.368018+0000", 2024-09-18T08:09:36.238 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.0781000000000001e-05 2024-09-18T08:09:36.238 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:09:36.238 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:09:36.238 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-09-18T08:09:36.238 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:09:28.367234+0000", 2024-09-18T08:09:36.238 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.99921584100000005 2024-09-18T08:09:36.238 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:09:36.238 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:09:36.239 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "started", 2024-09-18T08:09:36.239 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:09:28.367328+0000", 2024-09-18T08:09:36.239 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 9.4083e-05 2024-09-18T08:09:36.239 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:09:36.239 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:09:36.239 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "waiting for subops from 1,2", 2024-09-18T08:09:36.239 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:09:28.367448+0000", 2024-09-18T08:09:36.239 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.00012029800000000001 2024-09-18T08:09:36.239 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:09:36.239 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:09:36.239 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:09:36.239 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:09:28.373055+0000", 2024-09-18T08:09:36.239 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.0056071460000000004 2024-09-18T08:09:36.239 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:09:36.240 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:09:36.240 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:09:36.240 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:09:28.373134+0000", 2024-09-18T08:09:36.240 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 7.8740999999999997e-05 2024-09-18T08:09:36.240 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:09:36.240 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:09:36.240 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "op_commit", 2024-09-18T08:09:36.240 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:09:28.373213+0000", 2024-09-18T08:09:36.240 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 7.894e-05 2024-09-18T08:09:36.240 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:09:36.240 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:09:36.240 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "commit_sent", 2024-09-18T08:09:36.240 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:09:28.373255+0000", 2024-09-18T08:09:36.241 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 4.1718000000000003e-05 2024-09-18T08:09:36.241 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:09:36.241 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:09:36.241 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "done", 2024-09-18T08:09:36.241 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:09:28.373277+0000", 2024-09-18T08:09:36.241 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 2.1965000000000001e-05 2024-09-18T08:09:36.241 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-09-18T08:09:36.241 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-09-18T08:09:36.241 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-09-18T08:09:36.241 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-09-18T08:09:36.241 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-09-18T08:09:36.241 INFO:teuthology.orchestra.run.smithi007.stdout:} 2024-09-18T08:09:36.243 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:09:36.244 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:09:36.244 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6' 2024-09-18T08:09:36.345 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5 2024-09-18T08:09:36.378 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:09:36.379 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:09:36.380 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5' 2024-09-18T08:09:36.481 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5 2024-09-18T08:09:36.505 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:09:36.506 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:09:36.506 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5' 2024-09-18T08:09:36.607 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2 2024-09-18T08:09:36.642 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:09:36.642 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:09:36.643 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2' 2024-09-18T08:09:36.730 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:09:36 smithi007 ceph-mon[29459]: 6.12 scrub starts 2024-09-18T08:09:36.731 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:09:36 smithi007 ceph-mon[29459]: 6.12 scrub ok 2024-09-18T08:09:36.731 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:09:36 smithi007 ceph-mon[29459]: pgmap v332: 158 pgs: 1 active+recovering+undersized+remapped, 157 active+clean; 5.8 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 1.4 KiB/s rd, 34 MiB/s wr, 1.22k op/s; 2591/286947 objects misplaced (0.903%); 8.0 MiB/s, 127 objects/s recovering 2024-09-18T08:09:36.731 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Sep 18 08:09:36 smithi007 ceph-779349ca-7593-11ef-bceb-c7b262605968-mgr-y[29688]: ::ffff:172.21.15.50 - - [18/Sep/2024:08:09:36] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-09-18T08:09:36.731 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:09:36 smithi007 ceph-mon[39249]: 6.12 scrub starts 2024-09-18T08:09:36.731 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:09:36 smithi007 ceph-mon[39249]: 6.12 scrub ok 2024-09-18T08:09:36.731 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:09:36 smithi007 ceph-mon[39249]: pgmap v332: 158 pgs: 1 active+recovering+undersized+remapped, 157 active+clean; 5.8 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 1.4 KiB/s rd, 34 MiB/s wr, 1.22k op/s; 2591/286947 objects misplaced (0.903%); 8.0 MiB/s, 127 objects/s recovering 2024-09-18T08:09:36.744 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4 2024-09-18T08:09:36.776 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:09:36.777 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:09:36.778 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4' 2024-09-18T08:09:36.879 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2 2024-09-18T08:09:36.906 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:09:36 smithi050 ceph-mon[32371]: 6.12 scrub starts 2024-09-18T08:09:36.906 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:09:36 smithi050 ceph-mon[32371]: 6.12 scrub ok 2024-09-18T08:09:36.906 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:09:36 smithi050 ceph-mon[32371]: pgmap v332: 158 pgs: 1 active+recovering+undersized+remapped, 157 active+clean; 5.8 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 1.4 KiB/s rd, 34 MiB/s wr, 1.22k op/s; 2591/286947 objects misplaced (0.903%); 8.0 MiB/s, 127 objects/s recovering 2024-09-18T08:09:36.910 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:09:36.910 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:09:36.911 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2' 2024-09-18T08:09:37.012 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4 2024-09-18T08:09:37.044 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:09:37.045 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:09:37.049 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4' 2024-09-18T08:09:37.150 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6 2024-09-18T08:09:37.189 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:09:37.189 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:09:37.190 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6' 2024-09-18T08:09:37.291 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1 2024-09-18T08:09:37.313 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:09:37.314 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:09:37.314 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1' 2024-09-18T08:09:37.415 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7 2024-09-18T08:09:37.437 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:09:37.438 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:09:37.438 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7' 2024-09-18T08:09:37.539 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1 2024-09-18T08:09:37.561 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:09:37.562 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:09:37.563 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1' 2024-09-18T08:09:37.663 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4 2024-09-18T08:09:37.696 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:09:37.697 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:09:37.697 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4' 2024-09-18T08:09:37.730 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:09:37 smithi007 ceph-mon[29459]: 8.7 scrub ok 2024-09-18T08:09:37.730 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:09:37 smithi007 ceph-mon[39249]: 8.7 scrub ok 2024-09-18T08:09:37.798 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6 2024-09-18T08:09:37.828 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:09:37.829 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:09:37.830 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6' 2024-09-18T08:09:37.906 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:09:37 smithi050 ceph-mon[32371]: 8.7 scrub ok 2024-09-18T08:09:37.930 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4 2024-09-18T08:09:37.964 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:09:37.965 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:09:37.966 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4' 2024-09-18T08:09:38.067 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3 2024-09-18T08:09:38.109 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:09:38.109 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:09:38.110 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3' 2024-09-18T08:09:38.210 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5 2024-09-18T08:09:38.239 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:09:38.240 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:09:38.240 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5' 2024-09-18T08:09:38.341 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3 2024-09-18T08:09:38.381 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:09:38.381 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:09:38.382 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3' 2024-09-18T08:09:38.482 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3 2024-09-18T08:09:38.529 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:09:38.529 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:09:38.530 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3' 2024-09-18T08:09:38.632 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2 2024-09-18T08:09:38.659 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:09:38.660 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:09:38.660 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2' 2024-09-18T08:09:38.730 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:09:38 smithi007 ceph-mon[29459]: 6.1f scrub starts 2024-09-18T08:09:38.731 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:09:38 smithi007 ceph-mon[29459]: 6.1f scrub ok 2024-09-18T08:09:38.731 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:09:38 smithi007 ceph-mon[29459]: pgmap v333: 158 pgs: 1 active+recovering+undersized+remapped, 157 active+clean; 5.8 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 1.2 KiB/s rd, 18 MiB/s wr, 892 op/s; 2591/286947 objects misplaced (0.903%); 5.3 MiB/s, 84 objects/s recovering 2024-09-18T08:09:38.731 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:09:38 smithi007 ceph-mon[29459]: 3.15 scrub starts 2024-09-18T08:09:38.731 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:09:38 smithi007 ceph-mon[29459]: 3.15 scrub ok 2024-09-18T08:09:38.731 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:09:38 smithi007 ceph-mon[29459]: 8.1 scrub starts 2024-09-18T08:09:38.731 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:09:38 smithi007 ceph-mon[39249]: 6.1f scrub starts 2024-09-18T08:09:38.731 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:09:38 smithi007 ceph-mon[39249]: 6.1f scrub ok 2024-09-18T08:09:38.731 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:09:38 smithi007 ceph-mon[39249]: pgmap v333: 158 pgs: 1 active+recovering+undersized+remapped, 157 active+clean; 5.8 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 1.2 KiB/s rd, 18 MiB/s wr, 892 op/s; 2591/286947 objects misplaced (0.903%); 5.3 MiB/s, 84 objects/s recovering 2024-09-18T08:09:38.731 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:09:38 smithi007 ceph-mon[39249]: 3.15 scrub starts 2024-09-18T08:09:38.732 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:09:38 smithi007 ceph-mon[39249]: 3.15 scrub ok 2024-09-18T08:09:38.732 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:09:38 smithi007 ceph-mon[39249]: 8.1 scrub starts 2024-09-18T08:09:38.761 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7 2024-09-18T08:09:38.796 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:09:38.796 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:09:38.796 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7' 2024-09-18T08:09:38.897 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3 2024-09-18T08:09:38.905 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:09:38 smithi050 ceph-mon[32371]: 6.1f scrub starts 2024-09-18T08:09:38.906 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:09:38 smithi050 ceph-mon[32371]: 6.1f scrub ok 2024-09-18T08:09:38.906 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:09:38 smithi050 ceph-mon[32371]: pgmap v333: 158 pgs: 1 active+recovering+undersized+remapped, 157 active+clean; 5.8 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 1.2 KiB/s rd, 18 MiB/s wr, 892 op/s; 2591/286947 objects misplaced (0.903%); 5.3 MiB/s, 84 objects/s recovering 2024-09-18T08:09:38.906 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:09:38 smithi050 ceph-mon[32371]: 3.15 scrub starts 2024-09-18T08:09:38.906 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:09:38 smithi050 ceph-mon[32371]: 3.15 scrub ok 2024-09-18T08:09:38.906 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:09:38 smithi050 ceph-mon[32371]: 8.1 scrub starts 2024-09-18T08:09:38.922 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:09:38.922 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:09:38.923 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3' 2024-09-18T08:09:39.024 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7 2024-09-18T08:09:39.053 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:09:39.054 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:09:39.054 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7' 2024-09-18T08:09:39.155 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0 2024-09-18T08:09:39.186 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:09:39.187 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:09:39.187 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0' 2024-09-18T08:09:39.288 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7 2024-09-18T08:09:39.324 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:09:39.324 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:09:39.325 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7' 2024-09-18T08:09:39.406 INFO:journalctl@ceph.iscsi.iscsi.a.smithi050.stdout:Sep 18 08:09:39 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-iscsi-iscsi-a[57782]: debug there is no tcmu-runner data available 2024-09-18T08:09:39.426 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5 2024-09-18T08:09:39.452 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:09:39.453 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:09:39.453 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5' 2024-09-18T08:09:39.554 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1 2024-09-18T08:09:39.590 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:09:39.591 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:09:39.591 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1' 2024-09-18T08:09:39.692 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5 2024-09-18T08:09:39.721 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:09:39.722 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:09:39.722 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5' 2024-09-18T08:09:39.823 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0 2024-09-18T08:09:39.845 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:09:39.846 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:09:39.846 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0' 2024-09-18T08:09:39.947 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1 2024-09-18T08:09:39.986 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:09:39.987 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:09:39.987 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1' 2024-09-18T08:09:40.088 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7 2024-09-18T08:09:40.121 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:09:40.122 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:09:40.122 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7' 2024-09-18T08:09:40.223 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7 2024-09-18T08:09:40.259 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:09:40.260 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:09:40.260 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7' 2024-09-18T08:09:40.361 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0 2024-09-18T08:09:40.402 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:09:40.404 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:09:40.404 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0' 2024-09-18T08:09:40.505 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3 2024-09-18T08:09:40.544 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:09:40.545 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:09:40.545 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3' 2024-09-18T08:09:40.614 INFO:teuthology.orchestra.run.smithi007.stdout:osd.0: {} 2024-09-18T08:09:40.615 INFO:teuthology.orchestra.run.smithi007.stdout:osd.1: {} 2024-09-18T08:09:40.615 INFO:teuthology.orchestra.run.smithi007.stdout:osd.2: {} 2024-09-18T08:09:40.615 INFO:teuthology.orchestra.run.smithi007.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-18T08:09:40.615 INFO:teuthology.orchestra.run.smithi007.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-18T08:09:40.615 INFO:teuthology.orchestra.run.smithi007.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-18T08:09:40.615 INFO:teuthology.orchestra.run.smithi007.stdout:osd.3: {} 2024-09-18T08:09:40.616 INFO:teuthology.orchestra.run.smithi007.stdout:osd.4: {} 2024-09-18T08:09:40.616 INFO:teuthology.orchestra.run.smithi007.stdout:osd.5: {} 2024-09-18T08:09:40.616 INFO:teuthology.orchestra.run.smithi007.stdout:osd.6: {} 2024-09-18T08:09:40.616 INFO:teuthology.orchestra.run.smithi007.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-18T08:09:40.616 INFO:teuthology.orchestra.run.smithi007.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-18T08:09:40.616 INFO:teuthology.orchestra.run.smithi007.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-18T08:09:40.616 INFO:teuthology.orchestra.run.smithi007.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-18T08:09:40.617 INFO:teuthology.orchestra.run.smithi007.stdout:osd.7: {} 2024-09-18T08:09:40.617 INFO:teuthology.orchestra.run.smithi007.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-18T08:09:40.645 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5 2024-09-18T08:09:40.683 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:09:40.683 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:09:40.683 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5' 2024-09-18T08:09:40.730 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:09:40 smithi007 ceph-mon[29459]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-18T08:09:40.730 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:09:40 smithi007 ceph-mon[29459]: pgmap v334: 158 pgs: 1 active+recovering+undersized+remapped, 1 active+clean+scrubbing, 156 active+clean; 5.5 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 1.1 KiB/s rd, 102 B/s wr, 1.01k op/s; 2211/272622 objects misplaced (0.811%); 4.4 MiB/s, 69 objects/s recovering 2024-09-18T08:09:40.730 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:09:40 smithi007 ceph-mon[29459]: from='client.? 172.21.15.7:0/873874387' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-18T08:09:40.731 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:09:40 smithi007 ceph-mon[29459]: 8.1 scrub ok 2024-09-18T08:09:40.731 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:09:40 smithi007 ceph-mon[39249]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-18T08:09:40.731 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:09:40 smithi007 ceph-mon[39249]: pgmap v334: 158 pgs: 1 active+recovering+undersized+remapped, 1 active+clean+scrubbing, 156 active+clean; 5.5 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 1.1 KiB/s rd, 102 B/s wr, 1.01k op/s; 2211/272622 objects misplaced (0.811%); 4.4 MiB/s, 69 objects/s recovering 2024-09-18T08:09:40.731 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:09:40 smithi007 ceph-mon[39249]: from='client.? 172.21.15.7:0/873874387' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-18T08:09:40.731 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:09:40 smithi007 ceph-mon[39249]: 8.1 scrub ok 2024-09-18T08:09:40.785 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7 2024-09-18T08:09:40.814 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:09:40.815 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:09:40.815 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7' 2024-09-18T08:09:40.906 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:09:40 smithi050 ceph-mon[32371]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-18T08:09:40.906 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:09:40 smithi050 ceph-mon[32371]: pgmap v334: 158 pgs: 1 active+recovering+undersized+remapped, 1 active+clean+scrubbing, 156 active+clean; 5.5 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 1.1 KiB/s rd, 102 B/s wr, 1.01k op/s; 2211/272622 objects misplaced (0.811%); 4.4 MiB/s, 69 objects/s recovering 2024-09-18T08:09:40.906 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:09:40 smithi050 ceph-mon[32371]: from='client.? 172.21.15.7:0/873874387' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-18T08:09:40.906 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:09:40 smithi050 ceph-mon[32371]: 8.1 scrub ok 2024-09-18T08:09:40.916 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1 2024-09-18T08:09:40.956 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:09:40.956 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:09:40.957 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1' 2024-09-18T08:09:41.058 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4 2024-09-18T08:09:41.089 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:09:41.090 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:09:41.091 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4' 2024-09-18T08:09:41.192 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7 2024-09-18T08:09:41.227 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:09:41.228 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:09:41.228 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7' 2024-09-18T08:09:41.329 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3 2024-09-18T08:09:41.368 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:09:41.369 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:09:41.369 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3' 2024-09-18T08:09:41.470 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1 2024-09-18T08:09:41.507 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:09:41.508 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:09:41.508 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1' 2024-09-18T08:09:41.609 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1 2024-09-18T08:09:41.652 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:09:41.653 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:09:41.653 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1' 2024-09-18T08:09:41.730 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:09:41 smithi007 ceph-mon[29459]: 3.e scrub starts 2024-09-18T08:09:41.731 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:09:41 smithi007 ceph-mon[29459]: 3.e scrub ok 2024-09-18T08:09:41.731 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:09:41 smithi007 ceph-mon[39249]: 3.e scrub starts 2024-09-18T08:09:41.731 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:09:41 smithi007 ceph-mon[39249]: 3.e scrub ok 2024-09-18T08:09:41.755 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5 2024-09-18T08:09:41.786 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:09:41.787 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:09:41.787 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5' 2024-09-18T08:09:41.888 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6 2024-09-18T08:09:41.907 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:09:41 smithi050 ceph-mon[32371]: 3.e scrub starts 2024-09-18T08:09:41.907 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:09:41 smithi050 ceph-mon[32371]: 3.e scrub ok 2024-09-18T08:09:41.924 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:09:41.924 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:09:41.924 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6' 2024-09-18T08:09:42.025 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3 2024-09-18T08:09:42.067 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:09:42.068 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:09:42.068 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3' 2024-09-18T08:09:42.169 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7 2024-09-18T08:09:42.199 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:09:42.200 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:09:42.200 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7' 2024-09-18T08:09:42.226 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/779349ca-7593-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T08:09:42.247 DEBUG:teuthology.orchestra.run.smithi050:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid 779349ca-7593-11ef-bceb-c7b262605968 -- ceph daemon osd.4 dump_ops_in_flight 2024-09-18T08:09:42.279 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/779349ca-7593-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T08:09:42.301 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6 2024-09-18T08:09:42.344 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:09:42.345 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:09:42.345 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6' 2024-09-18T08:09:42.446 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5 2024-09-18T08:09:42.489 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:09:42.490 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:09:42.491 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5' 2024-09-18T08:09:42.591 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5 2024-09-18T08:09:42.636 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:09:42.637 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:09:42.637 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5' 2024-09-18T08:09:42.656 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:09:42 smithi050 ceph-mon[32371]: pgmap v335: 158 pgs: 1 active+recovering+undersized+remapped, 1 active+clean+scrubbing, 156 active+clean; 5.5 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 1.1 KiB/s rd, 102 B/s wr, 1.01k op/s; 2211/272622 objects misplaced (0.811%); 4.4 MiB/s, 69 objects/s recovering 2024-09-18T08:09:42.657 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:09:42 smithi050 ceph-mon[32371]: 1.3 scrub starts 2024-09-18T08:09:42.657 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:09:42 smithi050 ceph-mon[32371]: 1.3 scrub ok 2024-09-18T08:09:42.667 INFO:teuthology.orchestra.run.smithi050.stderr:Inferring config /var/lib/ceph/779349ca-7593-11ef-bceb-c7b262605968/mon.b/config 2024-09-18T08:09:42.731 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:09:42 smithi007 ceph-mon[29459]: pgmap v335: 158 pgs: 1 active+recovering+undersized+remapped, 1 active+clean+scrubbing, 156 active+clean; 5.5 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 1.1 KiB/s rd, 102 B/s wr, 1.01k op/s; 2211/272622 objects misplaced (0.811%); 4.4 MiB/s, 69 objects/s recovering 2024-09-18T08:09:42.731 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:09:42 smithi007 ceph-mon[29459]: 1.3 scrub starts 2024-09-18T08:09:42.731 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:09:42 smithi007 ceph-mon[29459]: 1.3 scrub ok 2024-09-18T08:09:42.731 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:09:42 smithi007 ceph-mon[39249]: pgmap v335: 158 pgs: 1 active+recovering+undersized+remapped, 1 active+clean+scrubbing, 156 active+clean; 5.5 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 1.1 KiB/s rd, 102 B/s wr, 1.01k op/s; 2211/272622 objects misplaced (0.811%); 4.4 MiB/s, 69 objects/s recovering 2024-09-18T08:09:42.731 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:09:42 smithi007 ceph-mon[39249]: 1.3 scrub starts 2024-09-18T08:09:42.731 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:09:42 smithi007 ceph-mon[39249]: 1.3 scrub ok 2024-09-18T08:09:42.738 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6 2024-09-18T08:09:42.776 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:09:42.776 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:09:42.777 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6' 2024-09-18T08:09:42.877 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5 2024-09-18T08:09:42.918 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:09:42.919 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:09:42.919 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5' 2024-09-18T08:09:43.020 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1 2024-09-18T08:09:43.059 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:09:43.059 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:09:43.060 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1' 2024-09-18T08:09:43.161 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6 2024-09-18T08:09:43.184 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:09:43.185 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:09:43.185 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6' 2024-09-18T08:09:43.286 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0 2024-09-18T08:09:43.327 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:09:43.327 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:09:43.328 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0' 2024-09-18T08:09:43.429 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0 2024-09-18T08:09:43.469 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:09:43.470 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:09:43.470 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0' 2024-09-18T08:09:43.571 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3 2024-09-18T08:09:43.604 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:09:43.605 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:09:43.605 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3' 2024-09-18T08:09:43.706 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6 2024-09-18T08:09:43.738 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid 779349ca-7593-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-18T08:09:43.745 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:09:43.746 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:09:43.746 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6' 2024-09-18T08:09:43.780 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:09:43 smithi007 ceph-mon[29459]: 6.16 scrub starts 2024-09-18T08:09:43.780 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:09:43 smithi007 ceph-mon[29459]: 6.16 scrub ok 2024-09-18T08:09:43.780 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:09:43 smithi007 ceph-mon[39249]: 6.16 scrub starts 2024-09-18T08:09:43.780 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:09:43 smithi007 ceph-mon[39249]: 6.16 scrub ok 2024-09-18T08:09:43.817 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:09:43 smithi050 ceph-mon[32371]: 6.16 scrub starts 2024-09-18T08:09:43.817 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:09:43 smithi050 ceph-mon[32371]: 6.16 scrub ok 2024-09-18T08:09:43.847 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6 2024-09-18T08:09:43.873 INFO:teuthology.orchestra.run.smithi050.stdout:ERROR: (22) Invalid argument 2024-09-18T08:09:43.873 INFO:teuthology.orchestra.run.smithi050.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-18T08:09:43.876 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:09:43.877 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:09:43.877 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6' 2024-09-18T08:09:43.978 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0 2024-09-18T08:09:44.036 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:09:44.037 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:09:44.037 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0' 2024-09-18T08:09:44.138 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5 2024-09-18T08:09:44.170 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:09:44.171 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:09:44.171 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5' 2024-09-18T08:09:44.245 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/779349ca-7593-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T08:09:44.272 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3 2024-09-18T08:09:44.321 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:09:44.322 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:09:44.322 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3' 2024-09-18T08:09:44.412 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-09-18T08:09:44.419 INFO:teuthology.orchestra.run.smithi007.stderr:dumped all 2024-09-18T08:09:44.423 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6 2024-09-18T08:09:44.455 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:09:44.455 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:09:44.456 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6' 2024-09-18T08:09:44.511 INFO:teuthology.orchestra.run.smithi007.stderr:nodeep-scrub is unset 2024-09-18T08:09:44.556 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5 2024-09-18T08:09:44.589 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:09:44.590 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:09:44.590 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5' 2024-09-18T08:09:44.656 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:09:44 smithi050 ceph-mon[32371]: 6.5 scrub starts 2024-09-18T08:09:44.656 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:09:44 smithi050 ceph-mon[32371]: 6.5 scrub ok 2024-09-18T08:09:44.656 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:09:44 smithi050 ceph-mon[32371]: pgmap v336: 158 pgs: 1 active+recovering+undersized+remapped, 1 active+clean+scrubbing, 156 active+clean; 5.5 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 1.1 KiB/s rd, 102 B/s wr, 1.01k op/s; 2211/270459 objects misplaced (0.817%); 4.4 MiB/s, 69 objects/s recovering 2024-09-18T08:09:44.656 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:09:44 smithi050 ceph-mon[32371]: from='client.? 172.21.15.7:0/2352709099' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-18T08:09:44.656 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:09:44 smithi050 ceph-mon[32371]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-18T08:09:44.691 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0 2024-09-18T08:09:44.707 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:09:44 smithi007 ceph-mon[29459]: 6.5 scrub starts 2024-09-18T08:09:44.708 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:09:44 smithi007 ceph-mon[29459]: 6.5 scrub ok 2024-09-18T08:09:44.708 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:09:44 smithi007 ceph-mon[29459]: pgmap v336: 158 pgs: 1 active+recovering+undersized+remapped, 1 active+clean+scrubbing, 156 active+clean; 5.5 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 1.1 KiB/s rd, 102 B/s wr, 1.01k op/s; 2211/270459 objects misplaced (0.817%); 4.4 MiB/s, 69 objects/s recovering 2024-09-18T08:09:44.708 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:09:44 smithi007 ceph-mon[29459]: from='client.? 172.21.15.7:0/2352709099' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-18T08:09:44.708 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:09:44 smithi007 ceph-mon[29459]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-18T08:09:44.708 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:09:44 smithi007 ceph-mon[39249]: 6.5 scrub starts 2024-09-18T08:09:44.709 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:09:44 smithi007 ceph-mon[39249]: 6.5 scrub ok 2024-09-18T08:09:44.709 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:09:44 smithi007 ceph-mon[39249]: pgmap v336: 158 pgs: 1 active+recovering+undersized+remapped, 1 active+clean+scrubbing, 156 active+clean; 5.5 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 1.1 KiB/s rd, 102 B/s wr, 1.01k op/s; 2211/270459 objects misplaced (0.817%); 4.4 MiB/s, 69 objects/s recovering 2024-09-18T08:09:44.709 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:09:44 smithi007 ceph-mon[39249]: from='client.? 172.21.15.7:0/2352709099' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-18T08:09:44.709 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:09:44 smithi007 ceph-mon[39249]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-18T08:09:44.722 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:09:44.722 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:09:44.722 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0' 2024-09-18T08:09:44.824 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7 2024-09-18T08:09:44.865 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:09:44.866 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:09:44.867 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7' 2024-09-18T08:09:44.912 DEBUG:teuthology.orchestra.run.smithi050:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid 779349ca-7593-11ef-bceb-c7b262605968 -- ceph daemon osd.4 dump_blocked_ops 2024-09-18T08:09:44.967 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0 2024-09-18T08:09:45.004 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:09:45.005 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:09:45.006 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0' 2024-09-18T08:09:45.106 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6 2024-09-18T08:09:45.136 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:09:45.136 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:09:45.137 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6' 2024-09-18T08:09:45.237 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5 2024-09-18T08:09:45.285 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:09:45.286 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:09:45.286 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5' 2024-09-18T08:09:45.332 INFO:teuthology.orchestra.run.smithi050.stderr:Inferring config /var/lib/ceph/779349ca-7593-11ef-bceb-c7b262605968/mon.b/config 2024-09-18T08:09:45.386 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3 2024-09-18T08:09:45.431 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:09:45.431 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:09:45.431 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3' 2024-09-18T08:09:45.532 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0 2024-09-18T08:09:45.580 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:09:45.581 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:09:45.582 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0' 2024-09-18T08:09:45.682 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1 2024-09-18T08:09:45.698 INFO:teuthology.orchestra.run.smithi007.stdout:osd.0: {} 2024-09-18T08:09:45.698 INFO:teuthology.orchestra.run.smithi007.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-18T08:09:45.735 INFO:teuthology.orchestra.run.smithi007.stdout:osd.1: {} 2024-09-18T08:09:45.735 INFO:teuthology.orchestra.run.smithi007.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-18T08:09:45.743 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:09:45.744 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:09:45.744 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1' 2024-09-18T08:09:45.774 INFO:teuthology.orchestra.run.smithi007.stdout:osd.2: {} 2024-09-18T08:09:45.775 INFO:teuthology.orchestra.run.smithi007.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-18T08:09:45.775 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:09:45 smithi007 ceph-mon[29459]: 4.16 scrub starts 2024-09-18T08:09:45.775 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:09:45 smithi007 ceph-mon[29459]: 4.16 scrub ok 2024-09-18T08:09:45.775 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:09:45 smithi007 ceph-mon[29459]: from='client.15369 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-18T08:09:45.775 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:09:45 smithi007 ceph-mon[29459]: purged_snaps scrub starts 2024-09-18T08:09:45.775 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:09:45 smithi007 ceph-mon[29459]: purged_snaps scrub ok 2024-09-18T08:09:45.775 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:09:45 smithi007 ceph-mon[29459]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-18T08:09:45.775 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:09:45 smithi007 ceph-mon[29459]: Cluster is now healthy 2024-09-18T08:09:45.776 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:09:45 smithi007 ceph-mon[29459]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-18T08:09:45.776 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:09:45 smithi007 ceph-mon[29459]: osdmap e214: 8 total, 8 up, 8 in 2024-09-18T08:09:45.776 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:09:45 smithi007 ceph-mon[29459]: from='mgr.24470 172.21.15.7:0/2594390261' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-18T08:09:45.776 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:09:45 smithi007 ceph-mon[39249]: 4.16 scrub starts 2024-09-18T08:09:45.776 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:09:45 smithi007 ceph-mon[39249]: 4.16 scrub ok 2024-09-18T08:09:45.776 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:09:45 smithi007 ceph-mon[39249]: from='client.15369 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-18T08:09:45.776 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:09:45 smithi007 ceph-mon[39249]: purged_snaps scrub starts 2024-09-18T08:09:45.776 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:09:45 smithi007 ceph-mon[39249]: purged_snaps scrub ok 2024-09-18T08:09:45.777 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:09:45 smithi007 ceph-mon[39249]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-18T08:09:45.777 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:09:45 smithi007 ceph-mon[39249]: Cluster is now healthy 2024-09-18T08:09:45.777 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:09:45 smithi007 ceph-mon[39249]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-18T08:09:45.777 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:09:45 smithi007 ceph-mon[39249]: osdmap e214: 8 total, 8 up, 8 in 2024-09-18T08:09:45.777 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:09:45 smithi007 ceph-mon[39249]: from='mgr.24470 172.21.15.7:0/2594390261' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-18T08:09:45.779 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:09:45 smithi050 ceph-mon[32371]: 4.16 scrub starts 2024-09-18T08:09:45.779 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:09:45 smithi050 ceph-mon[32371]: 4.16 scrub ok 2024-09-18T08:09:45.780 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:09:45 smithi050 ceph-mon[32371]: from='client.15369 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-18T08:09:45.780 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:09:45 smithi050 ceph-mon[32371]: purged_snaps scrub starts 2024-09-18T08:09:45.780 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:09:45 smithi050 ceph-mon[32371]: purged_snaps scrub ok 2024-09-18T08:09:45.780 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:09:45 smithi050 ceph-mon[32371]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-18T08:09:45.780 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:09:45 smithi050 ceph-mon[32371]: Cluster is now healthy 2024-09-18T08:09:45.780 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:09:45 smithi050 ceph-mon[32371]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-18T08:09:45.780 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:09:45 smithi050 ceph-mon[32371]: osdmap e214: 8 total, 8 up, 8 in 2024-09-18T08:09:45.780 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:09:45 smithi050 ceph-mon[32371]: from='mgr.24470 172.21.15.7:0/2594390261' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-18T08:09:45.810 INFO:teuthology.orchestra.run.smithi007.stdout:osd.3: {} 2024-09-18T08:09:45.810 INFO:teuthology.orchestra.run.smithi007.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-18T08:09:45.845 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0 2024-09-18T08:09:45.847 INFO:teuthology.orchestra.run.smithi007.stdout:osd.4: {} 2024-09-18T08:09:45.847 INFO:teuthology.orchestra.run.smithi007.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-18T08:09:45.883 INFO:teuthology.orchestra.run.smithi007.stdout:osd.5: {} 2024-09-18T08:09:45.884 INFO:teuthology.orchestra.run.smithi007.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-18T08:09:45.893 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:09:45.894 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:09:45.894 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0' 2024-09-18T08:09:45.922 INFO:teuthology.orchestra.run.smithi007.stdout:osd.6: {} 2024-09-18T08:09:45.923 INFO:teuthology.orchestra.run.smithi007.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-18T08:09:45.931 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2024-09-18T08:09:45.931 INFO:teuthology.orchestra.run.smithi050.stdout: "ops": [], 2024-09-18T08:09:45.931 INFO:teuthology.orchestra.run.smithi050.stdout: "complaint_time": 30, 2024-09-18T08:09:45.931 INFO:teuthology.orchestra.run.smithi050.stdout: "num_blocked_ops": 0 2024-09-18T08:09:45.931 INFO:teuthology.orchestra.run.smithi050.stdout:} 2024-09-18T08:09:45.961 INFO:teuthology.orchestra.run.smithi007.stdout:osd.7: {} 2024-09-18T08:09:45.961 INFO:teuthology.orchestra.run.smithi007.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-18T08:09:45.994 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1 2024-09-18T08:09:46.041 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:09:46.041 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:09:46.042 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1' 2024-09-18T08:09:46.142 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6 2024-09-18T08:09:46.173 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:09:46.174 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:09:46.174 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6' 2024-09-18T08:09:46.257 INFO:teuthology.orchestra.run.smithi007.stdout:{"pg_ready":true,"pg_map":{"version":336,"stamp":"2024-09-18T08:09:43.217358+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":5895756339,"num_objects":90153,"num_object_clones":0,"num_object_copies":270459,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":2211,"num_objects_unfound":0,"num_objects_dirty":90153,"num_whiteouts":0,"num_read":888,"num_read_kb":640,"num_write":111982,"num_write_kb":6447982,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":22275,"num_bytes_recovered":1443962066,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":79740,"ondisk_log_size":79740,"up":474,"acting":473,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":481,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":20069640,"kb_used_data":18228808,"kb_used_omap":12,"kb_used_meta":1840307,"kb_avail":729891576,"statfs":{"total":767960285184,"available":747408973824,"internally_reserved":0,"allocated":18666299392,"data_stored":18663207945,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12750,"internal_metadata":1884474930},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,2,0,1,3,0,3,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-665190376,"num_objects":-10149,"num_object_clones":0,"num_object_copies":-30447,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-697,"num_objects_unfound":0,"num_objects_dirty":-10149,"num_whiteouts":0,"num_read":11,"num_read_kb":11,"num_write":10151,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":697,"num_bytes_recovered":45678592,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.003433"},"pg_stats":[{"pgid":"3.1f","version":"0'0","reported_seq":346,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-09-18T08:09:34.404734+0000","last_change":"2024-09-18T08:08:52.697893+0000","last_active":"2024-09-18T08:09:34.404734+0000","last_peered":"2024-09-18T08:09:34.404734+0000","last_clean":"2024-09-18T08:09:34.404734+0000","last_became_active":"2024-09-18T08:08:52.697499+0000","last_became_peered":"2024-09-18T08:08:52.697499+0000","last_unstale":"2024-09-18T08:09:34.404734+0000","last_undegraded":"2024-09-18T08:09:34.404734+0000","last_fullsized":"2024-09-18T08:09:34.404734+0000","mapping_epoch":198,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":199,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:08:06.642662+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:46.860092+0000","last_clean_scrub_stamp":"2024-09-18T08:08:06.642662+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"69'9","reported_seq":84,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-09-18T08:09:34.393247+0000","last_change":"2024-09-18T08:08:18.575611+0000","last_active":"2024-09-18T08:09:34.393247+0000","last_peered":"2024-09-18T08:09:34.393247+0000","last_clean":"2024-09-18T08:09:34.393247+0000","last_became_active":"2024-09-18T08:08:18.537856+0000","last_became_peered":"2024-09-18T08:08:18.537856+0000","last_unstale":"2024-09-18T08:09:34.393247+0000","last_undegraded":"2024-09-18T08:09:34.393247+0000","last_fullsized":"2024-09-18T08:09:34.393247+0000","mapping_epoch":181,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":182,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'9","last_scrub_stamp":"2024-09-18T08:05:08.696796+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_clean_scrub_stamp":"2024-09-18T08:05:08.696796+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":16,"num_read_kb":10,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":331,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-09-18T08:09:34.410820+0000","last_change":"2024-09-18T08:07:18.737364+0000","last_active":"2024-09-18T08:09:34.410820+0000","last_peered":"2024-09-18T08:09:34.410820+0000","last_clean":"2024-09-18T08:09:34.410820+0000","last_became_active":"2024-09-18T08:02:51.907140+0000","last_became_peered":"2024-09-18T08:02:51.907140+0000","last_unstale":"2024-09-18T08:09:34.410820+0000","last_undegraded":"2024-09-18T08:09:34.410820+0000","last_fullsized":"2024-09-18T08:09:34.410820+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:07:18.737122+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:50.886830+0000","last_clean_scrub_stamp":"2024-09-18T08:07:18.737122+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:10:14.742960+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":74,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-09-18T08:09:34.393980+0000","last_change":"2024-09-18T08:08:18.545192+0000","last_active":"2024-09-18T08:09:34.393980+0000","last_peered":"2024-09-18T08:09:34.393980+0000","last_clean":"2024-09-18T08:09:34.393980+0000","last_became_active":"2024-09-18T08:08:18.540705+0000","last_became_peered":"2024-09-18T08:08:18.540705+0000","last_unstale":"2024-09-18T08:09:34.393980+0000","last_undegraded":"2024-09-18T08:09:34.393980+0000","last_fullsized":"2024-09-18T08:09:34.393980+0000","mapping_epoch":181,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":182,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:07:08.451865+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_clean_scrub_stamp":"2024-09-18T08:07:08.451865+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":334,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-09-18T08:09:34.398027+0000","last_change":"2024-09-18T08:07:13.244647+0000","last_active":"2024-09-18T08:09:34.398027+0000","last_peered":"2024-09-18T08:09:34.398027+0000","last_clean":"2024-09-18T08:09:34.398027+0000","last_became_active":"2024-09-18T08:06:24.128511+0000","last_became_peered":"2024-09-18T08:06:24.128511+0000","last_unstale":"2024-09-18T08:09:34.398027+0000","last_undegraded":"2024-09-18T08:09:34.398027+0000","last_fullsized":"2024-09-18T08:09:34.398027+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:07:13.244555+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_clean_scrub_stamp":"2024-09-18T08:07:13.244555+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":347,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-09-18T08:09:34.394389+0000","last_change":"2024-09-18T08:07:10.257236+0000","last_active":"2024-09-18T08:09:34.394389+0000","last_peered":"2024-09-18T08:09:34.394389+0000","last_clean":"2024-09-18T08:09:34.394389+0000","last_became_active":"2024-09-18T08:06:24.126194+0000","last_became_peered":"2024-09-18T08:06:24.126194+0000","last_unstale":"2024-09-18T08:09:34.394389+0000","last_undegraded":"2024-09-18T08:09:34.394389+0000","last_fullsized":"2024-09-18T08:09:34.394389+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:07:10.257170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:46.860092+0000","last_clean_scrub_stamp":"2024-09-18T08:07:10.257170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":37,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"69'15","reported_seq":345,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-09-18T08:09:34.396329+0000","last_change":"2024-09-18T08:09:32.575836+0000","last_active":"2024-09-18T08:09:34.396329+0000","last_peered":"2024-09-18T08:09:34.396329+0000","last_clean":"2024-09-18T08:09:34.396329+0000","last_became_active":"2024-09-18T08:02:49.886106+0000","last_became_peered":"2024-09-18T08:02:49.886106+0000","last_unstale":"2024-09-18T08:09:34.396329+0000","last_undegraded":"2024-09-18T08:09:34.396329+0000","last_fullsized":"2024-09-18T08:09:34.396329+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'15","last_scrub_stamp":"2024-09-18T08:09:32.575741+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_clean_scrub_stamp":"2024-09-18T08:09:32.575741+0000","objects_scrubbed":7,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:10:59.141685+0000","scrub_duration":40,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":74,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-09-18T08:09:34.393249+0000","last_change":"2024-09-18T08:08:18.543316+0000","last_active":"2024-09-18T08:09:34.393249+0000","last_peered":"2024-09-18T08:09:34.393249+0000","last_clean":"2024-09-18T08:09:34.393249+0000","last_became_active":"2024-09-18T08:08:18.538645+0000","last_became_peered":"2024-09-18T08:08:18.538645+0000","last_unstale":"2024-09-18T08:09:34.393249+0000","last_undegraded":"2024-09-18T08:09:34.393249+0000","last_fullsized":"2024-09-18T08:09:34.393249+0000","mapping_epoch":181,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":182,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:08:09.142188+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:50.886830+0000","last_clean_scrub_stamp":"2024-09-18T08:08:09.142188+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":359,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-09-18T08:09:34.396908+0000","last_change":"2024-09-18T08:08:18.548588+0000","last_active":"2024-09-18T08:09:34.396908+0000","last_peered":"2024-09-18T08:09:34.396908+0000","last_clean":"2024-09-18T08:09:34.396908+0000","last_became_active":"2024-09-18T08:08:18.548203+0000","last_became_peered":"2024-09-18T08:08:18.548203+0000","last_unstale":"2024-09-18T08:09:34.396908+0000","last_undegraded":"2024-09-18T08:09:34.396908+0000","last_fullsized":"2024-09-18T08:09:34.396908+0000","mapping_epoch":181,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":182,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:06:14.027077+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:46.860092+0000","last_clean_scrub_stamp":"2024-09-18T08:06:14.027077+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"69'9","reported_seq":84,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-09-18T08:09:34.394165+0000","last_change":"2024-09-18T08:08:18.631597+0000","last_active":"2024-09-18T08:09:34.394165+0000","last_peered":"2024-09-18T08:09:34.394165+0000","last_clean":"2024-09-18T08:09:34.394165+0000","last_became_active":"2024-09-18T08:08:18.550960+0000","last_became_peered":"2024-09-18T08:08:18.550960+0000","last_unstale":"2024-09-18T08:09:34.394165+0000","last_undegraded":"2024-09-18T08:09:34.394165+0000","last_fullsized":"2024-09-18T08:09:34.394165+0000","mapping_epoch":181,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":182,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_clean_scrub_stamp":"2024-09-18T08:02:48.870718+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":16,"num_read_kb":10,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":323,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-09-18T08:09:34.396118+0000","last_change":"2024-09-18T08:05:33.935844+0000","last_active":"2024-09-18T08:09:34.396118+0000","last_peered":"2024-09-18T08:09:34.396118+0000","last_clean":"2024-09-18T08:09:34.396118+0000","last_became_active":"2024-09-18T08:02:51.907296+0000","last_became_peered":"2024-09-18T08:02:51.907296+0000","last_unstale":"2024-09-18T08:09:34.396118+0000","last_undegraded":"2024-09-18T08:09:34.396118+0000","last_fullsized":"2024-09-18T08:09:34.396118+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:05:33.935798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:50.886830+0000","last_clean_scrub_stamp":"2024-09-18T08:05:33.935798+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":320,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-09-18T08:09:34.403934+0000","last_change":"2024-09-18T08:08:14.646073+0000","last_active":"2024-09-18T08:09:34.403934+0000","last_peered":"2024-09-18T08:09:34.403934+0000","last_clean":"2024-09-18T08:09:34.403934+0000","last_became_active":"2024-09-18T08:02:53.913519+0000","last_became_peered":"2024-09-18T08:02:53.913519+0000","last_unstale":"2024-09-18T08:09:34.403934+0000","last_undegraded":"2024-09-18T08:09:34.403934+0000","last_fullsized":"2024-09-18T08:09:34.403934+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:08:14.645993+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_clean_scrub_stamp":"2024-09-18T08:08:14.645993+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":337,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-09-18T08:09:34.397037+0000","last_change":"2024-09-18T08:08:18.624260+0000","last_active":"2024-09-18T08:09:34.397037+0000","last_peered":"2024-09-18T08:09:34.397037+0000","last_clean":"2024-09-18T08:09:34.397037+0000","last_became_active":"2024-09-18T08:08:18.623757+0000","last_became_peered":"2024-09-18T08:08:18.623757+0000","last_unstale":"2024-09-18T08:09:34.397037+0000","last_undegraded":"2024-09-18T08:09:34.397037+0000","last_fullsized":"2024-09-18T08:09:34.397037+0000","mapping_epoch":181,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":182,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:04:32.849119+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:46.860092+0000","last_clean_scrub_stamp":"2024-09-18T08:04:32.849119+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1b","version":"69'5","reported_seq":80,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-09-18T08:09:34.393925+0000","last_change":"2024-09-18T08:08:18.651427+0000","last_active":"2024-09-18T08:09:34.393925+0000","last_peered":"2024-09-18T08:09:34.393925+0000","last_clean":"2024-09-18T08:09:34.393925+0000","last_became_active":"2024-09-18T08:08:18.567126+0000","last_became_peered":"2024-09-18T08:08:18.567126+0000","last_unstale":"2024-09-18T08:09:34.393925+0000","last_undegraded":"2024-09-18T08:09:34.393925+0000","last_fullsized":"2024-09-18T08:09:34.393925+0000","mapping_epoch":181,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":182,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'5","last_scrub_stamp":"2024-09-18T08:06:09.465979+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_clean_scrub_stamp":"2024-09-18T08:06:09.465979+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":10,"num_read_kb":6,"num_write":6,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":353,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-09-18T08:09:34.408169+0000","last_change":"2024-09-18T08:09:12.674889+0000","last_active":"2024-09-18T08:09:34.408169+0000","last_peered":"2024-09-18T08:09:34.408169+0000","last_clean":"2024-09-18T08:09:34.408169+0000","last_became_active":"2024-09-18T08:08:18.524932+0000","last_became_peered":"2024-09-18T08:08:18.524932+0000","last_unstale":"2024-09-18T08:09:34.408169+0000","last_undegraded":"2024-09-18T08:09:34.408169+0000","last_fullsized":"2024-09-18T08:09:34.408169+0000","mapping_epoch":181,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":182,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:09:12.674843+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:50.886830+0000","last_clean_scrub_stamp":"2024-09-18T08:09:12.674843+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:10:16.027634+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":320,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-09-18T08:09:34.396440+0000","last_change":"2024-09-18T08:06:44.518882+0000","last_active":"2024-09-18T08:09:34.396440+0000","last_peered":"2024-09-18T08:09:34.396440+0000","last_clean":"2024-09-18T08:09:34.396440+0000","last_became_active":"2024-09-18T08:02:53.911648+0000","last_became_peered":"2024-09-18T08:02:53.911648+0000","last_unstale":"2024-09-18T08:09:34.396440+0000","last_undegraded":"2024-09-18T08:09:34.396440+0000","last_fullsized":"2024-09-18T08:09:34.396440+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:06:44.518835+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_clean_scrub_stamp":"2024-09-18T08:06:44.518835+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":74,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-09-18T08:09:34.395143+0000","last_change":"2024-09-18T08:08:18.544203+0000","last_active":"2024-09-18T08:09:34.395143+0000","last_peered":"2024-09-18T08:09:34.395143+0000","last_clean":"2024-09-18T08:09:34.395143+0000","last_became_active":"2024-09-18T08:08:18.539604+0000","last_became_peered":"2024-09-18T08:08:18.539604+0000","last_unstale":"2024-09-18T08:09:34.395143+0000","last_undegraded":"2024-09-18T08:09:34.395143+0000","last_fullsized":"2024-09-18T08:09:34.395143+0000","mapping_epoch":181,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":182,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:05:40.477965+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_clean_scrub_stamp":"2024-09-18T08:05:40.477965+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":345,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-09-18T08:09:34.405428+0000","last_change":"2024-09-18T08:08:18.524124+0000","last_active":"2024-09-18T08:09:34.405428+0000","last_peered":"2024-09-18T08:09:34.405428+0000","last_clean":"2024-09-18T08:09:34.405428+0000","last_became_active":"2024-09-18T08:08:18.523611+0000","last_became_peered":"2024-09-18T08:08:18.523611+0000","last_unstale":"2024-09-18T08:09:34.405428+0000","last_undegraded":"2024-09-18T08:09:34.405428+0000","last_fullsized":"2024-09-18T08:09:34.405428+0000","mapping_epoch":181,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":182,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:05:27.476144+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:46.860092+0000","last_clean_scrub_stamp":"2024-09-18T08:05:27.476144+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:10:23.059202+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1c","version":"69'15","reported_seq":56,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-09-18T08:09:34.400914+0000","last_change":"2024-09-18T08:08:52.708963+0000","last_active":"2024-09-18T08:09:34.400914+0000","last_peered":"2024-09-18T08:09:34.400914+0000","last_clean":"2024-09-18T08:09:34.400914+0000","last_became_active":"2024-09-18T08:08:52.690788+0000","last_became_peered":"2024-09-18T08:08:52.690788+0000","last_unstale":"2024-09-18T08:09:34.400914+0000","last_undegraded":"2024-09-18T08:09:34.400914+0000","last_fullsized":"2024-09-18T08:09:34.400914+0000","mapping_epoch":198,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":199,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'15","last_scrub_stamp":"2024-09-18T08:07:39.530340+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_clean_scrub_stamp":"2024-09-18T08:07:39.530340+0000","objects_scrubbed":7,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":353,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-09-18T08:09:34.410223+0000","last_change":"2024-09-18T08:09:11.693711+0000","last_active":"2024-09-18T08:09:34.410223+0000","last_peered":"2024-09-18T08:09:34.410223+0000","last_clean":"2024-09-18T08:09:34.410223+0000","last_became_active":"2024-09-18T08:08:18.538505+0000","last_became_peered":"2024-09-18T08:08:18.538505+0000","last_unstale":"2024-09-18T08:09:34.410223+0000","last_undegraded":"2024-09-18T08:09:34.410223+0000","last_fullsized":"2024-09-18T08:09:34.410223+0000","mapping_epoch":181,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":182,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:09:11.693643+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:50.886830+0000","last_clean_scrub_stamp":"2024-09-18T08:09:11.693643+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:10:34.429057+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":335,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-09-18T08:09:36.592348+0000","last_change":"2024-09-18T08:09:36.592348+0000","last_active":"2024-09-18T08:09:36.592348+0000","last_peered":"2024-09-18T08:09:36.592348+0000","last_clean":"2024-09-18T08:09:36.592348+0000","last_became_active":"2024-09-18T08:06:24.128659+0000","last_became_peered":"2024-09-18T08:06:24.128659+0000","last_unstale":"2024-09-18T08:09:36.592348+0000","last_undegraded":"2024-09-18T08:09:36.592348+0000","last_fullsized":"2024-09-18T08:09:36.592348+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:09:36.592281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_clean_scrub_stamp":"2024-09-18T08:09:36.592281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:10:46.573780+0000","scrub_duration":16,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":74,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-09-18T08:09:34.393832+0000","last_change":"2024-09-18T08:08:18.536559+0000","last_active":"2024-09-18T08:09:34.393832+0000","last_peered":"2024-09-18T08:09:34.393832+0000","last_clean":"2024-09-18T08:09:34.393832+0000","last_became_active":"2024-09-18T08:08:18.536084+0000","last_became_peered":"2024-09-18T08:08:18.536084+0000","last_unstale":"2024-09-18T08:09:34.393832+0000","last_undegraded":"2024-09-18T08:09:34.393832+0000","last_fullsized":"2024-09-18T08:09:34.393832+0000","mapping_epoch":181,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":182,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:07:13.543722+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:46.860092+0000","last_clean_scrub_stamp":"2024-09-18T08:07:13.543722+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"69'12","reported_seq":389,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-09-18T08:09:34.393666+0000","last_change":"2024-09-18T08:08:18.583915+0000","last_active":"2024-09-18T08:09:34.393666+0000","last_peered":"2024-09-18T08:09:34.393666+0000","last_clean":"2024-09-18T08:09:34.393666+0000","last_became_active":"2024-09-18T08:08:18.543675+0000","last_became_peered":"2024-09-18T08:08:18.543675+0000","last_unstale":"2024-09-18T08:09:34.393666+0000","last_undegraded":"2024-09-18T08:09:34.393666+0000","last_fullsized":"2024-09-18T08:09:34.393666+0000","mapping_epoch":181,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":182,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'12","last_scrub_stamp":"2024-09-18T08:07:20.251529+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_clean_scrub_stamp":"2024-09-18T08:07:20.251529+0000","objects_scrubbed":5,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":25,"num_read_kb":16,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":74,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-09-18T08:09:34.394829+0000","last_change":"2024-09-18T08:08:18.532482+0000","last_active":"2024-09-18T08:09:34.394829+0000","last_peered":"2024-09-18T08:09:34.394829+0000","last_clean":"2024-09-18T08:09:34.394829+0000","last_became_active":"2024-09-18T08:08:18.531438+0000","last_became_peered":"2024-09-18T08:08:18.531438+0000","last_unstale":"2024-09-18T08:09:34.394829+0000","last_undegraded":"2024-09-18T08:09:34.394829+0000","last_fullsized":"2024-09-18T08:09:34.394829+0000","mapping_epoch":181,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":182,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:06:12.434179+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:50.886830+0000","last_clean_scrub_stamp":"2024-09-18T08:06:12.434179+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1c","version":"69'1","reported_seq":163,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-09-18T08:09:34.396676+0000","last_change":"2024-09-18T08:06:46.142143+0000","last_active":"2024-09-18T08:09:34.396676+0000","last_peered":"2024-09-18T08:09:34.396676+0000","last_clean":"2024-09-18T08:09:34.396676+0000","last_became_active":"2024-09-18T08:06:46.141661+0000","last_became_peered":"2024-09-18T08:06:46.141661+0000","last_unstale":"2024-09-18T08:09:34.396676+0000","last_undegraded":"2024-09-18T08:09:34.396676+0000","last_fullsized":"2024-09-18T08:09:34.396676+0000","mapping_epoch":143,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":144,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'1","last_scrub_stamp":"2024-09-18T08:05:36.776369+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_clean_scrub_stamp":"2024-09-18T08:05:36.776369+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":403,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,2],"acting":[5,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"62'1","reported_seq":362,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-09-18T08:09:34.410915+0000","last_change":"2024-09-18T08:09:29.633875+0000","last_active":"2024-09-18T08:09:34.410915+0000","last_peered":"2024-09-18T08:09:34.410915+0000","last_clean":"2024-09-18T08:09:34.410915+0000","last_became_active":"2024-09-18T08:08:18.559125+0000","last_became_peered":"2024-09-18T08:08:18.559125+0000","last_unstale":"2024-09-18T08:09:34.410915+0000","last_undegraded":"2024-09-18T08:09:34.410915+0000","last_fullsized":"2024-09-18T08:09:34.410915+0000","mapping_epoch":181,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":182,"parent":"0.0","parent_split_bits":0,"last_scrub":"62'1","last_scrub_stamp":"2024-09-18T08:09:29.633520+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:46.860092+0000","last_clean_scrub_stamp":"2024-09-18T08:09:29.633520+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:10:38.890672+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":92,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1e","version":"69'10","reported_seq":327,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-09-18T08:09:34.405835+0000","last_change":"2024-09-18T08:02:49.892755+0000","last_active":"2024-09-18T08:09:34.405835+0000","last_peered":"2024-09-18T08:09:34.405835+0000","last_clean":"2024-09-18T08:09:34.405835+0000","last_became_active":"2024-09-18T08:02:49.892024+0000","last_became_peered":"2024-09-18T08:02:49.892024+0000","last_unstale":"2024-09-18T08:09:34.405835+0000","last_undegraded":"2024-09-18T08:09:34.405835+0000","last_fullsized":"2024-09-18T08:09:34.405835+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_clean_scrub_stamp":"2024-09-18T08:02:48.870718+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:10:17.106246+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1f","version":"182'13","reported_seq":96,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-09-18T08:09:38.009541+0000","last_change":"2024-09-18T08:08:18.627670+0000","last_active":"2024-09-18T08:09:38.009541+0000","last_peered":"2024-09-18T08:09:38.009541+0000","last_clean":"2024-09-18T08:09:38.009541+0000","last_became_active":"2024-09-18T08:08:18.543095+0000","last_became_peered":"2024-09-18T08:08:18.543095+0000","last_unstale":"2024-09-18T08:09:38.009541+0000","last_undegraded":"2024-09-18T08:09:38.009541+0000","last_fullsized":"2024-09-18T08:09:38.009541+0000","mapping_epoch":181,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":182,"parent":"0.0","parent_split_bits":0,"last_scrub":"173'12","last_scrub_stamp":"2024-09-18T08:08:07.553075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:50.886830+0000","last_clean_scrub_stamp":"2024-09-18T08:08:07.553075+0000","objects_scrubbed":0,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,7],"acting":[4,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.3","version":"213'7098","reported_seq":7277,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-09-18T08:09:38.516261+0000","last_change":"2024-09-18T08:09:28.263522+0000","last_active":"2024-09-18T08:09:38.516261+0000","last_peered":"2024-09-18T08:09:38.516261+0000","last_clean":"2024-09-18T08:09:38.516261+0000","last_became_active":"2024-09-18T08:07:53.672797+0000","last_became_peered":"2024-09-18T08:07:53.672797+0000","last_unstale":"2024-09-18T08:09:38.516261+0000","last_undegraded":"2024-09-18T08:09:38.516261+0000","last_fullsized":"2024-09-18T08:09:38.516261+0000","mapping_epoch":165,"log_start":"183'2300","ondisk_log_start":"183'2300","created":165,"last_epoch_clean":166,"parent":"0.0","parent_split_bits":0,"last_scrub":"211'6448","last_scrub_stamp":"2024-09-18T08:09:28.263434+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:07:52.307205+0000","last_clean_scrub_stamp":"2024-09-18T08:09:28.263434+0000","objects_scrubbed":6448,"log_size":4798,"log_dups_size":0,"ondisk_log_size":4798,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:10:29.843317+0000","scrub_duration":646,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":379977728,"num_objects":5798,"num_object_clones":0,"num_object_copies":17394,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5798,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7098,"num_write_kb":412672,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6],"acting":[1,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.a","version":"74'79","reported_seq":145,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-09-18T08:09:34.395931+0000","last_change":"2024-09-18T08:08:57.463205+0000","last_active":"2024-09-18T08:09:34.395931+0000","last_peered":"2024-09-18T08:09:34.395931+0000","last_clean":"2024-09-18T08:09:34.395931+0000","last_became_active":"2024-09-18T08:07:26.394584+0000","last_became_peered":"2024-09-18T08:07:26.394584+0000","last_unstale":"2024-09-18T08:09:34.395931+0000","last_undegraded":"2024-09-18T08:09:34.395931+0000","last_fullsized":"2024-09-18T08:09:34.395931+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":152,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":4,"last_scrub":"74'79","last_scrub_stamp":"2024-09-18T08:08:57.463113+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:00:11.893742+0000","last_clean_scrub_stamp":"2024-09-18T08:08:57.463113+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:09:57.971184+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,6],"acting":[5,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.f","version":"69'15","reported_seq":399,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-09-18T08:09:34.415322+0000","last_change":"2024-09-18T08:08:56.669262+0000","last_active":"2024-09-18T08:09:34.415322+0000","last_peered":"2024-09-18T08:09:34.415322+0000","last_clean":"2024-09-18T08:09:34.415322+0000","last_became_active":"2024-09-18T08:08:18.540211+0000","last_became_peered":"2024-09-18T08:08:18.540211+0000","last_unstale":"2024-09-18T08:09:34.415322+0000","last_undegraded":"2024-09-18T08:09:34.415322+0000","last_fullsized":"2024-09-18T08:09:34.415322+0000","mapping_epoch":181,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":182,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'15","last_scrub_stamp":"2024-09-18T08:08:56.668397+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_clean_scrub_stamp":"2024-09-18T08:08:56.668397+0000","objects_scrubbed":7,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:10:22.470704+0000","scrub_duration":36,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":333,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-09-18T08:09:34.395370+0000","last_change":"2024-09-18T08:08:18.389332+0000","last_active":"2024-09-18T08:09:34.395370+0000","last_peered":"2024-09-18T08:09:34.395370+0000","last_clean":"2024-09-18T08:09:34.395370+0000","last_became_active":"2024-09-18T08:02:47.879328+0000","last_became_peered":"2024-09-18T08:02:47.879328+0000","last_unstale":"2024-09-18T08:09:34.395370+0000","last_undegraded":"2024-09-18T08:09:34.395370+0000","last_fullsized":"2024-09-18T08:09:34.395370+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:08:18.389273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:46.860092+0000","last_clean_scrub_stamp":"2024-09-18T08:08:18.389273+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:09:46.276520+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.e","version":"182'12","reported_seq":96,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-09-18T08:09:38.009566+0000","last_change":"2024-09-18T08:08:18.560884+0000","last_active":"2024-09-18T08:09:38.009566+0000","last_peered":"2024-09-18T08:09:38.009566+0000","last_clean":"2024-09-18T08:09:38.009566+0000","last_became_active":"2024-09-18T08:08:18.525147+0000","last_became_peered":"2024-09-18T08:08:18.525147+0000","last_unstale":"2024-09-18T08:09:38.009566+0000","last_undegraded":"2024-09-18T08:09:38.009566+0000","last_fullsized":"2024-09-18T08:09:38.009566+0000","mapping_epoch":181,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":182,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'8","last_scrub_stamp":"2024-09-18T08:06:19.459375+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:50.886830+0000","last_clean_scrub_stamp":"2024-09-18T08:06:19.459375+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":328,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-09-18T08:09:34.396434+0000","last_change":"2024-09-18T08:07:19.714477+0000","last_active":"2024-09-18T08:09:34.396434+0000","last_peered":"2024-09-18T08:09:34.396434+0000","last_clean":"2024-09-18T08:09:34.396434+0000","last_became_active":"2024-09-18T08:02:53.909834+0000","last_became_peered":"2024-09-18T08:02:53.909834+0000","last_unstale":"2024-09-18T08:09:34.396434+0000","last_undegraded":"2024-09-18T08:09:34.396434+0000","last_fullsized":"2024-09-18T08:09:34.396434+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:07:19.714430+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_clean_scrub_stamp":"2024-09-18T08:07:19.714430+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.c","version":"213'6884","reported_seq":6991,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-09-18T08:09:38.780553+0000","last_change":"2024-09-18T08:07:53.670091+0000","last_active":"2024-09-18T08:09:38.780553+0000","last_peered":"2024-09-18T08:09:38.780553+0000","last_clean":"2024-09-18T08:09:38.780553+0000","last_became_active":"2024-09-18T08:07:53.669708+0000","last_became_peered":"2024-09-18T08:07:53.669708+0000","last_unstale":"2024-09-18T08:09:38.780553+0000","last_undegraded":"2024-09-18T08:09:38.780553+0000","last_fullsized":"2024-09-18T08:09:38.780553+0000","mapping_epoch":165,"log_start":"177'1500","ondisk_log_start":"177'1500","created":165,"last_epoch_clean":166,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:07:52.307205+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:07:52.307205+0000","last_clean_scrub_stamp":"2024-09-18T08:07:52.307205+0000","objects_scrubbed":0,"log_size":5384,"log_dups_size":0,"ondisk_log_size":5384,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":363462656,"num_objects":5546,"num_object_clones":0,"num_object_copies":16638,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5546,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6884,"num_write_kb":397760,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"74'79","reported_seq":89,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-09-18T08:09:34.393866+0000","last_change":"2024-09-18T08:08:52.690758+0000","last_active":"2024-09-18T08:09:34.393866+0000","last_peered":"2024-09-18T08:09:34.393866+0000","last_clean":"2024-09-18T08:09:34.393866+0000","last_became_active":"2024-09-18T08:08:52.690344+0000","last_became_peered":"2024-09-18T08:08:52.690344+0000","last_unstale":"2024-09-18T08:09:34.393866+0000","last_undegraded":"2024-09-18T08:09:34.393866+0000","last_fullsized":"2024-09-18T08:09:34.393866+0000","mapping_epoch":198,"log_start":"0'0","ondisk_log_start":"0'0","created":152,"last_epoch_clean":199,"parent":"0.0","parent_split_bits":3,"last_scrub":"74'79","last_scrub_stamp":"2024-09-18T08:08:19.036836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:00:11.893742+0000","last_clean_scrub_stamp":"2024-09-18T08:08:19.036836+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,6],"acting":[4,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.0","version":"69'18","reported_seq":361,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-09-18T08:09:34.395113+0000","last_change":"2024-09-18T08:06:44.301130+0000","last_active":"2024-09-18T08:09:34.395113+0000","last_peered":"2024-09-18T08:09:34.395113+0000","last_clean":"2024-09-18T08:09:34.395113+0000","last_became_active":"2024-09-18T08:02:49.886818+0000","last_became_peered":"2024-09-18T08:02:49.886818+0000","last_unstale":"2024-09-18T08:09:34.395113+0000","last_undegraded":"2024-09-18T08:09:34.395113+0000","last_fullsized":"2024-09-18T08:09:34.395113+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'18","last_scrub_stamp":"2024-09-18T08:06:44.300772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_clean_scrub_stamp":"2024-09-18T08:06:44.300772+0000","objects_scrubbed":8,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":34,"num_read_kb":22,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":341,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-09-18T08:09:34.395317+0000","last_change":"2024-09-18T08:07:21.242587+0000","last_active":"2024-09-18T08:09:34.395317+0000","last_peered":"2024-09-18T08:09:34.395317+0000","last_clean":"2024-09-18T08:09:34.395317+0000","last_became_active":"2024-09-18T08:02:47.880225+0000","last_became_peered":"2024-09-18T08:02:47.880225+0000","last_unstale":"2024-09-18T08:09:34.395317+0000","last_undegraded":"2024-09-18T08:09:34.395317+0000","last_fullsized":"2024-09-18T08:09:34.395317+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:07:21.242494+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:46.860092+0000","last_clean_scrub_stamp":"2024-09-18T08:07:21.242494+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":18,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":74,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-09-18T08:09:34.394484+0000","last_change":"2024-09-18T08:08:18.530658+0000","last_active":"2024-09-18T08:09:34.394484+0000","last_peered":"2024-09-18T08:09:34.394484+0000","last_clean":"2024-09-18T08:09:34.394484+0000","last_became_active":"2024-09-18T08:08:18.530341+0000","last_became_peered":"2024-09-18T08:08:18.530341+0000","last_unstale":"2024-09-18T08:09:34.394484+0000","last_undegraded":"2024-09-18T08:09:34.394484+0000","last_fullsized":"2024-09-18T08:09:34.394484+0000","mapping_epoch":181,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":182,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:06:10.468119+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:50.886830+0000","last_clean_scrub_stamp":"2024-09-18T08:06:10.468119+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":74,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-09-18T08:09:34.394076+0000","last_change":"2024-09-18T08:08:18.551527+0000","last_active":"2024-09-18T08:09:34.394076+0000","last_peered":"2024-09-18T08:09:34.394076+0000","last_clean":"2024-09-18T08:09:34.394076+0000","last_became_active":"2024-09-18T08:08:18.543694+0000","last_became_peered":"2024-09-18T08:08:18.543694+0000","last_unstale":"2024-09-18T08:09:34.394076+0000","last_undegraded":"2024-09-18T08:09:34.394076+0000","last_fullsized":"2024-09-18T08:09:34.394076+0000","mapping_epoch":181,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":182,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:04:16.427390+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_clean_scrub_stamp":"2024-09-18T08:04:16.427390+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.d","version":"213'6854","reported_seq":6962,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-09-18T08:09:38.217678+0000","last_change":"2024-09-18T08:07:53.671291+0000","last_active":"2024-09-18T08:09:38.217678+0000","last_peered":"2024-09-18T08:09:38.217678+0000","last_clean":"2024-09-18T08:09:38.217678+0000","last_became_active":"2024-09-18T08:07:53.670773+0000","last_became_peered":"2024-09-18T08:07:53.670773+0000","last_unstale":"2024-09-18T08:09:38.217678+0000","last_undegraded":"2024-09-18T08:09:38.217678+0000","last_fullsized":"2024-09-18T08:09:38.217678+0000","mapping_epoch":165,"log_start":"178'1700","ondisk_log_start":"178'1700","created":165,"last_epoch_clean":166,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:07:52.307205+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:07:52.307205+0000","last_clean_scrub_stamp":"2024-09-18T08:07:52.307205+0000","objects_scrubbed":0,"log_size":5154,"log_dups_size":0,"ondisk_log_size":5154,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":370606104,"num_objects":5656,"num_object_clones":0,"num_object_copies":16968,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5656,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":6854,"num_write_kb":400257,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,2],"acting":[5,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.4","version":"74'79","reported_seq":151,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-09-18T08:09:34.393693+0000","last_change":"2024-09-18T08:08:59.376417+0000","last_active":"2024-09-18T08:09:34.393693+0000","last_peered":"2024-09-18T08:09:34.393693+0000","last_clean":"2024-09-18T08:09:34.393693+0000","last_became_active":"2024-09-18T08:07:26.398930+0000","last_became_peered":"2024-09-18T08:07:26.398930+0000","last_unstale":"2024-09-18T08:09:34.393693+0000","last_undegraded":"2024-09-18T08:09:34.393693+0000","last_fullsized":"2024-09-18T08:09:34.393693+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":152,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":3,"last_scrub":"74'79","last_scrub_stamp":"2024-09-18T08:08:59.376337+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:00:11.893742+0000","last_clean_scrub_stamp":"2024-09-18T08:08:59.376337+0000","objects_scrubbed":1,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:10:26.641607+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":327680,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,7],"acting":[3,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"69'14","reported_seq":90,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-09-18T08:09:34.393441+0000","last_change":"2024-09-18T08:08:18.617734+0000","last_active":"2024-09-18T08:09:34.393441+0000","last_peered":"2024-09-18T08:09:34.393441+0000","last_clean":"2024-09-18T08:09:34.393441+0000","last_became_active":"2024-09-18T08:08:18.549671+0000","last_became_peered":"2024-09-18T08:08:18.549671+0000","last_unstale":"2024-09-18T08:09:34.393441+0000","last_undegraded":"2024-09-18T08:09:34.393441+0000","last_fullsized":"2024-09-18T08:09:34.393441+0000","mapping_epoch":181,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":182,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'14","last_scrub_stamp":"2024-09-18T08:08:08.629246+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_clean_scrub_stamp":"2024-09-18T08:08:08.629246+0000","objects_scrubbed":0,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":14,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"62'2","reported_seq":365,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-09-18T08:09:34.406149+0000","last_change":"2024-09-18T08:08:18.565279+0000","last_active":"2024-09-18T08:09:34.406149+0000","last_peered":"2024-09-18T08:09:34.406149+0000","last_clean":"2024-09-18T08:09:34.406149+0000","last_became_active":"2024-09-18T08:08:18.544872+0000","last_became_peered":"2024-09-18T08:08:18.544872+0000","last_unstale":"2024-09-18T08:09:34.406149+0000","last_undegraded":"2024-09-18T08:09:34.406149+0000","last_fullsized":"2024-09-18T08:09:34.406149+0000","mapping_epoch":181,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":182,"parent":"0.0","parent_split_bits":0,"last_scrub":"62'2","last_scrub_stamp":"2024-09-18T08:06:43.607152+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:46.860092+0000","last_clean_scrub_stamp":"2024-09-18T08:06:43.607152+0000","objects_scrubbed":2,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:10:26.053059+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1039,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":2078,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.0","version":"182'10","reported_seq":430,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-09-18T08:09:34.394935+0000","last_change":"2024-09-18T08:08:18.573278+0000","last_active":"2024-09-18T08:09:34.394935+0000","last_peered":"2024-09-18T08:09:34.394935+0000","last_clean":"2024-09-18T08:09:34.394935+0000","last_became_active":"2024-09-18T08:08:18.541756+0000","last_became_peered":"2024-09-18T08:08:18.541756+0000","last_unstale":"2024-09-18T08:09:34.394935+0000","last_undegraded":"2024-09-18T08:09:34.394935+0000","last_fullsized":"2024-09-18T08:09:34.394935+0000","mapping_epoch":181,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":182,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'8","last_scrub_stamp":"2024-09-18T08:04:23.863330+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:50.886830+0000","last_clean_scrub_stamp":"2024-09-18T08:04:23.863330+0000","objects_scrubbed":1,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:09:50.348476+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":343,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-09-18T08:09:34.408559+0000","last_change":"2024-09-18T08:07:49.650645+0000","last_active":"2024-09-18T08:09:34.408559+0000","last_peered":"2024-09-18T08:09:34.408559+0000","last_clean":"2024-09-18T08:09:34.408559+0000","last_became_active":"2024-09-18T08:06:24.140303+0000","last_became_peered":"2024-09-18T08:06:24.140303+0000","last_unstale":"2024-09-18T08:09:34.408559+0000","last_undegraded":"2024-09-18T08:09:34.408559+0000","last_fullsized":"2024-09-18T08:09:34.408559+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:07:49.650577+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_clean_scrub_stamp":"2024-09-18T08:07:49.650577+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:10:13.720079+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.e","version":"213'7020","reported_seq":11362,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-09-18T08:09:38.747948+0000","last_change":"2024-09-18T08:09:17.296111+0000","last_active":"2024-09-18T08:09:38.747948+0000","last_peered":"2024-09-18T08:09:38.747948+0000","last_clean":"2024-09-18T08:09:38.747948+0000","last_became_active":"2024-09-18T08:09:17.286866+0000","last_became_peered":"2024-09-18T08:09:17.286866+0000","last_unstale":"2024-09-18T08:09:38.747948+0000","last_undegraded":"2024-09-18T08:09:38.747948+0000","last_fullsized":"2024-09-18T08:09:38.747948+0000","mapping_epoch":207,"log_start":"177'1600","ondisk_log_start":"177'1600","created":165,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:07:52.307205+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:07:52.307205+0000","last_clean_scrub_stamp":"2024-09-18T08:07:52.307205+0000","objects_scrubbed":0,"log_size":5420,"log_dups_size":0,"ondisk_log_size":5420,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":376700928,"num_objects":5748,"num_object_clones":0,"num_object_copies":17244,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5748,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7020,"num_write_kb":408576,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2107,"num_bytes_recovered":138084352,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,4],"acting":[0,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.7","version":"74'79","reported_seq":146,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-09-18T08:09:34.394648+0000","last_change":"2024-09-18T08:08:57.438699+0000","last_active":"2024-09-18T08:09:34.394648+0000","last_peered":"2024-09-18T08:09:34.394648+0000","last_clean":"2024-09-18T08:09:34.394648+0000","last_became_active":"2024-09-18T08:07:26.398289+0000","last_became_peered":"2024-09-18T08:07:26.398289+0000","last_unstale":"2024-09-18T08:09:34.394648+0000","last_undegraded":"2024-09-18T08:09:34.394648+0000","last_fullsized":"2024-09-18T08:09:34.394648+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":152,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":3,"last_scrub":"74'79","last_scrub_stamp":"2024-09-18T08:08:57.438600+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:00:11.893742+0000","last_clean_scrub_stamp":"2024-09-18T08:08:57.438600+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:10:09.717533+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,7],"acting":[3,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.2","version":"69'10","reported_seq":389,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-09-18T08:09:34.410139+0000","last_change":"2024-09-18T08:08:18.580590+0000","last_active":"2024-09-18T08:09:34.410139+0000","last_peered":"2024-09-18T08:09:34.410139+0000","last_clean":"2024-09-18T08:09:34.410139+0000","last_became_active":"2024-09-18T08:08:18.538711+0000","last_became_peered":"2024-09-18T08:08:18.538711+0000","last_unstale":"2024-09-18T08:09:34.410139+0000","last_undegraded":"2024-09-18T08:09:34.410139+0000","last_fullsized":"2024-09-18T08:09:34.410139+0000","mapping_epoch":181,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":182,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'10","last_scrub_stamp":"2024-09-18T08:07:22.666042+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_clean_scrub_stamp":"2024-09-18T08:07:22.666042+0000","objects_scrubbed":5,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:10:17.688460+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":324,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-09-18T08:09:34.397009+0000","last_change":"2024-09-18T08:03:48.009689+0000","last_active":"2024-09-18T08:09:34.397009+0000","last_peered":"2024-09-18T08:09:34.397009+0000","last_clean":"2024-09-18T08:09:34.397009+0000","last_became_active":"2024-09-18T08:02:47.874727+0000","last_became_peered":"2024-09-18T08:02:47.874727+0000","last_unstale":"2024-09-18T08:09:34.397009+0000","last_undegraded":"2024-09-18T08:09:34.397009+0000","last_fullsized":"2024-09-18T08:09:34.397009+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:03:48.009611+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:46.860092+0000","last_clean_scrub_stamp":"2024-09-18T08:03:48.009611+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.3","version":"138'10","reported_seq":438,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-09-18T08:09:34.405038+0000","last_change":"2024-09-18T08:07:46.695714+0000","last_active":"2024-09-18T08:09:34.405038+0000","last_peered":"2024-09-18T08:09:34.405038+0000","last_clean":"2024-09-18T08:09:34.405038+0000","last_became_active":"2024-09-18T08:06:24.134194+0000","last_became_peered":"2024-09-18T08:06:24.134194+0000","last_unstale":"2024-09-18T08:09:34.405038+0000","last_undegraded":"2024-09-18T08:09:34.405038+0000","last_fullsized":"2024-09-18T08:09:34.405038+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"138'10","last_scrub_stamp":"2024-09-18T08:07:46.695656+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:50.886830+0000","last_clean_scrub_stamp":"2024-09-18T08:07:46.695656+0000","objects_scrubbed":1,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":343,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-09-18T08:09:34.404067+0000","last_change":"2024-09-18T08:08:52.700420+0000","last_active":"2024-09-18T08:09:34.404067+0000","last_peered":"2024-09-18T08:09:34.404067+0000","last_clean":"2024-09-18T08:09:34.404067+0000","last_became_active":"2024-09-18T08:08:52.700080+0000","last_became_peered":"2024-09-18T08:08:52.700080+0000","last_unstale":"2024-09-18T08:09:34.404067+0000","last_undegraded":"2024-09-18T08:09:34.404067+0000","last_fullsized":"2024-09-18T08:09:34.404067+0000","mapping_epoch":198,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":199,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:07:44.739578+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_clean_scrub_stamp":"2024-09-18T08:07:44.739578+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.f","version":"213'7117","reported_seq":23280,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-09-18T08:09:38.764060+0000","last_change":"2024-09-18T08:09:28.367000+0000","last_active":"2024-09-18T08:09:38.764060+0000","last_peered":"2024-09-18T08:09:38.764060+0000","last_clean":"2024-09-18T08:09:38.764060+0000","last_became_active":"2024-09-18T08:09:28.363033+0000","last_became_peered":"2024-09-18T08:09:28.363033+0000","last_unstale":"2024-09-18T08:09:38.764060+0000","last_undegraded":"2024-09-18T08:09:38.764060+0000","last_fullsized":"2024-09-18T08:09:38.764060+0000","mapping_epoch":211,"log_start":"183'2700","ondisk_log_start":"183'2700","created":165,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:07:52.307205+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:07:52.307205+0000","last_clean_scrub_stamp":"2024-09-18T08:07:52.307205+0000","objects_scrubbed":0,"log_size":4417,"log_dups_size":0,"ondisk_log_size":4417,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":379518976,"num_objects":5791,"num_object_clones":0,"num_object_copies":17373,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5791,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7117,"num_write_kb":413056,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8010,"num_bytes_recovered":524943360,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,1],"acting":[3,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"74'79","reported_seq":226,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-09-18T08:09:34.405459+0000","last_change":"2024-09-18T08:08:57.736557+0000","last_active":"2024-09-18T08:09:34.405459+0000","last_peered":"2024-09-18T08:09:34.405459+0000","last_clean":"2024-09-18T08:09:34.405459+0000","last_became_active":"2024-09-18T08:07:26.426532+0000","last_became_peered":"2024-09-18T08:07:26.426532+0000","last_unstale":"2024-09-18T08:09:34.405459+0000","last_undegraded":"2024-09-18T08:09:34.405459+0000","last_fullsized":"2024-09-18T08:09:34.405459+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":152,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":3,"last_scrub":"74'79","last_scrub_stamp":"2024-09-18T08:08:57.736355+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:00:11.893742+0000","last_clean_scrub_stamp":"2024-09-18T08:08:57.736355+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:10:19.314793+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,3],"acting":[0,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.3","version":"69'19","reported_seq":356,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-09-18T08:09:34.405177+0000","last_change":"2024-09-18T08:04:07.231406+0000","last_active":"2024-09-18T08:09:34.405177+0000","last_peered":"2024-09-18T08:09:34.405177+0000","last_clean":"2024-09-18T08:09:34.405177+0000","last_became_active":"2024-09-18T08:02:49.890318+0000","last_became_peered":"2024-09-18T08:02:49.890318+0000","last_unstale":"2024-09-18T08:09:34.405177+0000","last_undegraded":"2024-09-18T08:09:34.405177+0000","last_fullsized":"2024-09-18T08:09:34.405177+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'19","last_scrub_stamp":"2024-09-18T08:04:07.229846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_clean_scrub_stamp":"2024-09-18T08:04:07.229846+0000","objects_scrubbed":8,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:10:19.093125+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":39,"num_read_kb":25,"num_write":22,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.4","version":"62'1","reported_seq":360,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-09-18T08:09:34.411126+0000","last_change":"2024-09-18T08:09:02.569780+0000","last_active":"2024-09-18T08:09:34.411126+0000","last_peered":"2024-09-18T08:09:34.411126+0000","last_clean":"2024-09-18T08:09:34.411126+0000","last_became_active":"2024-09-18T08:08:52.688347+0000","last_became_peered":"2024-09-18T08:08:52.688347+0000","last_unstale":"2024-09-18T08:09:34.411126+0000","last_undegraded":"2024-09-18T08:09:34.411126+0000","last_fullsized":"2024-09-18T08:09:34.411126+0000","mapping_epoch":198,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":199,"parent":"0.0","parent_split_bits":0,"last_scrub":"62'1","last_scrub_stamp":"2024-09-18T08:09:02.569706+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:46.860092+0000","last_clean_scrub_stamp":"2024-09-18T08:09:02.569706+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:10:05.632401+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":436,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":872,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":250,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-09-18T08:09:34.404384+0000","last_change":"2024-09-18T08:06:24.133954+0000","last_active":"2024-09-18T08:09:34.404384+0000","last_peered":"2024-09-18T08:09:34.404384+0000","last_clean":"2024-09-18T08:09:34.404384+0000","last_became_active":"2024-09-18T08:06:24.133262+0000","last_became_peered":"2024-09-18T08:06:24.133262+0000","last_unstale":"2024-09-18T08:09:34.404384+0000","last_undegraded":"2024-09-18T08:09:34.404384+0000","last_fullsized":"2024-09-18T08:09:34.404384+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:04:13.377561+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:50.886830+0000","last_clean_scrub_stamp":"2024-09-18T08:04:13.377561+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:10:22.094192+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":356,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-09-18T08:09:34.410611+0000","last_change":"2024-09-18T08:08:52.688537+0000","last_active":"2024-09-18T08:09:34.410611+0000","last_peered":"2024-09-18T08:09:34.410611+0000","last_clean":"2024-09-18T08:09:34.410611+0000","last_became_active":"2024-09-18T08:08:52.688212+0000","last_became_peered":"2024-09-18T08:08:52.688212+0000","last_unstale":"2024-09-18T08:09:34.410611+0000","last_undegraded":"2024-09-18T08:09:34.410611+0000","last_fullsized":"2024-09-18T08:09:34.410611+0000","mapping_epoch":198,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":199,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:07:49.041138+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_clean_scrub_stamp":"2024-09-18T08:07:49.041138+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:10:25.679955+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.8","version":"213'6859","reported_seq":6966,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-09-18T08:09:38.231055+0000","last_change":"2024-09-18T08:07:53.671548+0000","last_active":"2024-09-18T08:09:38.231055+0000","last_peered":"2024-09-18T08:09:38.231055+0000","last_clean":"2024-09-18T08:09:38.231055+0000","last_became_active":"2024-09-18T08:07:53.670635+0000","last_became_peered":"2024-09-18T08:07:53.670635+0000","last_unstale":"2024-09-18T08:09:38.231055+0000","last_undegraded":"2024-09-18T08:09:38.231055+0000","last_fullsized":"2024-09-18T08:09:38.231055+0000","mapping_epoch":165,"log_start":"178'1700","ondisk_log_start":"178'1700","created":165,"last_epoch_clean":166,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:07:52.307205+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:07:52.307205+0000","last_clean_scrub_stamp":"2024-09-18T08:07:52.307205+0000","objects_scrubbed":0,"log_size":5159,"log_dups_size":0,"ondisk_log_size":5159,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":366542848,"num_objects":5593,"num_object_clones":0,"num_object_copies":16779,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5593,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6859,"num_write_kb":398464,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.1","version":"74'79","reported_seq":145,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-09-18T08:09:34.396131+0000","last_change":"2024-09-18T08:08:59.553431+0000","last_active":"2024-09-18T08:09:34.396131+0000","last_peered":"2024-09-18T08:09:34.396131+0000","last_clean":"2024-09-18T08:09:34.396131+0000","last_became_active":"2024-09-18T08:07:26.400643+0000","last_became_peered":"2024-09-18T08:07:26.400643+0000","last_unstale":"2024-09-18T08:09:34.396131+0000","last_undegraded":"2024-09-18T08:09:34.396131+0000","last_fullsized":"2024-09-18T08:09:34.396131+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":152,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":4,"last_scrub":"74'79","last_scrub_stamp":"2024-09-18T08:08:59.552592+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:00:11.893742+0000","last_clean_scrub_stamp":"2024-09-18T08:08:59.552592+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:10:15.399172+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,2],"acting":[5,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.4","version":"69'28","reported_seq":459,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-09-18T08:09:34.410751+0000","last_change":"2024-09-18T08:08:57.639985+0000","last_active":"2024-09-18T08:09:34.410751+0000","last_peered":"2024-09-18T08:09:34.410751+0000","last_clean":"2024-09-18T08:09:34.410751+0000","last_became_active":"2024-09-18T08:02:49.897221+0000","last_became_peered":"2024-09-18T08:02:49.897221+0000","last_unstale":"2024-09-18T08:09:34.410751+0000","last_undegraded":"2024-09-18T08:09:34.410751+0000","last_fullsized":"2024-09-18T08:09:34.410751+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'28","last_scrub_stamp":"2024-09-18T08:08:57.639628+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_clean_scrub_stamp":"2024-09-18T08:08:57.639628+0000","objects_scrubbed":10,"log_size":28,"log_dups_size":0,"ondisk_log_size":28,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:10:27.518247+0000","scrub_duration":18,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":48,"num_read_kb":33,"num_write":26,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":74,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-09-18T08:09:34.394010+0000","last_change":"2024-09-18T08:08:18.543579+0000","last_active":"2024-09-18T08:09:34.394010+0000","last_peered":"2024-09-18T08:09:34.394010+0000","last_clean":"2024-09-18T08:09:34.394010+0000","last_became_active":"2024-09-18T08:08:18.538852+0000","last_became_peered":"2024-09-18T08:08:18.538852+0000","last_unstale":"2024-09-18T08:09:34.394010+0000","last_undegraded":"2024-09-18T08:09:34.394010+0000","last_fullsized":"2024-09-18T08:09:34.394010+0000","mapping_epoch":181,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":182,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:05:39.490141+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:46.860092+0000","last_clean_scrub_stamp":"2024-09-18T08:05:39.490141+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"64'2","reported_seq":364,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-09-18T08:09:34.396539+0000","last_change":"2024-09-18T08:06:24.165227+0000","last_active":"2024-09-18T08:09:34.396539+0000","last_peered":"2024-09-18T08:09:34.396539+0000","last_clean":"2024-09-18T08:09:34.396539+0000","last_became_active":"2024-09-18T08:06:24.136080+0000","last_became_peered":"2024-09-18T08:06:24.136080+0000","last_unstale":"2024-09-18T08:09:34.396539+0000","last_undegraded":"2024-09-18T08:09:34.396539+0000","last_fullsized":"2024-09-18T08:09:34.396539+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"64'2","last_scrub_stamp":"2024-09-18T08:05:44.802942+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:45.855979+0000","last_clean_scrub_stamp":"2024-09-18T08:05:44.802942+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.001348349,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":38,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":330,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-09-18T08:09:34.404418+0000","last_change":"2024-09-18T08:08:18.544944+0000","last_active":"2024-09-18T08:09:34.404418+0000","last_peered":"2024-09-18T08:09:34.404418+0000","last_clean":"2024-09-18T08:09:34.404418+0000","last_became_active":"2024-09-18T08:08:18.544260+0000","last_became_peered":"2024-09-18T08:08:18.544260+0000","last_unstale":"2024-09-18T08:09:34.404418+0000","last_undegraded":"2024-09-18T08:09:34.404418+0000","last_fullsized":"2024-09-18T08:09:34.404418+0000","mapping_epoch":181,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":182,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:07:53.582095+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:50.886830+0000","last_clean_scrub_stamp":"2024-09-18T08:07:53.582095+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":336,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-09-18T08:09:34.394100+0000","last_change":"2024-09-18T08:08:18.543527+0000","last_active":"2024-09-18T08:09:34.394100+0000","last_peered":"2024-09-18T08:09:34.394100+0000","last_clean":"2024-09-18T08:09:34.394100+0000","last_became_active":"2024-09-18T08:08:18.542678+0000","last_became_peered":"2024-09-18T08:08:18.542678+0000","last_unstale":"2024-09-18T08:09:34.394100+0000","last_undegraded":"2024-09-18T08:09:34.394100+0000","last_fullsized":"2024-09-18T08:09:34.394100+0000","mapping_epoch":181,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":182,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:07:08.245063+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_clean_scrub_stamp":"2024-09-18T08:07:08.245063+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.b","version":"213'6961","reported_seq":12580,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-09-18T08:09:38.773547+0000","last_change":"2024-09-18T08:08:48.662821+0000","last_active":"2024-09-18T08:09:38.773547+0000","last_peered":"2024-09-18T08:09:38.773547+0000","last_clean":"2024-09-18T08:09:38.773547+0000","last_became_active":"2024-09-18T08:08:48.655676+0000","last_became_peered":"2024-09-18T08:08:48.655676+0000","last_unstale":"2024-09-18T08:09:38.773547+0000","last_undegraded":"2024-09-18T08:09:38.773547+0000","last_fullsized":"2024-09-18T08:09:38.773547+0000","mapping_epoch":194,"log_start":"183'2600","ondisk_log_start":"183'2600","created":165,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:07:52.307205+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:07:52.307205+0000","last_clean_scrub_stamp":"2024-09-18T08:07:52.307205+0000","objects_scrubbed":0,"log_size":4361,"log_dups_size":0,"ondisk_log_size":4361,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":374276096,"num_objects":5711,"num_object_clones":0,"num_object_copies":17133,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5711,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6961,"num_write_kb":405504,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2745,"num_bytes_recovered":179896320,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,1],"acting":[3,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"74'79","reported_seq":247,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-09-18T08:09:34.404234+0000","last_change":"2024-09-18T08:08:18.524000+0000","last_active":"2024-09-18T08:09:34.404234+0000","last_peered":"2024-09-18T08:09:34.404234+0000","last_clean":"2024-09-18T08:09:34.404234+0000","last_became_active":"2024-09-18T08:08:18.522977+0000","last_became_peered":"2024-09-18T08:08:18.522977+0000","last_unstale":"2024-09-18T08:09:34.404234+0000","last_undegraded":"2024-09-18T08:09:34.404234+0000","last_fullsized":"2024-09-18T08:09:34.404234+0000","mapping_epoch":181,"log_start":"0'0","ondisk_log_start":"0'0","created":152,"last_epoch_clean":182,"parent":"0.0","parent_split_bits":4,"last_scrub":"74'79","last_scrub_stamp":"2024-09-18T08:08:09.574144+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:00:11.893742+0000","last_clean_scrub_stamp":"2024-09-18T08:08:09.574144+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,4],"acting":[0,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.7","version":"69'13","reported_seq":266,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-09-18T08:09:34.403264+0000","last_change":"2024-09-18T08:04:18.164662+0000","last_active":"2024-09-18T08:09:34.403264+0000","last_peered":"2024-09-18T08:09:34.403264+0000","last_clean":"2024-09-18T08:09:34.403264+0000","last_became_active":"2024-09-18T08:04:18.144925+0000","last_became_peered":"2024-09-18T08:04:18.144925+0000","last_unstale":"2024-09-18T08:09:34.403264+0000","last_undegraded":"2024-09-18T08:09:34.403264+0000","last_fullsized":"2024-09-18T08:09:34.403264+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'13","last_scrub_stamp":"2024-09-18T08:03:50.700818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_clean_scrub_stamp":"2024-09-18T08:03:50.700818+0000","objects_scrubbed":0,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":29,"num_read_kb":18,"num_write":16,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,0],"acting":[2,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":346,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-09-18T08:09:34.409656+0000","last_change":"2024-09-18T08:08:53.649127+0000","last_active":"2024-09-18T08:09:34.409656+0000","last_peered":"2024-09-18T08:09:34.409656+0000","last_clean":"2024-09-18T08:09:34.409656+0000","last_became_active":"2024-09-18T08:06:24.126334+0000","last_became_peered":"2024-09-18T08:06:24.126334+0000","last_unstale":"2024-09-18T08:09:34.409656+0000","last_undegraded":"2024-09-18T08:09:34.409656+0000","last_fullsized":"2024-09-18T08:09:34.409656+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:08:53.649079+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:46.860092+0000","last_clean_scrub_stamp":"2024-09-18T08:08:53.649079+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:10:14.347881+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"62'1","reported_seq":334,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-09-18T08:09:34.400381+0000","last_change":"2024-09-18T08:04:09.732724+0000","last_active":"2024-09-18T08:09:34.400381+0000","last_peered":"2024-09-18T08:09:34.400381+0000","last_clean":"2024-09-18T08:09:34.400381+0000","last_became_active":"2024-09-18T08:02:46.868175+0000","last_became_peered":"2024-09-18T08:02:46.868175+0000","last_unstale":"2024-09-18T08:09:34.400381+0000","last_undegraded":"2024-09-18T08:09:34.400381+0000","last_fullsized":"2024-09-18T08:09:34.400381+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"62'1","last_scrub_stamp":"2024-09-18T08:04:09.732645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:45.855979+0000","last_clean_scrub_stamp":"2024-09-18T08:04:09.732645+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0.00066785499999999997,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":40,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-09-18T08:09:34.400255+0000","last_change":"2024-09-18T08:08:52.690656+0000","last_active":"2024-09-18T08:09:34.400255+0000","last_peered":"2024-09-18T08:09:34.400255+0000","last_clean":"2024-09-18T08:09:34.400255+0000","last_became_active":"2024-09-18T08:08:52.690286+0000","last_became_peered":"2024-09-18T08:08:52.690286+0000","last_unstale":"2024-09-18T08:09:34.400255+0000","last_undegraded":"2024-09-18T08:09:34.400255+0000","last_fullsized":"2024-09-18T08:09:34.400255+0000","mapping_epoch":198,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":199,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:07:08.806748+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:50.886830+0000","last_clean_scrub_stamp":"2024-09-18T08:07:08.806748+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":343,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-09-18T08:09:42.647525+0000","last_change":"2024-09-18T08:09:42.647525+0000","last_active":"2024-09-18T08:09:42.647525+0000","last_peered":"2024-09-18T08:09:42.647525+0000","last_clean":"2024-09-18T08:09:42.647525+0000","last_became_active":"2024-09-18T08:06:24.145382+0000","last_became_peered":"2024-09-18T08:06:24.145382+0000","last_unstale":"2024-09-18T08:09:42.647525+0000","last_undegraded":"2024-09-18T08:09:42.647525+0000","last_fullsized":"2024-09-18T08:09:42.647525+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:09:42.647317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_clean_scrub_stamp":"2024-09-18T08:09:42.647317+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:11:03.015274+0000","scrub_duration":26,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.a","version":"213'6930","reported_seq":7037,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-09-18T08:09:38.514906+0000","last_change":"2024-09-18T08:07:53.326651+0000","last_active":"2024-09-18T08:09:38.514906+0000","last_peered":"2024-09-18T08:09:38.514906+0000","last_clean":"2024-09-18T08:09:38.514906+0000","last_became_active":"2024-09-18T08:07:53.326361+0000","last_became_peered":"2024-09-18T08:07:53.326361+0000","last_unstale":"2024-09-18T08:09:38.514906+0000","last_undegraded":"2024-09-18T08:09:38.514906+0000","last_fullsized":"2024-09-18T08:09:38.514906+0000","mapping_epoch":165,"log_start":"180'2100","ondisk_log_start":"180'2100","created":165,"last_epoch_clean":166,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:07:52.307205+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:07:52.307205+0000","last_clean_scrub_stamp":"2024-09-18T08:07:52.307205+0000","objects_scrubbed":0,"log_size":4830,"log_dups_size":0,"ondisk_log_size":4830,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:10:20.719923+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":370540544,"num_objects":5654,"num_object_clones":0,"num_object_copies":16962,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5654,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6930,"num_write_kb":402688,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.3","version":"74'79","reported_seq":159,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-09-18T08:09:34.408805+0000","last_change":"2024-09-18T08:08:19.300155+0000","last_active":"2024-09-18T08:09:34.408805+0000","last_peered":"2024-09-18T08:09:34.408805+0000","last_clean":"2024-09-18T08:09:34.408805+0000","last_became_active":"2024-09-18T08:08:18.538929+0000","last_became_peered":"2024-09-18T08:08:18.538929+0000","last_unstale":"2024-09-18T08:09:34.408805+0000","last_undegraded":"2024-09-18T08:09:34.408805+0000","last_fullsized":"2024-09-18T08:09:34.408805+0000","mapping_epoch":181,"log_start":"0'0","ondisk_log_start":"0'0","created":152,"last_epoch_clean":182,"parent":"0.0","parent_split_bits":3,"last_scrub":"74'79","last_scrub_stamp":"2024-09-18T08:08:19.299929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:00:11.893742+0000","last_clean_scrub_stamp":"2024-09-18T08:08:19.299929+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:09:36.333261+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.6","version":"69'12","reported_seq":388,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-09-18T08:09:34.405957+0000","last_change":"2024-09-18T08:08:57.021693+0000","last_active":"2024-09-18T08:09:34.405957+0000","last_peered":"2024-09-18T08:09:34.405957+0000","last_clean":"2024-09-18T08:09:34.405957+0000","last_became_active":"2024-09-18T08:08:52.700130+0000","last_became_peered":"2024-09-18T08:08:52.700130+0000","last_unstale":"2024-09-18T08:09:34.405957+0000","last_undegraded":"2024-09-18T08:09:34.405957+0000","last_fullsized":"2024-09-18T08:09:34.405957+0000","mapping_epoch":198,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":200,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'12","last_scrub_stamp":"2024-09-18T08:07:39.603562+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_clean_scrub_stamp":"2024-09-18T08:07:39.603562+0000","objects_scrubbed":6,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":18,"num_read_kb":12,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":354,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-09-18T08:09:34.409420+0000","last_change":"2024-09-18T08:09:34.089548+0000","last_active":"2024-09-18T08:09:34.409420+0000","last_peered":"2024-09-18T08:09:34.409420+0000","last_clean":"2024-09-18T08:09:34.409420+0000","last_became_active":"2024-09-18T08:08:18.523135+0000","last_became_peered":"2024-09-18T08:08:18.523135+0000","last_unstale":"2024-09-18T08:09:34.409420+0000","last_undegraded":"2024-09-18T08:09:34.409420+0000","last_fullsized":"2024-09-18T08:09:34.409420+0000","mapping_epoch":181,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":182,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:09:34.089476+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:46.860092+0000","last_clean_scrub_stamp":"2024-09-18T08:09:34.089476+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:10:41.071862+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"70'5","reported_seq":382,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-09-18T08:09:36.712393+0000","last_change":"2024-09-18T08:09:09.649261+0000","last_active":"2024-09-18T08:09:36.712393+0000","last_peered":"2024-09-18T08:09:36.712393+0000","last_clean":"2024-09-18T08:09:36.712393+0000","last_became_active":"2024-09-18T08:06:46.141667+0000","last_became_peered":"2024-09-18T08:06:46.141667+0000","last_unstale":"2024-09-18T08:09:36.712393+0000","last_undegraded":"2024-09-18T08:09:36.712393+0000","last_fullsized":"2024-09-18T08:09:36.712393+0000","mapping_epoch":143,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":144,"parent":"0.0","parent_split_bits":0,"last_scrub":"70'5","last_scrub_stamp":"2024-09-18T08:09:09.649188+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:45.855979+0000","last_clean_scrub_stamp":"2024-09-18T08:09:09.649188+0000","objects_scrubbed":1,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:10:24.340859+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0.000504055,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":178,"num_read_kb":173,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":332,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-09-18T08:09:34.397093+0000","last_change":"2024-09-18T08:07:20.767916+0000","last_active":"2024-09-18T08:09:34.397093+0000","last_peered":"2024-09-18T08:09:34.397093+0000","last_clean":"2024-09-18T08:09:34.397093+0000","last_became_active":"2024-09-18T08:02:51.913112+0000","last_became_peered":"2024-09-18T08:02:51.913112+0000","last_unstale":"2024-09-18T08:09:34.397093+0000","last_undegraded":"2024-09-18T08:09:34.397093+0000","last_fullsized":"2024-09-18T08:09:34.397093+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:07:20.767841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:50.886830+0000","last_clean_scrub_stamp":"2024-09-18T08:07:20.767841+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":305,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-09-18T08:09:34.408656+0000","last_change":"2024-09-18T08:02:53.907689+0000","last_active":"2024-09-18T08:09:34.408656+0000","last_peered":"2024-09-18T08:09:34.408656+0000","last_clean":"2024-09-18T08:09:34.408656+0000","last_became_active":"2024-09-18T08:02:53.907404+0000","last_became_peered":"2024-09-18T08:02:53.907404+0000","last_unstale":"2024-09-18T08:09:34.408656+0000","last_undegraded":"2024-09-18T08:09:34.408656+0000","last_fullsized":"2024-09-18T08:09:34.408656+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_clean_scrub_stamp":"2024-09-18T08:02:52.890475+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:09:49.627839+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.9","version":"213'6686","reported_seq":6793,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-09-18T08:09:38.201707+0000","last_change":"2024-09-18T08:07:53.672294+0000","last_active":"2024-09-18T08:09:38.201707+0000","last_peered":"2024-09-18T08:09:38.201707+0000","last_clean":"2024-09-18T08:09:38.201707+0000","last_became_active":"2024-09-18T08:07:53.671994+0000","last_became_peered":"2024-09-18T08:07:53.671994+0000","last_unstale":"2024-09-18T08:09:38.201707+0000","last_undegraded":"2024-09-18T08:09:38.201707+0000","last_fullsized":"2024-09-18T08:09:38.201707+0000","mapping_epoch":165,"log_start":"177'1600","ondisk_log_start":"177'1600","created":165,"last_epoch_clean":166,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:07:52.307205+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:07:52.307205+0000","last_clean_scrub_stamp":"2024-09-18T08:07:52.307205+0000","objects_scrubbed":0,"log_size":5086,"log_dups_size":0,"ondisk_log_size":5086,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":362283008,"num_objects":5528,"num_object_clones":0,"num_object_copies":16584,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5528,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6686,"num_write_kb":390848,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.5","version":"182'21","reported_seq":110,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-09-18T08:09:38.009621+0000","last_change":"2024-09-18T08:08:18.609652+0000","last_active":"2024-09-18T08:09:38.009621+0000","last_peered":"2024-09-18T08:09:38.009621+0000","last_clean":"2024-09-18T08:09:38.009621+0000","last_became_active":"2024-09-18T08:08:18.547555+0000","last_became_peered":"2024-09-18T08:08:18.547555+0000","last_unstale":"2024-09-18T08:09:38.009621+0000","last_undegraded":"2024-09-18T08:09:38.009621+0000","last_fullsized":"2024-09-18T08:09:38.009621+0000","mapping_epoch":181,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":182,"parent":"0.0","parent_split_bits":0,"last_scrub":"130'18","last_scrub_stamp":"2024-09-18T08:06:15.455503+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_clean_scrub_stamp":"2024-09-18T08:06:15.455503+0000","objects_scrubbed":0,"log_size":21,"log_dups_size":0,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":24,"num_read_kb":14,"num_write":13,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":154,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,7],"acting":[4,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":339,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-09-18T08:09:34.394163+0000","last_change":"2024-09-18T08:06:24.134043+0000","last_active":"2024-09-18T08:09:34.394163+0000","last_peered":"2024-09-18T08:09:34.394163+0000","last_clean":"2024-09-18T08:09:34.394163+0000","last_became_active":"2024-09-18T08:06:24.133425+0000","last_became_peered":"2024-09-18T08:06:24.133425+0000","last_unstale":"2024-09-18T08:09:34.394163+0000","last_undegraded":"2024-09-18T08:09:34.394163+0000","last_fullsized":"2024-09-18T08:09:34.394163+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:05:38.080993+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:46.860092+0000","last_clean_scrub_stamp":"2024-09-18T08:05:38.080993+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"74'79","reported_seq":428,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-09-18T08:09:34.408404+0000","last_change":"2024-09-18T08:08:57.769104+0000","last_active":"2024-09-18T08:09:34.408404+0000","last_peered":"2024-09-18T08:09:34.408404+0000","last_clean":"2024-09-18T08:09:34.408404+0000","last_became_active":"2024-09-18T08:07:24.376931+0000","last_became_peered":"2024-09-18T08:07:24.376931+0000","last_unstale":"2024-09-18T08:09:34.408404+0000","last_undegraded":"2024-09-18T08:09:34.408404+0000","last_fullsized":"2024-09-18T08:09:34.408404+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":153,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'79","last_scrub_stamp":"2024-09-18T08:08:57.769048+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:00:11.893742+0000","last_clean_scrub_stamp":"2024-09-18T08:08:57.769048+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:10:19.097626+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":160,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-09-18T08:09:34.401128+0000","last_change":"2024-09-18T08:07:09.744206+0000","last_active":"2024-09-18T08:09:34.401128+0000","last_peered":"2024-09-18T08:09:34.401128+0000","last_clean":"2024-09-18T08:09:34.401128+0000","last_became_active":"2024-09-18T08:06:46.141046+0000","last_became_peered":"2024-09-18T08:06:46.141046+0000","last_unstale":"2024-09-18T08:09:34.401128+0000","last_undegraded":"2024-09-18T08:09:34.401128+0000","last_fullsized":"2024-09-18T08:09:34.401128+0000","mapping_epoch":143,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":144,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:07:09.744137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:50.886830+0000","last_clean_scrub_stamp":"2024-09-18T08:07:09.744137+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":335,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-09-18T08:09:34.396890+0000","last_change":"2024-09-18T08:08:18.548620+0000","last_active":"2024-09-18T08:09:34.396890+0000","last_peered":"2024-09-18T08:09:34.396890+0000","last_clean":"2024-09-18T08:09:34.396890+0000","last_became_active":"2024-09-18T08:08:18.548233+0000","last_became_peered":"2024-09-18T08:08:18.548233+0000","last_unstale":"2024-09-18T08:09:34.396890+0000","last_undegraded":"2024-09-18T08:09:34.396890+0000","last_fullsized":"2024-09-18T08:09:34.396890+0000","mapping_epoch":181,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":182,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:06:50.710565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_clean_scrub_stamp":"2024-09-18T08:06:50.710565+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.2","version":"213'7228","reported_seq":11612,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-09-18T08:09:43.128908+0000","last_change":"2024-09-18T08:08:58.743574+0000","last_active":"2024-09-18T08:09:43.128908+0000","last_peered":"2024-09-18T08:09:43.128908+0000","last_clean":"2024-09-18T08:09:43.128908+0000","last_became_active":"2024-09-18T08:08:58.738458+0000","last_became_peered":"2024-09-18T08:08:58.738458+0000","last_unstale":"2024-09-18T08:09:43.128908+0000","last_undegraded":"2024-09-18T08:09:43.128908+0000","last_fullsized":"2024-09-18T08:09:43.128908+0000","mapping_epoch":201,"log_start":"184'2700","ondisk_log_start":"184'2700","created":165,"last_epoch_clean":202,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:07:52.307205+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:07:52.307205+0000","last_clean_scrub_stamp":"2024-09-18T08:07:52.307205+0000","objects_scrubbed":0,"log_size":4528,"log_dups_size":0,"ondisk_log_size":4528,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":350224384,"num_objects":5344,"num_object_clones":0,"num_object_copies":16032,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5344,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7228,"num_write_kb":402304,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2129,"num_bytes_recovered":139526144,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,5],"acting":[4,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.e","version":"69'11","reported_seq":86,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-09-18T08:09:34.395037+0000","last_change":"2024-09-18T08:08:18.594670+0000","last_active":"2024-09-18T08:09:34.395037+0000","last_peered":"2024-09-18T08:09:34.395037+0000","last_clean":"2024-09-18T08:09:34.395037+0000","last_became_active":"2024-09-18T08:08:18.544581+0000","last_became_peered":"2024-09-18T08:08:18.544581+0000","last_unstale":"2024-09-18T08:09:34.395037+0000","last_undegraded":"2024-09-18T08:09:34.395037+0000","last_fullsized":"2024-09-18T08:09:34.395037+0000","mapping_epoch":181,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":182,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'11","last_scrub_stamp":"2024-09-18T08:06:14.472168+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_clean_scrub_stamp":"2024-09-18T08:06:14.472168+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":19,"num_read_kb":12,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":74,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-09-18T08:09:34.394659+0000","last_change":"2024-09-18T08:08:18.524516+0000","last_active":"2024-09-18T08:09:34.394659+0000","last_peered":"2024-09-18T08:09:34.394659+0000","last_clean":"2024-09-18T08:09:34.394659+0000","last_became_active":"2024-09-18T08:08:18.524229+0000","last_became_peered":"2024-09-18T08:08:18.524229+0000","last_unstale":"2024-09-18T08:09:34.394659+0000","last_undegraded":"2024-09-18T08:09:34.394659+0000","last_fullsized":"2024-09-18T08:09:34.394659+0000","mapping_epoch":181,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":182,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:02:46.860092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:46.860092+0000","last_clean_scrub_stamp":"2024-09-18T08:02:46.860092+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":352,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-09-18T08:09:34.396780+0000","last_change":"2024-09-18T08:08:18.547880+0000","last_active":"2024-09-18T08:09:34.396780+0000","last_peered":"2024-09-18T08:09:34.396780+0000","last_clean":"2024-09-18T08:09:34.396780+0000","last_became_active":"2024-09-18T08:08:18.547105+0000","last_became_peered":"2024-09-18T08:08:18.547105+0000","last_unstale":"2024-09-18T08:09:34.396780+0000","last_undegraded":"2024-09-18T08:09:34.396780+0000","last_fullsized":"2024-09-18T08:09:34.396780+0000","mapping_epoch":181,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":182,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:07:51.742090+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:50.886830+0000","last_clean_scrub_stamp":"2024-09-18T08:07:51.742090+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":334,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-09-18T08:09:34.393801+0000","last_change":"2024-09-18T08:07:09.197912+0000","last_active":"2024-09-18T08:09:34.393801+0000","last_peered":"2024-09-18T08:09:34.393801+0000","last_clean":"2024-09-18T08:09:34.393801+0000","last_became_active":"2024-09-18T08:06:24.130266+0000","last_became_peered":"2024-09-18T08:06:24.130266+0000","last_unstale":"2024-09-18T08:09:34.393801+0000","last_undegraded":"2024-09-18T08:09:34.393801+0000","last_fullsized":"2024-09-18T08:09:34.393801+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:07:09.197841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_clean_scrub_stamp":"2024-09-18T08:07:09.197841+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.1","version":"213'6961","reported_seq":7097,"reported_epoch":213,"state":"active+clean+scrubbing","last_fresh":"2024-09-18T08:09:38.521200+0000","last_change":"2024-09-18T08:09:37.622667+0000","last_active":"2024-09-18T08:09:38.521200+0000","last_peered":"2024-09-18T08:09:38.521200+0000","last_clean":"2024-09-18T08:09:38.521200+0000","last_became_active":"2024-09-18T08:07:53.327987+0000","last_became_peered":"2024-09-18T08:07:53.327987+0000","last_unstale":"2024-09-18T08:09:38.521200+0000","last_undegraded":"2024-09-18T08:09:38.521200+0000","last_fullsized":"2024-09-18T08:09:38.521200+0000","mapping_epoch":165,"log_start":"180'2100","ondisk_log_start":"180'2100","created":165,"last_epoch_clean":166,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:07:52.307205+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:07:52.307205+0000","last_clean_scrub_stamp":"2024-09-18T08:07:52.307205+0000","objects_scrubbed":2449,"log_size":4861,"log_dups_size":0,"ondisk_log_size":4861,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"scrubbing for 1s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":374669312,"num_objects":5717,"num_object_clones":0,"num_object_copies":17151,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5717,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6961,"num_write_kb":405696,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.8","version":"74'79","reported_seq":152,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-09-18T08:09:34.395848+0000","last_change":"2024-09-18T08:09:00.533593+0000","last_active":"2024-09-18T08:09:34.395848+0000","last_peered":"2024-09-18T08:09:34.395848+0000","last_clean":"2024-09-18T08:09:34.395848+0000","last_became_active":"2024-09-18T08:07:26.407298+0000","last_became_peered":"2024-09-18T08:07:26.407298+0000","last_unstale":"2024-09-18T08:09:34.395848+0000","last_undegraded":"2024-09-18T08:09:34.395848+0000","last_fullsized":"2024-09-18T08:09:34.395848+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":152,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":4,"last_scrub":"74'79","last_scrub_stamp":"2024-09-18T08:09:00.532813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:00:11.893742+0000","last_clean_scrub_stamp":"2024-09-18T08:09:00.532813+0000","objects_scrubbed":1,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:10:17.887770+0000","scrub_duration":25,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262688,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,3],"acting":[5,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.d","version":"69'17","reported_seq":92,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-09-18T08:09:34.394322+0000","last_change":"2024-09-18T08:08:18.588624+0000","last_active":"2024-09-18T08:09:34.394322+0000","last_peered":"2024-09-18T08:09:34.394322+0000","last_clean":"2024-09-18T08:09:34.394322+0000","last_became_active":"2024-09-18T08:08:18.540862+0000","last_became_peered":"2024-09-18T08:08:18.540862+0000","last_unstale":"2024-09-18T08:09:34.394322+0000","last_undegraded":"2024-09-18T08:09:34.394322+0000","last_fullsized":"2024-09-18T08:09:34.394322+0000","mapping_epoch":181,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":182,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'17","last_scrub_stamp":"2024-09-18T08:08:09.671248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_clean_scrub_stamp":"2024-09-18T08:08:09.671248+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":28,"num_read_kb":18,"num_write":18,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,1],"acting":[4,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":74,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-09-18T08:09:34.394252+0000","last_change":"2024-09-18T08:08:18.541029+0000","last_active":"2024-09-18T08:09:34.394252+0000","last_peered":"2024-09-18T08:09:34.394252+0000","last_clean":"2024-09-18T08:09:34.394252+0000","last_became_active":"2024-09-18T08:08:18.540285+0000","last_became_peered":"2024-09-18T08:08:18.540285+0000","last_unstale":"2024-09-18T08:09:34.394252+0000","last_undegraded":"2024-09-18T08:09:34.394252+0000","last_fullsized":"2024-09-18T08:09:34.394252+0000","mapping_epoch":181,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":182,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:05:12.622901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:46.860092+0000","last_clean_scrub_stamp":"2024-09-18T08:05:12.622901+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":346,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-09-18T08:09:34.411078+0000","last_change":"2024-09-18T08:09:06.610955+0000","last_active":"2024-09-18T08:09:34.411078+0000","last_peered":"2024-09-18T08:09:34.411078+0000","last_clean":"2024-09-18T08:09:34.411078+0000","last_became_active":"2024-09-18T08:08:18.623014+0000","last_became_peered":"2024-09-18T08:08:18.623014+0000","last_unstale":"2024-09-18T08:09:34.411078+0000","last_undegraded":"2024-09-18T08:09:34.411078+0000","last_fullsized":"2024-09-18T08:09:34.411078+0000","mapping_epoch":181,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":182,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:09:06.610879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:50.886830+0000","last_clean_scrub_stamp":"2024-09-18T08:09:06.610879+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:10:19.831235+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":40,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-09-18T08:09:34.403168+0000","last_change":"2024-09-18T08:08:52.696713+0000","last_active":"2024-09-18T08:09:34.403168+0000","last_peered":"2024-09-18T08:09:34.403168+0000","last_clean":"2024-09-18T08:09:34.403168+0000","last_became_active":"2024-09-18T08:08:52.696300+0000","last_became_peered":"2024-09-18T08:08:52.696300+0000","last_unstale":"2024-09-18T08:09:34.403168+0000","last_undegraded":"2024-09-18T08:09:34.403168+0000","last_fullsized":"2024-09-18T08:09:34.403168+0000","mapping_epoch":198,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":199,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:04:06.709631+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_clean_scrub_stamp":"2024-09-18T08:04:06.709631+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"8.0","version":"213'6902","reported_seq":7009,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-09-18T08:09:38.788074+0000","last_change":"2024-09-18T08:07:53.329182+0000","last_active":"2024-09-18T08:09:38.788074+0000","last_peered":"2024-09-18T08:09:38.788074+0000","last_clean":"2024-09-18T08:09:38.788074+0000","last_became_active":"2024-09-18T08:07:53.328914+0000","last_became_peered":"2024-09-18T08:07:53.328914+0000","last_unstale":"2024-09-18T08:09:38.788074+0000","last_undegraded":"2024-09-18T08:09:38.788074+0000","last_fullsized":"2024-09-18T08:09:38.788074+0000","mapping_epoch":165,"log_start":"183'2500","ondisk_log_start":"183'2500","created":165,"last_epoch_clean":166,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:07:52.307205+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:07:52.307205+0000","last_clean_scrub_stamp":"2024-09-18T08:07:52.307205+0000","objects_scrubbed":0,"log_size":4402,"log_dups_size":0,"ondisk_log_size":4402,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":370147328,"num_objects":5648,"num_object_clones":0,"num_object_copies":16944,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5648,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6902,"num_write_kb":401600,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.9","version":"74'79","reported_seq":450,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-09-18T08:09:34.408503+0000","last_change":"2024-09-18T08:09:27.676247+0000","last_active":"2024-09-18T08:09:34.408503+0000","last_peered":"2024-09-18T08:09:34.408503+0000","last_clean":"2024-09-18T08:09:34.408503+0000","last_became_active":"2024-09-18T08:08:52.691492+0000","last_became_peered":"2024-09-18T08:08:52.691492+0000","last_unstale":"2024-09-18T08:09:34.408503+0000","last_undegraded":"2024-09-18T08:09:34.408503+0000","last_fullsized":"2024-09-18T08:09:34.408503+0000","mapping_epoch":198,"log_start":"0'0","ondisk_log_start":"0'0","created":152,"last_epoch_clean":199,"parent":"0.0","parent_split_bits":4,"last_scrub":"74'79","last_scrub_stamp":"2024-09-18T08:09:27.676169+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:00:11.893742+0000","last_clean_scrub_stamp":"2024-09-18T08:09:27.676169+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:10:36.422152+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,2],"acting":[7,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.c","version":"69'10","reported_seq":86,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-09-18T08:09:34.394520+0000","last_change":"2024-09-18T08:08:18.642452+0000","last_active":"2024-09-18T08:09:34.394520+0000","last_peered":"2024-09-18T08:09:34.394520+0000","last_clean":"2024-09-18T08:09:34.394520+0000","last_became_active":"2024-09-18T08:08:18.622980+0000","last_became_peered":"2024-09-18T08:08:18.622980+0000","last_unstale":"2024-09-18T08:09:34.394520+0000","last_undegraded":"2024-09-18T08:09:34.394520+0000","last_fullsized":"2024-09-18T08:09:34.394520+0000","mapping_epoch":181,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":182,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'10","last_scrub_stamp":"2024-09-18T08:05:41.474813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_clean_scrub_stamp":"2024-09-18T08:05:41.474813+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":14,"num_read_kb":9,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":347,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-09-18T08:09:34.398464+0000","last_change":"2024-09-18T08:08:18.542800+0000","last_active":"2024-09-18T08:09:34.398464+0000","last_peered":"2024-09-18T08:09:34.398464+0000","last_clean":"2024-09-18T08:09:34.398464+0000","last_became_active":"2024-09-18T08:08:18.542424+0000","last_became_peered":"2024-09-18T08:08:18.542424+0000","last_unstale":"2024-09-18T08:09:34.398464+0000","last_undegraded":"2024-09-18T08:09:34.398464+0000","last_fullsized":"2024-09-18T08:09:34.398464+0000","mapping_epoch":181,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":182,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:06:46.339428+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:46.860092+0000","last_clean_scrub_stamp":"2024-09-18T08:06:46.339428+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.d","version":"69'8","reported_seq":416,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-09-18T08:09:38.011807+0000","last_change":"2024-09-18T08:05:33.027988+0000","last_active":"2024-09-18T08:09:38.011807+0000","last_peered":"2024-09-18T08:09:38.011807+0000","last_clean":"2024-09-18T08:09:38.011807+0000","last_became_active":"2024-09-18T08:02:51.909011+0000","last_became_peered":"2024-09-18T08:02:51.909011+0000","last_unstale":"2024-09-18T08:09:38.011807+0000","last_undegraded":"2024-09-18T08:09:38.011807+0000","last_fullsized":"2024-09-18T08:09:38.011807+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'8","last_scrub_stamp":"2024-09-18T08:05:33.027783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:50.886830+0000","last_clean_scrub_stamp":"2024-09-18T08:05:33.027783+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":81,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-09-18T08:09:34.395228+0000","last_change":"2024-09-18T08:08:52.685954+0000","last_active":"2024-09-18T08:09:34.395228+0000","last_peered":"2024-09-18T08:09:34.395228+0000","last_clean":"2024-09-18T08:09:34.395228+0000","last_became_active":"2024-09-18T08:08:52.685603+0000","last_became_peered":"2024-09-18T08:08:52.685603+0000","last_unstale":"2024-09-18T08:09:34.395228+0000","last_undegraded":"2024-09-18T08:09:34.395228+0000","last_fullsized":"2024-09-18T08:09:34.395228+0000","mapping_epoch":198,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":199,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:07:16.559597+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_clean_scrub_stamp":"2024-09-18T08:07:16.559597+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.7","version":"213'6956","reported_seq":7132,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-09-18T08:09:38.520503+0000","last_change":"2024-09-18T08:09:36.764450+0000","last_active":"2024-09-18T08:09:38.520503+0000","last_peered":"2024-09-18T08:09:38.520503+0000","last_clean":"2024-09-18T08:09:38.520503+0000","last_became_active":"2024-09-18T08:07:53.674433+0000","last_became_peered":"2024-09-18T08:07:53.674433+0000","last_unstale":"2024-09-18T08:09:38.520503+0000","last_undegraded":"2024-09-18T08:09:38.520503+0000","last_fullsized":"2024-09-18T08:09:38.520503+0000","mapping_epoch":165,"log_start":"180'2100","ondisk_log_start":"180'2100","created":165,"last_epoch_clean":166,"parent":"0.0","parent_split_bits":0,"last_scrub":"213'6848","last_scrub_stamp":"2024-09-18T08:09:36.764206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:07:52.307205+0000","last_clean_scrub_stamp":"2024-09-18T08:09:36.764206+0000","objects_scrubbed":5822,"log_size":4856,"log_dups_size":0,"ondisk_log_size":4856,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":4,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:11:06.120132+0000","scrub_duration":3143,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":368181248,"num_objects":5618,"num_object_clones":0,"num_object_copies":16854,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5618,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6956,"num_write_kb":402368,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,6],"acting":[1,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.b","version":"69'9","reported_seq":378,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-09-18T08:09:34.405085+0000","last_change":"2024-09-18T08:08:18.640380+0000","last_active":"2024-09-18T08:09:34.405085+0000","last_peered":"2024-09-18T08:09:34.405085+0000","last_clean":"2024-09-18T08:09:34.405085+0000","last_became_active":"2024-09-18T08:08:18.522995+0000","last_became_peered":"2024-09-18T08:08:18.522995+0000","last_unstale":"2024-09-18T08:09:34.405085+0000","last_undegraded":"2024-09-18T08:09:34.405085+0000","last_fullsized":"2024-09-18T08:09:34.405085+0000","mapping_epoch":181,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":182,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'9","last_scrub_stamp":"2024-09-18T08:08:18.640292+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_clean_scrub_stamp":"2024-09-18T08:08:18.640292+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:09:38.962419+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":346,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-09-18T08:09:34.397185+0000","last_change":"2024-09-18T08:07:40.708363+0000","last_active":"2024-09-18T08:09:34.397185+0000","last_peered":"2024-09-18T08:09:34.397185+0000","last_clean":"2024-09-18T08:09:34.397185+0000","last_became_active":"2024-09-18T08:06:24.135005+0000","last_became_peered":"2024-09-18T08:06:24.135005+0000","last_unstale":"2024-09-18T08:09:34.397185+0000","last_undegraded":"2024-09-18T08:09:34.397185+0000","last_fullsized":"2024-09-18T08:09:34.397185+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:07:40.708307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:46.860092+0000","last_clean_scrub_stamp":"2024-09-18T08:07:40.708307+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":347,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-09-18T08:09:34.402070+0000","last_change":"2024-09-18T08:08:18.509651+0000","last_active":"2024-09-18T08:09:34.402070+0000","last_peered":"2024-09-18T08:09:34.402070+0000","last_clean":"2024-09-18T08:09:34.402070+0000","last_became_active":"2024-09-18T08:08:18.509247+0000","last_became_peered":"2024-09-18T08:08:18.509247+0000","last_unstale":"2024-09-18T08:09:34.402070+0000","last_undegraded":"2024-09-18T08:09:34.402070+0000","last_fullsized":"2024-09-18T08:09:34.402070+0000","mapping_epoch":181,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":182,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:07:10.728679+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:50.886830+0000","last_clean_scrub_stamp":"2024-09-18T08:07:10.728679+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,3],"acting":[2,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":343,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-09-18T08:09:34.404817+0000","last_change":"2024-09-18T08:09:28.086782+0000","last_active":"2024-09-18T08:09:34.404817+0000","last_peered":"2024-09-18T08:09:34.404817+0000","last_clean":"2024-09-18T08:09:34.404817+0000","last_became_active":"2024-09-18T08:08:52.697337+0000","last_became_peered":"2024-09-18T08:08:52.697337+0000","last_unstale":"2024-09-18T08:09:34.404817+0000","last_undegraded":"2024-09-18T08:09:34.404817+0000","last_fullsized":"2024-09-18T08:09:34.404817+0000","mapping_epoch":198,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":199,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:09:28.086727+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_clean_scrub_stamp":"2024-09-18T08:09:28.086727+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:10:33.905647+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.6","version":"213'6903","reported_seq":7010,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-09-18T08:09:38.768322+0000","last_change":"2024-09-18T08:07:53.674440+0000","last_active":"2024-09-18T08:09:38.768322+0000","last_peered":"2024-09-18T08:09:38.768322+0000","last_clean":"2024-09-18T08:09:38.768322+0000","last_became_active":"2024-09-18T08:07:53.674139+0000","last_became_peered":"2024-09-18T08:07:53.674139+0000","last_unstale":"2024-09-18T08:09:38.768322+0000","last_undegraded":"2024-09-18T08:09:38.768322+0000","last_fullsized":"2024-09-18T08:09:38.768322+0000","mapping_epoch":165,"log_start":"177'1500","ondisk_log_start":"177'1500","created":165,"last_epoch_clean":166,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:07:52.307205+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:07:52.307205+0000","last_clean_scrub_stamp":"2024-09-18T08:07:52.307205+0000","objects_scrubbed":0,"log_size":5403,"log_dups_size":0,"ondisk_log_size":5403,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":368902144,"num_objects":5629,"num_object_clones":0,"num_object_copies":16887,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5629,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6903,"num_write_kb":401024,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,7],"acting":[0,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.a","version":"69'19","reported_seq":349,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-09-18T08:09:34.410056+0000","last_change":"2024-09-18T08:07:41.000210+0000","last_active":"2024-09-18T08:09:34.410056+0000","last_peered":"2024-09-18T08:09:34.410056+0000","last_clean":"2024-09-18T08:09:34.410056+0000","last_became_active":"2024-09-18T08:06:24.139909+0000","last_became_peered":"2024-09-18T08:06:24.139909+0000","last_unstale":"2024-09-18T08:09:34.410056+0000","last_undegraded":"2024-09-18T08:09:34.410056+0000","last_fullsized":"2024-09-18T08:09:34.410056+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'19","last_scrub_stamp":"2024-09-18T08:07:41.000146+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_clean_scrub_stamp":"2024-09-18T08:07:41.000146+0000","objects_scrubbed":9,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:10:18.661451+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":31,"num_read_kb":20,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,7],"acting":[1,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":334,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-09-18T08:09:34.396686+0000","last_change":"2024-09-18T08:06:47.638974+0000","last_active":"2024-09-18T08:09:34.396686+0000","last_peered":"2024-09-18T08:09:34.396686+0000","last_clean":"2024-09-18T08:09:34.396686+0000","last_became_active":"2024-09-18T08:06:46.171298+0000","last_became_peered":"2024-09-18T08:06:46.171298+0000","last_unstale":"2024-09-18T08:09:34.396686+0000","last_undegraded":"2024-09-18T08:09:34.396686+0000","last_fullsized":"2024-09-18T08:09:34.396686+0000","mapping_epoch":143,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":144,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:06:47.638911+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:46.860092+0000","last_clean_scrub_stamp":"2024-09-18T08:06:47.638911+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":40,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-09-18T08:09:34.399802+0000","last_change":"2024-09-18T08:08:52.698243+0000","last_active":"2024-09-18T08:09:34.399802+0000","last_peered":"2024-09-18T08:09:34.399802+0000","last_clean":"2024-09-18T08:09:34.399802+0000","last_became_active":"2024-09-18T08:08:52.697932+0000","last_became_peered":"2024-09-18T08:08:52.697932+0000","last_unstale":"2024-09-18T08:09:34.399802+0000","last_undegraded":"2024-09-18T08:09:34.399802+0000","last_fullsized":"2024-09-18T08:09:34.399802+0000","mapping_epoch":198,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":199,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:08:16.631007+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:50.886830+0000","last_clean_scrub_stamp":"2024-09-18T08:08:16.631007+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":160,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-09-18T08:09:34.397245+0000","last_change":"2024-09-18T08:06:50.902917+0000","last_active":"2024-09-18T08:09:34.397245+0000","last_peered":"2024-09-18T08:09:34.397245+0000","last_clean":"2024-09-18T08:09:34.397245+0000","last_became_active":"2024-09-18T08:06:46.137872+0000","last_became_peered":"2024-09-18T08:06:46.137872+0000","last_unstale":"2024-09-18T08:09:34.397245+0000","last_undegraded":"2024-09-18T08:09:34.397245+0000","last_fullsized":"2024-09-18T08:09:34.397245+0000","mapping_epoch":143,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":144,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:06:50.902835+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_clean_scrub_stamp":"2024-09-18T08:06:50.902835+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,3],"acting":[2,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"8.5","version":"213'7307","reported_seq":11727,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-09-18T08:09:43.101497+0000","last_change":"2024-09-18T08:09:13.170209+0000","last_active":"2024-09-18T08:09:43.101497+0000","last_peered":"2024-09-18T08:09:43.101497+0000","last_clean":"2024-09-18T08:09:43.101497+0000","last_became_active":"2024-09-18T08:09:13.166459+0000","last_became_peered":"2024-09-18T08:09:13.166459+0000","last_unstale":"2024-09-18T08:09:43.101497+0000","last_undegraded":"2024-09-18T08:09:43.101497+0000","last_fullsized":"2024-09-18T08:09:43.101497+0000","mapping_epoch":205,"log_start":"184'2774","ondisk_log_start":"184'2774","created":165,"last_epoch_clean":206,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:07:52.307205+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:07:52.307205+0000","last_clean_scrub_stamp":"2024-09-18T08:07:52.307205+0000","objects_scrubbed":0,"log_size":4533,"log_dups_size":0,"ondisk_log_size":4533,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":346226688,"num_objects":5283,"num_object_clones":0,"num_object_copies":15849,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5283,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7307,"num_write_kb":402880,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2146,"num_bytes_recovered":140640256,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,7],"acting":[4,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.9","version":"69'12","reported_seq":86,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-09-18T08:09:34.393344+0000","last_change":"2024-09-18T08:08:18.599669+0000","last_active":"2024-09-18T08:09:34.393344+0000","last_peered":"2024-09-18T08:09:34.393344+0000","last_clean":"2024-09-18T08:09:34.393344+0000","last_became_active":"2024-09-18T08:08:18.544489+0000","last_became_peered":"2024-09-18T08:08:18.544489+0000","last_unstale":"2024-09-18T08:09:34.393344+0000","last_undegraded":"2024-09-18T08:09:34.393344+0000","last_fullsized":"2024-09-18T08:09:34.393344+0000","mapping_epoch":181,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":182,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'12","last_scrub_stamp":"2024-09-18T08:05:33.544135+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_clean_scrub_stamp":"2024-09-18T08:05:33.544135+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":24,"num_read_kb":15,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":362,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-09-18T08:09:39.614251+0000","last_change":"2024-09-18T08:09:39.614251+0000","last_active":"2024-09-18T08:09:39.614251+0000","last_peered":"2024-09-18T08:09:39.614251+0000","last_clean":"2024-09-18T08:09:39.614251+0000","last_became_active":"2024-09-18T08:08:18.539573+0000","last_became_peered":"2024-09-18T08:08:18.539573+0000","last_unstale":"2024-09-18T08:09:39.614251+0000","last_undegraded":"2024-09-18T08:09:39.614251+0000","last_fullsized":"2024-09-18T08:09:39.614251+0000","mapping_epoch":181,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":182,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:09:39.614187+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:46.860092+0000","last_clean_scrub_stamp":"2024-09-18T08:09:39.614187+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:10:57.546531+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":325,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-09-18T08:09:34.397353+0000","last_change":"2024-09-18T08:08:15.744330+0000","last_active":"2024-09-18T08:09:34.397353+0000","last_peered":"2024-09-18T08:09:34.397353+0000","last_clean":"2024-09-18T08:09:34.397353+0000","last_became_active":"2024-09-18T08:02:51.902163+0000","last_became_peered":"2024-09-18T08:02:51.902163+0000","last_unstale":"2024-09-18T08:09:34.397353+0000","last_undegraded":"2024-09-18T08:09:34.397353+0000","last_fullsized":"2024-09-18T08:09:34.397353+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:08:15.744245+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:50.886830+0000","last_clean_scrub_stamp":"2024-09-18T08:08:15.744245+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:09:40.197923+0000","scrub_duration":74,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":322,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-09-18T08:09:34.398181+0000","last_change":"2024-09-18T08:07:44.113695+0000","last_active":"2024-09-18T08:09:34.398181+0000","last_peered":"2024-09-18T08:09:34.398181+0000","last_clean":"2024-09-18T08:09:34.398181+0000","last_became_active":"2024-09-18T08:02:53.907412+0000","last_became_peered":"2024-09-18T08:02:53.907412+0000","last_unstale":"2024-09-18T08:09:34.398181+0000","last_undegraded":"2024-09-18T08:09:34.398181+0000","last_fullsized":"2024-09-18T08:09:34.398181+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:07:44.113639+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_clean_scrub_stamp":"2024-09-18T08:07:44.113639+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.4","version":"213'6880","reported_seq":16806,"reported_epoch":213,"state":"active+recovering+undersized+remapped","last_fresh":"2024-09-18T08:09:38.219342+0000","last_change":"2024-09-18T08:08:56.896339+0000","last_active":"2024-09-18T08:09:38.219342+0000","last_peered":"2024-09-18T08:09:38.219342+0000","last_clean":"2024-09-18T08:08:51.672524+0000","last_became_active":"2024-09-18T08:08:53.828580+0000","last_became_peered":"2024-09-18T08:08:53.828580+0000","last_unstale":"2024-09-18T08:09:38.219342+0000","last_undegraded":"2024-09-18T08:09:38.219342+0000","last_fullsized":"2024-09-18T08:08:53.682963+0000","mapping_epoch":199,"log_start":"177'1700","ondisk_log_start":"177'1700","created":165,"last_epoch_clean":190,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:07:52.307205+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:07:52.307205+0000","last_clean_scrub_stamp":"2024-09-18T08:07:52.307205+0000","objects_scrubbed":0,"log_size":5180,"log_dups_size":0,"ondisk_log_size":5180,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":372899840,"num_objects":5690,"num_object_clones":0,"num_object_copies":17070,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":2211,"num_objects_unfound":0,"num_objects_dirty":5690,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6880,"num_write_kb":402240,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4896,"num_bytes_recovered":320864256,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,2],"acting":[5,4],"avail_no_missing":["5","3","4","6"],"object_location_counts":[{"shards":"4,5","objects":3479},{"shards":"4,5,6","objects":2211}],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.8","version":"69'15","reported_seq":391,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-09-18T08:09:34.395980+0000","last_change":"2024-09-18T08:06:24.162625+0000","last_active":"2024-09-18T08:09:34.395980+0000","last_peered":"2024-09-18T08:09:34.395980+0000","last_clean":"2024-09-18T08:09:34.395980+0000","last_became_active":"2024-09-18T08:06:24.138259+0000","last_became_peered":"2024-09-18T08:06:24.138259+0000","last_unstale":"2024-09-18T08:09:34.395980+0000","last_undegraded":"2024-09-18T08:09:34.395980+0000","last_fullsized":"2024-09-18T08:09:34.395980+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'15","last_scrub_stamp":"2024-09-18T08:04:26.859044+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_clean_scrub_stamp":"2024-09-18T08:04:26.859044+0000","objects_scrubbed":7,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.f","version":"62'2","reported_seq":378,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-09-18T08:09:34.409408+0000","last_change":"2024-09-18T08:08:18.629506+0000","last_active":"2024-09-18T08:09:34.409408+0000","last_peered":"2024-09-18T08:09:34.409408+0000","last_clean":"2024-09-18T08:09:34.409408+0000","last_became_active":"2024-09-18T08:08:18.524593+0000","last_became_peered":"2024-09-18T08:08:18.524593+0000","last_unstale":"2024-09-18T08:09:34.409408+0000","last_undegraded":"2024-09-18T08:09:34.409408+0000","last_fullsized":"2024-09-18T08:09:34.409408+0000","mapping_epoch":181,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":182,"parent":"0.0","parent_split_bits":0,"last_scrub":"62'2","last_scrub_stamp":"2024-09-18T08:07:43.540222+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:46.860092+0000","last_clean_scrub_stamp":"2024-09-18T08:07:43.540222+0000","objects_scrubbed":2,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:09:45.671982+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":4,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":184,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.9","version":"182'12","reported_seq":472,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-09-18T08:09:38.010195+0000","last_change":"2024-09-18T08:08:18.570029+0000","last_active":"2024-09-18T08:09:38.010195+0000","last_peered":"2024-09-18T08:09:38.010195+0000","last_clean":"2024-09-18T08:09:38.010195+0000","last_became_active":"2024-09-18T08:08:18.524858+0000","last_became_peered":"2024-09-18T08:08:18.524858+0000","last_unstale":"2024-09-18T08:09:38.010195+0000","last_undegraded":"2024-09-18T08:09:38.010195+0000","last_fullsized":"2024-09-18T08:09:38.010195+0000","mapping_epoch":181,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":182,"parent":"0.0","parent_split_bits":0,"last_scrub":"138'10","last_scrub_stamp":"2024-09-18T08:07:23.729715+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:50.886830+0000","last_clean_scrub_stamp":"2024-09-18T08:07:23.729715+0000","objects_scrubbed":1,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":334,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-09-18T08:09:34.396115+0000","last_change":"2024-09-18T08:07:45.707631+0000","last_active":"2024-09-18T08:09:34.396115+0000","last_peered":"2024-09-18T08:09:34.396115+0000","last_clean":"2024-09-18T08:09:34.396115+0000","last_became_active":"2024-09-18T08:06:24.133119+0000","last_became_peered":"2024-09-18T08:06:24.133119+0000","last_unstale":"2024-09-18T08:09:34.396115+0000","last_undegraded":"2024-09-18T08:09:34.396115+0000","last_fullsized":"2024-09-18T08:09:34.396115+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:07:45.707562+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_clean_scrub_stamp":"2024-09-18T08:07:45.707562+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":250,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-09-18T08:09:34.404149+0000","last_change":"2024-09-18T08:06:24.133955+0000","last_active":"2024-09-18T08:09:34.404149+0000","last_peered":"2024-09-18T08:09:34.404149+0000","last_clean":"2024-09-18T08:09:34.404149+0000","last_became_active":"2024-09-18T08:06:24.133644+0000","last_became_peered":"2024-09-18T08:06:24.133644+0000","last_unstale":"2024-09-18T08:09:34.404149+0000","last_undegraded":"2024-09-18T08:09:34.404149+0000","last_fullsized":"2024-09-18T08:09:34.404149+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:04:16.378205+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:46.860092+0000","last_clean_scrub_stamp":"2024-09-18T08:04:16.378205+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:10:20.052218+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.17","version":"69'6","reported_seq":345,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-09-18T08:09:34.394693+0000","last_change":"2024-09-18T08:07:19.309941+0000","last_active":"2024-09-18T08:09:34.394693+0000","last_peered":"2024-09-18T08:09:34.394693+0000","last_clean":"2024-09-18T08:09:34.394693+0000","last_became_active":"2024-09-18T08:02:49.895067+0000","last_became_peered":"2024-09-18T08:02:49.895067+0000","last_unstale":"2024-09-18T08:09:34.394693+0000","last_undegraded":"2024-09-18T08:09:34.394693+0000","last_fullsized":"2024-09-18T08:09:34.394693+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'6","last_scrub_stamp":"2024-09-18T08:07:19.309844+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_clean_scrub_stamp":"2024-09-18T08:07:19.309844+0000","objects_scrubbed":3,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":26,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":9,"num_read_kb":6,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":323,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-09-18T08:09:34.396336+0000","last_change":"2024-09-18T08:05:45.812436+0000","last_active":"2024-09-18T08:09:34.396336+0000","last_peered":"2024-09-18T08:09:34.396336+0000","last_clean":"2024-09-18T08:09:34.396336+0000","last_became_active":"2024-09-18T08:02:51.912557+0000","last_became_peered":"2024-09-18T08:02:51.912557+0000","last_unstale":"2024-09-18T08:09:34.396336+0000","last_undegraded":"2024-09-18T08:09:34.396336+0000","last_fullsized":"2024-09-18T08:09:34.396336+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:05:45.812389+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:50.886830+0000","last_clean_scrub_stamp":"2024-09-18T08:05:45.812389+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,1],"acting":[5,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":356,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-09-18T08:09:34.408407+0000","last_change":"2024-09-18T08:08:18.526113+0000","last_active":"2024-09-18T08:09:34.408407+0000","last_peered":"2024-09-18T08:09:34.408407+0000","last_clean":"2024-09-18T08:09:34.408407+0000","last_became_active":"2024-09-18T08:08:18.525260+0000","last_became_peered":"2024-09-18T08:08:18.525260+0000","last_unstale":"2024-09-18T08:09:34.408407+0000","last_undegraded":"2024-09-18T08:09:34.408407+0000","last_fullsized":"2024-09-18T08:09:34.408407+0000","mapping_epoch":181,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":182,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:07:53.670107+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_clean_scrub_stamp":"2024-09-18T08:07:53.670107+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:10:19.732636+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"69'9","reported_seq":343,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-09-18T08:09:34.406049+0000","last_change":"2024-09-18T08:07:23.638373+0000","last_active":"2024-09-18T08:09:34.406049+0000","last_peered":"2024-09-18T08:09:34.406049+0000","last_clean":"2024-09-18T08:09:34.406049+0000","last_became_active":"2024-09-18T08:02:49.891567+0000","last_became_peered":"2024-09-18T08:02:49.891567+0000","last_unstale":"2024-09-18T08:09:34.406049+0000","last_undegraded":"2024-09-18T08:09:34.406049+0000","last_fullsized":"2024-09-18T08:09:34.406049+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'9","last_scrub_stamp":"2024-09-18T08:07:23.638294+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_clean_scrub_stamp":"2024-09-18T08:07:23.638294+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":74,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-09-18T08:09:34.394399+0000","last_change":"2024-09-18T08:08:18.550531+0000","last_active":"2024-09-18T08:09:34.394399+0000","last_peered":"2024-09-18T08:09:34.394399+0000","last_clean":"2024-09-18T08:09:34.394399+0000","last_became_active":"2024-09-18T08:08:18.545953+0000","last_became_peered":"2024-09-18T08:08:18.545953+0000","last_unstale":"2024-09-18T08:09:34.394399+0000","last_undegraded":"2024-09-18T08:09:34.394399+0000","last_fullsized":"2024-09-18T08:09:34.394399+0000","mapping_epoch":181,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":182,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:06:13.214705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:46.860092+0000","last_clean_scrub_stamp":"2024-09-18T08:06:13.214705+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":316,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-09-18T08:09:34.394479+0000","last_change":"2024-09-18T08:05:42.061649+0000","last_active":"2024-09-18T08:09:34.394479+0000","last_peered":"2024-09-18T08:09:34.394479+0000","last_clean":"2024-09-18T08:09:34.394479+0000","last_became_active":"2024-09-18T08:02:51.907361+0000","last_became_peered":"2024-09-18T08:02:51.907361+0000","last_unstale":"2024-09-18T08:09:34.394479+0000","last_undegraded":"2024-09-18T08:09:34.394479+0000","last_fullsized":"2024-09-18T08:09:34.394479+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:05:42.061452+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:50.886830+0000","last_clean_scrub_stamp":"2024-09-18T08:05:42.061452+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":335,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-09-18T08:09:34.401048+0000","last_change":"2024-09-18T08:08:18.511719+0000","last_active":"2024-09-18T08:09:34.401048+0000","last_peered":"2024-09-18T08:09:34.401048+0000","last_clean":"2024-09-18T08:09:34.401048+0000","last_became_active":"2024-09-18T08:08:18.511019+0000","last_became_peered":"2024-09-18T08:08:18.511019+0000","last_unstale":"2024-09-18T08:09:34.401048+0000","last_undegraded":"2024-09-18T08:09:34.401048+0000","last_fullsized":"2024-09-18T08:09:34.401048+0000","mapping_epoch":181,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":182,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:08:10.660508+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_clean_scrub_stamp":"2024-09-18T08:08:10.660508+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.15","version":"69'9","reported_seq":343,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-09-18T08:09:34.396784+0000","last_change":"2024-09-18T08:05:36.896299+0000","last_active":"2024-09-18T08:09:34.396784+0000","last_peered":"2024-09-18T08:09:34.396784+0000","last_clean":"2024-09-18T08:09:34.396784+0000","last_became_active":"2024-09-18T08:02:49.890497+0000","last_became_peered":"2024-09-18T08:02:49.890497+0000","last_unstale":"2024-09-18T08:09:34.396784+0000","last_undegraded":"2024-09-18T08:09:34.396784+0000","last_fullsized":"2024-09-18T08:09:34.396784+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'9","last_scrub_stamp":"2024-09-18T08:05:36.896238+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_clean_scrub_stamp":"2024-09-18T08:05:36.896238+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":339,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-09-18T08:09:34.404537+0000","last_change":"2024-09-18T08:07:48.653905+0000","last_active":"2024-09-18T08:09:34.404537+0000","last_peered":"2024-09-18T08:09:34.404537+0000","last_clean":"2024-09-18T08:09:34.404537+0000","last_became_active":"2024-09-18T08:02:47.875983+0000","last_became_peered":"2024-09-18T08:02:47.875983+0000","last_unstale":"2024-09-18T08:09:34.404537+0000","last_undegraded":"2024-09-18T08:09:34.404537+0000","last_fullsized":"2024-09-18T08:09:34.404537+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:07:48.653856+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:46.860092+0000","last_clean_scrub_stamp":"2024-09-18T08:07:48.653856+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.14","version":"199'12","reported_seq":439,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-09-18T08:09:34.394016+0000","last_change":"2024-09-18T08:08:52.701287+0000","last_active":"2024-09-18T08:09:34.394016+0000","last_peered":"2024-09-18T08:09:34.394016+0000","last_clean":"2024-09-18T08:09:34.394016+0000","last_became_active":"2024-09-18T08:08:52.696201+0000","last_became_peered":"2024-09-18T08:08:52.696201+0000","last_unstale":"2024-09-18T08:09:34.394016+0000","last_undegraded":"2024-09-18T08:09:34.394016+0000","last_fullsized":"2024-09-18T08:09:34.394016+0000","mapping_epoch":198,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":199,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'8","last_scrub_stamp":"2024-09-18T08:07:16.241607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:50.886830+0000","last_clean_scrub_stamp":"2024-09-18T08:07:16.241607+0000","objects_scrubbed":1,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":74,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-09-18T08:09:34.394195+0000","last_change":"2024-09-18T08:08:18.523205+0000","last_active":"2024-09-18T08:09:34.394195+0000","last_peered":"2024-09-18T08:09:34.394195+0000","last_clean":"2024-09-18T08:09:34.394195+0000","last_became_active":"2024-09-18T08:08:18.522483+0000","last_became_peered":"2024-09-18T08:08:18.522483+0000","last_unstale":"2024-09-18T08:09:34.394195+0000","last_undegraded":"2024-09-18T08:09:34.394195+0000","last_fullsized":"2024-09-18T08:09:34.394195+0000","mapping_epoch":181,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":182,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:06:43.510513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_clean_scrub_stamp":"2024-09-18T08:06:43.510513+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.14","version":"69'10","reported_seq":362,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-09-18T08:09:34.394989+0000","last_change":"2024-09-18T08:08:08.493614+0000","last_active":"2024-09-18T08:09:34.394989+0000","last_peered":"2024-09-18T08:09:34.394989+0000","last_clean":"2024-09-18T08:09:34.394989+0000","last_became_active":"2024-09-18T08:04:18.150968+0000","last_became_peered":"2024-09-18T08:04:18.150968+0000","last_unstale":"2024-09-18T08:09:34.394989+0000","last_undegraded":"2024-09-18T08:09:34.394989+0000","last_fullsized":"2024-09-18T08:09:34.394989+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'10","last_scrub_stamp":"2024-09-18T08:08:08.493522+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_clean_scrub_stamp":"2024-09-18T08:08:08.493522+0000","objects_scrubbed":5,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,7],"acting":[3,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":353,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-09-18T08:09:34.409609+0000","last_change":"2024-09-18T08:09:31.658337+0000","last_active":"2024-09-18T08:09:34.409609+0000","last_peered":"2024-09-18T08:09:34.409609+0000","last_clean":"2024-09-18T08:09:34.409609+0000","last_became_active":"2024-09-18T08:08:18.524218+0000","last_became_peered":"2024-09-18T08:08:18.524218+0000","last_unstale":"2024-09-18T08:09:34.409609+0000","last_undegraded":"2024-09-18T08:09:34.409609+0000","last_fullsized":"2024-09-18T08:09:34.409609+0000","mapping_epoch":181,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":182,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:09:31.658259+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:46.860092+0000","last_clean_scrub_stamp":"2024-09-18T08:09:31.658259+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:10:50.143994+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.15","version":"69'8","reported_seq":422,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-09-18T08:09:34.396576+0000","last_change":"2024-09-18T08:06:45.555513+0000","last_active":"2024-09-18T08:09:34.396576+0000","last_peered":"2024-09-18T08:09:34.396576+0000","last_clean":"2024-09-18T08:09:34.396576+0000","last_became_active":"2024-09-18T08:02:51.914135+0000","last_became_peered":"2024-09-18T08:02:51.914135+0000","last_unstale":"2024-09-18T08:09:34.396576+0000","last_undegraded":"2024-09-18T08:09:34.396576+0000","last_fullsized":"2024-09-18T08:09:34.396576+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'8","last_scrub_stamp":"2024-09-18T08:06:45.555459+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:50.886830+0000","last_clean_scrub_stamp":"2024-09-18T08:06:45.555459+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":320,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-09-18T08:09:34.405510+0000","last_change":"2024-09-18T08:07:14.805012+0000","last_active":"2024-09-18T08:09:34.405510+0000","last_peered":"2024-09-18T08:09:34.405510+0000","last_clean":"2024-09-18T08:09:34.405510+0000","last_became_active":"2024-09-18T08:02:53.912112+0000","last_became_peered":"2024-09-18T08:02:53.912112+0000","last_unstale":"2024-09-18T08:09:34.405510+0000","last_undegraded":"2024-09-18T08:09:34.405510+0000","last_fullsized":"2024-09-18T08:09:34.405510+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:07:14.804953+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_clean_scrub_stamp":"2024-09-18T08:07:14.804953+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":46,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.13","version":"69'11","reported_seq":86,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-09-18T08:09:34.393829+0000","last_change":"2024-09-18T08:08:18.643772+0000","last_active":"2024-09-18T08:09:34.393829+0000","last_peered":"2024-09-18T08:09:34.393829+0000","last_clean":"2024-09-18T08:09:34.393829+0000","last_became_active":"2024-09-18T08:08:18.539712+0000","last_became_peered":"2024-09-18T08:08:18.539712+0000","last_unstale":"2024-09-18T08:09:34.393829+0000","last_undegraded":"2024-09-18T08:09:34.393829+0000","last_fullsized":"2024-09-18T08:09:34.393829+0000","mapping_epoch":181,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":182,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'11","last_scrub_stamp":"2024-09-18T08:07:53.824401+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_clean_scrub_stamp":"2024-09-18T08:07:53.824401+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":19,"num_read_kb":12,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":74,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-09-18T08:09:34.394922+0000","last_change":"2024-09-18T08:08:18.623643+0000","last_active":"2024-09-18T08:09:34.394922+0000","last_peered":"2024-09-18T08:09:34.394922+0000","last_clean":"2024-09-18T08:09:34.394922+0000","last_became_active":"2024-09-18T08:08:18.622893+0000","last_became_peered":"2024-09-18T08:08:18.622893+0000","last_unstale":"2024-09-18T08:09:34.394922+0000","last_undegraded":"2024-09-18T08:09:34.394922+0000","last_fullsized":"2024-09-18T08:09:34.394922+0000","mapping_epoch":181,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":182,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:06:46.502422+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:46.860092+0000","last_clean_scrub_stamp":"2024-09-18T08:06:46.502422+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":331,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-09-18T08:09:34.409979+0000","last_change":"2024-09-18T08:09:03.582984+0000","last_active":"2024-09-18T08:09:34.409979+0000","last_peered":"2024-09-18T08:09:34.409979+0000","last_clean":"2024-09-18T08:09:34.409979+0000","last_became_active":"2024-09-18T08:02:51.910752+0000","last_became_peered":"2024-09-18T08:02:51.910752+0000","last_unstale":"2024-09-18T08:09:34.409979+0000","last_undegraded":"2024-09-18T08:09:34.409979+0000","last_fullsized":"2024-09-18T08:09:34.409979+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:09:03.582914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:50.886830+0000","last_clean_scrub_stamp":"2024-09-18T08:09:03.582914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:10:14.069633+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":329,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-09-18T08:09:34.395894+0000","last_change":"2024-09-18T08:07:54.314026+0000","last_active":"2024-09-18T08:09:34.395894+0000","last_peered":"2024-09-18T08:09:34.395894+0000","last_clean":"2024-09-18T08:09:34.395894+0000","last_became_active":"2024-09-18T08:02:53.911429+0000","last_became_peered":"2024-09-18T08:02:53.911429+0000","last_unstale":"2024-09-18T08:09:34.395894+0000","last_undegraded":"2024-09-18T08:09:34.395894+0000","last_fullsized":"2024-09-18T08:09:34.395894+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:07:54.313971+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_clean_scrub_stamp":"2024-09-18T08:07:54.313971+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.12","version":"69'9","reported_seq":369,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-09-18T08:09:34.410224+0000","last_change":"2024-09-18T08:08:52.658353+0000","last_active":"2024-09-18T08:09:34.410224+0000","last_peered":"2024-09-18T08:09:34.410224+0000","last_clean":"2024-09-18T08:09:34.410224+0000","last_became_active":"2024-09-18T08:06:24.132270+0000","last_became_peered":"2024-09-18T08:06:24.132270+0000","last_unstale":"2024-09-18T08:09:34.410224+0000","last_undegraded":"2024-09-18T08:09:34.410224+0000","last_fullsized":"2024-09-18T08:09:34.410224+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'9","last_scrub_stamp":"2024-09-18T08:08:52.658282+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_clean_scrub_stamp":"2024-09-18T08:08:52.658282+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:10:04.461414+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":361,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-09-18T08:09:37.532792+0000","last_change":"2024-09-18T08:09:37.532792+0000","last_active":"2024-09-18T08:09:37.532792+0000","last_peered":"2024-09-18T08:09:37.532792+0000","last_clean":"2024-09-18T08:09:37.532792+0000","last_became_active":"2024-09-18T08:08:18.525527+0000","last_became_peered":"2024-09-18T08:08:18.525527+0000","last_unstale":"2024-09-18T08:09:37.532792+0000","last_undegraded":"2024-09-18T08:09:37.532792+0000","last_fullsized":"2024-09-18T08:09:37.532792+0000","mapping_epoch":181,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":182,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:09:37.532573+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:46.860092+0000","last_clean_scrub_stamp":"2024-09-18T08:09:37.532573+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:11:05.489513+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":324,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-09-18T08:09:34.398230+0000","last_change":"2024-09-18T08:07:42.165434+0000","last_active":"2024-09-18T08:09:34.398230+0000","last_peered":"2024-09-18T08:09:34.398230+0000","last_clean":"2024-09-18T08:09:34.398230+0000","last_became_active":"2024-09-18T08:02:51.906277+0000","last_became_peered":"2024-09-18T08:02:51.906277+0000","last_unstale":"2024-09-18T08:09:34.398230+0000","last_undegraded":"2024-09-18T08:09:34.398230+0000","last_fullsized":"2024-09-18T08:09:34.398230+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:07:42.165388+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:50.886830+0000","last_clean_scrub_stamp":"2024-09-18T08:07:42.165388+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.10","version":"69'1","reported_seq":308,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-09-18T08:09:34.403774+0000","last_change":"2024-09-18T08:05:51.429631+0000","last_active":"2024-09-18T08:09:34.403774+0000","last_peered":"2024-09-18T08:09:34.403774+0000","last_clean":"2024-09-18T08:09:34.403774+0000","last_became_active":"2024-09-18T08:02:53.912699+0000","last_became_peered":"2024-09-18T08:02:53.912699+0000","last_unstale":"2024-09-18T08:09:34.403774+0000","last_undegraded":"2024-09-18T08:09:34.403774+0000","last_fullsized":"2024-09-18T08:09:34.403774+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_clean_scrub_stamp":"2024-09-18T08:02:52.890475+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:10:16.084523+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.11","version":"69'11","reported_seq":393,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-09-18T08:09:34.396076+0000","last_change":"2024-09-18T08:07:18.282247+0000","last_active":"2024-09-18T08:09:34.396076+0000","last_peered":"2024-09-18T08:09:34.396076+0000","last_clean":"2024-09-18T08:09:34.396076+0000","last_became_active":"2024-09-18T08:06:24.132416+0000","last_became_peered":"2024-09-18T08:06:24.132416+0000","last_unstale":"2024-09-18T08:09:34.396076+0000","last_undegraded":"2024-09-18T08:09:34.396076+0000","last_fullsized":"2024-09-18T08:09:34.396076+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'11","last_scrub_stamp":"2024-09-18T08:07:18.282155+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_clean_scrub_stamp":"2024-09-18T08:07:18.282155+0000","objects_scrubbed":5,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":331,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-09-18T08:09:34.396419+0000","last_change":"2024-09-18T08:06:18.168254+0000","last_active":"2024-09-18T08:09:34.396419+0000","last_peered":"2024-09-18T08:09:34.396419+0000","last_clean":"2024-09-18T08:09:34.396419+0000","last_became_active":"2024-09-18T08:02:47.878886+0000","last_became_peered":"2024-09-18T08:02:47.878886+0000","last_unstale":"2024-09-18T08:09:34.396419+0000","last_undegraded":"2024-09-18T08:09:34.396419+0000","last_fullsized":"2024-09-18T08:09:34.396419+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:06:18.168206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:46.860092+0000","last_clean_scrub_stamp":"2024-09-18T08:06:18.168206+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":359,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-09-18T08:09:34.407989+0000","last_change":"2024-09-18T08:08:18.525683+0000","last_active":"2024-09-18T08:09:34.407989+0000","last_peered":"2024-09-18T08:09:34.407989+0000","last_clean":"2024-09-18T08:09:34.407989+0000","last_became_active":"2024-09-18T08:08:18.524966+0000","last_became_peered":"2024-09-18T08:08:18.524966+0000","last_unstale":"2024-09-18T08:09:34.407989+0000","last_undegraded":"2024-09-18T08:09:34.407989+0000","last_fullsized":"2024-09-18T08:09:34.407989+0000","mapping_epoch":181,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":182,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:07:21.724545+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:50.886830+0000","last_clean_scrub_stamp":"2024-09-18T08:07:21.724545+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:09:43.621477+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":334,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-09-18T08:09:34.398379+0000","last_change":"2024-09-18T08:07:48.218010+0000","last_active":"2024-09-18T08:09:34.398379+0000","last_peered":"2024-09-18T08:09:34.398379+0000","last_clean":"2024-09-18T08:09:34.398379+0000","last_became_active":"2024-09-18T08:06:24.128582+0000","last_became_peered":"2024-09-18T08:06:24.128582+0000","last_unstale":"2024-09-18T08:09:34.398379+0000","last_undegraded":"2024-09-18T08:09:34.398379+0000","last_fullsized":"2024-09-18T08:09:34.398379+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:07:48.217964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_clean_scrub_stamp":"2024-09-18T08:07:48.217964+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.10","version":"69'4","reported_seq":360,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-09-18T08:09:34.394990+0000","last_change":"2024-09-18T08:07:41.210418+0000","last_active":"2024-09-18T08:09:34.394990+0000","last_peered":"2024-09-18T08:09:34.394990+0000","last_clean":"2024-09-18T08:09:34.394990+0000","last_became_active":"2024-09-18T08:06:24.131309+0000","last_became_peered":"2024-09-18T08:06:24.131309+0000","last_unstale":"2024-09-18T08:09:34.394990+0000","last_undegraded":"2024-09-18T08:09:34.394990+0000","last_fullsized":"2024-09-18T08:09:34.394990+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'4","last_scrub_stamp":"2024-09-18T08:07:41.210347+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_clean_scrub_stamp":"2024-09-18T08:07:41.210347+0000","objects_scrubbed":2,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":6,"num_read_kb":4,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":339,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-09-18T08:09:34.404820+0000","last_change":"2024-09-18T08:07:52.551689+0000","last_active":"2024-09-18T08:09:34.404820+0000","last_peered":"2024-09-18T08:09:34.404820+0000","last_clean":"2024-09-18T08:09:34.404820+0000","last_became_active":"2024-09-18T08:02:47.877616+0000","last_became_peered":"2024-09-18T08:02:47.877616+0000","last_unstale":"2024-09-18T08:09:34.404820+0000","last_undegraded":"2024-09-18T08:09:34.404820+0000","last_fullsized":"2024-09-18T08:09:34.404820+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:07:52.551616+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:46.860092+0000","last_clean_scrub_stamp":"2024-09-18T08:07:52.551616+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":74,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-09-18T08:09:34.394578+0000","last_change":"2024-09-18T08:08:18.550244+0000","last_active":"2024-09-18T08:09:34.394578+0000","last_peered":"2024-09-18T08:09:34.394578+0000","last_clean":"2024-09-18T08:09:34.394578+0000","last_became_active":"2024-09-18T08:08:18.545361+0000","last_became_peered":"2024-09-18T08:08:18.545361+0000","last_unstale":"2024-09-18T08:09:34.394578+0000","last_undegraded":"2024-09-18T08:09:34.394578+0000","last_fullsized":"2024-09-18T08:09:34.394578+0000","mapping_epoch":181,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":182,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:07:17.605562+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:50.886830+0000","last_clean_scrub_stamp":"2024-09-18T08:07:17.605562+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,7],"acting":[4,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":350,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-09-18T08:09:34.607443+0000","last_change":"2024-09-18T08:09:34.607443+0000","last_active":"2024-09-18T08:09:34.607443+0000","last_peered":"2024-09-18T08:09:34.607443+0000","last_clean":"2024-09-18T08:09:34.607443+0000","last_became_active":"2024-09-18T08:08:18.525552+0000","last_became_peered":"2024-09-18T08:08:18.525552+0000","last_unstale":"2024-09-18T08:09:34.607443+0000","last_undegraded":"2024-09-18T08:09:34.607443+0000","last_fullsized":"2024-09-18T08:09:34.607443+0000","mapping_epoch":181,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":182,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:09:34.607374+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_clean_scrub_stamp":"2024-09-18T08:09:34.607374+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:10:39.783401+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,4],"acting":[7,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":318,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-09-18T08:09:34.409888+0000","last_change":"2024-09-18T08:08:18.539359+0000","last_active":"2024-09-18T08:09:34.409888+0000","last_peered":"2024-09-18T08:09:34.409888+0000","last_clean":"2024-09-18T08:09:34.409888+0000","last_became_active":"2024-09-18T08:08:18.538838+0000","last_became_peered":"2024-09-18T08:08:18.538838+0000","last_unstale":"2024-09-18T08:09:34.409888+0000","last_undegraded":"2024-09-18T08:09:34.409888+0000","last_fullsized":"2024-09-18T08:09:34.409888+0000","mapping_epoch":181,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":182,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_clean_scrub_stamp":"2024-09-18T08:02:52.890475+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:09:50.664384+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":332,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-09-18T08:09:34.395267+0000","last_change":"2024-09-18T08:07:17.277964+0000","last_active":"2024-09-18T08:09:34.395267+0000","last_peered":"2024-09-18T08:09:34.395267+0000","last_clean":"2024-09-18T08:09:34.395267+0000","last_became_active":"2024-09-18T08:02:47.878922+0000","last_became_peered":"2024-09-18T08:02:47.878922+0000","last_unstale":"2024-09-18T08:09:34.395267+0000","last_undegraded":"2024-09-18T08:09:34.395267+0000","last_fullsized":"2024-09-18T08:09:34.395267+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:07:17.277894+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:46.860092+0000","last_clean_scrub_stamp":"2024-09-18T08:07:17.277894+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":16,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"69'11","reported_seq":321,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-09-18T08:09:34.396992+0000","last_change":"2024-09-18T08:06:24.164086+0000","last_active":"2024-09-18T08:09:34.396992+0000","last_peered":"2024-09-18T08:09:34.396992+0000","last_clean":"2024-09-18T08:09:34.396992+0000","last_became_active":"2024-09-18T08:06:24.135232+0000","last_became_peered":"2024-09-18T08:06:24.135232+0000","last_unstale":"2024-09-18T08:09:34.396992+0000","last_undegraded":"2024-09-18T08:09:34.396992+0000","last_fullsized":"2024-09-18T08:09:34.396992+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'11","last_scrub_stamp":"2024-09-18T08:06:12.967462+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_clean_scrub_stamp":"2024-09-18T08:06:12.967462+0000","objects_scrubbed":5,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":19,"num_read_kb":12,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,1],"acting":[5,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":338,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-09-18T08:09:34.405747+0000","last_change":"2024-09-18T08:08:52.701053+0000","last_active":"2024-09-18T08:09:34.405747+0000","last_peered":"2024-09-18T08:09:34.405747+0000","last_clean":"2024-09-18T08:09:34.405747+0000","last_became_active":"2024-09-18T08:08:52.700595+0000","last_became_peered":"2024-09-18T08:08:52.700595+0000","last_unstale":"2024-09-18T08:09:34.405747+0000","last_undegraded":"2024-09-18T08:09:34.405747+0000","last_fullsized":"2024-09-18T08:09:34.405747+0000","mapping_epoch":198,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":199,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:08:12.639376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:50.886830+0000","last_clean_scrub_stamp":"2024-09-18T08:08:12.639376+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":8,"num_pg":16,"stat_sum":{"num_bytes":5895159832,"num_objects":89954,"num_object_clones":0,"num_object_copies":269862,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":2211,"num_objects_unfound":0,"num_objects_dirty":89954,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":111546,"num_write_kb":6447937,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":22033,"num_bytes_recovered":1443954688,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":18570424320,"data_stored":18570412104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":78372,"ondisk_log_size":78372,"up":48,"acting":47,"num_store_stats":8},{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":416,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":85,"ondisk_log_size":85,"up":96,"acting":96,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":178,"num_object_clones":0,"num_object_copies":534,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":178,"num_whiteouts":0,"num_read":685,"num_read_kb":442,"num_write":417,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":214,"num_bytes_recovered":4114,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":417792,"data_stored":11106,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":398,"ondisk_log_size":398,"up":96,"acting":96,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":1613,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":24,"num_read_kb":24,"num_write":10,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":3226,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":4839,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":6,"ondisk_log_size":6,"up":96,"acting":96,"num_store_stats":8},{"poolid":2,"num_pg":3,"stat_sum":{"num_bytes":408,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":178,"num_read_kb":173,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":38,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":8,"ondisk_log_size":8,"up":9,"acting":9,"num_store_stats":7},{"poolid":1,"num_pg":11,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":869,"ondisk_log_size":869,"up":33,"acting":33,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":56,"seq":240518168663,"num_pgs":63,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1965268,"kb_used_data":1785540,"kb_used_omap":1,"kb_used_meta":179710,"kb_avail":91779884,"statfs":{"total":95995035648,"available":93982601216,"internally_reserved":0,"allocated":1828392960,"data_stored":1828011296,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":184023498},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":47,"seq":201863463005,"num_pgs":55,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2236348,"kb_used_data":2048236,"kb_used_omap":1,"kb_used_meta":188030,"kb_avail":91508804,"statfs":{"total":95995035648,"available":93705015296,"internally_reserved":0,"allocated":2097393664,"data_stored":2097035467,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":192543174},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":39,"seq":167503724642,"num_pgs":59,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":3061136,"kb_used_data":2903616,"kb_used_omap":1,"kb_used_meta":157502,"kb_avail":90684016,"statfs":{"total":95995035648,"available":92860432384,"internally_reserved":0,"allocated":2973302784,"data_stored":2972925483,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":161282502},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,0,3],"upper_bound":32},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":32,"seq":137438953576,"num_pgs":67,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2328156,"kb_used_data":2098444,"kb_used_omap":1,"kb_used_meta":229694,"kb_avail":91416996,"statfs":{"total":95995035648,"available":93611003904,"internally_reserved":0,"allocated":2148806656,"data_stored":2148417289,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":235207109},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":26,"seq":111669149806,"num_pgs":69,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2583204,"kb_used_data":2281300,"kb_used_omap":1,"kb_used_meta":301886,"kb_avail":91161948,"statfs":{"total":95995035648,"available":93349834752,"internally_reserved":0,"allocated":2336051200,"data_stored":2335653478,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":309131720},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,1,0,1],"upper_bound":16},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411444,"num_pgs":49,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2333712,"kb_used_data":2053696,"kb_used_omap":1,"kb_used_meta":279678,"kb_avail":91411440,"statfs":{"total":95995035648,"available":93605314560,"internally_reserved":0,"allocated":2102984704,"data_stored":2102607509,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":286390725},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1],"upper_bound":4},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574968,"num_pgs":63,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2840252,"kb_used_data":2584172,"kb_used_omap":1,"kb_used_meta":256062,"kb_avail":90904900,"statfs":{"total":95995035648,"available":93086617600,"internally_reserved":0,"allocated":2646192128,"data_stored":2645771292,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":262207941},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,3],"upper_bound":128},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738495,"num_pgs":56,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2721564,"kb_used_data":2473804,"kb_used_omap":1,"kb_used_meta":247742,"kb_avail":91023588,"statfs":{"total":95995035648,"available":93208154112,"internally_reserved":0,"allocated":2533175296,"data_stored":2532786131,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":253688261},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,0,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":266240,"data_stored":262688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1131,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1521,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":436,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":46,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1177,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":436,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":77824,"data_stored":2008,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1128,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":1898,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1474,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":990,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1034,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1254,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":403,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":416,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":403,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":2513371136,"data_stored":2513371136,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":2619211776,"data_stored":2619211776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":2100957184,"data_stored":2100953112,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":2330918912,"data_stored":2330918912,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":2148007936,"data_stored":2148007936,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":2969899008,"data_stored":2969894936,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":2074542080,"data_stored":2074542080,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":1813516288,"data_stored":1813512216,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T08:09:46.265 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid 779349ca-7593-11ef-bceb-c7b262605968 -- ceph pg dump --format=json 2024-09-18T08:09:46.275 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2 2024-09-18T08:09:46.359 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:09:46.402 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:09:46.402 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2' 2024-09-18T08:09:46.503 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7 2024-09-18T08:09:46.539 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:09:46.540 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:09:46.541 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7' 2024-09-18T08:09:46.641 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4 2024-09-18T08:09:46.656 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:09:46 smithi050 ceph-mon[32371]: pgmap v338: 158 pgs: 1 active+recovering+undersized+remapped, 157 active+clean; 5.2 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.05k op/s; 1921/255555 objects misplaced (0.752%); 4.2 MiB/s, 66 objects/s recovering 2024-09-18T08:09:46.656 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:09:46 smithi050 ceph-mon[32371]: 3.2 scrub starts 2024-09-18T08:09:46.656 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:09:46 smithi050 ceph-mon[32371]: 3.2 scrub ok 2024-09-18T08:09:46.656 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:09:46 smithi050 ceph-mon[32371]: from='client.? 172.21.15.7:0/1213210907' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-18T08:09:46.656 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:09:46 smithi050 ceph-mon[32371]: from='mgr.24470 172.21.15.7:0/2594390261' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-18T08:09:46.656 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:09:46 smithi050 ceph-mon[32371]: from='mgr.24470 172.21.15.7:0/2594390261' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-18T08:09:46.657 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:09:46 smithi050 ceph-mon[32371]: from='mgr.24470 172.21.15.7:0/2594390261' entity='mgr.y' 2024-09-18T08:09:46.679 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:09:46.679 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:09:46.680 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4' 2024-09-18T08:09:46.781 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5 2024-09-18T08:09:46.791 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:09:46 smithi007 ceph-mon[29459]: pgmap v338: 158 pgs: 1 active+recovering+undersized+remapped, 157 active+clean; 5.2 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.05k op/s; 1921/255555 objects misplaced (0.752%); 4.2 MiB/s, 66 objects/s recovering 2024-09-18T08:09:46.791 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:09:46 smithi007 ceph-mon[29459]: 3.2 scrub starts 2024-09-18T08:09:46.791 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:09:46 smithi007 ceph-mon[29459]: 3.2 scrub ok 2024-09-18T08:09:46.791 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:09:46 smithi007 ceph-mon[29459]: from='client.? 172.21.15.7:0/1213210907' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-18T08:09:46.791 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:09:46 smithi007 ceph-mon[29459]: from='mgr.24470 172.21.15.7:0/2594390261' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-18T08:09:46.791 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:09:46 smithi007 ceph-mon[29459]: from='mgr.24470 172.21.15.7:0/2594390261' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-18T08:09:46.791 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:09:46 smithi007 ceph-mon[29459]: from='mgr.24470 172.21.15.7:0/2594390261' entity='mgr.y' 2024-09-18T08:09:46.791 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Sep 18 08:09:46 smithi007 ceph-779349ca-7593-11ef-bceb-c7b262605968-mgr-y[29688]: ::ffff:172.21.15.50 - - [18/Sep/2024:08:09:46] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-09-18T08:09:46.792 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:09:46 smithi007 ceph-mon[39249]: pgmap v338: 158 pgs: 1 active+recovering+undersized+remapped, 157 active+clean; 5.2 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.05k op/s; 1921/255555 objects misplaced (0.752%); 4.2 MiB/s, 66 objects/s recovering 2024-09-18T08:09:46.792 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:09:46 smithi007 ceph-mon[39249]: 3.2 scrub starts 2024-09-18T08:09:46.792 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:09:46 smithi007 ceph-mon[39249]: 3.2 scrub ok 2024-09-18T08:09:46.792 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:09:46 smithi007 ceph-mon[39249]: from='client.? 172.21.15.7:0/1213210907' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-18T08:09:46.792 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:09:46 smithi007 ceph-mon[39249]: from='mgr.24470 172.21.15.7:0/2594390261' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-18T08:09:46.792 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:09:46 smithi007 ceph-mon[39249]: from='mgr.24470 172.21.15.7:0/2594390261' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-18T08:09:46.792 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:09:46 smithi007 ceph-mon[39249]: from='mgr.24470 172.21.15.7:0/2594390261' entity='mgr.y' 2024-09-18T08:09:46.815 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:09:46.816 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:09:46.816 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5' 2024-09-18T08:09:46.917 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4 2024-09-18T08:09:46.952 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:09:46.952 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:09:46.953 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4' 2024-09-18T08:09:47.053 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7 2024-09-18T08:09:47.077 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:09:47.078 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:09:47.078 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7' 2024-09-18T08:09:47.179 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2 2024-09-18T08:09:47.220 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:09:47.221 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:09:47.222 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2' 2024-09-18T08:09:47.322 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3 2024-09-18T08:09:47.359 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:09:47.360 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:09:47.360 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3' 2024-09-18T08:09:47.461 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7 2024-09-18T08:09:47.497 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:09:47.498 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:09:47.498 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7' 2024-09-18T08:09:47.599 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6 2024-09-18T08:09:47.636 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:09:47.637 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:09:47.637 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6' 2024-09-18T08:09:47.738 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5 2024-09-18T08:09:47.755 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/779349ca-7593-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T08:09:47.770 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:09:47.771 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:09:47.772 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5' 2024-09-18T08:09:47.811 DEBUG:teuthology.orchestra.run.smithi050:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid 779349ca-7593-11ef-bceb-c7b262605968 -- ceph daemon osd.4 dump_historic_ops 2024-09-18T08:09:47.811 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:09:47 smithi050 ceph-mon[32371]: 3.12 scrub starts 2024-09-18T08:09:47.811 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:09:47 smithi050 ceph-mon[32371]: 3.12 scrub ok 2024-09-18T08:09:47.812 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:09:47 smithi050 ceph-mon[32371]: 5.17 scrub starts 2024-09-18T08:09:47.812 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:09:47 smithi050 ceph-mon[32371]: 5.17 scrub ok 2024-09-18T08:09:47.812 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:09:47 smithi050 ceph-mon[32371]: 6.0 scrub starts 2024-09-18T08:09:47.812 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:09:47 smithi050 ceph-mon[32371]: 6.0 scrub ok 2024-09-18T08:09:47.872 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1 2024-09-18T08:09:47.891 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:09:47 smithi007 ceph-mon[39249]: 3.12 scrub starts 2024-09-18T08:09:47.892 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:09:47 smithi007 ceph-mon[39249]: 3.12 scrub ok 2024-09-18T08:09:47.892 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:09:47 smithi007 ceph-mon[39249]: 5.17 scrub starts 2024-09-18T08:09:47.892 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:09:47 smithi007 ceph-mon[39249]: 5.17 scrub ok 2024-09-18T08:09:47.892 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:09:47 smithi007 ceph-mon[39249]: 6.0 scrub starts 2024-09-18T08:09:47.892 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:09:47 smithi007 ceph-mon[39249]: 6.0 scrub ok 2024-09-18T08:09:47.892 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:09:47 smithi007 ceph-mon[29459]: 3.12 scrub starts 2024-09-18T08:09:47.892 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:09:47 smithi007 ceph-mon[29459]: 3.12 scrub ok 2024-09-18T08:09:47.892 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:09:47 smithi007 ceph-mon[29459]: 5.17 scrub starts 2024-09-18T08:09:47.892 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:09:47 smithi007 ceph-mon[29459]: 5.17 scrub ok 2024-09-18T08:09:47.893 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:09:47 smithi007 ceph-mon[29459]: 6.0 scrub starts 2024-09-18T08:09:47.893 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:09:47 smithi007 ceph-mon[29459]: 6.0 scrub ok 2024-09-18T08:09:47.922 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:09:47.922 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:09:47.923 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1' 2024-09-18T08:09:48.024 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7 2024-09-18T08:09:48.064 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:09:48.065 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:09:48.065 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7' 2024-09-18T08:09:48.167 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7 2024-09-18T08:09:48.220 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:09:48.221 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:09:48.221 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7' 2024-09-18T08:09:48.281 INFO:teuthology.orchestra.run.smithi050.stderr:Inferring config /var/lib/ceph/779349ca-7593-11ef-bceb-c7b262605968/mon.b/config 2024-09-18T08:09:48.322 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6 2024-09-18T08:09:48.359 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:09:48.360 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:09:48.361 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6' 2024-09-18T08:09:48.462 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4 2024-09-18T08:09:48.499 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:09:48.500 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:09:48.500 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4' 2024-09-18T08:09:48.601 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5 2024-09-18T08:09:48.662 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:09:48.663 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:09:48.664 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5' 2024-09-18T08:09:48.752 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-09-18T08:09:48.754 INFO:teuthology.orchestra.run.smithi007.stderr:dumped all 2024-09-18T08:09:48.764 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2 2024-09-18T08:09:48.806 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:09:48.806 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:09:48.807 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2' 2024-09-18T08:09:48.846 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid 779349ca-7593-11ef-bceb-c7b262605968 -- ceph osd set noscrub 2024-09-18T08:09:48.886 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:09:48 smithi007 ceph-mon[39249]: 5.9 scrub starts 2024-09-18T08:09:48.887 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:09:48 smithi007 ceph-mon[39249]: 5.9 scrub ok 2024-09-18T08:09:48.887 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:09:48 smithi007 ceph-mon[39249]: pgmap v339: 158 pgs: 1 active+recovering+undersized+remapped, 157 active+clean; 5.2 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.05k op/s; 1921/255555 objects misplaced (0.752%); 4.2 MiB/s, 66 objects/s recovering 2024-09-18T08:09:48.887 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:09:48 smithi007 ceph-mon[39249]: from='mgr.24470 172.21.15.7:0/2594390261' entity='mgr.y' 2024-09-18T08:09:48.887 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:09:48 smithi007 ceph-mon[39249]: from='mgr.24470 172.21.15.7:0/2594390261' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-18T08:09:48.887 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:09:48 smithi007 ceph-mon[29459]: 5.9 scrub starts 2024-09-18T08:09:48.887 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:09:48 smithi007 ceph-mon[29459]: 5.9 scrub ok 2024-09-18T08:09:48.887 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:09:48 smithi007 ceph-mon[29459]: pgmap v339: 158 pgs: 1 active+recovering+undersized+remapped, 157 active+clean; 5.2 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.05k op/s; 1921/255555 objects misplaced (0.752%); 4.2 MiB/s, 66 objects/s recovering 2024-09-18T08:09:48.887 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:09:48 smithi007 ceph-mon[29459]: from='mgr.24470 172.21.15.7:0/2594390261' entity='mgr.y' 2024-09-18T08:09:48.887 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:09:48 smithi007 ceph-mon[29459]: from='mgr.24470 172.21.15.7:0/2594390261' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-18T08:09:48.906 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:09:48 smithi050 ceph-mon[32371]: 5.9 scrub starts 2024-09-18T08:09:48.906 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:09:48 smithi050 ceph-mon[32371]: 5.9 scrub ok 2024-09-18T08:09:48.906 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:09:48 smithi050 ceph-mon[32371]: pgmap v339: 158 pgs: 1 active+recovering+undersized+remapped, 157 active+clean; 5.2 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.05k op/s; 1921/255555 objects misplaced (0.752%); 4.2 MiB/s, 66 objects/s recovering 2024-09-18T08:09:48.906 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:09:48 smithi050 ceph-mon[32371]: from='mgr.24470 172.21.15.7:0/2594390261' entity='mgr.y' 2024-09-18T08:09:48.906 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:09:48 smithi050 ceph-mon[32371]: from='mgr.24470 172.21.15.7:0/2594390261' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-18T08:09:48.908 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5 2024-09-18T08:09:48.937 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:09:48.938 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:09:48.938 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5' 2024-09-18T08:09:49.039 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3 2024-09-18T08:09:49.070 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2024-09-18T08:09:49.070 INFO:teuthology.orchestra.run.smithi050.stdout: "size": 20, 2024-09-18T08:09:49.070 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 600, 2024-09-18T08:09:49.070 INFO:teuthology.orchestra.run.smithi050.stdout: "ops": [ 2024-09-18T08:09:49.070 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:09:49.070 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.24356.0:1183 5.1f 5:f95f44c2:::notify.0:head [watch reconnect cookie 94700888679552 gen 1] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e101)", 2024-09-18T08:09:49.070 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-09-18T08:04:42.950623+0000", 2024-09-18T08:09:49.070 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 306.11509653799999, 2024-09-18T08:09:49.071 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.35355955700000002, 2024-09-18T08:09:49.071 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-09-18T08:09:49.071 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-09-18T08:09:49.071 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "started", 2024-09-18T08:09:49.071 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2024-09-18T08:09:49.071 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.24356", 2024-09-18T08:09:49.071 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "172.21.15.7:0/219168083", 2024-09-18T08:09:49.071 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 1183 2024-09-18T08:09:49.071 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:09:49.071 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-09-18T08:09:49.071 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:09:49.071 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-09-18T08:09:49.071 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:42.950623+0000", 2024-09-18T08:09:49.072 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-18T08:09:49.072 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:09:49.072 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:09:49.072 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-09-18T08:09:49.072 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:42.950623+0000", 2024-09-18T08:09:49.072 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-18T08:09:49.072 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:09:49.072 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:09:49.072 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-09-18T08:09:49.072 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:42.950624+0000", 2024-09-18T08:09:49.072 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 8.6499999999999998e-07 2024-09-18T08:09:49.072 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:09:49.072 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:09:49.072 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-09-18T08:09:49.073 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:42.950628+0000", 2024-09-18T08:09:49.073 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4.0879999999999997e-06 2024-09-18T08:09:49.074 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:09:49.074 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:09:49.074 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-09-18T08:09:49.074 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:42.950644+0000", 2024-09-18T08:09:49.074 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.6507e-05 2024-09-18T08:09:49.074 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:09:49.074 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:09:49.074 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-09-18T08:09:49.074 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:42.950651+0000", 2024-09-18T08:09:49.074 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 6.5599999999999999e-06 2024-09-18T08:09:49.074 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:09:49.075 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:09:49.075 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-18T08:09:49.075 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:42.950713+0000", 2024-09-18T08:09:49.075 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 6.2360000000000006e-05 2024-09-18T08:09:49.075 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:09:49.075 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:09:49.075 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "waiting for peered", 2024-09-18T08:09:49.075 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:42.950724+0000", 2024-09-18T08:09:49.075 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.0329999999999999e-05 2024-09-18T08:09:49.075 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:09:49.075 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:09:49.075 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-18T08:09:49.075 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:43.301407+0000", 2024-09-18T08:09:49.076 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.35068303499999998 2024-09-18T08:09:49.076 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:09:49.076 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:09:49.076 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-09-18T08:09:49.076 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:43.301697+0000", 2024-09-18T08:09:49.076 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000289938 2024-09-18T08:09:49.076 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:09:49.076 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:09:49.076 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-09-18T08:09:49.076 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:43.304183+0000", 2024-09-18T08:09:49.076 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0024858739999999999 2024-09-18T08:09:49.076 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-18T08:09:49.077 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-09-18T08:09:49.077 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-18T08:09:49.077 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:09:49.077 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:09:49.077 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.24356.0:1184 4.5 4:a60f9269:lc::lua_package_allowlist:head [watch reconnect cookie 94700889442560 gen 1] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e101)", 2024-09-18T08:09:49.077 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-09-18T08:04:42.950665+0000", 2024-09-18T08:09:49.077 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 306.11505415900001, 2024-09-18T08:09:49.077 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.35372086800000002, 2024-09-18T08:09:49.077 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-09-18T08:09:49.077 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-09-18T08:09:49.077 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "started", 2024-09-18T08:09:49.077 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2024-09-18T08:09:49.078 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.24356", 2024-09-18T08:09:49.078 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "172.21.15.7:0/219168083", 2024-09-18T08:09:49.078 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 1184 2024-09-18T08:09:49.078 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:09:49.078 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-09-18T08:09:49.078 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:09:49.078 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-09-18T08:09:49.078 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:42.950665+0000", 2024-09-18T08:09:49.078 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-18T08:09:49.078 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:09:49.078 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:09:49.078 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-09-18T08:09:49.078 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:42.950665+0000", 2024-09-18T08:09:49.078 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-18T08:09:49.079 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:09:49.079 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:09:49.079 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-09-18T08:09:49.079 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:42.950666+0000", 2024-09-18T08:09:49.079 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 6.1099999999999995e-07 2024-09-18T08:09:49.079 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:09:49.079 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:09:49.079 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-09-18T08:09:49.079 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:42.950668+0000", 2024-09-18T08:09:49.079 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 2.514e-06 2024-09-18T08:09:49.079 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:09:49.079 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:09:49.079 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-09-18T08:09:49.080 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:42.950683+0000", 2024-09-18T08:09:49.080 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.4827000000000001e-05 2024-09-18T08:09:49.080 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:09:49.080 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:09:49.080 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-09-18T08:09:49.080 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:42.950689+0000", 2024-09-18T08:09:49.080 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 6.0789999999999997e-06 2024-09-18T08:09:49.080 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:09:49.080 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:09:49.080 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-18T08:09:49.080 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:42.950753+0000", 2024-09-18T08:09:49.080 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 6.3466999999999994e-05 2024-09-18T08:09:49.080 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:09:49.080 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:09:49.081 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "waiting for peered", 2024-09-18T08:09:49.081 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:42.950763+0000", 2024-09-18T08:09:49.081 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.0193e-05 2024-09-18T08:09:49.081 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:09:49.081 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:09:49.081 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-18T08:09:49.081 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:43.301959+0000", 2024-09-18T08:09:49.081 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.35119641299999999 2024-09-18T08:09:49.081 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:09:49.081 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:09:49.081 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-09-18T08:09:49.081 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:43.302091+0000", 2024-09-18T08:09:49.081 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00013126500000000001 2024-09-18T08:09:49.081 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:09:49.082 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:09:49.082 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-09-18T08:09:49.082 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:43.304386+0000", 2024-09-18T08:09:49.082 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0022954989999999999 2024-09-18T08:09:49.082 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-18T08:09:49.082 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-09-18T08:09:49.082 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-18T08:09:49.082 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:09:49.082 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:09:49.082 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.14805.0:147845 7.1 7:8a0a44bd:::benchmark_data_smithi007_73155_object29554:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e127)", 2024-09-18T08:09:49.082 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-09-18T08:06:04.708313+0000", 2024-09-18T08:09:49.082 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 224.35740672899999, 2024-09-18T08:09:49.082 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.073618763000000004, 2024-09-18T08:09:49.082 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-09-18T08:09:49.083 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-09-18T08:09:49.083 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T08:09:49.083 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2024-09-18T08:09:49.083 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.14805", 2024-09-18T08:09:49.083 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "172.21.15.7:0/640320436", 2024-09-18T08:09:49.083 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 147845 2024-09-18T08:09:49.083 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:09:49.083 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-09-18T08:09:49.083 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:09:49.083 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-09-18T08:09:49.083 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:06:04.708313+0000", 2024-09-18T08:09:49.083 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-18T08:09:49.083 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:09:49.083 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:09:49.084 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-09-18T08:09:49.084 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:06:04.708313+0000", 2024-09-18T08:09:49.084 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-18T08:09:49.084 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:09:49.084 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:09:49.084 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-09-18T08:09:49.084 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:06:04.708313+0000", 2024-09-18T08:09:49.084 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 6.9999999999999997e-07 2024-09-18T08:09:49.084 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:09:49.084 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:09:49.084 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-09-18T08:09:49.084 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:06:04.708317+0000", 2024-09-18T08:09:49.084 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 3.36e-06 2024-09-18T08:09:49.085 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:09:49.085 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:09:49.085 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-09-18T08:09:49.085 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:06:04.708324+0000", 2024-09-18T08:09:49.085 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 7.5739999999999998e-06 2024-09-18T08:09:49.085 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:09:49.085 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:09:49.085 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-09-18T08:09:49.085 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:06:04.708329+0000", 2024-09-18T08:09:49.085 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4.4209999999999997e-06 2024-09-18T08:09:49.085 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:09:49.085 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:09:49.085 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-18T08:09:49.086 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:06:04.709653+0000", 2024-09-18T08:09:49.086 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0013239090000000001 2024-09-18T08:09:49.086 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:09:49.086 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:09:49.087 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-09-18T08:09:49.087 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:06:04.710395+0000", 2024-09-18T08:09:49.087 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00074240000000000005 2024-09-18T08:09:49.087 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:09:49.087 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:09:49.087 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "waiting for subops from 3,6", 2024-09-18T08:09:49.087 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:06:04.710508+0000", 2024-09-18T08:09:49.087 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000113248 2024-09-18T08:09:49.087 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:09:49.087 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:09:49.087 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "op_commit", 2024-09-18T08:09:49.087 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:06:04.712970+0000", 2024-09-18T08:09:49.087 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0024617160000000001 2024-09-18T08:09:49.088 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:09:49.088 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:09:49.088 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:09:49.088 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:06:04.714318+0000", 2024-09-18T08:09:49.088 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0013482629999999999 2024-09-18T08:09:49.088 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:09:49.088 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:09:49.088 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:09:49.088 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:06:04.781826+0000", 2024-09-18T08:09:49.088 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.067507519000000002 2024-09-18T08:09:49.088 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:09:49.088 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:09:49.088 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "commit_sent", 2024-09-18T08:09:49.089 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:06:04.781893+0000", 2024-09-18T08:09:49.089 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 6.7374999999999998e-05 2024-09-18T08:09:49.089 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:09:49.089 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:09:49.089 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-09-18T08:09:49.089 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:06:04.781932+0000", 2024-09-18T08:09:49.089 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 3.8278000000000002e-05 2024-09-18T08:09:49.089 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-18T08:09:49.089 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-09-18T08:09:49.089 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-18T08:09:49.089 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:09:49.089 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:09:49.089 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.15216.0:86225 8.5 8:ae8df458:::benchmark_data_smithi007_101583_object86224:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e205)", 2024-09-18T08:09:49.090 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-09-18T08:09:12.178973+0000", 2024-09-18T08:09:49.090 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 36.886746097, 2024-09-18T08:09:49.090 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.99102768600000002, 2024-09-18T08:09:49.090 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-09-18T08:09:49.090 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-09-18T08:09:49.090 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "reached pg", 2024-09-18T08:09:49.090 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2024-09-18T08:09:49.090 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.15216", 2024-09-18T08:09:49.090 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "172.21.15.7:0/2596041177", 2024-09-18T08:09:49.090 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 86225 2024-09-18T08:09:49.090 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:09:49.090 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-09-18T08:09:49.090 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:09:49.091 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-09-18T08:09:49.091 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:12.178973+0000", 2024-09-18T08:09:49.091 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-18T08:09:49.091 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:09:49.091 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:09:49.091 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-09-18T08:09:49.091 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:12.178973+0000", 2024-09-18T08:09:49.091 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-18T08:09:49.091 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:09:49.091 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:09:49.091 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-09-18T08:09:49.091 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:12.178975+0000", 2024-09-18T08:09:49.091 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.3969999999999999e-06 2024-09-18T08:09:49.091 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:09:49.092 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:09:49.092 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-09-18T08:09:49.092 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:12.179022+0000", 2024-09-18T08:09:49.092 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4.7580000000000002e-05 2024-09-18T08:09:49.092 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:09:49.092 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:09:49.092 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-09-18T08:09:49.092 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:12.179034+0000", 2024-09-18T08:09:49.092 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.1477e-05 2024-09-18T08:09:49.092 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:09:49.092 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:09:49.092 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-09-18T08:09:49.092 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:12.179040+0000", 2024-09-18T08:09:49.093 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 5.7529999999999998e-06 2024-09-18T08:09:49.093 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:09:49.093 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:09:49.093 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-18T08:09:49.093 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:12.179092+0000", 2024-09-18T08:09:49.093 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 5.2840000000000002e-05 2024-09-18T08:09:49.093 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:09:49.093 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:09:49.093 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "waiting for peered", 2024-09-18T08:09:49.093 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:12.179102+0000", 2024-09-18T08:09:49.093 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 9.5210000000000008e-06 2024-09-18T08:09:49.093 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:09:49.093 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:09:49.093 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-18T08:09:49.094 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:13.169945+0000", 2024-09-18T08:09:49.094 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.99084323600000002 2024-09-18T08:09:49.094 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:09:49.094 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:09:49.094 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-09-18T08:09:49.094 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:13.170001+0000", 2024-09-18T08:09:49.094 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 5.5881999999999998e-05 2024-09-18T08:09:49.094 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-18T08:09:49.094 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-09-18T08:09:49.094 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-18T08:09:49.094 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:09:49.094 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:09:49.094 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.15216.0:86263 8.5 8:af593c7a:::benchmark_data_smithi007_101583_object86262:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e205)", 2024-09-18T08:09:49.094 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-09-18T08:09:12.204249+0000", 2024-09-18T08:09:49.095 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 36.861470091999998, 2024-09-18T08:09:49.095 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.98012944999999996, 2024-09-18T08:09:49.095 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-09-18T08:09:49.095 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-09-18T08:09:49.095 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T08:09:49.095 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2024-09-18T08:09:49.095 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.15216", 2024-09-18T08:09:49.095 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "172.21.15.7:0/2596041177", 2024-09-18T08:09:49.095 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 86263 2024-09-18T08:09:49.095 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:09:49.095 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-09-18T08:09:49.095 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:09:49.095 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-09-18T08:09:49.095 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:12.204249+0000", 2024-09-18T08:09:49.096 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-18T08:09:49.096 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:09:49.096 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:09:49.096 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-09-18T08:09:49.096 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:12.204249+0000", 2024-09-18T08:09:49.096 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-18T08:09:49.096 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:09:49.096 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:09:49.096 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-09-18T08:09:49.096 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:12.204251+0000", 2024-09-18T08:09:49.096 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.635e-06 2024-09-18T08:09:49.096 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:09:49.096 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:09:49.096 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-09-18T08:09:49.097 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:12.204299+0000", 2024-09-18T08:09:49.097 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4.8026999999999999e-05 2024-09-18T08:09:49.097 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:09:49.097 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:09:49.097 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-09-18T08:09:49.097 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:12.204313+0000", 2024-09-18T08:09:49.097 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.4066000000000001e-05 2024-09-18T08:09:49.097 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:09:49.097 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:09:49.097 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-09-18T08:09:49.097 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:12.204321+0000", 2024-09-18T08:09:49.097 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 8.1839999999999999e-06 2024-09-18T08:09:49.097 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:09:49.097 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:09:49.098 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-18T08:09:49.098 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:12.204392+0000", 2024-09-18T08:09:49.098 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 7.0340999999999996e-05 2024-09-18T08:09:49.098 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:09:49.098 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:09:49.098 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "waiting for peered", 2024-09-18T08:09:49.098 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:12.204403+0000", 2024-09-18T08:09:49.098 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.1654000000000001e-05 2024-09-18T08:09:49.098 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:09:49.098 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:09:49.098 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-18T08:09:49.098 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:13.170414+0000", 2024-09-18T08:09:49.098 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.96601117000000003 2024-09-18T08:09:49.098 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:09:49.099 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:09:49.099 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-09-18T08:09:49.099 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:13.170546+0000", 2024-09-18T08:09:49.099 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00013120100000000001 2024-09-18T08:09:49.099 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:09:49.099 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:09:49.099 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "waiting for subops from 6,7", 2024-09-18T08:09:49.099 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:13.170965+0000", 2024-09-18T08:09:49.099 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00041976200000000001 2024-09-18T08:09:49.099 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:09:49.099 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:09:49.099 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:09:49.099 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:13.181803+0000", 2024-09-18T08:09:49.099 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.010837167999999999 2024-09-18T08:09:49.100 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:09:49.100 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:09:49.100 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:09:49.100 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:13.181864+0000", 2024-09-18T08:09:49.100 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 6.1857000000000001e-05 2024-09-18T08:09:49.100 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:09:49.100 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:09:49.100 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "op_commit", 2024-09-18T08:09:49.100 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:13.184283+0000", 2024-09-18T08:09:49.100 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0024184340000000001 2024-09-18T08:09:49.100 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:09:49.100 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:09:49.100 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "commit_sent", 2024-09-18T08:09:49.100 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:13.184339+0000", 2024-09-18T08:09:49.101 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 5.6378999999999997e-05 2024-09-18T08:09:49.101 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:09:49.101 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:09:49.101 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-09-18T08:09:49.101 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:13.184379+0000", 2024-09-18T08:09:49.101 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 3.9572e-05 2024-09-18T08:09:49.101 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-18T08:09:49.101 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-09-18T08:09:49.101 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-18T08:09:49.101 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:09:49.101 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:09:49.101 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.15216.0:86267 8.5 8:a135397b:::benchmark_data_smithi007_101583_object86266:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e205)", 2024-09-18T08:09:49.101 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-09-18T08:09:12.207725+0000", 2024-09-18T08:09:49.101 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 36.857994042000001, 2024-09-18T08:09:49.102 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.97676783599999994, 2024-09-18T08:09:49.102 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-09-18T08:09:49.102 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-09-18T08:09:49.102 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T08:09:49.102 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2024-09-18T08:09:49.102 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.15216", 2024-09-18T08:09:49.102 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "172.21.15.7:0/2596041177", 2024-09-18T08:09:49.102 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 86267 2024-09-18T08:09:49.102 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:09:49.102 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-09-18T08:09:49.102 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:09:49.102 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-09-18T08:09:49.102 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:12.207725+0000", 2024-09-18T08:09:49.102 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-18T08:09:49.103 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:09:49.103 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:09:49.103 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-09-18T08:09:49.103 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:12.207725+0000", 2024-09-18T08:09:49.103 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-18T08:09:49.103 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:09:49.103 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:09:49.103 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-09-18T08:09:49.103 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:12.207727+0000", 2024-09-18T08:09:49.103 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.79e-06 2024-09-18T08:09:49.103 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:09:49.103 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:09:49.103 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-09-18T08:09:49.103 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:12.207765+0000", 2024-09-18T08:09:49.104 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 3.7864000000000002e-05 2024-09-18T08:09:49.104 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:09:49.104 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:09:49.104 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-09-18T08:09:49.104 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:12.207783+0000", 2024-09-18T08:09:49.104 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.7779999999999999e-05 2024-09-18T08:09:49.104 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:09:49.104 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:09:49.104 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-09-18T08:09:49.104 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:12.207791+0000", 2024-09-18T08:09:49.104 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 7.9729999999999997e-06 2024-09-18T08:09:49.104 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:09:49.104 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:09:49.105 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-18T08:09:49.105 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:12.207872+0000", 2024-09-18T08:09:49.105 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 8.1318999999999998e-05 2024-09-18T08:09:49.105 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:09:49.105 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:09:49.105 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "waiting for peered", 2024-09-18T08:09:49.105 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:12.207905+0000", 2024-09-18T08:09:49.105 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 3.2694000000000002e-05 2024-09-18T08:09:49.105 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:09:49.105 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:09:49.105 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-18T08:09:49.105 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:13.179332+0000", 2024-09-18T08:09:49.105 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.97142748800000001 2024-09-18T08:09:49.105 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:09:49.106 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:09:49.106 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-09-18T08:09:49.106 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:13.179470+0000", 2024-09-18T08:09:49.106 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00013747400000000001 2024-09-18T08:09:49.106 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:09:49.106 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:09:49.106 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "waiting for subops from 6,7", 2024-09-18T08:09:49.106 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:13.179634+0000", 2024-09-18T08:09:49.106 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00016417600000000001 2024-09-18T08:09:49.106 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:09:49.106 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:09:49.106 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:09:49.106 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:13.184267+0000", 2024-09-18T08:09:49.106 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0046332079999999998 2024-09-18T08:09:49.107 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:09:49.107 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:09:49.107 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "op_commit", 2024-09-18T08:09:49.107 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:13.184386+0000", 2024-09-18T08:09:49.107 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000118993 2024-09-18T08:09:49.107 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:09:49.107 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:09:49.107 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:09:49.107 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:13.184437+0000", 2024-09-18T08:09:49.107 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 5.0988e-05 2024-09-18T08:09:49.107 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:09:49.107 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:09:49.107 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "commit_sent", 2024-09-18T08:09:49.107 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:13.184469+0000", 2024-09-18T08:09:49.108 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 3.1603000000000002e-05 2024-09-18T08:09:49.108 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:09:49.108 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:09:49.108 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-09-18T08:09:49.108 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:13.184493+0000", 2024-09-18T08:09:49.108 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 2.4485999999999999e-05 2024-09-18T08:09:49.108 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-18T08:09:49.108 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-09-18T08:09:49.108 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-18T08:09:49.108 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:09:49.108 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:09:49.108 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.15216.0:86268 8.5 8:aa7096fb:::benchmark_data_smithi007_101583_object86267:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e205)", 2024-09-18T08:09:49.108 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-09-18T08:09:12.208946+0000", 2024-09-18T08:09:49.108 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 36.856773435000001, 2024-09-18T08:09:49.109 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.980827699, 2024-09-18T08:09:49.109 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-09-18T08:09:49.109 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-09-18T08:09:49.109 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T08:09:49.109 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2024-09-18T08:09:49.109 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.15216", 2024-09-18T08:09:49.109 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "172.21.15.7:0/2596041177", 2024-09-18T08:09:49.109 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 86268 2024-09-18T08:09:49.109 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:09:49.109 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-09-18T08:09:49.109 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:09:49.109 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-09-18T08:09:49.109 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:12.208946+0000", 2024-09-18T08:09:49.109 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-18T08:09:49.109 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:09:49.110 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:09:49.110 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-09-18T08:09:49.110 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:12.208946+0000", 2024-09-18T08:09:49.110 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-18T08:09:49.110 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:09:49.110 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:09:49.110 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-09-18T08:09:49.110 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:12.208947+0000", 2024-09-18T08:09:49.110 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.201e-06 2024-09-18T08:09:49.110 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:09:49.110 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:09:49.110 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-09-18T08:09:49.110 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:12.209036+0000", 2024-09-18T08:09:49.110 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 8.8865999999999999e-05 2024-09-18T08:09:49.111 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:09:49.111 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:09:49.111 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-09-18T08:09:49.111 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:12.209049+0000", 2024-09-18T08:09:49.111 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.2795e-05 2024-09-18T08:09:49.111 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:09:49.111 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:09:49.111 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-09-18T08:09:49.111 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:12.209056+0000", 2024-09-18T08:09:49.111 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 7.1160000000000001e-06 2024-09-18T08:09:49.111 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:09:49.111 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:09:49.111 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-18T08:09:49.111 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:12.209116+0000", 2024-09-18T08:09:49.112 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 5.9517e-05 2024-09-18T08:09:49.112 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:09:49.112 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:09:49.112 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "waiting for peered", 2024-09-18T08:09:49.112 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:12.209125+0000", 2024-09-18T08:09:49.112 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 9.6630000000000005e-06 2024-09-18T08:09:49.112 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:09:49.112 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:09:49.112 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-18T08:09:49.112 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:13.182004+0000", 2024-09-18T08:09:49.112 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.97287885799999996 2024-09-18T08:09:49.112 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:09:49.112 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:09:49.112 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-09-18T08:09:49.113 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:13.182091+0000", 2024-09-18T08:09:49.113 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 8.6907e-05 2024-09-18T08:09:49.113 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:09:49.113 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:09:49.113 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "waiting for subops from 6,7", 2024-09-18T08:09:49.115 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:13.182183+0000", 2024-09-18T08:09:49.116 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 9.2131000000000003e-05 2024-09-18T08:09:49.116 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:09:49.116 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:09:49.116 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "op_commit", 2024-09-18T08:09:49.116 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:13.187041+0000", 2024-09-18T08:09:49.116 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0048579210000000003 2024-09-18T08:09:49.116 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:09:49.116 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:09:49.116 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:09:49.116 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:13.189581+0000", 2024-09-18T08:09:49.116 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.002539707 2024-09-18T08:09:49.116 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:09:49.116 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:09:49.117 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:09:49.117 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:13.189646+0000", 2024-09-18T08:09:49.117 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 6.5699999999999998e-05 2024-09-18T08:09:49.117 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:09:49.117 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:09:49.117 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "commit_sent", 2024-09-18T08:09:49.117 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:13.189714+0000", 2024-09-18T08:09:49.117 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 6.7108999999999997e-05 2024-09-18T08:09:49.117 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:09:49.117 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:09:49.117 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-09-18T08:09:49.117 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:13.189774+0000", 2024-09-18T08:09:49.117 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 6.0207999999999997e-05 2024-09-18T08:09:49.117 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-18T08:09:49.118 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-09-18T08:09:49.118 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-18T08:09:49.118 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:09:49.118 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:09:49.118 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.15216.0:86303 8.5 8:aebc4459:::benchmark_data_smithi007_101583_object86302:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e205)", 2024-09-18T08:09:49.118 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-09-18T08:09:12.250052+0000", 2024-09-18T08:09:49.118 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 36.815667879000003, 2024-09-18T08:09:49.118 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.94992247699999999, 2024-09-18T08:09:49.118 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-09-18T08:09:49.118 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-09-18T08:09:49.118 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T08:09:49.118 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2024-09-18T08:09:49.118 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.15216", 2024-09-18T08:09:49.118 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "172.21.15.7:0/2596041177", 2024-09-18T08:09:49.119 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 86303 2024-09-18T08:09:49.119 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:09:49.119 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-09-18T08:09:49.119 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:09:49.119 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-09-18T08:09:49.119 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:12.250052+0000", 2024-09-18T08:09:49.119 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-18T08:09:49.119 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:09:49.119 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:09:49.119 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-09-18T08:09:49.119 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:12.250052+0000", 2024-09-18T08:09:49.119 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-18T08:09:49.119 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:09:49.119 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:09:49.120 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-09-18T08:09:49.120 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:12.250053+0000", 2024-09-18T08:09:49.120 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.6130000000000001e-06 2024-09-18T08:09:49.120 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:09:49.120 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:09:49.120 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-09-18T08:09:49.120 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:12.250090+0000", 2024-09-18T08:09:49.120 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 3.6307999999999999e-05 2024-09-18T08:09:49.120 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:09:49.120 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:09:49.120 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-09-18T08:09:49.120 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:12.250102+0000", 2024-09-18T08:09:49.120 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.2624e-05 2024-09-18T08:09:49.120 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:09:49.121 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:09:49.121 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-09-18T08:09:49.121 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:12.250109+0000", 2024-09-18T08:09:49.121 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 6.426e-06 2024-09-18T08:09:49.121 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:09:49.121 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:09:49.121 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-18T08:09:49.121 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:12.250173+0000", 2024-09-18T08:09:49.121 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 6.3966000000000004e-05 2024-09-18T08:09:49.121 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:09:49.121 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:09:49.121 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "waiting for peered", 2024-09-18T08:09:49.121 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:12.250184+0000", 2024-09-18T08:09:49.121 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.1401e-05 2024-09-18T08:09:49.122 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:09:49.122 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:09:49.122 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-18T08:09:49.122 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:13.184533+0000", 2024-09-18T08:09:49.122 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.93434866299999997 2024-09-18T08:09:49.122 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:09:49.122 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:09:49.122 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-09-18T08:09:49.122 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:13.184634+0000", 2024-09-18T08:09:49.122 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000101139 2024-09-18T08:09:49.122 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:09:49.122 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:09:49.122 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "waiting for subops from 6,7", 2024-09-18T08:09:49.123 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:13.184793+0000", 2024-09-18T08:09:49.123 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00015932000000000001 2024-09-18T08:09:49.123 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:09:49.123 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:09:49.123 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:09:49.123 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:13.194839+0000", 2024-09-18T08:09:49.123 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.010046144 2024-09-18T08:09:49.123 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:09:49.123 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:09:49.123 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:09:49.123 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:13.194928+0000", 2024-09-18T08:09:49.123 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 8.9037999999999997e-05 2024-09-18T08:09:49.123 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:09:49.123 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:09:49.124 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "op_commit", 2024-09-18T08:09:49.124 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:13.199865+0000", 2024-09-18T08:09:49.124 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0049365879999999996 2024-09-18T08:09:49.124 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:09:49.124 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:09:49.124 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "commit_sent", 2024-09-18T08:09:49.124 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:13.199924+0000", 2024-09-18T08:09:49.124 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 5.9383999999999999e-05 2024-09-18T08:09:49.124 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:09:49.124 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:09:49.124 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-09-18T08:09:49.124 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:13.199974+0000", 2024-09-18T08:09:49.124 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4.9863e-05 2024-09-18T08:09:49.124 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-18T08:09:49.125 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-09-18T08:09:49.125 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-18T08:09:49.125 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:09:49.125 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:09:49.125 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.15216.0:86344 8.5 8:ac157b17:::benchmark_data_smithi007_101583_object86343:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e205)", 2024-09-18T08:09:49.125 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-09-18T08:09:12.295743+0000", 2024-09-18T08:09:49.125 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 36.769976198999998, 2024-09-18T08:09:49.125 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.92374953400000004, 2024-09-18T08:09:49.125 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-09-18T08:09:49.125 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-09-18T08:09:49.125 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T08:09:49.125 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2024-09-18T08:09:49.125 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.15216", 2024-09-18T08:09:49.125 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "172.21.15.7:0/2596041177", 2024-09-18T08:09:49.126 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 86344 2024-09-18T08:09:49.126 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:09:49.126 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-09-18T08:09:49.126 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:09:49.126 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-09-18T08:09:49.126 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:12.295743+0000", 2024-09-18T08:09:49.126 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-18T08:09:49.126 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:09:49.126 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:09:49.126 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-09-18T08:09:49.126 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:12.295743+0000", 2024-09-18T08:09:49.126 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-18T08:09:49.126 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:09:49.126 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:09:49.127 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-09-18T08:09:49.127 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:12.295745+0000", 2024-09-18T08:09:49.127 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.57e-06 2024-09-18T08:09:49.127 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:09:49.127 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:09:49.127 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-09-18T08:09:49.127 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:12.295795+0000", 2024-09-18T08:09:49.127 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 5.0062999999999998e-05 2024-09-18T08:09:49.127 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:09:49.127 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:09:49.127 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-09-18T08:09:49.127 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:12.295807+0000", 2024-09-18T08:09:49.127 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.1578e-05 2024-09-18T08:09:49.127 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:09:49.128 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:09:49.128 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-09-18T08:09:49.128 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:12.295812+0000", 2024-09-18T08:09:49.128 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 5.8930000000000002e-06 2024-09-18T08:09:49.128 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:09:49.128 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:09:49.128 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-18T08:09:49.128 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:12.295875+0000", 2024-09-18T08:09:49.128 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 6.2168000000000005e-05 2024-09-18T08:09:49.128 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:09:49.128 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:09:49.128 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "waiting for peered", 2024-09-18T08:09:49.128 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:12.295883+0000", 2024-09-18T08:09:49.128 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 7.9969999999999998e-06 2024-09-18T08:09:49.129 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:09:49.129 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:09:49.129 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-18T08:09:49.129 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:13.187072+0000", 2024-09-18T08:09:49.129 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.89118984899999998 2024-09-18T08:09:49.129 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:09:49.129 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:09:49.129 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-09-18T08:09:49.129 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:13.187193+0000", 2024-09-18T08:09:49.129 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00012062599999999999 2024-09-18T08:09:49.129 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:09:49.129 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:09:49.129 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "waiting for subops from 6,7", 2024-09-18T08:09:49.129 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:13.187351+0000", 2024-09-18T08:09:49.131 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00015781500000000001 2024-09-18T08:09:49.131 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:09:49.131 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:09:49.131 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:09:49.131 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:13.194877+0000", 2024-09-18T08:09:49.131 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0075262139999999998 2024-09-18T08:09:49.131 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:09:49.131 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:09:49.131 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:09:49.132 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:13.195031+0000", 2024-09-18T08:09:49.132 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00015384799999999999 2024-09-18T08:09:49.132 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:09:49.132 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:09:49.132 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "op_commit", 2024-09-18T08:09:49.132 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:13.219386+0000", 2024-09-18T08:09:49.132 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.024355226000000001 2024-09-18T08:09:49.132 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:09:49.132 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:09:49.132 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "commit_sent", 2024-09-18T08:09:49.132 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:13.219440+0000", 2024-09-18T08:09:49.132 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 5.3786999999999998e-05 2024-09-18T08:09:49.132 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:09:49.133 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:09:49.133 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-09-18T08:09:49.133 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:13.219493+0000", 2024-09-18T08:09:49.133 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 5.2899999999999998e-05 2024-09-18T08:09:49.133 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-18T08:09:49.133 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-09-18T08:09:49.133 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-18T08:09:49.133 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:09:49.133 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:09:49.133 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.15216.0:86350 8.5 8:a14e8599:::benchmark_data_smithi007_101583_object86349:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e205)", 2024-09-18T08:09:49.133 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-09-18T08:09:12.297534+0000", 2024-09-18T08:09:49.133 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 36.768185715000001, 2024-09-18T08:09:49.133 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.922055505, 2024-09-18T08:09:49.133 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-09-18T08:09:49.134 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-09-18T08:09:49.134 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T08:09:49.134 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2024-09-18T08:09:49.134 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.15216", 2024-09-18T08:09:49.134 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "172.21.15.7:0/2596041177", 2024-09-18T08:09:49.134 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 86350 2024-09-18T08:09:49.134 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:09:49.134 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-09-18T08:09:49.134 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:09:49.134 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-09-18T08:09:49.134 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:12.297534+0000", 2024-09-18T08:09:49.134 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-18T08:09:49.134 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:09:49.134 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:09:49.135 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-09-18T08:09:49.135 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:12.297534+0000", 2024-09-18T08:09:49.135 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-18T08:09:49.135 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:09:49.135 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:09:49.135 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-09-18T08:09:49.135 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:12.297534+0000", 2024-09-18T08:09:49.135 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 7.0800000000000004e-07 2024-09-18T08:09:49.135 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:09:49.135 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:09:49.135 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-09-18T08:09:49.135 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:12.297606+0000", 2024-09-18T08:09:49.135 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 7.1075000000000006e-05 2024-09-18T08:09:49.136 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:09:49.136 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:09:49.136 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-09-18T08:09:49.136 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:12.297613+0000", 2024-09-18T08:09:49.136 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 7.6260000000000003e-06 2024-09-18T08:09:49.136 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:09:49.136 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:09:49.136 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-09-18T08:09:49.136 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:12.297618+0000", 2024-09-18T08:09:49.137 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4.8550000000000001e-06 2024-09-18T08:09:49.137 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:09:49.137 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:09:49.137 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-18T08:09:49.137 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:12.297660+0000", 2024-09-18T08:09:49.137 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4.1968000000000002e-05 2024-09-18T08:09:49.137 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:09:49.137 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:09:49.137 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "waiting for peered", 2024-09-18T08:09:49.137 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:12.297666+0000", 2024-09-18T08:09:49.137 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 5.806e-06 2024-09-18T08:09:49.137 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:09:49.137 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:09:49.138 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-18T08:09:49.138 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:13.189816+0000", 2024-09-18T08:09:49.138 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.89215055300000001 2024-09-18T08:09:49.138 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:09:49.138 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:09:49.138 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-09-18T08:09:49.138 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:13.190014+0000", 2024-09-18T08:09:49.138 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000197894 2024-09-18T08:09:49.138 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:09:49.138 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:09:49.138 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "waiting for subops from 6,7", 2024-09-18T08:09:49.138 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:13.190197+0000", 2024-09-18T08:09:49.138 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00018230399999999999 2024-09-18T08:09:49.138 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:09:49.139 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:09:49.139 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:09:49.139 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:13.194999+0000", 2024-09-18T08:09:49.139 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0048023019999999996 2024-09-18T08:09:49.139 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:09:49.139 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:09:49.139 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:09:49.139 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:13.197606+0000", 2024-09-18T08:09:49.139 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0026074890000000002 2024-09-18T08:09:49.139 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:09:49.139 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:09:49.139 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "op_commit", 2024-09-18T08:09:49.139 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:13.219502+0000", 2024-09-18T08:09:49.139 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.02189514 2024-09-18T08:09:49.140 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:09:49.140 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:09:49.140 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "commit_sent", 2024-09-18T08:09:49.140 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:13.219546+0000", 2024-09-18T08:09:49.140 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4.4574999999999999e-05 2024-09-18T08:09:49.140 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:09:49.140 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:09:49.140 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-09-18T08:09:49.140 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:13.219589+0000", 2024-09-18T08:09:49.140 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4.3210000000000001e-05 2024-09-18T08:09:49.140 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-18T08:09:49.140 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-09-18T08:09:49.140 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-18T08:09:49.140 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:09:49.141 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:09:49.141 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.15216.0:86366 8.5 8:a8c4536d:::benchmark_data_smithi007_101583_object86365:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e205)", 2024-09-18T08:09:49.141 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-09-18T08:09:12.318202+0000", 2024-09-18T08:09:49.141 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 36.747517039000002, 2024-09-18T08:09:49.141 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.90148185700000005, 2024-09-18T08:09:49.141 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-09-18T08:09:49.141 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-09-18T08:09:49.141 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T08:09:49.141 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2024-09-18T08:09:49.141 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.15216", 2024-09-18T08:09:49.141 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "172.21.15.7:0/2596041177", 2024-09-18T08:09:49.141 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 86366 2024-09-18T08:09:49.141 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:09:49.141 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-09-18T08:09:49.142 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:09:49.142 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-09-18T08:09:49.142 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:12.318202+0000", 2024-09-18T08:09:49.142 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-18T08:09:49.142 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:09:49.142 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:09:49.142 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-09-18T08:09:49.142 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:12.318202+0000", 2024-09-18T08:09:49.142 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-18T08:09:49.142 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:09:49.142 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:09:49.142 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-09-18T08:09:49.142 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:12.318205+0000", 2024-09-18T08:09:49.142 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 2.074e-06 2024-09-18T08:09:49.143 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:09:49.143 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:09:49.143 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-09-18T08:09:49.143 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:12.318242+0000", 2024-09-18T08:09:49.143 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 3.6996999999999998e-05 2024-09-18T08:09:49.143 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:09:49.143 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:09:49.143 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-09-18T08:09:49.143 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:12.318258+0000", 2024-09-18T08:09:49.143 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.6382e-05 2024-09-18T08:09:49.143 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:09:49.143 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:09:49.143 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-09-18T08:09:49.143 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:12.318266+0000", 2024-09-18T08:09:49.143 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 7.9820000000000008e-06 2024-09-18T08:09:49.144 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:09:49.144 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:09:49.144 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-18T08:09:49.144 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:12.318402+0000", 2024-09-18T08:09:49.144 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000136477 2024-09-18T08:09:49.144 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:09:49.144 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:09:49.144 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "waiting for peered", 2024-09-18T08:09:49.144 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:12.318414+0000", 2024-09-18T08:09:49.144 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.2096000000000001e-05 2024-09-18T08:09:49.144 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:09:49.144 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:09:49.144 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-18T08:09:49.144 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:13.192024+0000", 2024-09-18T08:09:49.145 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.87360952300000005 2024-09-18T08:09:49.145 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:09:49.145 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:09:49.145 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-09-18T08:09:49.145 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:13.192155+0000", 2024-09-18T08:09:49.145 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00013099499999999999 2024-09-18T08:09:49.145 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:09:49.145 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:09:49.145 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "waiting for subops from 6,7", 2024-09-18T08:09:49.145 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:13.192313+0000", 2024-09-18T08:09:49.145 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00015802500000000001 2024-09-18T08:09:49.145 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:09:49.145 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:09:49.145 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:09:49.146 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:13.195063+0000", 2024-09-18T08:09:49.146 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0027500279999999998 2024-09-18T08:09:49.146 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:09:49.146 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:09:49.146 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:09:49.146 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:13.200043+0000", 2024-09-18T08:09:49.146 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0049800929999999997 2024-09-18T08:09:49.146 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:09:49.146 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:09:49.146 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "op_commit", 2024-09-18T08:09:49.146 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:13.219598+0000", 2024-09-18T08:09:49.146 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.019554768 2024-09-18T08:09:49.146 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:09:49.147 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:09:49.147 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "commit_sent", 2024-09-18T08:09:49.147 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:13.219647+0000", 2024-09-18T08:09:49.147 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4.9327999999999999e-05 2024-09-18T08:09:49.147 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:09:49.147 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:09:49.147 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-09-18T08:09:49.147 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:13.219684+0000", 2024-09-18T08:09:49.147 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 3.7088999999999997e-05 2024-09-18T08:09:49.147 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-18T08:09:49.147 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-09-18T08:09:49.147 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-18T08:09:49.147 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:09:49.147 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:09:49.148 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.15216.0:86390 8.5 8:a8c7b3fd:::benchmark_data_smithi007_101583_object86389:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e205)", 2024-09-18T08:09:49.148 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-09-18T08:09:12.356640+0000", 2024-09-18T08:09:49.148 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 36.709079738, 2024-09-18T08:09:49.148 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.86366743700000004, 2024-09-18T08:09:49.148 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-09-18T08:09:49.148 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-09-18T08:09:49.148 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T08:09:49.148 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2024-09-18T08:09:49.148 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.15216", 2024-09-18T08:09:49.148 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "172.21.15.7:0/2596041177", 2024-09-18T08:09:49.148 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 86390 2024-09-18T08:09:49.148 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:09:49.148 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-09-18T08:09:49.148 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:09:49.149 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-09-18T08:09:49.149 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:12.356640+0000", 2024-09-18T08:09:49.149 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-18T08:09:49.149 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:09:49.149 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:09:49.149 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-09-18T08:09:49.149 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:12.356640+0000", 2024-09-18T08:09:49.149 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-18T08:09:49.149 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:09:49.149 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:09:49.149 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-09-18T08:09:49.149 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:12.356641+0000", 2024-09-18T08:09:49.149 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.5430000000000001e-06 2024-09-18T08:09:49.149 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:09:49.150 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:09:49.150 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-09-18T08:09:49.150 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:12.356684+0000", 2024-09-18T08:09:49.150 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4.2222999999999999e-05 2024-09-18T08:09:49.150 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:09:49.150 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:09:49.150 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-09-18T08:09:49.150 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:12.356696+0000", 2024-09-18T08:09:49.150 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.2663000000000001e-05 2024-09-18T08:09:49.150 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:09:49.150 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:09:49.150 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-09-18T08:09:49.150 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:12.356703+0000", 2024-09-18T08:09:49.150 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 6.6710000000000002e-06 2024-09-18T08:09:49.151 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:09:49.151 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:09:49.151 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-18T08:09:49.151 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:12.356759+0000", 2024-09-18T08:09:49.151 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 5.6382000000000003e-05 2024-09-18T08:09:49.151 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:09:49.151 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:09:49.151 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "waiting for peered", 2024-09-18T08:09:49.151 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:12.356767+0000", 2024-09-18T08:09:49.151 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 7.5190000000000003e-06 2024-09-18T08:09:49.151 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:09:49.151 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:09:49.151 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-18T08:09:49.151 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:13.195090+0000", 2024-09-18T08:09:49.152 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.83832288600000004 2024-09-18T08:09:49.152 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:09:49.152 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:09:49.152 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-09-18T08:09:49.152 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:13.195221+0000", 2024-09-18T08:09:49.152 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00013146699999999999 2024-09-18T08:09:49.152 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:09:49.152 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:09:49.152 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "waiting for subops from 6,7", 2024-09-18T08:09:49.152 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:13.195370+0000", 2024-09-18T08:09:49.152 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000148598 2024-09-18T08:09:49.152 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:09:49.152 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:09:49.152 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:09:49.153 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:13.199845+0000", 2024-09-18T08:09:49.153 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0044749999999999998 2024-09-18T08:09:49.153 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:09:49.153 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:09:49.153 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:09:49.153 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:13.207294+0000", 2024-09-18T08:09:49.153 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0074489309999999998 2024-09-18T08:09:49.153 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:09:49.153 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:09:49.153 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "op_commit", 2024-09-18T08:09:49.153 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:13.220218+0000", 2024-09-18T08:09:49.153 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.012924715 2024-09-18T08:09:49.153 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:09:49.153 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:09:49.154 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "commit_sent", 2024-09-18T08:09:49.154 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:13.220272+0000", 2024-09-18T08:09:49.154 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 5.3748000000000001e-05 2024-09-18T08:09:49.154 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:09:49.154 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:09:49.154 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-09-18T08:09:49.154 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:13.220307+0000", 2024-09-18T08:09:49.154 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 3.5091e-05 2024-09-18T08:09:49.154 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-18T08:09:49.154 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-09-18T08:09:49.154 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-18T08:09:49.154 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:09:49.154 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:09:49.154 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.15216.0:86394 8.5 8:a810cd9d:::benchmark_data_smithi007_101583_object86393:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e205)", 2024-09-18T08:09:49.155 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-09-18T08:09:12.360967+0000", 2024-09-18T08:09:49.155 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 36.704752161000002, 2024-09-18T08:09:49.155 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.85950780100000002, 2024-09-18T08:09:49.155 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-09-18T08:09:49.155 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-09-18T08:09:49.155 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T08:09:49.155 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2024-09-18T08:09:49.155 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.15216", 2024-09-18T08:09:49.155 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "172.21.15.7:0/2596041177", 2024-09-18T08:09:49.155 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 86394 2024-09-18T08:09:49.155 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:09:49.155 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-09-18T08:09:49.155 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:09:49.155 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-09-18T08:09:49.156 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:12.360967+0000", 2024-09-18T08:09:49.156 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-18T08:09:49.156 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:09:49.156 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:09:49.156 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-09-18T08:09:49.156 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:12.360967+0000", 2024-09-18T08:09:49.156 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-18T08:09:49.156 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:09:49.156 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:09:49.156 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-09-18T08:09:49.156 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:12.360969+0000", 2024-09-18T08:09:49.156 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.8500000000000001e-06 2024-09-18T08:09:49.156 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:09:49.157 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:09:49.157 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-09-18T08:09:49.157 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:12.361013+0000", 2024-09-18T08:09:49.157 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4.3693000000000003e-05 2024-09-18T08:09:49.157 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:09:49.157 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:09:49.157 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-09-18T08:09:49.157 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:12.361031+0000", 2024-09-18T08:09:49.157 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.8340000000000001e-05 2024-09-18T08:09:49.157 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:09:49.157 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:09:49.157 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-09-18T08:09:49.157 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:12.361040+0000", 2024-09-18T08:09:49.157 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 8.5320000000000006e-06 2024-09-18T08:09:49.158 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:09:49.158 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:09:49.158 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-18T08:09:49.158 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:12.361108+0000", 2024-09-18T08:09:49.158 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 6.7879000000000005e-05 2024-09-18T08:09:49.158 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:09:49.158 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:09:49.158 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "waiting for peered", 2024-09-18T08:09:49.158 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:12.361119+0000", 2024-09-18T08:09:49.158 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.1163e-05 2024-09-18T08:09:49.158 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:09:49.158 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:09:49.158 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-18T08:09:49.158 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:13.197641+0000", 2024-09-18T08:09:49.159 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.83652230299999997 2024-09-18T08:09:49.159 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:09:49.159 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:09:49.159 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-09-18T08:09:49.159 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:13.197776+0000", 2024-09-18T08:09:49.159 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00013499000000000001 2024-09-18T08:09:49.159 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:09:49.159 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:09:49.159 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "waiting for subops from 6,7", 2024-09-18T08:09:49.159 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:13.197954+0000", 2024-09-18T08:09:49.159 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00017745200000000001 2024-09-18T08:09:49.159 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:09:49.159 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:09:49.159 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:09:49.160 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:13.207329+0000", 2024-09-18T08:09:49.160 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0093754460000000008 2024-09-18T08:09:49.160 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:09:49.160 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:09:49.160 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:09:49.160 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:13.207363+0000", 2024-09-18T08:09:49.160 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 3.3535999999999999e-05 2024-09-18T08:09:49.160 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:09:49.160 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:09:49.160 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "op_commit", 2024-09-18T08:09:49.160 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:13.220323+0000", 2024-09-18T08:09:49.160 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.012960121999999999 2024-09-18T08:09:49.160 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:09:49.161 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:09:49.161 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "commit_sent", 2024-09-18T08:09:49.161 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:13.220445+0000", 2024-09-18T08:09:49.161 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00012217099999999999 2024-09-18T08:09:49.161 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:09:49.161 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:09:49.161 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-09-18T08:09:49.161 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:13.220475+0000", 2024-09-18T08:09:49.161 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 3.0324e-05 2024-09-18T08:09:49.161 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-18T08:09:49.161 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-09-18T08:09:49.161 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-18T08:09:49.161 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:09:49.161 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:09:49.162 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.15216.0:86399 8.5 8:a535189e:::benchmark_data_smithi007_101583_object86398:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e205)", 2024-09-18T08:09:49.162 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-09-18T08:09:12.364040+0000", 2024-09-18T08:09:49.162 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 36.701679097000003, 2024-09-18T08:09:49.162 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.85654366599999998, 2024-09-18T08:09:49.162 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-09-18T08:09:49.162 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-09-18T08:09:49.162 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T08:09:49.162 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2024-09-18T08:09:49.162 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.15216", 2024-09-18T08:09:49.162 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "172.21.15.7:0/2596041177", 2024-09-18T08:09:49.162 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 86399 2024-09-18T08:09:49.162 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:09:49.162 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-09-18T08:09:49.162 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:09:49.163 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-09-18T08:09:49.163 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:12.364040+0000", 2024-09-18T08:09:49.163 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-18T08:09:49.163 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:09:49.163 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:09:49.163 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-09-18T08:09:49.163 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:12.364040+0000", 2024-09-18T08:09:49.163 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-18T08:09:49.163 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:09:49.163 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:09:49.163 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-09-18T08:09:49.163 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:12.364042+0000", 2024-09-18T08:09:49.163 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.407e-06 2024-09-18T08:09:49.163 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:09:49.164 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:09:49.164 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-09-18T08:09:49.164 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:12.364174+0000", 2024-09-18T08:09:49.164 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000132545 2024-09-18T08:09:49.164 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:09:49.164 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:09:49.164 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-09-18T08:09:49.164 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:12.364186+0000", 2024-09-18T08:09:49.164 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.1912e-05 2024-09-18T08:09:49.164 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:09:49.164 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:09:49.164 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-09-18T08:09:49.165 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:12.364192+0000", 2024-09-18T08:09:49.165 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 5.4600000000000002e-06 2024-09-18T08:09:49.165 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:09:49.165 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:09:49.165 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-18T08:09:49.165 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:12.364257+0000", 2024-09-18T08:09:49.165 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 6.5606999999999997e-05 2024-09-18T08:09:49.165 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:09:49.165 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:09:49.165 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "waiting for peered", 2024-09-18T08:09:49.165 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:12.364271+0000", 2024-09-18T08:09:49.165 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.3433e-05 2024-09-18T08:09:49.165 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:09:49.166 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:09:49.166 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-18T08:09:49.166 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:13.200082+0000", 2024-09-18T08:09:49.166 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.83581135500000003 2024-09-18T08:09:49.166 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:09:49.166 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:09:49.166 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-09-18T08:09:49.166 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:13.200201+0000", 2024-09-18T08:09:49.166 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00011913099999999999 2024-09-18T08:09:49.166 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:09:49.166 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:09:49.166 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "waiting for subops from 6,7", 2024-09-18T08:09:49.166 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:13.200360+0000", 2024-09-18T08:09:49.167 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000159173 2024-09-18T08:09:49.167 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:09:49.167 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:09:49.167 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:09:49.167 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:13.207395+0000", 2024-09-18T08:09:49.167 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0070341090000000002 2024-09-18T08:09:49.167 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:09:49.167 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:09:49.167 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:09:49.167 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:13.207427+0000", 2024-09-18T08:09:49.167 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 3.2138000000000003e-05 2024-09-18T08:09:49.167 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:09:49.167 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:09:49.168 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "op_commit", 2024-09-18T08:09:49.168 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:13.220524+0000", 2024-09-18T08:09:49.168 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.013097698 2024-09-18T08:09:49.168 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:09:49.168 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:09:49.168 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "commit_sent", 2024-09-18T08:09:49.168 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:13.220557+0000", 2024-09-18T08:09:49.168 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 3.2682999999999999e-05 2024-09-18T08:09:49.168 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:09:49.168 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:09:49.168 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-09-18T08:09:49.168 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:13.220584+0000", 2024-09-18T08:09:49.168 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 2.7015000000000001e-05 2024-09-18T08:09:49.169 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-18T08:09:49.169 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-09-18T08:09:49.169 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-18T08:09:49.169 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:09:49.169 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:09:49.169 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.15216.0:86429 8.5 8:a7f1f687:::benchmark_data_smithi007_101583_object86428:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e205)", 2024-09-18T08:09:49.169 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-09-18T08:09:12.416915+0000", 2024-09-18T08:09:49.169 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 36.648804327999997, 2024-09-18T08:09:49.169 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.80373175900000005, 2024-09-18T08:09:49.169 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-09-18T08:09:49.169 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-09-18T08:09:49.169 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T08:09:49.169 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2024-09-18T08:09:49.170 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.15216", 2024-09-18T08:09:49.170 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "172.21.15.7:0/2596041177", 2024-09-18T08:09:49.170 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 86429 2024-09-18T08:09:49.170 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:09:49.170 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-09-18T08:09:49.170 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:09:49.170 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-09-18T08:09:49.170 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:12.416915+0000", 2024-09-18T08:09:49.170 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-18T08:09:49.170 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:09:49.170 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:09:49.170 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-09-18T08:09:49.170 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:12.416915+0000", 2024-09-18T08:09:49.170 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-18T08:09:49.171 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:09:49.171 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:09:49.171 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-09-18T08:09:49.171 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:12.416917+0000", 2024-09-18T08:09:49.171 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.7150000000000001e-06 2024-09-18T08:09:49.171 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:09:49.171 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:09:49.171 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-09-18T08:09:49.171 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:12.416952+0000", 2024-09-18T08:09:49.171 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 3.4860000000000002e-05 2024-09-18T08:09:49.171 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:09:49.171 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:09:49.171 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-09-18T08:09:49.171 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:12.416966+0000", 2024-09-18T08:09:49.172 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.3884e-05 2024-09-18T08:09:49.172 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:09:49.172 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:09:49.172 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-09-18T08:09:49.172 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:12.416973+0000", 2024-09-18T08:09:49.172 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 7.5809999999999999e-06 2024-09-18T08:09:49.172 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:09:49.172 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:09:49.172 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-18T08:09:49.172 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:12.417031+0000", 2024-09-18T08:09:49.172 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 5.7544999999999999e-05 2024-09-18T08:09:49.172 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:09:49.173 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:09:49.173 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:09:49.173 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3' 2024-09-18T08:09:49.174 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:09:49.174 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "waiting for peered", 2024-09-18T08:09:49.174 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:12.417039+0000", 2024-09-18T08:09:49.174 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 8.5369999999999997e-06 2024-09-18T08:09:49.174 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:09:49.175 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:09:49.175 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-18T08:09:49.175 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:13.202704+0000", 2024-09-18T08:09:49.175 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.78566510099999998 2024-09-18T08:09:49.175 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:09:49.175 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:09:49.175 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-09-18T08:09:49.175 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:13.202825+0000", 2024-09-18T08:09:49.175 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00012023600000000001 2024-09-18T08:09:49.175 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:09:49.175 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:09:49.175 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "waiting for subops from 6,7", 2024-09-18T08:09:49.175 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:13.202989+0000", 2024-09-18T08:09:49.176 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00016451000000000001 2024-09-18T08:09:49.176 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:09:49.176 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:09:49.176 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:09:49.176 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:13.207464+0000", 2024-09-18T08:09:49.176 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0044748909999999999 2024-09-18T08:09:49.176 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:09:49.176 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:09:49.176 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:09:49.176 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:13.214034+0000", 2024-09-18T08:09:49.176 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0065698229999999998 2024-09-18T08:09:49.176 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:09:49.176 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:09:49.177 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "op_commit", 2024-09-18T08:09:49.177 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:13.220590+0000", 2024-09-18T08:09:49.177 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0065556909999999998 2024-09-18T08:09:49.177 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:09:49.177 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:09:49.177 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "commit_sent", 2024-09-18T08:09:49.177 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:13.220620+0000", 2024-09-18T08:09:49.177 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 3.0429999999999998e-05 2024-09-18T08:09:49.177 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:09:49.177 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:09:49.177 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-09-18T08:09:49.177 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:13.220647+0000", 2024-09-18T08:09:49.177 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 2.6954999999999999e-05 2024-09-18T08:09:49.178 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-18T08:09:49.178 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-09-18T08:09:49.178 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-18T08:09:49.178 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:09:49.178 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:09:49.178 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.15216.0:86446 8.5 8:a0654d9b:::benchmark_data_smithi007_101583_object86445:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e205)", 2024-09-18T08:09:49.178 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-09-18T08:09:12.450214+0000", 2024-09-18T08:09:49.178 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 36.615505229, 2024-09-18T08:09:49.178 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.77050818399999998, 2024-09-18T08:09:49.178 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-09-18T08:09:49.178 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-09-18T08:09:49.178 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T08:09:49.179 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2024-09-18T08:09:49.179 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.15216", 2024-09-18T08:09:49.179 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "172.21.15.7:0/2596041177", 2024-09-18T08:09:49.179 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 86446 2024-09-18T08:09:49.179 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:09:49.179 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-09-18T08:09:49.179 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:09:49.179 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-09-18T08:09:49.179 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:12.450214+0000", 2024-09-18T08:09:49.179 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-18T08:09:49.179 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:09:49.179 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:09:49.179 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-09-18T08:09:49.179 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:12.450214+0000", 2024-09-18T08:09:49.180 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-18T08:09:49.180 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:09:49.180 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:09:49.180 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-09-18T08:09:49.180 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:12.450216+0000", 2024-09-18T08:09:49.180 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.8050000000000001e-06 2024-09-18T08:09:49.180 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:09:49.180 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:09:49.180 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-09-18T08:09:49.180 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:12.450274+0000", 2024-09-18T08:09:49.180 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 5.7865999999999998e-05 2024-09-18T08:09:49.180 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:09:49.180 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:09:49.181 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-09-18T08:09:49.181 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:12.450290+0000", 2024-09-18T08:09:49.181 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.5817000000000001e-05 2024-09-18T08:09:49.181 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:09:49.181 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:09:49.181 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-09-18T08:09:49.181 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:12.450299+0000", 2024-09-18T08:09:49.181 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 9.3759999999999997e-06 2024-09-18T08:09:49.181 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:09:49.181 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:09:49.181 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-18T08:09:49.181 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:12.450369+0000", 2024-09-18T08:09:49.181 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 6.9474000000000006e-05 2024-09-18T08:09:49.182 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:09:49.182 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:09:49.182 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "waiting for peered", 2024-09-18T08:09:49.182 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:12.450380+0000", 2024-09-18T08:09:49.182 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.168e-05 2024-09-18T08:09:49.182 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:09:49.182 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:09:49.182 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-18T08:09:49.182 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:13.207495+0000", 2024-09-18T08:09:49.182 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.75711499500000001 2024-09-18T08:09:49.182 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:09:49.182 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:09:49.182 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-09-18T08:09:49.182 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:13.207620+0000", 2024-09-18T08:09:49.183 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00012430199999999999 2024-09-18T08:09:49.183 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:09:49.183 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:09:49.183 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "waiting for subops from 6,7", 2024-09-18T08:09:49.183 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:13.207781+0000", 2024-09-18T08:09:49.183 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00016101099999999999 2024-09-18T08:09:49.183 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:09:49.183 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:09:49.183 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:09:49.183 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:13.214075+0000", 2024-09-18T08:09:49.183 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0062940419999999997 2024-09-18T08:09:49.183 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:09:49.183 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:09:49.184 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:09:49.184 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:13.214116+0000", 2024-09-18T08:09:49.184 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4.1517000000000002e-05 2024-09-18T08:09:49.184 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:09:49.184 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:09:49.184 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "op_commit", 2024-09-18T08:09:49.184 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:13.220658+0000", 2024-09-18T08:09:49.184 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0065420809999999999 2024-09-18T08:09:49.184 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:09:49.184 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:09:49.184 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "commit_sent", 2024-09-18T08:09:49.184 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:13.220695+0000", 2024-09-18T08:09:49.184 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 3.6896e-05 2024-09-18T08:09:49.185 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:09:49.185 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:09:49.185 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-09-18T08:09:49.185 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:13.220722+0000", 2024-09-18T08:09:49.185 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 2.7322e-05 2024-09-18T08:09:49.185 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-18T08:09:49.185 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-09-18T08:09:49.185 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-18T08:09:49.185 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:09:49.185 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:09:49.185 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.15216.0:86465 8.5 8:a7ecc445:::benchmark_data_smithi007_101583_object86464:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e205)", 2024-09-18T08:09:49.185 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-09-18T08:09:12.495165+0000", 2024-09-18T08:09:49.185 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 36.570554219000002, 2024-09-18T08:09:49.186 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.72562491600000001, 2024-09-18T08:09:49.186 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-09-18T08:09:49.186 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-09-18T08:09:49.186 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T08:09:49.186 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2024-09-18T08:09:49.186 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.15216", 2024-09-18T08:09:49.186 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "172.21.15.7:0/2596041177", 2024-09-18T08:09:49.186 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 86465 2024-09-18T08:09:49.186 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:09:49.186 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-09-18T08:09:49.186 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:09:49.186 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-09-18T08:09:49.186 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:12.495165+0000", 2024-09-18T08:09:49.186 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-18T08:09:49.187 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:09:49.187 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:09:49.187 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-09-18T08:09:49.187 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:12.495165+0000", 2024-09-18T08:09:49.187 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-18T08:09:49.187 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:09:49.187 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:09:49.187 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-09-18T08:09:49.187 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:12.495167+0000", 2024-09-18T08:09:49.187 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.9089999999999998e-06 2024-09-18T08:09:49.187 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:09:49.187 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:09:49.187 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-09-18T08:09:49.188 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:12.495284+0000", 2024-09-18T08:09:49.188 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000117228 2024-09-18T08:09:49.188 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:09:49.188 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:09:49.188 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-09-18T08:09:49.188 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:12.495301+0000", 2024-09-18T08:09:49.188 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.7065e-05 2024-09-18T08:09:49.188 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:09:49.188 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:09:49.188 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-09-18T08:09:49.188 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:12.495312+0000", 2024-09-18T08:09:49.188 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.0132999999999999e-05 2024-09-18T08:09:49.188 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:09:49.188 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:09:49.189 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-18T08:09:49.189 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:12.495388+0000", 2024-09-18T08:09:49.189 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 7.6512000000000002e-05 2024-09-18T08:09:49.189 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:09:49.189 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:09:49.189 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "waiting for peered", 2024-09-18T08:09:49.189 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:12.495402+0000", 2024-09-18T08:09:49.189 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.3376e-05 2024-09-18T08:09:49.189 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:09:49.189 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:09:49.189 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-18T08:09:49.189 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:13.211447+0000", 2024-09-18T08:09:49.189 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.71604507500000003 2024-09-18T08:09:49.190 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:09:49.190 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:09:49.190 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-09-18T08:09:49.190 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:13.211727+0000", 2024-09-18T08:09:49.190 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00028030299999999998 2024-09-18T08:09:49.190 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:09:49.190 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:09:49.190 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "waiting for subops from 6,7", 2024-09-18T08:09:49.190 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:13.211872+0000", 2024-09-18T08:09:49.190 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00014542799999999999 2024-09-18T08:09:49.190 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:09:49.190 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:09:49.190 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:09:49.191 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:13.219374+0000", 2024-09-18T08:09:49.191 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0075019090000000002 2024-09-18T08:09:49.191 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:09:49.191 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:09:49.191 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:09:49.191 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:13.220515+0000", 2024-09-18T08:09:49.191 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0011411839999999999 2024-09-18T08:09:49.191 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:09:49.191 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:09:49.191 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "op_commit", 2024-09-18T08:09:49.191 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:13.220727+0000", 2024-09-18T08:09:49.191 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00021189200000000001 2024-09-18T08:09:49.191 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:09:49.191 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:09:49.192 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "commit_sent", 2024-09-18T08:09:49.192 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:13.220765+0000", 2024-09-18T08:09:49.192 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 3.7319e-05 2024-09-18T08:09:49.192 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:09:49.192 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:09:49.192 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-09-18T08:09:49.192 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:13.220790+0000", 2024-09-18T08:09:49.192 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 2.5582999999999999e-05 2024-09-18T08:09:49.192 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-18T08:09:49.192 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-09-18T08:09:49.192 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-18T08:09:49.192 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:09:49.192 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:09:49.193 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.15216.0:86475 8.5 8:a4f19050:::benchmark_data_smithi007_101583_object86474:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e205)", 2024-09-18T08:09:49.193 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-09-18T08:09:12.523844+0000", 2024-09-18T08:09:49.193 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 36.541875900999997, 2024-09-18T08:09:49.193 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.69708282899999996, 2024-09-18T08:09:49.193 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-09-18T08:09:49.193 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-09-18T08:09:49.193 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T08:09:49.193 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2024-09-18T08:09:49.193 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.15216", 2024-09-18T08:09:49.193 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "172.21.15.7:0/2596041177", 2024-09-18T08:09:49.193 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 86475 2024-09-18T08:09:49.193 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:09:49.193 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-09-18T08:09:49.193 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:09:49.194 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-09-18T08:09:49.194 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:12.523844+0000", 2024-09-18T08:09:49.194 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-18T08:09:49.194 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:09:49.194 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:09:49.194 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-09-18T08:09:49.194 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:12.523844+0000", 2024-09-18T08:09:49.194 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-18T08:09:49.194 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:09:49.194 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:09:49.194 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-09-18T08:09:49.194 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:12.523846+0000", 2024-09-18T08:09:49.194 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 2.0320000000000002e-06 2024-09-18T08:09:49.194 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:09:49.195 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:09:49.195 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-09-18T08:09:49.195 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:12.523886+0000", 2024-09-18T08:09:49.195 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4.0763000000000003e-05 2024-09-18T08:09:49.195 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:09:49.195 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:09:49.195 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-09-18T08:09:49.195 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:12.523911+0000", 2024-09-18T08:09:49.195 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 2.4202e-05 2024-09-18T08:09:49.195 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:09:49.195 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:09:49.195 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-09-18T08:09:49.195 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:12.523920+0000", 2024-09-18T08:09:49.195 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 8.9330000000000007e-06 2024-09-18T08:09:49.196 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:09:49.196 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:09:49.196 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-18T08:09:49.196 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:12.523990+0000", 2024-09-18T08:09:49.196 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 7.0505000000000003e-05 2024-09-18T08:09:49.196 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:09:49.196 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:09:49.196 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "waiting for peered", 2024-09-18T08:09:49.196 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:12.524004+0000", 2024-09-18T08:09:49.196 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.3607e-05 2024-09-18T08:09:49.196 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:09:49.196 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:09:49.196 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-18T08:09:49.197 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:13.214163+0000", 2024-09-18T08:09:49.197 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.69015934999999995 2024-09-18T08:09:49.197 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:09:49.197 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:09:49.197 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-09-18T08:09:49.197 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:13.214291+0000", 2024-09-18T08:09:49.197 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00012824399999999999 2024-09-18T08:09:49.197 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:09:49.197 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:09:49.197 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "waiting for subops from 6,7", 2024-09-18T08:09:49.197 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:13.214455+0000", 2024-09-18T08:09:49.197 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00016422499999999999 2024-09-18T08:09:49.197 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:09:49.197 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:09:49.198 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "op_commit", 2024-09-18T08:09:49.198 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:13.220795+0000", 2024-09-18T08:09:49.198 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0063397560000000002 2024-09-18T08:09:49.198 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:09:49.198 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:09:49.198 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:09:49.198 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:13.220825+0000", 2024-09-18T08:09:49.198 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 2.976e-05 2024-09-18T08:09:49.198 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:09:49.198 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:09:49.198 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:09:49.198 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:13.220855+0000", 2024-09-18T08:09:49.198 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 3.0043999999999999e-05 2024-09-18T08:09:49.198 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:09:49.198 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:09:49.199 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "commit_sent", 2024-09-18T08:09:49.199 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:13.220889+0000", 2024-09-18T08:09:49.199 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 3.4251000000000001e-05 2024-09-18T08:09:49.199 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:09:49.199 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:09:49.199 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-09-18T08:09:49.199 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:13.220926+0000", 2024-09-18T08:09:49.199 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 3.7156999999999997e-05 2024-09-18T08:09:49.199 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-18T08:09:49.199 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-09-18T08:09:49.199 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-18T08:09:49.199 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:09:49.199 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:09:49.199 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.15216.0:86499 8.5 8:ab3bde63:::benchmark_data_smithi007_101583_object86498:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e205)", 2024-09-18T08:09:49.200 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-09-18T08:09:12.639820+0000", 2024-09-18T08:09:49.200 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 36.425899114000003, 2024-09-18T08:09:49.200 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.59097304500000003, 2024-09-18T08:09:49.200 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-09-18T08:09:49.200 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-09-18T08:09:49.200 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T08:09:49.200 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2024-09-18T08:09:49.200 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.15216", 2024-09-18T08:09:49.200 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "172.21.15.7:0/2596041177", 2024-09-18T08:09:49.200 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 86499 2024-09-18T08:09:49.200 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:09:49.200 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-09-18T08:09:49.200 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:09:49.200 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-09-18T08:09:49.201 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:12.639820+0000", 2024-09-18T08:09:49.201 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-18T08:09:49.201 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:09:49.201 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:09:49.201 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-09-18T08:09:49.201 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:12.639820+0000", 2024-09-18T08:09:49.201 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-18T08:09:49.201 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:09:49.201 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:09:49.201 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-09-18T08:09:49.201 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:12.639822+0000", 2024-09-18T08:09:49.201 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.4440000000000001e-06 2024-09-18T08:09:49.201 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:09:49.201 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:09:49.202 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-09-18T08:09:49.202 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:12.639872+0000", 2024-09-18T08:09:49.202 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 5.0296000000000001e-05 2024-09-18T08:09:49.202 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:09:49.202 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:09:49.202 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-09-18T08:09:49.202 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:12.639885+0000", 2024-09-18T08:09:49.202 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.3173000000000001e-05 2024-09-18T08:09:49.202 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:09:49.202 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:09:49.202 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-09-18T08:09:49.202 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:12.639894+0000", 2024-09-18T08:09:49.202 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 8.5310000000000001e-06 2024-09-18T08:09:49.202 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:09:49.203 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:09:49.203 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-18T08:09:49.203 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:12.639972+0000", 2024-09-18T08:09:49.203 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 7.8307999999999996e-05 2024-09-18T08:09:49.203 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:09:49.203 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:09:49.203 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "waiting for peered", 2024-09-18T08:09:49.203 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:12.639980+0000", 2024-09-18T08:09:49.203 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 8.0669999999999992e-06 2024-09-18T08:09:49.203 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:09:49.203 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:09:49.203 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-18T08:09:49.203 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:13.216786+0000", 2024-09-18T08:09:49.203 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.57680564199999995 2024-09-18T08:09:49.204 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:09:49.204 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:09:49.204 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-09-18T08:09:49.204 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:13.216933+0000", 2024-09-18T08:09:49.204 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00014732999999999999 2024-09-18T08:09:49.204 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:09:49.204 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:09:49.204 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "waiting for subops from 6,7", 2024-09-18T08:09:49.204 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:13.217061+0000", 2024-09-18T08:09:49.204 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000128042 2024-09-18T08:09:49.204 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:09:49.204 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:09:49.204 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:09:49.205 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:13.220967+0000", 2024-09-18T08:09:49.205 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0039059939999999999 2024-09-18T08:09:49.205 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:09:49.205 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:09:49.205 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "op_commit", 2024-09-18T08:09:49.205 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:13.228425+0000", 2024-09-18T08:09:49.205 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.007457623 2024-09-18T08:09:49.205 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:09:49.205 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:09:49.205 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:09:49.205 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:13.230681+0000", 2024-09-18T08:09:49.205 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0022558280000000001 2024-09-18T08:09:49.205 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:09:49.206 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:09:49.206 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "commit_sent", 2024-09-18T08:09:49.206 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:13.230747+0000", 2024-09-18T08:09:49.206 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 6.5881000000000002e-05 2024-09-18T08:09:49.206 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:09:49.206 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:09:49.206 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-09-18T08:09:49.206 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:13.230793+0000", 2024-09-18T08:09:49.206 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4.6885999999999998e-05 2024-09-18T08:09:49.206 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-18T08:09:49.206 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-09-18T08:09:49.206 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-18T08:09:49.206 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:09:49.206 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:09:49.207 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.15216.0:109892 8.2 8:49b39456:::benchmark_data_smithi007_101583_object9140:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e213)", 2024-09-18T08:09:49.207 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-09-18T08:09:37.528670+0000", 2024-09-18T08:09:49.207 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 11.537049803, 2024-09-18T08:09:49.207 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.080459064999999996, 2024-09-18T08:09:49.207 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-09-18T08:09:49.207 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-09-18T08:09:49.207 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T08:09:49.207 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2024-09-18T08:09:49.207 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.15216", 2024-09-18T08:09:49.207 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "172.21.15.7:0/2596041177", 2024-09-18T08:09:49.207 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 109892 2024-09-18T08:09:49.207 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:09:49.207 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-09-18T08:09:49.208 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:09:49.208 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-09-18T08:09:49.208 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:37.528670+0000", 2024-09-18T08:09:49.208 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-18T08:09:49.208 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:09:49.208 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:09:49.208 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-09-18T08:09:49.208 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:37.528670+0000", 2024-09-18T08:09:49.208 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-18T08:09:49.208 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:09:49.208 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:09:49.208 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-09-18T08:09:49.208 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:37.528671+0000", 2024-09-18T08:09:49.208 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.694e-06 2024-09-18T08:09:49.209 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:09:49.209 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:09:49.209 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-09-18T08:09:49.209 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:37.528677+0000", 2024-09-18T08:09:49.209 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 5.835e-06 2024-09-18T08:09:49.209 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:09:49.209 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:09:49.209 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-09-18T08:09:49.209 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:37.528693+0000", 2024-09-18T08:09:49.209 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.5764e-05 2024-09-18T08:09:49.209 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:09:49.209 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:09:49.209 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-09-18T08:09:49.210 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:37.528700+0000", 2024-09-18T08:09:49.210 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 7.2840000000000001e-06 2024-09-18T08:09:49.210 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:09:49.210 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:09:49.210 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-18T08:09:49.210 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:37.528764+0000", 2024-09-18T08:09:49.210 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 6.3433000000000001e-05 2024-09-18T08:09:49.210 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:09:49.210 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:09:49.210 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-09-18T08:09:49.210 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:37.528894+0000", 2024-09-18T08:09:49.210 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000129908 2024-09-18T08:09:49.210 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:09:49.210 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:09:49.211 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "waiting for subops from 0,5", 2024-09-18T08:09:49.211 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:37.529017+0000", 2024-09-18T08:09:49.211 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000123034 2024-09-18T08:09:49.211 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:09:49.211 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:09:49.211 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "op_commit", 2024-09-18T08:09:49.211 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:37.532304+0000", 2024-09-18T08:09:49.211 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0032873220000000001 2024-09-18T08:09:49.211 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:09:49.211 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:09:49.211 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:09:49.211 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:37.533683+0000", 2024-09-18T08:09:49.211 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0013787890000000001 2024-09-18T08:09:49.211 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:09:49.212 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:09:49.212 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:09:49.212 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:37.608992+0000", 2024-09-18T08:09:49.212 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.075309097000000005 2024-09-18T08:09:49.212 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:09:49.212 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:09:49.212 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "commit_sent", 2024-09-18T08:09:49.212 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:37.609083+0000", 2024-09-18T08:09:49.212 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 9.1189999999999999e-05 2024-09-18T08:09:49.212 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:09:49.212 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:09:49.212 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-09-18T08:09:49.212 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:37.609129+0000", 2024-09-18T08:09:49.212 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4.5714999999999999e-05 2024-09-18T08:09:49.213 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-18T08:09:49.213 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-09-18T08:09:49.213 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-18T08:09:49.213 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-18T08:09:49.213 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-09-18T08:09:49.213 INFO:teuthology.orchestra.run.smithi050.stdout:} 2024-09-18T08:09:49.213 INFO:journalctl@ceph.iscsi.iscsi.a.smithi050.stdout:Sep 18 08:09:49 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-iscsi-iscsi-a[57782]: debug there is no tcmu-runner data available 2024-09-18T08:09:49.274 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2 2024-09-18T08:09:49.315 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:09:49.316 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:09:49.316 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2' 2024-09-18T08:09:49.417 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5 2024-09-18T08:09:49.454 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:09:49.454 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:09:49.455 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5' 2024-09-18T08:09:49.555 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7 2024-09-18T08:09:49.591 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:09:49.592 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:09:49.592 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7' 2024-09-18T08:09:49.604 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid 779349ca-7593-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-18T08:09:49.690 INFO:teuthology.orchestra.run.smithi007.stdout:{"pg_ready":true,"pg_map":{"version":339,"stamp":"2024-09-18T08:09:47.218223+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":5570173491,"num_objects":85185,"num_object_clones":0,"num_object_copies":255555,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1921,"num_objects_unfound":0,"num_objects_dirty":85185,"num_whiteouts":0,"num_read":893,"num_read_kb":645,"num_write":116950,"num_write_kb":6447982,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":22565,"num_bytes_recovered":1462967506,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":79708,"ondisk_log_size":79708,"up":474,"acting":473,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":481,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":19330440,"kb_used_data":17464904,"kb_used_omap":12,"kb_used_meta":1865267,"kb_avail":730630776,"statfs":{"total":767960285184,"available":748165914624,"internally_reserved":0,"allocated":17884061696,"data_stored":17880970249,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12750,"internal_metadata":1910033970},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-685768704,"num_objects":-10464,"num_object_clones":0,"num_object_copies":-31392,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-670,"num_objects_unfound":0,"num_objects_dirty":-10464,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":10464,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":670,"num_bytes_recovered":43909120,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.002663"},"pg_stats":[{"pgid":"3.1f","version":"0'0","reported_seq":346,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-09-18T08:09:34.404734+0000","last_change":"2024-09-18T08:08:52.697893+0000","last_active":"2024-09-18T08:09:34.404734+0000","last_peered":"2024-09-18T08:09:34.404734+0000","last_clean":"2024-09-18T08:09:34.404734+0000","last_became_active":"2024-09-18T08:08:52.697499+0000","last_became_peered":"2024-09-18T08:08:52.697499+0000","last_unstale":"2024-09-18T08:09:34.404734+0000","last_undegraded":"2024-09-18T08:09:34.404734+0000","last_fullsized":"2024-09-18T08:09:34.404734+0000","mapping_epoch":198,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":199,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:08:06.642662+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:46.860092+0000","last_clean_scrub_stamp":"2024-09-18T08:08:06.642662+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"69'9","reported_seq":84,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-09-18T08:09:34.393247+0000","last_change":"2024-09-18T08:08:18.575611+0000","last_active":"2024-09-18T08:09:34.393247+0000","last_peered":"2024-09-18T08:09:34.393247+0000","last_clean":"2024-09-18T08:09:34.393247+0000","last_became_active":"2024-09-18T08:08:18.537856+0000","last_became_peered":"2024-09-18T08:08:18.537856+0000","last_unstale":"2024-09-18T08:09:34.393247+0000","last_undegraded":"2024-09-18T08:09:34.393247+0000","last_fullsized":"2024-09-18T08:09:34.393247+0000","mapping_epoch":181,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":182,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'9","last_scrub_stamp":"2024-09-18T08:05:08.696796+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_clean_scrub_stamp":"2024-09-18T08:05:08.696796+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":16,"num_read_kb":10,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":331,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-09-18T08:09:34.410820+0000","last_change":"2024-09-18T08:07:18.737364+0000","last_active":"2024-09-18T08:09:34.410820+0000","last_peered":"2024-09-18T08:09:34.410820+0000","last_clean":"2024-09-18T08:09:34.410820+0000","last_became_active":"2024-09-18T08:02:51.907140+0000","last_became_peered":"2024-09-18T08:02:51.907140+0000","last_unstale":"2024-09-18T08:09:34.410820+0000","last_undegraded":"2024-09-18T08:09:34.410820+0000","last_fullsized":"2024-09-18T08:09:34.410820+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:07:18.737122+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:50.886830+0000","last_clean_scrub_stamp":"2024-09-18T08:07:18.737122+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:10:14.742960+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":74,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-09-18T08:09:34.393980+0000","last_change":"2024-09-18T08:08:18.545192+0000","last_active":"2024-09-18T08:09:34.393980+0000","last_peered":"2024-09-18T08:09:34.393980+0000","last_clean":"2024-09-18T08:09:34.393980+0000","last_became_active":"2024-09-18T08:08:18.540705+0000","last_became_peered":"2024-09-18T08:08:18.540705+0000","last_unstale":"2024-09-18T08:09:34.393980+0000","last_undegraded":"2024-09-18T08:09:34.393980+0000","last_fullsized":"2024-09-18T08:09:34.393980+0000","mapping_epoch":181,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":182,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:07:08.451865+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_clean_scrub_stamp":"2024-09-18T08:07:08.451865+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":334,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-09-18T08:09:34.398027+0000","last_change":"2024-09-18T08:07:13.244647+0000","last_active":"2024-09-18T08:09:34.398027+0000","last_peered":"2024-09-18T08:09:34.398027+0000","last_clean":"2024-09-18T08:09:34.398027+0000","last_became_active":"2024-09-18T08:06:24.128511+0000","last_became_peered":"2024-09-18T08:06:24.128511+0000","last_unstale":"2024-09-18T08:09:34.398027+0000","last_undegraded":"2024-09-18T08:09:34.398027+0000","last_fullsized":"2024-09-18T08:09:34.398027+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:07:13.244555+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_clean_scrub_stamp":"2024-09-18T08:07:13.244555+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":347,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-09-18T08:09:34.394389+0000","last_change":"2024-09-18T08:07:10.257236+0000","last_active":"2024-09-18T08:09:34.394389+0000","last_peered":"2024-09-18T08:09:34.394389+0000","last_clean":"2024-09-18T08:09:34.394389+0000","last_became_active":"2024-09-18T08:06:24.126194+0000","last_became_peered":"2024-09-18T08:06:24.126194+0000","last_unstale":"2024-09-18T08:09:34.394389+0000","last_undegraded":"2024-09-18T08:09:34.394389+0000","last_fullsized":"2024-09-18T08:09:34.394389+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:07:10.257170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:46.860092+0000","last_clean_scrub_stamp":"2024-09-18T08:07:10.257170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":37,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"69'15","reported_seq":345,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-09-18T08:09:34.396329+0000","last_change":"2024-09-18T08:09:32.575836+0000","last_active":"2024-09-18T08:09:34.396329+0000","last_peered":"2024-09-18T08:09:34.396329+0000","last_clean":"2024-09-18T08:09:34.396329+0000","last_became_active":"2024-09-18T08:02:49.886106+0000","last_became_peered":"2024-09-18T08:02:49.886106+0000","last_unstale":"2024-09-18T08:09:34.396329+0000","last_undegraded":"2024-09-18T08:09:34.396329+0000","last_fullsized":"2024-09-18T08:09:34.396329+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'15","last_scrub_stamp":"2024-09-18T08:09:32.575741+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_clean_scrub_stamp":"2024-09-18T08:09:32.575741+0000","objects_scrubbed":7,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:10:59.141685+0000","scrub_duration":40,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":74,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-09-18T08:09:34.393249+0000","last_change":"2024-09-18T08:08:18.543316+0000","last_active":"2024-09-18T08:09:34.393249+0000","last_peered":"2024-09-18T08:09:34.393249+0000","last_clean":"2024-09-18T08:09:34.393249+0000","last_became_active":"2024-09-18T08:08:18.538645+0000","last_became_peered":"2024-09-18T08:08:18.538645+0000","last_unstale":"2024-09-18T08:09:34.393249+0000","last_undegraded":"2024-09-18T08:09:34.393249+0000","last_fullsized":"2024-09-18T08:09:34.393249+0000","mapping_epoch":181,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":182,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:08:09.142188+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:50.886830+0000","last_clean_scrub_stamp":"2024-09-18T08:08:09.142188+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":359,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-09-18T08:09:34.396908+0000","last_change":"2024-09-18T08:08:18.548588+0000","last_active":"2024-09-18T08:09:34.396908+0000","last_peered":"2024-09-18T08:09:34.396908+0000","last_clean":"2024-09-18T08:09:34.396908+0000","last_became_active":"2024-09-18T08:08:18.548203+0000","last_became_peered":"2024-09-18T08:08:18.548203+0000","last_unstale":"2024-09-18T08:09:34.396908+0000","last_undegraded":"2024-09-18T08:09:34.396908+0000","last_fullsized":"2024-09-18T08:09:34.396908+0000","mapping_epoch":181,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":182,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:06:14.027077+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:46.860092+0000","last_clean_scrub_stamp":"2024-09-18T08:06:14.027077+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"69'9","reported_seq":84,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-09-18T08:09:34.394165+0000","last_change":"2024-09-18T08:08:18.631597+0000","last_active":"2024-09-18T08:09:34.394165+0000","last_peered":"2024-09-18T08:09:34.394165+0000","last_clean":"2024-09-18T08:09:34.394165+0000","last_became_active":"2024-09-18T08:08:18.550960+0000","last_became_peered":"2024-09-18T08:08:18.550960+0000","last_unstale":"2024-09-18T08:09:34.394165+0000","last_undegraded":"2024-09-18T08:09:34.394165+0000","last_fullsized":"2024-09-18T08:09:34.394165+0000","mapping_epoch":181,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":182,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_clean_scrub_stamp":"2024-09-18T08:02:48.870718+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":16,"num_read_kb":10,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":323,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-09-18T08:09:34.396118+0000","last_change":"2024-09-18T08:05:33.935844+0000","last_active":"2024-09-18T08:09:34.396118+0000","last_peered":"2024-09-18T08:09:34.396118+0000","last_clean":"2024-09-18T08:09:34.396118+0000","last_became_active":"2024-09-18T08:02:51.907296+0000","last_became_peered":"2024-09-18T08:02:51.907296+0000","last_unstale":"2024-09-18T08:09:34.396118+0000","last_undegraded":"2024-09-18T08:09:34.396118+0000","last_fullsized":"2024-09-18T08:09:34.396118+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:05:33.935798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:50.886830+0000","last_clean_scrub_stamp":"2024-09-18T08:05:33.935798+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":320,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-09-18T08:09:34.403934+0000","last_change":"2024-09-18T08:08:14.646073+0000","last_active":"2024-09-18T08:09:34.403934+0000","last_peered":"2024-09-18T08:09:34.403934+0000","last_clean":"2024-09-18T08:09:34.403934+0000","last_became_active":"2024-09-18T08:02:53.913519+0000","last_became_peered":"2024-09-18T08:02:53.913519+0000","last_unstale":"2024-09-18T08:09:34.403934+0000","last_undegraded":"2024-09-18T08:09:34.403934+0000","last_fullsized":"2024-09-18T08:09:34.403934+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:08:14.645993+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_clean_scrub_stamp":"2024-09-18T08:08:14.645993+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":337,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-09-18T08:09:34.397037+0000","last_change":"2024-09-18T08:08:18.624260+0000","last_active":"2024-09-18T08:09:34.397037+0000","last_peered":"2024-09-18T08:09:34.397037+0000","last_clean":"2024-09-18T08:09:34.397037+0000","last_became_active":"2024-09-18T08:08:18.623757+0000","last_became_peered":"2024-09-18T08:08:18.623757+0000","last_unstale":"2024-09-18T08:09:34.397037+0000","last_undegraded":"2024-09-18T08:09:34.397037+0000","last_fullsized":"2024-09-18T08:09:34.397037+0000","mapping_epoch":181,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":182,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:04:32.849119+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:46.860092+0000","last_clean_scrub_stamp":"2024-09-18T08:04:32.849119+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1b","version":"69'5","reported_seq":80,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-09-18T08:09:34.393925+0000","last_change":"2024-09-18T08:08:18.651427+0000","last_active":"2024-09-18T08:09:34.393925+0000","last_peered":"2024-09-18T08:09:34.393925+0000","last_clean":"2024-09-18T08:09:34.393925+0000","last_became_active":"2024-09-18T08:08:18.567126+0000","last_became_peered":"2024-09-18T08:08:18.567126+0000","last_unstale":"2024-09-18T08:09:34.393925+0000","last_undegraded":"2024-09-18T08:09:34.393925+0000","last_fullsized":"2024-09-18T08:09:34.393925+0000","mapping_epoch":181,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":182,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'5","last_scrub_stamp":"2024-09-18T08:06:09.465979+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_clean_scrub_stamp":"2024-09-18T08:06:09.465979+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":10,"num_read_kb":6,"num_write":6,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":353,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-09-18T08:09:34.408169+0000","last_change":"2024-09-18T08:09:12.674889+0000","last_active":"2024-09-18T08:09:34.408169+0000","last_peered":"2024-09-18T08:09:34.408169+0000","last_clean":"2024-09-18T08:09:34.408169+0000","last_became_active":"2024-09-18T08:08:18.524932+0000","last_became_peered":"2024-09-18T08:08:18.524932+0000","last_unstale":"2024-09-18T08:09:34.408169+0000","last_undegraded":"2024-09-18T08:09:34.408169+0000","last_fullsized":"2024-09-18T08:09:34.408169+0000","mapping_epoch":181,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":182,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:09:12.674843+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:50.886830+0000","last_clean_scrub_stamp":"2024-09-18T08:09:12.674843+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:10:16.027634+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":320,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-09-18T08:09:34.396440+0000","last_change":"2024-09-18T08:06:44.518882+0000","last_active":"2024-09-18T08:09:34.396440+0000","last_peered":"2024-09-18T08:09:34.396440+0000","last_clean":"2024-09-18T08:09:34.396440+0000","last_became_active":"2024-09-18T08:02:53.911648+0000","last_became_peered":"2024-09-18T08:02:53.911648+0000","last_unstale":"2024-09-18T08:09:34.396440+0000","last_undegraded":"2024-09-18T08:09:34.396440+0000","last_fullsized":"2024-09-18T08:09:34.396440+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:06:44.518835+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_clean_scrub_stamp":"2024-09-18T08:06:44.518835+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":74,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-09-18T08:09:34.395143+0000","last_change":"2024-09-18T08:08:18.544203+0000","last_active":"2024-09-18T08:09:34.395143+0000","last_peered":"2024-09-18T08:09:34.395143+0000","last_clean":"2024-09-18T08:09:34.395143+0000","last_became_active":"2024-09-18T08:08:18.539604+0000","last_became_peered":"2024-09-18T08:08:18.539604+0000","last_unstale":"2024-09-18T08:09:34.395143+0000","last_undegraded":"2024-09-18T08:09:34.395143+0000","last_fullsized":"2024-09-18T08:09:34.395143+0000","mapping_epoch":181,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":182,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:05:40.477965+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_clean_scrub_stamp":"2024-09-18T08:05:40.477965+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":345,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-09-18T08:09:34.405428+0000","last_change":"2024-09-18T08:08:18.524124+0000","last_active":"2024-09-18T08:09:34.405428+0000","last_peered":"2024-09-18T08:09:34.405428+0000","last_clean":"2024-09-18T08:09:34.405428+0000","last_became_active":"2024-09-18T08:08:18.523611+0000","last_became_peered":"2024-09-18T08:08:18.523611+0000","last_unstale":"2024-09-18T08:09:34.405428+0000","last_undegraded":"2024-09-18T08:09:34.405428+0000","last_fullsized":"2024-09-18T08:09:34.405428+0000","mapping_epoch":181,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":182,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:05:27.476144+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:46.860092+0000","last_clean_scrub_stamp":"2024-09-18T08:05:27.476144+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:10:23.059202+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1c","version":"69'15","reported_seq":56,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-09-18T08:09:34.400914+0000","last_change":"2024-09-18T08:08:52.708963+0000","last_active":"2024-09-18T08:09:34.400914+0000","last_peered":"2024-09-18T08:09:34.400914+0000","last_clean":"2024-09-18T08:09:34.400914+0000","last_became_active":"2024-09-18T08:08:52.690788+0000","last_became_peered":"2024-09-18T08:08:52.690788+0000","last_unstale":"2024-09-18T08:09:34.400914+0000","last_undegraded":"2024-09-18T08:09:34.400914+0000","last_fullsized":"2024-09-18T08:09:34.400914+0000","mapping_epoch":198,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":199,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'15","last_scrub_stamp":"2024-09-18T08:07:39.530340+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_clean_scrub_stamp":"2024-09-18T08:07:39.530340+0000","objects_scrubbed":7,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":353,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-09-18T08:09:34.410223+0000","last_change":"2024-09-18T08:09:11.693711+0000","last_active":"2024-09-18T08:09:34.410223+0000","last_peered":"2024-09-18T08:09:34.410223+0000","last_clean":"2024-09-18T08:09:34.410223+0000","last_became_active":"2024-09-18T08:08:18.538505+0000","last_became_peered":"2024-09-18T08:08:18.538505+0000","last_unstale":"2024-09-18T08:09:34.410223+0000","last_undegraded":"2024-09-18T08:09:34.410223+0000","last_fullsized":"2024-09-18T08:09:34.410223+0000","mapping_epoch":181,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":182,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:09:11.693643+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:50.886830+0000","last_clean_scrub_stamp":"2024-09-18T08:09:11.693643+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:10:34.429057+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":335,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-09-18T08:09:36.592348+0000","last_change":"2024-09-18T08:09:36.592348+0000","last_active":"2024-09-18T08:09:36.592348+0000","last_peered":"2024-09-18T08:09:36.592348+0000","last_clean":"2024-09-18T08:09:36.592348+0000","last_became_active":"2024-09-18T08:06:24.128659+0000","last_became_peered":"2024-09-18T08:06:24.128659+0000","last_unstale":"2024-09-18T08:09:36.592348+0000","last_undegraded":"2024-09-18T08:09:36.592348+0000","last_fullsized":"2024-09-18T08:09:36.592348+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:09:36.592281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_clean_scrub_stamp":"2024-09-18T08:09:36.592281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:10:46.573780+0000","scrub_duration":16,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":74,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-09-18T08:09:34.393832+0000","last_change":"2024-09-18T08:08:18.536559+0000","last_active":"2024-09-18T08:09:34.393832+0000","last_peered":"2024-09-18T08:09:34.393832+0000","last_clean":"2024-09-18T08:09:34.393832+0000","last_became_active":"2024-09-18T08:08:18.536084+0000","last_became_peered":"2024-09-18T08:08:18.536084+0000","last_unstale":"2024-09-18T08:09:34.393832+0000","last_undegraded":"2024-09-18T08:09:34.393832+0000","last_fullsized":"2024-09-18T08:09:34.393832+0000","mapping_epoch":181,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":182,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:07:13.543722+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:46.860092+0000","last_clean_scrub_stamp":"2024-09-18T08:07:13.543722+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"69'12","reported_seq":389,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-09-18T08:09:34.393666+0000","last_change":"2024-09-18T08:08:18.583915+0000","last_active":"2024-09-18T08:09:34.393666+0000","last_peered":"2024-09-18T08:09:34.393666+0000","last_clean":"2024-09-18T08:09:34.393666+0000","last_became_active":"2024-09-18T08:08:18.543675+0000","last_became_peered":"2024-09-18T08:08:18.543675+0000","last_unstale":"2024-09-18T08:09:34.393666+0000","last_undegraded":"2024-09-18T08:09:34.393666+0000","last_fullsized":"2024-09-18T08:09:34.393666+0000","mapping_epoch":181,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":182,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'12","last_scrub_stamp":"2024-09-18T08:07:20.251529+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_clean_scrub_stamp":"2024-09-18T08:07:20.251529+0000","objects_scrubbed":5,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":25,"num_read_kb":16,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":74,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-09-18T08:09:34.394829+0000","last_change":"2024-09-18T08:08:18.532482+0000","last_active":"2024-09-18T08:09:34.394829+0000","last_peered":"2024-09-18T08:09:34.394829+0000","last_clean":"2024-09-18T08:09:34.394829+0000","last_became_active":"2024-09-18T08:08:18.531438+0000","last_became_peered":"2024-09-18T08:08:18.531438+0000","last_unstale":"2024-09-18T08:09:34.394829+0000","last_undegraded":"2024-09-18T08:09:34.394829+0000","last_fullsized":"2024-09-18T08:09:34.394829+0000","mapping_epoch":181,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":182,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:06:12.434179+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:50.886830+0000","last_clean_scrub_stamp":"2024-09-18T08:06:12.434179+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1c","version":"69'1","reported_seq":163,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-09-18T08:09:34.396676+0000","last_change":"2024-09-18T08:06:46.142143+0000","last_active":"2024-09-18T08:09:34.396676+0000","last_peered":"2024-09-18T08:09:34.396676+0000","last_clean":"2024-09-18T08:09:34.396676+0000","last_became_active":"2024-09-18T08:06:46.141661+0000","last_became_peered":"2024-09-18T08:06:46.141661+0000","last_unstale":"2024-09-18T08:09:34.396676+0000","last_undegraded":"2024-09-18T08:09:34.396676+0000","last_fullsized":"2024-09-18T08:09:34.396676+0000","mapping_epoch":143,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":144,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'1","last_scrub_stamp":"2024-09-18T08:05:36.776369+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_clean_scrub_stamp":"2024-09-18T08:05:36.776369+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":403,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,2],"acting":[5,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"62'1","reported_seq":362,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-09-18T08:09:34.410915+0000","last_change":"2024-09-18T08:09:29.633875+0000","last_active":"2024-09-18T08:09:34.410915+0000","last_peered":"2024-09-18T08:09:34.410915+0000","last_clean":"2024-09-18T08:09:34.410915+0000","last_became_active":"2024-09-18T08:08:18.559125+0000","last_became_peered":"2024-09-18T08:08:18.559125+0000","last_unstale":"2024-09-18T08:09:34.410915+0000","last_undegraded":"2024-09-18T08:09:34.410915+0000","last_fullsized":"2024-09-18T08:09:34.410915+0000","mapping_epoch":181,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":182,"parent":"0.0","parent_split_bits":0,"last_scrub":"62'1","last_scrub_stamp":"2024-09-18T08:09:29.633520+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:46.860092+0000","last_clean_scrub_stamp":"2024-09-18T08:09:29.633520+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:10:38.890672+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":92,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1e","version":"69'10","reported_seq":327,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-09-18T08:09:34.405835+0000","last_change":"2024-09-18T08:02:49.892755+0000","last_active":"2024-09-18T08:09:34.405835+0000","last_peered":"2024-09-18T08:09:34.405835+0000","last_clean":"2024-09-18T08:09:34.405835+0000","last_became_active":"2024-09-18T08:02:49.892024+0000","last_became_peered":"2024-09-18T08:02:49.892024+0000","last_unstale":"2024-09-18T08:09:34.405835+0000","last_undegraded":"2024-09-18T08:09:34.405835+0000","last_fullsized":"2024-09-18T08:09:34.405835+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_clean_scrub_stamp":"2024-09-18T08:02:48.870718+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:10:17.106246+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1f","version":"182'13","reported_seq":96,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-09-18T08:09:38.009541+0000","last_change":"2024-09-18T08:08:18.627670+0000","last_active":"2024-09-18T08:09:38.009541+0000","last_peered":"2024-09-18T08:09:38.009541+0000","last_clean":"2024-09-18T08:09:38.009541+0000","last_became_active":"2024-09-18T08:08:18.543095+0000","last_became_peered":"2024-09-18T08:08:18.543095+0000","last_unstale":"2024-09-18T08:09:38.009541+0000","last_undegraded":"2024-09-18T08:09:38.009541+0000","last_fullsized":"2024-09-18T08:09:38.009541+0000","mapping_epoch":181,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":182,"parent":"0.0","parent_split_bits":0,"last_scrub":"173'12","last_scrub_stamp":"2024-09-18T08:08:07.553075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:50.886830+0000","last_clean_scrub_stamp":"2024-09-18T08:08:07.553075+0000","objects_scrubbed":0,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,7],"acting":[4,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.3","version":"213'7457","reported_seq":7634,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-09-18T08:09:43.632746+0000","last_change":"2024-09-18T08:09:28.263522+0000","last_active":"2024-09-18T08:09:43.632746+0000","last_peered":"2024-09-18T08:09:43.632746+0000","last_clean":"2024-09-18T08:09:43.632746+0000","last_became_active":"2024-09-18T08:07:53.672797+0000","last_became_peered":"2024-09-18T08:07:53.672797+0000","last_unstale":"2024-09-18T08:09:43.632746+0000","last_undegraded":"2024-09-18T08:09:43.632746+0000","last_fullsized":"2024-09-18T08:09:43.632746+0000","mapping_epoch":165,"log_start":"183'2600","ondisk_log_start":"183'2600","created":165,"last_epoch_clean":166,"parent":"0.0","parent_split_bits":0,"last_scrub":"211'6448","last_scrub_stamp":"2024-09-18T08:09:28.263434+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:07:52.307205+0000","last_clean_scrub_stamp":"2024-09-18T08:09:28.263434+0000","objects_scrubbed":6448,"log_size":4857,"log_dups_size":0,"ondisk_log_size":4857,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:10:29.843317+0000","scrub_duration":646,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":356450304,"num_objects":5439,"num_object_clones":0,"num_object_copies":16317,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5439,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7457,"num_write_kb":412672,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6],"acting":[1,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.a","version":"74'79","reported_seq":145,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-09-18T08:09:34.395931+0000","last_change":"2024-09-18T08:08:57.463205+0000","last_active":"2024-09-18T08:09:34.395931+0000","last_peered":"2024-09-18T08:09:34.395931+0000","last_clean":"2024-09-18T08:09:34.395931+0000","last_became_active":"2024-09-18T08:07:26.394584+0000","last_became_peered":"2024-09-18T08:07:26.394584+0000","last_unstale":"2024-09-18T08:09:34.395931+0000","last_undegraded":"2024-09-18T08:09:34.395931+0000","last_fullsized":"2024-09-18T08:09:34.395931+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":152,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":4,"last_scrub":"74'79","last_scrub_stamp":"2024-09-18T08:08:57.463113+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:00:11.893742+0000","last_clean_scrub_stamp":"2024-09-18T08:08:57.463113+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:09:57.971184+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,6],"acting":[5,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.f","version":"69'15","reported_seq":399,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-09-18T08:09:34.415322+0000","last_change":"2024-09-18T08:08:56.669262+0000","last_active":"2024-09-18T08:09:34.415322+0000","last_peered":"2024-09-18T08:09:34.415322+0000","last_clean":"2024-09-18T08:09:34.415322+0000","last_became_active":"2024-09-18T08:08:18.540211+0000","last_became_peered":"2024-09-18T08:08:18.540211+0000","last_unstale":"2024-09-18T08:09:34.415322+0000","last_undegraded":"2024-09-18T08:09:34.415322+0000","last_fullsized":"2024-09-18T08:09:34.415322+0000","mapping_epoch":181,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":182,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'15","last_scrub_stamp":"2024-09-18T08:08:56.668397+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_clean_scrub_stamp":"2024-09-18T08:08:56.668397+0000","objects_scrubbed":7,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:10:22.470704+0000","scrub_duration":36,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":333,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-09-18T08:09:34.395370+0000","last_change":"2024-09-18T08:08:18.389332+0000","last_active":"2024-09-18T08:09:34.395370+0000","last_peered":"2024-09-18T08:09:34.395370+0000","last_clean":"2024-09-18T08:09:34.395370+0000","last_became_active":"2024-09-18T08:02:47.879328+0000","last_became_peered":"2024-09-18T08:02:47.879328+0000","last_unstale":"2024-09-18T08:09:34.395370+0000","last_undegraded":"2024-09-18T08:09:34.395370+0000","last_fullsized":"2024-09-18T08:09:34.395370+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:08:18.389273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:46.860092+0000","last_clean_scrub_stamp":"2024-09-18T08:08:18.389273+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:09:46.276520+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.e","version":"182'12","reported_seq":96,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-09-18T08:09:38.009566+0000","last_change":"2024-09-18T08:08:18.560884+0000","last_active":"2024-09-18T08:09:38.009566+0000","last_peered":"2024-09-18T08:09:38.009566+0000","last_clean":"2024-09-18T08:09:38.009566+0000","last_became_active":"2024-09-18T08:08:18.525147+0000","last_became_peered":"2024-09-18T08:08:18.525147+0000","last_unstale":"2024-09-18T08:09:38.009566+0000","last_undegraded":"2024-09-18T08:09:38.009566+0000","last_fullsized":"2024-09-18T08:09:38.009566+0000","mapping_epoch":181,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":182,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'8","last_scrub_stamp":"2024-09-18T08:06:19.459375+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:50.886830+0000","last_clean_scrub_stamp":"2024-09-18T08:06:19.459375+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":329,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-09-18T08:09:34.397110+0000","last_change":"2024-09-18T08:07:19.714477+0000","last_active":"2024-09-18T08:09:34.397110+0000","last_peered":"2024-09-18T08:09:34.397110+0000","last_clean":"2024-09-18T08:09:34.397110+0000","last_became_active":"2024-09-18T08:02:53.909834+0000","last_became_peered":"2024-09-18T08:02:53.909834+0000","last_unstale":"2024-09-18T08:09:34.397110+0000","last_undegraded":"2024-09-18T08:09:34.397110+0000","last_fullsized":"2024-09-18T08:09:34.397110+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:07:19.714430+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_clean_scrub_stamp":"2024-09-18T08:07:19.714430+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.c","version":"213'7246","reported_seq":7353,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-09-18T08:09:43.797195+0000","last_change":"2024-09-18T08:07:53.670091+0000","last_active":"2024-09-18T08:09:43.797195+0000","last_peered":"2024-09-18T08:09:43.797195+0000","last_clean":"2024-09-18T08:09:43.797195+0000","last_became_active":"2024-09-18T08:07:53.669708+0000","last_became_peered":"2024-09-18T08:07:53.669708+0000","last_unstale":"2024-09-18T08:09:43.797195+0000","last_undegraded":"2024-09-18T08:09:43.797195+0000","last_fullsized":"2024-09-18T08:09:43.797195+0000","mapping_epoch":165,"log_start":"178'1800","ondisk_log_start":"178'1800","created":165,"last_epoch_clean":166,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:07:52.307205+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:07:52.307205+0000","last_clean_scrub_stamp":"2024-09-18T08:07:52.307205+0000","objects_scrubbed":0,"log_size":5446,"log_dups_size":0,"ondisk_log_size":5446,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":339738624,"num_objects":5184,"num_object_clones":0,"num_object_copies":15552,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5184,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7246,"num_write_kb":397760,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"74'79","reported_seq":89,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-09-18T08:09:34.393866+0000","last_change":"2024-09-18T08:08:52.690758+0000","last_active":"2024-09-18T08:09:34.393866+0000","last_peered":"2024-09-18T08:09:34.393866+0000","last_clean":"2024-09-18T08:09:34.393866+0000","last_became_active":"2024-09-18T08:08:52.690344+0000","last_became_peered":"2024-09-18T08:08:52.690344+0000","last_unstale":"2024-09-18T08:09:34.393866+0000","last_undegraded":"2024-09-18T08:09:34.393866+0000","last_fullsized":"2024-09-18T08:09:34.393866+0000","mapping_epoch":198,"log_start":"0'0","ondisk_log_start":"0'0","created":152,"last_epoch_clean":199,"parent":"0.0","parent_split_bits":3,"last_scrub":"74'79","last_scrub_stamp":"2024-09-18T08:08:19.036836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:00:11.893742+0000","last_clean_scrub_stamp":"2024-09-18T08:08:19.036836+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,6],"acting":[4,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.0","version":"69'18","reported_seq":361,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-09-18T08:09:34.395113+0000","last_change":"2024-09-18T08:06:44.301130+0000","last_active":"2024-09-18T08:09:34.395113+0000","last_peered":"2024-09-18T08:09:34.395113+0000","last_clean":"2024-09-18T08:09:34.395113+0000","last_became_active":"2024-09-18T08:02:49.886818+0000","last_became_peered":"2024-09-18T08:02:49.886818+0000","last_unstale":"2024-09-18T08:09:34.395113+0000","last_undegraded":"2024-09-18T08:09:34.395113+0000","last_fullsized":"2024-09-18T08:09:34.395113+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'18","last_scrub_stamp":"2024-09-18T08:06:44.300772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_clean_scrub_stamp":"2024-09-18T08:06:44.300772+0000","objects_scrubbed":8,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":34,"num_read_kb":22,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":341,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-09-18T08:09:34.395317+0000","last_change":"2024-09-18T08:07:21.242587+0000","last_active":"2024-09-18T08:09:34.395317+0000","last_peered":"2024-09-18T08:09:34.395317+0000","last_clean":"2024-09-18T08:09:34.395317+0000","last_became_active":"2024-09-18T08:02:47.880225+0000","last_became_peered":"2024-09-18T08:02:47.880225+0000","last_unstale":"2024-09-18T08:09:34.395317+0000","last_undegraded":"2024-09-18T08:09:34.395317+0000","last_fullsized":"2024-09-18T08:09:34.395317+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:07:21.242494+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:46.860092+0000","last_clean_scrub_stamp":"2024-09-18T08:07:21.242494+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":18,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":74,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-09-18T08:09:34.394484+0000","last_change":"2024-09-18T08:08:18.530658+0000","last_active":"2024-09-18T08:09:34.394484+0000","last_peered":"2024-09-18T08:09:34.394484+0000","last_clean":"2024-09-18T08:09:34.394484+0000","last_became_active":"2024-09-18T08:08:18.530341+0000","last_became_peered":"2024-09-18T08:08:18.530341+0000","last_unstale":"2024-09-18T08:09:34.394484+0000","last_undegraded":"2024-09-18T08:09:34.394484+0000","last_fullsized":"2024-09-18T08:09:34.394484+0000","mapping_epoch":181,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":182,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:06:10.468119+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:50.886830+0000","last_clean_scrub_stamp":"2024-09-18T08:06:10.468119+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":74,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-09-18T08:09:34.394076+0000","last_change":"2024-09-18T08:08:18.551527+0000","last_active":"2024-09-18T08:09:34.394076+0000","last_peered":"2024-09-18T08:09:34.394076+0000","last_clean":"2024-09-18T08:09:34.394076+0000","last_became_active":"2024-09-18T08:08:18.543694+0000","last_became_peered":"2024-09-18T08:08:18.543694+0000","last_unstale":"2024-09-18T08:09:34.394076+0000","last_undegraded":"2024-09-18T08:09:34.394076+0000","last_fullsized":"2024-09-18T08:09:34.394076+0000","mapping_epoch":181,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":182,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:04:16.427390+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_clean_scrub_stamp":"2024-09-18T08:04:16.427390+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.d","version":"213'7207","reported_seq":7314,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-09-18T08:09:43.250334+0000","last_change":"2024-09-18T08:07:53.671291+0000","last_active":"2024-09-18T08:09:43.250334+0000","last_peered":"2024-09-18T08:09:43.250334+0000","last_clean":"2024-09-18T08:09:43.250334+0000","last_became_active":"2024-09-18T08:07:53.670773+0000","last_became_peered":"2024-09-18T08:07:53.670773+0000","last_unstale":"2024-09-18T08:09:43.250334+0000","last_undegraded":"2024-09-18T08:09:43.250334+0000","last_fullsized":"2024-09-18T08:09:43.250334+0000","mapping_epoch":165,"log_start":"180'2100","ondisk_log_start":"180'2100","created":165,"last_epoch_clean":166,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:07:52.307205+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:07:52.307205+0000","last_clean_scrub_stamp":"2024-09-18T08:07:52.307205+0000","objects_scrubbed":0,"log_size":5107,"log_dups_size":0,"ondisk_log_size":5107,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":347471896,"num_objects":5303,"num_object_clones":0,"num_object_copies":15909,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5303,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7207,"num_write_kb":400257,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,2],"acting":[5,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.4","version":"74'79","reported_seq":151,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-09-18T08:09:34.393693+0000","last_change":"2024-09-18T08:08:59.376417+0000","last_active":"2024-09-18T08:09:34.393693+0000","last_peered":"2024-09-18T08:09:34.393693+0000","last_clean":"2024-09-18T08:09:34.393693+0000","last_became_active":"2024-09-18T08:07:26.398930+0000","last_became_peered":"2024-09-18T08:07:26.398930+0000","last_unstale":"2024-09-18T08:09:34.393693+0000","last_undegraded":"2024-09-18T08:09:34.393693+0000","last_fullsized":"2024-09-18T08:09:34.393693+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":152,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":3,"last_scrub":"74'79","last_scrub_stamp":"2024-09-18T08:08:59.376337+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:00:11.893742+0000","last_clean_scrub_stamp":"2024-09-18T08:08:59.376337+0000","objects_scrubbed":1,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:10:26.641607+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":327680,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,7],"acting":[3,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"69'14","reported_seq":90,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-09-18T08:09:34.393441+0000","last_change":"2024-09-18T08:08:18.617734+0000","last_active":"2024-09-18T08:09:34.393441+0000","last_peered":"2024-09-18T08:09:34.393441+0000","last_clean":"2024-09-18T08:09:34.393441+0000","last_became_active":"2024-09-18T08:08:18.549671+0000","last_became_peered":"2024-09-18T08:08:18.549671+0000","last_unstale":"2024-09-18T08:09:34.393441+0000","last_undegraded":"2024-09-18T08:09:34.393441+0000","last_fullsized":"2024-09-18T08:09:34.393441+0000","mapping_epoch":181,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":182,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'14","last_scrub_stamp":"2024-09-18T08:08:08.629246+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_clean_scrub_stamp":"2024-09-18T08:08:08.629246+0000","objects_scrubbed":0,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":14,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"62'2","reported_seq":365,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-09-18T08:09:34.406149+0000","last_change":"2024-09-18T08:08:18.565279+0000","last_active":"2024-09-18T08:09:34.406149+0000","last_peered":"2024-09-18T08:09:34.406149+0000","last_clean":"2024-09-18T08:09:34.406149+0000","last_became_active":"2024-09-18T08:08:18.544872+0000","last_became_peered":"2024-09-18T08:08:18.544872+0000","last_unstale":"2024-09-18T08:09:34.406149+0000","last_undegraded":"2024-09-18T08:09:34.406149+0000","last_fullsized":"2024-09-18T08:09:34.406149+0000","mapping_epoch":181,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":182,"parent":"0.0","parent_split_bits":0,"last_scrub":"62'2","last_scrub_stamp":"2024-09-18T08:06:43.607152+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:46.860092+0000","last_clean_scrub_stamp":"2024-09-18T08:06:43.607152+0000","objects_scrubbed":2,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:10:26.053059+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1039,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":2078,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.0","version":"182'10","reported_seq":431,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-09-18T08:09:38.009373+0000","last_change":"2024-09-18T08:08:18.573278+0000","last_active":"2024-09-18T08:09:38.009373+0000","last_peered":"2024-09-18T08:09:38.009373+0000","last_clean":"2024-09-18T08:09:38.009373+0000","last_became_active":"2024-09-18T08:08:18.541756+0000","last_became_peered":"2024-09-18T08:08:18.541756+0000","last_unstale":"2024-09-18T08:09:38.009373+0000","last_undegraded":"2024-09-18T08:09:38.009373+0000","last_fullsized":"2024-09-18T08:09:38.009373+0000","mapping_epoch":181,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":182,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'8","last_scrub_stamp":"2024-09-18T08:04:23.863330+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:50.886830+0000","last_clean_scrub_stamp":"2024-09-18T08:04:23.863330+0000","objects_scrubbed":1,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:09:50.348476+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":343,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-09-18T08:09:34.408559+0000","last_change":"2024-09-18T08:07:49.650645+0000","last_active":"2024-09-18T08:09:34.408559+0000","last_peered":"2024-09-18T08:09:34.408559+0000","last_clean":"2024-09-18T08:09:34.408559+0000","last_became_active":"2024-09-18T08:06:24.140303+0000","last_became_peered":"2024-09-18T08:06:24.140303+0000","last_unstale":"2024-09-18T08:09:34.408559+0000","last_undegraded":"2024-09-18T08:09:34.408559+0000","last_fullsized":"2024-09-18T08:09:34.408559+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:07:49.650577+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_clean_scrub_stamp":"2024-09-18T08:07:49.650577+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:10:13.720079+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.e","version":"213'7359","reported_seq":11701,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-09-18T08:09:43.806589+0000","last_change":"2024-09-18T08:09:17.296111+0000","last_active":"2024-09-18T08:09:43.806589+0000","last_peered":"2024-09-18T08:09:43.806589+0000","last_clean":"2024-09-18T08:09:43.806589+0000","last_became_active":"2024-09-18T08:09:17.286866+0000","last_became_peered":"2024-09-18T08:09:17.286866+0000","last_unstale":"2024-09-18T08:09:43.806589+0000","last_undegraded":"2024-09-18T08:09:43.806589+0000","last_fullsized":"2024-09-18T08:09:43.806589+0000","mapping_epoch":207,"log_start":"180'2000","ondisk_log_start":"180'2000","created":165,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:07:52.307205+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:07:52.307205+0000","last_clean_scrub_stamp":"2024-09-18T08:07:52.307205+0000","objects_scrubbed":0,"log_size":5359,"log_dups_size":0,"ondisk_log_size":5359,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":354484224,"num_objects":5409,"num_object_clones":0,"num_object_copies":16227,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5409,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7359,"num_write_kb":408576,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2107,"num_bytes_recovered":138084352,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,4],"acting":[0,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.7","version":"74'79","reported_seq":146,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-09-18T08:09:34.394648+0000","last_change":"2024-09-18T08:08:57.438699+0000","last_active":"2024-09-18T08:09:34.394648+0000","last_peered":"2024-09-18T08:09:34.394648+0000","last_clean":"2024-09-18T08:09:34.394648+0000","last_became_active":"2024-09-18T08:07:26.398289+0000","last_became_peered":"2024-09-18T08:07:26.398289+0000","last_unstale":"2024-09-18T08:09:34.394648+0000","last_undegraded":"2024-09-18T08:09:34.394648+0000","last_fullsized":"2024-09-18T08:09:34.394648+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":152,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":3,"last_scrub":"74'79","last_scrub_stamp":"2024-09-18T08:08:57.438600+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:00:11.893742+0000","last_clean_scrub_stamp":"2024-09-18T08:08:57.438600+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:10:09.717533+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,7],"acting":[3,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.2","version":"69'10","reported_seq":389,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-09-18T08:09:34.410139+0000","last_change":"2024-09-18T08:08:18.580590+0000","last_active":"2024-09-18T08:09:34.410139+0000","last_peered":"2024-09-18T08:09:34.410139+0000","last_clean":"2024-09-18T08:09:34.410139+0000","last_became_active":"2024-09-18T08:08:18.538711+0000","last_became_peered":"2024-09-18T08:08:18.538711+0000","last_unstale":"2024-09-18T08:09:34.410139+0000","last_undegraded":"2024-09-18T08:09:34.410139+0000","last_fullsized":"2024-09-18T08:09:34.410139+0000","mapping_epoch":181,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":182,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'10","last_scrub_stamp":"2024-09-18T08:07:22.666042+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_clean_scrub_stamp":"2024-09-18T08:07:22.666042+0000","objects_scrubbed":5,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:10:17.688460+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":324,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-09-18T08:09:34.397009+0000","last_change":"2024-09-18T08:03:48.009689+0000","last_active":"2024-09-18T08:09:34.397009+0000","last_peered":"2024-09-18T08:09:34.397009+0000","last_clean":"2024-09-18T08:09:34.397009+0000","last_became_active":"2024-09-18T08:02:47.874727+0000","last_became_peered":"2024-09-18T08:02:47.874727+0000","last_unstale":"2024-09-18T08:09:34.397009+0000","last_undegraded":"2024-09-18T08:09:34.397009+0000","last_fullsized":"2024-09-18T08:09:34.397009+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:03:48.009611+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:46.860092+0000","last_clean_scrub_stamp":"2024-09-18T08:03:48.009611+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.3","version":"138'10","reported_seq":439,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-09-18T08:09:38.009408+0000","last_change":"2024-09-18T08:07:46.695714+0000","last_active":"2024-09-18T08:09:38.009408+0000","last_peered":"2024-09-18T08:09:38.009408+0000","last_clean":"2024-09-18T08:09:38.009408+0000","last_became_active":"2024-09-18T08:06:24.134194+0000","last_became_peered":"2024-09-18T08:06:24.134194+0000","last_unstale":"2024-09-18T08:09:38.009408+0000","last_undegraded":"2024-09-18T08:09:38.009408+0000","last_fullsized":"2024-09-18T08:09:38.009408+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"138'10","last_scrub_stamp":"2024-09-18T08:07:46.695656+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:50.886830+0000","last_clean_scrub_stamp":"2024-09-18T08:07:46.695656+0000","objects_scrubbed":1,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":343,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-09-18T08:09:34.404067+0000","last_change":"2024-09-18T08:08:52.700420+0000","last_active":"2024-09-18T08:09:34.404067+0000","last_peered":"2024-09-18T08:09:34.404067+0000","last_clean":"2024-09-18T08:09:34.404067+0000","last_became_active":"2024-09-18T08:08:52.700080+0000","last_became_peered":"2024-09-18T08:08:52.700080+0000","last_unstale":"2024-09-18T08:09:34.404067+0000","last_undegraded":"2024-09-18T08:09:34.404067+0000","last_fullsized":"2024-09-18T08:09:34.404067+0000","mapping_epoch":198,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":199,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:07:44.739578+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_clean_scrub_stamp":"2024-09-18T08:07:44.739578+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.f","version":"213'7487","reported_seq":23650,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-09-18T08:09:43.784871+0000","last_change":"2024-09-18T08:09:28.367000+0000","last_active":"2024-09-18T08:09:43.784871+0000","last_peered":"2024-09-18T08:09:43.784871+0000","last_clean":"2024-09-18T08:09:43.784871+0000","last_became_active":"2024-09-18T08:09:28.363033+0000","last_became_peered":"2024-09-18T08:09:28.363033+0000","last_unstale":"2024-09-18T08:09:43.784871+0000","last_undegraded":"2024-09-18T08:09:43.784871+0000","last_fullsized":"2024-09-18T08:09:43.784871+0000","mapping_epoch":211,"log_start":"185'3100","ondisk_log_start":"185'3100","created":165,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:07:52.307205+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:07:52.307205+0000","last_clean_scrub_stamp":"2024-09-18T08:07:52.307205+0000","objects_scrubbed":0,"log_size":4387,"log_dups_size":0,"ondisk_log_size":4387,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":355270656,"num_objects":5421,"num_object_clones":0,"num_object_copies":16263,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5421,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7487,"num_write_kb":413056,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8010,"num_bytes_recovered":524943360,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,1],"acting":[3,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"74'79","reported_seq":226,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-09-18T08:09:34.405459+0000","last_change":"2024-09-18T08:08:57.736557+0000","last_active":"2024-09-18T08:09:34.405459+0000","last_peered":"2024-09-18T08:09:34.405459+0000","last_clean":"2024-09-18T08:09:34.405459+0000","last_became_active":"2024-09-18T08:07:26.426532+0000","last_became_peered":"2024-09-18T08:07:26.426532+0000","last_unstale":"2024-09-18T08:09:34.405459+0000","last_undegraded":"2024-09-18T08:09:34.405459+0000","last_fullsized":"2024-09-18T08:09:34.405459+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":152,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":3,"last_scrub":"74'79","last_scrub_stamp":"2024-09-18T08:08:57.736355+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:00:11.893742+0000","last_clean_scrub_stamp":"2024-09-18T08:08:57.736355+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:10:19.314793+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,3],"acting":[0,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.3","version":"69'19","reported_seq":356,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-09-18T08:09:34.405177+0000","last_change":"2024-09-18T08:04:07.231406+0000","last_active":"2024-09-18T08:09:34.405177+0000","last_peered":"2024-09-18T08:09:34.405177+0000","last_clean":"2024-09-18T08:09:34.405177+0000","last_became_active":"2024-09-18T08:02:49.890318+0000","last_became_peered":"2024-09-18T08:02:49.890318+0000","last_unstale":"2024-09-18T08:09:34.405177+0000","last_undegraded":"2024-09-18T08:09:34.405177+0000","last_fullsized":"2024-09-18T08:09:34.405177+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'19","last_scrub_stamp":"2024-09-18T08:04:07.229846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_clean_scrub_stamp":"2024-09-18T08:04:07.229846+0000","objects_scrubbed":8,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:10:19.093125+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":39,"num_read_kb":25,"num_write":22,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.4","version":"62'1","reported_seq":360,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-09-18T08:09:34.411126+0000","last_change":"2024-09-18T08:09:02.569780+0000","last_active":"2024-09-18T08:09:34.411126+0000","last_peered":"2024-09-18T08:09:34.411126+0000","last_clean":"2024-09-18T08:09:34.411126+0000","last_became_active":"2024-09-18T08:08:52.688347+0000","last_became_peered":"2024-09-18T08:08:52.688347+0000","last_unstale":"2024-09-18T08:09:34.411126+0000","last_undegraded":"2024-09-18T08:09:34.411126+0000","last_fullsized":"2024-09-18T08:09:34.411126+0000","mapping_epoch":198,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":199,"parent":"0.0","parent_split_bits":0,"last_scrub":"62'1","last_scrub_stamp":"2024-09-18T08:09:02.569706+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:46.860092+0000","last_clean_scrub_stamp":"2024-09-18T08:09:02.569706+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:10:05.632401+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":436,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":872,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":250,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-09-18T08:09:34.404384+0000","last_change":"2024-09-18T08:06:24.133954+0000","last_active":"2024-09-18T08:09:34.404384+0000","last_peered":"2024-09-18T08:09:34.404384+0000","last_clean":"2024-09-18T08:09:34.404384+0000","last_became_active":"2024-09-18T08:06:24.133262+0000","last_became_peered":"2024-09-18T08:06:24.133262+0000","last_unstale":"2024-09-18T08:09:34.404384+0000","last_undegraded":"2024-09-18T08:09:34.404384+0000","last_fullsized":"2024-09-18T08:09:34.404384+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:04:13.377561+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:50.886830+0000","last_clean_scrub_stamp":"2024-09-18T08:04:13.377561+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:10:22.094192+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":356,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-09-18T08:09:34.410611+0000","last_change":"2024-09-18T08:08:52.688537+0000","last_active":"2024-09-18T08:09:34.410611+0000","last_peered":"2024-09-18T08:09:34.410611+0000","last_clean":"2024-09-18T08:09:34.410611+0000","last_became_active":"2024-09-18T08:08:52.688212+0000","last_became_peered":"2024-09-18T08:08:52.688212+0000","last_unstale":"2024-09-18T08:09:34.410611+0000","last_undegraded":"2024-09-18T08:09:34.410611+0000","last_fullsized":"2024-09-18T08:09:34.410611+0000","mapping_epoch":198,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":199,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:07:49.041138+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_clean_scrub_stamp":"2024-09-18T08:07:49.041138+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:10:25.679955+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.8","version":"213'7213","reported_seq":7320,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-09-18T08:09:43.129305+0000","last_change":"2024-09-18T08:07:53.671548+0000","last_active":"2024-09-18T08:09:43.129305+0000","last_peered":"2024-09-18T08:09:43.129305+0000","last_clean":"2024-09-18T08:09:43.129305+0000","last_became_active":"2024-09-18T08:07:53.670635+0000","last_became_peered":"2024-09-18T08:07:53.670635+0000","last_unstale":"2024-09-18T08:09:43.129305+0000","last_undegraded":"2024-09-18T08:09:43.129305+0000","last_fullsized":"2024-09-18T08:09:43.129305+0000","mapping_epoch":165,"log_start":"180'2100","ondisk_log_start":"180'2100","created":165,"last_epoch_clean":166,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:07:52.307205+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:07:52.307205+0000","last_clean_scrub_stamp":"2024-09-18T08:07:52.307205+0000","objects_scrubbed":0,"log_size":5113,"log_dups_size":0,"ondisk_log_size":5113,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":343343104,"num_objects":5239,"num_object_clones":0,"num_object_copies":15717,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5239,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7213,"num_write_kb":398464,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.1","version":"74'79","reported_seq":145,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-09-18T08:09:34.396131+0000","last_change":"2024-09-18T08:08:59.553431+0000","last_active":"2024-09-18T08:09:34.396131+0000","last_peered":"2024-09-18T08:09:34.396131+0000","last_clean":"2024-09-18T08:09:34.396131+0000","last_became_active":"2024-09-18T08:07:26.400643+0000","last_became_peered":"2024-09-18T08:07:26.400643+0000","last_unstale":"2024-09-18T08:09:34.396131+0000","last_undegraded":"2024-09-18T08:09:34.396131+0000","last_fullsized":"2024-09-18T08:09:34.396131+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":152,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":4,"last_scrub":"74'79","last_scrub_stamp":"2024-09-18T08:08:59.552592+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:00:11.893742+0000","last_clean_scrub_stamp":"2024-09-18T08:08:59.552592+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:10:15.399172+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,2],"acting":[5,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.4","version":"69'28","reported_seq":460,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-09-18T08:09:38.009531+0000","last_change":"2024-09-18T08:08:57.639985+0000","last_active":"2024-09-18T08:09:38.009531+0000","last_peered":"2024-09-18T08:09:38.009531+0000","last_clean":"2024-09-18T08:09:38.009531+0000","last_became_active":"2024-09-18T08:02:49.897221+0000","last_became_peered":"2024-09-18T08:02:49.897221+0000","last_unstale":"2024-09-18T08:09:38.009531+0000","last_undegraded":"2024-09-18T08:09:38.009531+0000","last_fullsized":"2024-09-18T08:09:38.009531+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'28","last_scrub_stamp":"2024-09-18T08:08:57.639628+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_clean_scrub_stamp":"2024-09-18T08:08:57.639628+0000","objects_scrubbed":10,"log_size":28,"log_dups_size":0,"ondisk_log_size":28,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:10:27.518247+0000","scrub_duration":18,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":48,"num_read_kb":33,"num_write":26,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":74,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-09-18T08:09:34.394010+0000","last_change":"2024-09-18T08:08:18.543579+0000","last_active":"2024-09-18T08:09:34.394010+0000","last_peered":"2024-09-18T08:09:34.394010+0000","last_clean":"2024-09-18T08:09:34.394010+0000","last_became_active":"2024-09-18T08:08:18.538852+0000","last_became_peered":"2024-09-18T08:08:18.538852+0000","last_unstale":"2024-09-18T08:09:34.394010+0000","last_undegraded":"2024-09-18T08:09:34.394010+0000","last_fullsized":"2024-09-18T08:09:34.394010+0000","mapping_epoch":181,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":182,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:05:39.490141+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:46.860092+0000","last_clean_scrub_stamp":"2024-09-18T08:05:39.490141+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"64'2","reported_seq":364,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-09-18T08:09:34.396539+0000","last_change":"2024-09-18T08:06:24.165227+0000","last_active":"2024-09-18T08:09:34.396539+0000","last_peered":"2024-09-18T08:09:34.396539+0000","last_clean":"2024-09-18T08:09:34.396539+0000","last_became_active":"2024-09-18T08:06:24.136080+0000","last_became_peered":"2024-09-18T08:06:24.136080+0000","last_unstale":"2024-09-18T08:09:34.396539+0000","last_undegraded":"2024-09-18T08:09:34.396539+0000","last_fullsized":"2024-09-18T08:09:34.396539+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"64'2","last_scrub_stamp":"2024-09-18T08:05:44.802942+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:45.855979+0000","last_clean_scrub_stamp":"2024-09-18T08:05:44.802942+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.001348349,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":38,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":330,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-09-18T08:09:34.404418+0000","last_change":"2024-09-18T08:08:18.544944+0000","last_active":"2024-09-18T08:09:34.404418+0000","last_peered":"2024-09-18T08:09:34.404418+0000","last_clean":"2024-09-18T08:09:34.404418+0000","last_became_active":"2024-09-18T08:08:18.544260+0000","last_became_peered":"2024-09-18T08:08:18.544260+0000","last_unstale":"2024-09-18T08:09:34.404418+0000","last_undegraded":"2024-09-18T08:09:34.404418+0000","last_fullsized":"2024-09-18T08:09:34.404418+0000","mapping_epoch":181,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":182,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:07:53.582095+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:50.886830+0000","last_clean_scrub_stamp":"2024-09-18T08:07:53.582095+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":336,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-09-18T08:09:34.394100+0000","last_change":"2024-09-18T08:08:18.543527+0000","last_active":"2024-09-18T08:09:34.394100+0000","last_peered":"2024-09-18T08:09:34.394100+0000","last_clean":"2024-09-18T08:09:34.394100+0000","last_became_active":"2024-09-18T08:08:18.542678+0000","last_became_peered":"2024-09-18T08:08:18.542678+0000","last_unstale":"2024-09-18T08:09:34.394100+0000","last_undegraded":"2024-09-18T08:09:34.394100+0000","last_fullsized":"2024-09-18T08:09:34.394100+0000","mapping_epoch":181,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":182,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:07:08.245063+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_clean_scrub_stamp":"2024-09-18T08:07:08.245063+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.b","version":"213'7336","reported_seq":12955,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-09-18T08:09:43.756412+0000","last_change":"2024-09-18T08:08:48.662821+0000","last_active":"2024-09-18T08:09:43.756412+0000","last_peered":"2024-09-18T08:09:43.756412+0000","last_clean":"2024-09-18T08:09:43.756412+0000","last_became_active":"2024-09-18T08:08:48.655676+0000","last_became_peered":"2024-09-18T08:08:48.655676+0000","last_unstale":"2024-09-18T08:09:43.756412+0000","last_undegraded":"2024-09-18T08:09:43.756412+0000","last_fullsized":"2024-09-18T08:09:43.756412+0000","mapping_epoch":194,"log_start":"184'2900","ondisk_log_start":"184'2900","created":165,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:07:52.307205+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:07:52.307205+0000","last_clean_scrub_stamp":"2024-09-18T08:07:52.307205+0000","objects_scrubbed":0,"log_size":4436,"log_dups_size":0,"ondisk_log_size":4436,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":349700096,"num_objects":5336,"num_object_clones":0,"num_object_copies":16008,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5336,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7336,"num_write_kb":405504,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2745,"num_bytes_recovered":179896320,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,1],"acting":[3,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"74'79","reported_seq":247,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-09-18T08:09:34.404234+0000","last_change":"2024-09-18T08:08:18.524000+0000","last_active":"2024-09-18T08:09:34.404234+0000","last_peered":"2024-09-18T08:09:34.404234+0000","last_clean":"2024-09-18T08:09:34.404234+0000","last_became_active":"2024-09-18T08:08:18.522977+0000","last_became_peered":"2024-09-18T08:08:18.522977+0000","last_unstale":"2024-09-18T08:09:34.404234+0000","last_undegraded":"2024-09-18T08:09:34.404234+0000","last_fullsized":"2024-09-18T08:09:34.404234+0000","mapping_epoch":181,"log_start":"0'0","ondisk_log_start":"0'0","created":152,"last_epoch_clean":182,"parent":"0.0","parent_split_bits":4,"last_scrub":"74'79","last_scrub_stamp":"2024-09-18T08:08:09.574144+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:00:11.893742+0000","last_clean_scrub_stamp":"2024-09-18T08:08:09.574144+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,4],"acting":[0,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.7","version":"69'13","reported_seq":266,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-09-18T08:09:34.403264+0000","last_change":"2024-09-18T08:04:18.164662+0000","last_active":"2024-09-18T08:09:34.403264+0000","last_peered":"2024-09-18T08:09:34.403264+0000","last_clean":"2024-09-18T08:09:34.403264+0000","last_became_active":"2024-09-18T08:04:18.144925+0000","last_became_peered":"2024-09-18T08:04:18.144925+0000","last_unstale":"2024-09-18T08:09:34.403264+0000","last_undegraded":"2024-09-18T08:09:34.403264+0000","last_fullsized":"2024-09-18T08:09:34.403264+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'13","last_scrub_stamp":"2024-09-18T08:03:50.700818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_clean_scrub_stamp":"2024-09-18T08:03:50.700818+0000","objects_scrubbed":0,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":29,"num_read_kb":18,"num_write":16,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,0],"acting":[2,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":346,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-09-18T08:09:34.409656+0000","last_change":"2024-09-18T08:08:53.649127+0000","last_active":"2024-09-18T08:09:34.409656+0000","last_peered":"2024-09-18T08:09:34.409656+0000","last_clean":"2024-09-18T08:09:34.409656+0000","last_became_active":"2024-09-18T08:06:24.126334+0000","last_became_peered":"2024-09-18T08:06:24.126334+0000","last_unstale":"2024-09-18T08:09:34.409656+0000","last_undegraded":"2024-09-18T08:09:34.409656+0000","last_fullsized":"2024-09-18T08:09:34.409656+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:08:53.649079+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:46.860092+0000","last_clean_scrub_stamp":"2024-09-18T08:08:53.649079+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:10:14.347881+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"62'1","reported_seq":334,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-09-18T08:09:34.400381+0000","last_change":"2024-09-18T08:04:09.732724+0000","last_active":"2024-09-18T08:09:34.400381+0000","last_peered":"2024-09-18T08:09:34.400381+0000","last_clean":"2024-09-18T08:09:34.400381+0000","last_became_active":"2024-09-18T08:02:46.868175+0000","last_became_peered":"2024-09-18T08:02:46.868175+0000","last_unstale":"2024-09-18T08:09:34.400381+0000","last_undegraded":"2024-09-18T08:09:34.400381+0000","last_fullsized":"2024-09-18T08:09:34.400381+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"62'1","last_scrub_stamp":"2024-09-18T08:04:09.732645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:45.855979+0000","last_clean_scrub_stamp":"2024-09-18T08:04:09.732645+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0.00066785499999999997,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":40,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-09-18T08:09:34.400255+0000","last_change":"2024-09-18T08:08:52.690656+0000","last_active":"2024-09-18T08:09:34.400255+0000","last_peered":"2024-09-18T08:09:34.400255+0000","last_clean":"2024-09-18T08:09:34.400255+0000","last_became_active":"2024-09-18T08:08:52.690286+0000","last_became_peered":"2024-09-18T08:08:52.690286+0000","last_unstale":"2024-09-18T08:09:34.400255+0000","last_undegraded":"2024-09-18T08:09:34.400255+0000","last_fullsized":"2024-09-18T08:09:34.400255+0000","mapping_epoch":198,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":199,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:07:08.806748+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:50.886830+0000","last_clean_scrub_stamp":"2024-09-18T08:07:08.806748+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":343,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-09-18T08:09:42.647525+0000","last_change":"2024-09-18T08:09:42.647525+0000","last_active":"2024-09-18T08:09:42.647525+0000","last_peered":"2024-09-18T08:09:42.647525+0000","last_clean":"2024-09-18T08:09:42.647525+0000","last_became_active":"2024-09-18T08:06:24.145382+0000","last_became_peered":"2024-09-18T08:06:24.145382+0000","last_unstale":"2024-09-18T08:09:42.647525+0000","last_undegraded":"2024-09-18T08:09:42.647525+0000","last_fullsized":"2024-09-18T08:09:42.647525+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:09:42.647317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_clean_scrub_stamp":"2024-09-18T08:09:42.647317+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:11:03.015274+0000","scrub_duration":26,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.a","version":"213'7289","reported_seq":7396,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-09-18T08:09:43.646330+0000","last_change":"2024-09-18T08:07:53.326651+0000","last_active":"2024-09-18T08:09:43.646330+0000","last_peered":"2024-09-18T08:09:43.646330+0000","last_clean":"2024-09-18T08:09:43.646330+0000","last_became_active":"2024-09-18T08:07:53.326361+0000","last_became_peered":"2024-09-18T08:07:53.326361+0000","last_unstale":"2024-09-18T08:09:43.646330+0000","last_undegraded":"2024-09-18T08:09:43.646330+0000","last_fullsized":"2024-09-18T08:09:43.646330+0000","mapping_epoch":165,"log_start":"183'2500","ondisk_log_start":"183'2500","created":165,"last_epoch_clean":166,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:07:52.307205+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:07:52.307205+0000","last_clean_scrub_stamp":"2024-09-18T08:07:52.307205+0000","objects_scrubbed":0,"log_size":4789,"log_dups_size":0,"ondisk_log_size":4789,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:10:20.719923+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":347013120,"num_objects":5295,"num_object_clones":0,"num_object_copies":15885,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5295,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7289,"num_write_kb":402688,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.3","version":"74'79","reported_seq":167,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-09-18T08:09:41.651800+0000","last_change":"2024-09-18T08:09:41.651800+0000","last_active":"2024-09-18T08:09:41.651800+0000","last_peered":"2024-09-18T08:09:41.651800+0000","last_clean":"2024-09-18T08:09:41.651800+0000","last_became_active":"2024-09-18T08:08:18.538929+0000","last_became_peered":"2024-09-18T08:08:18.538929+0000","last_unstale":"2024-09-18T08:09:41.651800+0000","last_undegraded":"2024-09-18T08:09:41.651800+0000","last_fullsized":"2024-09-18T08:09:41.651800+0000","mapping_epoch":181,"log_start":"0'0","ondisk_log_start":"0'0","created":152,"last_epoch_clean":182,"parent":"0.0","parent_split_bits":3,"last_scrub":"74'79","last_scrub_stamp":"2024-09-18T08:09:41.651729+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:00:11.893742+0000","last_clean_scrub_stamp":"2024-09-18T08:09:41.651729+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:10:55.722378+0000","scrub_duration":17,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.6","version":"69'12","reported_seq":388,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-09-18T08:09:34.405957+0000","last_change":"2024-09-18T08:08:57.021693+0000","last_active":"2024-09-18T08:09:34.405957+0000","last_peered":"2024-09-18T08:09:34.405957+0000","last_clean":"2024-09-18T08:09:34.405957+0000","last_became_active":"2024-09-18T08:08:52.700130+0000","last_became_peered":"2024-09-18T08:08:52.700130+0000","last_unstale":"2024-09-18T08:09:34.405957+0000","last_undegraded":"2024-09-18T08:09:34.405957+0000","last_fullsized":"2024-09-18T08:09:34.405957+0000","mapping_epoch":198,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":200,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'12","last_scrub_stamp":"2024-09-18T08:07:39.603562+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_clean_scrub_stamp":"2024-09-18T08:07:39.603562+0000","objects_scrubbed":6,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":18,"num_read_kb":12,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":354,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-09-18T08:09:34.409420+0000","last_change":"2024-09-18T08:09:34.089548+0000","last_active":"2024-09-18T08:09:34.409420+0000","last_peered":"2024-09-18T08:09:34.409420+0000","last_clean":"2024-09-18T08:09:34.409420+0000","last_became_active":"2024-09-18T08:08:18.523135+0000","last_became_peered":"2024-09-18T08:08:18.523135+0000","last_unstale":"2024-09-18T08:09:34.409420+0000","last_undegraded":"2024-09-18T08:09:34.409420+0000","last_fullsized":"2024-09-18T08:09:34.409420+0000","mapping_epoch":181,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":182,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:09:34.089476+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:46.860092+0000","last_clean_scrub_stamp":"2024-09-18T08:09:34.089476+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:10:41.071862+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"70'5","reported_seq":387,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-09-18T08:09:41.725322+0000","last_change":"2024-09-18T08:09:09.649261+0000","last_active":"2024-09-18T08:09:41.725322+0000","last_peered":"2024-09-18T08:09:41.725322+0000","last_clean":"2024-09-18T08:09:41.725322+0000","last_became_active":"2024-09-18T08:06:46.141667+0000","last_became_peered":"2024-09-18T08:06:46.141667+0000","last_unstale":"2024-09-18T08:09:41.725322+0000","last_undegraded":"2024-09-18T08:09:41.725322+0000","last_fullsized":"2024-09-18T08:09:41.725322+0000","mapping_epoch":143,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":144,"parent":"0.0","parent_split_bits":0,"last_scrub":"70'5","last_scrub_stamp":"2024-09-18T08:09:09.649188+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:45.855979+0000","last_clean_scrub_stamp":"2024-09-18T08:09:09.649188+0000","objects_scrubbed":1,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:10:24.340859+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0.000504055,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":183,"num_read_kb":178,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":332,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-09-18T08:09:34.397093+0000","last_change":"2024-09-18T08:07:20.767916+0000","last_active":"2024-09-18T08:09:34.397093+0000","last_peered":"2024-09-18T08:09:34.397093+0000","last_clean":"2024-09-18T08:09:34.397093+0000","last_became_active":"2024-09-18T08:02:51.913112+0000","last_became_peered":"2024-09-18T08:02:51.913112+0000","last_unstale":"2024-09-18T08:09:34.397093+0000","last_undegraded":"2024-09-18T08:09:34.397093+0000","last_fullsized":"2024-09-18T08:09:34.397093+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:07:20.767841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:50.886830+0000","last_clean_scrub_stamp":"2024-09-18T08:07:20.767841+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":305,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-09-18T08:09:34.408656+0000","last_change":"2024-09-18T08:02:53.907689+0000","last_active":"2024-09-18T08:09:34.408656+0000","last_peered":"2024-09-18T08:09:34.408656+0000","last_clean":"2024-09-18T08:09:34.408656+0000","last_became_active":"2024-09-18T08:02:53.907404+0000","last_became_peered":"2024-09-18T08:02:53.907404+0000","last_unstale":"2024-09-18T08:09:34.408656+0000","last_undegraded":"2024-09-18T08:09:34.408656+0000","last_fullsized":"2024-09-18T08:09:34.408656+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_clean_scrub_stamp":"2024-09-18T08:02:52.890475+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:09:49.627839+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.9","version":"213'7022","reported_seq":7128,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-09-18T08:09:43.246946+0000","last_change":"2024-09-18T08:07:53.672294+0000","last_active":"2024-09-18T08:09:43.246946+0000","last_peered":"2024-09-18T08:09:43.246946+0000","last_clean":"2024-09-18T08:09:43.246946+0000","last_became_active":"2024-09-18T08:07:53.671994+0000","last_became_peered":"2024-09-18T08:07:53.671994+0000","last_unstale":"2024-09-18T08:09:43.246946+0000","last_undegraded":"2024-09-18T08:09:43.246946+0000","last_fullsized":"2024-09-18T08:09:43.246946+0000","mapping_epoch":165,"log_start":"180'1900","ondisk_log_start":"180'1900","created":165,"last_epoch_clean":166,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:07:52.307205+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:07:52.307205+0000","last_clean_scrub_stamp":"2024-09-18T08:07:52.307205+0000","objects_scrubbed":0,"log_size":5122,"log_dups_size":0,"ondisk_log_size":5122,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":340262912,"num_objects":5192,"num_object_clones":0,"num_object_copies":15576,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5192,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7022,"num_write_kb":390848,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.5","version":"182'21","reported_seq":110,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-09-18T08:09:38.009621+0000","last_change":"2024-09-18T08:08:18.609652+0000","last_active":"2024-09-18T08:09:38.009621+0000","last_peered":"2024-09-18T08:09:38.009621+0000","last_clean":"2024-09-18T08:09:38.009621+0000","last_became_active":"2024-09-18T08:08:18.547555+0000","last_became_peered":"2024-09-18T08:08:18.547555+0000","last_unstale":"2024-09-18T08:09:38.009621+0000","last_undegraded":"2024-09-18T08:09:38.009621+0000","last_fullsized":"2024-09-18T08:09:38.009621+0000","mapping_epoch":181,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":182,"parent":"0.0","parent_split_bits":0,"last_scrub":"130'18","last_scrub_stamp":"2024-09-18T08:06:15.455503+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_clean_scrub_stamp":"2024-09-18T08:06:15.455503+0000","objects_scrubbed":0,"log_size":21,"log_dups_size":0,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":24,"num_read_kb":14,"num_write":13,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":154,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,7],"acting":[4,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":339,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-09-18T08:09:34.394163+0000","last_change":"2024-09-18T08:06:24.134043+0000","last_active":"2024-09-18T08:09:34.394163+0000","last_peered":"2024-09-18T08:09:34.394163+0000","last_clean":"2024-09-18T08:09:34.394163+0000","last_became_active":"2024-09-18T08:06:24.133425+0000","last_became_peered":"2024-09-18T08:06:24.133425+0000","last_unstale":"2024-09-18T08:09:34.394163+0000","last_undegraded":"2024-09-18T08:09:34.394163+0000","last_fullsized":"2024-09-18T08:09:34.394163+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:05:38.080993+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:46.860092+0000","last_clean_scrub_stamp":"2024-09-18T08:05:38.080993+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"74'79","reported_seq":428,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-09-18T08:09:34.408404+0000","last_change":"2024-09-18T08:08:57.769104+0000","last_active":"2024-09-18T08:09:34.408404+0000","last_peered":"2024-09-18T08:09:34.408404+0000","last_clean":"2024-09-18T08:09:34.408404+0000","last_became_active":"2024-09-18T08:07:24.376931+0000","last_became_peered":"2024-09-18T08:07:24.376931+0000","last_unstale":"2024-09-18T08:09:34.408404+0000","last_undegraded":"2024-09-18T08:09:34.408404+0000","last_fullsized":"2024-09-18T08:09:34.408404+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":153,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'79","last_scrub_stamp":"2024-09-18T08:08:57.769048+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:00:11.893742+0000","last_clean_scrub_stamp":"2024-09-18T08:08:57.769048+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:10:19.097626+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":160,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-09-18T08:09:34.401128+0000","last_change":"2024-09-18T08:07:09.744206+0000","last_active":"2024-09-18T08:09:34.401128+0000","last_peered":"2024-09-18T08:09:34.401128+0000","last_clean":"2024-09-18T08:09:34.401128+0000","last_became_active":"2024-09-18T08:06:46.141046+0000","last_became_peered":"2024-09-18T08:06:46.141046+0000","last_unstale":"2024-09-18T08:09:34.401128+0000","last_undegraded":"2024-09-18T08:09:34.401128+0000","last_fullsized":"2024-09-18T08:09:34.401128+0000","mapping_epoch":143,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":144,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:07:09.744137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:50.886830+0000","last_clean_scrub_stamp":"2024-09-18T08:07:09.744137+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":335,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-09-18T08:09:34.396890+0000","last_change":"2024-09-18T08:08:18.548620+0000","last_active":"2024-09-18T08:09:34.396890+0000","last_peered":"2024-09-18T08:09:34.396890+0000","last_clean":"2024-09-18T08:09:34.396890+0000","last_became_active":"2024-09-18T08:08:18.548233+0000","last_became_peered":"2024-09-18T08:08:18.548233+0000","last_unstale":"2024-09-18T08:09:34.396890+0000","last_undegraded":"2024-09-18T08:09:34.396890+0000","last_fullsized":"2024-09-18T08:09:34.396890+0000","mapping_epoch":181,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":182,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:06:50.710565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_clean_scrub_stamp":"2024-09-18T08:06:50.710565+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.2","version":"213'7228","reported_seq":11612,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-09-18T08:09:43.128908+0000","last_change":"2024-09-18T08:08:58.743574+0000","last_active":"2024-09-18T08:09:43.128908+0000","last_peered":"2024-09-18T08:09:43.128908+0000","last_clean":"2024-09-18T08:09:43.128908+0000","last_became_active":"2024-09-18T08:08:58.738458+0000","last_became_peered":"2024-09-18T08:08:58.738458+0000","last_unstale":"2024-09-18T08:09:43.128908+0000","last_undegraded":"2024-09-18T08:09:43.128908+0000","last_fullsized":"2024-09-18T08:09:43.128908+0000","mapping_epoch":201,"log_start":"184'2700","ondisk_log_start":"184'2700","created":165,"last_epoch_clean":202,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:07:52.307205+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:07:52.307205+0000","last_clean_scrub_stamp":"2024-09-18T08:07:52.307205+0000","objects_scrubbed":0,"log_size":4528,"log_dups_size":0,"ondisk_log_size":4528,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":350224384,"num_objects":5344,"num_object_clones":0,"num_object_copies":16032,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5344,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7228,"num_write_kb":402304,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2129,"num_bytes_recovered":139526144,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,5],"acting":[4,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.e","version":"69'11","reported_seq":86,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-09-18T08:09:34.395037+0000","last_change":"2024-09-18T08:08:18.594670+0000","last_active":"2024-09-18T08:09:34.395037+0000","last_peered":"2024-09-18T08:09:34.395037+0000","last_clean":"2024-09-18T08:09:34.395037+0000","last_became_active":"2024-09-18T08:08:18.544581+0000","last_became_peered":"2024-09-18T08:08:18.544581+0000","last_unstale":"2024-09-18T08:09:34.395037+0000","last_undegraded":"2024-09-18T08:09:34.395037+0000","last_fullsized":"2024-09-18T08:09:34.395037+0000","mapping_epoch":181,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":182,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'11","last_scrub_stamp":"2024-09-18T08:06:14.472168+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_clean_scrub_stamp":"2024-09-18T08:06:14.472168+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":19,"num_read_kb":12,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":74,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-09-18T08:09:34.394659+0000","last_change":"2024-09-18T08:08:18.524516+0000","last_active":"2024-09-18T08:09:34.394659+0000","last_peered":"2024-09-18T08:09:34.394659+0000","last_clean":"2024-09-18T08:09:34.394659+0000","last_became_active":"2024-09-18T08:08:18.524229+0000","last_became_peered":"2024-09-18T08:08:18.524229+0000","last_unstale":"2024-09-18T08:09:34.394659+0000","last_undegraded":"2024-09-18T08:09:34.394659+0000","last_fullsized":"2024-09-18T08:09:34.394659+0000","mapping_epoch":181,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":182,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:02:46.860092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:46.860092+0000","last_clean_scrub_stamp":"2024-09-18T08:02:46.860092+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":352,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-09-18T08:09:34.396780+0000","last_change":"2024-09-18T08:08:18.547880+0000","last_active":"2024-09-18T08:09:34.396780+0000","last_peered":"2024-09-18T08:09:34.396780+0000","last_clean":"2024-09-18T08:09:34.396780+0000","last_became_active":"2024-09-18T08:08:18.547105+0000","last_became_peered":"2024-09-18T08:08:18.547105+0000","last_unstale":"2024-09-18T08:09:34.396780+0000","last_undegraded":"2024-09-18T08:09:34.396780+0000","last_fullsized":"2024-09-18T08:09:34.396780+0000","mapping_epoch":181,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":182,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:07:51.742090+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:50.886830+0000","last_clean_scrub_stamp":"2024-09-18T08:07:51.742090+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":335,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-09-18T08:09:34.393846+0000","last_change":"2024-09-18T08:07:09.197912+0000","last_active":"2024-09-18T08:09:34.393846+0000","last_peered":"2024-09-18T08:09:34.393846+0000","last_clean":"2024-09-18T08:09:34.393846+0000","last_became_active":"2024-09-18T08:06:24.130266+0000","last_became_peered":"2024-09-18T08:06:24.130266+0000","last_unstale":"2024-09-18T08:09:34.393846+0000","last_undegraded":"2024-09-18T08:09:34.393846+0000","last_fullsized":"2024-09-18T08:09:34.393846+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:07:09.197841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_clean_scrub_stamp":"2024-09-18T08:07:09.197841+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.1","version":"213'7317","reported_seq":7491,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-09-18T08:09:43.652493+0000","last_change":"2024-09-18T08:09:40.130643+0000","last_active":"2024-09-18T08:09:43.652493+0000","last_peered":"2024-09-18T08:09:43.652493+0000","last_clean":"2024-09-18T08:09:43.652493+0000","last_became_active":"2024-09-18T08:07:53.327987+0000","last_became_peered":"2024-09-18T08:07:53.327987+0000","last_unstale":"2024-09-18T08:09:43.652493+0000","last_undegraded":"2024-09-18T08:09:43.652493+0000","last_fullsized":"2024-09-18T08:09:43.652493+0000","mapping_epoch":165,"log_start":"183'2500","ondisk_log_start":"183'2500","created":165,"last_epoch_clean":166,"parent":"0.0","parent_split_bits":0,"last_scrub":"213'7038","last_scrub_stamp":"2024-09-18T08:09:40.130353+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:07:52.307205+0000","last_clean_scrub_stamp":"2024-09-18T08:09:40.130353+0000","objects_scrubbed":5709,"log_size":4817,"log_dups_size":0,"ondisk_log_size":4817,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:11:06.670120+0000","scrub_duration":2501,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":351338496,"num_objects":5361,"num_object_clones":0,"num_object_copies":16083,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5361,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7317,"num_write_kb":405696,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.8","version":"74'79","reported_seq":152,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-09-18T08:09:34.395848+0000","last_change":"2024-09-18T08:09:00.533593+0000","last_active":"2024-09-18T08:09:34.395848+0000","last_peered":"2024-09-18T08:09:34.395848+0000","last_clean":"2024-09-18T08:09:34.395848+0000","last_became_active":"2024-09-18T08:07:26.407298+0000","last_became_peered":"2024-09-18T08:07:26.407298+0000","last_unstale":"2024-09-18T08:09:34.395848+0000","last_undegraded":"2024-09-18T08:09:34.395848+0000","last_fullsized":"2024-09-18T08:09:34.395848+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":152,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":4,"last_scrub":"74'79","last_scrub_stamp":"2024-09-18T08:09:00.532813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:00:11.893742+0000","last_clean_scrub_stamp":"2024-09-18T08:09:00.532813+0000","objects_scrubbed":1,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:10:17.887770+0000","scrub_duration":25,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262688,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,3],"acting":[5,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.d","version":"69'17","reported_seq":92,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-09-18T08:09:34.394322+0000","last_change":"2024-09-18T08:08:18.588624+0000","last_active":"2024-09-18T08:09:34.394322+0000","last_peered":"2024-09-18T08:09:34.394322+0000","last_clean":"2024-09-18T08:09:34.394322+0000","last_became_active":"2024-09-18T08:08:18.540862+0000","last_became_peered":"2024-09-18T08:08:18.540862+0000","last_unstale":"2024-09-18T08:09:34.394322+0000","last_undegraded":"2024-09-18T08:09:34.394322+0000","last_fullsized":"2024-09-18T08:09:34.394322+0000","mapping_epoch":181,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":182,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'17","last_scrub_stamp":"2024-09-18T08:08:09.671248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_clean_scrub_stamp":"2024-09-18T08:08:09.671248+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":28,"num_read_kb":18,"num_write":18,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,1],"acting":[4,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":74,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-09-18T08:09:34.394252+0000","last_change":"2024-09-18T08:08:18.541029+0000","last_active":"2024-09-18T08:09:34.394252+0000","last_peered":"2024-09-18T08:09:34.394252+0000","last_clean":"2024-09-18T08:09:34.394252+0000","last_became_active":"2024-09-18T08:08:18.540285+0000","last_became_peered":"2024-09-18T08:08:18.540285+0000","last_unstale":"2024-09-18T08:09:34.394252+0000","last_undegraded":"2024-09-18T08:09:34.394252+0000","last_fullsized":"2024-09-18T08:09:34.394252+0000","mapping_epoch":181,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":182,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:05:12.622901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:46.860092+0000","last_clean_scrub_stamp":"2024-09-18T08:05:12.622901+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":346,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-09-18T08:09:34.411078+0000","last_change":"2024-09-18T08:09:06.610955+0000","last_active":"2024-09-18T08:09:34.411078+0000","last_peered":"2024-09-18T08:09:34.411078+0000","last_clean":"2024-09-18T08:09:34.411078+0000","last_became_active":"2024-09-18T08:08:18.623014+0000","last_became_peered":"2024-09-18T08:08:18.623014+0000","last_unstale":"2024-09-18T08:09:34.411078+0000","last_undegraded":"2024-09-18T08:09:34.411078+0000","last_fullsized":"2024-09-18T08:09:34.411078+0000","mapping_epoch":181,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":182,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:09:06.610879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:50.886830+0000","last_clean_scrub_stamp":"2024-09-18T08:09:06.610879+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:10:19.831235+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":40,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-09-18T08:09:34.403168+0000","last_change":"2024-09-18T08:08:52.696713+0000","last_active":"2024-09-18T08:09:34.403168+0000","last_peered":"2024-09-18T08:09:34.403168+0000","last_clean":"2024-09-18T08:09:34.403168+0000","last_became_active":"2024-09-18T08:08:52.696300+0000","last_became_peered":"2024-09-18T08:08:52.696300+0000","last_unstale":"2024-09-18T08:09:34.403168+0000","last_undegraded":"2024-09-18T08:09:34.403168+0000","last_fullsized":"2024-09-18T08:09:34.403168+0000","mapping_epoch":198,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":199,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:04:06.709631+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_clean_scrub_stamp":"2024-09-18T08:04:06.709631+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"8.0","version":"213'7245","reported_seq":7352,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-09-18T08:09:43.764165+0000","last_change":"2024-09-18T08:07:53.329182+0000","last_active":"2024-09-18T08:09:43.764165+0000","last_peered":"2024-09-18T08:09:43.764165+0000","last_clean":"2024-09-18T08:09:43.764165+0000","last_became_active":"2024-09-18T08:07:53.328914+0000","last_became_peered":"2024-09-18T08:07:53.328914+0000","last_unstale":"2024-09-18T08:09:43.764165+0000","last_undegraded":"2024-09-18T08:09:43.764165+0000","last_fullsized":"2024-09-18T08:09:43.764165+0000","mapping_epoch":165,"log_start":"184'2800","ondisk_log_start":"184'2800","created":165,"last_epoch_clean":166,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:07:52.307205+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:07:52.307205+0000","last_clean_scrub_stamp":"2024-09-18T08:07:52.307205+0000","objects_scrubbed":0,"log_size":4445,"log_dups_size":0,"ondisk_log_size":4445,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":347668480,"num_objects":5305,"num_object_clones":0,"num_object_copies":15915,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5305,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7245,"num_write_kb":401600,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.9","version":"74'79","reported_seq":450,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-09-18T08:09:34.408503+0000","last_change":"2024-09-18T08:09:27.676247+0000","last_active":"2024-09-18T08:09:34.408503+0000","last_peered":"2024-09-18T08:09:34.408503+0000","last_clean":"2024-09-18T08:09:34.408503+0000","last_became_active":"2024-09-18T08:08:52.691492+0000","last_became_peered":"2024-09-18T08:08:52.691492+0000","last_unstale":"2024-09-18T08:09:34.408503+0000","last_undegraded":"2024-09-18T08:09:34.408503+0000","last_fullsized":"2024-09-18T08:09:34.408503+0000","mapping_epoch":198,"log_start":"0'0","ondisk_log_start":"0'0","created":152,"last_epoch_clean":199,"parent":"0.0","parent_split_bits":4,"last_scrub":"74'79","last_scrub_stamp":"2024-09-18T08:09:27.676169+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:00:11.893742+0000","last_clean_scrub_stamp":"2024-09-18T08:09:27.676169+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:10:36.422152+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,2],"acting":[7,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.c","version":"69'10","reported_seq":86,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-09-18T08:09:34.394520+0000","last_change":"2024-09-18T08:08:18.642452+0000","last_active":"2024-09-18T08:09:34.394520+0000","last_peered":"2024-09-18T08:09:34.394520+0000","last_clean":"2024-09-18T08:09:34.394520+0000","last_became_active":"2024-09-18T08:08:18.622980+0000","last_became_peered":"2024-09-18T08:08:18.622980+0000","last_unstale":"2024-09-18T08:09:34.394520+0000","last_undegraded":"2024-09-18T08:09:34.394520+0000","last_fullsized":"2024-09-18T08:09:34.394520+0000","mapping_epoch":181,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":182,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'10","last_scrub_stamp":"2024-09-18T08:05:41.474813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_clean_scrub_stamp":"2024-09-18T08:05:41.474813+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":14,"num_read_kb":9,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":347,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-09-18T08:09:34.398464+0000","last_change":"2024-09-18T08:08:18.542800+0000","last_active":"2024-09-18T08:09:34.398464+0000","last_peered":"2024-09-18T08:09:34.398464+0000","last_clean":"2024-09-18T08:09:34.398464+0000","last_became_active":"2024-09-18T08:08:18.542424+0000","last_became_peered":"2024-09-18T08:08:18.542424+0000","last_unstale":"2024-09-18T08:09:34.398464+0000","last_undegraded":"2024-09-18T08:09:34.398464+0000","last_fullsized":"2024-09-18T08:09:34.398464+0000","mapping_epoch":181,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":182,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:06:46.339428+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:46.860092+0000","last_clean_scrub_stamp":"2024-09-18T08:06:46.339428+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.d","version":"69'8","reported_seq":417,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-09-18T08:09:38.485793+0000","last_change":"2024-09-18T08:05:33.027988+0000","last_active":"2024-09-18T08:09:38.485793+0000","last_peered":"2024-09-18T08:09:38.485793+0000","last_clean":"2024-09-18T08:09:38.485793+0000","last_became_active":"2024-09-18T08:02:51.909011+0000","last_became_peered":"2024-09-18T08:02:51.909011+0000","last_unstale":"2024-09-18T08:09:38.485793+0000","last_undegraded":"2024-09-18T08:09:38.485793+0000","last_fullsized":"2024-09-18T08:09:38.485793+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'8","last_scrub_stamp":"2024-09-18T08:05:33.027783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:50.886830+0000","last_clean_scrub_stamp":"2024-09-18T08:05:33.027783+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":81,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-09-18T08:09:34.395228+0000","last_change":"2024-09-18T08:08:52.685954+0000","last_active":"2024-09-18T08:09:34.395228+0000","last_peered":"2024-09-18T08:09:34.395228+0000","last_clean":"2024-09-18T08:09:34.395228+0000","last_became_active":"2024-09-18T08:08:52.685603+0000","last_became_peered":"2024-09-18T08:08:52.685603+0000","last_unstale":"2024-09-18T08:09:34.395228+0000","last_undegraded":"2024-09-18T08:09:34.395228+0000","last_fullsized":"2024-09-18T08:09:34.395228+0000","mapping_epoch":198,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":199,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:07:16.559597+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_clean_scrub_stamp":"2024-09-18T08:07:16.559597+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.7","version":"213'7318","reported_seq":7495,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-09-18T08:09:43.617167+0000","last_change":"2024-09-18T08:09:36.764450+0000","last_active":"2024-09-18T08:09:43.617167+0000","last_peered":"2024-09-18T08:09:43.617167+0000","last_clean":"2024-09-18T08:09:43.617167+0000","last_became_active":"2024-09-18T08:07:53.674433+0000","last_became_peered":"2024-09-18T08:07:53.674433+0000","last_unstale":"2024-09-18T08:09:43.617167+0000","last_undegraded":"2024-09-18T08:09:43.617167+0000","last_fullsized":"2024-09-18T08:09:43.617167+0000","mapping_epoch":165,"log_start":"183'2500","ondisk_log_start":"183'2500","created":165,"last_epoch_clean":166,"parent":"0.0","parent_split_bits":0,"last_scrub":"213'6848","last_scrub_stamp":"2024-09-18T08:09:36.764206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:07:52.307205+0000","last_clean_scrub_stamp":"2024-09-18T08:09:36.764206+0000","objects_scrubbed":5822,"log_size":4818,"log_dups_size":0,"ondisk_log_size":4818,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":4,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:11:06.120132+0000","scrub_duration":3143,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":344457216,"num_objects":5256,"num_object_clones":0,"num_object_copies":15768,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5256,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7318,"num_write_kb":402368,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,6],"acting":[1,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.b","version":"69'9","reported_seq":378,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-09-18T08:09:34.405085+0000","last_change":"2024-09-18T08:08:18.640380+0000","last_active":"2024-09-18T08:09:34.405085+0000","last_peered":"2024-09-18T08:09:34.405085+0000","last_clean":"2024-09-18T08:09:34.405085+0000","last_became_active":"2024-09-18T08:08:18.522995+0000","last_became_peered":"2024-09-18T08:08:18.522995+0000","last_unstale":"2024-09-18T08:09:34.405085+0000","last_undegraded":"2024-09-18T08:09:34.405085+0000","last_fullsized":"2024-09-18T08:09:34.405085+0000","mapping_epoch":181,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":182,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'9","last_scrub_stamp":"2024-09-18T08:08:18.640292+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_clean_scrub_stamp":"2024-09-18T08:08:18.640292+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:09:38.962419+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":346,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-09-18T08:09:34.397185+0000","last_change":"2024-09-18T08:07:40.708363+0000","last_active":"2024-09-18T08:09:34.397185+0000","last_peered":"2024-09-18T08:09:34.397185+0000","last_clean":"2024-09-18T08:09:34.397185+0000","last_became_active":"2024-09-18T08:06:24.135005+0000","last_became_peered":"2024-09-18T08:06:24.135005+0000","last_unstale":"2024-09-18T08:09:34.397185+0000","last_undegraded":"2024-09-18T08:09:34.397185+0000","last_fullsized":"2024-09-18T08:09:34.397185+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:07:40.708307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:46.860092+0000","last_clean_scrub_stamp":"2024-09-18T08:07:40.708307+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":347,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-09-18T08:09:34.402070+0000","last_change":"2024-09-18T08:08:18.509651+0000","last_active":"2024-09-18T08:09:34.402070+0000","last_peered":"2024-09-18T08:09:34.402070+0000","last_clean":"2024-09-18T08:09:34.402070+0000","last_became_active":"2024-09-18T08:08:18.509247+0000","last_became_peered":"2024-09-18T08:08:18.509247+0000","last_unstale":"2024-09-18T08:09:34.402070+0000","last_undegraded":"2024-09-18T08:09:34.402070+0000","last_fullsized":"2024-09-18T08:09:34.402070+0000","mapping_epoch":181,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":182,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:07:10.728679+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:50.886830+0000","last_clean_scrub_stamp":"2024-09-18T08:07:10.728679+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,3],"acting":[2,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":343,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-09-18T08:09:34.404817+0000","last_change":"2024-09-18T08:09:28.086782+0000","last_active":"2024-09-18T08:09:34.404817+0000","last_peered":"2024-09-18T08:09:34.404817+0000","last_clean":"2024-09-18T08:09:34.404817+0000","last_became_active":"2024-09-18T08:08:52.697337+0000","last_became_peered":"2024-09-18T08:08:52.697337+0000","last_unstale":"2024-09-18T08:09:34.404817+0000","last_undegraded":"2024-09-18T08:09:34.404817+0000","last_fullsized":"2024-09-18T08:09:34.404817+0000","mapping_epoch":198,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":199,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:09:28.086727+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_clean_scrub_stamp":"2024-09-18T08:09:28.086727+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:10:33.905647+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.6","version":"213'7250","reported_seq":7357,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-09-18T08:09:43.797230+0000","last_change":"2024-09-18T08:07:53.674440+0000","last_active":"2024-09-18T08:09:43.797230+0000","last_peered":"2024-09-18T08:09:43.797230+0000","last_clean":"2024-09-18T08:09:43.797230+0000","last_became_active":"2024-09-18T08:07:53.674139+0000","last_became_peered":"2024-09-18T08:07:53.674139+0000","last_unstale":"2024-09-18T08:09:43.797230+0000","last_undegraded":"2024-09-18T08:09:43.797230+0000","last_fullsized":"2024-09-18T08:09:43.797230+0000","mapping_epoch":165,"log_start":"178'1800","ondisk_log_start":"178'1800","created":165,"last_epoch_clean":166,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:07:52.307205+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:07:52.307205+0000","last_clean_scrub_stamp":"2024-09-18T08:07:52.307205+0000","objects_scrubbed":0,"log_size":5450,"log_dups_size":0,"ondisk_log_size":5450,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":346161152,"num_objects":5282,"num_object_clones":0,"num_object_copies":15846,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5282,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7250,"num_write_kb":401024,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,7],"acting":[0,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.a","version":"69'19","reported_seq":349,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-09-18T08:09:34.410056+0000","last_change":"2024-09-18T08:07:41.000210+0000","last_active":"2024-09-18T08:09:34.410056+0000","last_peered":"2024-09-18T08:09:34.410056+0000","last_clean":"2024-09-18T08:09:34.410056+0000","last_became_active":"2024-09-18T08:06:24.139909+0000","last_became_peered":"2024-09-18T08:06:24.139909+0000","last_unstale":"2024-09-18T08:09:34.410056+0000","last_undegraded":"2024-09-18T08:09:34.410056+0000","last_fullsized":"2024-09-18T08:09:34.410056+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'19","last_scrub_stamp":"2024-09-18T08:07:41.000146+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_clean_scrub_stamp":"2024-09-18T08:07:41.000146+0000","objects_scrubbed":9,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:10:18.661451+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":31,"num_read_kb":20,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,7],"acting":[1,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":334,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-09-18T08:09:34.396686+0000","last_change":"2024-09-18T08:06:47.638974+0000","last_active":"2024-09-18T08:09:34.396686+0000","last_peered":"2024-09-18T08:09:34.396686+0000","last_clean":"2024-09-18T08:09:34.396686+0000","last_became_active":"2024-09-18T08:06:46.171298+0000","last_became_peered":"2024-09-18T08:06:46.171298+0000","last_unstale":"2024-09-18T08:09:34.396686+0000","last_undegraded":"2024-09-18T08:09:34.396686+0000","last_fullsized":"2024-09-18T08:09:34.396686+0000","mapping_epoch":143,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":144,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:06:47.638911+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:46.860092+0000","last_clean_scrub_stamp":"2024-09-18T08:06:47.638911+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":40,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-09-18T08:09:34.399802+0000","last_change":"2024-09-18T08:08:52.698243+0000","last_active":"2024-09-18T08:09:34.399802+0000","last_peered":"2024-09-18T08:09:34.399802+0000","last_clean":"2024-09-18T08:09:34.399802+0000","last_became_active":"2024-09-18T08:08:52.697932+0000","last_became_peered":"2024-09-18T08:08:52.697932+0000","last_unstale":"2024-09-18T08:09:34.399802+0000","last_undegraded":"2024-09-18T08:09:34.399802+0000","last_fullsized":"2024-09-18T08:09:34.399802+0000","mapping_epoch":198,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":199,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:08:16.631007+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:50.886830+0000","last_clean_scrub_stamp":"2024-09-18T08:08:16.631007+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":160,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-09-18T08:09:34.397245+0000","last_change":"2024-09-18T08:06:50.902917+0000","last_active":"2024-09-18T08:09:34.397245+0000","last_peered":"2024-09-18T08:09:34.397245+0000","last_clean":"2024-09-18T08:09:34.397245+0000","last_became_active":"2024-09-18T08:06:46.137872+0000","last_became_peered":"2024-09-18T08:06:46.137872+0000","last_unstale":"2024-09-18T08:09:34.397245+0000","last_undegraded":"2024-09-18T08:09:34.397245+0000","last_fullsized":"2024-09-18T08:09:34.397245+0000","mapping_epoch":143,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":144,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:06:50.902835+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_clean_scrub_stamp":"2024-09-18T08:06:50.902835+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,3],"acting":[2,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"8.5","version":"213'7307","reported_seq":11727,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-09-18T08:09:43.101497+0000","last_change":"2024-09-18T08:09:13.170209+0000","last_active":"2024-09-18T08:09:43.101497+0000","last_peered":"2024-09-18T08:09:43.101497+0000","last_clean":"2024-09-18T08:09:43.101497+0000","last_became_active":"2024-09-18T08:09:13.166459+0000","last_became_peered":"2024-09-18T08:09:13.166459+0000","last_unstale":"2024-09-18T08:09:43.101497+0000","last_undegraded":"2024-09-18T08:09:43.101497+0000","last_fullsized":"2024-09-18T08:09:43.101497+0000","mapping_epoch":205,"log_start":"184'2774","ondisk_log_start":"184'2774","created":165,"last_epoch_clean":206,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:07:52.307205+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:07:52.307205+0000","last_clean_scrub_stamp":"2024-09-18T08:07:52.307205+0000","objects_scrubbed":0,"log_size":4533,"log_dups_size":0,"ondisk_log_size":4533,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":346226688,"num_objects":5283,"num_object_clones":0,"num_object_copies":15849,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5283,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7307,"num_write_kb":402880,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2146,"num_bytes_recovered":140640256,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,7],"acting":[4,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.9","version":"69'12","reported_seq":86,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-09-18T08:09:34.393344+0000","last_change":"2024-09-18T08:08:18.599669+0000","last_active":"2024-09-18T08:09:34.393344+0000","last_peered":"2024-09-18T08:09:34.393344+0000","last_clean":"2024-09-18T08:09:34.393344+0000","last_became_active":"2024-09-18T08:08:18.544489+0000","last_became_peered":"2024-09-18T08:08:18.544489+0000","last_unstale":"2024-09-18T08:09:34.393344+0000","last_undegraded":"2024-09-18T08:09:34.393344+0000","last_fullsized":"2024-09-18T08:09:34.393344+0000","mapping_epoch":181,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":182,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'12","last_scrub_stamp":"2024-09-18T08:05:33.544135+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_clean_scrub_stamp":"2024-09-18T08:05:33.544135+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":24,"num_read_kb":15,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":362,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-09-18T08:09:39.614251+0000","last_change":"2024-09-18T08:09:39.614251+0000","last_active":"2024-09-18T08:09:39.614251+0000","last_peered":"2024-09-18T08:09:39.614251+0000","last_clean":"2024-09-18T08:09:39.614251+0000","last_became_active":"2024-09-18T08:08:18.539573+0000","last_became_peered":"2024-09-18T08:08:18.539573+0000","last_unstale":"2024-09-18T08:09:39.614251+0000","last_undegraded":"2024-09-18T08:09:39.614251+0000","last_fullsized":"2024-09-18T08:09:39.614251+0000","mapping_epoch":181,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":182,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:09:39.614187+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:46.860092+0000","last_clean_scrub_stamp":"2024-09-18T08:09:39.614187+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:10:57.546531+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":325,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-09-18T08:09:34.397353+0000","last_change":"2024-09-18T08:08:15.744330+0000","last_active":"2024-09-18T08:09:34.397353+0000","last_peered":"2024-09-18T08:09:34.397353+0000","last_clean":"2024-09-18T08:09:34.397353+0000","last_became_active":"2024-09-18T08:02:51.902163+0000","last_became_peered":"2024-09-18T08:02:51.902163+0000","last_unstale":"2024-09-18T08:09:34.397353+0000","last_undegraded":"2024-09-18T08:09:34.397353+0000","last_fullsized":"2024-09-18T08:09:34.397353+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:08:15.744245+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:50.886830+0000","last_clean_scrub_stamp":"2024-09-18T08:08:15.744245+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:09:40.197923+0000","scrub_duration":74,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":322,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-09-18T08:09:34.398181+0000","last_change":"2024-09-18T08:07:44.113695+0000","last_active":"2024-09-18T08:09:34.398181+0000","last_peered":"2024-09-18T08:09:34.398181+0000","last_clean":"2024-09-18T08:09:34.398181+0000","last_became_active":"2024-09-18T08:02:53.907412+0000","last_became_peered":"2024-09-18T08:02:53.907412+0000","last_unstale":"2024-09-18T08:09:34.398181+0000","last_undegraded":"2024-09-18T08:09:34.398181+0000","last_fullsized":"2024-09-18T08:09:34.398181+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:07:44.113639+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_clean_scrub_stamp":"2024-09-18T08:07:44.113639+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.4","version":"213'7233","reported_seq":17739,"reported_epoch":213,"state":"active+recovering+undersized+remapped","last_fresh":"2024-09-18T08:09:43.251528+0000","last_change":"2024-09-18T08:08:56.896339+0000","last_active":"2024-09-18T08:09:43.251528+0000","last_peered":"2024-09-18T08:09:43.251528+0000","last_clean":"2024-09-18T08:08:51.672524+0000","last_became_active":"2024-09-18T08:08:53.828580+0000","last_became_peered":"2024-09-18T08:08:53.828580+0000","last_unstale":"2024-09-18T08:09:43.251528+0000","last_undegraded":"2024-09-18T08:09:43.251528+0000","last_fullsized":"2024-09-18T08:08:53.682963+0000","mapping_epoch":199,"log_start":"180'2100","ondisk_log_start":"180'2100","created":165,"last_epoch_clean":190,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:07:52.307205+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:07:52.307205+0000","last_clean_scrub_stamp":"2024-09-18T08:07:52.307205+0000","objects_scrubbed":0,"log_size":5133,"log_dups_size":0,"ondisk_log_size":5133,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":349765632,"num_objects":5337,"num_object_clones":0,"num_object_copies":16011,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1921,"num_objects_unfound":0,"num_objects_dirty":5337,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7233,"num_write_kb":402240,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5186,"num_bytes_recovered":339869696,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,2],"acting":[5,4],"avail_no_missing":["5","3","4","6"],"object_location_counts":[{"shards":"4,5","objects":3416},{"shards":"4,5,6","objects":1921}],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.8","version":"69'15","reported_seq":391,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-09-18T08:09:34.395980+0000","last_change":"2024-09-18T08:06:24.162625+0000","last_active":"2024-09-18T08:09:34.395980+0000","last_peered":"2024-09-18T08:09:34.395980+0000","last_clean":"2024-09-18T08:09:34.395980+0000","last_became_active":"2024-09-18T08:06:24.138259+0000","last_became_peered":"2024-09-18T08:06:24.138259+0000","last_unstale":"2024-09-18T08:09:34.395980+0000","last_undegraded":"2024-09-18T08:09:34.395980+0000","last_fullsized":"2024-09-18T08:09:34.395980+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'15","last_scrub_stamp":"2024-09-18T08:04:26.859044+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_clean_scrub_stamp":"2024-09-18T08:04:26.859044+0000","objects_scrubbed":7,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.f","version":"62'2","reported_seq":378,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-09-18T08:09:34.409408+0000","last_change":"2024-09-18T08:08:18.629506+0000","last_active":"2024-09-18T08:09:34.409408+0000","last_peered":"2024-09-18T08:09:34.409408+0000","last_clean":"2024-09-18T08:09:34.409408+0000","last_became_active":"2024-09-18T08:08:18.524593+0000","last_became_peered":"2024-09-18T08:08:18.524593+0000","last_unstale":"2024-09-18T08:09:34.409408+0000","last_undegraded":"2024-09-18T08:09:34.409408+0000","last_fullsized":"2024-09-18T08:09:34.409408+0000","mapping_epoch":181,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":182,"parent":"0.0","parent_split_bits":0,"last_scrub":"62'2","last_scrub_stamp":"2024-09-18T08:07:43.540222+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:46.860092+0000","last_clean_scrub_stamp":"2024-09-18T08:07:43.540222+0000","objects_scrubbed":2,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:09:45.671982+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":4,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":184,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.9","version":"182'12","reported_seq":472,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-09-18T08:09:38.010195+0000","last_change":"2024-09-18T08:08:18.570029+0000","last_active":"2024-09-18T08:09:38.010195+0000","last_peered":"2024-09-18T08:09:38.010195+0000","last_clean":"2024-09-18T08:09:38.010195+0000","last_became_active":"2024-09-18T08:08:18.524858+0000","last_became_peered":"2024-09-18T08:08:18.524858+0000","last_unstale":"2024-09-18T08:09:38.010195+0000","last_undegraded":"2024-09-18T08:09:38.010195+0000","last_fullsized":"2024-09-18T08:09:38.010195+0000","mapping_epoch":181,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":182,"parent":"0.0","parent_split_bits":0,"last_scrub":"138'10","last_scrub_stamp":"2024-09-18T08:07:23.729715+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:50.886830+0000","last_clean_scrub_stamp":"2024-09-18T08:07:23.729715+0000","objects_scrubbed":1,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":334,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-09-18T08:09:34.396115+0000","last_change":"2024-09-18T08:07:45.707631+0000","last_active":"2024-09-18T08:09:34.396115+0000","last_peered":"2024-09-18T08:09:34.396115+0000","last_clean":"2024-09-18T08:09:34.396115+0000","last_became_active":"2024-09-18T08:06:24.133119+0000","last_became_peered":"2024-09-18T08:06:24.133119+0000","last_unstale":"2024-09-18T08:09:34.396115+0000","last_undegraded":"2024-09-18T08:09:34.396115+0000","last_fullsized":"2024-09-18T08:09:34.396115+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:07:45.707562+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_clean_scrub_stamp":"2024-09-18T08:07:45.707562+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":250,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-09-18T08:09:34.404149+0000","last_change":"2024-09-18T08:06:24.133955+0000","last_active":"2024-09-18T08:09:34.404149+0000","last_peered":"2024-09-18T08:09:34.404149+0000","last_clean":"2024-09-18T08:09:34.404149+0000","last_became_active":"2024-09-18T08:06:24.133644+0000","last_became_peered":"2024-09-18T08:06:24.133644+0000","last_unstale":"2024-09-18T08:09:34.404149+0000","last_undegraded":"2024-09-18T08:09:34.404149+0000","last_fullsized":"2024-09-18T08:09:34.404149+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:04:16.378205+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:46.860092+0000","last_clean_scrub_stamp":"2024-09-18T08:04:16.378205+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:10:20.052218+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.17","version":"69'6","reported_seq":345,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-09-18T08:09:34.394693+0000","last_change":"2024-09-18T08:07:19.309941+0000","last_active":"2024-09-18T08:09:34.394693+0000","last_peered":"2024-09-18T08:09:34.394693+0000","last_clean":"2024-09-18T08:09:34.394693+0000","last_became_active":"2024-09-18T08:02:49.895067+0000","last_became_peered":"2024-09-18T08:02:49.895067+0000","last_unstale":"2024-09-18T08:09:34.394693+0000","last_undegraded":"2024-09-18T08:09:34.394693+0000","last_fullsized":"2024-09-18T08:09:34.394693+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'6","last_scrub_stamp":"2024-09-18T08:07:19.309844+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_clean_scrub_stamp":"2024-09-18T08:07:19.309844+0000","objects_scrubbed":3,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":26,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":9,"num_read_kb":6,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":323,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-09-18T08:09:34.396336+0000","last_change":"2024-09-18T08:05:45.812436+0000","last_active":"2024-09-18T08:09:34.396336+0000","last_peered":"2024-09-18T08:09:34.396336+0000","last_clean":"2024-09-18T08:09:34.396336+0000","last_became_active":"2024-09-18T08:02:51.912557+0000","last_became_peered":"2024-09-18T08:02:51.912557+0000","last_unstale":"2024-09-18T08:09:34.396336+0000","last_undegraded":"2024-09-18T08:09:34.396336+0000","last_fullsized":"2024-09-18T08:09:34.396336+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:05:45.812389+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:50.886830+0000","last_clean_scrub_stamp":"2024-09-18T08:05:45.812389+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,1],"acting":[5,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":356,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-09-18T08:09:34.408407+0000","last_change":"2024-09-18T08:08:18.526113+0000","last_active":"2024-09-18T08:09:34.408407+0000","last_peered":"2024-09-18T08:09:34.408407+0000","last_clean":"2024-09-18T08:09:34.408407+0000","last_became_active":"2024-09-18T08:08:18.525260+0000","last_became_peered":"2024-09-18T08:08:18.525260+0000","last_unstale":"2024-09-18T08:09:34.408407+0000","last_undegraded":"2024-09-18T08:09:34.408407+0000","last_fullsized":"2024-09-18T08:09:34.408407+0000","mapping_epoch":181,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":182,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:07:53.670107+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_clean_scrub_stamp":"2024-09-18T08:07:53.670107+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:10:19.732636+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"69'9","reported_seq":343,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-09-18T08:09:34.406049+0000","last_change":"2024-09-18T08:07:23.638373+0000","last_active":"2024-09-18T08:09:34.406049+0000","last_peered":"2024-09-18T08:09:34.406049+0000","last_clean":"2024-09-18T08:09:34.406049+0000","last_became_active":"2024-09-18T08:02:49.891567+0000","last_became_peered":"2024-09-18T08:02:49.891567+0000","last_unstale":"2024-09-18T08:09:34.406049+0000","last_undegraded":"2024-09-18T08:09:34.406049+0000","last_fullsized":"2024-09-18T08:09:34.406049+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'9","last_scrub_stamp":"2024-09-18T08:07:23.638294+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_clean_scrub_stamp":"2024-09-18T08:07:23.638294+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":74,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-09-18T08:09:34.394399+0000","last_change":"2024-09-18T08:08:18.550531+0000","last_active":"2024-09-18T08:09:34.394399+0000","last_peered":"2024-09-18T08:09:34.394399+0000","last_clean":"2024-09-18T08:09:34.394399+0000","last_became_active":"2024-09-18T08:08:18.545953+0000","last_became_peered":"2024-09-18T08:08:18.545953+0000","last_unstale":"2024-09-18T08:09:34.394399+0000","last_undegraded":"2024-09-18T08:09:34.394399+0000","last_fullsized":"2024-09-18T08:09:34.394399+0000","mapping_epoch":181,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":182,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:06:13.214705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:46.860092+0000","last_clean_scrub_stamp":"2024-09-18T08:06:13.214705+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":316,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-09-18T08:09:34.394479+0000","last_change":"2024-09-18T08:05:42.061649+0000","last_active":"2024-09-18T08:09:34.394479+0000","last_peered":"2024-09-18T08:09:34.394479+0000","last_clean":"2024-09-18T08:09:34.394479+0000","last_became_active":"2024-09-18T08:02:51.907361+0000","last_became_peered":"2024-09-18T08:02:51.907361+0000","last_unstale":"2024-09-18T08:09:34.394479+0000","last_undegraded":"2024-09-18T08:09:34.394479+0000","last_fullsized":"2024-09-18T08:09:34.394479+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:05:42.061452+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:50.886830+0000","last_clean_scrub_stamp":"2024-09-18T08:05:42.061452+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":335,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-09-18T08:09:34.401048+0000","last_change":"2024-09-18T08:08:18.511719+0000","last_active":"2024-09-18T08:09:34.401048+0000","last_peered":"2024-09-18T08:09:34.401048+0000","last_clean":"2024-09-18T08:09:34.401048+0000","last_became_active":"2024-09-18T08:08:18.511019+0000","last_became_peered":"2024-09-18T08:08:18.511019+0000","last_unstale":"2024-09-18T08:09:34.401048+0000","last_undegraded":"2024-09-18T08:09:34.401048+0000","last_fullsized":"2024-09-18T08:09:34.401048+0000","mapping_epoch":181,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":182,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:08:10.660508+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_clean_scrub_stamp":"2024-09-18T08:08:10.660508+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.15","version":"69'9","reported_seq":343,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-09-18T08:09:34.396784+0000","last_change":"2024-09-18T08:05:36.896299+0000","last_active":"2024-09-18T08:09:34.396784+0000","last_peered":"2024-09-18T08:09:34.396784+0000","last_clean":"2024-09-18T08:09:34.396784+0000","last_became_active":"2024-09-18T08:02:49.890497+0000","last_became_peered":"2024-09-18T08:02:49.890497+0000","last_unstale":"2024-09-18T08:09:34.396784+0000","last_undegraded":"2024-09-18T08:09:34.396784+0000","last_fullsized":"2024-09-18T08:09:34.396784+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'9","last_scrub_stamp":"2024-09-18T08:05:36.896238+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_clean_scrub_stamp":"2024-09-18T08:05:36.896238+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":339,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-09-18T08:09:34.404537+0000","last_change":"2024-09-18T08:07:48.653905+0000","last_active":"2024-09-18T08:09:34.404537+0000","last_peered":"2024-09-18T08:09:34.404537+0000","last_clean":"2024-09-18T08:09:34.404537+0000","last_became_active":"2024-09-18T08:02:47.875983+0000","last_became_peered":"2024-09-18T08:02:47.875983+0000","last_unstale":"2024-09-18T08:09:34.404537+0000","last_undegraded":"2024-09-18T08:09:34.404537+0000","last_fullsized":"2024-09-18T08:09:34.404537+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:07:48.653856+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:46.860092+0000","last_clean_scrub_stamp":"2024-09-18T08:07:48.653856+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.14","version":"199'12","reported_seq":440,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-09-18T08:09:38.010980+0000","last_change":"2024-09-18T08:08:52.701287+0000","last_active":"2024-09-18T08:09:38.010980+0000","last_peered":"2024-09-18T08:09:38.010980+0000","last_clean":"2024-09-18T08:09:38.010980+0000","last_became_active":"2024-09-18T08:08:52.696201+0000","last_became_peered":"2024-09-18T08:08:52.696201+0000","last_unstale":"2024-09-18T08:09:38.010980+0000","last_undegraded":"2024-09-18T08:09:38.010980+0000","last_fullsized":"2024-09-18T08:09:38.010980+0000","mapping_epoch":198,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":199,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'8","last_scrub_stamp":"2024-09-18T08:07:16.241607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:50.886830+0000","last_clean_scrub_stamp":"2024-09-18T08:07:16.241607+0000","objects_scrubbed":1,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":74,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-09-18T08:09:34.394195+0000","last_change":"2024-09-18T08:08:18.523205+0000","last_active":"2024-09-18T08:09:34.394195+0000","last_peered":"2024-09-18T08:09:34.394195+0000","last_clean":"2024-09-18T08:09:34.394195+0000","last_became_active":"2024-09-18T08:08:18.522483+0000","last_became_peered":"2024-09-18T08:08:18.522483+0000","last_unstale":"2024-09-18T08:09:34.394195+0000","last_undegraded":"2024-09-18T08:09:34.394195+0000","last_fullsized":"2024-09-18T08:09:34.394195+0000","mapping_epoch":181,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":182,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:06:43.510513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_clean_scrub_stamp":"2024-09-18T08:06:43.510513+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.14","version":"69'10","reported_seq":362,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-09-18T08:09:34.394989+0000","last_change":"2024-09-18T08:08:08.493614+0000","last_active":"2024-09-18T08:09:34.394989+0000","last_peered":"2024-09-18T08:09:34.394989+0000","last_clean":"2024-09-18T08:09:34.394989+0000","last_became_active":"2024-09-18T08:04:18.150968+0000","last_became_peered":"2024-09-18T08:04:18.150968+0000","last_unstale":"2024-09-18T08:09:34.394989+0000","last_undegraded":"2024-09-18T08:09:34.394989+0000","last_fullsized":"2024-09-18T08:09:34.394989+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'10","last_scrub_stamp":"2024-09-18T08:08:08.493522+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_clean_scrub_stamp":"2024-09-18T08:08:08.493522+0000","objects_scrubbed":5,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,7],"acting":[3,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":353,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-09-18T08:09:34.409609+0000","last_change":"2024-09-18T08:09:31.658337+0000","last_active":"2024-09-18T08:09:34.409609+0000","last_peered":"2024-09-18T08:09:34.409609+0000","last_clean":"2024-09-18T08:09:34.409609+0000","last_became_active":"2024-09-18T08:08:18.524218+0000","last_became_peered":"2024-09-18T08:08:18.524218+0000","last_unstale":"2024-09-18T08:09:34.409609+0000","last_undegraded":"2024-09-18T08:09:34.409609+0000","last_fullsized":"2024-09-18T08:09:34.409609+0000","mapping_epoch":181,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":182,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:09:31.658259+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:46.860092+0000","last_clean_scrub_stamp":"2024-09-18T08:09:31.658259+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:10:50.143994+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.15","version":"69'8","reported_seq":423,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-09-18T08:09:38.009259+0000","last_change":"2024-09-18T08:06:45.555513+0000","last_active":"2024-09-18T08:09:38.009259+0000","last_peered":"2024-09-18T08:09:38.009259+0000","last_clean":"2024-09-18T08:09:38.009259+0000","last_became_active":"2024-09-18T08:02:51.914135+0000","last_became_peered":"2024-09-18T08:02:51.914135+0000","last_unstale":"2024-09-18T08:09:38.009259+0000","last_undegraded":"2024-09-18T08:09:38.009259+0000","last_fullsized":"2024-09-18T08:09:38.009259+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'8","last_scrub_stamp":"2024-09-18T08:06:45.555459+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:50.886830+0000","last_clean_scrub_stamp":"2024-09-18T08:06:45.555459+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":328,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-09-18T08:09:42.047337+0000","last_change":"2024-09-18T08:09:42.047337+0000","last_active":"2024-09-18T08:09:42.047337+0000","last_peered":"2024-09-18T08:09:42.047337+0000","last_clean":"2024-09-18T08:09:42.047337+0000","last_became_active":"2024-09-18T08:02:53.912112+0000","last_became_peered":"2024-09-18T08:02:53.912112+0000","last_unstale":"2024-09-18T08:09:42.047337+0000","last_undegraded":"2024-09-18T08:09:42.047337+0000","last_fullsized":"2024-09-18T08:09:42.047337+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:09:42.047270+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_clean_scrub_stamp":"2024-09-18T08:09:42.047270+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:10:59.660034+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.13","version":"69'11","reported_seq":86,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-09-18T08:09:34.393829+0000","last_change":"2024-09-18T08:08:18.643772+0000","last_active":"2024-09-18T08:09:34.393829+0000","last_peered":"2024-09-18T08:09:34.393829+0000","last_clean":"2024-09-18T08:09:34.393829+0000","last_became_active":"2024-09-18T08:08:18.539712+0000","last_became_peered":"2024-09-18T08:08:18.539712+0000","last_unstale":"2024-09-18T08:09:34.393829+0000","last_undegraded":"2024-09-18T08:09:34.393829+0000","last_fullsized":"2024-09-18T08:09:34.393829+0000","mapping_epoch":181,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":182,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'11","last_scrub_stamp":"2024-09-18T08:07:53.824401+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_clean_scrub_stamp":"2024-09-18T08:07:53.824401+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":19,"num_read_kb":12,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":74,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-09-18T08:09:34.394922+0000","last_change":"2024-09-18T08:08:18.623643+0000","last_active":"2024-09-18T08:09:34.394922+0000","last_peered":"2024-09-18T08:09:34.394922+0000","last_clean":"2024-09-18T08:09:34.394922+0000","last_became_active":"2024-09-18T08:08:18.622893+0000","last_became_peered":"2024-09-18T08:08:18.622893+0000","last_unstale":"2024-09-18T08:09:34.394922+0000","last_undegraded":"2024-09-18T08:09:34.394922+0000","last_fullsized":"2024-09-18T08:09:34.394922+0000","mapping_epoch":181,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":182,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:06:46.502422+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:46.860092+0000","last_clean_scrub_stamp":"2024-09-18T08:06:46.502422+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":331,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-09-18T08:09:34.409979+0000","last_change":"2024-09-18T08:09:03.582984+0000","last_active":"2024-09-18T08:09:34.409979+0000","last_peered":"2024-09-18T08:09:34.409979+0000","last_clean":"2024-09-18T08:09:34.409979+0000","last_became_active":"2024-09-18T08:02:51.910752+0000","last_became_peered":"2024-09-18T08:02:51.910752+0000","last_unstale":"2024-09-18T08:09:34.409979+0000","last_undegraded":"2024-09-18T08:09:34.409979+0000","last_fullsized":"2024-09-18T08:09:34.409979+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:09:03.582914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:50.886830+0000","last_clean_scrub_stamp":"2024-09-18T08:09:03.582914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:10:14.069633+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":329,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-09-18T08:09:34.395894+0000","last_change":"2024-09-18T08:07:54.314026+0000","last_active":"2024-09-18T08:09:34.395894+0000","last_peered":"2024-09-18T08:09:34.395894+0000","last_clean":"2024-09-18T08:09:34.395894+0000","last_became_active":"2024-09-18T08:02:53.911429+0000","last_became_peered":"2024-09-18T08:02:53.911429+0000","last_unstale":"2024-09-18T08:09:34.395894+0000","last_undegraded":"2024-09-18T08:09:34.395894+0000","last_fullsized":"2024-09-18T08:09:34.395894+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:07:54.313971+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_clean_scrub_stamp":"2024-09-18T08:07:54.313971+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.12","version":"69'9","reported_seq":369,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-09-18T08:09:34.410224+0000","last_change":"2024-09-18T08:08:52.658353+0000","last_active":"2024-09-18T08:09:34.410224+0000","last_peered":"2024-09-18T08:09:34.410224+0000","last_clean":"2024-09-18T08:09:34.410224+0000","last_became_active":"2024-09-18T08:06:24.132270+0000","last_became_peered":"2024-09-18T08:06:24.132270+0000","last_unstale":"2024-09-18T08:09:34.410224+0000","last_undegraded":"2024-09-18T08:09:34.410224+0000","last_fullsized":"2024-09-18T08:09:34.410224+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'9","last_scrub_stamp":"2024-09-18T08:08:52.658282+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_clean_scrub_stamp":"2024-09-18T08:08:52.658282+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:10:04.461414+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":361,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-09-18T08:09:37.532792+0000","last_change":"2024-09-18T08:09:37.532792+0000","last_active":"2024-09-18T08:09:37.532792+0000","last_peered":"2024-09-18T08:09:37.532792+0000","last_clean":"2024-09-18T08:09:37.532792+0000","last_became_active":"2024-09-18T08:08:18.525527+0000","last_became_peered":"2024-09-18T08:08:18.525527+0000","last_unstale":"2024-09-18T08:09:37.532792+0000","last_undegraded":"2024-09-18T08:09:37.532792+0000","last_fullsized":"2024-09-18T08:09:37.532792+0000","mapping_epoch":181,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":182,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:09:37.532573+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:46.860092+0000","last_clean_scrub_stamp":"2024-09-18T08:09:37.532573+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:11:05.489513+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":324,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-09-18T08:09:34.398230+0000","last_change":"2024-09-18T08:07:42.165434+0000","last_active":"2024-09-18T08:09:34.398230+0000","last_peered":"2024-09-18T08:09:34.398230+0000","last_clean":"2024-09-18T08:09:34.398230+0000","last_became_active":"2024-09-18T08:02:51.906277+0000","last_became_peered":"2024-09-18T08:02:51.906277+0000","last_unstale":"2024-09-18T08:09:34.398230+0000","last_undegraded":"2024-09-18T08:09:34.398230+0000","last_fullsized":"2024-09-18T08:09:34.398230+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:07:42.165388+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:50.886830+0000","last_clean_scrub_stamp":"2024-09-18T08:07:42.165388+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.10","version":"69'1","reported_seq":308,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-09-18T08:09:34.403774+0000","last_change":"2024-09-18T08:05:51.429631+0000","last_active":"2024-09-18T08:09:34.403774+0000","last_peered":"2024-09-18T08:09:34.403774+0000","last_clean":"2024-09-18T08:09:34.403774+0000","last_became_active":"2024-09-18T08:02:53.912699+0000","last_became_peered":"2024-09-18T08:02:53.912699+0000","last_unstale":"2024-09-18T08:09:34.403774+0000","last_undegraded":"2024-09-18T08:09:34.403774+0000","last_fullsized":"2024-09-18T08:09:34.403774+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_clean_scrub_stamp":"2024-09-18T08:02:52.890475+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:10:16.084523+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.11","version":"69'11","reported_seq":393,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-09-18T08:09:34.396076+0000","last_change":"2024-09-18T08:07:18.282247+0000","last_active":"2024-09-18T08:09:34.396076+0000","last_peered":"2024-09-18T08:09:34.396076+0000","last_clean":"2024-09-18T08:09:34.396076+0000","last_became_active":"2024-09-18T08:06:24.132416+0000","last_became_peered":"2024-09-18T08:06:24.132416+0000","last_unstale":"2024-09-18T08:09:34.396076+0000","last_undegraded":"2024-09-18T08:09:34.396076+0000","last_fullsized":"2024-09-18T08:09:34.396076+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'11","last_scrub_stamp":"2024-09-18T08:07:18.282155+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_clean_scrub_stamp":"2024-09-18T08:07:18.282155+0000","objects_scrubbed":5,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":331,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-09-18T08:09:34.396419+0000","last_change":"2024-09-18T08:06:18.168254+0000","last_active":"2024-09-18T08:09:34.396419+0000","last_peered":"2024-09-18T08:09:34.396419+0000","last_clean":"2024-09-18T08:09:34.396419+0000","last_became_active":"2024-09-18T08:02:47.878886+0000","last_became_peered":"2024-09-18T08:02:47.878886+0000","last_unstale":"2024-09-18T08:09:34.396419+0000","last_undegraded":"2024-09-18T08:09:34.396419+0000","last_fullsized":"2024-09-18T08:09:34.396419+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:06:18.168206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:46.860092+0000","last_clean_scrub_stamp":"2024-09-18T08:06:18.168206+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":359,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-09-18T08:09:34.407989+0000","last_change":"2024-09-18T08:08:18.525683+0000","last_active":"2024-09-18T08:09:34.407989+0000","last_peered":"2024-09-18T08:09:34.407989+0000","last_clean":"2024-09-18T08:09:34.407989+0000","last_became_active":"2024-09-18T08:08:18.524966+0000","last_became_peered":"2024-09-18T08:08:18.524966+0000","last_unstale":"2024-09-18T08:09:34.407989+0000","last_undegraded":"2024-09-18T08:09:34.407989+0000","last_fullsized":"2024-09-18T08:09:34.407989+0000","mapping_epoch":181,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":182,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:07:21.724545+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:50.886830+0000","last_clean_scrub_stamp":"2024-09-18T08:07:21.724545+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:09:43.621477+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":334,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-09-18T08:09:34.398379+0000","last_change":"2024-09-18T08:07:48.218010+0000","last_active":"2024-09-18T08:09:34.398379+0000","last_peered":"2024-09-18T08:09:34.398379+0000","last_clean":"2024-09-18T08:09:34.398379+0000","last_became_active":"2024-09-18T08:06:24.128582+0000","last_became_peered":"2024-09-18T08:06:24.128582+0000","last_unstale":"2024-09-18T08:09:34.398379+0000","last_undegraded":"2024-09-18T08:09:34.398379+0000","last_fullsized":"2024-09-18T08:09:34.398379+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:07:48.217964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_clean_scrub_stamp":"2024-09-18T08:07:48.217964+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.10","version":"69'4","reported_seq":360,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-09-18T08:09:34.394990+0000","last_change":"2024-09-18T08:07:41.210418+0000","last_active":"2024-09-18T08:09:34.394990+0000","last_peered":"2024-09-18T08:09:34.394990+0000","last_clean":"2024-09-18T08:09:34.394990+0000","last_became_active":"2024-09-18T08:06:24.131309+0000","last_became_peered":"2024-09-18T08:06:24.131309+0000","last_unstale":"2024-09-18T08:09:34.394990+0000","last_undegraded":"2024-09-18T08:09:34.394990+0000","last_fullsized":"2024-09-18T08:09:34.394990+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'4","last_scrub_stamp":"2024-09-18T08:07:41.210347+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_clean_scrub_stamp":"2024-09-18T08:07:41.210347+0000","objects_scrubbed":2,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":6,"num_read_kb":4,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":339,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-09-18T08:09:34.404820+0000","last_change":"2024-09-18T08:07:52.551689+0000","last_active":"2024-09-18T08:09:34.404820+0000","last_peered":"2024-09-18T08:09:34.404820+0000","last_clean":"2024-09-18T08:09:34.404820+0000","last_became_active":"2024-09-18T08:02:47.877616+0000","last_became_peered":"2024-09-18T08:02:47.877616+0000","last_unstale":"2024-09-18T08:09:34.404820+0000","last_undegraded":"2024-09-18T08:09:34.404820+0000","last_fullsized":"2024-09-18T08:09:34.404820+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:07:52.551616+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:46.860092+0000","last_clean_scrub_stamp":"2024-09-18T08:07:52.551616+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":74,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-09-18T08:09:34.394578+0000","last_change":"2024-09-18T08:08:18.550244+0000","last_active":"2024-09-18T08:09:34.394578+0000","last_peered":"2024-09-18T08:09:34.394578+0000","last_clean":"2024-09-18T08:09:34.394578+0000","last_became_active":"2024-09-18T08:08:18.545361+0000","last_became_peered":"2024-09-18T08:08:18.545361+0000","last_unstale":"2024-09-18T08:09:34.394578+0000","last_undegraded":"2024-09-18T08:09:34.394578+0000","last_fullsized":"2024-09-18T08:09:34.394578+0000","mapping_epoch":181,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":182,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:07:17.605562+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:50.886830+0000","last_clean_scrub_stamp":"2024-09-18T08:07:17.605562+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,7],"acting":[4,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":350,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-09-18T08:09:34.607443+0000","last_change":"2024-09-18T08:09:34.607443+0000","last_active":"2024-09-18T08:09:34.607443+0000","last_peered":"2024-09-18T08:09:34.607443+0000","last_clean":"2024-09-18T08:09:34.607443+0000","last_became_active":"2024-09-18T08:08:18.525552+0000","last_became_peered":"2024-09-18T08:08:18.525552+0000","last_unstale":"2024-09-18T08:09:34.607443+0000","last_undegraded":"2024-09-18T08:09:34.607443+0000","last_fullsized":"2024-09-18T08:09:34.607443+0000","mapping_epoch":181,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":182,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:09:34.607374+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_clean_scrub_stamp":"2024-09-18T08:09:34.607374+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:10:39.783401+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,4],"acting":[7,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":318,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-09-18T08:09:34.409888+0000","last_change":"2024-09-18T08:08:18.539359+0000","last_active":"2024-09-18T08:09:34.409888+0000","last_peered":"2024-09-18T08:09:34.409888+0000","last_clean":"2024-09-18T08:09:34.409888+0000","last_became_active":"2024-09-18T08:08:18.538838+0000","last_became_peered":"2024-09-18T08:08:18.538838+0000","last_unstale":"2024-09-18T08:09:34.409888+0000","last_undegraded":"2024-09-18T08:09:34.409888+0000","last_fullsized":"2024-09-18T08:09:34.409888+0000","mapping_epoch":181,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":182,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_clean_scrub_stamp":"2024-09-18T08:02:52.890475+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:09:50.664384+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":332,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-09-18T08:09:34.395267+0000","last_change":"2024-09-18T08:07:17.277964+0000","last_active":"2024-09-18T08:09:34.395267+0000","last_peered":"2024-09-18T08:09:34.395267+0000","last_clean":"2024-09-18T08:09:34.395267+0000","last_became_active":"2024-09-18T08:02:47.878922+0000","last_became_peered":"2024-09-18T08:02:47.878922+0000","last_unstale":"2024-09-18T08:09:34.395267+0000","last_undegraded":"2024-09-18T08:09:34.395267+0000","last_fullsized":"2024-09-18T08:09:34.395267+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:07:17.277894+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:46.860092+0000","last_clean_scrub_stamp":"2024-09-18T08:07:17.277894+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":16,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"69'11","reported_seq":321,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-09-18T08:09:34.396992+0000","last_change":"2024-09-18T08:06:24.164086+0000","last_active":"2024-09-18T08:09:34.396992+0000","last_peered":"2024-09-18T08:09:34.396992+0000","last_clean":"2024-09-18T08:09:34.396992+0000","last_became_active":"2024-09-18T08:06:24.135232+0000","last_became_peered":"2024-09-18T08:06:24.135232+0000","last_unstale":"2024-09-18T08:09:34.396992+0000","last_undegraded":"2024-09-18T08:09:34.396992+0000","last_fullsized":"2024-09-18T08:09:34.396992+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'11","last_scrub_stamp":"2024-09-18T08:06:12.967462+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_clean_scrub_stamp":"2024-09-18T08:06:12.967462+0000","objects_scrubbed":5,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":19,"num_read_kb":12,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,1],"acting":[5,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":338,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-09-18T08:09:34.405747+0000","last_change":"2024-09-18T08:08:52.701053+0000","last_active":"2024-09-18T08:09:34.405747+0000","last_peered":"2024-09-18T08:09:34.405747+0000","last_clean":"2024-09-18T08:09:34.405747+0000","last_became_active":"2024-09-18T08:08:52.700595+0000","last_became_peered":"2024-09-18T08:08:52.700595+0000","last_unstale":"2024-09-18T08:09:34.405747+0000","last_undegraded":"2024-09-18T08:09:34.405747+0000","last_fullsized":"2024-09-18T08:09:34.405747+0000","mapping_epoch":198,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":199,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:08:12.639376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:50.886830+0000","last_clean_scrub_stamp":"2024-09-18T08:08:12.639376+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":8,"num_pg":16,"stat_sum":{"num_bytes":5569576984,"num_objects":84986,"num_object_clones":0,"num_object_copies":254958,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1921,"num_objects_unfound":0,"num_objects_dirty":84986,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":116514,"num_write_kb":6447937,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":22323,"num_bytes_recovered":1462960128,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":17805619200,"data_stored":17805606984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":78340,"ondisk_log_size":78340,"up":48,"acting":47,"num_store_stats":8},{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":416,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":85,"ondisk_log_size":85,"up":96,"acting":96,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":178,"num_object_clones":0,"num_object_copies":534,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":178,"num_whiteouts":0,"num_read":685,"num_read_kb":442,"num_write":417,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":214,"num_bytes_recovered":4114,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":417792,"data_stored":11106,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":398,"ondisk_log_size":398,"up":96,"acting":96,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":1613,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":24,"num_read_kb":24,"num_write":10,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":3226,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":4839,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":6,"ondisk_log_size":6,"up":96,"acting":96,"num_store_stats":8},{"poolid":2,"num_pg":3,"stat_sum":{"num_bytes":408,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":183,"num_read_kb":178,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":38,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":8,"ondisk_log_size":8,"up":9,"acting":9,"num_store_stats":7},{"poolid":1,"num_pg":11,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":869,"ondisk_log_size":869,"up":33,"acting":33,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":56,"seq":240518168663,"num_pgs":63,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1965268,"kb_used_data":1785540,"kb_used_omap":1,"kb_used_meta":179710,"kb_avail":91779884,"statfs":{"total":95995035648,"available":93982601216,"internally_reserved":0,"allocated":1828392960,"data_stored":1828011296,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":184023498},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":47,"seq":201863463005,"num_pgs":55,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2236348,"kb_used_data":2048236,"kb_used_omap":1,"kb_used_meta":188030,"kb_avail":91508804,"statfs":{"total":95995035648,"available":93705015296,"internally_reserved":0,"allocated":2097393664,"data_stored":2097035467,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":192543174},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":39,"seq":167503724643,"num_pgs":59,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2892176,"kb_used_data":2726336,"kb_used_omap":1,"kb_used_meta":165822,"kb_avail":90852976,"statfs":{"total":95995035648,"available":93033447424,"internally_reserved":0,"allocated":2791768064,"data_stored":2791390763,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":169802182},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":32,"seq":137438953576,"num_pgs":67,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2328156,"kb_used_data":2098444,"kb_used_omap":1,"kb_used_meta":229694,"kb_avail":91416996,"statfs":{"total":95995035648,"available":93611003904,"internally_reserved":0,"allocated":2148806656,"data_stored":2148417289,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":235207109},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":26,"seq":111669149807,"num_pgs":69,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2445540,"kb_used_data":2135316,"kb_used_omap":1,"kb_used_meta":310206,"kb_avail":91299612,"statfs":{"total":95995035648,"available":93490802688,"internally_reserved":0,"allocated":2186563584,"data_stored":2186165862,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":317651400},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411445,"num_pgs":49,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2227536,"kb_used_data":1939520,"kb_used_omap":1,"kb_used_meta":287998,"kb_avail":91517616,"statfs":{"total":95995035648,"available":93714038784,"internally_reserved":0,"allocated":1986068480,"data_stored":1985691285,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":294910405},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574969,"num_pgs":63,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2651068,"kb_used_data":2399148,"kb_used_omap":1,"kb_used_meta":251902,"kb_avail":91094084,"statfs":{"total":95995035648,"available":93280342016,"internally_reserved":0,"allocated":2456727552,"data_stored":2456306716,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":257948101},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738496,"num_pgs":56,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2584348,"kb_used_data":2332364,"kb_used_omap":1,"kb_used_meta":251902,"kb_avail":91160804,"statfs":{"total":95995035648,"available":93348663296,"internally_reserved":0,"allocated":2388340736,"data_stored":2387951571,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":257948101},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":266240,"data_stored":262688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1131,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1521,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":436,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":46,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1177,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":436,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":77824,"data_stored":2008,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1128,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":1898,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1474,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":990,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1034,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1254,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":403,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":416,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":403,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":2366111744,"data_stored":2366111744,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":2453536768,"data_stored":2453536768,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":1982205952,"data_stored":1982201880,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":2178613248,"data_stored":2178613248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":2148007936,"data_stored":2148007936,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":2789085184,"data_stored":2789081112,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":2074542080,"data_stored":2074542080,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":1813516288,"data_stored":1813512216,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T08:09:49.697 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid 779349ca-7593-11ef-bceb-c7b262605968 -- ceph status --format=json 2024-09-18T08:09:49.697 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1 2024-09-18T08:09:49.745 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:09:49.746 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:09:49.746 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1' 2024-09-18T08:09:49.847 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7 2024-09-18T08:09:49.873 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/779349ca-7593-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T08:09:49.874 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:09:49.874 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:09:49.875 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7' 2024-09-18T08:09:49.975 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5 2024-09-18T08:09:50.011 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:09:50.012 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:09:50.012 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5' 2024-09-18T08:09:50.113 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7 2024-09-18T08:09:50.145 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:09:50.146 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:09:50.146 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7' 2024-09-18T08:09:50.247 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3 2024-09-18T08:09:50.301 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:09:50.302 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:09:50.302 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3' 2024-09-18T08:09:50.403 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0 2024-09-18T08:09:50.439 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:09:50.440 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:09:50.441 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0' 2024-09-18T08:09:50.486 DEBUG:teuthology.orchestra.run.smithi050:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid 779349ca-7593-11ef-bceb-c7b262605968 -- ceph daemon osd.5 dump_ops_in_flight 2024-09-18T08:09:50.522 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/779349ca-7593-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T08:09:50.527 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/779349ca-7593-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T08:09:50.542 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0 2024-09-18T08:09:50.621 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:09:50.622 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:09:50.622 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0' 2024-09-18T08:09:50.673 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:09:50 smithi007 ceph-mon[29459]: from='client.25156 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-18T08:09:50.673 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:09:50 smithi007 ceph-mon[29459]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-18T08:09:50.673 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:09:50 smithi007 ceph-mon[29459]: pgmap v340: 158 pgs: 1 active+recovering+undersized+remapped, 157 active+clean; 4.9 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.12k op/s; 1703/239049 objects misplaced (0.712%); 3.2 MiB/s, 50 objects/s recovering 2024-09-18T08:09:50.673 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:09:50 smithi007 ceph-mon[29459]: 6.4 scrub starts 2024-09-18T08:09:50.673 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:09:50 smithi007 ceph-mon[29459]: 6.4 scrub ok 2024-09-18T08:09:50.674 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:09:50 smithi007 ceph-mon[39249]: from='client.25156 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-18T08:09:50.674 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:09:50 smithi007 ceph-mon[39249]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-18T08:09:50.674 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:09:50 smithi007 ceph-mon[39249]: pgmap v340: 158 pgs: 1 active+recovering+undersized+remapped, 157 active+clean; 4.9 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.12k op/s; 1703/239049 objects misplaced (0.712%); 3.2 MiB/s, 50 objects/s recovering 2024-09-18T08:09:50.674 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:09:50 smithi007 ceph-mon[39249]: 6.4 scrub starts 2024-09-18T08:09:50.675 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:09:50 smithi007 ceph-mon[39249]: 6.4 scrub ok 2024-09-18T08:09:50.723 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1 2024-09-18T08:09:50.762 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:09:50.762 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:09:50.763 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1' 2024-09-18T08:09:50.864 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7 2024-09-18T08:09:50.867 INFO:teuthology.orchestra.run.smithi050.stderr:Inferring config /var/lib/ceph/779349ca-7593-11ef-bceb-c7b262605968/mon.b/config 2024-09-18T08:09:50.905 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:09:50.905 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:09:50.906 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7' 2024-09-18T08:09:50.906 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:09:50 smithi050 ceph-mon[32371]: from='client.25156 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-18T08:09:50.906 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:09:50 smithi050 ceph-mon[32371]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-18T08:09:50.906 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:09:50 smithi050 ceph-mon[32371]: pgmap v340: 158 pgs: 1 active+recovering+undersized+remapped, 157 active+clean; 4.9 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.12k op/s; 1703/239049 objects misplaced (0.712%); 3.2 MiB/s, 50 objects/s recovering 2024-09-18T08:09:50.906 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:09:50 smithi050 ceph-mon[32371]: 6.4 scrub starts 2024-09-18T08:09:50.906 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:09:50 smithi050 ceph-mon[32371]: 6.4 scrub ok 2024-09-18T08:09:51.007 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0 2024-09-18T08:09:51.044 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:09:51.044 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:09:51.045 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0' 2024-09-18T08:09:51.146 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6 2024-09-18T08:09:51.217 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:09:51.235 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:09:51.236 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6' 2024-09-18T08:09:51.337 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1 2024-09-18T08:09:51.377 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:09:51.377 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:09:51.377 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1' 2024-09-18T08:09:51.460 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2024-09-18T08:09:51.460 INFO:teuthology.orchestra.run.smithi050.stdout: "ops": [ 2024-09-18T08:09:51.460 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:09:51.461 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.15216.0:125257 8.4 8:2e1a66f9:::benchmark_data_smithi007_101583_object24505:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e214)", 2024-09-18T08:09:51.461 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-09-18T08:09:51.450876+0000", 2024-09-18T08:09:51.461 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 0.0061605490000000004, 2024-09-18T08:09:51.461 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0061898930000000001, 2024-09-18T08:09:51.461 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-09-18T08:09:51.461 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-09-18T08:09:51.462 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "waiting for sub ops", 2024-09-18T08:09:51.462 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2024-09-18T08:09:51.462 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.15216", 2024-09-18T08:09:51.462 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "172.21.15.7:0/2596041177", 2024-09-18T08:09:51.462 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 125257 2024-09-18T08:09:51.462 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:09:51.462 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-09-18T08:09:51.463 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:09:51.463 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-09-18T08:09:51.463 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:51.450876+0000", 2024-09-18T08:09:51.463 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-18T08:09:51.463 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:09:51.463 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:09:51.463 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-09-18T08:09:51.464 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:51.450876+0000", 2024-09-18T08:09:51.464 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-18T08:09:51.464 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:09:51.464 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:09:51.464 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-09-18T08:09:51.464 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:51.450877+0000", 2024-09-18T08:09:51.465 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.7400000000000001e-06 2024-09-18T08:09:51.465 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:09:51.465 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:09:51.465 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-09-18T08:09:51.465 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:51.450883+0000", 2024-09-18T08:09:51.466 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 5.2800000000000003e-06 2024-09-18T08:09:51.466 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:09:51.466 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:09:51.466 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-09-18T08:09:51.466 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:51.450893+0000", 2024-09-18T08:09:51.466 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.0468999999999999e-05 2024-09-18T08:09:51.466 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:09:51.467 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:09:51.467 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-09-18T08:09:51.467 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:51.450910+0000", 2024-09-18T08:09:51.467 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.6731e-05 2024-09-18T08:09:51.467 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:09:51.467 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:09:51.467 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-18T08:09:51.467 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:51.452409+0000", 2024-09-18T08:09:51.467 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00149899 2024-09-18T08:09:51.467 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:09:51.467 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:09:51.468 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-09-18T08:09:51.468 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:51.452555+0000", 2024-09-18T08:09:51.468 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000146417 2024-09-18T08:09:51.468 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:09:51.468 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:09:51.468 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "waiting for subops from 2,4", 2024-09-18T08:09:51.468 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:51.452693+0000", 2024-09-18T08:09:51.468 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00013783099999999999 2024-09-18T08:09:51.468 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-18T08:09:51.468 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-09-18T08:09:51.469 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-18T08:09:51.469 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:09:51.469 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:09:51.469 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.15216.0:125261 8.4 8:22f58231:::benchmark_data_smithi007_101583_object24509:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e214)", 2024-09-18T08:09:51.469 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-09-18T08:09:51.451399+0000", 2024-09-18T08:09:51.469 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 0.0056368770000000002, 2024-09-18T08:09:51.469 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0057505079999999997, 2024-09-18T08:09:51.470 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-09-18T08:09:51.470 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-09-18T08:09:51.470 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "waiting for sub ops", 2024-09-18T08:09:51.470 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2024-09-18T08:09:51.470 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.15216", 2024-09-18T08:09:51.470 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "172.21.15.7:0/2596041177", 2024-09-18T08:09:51.470 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 125261 2024-09-18T08:09:51.470 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:09:51.471 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-09-18T08:09:51.471 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:09:51.471 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-09-18T08:09:51.471 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:51.451399+0000", 2024-09-18T08:09:51.471 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-18T08:09:51.471 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:09:51.471 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:09:51.471 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-09-18T08:09:51.471 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:51.451399+0000", 2024-09-18T08:09:51.471 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-18T08:09:51.471 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:09:51.472 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:09:51.472 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-09-18T08:09:51.472 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:51.451400+0000", 2024-09-18T08:09:51.472 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.0219999999999999e-06 2024-09-18T08:09:51.472 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:09:51.472 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:09:51.472 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-09-18T08:09:51.472 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:51.451405+0000", 2024-09-18T08:09:51.472 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4.5349999999999998e-06 2024-09-18T08:09:51.472 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:09:51.473 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:09:51.473 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-09-18T08:09:51.473 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:51.451417+0000", 2024-09-18T08:09:51.473 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.2279000000000001e-05 2024-09-18T08:09:51.473 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:09:51.473 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:09:51.473 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-09-18T08:09:51.473 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:51.451424+0000", 2024-09-18T08:09:51.473 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 7.0990000000000001e-06 2024-09-18T08:09:51.474 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:09:51.474 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:09:51.474 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-18T08:09:51.474 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:51.456792+0000", 2024-09-18T08:09:51.474 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0053678170000000004 2024-09-18T08:09:51.474 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:09:51.475 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:09:51.475 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-09-18T08:09:51.475 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:51.456947+0000", 2024-09-18T08:09:51.475 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00015525000000000001 2024-09-18T08:09:51.475 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:09:51.475 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:09:51.475 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "waiting for subops from 2,4", 2024-09-18T08:09:51.475 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:51.457083+0000", 2024-09-18T08:09:51.475 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00013590600000000001 2024-09-18T08:09:51.476 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-18T08:09:51.476 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-09-18T08:09:51.476 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-18T08:09:51.476 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:09:51.476 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:09:51.476 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_repop_reply(client.15216.0:125257 8.4 e214/199)", 2024-09-18T08:09:51.476 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-09-18T08:09:51.455931+0000", 2024-09-18T08:09:51.476 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 0.00110559, 2024-09-18T08:09:51.476 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0012843869999999999, 2024-09-18T08:09:51.476 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-09-18T08:09:51.477 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-09-18T08:09:51.477 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "queued for pg", 2024-09-18T08:09:51.477 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-09-18T08:09:51.477 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:09:51.477 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-09-18T08:09:51.477 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:51.455931+0000", 2024-09-18T08:09:51.477 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-18T08:09:51.478 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:09:51.478 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:09:51.478 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-09-18T08:09:51.478 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:51.455931+0000", 2024-09-18T08:09:51.478 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-18T08:09:51.478 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:09:51.478 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:09:51.478 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-09-18T08:09:51.479 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:51.455931+0000", 2024-09-18T08:09:51.479 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 8.8100000000000001e-07 2024-09-18T08:09:51.479 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:09:51.479 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:09:51.479 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-09-18T08:09:51.479 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:51.455936+0000", 2024-09-18T08:09:51.479 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4.8439999999999997e-06 2024-09-18T08:09:51.479 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:09:51.479 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:09:51.480 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-09-18T08:09:51.480 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:51.455949+0000", 2024-09-18T08:09:51.480 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.3055e-05 2024-09-18T08:09:51.480 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:09:51.480 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:09:51.480 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-09-18T08:09:51.481 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:51.455956+0000", 2024-09-18T08:09:51.481 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 6.9399999999999996e-06 2024-09-18T08:09:51.481 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-18T08:09:51.481 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-09-18T08:09:51.481 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-18T08:09:51.481 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:09:51.481 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:09:51.481 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_repop(client.15216.0:125263 8.0 e214/165 8:0cc1c0f9:::benchmark_data_smithi007_101583_object24511:head v 214'7764, mlcod=214'7762)", 2024-09-18T08:09:51.482 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-09-18T08:09:51.456678+0000", 2024-09-18T08:09:51.482 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 0.00035833700000000002, 2024-09-18T08:09:51.482 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00058152099999999999, 2024-09-18T08:09:51.482 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-09-18T08:09:51.483 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-09-18T08:09:51.483 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "started", 2024-09-18T08:09:51.483 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-09-18T08:09:51.483 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:09:51.483 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-09-18T08:09:51.483 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:51.456678+0000", 2024-09-18T08:09:51.483 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-18T08:09:51.483 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:09:51.484 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:09:51.484 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-09-18T08:09:51.484 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:51.456678+0000", 2024-09-18T08:09:51.484 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-18T08:09:51.484 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:09:51.484 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:09:51.484 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-09-18T08:09:51.484 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:51.456679+0000", 2024-09-18T08:09:51.484 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 7.4000000000000001e-07 2024-09-18T08:09:51.485 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:09:51.485 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:09:51.485 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-09-18T08:09:51.485 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:51.456685+0000", 2024-09-18T08:09:51.485 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 6.2949999999999999e-06 2024-09-18T08:09:51.485 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:09:51.485 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:09:51.485 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-09-18T08:09:51.486 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:51.456700+0000", 2024-09-18T08:09:51.486 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.4851999999999999e-05 2024-09-18T08:09:51.486 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:09:51.486 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:09:51.486 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-09-18T08:09:51.486 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:51.456707+0000", 2024-09-18T08:09:51.486 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 7.571e-06 2024-09-18T08:09:51.486 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:09:51.487 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:09:51.487 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-18T08:09:51.487 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:51.456776+0000", 2024-09-18T08:09:51.487 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 6.9153999999999996e-05 2024-09-18T08:09:51.487 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:09:51.487 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:09:51.487 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-09-18T08:09:51.488 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:51.456798+0000", 2024-09-18T08:09:51.488 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 2.1914000000000001e-05 2024-09-18T08:09:51.488 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-18T08:09:51.488 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-09-18T08:09:51.488 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-18T08:09:51.488 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-18T08:09:51.488 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2024-09-18T08:09:51.488 INFO:teuthology.orchestra.run.smithi050.stdout: "num_ops": 4 2024-09-18T08:09:51.489 INFO:teuthology.orchestra.run.smithi050.stdout:} 2024-09-18T08:09:51.489 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3 2024-09-18T08:09:51.530 INFO:teuthology.orchestra.run.smithi007.stdout:osd.0: {} 2024-09-18T08:09:51.530 INFO:teuthology.orchestra.run.smithi007.stdout:osd.1: {} 2024-09-18T08:09:51.530 INFO:teuthology.orchestra.run.smithi007.stdout:osd.2: {} 2024-09-18T08:09:51.530 INFO:teuthology.orchestra.run.smithi007.stdout:osd.3: {} 2024-09-18T08:09:51.530 INFO:teuthology.orchestra.run.smithi007.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-18T08:09:51.530 INFO:teuthology.orchestra.run.smithi007.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-18T08:09:51.530 INFO:teuthology.orchestra.run.smithi007.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-18T08:09:51.530 INFO:teuthology.orchestra.run.smithi007.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-18T08:09:51.542 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:09:51.543 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:09:51.544 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3' 2024-09-18T08:09:51.549 INFO:teuthology.orchestra.run.smithi007.stdout:osd.4: {} 2024-09-18T08:09:51.549 INFO:teuthology.orchestra.run.smithi007.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-18T08:09:51.573 INFO:teuthology.orchestra.run.smithi007.stderr:noscrub is set 2024-09-18T08:09:51.587 INFO:teuthology.orchestra.run.smithi007.stdout:osd.5: {} 2024-09-18T08:09:51.587 INFO:teuthology.orchestra.run.smithi007.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-18T08:09:51.637 INFO:teuthology.orchestra.run.smithi007.stdout:osd.6: {} 2024-09-18T08:09:51.637 INFO:teuthology.orchestra.run.smithi007.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-18T08:09:51.645 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2 2024-09-18T08:09:51.656 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:09:51 smithi050 ceph-mon[32371]: 5.10 scrub starts 2024-09-18T08:09:51.656 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:09:51 smithi050 ceph-mon[32371]: 5.10 scrub ok 2024-09-18T08:09:51.656 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:09:51 smithi050 ceph-mon[32371]: purged_snaps scrub starts 2024-09-18T08:09:51.656 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:09:51 smithi050 ceph-mon[32371]: purged_snaps scrub ok 2024-09-18T08:09:51.656 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:09:51 smithi050 ceph-mon[32371]: from='client.? 172.21.15.7:0/4216175934' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-18T08:09:51.656 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:09:51 smithi050 ceph-mon[32371]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-18T08:09:51.656 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:09:51 smithi050 ceph-mon[32371]: from='client.? 172.21.15.7:0/190460052' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-18T08:09:51.682 INFO:teuthology.orchestra.run.smithi007.stdout:osd.7: {} 2024-09-18T08:09:51.682 INFO:teuthology.orchestra.run.smithi007.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-18T08:09:51.683 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:09:51.684 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:09:51.684 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2' 2024-09-18T08:09:51.785 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2 2024-09-18T08:09:51.821 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:09:51.822 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:09:51.822 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2' 2024-09-18T08:09:51.831 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:09:51 smithi007 ceph-mon[29459]: 5.10 scrub starts 2024-09-18T08:09:51.831 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:09:51 smithi007 ceph-mon[29459]: 5.10 scrub ok 2024-09-18T08:09:51.831 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:09:51 smithi007 ceph-mon[29459]: purged_snaps scrub starts 2024-09-18T08:09:51.831 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:09:51 smithi007 ceph-mon[29459]: purged_snaps scrub ok 2024-09-18T08:09:51.831 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:09:51 smithi007 ceph-mon[29459]: from='client.? 172.21.15.7:0/4216175934' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-18T08:09:51.831 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:09:51 smithi007 ceph-mon[29459]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-18T08:09:51.831 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:09:51 smithi007 ceph-mon[29459]: from='client.? 172.21.15.7:0/190460052' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-18T08:09:51.832 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:09:51 smithi007 ceph-mon[39249]: 5.10 scrub starts 2024-09-18T08:09:51.832 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:09:51 smithi007 ceph-mon[39249]: 5.10 scrub ok 2024-09-18T08:09:51.832 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:09:51 smithi007 ceph-mon[39249]: purged_snaps scrub starts 2024-09-18T08:09:51.832 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:09:51 smithi007 ceph-mon[39249]: purged_snaps scrub ok 2024-09-18T08:09:51.833 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:09:51 smithi007 ceph-mon[39249]: from='client.? 172.21.15.7:0/4216175934' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-18T08:09:51.833 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:09:51 smithi007 ceph-mon[39249]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-18T08:09:51.833 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:09:51 smithi007 ceph-mon[39249]: from='client.? 172.21.15.7:0/190460052' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-18T08:09:51.924 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6 2024-09-18T08:09:51.958 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:09:51.959 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:09:51.959 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6' 2024-09-18T08:09:51.964 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-09-18T08:09:51.964 INFO:teuthology.orchestra.run.smithi007.stdout:{"fsid":"779349ca-7593-11ef-bceb-c7b262605968","health":{"status":"HEALTH_WARN","checks":{"OSDMAP_FLAGS":{"severity":"HEALTH_WARN","summary":{"message":"noscrub flag(s) set","count":7},"muted":false}},"mutes":[]},"election_epoch":14,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":671,"monmap":{"epoch":3,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":215,"num_osds":8,"num_up_osds":8,"osd_up_since":1726646556,"num_in_osds":8,"osd_in_since":1726646897,"num_remapped_pgs":1},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":157},{"state_name":"active+recovering+undersized+remapped","count":1}],"num_pgs":158,"num_pools":7,"num_objects":79683,"data_bytes":5209594419,"bytes_used":18640904192,"bytes_avail":749319380992,"bytes_total":767960285184,"misplaced_objects":1703,"misplaced_total":239049,"misplaced_ratio":0.0071240624307150413,"recovering_objects_per_sec":50,"recovering_bytes_per_sec":3327375,"recovering_keys_per_sec":0,"num_objects_recovered":508,"num_bytes_recovered":33292288,"num_keys_recovered":0,"read_bytes_sec":1023,"write_bytes_sec":0,"read_op_per_sec":0,"write_op_per_sec":1118},"fsmap":{"epoch":1,"btime":"2024-09-18T07:57:09:295514+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":1,"modules":["cephadm","dashboard","iostat","nfs","prometheus","restful"],"services":{"dashboard":"https://172.21.15.7:8443/","prometheus":"http://172.21.15.7:9283/"}},"servicemap":{"epoch":14,"modified":"2024-09-18T08:04:19.081938+0000","services":{"rgw":{"daemons":{"summary":"","24356":{"start_epoch":12,"start_stamp":"2024-09-18T08:02:54.139075+0000","gid":24356,"addr":"172.21.15.7:0/219168083","metadata":{"arch":"x86_64","ceph_release":"squid","ceph_version":"ceph version 19.3.0-4983-ge7fb7b56 (e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e) squid (dev)","ceph_version_short":"19.3.0-4983-ge7fb7b56","container_hostname":"smithi007","container_image":"quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:1f5b5a3734f665ef333266f39143ac3ef1eb1951b21258f2307e10332e2d7ac9","cpu":"Intel(R) Xeon(R) CPU E5-1620 v3 @ 3.50GHz","distro":"centos","distro_description":"CentOS Stream 9","distro_version":"9","frontend_config#0":"beast port=80","frontend_type#0":"beast","hostname":"smithi007","id":"foo.a","kernel_description":"#1 SMP PREEMPT_DYNAMIC Thu Sep 12 15:49:37 UTC 2024","kernel_version":"5.14.0-508.el9.x86_64","mem_swap_kb":"0","mem_total_kb":"32493484","num_handles":"1","os":"Linux","pid":"2","realm_id":"","realm_name":"","zone_id":"412e9c99-1835-4c2d-b8ee-228c30c92752","zone_name":"default","zonegroup_id":"f0acf781-d0ae-4f13-8505-d87ee1ac7608","zonegroup_name":"default"},"task_status":{}}}}}},"progress_events":{}} 2024-09-18T08:09:52.060 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5 2024-09-18T08:09:52.089 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:09:52.089 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:09:52.090 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5' 2024-09-18T08:09:52.190 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5 2024-09-18T08:09:52.222 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:09:52.223 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:09:52.223 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5' 2024-09-18T08:09:52.303 DEBUG:teuthology.orchestra.run.smithi050:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid 779349ca-7593-11ef-bceb-c7b262605968 -- ceph daemon osd.5 dump_blocked_ops 2024-09-18T08:09:52.325 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7 2024-09-18T08:09:52.408 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:09:52.409 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:09:52.409 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7' 2024-09-18T08:09:52.510 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6 2024-09-18T08:09:52.550 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:09:52.550 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:09:52.551 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6' 2024-09-18T08:09:52.652 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5 2024-09-18T08:09:52.701 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:09:52.701 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:09:52.702 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5' 2024-09-18T08:09:52.730 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:09:52 smithi007 ceph-mon[29459]: pgmap v341: 158 pgs: 1 active+recovering+undersized+remapped, 157 active+clean; 4.9 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.12k op/s; 1703/239049 objects misplaced (0.712%); 3.2 MiB/s, 50 objects/s recovering 2024-09-18T08:09:52.730 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:09:52 smithi007 ceph-mon[29459]: 5.0 scrub starts 2024-09-18T08:09:52.730 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:09:52 smithi007 ceph-mon[29459]: 5.0 scrub ok 2024-09-18T08:09:52.730 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:09:52 smithi007 ceph-mon[29459]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-18T08:09:52.731 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:09:52 smithi007 ceph-mon[29459]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-18T08:09:52.731 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:09:52 smithi007 ceph-mon[29459]: osdmap e215: 8 total, 8 up, 8 in 2024-09-18T08:09:52.731 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:09:52 smithi007 ceph-mon[29459]: from='client.? 172.21.15.7:0/1362681327' entity='client.admin' cmd={"prefix": "status", "format": "json"} : dispatch 2024-09-18T08:09:52.731 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:09:52 smithi007 ceph-mon[39249]: pgmap v341: 158 pgs: 1 active+recovering+undersized+remapped, 157 active+clean; 4.9 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.12k op/s; 1703/239049 objects misplaced (0.712%); 3.2 MiB/s, 50 objects/s recovering 2024-09-18T08:09:52.731 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:09:52 smithi007 ceph-mon[39249]: 5.0 scrub starts 2024-09-18T08:09:52.731 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:09:52 smithi007 ceph-mon[39249]: 5.0 scrub ok 2024-09-18T08:09:52.731 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:09:52 smithi007 ceph-mon[39249]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-18T08:09:52.731 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:09:52 smithi007 ceph-mon[39249]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-18T08:09:52.731 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:09:52 smithi007 ceph-mon[39249]: osdmap e215: 8 total, 8 up, 8 in 2024-09-18T08:09:52.731 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:09:52 smithi007 ceph-mon[39249]: from='client.? 172.21.15.7:0/1362681327' entity='client.admin' cmd={"prefix": "status", "format": "json"} : dispatch 2024-09-18T08:09:52.753 INFO:teuthology.orchestra.run.smithi050.stderr:Inferring config /var/lib/ceph/779349ca-7593-11ef-bceb-c7b262605968/mon.b/config 2024-09-18T08:09:52.803 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5 2024-09-18T08:09:52.838 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:09:52.839 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:09:52.839 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5' 2024-09-18T08:09:52.888 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:09:52 smithi050 ceph-mon[32371]: pgmap v341: 158 pgs: 1 active+recovering+undersized+remapped, 157 active+clean; 4.9 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.12k op/s; 1703/239049 objects misplaced (0.712%); 3.2 MiB/s, 50 objects/s recovering 2024-09-18T08:09:52.888 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:09:52 smithi050 ceph-mon[32371]: 5.0 scrub starts 2024-09-18T08:09:52.888 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:09:52 smithi050 ceph-mon[32371]: 5.0 scrub ok 2024-09-18T08:09:52.888 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:09:52 smithi050 ceph-mon[32371]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-18T08:09:52.889 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:09:52 smithi050 ceph-mon[32371]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-18T08:09:52.889 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:09:52 smithi050 ceph-mon[32371]: osdmap e215: 8 total, 8 up, 8 in 2024-09-18T08:09:52.889 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:09:52 smithi050 ceph-mon[32371]: from='client.? 172.21.15.7:0/1362681327' entity='client.admin' cmd={"prefix": "status", "format": "json"} : dispatch 2024-09-18T08:09:52.940 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1 2024-09-18T08:09:52.986 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:09:52.987 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:09:52.988 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1' 2024-09-18T08:09:53.088 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6 2024-09-18T08:09:53.133 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:09:53.134 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:09:53.134 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6' 2024-09-18T08:09:53.235 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6 2024-09-18T08:09:53.269 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:09:53.269 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:09:53.270 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6' 2024-09-18T08:09:53.370 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7 2024-09-18T08:09:53.412 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:09:53.413 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:09:53.413 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7' 2024-09-18T08:09:53.514 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3 2024-09-18T08:09:53.550 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:09:53.551 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:09:53.551 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3' 2024-09-18T08:09:53.652 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7 2024-09-18T08:09:53.683 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:09:53.684 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:09:53.684 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7' 2024-09-18T08:09:53.785 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0 2024-09-18T08:09:53.824 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:09:53.825 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:09:53.825 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0' 2024-09-18T08:09:53.844 INFO:tasks.cephadm.ceph_manager.ceph:making progress, resetting timeout 2024-09-18T08:09:53.844 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid 779349ca-7593-11ef-bceb-c7b262605968 -- ceph pg dump --format=json 2024-09-18T08:09:53.927 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7 2024-09-18T08:09:53.968 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:09:53.969 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:09:53.969 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7' 2024-09-18T08:09:54.070 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4 2024-09-18T08:09:54.100 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:09:54.101 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:09:54.101 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4' 2024-09-18T08:09:54.202 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6 2024-09-18T08:09:54.236 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:09:54.237 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:09:54.237 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6' 2024-09-18T08:09:54.278 INFO:teuthology.orchestra.run.smithi050.stdout:ERROR: (22) Invalid argument 2024-09-18T08:09:54.278 INFO:teuthology.orchestra.run.smithi050.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-18T08:09:54.318 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/779349ca-7593-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T08:09:54.338 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5 2024-09-18T08:09:54.372 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:09:54.373 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:09:54.373 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5' 2024-09-18T08:09:54.474 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6 2024-09-18T08:09:54.505 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:09:54.506 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:09:54.506 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6' 2024-09-18T08:09:54.607 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2 2024-09-18T08:09:54.669 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:09:54.670 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:09:54.670 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2' 2024-09-18T08:09:54.730 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:09:54 smithi007 ceph-mon[29459]: pgmap v343: 158 pgs: 1 active+recovering+undersized+remapped, 157 active+clean; 4.8 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 587 B/s rd, 0 B/s wr, 708 op/s; 1703/237048 objects misplaced (0.718%); 1.6 MiB/s, 25 objects/s recovering 2024-09-18T08:09:54.730 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:09:54 smithi007 ceph-mon[39249]: pgmap v343: 158 pgs: 1 active+recovering+undersized+remapped, 157 active+clean; 4.8 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 587 B/s rd, 0 B/s wr, 708 op/s; 1703/237048 objects misplaced (0.718%); 1.6 MiB/s, 25 objects/s recovering 2024-09-18T08:09:54.771 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5 2024-09-18T08:09:54.794 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:09:54.795 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:09:54.796 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5' 2024-09-18T08:09:54.897 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2 2024-09-18T08:09:54.905 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:09:54 smithi050 ceph-mon[32371]: pgmap v343: 158 pgs: 1 active+recovering+undersized+remapped, 157 active+clean; 4.8 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 587 B/s rd, 0 B/s wr, 708 op/s; 1703/237048 objects misplaced (0.718%); 1.6 MiB/s, 25 objects/s recovering 2024-09-18T08:09:54.938 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:09:54.939 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:09:54.939 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2' 2024-09-18T08:09:54.975 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid 779349ca-7593-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-18T08:09:55.040 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7 2024-09-18T08:09:55.070 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:09:55.071 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:09:55.071 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7' 2024-09-18T08:09:55.172 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1 2024-09-18T08:09:55.212 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:09:55.213 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:09:55.213 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1' 2024-09-18T08:09:55.307 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-09-18T08:09:55.309 INFO:teuthology.orchestra.run.smithi007.stderr:dumped all 2024-09-18T08:09:55.311 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid 779349ca-7593-11ef-bceb-c7b262605968 -- ceph osd set nodeep-scrub 2024-09-18T08:09:55.314 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4 2024-09-18T08:09:55.344 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:09:55.345 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:09:55.345 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4' 2024-09-18T08:09:55.446 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5 2024-09-18T08:09:55.455 DEBUG:teuthology.orchestra.run.smithi050:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid 779349ca-7593-11ef-bceb-c7b262605968 -- ceph daemon osd.5 dump_historic_ops 2024-09-18T08:09:55.481 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:09:55.482 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:09:55.482 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5' 2024-09-18T08:09:55.583 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2 2024-09-18T08:09:55.618 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:09:55 smithi007 ceph-mon[29459]: Health check failed: Degraded data redundancy: 1 pg undersized (PG_DEGRADED) 2024-09-18T08:09:55.647 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:09:55.647 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:09:55.648 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2' 2024-09-18T08:09:55.749 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5 2024-09-18T08:09:55.788 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:09:55.788 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:09:55.789 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5' 2024-09-18T08:09:55.890 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0 2024-09-18T08:09:55.907 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:09:55 smithi050 ceph-mon[32371]: Health check failed: Degraded data redundancy: 1 pg undersized (PG_DEGRADED) 2024-09-18T08:09:55.925 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:09:55.925 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:09:55.925 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0' 2024-09-18T08:09:55.926 INFO:teuthology.orchestra.run.smithi050.stderr:Inferring config /var/lib/ceph/779349ca-7593-11ef-bceb-c7b262605968/mon.b/config 2024-09-18T08:09:55.980 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:09:55 smithi007 ceph-mon[39249]: Health check failed: Degraded data redundancy: 1 pg undersized (PG_DEGRADED) 2024-09-18T08:09:56.027 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2 2024-09-18T08:09:56.067 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:09:56.067 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:09:56.068 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2' 2024-09-18T08:09:56.168 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2 2024-09-18T08:09:56.203 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:09:56.204 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:09:56.204 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2' 2024-09-18T08:09:56.305 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7 2024-09-18T08:09:56.339 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:09:56.340 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:09:56.340 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7' 2024-09-18T08:09:56.427 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/779349ca-7593-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T08:09:56.433 INFO:teuthology.orchestra.run.smithi007.stdout:{"pg_ready":true,"pg_map":{"version":344,"stamp":"2024-09-18T08:09:55.223542+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":4827585075,"num_objects":73854,"num_object_clones":0,"num_object_copies":221562,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1453,"num_objects_unfound":0,"num_objects_dirty":73854,"num_whiteouts":0,"num_read":903,"num_read_kb":655,"num_write":128281,"num_write_kb":6447982,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":23033,"num_bytes_recovered":1493638354,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":79439,"ondisk_log_size":79439,"up":474,"acting":473,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":481,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":17096924,"kb_used_data":15260424,"kb_used_omap":12,"kb_used_meta":1836147,"kb_avail":732864292,"statfs":{"total":767960285184,"available":750453035008,"internally_reserved":0,"allocated":15626674176,"data_stored":15623496617,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12750,"internal_metadata":1880215090},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,1,2,2,3],"upper_bound":32},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-742588416,"num_objects":-11331,"num_object_clones":0,"num_object_copies":-33993,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-468,"num_objects_unfound":0,"num_objects_dirty":-11331,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":11331,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":468,"num_bytes_recovered":30670848,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.005755"},"pg_stats":[{"pgid":"3.1f","version":"0'0","reported_seq":350,"reported_epoch":215,"state":"active+clean","last_fresh":"2024-09-18T08:09:51.569270+0000","last_change":"2024-09-18T08:08:52.697893+0000","last_active":"2024-09-18T08:09:51.569270+0000","last_peered":"2024-09-18T08:09:51.569270+0000","last_clean":"2024-09-18T08:09:51.569270+0000","last_became_active":"2024-09-18T08:08:52.697499+0000","last_became_peered":"2024-09-18T08:08:52.697499+0000","last_unstale":"2024-09-18T08:09:51.569270+0000","last_undegraded":"2024-09-18T08:09:51.569270+0000","last_fullsized":"2024-09-18T08:09:51.569270+0000","mapping_epoch":198,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":199,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:08:06.642662+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:46.860092+0000","last_clean_scrub_stamp":"2024-09-18T08:08:06.642662+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"69'9","reported_seq":88,"reported_epoch":215,"state":"active+clean","last_fresh":"2024-09-18T08:09:51.557673+0000","last_change":"2024-09-18T08:08:18.575611+0000","last_active":"2024-09-18T08:09:51.557673+0000","last_peered":"2024-09-18T08:09:51.557673+0000","last_clean":"2024-09-18T08:09:51.557673+0000","last_became_active":"2024-09-18T08:08:18.537856+0000","last_became_peered":"2024-09-18T08:08:18.537856+0000","last_unstale":"2024-09-18T08:09:51.557673+0000","last_undegraded":"2024-09-18T08:09:51.557673+0000","last_fullsized":"2024-09-18T08:09:51.557673+0000","mapping_epoch":181,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":182,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'9","last_scrub_stamp":"2024-09-18T08:05:08.696796+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_clean_scrub_stamp":"2024-09-18T08:05:08.696796+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":16,"num_read_kb":10,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":335,"reported_epoch":215,"state":"active+clean","last_fresh":"2024-09-18T08:09:51.590045+0000","last_change":"2024-09-18T08:07:18.737364+0000","last_active":"2024-09-18T08:09:51.590045+0000","last_peered":"2024-09-18T08:09:51.590045+0000","last_clean":"2024-09-18T08:09:51.590045+0000","last_became_active":"2024-09-18T08:02:51.907140+0000","last_became_peered":"2024-09-18T08:02:51.907140+0000","last_unstale":"2024-09-18T08:09:51.590045+0000","last_undegraded":"2024-09-18T08:09:51.590045+0000","last_fullsized":"2024-09-18T08:09:51.590045+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:07:18.737122+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:50.886830+0000","last_clean_scrub_stamp":"2024-09-18T08:07:18.737122+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:10:14.742960+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":78,"reported_epoch":215,"state":"active+clean","last_fresh":"2024-09-18T08:09:51.558377+0000","last_change":"2024-09-18T08:08:18.545192+0000","last_active":"2024-09-18T08:09:51.558377+0000","last_peered":"2024-09-18T08:09:51.558377+0000","last_clean":"2024-09-18T08:09:51.558377+0000","last_became_active":"2024-09-18T08:08:18.540705+0000","last_became_peered":"2024-09-18T08:08:18.540705+0000","last_unstale":"2024-09-18T08:09:51.558377+0000","last_undegraded":"2024-09-18T08:09:51.558377+0000","last_fullsized":"2024-09-18T08:09:51.558377+0000","mapping_epoch":181,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":182,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:07:08.451865+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_clean_scrub_stamp":"2024-09-18T08:07:08.451865+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":338,"reported_epoch":215,"state":"active+clean","last_fresh":"2024-09-18T08:09:51.580794+0000","last_change":"2024-09-18T08:07:13.244647+0000","last_active":"2024-09-18T08:09:51.580794+0000","last_peered":"2024-09-18T08:09:51.580794+0000","last_clean":"2024-09-18T08:09:51.580794+0000","last_became_active":"2024-09-18T08:06:24.128511+0000","last_became_peered":"2024-09-18T08:06:24.128511+0000","last_unstale":"2024-09-18T08:09:51.580794+0000","last_undegraded":"2024-09-18T08:09:51.580794+0000","last_fullsized":"2024-09-18T08:09:51.580794+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:07:13.244555+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_clean_scrub_stamp":"2024-09-18T08:07:13.244555+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":351,"reported_epoch":215,"state":"active+clean","last_fresh":"2024-09-18T08:09:51.582303+0000","last_change":"2024-09-18T08:07:10.257236+0000","last_active":"2024-09-18T08:09:51.582303+0000","last_peered":"2024-09-18T08:09:51.582303+0000","last_clean":"2024-09-18T08:09:51.582303+0000","last_became_active":"2024-09-18T08:06:24.126194+0000","last_became_peered":"2024-09-18T08:06:24.126194+0000","last_unstale":"2024-09-18T08:09:51.582303+0000","last_undegraded":"2024-09-18T08:09:51.582303+0000","last_fullsized":"2024-09-18T08:09:51.582303+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:07:10.257170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:46.860092+0000","last_clean_scrub_stamp":"2024-09-18T08:07:10.257170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":37,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"69'15","reported_seq":349,"reported_epoch":215,"state":"active+clean","last_fresh":"2024-09-18T08:09:51.581180+0000","last_change":"2024-09-18T08:09:32.575836+0000","last_active":"2024-09-18T08:09:51.581180+0000","last_peered":"2024-09-18T08:09:51.581180+0000","last_clean":"2024-09-18T08:09:51.581180+0000","last_became_active":"2024-09-18T08:02:49.886106+0000","last_became_peered":"2024-09-18T08:02:49.886106+0000","last_unstale":"2024-09-18T08:09:51.581180+0000","last_undegraded":"2024-09-18T08:09:51.581180+0000","last_fullsized":"2024-09-18T08:09:51.581180+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'15","last_scrub_stamp":"2024-09-18T08:09:32.575741+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_clean_scrub_stamp":"2024-09-18T08:09:32.575741+0000","objects_scrubbed":7,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:10:59.141685+0000","scrub_duration":40,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":78,"reported_epoch":215,"state":"active+clean","last_fresh":"2024-09-18T08:09:51.557707+0000","last_change":"2024-09-18T08:08:18.543316+0000","last_active":"2024-09-18T08:09:51.557707+0000","last_peered":"2024-09-18T08:09:51.557707+0000","last_clean":"2024-09-18T08:09:51.557707+0000","last_became_active":"2024-09-18T08:08:18.538645+0000","last_became_peered":"2024-09-18T08:08:18.538645+0000","last_unstale":"2024-09-18T08:09:51.557707+0000","last_undegraded":"2024-09-18T08:09:51.557707+0000","last_fullsized":"2024-09-18T08:09:51.557707+0000","mapping_epoch":181,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":182,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:08:09.142188+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:50.886830+0000","last_clean_scrub_stamp":"2024-09-18T08:08:09.142188+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":363,"reported_epoch":215,"state":"active+clean","last_fresh":"2024-09-18T08:09:51.570329+0000","last_change":"2024-09-18T08:08:18.548588+0000","last_active":"2024-09-18T08:09:51.570329+0000","last_peered":"2024-09-18T08:09:51.570329+0000","last_clean":"2024-09-18T08:09:51.570329+0000","last_became_active":"2024-09-18T08:08:18.548203+0000","last_became_peered":"2024-09-18T08:08:18.548203+0000","last_unstale":"2024-09-18T08:09:51.570329+0000","last_undegraded":"2024-09-18T08:09:51.570329+0000","last_fullsized":"2024-09-18T08:09:51.570329+0000","mapping_epoch":181,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":182,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:06:14.027077+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:46.860092+0000","last_clean_scrub_stamp":"2024-09-18T08:06:14.027077+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"69'9","reported_seq":88,"reported_epoch":215,"state":"active+clean","last_fresh":"2024-09-18T08:09:51.558588+0000","last_change":"2024-09-18T08:08:18.631597+0000","last_active":"2024-09-18T08:09:51.558588+0000","last_peered":"2024-09-18T08:09:51.558588+0000","last_clean":"2024-09-18T08:09:51.558588+0000","last_became_active":"2024-09-18T08:08:18.550960+0000","last_became_peered":"2024-09-18T08:08:18.550960+0000","last_unstale":"2024-09-18T08:09:51.558588+0000","last_undegraded":"2024-09-18T08:09:51.558588+0000","last_fullsized":"2024-09-18T08:09:51.558588+0000","mapping_epoch":181,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":182,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_clean_scrub_stamp":"2024-09-18T08:02:48.870718+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":16,"num_read_kb":10,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":327,"reported_epoch":215,"state":"active+clean","last_fresh":"2024-09-18T08:09:51.568496+0000","last_change":"2024-09-18T08:05:33.935844+0000","last_active":"2024-09-18T08:09:51.568496+0000","last_peered":"2024-09-18T08:09:51.568496+0000","last_clean":"2024-09-18T08:09:51.568496+0000","last_became_active":"2024-09-18T08:02:51.907296+0000","last_became_peered":"2024-09-18T08:02:51.907296+0000","last_unstale":"2024-09-18T08:09:51.568496+0000","last_undegraded":"2024-09-18T08:09:51.568496+0000","last_fullsized":"2024-09-18T08:09:51.568496+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:05:33.935798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:50.886830+0000","last_clean_scrub_stamp":"2024-09-18T08:05:33.935798+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":324,"reported_epoch":215,"state":"active+clean","last_fresh":"2024-09-18T08:09:51.563842+0000","last_change":"2024-09-18T08:08:14.646073+0000","last_active":"2024-09-18T08:09:51.563842+0000","last_peered":"2024-09-18T08:09:51.563842+0000","last_clean":"2024-09-18T08:09:51.563842+0000","last_became_active":"2024-09-18T08:02:53.913519+0000","last_became_peered":"2024-09-18T08:02:53.913519+0000","last_unstale":"2024-09-18T08:09:51.563842+0000","last_undegraded":"2024-09-18T08:09:51.563842+0000","last_fullsized":"2024-09-18T08:09:51.563842+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:08:14.645993+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_clean_scrub_stamp":"2024-09-18T08:08:14.645993+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":341,"reported_epoch":215,"state":"active+clean","last_fresh":"2024-09-18T08:09:51.571412+0000","last_change":"2024-09-18T08:08:18.624260+0000","last_active":"2024-09-18T08:09:51.571412+0000","last_peered":"2024-09-18T08:09:51.571412+0000","last_clean":"2024-09-18T08:09:51.571412+0000","last_became_active":"2024-09-18T08:08:18.623757+0000","last_became_peered":"2024-09-18T08:08:18.623757+0000","last_unstale":"2024-09-18T08:09:51.571412+0000","last_undegraded":"2024-09-18T08:09:51.571412+0000","last_fullsized":"2024-09-18T08:09:51.571412+0000","mapping_epoch":181,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":182,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:04:32.849119+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:46.860092+0000","last_clean_scrub_stamp":"2024-09-18T08:04:32.849119+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1b","version":"69'5","reported_seq":84,"reported_epoch":215,"state":"active+clean","last_fresh":"2024-09-18T08:09:51.558337+0000","last_change":"2024-09-18T08:08:18.651427+0000","last_active":"2024-09-18T08:09:51.558337+0000","last_peered":"2024-09-18T08:09:51.558337+0000","last_clean":"2024-09-18T08:09:51.558337+0000","last_became_active":"2024-09-18T08:08:18.567126+0000","last_became_peered":"2024-09-18T08:08:18.567126+0000","last_unstale":"2024-09-18T08:09:51.558337+0000","last_undegraded":"2024-09-18T08:09:51.558337+0000","last_fullsized":"2024-09-18T08:09:51.558337+0000","mapping_epoch":181,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":182,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'5","last_scrub_stamp":"2024-09-18T08:06:09.465979+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_clean_scrub_stamp":"2024-09-18T08:06:09.465979+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":10,"num_read_kb":6,"num_write":6,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":357,"reported_epoch":215,"state":"active+clean","last_fresh":"2024-09-18T08:09:51.569846+0000","last_change":"2024-09-18T08:09:12.674889+0000","last_active":"2024-09-18T08:09:51.569846+0000","last_peered":"2024-09-18T08:09:51.569846+0000","last_clean":"2024-09-18T08:09:51.569846+0000","last_became_active":"2024-09-18T08:08:18.524932+0000","last_became_peered":"2024-09-18T08:08:18.524932+0000","last_unstale":"2024-09-18T08:09:51.569846+0000","last_undegraded":"2024-09-18T08:09:51.569846+0000","last_fullsized":"2024-09-18T08:09:51.569846+0000","mapping_epoch":181,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":182,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:09:12.674843+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:50.886830+0000","last_clean_scrub_stamp":"2024-09-18T08:09:12.674843+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:10:16.027634+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":324,"reported_epoch":215,"state":"active+clean","last_fresh":"2024-09-18T08:09:51.568750+0000","last_change":"2024-09-18T08:06:44.518882+0000","last_active":"2024-09-18T08:09:51.568750+0000","last_peered":"2024-09-18T08:09:51.568750+0000","last_clean":"2024-09-18T08:09:51.568750+0000","last_became_active":"2024-09-18T08:02:53.911648+0000","last_became_peered":"2024-09-18T08:02:53.911648+0000","last_unstale":"2024-09-18T08:09:51.568750+0000","last_undegraded":"2024-09-18T08:09:51.568750+0000","last_fullsized":"2024-09-18T08:09:51.568750+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:06:44.518835+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_clean_scrub_stamp":"2024-09-18T08:06:44.518835+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":78,"reported_epoch":215,"state":"active+clean","last_fresh":"2024-09-18T08:09:51.558770+0000","last_change":"2024-09-18T08:08:18.544203+0000","last_active":"2024-09-18T08:09:51.558770+0000","last_peered":"2024-09-18T08:09:51.558770+0000","last_clean":"2024-09-18T08:09:51.558770+0000","last_became_active":"2024-09-18T08:08:18.539604+0000","last_became_peered":"2024-09-18T08:08:18.539604+0000","last_unstale":"2024-09-18T08:09:51.558770+0000","last_undegraded":"2024-09-18T08:09:51.558770+0000","last_fullsized":"2024-09-18T08:09:51.558770+0000","mapping_epoch":181,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":182,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:05:40.477965+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_clean_scrub_stamp":"2024-09-18T08:05:40.477965+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":349,"reported_epoch":215,"state":"active+clean","last_fresh":"2024-09-18T08:09:51.564957+0000","last_change":"2024-09-18T08:08:18.524124+0000","last_active":"2024-09-18T08:09:51.564957+0000","last_peered":"2024-09-18T08:09:51.564957+0000","last_clean":"2024-09-18T08:09:51.564957+0000","last_became_active":"2024-09-18T08:08:18.523611+0000","last_became_peered":"2024-09-18T08:08:18.523611+0000","last_unstale":"2024-09-18T08:09:51.564957+0000","last_undegraded":"2024-09-18T08:09:51.564957+0000","last_fullsized":"2024-09-18T08:09:51.564957+0000","mapping_epoch":181,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":182,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:05:27.476144+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:46.860092+0000","last_clean_scrub_stamp":"2024-09-18T08:05:27.476144+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:10:23.059202+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1c","version":"69'15","reported_seq":60,"reported_epoch":215,"state":"active+clean","last_fresh":"2024-09-18T08:09:51.579799+0000","last_change":"2024-09-18T08:08:52.708963+0000","last_active":"2024-09-18T08:09:51.579799+0000","last_peered":"2024-09-18T08:09:51.579799+0000","last_clean":"2024-09-18T08:09:51.579799+0000","last_became_active":"2024-09-18T08:08:52.690788+0000","last_became_peered":"2024-09-18T08:08:52.690788+0000","last_unstale":"2024-09-18T08:09:51.579799+0000","last_undegraded":"2024-09-18T08:09:51.579799+0000","last_fullsized":"2024-09-18T08:09:51.579799+0000","mapping_epoch":198,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":199,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'15","last_scrub_stamp":"2024-09-18T08:07:39.530340+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_clean_scrub_stamp":"2024-09-18T08:07:39.530340+0000","objects_scrubbed":7,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":357,"reported_epoch":215,"state":"active+clean","last_fresh":"2024-09-18T08:09:51.589727+0000","last_change":"2024-09-18T08:09:11.693711+0000","last_active":"2024-09-18T08:09:51.589727+0000","last_peered":"2024-09-18T08:09:51.589727+0000","last_clean":"2024-09-18T08:09:51.589727+0000","last_became_active":"2024-09-18T08:08:18.538505+0000","last_became_peered":"2024-09-18T08:08:18.538505+0000","last_unstale":"2024-09-18T08:09:51.589727+0000","last_undegraded":"2024-09-18T08:09:51.589727+0000","last_fullsized":"2024-09-18T08:09:51.589727+0000","mapping_epoch":181,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":182,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:09:11.693643+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:50.886830+0000","last_clean_scrub_stamp":"2024-09-18T08:09:11.693643+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:10:34.429057+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":339,"reported_epoch":215,"state":"active+clean","last_fresh":"2024-09-18T08:09:51.583095+0000","last_change":"2024-09-18T08:09:36.592348+0000","last_active":"2024-09-18T08:09:51.583095+0000","last_peered":"2024-09-18T08:09:51.583095+0000","last_clean":"2024-09-18T08:09:51.583095+0000","last_became_active":"2024-09-18T08:06:24.128659+0000","last_became_peered":"2024-09-18T08:06:24.128659+0000","last_unstale":"2024-09-18T08:09:51.583095+0000","last_undegraded":"2024-09-18T08:09:51.583095+0000","last_fullsized":"2024-09-18T08:09:51.583095+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:09:36.592281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_clean_scrub_stamp":"2024-09-18T08:09:36.592281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:10:46.573780+0000","scrub_duration":16,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":78,"reported_epoch":215,"state":"active+clean","last_fresh":"2024-09-18T08:09:51.558224+0000","last_change":"2024-09-18T08:08:18.536559+0000","last_active":"2024-09-18T08:09:51.558224+0000","last_peered":"2024-09-18T08:09:51.558224+0000","last_clean":"2024-09-18T08:09:51.558224+0000","last_became_active":"2024-09-18T08:08:18.536084+0000","last_became_peered":"2024-09-18T08:08:18.536084+0000","last_unstale":"2024-09-18T08:09:51.558224+0000","last_undegraded":"2024-09-18T08:09:51.558224+0000","last_fullsized":"2024-09-18T08:09:51.558224+0000","mapping_epoch":181,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":182,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:07:13.543722+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:46.860092+0000","last_clean_scrub_stamp":"2024-09-18T08:07:13.543722+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"69'12","reported_seq":393,"reported_epoch":215,"state":"active+clean","last_fresh":"2024-09-18T08:09:51.582000+0000","last_change":"2024-09-18T08:08:18.583915+0000","last_active":"2024-09-18T08:09:51.582000+0000","last_peered":"2024-09-18T08:09:51.582000+0000","last_clean":"2024-09-18T08:09:51.582000+0000","last_became_active":"2024-09-18T08:08:18.543675+0000","last_became_peered":"2024-09-18T08:08:18.543675+0000","last_unstale":"2024-09-18T08:09:51.582000+0000","last_undegraded":"2024-09-18T08:09:51.582000+0000","last_fullsized":"2024-09-18T08:09:51.582000+0000","mapping_epoch":181,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":182,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'12","last_scrub_stamp":"2024-09-18T08:07:20.251529+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_clean_scrub_stamp":"2024-09-18T08:07:20.251529+0000","objects_scrubbed":5,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":25,"num_read_kb":16,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":78,"reported_epoch":215,"state":"active+clean","last_fresh":"2024-09-18T08:09:51.558745+0000","last_change":"2024-09-18T08:08:18.532482+0000","last_active":"2024-09-18T08:09:51.558745+0000","last_peered":"2024-09-18T08:09:51.558745+0000","last_clean":"2024-09-18T08:09:51.558745+0000","last_became_active":"2024-09-18T08:08:18.531438+0000","last_became_peered":"2024-09-18T08:08:18.531438+0000","last_unstale":"2024-09-18T08:09:51.558745+0000","last_undegraded":"2024-09-18T08:09:51.558745+0000","last_fullsized":"2024-09-18T08:09:51.558745+0000","mapping_epoch":181,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":182,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:06:12.434179+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:50.886830+0000","last_clean_scrub_stamp":"2024-09-18T08:06:12.434179+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1c","version":"69'1","reported_seq":167,"reported_epoch":215,"state":"active+clean","last_fresh":"2024-09-18T08:09:51.569684+0000","last_change":"2024-09-18T08:06:46.142143+0000","last_active":"2024-09-18T08:09:51.569684+0000","last_peered":"2024-09-18T08:09:51.569684+0000","last_clean":"2024-09-18T08:09:51.569684+0000","last_became_active":"2024-09-18T08:06:46.141661+0000","last_became_peered":"2024-09-18T08:06:46.141661+0000","last_unstale":"2024-09-18T08:09:51.569684+0000","last_undegraded":"2024-09-18T08:09:51.569684+0000","last_fullsized":"2024-09-18T08:09:51.569684+0000","mapping_epoch":143,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":144,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'1","last_scrub_stamp":"2024-09-18T08:05:36.776369+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_clean_scrub_stamp":"2024-09-18T08:05:36.776369+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":403,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,2],"acting":[5,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"62'1","reported_seq":366,"reported_epoch":215,"state":"active+clean","last_fresh":"2024-09-18T08:09:51.590185+0000","last_change":"2024-09-18T08:09:29.633875+0000","last_active":"2024-09-18T08:09:51.590185+0000","last_peered":"2024-09-18T08:09:51.590185+0000","last_clean":"2024-09-18T08:09:51.590185+0000","last_became_active":"2024-09-18T08:08:18.559125+0000","last_became_peered":"2024-09-18T08:08:18.559125+0000","last_unstale":"2024-09-18T08:09:51.590185+0000","last_undegraded":"2024-09-18T08:09:51.590185+0000","last_fullsized":"2024-09-18T08:09:51.590185+0000","mapping_epoch":181,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":182,"parent":"0.0","parent_split_bits":0,"last_scrub":"62'1","last_scrub_stamp":"2024-09-18T08:09:29.633520+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:46.860092+0000","last_clean_scrub_stamp":"2024-09-18T08:09:29.633520+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:10:38.890672+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":92,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1e","version":"69'10","reported_seq":331,"reported_epoch":215,"state":"active+clean","last_fresh":"2024-09-18T08:09:51.573640+0000","last_change":"2024-09-18T08:02:49.892755+0000","last_active":"2024-09-18T08:09:51.573640+0000","last_peered":"2024-09-18T08:09:51.573640+0000","last_clean":"2024-09-18T08:09:51.573640+0000","last_became_active":"2024-09-18T08:02:49.892024+0000","last_became_peered":"2024-09-18T08:02:49.892024+0000","last_unstale":"2024-09-18T08:09:51.573640+0000","last_undegraded":"2024-09-18T08:09:51.573640+0000","last_fullsized":"2024-09-18T08:09:51.573640+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_clean_scrub_stamp":"2024-09-18T08:02:48.870718+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:10:17.106246+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1f","version":"182'13","reported_seq":102,"reported_epoch":215,"state":"active+clean","last_fresh":"2024-09-18T08:09:51.559018+0000","last_change":"2024-09-18T08:08:18.627670+0000","last_active":"2024-09-18T08:09:51.559018+0000","last_peered":"2024-09-18T08:09:51.559018+0000","last_clean":"2024-09-18T08:09:51.559018+0000","last_became_active":"2024-09-18T08:08:18.543095+0000","last_became_peered":"2024-09-18T08:08:18.543095+0000","last_unstale":"2024-09-18T08:09:51.559018+0000","last_undegraded":"2024-09-18T08:09:51.559018+0000","last_fullsized":"2024-09-18T08:09:51.559018+0000","mapping_epoch":181,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":182,"parent":"0.0","parent_split_bits":0,"last_scrub":"173'12","last_scrub_stamp":"2024-09-18T08:08:07.553075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:50.886830+0000","last_clean_scrub_stamp":"2024-09-18T08:08:07.553075+0000","objects_scrubbed":0,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,7],"acting":[4,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.3","version":"215'8217","reported_seq":8400,"reported_epoch":215,"state":"active+clean","last_fresh":"2024-09-18T08:09:53.650099+0000","last_change":"2024-09-18T08:09:28.263522+0000","last_active":"2024-09-18T08:09:53.650099+0000","last_peered":"2024-09-18T08:09:53.650099+0000","last_clean":"2024-09-18T08:09:53.650099+0000","last_became_active":"2024-09-18T08:07:53.672797+0000","last_became_peered":"2024-09-18T08:07:53.672797+0000","last_unstale":"2024-09-18T08:09:53.650099+0000","last_undegraded":"2024-09-18T08:09:53.650099+0000","last_fullsized":"2024-09-18T08:09:53.650099+0000","mapping_epoch":165,"log_start":"186'3400","ondisk_log_start":"186'3400","created":165,"last_epoch_clean":166,"parent":"0.0","parent_split_bits":0,"last_scrub":"211'6448","last_scrub_stamp":"2024-09-18T08:09:28.263434+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:07:52.307205+0000","last_clean_scrub_stamp":"2024-09-18T08:09:28.263434+0000","objects_scrubbed":6448,"log_size":4817,"log_dups_size":0,"ondisk_log_size":4817,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:10:29.843317+0000","scrub_duration":646,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":306642944,"num_objects":4679,"num_object_clones":0,"num_object_copies":14037,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4679,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8217,"num_write_kb":412672,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6],"acting":[1,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.a","version":"74'79","reported_seq":149,"reported_epoch":215,"state":"active+clean","last_fresh":"2024-09-18T08:09:51.571668+0000","last_change":"2024-09-18T08:08:57.463205+0000","last_active":"2024-09-18T08:09:51.571668+0000","last_peered":"2024-09-18T08:09:51.571668+0000","last_clean":"2024-09-18T08:09:51.571668+0000","last_became_active":"2024-09-18T08:07:26.394584+0000","last_became_peered":"2024-09-18T08:07:26.394584+0000","last_unstale":"2024-09-18T08:09:51.571668+0000","last_undegraded":"2024-09-18T08:09:51.571668+0000","last_fullsized":"2024-09-18T08:09:51.571668+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":152,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":4,"last_scrub":"74'79","last_scrub_stamp":"2024-09-18T08:08:57.463113+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:00:11.893742+0000","last_clean_scrub_stamp":"2024-09-18T08:08:57.463113+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:09:57.971184+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,6],"acting":[5,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.f","version":"69'15","reported_seq":403,"reported_epoch":215,"state":"active+clean","last_fresh":"2024-09-18T08:09:51.592672+0000","last_change":"2024-09-18T08:08:56.669262+0000","last_active":"2024-09-18T08:09:51.592672+0000","last_peered":"2024-09-18T08:09:51.592672+0000","last_clean":"2024-09-18T08:09:51.592672+0000","last_became_active":"2024-09-18T08:08:18.540211+0000","last_became_peered":"2024-09-18T08:08:18.540211+0000","last_unstale":"2024-09-18T08:09:51.592672+0000","last_undegraded":"2024-09-18T08:09:51.592672+0000","last_fullsized":"2024-09-18T08:09:51.592672+0000","mapping_epoch":181,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":182,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'15","last_scrub_stamp":"2024-09-18T08:08:56.668397+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_clean_scrub_stamp":"2024-09-18T08:08:56.668397+0000","objects_scrubbed":7,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:10:22.470704+0000","scrub_duration":36,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":337,"reported_epoch":215,"state":"active+clean","last_fresh":"2024-09-18T08:09:51.584753+0000","last_change":"2024-09-18T08:08:18.389332+0000","last_active":"2024-09-18T08:09:51.584753+0000","last_peered":"2024-09-18T08:09:51.584753+0000","last_clean":"2024-09-18T08:09:51.584753+0000","last_became_active":"2024-09-18T08:02:47.879328+0000","last_became_peered":"2024-09-18T08:02:47.879328+0000","last_unstale":"2024-09-18T08:09:51.584753+0000","last_undegraded":"2024-09-18T08:09:51.584753+0000","last_fullsized":"2024-09-18T08:09:51.584753+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:08:18.389273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:46.860092+0000","last_clean_scrub_stamp":"2024-09-18T08:08:18.389273+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.e","version":"182'12","reported_seq":102,"reported_epoch":215,"state":"active+clean","last_fresh":"2024-09-18T08:09:51.558742+0000","last_change":"2024-09-18T08:08:18.560884+0000","last_active":"2024-09-18T08:09:51.558742+0000","last_peered":"2024-09-18T08:09:51.558742+0000","last_clean":"2024-09-18T08:09:51.558742+0000","last_became_active":"2024-09-18T08:08:18.525147+0000","last_became_peered":"2024-09-18T08:08:18.525147+0000","last_unstale":"2024-09-18T08:09:51.558742+0000","last_undegraded":"2024-09-18T08:09:51.558742+0000","last_fullsized":"2024-09-18T08:09:51.558742+0000","mapping_epoch":181,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":182,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'8","last_scrub_stamp":"2024-09-18T08:06:19.459375+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:50.886830+0000","last_clean_scrub_stamp":"2024-09-18T08:06:19.459375+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":333,"reported_epoch":215,"state":"active+clean","last_fresh":"2024-09-18T08:09:51.569038+0000","last_change":"2024-09-18T08:07:19.714477+0000","last_active":"2024-09-18T08:09:51.569038+0000","last_peered":"2024-09-18T08:09:51.569038+0000","last_clean":"2024-09-18T08:09:51.569038+0000","last_became_active":"2024-09-18T08:02:53.909834+0000","last_became_peered":"2024-09-18T08:02:53.909834+0000","last_unstale":"2024-09-18T08:09:51.569038+0000","last_undegraded":"2024-09-18T08:09:51.569038+0000","last_fullsized":"2024-09-18T08:09:51.569038+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:07:19.714430+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_clean_scrub_stamp":"2024-09-18T08:07:19.714430+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.c","version":"215'7983","reported_seq":8094,"reported_epoch":215,"state":"active+clean","last_fresh":"2024-09-18T08:09:53.811604+0000","last_change":"2024-09-18T08:07:53.670091+0000","last_active":"2024-09-18T08:09:53.811604+0000","last_peered":"2024-09-18T08:09:53.811604+0000","last_clean":"2024-09-18T08:09:53.811604+0000","last_became_active":"2024-09-18T08:07:53.669708+0000","last_became_peered":"2024-09-18T08:07:53.669708+0000","last_unstale":"2024-09-18T08:09:53.811604+0000","last_undegraded":"2024-09-18T08:09:53.811604+0000","last_fullsized":"2024-09-18T08:09:53.811604+0000","mapping_epoch":165,"log_start":"183'2600","ondisk_log_start":"183'2600","created":165,"last_epoch_clean":166,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:07:52.307205+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:07:52.307205+0000","last_clean_scrub_stamp":"2024-09-18T08:07:52.307205+0000","objects_scrubbed":0,"log_size":5383,"log_dups_size":0,"ondisk_log_size":5383,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":291438592,"num_objects":4447,"num_object_clones":0,"num_object_copies":13341,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4447,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7983,"num_write_kb":397760,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"74'79","reported_seq":93,"reported_epoch":215,"state":"active+clean","last_fresh":"2024-09-18T08:09:51.558312+0000","last_change":"2024-09-18T08:08:52.690758+0000","last_active":"2024-09-18T08:09:51.558312+0000","last_peered":"2024-09-18T08:09:51.558312+0000","last_clean":"2024-09-18T08:09:51.558312+0000","last_became_active":"2024-09-18T08:08:52.690344+0000","last_became_peered":"2024-09-18T08:08:52.690344+0000","last_unstale":"2024-09-18T08:09:51.558312+0000","last_undegraded":"2024-09-18T08:09:51.558312+0000","last_fullsized":"2024-09-18T08:09:51.558312+0000","mapping_epoch":198,"log_start":"0'0","ondisk_log_start":"0'0","created":152,"last_epoch_clean":199,"parent":"0.0","parent_split_bits":3,"last_scrub":"74'79","last_scrub_stamp":"2024-09-18T08:08:19.036836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:00:11.893742+0000","last_clean_scrub_stamp":"2024-09-18T08:08:19.036836+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,6],"acting":[4,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.0","version":"69'18","reported_seq":365,"reported_epoch":215,"state":"active+clean","last_fresh":"2024-09-18T08:09:51.584410+0000","last_change":"2024-09-18T08:06:44.301130+0000","last_active":"2024-09-18T08:09:51.584410+0000","last_peered":"2024-09-18T08:09:51.584410+0000","last_clean":"2024-09-18T08:09:51.584410+0000","last_became_active":"2024-09-18T08:02:49.886818+0000","last_became_peered":"2024-09-18T08:02:49.886818+0000","last_unstale":"2024-09-18T08:09:51.584410+0000","last_undegraded":"2024-09-18T08:09:51.584410+0000","last_fullsized":"2024-09-18T08:09:51.584410+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'18","last_scrub_stamp":"2024-09-18T08:06:44.300772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_clean_scrub_stamp":"2024-09-18T08:06:44.300772+0000","objects_scrubbed":8,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":34,"num_read_kb":22,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":345,"reported_epoch":215,"state":"active+clean","last_fresh":"2024-09-18T08:09:51.583693+0000","last_change":"2024-09-18T08:07:21.242587+0000","last_active":"2024-09-18T08:09:51.583693+0000","last_peered":"2024-09-18T08:09:51.583693+0000","last_clean":"2024-09-18T08:09:51.583693+0000","last_became_active":"2024-09-18T08:02:47.880225+0000","last_became_peered":"2024-09-18T08:02:47.880225+0000","last_unstale":"2024-09-18T08:09:51.583693+0000","last_undegraded":"2024-09-18T08:09:51.583693+0000","last_fullsized":"2024-09-18T08:09:51.583693+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:07:21.242494+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:46.860092+0000","last_clean_scrub_stamp":"2024-09-18T08:07:21.242494+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":18,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":78,"reported_epoch":215,"state":"active+clean","last_fresh":"2024-09-18T08:09:51.557985+0000","last_change":"2024-09-18T08:08:18.530658+0000","last_active":"2024-09-18T08:09:51.557985+0000","last_peered":"2024-09-18T08:09:51.557985+0000","last_clean":"2024-09-18T08:09:51.557985+0000","last_became_active":"2024-09-18T08:08:18.530341+0000","last_became_peered":"2024-09-18T08:08:18.530341+0000","last_unstale":"2024-09-18T08:09:51.557985+0000","last_undegraded":"2024-09-18T08:09:51.557985+0000","last_fullsized":"2024-09-18T08:09:51.557985+0000","mapping_epoch":181,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":182,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:06:10.468119+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:50.886830+0000","last_clean_scrub_stamp":"2024-09-18T08:06:10.468119+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":78,"reported_epoch":215,"state":"active+clean","last_fresh":"2024-09-18T08:09:51.558490+0000","last_change":"2024-09-18T08:08:18.551527+0000","last_active":"2024-09-18T08:09:51.558490+0000","last_peered":"2024-09-18T08:09:51.558490+0000","last_clean":"2024-09-18T08:09:51.558490+0000","last_became_active":"2024-09-18T08:08:18.543694+0000","last_became_peered":"2024-09-18T08:08:18.543694+0000","last_unstale":"2024-09-18T08:09:51.558490+0000","last_undegraded":"2024-09-18T08:09:51.558490+0000","last_fullsized":"2024-09-18T08:09:51.558490+0000","mapping_epoch":181,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":182,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:04:16.427390+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_clean_scrub_stamp":"2024-09-18T08:04:16.427390+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.d","version":"215'7936","reported_seq":8048,"reported_epoch":215,"state":"active+clean","last_fresh":"2024-09-18T08:09:53.277991+0000","last_change":"2024-09-18T08:07:53.671291+0000","last_active":"2024-09-18T08:09:53.277991+0000","last_peered":"2024-09-18T08:09:53.277991+0000","last_clean":"2024-09-18T08:09:53.277991+0000","last_became_active":"2024-09-18T08:07:53.670773+0000","last_became_peered":"2024-09-18T08:07:53.670773+0000","last_unstale":"2024-09-18T08:09:53.277991+0000","last_undegraded":"2024-09-18T08:09:53.277991+0000","last_fullsized":"2024-09-18T08:09:53.277991+0000","mapping_epoch":165,"log_start":"184'2800","ondisk_log_start":"184'2800","created":165,"last_epoch_clean":166,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:07:52.307205+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:07:52.307205+0000","last_clean_scrub_stamp":"2024-09-18T08:07:52.307205+0000","objects_scrubbed":0,"log_size":5136,"log_dups_size":0,"ondisk_log_size":5136,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":299696152,"num_objects":4574,"num_object_clones":0,"num_object_copies":13722,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4574,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7936,"num_write_kb":400257,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,2],"acting":[5,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.4","version":"74'79","reported_seq":155,"reported_epoch":215,"state":"active+clean","last_fresh":"2024-09-18T08:09:51.581774+0000","last_change":"2024-09-18T08:08:59.376417+0000","last_active":"2024-09-18T08:09:51.581774+0000","last_peered":"2024-09-18T08:09:51.581774+0000","last_clean":"2024-09-18T08:09:51.581774+0000","last_became_active":"2024-09-18T08:07:26.398930+0000","last_became_peered":"2024-09-18T08:07:26.398930+0000","last_unstale":"2024-09-18T08:09:51.581774+0000","last_undegraded":"2024-09-18T08:09:51.581774+0000","last_fullsized":"2024-09-18T08:09:51.581774+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":152,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":3,"last_scrub":"74'79","last_scrub_stamp":"2024-09-18T08:08:59.376337+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:00:11.893742+0000","last_clean_scrub_stamp":"2024-09-18T08:08:59.376337+0000","objects_scrubbed":1,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:10:26.641607+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":327680,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,7],"acting":[3,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"69'14","reported_seq":94,"reported_epoch":215,"state":"active+clean","last_fresh":"2024-09-18T08:09:51.557860+0000","last_change":"2024-09-18T08:08:18.617734+0000","last_active":"2024-09-18T08:09:51.557860+0000","last_peered":"2024-09-18T08:09:51.557860+0000","last_clean":"2024-09-18T08:09:51.557860+0000","last_became_active":"2024-09-18T08:08:18.549671+0000","last_became_peered":"2024-09-18T08:08:18.549671+0000","last_unstale":"2024-09-18T08:09:51.557860+0000","last_undegraded":"2024-09-18T08:09:51.557860+0000","last_fullsized":"2024-09-18T08:09:51.557860+0000","mapping_epoch":181,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":182,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'14","last_scrub_stamp":"2024-09-18T08:08:08.629246+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_clean_scrub_stamp":"2024-09-18T08:08:08.629246+0000","objects_scrubbed":0,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":14,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"62'2","reported_seq":369,"reported_epoch":215,"state":"active+clean","last_fresh":"2024-09-18T08:09:51.577843+0000","last_change":"2024-09-18T08:08:18.565279+0000","last_active":"2024-09-18T08:09:51.577843+0000","last_peered":"2024-09-18T08:09:51.577843+0000","last_clean":"2024-09-18T08:09:51.577843+0000","last_became_active":"2024-09-18T08:08:18.544872+0000","last_became_peered":"2024-09-18T08:08:18.544872+0000","last_unstale":"2024-09-18T08:09:51.577843+0000","last_undegraded":"2024-09-18T08:09:51.577843+0000","last_fullsized":"2024-09-18T08:09:51.577843+0000","mapping_epoch":181,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":182,"parent":"0.0","parent_split_bits":0,"last_scrub":"62'2","last_scrub_stamp":"2024-09-18T08:06:43.607152+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:46.860092+0000","last_clean_scrub_stamp":"2024-09-18T08:06:43.607152+0000","objects_scrubbed":2,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:10:26.053059+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1039,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":2078,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.0","version":"182'10","reported_seq":445,"reported_epoch":215,"state":"active+clean","last_fresh":"2024-09-18T08:09:51.584248+0000","last_change":"2024-09-18T08:09:51.467488+0000","last_active":"2024-09-18T08:09:51.584248+0000","last_peered":"2024-09-18T08:09:51.584248+0000","last_clean":"2024-09-18T08:09:51.584248+0000","last_became_active":"2024-09-18T08:08:18.541756+0000","last_became_peered":"2024-09-18T08:08:18.541756+0000","last_unstale":"2024-09-18T08:09:51.584248+0000","last_undegraded":"2024-09-18T08:09:51.584248+0000","last_fullsized":"2024-09-18T08:09:51.584248+0000","mapping_epoch":181,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":182,"parent":"0.0","parent_split_bits":0,"last_scrub":"182'10","last_scrub_stamp":"2024-09-18T08:09:51.467349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:50.886830+0000","last_clean_scrub_stamp":"2024-09-18T08:09:51.467349+0000","objects_scrubbed":1,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:10:51.919411+0000","scrub_duration":19,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":347,"reported_epoch":215,"state":"active+clean","last_fresh":"2024-09-18T08:09:51.570089+0000","last_change":"2024-09-18T08:07:49.650645+0000","last_active":"2024-09-18T08:09:51.570089+0000","last_peered":"2024-09-18T08:09:51.570089+0000","last_clean":"2024-09-18T08:09:51.570089+0000","last_became_active":"2024-09-18T08:06:24.140303+0000","last_became_peered":"2024-09-18T08:06:24.140303+0000","last_unstale":"2024-09-18T08:09:51.570089+0000","last_undegraded":"2024-09-18T08:09:51.570089+0000","last_fullsized":"2024-09-18T08:09:51.570089+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:07:49.650577+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_clean_scrub_stamp":"2024-09-18T08:07:49.650577+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:10:13.720079+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.e","version":"215'8060","reported_seq":12406,"reported_epoch":215,"state":"active+clean","last_fresh":"2024-09-18T08:09:53.801270+0000","last_change":"2024-09-18T08:09:17.296111+0000","last_active":"2024-09-18T08:09:53.801270+0000","last_peered":"2024-09-18T08:09:53.801270+0000","last_clean":"2024-09-18T08:09:53.801270+0000","last_became_active":"2024-09-18T08:09:17.286866+0000","last_became_peered":"2024-09-18T08:09:17.286866+0000","last_unstale":"2024-09-18T08:09:53.801270+0000","last_undegraded":"2024-09-18T08:09:53.801270+0000","last_fullsized":"2024-09-18T08:09:53.801270+0000","mapping_epoch":207,"log_start":"184'2700","ondisk_log_start":"184'2700","created":165,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:07:52.307205+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:07:52.307205+0000","last_clean_scrub_stamp":"2024-09-18T08:07:52.307205+0000","objects_scrubbed":0,"log_size":5360,"log_dups_size":0,"ondisk_log_size":5360,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":308543488,"num_objects":4708,"num_object_clones":0,"num_object_copies":14124,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4708,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8060,"num_write_kb":408576,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2107,"num_bytes_recovered":138084352,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,4],"acting":[0,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.7","version":"74'79","reported_seq":150,"reported_epoch":215,"state":"active+clean","last_fresh":"2024-09-18T08:09:51.583384+0000","last_change":"2024-09-18T08:08:57.438699+0000","last_active":"2024-09-18T08:09:51.583384+0000","last_peered":"2024-09-18T08:09:51.583384+0000","last_clean":"2024-09-18T08:09:51.583384+0000","last_became_active":"2024-09-18T08:07:26.398289+0000","last_became_peered":"2024-09-18T08:07:26.398289+0000","last_unstale":"2024-09-18T08:09:51.583384+0000","last_undegraded":"2024-09-18T08:09:51.583384+0000","last_fullsized":"2024-09-18T08:09:51.583384+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":152,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":3,"last_scrub":"74'79","last_scrub_stamp":"2024-09-18T08:08:57.438600+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:00:11.893742+0000","last_clean_scrub_stamp":"2024-09-18T08:08:57.438600+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:10:09.717533+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,7],"acting":[3,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.2","version":"69'10","reported_seq":393,"reported_epoch":215,"state":"active+clean","last_fresh":"2024-09-18T08:09:51.589506+0000","last_change":"2024-09-18T08:08:18.580590+0000","last_active":"2024-09-18T08:09:51.589506+0000","last_peered":"2024-09-18T08:09:51.589506+0000","last_clean":"2024-09-18T08:09:51.589506+0000","last_became_active":"2024-09-18T08:08:18.538711+0000","last_became_peered":"2024-09-18T08:08:18.538711+0000","last_unstale":"2024-09-18T08:09:51.589506+0000","last_undegraded":"2024-09-18T08:09:51.589506+0000","last_fullsized":"2024-09-18T08:09:51.589506+0000","mapping_epoch":181,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":182,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'10","last_scrub_stamp":"2024-09-18T08:07:22.666042+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_clean_scrub_stamp":"2024-09-18T08:07:22.666042+0000","objects_scrubbed":5,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:10:17.688460+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":328,"reported_epoch":215,"state":"active+clean","last_fresh":"2024-09-18T08:09:51.570393+0000","last_change":"2024-09-18T08:03:48.009689+0000","last_active":"2024-09-18T08:09:51.570393+0000","last_peered":"2024-09-18T08:09:51.570393+0000","last_clean":"2024-09-18T08:09:51.570393+0000","last_became_active":"2024-09-18T08:02:47.874727+0000","last_became_peered":"2024-09-18T08:02:47.874727+0000","last_unstale":"2024-09-18T08:09:51.570393+0000","last_undegraded":"2024-09-18T08:09:51.570393+0000","last_fullsized":"2024-09-18T08:09:51.570393+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:03:48.009611+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:46.860092+0000","last_clean_scrub_stamp":"2024-09-18T08:03:48.009611+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.3","version":"138'10","reported_seq":445,"reported_epoch":215,"state":"active+clean","last_fresh":"2024-09-18T08:09:51.564537+0000","last_change":"2024-09-18T08:07:46.695714+0000","last_active":"2024-09-18T08:09:51.564537+0000","last_peered":"2024-09-18T08:09:51.564537+0000","last_clean":"2024-09-18T08:09:51.564537+0000","last_became_active":"2024-09-18T08:06:24.134194+0000","last_became_peered":"2024-09-18T08:06:24.134194+0000","last_unstale":"2024-09-18T08:09:51.564537+0000","last_undegraded":"2024-09-18T08:09:51.564537+0000","last_fullsized":"2024-09-18T08:09:51.564537+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"138'10","last_scrub_stamp":"2024-09-18T08:07:46.695656+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:50.886830+0000","last_clean_scrub_stamp":"2024-09-18T08:07:46.695656+0000","objects_scrubbed":1,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":355,"reported_epoch":215,"state":"active+clean","last_fresh":"2024-09-18T08:09:51.564006+0000","last_change":"2024-09-18T08:09:46.989395+0000","last_active":"2024-09-18T08:09:51.564006+0000","last_peered":"2024-09-18T08:09:51.564006+0000","last_clean":"2024-09-18T08:09:51.564006+0000","last_became_active":"2024-09-18T08:08:52.700080+0000","last_became_peered":"2024-09-18T08:08:52.700080+0000","last_unstale":"2024-09-18T08:09:51.564006+0000","last_undegraded":"2024-09-18T08:09:51.564006+0000","last_fullsized":"2024-09-18T08:09:51.564006+0000","mapping_epoch":198,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":199,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:09:46.989329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_clean_scrub_stamp":"2024-09-18T08:09:46.989329+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:10:55.582764+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.f","version":"215'8175","reported_seq":24342,"reported_epoch":215,"state":"active+clean","last_fresh":"2024-09-18T08:09:53.784573+0000","last_change":"2024-09-18T08:09:28.367000+0000","last_active":"2024-09-18T08:09:53.784573+0000","last_peered":"2024-09-18T08:09:53.784573+0000","last_clean":"2024-09-18T08:09:53.784573+0000","last_became_active":"2024-09-18T08:09:28.363033+0000","last_became_peered":"2024-09-18T08:09:28.363033+0000","last_unstale":"2024-09-18T08:09:53.784573+0000","last_undegraded":"2024-09-18T08:09:53.784573+0000","last_fullsized":"2024-09-18T08:09:53.784573+0000","mapping_epoch":211,"log_start":"191'3800","ondisk_log_start":"191'3800","created":165,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:07:52.307205+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:07:52.307205+0000","last_clean_scrub_stamp":"2024-09-18T08:07:52.307205+0000","objects_scrubbed":0,"log_size":4375,"log_dups_size":0,"ondisk_log_size":4375,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":310181888,"num_objects":4733,"num_object_clones":0,"num_object_copies":14199,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4733,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8175,"num_write_kb":413056,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8010,"num_bytes_recovered":524943360,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,1],"acting":[3,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"74'79","reported_seq":230,"reported_epoch":215,"state":"active+clean","last_fresh":"2024-09-18T08:09:51.573198+0000","last_change":"2024-09-18T08:08:57.736557+0000","last_active":"2024-09-18T08:09:51.573198+0000","last_peered":"2024-09-18T08:09:51.573198+0000","last_clean":"2024-09-18T08:09:51.573198+0000","last_became_active":"2024-09-18T08:07:26.426532+0000","last_became_peered":"2024-09-18T08:07:26.426532+0000","last_unstale":"2024-09-18T08:09:51.573198+0000","last_undegraded":"2024-09-18T08:09:51.573198+0000","last_fullsized":"2024-09-18T08:09:51.573198+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":152,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":3,"last_scrub":"74'79","last_scrub_stamp":"2024-09-18T08:08:57.736355+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:00:11.893742+0000","last_clean_scrub_stamp":"2024-09-18T08:08:57.736355+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:10:19.314793+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,3],"acting":[0,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.3","version":"69'19","reported_seq":360,"reported_epoch":215,"state":"active+clean","last_fresh":"2024-09-18T08:09:51.564696+0000","last_change":"2024-09-18T08:04:07.231406+0000","last_active":"2024-09-18T08:09:51.564696+0000","last_peered":"2024-09-18T08:09:51.564696+0000","last_clean":"2024-09-18T08:09:51.564696+0000","last_became_active":"2024-09-18T08:02:49.890318+0000","last_became_peered":"2024-09-18T08:02:49.890318+0000","last_unstale":"2024-09-18T08:09:51.564696+0000","last_undegraded":"2024-09-18T08:09:51.564696+0000","last_fullsized":"2024-09-18T08:09:51.564696+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'19","last_scrub_stamp":"2024-09-18T08:04:07.229846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_clean_scrub_stamp":"2024-09-18T08:04:07.229846+0000","objects_scrubbed":8,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:10:19.093125+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":39,"num_read_kb":25,"num_write":22,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.4","version":"62'1","reported_seq":364,"reported_epoch":215,"state":"active+clean","last_fresh":"2024-09-18T08:09:51.590156+0000","last_change":"2024-09-18T08:09:02.569780+0000","last_active":"2024-09-18T08:09:51.590156+0000","last_peered":"2024-09-18T08:09:51.590156+0000","last_clean":"2024-09-18T08:09:51.590156+0000","last_became_active":"2024-09-18T08:08:52.688347+0000","last_became_peered":"2024-09-18T08:08:52.688347+0000","last_unstale":"2024-09-18T08:09:51.590156+0000","last_undegraded":"2024-09-18T08:09:51.590156+0000","last_fullsized":"2024-09-18T08:09:51.590156+0000","mapping_epoch":198,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":199,"parent":"0.0","parent_split_bits":0,"last_scrub":"62'1","last_scrub_stamp":"2024-09-18T08:09:02.569706+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:46.860092+0000","last_clean_scrub_stamp":"2024-09-18T08:09:02.569706+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:10:05.632401+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":436,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":872,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":254,"reported_epoch":215,"state":"active+clean","last_fresh":"2024-09-18T08:09:51.565152+0000","last_change":"2024-09-18T08:06:24.133954+0000","last_active":"2024-09-18T08:09:51.565152+0000","last_peered":"2024-09-18T08:09:51.565152+0000","last_clean":"2024-09-18T08:09:51.565152+0000","last_became_active":"2024-09-18T08:06:24.133262+0000","last_became_peered":"2024-09-18T08:06:24.133262+0000","last_unstale":"2024-09-18T08:09:51.565152+0000","last_undegraded":"2024-09-18T08:09:51.565152+0000","last_fullsized":"2024-09-18T08:09:51.565152+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:04:13.377561+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:50.886830+0000","last_clean_scrub_stamp":"2024-09-18T08:04:13.377561+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:10:22.094192+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":360,"reported_epoch":215,"state":"active+clean","last_fresh":"2024-09-18T08:09:51.589962+0000","last_change":"2024-09-18T08:08:52.688537+0000","last_active":"2024-09-18T08:09:51.589962+0000","last_peered":"2024-09-18T08:09:51.589962+0000","last_clean":"2024-09-18T08:09:51.589962+0000","last_became_active":"2024-09-18T08:08:52.688212+0000","last_became_peered":"2024-09-18T08:08:52.688212+0000","last_unstale":"2024-09-18T08:09:51.589962+0000","last_undegraded":"2024-09-18T08:09:51.589962+0000","last_fullsized":"2024-09-18T08:09:51.589962+0000","mapping_epoch":198,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":199,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:07:49.041138+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_clean_scrub_stamp":"2024-09-18T08:07:49.041138+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:10:25.679955+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.8","version":"215'7926","reported_seq":8037,"reported_epoch":215,"state":"active+clean","last_fresh":"2024-09-18T08:09:53.239077+0000","last_change":"2024-09-18T08:07:53.671548+0000","last_active":"2024-09-18T08:09:53.239077+0000","last_peered":"2024-09-18T08:09:53.239077+0000","last_clean":"2024-09-18T08:09:53.239077+0000","last_became_active":"2024-09-18T08:07:53.670635+0000","last_became_peered":"2024-09-18T08:07:53.670635+0000","last_unstale":"2024-09-18T08:09:53.239077+0000","last_undegraded":"2024-09-18T08:09:53.239077+0000","last_fullsized":"2024-09-18T08:09:53.239077+0000","mapping_epoch":165,"log_start":"184'2800","ondisk_log_start":"184'2800","created":165,"last_epoch_clean":166,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:07:52.307205+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:07:52.307205+0000","last_clean_scrub_stamp":"2024-09-18T08:07:52.307205+0000","objects_scrubbed":0,"log_size":5126,"log_dups_size":0,"ondisk_log_size":5126,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":296615936,"num_objects":4526,"num_object_clones":0,"num_object_copies":13578,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4526,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7926,"num_write_kb":398464,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.1","version":"74'79","reported_seq":149,"reported_epoch":215,"state":"active+clean","last_fresh":"2024-09-18T08:09:51.568338+0000","last_change":"2024-09-18T08:08:59.553431+0000","last_active":"2024-09-18T08:09:51.568338+0000","last_peered":"2024-09-18T08:09:51.568338+0000","last_clean":"2024-09-18T08:09:51.568338+0000","last_became_active":"2024-09-18T08:07:26.400643+0000","last_became_peered":"2024-09-18T08:07:26.400643+0000","last_unstale":"2024-09-18T08:09:51.568338+0000","last_undegraded":"2024-09-18T08:09:51.568338+0000","last_fullsized":"2024-09-18T08:09:51.568338+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":152,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":4,"last_scrub":"74'79","last_scrub_stamp":"2024-09-18T08:08:59.552592+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:00:11.893742+0000","last_clean_scrub_stamp":"2024-09-18T08:08:59.552592+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:10:15.399172+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,2],"acting":[5,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.4","version":"69'28","reported_seq":466,"reported_epoch":215,"state":"active+clean","last_fresh":"2024-09-18T08:09:51.590246+0000","last_change":"2024-09-18T08:08:57.639985+0000","last_active":"2024-09-18T08:09:51.590246+0000","last_peered":"2024-09-18T08:09:51.590246+0000","last_clean":"2024-09-18T08:09:51.590246+0000","last_became_active":"2024-09-18T08:02:49.897221+0000","last_became_peered":"2024-09-18T08:02:49.897221+0000","last_unstale":"2024-09-18T08:09:51.590246+0000","last_undegraded":"2024-09-18T08:09:51.590246+0000","last_fullsized":"2024-09-18T08:09:51.590246+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'28","last_scrub_stamp":"2024-09-18T08:08:57.639628+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_clean_scrub_stamp":"2024-09-18T08:08:57.639628+0000","objects_scrubbed":10,"log_size":28,"log_dups_size":0,"ondisk_log_size":28,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:10:27.518247+0000","scrub_duration":18,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":48,"num_read_kb":33,"num_write":26,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":78,"reported_epoch":215,"state":"active+clean","last_fresh":"2024-09-18T08:09:51.558428+0000","last_change":"2024-09-18T08:08:18.543579+0000","last_active":"2024-09-18T08:09:51.558428+0000","last_peered":"2024-09-18T08:09:51.558428+0000","last_clean":"2024-09-18T08:09:51.558428+0000","last_became_active":"2024-09-18T08:08:18.538852+0000","last_became_peered":"2024-09-18T08:08:18.538852+0000","last_unstale":"2024-09-18T08:09:51.558428+0000","last_undegraded":"2024-09-18T08:09:51.558428+0000","last_fullsized":"2024-09-18T08:09:51.558428+0000","mapping_epoch":181,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":182,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:05:39.490141+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:46.860092+0000","last_clean_scrub_stamp":"2024-09-18T08:05:39.490141+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"64'2","reported_seq":368,"reported_epoch":215,"state":"active+clean","last_fresh":"2024-09-18T08:09:51.572204+0000","last_change":"2024-09-18T08:06:24.165227+0000","last_active":"2024-09-18T08:09:51.572204+0000","last_peered":"2024-09-18T08:09:51.572204+0000","last_clean":"2024-09-18T08:09:51.572204+0000","last_became_active":"2024-09-18T08:06:24.136080+0000","last_became_peered":"2024-09-18T08:06:24.136080+0000","last_unstale":"2024-09-18T08:09:51.572204+0000","last_undegraded":"2024-09-18T08:09:51.572204+0000","last_fullsized":"2024-09-18T08:09:51.572204+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"64'2","last_scrub_stamp":"2024-09-18T08:05:44.802942+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:45.855979+0000","last_clean_scrub_stamp":"2024-09-18T08:05:44.802942+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.001348349,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":38,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":334,"reported_epoch":215,"state":"active+clean","last_fresh":"2024-09-18T08:09:51.569607+0000","last_change":"2024-09-18T08:08:18.544944+0000","last_active":"2024-09-18T08:09:51.569607+0000","last_peered":"2024-09-18T08:09:51.569607+0000","last_clean":"2024-09-18T08:09:51.569607+0000","last_became_active":"2024-09-18T08:08:18.544260+0000","last_became_peered":"2024-09-18T08:08:18.544260+0000","last_unstale":"2024-09-18T08:09:51.569607+0000","last_undegraded":"2024-09-18T08:09:51.569607+0000","last_fullsized":"2024-09-18T08:09:51.569607+0000","mapping_epoch":181,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":182,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:07:53.582095+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:50.886830+0000","last_clean_scrub_stamp":"2024-09-18T08:07:53.582095+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":340,"reported_epoch":215,"state":"active+clean","last_fresh":"2024-09-18T08:09:51.582042+0000","last_change":"2024-09-18T08:08:18.543527+0000","last_active":"2024-09-18T08:09:51.582042+0000","last_peered":"2024-09-18T08:09:51.582042+0000","last_clean":"2024-09-18T08:09:51.582042+0000","last_became_active":"2024-09-18T08:08:18.542678+0000","last_became_peered":"2024-09-18T08:08:18.542678+0000","last_unstale":"2024-09-18T08:09:51.582042+0000","last_undegraded":"2024-09-18T08:09:51.582042+0000","last_fullsized":"2024-09-18T08:09:51.582042+0000","mapping_epoch":181,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":182,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:07:08.245063+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_clean_scrub_stamp":"2024-09-18T08:07:08.245063+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.b","version":"215'8043","reported_seq":13666,"reported_epoch":215,"state":"active+clean","last_fresh":"2024-09-18T08:09:53.768353+0000","last_change":"2024-09-18T08:08:48.662821+0000","last_active":"2024-09-18T08:09:53.768353+0000","last_peered":"2024-09-18T08:09:53.768353+0000","last_clean":"2024-09-18T08:09:53.768353+0000","last_became_active":"2024-09-18T08:08:48.655676+0000","last_became_peered":"2024-09-18T08:08:48.655676+0000","last_unstale":"2024-09-18T08:09:53.768353+0000","last_undegraded":"2024-09-18T08:09:53.768353+0000","last_fullsized":"2024-09-18T08:09:53.768353+0000","mapping_epoch":194,"log_start":"190'3600","ondisk_log_start":"190'3600","created":165,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:07:52.307205+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:07:52.307205+0000","last_clean_scrub_stamp":"2024-09-18T08:07:52.307205+0000","objects_scrubbed":0,"log_size":4443,"log_dups_size":0,"ondisk_log_size":4443,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":303366144,"num_objects":4629,"num_object_clones":0,"num_object_copies":13887,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4629,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8043,"num_write_kb":405504,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2745,"num_bytes_recovered":179896320,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,1],"acting":[3,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"74'79","reported_seq":251,"reported_epoch":215,"state":"active+clean","last_fresh":"2024-09-18T08:09:51.565007+0000","last_change":"2024-09-18T08:08:18.524000+0000","last_active":"2024-09-18T08:09:51.565007+0000","last_peered":"2024-09-18T08:09:51.565007+0000","last_clean":"2024-09-18T08:09:51.565007+0000","last_became_active":"2024-09-18T08:08:18.522977+0000","last_became_peered":"2024-09-18T08:08:18.522977+0000","last_unstale":"2024-09-18T08:09:51.565007+0000","last_undegraded":"2024-09-18T08:09:51.565007+0000","last_fullsized":"2024-09-18T08:09:51.565007+0000","mapping_epoch":181,"log_start":"0'0","ondisk_log_start":"0'0","created":152,"last_epoch_clean":182,"parent":"0.0","parent_split_bits":4,"last_scrub":"74'79","last_scrub_stamp":"2024-09-18T08:08:09.574144+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:00:11.893742+0000","last_clean_scrub_stamp":"2024-09-18T08:08:09.574144+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,4],"acting":[0,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.7","version":"69'13","reported_seq":270,"reported_epoch":215,"state":"active+clean","last_fresh":"2024-09-18T08:09:51.578742+0000","last_change":"2024-09-18T08:04:18.164662+0000","last_active":"2024-09-18T08:09:51.578742+0000","last_peered":"2024-09-18T08:09:51.578742+0000","last_clean":"2024-09-18T08:09:51.578742+0000","last_became_active":"2024-09-18T08:04:18.144925+0000","last_became_peered":"2024-09-18T08:04:18.144925+0000","last_unstale":"2024-09-18T08:09:51.578742+0000","last_undegraded":"2024-09-18T08:09:51.578742+0000","last_fullsized":"2024-09-18T08:09:51.578742+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'13","last_scrub_stamp":"2024-09-18T08:03:50.700818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_clean_scrub_stamp":"2024-09-18T08:03:50.700818+0000","objects_scrubbed":0,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":29,"num_read_kb":18,"num_write":16,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,0],"acting":[2,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":350,"reported_epoch":215,"state":"active+clean","last_fresh":"2024-09-18T08:09:51.590913+0000","last_change":"2024-09-18T08:08:53.649127+0000","last_active":"2024-09-18T08:09:51.590913+0000","last_peered":"2024-09-18T08:09:51.590913+0000","last_clean":"2024-09-18T08:09:51.590913+0000","last_became_active":"2024-09-18T08:06:24.126334+0000","last_became_peered":"2024-09-18T08:06:24.126334+0000","last_unstale":"2024-09-18T08:09:51.590913+0000","last_undegraded":"2024-09-18T08:09:51.590913+0000","last_fullsized":"2024-09-18T08:09:51.590913+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:08:53.649079+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:46.860092+0000","last_clean_scrub_stamp":"2024-09-18T08:08:53.649079+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:10:14.347881+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"62'1","reported_seq":338,"reported_epoch":215,"state":"active+clean","last_fresh":"2024-09-18T08:09:51.574919+0000","last_change":"2024-09-18T08:04:09.732724+0000","last_active":"2024-09-18T08:09:51.574919+0000","last_peered":"2024-09-18T08:09:51.574919+0000","last_clean":"2024-09-18T08:09:51.574919+0000","last_became_active":"2024-09-18T08:02:46.868175+0000","last_became_peered":"2024-09-18T08:02:46.868175+0000","last_unstale":"2024-09-18T08:09:51.574919+0000","last_undegraded":"2024-09-18T08:09:51.574919+0000","last_fullsized":"2024-09-18T08:09:51.574919+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"62'1","last_scrub_stamp":"2024-09-18T08:04:09.732645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:45.855979+0000","last_clean_scrub_stamp":"2024-09-18T08:04:09.732645+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0.00066785499999999997,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":44,"reported_epoch":215,"state":"active+clean","last_fresh":"2024-09-18T08:09:51.578343+0000","last_change":"2024-09-18T08:08:52.690656+0000","last_active":"2024-09-18T08:09:51.578343+0000","last_peered":"2024-09-18T08:09:51.578343+0000","last_clean":"2024-09-18T08:09:51.578343+0000","last_became_active":"2024-09-18T08:08:52.690286+0000","last_became_peered":"2024-09-18T08:08:52.690286+0000","last_unstale":"2024-09-18T08:09:51.578343+0000","last_undegraded":"2024-09-18T08:09:51.578343+0000","last_fullsized":"2024-09-18T08:09:51.578343+0000","mapping_epoch":198,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":199,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:07:08.806748+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:50.886830+0000","last_clean_scrub_stamp":"2024-09-18T08:07:08.806748+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":347,"reported_epoch":215,"state":"active+clean","last_fresh":"2024-09-18T08:09:51.571089+0000","last_change":"2024-09-18T08:09:42.647525+0000","last_active":"2024-09-18T08:09:51.571089+0000","last_peered":"2024-09-18T08:09:51.571089+0000","last_clean":"2024-09-18T08:09:51.571089+0000","last_became_active":"2024-09-18T08:06:24.145382+0000","last_became_peered":"2024-09-18T08:06:24.145382+0000","last_unstale":"2024-09-18T08:09:51.571089+0000","last_undegraded":"2024-09-18T08:09:51.571089+0000","last_fullsized":"2024-09-18T08:09:51.571089+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:09:42.647317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_clean_scrub_stamp":"2024-09-18T08:09:42.647317+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:11:03.015274+0000","scrub_duration":26,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.a","version":"215'7978","reported_seq":8089,"reported_epoch":215,"state":"active+clean","last_fresh":"2024-09-18T08:09:53.643063+0000","last_change":"2024-09-18T08:07:53.326651+0000","last_active":"2024-09-18T08:09:53.643063+0000","last_peered":"2024-09-18T08:09:53.643063+0000","last_clean":"2024-09-18T08:09:53.643063+0000","last_became_active":"2024-09-18T08:07:53.326361+0000","last_became_peered":"2024-09-18T08:07:53.326361+0000","last_unstale":"2024-09-18T08:09:53.643063+0000","last_undegraded":"2024-09-18T08:09:53.643063+0000","last_fullsized":"2024-09-18T08:09:53.643063+0000","mapping_epoch":165,"log_start":"186'3200","ondisk_log_start":"186'3200","created":165,"last_epoch_clean":166,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:07:52.307205+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:07:52.307205+0000","last_clean_scrub_stamp":"2024-09-18T08:07:52.307205+0000","objects_scrubbed":0,"log_size":4778,"log_dups_size":0,"ondisk_log_size":4778,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:10:20.719923+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":301858816,"num_objects":4606,"num_object_clones":0,"num_object_copies":13818,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4606,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7978,"num_write_kb":402688,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.3","version":"74'79","reported_seq":171,"reported_epoch":215,"state":"active+clean","last_fresh":"2024-09-18T08:09:51.592112+0000","last_change":"2024-09-18T08:09:41.651800+0000","last_active":"2024-09-18T08:09:51.592112+0000","last_peered":"2024-09-18T08:09:51.592112+0000","last_clean":"2024-09-18T08:09:51.592112+0000","last_became_active":"2024-09-18T08:08:18.538929+0000","last_became_peered":"2024-09-18T08:08:18.538929+0000","last_unstale":"2024-09-18T08:09:51.592112+0000","last_undegraded":"2024-09-18T08:09:51.592112+0000","last_fullsized":"2024-09-18T08:09:51.592112+0000","mapping_epoch":181,"log_start":"0'0","ondisk_log_start":"0'0","created":152,"last_epoch_clean":182,"parent":"0.0","parent_split_bits":3,"last_scrub":"74'79","last_scrub_stamp":"2024-09-18T08:09:41.651729+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:00:11.893742+0000","last_clean_scrub_stamp":"2024-09-18T08:09:41.651729+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:10:55.722378+0000","scrub_duration":17,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.6","version":"69'12","reported_seq":392,"reported_epoch":215,"state":"active+clean","last_fresh":"2024-09-18T08:09:51.577623+0000","last_change":"2024-09-18T08:08:57.021693+0000","last_active":"2024-09-18T08:09:51.577623+0000","last_peered":"2024-09-18T08:09:51.577623+0000","last_clean":"2024-09-18T08:09:51.577623+0000","last_became_active":"2024-09-18T08:08:52.700130+0000","last_became_peered":"2024-09-18T08:08:52.700130+0000","last_unstale":"2024-09-18T08:09:51.577623+0000","last_undegraded":"2024-09-18T08:09:51.577623+0000","last_fullsized":"2024-09-18T08:09:51.577623+0000","mapping_epoch":198,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":200,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'12","last_scrub_stamp":"2024-09-18T08:07:39.603562+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_clean_scrub_stamp":"2024-09-18T08:07:39.603562+0000","objects_scrubbed":6,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":18,"num_read_kb":12,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":358,"reported_epoch":215,"state":"active+clean","last_fresh":"2024-09-18T08:09:51.564634+0000","last_change":"2024-09-18T08:09:34.089548+0000","last_active":"2024-09-18T08:09:51.564634+0000","last_peered":"2024-09-18T08:09:51.564634+0000","last_clean":"2024-09-18T08:09:51.564634+0000","last_became_active":"2024-09-18T08:08:18.523135+0000","last_became_peered":"2024-09-18T08:08:18.523135+0000","last_unstale":"2024-09-18T08:09:51.564634+0000","last_undegraded":"2024-09-18T08:09:51.564634+0000","last_fullsized":"2024-09-18T08:09:51.564634+0000","mapping_epoch":181,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":182,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:09:34.089476+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:46.860092+0000","last_clean_scrub_stamp":"2024-09-18T08:09:34.089476+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:10:41.071862+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"70'5","reported_seq":401,"reported_epoch":215,"state":"active+clean","last_fresh":"2024-09-18T08:09:51.753806+0000","last_change":"2024-09-18T08:09:09.649261+0000","last_active":"2024-09-18T08:09:51.753806+0000","last_peered":"2024-09-18T08:09:51.753806+0000","last_clean":"2024-09-18T08:09:51.753806+0000","last_became_active":"2024-09-18T08:06:46.141667+0000","last_became_peered":"2024-09-18T08:06:46.141667+0000","last_unstale":"2024-09-18T08:09:51.753806+0000","last_undegraded":"2024-09-18T08:09:51.753806+0000","last_fullsized":"2024-09-18T08:09:51.753806+0000","mapping_epoch":143,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":144,"parent":"0.0","parent_split_bits":0,"last_scrub":"70'5","last_scrub_stamp":"2024-09-18T08:09:09.649188+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:45.855979+0000","last_clean_scrub_stamp":"2024-09-18T08:09:09.649188+0000","objects_scrubbed":1,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:10:24.340859+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0.000504055,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":193,"num_read_kb":188,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":336,"reported_epoch":215,"state":"active+clean","last_fresh":"2024-09-18T08:09:51.569839+0000","last_change":"2024-09-18T08:07:20.767916+0000","last_active":"2024-09-18T08:09:51.569839+0000","last_peered":"2024-09-18T08:09:51.569839+0000","last_clean":"2024-09-18T08:09:51.569839+0000","last_became_active":"2024-09-18T08:02:51.913112+0000","last_became_peered":"2024-09-18T08:02:51.913112+0000","last_unstale":"2024-09-18T08:09:51.569839+0000","last_undegraded":"2024-09-18T08:09:51.569839+0000","last_fullsized":"2024-09-18T08:09:51.569839+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:07:20.767841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:50.886830+0000","last_clean_scrub_stamp":"2024-09-18T08:07:20.767841+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":317,"reported_epoch":215,"state":"active+clean","last_fresh":"2024-09-18T08:09:51.589994+0000","last_change":"2024-09-18T08:09:49.702132+0000","last_active":"2024-09-18T08:09:51.589994+0000","last_peered":"2024-09-18T08:09:51.589994+0000","last_clean":"2024-09-18T08:09:51.589994+0000","last_became_active":"2024-09-18T08:02:53.907404+0000","last_became_peered":"2024-09-18T08:02:53.907404+0000","last_unstale":"2024-09-18T08:09:51.589994+0000","last_undegraded":"2024-09-18T08:09:51.589994+0000","last_fullsized":"2024-09-18T08:09:51.589994+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:09:49.702055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_clean_scrub_stamp":"2024-09-18T08:09:49.702055+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:11:01.853786+0000","scrub_duration":23,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.9","version":"215'7724","reported_seq":7835,"reported_epoch":215,"state":"active+clean","last_fresh":"2024-09-18T08:09:53.214037+0000","last_change":"2024-09-18T08:07:53.672294+0000","last_active":"2024-09-18T08:09:53.214037+0000","last_peered":"2024-09-18T08:09:53.214037+0000","last_clean":"2024-09-18T08:09:53.214037+0000","last_became_active":"2024-09-18T08:07:53.671994+0000","last_became_peered":"2024-09-18T08:07:53.671994+0000","last_unstale":"2024-09-18T08:09:53.214037+0000","last_undegraded":"2024-09-18T08:09:53.214037+0000","last_fullsized":"2024-09-18T08:09:53.214037+0000","mapping_epoch":165,"log_start":"183'2600","ondisk_log_start":"183'2600","created":165,"last_epoch_clean":166,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:07:52.307205+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:07:52.307205+0000","last_clean_scrub_stamp":"2024-09-18T08:07:52.307205+0000","objects_scrubbed":0,"log_size":5124,"log_dups_size":0,"ondisk_log_size":5124,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":294256640,"num_objects":4490,"num_object_clones":0,"num_object_copies":13470,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4490,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7724,"num_write_kb":390848,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.5","version":"182'21","reported_seq":116,"reported_epoch":215,"state":"active+clean","last_fresh":"2024-09-18T08:09:51.558884+0000","last_change":"2024-09-18T08:08:18.609652+0000","last_active":"2024-09-18T08:09:51.558884+0000","last_peered":"2024-09-18T08:09:51.558884+0000","last_clean":"2024-09-18T08:09:51.558884+0000","last_became_active":"2024-09-18T08:08:18.547555+0000","last_became_peered":"2024-09-18T08:08:18.547555+0000","last_unstale":"2024-09-18T08:09:51.558884+0000","last_undegraded":"2024-09-18T08:09:51.558884+0000","last_fullsized":"2024-09-18T08:09:51.558884+0000","mapping_epoch":181,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":182,"parent":"0.0","parent_split_bits":0,"last_scrub":"130'18","last_scrub_stamp":"2024-09-18T08:06:15.455503+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_clean_scrub_stamp":"2024-09-18T08:06:15.455503+0000","objects_scrubbed":0,"log_size":21,"log_dups_size":0,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":24,"num_read_kb":14,"num_write":13,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":154,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,7],"acting":[4,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":351,"reported_epoch":215,"state":"active+clean","last_fresh":"2024-09-18T08:09:51.581604+0000","last_change":"2024-09-18T08:09:45.572278+0000","last_active":"2024-09-18T08:09:51.581604+0000","last_peered":"2024-09-18T08:09:51.581604+0000","last_clean":"2024-09-18T08:09:51.581604+0000","last_became_active":"2024-09-18T08:06:24.133425+0000","last_became_peered":"2024-09-18T08:06:24.133425+0000","last_unstale":"2024-09-18T08:09:51.581604+0000","last_undegraded":"2024-09-18T08:09:51.581604+0000","last_fullsized":"2024-09-18T08:09:51.581604+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:09:45.572201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:46.860092+0000","last_clean_scrub_stamp":"2024-09-18T08:09:45.572201+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:11:00.131734+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"74'79","reported_seq":432,"reported_epoch":215,"state":"active+clean","last_fresh":"2024-09-18T08:09:51.570268+0000","last_change":"2024-09-18T08:08:57.769104+0000","last_active":"2024-09-18T08:09:51.570268+0000","last_peered":"2024-09-18T08:09:51.570268+0000","last_clean":"2024-09-18T08:09:51.570268+0000","last_became_active":"2024-09-18T08:07:24.376931+0000","last_became_peered":"2024-09-18T08:07:24.376931+0000","last_unstale":"2024-09-18T08:09:51.570268+0000","last_undegraded":"2024-09-18T08:09:51.570268+0000","last_fullsized":"2024-09-18T08:09:51.570268+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":153,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'79","last_scrub_stamp":"2024-09-18T08:08:57.769048+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:00:11.893742+0000","last_clean_scrub_stamp":"2024-09-18T08:08:57.769048+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:10:19.097626+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":164,"reported_epoch":215,"state":"active+clean","last_fresh":"2024-09-18T08:09:51.580257+0000","last_change":"2024-09-18T08:07:09.744206+0000","last_active":"2024-09-18T08:09:51.580257+0000","last_peered":"2024-09-18T08:09:51.580257+0000","last_clean":"2024-09-18T08:09:51.580257+0000","last_became_active":"2024-09-18T08:06:46.141046+0000","last_became_peered":"2024-09-18T08:06:46.141046+0000","last_unstale":"2024-09-18T08:09:51.580257+0000","last_undegraded":"2024-09-18T08:09:51.580257+0000","last_fullsized":"2024-09-18T08:09:51.580257+0000","mapping_epoch":143,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":144,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:07:09.744137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:50.886830+0000","last_clean_scrub_stamp":"2024-09-18T08:07:09.744137+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":339,"reported_epoch":215,"state":"active+clean","last_fresh":"2024-09-18T08:09:51.569519+0000","last_change":"2024-09-18T08:08:18.548620+0000","last_active":"2024-09-18T08:09:51.569519+0000","last_peered":"2024-09-18T08:09:51.569519+0000","last_clean":"2024-09-18T08:09:51.569519+0000","last_became_active":"2024-09-18T08:08:18.548233+0000","last_became_peered":"2024-09-18T08:08:18.548233+0000","last_unstale":"2024-09-18T08:09:51.569519+0000","last_undegraded":"2024-09-18T08:09:51.569519+0000","last_fullsized":"2024-09-18T08:09:51.569519+0000","mapping_epoch":181,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":182,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:06:50.710565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_clean_scrub_stamp":"2024-09-18T08:06:50.710565+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.2","version":"215'7882","reported_seq":12270,"reported_epoch":215,"state":"active+clean","last_fresh":"2024-09-18T08:09:53.167778+0000","last_change":"2024-09-18T08:08:58.743574+0000","last_active":"2024-09-18T08:09:53.167778+0000","last_peered":"2024-09-18T08:09:53.167778+0000","last_clean":"2024-09-18T08:09:53.167778+0000","last_became_active":"2024-09-18T08:08:58.738458+0000","last_became_peered":"2024-09-18T08:08:58.738458+0000","last_unstale":"2024-09-18T08:09:53.167778+0000","last_undegraded":"2024-09-18T08:09:53.167778+0000","last_fullsized":"2024-09-18T08:09:53.167778+0000","mapping_epoch":201,"log_start":"188'3400","ondisk_log_start":"188'3400","created":165,"last_epoch_clean":202,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:07:52.307205+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:07:52.307205+0000","last_clean_scrub_stamp":"2024-09-18T08:07:52.307205+0000","objects_scrubbed":0,"log_size":4482,"log_dups_size":0,"ondisk_log_size":4482,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":307363840,"num_objects":4690,"num_object_clones":0,"num_object_copies":14070,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4690,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7882,"num_write_kb":402304,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2129,"num_bytes_recovered":139526144,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,5],"acting":[4,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.e","version":"69'11","reported_seq":90,"reported_epoch":215,"state":"active+clean","last_fresh":"2024-09-18T08:09:51.558662+0000","last_change":"2024-09-18T08:08:18.594670+0000","last_active":"2024-09-18T08:09:51.558662+0000","last_peered":"2024-09-18T08:09:51.558662+0000","last_clean":"2024-09-18T08:09:51.558662+0000","last_became_active":"2024-09-18T08:08:18.544581+0000","last_became_peered":"2024-09-18T08:08:18.544581+0000","last_unstale":"2024-09-18T08:09:51.558662+0000","last_undegraded":"2024-09-18T08:09:51.558662+0000","last_fullsized":"2024-09-18T08:09:51.558662+0000","mapping_epoch":181,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":182,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'11","last_scrub_stamp":"2024-09-18T08:06:14.472168+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_clean_scrub_stamp":"2024-09-18T08:06:14.472168+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":19,"num_read_kb":12,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":78,"reported_epoch":215,"state":"active+clean","last_fresh":"2024-09-18T08:09:51.558081+0000","last_change":"2024-09-18T08:08:18.524516+0000","last_active":"2024-09-18T08:09:51.558081+0000","last_peered":"2024-09-18T08:09:51.558081+0000","last_clean":"2024-09-18T08:09:51.558081+0000","last_became_active":"2024-09-18T08:08:18.524229+0000","last_became_peered":"2024-09-18T08:08:18.524229+0000","last_unstale":"2024-09-18T08:09:51.558081+0000","last_undegraded":"2024-09-18T08:09:51.558081+0000","last_fullsized":"2024-09-18T08:09:51.558081+0000","mapping_epoch":181,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":182,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:02:46.860092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:46.860092+0000","last_clean_scrub_stamp":"2024-09-18T08:02:46.860092+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":356,"reported_epoch":215,"state":"active+clean","last_fresh":"2024-09-18T08:09:51.569373+0000","last_change":"2024-09-18T08:08:18.547880+0000","last_active":"2024-09-18T08:09:51.569373+0000","last_peered":"2024-09-18T08:09:51.569373+0000","last_clean":"2024-09-18T08:09:51.569373+0000","last_became_active":"2024-09-18T08:08:18.547105+0000","last_became_peered":"2024-09-18T08:08:18.547105+0000","last_unstale":"2024-09-18T08:09:51.569373+0000","last_undegraded":"2024-09-18T08:09:51.569373+0000","last_fullsized":"2024-09-18T08:09:51.569373+0000","mapping_epoch":181,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":182,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:07:51.742090+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:50.886830+0000","last_clean_scrub_stamp":"2024-09-18T08:07:51.742090+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":339,"reported_epoch":215,"state":"active+clean","last_fresh":"2024-09-18T08:09:51.581889+0000","last_change":"2024-09-18T08:07:09.197912+0000","last_active":"2024-09-18T08:09:51.581889+0000","last_peered":"2024-09-18T08:09:51.581889+0000","last_clean":"2024-09-18T08:09:51.581889+0000","last_became_active":"2024-09-18T08:06:24.130266+0000","last_became_peered":"2024-09-18T08:06:24.130266+0000","last_unstale":"2024-09-18T08:09:51.581889+0000","last_undegraded":"2024-09-18T08:09:51.581889+0000","last_fullsized":"2024-09-18T08:09:51.581889+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:07:09.197841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_clean_scrub_stamp":"2024-09-18T08:07:09.197841+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.1","version":"215'8016","reported_seq":8193,"reported_epoch":215,"state":"active+clean","last_fresh":"2024-09-18T08:09:53.675165+0000","last_change":"2024-09-18T08:09:40.130643+0000","last_active":"2024-09-18T08:09:53.675165+0000","last_peered":"2024-09-18T08:09:53.675165+0000","last_clean":"2024-09-18T08:09:53.675165+0000","last_became_active":"2024-09-18T08:07:53.327987+0000","last_became_peered":"2024-09-18T08:07:53.327987+0000","last_unstale":"2024-09-18T08:09:53.675165+0000","last_undegraded":"2024-09-18T08:09:53.675165+0000","last_fullsized":"2024-09-18T08:09:53.675165+0000","mapping_epoch":165,"log_start":"186'3200","ondisk_log_start":"186'3200","created":165,"last_epoch_clean":166,"parent":"0.0","parent_split_bits":0,"last_scrub":"213'7038","last_scrub_stamp":"2024-09-18T08:09:40.130353+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:07:52.307205+0000","last_clean_scrub_stamp":"2024-09-18T08:09:40.130353+0000","objects_scrubbed":5709,"log_size":4816,"log_dups_size":0,"ondisk_log_size":4816,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:11:06.670120+0000","scrub_duration":2501,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":305528832,"num_objects":4662,"num_object_clones":0,"num_object_copies":13986,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4662,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8016,"num_write_kb":405696,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.8","version":"74'79","reported_seq":156,"reported_epoch":215,"state":"active+clean","last_fresh":"2024-09-18T08:09:51.568791+0000","last_change":"2024-09-18T08:09:00.533593+0000","last_active":"2024-09-18T08:09:51.568791+0000","last_peered":"2024-09-18T08:09:51.568791+0000","last_clean":"2024-09-18T08:09:51.568791+0000","last_became_active":"2024-09-18T08:07:26.407298+0000","last_became_peered":"2024-09-18T08:07:26.407298+0000","last_unstale":"2024-09-18T08:09:51.568791+0000","last_undegraded":"2024-09-18T08:09:51.568791+0000","last_fullsized":"2024-09-18T08:09:51.568791+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":152,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":4,"last_scrub":"74'79","last_scrub_stamp":"2024-09-18T08:09:00.532813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:00:11.893742+0000","last_clean_scrub_stamp":"2024-09-18T08:09:00.532813+0000","objects_scrubbed":1,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:10:17.887770+0000","scrub_duration":25,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262688,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,3],"acting":[5,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.d","version":"69'17","reported_seq":96,"reported_epoch":215,"state":"active+clean","last_fresh":"2024-09-18T08:09:51.558665+0000","last_change":"2024-09-18T08:08:18.588624+0000","last_active":"2024-09-18T08:09:51.558665+0000","last_peered":"2024-09-18T08:09:51.558665+0000","last_clean":"2024-09-18T08:09:51.558665+0000","last_became_active":"2024-09-18T08:08:18.540862+0000","last_became_peered":"2024-09-18T08:08:18.540862+0000","last_unstale":"2024-09-18T08:09:51.558665+0000","last_undegraded":"2024-09-18T08:09:51.558665+0000","last_fullsized":"2024-09-18T08:09:51.558665+0000","mapping_epoch":181,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":182,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'17","last_scrub_stamp":"2024-09-18T08:08:09.671248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_clean_scrub_stamp":"2024-09-18T08:08:09.671248+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":28,"num_read_kb":18,"num_write":18,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,1],"acting":[4,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":78,"reported_epoch":215,"state":"active+clean","last_fresh":"2024-09-18T08:09:51.557999+0000","last_change":"2024-09-18T08:08:18.541029+0000","last_active":"2024-09-18T08:09:51.557999+0000","last_peered":"2024-09-18T08:09:51.557999+0000","last_clean":"2024-09-18T08:09:51.557999+0000","last_became_active":"2024-09-18T08:08:18.540285+0000","last_became_peered":"2024-09-18T08:08:18.540285+0000","last_unstale":"2024-09-18T08:09:51.557999+0000","last_undegraded":"2024-09-18T08:09:51.557999+0000","last_fullsized":"2024-09-18T08:09:51.557999+0000","mapping_epoch":181,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":182,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:05:12.622901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:46.860092+0000","last_clean_scrub_stamp":"2024-09-18T08:05:12.622901+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":350,"reported_epoch":215,"state":"active+clean","last_fresh":"2024-09-18T08:09:51.589297+0000","last_change":"2024-09-18T08:09:06.610955+0000","last_active":"2024-09-18T08:09:51.589297+0000","last_peered":"2024-09-18T08:09:51.589297+0000","last_clean":"2024-09-18T08:09:51.589297+0000","last_became_active":"2024-09-18T08:08:18.623014+0000","last_became_peered":"2024-09-18T08:08:18.623014+0000","last_unstale":"2024-09-18T08:09:51.589297+0000","last_undegraded":"2024-09-18T08:09:51.589297+0000","last_fullsized":"2024-09-18T08:09:51.589297+0000","mapping_epoch":181,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":182,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:09:06.610879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:50.886830+0000","last_clean_scrub_stamp":"2024-09-18T08:09:06.610879+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:10:19.831235+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":44,"reported_epoch":215,"state":"active+clean","last_fresh":"2024-09-18T08:09:51.578654+0000","last_change":"2024-09-18T08:08:52.696713+0000","last_active":"2024-09-18T08:09:51.578654+0000","last_peered":"2024-09-18T08:09:51.578654+0000","last_clean":"2024-09-18T08:09:51.578654+0000","last_became_active":"2024-09-18T08:08:52.696300+0000","last_became_peered":"2024-09-18T08:08:52.696300+0000","last_unstale":"2024-09-18T08:09:51.578654+0000","last_undegraded":"2024-09-18T08:09:51.578654+0000","last_fullsized":"2024-09-18T08:09:51.578654+0000","mapping_epoch":198,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":199,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:04:06.709631+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_clean_scrub_stamp":"2024-09-18T08:04:06.709631+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"8.0","version":"215'7946","reported_seq":8058,"reported_epoch":215,"state":"active+clean","last_fresh":"2024-09-18T08:09:53.799393+0000","last_change":"2024-09-18T08:07:53.329182+0000","last_active":"2024-09-18T08:09:53.799393+0000","last_peered":"2024-09-18T08:09:53.799393+0000","last_clean":"2024-09-18T08:09:53.799393+0000","last_became_active":"2024-09-18T08:07:53.328914+0000","last_became_peered":"2024-09-18T08:07:53.328914+0000","last_unstale":"2024-09-18T08:09:53.799393+0000","last_undegraded":"2024-09-18T08:09:53.799393+0000","last_fullsized":"2024-09-18T08:09:53.799393+0000","mapping_epoch":165,"log_start":"190'3500","ondisk_log_start":"190'3500","created":165,"last_epoch_clean":166,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:07:52.307205+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:07:52.307205+0000","last_clean_scrub_stamp":"2024-09-18T08:07:52.307205+0000","objects_scrubbed":0,"log_size":4446,"log_dups_size":0,"ondisk_log_size":4446,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":301727744,"num_objects":4604,"num_object_clones":0,"num_object_copies":13812,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4604,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7946,"num_write_kb":401600,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.9","version":"74'79","reported_seq":454,"reported_epoch":215,"state":"active+clean","last_fresh":"2024-09-18T08:09:51.569476+0000","last_change":"2024-09-18T08:09:27.676247+0000","last_active":"2024-09-18T08:09:51.569476+0000","last_peered":"2024-09-18T08:09:51.569476+0000","last_clean":"2024-09-18T08:09:51.569476+0000","last_became_active":"2024-09-18T08:08:52.691492+0000","last_became_peered":"2024-09-18T08:08:52.691492+0000","last_unstale":"2024-09-18T08:09:51.569476+0000","last_undegraded":"2024-09-18T08:09:51.569476+0000","last_fullsized":"2024-09-18T08:09:51.569476+0000","mapping_epoch":198,"log_start":"0'0","ondisk_log_start":"0'0","created":152,"last_epoch_clean":199,"parent":"0.0","parent_split_bits":4,"last_scrub":"74'79","last_scrub_stamp":"2024-09-18T08:09:27.676169+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:00:11.893742+0000","last_clean_scrub_stamp":"2024-09-18T08:09:27.676169+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:10:36.422152+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,2],"acting":[7,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.c","version":"69'10","reported_seq":90,"reported_epoch":215,"state":"active+clean","last_fresh":"2024-09-18T08:09:51.558312+0000","last_change":"2024-09-18T08:08:18.642452+0000","last_active":"2024-09-18T08:09:51.558312+0000","last_peered":"2024-09-18T08:09:51.558312+0000","last_clean":"2024-09-18T08:09:51.558312+0000","last_became_active":"2024-09-18T08:08:18.622980+0000","last_became_peered":"2024-09-18T08:08:18.622980+0000","last_unstale":"2024-09-18T08:09:51.558312+0000","last_undegraded":"2024-09-18T08:09:51.558312+0000","last_fullsized":"2024-09-18T08:09:51.558312+0000","mapping_epoch":181,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":182,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'10","last_scrub_stamp":"2024-09-18T08:05:41.474813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_clean_scrub_stamp":"2024-09-18T08:05:41.474813+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":14,"num_read_kb":9,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":351,"reported_epoch":215,"state":"active+clean","last_fresh":"2024-09-18T08:09:51.582858+0000","last_change":"2024-09-18T08:08:18.542800+0000","last_active":"2024-09-18T08:09:51.582858+0000","last_peered":"2024-09-18T08:09:51.582858+0000","last_clean":"2024-09-18T08:09:51.582858+0000","last_became_active":"2024-09-18T08:08:18.542424+0000","last_became_peered":"2024-09-18T08:08:18.542424+0000","last_unstale":"2024-09-18T08:09:51.582858+0000","last_undegraded":"2024-09-18T08:09:51.582858+0000","last_fullsized":"2024-09-18T08:09:51.582858+0000","mapping_epoch":181,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":182,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:06:46.339428+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:46.860092+0000","last_clean_scrub_stamp":"2024-09-18T08:06:46.339428+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.d","version":"69'8","reported_seq":423,"reported_epoch":215,"state":"active+clean","last_fresh":"2024-09-18T08:09:51.585440+0000","last_change":"2024-09-18T08:05:33.027988+0000","last_active":"2024-09-18T08:09:51.585440+0000","last_peered":"2024-09-18T08:09:51.585440+0000","last_clean":"2024-09-18T08:09:51.585440+0000","last_became_active":"2024-09-18T08:02:51.909011+0000","last_became_peered":"2024-09-18T08:02:51.909011+0000","last_unstale":"2024-09-18T08:09:51.585440+0000","last_undegraded":"2024-09-18T08:09:51.585440+0000","last_fullsized":"2024-09-18T08:09:51.585440+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'8","last_scrub_stamp":"2024-09-18T08:05:33.027783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:50.886830+0000","last_clean_scrub_stamp":"2024-09-18T08:05:33.027783+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":85,"reported_epoch":215,"state":"active+clean","last_fresh":"2024-09-18T08:09:51.558793+0000","last_change":"2024-09-18T08:08:52.685954+0000","last_active":"2024-09-18T08:09:51.558793+0000","last_peered":"2024-09-18T08:09:51.558793+0000","last_clean":"2024-09-18T08:09:51.558793+0000","last_became_active":"2024-09-18T08:08:52.685603+0000","last_became_peered":"2024-09-18T08:08:52.685603+0000","last_unstale":"2024-09-18T08:09:51.558793+0000","last_undegraded":"2024-09-18T08:09:51.558793+0000","last_fullsized":"2024-09-18T08:09:51.558793+0000","mapping_epoch":198,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":199,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:07:16.559597+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_clean_scrub_stamp":"2024-09-18T08:07:16.559597+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.7","version":"215'8013","reported_seq":8194,"reported_epoch":215,"state":"active+clean","last_fresh":"2024-09-18T08:09:53.673977+0000","last_change":"2024-09-18T08:09:36.764450+0000","last_active":"2024-09-18T08:09:53.673977+0000","last_peered":"2024-09-18T08:09:53.673977+0000","last_clean":"2024-09-18T08:09:53.673977+0000","last_became_active":"2024-09-18T08:07:53.674433+0000","last_became_peered":"2024-09-18T08:07:53.674433+0000","last_unstale":"2024-09-18T08:09:53.673977+0000","last_undegraded":"2024-09-18T08:09:53.673977+0000","last_fullsized":"2024-09-18T08:09:53.673977+0000","mapping_epoch":165,"log_start":"186'3200","ondisk_log_start":"186'3200","created":165,"last_epoch_clean":166,"parent":"0.0","parent_split_bits":0,"last_scrub":"213'6848","last_scrub_stamp":"2024-09-18T08:09:36.764206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:07:52.307205+0000","last_clean_scrub_stamp":"2024-09-18T08:09:36.764206+0000","objects_scrubbed":5822,"log_size":4813,"log_dups_size":0,"ondisk_log_size":4813,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":4,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:11:06.120132+0000","scrub_duration":3143,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":298909696,"num_objects":4561,"num_object_clones":0,"num_object_copies":13683,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4561,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8013,"num_write_kb":402368,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,6],"acting":[1,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.b","version":"69'9","reported_seq":382,"reported_epoch":215,"state":"active+clean","last_fresh":"2024-09-18T08:09:51.564594+0000","last_change":"2024-09-18T08:08:18.640380+0000","last_active":"2024-09-18T08:09:51.564594+0000","last_peered":"2024-09-18T08:09:51.564594+0000","last_clean":"2024-09-18T08:09:51.564594+0000","last_became_active":"2024-09-18T08:08:18.522995+0000","last_became_peered":"2024-09-18T08:08:18.522995+0000","last_unstale":"2024-09-18T08:09:51.564594+0000","last_undegraded":"2024-09-18T08:09:51.564594+0000","last_fullsized":"2024-09-18T08:09:51.564594+0000","mapping_epoch":181,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":182,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'9","last_scrub_stamp":"2024-09-18T08:08:18.640292+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_clean_scrub_stamp":"2024-09-18T08:08:18.640292+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":350,"reported_epoch":215,"state":"active+clean","last_fresh":"2024-09-18T08:09:51.571563+0000","last_change":"2024-09-18T08:07:40.708363+0000","last_active":"2024-09-18T08:09:51.571563+0000","last_peered":"2024-09-18T08:09:51.571563+0000","last_clean":"2024-09-18T08:09:51.571563+0000","last_became_active":"2024-09-18T08:06:24.135005+0000","last_became_peered":"2024-09-18T08:06:24.135005+0000","last_unstale":"2024-09-18T08:09:51.571563+0000","last_undegraded":"2024-09-18T08:09:51.571563+0000","last_fullsized":"2024-09-18T08:09:51.571563+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:07:40.708307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:46.860092+0000","last_clean_scrub_stamp":"2024-09-18T08:07:40.708307+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":351,"reported_epoch":215,"state":"active+clean","last_fresh":"2024-09-18T08:09:51.577497+0000","last_change":"2024-09-18T08:08:18.509651+0000","last_active":"2024-09-18T08:09:51.577497+0000","last_peered":"2024-09-18T08:09:51.577497+0000","last_clean":"2024-09-18T08:09:51.577497+0000","last_became_active":"2024-09-18T08:08:18.509247+0000","last_became_peered":"2024-09-18T08:08:18.509247+0000","last_unstale":"2024-09-18T08:09:51.577497+0000","last_undegraded":"2024-09-18T08:09:51.577497+0000","last_fullsized":"2024-09-18T08:09:51.577497+0000","mapping_epoch":181,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":182,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:07:10.728679+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:50.886830+0000","last_clean_scrub_stamp":"2024-09-18T08:07:10.728679+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,3],"acting":[2,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":347,"reported_epoch":215,"state":"active+clean","last_fresh":"2024-09-18T08:09:51.564485+0000","last_change":"2024-09-18T08:09:28.086782+0000","last_active":"2024-09-18T08:09:51.564485+0000","last_peered":"2024-09-18T08:09:51.564485+0000","last_clean":"2024-09-18T08:09:51.564485+0000","last_became_active":"2024-09-18T08:08:52.697337+0000","last_became_peered":"2024-09-18T08:08:52.697337+0000","last_unstale":"2024-09-18T08:09:51.564485+0000","last_undegraded":"2024-09-18T08:09:51.564485+0000","last_fullsized":"2024-09-18T08:09:51.564485+0000","mapping_epoch":198,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":199,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:09:28.086727+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_clean_scrub_stamp":"2024-09-18T08:09:28.086727+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:10:33.905647+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.6","version":"215'7977","reported_seq":8088,"reported_epoch":215,"state":"active+clean","last_fresh":"2024-09-18T08:09:53.807470+0000","last_change":"2024-09-18T08:07:53.674440+0000","last_active":"2024-09-18T08:09:53.807470+0000","last_peered":"2024-09-18T08:09:53.807470+0000","last_clean":"2024-09-18T08:09:53.807470+0000","last_became_active":"2024-09-18T08:07:53.674139+0000","last_became_peered":"2024-09-18T08:07:53.674139+0000","last_unstale":"2024-09-18T08:09:53.807470+0000","last_undegraded":"2024-09-18T08:09:53.807470+0000","last_fullsized":"2024-09-18T08:09:53.807470+0000","mapping_epoch":165,"log_start":"183'2600","ondisk_log_start":"183'2600","created":165,"last_epoch_clean":166,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:07:52.307205+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:07:52.307205+0000","last_clean_scrub_stamp":"2024-09-18T08:07:52.307205+0000","objects_scrubbed":0,"log_size":5377,"log_dups_size":0,"ondisk_log_size":5377,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":298516480,"num_objects":4555,"num_object_clones":0,"num_object_copies":13665,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4555,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7977,"num_write_kb":401024,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,7],"acting":[0,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.a","version":"69'19","reported_seq":353,"reported_epoch":215,"state":"active+clean","last_fresh":"2024-09-18T08:09:51.589409+0000","last_change":"2024-09-18T08:07:41.000210+0000","last_active":"2024-09-18T08:09:51.589409+0000","last_peered":"2024-09-18T08:09:51.589409+0000","last_clean":"2024-09-18T08:09:51.589409+0000","last_became_active":"2024-09-18T08:06:24.139909+0000","last_became_peered":"2024-09-18T08:06:24.139909+0000","last_unstale":"2024-09-18T08:09:51.589409+0000","last_undegraded":"2024-09-18T08:09:51.589409+0000","last_fullsized":"2024-09-18T08:09:51.589409+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'19","last_scrub_stamp":"2024-09-18T08:07:41.000146+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_clean_scrub_stamp":"2024-09-18T08:07:41.000146+0000","objects_scrubbed":9,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:10:18.661451+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":31,"num_read_kb":20,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,7],"acting":[1,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":338,"reported_epoch":215,"state":"active+clean","last_fresh":"2024-09-18T08:09:51.569435+0000","last_change":"2024-09-18T08:06:47.638974+0000","last_active":"2024-09-18T08:09:51.569435+0000","last_peered":"2024-09-18T08:09:51.569435+0000","last_clean":"2024-09-18T08:09:51.569435+0000","last_became_active":"2024-09-18T08:06:46.171298+0000","last_became_peered":"2024-09-18T08:06:46.171298+0000","last_unstale":"2024-09-18T08:09:51.569435+0000","last_undegraded":"2024-09-18T08:09:51.569435+0000","last_fullsized":"2024-09-18T08:09:51.569435+0000","mapping_epoch":143,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":144,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:06:47.638911+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:46.860092+0000","last_clean_scrub_stamp":"2024-09-18T08:06:47.638911+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":44,"reported_epoch":215,"state":"active+clean","last_fresh":"2024-09-18T08:09:51.573100+0000","last_change":"2024-09-18T08:08:52.698243+0000","last_active":"2024-09-18T08:09:51.573100+0000","last_peered":"2024-09-18T08:09:51.573100+0000","last_clean":"2024-09-18T08:09:51.573100+0000","last_became_active":"2024-09-18T08:08:52.697932+0000","last_became_peered":"2024-09-18T08:08:52.697932+0000","last_unstale":"2024-09-18T08:09:51.573100+0000","last_undegraded":"2024-09-18T08:09:51.573100+0000","last_fullsized":"2024-09-18T08:09:51.573100+0000","mapping_epoch":198,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":199,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:08:16.631007+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:50.886830+0000","last_clean_scrub_stamp":"2024-09-18T08:08:16.631007+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":164,"reported_epoch":215,"state":"active+clean","last_fresh":"2024-09-18T08:09:51.573599+0000","last_change":"2024-09-18T08:06:50.902917+0000","last_active":"2024-09-18T08:09:51.573599+0000","last_peered":"2024-09-18T08:09:51.573599+0000","last_clean":"2024-09-18T08:09:51.573599+0000","last_became_active":"2024-09-18T08:06:46.137872+0000","last_became_peered":"2024-09-18T08:06:46.137872+0000","last_unstale":"2024-09-18T08:09:51.573599+0000","last_undegraded":"2024-09-18T08:09:51.573599+0000","last_fullsized":"2024-09-18T08:09:51.573599+0000","mapping_epoch":143,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":144,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:06:50.902835+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_clean_scrub_stamp":"2024-09-18T08:06:50.902835+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,3],"acting":[2,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"8.5","version":"215'7967","reported_seq":12390,"reported_epoch":215,"state":"active+clean","last_fresh":"2024-09-18T08:09:53.180002+0000","last_change":"2024-09-18T08:09:13.170209+0000","last_active":"2024-09-18T08:09:53.180002+0000","last_peered":"2024-09-18T08:09:53.180002+0000","last_clean":"2024-09-18T08:09:53.180002+0000","last_became_active":"2024-09-18T08:09:13.166459+0000","last_became_peered":"2024-09-18T08:09:13.166459+0000","last_unstale":"2024-09-18T08:09:53.180002+0000","last_undegraded":"2024-09-18T08:09:53.180002+0000","last_fullsized":"2024-09-18T08:09:53.180002+0000","mapping_epoch":205,"log_start":"190'3474","ondisk_log_start":"190'3474","created":165,"last_epoch_clean":206,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:07:52.307205+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:07:52.307205+0000","last_clean_scrub_stamp":"2024-09-18T08:07:52.307205+0000","objects_scrubbed":0,"log_size":4493,"log_dups_size":0,"ondisk_log_size":4493,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":302972928,"num_objects":4623,"num_object_clones":0,"num_object_copies":13869,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4623,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7967,"num_write_kb":402880,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2146,"num_bytes_recovered":140640256,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,7],"acting":[4,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.9","version":"69'12","reported_seq":90,"reported_epoch":215,"state":"active+clean","last_fresh":"2024-09-18T08:09:51.557833+0000","last_change":"2024-09-18T08:08:18.599669+0000","last_active":"2024-09-18T08:09:51.557833+0000","last_peered":"2024-09-18T08:09:51.557833+0000","last_clean":"2024-09-18T08:09:51.557833+0000","last_became_active":"2024-09-18T08:08:18.544489+0000","last_became_peered":"2024-09-18T08:08:18.544489+0000","last_unstale":"2024-09-18T08:09:51.557833+0000","last_undegraded":"2024-09-18T08:09:51.557833+0000","last_fullsized":"2024-09-18T08:09:51.557833+0000","mapping_epoch":181,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":182,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'12","last_scrub_stamp":"2024-09-18T08:05:33.544135+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_clean_scrub_stamp":"2024-09-18T08:05:33.544135+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":24,"num_read_kb":15,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":366,"reported_epoch":215,"state":"active+clean","last_fresh":"2024-09-18T08:09:51.571226+0000","last_change":"2024-09-18T08:09:39.614251+0000","last_active":"2024-09-18T08:09:51.571226+0000","last_peered":"2024-09-18T08:09:51.571226+0000","last_clean":"2024-09-18T08:09:51.571226+0000","last_became_active":"2024-09-18T08:08:18.539573+0000","last_became_peered":"2024-09-18T08:08:18.539573+0000","last_unstale":"2024-09-18T08:09:51.571226+0000","last_undegraded":"2024-09-18T08:09:51.571226+0000","last_fullsized":"2024-09-18T08:09:51.571226+0000","mapping_epoch":181,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":182,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:09:39.614187+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:46.860092+0000","last_clean_scrub_stamp":"2024-09-18T08:09:39.614187+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:10:57.546531+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":329,"reported_epoch":215,"state":"active+clean","last_fresh":"2024-09-18T08:09:51.573815+0000","last_change":"2024-09-18T08:08:15.744330+0000","last_active":"2024-09-18T08:09:51.573815+0000","last_peered":"2024-09-18T08:09:51.573815+0000","last_clean":"2024-09-18T08:09:51.573815+0000","last_became_active":"2024-09-18T08:02:51.902163+0000","last_became_peered":"2024-09-18T08:02:51.902163+0000","last_unstale":"2024-09-18T08:09:51.573815+0000","last_undegraded":"2024-09-18T08:09:51.573815+0000","last_fullsized":"2024-09-18T08:09:51.573815+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:08:15.744245+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:50.886830+0000","last_clean_scrub_stamp":"2024-09-18T08:08:15.744245+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":74,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":326,"reported_epoch":215,"state":"active+clean","last_fresh":"2024-09-18T08:09:51.580907+0000","last_change":"2024-09-18T08:07:44.113695+0000","last_active":"2024-09-18T08:09:51.580907+0000","last_peered":"2024-09-18T08:09:51.580907+0000","last_clean":"2024-09-18T08:09:51.580907+0000","last_became_active":"2024-09-18T08:02:53.907412+0000","last_became_peered":"2024-09-18T08:02:53.907412+0000","last_unstale":"2024-09-18T08:09:51.580907+0000","last_undegraded":"2024-09-18T08:09:51.580907+0000","last_fullsized":"2024-09-18T08:09:51.580907+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:07:44.113639+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_clean_scrub_stamp":"2024-09-18T08:07:44.113639+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.4","version":"215'8002","reported_seq":19448,"reported_epoch":215,"state":"active+recovering+undersized+remapped","last_fresh":"2024-09-18T08:09:53.271934+0000","last_change":"2024-09-18T08:08:56.896339+0000","last_active":"2024-09-18T08:09:53.271934+0000","last_peered":"2024-09-18T08:09:53.271934+0000","last_clean":"2024-09-18T08:08:51.672524+0000","last_became_active":"2024-09-18T08:08:53.828580+0000","last_became_peered":"2024-09-18T08:08:53.828580+0000","last_unstale":"2024-09-18T08:09:53.271934+0000","last_undegraded":"2024-09-18T08:09:53.271934+0000","last_fullsized":"2024-09-18T08:08:53.682963+0000","mapping_epoch":199,"log_start":"184'2900","ondisk_log_start":"184'2900","created":165,"last_epoch_clean":190,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:07:52.307205+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:07:52.307205+0000","last_clean_scrub_stamp":"2024-09-18T08:07:52.307205+0000","objects_scrubbed":0,"log_size":5102,"log_dups_size":0,"ondisk_log_size":5102,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":299368448,"num_objects":4568,"num_object_clones":0,"num_object_copies":13704,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1453,"num_objects_unfound":0,"num_objects_dirty":4568,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8002,"num_write_kb":402240,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5654,"num_bytes_recovered":370540544,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,2],"acting":[5,4],"avail_no_missing":["5","3","4","6"],"object_location_counts":[{"shards":"4,5","objects":3115},{"shards":"4,5,6","objects":1453}],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.8","version":"69'15","reported_seq":395,"reported_epoch":215,"state":"active+clean","last_fresh":"2024-09-18T08:09:51.570067+0000","last_change":"2024-09-18T08:06:24.162625+0000","last_active":"2024-09-18T08:09:51.570067+0000","last_peered":"2024-09-18T08:09:51.570067+0000","last_clean":"2024-09-18T08:09:51.570067+0000","last_became_active":"2024-09-18T08:06:24.138259+0000","last_became_peered":"2024-09-18T08:06:24.138259+0000","last_unstale":"2024-09-18T08:09:51.570067+0000","last_undegraded":"2024-09-18T08:09:51.570067+0000","last_fullsized":"2024-09-18T08:09:51.570067+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'15","last_scrub_stamp":"2024-09-18T08:04:26.859044+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_clean_scrub_stamp":"2024-09-18T08:04:26.859044+0000","objects_scrubbed":7,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.f","version":"62'2","reported_seq":382,"reported_epoch":215,"state":"active+clean","last_fresh":"2024-09-18T08:09:51.570918+0000","last_change":"2024-09-18T08:08:18.629506+0000","last_active":"2024-09-18T08:09:51.570918+0000","last_peered":"2024-09-18T08:09:51.570918+0000","last_clean":"2024-09-18T08:09:51.570918+0000","last_became_active":"2024-09-18T08:08:18.524593+0000","last_became_peered":"2024-09-18T08:08:18.524593+0000","last_unstale":"2024-09-18T08:09:51.570918+0000","last_undegraded":"2024-09-18T08:09:51.570918+0000","last_fullsized":"2024-09-18T08:09:51.570918+0000","mapping_epoch":181,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":182,"parent":"0.0","parent_split_bits":0,"last_scrub":"62'2","last_scrub_stamp":"2024-09-18T08:07:43.540222+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:46.860092+0000","last_clean_scrub_stamp":"2024-09-18T08:07:43.540222+0000","objects_scrubbed":2,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":4,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":184,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.9","version":"182'12","reported_seq":486,"reported_epoch":215,"state":"active+clean","last_fresh":"2024-09-18T08:09:51.569431+0000","last_change":"2024-09-18T08:09:46.742515+0000","last_active":"2024-09-18T08:09:51.569431+0000","last_peered":"2024-09-18T08:09:51.569431+0000","last_clean":"2024-09-18T08:09:51.569431+0000","last_became_active":"2024-09-18T08:08:18.524858+0000","last_became_peered":"2024-09-18T08:08:18.524858+0000","last_unstale":"2024-09-18T08:09:51.569431+0000","last_undegraded":"2024-09-18T08:09:51.569431+0000","last_fullsized":"2024-09-18T08:09:51.569431+0000","mapping_epoch":181,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":182,"parent":"0.0","parent_split_bits":0,"last_scrub":"182'12","last_scrub_stamp":"2024-09-18T08:09:46.742442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:50.886830+0000","last_clean_scrub_stamp":"2024-09-18T08:09:46.742442+0000","objects_scrubbed":1,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:11:07.487575+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":338,"reported_epoch":215,"state":"active+clean","last_fresh":"2024-09-18T08:09:51.571883+0000","last_change":"2024-09-18T08:07:45.707631+0000","last_active":"2024-09-18T08:09:51.571883+0000","last_peered":"2024-09-18T08:09:51.571883+0000","last_clean":"2024-09-18T08:09:51.571883+0000","last_became_active":"2024-09-18T08:06:24.133119+0000","last_became_peered":"2024-09-18T08:06:24.133119+0000","last_unstale":"2024-09-18T08:09:51.571883+0000","last_undegraded":"2024-09-18T08:09:51.571883+0000","last_fullsized":"2024-09-18T08:09:51.571883+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:07:45.707562+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_clean_scrub_stamp":"2024-09-18T08:07:45.707562+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":254,"reported_epoch":215,"state":"active+clean","last_fresh":"2024-09-18T08:09:51.564097+0000","last_change":"2024-09-18T08:06:24.133955+0000","last_active":"2024-09-18T08:09:51.564097+0000","last_peered":"2024-09-18T08:09:51.564097+0000","last_clean":"2024-09-18T08:09:51.564097+0000","last_became_active":"2024-09-18T08:06:24.133644+0000","last_became_peered":"2024-09-18T08:06:24.133644+0000","last_unstale":"2024-09-18T08:09:51.564097+0000","last_undegraded":"2024-09-18T08:09:51.564097+0000","last_fullsized":"2024-09-18T08:09:51.564097+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:04:16.378205+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:46.860092+0000","last_clean_scrub_stamp":"2024-09-18T08:04:16.378205+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:10:20.052218+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.17","version":"69'6","reported_seq":349,"reported_epoch":215,"state":"active+clean","last_fresh":"2024-09-18T08:09:51.583469+0000","last_change":"2024-09-18T08:07:19.309941+0000","last_active":"2024-09-18T08:09:51.583469+0000","last_peered":"2024-09-18T08:09:51.583469+0000","last_clean":"2024-09-18T08:09:51.583469+0000","last_became_active":"2024-09-18T08:02:49.895067+0000","last_became_peered":"2024-09-18T08:02:49.895067+0000","last_unstale":"2024-09-18T08:09:51.583469+0000","last_undegraded":"2024-09-18T08:09:51.583469+0000","last_fullsized":"2024-09-18T08:09:51.583469+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'6","last_scrub_stamp":"2024-09-18T08:07:19.309844+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_clean_scrub_stamp":"2024-09-18T08:07:19.309844+0000","objects_scrubbed":3,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":26,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":9,"num_read_kb":6,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":327,"reported_epoch":215,"state":"active+clean","last_fresh":"2024-09-18T08:09:51.569024+0000","last_change":"2024-09-18T08:05:45.812436+0000","last_active":"2024-09-18T08:09:51.569024+0000","last_peered":"2024-09-18T08:09:51.569024+0000","last_clean":"2024-09-18T08:09:51.569024+0000","last_became_active":"2024-09-18T08:02:51.912557+0000","last_became_peered":"2024-09-18T08:02:51.912557+0000","last_unstale":"2024-09-18T08:09:51.569024+0000","last_undegraded":"2024-09-18T08:09:51.569024+0000","last_fullsized":"2024-09-18T08:09:51.569024+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:05:45.812389+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:50.886830+0000","last_clean_scrub_stamp":"2024-09-18T08:05:45.812389+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,1],"acting":[5,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":360,"reported_epoch":215,"state":"active+clean","last_fresh":"2024-09-18T08:09:51.570598+0000","last_change":"2024-09-18T08:08:18.526113+0000","last_active":"2024-09-18T08:09:51.570598+0000","last_peered":"2024-09-18T08:09:51.570598+0000","last_clean":"2024-09-18T08:09:51.570598+0000","last_became_active":"2024-09-18T08:08:18.525260+0000","last_became_peered":"2024-09-18T08:08:18.525260+0000","last_unstale":"2024-09-18T08:09:51.570598+0000","last_undegraded":"2024-09-18T08:09:51.570598+0000","last_fullsized":"2024-09-18T08:09:51.570598+0000","mapping_epoch":181,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":182,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:07:53.670107+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_clean_scrub_stamp":"2024-09-18T08:07:53.670107+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:10:19.732636+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"69'9","reported_seq":355,"reported_epoch":215,"state":"active+clean","last_fresh":"2024-09-18T08:09:51.577745+0000","last_change":"2024-09-18T08:09:44.116395+0000","last_active":"2024-09-18T08:09:51.577745+0000","last_peered":"2024-09-18T08:09:51.577745+0000","last_clean":"2024-09-18T08:09:51.577745+0000","last_became_active":"2024-09-18T08:02:49.891567+0000","last_became_peered":"2024-09-18T08:02:49.891567+0000","last_unstale":"2024-09-18T08:09:51.577745+0000","last_undegraded":"2024-09-18T08:09:51.577745+0000","last_fullsized":"2024-09-18T08:09:51.577745+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'9","last_scrub_stamp":"2024-09-18T08:09:44.116308+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_clean_scrub_stamp":"2024-09-18T08:09:44.116308+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:11:07.916395+0000","scrub_duration":43,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":78,"reported_epoch":215,"state":"active+clean","last_fresh":"2024-09-18T08:09:51.557956+0000","last_change":"2024-09-18T08:08:18.550531+0000","last_active":"2024-09-18T08:09:51.557956+0000","last_peered":"2024-09-18T08:09:51.557956+0000","last_clean":"2024-09-18T08:09:51.557956+0000","last_became_active":"2024-09-18T08:08:18.545953+0000","last_became_peered":"2024-09-18T08:08:18.545953+0000","last_unstale":"2024-09-18T08:09:51.557956+0000","last_undegraded":"2024-09-18T08:09:51.557956+0000","last_fullsized":"2024-09-18T08:09:51.557956+0000","mapping_epoch":181,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":182,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:06:13.214705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:46.860092+0000","last_clean_scrub_stamp":"2024-09-18T08:06:13.214705+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":328,"reported_epoch":215,"state":"active+clean","last_fresh":"2024-09-18T08:09:51.583255+0000","last_change":"2024-09-18T08:09:46.560057+0000","last_active":"2024-09-18T08:09:51.583255+0000","last_peered":"2024-09-18T08:09:51.583255+0000","last_clean":"2024-09-18T08:09:51.583255+0000","last_became_active":"2024-09-18T08:02:51.907361+0000","last_became_peered":"2024-09-18T08:02:51.907361+0000","last_unstale":"2024-09-18T08:09:51.583255+0000","last_undegraded":"2024-09-18T08:09:51.583255+0000","last_fullsized":"2024-09-18T08:09:51.583255+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:09:46.557349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:50.886830+0000","last_clean_scrub_stamp":"2024-09-18T08:09:46.557349+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:11:08.318311+0000","scrub_duration":22,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":339,"reported_epoch":215,"state":"active+clean","last_fresh":"2024-09-18T08:09:51.579990+0000","last_change":"2024-09-18T08:08:18.511719+0000","last_active":"2024-09-18T08:09:51.579990+0000","last_peered":"2024-09-18T08:09:51.579990+0000","last_clean":"2024-09-18T08:09:51.579990+0000","last_became_active":"2024-09-18T08:08:18.511019+0000","last_became_peered":"2024-09-18T08:08:18.511019+0000","last_unstale":"2024-09-18T08:09:51.579990+0000","last_undegraded":"2024-09-18T08:09:51.579990+0000","last_fullsized":"2024-09-18T08:09:51.579990+0000","mapping_epoch":181,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":182,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:08:10.660508+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_clean_scrub_stamp":"2024-09-18T08:08:10.660508+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.15","version":"69'9","reported_seq":347,"reported_epoch":215,"state":"active+clean","last_fresh":"2024-09-18T08:09:51.569550+0000","last_change":"2024-09-18T08:05:36.896299+0000","last_active":"2024-09-18T08:09:51.569550+0000","last_peered":"2024-09-18T08:09:51.569550+0000","last_clean":"2024-09-18T08:09:51.569550+0000","last_became_active":"2024-09-18T08:02:49.890497+0000","last_became_peered":"2024-09-18T08:02:49.890497+0000","last_unstale":"2024-09-18T08:09:51.569550+0000","last_undegraded":"2024-09-18T08:09:51.569550+0000","last_fullsized":"2024-09-18T08:09:51.569550+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'9","last_scrub_stamp":"2024-09-18T08:05:36.896238+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_clean_scrub_stamp":"2024-09-18T08:05:36.896238+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":352,"reported_epoch":215,"state":"active+clean","last_fresh":"2024-09-18T08:09:51.572748+0000","last_change":"2024-09-18T08:09:46.035501+0000","last_active":"2024-09-18T08:09:51.572748+0000","last_peered":"2024-09-18T08:09:51.572748+0000","last_clean":"2024-09-18T08:09:51.572748+0000","last_became_active":"2024-09-18T08:02:47.875983+0000","last_became_peered":"2024-09-18T08:02:47.875983+0000","last_unstale":"2024-09-18T08:09:51.572748+0000","last_undegraded":"2024-09-18T08:09:51.572748+0000","last_fullsized":"2024-09-18T08:09:51.572748+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:09:46.035332+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:46.860092+0000","last_clean_scrub_stamp":"2024-09-18T08:09:46.035332+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:10:47.519129+0000","scrub_duration":18,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.14","version":"199'12","reported_seq":447,"reported_epoch":215,"state":"active+clean","last_fresh":"2024-09-18T08:09:53.016185+0000","last_change":"2024-09-18T08:08:52.701287+0000","last_active":"2024-09-18T08:09:53.016185+0000","last_peered":"2024-09-18T08:09:53.016185+0000","last_clean":"2024-09-18T08:09:53.016185+0000","last_became_active":"2024-09-18T08:08:52.696201+0000","last_became_peered":"2024-09-18T08:08:52.696201+0000","last_unstale":"2024-09-18T08:09:53.016185+0000","last_undegraded":"2024-09-18T08:09:53.016185+0000","last_fullsized":"2024-09-18T08:09:53.016185+0000","mapping_epoch":198,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":199,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'8","last_scrub_stamp":"2024-09-18T08:07:16.241607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:50.886830+0000","last_clean_scrub_stamp":"2024-09-18T08:07:16.241607+0000","objects_scrubbed":1,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":78,"reported_epoch":215,"state":"active+clean","last_fresh":"2024-09-18T08:09:51.558869+0000","last_change":"2024-09-18T08:08:18.523205+0000","last_active":"2024-09-18T08:09:51.558869+0000","last_peered":"2024-09-18T08:09:51.558869+0000","last_clean":"2024-09-18T08:09:51.558869+0000","last_became_active":"2024-09-18T08:08:18.522483+0000","last_became_peered":"2024-09-18T08:08:18.522483+0000","last_unstale":"2024-09-18T08:09:51.558869+0000","last_undegraded":"2024-09-18T08:09:51.558869+0000","last_fullsized":"2024-09-18T08:09:51.558869+0000","mapping_epoch":181,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":182,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:06:43.510513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_clean_scrub_stamp":"2024-09-18T08:06:43.510513+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.14","version":"69'10","reported_seq":367,"reported_epoch":215,"state":"active+clean","last_fresh":"2024-09-18T08:09:51.582490+0000","last_change":"2024-09-18T08:08:08.493614+0000","last_active":"2024-09-18T08:09:51.582490+0000","last_peered":"2024-09-18T08:09:51.582490+0000","last_clean":"2024-09-18T08:09:51.582490+0000","last_became_active":"2024-09-18T08:04:18.150968+0000","last_became_peered":"2024-09-18T08:04:18.150968+0000","last_unstale":"2024-09-18T08:09:51.582490+0000","last_undegraded":"2024-09-18T08:09:51.582490+0000","last_fullsized":"2024-09-18T08:09:51.582490+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'10","last_scrub_stamp":"2024-09-18T08:08:08.493522+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_clean_scrub_stamp":"2024-09-18T08:08:08.493522+0000","objects_scrubbed":5,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,7],"acting":[3,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":358,"reported_epoch":215,"state":"active+clean","last_fresh":"2024-09-18T08:09:51.570355+0000","last_change":"2024-09-18T08:09:31.658337+0000","last_active":"2024-09-18T08:09:51.570355+0000","last_peered":"2024-09-18T08:09:51.570355+0000","last_clean":"2024-09-18T08:09:51.570355+0000","last_became_active":"2024-09-18T08:08:18.524218+0000","last_became_peered":"2024-09-18T08:08:18.524218+0000","last_unstale":"2024-09-18T08:09:51.570355+0000","last_undegraded":"2024-09-18T08:09:51.570355+0000","last_fullsized":"2024-09-18T08:09:51.570355+0000","mapping_epoch":181,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":182,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:09:31.658259+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:46.860092+0000","last_clean_scrub_stamp":"2024-09-18T08:09:31.658259+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:10:50.143994+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.15","version":"69'8","reported_seq":429,"reported_epoch":215,"state":"active+clean","last_fresh":"2024-09-18T08:09:51.569324+0000","last_change":"2024-09-18T08:06:45.555513+0000","last_active":"2024-09-18T08:09:51.569324+0000","last_peered":"2024-09-18T08:09:51.569324+0000","last_clean":"2024-09-18T08:09:51.569324+0000","last_became_active":"2024-09-18T08:02:51.914135+0000","last_became_peered":"2024-09-18T08:02:51.914135+0000","last_unstale":"2024-09-18T08:09:51.569324+0000","last_undegraded":"2024-09-18T08:09:51.569324+0000","last_fullsized":"2024-09-18T08:09:51.569324+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'8","last_scrub_stamp":"2024-09-18T08:06:45.555459+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:50.886830+0000","last_clean_scrub_stamp":"2024-09-18T08:06:45.555459+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":332,"reported_epoch":215,"state":"active+clean","last_fresh":"2024-09-18T08:09:51.573302+0000","last_change":"2024-09-18T08:09:42.047337+0000","last_active":"2024-09-18T08:09:51.573302+0000","last_peered":"2024-09-18T08:09:51.573302+0000","last_clean":"2024-09-18T08:09:51.573302+0000","last_became_active":"2024-09-18T08:02:53.912112+0000","last_became_peered":"2024-09-18T08:02:53.912112+0000","last_unstale":"2024-09-18T08:09:51.573302+0000","last_undegraded":"2024-09-18T08:09:51.573302+0000","last_fullsized":"2024-09-18T08:09:51.573302+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:09:42.047270+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_clean_scrub_stamp":"2024-09-18T08:09:42.047270+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:10:59.660034+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.13","version":"69'11","reported_seq":90,"reported_epoch":215,"state":"active+clean","last_fresh":"2024-09-18T08:09:51.558237+0000","last_change":"2024-09-18T08:08:18.643772+0000","last_active":"2024-09-18T08:09:51.558237+0000","last_peered":"2024-09-18T08:09:51.558237+0000","last_clean":"2024-09-18T08:09:51.558237+0000","last_became_active":"2024-09-18T08:08:18.539712+0000","last_became_peered":"2024-09-18T08:08:18.539712+0000","last_unstale":"2024-09-18T08:09:51.558237+0000","last_undegraded":"2024-09-18T08:09:51.558237+0000","last_fullsized":"2024-09-18T08:09:51.558237+0000","mapping_epoch":181,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":182,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'11","last_scrub_stamp":"2024-09-18T08:07:53.824401+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_clean_scrub_stamp":"2024-09-18T08:07:53.824401+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":19,"num_read_kb":12,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":78,"reported_epoch":215,"state":"active+clean","last_fresh":"2024-09-18T08:09:51.558840+0000","last_change":"2024-09-18T08:08:18.623643+0000","last_active":"2024-09-18T08:09:51.558840+0000","last_peered":"2024-09-18T08:09:51.558840+0000","last_clean":"2024-09-18T08:09:51.558840+0000","last_became_active":"2024-09-18T08:08:18.622893+0000","last_became_peered":"2024-09-18T08:08:18.622893+0000","last_unstale":"2024-09-18T08:09:51.558840+0000","last_undegraded":"2024-09-18T08:09:51.558840+0000","last_fullsized":"2024-09-18T08:09:51.558840+0000","mapping_epoch":181,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":182,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:06:46.502422+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:46.860092+0000","last_clean_scrub_stamp":"2024-09-18T08:06:46.502422+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":336,"reported_epoch":215,"state":"active+clean","last_fresh":"2024-09-18T08:09:51.589252+0000","last_change":"2024-09-18T08:09:03.582984+0000","last_active":"2024-09-18T08:09:51.589252+0000","last_peered":"2024-09-18T08:09:51.589252+0000","last_clean":"2024-09-18T08:09:51.589252+0000","last_became_active":"2024-09-18T08:02:51.910752+0000","last_became_peered":"2024-09-18T08:02:51.910752+0000","last_unstale":"2024-09-18T08:09:51.589252+0000","last_undegraded":"2024-09-18T08:09:51.589252+0000","last_fullsized":"2024-09-18T08:09:51.589252+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:09:03.582914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:50.886830+0000","last_clean_scrub_stamp":"2024-09-18T08:09:03.582914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:10:14.069633+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":333,"reported_epoch":215,"state":"active+clean","last_fresh":"2024-09-18T08:09:51.580743+0000","last_change":"2024-09-18T08:07:54.314026+0000","last_active":"2024-09-18T08:09:51.580743+0000","last_peered":"2024-09-18T08:09:51.580743+0000","last_clean":"2024-09-18T08:09:51.580743+0000","last_became_active":"2024-09-18T08:02:53.911429+0000","last_became_peered":"2024-09-18T08:02:53.911429+0000","last_unstale":"2024-09-18T08:09:51.580743+0000","last_undegraded":"2024-09-18T08:09:51.580743+0000","last_fullsized":"2024-09-18T08:09:51.580743+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:07:54.313971+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_clean_scrub_stamp":"2024-09-18T08:07:54.313971+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.12","version":"69'9","reported_seq":374,"reported_epoch":215,"state":"active+clean","last_fresh":"2024-09-18T08:09:51.589636+0000","last_change":"2024-09-18T08:08:52.658353+0000","last_active":"2024-09-18T08:09:51.589636+0000","last_peered":"2024-09-18T08:09:51.589636+0000","last_clean":"2024-09-18T08:09:51.589636+0000","last_became_active":"2024-09-18T08:06:24.132270+0000","last_became_peered":"2024-09-18T08:06:24.132270+0000","last_unstale":"2024-09-18T08:09:51.589636+0000","last_undegraded":"2024-09-18T08:09:51.589636+0000","last_fullsized":"2024-09-18T08:09:51.589636+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'9","last_scrub_stamp":"2024-09-18T08:08:52.658282+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_clean_scrub_stamp":"2024-09-18T08:08:52.658282+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:10:04.461414+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":365,"reported_epoch":215,"state":"active+clean","last_fresh":"2024-09-18T08:09:51.571298+0000","last_change":"2024-09-18T08:09:37.532792+0000","last_active":"2024-09-18T08:09:51.571298+0000","last_peered":"2024-09-18T08:09:51.571298+0000","last_clean":"2024-09-18T08:09:51.571298+0000","last_became_active":"2024-09-18T08:08:18.525527+0000","last_became_peered":"2024-09-18T08:08:18.525527+0000","last_unstale":"2024-09-18T08:09:51.571298+0000","last_undegraded":"2024-09-18T08:09:51.571298+0000","last_fullsized":"2024-09-18T08:09:51.571298+0000","mapping_epoch":181,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":182,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:09:37.532573+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:46.860092+0000","last_clean_scrub_stamp":"2024-09-18T08:09:37.532573+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:11:05.489513+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":329,"reported_epoch":215,"state":"active+clean","last_fresh":"2024-09-18T08:09:51.582656+0000","last_change":"2024-09-18T08:07:42.165434+0000","last_active":"2024-09-18T08:09:51.582656+0000","last_peered":"2024-09-18T08:09:51.582656+0000","last_clean":"2024-09-18T08:09:51.582656+0000","last_became_active":"2024-09-18T08:02:51.906277+0000","last_became_peered":"2024-09-18T08:02:51.906277+0000","last_unstale":"2024-09-18T08:09:51.582656+0000","last_undegraded":"2024-09-18T08:09:51.582656+0000","last_fullsized":"2024-09-18T08:09:51.582656+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:07:42.165388+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:50.886830+0000","last_clean_scrub_stamp":"2024-09-18T08:07:42.165388+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.10","version":"69'1","reported_seq":313,"reported_epoch":215,"state":"active+clean","last_fresh":"2024-09-18T08:09:51.563725+0000","last_change":"2024-09-18T08:05:51.429631+0000","last_active":"2024-09-18T08:09:51.563725+0000","last_peered":"2024-09-18T08:09:51.563725+0000","last_clean":"2024-09-18T08:09:51.563725+0000","last_became_active":"2024-09-18T08:02:53.912699+0000","last_became_peered":"2024-09-18T08:02:53.912699+0000","last_unstale":"2024-09-18T08:09:51.563725+0000","last_undegraded":"2024-09-18T08:09:51.563725+0000","last_fullsized":"2024-09-18T08:09:51.563725+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_clean_scrub_stamp":"2024-09-18T08:02:52.890475+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:10:16.084523+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.11","version":"69'11","reported_seq":398,"reported_epoch":215,"state":"active+clean","last_fresh":"2024-09-18T08:09:51.580996+0000","last_change":"2024-09-18T08:07:18.282247+0000","last_active":"2024-09-18T08:09:51.580996+0000","last_peered":"2024-09-18T08:09:51.580996+0000","last_clean":"2024-09-18T08:09:51.580996+0000","last_became_active":"2024-09-18T08:06:24.132416+0000","last_became_peered":"2024-09-18T08:06:24.132416+0000","last_unstale":"2024-09-18T08:09:51.580996+0000","last_undegraded":"2024-09-18T08:09:51.580996+0000","last_fullsized":"2024-09-18T08:09:51.580996+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'11","last_scrub_stamp":"2024-09-18T08:07:18.282155+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_clean_scrub_stamp":"2024-09-18T08:07:18.282155+0000","objects_scrubbed":5,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":335,"reported_epoch":215,"state":"active+clean","last_fresh":"2024-09-18T08:09:51.569112+0000","last_change":"2024-09-18T08:06:18.168254+0000","last_active":"2024-09-18T08:09:51.569112+0000","last_peered":"2024-09-18T08:09:51.569112+0000","last_clean":"2024-09-18T08:09:51.569112+0000","last_became_active":"2024-09-18T08:02:47.878886+0000","last_became_peered":"2024-09-18T08:02:47.878886+0000","last_unstale":"2024-09-18T08:09:51.569112+0000","last_undegraded":"2024-09-18T08:09:51.569112+0000","last_fullsized":"2024-09-18T08:09:51.569112+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:06:18.168206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:46.860092+0000","last_clean_scrub_stamp":"2024-09-18T08:06:18.168206+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":372,"reported_epoch":215,"state":"active+clean","last_fresh":"2024-09-18T08:09:51.569868+0000","last_change":"2024-09-18T08:09:49.824264+0000","last_active":"2024-09-18T08:09:51.569868+0000","last_peered":"2024-09-18T08:09:51.569868+0000","last_clean":"2024-09-18T08:09:51.569868+0000","last_became_active":"2024-09-18T08:08:18.524966+0000","last_became_peered":"2024-09-18T08:08:18.524966+0000","last_unstale":"2024-09-18T08:09:51.569868+0000","last_undegraded":"2024-09-18T08:09:51.569868+0000","last_fullsized":"2024-09-18T08:09:51.569868+0000","mapping_epoch":181,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":182,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:09:49.824196+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:50.886830+0000","last_clean_scrub_stamp":"2024-09-18T08:09:49.824196+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:10:51.928894+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":338,"reported_epoch":215,"state":"active+clean","last_fresh":"2024-09-18T08:09:51.582762+0000","last_change":"2024-09-18T08:07:48.218010+0000","last_active":"2024-09-18T08:09:51.582762+0000","last_peered":"2024-09-18T08:09:51.582762+0000","last_clean":"2024-09-18T08:09:51.582762+0000","last_became_active":"2024-09-18T08:06:24.128582+0000","last_became_peered":"2024-09-18T08:06:24.128582+0000","last_unstale":"2024-09-18T08:09:51.582762+0000","last_undegraded":"2024-09-18T08:09:51.582762+0000","last_fullsized":"2024-09-18T08:09:51.582762+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:07:48.217964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_clean_scrub_stamp":"2024-09-18T08:07:48.217964+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.10","version":"69'4","reported_seq":365,"reported_epoch":215,"state":"active+clean","last_fresh":"2024-09-18T08:09:51.584306+0000","last_change":"2024-09-18T08:07:41.210418+0000","last_active":"2024-09-18T08:09:51.584306+0000","last_peered":"2024-09-18T08:09:51.584306+0000","last_clean":"2024-09-18T08:09:51.584306+0000","last_became_active":"2024-09-18T08:06:24.131309+0000","last_became_peered":"2024-09-18T08:06:24.131309+0000","last_unstale":"2024-09-18T08:09:51.584306+0000","last_undegraded":"2024-09-18T08:09:51.584306+0000","last_fullsized":"2024-09-18T08:09:51.584306+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'4","last_scrub_stamp":"2024-09-18T08:07:41.210347+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_clean_scrub_stamp":"2024-09-18T08:07:41.210347+0000","objects_scrubbed":2,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":6,"num_read_kb":4,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":343,"reported_epoch":215,"state":"active+clean","last_fresh":"2024-09-18T08:09:51.569370+0000","last_change":"2024-09-18T08:07:52.551689+0000","last_active":"2024-09-18T08:09:51.569370+0000","last_peered":"2024-09-18T08:09:51.569370+0000","last_clean":"2024-09-18T08:09:51.569370+0000","last_became_active":"2024-09-18T08:02:47.877616+0000","last_became_peered":"2024-09-18T08:02:47.877616+0000","last_unstale":"2024-09-18T08:09:51.569370+0000","last_undegraded":"2024-09-18T08:09:51.569370+0000","last_fullsized":"2024-09-18T08:09:51.569370+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:07:52.551616+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:46.860092+0000","last_clean_scrub_stamp":"2024-09-18T08:07:52.551616+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":78,"reported_epoch":215,"state":"active+clean","last_fresh":"2024-09-18T08:09:51.558050+0000","last_change":"2024-09-18T08:08:18.550244+0000","last_active":"2024-09-18T08:09:51.558050+0000","last_peered":"2024-09-18T08:09:51.558050+0000","last_clean":"2024-09-18T08:09:51.558050+0000","last_became_active":"2024-09-18T08:08:18.545361+0000","last_became_peered":"2024-09-18T08:08:18.545361+0000","last_unstale":"2024-09-18T08:09:51.558050+0000","last_undegraded":"2024-09-18T08:09:51.558050+0000","last_fullsized":"2024-09-18T08:09:51.558050+0000","mapping_epoch":181,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":182,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:07:17.605562+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:50.886830+0000","last_clean_scrub_stamp":"2024-09-18T08:07:17.605562+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,7],"acting":[4,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":354,"reported_epoch":215,"state":"active+clean","last_fresh":"2024-09-18T08:09:51.569764+0000","last_change":"2024-09-18T08:09:34.607443+0000","last_active":"2024-09-18T08:09:51.569764+0000","last_peered":"2024-09-18T08:09:51.569764+0000","last_clean":"2024-09-18T08:09:51.569764+0000","last_became_active":"2024-09-18T08:08:18.525552+0000","last_became_peered":"2024-09-18T08:08:18.525552+0000","last_unstale":"2024-09-18T08:09:51.569764+0000","last_undegraded":"2024-09-18T08:09:51.569764+0000","last_fullsized":"2024-09-18T08:09:51.569764+0000","mapping_epoch":181,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":182,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:09:34.607374+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_clean_scrub_stamp":"2024-09-18T08:09:34.607374+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:10:39.783401+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,4],"acting":[7,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":322,"reported_epoch":215,"state":"active+clean","last_fresh":"2024-09-18T08:09:51.589500+0000","last_change":"2024-09-18T08:08:18.539359+0000","last_active":"2024-09-18T08:09:51.589500+0000","last_peered":"2024-09-18T08:09:51.589500+0000","last_clean":"2024-09-18T08:09:51.589500+0000","last_became_active":"2024-09-18T08:08:18.538838+0000","last_became_peered":"2024-09-18T08:08:18.538838+0000","last_unstale":"2024-09-18T08:09:51.589500+0000","last_undegraded":"2024-09-18T08:09:51.589500+0000","last_fullsized":"2024-09-18T08:09:51.589500+0000","mapping_epoch":181,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":182,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_clean_scrub_stamp":"2024-09-18T08:02:52.890475+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":336,"reported_epoch":215,"state":"active+clean","last_fresh":"2024-09-18T08:09:51.584565+0000","last_change":"2024-09-18T08:07:17.277964+0000","last_active":"2024-09-18T08:09:51.584565+0000","last_peered":"2024-09-18T08:09:51.584565+0000","last_clean":"2024-09-18T08:09:51.584565+0000","last_became_active":"2024-09-18T08:02:47.878922+0000","last_became_peered":"2024-09-18T08:02:47.878922+0000","last_unstale":"2024-09-18T08:09:51.584565+0000","last_undegraded":"2024-09-18T08:09:51.584565+0000","last_fullsized":"2024-09-18T08:09:51.584565+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:07:17.277894+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:46.860092+0000","last_clean_scrub_stamp":"2024-09-18T08:07:17.277894+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":16,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"69'11","reported_seq":325,"reported_epoch":215,"state":"active+clean","last_fresh":"2024-09-18T08:09:51.569678+0000","last_change":"2024-09-18T08:06:24.164086+0000","last_active":"2024-09-18T08:09:51.569678+0000","last_peered":"2024-09-18T08:09:51.569678+0000","last_clean":"2024-09-18T08:09:51.569678+0000","last_became_active":"2024-09-18T08:06:24.135232+0000","last_became_peered":"2024-09-18T08:06:24.135232+0000","last_unstale":"2024-09-18T08:09:51.569678+0000","last_undegraded":"2024-09-18T08:09:51.569678+0000","last_fullsized":"2024-09-18T08:09:51.569678+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'11","last_scrub_stamp":"2024-09-18T08:06:12.967462+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_clean_scrub_stamp":"2024-09-18T08:06:12.967462+0000","objects_scrubbed":5,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":19,"num_read_kb":12,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,1],"acting":[5,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":342,"reported_epoch":215,"state":"active+clean","last_fresh":"2024-09-18T08:09:51.573553+0000","last_change":"2024-09-18T08:08:52.701053+0000","last_active":"2024-09-18T08:09:51.573553+0000","last_peered":"2024-09-18T08:09:51.573553+0000","last_clean":"2024-09-18T08:09:51.573553+0000","last_became_active":"2024-09-18T08:08:52.700595+0000","last_became_peered":"2024-09-18T08:08:52.700595+0000","last_unstale":"2024-09-18T08:09:51.573553+0000","last_undegraded":"2024-09-18T08:09:51.573553+0000","last_fullsized":"2024-09-18T08:09:51.573553+0000","mapping_epoch":198,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":199,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:08:12.639376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:50.886830+0000","last_clean_scrub_stamp":"2024-09-18T08:08:12.639376+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":8,"num_pg":16,"stat_sum":{"num_bytes":4826988568,"num_objects":73655,"num_object_clones":0,"num_object_copies":220965,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1453,"num_objects_unfound":0,"num_objects_dirty":73655,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":127845,"num_write_kb":6447937,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":22791,"num_bytes_recovered":1493630976,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":15512055808,"data_stored":15512043592,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":78071,"ondisk_log_size":78071,"up":48,"acting":47,"num_store_stats":8},{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":416,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":85,"ondisk_log_size":85,"up":96,"acting":96,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":178,"num_object_clones":0,"num_object_copies":534,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":178,"num_whiteouts":0,"num_read":685,"num_read_kb":442,"num_write":417,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":214,"num_bytes_recovered":4114,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":417792,"data_stored":11106,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":398,"ondisk_log_size":398,"up":96,"acting":96,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":1613,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":24,"num_read_kb":24,"num_write":10,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":3226,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":4839,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":6,"ondisk_log_size":6,"up":96,"acting":96,"num_store_stats":8},{"poolid":2,"num_pg":3,"stat_sum":{"num_bytes":408,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":193,"num_read_kb":188,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":38,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":8,"ondisk_log_size":8,"up":9,"acting":9,"num_store_stats":7},{"poolid":1,"num_pg":11,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":869,"ondisk_log_size":869,"up":33,"acting":33,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":56,"seq":240518168665,"num_pgs":63,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1714864,"kb_used_data":1539100,"kb_used_omap":1,"kb_used_meta":175550,"kb_avail":92030288,"statfs":{"total":95995035648,"available":94239014912,"internally_reserved":0,"allocated":1576038400,"data_stored":1575645972,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":179763658},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1],"upper_bound":4},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":47,"seq":201863463007,"num_pgs":55,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1991768,"kb_used_data":1807876,"kb_used_omap":1,"kb_used_meta":183870,"kb_avail":91753384,"statfs":{"total":95995035648,"available":93955465216,"internally_reserved":0,"allocated":1851265024,"data_stored":1850896063,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":188283334},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":39,"seq":167503724645,"num_pgs":59,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2523692,"kb_used_data":2370328,"kb_used_omap":1,"kb_used_meta":153342,"kb_avail":91221460,"statfs":{"total":95995035648,"available":93410775040,"internally_reserved":0,"allocated":2427215872,"data_stored":2426827807,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":157022662},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":32,"seq":137438953578,"num_pgs":67,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2086324,"kb_used_data":1860772,"kb_used_omap":1,"kb_used_meta":225534,"kb_avail":91658828,"statfs":{"total":95995035648,"available":93858639872,"internally_reserved":0,"allocated":1905430528,"data_stored":1905030397,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":230947269},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1],"upper_bound":2},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":26,"seq":111669149809,"num_pgs":69,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2155712,"kb_used_data":1849644,"kb_used_omap":1,"kb_used_meta":306046,"kb_avail":91589440,"statfs":{"total":95995035648,"available":93787586560,"internally_reserved":0,"allocated":1894035456,"data_stored":1893626970,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":313391560},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,2,3],"upper_bound":32},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411447,"num_pgs":49,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1985960,"kb_used_data":1697944,"kb_used_omap":1,"kb_used_meta":287998,"kb_avail":91759192,"statfs":{"total":95995035648,"available":93961412608,"internally_reserved":0,"allocated":1738694656,"data_stored":1738306697,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":294910405},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574971,"num_pgs":63,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2335892,"kb_used_data":2083972,"kb_used_omap":1,"kb_used_meta":251902,"kb_avail":91409260,"statfs":{"total":95995035648,"available":93603082240,"internally_reserved":0,"allocated":2133987328,"data_stored":2133555728,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":257948101},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738498,"num_pgs":56,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2302712,"kb_used_data":2050788,"kb_used_omap":1,"kb_used_meta":251902,"kb_avail":91442440,"statfs":{"total":95995035648,"available":93637058560,"internally_reserved":0,"allocated":2100006912,"data_stored":2099606983,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":257948101},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":266240,"data_stored":262688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1131,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1521,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":436,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":46,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1177,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":436,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":77824,"data_stored":2008,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1128,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":1898,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1474,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":990,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1034,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1254,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":403,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":416,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":403,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":2068774912,"data_stored":2068774912,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":2129854464,"data_stored":2129854464,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":1733038080,"data_stored":1733034008,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":1881931776,"data_stored":1881931776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":1901723648,"data_stored":1901723648,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":2413301760,"data_stored":2413297688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":1822162944,"data_stored":1822162944,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":1561268224,"data_stored":1561264152,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T08:09:56.441 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5 2024-09-18T08:09:56.476 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:09:56.476 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:09:56.476 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5' 2024-09-18T08:09:56.508 INFO:teuthology.orchestra.run.smithi050.stdout:ERROR: (22) Invalid argument 2024-09-18T08:09:56.509 INFO:teuthology.orchestra.run.smithi050.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-18T08:09:56.577 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1 2024-09-18T08:09:56.621 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:09:56.622 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:09:56.622 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1' 2024-09-18T08:09:56.698 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:09:56 smithi007 ceph-mon[29459]: pgmap v344: 158 pgs: 1 active+recovering+undersized+remapped, 157 active+clean; 4.5 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.13k op/s; 1453/221562 objects misplaced (0.656%); 2.9 MiB/s, 46 objects/s recovering 2024-09-18T08:09:56.698 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:09:56 smithi007 ceph-mon[29459]: from='client.25226 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-18T08:09:56.699 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:09:56 smithi007 ceph-mon[39249]: pgmap v344: 158 pgs: 1 active+recovering+undersized+remapped, 157 active+clean; 4.5 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.13k op/s; 1453/221562 objects misplaced (0.656%); 2.9 MiB/s, 46 objects/s recovering 2024-09-18T08:09:56.699 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:09:56 smithi007 ceph-mon[39249]: from='client.25226 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-18T08:09:56.723 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4 2024-09-18T08:09:56.752 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:09:56.752 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:09:56.753 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4' 2024-09-18T08:09:56.854 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7 2024-09-18T08:09:56.896 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:09:56.897 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:09:56.897 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7' 2024-09-18T08:09:56.905 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:09:56 smithi050 ceph-mon[32371]: pgmap v344: 158 pgs: 1 active+recovering+undersized+remapped, 157 active+clean; 4.5 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.13k op/s; 1453/221562 objects misplaced (0.656%); 2.9 MiB/s, 46 objects/s recovering 2024-09-18T08:09:56.906 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:09:56 smithi050 ceph-mon[32371]: from='client.25226 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-18T08:09:56.981 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Sep 18 08:09:56 smithi007 ceph-779349ca-7593-11ef-bceb-c7b262605968-mgr-y[29688]: ::ffff:172.21.15.50 - - [18/Sep/2024:08:09:56] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-09-18T08:09:56.999 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1 2024-09-18T08:09:57.020 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/779349ca-7593-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T08:09:57.047 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:09:57.048 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:09:57.048 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1' 2024-09-18T08:09:57.092 INFO:teuthology.orchestra.run.smithi007.stdout:osd.0: {} 2024-09-18T08:09:57.092 INFO:teuthology.orchestra.run.smithi007.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-18T08:09:57.133 INFO:teuthology.orchestra.run.smithi007.stdout:osd.1: {} 2024-09-18T08:09:57.134 INFO:teuthology.orchestra.run.smithi007.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-18T08:09:57.149 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6 2024-09-18T08:09:57.168 INFO:teuthology.orchestra.run.smithi007.stdout:osd.2: {} 2024-09-18T08:09:57.168 INFO:teuthology.orchestra.run.smithi007.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-18T08:09:57.185 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:09:57.185 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:09:57.186 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6' 2024-09-18T08:09:57.208 INFO:teuthology.orchestra.run.smithi007.stdout:osd.3: {} 2024-09-18T08:09:57.208 INFO:teuthology.orchestra.run.smithi007.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-18T08:09:57.244 INFO:teuthology.orchestra.run.smithi007.stdout:osd.4: {} 2024-09-18T08:09:57.245 INFO:teuthology.orchestra.run.smithi007.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-18T08:09:57.282 INFO:teuthology.orchestra.run.smithi007.stdout:osd.5: {} 2024-09-18T08:09:57.283 INFO:teuthology.orchestra.run.smithi007.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-18T08:09:57.286 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7 2024-09-18T08:09:57.315 INFO:teuthology.orchestra.run.smithi007.stdout:osd.6: {} 2024-09-18T08:09:57.315 INFO:teuthology.orchestra.run.smithi007.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-18T08:09:57.324 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:09:57.325 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:09:57.325 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7' 2024-09-18T08:09:57.340 DEBUG:teuthology.orchestra.run.smithi050:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid 779349ca-7593-11ef-bceb-c7b262605968 -- ceph daemon osd.6 dump_ops_in_flight 2024-09-18T08:09:57.359 INFO:teuthology.orchestra.run.smithi007.stdout:osd.7: {} 2024-09-18T08:09:57.359 INFO:teuthology.orchestra.run.smithi007.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-18T08:09:57.426 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4 2024-09-18T08:09:57.459 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:09:57.459 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:09:57.460 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4' 2024-09-18T08:09:57.560 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0 2024-09-18T08:09:57.603 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:09:57.603 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:09:57.603 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0' 2024-09-18T08:09:57.704 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4 2024-09-18T08:09:57.740 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:09:57.740 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:09:57.741 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4' 2024-09-18T08:09:57.772 INFO:teuthology.orchestra.run.smithi050.stderr:Inferring config /var/lib/ceph/779349ca-7593-11ef-bceb-c7b262605968/mon.b/config 2024-09-18T08:09:57.842 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0 2024-09-18T08:09:57.879 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:09:57.879 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:09:57.880 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0' 2024-09-18T08:09:57.901 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:09:57 smithi050 ceph-mon[32371]: from='client.? 172.21.15.7:0/3079059275' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-18T08:09:57.901 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:09:57 smithi050 ceph-mon[32371]: pgmap v345: 158 pgs: 1 active+recovering+undersized+remapped, 157 active+clean; 4.5 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.13k op/s; 1453/221562 objects misplaced (0.656%); 2.9 MiB/s, 46 objects/s recovering 2024-09-18T08:09:57.953 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:09:57 smithi007 ceph-mon[39249]: from='client.? 172.21.15.7:0/3079059275' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-18T08:09:57.953 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:09:57 smithi007 ceph-mon[39249]: pgmap v345: 158 pgs: 1 active+recovering+undersized+remapped, 157 active+clean; 4.5 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.13k op/s; 1453/221562 objects misplaced (0.656%); 2.9 MiB/s, 46 objects/s recovering 2024-09-18T08:09:57.954 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:09:57 smithi007 ceph-mon[29459]: from='client.? 172.21.15.7:0/3079059275' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-18T08:09:57.954 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:09:57 smithi007 ceph-mon[29459]: pgmap v345: 158 pgs: 1 active+recovering+undersized+remapped, 157 active+clean; 4.5 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.13k op/s; 1453/221562 objects misplaced (0.656%); 2.9 MiB/s, 46 objects/s recovering 2024-09-18T08:09:57.981 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0 2024-09-18T08:09:58.025 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:09:58.025 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:09:58.026 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0' 2024-09-18T08:09:58.127 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5 2024-09-18T08:09:58.168 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:09:58.169 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:09:58.169 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5' 2024-09-18T08:09:58.270 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5 2024-09-18T08:09:58.306 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:09:58.307 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:09:58.308 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5' 2024-09-18T08:09:58.375 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2024-09-18T08:09:58.375 INFO:teuthology.orchestra.run.smithi050.stdout: "ops": [ 2024-09-18T08:09:58.375 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:09:58.375 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_repop(client.15216.0:133477 8.6 e215/165 8:6ea8e347:::benchmark_data_smithi007_101583_object32725:head v 215'8297, mlcod=215'8295)", 2024-09-18T08:09:58.376 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-09-18T08:09:58.371128+0000", 2024-09-18T08:09:58.376 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 0.001447928, 2024-09-18T08:09:58.376 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0014757699999999999, 2024-09-18T08:09:58.376 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-09-18T08:09:58.376 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-09-18T08:09:58.376 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "started", 2024-09-18T08:09:58.376 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-09-18T08:09:58.376 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:09:58.376 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-09-18T08:09:58.376 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:58.371128+0000", 2024-09-18T08:09:58.376 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-18T08:09:58.376 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:09:58.376 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:09:58.377 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-09-18T08:09:58.377 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:58.371128+0000", 2024-09-18T08:09:58.377 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-18T08:09:58.377 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:09:58.377 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:09:58.377 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-09-18T08:09:58.377 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:58.371129+0000", 2024-09-18T08:09:58.377 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.0070000000000001e-06 2024-09-18T08:09:58.377 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:09:58.377 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:09:58.377 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-09-18T08:09:58.377 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:58.371139+0000", 2024-09-18T08:09:58.378 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.0545e-05 2024-09-18T08:09:58.378 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:09:58.378 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:09:58.378 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-09-18T08:09:58.378 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:58.371154+0000", 2024-09-18T08:09:58.378 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.5109999999999999e-05 2024-09-18T08:09:58.378 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:09:58.378 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:09:58.378 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-09-18T08:09:58.378 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:58.371162+0000", 2024-09-18T08:09:58.378 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 7.8820000000000001e-06 2024-09-18T08:09:58.378 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:09:58.378 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:09:58.379 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-18T08:09:58.379 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:58.371230+0000", 2024-09-18T08:09:58.379 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 6.7817999999999993e-05 2024-09-18T08:09:58.379 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:09:58.379 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:09:58.379 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-09-18T08:09:58.379 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:58.371250+0000", 2024-09-18T08:09:58.379 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.9690999999999998e-05 2024-09-18T08:09:58.379 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-18T08:09:58.379 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-09-18T08:09:58.379 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-18T08:09:58.379 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-18T08:09:58.380 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2024-09-18T08:09:58.380 INFO:teuthology.orchestra.run.smithi050.stdout: "num_ops": 1 2024-09-18T08:09:58.380 INFO:teuthology.orchestra.run.smithi050.stdout:} 2024-09-18T08:09:58.408 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2 2024-09-18T08:09:58.446 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:09:58.447 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:09:58.448 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2' 2024-09-18T08:09:58.549 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3 2024-09-18T08:09:58.588 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:09:58.589 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:09:58.589 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3' 2024-09-18T08:09:58.638 INFO:teuthology.orchestra.run.smithi007.stderr:nodeep-scrub is set 2024-09-18T08:09:58.691 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6 2024-09-18T08:09:58.722 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:09:58.723 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:09:58.723 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6' 2024-09-18T08:09:58.732 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:09:58 smithi007 ceph-mon[29459]: purged_snaps scrub starts 2024-09-18T08:09:58.732 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:09:58 smithi007 ceph-mon[29459]: purged_snaps scrub ok 2024-09-18T08:09:58.732 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:09:58 smithi007 ceph-mon[29459]: from='client.? 172.21.15.7:0/2196374951' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-18T08:09:58.733 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:09:58 smithi007 ceph-mon[39249]: purged_snaps scrub starts 2024-09-18T08:09:58.733 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:09:58 smithi007 ceph-mon[39249]: purged_snaps scrub ok 2024-09-18T08:09:58.733 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:09:58 smithi007 ceph-mon[39249]: from='client.? 172.21.15.7:0/2196374951' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-18T08:09:58.824 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1 2024-09-18T08:09:58.861 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:09:58.861 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:09:58.862 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1' 2024-09-18T08:09:58.963 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5 2024-09-18T08:09:58.998 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:09:58.999 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:09:58.999 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5' 2024-09-18T08:09:59.085 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:09:58 smithi050 ceph-mon[32371]: purged_snaps scrub starts 2024-09-18T08:09:59.086 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:09:58 smithi050 ceph-mon[32371]: purged_snaps scrub ok 2024-09-18T08:09:59.086 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:09:58 smithi050 ceph-mon[32371]: from='client.? 172.21.15.7:0/2196374951' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-18T08:09:59.100 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6 2024-09-18T08:09:59.133 DEBUG:teuthology.orchestra.run.smithi050:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid 779349ca-7593-11ef-bceb-c7b262605968 -- ceph daemon osd.6 dump_blocked_ops 2024-09-18T08:09:59.134 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:09:59.135 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:09:59.135 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6' 2024-09-18T08:09:59.236 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0 2024-09-18T08:09:59.279 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:09:59.279 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:09:59.280 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0' 2024-09-18T08:09:59.380 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6 2024-09-18T08:09:59.406 INFO:journalctl@ceph.iscsi.iscsi.a.smithi050.stdout:Sep 18 08:09:59 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-iscsi-iscsi-a[57782]: debug there is no tcmu-runner data available 2024-09-18T08:09:59.431 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:09:59.433 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:09:59.433 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6' 2024-09-18T08:09:59.441 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid 779349ca-7593-11ef-bceb-c7b262605968 -- ceph pg dump --format=json 2024-09-18T08:09:59.534 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2 2024-09-18T08:09:59.557 INFO:teuthology.orchestra.run.smithi050.stderr:Inferring config /var/lib/ceph/779349ca-7593-11ef-bceb-c7b262605968/mon.b/config 2024-09-18T08:09:59.581 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:09:59.582 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:09:59.582 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2' 2024-09-18T08:09:59.683 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6 2024-09-18T08:09:59.718 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:09:59.719 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:09:59.719 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6' 2024-09-18T08:09:59.820 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6 2024-09-18T08:09:59.856 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:09:59.857 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:09:59.857 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6' 2024-09-18T08:09:59.881 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/779349ca-7593-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T08:09:59.958 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4 2024-09-18T08:09:59.981 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:09:59 smithi007 ceph-mon[29459]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-18T08:09:59.981 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:09:59 smithi007 ceph-mon[29459]: from='client.? 172.21.15.7:0/2196374951' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-18T08:09:59.981 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:09:59 smithi007 ceph-mon[29459]: osdmap e216: 8 total, 8 up, 8 in 2024-09-18T08:09:59.982 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:09:59 smithi007 ceph-mon[29459]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-18T08:09:59.982 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:09:59 smithi007 ceph-mon[29459]: pgmap v347: 158 pgs: 1 active+recovering+undersized+remapped, 157 active+clean; 4.1 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.46k op/s; 1178/203937 objects misplaced (0.578%); 4.1 MiB/s, 65 objects/s recovering 2024-09-18T08:09:59.982 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:09:59 smithi007 ceph-mon[39249]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-18T08:09:59.982 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:09:59 smithi007 ceph-mon[39249]: from='client.? 172.21.15.7:0/2196374951' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-18T08:09:59.982 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:09:59 smithi007 ceph-mon[39249]: osdmap e216: 8 total, 8 up, 8 in 2024-09-18T08:09:59.982 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:09:59 smithi007 ceph-mon[39249]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-18T08:09:59.982 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:09:59 smithi007 ceph-mon[39249]: pgmap v347: 158 pgs: 1 active+recovering+undersized+remapped, 157 active+clean; 4.1 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.46k op/s; 1178/203937 objects misplaced (0.578%); 4.1 MiB/s, 65 objects/s recovering 2024-09-18T08:09:59.986 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:09:59.987 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:09:59.987 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4' 2024-09-18T08:10:00.005 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:09:59 smithi050 ceph-mon[32371]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-18T08:10:00.005 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:09:59 smithi050 ceph-mon[32371]: from='client.? 172.21.15.7:0/2196374951' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-18T08:10:00.005 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:09:59 smithi050 ceph-mon[32371]: osdmap e216: 8 total, 8 up, 8 in 2024-09-18T08:10:00.005 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:09:59 smithi050 ceph-mon[32371]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-18T08:10:00.006 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:09:59 smithi050 ceph-mon[32371]: pgmap v347: 158 pgs: 1 active+recovering+undersized+remapped, 157 active+clean; 4.1 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.46k op/s; 1178/203937 objects misplaced (0.578%); 4.1 MiB/s, 65 objects/s recovering 2024-09-18T08:10:00.088 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3 2024-09-18T08:10:00.128 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:10:00.128 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:10:00.129 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3' 2024-09-18T08:10:00.172 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2024-09-18T08:10:00.172 INFO:teuthology.orchestra.run.smithi050.stdout: "ops": [], 2024-09-18T08:10:00.172 INFO:teuthology.orchestra.run.smithi050.stdout: "complaint_time": 30, 2024-09-18T08:10:00.172 INFO:teuthology.orchestra.run.smithi050.stdout: "num_blocked_ops": 0 2024-09-18T08:10:00.172 INFO:teuthology.orchestra.run.smithi050.stdout:} 2024-09-18T08:10:00.230 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5 2024-09-18T08:10:00.263 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:10:00.264 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:10:00.265 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5' 2024-09-18T08:10:00.366 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2 2024-09-18T08:10:00.406 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:10:00.407 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:10:00.407 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2' 2024-09-18T08:10:00.469 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid 779349ca-7593-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-18T08:10:00.508 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5 2024-09-18T08:10:00.539 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:10:00.539 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:10:00.540 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5' 2024-09-18T08:10:00.641 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3 2024-09-18T08:10:00.696 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:10:00.697 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:10:00.697 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3' 2024-09-18T08:10:00.798 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2 2024-09-18T08:10:00.846 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:10:00.847 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:10:00.847 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2' 2024-09-18T08:10:00.867 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-09-18T08:10:00.873 INFO:teuthology.orchestra.run.smithi007.stderr:dumped all 2024-09-18T08:10:00.948 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3 2024-09-18T08:10:00.981 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:10:00 smithi007 ceph-mon[29459]: Health detail: HEALTH_WARN noscrub,nodeep-scrub flag(s) set; Degraded data redundancy: 1 pg undersized 2024-09-18T08:10:00.981 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:10:00 smithi007 ceph-mon[29459]: [WRN] OSDMAP_FLAGS: noscrub,nodeep-scrub flag(s) set 2024-09-18T08:10:00.981 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:10:00 smithi007 ceph-mon[29459]: [WRN] PG_DEGRADED: Degraded data redundancy: 1 pg undersized 2024-09-18T08:10:00.981 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:10:00 smithi007 ceph-mon[29459]: pg 8.4 is stuck undersized for 63s, current state active+recovering+undersized+remapped, last acting [5,4] 2024-09-18T08:10:00.981 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:10:00 smithi007 ceph-mon[39249]: Health detail: HEALTH_WARN noscrub,nodeep-scrub flag(s) set; Degraded data redundancy: 1 pg undersized 2024-09-18T08:10:00.982 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:10:00 smithi007 ceph-mon[39249]: [WRN] OSDMAP_FLAGS: noscrub,nodeep-scrub flag(s) set 2024-09-18T08:10:00.982 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:10:00 smithi007 ceph-mon[39249]: [WRN] PG_DEGRADED: Degraded data redundancy: 1 pg undersized 2024-09-18T08:10:00.982 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:10:00 smithi007 ceph-mon[39249]: pg 8.4 is stuck undersized for 63s, current state active+recovering+undersized+remapped, last acting [5,4] 2024-09-18T08:10:00.988 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:10:00.988 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:10:00.989 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3' 2024-09-18T08:10:01.064 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:10:00 smithi050 ceph-mon[32371]: Health detail: HEALTH_WARN noscrub,nodeep-scrub flag(s) set; Degraded data redundancy: 1 pg undersized 2024-09-18T08:10:01.064 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:10:00 smithi050 ceph-mon[32371]: [WRN] OSDMAP_FLAGS: noscrub,nodeep-scrub flag(s) set 2024-09-18T08:10:01.064 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:10:00 smithi050 ceph-mon[32371]: [WRN] PG_DEGRADED: Degraded data redundancy: 1 pg undersized 2024-09-18T08:10:01.064 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:10:00 smithi050 ceph-mon[32371]: pg 8.4 is stuck undersized for 63s, current state active+recovering+undersized+remapped, last acting [5,4] 2024-09-18T08:10:01.090 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7 2024-09-18T08:10:01.111 DEBUG:teuthology.orchestra.run.smithi050:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid 779349ca-7593-11ef-bceb-c7b262605968 -- ceph daemon osd.6 dump_historic_ops 2024-09-18T08:10:01.123 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:10:01.123 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:10:01.123 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7' 2024-09-18T08:10:01.225 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0 2024-09-18T08:10:01.260 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:10:01.261 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:10:01.261 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0' 2024-09-18T08:10:01.362 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1 2024-09-18T08:10:01.376 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid 779349ca-7593-11ef-bceb-c7b262605968 -- ceph osd unset noscrub 2024-09-18T08:10:01.397 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:10:01.397 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:10:01.398 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1' 2024-09-18T08:10:01.499 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0 2024-09-18T08:10:01.536 INFO:teuthology.orchestra.run.smithi050.stderr:Inferring config /var/lib/ceph/779349ca-7593-11ef-bceb-c7b262605968/mon.b/config 2024-09-18T08:10:01.538 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:10:01.539 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:10:01.539 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0' 2024-09-18T08:10:01.640 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5 2024-09-18T08:10:01.669 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:10:01.670 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:10:01.670 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5' 2024-09-18T08:10:01.730 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:10:01 smithi007 ceph-mon[29459]: from='client.15414 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-18T08:10:01.730 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:10:01 smithi007 ceph-mon[29459]: pgmap v348: 158 pgs: 1 active+recovering+undersized+remapped, 157 active+clean; 4.1 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 1.0 KiB/s rd, 0 B/s wr, 1.21k op/s; 1178/203937 objects misplaced (0.578%); 3.4 MiB/s, 54 objects/s recovering 2024-09-18T08:10:01.731 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:10:01 smithi007 ceph-mon[39249]: from='client.15414 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-18T08:10:01.731 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:10:01 smithi007 ceph-mon[39249]: pgmap v348: 158 pgs: 1 active+recovering+undersized+remapped, 157 active+clean; 4.1 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 1.0 KiB/s rd, 0 B/s wr, 1.21k op/s; 1178/203937 objects misplaced (0.578%); 3.4 MiB/s, 54 objects/s recovering 2024-09-18T08:10:01.771 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1 2024-09-18T08:10:01.814 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:10:01.815 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:10:01.815 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1' 2024-09-18T08:10:01.916 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2 2024-09-18T08:10:01.963 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:10:01.963 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:10:01.963 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2' 2024-09-18T08:10:01.975 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:10:01 smithi050 ceph-mon[32371]: from='client.15414 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-18T08:10:01.975 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:10:01 smithi050 ceph-mon[32371]: pgmap v348: 158 pgs: 1 active+recovering+undersized+remapped, 157 active+clean; 4.1 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 1.0 KiB/s rd, 0 B/s wr, 1.21k op/s; 1178/203937 objects misplaced (0.578%); 3.4 MiB/s, 54 objects/s recovering 2024-09-18T08:10:02.064 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6 2024-09-18T08:10:02.101 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:10:02.102 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:10:02.102 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6' 2024-09-18T08:10:02.138 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2024-09-18T08:10:02.138 INFO:teuthology.orchestra.run.smithi050.stdout: "size": 20, 2024-09-18T08:10:02.138 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 600, 2024-09-18T08:10:02.138 INFO:teuthology.orchestra.run.smithi050.stdout: "ops": [ 2024-09-18T08:10:02.138 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:10:02.138 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.14805.0:38000 7.7 7:e509d896:::benchmark_data_smithi007_73155_object37999:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e94)", 2024-09-18T08:10:02.138 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-09-18T08:04:23.049009+0000", 2024-09-18T08:10:02.138 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 339.08544633999998, 2024-09-18T08:10:02.139 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.027565702000000001, 2024-09-18T08:10:02.139 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-09-18T08:10:02.139 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-09-18T08:10:02.139 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T08:10:02.139 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2024-09-18T08:10:02.139 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.14805", 2024-09-18T08:10:02.139 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "172.21.15.7:0/640320436", 2024-09-18T08:10:02.139 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 38000 2024-09-18T08:10:02.139 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:10:02.139 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-09-18T08:10:02.139 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:10:02.139 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-09-18T08:10:02.139 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:23.049009+0000", 2024-09-18T08:10:02.140 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-18T08:10:02.140 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:10:02.140 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:10:02.140 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-09-18T08:10:02.140 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:23.049009+0000", 2024-09-18T08:10:02.140 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-18T08:10:02.140 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:10:02.140 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:10:02.140 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-09-18T08:10:02.140 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:23.049010+0000", 2024-09-18T08:10:02.140 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.11e-06 2024-09-18T08:10:02.140 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:10:02.140 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:10:02.141 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-09-18T08:10:02.141 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:23.068284+0000", 2024-09-18T08:10:02.141 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.019274264999999999 2024-09-18T08:10:02.141 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:10:02.141 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:10:02.141 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-09-18T08:10:02.141 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:23.068323+0000", 2024-09-18T08:10:02.141 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 3.8167999999999998e-05 2024-09-18T08:10:02.141 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:10:02.141 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:10:02.141 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-09-18T08:10:02.141 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:23.068329+0000", 2024-09-18T08:10:02.141 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 6.331e-06 2024-09-18T08:10:02.141 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:10:02.142 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:10:02.142 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-18T08:10:02.142 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:23.068399+0000", 2024-09-18T08:10:02.142 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 7.0055999999999995e-05 2024-09-18T08:10:02.142 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:10:02.142 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:10:02.142 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-09-18T08:10:02.142 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:23.068503+0000", 2024-09-18T08:10:02.142 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000104206 2024-09-18T08:10:02.142 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:10:02.143 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:10:02.143 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "waiting for subops from 0,3", 2024-09-18T08:10:02.143 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:23.068640+0000", 2024-09-18T08:10:02.143 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00013645000000000001 2024-09-18T08:10:02.143 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:10:02.143 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:10:02.143 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "op_commit", 2024-09-18T08:10:02.143 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:23.073270+0000", 2024-09-18T08:10:02.143 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0046298950000000002 2024-09-18T08:10:02.143 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:10:02.143 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:10:02.143 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:10:02.143 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:23.073311+0000", 2024-09-18T08:10:02.143 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4.1406999999999998e-05 2024-09-18T08:10:02.144 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:10:02.144 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:10:02.144 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:10:02.144 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:23.076521+0000", 2024-09-18T08:10:02.144 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.003209862 2024-09-18T08:10:02.144 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:10:02.144 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:10:02.144 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "commit_sent", 2024-09-18T08:10:02.144 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:23.076552+0000", 2024-09-18T08:10:02.144 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 3.1698e-05 2024-09-18T08:10:02.145 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:10:02.145 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:10:02.145 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-09-18T08:10:02.145 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:23.076575+0000", 2024-09-18T08:10:02.145 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 2.2254000000000001e-05 2024-09-18T08:10:02.145 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-18T08:10:02.145 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-09-18T08:10:02.146 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-18T08:10:02.146 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:10:02.146 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:10:02.146 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.14805.0:39064 7.7 7:ef833039:::benchmark_data_smithi007_73155_object39063:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e95)", 2024-09-18T08:10:02.146 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-09-18T08:04:23.658960+0000", 2024-09-18T08:10:02.146 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 338.47549506799999, 2024-09-18T08:10:02.146 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.029075624000000001, 2024-09-18T08:10:02.146 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-09-18T08:10:02.146 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-09-18T08:10:02.146 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T08:10:02.146 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2024-09-18T08:10:02.146 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.14805", 2024-09-18T08:10:02.146 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "172.21.15.7:0/640320436", 2024-09-18T08:10:02.147 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 39064 2024-09-18T08:10:02.147 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:10:02.147 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-09-18T08:10:02.147 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:10:02.147 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-09-18T08:10:02.147 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:23.658960+0000", 2024-09-18T08:10:02.147 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-18T08:10:02.147 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:10:02.147 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:10:02.147 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-09-18T08:10:02.147 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:23.658960+0000", 2024-09-18T08:10:02.147 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-18T08:10:02.147 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:10:02.147 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:10:02.148 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-09-18T08:10:02.148 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:23.658961+0000", 2024-09-18T08:10:02.148 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 6.7899999999999998e-07 2024-09-18T08:10:02.148 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:10:02.148 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:10:02.148 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-09-18T08:10:02.148 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:23.658991+0000", 2024-09-18T08:10:02.148 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 3.0168999999999999e-05 2024-09-18T08:10:02.148 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:10:02.148 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:10:02.148 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-09-18T08:10:02.148 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:23.659001+0000", 2024-09-18T08:10:02.148 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 9.6220000000000004e-06 2024-09-18T08:10:02.148 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:10:02.149 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:10:02.149 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-09-18T08:10:02.149 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:23.659010+0000", 2024-09-18T08:10:02.149 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 8.9160000000000007e-06 2024-09-18T08:10:02.149 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:10:02.149 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:10:02.149 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-18T08:10:02.149 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:23.661923+0000", 2024-09-18T08:10:02.149 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.002913279 2024-09-18T08:10:02.149 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:10:02.149 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:10:02.149 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-09-18T08:10:02.149 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:23.662030+0000", 2024-09-18T08:10:02.149 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00010665700000000001 2024-09-18T08:10:02.150 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:10:02.150 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:10:02.150 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "waiting for subops from 0,3", 2024-09-18T08:10:02.150 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:23.662167+0000", 2024-09-18T08:10:02.150 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00013728300000000001 2024-09-18T08:10:02.150 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:10:02.150 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:10:02.150 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "op_commit", 2024-09-18T08:10:02.150 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:23.663255+0000", 2024-09-18T08:10:02.150 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0010883900000000001 2024-09-18T08:10:02.150 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:10:02.150 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:10:02.150 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:10:02.150 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:23.671279+0000", 2024-09-18T08:10:02.150 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0080233749999999993 2024-09-18T08:10:02.151 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:10:02.151 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:10:02.151 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:10:02.151 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:23.687965+0000", 2024-09-18T08:10:02.151 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.016686590000000001 2024-09-18T08:10:02.151 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:10:02.151 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:10:02.151 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "commit_sent", 2024-09-18T08:10:02.151 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:23.688005+0000", 2024-09-18T08:10:02.151 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 3.9604000000000002e-05 2024-09-18T08:10:02.151 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:10:02.151 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:10:02.151 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-09-18T08:10:02.151 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:23.688036+0000", 2024-09-18T08:10:02.152 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 3.1059999999999997e-05 2024-09-18T08:10:02.152 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-18T08:10:02.152 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-09-18T08:10:02.152 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-18T08:10:02.152 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:10:02.152 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:10:02.152 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.14805.0:39805 7.7 7:e9cd36b3:::benchmark_data_smithi007_73155_object39804:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e95)", 2024-09-18T08:10:02.152 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-09-18T08:04:24.061655+0000", 2024-09-18T08:10:02.152 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 338.07280038699997, 2024-09-18T08:10:02.152 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.036883131, 2024-09-18T08:10:02.152 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-09-18T08:10:02.152 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-09-18T08:10:02.152 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T08:10:02.153 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2024-09-18T08:10:02.153 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.14805", 2024-09-18T08:10:02.153 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "172.21.15.7:0/640320436", 2024-09-18T08:10:02.153 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 39805 2024-09-18T08:10:02.153 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:10:02.154 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-09-18T08:10:02.154 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:10:02.154 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-09-18T08:10:02.154 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:24.061655+0000", 2024-09-18T08:10:02.154 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-18T08:10:02.154 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:10:02.154 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:10:02.154 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-09-18T08:10:02.154 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:24.061655+0000", 2024-09-18T08:10:02.154 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-18T08:10:02.154 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:10:02.154 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:10:02.154 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-09-18T08:10:02.155 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:24.061656+0000", 2024-09-18T08:10:02.155 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.4449999999999999e-06 2024-09-18T08:10:02.155 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:10:02.155 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:10:02.155 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-09-18T08:10:02.155 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:24.061686+0000", 2024-09-18T08:10:02.155 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 2.9984e-05 2024-09-18T08:10:02.155 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:10:02.155 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:10:02.155 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-09-18T08:10:02.155 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:24.061699+0000", 2024-09-18T08:10:02.155 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.2850000000000001e-05 2024-09-18T08:10:02.155 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:10:02.155 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:10:02.156 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-09-18T08:10:02.156 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:24.061706+0000", 2024-09-18T08:10:02.156 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 6.8560000000000001e-06 2024-09-18T08:10:02.156 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:10:02.156 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:10:02.156 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-18T08:10:02.156 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:24.061755+0000", 2024-09-18T08:10:02.156 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4.9001000000000001e-05 2024-09-18T08:10:02.156 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:10:02.156 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:10:02.156 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-09-18T08:10:02.156 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:24.061896+0000", 2024-09-18T08:10:02.156 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00014040499999999999 2024-09-18T08:10:02.157 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:10:02.157 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:10:02.157 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "waiting for subops from 0,3", 2024-09-18T08:10:02.157 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:24.062006+0000", 2024-09-18T08:10:02.157 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000110813 2024-09-18T08:10:02.157 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:10:02.157 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:10:02.157 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:10:02.157 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:24.065677+0000", 2024-09-18T08:10:02.157 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0036702800000000002 2024-09-18T08:10:02.157 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:10:02.157 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:10:02.157 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "op_commit", 2024-09-18T08:10:02.157 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:24.065694+0000", 2024-09-18T08:10:02.158 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.7876e-05 2024-09-18T08:10:02.158 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:10:02.158 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:10:02.158 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:10:02.158 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:24.098357+0000", 2024-09-18T08:10:02.158 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.032662140999999999 2024-09-18T08:10:02.158 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:10:02.158 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:10:02.158 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "commit_sent", 2024-09-18T08:10:02.158 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:24.098447+0000", 2024-09-18T08:10:02.158 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 9.0724999999999996e-05 2024-09-18T08:10:02.158 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:10:02.158 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:10:02.158 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-09-18T08:10:02.159 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:24.098538+0000", 2024-09-18T08:10:02.159 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 9.0754999999999994e-05 2024-09-18T08:10:02.159 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-18T08:10:02.159 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-09-18T08:10:02.159 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-18T08:10:02.159 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:10:02.159 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:10:02.159 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.14805.0:39807 7.d 7:bf1e27dc:::benchmark_data_smithi007_73155_object39806:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e95)", 2024-09-18T08:10:02.159 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-09-18T08:04:24.061827+0000", 2024-09-18T08:10:02.159 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 338.072628092, 2024-09-18T08:10:02.159 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.037358672000000002, 2024-09-18T08:10:02.159 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-09-18T08:10:02.159 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-09-18T08:10:02.159 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T08:10:02.160 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2024-09-18T08:10:02.160 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.14805", 2024-09-18T08:10:02.160 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "172.21.15.7:0/640320436", 2024-09-18T08:10:02.160 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 39807 2024-09-18T08:10:02.160 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:10:02.160 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-09-18T08:10:02.160 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:10:02.160 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-09-18T08:10:02.160 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:24.061827+0000", 2024-09-18T08:10:02.160 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-18T08:10:02.160 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:10:02.160 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:10:02.160 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-09-18T08:10:02.160 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:24.061827+0000", 2024-09-18T08:10:02.161 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-18T08:10:02.161 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:10:02.161 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:10:02.161 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-09-18T08:10:02.161 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:24.061828+0000", 2024-09-18T08:10:02.161 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 7.9899999999999999e-07 2024-09-18T08:10:02.161 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:10:02.161 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:10:02.161 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-09-18T08:10:02.161 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:24.061888+0000", 2024-09-18T08:10:02.161 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 5.9497000000000003e-05 2024-09-18T08:10:02.161 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:10:02.161 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:10:02.161 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-09-18T08:10:02.162 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:24.061904+0000", 2024-09-18T08:10:02.162 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.6393999999999999e-05 2024-09-18T08:10:02.162 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:10:02.162 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:10:02.162 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-09-18T08:10:02.162 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:24.061912+0000", 2024-09-18T08:10:02.162 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 8.0909999999999993e-06 2024-09-18T08:10:02.162 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:10:02.162 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:10:02.162 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-18T08:10:02.162 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:24.061968+0000", 2024-09-18T08:10:02.162 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 5.5543e-05 2024-09-18T08:10:02.162 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:10:02.162 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:10:02.163 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-09-18T08:10:02.163 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:24.062061+0000", 2024-09-18T08:10:02.163 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 9.3558000000000001e-05 2024-09-18T08:10:02.163 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:10:02.163 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:10:02.163 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "waiting for subops from 0,1", 2024-09-18T08:10:02.163 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:24.062163+0000", 2024-09-18T08:10:02.163 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000101903 2024-09-18T08:10:02.163 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:10:02.163 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:10:02.163 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "op_commit", 2024-09-18T08:10:02.163 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:24.064018+0000", 2024-09-18T08:10:02.163 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.001855071 2024-09-18T08:10:02.163 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:10:02.163 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:10:02.164 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:10:02.164 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:24.064283+0000", 2024-09-18T08:10:02.164 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000265173 2024-09-18T08:10:02.164 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:10:02.164 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:10:02.164 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:10:02.164 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:24.099043+0000", 2024-09-18T08:10:02.164 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.034760159999999998 2024-09-18T08:10:02.164 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:10:02.164 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:10:02.164 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "commit_sent", 2024-09-18T08:10:02.164 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:24.099100+0000", 2024-09-18T08:10:02.164 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 5.6147000000000003e-05 2024-09-18T08:10:02.164 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:10:02.165 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:10:02.165 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-09-18T08:10:02.165 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:24.099186+0000", 2024-09-18T08:10:02.165 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 8.6335999999999995e-05 2024-09-18T08:10:02.165 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-18T08:10:02.165 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-09-18T08:10:02.165 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-18T08:10:02.165 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:10:02.165 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:10:02.165 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.14805.0:39808 7.d 7:b98cab8f:::benchmark_data_smithi007_73155_object39807:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e95)", 2024-09-18T08:10:02.165 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-09-18T08:04:24.067964+0000", 2024-09-18T08:10:02.165 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 338.06649165499999, 2024-09-18T08:10:02.165 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.034258180999999999, 2024-09-18T08:10:02.165 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-09-18T08:10:02.166 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-09-18T08:10:02.166 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T08:10:02.166 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2024-09-18T08:10:02.166 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.14805", 2024-09-18T08:10:02.166 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "172.21.15.7:0/640320436", 2024-09-18T08:10:02.166 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 39808 2024-09-18T08:10:02.166 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:10:02.166 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-09-18T08:10:02.166 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:10:02.166 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-09-18T08:10:02.166 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:24.067964+0000", 2024-09-18T08:10:02.166 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-18T08:10:02.166 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:10:02.166 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:10:02.166 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-09-18T08:10:02.167 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:24.067964+0000", 2024-09-18T08:10:02.167 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-18T08:10:02.167 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:10:02.167 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:10:02.167 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-09-18T08:10:02.167 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:24.067965+0000", 2024-09-18T08:10:02.167 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.4750000000000001e-06 2024-09-18T08:10:02.167 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:10:02.167 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:10:02.167 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-09-18T08:10:02.167 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:24.067992+0000", 2024-09-18T08:10:02.167 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 2.6760000000000001e-05 2024-09-18T08:10:02.167 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:10:02.167 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:10:02.168 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-09-18T08:10:02.168 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:24.068004+0000", 2024-09-18T08:10:02.168 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.2456000000000001e-05 2024-09-18T08:10:02.168 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:10:02.168 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:10:02.168 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-09-18T08:10:02.168 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:24.068012+0000", 2024-09-18T08:10:02.168 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 8.0779999999999996e-06 2024-09-18T08:10:02.168 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:10:02.168 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:10:02.168 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-18T08:10:02.168 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:24.068072+0000", 2024-09-18T08:10:02.168 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 5.9299000000000003e-05 2024-09-18T08:10:02.168 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:10:02.169 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:10:02.169 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-09-18T08:10:02.169 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:24.068164+0000", 2024-09-18T08:10:02.169 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 9.2053999999999997e-05 2024-09-18T08:10:02.169 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:10:02.169 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:10:02.169 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "waiting for subops from 0,1", 2024-09-18T08:10:02.169 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:24.096945+0000", 2024-09-18T08:10:02.169 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.028781128 2024-09-18T08:10:02.169 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:10:02.170 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:10:02.170 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "op_commit", 2024-09-18T08:10:02.170 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:24.099254+0000", 2024-09-18T08:10:02.170 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.002309218 2024-09-18T08:10:02.170 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:10:02.170 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:10:02.170 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:10:02.170 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:24.100291+0000", 2024-09-18T08:10:02.170 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0010365280000000001 2024-09-18T08:10:02.170 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:10:02.170 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:10:02.170 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:10:02.170 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:24.102125+0000", 2024-09-18T08:10:02.170 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0018346599999999999 2024-09-18T08:10:02.171 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:10:02.171 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:10:02.171 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "commit_sent", 2024-09-18T08:10:02.171 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:24.102184+0000", 2024-09-18T08:10:02.171 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 5.8965000000000002e-05 2024-09-18T08:10:02.171 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:10:02.171 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:10:02.171 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-09-18T08:10:02.171 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:24.102222+0000", 2024-09-18T08:10:02.171 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 3.756e-05 2024-09-18T08:10:02.171 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-18T08:10:02.171 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-09-18T08:10:02.171 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-18T08:10:02.171 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:10:02.172 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:10:02.172 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_repop_reply(client.14805.0:39807 7.d e95/83 ondisk, result = 0)", 2024-09-18T08:10:02.172 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-09-18T08:04:24.069321+0000", 2024-09-18T08:10:02.172 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 338.06513429099999, 2024-09-18T08:10:02.172 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.029872724999999999, 2024-09-18T08:10:02.172 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-09-18T08:10:02.172 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-09-18T08:10:02.172 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "started", 2024-09-18T08:10:02.172 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-09-18T08:10:02.172 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:10:02.172 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-09-18T08:10:02.172 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:24.069321+0000", 2024-09-18T08:10:02.172 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-18T08:10:02.172 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:10:02.173 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:10:02.173 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-09-18T08:10:02.173 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:24.069321+0000", 2024-09-18T08:10:02.173 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-18T08:10:02.173 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:10:02.173 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:10:02.173 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-09-18T08:10:02.173 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:24.069322+0000", 2024-09-18T08:10:02.173 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 8.3799999999999996e-07 2024-09-18T08:10:02.173 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:10:02.173 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:10:02.173 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-09-18T08:10:02.173 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:24.069327+0000", 2024-09-18T08:10:02.174 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4.8679999999999998e-06 2024-09-18T08:10:02.174 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:10:02.174 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:10:02.174 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-09-18T08:10:02.174 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:24.098180+0000", 2024-09-18T08:10:02.174 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.028853407000000001 2024-09-18T08:10:02.174 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:10:02.174 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:10:02.174 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-09-18T08:10:02.174 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:24.098189+0000", 2024-09-18T08:10:02.174 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 8.8850000000000005e-06 2024-09-18T08:10:02.174 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:10:02.174 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:10:02.174 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-18T08:10:02.174 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:24.099029+0000", 2024-09-18T08:10:02.175 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00084036500000000004 2024-09-18T08:10:02.175 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:10:02.175 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:10:02.175 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-09-18T08:10:02.175 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:24.099037+0000", 2024-09-18T08:10:02.175 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 8.0150000000000003e-06 2024-09-18T08:10:02.175 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:10:02.175 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:10:02.175 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-09-18T08:10:02.175 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:24.099194+0000", 2024-09-18T08:10:02.175 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00015634700000000001 2024-09-18T08:10:02.175 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-18T08:10:02.175 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-09-18T08:10:02.175 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-18T08:10:02.176 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:10:02.176 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:10:02.176 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.14805.0:39817 7.c 7:3826d3a8:::benchmark_data_smithi007_73155_object39816:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e96)", 2024-09-18T08:10:02.176 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-09-18T08:04:24.077849+0000", 2024-09-18T08:10:02.176 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 338.056606343, 2024-09-18T08:10:02.176 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.028025705000000001, 2024-09-18T08:10:02.176 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-09-18T08:10:02.176 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-09-18T08:10:02.176 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T08:10:02.176 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2024-09-18T08:10:02.176 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.14805", 2024-09-18T08:10:02.176 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "172.21.15.7:0/640320436", 2024-09-18T08:10:02.177 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 39817 2024-09-18T08:10:02.177 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:10:02.177 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-09-18T08:10:02.177 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:10:02.177 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-09-18T08:10:02.177 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:24.077849+0000", 2024-09-18T08:10:02.177 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-18T08:10:02.177 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:10:02.177 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:10:02.177 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-09-18T08:10:02.177 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:24.077849+0000", 2024-09-18T08:10:02.177 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-18T08:10:02.178 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:10:02.178 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:10:02.178 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-09-18T08:10:02.178 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:24.077851+0000", 2024-09-18T08:10:02.178 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 2.322e-06 2024-09-18T08:10:02.178 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:10:02.178 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:10:02.178 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-09-18T08:10:02.178 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:24.077879+0000", 2024-09-18T08:10:02.178 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 2.7596000000000001e-05 2024-09-18T08:10:02.178 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:10:02.178 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:10:02.178 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-09-18T08:10:02.178 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:24.097805+0000", 2024-09-18T08:10:02.178 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.019925552999999999 2024-09-18T08:10:02.179 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:10:02.179 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:10:02.179 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-09-18T08:10:02.179 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:24.097820+0000", 2024-09-18T08:10:02.179 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.5705999999999998e-05 2024-09-18T08:10:02.179 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:10:02.179 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:10:02.179 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-18T08:10:02.179 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:24.097984+0000", 2024-09-18T08:10:02.179 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00016333499999999999 2024-09-18T08:10:02.179 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:10:02.179 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:10:02.180 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "op must wait for map", 2024-09-18T08:10:02.180 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:24.097990+0000", 2024-09-18T08:10:02.180 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 6.81e-06 2024-09-18T08:10:02.180 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:10:02.180 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:10:02.180 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-18T08:10:02.180 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:24.102003+0000", 2024-09-18T08:10:02.180 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0040126160000000001 2024-09-18T08:10:02.180 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:10:02.180 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:10:02.180 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-09-18T08:10:02.180 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:24.102155+0000", 2024-09-18T08:10:02.180 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00015247500000000001 2024-09-18T08:10:02.181 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:10:02.181 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:10:02.181 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "waiting for subops from 3,7", 2024-09-18T08:10:02.181 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:24.102305+0000", 2024-09-18T08:10:02.181 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000149872 2024-09-18T08:10:02.181 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:10:02.181 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:10:02.181 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "op_commit", 2024-09-18T08:10:02.181 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:24.105200+0000", 2024-09-18T08:10:02.181 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0028950730000000002 2024-09-18T08:10:02.181 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:10:02.181 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:10:02.181 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:10:02.181 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:24.105248+0000", 2024-09-18T08:10:02.182 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4.7787999999999997e-05 2024-09-18T08:10:02.182 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:10:02.182 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:10:02.183 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:10:02.183 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:24.105772+0000", 2024-09-18T08:10:02.183 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00052425699999999998 2024-09-18T08:10:02.183 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:10:02.183 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:10:02.183 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "commit_sent", 2024-09-18T08:10:02.183 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:24.105826+0000", 2024-09-18T08:10:02.183 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 5.393e-05 2024-09-18T08:10:02.183 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:10:02.183 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:10:02.184 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-09-18T08:10:02.184 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:24.105875+0000", 2024-09-18T08:10:02.184 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4.8371999999999997e-05 2024-09-18T08:10:02.184 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-18T08:10:02.184 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-09-18T08:10:02.185 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-18T08:10:02.185 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:10:02.185 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:10:02.185 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.14805.0:43273 7.c 7:38821c7c:::benchmark_data_smithi007_73155_object43272:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e96)", 2024-09-18T08:10:02.185 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-09-18T08:04:26.167716+0000", 2024-09-18T08:10:02.185 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 335.96673941400002, 2024-09-18T08:10:02.185 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.027026471999999999, 2024-09-18T08:10:02.185 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-09-18T08:10:02.185 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-09-18T08:10:02.185 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T08:10:02.185 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2024-09-18T08:10:02.185 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.14805", 2024-09-18T08:10:02.185 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "172.21.15.7:0/640320436", 2024-09-18T08:10:02.185 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 43273 2024-09-18T08:10:02.186 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:10:02.186 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-09-18T08:10:02.186 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:10:02.186 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-09-18T08:10:02.186 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:26.167716+0000", 2024-09-18T08:10:02.186 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-18T08:10:02.186 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:10:02.186 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:10:02.186 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-09-18T08:10:02.186 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:26.167716+0000", 2024-09-18T08:10:02.186 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-18T08:10:02.186 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:10:02.186 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:10:02.186 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-09-18T08:10:02.187 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:26.167717+0000", 2024-09-18T08:10:02.187 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.3629999999999999e-06 2024-09-18T08:10:02.187 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:10:02.187 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:10:02.187 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-09-18T08:10:02.187 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:26.187242+0000", 2024-09-18T08:10:02.187 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.019525094999999999 2024-09-18T08:10:02.187 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:10:02.187 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:10:02.187 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-09-18T08:10:02.187 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:26.187255+0000", 2024-09-18T08:10:02.187 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.2669e-05 2024-09-18T08:10:02.187 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:10:02.187 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:10:02.188 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-09-18T08:10:02.188 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:26.187264+0000", 2024-09-18T08:10:02.188 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 9.3770000000000002e-06 2024-09-18T08:10:02.188 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:10:02.188 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:10:02.188 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-18T08:10:02.188 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:26.187321+0000", 2024-09-18T08:10:02.188 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 5.6379999999999999e-05 2024-09-18T08:10:02.188 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:10:02.188 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:10:02.188 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-09-18T08:10:02.188 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:26.187444+0000", 2024-09-18T08:10:02.188 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00012314699999999999 2024-09-18T08:10:02.188 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:10:02.189 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:10:02.189 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "waiting for subops from 3,7", 2024-09-18T08:10:02.189 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:26.187575+0000", 2024-09-18T08:10:02.189 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00013138700000000001 2024-09-18T08:10:02.189 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:10:02.189 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:10:02.189 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "op_commit", 2024-09-18T08:10:02.189 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:26.189195+0000", 2024-09-18T08:10:02.189 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0016195770000000001 2024-09-18T08:10:02.189 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:10:02.190 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:10:02.190 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:10:02.190 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:26.189327+0000", 2024-09-18T08:10:02.190 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00013181099999999999 2024-09-18T08:10:02.190 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:10:02.190 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:10:02.190 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:10:02.190 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:26.194627+0000", 2024-09-18T08:10:02.190 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0052998799999999999 2024-09-18T08:10:02.190 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:10:02.190 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:10:02.190 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "commit_sent", 2024-09-18T08:10:02.190 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:26.194693+0000", 2024-09-18T08:10:02.190 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 6.6748999999999994e-05 2024-09-18T08:10:02.191 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:10:02.191 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:10:02.191 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-09-18T08:10:02.191 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:26.194742+0000", 2024-09-18T08:10:02.191 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4.9036999999999998e-05 2024-09-18T08:10:02.191 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-18T08:10:02.191 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-09-18T08:10:02.191 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-18T08:10:02.191 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:10:02.191 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:10:02.191 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.14805.0:43281 7.c 7:35a814bc:::benchmark_data_smithi007_73155_object43280:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e96)", 2024-09-18T08:10:02.191 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-09-18T08:04:26.187285+0000", 2024-09-18T08:10:02.191 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 335.94717022600003, 2024-09-18T08:10:02.191 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.026194973999999999, 2024-09-18T08:10:02.192 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-09-18T08:10:02.192 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-09-18T08:10:02.192 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T08:10:02.192 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2024-09-18T08:10:02.192 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.14805", 2024-09-18T08:10:02.192 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "172.21.15.7:0/640320436", 2024-09-18T08:10:02.192 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 43281 2024-09-18T08:10:02.192 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:10:02.192 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-09-18T08:10:02.192 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:10:02.192 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-09-18T08:10:02.192 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:26.187285+0000", 2024-09-18T08:10:02.192 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-18T08:10:02.192 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:10:02.193 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:10:02.193 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-09-18T08:10:02.193 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:26.187285+0000", 2024-09-18T08:10:02.193 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-18T08:10:02.193 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:10:02.193 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:10:02.193 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-09-18T08:10:02.193 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:26.187286+0000", 2024-09-18T08:10:02.193 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.2190000000000001e-06 2024-09-18T08:10:02.193 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:10:02.193 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:10:02.193 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-09-18T08:10:02.193 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:26.206523+0000", 2024-09-18T08:10:02.193 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.019236797999999999 2024-09-18T08:10:02.194 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:10:02.194 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:10:02.194 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-09-18T08:10:02.194 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:26.206550+0000", 2024-09-18T08:10:02.194 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 2.6509999999999999e-05 2024-09-18T08:10:02.194 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:10:02.194 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:10:02.194 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-09-18T08:10:02.194 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:26.206567+0000", 2024-09-18T08:10:02.194 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.7401000000000002e-05 2024-09-18T08:10:02.194 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:10:02.194 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:10:02.194 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-18T08:10:02.194 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:26.206657+0000", 2024-09-18T08:10:02.194 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 8.9485000000000001e-05 2024-09-18T08:10:02.195 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:10:02.195 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:10:02.195 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-09-18T08:10:02.195 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:26.206738+0000", 2024-09-18T08:10:02.195 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 8.1077999999999999e-05 2024-09-18T08:10:02.195 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:10:02.195 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:10:02.195 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "waiting for subops from 3,7", 2024-09-18T08:10:02.195 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:26.206832+0000", 2024-09-18T08:10:02.195 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 9.4160000000000006e-05 2024-09-18T08:10:02.195 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:10:02.195 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:10:02.195 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:10:02.195 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:26.213326+0000", 2024-09-18T08:10:02.195 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0064943099999999997 2024-09-18T08:10:02.196 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:10:02.196 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:10:02.196 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "op_commit", 2024-09-18T08:10:02.196 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:26.213342+0000", 2024-09-18T08:10:02.196 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.5415000000000001e-05 2024-09-18T08:10:02.196 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:10:02.196 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:10:02.196 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:10:02.196 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:26.213393+0000", 2024-09-18T08:10:02.196 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 5.1038999999999997e-05 2024-09-18T08:10:02.196 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:10:02.196 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:10:02.197 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "commit_sent", 2024-09-18T08:10:02.197 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:26.213443+0000", 2024-09-18T08:10:02.197 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4.9938999999999997e-05 2024-09-18T08:10:02.197 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:10:02.198 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:10:02.198 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-09-18T08:10:02.198 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:26.213480+0000", 2024-09-18T08:10:02.198 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 3.7620000000000002e-05 2024-09-18T08:10:02.198 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-18T08:10:02.198 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-09-18T08:10:02.198 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-18T08:10:02.198 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:10:02.198 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:10:02.198 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.14805.0:43731 7.d 7:b3fc8faf:::benchmark_data_smithi007_73155_object43730:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e97)", 2024-09-18T08:10:02.198 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-09-18T08:04:26.432079+0000", 2024-09-18T08:10:02.198 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 335.70237606900002, 2024-09-18T08:10:02.198 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.027497293999999999, 2024-09-18T08:10:02.198 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-09-18T08:10:02.199 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-09-18T08:10:02.199 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T08:10:02.199 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2024-09-18T08:10:02.199 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.14805", 2024-09-18T08:10:02.199 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "172.21.15.7:0/640320436", 2024-09-18T08:10:02.199 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 43731 2024-09-18T08:10:02.199 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:10:02.199 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-09-18T08:10:02.199 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:10:02.199 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-09-18T08:10:02.199 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:26.432079+0000", 2024-09-18T08:10:02.199 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-18T08:10:02.199 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:10:02.200 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:10:02.200 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-09-18T08:10:02.200 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:26.432079+0000", 2024-09-18T08:10:02.200 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-18T08:10:02.200 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:10:02.200 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:10:02.200 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-09-18T08:10:02.200 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:26.432080+0000", 2024-09-18T08:10:02.200 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.128e-06 2024-09-18T08:10:02.200 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:10:02.200 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:10:02.200 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-09-18T08:10:02.200 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:26.432153+0000", 2024-09-18T08:10:02.200 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 7.2521999999999994e-05 2024-09-18T08:10:02.201 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:10:02.201 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:10:02.201 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-09-18T08:10:02.201 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:26.432163+0000", 2024-09-18T08:10:02.201 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.0115999999999999e-05 2024-09-18T08:10:02.201 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:10:02.201 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:10:02.201 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-09-18T08:10:02.201 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:26.432170+0000", 2024-09-18T08:10:02.201 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 7.0729999999999999e-06 2024-09-18T08:10:02.201 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:10:02.201 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:10:02.201 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-18T08:10:02.201 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:26.432225+0000", 2024-09-18T08:10:02.202 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 5.4463999999999999e-05 2024-09-18T08:10:02.202 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:10:02.202 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:10:02.202 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-09-18T08:10:02.202 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:26.432342+0000", 2024-09-18T08:10:02.202 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000117118 2024-09-18T08:10:02.202 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:10:02.202 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:10:02.202 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "waiting for subops from 0,1", 2024-09-18T08:10:02.202 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:26.432467+0000", 2024-09-18T08:10:02.202 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00012505700000000001 2024-09-18T08:10:02.202 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:10:02.202 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:10:02.203 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "op_commit", 2024-09-18T08:10:02.203 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:26.433881+0000", 2024-09-18T08:10:02.203 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.001413812 2024-09-18T08:10:02.203 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:10:02.203 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:10:02.203 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:10:02.203 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:26.434589+0000", 2024-09-18T08:10:02.203 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00070883600000000001 2024-09-18T08:10:02.203 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:10:02.203 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:10:02.203 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:10:02.203 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:26.459469+0000", 2024-09-18T08:10:02.203 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.024879951000000001 2024-09-18T08:10:02.203 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:10:02.203 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:10:02.204 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "commit_sent", 2024-09-18T08:10:02.204 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:26.459527+0000", 2024-09-18T08:10:02.204 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 5.7126999999999997e-05 2024-09-18T08:10:02.204 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:10:02.204 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:10:02.204 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-09-18T08:10:02.204 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:26.459577+0000", 2024-09-18T08:10:02.204 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 5.0090000000000003e-05 2024-09-18T08:10:02.204 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-18T08:10:02.205 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-09-18T08:10:02.205 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-18T08:10:02.205 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:10:02.205 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:10:02.205 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.14805.0:63861 7.d 7:bf91474b:::benchmark_data_smithi007_73155_object63860:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e100)", 2024-09-18T08:10:02.205 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-09-18T08:04:41.125989+0000", 2024-09-18T08:10:02.205 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 321.00846677700002, 2024-09-18T08:10:02.205 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.031566575999999999, 2024-09-18T08:10:02.205 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-09-18T08:10:02.205 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-09-18T08:10:02.205 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T08:10:02.205 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2024-09-18T08:10:02.205 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.14805", 2024-09-18T08:10:02.206 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "172.21.15.7:0/640320436", 2024-09-18T08:10:02.206 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 63861 2024-09-18T08:10:02.206 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:10:02.206 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-09-18T08:10:02.206 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:10:02.206 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-09-18T08:10:02.206 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:41.125989+0000", 2024-09-18T08:10:02.206 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-18T08:10:02.206 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:10:02.206 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:10:02.206 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-09-18T08:10:02.206 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:41.125989+0000", 2024-09-18T08:10:02.206 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-18T08:10:02.206 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:10:02.207 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:10:02.207 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-09-18T08:10:02.207 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:41.125990+0000", 2024-09-18T08:10:02.207 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.6109999999999999e-06 2024-09-18T08:10:02.207 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:10:02.207 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:10:02.207 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-09-18T08:10:02.207 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:41.126017+0000", 2024-09-18T08:10:02.207 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 2.6594999999999999e-05 2024-09-18T08:10:02.207 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:10:02.207 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:10:02.207 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-09-18T08:10:02.208 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:41.126035+0000", 2024-09-18T08:10:02.208 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.7842999999999998e-05 2024-09-18T08:10:02.208 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:10:02.208 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:10:02.208 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-09-18T08:10:02.208 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:41.126043+0000", 2024-09-18T08:10:02.208 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 8.8489999999999995e-06 2024-09-18T08:10:02.208 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:10:02.208 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:10:02.208 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-18T08:10:02.208 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:41.126104+0000", 2024-09-18T08:10:02.208 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 6.0448999999999997e-05 2024-09-18T08:10:02.208 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:10:02.208 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:10:02.209 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-09-18T08:10:02.209 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:41.126225+0000", 2024-09-18T08:10:02.209 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000121101 2024-09-18T08:10:02.209 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:10:02.209 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:10:02.209 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "waiting for subops from 0,1", 2024-09-18T08:10:02.209 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:41.126366+0000", 2024-09-18T08:10:02.209 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00014059899999999999 2024-09-18T08:10:02.209 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:10:02.209 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:10:02.209 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "op_commit", 2024-09-18T08:10:02.209 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:41.129750+0000", 2024-09-18T08:10:02.209 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.003383863 2024-09-18T08:10:02.209 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:10:02.209 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:10:02.210 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:10:02.210 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:41.140266+0000", 2024-09-18T08:10:02.210 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.010516266999999999 2024-09-18T08:10:02.212 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:10:02.213 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:10:02.213 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:10:02.213 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:41.157441+0000", 2024-09-18T08:10:02.213 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.017174891000000001 2024-09-18T08:10:02.213 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:10:02.213 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:10:02.213 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "commit_sent", 2024-09-18T08:10:02.213 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:41.157509+0000", 2024-09-18T08:10:02.213 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 6.8627999999999994e-05 2024-09-18T08:10:02.213 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:10:02.213 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:10:02.213 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-09-18T08:10:02.213 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:41.157555+0000", 2024-09-18T08:10:02.213 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4.5880000000000001e-05 2024-09-18T08:10:02.214 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-18T08:10:02.214 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-09-18T08:10:02.214 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-18T08:10:02.214 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:10:02.214 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:10:02.214 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.14805.0:63862 7.d 7:b1f318bc:::benchmark_data_smithi007_73155_object63861:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e100)", 2024-09-18T08:10:02.214 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-09-18T08:04:41.126934+0000", 2024-09-18T08:10:02.214 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 321.00752163999999, 2024-09-18T08:10:02.214 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.030729600999999999, 2024-09-18T08:10:02.214 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-09-18T08:10:02.214 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-09-18T08:10:02.214 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T08:10:02.214 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2024-09-18T08:10:02.214 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.14805", 2024-09-18T08:10:02.215 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "172.21.15.7:0/640320436", 2024-09-18T08:10:02.215 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 63862 2024-09-18T08:10:02.215 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:10:02.215 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-09-18T08:10:02.215 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:10:02.215 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-09-18T08:10:02.215 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:41.126934+0000", 2024-09-18T08:10:02.215 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-18T08:10:02.215 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:10:02.215 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:10:02.215 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-09-18T08:10:02.215 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:41.126934+0000", 2024-09-18T08:10:02.215 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-18T08:10:02.215 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:10:02.216 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:10:02.216 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-09-18T08:10:02.216 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:41.126935+0000", 2024-09-18T08:10:02.216 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 7.9599999999999998e-07 2024-09-18T08:10:02.216 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:10:02.216 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:10:02.216 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-09-18T08:10:02.216 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:41.126989+0000", 2024-09-18T08:10:02.216 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 5.4783e-05 2024-09-18T08:10:02.216 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:10:02.216 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:10:02.216 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-09-18T08:10:02.216 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:41.126999+0000", 2024-09-18T08:10:02.216 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 9.8889999999999997e-06 2024-09-18T08:10:02.217 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:10:02.217 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:10:02.217 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-09-18T08:10:02.217 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:41.127007+0000", 2024-09-18T08:10:02.217 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 7.4399999999999999e-06 2024-09-18T08:10:02.217 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:10:02.217 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:10:02.217 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-18T08:10:02.217 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:41.128171+0000", 2024-09-18T08:10:02.217 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0011642600000000001 2024-09-18T08:10:02.218 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:10:02.218 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:10:02.218 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-09-18T08:10:02.218 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:41.128287+0000", 2024-09-18T08:10:02.218 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000116508 2024-09-18T08:10:02.218 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:10:02.218 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:10:02.218 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "waiting for subops from 0,1", 2024-09-18T08:10:02.218 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:41.128375+0000", 2024-09-18T08:10:02.218 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 8.7300999999999996e-05 2024-09-18T08:10:02.218 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:10:02.218 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:10:02.218 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "op_commit", 2024-09-18T08:10:02.218 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:41.130919+0000", 2024-09-18T08:10:02.219 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0025447450000000002 2024-09-18T08:10:02.219 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:10:02.219 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:10:02.219 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:10:02.219 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:41.140334+0000", 2024-09-18T08:10:02.219 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0094146549999999992 2024-09-18T08:10:02.219 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:10:02.219 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:10:02.219 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:10:02.219 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:41.157602+0000", 2024-09-18T08:10:02.219 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.017267966999999999 2024-09-18T08:10:02.219 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:10:02.220 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:10:02.220 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "commit_sent", 2024-09-18T08:10:02.220 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:41.157636+0000", 2024-09-18T08:10:02.220 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 3.4214000000000001e-05 2024-09-18T08:10:02.220 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:10:02.220 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:10:02.220 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-09-18T08:10:02.220 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:41.157663+0000", 2024-09-18T08:10:02.220 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 2.7042999999999998e-05 2024-09-18T08:10:02.220 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-18T08:10:02.220 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-09-18T08:10:02.220 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-18T08:10:02.221 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:10:02.221 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:10:02.221 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.14805.0:63864 7.d 7:bb7b1761:::benchmark_data_smithi007_73155_object63863:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e100)", 2024-09-18T08:10:02.221 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-09-18T08:04:41.127581+0000", 2024-09-18T08:10:02.221 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 321.00687417699999, 2024-09-18T08:10:02.221 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.030231638000000002, 2024-09-18T08:10:02.221 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-09-18T08:10:02.221 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-09-18T08:10:02.221 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T08:10:02.221 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2024-09-18T08:10:02.221 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.14805", 2024-09-18T08:10:02.221 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "172.21.15.7:0/640320436", 2024-09-18T08:10:02.221 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 63864 2024-09-18T08:10:02.222 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:10:02.222 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-09-18T08:10:02.222 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:10:02.222 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-09-18T08:10:02.222 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:41.127581+0000", 2024-09-18T08:10:02.222 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-18T08:10:02.222 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:10:02.222 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:10:02.222 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-09-18T08:10:02.222 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:41.127581+0000", 2024-09-18T08:10:02.222 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-18T08:10:02.222 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:10:02.222 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:10:02.222 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-09-18T08:10:02.222 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:41.127582+0000", 2024-09-18T08:10:02.223 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 7.8999999999999995e-07 2024-09-18T08:10:02.223 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:10:02.223 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:10:02.223 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-09-18T08:10:02.223 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:41.127648+0000", 2024-09-18T08:10:02.223 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 6.5618999999999996e-05 2024-09-18T08:10:02.223 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:10:02.223 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:10:02.223 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-09-18T08:10:02.223 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:41.127658+0000", 2024-09-18T08:10:02.223 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.0192e-05 2024-09-18T08:10:02.223 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:10:02.223 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:10:02.223 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-09-18T08:10:02.224 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:41.127663+0000", 2024-09-18T08:10:02.224 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 5.2800000000000003e-06 2024-09-18T08:10:02.224 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:10:02.224 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:10:02.224 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-18T08:10:02.224 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:41.129783+0000", 2024-09-18T08:10:02.224 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0021199740000000002 2024-09-18T08:10:02.224 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:10:02.224 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:10:02.224 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-09-18T08:10:02.224 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:41.129885+0000", 2024-09-18T08:10:02.224 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000101987 2024-09-18T08:10:02.224 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:10:02.224 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:10:02.225 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "waiting for subops from 0,1", 2024-09-18T08:10:02.225 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:41.129980+0000", 2024-09-18T08:10:02.225 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 9.4480000000000003e-05 2024-09-18T08:10:02.225 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:10:02.225 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:10:02.225 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "op_commit", 2024-09-18T08:10:02.225 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:41.131035+0000", 2024-09-18T08:10:02.225 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.001055884 2024-09-18T08:10:02.225 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:10:02.225 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:10:02.225 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:10:02.225 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:41.140376+0000", 2024-09-18T08:10:02.225 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0093401439999999999 2024-09-18T08:10:02.225 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:10:02.226 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:10:02.226 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:10:02.226 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:41.157713+0000", 2024-09-18T08:10:02.226 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.017337676999999999 2024-09-18T08:10:02.226 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:10:02.226 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:10:02.226 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "commit_sent", 2024-09-18T08:10:02.226 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:41.157771+0000", 2024-09-18T08:10:02.226 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 5.7674999999999999e-05 2024-09-18T08:10:02.226 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:10:02.226 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:10:02.226 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-09-18T08:10:02.226 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:41.157813+0000", 2024-09-18T08:10:02.227 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4.1936e-05 2024-09-18T08:10:02.227 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-18T08:10:02.227 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-09-18T08:10:02.227 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-18T08:10:02.227 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:10:02.227 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:10:02.227 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.14805.0:63883 7.7 7:ebafa8b8:::benchmark_data_smithi007_73155_object63882:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e100)", 2024-09-18T08:10:02.227 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-09-18T08:04:41.154209+0000", 2024-09-18T08:10:02.227 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 320.98024598500001, 2024-09-18T08:10:02.227 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.042338318999999999, 2024-09-18T08:10:02.227 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-09-18T08:10:02.227 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-09-18T08:10:02.227 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T08:10:02.227 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2024-09-18T08:10:02.227 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.14805", 2024-09-18T08:10:02.228 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "172.21.15.7:0/640320436", 2024-09-18T08:10:02.228 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 63883 2024-09-18T08:10:02.228 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:10:02.228 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-09-18T08:10:02.228 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:10:02.228 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-09-18T08:10:02.228 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:41.154209+0000", 2024-09-18T08:10:02.228 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-18T08:10:02.228 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:10:02.228 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:10:02.228 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-09-18T08:10:02.228 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:41.154209+0000", 2024-09-18T08:10:02.228 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-18T08:10:02.229 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:10:02.229 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:10:02.229 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-09-18T08:10:02.229 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:41.154211+0000", 2024-09-18T08:10:02.229 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.2109999999999999e-06 2024-09-18T08:10:02.229 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:10:02.229 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:10:02.229 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-09-18T08:10:02.229 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:41.154259+0000", 2024-09-18T08:10:02.229 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4.8412000000000003e-05 2024-09-18T08:10:02.229 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:10:02.229 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:10:02.229 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-09-18T08:10:02.229 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:41.154270+0000", 2024-09-18T08:10:02.230 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.0515e-05 2024-09-18T08:10:02.230 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:10:02.230 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:10:02.230 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-09-18T08:10:02.230 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:41.154275+0000", 2024-09-18T08:10:02.230 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 5.4469999999999997e-06 2024-09-18T08:10:02.230 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:10:02.230 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:10:02.230 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-18T08:10:02.230 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:41.154323+0000", 2024-09-18T08:10:02.230 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4.8297999999999997e-05 2024-09-18T08:10:02.230 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:10:02.230 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:10:02.230 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-09-18T08:10:02.231 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:41.154456+0000", 2024-09-18T08:10:02.231 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000132593 2024-09-18T08:10:02.231 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:10:02.231 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:10:02.231 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "waiting for subops from 0,3", 2024-09-18T08:10:02.231 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:41.154615+0000", 2024-09-18T08:10:02.231 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000159049 2024-09-18T08:10:02.231 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:10:02.231 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:10:02.231 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "op_commit", 2024-09-18T08:10:02.231 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:41.156649+0000", 2024-09-18T08:10:02.231 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0020336709999999999 2024-09-18T08:10:02.231 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:10:02.231 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:10:02.232 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:10:02.232 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:41.165163+0000", 2024-09-18T08:10:02.232 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0085141100000000001 2024-09-18T08:10:02.232 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:10:02.232 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:10:02.232 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:10:02.232 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:41.196483+0000", 2024-09-18T08:10:02.232 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.031320300000000002 2024-09-18T08:10:02.232 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:10:02.232 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:10:02.232 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "commit_sent", 2024-09-18T08:10:02.232 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:41.196519+0000", 2024-09-18T08:10:02.232 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 3.6109999999999998e-05 2024-09-18T08:10:02.232 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:10:02.233 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:10:02.233 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-09-18T08:10:02.233 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:41.196548+0000", 2024-09-18T08:10:02.233 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 2.8603e-05 2024-09-18T08:10:02.233 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-18T08:10:02.233 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-09-18T08:10:02.233 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-18T08:10:02.233 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:10:02.233 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:10:02.233 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.14805.0:63908 7.d 7:bd06bac1:::benchmark_data_smithi007_73155_object63907:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e100)", 2024-09-18T08:10:02.233 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-09-18T08:04:41.183375+0000", 2024-09-18T08:10:02.233 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 320.95108022300002, 2024-09-18T08:10:02.233 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.033452764000000003, 2024-09-18T08:10:02.234 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-09-18T08:10:02.234 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-09-18T08:10:02.234 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T08:10:02.234 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2024-09-18T08:10:02.234 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.14805", 2024-09-18T08:10:02.234 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "172.21.15.7:0/640320436", 2024-09-18T08:10:02.234 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 63908 2024-09-18T08:10:02.234 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:10:02.234 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-09-18T08:10:02.234 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:10:02.234 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-09-18T08:10:02.234 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:41.183375+0000", 2024-09-18T08:10:02.234 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-18T08:10:02.235 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:10:02.235 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:10:02.235 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-09-18T08:10:02.235 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:41.183375+0000", 2024-09-18T08:10:02.235 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-18T08:10:02.235 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:10:02.235 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:10:02.235 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-09-18T08:10:02.235 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:41.183376+0000", 2024-09-18T08:10:02.235 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.226e-06 2024-09-18T08:10:02.235 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:10:02.235 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:10:02.235 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-09-18T08:10:02.235 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:41.183403+0000", 2024-09-18T08:10:02.236 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 2.6404e-05 2024-09-18T08:10:02.236 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:10:02.236 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:10:02.236 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-09-18T08:10:02.236 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:41.183417+0000", 2024-09-18T08:10:02.236 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.4399000000000001e-05 2024-09-18T08:10:02.236 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:10:02.236 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:10:02.236 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-09-18T08:10:02.236 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:41.183425+0000", 2024-09-18T08:10:02.236 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 8.1459999999999996e-06 2024-09-18T08:10:02.236 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:10:02.236 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:10:02.236 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-18T08:10:02.237 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:41.183485+0000", 2024-09-18T08:10:02.237 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 5.9673999999999998e-05 2024-09-18T08:10:02.237 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:10:02.237 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:10:02.237 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-09-18T08:10:02.237 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:41.183587+0000", 2024-09-18T08:10:02.237 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000101611 2024-09-18T08:10:02.237 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:10:02.237 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:10:02.237 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "waiting for subops from 0,1", 2024-09-18T08:10:02.237 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:41.183686+0000", 2024-09-18T08:10:02.237 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 9.9823999999999998e-05 2024-09-18T08:10:02.237 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:10:02.238 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:10:02.238 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "op_commit", 2024-09-18T08:10:02.238 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:41.185240+0000", 2024-09-18T08:10:02.238 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0015539340000000001 2024-09-18T08:10:02.238 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:10:02.238 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:10:02.238 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:10:02.238 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:41.191524+0000", 2024-09-18T08:10:02.238 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0062835690000000001 2024-09-18T08:10:02.238 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:10:02.238 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:10:02.238 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:10:02.238 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:41.216746+0000", 2024-09-18T08:10:02.238 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.025222364000000001 2024-09-18T08:10:02.238 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:10:02.239 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:10:02.239 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "commit_sent", 2024-09-18T08:10:02.239 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:41.216782+0000", 2024-09-18T08:10:02.239 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 3.6069999999999999e-05 2024-09-18T08:10:02.239 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:10:02.239 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:10:02.239 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-09-18T08:10:02.239 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:41.216828+0000", 2024-09-18T08:10:02.239 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4.5543000000000001e-05 2024-09-18T08:10:02.239 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-18T08:10:02.239 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-09-18T08:10:02.239 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-18T08:10:02.239 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:10:02.239 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:10:02.240 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.14805.0:63960 7.c 7:30cc2060:::benchmark_data_smithi007_73155_object63959:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e100)", 2024-09-18T08:10:02.240 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-09-18T08:04:41.252452+0000", 2024-09-18T08:10:02.240 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 320.88200349700003, 2024-09-18T08:10:02.240 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.027634619999999999, 2024-09-18T08:10:02.240 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-09-18T08:10:02.240 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-09-18T08:10:02.240 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T08:10:02.240 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2024-09-18T08:10:02.240 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.14805", 2024-09-18T08:10:02.240 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "172.21.15.7:0/640320436", 2024-09-18T08:10:02.240 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 63960 2024-09-18T08:10:02.240 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:10:02.240 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-09-18T08:10:02.240 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:10:02.241 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-09-18T08:10:02.241 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:41.252452+0000", 2024-09-18T08:10:02.241 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-18T08:10:02.241 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:10:02.241 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:10:02.241 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-09-18T08:10:02.241 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:41.252452+0000", 2024-09-18T08:10:02.241 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-18T08:10:02.241 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:10:02.241 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:10:02.241 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-09-18T08:10:02.241 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:41.252453+0000", 2024-09-18T08:10:02.241 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.063e-06 2024-09-18T08:10:02.241 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:10:02.242 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:10:02.242 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7 2024-09-18T08:10:02.243 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-09-18T08:10:02.243 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:41.252503+0000", 2024-09-18T08:10:02.243 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 5.024e-05 2024-09-18T08:10:02.243 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:10:02.243 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:10:02.243 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-09-18T08:10:02.243 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:41.252513+0000", 2024-09-18T08:10:02.243 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.0137e-05 2024-09-18T08:10:02.244 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:10:02.244 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:10:02.244 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-09-18T08:10:02.244 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:41.252519+0000", 2024-09-18T08:10:02.244 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 5.2630000000000003e-06 2024-09-18T08:10:02.244 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:10:02.244 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:10:02.244 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-18T08:10:02.244 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:41.252564+0000", 2024-09-18T08:10:02.244 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4.5528999999999997e-05 2024-09-18T08:10:02.244 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:10:02.245 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:10:02.245 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-09-18T08:10:02.245 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:41.252645+0000", 2024-09-18T08:10:02.245 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 8.1118000000000005e-05 2024-09-18T08:10:02.245 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:10:02.245 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:10:02.245 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "waiting for subops from 3,7", 2024-09-18T08:10:02.245 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:41.252726+0000", 2024-09-18T08:10:02.245 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 8.0912000000000001e-05 2024-09-18T08:10:02.245 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:10:02.246 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:10:02.246 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "op_commit", 2024-09-18T08:10:02.246 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:41.254200+0000", 2024-09-18T08:10:02.246 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0014738010000000001 2024-09-18T08:10:02.246 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:10:02.246 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:10:02.246 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:10:02.246 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:41.254525+0000", 2024-09-18T08:10:02.246 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00032529599999999998 2024-09-18T08:10:02.246 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:10:02.246 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:10:02.246 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:10:02.247 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:41.280019+0000", 2024-09-18T08:10:02.247 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.025493512999999999 2024-09-18T08:10:02.247 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:10:02.247 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:10:02.247 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "commit_sent", 2024-09-18T08:10:02.247 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:41.280057+0000", 2024-09-18T08:10:02.247 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 3.8287000000000001e-05 2024-09-18T08:10:02.247 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:10:02.247 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:10:02.247 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-09-18T08:10:02.247 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:41.280086+0000", 2024-09-18T08:10:02.247 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 2.9461000000000001e-05 2024-09-18T08:10:02.248 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-18T08:10:02.248 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-09-18T08:10:02.248 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-18T08:10:02.248 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:10:02.248 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:10:02.248 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.14805.0:65051 7.d 7:bcc5331c:::benchmark_data_smithi007_73155_object65050:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e100)", 2024-09-18T08:10:02.248 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-09-18T08:04:42.227551+0000", 2024-09-18T08:10:02.248 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 319.906904736, 2024-09-18T08:10:02.248 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.057084449000000002, 2024-09-18T08:10:02.248 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-09-18T08:10:02.249 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-09-18T08:10:02.249 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "waiting for sub ops", 2024-09-18T08:10:02.249 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2024-09-18T08:10:02.249 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.14805", 2024-09-18T08:10:02.249 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "172.21.15.7:0/640320436", 2024-09-18T08:10:02.249 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 65051 2024-09-18T08:10:02.249 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:10:02.249 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-09-18T08:10:02.249 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:10:02.249 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-09-18T08:10:02.249 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:42.227551+0000", 2024-09-18T08:10:02.249 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-18T08:10:02.249 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:10:02.250 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:10:02.250 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-09-18T08:10:02.250 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:42.227551+0000", 2024-09-18T08:10:02.250 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-18T08:10:02.250 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:10:02.250 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:10:02.250 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-09-18T08:10:02.250 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:42.227552+0000", 2024-09-18T08:10:02.250 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.5319999999999999e-06 2024-09-18T08:10:02.250 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:10:02.250 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:10:02.250 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-09-18T08:10:02.250 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:42.227606+0000", 2024-09-18T08:10:02.250 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 5.4290999999999999e-05 2024-09-18T08:10:02.251 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:10:02.251 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:10:02.251 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-09-18T08:10:02.251 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:42.227619+0000", 2024-09-18T08:10:02.251 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.2937e-05 2024-09-18T08:10:02.251 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:10:02.251 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:10:02.251 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-09-18T08:10:02.251 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:42.227627+0000", 2024-09-18T08:10:02.251 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 7.9340000000000006e-06 2024-09-18T08:10:02.251 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:10:02.251 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:10:02.251 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-18T08:10:02.251 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:42.227681+0000", 2024-09-18T08:10:02.252 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 5.3771999999999999e-05 2024-09-18T08:10:02.252 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:10:02.252 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:10:02.252 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-09-18T08:10:02.252 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:42.227809+0000", 2024-09-18T08:10:02.252 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00012750399999999999 2024-09-18T08:10:02.252 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:10:02.252 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:10:02.252 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "waiting for subops from 0,1", 2024-09-18T08:10:02.252 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:42.227961+0000", 2024-09-18T08:10:02.252 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00015280599999999999 2024-09-18T08:10:02.252 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:10:02.252 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:10:02.253 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "op_commit", 2024-09-18T08:10:02.253 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:42.229971+0000", 2024-09-18T08:10:02.253 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0020099179999999999 2024-09-18T08:10:02.253 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:10:02.253 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:10:02.253 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:10:02.253 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:42.232646+0000", 2024-09-18T08:10:02.253 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0026748359999999999 2024-09-18T08:10:02.253 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:10:02.253 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:10:02.253 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-09-18T08:10:02.253 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:42.284635+0000", 2024-09-18T08:10:02.254 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.051988919000000001 2024-09-18T08:10:02.254 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-18T08:10:02.254 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-09-18T08:10:02.254 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-18T08:10:02.254 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:10:02.254 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:10:02.254 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.14805.0:65057 7.d 7:bffbe46c:::benchmark_data_smithi007_73155_object65056:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e100)", 2024-09-18T08:10:02.254 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-09-18T08:04:42.229929+0000", 2024-09-18T08:10:02.254 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 319.90452604000001, 2024-09-18T08:10:02.254 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.054704183000000003, 2024-09-18T08:10:02.254 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-09-18T08:10:02.254 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-09-18T08:10:02.254 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "waiting for sub ops", 2024-09-18T08:10:02.254 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2024-09-18T08:10:02.255 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.14805", 2024-09-18T08:10:02.255 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "172.21.15.7:0/640320436", 2024-09-18T08:10:02.255 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 65057 2024-09-18T08:10:02.255 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:10:02.255 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-09-18T08:10:02.255 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:10:02.255 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-09-18T08:10:02.255 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:42.229929+0000", 2024-09-18T08:10:02.255 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-18T08:10:02.255 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:10:02.255 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:10:02.255 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-09-18T08:10:02.256 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:42.229929+0000", 2024-09-18T08:10:02.256 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-18T08:10:02.256 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:10:02.256 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:10:02.256 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-09-18T08:10:02.256 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:42.229930+0000", 2024-09-18T08:10:02.256 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.0219999999999999e-06 2024-09-18T08:10:02.256 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:10:02.256 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:10:02.256 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-09-18T08:10:02.256 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:42.229973+0000", 2024-09-18T08:10:02.256 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4.2904000000000001e-05 2024-09-18T08:10:02.256 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:10:02.256 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:10:02.257 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-09-18T08:10:02.257 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:42.229987+0000", 2024-09-18T08:10:02.257 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.3405e-05 2024-09-18T08:10:02.257 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:10:02.257 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:10:02.257 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-09-18T08:10:02.257 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:42.229996+0000", 2024-09-18T08:10:02.257 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 9.6299999999999993e-06 2024-09-18T08:10:02.257 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:10:02.257 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:10:02.257 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-18T08:10:02.257 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:42.230048+0000", 2024-09-18T08:10:02.257 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 5.1839000000000003e-05 2024-09-18T08:10:02.257 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:10:02.258 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:10:02.258 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-09-18T08:10:02.258 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:42.230146+0000", 2024-09-18T08:10:02.258 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 9.8101000000000001e-05 2024-09-18T08:10:02.258 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:10:02.258 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:10:02.258 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "waiting for subops from 0,1", 2024-09-18T08:10:02.258 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:42.230237+0000", 2024-09-18T08:10:02.258 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 9.0634e-05 2024-09-18T08:10:02.258 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:10:02.258 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:10:02.258 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "op_commit", 2024-09-18T08:10:02.258 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:42.231752+0000", 2024-09-18T08:10:02.259 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.001515548 2024-09-18T08:10:02.259 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:10:02.259 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:10:02.259 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:10:02.259 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:42.232681+0000", 2024-09-18T08:10:02.259 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00092884 2024-09-18T08:10:02.259 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:10:02.259 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:10:02.259 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-09-18T08:10:02.259 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:42.284634+0000", 2024-09-18T08:10:02.259 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.05195226 2024-09-18T08:10:02.259 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-18T08:10:02.259 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-09-18T08:10:02.259 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-18T08:10:02.260 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:10:02.260 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:10:02.260 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.14805.0:65062 7.d 7:b1e686a2:::benchmark_data_smithi007_73155_object65061:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e100)", 2024-09-18T08:10:02.260 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-09-18T08:04:42.236979+0000", 2024-09-18T08:10:02.260 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 319.89747672499999, 2024-09-18T08:10:02.260 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.047653305999999999, 2024-09-18T08:10:02.260 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-09-18T08:10:02.260 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-09-18T08:10:02.260 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "waiting for sub ops", 2024-09-18T08:10:02.260 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2024-09-18T08:10:02.260 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.14805", 2024-09-18T08:10:02.260 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "172.21.15.7:0/640320436", 2024-09-18T08:10:02.260 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 65062 2024-09-18T08:10:02.261 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:10:02.261 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-09-18T08:10:02.261 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:10:02.261 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-09-18T08:10:02.261 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:42.236979+0000", 2024-09-18T08:10:02.261 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-18T08:10:02.261 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:10:02.261 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:10:02.261 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-09-18T08:10:02.261 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:42.236979+0000", 2024-09-18T08:10:02.261 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-18T08:10:02.261 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:10:02.261 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:10:02.261 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-09-18T08:10:02.262 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:42.236980+0000", 2024-09-18T08:10:02.262 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.124e-06 2024-09-18T08:10:02.262 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:10:02.262 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:10:02.262 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-09-18T08:10:02.262 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:42.237029+0000", 2024-09-18T08:10:02.262 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4.9450999999999998e-05 2024-09-18T08:10:02.262 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:10:02.262 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:10:02.262 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-09-18T08:10:02.262 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:42.237041+0000", 2024-09-18T08:10:02.262 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.1418e-05 2024-09-18T08:10:02.262 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:10:02.263 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:10:02.263 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-09-18T08:10:02.263 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:42.237049+0000", 2024-09-18T08:10:02.263 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 8.0059999999999992e-06 2024-09-18T08:10:02.263 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:10:02.263 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:10:02.263 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-18T08:10:02.263 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:42.237104+0000", 2024-09-18T08:10:02.263 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 5.4870999999999997e-05 2024-09-18T08:10:02.263 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:10:02.263 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:10:02.263 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-09-18T08:10:02.263 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:42.237258+0000", 2024-09-18T08:10:02.263 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000154857 2024-09-18T08:10:02.264 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:10:02.264 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:10:02.264 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "waiting for subops from 0,1", 2024-09-18T08:10:02.264 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:42.237413+0000", 2024-09-18T08:10:02.264 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000154764 2024-09-18T08:10:02.264 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:10:02.264 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:10:02.264 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "op_commit", 2024-09-18T08:10:02.264 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:42.240054+0000", 2024-09-18T08:10:02.264 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0026404549999999999 2024-09-18T08:10:02.264 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:10:02.264 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:10:02.264 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-09-18T08:10:02.264 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:42.284632+0000", 2024-09-18T08:10:02.265 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.044578359999999997 2024-09-18T08:10:02.265 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-18T08:10:02.265 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-09-18T08:10:02.266 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-18T08:10:02.266 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:10:02.266 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:10:02.266 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.14805.0:65063 7.d 7:b037f784:::benchmark_data_smithi007_73155_object65062:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e100)", 2024-09-18T08:10:02.266 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-09-18T08:04:42.237116+0000", 2024-09-18T08:10:02.266 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 319.89733925899998, 2024-09-18T08:10:02.266 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.047512190000000003, 2024-09-18T08:10:02.266 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-09-18T08:10:02.266 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-09-18T08:10:02.266 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "waiting for sub ops", 2024-09-18T08:10:02.266 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2024-09-18T08:10:02.266 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.14805", 2024-09-18T08:10:02.266 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "172.21.15.7:0/640320436", 2024-09-18T08:10:02.266 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 65063 2024-09-18T08:10:02.267 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:10:02.267 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-09-18T08:10:02.267 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:10:02.267 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-09-18T08:10:02.267 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:42.237116+0000", 2024-09-18T08:10:02.267 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-18T08:10:02.267 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:10:02.267 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:10:02.267 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-09-18T08:10:02.267 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:42.237116+0000", 2024-09-18T08:10:02.267 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-18T08:10:02.267 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:10:02.267 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:10:02.268 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-09-18T08:10:02.268 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:42.237117+0000", 2024-09-18T08:10:02.268 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 6.7000000000000004e-07 2024-09-18T08:10:02.268 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:10:02.268 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:10:02.268 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-09-18T08:10:02.268 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:42.237190+0000", 2024-09-18T08:10:02.268 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 7.2972000000000005e-05 2024-09-18T08:10:02.268 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:10:02.268 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:10:02.268 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-09-18T08:10:02.268 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:42.237197+0000", 2024-09-18T08:10:02.268 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 7.2729999999999997e-06 2024-09-18T08:10:02.269 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:10:02.269 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:10:02.269 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-09-18T08:10:02.269 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:42.237203+0000", 2024-09-18T08:10:02.269 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 5.5289999999999999e-06 2024-09-18T08:10:02.269 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:10:02.269 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:10:02.269 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-18T08:10:02.269 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:42.238960+0000", 2024-09-18T08:10:02.269 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.001757759 2024-09-18T08:10:02.269 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:10:02.269 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:10:02.269 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-09-18T08:10:02.269 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:42.239038+0000", 2024-09-18T08:10:02.270 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 7.8054999999999997e-05 2024-09-18T08:10:02.270 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:10:02.270 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:10:02.270 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "waiting for subops from 0,1", 2024-09-18T08:10:02.270 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:42.239122+0000", 2024-09-18T08:10:02.270 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 8.3963999999999998e-05 2024-09-18T08:10:02.270 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:10:02.270 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:10:02.270 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "op_commit", 2024-09-18T08:10:02.270 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:42.240157+0000", 2024-09-18T08:10:02.270 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.001034372 2024-09-18T08:10:02.270 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:10:02.270 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:10:02.270 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-09-18T08:10:02.271 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:42.284628+0000", 2024-09-18T08:10:02.271 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.044471596000000002 2024-09-18T08:10:02.271 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-18T08:10:02.271 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-09-18T08:10:02.271 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-18T08:10:02.271 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-18T08:10:02.271 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-09-18T08:10:02.271 INFO:teuthology.orchestra.run.smithi050.stdout:} 2024-09-18T08:10:02.307 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:10:02.307 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:10:02.308 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7' 2024-09-18T08:10:02.408 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3 2024-09-18T08:10:02.414 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/779349ca-7593-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T08:10:02.463 INFO:teuthology.orchestra.run.smithi007.stdout:{"pg_ready":true,"pg_map":{"version":347,"stamp":"2024-09-18T08:09:59.224858+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":4442561075,"num_objects":67979,"num_object_clones":0,"num_object_copies":203937,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1178,"num_objects_unfound":0,"num_objects_dirty":67979,"num_whiteouts":0,"num_read":908,"num_read_kb":660,"num_write":134156,"num_write_kb":6447982,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":23308,"num_bytes_recovered":1511660754,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":79714,"ondisk_log_size":79714,"up":474,"acting":473,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":481,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":15948696,"kb_used_data":14070740,"kb_used_omap":12,"kb_used_meta":1877811,"kb_avail":734012520,"statfs":{"total":767960285184,"available":751628820480,"internally_reserved":0,"allocated":14408437760,"data_stored":14405254819,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12750,"internal_metadata":1922879026},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,2,0,1],"upper_bound":16},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-767033344,"num_objects":-11704,"num_object_clones":0,"num_object_copies":-35112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-525,"num_objects_unfound":0,"num_objects_dirty":-11704,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":11704,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":525,"num_bytes_recovered":34406400,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.002444"},"pg_stats":[{"pgid":"3.1f","version":"0'0","reported_seq":352,"reported_epoch":216,"state":"active+clean","last_fresh":"2024-09-18T08:09:58.644609+0000","last_change":"2024-09-18T08:08:52.697893+0000","last_active":"2024-09-18T08:09:58.644609+0000","last_peered":"2024-09-18T08:09:58.644609+0000","last_clean":"2024-09-18T08:09:58.644609+0000","last_became_active":"2024-09-18T08:08:52.697499+0000","last_became_peered":"2024-09-18T08:08:52.697499+0000","last_unstale":"2024-09-18T08:09:58.644609+0000","last_undegraded":"2024-09-18T08:09:58.644609+0000","last_fullsized":"2024-09-18T08:09:58.644609+0000","mapping_epoch":198,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":199,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:08:06.642662+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:46.860092+0000","last_clean_scrub_stamp":"2024-09-18T08:08:06.642662+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"69'9","reported_seq":88,"reported_epoch":215,"state":"active+clean","last_fresh":"2024-09-18T08:09:51.557673+0000","last_change":"2024-09-18T08:08:18.575611+0000","last_active":"2024-09-18T08:09:51.557673+0000","last_peered":"2024-09-18T08:09:51.557673+0000","last_clean":"2024-09-18T08:09:51.557673+0000","last_became_active":"2024-09-18T08:08:18.537856+0000","last_became_peered":"2024-09-18T08:08:18.537856+0000","last_unstale":"2024-09-18T08:09:51.557673+0000","last_undegraded":"2024-09-18T08:09:51.557673+0000","last_fullsized":"2024-09-18T08:09:51.557673+0000","mapping_epoch":181,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":182,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'9","last_scrub_stamp":"2024-09-18T08:05:08.696796+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_clean_scrub_stamp":"2024-09-18T08:05:08.696796+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":16,"num_read_kb":10,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":337,"reported_epoch":216,"state":"active+clean","last_fresh":"2024-09-18T08:09:58.660501+0000","last_change":"2024-09-18T08:07:18.737364+0000","last_active":"2024-09-18T08:09:58.660501+0000","last_peered":"2024-09-18T08:09:58.660501+0000","last_clean":"2024-09-18T08:09:58.660501+0000","last_became_active":"2024-09-18T08:02:51.907140+0000","last_became_peered":"2024-09-18T08:02:51.907140+0000","last_unstale":"2024-09-18T08:09:58.660501+0000","last_undegraded":"2024-09-18T08:09:58.660501+0000","last_fullsized":"2024-09-18T08:09:58.660501+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:07:18.737122+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:50.886830+0000","last_clean_scrub_stamp":"2024-09-18T08:07:18.737122+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:10:14.742960+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":78,"reported_epoch":215,"state":"active+clean","last_fresh":"2024-09-18T08:09:51.558377+0000","last_change":"2024-09-18T08:08:18.545192+0000","last_active":"2024-09-18T08:09:51.558377+0000","last_peered":"2024-09-18T08:09:51.558377+0000","last_clean":"2024-09-18T08:09:51.558377+0000","last_became_active":"2024-09-18T08:08:18.540705+0000","last_became_peered":"2024-09-18T08:08:18.540705+0000","last_unstale":"2024-09-18T08:09:51.558377+0000","last_undegraded":"2024-09-18T08:09:51.558377+0000","last_fullsized":"2024-09-18T08:09:51.558377+0000","mapping_epoch":181,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":182,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:07:08.451865+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_clean_scrub_stamp":"2024-09-18T08:07:08.451865+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":340,"reported_epoch":216,"state":"active+clean","last_fresh":"2024-09-18T08:09:58.652981+0000","last_change":"2024-09-18T08:07:13.244647+0000","last_active":"2024-09-18T08:09:58.652981+0000","last_peered":"2024-09-18T08:09:58.652981+0000","last_clean":"2024-09-18T08:09:58.652981+0000","last_became_active":"2024-09-18T08:06:24.128511+0000","last_became_peered":"2024-09-18T08:06:24.128511+0000","last_unstale":"2024-09-18T08:09:58.652981+0000","last_undegraded":"2024-09-18T08:09:58.652981+0000","last_fullsized":"2024-09-18T08:09:58.652981+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:07:13.244555+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_clean_scrub_stamp":"2024-09-18T08:07:13.244555+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:10:55.479885+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":353,"reported_epoch":216,"state":"active+clean","last_fresh":"2024-09-18T08:09:58.659372+0000","last_change":"2024-09-18T08:07:10.257236+0000","last_active":"2024-09-18T08:09:58.659372+0000","last_peered":"2024-09-18T08:09:58.659372+0000","last_clean":"2024-09-18T08:09:58.659372+0000","last_became_active":"2024-09-18T08:06:24.126194+0000","last_became_peered":"2024-09-18T08:06:24.126194+0000","last_unstale":"2024-09-18T08:09:58.659372+0000","last_undegraded":"2024-09-18T08:09:58.659372+0000","last_fullsized":"2024-09-18T08:09:58.659372+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:07:10.257170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:46.860092+0000","last_clean_scrub_stamp":"2024-09-18T08:07:10.257170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":37,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"69'15","reported_seq":351,"reported_epoch":216,"state":"active+clean","last_fresh":"2024-09-18T08:09:58.652968+0000","last_change":"2024-09-18T08:09:32.575836+0000","last_active":"2024-09-18T08:09:58.652968+0000","last_peered":"2024-09-18T08:09:58.652968+0000","last_clean":"2024-09-18T08:09:58.652968+0000","last_became_active":"2024-09-18T08:02:49.886106+0000","last_became_peered":"2024-09-18T08:02:49.886106+0000","last_unstale":"2024-09-18T08:09:58.652968+0000","last_undegraded":"2024-09-18T08:09:58.652968+0000","last_fullsized":"2024-09-18T08:09:58.652968+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'15","last_scrub_stamp":"2024-09-18T08:09:32.575741+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_clean_scrub_stamp":"2024-09-18T08:09:32.575741+0000","objects_scrubbed":7,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:10:59.141685+0000","scrub_duration":40,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":78,"reported_epoch":215,"state":"active+clean","last_fresh":"2024-09-18T08:09:51.557707+0000","last_change":"2024-09-18T08:08:18.543316+0000","last_active":"2024-09-18T08:09:51.557707+0000","last_peered":"2024-09-18T08:09:51.557707+0000","last_clean":"2024-09-18T08:09:51.557707+0000","last_became_active":"2024-09-18T08:08:18.538645+0000","last_became_peered":"2024-09-18T08:08:18.538645+0000","last_unstale":"2024-09-18T08:09:51.557707+0000","last_undegraded":"2024-09-18T08:09:51.557707+0000","last_fullsized":"2024-09-18T08:09:51.557707+0000","mapping_epoch":181,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":182,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:08:09.142188+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:50.886830+0000","last_clean_scrub_stamp":"2024-09-18T08:08:09.142188+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":363,"reported_epoch":215,"state":"active+clean","last_fresh":"2024-09-18T08:09:51.570329+0000","last_change":"2024-09-18T08:08:18.548588+0000","last_active":"2024-09-18T08:09:51.570329+0000","last_peered":"2024-09-18T08:09:51.570329+0000","last_clean":"2024-09-18T08:09:51.570329+0000","last_became_active":"2024-09-18T08:08:18.548203+0000","last_became_peered":"2024-09-18T08:08:18.548203+0000","last_unstale":"2024-09-18T08:09:51.570329+0000","last_undegraded":"2024-09-18T08:09:51.570329+0000","last_fullsized":"2024-09-18T08:09:51.570329+0000","mapping_epoch":181,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":182,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:06:14.027077+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:46.860092+0000","last_clean_scrub_stamp":"2024-09-18T08:06:14.027077+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"69'9","reported_seq":88,"reported_epoch":215,"state":"active+clean","last_fresh":"2024-09-18T08:09:51.558588+0000","last_change":"2024-09-18T08:08:18.631597+0000","last_active":"2024-09-18T08:09:51.558588+0000","last_peered":"2024-09-18T08:09:51.558588+0000","last_clean":"2024-09-18T08:09:51.558588+0000","last_became_active":"2024-09-18T08:08:18.550960+0000","last_became_peered":"2024-09-18T08:08:18.550960+0000","last_unstale":"2024-09-18T08:09:51.558588+0000","last_undegraded":"2024-09-18T08:09:51.558588+0000","last_fullsized":"2024-09-18T08:09:51.558588+0000","mapping_epoch":181,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":182,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_clean_scrub_stamp":"2024-09-18T08:02:48.870718+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":16,"num_read_kb":10,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":327,"reported_epoch":215,"state":"active+clean","last_fresh":"2024-09-18T08:09:51.568496+0000","last_change":"2024-09-18T08:05:33.935844+0000","last_active":"2024-09-18T08:09:51.568496+0000","last_peered":"2024-09-18T08:09:51.568496+0000","last_clean":"2024-09-18T08:09:51.568496+0000","last_became_active":"2024-09-18T08:02:51.907296+0000","last_became_peered":"2024-09-18T08:02:51.907296+0000","last_unstale":"2024-09-18T08:09:51.568496+0000","last_undegraded":"2024-09-18T08:09:51.568496+0000","last_fullsized":"2024-09-18T08:09:51.568496+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:05:33.935798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:50.886830+0000","last_clean_scrub_stamp":"2024-09-18T08:05:33.935798+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":326,"reported_epoch":216,"state":"active+clean","last_fresh":"2024-09-18T08:09:58.642812+0000","last_change":"2024-09-18T08:08:14.646073+0000","last_active":"2024-09-18T08:09:58.642812+0000","last_peered":"2024-09-18T08:09:58.642812+0000","last_clean":"2024-09-18T08:09:58.642812+0000","last_became_active":"2024-09-18T08:02:53.913519+0000","last_became_peered":"2024-09-18T08:02:53.913519+0000","last_unstale":"2024-09-18T08:09:58.642812+0000","last_undegraded":"2024-09-18T08:09:58.642812+0000","last_fullsized":"2024-09-18T08:09:58.642812+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:08:14.645993+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_clean_scrub_stamp":"2024-09-18T08:08:14.645993+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":341,"reported_epoch":215,"state":"active+clean","last_fresh":"2024-09-18T08:09:51.571412+0000","last_change":"2024-09-18T08:08:18.624260+0000","last_active":"2024-09-18T08:09:51.571412+0000","last_peered":"2024-09-18T08:09:51.571412+0000","last_clean":"2024-09-18T08:09:51.571412+0000","last_became_active":"2024-09-18T08:08:18.623757+0000","last_became_peered":"2024-09-18T08:08:18.623757+0000","last_unstale":"2024-09-18T08:09:51.571412+0000","last_undegraded":"2024-09-18T08:09:51.571412+0000","last_fullsized":"2024-09-18T08:09:51.571412+0000","mapping_epoch":181,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":182,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:04:32.849119+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:46.860092+0000","last_clean_scrub_stamp":"2024-09-18T08:04:32.849119+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1b","version":"69'5","reported_seq":84,"reported_epoch":215,"state":"active+clean","last_fresh":"2024-09-18T08:09:51.558337+0000","last_change":"2024-09-18T08:08:18.651427+0000","last_active":"2024-09-18T08:09:51.558337+0000","last_peered":"2024-09-18T08:09:51.558337+0000","last_clean":"2024-09-18T08:09:51.558337+0000","last_became_active":"2024-09-18T08:08:18.567126+0000","last_became_peered":"2024-09-18T08:08:18.567126+0000","last_unstale":"2024-09-18T08:09:51.558337+0000","last_undegraded":"2024-09-18T08:09:51.558337+0000","last_fullsized":"2024-09-18T08:09:51.558337+0000","mapping_epoch":181,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":182,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'5","last_scrub_stamp":"2024-09-18T08:06:09.465979+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_clean_scrub_stamp":"2024-09-18T08:06:09.465979+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":10,"num_read_kb":6,"num_write":6,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":357,"reported_epoch":215,"state":"active+clean","last_fresh":"2024-09-18T08:09:51.569846+0000","last_change":"2024-09-18T08:09:12.674889+0000","last_active":"2024-09-18T08:09:51.569846+0000","last_peered":"2024-09-18T08:09:51.569846+0000","last_clean":"2024-09-18T08:09:51.569846+0000","last_became_active":"2024-09-18T08:08:18.524932+0000","last_became_peered":"2024-09-18T08:08:18.524932+0000","last_unstale":"2024-09-18T08:09:51.569846+0000","last_undegraded":"2024-09-18T08:09:51.569846+0000","last_fullsized":"2024-09-18T08:09:51.569846+0000","mapping_epoch":181,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":182,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:09:12.674843+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:50.886830+0000","last_clean_scrub_stamp":"2024-09-18T08:09:12.674843+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:10:16.027634+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":324,"reported_epoch":215,"state":"active+clean","last_fresh":"2024-09-18T08:09:51.568750+0000","last_change":"2024-09-18T08:06:44.518882+0000","last_active":"2024-09-18T08:09:51.568750+0000","last_peered":"2024-09-18T08:09:51.568750+0000","last_clean":"2024-09-18T08:09:51.568750+0000","last_became_active":"2024-09-18T08:02:53.911648+0000","last_became_peered":"2024-09-18T08:02:53.911648+0000","last_unstale":"2024-09-18T08:09:51.568750+0000","last_undegraded":"2024-09-18T08:09:51.568750+0000","last_fullsized":"2024-09-18T08:09:51.568750+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:06:44.518835+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_clean_scrub_stamp":"2024-09-18T08:06:44.518835+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":78,"reported_epoch":215,"state":"active+clean","last_fresh":"2024-09-18T08:09:51.558770+0000","last_change":"2024-09-18T08:08:18.544203+0000","last_active":"2024-09-18T08:09:51.558770+0000","last_peered":"2024-09-18T08:09:51.558770+0000","last_clean":"2024-09-18T08:09:51.558770+0000","last_became_active":"2024-09-18T08:08:18.539604+0000","last_became_peered":"2024-09-18T08:08:18.539604+0000","last_unstale":"2024-09-18T08:09:51.558770+0000","last_undegraded":"2024-09-18T08:09:51.558770+0000","last_fullsized":"2024-09-18T08:09:51.558770+0000","mapping_epoch":181,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":182,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:05:40.477965+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_clean_scrub_stamp":"2024-09-18T08:05:40.477965+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":351,"reported_epoch":216,"state":"active+clean","last_fresh":"2024-09-18T08:09:58.645139+0000","last_change":"2024-09-18T08:08:18.524124+0000","last_active":"2024-09-18T08:09:58.645139+0000","last_peered":"2024-09-18T08:09:58.645139+0000","last_clean":"2024-09-18T08:09:58.645139+0000","last_became_active":"2024-09-18T08:08:18.523611+0000","last_became_peered":"2024-09-18T08:08:18.523611+0000","last_unstale":"2024-09-18T08:09:58.645139+0000","last_undegraded":"2024-09-18T08:09:58.645139+0000","last_fullsized":"2024-09-18T08:09:58.645139+0000","mapping_epoch":181,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":182,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:05:27.476144+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:46.860092+0000","last_clean_scrub_stamp":"2024-09-18T08:05:27.476144+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:10:23.059202+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1c","version":"69'15","reported_seq":60,"reported_epoch":215,"state":"active+clean","last_fresh":"2024-09-18T08:09:51.579799+0000","last_change":"2024-09-18T08:08:52.708963+0000","last_active":"2024-09-18T08:09:51.579799+0000","last_peered":"2024-09-18T08:09:51.579799+0000","last_clean":"2024-09-18T08:09:51.579799+0000","last_became_active":"2024-09-18T08:08:52.690788+0000","last_became_peered":"2024-09-18T08:08:52.690788+0000","last_unstale":"2024-09-18T08:09:51.579799+0000","last_undegraded":"2024-09-18T08:09:51.579799+0000","last_fullsized":"2024-09-18T08:09:51.579799+0000","mapping_epoch":198,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":199,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'15","last_scrub_stamp":"2024-09-18T08:07:39.530340+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_clean_scrub_stamp":"2024-09-18T08:07:39.530340+0000","objects_scrubbed":7,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":359,"reported_epoch":216,"state":"active+clean","last_fresh":"2024-09-18T08:09:58.661201+0000","last_change":"2024-09-18T08:09:11.693711+0000","last_active":"2024-09-18T08:09:58.661201+0000","last_peered":"2024-09-18T08:09:58.661201+0000","last_clean":"2024-09-18T08:09:58.661201+0000","last_became_active":"2024-09-18T08:08:18.538505+0000","last_became_peered":"2024-09-18T08:08:18.538505+0000","last_unstale":"2024-09-18T08:09:58.661201+0000","last_undegraded":"2024-09-18T08:09:58.661201+0000","last_fullsized":"2024-09-18T08:09:58.661201+0000","mapping_epoch":181,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":182,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:09:11.693643+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:50.886830+0000","last_clean_scrub_stamp":"2024-09-18T08:09:11.693643+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:10:34.429057+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":341,"reported_epoch":216,"state":"active+clean","last_fresh":"2024-09-18T08:09:58.653268+0000","last_change":"2024-09-18T08:09:36.592348+0000","last_active":"2024-09-18T08:09:58.653268+0000","last_peered":"2024-09-18T08:09:58.653268+0000","last_clean":"2024-09-18T08:09:58.653268+0000","last_became_active":"2024-09-18T08:06:24.128659+0000","last_became_peered":"2024-09-18T08:06:24.128659+0000","last_unstale":"2024-09-18T08:09:58.653268+0000","last_undegraded":"2024-09-18T08:09:58.653268+0000","last_fullsized":"2024-09-18T08:09:58.653268+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:09:36.592281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_clean_scrub_stamp":"2024-09-18T08:09:36.592281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:10:46.573780+0000","scrub_duration":16,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":78,"reported_epoch":215,"state":"active+clean","last_fresh":"2024-09-18T08:09:51.558224+0000","last_change":"2024-09-18T08:08:18.536559+0000","last_active":"2024-09-18T08:09:51.558224+0000","last_peered":"2024-09-18T08:09:51.558224+0000","last_clean":"2024-09-18T08:09:51.558224+0000","last_became_active":"2024-09-18T08:08:18.536084+0000","last_became_peered":"2024-09-18T08:08:18.536084+0000","last_unstale":"2024-09-18T08:09:51.558224+0000","last_undegraded":"2024-09-18T08:09:51.558224+0000","last_fullsized":"2024-09-18T08:09:51.558224+0000","mapping_epoch":181,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":182,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:07:13.543722+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:46.860092+0000","last_clean_scrub_stamp":"2024-09-18T08:07:13.543722+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"69'12","reported_seq":395,"reported_epoch":216,"state":"active+clean","last_fresh":"2024-09-18T08:09:58.652629+0000","last_change":"2024-09-18T08:08:18.583915+0000","last_active":"2024-09-18T08:09:58.652629+0000","last_peered":"2024-09-18T08:09:58.652629+0000","last_clean":"2024-09-18T08:09:58.652629+0000","last_became_active":"2024-09-18T08:08:18.543675+0000","last_became_peered":"2024-09-18T08:08:18.543675+0000","last_unstale":"2024-09-18T08:09:58.652629+0000","last_undegraded":"2024-09-18T08:09:58.652629+0000","last_fullsized":"2024-09-18T08:09:58.652629+0000","mapping_epoch":181,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":182,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'12","last_scrub_stamp":"2024-09-18T08:07:20.251529+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_clean_scrub_stamp":"2024-09-18T08:07:20.251529+0000","objects_scrubbed":5,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":25,"num_read_kb":16,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":78,"reported_epoch":215,"state":"active+clean","last_fresh":"2024-09-18T08:09:51.558745+0000","last_change":"2024-09-18T08:08:18.532482+0000","last_active":"2024-09-18T08:09:51.558745+0000","last_peered":"2024-09-18T08:09:51.558745+0000","last_clean":"2024-09-18T08:09:51.558745+0000","last_became_active":"2024-09-18T08:08:18.531438+0000","last_became_peered":"2024-09-18T08:08:18.531438+0000","last_unstale":"2024-09-18T08:09:51.558745+0000","last_undegraded":"2024-09-18T08:09:51.558745+0000","last_fullsized":"2024-09-18T08:09:51.558745+0000","mapping_epoch":181,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":182,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:06:12.434179+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:50.886830+0000","last_clean_scrub_stamp":"2024-09-18T08:06:12.434179+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1c","version":"69'1","reported_seq":167,"reported_epoch":215,"state":"active+clean","last_fresh":"2024-09-18T08:09:51.569684+0000","last_change":"2024-09-18T08:06:46.142143+0000","last_active":"2024-09-18T08:09:51.569684+0000","last_peered":"2024-09-18T08:09:51.569684+0000","last_clean":"2024-09-18T08:09:51.569684+0000","last_became_active":"2024-09-18T08:06:46.141661+0000","last_became_peered":"2024-09-18T08:06:46.141661+0000","last_unstale":"2024-09-18T08:09:51.569684+0000","last_undegraded":"2024-09-18T08:09:51.569684+0000","last_fullsized":"2024-09-18T08:09:51.569684+0000","mapping_epoch":143,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":144,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'1","last_scrub_stamp":"2024-09-18T08:05:36.776369+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_clean_scrub_stamp":"2024-09-18T08:05:36.776369+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":403,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,2],"acting":[5,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"62'1","reported_seq":368,"reported_epoch":216,"state":"active+clean","last_fresh":"2024-09-18T08:09:58.670004+0000","last_change":"2024-09-18T08:09:29.633875+0000","last_active":"2024-09-18T08:09:58.670004+0000","last_peered":"2024-09-18T08:09:58.670004+0000","last_clean":"2024-09-18T08:09:58.670004+0000","last_became_active":"2024-09-18T08:08:18.559125+0000","last_became_peered":"2024-09-18T08:08:18.559125+0000","last_unstale":"2024-09-18T08:09:58.670004+0000","last_undegraded":"2024-09-18T08:09:58.670004+0000","last_fullsized":"2024-09-18T08:09:58.670004+0000","mapping_epoch":181,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":182,"parent":"0.0","parent_split_bits":0,"last_scrub":"62'1","last_scrub_stamp":"2024-09-18T08:09:29.633520+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:46.860092+0000","last_clean_scrub_stamp":"2024-09-18T08:09:29.633520+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:10:38.890672+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":92,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1e","version":"69'10","reported_seq":333,"reported_epoch":216,"state":"active+clean","last_fresh":"2024-09-18T08:09:58.644979+0000","last_change":"2024-09-18T08:02:49.892755+0000","last_active":"2024-09-18T08:09:58.644979+0000","last_peered":"2024-09-18T08:09:58.644979+0000","last_clean":"2024-09-18T08:09:58.644979+0000","last_became_active":"2024-09-18T08:02:49.892024+0000","last_became_peered":"2024-09-18T08:02:49.892024+0000","last_unstale":"2024-09-18T08:09:58.644979+0000","last_undegraded":"2024-09-18T08:09:58.644979+0000","last_fullsized":"2024-09-18T08:09:58.644979+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_clean_scrub_stamp":"2024-09-18T08:02:48.870718+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:10:17.106246+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1f","version":"182'13","reported_seq":103,"reported_epoch":215,"state":"active+clean","last_fresh":"2024-09-18T08:09:53.012728+0000","last_change":"2024-09-18T08:08:18.627670+0000","last_active":"2024-09-18T08:09:53.012728+0000","last_peered":"2024-09-18T08:09:53.012728+0000","last_clean":"2024-09-18T08:09:53.012728+0000","last_became_active":"2024-09-18T08:08:18.543095+0000","last_became_peered":"2024-09-18T08:08:18.543095+0000","last_unstale":"2024-09-18T08:09:53.012728+0000","last_undegraded":"2024-09-18T08:09:53.012728+0000","last_fullsized":"2024-09-18T08:09:53.012728+0000","mapping_epoch":181,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":182,"parent":"0.0","parent_split_bits":0,"last_scrub":"173'12","last_scrub_stamp":"2024-09-18T08:08:07.553075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:50.886830+0000","last_clean_scrub_stamp":"2024-09-18T08:08:07.553075+0000","objects_scrubbed":0,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,7],"acting":[4,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.3","version":"216'8581","reported_seq":8766,"reported_epoch":216,"state":"active+clean","last_fresh":"2024-09-18T08:09:58.676400+0000","last_change":"2024-09-18T08:09:28.263522+0000","last_active":"2024-09-18T08:09:58.676400+0000","last_peered":"2024-09-18T08:09:58.676400+0000","last_clean":"2024-09-18T08:09:58.676400+0000","last_became_active":"2024-09-18T08:07:53.672797+0000","last_became_peered":"2024-09-18T08:07:53.672797+0000","last_unstale":"2024-09-18T08:09:58.676400+0000","last_undegraded":"2024-09-18T08:09:58.676400+0000","last_fullsized":"2024-09-18T08:09:58.676400+0000","mapping_epoch":165,"log_start":"191'3800","ondisk_log_start":"191'3800","created":165,"last_epoch_clean":166,"parent":"0.0","parent_split_bits":0,"last_scrub":"211'6448","last_scrub_stamp":"2024-09-18T08:09:28.263434+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:07:52.307205+0000","last_clean_scrub_stamp":"2024-09-18T08:09:28.263434+0000","objects_scrubbed":6448,"log_size":4781,"log_dups_size":0,"ondisk_log_size":4781,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:10:29.843317+0000","scrub_duration":646,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":282787840,"num_objects":4315,"num_object_clones":0,"num_object_copies":12945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4315,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8581,"num_write_kb":412672,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6],"acting":[1,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.a","version":"74'79","reported_seq":149,"reported_epoch":215,"state":"active+clean","last_fresh":"2024-09-18T08:09:51.571668+0000","last_change":"2024-09-18T08:08:57.463205+0000","last_active":"2024-09-18T08:09:51.571668+0000","last_peered":"2024-09-18T08:09:51.571668+0000","last_clean":"2024-09-18T08:09:51.571668+0000","last_became_active":"2024-09-18T08:07:26.394584+0000","last_became_peered":"2024-09-18T08:07:26.394584+0000","last_unstale":"2024-09-18T08:09:51.571668+0000","last_undegraded":"2024-09-18T08:09:51.571668+0000","last_fullsized":"2024-09-18T08:09:51.571668+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":152,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":4,"last_scrub":"74'79","last_scrub_stamp":"2024-09-18T08:08:57.463113+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:00:11.893742+0000","last_clean_scrub_stamp":"2024-09-18T08:08:57.463113+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:09:57.971184+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,6],"acting":[5,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.f","version":"69'15","reported_seq":405,"reported_epoch":216,"state":"active+clean","last_fresh":"2024-09-18T08:09:58.663337+0000","last_change":"2024-09-18T08:08:56.669262+0000","last_active":"2024-09-18T08:09:58.663337+0000","last_peered":"2024-09-18T08:09:58.663337+0000","last_clean":"2024-09-18T08:09:58.663337+0000","last_became_active":"2024-09-18T08:08:18.540211+0000","last_became_peered":"2024-09-18T08:08:18.540211+0000","last_unstale":"2024-09-18T08:09:58.663337+0000","last_undegraded":"2024-09-18T08:09:58.663337+0000","last_fullsized":"2024-09-18T08:09:58.663337+0000","mapping_epoch":181,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":182,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'15","last_scrub_stamp":"2024-09-18T08:08:56.668397+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_clean_scrub_stamp":"2024-09-18T08:08:56.668397+0000","objects_scrubbed":7,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:10:22.470704+0000","scrub_duration":36,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":339,"reported_epoch":216,"state":"active+clean","last_fresh":"2024-09-18T08:09:58.660505+0000","last_change":"2024-09-18T08:08:18.389332+0000","last_active":"2024-09-18T08:09:58.660505+0000","last_peered":"2024-09-18T08:09:58.660505+0000","last_clean":"2024-09-18T08:09:58.660505+0000","last_became_active":"2024-09-18T08:02:47.879328+0000","last_became_peered":"2024-09-18T08:02:47.879328+0000","last_unstale":"2024-09-18T08:09:58.660505+0000","last_undegraded":"2024-09-18T08:09:58.660505+0000","last_fullsized":"2024-09-18T08:09:58.660505+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:08:18.389273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:46.860092+0000","last_clean_scrub_stamp":"2024-09-18T08:08:18.389273+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.e","version":"182'12","reported_seq":103,"reported_epoch":215,"state":"active+clean","last_fresh":"2024-09-18T08:09:53.012367+0000","last_change":"2024-09-18T08:08:18.560884+0000","last_active":"2024-09-18T08:09:53.012367+0000","last_peered":"2024-09-18T08:09:53.012367+0000","last_clean":"2024-09-18T08:09:53.012367+0000","last_became_active":"2024-09-18T08:08:18.525147+0000","last_became_peered":"2024-09-18T08:08:18.525147+0000","last_unstale":"2024-09-18T08:09:53.012367+0000","last_undegraded":"2024-09-18T08:09:53.012367+0000","last_fullsized":"2024-09-18T08:09:53.012367+0000","mapping_epoch":181,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":182,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'8","last_scrub_stamp":"2024-09-18T08:06:19.459375+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:50.886830+0000","last_clean_scrub_stamp":"2024-09-18T08:06:19.459375+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":333,"reported_epoch":215,"state":"active+clean","last_fresh":"2024-09-18T08:09:51.569038+0000","last_change":"2024-09-18T08:07:19.714477+0000","last_active":"2024-09-18T08:09:51.569038+0000","last_peered":"2024-09-18T08:09:51.569038+0000","last_clean":"2024-09-18T08:09:51.569038+0000","last_became_active":"2024-09-18T08:02:53.909834+0000","last_became_peered":"2024-09-18T08:02:53.909834+0000","last_unstale":"2024-09-18T08:09:51.569038+0000","last_undegraded":"2024-09-18T08:09:51.569038+0000","last_fullsized":"2024-09-18T08:09:51.569038+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:07:19.714430+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_clean_scrub_stamp":"2024-09-18T08:07:19.714430+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.c","version":"216'8345","reported_seq":8458,"reported_epoch":216,"state":"active+clean","last_fresh":"2024-09-18T08:09:58.810981+0000","last_change":"2024-09-18T08:07:53.670091+0000","last_active":"2024-09-18T08:09:58.810981+0000","last_peered":"2024-09-18T08:09:58.810981+0000","last_clean":"2024-09-18T08:09:58.810981+0000","last_became_active":"2024-09-18T08:07:53.669708+0000","last_became_peered":"2024-09-18T08:07:53.669708+0000","last_unstale":"2024-09-18T08:09:58.810981+0000","last_undegraded":"2024-09-18T08:09:58.810981+0000","last_fullsized":"2024-09-18T08:09:58.810981+0000","mapping_epoch":165,"log_start":"184'2900","ondisk_log_start":"184'2900","created":165,"last_epoch_clean":166,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:07:52.307205+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:07:52.307205+0000","last_clean_scrub_stamp":"2024-09-18T08:07:52.307205+0000","objects_scrubbed":0,"log_size":5445,"log_dups_size":0,"ondisk_log_size":5445,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":267714560,"num_objects":4085,"num_object_clones":0,"num_object_copies":12255,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4085,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8345,"num_write_kb":397760,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"74'79","reported_seq":93,"reported_epoch":215,"state":"active+clean","last_fresh":"2024-09-18T08:09:51.558312+0000","last_change":"2024-09-18T08:08:52.690758+0000","last_active":"2024-09-18T08:09:51.558312+0000","last_peered":"2024-09-18T08:09:51.558312+0000","last_clean":"2024-09-18T08:09:51.558312+0000","last_became_active":"2024-09-18T08:08:52.690344+0000","last_became_peered":"2024-09-18T08:08:52.690344+0000","last_unstale":"2024-09-18T08:09:51.558312+0000","last_undegraded":"2024-09-18T08:09:51.558312+0000","last_fullsized":"2024-09-18T08:09:51.558312+0000","mapping_epoch":198,"log_start":"0'0","ondisk_log_start":"0'0","created":152,"last_epoch_clean":199,"parent":"0.0","parent_split_bits":3,"last_scrub":"74'79","last_scrub_stamp":"2024-09-18T08:08:19.036836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:00:11.893742+0000","last_clean_scrub_stamp":"2024-09-18T08:08:19.036836+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,6],"acting":[4,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.0","version":"69'18","reported_seq":367,"reported_epoch":216,"state":"active+clean","last_fresh":"2024-09-18T08:09:58.660258+0000","last_change":"2024-09-18T08:06:44.301130+0000","last_active":"2024-09-18T08:09:58.660258+0000","last_peered":"2024-09-18T08:09:58.660258+0000","last_clean":"2024-09-18T08:09:58.660258+0000","last_became_active":"2024-09-18T08:02:49.886818+0000","last_became_peered":"2024-09-18T08:02:49.886818+0000","last_unstale":"2024-09-18T08:09:58.660258+0000","last_undegraded":"2024-09-18T08:09:58.660258+0000","last_fullsized":"2024-09-18T08:09:58.660258+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'18","last_scrub_stamp":"2024-09-18T08:06:44.300772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_clean_scrub_stamp":"2024-09-18T08:06:44.300772+0000","objects_scrubbed":8,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:10:52.429408+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":34,"num_read_kb":22,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":347,"reported_epoch":216,"state":"active+clean","last_fresh":"2024-09-18T08:09:58.659826+0000","last_change":"2024-09-18T08:07:21.242587+0000","last_active":"2024-09-18T08:09:58.659826+0000","last_peered":"2024-09-18T08:09:58.659826+0000","last_clean":"2024-09-18T08:09:58.659826+0000","last_became_active":"2024-09-18T08:02:47.880225+0000","last_became_peered":"2024-09-18T08:02:47.880225+0000","last_unstale":"2024-09-18T08:09:58.659826+0000","last_undegraded":"2024-09-18T08:09:58.659826+0000","last_fullsized":"2024-09-18T08:09:58.659826+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:07:21.242494+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:46.860092+0000","last_clean_scrub_stamp":"2024-09-18T08:07:21.242494+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":18,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":78,"reported_epoch":215,"state":"active+clean","last_fresh":"2024-09-18T08:09:51.557985+0000","last_change":"2024-09-18T08:08:18.530658+0000","last_active":"2024-09-18T08:09:51.557985+0000","last_peered":"2024-09-18T08:09:51.557985+0000","last_clean":"2024-09-18T08:09:51.557985+0000","last_became_active":"2024-09-18T08:08:18.530341+0000","last_became_peered":"2024-09-18T08:08:18.530341+0000","last_unstale":"2024-09-18T08:09:51.557985+0000","last_undegraded":"2024-09-18T08:09:51.557985+0000","last_fullsized":"2024-09-18T08:09:51.557985+0000","mapping_epoch":181,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":182,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:06:10.468119+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:50.886830+0000","last_clean_scrub_stamp":"2024-09-18T08:06:10.468119+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":78,"reported_epoch":215,"state":"active+clean","last_fresh":"2024-09-18T08:09:51.558490+0000","last_change":"2024-09-18T08:08:18.551527+0000","last_active":"2024-09-18T08:09:51.558490+0000","last_peered":"2024-09-18T08:09:51.558490+0000","last_clean":"2024-09-18T08:09:51.558490+0000","last_became_active":"2024-09-18T08:08:18.543694+0000","last_became_peered":"2024-09-18T08:08:18.543694+0000","last_unstale":"2024-09-18T08:09:51.558490+0000","last_undegraded":"2024-09-18T08:09:51.558490+0000","last_fullsized":"2024-09-18T08:09:51.558490+0000","mapping_epoch":181,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":182,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:04:16.427390+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_clean_scrub_stamp":"2024-09-18T08:04:16.427390+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.d","version":"215'8294","reported_seq":8406,"reported_epoch":215,"state":"active+clean","last_fresh":"2024-09-18T08:09:58.281202+0000","last_change":"2024-09-18T08:07:53.671291+0000","last_active":"2024-09-18T08:09:58.281202+0000","last_peered":"2024-09-18T08:09:58.281202+0000","last_clean":"2024-09-18T08:09:58.281202+0000","last_became_active":"2024-09-18T08:07:53.670773+0000","last_became_peered":"2024-09-18T08:07:53.670773+0000","last_unstale":"2024-09-18T08:09:58.281202+0000","last_undegraded":"2024-09-18T08:09:58.281202+0000","last_fullsized":"2024-09-18T08:09:58.281202+0000","mapping_epoch":165,"log_start":"186'3200","ondisk_log_start":"186'3200","created":165,"last_epoch_clean":166,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:07:52.307205+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:07:52.307205+0000","last_clean_scrub_stamp":"2024-09-18T08:07:52.307205+0000","objects_scrubbed":0,"log_size":5094,"log_dups_size":0,"ondisk_log_size":5094,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":276234264,"num_objects":4216,"num_object_clones":0,"num_object_copies":12648,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4216,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":8294,"num_write_kb":400257,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,2],"acting":[5,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.4","version":"74'79","reported_seq":157,"reported_epoch":216,"state":"active+clean","last_fresh":"2024-09-18T08:09:58.660015+0000","last_change":"2024-09-18T08:08:59.376417+0000","last_active":"2024-09-18T08:09:58.660015+0000","last_peered":"2024-09-18T08:09:58.660015+0000","last_clean":"2024-09-18T08:09:58.660015+0000","last_became_active":"2024-09-18T08:07:26.398930+0000","last_became_peered":"2024-09-18T08:07:26.398930+0000","last_unstale":"2024-09-18T08:09:58.660015+0000","last_undegraded":"2024-09-18T08:09:58.660015+0000","last_fullsized":"2024-09-18T08:09:58.660015+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":152,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":3,"last_scrub":"74'79","last_scrub_stamp":"2024-09-18T08:08:59.376337+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:00:11.893742+0000","last_clean_scrub_stamp":"2024-09-18T08:08:59.376337+0000","objects_scrubbed":1,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:10:26.641607+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":327680,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,7],"acting":[3,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"69'14","reported_seq":94,"reported_epoch":215,"state":"active+clean","last_fresh":"2024-09-18T08:09:51.557860+0000","last_change":"2024-09-18T08:08:18.617734+0000","last_active":"2024-09-18T08:09:51.557860+0000","last_peered":"2024-09-18T08:09:51.557860+0000","last_clean":"2024-09-18T08:09:51.557860+0000","last_became_active":"2024-09-18T08:08:18.549671+0000","last_became_peered":"2024-09-18T08:08:18.549671+0000","last_unstale":"2024-09-18T08:09:51.557860+0000","last_undegraded":"2024-09-18T08:09:51.557860+0000","last_fullsized":"2024-09-18T08:09:51.557860+0000","mapping_epoch":181,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":182,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'14","last_scrub_stamp":"2024-09-18T08:08:08.629246+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_clean_scrub_stamp":"2024-09-18T08:08:08.629246+0000","objects_scrubbed":0,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":14,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"62'2","reported_seq":371,"reported_epoch":216,"state":"active+clean","last_fresh":"2024-09-18T08:09:58.645274+0000","last_change":"2024-09-18T08:08:18.565279+0000","last_active":"2024-09-18T08:09:58.645274+0000","last_peered":"2024-09-18T08:09:58.645274+0000","last_clean":"2024-09-18T08:09:58.645274+0000","last_became_active":"2024-09-18T08:08:18.544872+0000","last_became_peered":"2024-09-18T08:08:18.544872+0000","last_unstale":"2024-09-18T08:09:58.645274+0000","last_undegraded":"2024-09-18T08:09:58.645274+0000","last_fullsized":"2024-09-18T08:09:58.645274+0000","mapping_epoch":181,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":182,"parent":"0.0","parent_split_bits":0,"last_scrub":"62'2","last_scrub_stamp":"2024-09-18T08:06:43.607152+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:46.860092+0000","last_clean_scrub_stamp":"2024-09-18T08:06:43.607152+0000","objects_scrubbed":2,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:10:26.053059+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1039,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":2078,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.0","version":"182'10","reported_seq":448,"reported_epoch":216,"state":"active+clean","last_fresh":"2024-09-18T08:09:58.660056+0000","last_change":"2024-09-18T08:09:51.467488+0000","last_active":"2024-09-18T08:09:58.660056+0000","last_peered":"2024-09-18T08:09:58.660056+0000","last_clean":"2024-09-18T08:09:58.660056+0000","last_became_active":"2024-09-18T08:08:18.541756+0000","last_became_peered":"2024-09-18T08:08:18.541756+0000","last_unstale":"2024-09-18T08:09:58.660056+0000","last_undegraded":"2024-09-18T08:09:58.660056+0000","last_fullsized":"2024-09-18T08:09:58.660056+0000","mapping_epoch":181,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":182,"parent":"0.0","parent_split_bits":0,"last_scrub":"182'10","last_scrub_stamp":"2024-09-18T08:09:51.467349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:50.886830+0000","last_clean_scrub_stamp":"2024-09-18T08:09:51.467349+0000","objects_scrubbed":1,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:10:51.919411+0000","scrub_duration":19,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":347,"reported_epoch":215,"state":"active+clean","last_fresh":"2024-09-18T08:09:51.570089+0000","last_change":"2024-09-18T08:07:49.650645+0000","last_active":"2024-09-18T08:09:51.570089+0000","last_peered":"2024-09-18T08:09:51.570089+0000","last_clean":"2024-09-18T08:09:51.570089+0000","last_became_active":"2024-09-18T08:06:24.140303+0000","last_became_peered":"2024-09-18T08:06:24.140303+0000","last_unstale":"2024-09-18T08:09:51.570089+0000","last_undegraded":"2024-09-18T08:09:51.570089+0000","last_fullsized":"2024-09-18T08:09:51.570089+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:07:49.650577+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_clean_scrub_stamp":"2024-09-18T08:07:49.650577+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:10:13.720079+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.e","version":"216'8402","reported_seq":12750,"reported_epoch":216,"state":"active+clean","last_fresh":"2024-09-18T08:09:58.790938+0000","last_change":"2024-09-18T08:09:17.296111+0000","last_active":"2024-09-18T08:09:58.790938+0000","last_peered":"2024-09-18T08:09:58.790938+0000","last_clean":"2024-09-18T08:09:58.790938+0000","last_became_active":"2024-09-18T08:09:17.286866+0000","last_became_peered":"2024-09-18T08:09:17.286866+0000","last_unstale":"2024-09-18T08:09:58.790938+0000","last_undegraded":"2024-09-18T08:09:58.790938+0000","last_fullsized":"2024-09-18T08:09:58.790938+0000","mapping_epoch":207,"log_start":"185'3000","ondisk_log_start":"185'3000","created":165,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:07:52.307205+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:07:52.307205+0000","last_clean_scrub_stamp":"2024-09-18T08:07:52.307205+0000","objects_scrubbed":0,"log_size":5402,"log_dups_size":0,"ondisk_log_size":5402,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":286130176,"num_objects":4366,"num_object_clones":0,"num_object_copies":13098,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4366,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8402,"num_write_kb":408576,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2107,"num_bytes_recovered":138084352,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,4],"acting":[0,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.7","version":"74'79","reported_seq":152,"reported_epoch":216,"state":"active+clean","last_fresh":"2024-09-18T08:09:58.659508+0000","last_change":"2024-09-18T08:08:57.438699+0000","last_active":"2024-09-18T08:09:58.659508+0000","last_peered":"2024-09-18T08:09:58.659508+0000","last_clean":"2024-09-18T08:09:58.659508+0000","last_became_active":"2024-09-18T08:07:26.398289+0000","last_became_peered":"2024-09-18T08:07:26.398289+0000","last_unstale":"2024-09-18T08:09:58.659508+0000","last_undegraded":"2024-09-18T08:09:58.659508+0000","last_fullsized":"2024-09-18T08:09:58.659508+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":152,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":3,"last_scrub":"74'79","last_scrub_stamp":"2024-09-18T08:08:57.438600+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:00:11.893742+0000","last_clean_scrub_stamp":"2024-09-18T08:08:57.438600+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:10:09.717533+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,7],"acting":[3,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.2","version":"69'10","reported_seq":395,"reported_epoch":216,"state":"active+clean","last_fresh":"2024-09-18T08:09:58.660801+0000","last_change":"2024-09-18T08:08:18.580590+0000","last_active":"2024-09-18T08:09:58.660801+0000","last_peered":"2024-09-18T08:09:58.660801+0000","last_clean":"2024-09-18T08:09:58.660801+0000","last_became_active":"2024-09-18T08:08:18.538711+0000","last_became_peered":"2024-09-18T08:08:18.538711+0000","last_unstale":"2024-09-18T08:09:58.660801+0000","last_undegraded":"2024-09-18T08:09:58.660801+0000","last_fullsized":"2024-09-18T08:09:58.660801+0000","mapping_epoch":181,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":182,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'10","last_scrub_stamp":"2024-09-18T08:07:22.666042+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_clean_scrub_stamp":"2024-09-18T08:07:22.666042+0000","objects_scrubbed":5,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:10:17.688460+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":328,"reported_epoch":215,"state":"active+clean","last_fresh":"2024-09-18T08:09:51.570393+0000","last_change":"2024-09-18T08:03:48.009689+0000","last_active":"2024-09-18T08:09:51.570393+0000","last_peered":"2024-09-18T08:09:51.570393+0000","last_clean":"2024-09-18T08:09:51.570393+0000","last_became_active":"2024-09-18T08:02:47.874727+0000","last_became_peered":"2024-09-18T08:02:47.874727+0000","last_unstale":"2024-09-18T08:09:51.570393+0000","last_undegraded":"2024-09-18T08:09:51.570393+0000","last_fullsized":"2024-09-18T08:09:51.570393+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:03:48.009611+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:46.860092+0000","last_clean_scrub_stamp":"2024-09-18T08:03:48.009611+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.3","version":"138'10","reported_seq":448,"reported_epoch":216,"state":"active+clean","last_fresh":"2024-09-18T08:09:58.642771+0000","last_change":"2024-09-18T08:07:46.695714+0000","last_active":"2024-09-18T08:09:58.642771+0000","last_peered":"2024-09-18T08:09:58.642771+0000","last_clean":"2024-09-18T08:09:58.642771+0000","last_became_active":"2024-09-18T08:06:24.134194+0000","last_became_peered":"2024-09-18T08:06:24.134194+0000","last_unstale":"2024-09-18T08:09:58.642771+0000","last_undegraded":"2024-09-18T08:09:58.642771+0000","last_fullsized":"2024-09-18T08:09:58.642771+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"138'10","last_scrub_stamp":"2024-09-18T08:07:46.695656+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:50.886830+0000","last_clean_scrub_stamp":"2024-09-18T08:07:46.695656+0000","objects_scrubbed":1,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":357,"reported_epoch":216,"state":"active+clean","last_fresh":"2024-09-18T08:09:58.643571+0000","last_change":"2024-09-18T08:09:46.989395+0000","last_active":"2024-09-18T08:09:58.643571+0000","last_peered":"2024-09-18T08:09:58.643571+0000","last_clean":"2024-09-18T08:09:58.643571+0000","last_became_active":"2024-09-18T08:08:52.700080+0000","last_became_peered":"2024-09-18T08:08:52.700080+0000","last_unstale":"2024-09-18T08:09:58.643571+0000","last_undegraded":"2024-09-18T08:09:58.643571+0000","last_fullsized":"2024-09-18T08:09:58.643571+0000","mapping_epoch":198,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":199,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:09:46.989329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_clean_scrub_stamp":"2024-09-18T08:09:46.989329+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:10:55.582764+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.f","version":"216'8573","reported_seq":24742,"reported_epoch":216,"state":"active+clean","last_fresh":"2024-09-18T08:09:58.789796+0000","last_change":"2024-09-18T08:09:28.367000+0000","last_active":"2024-09-18T08:09:58.789796+0000","last_peered":"2024-09-18T08:09:58.789796+0000","last_clean":"2024-09-18T08:09:58.789796+0000","last_became_active":"2024-09-18T08:09:28.363033+0000","last_became_peered":"2024-09-18T08:09:28.363033+0000","last_unstale":"2024-09-18T08:09:58.789796+0000","last_undegraded":"2024-09-18T08:09:58.789796+0000","last_fullsized":"2024-09-18T08:09:58.789796+0000","mapping_epoch":211,"log_start":"193'4200","ondisk_log_start":"193'4200","created":165,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:07:52.307205+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:07:52.307205+0000","last_clean_scrub_stamp":"2024-09-18T08:07:52.307205+0000","objects_scrubbed":0,"log_size":4373,"log_dups_size":0,"ondisk_log_size":4373,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":284098560,"num_objects":4335,"num_object_clones":0,"num_object_copies":13005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4335,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8573,"num_write_kb":413056,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8010,"num_bytes_recovered":524943360,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,1],"acting":[3,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"74'79","reported_seq":232,"reported_epoch":216,"state":"active+clean","last_fresh":"2024-09-18T08:09:58.644538+0000","last_change":"2024-09-18T08:08:57.736557+0000","last_active":"2024-09-18T08:09:58.644538+0000","last_peered":"2024-09-18T08:09:58.644538+0000","last_clean":"2024-09-18T08:09:58.644538+0000","last_became_active":"2024-09-18T08:07:26.426532+0000","last_became_peered":"2024-09-18T08:07:26.426532+0000","last_unstale":"2024-09-18T08:09:58.644538+0000","last_undegraded":"2024-09-18T08:09:58.644538+0000","last_fullsized":"2024-09-18T08:09:58.644538+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":152,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":3,"last_scrub":"74'79","last_scrub_stamp":"2024-09-18T08:08:57.736355+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:00:11.893742+0000","last_clean_scrub_stamp":"2024-09-18T08:08:57.736355+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:10:19.314793+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,3],"acting":[0,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.3","version":"69'19","reported_seq":362,"reported_epoch":216,"state":"active+clean","last_fresh":"2024-09-18T08:09:58.644883+0000","last_change":"2024-09-18T08:04:07.231406+0000","last_active":"2024-09-18T08:09:58.644883+0000","last_peered":"2024-09-18T08:09:58.644883+0000","last_clean":"2024-09-18T08:09:58.644883+0000","last_became_active":"2024-09-18T08:02:49.890318+0000","last_became_peered":"2024-09-18T08:02:49.890318+0000","last_unstale":"2024-09-18T08:09:58.644883+0000","last_undegraded":"2024-09-18T08:09:58.644883+0000","last_fullsized":"2024-09-18T08:09:58.644883+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'19","last_scrub_stamp":"2024-09-18T08:04:07.229846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_clean_scrub_stamp":"2024-09-18T08:04:07.229846+0000","objects_scrubbed":8,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:10:19.093125+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":39,"num_read_kb":25,"num_write":22,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.4","version":"62'1","reported_seq":366,"reported_epoch":216,"state":"active+clean","last_fresh":"2024-09-18T08:09:58.660838+0000","last_change":"2024-09-18T08:09:02.569780+0000","last_active":"2024-09-18T08:09:58.660838+0000","last_peered":"2024-09-18T08:09:58.660838+0000","last_clean":"2024-09-18T08:09:58.660838+0000","last_became_active":"2024-09-18T08:08:52.688347+0000","last_became_peered":"2024-09-18T08:08:52.688347+0000","last_unstale":"2024-09-18T08:09:58.660838+0000","last_undegraded":"2024-09-18T08:09:58.660838+0000","last_fullsized":"2024-09-18T08:09:58.660838+0000","mapping_epoch":198,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":199,"parent":"0.0","parent_split_bits":0,"last_scrub":"62'1","last_scrub_stamp":"2024-09-18T08:09:02.569706+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:46.860092+0000","last_clean_scrub_stamp":"2024-09-18T08:09:02.569706+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:10:05.632401+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":436,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":872,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":256,"reported_epoch":216,"state":"active+clean","last_fresh":"2024-09-18T08:09:58.644299+0000","last_change":"2024-09-18T08:06:24.133954+0000","last_active":"2024-09-18T08:09:58.644299+0000","last_peered":"2024-09-18T08:09:58.644299+0000","last_clean":"2024-09-18T08:09:58.644299+0000","last_became_active":"2024-09-18T08:06:24.133262+0000","last_became_peered":"2024-09-18T08:06:24.133262+0000","last_unstale":"2024-09-18T08:09:58.644299+0000","last_undegraded":"2024-09-18T08:09:58.644299+0000","last_fullsized":"2024-09-18T08:09:58.644299+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:04:13.377561+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:50.886830+0000","last_clean_scrub_stamp":"2024-09-18T08:04:13.377561+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:10:22.094192+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":362,"reported_epoch":216,"state":"active+clean","last_fresh":"2024-09-18T08:09:58.660399+0000","last_change":"2024-09-18T08:08:52.688537+0000","last_active":"2024-09-18T08:09:58.660399+0000","last_peered":"2024-09-18T08:09:58.660399+0000","last_clean":"2024-09-18T08:09:58.660399+0000","last_became_active":"2024-09-18T08:08:52.688212+0000","last_became_peered":"2024-09-18T08:08:52.688212+0000","last_unstale":"2024-09-18T08:09:58.660399+0000","last_undegraded":"2024-09-18T08:09:58.660399+0000","last_fullsized":"2024-09-18T08:09:58.660399+0000","mapping_epoch":198,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":199,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:07:49.041138+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_clean_scrub_stamp":"2024-09-18T08:07:49.041138+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:10:25.679955+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.8","version":"215'8268","reported_seq":8378,"reported_epoch":215,"state":"active+clean","last_fresh":"2024-09-18T08:09:58.291969+0000","last_change":"2024-09-18T08:07:53.671548+0000","last_active":"2024-09-18T08:09:58.291969+0000","last_peered":"2024-09-18T08:09:58.291969+0000","last_clean":"2024-09-18T08:09:58.291969+0000","last_became_active":"2024-09-18T08:07:53.670635+0000","last_became_peered":"2024-09-18T08:07:53.670635+0000","last_unstale":"2024-09-18T08:09:58.291969+0000","last_undegraded":"2024-09-18T08:09:58.291969+0000","last_fullsized":"2024-09-18T08:09:58.291969+0000","mapping_epoch":165,"log_start":"185'3100","ondisk_log_start":"185'3100","created":165,"last_epoch_clean":166,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:07:52.307205+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:07:52.307205+0000","last_clean_scrub_stamp":"2024-09-18T08:07:52.307205+0000","objects_scrubbed":0,"log_size":5168,"log_dups_size":0,"ondisk_log_size":5168,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":274202624,"num_objects":4184,"num_object_clones":0,"num_object_copies":12552,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4184,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8268,"num_write_kb":398464,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.1","version":"74'79","reported_seq":149,"reported_epoch":215,"state":"active+clean","last_fresh":"2024-09-18T08:09:51.568338+0000","last_change":"2024-09-18T08:08:59.553431+0000","last_active":"2024-09-18T08:09:51.568338+0000","last_peered":"2024-09-18T08:09:51.568338+0000","last_clean":"2024-09-18T08:09:51.568338+0000","last_became_active":"2024-09-18T08:07:26.400643+0000","last_became_peered":"2024-09-18T08:07:26.400643+0000","last_unstale":"2024-09-18T08:09:51.568338+0000","last_undegraded":"2024-09-18T08:09:51.568338+0000","last_fullsized":"2024-09-18T08:09:51.568338+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":152,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":4,"last_scrub":"74'79","last_scrub_stamp":"2024-09-18T08:08:59.552592+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:00:11.893742+0000","last_clean_scrub_stamp":"2024-09-18T08:08:59.552592+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:10:15.399172+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,2],"acting":[5,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.4","version":"69'28","reported_seq":469,"reported_epoch":216,"state":"active+clean","last_fresh":"2024-09-18T08:09:58.660674+0000","last_change":"2024-09-18T08:08:57.639985+0000","last_active":"2024-09-18T08:09:58.660674+0000","last_peered":"2024-09-18T08:09:58.660674+0000","last_clean":"2024-09-18T08:09:58.660674+0000","last_became_active":"2024-09-18T08:02:49.897221+0000","last_became_peered":"2024-09-18T08:02:49.897221+0000","last_unstale":"2024-09-18T08:09:58.660674+0000","last_undegraded":"2024-09-18T08:09:58.660674+0000","last_fullsized":"2024-09-18T08:09:58.660674+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'28","last_scrub_stamp":"2024-09-18T08:08:57.639628+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_clean_scrub_stamp":"2024-09-18T08:08:57.639628+0000","objects_scrubbed":10,"log_size":28,"log_dups_size":0,"ondisk_log_size":28,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:10:27.518247+0000","scrub_duration":18,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":48,"num_read_kb":33,"num_write":26,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":78,"reported_epoch":215,"state":"active+clean","last_fresh":"2024-09-18T08:09:51.558428+0000","last_change":"2024-09-18T08:08:18.543579+0000","last_active":"2024-09-18T08:09:51.558428+0000","last_peered":"2024-09-18T08:09:51.558428+0000","last_clean":"2024-09-18T08:09:51.558428+0000","last_became_active":"2024-09-18T08:08:18.538852+0000","last_became_peered":"2024-09-18T08:08:18.538852+0000","last_unstale":"2024-09-18T08:09:51.558428+0000","last_undegraded":"2024-09-18T08:09:51.558428+0000","last_fullsized":"2024-09-18T08:09:51.558428+0000","mapping_epoch":181,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":182,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:05:39.490141+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:46.860092+0000","last_clean_scrub_stamp":"2024-09-18T08:05:39.490141+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"64'2","reported_seq":368,"reported_epoch":215,"state":"active+clean","last_fresh":"2024-09-18T08:09:51.572204+0000","last_change":"2024-09-18T08:06:24.165227+0000","last_active":"2024-09-18T08:09:51.572204+0000","last_peered":"2024-09-18T08:09:51.572204+0000","last_clean":"2024-09-18T08:09:51.572204+0000","last_became_active":"2024-09-18T08:06:24.136080+0000","last_became_peered":"2024-09-18T08:06:24.136080+0000","last_unstale":"2024-09-18T08:09:51.572204+0000","last_undegraded":"2024-09-18T08:09:51.572204+0000","last_fullsized":"2024-09-18T08:09:51.572204+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"64'2","last_scrub_stamp":"2024-09-18T08:05:44.802942+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:45.855979+0000","last_clean_scrub_stamp":"2024-09-18T08:05:44.802942+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.001348349,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":38,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":336,"reported_epoch":216,"state":"active+clean","last_fresh":"2024-09-18T08:09:58.644104+0000","last_change":"2024-09-18T08:08:18.544944+0000","last_active":"2024-09-18T08:09:58.644104+0000","last_peered":"2024-09-18T08:09:58.644104+0000","last_clean":"2024-09-18T08:09:58.644104+0000","last_became_active":"2024-09-18T08:08:18.544260+0000","last_became_peered":"2024-09-18T08:08:18.544260+0000","last_unstale":"2024-09-18T08:09:58.644104+0000","last_undegraded":"2024-09-18T08:09:58.644104+0000","last_fullsized":"2024-09-18T08:09:58.644104+0000","mapping_epoch":181,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":182,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:07:53.582095+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:50.886830+0000","last_clean_scrub_stamp":"2024-09-18T08:07:53.582095+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:10:51.897980+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":342,"reported_epoch":216,"state":"active+clean","last_fresh":"2024-09-18T08:09:58.653265+0000","last_change":"2024-09-18T08:08:18.543527+0000","last_active":"2024-09-18T08:09:58.653265+0000","last_peered":"2024-09-18T08:09:58.653265+0000","last_clean":"2024-09-18T08:09:58.653265+0000","last_became_active":"2024-09-18T08:08:18.542678+0000","last_became_peered":"2024-09-18T08:08:18.542678+0000","last_unstale":"2024-09-18T08:09:58.653265+0000","last_undegraded":"2024-09-18T08:09:58.653265+0000","last_fullsized":"2024-09-18T08:09:58.653265+0000","mapping_epoch":181,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":182,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:07:08.245063+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_clean_scrub_stamp":"2024-09-18T08:07:08.245063+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:10:56.470037+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.b","version":"216'8423","reported_seq":14047,"reported_epoch":216,"state":"active+clean","last_fresh":"2024-09-18T08:09:58.791073+0000","last_change":"2024-09-18T08:08:48.662821+0000","last_active":"2024-09-18T08:09:58.791073+0000","last_peered":"2024-09-18T08:09:58.791073+0000","last_clean":"2024-09-18T08:09:58.791073+0000","last_became_active":"2024-09-18T08:08:48.655676+0000","last_became_peered":"2024-09-18T08:08:48.655676+0000","last_unstale":"2024-09-18T08:09:58.791073+0000","last_undegraded":"2024-09-18T08:09:58.791073+0000","last_fullsized":"2024-09-18T08:09:58.791073+0000","mapping_epoch":194,"log_start":"191'4000","ondisk_log_start":"191'4000","created":165,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:07:52.307205+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:07:52.307205+0000","last_clean_scrub_stamp":"2024-09-18T08:07:52.307205+0000","objects_scrubbed":0,"log_size":4423,"log_dups_size":0,"ondisk_log_size":4423,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":278462464,"num_objects":4249,"num_object_clones":0,"num_object_copies":12747,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4249,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8423,"num_write_kb":405504,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2745,"num_bytes_recovered":179896320,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,1],"acting":[3,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"74'79","reported_seq":253,"reported_epoch":216,"state":"active+clean","last_fresh":"2024-09-18T08:09:58.642703+0000","last_change":"2024-09-18T08:08:18.524000+0000","last_active":"2024-09-18T08:09:58.642703+0000","last_peered":"2024-09-18T08:09:58.642703+0000","last_clean":"2024-09-18T08:09:58.642703+0000","last_became_active":"2024-09-18T08:08:18.522977+0000","last_became_peered":"2024-09-18T08:08:18.522977+0000","last_unstale":"2024-09-18T08:09:58.642703+0000","last_undegraded":"2024-09-18T08:09:58.642703+0000","last_fullsized":"2024-09-18T08:09:58.642703+0000","mapping_epoch":181,"log_start":"0'0","ondisk_log_start":"0'0","created":152,"last_epoch_clean":182,"parent":"0.0","parent_split_bits":4,"last_scrub":"74'79","last_scrub_stamp":"2024-09-18T08:08:09.574144+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:00:11.893742+0000","last_clean_scrub_stamp":"2024-09-18T08:08:09.574144+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,4],"acting":[0,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.7","version":"69'13","reported_seq":270,"reported_epoch":215,"state":"active+clean","last_fresh":"2024-09-18T08:09:51.578742+0000","last_change":"2024-09-18T08:04:18.164662+0000","last_active":"2024-09-18T08:09:51.578742+0000","last_peered":"2024-09-18T08:09:51.578742+0000","last_clean":"2024-09-18T08:09:51.578742+0000","last_became_active":"2024-09-18T08:04:18.144925+0000","last_became_peered":"2024-09-18T08:04:18.144925+0000","last_unstale":"2024-09-18T08:09:51.578742+0000","last_undegraded":"2024-09-18T08:09:51.578742+0000","last_fullsized":"2024-09-18T08:09:51.578742+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'13","last_scrub_stamp":"2024-09-18T08:03:50.700818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_clean_scrub_stamp":"2024-09-18T08:03:50.700818+0000","objects_scrubbed":0,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":29,"num_read_kb":18,"num_write":16,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,0],"acting":[2,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":352,"reported_epoch":216,"state":"active+clean","last_fresh":"2024-09-18T08:09:58.667282+0000","last_change":"2024-09-18T08:08:53.649127+0000","last_active":"2024-09-18T08:09:58.667282+0000","last_peered":"2024-09-18T08:09:58.667282+0000","last_clean":"2024-09-18T08:09:58.667282+0000","last_became_active":"2024-09-18T08:06:24.126334+0000","last_became_peered":"2024-09-18T08:06:24.126334+0000","last_unstale":"2024-09-18T08:09:58.667282+0000","last_undegraded":"2024-09-18T08:09:58.667282+0000","last_fullsized":"2024-09-18T08:09:58.667282+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:08:53.649079+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:46.860092+0000","last_clean_scrub_stamp":"2024-09-18T08:08:53.649079+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:10:14.347881+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"62'1","reported_seq":338,"reported_epoch":215,"state":"active+clean","last_fresh":"2024-09-18T08:09:51.574919+0000","last_change":"2024-09-18T08:04:09.732724+0000","last_active":"2024-09-18T08:09:51.574919+0000","last_peered":"2024-09-18T08:09:51.574919+0000","last_clean":"2024-09-18T08:09:51.574919+0000","last_became_active":"2024-09-18T08:02:46.868175+0000","last_became_peered":"2024-09-18T08:02:46.868175+0000","last_unstale":"2024-09-18T08:09:51.574919+0000","last_undegraded":"2024-09-18T08:09:51.574919+0000","last_fullsized":"2024-09-18T08:09:51.574919+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"62'1","last_scrub_stamp":"2024-09-18T08:04:09.732645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:45.855979+0000","last_clean_scrub_stamp":"2024-09-18T08:04:09.732645+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0.00066785499999999997,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":44,"reported_epoch":215,"state":"active+clean","last_fresh":"2024-09-18T08:09:51.578343+0000","last_change":"2024-09-18T08:08:52.690656+0000","last_active":"2024-09-18T08:09:51.578343+0000","last_peered":"2024-09-18T08:09:51.578343+0000","last_clean":"2024-09-18T08:09:51.578343+0000","last_became_active":"2024-09-18T08:08:52.690286+0000","last_became_peered":"2024-09-18T08:08:52.690286+0000","last_unstale":"2024-09-18T08:09:51.578343+0000","last_undegraded":"2024-09-18T08:09:51.578343+0000","last_fullsized":"2024-09-18T08:09:51.578343+0000","mapping_epoch":198,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":199,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:07:08.806748+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:50.886830+0000","last_clean_scrub_stamp":"2024-09-18T08:07:08.806748+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":347,"reported_epoch":215,"state":"active+clean","last_fresh":"2024-09-18T08:09:51.571089+0000","last_change":"2024-09-18T08:09:42.647525+0000","last_active":"2024-09-18T08:09:51.571089+0000","last_peered":"2024-09-18T08:09:51.571089+0000","last_clean":"2024-09-18T08:09:51.571089+0000","last_became_active":"2024-09-18T08:06:24.145382+0000","last_became_peered":"2024-09-18T08:06:24.145382+0000","last_unstale":"2024-09-18T08:09:51.571089+0000","last_undegraded":"2024-09-18T08:09:51.571089+0000","last_fullsized":"2024-09-18T08:09:51.571089+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:09:42.647317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_clean_scrub_stamp":"2024-09-18T08:09:42.647317+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:11:03.015274+0000","scrub_duration":26,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.a","version":"216'8404","reported_seq":8517,"reported_epoch":216,"state":"active+clean","last_fresh":"2024-09-18T08:09:58.676188+0000","last_change":"2024-09-18T08:07:53.326651+0000","last_active":"2024-09-18T08:09:58.676188+0000","last_peered":"2024-09-18T08:09:58.676188+0000","last_clean":"2024-09-18T08:09:58.676188+0000","last_became_active":"2024-09-18T08:07:53.326361+0000","last_became_peered":"2024-09-18T08:07:53.326361+0000","last_unstale":"2024-09-18T08:09:58.676188+0000","last_undegraded":"2024-09-18T08:09:58.676188+0000","last_fullsized":"2024-09-18T08:09:58.676188+0000","mapping_epoch":165,"log_start":"190'3600","ondisk_log_start":"190'3600","created":165,"last_epoch_clean":166,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:07:52.307205+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:07:52.307205+0000","last_clean_scrub_stamp":"2024-09-18T08:07:52.307205+0000","objects_scrubbed":0,"log_size":4804,"log_dups_size":0,"ondisk_log_size":4804,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:10:20.719923+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":273940480,"num_objects":4180,"num_object_clones":0,"num_object_copies":12540,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4180,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8404,"num_write_kb":402688,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.3","version":"74'79","reported_seq":173,"reported_epoch":216,"state":"active+clean","last_fresh":"2024-09-18T08:09:58.661162+0000","last_change":"2024-09-18T08:09:41.651800+0000","last_active":"2024-09-18T08:09:58.661162+0000","last_peered":"2024-09-18T08:09:58.661162+0000","last_clean":"2024-09-18T08:09:58.661162+0000","last_became_active":"2024-09-18T08:08:18.538929+0000","last_became_peered":"2024-09-18T08:08:18.538929+0000","last_unstale":"2024-09-18T08:09:58.661162+0000","last_undegraded":"2024-09-18T08:09:58.661162+0000","last_fullsized":"2024-09-18T08:09:58.661162+0000","mapping_epoch":181,"log_start":"0'0","ondisk_log_start":"0'0","created":152,"last_epoch_clean":182,"parent":"0.0","parent_split_bits":3,"last_scrub":"74'79","last_scrub_stamp":"2024-09-18T08:09:41.651729+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:00:11.893742+0000","last_clean_scrub_stamp":"2024-09-18T08:09:41.651729+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:10:55.722378+0000","scrub_duration":17,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.6","version":"69'12","reported_seq":394,"reported_epoch":216,"state":"active+clean","last_fresh":"2024-09-18T08:09:58.645074+0000","last_change":"2024-09-18T08:08:57.021693+0000","last_active":"2024-09-18T08:09:58.645074+0000","last_peered":"2024-09-18T08:09:58.645074+0000","last_clean":"2024-09-18T08:09:58.645074+0000","last_became_active":"2024-09-18T08:08:52.700130+0000","last_became_peered":"2024-09-18T08:08:52.700130+0000","last_unstale":"2024-09-18T08:09:58.645074+0000","last_undegraded":"2024-09-18T08:09:58.645074+0000","last_fullsized":"2024-09-18T08:09:58.645074+0000","mapping_epoch":198,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":200,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'12","last_scrub_stamp":"2024-09-18T08:07:39.603562+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_clean_scrub_stamp":"2024-09-18T08:07:39.603562+0000","objects_scrubbed":6,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:10:57.784539+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":18,"num_read_kb":12,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":360,"reported_epoch":216,"state":"active+clean","last_fresh":"2024-09-18T08:09:58.646030+0000","last_change":"2024-09-18T08:09:34.089548+0000","last_active":"2024-09-18T08:09:58.646030+0000","last_peered":"2024-09-18T08:09:58.646030+0000","last_clean":"2024-09-18T08:09:58.646030+0000","last_became_active":"2024-09-18T08:08:18.523135+0000","last_became_peered":"2024-09-18T08:08:18.523135+0000","last_unstale":"2024-09-18T08:09:58.646030+0000","last_undegraded":"2024-09-18T08:09:58.646030+0000","last_fullsized":"2024-09-18T08:09:58.646030+0000","mapping_epoch":181,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":182,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:09:34.089476+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:46.860092+0000","last_clean_scrub_stamp":"2024-09-18T08:09:34.089476+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:10:41.071862+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"70'5","reported_seq":408,"reported_epoch":216,"state":"active+clean","last_fresh":"2024-09-18T08:09:58.661253+0000","last_change":"2024-09-18T08:09:09.649261+0000","last_active":"2024-09-18T08:09:58.661253+0000","last_peered":"2024-09-18T08:09:58.661253+0000","last_clean":"2024-09-18T08:09:58.661253+0000","last_became_active":"2024-09-18T08:06:46.141667+0000","last_became_peered":"2024-09-18T08:06:46.141667+0000","last_unstale":"2024-09-18T08:09:58.661253+0000","last_undegraded":"2024-09-18T08:09:58.661253+0000","last_fullsized":"2024-09-18T08:09:58.661253+0000","mapping_epoch":143,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":144,"parent":"0.0","parent_split_bits":0,"last_scrub":"70'5","last_scrub_stamp":"2024-09-18T08:09:09.649188+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:45.855979+0000","last_clean_scrub_stamp":"2024-09-18T08:09:09.649188+0000","objects_scrubbed":1,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:10:24.340859+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0.000504055,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":198,"num_read_kb":193,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":336,"reported_epoch":215,"state":"active+clean","last_fresh":"2024-09-18T08:09:51.569839+0000","last_change":"2024-09-18T08:07:20.767916+0000","last_active":"2024-09-18T08:09:51.569839+0000","last_peered":"2024-09-18T08:09:51.569839+0000","last_clean":"2024-09-18T08:09:51.569839+0000","last_became_active":"2024-09-18T08:02:51.913112+0000","last_became_peered":"2024-09-18T08:02:51.913112+0000","last_unstale":"2024-09-18T08:09:51.569839+0000","last_undegraded":"2024-09-18T08:09:51.569839+0000","last_fullsized":"2024-09-18T08:09:51.569839+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:07:20.767841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:50.886830+0000","last_clean_scrub_stamp":"2024-09-18T08:07:20.767841+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":319,"reported_epoch":216,"state":"active+clean","last_fresh":"2024-09-18T08:09:58.660462+0000","last_change":"2024-09-18T08:09:49.702132+0000","last_active":"2024-09-18T08:09:58.660462+0000","last_peered":"2024-09-18T08:09:58.660462+0000","last_clean":"2024-09-18T08:09:58.660462+0000","last_became_active":"2024-09-18T08:02:53.907404+0000","last_became_peered":"2024-09-18T08:02:53.907404+0000","last_unstale":"2024-09-18T08:09:58.660462+0000","last_undegraded":"2024-09-18T08:09:58.660462+0000","last_fullsized":"2024-09-18T08:09:58.660462+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:09:49.702055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_clean_scrub_stamp":"2024-09-18T08:09:49.702055+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:11:01.853786+0000","scrub_duration":23,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.9","version":"215'8066","reported_seq":8177,"reported_epoch":215,"state":"active+clean","last_fresh":"2024-09-18T08:09:58.206084+0000","last_change":"2024-09-18T08:07:53.672294+0000","last_active":"2024-09-18T08:09:58.206084+0000","last_peered":"2024-09-18T08:09:58.206084+0000","last_clean":"2024-09-18T08:09:58.206084+0000","last_became_active":"2024-09-18T08:07:53.671994+0000","last_became_peered":"2024-09-18T08:07:53.671994+0000","last_unstale":"2024-09-18T08:09:58.206084+0000","last_undegraded":"2024-09-18T08:09:58.206084+0000","last_fullsized":"2024-09-18T08:09:58.206084+0000","mapping_epoch":165,"log_start":"185'2900","ondisk_log_start":"185'2900","created":165,"last_epoch_clean":166,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:07:52.307205+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:07:52.307205+0000","last_clean_scrub_stamp":"2024-09-18T08:07:52.307205+0000","objects_scrubbed":0,"log_size":5166,"log_dups_size":0,"ondisk_log_size":5166,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":271843328,"num_objects":4148,"num_object_clones":0,"num_object_copies":12444,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4148,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8066,"num_write_kb":390848,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.5","version":"182'21","reported_seq":117,"reported_epoch":215,"state":"active+clean","last_fresh":"2024-09-18T08:09:53.012329+0000","last_change":"2024-09-18T08:08:18.609652+0000","last_active":"2024-09-18T08:09:53.012329+0000","last_peered":"2024-09-18T08:09:53.012329+0000","last_clean":"2024-09-18T08:09:53.012329+0000","last_became_active":"2024-09-18T08:08:18.547555+0000","last_became_peered":"2024-09-18T08:08:18.547555+0000","last_unstale":"2024-09-18T08:09:53.012329+0000","last_undegraded":"2024-09-18T08:09:53.012329+0000","last_fullsized":"2024-09-18T08:09:53.012329+0000","mapping_epoch":181,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":182,"parent":"0.0","parent_split_bits":0,"last_scrub":"130'18","last_scrub_stamp":"2024-09-18T08:06:15.455503+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_clean_scrub_stamp":"2024-09-18T08:06:15.455503+0000","objects_scrubbed":0,"log_size":21,"log_dups_size":0,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":24,"num_read_kb":14,"num_write":13,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":154,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,7],"acting":[4,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":353,"reported_epoch":216,"state":"active+clean","last_fresh":"2024-09-18T08:09:58.659496+0000","last_change":"2024-09-18T08:09:45.572278+0000","last_active":"2024-09-18T08:09:58.659496+0000","last_peered":"2024-09-18T08:09:58.659496+0000","last_clean":"2024-09-18T08:09:58.659496+0000","last_became_active":"2024-09-18T08:06:24.133425+0000","last_became_peered":"2024-09-18T08:06:24.133425+0000","last_unstale":"2024-09-18T08:09:58.659496+0000","last_undegraded":"2024-09-18T08:09:58.659496+0000","last_fullsized":"2024-09-18T08:09:58.659496+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:09:45.572201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:46.860092+0000","last_clean_scrub_stamp":"2024-09-18T08:09:45.572201+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:11:00.131734+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"74'79","reported_seq":432,"reported_epoch":215,"state":"active+clean","last_fresh":"2024-09-18T08:09:51.570268+0000","last_change":"2024-09-18T08:08:57.769104+0000","last_active":"2024-09-18T08:09:51.570268+0000","last_peered":"2024-09-18T08:09:51.570268+0000","last_clean":"2024-09-18T08:09:51.570268+0000","last_became_active":"2024-09-18T08:07:24.376931+0000","last_became_peered":"2024-09-18T08:07:24.376931+0000","last_unstale":"2024-09-18T08:09:51.570268+0000","last_undegraded":"2024-09-18T08:09:51.570268+0000","last_fullsized":"2024-09-18T08:09:51.570268+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":153,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'79","last_scrub_stamp":"2024-09-18T08:08:57.769048+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:00:11.893742+0000","last_clean_scrub_stamp":"2024-09-18T08:08:57.769048+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:10:19.097626+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":164,"reported_epoch":215,"state":"active+clean","last_fresh":"2024-09-18T08:09:51.580257+0000","last_change":"2024-09-18T08:07:09.744206+0000","last_active":"2024-09-18T08:09:51.580257+0000","last_peered":"2024-09-18T08:09:51.580257+0000","last_clean":"2024-09-18T08:09:51.580257+0000","last_became_active":"2024-09-18T08:06:46.141046+0000","last_became_peered":"2024-09-18T08:06:46.141046+0000","last_unstale":"2024-09-18T08:09:51.580257+0000","last_undegraded":"2024-09-18T08:09:51.580257+0000","last_fullsized":"2024-09-18T08:09:51.580257+0000","mapping_epoch":143,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":144,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:07:09.744137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:50.886830+0000","last_clean_scrub_stamp":"2024-09-18T08:07:09.744137+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":339,"reported_epoch":215,"state":"active+clean","last_fresh":"2024-09-18T08:09:51.569519+0000","last_change":"2024-09-18T08:08:18.548620+0000","last_active":"2024-09-18T08:09:51.569519+0000","last_peered":"2024-09-18T08:09:51.569519+0000","last_clean":"2024-09-18T08:09:51.569519+0000","last_became_active":"2024-09-18T08:08:18.548233+0000","last_became_peered":"2024-09-18T08:08:18.548233+0000","last_unstale":"2024-09-18T08:09:51.569519+0000","last_undegraded":"2024-09-18T08:09:51.569519+0000","last_fullsized":"2024-09-18T08:09:51.569519+0000","mapping_epoch":181,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":182,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:06:50.710565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_clean_scrub_stamp":"2024-09-18T08:06:50.710565+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.2","version":"215'8281","reported_seq":12669,"reported_epoch":215,"state":"active+clean","last_fresh":"2024-09-18T08:09:58.162394+0000","last_change":"2024-09-18T08:08:58.743574+0000","last_active":"2024-09-18T08:09:58.162394+0000","last_peered":"2024-09-18T08:09:58.162394+0000","last_clean":"2024-09-18T08:09:58.162394+0000","last_became_active":"2024-09-18T08:08:58.738458+0000","last_became_peered":"2024-09-18T08:08:58.738458+0000","last_unstale":"2024-09-18T08:09:58.162394+0000","last_undegraded":"2024-09-18T08:09:58.162394+0000","last_fullsized":"2024-09-18T08:09:58.162394+0000","mapping_epoch":201,"log_start":"191'3800","ondisk_log_start":"191'3800","created":165,"last_epoch_clean":202,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:07:52.307205+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:07:52.307205+0000","last_clean_scrub_stamp":"2024-09-18T08:07:52.307205+0000","objects_scrubbed":0,"log_size":4481,"log_dups_size":0,"ondisk_log_size":4481,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":281214976,"num_objects":4291,"num_object_clones":0,"num_object_copies":12873,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4291,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8281,"num_write_kb":402304,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2129,"num_bytes_recovered":139526144,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,5],"acting":[4,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.e","version":"69'11","reported_seq":90,"reported_epoch":215,"state":"active+clean","last_fresh":"2024-09-18T08:09:51.558662+0000","last_change":"2024-09-18T08:08:18.594670+0000","last_active":"2024-09-18T08:09:51.558662+0000","last_peered":"2024-09-18T08:09:51.558662+0000","last_clean":"2024-09-18T08:09:51.558662+0000","last_became_active":"2024-09-18T08:08:18.544581+0000","last_became_peered":"2024-09-18T08:08:18.544581+0000","last_unstale":"2024-09-18T08:09:51.558662+0000","last_undegraded":"2024-09-18T08:09:51.558662+0000","last_fullsized":"2024-09-18T08:09:51.558662+0000","mapping_epoch":181,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":182,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'11","last_scrub_stamp":"2024-09-18T08:06:14.472168+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_clean_scrub_stamp":"2024-09-18T08:06:14.472168+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":19,"num_read_kb":12,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":78,"reported_epoch":215,"state":"active+clean","last_fresh":"2024-09-18T08:09:51.558081+0000","last_change":"2024-09-18T08:08:18.524516+0000","last_active":"2024-09-18T08:09:51.558081+0000","last_peered":"2024-09-18T08:09:51.558081+0000","last_clean":"2024-09-18T08:09:51.558081+0000","last_became_active":"2024-09-18T08:08:18.524229+0000","last_became_peered":"2024-09-18T08:08:18.524229+0000","last_unstale":"2024-09-18T08:09:51.558081+0000","last_undegraded":"2024-09-18T08:09:51.558081+0000","last_fullsized":"2024-09-18T08:09:51.558081+0000","mapping_epoch":181,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":182,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:02:46.860092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:46.860092+0000","last_clean_scrub_stamp":"2024-09-18T08:02:46.860092+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":356,"reported_epoch":215,"state":"active+clean","last_fresh":"2024-09-18T08:09:51.569373+0000","last_change":"2024-09-18T08:08:18.547880+0000","last_active":"2024-09-18T08:09:51.569373+0000","last_peered":"2024-09-18T08:09:51.569373+0000","last_clean":"2024-09-18T08:09:51.569373+0000","last_became_active":"2024-09-18T08:08:18.547105+0000","last_became_peered":"2024-09-18T08:08:18.547105+0000","last_unstale":"2024-09-18T08:09:51.569373+0000","last_undegraded":"2024-09-18T08:09:51.569373+0000","last_fullsized":"2024-09-18T08:09:51.569373+0000","mapping_epoch":181,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":182,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:07:51.742090+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:50.886830+0000","last_clean_scrub_stamp":"2024-09-18T08:07:51.742090+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":341,"reported_epoch":216,"state":"active+clean","last_fresh":"2024-09-18T08:09:58.660111+0000","last_change":"2024-09-18T08:07:09.197912+0000","last_active":"2024-09-18T08:09:58.660111+0000","last_peered":"2024-09-18T08:09:58.660111+0000","last_clean":"2024-09-18T08:09:58.660111+0000","last_became_active":"2024-09-18T08:06:24.130266+0000","last_became_peered":"2024-09-18T08:06:24.130266+0000","last_unstale":"2024-09-18T08:09:58.660111+0000","last_undegraded":"2024-09-18T08:09:58.660111+0000","last_fullsized":"2024-09-18T08:09:58.660111+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:07:09.197841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_clean_scrub_stamp":"2024-09-18T08:07:09.197841+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:10:58.431705+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.1","version":"215'8417","reported_seq":8597,"reported_epoch":216,"state":"active+clean","last_fresh":"2024-09-18T08:09:58.660206+0000","last_change":"2024-09-18T08:09:40.130643+0000","last_active":"2024-09-18T08:09:58.660206+0000","last_peered":"2024-09-18T08:09:58.660206+0000","last_clean":"2024-09-18T08:09:58.660206+0000","last_became_active":"2024-09-18T08:07:53.327987+0000","last_became_peered":"2024-09-18T08:07:53.327987+0000","last_unstale":"2024-09-18T08:09:58.660206+0000","last_undegraded":"2024-09-18T08:09:58.660206+0000","last_fullsized":"2024-09-18T08:09:58.660206+0000","mapping_epoch":165,"log_start":"190'3600","ondisk_log_start":"190'3600","created":165,"last_epoch_clean":166,"parent":"0.0","parent_split_bits":0,"last_scrub":"213'7038","last_scrub_stamp":"2024-09-18T08:09:40.130353+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:07:52.307205+0000","last_clean_scrub_stamp":"2024-09-18T08:09:40.130353+0000","objects_scrubbed":5709,"log_size":4817,"log_dups_size":0,"ondisk_log_size":4817,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:11:06.670120+0000","scrub_duration":2501,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":279248896,"num_objects":4261,"num_object_clones":0,"num_object_copies":12783,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4261,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8417,"num_write_kb":405696,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.8","version":"74'79","reported_seq":156,"reported_epoch":215,"state":"active+clean","last_fresh":"2024-09-18T08:09:51.568791+0000","last_change":"2024-09-18T08:09:00.533593+0000","last_active":"2024-09-18T08:09:51.568791+0000","last_peered":"2024-09-18T08:09:51.568791+0000","last_clean":"2024-09-18T08:09:51.568791+0000","last_became_active":"2024-09-18T08:07:26.407298+0000","last_became_peered":"2024-09-18T08:07:26.407298+0000","last_unstale":"2024-09-18T08:09:51.568791+0000","last_undegraded":"2024-09-18T08:09:51.568791+0000","last_fullsized":"2024-09-18T08:09:51.568791+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":152,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":4,"last_scrub":"74'79","last_scrub_stamp":"2024-09-18T08:09:00.532813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:00:11.893742+0000","last_clean_scrub_stamp":"2024-09-18T08:09:00.532813+0000","objects_scrubbed":1,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:10:17.887770+0000","scrub_duration":25,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262688,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,3],"acting":[5,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.d","version":"69'17","reported_seq":96,"reported_epoch":215,"state":"active+clean","last_fresh":"2024-09-18T08:09:51.558665+0000","last_change":"2024-09-18T08:08:18.588624+0000","last_active":"2024-09-18T08:09:51.558665+0000","last_peered":"2024-09-18T08:09:51.558665+0000","last_clean":"2024-09-18T08:09:51.558665+0000","last_became_active":"2024-09-18T08:08:18.540862+0000","last_became_peered":"2024-09-18T08:08:18.540862+0000","last_unstale":"2024-09-18T08:09:51.558665+0000","last_undegraded":"2024-09-18T08:09:51.558665+0000","last_fullsized":"2024-09-18T08:09:51.558665+0000","mapping_epoch":181,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":182,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'17","last_scrub_stamp":"2024-09-18T08:08:09.671248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_clean_scrub_stamp":"2024-09-18T08:08:09.671248+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":28,"num_read_kb":18,"num_write":18,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,1],"acting":[4,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":78,"reported_epoch":215,"state":"active+clean","last_fresh":"2024-09-18T08:09:51.557999+0000","last_change":"2024-09-18T08:08:18.541029+0000","last_active":"2024-09-18T08:09:51.557999+0000","last_peered":"2024-09-18T08:09:51.557999+0000","last_clean":"2024-09-18T08:09:51.557999+0000","last_became_active":"2024-09-18T08:08:18.540285+0000","last_became_peered":"2024-09-18T08:08:18.540285+0000","last_unstale":"2024-09-18T08:09:51.557999+0000","last_undegraded":"2024-09-18T08:09:51.557999+0000","last_fullsized":"2024-09-18T08:09:51.557999+0000","mapping_epoch":181,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":182,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:05:12.622901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:46.860092+0000","last_clean_scrub_stamp":"2024-09-18T08:05:12.622901+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":352,"reported_epoch":216,"state":"active+clean","last_fresh":"2024-09-18T08:09:58.660570+0000","last_change":"2024-09-18T08:09:06.610955+0000","last_active":"2024-09-18T08:09:58.660570+0000","last_peered":"2024-09-18T08:09:58.660570+0000","last_clean":"2024-09-18T08:09:58.660570+0000","last_became_active":"2024-09-18T08:08:18.623014+0000","last_became_peered":"2024-09-18T08:08:18.623014+0000","last_unstale":"2024-09-18T08:09:58.660570+0000","last_undegraded":"2024-09-18T08:09:58.660570+0000","last_fullsized":"2024-09-18T08:09:58.660570+0000","mapping_epoch":181,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":182,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:09:06.610879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:50.886830+0000","last_clean_scrub_stamp":"2024-09-18T08:09:06.610879+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:10:19.831235+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":44,"reported_epoch":215,"state":"active+clean","last_fresh":"2024-09-18T08:09:51.578654+0000","last_change":"2024-09-18T08:08:52.696713+0000","last_active":"2024-09-18T08:09:51.578654+0000","last_peered":"2024-09-18T08:09:51.578654+0000","last_clean":"2024-09-18T08:09:51.578654+0000","last_became_active":"2024-09-18T08:08:52.696300+0000","last_became_peered":"2024-09-18T08:08:52.696300+0000","last_unstale":"2024-09-18T08:09:51.578654+0000","last_undegraded":"2024-09-18T08:09:51.578654+0000","last_fullsized":"2024-09-18T08:09:51.578654+0000","mapping_epoch":198,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":199,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:04:06.709631+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_clean_scrub_stamp":"2024-09-18T08:04:06.709631+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"8.0","version":"216'8308","reported_seq":8422,"reported_epoch":216,"state":"active+clean","last_fresh":"2024-09-18T08:09:58.778104+0000","last_change":"2024-09-18T08:07:53.329182+0000","last_active":"2024-09-18T08:09:58.778104+0000","last_peered":"2024-09-18T08:09:58.778104+0000","last_clean":"2024-09-18T08:09:58.778104+0000","last_became_active":"2024-09-18T08:07:53.328914+0000","last_became_peered":"2024-09-18T08:07:53.328914+0000","last_unstale":"2024-09-18T08:09:58.778104+0000","last_undegraded":"2024-09-18T08:09:58.778104+0000","last_fullsized":"2024-09-18T08:09:58.778104+0000","mapping_epoch":165,"log_start":"191'3900","ondisk_log_start":"191'3900","created":165,"last_epoch_clean":166,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:07:52.307205+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:07:52.307205+0000","last_clean_scrub_stamp":"2024-09-18T08:07:52.307205+0000","objects_scrubbed":0,"log_size":4408,"log_dups_size":0,"ondisk_log_size":4408,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":278003712,"num_objects":4242,"num_object_clones":0,"num_object_copies":12726,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4242,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8308,"num_write_kb":401600,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.9","version":"74'79","reported_seq":454,"reported_epoch":215,"state":"active+clean","last_fresh":"2024-09-18T08:09:51.569476+0000","last_change":"2024-09-18T08:09:27.676247+0000","last_active":"2024-09-18T08:09:51.569476+0000","last_peered":"2024-09-18T08:09:51.569476+0000","last_clean":"2024-09-18T08:09:51.569476+0000","last_became_active":"2024-09-18T08:08:52.691492+0000","last_became_peered":"2024-09-18T08:08:52.691492+0000","last_unstale":"2024-09-18T08:09:51.569476+0000","last_undegraded":"2024-09-18T08:09:51.569476+0000","last_fullsized":"2024-09-18T08:09:51.569476+0000","mapping_epoch":198,"log_start":"0'0","ondisk_log_start":"0'0","created":152,"last_epoch_clean":199,"parent":"0.0","parent_split_bits":4,"last_scrub":"74'79","last_scrub_stamp":"2024-09-18T08:09:27.676169+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:00:11.893742+0000","last_clean_scrub_stamp":"2024-09-18T08:09:27.676169+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:10:36.422152+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,2],"acting":[7,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.c","version":"69'10","reported_seq":90,"reported_epoch":215,"state":"active+clean","last_fresh":"2024-09-18T08:09:51.558312+0000","last_change":"2024-09-18T08:08:18.642452+0000","last_active":"2024-09-18T08:09:51.558312+0000","last_peered":"2024-09-18T08:09:51.558312+0000","last_clean":"2024-09-18T08:09:51.558312+0000","last_became_active":"2024-09-18T08:08:18.622980+0000","last_became_peered":"2024-09-18T08:08:18.622980+0000","last_unstale":"2024-09-18T08:09:51.558312+0000","last_undegraded":"2024-09-18T08:09:51.558312+0000","last_fullsized":"2024-09-18T08:09:51.558312+0000","mapping_epoch":181,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":182,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'10","last_scrub_stamp":"2024-09-18T08:05:41.474813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_clean_scrub_stamp":"2024-09-18T08:05:41.474813+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":14,"num_read_kb":9,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":353,"reported_epoch":216,"state":"active+clean","last_fresh":"2024-09-18T08:09:58.659895+0000","last_change":"2024-09-18T08:08:18.542800+0000","last_active":"2024-09-18T08:09:58.659895+0000","last_peered":"2024-09-18T08:09:58.659895+0000","last_clean":"2024-09-18T08:09:58.659895+0000","last_became_active":"2024-09-18T08:08:18.542424+0000","last_became_peered":"2024-09-18T08:08:18.542424+0000","last_unstale":"2024-09-18T08:09:58.659895+0000","last_undegraded":"2024-09-18T08:09:58.659895+0000","last_fullsized":"2024-09-18T08:09:58.659895+0000","mapping_epoch":181,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":182,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:06:46.339428+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:46.860092+0000","last_clean_scrub_stamp":"2024-09-18T08:06:46.339428+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:10:53.446520+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.d","version":"69'8","reported_seq":424,"reported_epoch":215,"state":"active+clean","last_fresh":"2024-09-18T08:09:53.018291+0000","last_change":"2024-09-18T08:05:33.027988+0000","last_active":"2024-09-18T08:09:53.018291+0000","last_peered":"2024-09-18T08:09:53.018291+0000","last_clean":"2024-09-18T08:09:53.018291+0000","last_became_active":"2024-09-18T08:02:51.909011+0000","last_became_peered":"2024-09-18T08:02:51.909011+0000","last_unstale":"2024-09-18T08:09:53.018291+0000","last_undegraded":"2024-09-18T08:09:53.018291+0000","last_fullsized":"2024-09-18T08:09:53.018291+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'8","last_scrub_stamp":"2024-09-18T08:05:33.027783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:50.886830+0000","last_clean_scrub_stamp":"2024-09-18T08:05:33.027783+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":85,"reported_epoch":215,"state":"active+clean","last_fresh":"2024-09-18T08:09:51.558793+0000","last_change":"2024-09-18T08:08:52.685954+0000","last_active":"2024-09-18T08:09:51.558793+0000","last_peered":"2024-09-18T08:09:51.558793+0000","last_clean":"2024-09-18T08:09:51.558793+0000","last_became_active":"2024-09-18T08:08:52.685603+0000","last_became_peered":"2024-09-18T08:08:52.685603+0000","last_unstale":"2024-09-18T08:09:51.558793+0000","last_undegraded":"2024-09-18T08:09:51.558793+0000","last_fullsized":"2024-09-18T08:09:51.558793+0000","mapping_epoch":198,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":199,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:07:16.559597+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_clean_scrub_stamp":"2024-09-18T08:07:16.559597+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.7","version":"216'8356","reported_seq":8539,"reported_epoch":216,"state":"active+clean","last_fresh":"2024-09-18T08:09:58.678244+0000","last_change":"2024-09-18T08:09:36.764450+0000","last_active":"2024-09-18T08:09:58.678244+0000","last_peered":"2024-09-18T08:09:58.678244+0000","last_clean":"2024-09-18T08:09:58.678244+0000","last_became_active":"2024-09-18T08:07:53.674433+0000","last_became_peered":"2024-09-18T08:07:53.674433+0000","last_unstale":"2024-09-18T08:09:58.678244+0000","last_undegraded":"2024-09-18T08:09:58.678244+0000","last_fullsized":"2024-09-18T08:09:58.678244+0000","mapping_epoch":165,"log_start":"190'3500","ondisk_log_start":"190'3500","created":165,"last_epoch_clean":166,"parent":"0.0","parent_split_bits":0,"last_scrub":"213'6848","last_scrub_stamp":"2024-09-18T08:09:36.764206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:07:52.307205+0000","last_clean_scrub_stamp":"2024-09-18T08:09:36.764206+0000","objects_scrubbed":5822,"log_size":4856,"log_dups_size":0,"ondisk_log_size":4856,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":4,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:11:06.120132+0000","scrub_duration":3143,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":276430848,"num_objects":4218,"num_object_clones":0,"num_object_copies":12654,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4218,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8356,"num_write_kb":402368,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,6],"acting":[1,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.b","version":"69'9","reported_seq":384,"reported_epoch":216,"state":"active+clean","last_fresh":"2024-09-18T08:09:58.644773+0000","last_change":"2024-09-18T08:08:18.640380+0000","last_active":"2024-09-18T08:09:58.644773+0000","last_peered":"2024-09-18T08:09:58.644773+0000","last_clean":"2024-09-18T08:09:58.644773+0000","last_became_active":"2024-09-18T08:08:18.522995+0000","last_became_peered":"2024-09-18T08:08:18.522995+0000","last_unstale":"2024-09-18T08:09:58.644773+0000","last_undegraded":"2024-09-18T08:09:58.644773+0000","last_fullsized":"2024-09-18T08:09:58.644773+0000","mapping_epoch":181,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":182,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'9","last_scrub_stamp":"2024-09-18T08:08:18.640292+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_clean_scrub_stamp":"2024-09-18T08:08:18.640292+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":350,"reported_epoch":215,"state":"active+clean","last_fresh":"2024-09-18T08:09:51.571563+0000","last_change":"2024-09-18T08:07:40.708363+0000","last_active":"2024-09-18T08:09:51.571563+0000","last_peered":"2024-09-18T08:09:51.571563+0000","last_clean":"2024-09-18T08:09:51.571563+0000","last_became_active":"2024-09-18T08:06:24.135005+0000","last_became_peered":"2024-09-18T08:06:24.135005+0000","last_unstale":"2024-09-18T08:09:51.571563+0000","last_undegraded":"2024-09-18T08:09:51.571563+0000","last_fullsized":"2024-09-18T08:09:51.571563+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:07:40.708307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:46.860092+0000","last_clean_scrub_stamp":"2024-09-18T08:07:40.708307+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":351,"reported_epoch":215,"state":"active+clean","last_fresh":"2024-09-18T08:09:51.577497+0000","last_change":"2024-09-18T08:08:18.509651+0000","last_active":"2024-09-18T08:09:51.577497+0000","last_peered":"2024-09-18T08:09:51.577497+0000","last_clean":"2024-09-18T08:09:51.577497+0000","last_became_active":"2024-09-18T08:08:18.509247+0000","last_became_peered":"2024-09-18T08:08:18.509247+0000","last_unstale":"2024-09-18T08:09:51.577497+0000","last_undegraded":"2024-09-18T08:09:51.577497+0000","last_fullsized":"2024-09-18T08:09:51.577497+0000","mapping_epoch":181,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":182,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:07:10.728679+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:50.886830+0000","last_clean_scrub_stamp":"2024-09-18T08:07:10.728679+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,3],"acting":[2,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":349,"reported_epoch":216,"state":"active+clean","last_fresh":"2024-09-18T08:09:58.645931+0000","last_change":"2024-09-18T08:09:28.086782+0000","last_active":"2024-09-18T08:09:58.645931+0000","last_peered":"2024-09-18T08:09:58.645931+0000","last_clean":"2024-09-18T08:09:58.645931+0000","last_became_active":"2024-09-18T08:08:52.697337+0000","last_became_peered":"2024-09-18T08:08:52.697337+0000","last_unstale":"2024-09-18T08:09:58.645931+0000","last_undegraded":"2024-09-18T08:09:58.645931+0000","last_fullsized":"2024-09-18T08:09:58.645931+0000","mapping_epoch":198,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":199,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:09:28.086727+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_clean_scrub_stamp":"2024-09-18T08:09:28.086727+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:10:33.905647+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.6","version":"216'8332","reported_seq":8444,"reported_epoch":216,"state":"active+clean","last_fresh":"2024-09-18T08:09:58.824474+0000","last_change":"2024-09-18T08:07:53.674440+0000","last_active":"2024-09-18T08:09:58.824474+0000","last_peered":"2024-09-18T08:09:58.824474+0000","last_clean":"2024-09-18T08:09:58.824474+0000","last_became_active":"2024-09-18T08:07:53.674139+0000","last_became_peered":"2024-09-18T08:07:53.674139+0000","last_unstale":"2024-09-18T08:09:58.824474+0000","last_undegraded":"2024-09-18T08:09:58.824474+0000","last_fullsized":"2024-09-18T08:09:58.824474+0000","mapping_epoch":165,"log_start":"184'2900","ondisk_log_start":"184'2900","created":165,"last_epoch_clean":166,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:07:52.307205+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:07:52.307205+0000","last_clean_scrub_stamp":"2024-09-18T08:07:52.307205+0000","objects_scrubbed":0,"log_size":5432,"log_dups_size":0,"ondisk_log_size":5432,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":275251200,"num_objects":4200,"num_object_clones":0,"num_object_copies":12600,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4200,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8332,"num_write_kb":401024,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,7],"acting":[0,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.a","version":"69'19","reported_seq":355,"reported_epoch":216,"state":"active+clean","last_fresh":"2024-09-18T08:09:58.660672+0000","last_change":"2024-09-18T08:07:41.000210+0000","last_active":"2024-09-18T08:09:58.660672+0000","last_peered":"2024-09-18T08:09:58.660672+0000","last_clean":"2024-09-18T08:09:58.660672+0000","last_became_active":"2024-09-18T08:06:24.139909+0000","last_became_peered":"2024-09-18T08:06:24.139909+0000","last_unstale":"2024-09-18T08:09:58.660672+0000","last_undegraded":"2024-09-18T08:09:58.660672+0000","last_fullsized":"2024-09-18T08:09:58.660672+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'19","last_scrub_stamp":"2024-09-18T08:07:41.000146+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_clean_scrub_stamp":"2024-09-18T08:07:41.000146+0000","objects_scrubbed":9,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:10:18.661451+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":31,"num_read_kb":20,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,7],"acting":[1,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":338,"reported_epoch":215,"state":"active+clean","last_fresh":"2024-09-18T08:09:51.569435+0000","last_change":"2024-09-18T08:06:47.638974+0000","last_active":"2024-09-18T08:09:51.569435+0000","last_peered":"2024-09-18T08:09:51.569435+0000","last_clean":"2024-09-18T08:09:51.569435+0000","last_became_active":"2024-09-18T08:06:46.171298+0000","last_became_peered":"2024-09-18T08:06:46.171298+0000","last_unstale":"2024-09-18T08:09:51.569435+0000","last_undegraded":"2024-09-18T08:09:51.569435+0000","last_fullsized":"2024-09-18T08:09:51.569435+0000","mapping_epoch":143,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":144,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:06:47.638911+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:46.860092+0000","last_clean_scrub_stamp":"2024-09-18T08:06:47.638911+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":44,"reported_epoch":215,"state":"active+clean","last_fresh":"2024-09-18T08:09:51.573100+0000","last_change":"2024-09-18T08:08:52.698243+0000","last_active":"2024-09-18T08:09:51.573100+0000","last_peered":"2024-09-18T08:09:51.573100+0000","last_clean":"2024-09-18T08:09:51.573100+0000","last_became_active":"2024-09-18T08:08:52.697932+0000","last_became_peered":"2024-09-18T08:08:52.697932+0000","last_unstale":"2024-09-18T08:09:51.573100+0000","last_undegraded":"2024-09-18T08:09:51.573100+0000","last_fullsized":"2024-09-18T08:09:51.573100+0000","mapping_epoch":198,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":199,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:08:16.631007+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:50.886830+0000","last_clean_scrub_stamp":"2024-09-18T08:08:16.631007+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":164,"reported_epoch":215,"state":"active+clean","last_fresh":"2024-09-18T08:09:51.573599+0000","last_change":"2024-09-18T08:06:50.902917+0000","last_active":"2024-09-18T08:09:51.573599+0000","last_peered":"2024-09-18T08:09:51.573599+0000","last_clean":"2024-09-18T08:09:51.573599+0000","last_became_active":"2024-09-18T08:06:46.137872+0000","last_became_peered":"2024-09-18T08:06:46.137872+0000","last_unstale":"2024-09-18T08:09:51.573599+0000","last_undegraded":"2024-09-18T08:09:51.573599+0000","last_fullsized":"2024-09-18T08:09:51.573599+0000","mapping_epoch":143,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":144,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:06:50.902835+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_clean_scrub_stamp":"2024-09-18T08:06:50.902835+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,3],"acting":[2,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"8.5","version":"215'8312","reported_seq":12735,"reported_epoch":215,"state":"active+clean","last_fresh":"2024-09-18T08:09:58.186915+0000","last_change":"2024-09-18T08:09:13.170209+0000","last_active":"2024-09-18T08:09:58.186915+0000","last_peered":"2024-09-18T08:09:58.186915+0000","last_clean":"2024-09-18T08:09:58.186915+0000","last_became_active":"2024-09-18T08:09:13.166459+0000","last_became_peered":"2024-09-18T08:09:13.166459+0000","last_unstale":"2024-09-18T08:09:58.186915+0000","last_undegraded":"2024-09-18T08:09:58.186915+0000","last_fullsized":"2024-09-18T08:09:58.186915+0000","mapping_epoch":205,"log_start":"191'3774","ondisk_log_start":"191'3774","created":165,"last_epoch_clean":206,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:07:52.307205+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:07:52.307205+0000","last_clean_scrub_stamp":"2024-09-18T08:07:52.307205+0000","objects_scrubbed":0,"log_size":4538,"log_dups_size":0,"ondisk_log_size":4538,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":280363008,"num_objects":4278,"num_object_clones":0,"num_object_copies":12834,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4278,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8312,"num_write_kb":402880,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2146,"num_bytes_recovered":140640256,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,7],"acting":[4,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.9","version":"69'12","reported_seq":90,"reported_epoch":215,"state":"active+clean","last_fresh":"2024-09-18T08:09:51.557833+0000","last_change":"2024-09-18T08:08:18.599669+0000","last_active":"2024-09-18T08:09:51.557833+0000","last_peered":"2024-09-18T08:09:51.557833+0000","last_clean":"2024-09-18T08:09:51.557833+0000","last_became_active":"2024-09-18T08:08:18.544489+0000","last_became_peered":"2024-09-18T08:08:18.544489+0000","last_unstale":"2024-09-18T08:09:51.557833+0000","last_undegraded":"2024-09-18T08:09:51.557833+0000","last_fullsized":"2024-09-18T08:09:51.557833+0000","mapping_epoch":181,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":182,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'12","last_scrub_stamp":"2024-09-18T08:05:33.544135+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_clean_scrub_stamp":"2024-09-18T08:05:33.544135+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":24,"num_read_kb":15,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":366,"reported_epoch":215,"state":"active+clean","last_fresh":"2024-09-18T08:09:51.571226+0000","last_change":"2024-09-18T08:09:39.614251+0000","last_active":"2024-09-18T08:09:51.571226+0000","last_peered":"2024-09-18T08:09:51.571226+0000","last_clean":"2024-09-18T08:09:51.571226+0000","last_became_active":"2024-09-18T08:08:18.539573+0000","last_became_peered":"2024-09-18T08:08:18.539573+0000","last_unstale":"2024-09-18T08:09:51.571226+0000","last_undegraded":"2024-09-18T08:09:51.571226+0000","last_fullsized":"2024-09-18T08:09:51.571226+0000","mapping_epoch":181,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":182,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:09:39.614187+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:46.860092+0000","last_clean_scrub_stamp":"2024-09-18T08:09:39.614187+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:10:57.546531+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":329,"reported_epoch":215,"state":"active+clean","last_fresh":"2024-09-18T08:09:51.573815+0000","last_change":"2024-09-18T08:08:15.744330+0000","last_active":"2024-09-18T08:09:51.573815+0000","last_peered":"2024-09-18T08:09:51.573815+0000","last_clean":"2024-09-18T08:09:51.573815+0000","last_became_active":"2024-09-18T08:02:51.902163+0000","last_became_peered":"2024-09-18T08:02:51.902163+0000","last_unstale":"2024-09-18T08:09:51.573815+0000","last_undegraded":"2024-09-18T08:09:51.573815+0000","last_fullsized":"2024-09-18T08:09:51.573815+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:08:15.744245+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:50.886830+0000","last_clean_scrub_stamp":"2024-09-18T08:08:15.744245+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":74,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":328,"reported_epoch":216,"state":"active+clean","last_fresh":"2024-09-18T08:09:58.653066+0000","last_change":"2024-09-18T08:07:44.113695+0000","last_active":"2024-09-18T08:09:58.653066+0000","last_peered":"2024-09-18T08:09:58.653066+0000","last_clean":"2024-09-18T08:09:58.653066+0000","last_became_active":"2024-09-18T08:02:53.907412+0000","last_became_peered":"2024-09-18T08:02:53.907412+0000","last_unstale":"2024-09-18T08:09:58.653066+0000","last_undegraded":"2024-09-18T08:09:58.653066+0000","last_fullsized":"2024-09-18T08:09:58.653066+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:07:44.113639+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_clean_scrub_stamp":"2024-09-18T08:07:44.113639+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.4","version":"215'8358","reported_seq":20355,"reported_epoch":215,"state":"active+recovering+undersized+remapped","last_fresh":"2024-09-18T08:09:58.293344+0000","last_change":"2024-09-18T08:08:56.896339+0000","last_active":"2024-09-18T08:09:58.293344+0000","last_peered":"2024-09-18T08:09:58.293344+0000","last_clean":"2024-09-18T08:08:51.672524+0000","last_became_active":"2024-09-18T08:08:53.828580+0000","last_became_peered":"2024-09-18T08:08:53.828580+0000","last_unstale":"2024-09-18T08:09:58.293344+0000","last_undegraded":"2024-09-18T08:09:58.293344+0000","last_fullsized":"2024-09-18T08:08:53.682963+0000","mapping_epoch":199,"log_start":"186'3200","ondisk_log_start":"186'3200","created":165,"last_epoch_clean":190,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:07:52.307205+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:07:52.307205+0000","last_clean_scrub_stamp":"2024-09-18T08:07:52.307205+0000","objects_scrubbed":0,"log_size":5158,"log_dups_size":0,"ondisk_log_size":5158,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":276037632,"num_objects":4212,"num_object_clones":0,"num_object_copies":12636,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1178,"num_objects_unfound":0,"num_objects_dirty":4212,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8358,"num_write_kb":402240,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5929,"num_bytes_recovered":388562944,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,2],"acting":[5,4],"avail_no_missing":["5","3","4","6"],"object_location_counts":[{"shards":"4,5","objects":3034},{"shards":"4,5,6","objects":1178}],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.8","version":"69'15","reported_seq":395,"reported_epoch":215,"state":"active+clean","last_fresh":"2024-09-18T08:09:51.570067+0000","last_change":"2024-09-18T08:06:24.162625+0000","last_active":"2024-09-18T08:09:51.570067+0000","last_peered":"2024-09-18T08:09:51.570067+0000","last_clean":"2024-09-18T08:09:51.570067+0000","last_became_active":"2024-09-18T08:06:24.138259+0000","last_became_peered":"2024-09-18T08:06:24.138259+0000","last_unstale":"2024-09-18T08:09:51.570067+0000","last_undegraded":"2024-09-18T08:09:51.570067+0000","last_fullsized":"2024-09-18T08:09:51.570067+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'15","last_scrub_stamp":"2024-09-18T08:04:26.859044+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_clean_scrub_stamp":"2024-09-18T08:04:26.859044+0000","objects_scrubbed":7,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.f","version":"62'2","reported_seq":382,"reported_epoch":215,"state":"active+clean","last_fresh":"2024-09-18T08:09:51.570918+0000","last_change":"2024-09-18T08:08:18.629506+0000","last_active":"2024-09-18T08:09:51.570918+0000","last_peered":"2024-09-18T08:09:51.570918+0000","last_clean":"2024-09-18T08:09:51.570918+0000","last_became_active":"2024-09-18T08:08:18.524593+0000","last_became_peered":"2024-09-18T08:08:18.524593+0000","last_unstale":"2024-09-18T08:09:51.570918+0000","last_undegraded":"2024-09-18T08:09:51.570918+0000","last_fullsized":"2024-09-18T08:09:51.570918+0000","mapping_epoch":181,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":182,"parent":"0.0","parent_split_bits":0,"last_scrub":"62'2","last_scrub_stamp":"2024-09-18T08:07:43.540222+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:46.860092+0000","last_clean_scrub_stamp":"2024-09-18T08:07:43.540222+0000","objects_scrubbed":2,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":4,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":184,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.9","version":"182'12","reported_seq":487,"reported_epoch":215,"state":"active+clean","last_fresh":"2024-09-18T08:09:53.012234+0000","last_change":"2024-09-18T08:09:46.742515+0000","last_active":"2024-09-18T08:09:53.012234+0000","last_peered":"2024-09-18T08:09:53.012234+0000","last_clean":"2024-09-18T08:09:53.012234+0000","last_became_active":"2024-09-18T08:08:18.524858+0000","last_became_peered":"2024-09-18T08:08:18.524858+0000","last_unstale":"2024-09-18T08:09:53.012234+0000","last_undegraded":"2024-09-18T08:09:53.012234+0000","last_fullsized":"2024-09-18T08:09:53.012234+0000","mapping_epoch":181,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":182,"parent":"0.0","parent_split_bits":0,"last_scrub":"182'12","last_scrub_stamp":"2024-09-18T08:09:46.742442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:50.886830+0000","last_clean_scrub_stamp":"2024-09-18T08:09:46.742442+0000","objects_scrubbed":1,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:11:07.487575+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":338,"reported_epoch":215,"state":"active+clean","last_fresh":"2024-09-18T08:09:51.571883+0000","last_change":"2024-09-18T08:07:45.707631+0000","last_active":"2024-09-18T08:09:51.571883+0000","last_peered":"2024-09-18T08:09:51.571883+0000","last_clean":"2024-09-18T08:09:51.571883+0000","last_became_active":"2024-09-18T08:06:24.133119+0000","last_became_peered":"2024-09-18T08:06:24.133119+0000","last_unstale":"2024-09-18T08:09:51.571883+0000","last_undegraded":"2024-09-18T08:09:51.571883+0000","last_fullsized":"2024-09-18T08:09:51.571883+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:07:45.707562+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_clean_scrub_stamp":"2024-09-18T08:07:45.707562+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":256,"reported_epoch":216,"state":"active+clean","last_fresh":"2024-09-18T08:09:58.643243+0000","last_change":"2024-09-18T08:06:24.133955+0000","last_active":"2024-09-18T08:09:58.643243+0000","last_peered":"2024-09-18T08:09:58.643243+0000","last_clean":"2024-09-18T08:09:58.643243+0000","last_became_active":"2024-09-18T08:06:24.133644+0000","last_became_peered":"2024-09-18T08:06:24.133644+0000","last_unstale":"2024-09-18T08:09:58.643243+0000","last_undegraded":"2024-09-18T08:09:58.643243+0000","last_fullsized":"2024-09-18T08:09:58.643243+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:04:16.378205+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:46.860092+0000","last_clean_scrub_stamp":"2024-09-18T08:04:16.378205+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:10:20.052218+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.17","version":"69'6","reported_seq":351,"reported_epoch":216,"state":"active+clean","last_fresh":"2024-09-18T08:09:58.659597+0000","last_change":"2024-09-18T08:07:19.309941+0000","last_active":"2024-09-18T08:09:58.659597+0000","last_peered":"2024-09-18T08:09:58.659597+0000","last_clean":"2024-09-18T08:09:58.659597+0000","last_became_active":"2024-09-18T08:02:49.895067+0000","last_became_peered":"2024-09-18T08:02:49.895067+0000","last_unstale":"2024-09-18T08:09:58.659597+0000","last_undegraded":"2024-09-18T08:09:58.659597+0000","last_fullsized":"2024-09-18T08:09:58.659597+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'6","last_scrub_stamp":"2024-09-18T08:07:19.309844+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_clean_scrub_stamp":"2024-09-18T08:07:19.309844+0000","objects_scrubbed":3,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":26,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":9,"num_read_kb":6,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":327,"reported_epoch":215,"state":"active+clean","last_fresh":"2024-09-18T08:09:51.569024+0000","last_change":"2024-09-18T08:05:45.812436+0000","last_active":"2024-09-18T08:09:51.569024+0000","last_peered":"2024-09-18T08:09:51.569024+0000","last_clean":"2024-09-18T08:09:51.569024+0000","last_became_active":"2024-09-18T08:02:51.912557+0000","last_became_peered":"2024-09-18T08:02:51.912557+0000","last_unstale":"2024-09-18T08:09:51.569024+0000","last_undegraded":"2024-09-18T08:09:51.569024+0000","last_fullsized":"2024-09-18T08:09:51.569024+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:05:45.812389+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:50.886830+0000","last_clean_scrub_stamp":"2024-09-18T08:05:45.812389+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,1],"acting":[5,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":360,"reported_epoch":215,"state":"active+clean","last_fresh":"2024-09-18T08:09:51.570598+0000","last_change":"2024-09-18T08:08:18.526113+0000","last_active":"2024-09-18T08:09:51.570598+0000","last_peered":"2024-09-18T08:09:51.570598+0000","last_clean":"2024-09-18T08:09:51.570598+0000","last_became_active":"2024-09-18T08:08:18.525260+0000","last_became_peered":"2024-09-18T08:08:18.525260+0000","last_unstale":"2024-09-18T08:09:51.570598+0000","last_undegraded":"2024-09-18T08:09:51.570598+0000","last_fullsized":"2024-09-18T08:09:51.570598+0000","mapping_epoch":181,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":182,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:07:53.670107+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_clean_scrub_stamp":"2024-09-18T08:07:53.670107+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:10:19.732636+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"69'9","reported_seq":357,"reported_epoch":216,"state":"active+clean","last_fresh":"2024-09-18T08:09:58.645172+0000","last_change":"2024-09-18T08:09:44.116395+0000","last_active":"2024-09-18T08:09:58.645172+0000","last_peered":"2024-09-18T08:09:58.645172+0000","last_clean":"2024-09-18T08:09:58.645172+0000","last_became_active":"2024-09-18T08:02:49.891567+0000","last_became_peered":"2024-09-18T08:02:49.891567+0000","last_unstale":"2024-09-18T08:09:58.645172+0000","last_undegraded":"2024-09-18T08:09:58.645172+0000","last_fullsized":"2024-09-18T08:09:58.645172+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'9","last_scrub_stamp":"2024-09-18T08:09:44.116308+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_clean_scrub_stamp":"2024-09-18T08:09:44.116308+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:11:07.916395+0000","scrub_duration":43,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":78,"reported_epoch":215,"state":"active+clean","last_fresh":"2024-09-18T08:09:51.557956+0000","last_change":"2024-09-18T08:08:18.550531+0000","last_active":"2024-09-18T08:09:51.557956+0000","last_peered":"2024-09-18T08:09:51.557956+0000","last_clean":"2024-09-18T08:09:51.557956+0000","last_became_active":"2024-09-18T08:08:18.545953+0000","last_became_peered":"2024-09-18T08:08:18.545953+0000","last_unstale":"2024-09-18T08:09:51.557956+0000","last_undegraded":"2024-09-18T08:09:51.557956+0000","last_fullsized":"2024-09-18T08:09:51.557956+0000","mapping_epoch":181,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":182,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:06:13.214705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:46.860092+0000","last_clean_scrub_stamp":"2024-09-18T08:06:13.214705+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":330,"reported_epoch":216,"state":"active+clean","last_fresh":"2024-09-18T08:09:58.659350+0000","last_change":"2024-09-18T08:09:46.560057+0000","last_active":"2024-09-18T08:09:58.659350+0000","last_peered":"2024-09-18T08:09:58.659350+0000","last_clean":"2024-09-18T08:09:58.659350+0000","last_became_active":"2024-09-18T08:02:51.907361+0000","last_became_peered":"2024-09-18T08:02:51.907361+0000","last_unstale":"2024-09-18T08:09:58.659350+0000","last_undegraded":"2024-09-18T08:09:58.659350+0000","last_fullsized":"2024-09-18T08:09:58.659350+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:09:46.557349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:50.886830+0000","last_clean_scrub_stamp":"2024-09-18T08:09:46.557349+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:11:08.318311+0000","scrub_duration":22,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":340,"reported_epoch":215,"state":"active+clean","last_fresh":"2024-09-18T08:09:51.580201+0000","last_change":"2024-09-18T08:08:18.511719+0000","last_active":"2024-09-18T08:09:51.580201+0000","last_peered":"2024-09-18T08:09:51.580201+0000","last_clean":"2024-09-18T08:09:51.580201+0000","last_became_active":"2024-09-18T08:08:18.511019+0000","last_became_peered":"2024-09-18T08:08:18.511019+0000","last_unstale":"2024-09-18T08:09:51.580201+0000","last_undegraded":"2024-09-18T08:09:51.580201+0000","last_fullsized":"2024-09-18T08:09:51.580201+0000","mapping_epoch":181,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":182,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:08:10.660508+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_clean_scrub_stamp":"2024-09-18T08:08:10.660508+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.15","version":"69'9","reported_seq":347,"reported_epoch":215,"state":"active+clean","last_fresh":"2024-09-18T08:09:51.569550+0000","last_change":"2024-09-18T08:05:36.896299+0000","last_active":"2024-09-18T08:09:51.569550+0000","last_peered":"2024-09-18T08:09:51.569550+0000","last_clean":"2024-09-18T08:09:51.569550+0000","last_became_active":"2024-09-18T08:02:49.890497+0000","last_became_peered":"2024-09-18T08:02:49.890497+0000","last_unstale":"2024-09-18T08:09:51.569550+0000","last_undegraded":"2024-09-18T08:09:51.569550+0000","last_fullsized":"2024-09-18T08:09:51.569550+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'9","last_scrub_stamp":"2024-09-18T08:05:36.896238+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_clean_scrub_stamp":"2024-09-18T08:05:36.896238+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":354,"reported_epoch":216,"state":"active+clean","last_fresh":"2024-09-18T08:09:58.644444+0000","last_change":"2024-09-18T08:09:46.035501+0000","last_active":"2024-09-18T08:09:58.644444+0000","last_peered":"2024-09-18T08:09:58.644444+0000","last_clean":"2024-09-18T08:09:58.644444+0000","last_became_active":"2024-09-18T08:02:47.875983+0000","last_became_peered":"2024-09-18T08:02:47.875983+0000","last_unstale":"2024-09-18T08:09:58.644444+0000","last_undegraded":"2024-09-18T08:09:58.644444+0000","last_fullsized":"2024-09-18T08:09:58.644444+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:09:46.035332+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:46.860092+0000","last_clean_scrub_stamp":"2024-09-18T08:09:46.035332+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:10:47.519129+0000","scrub_duration":18,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.14","version":"199'12","reported_seq":450,"reported_epoch":216,"state":"active+clean","last_fresh":"2024-09-18T08:09:58.660381+0000","last_change":"2024-09-18T08:08:52.701287+0000","last_active":"2024-09-18T08:09:58.660381+0000","last_peered":"2024-09-18T08:09:58.660381+0000","last_clean":"2024-09-18T08:09:58.660381+0000","last_became_active":"2024-09-18T08:08:52.696201+0000","last_became_peered":"2024-09-18T08:08:52.696201+0000","last_unstale":"2024-09-18T08:09:58.660381+0000","last_undegraded":"2024-09-18T08:09:58.660381+0000","last_fullsized":"2024-09-18T08:09:58.660381+0000","mapping_epoch":198,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":199,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'8","last_scrub_stamp":"2024-09-18T08:07:16.241607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:50.886830+0000","last_clean_scrub_stamp":"2024-09-18T08:07:16.241607+0000","objects_scrubbed":1,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":78,"reported_epoch":215,"state":"active+clean","last_fresh":"2024-09-18T08:09:51.558869+0000","last_change":"2024-09-18T08:08:18.523205+0000","last_active":"2024-09-18T08:09:51.558869+0000","last_peered":"2024-09-18T08:09:51.558869+0000","last_clean":"2024-09-18T08:09:51.558869+0000","last_became_active":"2024-09-18T08:08:18.522483+0000","last_became_peered":"2024-09-18T08:08:18.522483+0000","last_unstale":"2024-09-18T08:09:51.558869+0000","last_undegraded":"2024-09-18T08:09:51.558869+0000","last_fullsized":"2024-09-18T08:09:51.558869+0000","mapping_epoch":181,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":182,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:06:43.510513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_clean_scrub_stamp":"2024-09-18T08:06:43.510513+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.14","version":"69'10","reported_seq":369,"reported_epoch":216,"state":"active+clean","last_fresh":"2024-09-18T08:09:58.660887+0000","last_change":"2024-09-18T08:08:08.493614+0000","last_active":"2024-09-18T08:09:58.660887+0000","last_peered":"2024-09-18T08:09:58.660887+0000","last_clean":"2024-09-18T08:09:58.660887+0000","last_became_active":"2024-09-18T08:04:18.150968+0000","last_became_peered":"2024-09-18T08:04:18.150968+0000","last_unstale":"2024-09-18T08:09:58.660887+0000","last_undegraded":"2024-09-18T08:09:58.660887+0000","last_fullsized":"2024-09-18T08:09:58.660887+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'10","last_scrub_stamp":"2024-09-18T08:08:08.493522+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_clean_scrub_stamp":"2024-09-18T08:08:08.493522+0000","objects_scrubbed":5,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,7],"acting":[3,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":358,"reported_epoch":215,"state":"active+clean","last_fresh":"2024-09-18T08:09:51.570355+0000","last_change":"2024-09-18T08:09:31.658337+0000","last_active":"2024-09-18T08:09:51.570355+0000","last_peered":"2024-09-18T08:09:51.570355+0000","last_clean":"2024-09-18T08:09:51.570355+0000","last_became_active":"2024-09-18T08:08:18.524218+0000","last_became_peered":"2024-09-18T08:08:18.524218+0000","last_unstale":"2024-09-18T08:09:51.570355+0000","last_undegraded":"2024-09-18T08:09:51.570355+0000","last_fullsized":"2024-09-18T08:09:51.570355+0000","mapping_epoch":181,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":182,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:09:31.658259+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:46.860092+0000","last_clean_scrub_stamp":"2024-09-18T08:09:31.658259+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:10:50.143994+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.15","version":"69'8","reported_seq":430,"reported_epoch":215,"state":"active+clean","last_fresh":"2024-09-18T08:09:53.012230+0000","last_change":"2024-09-18T08:06:45.555513+0000","last_active":"2024-09-18T08:09:53.012230+0000","last_peered":"2024-09-18T08:09:53.012230+0000","last_clean":"2024-09-18T08:09:53.012230+0000","last_became_active":"2024-09-18T08:02:51.914135+0000","last_became_peered":"2024-09-18T08:02:51.914135+0000","last_unstale":"2024-09-18T08:09:53.012230+0000","last_undegraded":"2024-09-18T08:09:53.012230+0000","last_fullsized":"2024-09-18T08:09:53.012230+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'8","last_scrub_stamp":"2024-09-18T08:06:45.555459+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:50.886830+0000","last_clean_scrub_stamp":"2024-09-18T08:06:45.555459+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":334,"reported_epoch":216,"state":"active+clean","last_fresh":"2024-09-18T08:09:58.644632+0000","last_change":"2024-09-18T08:09:42.047337+0000","last_active":"2024-09-18T08:09:58.644632+0000","last_peered":"2024-09-18T08:09:58.644632+0000","last_clean":"2024-09-18T08:09:58.644632+0000","last_became_active":"2024-09-18T08:02:53.912112+0000","last_became_peered":"2024-09-18T08:02:53.912112+0000","last_unstale":"2024-09-18T08:09:58.644632+0000","last_undegraded":"2024-09-18T08:09:58.644632+0000","last_fullsized":"2024-09-18T08:09:58.644632+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:09:42.047270+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_clean_scrub_stamp":"2024-09-18T08:09:42.047270+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:10:59.660034+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.13","version":"69'11","reported_seq":90,"reported_epoch":215,"state":"active+clean","last_fresh":"2024-09-18T08:09:51.558237+0000","last_change":"2024-09-18T08:08:18.643772+0000","last_active":"2024-09-18T08:09:51.558237+0000","last_peered":"2024-09-18T08:09:51.558237+0000","last_clean":"2024-09-18T08:09:51.558237+0000","last_became_active":"2024-09-18T08:08:18.539712+0000","last_became_peered":"2024-09-18T08:08:18.539712+0000","last_unstale":"2024-09-18T08:09:51.558237+0000","last_undegraded":"2024-09-18T08:09:51.558237+0000","last_fullsized":"2024-09-18T08:09:51.558237+0000","mapping_epoch":181,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":182,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'11","last_scrub_stamp":"2024-09-18T08:07:53.824401+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_clean_scrub_stamp":"2024-09-18T08:07:53.824401+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":19,"num_read_kb":12,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":78,"reported_epoch":215,"state":"active+clean","last_fresh":"2024-09-18T08:09:51.558840+0000","last_change":"2024-09-18T08:08:18.623643+0000","last_active":"2024-09-18T08:09:51.558840+0000","last_peered":"2024-09-18T08:09:51.558840+0000","last_clean":"2024-09-18T08:09:51.558840+0000","last_became_active":"2024-09-18T08:08:18.622893+0000","last_became_peered":"2024-09-18T08:08:18.622893+0000","last_unstale":"2024-09-18T08:09:51.558840+0000","last_undegraded":"2024-09-18T08:09:51.558840+0000","last_fullsized":"2024-09-18T08:09:51.558840+0000","mapping_epoch":181,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":182,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:06:46.502422+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:46.860092+0000","last_clean_scrub_stamp":"2024-09-18T08:06:46.502422+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":338,"reported_epoch":216,"state":"active+clean","last_fresh":"2024-09-18T08:09:58.660538+0000","last_change":"2024-09-18T08:09:03.582984+0000","last_active":"2024-09-18T08:09:58.660538+0000","last_peered":"2024-09-18T08:09:58.660538+0000","last_clean":"2024-09-18T08:09:58.660538+0000","last_became_active":"2024-09-18T08:02:51.910752+0000","last_became_peered":"2024-09-18T08:02:51.910752+0000","last_unstale":"2024-09-18T08:09:58.660538+0000","last_undegraded":"2024-09-18T08:09:58.660538+0000","last_fullsized":"2024-09-18T08:09:58.660538+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:09:03.582914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:50.886830+0000","last_clean_scrub_stamp":"2024-09-18T08:09:03.582914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:10:14.069633+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":336,"reported_epoch":216,"state":"active+clean","last_fresh":"2024-09-18T08:09:58.652522+0000","last_change":"2024-09-18T08:07:54.314026+0000","last_active":"2024-09-18T08:09:58.652522+0000","last_peered":"2024-09-18T08:09:58.652522+0000","last_clean":"2024-09-18T08:09:58.652522+0000","last_became_active":"2024-09-18T08:02:53.911429+0000","last_became_peered":"2024-09-18T08:02:53.911429+0000","last_unstale":"2024-09-18T08:09:58.652522+0000","last_undegraded":"2024-09-18T08:09:58.652522+0000","last_fullsized":"2024-09-18T08:09:58.652522+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:07:54.313971+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_clean_scrub_stamp":"2024-09-18T08:07:54.313971+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.12","version":"69'9","reported_seq":376,"reported_epoch":216,"state":"active+clean","last_fresh":"2024-09-18T08:09:58.660920+0000","last_change":"2024-09-18T08:08:52.658353+0000","last_active":"2024-09-18T08:09:58.660920+0000","last_peered":"2024-09-18T08:09:58.660920+0000","last_clean":"2024-09-18T08:09:58.660920+0000","last_became_active":"2024-09-18T08:06:24.132270+0000","last_became_peered":"2024-09-18T08:06:24.132270+0000","last_unstale":"2024-09-18T08:09:58.660920+0000","last_undegraded":"2024-09-18T08:09:58.660920+0000","last_fullsized":"2024-09-18T08:09:58.660920+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'9","last_scrub_stamp":"2024-09-18T08:08:52.658282+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_clean_scrub_stamp":"2024-09-18T08:08:52.658282+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:10:04.461414+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":366,"reported_epoch":215,"state":"active+clean","last_fresh":"2024-09-18T08:09:51.571333+0000","last_change":"2024-09-18T08:09:37.532792+0000","last_active":"2024-09-18T08:09:51.571333+0000","last_peered":"2024-09-18T08:09:51.571333+0000","last_clean":"2024-09-18T08:09:51.571333+0000","last_became_active":"2024-09-18T08:08:18.525527+0000","last_became_peered":"2024-09-18T08:08:18.525527+0000","last_unstale":"2024-09-18T08:09:51.571333+0000","last_undegraded":"2024-09-18T08:09:51.571333+0000","last_fullsized":"2024-09-18T08:09:51.571333+0000","mapping_epoch":181,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":182,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:09:37.532573+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:46.860092+0000","last_clean_scrub_stamp":"2024-09-18T08:09:37.532573+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:11:05.489513+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":331,"reported_epoch":216,"state":"active+clean","last_fresh":"2024-09-18T08:09:58.653154+0000","last_change":"2024-09-18T08:07:42.165434+0000","last_active":"2024-09-18T08:09:58.653154+0000","last_peered":"2024-09-18T08:09:58.653154+0000","last_clean":"2024-09-18T08:09:58.653154+0000","last_became_active":"2024-09-18T08:02:51.906277+0000","last_became_peered":"2024-09-18T08:02:51.906277+0000","last_unstale":"2024-09-18T08:09:58.653154+0000","last_undegraded":"2024-09-18T08:09:58.653154+0000","last_fullsized":"2024-09-18T08:09:58.653154+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:07:42.165388+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:50.886830+0000","last_clean_scrub_stamp":"2024-09-18T08:07:42.165388+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.10","version":"69'1","reported_seq":315,"reported_epoch":216,"state":"active+clean","last_fresh":"2024-09-18T08:09:58.642614+0000","last_change":"2024-09-18T08:05:51.429631+0000","last_active":"2024-09-18T08:09:58.642614+0000","last_peered":"2024-09-18T08:09:58.642614+0000","last_clean":"2024-09-18T08:09:58.642614+0000","last_became_active":"2024-09-18T08:02:53.912699+0000","last_became_peered":"2024-09-18T08:02:53.912699+0000","last_unstale":"2024-09-18T08:09:58.642614+0000","last_undegraded":"2024-09-18T08:09:58.642614+0000","last_fullsized":"2024-09-18T08:09:58.642614+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_clean_scrub_stamp":"2024-09-18T08:02:52.890475+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:10:16.084523+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.11","version":"69'11","reported_seq":400,"reported_epoch":216,"state":"active+clean","last_fresh":"2024-09-18T08:09:58.652699+0000","last_change":"2024-09-18T08:07:18.282247+0000","last_active":"2024-09-18T08:09:58.652699+0000","last_peered":"2024-09-18T08:09:58.652699+0000","last_clean":"2024-09-18T08:09:58.652699+0000","last_became_active":"2024-09-18T08:06:24.132416+0000","last_became_peered":"2024-09-18T08:06:24.132416+0000","last_unstale":"2024-09-18T08:09:58.652699+0000","last_undegraded":"2024-09-18T08:09:58.652699+0000","last_fullsized":"2024-09-18T08:09:58.652699+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'11","last_scrub_stamp":"2024-09-18T08:07:18.282155+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_clean_scrub_stamp":"2024-09-18T08:07:18.282155+0000","objects_scrubbed":5,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":335,"reported_epoch":215,"state":"active+clean","last_fresh":"2024-09-18T08:09:51.569112+0000","last_change":"2024-09-18T08:06:18.168254+0000","last_active":"2024-09-18T08:09:51.569112+0000","last_peered":"2024-09-18T08:09:51.569112+0000","last_clean":"2024-09-18T08:09:51.569112+0000","last_became_active":"2024-09-18T08:02:47.878886+0000","last_became_peered":"2024-09-18T08:02:47.878886+0000","last_unstale":"2024-09-18T08:09:51.569112+0000","last_undegraded":"2024-09-18T08:09:51.569112+0000","last_fullsized":"2024-09-18T08:09:51.569112+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:06:18.168206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:46.860092+0000","last_clean_scrub_stamp":"2024-09-18T08:06:18.168206+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":372,"reported_epoch":215,"state":"active+clean","last_fresh":"2024-09-18T08:09:51.569868+0000","last_change":"2024-09-18T08:09:49.824264+0000","last_active":"2024-09-18T08:09:51.569868+0000","last_peered":"2024-09-18T08:09:51.569868+0000","last_clean":"2024-09-18T08:09:51.569868+0000","last_became_active":"2024-09-18T08:08:18.524966+0000","last_became_peered":"2024-09-18T08:08:18.524966+0000","last_unstale":"2024-09-18T08:09:51.569868+0000","last_undegraded":"2024-09-18T08:09:51.569868+0000","last_fullsized":"2024-09-18T08:09:51.569868+0000","mapping_epoch":181,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":182,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:09:49.824196+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:50.886830+0000","last_clean_scrub_stamp":"2024-09-18T08:09:49.824196+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:10:51.928894+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":341,"reported_epoch":216,"state":"active+clean","last_fresh":"2024-09-18T08:09:58.659779+0000","last_change":"2024-09-18T08:07:48.218010+0000","last_active":"2024-09-18T08:09:58.659779+0000","last_peered":"2024-09-18T08:09:58.659779+0000","last_clean":"2024-09-18T08:09:58.659779+0000","last_became_active":"2024-09-18T08:06:24.128582+0000","last_became_peered":"2024-09-18T08:06:24.128582+0000","last_unstale":"2024-09-18T08:09:58.659779+0000","last_undegraded":"2024-09-18T08:09:58.659779+0000","last_fullsized":"2024-09-18T08:09:58.659779+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:07:48.217964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_clean_scrub_stamp":"2024-09-18T08:07:48.217964+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.10","version":"69'4","reported_seq":367,"reported_epoch":216,"state":"active+clean","last_fresh":"2024-09-18T08:09:58.660158+0000","last_change":"2024-09-18T08:07:41.210418+0000","last_active":"2024-09-18T08:09:58.660158+0000","last_peered":"2024-09-18T08:09:58.660158+0000","last_clean":"2024-09-18T08:09:58.660158+0000","last_became_active":"2024-09-18T08:06:24.131309+0000","last_became_peered":"2024-09-18T08:06:24.131309+0000","last_unstale":"2024-09-18T08:09:58.660158+0000","last_undegraded":"2024-09-18T08:09:58.660158+0000","last_fullsized":"2024-09-18T08:09:58.660158+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'4","last_scrub_stamp":"2024-09-18T08:07:41.210347+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_clean_scrub_stamp":"2024-09-18T08:07:41.210347+0000","objects_scrubbed":2,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":6,"num_read_kb":4,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":345,"reported_epoch":216,"state":"active+clean","last_fresh":"2024-09-18T08:09:58.644696+0000","last_change":"2024-09-18T08:07:52.551689+0000","last_active":"2024-09-18T08:09:58.644696+0000","last_peered":"2024-09-18T08:09:58.644696+0000","last_clean":"2024-09-18T08:09:58.644696+0000","last_became_active":"2024-09-18T08:02:47.877616+0000","last_became_peered":"2024-09-18T08:02:47.877616+0000","last_unstale":"2024-09-18T08:09:58.644696+0000","last_undegraded":"2024-09-18T08:09:58.644696+0000","last_fullsized":"2024-09-18T08:09:58.644696+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:07:52.551616+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:46.860092+0000","last_clean_scrub_stamp":"2024-09-18T08:07:52.551616+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":78,"reported_epoch":215,"state":"active+clean","last_fresh":"2024-09-18T08:09:51.558050+0000","last_change":"2024-09-18T08:08:18.550244+0000","last_active":"2024-09-18T08:09:51.558050+0000","last_peered":"2024-09-18T08:09:51.558050+0000","last_clean":"2024-09-18T08:09:51.558050+0000","last_became_active":"2024-09-18T08:08:18.545361+0000","last_became_peered":"2024-09-18T08:08:18.545361+0000","last_unstale":"2024-09-18T08:09:51.558050+0000","last_undegraded":"2024-09-18T08:09:51.558050+0000","last_fullsized":"2024-09-18T08:09:51.558050+0000","mapping_epoch":181,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":182,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:07:17.605562+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:50.886830+0000","last_clean_scrub_stamp":"2024-09-18T08:07:17.605562+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,7],"acting":[4,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":355,"reported_epoch":215,"state":"active+clean","last_fresh":"2024-09-18T08:09:51.569802+0000","last_change":"2024-09-18T08:09:34.607443+0000","last_active":"2024-09-18T08:09:51.569802+0000","last_peered":"2024-09-18T08:09:51.569802+0000","last_clean":"2024-09-18T08:09:51.569802+0000","last_became_active":"2024-09-18T08:08:18.525552+0000","last_became_peered":"2024-09-18T08:08:18.525552+0000","last_unstale":"2024-09-18T08:09:51.569802+0000","last_undegraded":"2024-09-18T08:09:51.569802+0000","last_fullsized":"2024-09-18T08:09:51.569802+0000","mapping_epoch":181,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":182,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:09:34.607374+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_clean_scrub_stamp":"2024-09-18T08:09:34.607374+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:10:39.783401+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,4],"acting":[7,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":324,"reported_epoch":216,"state":"active+clean","last_fresh":"2024-09-18T08:09:58.660964+0000","last_change":"2024-09-18T08:08:18.539359+0000","last_active":"2024-09-18T08:09:58.660964+0000","last_peered":"2024-09-18T08:09:58.660964+0000","last_clean":"2024-09-18T08:09:58.660964+0000","last_became_active":"2024-09-18T08:08:18.538838+0000","last_became_peered":"2024-09-18T08:08:18.538838+0000","last_unstale":"2024-09-18T08:09:58.660964+0000","last_undegraded":"2024-09-18T08:09:58.660964+0000","last_fullsized":"2024-09-18T08:09:58.660964+0000","mapping_epoch":181,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":182,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_clean_scrub_stamp":"2024-09-18T08:02:52.890475+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:10:51.616752+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":338,"reported_epoch":216,"state":"active+clean","last_fresh":"2024-09-18T08:09:58.660413+0000","last_change":"2024-09-18T08:07:17.277964+0000","last_active":"2024-09-18T08:09:58.660413+0000","last_peered":"2024-09-18T08:09:58.660413+0000","last_clean":"2024-09-18T08:09:58.660413+0000","last_became_active":"2024-09-18T08:02:47.878922+0000","last_became_peered":"2024-09-18T08:02:47.878922+0000","last_unstale":"2024-09-18T08:09:58.660413+0000","last_undegraded":"2024-09-18T08:09:58.660413+0000","last_fullsized":"2024-09-18T08:09:58.660413+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:07:17.277894+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:46.860092+0000","last_clean_scrub_stamp":"2024-09-18T08:07:17.277894+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":16,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"69'11","reported_seq":325,"reported_epoch":215,"state":"active+clean","last_fresh":"2024-09-18T08:09:51.569678+0000","last_change":"2024-09-18T08:06:24.164086+0000","last_active":"2024-09-18T08:09:51.569678+0000","last_peered":"2024-09-18T08:09:51.569678+0000","last_clean":"2024-09-18T08:09:51.569678+0000","last_became_active":"2024-09-18T08:06:24.135232+0000","last_became_peered":"2024-09-18T08:06:24.135232+0000","last_unstale":"2024-09-18T08:09:51.569678+0000","last_undegraded":"2024-09-18T08:09:51.569678+0000","last_fullsized":"2024-09-18T08:09:51.569678+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'11","last_scrub_stamp":"2024-09-18T08:06:12.967462+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_clean_scrub_stamp":"2024-09-18T08:06:12.967462+0000","objects_scrubbed":5,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":19,"num_read_kb":12,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,1],"acting":[5,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":344,"reported_epoch":216,"state":"active+clean","last_fresh":"2024-09-18T08:09:58.644892+0000","last_change":"2024-09-18T08:08:52.701053+0000","last_active":"2024-09-18T08:09:58.644892+0000","last_peered":"2024-09-18T08:09:58.644892+0000","last_clean":"2024-09-18T08:09:58.644892+0000","last_became_active":"2024-09-18T08:08:52.700595+0000","last_became_peered":"2024-09-18T08:08:52.700595+0000","last_unstale":"2024-09-18T08:09:58.644892+0000","last_undegraded":"2024-09-18T08:09:58.644892+0000","last_fullsized":"2024-09-18T08:09:58.644892+0000","mapping_epoch":198,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":199,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:08:12.639376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:50.886830+0000","last_clean_scrub_stamp":"2024-09-18T08:08:12.639376+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":8,"num_pg":16,"stat_sum":{"num_bytes":4441964568,"num_objects":67780,"num_object_clones":0,"num_object_copies":203340,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1178,"num_objects_unfound":0,"num_objects_dirty":67780,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":133720,"num_write_kb":6447937,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":23066,"num_bytes_recovered":1511653376,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":14326640640,"data_stored":14326628424,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":78346,"ondisk_log_size":78346,"up":48,"acting":47,"num_store_stats":8},{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":416,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":85,"ondisk_log_size":85,"up":96,"acting":96,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":178,"num_object_clones":0,"num_object_copies":534,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":178,"num_whiteouts":0,"num_read":685,"num_read_kb":442,"num_write":417,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":214,"num_bytes_recovered":4114,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":417792,"data_stored":11106,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":398,"ondisk_log_size":398,"up":96,"acting":96,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":1613,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":24,"num_read_kb":24,"num_write":10,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":3226,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":4839,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":6,"ondisk_log_size":6,"up":96,"acting":96,"num_store_stats":8},{"poolid":2,"num_pg":3,"stat_sum":{"num_bytes":408,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":198,"num_read_kb":193,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":38,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":8,"ondisk_log_size":8,"up":9,"acting":9,"num_store_stats":7},{"poolid":1,"num_pg":11,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":869,"ondisk_log_size":869,"up":33,"acting":33,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":56,"seq":240518168666,"num_pgs":63,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1599340,"kb_used_data":1415452,"kb_used_omap":1,"kb_used_meta":183870,"kb_avail":92145812,"statfs":{"total":95995035648,"available":94357311488,"internally_reserved":0,"allocated":1449422848,"data_stored":1449030420,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":188283338},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1],"upper_bound":4},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":47,"seq":201863463008,"num_pgs":55,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1874068,"kb_used_data":1680452,"kb_used_omap":1,"kb_used_meta":193598,"kb_avail":91871084,"statfs":{"total":95995035648,"available":94075990016,"internally_reserved":0,"allocated":1720782848,"data_stored":1720413887,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":198244806},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":39,"seq":167503724646,"num_pgs":59,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2348712,"kb_used_data":2182872,"kb_used_omap":1,"kb_used_meta":165822,"kb_avail":91396440,"statfs":{"total":95995035648,"available":93589954560,"internally_reserved":0,"allocated":2235260928,"data_stored":2234872863,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":169802182},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":32,"seq":137438953579,"num_pgs":67,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1966964,"kb_used_data":1733092,"kb_used_omap":1,"kb_used_meta":233854,"kb_avail":91778188,"statfs":{"total":95995035648,"available":93980864512,"internally_reserved":0,"allocated":1774686208,"data_stored":1774286077,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":239466949},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":26,"seq":111669149810,"num_pgs":69,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1998652,"kb_used_data":1700908,"kb_used_omap":1,"kb_used_meta":297726,"kb_avail":91746500,"statfs":{"total":95995035648,"available":93948416000,"internally_reserved":0,"allocated":1741729792,"data_stored":1741321306,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":304871880},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411448,"num_pgs":49,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1867304,"kb_used_data":1569560,"kb_used_omap":1,"kb_used_meta":297726,"kb_avail":91877848,"statfs":{"total":95995035648,"available":94082916352,"internally_reserved":0,"allocated":1607229440,"data_stored":1606841481,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":304871877},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,1],"upper_bound":16},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574972,"num_pgs":63,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2176468,"kb_used_data":1914820,"kb_used_omap":1,"kb_used_meta":261630,"kb_avail":91568684,"statfs":{"total":95995035648,"available":93766332416,"internally_reserved":0,"allocated":1960775680,"data_stored":1960344080,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":267909573},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738499,"num_pgs":56,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2117188,"kb_used_data":1873584,"kb_used_omap":1,"kb_used_meta":243582,"kb_avail":91627964,"statfs":{"total":95995035648,"available":93827035136,"internally_reserved":0,"allocated":1918550016,"data_stored":1918144705,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":249428421},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":266240,"data_stored":262688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1131,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1521,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":436,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":46,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1177,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":436,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":77824,"data_stored":2008,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1128,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":1898,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1474,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":990,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1034,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1254,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":403,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":416,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":403,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":1915355136,"data_stored":1915355136,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":1955266560,"data_stored":1955266560,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":1601638400,"data_stored":1601634328,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":1728577536,"data_stored":1728577536,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":1771700224,"data_stored":1771700224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":2220363776,"data_stored":2220359704,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":1697120256,"data_stored":1697120256,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":1436618752,"data_stored":1436614680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T08:10:02.471 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid 779349ca-7593-11ef-bceb-c7b262605968 -- ceph status --format=json 2024-09-18T08:10:02.477 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:10:02.478 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:10:02.478 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3' 2024-09-18T08:10:02.579 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2 2024-09-18T08:10:02.625 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:10:02.625 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:10:02.626 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2' 2024-09-18T08:10:02.727 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4 2024-09-18T08:10:02.765 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:10:02.766 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:10:02.766 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4' 2024-09-18T08:10:02.867 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1 2024-09-18T08:10:02.904 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:10:02.905 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:10:02.905 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1' 2024-09-18T08:10:03.006 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3 2024-09-18T08:10:03.061 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:10:03.062 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:10:03.063 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3' 2024-09-18T08:10:03.135 INFO:teuthology.orchestra.run.smithi007.stdout:osd.0: {} 2024-09-18T08:10:03.135 INFO:teuthology.orchestra.run.smithi007.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-18T08:10:03.163 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5 2024-09-18T08:10:03.180 INFO:teuthology.orchestra.run.smithi007.stdout:osd.1: {} 2024-09-18T08:10:03.180 INFO:teuthology.orchestra.run.smithi007.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-18T08:10:03.198 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:10:03.199 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:10:03.199 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5' 2024-09-18T08:10:03.204 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/779349ca-7593-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T08:10:03.207 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/779349ca-7593-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T08:10:03.225 INFO:teuthology.orchestra.run.smithi007.stdout:osd.2: {} 2024-09-18T08:10:03.225 INFO:teuthology.orchestra.run.smithi007.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-18T08:10:03.263 INFO:teuthology.orchestra.run.smithi007.stdout:osd.3: {} 2024-09-18T08:10:03.263 INFO:teuthology.orchestra.run.smithi007.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-18T08:10:03.294 INFO:teuthology.orchestra.run.smithi007.stdout:osd.4: {} 2024-09-18T08:10:03.294 INFO:teuthology.orchestra.run.smithi007.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-18T08:10:03.300 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4 2024-09-18T08:10:03.331 INFO:teuthology.orchestra.run.smithi007.stdout:osd.5: {} 2024-09-18T08:10:03.331 INFO:teuthology.orchestra.run.smithi007.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-18T08:10:03.332 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:10:03 smithi007 ceph-mon[29459]: from='client.? 172.21.15.7:0/3394318943' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-18T08:10:03.332 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:10:03 smithi007 ceph-mon[39249]: from='client.? 172.21.15.7:0/3394318943' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-18T08:10:03.333 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:10:03.334 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:10:03.334 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4' 2024-09-18T08:10:03.358 DEBUG:teuthology.orchestra.run.smithi050:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid 779349ca-7593-11ef-bceb-c7b262605968 -- ceph daemon osd.7 dump_ops_in_flight 2024-09-18T08:10:03.369 INFO:teuthology.orchestra.run.smithi007.stdout:osd.6: {} 2024-09-18T08:10:03.369 INFO:teuthology.orchestra.run.smithi007.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-18T08:10:03.401 INFO:teuthology.orchestra.run.smithi007.stdout:osd.7: {} 2024-09-18T08:10:03.401 INFO:teuthology.orchestra.run.smithi007.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-18T08:10:03.405 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:10:03 smithi050 ceph-mon[32371]: from='client.? 172.21.15.7:0/3394318943' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-18T08:10:03.435 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6 2024-09-18T08:10:03.470 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:10:03.470 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:10:03.471 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6' 2024-09-18T08:10:03.571 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4 2024-09-18T08:10:03.602 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:10:03.603 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:10:03.603 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4' 2024-09-18T08:10:03.704 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0 2024-09-18T08:10:03.754 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:10:03.755 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:10:03.755 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0' 2024-09-18T08:10:03.785 INFO:teuthology.orchestra.run.smithi050.stderr:Inferring config /var/lib/ceph/779349ca-7593-11ef-bceb-c7b262605968/mon.b/config 2024-09-18T08:10:03.857 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1 2024-09-18T08:10:03.900 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:10:03.901 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:10:03.901 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1' 2024-09-18T08:10:04.002 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4 2024-09-18T08:10:04.038 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:10:04.039 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:10:04.040 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4' 2024-09-18T08:10:04.140 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5 2024-09-18T08:10:04.171 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:10:04.172 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:10:04.173 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5' 2024-09-18T08:10:04.224 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:10:04 smithi007 ceph-mon[29459]: from='mgr.24470 172.21.15.7:0/2594390261' entity='mgr.y' 2024-09-18T08:10:04.224 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:10:04 smithi007 ceph-mon[39249]: from='mgr.24470 172.21.15.7:0/2594390261' entity='mgr.y' 2024-09-18T08:10:04.224 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:10:04 smithi007 ceph-mon[39249]: from='mgr.24470 172.21.15.7:0/2594390261' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-18T08:10:04.224 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:10:04 smithi007 ceph-mon[39249]: pgmap v349: 158 pgs: 1 active+recovering+undersized+remapped, 157 active+clean; 4.1 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.18k op/s; 1178/201756 objects misplaced (0.584%); 3.3 MiB/s, 52 objects/s recovering 2024-09-18T08:10:04.273 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7 2024-09-18T08:10:04.305 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:10:04.306 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:10:04.306 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7' 2024-09-18T08:10:04.382 INFO:teuthology.orchestra.run.smithi050.stdout:ERROR: (22) Invalid argument 2024-09-18T08:10:04.383 INFO:teuthology.orchestra.run.smithi050.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-18T08:10:04.407 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2 2024-09-18T08:10:04.442 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:10:04.443 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:10:04.443 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2' 2024-09-18T08:10:04.444 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:10:04 smithi050 ceph-mon[32371]: from='mgr.24470 172.21.15.7:0/2594390261' entity='mgr.y' 2024-09-18T08:10:04.444 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:10:04 smithi050 ceph-mon[32371]: from='mgr.24470 172.21.15.7:0/2594390261' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-18T08:10:04.444 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:10:04 smithi050 ceph-mon[32371]: pgmap v349: 158 pgs: 1 active+recovering+undersized+remapped, 157 active+clean; 4.1 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.18k op/s; 1178/201756 objects misplaced (0.584%); 3.3 MiB/s, 52 objects/s recovering 2024-09-18T08:10:04.480 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:10:04 smithi007 ceph-mon[29459]: from='mgr.24470 172.21.15.7:0/2594390261' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-18T08:10:04.480 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:10:04 smithi007 ceph-mon[29459]: pgmap v349: 158 pgs: 1 active+recovering+undersized+remapped, 157 active+clean; 4.1 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.18k op/s; 1178/201756 objects misplaced (0.584%); 3.3 MiB/s, 52 objects/s recovering 2024-09-18T08:10:04.545 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7 2024-09-18T08:10:04.575 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:10:04.576 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:10:04.576 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7' 2024-09-18T08:10:04.681 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3 2024-09-18T08:10:04.725 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:10:04.726 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:10:04.727 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3' 2024-09-18T08:10:04.827 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4 2024-09-18T08:10:04.860 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:10:04.861 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:10:04.861 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4' 2024-09-18T08:10:04.962 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1 2024-09-18T08:10:04.999 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:10:04.999 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:10:05.000 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1' 2024-09-18T08:10:05.101 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4 2024-09-18T08:10:05.129 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:10:05.130 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:10:05.130 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4' 2024-09-18T08:10:05.231 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4 2024-09-18T08:10:05.233 INFO:teuthology.orchestra.run.smithi007.stderr:noscrub is unset 2024-09-18T08:10:05.253 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:10:05 smithi007 ceph-mon[29459]: from='client.? 172.21.15.7:0/2742465668' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-18T08:10:05.253 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:10:05 smithi007 ceph-mon[39249]: from='client.? 172.21.15.7:0/2742465668' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-18T08:10:05.260 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:10:05.261 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:10:05.261 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4' 2024-09-18T08:10:05.362 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2 2024-09-18T08:10:05.400 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:10:05.400 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:10:05.401 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2' 2024-09-18T08:10:05.405 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:10:05 smithi050 ceph-mon[32371]: from='client.? 172.21.15.7:0/2742465668' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-18T08:10:05.502 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5 2024-09-18T08:10:05.531 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:10:05.531 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:10:05.532 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5' 2024-09-18T08:10:05.599 DEBUG:teuthology.orchestra.run.smithi050:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid 779349ca-7593-11ef-bceb-c7b262605968 -- ceph daemon osd.7 dump_blocked_ops 2024-09-18T08:10:05.633 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7 2024-09-18T08:10:05.679 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:10:05.681 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:10:05.681 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7' 2024-09-18T08:10:05.782 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7 2024-09-18T08:10:05.815 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:10:05.816 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:10:05.816 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7' 2024-09-18T08:10:05.917 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5 2024-09-18T08:10:05.940 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:10:05.940 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:10:05.941 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5' 2024-09-18T08:10:05.957 INFO:teuthology.orchestra.run.smithi050.stderr:Inferring config /var/lib/ceph/779349ca-7593-11ef-bceb-c7b262605968/mon.b/config 2024-09-18T08:10:06.042 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7 2024-09-18T08:10:06.087 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:10:06.087 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:10:06.088 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7' 2024-09-18T08:10:06.111 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-09-18T08:10:06.112 INFO:teuthology.orchestra.run.smithi007.stdout:{"fsid":"779349ca-7593-11ef-bceb-c7b262605968","health":{"status":"HEALTH_WARN","checks":{"OSDMAP_FLAGS":{"severity":"HEALTH_WARN","summary":{"message":"nodeep-scrub flag(s) set","count":12},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 1 pg undersized","count":1},"muted":false}},"mutes":[]},"election_epoch":14,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":685,"monmap":{"epoch":3,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":217,"num_osds":8,"num_up_osds":8,"osd_up_since":1726646556,"num_in_osds":8,"osd_in_since":1726646897,"num_remapped_pgs":1},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":157},{"state_name":"active+recovering+undersized+remapped","count":1}],"num_pgs":158,"num_pools":7,"num_objects":62088,"data_bytes":4056488499,"bytes_used":15212191744,"bytes_avail":752748093440,"bytes_total":767960285184,"misplaced_objects":868,"misplaced_total":186264,"misplaced_ratio":0.0046600523987458658,"recovering_objects_per_sec":58,"recovering_bytes_per_sec":3832076,"recovering_keys_per_sec":0,"num_objects_recovered":585,"num_bytes_recovered":38338560,"num_keys_recovered":0,"read_bytes_sec":1023,"write_bytes_sec":0,"read_op_per_sec":0,"write_op_per_sec":1176},"fsmap":{"epoch":1,"btime":"2024-09-18T07:57:09:295514+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":1,"modules":["cephadm","dashboard","iostat","nfs","prometheus","restful"],"services":{"dashboard":"https://172.21.15.7:8443/","prometheus":"http://172.21.15.7:9283/"}},"servicemap":{"epoch":14,"modified":"2024-09-18T08:04:19.081938+0000","services":{"rgw":{"daemons":{"summary":"","24356":{"start_epoch":12,"start_stamp":"2024-09-18T08:02:54.139075+0000","gid":24356,"addr":"172.21.15.7:0/219168083","metadata":{"arch":"x86_64","ceph_release":"squid","ceph_version":"ceph version 19.3.0-4983-ge7fb7b56 (e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e) squid (dev)","ceph_version_short":"19.3.0-4983-ge7fb7b56","container_hostname":"smithi007","container_image":"quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:1f5b5a3734f665ef333266f39143ac3ef1eb1951b21258f2307e10332e2d7ac9","cpu":"Intel(R) Xeon(R) CPU E5-1620 v3 @ 3.50GHz","distro":"centos","distro_description":"CentOS Stream 9","distro_version":"9","frontend_config#0":"beast port=80","frontend_type#0":"beast","hostname":"smithi007","id":"foo.a","kernel_description":"#1 SMP PREEMPT_DYNAMIC Thu Sep 12 15:49:37 UTC 2024","kernel_version":"5.14.0-508.el9.x86_64","mem_swap_kb":"0","mem_total_kb":"32493484","num_handles":"1","os":"Linux","pid":"2","realm_id":"","realm_name":"","zone_id":"412e9c99-1835-4c2d-b8ee-228c30c92752","zone_name":"default","zonegroup_id":"f0acf781-d0ae-4f13-8505-d87ee1ac7608","zonegroup_name":"default"},"task_status":{}}}}}},"progress_events":{}} 2024-09-18T08:10:06.188 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2 2024-09-18T08:10:06.232 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:10:06.233 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:10:06.233 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2' 2024-09-18T08:10:06.334 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7 2024-09-18T08:10:06.352 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:10:06 smithi050 ceph-mon[32371]: pgmap v350: 158 pgs: 1 active+recovering+undersized+remapped, 157 active+clean; 3.8 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.18k op/s; 868/186264 objects misplaced (0.466%); 3.7 MiB/s, 58 objects/s recovering 2024-09-18T08:10:06.352 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:10:06 smithi050 ceph-mon[32371]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-18T08:10:06.352 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:10:06 smithi050 ceph-mon[32371]: from='client.? 172.21.15.7:0/2742465668' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-18T08:10:06.353 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:10:06 smithi050 ceph-mon[32371]: osdmap e217: 8 total, 8 up, 8 in 2024-09-18T08:10:06.353 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:10:06 smithi050 ceph-mon[32371]: from='client.? 172.21.15.7:0/2824404460' entity='client.admin' cmd={"prefix": "status", "format": "json"} : dispatch 2024-09-18T08:10:06.375 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:10:06.376 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:10:06.376 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7' 2024-09-18T08:10:06.477 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0 2024-09-18T08:10:06.480 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:10:06 smithi007 ceph-mon[29459]: pgmap v350: 158 pgs: 1 active+recovering+undersized+remapped, 157 active+clean; 3.8 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.18k op/s; 868/186264 objects misplaced (0.466%); 3.7 MiB/s, 58 objects/s recovering 2024-09-18T08:10:06.480 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:10:06 smithi007 ceph-mon[29459]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-18T08:10:06.480 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:10:06 smithi007 ceph-mon[29459]: from='client.? 172.21.15.7:0/2742465668' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-18T08:10:06.481 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:10:06 smithi007 ceph-mon[29459]: osdmap e217: 8 total, 8 up, 8 in 2024-09-18T08:10:06.481 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:10:06 smithi007 ceph-mon[29459]: from='client.? 172.21.15.7:0/2824404460' entity='client.admin' cmd={"prefix": "status", "format": "json"} : dispatch 2024-09-18T08:10:06.481 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:10:06 smithi007 ceph-mon[39249]: pgmap v350: 158 pgs: 1 active+recovering+undersized+remapped, 157 active+clean; 3.8 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.18k op/s; 868/186264 objects misplaced (0.466%); 3.7 MiB/s, 58 objects/s recovering 2024-09-18T08:10:06.482 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:10:06 smithi007 ceph-mon[39249]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-18T08:10:06.482 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:10:06 smithi007 ceph-mon[39249]: from='client.? 172.21.15.7:0/2742465668' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-18T08:10:06.482 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:10:06 smithi007 ceph-mon[39249]: osdmap e217: 8 total, 8 up, 8 in 2024-09-18T08:10:06.482 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:10:06 smithi007 ceph-mon[39249]: from='client.? 172.21.15.7:0/2824404460' entity='client.admin' cmd={"prefix": "status", "format": "json"} : dispatch 2024-09-18T08:10:06.491 INFO:teuthology.orchestra.run.smithi050.stdout:ERROR: (22) Invalid argument 2024-09-18T08:10:06.491 INFO:teuthology.orchestra.run.smithi050.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-18T08:10:06.509 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:10:06.510 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:10:06.510 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0' 2024-09-18T08:10:06.611 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2 2024-09-18T08:10:06.648 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:10:06.649 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:10:06.650 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2' 2024-09-18T08:10:06.750 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3 2024-09-18T08:10:06.788 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:10:06.788 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:10:06.789 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3' 2024-09-18T08:10:06.890 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6 2024-09-18T08:10:06.924 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:10:06.925 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:10:06.925 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6' 2024-09-18T08:10:06.981 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Sep 18 08:10:06 smithi007 ceph-779349ca-7593-11ef-bceb-c7b262605968-mgr-y[29688]: ::ffff:172.21.15.50 - - [18/Sep/2024:08:10:06] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-09-18T08:10:07.026 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1 2024-09-18T08:10:07.047 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid 779349ca-7593-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-18T08:10:07.126 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:10:07.126 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:10:07.127 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1' 2024-09-18T08:10:07.228 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6 2024-09-18T08:10:07.259 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:10:07.260 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:10:07.261 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6' 2024-09-18T08:10:07.348 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:10:07 smithi050 ceph-mon[32371]: 4.12 scrub starts 2024-09-18T08:10:07.348 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:10:07 smithi050 ceph-mon[32371]: 4.12 scrub ok 2024-09-18T08:10:07.349 DEBUG:teuthology.orchestra.run.smithi050:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid 779349ca-7593-11ef-bceb-c7b262605968 -- ceph daemon osd.7 dump_historic_ops 2024-09-18T08:10:07.361 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7 2024-09-18T08:10:07.434 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:10:07.435 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:10:07.435 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7' 2024-09-18T08:10:07.480 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:10:07 smithi007 ceph-mon[29459]: 4.12 scrub starts 2024-09-18T08:10:07.480 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:10:07 smithi007 ceph-mon[29459]: 4.12 scrub ok 2024-09-18T08:10:07.481 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:10:07 smithi007 ceph-mon[39249]: 4.12 scrub starts 2024-09-18T08:10:07.481 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:10:07 smithi007 ceph-mon[39249]: 4.12 scrub ok 2024-09-18T08:10:07.536 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5 2024-09-18T08:10:07.574 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:10:07.575 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:10:07.575 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5' 2024-09-18T08:10:07.676 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1 2024-09-18T08:10:07.713 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:10:07.714 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:10:07.714 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1' 2024-09-18T08:10:07.720 INFO:teuthology.orchestra.run.smithi050.stderr:Inferring config /var/lib/ceph/779349ca-7593-11ef-bceb-c7b262605968/mon.b/config 2024-09-18T08:10:07.814 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2 2024-09-18T08:10:07.855 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:10:07.855 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:10:07.856 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2' 2024-09-18T08:10:07.957 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6 2024-09-18T08:10:07.994 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:10:07.994 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:10:07.995 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6' 2024-09-18T08:10:08.096 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2 2024-09-18T08:10:08.134 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:10:08.135 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:10:08.135 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2' 2024-09-18T08:10:08.236 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2 2024-09-18T08:10:08.278 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:10:08.278 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:10:08.279 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2' 2024-09-18T08:10:08.301 INFO:teuthology.orchestra.run.smithi050.stdout:ERROR: (22) Invalid argument 2024-09-18T08:10:08.301 INFO:teuthology.orchestra.run.smithi050.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-18T08:10:08.380 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2 2024-09-18T08:10:08.418 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:10:08.419 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:10:08.419 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2' 2024-09-18T08:10:08.520 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0 2024-09-18T08:10:08.555 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:10:08.555 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:10:08.555 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0' 2024-09-18T08:10:08.652 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:10:08 smithi050 ceph-mon[32371]: pgmap v352: 158 pgs: 1 active+recovering+undersized+remapped, 157 active+clean; 3.8 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.12k op/s; 868/186264 objects misplaced (0.466%); 2.3 MiB/s, 36 objects/s recovering 2024-09-18T08:10:08.652 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:10:08 smithi050 ceph-mon[32371]: 3.4 scrub starts 2024-09-18T08:10:08.652 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:10:08 smithi050 ceph-mon[32371]: 3.4 scrub ok 2024-09-18T08:10:08.656 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1 2024-09-18T08:10:08.678 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:10:08.679 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:10:08.679 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1' 2024-09-18T08:10:08.730 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:10:08 smithi007 ceph-mon[29459]: pgmap v352: 158 pgs: 1 active+recovering+undersized+remapped, 157 active+clean; 3.8 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.12k op/s; 868/186264 objects misplaced (0.466%); 2.3 MiB/s, 36 objects/s recovering 2024-09-18T08:10:08.731 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:10:08 smithi007 ceph-mon[29459]: 3.4 scrub starts 2024-09-18T08:10:08.731 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:10:08 smithi007 ceph-mon[29459]: 3.4 scrub ok 2024-09-18T08:10:08.731 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:10:08 smithi007 ceph-mon[39249]: pgmap v352: 158 pgs: 1 active+recovering+undersized+remapped, 157 active+clean; 3.8 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.12k op/s; 868/186264 objects misplaced (0.466%); 2.3 MiB/s, 36 objects/s recovering 2024-09-18T08:10:08.732 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:10:08 smithi007 ceph-mon[39249]: 3.4 scrub starts 2024-09-18T08:10:08.732 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:10:08 smithi007 ceph-mon[39249]: 3.4 scrub ok 2024-09-18T08:10:08.780 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2 2024-09-18T08:10:08.819 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:10:08.820 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:10:08.820 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2' 2024-09-18T08:10:08.921 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0 2024-09-18T08:10:08.960 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:10:08.961 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:10:08.961 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0' 2024-09-18T08:10:09.018 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid 779349ca-7593-11ef-bceb-c7b262605968 -- ceph daemon osd.0 dump_ops_in_flight 2024-09-18T08:10:09.062 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0 2024-09-18T08:10:09.105 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:10:09.106 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:10:09.107 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0' 2024-09-18T08:10:09.207 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2 2024-09-18T08:10:09.248 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:10:09.248 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:10:09.249 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2' 2024-09-18T08:10:09.266 INFO:journalctl@ceph.iscsi.iscsi.a.smithi050.stdout:Sep 18 08:10:09 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-iscsi-iscsi-a[57782]: debug there is no tcmu-runner data available 2024-09-18T08:10:09.350 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0 2024-09-18T08:10:09.393 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:10:09.393 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:10:09.394 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0' 2024-09-18T08:10:09.480 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:10:09 smithi007 ceph-mon[29459]: 3.1e scrub starts 2024-09-18T08:10:09.480 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:10:09 smithi007 ceph-mon[29459]: 3.1e scrub ok 2024-09-18T08:10:09.481 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:10:09 smithi007 ceph-mon[39249]: 3.1e scrub starts 2024-09-18T08:10:09.481 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:10:09 smithi007 ceph-mon[39249]: 3.1e scrub ok 2024-09-18T08:10:09.494 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4 2024-09-18T08:10:09.525 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:10:09.526 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:10:09.527 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4' 2024-09-18T08:10:09.627 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7 2024-09-18T08:10:09.656 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:10:09 smithi050 ceph-mon[32371]: 3.1e scrub starts 2024-09-18T08:10:09.656 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:10:09 smithi050 ceph-mon[32371]: 3.1e scrub ok 2024-09-18T08:10:09.657 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:10:09.657 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:10:09.658 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7' 2024-09-18T08:10:09.759 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3 2024-09-18T08:10:09.798 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:10:09.798 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:10:09.798 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3' 2024-09-18T08:10:09.899 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0 2024-09-18T08:10:09.939 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:10:09.940 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:10:09.940 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0' 2024-09-18T08:10:10.041 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0 2024-09-18T08:10:10.063 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:10:10.064 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:10:10.064 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0' 2024-09-18T08:10:10.165 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6 2024-09-18T08:10:10.196 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:10:10.197 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:10:10.197 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6' 2024-09-18T08:10:10.298 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3 2024-09-18T08:10:10.333 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:10:10.334 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:10:10.334 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3' 2024-09-18T08:10:10.435 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7 2024-09-18T08:10:10.469 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:10:10.469 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:10:10.470 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7' 2024-09-18T08:10:10.571 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1 2024-09-18T08:10:10.608 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:10:10.609 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:10:10.609 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1' 2024-09-18T08:10:10.656 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:10:10 smithi050 ceph-mon[32371]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-18T08:10:10.656 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:10:10 smithi050 ceph-mon[32371]: pgmap v353: 158 pgs: 1 active+recovering+undersized+remapped, 157 active+clean; 3.4 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.15k op/s; 486/169542 objects misplaced (0.287%); 4.3 MiB/s, 69 objects/s recovering 2024-09-18T08:10:10.656 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:10:10 smithi050 ceph-mon[32371]: 3.7 scrub starts 2024-09-18T08:10:10.656 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:10:10 smithi050 ceph-mon[32371]: 3.7 scrub ok 2024-09-18T08:10:10.656 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:10:10 smithi050 ceph-mon[32371]: 8.c scrub starts 2024-09-18T08:10:10.710 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6 2024-09-18T08:10:10.730 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:10:10 smithi007 ceph-mon[29459]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-18T08:10:10.730 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:10:10 smithi007 ceph-mon[29459]: pgmap v353: 158 pgs: 1 active+recovering+undersized+remapped, 157 active+clean; 3.4 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.15k op/s; 486/169542 objects misplaced (0.287%); 4.3 MiB/s, 69 objects/s recovering 2024-09-18T08:10:10.730 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:10:10 smithi007 ceph-mon[29459]: 3.7 scrub starts 2024-09-18T08:10:10.730 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:10:10 smithi007 ceph-mon[29459]: 3.7 scrub ok 2024-09-18T08:10:10.730 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:10:10 smithi007 ceph-mon[29459]: 8.c scrub starts 2024-09-18T08:10:10.731 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:10:10 smithi007 ceph-mon[39249]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-18T08:10:10.731 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:10:10 smithi007 ceph-mon[39249]: pgmap v353: 158 pgs: 1 active+recovering+undersized+remapped, 157 active+clean; 3.4 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.15k op/s; 486/169542 objects misplaced (0.287%); 4.3 MiB/s, 69 objects/s recovering 2024-09-18T08:10:10.731 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:10:10 smithi007 ceph-mon[39249]: 3.7 scrub starts 2024-09-18T08:10:10.731 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:10:10 smithi007 ceph-mon[39249]: 3.7 scrub ok 2024-09-18T08:10:10.731 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:10:10 smithi007 ceph-mon[39249]: 8.c scrub starts 2024-09-18T08:10:10.734 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:10:10.734 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:10:10.735 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6' 2024-09-18T08:10:10.835 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2 2024-09-18T08:10:10.877 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:10:10.878 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:10:10.879 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2' 2024-09-18T08:10:10.979 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6 2024-09-18T08:10:11.004 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:10:11.004 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:10:11.005 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6' 2024-09-18T08:10:11.106 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2 2024-09-18T08:10:11.144 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:10:11.145 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:10:11.145 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2' 2024-09-18T08:10:11.246 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3 2024-09-18T08:10:11.282 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:10:11.283 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:10:11.283 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3' 2024-09-18T08:10:11.384 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0 2024-09-18T08:10:11.420 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:10:11.421 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:10:11.421 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0' 2024-09-18T08:10:11.522 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2 2024-09-18T08:10:11.559 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:10:11.560 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:10:11.560 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2' 2024-09-18T08:10:11.572 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid 779349ca-7593-11ef-bceb-c7b262605968 -- ceph osd unset nodeep-scrub 2024-09-18T08:10:11.636 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:10:11 smithi007 ceph-mon[29459]: 5.14 scrub starts 2024-09-18T08:10:11.636 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:10:11 smithi007 ceph-mon[29459]: 5.14 scrub ok 2024-09-18T08:10:11.636 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:10:11 smithi007 ceph-mon[29459]: 8.6 scrub starts 2024-09-18T08:10:11.636 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:10:11 smithi007 ceph-mon[39249]: 5.14 scrub starts 2024-09-18T08:10:11.636 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:10:11 smithi007 ceph-mon[39249]: 5.14 scrub ok 2024-09-18T08:10:11.637 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:10:11 smithi007 ceph-mon[39249]: 8.6 scrub starts 2024-09-18T08:10:11.656 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:10:11 smithi050 ceph-mon[32371]: 5.14 scrub starts 2024-09-18T08:10:11.656 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:10:11 smithi050 ceph-mon[32371]: 5.14 scrub ok 2024-09-18T08:10:11.656 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:10:11 smithi050 ceph-mon[32371]: 8.6 scrub starts 2024-09-18T08:10:11.661 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2 2024-09-18T08:10:11.710 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:10:11.710 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:10:11.711 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2' 2024-09-18T08:10:11.812 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1 2024-09-18T08:10:11.851 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:10:11.852 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:10:11.852 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1' 2024-09-18T08:10:11.953 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4 2024-09-18T08:10:11.969 INFO:tasks.cephadm.ceph_manager.ceph:making progress, resetting timeout 2024-09-18T08:10:11.970 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid 779349ca-7593-11ef-bceb-c7b262605968 -- ceph pg dump --format=json 2024-09-18T08:10:11.984 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:10:11.984 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:10:11.985 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4' 2024-09-18T08:10:12.086 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2 2024-09-18T08:10:12.091 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/779349ca-7593-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T08:10:12.129 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/779349ca-7593-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T08:10:12.135 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/779349ca-7593-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T08:10:12.160 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:10:12.161 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:10:12.162 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2' 2024-09-18T08:10:12.262 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4 2024-09-18T08:10:12.291 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:10:12.292 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:10:12.292 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4' 2024-09-18T08:10:12.393 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3 2024-09-18T08:10:12.443 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:10:12.445 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:10:12.445 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3' 2024-09-18T08:10:12.505 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:10:12 smithi007 ceph-mon[29459]: 8.c scrub ok 2024-09-18T08:10:12.505 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:10:12 smithi007 ceph-mon[29459]: pgmap v354: 158 pgs: 1 active+recovering+undersized+remapped, 157 active+clean; 3.4 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.15k op/s; 486/169542 objects misplaced (0.287%); 4.3 MiB/s, 69 objects/s recovering 2024-09-18T08:10:12.505 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:10:12 smithi007 ceph-mon[29459]: purged_snaps scrub starts 2024-09-18T08:10:12.505 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:10:12 smithi007 ceph-mon[29459]: purged_snaps scrub ok 2024-09-18T08:10:12.505 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:10:12 smithi007 ceph-mon[29459]: purged_snaps scrub starts 2024-09-18T08:10:12.505 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:10:12 smithi007 ceph-mon[29459]: purged_snaps scrub ok 2024-09-18T08:10:12.506 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:10:12 smithi007 ceph-mon[39249]: 8.c scrub ok 2024-09-18T08:10:12.506 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:10:12 smithi007 ceph-mon[39249]: pgmap v354: 158 pgs: 1 active+recovering+undersized+remapped, 157 active+clean; 3.4 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.15k op/s; 486/169542 objects misplaced (0.287%); 4.3 MiB/s, 69 objects/s recovering 2024-09-18T08:10:12.506 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:10:12 smithi007 ceph-mon[39249]: purged_snaps scrub starts 2024-09-18T08:10:12.506 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:10:12 smithi007 ceph-mon[39249]: purged_snaps scrub ok 2024-09-18T08:10:12.506 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:10:12 smithi007 ceph-mon[39249]: purged_snaps scrub starts 2024-09-18T08:10:12.506 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:10:12 smithi007 ceph-mon[39249]: purged_snaps scrub ok 2024-09-18T08:10:12.546 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5 2024-09-18T08:10:12.580 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:10:12.581 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:10:12.581 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5' 2024-09-18T08:10:12.593 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:10:12 smithi050 ceph-mon[32371]: 8.c scrub ok 2024-09-18T08:10:12.594 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:10:12 smithi050 ceph-mon[32371]: pgmap v354: 158 pgs: 1 active+recovering+undersized+remapped, 157 active+clean; 3.4 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.15k op/s; 486/169542 objects misplaced (0.287%); 4.3 MiB/s, 69 objects/s recovering 2024-09-18T08:10:12.594 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:10:12 smithi050 ceph-mon[32371]: purged_snaps scrub starts 2024-09-18T08:10:12.594 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:10:12 smithi050 ceph-mon[32371]: purged_snaps scrub ok 2024-09-18T08:10:12.594 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:10:12 smithi050 ceph-mon[32371]: purged_snaps scrub starts 2024-09-18T08:10:12.594 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:10:12 smithi050 ceph-mon[32371]: purged_snaps scrub ok 2024-09-18T08:10:12.682 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7 2024-09-18T08:10:12.721 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:10:12.721 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:10:12.722 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7' 2024-09-18T08:10:12.823 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7 2024-09-18T08:10:12.856 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:10:12.857 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:10:12.857 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7' 2024-09-18T08:10:12.958 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4 2024-09-18T08:10:12.988 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:10:12.989 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:10:12.989 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4' 2024-09-18T08:10:13.071 INFO:teuthology.orchestra.run.smithi007.stdout:osd.0: {} 2024-09-18T08:10:13.071 INFO:teuthology.orchestra.run.smithi007.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-18T08:10:13.090 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1 2024-09-18T08:10:13.113 INFO:teuthology.orchestra.run.smithi007.stdout:osd.1: {} 2024-09-18T08:10:13.113 INFO:teuthology.orchestra.run.smithi007.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-18T08:10:13.134 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:10:13.135 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:10:13.135 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1' 2024-09-18T08:10:13.166 INFO:teuthology.orchestra.run.smithi007.stdout:osd.2: {} 2024-09-18T08:10:13.166 INFO:teuthology.orchestra.run.smithi007.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-18T08:10:13.206 INFO:teuthology.orchestra.run.smithi007.stdout:osd.3: {} 2024-09-18T08:10:13.206 INFO:teuthology.orchestra.run.smithi007.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-18T08:10:13.236 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0 2024-09-18T08:10:13.246 INFO:teuthology.orchestra.run.smithi007.stdout:osd.4: {} 2024-09-18T08:10:13.246 INFO:teuthology.orchestra.run.smithi007.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-18T08:10:13.282 INFO:teuthology.orchestra.run.smithi007.stdout:osd.5: {} 2024-09-18T08:10:13.283 INFO:teuthology.orchestra.run.smithi007.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-18T08:10:13.299 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:10:13.299 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:10:13.300 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0' 2024-09-18T08:10:13.326 INFO:teuthology.orchestra.run.smithi007.stdout:osd.6: {} 2024-09-18T08:10:13.326 INFO:teuthology.orchestra.run.smithi007.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-18T08:10:13.326 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:10:13 smithi007 ceph-mon[39249]: 8.6 scrub ok 2024-09-18T08:10:13.326 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:10:13 smithi007 ceph-mon[39249]: osdmap e218: 8 total, 8 up, 8 in 2024-09-18T08:10:13.327 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:10:13 smithi007 ceph-mon[39249]: 4.1d scrub starts 2024-09-18T08:10:13.327 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:10:13 smithi007 ceph-mon[39249]: 4.1d scrub ok 2024-09-18T08:10:13.327 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:10:13 smithi007 ceph-mon[39249]: from='client.? 172.21.15.7:0/1702182037' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-18T08:10:13.327 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:10:13 smithi007 ceph-mon[29459]: 8.6 scrub ok 2024-09-18T08:10:13.327 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:10:13 smithi007 ceph-mon[29459]: osdmap e218: 8 total, 8 up, 8 in 2024-09-18T08:10:13.327 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:10:13 smithi007 ceph-mon[29459]: 4.1d scrub starts 2024-09-18T08:10:13.327 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:10:13 smithi007 ceph-mon[29459]: 4.1d scrub ok 2024-09-18T08:10:13.327 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:10:13 smithi007 ceph-mon[29459]: from='client.? 172.21.15.7:0/1702182037' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-18T08:10:13.370 INFO:teuthology.orchestra.run.smithi007.stdout:osd.7: {} 2024-09-18T08:10:13.371 INFO:teuthology.orchestra.run.smithi007.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-18T08:10:13.400 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7 2024-09-18T08:10:13.433 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:10:13.433 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:10:13.434 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7' 2024-09-18T08:10:13.535 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5 2024-09-18T08:10:13.573 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:10:13.574 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:10:13.574 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5' 2024-09-18T08:10:13.656 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:10:13 smithi050 ceph-mon[32371]: 8.6 scrub ok 2024-09-18T08:10:13.656 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:10:13 smithi050 ceph-mon[32371]: osdmap e218: 8 total, 8 up, 8 in 2024-09-18T08:10:13.656 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:10:13 smithi050 ceph-mon[32371]: 4.1d scrub starts 2024-09-18T08:10:13.656 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:10:13 smithi050 ceph-mon[32371]: 4.1d scrub ok 2024-09-18T08:10:13.656 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:10:13 smithi050 ceph-mon[32371]: from='client.? 172.21.15.7:0/1702182037' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-18T08:10:13.676 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1 2024-09-18T08:10:13.721 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:10:13.722 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:10:13.722 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1' 2024-09-18T08:10:13.823 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2 2024-09-18T08:10:13.862 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:10:13.862 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:10:13.863 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2' 2024-09-18T08:10:13.964 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0 2024-09-18T08:10:13.999 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:10:13.999 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:10:14.000 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0' 2024-09-18T08:10:14.100 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4 2024-09-18T08:10:14.134 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:10:14.134 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:10:14.135 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4' 2024-09-18T08:10:14.236 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3 2024-09-18T08:10:14.246 INFO:teuthology.orchestra.run.smithi007.stderr:nodeep-scrub is unset 2024-09-18T08:10:14.274 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:10:14.275 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:10:14.275 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3' 2024-09-18T08:10:14.376 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5 2024-09-18T08:10:14.404 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:10:14.404 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:10:14.405 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5' 2024-09-18T08:10:14.505 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5 2024-09-18T08:10:14.528 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:10:14.528 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:10:14.529 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5' 2024-09-18T08:10:14.629 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2 2024-09-18T08:10:14.652 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:10:14.653 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:10:14.653 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2' 2024-09-18T08:10:14.655 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:10:14 smithi050 ceph-mon[32371]: pgmap v356: 158 pgs: 1 active+recovering+undersized+remapped, 157 active+clean; 3.4 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 773 op/s; 486/167706 objects misplaced (0.290%); 3.0 MiB/s, 47 objects/s recovering 2024-09-18T08:10:14.655 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:10:14 smithi050 ceph-mon[32371]: from='client.? 172.21.15.7:0/223196652' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-18T08:10:14.656 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:10:14 smithi050 ceph-mon[32371]: 3.17 scrub starts 2024-09-18T08:10:14.656 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:10:14 smithi050 ceph-mon[32371]: 3.17 scrub ok 2024-09-18T08:10:14.656 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:10:14 smithi050 ceph-mon[32371]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-18T08:10:14.656 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:10:14 smithi050 ceph-mon[32371]: from='client.? 172.21.15.7:0/223196652' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-18T08:10:14.656 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:10:14 smithi050 ceph-mon[32371]: osdmap e219: 8 total, 8 up, 8 in 2024-09-18T08:10:14.730 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:10:14 smithi007 ceph-mon[29459]: pgmap v356: 158 pgs: 1 active+recovering+undersized+remapped, 157 active+clean; 3.4 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 773 op/s; 486/167706 objects misplaced (0.290%); 3.0 MiB/s, 47 objects/s recovering 2024-09-18T08:10:14.731 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:10:14 smithi007 ceph-mon[29459]: from='client.? 172.21.15.7:0/223196652' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-18T08:10:14.731 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:10:14 smithi007 ceph-mon[29459]: 3.17 scrub starts 2024-09-18T08:10:14.731 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:10:14 smithi007 ceph-mon[29459]: 3.17 scrub ok 2024-09-18T08:10:14.731 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:10:14 smithi007 ceph-mon[29459]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-18T08:10:14.731 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:10:14 smithi007 ceph-mon[29459]: from='client.? 172.21.15.7:0/223196652' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-18T08:10:14.731 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:10:14 smithi007 ceph-mon[29459]: osdmap e219: 8 total, 8 up, 8 in 2024-09-18T08:10:14.731 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:10:14 smithi007 ceph-mon[39249]: pgmap v356: 158 pgs: 1 active+recovering+undersized+remapped, 157 active+clean; 3.4 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 773 op/s; 486/167706 objects misplaced (0.290%); 3.0 MiB/s, 47 objects/s recovering 2024-09-18T08:10:14.731 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:10:14 smithi007 ceph-mon[39249]: from='client.? 172.21.15.7:0/223196652' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-18T08:10:14.731 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:10:14 smithi007 ceph-mon[39249]: 3.17 scrub starts 2024-09-18T08:10:14.731 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:10:14 smithi007 ceph-mon[39249]: 3.17 scrub ok 2024-09-18T08:10:14.732 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:10:14 smithi007 ceph-mon[39249]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-18T08:10:14.732 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:10:14 smithi007 ceph-mon[39249]: from='client.? 172.21.15.7:0/223196652' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-18T08:10:14.732 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:10:14 smithi007 ceph-mon[39249]: osdmap e219: 8 total, 8 up, 8 in 2024-09-18T08:10:14.754 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2 2024-09-18T08:10:14.776 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:10:14.777 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:10:14.777 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2' 2024-09-18T08:10:14.836 INFO:teuthology.orchestra.run.smithi007.stdout:{ 2024-09-18T08:10:14.836 INFO:teuthology.orchestra.run.smithi007.stdout: "ops": [], 2024-09-18T08:10:14.837 INFO:teuthology.orchestra.run.smithi007.stdout: "num_ops": 0 2024-09-18T08:10:14.837 INFO:teuthology.orchestra.run.smithi007.stdout:} 2024-09-18T08:10:14.878 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3 2024-09-18T08:10:14.899 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:10:14.900 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:10:14.900 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3' 2024-09-18T08:10:15.001 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4 2024-09-18T08:10:15.025 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:10:15.025 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:10:15.026 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4' 2024-09-18T08:10:15.127 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2 2024-09-18T08:10:15.150 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:10:15.150 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:10:15.150 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2' 2024-09-18T08:10:15.251 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0 2024-09-18T08:10:15.277 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:10:15.278 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:10:15.279 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0' 2024-09-18T08:10:15.379 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0 2024-09-18T08:10:15.421 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:10:15.422 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:10:15.422 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0' 2024-09-18T08:10:15.480 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:10:15 smithi007 ceph-mon[29459]: 3.9 scrub starts 2024-09-18T08:10:15.480 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:10:15 smithi007 ceph-mon[29459]: 3.9 scrub ok 2024-09-18T08:10:15.481 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:10:15 smithi007 ceph-mon[29459]: osdmap e220: 8 total, 8 up, 8 in 2024-09-18T08:10:15.481 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:10:15 smithi007 ceph-mon[39249]: 3.9 scrub starts 2024-09-18T08:10:15.481 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:10:15 smithi007 ceph-mon[39249]: 3.9 scrub ok 2024-09-18T08:10:15.481 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:10:15 smithi007 ceph-mon[39249]: osdmap e220: 8 total, 8 up, 8 in 2024-09-18T08:10:15.524 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2 2024-09-18T08:10:15.559 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:10:15.560 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:10:15.560 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2' 2024-09-18T08:10:15.616 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/779349ca-7593-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T08:10:15.655 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:10:15 smithi050 ceph-mon[32371]: 3.9 scrub starts 2024-09-18T08:10:15.656 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:10:15 smithi050 ceph-mon[32371]: 3.9 scrub ok 2024-09-18T08:10:15.656 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:10:15 smithi050 ceph-mon[32371]: osdmap e220: 8 total, 8 up, 8 in 2024-09-18T08:10:15.661 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7 2024-09-18T08:10:15.696 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:10:15.697 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:10:15.697 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7' 2024-09-18T08:10:15.798 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1 2024-09-18T08:10:15.836 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:10:15.836 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:10:15.836 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1' 2024-09-18T08:10:15.937 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4 2024-09-18T08:10:15.968 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:10:15.968 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:10:15.969 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4' 2024-09-18T08:10:16.070 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2 2024-09-18T08:10:16.082 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid 779349ca-7593-11ef-bceb-c7b262605968 -- ceph daemon osd.0 dump_blocked_ops 2024-09-18T08:10:16.106 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:10:16.107 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:10:16.107 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2' 2024-09-18T08:10:16.207 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7 2024-09-18T08:10:16.238 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:10:16.238 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:10:16.239 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7' 2024-09-18T08:10:16.341 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1 2024-09-18T08:10:16.397 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:10:16.397 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:10:16.398 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1' 2024-09-18T08:10:16.404 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid 779349ca-7593-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-18T08:10:16.498 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2 2024-09-18T08:10:16.546 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:10:16.546 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:10:16.546 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2' 2024-09-18T08:10:16.610 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:10:16 smithi007 ceph-mon[29459]: pgmap v358: 158 pgs: 1 active+recovering+undersized+remapped, 157 active+clean; 3.1 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.34k op/s; 51/154086 objects misplaced (0.033%); 6.4 MiB/s, 102 objects/s recovering 2024-09-18T08:10:16.610 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:10:16 smithi007 ceph-mon[29459]: 4.17 scrub starts 2024-09-18T08:10:16.610 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:10:16 smithi007 ceph-mon[29459]: 4.17 scrub ok 2024-09-18T08:10:16.610 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:10:16 smithi007 ceph-mon[29459]: 5.19 scrub starts 2024-09-18T08:10:16.610 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:10:16 smithi007 ceph-mon[29459]: 5.19 scrub ok 2024-09-18T08:10:16.610 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:10:16 smithi007 ceph-mon[29459]: 3.1f scrub starts 2024-09-18T08:10:16.610 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:10:16 smithi007 ceph-mon[29459]: 3.1f scrub ok 2024-09-18T08:10:16.610 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:10:16 smithi007 ceph-mon[39249]: pgmap v358: 158 pgs: 1 active+recovering+undersized+remapped, 157 active+clean; 3.1 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.34k op/s; 51/154086 objects misplaced (0.033%); 6.4 MiB/s, 102 objects/s recovering 2024-09-18T08:10:16.611 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:10:16 smithi007 ceph-mon[39249]: 4.17 scrub starts 2024-09-18T08:10:16.611 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:10:16 smithi007 ceph-mon[39249]: 4.17 scrub ok 2024-09-18T08:10:16.611 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:10:16 smithi007 ceph-mon[39249]: 5.19 scrub starts 2024-09-18T08:10:16.611 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:10:16 smithi007 ceph-mon[39249]: 5.19 scrub ok 2024-09-18T08:10:16.611 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:10:16 smithi007 ceph-mon[39249]: 3.1f scrub starts 2024-09-18T08:10:16.611 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:10:16 smithi007 ceph-mon[39249]: 3.1f scrub ok 2024-09-18T08:10:16.648 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2 2024-09-18T08:10:16.655 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:10:16 smithi050 ceph-mon[32371]: pgmap v358: 158 pgs: 1 active+recovering+undersized+remapped, 157 active+clean; 3.1 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.34k op/s; 51/154086 objects misplaced (0.033%); 6.4 MiB/s, 102 objects/s recovering 2024-09-18T08:10:16.656 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:10:16 smithi050 ceph-mon[32371]: 4.17 scrub starts 2024-09-18T08:10:16.656 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:10:16 smithi050 ceph-mon[32371]: 4.17 scrub ok 2024-09-18T08:10:16.656 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:10:16 smithi050 ceph-mon[32371]: 5.19 scrub starts 2024-09-18T08:10:16.656 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:10:16 smithi050 ceph-mon[32371]: 5.19 scrub ok 2024-09-18T08:10:16.656 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:10:16 smithi050 ceph-mon[32371]: 3.1f scrub starts 2024-09-18T08:10:16.656 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:10:16 smithi050 ceph-mon[32371]: 3.1f scrub ok 2024-09-18T08:10:16.713 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:10:16.714 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:10:16.714 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2' 2024-09-18T08:10:16.815 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7 2024-09-18T08:10:16.850 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:10:16.851 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:10:16.851 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7' 2024-09-18T08:10:16.922 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/779349ca-7593-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T08:10:16.928 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/779349ca-7593-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T08:10:16.952 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0 2024-09-18T08:10:16.981 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Sep 18 08:10:16 smithi007 ceph-779349ca-7593-11ef-bceb-c7b262605968-mgr-y[29688]: ::ffff:172.21.15.50 - - [18/Sep/2024:08:10:16] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-09-18T08:10:17.006 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:10:17.006 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:10:17.007 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0' 2024-09-18T08:10:17.108 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6 2024-09-18T08:10:17.137 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:10:17.138 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:10:17.138 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6' 2024-09-18T08:10:17.221 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-09-18T08:10:17.223 INFO:teuthology.orchestra.run.smithi007.stderr:dumped all 2024-09-18T08:10:17.239 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3 2024-09-18T08:10:17.287 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:10:17.288 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:10:17.289 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3' 2024-09-18T08:10:17.389 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4 2024-09-18T08:10:17.421 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:10:17.422 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:10:17.422 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4' 2024-09-18T08:10:17.523 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0 2024-09-18T08:10:17.565 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:10:17.566 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:10:17.566 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0' 2024-09-18T08:10:17.596 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:10:17 smithi007 ceph-mon[29459]: 6.3 scrub starts 2024-09-18T08:10:17.596 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:10:17 smithi007 ceph-mon[29459]: 6.3 scrub ok 2024-09-18T08:10:17.596 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:10:17 smithi007 ceph-mon[29459]: 3.5 scrub starts 2024-09-18T08:10:17.596 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:10:17 smithi007 ceph-mon[29459]: 3.5 scrub ok 2024-09-18T08:10:17.597 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:10:17 smithi007 ceph-mon[29459]: 4.1a scrub starts 2024-09-18T08:10:17.597 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:10:17 smithi007 ceph-mon[29459]: 4.1a scrub ok 2024-09-18T08:10:17.597 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:10:17 smithi007 ceph-mon[29459]: 4.11 scrub starts 2024-09-18T08:10:17.597 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:10:17 smithi007 ceph-mon[29459]: 4.11 scrub ok 2024-09-18T08:10:17.597 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:10:17 smithi007 ceph-mon[29459]: from='mgr.24470 172.21.15.7:0/2594390261' entity='mgr.y' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.9", "id": [1, 5]} : dispatch 2024-09-18T08:10:17.597 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:10:17 smithi007 ceph-mon[39249]: 6.3 scrub starts 2024-09-18T08:10:17.597 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:10:17 smithi007 ceph-mon[39249]: 6.3 scrub ok 2024-09-18T08:10:17.597 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:10:17 smithi007 ceph-mon[39249]: 3.5 scrub starts 2024-09-18T08:10:17.597 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:10:17 smithi007 ceph-mon[39249]: 3.5 scrub ok 2024-09-18T08:10:17.597 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:10:17 smithi007 ceph-mon[39249]: 4.1a scrub starts 2024-09-18T08:10:17.597 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:10:17 smithi007 ceph-mon[39249]: 4.1a scrub ok 2024-09-18T08:10:17.598 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:10:17 smithi007 ceph-mon[39249]: 4.11 scrub starts 2024-09-18T08:10:17.598 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:10:17 smithi007 ceph-mon[39249]: 4.11 scrub ok 2024-09-18T08:10:17.598 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:10:17 smithi007 ceph-mon[39249]: from='mgr.24470 172.21.15.7:0/2594390261' entity='mgr.y' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.9", "id": [1, 5]} : dispatch 2024-09-18T08:10:17.618 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid 779349ca-7593-11ef-bceb-c7b262605968 -- ceph osd set noscrub 2024-09-18T08:10:17.656 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:10:17 smithi050 ceph-mon[32371]: 6.3 scrub starts 2024-09-18T08:10:17.656 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:10:17 smithi050 ceph-mon[32371]: 6.3 scrub ok 2024-09-18T08:10:17.656 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:10:17 smithi050 ceph-mon[32371]: 3.5 scrub starts 2024-09-18T08:10:17.656 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:10:17 smithi050 ceph-mon[32371]: 3.5 scrub ok 2024-09-18T08:10:17.656 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:10:17 smithi050 ceph-mon[32371]: 4.1a scrub starts 2024-09-18T08:10:17.656 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:10:17 smithi050 ceph-mon[32371]: 4.1a scrub ok 2024-09-18T08:10:17.656 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:10:17 smithi050 ceph-mon[32371]: 4.11 scrub starts 2024-09-18T08:10:17.656 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:10:17 smithi050 ceph-mon[32371]: 4.11 scrub ok 2024-09-18T08:10:17.656 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:10:17 smithi050 ceph-mon[32371]: from='mgr.24470 172.21.15.7:0/2594390261' entity='mgr.y' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.9", "id": [1, 5]} : dispatch 2024-09-18T08:10:17.661 INFO:teuthology.orchestra.run.smithi007.stdout:{ 2024-09-18T08:10:17.661 INFO:teuthology.orchestra.run.smithi007.stdout: "ops": [], 2024-09-18T08:10:17.661 INFO:teuthology.orchestra.run.smithi007.stdout: "complaint_time": 30, 2024-09-18T08:10:17.661 INFO:teuthology.orchestra.run.smithi007.stdout: "num_blocked_ops": 0 2024-09-18T08:10:17.661 INFO:teuthology.orchestra.run.smithi007.stdout:} 2024-09-18T08:10:17.667 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1 2024-09-18T08:10:17.727 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:10:17.728 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:10:17.728 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1' 2024-09-18T08:10:17.829 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2 2024-09-18T08:10:17.874 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:10:17.875 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:10:17.875 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2' 2024-09-18T08:10:17.976 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3 2024-09-18T08:10:18.020 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:10:18.020 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:10:18.020 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3' 2024-09-18T08:10:18.121 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6 2024-09-18T08:10:18.160 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:10:18.160 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:10:18.161 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6' 2024-09-18T08:10:18.261 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2 2024-09-18T08:10:18.301 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:10:18.301 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:10:18.302 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2' 2024-09-18T08:10:18.403 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2 2024-09-18T08:10:18.438 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:10:18.439 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:10:18.439 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2' 2024-09-18T08:10:18.540 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4 2024-09-18T08:10:18.576 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:10:18.576 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:10:18.577 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4' 2024-09-18T08:10:18.656 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:10:18 smithi050 ceph-mon[32371]: 5.1a scrub starts 2024-09-18T08:10:18.656 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:10:18 smithi050 ceph-mon[32371]: 5.1a scrub ok 2024-09-18T08:10:18.656 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:10:18 smithi050 ceph-mon[32371]: from='client.25207 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-18T08:10:18.656 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:10:18 smithi050 ceph-mon[32371]: 6.2 scrub starts 2024-09-18T08:10:18.656 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:10:18 smithi050 ceph-mon[32371]: 6.2 scrub ok 2024-09-18T08:10:18.656 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:10:18 smithi050 ceph-mon[32371]: pgmap v360: 158 pgs: 1 active+recovering+undersized+remapped, 157 active+clean; 3.1 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 852 B/s rd, 0 B/s wr, 859 op/s; 51/154086 objects misplaced (0.033%); 4.5 MiB/s, 72 objects/s recovering 2024-09-18T08:10:18.656 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:10:18 smithi050 ceph-mon[32371]: from='mgr.24470 172.21.15.7:0/2594390261' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.9", "id": [1, 5]}]': finished 2024-09-18T08:10:18.657 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:10:18 smithi050 ceph-mon[32371]: osdmap e221: 8 total, 8 up, 8 in 2024-09-18T08:10:18.657 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:10:18 smithi050 ceph-mon[32371]: 6.b scrub starts 2024-09-18T08:10:18.657 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:10:18 smithi050 ceph-mon[32371]: 6.b scrub ok 2024-09-18T08:10:18.657 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:10:18 smithi050 ceph-mon[32371]: 6.f scrub starts 2024-09-18T08:10:18.657 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:10:18 smithi050 ceph-mon[32371]: 6.f scrub ok 2024-09-18T08:10:18.657 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:10:18 smithi050 ceph-mon[32371]: 6.10 scrub starts 2024-09-18T08:10:18.657 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:10:18 smithi050 ceph-mon[32371]: 6.10 scrub ok 2024-09-18T08:10:18.657 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:10:18 smithi050 ceph-mon[32371]: from='mgr.24470 172.21.15.7:0/2594390261' entity='mgr.y' 2024-09-18T08:10:18.657 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:10:18 smithi050 ceph-mon[32371]: from='mgr.24470 172.21.15.7:0/2594390261' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-18T08:10:18.677 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0 2024-09-18T08:10:18.717 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:10:18.718 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:10:18.718 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0' 2024-09-18T08:10:18.731 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:10:18 smithi007 ceph-mon[39249]: 5.1a scrub starts 2024-09-18T08:10:18.731 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:10:18 smithi007 ceph-mon[39249]: 5.1a scrub ok 2024-09-18T08:10:18.731 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:10:18 smithi007 ceph-mon[39249]: from='client.25207 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-18T08:10:18.731 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:10:18 smithi007 ceph-mon[39249]: 6.2 scrub starts 2024-09-18T08:10:18.731 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:10:18 smithi007 ceph-mon[39249]: 6.2 scrub ok 2024-09-18T08:10:18.731 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:10:18 smithi007 ceph-mon[39249]: pgmap v360: 158 pgs: 1 active+recovering+undersized+remapped, 157 active+clean; 3.1 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 852 B/s rd, 0 B/s wr, 859 op/s; 51/154086 objects misplaced (0.033%); 4.5 MiB/s, 72 objects/s recovering 2024-09-18T08:10:18.731 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:10:18 smithi007 ceph-mon[39249]: from='mgr.24470 172.21.15.7:0/2594390261' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.9", "id": [1, 5]}]': finished 2024-09-18T08:10:18.731 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:10:18 smithi007 ceph-mon[39249]: osdmap e221: 8 total, 8 up, 8 in 2024-09-18T08:10:18.731 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:10:18 smithi007 ceph-mon[39249]: 6.b scrub starts 2024-09-18T08:10:18.731 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:10:18 smithi007 ceph-mon[39249]: 6.b scrub ok 2024-09-18T08:10:18.731 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:10:18 smithi007 ceph-mon[39249]: 6.f scrub starts 2024-09-18T08:10:18.731 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:10:18 smithi007 ceph-mon[39249]: 6.f scrub ok 2024-09-18T08:10:18.731 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:10:18 smithi007 ceph-mon[39249]: 6.10 scrub starts 2024-09-18T08:10:18.732 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:10:18 smithi007 ceph-mon[39249]: 6.10 scrub ok 2024-09-18T08:10:18.732 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:10:18 smithi007 ceph-mon[39249]: from='mgr.24470 172.21.15.7:0/2594390261' entity='mgr.y' 2024-09-18T08:10:18.732 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:10:18 smithi007 ceph-mon[39249]: from='mgr.24470 172.21.15.7:0/2594390261' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-18T08:10:18.732 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:10:18 smithi007 ceph-mon[29459]: 5.1a scrub starts 2024-09-18T08:10:18.732 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:10:18 smithi007 ceph-mon[29459]: 5.1a scrub ok 2024-09-18T08:10:18.732 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:10:18 smithi007 ceph-mon[29459]: from='client.25207 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-18T08:10:18.732 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:10:18 smithi007 ceph-mon[29459]: 6.2 scrub starts 2024-09-18T08:10:18.732 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:10:18 smithi007 ceph-mon[29459]: 6.2 scrub ok 2024-09-18T08:10:18.732 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:10:18 smithi007 ceph-mon[29459]: pgmap v360: 158 pgs: 1 active+recovering+undersized+remapped, 157 active+clean; 3.1 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 852 B/s rd, 0 B/s wr, 859 op/s; 51/154086 objects misplaced (0.033%); 4.5 MiB/s, 72 objects/s recovering 2024-09-18T08:10:18.732 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:10:18 smithi007 ceph-mon[29459]: from='mgr.24470 172.21.15.7:0/2594390261' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.9", "id": [1, 5]}]': finished 2024-09-18T08:10:18.732 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:10:18 smithi007 ceph-mon[29459]: osdmap e221: 8 total, 8 up, 8 in 2024-09-18T08:10:18.733 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:10:18 smithi007 ceph-mon[29459]: 6.b scrub starts 2024-09-18T08:10:18.733 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:10:18 smithi007 ceph-mon[29459]: 6.b scrub ok 2024-09-18T08:10:18.733 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:10:18 smithi007 ceph-mon[29459]: 6.f scrub starts 2024-09-18T08:10:18.733 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:10:18 smithi007 ceph-mon[29459]: 6.f scrub ok 2024-09-18T08:10:18.733 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:10:18 smithi007 ceph-mon[29459]: 6.10 scrub starts 2024-09-18T08:10:18.733 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:10:18 smithi007 ceph-mon[29459]: 6.10 scrub ok 2024-09-18T08:10:18.733 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:10:18 smithi007 ceph-mon[29459]: from='mgr.24470 172.21.15.7:0/2594390261' entity='mgr.y' 2024-09-18T08:10:18.733 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:10:18 smithi007 ceph-mon[29459]: from='mgr.24470 172.21.15.7:0/2594390261' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-18T08:10:18.819 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3 2024-09-18T08:10:18.853 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:10:18.854 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:10:18.854 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3' 2024-09-18T08:10:18.938 INFO:teuthology.orchestra.run.smithi007.stdout:{"pg_ready":true,"pg_map":{"version":359,"stamp":"2024-09-18T08:10:15.241663+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":3353549363,"num_objects":51362,"num_object_clones":0,"num_object_copies":154086,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":51,"num_objects_unfound":0,"num_objects_dirty":51362,"num_whiteouts":0,"num_read":923,"num_read_kb":675,"num_write":150773,"num_write_kb":6447982,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24435,"num_bytes_recovered":1585519826,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":79931,"ondisk_log_size":79931,"up":474,"acting":473,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":479,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":12796656,"kb_used_data":10821340,"kb_used_omap":12,"kb_used_meta":1974963,"kb_avail":737164560,"statfs":{"total":767960285184,"available":754856509440,"internally_reserved":0,"allocated":11081052160,"data_stored":11078191277,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12750,"internal_metadata":2022362674},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,1,3,1],"upper_bound":32},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-337641472,"num_objects":-5152,"num_object_clones":0,"num_object_copies":-15456,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-435,"num_objects_unfound":0,"num_objects_dirty":-5152,"num_whiteouts":0,"num_read":5,"num_read_kb":5,"num_write":5152,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":435,"num_bytes_recovered":28508160,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.012098"},"pg_stats":[{"pgid":"3.1f","version":"0'0","reported_seq":356,"reported_epoch":218,"state":"active+clean","last_fresh":"2024-09-18T08:10:12.297355+0000","last_change":"2024-09-18T08:08:52.697893+0000","last_active":"2024-09-18T08:10:12.297355+0000","last_peered":"2024-09-18T08:10:12.297355+0000","last_clean":"2024-09-18T08:10:12.297355+0000","last_became_active":"2024-09-18T08:08:52.697499+0000","last_became_peered":"2024-09-18T08:08:52.697499+0000","last_unstale":"2024-09-18T08:10:12.297355+0000","last_undegraded":"2024-09-18T08:10:12.297355+0000","last_fullsized":"2024-09-18T08:10:12.297355+0000","mapping_epoch":198,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":199,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:08:06.642662+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:46.860092+0000","last_clean_scrub_stamp":"2024-09-18T08:08:06.642662+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"69'9","reported_seq":94,"reported_epoch":218,"state":"active+clean","last_fresh":"2024-09-18T08:10:12.296035+0000","last_change":"2024-09-18T08:08:18.575611+0000","last_active":"2024-09-18T08:10:12.296035+0000","last_peered":"2024-09-18T08:10:12.296035+0000","last_clean":"2024-09-18T08:10:12.296035+0000","last_became_active":"2024-09-18T08:08:18.537856+0000","last_became_peered":"2024-09-18T08:08:18.537856+0000","last_unstale":"2024-09-18T08:10:12.296035+0000","last_undegraded":"2024-09-18T08:10:12.296035+0000","last_fullsized":"2024-09-18T08:10:12.296035+0000","mapping_epoch":181,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":182,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'9","last_scrub_stamp":"2024-09-18T08:05:08.696796+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_clean_scrub_stamp":"2024-09-18T08:05:08.696796+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":16,"num_read_kb":10,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":342,"reported_epoch":218,"state":"active+clean","last_fresh":"2024-09-18T08:10:12.301244+0000","last_change":"2024-09-18T08:07:18.737364+0000","last_active":"2024-09-18T08:10:12.301244+0000","last_peered":"2024-09-18T08:10:12.301244+0000","last_clean":"2024-09-18T08:10:12.301244+0000","last_became_active":"2024-09-18T08:02:51.907140+0000","last_became_peered":"2024-09-18T08:02:51.907140+0000","last_unstale":"2024-09-18T08:10:12.301244+0000","last_undegraded":"2024-09-18T08:10:12.301244+0000","last_fullsized":"2024-09-18T08:10:12.301244+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:07:18.737122+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:50.886830+0000","last_clean_scrub_stamp":"2024-09-18T08:07:18.737122+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:10:14.742960+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":84,"reported_epoch":218,"state":"active+clean","last_fresh":"2024-09-18T08:10:12.296437+0000","last_change":"2024-09-18T08:08:18.545192+0000","last_active":"2024-09-18T08:10:12.296437+0000","last_peered":"2024-09-18T08:10:12.296437+0000","last_clean":"2024-09-18T08:10:12.296437+0000","last_became_active":"2024-09-18T08:08:18.540705+0000","last_became_peered":"2024-09-18T08:08:18.540705+0000","last_unstale":"2024-09-18T08:10:12.296437+0000","last_undegraded":"2024-09-18T08:10:12.296437+0000","last_fullsized":"2024-09-18T08:10:12.296437+0000","mapping_epoch":181,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":182,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:07:08.451865+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_clean_scrub_stamp":"2024-09-18T08:07:08.451865+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":344,"reported_epoch":218,"state":"active+clean","last_fresh":"2024-09-18T08:10:12.321843+0000","last_change":"2024-09-18T08:07:13.244647+0000","last_active":"2024-09-18T08:10:12.321843+0000","last_peered":"2024-09-18T08:10:12.321843+0000","last_clean":"2024-09-18T08:10:12.321843+0000","last_became_active":"2024-09-18T08:06:24.128511+0000","last_became_peered":"2024-09-18T08:06:24.128511+0000","last_unstale":"2024-09-18T08:10:12.321843+0000","last_undegraded":"2024-09-18T08:10:12.321843+0000","last_fullsized":"2024-09-18T08:10:12.321843+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:07:13.244555+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_clean_scrub_stamp":"2024-09-18T08:07:13.244555+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:10:55.479885+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":365,"reported_epoch":218,"state":"active+clean","last_fresh":"2024-09-18T08:10:12.321598+0000","last_change":"2024-09-18T08:10:08.473695+0000","last_active":"2024-09-18T08:10:12.321598+0000","last_peered":"2024-09-18T08:10:12.321598+0000","last_clean":"2024-09-18T08:10:12.321598+0000","last_became_active":"2024-09-18T08:06:24.126194+0000","last_became_peered":"2024-09-18T08:06:24.126194+0000","last_unstale":"2024-09-18T08:10:12.321598+0000","last_undegraded":"2024-09-18T08:10:12.321598+0000","last_fullsized":"2024-09-18T08:10:12.321598+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:10:08.473626+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:46.860092+0000","last_clean_scrub_stamp":"2024-09-18T08:10:08.473626+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:11:38.009160+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"69'15","reported_seq":356,"reported_epoch":218,"state":"active+clean","last_fresh":"2024-09-18T08:10:12.320876+0000","last_change":"2024-09-18T08:09:32.575836+0000","last_active":"2024-09-18T08:10:12.320876+0000","last_peered":"2024-09-18T08:10:12.320876+0000","last_clean":"2024-09-18T08:10:12.320876+0000","last_became_active":"2024-09-18T08:02:49.886106+0000","last_became_peered":"2024-09-18T08:02:49.886106+0000","last_unstale":"2024-09-18T08:10:12.320876+0000","last_undegraded":"2024-09-18T08:10:12.320876+0000","last_fullsized":"2024-09-18T08:10:12.320876+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'15","last_scrub_stamp":"2024-09-18T08:09:32.575741+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_clean_scrub_stamp":"2024-09-18T08:09:32.575741+0000","objects_scrubbed":7,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:10:59.141685+0000","scrub_duration":40,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":84,"reported_epoch":218,"state":"active+clean","last_fresh":"2024-09-18T08:10:12.296149+0000","last_change":"2024-09-18T08:08:18.543316+0000","last_active":"2024-09-18T08:10:12.296149+0000","last_peered":"2024-09-18T08:10:12.296149+0000","last_clean":"2024-09-18T08:10:12.296149+0000","last_became_active":"2024-09-18T08:08:18.538645+0000","last_became_peered":"2024-09-18T08:08:18.538645+0000","last_unstale":"2024-09-18T08:10:12.296149+0000","last_undegraded":"2024-09-18T08:10:12.296149+0000","last_fullsized":"2024-09-18T08:10:12.296149+0000","mapping_epoch":181,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":182,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:08:09.142188+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:50.886830+0000","last_clean_scrub_stamp":"2024-09-18T08:08:09.142188+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":369,"reported_epoch":218,"state":"active+clean","last_fresh":"2024-09-18T08:10:12.302748+0000","last_change":"2024-09-18T08:08:18.548588+0000","last_active":"2024-09-18T08:10:12.302748+0000","last_peered":"2024-09-18T08:10:12.302748+0000","last_clean":"2024-09-18T08:10:12.302748+0000","last_became_active":"2024-09-18T08:08:18.548203+0000","last_became_peered":"2024-09-18T08:08:18.548203+0000","last_unstale":"2024-09-18T08:10:12.302748+0000","last_undegraded":"2024-09-18T08:10:12.302748+0000","last_fullsized":"2024-09-18T08:10:12.302748+0000","mapping_epoch":181,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":182,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:06:14.027077+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:46.860092+0000","last_clean_scrub_stamp":"2024-09-18T08:06:14.027077+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"69'9","reported_seq":94,"reported_epoch":218,"state":"active+clean","last_fresh":"2024-09-18T08:10:12.296672+0000","last_change":"2024-09-18T08:08:18.631597+0000","last_active":"2024-09-18T08:10:12.296672+0000","last_peered":"2024-09-18T08:10:12.296672+0000","last_clean":"2024-09-18T08:10:12.296672+0000","last_became_active":"2024-09-18T08:08:18.550960+0000","last_became_peered":"2024-09-18T08:08:18.550960+0000","last_unstale":"2024-09-18T08:10:12.296672+0000","last_undegraded":"2024-09-18T08:10:12.296672+0000","last_fullsized":"2024-09-18T08:10:12.296672+0000","mapping_epoch":181,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":182,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_clean_scrub_stamp":"2024-09-18T08:02:48.870718+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":16,"num_read_kb":10,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":333,"reported_epoch":218,"state":"active+clean","last_fresh":"2024-09-18T08:10:12.302615+0000","last_change":"2024-09-18T08:05:33.935844+0000","last_active":"2024-09-18T08:10:12.302615+0000","last_peered":"2024-09-18T08:10:12.302615+0000","last_clean":"2024-09-18T08:10:12.302615+0000","last_became_active":"2024-09-18T08:02:51.907296+0000","last_became_peered":"2024-09-18T08:02:51.907296+0000","last_unstale":"2024-09-18T08:10:12.302615+0000","last_undegraded":"2024-09-18T08:10:12.302615+0000","last_fullsized":"2024-09-18T08:10:12.302615+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:05:33.935798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:50.886830+0000","last_clean_scrub_stamp":"2024-09-18T08:05:33.935798+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":331,"reported_epoch":218,"state":"active+clean","last_fresh":"2024-09-18T08:10:12.296578+0000","last_change":"2024-09-18T08:08:14.646073+0000","last_active":"2024-09-18T08:10:12.296578+0000","last_peered":"2024-09-18T08:10:12.296578+0000","last_clean":"2024-09-18T08:10:12.296578+0000","last_became_active":"2024-09-18T08:02:53.913519+0000","last_became_peered":"2024-09-18T08:02:53.913519+0000","last_unstale":"2024-09-18T08:10:12.296578+0000","last_undegraded":"2024-09-18T08:10:12.296578+0000","last_fullsized":"2024-09-18T08:10:12.296578+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:08:14.645993+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_clean_scrub_stamp":"2024-09-18T08:08:14.645993+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":347,"reported_epoch":218,"state":"active+clean","last_fresh":"2024-09-18T08:10:12.303085+0000","last_change":"2024-09-18T08:08:18.624260+0000","last_active":"2024-09-18T08:10:12.303085+0000","last_peered":"2024-09-18T08:10:12.303085+0000","last_clean":"2024-09-18T08:10:12.303085+0000","last_became_active":"2024-09-18T08:08:18.623757+0000","last_became_peered":"2024-09-18T08:08:18.623757+0000","last_unstale":"2024-09-18T08:10:12.303085+0000","last_undegraded":"2024-09-18T08:10:12.303085+0000","last_fullsized":"2024-09-18T08:10:12.303085+0000","mapping_epoch":181,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":182,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:04:32.849119+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:46.860092+0000","last_clean_scrub_stamp":"2024-09-18T08:04:32.849119+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1b","version":"69'5","reported_seq":90,"reported_epoch":218,"state":"active+clean","last_fresh":"2024-09-18T08:10:12.296768+0000","last_change":"2024-09-18T08:08:18.651427+0000","last_active":"2024-09-18T08:10:12.296768+0000","last_peered":"2024-09-18T08:10:12.296768+0000","last_clean":"2024-09-18T08:10:12.296768+0000","last_became_active":"2024-09-18T08:08:18.567126+0000","last_became_peered":"2024-09-18T08:08:18.567126+0000","last_unstale":"2024-09-18T08:10:12.296768+0000","last_undegraded":"2024-09-18T08:10:12.296768+0000","last_fullsized":"2024-09-18T08:10:12.296768+0000","mapping_epoch":181,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":182,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'5","last_scrub_stamp":"2024-09-18T08:06:09.465979+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_clean_scrub_stamp":"2024-09-18T08:06:09.465979+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":10,"num_read_kb":6,"num_write":6,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":363,"reported_epoch":218,"state":"active+clean","last_fresh":"2024-09-18T08:10:12.300807+0000","last_change":"2024-09-18T08:09:12.674889+0000","last_active":"2024-09-18T08:10:12.300807+0000","last_peered":"2024-09-18T08:10:12.300807+0000","last_clean":"2024-09-18T08:10:12.300807+0000","last_became_active":"2024-09-18T08:08:18.524932+0000","last_became_peered":"2024-09-18T08:08:18.524932+0000","last_unstale":"2024-09-18T08:10:12.300807+0000","last_undegraded":"2024-09-18T08:10:12.300807+0000","last_fullsized":"2024-09-18T08:10:12.300807+0000","mapping_epoch":181,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":182,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:09:12.674843+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:50.886830+0000","last_clean_scrub_stamp":"2024-09-18T08:09:12.674843+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:10:16.027634+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":331,"reported_epoch":218,"state":"active+clean","last_fresh":"2024-09-18T08:10:12.302645+0000","last_change":"2024-09-18T08:06:44.518882+0000","last_active":"2024-09-18T08:10:12.302645+0000","last_peered":"2024-09-18T08:10:12.302645+0000","last_clean":"2024-09-18T08:10:12.302645+0000","last_became_active":"2024-09-18T08:02:53.911648+0000","last_became_peered":"2024-09-18T08:02:53.911648+0000","last_unstale":"2024-09-18T08:10:12.302645+0000","last_undegraded":"2024-09-18T08:10:12.302645+0000","last_fullsized":"2024-09-18T08:10:12.302645+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:06:44.518835+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_clean_scrub_stamp":"2024-09-18T08:06:44.518835+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":84,"reported_epoch":218,"state":"active+clean","last_fresh":"2024-09-18T08:10:12.297278+0000","last_change":"2024-09-18T08:08:18.544203+0000","last_active":"2024-09-18T08:10:12.297278+0000","last_peered":"2024-09-18T08:10:12.297278+0000","last_clean":"2024-09-18T08:10:12.297278+0000","last_became_active":"2024-09-18T08:08:18.539604+0000","last_became_peered":"2024-09-18T08:08:18.539604+0000","last_unstale":"2024-09-18T08:10:12.297278+0000","last_undegraded":"2024-09-18T08:10:12.297278+0000","last_fullsized":"2024-09-18T08:10:12.297278+0000","mapping_epoch":181,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":182,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:05:40.477965+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_clean_scrub_stamp":"2024-09-18T08:05:40.477965+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":355,"reported_epoch":218,"state":"active+clean","last_fresh":"2024-09-18T08:10:12.297131+0000","last_change":"2024-09-18T08:08:18.524124+0000","last_active":"2024-09-18T08:10:12.297131+0000","last_peered":"2024-09-18T08:10:12.297131+0000","last_clean":"2024-09-18T08:10:12.297131+0000","last_became_active":"2024-09-18T08:08:18.523611+0000","last_became_peered":"2024-09-18T08:08:18.523611+0000","last_unstale":"2024-09-18T08:10:12.297131+0000","last_undegraded":"2024-09-18T08:10:12.297131+0000","last_fullsized":"2024-09-18T08:10:12.297131+0000","mapping_epoch":181,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":182,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:05:27.476144+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:46.860092+0000","last_clean_scrub_stamp":"2024-09-18T08:05:27.476144+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:10:23.059202+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1c","version":"69'15","reported_seq":66,"reported_epoch":218,"state":"active+clean","last_fresh":"2024-09-18T08:10:12.297993+0000","last_change":"2024-09-18T08:08:52.708963+0000","last_active":"2024-09-18T08:10:12.297993+0000","last_peered":"2024-09-18T08:10:12.297993+0000","last_clean":"2024-09-18T08:10:12.297993+0000","last_became_active":"2024-09-18T08:08:52.690788+0000","last_became_peered":"2024-09-18T08:08:52.690788+0000","last_unstale":"2024-09-18T08:10:12.297993+0000","last_undegraded":"2024-09-18T08:10:12.297993+0000","last_fullsized":"2024-09-18T08:10:12.297993+0000","mapping_epoch":198,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":199,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'15","last_scrub_stamp":"2024-09-18T08:07:39.530340+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_clean_scrub_stamp":"2024-09-18T08:07:39.530340+0000","objects_scrubbed":7,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":363,"reported_epoch":218,"state":"active+clean","last_fresh":"2024-09-18T08:10:12.302250+0000","last_change":"2024-09-18T08:09:11.693711+0000","last_active":"2024-09-18T08:10:12.302250+0000","last_peered":"2024-09-18T08:10:12.302250+0000","last_clean":"2024-09-18T08:10:12.302250+0000","last_became_active":"2024-09-18T08:08:18.538505+0000","last_became_peered":"2024-09-18T08:08:18.538505+0000","last_unstale":"2024-09-18T08:10:12.302250+0000","last_undegraded":"2024-09-18T08:10:12.302250+0000","last_fullsized":"2024-09-18T08:10:12.302250+0000","mapping_epoch":181,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":182,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:09:11.693643+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:50.886830+0000","last_clean_scrub_stamp":"2024-09-18T08:09:11.693643+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:10:34.429057+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":345,"reported_epoch":218,"state":"active+clean","last_fresh":"2024-09-18T08:10:12.322499+0000","last_change":"2024-09-18T08:09:36.592348+0000","last_active":"2024-09-18T08:10:12.322499+0000","last_peered":"2024-09-18T08:10:12.322499+0000","last_clean":"2024-09-18T08:10:12.322499+0000","last_became_active":"2024-09-18T08:06:24.128659+0000","last_became_peered":"2024-09-18T08:06:24.128659+0000","last_unstale":"2024-09-18T08:10:12.322499+0000","last_undegraded":"2024-09-18T08:10:12.322499+0000","last_fullsized":"2024-09-18T08:10:12.322499+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:09:36.592281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_clean_scrub_stamp":"2024-09-18T08:09:36.592281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:10:46.573780+0000","scrub_duration":16,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":84,"reported_epoch":218,"state":"active+clean","last_fresh":"2024-09-18T08:10:12.296317+0000","last_change":"2024-09-18T08:08:18.536559+0000","last_active":"2024-09-18T08:10:12.296317+0000","last_peered":"2024-09-18T08:10:12.296317+0000","last_clean":"2024-09-18T08:10:12.296317+0000","last_became_active":"2024-09-18T08:08:18.536084+0000","last_became_peered":"2024-09-18T08:08:18.536084+0000","last_unstale":"2024-09-18T08:10:12.296317+0000","last_undegraded":"2024-09-18T08:10:12.296317+0000","last_fullsized":"2024-09-18T08:10:12.296317+0000","mapping_epoch":181,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":182,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:07:13.543722+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:46.860092+0000","last_clean_scrub_stamp":"2024-09-18T08:07:13.543722+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"69'12","reported_seq":407,"reported_epoch":218,"state":"active+clean","last_fresh":"2024-09-18T08:10:12.443637+0000","last_change":"2024-09-18T08:10:12.443637+0000","last_active":"2024-09-18T08:10:12.443637+0000","last_peered":"2024-09-18T08:10:12.443637+0000","last_clean":"2024-09-18T08:10:12.443637+0000","last_became_active":"2024-09-18T08:08:18.543675+0000","last_became_peered":"2024-09-18T08:08:18.543675+0000","last_unstale":"2024-09-18T08:10:12.443637+0000","last_undegraded":"2024-09-18T08:10:12.443637+0000","last_fullsized":"2024-09-18T08:10:12.443637+0000","mapping_epoch":181,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":182,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'12","last_scrub_stamp":"2024-09-18T08:10:12.443558+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_clean_scrub_stamp":"2024-09-18T08:10:12.443558+0000","objects_scrubbed":5,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:11:22.178344+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":25,"num_read_kb":16,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":84,"reported_epoch":218,"state":"active+clean","last_fresh":"2024-09-18T08:10:12.296854+0000","last_change":"2024-09-18T08:08:18.532482+0000","last_active":"2024-09-18T08:10:12.296854+0000","last_peered":"2024-09-18T08:10:12.296854+0000","last_clean":"2024-09-18T08:10:12.296854+0000","last_became_active":"2024-09-18T08:08:18.531438+0000","last_became_peered":"2024-09-18T08:08:18.531438+0000","last_unstale":"2024-09-18T08:10:12.296854+0000","last_undegraded":"2024-09-18T08:10:12.296854+0000","last_fullsized":"2024-09-18T08:10:12.296854+0000","mapping_epoch":181,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":182,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:06:12.434179+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:50.886830+0000","last_clean_scrub_stamp":"2024-09-18T08:06:12.434179+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1c","version":"69'1","reported_seq":173,"reported_epoch":218,"state":"active+clean","last_fresh":"2024-09-18T08:10:12.302699+0000","last_change":"2024-09-18T08:06:46.142143+0000","last_active":"2024-09-18T08:10:12.302699+0000","last_peered":"2024-09-18T08:10:12.302699+0000","last_clean":"2024-09-18T08:10:12.302699+0000","last_became_active":"2024-09-18T08:06:46.141661+0000","last_became_peered":"2024-09-18T08:06:46.141661+0000","last_unstale":"2024-09-18T08:10:12.302699+0000","last_undegraded":"2024-09-18T08:10:12.302699+0000","last_fullsized":"2024-09-18T08:10:12.302699+0000","mapping_epoch":143,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":144,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'1","last_scrub_stamp":"2024-09-18T08:05:36.776369+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_clean_scrub_stamp":"2024-09-18T08:05:36.776369+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":403,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,2],"acting":[5,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"62'1","reported_seq":373,"reported_epoch":218,"state":"active+clean","last_fresh":"2024-09-18T08:10:12.301398+0000","last_change":"2024-09-18T08:09:29.633875+0000","last_active":"2024-09-18T08:10:12.301398+0000","last_peered":"2024-09-18T08:10:12.301398+0000","last_clean":"2024-09-18T08:10:12.301398+0000","last_became_active":"2024-09-18T08:08:18.559125+0000","last_became_peered":"2024-09-18T08:08:18.559125+0000","last_unstale":"2024-09-18T08:10:12.301398+0000","last_undegraded":"2024-09-18T08:10:12.301398+0000","last_fullsized":"2024-09-18T08:10:12.301398+0000","mapping_epoch":181,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":182,"parent":"0.0","parent_split_bits":0,"last_scrub":"62'1","last_scrub_stamp":"2024-09-18T08:09:29.633520+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:46.860092+0000","last_clean_scrub_stamp":"2024-09-18T08:09:29.633520+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:10:38.890672+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":92,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1e","version":"69'10","reported_seq":337,"reported_epoch":218,"state":"active+clean","last_fresh":"2024-09-18T08:10:12.298622+0000","last_change":"2024-09-18T08:02:49.892755+0000","last_active":"2024-09-18T08:10:12.298622+0000","last_peered":"2024-09-18T08:10:12.298622+0000","last_clean":"2024-09-18T08:10:12.298622+0000","last_became_active":"2024-09-18T08:02:49.892024+0000","last_became_peered":"2024-09-18T08:02:49.892024+0000","last_unstale":"2024-09-18T08:10:12.298622+0000","last_undegraded":"2024-09-18T08:10:12.298622+0000","last_fullsized":"2024-09-18T08:10:12.298622+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_clean_scrub_stamp":"2024-09-18T08:02:48.870718+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:10:17.106246+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1f","version":"182'13","reported_seq":112,"reported_epoch":218,"state":"active+clean","last_fresh":"2024-09-18T08:10:12.297260+0000","last_change":"2024-09-18T08:08:18.627670+0000","last_active":"2024-09-18T08:10:12.297260+0000","last_peered":"2024-09-18T08:10:12.297260+0000","last_clean":"2024-09-18T08:10:12.297260+0000","last_became_active":"2024-09-18T08:08:18.543095+0000","last_became_peered":"2024-09-18T08:08:18.543095+0000","last_unstale":"2024-09-18T08:10:12.297260+0000","last_undegraded":"2024-09-18T08:10:12.297260+0000","last_fullsized":"2024-09-18T08:10:12.297260+0000","mapping_epoch":181,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":182,"parent":"0.0","parent_split_bits":0,"last_scrub":"173'12","last_scrub_stamp":"2024-09-18T08:08:07.553075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:50.886830+0000","last_clean_scrub_stamp":"2024-09-18T08:08:07.553075+0000","objects_scrubbed":0,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,7],"acting":[4,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.3","version":"218'9652","reported_seq":9842,"reported_epoch":218,"state":"active+clean","last_fresh":"2024-09-18T08:10:13.690124+0000","last_change":"2024-09-18T08:09:28.263522+0000","last_active":"2024-09-18T08:10:13.690124+0000","last_peered":"2024-09-18T08:10:13.690124+0000","last_clean":"2024-09-18T08:10:13.690124+0000","last_became_active":"2024-09-18T08:07:53.672797+0000","last_became_peered":"2024-09-18T08:07:53.672797+0000","last_unstale":"2024-09-18T08:10:13.690124+0000","last_undegraded":"2024-09-18T08:10:13.690124+0000","last_fullsized":"2024-09-18T08:10:13.690124+0000","mapping_epoch":165,"log_start":"202'4800","ondisk_log_start":"202'4800","created":165,"last_epoch_clean":166,"parent":"0.0","parent_split_bits":0,"last_scrub":"211'6448","last_scrub_stamp":"2024-09-18T08:09:28.263434+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:07:52.307205+0000","last_clean_scrub_stamp":"2024-09-18T08:09:28.263434+0000","objects_scrubbed":6448,"log_size":4852,"log_dups_size":0,"ondisk_log_size":4852,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:10:29.843317+0000","scrub_duration":646,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":212598784,"num_objects":3244,"num_object_clones":0,"num_object_copies":9732,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3244,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9652,"num_write_kb":412672,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6],"acting":[1,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.a","version":"74'79","reported_seq":155,"reported_epoch":218,"state":"active+clean","last_fresh":"2024-09-18T08:10:12.299857+0000","last_change":"2024-09-18T08:08:57.463205+0000","last_active":"2024-09-18T08:10:12.299857+0000","last_peered":"2024-09-18T08:10:12.299857+0000","last_clean":"2024-09-18T08:10:12.299857+0000","last_became_active":"2024-09-18T08:07:26.394584+0000","last_became_peered":"2024-09-18T08:07:26.394584+0000","last_unstale":"2024-09-18T08:10:12.299857+0000","last_undegraded":"2024-09-18T08:10:12.299857+0000","last_fullsized":"2024-09-18T08:10:12.299857+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":152,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":4,"last_scrub":"74'79","last_scrub_stamp":"2024-09-18T08:08:57.463113+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:00:11.893742+0000","last_clean_scrub_stamp":"2024-09-18T08:08:57.463113+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,6],"acting":[5,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.f","version":"69'15","reported_seq":409,"reported_epoch":218,"state":"active+clean","last_fresh":"2024-09-18T08:10:12.304773+0000","last_change":"2024-09-18T08:08:56.669262+0000","last_active":"2024-09-18T08:10:12.304773+0000","last_peered":"2024-09-18T08:10:12.304773+0000","last_clean":"2024-09-18T08:10:12.304773+0000","last_became_active":"2024-09-18T08:08:18.540211+0000","last_became_peered":"2024-09-18T08:08:18.540211+0000","last_unstale":"2024-09-18T08:10:12.304773+0000","last_undegraded":"2024-09-18T08:10:12.304773+0000","last_fullsized":"2024-09-18T08:10:12.304773+0000","mapping_epoch":181,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":182,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'15","last_scrub_stamp":"2024-09-18T08:08:56.668397+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_clean_scrub_stamp":"2024-09-18T08:08:56.668397+0000","objects_scrubbed":7,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:10:22.470704+0000","scrub_duration":36,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":343,"reported_epoch":218,"state":"active+clean","last_fresh":"2024-09-18T08:10:12.320700+0000","last_change":"2024-09-18T08:08:18.389332+0000","last_active":"2024-09-18T08:10:12.320700+0000","last_peered":"2024-09-18T08:10:12.320700+0000","last_clean":"2024-09-18T08:10:12.320700+0000","last_became_active":"2024-09-18T08:02:47.879328+0000","last_became_peered":"2024-09-18T08:02:47.879328+0000","last_unstale":"2024-09-18T08:10:12.320700+0000","last_undegraded":"2024-09-18T08:10:12.320700+0000","last_fullsized":"2024-09-18T08:10:12.320700+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:08:18.389273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:46.860092+0000","last_clean_scrub_stamp":"2024-09-18T08:08:18.389273+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.e","version":"182'12","reported_seq":112,"reported_epoch":218,"state":"active+clean","last_fresh":"2024-09-18T08:10:12.297294+0000","last_change":"2024-09-18T08:08:18.560884+0000","last_active":"2024-09-18T08:10:12.297294+0000","last_peered":"2024-09-18T08:10:12.297294+0000","last_clean":"2024-09-18T08:10:12.297294+0000","last_became_active":"2024-09-18T08:08:18.525147+0000","last_became_peered":"2024-09-18T08:08:18.525147+0000","last_unstale":"2024-09-18T08:10:12.297294+0000","last_undegraded":"2024-09-18T08:10:12.297294+0000","last_fullsized":"2024-09-18T08:10:12.297294+0000","mapping_epoch":181,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":182,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'8","last_scrub_stamp":"2024-09-18T08:06:19.459375+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:50.886830+0000","last_clean_scrub_stamp":"2024-09-18T08:06:19.459375+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":339,"reported_epoch":218,"state":"active+clean","last_fresh":"2024-09-18T08:10:12.302041+0000","last_change":"2024-09-18T08:07:19.714477+0000","last_active":"2024-09-18T08:10:12.302041+0000","last_peered":"2024-09-18T08:10:12.302041+0000","last_clean":"2024-09-18T08:10:12.302041+0000","last_became_active":"2024-09-18T08:02:53.909834+0000","last_became_peered":"2024-09-18T08:02:53.909834+0000","last_unstale":"2024-09-18T08:10:12.302041+0000","last_undegraded":"2024-09-18T08:10:12.302041+0000","last_fullsized":"2024-09-18T08:10:12.302041+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:07:19.714430+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_clean_scrub_stamp":"2024-09-18T08:07:19.714430+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.c","version":"218'9332","reported_seq":9490,"reported_epoch":218,"state":"active+clean","last_fresh":"2024-09-18T08:10:13.829762+0000","last_change":"2024-09-18T08:10:11.144727+0000","last_active":"2024-09-18T08:10:13.829762+0000","last_peered":"2024-09-18T08:10:13.829762+0000","last_clean":"2024-09-18T08:10:13.829762+0000","last_became_active":"2024-09-18T08:07:53.669708+0000","last_became_peered":"2024-09-18T08:07:53.669708+0000","last_unstale":"2024-09-18T08:10:13.829762+0000","last_undegraded":"2024-09-18T08:10:13.829762+0000","last_fullsized":"2024-09-18T08:10:13.829762+0000","mapping_epoch":165,"log_start":"191'3900","ondisk_log_start":"191'3900","created":165,"last_epoch_clean":166,"parent":"0.0","parent_split_bits":0,"last_scrub":"217'9165","last_scrub_stamp":"2024-09-18T08:10:11.144412+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:07:52.307205+0000","last_clean_scrub_stamp":"2024-09-18T08:10:11.144412+0000","objects_scrubbed":3298,"log_size":5432,"log_dups_size":0,"ondisk_log_size":5432,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:11:37.247660+0000","scrub_duration":1357,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":203030528,"num_objects":3098,"num_object_clones":0,"num_object_copies":9294,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3098,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9332,"num_write_kb":397760,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"74'79","reported_seq":99,"reported_epoch":218,"state":"active+clean","last_fresh":"2024-09-18T08:10:12.296832+0000","last_change":"2024-09-18T08:08:52.690758+0000","last_active":"2024-09-18T08:10:12.296832+0000","last_peered":"2024-09-18T08:10:12.296832+0000","last_clean":"2024-09-18T08:10:12.296832+0000","last_became_active":"2024-09-18T08:08:52.690344+0000","last_became_peered":"2024-09-18T08:08:52.690344+0000","last_unstale":"2024-09-18T08:10:12.296832+0000","last_undegraded":"2024-09-18T08:10:12.296832+0000","last_fullsized":"2024-09-18T08:10:12.296832+0000","mapping_epoch":198,"log_start":"0'0","ondisk_log_start":"0'0","created":152,"last_epoch_clean":199,"parent":"0.0","parent_split_bits":3,"last_scrub":"74'79","last_scrub_stamp":"2024-09-18T08:08:19.036836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:00:11.893742+0000","last_clean_scrub_stamp":"2024-09-18T08:08:19.036836+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,6],"acting":[4,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.0","version":"69'18","reported_seq":371,"reported_epoch":218,"state":"active+clean","last_fresh":"2024-09-18T08:10:12.320486+0000","last_change":"2024-09-18T08:06:44.301130+0000","last_active":"2024-09-18T08:10:12.320486+0000","last_peered":"2024-09-18T08:10:12.320486+0000","last_clean":"2024-09-18T08:10:12.320486+0000","last_became_active":"2024-09-18T08:02:49.886818+0000","last_became_peered":"2024-09-18T08:02:49.886818+0000","last_unstale":"2024-09-18T08:10:12.320486+0000","last_undegraded":"2024-09-18T08:10:12.320486+0000","last_fullsized":"2024-09-18T08:10:12.320486+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'18","last_scrub_stamp":"2024-09-18T08:06:44.300772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_clean_scrub_stamp":"2024-09-18T08:06:44.300772+0000","objects_scrubbed":8,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:10:52.429408+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":34,"num_read_kb":22,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":359,"reported_epoch":218,"state":"active+clean","last_fresh":"2024-09-18T08:10:12.323064+0000","last_change":"2024-09-18T08:10:09.469428+0000","last_active":"2024-09-18T08:10:12.323064+0000","last_peered":"2024-09-18T08:10:12.323064+0000","last_clean":"2024-09-18T08:10:12.323064+0000","last_became_active":"2024-09-18T08:02:47.880225+0000","last_became_peered":"2024-09-18T08:02:47.880225+0000","last_unstale":"2024-09-18T08:10:12.323064+0000","last_undegraded":"2024-09-18T08:10:12.323064+0000","last_fullsized":"2024-09-18T08:10:12.323064+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:10:09.469351+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:46.860092+0000","last_clean_scrub_stamp":"2024-09-18T08:10:09.469351+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:11:16.425431+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":84,"reported_epoch":218,"state":"active+clean","last_fresh":"2024-09-18T08:10:12.296378+0000","last_change":"2024-09-18T08:08:18.530658+0000","last_active":"2024-09-18T08:10:12.296378+0000","last_peered":"2024-09-18T08:10:12.296378+0000","last_clean":"2024-09-18T08:10:12.296378+0000","last_became_active":"2024-09-18T08:08:18.530341+0000","last_became_peered":"2024-09-18T08:08:18.530341+0000","last_unstale":"2024-09-18T08:10:12.296378+0000","last_undegraded":"2024-09-18T08:10:12.296378+0000","last_fullsized":"2024-09-18T08:10:12.296378+0000","mapping_epoch":181,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":182,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:06:10.468119+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:50.886830+0000","last_clean_scrub_stamp":"2024-09-18T08:06:10.468119+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":84,"reported_epoch":218,"state":"active+clean","last_fresh":"2024-09-18T08:10:12.296572+0000","last_change":"2024-09-18T08:08:18.551527+0000","last_active":"2024-09-18T08:10:12.296572+0000","last_peered":"2024-09-18T08:10:12.296572+0000","last_clean":"2024-09-18T08:10:12.296572+0000","last_became_active":"2024-09-18T08:08:18.543694+0000","last_became_peered":"2024-09-18T08:08:18.543694+0000","last_unstale":"2024-09-18T08:10:12.296572+0000","last_undegraded":"2024-09-18T08:10:12.296572+0000","last_fullsized":"2024-09-18T08:10:12.296572+0000","mapping_epoch":181,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":182,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:04:16.427390+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_clean_scrub_stamp":"2024-09-18T08:04:16.427390+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.d","version":"218'9303","reported_seq":9422,"reported_epoch":218,"state":"active+clean","last_fresh":"2024-09-18T08:10:13.299509+0000","last_change":"2024-09-18T08:07:53.671291+0000","last_active":"2024-09-18T08:10:13.299509+0000","last_peered":"2024-09-18T08:10:13.299509+0000","last_clean":"2024-09-18T08:10:13.299509+0000","last_became_active":"2024-09-18T08:07:53.670773+0000","last_became_peered":"2024-09-18T08:07:53.670773+0000","last_unstale":"2024-09-18T08:10:13.299509+0000","last_undegraded":"2024-09-18T08:10:13.299509+0000","last_fullsized":"2024-09-18T08:10:13.299509+0000","mapping_epoch":165,"log_start":"196'4200","ondisk_log_start":"196'4200","created":165,"last_epoch_clean":166,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:07:52.307205+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:07:52.307205+0000","last_clean_scrub_stamp":"2024-09-18T08:07:52.307205+0000","objects_scrubbed":0,"log_size":5103,"log_dups_size":0,"ondisk_log_size":5103,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":210108440,"num_objects":3207,"num_object_clones":0,"num_object_copies":9621,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3207,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":9303,"num_write_kb":400257,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,2],"acting":[5,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.4","version":"74'79","reported_seq":161,"reported_epoch":218,"state":"active+clean","last_fresh":"2024-09-18T08:10:12.320790+0000","last_change":"2024-09-18T08:08:59.376417+0000","last_active":"2024-09-18T08:10:12.320790+0000","last_peered":"2024-09-18T08:10:12.320790+0000","last_clean":"2024-09-18T08:10:12.320790+0000","last_became_active":"2024-09-18T08:07:26.398930+0000","last_became_peered":"2024-09-18T08:07:26.398930+0000","last_unstale":"2024-09-18T08:10:12.320790+0000","last_undegraded":"2024-09-18T08:10:12.320790+0000","last_fullsized":"2024-09-18T08:10:12.320790+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":152,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":3,"last_scrub":"74'79","last_scrub_stamp":"2024-09-18T08:08:59.376337+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:00:11.893742+0000","last_clean_scrub_stamp":"2024-09-18T08:08:59.376337+0000","objects_scrubbed":1,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:10:26.641607+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":327680,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,7],"acting":[3,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"69'14","reported_seq":100,"reported_epoch":218,"state":"active+clean","last_fresh":"2024-09-18T08:10:12.296207+0000","last_change":"2024-09-18T08:08:18.617734+0000","last_active":"2024-09-18T08:10:12.296207+0000","last_peered":"2024-09-18T08:10:12.296207+0000","last_clean":"2024-09-18T08:10:12.296207+0000","last_became_active":"2024-09-18T08:08:18.549671+0000","last_became_peered":"2024-09-18T08:08:18.549671+0000","last_unstale":"2024-09-18T08:10:12.296207+0000","last_undegraded":"2024-09-18T08:10:12.296207+0000","last_fullsized":"2024-09-18T08:10:12.296207+0000","mapping_epoch":181,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":182,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'14","last_scrub_stamp":"2024-09-18T08:08:08.629246+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_clean_scrub_stamp":"2024-09-18T08:08:08.629246+0000","objects_scrubbed":0,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":14,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"62'2","reported_seq":375,"reported_epoch":218,"state":"active+clean","last_fresh":"2024-09-18T08:10:12.298869+0000","last_change":"2024-09-18T08:08:18.565279+0000","last_active":"2024-09-18T08:10:12.298869+0000","last_peered":"2024-09-18T08:10:12.298869+0000","last_clean":"2024-09-18T08:10:12.298869+0000","last_became_active":"2024-09-18T08:08:18.544872+0000","last_became_peered":"2024-09-18T08:08:18.544872+0000","last_unstale":"2024-09-18T08:10:12.298869+0000","last_undegraded":"2024-09-18T08:10:12.298869+0000","last_fullsized":"2024-09-18T08:10:12.298869+0000","mapping_epoch":181,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":182,"parent":"0.0","parent_split_bits":0,"last_scrub":"62'2","last_scrub_stamp":"2024-09-18T08:06:43.607152+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:46.860092+0000","last_clean_scrub_stamp":"2024-09-18T08:06:43.607152+0000","objects_scrubbed":2,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:10:26.053059+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1039,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":2078,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.0","version":"182'10","reported_seq":455,"reported_epoch":218,"state":"active+clean","last_fresh":"2024-09-18T08:10:12.320352+0000","last_change":"2024-09-18T08:09:51.467488+0000","last_active":"2024-09-18T08:10:12.320352+0000","last_peered":"2024-09-18T08:10:12.320352+0000","last_clean":"2024-09-18T08:10:12.320352+0000","last_became_active":"2024-09-18T08:08:18.541756+0000","last_became_peered":"2024-09-18T08:08:18.541756+0000","last_unstale":"2024-09-18T08:10:12.320352+0000","last_undegraded":"2024-09-18T08:10:12.320352+0000","last_fullsized":"2024-09-18T08:10:12.320352+0000","mapping_epoch":181,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":182,"parent":"0.0","parent_split_bits":0,"last_scrub":"182'10","last_scrub_stamp":"2024-09-18T08:09:51.467349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:50.886830+0000","last_clean_scrub_stamp":"2024-09-18T08:09:51.467349+0000","objects_scrubbed":1,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:10:51.919411+0000","scrub_duration":19,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":353,"reported_epoch":218,"state":"active+clean","last_fresh":"2024-09-18T08:10:12.301076+0000","last_change":"2024-09-18T08:07:49.650645+0000","last_active":"2024-09-18T08:10:12.301076+0000","last_peered":"2024-09-18T08:10:12.301076+0000","last_clean":"2024-09-18T08:10:12.301076+0000","last_became_active":"2024-09-18T08:06:24.140303+0000","last_became_peered":"2024-09-18T08:06:24.140303+0000","last_unstale":"2024-09-18T08:10:12.301076+0000","last_undegraded":"2024-09-18T08:10:12.301076+0000","last_fullsized":"2024-09-18T08:10:12.301076+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:07:49.650577+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_clean_scrub_stamp":"2024-09-18T08:07:49.650577+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:10:13.720079+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.e","version":"218'9431","reported_seq":13784,"reported_epoch":218,"state":"active+clean","last_fresh":"2024-09-18T08:10:13.813064+0000","last_change":"2024-09-18T08:09:17.296111+0000","last_active":"2024-09-18T08:10:13.813064+0000","last_peered":"2024-09-18T08:10:13.813064+0000","last_clean":"2024-09-18T08:10:13.813064+0000","last_became_active":"2024-09-18T08:09:17.286866+0000","last_became_peered":"2024-09-18T08:09:17.286866+0000","last_unstale":"2024-09-18T08:10:13.813064+0000","last_undegraded":"2024-09-18T08:10:13.813064+0000","last_fullsized":"2024-09-18T08:10:13.813064+0000","mapping_epoch":207,"log_start":"192'4000","ondisk_log_start":"192'4000","created":165,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:07:52.307205+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:07:52.307205+0000","last_clean_scrub_stamp":"2024-09-18T08:07:52.307205+0000","objects_scrubbed":0,"log_size":5431,"log_dups_size":0,"ondisk_log_size":5431,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":218693632,"num_objects":3337,"num_object_clones":0,"num_object_copies":10011,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3337,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9431,"num_write_kb":408576,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2107,"num_bytes_recovered":138084352,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,4],"acting":[0,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.7","version":"74'79","reported_seq":156,"reported_epoch":218,"state":"active+clean","last_fresh":"2024-09-18T08:10:12.322749+0000","last_change":"2024-09-18T08:08:57.438699+0000","last_active":"2024-09-18T08:10:12.322749+0000","last_peered":"2024-09-18T08:10:12.322749+0000","last_clean":"2024-09-18T08:10:12.322749+0000","last_became_active":"2024-09-18T08:07:26.398289+0000","last_became_peered":"2024-09-18T08:07:26.398289+0000","last_unstale":"2024-09-18T08:10:12.322749+0000","last_undegraded":"2024-09-18T08:10:12.322749+0000","last_fullsized":"2024-09-18T08:10:12.322749+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":152,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":3,"last_scrub":"74'79","last_scrub_stamp":"2024-09-18T08:08:57.438600+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:00:11.893742+0000","last_clean_scrub_stamp":"2024-09-18T08:08:57.438600+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,7],"acting":[3,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.2","version":"69'10","reported_seq":399,"reported_epoch":218,"state":"active+clean","last_fresh":"2024-09-18T08:10:12.303412+0000","last_change":"2024-09-18T08:08:18.580590+0000","last_active":"2024-09-18T08:10:12.303412+0000","last_peered":"2024-09-18T08:10:12.303412+0000","last_clean":"2024-09-18T08:10:12.303412+0000","last_became_active":"2024-09-18T08:08:18.538711+0000","last_became_peered":"2024-09-18T08:08:18.538711+0000","last_unstale":"2024-09-18T08:10:12.303412+0000","last_undegraded":"2024-09-18T08:10:12.303412+0000","last_fullsized":"2024-09-18T08:10:12.303412+0000","mapping_epoch":181,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":182,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'10","last_scrub_stamp":"2024-09-18T08:07:22.666042+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_clean_scrub_stamp":"2024-09-18T08:07:22.666042+0000","objects_scrubbed":5,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:10:17.688460+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":334,"reported_epoch":218,"state":"active+clean","last_fresh":"2024-09-18T08:10:12.302857+0000","last_change":"2024-09-18T08:03:48.009689+0000","last_active":"2024-09-18T08:10:12.302857+0000","last_peered":"2024-09-18T08:10:12.302857+0000","last_clean":"2024-09-18T08:10:12.302857+0000","last_became_active":"2024-09-18T08:02:47.874727+0000","last_became_peered":"2024-09-18T08:02:47.874727+0000","last_unstale":"2024-09-18T08:10:12.302857+0000","last_undegraded":"2024-09-18T08:10:12.302857+0000","last_fullsized":"2024-09-18T08:10:12.302857+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:03:48.009611+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:46.860092+0000","last_clean_scrub_stamp":"2024-09-18T08:03:48.009611+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.3","version":"138'10","reported_seq":455,"reported_epoch":218,"state":"active+clean","last_fresh":"2024-09-18T08:10:12.296786+0000","last_change":"2024-09-18T08:07:46.695714+0000","last_active":"2024-09-18T08:10:12.296786+0000","last_peered":"2024-09-18T08:10:12.296786+0000","last_clean":"2024-09-18T08:10:12.296786+0000","last_became_active":"2024-09-18T08:06:24.134194+0000","last_became_peered":"2024-09-18T08:06:24.134194+0000","last_unstale":"2024-09-18T08:10:12.296786+0000","last_undegraded":"2024-09-18T08:10:12.296786+0000","last_fullsized":"2024-09-18T08:10:12.296786+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"138'10","last_scrub_stamp":"2024-09-18T08:07:46.695656+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:50.886830+0000","last_clean_scrub_stamp":"2024-09-18T08:07:46.695656+0000","objects_scrubbed":1,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":361,"reported_epoch":218,"state":"active+clean","last_fresh":"2024-09-18T08:10:12.296637+0000","last_change":"2024-09-18T08:09:46.989395+0000","last_active":"2024-09-18T08:10:12.296637+0000","last_peered":"2024-09-18T08:10:12.296637+0000","last_clean":"2024-09-18T08:10:12.296637+0000","last_became_active":"2024-09-18T08:08:52.700080+0000","last_became_peered":"2024-09-18T08:08:52.700080+0000","last_unstale":"2024-09-18T08:10:12.296637+0000","last_undegraded":"2024-09-18T08:10:12.296637+0000","last_fullsized":"2024-09-18T08:10:12.296637+0000","mapping_epoch":198,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":199,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:09:46.989329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_clean_scrub_stamp":"2024-09-18T08:09:46.989329+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:10:55.582764+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.f","version":"218'9624","reported_seq":25798,"reported_epoch":218,"state":"active+clean","last_fresh":"2024-09-18T08:10:13.779058+0000","last_change":"2024-09-18T08:09:28.367000+0000","last_active":"2024-09-18T08:10:13.779058+0000","last_peered":"2024-09-18T08:10:13.779058+0000","last_clean":"2024-09-18T08:10:13.779058+0000","last_became_active":"2024-09-18T08:09:28.363033+0000","last_became_peered":"2024-09-18T08:09:28.363033+0000","last_unstale":"2024-09-18T08:10:13.779058+0000","last_undegraded":"2024-09-18T08:10:13.779058+0000","last_fullsized":"2024-09-18T08:10:13.779058+0000","mapping_epoch":211,"log_start":"202'5200","ondisk_log_start":"202'5200","created":165,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:07:52.307205+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:07:52.307205+0000","last_clean_scrub_stamp":"2024-09-18T08:07:52.307205+0000","objects_scrubbed":0,"log_size":4424,"log_dups_size":0,"ondisk_log_size":4424,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":215220224,"num_objects":3284,"num_object_clones":0,"num_object_copies":9852,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3284,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9624,"num_write_kb":413056,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8010,"num_bytes_recovered":524943360,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,1],"acting":[3,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"74'79","reported_seq":236,"reported_epoch":218,"state":"active+clean","last_fresh":"2024-09-18T08:10:12.298260+0000","last_change":"2024-09-18T08:08:57.736557+0000","last_active":"2024-09-18T08:10:12.298260+0000","last_peered":"2024-09-18T08:10:12.298260+0000","last_clean":"2024-09-18T08:10:12.298260+0000","last_became_active":"2024-09-18T08:07:26.426532+0000","last_became_peered":"2024-09-18T08:07:26.426532+0000","last_unstale":"2024-09-18T08:10:12.298260+0000","last_undegraded":"2024-09-18T08:10:12.298260+0000","last_fullsized":"2024-09-18T08:10:12.298260+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":152,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":3,"last_scrub":"74'79","last_scrub_stamp":"2024-09-18T08:08:57.736355+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:00:11.893742+0000","last_clean_scrub_stamp":"2024-09-18T08:08:57.736355+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:10:19.314793+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,3],"acting":[0,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.3","version":"69'19","reported_seq":366,"reported_epoch":218,"state":"active+clean","last_fresh":"2024-09-18T08:10:12.296893+0000","last_change":"2024-09-18T08:04:07.231406+0000","last_active":"2024-09-18T08:10:12.296893+0000","last_peered":"2024-09-18T08:10:12.296893+0000","last_clean":"2024-09-18T08:10:12.296893+0000","last_became_active":"2024-09-18T08:02:49.890318+0000","last_became_peered":"2024-09-18T08:02:49.890318+0000","last_unstale":"2024-09-18T08:10:12.296893+0000","last_undegraded":"2024-09-18T08:10:12.296893+0000","last_fullsized":"2024-09-18T08:10:12.296893+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'19","last_scrub_stamp":"2024-09-18T08:04:07.229846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_clean_scrub_stamp":"2024-09-18T08:04:07.229846+0000","objects_scrubbed":8,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:10:19.093125+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":39,"num_read_kb":25,"num_write":22,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.4","version":"62'1","reported_seq":378,"reported_epoch":218,"state":"active+clean","last_fresh":"2024-09-18T08:10:12.301816+0000","last_change":"2024-09-18T08:10:07.593068+0000","last_active":"2024-09-18T08:10:12.301816+0000","last_peered":"2024-09-18T08:10:12.301816+0000","last_clean":"2024-09-18T08:10:12.301816+0000","last_became_active":"2024-09-18T08:08:52.688347+0000","last_became_peered":"2024-09-18T08:08:52.688347+0000","last_unstale":"2024-09-18T08:10:12.301816+0000","last_undegraded":"2024-09-18T08:10:12.301816+0000","last_fullsized":"2024-09-18T08:10:12.301816+0000","mapping_epoch":198,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":199,"parent":"0.0","parent_split_bits":0,"last_scrub":"62'1","last_scrub_stamp":"2024-09-18T08:10:07.593004+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:46.860092+0000","last_clean_scrub_stamp":"2024-09-18T08:10:07.593004+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:11:36.610081+0000","scrub_duration":26,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":436,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":872,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":260,"reported_epoch":218,"state":"active+clean","last_fresh":"2024-09-18T08:10:12.297113+0000","last_change":"2024-09-18T08:06:24.133954+0000","last_active":"2024-09-18T08:10:12.297113+0000","last_peered":"2024-09-18T08:10:12.297113+0000","last_clean":"2024-09-18T08:10:12.297113+0000","last_became_active":"2024-09-18T08:06:24.133262+0000","last_became_peered":"2024-09-18T08:06:24.133262+0000","last_unstale":"2024-09-18T08:10:12.297113+0000","last_undegraded":"2024-09-18T08:10:12.297113+0000","last_fullsized":"2024-09-18T08:10:12.297113+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:04:13.377561+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:50.886830+0000","last_clean_scrub_stamp":"2024-09-18T08:04:13.377561+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:10:22.094192+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":366,"reported_epoch":218,"state":"active+clean","last_fresh":"2024-09-18T08:10:12.301147+0000","last_change":"2024-09-18T08:08:52.688537+0000","last_active":"2024-09-18T08:10:12.301147+0000","last_peered":"2024-09-18T08:10:12.301147+0000","last_clean":"2024-09-18T08:10:12.301147+0000","last_became_active":"2024-09-18T08:08:52.688212+0000","last_became_peered":"2024-09-18T08:08:52.688212+0000","last_unstale":"2024-09-18T08:10:12.301147+0000","last_undegraded":"2024-09-18T08:10:12.301147+0000","last_fullsized":"2024-09-18T08:10:12.301147+0000","mapping_epoch":198,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":199,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:07:49.041138+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_clean_scrub_stamp":"2024-09-18T08:07:49.041138+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:10:25.679955+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.8","version":"218'9309","reported_seq":9427,"reported_epoch":218,"state":"active+clean","last_fresh":"2024-09-18T08:10:13.282627+0000","last_change":"2024-09-18T08:07:53.671548+0000","last_active":"2024-09-18T08:10:13.282627+0000","last_peered":"2024-09-18T08:10:13.282627+0000","last_clean":"2024-09-18T08:10:13.282627+0000","last_became_active":"2024-09-18T08:07:53.670635+0000","last_became_peered":"2024-09-18T08:07:53.670635+0000","last_unstale":"2024-09-18T08:10:13.282627+0000","last_undegraded":"2024-09-18T08:10:13.282627+0000","last_fullsized":"2024-09-18T08:10:13.282627+0000","mapping_epoch":165,"log_start":"196'4200","ondisk_log_start":"196'4200","created":165,"last_epoch_clean":166,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:07:52.307205+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:07:52.307205+0000","last_clean_scrub_stamp":"2024-09-18T08:07:52.307205+0000","objects_scrubbed":0,"log_size":5109,"log_dups_size":0,"ondisk_log_size":5109,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":205979648,"num_objects":3143,"num_object_clones":0,"num_object_copies":9429,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3143,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9309,"num_write_kb":398464,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.1","version":"74'79","reported_seq":155,"reported_epoch":218,"state":"active+clean","last_fresh":"2024-09-18T08:10:12.302306+0000","last_change":"2024-09-18T08:08:59.553431+0000","last_active":"2024-09-18T08:10:12.302306+0000","last_peered":"2024-09-18T08:10:12.302306+0000","last_clean":"2024-09-18T08:10:12.302306+0000","last_became_active":"2024-09-18T08:07:26.400643+0000","last_became_peered":"2024-09-18T08:07:26.400643+0000","last_unstale":"2024-09-18T08:10:12.302306+0000","last_undegraded":"2024-09-18T08:10:12.302306+0000","last_fullsized":"2024-09-18T08:10:12.302306+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":152,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":4,"last_scrub":"74'79","last_scrub_stamp":"2024-09-18T08:08:59.552592+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:00:11.893742+0000","last_clean_scrub_stamp":"2024-09-18T08:08:59.552592+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:10:15.399172+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,2],"acting":[5,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.4","version":"69'28","reported_seq":476,"reported_epoch":218,"state":"active+clean","last_fresh":"2024-09-18T08:10:12.301725+0000","last_change":"2024-09-18T08:08:57.639985+0000","last_active":"2024-09-18T08:10:12.301725+0000","last_peered":"2024-09-18T08:10:12.301725+0000","last_clean":"2024-09-18T08:10:12.301725+0000","last_became_active":"2024-09-18T08:02:49.897221+0000","last_became_peered":"2024-09-18T08:02:49.897221+0000","last_unstale":"2024-09-18T08:10:12.301725+0000","last_undegraded":"2024-09-18T08:10:12.301725+0000","last_fullsized":"2024-09-18T08:10:12.301725+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'28","last_scrub_stamp":"2024-09-18T08:08:57.639628+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_clean_scrub_stamp":"2024-09-18T08:08:57.639628+0000","objects_scrubbed":10,"log_size":28,"log_dups_size":0,"ondisk_log_size":28,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:10:27.518247+0000","scrub_duration":18,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":48,"num_read_kb":33,"num_write":26,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":84,"reported_epoch":218,"state":"active+clean","last_fresh":"2024-09-18T08:10:12.296870+0000","last_change":"2024-09-18T08:08:18.543579+0000","last_active":"2024-09-18T08:10:12.296870+0000","last_peered":"2024-09-18T08:10:12.296870+0000","last_clean":"2024-09-18T08:10:12.296870+0000","last_became_active":"2024-09-18T08:08:18.538852+0000","last_became_peered":"2024-09-18T08:08:18.538852+0000","last_unstale":"2024-09-18T08:10:12.296870+0000","last_undegraded":"2024-09-18T08:10:12.296870+0000","last_fullsized":"2024-09-18T08:10:12.296870+0000","mapping_epoch":181,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":182,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:05:39.490141+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:46.860092+0000","last_clean_scrub_stamp":"2024-09-18T08:05:39.490141+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"64'2","reported_seq":374,"reported_epoch":218,"state":"active+clean","last_fresh":"2024-09-18T08:10:12.300264+0000","last_change":"2024-09-18T08:06:24.165227+0000","last_active":"2024-09-18T08:10:12.300264+0000","last_peered":"2024-09-18T08:10:12.300264+0000","last_clean":"2024-09-18T08:10:12.300264+0000","last_became_active":"2024-09-18T08:06:24.136080+0000","last_became_peered":"2024-09-18T08:06:24.136080+0000","last_unstale":"2024-09-18T08:10:12.300264+0000","last_undegraded":"2024-09-18T08:10:12.300264+0000","last_fullsized":"2024-09-18T08:10:12.300264+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"64'2","last_scrub_stamp":"2024-09-18T08:05:44.802942+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:45.855979+0000","last_clean_scrub_stamp":"2024-09-18T08:05:44.802942+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.001348349,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":38,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":340,"reported_epoch":218,"state":"active+clean","last_fresh":"2024-09-18T08:10:12.297088+0000","last_change":"2024-09-18T08:08:18.544944+0000","last_active":"2024-09-18T08:10:12.297088+0000","last_peered":"2024-09-18T08:10:12.297088+0000","last_clean":"2024-09-18T08:10:12.297088+0000","last_became_active":"2024-09-18T08:08:18.544260+0000","last_became_peered":"2024-09-18T08:08:18.544260+0000","last_unstale":"2024-09-18T08:10:12.297088+0000","last_undegraded":"2024-09-18T08:10:12.297088+0000","last_fullsized":"2024-09-18T08:10:12.297088+0000","mapping_epoch":181,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":182,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:07:53.582095+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:50.886830+0000","last_clean_scrub_stamp":"2024-09-18T08:07:53.582095+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:10:51.897980+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":346,"reported_epoch":218,"state":"active+clean","last_fresh":"2024-09-18T08:10:12.321411+0000","last_change":"2024-09-18T08:08:18.543527+0000","last_active":"2024-09-18T08:10:12.321411+0000","last_peered":"2024-09-18T08:10:12.321411+0000","last_clean":"2024-09-18T08:10:12.321411+0000","last_became_active":"2024-09-18T08:08:18.542678+0000","last_became_peered":"2024-09-18T08:08:18.542678+0000","last_unstale":"2024-09-18T08:10:12.321411+0000","last_undegraded":"2024-09-18T08:10:12.321411+0000","last_fullsized":"2024-09-18T08:10:12.321411+0000","mapping_epoch":181,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":182,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:07:08.245063+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_clean_scrub_stamp":"2024-09-18T08:07:08.245063+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:10:56.470037+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.b","version":"218'9468","reported_seq":15097,"reported_epoch":218,"state":"active+clean","last_fresh":"2024-09-18T08:10:13.813104+0000","last_change":"2024-09-18T08:08:48.662821+0000","last_active":"2024-09-18T08:10:13.813104+0000","last_peered":"2024-09-18T08:10:13.813104+0000","last_clean":"2024-09-18T08:10:13.813104+0000","last_became_active":"2024-09-18T08:08:48.655676+0000","last_became_peered":"2024-09-18T08:08:48.655676+0000","last_unstale":"2024-09-18T08:10:13.813104+0000","last_undegraded":"2024-09-18T08:10:13.813104+0000","last_fullsized":"2024-09-18T08:10:13.813104+0000","mapping_epoch":194,"log_start":"202'5000","ondisk_log_start":"202'5000","created":165,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:07:52.307205+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:07:52.307205+0000","last_clean_scrub_stamp":"2024-09-18T08:07:52.307205+0000","objects_scrubbed":0,"log_size":4468,"log_dups_size":0,"ondisk_log_size":4468,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":209977344,"num_objects":3204,"num_object_clones":0,"num_object_copies":9612,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3204,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9468,"num_write_kb":405504,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2745,"num_bytes_recovered":179896320,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,1],"acting":[3,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"74'79","reported_seq":257,"reported_epoch":218,"state":"active+clean","last_fresh":"2024-09-18T08:10:12.296977+0000","last_change":"2024-09-18T08:08:18.524000+0000","last_active":"2024-09-18T08:10:12.296977+0000","last_peered":"2024-09-18T08:10:12.296977+0000","last_clean":"2024-09-18T08:10:12.296977+0000","last_became_active":"2024-09-18T08:08:18.522977+0000","last_became_peered":"2024-09-18T08:08:18.522977+0000","last_unstale":"2024-09-18T08:10:12.296977+0000","last_undegraded":"2024-09-18T08:10:12.296977+0000","last_fullsized":"2024-09-18T08:10:12.296977+0000","mapping_epoch":181,"log_start":"0'0","ondisk_log_start":"0'0","created":152,"last_epoch_clean":182,"parent":"0.0","parent_split_bits":4,"last_scrub":"74'79","last_scrub_stamp":"2024-09-18T08:08:09.574144+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:00:11.893742+0000","last_clean_scrub_stamp":"2024-09-18T08:08:09.574144+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,4],"acting":[0,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.7","version":"69'13","reported_seq":276,"reported_epoch":218,"state":"active+clean","last_fresh":"2024-09-18T08:10:12.302076+0000","last_change":"2024-09-18T08:04:18.164662+0000","last_active":"2024-09-18T08:10:12.302076+0000","last_peered":"2024-09-18T08:10:12.302076+0000","last_clean":"2024-09-18T08:10:12.302076+0000","last_became_active":"2024-09-18T08:04:18.144925+0000","last_became_peered":"2024-09-18T08:04:18.144925+0000","last_unstale":"2024-09-18T08:10:12.302076+0000","last_undegraded":"2024-09-18T08:10:12.302076+0000","last_fullsized":"2024-09-18T08:10:12.302076+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'13","last_scrub_stamp":"2024-09-18T08:03:50.700818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_clean_scrub_stamp":"2024-09-18T08:03:50.700818+0000","objects_scrubbed":0,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":29,"num_read_kb":18,"num_write":16,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,0],"acting":[2,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":356,"reported_epoch":218,"state":"active+clean","last_fresh":"2024-09-18T08:10:12.301515+0000","last_change":"2024-09-18T08:08:53.649127+0000","last_active":"2024-09-18T08:10:12.301515+0000","last_peered":"2024-09-18T08:10:12.301515+0000","last_clean":"2024-09-18T08:10:12.301515+0000","last_became_active":"2024-09-18T08:06:24.126334+0000","last_became_peered":"2024-09-18T08:06:24.126334+0000","last_unstale":"2024-09-18T08:10:12.301515+0000","last_undegraded":"2024-09-18T08:10:12.301515+0000","last_fullsized":"2024-09-18T08:10:12.301515+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:08:53.649079+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:46.860092+0000","last_clean_scrub_stamp":"2024-09-18T08:08:53.649079+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:10:14.347881+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"62'1","reported_seq":344,"reported_epoch":218,"state":"active+clean","last_fresh":"2024-09-18T08:10:12.298342+0000","last_change":"2024-09-18T08:04:09.732724+0000","last_active":"2024-09-18T08:10:12.298342+0000","last_peered":"2024-09-18T08:10:12.298342+0000","last_clean":"2024-09-18T08:10:12.298342+0000","last_became_active":"2024-09-18T08:02:46.868175+0000","last_became_peered":"2024-09-18T08:02:46.868175+0000","last_unstale":"2024-09-18T08:10:12.298342+0000","last_undegraded":"2024-09-18T08:10:12.298342+0000","last_fullsized":"2024-09-18T08:10:12.298342+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"62'1","last_scrub_stamp":"2024-09-18T08:04:09.732645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:45.855979+0000","last_clean_scrub_stamp":"2024-09-18T08:04:09.732645+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0.00066785499999999997,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":50,"reported_epoch":218,"state":"active+clean","last_fresh":"2024-09-18T08:10:12.301569+0000","last_change":"2024-09-18T08:08:52.690656+0000","last_active":"2024-09-18T08:10:12.301569+0000","last_peered":"2024-09-18T08:10:12.301569+0000","last_clean":"2024-09-18T08:10:12.301569+0000","last_became_active":"2024-09-18T08:08:52.690286+0000","last_became_peered":"2024-09-18T08:08:52.690286+0000","last_unstale":"2024-09-18T08:10:12.301569+0000","last_undegraded":"2024-09-18T08:10:12.301569+0000","last_fullsized":"2024-09-18T08:10:12.301569+0000","mapping_epoch":198,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":199,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:07:08.806748+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:50.886830+0000","last_clean_scrub_stamp":"2024-09-18T08:07:08.806748+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":353,"reported_epoch":218,"state":"active+clean","last_fresh":"2024-09-18T08:10:12.301774+0000","last_change":"2024-09-18T08:09:42.647525+0000","last_active":"2024-09-18T08:10:12.301774+0000","last_peered":"2024-09-18T08:10:12.301774+0000","last_clean":"2024-09-18T08:10:12.301774+0000","last_became_active":"2024-09-18T08:06:24.145382+0000","last_became_peered":"2024-09-18T08:06:24.145382+0000","last_unstale":"2024-09-18T08:10:12.301774+0000","last_undegraded":"2024-09-18T08:10:12.301774+0000","last_fullsized":"2024-09-18T08:10:12.301774+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:09:42.647317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_clean_scrub_stamp":"2024-09-18T08:09:42.647317+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:11:03.015274+0000","scrub_duration":26,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.a","version":"218'9457","reported_seq":9575,"reported_epoch":218,"state":"active+clean","last_fresh":"2024-09-18T08:10:13.691092+0000","last_change":"2024-09-18T08:07:53.326651+0000","last_active":"2024-09-18T08:10:13.691092+0000","last_peered":"2024-09-18T08:10:13.691092+0000","last_clean":"2024-09-18T08:10:13.691092+0000","last_became_active":"2024-09-18T08:07:53.326361+0000","last_became_peered":"2024-09-18T08:07:53.326361+0000","last_unstale":"2024-09-18T08:10:13.691092+0000","last_undegraded":"2024-09-18T08:10:13.691092+0000","last_fullsized":"2024-09-18T08:10:13.691092+0000","mapping_epoch":165,"log_start":"200'4600","ondisk_log_start":"200'4600","created":165,"last_epoch_clean":166,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:07:52.307205+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:07:52.307205+0000","last_clean_scrub_stamp":"2024-09-18T08:07:52.307205+0000","objects_scrubbed":0,"log_size":4857,"log_dups_size":0,"ondisk_log_size":4857,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:10:20.719923+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":204931072,"num_objects":3127,"num_object_clones":0,"num_object_copies":9381,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3127,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9457,"num_write_kb":402688,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.3","version":"74'79","reported_seq":177,"reported_epoch":218,"state":"active+clean","last_fresh":"2024-09-18T08:10:12.301722+0000","last_change":"2024-09-18T08:09:41.651800+0000","last_active":"2024-09-18T08:10:12.301722+0000","last_peered":"2024-09-18T08:10:12.301722+0000","last_clean":"2024-09-18T08:10:12.301722+0000","last_became_active":"2024-09-18T08:08:18.538929+0000","last_became_peered":"2024-09-18T08:08:18.538929+0000","last_unstale":"2024-09-18T08:10:12.301722+0000","last_undegraded":"2024-09-18T08:10:12.301722+0000","last_fullsized":"2024-09-18T08:10:12.301722+0000","mapping_epoch":181,"log_start":"0'0","ondisk_log_start":"0'0","created":152,"last_epoch_clean":182,"parent":"0.0","parent_split_bits":3,"last_scrub":"74'79","last_scrub_stamp":"2024-09-18T08:09:41.651729+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:00:11.893742+0000","last_clean_scrub_stamp":"2024-09-18T08:09:41.651729+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:10:55.722378+0000","scrub_duration":17,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.6","version":"69'12","reported_seq":398,"reported_epoch":218,"state":"active+clean","last_fresh":"2024-09-18T08:10:12.298702+0000","last_change":"2024-09-18T08:08:57.021693+0000","last_active":"2024-09-18T08:10:12.298702+0000","last_peered":"2024-09-18T08:10:12.298702+0000","last_clean":"2024-09-18T08:10:12.298702+0000","last_became_active":"2024-09-18T08:08:52.700130+0000","last_became_peered":"2024-09-18T08:08:52.700130+0000","last_unstale":"2024-09-18T08:10:12.298702+0000","last_undegraded":"2024-09-18T08:10:12.298702+0000","last_fullsized":"2024-09-18T08:10:12.298702+0000","mapping_epoch":198,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":200,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'12","last_scrub_stamp":"2024-09-18T08:07:39.603562+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_clean_scrub_stamp":"2024-09-18T08:07:39.603562+0000","objects_scrubbed":6,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:10:57.784539+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":18,"num_read_kb":12,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":364,"reported_epoch":218,"state":"active+clean","last_fresh":"2024-09-18T08:10:12.302400+0000","last_change":"2024-09-18T08:09:34.089548+0000","last_active":"2024-09-18T08:10:12.302400+0000","last_peered":"2024-09-18T08:10:12.302400+0000","last_clean":"2024-09-18T08:10:12.302400+0000","last_became_active":"2024-09-18T08:08:18.523135+0000","last_became_peered":"2024-09-18T08:08:18.523135+0000","last_unstale":"2024-09-18T08:10:12.302400+0000","last_undegraded":"2024-09-18T08:10:12.302400+0000","last_fullsized":"2024-09-18T08:10:12.302400+0000","mapping_epoch":181,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":182,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:09:34.089476+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:46.860092+0000","last_clean_scrub_stamp":"2024-09-18T08:09:34.089476+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:10:41.071862+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"70'5","reported_seq":427,"reported_epoch":218,"state":"active+clean","last_fresh":"2024-09-18T08:10:12.301417+0000","last_change":"2024-09-18T08:09:09.649261+0000","last_active":"2024-09-18T08:10:12.301417+0000","last_peered":"2024-09-18T08:10:12.301417+0000","last_clean":"2024-09-18T08:10:12.301417+0000","last_became_active":"2024-09-18T08:06:46.141667+0000","last_became_peered":"2024-09-18T08:06:46.141667+0000","last_unstale":"2024-09-18T08:10:12.301417+0000","last_undegraded":"2024-09-18T08:10:12.301417+0000","last_fullsized":"2024-09-18T08:10:12.301417+0000","mapping_epoch":143,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":144,"parent":"0.0","parent_split_bits":0,"last_scrub":"70'5","last_scrub_stamp":"2024-09-18T08:09:09.649188+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:45.855979+0000","last_clean_scrub_stamp":"2024-09-18T08:09:09.649188+0000","objects_scrubbed":1,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:10:24.340859+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0.000504055,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":213,"num_read_kb":208,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":342,"reported_epoch":218,"state":"active+clean","last_fresh":"2024-09-18T08:10:12.303892+0000","last_change":"2024-09-18T08:07:20.767916+0000","last_active":"2024-09-18T08:10:12.303892+0000","last_peered":"2024-09-18T08:10:12.303892+0000","last_clean":"2024-09-18T08:10:12.303892+0000","last_became_active":"2024-09-18T08:02:51.913112+0000","last_became_peered":"2024-09-18T08:02:51.913112+0000","last_unstale":"2024-09-18T08:10:12.303892+0000","last_undegraded":"2024-09-18T08:10:12.303892+0000","last_fullsized":"2024-09-18T08:10:12.303892+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:07:20.767841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:50.886830+0000","last_clean_scrub_stamp":"2024-09-18T08:07:20.767841+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":323,"reported_epoch":218,"state":"active+clean","last_fresh":"2024-09-18T08:10:12.301503+0000","last_change":"2024-09-18T08:09:49.702132+0000","last_active":"2024-09-18T08:10:12.301503+0000","last_peered":"2024-09-18T08:10:12.301503+0000","last_clean":"2024-09-18T08:10:12.301503+0000","last_became_active":"2024-09-18T08:02:53.907404+0000","last_became_peered":"2024-09-18T08:02:53.907404+0000","last_unstale":"2024-09-18T08:10:12.301503+0000","last_undegraded":"2024-09-18T08:10:12.301503+0000","last_fullsized":"2024-09-18T08:10:12.301503+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:09:49.702055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_clean_scrub_stamp":"2024-09-18T08:09:49.702055+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:11:01.853786+0000","scrub_duration":23,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.9","version":"218'9080","reported_seq":9198,"reported_epoch":218,"state":"active+clean","last_fresh":"2024-09-18T08:10:13.237366+0000","last_change":"2024-09-18T08:07:53.672294+0000","last_active":"2024-09-18T08:10:13.237366+0000","last_peered":"2024-09-18T08:10:13.237366+0000","last_clean":"2024-09-18T08:10:13.237366+0000","last_became_active":"2024-09-18T08:07:53.671994+0000","last_became_peered":"2024-09-18T08:07:53.671994+0000","last_unstale":"2024-09-18T08:10:13.237366+0000","last_undegraded":"2024-09-18T08:10:13.237366+0000","last_fullsized":"2024-09-18T08:10:13.237366+0000","mapping_epoch":165,"log_start":"192'3900","ondisk_log_start":"192'3900","created":165,"last_epoch_clean":166,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:07:52.307205+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:07:52.307205+0000","last_clean_scrub_stamp":"2024-09-18T08:07:52.307205+0000","objects_scrubbed":0,"log_size":5180,"log_dups_size":0,"ondisk_log_size":5180,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":205389824,"num_objects":3134,"num_object_clones":0,"num_object_copies":9402,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3134,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9080,"num_write_kb":390848,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.5","version":"182'21","reported_seq":126,"reported_epoch":218,"state":"active+clean","last_fresh":"2024-09-18T08:10:12.297042+0000","last_change":"2024-09-18T08:08:18.609652+0000","last_active":"2024-09-18T08:10:12.297042+0000","last_peered":"2024-09-18T08:10:12.297042+0000","last_clean":"2024-09-18T08:10:12.297042+0000","last_became_active":"2024-09-18T08:08:18.547555+0000","last_became_peered":"2024-09-18T08:08:18.547555+0000","last_unstale":"2024-09-18T08:10:12.297042+0000","last_undegraded":"2024-09-18T08:10:12.297042+0000","last_fullsized":"2024-09-18T08:10:12.297042+0000","mapping_epoch":181,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":182,"parent":"0.0","parent_split_bits":0,"last_scrub":"130'18","last_scrub_stamp":"2024-09-18T08:06:15.455503+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_clean_scrub_stamp":"2024-09-18T08:06:15.455503+0000","objects_scrubbed":0,"log_size":21,"log_dups_size":0,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":24,"num_read_kb":14,"num_write":13,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":154,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,7],"acting":[4,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":357,"reported_epoch":218,"state":"active+clean","last_fresh":"2024-09-18T08:10:12.321597+0000","last_change":"2024-09-18T08:09:45.572278+0000","last_active":"2024-09-18T08:10:12.321597+0000","last_peered":"2024-09-18T08:10:12.321597+0000","last_clean":"2024-09-18T08:10:12.321597+0000","last_became_active":"2024-09-18T08:06:24.133425+0000","last_became_peered":"2024-09-18T08:06:24.133425+0000","last_unstale":"2024-09-18T08:10:12.321597+0000","last_undegraded":"2024-09-18T08:10:12.321597+0000","last_fullsized":"2024-09-18T08:10:12.321597+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:09:45.572201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:46.860092+0000","last_clean_scrub_stamp":"2024-09-18T08:09:45.572201+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:11:00.131734+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"74'79","reported_seq":438,"reported_epoch":218,"state":"active+clean","last_fresh":"2024-09-18T08:10:12.300963+0000","last_change":"2024-09-18T08:08:57.769104+0000","last_active":"2024-09-18T08:10:12.300963+0000","last_peered":"2024-09-18T08:10:12.300963+0000","last_clean":"2024-09-18T08:10:12.300963+0000","last_became_active":"2024-09-18T08:07:24.376931+0000","last_became_peered":"2024-09-18T08:07:24.376931+0000","last_unstale":"2024-09-18T08:10:12.300963+0000","last_undegraded":"2024-09-18T08:10:12.300963+0000","last_fullsized":"2024-09-18T08:10:12.300963+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":153,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'79","last_scrub_stamp":"2024-09-18T08:08:57.769048+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:00:11.893742+0000","last_clean_scrub_stamp":"2024-09-18T08:08:57.769048+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:10:19.097626+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":170,"reported_epoch":218,"state":"active+clean","last_fresh":"2024-09-18T08:10:12.298625+0000","last_change":"2024-09-18T08:07:09.744206+0000","last_active":"2024-09-18T08:10:12.298625+0000","last_peered":"2024-09-18T08:10:12.298625+0000","last_clean":"2024-09-18T08:10:12.298625+0000","last_became_active":"2024-09-18T08:06:46.141046+0000","last_became_peered":"2024-09-18T08:06:46.141046+0000","last_unstale":"2024-09-18T08:10:12.298625+0000","last_undegraded":"2024-09-18T08:10:12.298625+0000","last_fullsized":"2024-09-18T08:10:12.298625+0000","mapping_epoch":143,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":144,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:07:09.744137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:50.886830+0000","last_clean_scrub_stamp":"2024-09-18T08:07:09.744137+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":345,"reported_epoch":218,"state":"active+clean","last_fresh":"2024-09-18T08:10:12.302883+0000","last_change":"2024-09-18T08:08:18.548620+0000","last_active":"2024-09-18T08:10:12.302883+0000","last_peered":"2024-09-18T08:10:12.302883+0000","last_clean":"2024-09-18T08:10:12.302883+0000","last_became_active":"2024-09-18T08:08:18.548233+0000","last_became_peered":"2024-09-18T08:08:18.548233+0000","last_unstale":"2024-09-18T08:10:12.302883+0000","last_undegraded":"2024-09-18T08:10:12.302883+0000","last_fullsized":"2024-09-18T08:10:12.302883+0000","mapping_epoch":181,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":182,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:06:50.710565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_clean_scrub_stamp":"2024-09-18T08:06:50.710565+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.2","version":"218'9324","reported_seq":13718,"reported_epoch":218,"state":"active+clean","last_fresh":"2024-09-18T08:10:13.181776+0000","last_change":"2024-09-18T08:08:58.743574+0000","last_active":"2024-09-18T08:10:13.181776+0000","last_peered":"2024-09-18T08:10:13.181776+0000","last_clean":"2024-09-18T08:10:13.181776+0000","last_became_active":"2024-09-18T08:08:58.738458+0000","last_became_peered":"2024-09-18T08:08:58.738458+0000","last_unstale":"2024-09-18T08:10:13.181776+0000","last_undegraded":"2024-09-18T08:10:13.181776+0000","last_fullsized":"2024-09-18T08:10:13.181776+0000","mapping_epoch":201,"log_start":"202'4800","ondisk_log_start":"202'4800","created":165,"last_epoch_clean":202,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:07:52.307205+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:07:52.307205+0000","last_clean_scrub_stamp":"2024-09-18T08:07:52.307205+0000","objects_scrubbed":0,"log_size":4524,"log_dups_size":0,"ondisk_log_size":4524,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":212860928,"num_objects":3248,"num_object_clones":0,"num_object_copies":9744,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3248,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9324,"num_write_kb":402304,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2129,"num_bytes_recovered":139526144,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,5],"acting":[4,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.e","version":"69'11","reported_seq":96,"reported_epoch":218,"state":"active+clean","last_fresh":"2024-09-18T08:10:12.297143+0000","last_change":"2024-09-18T08:08:18.594670+0000","last_active":"2024-09-18T08:10:12.297143+0000","last_peered":"2024-09-18T08:10:12.297143+0000","last_clean":"2024-09-18T08:10:12.297143+0000","last_became_active":"2024-09-18T08:08:18.544581+0000","last_became_peered":"2024-09-18T08:08:18.544581+0000","last_unstale":"2024-09-18T08:10:12.297143+0000","last_undegraded":"2024-09-18T08:10:12.297143+0000","last_fullsized":"2024-09-18T08:10:12.297143+0000","mapping_epoch":181,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":182,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'11","last_scrub_stamp":"2024-09-18T08:06:14.472168+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_clean_scrub_stamp":"2024-09-18T08:06:14.472168+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":19,"num_read_kb":12,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":84,"reported_epoch":218,"state":"active+clean","last_fresh":"2024-09-18T08:10:12.296628+0000","last_change":"2024-09-18T08:08:18.524516+0000","last_active":"2024-09-18T08:10:12.296628+0000","last_peered":"2024-09-18T08:10:12.296628+0000","last_clean":"2024-09-18T08:10:12.296628+0000","last_became_active":"2024-09-18T08:08:18.524229+0000","last_became_peered":"2024-09-18T08:08:18.524229+0000","last_unstale":"2024-09-18T08:10:12.296628+0000","last_undegraded":"2024-09-18T08:10:12.296628+0000","last_fullsized":"2024-09-18T08:10:12.296628+0000","mapping_epoch":181,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":182,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:02:46.860092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:46.860092+0000","last_clean_scrub_stamp":"2024-09-18T08:02:46.860092+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":362,"reported_epoch":218,"state":"active+clean","last_fresh":"2024-09-18T08:10:12.302657+0000","last_change":"2024-09-18T08:08:18.547880+0000","last_active":"2024-09-18T08:10:12.302657+0000","last_peered":"2024-09-18T08:10:12.302657+0000","last_clean":"2024-09-18T08:10:12.302657+0000","last_became_active":"2024-09-18T08:08:18.547105+0000","last_became_peered":"2024-09-18T08:08:18.547105+0000","last_unstale":"2024-09-18T08:10:12.302657+0000","last_undegraded":"2024-09-18T08:10:12.302657+0000","last_fullsized":"2024-09-18T08:10:12.302657+0000","mapping_epoch":181,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":182,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:07:51.742090+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:50.886830+0000","last_clean_scrub_stamp":"2024-09-18T08:07:51.742090+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":345,"reported_epoch":218,"state":"active+clean","last_fresh":"2024-09-18T08:10:12.320898+0000","last_change":"2024-09-18T08:07:09.197912+0000","last_active":"2024-09-18T08:10:12.320898+0000","last_peered":"2024-09-18T08:10:12.320898+0000","last_clean":"2024-09-18T08:10:12.320898+0000","last_became_active":"2024-09-18T08:06:24.130266+0000","last_became_peered":"2024-09-18T08:06:24.130266+0000","last_unstale":"2024-09-18T08:10:12.320898+0000","last_undegraded":"2024-09-18T08:10:12.320898+0000","last_fullsized":"2024-09-18T08:10:12.320898+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:07:09.197841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_clean_scrub_stamp":"2024-09-18T08:07:09.197841+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:10:58.431705+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.1","version":"218'9505","reported_seq":9689,"reported_epoch":218,"state":"active+clean","last_fresh":"2024-09-18T08:10:13.694272+0000","last_change":"2024-09-18T08:09:40.130643+0000","last_active":"2024-09-18T08:10:13.694272+0000","last_peered":"2024-09-18T08:10:13.694272+0000","last_clean":"2024-09-18T08:10:13.694272+0000","last_became_active":"2024-09-18T08:07:53.327987+0000","last_became_peered":"2024-09-18T08:07:53.327987+0000","last_unstale":"2024-09-18T08:10:13.694272+0000","last_undegraded":"2024-09-18T08:10:13.694272+0000","last_fullsized":"2024-09-18T08:10:13.694272+0000","mapping_epoch":165,"log_start":"202'4700","ondisk_log_start":"202'4700","created":165,"last_epoch_clean":166,"parent":"0.0","parent_split_bits":0,"last_scrub":"213'7038","last_scrub_stamp":"2024-09-18T08:09:40.130353+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:07:52.307205+0000","last_clean_scrub_stamp":"2024-09-18T08:09:40.130353+0000","objects_scrubbed":5709,"log_size":4805,"log_dups_size":0,"ondisk_log_size":4805,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:11:06.670120+0000","scrub_duration":2501,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":207945728,"num_objects":3173,"num_object_clones":0,"num_object_copies":9519,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3173,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9505,"num_write_kb":405696,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.8","version":"74'79","reported_seq":162,"reported_epoch":218,"state":"active+clean","last_fresh":"2024-09-18T08:10:12.300131+0000","last_change":"2024-09-18T08:09:00.533593+0000","last_active":"2024-09-18T08:10:12.300131+0000","last_peered":"2024-09-18T08:10:12.300131+0000","last_clean":"2024-09-18T08:10:12.300131+0000","last_became_active":"2024-09-18T08:07:26.407298+0000","last_became_peered":"2024-09-18T08:07:26.407298+0000","last_unstale":"2024-09-18T08:10:12.300131+0000","last_undegraded":"2024-09-18T08:10:12.300131+0000","last_fullsized":"2024-09-18T08:10:12.300131+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":152,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":4,"last_scrub":"74'79","last_scrub_stamp":"2024-09-18T08:09:00.532813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:00:11.893742+0000","last_clean_scrub_stamp":"2024-09-18T08:09:00.532813+0000","objects_scrubbed":1,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:10:17.887770+0000","scrub_duration":25,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262688,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,3],"acting":[5,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.d","version":"69'17","reported_seq":102,"reported_epoch":218,"state":"active+clean","last_fresh":"2024-09-18T08:10:12.297161+0000","last_change":"2024-09-18T08:08:18.588624+0000","last_active":"2024-09-18T08:10:12.297161+0000","last_peered":"2024-09-18T08:10:12.297161+0000","last_clean":"2024-09-18T08:10:12.297161+0000","last_became_active":"2024-09-18T08:08:18.540862+0000","last_became_peered":"2024-09-18T08:08:18.540862+0000","last_unstale":"2024-09-18T08:10:12.297161+0000","last_undegraded":"2024-09-18T08:10:12.297161+0000","last_fullsized":"2024-09-18T08:10:12.297161+0000","mapping_epoch":181,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":182,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'17","last_scrub_stamp":"2024-09-18T08:08:09.671248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_clean_scrub_stamp":"2024-09-18T08:08:09.671248+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":28,"num_read_kb":18,"num_write":18,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,1],"acting":[4,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":84,"reported_epoch":218,"state":"active+clean","last_fresh":"2024-09-18T08:10:12.296196+0000","last_change":"2024-09-18T08:08:18.541029+0000","last_active":"2024-09-18T08:10:12.296196+0000","last_peered":"2024-09-18T08:10:12.296196+0000","last_clean":"2024-09-18T08:10:12.296196+0000","last_became_active":"2024-09-18T08:08:18.540285+0000","last_became_peered":"2024-09-18T08:08:18.540285+0000","last_unstale":"2024-09-18T08:10:12.296196+0000","last_undegraded":"2024-09-18T08:10:12.296196+0000","last_fullsized":"2024-09-18T08:10:12.296196+0000","mapping_epoch":181,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":182,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:05:12.622901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:46.860092+0000","last_clean_scrub_stamp":"2024-09-18T08:05:12.622901+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":356,"reported_epoch":218,"state":"active+clean","last_fresh":"2024-09-18T08:10:12.301603+0000","last_change":"2024-09-18T08:09:06.610955+0000","last_active":"2024-09-18T08:10:12.301603+0000","last_peered":"2024-09-18T08:10:12.301603+0000","last_clean":"2024-09-18T08:10:12.301603+0000","last_became_active":"2024-09-18T08:08:18.623014+0000","last_became_peered":"2024-09-18T08:08:18.623014+0000","last_unstale":"2024-09-18T08:10:12.301603+0000","last_undegraded":"2024-09-18T08:10:12.301603+0000","last_fullsized":"2024-09-18T08:10:12.301603+0000","mapping_epoch":181,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":182,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:09:06.610879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:50.886830+0000","last_clean_scrub_stamp":"2024-09-18T08:09:06.610879+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:10:19.831235+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":50,"reported_epoch":218,"state":"active+clean","last_fresh":"2024-09-18T08:10:12.301981+0000","last_change":"2024-09-18T08:08:52.696713+0000","last_active":"2024-09-18T08:10:12.301981+0000","last_peered":"2024-09-18T08:10:12.301981+0000","last_clean":"2024-09-18T08:10:12.301981+0000","last_became_active":"2024-09-18T08:08:52.696300+0000","last_became_peered":"2024-09-18T08:08:52.696300+0000","last_unstale":"2024-09-18T08:10:12.301981+0000","last_undegraded":"2024-09-18T08:10:12.301981+0000","last_fullsized":"2024-09-18T08:10:12.301981+0000","mapping_epoch":198,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":199,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:04:06.709631+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_clean_scrub_stamp":"2024-09-18T08:04:06.709631+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"8.0","version":"218'9352","reported_seq":9470,"reported_epoch":218,"state":"active+clean","last_fresh":"2024-09-18T08:10:13.767229+0000","last_change":"2024-09-18T08:07:53.329182+0000","last_active":"2024-09-18T08:10:13.767229+0000","last_peered":"2024-09-18T08:10:13.767229+0000","last_clean":"2024-09-18T08:10:13.767229+0000","last_became_active":"2024-09-18T08:07:53.328914+0000","last_became_peered":"2024-09-18T08:07:53.328914+0000","last_unstale":"2024-09-18T08:10:13.767229+0000","last_undegraded":"2024-09-18T08:10:13.767229+0000","last_fullsized":"2024-09-18T08:10:13.767229+0000","mapping_epoch":165,"log_start":"202'4900","ondisk_log_start":"202'4900","created":165,"last_epoch_clean":166,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:07:52.307205+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:07:52.307205+0000","last_clean_scrub_stamp":"2024-09-18T08:07:52.307205+0000","objects_scrubbed":0,"log_size":4452,"log_dups_size":0,"ondisk_log_size":4452,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":209584128,"num_objects":3198,"num_object_clones":0,"num_object_copies":9594,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3198,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9352,"num_write_kb":401600,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.9","version":"74'79","reported_seq":460,"reported_epoch":218,"state":"active+clean","last_fresh":"2024-09-18T08:10:12.300845+0000","last_change":"2024-09-18T08:09:27.676247+0000","last_active":"2024-09-18T08:10:12.300845+0000","last_peered":"2024-09-18T08:10:12.300845+0000","last_clean":"2024-09-18T08:10:12.300845+0000","last_became_active":"2024-09-18T08:08:52.691492+0000","last_became_peered":"2024-09-18T08:08:52.691492+0000","last_unstale":"2024-09-18T08:10:12.300845+0000","last_undegraded":"2024-09-18T08:10:12.300845+0000","last_fullsized":"2024-09-18T08:10:12.300845+0000","mapping_epoch":198,"log_start":"0'0","ondisk_log_start":"0'0","created":152,"last_epoch_clean":199,"parent":"0.0","parent_split_bits":4,"last_scrub":"74'79","last_scrub_stamp":"2024-09-18T08:09:27.676169+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:00:11.893742+0000","last_clean_scrub_stamp":"2024-09-18T08:09:27.676169+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:10:36.422152+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,2],"acting":[7,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.c","version":"69'10","reported_seq":96,"reported_epoch":218,"state":"active+clean","last_fresh":"2024-09-18T08:10:12.296663+0000","last_change":"2024-09-18T08:08:18.642452+0000","last_active":"2024-09-18T08:10:12.296663+0000","last_peered":"2024-09-18T08:10:12.296663+0000","last_clean":"2024-09-18T08:10:12.296663+0000","last_became_active":"2024-09-18T08:08:18.622980+0000","last_became_peered":"2024-09-18T08:08:18.622980+0000","last_unstale":"2024-09-18T08:10:12.296663+0000","last_undegraded":"2024-09-18T08:10:12.296663+0000","last_fullsized":"2024-09-18T08:10:12.296663+0000","mapping_epoch":181,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":182,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'10","last_scrub_stamp":"2024-09-18T08:05:41.474813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_clean_scrub_stamp":"2024-09-18T08:05:41.474813+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":14,"num_read_kb":9,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":357,"reported_epoch":218,"state":"active+clean","last_fresh":"2024-09-18T08:10:12.322263+0000","last_change":"2024-09-18T08:08:18.542800+0000","last_active":"2024-09-18T08:10:12.322263+0000","last_peered":"2024-09-18T08:10:12.322263+0000","last_clean":"2024-09-18T08:10:12.322263+0000","last_became_active":"2024-09-18T08:08:18.542424+0000","last_became_peered":"2024-09-18T08:08:18.542424+0000","last_unstale":"2024-09-18T08:10:12.322263+0000","last_undegraded":"2024-09-18T08:10:12.322263+0000","last_fullsized":"2024-09-18T08:10:12.322263+0000","mapping_epoch":181,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":182,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:06:46.339428+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:46.860092+0000","last_clean_scrub_stamp":"2024-09-18T08:06:46.339428+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:10:53.446520+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.d","version":"69'8","reported_seq":433,"reported_epoch":218,"state":"active+clean","last_fresh":"2024-09-18T08:10:12.300406+0000","last_change":"2024-09-18T08:05:33.027988+0000","last_active":"2024-09-18T08:10:12.300406+0000","last_peered":"2024-09-18T08:10:12.300406+0000","last_clean":"2024-09-18T08:10:12.300406+0000","last_became_active":"2024-09-18T08:02:51.909011+0000","last_became_peered":"2024-09-18T08:02:51.909011+0000","last_unstale":"2024-09-18T08:10:12.300406+0000","last_undegraded":"2024-09-18T08:10:12.300406+0000","last_fullsized":"2024-09-18T08:10:12.300406+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'8","last_scrub_stamp":"2024-09-18T08:05:33.027783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:50.886830+0000","last_clean_scrub_stamp":"2024-09-18T08:05:33.027783+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":91,"reported_epoch":218,"state":"active+clean","last_fresh":"2024-09-18T08:10:12.297344+0000","last_change":"2024-09-18T08:08:52.685954+0000","last_active":"2024-09-18T08:10:12.297344+0000","last_peered":"2024-09-18T08:10:12.297344+0000","last_clean":"2024-09-18T08:10:12.297344+0000","last_became_active":"2024-09-18T08:08:52.685603+0000","last_became_peered":"2024-09-18T08:08:52.685603+0000","last_unstale":"2024-09-18T08:10:12.297344+0000","last_undegraded":"2024-09-18T08:10:12.297344+0000","last_fullsized":"2024-09-18T08:10:12.297344+0000","mapping_epoch":198,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":199,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:07:16.559597+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_clean_scrub_stamp":"2024-09-18T08:07:16.559597+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.7","version":"218'9394","reported_seq":9581,"reported_epoch":218,"state":"active+clean","last_fresh":"2024-09-18T08:10:13.691543+0000","last_change":"2024-09-18T08:09:36.764450+0000","last_active":"2024-09-18T08:10:13.691543+0000","last_peered":"2024-09-18T08:10:13.691543+0000","last_clean":"2024-09-18T08:10:13.691543+0000","last_became_active":"2024-09-18T08:07:53.674433+0000","last_became_peered":"2024-09-18T08:07:53.674433+0000","last_unstale":"2024-09-18T08:10:13.691543+0000","last_undegraded":"2024-09-18T08:10:13.691543+0000","last_fullsized":"2024-09-18T08:10:13.691543+0000","mapping_epoch":165,"log_start":"202'4600","ondisk_log_start":"202'4600","created":165,"last_epoch_clean":166,"parent":"0.0","parent_split_bits":0,"last_scrub":"213'6848","last_scrub_stamp":"2024-09-18T08:09:36.764206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:07:52.307205+0000","last_clean_scrub_stamp":"2024-09-18T08:09:36.764206+0000","objects_scrubbed":5822,"log_size":4794,"log_dups_size":0,"ondisk_log_size":4794,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":4,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:11:06.120132+0000","scrub_duration":3143,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":208404480,"num_objects":3180,"num_object_clones":0,"num_object_copies":9540,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3180,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9394,"num_write_kb":402368,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,6],"acting":[1,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.b","version":"69'9","reported_seq":388,"reported_epoch":218,"state":"active+clean","last_fresh":"2024-09-18T08:10:12.296834+0000","last_change":"2024-09-18T08:08:18.640380+0000","last_active":"2024-09-18T08:10:12.296834+0000","last_peered":"2024-09-18T08:10:12.296834+0000","last_clean":"2024-09-18T08:10:12.296834+0000","last_became_active":"2024-09-18T08:08:18.522995+0000","last_became_peered":"2024-09-18T08:08:18.522995+0000","last_unstale":"2024-09-18T08:10:12.296834+0000","last_undegraded":"2024-09-18T08:10:12.296834+0000","last_fullsized":"2024-09-18T08:10:12.296834+0000","mapping_epoch":181,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":182,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'9","last_scrub_stamp":"2024-09-18T08:08:18.640292+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_clean_scrub_stamp":"2024-09-18T08:08:18.640292+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":356,"reported_epoch":218,"state":"active+clean","last_fresh":"2024-09-18T08:10:12.303182+0000","last_change":"2024-09-18T08:07:40.708363+0000","last_active":"2024-09-18T08:10:12.303182+0000","last_peered":"2024-09-18T08:10:12.303182+0000","last_clean":"2024-09-18T08:10:12.303182+0000","last_became_active":"2024-09-18T08:06:24.135005+0000","last_became_peered":"2024-09-18T08:06:24.135005+0000","last_unstale":"2024-09-18T08:10:12.303182+0000","last_undegraded":"2024-09-18T08:10:12.303182+0000","last_fullsized":"2024-09-18T08:10:12.303182+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:07:40.708307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:46.860092+0000","last_clean_scrub_stamp":"2024-09-18T08:07:40.708307+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":357,"reported_epoch":218,"state":"active+clean","last_fresh":"2024-09-18T08:10:12.298230+0000","last_change":"2024-09-18T08:08:18.509651+0000","last_active":"2024-09-18T08:10:12.298230+0000","last_peered":"2024-09-18T08:10:12.298230+0000","last_clean":"2024-09-18T08:10:12.298230+0000","last_became_active":"2024-09-18T08:08:18.509247+0000","last_became_peered":"2024-09-18T08:08:18.509247+0000","last_unstale":"2024-09-18T08:10:12.298230+0000","last_undegraded":"2024-09-18T08:10:12.298230+0000","last_fullsized":"2024-09-18T08:10:12.298230+0000","mapping_epoch":181,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":182,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:07:10.728679+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:50.886830+0000","last_clean_scrub_stamp":"2024-09-18T08:07:10.728679+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,3],"acting":[2,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":353,"reported_epoch":218,"state":"active+clean","last_fresh":"2024-09-18T08:10:12.302288+0000","last_change":"2024-09-18T08:09:28.086782+0000","last_active":"2024-09-18T08:10:12.302288+0000","last_peered":"2024-09-18T08:10:12.302288+0000","last_clean":"2024-09-18T08:10:12.302288+0000","last_became_active":"2024-09-18T08:08:52.697337+0000","last_became_peered":"2024-09-18T08:08:52.697337+0000","last_unstale":"2024-09-18T08:10:12.302288+0000","last_undegraded":"2024-09-18T08:10:12.302288+0000","last_fullsized":"2024-09-18T08:10:12.302288+0000","mapping_epoch":198,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":199,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:09:28.086727+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_clean_scrub_stamp":"2024-09-18T08:09:28.086727+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:10:33.905647+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.6","version":"218'9335","reported_seq":9494,"reported_epoch":218,"state":"active+clean","last_fresh":"2024-09-18T08:10:13.813059+0000","last_change":"2024-09-18T08:10:12.178469+0000","last_active":"2024-09-18T08:10:13.813059+0000","last_peered":"2024-09-18T08:10:13.813059+0000","last_clean":"2024-09-18T08:10:13.813059+0000","last_became_active":"2024-09-18T08:07:53.674139+0000","last_became_peered":"2024-09-18T08:07:53.674139+0000","last_unstale":"2024-09-18T08:10:13.813059+0000","last_undegraded":"2024-09-18T08:10:13.813059+0000","last_fullsized":"2024-09-18T08:10:13.813059+0000","mapping_epoch":165,"log_start":"192'3900","ondisk_log_start":"192'3900","created":165,"last_epoch_clean":166,"parent":"0.0","parent_split_bits":0,"last_scrub":"217'9226","last_scrub_stamp":"2024-09-18T08:10:12.178143+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:07:52.307205+0000","last_clean_scrub_stamp":"2024-09-18T08:10:12.178143+0000","objects_scrubbed":3346,"log_size":5435,"log_dups_size":0,"ondisk_log_size":5435,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:11:40.476010+0000","scrub_duration":1358,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":209518592,"num_objects":3197,"num_object_clones":0,"num_object_copies":9591,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3197,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9335,"num_write_kb":401024,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,7],"acting":[0,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.a","version":"69'19","reported_seq":359,"reported_epoch":218,"state":"active+clean","last_fresh":"2024-09-18T08:10:12.303304+0000","last_change":"2024-09-18T08:07:41.000210+0000","last_active":"2024-09-18T08:10:12.303304+0000","last_peered":"2024-09-18T08:10:12.303304+0000","last_clean":"2024-09-18T08:10:12.303304+0000","last_became_active":"2024-09-18T08:06:24.139909+0000","last_became_peered":"2024-09-18T08:06:24.139909+0000","last_unstale":"2024-09-18T08:10:12.303304+0000","last_undegraded":"2024-09-18T08:10:12.303304+0000","last_fullsized":"2024-09-18T08:10:12.303304+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'19","last_scrub_stamp":"2024-09-18T08:07:41.000146+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_clean_scrub_stamp":"2024-09-18T08:07:41.000146+0000","objects_scrubbed":9,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:10:18.661451+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":31,"num_read_kb":20,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,7],"acting":[1,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":344,"reported_epoch":218,"state":"active+clean","last_fresh":"2024-09-18T08:10:12.302557+0000","last_change":"2024-09-18T08:06:47.638974+0000","last_active":"2024-09-18T08:10:12.302557+0000","last_peered":"2024-09-18T08:10:12.302557+0000","last_clean":"2024-09-18T08:10:12.302557+0000","last_became_active":"2024-09-18T08:06:46.171298+0000","last_became_peered":"2024-09-18T08:06:46.171298+0000","last_unstale":"2024-09-18T08:10:12.302557+0000","last_undegraded":"2024-09-18T08:10:12.302557+0000","last_fullsized":"2024-09-18T08:10:12.302557+0000","mapping_epoch":143,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":144,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:06:47.638911+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:46.860092+0000","last_clean_scrub_stamp":"2024-09-18T08:06:47.638911+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":50,"reported_epoch":218,"state":"active+clean","last_fresh":"2024-09-18T08:10:12.298268+0000","last_change":"2024-09-18T08:08:52.698243+0000","last_active":"2024-09-18T08:10:12.298268+0000","last_peered":"2024-09-18T08:10:12.298268+0000","last_clean":"2024-09-18T08:10:12.298268+0000","last_became_active":"2024-09-18T08:08:52.697932+0000","last_became_peered":"2024-09-18T08:08:52.697932+0000","last_unstale":"2024-09-18T08:10:12.298268+0000","last_undegraded":"2024-09-18T08:10:12.298268+0000","last_fullsized":"2024-09-18T08:10:12.298268+0000","mapping_epoch":198,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":199,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:08:16.631007+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:50.886830+0000","last_clean_scrub_stamp":"2024-09-18T08:08:16.631007+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":170,"reported_epoch":218,"state":"active+clean","last_fresh":"2024-09-18T08:10:12.299354+0000","last_change":"2024-09-18T08:06:50.902917+0000","last_active":"2024-09-18T08:10:12.299354+0000","last_peered":"2024-09-18T08:10:12.299354+0000","last_clean":"2024-09-18T08:10:12.299354+0000","last_became_active":"2024-09-18T08:06:46.137872+0000","last_became_peered":"2024-09-18T08:06:46.137872+0000","last_unstale":"2024-09-18T08:10:12.299354+0000","last_undegraded":"2024-09-18T08:10:12.299354+0000","last_fullsized":"2024-09-18T08:10:12.299354+0000","mapping_epoch":143,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":144,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:06:50.902835+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_clean_scrub_stamp":"2024-09-18T08:06:50.902835+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,3],"acting":[2,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"8.5","version":"218'9330","reported_seq":13760,"reported_epoch":218,"state":"active+clean","last_fresh":"2024-09-18T08:10:13.138420+0000","last_change":"2024-09-18T08:09:13.170209+0000","last_active":"2024-09-18T08:10:13.138420+0000","last_peered":"2024-09-18T08:10:13.138420+0000","last_clean":"2024-09-18T08:10:13.138420+0000","last_became_active":"2024-09-18T08:09:13.166459+0000","last_became_peered":"2024-09-18T08:09:13.166459+0000","last_unstale":"2024-09-18T08:10:13.138420+0000","last_undegraded":"2024-09-18T08:10:13.138420+0000","last_fullsized":"2024-09-18T08:10:13.138420+0000","mapping_epoch":205,"log_start":"202'4774","ondisk_log_start":"202'4774","created":165,"last_epoch_clean":206,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:07:52.307205+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:07:52.307205+0000","last_clean_scrub_stamp":"2024-09-18T08:07:52.307205+0000","objects_scrubbed":0,"log_size":4556,"log_dups_size":0,"ondisk_log_size":4556,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":213647360,"num_objects":3260,"num_object_clones":0,"num_object_copies":9780,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3260,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9330,"num_write_kb":402880,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2146,"num_bytes_recovered":140640256,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,7],"acting":[4,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.9","version":"69'12","reported_seq":96,"reported_epoch":218,"state":"active+clean","last_fresh":"2024-09-18T08:10:12.296168+0000","last_change":"2024-09-18T08:08:18.599669+0000","last_active":"2024-09-18T08:10:12.296168+0000","last_peered":"2024-09-18T08:10:12.296168+0000","last_clean":"2024-09-18T08:10:12.296168+0000","last_became_active":"2024-09-18T08:08:18.544489+0000","last_became_peered":"2024-09-18T08:08:18.544489+0000","last_unstale":"2024-09-18T08:10:12.296168+0000","last_undegraded":"2024-09-18T08:10:12.296168+0000","last_fullsized":"2024-09-18T08:10:12.296168+0000","mapping_epoch":181,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":182,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'12","last_scrub_stamp":"2024-09-18T08:05:33.544135+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_clean_scrub_stamp":"2024-09-18T08:05:33.544135+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":24,"num_read_kb":15,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":372,"reported_epoch":218,"state":"active+clean","last_fresh":"2024-09-18T08:10:12.304474+0000","last_change":"2024-09-18T08:09:39.614251+0000","last_active":"2024-09-18T08:10:12.304474+0000","last_peered":"2024-09-18T08:10:12.304474+0000","last_clean":"2024-09-18T08:10:12.304474+0000","last_became_active":"2024-09-18T08:08:18.539573+0000","last_became_peered":"2024-09-18T08:08:18.539573+0000","last_unstale":"2024-09-18T08:10:12.304474+0000","last_undegraded":"2024-09-18T08:10:12.304474+0000","last_fullsized":"2024-09-18T08:10:12.304474+0000","mapping_epoch":181,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":182,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:09:39.614187+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:46.860092+0000","last_clean_scrub_stamp":"2024-09-18T08:09:39.614187+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:10:57.546531+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":335,"reported_epoch":218,"state":"active+clean","last_fresh":"2024-09-18T08:10:12.299439+0000","last_change":"2024-09-18T08:08:15.744330+0000","last_active":"2024-09-18T08:10:12.299439+0000","last_peered":"2024-09-18T08:10:12.299439+0000","last_clean":"2024-09-18T08:10:12.299439+0000","last_became_active":"2024-09-18T08:02:51.902163+0000","last_became_peered":"2024-09-18T08:02:51.902163+0000","last_unstale":"2024-09-18T08:10:12.299439+0000","last_undegraded":"2024-09-18T08:10:12.299439+0000","last_fullsized":"2024-09-18T08:10:12.299439+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:08:15.744245+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:50.886830+0000","last_clean_scrub_stamp":"2024-09-18T08:08:15.744245+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":74,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":332,"reported_epoch":218,"state":"active+clean","last_fresh":"2024-09-18T08:10:12.321950+0000","last_change":"2024-09-18T08:07:44.113695+0000","last_active":"2024-09-18T08:10:12.321950+0000","last_peered":"2024-09-18T08:10:12.321950+0000","last_clean":"2024-09-18T08:10:12.321950+0000","last_became_active":"2024-09-18T08:02:53.907412+0000","last_became_peered":"2024-09-18T08:02:53.907412+0000","last_unstale":"2024-09-18T08:10:12.321950+0000","last_undegraded":"2024-09-18T08:10:12.321950+0000","last_fullsized":"2024-09-18T08:10:12.321950+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:07:44.113639+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_clean_scrub_stamp":"2024-09-18T08:07:44.113639+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.4","version":"218'9441","reported_seq":23698,"reported_epoch":218,"state":"active+recovering+undersized+remapped","last_fresh":"2024-09-18T08:10:13.301970+0000","last_change":"2024-09-18T08:08:56.896339+0000","last_active":"2024-09-18T08:10:13.301970+0000","last_peered":"2024-09-18T08:10:13.301970+0000","last_clean":"2024-09-18T08:08:51.672524+0000","last_became_active":"2024-09-18T08:08:53.828580+0000","last_became_peered":"2024-09-18T08:08:53.828580+0000","last_unstale":"2024-09-18T08:10:13.301970+0000","last_undegraded":"2024-09-18T08:10:13.301970+0000","last_fullsized":"2024-09-18T08:08:53.682963+0000","mapping_epoch":199,"log_start":"196'4300","ondisk_log_start":"196'4300","created":165,"last_epoch_clean":190,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:07:52.307205+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:07:52.307205+0000","last_clean_scrub_stamp":"2024-09-18T08:07:52.307205+0000","objects_scrubbed":0,"log_size":5141,"log_dups_size":0,"ondisk_log_size":5141,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":205062144,"num_objects":3129,"num_object_clones":0,"num_object_copies":9387,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":51,"num_objects_unfound":0,"num_objects_dirty":3129,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9441,"num_write_kb":402240,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7056,"num_bytes_recovered":462422016,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,2],"acting":[5,4],"avail_no_missing":["5","3","4","6"],"object_location_counts":[{"shards":"4,5","objects":3078},{"shards":"4,5,6","objects":51}],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.8","version":"69'15","reported_seq":401,"reported_epoch":218,"state":"active+clean","last_fresh":"2024-09-18T08:10:12.300304+0000","last_change":"2024-09-18T08:06:24.162625+0000","last_active":"2024-09-18T08:10:12.300304+0000","last_peered":"2024-09-18T08:10:12.300304+0000","last_clean":"2024-09-18T08:10:12.300304+0000","last_became_active":"2024-09-18T08:06:24.138259+0000","last_became_peered":"2024-09-18T08:06:24.138259+0000","last_unstale":"2024-09-18T08:10:12.300304+0000","last_undegraded":"2024-09-18T08:10:12.300304+0000","last_fullsized":"2024-09-18T08:10:12.300304+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'15","last_scrub_stamp":"2024-09-18T08:04:26.859044+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_clean_scrub_stamp":"2024-09-18T08:04:26.859044+0000","objects_scrubbed":7,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.f","version":"62'2","reported_seq":388,"reported_epoch":218,"state":"active+clean","last_fresh":"2024-09-18T08:10:12.302408+0000","last_change":"2024-09-18T08:08:18.629506+0000","last_active":"2024-09-18T08:10:12.302408+0000","last_peered":"2024-09-18T08:10:12.302408+0000","last_clean":"2024-09-18T08:10:12.302408+0000","last_became_active":"2024-09-18T08:08:18.524593+0000","last_became_peered":"2024-09-18T08:08:18.524593+0000","last_unstale":"2024-09-18T08:10:12.302408+0000","last_undegraded":"2024-09-18T08:10:12.302408+0000","last_fullsized":"2024-09-18T08:10:12.302408+0000","mapping_epoch":181,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":182,"parent":"0.0","parent_split_bits":0,"last_scrub":"62'2","last_scrub_stamp":"2024-09-18T08:07:43.540222+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:46.860092+0000","last_clean_scrub_stamp":"2024-09-18T08:07:43.540222+0000","objects_scrubbed":2,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:10:52.700103+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":4,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":184,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.9","version":"182'12","reported_seq":496,"reported_epoch":218,"state":"active+clean","last_fresh":"2024-09-18T08:10:12.300796+0000","last_change":"2024-09-18T08:09:46.742515+0000","last_active":"2024-09-18T08:10:12.300796+0000","last_peered":"2024-09-18T08:10:12.300796+0000","last_clean":"2024-09-18T08:10:12.300796+0000","last_became_active":"2024-09-18T08:08:18.524858+0000","last_became_peered":"2024-09-18T08:08:18.524858+0000","last_unstale":"2024-09-18T08:10:12.300796+0000","last_undegraded":"2024-09-18T08:10:12.300796+0000","last_fullsized":"2024-09-18T08:10:12.300796+0000","mapping_epoch":181,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":182,"parent":"0.0","parent_split_bits":0,"last_scrub":"182'12","last_scrub_stamp":"2024-09-18T08:09:46.742442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:50.886830+0000","last_clean_scrub_stamp":"2024-09-18T08:09:46.742442+0000","objects_scrubbed":1,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:11:07.487575+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":344,"reported_epoch":218,"state":"active+clean","last_fresh":"2024-09-18T08:10:12.300068+0000","last_change":"2024-09-18T08:07:45.707631+0000","last_active":"2024-09-18T08:10:12.300068+0000","last_peered":"2024-09-18T08:10:12.300068+0000","last_clean":"2024-09-18T08:10:12.300068+0000","last_became_active":"2024-09-18T08:06:24.133119+0000","last_became_peered":"2024-09-18T08:06:24.133119+0000","last_unstale":"2024-09-18T08:10:12.300068+0000","last_undegraded":"2024-09-18T08:10:12.300068+0000","last_fullsized":"2024-09-18T08:10:12.300068+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:07:45.707562+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_clean_scrub_stamp":"2024-09-18T08:07:45.707562+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":260,"reported_epoch":218,"state":"active+clean","last_fresh":"2024-09-18T08:10:12.296649+0000","last_change":"2024-09-18T08:06:24.133955+0000","last_active":"2024-09-18T08:10:12.296649+0000","last_peered":"2024-09-18T08:10:12.296649+0000","last_clean":"2024-09-18T08:10:12.296649+0000","last_became_active":"2024-09-18T08:06:24.133644+0000","last_became_peered":"2024-09-18T08:06:24.133644+0000","last_unstale":"2024-09-18T08:10:12.296649+0000","last_undegraded":"2024-09-18T08:10:12.296649+0000","last_fullsized":"2024-09-18T08:10:12.296649+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:04:16.378205+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:46.860092+0000","last_clean_scrub_stamp":"2024-09-18T08:04:16.378205+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:10:20.052218+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.17","version":"69'6","reported_seq":356,"reported_epoch":218,"state":"active+clean","last_fresh":"2024-09-18T08:10:12.322833+0000","last_change":"2024-09-18T08:07:19.309941+0000","last_active":"2024-09-18T08:10:12.322833+0000","last_peered":"2024-09-18T08:10:12.322833+0000","last_clean":"2024-09-18T08:10:12.322833+0000","last_became_active":"2024-09-18T08:02:49.895067+0000","last_became_peered":"2024-09-18T08:02:49.895067+0000","last_unstale":"2024-09-18T08:10:12.322833+0000","last_undegraded":"2024-09-18T08:10:12.322833+0000","last_fullsized":"2024-09-18T08:10:12.322833+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'6","last_scrub_stamp":"2024-09-18T08:07:19.309844+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_clean_scrub_stamp":"2024-09-18T08:07:19.309844+0000","objects_scrubbed":3,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":26,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":9,"num_read_kb":6,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":334,"reported_epoch":218,"state":"active+clean","last_fresh":"2024-09-18T08:10:12.302266+0000","last_change":"2024-09-18T08:05:45.812436+0000","last_active":"2024-09-18T08:10:12.302266+0000","last_peered":"2024-09-18T08:10:12.302266+0000","last_clean":"2024-09-18T08:10:12.302266+0000","last_became_active":"2024-09-18T08:02:51.912557+0000","last_became_peered":"2024-09-18T08:02:51.912557+0000","last_unstale":"2024-09-18T08:10:12.302266+0000","last_undegraded":"2024-09-18T08:10:12.302266+0000","last_fullsized":"2024-09-18T08:10:12.302266+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:05:45.812389+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:50.886830+0000","last_clean_scrub_stamp":"2024-09-18T08:05:45.812389+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,1],"acting":[5,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":367,"reported_epoch":218,"state":"active+clean","last_fresh":"2024-09-18T08:10:12.301704+0000","last_change":"2024-09-18T08:08:18.526113+0000","last_active":"2024-09-18T08:10:12.301704+0000","last_peered":"2024-09-18T08:10:12.301704+0000","last_clean":"2024-09-18T08:10:12.301704+0000","last_became_active":"2024-09-18T08:08:18.525260+0000","last_became_peered":"2024-09-18T08:08:18.525260+0000","last_unstale":"2024-09-18T08:10:12.301704+0000","last_undegraded":"2024-09-18T08:10:12.301704+0000","last_fullsized":"2024-09-18T08:10:12.301704+0000","mapping_epoch":181,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":182,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:07:53.670107+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_clean_scrub_stamp":"2024-09-18T08:07:53.670107+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:10:19.732636+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"69'9","reported_seq":362,"reported_epoch":218,"state":"active+clean","last_fresh":"2024-09-18T08:10:12.298782+0000","last_change":"2024-09-18T08:09:44.116395+0000","last_active":"2024-09-18T08:10:12.298782+0000","last_peered":"2024-09-18T08:10:12.298782+0000","last_clean":"2024-09-18T08:10:12.298782+0000","last_became_active":"2024-09-18T08:02:49.891567+0000","last_became_peered":"2024-09-18T08:02:49.891567+0000","last_unstale":"2024-09-18T08:10:12.298782+0000","last_undegraded":"2024-09-18T08:10:12.298782+0000","last_fullsized":"2024-09-18T08:10:12.298782+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'9","last_scrub_stamp":"2024-09-18T08:09:44.116308+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_clean_scrub_stamp":"2024-09-18T08:09:44.116308+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:11:07.916395+0000","scrub_duration":43,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":84,"reported_epoch":218,"state":"active+clean","last_fresh":"2024-09-18T08:10:12.296268+0000","last_change":"2024-09-18T08:08:18.550531+0000","last_active":"2024-09-18T08:10:12.296268+0000","last_peered":"2024-09-18T08:10:12.296268+0000","last_clean":"2024-09-18T08:10:12.296268+0000","last_became_active":"2024-09-18T08:08:18.545953+0000","last_became_peered":"2024-09-18T08:08:18.545953+0000","last_unstale":"2024-09-18T08:10:12.296268+0000","last_undegraded":"2024-09-18T08:10:12.296268+0000","last_fullsized":"2024-09-18T08:10:12.296268+0000","mapping_epoch":181,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":182,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:06:13.214705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:46.860092+0000","last_clean_scrub_stamp":"2024-09-18T08:06:13.214705+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":335,"reported_epoch":218,"state":"active+clean","last_fresh":"2024-09-18T08:10:12.322623+0000","last_change":"2024-09-18T08:09:46.560057+0000","last_active":"2024-09-18T08:10:12.322623+0000","last_peered":"2024-09-18T08:10:12.322623+0000","last_clean":"2024-09-18T08:10:12.322623+0000","last_became_active":"2024-09-18T08:02:51.907361+0000","last_became_peered":"2024-09-18T08:02:51.907361+0000","last_unstale":"2024-09-18T08:10:12.322623+0000","last_undegraded":"2024-09-18T08:10:12.322623+0000","last_fullsized":"2024-09-18T08:10:12.322623+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:09:46.557349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:50.886830+0000","last_clean_scrub_stamp":"2024-09-18T08:09:46.557349+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:11:08.318311+0000","scrub_duration":22,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":346,"reported_epoch":218,"state":"active+clean","last_fresh":"2024-09-18T08:10:12.298532+0000","last_change":"2024-09-18T08:08:18.511719+0000","last_active":"2024-09-18T08:10:12.298532+0000","last_peered":"2024-09-18T08:10:12.298532+0000","last_clean":"2024-09-18T08:10:12.298532+0000","last_became_active":"2024-09-18T08:08:18.511019+0000","last_became_peered":"2024-09-18T08:08:18.511019+0000","last_unstale":"2024-09-18T08:10:12.298532+0000","last_undegraded":"2024-09-18T08:10:12.298532+0000","last_fullsized":"2024-09-18T08:10:12.298532+0000","mapping_epoch":181,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":182,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:08:10.660508+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_clean_scrub_stamp":"2024-09-18T08:08:10.660508+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.15","version":"69'9","reported_seq":354,"reported_epoch":218,"state":"active+clean","last_fresh":"2024-09-18T08:10:12.302649+0000","last_change":"2024-09-18T08:05:36.896299+0000","last_active":"2024-09-18T08:10:12.302649+0000","last_peered":"2024-09-18T08:10:12.302649+0000","last_clean":"2024-09-18T08:10:12.302649+0000","last_became_active":"2024-09-18T08:02:49.890497+0000","last_became_peered":"2024-09-18T08:02:49.890497+0000","last_unstale":"2024-09-18T08:10:12.302649+0000","last_undegraded":"2024-09-18T08:10:12.302649+0000","last_fullsized":"2024-09-18T08:10:12.302649+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'9","last_scrub_stamp":"2024-09-18T08:05:36.896238+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_clean_scrub_stamp":"2024-09-18T08:05:36.896238+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":358,"reported_epoch":218,"state":"active+clean","last_fresh":"2024-09-18T08:10:12.297242+0000","last_change":"2024-09-18T08:09:46.035501+0000","last_active":"2024-09-18T08:10:12.297242+0000","last_peered":"2024-09-18T08:10:12.297242+0000","last_clean":"2024-09-18T08:10:12.297242+0000","last_became_active":"2024-09-18T08:02:47.875983+0000","last_became_peered":"2024-09-18T08:02:47.875983+0000","last_unstale":"2024-09-18T08:10:12.297242+0000","last_undegraded":"2024-09-18T08:10:12.297242+0000","last_fullsized":"2024-09-18T08:10:12.297242+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:09:46.035332+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:46.860092+0000","last_clean_scrub_stamp":"2024-09-18T08:09:46.035332+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:10:47.519129+0000","scrub_duration":18,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.14","version":"199'12","reported_seq":465,"reported_epoch":218,"state":"active+clean","last_fresh":"2024-09-18T08:10:12.321107+0000","last_change":"2024-09-18T08:10:10.449130+0000","last_active":"2024-09-18T08:10:12.321107+0000","last_peered":"2024-09-18T08:10:12.321107+0000","last_clean":"2024-09-18T08:10:12.321107+0000","last_became_active":"2024-09-18T08:08:52.696201+0000","last_became_peered":"2024-09-18T08:08:52.696201+0000","last_unstale":"2024-09-18T08:10:12.321107+0000","last_undegraded":"2024-09-18T08:10:12.321107+0000","last_fullsized":"2024-09-18T08:10:12.321107+0000","mapping_epoch":198,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":199,"parent":"0.0","parent_split_bits":0,"last_scrub":"199'12","last_scrub_stamp":"2024-09-18T08:10:10.449024+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:50.886830+0000","last_clean_scrub_stamp":"2024-09-18T08:10:10.449024+0000","objects_scrubbed":1,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:11:36.934868+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":84,"reported_epoch":218,"state":"active+clean","last_fresh":"2024-09-18T08:10:12.297150+0000","last_change":"2024-09-18T08:08:18.523205+0000","last_active":"2024-09-18T08:10:12.297150+0000","last_peered":"2024-09-18T08:10:12.297150+0000","last_clean":"2024-09-18T08:10:12.297150+0000","last_became_active":"2024-09-18T08:08:18.522483+0000","last_became_peered":"2024-09-18T08:08:18.522483+0000","last_unstale":"2024-09-18T08:10:12.297150+0000","last_undegraded":"2024-09-18T08:10:12.297150+0000","last_fullsized":"2024-09-18T08:10:12.297150+0000","mapping_epoch":181,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":182,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:06:43.510513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_clean_scrub_stamp":"2024-09-18T08:06:43.510513+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.14","version":"69'10","reported_seq":373,"reported_epoch":218,"state":"active+clean","last_fresh":"2024-09-18T08:10:12.322005+0000","last_change":"2024-09-18T08:08:08.493614+0000","last_active":"2024-09-18T08:10:12.322005+0000","last_peered":"2024-09-18T08:10:12.322005+0000","last_clean":"2024-09-18T08:10:12.322005+0000","last_became_active":"2024-09-18T08:04:18.150968+0000","last_became_peered":"2024-09-18T08:04:18.150968+0000","last_unstale":"2024-09-18T08:10:12.322005+0000","last_undegraded":"2024-09-18T08:10:12.322005+0000","last_fullsized":"2024-09-18T08:10:12.322005+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'10","last_scrub_stamp":"2024-09-18T08:08:08.493522+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_clean_scrub_stamp":"2024-09-18T08:08:08.493522+0000","objects_scrubbed":5,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,7],"acting":[3,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":364,"reported_epoch":218,"state":"active+clean","last_fresh":"2024-09-18T08:10:12.301354+0000","last_change":"2024-09-18T08:09:31.658337+0000","last_active":"2024-09-18T08:10:12.301354+0000","last_peered":"2024-09-18T08:10:12.301354+0000","last_clean":"2024-09-18T08:10:12.301354+0000","last_became_active":"2024-09-18T08:08:18.524218+0000","last_became_peered":"2024-09-18T08:08:18.524218+0000","last_unstale":"2024-09-18T08:10:12.301354+0000","last_undegraded":"2024-09-18T08:10:12.301354+0000","last_fullsized":"2024-09-18T08:10:12.301354+0000","mapping_epoch":181,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":182,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:09:31.658259+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:46.860092+0000","last_clean_scrub_stamp":"2024-09-18T08:09:31.658259+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:10:50.143994+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.15","version":"69'8","reported_seq":440,"reported_epoch":218,"state":"active+clean","last_fresh":"2024-09-18T08:10:12.302467+0000","last_change":"2024-09-18T08:06:45.555513+0000","last_active":"2024-09-18T08:10:12.302467+0000","last_peered":"2024-09-18T08:10:12.302467+0000","last_clean":"2024-09-18T08:10:12.302467+0000","last_became_active":"2024-09-18T08:02:51.914135+0000","last_became_peered":"2024-09-18T08:02:51.914135+0000","last_unstale":"2024-09-18T08:10:12.302467+0000","last_undegraded":"2024-09-18T08:10:12.302467+0000","last_fullsized":"2024-09-18T08:10:12.302467+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'8","last_scrub_stamp":"2024-09-18T08:06:45.555459+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:50.886830+0000","last_clean_scrub_stamp":"2024-09-18T08:06:45.555459+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":339,"reported_epoch":218,"state":"active+clean","last_fresh":"2024-09-18T08:10:12.298341+0000","last_change":"2024-09-18T08:09:42.047337+0000","last_active":"2024-09-18T08:10:12.298341+0000","last_peered":"2024-09-18T08:10:12.298341+0000","last_clean":"2024-09-18T08:10:12.298341+0000","last_became_active":"2024-09-18T08:02:53.912112+0000","last_became_peered":"2024-09-18T08:02:53.912112+0000","last_unstale":"2024-09-18T08:10:12.298341+0000","last_undegraded":"2024-09-18T08:10:12.298341+0000","last_fullsized":"2024-09-18T08:10:12.298341+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:09:42.047270+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_clean_scrub_stamp":"2024-09-18T08:09:42.047270+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:10:59.660034+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.13","version":"69'11","reported_seq":96,"reported_epoch":218,"state":"active+clean","last_fresh":"2024-09-18T08:10:12.296671+0000","last_change":"2024-09-18T08:08:18.643772+0000","last_active":"2024-09-18T08:10:12.296671+0000","last_peered":"2024-09-18T08:10:12.296671+0000","last_clean":"2024-09-18T08:10:12.296671+0000","last_became_active":"2024-09-18T08:08:18.539712+0000","last_became_peered":"2024-09-18T08:08:18.539712+0000","last_unstale":"2024-09-18T08:10:12.296671+0000","last_undegraded":"2024-09-18T08:10:12.296671+0000","last_fullsized":"2024-09-18T08:10:12.296671+0000","mapping_epoch":181,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":182,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'11","last_scrub_stamp":"2024-09-18T08:07:53.824401+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_clean_scrub_stamp":"2024-09-18T08:07:53.824401+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":19,"num_read_kb":12,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":84,"reported_epoch":218,"state":"active+clean","last_fresh":"2024-09-18T08:10:12.297009+0000","last_change":"2024-09-18T08:08:18.623643+0000","last_active":"2024-09-18T08:10:12.297009+0000","last_peered":"2024-09-18T08:10:12.297009+0000","last_clean":"2024-09-18T08:10:12.297009+0000","last_became_active":"2024-09-18T08:08:18.622893+0000","last_became_peered":"2024-09-18T08:08:18.622893+0000","last_unstale":"2024-09-18T08:10:12.297009+0000","last_undegraded":"2024-09-18T08:10:12.297009+0000","last_fullsized":"2024-09-18T08:10:12.297009+0000","mapping_epoch":181,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":182,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:06:46.502422+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:46.860092+0000","last_clean_scrub_stamp":"2024-09-18T08:06:46.502422+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":342,"reported_epoch":218,"state":"active+clean","last_fresh":"2024-09-18T08:10:12.303215+0000","last_change":"2024-09-18T08:09:03.582984+0000","last_active":"2024-09-18T08:10:12.303215+0000","last_peered":"2024-09-18T08:10:12.303215+0000","last_clean":"2024-09-18T08:10:12.303215+0000","last_became_active":"2024-09-18T08:02:51.910752+0000","last_became_peered":"2024-09-18T08:02:51.910752+0000","last_unstale":"2024-09-18T08:10:12.303215+0000","last_undegraded":"2024-09-18T08:10:12.303215+0000","last_fullsized":"2024-09-18T08:10:12.303215+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:09:03.582914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:50.886830+0000","last_clean_scrub_stamp":"2024-09-18T08:09:03.582914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:10:14.069633+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":340,"reported_epoch":218,"state":"active+clean","last_fresh":"2024-09-18T08:10:12.320427+0000","last_change":"2024-09-18T08:07:54.314026+0000","last_active":"2024-09-18T08:10:12.320427+0000","last_peered":"2024-09-18T08:10:12.320427+0000","last_clean":"2024-09-18T08:10:12.320427+0000","last_became_active":"2024-09-18T08:02:53.911429+0000","last_became_peered":"2024-09-18T08:02:53.911429+0000","last_unstale":"2024-09-18T08:10:12.320427+0000","last_undegraded":"2024-09-18T08:10:12.320427+0000","last_fullsized":"2024-09-18T08:10:12.320427+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:07:54.313971+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_clean_scrub_stamp":"2024-09-18T08:07:54.313971+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.12","version":"69'9","reported_seq":388,"reported_epoch":218,"state":"active+clean","last_fresh":"2024-09-18T08:10:12.303514+0000","last_change":"2024-09-18T08:10:06.582865+0000","last_active":"2024-09-18T08:10:12.303514+0000","last_peered":"2024-09-18T08:10:12.303514+0000","last_clean":"2024-09-18T08:10:12.303514+0000","last_became_active":"2024-09-18T08:06:24.132270+0000","last_became_peered":"2024-09-18T08:06:24.132270+0000","last_unstale":"2024-09-18T08:10:12.303514+0000","last_undegraded":"2024-09-18T08:10:12.303514+0000","last_fullsized":"2024-09-18T08:10:12.303514+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'9","last_scrub_stamp":"2024-09-18T08:10:06.582804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_clean_scrub_stamp":"2024-09-18T08:10:06.582804+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:11:13.834917+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":372,"reported_epoch":218,"state":"active+clean","last_fresh":"2024-09-18T08:10:12.301895+0000","last_change":"2024-09-18T08:09:37.532792+0000","last_active":"2024-09-18T08:10:12.301895+0000","last_peered":"2024-09-18T08:10:12.301895+0000","last_clean":"2024-09-18T08:10:12.301895+0000","last_became_active":"2024-09-18T08:08:18.525527+0000","last_became_peered":"2024-09-18T08:08:18.525527+0000","last_unstale":"2024-09-18T08:10:12.301895+0000","last_undegraded":"2024-09-18T08:10:12.301895+0000","last_fullsized":"2024-09-18T08:10:12.301895+0000","mapping_epoch":181,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":182,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:09:37.532573+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:46.860092+0000","last_clean_scrub_stamp":"2024-09-18T08:09:37.532573+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:11:05.489513+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":335,"reported_epoch":218,"state":"active+clean","last_fresh":"2024-09-18T08:10:12.322033+0000","last_change":"2024-09-18T08:07:42.165434+0000","last_active":"2024-09-18T08:10:12.322033+0000","last_peered":"2024-09-18T08:10:12.322033+0000","last_clean":"2024-09-18T08:10:12.322033+0000","last_became_active":"2024-09-18T08:02:51.906277+0000","last_became_peered":"2024-09-18T08:02:51.906277+0000","last_unstale":"2024-09-18T08:10:12.322033+0000","last_undegraded":"2024-09-18T08:10:12.322033+0000","last_fullsized":"2024-09-18T08:10:12.322033+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:07:42.165388+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:50.886830+0000","last_clean_scrub_stamp":"2024-09-18T08:07:42.165388+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.10","version":"69'1","reported_seq":319,"reported_epoch":218,"state":"active+clean","last_fresh":"2024-09-18T08:10:12.296376+0000","last_change":"2024-09-18T08:05:51.429631+0000","last_active":"2024-09-18T08:10:12.296376+0000","last_peered":"2024-09-18T08:10:12.296376+0000","last_clean":"2024-09-18T08:10:12.296376+0000","last_became_active":"2024-09-18T08:02:53.912699+0000","last_became_peered":"2024-09-18T08:02:53.912699+0000","last_unstale":"2024-09-18T08:10:12.296376+0000","last_undegraded":"2024-09-18T08:10:12.296376+0000","last_fullsized":"2024-09-18T08:10:12.296376+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_clean_scrub_stamp":"2024-09-18T08:02:52.890475+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:10:16.084523+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.11","version":"69'11","reported_seq":404,"reported_epoch":218,"state":"active+clean","last_fresh":"2024-09-18T08:10:12.320624+0000","last_change":"2024-09-18T08:07:18.282247+0000","last_active":"2024-09-18T08:10:12.320624+0000","last_peered":"2024-09-18T08:10:12.320624+0000","last_clean":"2024-09-18T08:10:12.320624+0000","last_became_active":"2024-09-18T08:06:24.132416+0000","last_became_peered":"2024-09-18T08:06:24.132416+0000","last_unstale":"2024-09-18T08:10:12.320624+0000","last_undegraded":"2024-09-18T08:10:12.320624+0000","last_fullsized":"2024-09-18T08:10:12.320624+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'11","last_scrub_stamp":"2024-09-18T08:07:18.282155+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_clean_scrub_stamp":"2024-09-18T08:07:18.282155+0000","objects_scrubbed":5,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":342,"reported_epoch":218,"state":"active+clean","last_fresh":"2024-09-18T08:10:12.302361+0000","last_change":"2024-09-18T08:06:18.168254+0000","last_active":"2024-09-18T08:10:12.302361+0000","last_peered":"2024-09-18T08:10:12.302361+0000","last_clean":"2024-09-18T08:10:12.302361+0000","last_became_active":"2024-09-18T08:02:47.878886+0000","last_became_peered":"2024-09-18T08:02:47.878886+0000","last_unstale":"2024-09-18T08:10:12.302361+0000","last_undegraded":"2024-09-18T08:10:12.302361+0000","last_fullsized":"2024-09-18T08:10:12.302361+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:06:18.168206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:46.860092+0000","last_clean_scrub_stamp":"2024-09-18T08:06:18.168206+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":378,"reported_epoch":218,"state":"active+clean","last_fresh":"2024-09-18T08:10:12.300490+0000","last_change":"2024-09-18T08:09:49.824264+0000","last_active":"2024-09-18T08:10:12.300490+0000","last_peered":"2024-09-18T08:10:12.300490+0000","last_clean":"2024-09-18T08:10:12.300490+0000","last_became_active":"2024-09-18T08:08:18.524966+0000","last_became_peered":"2024-09-18T08:08:18.524966+0000","last_unstale":"2024-09-18T08:10:12.300490+0000","last_undegraded":"2024-09-18T08:10:12.300490+0000","last_fullsized":"2024-09-18T08:10:12.300490+0000","mapping_epoch":181,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":182,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:09:49.824196+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:50.886830+0000","last_clean_scrub_stamp":"2024-09-18T08:09:49.824196+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:10:51.928894+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":345,"reported_epoch":218,"state":"active+clean","last_fresh":"2024-09-18T08:10:12.322174+0000","last_change":"2024-09-18T08:07:48.218010+0000","last_active":"2024-09-18T08:10:12.322174+0000","last_peered":"2024-09-18T08:10:12.322174+0000","last_clean":"2024-09-18T08:10:12.322174+0000","last_became_active":"2024-09-18T08:06:24.128582+0000","last_became_peered":"2024-09-18T08:06:24.128582+0000","last_unstale":"2024-09-18T08:10:12.322174+0000","last_undegraded":"2024-09-18T08:10:12.322174+0000","last_fullsized":"2024-09-18T08:10:12.322174+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:07:48.217964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_clean_scrub_stamp":"2024-09-18T08:07:48.217964+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.10","version":"69'4","reported_seq":371,"reported_epoch":218,"state":"active+clean","last_fresh":"2024-09-18T08:10:12.320399+0000","last_change":"2024-09-18T08:07:41.210418+0000","last_active":"2024-09-18T08:10:12.320399+0000","last_peered":"2024-09-18T08:10:12.320399+0000","last_clean":"2024-09-18T08:10:12.320399+0000","last_became_active":"2024-09-18T08:06:24.131309+0000","last_became_peered":"2024-09-18T08:06:24.131309+0000","last_unstale":"2024-09-18T08:10:12.320399+0000","last_undegraded":"2024-09-18T08:10:12.320399+0000","last_fullsized":"2024-09-18T08:10:12.320399+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'4","last_scrub_stamp":"2024-09-18T08:07:41.210347+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_clean_scrub_stamp":"2024-09-18T08:07:41.210347+0000","objects_scrubbed":2,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":6,"num_read_kb":4,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":358,"reported_epoch":218,"state":"active+clean","last_fresh":"2024-09-18T08:10:13.820205+0000","last_change":"2024-09-18T08:10:13.820205+0000","last_active":"2024-09-18T08:10:13.820205+0000","last_peered":"2024-09-18T08:10:13.820205+0000","last_clean":"2024-09-18T08:10:13.820205+0000","last_became_active":"2024-09-18T08:02:47.877616+0000","last_became_peered":"2024-09-18T08:02:47.877616+0000","last_unstale":"2024-09-18T08:10:13.820205+0000","last_undegraded":"2024-09-18T08:10:13.820205+0000","last_fullsized":"2024-09-18T08:10:13.820205+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:10:13.820142+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:46.860092+0000","last_clean_scrub_stamp":"2024-09-18T08:10:13.820142+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:11:32.099137+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":84,"reported_epoch":218,"state":"active+clean","last_fresh":"2024-09-18T08:10:12.296507+0000","last_change":"2024-09-18T08:08:18.550244+0000","last_active":"2024-09-18T08:10:12.296507+0000","last_peered":"2024-09-18T08:10:12.296507+0000","last_clean":"2024-09-18T08:10:12.296507+0000","last_became_active":"2024-09-18T08:08:18.545361+0000","last_became_peered":"2024-09-18T08:08:18.545361+0000","last_unstale":"2024-09-18T08:10:12.296507+0000","last_undegraded":"2024-09-18T08:10:12.296507+0000","last_fullsized":"2024-09-18T08:10:12.296507+0000","mapping_epoch":181,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":182,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:07:17.605562+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:50.886830+0000","last_clean_scrub_stamp":"2024-09-18T08:07:17.605562+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,7],"acting":[4,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":361,"reported_epoch":218,"state":"active+clean","last_fresh":"2024-09-18T08:10:12.300711+0000","last_change":"2024-09-18T08:09:34.607443+0000","last_active":"2024-09-18T08:10:12.300711+0000","last_peered":"2024-09-18T08:10:12.300711+0000","last_clean":"2024-09-18T08:10:12.300711+0000","last_became_active":"2024-09-18T08:08:18.525552+0000","last_became_peered":"2024-09-18T08:08:18.525552+0000","last_unstale":"2024-09-18T08:10:12.300711+0000","last_undegraded":"2024-09-18T08:10:12.300711+0000","last_fullsized":"2024-09-18T08:10:12.300711+0000","mapping_epoch":181,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":182,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:09:34.607374+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_clean_scrub_stamp":"2024-09-18T08:09:34.607374+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:10:39.783401+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,4],"acting":[7,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":328,"reported_epoch":218,"state":"active+clean","last_fresh":"2024-09-18T08:10:12.302048+0000","last_change":"2024-09-18T08:08:18.539359+0000","last_active":"2024-09-18T08:10:12.302048+0000","last_peered":"2024-09-18T08:10:12.302048+0000","last_clean":"2024-09-18T08:10:12.302048+0000","last_became_active":"2024-09-18T08:08:18.538838+0000","last_became_peered":"2024-09-18T08:08:18.538838+0000","last_unstale":"2024-09-18T08:10:12.302048+0000","last_undegraded":"2024-09-18T08:10:12.302048+0000","last_fullsized":"2024-09-18T08:10:12.302048+0000","mapping_epoch":181,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":182,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_clean_scrub_stamp":"2024-09-18T08:02:52.890475+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:10:51.616752+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":343,"reported_epoch":218,"state":"active+clean","last_fresh":"2024-09-18T08:10:12.320624+0000","last_change":"2024-09-18T08:07:17.277964+0000","last_active":"2024-09-18T08:10:12.320624+0000","last_peered":"2024-09-18T08:10:12.320624+0000","last_clean":"2024-09-18T08:10:12.320624+0000","last_became_active":"2024-09-18T08:02:47.878922+0000","last_became_peered":"2024-09-18T08:02:47.878922+0000","last_unstale":"2024-09-18T08:10:12.320624+0000","last_undegraded":"2024-09-18T08:10:12.320624+0000","last_fullsized":"2024-09-18T08:10:12.320624+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:07:17.277894+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:46.860092+0000","last_clean_scrub_stamp":"2024-09-18T08:07:17.277894+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:11:04.482372+0000","scrub_duration":16,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"69'11","reported_seq":331,"reported_epoch":218,"state":"active+clean","last_fresh":"2024-09-18T08:10:12.302991+0000","last_change":"2024-09-18T08:06:24.164086+0000","last_active":"2024-09-18T08:10:12.302991+0000","last_peered":"2024-09-18T08:10:12.302991+0000","last_clean":"2024-09-18T08:10:12.302991+0000","last_became_active":"2024-09-18T08:06:24.135232+0000","last_became_peered":"2024-09-18T08:06:24.135232+0000","last_unstale":"2024-09-18T08:10:12.302991+0000","last_undegraded":"2024-09-18T08:10:12.302991+0000","last_fullsized":"2024-09-18T08:10:12.302991+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'11","last_scrub_stamp":"2024-09-18T08:06:12.967462+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_clean_scrub_stamp":"2024-09-18T08:06:12.967462+0000","objects_scrubbed":5,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":19,"num_read_kb":12,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,1],"acting":[5,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":348,"reported_epoch":218,"state":"active+clean","last_fresh":"2024-09-18T08:10:12.298550+0000","last_change":"2024-09-18T08:08:52.701053+0000","last_active":"2024-09-18T08:10:12.298550+0000","last_peered":"2024-09-18T08:10:12.298550+0000","last_clean":"2024-09-18T08:10:12.298550+0000","last_became_active":"2024-09-18T08:08:52.700595+0000","last_became_peered":"2024-09-18T08:08:52.700595+0000","last_unstale":"2024-09-18T08:10:12.298550+0000","last_undegraded":"2024-09-18T08:10:12.298550+0000","last_fullsized":"2024-09-18T08:10:12.298550+0000","mapping_epoch":198,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":199,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:08:12.639376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:50.886830+0000","last_clean_scrub_stamp":"2024-09-18T08:08:12.639376+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":8,"num_pg":16,"stat_sum":{"num_bytes":3352952856,"num_objects":51163,"num_object_clones":0,"num_object_copies":153489,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":51,"num_objects_unfound":0,"num_objects_dirty":51163,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":150337,"num_write_kb":6447937,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24193,"num_bytes_recovered":1585512448,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":11002392576,"data_stored":11002380360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":78563,"ondisk_log_size":78563,"up":48,"acting":47,"num_store_stats":8},{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":416,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":85,"ondisk_log_size":85,"up":96,"acting":96,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":178,"num_object_clones":0,"num_object_copies":534,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":178,"num_whiteouts":0,"num_read":685,"num_read_kb":442,"num_write":417,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":214,"num_bytes_recovered":4114,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":417792,"data_stored":11106,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":398,"ondisk_log_size":398,"up":96,"acting":96,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":1613,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":24,"num_read_kb":24,"num_write":10,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":3226,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":4839,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":6,"ondisk_log_size":6,"up":96,"acting":96,"num_store_stats":8},{"poolid":2,"num_pg":3,"stat_sum":{"num_bytes":408,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":213,"num_read_kb":208,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":38,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":8,"ondisk_log_size":8,"up":9,"acting":9,"num_store_stats":7},{"poolid":1,"num_pg":11,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":869,"ondisk_log_size":869,"up":33,"acting":33,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":56,"seq":240518168669,"num_pgs":63,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1232796,"kb_used_data":1061320,"kb_used_omap":1,"kb_used_meta":171390,"kb_avail":92512356,"statfs":{"total":95995035648,"available":94732652544,"internally_reserved":0,"allocated":1086791680,"data_stored":1086438170,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":175503818},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":47,"seq":201863463011,"num_pgs":54,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1546100,"kb_used_data":1334436,"kb_used_omap":1,"kb_used_meta":211646,"kb_avail":92199052,"statfs":{"total":95995035648,"available":94411829248,"internally_reserved":0,"allocated":1366462464,"data_stored":1366137753,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":216725958},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":39,"seq":167503724649,"num_pgs":59,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1843092,"kb_used_data":1645316,"kb_used_omap":1,"kb_used_meta":197758,"kb_avail":91902060,"statfs":{"total":95995035648,"available":94107709440,"internally_reserved":0,"allocated":1684803584,"data_stored":1684454437,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":202504646},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":32,"seq":137438953582,"num_pgs":67,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1607008,"kb_used_data":1368912,"kb_used_omap":1,"kb_used_meta":238014,"kb_avail":92138144,"statfs":{"total":95995035648,"available":94349459456,"internally_reserved":0,"allocated":1401765888,"data_stored":1401404675,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":243726789},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":26,"seq":111669149813,"num_pgs":68,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1590700,"kb_used_data":1284632,"kb_used_omap":1,"kb_used_meta":306046,"kb_avail":92154452,"statfs":{"total":95995035648,"available":94366158848,"internally_reserved":0,"allocated":1315463168,"data_stored":1315093600,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":313391560},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411451,"num_pgs":49,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1564632,"kb_used_data":1236292,"kb_used_omap":1,"kb_used_meta":328254,"kb_avail":92180520,"statfs":{"total":95995035648,"available":94392852480,"internally_reserved":0,"allocated":1265963008,"data_stored":1265613967,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":336132549},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,1,3,1],"upper_bound":32},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574975,"num_pgs":63,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1712068,"kb_used_data":1442096,"kb_used_omap":1,"kb_used_meta":269950,"kb_avail":92033084,"statfs":{"total":95995035648,"available":94241878016,"internally_reserved":0,"allocated":1476706304,"data_stored":1476313622,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":276429253},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738502,"num_pgs":56,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1700260,"kb_used_data":1448336,"kb_used_omap":1,"kb_used_meta":251902,"kb_avail":92044892,"statfs":{"total":95995035648,"available":94253969408,"internally_reserved":0,"allocated":1483096064,"data_stored":1482735053,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":257948101},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":266240,"data_stored":262688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1131,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1521,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":436,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":46,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1177,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":436,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":77824,"data_stored":2008,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1128,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":1898,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1474,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":990,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1034,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1254,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":403,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":416,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":403,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":1481834496,"data_stored":1481834496,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":1469841408,"data_stored":1469841408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":1258426368,"data_stored":1258422296,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":1303314432,"data_stored":1303314432,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":1400111104,"data_stored":1400111104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":1672089600,"data_stored":1672085528,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":1343160320,"data_stored":1343160320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":1073614848,"data_stored":1073610776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T08:10:19.006 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0 2024-09-18T08:10:19.047 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:10:19.048 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:10:19.048 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0' 2024-09-18T08:10:19.149 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3 2024-09-18T08:10:19.184 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:10:19.185 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:10:19.185 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3' 2024-09-18T08:10:19.202 INFO:teuthology.orchestra.run.smithi007.stdout:osd.0: {} 2024-09-18T08:10:19.202 INFO:teuthology.orchestra.run.smithi007.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-18T08:10:19.245 INFO:teuthology.orchestra.run.smithi007.stdout:osd.1: {} 2024-09-18T08:10:19.245 INFO:teuthology.orchestra.run.smithi007.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-18T08:10:19.280 INFO:teuthology.orchestra.run.smithi007.stdout:osd.2: {} 2024-09-18T08:10:19.281 INFO:teuthology.orchestra.run.smithi007.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-18T08:10:19.286 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4 2024-09-18T08:10:19.314 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:10:19.315 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:10:19.315 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4' 2024-09-18T08:10:19.319 INFO:teuthology.orchestra.run.smithi007.stdout:osd.3: {} 2024-09-18T08:10:19.319 INFO:teuthology.orchestra.run.smithi007.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-18T08:10:19.362 INFO:teuthology.orchestra.run.smithi007.stdout:osd.4: {} 2024-09-18T08:10:19.362 INFO:teuthology.orchestra.run.smithi007.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-18T08:10:19.395 INFO:teuthology.orchestra.run.smithi007.stdout:osd.5: {} 2024-09-18T08:10:19.396 INFO:teuthology.orchestra.run.smithi007.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-18T08:10:19.406 INFO:journalctl@ceph.iscsi.iscsi.a.smithi050.stdout:Sep 18 08:10:19 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-iscsi-iscsi-a[57782]: debug there is no tcmu-runner data available 2024-09-18T08:10:19.406 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:10:19 smithi050 ceph-mon[32371]: osdmap e222: 8 total, 8 up, 8 in 2024-09-18T08:10:19.406 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:10:19 smithi050 ceph-mon[32371]: 4.10 scrub starts 2024-09-18T08:10:19.406 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:10:19 smithi050 ceph-mon[32371]: 4.10 scrub ok 2024-09-18T08:10:19.406 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:10:19 smithi050 ceph-mon[32371]: 4.7 scrub starts 2024-09-18T08:10:19.406 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:10:19 smithi050 ceph-mon[32371]: 4.7 scrub ok 2024-09-18T08:10:19.406 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:10:19 smithi050 ceph-mon[32371]: 4.2 scrub starts 2024-09-18T08:10:19.406 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:10:19 smithi050 ceph-mon[32371]: 4.2 scrub ok 2024-09-18T08:10:19.406 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:10:19 smithi050 ceph-mon[32371]: from='client.? 172.21.15.7:0/1557137373' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-18T08:10:19.407 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:10:19 smithi050 ceph-mon[32371]: 4.18 scrub starts 2024-09-18T08:10:19.407 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:10:19 smithi050 ceph-mon[32371]: 4.18 scrub ok 2024-09-18T08:10:19.416 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1 2024-09-18T08:10:19.431 INFO:teuthology.orchestra.run.smithi007.stdout:osd.6: {} 2024-09-18T08:10:19.431 INFO:teuthology.orchestra.run.smithi007.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-18T08:10:19.454 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:10:19.454 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:10:19.455 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1' 2024-09-18T08:10:19.464 INFO:teuthology.orchestra.run.smithi007.stdout:osd.7: {} 2024-09-18T08:10:19.465 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:10:19 smithi007 ceph-mon[29459]: osdmap e222: 8 total, 8 up, 8 in 2024-09-18T08:10:19.465 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:10:19 smithi007 ceph-mon[29459]: 4.10 scrub starts 2024-09-18T08:10:19.465 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:10:19 smithi007 ceph-mon[29459]: 4.10 scrub ok 2024-09-18T08:10:19.465 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:10:19 smithi007 ceph-mon[29459]: 4.7 scrub starts 2024-09-18T08:10:19.465 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:10:19 smithi007 ceph-mon[29459]: 4.7 scrub ok 2024-09-18T08:10:19.465 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:10:19 smithi007 ceph-mon[29459]: 4.2 scrub starts 2024-09-18T08:10:19.465 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:10:19 smithi007 ceph-mon[29459]: 4.2 scrub ok 2024-09-18T08:10:19.465 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:10:19 smithi007 ceph-mon[29459]: from='client.? 172.21.15.7:0/1557137373' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-18T08:10:19.466 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:10:19 smithi007 ceph-mon[29459]: 4.18 scrub starts 2024-09-18T08:10:19.466 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:10:19 smithi007 ceph-mon[29459]: 4.18 scrub ok 2024-09-18T08:10:19.466 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:10:19 smithi007 ceph-mon[39249]: osdmap e222: 8 total, 8 up, 8 in 2024-09-18T08:10:19.466 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:10:19 smithi007 ceph-mon[39249]: 4.10 scrub starts 2024-09-18T08:10:19.466 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:10:19 smithi007 ceph-mon[39249]: 4.10 scrub ok 2024-09-18T08:10:19.466 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:10:19 smithi007 ceph-mon[39249]: 4.7 scrub starts 2024-09-18T08:10:19.466 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:10:19 smithi007 ceph-mon[39249]: 4.7 scrub ok 2024-09-18T08:10:19.466 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:10:19 smithi007 ceph-mon[39249]: 4.2 scrub starts 2024-09-18T08:10:19.466 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:10:19 smithi007 ceph-mon[39249]: 4.2 scrub ok 2024-09-18T08:10:19.466 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:10:19 smithi007 ceph-mon[39249]: from='client.? 172.21.15.7:0/1557137373' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-18T08:10:19.466 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:10:19 smithi007 ceph-mon[39249]: 4.18 scrub starts 2024-09-18T08:10:19.467 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:10:19 smithi007 ceph-mon[39249]: 4.18 scrub ok 2024-09-18T08:10:19.467 INFO:teuthology.orchestra.run.smithi007.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-18T08:10:19.557 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7 2024-09-18T08:10:19.589 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:10:19.589 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:10:19.589 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7' 2024-09-18T08:10:19.691 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2 2024-09-18T08:10:19.736 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:10:19.737 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:10:19.737 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2' 2024-09-18T08:10:19.758 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid 779349ca-7593-11ef-bceb-c7b262605968 -- ceph daemon osd.0 dump_historic_ops 2024-09-18T08:10:19.838 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5 2024-09-18T08:10:19.868 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:10:19.869 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:10:19.869 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5' 2024-09-18T08:10:19.970 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2 2024-09-18T08:10:20.008 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/779349ca-7593-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T08:10:20.027 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:10:20.027 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:10:20.028 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2' 2024-09-18T08:10:20.128 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4 2024-09-18T08:10:20.157 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:10:20.158 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:10:20.158 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4' 2024-09-18T08:10:20.259 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6 2024-09-18T08:10:20.288 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:10:20.289 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:10:20.289 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6' 2024-09-18T08:10:20.390 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1 2024-09-18T08:10:20.432 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:10:20.433 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:10:20.433 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1' 2024-09-18T08:10:20.534 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2 2024-09-18T08:10:20.572 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:10:20.576 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:10:20.577 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2' 2024-09-18T08:10:20.656 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:10:20 smithi050 ceph-mon[32371]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-18T08:10:20.656 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:10:20 smithi050 ceph-mon[32371]: pgmap v363: 158 pgs: 1 peering, 157 active+clean; 2.8 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 1.0 KiB/s rd, 0 B/s wr, 1.04k op/s; 654 KiB/s, 10 objects/s recovering 2024-09-18T08:10:20.656 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:10:20 smithi050 ceph-mon[32371]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 1 pg undersized) 2024-09-18T08:10:20.656 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:10:20 smithi050 ceph-mon[32371]: Cluster is now healthy 2024-09-18T08:10:20.656 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:10:20 smithi050 ceph-mon[32371]: 8.f scrub starts 2024-09-18T08:10:20.656 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:10:20 smithi050 ceph-mon[32371]: 4.a scrub starts 2024-09-18T08:10:20.656 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:10:20 smithi050 ceph-mon[32371]: 4.a scrub ok 2024-09-18T08:10:20.656 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:10:20 smithi050 ceph-mon[32371]: 4.1e scrub starts 2024-09-18T08:10:20.656 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:10:20 smithi050 ceph-mon[32371]: 4.1e scrub ok 2024-09-18T08:10:20.677 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7 2024-09-18T08:10:20.690 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:10:20 smithi007 ceph-mon[29459]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-18T08:10:20.690 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:10:20 smithi007 ceph-mon[29459]: pgmap v363: 158 pgs: 1 peering, 157 active+clean; 2.8 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 1.0 KiB/s rd, 0 B/s wr, 1.04k op/s; 654 KiB/s, 10 objects/s recovering 2024-09-18T08:10:20.690 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:10:20 smithi007 ceph-mon[29459]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 1 pg undersized) 2024-09-18T08:10:20.690 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:10:20 smithi007 ceph-mon[29459]: Cluster is now healthy 2024-09-18T08:10:20.690 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:10:20 smithi007 ceph-mon[29459]: 8.f scrub starts 2024-09-18T08:10:20.690 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:10:20 smithi007 ceph-mon[29459]: 4.a scrub starts 2024-09-18T08:10:20.690 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:10:20 smithi007 ceph-mon[29459]: 4.a scrub ok 2024-09-18T08:10:20.690 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:10:20 smithi007 ceph-mon[29459]: 4.1e scrub starts 2024-09-18T08:10:20.690 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:10:20 smithi007 ceph-mon[29459]: 4.1e scrub ok 2024-09-18T08:10:20.691 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:10:20 smithi007 ceph-mon[39249]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-18T08:10:20.691 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:10:20 smithi007 ceph-mon[39249]: pgmap v363: 158 pgs: 1 peering, 157 active+clean; 2.8 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 1.0 KiB/s rd, 0 B/s wr, 1.04k op/s; 654 KiB/s, 10 objects/s recovering 2024-09-18T08:10:20.691 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:10:20 smithi007 ceph-mon[39249]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 1 pg undersized) 2024-09-18T08:10:20.691 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:10:20 smithi007 ceph-mon[39249]: Cluster is now healthy 2024-09-18T08:10:20.691 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:10:20 smithi007 ceph-mon[39249]: 8.f scrub starts 2024-09-18T08:10:20.691 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:10:20 smithi007 ceph-mon[39249]: 4.a scrub starts 2024-09-18T08:10:20.691 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:10:20 smithi007 ceph-mon[39249]: 4.a scrub ok 2024-09-18T08:10:20.691 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:10:20 smithi007 ceph-mon[39249]: 4.1e scrub starts 2024-09-18T08:10:20.691 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:10:20 smithi007 ceph-mon[39249]: 4.1e scrub ok 2024-09-18T08:10:20.707 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:10:20.708 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:10:20.708 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7' 2024-09-18T08:10:20.809 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5 2024-09-18T08:10:20.840 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:10:20.841 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:10:20.841 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5' 2024-09-18T08:10:20.942 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0 2024-09-18T08:10:20.986 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:10:20.987 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:10:20.987 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0' 2024-09-18T08:10:21.088 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1 2024-09-18T08:10:21.136 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:10:21.136 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:10:21.137 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1' 2024-09-18T08:10:21.191 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/779349ca-7593-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T08:10:21.241 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2 2024-09-18T08:10:21.291 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:10:21.292 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:10:21.292 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2' 2024-09-18T08:10:21.393 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1 2024-09-18T08:10:21.429 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:10:21.430 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:10:21.430 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1' 2024-09-18T08:10:21.531 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7 2024-09-18T08:10:21.557 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:10:21.558 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:10:21.558 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7' 2024-09-18T08:10:21.629 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:10:21 smithi007 ceph-mon[39249]: 6.11 scrub starts 2024-09-18T08:10:21.630 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:10:21 smithi007 ceph-mon[39249]: 6.11 scrub ok 2024-09-18T08:10:21.630 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:10:21 smithi007 ceph-mon[29459]: 6.11 scrub starts 2024-09-18T08:10:21.630 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:10:21 smithi007 ceph-mon[29459]: 6.11 scrub ok 2024-09-18T08:10:21.655 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:10:21 smithi050 ceph-mon[32371]: 6.11 scrub starts 2024-09-18T08:10:21.656 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:10:21 smithi050 ceph-mon[32371]: 6.11 scrub ok 2024-09-18T08:10:21.659 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0 2024-09-18T08:10:21.706 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:10:21.707 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:10:21.707 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0' 2024-09-18T08:10:21.791 INFO:teuthology.orchestra.run.smithi007.stdout:ERROR: (22) Invalid argument 2024-09-18T08:10:21.791 INFO:teuthology.orchestra.run.smithi007.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-18T08:10:21.808 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2 2024-09-18T08:10:21.846 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:10:21.847 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:10:21.848 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2' 2024-09-18T08:10:21.948 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4 2024-09-18T08:10:21.988 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:10:21.988 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:10:21.989 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4' 2024-09-18T08:10:22.006 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid 779349ca-7593-11ef-bceb-c7b262605968 -- ceph pg dump --format=json 2024-09-18T08:10:22.089 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6 2024-09-18T08:10:22.123 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:10:22.124 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:10:22.124 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6' 2024-09-18T08:10:22.225 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7 2024-09-18T08:10:22.253 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:10:22.253 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:10:22.254 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7' 2024-09-18T08:10:22.354 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5 2024-09-18T08:10:22.383 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:10:22.384 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:10:22.384 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5' 2024-09-18T08:10:22.395 INFO:teuthology.orchestra.run.smithi007.stderr:noscrub is set 2024-09-18T08:10:22.485 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1 2024-09-18T08:10:22.525 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:10:22.525 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:10:22.525 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1' 2024-09-18T08:10:22.627 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5 2024-09-18T08:10:22.656 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:10:22 smithi050 ceph-mon[32371]: 8.f scrub ok 2024-09-18T08:10:22.656 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:10:22 smithi050 ceph-mon[32371]: pgmap v364: 158 pgs: 1 peering, 157 active+clean; 2.8 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 852 B/s rd, 0 B/s wr, 861 op/s; 544 KiB/s, 8 objects/s recovering 2024-09-18T08:10:22.656 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:10:22 smithi050 ceph-mon[32371]: from='client.? 172.21.15.7:0/68362751' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-18T08:10:22.656 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:10:22 smithi050 ceph-mon[32371]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-18T08:10:22.656 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:10:22 smithi050 ceph-mon[32371]: 2.1 scrub starts 2024-09-18T08:10:22.656 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:10:22 smithi050 ceph-mon[32371]: 2.1 scrub ok 2024-09-18T08:10:22.656 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:10:22 smithi050 ceph-mon[32371]: 4.3 scrub starts 2024-09-18T08:10:22.656 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:10:22 smithi050 ceph-mon[32371]: 4.3 scrub ok 2024-09-18T08:10:22.656 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:10:22 smithi050 ceph-mon[32371]: 3.a scrub starts 2024-09-18T08:10:22.656 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:10:22 smithi050 ceph-mon[32371]: 3.a scrub ok 2024-09-18T08:10:22.660 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:10:22.661 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:10:22.661 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5' 2024-09-18T08:10:22.730 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:10:22 smithi007 ceph-mon[29459]: 8.f scrub ok 2024-09-18T08:10:22.731 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:10:22 smithi007 ceph-mon[29459]: pgmap v364: 158 pgs: 1 peering, 157 active+clean; 2.8 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 852 B/s rd, 0 B/s wr, 861 op/s; 544 KiB/s, 8 objects/s recovering 2024-09-18T08:10:22.731 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:10:22 smithi007 ceph-mon[29459]: from='client.? 172.21.15.7:0/68362751' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-18T08:10:22.731 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:10:22 smithi007 ceph-mon[29459]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-18T08:10:22.731 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:10:22 smithi007 ceph-mon[29459]: 2.1 scrub starts 2024-09-18T08:10:22.731 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:10:22 smithi007 ceph-mon[29459]: 2.1 scrub ok 2024-09-18T08:10:22.731 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:10:22 smithi007 ceph-mon[29459]: 4.3 scrub starts 2024-09-18T08:10:22.731 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:10:22 smithi007 ceph-mon[29459]: 4.3 scrub ok 2024-09-18T08:10:22.731 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:10:22 smithi007 ceph-mon[29459]: 3.a scrub starts 2024-09-18T08:10:22.731 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:10:22 smithi007 ceph-mon[29459]: 3.a scrub ok 2024-09-18T08:10:22.731 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:10:22 smithi007 ceph-mon[39249]: 8.f scrub ok 2024-09-18T08:10:22.732 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:10:22 smithi007 ceph-mon[39249]: pgmap v364: 158 pgs: 1 peering, 157 active+clean; 2.8 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 852 B/s rd, 0 B/s wr, 861 op/s; 544 KiB/s, 8 objects/s recovering 2024-09-18T08:10:22.732 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:10:22 smithi007 ceph-mon[39249]: from='client.? 172.21.15.7:0/68362751' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-18T08:10:22.732 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:10:22 smithi007 ceph-mon[39249]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-18T08:10:22.732 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:10:22 smithi007 ceph-mon[39249]: 2.1 scrub starts 2024-09-18T08:10:22.732 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:10:22 smithi007 ceph-mon[39249]: 2.1 scrub ok 2024-09-18T08:10:22.732 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:10:22 smithi007 ceph-mon[39249]: 4.3 scrub starts 2024-09-18T08:10:22.732 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:10:22 smithi007 ceph-mon[39249]: 4.3 scrub ok 2024-09-18T08:10:22.732 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:10:22 smithi007 ceph-mon[39249]: 3.a scrub starts 2024-09-18T08:10:22.732 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:10:22 smithi007 ceph-mon[39249]: 3.a scrub ok 2024-09-18T08:10:22.745 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid 779349ca-7593-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-18T08:10:22.762 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4 2024-09-18T08:10:22.789 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:10:22.790 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:10:22.790 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4' 2024-09-18T08:10:22.891 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6 2024-09-18T08:10:22.919 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:10:22.920 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:10:22.921 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6' 2024-09-18T08:10:23.021 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2 2024-09-18T08:10:23.072 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:10:23.073 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:10:23.074 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2' 2024-09-18T08:10:23.174 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1 2024-09-18T08:10:23.243 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:10:23.244 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:10:23.244 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1' 2024-09-18T08:10:23.272 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid 779349ca-7593-11ef-bceb-c7b262605968 -- ceph daemon osd.1 dump_ops_in_flight 2024-09-18T08:10:23.345 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7 2024-09-18T08:10:23.378 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:10:23.379 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:10:23.379 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7' 2024-09-18T08:10:23.480 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0 2024-09-18T08:10:23.480 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:10:23 smithi007 ceph-mon[39249]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-18T08:10:23.481 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:10:23 smithi007 ceph-mon[39249]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-18T08:10:23.481 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:10:23 smithi007 ceph-mon[39249]: osdmap e223: 8 total, 8 up, 8 in 2024-09-18T08:10:23.481 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:10:23 smithi007 ceph-mon[29459]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-18T08:10:23.482 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:10:23 smithi007 ceph-mon[29459]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-18T08:10:23.482 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:10:23 smithi007 ceph-mon[29459]: osdmap e223: 8 total, 8 up, 8 in 2024-09-18T08:10:23.532 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:10:23.533 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:10:23.533 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0' 2024-09-18T08:10:23.634 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0 2024-09-18T08:10:23.655 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:10:23 smithi050 ceph-mon[32371]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-18T08:10:23.656 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:10:23 smithi050 ceph-mon[32371]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-18T08:10:23.656 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:10:23 smithi050 ceph-mon[32371]: osdmap e223: 8 total, 8 up, 8 in 2024-09-18T08:10:23.685 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:10:23.685 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:10:23.686 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0' 2024-09-18T08:10:23.786 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6 2024-09-18T08:10:23.814 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:10:23.815 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:10:23.815 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6' 2024-09-18T08:10:23.848 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/779349ca-7593-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T08:10:23.871 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/779349ca-7593-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T08:10:23.889 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/779349ca-7593-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T08:10:23.916 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1 2024-09-18T08:10:23.976 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:10:23.976 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:10:23.977 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1' 2024-09-18T08:10:24.077 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0 2024-09-18T08:10:24.117 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:10:24.118 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:10:24.118 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0' 2024-09-18T08:10:24.219 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5 2024-09-18T08:10:24.252 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:10:24.252 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:10:24.253 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5' 2024-09-18T08:10:24.354 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4 2024-09-18T08:10:24.388 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:10:24.388 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:10:24.389 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4' 2024-09-18T08:10:24.490 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3 2024-09-18T08:10:24.525 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:10:24.526 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:10:24.526 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3' 2024-09-18T08:10:24.553 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:10:24 smithi007 ceph-mon[29459]: pgmap v366: 158 pgs: 158 active+clean; 2.8 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 853 B/s rd, 0 B/s wr, 988 op/s; 544 KiB/s, 9 objects/s recovering 2024-09-18T08:10:24.554 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:10:24 smithi007 ceph-mon[39249]: pgmap v366: 158 pgs: 158 active+clean; 2.8 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 853 B/s rd, 0 B/s wr, 988 op/s; 544 KiB/s, 9 objects/s recovering 2024-09-18T08:10:24.627 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1 2024-09-18T08:10:24.655 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:10:24 smithi050 ceph-mon[32371]: pgmap v366: 158 pgs: 158 active+clean; 2.8 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 853 B/s rd, 0 B/s wr, 988 op/s; 544 KiB/s, 9 objects/s recovering 2024-09-18T08:10:24.664 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:10:24.665 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:10:24.665 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1' 2024-09-18T08:10:24.766 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3 2024-09-18T08:10:24.805 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:10:24.805 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:10:24.805 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3' 2024-09-18T08:10:24.906 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4 2024-09-18T08:10:24.936 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:10:24.937 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:10:24.937 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4' 2024-09-18T08:10:25.038 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2 2024-09-18T08:10:25.076 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:10:25.077 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:10:25.077 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2' 2024-09-18T08:10:25.160 INFO:teuthology.orchestra.run.smithi007.stdout:osd.0: {} 2024-09-18T08:10:25.160 INFO:teuthology.orchestra.run.smithi007.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-18T08:10:25.178 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7 2024-09-18T08:10:25.198 INFO:teuthology.orchestra.run.smithi007.stdout:osd.1: {} 2024-09-18T08:10:25.198 INFO:teuthology.orchestra.run.smithi007.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-18T08:10:25.211 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:10:25.212 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:10:25.212 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7' 2024-09-18T08:10:25.236 INFO:teuthology.orchestra.run.smithi007.stdout:osd.2: {} 2024-09-18T08:10:25.236 INFO:teuthology.orchestra.run.smithi007.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-18T08:10:25.269 INFO:teuthology.orchestra.run.smithi007.stdout:osd.3: {} 2024-09-18T08:10:25.269 INFO:teuthology.orchestra.run.smithi007.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-18T08:10:25.309 INFO:teuthology.orchestra.run.smithi007.stdout:osd.4: {} 2024-09-18T08:10:25.310 INFO:teuthology.orchestra.run.smithi007.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-18T08:10:25.313 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1 2024-09-18T08:10:25.345 INFO:teuthology.orchestra.run.smithi007.stdout:osd.5: {} 2024-09-18T08:10:25.345 INFO:teuthology.orchestra.run.smithi007.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-18T08:10:25.360 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:10:25.360 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:10:25.361 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1' 2024-09-18T08:10:25.388 INFO:teuthology.orchestra.run.smithi007.stdout:osd.6: {} 2024-09-18T08:10:25.388 INFO:teuthology.orchestra.run.smithi007.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-18T08:10:25.422 INFO:teuthology.orchestra.run.smithi007.stdout:osd.7: {} 2024-09-18T08:10:25.422 INFO:teuthology.orchestra.run.smithi007.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-18T08:10:25.461 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3 2024-09-18T08:10:25.487 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:10:25 smithi007 ceph-mon[29459]: from='client.? 172.21.15.7:0/893231062' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-18T08:10:25.487 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:10:25 smithi007 ceph-mon[39249]: from='client.? 172.21.15.7:0/893231062' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-18T08:10:25.499 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:10:25.500 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:10:25.500 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3' 2024-09-18T08:10:25.601 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3 2024-09-18T08:10:25.643 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:10:25.644 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:10:25.644 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3' 2024-09-18T08:10:25.745 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7 2024-09-18T08:10:25.778 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:10:25.779 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:10:25.779 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7' 2024-09-18T08:10:25.813 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid 779349ca-7593-11ef-bceb-c7b262605968 -- ceph osd set nodeep-scrub 2024-09-18T08:10:25.818 INFO:teuthology.orchestra.run.smithi007.stdout:{ 2024-09-18T08:10:25.818 INFO:teuthology.orchestra.run.smithi007.stdout: "ops": [ 2024-09-18T08:10:25.818 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:10:25.819 INFO:teuthology.orchestra.run.smithi007.stdout: "description": "osd_op(client.15216.0:164009 8.7 8.b382efc7 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e223)", 2024-09-18T08:10:25.819 INFO:teuthology.orchestra.run.smithi007.stdout: "initiated_at": "2024-09-18T08:10:25.643972+0000", 2024-09-18T08:10:25.819 INFO:teuthology.orchestra.run.smithi007.stdout: "age": 0.013812253, 2024-09-18T08:10:25.819 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.013837194000000001, 2024-09-18T08:10:25.819 INFO:teuthology.orchestra.run.smithi007.stdout: "continuous": false, 2024-09-18T08:10:25.819 INFO:teuthology.orchestra.run.smithi007.stdout: "type_data": { 2024-09-18T08:10:25.819 INFO:teuthology.orchestra.run.smithi007.stdout: "flag_point": "queued for pg", 2024-09-18T08:10:25.819 INFO:teuthology.orchestra.run.smithi007.stdout: "client_info": { 2024-09-18T08:10:25.819 INFO:teuthology.orchestra.run.smithi007.stdout: "client": "client.15216", 2024-09-18T08:10:25.819 INFO:teuthology.orchestra.run.smithi007.stdout: "client_addr": "172.21.15.7:0/2596041177", 2024-09-18T08:10:25.819 INFO:teuthology.orchestra.run.smithi007.stdout: "tid": 164009 2024-09-18T08:10:25.819 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:10:25.819 INFO:teuthology.orchestra.run.smithi007.stdout: "events": [ 2024-09-18T08:10:25.820 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:10:25.820 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "initiated", 2024-09-18T08:10:25.820 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:10:25.643972+0000", 2024-09-18T08:10:25.820 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-09-18T08:10:25.820 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:10:25.820 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:10:25.820 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "header_read", 2024-09-18T08:10:25.820 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:10:25.643972+0000", 2024-09-18T08:10:25.820 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-09-18T08:10:25.820 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:10:25.820 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:10:25.820 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "throttled", 2024-09-18T08:10:25.821 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:10:25.643973+0000", 2024-09-18T08:10:25.821 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.082e-06 2024-09-18T08:10:25.821 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:10:25.821 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:10:25.821 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "all_read", 2024-09-18T08:10:25.821 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:10:25.643978+0000", 2024-09-18T08:10:25.821 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 4.8770000000000001e-06 2024-09-18T08:10:25.821 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:10:25.821 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:10:25.821 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "dispatched", 2024-09-18T08:10:25.821 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:10:25.643989+0000", 2024-09-18T08:10:25.822 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.1291000000000001e-05 2024-09-18T08:10:25.822 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:10:25.822 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:10:25.822 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "queued_for_pg", 2024-09-18T08:10:25.822 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:10:25.643997+0000", 2024-09-18T08:10:25.822 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 8.2689999999999999e-06 2024-09-18T08:10:25.822 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-09-18T08:10:25.822 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-09-18T08:10:25.822 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-09-18T08:10:25.822 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:10:25.822 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:10:25.822 INFO:teuthology.orchestra.run.smithi007.stdout: "description": "osd_repop(client.15216.0:164011 8.b e223/194 8:d3deaef9:::benchmark_data_smithi007_101583_object63259:head v 223'10322, mlcod=223'10320)", 2024-09-18T08:10:25.822 INFO:teuthology.orchestra.run.smithi007.stdout: "initiated_at": "2024-09-18T08:10:25.647391+0000", 2024-09-18T08:10:25.823 INFO:teuthology.orchestra.run.smithi007.stdout: "age": 0.010392670999999999, 2024-09-18T08:10:25.823 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.010485822000000001, 2024-09-18T08:10:25.823 INFO:teuthology.orchestra.run.smithi007.stdout: "continuous": false, 2024-09-18T08:10:25.823 INFO:teuthology.orchestra.run.smithi007.stdout: "type_data": { 2024-09-18T08:10:25.823 INFO:teuthology.orchestra.run.smithi007.stdout: "flag_point": "started", 2024-09-18T08:10:25.823 INFO:teuthology.orchestra.run.smithi007.stdout: "events": [ 2024-09-18T08:10:25.823 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:10:25.823 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "initiated", 2024-09-18T08:10:25.823 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:10:25.647391+0000", 2024-09-18T08:10:25.823 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-09-18T08:10:25.823 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:10:25.823 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:10:25.823 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "header_read", 2024-09-18T08:10:25.823 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:10:25.647391+0000", 2024-09-18T08:10:25.824 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-09-18T08:10:25.824 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:10:25.824 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:10:25.824 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "throttled", 2024-09-18T08:10:25.824 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:10:25.647392+0000", 2024-09-18T08:10:25.824 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 7.1900000000000002e-07 2024-09-18T08:10:25.824 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:10:25.824 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:10:25.824 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "all_read", 2024-09-18T08:10:25.824 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:10:25.647398+0000", 2024-09-18T08:10:25.824 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 5.4890000000000003e-06 2024-09-18T08:10:25.824 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:10:25.825 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:10:25.825 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "dispatched", 2024-09-18T08:10:25.825 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:10:25.647418+0000", 2024-09-18T08:10:25.825 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 2.0152999999999998e-05 2024-09-18T08:10:25.825 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:10:25.825 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:10:25.825 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "queued_for_pg", 2024-09-18T08:10:25.825 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:10:25.647425+0000", 2024-09-18T08:10:25.825 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 7.0389999999999999e-06 2024-09-18T08:10:25.825 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:10:25.826 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:10:25.826 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-09-18T08:10:25.826 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:10:25.647473+0000", 2024-09-18T08:10:25.826 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 4.8059000000000002e-05 2024-09-18T08:10:25.826 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:10:25.826 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:10:25.826 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "started", 2024-09-18T08:10:25.826 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:10:25.647492+0000", 2024-09-18T08:10:25.826 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.8873e-05 2024-09-18T08:10:25.826 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-09-18T08:10:25.827 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-09-18T08:10:25.827 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-09-18T08:10:25.827 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:10:25.827 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:10:25.827 INFO:teuthology.orchestra.run.smithi007.stdout: "description": "osd_op(client.15216.0:164012 8.3 8:cf478d61:::benchmark_data_smithi007_101583_object63260:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e223)", 2024-09-18T08:10:25.827 INFO:teuthology.orchestra.run.smithi007.stdout: "initiated_at": "2024-09-18T08:10:25.649493+0000", 2024-09-18T08:10:25.827 INFO:teuthology.orchestra.run.smithi007.stdout: "age": 0.0082907299999999996, 2024-09-18T08:10:25.827 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.0084350980000000002, 2024-09-18T08:10:25.827 INFO:teuthology.orchestra.run.smithi007.stdout: "continuous": false, 2024-09-18T08:10:25.827 INFO:teuthology.orchestra.run.smithi007.stdout: "type_data": { 2024-09-18T08:10:25.827 INFO:teuthology.orchestra.run.smithi007.stdout: "flag_point": "waiting for sub ops", 2024-09-18T08:10:25.828 INFO:teuthology.orchestra.run.smithi007.stdout: "client_info": { 2024-09-18T08:10:25.828 INFO:teuthology.orchestra.run.smithi007.stdout: "client": "client.15216", 2024-09-18T08:10:25.828 INFO:teuthology.orchestra.run.smithi007.stdout: "client_addr": "172.21.15.7:0/2596041177", 2024-09-18T08:10:25.828 INFO:teuthology.orchestra.run.smithi007.stdout: "tid": 164012 2024-09-18T08:10:25.828 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:10:25.828 INFO:teuthology.orchestra.run.smithi007.stdout: "events": [ 2024-09-18T08:10:25.828 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:10:25.828 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "initiated", 2024-09-18T08:10:25.828 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:10:25.649493+0000", 2024-09-18T08:10:25.828 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-09-18T08:10:25.828 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:10:25.829 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:10:25.829 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "header_read", 2024-09-18T08:10:25.829 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:10:25.649493+0000", 2024-09-18T08:10:25.829 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-09-18T08:10:25.829 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:10:25.829 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:10:25.829 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "throttled", 2024-09-18T08:10:25.829 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:10:25.649495+0000", 2024-09-18T08:10:25.829 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.751e-06 2024-09-18T08:10:25.830 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:10:25.830 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:10:25.830 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "all_read", 2024-09-18T08:10:25.830 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:10:25.649501+0000", 2024-09-18T08:10:25.831 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 6.2720000000000003e-06 2024-09-18T08:10:25.831 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:10:25.831 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:10:25.831 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "dispatched", 2024-09-18T08:10:25.831 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:10:25.649518+0000", 2024-09-18T08:10:25.831 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.6869999999999999e-05 2024-09-18T08:10:25.831 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:10:25.831 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:10:25.831 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "queued_for_pg", 2024-09-18T08:10:25.831 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:10:25.649527+0000", 2024-09-18T08:10:25.831 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 8.4279999999999996e-06 2024-09-18T08:10:25.831 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:10:25.831 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:10:25.832 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-09-18T08:10:25.832 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:10:25.649720+0000", 2024-09-18T08:10:25.832 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.000193315 2024-09-18T08:10:25.832 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:10:25.832 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:10:25.832 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "started", 2024-09-18T08:10:25.832 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:10:25.650467+0000", 2024-09-18T08:10:25.832 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.000746604 2024-09-18T08:10:25.832 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:10:25.832 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:10:25.832 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "waiting for subops from 5,6", 2024-09-18T08:10:25.832 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:10:25.651242+0000", 2024-09-18T08:10:25.832 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.00077514900000000004 2024-09-18T08:10:25.832 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:10:25.833 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:10:25.833 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:10:25.833 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:10:25.656345+0000", 2024-09-18T08:10:25.833 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.0051038209999999997 2024-09-18T08:10:25.833 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-09-18T08:10:25.833 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-09-18T08:10:25.833 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-09-18T08:10:25.833 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:10:25.833 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:10:25.833 INFO:teuthology.orchestra.run.smithi007.stdout: "description": "osd_repop(client.15216.0:164016 8.b e223/194 8:d5f6d498:::benchmark_data_smithi007_101583_object63264:head v 223'10323, mlcod=223'10320)", 2024-09-18T08:10:25.833 INFO:teuthology.orchestra.run.smithi007.stdout: "initiated_at": "2024-09-18T08:10:25.650713+0000", 2024-09-18T08:10:25.833 INFO:teuthology.orchestra.run.smithi007.stdout: "age": 0.0070705949999999998, 2024-09-18T08:10:25.833 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.0072801100000000002, 2024-09-18T08:10:25.834 INFO:teuthology.orchestra.run.smithi007.stdout: "continuous": false, 2024-09-18T08:10:25.834 INFO:teuthology.orchestra.run.smithi007.stdout: "type_data": { 2024-09-18T08:10:25.834 INFO:teuthology.orchestra.run.smithi007.stdout: "flag_point": "started", 2024-09-18T08:10:25.834 INFO:teuthology.orchestra.run.smithi007.stdout: "events": [ 2024-09-18T08:10:25.834 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:10:25.834 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "initiated", 2024-09-18T08:10:25.834 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:10:25.650713+0000", 2024-09-18T08:10:25.834 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-09-18T08:10:25.834 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:10:25.834 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:10:25.834 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "header_read", 2024-09-18T08:10:25.834 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:10:25.650713+0000", 2024-09-18T08:10:25.834 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-09-18T08:10:25.834 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:10:25.835 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:10:25.835 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "throttled", 2024-09-18T08:10:25.835 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:10:25.650714+0000", 2024-09-18T08:10:25.835 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 7.2099999999999996e-07 2024-09-18T08:10:25.835 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:10:25.835 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:10:25.835 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "all_read", 2024-09-18T08:10:25.835 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:10:25.650720+0000", 2024-09-18T08:10:25.835 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 5.536e-06 2024-09-18T08:10:25.835 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:10:25.835 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:10:25.835 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "dispatched", 2024-09-18T08:10:25.835 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:10:25.650728+0000", 2024-09-18T08:10:25.836 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 7.9019999999999999e-06 2024-09-18T08:10:25.836 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:10:25.836 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:10:25.836 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "queued_for_pg", 2024-09-18T08:10:25.836 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:10:25.650737+0000", 2024-09-18T08:10:25.836 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 9.8290000000000003e-06 2024-09-18T08:10:25.836 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:10:25.836 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:10:25.836 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-09-18T08:10:25.836 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:10:25.655269+0000", 2024-09-18T08:10:25.836 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.0045317389999999999 2024-09-18T08:10:25.836 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:10:25.836 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:10:25.837 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "started", 2024-09-18T08:10:25.837 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:10:25.655326+0000", 2024-09-18T08:10:25.837 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 5.6648999999999999e-05 2024-09-18T08:10:25.837 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-09-18T08:10:25.837 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-09-18T08:10:25.837 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-09-18T08:10:25.837 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:10:25.837 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:10:25.837 INFO:teuthology.orchestra.run.smithi007.stdout: "description": "osd_repop_reply(client.15216.0:164012 8.3 e223/165 ondisk, result = 0)", 2024-09-18T08:10:25.837 INFO:teuthology.orchestra.run.smithi007.stdout: "initiated_at": "2024-09-18T08:10:25.656085+0000", 2024-09-18T08:10:25.837 INFO:teuthology.orchestra.run.smithi007.stdout: "age": 0.00169859, 2024-09-18T08:10:25.838 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.0019565989999999998, 2024-09-18T08:10:25.838 INFO:teuthology.orchestra.run.smithi007.stdout: "continuous": false, 2024-09-18T08:10:25.838 INFO:teuthology.orchestra.run.smithi007.stdout: "type_data": { 2024-09-18T08:10:25.838 INFO:teuthology.orchestra.run.smithi007.stdout: "flag_point": "started", 2024-09-18T08:10:25.838 INFO:teuthology.orchestra.run.smithi007.stdout: "events": [ 2024-09-18T08:10:25.838 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:10:25.838 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "initiated", 2024-09-18T08:10:25.838 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:10:25.656085+0000", 2024-09-18T08:10:25.838 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-09-18T08:10:25.838 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:10:25.838 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:10:25.838 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "header_read", 2024-09-18T08:10:25.838 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:10:25.656085+0000", 2024-09-18T08:10:25.838 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-09-18T08:10:25.839 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:10:25.839 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:10:25.839 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "throttled", 2024-09-18T08:10:25.839 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:10:25.656087+0000", 2024-09-18T08:10:25.839 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.1060000000000001e-06 2024-09-18T08:10:25.839 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:10:25.839 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:10:25.839 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "all_read", 2024-09-18T08:10:25.839 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:10:25.656092+0000", 2024-09-18T08:10:25.839 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 5.9529999999999996e-06 2024-09-18T08:10:25.839 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:10:25.839 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:10:25.839 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "dispatched", 2024-09-18T08:10:25.839 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:10:25.656232+0000", 2024-09-18T08:10:25.840 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.00013931300000000001 2024-09-18T08:10:25.840 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:10:25.840 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:10:25.840 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "queued_for_pg", 2024-09-18T08:10:25.840 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:10:25.656246+0000", 2024-09-18T08:10:25.840 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.4487e-05 2024-09-18T08:10:25.840 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:10:25.840 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:10:25.840 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-09-18T08:10:25.840 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:10:25.656332+0000", 2024-09-18T08:10:25.840 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 8.6176999999999998e-05 2024-09-18T08:10:25.840 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:10:25.840 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:10:25.841 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "started", 2024-09-18T08:10:25.841 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:10:25.656340+0000", 2024-09-18T08:10:25.841 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 7.0940000000000002e-06 2024-09-18T08:10:25.841 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-09-18T08:10:25.841 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-09-18T08:10:25.841 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-09-18T08:10:25.841 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:10:25.841 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:10:25.841 INFO:teuthology.orchestra.run.smithi007.stdout: "description": "osd_repop(client.15216.0:164017 8.f e223/211)", 2024-09-18T08:10:25.841 INFO:teuthology.orchestra.run.smithi007.stdout: "initiated_at": "2024-09-18T08:10:25.656295+0000", 2024-09-18T08:10:25.841 INFO:teuthology.orchestra.run.smithi007.stdout: "age": 0.0014889879999999999, 2024-09-18T08:10:25.841 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.0017972330000000001, 2024-09-18T08:10:25.841 INFO:teuthology.orchestra.run.smithi007.stdout: "continuous": false, 2024-09-18T08:10:25.842 INFO:teuthology.orchestra.run.smithi007.stdout: "type_data": { 2024-09-18T08:10:25.842 INFO:teuthology.orchestra.run.smithi007.stdout: "flag_point": "queued for pg", 2024-09-18T08:10:25.842 INFO:teuthology.orchestra.run.smithi007.stdout: "events": [ 2024-09-18T08:10:25.842 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:10:25.842 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "initiated", 2024-09-18T08:10:25.842 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:10:25.656295+0000", 2024-09-18T08:10:25.842 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-09-18T08:10:25.842 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:10:25.842 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:10:25.842 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "header_read", 2024-09-18T08:10:25.842 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:10:25.656295+0000", 2024-09-18T08:10:25.842 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-09-18T08:10:25.842 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:10:25.843 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:10:25.843 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "throttled", 2024-09-18T08:10:25.843 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:10:25.656296+0000", 2024-09-18T08:10:25.843 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 5.7800000000000001e-07 2024-09-18T08:10:25.843 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:10:25.843 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:10:25.843 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "all_read", 2024-09-18T08:10:25.843 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:10:25.656302+0000", 2024-09-18T08:10:25.843 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 6.3409999999999999e-06 2024-09-18T08:10:25.843 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:10:25.843 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:10:25.843 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "dispatched", 2024-09-18T08:10:25.843 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:10:25.656852+0000", 2024-09-18T08:10:25.844 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.00055026300000000005 2024-09-18T08:10:25.844 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:10:25.844 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:10:25.844 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "queued_for_pg", 2024-09-18T08:10:25.844 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:10:25.656862+0000", 2024-09-18T08:10:25.844 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 9.3370000000000006e-06 2024-09-18T08:10:25.844 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-09-18T08:10:25.844 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-09-18T08:10:25.844 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-09-18T08:10:25.844 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:10:25.844 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:10:25.844 INFO:teuthology.orchestra.run.smithi007.stdout: "description": "osd_repop(client.15216.0:164022 8.0 e223/165 8:006d340c:::benchmark_data_smithi007_101583_object63270:head v 223'10186, mlcod=223'10184)", 2024-09-18T08:10:25.845 INFO:teuthology.orchestra.run.smithi007.stdout: "initiated_at": "2024-09-18T08:10:25.656880+0000", 2024-09-18T08:10:25.845 INFO:teuthology.orchestra.run.smithi007.stdout: "age": 0.00090381800000000003, 2024-09-18T08:10:25.845 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.001259725, 2024-09-18T08:10:25.845 INFO:teuthology.orchestra.run.smithi007.stdout: "continuous": false, 2024-09-18T08:10:25.845 INFO:teuthology.orchestra.run.smithi007.stdout: "type_data": { 2024-09-18T08:10:25.845 INFO:teuthology.orchestra.run.smithi007.stdout: "flag_point": "started", 2024-09-18T08:10:25.845 INFO:teuthology.orchestra.run.smithi007.stdout: "events": [ 2024-09-18T08:10:25.845 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:10:25.845 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "initiated", 2024-09-18T08:10:25.845 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:10:25.656880+0000", 2024-09-18T08:10:25.845 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-09-18T08:10:25.845 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:10:25.845 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:10:25.845 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "header_read", 2024-09-18T08:10:25.846 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:10:25.656880+0000", 2024-09-18T08:10:25.846 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-09-18T08:10:25.846 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:10:25.846 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:10:25.846 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "throttled", 2024-09-18T08:10:25.846 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:10:25.656881+0000", 2024-09-18T08:10:25.846 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 4.1699999999999999e-07 2024-09-18T08:10:25.846 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:10:25.846 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:10:25.846 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "all_read", 2024-09-18T08:10:25.846 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:10:25.656886+0000", 2024-09-18T08:10:25.846 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 5.3120000000000001e-06 2024-09-18T08:10:25.846 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:10:25.846 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:10:25.847 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "dispatched", 2024-09-18T08:10:25.847 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:10:25.656894+0000", 2024-09-18T08:10:25.847 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 7.7619999999999995e-06 2024-09-18T08:10:25.847 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:10:25.847 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:10:25.847 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "queued_for_pg", 2024-09-18T08:10:25.847 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:10:25.656899+0000", 2024-09-18T08:10:25.847 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 5.118e-06 2024-09-18T08:10:25.847 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:10:25.847 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:10:25.847 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-09-18T08:10:25.847 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:10:25.657033+0000", 2024-09-18T08:10:25.847 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.00013463799999999999 2024-09-18T08:10:25.847 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:10:25.848 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:10:25.848 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "started", 2024-09-18T08:10:25.848 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:10:25.657052+0000", 2024-09-18T08:10:25.848 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.893e-05 2024-09-18T08:10:25.848 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-09-18T08:10:25.848 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-09-18T08:10:25.848 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-09-18T08:10:25.848 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:10:25.848 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:10:25.848 INFO:teuthology.orchestra.run.smithi007.stdout: "description": "osd_repop_reply(client.15216.0:164012 8.3 e223/165)", 2024-09-18T08:10:25.848 INFO:teuthology.orchestra.run.smithi007.stdout: "initiated_at": "2024-09-18T08:10:25.656925+0000", 2024-09-18T08:10:25.848 INFO:teuthology.orchestra.run.smithi007.stdout: "age": 0.00085905599999999997, 2024-09-18T08:10:25.848 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.0012636419999999999, 2024-09-18T08:10:25.849 INFO:teuthology.orchestra.run.smithi007.stdout: "continuous": false, 2024-09-18T08:10:25.849 INFO:teuthology.orchestra.run.smithi007.stdout: "type_data": { 2024-09-18T08:10:25.849 INFO:teuthology.orchestra.run.smithi007.stdout: "flag_point": "queued for pg", 2024-09-18T08:10:25.849 INFO:teuthology.orchestra.run.smithi007.stdout: "events": [ 2024-09-18T08:10:25.849 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:10:25.849 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "initiated", 2024-09-18T08:10:25.849 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:10:25.656925+0000", 2024-09-18T08:10:25.849 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-09-18T08:10:25.849 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:10:25.849 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:10:25.849 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "header_read", 2024-09-18T08:10:25.849 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:10:25.656925+0000", 2024-09-18T08:10:25.849 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-09-18T08:10:25.850 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:10:25.850 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:10:25.850 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "throttled", 2024-09-18T08:10:25.850 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:10:25.656925+0000", 2024-09-18T08:10:25.850 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 4.8800000000000003e-07 2024-09-18T08:10:25.850 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:10:25.850 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:10:25.850 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "all_read", 2024-09-18T08:10:25.850 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:10:25.656929+0000", 2024-09-18T08:10:25.850 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 3.3110000000000001e-06 2024-09-18T08:10:25.850 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:10:25.850 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:10:25.850 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "dispatched", 2024-09-18T08:10:25.851 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:10:25.656936+0000", 2024-09-18T08:10:25.851 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 7.6469999999999998e-06 2024-09-18T08:10:25.851 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:10:25.851 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:10:25.851 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "queued_for_pg", 2024-09-18T08:10:25.851 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:10:25.656942+0000", 2024-09-18T08:10:25.851 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 5.5339999999999999e-06 2024-09-18T08:10:25.851 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-09-18T08:10:25.851 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-09-18T08:10:25.851 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-09-18T08:10:25.851 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:10:25.851 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:10:25.851 INFO:teuthology.orchestra.run.smithi007.stdout: "description": "osd_repop_reply(client.15216.0:164006 8.7 e223/165)", 2024-09-18T08:10:25.851 INFO:teuthology.orchestra.run.smithi007.stdout: "initiated_at": "2024-09-18T08:10:25.656952+0000", 2024-09-18T08:10:25.851 INFO:teuthology.orchestra.run.smithi007.stdout: "age": 0.00083191899999999997, 2024-09-18T08:10:25.852 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.0012810899999999999, 2024-09-18T08:10:25.852 INFO:teuthology.orchestra.run.smithi007.stdout: "continuous": false, 2024-09-18T08:10:25.852 INFO:teuthology.orchestra.run.smithi007.stdout: "type_data": { 2024-09-18T08:10:25.852 INFO:teuthology.orchestra.run.smithi007.stdout: "flag_point": "queued for pg", 2024-09-18T08:10:25.852 INFO:teuthology.orchestra.run.smithi007.stdout: "events": [ 2024-09-18T08:10:25.852 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:10:25.852 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "initiated", 2024-09-18T08:10:25.852 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:10:25.656952+0000", 2024-09-18T08:10:25.852 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-09-18T08:10:25.852 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:10:25.852 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:10:25.852 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "header_read", 2024-09-18T08:10:25.852 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:10:25.656952+0000", 2024-09-18T08:10:25.853 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-09-18T08:10:25.853 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:10:25.853 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:10:25.853 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "throttled", 2024-09-18T08:10:25.853 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:10:25.656952+0000", 2024-09-18T08:10:25.853 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 3.1600000000000002e-07 2024-09-18T08:10:25.853 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:10:25.853 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:10:25.853 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "all_read", 2024-09-18T08:10:25.853 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:10:25.656955+0000", 2024-09-18T08:10:25.853 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 2.835e-06 2024-09-18T08:10:25.853 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:10:25.853 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:10:25.854 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "dispatched", 2024-09-18T08:10:25.854 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:10:25.656961+0000", 2024-09-18T08:10:25.854 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 5.5960000000000003e-06 2024-09-18T08:10:25.854 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:10:25.854 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:10:25.854 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "queued_for_pg", 2024-09-18T08:10:25.854 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:10:25.656966+0000", 2024-09-18T08:10:25.854 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 4.899e-06 2024-09-18T08:10:25.854 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-09-18T08:10:25.854 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-09-18T08:10:25.854 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-09-18T08:10:25.854 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:10:25.854 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:10:25.854 INFO:teuthology.orchestra.run.smithi007.stdout: "description": "osd_op(client.15216.0:164005 8.7 8:e5f0db59:::benchmark_data_smithi007_101583_object63253:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e223)", 2024-09-18T08:10:25.855 INFO:teuthology.orchestra.run.smithi007.stdout: "initiated_at": "2024-09-18T08:10:25.642718+0000", 2024-09-18T08:10:25.855 INFO:teuthology.orchestra.run.smithi007.stdout: "age": 0.015066298000000001, 2024-09-18T08:10:25.855 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.015564789000000001, 2024-09-18T08:10:25.855 INFO:teuthology.orchestra.run.smithi007.stdout: "continuous": false, 2024-09-18T08:10:25.855 INFO:teuthology.orchestra.run.smithi007.stdout: "type_data": { 2024-09-18T08:10:25.855 INFO:teuthology.orchestra.run.smithi007.stdout: "flag_point": "waiting for sub ops", 2024-09-18T08:10:25.855 INFO:teuthology.orchestra.run.smithi007.stdout: "client_info": { 2024-09-18T08:10:25.855 INFO:teuthology.orchestra.run.smithi007.stdout: "client": "client.15216", 2024-09-18T08:10:25.855 INFO:teuthology.orchestra.run.smithi007.stdout: "client_addr": "172.21.15.7:0/2596041177", 2024-09-18T08:10:25.855 INFO:teuthology.orchestra.run.smithi007.stdout: "tid": 164005 2024-09-18T08:10:25.855 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:10:25.855 INFO:teuthology.orchestra.run.smithi007.stdout: "events": [ 2024-09-18T08:10:25.855 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:10:25.855 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "initiated", 2024-09-18T08:10:25.855 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:10:25.642718+0000", 2024-09-18T08:10:25.856 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-09-18T08:10:25.856 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:10:25.856 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:10:25.856 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "header_read", 2024-09-18T08:10:25.856 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:10:25.642718+0000", 2024-09-18T08:10:25.856 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-09-18T08:10:25.856 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:10:25.856 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:10:25.856 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "throttled", 2024-09-18T08:10:25.856 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:10:25.642718+0000", 2024-09-18T08:10:25.856 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 5.9699999999999996e-07 2024-09-18T08:10:25.856 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:10:25.856 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:10:25.856 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "all_read", 2024-09-18T08:10:25.856 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:10:25.642722+0000", 2024-09-18T08:10:25.857 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 3.281e-06 2024-09-18T08:10:25.857 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:10:25.857 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:10:25.857 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "dispatched", 2024-09-18T08:10:25.857 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:10:25.642729+0000", 2024-09-18T08:10:25.857 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 7.3939999999999998e-06 2024-09-18T08:10:25.857 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:10:25.857 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:10:25.857 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "queued_for_pg", 2024-09-18T08:10:25.857 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:10:25.642768+0000", 2024-09-18T08:10:25.857 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 3.9499999999999998e-05 2024-09-18T08:10:25.857 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:10:25.858 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:10:25.858 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-09-18T08:10:25.858 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:10:25.646549+0000", 2024-09-18T08:10:25.858 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.0037804729999999999 2024-09-18T08:10:25.858 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:10:25.858 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:10:25.858 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "started", 2024-09-18T08:10:25.858 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:10:25.646813+0000", 2024-09-18T08:10:25.858 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.00026449900000000002 2024-09-18T08:10:25.858 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:10:25.858 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:10:25.858 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "waiting for subops from 3,6", 2024-09-18T08:10:25.858 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:10:25.647017+0000", 2024-09-18T08:10:25.858 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.00020378799999999999 2024-09-18T08:10:25.858 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:10:25.858 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:10:25.859 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:10:25.859 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:10:25.651382+0000", 2024-09-18T08:10:25.859 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.0043650349999999997 2024-09-18T08:10:25.859 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:10:25.859 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:10:25.859 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:10:25.859 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:10:25.651422+0000", 2024-09-18T08:10:25.859 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 3.9422999999999998e-05 2024-09-18T08:10:25.859 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-09-18T08:10:25.859 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-09-18T08:10:25.859 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-09-18T08:10:25.859 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:10:25.859 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:10:25.859 INFO:teuthology.orchestra.run.smithi007.stdout: "description": "osd_op(client.15216.0:164006 8.7 8:e634521e:::benchmark_data_smithi007_101583_object63254:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e223)", 2024-09-18T08:10:25.859 INFO:teuthology.orchestra.run.smithi007.stdout: "initiated_at": "2024-09-18T08:10:25.643116+0000", 2024-09-18T08:10:25.860 INFO:teuthology.orchestra.run.smithi007.stdout: "age": 0.01466783, 2024-09-18T08:10:25.860 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.015247251, 2024-09-18T08:10:25.860 INFO:teuthology.orchestra.run.smithi007.stdout: "continuous": false, 2024-09-18T08:10:25.860 INFO:teuthology.orchestra.run.smithi007.stdout: "type_data": { 2024-09-18T08:10:25.860 INFO:teuthology.orchestra.run.smithi007.stdout: "flag_point": "waiting for sub ops", 2024-09-18T08:10:25.860 INFO:teuthology.orchestra.run.smithi007.stdout: "client_info": { 2024-09-18T08:10:25.860 INFO:teuthology.orchestra.run.smithi007.stdout: "client": "client.15216", 2024-09-18T08:10:25.860 INFO:teuthology.orchestra.run.smithi007.stdout: "client_addr": "172.21.15.7:0/2596041177", 2024-09-18T08:10:25.860 INFO:teuthology.orchestra.run.smithi007.stdout: "tid": 164006 2024-09-18T08:10:25.860 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:10:25.860 INFO:teuthology.orchestra.run.smithi007.stdout: "events": [ 2024-09-18T08:10:25.860 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:10:25.861 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "initiated", 2024-09-18T08:10:25.861 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:10:25.643116+0000", 2024-09-18T08:10:25.861 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-09-18T08:10:25.861 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:10:25.861 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:10:25.861 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "header_read", 2024-09-18T08:10:25.861 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:10:25.643116+0000", 2024-09-18T08:10:25.861 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-09-18T08:10:25.861 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:10:25.861 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:10:25.861 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "throttled", 2024-09-18T08:10:25.861 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:10:25.643117+0000", 2024-09-18T08:10:25.862 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 9.3600000000000002e-07 2024-09-18T08:10:25.862 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:10:25.862 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:10:25.862 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "all_read", 2024-09-18T08:10:25.862 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:10:25.643128+0000", 2024-09-18T08:10:25.862 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.0859000000000001e-05 2024-09-18T08:10:25.862 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:10:25.862 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:10:25.862 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "dispatched", 2024-09-18T08:10:25.862 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:10:25.643146+0000", 2024-09-18T08:10:25.862 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.7991000000000001e-05 2024-09-18T08:10:25.862 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:10:25.862 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:10:25.862 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "queued_for_pg", 2024-09-18T08:10:25.862 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:10:25.643154+0000", 2024-09-18T08:10:25.863 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 7.6040000000000004e-06 2024-09-18T08:10:25.863 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:10:25.863 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:10:25.863 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-09-18T08:10:25.863 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:10:25.651449+0000", 2024-09-18T08:10:25.863 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.0082950379999999994 2024-09-18T08:10:25.863 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:10:25.863 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:10:25.863 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "started", 2024-09-18T08:10:25.863 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:10:25.651631+0000", 2024-09-18T08:10:25.863 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.00018237599999999999 2024-09-18T08:10:25.863 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:10:25.863 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:10:25.863 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "waiting for subops from 3,6", 2024-09-18T08:10:25.863 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:10:25.651738+0000", 2024-09-18T08:10:25.863 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.000107366 2024-09-18T08:10:25.864 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-09-18T08:10:25.864 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-09-18T08:10:25.864 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-09-18T08:10:25.864 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-09-18T08:10:25.864 INFO:teuthology.orchestra.run.smithi007.stdout: ], 2024-09-18T08:10:25.864 INFO:teuthology.orchestra.run.smithi007.stdout: "num_ops": 11 2024-09-18T08:10:25.864 INFO:teuthology.orchestra.run.smithi007.stdout:} 2024-09-18T08:10:25.881 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4 2024-09-18T08:10:25.905 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:10:25 smithi050 ceph-mon[32371]: from='client.? 172.21.15.7:0/893231062' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-18T08:10:25.908 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:10:25.908 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:10:25.909 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4' 2024-09-18T08:10:26.009 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2 2024-09-18T08:10:26.050 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:10:26.051 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:10:26.051 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2' 2024-09-18T08:10:26.152 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4 2024-09-18T08:10:26.186 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:10:26.187 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:10:26.187 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4' 2024-09-18T08:10:26.288 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5 2024-09-18T08:10:26.320 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:10:26.321 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:10:26.321 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5' 2024-09-18T08:10:26.422 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0 2024-09-18T08:10:26.467 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:10:26.468 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:10:26.468 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0' 2024-09-18T08:10:26.569 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1 2024-09-18T08:10:26.608 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:10:26.609 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:10:26.609 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1' 2024-09-18T08:10:26.698 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:10:26 smithi007 ceph-mon[29459]: from='client.25219 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-18T08:10:26.698 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:10:26 smithi007 ceph-mon[29459]: pgmap v367: 158 pgs: 158 active+clean; 2.4 GiB data, 10 GiB used, 705 GiB / 715 GiB avail; 1.3 KiB/s rd, 0 B/s wr, 1.43k op/s; 414 KiB/s, 7 objects/s recovering 2024-09-18T08:10:26.699 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:10:26 smithi007 ceph-mon[39249]: from='client.25219 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-18T08:10:26.699 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:10:26 smithi007 ceph-mon[39249]: pgmap v367: 158 pgs: 158 active+clean; 2.4 GiB data, 10 GiB used, 705 GiB / 715 GiB avail; 1.3 KiB/s rd, 0 B/s wr, 1.43k op/s; 414 KiB/s, 7 objects/s recovering 2024-09-18T08:10:26.710 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3 2024-09-18T08:10:26.754 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:10:26.755 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:10:26.756 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3' 2024-09-18T08:10:26.857 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid 779349ca-7593-11ef-bceb-c7b262605968 -- ceph daemon osd.1 dump_blocked_ops 2024-09-18T08:10:26.857 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6 2024-09-18T08:10:26.887 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:10:26.888 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:10:26.889 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6' 2024-09-18T08:10:26.905 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:10:26 smithi050 ceph-mon[32371]: from='client.25219 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-18T08:10:26.906 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:10:26 smithi050 ceph-mon[32371]: pgmap v367: 158 pgs: 158 active+clean; 2.4 GiB data, 10 GiB used, 705 GiB / 715 GiB avail; 1.3 KiB/s rd, 0 B/s wr, 1.43k op/s; 414 KiB/s, 7 objects/s recovering 2024-09-18T08:10:26.980 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Sep 18 08:10:26 smithi007 ceph-779349ca-7593-11ef-bceb-c7b262605968-mgr-y[29688]: ::ffff:172.21.15.50 - - [18/Sep/2024:08:10:26] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-09-18T08:10:26.990 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0 2024-09-18T08:10:27.041 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:10:27.041 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:10:27.042 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0' 2024-09-18T08:10:27.142 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4 2024-09-18T08:10:27.177 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:10:27.177 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:10:27.178 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4' 2024-09-18T08:10:27.278 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0 2024-09-18T08:10:27.321 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:10:27.322 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:10:27.322 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0' 2024-09-18T08:10:27.422 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6 2024-09-18T08:10:27.455 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:10:27.456 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:10:27.456 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6' 2024-09-18T08:10:27.504 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-09-18T08:10:27.506 INFO:teuthology.orchestra.run.smithi007.stderr:dumped all 2024-09-18T08:10:27.557 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6 2024-09-18T08:10:27.585 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:10:27.586 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:10:27.587 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6' 2024-09-18T08:10:27.687 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0 2024-09-18T08:10:27.710 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:10:27.711 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:10:27.711 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0' 2024-09-18T08:10:27.812 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4 2024-09-18T08:10:27.840 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:10:27.841 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:10:27.841 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4' 2024-09-18T08:10:27.942 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7 2024-09-18T08:10:27.977 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:10:27.977 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:10:27.978 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7' 2024-09-18T08:10:28.078 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5 2024-09-18T08:10:28.108 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:10:28.109 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:10:28.109 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5' 2024-09-18T08:10:28.210 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7 2024-09-18T08:10:28.242 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:10:28.243 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:10:28.243 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7' 2024-09-18T08:10:28.344 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3 2024-09-18T08:10:28.379 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:10:28.380 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:10:28.380 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3' 2024-09-18T08:10:28.481 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7 2024-09-18T08:10:28.510 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:10:28.511 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:10:28.511 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7' 2024-09-18T08:10:28.614 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2 2024-09-18T08:10:28.642 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:10:28.643 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:10:28.643 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2' 2024-09-18T08:10:28.730 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:10:28 smithi007 ceph-mon[29459]: pgmap v368: 158 pgs: 158 active+clean; 2.4 GiB data, 10 GiB used, 705 GiB / 715 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 1.05k op/s; 24 B/s, 0 objects/s recovering 2024-09-18T08:10:28.730 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:10:28 smithi007 ceph-mon[39249]: pgmap v368: 158 pgs: 158 active+clean; 2.4 GiB data, 10 GiB used, 705 GiB / 715 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 1.05k op/s; 24 B/s, 0 objects/s recovering 2024-09-18T08:10:28.744 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2 2024-09-18T08:10:28.779 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:10:28.780 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:10:28.781 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2' 2024-09-18T08:10:28.881 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6 2024-09-18T08:10:28.905 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:10:28 smithi050 ceph-mon[32371]: pgmap v368: 158 pgs: 158 active+clean; 2.4 GiB data, 10 GiB used, 705 GiB / 715 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 1.05k op/s; 24 B/s, 0 objects/s recovering 2024-09-18T08:10:28.917 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:10:28.917 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:10:28.918 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6' 2024-09-18T08:10:29.018 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1 2024-09-18T08:10:29.057 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:10:29.057 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:10:29.058 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1' 2024-09-18T08:10:29.158 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3 2024-09-18T08:10:29.196 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:10:29.197 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:10:29.197 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3' 2024-09-18T08:10:29.298 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1 2024-09-18T08:10:29.336 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:10:29.337 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:10:29.338 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1' 2024-09-18T08:10:29.406 INFO:journalctl@ceph.iscsi.iscsi.a.smithi050.stdout:Sep 18 08:10:29 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-iscsi-iscsi-a[57782]: debug there is no tcmu-runner data available 2024-09-18T08:10:29.426 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid 779349ca-7593-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-18T08:10:29.439 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2 2024-09-18T08:10:29.519 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:10:29.520 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:10:29.520 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2' 2024-09-18T08:10:29.621 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7 2024-09-18T08:10:29.648 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:10:29.649 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:10:29.649 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7' 2024-09-18T08:10:29.658 INFO:teuthology.orchestra.run.smithi007.stdout:{"pg_ready":true,"pg_map":{"version":366,"stamp":"2024-09-18T08:10:23.234845+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":2965248563,"num_objects":45437,"num_object_clones":0,"num_object_copies":136311,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":45437,"num_whiteouts":0,"num_read":928,"num_read_kb":680,"num_write":156698,"num_write_kb":6447982,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24491,"num_bytes_recovered":1588862382,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":79956,"ondisk_log_size":79956,"up":474,"acting":474,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":479,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":11419968,"kb_used_data":9430880,"kb_used_omap":12,"kb_used_meta":1988851,"kb_avail":738541248,"statfs":{"total":767960285184,"available":756266237952,"internally_reserved":0,"allocated":9657221120,"data_stored":9654189165,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12750,"internal_metadata":2036583986},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,5,3,7,6],"upper_bound":32},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-388300800,"num_objects":-5925,"num_object_clones":0,"num_object_copies":-17775,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-51,"num_objects_unfound":0,"num_objects_dirty":-5925,"num_whiteouts":0,"num_read":5,"num_read_kb":5,"num_write":5925,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":56,"num_bytes_recovered":3342556,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.001928"},"pg_stats":[{"pgid":"3.1f","version":"0'0","reported_seq":372,"reported_epoch":222,"state":"active+clean","last_fresh":"2024-09-18T08:10:18.362042+0000","last_change":"2024-09-18T08:10:15.850596+0000","last_active":"2024-09-18T08:10:18.362042+0000","last_peered":"2024-09-18T08:10:18.362042+0000","last_clean":"2024-09-18T08:10:18.362042+0000","last_became_active":"2024-09-18T08:08:52.697499+0000","last_became_peered":"2024-09-18T08:08:52.697499+0000","last_unstale":"2024-09-18T08:10:18.362042+0000","last_undegraded":"2024-09-18T08:10:18.362042+0000","last_fullsized":"2024-09-18T08:10:18.362042+0000","mapping_epoch":198,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":199,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:10:15.850515+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:46.860092+0000","last_clean_scrub_stamp":"2024-09-18T08:10:15.850515+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:11:28.482967+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"69'9","reported_seq":112,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-09-18T08:10:22.392610+0000","last_change":"2024-09-18T08:10:19.271321+0000","last_active":"2024-09-18T08:10:22.392610+0000","last_peered":"2024-09-18T08:10:22.392610+0000","last_clean":"2024-09-18T08:10:22.392610+0000","last_became_active":"2024-09-18T08:08:18.537856+0000","last_became_peered":"2024-09-18T08:08:18.537856+0000","last_unstale":"2024-09-18T08:10:22.392610+0000","last_undegraded":"2024-09-18T08:10:22.392610+0000","last_fullsized":"2024-09-18T08:10:22.392610+0000","mapping_epoch":181,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":182,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'9","last_scrub_stamp":"2024-09-18T08:10:19.271241+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_clean_scrub_stamp":"2024-09-18T08:10:19.271241+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:11:37.987398+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":16,"num_read_kb":10,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":358,"reported_epoch":222,"state":"active+clean","last_fresh":"2024-09-18T08:10:18.358977+0000","last_change":"2024-09-18T08:10:15.603362+0000","last_active":"2024-09-18T08:10:18.358977+0000","last_peered":"2024-09-18T08:10:18.358977+0000","last_clean":"2024-09-18T08:10:18.358977+0000","last_became_active":"2024-09-18T08:02:51.907140+0000","last_became_peered":"2024-09-18T08:02:51.907140+0000","last_unstale":"2024-09-18T08:10:18.358977+0000","last_undegraded":"2024-09-18T08:10:18.358977+0000","last_fullsized":"2024-09-18T08:10:18.358977+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:10:15.603289+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:50.886830+0000","last_clean_scrub_stamp":"2024-09-18T08:10:15.603289+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:11:27.298716+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":94,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-09-18T08:10:22.393633+0000","last_change":"2024-09-18T08:08:18.545192+0000","last_active":"2024-09-18T08:10:22.393633+0000","last_peered":"2024-09-18T08:10:22.393633+0000","last_clean":"2024-09-18T08:10:22.393633+0000","last_became_active":"2024-09-18T08:08:18.540705+0000","last_became_peered":"2024-09-18T08:08:18.540705+0000","last_unstale":"2024-09-18T08:10:22.393633+0000","last_undegraded":"2024-09-18T08:10:22.393633+0000","last_fullsized":"2024-09-18T08:10:22.393633+0000","mapping_epoch":181,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":182,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:07:08.451865+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_clean_scrub_stamp":"2024-09-18T08:07:08.451865+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":352,"reported_epoch":222,"state":"active+clean","last_fresh":"2024-09-18T08:10:18.364224+0000","last_change":"2024-09-18T08:07:13.244647+0000","last_active":"2024-09-18T08:10:18.364224+0000","last_peered":"2024-09-18T08:10:18.364224+0000","last_clean":"2024-09-18T08:10:18.364224+0000","last_became_active":"2024-09-18T08:06:24.128511+0000","last_became_peered":"2024-09-18T08:06:24.128511+0000","last_unstale":"2024-09-18T08:10:18.364224+0000","last_undegraded":"2024-09-18T08:10:18.364224+0000","last_fullsized":"2024-09-18T08:10:18.364224+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:07:13.244555+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_clean_scrub_stamp":"2024-09-18T08:07:13.244555+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:10:55.479885+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":373,"reported_epoch":222,"state":"active+clean","last_fresh":"2024-09-18T08:10:18.362789+0000","last_change":"2024-09-18T08:10:08.473695+0000","last_active":"2024-09-18T08:10:18.362789+0000","last_peered":"2024-09-18T08:10:18.362789+0000","last_clean":"2024-09-18T08:10:18.362789+0000","last_became_active":"2024-09-18T08:06:24.126194+0000","last_became_peered":"2024-09-18T08:06:24.126194+0000","last_unstale":"2024-09-18T08:10:18.362789+0000","last_undegraded":"2024-09-18T08:10:18.362789+0000","last_fullsized":"2024-09-18T08:10:18.362789+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:10:08.473626+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:46.860092+0000","last_clean_scrub_stamp":"2024-09-18T08:10:08.473626+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:11:38.009160+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"69'15","reported_seq":364,"reported_epoch":222,"state":"active+clean","last_fresh":"2024-09-18T08:10:18.361967+0000","last_change":"2024-09-18T08:09:32.575836+0000","last_active":"2024-09-18T08:10:18.361967+0000","last_peered":"2024-09-18T08:10:18.361967+0000","last_clean":"2024-09-18T08:10:18.361967+0000","last_became_active":"2024-09-18T08:02:49.886106+0000","last_became_peered":"2024-09-18T08:02:49.886106+0000","last_unstale":"2024-09-18T08:10:18.361967+0000","last_undegraded":"2024-09-18T08:10:18.361967+0000","last_fullsized":"2024-09-18T08:10:18.361967+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'15","last_scrub_stamp":"2024-09-18T08:09:32.575741+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_clean_scrub_stamp":"2024-09-18T08:09:32.575741+0000","objects_scrubbed":7,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:10:59.141685+0000","scrub_duration":40,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":94,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-09-18T08:10:22.392612+0000","last_change":"2024-09-18T08:08:18.543316+0000","last_active":"2024-09-18T08:10:22.392612+0000","last_peered":"2024-09-18T08:10:22.392612+0000","last_clean":"2024-09-18T08:10:22.392612+0000","last_became_active":"2024-09-18T08:08:18.538645+0000","last_became_peered":"2024-09-18T08:08:18.538645+0000","last_unstale":"2024-09-18T08:10:22.392612+0000","last_undegraded":"2024-09-18T08:10:22.392612+0000","last_fullsized":"2024-09-18T08:10:22.392612+0000","mapping_epoch":181,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":182,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:08:09.142188+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:50.886830+0000","last_clean_scrub_stamp":"2024-09-18T08:08:09.142188+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":376,"reported_epoch":221,"state":"active+clean","last_fresh":"2024-09-18T08:10:17.354297+0000","last_change":"2024-09-18T08:08:18.548588+0000","last_active":"2024-09-18T08:10:17.354297+0000","last_peered":"2024-09-18T08:10:17.354297+0000","last_clean":"2024-09-18T08:10:17.354297+0000","last_became_active":"2024-09-18T08:08:18.548203+0000","last_became_peered":"2024-09-18T08:08:18.548203+0000","last_unstale":"2024-09-18T08:10:17.354297+0000","last_undegraded":"2024-09-18T08:10:17.354297+0000","last_fullsized":"2024-09-18T08:10:17.354297+0000","mapping_epoch":181,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":182,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:06:14.027077+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:46.860092+0000","last_clean_scrub_stamp":"2024-09-18T08:06:14.027077+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"69'9","reported_seq":112,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-09-18T08:10:22.393801+0000","last_change":"2024-09-18T08:10:16.220584+0000","last_active":"2024-09-18T08:10:22.393801+0000","last_peered":"2024-09-18T08:10:22.393801+0000","last_clean":"2024-09-18T08:10:22.393801+0000","last_became_active":"2024-09-18T08:08:18.550960+0000","last_became_peered":"2024-09-18T08:08:18.550960+0000","last_unstale":"2024-09-18T08:10:22.393801+0000","last_undegraded":"2024-09-18T08:10:22.393801+0000","last_fullsized":"2024-09-18T08:10:22.393801+0000","mapping_epoch":181,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":182,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'9","last_scrub_stamp":"2024-09-18T08:10:16.220503+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_clean_scrub_stamp":"2024-09-18T08:10:16.220503+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:11:28.084018+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":16,"num_read_kb":10,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":339,"reported_epoch":221,"state":"active+clean","last_fresh":"2024-09-18T08:10:17.353844+0000","last_change":"2024-09-18T08:05:33.935844+0000","last_active":"2024-09-18T08:10:17.353844+0000","last_peered":"2024-09-18T08:10:17.353844+0000","last_clean":"2024-09-18T08:10:17.353844+0000","last_became_active":"2024-09-18T08:02:51.907296+0000","last_became_peered":"2024-09-18T08:02:51.907296+0000","last_unstale":"2024-09-18T08:10:17.353844+0000","last_undegraded":"2024-09-18T08:10:17.353844+0000","last_fullsized":"2024-09-18T08:10:17.353844+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:05:33.935798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:50.886830+0000","last_clean_scrub_stamp":"2024-09-18T08:05:33.935798+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":339,"reported_epoch":222,"state":"active+clean","last_fresh":"2024-09-18T08:10:18.360479+0000","last_change":"2024-09-18T08:08:14.646073+0000","last_active":"2024-09-18T08:10:18.360479+0000","last_peered":"2024-09-18T08:10:18.360479+0000","last_clean":"2024-09-18T08:10:18.360479+0000","last_became_active":"2024-09-18T08:02:53.913519+0000","last_became_peered":"2024-09-18T08:02:53.913519+0000","last_unstale":"2024-09-18T08:10:18.360479+0000","last_undegraded":"2024-09-18T08:10:18.360479+0000","last_fullsized":"2024-09-18T08:10:18.360479+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:08:14.645993+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_clean_scrub_stamp":"2024-09-18T08:08:14.645993+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":354,"reported_epoch":221,"state":"active+clean","last_fresh":"2024-09-18T08:10:17.353597+0000","last_change":"2024-09-18T08:08:18.624260+0000","last_active":"2024-09-18T08:10:17.353597+0000","last_peered":"2024-09-18T08:10:17.353597+0000","last_clean":"2024-09-18T08:10:17.353597+0000","last_became_active":"2024-09-18T08:08:18.623757+0000","last_became_peered":"2024-09-18T08:08:18.623757+0000","last_unstale":"2024-09-18T08:10:17.353597+0000","last_undegraded":"2024-09-18T08:10:17.353597+0000","last_fullsized":"2024-09-18T08:10:17.353597+0000","mapping_epoch":181,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":182,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:04:32.849119+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:46.860092+0000","last_clean_scrub_stamp":"2024-09-18T08:04:32.849119+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1b","version":"69'5","reported_seq":100,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-09-18T08:10:22.393136+0000","last_change":"2024-09-18T08:08:18.651427+0000","last_active":"2024-09-18T08:10:22.393136+0000","last_peered":"2024-09-18T08:10:22.393136+0000","last_clean":"2024-09-18T08:10:22.393136+0000","last_became_active":"2024-09-18T08:08:18.567126+0000","last_became_peered":"2024-09-18T08:08:18.567126+0000","last_unstale":"2024-09-18T08:10:22.393136+0000","last_undegraded":"2024-09-18T08:10:22.393136+0000","last_fullsized":"2024-09-18T08:10:22.393136+0000","mapping_epoch":181,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":182,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'5","last_scrub_stamp":"2024-09-18T08:06:09.465979+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_clean_scrub_stamp":"2024-09-18T08:06:09.465979+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":10,"num_read_kb":6,"num_write":6,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":382,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-09-18T08:10:22.414023+0000","last_change":"2024-09-18T08:10:16.664655+0000","last_active":"2024-09-18T08:10:22.414023+0000","last_peered":"2024-09-18T08:10:22.414023+0000","last_clean":"2024-09-18T08:10:22.414023+0000","last_became_active":"2024-09-18T08:08:18.524932+0000","last_became_peered":"2024-09-18T08:08:18.524932+0000","last_unstale":"2024-09-18T08:10:22.414023+0000","last_undegraded":"2024-09-18T08:10:22.414023+0000","last_fullsized":"2024-09-18T08:10:22.414023+0000","mapping_epoch":181,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":182,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:10:16.664597+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:50.886830+0000","last_clean_scrub_stamp":"2024-09-18T08:10:16.664597+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:11:29.391762+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":337,"reported_epoch":221,"state":"active+clean","last_fresh":"2024-09-18T08:10:17.353214+0000","last_change":"2024-09-18T08:06:44.518882+0000","last_active":"2024-09-18T08:10:17.353214+0000","last_peered":"2024-09-18T08:10:17.353214+0000","last_clean":"2024-09-18T08:10:17.353214+0000","last_became_active":"2024-09-18T08:02:53.911648+0000","last_became_peered":"2024-09-18T08:02:53.911648+0000","last_unstale":"2024-09-18T08:10:17.353214+0000","last_undegraded":"2024-09-18T08:10:17.353214+0000","last_fullsized":"2024-09-18T08:10:17.353214+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:06:44.518835+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_clean_scrub_stamp":"2024-09-18T08:06:44.518835+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":95,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-09-18T08:10:22.393728+0000","last_change":"2024-09-18T08:08:18.544203+0000","last_active":"2024-09-18T08:10:22.393728+0000","last_peered":"2024-09-18T08:10:22.393728+0000","last_clean":"2024-09-18T08:10:22.393728+0000","last_became_active":"2024-09-18T08:08:18.539604+0000","last_became_peered":"2024-09-18T08:08:18.539604+0000","last_unstale":"2024-09-18T08:10:22.393728+0000","last_undegraded":"2024-09-18T08:10:22.393728+0000","last_fullsized":"2024-09-18T08:10:22.393728+0000","mapping_epoch":181,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":182,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:05:40.477965+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_clean_scrub_stamp":"2024-09-18T08:05:40.477965+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":364,"reported_epoch":222,"state":"active+clean","last_fresh":"2024-09-18T08:10:18.361673+0000","last_change":"2024-09-18T08:08:18.524124+0000","last_active":"2024-09-18T08:10:18.361673+0000","last_peered":"2024-09-18T08:10:18.361673+0000","last_clean":"2024-09-18T08:10:18.361673+0000","last_became_active":"2024-09-18T08:08:18.523611+0000","last_became_peered":"2024-09-18T08:08:18.523611+0000","last_unstale":"2024-09-18T08:10:18.361673+0000","last_undegraded":"2024-09-18T08:10:18.361673+0000","last_fullsized":"2024-09-18T08:10:18.361673+0000","mapping_epoch":181,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":182,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:05:27.476144+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:46.860092+0000","last_clean_scrub_stamp":"2024-09-18T08:05:27.476144+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:10:23.059202+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1c","version":"69'15","reported_seq":74,"reported_epoch":222,"state":"active+clean","last_fresh":"2024-09-18T08:10:18.355750+0000","last_change":"2024-09-18T08:08:52.708963+0000","last_active":"2024-09-18T08:10:18.355750+0000","last_peered":"2024-09-18T08:10:18.355750+0000","last_clean":"2024-09-18T08:10:18.355750+0000","last_became_active":"2024-09-18T08:08:52.690788+0000","last_became_peered":"2024-09-18T08:08:52.690788+0000","last_unstale":"2024-09-18T08:10:18.355750+0000","last_undegraded":"2024-09-18T08:10:18.355750+0000","last_fullsized":"2024-09-18T08:10:18.355750+0000","mapping_epoch":198,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":199,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'15","last_scrub_stamp":"2024-09-18T08:07:39.530340+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_clean_scrub_stamp":"2024-09-18T08:07:39.530340+0000","objects_scrubbed":7,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":371,"reported_epoch":222,"state":"active+clean","last_fresh":"2024-09-18T08:10:18.357308+0000","last_change":"2024-09-18T08:09:11.693711+0000","last_active":"2024-09-18T08:10:18.357308+0000","last_peered":"2024-09-18T08:10:18.357308+0000","last_clean":"2024-09-18T08:10:18.357308+0000","last_became_active":"2024-09-18T08:08:18.538505+0000","last_became_peered":"2024-09-18T08:08:18.538505+0000","last_unstale":"2024-09-18T08:10:18.357308+0000","last_undegraded":"2024-09-18T08:10:18.357308+0000","last_fullsized":"2024-09-18T08:10:18.357308+0000","mapping_epoch":181,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":182,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:09:11.693643+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:50.886830+0000","last_clean_scrub_stamp":"2024-09-18T08:09:11.693643+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:10:34.429057+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":353,"reported_epoch":222,"state":"active+clean","last_fresh":"2024-09-18T08:10:18.357202+0000","last_change":"2024-09-18T08:09:36.592348+0000","last_active":"2024-09-18T08:10:18.357202+0000","last_peered":"2024-09-18T08:10:18.357202+0000","last_clean":"2024-09-18T08:10:18.357202+0000","last_became_active":"2024-09-18T08:06:24.128659+0000","last_became_peered":"2024-09-18T08:06:24.128659+0000","last_unstale":"2024-09-18T08:10:18.357202+0000","last_undegraded":"2024-09-18T08:10:18.357202+0000","last_fullsized":"2024-09-18T08:10:18.357202+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:09:36.592281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_clean_scrub_stamp":"2024-09-18T08:09:36.592281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:10:46.573780+0000","scrub_duration":16,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":94,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-09-18T08:10:22.393064+0000","last_change":"2024-09-18T08:08:18.536559+0000","last_active":"2024-09-18T08:10:22.393064+0000","last_peered":"2024-09-18T08:10:22.393064+0000","last_clean":"2024-09-18T08:10:22.393064+0000","last_became_active":"2024-09-18T08:08:18.536084+0000","last_became_peered":"2024-09-18T08:08:18.536084+0000","last_unstale":"2024-09-18T08:10:22.393064+0000","last_undegraded":"2024-09-18T08:10:22.393064+0000","last_fullsized":"2024-09-18T08:10:22.393064+0000","mapping_epoch":181,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":182,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:07:13.543722+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:46.860092+0000","last_clean_scrub_stamp":"2024-09-18T08:07:13.543722+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"69'12","reported_seq":415,"reported_epoch":222,"state":"active+clean","last_fresh":"2024-09-18T08:10:18.362428+0000","last_change":"2024-09-18T08:10:12.443637+0000","last_active":"2024-09-18T08:10:18.362428+0000","last_peered":"2024-09-18T08:10:18.362428+0000","last_clean":"2024-09-18T08:10:18.362428+0000","last_became_active":"2024-09-18T08:08:18.543675+0000","last_became_peered":"2024-09-18T08:08:18.543675+0000","last_unstale":"2024-09-18T08:10:18.362428+0000","last_undegraded":"2024-09-18T08:10:18.362428+0000","last_fullsized":"2024-09-18T08:10:18.362428+0000","mapping_epoch":181,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":182,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'12","last_scrub_stamp":"2024-09-18T08:10:12.443558+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_clean_scrub_stamp":"2024-09-18T08:10:12.443558+0000","objects_scrubbed":5,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:11:22.178344+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":25,"num_read_kb":16,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":94,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-09-18T08:10:22.393531+0000","last_change":"2024-09-18T08:08:18.532482+0000","last_active":"2024-09-18T08:10:22.393531+0000","last_peered":"2024-09-18T08:10:22.393531+0000","last_clean":"2024-09-18T08:10:22.393531+0000","last_became_active":"2024-09-18T08:08:18.531438+0000","last_became_peered":"2024-09-18T08:08:18.531438+0000","last_unstale":"2024-09-18T08:10:22.393531+0000","last_undegraded":"2024-09-18T08:10:22.393531+0000","last_fullsized":"2024-09-18T08:10:22.393531+0000","mapping_epoch":181,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":182,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:06:12.434179+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:50.886830+0000","last_clean_scrub_stamp":"2024-09-18T08:06:12.434179+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1c","version":"69'1","reported_seq":179,"reported_epoch":221,"state":"active+clean","last_fresh":"2024-09-18T08:10:17.353379+0000","last_change":"2024-09-18T08:06:46.142143+0000","last_active":"2024-09-18T08:10:17.353379+0000","last_peered":"2024-09-18T08:10:17.353379+0000","last_clean":"2024-09-18T08:10:17.353379+0000","last_became_active":"2024-09-18T08:06:46.141661+0000","last_became_peered":"2024-09-18T08:06:46.141661+0000","last_unstale":"2024-09-18T08:10:17.353379+0000","last_undegraded":"2024-09-18T08:10:17.353379+0000","last_fullsized":"2024-09-18T08:10:17.353379+0000","mapping_epoch":143,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":144,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'1","last_scrub_stamp":"2024-09-18T08:05:36.776369+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_clean_scrub_stamp":"2024-09-18T08:05:36.776369+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":403,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,2],"acting":[5,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"62'1","reported_seq":381,"reported_epoch":222,"state":"active+clean","last_fresh":"2024-09-18T08:10:18.359091+0000","last_change":"2024-09-18T08:09:29.633875+0000","last_active":"2024-09-18T08:10:18.359091+0000","last_peered":"2024-09-18T08:10:18.359091+0000","last_clean":"2024-09-18T08:10:18.359091+0000","last_became_active":"2024-09-18T08:08:18.559125+0000","last_became_peered":"2024-09-18T08:08:18.559125+0000","last_unstale":"2024-09-18T08:10:18.359091+0000","last_undegraded":"2024-09-18T08:10:18.359091+0000","last_fullsized":"2024-09-18T08:10:18.359091+0000","mapping_epoch":181,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":182,"parent":"0.0","parent_split_bits":0,"last_scrub":"62'1","last_scrub_stamp":"2024-09-18T08:09:29.633520+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:46.860092+0000","last_clean_scrub_stamp":"2024-09-18T08:09:29.633520+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:10:38.890672+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":92,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1e","version":"69'10","reported_seq":345,"reported_epoch":222,"state":"active+clean","last_fresh":"2024-09-18T08:10:18.366263+0000","last_change":"2024-09-18T08:02:49.892755+0000","last_active":"2024-09-18T08:10:18.366263+0000","last_peered":"2024-09-18T08:10:18.366263+0000","last_clean":"2024-09-18T08:10:18.366263+0000","last_became_active":"2024-09-18T08:02:49.892024+0000","last_became_peered":"2024-09-18T08:02:49.892024+0000","last_unstale":"2024-09-18T08:10:18.366263+0000","last_undegraded":"2024-09-18T08:10:18.366263+0000","last_fullsized":"2024-09-18T08:10:18.366263+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_clean_scrub_stamp":"2024-09-18T08:02:48.870718+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1f","version":"182'13","reported_seq":125,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-09-18T08:10:23.017306+0000","last_change":"2024-09-18T08:08:18.627670+0000","last_active":"2024-09-18T08:10:23.017306+0000","last_peered":"2024-09-18T08:10:23.017306+0000","last_clean":"2024-09-18T08:10:23.017306+0000","last_became_active":"2024-09-18T08:08:18.543095+0000","last_became_peered":"2024-09-18T08:08:18.543095+0000","last_unstale":"2024-09-18T08:10:23.017306+0000","last_undegraded":"2024-09-18T08:10:23.017306+0000","last_fullsized":"2024-09-18T08:10:23.017306+0000","mapping_epoch":181,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":182,"parent":"0.0","parent_split_bits":0,"last_scrub":"173'12","last_scrub_stamp":"2024-09-18T08:08:07.553075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:50.886830+0000","last_clean_scrub_stamp":"2024-09-18T08:08:07.553075+0000","objects_scrubbed":0,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,7],"acting":[4,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.3","version":"222'9981","reported_seq":10179,"reported_epoch":222,"state":"active+clean","last_fresh":"2024-09-18T08:10:18.696606+0000","last_change":"2024-09-18T08:09:28.263522+0000","last_active":"2024-09-18T08:10:18.696606+0000","last_peered":"2024-09-18T08:10:18.696606+0000","last_clean":"2024-09-18T08:10:18.696606+0000","last_became_active":"2024-09-18T08:07:53.672797+0000","last_became_peered":"2024-09-18T08:07:53.672797+0000","last_unstale":"2024-09-18T08:10:18.696606+0000","last_undegraded":"2024-09-18T08:10:18.696606+0000","last_fullsized":"2024-09-18T08:10:18.696606+0000","mapping_epoch":165,"log_start":"202'5200","ondisk_log_start":"202'5200","created":165,"last_epoch_clean":166,"parent":"0.0","parent_split_bits":0,"last_scrub":"211'6448","last_scrub_stamp":"2024-09-18T08:09:28.263434+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:07:52.307205+0000","last_clean_scrub_stamp":"2024-09-18T08:09:28.263434+0000","objects_scrubbed":6448,"log_size":4781,"log_dups_size":0,"ondisk_log_size":4781,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:10:29.843317+0000","scrub_duration":646,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":191037440,"num_objects":2915,"num_object_clones":0,"num_object_copies":8745,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2915,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9981,"num_write_kb":412672,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6],"acting":[1,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.a","version":"74'79","reported_seq":161,"reported_epoch":221,"state":"active+clean","last_fresh":"2024-09-18T08:10:17.353174+0000","last_change":"2024-09-18T08:08:57.463205+0000","last_active":"2024-09-18T08:10:17.353174+0000","last_peered":"2024-09-18T08:10:17.353174+0000","last_clean":"2024-09-18T08:10:17.353174+0000","last_became_active":"2024-09-18T08:07:26.394584+0000","last_became_peered":"2024-09-18T08:07:26.394584+0000","last_unstale":"2024-09-18T08:10:17.353174+0000","last_undegraded":"2024-09-18T08:10:17.353174+0000","last_fullsized":"2024-09-18T08:10:17.353174+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":152,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":4,"last_scrub":"74'79","last_scrub_stamp":"2024-09-18T08:08:57.463113+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:00:11.893742+0000","last_clean_scrub_stamp":"2024-09-18T08:08:57.463113+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,6],"acting":[5,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.f","version":"69'15","reported_seq":417,"reported_epoch":222,"state":"active+clean","last_fresh":"2024-09-18T08:10:18.361883+0000","last_change":"2024-09-18T08:08:56.669262+0000","last_active":"2024-09-18T08:10:18.361883+0000","last_peered":"2024-09-18T08:10:18.361883+0000","last_clean":"2024-09-18T08:10:18.361883+0000","last_became_active":"2024-09-18T08:08:18.540211+0000","last_became_peered":"2024-09-18T08:08:18.540211+0000","last_unstale":"2024-09-18T08:10:18.361883+0000","last_undegraded":"2024-09-18T08:10:18.361883+0000","last_fullsized":"2024-09-18T08:10:18.361883+0000","mapping_epoch":181,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":182,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'15","last_scrub_stamp":"2024-09-18T08:08:56.668397+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_clean_scrub_stamp":"2024-09-18T08:08:56.668397+0000","objects_scrubbed":7,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:10:22.470704+0000","scrub_duration":36,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":351,"reported_epoch":222,"state":"active+clean","last_fresh":"2024-09-18T08:10:18.363597+0000","last_change":"2024-09-18T08:08:18.389332+0000","last_active":"2024-09-18T08:10:18.363597+0000","last_peered":"2024-09-18T08:10:18.363597+0000","last_clean":"2024-09-18T08:10:18.363597+0000","last_became_active":"2024-09-18T08:02:47.879328+0000","last_became_peered":"2024-09-18T08:02:47.879328+0000","last_unstale":"2024-09-18T08:10:18.363597+0000","last_undegraded":"2024-09-18T08:10:18.363597+0000","last_fullsized":"2024-09-18T08:10:18.363597+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:08:18.389273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:46.860092+0000","last_clean_scrub_stamp":"2024-09-18T08:08:18.389273+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.e","version":"182'12","reported_seq":124,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-09-18T08:10:22.393617+0000","last_change":"2024-09-18T08:08:18.560884+0000","last_active":"2024-09-18T08:10:22.393617+0000","last_peered":"2024-09-18T08:10:22.393617+0000","last_clean":"2024-09-18T08:10:22.393617+0000","last_became_active":"2024-09-18T08:08:18.525147+0000","last_became_peered":"2024-09-18T08:08:18.525147+0000","last_unstale":"2024-09-18T08:10:22.393617+0000","last_undegraded":"2024-09-18T08:10:22.393617+0000","last_fullsized":"2024-09-18T08:10:22.393617+0000","mapping_epoch":181,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":182,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'8","last_scrub_stamp":"2024-09-18T08:06:19.459375+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:50.886830+0000","last_clean_scrub_stamp":"2024-09-18T08:06:19.459375+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":345,"reported_epoch":221,"state":"active+clean","last_fresh":"2024-09-18T08:10:17.353707+0000","last_change":"2024-09-18T08:07:19.714477+0000","last_active":"2024-09-18T08:10:17.353707+0000","last_peered":"2024-09-18T08:10:17.353707+0000","last_clean":"2024-09-18T08:10:17.353707+0000","last_became_active":"2024-09-18T08:02:53.909834+0000","last_became_peered":"2024-09-18T08:02:53.909834+0000","last_unstale":"2024-09-18T08:10:17.353707+0000","last_undegraded":"2024-09-18T08:10:17.353707+0000","last_fullsized":"2024-09-18T08:10:17.353707+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:07:19.714430+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_clean_scrub_stamp":"2024-09-18T08:07:19.714430+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.c","version":"222'9624","reported_seq":9789,"reported_epoch":222,"state":"active+clean","last_fresh":"2024-09-18T08:10:18.821167+0000","last_change":"2024-09-18T08:10:11.144727+0000","last_active":"2024-09-18T08:10:18.821167+0000","last_peered":"2024-09-18T08:10:18.821167+0000","last_clean":"2024-09-18T08:10:18.821167+0000","last_became_active":"2024-09-18T08:07:53.669708+0000","last_became_peered":"2024-09-18T08:07:53.669708+0000","last_unstale":"2024-09-18T08:10:18.821167+0000","last_undegraded":"2024-09-18T08:10:18.821167+0000","last_fullsized":"2024-09-18T08:10:18.821167+0000","mapping_epoch":165,"log_start":"197'4200","ondisk_log_start":"197'4200","created":165,"last_epoch_clean":166,"parent":"0.0","parent_split_bits":0,"last_scrub":"217'9165","last_scrub_stamp":"2024-09-18T08:10:11.144412+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:07:52.307205+0000","last_clean_scrub_stamp":"2024-09-18T08:10:11.144412+0000","objects_scrubbed":3298,"log_size":5424,"log_dups_size":0,"ondisk_log_size":5424,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:11:37.247660+0000","scrub_duration":1357,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":183894016,"num_objects":2806,"num_object_clones":0,"num_object_copies":8418,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2806,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9624,"num_write_kb":397760,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"74'79","reported_seq":110,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-09-18T08:10:22.393504+0000","last_change":"2024-09-18T08:08:52.690758+0000","last_active":"2024-09-18T08:10:22.393504+0000","last_peered":"2024-09-18T08:10:22.393504+0000","last_clean":"2024-09-18T08:10:22.393504+0000","last_became_active":"2024-09-18T08:08:52.690344+0000","last_became_peered":"2024-09-18T08:08:52.690344+0000","last_unstale":"2024-09-18T08:10:22.393504+0000","last_undegraded":"2024-09-18T08:10:22.393504+0000","last_fullsized":"2024-09-18T08:10:22.393504+0000","mapping_epoch":198,"log_start":"0'0","ondisk_log_start":"0'0","created":152,"last_epoch_clean":199,"parent":"0.0","parent_split_bits":3,"last_scrub":"74'79","last_scrub_stamp":"2024-09-18T08:08:19.036836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:00:11.893742+0000","last_clean_scrub_stamp":"2024-09-18T08:08:19.036836+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,6],"acting":[4,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.0","version":"69'18","reported_seq":379,"reported_epoch":222,"state":"active+clean","last_fresh":"2024-09-18T08:10:18.363378+0000","last_change":"2024-09-18T08:06:44.301130+0000","last_active":"2024-09-18T08:10:18.363378+0000","last_peered":"2024-09-18T08:10:18.363378+0000","last_clean":"2024-09-18T08:10:18.363378+0000","last_became_active":"2024-09-18T08:02:49.886818+0000","last_became_peered":"2024-09-18T08:02:49.886818+0000","last_unstale":"2024-09-18T08:10:18.363378+0000","last_undegraded":"2024-09-18T08:10:18.363378+0000","last_fullsized":"2024-09-18T08:10:18.363378+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'18","last_scrub_stamp":"2024-09-18T08:06:44.300772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_clean_scrub_stamp":"2024-09-18T08:06:44.300772+0000","objects_scrubbed":8,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:10:52.429408+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":34,"num_read_kb":22,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":367,"reported_epoch":222,"state":"active+clean","last_fresh":"2024-09-18T08:10:18.365828+0000","last_change":"2024-09-18T08:10:09.469428+0000","last_active":"2024-09-18T08:10:18.365828+0000","last_peered":"2024-09-18T08:10:18.365828+0000","last_clean":"2024-09-18T08:10:18.365828+0000","last_became_active":"2024-09-18T08:02:47.880225+0000","last_became_peered":"2024-09-18T08:02:47.880225+0000","last_unstale":"2024-09-18T08:10:18.365828+0000","last_undegraded":"2024-09-18T08:10:18.365828+0000","last_fullsized":"2024-09-18T08:10:18.365828+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:10:09.469351+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:46.860092+0000","last_clean_scrub_stamp":"2024-09-18T08:10:09.469351+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:11:16.425431+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":95,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-09-18T08:10:22.392924+0000","last_change":"2024-09-18T08:08:18.530658+0000","last_active":"2024-09-18T08:10:22.392924+0000","last_peered":"2024-09-18T08:10:22.392924+0000","last_clean":"2024-09-18T08:10:22.392924+0000","last_became_active":"2024-09-18T08:08:18.530341+0000","last_became_peered":"2024-09-18T08:08:18.530341+0000","last_unstale":"2024-09-18T08:10:22.392924+0000","last_undegraded":"2024-09-18T08:10:22.392924+0000","last_fullsized":"2024-09-18T08:10:22.392924+0000","mapping_epoch":181,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":182,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:06:10.468119+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:50.886830+0000","last_clean_scrub_stamp":"2024-09-18T08:06:10.468119+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":103,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-09-18T08:10:22.393847+0000","last_change":"2024-09-18T08:10:17.226945+0000","last_active":"2024-09-18T08:10:22.393847+0000","last_peered":"2024-09-18T08:10:22.393847+0000","last_clean":"2024-09-18T08:10:22.393847+0000","last_became_active":"2024-09-18T08:08:18.543694+0000","last_became_peered":"2024-09-18T08:08:18.543694+0000","last_unstale":"2024-09-18T08:10:22.393847+0000","last_undegraded":"2024-09-18T08:10:22.393847+0000","last_fullsized":"2024-09-18T08:10:22.393847+0000","mapping_epoch":181,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":182,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:10:17.226867+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_clean_scrub_stamp":"2024-09-18T08:10:17.226867+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:11:38.990188+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.d","version":"221'9612","reported_seq":9736,"reported_epoch":221,"state":"active+clean","last_fresh":"2024-09-18T08:10:18.299324+0000","last_change":"2024-09-18T08:07:53.671291+0000","last_active":"2024-09-18T08:10:18.299324+0000","last_peered":"2024-09-18T08:10:18.299324+0000","last_clean":"2024-09-18T08:10:18.299324+0000","last_became_active":"2024-09-18T08:07:53.670773+0000","last_became_peered":"2024-09-18T08:07:53.670773+0000","last_unstale":"2024-09-18T08:10:18.299324+0000","last_undegraded":"2024-09-18T08:10:18.299324+0000","last_fullsized":"2024-09-18T08:10:18.299324+0000","mapping_epoch":165,"log_start":"200'4500","ondisk_log_start":"200'4500","created":165,"last_epoch_clean":166,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:07:52.307205+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:07:52.307205+0000","last_clean_scrub_stamp":"2024-09-18T08:07:52.307205+0000","objects_scrubbed":0,"log_size":5112,"log_dups_size":0,"ondisk_log_size":5112,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":189857816,"num_objects":2898,"num_object_clones":0,"num_object_copies":8694,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2898,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":9612,"num_write_kb":400257,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,2],"acting":[5,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.4","version":"74'79","reported_seq":169,"reported_epoch":222,"state":"active+clean","last_fresh":"2024-09-18T08:10:18.356956+0000","last_change":"2024-09-18T08:08:59.376417+0000","last_active":"2024-09-18T08:10:18.356956+0000","last_peered":"2024-09-18T08:10:18.356956+0000","last_clean":"2024-09-18T08:10:18.356956+0000","last_became_active":"2024-09-18T08:07:26.398930+0000","last_became_peered":"2024-09-18T08:07:26.398930+0000","last_unstale":"2024-09-18T08:10:18.356956+0000","last_undegraded":"2024-09-18T08:10:18.356956+0000","last_fullsized":"2024-09-18T08:10:18.356956+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":152,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":3,"last_scrub":"74'79","last_scrub_stamp":"2024-09-18T08:08:59.376337+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:00:11.893742+0000","last_clean_scrub_stamp":"2024-09-18T08:08:59.376337+0000","objects_scrubbed":1,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:10:26.641607+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":327680,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,7],"acting":[3,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"69'14","reported_seq":111,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-09-18T08:10:22.392771+0000","last_change":"2024-09-18T08:08:18.617734+0000","last_active":"2024-09-18T08:10:22.392771+0000","last_peered":"2024-09-18T08:10:22.392771+0000","last_clean":"2024-09-18T08:10:22.392771+0000","last_became_active":"2024-09-18T08:08:18.549671+0000","last_became_peered":"2024-09-18T08:08:18.549671+0000","last_unstale":"2024-09-18T08:10:22.392771+0000","last_undegraded":"2024-09-18T08:10:22.392771+0000","last_fullsized":"2024-09-18T08:10:22.392771+0000","mapping_epoch":181,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":182,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'14","last_scrub_stamp":"2024-09-18T08:08:08.629246+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_clean_scrub_stamp":"2024-09-18T08:08:08.629246+0000","objects_scrubbed":0,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":14,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"62'2","reported_seq":383,"reported_epoch":222,"state":"active+clean","last_fresh":"2024-09-18T08:10:18.366561+0000","last_change":"2024-09-18T08:08:18.565279+0000","last_active":"2024-09-18T08:10:18.366561+0000","last_peered":"2024-09-18T08:10:18.366561+0000","last_clean":"2024-09-18T08:10:18.366561+0000","last_became_active":"2024-09-18T08:08:18.544872+0000","last_became_peered":"2024-09-18T08:08:18.544872+0000","last_unstale":"2024-09-18T08:10:18.366561+0000","last_undegraded":"2024-09-18T08:10:18.366561+0000","last_fullsized":"2024-09-18T08:10:18.366561+0000","mapping_epoch":181,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":182,"parent":"0.0","parent_split_bits":0,"last_scrub":"62'2","last_scrub_stamp":"2024-09-18T08:06:43.607152+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:46.860092+0000","last_clean_scrub_stamp":"2024-09-18T08:06:43.607152+0000","objects_scrubbed":2,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:10:26.053059+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1039,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":2078,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.0","version":"182'10","reported_seq":464,"reported_epoch":222,"state":"active+clean","last_fresh":"2024-09-18T08:10:18.363201+0000","last_change":"2024-09-18T08:09:51.467488+0000","last_active":"2024-09-18T08:10:18.363201+0000","last_peered":"2024-09-18T08:10:18.363201+0000","last_clean":"2024-09-18T08:10:18.363201+0000","last_became_active":"2024-09-18T08:08:18.541756+0000","last_became_peered":"2024-09-18T08:08:18.541756+0000","last_unstale":"2024-09-18T08:10:18.363201+0000","last_undegraded":"2024-09-18T08:10:18.363201+0000","last_fullsized":"2024-09-18T08:10:18.363201+0000","mapping_epoch":181,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":182,"parent":"0.0","parent_split_bits":0,"last_scrub":"182'10","last_scrub_stamp":"2024-09-18T08:09:51.467349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:50.886830+0000","last_clean_scrub_stamp":"2024-09-18T08:09:51.467349+0000","objects_scrubbed":1,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:10:51.919411+0000","scrub_duration":19,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":371,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-09-18T08:10:22.414380+0000","last_change":"2024-09-18T08:10:15.663125+0000","last_active":"2024-09-18T08:10:22.414380+0000","last_peered":"2024-09-18T08:10:22.414380+0000","last_clean":"2024-09-18T08:10:22.414380+0000","last_became_active":"2024-09-18T08:06:24.140303+0000","last_became_peered":"2024-09-18T08:06:24.140303+0000","last_unstale":"2024-09-18T08:10:22.414380+0000","last_undegraded":"2024-09-18T08:10:22.414380+0000","last_fullsized":"2024-09-18T08:10:22.414380+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:10:15.663047+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_clean_scrub_stamp":"2024-09-18T08:10:15.663047+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:11:20.567559+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.e","version":"222'9751","reported_seq":14112,"reported_epoch":222,"state":"active+clean","last_fresh":"2024-09-18T08:10:18.811341+0000","last_change":"2024-09-18T08:09:17.296111+0000","last_active":"2024-09-18T08:10:18.811341+0000","last_peered":"2024-09-18T08:10:18.811341+0000","last_clean":"2024-09-18T08:10:18.811341+0000","last_became_active":"2024-09-18T08:09:17.286866+0000","last_became_peered":"2024-09-18T08:09:17.286866+0000","last_unstale":"2024-09-18T08:10:18.811341+0000","last_undegraded":"2024-09-18T08:10:18.811341+0000","last_fullsized":"2024-09-18T08:10:18.811341+0000","mapping_epoch":207,"log_start":"197'4300","ondisk_log_start":"197'4300","created":165,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:07:52.307205+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:07:52.307205+0000","last_clean_scrub_stamp":"2024-09-18T08:07:52.307205+0000","objects_scrubbed":0,"log_size":5451,"log_dups_size":0,"ondisk_log_size":5451,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":197722112,"num_objects":3017,"num_object_clones":0,"num_object_copies":9051,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3017,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9751,"num_write_kb":408576,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2107,"num_bytes_recovered":138084352,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,4],"acting":[0,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.7","version":"74'79","reported_seq":164,"reported_epoch":222,"state":"active+clean","last_fresh":"2024-09-18T08:10:18.357703+0000","last_change":"2024-09-18T08:08:57.438699+0000","last_active":"2024-09-18T08:10:18.357703+0000","last_peered":"2024-09-18T08:10:18.357703+0000","last_clean":"2024-09-18T08:10:18.357703+0000","last_became_active":"2024-09-18T08:07:26.398289+0000","last_became_peered":"2024-09-18T08:07:26.398289+0000","last_unstale":"2024-09-18T08:10:18.357703+0000","last_undegraded":"2024-09-18T08:10:18.357703+0000","last_fullsized":"2024-09-18T08:10:18.357703+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":152,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":3,"last_scrub":"74'79","last_scrub_stamp":"2024-09-18T08:08:57.438600+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:00:11.893742+0000","last_clean_scrub_stamp":"2024-09-18T08:08:57.438600+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,7],"acting":[3,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.2","version":"69'10","reported_seq":415,"reported_epoch":222,"state":"active+clean","last_fresh":"2024-09-18T08:10:18.637715+0000","last_change":"2024-09-18T08:10:18.637715+0000","last_active":"2024-09-18T08:10:18.637715+0000","last_peered":"2024-09-18T08:10:18.637715+0000","last_clean":"2024-09-18T08:10:18.637715+0000","last_became_active":"2024-09-18T08:08:18.538711+0000","last_became_peered":"2024-09-18T08:08:18.538711+0000","last_unstale":"2024-09-18T08:10:18.637715+0000","last_undegraded":"2024-09-18T08:10:18.637715+0000","last_fullsized":"2024-09-18T08:10:18.637715+0000","mapping_epoch":181,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":182,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'10","last_scrub_stamp":"2024-09-18T08:10:18.636906+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_clean_scrub_stamp":"2024-09-18T08:10:18.636906+0000","objects_scrubbed":5,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:11:21.146399+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":348,"reported_epoch":221,"state":"active+clean","last_fresh":"2024-09-18T08:10:17.354345+0000","last_change":"2024-09-18T08:10:15.965550+0000","last_active":"2024-09-18T08:10:17.354345+0000","last_peered":"2024-09-18T08:10:17.354345+0000","last_clean":"2024-09-18T08:10:17.354345+0000","last_became_active":"2024-09-18T08:02:47.874727+0000","last_became_peered":"2024-09-18T08:02:47.874727+0000","last_unstale":"2024-09-18T08:10:17.354345+0000","last_undegraded":"2024-09-18T08:10:17.354345+0000","last_fullsized":"2024-09-18T08:10:17.354345+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:10:15.965485+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:46.860092+0000","last_clean_scrub_stamp":"2024-09-18T08:10:15.965485+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:11:38.102213+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.3","version":"138'10","reported_seq":464,"reported_epoch":222,"state":"active+clean","last_fresh":"2024-09-18T08:10:18.361249+0000","last_change":"2024-09-18T08:07:46.695714+0000","last_active":"2024-09-18T08:10:18.361249+0000","last_peered":"2024-09-18T08:10:18.361249+0000","last_clean":"2024-09-18T08:10:18.361249+0000","last_became_active":"2024-09-18T08:06:24.134194+0000","last_became_peered":"2024-09-18T08:06:24.134194+0000","last_unstale":"2024-09-18T08:10:18.361249+0000","last_undegraded":"2024-09-18T08:10:18.361249+0000","last_fullsized":"2024-09-18T08:10:18.361249+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"138'10","last_scrub_stamp":"2024-09-18T08:07:46.695656+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:50.886830+0000","last_clean_scrub_stamp":"2024-09-18T08:07:46.695656+0000","objects_scrubbed":1,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":369,"reported_epoch":222,"state":"active+clean","last_fresh":"2024-09-18T08:10:18.360161+0000","last_change":"2024-09-18T08:09:46.989395+0000","last_active":"2024-09-18T08:10:18.360161+0000","last_peered":"2024-09-18T08:10:18.360161+0000","last_clean":"2024-09-18T08:10:18.360161+0000","last_became_active":"2024-09-18T08:08:52.700080+0000","last_became_peered":"2024-09-18T08:08:52.700080+0000","last_unstale":"2024-09-18T08:10:18.360161+0000","last_undegraded":"2024-09-18T08:10:18.360161+0000","last_fullsized":"2024-09-18T08:10:18.360161+0000","mapping_epoch":198,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":199,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:09:46.989329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_clean_scrub_stamp":"2024-09-18T08:09:46.989329+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:10:55.582764+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.f","version":"222'9956","reported_seq":26138,"reported_epoch":222,"state":"active+clean","last_fresh":"2024-09-18T08:10:18.803680+0000","last_change":"2024-09-18T08:09:28.367000+0000","last_active":"2024-09-18T08:10:18.803680+0000","last_peered":"2024-09-18T08:10:18.803680+0000","last_clean":"2024-09-18T08:10:18.803680+0000","last_became_active":"2024-09-18T08:09:28.363033+0000","last_became_peered":"2024-09-18T08:09:28.363033+0000","last_unstale":"2024-09-18T08:10:18.803680+0000","last_undegraded":"2024-09-18T08:10:18.803680+0000","last_fullsized":"2024-09-18T08:10:18.803680+0000","mapping_epoch":211,"log_start":"204'5500","ondisk_log_start":"204'5500","created":165,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:07:52.307205+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:07:52.307205+0000","last_clean_scrub_stamp":"2024-09-18T08:07:52.307205+0000","objects_scrubbed":0,"log_size":4456,"log_dups_size":0,"ondisk_log_size":4456,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":193462272,"num_objects":2952,"num_object_clones":0,"num_object_copies":8856,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2952,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9956,"num_write_kb":413056,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8010,"num_bytes_recovered":524943360,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,1],"acting":[3,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"74'79","reported_seq":244,"reported_epoch":222,"state":"active+clean","last_fresh":"2024-09-18T08:10:18.360998+0000","last_change":"2024-09-18T08:08:57.736557+0000","last_active":"2024-09-18T08:10:18.360998+0000","last_peered":"2024-09-18T08:10:18.360998+0000","last_clean":"2024-09-18T08:10:18.360998+0000","last_became_active":"2024-09-18T08:07:26.426532+0000","last_became_peered":"2024-09-18T08:07:26.426532+0000","last_unstale":"2024-09-18T08:10:18.360998+0000","last_undegraded":"2024-09-18T08:10:18.360998+0000","last_fullsized":"2024-09-18T08:10:18.360998+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":152,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":3,"last_scrub":"74'79","last_scrub_stamp":"2024-09-18T08:08:57.736355+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:00:11.893742+0000","last_clean_scrub_stamp":"2024-09-18T08:08:57.736355+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:10:19.314793+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,3],"acting":[0,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.3","version":"69'19","reported_seq":374,"reported_epoch":222,"state":"active+clean","last_fresh":"2024-09-18T08:10:18.361418+0000","last_change":"2024-09-18T08:04:07.231406+0000","last_active":"2024-09-18T08:10:18.361418+0000","last_peered":"2024-09-18T08:10:18.361418+0000","last_clean":"2024-09-18T08:10:18.361418+0000","last_became_active":"2024-09-18T08:02:49.890318+0000","last_became_peered":"2024-09-18T08:02:49.890318+0000","last_unstale":"2024-09-18T08:10:18.361418+0000","last_undegraded":"2024-09-18T08:10:18.361418+0000","last_fullsized":"2024-09-18T08:10:18.361418+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'19","last_scrub_stamp":"2024-09-18T08:04:07.229846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_clean_scrub_stamp":"2024-09-18T08:04:07.229846+0000","objects_scrubbed":8,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:10:19.093125+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":39,"num_read_kb":25,"num_write":22,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.4","version":"62'1","reported_seq":386,"reported_epoch":222,"state":"active+clean","last_fresh":"2024-09-18T08:10:18.359258+0000","last_change":"2024-09-18T08:10:07.593068+0000","last_active":"2024-09-18T08:10:18.359258+0000","last_peered":"2024-09-18T08:10:18.359258+0000","last_clean":"2024-09-18T08:10:18.359258+0000","last_became_active":"2024-09-18T08:08:52.688347+0000","last_became_peered":"2024-09-18T08:08:52.688347+0000","last_unstale":"2024-09-18T08:10:18.359258+0000","last_undegraded":"2024-09-18T08:10:18.359258+0000","last_fullsized":"2024-09-18T08:10:18.359258+0000","mapping_epoch":198,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":199,"parent":"0.0","parent_split_bits":0,"last_scrub":"62'1","last_scrub_stamp":"2024-09-18T08:10:07.593004+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:46.860092+0000","last_clean_scrub_stamp":"2024-09-18T08:10:07.593004+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:11:36.610081+0000","scrub_duration":26,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":436,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":872,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":268,"reported_epoch":222,"state":"active+clean","last_fresh":"2024-09-18T08:10:18.360832+0000","last_change":"2024-09-18T08:06:24.133954+0000","last_active":"2024-09-18T08:10:18.360832+0000","last_peered":"2024-09-18T08:10:18.360832+0000","last_clean":"2024-09-18T08:10:18.360832+0000","last_became_active":"2024-09-18T08:06:24.133262+0000","last_became_peered":"2024-09-18T08:06:24.133262+0000","last_unstale":"2024-09-18T08:10:18.360832+0000","last_undegraded":"2024-09-18T08:10:18.360832+0000","last_fullsized":"2024-09-18T08:10:18.360832+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:04:13.377561+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:50.886830+0000","last_clean_scrub_stamp":"2024-09-18T08:04:13.377561+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:10:22.094192+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":374,"reported_epoch":222,"state":"active+clean","last_fresh":"2024-09-18T08:10:18.358364+0000","last_change":"2024-09-18T08:08:52.688537+0000","last_active":"2024-09-18T08:10:18.358364+0000","last_peered":"2024-09-18T08:10:18.358364+0000","last_clean":"2024-09-18T08:10:18.358364+0000","last_became_active":"2024-09-18T08:08:52.688212+0000","last_became_peered":"2024-09-18T08:08:52.688212+0000","last_unstale":"2024-09-18T08:10:18.358364+0000","last_undegraded":"2024-09-18T08:10:18.358364+0000","last_fullsized":"2024-09-18T08:10:18.358364+0000","mapping_epoch":198,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":199,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:07:49.041138+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_clean_scrub_stamp":"2024-09-18T08:07:49.041138+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:10:25.679955+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.8","version":"221'9633","reported_seq":9757,"reported_epoch":221,"state":"active+clean","last_fresh":"2024-09-18T08:10:18.285308+0000","last_change":"2024-09-18T08:07:53.671548+0000","last_active":"2024-09-18T08:10:18.285308+0000","last_peered":"2024-09-18T08:10:18.285308+0000","last_clean":"2024-09-18T08:10:18.285308+0000","last_became_active":"2024-09-18T08:07:53.670635+0000","last_became_peered":"2024-09-18T08:07:53.670635+0000","last_unstale":"2024-09-18T08:10:18.285308+0000","last_undegraded":"2024-09-18T08:10:18.285308+0000","last_fullsized":"2024-09-18T08:10:18.285308+0000","mapping_epoch":165,"log_start":"200'4500","ondisk_log_start":"200'4500","created":165,"last_epoch_clean":166,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:07:52.307205+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:07:52.307205+0000","last_clean_scrub_stamp":"2024-09-18T08:07:52.307205+0000","objects_scrubbed":0,"log_size":5133,"log_dups_size":0,"ondisk_log_size":5133,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":184745984,"num_objects":2819,"num_object_clones":0,"num_object_copies":8457,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2819,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9633,"num_write_kb":398464,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.1","version":"74'79","reported_seq":161,"reported_epoch":221,"state":"active+clean","last_fresh":"2024-09-18T08:10:17.353104+0000","last_change":"2024-09-18T08:08:59.553431+0000","last_active":"2024-09-18T08:10:17.353104+0000","last_peered":"2024-09-18T08:10:17.353104+0000","last_clean":"2024-09-18T08:10:17.353104+0000","last_became_active":"2024-09-18T08:07:26.400643+0000","last_became_peered":"2024-09-18T08:07:26.400643+0000","last_unstale":"2024-09-18T08:10:17.353104+0000","last_undegraded":"2024-09-18T08:10:17.353104+0000","last_fullsized":"2024-09-18T08:10:17.353104+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":152,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":4,"last_scrub":"74'79","last_scrub_stamp":"2024-09-18T08:08:59.552592+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:00:11.893742+0000","last_clean_scrub_stamp":"2024-09-18T08:08:59.552592+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,2],"acting":[5,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.4","version":"69'28","reported_seq":485,"reported_epoch":222,"state":"active+clean","last_fresh":"2024-09-18T08:10:18.359129+0000","last_change":"2024-09-18T08:08:57.639985+0000","last_active":"2024-09-18T08:10:18.359129+0000","last_peered":"2024-09-18T08:10:18.359129+0000","last_clean":"2024-09-18T08:10:18.359129+0000","last_became_active":"2024-09-18T08:02:49.897221+0000","last_became_peered":"2024-09-18T08:02:49.897221+0000","last_unstale":"2024-09-18T08:10:18.359129+0000","last_undegraded":"2024-09-18T08:10:18.359129+0000","last_fullsized":"2024-09-18T08:10:18.359129+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'28","last_scrub_stamp":"2024-09-18T08:08:57.639628+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_clean_scrub_stamp":"2024-09-18T08:08:57.639628+0000","objects_scrubbed":10,"log_size":28,"log_dups_size":0,"ondisk_log_size":28,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:10:27.518247+0000","scrub_duration":18,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":48,"num_read_kb":33,"num_write":26,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":95,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-09-18T08:10:22.393357+0000","last_change":"2024-09-18T08:08:18.543579+0000","last_active":"2024-09-18T08:10:22.393357+0000","last_peered":"2024-09-18T08:10:22.393357+0000","last_clean":"2024-09-18T08:10:22.393357+0000","last_became_active":"2024-09-18T08:08:18.538852+0000","last_became_peered":"2024-09-18T08:08:18.538852+0000","last_unstale":"2024-09-18T08:10:22.393357+0000","last_undegraded":"2024-09-18T08:10:22.393357+0000","last_fullsized":"2024-09-18T08:10:22.393357+0000","mapping_epoch":181,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":182,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:05:39.490141+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:46.860092+0000","last_clean_scrub_stamp":"2024-09-18T08:05:39.490141+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"64'2","reported_seq":380,"reported_epoch":221,"state":"active+clean","last_fresh":"2024-09-18T08:10:17.353802+0000","last_change":"2024-09-18T08:06:24.165227+0000","last_active":"2024-09-18T08:10:17.353802+0000","last_peered":"2024-09-18T08:10:17.353802+0000","last_clean":"2024-09-18T08:10:17.353802+0000","last_became_active":"2024-09-18T08:06:24.136080+0000","last_became_peered":"2024-09-18T08:06:24.136080+0000","last_unstale":"2024-09-18T08:10:17.353802+0000","last_undegraded":"2024-09-18T08:10:17.353802+0000","last_fullsized":"2024-09-18T08:10:17.353802+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"64'2","last_scrub_stamp":"2024-09-18T08:05:44.802942+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:45.855979+0000","last_clean_scrub_stamp":"2024-09-18T08:05:44.802942+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.001348349,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":38,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":348,"reported_epoch":222,"state":"active+clean","last_fresh":"2024-09-18T08:10:18.361104+0000","last_change":"2024-09-18T08:08:18.544944+0000","last_active":"2024-09-18T08:10:18.361104+0000","last_peered":"2024-09-18T08:10:18.361104+0000","last_clean":"2024-09-18T08:10:18.361104+0000","last_became_active":"2024-09-18T08:08:18.544260+0000","last_became_peered":"2024-09-18T08:08:18.544260+0000","last_unstale":"2024-09-18T08:10:18.361104+0000","last_undegraded":"2024-09-18T08:10:18.361104+0000","last_fullsized":"2024-09-18T08:10:18.361104+0000","mapping_epoch":181,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":182,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:07:53.582095+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:50.886830+0000","last_clean_scrub_stamp":"2024-09-18T08:07:53.582095+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:10:51.897980+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":354,"reported_epoch":222,"state":"active+clean","last_fresh":"2024-09-18T08:10:18.361679+0000","last_change":"2024-09-18T08:08:18.543527+0000","last_active":"2024-09-18T08:10:18.361679+0000","last_peered":"2024-09-18T08:10:18.361679+0000","last_clean":"2024-09-18T08:10:18.361679+0000","last_became_active":"2024-09-18T08:08:18.542678+0000","last_became_peered":"2024-09-18T08:08:18.542678+0000","last_unstale":"2024-09-18T08:10:18.361679+0000","last_undegraded":"2024-09-18T08:10:18.361679+0000","last_fullsized":"2024-09-18T08:10:18.361679+0000","mapping_epoch":181,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":182,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:07:08.245063+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_clean_scrub_stamp":"2024-09-18T08:07:08.245063+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:10:56.470037+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.b","version":"222'9811","reported_seq":15449,"reported_epoch":222,"state":"active+clean","last_fresh":"2024-09-18T08:10:18.822692+0000","last_change":"2024-09-18T08:08:48.662821+0000","last_active":"2024-09-18T08:10:18.822692+0000","last_peered":"2024-09-18T08:10:18.822692+0000","last_clean":"2024-09-18T08:10:18.822692+0000","last_became_active":"2024-09-18T08:08:48.655676+0000","last_became_peered":"2024-09-18T08:08:48.655676+0000","last_unstale":"2024-09-18T08:10:18.822692+0000","last_undegraded":"2024-09-18T08:10:18.822692+0000","last_fullsized":"2024-09-18T08:10:18.822692+0000","mapping_epoch":194,"log_start":"204'5300","ondisk_log_start":"204'5300","created":165,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:07:52.307205+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:07:52.307205+0000","last_clean_scrub_stamp":"2024-09-18T08:07:52.307205+0000","objects_scrubbed":0,"log_size":4511,"log_dups_size":0,"ondisk_log_size":4511,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":187498496,"num_objects":2861,"num_object_clones":0,"num_object_copies":8583,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2861,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9811,"num_write_kb":405504,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2745,"num_bytes_recovered":179896320,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,1],"acting":[3,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"74'79","reported_seq":265,"reported_epoch":222,"state":"active+clean","last_fresh":"2024-09-18T08:10:18.360701+0000","last_change":"2024-09-18T08:08:18.524000+0000","last_active":"2024-09-18T08:10:18.360701+0000","last_peered":"2024-09-18T08:10:18.360701+0000","last_clean":"2024-09-18T08:10:18.360701+0000","last_became_active":"2024-09-18T08:08:18.522977+0000","last_became_peered":"2024-09-18T08:08:18.522977+0000","last_unstale":"2024-09-18T08:10:18.360701+0000","last_undegraded":"2024-09-18T08:10:18.360701+0000","last_fullsized":"2024-09-18T08:10:18.360701+0000","mapping_epoch":181,"log_start":"0'0","ondisk_log_start":"0'0","created":152,"last_epoch_clean":182,"parent":"0.0","parent_split_bits":4,"last_scrub":"74'79","last_scrub_stamp":"2024-09-18T08:08:09.574144+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:00:11.893742+0000","last_clean_scrub_stamp":"2024-09-18T08:08:09.574144+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,4],"acting":[0,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.7","version":"69'13","reported_seq":292,"reported_epoch":222,"state":"active+clean","last_fresh":"2024-09-18T08:10:18.596235+0000","last_change":"2024-09-18T08:10:18.596235+0000","last_active":"2024-09-18T08:10:18.596235+0000","last_peered":"2024-09-18T08:10:18.596235+0000","last_clean":"2024-09-18T08:10:18.596235+0000","last_became_active":"2024-09-18T08:04:18.144925+0000","last_became_peered":"2024-09-18T08:04:18.144925+0000","last_unstale":"2024-09-18T08:10:18.596235+0000","last_undegraded":"2024-09-18T08:10:18.596235+0000","last_fullsized":"2024-09-18T08:10:18.596235+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'13","last_scrub_stamp":"2024-09-18T08:10:18.596150+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_clean_scrub_stamp":"2024-09-18T08:10:18.596150+0000","objects_scrubbed":5,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:11:34.602875+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":29,"num_read_kb":18,"num_write":16,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,0],"acting":[2,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":364,"reported_epoch":222,"state":"active+clean","last_fresh":"2024-09-18T08:10:18.365515+0000","last_change":"2024-09-18T08:08:53.649127+0000","last_active":"2024-09-18T08:10:18.365515+0000","last_peered":"2024-09-18T08:10:18.365515+0000","last_clean":"2024-09-18T08:10:18.365515+0000","last_became_active":"2024-09-18T08:06:24.126334+0000","last_became_peered":"2024-09-18T08:06:24.126334+0000","last_unstale":"2024-09-18T08:10:18.365515+0000","last_undegraded":"2024-09-18T08:10:18.365515+0000","last_fullsized":"2024-09-18T08:10:18.365515+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:08:53.649079+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:46.860092+0000","last_clean_scrub_stamp":"2024-09-18T08:08:53.649079+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"62'1","reported_seq":352,"reported_epoch":222,"state":"active+clean","last_fresh":"2024-09-18T08:10:18.355281+0000","last_change":"2024-09-18T08:04:09.732724+0000","last_active":"2024-09-18T08:10:18.355281+0000","last_peered":"2024-09-18T08:10:18.355281+0000","last_clean":"2024-09-18T08:10:18.355281+0000","last_became_active":"2024-09-18T08:02:46.868175+0000","last_became_peered":"2024-09-18T08:02:46.868175+0000","last_unstale":"2024-09-18T08:10:18.355281+0000","last_undegraded":"2024-09-18T08:10:18.355281+0000","last_fullsized":"2024-09-18T08:10:18.355281+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"62'1","last_scrub_stamp":"2024-09-18T08:04:09.732645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:45.855979+0000","last_clean_scrub_stamp":"2024-09-18T08:04:09.732645+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0.00066785499999999997,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":58,"reported_epoch":222,"state":"active+clean","last_fresh":"2024-09-18T08:10:18.360415+0000","last_change":"2024-09-18T08:08:52.690656+0000","last_active":"2024-09-18T08:10:18.360415+0000","last_peered":"2024-09-18T08:10:18.360415+0000","last_clean":"2024-09-18T08:10:18.360415+0000","last_became_active":"2024-09-18T08:08:52.690286+0000","last_became_peered":"2024-09-18T08:08:52.690286+0000","last_unstale":"2024-09-18T08:10:18.360415+0000","last_undegraded":"2024-09-18T08:10:18.360415+0000","last_fullsized":"2024-09-18T08:10:18.360415+0000","mapping_epoch":198,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":199,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:07:08.806748+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:50.886830+0000","last_clean_scrub_stamp":"2024-09-18T08:07:08.806748+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":363,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-09-18T08:10:22.415341+0000","last_change":"2024-09-18T08:09:42.647525+0000","last_active":"2024-09-18T08:10:22.415341+0000","last_peered":"2024-09-18T08:10:22.415341+0000","last_clean":"2024-09-18T08:10:22.415341+0000","last_became_active":"2024-09-18T08:06:24.145382+0000","last_became_peered":"2024-09-18T08:06:24.145382+0000","last_unstale":"2024-09-18T08:10:22.415341+0000","last_undegraded":"2024-09-18T08:10:22.415341+0000","last_fullsized":"2024-09-18T08:10:22.415341+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:09:42.647317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_clean_scrub_stamp":"2024-09-18T08:09:42.647317+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:11:03.015274+0000","scrub_duration":26,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.a","version":"222'9777","reported_seq":9903,"reported_epoch":222,"state":"active+clean","last_fresh":"2024-09-18T08:10:18.717834+0000","last_change":"2024-09-18T08:07:53.326651+0000","last_active":"2024-09-18T08:10:18.717834+0000","last_peered":"2024-09-18T08:10:18.717834+0000","last_clean":"2024-09-18T08:10:18.717834+0000","last_became_active":"2024-09-18T08:07:53.326361+0000","last_became_peered":"2024-09-18T08:07:53.326361+0000","last_unstale":"2024-09-18T08:10:18.717834+0000","last_undegraded":"2024-09-18T08:10:18.717834+0000","last_fullsized":"2024-09-18T08:10:18.717834+0000","mapping_epoch":165,"log_start":"202'5000","ondisk_log_start":"202'5000","created":165,"last_epoch_clean":166,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:07:52.307205+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:07:52.307205+0000","last_clean_scrub_stamp":"2024-09-18T08:07:52.307205+0000","objects_scrubbed":0,"log_size":4777,"log_dups_size":0,"ondisk_log_size":4777,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:10:20.719923+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":183959552,"num_objects":2807,"num_object_clones":0,"num_object_copies":8421,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2807,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9777,"num_write_kb":402688,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.3","version":"74'79","reported_seq":185,"reported_epoch":222,"state":"active+clean","last_fresh":"2024-09-18T08:10:18.361266+0000","last_change":"2024-09-18T08:09:41.651800+0000","last_active":"2024-09-18T08:10:18.361266+0000","last_peered":"2024-09-18T08:10:18.361266+0000","last_clean":"2024-09-18T08:10:18.361266+0000","last_became_active":"2024-09-18T08:08:18.538929+0000","last_became_peered":"2024-09-18T08:08:18.538929+0000","last_unstale":"2024-09-18T08:10:18.361266+0000","last_undegraded":"2024-09-18T08:10:18.361266+0000","last_fullsized":"2024-09-18T08:10:18.361266+0000","mapping_epoch":181,"log_start":"0'0","ondisk_log_start":"0'0","created":152,"last_epoch_clean":182,"parent":"0.0","parent_split_bits":3,"last_scrub":"74'79","last_scrub_stamp":"2024-09-18T08:09:41.651729+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:00:11.893742+0000","last_clean_scrub_stamp":"2024-09-18T08:09:41.651729+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:10:55.722378+0000","scrub_duration":17,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.6","version":"69'12","reported_seq":406,"reported_epoch":222,"state":"active+clean","last_fresh":"2024-09-18T08:10:18.366360+0000","last_change":"2024-09-18T08:08:57.021693+0000","last_active":"2024-09-18T08:10:18.366360+0000","last_peered":"2024-09-18T08:10:18.366360+0000","last_clean":"2024-09-18T08:10:18.366360+0000","last_became_active":"2024-09-18T08:08:52.700130+0000","last_became_peered":"2024-09-18T08:08:52.700130+0000","last_unstale":"2024-09-18T08:10:18.366360+0000","last_undegraded":"2024-09-18T08:10:18.366360+0000","last_fullsized":"2024-09-18T08:10:18.366360+0000","mapping_epoch":198,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":200,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'12","last_scrub_stamp":"2024-09-18T08:07:39.603562+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_clean_scrub_stamp":"2024-09-18T08:07:39.603562+0000","objects_scrubbed":6,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:10:57.784539+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":18,"num_read_kb":12,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":372,"reported_epoch":222,"state":"active+clean","last_fresh":"2024-09-18T08:10:18.361499+0000","last_change":"2024-09-18T08:09:34.089548+0000","last_active":"2024-09-18T08:10:18.361499+0000","last_peered":"2024-09-18T08:10:18.361499+0000","last_clean":"2024-09-18T08:10:18.361499+0000","last_became_active":"2024-09-18T08:08:18.523135+0000","last_became_peered":"2024-09-18T08:08:18.523135+0000","last_unstale":"2024-09-18T08:10:18.361499+0000","last_undegraded":"2024-09-18T08:10:18.361499+0000","last_fullsized":"2024-09-18T08:10:18.361499+0000","mapping_epoch":181,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":182,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:09:34.089476+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:46.860092+0000","last_clean_scrub_stamp":"2024-09-18T08:09:34.089476+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:10:41.071862+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"70'5","reported_seq":440,"reported_epoch":222,"state":"active+clean","last_fresh":"2024-09-18T08:10:18.365344+0000","last_change":"2024-09-18T08:09:09.649261+0000","last_active":"2024-09-18T08:10:18.365344+0000","last_peered":"2024-09-18T08:10:18.365344+0000","last_clean":"2024-09-18T08:10:18.365344+0000","last_became_active":"2024-09-18T08:06:46.141667+0000","last_became_peered":"2024-09-18T08:06:46.141667+0000","last_unstale":"2024-09-18T08:10:18.365344+0000","last_undegraded":"2024-09-18T08:10:18.365344+0000","last_fullsized":"2024-09-18T08:10:18.365344+0000","mapping_epoch":143,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":144,"parent":"0.0","parent_split_bits":0,"last_scrub":"70'5","last_scrub_stamp":"2024-09-18T08:09:09.649188+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:45.855979+0000","last_clean_scrub_stamp":"2024-09-18T08:09:09.649188+0000","objects_scrubbed":1,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:10:24.340859+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0.000504055,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":218,"num_read_kb":213,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":348,"reported_epoch":221,"state":"active+clean","last_fresh":"2024-09-18T08:10:17.354229+0000","last_change":"2024-09-18T08:07:20.767916+0000","last_active":"2024-09-18T08:10:17.354229+0000","last_peered":"2024-09-18T08:10:17.354229+0000","last_clean":"2024-09-18T08:10:17.354229+0000","last_became_active":"2024-09-18T08:02:51.913112+0000","last_became_peered":"2024-09-18T08:02:51.913112+0000","last_unstale":"2024-09-18T08:10:17.354229+0000","last_undegraded":"2024-09-18T08:10:17.354229+0000","last_fullsized":"2024-09-18T08:10:17.354229+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:07:20.767841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:50.886830+0000","last_clean_scrub_stamp":"2024-09-18T08:07:20.767841+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":331,"reported_epoch":222,"state":"active+clean","last_fresh":"2024-09-18T08:10:18.358923+0000","last_change":"2024-09-18T08:09:49.702132+0000","last_active":"2024-09-18T08:10:18.358923+0000","last_peered":"2024-09-18T08:10:18.358923+0000","last_clean":"2024-09-18T08:10:18.358923+0000","last_became_active":"2024-09-18T08:02:53.907404+0000","last_became_peered":"2024-09-18T08:02:53.907404+0000","last_unstale":"2024-09-18T08:10:18.358923+0000","last_undegraded":"2024-09-18T08:10:18.358923+0000","last_fullsized":"2024-09-18T08:10:18.358923+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:09:49.702055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_clean_scrub_stamp":"2024-09-18T08:09:49.702055+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:11:01.853786+0000","scrub_duration":23,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.9","version":"221'9415","reported_seq":9538,"reported_epoch":221,"state":"active+clean","last_fresh":"2024-09-18T08:10:18.291249+0000","last_change":"2024-09-18T08:07:53.672294+0000","last_active":"2024-09-18T08:10:18.291249+0000","last_peered":"2024-09-18T08:10:18.291249+0000","last_clean":"2024-09-18T08:10:18.291249+0000","last_became_active":"2024-09-18T08:07:53.671994+0000","last_became_peered":"2024-09-18T08:07:53.671994+0000","last_unstale":"2024-09-18T08:10:18.291249+0000","last_undegraded":"2024-09-18T08:10:18.291249+0000","last_fullsized":"2024-09-18T08:10:18.291249+0000","mapping_epoch":165,"log_start":"200'4300","ondisk_log_start":"200'4300","created":165,"last_epoch_clean":166,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:07:52.307205+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:07:52.307205+0000","last_clean_scrub_stamp":"2024-09-18T08:07:52.307205+0000","objects_scrubbed":0,"log_size":5115,"log_dups_size":0,"ondisk_log_size":5115,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":183435264,"num_objects":2799,"num_object_clones":0,"num_object_copies":8397,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2799,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9415,"num_write_kb":390848,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.5","version":"182'21","reported_seq":139,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-09-18T08:10:23.017638+0000","last_change":"2024-09-18T08:08:18.609652+0000","last_active":"2024-09-18T08:10:23.017638+0000","last_peered":"2024-09-18T08:10:23.017638+0000","last_clean":"2024-09-18T08:10:23.017638+0000","last_became_active":"2024-09-18T08:08:18.547555+0000","last_became_peered":"2024-09-18T08:08:18.547555+0000","last_unstale":"2024-09-18T08:10:23.017638+0000","last_undegraded":"2024-09-18T08:10:23.017638+0000","last_fullsized":"2024-09-18T08:10:23.017638+0000","mapping_epoch":181,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":182,"parent":"0.0","parent_split_bits":0,"last_scrub":"130'18","last_scrub_stamp":"2024-09-18T08:06:15.455503+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_clean_scrub_stamp":"2024-09-18T08:06:15.455503+0000","objects_scrubbed":0,"log_size":21,"log_dups_size":0,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":24,"num_read_kb":14,"num_write":13,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":154,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,7],"acting":[4,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":365,"reported_epoch":222,"state":"active+clean","last_fresh":"2024-09-18T08:10:18.362478+0000","last_change":"2024-09-18T08:09:45.572278+0000","last_active":"2024-09-18T08:10:18.362478+0000","last_peered":"2024-09-18T08:10:18.362478+0000","last_clean":"2024-09-18T08:10:18.362478+0000","last_became_active":"2024-09-18T08:06:24.133425+0000","last_became_peered":"2024-09-18T08:06:24.133425+0000","last_unstale":"2024-09-18T08:10:18.362478+0000","last_undegraded":"2024-09-18T08:10:18.362478+0000","last_fullsized":"2024-09-18T08:10:18.362478+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:09:45.572201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:46.860092+0000","last_clean_scrub_stamp":"2024-09-18T08:09:45.572201+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:11:00.131734+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"74'79","reported_seq":448,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-09-18T08:10:22.413974+0000","last_change":"2024-09-18T08:08:57.769104+0000","last_active":"2024-09-18T08:10:22.413974+0000","last_peered":"2024-09-18T08:10:22.413974+0000","last_clean":"2024-09-18T08:10:22.413974+0000","last_became_active":"2024-09-18T08:07:24.376931+0000","last_became_peered":"2024-09-18T08:07:24.376931+0000","last_unstale":"2024-09-18T08:10:22.413974+0000","last_undegraded":"2024-09-18T08:10:22.413974+0000","last_fullsized":"2024-09-18T08:10:22.413974+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":153,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'79","last_scrub_stamp":"2024-09-18T08:08:57.769048+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:00:11.893742+0000","last_clean_scrub_stamp":"2024-09-18T08:08:57.769048+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":178,"reported_epoch":222,"state":"active+clean","last_fresh":"2024-09-18T08:10:18.358044+0000","last_change":"2024-09-18T08:07:09.744206+0000","last_active":"2024-09-18T08:10:18.358044+0000","last_peered":"2024-09-18T08:10:18.358044+0000","last_clean":"2024-09-18T08:10:18.358044+0000","last_became_active":"2024-09-18T08:06:46.141046+0000","last_became_peered":"2024-09-18T08:06:46.141046+0000","last_unstale":"2024-09-18T08:10:18.358044+0000","last_undegraded":"2024-09-18T08:10:18.358044+0000","last_fullsized":"2024-09-18T08:10:18.358044+0000","mapping_epoch":143,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":144,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:07:09.744137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:50.886830+0000","last_clean_scrub_stamp":"2024-09-18T08:07:09.744137+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":351,"reported_epoch":221,"state":"active+clean","last_fresh":"2024-09-18T08:10:17.354040+0000","last_change":"2024-09-18T08:08:18.548620+0000","last_active":"2024-09-18T08:10:17.354040+0000","last_peered":"2024-09-18T08:10:17.354040+0000","last_clean":"2024-09-18T08:10:17.354040+0000","last_became_active":"2024-09-18T08:08:18.548233+0000","last_became_peered":"2024-09-18T08:08:18.548233+0000","last_unstale":"2024-09-18T08:10:17.354040+0000","last_undegraded":"2024-09-18T08:10:17.354040+0000","last_fullsized":"2024-09-18T08:10:17.354040+0000","mapping_epoch":181,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":182,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:06:50.710565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_clean_scrub_stamp":"2024-09-18T08:06:50.710565+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.2","version":"223'10054","reported_seq":14458,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-09-18T08:10:23.190204+0000","last_change":"2024-09-18T08:08:58.743574+0000","last_active":"2024-09-18T08:10:23.190204+0000","last_peered":"2024-09-18T08:10:23.190204+0000","last_clean":"2024-09-18T08:10:23.190204+0000","last_became_active":"2024-09-18T08:08:58.738458+0000","last_became_peered":"2024-09-18T08:08:58.738458+0000","last_unstale":"2024-09-18T08:10:23.190204+0000","last_undegraded":"2024-09-18T08:10:23.190204+0000","last_fullsized":"2024-09-18T08:10:23.190204+0000","mapping_epoch":201,"log_start":"206'5500","ondisk_log_start":"206'5500","created":165,"last_epoch_clean":202,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:07:52.307205+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:07:52.307205+0000","last_clean_scrub_stamp":"2024-09-18T08:07:52.307205+0000","objects_scrubbed":0,"log_size":4554,"log_dups_size":0,"ondisk_log_size":4554,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":165019648,"num_objects":2518,"num_object_clones":0,"num_object_copies":7554,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2518,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10054,"num_write_kb":402304,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2129,"num_bytes_recovered":139526144,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,5],"acting":[4,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.e","version":"69'11","reported_seq":106,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-09-18T08:10:22.393436+0000","last_change":"2024-09-18T08:08:18.594670+0000","last_active":"2024-09-18T08:10:22.393436+0000","last_peered":"2024-09-18T08:10:22.393436+0000","last_clean":"2024-09-18T08:10:22.393436+0000","last_became_active":"2024-09-18T08:08:18.544581+0000","last_became_peered":"2024-09-18T08:08:18.544581+0000","last_unstale":"2024-09-18T08:10:22.393436+0000","last_undegraded":"2024-09-18T08:10:22.393436+0000","last_fullsized":"2024-09-18T08:10:22.393436+0000","mapping_epoch":181,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":182,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'11","last_scrub_stamp":"2024-09-18T08:06:14.472168+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_clean_scrub_stamp":"2024-09-18T08:06:14.472168+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":19,"num_read_kb":12,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":102,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-09-18T08:10:22.393010+0000","last_change":"2024-09-18T08:10:14.190094+0000","last_active":"2024-09-18T08:10:22.393010+0000","last_peered":"2024-09-18T08:10:22.393010+0000","last_clean":"2024-09-18T08:10:22.393010+0000","last_became_active":"2024-09-18T08:08:18.524229+0000","last_became_peered":"2024-09-18T08:08:18.524229+0000","last_unstale":"2024-09-18T08:10:22.393010+0000","last_undegraded":"2024-09-18T08:10:22.393010+0000","last_fullsized":"2024-09-18T08:10:22.393010+0000","mapping_epoch":181,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":182,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:10:14.190017+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:46.860092+0000","last_clean_scrub_stamp":"2024-09-18T08:10:14.190017+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:11:19.091896+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":368,"reported_epoch":221,"state":"active+clean","last_fresh":"2024-09-18T08:10:17.353879+0000","last_change":"2024-09-18T08:08:18.547880+0000","last_active":"2024-09-18T08:10:17.353879+0000","last_peered":"2024-09-18T08:10:17.353879+0000","last_clean":"2024-09-18T08:10:17.353879+0000","last_became_active":"2024-09-18T08:08:18.547105+0000","last_became_peered":"2024-09-18T08:08:18.547105+0000","last_unstale":"2024-09-18T08:10:17.353879+0000","last_undegraded":"2024-09-18T08:10:17.353879+0000","last_fullsized":"2024-09-18T08:10:17.353879+0000","mapping_epoch":181,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":182,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:07:51.742090+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:50.886830+0000","last_clean_scrub_stamp":"2024-09-18T08:07:51.742090+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":353,"reported_epoch":222,"state":"active+clean","last_fresh":"2024-09-18T08:10:18.357055+0000","last_change":"2024-09-18T08:07:09.197912+0000","last_active":"2024-09-18T08:10:18.357055+0000","last_peered":"2024-09-18T08:10:18.357055+0000","last_clean":"2024-09-18T08:10:18.357055+0000","last_became_active":"2024-09-18T08:06:24.130266+0000","last_became_peered":"2024-09-18T08:06:24.130266+0000","last_unstale":"2024-09-18T08:10:18.357055+0000","last_undegraded":"2024-09-18T08:10:18.357055+0000","last_fullsized":"2024-09-18T08:10:18.357055+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:07:09.197841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_clean_scrub_stamp":"2024-09-18T08:07:09.197841+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:10:58.431705+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.1","version":"222'9817","reported_seq":10009,"reported_epoch":222,"state":"active+clean","last_fresh":"2024-09-18T08:10:18.694303+0000","last_change":"2024-09-18T08:09:40.130643+0000","last_active":"2024-09-18T08:10:18.694303+0000","last_peered":"2024-09-18T08:10:18.694303+0000","last_clean":"2024-09-18T08:10:18.694303+0000","last_became_active":"2024-09-18T08:07:53.327987+0000","last_became_peered":"2024-09-18T08:07:53.327987+0000","last_unstale":"2024-09-18T08:10:18.694303+0000","last_undegraded":"2024-09-18T08:10:18.694303+0000","last_fullsized":"2024-09-18T08:10:18.694303+0000","mapping_epoch":165,"log_start":"202'5000","ondisk_log_start":"202'5000","created":165,"last_epoch_clean":166,"parent":"0.0","parent_split_bits":0,"last_scrub":"213'7038","last_scrub_stamp":"2024-09-18T08:09:40.130353+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:07:52.307205+0000","last_clean_scrub_stamp":"2024-09-18T08:09:40.130353+0000","objects_scrubbed":5709,"log_size":4817,"log_dups_size":0,"ondisk_log_size":4817,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:11:06.670120+0000","scrub_duration":2501,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":187498496,"num_objects":2861,"num_object_clones":0,"num_object_copies":8583,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2861,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9817,"num_write_kb":405696,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.8","version":"74'79","reported_seq":168,"reported_epoch":221,"state":"active+clean","last_fresh":"2024-09-18T08:10:17.352500+0000","last_change":"2024-09-18T08:09:00.533593+0000","last_active":"2024-09-18T08:10:17.352500+0000","last_peered":"2024-09-18T08:10:17.352500+0000","last_clean":"2024-09-18T08:10:17.352500+0000","last_became_active":"2024-09-18T08:07:26.407298+0000","last_became_peered":"2024-09-18T08:07:26.407298+0000","last_unstale":"2024-09-18T08:10:17.352500+0000","last_undegraded":"2024-09-18T08:10:17.352500+0000","last_fullsized":"2024-09-18T08:10:17.352500+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":152,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":4,"last_scrub":"74'79","last_scrub_stamp":"2024-09-18T08:09:00.532813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:00:11.893742+0000","last_clean_scrub_stamp":"2024-09-18T08:09:00.532813+0000","objects_scrubbed":1,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:10:17.887770+0000","scrub_duration":25,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262688,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,3],"acting":[5,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.d","version":"69'17","reported_seq":112,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-09-18T08:10:22.393952+0000","last_change":"2024-09-18T08:08:18.588624+0000","last_active":"2024-09-18T08:10:22.393952+0000","last_peered":"2024-09-18T08:10:22.393952+0000","last_clean":"2024-09-18T08:10:22.393952+0000","last_became_active":"2024-09-18T08:08:18.540862+0000","last_became_peered":"2024-09-18T08:08:18.540862+0000","last_unstale":"2024-09-18T08:10:22.393952+0000","last_undegraded":"2024-09-18T08:10:22.393952+0000","last_fullsized":"2024-09-18T08:10:22.393952+0000","mapping_epoch":181,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":182,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'17","last_scrub_stamp":"2024-09-18T08:08:09.671248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_clean_scrub_stamp":"2024-09-18T08:08:09.671248+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":28,"num_read_kb":18,"num_write":18,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,1],"acting":[4,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":102,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-09-18T08:10:22.393646+0000","last_change":"2024-09-18T08:10:22.243240+0000","last_active":"2024-09-18T08:10:22.393646+0000","last_peered":"2024-09-18T08:10:22.393646+0000","last_clean":"2024-09-18T08:10:22.393646+0000","last_became_active":"2024-09-18T08:08:18.540285+0000","last_became_peered":"2024-09-18T08:08:18.540285+0000","last_unstale":"2024-09-18T08:10:22.393646+0000","last_undegraded":"2024-09-18T08:10:22.393646+0000","last_fullsized":"2024-09-18T08:10:22.393646+0000","mapping_epoch":181,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":182,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:10:22.243148+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:46.860092+0000","last_clean_scrub_stamp":"2024-09-18T08:10:22.243148+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:11:51.567698+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":364,"reported_epoch":222,"state":"active+clean","last_fresh":"2024-09-18T08:10:18.359034+0000","last_change":"2024-09-18T08:09:06.610955+0000","last_active":"2024-09-18T08:10:18.359034+0000","last_peered":"2024-09-18T08:10:18.359034+0000","last_clean":"2024-09-18T08:10:18.359034+0000","last_became_active":"2024-09-18T08:08:18.623014+0000","last_became_peered":"2024-09-18T08:08:18.623014+0000","last_unstale":"2024-09-18T08:10:18.359034+0000","last_undegraded":"2024-09-18T08:10:18.359034+0000","last_fullsized":"2024-09-18T08:10:18.359034+0000","mapping_epoch":181,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":182,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:09:06.610879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:50.886830+0000","last_clean_scrub_stamp":"2024-09-18T08:09:06.610879+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:10:19.831235+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":66,"reported_epoch":222,"state":"active+clean","last_fresh":"2024-09-18T08:10:18.354224+0000","last_change":"2024-09-18T08:10:17.574938+0000","last_active":"2024-09-18T08:10:18.354224+0000","last_peered":"2024-09-18T08:10:18.354224+0000","last_clean":"2024-09-18T08:10:18.354224+0000","last_became_active":"2024-09-18T08:08:52.696300+0000","last_became_peered":"2024-09-18T08:08:52.696300+0000","last_unstale":"2024-09-18T08:10:18.354224+0000","last_undegraded":"2024-09-18T08:10:18.354224+0000","last_fullsized":"2024-09-18T08:10:18.354224+0000","mapping_epoch":198,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":199,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:10:17.574551+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_clean_scrub_stamp":"2024-09-18T08:10:17.574551+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:11:25.310756+0000","scrub_duration":20,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"8.0","version":"222'9700","reported_seq":9826,"reported_epoch":222,"state":"active+clean","last_fresh":"2024-09-18T08:10:18.817651+0000","last_change":"2024-09-18T08:07:53.329182+0000","last_active":"2024-09-18T08:10:18.817651+0000","last_peered":"2024-09-18T08:10:18.817651+0000","last_clean":"2024-09-18T08:10:18.817651+0000","last_became_active":"2024-09-18T08:07:53.328914+0000","last_became_peered":"2024-09-18T08:07:53.328914+0000","last_unstale":"2024-09-18T08:10:18.817651+0000","last_undegraded":"2024-09-18T08:10:18.817651+0000","last_fullsized":"2024-09-18T08:10:18.817651+0000","mapping_epoch":165,"log_start":"203'5200","ondisk_log_start":"203'5200","created":165,"last_epoch_clean":166,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:07:52.307205+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:07:52.307205+0000","last_clean_scrub_stamp":"2024-09-18T08:07:52.307205+0000","objects_scrubbed":0,"log_size":4500,"log_dups_size":0,"ondisk_log_size":4500,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":186777600,"num_objects":2850,"num_object_clones":0,"num_object_copies":8550,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2850,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9700,"num_write_kb":401600,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.9","version":"74'79","reported_seq":470,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-09-18T08:10:22.414267+0000","last_change":"2024-09-18T08:09:27.676247+0000","last_active":"2024-09-18T08:10:22.414267+0000","last_peered":"2024-09-18T08:10:22.414267+0000","last_clean":"2024-09-18T08:10:22.414267+0000","last_became_active":"2024-09-18T08:08:52.691492+0000","last_became_peered":"2024-09-18T08:08:52.691492+0000","last_unstale":"2024-09-18T08:10:22.414267+0000","last_undegraded":"2024-09-18T08:10:22.414267+0000","last_fullsized":"2024-09-18T08:10:22.414267+0000","mapping_epoch":198,"log_start":"0'0","ondisk_log_start":"0'0","created":152,"last_epoch_clean":199,"parent":"0.0","parent_split_bits":4,"last_scrub":"74'79","last_scrub_stamp":"2024-09-18T08:09:27.676169+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:00:11.893742+0000","last_clean_scrub_stamp":"2024-09-18T08:09:27.676169+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:10:36.422152+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,2],"acting":[7,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.c","version":"69'10","reported_seq":106,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-09-18T08:10:22.393343+0000","last_change":"2024-09-18T08:08:18.642452+0000","last_active":"2024-09-18T08:10:22.393343+0000","last_peered":"2024-09-18T08:10:22.393343+0000","last_clean":"2024-09-18T08:10:22.393343+0000","last_became_active":"2024-09-18T08:08:18.622980+0000","last_became_peered":"2024-09-18T08:08:18.622980+0000","last_unstale":"2024-09-18T08:10:22.393343+0000","last_undegraded":"2024-09-18T08:10:22.393343+0000","last_fullsized":"2024-09-18T08:10:22.393343+0000","mapping_epoch":181,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":182,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'10","last_scrub_stamp":"2024-09-18T08:05:41.474813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_clean_scrub_stamp":"2024-09-18T08:05:41.474813+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":14,"num_read_kb":9,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":365,"reported_epoch":222,"state":"active+clean","last_fresh":"2024-09-18T08:10:18.364681+0000","last_change":"2024-09-18T08:08:18.542800+0000","last_active":"2024-09-18T08:10:18.364681+0000","last_peered":"2024-09-18T08:10:18.364681+0000","last_clean":"2024-09-18T08:10:18.364681+0000","last_became_active":"2024-09-18T08:08:18.542424+0000","last_became_peered":"2024-09-18T08:08:18.542424+0000","last_unstale":"2024-09-18T08:10:18.364681+0000","last_undegraded":"2024-09-18T08:10:18.364681+0000","last_fullsized":"2024-09-18T08:10:18.364681+0000","mapping_epoch":181,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":182,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:06:46.339428+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:46.860092+0000","last_clean_scrub_stamp":"2024-09-18T08:06:46.339428+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:10:53.446520+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.d","version":"69'8","reported_seq":442,"reported_epoch":222,"state":"active+clean","last_fresh":"2024-09-18T08:10:18.354737+0000","last_change":"2024-09-18T08:05:33.027988+0000","last_active":"2024-09-18T08:10:18.354737+0000","last_peered":"2024-09-18T08:10:18.354737+0000","last_clean":"2024-09-18T08:10:18.354737+0000","last_became_active":"2024-09-18T08:02:51.909011+0000","last_became_peered":"2024-09-18T08:02:51.909011+0000","last_unstale":"2024-09-18T08:10:18.354737+0000","last_undegraded":"2024-09-18T08:10:18.354737+0000","last_fullsized":"2024-09-18T08:10:18.354737+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'8","last_scrub_stamp":"2024-09-18T08:05:33.027783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:50.886830+0000","last_clean_scrub_stamp":"2024-09-18T08:05:33.027783+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":101,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-09-18T08:10:22.393812+0000","last_change":"2024-09-18T08:08:52.685954+0000","last_active":"2024-09-18T08:10:22.393812+0000","last_peered":"2024-09-18T08:10:22.393812+0000","last_clean":"2024-09-18T08:10:22.393812+0000","last_became_active":"2024-09-18T08:08:52.685603+0000","last_became_peered":"2024-09-18T08:08:52.685603+0000","last_unstale":"2024-09-18T08:10:22.393812+0000","last_undegraded":"2024-09-18T08:10:22.393812+0000","last_fullsized":"2024-09-18T08:10:22.393812+0000","mapping_epoch":198,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":199,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:07:16.559597+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_clean_scrub_stamp":"2024-09-18T08:07:16.559597+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.7","version":"222'9719","reported_seq":9915,"reported_epoch":222,"state":"active+clean","last_fresh":"2024-09-18T08:10:18.669643+0000","last_change":"2024-09-18T08:09:36.764450+0000","last_active":"2024-09-18T08:10:18.669643+0000","last_peered":"2024-09-18T08:10:18.669643+0000","last_clean":"2024-09-18T08:10:18.669643+0000","last_became_active":"2024-09-18T08:07:53.674433+0000","last_became_peered":"2024-09-18T08:07:53.674433+0000","last_unstale":"2024-09-18T08:10:18.669643+0000","last_undegraded":"2024-09-18T08:10:18.669643+0000","last_fullsized":"2024-09-18T08:10:18.669643+0000","mapping_epoch":165,"log_start":"202'4900","ondisk_log_start":"202'4900","created":165,"last_epoch_clean":166,"parent":"0.0","parent_split_bits":0,"last_scrub":"213'6848","last_scrub_stamp":"2024-09-18T08:09:36.764206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:07:52.307205+0000","last_clean_scrub_stamp":"2024-09-18T08:09:36.764206+0000","objects_scrubbed":5822,"log_size":4819,"log_dups_size":0,"ondisk_log_size":4819,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":4,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:11:06.120132+0000","scrub_duration":3143,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":187105280,"num_objects":2855,"num_object_clones":0,"num_object_copies":8565,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2855,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9719,"num_write_kb":402368,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,6],"acting":[1,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.b","version":"69'9","reported_seq":396,"reported_epoch":222,"state":"active+clean","last_fresh":"2024-09-18T08:10:18.361337+0000","last_change":"2024-09-18T08:08:18.640380+0000","last_active":"2024-09-18T08:10:18.361337+0000","last_peered":"2024-09-18T08:10:18.361337+0000","last_clean":"2024-09-18T08:10:18.361337+0000","last_became_active":"2024-09-18T08:08:18.522995+0000","last_became_peered":"2024-09-18T08:08:18.522995+0000","last_unstale":"2024-09-18T08:10:18.361337+0000","last_undegraded":"2024-09-18T08:10:18.361337+0000","last_fullsized":"2024-09-18T08:10:18.361337+0000","mapping_epoch":181,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":182,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'9","last_scrub_stamp":"2024-09-18T08:08:18.640292+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_clean_scrub_stamp":"2024-09-18T08:08:18.640292+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":362,"reported_epoch":221,"state":"active+clean","last_fresh":"2024-09-18T08:10:17.353660+0000","last_change":"2024-09-18T08:07:40.708363+0000","last_active":"2024-09-18T08:10:17.353660+0000","last_peered":"2024-09-18T08:10:17.353660+0000","last_clean":"2024-09-18T08:10:17.353660+0000","last_became_active":"2024-09-18T08:06:24.135005+0000","last_became_peered":"2024-09-18T08:06:24.135005+0000","last_unstale":"2024-09-18T08:10:17.353660+0000","last_undegraded":"2024-09-18T08:10:17.353660+0000","last_fullsized":"2024-09-18T08:10:17.353660+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:07:40.708307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:46.860092+0000","last_clean_scrub_stamp":"2024-09-18T08:07:40.708307+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":365,"reported_epoch":222,"state":"active+clean","last_fresh":"2024-09-18T08:10:18.356009+0000","last_change":"2024-09-18T08:08:18.509651+0000","last_active":"2024-09-18T08:10:18.356009+0000","last_peered":"2024-09-18T08:10:18.356009+0000","last_clean":"2024-09-18T08:10:18.356009+0000","last_became_active":"2024-09-18T08:08:18.509247+0000","last_became_peered":"2024-09-18T08:08:18.509247+0000","last_unstale":"2024-09-18T08:10:18.356009+0000","last_undegraded":"2024-09-18T08:10:18.356009+0000","last_fullsized":"2024-09-18T08:10:18.356009+0000","mapping_epoch":181,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":182,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:07:10.728679+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:50.886830+0000","last_clean_scrub_stamp":"2024-09-18T08:07:10.728679+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,3],"acting":[2,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":361,"reported_epoch":222,"state":"active+clean","last_fresh":"2024-09-18T08:10:18.361303+0000","last_change":"2024-09-18T08:09:28.086782+0000","last_active":"2024-09-18T08:10:18.361303+0000","last_peered":"2024-09-18T08:10:18.361303+0000","last_clean":"2024-09-18T08:10:18.361303+0000","last_became_active":"2024-09-18T08:08:52.697337+0000","last_became_peered":"2024-09-18T08:08:52.697337+0000","last_unstale":"2024-09-18T08:10:18.361303+0000","last_undegraded":"2024-09-18T08:10:18.361303+0000","last_fullsized":"2024-09-18T08:10:18.361303+0000","mapping_epoch":198,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":199,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:09:28.086727+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_clean_scrub_stamp":"2024-09-18T08:09:28.086727+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:10:33.905647+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.6","version":"222'9643","reported_seq":9810,"reported_epoch":222,"state":"active+clean","last_fresh":"2024-09-18T08:10:18.811257+0000","last_change":"2024-09-18T08:10:12.178469+0000","last_active":"2024-09-18T08:10:18.811257+0000","last_peered":"2024-09-18T08:10:18.811257+0000","last_clean":"2024-09-18T08:10:18.811257+0000","last_became_active":"2024-09-18T08:07:53.674139+0000","last_became_peered":"2024-09-18T08:07:53.674139+0000","last_unstale":"2024-09-18T08:10:18.811257+0000","last_undegraded":"2024-09-18T08:10:18.811257+0000","last_fullsized":"2024-09-18T08:10:18.811257+0000","mapping_epoch":165,"log_start":"197'4200","ondisk_log_start":"197'4200","created":165,"last_epoch_clean":166,"parent":"0.0","parent_split_bits":0,"last_scrub":"217'9226","last_scrub_stamp":"2024-09-18T08:10:12.178143+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:07:52.307205+0000","last_clean_scrub_stamp":"2024-09-18T08:10:12.178143+0000","objects_scrubbed":3346,"log_size":5443,"log_dups_size":0,"ondisk_log_size":5443,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:11:40.476010+0000","scrub_duration":1358,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":189333504,"num_objects":2889,"num_object_clones":0,"num_object_copies":8667,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2889,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9643,"num_write_kb":401024,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,7],"acting":[0,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.a","version":"69'19","reported_seq":367,"reported_epoch":222,"state":"active+clean","last_fresh":"2024-09-18T08:10:18.358243+0000","last_change":"2024-09-18T08:07:41.000210+0000","last_active":"2024-09-18T08:10:18.358243+0000","last_peered":"2024-09-18T08:10:18.358243+0000","last_clean":"2024-09-18T08:10:18.358243+0000","last_became_active":"2024-09-18T08:06:24.139909+0000","last_became_peered":"2024-09-18T08:06:24.139909+0000","last_unstale":"2024-09-18T08:10:18.358243+0000","last_undegraded":"2024-09-18T08:10:18.358243+0000","last_fullsized":"2024-09-18T08:10:18.358243+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'19","last_scrub_stamp":"2024-09-18T08:07:41.000146+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_clean_scrub_stamp":"2024-09-18T08:07:41.000146+0000","objects_scrubbed":9,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:10:18.661451+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":31,"num_read_kb":20,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,7],"acting":[1,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":350,"reported_epoch":221,"state":"active+clean","last_fresh":"2024-09-18T08:10:17.354064+0000","last_change":"2024-09-18T08:06:47.638974+0000","last_active":"2024-09-18T08:10:17.354064+0000","last_peered":"2024-09-18T08:10:17.354064+0000","last_clean":"2024-09-18T08:10:17.354064+0000","last_became_active":"2024-09-18T08:06:46.171298+0000","last_became_peered":"2024-09-18T08:06:46.171298+0000","last_unstale":"2024-09-18T08:10:17.354064+0000","last_undegraded":"2024-09-18T08:10:17.354064+0000","last_fullsized":"2024-09-18T08:10:17.354064+0000","mapping_epoch":143,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":144,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:06:47.638911+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:46.860092+0000","last_clean_scrub_stamp":"2024-09-18T08:06:47.638911+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":58,"reported_epoch":222,"state":"active+clean","last_fresh":"2024-09-18T08:10:18.353105+0000","last_change":"2024-09-18T08:08:52.698243+0000","last_active":"2024-09-18T08:10:18.353105+0000","last_peered":"2024-09-18T08:10:18.353105+0000","last_clean":"2024-09-18T08:10:18.353105+0000","last_became_active":"2024-09-18T08:08:52.697932+0000","last_became_peered":"2024-09-18T08:08:52.697932+0000","last_unstale":"2024-09-18T08:10:18.353105+0000","last_undegraded":"2024-09-18T08:10:18.353105+0000","last_fullsized":"2024-09-18T08:10:18.353105+0000","mapping_epoch":198,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":199,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:08:16.631007+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:50.886830+0000","last_clean_scrub_stamp":"2024-09-18T08:08:16.631007+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":178,"reported_epoch":222,"state":"active+clean","last_fresh":"2024-09-18T08:10:18.353753+0000","last_change":"2024-09-18T08:06:50.902917+0000","last_active":"2024-09-18T08:10:18.353753+0000","last_peered":"2024-09-18T08:10:18.353753+0000","last_clean":"2024-09-18T08:10:18.353753+0000","last_became_active":"2024-09-18T08:06:46.137872+0000","last_became_peered":"2024-09-18T08:06:46.137872+0000","last_unstale":"2024-09-18T08:10:18.353753+0000","last_undegraded":"2024-09-18T08:10:18.353753+0000","last_fullsized":"2024-09-18T08:10:18.353753+0000","mapping_epoch":143,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":144,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:06:50.902835+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_clean_scrub_stamp":"2024-09-18T08:06:50.902835+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,3],"acting":[2,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"8.5","version":"223'10004","reported_seq":14445,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-09-18T08:10:23.186523+0000","last_change":"2024-09-18T08:09:13.170209+0000","last_active":"2024-09-18T08:10:23.186523+0000","last_peered":"2024-09-18T08:10:23.186523+0000","last_clean":"2024-09-18T08:10:23.186523+0000","last_became_active":"2024-09-18T08:09:13.166459+0000","last_became_peered":"2024-09-18T08:09:13.166459+0000","last_unstale":"2024-09-18T08:10:23.186523+0000","last_undegraded":"2024-09-18T08:10:23.186523+0000","last_fullsized":"2024-09-18T08:10:23.186523+0000","mapping_epoch":205,"log_start":"206'5474","ondisk_log_start":"206'5474","created":165,"last_epoch_clean":206,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:07:52.307205+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:07:52.307205+0000","last_clean_scrub_stamp":"2024-09-18T08:07:52.307205+0000","objects_scrubbed":0,"log_size":4530,"log_dups_size":0,"ondisk_log_size":4530,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":169476096,"num_objects":2586,"num_object_clones":0,"num_object_copies":7758,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2586,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10004,"num_write_kb":402880,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2146,"num_bytes_recovered":140640256,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,7],"acting":[4,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.9","version":"69'12","reported_seq":125,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-09-18T08:10:22.392699+0000","last_change":"2024-09-18T08:10:18.373828+0000","last_active":"2024-09-18T08:10:22.392699+0000","last_peered":"2024-09-18T08:10:22.392699+0000","last_clean":"2024-09-18T08:10:22.392699+0000","last_became_active":"2024-09-18T08:10:18.362984+0000","last_became_peered":"2024-09-18T08:10:18.362984+0000","last_unstale":"2024-09-18T08:10:22.392699+0000","last_undegraded":"2024-09-18T08:10:22.392699+0000","last_fullsized":"2024-09-18T08:10:22.392699+0000","mapping_epoch":221,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":222,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'12","last_scrub_stamp":"2024-09-18T08:05:33.544135+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_clean_scrub_stamp":"2024-09-18T08:05:33.544135+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":24,"num_read_kb":15,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":660,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,3],"acting":[4,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":382,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-09-18T08:10:22.416680+0000","last_change":"2024-09-18T08:09:39.614251+0000","last_active":"2024-09-18T08:10:22.416680+0000","last_peered":"2024-09-18T08:10:22.416680+0000","last_clean":"2024-09-18T08:10:22.416680+0000","last_became_active":"2024-09-18T08:08:18.539573+0000","last_became_peered":"2024-09-18T08:08:18.539573+0000","last_unstale":"2024-09-18T08:10:22.416680+0000","last_undegraded":"2024-09-18T08:10:22.416680+0000","last_fullsized":"2024-09-18T08:10:22.416680+0000","mapping_epoch":181,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":182,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:09:39.614187+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:46.860092+0000","last_clean_scrub_stamp":"2024-09-18T08:09:39.614187+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:10:57.546531+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":343,"reported_epoch":222,"state":"active+clean","last_fresh":"2024-09-18T08:10:18.353840+0000","last_change":"2024-09-18T08:08:15.744330+0000","last_active":"2024-09-18T08:10:18.353840+0000","last_peered":"2024-09-18T08:10:18.353840+0000","last_clean":"2024-09-18T08:10:18.353840+0000","last_became_active":"2024-09-18T08:02:51.902163+0000","last_became_peered":"2024-09-18T08:02:51.902163+0000","last_unstale":"2024-09-18T08:10:18.353840+0000","last_undegraded":"2024-09-18T08:10:18.353840+0000","last_fullsized":"2024-09-18T08:10:18.353840+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:08:15.744245+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:50.886830+0000","last_clean_scrub_stamp":"2024-09-18T08:08:15.744245+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":74,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":348,"reported_epoch":222,"state":"active+clean","last_fresh":"2024-09-18T08:10:18.364309+0000","last_change":"2024-09-18T08:10:17.494803+0000","last_active":"2024-09-18T08:10:18.364309+0000","last_peered":"2024-09-18T08:10:18.364309+0000","last_clean":"2024-09-18T08:10:18.364309+0000","last_became_active":"2024-09-18T08:02:53.907412+0000","last_became_peered":"2024-09-18T08:02:53.907412+0000","last_unstale":"2024-09-18T08:10:18.364309+0000","last_undegraded":"2024-09-18T08:10:18.364309+0000","last_fullsized":"2024-09-18T08:10:18.364309+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:10:17.494741+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_clean_scrub_stamp":"2024-09-18T08:10:17.494741+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:11:19.219802+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.4","version":"221'9765","reported_seq":24137,"reported_epoch":221,"state":"active+clean","last_fresh":"2024-09-18T08:10:18.307219+0000","last_change":"2024-09-18T08:10:15.264328+0000","last_active":"2024-09-18T08:10:18.307219+0000","last_peered":"2024-09-18T08:10:18.307219+0000","last_clean":"2024-09-18T08:10:18.307219+0000","last_became_active":"2024-09-18T08:10:15.246282+0000","last_became_peered":"2024-09-18T08:10:15.246282+0000","last_unstale":"2024-09-18T08:10:18.307219+0000","last_undegraded":"2024-09-18T08:10:18.307219+0000","last_fullsized":"2024-09-18T08:10:18.307219+0000","mapping_epoch":219,"log_start":"202'4600","ondisk_log_start":"202'4600","created":165,"last_epoch_clean":220,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:07:52.307205+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:07:52.307205+0000","last_clean_scrub_stamp":"2024-09-18T08:07:52.307205+0000","objects_scrubbed":0,"log_size":5165,"log_dups_size":0,"ondisk_log_size":5165,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":183828480,"num_objects":2805,"num_object_clones":0,"num_object_copies":8415,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2805,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9765,"num_write_kb":402240,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7107,"num_bytes_recovered":465764352,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,2],"acting":[5,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.8","version":"69'15","reported_seq":407,"reported_epoch":221,"state":"active+clean","last_fresh":"2024-09-18T08:10:17.352365+0000","last_change":"2024-09-18T08:06:24.162625+0000","last_active":"2024-09-18T08:10:17.352365+0000","last_peered":"2024-09-18T08:10:17.352365+0000","last_clean":"2024-09-18T08:10:17.352365+0000","last_became_active":"2024-09-18T08:06:24.138259+0000","last_became_peered":"2024-09-18T08:06:24.138259+0000","last_unstale":"2024-09-18T08:10:17.352365+0000","last_undegraded":"2024-09-18T08:10:17.352365+0000","last_fullsized":"2024-09-18T08:10:17.352365+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'15","last_scrub_stamp":"2024-09-18T08:04:26.859044+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_clean_scrub_stamp":"2024-09-18T08:04:26.859044+0000","objects_scrubbed":7,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.f","version":"62'2","reported_seq":398,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-09-18T08:10:22.415376+0000","last_change":"2024-09-18T08:08:18.629506+0000","last_active":"2024-09-18T08:10:22.415376+0000","last_peered":"2024-09-18T08:10:22.415376+0000","last_clean":"2024-09-18T08:10:22.415376+0000","last_became_active":"2024-09-18T08:08:18.524593+0000","last_became_peered":"2024-09-18T08:08:18.524593+0000","last_unstale":"2024-09-18T08:10:22.415376+0000","last_undegraded":"2024-09-18T08:10:22.415376+0000","last_fullsized":"2024-09-18T08:10:22.415376+0000","mapping_epoch":181,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":182,"parent":"0.0","parent_split_bits":0,"last_scrub":"62'2","last_scrub_stamp":"2024-09-18T08:07:43.540222+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:46.860092+0000","last_clean_scrub_stamp":"2024-09-18T08:07:43.540222+0000","objects_scrubbed":2,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:10:52.700103+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":4,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":184,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.9","version":"182'12","reported_seq":508,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-09-18T08:10:22.415108+0000","last_change":"2024-09-18T08:09:46.742515+0000","last_active":"2024-09-18T08:10:22.415108+0000","last_peered":"2024-09-18T08:10:22.415108+0000","last_clean":"2024-09-18T08:10:22.415108+0000","last_became_active":"2024-09-18T08:08:18.524858+0000","last_became_peered":"2024-09-18T08:08:18.524858+0000","last_unstale":"2024-09-18T08:10:22.415108+0000","last_undegraded":"2024-09-18T08:10:22.415108+0000","last_fullsized":"2024-09-18T08:10:22.415108+0000","mapping_epoch":181,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":182,"parent":"0.0","parent_split_bits":0,"last_scrub":"182'12","last_scrub_stamp":"2024-09-18T08:09:46.742442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:50.886830+0000","last_clean_scrub_stamp":"2024-09-18T08:09:46.742442+0000","objects_scrubbed":1,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:11:07.487575+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":350,"reported_epoch":221,"state":"active+clean","last_fresh":"2024-09-18T08:10:17.353392+0000","last_change":"2024-09-18T08:07:45.707631+0000","last_active":"2024-09-18T08:10:17.353392+0000","last_peered":"2024-09-18T08:10:17.353392+0000","last_clean":"2024-09-18T08:10:17.353392+0000","last_became_active":"2024-09-18T08:06:24.133119+0000","last_became_peered":"2024-09-18T08:06:24.133119+0000","last_unstale":"2024-09-18T08:10:17.353392+0000","last_undegraded":"2024-09-18T08:10:17.353392+0000","last_fullsized":"2024-09-18T08:10:17.353392+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:07:45.707562+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_clean_scrub_stamp":"2024-09-18T08:07:45.707562+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":268,"reported_epoch":222,"state":"active+clean","last_fresh":"2024-09-18T08:10:18.360247+0000","last_change":"2024-09-18T08:06:24.133955+0000","last_active":"2024-09-18T08:10:18.360247+0000","last_peered":"2024-09-18T08:10:18.360247+0000","last_clean":"2024-09-18T08:10:18.360247+0000","last_became_active":"2024-09-18T08:06:24.133644+0000","last_became_peered":"2024-09-18T08:06:24.133644+0000","last_unstale":"2024-09-18T08:10:18.360247+0000","last_undegraded":"2024-09-18T08:10:18.360247+0000","last_fullsized":"2024-09-18T08:10:18.360247+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:04:16.378205+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:46.860092+0000","last_clean_scrub_stamp":"2024-09-18T08:04:16.378205+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:10:20.052218+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.17","version":"69'6","reported_seq":372,"reported_epoch":222,"state":"active+clean","last_fresh":"2024-09-18T08:10:18.361696+0000","last_change":"2024-09-18T08:10:15.516230+0000","last_active":"2024-09-18T08:10:18.361696+0000","last_peered":"2024-09-18T08:10:18.361696+0000","last_clean":"2024-09-18T08:10:18.361696+0000","last_became_active":"2024-09-18T08:02:49.895067+0000","last_became_peered":"2024-09-18T08:02:49.895067+0000","last_unstale":"2024-09-18T08:10:18.361696+0000","last_undegraded":"2024-09-18T08:10:18.361696+0000","last_fullsized":"2024-09-18T08:10:18.361696+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'6","last_scrub_stamp":"2024-09-18T08:10:15.515914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_clean_scrub_stamp":"2024-09-18T08:10:15.515914+0000","objects_scrubbed":3,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:11:26.692514+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":9,"num_read_kb":6,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":340,"reported_epoch":221,"state":"active+clean","last_fresh":"2024-09-18T08:10:17.353706+0000","last_change":"2024-09-18T08:05:45.812436+0000","last_active":"2024-09-18T08:10:17.353706+0000","last_peered":"2024-09-18T08:10:17.353706+0000","last_clean":"2024-09-18T08:10:17.353706+0000","last_became_active":"2024-09-18T08:02:51.912557+0000","last_became_peered":"2024-09-18T08:02:51.912557+0000","last_unstale":"2024-09-18T08:10:17.353706+0000","last_undegraded":"2024-09-18T08:10:17.353706+0000","last_fullsized":"2024-09-18T08:10:17.353706+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:05:45.812389+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:50.886830+0000","last_clean_scrub_stamp":"2024-09-18T08:05:45.812389+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,1],"acting":[5,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":377,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-09-18T08:10:22.414595+0000","last_change":"2024-09-18T08:08:18.526113+0000","last_active":"2024-09-18T08:10:22.414595+0000","last_peered":"2024-09-18T08:10:22.414595+0000","last_clean":"2024-09-18T08:10:22.414595+0000","last_became_active":"2024-09-18T08:08:18.525260+0000","last_became_peered":"2024-09-18T08:08:18.525260+0000","last_unstale":"2024-09-18T08:10:22.414595+0000","last_undegraded":"2024-09-18T08:10:22.414595+0000","last_fullsized":"2024-09-18T08:10:22.414595+0000","mapping_epoch":181,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":182,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:07:53.670107+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_clean_scrub_stamp":"2024-09-18T08:07:53.670107+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"69'9","reported_seq":370,"reported_epoch":222,"state":"active+clean","last_fresh":"2024-09-18T08:10:18.366458+0000","last_change":"2024-09-18T08:09:44.116395+0000","last_active":"2024-09-18T08:10:18.366458+0000","last_peered":"2024-09-18T08:10:18.366458+0000","last_clean":"2024-09-18T08:10:18.366458+0000","last_became_active":"2024-09-18T08:02:49.891567+0000","last_became_peered":"2024-09-18T08:02:49.891567+0000","last_unstale":"2024-09-18T08:10:18.366458+0000","last_undegraded":"2024-09-18T08:10:18.366458+0000","last_fullsized":"2024-09-18T08:10:18.366458+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'9","last_scrub_stamp":"2024-09-18T08:09:44.116308+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_clean_scrub_stamp":"2024-09-18T08:09:44.116308+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:11:07.916395+0000","scrub_duration":43,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":94,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-09-18T08:10:22.392827+0000","last_change":"2024-09-18T08:08:18.550531+0000","last_active":"2024-09-18T08:10:22.392827+0000","last_peered":"2024-09-18T08:10:22.392827+0000","last_clean":"2024-09-18T08:10:22.392827+0000","last_became_active":"2024-09-18T08:08:18.545953+0000","last_became_peered":"2024-09-18T08:08:18.545953+0000","last_unstale":"2024-09-18T08:10:22.392827+0000","last_undegraded":"2024-09-18T08:10:22.392827+0000","last_fullsized":"2024-09-18T08:10:22.392827+0000","mapping_epoch":181,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":182,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:06:13.214705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:46.860092+0000","last_clean_scrub_stamp":"2024-09-18T08:06:13.214705+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":343,"reported_epoch":222,"state":"active+clean","last_fresh":"2024-09-18T08:10:18.357522+0000","last_change":"2024-09-18T08:09:46.560057+0000","last_active":"2024-09-18T08:10:18.357522+0000","last_peered":"2024-09-18T08:10:18.357522+0000","last_clean":"2024-09-18T08:10:18.357522+0000","last_became_active":"2024-09-18T08:02:51.907361+0000","last_became_peered":"2024-09-18T08:02:51.907361+0000","last_unstale":"2024-09-18T08:10:18.357522+0000","last_undegraded":"2024-09-18T08:10:18.357522+0000","last_fullsized":"2024-09-18T08:10:18.357522+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:09:46.557349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:50.886830+0000","last_clean_scrub_stamp":"2024-09-18T08:09:46.557349+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:11:08.318311+0000","scrub_duration":22,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":354,"reported_epoch":222,"state":"active+clean","last_fresh":"2024-09-18T08:10:18.357943+0000","last_change":"2024-09-18T08:08:18.511719+0000","last_active":"2024-09-18T08:10:18.357943+0000","last_peered":"2024-09-18T08:10:18.357943+0000","last_clean":"2024-09-18T08:10:18.357943+0000","last_became_active":"2024-09-18T08:08:18.511019+0000","last_became_peered":"2024-09-18T08:08:18.511019+0000","last_unstale":"2024-09-18T08:10:18.357943+0000","last_undegraded":"2024-09-18T08:10:18.357943+0000","last_fullsized":"2024-09-18T08:10:18.357943+0000","mapping_epoch":181,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":182,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:08:10.660508+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_clean_scrub_stamp":"2024-09-18T08:08:10.660508+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.15","version":"69'9","reported_seq":360,"reported_epoch":221,"state":"active+clean","last_fresh":"2024-09-18T08:10:17.354163+0000","last_change":"2024-09-18T08:05:36.896299+0000","last_active":"2024-09-18T08:10:17.354163+0000","last_peered":"2024-09-18T08:10:17.354163+0000","last_clean":"2024-09-18T08:10:17.354163+0000","last_became_active":"2024-09-18T08:02:49.890497+0000","last_became_peered":"2024-09-18T08:02:49.890497+0000","last_unstale":"2024-09-18T08:10:17.354163+0000","last_undegraded":"2024-09-18T08:10:17.354163+0000","last_fullsized":"2024-09-18T08:10:17.354163+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'9","last_scrub_stamp":"2024-09-18T08:05:36.896238+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_clean_scrub_stamp":"2024-09-18T08:05:36.896238+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":366,"reported_epoch":222,"state":"active+clean","last_fresh":"2024-09-18T08:10:18.360985+0000","last_change":"2024-09-18T08:09:46.035501+0000","last_active":"2024-09-18T08:10:18.360985+0000","last_peered":"2024-09-18T08:10:18.360985+0000","last_clean":"2024-09-18T08:10:18.360985+0000","last_became_active":"2024-09-18T08:02:47.875983+0000","last_became_peered":"2024-09-18T08:02:47.875983+0000","last_unstale":"2024-09-18T08:10:18.360985+0000","last_undegraded":"2024-09-18T08:10:18.360985+0000","last_fullsized":"2024-09-18T08:10:18.360985+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:09:46.035332+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:46.860092+0000","last_clean_scrub_stamp":"2024-09-18T08:09:46.035332+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:10:47.519129+0000","scrub_duration":18,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.14","version":"199'12","reported_seq":474,"reported_epoch":222,"state":"active+clean","last_fresh":"2024-09-18T08:10:18.357209+0000","last_change":"2024-09-18T08:10:10.449130+0000","last_active":"2024-09-18T08:10:18.357209+0000","last_peered":"2024-09-18T08:10:18.357209+0000","last_clean":"2024-09-18T08:10:18.357209+0000","last_became_active":"2024-09-18T08:08:52.696201+0000","last_became_peered":"2024-09-18T08:08:52.696201+0000","last_unstale":"2024-09-18T08:10:18.357209+0000","last_undegraded":"2024-09-18T08:10:18.357209+0000","last_fullsized":"2024-09-18T08:10:18.357209+0000","mapping_epoch":198,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":199,"parent":"0.0","parent_split_bits":0,"last_scrub":"199'12","last_scrub_stamp":"2024-09-18T08:10:10.449024+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:50.886830+0000","last_clean_scrub_stamp":"2024-09-18T08:10:10.449024+0000","objects_scrubbed":1,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:11:36.934868+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":94,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-09-18T08:10:22.393661+0000","last_change":"2024-09-18T08:08:18.523205+0000","last_active":"2024-09-18T08:10:22.393661+0000","last_peered":"2024-09-18T08:10:22.393661+0000","last_clean":"2024-09-18T08:10:22.393661+0000","last_became_active":"2024-09-18T08:08:18.522483+0000","last_became_peered":"2024-09-18T08:08:18.522483+0000","last_unstale":"2024-09-18T08:10:22.393661+0000","last_undegraded":"2024-09-18T08:10:22.393661+0000","last_fullsized":"2024-09-18T08:10:22.393661+0000","mapping_epoch":181,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":182,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:06:43.510513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_clean_scrub_stamp":"2024-09-18T08:06:43.510513+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.14","version":"69'10","reported_seq":381,"reported_epoch":222,"state":"active+clean","last_fresh":"2024-09-18T08:10:18.357555+0000","last_change":"2024-09-18T08:08:08.493614+0000","last_active":"2024-09-18T08:10:18.357555+0000","last_peered":"2024-09-18T08:10:18.357555+0000","last_clean":"2024-09-18T08:10:18.357555+0000","last_became_active":"2024-09-18T08:04:18.150968+0000","last_became_peered":"2024-09-18T08:04:18.150968+0000","last_unstale":"2024-09-18T08:10:18.357555+0000","last_undegraded":"2024-09-18T08:10:18.357555+0000","last_fullsized":"2024-09-18T08:10:18.357555+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'10","last_scrub_stamp":"2024-09-18T08:08:08.493522+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_clean_scrub_stamp":"2024-09-18T08:08:08.493522+0000","objects_scrubbed":5,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,7],"acting":[3,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":374,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-09-18T08:10:22.415173+0000","last_change":"2024-09-18T08:09:31.658337+0000","last_active":"2024-09-18T08:10:22.415173+0000","last_peered":"2024-09-18T08:10:22.415173+0000","last_clean":"2024-09-18T08:10:22.415173+0000","last_became_active":"2024-09-18T08:08:18.524218+0000","last_became_peered":"2024-09-18T08:08:18.524218+0000","last_unstale":"2024-09-18T08:10:22.415173+0000","last_undegraded":"2024-09-18T08:10:22.415173+0000","last_fullsized":"2024-09-18T08:10:22.415173+0000","mapping_epoch":181,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":182,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:09:31.658259+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:46.860092+0000","last_clean_scrub_stamp":"2024-09-18T08:09:31.658259+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:10:50.143994+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.15","version":"69'8","reported_seq":447,"reported_epoch":221,"state":"active+clean","last_fresh":"2024-09-18T08:10:17.353942+0000","last_change":"2024-09-18T08:06:45.555513+0000","last_active":"2024-09-18T08:10:17.353942+0000","last_peered":"2024-09-18T08:10:17.353942+0000","last_clean":"2024-09-18T08:10:17.353942+0000","last_became_active":"2024-09-18T08:02:51.914135+0000","last_became_peered":"2024-09-18T08:02:51.914135+0000","last_unstale":"2024-09-18T08:10:17.353942+0000","last_undegraded":"2024-09-18T08:10:17.353942+0000","last_fullsized":"2024-09-18T08:10:17.353942+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'8","last_scrub_stamp":"2024-09-18T08:06:45.555459+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:50.886830+0000","last_clean_scrub_stamp":"2024-09-18T08:06:45.555459+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":347,"reported_epoch":222,"state":"active+clean","last_fresh":"2024-09-18T08:10:18.363768+0000","last_change":"2024-09-18T08:09:42.047337+0000","last_active":"2024-09-18T08:10:18.363768+0000","last_peered":"2024-09-18T08:10:18.363768+0000","last_clean":"2024-09-18T08:10:18.363768+0000","last_became_active":"2024-09-18T08:02:53.912112+0000","last_became_peered":"2024-09-18T08:02:53.912112+0000","last_unstale":"2024-09-18T08:10:18.363768+0000","last_undegraded":"2024-09-18T08:10:18.363768+0000","last_fullsized":"2024-09-18T08:10:18.363768+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:09:42.047270+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_clean_scrub_stamp":"2024-09-18T08:09:42.047270+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:10:59.660034+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.13","version":"69'11","reported_seq":106,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-09-18T08:10:22.393503+0000","last_change":"2024-09-18T08:08:18.643772+0000","last_active":"2024-09-18T08:10:22.393503+0000","last_peered":"2024-09-18T08:10:22.393503+0000","last_clean":"2024-09-18T08:10:22.393503+0000","last_became_active":"2024-09-18T08:08:18.539712+0000","last_became_peered":"2024-09-18T08:08:18.539712+0000","last_unstale":"2024-09-18T08:10:22.393503+0000","last_undegraded":"2024-09-18T08:10:22.393503+0000","last_fullsized":"2024-09-18T08:10:22.393503+0000","mapping_epoch":181,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":182,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'11","last_scrub_stamp":"2024-09-18T08:07:53.824401+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_clean_scrub_stamp":"2024-09-18T08:07:53.824401+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":19,"num_read_kb":12,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":94,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-09-18T08:10:22.393586+0000","last_change":"2024-09-18T08:08:18.623643+0000","last_active":"2024-09-18T08:10:22.393586+0000","last_peered":"2024-09-18T08:10:22.393586+0000","last_clean":"2024-09-18T08:10:22.393586+0000","last_became_active":"2024-09-18T08:08:18.622893+0000","last_became_peered":"2024-09-18T08:08:18.622893+0000","last_unstale":"2024-09-18T08:10:22.393586+0000","last_undegraded":"2024-09-18T08:10:22.393586+0000","last_fullsized":"2024-09-18T08:10:22.393586+0000","mapping_epoch":181,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":182,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:06:46.502422+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:46.860092+0000","last_clean_scrub_stamp":"2024-09-18T08:06:46.502422+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":350,"reported_epoch":222,"state":"active+clean","last_fresh":"2024-09-18T08:10:18.358148+0000","last_change":"2024-09-18T08:09:03.582984+0000","last_active":"2024-09-18T08:10:18.358148+0000","last_peered":"2024-09-18T08:10:18.358148+0000","last_clean":"2024-09-18T08:10:18.358148+0000","last_became_active":"2024-09-18T08:02:51.910752+0000","last_became_peered":"2024-09-18T08:02:51.910752+0000","last_unstale":"2024-09-18T08:10:18.358148+0000","last_undegraded":"2024-09-18T08:10:18.358148+0000","last_fullsized":"2024-09-18T08:10:18.358148+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:09:03.582914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:50.886830+0000","last_clean_scrub_stamp":"2024-09-18T08:09:03.582914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":348,"reported_epoch":222,"state":"active+clean","last_fresh":"2024-09-18T08:10:18.357333+0000","last_change":"2024-09-18T08:07:54.314026+0000","last_active":"2024-09-18T08:10:18.357333+0000","last_peered":"2024-09-18T08:10:18.357333+0000","last_clean":"2024-09-18T08:10:18.357333+0000","last_became_active":"2024-09-18T08:02:53.911429+0000","last_became_peered":"2024-09-18T08:02:53.911429+0000","last_unstale":"2024-09-18T08:10:18.357333+0000","last_undegraded":"2024-09-18T08:10:18.357333+0000","last_fullsized":"2024-09-18T08:10:18.357333+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:07:54.313971+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_clean_scrub_stamp":"2024-09-18T08:07:54.313971+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.12","version":"69'9","reported_seq":396,"reported_epoch":222,"state":"active+clean","last_fresh":"2024-09-18T08:10:18.359233+0000","last_change":"2024-09-18T08:10:06.582865+0000","last_active":"2024-09-18T08:10:18.359233+0000","last_peered":"2024-09-18T08:10:18.359233+0000","last_clean":"2024-09-18T08:10:18.359233+0000","last_became_active":"2024-09-18T08:06:24.132270+0000","last_became_peered":"2024-09-18T08:06:24.132270+0000","last_unstale":"2024-09-18T08:10:18.359233+0000","last_undegraded":"2024-09-18T08:10:18.359233+0000","last_fullsized":"2024-09-18T08:10:18.359233+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'9","last_scrub_stamp":"2024-09-18T08:10:06.582804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_clean_scrub_stamp":"2024-09-18T08:10:06.582804+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:11:13.834917+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":382,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-09-18T08:10:22.415533+0000","last_change":"2024-09-18T08:09:37.532792+0000","last_active":"2024-09-18T08:10:22.415533+0000","last_peered":"2024-09-18T08:10:22.415533+0000","last_clean":"2024-09-18T08:10:22.415533+0000","last_became_active":"2024-09-18T08:08:18.525527+0000","last_became_peered":"2024-09-18T08:08:18.525527+0000","last_unstale":"2024-09-18T08:10:22.415533+0000","last_undegraded":"2024-09-18T08:10:22.415533+0000","last_fullsized":"2024-09-18T08:10:22.415533+0000","mapping_epoch":181,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":182,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:09:37.532573+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:46.860092+0000","last_clean_scrub_stamp":"2024-09-18T08:09:37.532573+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:11:05.489513+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":343,"reported_epoch":222,"state":"active+clean","last_fresh":"2024-09-18T08:10:18.364424+0000","last_change":"2024-09-18T08:07:42.165434+0000","last_active":"2024-09-18T08:10:18.364424+0000","last_peered":"2024-09-18T08:10:18.364424+0000","last_clean":"2024-09-18T08:10:18.364424+0000","last_became_active":"2024-09-18T08:02:51.906277+0000","last_became_peered":"2024-09-18T08:02:51.906277+0000","last_unstale":"2024-09-18T08:10:18.364424+0000","last_undegraded":"2024-09-18T08:10:18.364424+0000","last_fullsized":"2024-09-18T08:10:18.364424+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:07:42.165388+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:50.886830+0000","last_clean_scrub_stamp":"2024-09-18T08:07:42.165388+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.10","version":"69'1","reported_seq":335,"reported_epoch":222,"state":"active+clean","last_fresh":"2024-09-18T08:10:18.359935+0000","last_change":"2024-09-18T08:10:17.879036+0000","last_active":"2024-09-18T08:10:18.359935+0000","last_peered":"2024-09-18T08:10:18.359935+0000","last_clean":"2024-09-18T08:10:18.359935+0000","last_became_active":"2024-09-18T08:02:53.912699+0000","last_became_peered":"2024-09-18T08:02:53.912699+0000","last_unstale":"2024-09-18T08:10:18.359935+0000","last_undegraded":"2024-09-18T08:10:18.359935+0000","last_fullsized":"2024-09-18T08:10:18.359935+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'1","last_scrub_stamp":"2024-09-18T08:10:17.878965+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_clean_scrub_stamp":"2024-09-18T08:10:17.878965+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:11:23.854550+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.11","version":"69'11","reported_seq":420,"reported_epoch":222,"state":"active+clean","last_fresh":"2024-09-18T08:10:18.357575+0000","last_change":"2024-09-18T08:10:16.510963+0000","last_active":"2024-09-18T08:10:18.357575+0000","last_peered":"2024-09-18T08:10:18.357575+0000","last_clean":"2024-09-18T08:10:18.357575+0000","last_became_active":"2024-09-18T08:06:24.132416+0000","last_became_peered":"2024-09-18T08:06:24.132416+0000","last_unstale":"2024-09-18T08:10:18.357575+0000","last_undegraded":"2024-09-18T08:10:18.357575+0000","last_fullsized":"2024-09-18T08:10:18.357575+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'11","last_scrub_stamp":"2024-09-18T08:10:16.510871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_clean_scrub_stamp":"2024-09-18T08:10:16.510871+0000","objects_scrubbed":5,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:11:20.303978+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":348,"reported_epoch":221,"state":"active+clean","last_fresh":"2024-09-18T08:10:17.354419+0000","last_change":"2024-09-18T08:06:18.168254+0000","last_active":"2024-09-18T08:10:17.354419+0000","last_peered":"2024-09-18T08:10:17.354419+0000","last_clean":"2024-09-18T08:10:17.354419+0000","last_became_active":"2024-09-18T08:02:47.878886+0000","last_became_peered":"2024-09-18T08:02:47.878886+0000","last_unstale":"2024-09-18T08:10:17.354419+0000","last_undegraded":"2024-09-18T08:10:17.354419+0000","last_fullsized":"2024-09-18T08:10:17.354419+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:06:18.168206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:46.860092+0000","last_clean_scrub_stamp":"2024-09-18T08:06:18.168206+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":388,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-09-18T08:10:22.413683+0000","last_change":"2024-09-18T08:09:49.824264+0000","last_active":"2024-09-18T08:10:22.413683+0000","last_peered":"2024-09-18T08:10:22.413683+0000","last_clean":"2024-09-18T08:10:22.413683+0000","last_became_active":"2024-09-18T08:08:18.524966+0000","last_became_peered":"2024-09-18T08:08:18.524966+0000","last_unstale":"2024-09-18T08:10:22.413683+0000","last_undegraded":"2024-09-18T08:10:22.413683+0000","last_fullsized":"2024-09-18T08:10:22.413683+0000","mapping_epoch":181,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":182,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:09:49.824196+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:50.886830+0000","last_clean_scrub_stamp":"2024-09-18T08:09:49.824196+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:10:51.928894+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":353,"reported_epoch":222,"state":"active+clean","last_fresh":"2024-09-18T08:10:18.364604+0000","last_change":"2024-09-18T08:07:48.218010+0000","last_active":"2024-09-18T08:10:18.364604+0000","last_peered":"2024-09-18T08:10:18.364604+0000","last_clean":"2024-09-18T08:10:18.364604+0000","last_became_active":"2024-09-18T08:06:24.128582+0000","last_became_peered":"2024-09-18T08:06:24.128582+0000","last_unstale":"2024-09-18T08:10:18.364604+0000","last_undegraded":"2024-09-18T08:10:18.364604+0000","last_fullsized":"2024-09-18T08:10:18.364604+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:07:48.217964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_clean_scrub_stamp":"2024-09-18T08:07:48.217964+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.10","version":"69'4","reported_seq":387,"reported_epoch":222,"state":"active+clean","last_fresh":"2024-09-18T08:10:18.543267+0000","last_change":"2024-09-18T08:10:18.543267+0000","last_active":"2024-09-18T08:10:18.543267+0000","last_peered":"2024-09-18T08:10:18.543267+0000","last_clean":"2024-09-18T08:10:18.543267+0000","last_became_active":"2024-09-18T08:06:24.131309+0000","last_became_peered":"2024-09-18T08:06:24.131309+0000","last_unstale":"2024-09-18T08:10:18.543267+0000","last_undegraded":"2024-09-18T08:10:18.543267+0000","last_fullsized":"2024-09-18T08:10:18.543267+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'4","last_scrub_stamp":"2024-09-18T08:10:18.542984+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_clean_scrub_stamp":"2024-09-18T08:10:18.542984+0000","objects_scrubbed":2,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:11:33.720540+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":6,"num_read_kb":4,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":366,"reported_epoch":222,"state":"active+clean","last_fresh":"2024-09-18T08:10:18.362124+0000","last_change":"2024-09-18T08:10:13.820205+0000","last_active":"2024-09-18T08:10:18.362124+0000","last_peered":"2024-09-18T08:10:18.362124+0000","last_clean":"2024-09-18T08:10:18.362124+0000","last_became_active":"2024-09-18T08:02:47.877616+0000","last_became_peered":"2024-09-18T08:02:47.877616+0000","last_unstale":"2024-09-18T08:10:18.362124+0000","last_undegraded":"2024-09-18T08:10:18.362124+0000","last_fullsized":"2024-09-18T08:10:18.362124+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:10:13.820142+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:46.860092+0000","last_clean_scrub_stamp":"2024-09-18T08:10:13.820142+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:11:32.099137+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":94,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-09-18T08:10:22.393002+0000","last_change":"2024-09-18T08:08:18.550244+0000","last_active":"2024-09-18T08:10:22.393002+0000","last_peered":"2024-09-18T08:10:22.393002+0000","last_clean":"2024-09-18T08:10:22.393002+0000","last_became_active":"2024-09-18T08:08:18.545361+0000","last_became_peered":"2024-09-18T08:08:18.545361+0000","last_unstale":"2024-09-18T08:10:22.393002+0000","last_undegraded":"2024-09-18T08:10:22.393002+0000","last_fullsized":"2024-09-18T08:10:22.393002+0000","mapping_epoch":181,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":182,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:07:17.605562+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:50.886830+0000","last_clean_scrub_stamp":"2024-09-18T08:07:17.605562+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,7],"acting":[4,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":371,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-09-18T08:10:22.413989+0000","last_change":"2024-09-18T08:09:34.607443+0000","last_active":"2024-09-18T08:10:22.413989+0000","last_peered":"2024-09-18T08:10:22.413989+0000","last_clean":"2024-09-18T08:10:22.413989+0000","last_became_active":"2024-09-18T08:08:18.525552+0000","last_became_peered":"2024-09-18T08:08:18.525552+0000","last_unstale":"2024-09-18T08:10:22.413989+0000","last_undegraded":"2024-09-18T08:10:22.413989+0000","last_fullsized":"2024-09-18T08:10:22.413989+0000","mapping_epoch":181,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":182,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:09:34.607374+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_clean_scrub_stamp":"2024-09-18T08:09:34.607374+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:10:39.783401+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,4],"acting":[7,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":336,"reported_epoch":222,"state":"active+clean","last_fresh":"2024-09-18T08:10:18.357087+0000","last_change":"2024-09-18T08:08:18.539359+0000","last_active":"2024-09-18T08:10:18.357087+0000","last_peered":"2024-09-18T08:10:18.357087+0000","last_clean":"2024-09-18T08:10:18.357087+0000","last_became_active":"2024-09-18T08:08:18.538838+0000","last_became_peered":"2024-09-18T08:08:18.538838+0000","last_unstale":"2024-09-18T08:10:18.357087+0000","last_undegraded":"2024-09-18T08:10:18.357087+0000","last_fullsized":"2024-09-18T08:10:18.357087+0000","mapping_epoch":181,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":182,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_clean_scrub_stamp":"2024-09-18T08:02:52.890475+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:10:51.616752+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":351,"reported_epoch":222,"state":"active+clean","last_fresh":"2024-09-18T08:10:18.363519+0000","last_change":"2024-09-18T08:07:17.277964+0000","last_active":"2024-09-18T08:10:18.363519+0000","last_peered":"2024-09-18T08:10:18.363519+0000","last_clean":"2024-09-18T08:10:18.363519+0000","last_became_active":"2024-09-18T08:02:47.878922+0000","last_became_peered":"2024-09-18T08:02:47.878922+0000","last_unstale":"2024-09-18T08:10:18.363519+0000","last_undegraded":"2024-09-18T08:10:18.363519+0000","last_fullsized":"2024-09-18T08:10:18.363519+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:07:17.277894+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:46.860092+0000","last_clean_scrub_stamp":"2024-09-18T08:07:17.277894+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:11:04.482372+0000","scrub_duration":16,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"69'11","reported_seq":337,"reported_epoch":221,"state":"active+clean","last_fresh":"2024-09-18T08:10:17.354139+0000","last_change":"2024-09-18T08:06:24.164086+0000","last_active":"2024-09-18T08:10:17.354139+0000","last_peered":"2024-09-18T08:10:17.354139+0000","last_clean":"2024-09-18T08:10:17.354139+0000","last_became_active":"2024-09-18T08:06:24.135232+0000","last_became_peered":"2024-09-18T08:06:24.135232+0000","last_unstale":"2024-09-18T08:10:17.354139+0000","last_undegraded":"2024-09-18T08:10:17.354139+0000","last_fullsized":"2024-09-18T08:10:17.354139+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'11","last_scrub_stamp":"2024-09-18T08:06:12.967462+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_clean_scrub_stamp":"2024-09-18T08:06:12.967462+0000","objects_scrubbed":5,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":19,"num_read_kb":12,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,1],"acting":[5,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":356,"reported_epoch":222,"state":"active+clean","last_fresh":"2024-09-18T08:10:18.366171+0000","last_change":"2024-09-18T08:08:52.701053+0000","last_active":"2024-09-18T08:10:18.366171+0000","last_peered":"2024-09-18T08:10:18.366171+0000","last_clean":"2024-09-18T08:10:18.366171+0000","last_became_active":"2024-09-18T08:08:52.700595+0000","last_became_peered":"2024-09-18T08:08:52.700595+0000","last_unstale":"2024-09-18T08:10:18.366171+0000","last_undegraded":"2024-09-18T08:10:18.366171+0000","last_fullsized":"2024-09-18T08:10:18.366171+0000","mapping_epoch":198,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":199,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:08:12.639376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:50.886830+0000","last_clean_scrub_stamp":"2024-09-18T08:08:12.639376+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":8,"num_pg":16,"stat_sum":{"num_bytes":2964652056,"num_objects":45238,"num_object_clones":0,"num_object_copies":135714,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":45238,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":156262,"num_write_kb":6447937,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24244,"num_bytes_recovered":1588854784,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":9545396224,"data_stored":9545384008,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":78588,"ondisk_log_size":78588,"up":48,"acting":48,"num_store_stats":8},{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":416,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":85,"ondisk_log_size":85,"up":96,"acting":96,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":178,"num_object_clones":0,"num_object_copies":534,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":178,"num_whiteouts":0,"num_read":685,"num_read_kb":442,"num_write":417,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":219,"num_bytes_recovered":4334,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":409600,"data_stored":10886,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":398,"ondisk_log_size":398,"up":96,"acting":96,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":1613,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":24,"num_read_kb":24,"num_write":10,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":3226,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":4839,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":6,"ondisk_log_size":6,"up":96,"acting":96,"num_store_stats":8},{"poolid":2,"num_pg":3,"stat_sum":{"num_bytes":408,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":218,"num_read_kb":213,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":38,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":8,"ondisk_log_size":8,"up":9,"acting":9,"num_store_stats":7},{"poolid":1,"num_pg":11,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":869,"ondisk_log_size":869,"up":33,"acting":33,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":56,"seq":240518168671,"num_pgs":63,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1004760,"kb_used_data":820868,"kb_used_omap":1,"kb_used_meta":183870,"kb_avail":92740392,"statfs":{"total":95995035648,"available":94966161408,"internally_reserved":0,"allocated":840568832,"data_stored":840188178,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":188283338},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":47,"seq":201863463013,"num_pgs":54,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1286784,"kb_used_data":1075116,"kb_used_omap":1,"kb_used_meta":211646,"kb_avail":92458368,"statfs":{"total":95995035648,"available":94677368832,"internally_reserved":0,"allocated":1100918784,"data_stored":1100561597,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":216725958},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":39,"seq":167503724650,"num_pgs":59,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1676476,"kb_used_data":1482792,"kb_used_omap":1,"kb_used_meta":193598,"kb_avail":92068676,"statfs":{"total":95995035648,"available":94278324224,"internally_reserved":0,"allocated":1518379008,"data_stored":1518013573,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":198244806},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,2,1,3],"upper_bound":16},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":32,"seq":137438953584,"num_pgs":67,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1382608,"kb_used_data":1148736,"kb_used_omap":1,"kb_used_meta":233854,"kb_avail":92362544,"statfs":{"total":95995035648,"available":94579245056,"internally_reserved":0,"allocated":1176305664,"data_stored":1175922739,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":239466949},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":26,"seq":111669149814,"num_pgs":68,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1459740,"kb_used_data":1149512,"kb_used_omap":1,"kb_used_meta":310206,"kb_avail":92285412,"statfs":{"total":95995035648,"available":94500261888,"internally_reserved":0,"allocated":1177100288,"data_stored":1176709008,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":317651400},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,2,0,2],"upper_bound":16},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411452,"num_pgs":49,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1431756,"kb_used_data":1111796,"kb_used_omap":1,"kb_used_meta":319934,"kb_avail":92313396,"statfs":{"total":95995035648,"available":94528917504,"internally_reserved":0,"allocated":1138479104,"data_stored":1138108351,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":327612869},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,1,1,1],"upper_bound":16},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574976,"num_pgs":63,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1554096,"kb_used_data":1292440,"kb_used_omap":1,"kb_used_meta":261630,"kb_avail":92191056,"statfs":{"total":95995035648,"available":94403641344,"internally_reserved":0,"allocated":1323458560,"data_stored":1323052138,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":267909573},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,1,1,6],"upper_bound":32},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738503,"num_pgs":56,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1623748,"kb_used_data":1349620,"kb_used_omap":1,"kb_used_meta":274110,"kb_avail":92121404,"statfs":{"total":95995035648,"available":94332317696,"internally_reserved":0,"allocated":1382010880,"data_stored":1381633581,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":280689093},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":266240,"data_stored":262688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1131,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1521,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":436,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":46,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1177,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":436,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":69632,"data_stored":1788,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1128,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":1898,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1474,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":990,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1034,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1254,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":403,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":416,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":403,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":1346568192,"data_stored":1346568192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":1318453248,"data_stored":1318453248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":1133449216,"data_stored":1133445144,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":1165492224,"data_stored":1165492224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":1152188416,"data_stored":1152188416,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":1500844032,"data_stored":1500839960,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":1099890688,"data_stored":1099890688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":828510208,"data_stored":828506136,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T08:10:29.665 INFO:tasks.cephadm.ceph_manager.ceph:recovered! 2024-09-18T08:10:29.665 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-18T08:10:29.665 INFO:tasks.thrashosds.thrasher:inject_pause on osd.3 2024-09-18T08:10:29.665 INFO:tasks.thrashosds.thrasher:Testing filestore_inject_stall pause injection for duration 3 2024-09-18T08:10:29.665 INFO:tasks.thrashosds.thrasher:Checking after 0, should_be_down=False 2024-09-18T08:10:29.666 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid 779349ca-7593-11ef-bceb-c7b262605968 -- ceph daemon osd.3 config set filestore_inject_stall 3 2024-09-18T08:10:29.693 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/779349ca-7593-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T08:10:29.750 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2 2024-09-18T08:10:29.800 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:10:29.800 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:10:29.800 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2' 2024-09-18T08:10:29.875 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/779349ca-7593-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T08:10:29.902 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6 2024-09-18T08:10:29.927 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:10:29.927 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:10:29.928 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6' 2024-09-18T08:10:30.029 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1 2024-09-18T08:10:30.089 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:10:30.089 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:10:30.090 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1' 2024-09-18T08:10:30.191 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2 2024-09-18T08:10:30.236 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:10:30.237 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:10:30.237 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2' 2024-09-18T08:10:30.338 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1 2024-09-18T08:10:30.374 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:10:30.375 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:10:30.375 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1' 2024-09-18T08:10:30.476 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2 2024-09-18T08:10:30.499 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:10:30.500 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:10:30.500 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2' 2024-09-18T08:10:30.601 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2 2024-09-18T08:10:30.641 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:10:30.641 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:10:30.642 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2' 2024-09-18T08:10:30.730 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:10:30 smithi007 ceph-mon[29459]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-18T08:10:30.730 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:10:30 smithi007 ceph-mon[29459]: pgmap v369: 158 pgs: 158 active+clean; 2.1 GiB data, 8.9 GiB used, 706 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.23k op/s; 21 B/s, 0 objects/s recovering 2024-09-18T08:10:30.731 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:10:30 smithi007 ceph-mon[39249]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-18T08:10:30.731 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:10:30 smithi007 ceph-mon[39249]: pgmap v369: 158 pgs: 158 active+clean; 2.1 GiB data, 8.9 GiB used, 706 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.23k op/s; 21 B/s, 0 objects/s recovering 2024-09-18T08:10:30.743 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1 2024-09-18T08:10:30.785 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:10:30.786 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:10:30.786 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1' 2024-09-18T08:10:30.887 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1 2024-09-18T08:10:30.905 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:10:30 smithi050 ceph-mon[32371]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-18T08:10:30.906 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:10:30 smithi050 ceph-mon[32371]: pgmap v369: 158 pgs: 158 active+clean; 2.1 GiB data, 8.9 GiB used, 706 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.23k op/s; 21 B/s, 0 objects/s recovering 2024-09-18T08:10:30.921 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:10:30.922 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:10:30.922 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1' 2024-09-18T08:10:31.022 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4 2024-09-18T08:10:31.053 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:10:31.054 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:10:31.054 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4' 2024-09-18T08:10:31.155 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6 2024-09-18T08:10:31.190 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:10:31.191 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:10:31.191 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6' 2024-09-18T08:10:31.292 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0 2024-09-18T08:10:31.320 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:10:31.321 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:10:31.321 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0' 2024-09-18T08:10:31.422 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7 2024-09-18T08:10:31.455 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:10:31.455 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:10:31.456 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7' 2024-09-18T08:10:31.557 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3 2024-09-18T08:10:31.581 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:10:31.581 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:10:31.582 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3' 2024-09-18T08:10:31.683 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1 2024-09-18T08:10:31.719 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:10:31.720 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:10:31.720 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1' 2024-09-18T08:10:31.821 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5 2024-09-18T08:10:31.850 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:10:31.851 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:10:31.851 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5' 2024-09-18T08:10:31.952 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3 2024-09-18T08:10:31.995 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:10:31.996 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:10:31.996 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3' 2024-09-18T08:10:32.097 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6 2024-09-18T08:10:32.123 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:10:32.124 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:10:32.124 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6' 2024-09-18T08:10:32.212 INFO:teuthology.orchestra.run.smithi007.stdout:{ 2024-09-18T08:10:32.213 INFO:teuthology.orchestra.run.smithi007.stdout: "ops": [], 2024-09-18T08:10:32.213 INFO:teuthology.orchestra.run.smithi007.stdout: "complaint_time": 30, 2024-09-18T08:10:32.213 INFO:teuthology.orchestra.run.smithi007.stdout: "num_blocked_ops": 0 2024-09-18T08:10:32.213 INFO:teuthology.orchestra.run.smithi007.stdout:} 2024-09-18T08:10:32.225 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1 2024-09-18T08:10:32.271 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:10:32.271 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:10:32.271 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1' 2024-09-18T08:10:32.372 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6 2024-09-18T08:10:32.400 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:10:32.400 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:10:32.401 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6' 2024-09-18T08:10:32.502 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4 2024-09-18T08:10:32.536 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:10:32.537 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:10:32.537 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4' 2024-09-18T08:10:32.638 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3 2024-09-18T08:10:32.676 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:10:32.676 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:10:32.676 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3' 2024-09-18T08:10:32.730 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:10:32 smithi007 ceph-mon[29459]: pgmap v370: 158 pgs: 158 active+clean; 2.1 GiB data, 8.9 GiB used, 706 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.23k op/s; 21 B/s, 0 objects/s recovering 2024-09-18T08:10:32.730 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:10:32 smithi007 ceph-mon[39249]: pgmap v370: 158 pgs: 158 active+clean; 2.1 GiB data, 8.9 GiB used, 706 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.23k op/s; 21 B/s, 0 objects/s recovering 2024-09-18T08:10:32.777 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5 2024-09-18T08:10:32.808 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:10:32.809 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:10:32.810 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5' 2024-09-18T08:10:32.906 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:10:32 smithi050 ceph-mon[32371]: pgmap v370: 158 pgs: 158 active+clean; 2.1 GiB data, 8.9 GiB used, 706 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.23k op/s; 21 B/s, 0 objects/s recovering 2024-09-18T08:10:32.911 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4 2024-09-18T08:10:32.947 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:10:32.948 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:10:32.948 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4' 2024-09-18T08:10:33.049 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0 2024-09-18T08:10:33.083 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:10:33.084 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:10:33.085 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0' 2024-09-18T08:10:33.185 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0 2024-09-18T08:10:33.227 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:10:33.227 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:10:33.228 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0' 2024-09-18T08:10:33.299 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/779349ca-7593-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T08:10:33.311 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid 779349ca-7593-11ef-bceb-c7b262605968 -- ceph daemon osd.1 dump_historic_ops 2024-09-18T08:10:33.328 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2 2024-09-18T08:10:33.347 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/779349ca-7593-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T08:10:33.401 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:10:33.402 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:10:33.402 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2' 2024-09-18T08:10:33.455 INFO:teuthology.orchestra.run.smithi007.stderr:nodeep-scrub is set 2024-09-18T08:10:33.504 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2 2024-09-18T08:10:33.519 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:10:33 smithi007 ceph-mon[39249]: from='client.? 172.21.15.7:0/187319810' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-18T08:10:33.519 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:10:33 smithi007 ceph-mon[39249]: from='mgr.24470 172.21.15.7:0/2594390261' entity='mgr.y' 2024-09-18T08:10:33.519 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:10:33 smithi007 ceph-mon[39249]: from='mgr.24470 172.21.15.7:0/2594390261' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-18T08:10:33.519 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:10:33 smithi007 ceph-mon[29459]: from='client.? 172.21.15.7:0/187319810' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-18T08:10:33.520 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:10:33 smithi007 ceph-mon[29459]: from='mgr.24470 172.21.15.7:0/2594390261' entity='mgr.y' 2024-09-18T08:10:33.520 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:10:33 smithi007 ceph-mon[29459]: from='mgr.24470 172.21.15.7:0/2594390261' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-18T08:10:33.548 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:10:33.549 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:10:33.549 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2' 2024-09-18T08:10:33.650 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2 2024-09-18T08:10:33.710 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:10:33.711 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:10:33.711 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2' 2024-09-18T08:10:33.812 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3 2024-09-18T08:10:33.852 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:10:33.853 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:10:33.853 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3' 2024-09-18T08:10:33.906 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:10:33 smithi050 ceph-mon[32371]: from='client.? 172.21.15.7:0/187319810' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-18T08:10:33.906 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:10:33 smithi050 ceph-mon[32371]: from='mgr.24470 172.21.15.7:0/2594390261' entity='mgr.y' 2024-09-18T08:10:33.906 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:10:33 smithi050 ceph-mon[32371]: from='mgr.24470 172.21.15.7:0/2594390261' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-18T08:10:33.954 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3 2024-09-18T08:10:33.992 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:10:33.993 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:10:33.993 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3' 2024-09-18T08:10:34.094 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2 2024-09-18T08:10:34.134 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:10:34.135 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:10:34.135 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2' 2024-09-18T08:10:34.236 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1 2024-09-18T08:10:34.271 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:10:34.272 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:10:34.272 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1' 2024-09-18T08:10:34.373 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6 2024-09-18T08:10:34.405 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:10:34.405 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:10:34.406 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6' 2024-09-18T08:10:34.507 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6 2024-09-18T08:10:34.536 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:10:34.537 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:10:34.538 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6' 2024-09-18T08:10:34.638 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2 2024-09-18T08:10:34.644 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:10:34 smithi007 ceph-mon[29459]: pgmap v371: 158 pgs: 158 active+clean; 2.0 GiB data, 8.6 GiB used, 707 GiB / 715 GiB avail; 943 B/s rd, 0 B/s wr, 1.21k op/s; 20 B/s, 0 objects/s recovering 2024-09-18T08:10:34.644 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:10:34 smithi007 ceph-mon[29459]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-18T08:10:34.644 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:10:34 smithi007 ceph-mon[29459]: from='client.? 172.21.15.7:0/187319810' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-18T08:10:34.644 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:10:34 smithi007 ceph-mon[29459]: osdmap e224: 8 total, 8 up, 8 in 2024-09-18T08:10:34.645 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:10:34 smithi007 ceph-mon[39249]: pgmap v371: 158 pgs: 158 active+clean; 2.0 GiB data, 8.6 GiB used, 707 GiB / 715 GiB avail; 943 B/s rd, 0 B/s wr, 1.21k op/s; 20 B/s, 0 objects/s recovering 2024-09-18T08:10:34.645 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:10:34 smithi007 ceph-mon[39249]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-18T08:10:34.645 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:10:34 smithi007 ceph-mon[39249]: from='client.? 172.21.15.7:0/187319810' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-18T08:10:34.645 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:10:34 smithi007 ceph-mon[39249]: osdmap e224: 8 total, 8 up, 8 in 2024-09-18T08:10:34.697 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:10:34.698 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:10:34.698 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2' 2024-09-18T08:10:34.799 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0 2024-09-18T08:10:34.836 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:10:34.837 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:10:34.837 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0' 2024-09-18T08:10:34.906 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:10:34 smithi050 ceph-mon[32371]: pgmap v371: 158 pgs: 158 active+clean; 2.0 GiB data, 8.6 GiB used, 707 GiB / 715 GiB avail; 943 B/s rd, 0 B/s wr, 1.21k op/s; 20 B/s, 0 objects/s recovering 2024-09-18T08:10:34.906 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:10:34 smithi050 ceph-mon[32371]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-18T08:10:34.906 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:10:34 smithi050 ceph-mon[32371]: from='client.? 172.21.15.7:0/187319810' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-18T08:10:34.906 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:10:34 smithi050 ceph-mon[32371]: osdmap e224: 8 total, 8 up, 8 in 2024-09-18T08:10:34.914 INFO:teuthology.orchestra.run.smithi007.stdout:{ 2024-09-18T08:10:34.914 INFO:teuthology.orchestra.run.smithi007.stdout: "success": "filestore_inject_stall = '' (not observed, change may require restart) " 2024-09-18T08:10:34.914 INFO:teuthology.orchestra.run.smithi007.stdout:} 2024-09-18T08:10:34.937 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4 2024-09-18T08:10:34.964 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:10:34.964 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:10:34.965 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4' 2024-09-18T08:10:35.065 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1 2024-09-18T08:10:35.106 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:10:35.108 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:10:35.108 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1' 2024-09-18T08:10:35.179 INFO:teuthology.orchestra.run.smithi007.stdout:osd.0: {} 2024-09-18T08:10:35.179 INFO:teuthology.orchestra.run.smithi007.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-18T08:10:35.209 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7 2024-09-18T08:10:35.217 INFO:teuthology.orchestra.run.smithi007.stdout:osd.1: {} 2024-09-18T08:10:35.217 INFO:teuthology.orchestra.run.smithi007.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-18T08:10:35.239 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:10:35.239 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:10:35.240 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7' 2024-09-18T08:10:35.251 INFO:teuthology.orchestra.run.smithi007.stdout:osd.2: {} 2024-09-18T08:10:35.252 INFO:teuthology.orchestra.run.smithi007.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-18T08:10:35.289 INFO:teuthology.orchestra.run.smithi007.stdout:osd.3: {} 2024-09-18T08:10:35.289 INFO:teuthology.orchestra.run.smithi007.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-18T08:10:35.322 INFO:teuthology.orchestra.run.smithi007.stdout:osd.4: {} 2024-09-18T08:10:35.322 INFO:teuthology.orchestra.run.smithi007.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-18T08:10:35.341 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2 2024-09-18T08:10:35.356 INFO:teuthology.orchestra.run.smithi007.stdout:osd.5: {} 2024-09-18T08:10:35.356 INFO:teuthology.orchestra.run.smithi007.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-18T08:10:35.386 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:10:35.387 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:10:35.387 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2' 2024-09-18T08:10:35.395 INFO:teuthology.orchestra.run.smithi007.stdout:osd.6: {} 2024-09-18T08:10:35.395 INFO:teuthology.orchestra.run.smithi007.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-18T08:10:35.404 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/779349ca-7593-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T08:10:35.436 INFO:teuthology.orchestra.run.smithi007.stdout:osd.7: {} 2024-09-18T08:10:35.436 INFO:teuthology.orchestra.run.smithi007.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-18T08:10:35.488 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6 2024-09-18T08:10:35.520 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:10:35.521 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:10:35.521 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6' 2024-09-18T08:10:35.622 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7 2024-09-18T08:10:35.655 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:10:35.656 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:10:35.656 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7' 2024-09-18T08:10:35.730 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:10:35 smithi007 ceph-mon[29459]: from='client.? 172.21.15.7:0/1182049375' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-18T08:10:35.731 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:10:35 smithi007 ceph-mon[39249]: from='client.? 172.21.15.7:0/1182049375' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-18T08:10:35.757 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3 2024-09-18T08:10:35.792 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:10:35.793 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:10:35.793 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3' 2024-09-18T08:10:35.894 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3 2024-09-18T08:10:35.906 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:10:35 smithi050 ceph-mon[32371]: from='client.? 172.21.15.7:0/1182049375' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-18T08:10:35.931 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:10:35.932 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:10:35.933 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3' 2024-09-18T08:10:36.033 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6 2024-09-18T08:10:36.062 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:10:36.063 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:10:36.063 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6' 2024-09-18T08:10:36.164 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6 2024-09-18T08:10:36.192 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:10:36.193 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:10:36.194 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6' 2024-09-18T08:10:36.294 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3 2024-09-18T08:10:36.348 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:10:36.348 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:10:36.349 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3' 2024-09-18T08:10:36.411 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid 779349ca-7593-11ef-bceb-c7b262605968 -- ceph osd unset noscrub 2024-09-18T08:10:36.450 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3 2024-09-18T08:10:36.505 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:10:36.506 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:10:36.506 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3' 2024-09-18T08:10:36.607 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6 2024-09-18T08:10:36.636 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:10:36.637 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:10:36.638 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6' 2024-09-18T08:10:36.638 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:10:36 smithi007 ceph-mon[39249]: pgmap v373: 158 pgs: 158 active+clean; 1.7 GiB data, 7.8 GiB used, 707 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.24k op/s 2024-09-18T08:10:36.639 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:10:36 smithi007 ceph-mon[29459]: pgmap v373: 158 pgs: 158 active+clean; 1.7 GiB data, 7.8 GiB used, 707 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.24k op/s 2024-09-18T08:10:36.738 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0 2024-09-18T08:10:36.790 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:10:36.791 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:10:36.792 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0' 2024-09-18T08:10:36.892 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5 2024-09-18T08:10:36.905 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:10:36 smithi050 ceph-mon[32371]: pgmap v373: 158 pgs: 158 active+clean; 1.7 GiB data, 7.8 GiB used, 707 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.24k op/s 2024-09-18T08:10:36.925 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:10:36.926 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:10:36.926 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5' 2024-09-18T08:10:36.981 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Sep 18 08:10:36 smithi007 ceph-779349ca-7593-11ef-bceb-c7b262605968-mgr-y[29688]: ::ffff:172.21.15.50 - - [18/Sep/2024:08:10:36] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-09-18T08:10:37.026 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5 2024-09-18T08:10:37.065 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:10:37.065 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:10:37.066 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5' 2024-09-18T08:10:37.166 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4 2024-09-18T08:10:37.201 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:10:37.202 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:10:37.202 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4' 2024-09-18T08:10:37.303 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5 2024-09-18T08:10:37.334 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:10:37.334 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:10:37.335 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5' 2024-09-18T08:10:37.436 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3 2024-09-18T08:10:37.440 INFO:teuthology.orchestra.run.smithi007.stdout:ERROR: (22) Invalid argument 2024-09-18T08:10:37.440 INFO:teuthology.orchestra.run.smithi007.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-18T08:10:37.472 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:10:37.472 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:10:37.473 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3' 2024-09-18T08:10:37.573 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7 2024-09-18T08:10:37.603 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:10:37.604 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:10:37.605 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7' 2024-09-18T08:10:37.705 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3 2024-09-18T08:10:37.743 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:10:37.744 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:10:37.744 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3' 2024-09-18T08:10:37.845 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1 2024-09-18T08:10:37.887 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:10:37.888 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:10:37.888 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1' 2024-09-18T08:10:37.989 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7 2024-09-18T08:10:38.022 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:10:38.023 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:10:38.023 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7' 2024-09-18T08:10:38.124 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7 2024-09-18T08:10:38.155 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:10:38.156 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:10:38.156 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7' 2024-09-18T08:10:38.257 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1 2024-09-18T08:10:38.297 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:10:38.298 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:10:38.298 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1' 2024-09-18T08:10:38.399 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1 2024-09-18T08:10:38.435 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:10:38.435 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:10:38.436 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1' 2024-09-18T08:10:38.536 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7 2024-09-18T08:10:38.564 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:10:38.565 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:10:38.565 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7' 2024-09-18T08:10:38.666 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3 2024-09-18T08:10:38.702 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:10:38.702 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:10:38.703 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3' 2024-09-18T08:10:38.730 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:10:38 smithi007 ceph-mon[29459]: pgmap v374: 158 pgs: 158 active+clean; 1.7 GiB data, 7.8 GiB used, 707 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.24k op/s 2024-09-18T08:10:38.730 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:10:38 smithi007 ceph-mon[29459]: purged_snaps scrub starts 2024-09-18T08:10:38.731 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:10:38 smithi007 ceph-mon[29459]: purged_snaps scrub ok 2024-09-18T08:10:38.731 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:10:38 smithi007 ceph-mon[39249]: pgmap v374: 158 pgs: 158 active+clean; 1.7 GiB data, 7.8 GiB used, 707 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.24k op/s 2024-09-18T08:10:38.731 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:10:38 smithi007 ceph-mon[39249]: purged_snaps scrub starts 2024-09-18T08:10:38.731 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:10:38 smithi007 ceph-mon[39249]: purged_snaps scrub ok 2024-09-18T08:10:38.804 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6 2024-09-18T08:10:38.826 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:10:38.827 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:10:38.827 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6' 2024-09-18T08:10:38.906 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:10:38 smithi050 ceph-mon[32371]: pgmap v374: 158 pgs: 158 active+clean; 1.7 GiB data, 7.8 GiB used, 707 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.24k op/s 2024-09-18T08:10:38.906 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:10:38 smithi050 ceph-mon[32371]: purged_snaps scrub starts 2024-09-18T08:10:38.906 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:10:38 smithi050 ceph-mon[32371]: purged_snaps scrub ok 2024-09-18T08:10:38.928 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0 2024-09-18T08:10:38.964 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:10:38.965 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:10:38.965 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0' 2024-09-18T08:10:39.066 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1 2024-09-18T08:10:39.105 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:10:39.106 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:10:39.106 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1' 2024-09-18T08:10:39.207 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5 2024-09-18T08:10:39.240 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:10:39.240 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:10:39.241 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5' 2024-09-18T08:10:39.341 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5 2024-09-18T08:10:39.375 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:10:39.376 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:10:39.376 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5' 2024-09-18T08:10:39.406 INFO:journalctl@ceph.iscsi.iscsi.a.smithi050.stdout:Sep 18 08:10:39 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-iscsi-iscsi-a[57782]: debug there is no tcmu-runner data available 2024-09-18T08:10:39.477 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1 2024-09-18T08:10:39.521 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:10:39.521 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:10:39.521 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1' 2024-09-18T08:10:39.550 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid 779349ca-7593-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-18T08:10:39.622 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7 2024-09-18T08:10:39.650 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:10:39.650 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:10:39.651 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7' 2024-09-18T08:10:39.730 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:10:39 smithi007 ceph-mon[29459]: osdmap e225: 8 total, 8 up, 8 in 2024-09-18T08:10:39.730 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:10:39 smithi007 ceph-mon[39249]: osdmap e225: 8 total, 8 up, 8 in 2024-09-18T08:10:39.752 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6 2024-09-18T08:10:39.782 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:10:39.782 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:10:39.783 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6' 2024-09-18T08:10:39.884 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3 2024-09-18T08:10:39.905 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:10:39 smithi050 ceph-mon[32371]: osdmap e225: 8 total, 8 up, 8 in 2024-09-18T08:10:39.924 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:10:39.925 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:10:39.925 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3' 2024-09-18T08:10:40.026 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2 2024-09-18T08:10:40.050 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:10:40.050 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:10:40.051 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2' 2024-09-18T08:10:40.152 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0 2024-09-18T08:10:40.188 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:10:40.189 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:10:40.189 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0' 2024-09-18T08:10:40.290 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3 2024-09-18T08:10:40.296 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid 779349ca-7593-11ef-bceb-c7b262605968 -- ceph daemon osd.2 dump_ops_in_flight 2024-09-18T08:10:40.313 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/779349ca-7593-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T08:10:40.353 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:10:40.353 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:10:40.354 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3' 2024-09-18T08:10:40.454 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4 2024-09-18T08:10:40.481 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:10:40.482 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:10:40.482 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4' 2024-09-18T08:10:40.583 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3 2024-09-18T08:10:40.627 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:10:40.629 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:10:40.629 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3' 2024-09-18T08:10:40.730 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5 2024-09-18T08:10:40.763 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:10:40.764 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:10:40.764 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5' 2024-09-18T08:10:40.827 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:10:40 smithi007 ceph-mon[29459]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-18T08:10:40.827 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:10:40 smithi007 ceph-mon[29459]: pgmap v376: 158 pgs: 158 active+clean; 1.3 GiB data, 6.6 GiB used, 709 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.56k op/s 2024-09-18T08:10:40.827 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:10:40 smithi007 ceph-mon[39249]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-18T08:10:40.827 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:10:40 smithi007 ceph-mon[39249]: pgmap v376: 158 pgs: 158 active+clean; 1.3 GiB data, 6.6 GiB used, 709 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.56k op/s 2024-09-18T08:10:40.865 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0 2024-09-18T08:10:40.905 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:10:40 smithi050 ceph-mon[32371]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-18T08:10:40.905 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:10:40 smithi050 ceph-mon[32371]: pgmap v376: 158 pgs: 158 active+clean; 1.3 GiB data, 6.6 GiB used, 709 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.56k op/s 2024-09-18T08:10:40.909 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:10:40.912 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:10:40.912 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0' 2024-09-18T08:10:40.953 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/779349ca-7593-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T08:10:41.013 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0 2024-09-18T08:10:41.065 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:10:41.066 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:10:41.066 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0' 2024-09-18T08:10:41.167 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4 2024-09-18T08:10:41.206 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:10:41.207 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:10:41.207 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4' 2024-09-18T08:10:41.277 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/779349ca-7593-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T08:10:41.308 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7 2024-09-18T08:10:41.332 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 5, 7, 6, 4] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-09-18T08:10:41.332 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-18T08:10:41.333 INFO:tasks.thrashosds.thrasher:Shrinking pool unique_pool_1 2024-09-18T08:10:41.333 INFO:tasks.cephadm.ceph_manager.ceph:contract_pool unique_pool_1 by 10 min 8 2024-09-18T08:10:41.333 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid 779349ca-7593-11ef-bceb-c7b262605968 -- ceph pg dump --format=json 2024-09-18T08:10:41.334 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:10:41.334 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:10:41.334 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7' 2024-09-18T08:10:41.435 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5 2024-09-18T08:10:41.468 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:10:41.469 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:10:41.470 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5' 2024-09-18T08:10:41.492 INFO:teuthology.orchestra.run.smithi007.stderr:noscrub is unset 2024-09-18T08:10:41.570 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3 2024-09-18T08:10:41.613 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:10:41.614 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:10:41.615 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3' 2024-09-18T08:10:41.644 INFO:teuthology.orchestra.run.smithi007.stdout:osd.0: {} 2024-09-18T08:10:41.645 INFO:teuthology.orchestra.run.smithi007.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-18T08:10:41.645 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:10:41 smithi007 ceph-mon[29459]: from='client.? 172.21.15.7:0/2909632010' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-18T08:10:41.645 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:10:41 smithi007 ceph-mon[29459]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-18T08:10:41.646 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:10:41 smithi007 ceph-mon[39249]: from='client.? 172.21.15.7:0/2909632010' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-18T08:10:41.646 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:10:41 smithi007 ceph-mon[39249]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-18T08:10:41.685 INFO:teuthology.orchestra.run.smithi007.stdout:osd.1: {} 2024-09-18T08:10:41.685 INFO:teuthology.orchestra.run.smithi007.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-18T08:10:41.716 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0 2024-09-18T08:10:41.720 INFO:teuthology.orchestra.run.smithi007.stdout:osd.2: {} 2024-09-18T08:10:41.721 INFO:teuthology.orchestra.run.smithi007.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-18T08:10:41.753 INFO:teuthology.orchestra.run.smithi007.stdout:osd.3: {} 2024-09-18T08:10:41.753 INFO:teuthology.orchestra.run.smithi007.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-18T08:10:41.754 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:10:41.756 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:10:41.756 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0' 2024-09-18T08:10:41.789 INFO:teuthology.orchestra.run.smithi007.stdout:osd.4: {} 2024-09-18T08:10:41.789 INFO:teuthology.orchestra.run.smithi007.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-18T08:10:41.825 INFO:teuthology.orchestra.run.smithi007.stdout:osd.5: {} 2024-09-18T08:10:41.826 INFO:teuthology.orchestra.run.smithi007.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-18T08:10:41.857 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4 2024-09-18T08:10:41.861 INFO:teuthology.orchestra.run.smithi007.stdout:osd.6: {} 2024-09-18T08:10:41.861 INFO:teuthology.orchestra.run.smithi007.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-18T08:10:41.894 INFO:teuthology.orchestra.run.smithi007.stdout:osd.7: {} 2024-09-18T08:10:41.894 INFO:teuthology.orchestra.run.smithi007.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-18T08:10:41.896 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:10:41.897 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:10:41.897 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4' 2024-09-18T08:10:41.905 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:10:41 smithi050 ceph-mon[32371]: from='client.? 172.21.15.7:0/2909632010' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-18T08:10:41.906 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:10:41 smithi050 ceph-mon[32371]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-18T08:10:41.998 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0 2024-09-18T08:10:42.038 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:10:42.038 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:10:42.039 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0' 2024-09-18T08:10:42.140 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3 2024-09-18T08:10:42.162 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:10:42.162 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:10:42.162 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3' 2024-09-18T08:10:42.263 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3 2024-09-18T08:10:42.308 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:10:42.308 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:10:42.309 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3' 2024-09-18T08:10:42.409 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7 2024-09-18T08:10:42.443 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:10:42.443 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:10:42.443 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7' 2024-09-18T08:10:42.544 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1 2024-09-18T08:10:42.580 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:10:42.581 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:10:42.581 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1' 2024-09-18T08:10:42.682 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5 2024-09-18T08:10:42.716 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:10:42.717 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:10:42.718 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5' 2024-09-18T08:10:42.806 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:10:42 smithi050 ceph-mon[32371]: pgmap v377: 158 pgs: 158 active+clean; 1.3 GiB data, 6.6 GiB used, 709 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.46k op/s 2024-09-18T08:10:42.806 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:10:42 smithi050 ceph-mon[32371]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-18T08:10:42.806 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:10:42 smithi050 ceph-mon[32371]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-18T08:10:42.806 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:10:42 smithi050 ceph-mon[32371]: osdmap e226: 8 total, 8 up, 8 in 2024-09-18T08:10:42.807 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:10:42 smithi050 ceph-mon[32371]: 1.7 scrub starts 2024-09-18T08:10:42.807 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:10:42 smithi050 ceph-mon[32371]: 1.7 scrub ok 2024-09-18T08:10:42.807 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:10:42 smithi050 ceph-mon[32371]: 4.f scrub starts 2024-09-18T08:10:42.807 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:10:42 smithi050 ceph-mon[32371]: 4.f scrub ok 2024-09-18T08:10:42.807 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:10:42 smithi050 ceph-mon[32371]: from='client.? 172.21.15.7:0/1371907270' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-18T08:10:42.807 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:10:42 smithi050 ceph-mon[32371]: 5.b scrub starts 2024-09-18T08:10:42.807 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:10:42 smithi050 ceph-mon[32371]: 5.b scrub ok 2024-09-18T08:10:42.807 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:10:42 smithi050 ceph-mon[32371]: 3.11 scrub starts 2024-09-18T08:10:42.807 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:10:42 smithi050 ceph-mon[32371]: 3.11 scrub ok 2024-09-18T08:10:42.818 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0 2024-09-18T08:10:42.859 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:10:42.859 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:10:42.860 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0' 2024-09-18T08:10:42.961 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4 2024-09-18T08:10:42.981 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:10:42 smithi007 ceph-mon[29459]: pgmap v377: 158 pgs: 158 active+clean; 1.3 GiB data, 6.6 GiB used, 709 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.46k op/s 2024-09-18T08:10:42.981 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:10:42 smithi007 ceph-mon[29459]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-18T08:10:42.981 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:10:42 smithi007 ceph-mon[29459]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-18T08:10:42.981 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:10:42 smithi007 ceph-mon[29459]: osdmap e226: 8 total, 8 up, 8 in 2024-09-18T08:10:42.981 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:10:42 smithi007 ceph-mon[29459]: 1.7 scrub starts 2024-09-18T08:10:42.981 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:10:42 smithi007 ceph-mon[29459]: 1.7 scrub ok 2024-09-18T08:10:42.982 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:10:42 smithi007 ceph-mon[29459]: 4.f scrub starts 2024-09-18T08:10:42.982 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:10:42 smithi007 ceph-mon[29459]: 4.f scrub ok 2024-09-18T08:10:42.982 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:10:42 smithi007 ceph-mon[29459]: from='client.? 172.21.15.7:0/1371907270' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-18T08:10:42.982 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:10:42 smithi007 ceph-mon[29459]: 5.b scrub starts 2024-09-18T08:10:42.982 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:10:42 smithi007 ceph-mon[29459]: 5.b scrub ok 2024-09-18T08:10:42.983 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:10:42 smithi007 ceph-mon[29459]: 3.11 scrub starts 2024-09-18T08:10:42.983 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:10:42 smithi007 ceph-mon[29459]: 3.11 scrub ok 2024-09-18T08:10:42.983 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:10:42 smithi007 ceph-mon[39249]: pgmap v377: 158 pgs: 158 active+clean; 1.3 GiB data, 6.6 GiB used, 709 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.46k op/s 2024-09-18T08:10:42.983 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:10:42 smithi007 ceph-mon[39249]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-18T08:10:42.983 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:10:42 smithi007 ceph-mon[39249]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-18T08:10:42.984 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:10:42 smithi007 ceph-mon[39249]: osdmap e226: 8 total, 8 up, 8 in 2024-09-18T08:10:42.984 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:10:42 smithi007 ceph-mon[39249]: 1.7 scrub starts 2024-09-18T08:10:42.984 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:10:42 smithi007 ceph-mon[39249]: 1.7 scrub ok 2024-09-18T08:10:42.984 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:10:42 smithi007 ceph-mon[39249]: 4.f scrub starts 2024-09-18T08:10:42.984 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:10:42 smithi007 ceph-mon[39249]: 4.f scrub ok 2024-09-18T08:10:42.984 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:10:42 smithi007 ceph-mon[39249]: from='client.? 172.21.15.7:0/1371907270' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-18T08:10:42.984 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:10:42 smithi007 ceph-mon[39249]: 5.b scrub starts 2024-09-18T08:10:42.985 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:10:42 smithi007 ceph-mon[39249]: 5.b scrub ok 2024-09-18T08:10:42.985 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:10:42 smithi007 ceph-mon[39249]: 3.11 scrub starts 2024-09-18T08:10:42.985 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:10:42 smithi007 ceph-mon[39249]: 3.11 scrub ok 2024-09-18T08:10:42.999 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:10:43.000 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:10:43.000 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4' 2024-09-18T08:10:43.101 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7 2024-09-18T08:10:43.125 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:10:43.126 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:10:43.126 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7' 2024-09-18T08:10:43.227 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2 2024-09-18T08:10:43.273 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:10:43.273 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:10:43.274 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2' 2024-09-18T08:10:43.374 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1 2024-09-18T08:10:43.411 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:10:43.412 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:10:43.412 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1' 2024-09-18T08:10:43.513 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0 2024-09-18T08:10:43.562 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:10:43.563 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:10:43.563 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0' 2024-09-18T08:10:43.664 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2 2024-09-18T08:10:43.722 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:10:43.722 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:10:43.723 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2' 2024-09-18T08:10:43.730 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:10:43 smithi007 ceph-mon[29459]: 6.14 scrub starts 2024-09-18T08:10:43.730 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:10:43 smithi007 ceph-mon[29459]: 6.14 scrub ok 2024-09-18T08:10:43.731 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:10:43 smithi007 ceph-mon[39249]: 6.14 scrub starts 2024-09-18T08:10:43.731 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:10:43 smithi007 ceph-mon[39249]: 6.14 scrub ok 2024-09-18T08:10:43.753 INFO:teuthology.orchestra.run.smithi007.stdout:{ 2024-09-18T08:10:43.753 INFO:teuthology.orchestra.run.smithi007.stdout: "ops": [ 2024-09-18T08:10:43.754 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:10:43.754 INFO:teuthology.orchestra.run.smithi007.stdout: "description": "osd_repop(client.15216.0:185954 8.e e226/207 8:7d7e7d4e:::benchmark_data_smithi007_101583_object85202:head v 226'11739, mlcod=226'11737)", 2024-09-18T08:10:43.754 INFO:teuthology.orchestra.run.smithi007.stdout: "initiated_at": "2024-09-18T08:10:43.747623+0000", 2024-09-18T08:10:43.754 INFO:teuthology.orchestra.run.smithi007.stdout: "age": 0.0034421399999999998, 2024-09-18T08:10:43.754 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.0034685620000000001, 2024-09-18T08:10:43.754 INFO:teuthology.orchestra.run.smithi007.stdout: "continuous": false, 2024-09-18T08:10:43.754 INFO:teuthology.orchestra.run.smithi007.stdout: "type_data": { 2024-09-18T08:10:43.754 INFO:teuthology.orchestra.run.smithi007.stdout: "flag_point": "started", 2024-09-18T08:10:43.754 INFO:teuthology.orchestra.run.smithi007.stdout: "events": [ 2024-09-18T08:10:43.754 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:10:43.754 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "initiated", 2024-09-18T08:10:43.755 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:10:43.747623+0000", 2024-09-18T08:10:43.755 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-09-18T08:10:43.755 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:10:43.755 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:10:43.755 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "header_read", 2024-09-18T08:10:43.755 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:10:43.747623+0000", 2024-09-18T08:10:43.755 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-09-18T08:10:43.755 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:10:43.756 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:10:43.756 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "throttled", 2024-09-18T08:10:43.756 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:10:43.747624+0000", 2024-09-18T08:10:43.756 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 8.8100000000000001e-07 2024-09-18T08:10:43.756 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:10:43.756 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:10:43.756 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "all_read", 2024-09-18T08:10:43.756 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:10:43.747630+0000", 2024-09-18T08:10:43.756 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 5.8549999999999998e-06 2024-09-18T08:10:43.756 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:10:43.757 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:10:43.757 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "dispatched", 2024-09-18T08:10:43.757 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:10:43.747640+0000", 2024-09-18T08:10:43.757 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.0166000000000001e-05 2024-09-18T08:10:43.757 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:10:43.757 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:10:43.757 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "queued_for_pg", 2024-09-18T08:10:43.757 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:10:43.747653+0000", 2024-09-18T08:10:43.757 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.2645e-05 2024-09-18T08:10:43.757 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:10:43.757 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:10:43.757 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-09-18T08:10:43.757 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:10:43.747885+0000", 2024-09-18T08:10:43.758 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.00023239800000000001 2024-09-18T08:10:43.758 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:10:43.758 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:10:43.758 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "started", 2024-09-18T08:10:43.758 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:10:43.747935+0000", 2024-09-18T08:10:43.758 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 4.9336999999999999e-05 2024-09-18T08:10:43.758 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-09-18T08:10:43.758 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-09-18T08:10:43.758 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-09-18T08:10:43.758 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:10:43.758 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:10:43.758 INFO:teuthology.orchestra.run.smithi007.stdout: "description": "osd_repop(client.15216.0:185952 8.f e226/211 8:f4272e0d:::benchmark_data_smithi007_101583_object85200:head v 226'11905, mlcod=226'11902)", 2024-09-18T08:10:43.758 INFO:teuthology.orchestra.run.smithi007.stdout: "initiated_at": "2024-09-18T08:10:43.749146+0000", 2024-09-18T08:10:43.759 INFO:teuthology.orchestra.run.smithi007.stdout: "age": 0.0019196409999999999, 2024-09-18T08:10:43.759 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.0020030529999999999, 2024-09-18T08:10:43.759 INFO:teuthology.orchestra.run.smithi007.stdout: "continuous": false, 2024-09-18T08:10:43.759 INFO:teuthology.orchestra.run.smithi007.stdout: "type_data": { 2024-09-18T08:10:43.759 INFO:teuthology.orchestra.run.smithi007.stdout: "flag_point": "started", 2024-09-18T08:10:43.760 INFO:teuthology.orchestra.run.smithi007.stdout: "events": [ 2024-09-18T08:10:43.760 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:10:43.760 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "initiated", 2024-09-18T08:10:43.760 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:10:43.749146+0000", 2024-09-18T08:10:43.760 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-09-18T08:10:43.760 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:10:43.760 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:10:43.760 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "header_read", 2024-09-18T08:10:43.760 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:10:43.749146+0000", 2024-09-18T08:10:43.760 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-09-18T08:10:43.760 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:10:43.760 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:10:43.761 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "throttled", 2024-09-18T08:10:43.761 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:10:43.749147+0000", 2024-09-18T08:10:43.761 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 7.6899999999999996e-07 2024-09-18T08:10:43.761 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:10:43.761 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:10:43.761 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "all_read", 2024-09-18T08:10:43.761 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:10:43.749153+0000", 2024-09-18T08:10:43.761 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 5.8880000000000002e-06 2024-09-18T08:10:43.761 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:10:43.761 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:10:43.761 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "dispatched", 2024-09-18T08:10:43.761 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:10:43.749172+0000", 2024-09-18T08:10:43.762 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.9843999999999998e-05 2024-09-18T08:10:43.762 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:10:43.762 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:10:43.762 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "queued_for_pg", 2024-09-18T08:10:43.762 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:10:43.749182+0000", 2024-09-18T08:10:43.762 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 9.2410000000000001e-06 2024-09-18T08:10:43.762 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:10:43.762 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:10:43.762 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-09-18T08:10:43.762 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:10:43.749269+0000", 2024-09-18T08:10:43.762 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 8.7763000000000007e-05 2024-09-18T08:10:43.762 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:10:43.762 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:10:43.762 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "started", 2024-09-18T08:10:43.762 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:10:43.749297+0000", 2024-09-18T08:10:43.763 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 2.7242000000000001e-05 2024-09-18T08:10:43.763 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-09-18T08:10:43.763 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-09-18T08:10:43.763 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-09-18T08:10:43.763 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-09-18T08:10:43.763 INFO:teuthology.orchestra.run.smithi007.stdout: ], 2024-09-18T08:10:43.763 INFO:teuthology.orchestra.run.smithi007.stdout: "num_ops": 2 2024-09-18T08:10:43.763 INFO:teuthology.orchestra.run.smithi007.stdout:} 2024-09-18T08:10:43.824 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2 2024-09-18T08:10:43.863 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:10:43.864 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:10:43.864 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2' 2024-09-18T08:10:43.906 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:10:43 smithi050 ceph-mon[32371]: 6.14 scrub starts 2024-09-18T08:10:43.906 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:10:43 smithi050 ceph-mon[32371]: 6.14 scrub ok 2024-09-18T08:10:43.965 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3 2024-09-18T08:10:44.007 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:10:44.008 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:10:44.008 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3' 2024-09-18T08:10:44.109 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2 2024-09-18T08:10:44.144 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:10:44.144 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:10:44.144 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2' 2024-09-18T08:10:44.246 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2 2024-09-18T08:10:44.285 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:10:44.286 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:10:44.288 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2' 2024-09-18T08:10:44.388 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5 2024-09-18T08:10:44.426 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:10:44.428 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:10:44.428 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5' 2024-09-18T08:10:44.529 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2 2024-09-18T08:10:44.564 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:10:44.565 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:10:44.565 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2' 2024-09-18T08:10:44.666 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4 2024-09-18T08:10:44.707 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:10:44.708 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:10:44.708 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4' 2024-09-18T08:10:44.809 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5 2024-09-18T08:10:44.834 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:10:44.834 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:10:44.835 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5' 2024-09-18T08:10:44.906 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:10:44 smithi050 ceph-mon[32371]: pgmap v379: 158 pgs: 158 active+clean; 1.3 GiB data, 6.3 GiB used, 709 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 877 op/s 2024-09-18T08:10:44.906 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:10:44 smithi050 ceph-mon[32371]: 1.4 scrub starts 2024-09-18T08:10:44.906 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:10:44 smithi050 ceph-mon[32371]: 1.4 scrub ok 2024-09-18T08:10:44.906 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:10:44 smithi050 ceph-mon[32371]: 5.1e scrub starts 2024-09-18T08:10:44.906 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:10:44 smithi050 ceph-mon[32371]: 5.1e scrub ok 2024-09-18T08:10:44.936 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6 2024-09-18T08:10:44.967 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:10:44.967 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:10:44.968 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6' 2024-09-18T08:10:44.980 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:10:44 smithi007 ceph-mon[29459]: pgmap v379: 158 pgs: 158 active+clean; 1.3 GiB data, 6.3 GiB used, 709 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 877 op/s 2024-09-18T08:10:44.980 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:10:44 smithi007 ceph-mon[29459]: 1.4 scrub starts 2024-09-18T08:10:44.981 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:10:44 smithi007 ceph-mon[29459]: 1.4 scrub ok 2024-09-18T08:10:44.981 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:10:44 smithi007 ceph-mon[29459]: 5.1e scrub starts 2024-09-18T08:10:44.981 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:10:44 smithi007 ceph-mon[29459]: 5.1e scrub ok 2024-09-18T08:10:44.981 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:10:44 smithi007 ceph-mon[39249]: pgmap v379: 158 pgs: 158 active+clean; 1.3 GiB data, 6.3 GiB used, 709 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 877 op/s 2024-09-18T08:10:44.981 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:10:44 smithi007 ceph-mon[39249]: 1.4 scrub starts 2024-09-18T08:10:44.981 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:10:44 smithi007 ceph-mon[39249]: 1.4 scrub ok 2024-09-18T08:10:44.981 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:10:44 smithi007 ceph-mon[39249]: 5.1e scrub starts 2024-09-18T08:10:44.981 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:10:44 smithi007 ceph-mon[39249]: 5.1e scrub ok 2024-09-18T08:10:45.069 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5 2024-09-18T08:10:45.104 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:10:45.104 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:10:45.104 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5' 2024-09-18T08:10:45.205 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0 2024-09-18T08:10:45.245 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:10:45.246 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:10:45.247 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0' 2024-09-18T08:10:45.348 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1 2024-09-18T08:10:45.394 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:10:45.396 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:10:45.396 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1' 2024-09-18T08:10:45.414 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/779349ca-7593-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T08:10:45.431 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid 779349ca-7593-11ef-bceb-c7b262605968 -- ceph daemon osd.2 dump_blocked_ops 2024-09-18T08:10:45.497 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0 2024-09-18T08:10:45.535 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid 779349ca-7593-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-18T08:10:45.565 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:10:45.566 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:10:45.566 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0' 2024-09-18T08:10:45.667 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1 2024-09-18T08:10:45.725 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:10:45.726 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:10:45.726 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1' 2024-09-18T08:10:45.730 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:10:45 smithi007 ceph-mon[39249]: 2.0 scrub starts 2024-09-18T08:10:45.730 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:10:45 smithi007 ceph-mon[39249]: 2.0 scrub ok 2024-09-18T08:10:45.730 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:10:45 smithi007 ceph-mon[29459]: 2.0 scrub starts 2024-09-18T08:10:45.731 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:10:45 smithi007 ceph-mon[29459]: 2.0 scrub ok 2024-09-18T08:10:45.827 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6 2024-09-18T08:10:45.854 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:10:45.855 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:10:45.856 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6' 2024-09-18T08:10:45.906 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:10:45 smithi050 ceph-mon[32371]: 2.0 scrub starts 2024-09-18T08:10:45.906 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:10:45 smithi050 ceph-mon[32371]: 2.0 scrub ok 2024-09-18T08:10:45.957 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5 2024-09-18T08:10:45.994 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:10:45.994 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:10:45.995 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5' 2024-09-18T08:10:46.095 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3 2024-09-18T08:10:46.131 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:10:46.131 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:10:46.132 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3' 2024-09-18T08:10:46.224 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid 779349ca-7593-11ef-bceb-c7b262605968 -- ceph osd unset nodeep-scrub 2024-09-18T08:10:46.233 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0 2024-09-18T08:10:46.326 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:10:46.326 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:10:46.326 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0' 2024-09-18T08:10:46.427 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5 2024-09-18T08:10:46.456 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:10:46.457 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:10:46.457 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5' 2024-09-18T08:10:46.558 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2 2024-09-18T08:10:46.607 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/779349ca-7593-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T08:10:46.635 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:10:46.636 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:10:46.636 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2' 2024-09-18T08:10:46.659 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/779349ca-7593-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T08:10:46.737 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2 2024-09-18T08:10:46.775 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:10:46 smithi007 ceph-mon[29459]: pgmap v380: 158 pgs: 158 active+clean; 980 MiB data, 5.6 GiB used, 710 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.48k op/s 2024-09-18T08:10:46.775 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:10:46 smithi007 ceph-mon[29459]: 5.1c scrub starts 2024-09-18T08:10:46.775 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:10:46 smithi007 ceph-mon[29459]: 5.1c scrub ok 2024-09-18T08:10:46.775 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Sep 18 08:10:46 smithi007 ceph-779349ca-7593-11ef-bceb-c7b262605968-mgr-y[29688]: ::ffff:172.21.15.50 - - [18/Sep/2024:08:10:46] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-09-18T08:10:46.775 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:10:46 smithi007 ceph-mon[39249]: pgmap v380: 158 pgs: 158 active+clean; 980 MiB data, 5.6 GiB used, 710 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.48k op/s 2024-09-18T08:10:46.775 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:10:46 smithi007 ceph-mon[39249]: 5.1c scrub starts 2024-09-18T08:10:46.775 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:10:46 smithi007 ceph-mon[39249]: 5.1c scrub ok 2024-09-18T08:10:46.810 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:10:46.811 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:10:46.811 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2' 2024-09-18T08:10:46.906 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:10:46 smithi050 ceph-mon[32371]: pgmap v380: 158 pgs: 158 active+clean; 980 MiB data, 5.6 GiB used, 710 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.48k op/s 2024-09-18T08:10:46.906 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:10:46 smithi050 ceph-mon[32371]: 5.1c scrub starts 2024-09-18T08:10:46.906 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:10:46 smithi050 ceph-mon[32371]: 5.1c scrub ok 2024-09-18T08:10:46.912 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4 2024-09-18T08:10:46.943 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:10:46.943 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:10:46.944 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4' 2024-09-18T08:10:46.967 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-09-18T08:10:46.969 INFO:teuthology.orchestra.run.smithi007.stderr:dumped all 2024-09-18T08:10:47.044 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3 2024-09-18T08:10:47.090 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:10:47.091 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:10:47.091 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3' 2024-09-18T08:10:47.192 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7 2024-09-18T08:10:47.223 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:10:47.223 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:10:47.224 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7' 2024-09-18T08:10:47.325 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0 2024-09-18T08:10:47.372 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:10:47.373 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:10:47.374 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0' 2024-09-18T08:10:47.475 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2 2024-09-18T08:10:47.532 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:10:47.533 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:10:47.533 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2' 2024-09-18T08:10:47.634 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5 2024-09-18T08:10:47.667 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:10:47.668 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:10:47.668 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5' 2024-09-18T08:10:47.668 INFO:teuthology.orchestra.run.smithi007.stdout:osd.0: {} 2024-09-18T08:10:47.669 INFO:teuthology.orchestra.run.smithi007.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-18T08:10:47.669 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:10:47 smithi007 ceph-mon[29459]: 6.1c scrub starts 2024-09-18T08:10:47.669 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:10:47 smithi007 ceph-mon[29459]: 6.1c scrub ok 2024-09-18T08:10:47.669 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:10:47 smithi007 ceph-mon[29459]: 4.4 scrub starts 2024-09-18T08:10:47.669 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:10:47 smithi007 ceph-mon[29459]: 4.4 scrub ok 2024-09-18T08:10:47.669 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:10:47 smithi007 ceph-mon[29459]: from='mgr.24470 172.21.15.7:0/2594390261' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-18T08:10:47.669 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:10:47 smithi007 ceph-mon[29459]: from='mgr.24470 172.21.15.7:0/2594390261' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-18T08:10:47.670 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:10:47 smithi007 ceph-mon[29459]: from='mgr.24470 172.21.15.7:0/2594390261' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-18T08:10:47.670 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:10:47 smithi007 ceph-mon[39249]: 6.1c scrub starts 2024-09-18T08:10:47.670 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:10:47 smithi007 ceph-mon[39249]: 6.1c scrub ok 2024-09-18T08:10:47.670 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:10:47 smithi007 ceph-mon[39249]: 4.4 scrub starts 2024-09-18T08:10:47.670 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:10:47 smithi007 ceph-mon[39249]: 4.4 scrub ok 2024-09-18T08:10:47.670 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:10:47 smithi007 ceph-mon[39249]: from='mgr.24470 172.21.15.7:0/2594390261' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-18T08:10:47.670 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:10:47 smithi007 ceph-mon[39249]: from='mgr.24470 172.21.15.7:0/2594390261' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-18T08:10:47.670 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:10:47 smithi007 ceph-mon[39249]: from='mgr.24470 172.21.15.7:0/2594390261' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-18T08:10:47.705 INFO:teuthology.orchestra.run.smithi007.stdout:{ 2024-09-18T08:10:47.705 INFO:teuthology.orchestra.run.smithi007.stdout: "ops": [], 2024-09-18T08:10:47.705 INFO:teuthology.orchestra.run.smithi007.stdout: "complaint_time": 30, 2024-09-18T08:10:47.705 INFO:teuthology.orchestra.run.smithi007.stdout: "num_blocked_ops": 0 2024-09-18T08:10:47.705 INFO:teuthology.orchestra.run.smithi007.stdout:} 2024-09-18T08:10:47.712 INFO:teuthology.orchestra.run.smithi007.stdout:osd.1: {} 2024-09-18T08:10:47.712 INFO:teuthology.orchestra.run.smithi007.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-18T08:10:47.757 INFO:teuthology.orchestra.run.smithi007.stdout:osd.2: {} 2024-09-18T08:10:47.757 INFO:teuthology.orchestra.run.smithi007.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-18T08:10:47.768 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5 2024-09-18T08:10:47.793 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:10:47.794 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:10:47.794 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5' 2024-09-18T08:10:47.799 INFO:teuthology.orchestra.run.smithi007.stdout:osd.3: {} 2024-09-18T08:10:47.799 INFO:teuthology.orchestra.run.smithi007.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-18T08:10:47.837 INFO:teuthology.orchestra.run.smithi007.stdout:osd.4: {} 2024-09-18T08:10:47.838 INFO:teuthology.orchestra.run.smithi007.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-18T08:10:47.873 INFO:teuthology.orchestra.run.smithi007.stdout:osd.5: {} 2024-09-18T08:10:47.874 INFO:teuthology.orchestra.run.smithi007.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-18T08:10:47.895 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5 2024-09-18T08:10:47.905 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:10:47 smithi050 ceph-mon[32371]: 6.1c scrub starts 2024-09-18T08:10:47.906 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:10:47 smithi050 ceph-mon[32371]: 6.1c scrub ok 2024-09-18T08:10:47.906 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:10:47 smithi050 ceph-mon[32371]: 4.4 scrub starts 2024-09-18T08:10:47.906 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:10:47 smithi050 ceph-mon[32371]: 4.4 scrub ok 2024-09-18T08:10:47.906 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:10:47 smithi050 ceph-mon[32371]: from='mgr.24470 172.21.15.7:0/2594390261' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-18T08:10:47.906 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:10:47 smithi050 ceph-mon[32371]: from='mgr.24470 172.21.15.7:0/2594390261' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-18T08:10:47.906 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:10:47 smithi050 ceph-mon[32371]: from='mgr.24470 172.21.15.7:0/2594390261' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-18T08:10:47.912 INFO:teuthology.orchestra.run.smithi007.stdout:osd.6: {} 2024-09-18T08:10:47.912 INFO:teuthology.orchestra.run.smithi007.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-18T08:10:47.924 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:10:47.925 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:10:47.925 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5' 2024-09-18T08:10:47.945 INFO:teuthology.orchestra.run.smithi007.stdout:osd.7: {} 2024-09-18T08:10:47.945 INFO:teuthology.orchestra.run.smithi007.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-18T08:10:48.026 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1 2024-09-18T08:10:48.063 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:10:48.063 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:10:48.064 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1' 2024-09-18T08:10:48.165 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1 2024-09-18T08:10:48.200 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:10:48.201 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:10:48.201 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1' 2024-09-18T08:10:48.302 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0 2024-09-18T08:10:48.338 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:10:48.339 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:10:48.339 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0' 2024-09-18T08:10:48.440 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5 2024-09-18T08:10:48.475 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:10:48.476 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:10:48.477 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5' 2024-09-18T08:10:48.502 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid 779349ca-7593-11ef-bceb-c7b262605968 -- ceph daemon osd.2 dump_historic_ops 2024-09-18T08:10:48.578 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4 2024-09-18T08:10:48.610 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:10:48.610 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:10:48.611 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4' 2024-09-18T08:10:48.711 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7 2024-09-18T08:10:48.730 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:10:48 smithi007 ceph-mon[39249]: from='client.25304 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-18T08:10:48.730 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:10:48 smithi007 ceph-mon[39249]: pgmap v381: 158 pgs: 158 active+clean; 980 MiB data, 5.6 GiB used, 710 GiB / 715 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 1.08k op/s 2024-09-18T08:10:48.731 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:10:48 smithi007 ceph-mon[39249]: from='mgr.24470 172.21.15.7:0/2594390261' entity='mgr.y' 2024-09-18T08:10:48.731 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:10:48 smithi007 ceph-mon[39249]: from='client.? 172.21.15.7:0/3871671028' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-18T08:10:48.731 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:10:48 smithi007 ceph-mon[39249]: 5.8 scrub starts 2024-09-18T08:10:48.731 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:10:48 smithi007 ceph-mon[39249]: 5.8 scrub ok 2024-09-18T08:10:48.731 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:10:48 smithi007 ceph-mon[39249]: 4.b scrub starts 2024-09-18T08:10:48.731 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:10:48 smithi007 ceph-mon[39249]: 4.b scrub ok 2024-09-18T08:10:48.731 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:10:48 smithi007 ceph-mon[39249]: from='mgr.24470 172.21.15.7:0/2594390261' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-18T08:10:48.731 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:10:48 smithi007 ceph-mon[29459]: from='client.25304 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-18T08:10:48.731 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:10:48 smithi007 ceph-mon[29459]: pgmap v381: 158 pgs: 158 active+clean; 980 MiB data, 5.6 GiB used, 710 GiB / 715 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 1.08k op/s 2024-09-18T08:10:48.731 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:10:48 smithi007 ceph-mon[29459]: from='mgr.24470 172.21.15.7:0/2594390261' entity='mgr.y' 2024-09-18T08:10:48.731 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:10:48 smithi007 ceph-mon[29459]: from='client.? 172.21.15.7:0/3871671028' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-18T08:10:48.732 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:10:48 smithi007 ceph-mon[29459]: 5.8 scrub starts 2024-09-18T08:10:48.732 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:10:48 smithi007 ceph-mon[29459]: 5.8 scrub ok 2024-09-18T08:10:48.732 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:10:48 smithi007 ceph-mon[29459]: 4.b scrub starts 2024-09-18T08:10:48.732 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:10:48 smithi007 ceph-mon[29459]: 4.b scrub ok 2024-09-18T08:10:48.732 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:10:48 smithi007 ceph-mon[29459]: from='mgr.24470 172.21.15.7:0/2594390261' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-18T08:10:48.738 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:10:48.739 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:10:48.739 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7' 2024-09-18T08:10:48.840 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6 2024-09-18T08:10:48.874 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:10:48.875 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:10:48.875 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6' 2024-09-18T08:10:48.906 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:10:48 smithi050 ceph-mon[32371]: from='client.25304 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-18T08:10:48.906 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:10:48 smithi050 ceph-mon[32371]: pgmap v381: 158 pgs: 158 active+clean; 980 MiB data, 5.6 GiB used, 710 GiB / 715 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 1.08k op/s 2024-09-18T08:10:48.906 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:10:48 smithi050 ceph-mon[32371]: from='mgr.24470 172.21.15.7:0/2594390261' entity='mgr.y' 2024-09-18T08:10:48.906 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:10:48 smithi050 ceph-mon[32371]: from='client.? 172.21.15.7:0/3871671028' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-18T08:10:48.906 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:10:48 smithi050 ceph-mon[32371]: 5.8 scrub starts 2024-09-18T08:10:48.906 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:10:48 smithi050 ceph-mon[32371]: 5.8 scrub ok 2024-09-18T08:10:48.906 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:10:48 smithi050 ceph-mon[32371]: 4.b scrub starts 2024-09-18T08:10:48.906 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:10:48 smithi050 ceph-mon[32371]: 4.b scrub ok 2024-09-18T08:10:48.906 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:10:48 smithi050 ceph-mon[32371]: from='mgr.24470 172.21.15.7:0/2594390261' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-18T08:10:48.976 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1 2024-09-18T08:10:49.018 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:10:49.018 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:10:49.018 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1' 2024-09-18T08:10:49.109 INFO:teuthology.orchestra.run.smithi007.stdout:{"pg_ready":true,"pg_map":{"version":380,"stamp":"2024-09-18T08:10:45.242754+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":1028004403,"num_objects":15877,"num_object_clones":0,"num_object_copies":47631,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15877,"num_whiteouts":0,"num_read":953,"num_read_kb":705,"num_write":186258,"num_write_kb":6447982,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24491,"num_bytes_recovered":1588862382,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":79976,"ondisk_log_size":79976,"up":474,"acting":474,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":478,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":5921524,"kb_used_data":3855912,"kb_used_omap":12,"kb_used_meta":2065203,"kb_avail":744039692,"statfs":{"total":767960285184,"available":761896644608,"internally_reserved":0,"allocated":3948453888,"data_stored":3945240553,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12750,"internal_metadata":2114768434},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,2,4,7,1],"upper_bound":32},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-774569984,"num_objects":-11819,"num_object_clones":0,"num_object_copies":-35457,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-11819,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":11819,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.002039"},"pg_stats":[{"pgid":"3.1f","version":"0'0","reported_seq":380,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-09-18T08:10:41.505819+0000","last_change":"2024-09-18T08:10:15.850596+0000","last_active":"2024-09-18T08:10:41.505819+0000","last_peered":"2024-09-18T08:10:41.505819+0000","last_clean":"2024-09-18T08:10:41.505819+0000","last_became_active":"2024-09-18T08:08:52.697499+0000","last_became_peered":"2024-09-18T08:08:52.697499+0000","last_unstale":"2024-09-18T08:10:41.505819+0000","last_undegraded":"2024-09-18T08:10:41.505819+0000","last_fullsized":"2024-09-18T08:10:41.505819+0000","mapping_epoch":198,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":199,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:10:15.850515+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:46.860092+0000","last_clean_scrub_stamp":"2024-09-18T08:10:15.850515+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:11:28.482967+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"69'9","reported_seq":119,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-09-18T08:10:41.487600+0000","last_change":"2024-09-18T08:10:19.271321+0000","last_active":"2024-09-18T08:10:41.487600+0000","last_peered":"2024-09-18T08:10:41.487600+0000","last_clean":"2024-09-18T08:10:41.487600+0000","last_became_active":"2024-09-18T08:08:18.537856+0000","last_became_peered":"2024-09-18T08:08:18.537856+0000","last_unstale":"2024-09-18T08:10:41.487600+0000","last_undegraded":"2024-09-18T08:10:41.487600+0000","last_fullsized":"2024-09-18T08:10:41.487600+0000","mapping_epoch":181,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":182,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'9","last_scrub_stamp":"2024-09-18T08:10:19.271241+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_clean_scrub_stamp":"2024-09-18T08:10:19.271241+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:11:37.987398+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":16,"num_read_kb":10,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":366,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-09-18T08:10:41.496470+0000","last_change":"2024-09-18T08:10:15.603362+0000","last_active":"2024-09-18T08:10:41.496470+0000","last_peered":"2024-09-18T08:10:41.496470+0000","last_clean":"2024-09-18T08:10:41.496470+0000","last_became_active":"2024-09-18T08:02:51.907140+0000","last_became_peered":"2024-09-18T08:02:51.907140+0000","last_unstale":"2024-09-18T08:10:41.496470+0000","last_undegraded":"2024-09-18T08:10:41.496470+0000","last_fullsized":"2024-09-18T08:10:41.496470+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:10:15.603289+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:50.886830+0000","last_clean_scrub_stamp":"2024-09-18T08:10:15.603289+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:11:27.298716+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":100,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-09-18T08:10:41.489162+0000","last_change":"2024-09-18T08:08:18.545192+0000","last_active":"2024-09-18T08:10:41.489162+0000","last_peered":"2024-09-18T08:10:41.489162+0000","last_clean":"2024-09-18T08:10:41.489162+0000","last_became_active":"2024-09-18T08:08:18.540705+0000","last_became_peered":"2024-09-18T08:08:18.540705+0000","last_unstale":"2024-09-18T08:10:41.489162+0000","last_undegraded":"2024-09-18T08:10:41.489162+0000","last_fullsized":"2024-09-18T08:10:41.489162+0000","mapping_epoch":181,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":182,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:07:08.451865+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_clean_scrub_stamp":"2024-09-18T08:07:08.451865+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":361,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-09-18T08:10:41.500790+0000","last_change":"2024-09-18T08:07:13.244647+0000","last_active":"2024-09-18T08:10:41.500790+0000","last_peered":"2024-09-18T08:10:41.500790+0000","last_clean":"2024-09-18T08:10:41.500790+0000","last_became_active":"2024-09-18T08:06:24.128511+0000","last_became_peered":"2024-09-18T08:06:24.128511+0000","last_unstale":"2024-09-18T08:10:41.500790+0000","last_undegraded":"2024-09-18T08:10:41.500790+0000","last_fullsized":"2024-09-18T08:10:41.500790+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:07:13.244555+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_clean_scrub_stamp":"2024-09-18T08:07:13.244555+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:10:55.479885+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":381,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-09-18T08:10:41.493387+0000","last_change":"2024-09-18T08:10:08.473695+0000","last_active":"2024-09-18T08:10:41.493387+0000","last_peered":"2024-09-18T08:10:41.493387+0000","last_clean":"2024-09-18T08:10:41.493387+0000","last_became_active":"2024-09-18T08:06:24.126194+0000","last_became_peered":"2024-09-18T08:06:24.126194+0000","last_unstale":"2024-09-18T08:10:41.493387+0000","last_undegraded":"2024-09-18T08:10:41.493387+0000","last_fullsized":"2024-09-18T08:10:41.493387+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:10:08.473626+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:46.860092+0000","last_clean_scrub_stamp":"2024-09-18T08:10:08.473626+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:11:38.009160+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"69'15","reported_seq":372,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-09-18T08:10:41.492295+0000","last_change":"2024-09-18T08:09:32.575836+0000","last_active":"2024-09-18T08:10:41.492295+0000","last_peered":"2024-09-18T08:10:41.492295+0000","last_clean":"2024-09-18T08:10:41.492295+0000","last_became_active":"2024-09-18T08:02:49.886106+0000","last_became_peered":"2024-09-18T08:02:49.886106+0000","last_unstale":"2024-09-18T08:10:41.492295+0000","last_undegraded":"2024-09-18T08:10:41.492295+0000","last_fullsized":"2024-09-18T08:10:41.492295+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'15","last_scrub_stamp":"2024-09-18T08:09:32.575741+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_clean_scrub_stamp":"2024-09-18T08:09:32.575741+0000","objects_scrubbed":7,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:10:59.141685+0000","scrub_duration":40,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":101,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-09-18T08:10:41.488621+0000","last_change":"2024-09-18T08:08:18.543316+0000","last_active":"2024-09-18T08:10:41.488621+0000","last_peered":"2024-09-18T08:10:41.488621+0000","last_clean":"2024-09-18T08:10:41.488621+0000","last_became_active":"2024-09-18T08:08:18.538645+0000","last_became_peered":"2024-09-18T08:08:18.538645+0000","last_unstale":"2024-09-18T08:10:41.488621+0000","last_undegraded":"2024-09-18T08:10:41.488621+0000","last_fullsized":"2024-09-18T08:10:41.488621+0000","mapping_epoch":181,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":182,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:08:09.142188+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:50.886830+0000","last_clean_scrub_stamp":"2024-09-18T08:08:09.142188+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":386,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-09-18T08:10:41.499908+0000","last_change":"2024-09-18T08:08:18.548588+0000","last_active":"2024-09-18T08:10:41.499908+0000","last_peered":"2024-09-18T08:10:41.499908+0000","last_clean":"2024-09-18T08:10:41.499908+0000","last_became_active":"2024-09-18T08:08:18.548203+0000","last_became_peered":"2024-09-18T08:08:18.548203+0000","last_unstale":"2024-09-18T08:10:41.499908+0000","last_undegraded":"2024-09-18T08:10:41.499908+0000","last_fullsized":"2024-09-18T08:10:41.499908+0000","mapping_epoch":181,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":182,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:06:14.027077+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:46.860092+0000","last_clean_scrub_stamp":"2024-09-18T08:06:14.027077+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"69'9","reported_seq":118,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-09-18T08:10:41.489328+0000","last_change":"2024-09-18T08:10:16.220584+0000","last_active":"2024-09-18T08:10:41.489328+0000","last_peered":"2024-09-18T08:10:41.489328+0000","last_clean":"2024-09-18T08:10:41.489328+0000","last_became_active":"2024-09-18T08:08:18.550960+0000","last_became_peered":"2024-09-18T08:08:18.550960+0000","last_unstale":"2024-09-18T08:10:41.489328+0000","last_undegraded":"2024-09-18T08:10:41.489328+0000","last_fullsized":"2024-09-18T08:10:41.489328+0000","mapping_epoch":181,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":182,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'9","last_scrub_stamp":"2024-09-18T08:10:16.220503+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_clean_scrub_stamp":"2024-09-18T08:10:16.220503+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:11:28.084018+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":16,"num_read_kb":10,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":350,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-09-18T08:10:41.499437+0000","last_change":"2024-09-18T08:05:33.935844+0000","last_active":"2024-09-18T08:10:41.499437+0000","last_peered":"2024-09-18T08:10:41.499437+0000","last_clean":"2024-09-18T08:10:41.499437+0000","last_became_active":"2024-09-18T08:02:51.907296+0000","last_became_peered":"2024-09-18T08:02:51.907296+0000","last_unstale":"2024-09-18T08:10:41.499437+0000","last_undegraded":"2024-09-18T08:10:41.499437+0000","last_fullsized":"2024-09-18T08:10:41.499437+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:05:33.935798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:50.886830+0000","last_clean_scrub_stamp":"2024-09-18T08:05:33.935798+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:11:39.047719+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":347,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-09-18T08:10:41.499682+0000","last_change":"2024-09-18T08:08:14.646073+0000","last_active":"2024-09-18T08:10:41.499682+0000","last_peered":"2024-09-18T08:10:41.499682+0000","last_clean":"2024-09-18T08:10:41.499682+0000","last_became_active":"2024-09-18T08:02:53.913519+0000","last_became_peered":"2024-09-18T08:02:53.913519+0000","last_unstale":"2024-09-18T08:10:41.499682+0000","last_undegraded":"2024-09-18T08:10:41.499682+0000","last_fullsized":"2024-09-18T08:10:41.499682+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:08:14.645993+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_clean_scrub_stamp":"2024-09-18T08:08:14.645993+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:11:39.786325+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":364,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-09-18T08:10:41.499220+0000","last_change":"2024-09-18T08:08:18.624260+0000","last_active":"2024-09-18T08:10:41.499220+0000","last_peered":"2024-09-18T08:10:41.499220+0000","last_clean":"2024-09-18T08:10:41.499220+0000","last_became_active":"2024-09-18T08:08:18.623757+0000","last_became_peered":"2024-09-18T08:08:18.623757+0000","last_unstale":"2024-09-18T08:10:41.499220+0000","last_undegraded":"2024-09-18T08:10:41.499220+0000","last_fullsized":"2024-09-18T08:10:41.499220+0000","mapping_epoch":181,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":182,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:04:32.849119+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:46.860092+0000","last_clean_scrub_stamp":"2024-09-18T08:04:32.849119+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:11:33.035769+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1b","version":"69'5","reported_seq":106,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-09-18T08:10:41.488313+0000","last_change":"2024-09-18T08:08:18.651427+0000","last_active":"2024-09-18T08:10:41.488313+0000","last_peered":"2024-09-18T08:10:41.488313+0000","last_clean":"2024-09-18T08:10:41.488313+0000","last_became_active":"2024-09-18T08:08:18.567126+0000","last_became_peered":"2024-09-18T08:08:18.567126+0000","last_unstale":"2024-09-18T08:10:41.488313+0000","last_undegraded":"2024-09-18T08:10:41.488313+0000","last_fullsized":"2024-09-18T08:10:41.488313+0000","mapping_epoch":181,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":182,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'5","last_scrub_stamp":"2024-09-18T08:06:09.465979+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_clean_scrub_stamp":"2024-09-18T08:06:09.465979+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":10,"num_read_kb":6,"num_write":6,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":388,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-09-18T08:10:41.496076+0000","last_change":"2024-09-18T08:10:16.664655+0000","last_active":"2024-09-18T08:10:41.496076+0000","last_peered":"2024-09-18T08:10:41.496076+0000","last_clean":"2024-09-18T08:10:41.496076+0000","last_became_active":"2024-09-18T08:08:18.524932+0000","last_became_peered":"2024-09-18T08:08:18.524932+0000","last_unstale":"2024-09-18T08:10:41.496076+0000","last_undegraded":"2024-09-18T08:10:41.496076+0000","last_fullsized":"2024-09-18T08:10:41.496076+0000","mapping_epoch":181,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":182,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:10:16.664597+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:50.886830+0000","last_clean_scrub_stamp":"2024-09-18T08:10:16.664597+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:11:29.391762+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":347,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-09-18T08:10:41.499186+0000","last_change":"2024-09-18T08:06:44.518882+0000","last_active":"2024-09-18T08:10:41.499186+0000","last_peered":"2024-09-18T08:10:41.499186+0000","last_clean":"2024-09-18T08:10:41.499186+0000","last_became_active":"2024-09-18T08:02:53.911648+0000","last_became_peered":"2024-09-18T08:02:53.911648+0000","last_unstale":"2024-09-18T08:10:41.499186+0000","last_undegraded":"2024-09-18T08:10:41.499186+0000","last_fullsized":"2024-09-18T08:10:41.499186+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:06:44.518835+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_clean_scrub_stamp":"2024-09-18T08:06:44.518835+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":101,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-09-18T08:10:41.490288+0000","last_change":"2024-09-18T08:08:18.544203+0000","last_active":"2024-09-18T08:10:41.490288+0000","last_peered":"2024-09-18T08:10:41.490288+0000","last_clean":"2024-09-18T08:10:41.490288+0000","last_became_active":"2024-09-18T08:08:18.539604+0000","last_became_peered":"2024-09-18T08:08:18.539604+0000","last_unstale":"2024-09-18T08:10:41.490288+0000","last_undegraded":"2024-09-18T08:10:41.490288+0000","last_fullsized":"2024-09-18T08:10:41.490288+0000","mapping_epoch":181,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":182,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:05:40.477965+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_clean_scrub_stamp":"2024-09-18T08:05:40.477965+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:11:29.146093+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":372,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-09-18T08:10:41.506284+0000","last_change":"2024-09-18T08:08:18.524124+0000","last_active":"2024-09-18T08:10:41.506284+0000","last_peered":"2024-09-18T08:10:41.506284+0000","last_clean":"2024-09-18T08:10:41.506284+0000","last_became_active":"2024-09-18T08:08:18.523611+0000","last_became_peered":"2024-09-18T08:08:18.523611+0000","last_unstale":"2024-09-18T08:10:41.506284+0000","last_undegraded":"2024-09-18T08:10:41.506284+0000","last_fullsized":"2024-09-18T08:10:41.506284+0000","mapping_epoch":181,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":182,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:05:27.476144+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:46.860092+0000","last_clean_scrub_stamp":"2024-09-18T08:05:27.476144+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:11:25.862158+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1c","version":"69'15","reported_seq":82,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-09-18T08:10:41.497250+0000","last_change":"2024-09-18T08:08:52.708963+0000","last_active":"2024-09-18T08:10:41.497250+0000","last_peered":"2024-09-18T08:10:41.497250+0000","last_clean":"2024-09-18T08:10:41.497250+0000","last_became_active":"2024-09-18T08:08:52.690788+0000","last_became_peered":"2024-09-18T08:08:52.690788+0000","last_unstale":"2024-09-18T08:10:41.497250+0000","last_undegraded":"2024-09-18T08:10:41.497250+0000","last_fullsized":"2024-09-18T08:10:41.497250+0000","mapping_epoch":198,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":199,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'15","last_scrub_stamp":"2024-09-18T08:07:39.530340+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_clean_scrub_stamp":"2024-09-18T08:07:39.530340+0000","objects_scrubbed":7,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:11:34.605598+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":380,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-09-18T08:10:41.501918+0000","last_change":"2024-09-18T08:09:11.693711+0000","last_active":"2024-09-18T08:10:41.501918+0000","last_peered":"2024-09-18T08:10:41.501918+0000","last_clean":"2024-09-18T08:10:41.501918+0000","last_became_active":"2024-09-18T08:08:18.538505+0000","last_became_peered":"2024-09-18T08:08:18.538505+0000","last_unstale":"2024-09-18T08:10:41.501918+0000","last_undegraded":"2024-09-18T08:10:41.501918+0000","last_fullsized":"2024-09-18T08:10:41.501918+0000","mapping_epoch":181,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":182,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:09:11.693643+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:50.886830+0000","last_clean_scrub_stamp":"2024-09-18T08:09:11.693643+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":361,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-09-18T08:10:41.510117+0000","last_change":"2024-09-18T08:09:36.592348+0000","last_active":"2024-09-18T08:10:41.510117+0000","last_peered":"2024-09-18T08:10:41.510117+0000","last_clean":"2024-09-18T08:10:41.510117+0000","last_became_active":"2024-09-18T08:06:24.128659+0000","last_became_peered":"2024-09-18T08:06:24.128659+0000","last_unstale":"2024-09-18T08:10:41.510117+0000","last_undegraded":"2024-09-18T08:10:41.510117+0000","last_fullsized":"2024-09-18T08:10:41.510117+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:09:36.592281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_clean_scrub_stamp":"2024-09-18T08:09:36.592281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:10:46.573780+0000","scrub_duration":16,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":100,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-09-18T08:10:41.489045+0000","last_change":"2024-09-18T08:08:18.536559+0000","last_active":"2024-09-18T08:10:41.489045+0000","last_peered":"2024-09-18T08:10:41.489045+0000","last_clean":"2024-09-18T08:10:41.489045+0000","last_became_active":"2024-09-18T08:08:18.536084+0000","last_became_peered":"2024-09-18T08:08:18.536084+0000","last_unstale":"2024-09-18T08:10:41.489045+0000","last_undegraded":"2024-09-18T08:10:41.489045+0000","last_fullsized":"2024-09-18T08:10:41.489045+0000","mapping_epoch":181,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":182,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:07:13.543722+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:46.860092+0000","last_clean_scrub_stamp":"2024-09-18T08:07:13.543722+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"69'12","reported_seq":424,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-09-18T08:10:41.492871+0000","last_change":"2024-09-18T08:10:12.443637+0000","last_active":"2024-09-18T08:10:41.492871+0000","last_peered":"2024-09-18T08:10:41.492871+0000","last_clean":"2024-09-18T08:10:41.492871+0000","last_became_active":"2024-09-18T08:08:18.543675+0000","last_became_peered":"2024-09-18T08:08:18.543675+0000","last_unstale":"2024-09-18T08:10:41.492871+0000","last_undegraded":"2024-09-18T08:10:41.492871+0000","last_fullsized":"2024-09-18T08:10:41.492871+0000","mapping_epoch":181,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":182,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'12","last_scrub_stamp":"2024-09-18T08:10:12.443558+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_clean_scrub_stamp":"2024-09-18T08:10:12.443558+0000","objects_scrubbed":5,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:11:22.178344+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":25,"num_read_kb":16,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":100,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-09-18T08:10:41.489823+0000","last_change":"2024-09-18T08:08:18.532482+0000","last_active":"2024-09-18T08:10:41.489823+0000","last_peered":"2024-09-18T08:10:41.489823+0000","last_clean":"2024-09-18T08:10:41.489823+0000","last_became_active":"2024-09-18T08:08:18.531438+0000","last_became_peered":"2024-09-18T08:08:18.531438+0000","last_unstale":"2024-09-18T08:10:41.489823+0000","last_undegraded":"2024-09-18T08:10:41.489823+0000","last_fullsized":"2024-09-18T08:10:41.489823+0000","mapping_epoch":181,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":182,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:06:12.434179+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:50.886830+0000","last_clean_scrub_stamp":"2024-09-18T08:06:12.434179+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1c","version":"69'1","reported_seq":189,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-09-18T08:10:41.498980+0000","last_change":"2024-09-18T08:06:46.142143+0000","last_active":"2024-09-18T08:10:41.498980+0000","last_peered":"2024-09-18T08:10:41.498980+0000","last_clean":"2024-09-18T08:10:41.498980+0000","last_became_active":"2024-09-18T08:06:46.141661+0000","last_became_peered":"2024-09-18T08:06:46.141661+0000","last_unstale":"2024-09-18T08:10:41.498980+0000","last_undegraded":"2024-09-18T08:10:41.498980+0000","last_fullsized":"2024-09-18T08:10:41.498980+0000","mapping_epoch":143,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":144,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'1","last_scrub_stamp":"2024-09-18T08:05:36.776369+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_clean_scrub_stamp":"2024-09-18T08:05:36.776369+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":403,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,2],"acting":[5,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"62'1","reported_seq":389,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-09-18T08:10:41.510238+0000","last_change":"2024-09-18T08:09:29.633875+0000","last_active":"2024-09-18T08:10:41.510238+0000","last_peered":"2024-09-18T08:10:41.510238+0000","last_clean":"2024-09-18T08:10:41.510238+0000","last_became_active":"2024-09-18T08:08:18.559125+0000","last_became_peered":"2024-09-18T08:08:18.559125+0000","last_unstale":"2024-09-18T08:10:41.510238+0000","last_undegraded":"2024-09-18T08:10:41.510238+0000","last_fullsized":"2024-09-18T08:10:41.510238+0000","mapping_epoch":181,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":182,"parent":"0.0","parent_split_bits":0,"last_scrub":"62'1","last_scrub_stamp":"2024-09-18T08:09:29.633520+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:46.860092+0000","last_clean_scrub_stamp":"2024-09-18T08:09:29.633520+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":92,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1e","version":"69'10","reported_seq":361,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-09-18T08:10:41.499264+0000","last_change":"2024-09-18T08:10:19.932544+0000","last_active":"2024-09-18T08:10:41.499264+0000","last_peered":"2024-09-18T08:10:41.499264+0000","last_clean":"2024-09-18T08:10:41.499264+0000","last_became_active":"2024-09-18T08:02:49.892024+0000","last_became_peered":"2024-09-18T08:02:49.892024+0000","last_unstale":"2024-09-18T08:10:41.499264+0000","last_undegraded":"2024-09-18T08:10:41.499264+0000","last_fullsized":"2024-09-18T08:10:41.499264+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'10","last_scrub_stamp":"2024-09-18T08:10:19.932465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_clean_scrub_stamp":"2024-09-18T08:10:19.932465+0000","objects_scrubbed":5,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:11:44.918707+0000","scrub_duration":19,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1f","version":"182'13","reported_seq":135,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-09-18T08:10:41.490596+0000","last_change":"2024-09-18T08:08:18.627670+0000","last_active":"2024-09-18T08:10:41.490596+0000","last_peered":"2024-09-18T08:10:41.490596+0000","last_clean":"2024-09-18T08:10:41.490596+0000","last_became_active":"2024-09-18T08:08:18.543095+0000","last_became_peered":"2024-09-18T08:08:18.543095+0000","last_unstale":"2024-09-18T08:10:41.490596+0000","last_undegraded":"2024-09-18T08:10:41.490596+0000","last_fullsized":"2024-09-18T08:10:41.490596+0000","mapping_epoch":181,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":182,"parent":"0.0","parent_split_bits":0,"last_scrub":"173'12","last_scrub_stamp":"2024-09-18T08:08:07.553075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:50.886830+0000","last_clean_scrub_stamp":"2024-09-18T08:08:07.553075+0000","objects_scrubbed":0,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,7],"acting":[4,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.3","version":"226'11894","reported_seq":12100,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-09-18T08:10:43.807473+0000","last_change":"2024-09-18T08:09:28.263522+0000","last_active":"2024-09-18T08:10:43.807473+0000","last_peered":"2024-09-18T08:10:43.807473+0000","last_clean":"2024-09-18T08:10:43.807473+0000","last_became_active":"2024-09-18T08:07:53.672797+0000","last_became_peered":"2024-09-18T08:07:53.672797+0000","last_unstale":"2024-09-18T08:10:43.807473+0000","last_undegraded":"2024-09-18T08:10:43.807473+0000","last_fullsized":"2024-09-18T08:10:43.807473+0000","mapping_epoch":165,"log_start":"213'7000","ondisk_log_start":"213'7000","created":165,"last_epoch_clean":166,"parent":"0.0","parent_split_bits":0,"last_scrub":"211'6448","last_scrub_stamp":"2024-09-18T08:09:28.263434+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:07:52.307205+0000","last_clean_scrub_stamp":"2024-09-18T08:09:28.263434+0000","objects_scrubbed":6448,"log_size":4894,"log_dups_size":0,"ondisk_log_size":4894,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":646,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":65667072,"num_objects":1002,"num_object_clones":0,"num_object_copies":3006,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1002,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11894,"num_write_kb":412672,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6],"acting":[1,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.a","version":"74'79","reported_seq":171,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-09-18T08:10:41.500262+0000","last_change":"2024-09-18T08:08:57.463205+0000","last_active":"2024-09-18T08:10:41.500262+0000","last_peered":"2024-09-18T08:10:41.500262+0000","last_clean":"2024-09-18T08:10:41.500262+0000","last_became_active":"2024-09-18T08:07:26.394584+0000","last_became_peered":"2024-09-18T08:07:26.394584+0000","last_unstale":"2024-09-18T08:10:41.500262+0000","last_undegraded":"2024-09-18T08:10:41.500262+0000","last_fullsized":"2024-09-18T08:10:41.500262+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":152,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":4,"last_scrub":"74'79","last_scrub_stamp":"2024-09-18T08:08:57.463113+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:00:11.893742+0000","last_clean_scrub_stamp":"2024-09-18T08:08:57.463113+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,6],"acting":[5,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.f","version":"69'15","reported_seq":433,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-09-18T08:10:41.548698+0000","last_change":"2024-09-18T08:10:41.548698+0000","last_active":"2024-09-18T08:10:41.548698+0000","last_peered":"2024-09-18T08:10:41.548698+0000","last_clean":"2024-09-18T08:10:41.548698+0000","last_became_active":"2024-09-18T08:08:18.540211+0000","last_became_peered":"2024-09-18T08:08:18.540211+0000","last_unstale":"2024-09-18T08:10:41.548698+0000","last_undegraded":"2024-09-18T08:10:41.548698+0000","last_fullsized":"2024-09-18T08:10:41.548698+0000","mapping_epoch":181,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":182,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'15","last_scrub_stamp":"2024-09-18T08:10:41.548188+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_clean_scrub_stamp":"2024-09-18T08:10:41.548188+0000","objects_scrubbed":7,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:11:52.798979+0000","scrub_duration":29,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":359,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-09-18T08:10:41.498827+0000","last_change":"2024-09-18T08:08:18.389332+0000","last_active":"2024-09-18T08:10:41.498827+0000","last_peered":"2024-09-18T08:10:41.498827+0000","last_clean":"2024-09-18T08:10:41.498827+0000","last_became_active":"2024-09-18T08:02:47.879328+0000","last_became_peered":"2024-09-18T08:02:47.879328+0000","last_unstale":"2024-09-18T08:10:41.498827+0000","last_undegraded":"2024-09-18T08:10:41.498827+0000","last_fullsized":"2024-09-18T08:10:41.498827+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:08:18.389273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:46.860092+0000","last_clean_scrub_stamp":"2024-09-18T08:08:18.389273+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:11:39.463753+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.e","version":"182'12","reported_seq":135,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-09-18T08:10:41.490232+0000","last_change":"2024-09-18T08:08:18.560884+0000","last_active":"2024-09-18T08:10:41.490232+0000","last_peered":"2024-09-18T08:10:41.490232+0000","last_clean":"2024-09-18T08:10:41.490232+0000","last_became_active":"2024-09-18T08:08:18.525147+0000","last_became_peered":"2024-09-18T08:08:18.525147+0000","last_unstale":"2024-09-18T08:10:41.490232+0000","last_undegraded":"2024-09-18T08:10:41.490232+0000","last_fullsized":"2024-09-18T08:10:41.490232+0000","mapping_epoch":181,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":182,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'8","last_scrub_stamp":"2024-09-18T08:06:19.459375+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:50.886830+0000","last_clean_scrub_stamp":"2024-09-18T08:06:19.459375+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":355,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-09-18T08:10:41.499298+0000","last_change":"2024-09-18T08:07:19.714477+0000","last_active":"2024-09-18T08:10:41.499298+0000","last_peered":"2024-09-18T08:10:41.499298+0000","last_clean":"2024-09-18T08:10:41.499298+0000","last_became_active":"2024-09-18T08:02:53.909834+0000","last_became_peered":"2024-09-18T08:02:53.909834+0000","last_unstale":"2024-09-18T08:10:41.499298+0000","last_undegraded":"2024-09-18T08:10:41.499298+0000","last_fullsized":"2024-09-18T08:10:41.499298+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:07:19.714430+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_clean_scrub_stamp":"2024-09-18T08:07:19.714430+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.c","version":"226'11451","reported_seq":11625,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-09-18T08:10:43.826264+0000","last_change":"2024-09-18T08:10:11.144727+0000","last_active":"2024-09-18T08:10:43.826264+0000","last_peered":"2024-09-18T08:10:43.826264+0000","last_clean":"2024-09-18T08:10:43.826264+0000","last_became_active":"2024-09-18T08:07:53.669708+0000","last_became_peered":"2024-09-18T08:07:53.669708+0000","last_unstale":"2024-09-18T08:10:43.826264+0000","last_undegraded":"2024-09-18T08:10:43.826264+0000","last_fullsized":"2024-09-18T08:10:43.826264+0000","mapping_epoch":165,"log_start":"210'6000","ondisk_log_start":"210'6000","created":165,"last_epoch_clean":166,"parent":"0.0","parent_split_bits":0,"last_scrub":"217'9165","last_scrub_stamp":"2024-09-18T08:10:11.144412+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:07:52.307205+0000","last_clean_scrub_stamp":"2024-09-18T08:10:11.144412+0000","objects_scrubbed":3298,"log_size":5451,"log_dups_size":0,"ondisk_log_size":5451,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:11:37.247660+0000","scrub_duration":1357,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":64159744,"num_objects":979,"num_object_clones":0,"num_object_copies":2937,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":979,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11451,"num_write_kb":397760,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"74'79","reported_seq":116,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-09-18T08:10:41.489427+0000","last_change":"2024-09-18T08:08:52.690758+0000","last_active":"2024-09-18T08:10:41.489427+0000","last_peered":"2024-09-18T08:10:41.489427+0000","last_clean":"2024-09-18T08:10:41.489427+0000","last_became_active":"2024-09-18T08:08:52.690344+0000","last_became_peered":"2024-09-18T08:08:52.690344+0000","last_unstale":"2024-09-18T08:10:41.489427+0000","last_undegraded":"2024-09-18T08:10:41.489427+0000","last_fullsized":"2024-09-18T08:10:41.489427+0000","mapping_epoch":198,"log_start":"0'0","ondisk_log_start":"0'0","created":152,"last_epoch_clean":199,"parent":"0.0","parent_split_bits":3,"last_scrub":"74'79","last_scrub_stamp":"2024-09-18T08:08:19.036836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:00:11.893742+0000","last_clean_scrub_stamp":"2024-09-18T08:08:19.036836+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,6],"acting":[4,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.0","version":"69'18","reported_seq":387,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-09-18T08:10:41.498544+0000","last_change":"2024-09-18T08:06:44.301130+0000","last_active":"2024-09-18T08:10:41.498544+0000","last_peered":"2024-09-18T08:10:41.498544+0000","last_clean":"2024-09-18T08:10:41.498544+0000","last_became_active":"2024-09-18T08:02:49.886818+0000","last_became_peered":"2024-09-18T08:02:49.886818+0000","last_unstale":"2024-09-18T08:10:41.498544+0000","last_undegraded":"2024-09-18T08:10:41.498544+0000","last_fullsized":"2024-09-18T08:10:41.498544+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'18","last_scrub_stamp":"2024-09-18T08:06:44.300772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_clean_scrub_stamp":"2024-09-18T08:06:44.300772+0000","objects_scrubbed":8,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:10:52.429408+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":34,"num_read_kb":22,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":375,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-09-18T08:10:41.512353+0000","last_change":"2024-09-18T08:10:09.469428+0000","last_active":"2024-09-18T08:10:41.512353+0000","last_peered":"2024-09-18T08:10:41.512353+0000","last_clean":"2024-09-18T08:10:41.512353+0000","last_became_active":"2024-09-18T08:02:47.880225+0000","last_became_peered":"2024-09-18T08:02:47.880225+0000","last_unstale":"2024-09-18T08:10:41.512353+0000","last_undegraded":"2024-09-18T08:10:41.512353+0000","last_fullsized":"2024-09-18T08:10:41.512353+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:10:09.469351+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:46.860092+0000","last_clean_scrub_stamp":"2024-09-18T08:10:09.469351+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:11:16.425431+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":101,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-09-18T08:10:41.488357+0000","last_change":"2024-09-18T08:08:18.530658+0000","last_active":"2024-09-18T08:10:41.488357+0000","last_peered":"2024-09-18T08:10:41.488357+0000","last_clean":"2024-09-18T08:10:41.488357+0000","last_became_active":"2024-09-18T08:08:18.530341+0000","last_became_peered":"2024-09-18T08:08:18.530341+0000","last_unstale":"2024-09-18T08:10:41.488357+0000","last_undegraded":"2024-09-18T08:10:41.488357+0000","last_fullsized":"2024-09-18T08:10:41.488357+0000","mapping_epoch":181,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":182,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:06:10.468119+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:50.886830+0000","last_clean_scrub_stamp":"2024-09-18T08:06:10.468119+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":109,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-09-18T08:10:41.489249+0000","last_change":"2024-09-18T08:10:17.226945+0000","last_active":"2024-09-18T08:10:41.489249+0000","last_peered":"2024-09-18T08:10:41.489249+0000","last_clean":"2024-09-18T08:10:41.489249+0000","last_became_active":"2024-09-18T08:08:18.543694+0000","last_became_peered":"2024-09-18T08:08:18.543694+0000","last_unstale":"2024-09-18T08:10:41.489249+0000","last_undegraded":"2024-09-18T08:10:41.489249+0000","last_fullsized":"2024-09-18T08:10:41.489249+0000","mapping_epoch":181,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":182,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:10:17.226867+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_clean_scrub_stamp":"2024-09-18T08:10:17.226867+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:11:38.990188+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.d","version":"226'11541","reported_seq":11676,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-09-18T08:10:43.282776+0000","last_change":"2024-09-18T08:07:53.671291+0000","last_active":"2024-09-18T08:10:43.282776+0000","last_peered":"2024-09-18T08:10:43.282776+0000","last_clean":"2024-09-18T08:10:43.282776+0000","last_became_active":"2024-09-18T08:07:53.670773+0000","last_became_peered":"2024-09-18T08:07:53.670773+0000","last_unstale":"2024-09-18T08:10:43.282776+0000","last_undegraded":"2024-09-18T08:10:43.282776+0000","last_fullsized":"2024-09-18T08:10:43.282776+0000","mapping_epoch":165,"log_start":"212'6514","ondisk_log_start":"212'6514","created":165,"last_epoch_clean":166,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:07:52.307205+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:07:52.307205+0000","last_clean_scrub_stamp":"2024-09-18T08:07:52.307205+0000","objects_scrubbed":0,"log_size":5027,"log_dups_size":0,"ondisk_log_size":5027,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":63438872,"num_objects":969,"num_object_clones":0,"num_object_copies":2907,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":969,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":11541,"num_write_kb":400257,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,2],"acting":[5,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.4","version":"74'79","reported_seq":185,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-09-18T08:10:43.473244+0000","last_change":"2024-09-18T08:10:43.473244+0000","last_active":"2024-09-18T08:10:43.473244+0000","last_peered":"2024-09-18T08:10:43.473244+0000","last_clean":"2024-09-18T08:10:43.473244+0000","last_became_active":"2024-09-18T08:07:26.398930+0000","last_became_peered":"2024-09-18T08:07:26.398930+0000","last_unstale":"2024-09-18T08:10:43.473244+0000","last_undegraded":"2024-09-18T08:10:43.473244+0000","last_fullsized":"2024-09-18T08:10:43.473244+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":152,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":3,"last_scrub":"74'79","last_scrub_stamp":"2024-09-18T08:10:43.473169+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:00:11.893742+0000","last_clean_scrub_stamp":"2024-09-18T08:10:43.473169+0000","objects_scrubbed":1,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:11:58.485306+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":327680,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,7],"acting":[3,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"69'14","reported_seq":117,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-09-18T08:10:41.488202+0000","last_change":"2024-09-18T08:08:18.617734+0000","last_active":"2024-09-18T08:10:41.488202+0000","last_peered":"2024-09-18T08:10:41.488202+0000","last_clean":"2024-09-18T08:10:41.488202+0000","last_became_active":"2024-09-18T08:08:18.549671+0000","last_became_peered":"2024-09-18T08:08:18.549671+0000","last_unstale":"2024-09-18T08:10:41.488202+0000","last_undegraded":"2024-09-18T08:10:41.488202+0000","last_fullsized":"2024-09-18T08:10:41.488202+0000","mapping_epoch":181,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":182,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'14","last_scrub_stamp":"2024-09-18T08:08:08.629246+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_clean_scrub_stamp":"2024-09-18T08:08:08.629246+0000","objects_scrubbed":0,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":14,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"62'2","reported_seq":391,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-09-18T08:10:41.499565+0000","last_change":"2024-09-18T08:08:18.565279+0000","last_active":"2024-09-18T08:10:41.499565+0000","last_peered":"2024-09-18T08:10:41.499565+0000","last_clean":"2024-09-18T08:10:41.499565+0000","last_became_active":"2024-09-18T08:08:18.544872+0000","last_became_peered":"2024-09-18T08:08:18.544872+0000","last_unstale":"2024-09-18T08:10:41.499565+0000","last_undegraded":"2024-09-18T08:10:41.499565+0000","last_fullsized":"2024-09-18T08:10:41.499565+0000","mapping_epoch":181,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":182,"parent":"0.0","parent_split_bits":0,"last_scrub":"62'2","last_scrub_stamp":"2024-09-18T08:06:43.607152+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:46.860092+0000","last_clean_scrub_stamp":"2024-09-18T08:06:43.607152+0000","objects_scrubbed":2,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:11:27.842327+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1039,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":2078,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.0","version":"182'10","reported_seq":477,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-09-18T08:10:41.498399+0000","last_change":"2024-09-18T08:09:51.467488+0000","last_active":"2024-09-18T08:10:41.498399+0000","last_peered":"2024-09-18T08:10:41.498399+0000","last_clean":"2024-09-18T08:10:41.498399+0000","last_became_active":"2024-09-18T08:08:18.541756+0000","last_became_peered":"2024-09-18T08:08:18.541756+0000","last_unstale":"2024-09-18T08:10:41.498399+0000","last_undegraded":"2024-09-18T08:10:41.498399+0000","last_fullsized":"2024-09-18T08:10:41.498399+0000","mapping_epoch":181,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":182,"parent":"0.0","parent_split_bits":0,"last_scrub":"182'10","last_scrub_stamp":"2024-09-18T08:09:51.467349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:50.886830+0000","last_clean_scrub_stamp":"2024-09-18T08:09:51.467349+0000","objects_scrubbed":1,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:10:51.919411+0000","scrub_duration":19,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":377,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-09-18T08:10:41.495954+0000","last_change":"2024-09-18T08:10:15.663125+0000","last_active":"2024-09-18T08:10:41.495954+0000","last_peered":"2024-09-18T08:10:41.495954+0000","last_clean":"2024-09-18T08:10:41.495954+0000","last_became_active":"2024-09-18T08:06:24.140303+0000","last_became_peered":"2024-09-18T08:06:24.140303+0000","last_unstale":"2024-09-18T08:10:41.495954+0000","last_undegraded":"2024-09-18T08:10:41.495954+0000","last_fullsized":"2024-09-18T08:10:41.495954+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:10:15.663047+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_clean_scrub_stamp":"2024-09-18T08:10:15.663047+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:11:20.567559+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.e","version":"226'11757","reported_seq":16125,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-09-18T08:10:43.868612+0000","last_change":"2024-09-18T08:09:17.296111+0000","last_active":"2024-09-18T08:10:43.868612+0000","last_peered":"2024-09-18T08:10:43.868612+0000","last_clean":"2024-09-18T08:10:43.868612+0000","last_became_active":"2024-09-18T08:09:17.286866+0000","last_became_peered":"2024-09-18T08:09:17.286866+0000","last_unstale":"2024-09-18T08:10:43.868612+0000","last_undegraded":"2024-09-18T08:10:43.868612+0000","last_fullsized":"2024-09-18T08:10:43.868612+0000","mapping_epoch":207,"log_start":"210'6300","ondisk_log_start":"210'6300","created":165,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:07:52.307205+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:07:52.307205+0000","last_clean_scrub_stamp":"2024-09-18T08:07:52.307205+0000","objects_scrubbed":0,"log_size":5457,"log_dups_size":0,"ondisk_log_size":5457,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:11:32.807550+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":66256896,"num_objects":1011,"num_object_clones":0,"num_object_copies":3033,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1011,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11757,"num_write_kb":408576,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2107,"num_bytes_recovered":138084352,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,4],"acting":[0,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.7","version":"74'79","reported_seq":180,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-09-18T08:10:41.528597+0000","last_change":"2024-09-18T08:10:41.528597+0000","last_active":"2024-09-18T08:10:41.528597+0000","last_peered":"2024-09-18T08:10:41.528597+0000","last_clean":"2024-09-18T08:10:41.528597+0000","last_became_active":"2024-09-18T08:07:26.398289+0000","last_became_peered":"2024-09-18T08:07:26.398289+0000","last_unstale":"2024-09-18T08:10:41.528597+0000","last_undegraded":"2024-09-18T08:10:41.528597+0000","last_fullsized":"2024-09-18T08:10:41.528597+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":152,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":3,"last_scrub":"74'79","last_scrub_stamp":"2024-09-18T08:10:41.528526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:00:11.893742+0000","last_clean_scrub_stamp":"2024-09-18T08:10:41.528526+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:11:44.978584+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,7],"acting":[3,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.2","version":"69'10","reported_seq":423,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-09-18T08:10:41.509282+0000","last_change":"2024-09-18T08:10:18.637715+0000","last_active":"2024-09-18T08:10:41.509282+0000","last_peered":"2024-09-18T08:10:41.509282+0000","last_clean":"2024-09-18T08:10:41.509282+0000","last_became_active":"2024-09-18T08:08:18.538711+0000","last_became_peered":"2024-09-18T08:08:18.538711+0000","last_unstale":"2024-09-18T08:10:41.509282+0000","last_undegraded":"2024-09-18T08:10:41.509282+0000","last_fullsized":"2024-09-18T08:10:41.509282+0000","mapping_epoch":181,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":182,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'10","last_scrub_stamp":"2024-09-18T08:10:18.636906+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_clean_scrub_stamp":"2024-09-18T08:10:18.636906+0000","objects_scrubbed":5,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:11:21.146399+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":358,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-09-18T08:10:41.500008+0000","last_change":"2024-09-18T08:10:15.965550+0000","last_active":"2024-09-18T08:10:41.500008+0000","last_peered":"2024-09-18T08:10:41.500008+0000","last_clean":"2024-09-18T08:10:41.500008+0000","last_became_active":"2024-09-18T08:02:47.874727+0000","last_became_peered":"2024-09-18T08:02:47.874727+0000","last_unstale":"2024-09-18T08:10:41.500008+0000","last_undegraded":"2024-09-18T08:10:41.500008+0000","last_fullsized":"2024-09-18T08:10:41.500008+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:10:15.965485+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:46.860092+0000","last_clean_scrub_stamp":"2024-09-18T08:10:15.965485+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:11:38.102213+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.3","version":"138'10","reported_seq":477,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-09-18T08:10:41.501085+0000","last_change":"2024-09-18T08:07:46.695714+0000","last_active":"2024-09-18T08:10:41.501085+0000","last_peered":"2024-09-18T08:10:41.501085+0000","last_clean":"2024-09-18T08:10:41.501085+0000","last_became_active":"2024-09-18T08:06:24.134194+0000","last_became_peered":"2024-09-18T08:06:24.134194+0000","last_unstale":"2024-09-18T08:10:41.501085+0000","last_undegraded":"2024-09-18T08:10:41.501085+0000","last_fullsized":"2024-09-18T08:10:41.501085+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"138'10","last_scrub_stamp":"2024-09-18T08:07:46.695656+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:50.886830+0000","last_clean_scrub_stamp":"2024-09-18T08:07:46.695656+0000","objects_scrubbed":1,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:11:30.775523+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":377,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-09-18T08:10:41.499823+0000","last_change":"2024-09-18T08:09:46.989395+0000","last_active":"2024-09-18T08:10:41.499823+0000","last_peered":"2024-09-18T08:10:41.499823+0000","last_clean":"2024-09-18T08:10:41.499823+0000","last_became_active":"2024-09-18T08:08:52.700080+0000","last_became_peered":"2024-09-18T08:08:52.700080+0000","last_unstale":"2024-09-18T08:10:41.499823+0000","last_undegraded":"2024-09-18T08:10:41.499823+0000","last_fullsized":"2024-09-18T08:10:41.499823+0000","mapping_epoch":198,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":199,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:09:46.989329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_clean_scrub_stamp":"2024-09-18T08:09:46.989329+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:10:55.582764+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.f","version":"226'11912","reported_seq":28141,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-09-18T08:10:43.828043+0000","last_change":"2024-09-18T08:10:20.726308+0000","last_active":"2024-09-18T08:10:43.828043+0000","last_peered":"2024-09-18T08:10:43.828043+0000","last_clean":"2024-09-18T08:10:43.828043+0000","last_became_active":"2024-09-18T08:09:28.363033+0000","last_became_peered":"2024-09-18T08:09:28.363033+0000","last_unstale":"2024-09-18T08:10:43.828043+0000","last_undegraded":"2024-09-18T08:10:43.828043+0000","last_fullsized":"2024-09-18T08:10:43.828043+0000","mapping_epoch":211,"log_start":"213'7400","ondisk_log_start":"213'7400","created":165,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"222'10103","last_scrub_stamp":"2024-09-18T08:10:20.725996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:07:52.307205+0000","last_clean_scrub_stamp":"2024-09-18T08:10:20.725996+0000","objects_scrubbed":2844,"log_size":4512,"log_dups_size":0,"ondisk_log_size":4512,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:11:40.364753+0000","scrub_duration":1160,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":65273856,"num_objects":996,"num_object_clones":0,"num_object_copies":2988,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":996,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11912,"num_write_kb":413056,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8010,"num_bytes_recovered":524943360,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,1],"acting":[3,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"74'79","reported_seq":252,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-09-18T08:10:41.498759+0000","last_change":"2024-09-18T08:08:57.736557+0000","last_active":"2024-09-18T08:10:41.498759+0000","last_peered":"2024-09-18T08:10:41.498759+0000","last_clean":"2024-09-18T08:10:41.498759+0000","last_became_active":"2024-09-18T08:07:26.426532+0000","last_became_peered":"2024-09-18T08:07:26.426532+0000","last_unstale":"2024-09-18T08:10:41.498759+0000","last_undegraded":"2024-09-18T08:10:41.498759+0000","last_fullsized":"2024-09-18T08:10:41.498759+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":152,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":3,"last_scrub":"74'79","last_scrub_stamp":"2024-09-18T08:08:57.736355+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:00:11.893742+0000","last_clean_scrub_stamp":"2024-09-18T08:08:57.736355+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,3],"acting":[0,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.3","version":"69'19","reported_seq":390,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-09-18T08:10:41.506065+0000","last_change":"2024-09-18T08:10:21.940118+0000","last_active":"2024-09-18T08:10:41.506065+0000","last_peered":"2024-09-18T08:10:41.506065+0000","last_clean":"2024-09-18T08:10:41.506065+0000","last_became_active":"2024-09-18T08:02:49.890318+0000","last_became_peered":"2024-09-18T08:02:49.890318+0000","last_unstale":"2024-09-18T08:10:41.506065+0000","last_undegraded":"2024-09-18T08:10:41.506065+0000","last_fullsized":"2024-09-18T08:10:41.506065+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'19","last_scrub_stamp":"2024-09-18T08:10:21.940027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_clean_scrub_stamp":"2024-09-18T08:10:21.940027+0000","objects_scrubbed":8,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:11:50.373134+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":39,"num_read_kb":25,"num_write":22,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.4","version":"62'1","reported_seq":394,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-09-18T08:10:41.501590+0000","last_change":"2024-09-18T08:10:07.593068+0000","last_active":"2024-09-18T08:10:41.501590+0000","last_peered":"2024-09-18T08:10:41.501590+0000","last_clean":"2024-09-18T08:10:41.501590+0000","last_became_active":"2024-09-18T08:08:52.688347+0000","last_became_peered":"2024-09-18T08:08:52.688347+0000","last_unstale":"2024-09-18T08:10:41.501590+0000","last_undegraded":"2024-09-18T08:10:41.501590+0000","last_fullsized":"2024-09-18T08:10:41.501590+0000","mapping_epoch":198,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":199,"parent":"0.0","parent_split_bits":0,"last_scrub":"62'1","last_scrub_stamp":"2024-09-18T08:10:07.593004+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:46.860092+0000","last_clean_scrub_stamp":"2024-09-18T08:10:07.593004+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:11:36.610081+0000","scrub_duration":26,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":436,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":872,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":276,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-09-18T08:10:41.517113+0000","last_change":"2024-09-18T08:06:24.133954+0000","last_active":"2024-09-18T08:10:41.517113+0000","last_peered":"2024-09-18T08:10:41.517113+0000","last_clean":"2024-09-18T08:10:41.517113+0000","last_became_active":"2024-09-18T08:06:24.133262+0000","last_became_peered":"2024-09-18T08:06:24.133262+0000","last_unstale":"2024-09-18T08:10:41.517113+0000","last_undegraded":"2024-09-18T08:10:41.517113+0000","last_fullsized":"2024-09-18T08:10:41.517113+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:04:13.377561+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:50.886830+0000","last_clean_scrub_stamp":"2024-09-18T08:04:13.377561+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:11:23.886465+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":382,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-09-18T08:10:41.496381+0000","last_change":"2024-09-18T08:08:52.688537+0000","last_active":"2024-09-18T08:10:41.496381+0000","last_peered":"2024-09-18T08:10:41.496381+0000","last_clean":"2024-09-18T08:10:41.496381+0000","last_became_active":"2024-09-18T08:08:52.688212+0000","last_became_peered":"2024-09-18T08:08:52.688212+0000","last_unstale":"2024-09-18T08:10:41.496381+0000","last_undegraded":"2024-09-18T08:10:41.496381+0000","last_fullsized":"2024-09-18T08:10:41.496381+0000","mapping_epoch":198,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":199,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:07:49.041138+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_clean_scrub_stamp":"2024-09-18T08:07:49.041138+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:11:27.552078+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.8","version":"226'11476","reported_seq":11610,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-09-18T08:10:43.306946+0000","last_change":"2024-09-18T08:07:53.671548+0000","last_active":"2024-09-18T08:10:43.306946+0000","last_peered":"2024-09-18T08:10:43.306946+0000","last_clean":"2024-09-18T08:10:43.306946+0000","last_became_active":"2024-09-18T08:07:53.670635+0000","last_became_peered":"2024-09-18T08:07:53.670635+0000","last_unstale":"2024-09-18T08:10:43.306946+0000","last_undegraded":"2024-09-18T08:10:43.306946+0000","last_fullsized":"2024-09-18T08:10:43.306946+0000","mapping_epoch":165,"log_start":"212'6440","ondisk_log_start":"212'6440","created":165,"last_epoch_clean":166,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:07:52.307205+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:07:52.307205+0000","last_clean_scrub_stamp":"2024-09-18T08:07:52.307205+0000","objects_scrubbed":0,"log_size":5036,"log_dups_size":0,"ondisk_log_size":5036,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":63963136,"num_objects":976,"num_object_clones":0,"num_object_copies":2928,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":976,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11476,"num_write_kb":398464,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.1","version":"74'79","reported_seq":171,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-09-18T08:10:41.498971+0000","last_change":"2024-09-18T08:08:59.553431+0000","last_active":"2024-09-18T08:10:41.498971+0000","last_peered":"2024-09-18T08:10:41.498971+0000","last_clean":"2024-09-18T08:10:41.498971+0000","last_became_active":"2024-09-18T08:07:26.400643+0000","last_became_peered":"2024-09-18T08:07:26.400643+0000","last_unstale":"2024-09-18T08:10:41.498971+0000","last_undegraded":"2024-09-18T08:10:41.498971+0000","last_fullsized":"2024-09-18T08:10:41.498971+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":152,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":4,"last_scrub":"74'79","last_scrub_stamp":"2024-09-18T08:08:59.552592+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:00:11.893742+0000","last_clean_scrub_stamp":"2024-09-18T08:08:59.552592+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,2],"acting":[5,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.4","version":"69'28","reported_seq":498,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-09-18T08:10:41.501487+0000","last_change":"2024-09-18T08:08:57.639985+0000","last_active":"2024-09-18T08:10:41.501487+0000","last_peered":"2024-09-18T08:10:41.501487+0000","last_clean":"2024-09-18T08:10:41.501487+0000","last_became_active":"2024-09-18T08:02:49.897221+0000","last_became_peered":"2024-09-18T08:02:49.897221+0000","last_unstale":"2024-09-18T08:10:41.501487+0000","last_undegraded":"2024-09-18T08:10:41.501487+0000","last_fullsized":"2024-09-18T08:10:41.501487+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'28","last_scrub_stamp":"2024-09-18T08:08:57.639628+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_clean_scrub_stamp":"2024-09-18T08:08:57.639628+0000","objects_scrubbed":10,"log_size":28,"log_dups_size":0,"ondisk_log_size":28,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":18,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":48,"num_read_kb":33,"num_write":26,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":101,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-09-18T08:10:41.488394+0000","last_change":"2024-09-18T08:08:18.543579+0000","last_active":"2024-09-18T08:10:41.488394+0000","last_peered":"2024-09-18T08:10:41.488394+0000","last_clean":"2024-09-18T08:10:41.488394+0000","last_became_active":"2024-09-18T08:08:18.538852+0000","last_became_peered":"2024-09-18T08:08:18.538852+0000","last_unstale":"2024-09-18T08:10:41.488394+0000","last_undegraded":"2024-09-18T08:10:41.488394+0000","last_fullsized":"2024-09-18T08:10:41.488394+0000","mapping_epoch":181,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":182,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:05:39.490141+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:46.860092+0000","last_clean_scrub_stamp":"2024-09-18T08:05:39.490141+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:11:40.278193+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"64'2","reported_seq":390,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-09-18T08:10:41.502121+0000","last_change":"2024-09-18T08:06:24.165227+0000","last_active":"2024-09-18T08:10:41.502121+0000","last_peered":"2024-09-18T08:10:41.502121+0000","last_clean":"2024-09-18T08:10:41.502121+0000","last_became_active":"2024-09-18T08:06:24.136080+0000","last_became_peered":"2024-09-18T08:06:24.136080+0000","last_unstale":"2024-09-18T08:10:41.502121+0000","last_undegraded":"2024-09-18T08:10:41.502121+0000","last_fullsized":"2024-09-18T08:10:41.502121+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"64'2","last_scrub_stamp":"2024-09-18T08:05:44.802942+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:45.855979+0000","last_clean_scrub_stamp":"2024-09-18T08:05:44.802942+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:11:40.093235+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.001348349,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":38,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":356,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-09-18T08:10:41.500057+0000","last_change":"2024-09-18T08:08:18.544944+0000","last_active":"2024-09-18T08:10:41.500057+0000","last_peered":"2024-09-18T08:10:41.500057+0000","last_clean":"2024-09-18T08:10:41.500057+0000","last_became_active":"2024-09-18T08:08:18.544260+0000","last_became_peered":"2024-09-18T08:08:18.544260+0000","last_unstale":"2024-09-18T08:10:41.500057+0000","last_undegraded":"2024-09-18T08:10:41.500057+0000","last_fullsized":"2024-09-18T08:10:41.500057+0000","mapping_epoch":181,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":182,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:07:53.582095+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:50.886830+0000","last_clean_scrub_stamp":"2024-09-18T08:07:53.582095+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:10:51.897980+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":362,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-09-18T08:10:41.493160+0000","last_change":"2024-09-18T08:08:18.543527+0000","last_active":"2024-09-18T08:10:41.493160+0000","last_peered":"2024-09-18T08:10:41.493160+0000","last_clean":"2024-09-18T08:10:41.493160+0000","last_became_active":"2024-09-18T08:08:18.542678+0000","last_became_peered":"2024-09-18T08:08:18.542678+0000","last_unstale":"2024-09-18T08:10:41.493160+0000","last_undegraded":"2024-09-18T08:10:41.493160+0000","last_fullsized":"2024-09-18T08:10:41.493160+0000","mapping_epoch":181,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":182,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:07:08.245063+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_clean_scrub_stamp":"2024-09-18T08:07:08.245063+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:10:56.470037+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.b","version":"226'11680","reported_seq":17326,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-09-18T08:10:43.851544+0000","last_change":"2024-09-18T08:08:48.662821+0000","last_active":"2024-09-18T08:10:43.851544+0000","last_peered":"2024-09-18T08:10:43.851544+0000","last_clean":"2024-09-18T08:10:43.851544+0000","last_became_active":"2024-09-18T08:08:48.655676+0000","last_became_peered":"2024-09-18T08:08:48.655676+0000","last_unstale":"2024-09-18T08:10:43.851544+0000","last_undegraded":"2024-09-18T08:10:43.851544+0000","last_fullsized":"2024-09-18T08:10:43.851544+0000","mapping_epoch":194,"log_start":"213'7200","ondisk_log_start":"213'7200","created":165,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:07:52.307205+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:07:52.307205+0000","last_clean_scrub_stamp":"2024-09-18T08:07:52.307205+0000","objects_scrubbed":0,"log_size":4480,"log_dups_size":0,"ondisk_log_size":4480,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:11:28.533710+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":65011712,"num_objects":992,"num_object_clones":0,"num_object_copies":2976,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":992,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11680,"num_write_kb":405504,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2745,"num_bytes_recovered":179896320,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,1],"acting":[3,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"74'79","reported_seq":273,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-09-18T08:10:41.516946+0000","last_change":"2024-09-18T08:08:18.524000+0000","last_active":"2024-09-18T08:10:41.516946+0000","last_peered":"2024-09-18T08:10:41.516946+0000","last_clean":"2024-09-18T08:10:41.516946+0000","last_became_active":"2024-09-18T08:08:18.522977+0000","last_became_peered":"2024-09-18T08:08:18.522977+0000","last_unstale":"2024-09-18T08:10:41.516946+0000","last_undegraded":"2024-09-18T08:10:41.516946+0000","last_fullsized":"2024-09-18T08:10:41.516946+0000","mapping_epoch":181,"log_start":"0'0","ondisk_log_start":"0'0","created":152,"last_epoch_clean":182,"parent":"0.0","parent_split_bits":4,"last_scrub":"74'79","last_scrub_stamp":"2024-09-18T08:08:09.574144+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:00:11.893742+0000","last_clean_scrub_stamp":"2024-09-18T08:08:09.574144+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:11:35.798241+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,4],"acting":[0,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.7","version":"69'13","reported_seq":300,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-09-18T08:10:41.500234+0000","last_change":"2024-09-18T08:10:18.596235+0000","last_active":"2024-09-18T08:10:41.500234+0000","last_peered":"2024-09-18T08:10:41.500234+0000","last_clean":"2024-09-18T08:10:41.500234+0000","last_became_active":"2024-09-18T08:04:18.144925+0000","last_became_peered":"2024-09-18T08:04:18.144925+0000","last_unstale":"2024-09-18T08:10:41.500234+0000","last_undegraded":"2024-09-18T08:10:41.500234+0000","last_fullsized":"2024-09-18T08:10:41.500234+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'13","last_scrub_stamp":"2024-09-18T08:10:18.596150+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_clean_scrub_stamp":"2024-09-18T08:10:18.596150+0000","objects_scrubbed":5,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:11:34.602875+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":29,"num_read_kb":18,"num_write":16,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,0],"acting":[2,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":372,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-09-18T08:10:41.497760+0000","last_change":"2024-09-18T08:08:53.649127+0000","last_active":"2024-09-18T08:10:41.497760+0000","last_peered":"2024-09-18T08:10:41.497760+0000","last_clean":"2024-09-18T08:10:41.497760+0000","last_became_active":"2024-09-18T08:06:24.126334+0000","last_became_peered":"2024-09-18T08:06:24.126334+0000","last_unstale":"2024-09-18T08:10:41.497760+0000","last_undegraded":"2024-09-18T08:10:41.497760+0000","last_fullsized":"2024-09-18T08:10:41.497760+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:08:53.649079+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:46.860092+0000","last_clean_scrub_stamp":"2024-09-18T08:08:53.649079+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:11:28.555391+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"62'1","reported_seq":368,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-09-18T08:10:41.496910+0000","last_change":"2024-09-18T08:10:21.589893+0000","last_active":"2024-09-18T08:10:41.496910+0000","last_peered":"2024-09-18T08:10:41.496910+0000","last_clean":"2024-09-18T08:10:41.496910+0000","last_became_active":"2024-09-18T08:02:46.868175+0000","last_became_peered":"2024-09-18T08:02:46.868175+0000","last_unstale":"2024-09-18T08:10:41.496910+0000","last_undegraded":"2024-09-18T08:10:41.496910+0000","last_fullsized":"2024-09-18T08:10:41.496910+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"62'1","last_scrub_stamp":"2024-09-18T08:10:21.589814+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:45.855979+0000","last_clean_scrub_stamp":"2024-09-18T08:10:21.589814+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:11:27.797703+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0.00066785499999999997,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":66,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-09-18T08:10:41.497171+0000","last_change":"2024-09-18T08:08:52.690656+0000","last_active":"2024-09-18T08:10:41.497171+0000","last_peered":"2024-09-18T08:10:41.497171+0000","last_clean":"2024-09-18T08:10:41.497171+0000","last_became_active":"2024-09-18T08:08:52.690286+0000","last_became_peered":"2024-09-18T08:08:52.690286+0000","last_unstale":"2024-09-18T08:10:41.497171+0000","last_undegraded":"2024-09-18T08:10:41.497171+0000","last_fullsized":"2024-09-18T08:10:41.497171+0000","mapping_epoch":198,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":199,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:07:08.806748+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:50.886830+0000","last_clean_scrub_stamp":"2024-09-18T08:07:08.806748+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:11:28.580473+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":369,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-09-18T08:10:41.497257+0000","last_change":"2024-09-18T08:09:42.647525+0000","last_active":"2024-09-18T08:10:41.497257+0000","last_peered":"2024-09-18T08:10:41.497257+0000","last_clean":"2024-09-18T08:10:41.497257+0000","last_became_active":"2024-09-18T08:06:24.145382+0000","last_became_peered":"2024-09-18T08:06:24.145382+0000","last_unstale":"2024-09-18T08:10:41.497257+0000","last_undegraded":"2024-09-18T08:10:41.497257+0000","last_fullsized":"2024-09-18T08:10:41.497257+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:09:42.647317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_clean_scrub_stamp":"2024-09-18T08:09:42.647317+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:11:03.015274+0000","scrub_duration":26,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.a","version":"226'11676","reported_seq":11810,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-09-18T08:10:43.821061+0000","last_change":"2024-09-18T08:07:53.326651+0000","last_active":"2024-09-18T08:10:43.821061+0000","last_peered":"2024-09-18T08:10:43.821061+0000","last_clean":"2024-09-18T08:10:43.821061+0000","last_became_active":"2024-09-18T08:07:53.326361+0000","last_became_peered":"2024-09-18T08:07:53.326361+0000","last_unstale":"2024-09-18T08:10:43.821061+0000","last_undegraded":"2024-09-18T08:10:43.821061+0000","last_fullsized":"2024-09-18T08:10:43.821061+0000","mapping_epoch":165,"log_start":"213'6800","ondisk_log_start":"213'6800","created":165,"last_epoch_clean":166,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:07:52.307205+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:07:52.307205+0000","last_clean_scrub_stamp":"2024-09-18T08:07:52.307205+0000","objects_scrubbed":0,"log_size":4876,"log_dups_size":0,"ondisk_log_size":4876,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:11:22.601448+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":59506688,"num_objects":908,"num_object_clones":0,"num_object_copies":2724,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":908,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11676,"num_write_kb":402688,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.3","version":"74'79","reported_seq":193,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-09-18T08:10:41.505930+0000","last_change":"2024-09-18T08:09:41.651800+0000","last_active":"2024-09-18T08:10:41.505930+0000","last_peered":"2024-09-18T08:10:41.505930+0000","last_clean":"2024-09-18T08:10:41.505930+0000","last_became_active":"2024-09-18T08:08:18.538929+0000","last_became_peered":"2024-09-18T08:08:18.538929+0000","last_unstale":"2024-09-18T08:10:41.505930+0000","last_undegraded":"2024-09-18T08:10:41.505930+0000","last_fullsized":"2024-09-18T08:10:41.505930+0000","mapping_epoch":181,"log_start":"0'0","ondisk_log_start":"0'0","created":152,"last_epoch_clean":182,"parent":"0.0","parent_split_bits":3,"last_scrub":"74'79","last_scrub_stamp":"2024-09-18T08:09:41.651729+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:00:11.893742+0000","last_clean_scrub_stamp":"2024-09-18T08:09:41.651729+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:10:55.722378+0000","scrub_duration":17,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.6","version":"69'12","reported_seq":414,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-09-18T08:10:41.499363+0000","last_change":"2024-09-18T08:08:57.021693+0000","last_active":"2024-09-18T08:10:41.499363+0000","last_peered":"2024-09-18T08:10:41.499363+0000","last_clean":"2024-09-18T08:10:41.499363+0000","last_became_active":"2024-09-18T08:08:52.700130+0000","last_became_peered":"2024-09-18T08:08:52.700130+0000","last_unstale":"2024-09-18T08:10:41.499363+0000","last_undegraded":"2024-09-18T08:10:41.499363+0000","last_fullsized":"2024-09-18T08:10:41.499363+0000","mapping_epoch":198,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":200,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'12","last_scrub_stamp":"2024-09-18T08:07:39.603562+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_clean_scrub_stamp":"2024-09-18T08:07:39.603562+0000","objects_scrubbed":6,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:10:57.784539+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":18,"num_read_kb":12,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":380,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-09-18T08:10:41.503612+0000","last_change":"2024-09-18T08:09:34.089548+0000","last_active":"2024-09-18T08:10:41.503612+0000","last_peered":"2024-09-18T08:10:41.503612+0000","last_clean":"2024-09-18T08:10:41.503612+0000","last_became_active":"2024-09-18T08:08:18.523135+0000","last_became_peered":"2024-09-18T08:08:18.523135+0000","last_unstale":"2024-09-18T08:10:41.503612+0000","last_undegraded":"2024-09-18T08:10:41.503612+0000","last_fullsized":"2024-09-18T08:10:41.503612+0000","mapping_epoch":181,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":182,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:09:34.089476+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:46.860092+0000","last_clean_scrub_stamp":"2024-09-18T08:09:34.089476+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"70'5","reported_seq":473,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-09-18T08:10:41.911594+0000","last_change":"2024-09-18T08:09:09.649261+0000","last_active":"2024-09-18T08:10:41.911594+0000","last_peered":"2024-09-18T08:10:41.911594+0000","last_clean":"2024-09-18T08:10:41.911594+0000","last_became_active":"2024-09-18T08:06:46.141667+0000","last_became_peered":"2024-09-18T08:06:46.141667+0000","last_unstale":"2024-09-18T08:10:41.911594+0000","last_undegraded":"2024-09-18T08:10:41.911594+0000","last_fullsized":"2024-09-18T08:10:41.911594+0000","mapping_epoch":143,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":144,"parent":"0.0","parent_split_bits":0,"last_scrub":"70'5","last_scrub_stamp":"2024-09-18T08:09:09.649188+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:45.855979+0000","last_clean_scrub_stamp":"2024-09-18T08:09:09.649188+0000","objects_scrubbed":1,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0.000504055,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":243,"num_read_kb":238,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":358,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-09-18T08:10:41.500367+0000","last_change":"2024-09-18T08:07:20.767916+0000","last_active":"2024-09-18T08:10:41.500367+0000","last_peered":"2024-09-18T08:10:41.500367+0000","last_clean":"2024-09-18T08:10:41.500367+0000","last_became_active":"2024-09-18T08:02:51.913112+0000","last_became_peered":"2024-09-18T08:02:51.913112+0000","last_unstale":"2024-09-18T08:10:41.500367+0000","last_undegraded":"2024-09-18T08:10:41.500367+0000","last_fullsized":"2024-09-18T08:10:41.500367+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:07:20.767841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:50.886830+0000","last_clean_scrub_stamp":"2024-09-18T08:07:20.767841+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":339,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-09-18T08:10:41.501265+0000","last_change":"2024-09-18T08:09:49.702132+0000","last_active":"2024-09-18T08:10:41.501265+0000","last_peered":"2024-09-18T08:10:41.501265+0000","last_clean":"2024-09-18T08:10:41.501265+0000","last_became_active":"2024-09-18T08:02:53.907404+0000","last_became_peered":"2024-09-18T08:02:53.907404+0000","last_unstale":"2024-09-18T08:10:41.501265+0000","last_undegraded":"2024-09-18T08:10:41.501265+0000","last_fullsized":"2024-09-18T08:10:41.501265+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:09:49.702055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_clean_scrub_stamp":"2024-09-18T08:09:49.702055+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:11:01.853786+0000","scrub_duration":23,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.9","version":"226'11276","reported_seq":11410,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-09-18T08:10:43.292239+0000","last_change":"2024-09-18T08:07:53.672294+0000","last_active":"2024-09-18T08:10:43.292239+0000","last_peered":"2024-09-18T08:10:43.292239+0000","last_clean":"2024-09-18T08:10:43.292239+0000","last_became_active":"2024-09-18T08:07:53.671994+0000","last_became_peered":"2024-09-18T08:07:53.671994+0000","last_unstale":"2024-09-18T08:10:43.292239+0000","last_undegraded":"2024-09-18T08:10:43.292239+0000","last_fullsized":"2024-09-18T08:10:43.292239+0000","mapping_epoch":165,"log_start":"212'6218","ondisk_log_start":"212'6218","created":165,"last_epoch_clean":166,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:07:52.307205+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:07:52.307205+0000","last_clean_scrub_stamp":"2024-09-18T08:07:52.307205+0000","objects_scrubbed":0,"log_size":5058,"log_dups_size":0,"ondisk_log_size":5058,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":61472768,"num_objects":938,"num_object_clones":0,"num_object_copies":2814,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":938,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11276,"num_write_kb":390848,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.5","version":"182'21","reported_seq":149,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-09-18T08:10:41.489624+0000","last_change":"2024-09-18T08:08:18.609652+0000","last_active":"2024-09-18T08:10:41.489624+0000","last_peered":"2024-09-18T08:10:41.489624+0000","last_clean":"2024-09-18T08:10:41.489624+0000","last_became_active":"2024-09-18T08:08:18.547555+0000","last_became_peered":"2024-09-18T08:08:18.547555+0000","last_unstale":"2024-09-18T08:10:41.489624+0000","last_undegraded":"2024-09-18T08:10:41.489624+0000","last_fullsized":"2024-09-18T08:10:41.489624+0000","mapping_epoch":181,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":182,"parent":"0.0","parent_split_bits":0,"last_scrub":"130'18","last_scrub_stamp":"2024-09-18T08:06:15.455503+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_clean_scrub_stamp":"2024-09-18T08:06:15.455503+0000","objects_scrubbed":0,"log_size":21,"log_dups_size":0,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":24,"num_read_kb":14,"num_write":13,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":154,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,7],"acting":[4,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":373,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-09-18T08:10:41.492669+0000","last_change":"2024-09-18T08:09:45.572278+0000","last_active":"2024-09-18T08:10:41.492669+0000","last_peered":"2024-09-18T08:10:41.492669+0000","last_clean":"2024-09-18T08:10:41.492669+0000","last_became_active":"2024-09-18T08:06:24.133425+0000","last_became_peered":"2024-09-18T08:06:24.133425+0000","last_unstale":"2024-09-18T08:10:41.492669+0000","last_undegraded":"2024-09-18T08:10:41.492669+0000","last_fullsized":"2024-09-18T08:10:41.492669+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:09:45.572201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:46.860092+0000","last_clean_scrub_stamp":"2024-09-18T08:09:45.572201+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:11:00.131734+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"74'79","reported_seq":455,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-09-18T08:10:41.498019+0000","last_change":"2024-09-18T08:08:57.769104+0000","last_active":"2024-09-18T08:10:41.498019+0000","last_peered":"2024-09-18T08:10:41.498019+0000","last_clean":"2024-09-18T08:10:41.498019+0000","last_became_active":"2024-09-18T08:07:24.376931+0000","last_became_peered":"2024-09-18T08:07:24.376931+0000","last_unstale":"2024-09-18T08:10:41.498019+0000","last_undegraded":"2024-09-18T08:10:41.498019+0000","last_fullsized":"2024-09-18T08:10:41.498019+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":153,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'79","last_scrub_stamp":"2024-09-18T08:08:57.769048+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:00:11.893742+0000","last_clean_scrub_stamp":"2024-09-18T08:08:57.769048+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:11:32.632964+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":186,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-09-18T08:10:41.497521+0000","last_change":"2024-09-18T08:07:09.744206+0000","last_active":"2024-09-18T08:10:41.497521+0000","last_peered":"2024-09-18T08:10:41.497521+0000","last_clean":"2024-09-18T08:10:41.497521+0000","last_became_active":"2024-09-18T08:06:46.141046+0000","last_became_peered":"2024-09-18T08:06:46.141046+0000","last_unstale":"2024-09-18T08:10:41.497521+0000","last_undegraded":"2024-09-18T08:10:41.497521+0000","last_fullsized":"2024-09-18T08:10:41.497521+0000","mapping_epoch":143,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":144,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:07:09.744137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:50.886830+0000","last_clean_scrub_stamp":"2024-09-18T08:07:09.744137+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:11:25.618649+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":361,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-09-18T08:10:41.500178+0000","last_change":"2024-09-18T08:08:18.548620+0000","last_active":"2024-09-18T08:10:41.500178+0000","last_peered":"2024-09-18T08:10:41.500178+0000","last_clean":"2024-09-18T08:10:41.500178+0000","last_became_active":"2024-09-18T08:08:18.548233+0000","last_became_peered":"2024-09-18T08:08:18.548233+0000","last_unstale":"2024-09-18T08:10:41.500178+0000","last_undegraded":"2024-09-18T08:10:41.500178+0000","last_fullsized":"2024-09-18T08:10:41.500178+0000","mapping_epoch":181,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":182,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:06:50.710565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_clean_scrub_stamp":"2024-09-18T08:06:50.710565+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.2","version":"226'11584","reported_seq":15995,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-09-18T08:10:43.196298+0000","last_change":"2024-09-18T08:08:58.743574+0000","last_active":"2024-09-18T08:10:43.196298+0000","last_peered":"2024-09-18T08:10:43.196298+0000","last_clean":"2024-09-18T08:10:43.196298+0000","last_became_active":"2024-09-18T08:08:58.738458+0000","last_became_peered":"2024-09-18T08:08:58.738458+0000","last_unstale":"2024-09-18T08:10:43.196298+0000","last_undegraded":"2024-09-18T08:10:43.196298+0000","last_fullsized":"2024-09-18T08:10:43.196298+0000","mapping_epoch":201,"log_start":"213'7100","ondisk_log_start":"213'7100","created":165,"last_epoch_clean":202,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:07:52.307205+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:07:52.307205+0000","last_clean_scrub_stamp":"2024-09-18T08:07:52.307205+0000","objects_scrubbed":0,"log_size":4484,"log_dups_size":0,"ondisk_log_size":4484,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":64749568,"num_objects":988,"num_object_clones":0,"num_object_copies":2964,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":988,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11584,"num_write_kb":402304,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2129,"num_bytes_recovered":139526144,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,5],"acting":[4,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.e","version":"69'11","reported_seq":113,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-09-18T08:10:41.490473+0000","last_change":"2024-09-18T08:08:18.594670+0000","last_active":"2024-09-18T08:10:41.490473+0000","last_peered":"2024-09-18T08:10:41.490473+0000","last_clean":"2024-09-18T08:10:41.490473+0000","last_became_active":"2024-09-18T08:08:18.544581+0000","last_became_peered":"2024-09-18T08:08:18.544581+0000","last_unstale":"2024-09-18T08:10:41.490473+0000","last_undegraded":"2024-09-18T08:10:41.490473+0000","last_fullsized":"2024-09-18T08:10:41.490473+0000","mapping_epoch":181,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":182,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'11","last_scrub_stamp":"2024-09-18T08:06:14.472168+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_clean_scrub_stamp":"2024-09-18T08:06:14.472168+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":19,"num_read_kb":12,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":109,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-09-18T08:10:41.488501+0000","last_change":"2024-09-18T08:10:14.190094+0000","last_active":"2024-09-18T08:10:41.488501+0000","last_peered":"2024-09-18T08:10:41.488501+0000","last_clean":"2024-09-18T08:10:41.488501+0000","last_became_active":"2024-09-18T08:08:18.524229+0000","last_became_peered":"2024-09-18T08:08:18.524229+0000","last_unstale":"2024-09-18T08:10:41.488501+0000","last_undegraded":"2024-09-18T08:10:41.488501+0000","last_fullsized":"2024-09-18T08:10:41.488501+0000","mapping_epoch":181,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":182,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:10:14.190017+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:46.860092+0000","last_clean_scrub_stamp":"2024-09-18T08:10:14.190017+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:11:19.091896+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":378,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-09-18T08:10:41.500021+0000","last_change":"2024-09-18T08:08:18.547880+0000","last_active":"2024-09-18T08:10:41.500021+0000","last_peered":"2024-09-18T08:10:41.500021+0000","last_clean":"2024-09-18T08:10:41.500021+0000","last_became_active":"2024-09-18T08:08:18.547105+0000","last_became_peered":"2024-09-18T08:08:18.547105+0000","last_unstale":"2024-09-18T08:10:41.500021+0000","last_undegraded":"2024-09-18T08:10:41.500021+0000","last_fullsized":"2024-09-18T08:10:41.500021+0000","mapping_epoch":181,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":182,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:07:51.742090+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:50.886830+0000","last_clean_scrub_stamp":"2024-09-18T08:07:51.742090+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":361,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-09-18T08:10:41.494104+0000","last_change":"2024-09-18T08:07:09.197912+0000","last_active":"2024-09-18T08:10:41.494104+0000","last_peered":"2024-09-18T08:10:41.494104+0000","last_clean":"2024-09-18T08:10:41.494104+0000","last_became_active":"2024-09-18T08:06:24.130266+0000","last_became_peered":"2024-09-18T08:06:24.130266+0000","last_unstale":"2024-09-18T08:10:41.494104+0000","last_undegraded":"2024-09-18T08:10:41.494104+0000","last_fullsized":"2024-09-18T08:10:41.494104+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:07:09.197841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_clean_scrub_stamp":"2024-09-18T08:07:09.197841+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:10:58.431705+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.1","version":"226'11713","reported_seq":11912,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-09-18T08:10:43.817388+0000","last_change":"2024-09-18T08:09:40.130643+0000","last_active":"2024-09-18T08:10:43.817388+0000","last_peered":"2024-09-18T08:10:43.817388+0000","last_clean":"2024-09-18T08:10:43.817388+0000","last_became_active":"2024-09-18T08:07:53.327987+0000","last_became_peered":"2024-09-18T08:07:53.327987+0000","last_unstale":"2024-09-18T08:10:43.817388+0000","last_undegraded":"2024-09-18T08:10:43.817388+0000","last_fullsized":"2024-09-18T08:10:43.817388+0000","mapping_epoch":165,"log_start":"213'6800","ondisk_log_start":"213'6800","created":165,"last_epoch_clean":166,"parent":"0.0","parent_split_bits":0,"last_scrub":"213'7038","last_scrub_stamp":"2024-09-18T08:09:40.130353+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:07:52.307205+0000","last_clean_scrub_stamp":"2024-09-18T08:09:40.130353+0000","objects_scrubbed":5709,"log_size":4913,"log_dups_size":0,"ondisk_log_size":4913,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:11:06.670120+0000","scrub_duration":2501,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":63242240,"num_objects":965,"num_object_clones":0,"num_object_copies":2895,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":965,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11713,"num_write_kb":405696,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.8","version":"74'79","reported_seq":178,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-09-18T08:10:41.495363+0000","last_change":"2024-09-18T08:09:00.533593+0000","last_active":"2024-09-18T08:10:41.495363+0000","last_peered":"2024-09-18T08:10:41.495363+0000","last_clean":"2024-09-18T08:10:41.495363+0000","last_became_active":"2024-09-18T08:07:26.407298+0000","last_became_peered":"2024-09-18T08:07:26.407298+0000","last_unstale":"2024-09-18T08:10:41.495363+0000","last_undegraded":"2024-09-18T08:10:41.495363+0000","last_fullsized":"2024-09-18T08:10:41.495363+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":152,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":4,"last_scrub":"74'79","last_scrub_stamp":"2024-09-18T08:09:00.532813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:00:11.893742+0000","last_clean_scrub_stamp":"2024-09-18T08:09:00.532813+0000","objects_scrubbed":1,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":25,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262688,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,3],"acting":[5,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.d","version":"69'17","reported_seq":119,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-09-18T08:10:41.493043+0000","last_change":"2024-09-18T08:08:18.588624+0000","last_active":"2024-09-18T08:10:41.493043+0000","last_peered":"2024-09-18T08:10:41.493043+0000","last_clean":"2024-09-18T08:10:41.493043+0000","last_became_active":"2024-09-18T08:08:18.540862+0000","last_became_peered":"2024-09-18T08:08:18.540862+0000","last_unstale":"2024-09-18T08:10:41.493043+0000","last_undegraded":"2024-09-18T08:10:41.493043+0000","last_fullsized":"2024-09-18T08:10:41.493043+0000","mapping_epoch":181,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":182,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'17","last_scrub_stamp":"2024-09-18T08:08:09.671248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_clean_scrub_stamp":"2024-09-18T08:08:09.671248+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":28,"num_read_kb":18,"num_write":18,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,1],"acting":[4,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":109,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-09-18T08:10:41.488878+0000","last_change":"2024-09-18T08:10:22.243240+0000","last_active":"2024-09-18T08:10:41.488878+0000","last_peered":"2024-09-18T08:10:41.488878+0000","last_clean":"2024-09-18T08:10:41.488878+0000","last_became_active":"2024-09-18T08:08:18.540285+0000","last_became_peered":"2024-09-18T08:08:18.540285+0000","last_unstale":"2024-09-18T08:10:41.488878+0000","last_undegraded":"2024-09-18T08:10:41.488878+0000","last_fullsized":"2024-09-18T08:10:41.488878+0000","mapping_epoch":181,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":182,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:10:22.243148+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:46.860092+0000","last_clean_scrub_stamp":"2024-09-18T08:10:22.243148+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:11:51.567698+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":372,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-09-18T08:10:41.501360+0000","last_change":"2024-09-18T08:09:06.610955+0000","last_active":"2024-09-18T08:10:41.501360+0000","last_peered":"2024-09-18T08:10:41.501360+0000","last_clean":"2024-09-18T08:10:41.501360+0000","last_became_active":"2024-09-18T08:08:18.623014+0000","last_became_peered":"2024-09-18T08:08:18.623014+0000","last_unstale":"2024-09-18T08:10:41.501360+0000","last_undegraded":"2024-09-18T08:10:41.501360+0000","last_fullsized":"2024-09-18T08:10:41.501360+0000","mapping_epoch":181,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":182,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:09:06.610879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:50.886830+0000","last_clean_scrub_stamp":"2024-09-18T08:09:06.610879+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:11:40.561760+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":74,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-09-18T08:10:41.498327+0000","last_change":"2024-09-18T08:10:17.574938+0000","last_active":"2024-09-18T08:10:41.498327+0000","last_peered":"2024-09-18T08:10:41.498327+0000","last_clean":"2024-09-18T08:10:41.498327+0000","last_became_active":"2024-09-18T08:08:52.696300+0000","last_became_peered":"2024-09-18T08:08:52.696300+0000","last_unstale":"2024-09-18T08:10:41.498327+0000","last_undegraded":"2024-09-18T08:10:41.498327+0000","last_fullsized":"2024-09-18T08:10:41.498327+0000","mapping_epoch":198,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":199,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:10:17.574551+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_clean_scrub_stamp":"2024-09-18T08:10:17.574551+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:11:25.310756+0000","scrub_duration":20,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"8.0","version":"226'11584","reported_seq":11718,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-09-18T08:10:43.808097+0000","last_change":"2024-09-18T08:07:53.329182+0000","last_active":"2024-09-18T08:10:43.808097+0000","last_peered":"2024-09-18T08:10:43.808097+0000","last_clean":"2024-09-18T08:10:43.808097+0000","last_became_active":"2024-09-18T08:07:53.328914+0000","last_became_peered":"2024-09-18T08:07:53.328914+0000","last_unstale":"2024-09-18T08:10:43.808097+0000","last_undegraded":"2024-09-18T08:10:43.808097+0000","last_fullsized":"2024-09-18T08:10:43.808097+0000","mapping_epoch":165,"log_start":"213'7100","ondisk_log_start":"213'7100","created":165,"last_epoch_clean":166,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:07:52.307205+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:07:52.307205+0000","last_clean_scrub_stamp":"2024-09-18T08:07:52.307205+0000","objects_scrubbed":0,"log_size":4484,"log_dups_size":0,"ondisk_log_size":4484,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:11:31.524866+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":63307776,"num_objects":966,"num_object_clones":0,"num_object_copies":2898,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":966,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11584,"num_write_kb":401600,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.9","version":"74'79","reported_seq":476,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-09-18T08:10:41.496286+0000","last_change":"2024-09-18T08:09:27.676247+0000","last_active":"2024-09-18T08:10:41.496286+0000","last_peered":"2024-09-18T08:10:41.496286+0000","last_clean":"2024-09-18T08:10:41.496286+0000","last_became_active":"2024-09-18T08:08:52.691492+0000","last_became_peered":"2024-09-18T08:08:52.691492+0000","last_unstale":"2024-09-18T08:10:41.496286+0000","last_undegraded":"2024-09-18T08:10:41.496286+0000","last_fullsized":"2024-09-18T08:10:41.496286+0000","mapping_epoch":198,"log_start":"0'0","ondisk_log_start":"0'0","created":152,"last_epoch_clean":199,"parent":"0.0","parent_split_bits":4,"last_scrub":"74'79","last_scrub_stamp":"2024-09-18T08:09:27.676169+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:00:11.893742+0000","last_clean_scrub_stamp":"2024-09-18T08:09:27.676169+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:11:36.565835+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,2],"acting":[7,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.c","version":"69'10","reported_seq":113,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-09-18T08:10:41.489687+0000","last_change":"2024-09-18T08:08:18.642452+0000","last_active":"2024-09-18T08:10:41.489687+0000","last_peered":"2024-09-18T08:10:41.489687+0000","last_clean":"2024-09-18T08:10:41.489687+0000","last_became_active":"2024-09-18T08:08:18.622980+0000","last_became_peered":"2024-09-18T08:08:18.622980+0000","last_unstale":"2024-09-18T08:10:41.489687+0000","last_undegraded":"2024-09-18T08:10:41.489687+0000","last_fullsized":"2024-09-18T08:10:41.489687+0000","mapping_epoch":181,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":182,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'10","last_scrub_stamp":"2024-09-18T08:05:41.474813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_clean_scrub_stamp":"2024-09-18T08:05:41.474813+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:11:31.237070+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":14,"num_read_kb":9,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":373,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-09-18T08:10:41.501230+0000","last_change":"2024-09-18T08:08:18.542800+0000","last_active":"2024-09-18T08:10:41.501230+0000","last_peered":"2024-09-18T08:10:41.501230+0000","last_clean":"2024-09-18T08:10:41.501230+0000","last_became_active":"2024-09-18T08:08:18.542424+0000","last_became_peered":"2024-09-18T08:08:18.542424+0000","last_unstale":"2024-09-18T08:10:41.501230+0000","last_undegraded":"2024-09-18T08:10:41.501230+0000","last_fullsized":"2024-09-18T08:10:41.501230+0000","mapping_epoch":181,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":182,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:06:46.339428+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:46.860092+0000","last_clean_scrub_stamp":"2024-09-18T08:06:46.339428+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:10:53.446520+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.d","version":"69'8","reported_seq":455,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-09-18T08:10:41.498085+0000","last_change":"2024-09-18T08:05:33.027988+0000","last_active":"2024-09-18T08:10:41.498085+0000","last_peered":"2024-09-18T08:10:41.498085+0000","last_clean":"2024-09-18T08:10:41.498085+0000","last_became_active":"2024-09-18T08:02:51.909011+0000","last_became_peered":"2024-09-18T08:02:51.909011+0000","last_unstale":"2024-09-18T08:10:41.498085+0000","last_undegraded":"2024-09-18T08:10:41.498085+0000","last_fullsized":"2024-09-18T08:10:41.498085+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'8","last_scrub_stamp":"2024-09-18T08:05:33.027783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:50.886830+0000","last_clean_scrub_stamp":"2024-09-18T08:05:33.027783+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:11:22.620829+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":108,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-09-18T08:10:41.490383+0000","last_change":"2024-09-18T08:08:52.685954+0000","last_active":"2024-09-18T08:10:41.490383+0000","last_peered":"2024-09-18T08:10:41.490383+0000","last_clean":"2024-09-18T08:10:41.490383+0000","last_became_active":"2024-09-18T08:08:52.685603+0000","last_became_peered":"2024-09-18T08:08:52.685603+0000","last_unstale":"2024-09-18T08:10:41.490383+0000","last_undegraded":"2024-09-18T08:10:41.490383+0000","last_fullsized":"2024-09-18T08:10:41.490383+0000","mapping_epoch":198,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":199,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:07:16.559597+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_clean_scrub_stamp":"2024-09-18T08:07:16.559597+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.7","version":"226'11614","reported_seq":11817,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-09-18T08:10:43.791105+0000","last_change":"2024-09-18T08:09:36.764450+0000","last_active":"2024-09-18T08:10:43.791105+0000","last_peered":"2024-09-18T08:10:43.791105+0000","last_clean":"2024-09-18T08:10:43.791105+0000","last_became_active":"2024-09-18T08:07:53.674433+0000","last_became_peered":"2024-09-18T08:07:53.674433+0000","last_unstale":"2024-09-18T08:10:43.791105+0000","last_undegraded":"2024-09-18T08:10:43.791105+0000","last_fullsized":"2024-09-18T08:10:43.791105+0000","mapping_epoch":165,"log_start":"213'6700","ondisk_log_start":"213'6700","created":165,"last_epoch_clean":166,"parent":"0.0","parent_split_bits":0,"last_scrub":"213'6848","last_scrub_stamp":"2024-09-18T08:09:36.764206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:07:52.307205+0000","last_clean_scrub_stamp":"2024-09-18T08:09:36.764206+0000","objects_scrubbed":5822,"log_size":4914,"log_dups_size":0,"ondisk_log_size":4914,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":4,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:11:06.120132+0000","scrub_duration":3143,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":62914560,"num_objects":960,"num_object_clones":0,"num_object_copies":2880,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":960,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11614,"num_write_kb":402368,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,6],"acting":[1,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.b","version":"69'9","reported_seq":404,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-09-18T08:10:41.505973+0000","last_change":"2024-09-18T08:08:18.640380+0000","last_active":"2024-09-18T08:10:41.505973+0000","last_peered":"2024-09-18T08:10:41.505973+0000","last_clean":"2024-09-18T08:10:41.505973+0000","last_became_active":"2024-09-18T08:08:18.522995+0000","last_became_peered":"2024-09-18T08:08:18.522995+0000","last_unstale":"2024-09-18T08:10:41.505973+0000","last_undegraded":"2024-09-18T08:10:41.505973+0000","last_fullsized":"2024-09-18T08:10:41.505973+0000","mapping_epoch":181,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":182,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'9","last_scrub_stamp":"2024-09-18T08:08:18.640292+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_clean_scrub_stamp":"2024-09-18T08:08:18.640292+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":372,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-09-18T08:10:41.499314+0000","last_change":"2024-09-18T08:07:40.708363+0000","last_active":"2024-09-18T08:10:41.499314+0000","last_peered":"2024-09-18T08:10:41.499314+0000","last_clean":"2024-09-18T08:10:41.499314+0000","last_became_active":"2024-09-18T08:06:24.135005+0000","last_became_peered":"2024-09-18T08:06:24.135005+0000","last_unstale":"2024-09-18T08:10:41.499314+0000","last_undegraded":"2024-09-18T08:10:41.499314+0000","last_fullsized":"2024-09-18T08:10:41.499314+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:07:40.708307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:46.860092+0000","last_clean_scrub_stamp":"2024-09-18T08:07:40.708307+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":373,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-09-18T08:10:41.510405+0000","last_change":"2024-09-18T08:08:18.509651+0000","last_active":"2024-09-18T08:10:41.510405+0000","last_peered":"2024-09-18T08:10:41.510405+0000","last_clean":"2024-09-18T08:10:41.510405+0000","last_became_active":"2024-09-18T08:08:18.509247+0000","last_became_peered":"2024-09-18T08:08:18.509247+0000","last_unstale":"2024-09-18T08:10:41.510405+0000","last_undegraded":"2024-09-18T08:10:41.510405+0000","last_fullsized":"2024-09-18T08:10:41.510405+0000","mapping_epoch":181,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":182,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:07:10.728679+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:50.886830+0000","last_clean_scrub_stamp":"2024-09-18T08:07:10.728679+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:11:32.611222+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,3],"acting":[2,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":369,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-09-18T08:10:41.503526+0000","last_change":"2024-09-18T08:09:28.086782+0000","last_active":"2024-09-18T08:10:41.503526+0000","last_peered":"2024-09-18T08:10:41.503526+0000","last_clean":"2024-09-18T08:10:41.503526+0000","last_became_active":"2024-09-18T08:08:52.697337+0000","last_became_peered":"2024-09-18T08:08:52.697337+0000","last_unstale":"2024-09-18T08:10:41.503526+0000","last_undegraded":"2024-09-18T08:10:41.503526+0000","last_fullsized":"2024-09-18T08:10:41.503526+0000","mapping_epoch":198,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":199,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:09:28.086727+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_clean_scrub_stamp":"2024-09-18T08:09:28.086727+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.6","version":"226'11562","reported_seq":11737,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-09-18T08:10:43.847704+0000","last_change":"2024-09-18T08:10:12.178469+0000","last_active":"2024-09-18T08:10:43.847704+0000","last_peered":"2024-09-18T08:10:43.847704+0000","last_clean":"2024-09-18T08:10:43.847704+0000","last_became_active":"2024-09-18T08:07:53.674139+0000","last_became_peered":"2024-09-18T08:07:53.674139+0000","last_unstale":"2024-09-18T08:10:43.847704+0000","last_undegraded":"2024-09-18T08:10:43.847704+0000","last_fullsized":"2024-09-18T08:10:43.847704+0000","mapping_epoch":165,"log_start":"210'6200","ondisk_log_start":"210'6200","created":165,"last_epoch_clean":166,"parent":"0.0","parent_split_bits":0,"last_scrub":"217'9226","last_scrub_stamp":"2024-09-18T08:10:12.178143+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:07:52.307205+0000","last_clean_scrub_stamp":"2024-09-18T08:10:12.178143+0000","objects_scrubbed":3346,"log_size":5362,"log_dups_size":0,"ondisk_log_size":5362,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:11:40.476010+0000","scrub_duration":1358,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":63569920,"num_objects":970,"num_object_clones":0,"num_object_copies":2910,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":970,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11562,"num_write_kb":401024,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,7],"acting":[0,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.a","version":"69'19","reported_seq":383,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-09-18T08:10:41.509200+0000","last_change":"2024-09-18T08:10:19.603418+0000","last_active":"2024-09-18T08:10:41.509200+0000","last_peered":"2024-09-18T08:10:41.509200+0000","last_clean":"2024-09-18T08:10:41.509200+0000","last_became_active":"2024-09-18T08:06:24.139909+0000","last_became_peered":"2024-09-18T08:06:24.139909+0000","last_unstale":"2024-09-18T08:10:41.509200+0000","last_undegraded":"2024-09-18T08:10:41.509200+0000","last_fullsized":"2024-09-18T08:10:41.509200+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'19","last_scrub_stamp":"2024-09-18T08:10:19.603091+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_clean_scrub_stamp":"2024-09-18T08:10:19.603091+0000","objects_scrubbed":9,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:11:29.816020+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":31,"num_read_kb":20,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,7],"acting":[1,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":360,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-09-18T08:10:41.499641+0000","last_change":"2024-09-18T08:06:47.638974+0000","last_active":"2024-09-18T08:10:41.499641+0000","last_peered":"2024-09-18T08:10:41.499641+0000","last_clean":"2024-09-18T08:10:41.499641+0000","last_became_active":"2024-09-18T08:06:46.171298+0000","last_became_peered":"2024-09-18T08:06:46.171298+0000","last_unstale":"2024-09-18T08:10:41.499641+0000","last_undegraded":"2024-09-18T08:10:41.499641+0000","last_fullsized":"2024-09-18T08:10:41.499641+0000","mapping_epoch":143,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":144,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:06:47.638911+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:46.860092+0000","last_clean_scrub_stamp":"2024-09-18T08:06:47.638911+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":74,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-09-18T08:10:41.682366+0000","last_change":"2024-09-18T08:10:41.682366+0000","last_active":"2024-09-18T08:10:41.682366+0000","last_peered":"2024-09-18T08:10:41.682366+0000","last_clean":"2024-09-18T08:10:41.682366+0000","last_became_active":"2024-09-18T08:08:52.697932+0000","last_became_peered":"2024-09-18T08:08:52.697932+0000","last_unstale":"2024-09-18T08:10:41.682366+0000","last_undegraded":"2024-09-18T08:10:41.682366+0000","last_fullsized":"2024-09-18T08:10:41.682366+0000","mapping_epoch":198,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":199,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:10:41.682297+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:50.886830+0000","last_clean_scrub_stamp":"2024-09-18T08:10:41.682297+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:11:53.645609+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":186,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-09-18T08:10:41.497168+0000","last_change":"2024-09-18T08:06:50.902917+0000","last_active":"2024-09-18T08:10:41.497168+0000","last_peered":"2024-09-18T08:10:41.497168+0000","last_clean":"2024-09-18T08:10:41.497168+0000","last_became_active":"2024-09-18T08:06:46.137872+0000","last_became_peered":"2024-09-18T08:06:46.137872+0000","last_unstale":"2024-09-18T08:10:41.497168+0000","last_undegraded":"2024-09-18T08:10:41.497168+0000","last_fullsized":"2024-09-18T08:10:41.497168+0000","mapping_epoch":143,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":144,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:06:50.902835+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_clean_scrub_stamp":"2024-09-18T08:06:50.902835+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:11:24.595186+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,3],"acting":[2,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"8.5","version":"226'11541","reported_seq":15988,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-09-18T08:10:43.214535+0000","last_change":"2024-09-18T08:09:13.170209+0000","last_active":"2024-09-18T08:10:43.214535+0000","last_peered":"2024-09-18T08:10:43.214535+0000","last_clean":"2024-09-18T08:10:43.214535+0000","last_became_active":"2024-09-18T08:09:13.166459+0000","last_became_peered":"2024-09-18T08:09:13.166459+0000","last_unstale":"2024-09-18T08:10:43.214535+0000","last_undegraded":"2024-09-18T08:10:43.214535+0000","last_fullsized":"2024-09-18T08:10:43.214535+0000","mapping_epoch":205,"log_start":"213'6974","ondisk_log_start":"213'6974","created":165,"last_epoch_clean":206,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:07:52.307205+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:07:52.307205+0000","last_clean_scrub_stamp":"2024-09-18T08:07:52.307205+0000","objects_scrubbed":0,"log_size":4567,"log_dups_size":0,"ondisk_log_size":4567,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68747264,"num_objects":1049,"num_object_clones":0,"num_object_copies":3147,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1049,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11541,"num_write_kb":402880,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2146,"num_bytes_recovered":140640256,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,7],"acting":[4,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.9","version":"69'12","reported_seq":132,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-09-18T08:10:41.488115+0000","last_change":"2024-09-18T08:10:18.373828+0000","last_active":"2024-09-18T08:10:41.488115+0000","last_peered":"2024-09-18T08:10:41.488115+0000","last_clean":"2024-09-18T08:10:41.488115+0000","last_became_active":"2024-09-18T08:10:18.362984+0000","last_became_peered":"2024-09-18T08:10:18.362984+0000","last_unstale":"2024-09-18T08:10:41.488115+0000","last_undegraded":"2024-09-18T08:10:41.488115+0000","last_fullsized":"2024-09-18T08:10:41.488115+0000","mapping_epoch":221,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":222,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'12","last_scrub_stamp":"2024-09-18T08:05:33.544135+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_clean_scrub_stamp":"2024-09-18T08:05:33.544135+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:11:26.262794+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":24,"num_read_kb":15,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":660,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,3],"acting":[4,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":388,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-09-18T08:10:41.499629+0000","last_change":"2024-09-18T08:09:39.614251+0000","last_active":"2024-09-18T08:10:41.499629+0000","last_peered":"2024-09-18T08:10:41.499629+0000","last_clean":"2024-09-18T08:10:41.499629+0000","last_became_active":"2024-09-18T08:08:18.539573+0000","last_became_peered":"2024-09-18T08:08:18.539573+0000","last_unstale":"2024-09-18T08:10:41.499629+0000","last_undegraded":"2024-09-18T08:10:41.499629+0000","last_fullsized":"2024-09-18T08:10:41.499629+0000","mapping_epoch":181,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":182,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:09:39.614187+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:46.860092+0000","last_clean_scrub_stamp":"2024-09-18T08:09:39.614187+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:10:57.546531+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":351,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-09-18T08:10:41.497318+0000","last_change":"2024-09-18T08:08:15.744330+0000","last_active":"2024-09-18T08:10:41.497318+0000","last_peered":"2024-09-18T08:10:41.497318+0000","last_clean":"2024-09-18T08:10:41.497318+0000","last_became_active":"2024-09-18T08:02:51.902163+0000","last_became_peered":"2024-09-18T08:02:51.902163+0000","last_unstale":"2024-09-18T08:10:41.497318+0000","last_undegraded":"2024-09-18T08:10:41.497318+0000","last_fullsized":"2024-09-18T08:10:41.497318+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:08:15.744245+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:50.886830+0000","last_clean_scrub_stamp":"2024-09-18T08:08:15.744245+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":74,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":356,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-09-18T08:10:41.500900+0000","last_change":"2024-09-18T08:10:17.494803+0000","last_active":"2024-09-18T08:10:41.500900+0000","last_peered":"2024-09-18T08:10:41.500900+0000","last_clean":"2024-09-18T08:10:41.500900+0000","last_became_active":"2024-09-18T08:02:53.907412+0000","last_became_peered":"2024-09-18T08:02:53.907412+0000","last_unstale":"2024-09-18T08:10:41.500900+0000","last_undegraded":"2024-09-18T08:10:41.500900+0000","last_fullsized":"2024-09-18T08:10:41.500900+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:10:17.494741+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_clean_scrub_stamp":"2024-09-18T08:10:17.494741+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:11:19.219802+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.4","version":"226'11561","reported_seq":25943,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-09-18T08:10:43.307562+0000","last_change":"2024-09-18T08:10:15.264328+0000","last_active":"2024-09-18T08:10:43.307562+0000","last_peered":"2024-09-18T08:10:43.307562+0000","last_clean":"2024-09-18T08:10:43.307562+0000","last_became_active":"2024-09-18T08:10:15.246282+0000","last_became_peered":"2024-09-18T08:10:15.246282+0000","last_unstale":"2024-09-18T08:10:43.307562+0000","last_undegraded":"2024-09-18T08:10:43.307562+0000","last_fullsized":"2024-09-18T08:10:43.307562+0000","mapping_epoch":219,"log_start":"212'6468","ondisk_log_start":"212'6468","created":165,"last_epoch_clean":220,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:07:52.307205+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:07:52.307205+0000","last_clean_scrub_stamp":"2024-09-18T08:07:52.307205+0000","objects_scrubbed":0,"log_size":5093,"log_dups_size":0,"ondisk_log_size":5093,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":66125824,"num_objects":1009,"num_object_clones":0,"num_object_copies":3027,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1009,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11561,"num_write_kb":402240,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7107,"num_bytes_recovered":465764352,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,2],"acting":[5,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.8","version":"69'15","reported_seq":417,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-09-18T08:10:41.495669+0000","last_change":"2024-09-18T08:06:24.162625+0000","last_active":"2024-09-18T08:10:41.495669+0000","last_peered":"2024-09-18T08:10:41.495669+0000","last_clean":"2024-09-18T08:10:41.495669+0000","last_became_active":"2024-09-18T08:06:24.138259+0000","last_became_peered":"2024-09-18T08:06:24.138259+0000","last_unstale":"2024-09-18T08:10:41.495669+0000","last_undegraded":"2024-09-18T08:10:41.495669+0000","last_fullsized":"2024-09-18T08:10:41.495669+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'15","last_scrub_stamp":"2024-09-18T08:04:26.859044+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_clean_scrub_stamp":"2024-09-18T08:04:26.859044+0000","objects_scrubbed":7,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:11:32.049514+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.f","version":"62'2","reported_seq":404,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-09-18T08:10:41.496888+0000","last_change":"2024-09-18T08:08:18.629506+0000","last_active":"2024-09-18T08:10:41.496888+0000","last_peered":"2024-09-18T08:10:41.496888+0000","last_clean":"2024-09-18T08:10:41.496888+0000","last_became_active":"2024-09-18T08:08:18.524593+0000","last_became_peered":"2024-09-18T08:08:18.524593+0000","last_unstale":"2024-09-18T08:10:41.496888+0000","last_undegraded":"2024-09-18T08:10:41.496888+0000","last_fullsized":"2024-09-18T08:10:41.496888+0000","mapping_epoch":181,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":182,"parent":"0.0","parent_split_bits":0,"last_scrub":"62'2","last_scrub_stamp":"2024-09-18T08:07:43.540222+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:46.860092+0000","last_clean_scrub_stamp":"2024-09-18T08:07:43.540222+0000","objects_scrubbed":2,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:10:52.700103+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":4,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":184,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.9","version":"182'12","reported_seq":518,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-09-18T08:10:41.496232+0000","last_change":"2024-09-18T08:09:46.742515+0000","last_active":"2024-09-18T08:10:41.496232+0000","last_peered":"2024-09-18T08:10:41.496232+0000","last_clean":"2024-09-18T08:10:41.496232+0000","last_became_active":"2024-09-18T08:08:18.524858+0000","last_became_peered":"2024-09-18T08:08:18.524858+0000","last_unstale":"2024-09-18T08:10:41.496232+0000","last_undegraded":"2024-09-18T08:10:41.496232+0000","last_fullsized":"2024-09-18T08:10:41.496232+0000","mapping_epoch":181,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":182,"parent":"0.0","parent_split_bits":0,"last_scrub":"182'12","last_scrub_stamp":"2024-09-18T08:09:46.742442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:50.886830+0000","last_clean_scrub_stamp":"2024-09-18T08:09:46.742442+0000","objects_scrubbed":1,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:11:07.487575+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":360,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-09-18T08:10:41.500498+0000","last_change":"2024-09-18T08:07:45.707631+0000","last_active":"2024-09-18T08:10:41.500498+0000","last_peered":"2024-09-18T08:10:41.500498+0000","last_clean":"2024-09-18T08:10:41.500498+0000","last_became_active":"2024-09-18T08:06:24.133119+0000","last_became_peered":"2024-09-18T08:06:24.133119+0000","last_unstale":"2024-09-18T08:10:41.500498+0000","last_undegraded":"2024-09-18T08:10:41.500498+0000","last_fullsized":"2024-09-18T08:10:41.500498+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:07:45.707562+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_clean_scrub_stamp":"2024-09-18T08:07:45.707562+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":276,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-09-18T08:10:41.499925+0000","last_change":"2024-09-18T08:06:24.133955+0000","last_active":"2024-09-18T08:10:41.499925+0000","last_peered":"2024-09-18T08:10:41.499925+0000","last_clean":"2024-09-18T08:10:41.499925+0000","last_became_active":"2024-09-18T08:06:24.133644+0000","last_became_peered":"2024-09-18T08:06:24.133644+0000","last_unstale":"2024-09-18T08:10:41.499925+0000","last_undegraded":"2024-09-18T08:10:41.499925+0000","last_fullsized":"2024-09-18T08:10:41.499925+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:04:16.378205+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:46.860092+0000","last_clean_scrub_stamp":"2024-09-18T08:04:16.378205+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:11:22.917471+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.17","version":"69'6","reported_seq":380,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-09-18T08:10:41.512130+0000","last_change":"2024-09-18T08:10:15.516230+0000","last_active":"2024-09-18T08:10:41.512130+0000","last_peered":"2024-09-18T08:10:41.512130+0000","last_clean":"2024-09-18T08:10:41.512130+0000","last_became_active":"2024-09-18T08:02:49.895067+0000","last_became_peered":"2024-09-18T08:02:49.895067+0000","last_unstale":"2024-09-18T08:10:41.512130+0000","last_undegraded":"2024-09-18T08:10:41.512130+0000","last_fullsized":"2024-09-18T08:10:41.512130+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'6","last_scrub_stamp":"2024-09-18T08:10:15.515914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_clean_scrub_stamp":"2024-09-18T08:10:15.515914+0000","objects_scrubbed":3,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:11:26.692514+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":9,"num_read_kb":6,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":350,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-09-18T08:10:41.499881+0000","last_change":"2024-09-18T08:05:45.812436+0000","last_active":"2024-09-18T08:10:41.499881+0000","last_peered":"2024-09-18T08:10:41.499881+0000","last_clean":"2024-09-18T08:10:41.499881+0000","last_became_active":"2024-09-18T08:02:51.912557+0000","last_became_peered":"2024-09-18T08:02:51.912557+0000","last_unstale":"2024-09-18T08:10:41.499881+0000","last_undegraded":"2024-09-18T08:10:41.499881+0000","last_fullsized":"2024-09-18T08:10:41.499881+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:05:45.812389+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:50.886830+0000","last_clean_scrub_stamp":"2024-09-18T08:05:45.812389+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,1],"acting":[5,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":383,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-09-18T08:10:41.497176+0000","last_change":"2024-09-18T08:08:18.526113+0000","last_active":"2024-09-18T08:10:41.497176+0000","last_peered":"2024-09-18T08:10:41.497176+0000","last_clean":"2024-09-18T08:10:41.497176+0000","last_became_active":"2024-09-18T08:08:18.525260+0000","last_became_peered":"2024-09-18T08:08:18.525260+0000","last_unstale":"2024-09-18T08:10:41.497176+0000","last_undegraded":"2024-09-18T08:10:41.497176+0000","last_fullsized":"2024-09-18T08:10:41.497176+0000","mapping_epoch":181,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":182,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:07:53.670107+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_clean_scrub_stamp":"2024-09-18T08:07:53.670107+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:11:22.750433+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"69'9","reported_seq":378,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-09-18T08:10:41.499462+0000","last_change":"2024-09-18T08:09:44.116395+0000","last_active":"2024-09-18T08:10:41.499462+0000","last_peered":"2024-09-18T08:10:41.499462+0000","last_clean":"2024-09-18T08:10:41.499462+0000","last_became_active":"2024-09-18T08:02:49.891567+0000","last_became_peered":"2024-09-18T08:02:49.891567+0000","last_unstale":"2024-09-18T08:10:41.499462+0000","last_undegraded":"2024-09-18T08:10:41.499462+0000","last_fullsized":"2024-09-18T08:10:41.499462+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'9","last_scrub_stamp":"2024-09-18T08:09:44.116308+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_clean_scrub_stamp":"2024-09-18T08:09:44.116308+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:11:07.916395+0000","scrub_duration":43,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":109,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-09-18T08:10:42.301469+0000","last_change":"2024-09-18T08:10:42.301469+0000","last_active":"2024-09-18T08:10:42.301469+0000","last_peered":"2024-09-18T08:10:42.301469+0000","last_clean":"2024-09-18T08:10:42.301469+0000","last_became_active":"2024-09-18T08:08:18.545953+0000","last_became_peered":"2024-09-18T08:08:18.545953+0000","last_unstale":"2024-09-18T08:10:42.301469+0000","last_undegraded":"2024-09-18T08:10:42.301469+0000","last_fullsized":"2024-09-18T08:10:42.301469+0000","mapping_epoch":181,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":182,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:10:42.301416+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:46.860092+0000","last_clean_scrub_stamp":"2024-09-18T08:10:42.301416+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:11:44.984541+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":351,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-09-18T08:10:41.511893+0000","last_change":"2024-09-18T08:09:46.560057+0000","last_active":"2024-09-18T08:10:41.511893+0000","last_peered":"2024-09-18T08:10:41.511893+0000","last_clean":"2024-09-18T08:10:41.511893+0000","last_became_active":"2024-09-18T08:02:51.907361+0000","last_became_peered":"2024-09-18T08:02:51.907361+0000","last_unstale":"2024-09-18T08:10:41.511893+0000","last_undegraded":"2024-09-18T08:10:41.511893+0000","last_fullsized":"2024-09-18T08:10:41.511893+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:09:46.557349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:50.886830+0000","last_clean_scrub_stamp":"2024-09-18T08:09:46.557349+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:11:08.318311+0000","scrub_duration":22,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":370,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-09-18T08:10:42.731922+0000","last_change":"2024-09-18T08:10:42.731922+0000","last_active":"2024-09-18T08:10:42.731922+0000","last_peered":"2024-09-18T08:10:42.731922+0000","last_clean":"2024-09-18T08:10:42.731922+0000","last_became_active":"2024-09-18T08:08:18.511019+0000","last_became_peered":"2024-09-18T08:08:18.511019+0000","last_unstale":"2024-09-18T08:10:42.731922+0000","last_undegraded":"2024-09-18T08:10:42.731922+0000","last_fullsized":"2024-09-18T08:10:42.731922+0000","mapping_epoch":181,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":182,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:10:42.731845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_clean_scrub_stamp":"2024-09-18T08:10:42.731845+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:11:54.306475+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.15","version":"69'9","reported_seq":370,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-09-18T08:10:41.499799+0000","last_change":"2024-09-18T08:05:36.896299+0000","last_active":"2024-09-18T08:10:41.499799+0000","last_peered":"2024-09-18T08:10:41.499799+0000","last_clean":"2024-09-18T08:10:41.499799+0000","last_became_active":"2024-09-18T08:02:49.890497+0000","last_became_peered":"2024-09-18T08:02:49.890497+0000","last_unstale":"2024-09-18T08:10:41.499799+0000","last_undegraded":"2024-09-18T08:10:41.499799+0000","last_fullsized":"2024-09-18T08:10:41.499799+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'9","last_scrub_stamp":"2024-09-18T08:05:36.896238+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_clean_scrub_stamp":"2024-09-18T08:05:36.896238+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:11:41.088262+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":374,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-09-18T08:10:41.517239+0000","last_change":"2024-09-18T08:09:46.035501+0000","last_active":"2024-09-18T08:10:41.517239+0000","last_peered":"2024-09-18T08:10:41.517239+0000","last_clean":"2024-09-18T08:10:41.517239+0000","last_became_active":"2024-09-18T08:02:47.875983+0000","last_became_peered":"2024-09-18T08:02:47.875983+0000","last_unstale":"2024-09-18T08:10:41.517239+0000","last_undegraded":"2024-09-18T08:10:41.517239+0000","last_fullsized":"2024-09-18T08:10:41.517239+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:09:46.035332+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:46.860092+0000","last_clean_scrub_stamp":"2024-09-18T08:09:46.035332+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:10:47.519129+0000","scrub_duration":18,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.14","version":"199'12","reported_seq":487,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-09-18T08:10:41.494300+0000","last_change":"2024-09-18T08:10:10.449130+0000","last_active":"2024-09-18T08:10:41.494300+0000","last_peered":"2024-09-18T08:10:41.494300+0000","last_clean":"2024-09-18T08:10:41.494300+0000","last_became_active":"2024-09-18T08:08:52.696201+0000","last_became_peered":"2024-09-18T08:08:52.696201+0000","last_unstale":"2024-09-18T08:10:41.494300+0000","last_undegraded":"2024-09-18T08:10:41.494300+0000","last_fullsized":"2024-09-18T08:10:41.494300+0000","mapping_epoch":198,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":199,"parent":"0.0","parent_split_bits":0,"last_scrub":"199'12","last_scrub_stamp":"2024-09-18T08:10:10.449024+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:50.886830+0000","last_clean_scrub_stamp":"2024-09-18T08:10:10.449024+0000","objects_scrubbed":1,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:11:36.934868+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":101,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-09-18T08:10:41.490502+0000","last_change":"2024-09-18T08:08:18.523205+0000","last_active":"2024-09-18T08:10:41.490502+0000","last_peered":"2024-09-18T08:10:41.490502+0000","last_clean":"2024-09-18T08:10:41.490502+0000","last_became_active":"2024-09-18T08:08:18.522483+0000","last_became_peered":"2024-09-18T08:08:18.522483+0000","last_unstale":"2024-09-18T08:10:41.490502+0000","last_undegraded":"2024-09-18T08:10:41.490502+0000","last_fullsized":"2024-09-18T08:10:41.490502+0000","mapping_epoch":181,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":182,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:06:43.510513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_clean_scrub_stamp":"2024-09-18T08:06:43.510513+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.14","version":"69'10","reported_seq":389,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-09-18T08:10:41.494605+0000","last_change":"2024-09-18T08:08:08.493614+0000","last_active":"2024-09-18T08:10:41.494605+0000","last_peered":"2024-09-18T08:10:41.494605+0000","last_clean":"2024-09-18T08:10:41.494605+0000","last_became_active":"2024-09-18T08:04:18.150968+0000","last_became_peered":"2024-09-18T08:04:18.150968+0000","last_unstale":"2024-09-18T08:10:41.494605+0000","last_undegraded":"2024-09-18T08:10:41.494605+0000","last_fullsized":"2024-09-18T08:10:41.494605+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'10","last_scrub_stamp":"2024-09-18T08:08:08.493522+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_clean_scrub_stamp":"2024-09-18T08:08:08.493522+0000","objects_scrubbed":5,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:11:33.537649+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,7],"acting":[3,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":380,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-09-18T08:10:41.496474+0000","last_change":"2024-09-18T08:09:31.658337+0000","last_active":"2024-09-18T08:10:41.496474+0000","last_peered":"2024-09-18T08:10:41.496474+0000","last_clean":"2024-09-18T08:10:41.496474+0000","last_became_active":"2024-09-18T08:08:18.524218+0000","last_became_peered":"2024-09-18T08:08:18.524218+0000","last_unstale":"2024-09-18T08:10:41.496474+0000","last_undegraded":"2024-09-18T08:10:41.496474+0000","last_fullsized":"2024-09-18T08:10:41.496474+0000","mapping_epoch":181,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":182,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:09:31.658259+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:46.860092+0000","last_clean_scrub_stamp":"2024-09-18T08:09:31.658259+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:10:50.143994+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.15","version":"69'8","reported_seq":462,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-09-18T08:10:41.499524+0000","last_change":"2024-09-18T08:06:45.555513+0000","last_active":"2024-09-18T08:10:41.499524+0000","last_peered":"2024-09-18T08:10:41.499524+0000","last_clean":"2024-09-18T08:10:41.499524+0000","last_became_active":"2024-09-18T08:02:51.914135+0000","last_became_peered":"2024-09-18T08:02:51.914135+0000","last_unstale":"2024-09-18T08:10:41.499524+0000","last_undegraded":"2024-09-18T08:10:41.499524+0000","last_fullsized":"2024-09-18T08:10:41.499524+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'8","last_scrub_stamp":"2024-09-18T08:06:45.555459+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:50.886830+0000","last_clean_scrub_stamp":"2024-09-18T08:06:45.555459+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":355,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-09-18T08:10:41.498873+0000","last_change":"2024-09-18T08:09:42.047337+0000","last_active":"2024-09-18T08:10:41.498873+0000","last_peered":"2024-09-18T08:10:41.498873+0000","last_clean":"2024-09-18T08:10:41.498873+0000","last_became_active":"2024-09-18T08:02:53.912112+0000","last_became_peered":"2024-09-18T08:02:53.912112+0000","last_unstale":"2024-09-18T08:10:41.498873+0000","last_undegraded":"2024-09-18T08:10:41.498873+0000","last_fullsized":"2024-09-18T08:10:41.498873+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:09:42.047270+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_clean_scrub_stamp":"2024-09-18T08:09:42.047270+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:10:59.660034+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.13","version":"69'11","reported_seq":113,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-09-18T08:10:41.488225+0000","last_change":"2024-09-18T08:08:18.643772+0000","last_active":"2024-09-18T08:10:41.488225+0000","last_peered":"2024-09-18T08:10:41.488225+0000","last_clean":"2024-09-18T08:10:41.488225+0000","last_became_active":"2024-09-18T08:08:18.539712+0000","last_became_peered":"2024-09-18T08:08:18.539712+0000","last_unstale":"2024-09-18T08:10:41.488225+0000","last_undegraded":"2024-09-18T08:10:41.488225+0000","last_fullsized":"2024-09-18T08:10:41.488225+0000","mapping_epoch":181,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":182,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'11","last_scrub_stamp":"2024-09-18T08:07:53.824401+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_clean_scrub_stamp":"2024-09-18T08:07:53.824401+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":19,"num_read_kb":12,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":101,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-09-18T08:10:41.489905+0000","last_change":"2024-09-18T08:08:18.623643+0000","last_active":"2024-09-18T08:10:41.489905+0000","last_peered":"2024-09-18T08:10:41.489905+0000","last_clean":"2024-09-18T08:10:41.489905+0000","last_became_active":"2024-09-18T08:08:18.622893+0000","last_became_peered":"2024-09-18T08:08:18.622893+0000","last_unstale":"2024-09-18T08:10:41.489905+0000","last_undegraded":"2024-09-18T08:10:41.489905+0000","last_fullsized":"2024-09-18T08:10:41.489905+0000","mapping_epoch":181,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":182,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:06:46.502422+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:46.860092+0000","last_clean_scrub_stamp":"2024-09-18T08:06:46.502422+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":358,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-09-18T08:10:41.509122+0000","last_change":"2024-09-18T08:09:03.582984+0000","last_active":"2024-09-18T08:10:41.509122+0000","last_peered":"2024-09-18T08:10:41.509122+0000","last_clean":"2024-09-18T08:10:41.509122+0000","last_became_active":"2024-09-18T08:02:51.910752+0000","last_became_peered":"2024-09-18T08:02:51.910752+0000","last_unstale":"2024-09-18T08:10:41.509122+0000","last_undegraded":"2024-09-18T08:10:41.509122+0000","last_fullsized":"2024-09-18T08:10:41.509122+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:09:03.582914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:50.886830+0000","last_clean_scrub_stamp":"2024-09-18T08:09:03.582914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:11:24.544180+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":364,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-09-18T08:10:41.491882+0000","last_change":"2024-09-18T08:10:20.525295+0000","last_active":"2024-09-18T08:10:41.491882+0000","last_peered":"2024-09-18T08:10:41.491882+0000","last_clean":"2024-09-18T08:10:41.491882+0000","last_became_active":"2024-09-18T08:02:53.911429+0000","last_became_peered":"2024-09-18T08:02:53.911429+0000","last_unstale":"2024-09-18T08:10:41.491882+0000","last_undegraded":"2024-09-18T08:10:41.491882+0000","last_fullsized":"2024-09-18T08:10:41.491882+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:10:20.525222+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_clean_scrub_stamp":"2024-09-18T08:10:20.525222+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:11:38.979849+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.12","version":"69'9","reported_seq":404,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-09-18T08:10:41.509362+0000","last_change":"2024-09-18T08:10:06.582865+0000","last_active":"2024-09-18T08:10:41.509362+0000","last_peered":"2024-09-18T08:10:41.509362+0000","last_clean":"2024-09-18T08:10:41.509362+0000","last_became_active":"2024-09-18T08:06:24.132270+0000","last_became_peered":"2024-09-18T08:06:24.132270+0000","last_unstale":"2024-09-18T08:10:41.509362+0000","last_undegraded":"2024-09-18T08:10:41.509362+0000","last_fullsized":"2024-09-18T08:10:41.509362+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'9","last_scrub_stamp":"2024-09-18T08:10:06.582804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_clean_scrub_stamp":"2024-09-18T08:10:06.582804+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:11:13.834917+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":388,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-09-18T08:10:41.497429+0000","last_change":"2024-09-18T08:09:37.532792+0000","last_active":"2024-09-18T08:10:41.497429+0000","last_peered":"2024-09-18T08:10:41.497429+0000","last_clean":"2024-09-18T08:10:41.497429+0000","last_became_active":"2024-09-18T08:08:18.525527+0000","last_became_peered":"2024-09-18T08:08:18.525527+0000","last_unstale":"2024-09-18T08:10:41.497429+0000","last_undegraded":"2024-09-18T08:10:41.497429+0000","last_fullsized":"2024-09-18T08:10:41.497429+0000","mapping_epoch":181,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":182,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:09:37.532573+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:46.860092+0000","last_clean_scrub_stamp":"2024-09-18T08:09:37.532573+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:11:05.489513+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":351,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-09-18T08:10:41.501000+0000","last_change":"2024-09-18T08:07:42.165434+0000","last_active":"2024-09-18T08:10:41.501000+0000","last_peered":"2024-09-18T08:10:41.501000+0000","last_clean":"2024-09-18T08:10:41.501000+0000","last_became_active":"2024-09-18T08:02:51.906277+0000","last_became_peered":"2024-09-18T08:02:51.906277+0000","last_unstale":"2024-09-18T08:10:41.501000+0000","last_undegraded":"2024-09-18T08:10:41.501000+0000","last_fullsized":"2024-09-18T08:10:41.501000+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:07:42.165388+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:50.886830+0000","last_clean_scrub_stamp":"2024-09-18T08:07:42.165388+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:11:26.454454+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.10","version":"69'1","reported_seq":343,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-09-18T08:10:41.499516+0000","last_change":"2024-09-18T08:10:17.879036+0000","last_active":"2024-09-18T08:10:41.499516+0000","last_peered":"2024-09-18T08:10:41.499516+0000","last_clean":"2024-09-18T08:10:41.499516+0000","last_became_active":"2024-09-18T08:02:53.912699+0000","last_became_peered":"2024-09-18T08:02:53.912699+0000","last_unstale":"2024-09-18T08:10:41.499516+0000","last_undegraded":"2024-09-18T08:10:41.499516+0000","last_fullsized":"2024-09-18T08:10:41.499516+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'1","last_scrub_stamp":"2024-09-18T08:10:17.878965+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_clean_scrub_stamp":"2024-09-18T08:10:17.878965+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:11:23.854550+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.11","version":"69'11","reported_seq":428,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-09-18T08:10:41.492072+0000","last_change":"2024-09-18T08:10:16.510963+0000","last_active":"2024-09-18T08:10:41.492072+0000","last_peered":"2024-09-18T08:10:41.492072+0000","last_clean":"2024-09-18T08:10:41.492072+0000","last_became_active":"2024-09-18T08:06:24.132416+0000","last_became_peered":"2024-09-18T08:06:24.132416+0000","last_unstale":"2024-09-18T08:10:41.492072+0000","last_undegraded":"2024-09-18T08:10:41.492072+0000","last_fullsized":"2024-09-18T08:10:41.492072+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'11","last_scrub_stamp":"2024-09-18T08:10:16.510871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_clean_scrub_stamp":"2024-09-18T08:10:16.510871+0000","objects_scrubbed":5,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:11:20.303978+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":358,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-09-18T08:10:41.499992+0000","last_change":"2024-09-18T08:06:18.168254+0000","last_active":"2024-09-18T08:10:41.499992+0000","last_peered":"2024-09-18T08:10:41.499992+0000","last_clean":"2024-09-18T08:10:41.499992+0000","last_became_active":"2024-09-18T08:02:47.878886+0000","last_became_peered":"2024-09-18T08:02:47.878886+0000","last_unstale":"2024-09-18T08:10:41.499992+0000","last_undegraded":"2024-09-18T08:10:41.499992+0000","last_fullsized":"2024-09-18T08:10:41.499992+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:06:18.168206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:46.860092+0000","last_clean_scrub_stamp":"2024-09-18T08:06:18.168206+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":394,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-09-18T08:10:41.497537+0000","last_change":"2024-09-18T08:09:49.824264+0000","last_active":"2024-09-18T08:10:41.497537+0000","last_peered":"2024-09-18T08:10:41.497537+0000","last_clean":"2024-09-18T08:10:41.497537+0000","last_became_active":"2024-09-18T08:08:18.524966+0000","last_became_peered":"2024-09-18T08:08:18.524966+0000","last_unstale":"2024-09-18T08:10:41.497537+0000","last_undegraded":"2024-09-18T08:10:41.497537+0000","last_fullsized":"2024-09-18T08:10:41.497537+0000","mapping_epoch":181,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":182,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:09:49.824196+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:50.886830+0000","last_clean_scrub_stamp":"2024-09-18T08:09:49.824196+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:10:51.928894+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":361,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-09-18T08:10:41.501151+0000","last_change":"2024-09-18T08:07:48.218010+0000","last_active":"2024-09-18T08:10:41.501151+0000","last_peered":"2024-09-18T08:10:41.501151+0000","last_clean":"2024-09-18T08:10:41.501151+0000","last_became_active":"2024-09-18T08:06:24.128582+0000","last_became_peered":"2024-09-18T08:06:24.128582+0000","last_unstale":"2024-09-18T08:10:41.501151+0000","last_undegraded":"2024-09-18T08:10:41.501151+0000","last_fullsized":"2024-09-18T08:10:41.501151+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:07:48.217964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_clean_scrub_stamp":"2024-09-18T08:07:48.217964+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:11:24.500466+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.10","version":"69'4","reported_seq":395,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-09-18T08:10:41.498452+0000","last_change":"2024-09-18T08:10:18.543267+0000","last_active":"2024-09-18T08:10:41.498452+0000","last_peered":"2024-09-18T08:10:41.498452+0000","last_clean":"2024-09-18T08:10:41.498452+0000","last_became_active":"2024-09-18T08:06:24.131309+0000","last_became_peered":"2024-09-18T08:06:24.131309+0000","last_unstale":"2024-09-18T08:10:41.498452+0000","last_undegraded":"2024-09-18T08:10:41.498452+0000","last_fullsized":"2024-09-18T08:10:41.498452+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'4","last_scrub_stamp":"2024-09-18T08:10:18.542984+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_clean_scrub_stamp":"2024-09-18T08:10:18.542984+0000","objects_scrubbed":2,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:11:33.720540+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":6,"num_read_kb":4,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":374,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-09-18T08:10:41.505911+0000","last_change":"2024-09-18T08:10:13.820205+0000","last_active":"2024-09-18T08:10:41.505911+0000","last_peered":"2024-09-18T08:10:41.505911+0000","last_clean":"2024-09-18T08:10:41.505911+0000","last_became_active":"2024-09-18T08:02:47.877616+0000","last_became_peered":"2024-09-18T08:02:47.877616+0000","last_unstale":"2024-09-18T08:10:41.505911+0000","last_undegraded":"2024-09-18T08:10:41.505911+0000","last_fullsized":"2024-09-18T08:10:41.505911+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:10:13.820142+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:46.860092+0000","last_clean_scrub_stamp":"2024-09-18T08:10:13.820142+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:11:32.099137+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":101,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-09-18T08:10:41.488429+0000","last_change":"2024-09-18T08:08:18.550244+0000","last_active":"2024-09-18T08:10:41.488429+0000","last_peered":"2024-09-18T08:10:41.488429+0000","last_clean":"2024-09-18T08:10:41.488429+0000","last_became_active":"2024-09-18T08:08:18.545361+0000","last_became_peered":"2024-09-18T08:08:18.545361+0000","last_unstale":"2024-09-18T08:10:41.488429+0000","last_undegraded":"2024-09-18T08:10:41.488429+0000","last_fullsized":"2024-09-18T08:10:41.488429+0000","mapping_epoch":181,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":182,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:07:17.605562+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:50.886830+0000","last_clean_scrub_stamp":"2024-09-18T08:07:17.605562+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,7],"acting":[4,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":377,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-09-18T08:10:41.495972+0000","last_change":"2024-09-18T08:09:34.607443+0000","last_active":"2024-09-18T08:10:41.495972+0000","last_peered":"2024-09-18T08:10:41.495972+0000","last_clean":"2024-09-18T08:10:41.495972+0000","last_became_active":"2024-09-18T08:08:18.525552+0000","last_became_peered":"2024-09-18T08:08:18.525552+0000","last_unstale":"2024-09-18T08:10:41.495972+0000","last_undegraded":"2024-09-18T08:10:41.495972+0000","last_fullsized":"2024-09-18T08:10:41.495972+0000","mapping_epoch":181,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":182,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:09:34.607374+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_clean_scrub_stamp":"2024-09-18T08:09:34.607374+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:11:40.565887+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,4],"acting":[7,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":345,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-09-18T08:10:41.501704+0000","last_change":"2024-09-18T08:08:18.539359+0000","last_active":"2024-09-18T08:10:41.501704+0000","last_peered":"2024-09-18T08:10:41.501704+0000","last_clean":"2024-09-18T08:10:41.501704+0000","last_became_active":"2024-09-18T08:08:18.538838+0000","last_became_peered":"2024-09-18T08:08:18.538838+0000","last_unstale":"2024-09-18T08:10:41.501704+0000","last_undegraded":"2024-09-18T08:10:41.501704+0000","last_fullsized":"2024-09-18T08:10:41.501704+0000","mapping_epoch":181,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":182,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_clean_scrub_stamp":"2024-09-18T08:02:52.890475+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:10:51.616752+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":359,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-09-18T08:10:41.498743+0000","last_change":"2024-09-18T08:07:17.277964+0000","last_active":"2024-09-18T08:10:41.498743+0000","last_peered":"2024-09-18T08:10:41.498743+0000","last_clean":"2024-09-18T08:10:41.498743+0000","last_became_active":"2024-09-18T08:02:47.878922+0000","last_became_peered":"2024-09-18T08:02:47.878922+0000","last_unstale":"2024-09-18T08:10:41.498743+0000","last_undegraded":"2024-09-18T08:10:41.498743+0000","last_fullsized":"2024-09-18T08:10:41.498743+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:07:17.277894+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:46.860092+0000","last_clean_scrub_stamp":"2024-09-18T08:07:17.277894+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:11:04.482372+0000","scrub_duration":16,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"69'11","reported_seq":347,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-09-18T08:10:41.500275+0000","last_change":"2024-09-18T08:06:24.164086+0000","last_active":"2024-09-18T08:10:41.500275+0000","last_peered":"2024-09-18T08:10:41.500275+0000","last_clean":"2024-09-18T08:10:41.500275+0000","last_became_active":"2024-09-18T08:06:24.135232+0000","last_became_peered":"2024-09-18T08:06:24.135232+0000","last_unstale":"2024-09-18T08:10:41.500275+0000","last_undegraded":"2024-09-18T08:10:41.500275+0000","last_fullsized":"2024-09-18T08:10:41.500275+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'11","last_scrub_stamp":"2024-09-18T08:06:12.967462+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_clean_scrub_stamp":"2024-09-18T08:06:12.967462+0000","objects_scrubbed":5,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":19,"num_read_kb":12,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,1],"acting":[5,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":364,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-09-18T08:10:41.499171+0000","last_change":"2024-09-18T08:08:52.701053+0000","last_active":"2024-09-18T08:10:41.499171+0000","last_peered":"2024-09-18T08:10:41.499171+0000","last_clean":"2024-09-18T08:10:41.499171+0000","last_became_active":"2024-09-18T08:08:52.700595+0000","last_became_peered":"2024-09-18T08:08:52.700595+0000","last_unstale":"2024-09-18T08:10:41.499171+0000","last_undegraded":"2024-09-18T08:10:41.499171+0000","last_fullsized":"2024-09-18T08:10:41.499171+0000","mapping_epoch":198,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":199,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:08:12.639376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:50.886830+0000","last_clean_scrub_stamp":"2024-09-18T08:08:12.639376+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":8,"num_pg":16,"stat_sum":{"num_bytes":1027407896,"num_objects":15678,"num_object_clones":0,"num_object_copies":47034,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15678,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":185822,"num_write_kb":6447937,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24244,"num_bytes_recovered":1588854784,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":3781111808,"data_stored":3781099592,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":78608,"ondisk_log_size":78608,"up":48,"acting":48,"num_store_stats":8},{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":416,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":85,"ondisk_log_size":85,"up":96,"acting":96,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":178,"num_object_clones":0,"num_object_copies":534,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":178,"num_whiteouts":0,"num_read":685,"num_read_kb":442,"num_write":417,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":219,"num_bytes_recovered":4334,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":417792,"data_stored":11106,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":398,"ondisk_log_size":398,"up":96,"acting":96,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":1613,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":24,"num_read_kb":24,"num_write":10,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":3226,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":4839,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":6,"ondisk_log_size":6,"up":96,"acting":96,"num_store_stats":8},{"poolid":2,"num_pg":3,"stat_sum":{"num_bytes":408,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":243,"num_read_kb":238,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":38,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":8,"ondisk_log_size":8,"up":9,"acting":9,"num_store_stats":7},{"poolid":1,"num_pg":11,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":869,"ondisk_log_size":869,"up":33,"acting":33,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":56,"seq":240518168675,"num_pgs":62,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":556792,"kb_used_data":340968,"kb_used_omap":1,"kb_used_meta":215806,"kb_avail":93188360,"statfs":{"total":95995035648,"available":95424880640,"internally_reserved":0,"allocated":349151232,"data_stored":348754332,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":220985802},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":47,"seq":201863463017,"num_pgs":54,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":783200,"kb_used_data":575696,"kb_used_omap":1,"kb_used_meta":207486,"kb_avail":92961952,"statfs":{"total":95995035648,"available":95193038848,"internally_reserved":0,"allocated":589512704,"data_stored":589139271,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":212466118},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1],"upper_bound":4},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":39,"seq":167503724655,"num_pgs":60,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":712636,"kb_used_data":510700,"kb_used_omap":1,"kb_used_meta":201918,"kb_avail":93032516,"statfs":{"total":95995035648,"available":95265296384,"internally_reserved":0,"allocated":522956800,"data_stored":522556291,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":206764486},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,4,2,1],"upper_bound":32},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":32,"seq":137438953588,"num_pgs":67,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":851588,"kb_used_data":621872,"kb_used_omap":1,"kb_used_meta":229694,"kb_avail":92893564,"statfs":{"total":95995035648,"available":95123009536,"internally_reserved":0,"allocated":636796928,"data_stored":636392325,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":235207109},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":26,"seq":111669149819,"num_pgs":68,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":705544,"kb_used_data":385592,"kb_used_omap":1,"kb_used_meta":319934,"kb_avail":93039608,"statfs":{"total":95995035648,"available":95272558592,"internally_reserved":0,"allocated":394846208,"data_stored":394433250,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":327612872},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411457,"num_pgs":49,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":733048,"kb_used_data":404772,"kb_used_omap":1,"kb_used_meta":328254,"kb_avail":93012104,"statfs":{"total":95995035648,"available":95244394496,"internally_reserved":0,"allocated":414486528,"data_stored":414094097,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":336132549},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1],"upper_bound":4},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574981,"num_pgs":62,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":728092,"kb_used_data":449672,"kb_used_omap":1,"kb_used_meta":278270,"kb_avail":93017060,"statfs":{"total":95995035648,"available":95249469440,"internally_reserved":0,"allocated":460464128,"data_stored":460036028,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":284948933},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738508,"num_pgs":56,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":850624,"kb_used_data":566640,"kb_used_omap":1,"kb_used_meta":283838,"kb_avail":92894528,"statfs":{"total":95995035648,"available":95123996672,"internally_reserved":0,"allocated":580239360,"data_stored":579834959,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":290650565},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,5],"upper_bound":16},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":266240,"data_stored":262688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1131,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1521,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":436,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":46,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1177,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":436,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":69632,"data_stored":1788,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1128,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":1898,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1474,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1210,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1034,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1254,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":403,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":416,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":403,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":548405248,"data_stored":548405248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":445120512,"data_stored":445120512,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":383324160,"data_stored":383320088,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":382402560,"data_stored":382402560,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":612237312,"data_stored":612237312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":515837952,"data_stored":515833880,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":563216384,"data_stored":563216384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":330567680,"data_stored":330563608,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T08:10:49.115 INFO:tasks.cephadm.ceph_manager.ceph:would drop below min_pgs, proj 6, currently 16 2024-09-18T08:10:49.119 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3 2024-09-18T08:10:49.166 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:10:49.167 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:10:49.167 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3' 2024-09-18T08:10:49.268 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7 2024-09-18T08:10:49.295 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:10:49.295 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:10:49.296 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7' 2024-09-18T08:10:49.397 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5 2024-09-18T08:10:49.433 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:10:49.433 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:10:49.434 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5' 2024-09-18T08:10:49.534 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0 2024-09-18T08:10:49.579 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:10:49.580 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:10:49.580 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0' 2024-09-18T08:10:49.656 INFO:journalctl@ceph.iscsi.iscsi.a.smithi050.stdout:Sep 18 08:10:49 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-iscsi-iscsi-a[57782]: debug there is no tcmu-runner data available 2024-09-18T08:10:49.681 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1 2024-09-18T08:10:49.727 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:10:49.728 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:10:49.728 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1' 2024-09-18T08:10:49.830 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7 2024-09-18T08:10:49.856 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:10:49.856 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:10:49.857 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7' 2024-09-18T08:10:49.862 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/779349ca-7593-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T08:10:49.923 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/779349ca-7593-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T08:10:49.957 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1 2024-09-18T08:10:50.003 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:10:50.004 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:10:50.005 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1' 2024-09-18T08:10:50.105 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5 2024-09-18T08:10:50.141 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:10:50.141 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:10:50.142 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5' 2024-09-18T08:10:50.242 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7 2024-09-18T08:10:50.272 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:10:50.273 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:10:50.273 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7' 2024-09-18T08:10:50.374 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6 2024-09-18T08:10:50.397 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:10:50.398 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:10:50.398 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6' 2024-09-18T08:10:50.499 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6 2024-09-18T08:10:50.530 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:10:50.531 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:10:50.531 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6' 2024-09-18T08:10:50.632 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6 2024-09-18T08:10:50.666 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:10:50.667 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:10:50.667 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6' 2024-09-18T08:10:50.768 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3 2024-09-18T08:10:50.804 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:10:50.804 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:10:50.805 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3' 2024-09-18T08:10:50.905 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7 2024-09-18T08:10:50.906 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:10:50 smithi050 ceph-mon[32371]: purged_snaps scrub starts 2024-09-18T08:10:50.907 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:10:50 smithi050 ceph-mon[32371]: purged_snaps scrub ok 2024-09-18T08:10:50.907 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:10:50 smithi050 ceph-mon[32371]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-18T08:10:50.907 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:10:50 smithi050 ceph-mon[32371]: 5.16 scrub starts 2024-09-18T08:10:50.907 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:10:50 smithi050 ceph-mon[32371]: 5.16 scrub ok 2024-09-18T08:10:50.907 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:10:50 smithi050 ceph-mon[32371]: pgmap v382: 158 pgs: 158 active+clean; 618 MiB data, 4.5 GiB used, 711 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.13k op/s 2024-09-18T08:10:50.907 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:10:50 smithi050 ceph-mon[32371]: 8.3 scrub starts 2024-09-18T08:10:50.907 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:10:50 smithi050 ceph-mon[32371]: osdmap e227: 8 total, 8 up, 8 in 2024-09-18T08:10:50.907 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:10:50 smithi050 ceph-mon[32371]: 8.3 scrub ok 2024-09-18T08:10:50.943 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:10:50.944 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:10:50.944 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7' 2024-09-18T08:10:50.980 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:10:50 smithi007 ceph-mon[29459]: purged_snaps scrub starts 2024-09-18T08:10:50.980 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:10:50 smithi007 ceph-mon[29459]: purged_snaps scrub ok 2024-09-18T08:10:50.981 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:10:50 smithi007 ceph-mon[29459]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-18T08:10:50.981 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:10:50 smithi007 ceph-mon[29459]: 5.16 scrub starts 2024-09-18T08:10:50.981 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:10:50 smithi007 ceph-mon[29459]: 5.16 scrub ok 2024-09-18T08:10:50.981 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:10:50 smithi007 ceph-mon[29459]: pgmap v382: 158 pgs: 158 active+clean; 618 MiB data, 4.5 GiB used, 711 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.13k op/s 2024-09-18T08:10:50.981 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:10:50 smithi007 ceph-mon[29459]: 8.3 scrub starts 2024-09-18T08:10:50.981 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:10:50 smithi007 ceph-mon[29459]: osdmap e227: 8 total, 8 up, 8 in 2024-09-18T08:10:50.981 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:10:50 smithi007 ceph-mon[29459]: 8.3 scrub ok 2024-09-18T08:10:50.981 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:10:50 smithi007 ceph-mon[39249]: purged_snaps scrub starts 2024-09-18T08:10:50.981 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:10:50 smithi007 ceph-mon[39249]: purged_snaps scrub ok 2024-09-18T08:10:50.981 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:10:50 smithi007 ceph-mon[39249]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-18T08:10:50.981 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:10:50 smithi007 ceph-mon[39249]: 5.16 scrub starts 2024-09-18T08:10:50.982 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:10:50 smithi007 ceph-mon[39249]: 5.16 scrub ok 2024-09-18T08:10:50.982 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:10:50 smithi007 ceph-mon[39249]: pgmap v382: 158 pgs: 158 active+clean; 618 MiB data, 4.5 GiB used, 711 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.13k op/s 2024-09-18T08:10:50.982 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:10:50 smithi007 ceph-mon[39249]: 8.3 scrub starts 2024-09-18T08:10:50.982 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:10:50 smithi007 ceph-mon[39249]: osdmap e227: 8 total, 8 up, 8 in 2024-09-18T08:10:50.982 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:10:50 smithi007 ceph-mon[39249]: 8.3 scrub ok 2024-09-18T08:10:51.045 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5 2024-09-18T08:10:51.074 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:10:51.074 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:10:51.075 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5' 2024-09-18T08:10:51.175 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0 2024-09-18T08:10:51.213 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:10:51.213 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:10:51.213 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0' 2024-09-18T08:10:51.314 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0 2024-09-18T08:10:51.347 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:10:51.348 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:10:51.348 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0' 2024-09-18T08:10:51.449 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3 2024-09-18T08:10:51.486 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:10:51.486 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:10:51.486 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3' 2024-09-18T08:10:51.587 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0 2024-09-18T08:10:51.610 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:10:51.611 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:10:51.611 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0' 2024-09-18T08:10:51.712 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4 2024-09-18T08:10:51.744 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:10:51.745 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:10:51.745 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4' 2024-09-18T08:10:51.804 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid 779349ca-7593-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-18T08:10:51.843 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:10:51 smithi007 ceph-mon[29459]: 4.1f scrub starts 2024-09-18T08:10:51.843 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:10:51 smithi007 ceph-mon[29459]: 4.1f scrub ok 2024-09-18T08:10:51.844 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:10:51 smithi007 ceph-mon[29459]: 4.5 scrub starts 2024-09-18T08:10:51.844 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:10:51 smithi007 ceph-mon[29459]: 4.5 scrub ok 2024-09-18T08:10:51.844 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:10:51 smithi007 ceph-mon[29459]: 3.13 scrub starts 2024-09-18T08:10:51.844 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:10:51 smithi007 ceph-mon[29459]: 3.13 scrub ok 2024-09-18T08:10:51.844 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:10:51 smithi007 ceph-mon[29459]: 1.6 scrub starts 2024-09-18T08:10:51.844 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:10:51 smithi007 ceph-mon[29459]: 1.6 scrub ok 2024-09-18T08:10:51.844 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:10:51 smithi007 ceph-mon[39249]: 4.1f scrub starts 2024-09-18T08:10:51.844 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:10:51 smithi007 ceph-mon[39249]: 4.1f scrub ok 2024-09-18T08:10:51.844 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:10:51 smithi007 ceph-mon[39249]: 4.5 scrub starts 2024-09-18T08:10:51.844 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:10:51 smithi007 ceph-mon[39249]: 4.5 scrub ok 2024-09-18T08:10:51.845 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:10:51 smithi007 ceph-mon[39249]: 3.13 scrub starts 2024-09-18T08:10:51.845 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:10:51 smithi007 ceph-mon[39249]: 3.13 scrub ok 2024-09-18T08:10:51.845 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:10:51 smithi007 ceph-mon[39249]: 1.6 scrub starts 2024-09-18T08:10:51.845 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:10:51 smithi007 ceph-mon[39249]: 1.6 scrub ok 2024-09-18T08:10:51.846 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3 2024-09-18T08:10:51.884 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:10:51.884 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:10:51.885 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3' 2024-09-18T08:10:51.906 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:10:51 smithi050 ceph-mon[32371]: 4.1f scrub starts 2024-09-18T08:10:51.906 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:10:51 smithi050 ceph-mon[32371]: 4.1f scrub ok 2024-09-18T08:10:51.906 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:10:51 smithi050 ceph-mon[32371]: 4.5 scrub starts 2024-09-18T08:10:51.906 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:10:51 smithi050 ceph-mon[32371]: 4.5 scrub ok 2024-09-18T08:10:51.906 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:10:51 smithi050 ceph-mon[32371]: 3.13 scrub starts 2024-09-18T08:10:51.906 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:10:51 smithi050 ceph-mon[32371]: 3.13 scrub ok 2024-09-18T08:10:51.906 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:10:51 smithi050 ceph-mon[32371]: 1.6 scrub starts 2024-09-18T08:10:51.906 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:10:51 smithi050 ceph-mon[32371]: 1.6 scrub ok 2024-09-18T08:10:51.986 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4 2024-09-18T08:10:52.014 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:10:52.015 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:10:52.015 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4' 2024-09-18T08:10:52.116 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7 2024-09-18T08:10:52.145 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:10:52.145 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:10:52.146 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7' 2024-09-18T08:10:52.246 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5 2024-09-18T08:10:52.276 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:10:52.277 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:10:52.277 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5' 2024-09-18T08:10:52.378 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6 2024-09-18T08:10:52.411 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:10:52.412 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:10:52.412 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6' 2024-09-18T08:10:52.513 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4 2024-09-18T08:10:52.541 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:10:52.542 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:10:52.542 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4' 2024-09-18T08:10:52.589 INFO:teuthology.orchestra.run.smithi007.stderr:nodeep-scrub is unset 2024-09-18T08:10:52.643 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1 2024-09-18T08:10:52.678 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:10:52.679 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:10:52.680 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1' 2024-09-18T08:10:52.780 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0 2024-09-18T08:10:52.819 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:10:52.820 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:10:52.820 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0' 2024-09-18T08:10:52.906 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:10:52 smithi050 ceph-mon[32371]: 3.1d scrub starts 2024-09-18T08:10:52.906 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:10:52 smithi050 ceph-mon[32371]: 3.1d scrub ok 2024-09-18T08:10:52.906 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:10:52 smithi050 ceph-mon[32371]: pgmap v384: 158 pgs: 158 active+clean; 618 MiB data, 4.5 GiB used, 711 GiB / 715 GiB avail; 1.0 KiB/s rd, 0 B/s wr, 1.16k op/s 2024-09-18T08:10:52.906 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:10:52 smithi050 ceph-mon[32371]: from='client.? 172.21.15.7:0/3709151645' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-18T08:10:52.921 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6 2024-09-18T08:10:52.926 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:10:52 smithi007 ceph-mon[29459]: 3.1d scrub starts 2024-09-18T08:10:52.926 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:10:52 smithi007 ceph-mon[29459]: 3.1d scrub ok 2024-09-18T08:10:52.926 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:10:52 smithi007 ceph-mon[29459]: pgmap v384: 158 pgs: 158 active+clean; 618 MiB data, 4.5 GiB used, 711 GiB / 715 GiB avail; 1.0 KiB/s rd, 0 B/s wr, 1.16k op/s 2024-09-18T08:10:52.926 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:10:52 smithi007 ceph-mon[29459]: from='client.? 172.21.15.7:0/3709151645' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-18T08:10:52.926 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:10:52 smithi007 ceph-mon[39249]: 3.1d scrub starts 2024-09-18T08:10:52.926 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:10:52 smithi007 ceph-mon[39249]: 3.1d scrub ok 2024-09-18T08:10:52.926 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:10:52 smithi007 ceph-mon[39249]: pgmap v384: 158 pgs: 158 active+clean; 618 MiB data, 4.5 GiB used, 711 GiB / 715 GiB avail; 1.0 KiB/s rd, 0 B/s wr, 1.16k op/s 2024-09-18T08:10:52.927 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:10:52 smithi007 ceph-mon[39249]: from='client.? 172.21.15.7:0/3709151645' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-18T08:10:52.949 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:10:52.949 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:10:52.950 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6' 2024-09-18T08:10:53.051 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2 2024-09-18T08:10:53.087 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:10:53.087 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:10:53.088 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2' 2024-09-18T08:10:53.189 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1 2024-09-18T08:10:53.203 INFO:teuthology.orchestra.run.smithi007.stdout:ERROR: (22) Invalid argument 2024-09-18T08:10:53.204 INFO:teuthology.orchestra.run.smithi007.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-18T08:10:53.221 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:10:53.222 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:10:53.222 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1' 2024-09-18T08:10:53.323 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3 2024-09-18T08:10:53.354 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:10:53.355 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:10:53.355 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3' 2024-09-18T08:10:53.456 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1 2024-09-18T08:10:53.493 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:10:53.494 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:10:53.494 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1' 2024-09-18T08:10:53.595 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7 2024-09-18T08:10:53.624 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:10:53.624 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:10:53.625 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7' 2024-09-18T08:10:53.725 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7 2024-09-18T08:10:53.766 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:10:53.767 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:10:53.767 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7' 2024-09-18T08:10:53.868 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6 2024-09-18T08:10:53.902 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:10:53.903 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:10:53.903 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6' 2024-09-18T08:10:53.905 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:10:53 smithi050 ceph-mon[32371]: 4.1b scrub starts 2024-09-18T08:10:53.906 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:10:53 smithi050 ceph-mon[32371]: 4.1b scrub ok 2024-09-18T08:10:53.906 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:10:53 smithi050 ceph-mon[32371]: 5.10 scrub starts 2024-09-18T08:10:53.906 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:10:53 smithi050 ceph-mon[32371]: 5.10 scrub ok 2024-09-18T08:10:53.906 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:10:53 smithi050 ceph-mon[32371]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-18T08:10:53.906 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:10:53 smithi050 ceph-mon[32371]: Cluster is now healthy 2024-09-18T08:10:53.906 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:10:53 smithi050 ceph-mon[32371]: from='client.? 172.21.15.7:0/3709151645' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-18T08:10:53.906 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:10:53 smithi050 ceph-mon[32371]: osdmap e228: 8 total, 8 up, 8 in 2024-09-18T08:10:53.906 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:10:53 smithi050 ceph-mon[32371]: 6.1d scrub starts 2024-09-18T08:10:53.906 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:10:53 smithi050 ceph-mon[32371]: 6.1d scrub ok 2024-09-18T08:10:53.927 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:10:53 smithi007 ceph-mon[29459]: 4.1b scrub starts 2024-09-18T08:10:53.927 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:10:53 smithi007 ceph-mon[29459]: 4.1b scrub ok 2024-09-18T08:10:53.927 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:10:53 smithi007 ceph-mon[29459]: 5.10 scrub starts 2024-09-18T08:10:53.927 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:10:53 smithi007 ceph-mon[29459]: 5.10 scrub ok 2024-09-18T08:10:53.927 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:10:53 smithi007 ceph-mon[29459]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-18T08:10:53.927 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:10:53 smithi007 ceph-mon[29459]: Cluster is now healthy 2024-09-18T08:10:53.927 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:10:53 smithi007 ceph-mon[29459]: from='client.? 172.21.15.7:0/3709151645' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-18T08:10:53.927 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:10:53 smithi007 ceph-mon[29459]: osdmap e228: 8 total, 8 up, 8 in 2024-09-18T08:10:53.927 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:10:53 smithi007 ceph-mon[29459]: 6.1d scrub starts 2024-09-18T08:10:53.927 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:10:53 smithi007 ceph-mon[29459]: 6.1d scrub ok 2024-09-18T08:10:53.928 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:10:53 smithi007 ceph-mon[39249]: 4.1b scrub starts 2024-09-18T08:10:53.928 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:10:53 smithi007 ceph-mon[39249]: 4.1b scrub ok 2024-09-18T08:10:53.928 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:10:53 smithi007 ceph-mon[39249]: 5.10 scrub starts 2024-09-18T08:10:53.928 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:10:53 smithi007 ceph-mon[39249]: 5.10 scrub ok 2024-09-18T08:10:53.928 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:10:53 smithi007 ceph-mon[39249]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-18T08:10:53.928 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:10:53 smithi007 ceph-mon[39249]: Cluster is now healthy 2024-09-18T08:10:53.928 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:10:53 smithi007 ceph-mon[39249]: from='client.? 172.21.15.7:0/3709151645' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-18T08:10:53.928 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:10:53 smithi007 ceph-mon[39249]: osdmap e228: 8 total, 8 up, 8 in 2024-09-18T08:10:53.928 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:10:53 smithi007 ceph-mon[39249]: 6.1d scrub starts 2024-09-18T08:10:53.929 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:10:53 smithi007 ceph-mon[39249]: 6.1d scrub ok 2024-09-18T08:10:54.004 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6 2024-09-18T08:10:54.039 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:10:54.039 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:10:54.040 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6' 2024-09-18T08:10:54.116 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 5, 7, 6, 4] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-09-18T08:10:54.117 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-18T08:10:54.117 INFO:tasks.thrashosds.thrasher:trim_stale_osdmaps 2024-09-18T08:10:54.117 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid 779349ca-7593-11ef-bceb-c7b262605968 -- ceph tell osd.0 'trim stale osdmaps' 2024-09-18T08:10:54.140 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7 2024-09-18T08:10:54.167 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:10:54.168 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:10:54.169 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7' 2024-09-18T08:10:54.269 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4 2024-09-18T08:10:54.303 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:10:54.303 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:10:54.304 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4' 2024-09-18T08:10:54.405 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0 2024-09-18T08:10:54.457 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:10:54.458 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:10:54.458 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0' 2024-09-18T08:10:54.559 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7 2024-09-18T08:10:54.589 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:10:54.590 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:10:54.590 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7' 2024-09-18T08:10:54.691 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3 2024-09-18T08:10:54.726 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:10:54.726 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:10:54.727 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3' 2024-09-18T08:10:54.827 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7 2024-09-18T08:10:54.836 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid 779349ca-7593-11ef-bceb-c7b262605968 -- ceph daemon osd.3 dump_ops_in_flight 2024-09-18T08:10:54.856 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:10:54.856 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:10:54.857 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7' 2024-09-18T08:10:54.857 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/779349ca-7593-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T08:10:54.893 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:10:54 smithi007 ceph-mon[29459]: 3.16 scrub starts 2024-09-18T08:10:54.893 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:10:54 smithi007 ceph-mon[29459]: 3.16 scrub ok 2024-09-18T08:10:54.893 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:10:54 smithi007 ceph-mon[29459]: pgmap v386: 158 pgs: 158 active+clean; 570 MiB data, 4.2 GiB used, 711 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 821 op/s 2024-09-18T08:10:54.894 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:10:54 smithi007 ceph-mon[29459]: 4.e scrub starts 2024-09-18T08:10:54.894 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:10:54 smithi007 ceph-mon[29459]: 4.e scrub ok 2024-09-18T08:10:54.894 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:10:54 smithi007 ceph-mon[29459]: 3.f scrub starts 2024-09-18T08:10:54.894 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:10:54 smithi007 ceph-mon[29459]: 3.f scrub ok 2024-09-18T08:10:54.894 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:10:54 smithi007 ceph-mon[29459]: 5.5 scrub starts 2024-09-18T08:10:54.894 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:10:54 smithi007 ceph-mon[29459]: 5.5 scrub ok 2024-09-18T08:10:54.894 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:10:54 smithi007 ceph-mon[39249]: 3.16 scrub starts 2024-09-18T08:10:54.894 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:10:54 smithi007 ceph-mon[39249]: 3.16 scrub ok 2024-09-18T08:10:54.894 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:10:54 smithi007 ceph-mon[39249]: pgmap v386: 158 pgs: 158 active+clean; 570 MiB data, 4.2 GiB used, 711 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 821 op/s 2024-09-18T08:10:54.894 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:10:54 smithi007 ceph-mon[39249]: 4.e scrub starts 2024-09-18T08:10:54.895 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:10:54 smithi007 ceph-mon[39249]: 4.e scrub ok 2024-09-18T08:10:54.895 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:10:54 smithi007 ceph-mon[39249]: 3.f scrub starts 2024-09-18T08:10:54.895 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:10:54 smithi007 ceph-mon[39249]: 3.f scrub ok 2024-09-18T08:10:54.895 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:10:54 smithi007 ceph-mon[39249]: 5.5 scrub starts 2024-09-18T08:10:54.895 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:10:54 smithi007 ceph-mon[39249]: 5.5 scrub ok 2024-09-18T08:10:54.906 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:10:54 smithi050 ceph-mon[32371]: 3.16 scrub starts 2024-09-18T08:10:54.906 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:10:54 smithi050 ceph-mon[32371]: 3.16 scrub ok 2024-09-18T08:10:54.906 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:10:54 smithi050 ceph-mon[32371]: pgmap v386: 158 pgs: 158 active+clean; 570 MiB data, 4.2 GiB used, 711 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 821 op/s 2024-09-18T08:10:54.906 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:10:54 smithi050 ceph-mon[32371]: 4.e scrub starts 2024-09-18T08:10:54.906 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:10:54 smithi050 ceph-mon[32371]: 4.e scrub ok 2024-09-18T08:10:54.906 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:10:54 smithi050 ceph-mon[32371]: 3.f scrub starts 2024-09-18T08:10:54.906 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:10:54 smithi050 ceph-mon[32371]: 3.f scrub ok 2024-09-18T08:10:54.906 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:10:54 smithi050 ceph-mon[32371]: 5.5 scrub starts 2024-09-18T08:10:54.906 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:10:54 smithi050 ceph-mon[32371]: 5.5 scrub ok 2024-09-18T08:10:54.958 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5 2024-09-18T08:10:54.988 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:10:54.989 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:10:54.989 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5' 2024-09-18T08:10:55.090 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4 2024-09-18T08:10:55.118 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:10:55.118 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:10:55.119 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4' 2024-09-18T08:10:55.220 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4 2024-09-18T08:10:55.248 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:10:55.249 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:10:55.250 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4' 2024-09-18T08:10:55.350 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1 2024-09-18T08:10:55.390 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:10:55.391 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:10:55.391 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1' 2024-09-18T08:10:55.473 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/779349ca-7593-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T08:10:55.492 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2 2024-09-18T08:10:55.552 INFO:teuthology.orchestra.run.smithi007.stdout:osd.0: {} 2024-09-18T08:10:55.552 INFO:teuthology.orchestra.run.smithi007.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-18T08:10:55.560 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:10:55.560 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:10:55.561 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2' 2024-09-18T08:10:55.591 INFO:teuthology.orchestra.run.smithi007.stdout:osd.1: {} 2024-09-18T08:10:55.592 INFO:teuthology.orchestra.run.smithi007.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-18T08:10:55.625 INFO:teuthology.orchestra.run.smithi007.stdout:osd.2: {} 2024-09-18T08:10:55.626 INFO:teuthology.orchestra.run.smithi007.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-18T08:10:55.661 INFO:teuthology.orchestra.run.smithi007.stdout:osd.3: {} 2024-09-18T08:10:55.662 INFO:teuthology.orchestra.run.smithi007.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-18T08:10:55.662 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4 2024-09-18T08:10:55.686 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:10:55.687 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:10:55.688 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4' 2024-09-18T08:10:55.696 INFO:teuthology.orchestra.run.smithi007.stdout:osd.4: {} 2024-09-18T08:10:55.696 INFO:teuthology.orchestra.run.smithi007.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-18T08:10:55.730 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:10:55 smithi007 ceph-mon[29459]: 5.15 scrub starts 2024-09-18T08:10:55.731 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:10:55 smithi007 ceph-mon[29459]: 5.15 scrub ok 2024-09-18T08:10:55.731 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:10:55 smithi007 ceph-mon[29459]: 5.1 scrub starts 2024-09-18T08:10:55.731 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:10:55 smithi007 ceph-mon[29459]: 5.1 scrub ok 2024-09-18T08:10:55.731 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:10:55 smithi007 ceph-mon[29459]: from='client.? 172.21.15.7:0/451355636' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-18T08:10:55.731 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:10:55 smithi007 ceph-mon[39249]: 5.15 scrub starts 2024-09-18T08:10:55.731 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:10:55 smithi007 ceph-mon[39249]: 5.15 scrub ok 2024-09-18T08:10:55.731 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:10:55 smithi007 ceph-mon[39249]: 5.1 scrub starts 2024-09-18T08:10:55.732 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:10:55 smithi007 ceph-mon[39249]: 5.1 scrub ok 2024-09-18T08:10:55.732 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:10:55 smithi007 ceph-mon[39249]: from='client.? 172.21.15.7:0/451355636' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-18T08:10:55.733 INFO:teuthology.orchestra.run.smithi007.stdout:osd.5: {} 2024-09-18T08:10:55.734 INFO:teuthology.orchestra.run.smithi007.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-18T08:10:55.767 INFO:teuthology.orchestra.run.smithi007.stdout:osd.6: {} 2024-09-18T08:10:55.767 INFO:teuthology.orchestra.run.smithi007.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-18T08:10:55.789 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1 2024-09-18T08:10:55.803 INFO:teuthology.orchestra.run.smithi007.stdout:osd.7: {} 2024-09-18T08:10:55.804 INFO:teuthology.orchestra.run.smithi007.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-18T08:10:55.821 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/779349ca-7593-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T08:10:55.852 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:10:55.871 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:10:55.872 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1' 2024-09-18T08:10:55.906 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:10:55 smithi050 ceph-mon[32371]: 5.15 scrub starts 2024-09-18T08:10:55.906 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:10:55 smithi050 ceph-mon[32371]: 5.15 scrub ok 2024-09-18T08:10:55.906 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:10:55 smithi050 ceph-mon[32371]: 5.1 scrub starts 2024-09-18T08:10:55.906 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:10:55 smithi050 ceph-mon[32371]: 5.1 scrub ok 2024-09-18T08:10:55.906 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:10:55 smithi050 ceph-mon[32371]: from='client.? 172.21.15.7:0/451355636' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-18T08:10:55.973 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5 2024-09-18T08:10:56.006 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:10:56.007 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:10:56.007 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5' 2024-09-18T08:10:56.108 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6 2024-09-18T08:10:56.136 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:10:56.137 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:10:56.137 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6' 2024-09-18T08:10:56.205 INFO:teuthology.orchestra.run.smithi007.stderr:no valid command found; 10 closest matches: 2024-09-18T08:10:56.205 INFO:teuthology.orchestra.run.smithi007.stderr:0 2024-09-18T08:10:56.205 INFO:teuthology.orchestra.run.smithi007.stderr:1 2024-09-18T08:10:56.205 INFO:teuthology.orchestra.run.smithi007.stderr:2 2024-09-18T08:10:56.205 INFO:teuthology.orchestra.run.smithi007.stderr:abort 2024-09-18T08:10:56.206 INFO:teuthology.orchestra.run.smithi007.stderr:assert 2024-09-18T08:10:56.206 INFO:teuthology.orchestra.run.smithi007.stderr:bench [] [] [] [] 2024-09-18T08:10:56.206 INFO:teuthology.orchestra.run.smithi007.stderr:bluefs debug_inject_read_zeros 2024-09-18T08:10:56.206 INFO:teuthology.orchestra.run.smithi007.stderr:bluefs files list 2024-09-18T08:10:56.206 INFO:teuthology.orchestra.run.smithi007.stderr:bluefs stats 2024-09-18T08:10:56.206 INFO:teuthology.orchestra.run.smithi007.stderr:bluestore allocator dump block 2024-09-18T08:10:56.206 INFO:teuthology.orchestra.run.smithi007.stderr:Error EINVAL: invalid command 2024-09-18T08:10:56.238 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0 2024-09-18T08:10:56.277 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:10:56.278 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:10:56.278 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0' 2024-09-18T08:10:56.379 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5 2024-09-18T08:10:56.411 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid 779349ca-7593-11ef-bceb-c7b262605968 -- ceph osd set noscrub 2024-09-18T08:10:56.413 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:10:56.414 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:10:56.415 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5' 2024-09-18T08:10:56.515 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2 2024-09-18T08:10:56.561 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:10:56.562 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:10:56.562 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2' 2024-09-18T08:10:56.663 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2 2024-09-18T08:10:56.717 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:10:56.717 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:10:56.718 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2' 2024-09-18T08:10:56.776 INFO:tasks.radosbench.radosbench.0.smithi007.stdout:Removed 100749 objects 2024-09-18T08:10:56.776 INFO:tasks.radosbench.radosbench.0.smithi007.stdout:Clean up completed and total clean up time :87.5821 2024-09-18T08:10:56.819 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7 2024-09-18T08:10:56.841 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:10:56.842 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:10:56.843 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7' 2024-09-18T08:10:56.906 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:10:56 smithi050 ceph-mon[32371]: pgmap v387: 158 pgs: 158 active+clean; 237 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.49k op/s 2024-09-18T08:10:56.906 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:10:56 smithi050 ceph-mon[32371]: 4.0 scrub starts 2024-09-18T08:10:56.906 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:10:56 smithi050 ceph-mon[32371]: 4.0 scrub ok 2024-09-18T08:10:56.943 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7 2024-09-18T08:10:56.965 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:10:56.966 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:10:56.966 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7' 2024-09-18T08:10:56.980 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:10:56 smithi007 ceph-mon[29459]: pgmap v387: 158 pgs: 158 active+clean; 237 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.49k op/s 2024-09-18T08:10:56.980 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:10:56 smithi007 ceph-mon[29459]: 4.0 scrub starts 2024-09-18T08:10:56.980 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:10:56 smithi007 ceph-mon[29459]: 4.0 scrub ok 2024-09-18T08:10:56.981 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Sep 18 08:10:56 smithi007 ceph-779349ca-7593-11ef-bceb-c7b262605968-mgr-y[29688]: ::ffff:172.21.15.50 - - [18/Sep/2024:08:10:56] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-09-18T08:10:56.981 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:10:56 smithi007 ceph-mon[39249]: pgmap v387: 158 pgs: 158 active+clean; 237 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.49k op/s 2024-09-18T08:10:56.981 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:10:56 smithi007 ceph-mon[39249]: 4.0 scrub starts 2024-09-18T08:10:56.981 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:10:56 smithi007 ceph-mon[39249]: 4.0 scrub ok 2024-09-18T08:10:57.067 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6 2024-09-18T08:10:57.089 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:10:57.089 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:10:57.090 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6' 2024-09-18T08:10:57.190 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4 2024-09-18T08:10:57.213 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:10:57.214 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:10:57.214 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4' 2024-09-18T08:10:57.315 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0 2024-09-18T08:10:57.339 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:10:57.340 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:10:57.340 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0' 2024-09-18T08:10:57.441 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0 2024-09-18T08:10:57.465 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:10:57.465 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:10:57.465 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0' 2024-09-18T08:10:57.566 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3 2024-09-18T08:10:57.591 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:10:57.591 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:10:57.592 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3' 2024-09-18T08:10:57.693 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5 2024-09-18T08:10:57.718 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:10:57.719 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:10:57.719 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5' 2024-09-18T08:10:57.789 INFO:teuthology.orchestra.run.smithi007.stdout:{ 2024-09-18T08:10:57.789 INFO:teuthology.orchestra.run.smithi007.stdout: "ops": [], 2024-09-18T08:10:57.789 INFO:teuthology.orchestra.run.smithi007.stdout: "num_ops": 0 2024-09-18T08:10:57.789 INFO:teuthology.orchestra.run.smithi007.stdout:} 2024-09-18T08:10:57.820 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3 2024-09-18T08:10:57.842 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:10:57.843 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:10:57.844 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3' 2024-09-18T08:10:57.906 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:10:57 smithi050 ceph-mon[32371]: 3.d scrub starts 2024-09-18T08:10:57.906 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:10:57 smithi050 ceph-mon[32371]: 3.d scrub ok 2024-09-18T08:10:57.906 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:10:57 smithi050 ceph-mon[32371]: 5.e scrub starts 2024-09-18T08:10:57.906 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:10:57 smithi050 ceph-mon[32371]: 5.e scrub ok 2024-09-18T08:10:57.906 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:10:57 smithi050 ceph-mon[32371]: purged_snaps scrub starts 2024-09-18T08:10:57.906 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:10:57 smithi050 ceph-mon[32371]: purged_snaps scrub ok 2024-09-18T08:10:57.944 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0 2024-09-18T08:10:57.967 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:10:57.968 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:10:57.968 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0' 2024-09-18T08:10:57.980 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:10:57 smithi007 ceph-mon[29459]: 3.d scrub starts 2024-09-18T08:10:57.980 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:10:57 smithi007 ceph-mon[29459]: 3.d scrub ok 2024-09-18T08:10:57.980 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:10:57 smithi007 ceph-mon[29459]: 5.e scrub starts 2024-09-18T08:10:57.980 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:10:57 smithi007 ceph-mon[29459]: 5.e scrub ok 2024-09-18T08:10:57.980 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:10:57 smithi007 ceph-mon[29459]: purged_snaps scrub starts 2024-09-18T08:10:57.981 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:10:57 smithi007 ceph-mon[29459]: purged_snaps scrub ok 2024-09-18T08:10:57.981 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:10:57 smithi007 ceph-mon[39249]: 3.d scrub starts 2024-09-18T08:10:57.981 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:10:57 smithi007 ceph-mon[39249]: 3.d scrub ok 2024-09-18T08:10:57.981 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:10:57 smithi007 ceph-mon[39249]: 5.e scrub starts 2024-09-18T08:10:57.981 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:10:57 smithi007 ceph-mon[39249]: 5.e scrub ok 2024-09-18T08:10:57.981 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:10:57 smithi007 ceph-mon[39249]: purged_snaps scrub starts 2024-09-18T08:10:57.982 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:10:57 smithi007 ceph-mon[39249]: purged_snaps scrub ok 2024-09-18T08:10:58.069 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6 2024-09-18T08:10:58.096 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:10:58.097 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:10:58.097 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6' 2024-09-18T08:10:58.198 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5 2024-09-18T08:10:58.225 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:10:58.225 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:10:58.226 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5' 2024-09-18T08:10:58.327 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4 2024-09-18T08:10:58.489 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:10:58.490 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:10:58.491 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4' 2024-09-18T08:10:58.591 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3 2024-09-18T08:10:58.619 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:10:58.620 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:10:58.620 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3' 2024-09-18T08:10:58.662 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid 779349ca-7593-11ef-bceb-c7b262605968 -- ceph daemon osd.3 dump_blocked_ops 2024-09-18T08:10:58.722 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1 2024-09-18T08:10:58.769 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:10:58.769 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:10:58.770 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1' 2024-09-18T08:10:58.871 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5 2024-09-18T08:10:58.899 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:10:58.900 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:10:58.900 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5' 2024-09-18T08:10:58.906 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:10:58 smithi050 ceph-mon[32371]: pgmap v388: 158 pgs: 158 active+clean; 237 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 762 op/s 2024-09-18T08:10:58.906 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:10:58 smithi050 ceph-mon[32371]: 3.b scrub starts 2024-09-18T08:10:58.906 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:10:58 smithi050 ceph-mon[32371]: 3.b scrub ok 2024-09-18T08:10:58.906 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:10:58 smithi050 ceph-mon[32371]: 3.e scrub starts 2024-09-18T08:10:58.906 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:10:58 smithi050 ceph-mon[32371]: 3.e scrub ok 2024-09-18T08:10:58.906 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:10:58 smithi050 ceph-mon[32371]: osdmap e229: 8 total, 8 up, 8 in 2024-09-18T08:10:58.981 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:10:58 smithi007 ceph-mon[39249]: pgmap v388: 158 pgs: 158 active+clean; 237 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 762 op/s 2024-09-18T08:10:58.981 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:10:58 smithi007 ceph-mon[39249]: 3.b scrub starts 2024-09-18T08:10:58.981 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:10:58 smithi007 ceph-mon[39249]: 3.b scrub ok 2024-09-18T08:10:58.981 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:10:58 smithi007 ceph-mon[39249]: 3.e scrub starts 2024-09-18T08:10:58.981 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:10:58 smithi007 ceph-mon[39249]: 3.e scrub ok 2024-09-18T08:10:58.981 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:10:58 smithi007 ceph-mon[39249]: osdmap e229: 8 total, 8 up, 8 in 2024-09-18T08:10:58.981 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:10:58 smithi007 ceph-mon[29459]: pgmap v388: 158 pgs: 158 active+clean; 237 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 762 op/s 2024-09-18T08:10:58.981 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:10:58 smithi007 ceph-mon[29459]: 3.b scrub starts 2024-09-18T08:10:58.981 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:10:58 smithi007 ceph-mon[29459]: 3.b scrub ok 2024-09-18T08:10:58.982 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:10:58 smithi007 ceph-mon[29459]: 3.e scrub starts 2024-09-18T08:10:58.982 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:10:58 smithi007 ceph-mon[29459]: 3.e scrub ok 2024-09-18T08:10:58.982 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:10:58 smithi007 ceph-mon[29459]: osdmap e229: 8 total, 8 up, 8 in 2024-09-18T08:10:59.001 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4 2024-09-18T08:10:59.028 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:10:59.029 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:10:59.029 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4' 2024-09-18T08:10:59.132 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2 2024-09-18T08:10:59.155 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:10:59.156 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:10:59.156 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2' 2024-09-18T08:10:59.257 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0 2024-09-18T08:10:59.260 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid 779349ca-7593-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-18T08:10:59.282 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:10:59.282 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:10:59.283 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0' 2024-09-18T08:10:59.384 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2 2024-09-18T08:10:59.405 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:10:59.406 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:10:59.407 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2' 2024-09-18T08:10:59.507 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2 2024-09-18T08:10:59.530 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:10:59.530 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:10:59.531 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2' 2024-09-18T08:10:59.631 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1 2024-09-18T08:10:59.637 INFO:journalctl@ceph.iscsi.iscsi.a.smithi050.stdout:Sep 18 08:10:59 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-iscsi-iscsi-a[57782]: debug there is no tcmu-runner data available 2024-09-18T08:10:59.655 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:10:59.655 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:10:59.656 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1' 2024-09-18T08:10:59.693 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-09-18T08:10:59.693 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-09-18T08:10:59.693 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid 779349ca-7593-11ef-bceb-c7b262605968 -- ceph tell osd.1 'trim stale osdmaps' 2024-09-18T08:10:59.757 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0 2024-09-18T08:10:59.758 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/779349ca-7593-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T08:10:59.758 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/779349ca-7593-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T08:10:59.791 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/779349ca-7593-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T08:10:59.791 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:10:59.792 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:10:59.793 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0' 2024-09-18T08:10:59.894 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2 2024-09-18T08:10:59.905 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:10:59 smithi050 ceph-mon[32371]: 3.14 scrub starts 2024-09-18T08:10:59.906 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:10:59 smithi050 ceph-mon[32371]: 3.14 scrub ok 2024-09-18T08:10:59.906 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:10:59 smithi050 ceph-mon[32371]: 4.6 scrub starts 2024-09-18T08:10:59.906 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:10:59 smithi050 ceph-mon[32371]: 4.6 scrub ok 2024-09-18T08:10:59.909 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:10:59 smithi007 ceph-mon[29459]: 3.14 scrub starts 2024-09-18T08:10:59.909 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:10:59 smithi007 ceph-mon[29459]: 3.14 scrub ok 2024-09-18T08:10:59.909 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:10:59 smithi007 ceph-mon[29459]: 4.6 scrub starts 2024-09-18T08:10:59.909 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:10:59 smithi007 ceph-mon[29459]: 4.6 scrub ok 2024-09-18T08:10:59.910 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:10:59 smithi007 ceph-mon[39249]: 3.14 scrub starts 2024-09-18T08:10:59.910 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:10:59 smithi007 ceph-mon[39249]: 3.14 scrub ok 2024-09-18T08:10:59.910 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:10:59 smithi007 ceph-mon[39249]: 4.6 scrub starts 2024-09-18T08:10:59.910 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:10:59 smithi007 ceph-mon[39249]: 4.6 scrub ok 2024-09-18T08:10:59.917 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:10:59.917 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:10:59.918 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2' 2024-09-18T08:11:00.019 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1 2024-09-18T08:11:00.044 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:11:00.044 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:11:00.044 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1' 2024-09-18T08:11:00.145 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0 2024-09-18T08:11:00.167 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:11:00.168 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:11:00.168 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0' 2024-09-18T08:11:00.269 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5 2024-09-18T08:11:00.297 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:11:00.298 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:11:00.298 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5' 2024-09-18T08:11:00.400 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5 2024-09-18T08:11:00.427 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:11:00.427 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:11:00.428 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5' 2024-09-18T08:11:00.529 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7 2024-09-18T08:11:00.556 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:11:00.556 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:11:00.557 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7' 2024-09-18T08:11:00.658 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4 2024-09-18T08:11:00.684 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:11:00.684 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:11:00.685 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4' 2024-09-18T08:11:00.699 INFO:teuthology.orchestra.run.smithi007.stdout:osd.0: {} 2024-09-18T08:11:00.699 INFO:teuthology.orchestra.run.smithi007.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-18T08:11:00.710 INFO:teuthology.orchestra.run.smithi007.stdout:osd.1: {} 2024-09-18T08:11:00.710 INFO:teuthology.orchestra.run.smithi007.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-18T08:11:00.727 INFO:teuthology.orchestra.run.smithi007.stdout:osd.2: {} 2024-09-18T08:11:00.727 INFO:teuthology.orchestra.run.smithi007.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-18T08:11:00.743 INFO:teuthology.orchestra.run.smithi007.stdout:osd.3: {} 2024-09-18T08:11:00.743 INFO:teuthology.orchestra.run.smithi007.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-18T08:11:00.760 INFO:teuthology.orchestra.run.smithi007.stdout:osd.4: {} 2024-09-18T08:11:00.760 INFO:teuthology.orchestra.run.smithi007.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-18T08:11:00.778 INFO:teuthology.orchestra.run.smithi007.stdout:osd.5: {} 2024-09-18T08:11:00.779 INFO:teuthology.orchestra.run.smithi007.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-18T08:11:00.785 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0 2024-09-18T08:11:00.796 INFO:teuthology.orchestra.run.smithi007.stdout:osd.6: {} 2024-09-18T08:11:00.797 INFO:teuthology.orchestra.run.smithi007.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-18T08:11:00.807 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:11:00.808 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:11:00.808 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0' 2024-09-18T08:11:00.814 INFO:teuthology.orchestra.run.smithi007.stdout:osd.7: {} 2024-09-18T08:11:00.814 INFO:teuthology.orchestra.run.smithi007.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-18T08:11:00.872 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:11:00 smithi007 ceph-mon[39249]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-18T08:11:00.872 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:11:00 smithi007 ceph-mon[39249]: pgmap v390: 158 pgs: 158 active+clean; 583 KiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.24k op/s 2024-09-18T08:11:00.872 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:11:00 smithi007 ceph-mon[39249]: 5.1d scrub starts 2024-09-18T08:11:00.872 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:11:00 smithi007 ceph-mon[39249]: 5.1d scrub ok 2024-09-18T08:11:00.872 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:11:00 smithi007 ceph-mon[39249]: 6.9 scrub starts 2024-09-18T08:11:00.872 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:11:00 smithi007 ceph-mon[39249]: 6.9 scrub ok 2024-09-18T08:11:00.872 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:11:00 smithi007 ceph-mon[29459]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-18T08:11:00.873 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:11:00 smithi007 ceph-mon[29459]: pgmap v390: 158 pgs: 158 active+clean; 583 KiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.24k op/s 2024-09-18T08:11:00.873 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:11:00 smithi007 ceph-mon[29459]: 5.1d scrub starts 2024-09-18T08:11:00.873 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:11:00 smithi007 ceph-mon[29459]: 5.1d scrub ok 2024-09-18T08:11:00.873 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:11:00 smithi007 ceph-mon[29459]: 6.9 scrub starts 2024-09-18T08:11:00.873 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:11:00 smithi007 ceph-mon[29459]: 6.9 scrub ok 2024-09-18T08:11:00.906 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:11:00 smithi050 ceph-mon[32371]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-18T08:11:00.906 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:11:00 smithi050 ceph-mon[32371]: pgmap v390: 158 pgs: 158 active+clean; 583 KiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.24k op/s 2024-09-18T08:11:00.907 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:11:00 smithi050 ceph-mon[32371]: 5.1d scrub starts 2024-09-18T08:11:00.907 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:11:00 smithi050 ceph-mon[32371]: 5.1d scrub ok 2024-09-18T08:11:00.907 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:11:00 smithi050 ceph-mon[32371]: 6.9 scrub starts 2024-09-18T08:11:00.907 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:11:00 smithi050 ceph-mon[32371]: 6.9 scrub ok 2024-09-18T08:11:00.909 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3 2024-09-18T08:11:00.932 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:11:00.933 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:11:00.933 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3' 2024-09-18T08:11:01.034 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0 2024-09-18T08:11:01.057 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:11:01.057 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:11:01.058 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0' 2024-09-18T08:11:01.143 INFO:tasks.cephadm.ceph_manager.ceph:removing pool_name unique_pool_1 2024-09-18T08:11:01.144 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid 779349ca-7593-11ef-bceb-c7b262605968 -- ceph osd pool rm unique_pool_1 unique_pool_1 --yes-i-really-really-mean-it 2024-09-18T08:11:01.159 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5 2024-09-18T08:11:01.185 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:11:01.186 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:11:01.186 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5' 2024-09-18T08:11:01.287 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1 2024-09-18T08:11:01.311 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:11:01.312 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:11:01.313 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1' 2024-09-18T08:11:01.413 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6 2024-09-18T08:11:01.441 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:11:01.442 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:11:01.442 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6' 2024-09-18T08:11:01.543 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7 2024-09-18T08:11:01.570 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:11:01.570 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:11:01.571 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7' 2024-09-18T08:11:01.644 INFO:teuthology.orchestra.run.smithi007.stderr:noscrub is set 2024-09-18T08:11:01.671 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4 2024-09-18T08:11:01.699 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:11:01.700 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:11:01.700 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4' 2024-09-18T08:11:01.801 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7 2024-09-18T08:11:01.828 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:11:01.829 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:11:01.829 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7' 2024-09-18T08:11:01.906 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:11:01 smithi050 ceph-mon[32371]: 3.19 scrub starts 2024-09-18T08:11:01.906 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:11:01 smithi050 ceph-mon[32371]: 3.19 scrub ok 2024-09-18T08:11:01.906 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:11:01 smithi050 ceph-mon[32371]: from='client.? 172.21.15.7:0/1669179423' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-18T08:11:01.906 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:11:01 smithi050 ceph-mon[32371]: from='client.? 172.21.15.7:0/3647478401' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-18T08:11:01.932 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2 2024-09-18T08:11:01.956 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:11:01.958 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:11:01.958 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2' 2024-09-18T08:11:01.980 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:11:01 smithi007 ceph-mon[29459]: 3.19 scrub starts 2024-09-18T08:11:01.980 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:11:01 smithi007 ceph-mon[29459]: 3.19 scrub ok 2024-09-18T08:11:01.980 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:11:01 smithi007 ceph-mon[29459]: from='client.? 172.21.15.7:0/1669179423' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-18T08:11:01.980 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:11:01 smithi007 ceph-mon[29459]: from='client.? 172.21.15.7:0/3647478401' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-18T08:11:01.981 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:11:01 smithi007 ceph-mon[39249]: 3.19 scrub starts 2024-09-18T08:11:01.981 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:11:01 smithi007 ceph-mon[39249]: 3.19 scrub ok 2024-09-18T08:11:01.981 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:11:01 smithi007 ceph-mon[39249]: from='client.? 172.21.15.7:0/1669179423' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-18T08:11:01.981 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:11:01 smithi007 ceph-mon[39249]: from='client.? 172.21.15.7:0/3647478401' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-18T08:11:02.060 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7 2024-09-18T08:11:02.086 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:11:02.087 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:11:02.088 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7' 2024-09-18T08:11:02.188 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4 2024-09-18T08:11:02.214 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:11:02.215 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:11:02.215 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4' 2024-09-18T08:11:02.316 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5 2024-09-18T08:11:02.343 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:11:02.344 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:11:02.344 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5' 2024-09-18T08:11:02.445 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5 2024-09-18T08:11:02.450 INFO:teuthology.orchestra.run.smithi007.stdout:ERROR: (22) Invalid argument 2024-09-18T08:11:02.450 INFO:teuthology.orchestra.run.smithi007.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-18T08:11:02.472 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:11:02.472 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:11:02.473 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5' 2024-09-18T08:11:02.574 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4 2024-09-18T08:11:02.601 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:11:02.601 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:11:02.602 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4' 2024-09-18T08:11:02.702 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2 2024-09-18T08:11:02.730 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:11:02 smithi007 ceph-mon[29459]: 6.19 scrub starts 2024-09-18T08:11:02.730 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:11:02 smithi007 ceph-mon[29459]: 6.19 scrub ok 2024-09-18T08:11:02.730 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:11:02 smithi007 ceph-mon[29459]: pgmap v391: 158 pgs: 158 active+clean; 583 KiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.14k op/s 2024-09-18T08:11:02.730 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:11:02 smithi007 ceph-mon[29459]: 6.1b scrub starts 2024-09-18T08:11:02.731 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:11:02 smithi007 ceph-mon[29459]: 6.1b scrub ok 2024-09-18T08:11:02.731 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:11:02 smithi007 ceph-mon[29459]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-18T08:11:02.731 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:11:02 smithi007 ceph-mon[29459]: from='client.? 172.21.15.7:0/3647478401' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-18T08:11:02.731 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:11:02 smithi007 ceph-mon[29459]: osdmap e230: 8 total, 8 up, 8 in 2024-09-18T08:11:02.731 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:11:02 smithi007 ceph-mon[39249]: 6.19 scrub starts 2024-09-18T08:11:02.731 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:11:02 smithi007 ceph-mon[39249]: 6.19 scrub ok 2024-09-18T08:11:02.731 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:11:02 smithi007 ceph-mon[39249]: pgmap v391: 158 pgs: 158 active+clean; 583 KiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.14k op/s 2024-09-18T08:11:02.731 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:11:02 smithi007 ceph-mon[39249]: 6.1b scrub starts 2024-09-18T08:11:02.731 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:11:02 smithi007 ceph-mon[39249]: 6.1b scrub ok 2024-09-18T08:11:02.731 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:11:02 smithi007 ceph-mon[39249]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-18T08:11:02.732 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:11:02 smithi007 ceph-mon[39249]: from='client.? 172.21.15.7:0/3647478401' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-18T08:11:02.732 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:11:02 smithi007 ceph-mon[39249]: osdmap e230: 8 total, 8 up, 8 in 2024-09-18T08:11:02.732 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:11:02.733 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:11:02.733 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2' 2024-09-18T08:11:02.834 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0 2024-09-18T08:11:02.861 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:11:02.862 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:11:02.862 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0' 2024-09-18T08:11:02.906 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:11:02 smithi050 ceph-mon[32371]: 6.19 scrub starts 2024-09-18T08:11:02.906 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:11:02 smithi050 ceph-mon[32371]: 6.19 scrub ok 2024-09-18T08:11:02.906 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:11:02 smithi050 ceph-mon[32371]: pgmap v391: 158 pgs: 158 active+clean; 583 KiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.14k op/s 2024-09-18T08:11:02.906 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:11:02 smithi050 ceph-mon[32371]: 6.1b scrub starts 2024-09-18T08:11:02.906 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:11:02 smithi050 ceph-mon[32371]: 6.1b scrub ok 2024-09-18T08:11:02.906 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:11:02 smithi050 ceph-mon[32371]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-18T08:11:02.907 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:11:02 smithi050 ceph-mon[32371]: from='client.? 172.21.15.7:0/3647478401' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-18T08:11:02.907 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:11:02 smithi050 ceph-mon[32371]: osdmap e230: 8 total, 8 up, 8 in 2024-09-18T08:11:02.963 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6 2024-09-18T08:11:02.990 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:11:02.991 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:11:02.991 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6' 2024-09-18T08:11:03.092 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7 2024-09-18T08:11:03.117 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:11:03.118 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:11:03.119 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7' 2024-09-18T08:11:03.219 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0 2024-09-18T08:11:03.247 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:11:03.248 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:11:03.248 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0' 2024-09-18T08:11:03.349 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1 2024-09-18T08:11:03.376 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:11:03.377 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:11:03.377 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1' 2024-09-18T08:11:03.378 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid 779349ca-7593-11ef-bceb-c7b262605968 -- ceph daemon osd.3 dump_historic_ops 2024-09-18T08:11:03.478 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2 2024-09-18T08:11:03.502 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:11:03.503 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:11:03.503 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2' 2024-09-18T08:11:03.604 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3 2024-09-18T08:11:03.627 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:11:03.627 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:11:03.628 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3' 2024-09-18T08:11:03.729 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6 2024-09-18T08:11:03.756 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:11:03.757 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:11:03.757 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6' 2024-09-18T08:11:03.842 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/779349ca-7593-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T08:11:03.858 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3 2024-09-18T08:11:03.888 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:11:03.888 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/779349ca-7593-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T08:11:03.889 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:11:03.889 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3' 2024-09-18T08:11:03.905 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:11:03 smithi050 ceph-mon[32371]: pgmap v393: 158 pgs: 158 active+clean; 583 KiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 474 op/s 2024-09-18T08:11:03.906 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:11:03 smithi050 ceph-mon[32371]: from='mgr.24470 172.21.15.7:0/2594390261' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-18T08:11:03.966 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:11:03 smithi007 ceph-mon[29459]: pgmap v393: 158 pgs: 158 active+clean; 583 KiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 474 op/s 2024-09-18T08:11:03.967 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:11:03 smithi007 ceph-mon[29459]: from='mgr.24470 172.21.15.7:0/2594390261' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-18T08:11:03.967 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:11:03 smithi007 ceph-mon[39249]: pgmap v393: 158 pgs: 158 active+clean; 583 KiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 474 op/s 2024-09-18T08:11:03.967 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:11:03 smithi007 ceph-mon[39249]: from='mgr.24470 172.21.15.7:0/2594390261' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-18T08:11:03.990 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5 2024-09-18T08:11:04.017 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:11:04.017 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:11:04.018 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5' 2024-09-18T08:11:04.118 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7 2024-09-18T08:11:04.145 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:11:04.146 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:11:04.147 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7' 2024-09-18T08:11:04.248 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1 2024-09-18T08:11:04.272 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:11:04.273 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:11:04.273 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1' 2024-09-18T08:11:04.374 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2 2024-09-18T08:11:04.397 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:11:04.397 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:11:04.398 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2' 2024-09-18T08:11:04.498 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7 2024-09-18T08:11:04.525 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:11:04.526 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:11:04.527 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7' 2024-09-18T08:11:04.627 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7 2024-09-18T08:11:04.654 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:11:04.654 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:11:04.655 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7' 2024-09-18T08:11:04.756 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0 2024-09-18T08:11:04.777 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:11:04.778 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:11:04.778 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0' 2024-09-18T08:11:04.793 INFO:teuthology.orchestra.run.smithi007.stderr:no valid command found; 10 closest matches: 2024-09-18T08:11:04.793 INFO:teuthology.orchestra.run.smithi007.stderr:0 2024-09-18T08:11:04.793 INFO:teuthology.orchestra.run.smithi007.stderr:1 2024-09-18T08:11:04.793 INFO:teuthology.orchestra.run.smithi007.stderr:2 2024-09-18T08:11:04.794 INFO:teuthology.orchestra.run.smithi007.stderr:abort 2024-09-18T08:11:04.794 INFO:teuthology.orchestra.run.smithi007.stderr:assert 2024-09-18T08:11:04.794 INFO:teuthology.orchestra.run.smithi007.stderr:bench [] [] [] [] 2024-09-18T08:11:04.794 INFO:teuthology.orchestra.run.smithi007.stderr:bluefs debug_inject_read_zeros 2024-09-18T08:11:04.794 INFO:teuthology.orchestra.run.smithi007.stderr:bluefs files list 2024-09-18T08:11:04.794 INFO:teuthology.orchestra.run.smithi007.stderr:bluefs stats 2024-09-18T08:11:04.794 INFO:teuthology.orchestra.run.smithi007.stderr:bluestore allocator dump block 2024-09-18T08:11:04.794 INFO:teuthology.orchestra.run.smithi007.stderr:Error EINVAL: invalid command 2024-09-18T08:11:04.878 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid 779349ca-7593-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-18T08:11:04.879 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5 2024-09-18T08:11:04.905 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:11:04.906 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:11:04.906 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5' 2024-09-18T08:11:04.993 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/779349ca-7593-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T08:11:05.007 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1 2024-09-18T08:11:05.029 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:11:05.030 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:11:05.030 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1' 2024-09-18T08:11:05.131 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1 2024-09-18T08:11:05.153 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:11:05.154 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:11:05.154 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1' 2024-09-18T08:11:05.255 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6 2024-09-18T08:11:05.267 INFO:teuthology.orchestra.run.smithi007.stderr:pool 'unique_pool_1' removed 2024-09-18T08:11:05.285 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:11:05.286 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:11:05.286 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6' 2024-09-18T08:11:05.387 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2 2024-09-18T08:11:05.445 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:11:05.446 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:11:05.447 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2' 2024-09-18T08:11:05.457 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-09-18T08:11:05.457 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-09-18T08:11:05.458 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid 779349ca-7593-11ef-bceb-c7b262605968 -- ceph tell osd.2 'trim stale osdmaps' 2024-09-18T08:11:05.520 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:11:05 smithi007 ceph-mon[29459]: from='client.? 172.21.15.7:0/1344574282' entity='client.admin' cmd={"prefix": "osd pool rm", "pool": "unique_pool_1", "pool2": "unique_pool_1", "yes_i_really_really_mean_it": true} : dispatch 2024-09-18T08:11:05.520 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:11:05 smithi007 ceph-mon[29459]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool rm", "pool": "unique_pool_1", "pool2": "unique_pool_1", "yes_i_really_really_mean_it": true} : dispatch 2024-09-18T08:11:05.521 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:11:05 smithi007 ceph-mon[39249]: from='client.? 172.21.15.7:0/1344574282' entity='client.admin' cmd={"prefix": "osd pool rm", "pool": "unique_pool_1", "pool2": "unique_pool_1", "yes_i_really_really_mean_it": true} : dispatch 2024-09-18T08:11:05.521 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:11:05 smithi007 ceph-mon[39249]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool rm", "pool": "unique_pool_1", "pool2": "unique_pool_1", "yes_i_really_really_mean_it": true} : dispatch 2024-09-18T08:11:05.547 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1 2024-09-18T08:11:05.569 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:11:05.570 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:11:05.570 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1' 2024-09-18T08:11:05.655 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:11:05 smithi050 ceph-mon[32371]: from='client.? 172.21.15.7:0/1344574282' entity='client.admin' cmd={"prefix": "osd pool rm", "pool": "unique_pool_1", "pool2": "unique_pool_1", "yes_i_really_really_mean_it": true} : dispatch 2024-09-18T08:11:05.656 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:11:05 smithi050 ceph-mon[32371]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool rm", "pool": "unique_pool_1", "pool2": "unique_pool_1", "yes_i_really_really_mean_it": true} : dispatch 2024-09-18T08:11:05.671 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4 2024-09-18T08:11:05.694 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:11:05.695 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:11:05.695 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4' 2024-09-18T08:11:05.796 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7 2024-09-18T08:11:05.820 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:11:05.820 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:11:05.821 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7' 2024-09-18T08:11:05.868 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid 779349ca-7593-11ef-bceb-c7b262605968 -- ceph osd set nodeep-scrub 2024-09-18T08:11:05.922 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6 2024-09-18T08:11:05.945 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:11:05.945 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:11:05.946 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6' 2024-09-18T08:11:06.046 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7 2024-09-18T08:11:06.049 INFO:teuthology.task.full_sequential:In full_sequential, running task radosbench... 2024-09-18T08:11:06.050 INFO:tasks.radosbench:Beginning radosbench... 2024-09-18T08:11:06.050 INFO:tasks.cephadm.ceph_manager.ceph:creating pool_name unique_pool_2 2024-09-18T08:11:06.050 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid 779349ca-7593-11ef-bceb-c7b262605968 -- ceph osd pool create unique_pool_2 16 2024-09-18T08:11:06.070 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:11:06.070 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:11:06.071 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7' 2024-09-18T08:11:06.172 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1 2024-09-18T08:11:06.193 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:11:06.194 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:11:06.194 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1' 2024-09-18T08:11:06.295 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5 2024-09-18T08:11:06.319 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:11:06.320 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:11:06.320 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5' 2024-09-18T08:11:06.415 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:11:06 smithi007 ceph-mon[39249]: pgmap v394: 158 pgs: 158 active+clean; 583 KiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 474 op/s 2024-09-18T08:11:06.415 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:11:06 smithi007 ceph-mon[39249]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "unique_pool_1", "pool2": "unique_pool_1", "yes_i_really_really_mean_it": true}]': finished 2024-09-18T08:11:06.415 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:11:06 smithi007 ceph-mon[39249]: osdmap e231: 8 total, 8 up, 8 in 2024-09-18T08:11:06.416 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:11:06 smithi007 ceph-mon[29459]: pgmap v394: 158 pgs: 158 active+clean; 583 KiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 474 op/s 2024-09-18T08:11:06.416 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:11:06 smithi007 ceph-mon[29459]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "unique_pool_1", "pool2": "unique_pool_1", "yes_i_really_really_mean_it": true}]': finished 2024-09-18T08:11:06.416 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:11:06 smithi007 ceph-mon[29459]: osdmap e231: 8 total, 8 up, 8 in 2024-09-18T08:11:06.421 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4 2024-09-18T08:11:06.445 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:11:06.446 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:11:06.446 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4' 2024-09-18T08:11:06.521 INFO:teuthology.orchestra.run.smithi007.stdout:ERROR: (22) Invalid argument 2024-09-18T08:11:06.521 INFO:teuthology.orchestra.run.smithi007.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-18T08:11:06.547 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1 2024-09-18T08:11:06.571 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:11:06.572 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:11:06.572 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1' 2024-09-18T08:11:06.656 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:11:06 smithi050 ceph-mon[32371]: pgmap v394: 158 pgs: 158 active+clean; 583 KiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 474 op/s 2024-09-18T08:11:06.656 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:11:06 smithi050 ceph-mon[32371]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "unique_pool_1", "pool2": "unique_pool_1", "yes_i_really_really_mean_it": true}]': finished 2024-09-18T08:11:06.656 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:11:06 smithi050 ceph-mon[32371]: osdmap e231: 8 total, 8 up, 8 in 2024-09-18T08:11:06.672 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1 2024-09-18T08:11:06.694 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:11:06.695 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:11:06.695 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1' 2024-09-18T08:11:06.783 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/779349ca-7593-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T08:11:06.796 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5 2024-09-18T08:11:06.799 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/779349ca-7593-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T08:11:06.823 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:11:06.823 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:11:06.824 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5' 2024-09-18T08:11:06.924 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2 2024-09-18T08:11:06.947 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:11:06.947 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:11:06.948 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2' 2024-09-18T08:11:06.980 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Sep 18 08:11:06 smithi007 ceph-779349ca-7593-11ef-bceb-c7b262605968-mgr-y[29688]: ::ffff:172.21.15.50 - - [18/Sep/2024:08:11:06] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-09-18T08:11:07.049 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2 2024-09-18T08:11:07.071 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:11:07.072 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:11:07.072 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2' 2024-09-18T08:11:07.174 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0 2024-09-18T08:11:07.196 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:11:07.197 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:11:07.197 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0' 2024-09-18T08:11:07.298 DEBUG:teuthology.orchestra.run.smithi050:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid 779349ca-7593-11ef-bceb-c7b262605968 -- ceph daemon osd.4 dump_ops_in_flight 2024-09-18T08:11:07.299 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4 2024-09-18T08:11:07.325 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/779349ca-7593-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T08:11:07.326 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:11:07.327 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:11:07.327 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4' 2024-09-18T08:11:07.359 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/779349ca-7593-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T08:11:07.427 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1 2024-09-18T08:11:07.450 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:11:07.451 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:11:07.451 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1' 2024-09-18T08:11:07.552 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0 2024-09-18T08:11:07.569 INFO:teuthology.orchestra.run.smithi050.stderr:Inferring config /var/lib/ceph/779349ca-7593-11ef-bceb-c7b262605968/mon.b/config 2024-09-18T08:11:07.574 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:11:07.575 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:11:07.575 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0' 2024-09-18T08:11:07.676 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6 2024-09-18T08:11:07.699 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:11:07.700 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:11:07.700 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6' 2024-09-18T08:11:07.801 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3 2024-09-18T08:11:07.825 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:11:07.826 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:11:07.826 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3' 2024-09-18T08:11:07.927 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2 2024-09-18T08:11:07.949 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:11:07.950 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:11:07.950 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2' 2024-09-18T08:11:08.051 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2 2024-09-18T08:11:08.060 INFO:teuthology.orchestra.run.smithi050.stdout:ERROR: (22) Invalid argument 2024-09-18T08:11:08.060 INFO:teuthology.orchestra.run.smithi050.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-18T08:11:08.074 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:11:08.075 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:11:08.075 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2' 2024-09-18T08:11:08.177 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1 2024-09-18T08:11:08.199 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:11:08.199 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:11:08.200 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1' 2024-09-18T08:11:08.269 INFO:teuthology.orchestra.run.smithi007.stderr:pool 'unique_pool_2' created 2024-09-18T08:11:08.301 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2 2024-09-18T08:11:08.326 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:11:08.327 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:11:08.327 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2' 2024-09-18T08:11:08.338 INFO:teuthology.orchestra.run.smithi007.stdout:osd.0: {} 2024-09-18T08:11:08.339 INFO:teuthology.orchestra.run.smithi007.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-18T08:11:08.358 INFO:teuthology.orchestra.run.smithi007.stdout:osd.1: {} 2024-09-18T08:11:08.358 INFO:teuthology.orchestra.run.smithi007.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-18T08:11:08.374 INFO:teuthology.orchestra.run.smithi007.stdout:osd.2: {} 2024-09-18T08:11:08.374 INFO:teuthology.orchestra.run.smithi007.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-18T08:11:08.391 INFO:teuthology.orchestra.run.smithi007.stdout:osd.3: {} 2024-09-18T08:11:08.391 INFO:teuthology.orchestra.run.smithi007.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-18T08:11:08.408 INFO:teuthology.orchestra.run.smithi007.stdout:osd.4: {} 2024-09-18T08:11:08.408 INFO:teuthology.orchestra.run.smithi007.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-18T08:11:08.425 INFO:teuthology.orchestra.run.smithi007.stdout:osd.5: {} 2024-09-18T08:11:08.425 INFO:teuthology.orchestra.run.smithi007.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-18T08:11:08.428 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1 2024-09-18T08:11:08.441 INFO:teuthology.orchestra.run.smithi007.stdout:osd.6: {} 2024-09-18T08:11:08.442 INFO:teuthology.orchestra.run.smithi007.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-18T08:11:08.451 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:11:08.452 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:11:08.453 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1' 2024-09-18T08:11:08.461 INFO:teuthology.orchestra.run.smithi007.stdout:osd.7: {} 2024-09-18T08:11:08.461 INFO:teuthology.orchestra.run.smithi007.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-18T08:11:08.462 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:11:08 smithi007 ceph-mon[29459]: pgmap v396: 142 pgs: 142 active+clean; 583 KiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 511 B/s rd, 0 op/s 2024-09-18T08:11:08.462 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:11:08 smithi007 ceph-mon[29459]: from='client.? 172.21.15.7:0/536683699' entity='client.admin' cmd={"prefix": "osd pool create", "pool": "unique_pool_2", "pg_num": 16} : dispatch 2024-09-18T08:11:08.463 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:11:08 smithi007 ceph-mon[39249]: pgmap v396: 142 pgs: 142 active+clean; 583 KiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 511 B/s rd, 0 op/s 2024-09-18T08:11:08.463 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:11:08 smithi007 ceph-mon[39249]: from='client.? 172.21.15.7:0/536683699' entity='client.admin' cmd={"prefix": "osd pool create", "pool": "unique_pool_2", "pg_num": 16} : dispatch 2024-09-18T08:11:08.554 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7 2024-09-18T08:11:08.578 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:11:08.578 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:11:08.579 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7' 2024-09-18T08:11:08.655 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:11:08 smithi050 ceph-mon[32371]: pgmap v396: 142 pgs: 142 active+clean; 583 KiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 511 B/s rd, 0 op/s 2024-09-18T08:11:08.655 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:11:08 smithi050 ceph-mon[32371]: from='client.? 172.21.15.7:0/536683699' entity='client.admin' cmd={"prefix": "osd pool create", "pool": "unique_pool_2", "pg_num": 16} : dispatch 2024-09-18T08:11:08.680 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7 2024-09-18T08:11:08.703 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:11:08.704 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:11:08.704 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7' 2024-09-18T08:11:08.805 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6 2024-09-18T08:11:08.828 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:11:08.829 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:11:08.830 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6' 2024-09-18T08:11:08.839 DEBUG:teuthology.orchestra.run.smithi050:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid 779349ca-7593-11ef-bceb-c7b262605968 -- ceph daemon osd.4 dump_blocked_ops 2024-09-18T08:11:08.930 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3 2024-09-18T08:11:08.953 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:11:08.954 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:11:08.954 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3' 2024-09-18T08:11:09.055 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1 2024-09-18T08:11:09.078 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:11:09.078 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:11:09.079 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1' 2024-09-18T08:11:09.145 INFO:teuthology.orchestra.run.smithi050.stderr:Inferring config /var/lib/ceph/779349ca-7593-11ef-bceb-c7b262605968/mon.b/config 2024-09-18T08:11:09.179 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5 2024-09-18T08:11:09.201 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:11:09.202 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:11:09.202 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5' 2024-09-18T08:11:09.303 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7 2024-09-18T08:11:09.325 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:11:09.326 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:11:09.326 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7' 2024-09-18T08:11:09.427 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3 2024-09-18T08:11:09.430 INFO:journalctl@ceph.iscsi.iscsi.a.smithi050.stdout:Sep 18 08:11:09 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-iscsi-iscsi-a[57782]: debug there is no tcmu-runner data available 2024-09-18T08:11:09.431 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:11:09 smithi050 ceph-mon[32371]: from='client.? 172.21.15.7:0/536683699' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "unique_pool_2", "pg_num": 16}]': finished 2024-09-18T08:11:09.431 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:11:09 smithi050 ceph-mon[32371]: osdmap e232: 8 total, 8 up, 8 in 2024-09-18T08:11:09.431 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:11:09 smithi050 ceph-mon[32371]: from='client.? 172.21.15.7:0/1940580329' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-18T08:11:09.451 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:11:09.451 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:11:09.452 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3' 2024-09-18T08:11:09.538 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:11:09 smithi007 ceph-mon[29459]: from='client.? 172.21.15.7:0/536683699' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "unique_pool_2", "pg_num": 16}]': finished 2024-09-18T08:11:09.538 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:11:09 smithi007 ceph-mon[29459]: osdmap e232: 8 total, 8 up, 8 in 2024-09-18T08:11:09.538 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:11:09 smithi007 ceph-mon[29459]: from='client.? 172.21.15.7:0/1940580329' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-18T08:11:09.538 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:11:09 smithi007 ceph-mon[39249]: from='client.? 172.21.15.7:0/536683699' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "unique_pool_2", "pg_num": 16}]': finished 2024-09-18T08:11:09.538 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:11:09 smithi007 ceph-mon[39249]: osdmap e232: 8 total, 8 up, 8 in 2024-09-18T08:11:09.538 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:11:09 smithi007 ceph-mon[39249]: from='client.? 172.21.15.7:0/1940580329' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-18T08:11:09.553 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3 2024-09-18T08:11:09.577 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:11:09.577 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:11:09.578 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3' 2024-09-18T08:11:09.588 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid 779349ca-7593-11ef-bceb-c7b262605968 -- ceph osd pool application enable unique_pool_2 rados --yes-i-really-mean-it || true 2024-09-18T08:11:09.679 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4 2024-09-18T08:11:09.703 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:11:09.703 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:11:09.704 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4' 2024-09-18T08:11:09.799 INFO:teuthology.orchestra.run.smithi007.stderr:no valid command found; 10 closest matches: 2024-09-18T08:11:09.799 INFO:teuthology.orchestra.run.smithi007.stderr:0 2024-09-18T08:11:09.799 INFO:teuthology.orchestra.run.smithi007.stderr:1 2024-09-18T08:11:09.799 INFO:teuthology.orchestra.run.smithi007.stderr:2 2024-09-18T08:11:09.799 INFO:teuthology.orchestra.run.smithi007.stderr:abort 2024-09-18T08:11:09.799 INFO:teuthology.orchestra.run.smithi007.stderr:assert 2024-09-18T08:11:09.800 INFO:teuthology.orchestra.run.smithi007.stderr:bench [] [] [] [] 2024-09-18T08:11:09.800 INFO:teuthology.orchestra.run.smithi007.stderr:bluefs debug_inject_read_zeros 2024-09-18T08:11:09.800 INFO:teuthology.orchestra.run.smithi007.stderr:bluefs files list 2024-09-18T08:11:09.800 INFO:teuthology.orchestra.run.smithi007.stderr:bluefs stats 2024-09-18T08:11:09.800 INFO:teuthology.orchestra.run.smithi007.stderr:bluestore allocator dump block 2024-09-18T08:11:09.800 INFO:teuthology.orchestra.run.smithi007.stderr:Error EINVAL: invalid command 2024-09-18T08:11:09.804 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4 2024-09-18T08:11:09.827 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:11:09.828 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:11:09.828 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4' 2024-09-18T08:11:09.928 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2024-09-18T08:11:09.928 INFO:teuthology.orchestra.run.smithi050.stdout: "ops": [], 2024-09-18T08:11:09.928 INFO:teuthology.orchestra.run.smithi050.stdout: "complaint_time": 30, 2024-09-18T08:11:09.928 INFO:teuthology.orchestra.run.smithi050.stdout: "num_blocked_ops": 0 2024-09-18T08:11:09.928 INFO:teuthology.orchestra.run.smithi050.stdout:} 2024-09-18T08:11:09.929 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1 2024-09-18T08:11:09.953 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:11:09.953 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:11:09.953 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1' 2024-09-18T08:11:10.054 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4 2024-09-18T08:11:10.077 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:11:10.078 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:11:10.078 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4' 2024-09-18T08:11:10.179 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7 2024-09-18T08:11:10.201 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:11:10.202 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:11:10.202 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7' 2024-09-18T08:11:10.303 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0 2024-09-18T08:11:10.325 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:11:10.325 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:11:10.326 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0' 2024-09-18T08:11:10.426 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6 2024-09-18T08:11:10.450 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:11:10.450 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:11:10.451 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6' 2024-09-18T08:11:10.480 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:11:10 smithi007 ceph-mon[29459]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-18T08:11:10.480 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:11:10 smithi007 ceph-mon[29459]: pgmap v398: 158 pgs: 13 creating+peering, 3 unknown, 142 active+clean; 583 KiB data, 2.5 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-18T08:11:10.480 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:11:10 smithi007 ceph-mon[29459]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-18T08:11:10.480 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:11:10 smithi007 ceph-mon[29459]: osdmap e233: 8 total, 8 up, 8 in 2024-09-18T08:11:10.481 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:11:10 smithi007 ceph-mon[39249]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-18T08:11:10.481 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:11:10 smithi007 ceph-mon[39249]: pgmap v398: 158 pgs: 13 creating+peering, 3 unknown, 142 active+clean; 583 KiB data, 2.5 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-18T08:11:10.481 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:11:10 smithi007 ceph-mon[39249]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-18T08:11:10.481 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:11:10 smithi007 ceph-mon[39249]: osdmap e233: 8 total, 8 up, 8 in 2024-09-18T08:11:10.551 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2 2024-09-18T08:11:10.565 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:11:10 smithi050 ceph-mon[32371]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-18T08:11:10.565 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:11:10 smithi050 ceph-mon[32371]: pgmap v398: 158 pgs: 13 creating+peering, 3 unknown, 142 active+clean; 583 KiB data, 2.5 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-18T08:11:10.565 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:11:10 smithi050 ceph-mon[32371]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-18T08:11:10.565 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:11:10 smithi050 ceph-mon[32371]: osdmap e233: 8 total, 8 up, 8 in 2024-09-18T08:11:10.574 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:11:10.575 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:11:10.575 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2' 2024-09-18T08:11:10.676 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3 2024-09-18T08:11:10.699 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:11:10.700 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:11:10.700 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3' 2024-09-18T08:11:10.802 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4 2024-09-18T08:11:10.818 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-09-18T08:11:10.818 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-09-18T08:11:10.819 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid 779349ca-7593-11ef-bceb-c7b262605968 -- ceph tell osd.3 'trim stale osdmaps' 2024-09-18T08:11:10.825 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:11:10.826 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:11:10.826 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4' 2024-09-18T08:11:10.878 DEBUG:teuthology.orchestra.run.smithi050:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid 779349ca-7593-11ef-bceb-c7b262605968 -- ceph daemon osd.4 dump_historic_ops 2024-09-18T08:11:10.927 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0 2024-09-18T08:11:10.951 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:11:10.952 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:11:10.952 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0' 2024-09-18T08:11:11.053 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5 2024-09-18T08:11:11.077 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:11:11.078 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:11:11.078 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5' 2024-09-18T08:11:11.136 INFO:teuthology.orchestra.run.smithi050.stderr:Inferring config /var/lib/ceph/779349ca-7593-11ef-bceb-c7b262605968/mon.b/config 2024-09-18T08:11:11.179 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6 2024-09-18T08:11:11.201 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:11:11.202 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:11:11.202 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6' 2024-09-18T08:11:11.290 INFO:teuthology.orchestra.run.smithi007.stderr:nodeep-scrub is set 2024-09-18T08:11:11.302 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:11:11 smithi007 ceph-mon[29459]: purged_snaps scrub starts 2024-09-18T08:11:11.302 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:11:11 smithi007 ceph-mon[29459]: purged_snaps scrub ok 2024-09-18T08:11:11.303 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:11:11 smithi007 ceph-mon[29459]: osdmap e234: 8 total, 8 up, 8 in 2024-09-18T08:11:11.303 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:11:11 smithi007 ceph-mon[29459]: from='client.? 172.21.15.7:0/2107740216' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-18T08:11:11.303 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:11:11 smithi007 ceph-mon[29459]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-18T08:11:11.303 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:11:11 smithi007 ceph-mon[39249]: purged_snaps scrub starts 2024-09-18T08:11:11.304 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:11:11 smithi007 ceph-mon[39249]: purged_snaps scrub ok 2024-09-18T08:11:11.304 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:11:11 smithi007 ceph-mon[39249]: osdmap e234: 8 total, 8 up, 8 in 2024-09-18T08:11:11.304 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:11:11 smithi007 ceph-mon[39249]: from='client.? 172.21.15.7:0/2107740216' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-18T08:11:11.304 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:11:11 smithi007 ceph-mon[39249]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-18T08:11:11.304 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5 2024-09-18T08:11:11.327 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:11:11.328 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:11:11.328 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5' 2024-09-18T08:11:11.429 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7 2024-09-18T08:11:11.489 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:11:11.490 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:11:11.490 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7' 2024-09-18T08:11:11.519 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:11:11 smithi050 ceph-mon[32371]: purged_snaps scrub starts 2024-09-18T08:11:11.519 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:11:11 smithi050 ceph-mon[32371]: purged_snaps scrub ok 2024-09-18T08:11:11.519 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:11:11 smithi050 ceph-mon[32371]: osdmap e234: 8 total, 8 up, 8 in 2024-09-18T08:11:11.519 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:11:11 smithi050 ceph-mon[32371]: from='client.? 172.21.15.7:0/2107740216' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-18T08:11:11.519 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:11:11 smithi050 ceph-mon[32371]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-18T08:11:11.591 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1 2024-09-18T08:11:11.614 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:11:11.615 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:11:11.615 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1' 2024-09-18T08:11:11.625 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2024-09-18T08:11:11.625 INFO:teuthology.orchestra.run.smithi050.stdout: "size": 20, 2024-09-18T08:11:11.625 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 600, 2024-09-18T08:11:11.625 INFO:teuthology.orchestra.run.smithi050.stdout: "ops": [ 2024-09-18T08:11:11.625 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:11:11.625 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.24356.0:1183 5.1f 5:f95f44c2:::notify.0:head [watch reconnect cookie 94700888679552 gen 1] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e101)", 2024-09-18T08:11:11.625 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-09-18T08:04:42.950623+0000", 2024-09-18T08:11:11.625 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 388.67028125100001, 2024-09-18T08:11:11.625 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.35355955700000002, 2024-09-18T08:11:11.625 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-09-18T08:11:11.625 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-09-18T08:11:11.625 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "started", 2024-09-18T08:11:11.625 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2024-09-18T08:11:11.626 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.24356", 2024-09-18T08:11:11.626 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "172.21.15.7:0/219168083", 2024-09-18T08:11:11.626 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 1183 2024-09-18T08:11:11.626 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:11:11.626 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-09-18T08:11:11.626 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:11:11.626 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-09-18T08:11:11.626 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:42.950623+0000", 2024-09-18T08:11:11.626 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-18T08:11:11.626 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:11:11.626 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:11:11.626 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-09-18T08:11:11.626 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:42.950623+0000", 2024-09-18T08:11:11.627 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-18T08:11:11.627 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:11:11.627 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:11:11.627 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-09-18T08:11:11.627 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:42.950624+0000", 2024-09-18T08:11:11.627 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 8.6499999999999998e-07 2024-09-18T08:11:11.627 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:11:11.627 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:11:11.627 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-09-18T08:11:11.627 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:42.950628+0000", 2024-09-18T08:11:11.627 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4.0879999999999997e-06 2024-09-18T08:11:11.627 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:11:11.627 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:11:11.628 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-09-18T08:11:11.628 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:42.950644+0000", 2024-09-18T08:11:11.628 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.6507e-05 2024-09-18T08:11:11.628 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:11:11.628 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:11:11.628 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-09-18T08:11:11.628 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:42.950651+0000", 2024-09-18T08:11:11.628 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 6.5599999999999999e-06 2024-09-18T08:11:11.628 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:11:11.628 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:11:11.628 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-18T08:11:11.628 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:42.950713+0000", 2024-09-18T08:11:11.629 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 6.2360000000000006e-05 2024-09-18T08:11:11.629 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:11:11.629 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:11:11.629 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "waiting for peered", 2024-09-18T08:11:11.629 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:42.950724+0000", 2024-09-18T08:11:11.629 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.0329999999999999e-05 2024-09-18T08:11:11.629 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:11:11.629 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:11:11.629 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-18T08:11:11.629 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:43.301407+0000", 2024-09-18T08:11:11.629 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.35068303499999998 2024-09-18T08:11:11.629 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:11:11.629 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:11:11.630 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-09-18T08:11:11.630 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:43.301697+0000", 2024-09-18T08:11:11.630 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000289938 2024-09-18T08:11:11.630 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:11:11.630 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:11:11.630 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-09-18T08:11:11.630 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:43.304183+0000", 2024-09-18T08:11:11.630 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0024858739999999999 2024-09-18T08:11:11.630 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-18T08:11:11.630 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-09-18T08:11:11.630 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-18T08:11:11.630 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:11:11.630 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:11:11.631 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.24356.0:1184 4.5 4:a60f9269:lc::lua_package_allowlist:head [watch reconnect cookie 94700889442560 gen 1] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e101)", 2024-09-18T08:11:11.631 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-09-18T08:04:42.950665+0000", 2024-09-18T08:11:11.631 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 388.67023887200003, 2024-09-18T08:11:11.631 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.35372086800000002, 2024-09-18T08:11:11.631 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-09-18T08:11:11.631 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-09-18T08:11:11.631 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "started", 2024-09-18T08:11:11.631 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2024-09-18T08:11:11.631 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.24356", 2024-09-18T08:11:11.631 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "172.21.15.7:0/219168083", 2024-09-18T08:11:11.631 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 1184 2024-09-18T08:11:11.631 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:11:11.631 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-09-18T08:11:11.632 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:11:11.632 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-09-18T08:11:11.632 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:42.950665+0000", 2024-09-18T08:11:11.632 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-18T08:11:11.632 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:11:11.632 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:11:11.632 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-09-18T08:11:11.632 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:42.950665+0000", 2024-09-18T08:11:11.632 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-18T08:11:11.632 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:11:11.632 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:11:11.632 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-09-18T08:11:11.632 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:42.950666+0000", 2024-09-18T08:11:11.633 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 6.1099999999999995e-07 2024-09-18T08:11:11.633 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:11:11.633 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:11:11.633 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-09-18T08:11:11.633 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:42.950668+0000", 2024-09-18T08:11:11.633 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 2.514e-06 2024-09-18T08:11:11.633 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:11:11.633 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:11:11.633 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-09-18T08:11:11.633 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:42.950683+0000", 2024-09-18T08:11:11.633 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.4827000000000001e-05 2024-09-18T08:11:11.633 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:11:11.633 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:11:11.634 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-09-18T08:11:11.634 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:42.950689+0000", 2024-09-18T08:11:11.634 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 6.0789999999999997e-06 2024-09-18T08:11:11.634 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:11:11.634 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:11:11.634 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-18T08:11:11.634 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:42.950753+0000", 2024-09-18T08:11:11.634 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 6.3466999999999994e-05 2024-09-18T08:11:11.634 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:11:11.634 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:11:11.634 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "waiting for peered", 2024-09-18T08:11:11.634 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:42.950763+0000", 2024-09-18T08:11:11.634 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.0193e-05 2024-09-18T08:11:11.634 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:11:11.635 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:11:11.635 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-18T08:11:11.635 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:43.301959+0000", 2024-09-18T08:11:11.635 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.35119641299999999 2024-09-18T08:11:11.635 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:11:11.635 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:11:11.635 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-09-18T08:11:11.635 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:43.302091+0000", 2024-09-18T08:11:11.635 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00013126500000000001 2024-09-18T08:11:11.635 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:11:11.635 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:11:11.635 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-09-18T08:11:11.635 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:43.304386+0000", 2024-09-18T08:11:11.636 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0022954989999999999 2024-09-18T08:11:11.636 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-18T08:11:11.636 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-09-18T08:11:11.636 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-18T08:11:11.636 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:11:11.636 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:11:11.636 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.15216.0:86225 8.5 8:ae8df458:::benchmark_data_smithi007_101583_object86224:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e205)", 2024-09-18T08:11:11.636 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-09-18T08:09:12.178973+0000", 2024-09-18T08:11:11.636 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 119.44193081, 2024-09-18T08:11:11.636 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.99102768600000002, 2024-09-18T08:11:11.636 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-09-18T08:11:11.636 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-09-18T08:11:11.636 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "reached pg", 2024-09-18T08:11:11.636 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2024-09-18T08:11:11.637 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.15216", 2024-09-18T08:11:11.637 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "172.21.15.7:0/2596041177", 2024-09-18T08:11:11.637 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 86225 2024-09-18T08:11:11.637 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:11:11.637 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-09-18T08:11:11.637 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:11:11.637 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-09-18T08:11:11.637 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:12.178973+0000", 2024-09-18T08:11:11.637 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-18T08:11:11.637 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:11:11.637 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:11:11.637 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-09-18T08:11:11.637 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:12.178973+0000", 2024-09-18T08:11:11.638 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-18T08:11:11.638 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:11:11.638 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:11:11.638 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-09-18T08:11:11.638 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:12.178975+0000", 2024-09-18T08:11:11.638 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.3969999999999999e-06 2024-09-18T08:11:11.638 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:11:11.638 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:11:11.638 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-09-18T08:11:11.638 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:12.179022+0000", 2024-09-18T08:11:11.638 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4.7580000000000002e-05 2024-09-18T08:11:11.638 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:11:11.638 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:11:11.639 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-09-18T08:11:11.639 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:12.179034+0000", 2024-09-18T08:11:11.639 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.1477e-05 2024-09-18T08:11:11.639 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:11:11.639 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:11:11.639 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-09-18T08:11:11.639 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:12.179040+0000", 2024-09-18T08:11:11.639 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 5.7529999999999998e-06 2024-09-18T08:11:11.639 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:11:11.639 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:11:11.640 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-18T08:11:11.640 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:12.179092+0000", 2024-09-18T08:11:11.640 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 5.2840000000000002e-05 2024-09-18T08:11:11.640 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:11:11.640 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:11:11.641 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "waiting for peered", 2024-09-18T08:11:11.641 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:12.179102+0000", 2024-09-18T08:11:11.641 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 9.5210000000000008e-06 2024-09-18T08:11:11.641 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:11:11.641 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:11:11.641 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-18T08:11:11.641 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:13.169945+0000", 2024-09-18T08:11:11.641 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.99084323600000002 2024-09-18T08:11:11.641 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:11:11.641 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:11:11.641 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-09-18T08:11:11.641 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:13.170001+0000", 2024-09-18T08:11:11.641 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 5.5881999999999998e-05 2024-09-18T08:11:11.642 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-18T08:11:11.642 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-09-18T08:11:11.642 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-18T08:11:11.642 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:11:11.642 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:11:11.642 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.15216.0:86263 8.5 8:af593c7a:::benchmark_data_smithi007_101583_object86262:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e205)", 2024-09-18T08:11:11.642 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-09-18T08:09:12.204249+0000", 2024-09-18T08:11:11.642 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 119.41665480499999, 2024-09-18T08:11:11.642 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.98012944999999996, 2024-09-18T08:11:11.642 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-09-18T08:11:11.642 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-09-18T08:11:11.642 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T08:11:11.642 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2024-09-18T08:11:11.642 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.15216", 2024-09-18T08:11:11.643 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "172.21.15.7:0/2596041177", 2024-09-18T08:11:11.643 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 86263 2024-09-18T08:11:11.643 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:11:11.643 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-09-18T08:11:11.643 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:11:11.643 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-09-18T08:11:11.643 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:12.204249+0000", 2024-09-18T08:11:11.643 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-18T08:11:11.643 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:11:11.643 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:11:11.643 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-09-18T08:11:11.643 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:12.204249+0000", 2024-09-18T08:11:11.643 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-18T08:11:11.643 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:11:11.644 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:11:11.644 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-09-18T08:11:11.644 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:12.204251+0000", 2024-09-18T08:11:11.644 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.635e-06 2024-09-18T08:11:11.644 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:11:11.644 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:11:11.644 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-09-18T08:11:11.644 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:12.204299+0000", 2024-09-18T08:11:11.644 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4.8026999999999999e-05 2024-09-18T08:11:11.644 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:11:11.644 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:11:11.644 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-09-18T08:11:11.644 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:12.204313+0000", 2024-09-18T08:11:11.645 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.4066000000000001e-05 2024-09-18T08:11:11.645 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:11:11.645 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:11:11.645 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-09-18T08:11:11.645 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:12.204321+0000", 2024-09-18T08:11:11.645 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 8.1839999999999999e-06 2024-09-18T08:11:11.645 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:11:11.645 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:11:11.645 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-18T08:11:11.645 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:12.204392+0000", 2024-09-18T08:11:11.645 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 7.0340999999999996e-05 2024-09-18T08:11:11.645 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:11:11.645 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:11:11.646 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "waiting for peered", 2024-09-18T08:11:11.646 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:12.204403+0000", 2024-09-18T08:11:11.646 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.1654000000000001e-05 2024-09-18T08:11:11.646 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:11:11.646 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:11:11.646 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-18T08:11:11.646 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:13.170414+0000", 2024-09-18T08:11:11.646 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.96601117000000003 2024-09-18T08:11:11.646 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:11:11.646 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:11:11.646 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-09-18T08:11:11.646 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:13.170546+0000", 2024-09-18T08:11:11.646 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00013120100000000001 2024-09-18T08:11:11.646 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:11:11.647 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:11:11.647 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "waiting for subops from 6,7", 2024-09-18T08:11:11.647 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:13.170965+0000", 2024-09-18T08:11:11.647 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00041976200000000001 2024-09-18T08:11:11.647 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:11:11.647 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:11:11.647 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:11:11.647 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:13.181803+0000", 2024-09-18T08:11:11.647 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.010837167999999999 2024-09-18T08:11:11.647 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:11:11.647 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:11:11.647 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:11:11.647 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:13.181864+0000", 2024-09-18T08:11:11.647 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 6.1857000000000001e-05 2024-09-18T08:11:11.648 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:11:11.648 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:11:11.648 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "op_commit", 2024-09-18T08:11:11.648 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:13.184283+0000", 2024-09-18T08:11:11.648 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0024184340000000001 2024-09-18T08:11:11.648 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:11:11.648 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:11:11.648 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "commit_sent", 2024-09-18T08:11:11.648 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:13.184339+0000", 2024-09-18T08:11:11.648 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 5.6378999999999997e-05 2024-09-18T08:11:11.648 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:11:11.648 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:11:11.648 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-09-18T08:11:11.649 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:13.184379+0000", 2024-09-18T08:11:11.649 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 3.9572e-05 2024-09-18T08:11:11.649 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-18T08:11:11.649 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-09-18T08:11:11.649 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-18T08:11:11.649 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:11:11.649 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:11:11.649 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.15216.0:86267 8.5 8:a135397b:::benchmark_data_smithi007_101583_object86266:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e205)", 2024-09-18T08:11:11.649 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-09-18T08:09:12.207725+0000", 2024-09-18T08:11:11.649 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 119.413178755, 2024-09-18T08:11:11.649 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.97676783599999994, 2024-09-18T08:11:11.649 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-09-18T08:11:11.649 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-09-18T08:11:11.650 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T08:11:11.650 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2024-09-18T08:11:11.650 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.15216", 2024-09-18T08:11:11.650 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "172.21.15.7:0/2596041177", 2024-09-18T08:11:11.650 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 86267 2024-09-18T08:11:11.650 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:11:11.650 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-09-18T08:11:11.650 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:11:11.650 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-09-18T08:11:11.650 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:12.207725+0000", 2024-09-18T08:11:11.650 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-18T08:11:11.650 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:11:11.650 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:11:11.650 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-09-18T08:11:11.650 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:12.207725+0000", 2024-09-18T08:11:11.651 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-18T08:11:11.651 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:11:11.651 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:11:11.651 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-09-18T08:11:11.651 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:12.207727+0000", 2024-09-18T08:11:11.651 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.79e-06 2024-09-18T08:11:11.651 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:11:11.651 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:11:11.651 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-09-18T08:11:11.651 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:12.207765+0000", 2024-09-18T08:11:11.651 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 3.7864000000000002e-05 2024-09-18T08:11:11.651 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:11:11.651 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:11:11.651 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-09-18T08:11:11.652 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:12.207783+0000", 2024-09-18T08:11:11.652 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.7779999999999999e-05 2024-09-18T08:11:11.652 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:11:11.652 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:11:11.652 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-09-18T08:11:11.652 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:12.207791+0000", 2024-09-18T08:11:11.652 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 7.9729999999999997e-06 2024-09-18T08:11:11.652 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:11:11.652 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:11:11.652 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-18T08:11:11.652 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:12.207872+0000", 2024-09-18T08:11:11.652 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 8.1318999999999998e-05 2024-09-18T08:11:11.652 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:11:11.653 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:11:11.653 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "waiting for peered", 2024-09-18T08:11:11.653 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:12.207905+0000", 2024-09-18T08:11:11.653 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 3.2694000000000002e-05 2024-09-18T08:11:11.653 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:11:11.653 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:11:11.653 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-18T08:11:11.653 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:13.179332+0000", 2024-09-18T08:11:11.653 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.97142748800000001 2024-09-18T08:11:11.653 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:11:11.653 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:11:11.653 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-09-18T08:11:11.653 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:13.179470+0000", 2024-09-18T08:11:11.653 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00013747400000000001 2024-09-18T08:11:11.653 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:11:11.654 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:11:11.654 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "waiting for subops from 6,7", 2024-09-18T08:11:11.654 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:13.179634+0000", 2024-09-18T08:11:11.654 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00016417600000000001 2024-09-18T08:11:11.654 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:11:11.654 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:11:11.654 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:11:11.654 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:13.184267+0000", 2024-09-18T08:11:11.654 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0046332079999999998 2024-09-18T08:11:11.654 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:11:11.654 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:11:11.654 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "op_commit", 2024-09-18T08:11:11.654 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:13.184386+0000", 2024-09-18T08:11:11.655 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000118993 2024-09-18T08:11:11.655 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:11:11.655 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:11:11.655 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:11:11.655 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:13.184437+0000", 2024-09-18T08:11:11.655 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 5.0988e-05 2024-09-18T08:11:11.655 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:11:11.655 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:11:11.655 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "commit_sent", 2024-09-18T08:11:11.655 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:13.184469+0000", 2024-09-18T08:11:11.655 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 3.1603000000000002e-05 2024-09-18T08:11:11.655 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:11:11.655 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:11:11.655 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-09-18T08:11:11.656 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:13.184493+0000", 2024-09-18T08:11:11.656 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 2.4485999999999999e-05 2024-09-18T08:11:11.656 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-18T08:11:11.656 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-09-18T08:11:11.656 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-18T08:11:11.656 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:11:11.656 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:11:11.656 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.15216.0:86268 8.5 8:aa7096fb:::benchmark_data_smithi007_101583_object86267:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e205)", 2024-09-18T08:11:11.656 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-09-18T08:09:12.208946+0000", 2024-09-18T08:11:11.656 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 119.411958148, 2024-09-18T08:11:11.656 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.980827699, 2024-09-18T08:11:11.656 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-09-18T08:11:11.656 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-09-18T08:11:11.657 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T08:11:11.657 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2024-09-18T08:11:11.657 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.15216", 2024-09-18T08:11:11.657 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "172.21.15.7:0/2596041177", 2024-09-18T08:11:11.657 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 86268 2024-09-18T08:11:11.657 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:11:11.657 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-09-18T08:11:11.657 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:11:11.657 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-09-18T08:11:11.657 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:12.208946+0000", 2024-09-18T08:11:11.657 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-18T08:11:11.658 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:11:11.658 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:11:11.658 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-09-18T08:11:11.658 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:12.208946+0000", 2024-09-18T08:11:11.658 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-18T08:11:11.658 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:11:11.658 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:11:11.658 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-09-18T08:11:11.658 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:12.208947+0000", 2024-09-18T08:11:11.658 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.201e-06 2024-09-18T08:11:11.658 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:11:11.658 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:11:11.659 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-09-18T08:11:11.659 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:12.209036+0000", 2024-09-18T08:11:11.659 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 8.8865999999999999e-05 2024-09-18T08:11:11.659 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:11:11.659 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:11:11.659 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-09-18T08:11:11.659 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:12.209049+0000", 2024-09-18T08:11:11.659 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.2795e-05 2024-09-18T08:11:11.659 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:11:11.659 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:11:11.659 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-09-18T08:11:11.659 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:12.209056+0000", 2024-09-18T08:11:11.659 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 7.1160000000000001e-06 2024-09-18T08:11:11.659 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:11:11.660 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:11:11.660 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-18T08:11:11.660 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:12.209116+0000", 2024-09-18T08:11:11.660 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 5.9517e-05 2024-09-18T08:11:11.660 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:11:11.660 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:11:11.660 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "waiting for peered", 2024-09-18T08:11:11.660 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:12.209125+0000", 2024-09-18T08:11:11.660 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 9.6630000000000005e-06 2024-09-18T08:11:11.660 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:11:11.660 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:11:11.660 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-18T08:11:11.660 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:13.182004+0000", 2024-09-18T08:11:11.661 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.97287885799999996 2024-09-18T08:11:11.661 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:11:11.661 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:11:11.661 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-09-18T08:11:11.661 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:13.182091+0000", 2024-09-18T08:11:11.661 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 8.6907e-05 2024-09-18T08:11:11.661 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:11:11.661 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:11:11.661 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "waiting for subops from 6,7", 2024-09-18T08:11:11.661 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:13.182183+0000", 2024-09-18T08:11:11.661 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 9.2131000000000003e-05 2024-09-18T08:11:11.661 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:11:11.661 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:11:11.661 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "op_commit", 2024-09-18T08:11:11.662 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:13.187041+0000", 2024-09-18T08:11:11.662 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0048579210000000003 2024-09-18T08:11:11.662 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:11:11.662 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:11:11.662 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:11:11.662 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:13.189581+0000", 2024-09-18T08:11:11.662 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.002539707 2024-09-18T08:11:11.662 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:11:11.662 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:11:11.662 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:11:11.662 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:13.189646+0000", 2024-09-18T08:11:11.662 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 6.5699999999999998e-05 2024-09-18T08:11:11.662 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:11:11.663 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:11:11.663 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "commit_sent", 2024-09-18T08:11:11.663 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:13.189714+0000", 2024-09-18T08:11:11.663 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 6.7108999999999997e-05 2024-09-18T08:11:11.663 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:11:11.663 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:11:11.663 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-09-18T08:11:11.663 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:13.189774+0000", 2024-09-18T08:11:11.663 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 6.0207999999999997e-05 2024-09-18T08:11:11.663 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-18T08:11:11.663 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-09-18T08:11:11.663 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-18T08:11:11.663 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:11:11.663 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:11:11.663 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.15216.0:86303 8.5 8:aebc4459:::benchmark_data_smithi007_101583_object86302:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e205)", 2024-09-18T08:11:11.664 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-09-18T08:09:12.250052+0000", 2024-09-18T08:11:11.664 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 119.37085259200001, 2024-09-18T08:11:11.664 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.94992247699999999, 2024-09-18T08:11:11.664 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-09-18T08:11:11.664 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-09-18T08:11:11.664 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T08:11:11.664 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2024-09-18T08:11:11.664 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.15216", 2024-09-18T08:11:11.664 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "172.21.15.7:0/2596041177", 2024-09-18T08:11:11.664 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 86303 2024-09-18T08:11:11.664 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:11:11.664 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-09-18T08:11:11.664 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:11:11.665 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-09-18T08:11:11.665 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:12.250052+0000", 2024-09-18T08:11:11.665 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-18T08:11:11.665 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:11:11.665 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:11:11.665 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-09-18T08:11:11.665 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:12.250052+0000", 2024-09-18T08:11:11.665 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-18T08:11:11.665 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:11:11.665 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:11:11.665 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-09-18T08:11:11.665 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:12.250053+0000", 2024-09-18T08:11:11.665 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.6130000000000001e-06 2024-09-18T08:11:11.665 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:11:11.666 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:11:11.666 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-09-18T08:11:11.666 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:12.250090+0000", 2024-09-18T08:11:11.666 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 3.6307999999999999e-05 2024-09-18T08:11:11.666 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:11:11.666 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:11:11.666 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-09-18T08:11:11.666 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:12.250102+0000", 2024-09-18T08:11:11.666 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.2624e-05 2024-09-18T08:11:11.666 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:11:11.666 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:11:11.666 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-09-18T08:11:11.666 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:12.250109+0000", 2024-09-18T08:11:11.666 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 6.426e-06 2024-09-18T08:11:11.667 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:11:11.667 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:11:11.667 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-18T08:11:11.667 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:12.250173+0000", 2024-09-18T08:11:11.667 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 6.3966000000000004e-05 2024-09-18T08:11:11.667 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:11:11.667 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:11:11.667 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "waiting for peered", 2024-09-18T08:11:11.667 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:12.250184+0000", 2024-09-18T08:11:11.667 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.1401e-05 2024-09-18T08:11:11.667 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:11:11.667 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:11:11.667 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-18T08:11:11.667 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:13.184533+0000", 2024-09-18T08:11:11.668 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.93434866299999997 2024-09-18T08:11:11.668 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:11:11.668 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:11:11.668 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-09-18T08:11:11.668 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:13.184634+0000", 2024-09-18T08:11:11.668 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000101139 2024-09-18T08:11:11.668 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:11:11.668 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:11:11.668 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "waiting for subops from 6,7", 2024-09-18T08:11:11.668 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:13.184793+0000", 2024-09-18T08:11:11.668 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00015932000000000001 2024-09-18T08:11:11.668 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:11:11.668 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:11:11.669 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:11:11.669 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:13.194839+0000", 2024-09-18T08:11:11.669 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.010046144 2024-09-18T08:11:11.669 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:11:11.669 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:11:11.669 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:11:11.669 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:13.194928+0000", 2024-09-18T08:11:11.669 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 8.9037999999999997e-05 2024-09-18T08:11:11.669 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:11:11.669 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:11:11.669 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "op_commit", 2024-09-18T08:11:11.669 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:13.199865+0000", 2024-09-18T08:11:11.669 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0049365879999999996 2024-09-18T08:11:11.670 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:11:11.670 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:11:11.670 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "commit_sent", 2024-09-18T08:11:11.670 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:13.199924+0000", 2024-09-18T08:11:11.670 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 5.9383999999999999e-05 2024-09-18T08:11:11.670 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:11:11.670 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:11:11.670 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-09-18T08:11:11.670 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:13.199974+0000", 2024-09-18T08:11:11.670 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4.9863e-05 2024-09-18T08:11:11.670 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-18T08:11:11.670 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-09-18T08:11:11.670 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-18T08:11:11.671 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:11:11.671 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:11:11.671 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.15216.0:86344 8.5 8:ac157b17:::benchmark_data_smithi007_101583_object86343:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e205)", 2024-09-18T08:11:11.671 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-09-18T08:09:12.295743+0000", 2024-09-18T08:11:11.671 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 119.325160912, 2024-09-18T08:11:11.671 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.92374953400000004, 2024-09-18T08:11:11.671 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-09-18T08:11:11.671 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-09-18T08:11:11.671 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T08:11:11.671 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2024-09-18T08:11:11.671 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.15216", 2024-09-18T08:11:11.671 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "172.21.15.7:0/2596041177", 2024-09-18T08:11:11.671 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 86344 2024-09-18T08:11:11.672 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:11:11.672 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-09-18T08:11:11.672 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:11:11.672 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-09-18T08:11:11.672 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:12.295743+0000", 2024-09-18T08:11:11.672 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-18T08:11:11.672 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:11:11.672 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:11:11.672 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-09-18T08:11:11.672 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:12.295743+0000", 2024-09-18T08:11:11.672 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-18T08:11:11.672 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:11:11.672 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:11:11.672 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-09-18T08:11:11.673 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:12.295745+0000", 2024-09-18T08:11:11.673 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.57e-06 2024-09-18T08:11:11.673 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:11:11.673 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:11:11.673 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-09-18T08:11:11.673 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:12.295795+0000", 2024-09-18T08:11:11.673 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 5.0062999999999998e-05 2024-09-18T08:11:11.673 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:11:11.673 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:11:11.673 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-09-18T08:11:11.673 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:12.295807+0000", 2024-09-18T08:11:11.673 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.1578e-05 2024-09-18T08:11:11.673 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:11:11.674 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:11:11.674 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-09-18T08:11:11.674 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:12.295812+0000", 2024-09-18T08:11:11.674 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 5.8930000000000002e-06 2024-09-18T08:11:11.674 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:11:11.674 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:11:11.674 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-18T08:11:11.674 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:12.295875+0000", 2024-09-18T08:11:11.674 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 6.2168000000000005e-05 2024-09-18T08:11:11.674 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:11:11.674 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:11:11.674 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "waiting for peered", 2024-09-18T08:11:11.674 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:12.295883+0000", 2024-09-18T08:11:11.675 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 7.9969999999999998e-06 2024-09-18T08:11:11.675 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:11:11.675 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:11:11.675 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-18T08:11:11.675 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:13.187072+0000", 2024-09-18T08:11:11.675 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.89118984899999998 2024-09-18T08:11:11.675 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:11:11.675 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:11:11.675 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-09-18T08:11:11.675 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:13.187193+0000", 2024-09-18T08:11:11.675 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00012062599999999999 2024-09-18T08:11:11.675 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:11:11.675 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:11:11.676 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "waiting for subops from 6,7", 2024-09-18T08:11:11.676 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:13.187351+0000", 2024-09-18T08:11:11.676 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00015781500000000001 2024-09-18T08:11:11.676 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:11:11.676 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:11:11.676 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:11:11.676 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:13.194877+0000", 2024-09-18T08:11:11.676 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0075262139999999998 2024-09-18T08:11:11.676 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:11:11.676 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:11:11.676 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:11:11.676 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:13.195031+0000", 2024-09-18T08:11:11.676 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00015384799999999999 2024-09-18T08:11:11.677 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:11:11.677 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:11:11.677 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "op_commit", 2024-09-18T08:11:11.677 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:13.219386+0000", 2024-09-18T08:11:11.677 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.024355226000000001 2024-09-18T08:11:11.677 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:11:11.677 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:11:11.677 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "commit_sent", 2024-09-18T08:11:11.677 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:13.219440+0000", 2024-09-18T08:11:11.677 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 5.3786999999999998e-05 2024-09-18T08:11:11.677 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:11:11.677 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:11:11.677 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-09-18T08:11:11.677 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:13.219493+0000", 2024-09-18T08:11:11.678 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 5.2899999999999998e-05 2024-09-18T08:11:11.678 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-18T08:11:11.678 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-09-18T08:11:11.678 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-18T08:11:11.678 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:11:11.678 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:11:11.678 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.15216.0:86350 8.5 8:a14e8599:::benchmark_data_smithi007_101583_object86349:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e205)", 2024-09-18T08:11:11.678 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-09-18T08:09:12.297534+0000", 2024-09-18T08:11:11.678 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 119.323370428, 2024-09-18T08:11:11.678 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.922055505, 2024-09-18T08:11:11.678 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-09-18T08:11:11.678 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-09-18T08:11:11.678 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T08:11:11.679 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2024-09-18T08:11:11.679 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.15216", 2024-09-18T08:11:11.679 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "172.21.15.7:0/2596041177", 2024-09-18T08:11:11.679 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 86350 2024-09-18T08:11:11.679 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:11:11.679 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-09-18T08:11:11.679 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:11:11.679 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-09-18T08:11:11.679 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:12.297534+0000", 2024-09-18T08:11:11.679 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-18T08:11:11.679 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:11:11.679 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:11:11.679 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-09-18T08:11:11.679 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:12.297534+0000", 2024-09-18T08:11:11.680 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-18T08:11:11.680 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:11:11.680 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:11:11.680 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-09-18T08:11:11.680 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:12.297534+0000", 2024-09-18T08:11:11.680 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 7.0800000000000004e-07 2024-09-18T08:11:11.680 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:11:11.680 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:11:11.680 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-09-18T08:11:11.680 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:12.297606+0000", 2024-09-18T08:11:11.680 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 7.1075000000000006e-05 2024-09-18T08:11:11.680 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:11:11.680 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:11:11.680 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-09-18T08:11:11.681 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:12.297613+0000", 2024-09-18T08:11:11.681 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 7.6260000000000003e-06 2024-09-18T08:11:11.681 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:11:11.681 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:11:11.681 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-09-18T08:11:11.681 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:12.297618+0000", 2024-09-18T08:11:11.681 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4.8550000000000001e-06 2024-09-18T08:11:11.681 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:11:11.681 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:11:11.681 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-18T08:11:11.681 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:12.297660+0000", 2024-09-18T08:11:11.681 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4.1968000000000002e-05 2024-09-18T08:11:11.681 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:11:11.681 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:11:11.682 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "waiting for peered", 2024-09-18T08:11:11.682 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:12.297666+0000", 2024-09-18T08:11:11.682 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 5.806e-06 2024-09-18T08:11:11.682 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:11:11.682 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:11:11.682 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-18T08:11:11.682 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:13.189816+0000", 2024-09-18T08:11:11.682 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.89215055300000001 2024-09-18T08:11:11.682 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:11:11.682 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:11:11.682 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-09-18T08:11:11.682 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:13.190014+0000", 2024-09-18T08:11:11.682 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000197894 2024-09-18T08:11:11.683 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:11:11.683 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:11:11.683 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "waiting for subops from 6,7", 2024-09-18T08:11:11.683 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:13.190197+0000", 2024-09-18T08:11:11.683 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00018230399999999999 2024-09-18T08:11:11.683 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:11:11.683 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:11:11.683 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:11:11.683 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:13.194999+0000", 2024-09-18T08:11:11.683 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0048023019999999996 2024-09-18T08:11:11.683 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:11:11.683 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:11:11.683 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:11:11.683 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:13.197606+0000", 2024-09-18T08:11:11.684 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0026074890000000002 2024-09-18T08:11:11.684 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:11:11.684 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:11:11.684 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "op_commit", 2024-09-18T08:11:11.684 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:13.219502+0000", 2024-09-18T08:11:11.684 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.02189514 2024-09-18T08:11:11.684 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:11:11.684 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:11:11.684 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "commit_sent", 2024-09-18T08:11:11.684 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:13.219546+0000", 2024-09-18T08:11:11.684 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4.4574999999999999e-05 2024-09-18T08:11:11.684 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:11:11.684 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:11:11.685 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-09-18T08:11:11.685 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:13.219589+0000", 2024-09-18T08:11:11.685 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4.3210000000000001e-05 2024-09-18T08:11:11.685 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-18T08:11:11.685 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-09-18T08:11:11.685 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-18T08:11:11.685 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:11:11.685 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:11:11.685 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.15216.0:86366 8.5 8:a8c4536d:::benchmark_data_smithi007_101583_object86365:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e205)", 2024-09-18T08:11:11.685 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-09-18T08:09:12.318202+0000", 2024-09-18T08:11:11.685 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 119.302701752, 2024-09-18T08:11:11.685 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.90148185700000005, 2024-09-18T08:11:11.685 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-09-18T08:11:11.685 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-09-18T08:11:11.686 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T08:11:11.686 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2024-09-18T08:11:11.686 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.15216", 2024-09-18T08:11:11.686 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "172.21.15.7:0/2596041177", 2024-09-18T08:11:11.686 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 86366 2024-09-18T08:11:11.686 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:11:11.686 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-09-18T08:11:11.686 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:11:11.686 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-09-18T08:11:11.686 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:12.318202+0000", 2024-09-18T08:11:11.686 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-18T08:11:11.686 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:11:11.686 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:11:11.687 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-09-18T08:11:11.687 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:12.318202+0000", 2024-09-18T08:11:11.687 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-18T08:11:11.687 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:11:11.687 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:11:11.687 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-09-18T08:11:11.687 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:12.318205+0000", 2024-09-18T08:11:11.687 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 2.074e-06 2024-09-18T08:11:11.687 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:11:11.687 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:11:11.687 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-09-18T08:11:11.687 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:12.318242+0000", 2024-09-18T08:11:11.687 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 3.6996999999999998e-05 2024-09-18T08:11:11.687 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:11:11.688 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:11:11.688 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-09-18T08:11:11.688 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:12.318258+0000", 2024-09-18T08:11:11.688 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.6382e-05 2024-09-18T08:11:11.688 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:11:11.688 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:11:11.688 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-09-18T08:11:11.688 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:12.318266+0000", 2024-09-18T08:11:11.688 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 7.9820000000000008e-06 2024-09-18T08:11:11.688 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:11:11.688 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:11:11.688 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-18T08:11:11.688 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:12.318402+0000", 2024-09-18T08:11:11.688 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000136477 2024-09-18T08:11:11.689 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:11:11.689 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:11:11.689 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "waiting for peered", 2024-09-18T08:11:11.689 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:12.318414+0000", 2024-09-18T08:11:11.689 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.2096000000000001e-05 2024-09-18T08:11:11.689 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:11:11.689 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:11:11.689 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-18T08:11:11.689 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:13.192024+0000", 2024-09-18T08:11:11.689 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.87360952300000005 2024-09-18T08:11:11.689 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:11:11.689 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:11:11.689 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-09-18T08:11:11.690 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:13.192155+0000", 2024-09-18T08:11:11.690 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00013099499999999999 2024-09-18T08:11:11.690 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:11:11.690 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:11:11.690 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "waiting for subops from 6,7", 2024-09-18T08:11:11.690 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:13.192313+0000", 2024-09-18T08:11:11.690 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00015802500000000001 2024-09-18T08:11:11.690 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:11:11.690 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:11:11.690 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:11:11.690 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:13.195063+0000", 2024-09-18T08:11:11.690 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0027500279999999998 2024-09-18T08:11:11.690 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:11:11.690 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:11:11.691 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:11:11.691 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:13.200043+0000", 2024-09-18T08:11:11.691 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0049800929999999997 2024-09-18T08:11:11.691 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:11:11.691 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:11:11.691 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "op_commit", 2024-09-18T08:11:11.691 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:13.219598+0000", 2024-09-18T08:11:11.691 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.019554768 2024-09-18T08:11:11.691 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:11:11.691 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:11:11.691 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "commit_sent", 2024-09-18T08:11:11.691 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:13.219647+0000", 2024-09-18T08:11:11.691 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4.9327999999999999e-05 2024-09-18T08:11:11.691 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:11:11.692 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:11:11.692 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-09-18T08:11:11.692 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:13.219684+0000", 2024-09-18T08:11:11.692 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 3.7088999999999997e-05 2024-09-18T08:11:11.692 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-18T08:11:11.692 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-09-18T08:11:11.692 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-18T08:11:11.692 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:11:11.692 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:11:11.692 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.15216.0:86390 8.5 8:a8c7b3fd:::benchmark_data_smithi007_101583_object86389:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e205)", 2024-09-18T08:11:11.692 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-09-18T08:09:12.356640+0000", 2024-09-18T08:11:11.692 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 119.264264451, 2024-09-18T08:11:11.692 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.86366743700000004, 2024-09-18T08:11:11.692 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-09-18T08:11:11.693 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-09-18T08:11:11.693 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T08:11:11.693 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2024-09-18T08:11:11.693 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.15216", 2024-09-18T08:11:11.693 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "172.21.15.7:0/2596041177", 2024-09-18T08:11:11.693 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 86390 2024-09-18T08:11:11.693 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:11:11.693 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-09-18T08:11:11.693 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:11:11.693 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-09-18T08:11:11.693 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:12.356640+0000", 2024-09-18T08:11:11.693 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-18T08:11:11.693 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:11:11.694 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:11:11.694 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-09-18T08:11:11.694 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:12.356640+0000", 2024-09-18T08:11:11.694 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-18T08:11:11.694 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:11:11.694 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:11:11.694 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-09-18T08:11:11.694 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:12.356641+0000", 2024-09-18T08:11:11.694 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.5430000000000001e-06 2024-09-18T08:11:11.694 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:11:11.694 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:11:11.694 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-09-18T08:11:11.694 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:12.356684+0000", 2024-09-18T08:11:11.694 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4.2222999999999999e-05 2024-09-18T08:11:11.695 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:11:11.695 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:11:11.695 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-09-18T08:11:11.695 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:12.356696+0000", 2024-09-18T08:11:11.695 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.2663000000000001e-05 2024-09-18T08:11:11.695 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:11:11.695 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:11:11.695 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-09-18T08:11:11.695 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:12.356703+0000", 2024-09-18T08:11:11.695 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 6.6710000000000002e-06 2024-09-18T08:11:11.695 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:11:11.695 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:11:11.695 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-18T08:11:11.695 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:12.356759+0000", 2024-09-18T08:11:11.696 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 5.6382000000000003e-05 2024-09-18T08:11:11.696 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:11:11.696 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:11:11.697 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "waiting for peered", 2024-09-18T08:11:11.697 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:12.356767+0000", 2024-09-18T08:11:11.697 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 7.5190000000000003e-06 2024-09-18T08:11:11.697 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:11:11.697 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:11:11.697 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-18T08:11:11.697 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:13.195090+0000", 2024-09-18T08:11:11.697 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.83832288600000004 2024-09-18T08:11:11.697 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:11:11.698 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:11:11.698 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-09-18T08:11:11.698 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:13.195221+0000", 2024-09-18T08:11:11.698 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00013146699999999999 2024-09-18T08:11:11.698 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:11:11.698 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:11:11.698 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "waiting for subops from 6,7", 2024-09-18T08:11:11.698 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:13.195370+0000", 2024-09-18T08:11:11.698 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000148598 2024-09-18T08:11:11.698 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:11:11.698 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:11:11.698 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:11:11.698 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:13.199845+0000", 2024-09-18T08:11:11.698 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0044749999999999998 2024-09-18T08:11:11.699 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:11:11.699 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:11:11.699 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:11:11.699 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:13.207294+0000", 2024-09-18T08:11:11.699 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0074489309999999998 2024-09-18T08:11:11.699 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:11:11.699 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:11:11.699 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "op_commit", 2024-09-18T08:11:11.699 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:13.220218+0000", 2024-09-18T08:11:11.699 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.012924715 2024-09-18T08:11:11.699 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:11:11.699 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:11:11.699 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "commit_sent", 2024-09-18T08:11:11.700 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:13.220272+0000", 2024-09-18T08:11:11.700 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 5.3748000000000001e-05 2024-09-18T08:11:11.700 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:11:11.700 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:11:11.700 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-09-18T08:11:11.700 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:13.220307+0000", 2024-09-18T08:11:11.700 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 3.5091e-05 2024-09-18T08:11:11.700 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-18T08:11:11.700 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-09-18T08:11:11.700 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-18T08:11:11.700 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:11:11.700 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:11:11.700 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.15216.0:86394 8.5 8:a810cd9d:::benchmark_data_smithi007_101583_object86393:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e205)", 2024-09-18T08:11:11.701 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-09-18T08:09:12.360967+0000", 2024-09-18T08:11:11.701 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 119.259936874, 2024-09-18T08:11:11.701 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.85950780100000002, 2024-09-18T08:11:11.701 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-09-18T08:11:11.701 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-09-18T08:11:11.701 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T08:11:11.701 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2024-09-18T08:11:11.701 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.15216", 2024-09-18T08:11:11.701 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "172.21.15.7:0/2596041177", 2024-09-18T08:11:11.701 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 86394 2024-09-18T08:11:11.701 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:11:11.701 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-09-18T08:11:11.701 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:11:11.701 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-09-18T08:11:11.702 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:12.360967+0000", 2024-09-18T08:11:11.702 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-18T08:11:11.702 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:11:11.702 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:11:11.702 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-09-18T08:11:11.702 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:12.360967+0000", 2024-09-18T08:11:11.702 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-18T08:11:11.702 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:11:11.702 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:11:11.702 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-09-18T08:11:11.702 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:12.360969+0000", 2024-09-18T08:11:11.702 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.8500000000000001e-06 2024-09-18T08:11:11.702 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:11:11.703 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:11:11.703 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-09-18T08:11:11.703 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:12.361013+0000", 2024-09-18T08:11:11.703 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4.3693000000000003e-05 2024-09-18T08:11:11.703 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:11:11.703 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:11:11.703 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-09-18T08:11:11.703 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:12.361031+0000", 2024-09-18T08:11:11.703 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.8340000000000001e-05 2024-09-18T08:11:11.703 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:11:11.703 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:11:11.703 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-09-18T08:11:11.703 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:12.361040+0000", 2024-09-18T08:11:11.703 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 8.5320000000000006e-06 2024-09-18T08:11:11.704 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:11:11.704 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:11:11.704 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-18T08:11:11.704 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:12.361108+0000", 2024-09-18T08:11:11.704 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 6.7879000000000005e-05 2024-09-18T08:11:11.704 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:11:11.704 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:11:11.704 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "waiting for peered", 2024-09-18T08:11:11.704 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:12.361119+0000", 2024-09-18T08:11:11.704 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.1163e-05 2024-09-18T08:11:11.704 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:11:11.704 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:11:11.704 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-18T08:11:11.705 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:13.197641+0000", 2024-09-18T08:11:11.705 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.83652230299999997 2024-09-18T08:11:11.705 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:11:11.705 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:11:11.705 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-09-18T08:11:11.705 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:13.197776+0000", 2024-09-18T08:11:11.705 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00013499000000000001 2024-09-18T08:11:11.705 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:11:11.705 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:11:11.705 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "waiting for subops from 6,7", 2024-09-18T08:11:11.705 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:13.197954+0000", 2024-09-18T08:11:11.705 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00017745200000000001 2024-09-18T08:11:11.705 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:11:11.706 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:11:11.706 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:11:11.706 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:13.207329+0000", 2024-09-18T08:11:11.706 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0093754460000000008 2024-09-18T08:11:11.706 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:11:11.706 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:11:11.706 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:11:11.706 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:13.207363+0000", 2024-09-18T08:11:11.706 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 3.3535999999999999e-05 2024-09-18T08:11:11.706 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:11:11.706 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:11:11.706 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "op_commit", 2024-09-18T08:11:11.706 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:13.220323+0000", 2024-09-18T08:11:11.706 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.012960121999999999 2024-09-18T08:11:11.707 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:11:11.707 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:11:11.707 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "commit_sent", 2024-09-18T08:11:11.707 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:13.220445+0000", 2024-09-18T08:11:11.707 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00012217099999999999 2024-09-18T08:11:11.707 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:11:11.707 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:11:11.707 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-09-18T08:11:11.707 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:13.220475+0000", 2024-09-18T08:11:11.707 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 3.0324e-05 2024-09-18T08:11:11.707 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-18T08:11:11.707 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-09-18T08:11:11.707 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-18T08:11:11.707 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:11:11.708 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:11:11.708 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.15216.0:86399 8.5 8:a535189e:::benchmark_data_smithi007_101583_object86398:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e205)", 2024-09-18T08:11:11.708 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-09-18T08:09:12.364040+0000", 2024-09-18T08:11:11.708 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 119.25686381, 2024-09-18T08:11:11.708 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.85654366599999998, 2024-09-18T08:11:11.708 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-09-18T08:11:11.708 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-09-18T08:11:11.708 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T08:11:11.708 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2024-09-18T08:11:11.708 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.15216", 2024-09-18T08:11:11.708 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "172.21.15.7:0/2596041177", 2024-09-18T08:11:11.708 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 86399 2024-09-18T08:11:11.708 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:11:11.708 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-09-18T08:11:11.709 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:11:11.709 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-09-18T08:11:11.709 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:12.364040+0000", 2024-09-18T08:11:11.709 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-18T08:11:11.709 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:11:11.709 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:11:11.709 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-09-18T08:11:11.709 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:12.364040+0000", 2024-09-18T08:11:11.709 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-18T08:11:11.709 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:11:11.709 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:11:11.709 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-09-18T08:11:11.709 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:12.364042+0000", 2024-09-18T08:11:11.710 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.407e-06 2024-09-18T08:11:11.710 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:11:11.710 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:11:11.710 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-09-18T08:11:11.710 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:12.364174+0000", 2024-09-18T08:11:11.710 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000132545 2024-09-18T08:11:11.710 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:11:11.710 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:11:11.710 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-09-18T08:11:11.710 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:12.364186+0000", 2024-09-18T08:11:11.710 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.1912e-05 2024-09-18T08:11:11.710 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:11:11.710 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:11:11.710 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-09-18T08:11:11.711 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:12.364192+0000", 2024-09-18T08:11:11.711 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 5.4600000000000002e-06 2024-09-18T08:11:11.711 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:11:11.711 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:11:11.711 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-18T08:11:11.711 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:12.364257+0000", 2024-09-18T08:11:11.711 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 6.5606999999999997e-05 2024-09-18T08:11:11.711 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:11:11.711 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:11:11.711 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "waiting for peered", 2024-09-18T08:11:11.711 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:12.364271+0000", 2024-09-18T08:11:11.711 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.3433e-05 2024-09-18T08:11:11.711 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:11:11.712 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:11:11.712 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-18T08:11:11.712 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:13.200082+0000", 2024-09-18T08:11:11.712 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.83581135500000003 2024-09-18T08:11:11.712 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:11:11.712 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:11:11.712 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-09-18T08:11:11.712 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:13.200201+0000", 2024-09-18T08:11:11.712 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00011913099999999999 2024-09-18T08:11:11.712 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:11:11.712 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:11:11.712 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "waiting for subops from 6,7", 2024-09-18T08:11:11.712 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:13.200360+0000", 2024-09-18T08:11:11.713 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000159173 2024-09-18T08:11:11.713 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:11:11.713 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:11:11.713 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:11:11.713 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:13.207395+0000", 2024-09-18T08:11:11.713 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0070341090000000002 2024-09-18T08:11:11.713 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:11:11.713 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:11:11.713 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:11:11.713 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:13.207427+0000", 2024-09-18T08:11:11.713 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 3.2138000000000003e-05 2024-09-18T08:11:11.713 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:11:11.713 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:11:11.713 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "op_commit", 2024-09-18T08:11:11.714 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:13.220524+0000", 2024-09-18T08:11:11.714 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.013097698 2024-09-18T08:11:11.714 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:11:11.714 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:11:11.714 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "commit_sent", 2024-09-18T08:11:11.714 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:13.220557+0000", 2024-09-18T08:11:11.714 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 3.2682999999999999e-05 2024-09-18T08:11:11.714 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:11:11.714 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:11:11.714 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-09-18T08:11:11.714 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:13.220584+0000", 2024-09-18T08:11:11.714 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 2.7015000000000001e-05 2024-09-18T08:11:11.714 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-18T08:11:11.714 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-09-18T08:11:11.715 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-18T08:11:11.715 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:11:11.715 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:11:11.715 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.15216.0:86429 8.5 8:a7f1f687:::benchmark_data_smithi007_101583_object86428:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e205)", 2024-09-18T08:11:11.715 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-09-18T08:09:12.416915+0000", 2024-09-18T08:11:11.715 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 119.203989041, 2024-09-18T08:11:11.715 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.80373175900000005, 2024-09-18T08:11:11.715 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-09-18T08:11:11.715 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-09-18T08:11:11.715 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T08:11:11.715 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2024-09-18T08:11:11.715 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.15216", 2024-09-18T08:11:11.715 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "172.21.15.7:0/2596041177", 2024-09-18T08:11:11.715 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 86429 2024-09-18T08:11:11.715 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:11:11.716 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-09-18T08:11:11.716 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:11:11.716 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-09-18T08:11:11.716 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:12.416915+0000", 2024-09-18T08:11:11.716 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-18T08:11:11.716 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:11:11.716 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:11:11.716 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-09-18T08:11:11.716 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:12.416915+0000", 2024-09-18T08:11:11.716 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-18T08:11:11.716 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:11:11.716 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:11:11.716 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-09-18T08:11:11.716 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:12.416917+0000", 2024-09-18T08:11:11.717 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.7150000000000001e-06 2024-09-18T08:11:11.717 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:11:11.717 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:11:11.717 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-09-18T08:11:11.717 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:12.416952+0000", 2024-09-18T08:11:11.717 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 3.4860000000000002e-05 2024-09-18T08:11:11.717 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:11:11.717 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:11:11.717 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-09-18T08:11:11.717 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:12.416966+0000", 2024-09-18T08:11:11.717 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.3884e-05 2024-09-18T08:11:11.717 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:11:11.717 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:11:11.717 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-09-18T08:11:11.718 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:12.416973+0000", 2024-09-18T08:11:11.718 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 7.5809999999999999e-06 2024-09-18T08:11:11.718 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:11:11.718 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:11:11.718 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-18T08:11:11.718 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:12.417031+0000", 2024-09-18T08:11:11.718 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 5.7544999999999999e-05 2024-09-18T08:11:11.718 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:11:11.718 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:11:11.718 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "waiting for peered", 2024-09-18T08:11:11.718 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:12.417039+0000", 2024-09-18T08:11:11.718 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 8.5369999999999997e-06 2024-09-18T08:11:11.718 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:11:11.718 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:11:11.718 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-18T08:11:11.719 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:13.202704+0000", 2024-09-18T08:11:11.719 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.78566510099999998 2024-09-18T08:11:11.719 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:11:11.719 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:11:11.719 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-09-18T08:11:11.719 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:13.202825+0000", 2024-09-18T08:11:11.719 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00012023600000000001 2024-09-18T08:11:11.719 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:11:11.719 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:11:11.719 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "waiting for subops from 6,7", 2024-09-18T08:11:11.719 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:13.202989+0000", 2024-09-18T08:11:11.719 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00016451000000000001 2024-09-18T08:11:11.719 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:11:11.720 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:11:11.720 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:11:11.720 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:13.207464+0000", 2024-09-18T08:11:11.720 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0044748909999999999 2024-09-18T08:11:11.720 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:11:11.720 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:11:11.720 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:11:11.720 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:13.214034+0000", 2024-09-18T08:11:11.720 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0065698229999999998 2024-09-18T08:11:11.720 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:11:11.720 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:11:11.720 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "op_commit", 2024-09-18T08:11:11.720 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:13.220590+0000", 2024-09-18T08:11:11.720 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0065556909999999998 2024-09-18T08:11:11.721 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:11:11.721 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:11:11.721 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "commit_sent", 2024-09-18T08:11:11.721 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:13.220620+0000", 2024-09-18T08:11:11.721 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 3.0429999999999998e-05 2024-09-18T08:11:11.721 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:11:11.721 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:11:11.721 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-09-18T08:11:11.721 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:13.220647+0000", 2024-09-18T08:11:11.721 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 2.6954999999999999e-05 2024-09-18T08:11:11.721 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-18T08:11:11.721 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-09-18T08:11:11.721 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-18T08:11:11.721 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:11:11.722 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:11:11.722 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.15216.0:86446 8.5 8:a0654d9b:::benchmark_data_smithi007_101583_object86445:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e205)", 2024-09-18T08:11:11.722 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-09-18T08:09:12.450214+0000", 2024-09-18T08:11:11.722 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 119.170689942, 2024-09-18T08:11:11.722 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.77050818399999998, 2024-09-18T08:11:11.722 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-09-18T08:11:11.722 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-09-18T08:11:11.722 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T08:11:11.722 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2024-09-18T08:11:11.722 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.15216", 2024-09-18T08:11:11.722 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "172.21.15.7:0/2596041177", 2024-09-18T08:11:11.722 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 86446 2024-09-18T08:11:11.722 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:11:11.723 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-09-18T08:11:11.723 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:11:11.723 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-09-18T08:11:11.723 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:12.450214+0000", 2024-09-18T08:11:11.723 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-18T08:11:11.723 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:11:11.723 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:11:11.723 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-09-18T08:11:11.723 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:12.450214+0000", 2024-09-18T08:11:11.723 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-18T08:11:11.723 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:11:11.723 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:11:11.723 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-09-18T08:11:11.723 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:12.450216+0000", 2024-09-18T08:11:11.724 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.8050000000000001e-06 2024-09-18T08:11:11.724 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:11:11.724 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:11:11.724 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-09-18T08:11:11.724 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:12.450274+0000", 2024-09-18T08:11:11.724 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 5.7865999999999998e-05 2024-09-18T08:11:11.724 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:11:11.724 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:11:11.724 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-09-18T08:11:11.724 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:12.450290+0000", 2024-09-18T08:11:11.724 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.5817000000000001e-05 2024-09-18T08:11:11.724 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:11:11.724 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:11:11.725 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-09-18T08:11:11.725 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:12.450299+0000", 2024-09-18T08:11:11.725 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 9.3759999999999997e-06 2024-09-18T08:11:11.725 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:11:11.725 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:11:11.725 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-18T08:11:11.725 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:12.450369+0000", 2024-09-18T08:11:11.725 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 6.9474000000000006e-05 2024-09-18T08:11:11.725 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:11:11.725 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:11:11.725 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "waiting for peered", 2024-09-18T08:11:11.725 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:12.450380+0000", 2024-09-18T08:11:11.725 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.168e-05 2024-09-18T08:11:11.726 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:11:11.726 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:11:11.726 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-18T08:11:11.726 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:13.207495+0000", 2024-09-18T08:11:11.726 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.75711499500000001 2024-09-18T08:11:11.726 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:11:11.726 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:11:11.726 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-09-18T08:11:11.726 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:13.207620+0000", 2024-09-18T08:11:11.726 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00012430199999999999 2024-09-18T08:11:11.726 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:11:11.726 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:11:11.726 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "waiting for subops from 6,7", 2024-09-18T08:11:11.727 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:13.207781+0000", 2024-09-18T08:11:11.727 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00016101099999999999 2024-09-18T08:11:11.727 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:11:11.727 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:11:11.727 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:11:11.727 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:13.214075+0000", 2024-09-18T08:11:11.727 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0062940419999999997 2024-09-18T08:11:11.727 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:11:11.727 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:11:11.727 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:11:11.727 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:13.214116+0000", 2024-09-18T08:11:11.727 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4.1517000000000002e-05 2024-09-18T08:11:11.727 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:11:11.727 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:11:11.728 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "op_commit", 2024-09-18T08:11:11.728 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:13.220658+0000", 2024-09-18T08:11:11.728 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0065420809999999999 2024-09-18T08:11:11.728 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:11:11.728 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:11:11.728 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "commit_sent", 2024-09-18T08:11:11.728 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:13.220695+0000", 2024-09-18T08:11:11.728 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 3.6896e-05 2024-09-18T08:11:11.728 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:11:11.728 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:11:11.728 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-09-18T08:11:11.728 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:13.220722+0000", 2024-09-18T08:11:11.728 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 2.7322e-05 2024-09-18T08:11:11.729 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-18T08:11:11.729 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-09-18T08:11:11.729 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-18T08:11:11.729 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:11:11.729 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:11:11.729 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.15216.0:86465 8.5 8:a7ecc445:::benchmark_data_smithi007_101583_object86464:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e205)", 2024-09-18T08:11:11.729 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-09-18T08:09:12.495165+0000", 2024-09-18T08:11:11.729 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 119.125738932, 2024-09-18T08:11:11.729 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.72562491600000001, 2024-09-18T08:11:11.729 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-09-18T08:11:11.729 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-09-18T08:11:11.729 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T08:11:11.729 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2024-09-18T08:11:11.729 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.15216", 2024-09-18T08:11:11.730 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "172.21.15.7:0/2596041177", 2024-09-18T08:11:11.730 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 86465 2024-09-18T08:11:11.730 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:11:11.730 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-09-18T08:11:11.730 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:11:11.730 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-09-18T08:11:11.730 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:12.495165+0000", 2024-09-18T08:11:11.730 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-18T08:11:11.730 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:11:11.730 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:11:11.730 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-09-18T08:11:11.730 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:12.495165+0000", 2024-09-18T08:11:11.730 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-18T08:11:11.730 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:11:11.731 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:11:11.731 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-09-18T08:11:11.731 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:12.495167+0000", 2024-09-18T08:11:11.731 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.9089999999999998e-06 2024-09-18T08:11:11.731 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:11:11.731 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:11:11.731 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-09-18T08:11:11.731 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:12.495284+0000", 2024-09-18T08:11:11.731 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000117228 2024-09-18T08:11:11.731 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:11:11.731 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:11:11.731 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-09-18T08:11:11.731 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:12.495301+0000", 2024-09-18T08:11:11.732 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.7065e-05 2024-09-18T08:11:11.732 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:11:11.732 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:11:11.732 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-09-18T08:11:11.732 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:12.495312+0000", 2024-09-18T08:11:11.732 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.0132999999999999e-05 2024-09-18T08:11:11.732 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:11:11.732 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:11:11.732 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-18T08:11:11.732 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:12.495388+0000", 2024-09-18T08:11:11.732 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 7.6512000000000002e-05 2024-09-18T08:11:11.732 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:11:11.732 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:11:11.732 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "waiting for peered", 2024-09-18T08:11:11.733 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:12.495402+0000", 2024-09-18T08:11:11.733 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.3376e-05 2024-09-18T08:11:11.733 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:11:11.733 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:11:11.733 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-18T08:11:11.733 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:13.211447+0000", 2024-09-18T08:11:11.733 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.71604507500000003 2024-09-18T08:11:11.733 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:11:11.733 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:11:11.733 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-09-18T08:11:11.733 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:13.211727+0000", 2024-09-18T08:11:11.733 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00028030299999999998 2024-09-18T08:11:11.733 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:11:11.733 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:11:11.734 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "waiting for subops from 6,7", 2024-09-18T08:11:11.734 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:13.211872+0000", 2024-09-18T08:11:11.734 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00014542799999999999 2024-09-18T08:11:11.734 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:11:11.734 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:11:11.734 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:11:11.734 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:13.219374+0000", 2024-09-18T08:11:11.734 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0075019090000000002 2024-09-18T08:11:11.734 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:11:11.734 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:11:11.734 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:11:11.734 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:13.220515+0000", 2024-09-18T08:11:11.734 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0011411839999999999 2024-09-18T08:11:11.734 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:11:11.735 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:11:11.735 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "op_commit", 2024-09-18T08:11:11.735 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:13.220727+0000", 2024-09-18T08:11:11.735 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00021189200000000001 2024-09-18T08:11:11.735 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:11:11.735 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:11:11.735 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "commit_sent", 2024-09-18T08:11:11.735 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:13.220765+0000", 2024-09-18T08:11:11.735 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 3.7319e-05 2024-09-18T08:11:11.735 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:11:11.735 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:11:11.735 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-09-18T08:11:11.735 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:13.220790+0000", 2024-09-18T08:11:11.735 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 2.5582999999999999e-05 2024-09-18T08:11:11.736 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-18T08:11:11.736 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-09-18T08:11:11.736 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-18T08:11:11.736 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:11:11.736 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:11:11.736 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.15216.0:86475 8.5 8:a4f19050:::benchmark_data_smithi007_101583_object86474:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e205)", 2024-09-18T08:11:11.736 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-09-18T08:09:12.523844+0000", 2024-09-18T08:11:11.736 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 119.097060614, 2024-09-18T08:11:11.736 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.69708282899999996, 2024-09-18T08:11:11.736 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-09-18T08:11:11.736 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-09-18T08:11:11.736 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T08:11:11.736 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2024-09-18T08:11:11.736 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.15216", 2024-09-18T08:11:11.737 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "172.21.15.7:0/2596041177", 2024-09-18T08:11:11.737 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 86475 2024-09-18T08:11:11.737 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:11:11.737 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-09-18T08:11:11.737 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:11:11.737 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-09-18T08:11:11.737 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:12.523844+0000", 2024-09-18T08:11:11.737 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-18T08:11:11.737 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:11:11.737 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:11:11.737 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-09-18T08:11:11.737 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:12.523844+0000", 2024-09-18T08:11:11.737 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-18T08:11:11.737 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:11:11.738 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:11:11.738 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-09-18T08:11:11.738 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:12.523846+0000", 2024-09-18T08:11:11.738 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 2.0320000000000002e-06 2024-09-18T08:11:11.738 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:11:11.738 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:11:11.738 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-09-18T08:11:11.738 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:12.523886+0000", 2024-09-18T08:11:11.738 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4.0763000000000003e-05 2024-09-18T08:11:11.738 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:11:11.738 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:11:11.738 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-09-18T08:11:11.738 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:12.523911+0000", 2024-09-18T08:11:11.739 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 2.4202e-05 2024-09-18T08:11:11.739 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:11:11.739 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:11:11.739 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-09-18T08:11:11.739 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:12.523920+0000", 2024-09-18T08:11:11.739 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 8.9330000000000007e-06 2024-09-18T08:11:11.739 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:11:11.739 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:11:11.739 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-18T08:11:11.739 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:12.523990+0000", 2024-09-18T08:11:11.739 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 7.0505000000000003e-05 2024-09-18T08:11:11.739 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:11:11.739 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:11:11.739 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "waiting for peered", 2024-09-18T08:11:11.740 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:12.524004+0000", 2024-09-18T08:11:11.740 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.3607e-05 2024-09-18T08:11:11.740 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:11:11.740 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:11:11.740 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-18T08:11:11.740 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:13.214163+0000", 2024-09-18T08:11:11.740 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.69015934999999995 2024-09-18T08:11:11.740 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:11:11.740 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:11:11.740 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-09-18T08:11:11.740 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:13.214291+0000", 2024-09-18T08:11:11.740 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00012824399999999999 2024-09-18T08:11:11.740 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:11:11.741 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:11:11.741 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "waiting for subops from 6,7", 2024-09-18T08:11:11.741 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:13.214455+0000", 2024-09-18T08:11:11.741 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00016422499999999999 2024-09-18T08:11:11.741 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:11:11.741 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:11:11.741 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "op_commit", 2024-09-18T08:11:11.741 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:13.220795+0000", 2024-09-18T08:11:11.741 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0063397560000000002 2024-09-18T08:11:11.741 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:11:11.741 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:11:11.741 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:11:11.741 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:13.220825+0000", 2024-09-18T08:11:11.741 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 2.976e-05 2024-09-18T08:11:11.742 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:11:11.742 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:11:11.742 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:11:11.742 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:13.220855+0000", 2024-09-18T08:11:11.742 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 3.0043999999999999e-05 2024-09-18T08:11:11.742 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:11:11.742 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:11:11.742 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "commit_sent", 2024-09-18T08:11:11.742 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:13.220889+0000", 2024-09-18T08:11:11.742 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 3.4251000000000001e-05 2024-09-18T08:11:11.742 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:11:11.742 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:11:11.742 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-09-18T08:11:11.742 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:13.220926+0000", 2024-09-18T08:11:11.743 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 3.7156999999999997e-05 2024-09-18T08:11:11.743 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-18T08:11:11.743 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-09-18T08:11:11.743 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-18T08:11:11.743 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:11:11.743 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:11:11.743 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.15216.0:86499 8.5 8:ab3bde63:::benchmark_data_smithi007_101583_object86498:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e205)", 2024-09-18T08:11:11.743 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-09-18T08:09:12.639820+0000", 2024-09-18T08:11:11.743 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 118.98108382700001, 2024-09-18T08:11:11.743 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.59097304500000003, 2024-09-18T08:11:11.743 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-09-18T08:11:11.743 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-09-18T08:11:11.743 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T08:11:11.744 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2024-09-18T08:11:11.744 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.15216", 2024-09-18T08:11:11.744 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "172.21.15.7:0/2596041177", 2024-09-18T08:11:11.744 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 86499 2024-09-18T08:11:11.744 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:11:11.744 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-09-18T08:11:11.744 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:11:11.744 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-09-18T08:11:11.744 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:12.639820+0000", 2024-09-18T08:11:11.744 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-18T08:11:11.744 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:11:11.744 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:11:11.744 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-09-18T08:11:11.745 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:12.639820+0000", 2024-09-18T08:11:11.745 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-18T08:11:11.745 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:11:11.745 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:11:11.745 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-09-18T08:11:11.745 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:12.639822+0000", 2024-09-18T08:11:11.745 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.4440000000000001e-06 2024-09-18T08:11:11.745 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:11:11.745 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:11:11.745 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-09-18T08:11:11.745 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:12.639872+0000", 2024-09-18T08:11:11.745 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 5.0296000000000001e-05 2024-09-18T08:11:11.745 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:11:11.745 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:11:11.746 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-09-18T08:11:11.746 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:12.639885+0000", 2024-09-18T08:11:11.746 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.3173000000000001e-05 2024-09-18T08:11:11.746 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:11:11.746 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:11:11.746 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-09-18T08:11:11.746 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:12.639894+0000", 2024-09-18T08:11:11.746 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 8.5310000000000001e-06 2024-09-18T08:11:11.746 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:11:11.746 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:11:11.746 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-18T08:11:11.746 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:12.639972+0000", 2024-09-18T08:11:11.746 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 7.8307999999999996e-05 2024-09-18T08:11:11.746 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:11:11.747 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:11:11.747 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "waiting for peered", 2024-09-18T08:11:11.747 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:12.639980+0000", 2024-09-18T08:11:11.747 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 8.0669999999999992e-06 2024-09-18T08:11:11.747 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:11:11.747 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:11:11.747 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-18T08:11:11.747 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:13.216786+0000", 2024-09-18T08:11:11.747 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.57680564199999995 2024-09-18T08:11:11.747 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:11:11.747 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:11:11.747 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-09-18T08:11:11.747 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:13.216933+0000", 2024-09-18T08:11:11.747 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00014732999999999999 2024-09-18T08:11:11.747 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:11:11.748 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:11:11.748 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "waiting for subops from 6,7", 2024-09-18T08:11:11.748 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:13.217061+0000", 2024-09-18T08:11:11.748 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000128042 2024-09-18T08:11:11.748 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:11:11.748 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:11:11.748 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:11:11.748 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:13.220967+0000", 2024-09-18T08:11:11.748 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0039059939999999999 2024-09-18T08:11:11.748 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:11:11.748 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:11:11.748 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "op_commit", 2024-09-18T08:11:11.748 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:13.228425+0000", 2024-09-18T08:11:11.749 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.007457623 2024-09-18T08:11:11.749 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:11:11.749 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:11:11.749 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:11:11.749 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:13.230681+0000", 2024-09-18T08:11:11.749 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0022558280000000001 2024-09-18T08:11:11.749 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:11:11.749 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:11:11.749 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "commit_sent", 2024-09-18T08:11:11.749 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:13.230747+0000", 2024-09-18T08:11:11.749 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 6.5881000000000002e-05 2024-09-18T08:11:11.749 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:11:11.749 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:11:11.749 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-09-18T08:11:11.750 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:13.230793+0000", 2024-09-18T08:11:11.750 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4.6885999999999998e-05 2024-09-18T08:11:11.750 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-18T08:11:11.750 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-09-18T08:11:11.750 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-18T08:11:11.750 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:11:11.750 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:11:11.750 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.15216.0:185198 8.2 8:4b2bc7f8:::benchmark_data_smithi007_101583_object84446:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e226)", 2024-09-18T08:11:11.750 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-09-18T08:10:43.088217+0000", 2024-09-18T08:11:11.750 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 28.532686753, 2024-09-18T08:11:11.750 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.094031926000000002, 2024-09-18T08:11:11.750 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-09-18T08:11:11.750 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-09-18T08:11:11.750 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T08:11:11.751 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2024-09-18T08:11:11.751 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.15216", 2024-09-18T08:11:11.751 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "172.21.15.7:0/2596041177", 2024-09-18T08:11:11.751 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 185198 2024-09-18T08:11:11.751 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:11:11.751 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-09-18T08:11:11.751 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:11:11.751 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-09-18T08:11:11.751 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:10:43.088217+0000", 2024-09-18T08:11:11.751 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-18T08:11:11.751 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:11:11.751 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:11:11.751 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-09-18T08:11:11.751 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:10:43.088217+0000", 2024-09-18T08:11:11.752 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-18T08:11:11.752 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:11:11.752 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:11:11.752 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-09-18T08:11:11.752 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:10:43.088219+0000", 2024-09-18T08:11:11.752 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.5510000000000001e-06 2024-09-18T08:11:11.752 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:11:11.752 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:11:11.752 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-09-18T08:11:11.752 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:10:43.088225+0000", 2024-09-18T08:11:11.752 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 5.75e-06 2024-09-18T08:11:11.752 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:11:11.753 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:11:11.753 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-09-18T08:11:11.753 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:10:43.088238+0000", 2024-09-18T08:11:11.753 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.2908000000000001e-05 2024-09-18T08:11:11.753 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:11:11.753 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:11:11.753 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-09-18T08:11:11.753 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:10:43.088246+0000", 2024-09-18T08:11:11.753 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 8.1960000000000008e-06 2024-09-18T08:11:11.753 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:11:11.753 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:11:11.753 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-18T08:11:11.753 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:10:43.088307+0000", 2024-09-18T08:11:11.754 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 6.0819000000000002e-05 2024-09-18T08:11:11.754 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:11:11.754 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:11:11.754 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-09-18T08:11:11.754 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:10:43.088419+0000", 2024-09-18T08:11:11.754 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000112339 2024-09-18T08:11:11.754 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:11:11.754 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:11:11.754 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "waiting for subops from 0,5", 2024-09-18T08:11:11.754 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:10:43.088527+0000", 2024-09-18T08:11:11.754 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000108262 2024-09-18T08:11:11.754 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:11:11.754 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:11:11.754 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "op_commit", 2024-09-18T08:11:11.755 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:10:43.091325+0000", 2024-09-18T08:11:11.755 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0027975930000000001 2024-09-18T08:11:11.755 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:11:11.755 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:11:11.755 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:11:11.755 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:10:43.092275+0000", 2024-09-18T08:11:11.755 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00095057699999999998 2024-09-18T08:11:11.755 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:11:11.755 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:11:11.755 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:11:11.755 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:10:43.182095+0000", 2024-09-18T08:11:11.755 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.089819484000000005 2024-09-18T08:11:11.755 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:11:11.755 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:11:11.756 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "commit_sent", 2024-09-18T08:11:11.756 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:10:43.182200+0000", 2024-09-18T08:11:11.756 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00010474199999999999 2024-09-18T08:11:11.756 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:11:11.756 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:11:11.756 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-09-18T08:11:11.756 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:10:43.182249+0000", 2024-09-18T08:11:11.756 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4.9704999999999999e-05 2024-09-18T08:11:11.756 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-18T08:11:11.756 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-09-18T08:11:11.756 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-18T08:11:11.756 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:11:11.756 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:11:11.756 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.15216.0:185209 8.2 8:4f8d20eb:::benchmark_data_smithi007_101583_object84457:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e226)", 2024-09-18T08:11:11.757 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-09-18T08:10:43.100856+0000", 2024-09-18T08:11:11.757 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 28.520047796, 2024-09-18T08:11:11.757 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.081572924000000005, 2024-09-18T08:11:11.757 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-09-18T08:11:11.757 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-09-18T08:11:11.757 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T08:11:11.757 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2024-09-18T08:11:11.757 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.15216", 2024-09-18T08:11:11.757 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "172.21.15.7:0/2596041177", 2024-09-18T08:11:11.757 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 185209 2024-09-18T08:11:11.757 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:11:11.757 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-09-18T08:11:11.757 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:11:11.758 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-09-18T08:11:11.758 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:10:43.100856+0000", 2024-09-18T08:11:11.758 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-18T08:11:11.758 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:11:11.758 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:11:11.758 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-09-18T08:11:11.758 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:10:43.100856+0000", 2024-09-18T08:11:11.758 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-18T08:11:11.758 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:11:11.759 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:11:11.759 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-09-18T08:11:11.759 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:10:43.100858+0000", 2024-09-18T08:11:11.759 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.5579999999999999e-06 2024-09-18T08:11:11.759 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:11:11.759 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:11:11.759 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-09-18T08:11:11.759 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:10:43.100865+0000", 2024-09-18T08:11:11.759 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 7.08e-06 2024-09-18T08:11:11.759 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:11:11.759 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:11:11.759 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-09-18T08:11:11.759 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:10:43.100878+0000", 2024-09-18T08:11:11.759 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.2512e-05 2024-09-18T08:11:11.760 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:11:11.760 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:11:11.760 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-09-18T08:11:11.760 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:10:43.100885+0000", 2024-09-18T08:11:11.760 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 7.2189999999999998e-06 2024-09-18T08:11:11.760 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:11:11.760 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:11:11.760 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-18T08:11:11.760 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:10:43.100937+0000", 2024-09-18T08:11:11.760 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 5.2437e-05 2024-09-18T08:11:11.760 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:11:11.760 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:11:11.760 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-09-18T08:11:11.761 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:10:43.101063+0000", 2024-09-18T08:11:11.761 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000125582 2024-09-18T08:11:11.761 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:11:11.761 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:11:11.761 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "waiting for subops from 0,5", 2024-09-18T08:11:11.761 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:10:43.101180+0000", 2024-09-18T08:11:11.761 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000117268 2024-09-18T08:11:11.761 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:11:11.761 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:11:11.761 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "op_commit", 2024-09-18T08:11:11.761 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:10:43.103288+0000", 2024-09-18T08:11:11.761 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0021079319999999999 2024-09-18T08:11:11.761 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:11:11.762 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:11:11.762 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:11:11.762 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:10:43.103946+0000", 2024-09-18T08:11:11.762 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00065811400000000001 2024-09-18T08:11:11.762 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:11:11.762 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:11:11.762 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:11:11.762 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:10:43.182296+0000", 2024-09-18T08:11:11.762 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.078349715 2024-09-18T08:11:11.762 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:11:11.762 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:11:11.762 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "commit_sent", 2024-09-18T08:11:11.762 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:10:43.182379+0000", 2024-09-18T08:11:11.762 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 8.3358999999999999e-05 2024-09-18T08:11:11.762 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:11:11.763 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:11:11.763 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-09-18T08:11:11.763 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:10:43.182429+0000", 2024-09-18T08:11:11.763 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 5.0148000000000002e-05 2024-09-18T08:11:11.763 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-18T08:11:11.763 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-09-18T08:11:11.763 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-18T08:11:11.763 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-18T08:11:11.763 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-09-18T08:11:11.763 INFO:teuthology.orchestra.run.smithi050.stdout:} 2024-09-18T08:11:11.763 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1 2024-09-18T08:11:11.786 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:11:11.786 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:11:11.786 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1' 2024-09-18T08:11:11.887 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1 2024-09-18T08:11:11.910 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:11:11.910 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:11:11.911 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1' 2024-09-18T08:11:12.012 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5 2024-09-18T08:11:12.035 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:11:12.035 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:11:12.036 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5' 2024-09-18T08:11:12.137 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7 2024-09-18T08:11:12.159 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:11:12.160 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:11:12.160 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7' 2024-09-18T08:11:12.219 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/779349ca-7593-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T08:11:12.260 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/779349ca-7593-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T08:11:12.260 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3 2024-09-18T08:11:12.285 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:11:12.286 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:11:12.286 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3' 2024-09-18T08:11:12.387 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6 2024-09-18T08:11:12.409 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:11:12.410 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:11:12.410 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6' 2024-09-18T08:11:12.480 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:11:12 smithi007 ceph-mon[29459]: pgmap v401: 158 pgs: 13 creating+peering, 3 unknown, 142 active+clean; 583 KiB data, 2.5 GiB used, 713 GiB / 715 GiB avail; 854 B/s rd, 0 op/s 2024-09-18T08:11:12.480 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:11:12 smithi007 ceph-mon[29459]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-18T08:11:12.480 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:11:12 smithi007 ceph-mon[29459]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-18T08:11:12.481 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:11:12 smithi007 ceph-mon[29459]: osdmap e235: 8 total, 8 up, 8 in 2024-09-18T08:11:12.481 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:11:12 smithi007 ceph-mon[39249]: pgmap v401: 158 pgs: 13 creating+peering, 3 unknown, 142 active+clean; 583 KiB data, 2.5 GiB used, 713 GiB / 715 GiB avail; 854 B/s rd, 0 op/s 2024-09-18T08:11:12.481 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:11:12 smithi007 ceph-mon[39249]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-18T08:11:12.481 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:11:12 smithi007 ceph-mon[39249]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-18T08:11:12.481 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:11:12 smithi007 ceph-mon[39249]: osdmap e235: 8 total, 8 up, 8 in 2024-09-18T08:11:12.511 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1 2024-09-18T08:11:12.533 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:11:12.534 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:11:12.534 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1' 2024-09-18T08:11:12.635 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6 2024-09-18T08:11:12.655 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:11:12 smithi050 ceph-mon[32371]: pgmap v401: 158 pgs: 13 creating+peering, 3 unknown, 142 active+clean; 583 KiB data, 2.5 GiB used, 713 GiB / 715 GiB avail; 854 B/s rd, 0 op/s 2024-09-18T08:11:12.656 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:11:12 smithi050 ceph-mon[32371]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-18T08:11:12.656 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:11:12 smithi050 ceph-mon[32371]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-18T08:11:12.656 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:11:12 smithi050 ceph-mon[32371]: osdmap e235: 8 total, 8 up, 8 in 2024-09-18T08:11:12.660 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:11:12.661 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:11:12.662 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6' 2024-09-18T08:11:12.763 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7 2024-09-18T08:11:12.788 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:11:12.788 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:11:12.789 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7' 2024-09-18T08:11:12.889 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1 2024-09-18T08:11:12.912 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:11:12.913 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:11:12.914 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1' 2024-09-18T08:11:13.015 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2 2024-09-18T08:11:13.038 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:11:13.039 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:11:13.039 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2' 2024-09-18T08:11:13.140 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1 2024-09-18T08:11:13.165 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:11:13.166 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:11:13.166 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1' 2024-09-18T08:11:13.175 INFO:teuthology.orchestra.run.smithi007.stderr:no valid command found; 10 closest matches: 2024-09-18T08:11:13.175 INFO:teuthology.orchestra.run.smithi007.stderr:0 2024-09-18T08:11:13.175 INFO:teuthology.orchestra.run.smithi007.stderr:1 2024-09-18T08:11:13.175 INFO:teuthology.orchestra.run.smithi007.stderr:2 2024-09-18T08:11:13.175 INFO:teuthology.orchestra.run.smithi007.stderr:abort 2024-09-18T08:11:13.175 INFO:teuthology.orchestra.run.smithi007.stderr:assert 2024-09-18T08:11:13.175 INFO:teuthology.orchestra.run.smithi007.stderr:bench [] [] [] [] 2024-09-18T08:11:13.175 INFO:teuthology.orchestra.run.smithi007.stderr:bluefs debug_inject_read_zeros 2024-09-18T08:11:13.175 INFO:teuthology.orchestra.run.smithi007.stderr:bluefs files list 2024-09-18T08:11:13.175 INFO:teuthology.orchestra.run.smithi007.stderr:bluefs stats 2024-09-18T08:11:13.176 INFO:teuthology.orchestra.run.smithi007.stderr:bluestore allocator dump block 2024-09-18T08:11:13.176 INFO:teuthology.orchestra.run.smithi007.stderr:Error EINVAL: invalid command 2024-09-18T08:11:13.267 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7 2024-09-18T08:11:13.292 INFO:teuthology.orchestra.run.smithi007.stderr:enabled application 'rados' on pool 'unique_pool_2' 2024-09-18T08:11:13.298 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:11:13.299 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:11:13.299 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7' 2024-09-18T08:11:13.400 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5 2024-09-18T08:11:13.427 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:11:13.428 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:11:13.428 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5' 2024-09-18T08:11:13.529 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4 2024-09-18T08:11:13.550 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid 779349ca-7593-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-18T08:11:13.555 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:11:13.555 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:11:13.556 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4' 2024-09-18T08:11:13.575 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:11:13 smithi007 ceph-mon[29459]: from='client.? 172.21.15.7:0/1739580982' entity='client.admin' cmd={"prefix": "osd pool application enable", "pool": "unique_pool_2", "app": "rados", "yes_i_really_mean_it": true} : dispatch 2024-09-18T08:11:13.575 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:11:13 smithi007 ceph-mon[39249]: from='client.? 172.21.15.7:0/1739580982' entity='client.admin' cmd={"prefix": "osd pool application enable", "pool": "unique_pool_2", "app": "rados", "yes_i_really_mean_it": true} : dispatch 2024-09-18T08:11:13.593 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:11:13 smithi050 ceph-mon[32371]: from='client.? 172.21.15.7:0/1739580982' entity='client.admin' cmd={"prefix": "osd pool application enable", "pool": "unique_pool_2", "app": "rados", "yes_i_really_mean_it": true} : dispatch 2024-09-18T08:11:13.628 DEBUG:teuthology.orchestra.run.smithi050:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid 779349ca-7593-11ef-bceb-c7b262605968 -- ceph daemon osd.5 dump_ops_in_flight 2024-09-18T08:11:13.656 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5 2024-09-18T08:11:13.691 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:11:13.692 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:11:13.692 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5' 2024-09-18T08:11:13.793 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1 2024-09-18T08:11:13.815 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:11:13.816 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:11:13.816 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1' 2024-09-18T08:11:13.917 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3 2024-09-18T08:11:13.927 INFO:teuthology.orchestra.run.smithi050.stderr:Inferring config /var/lib/ceph/779349ca-7593-11ef-bceb-c7b262605968/mon.b/config 2024-09-18T08:11:13.940 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:11:13.941 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:11:13.941 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3' 2024-09-18T08:11:14.042 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4 2024-09-18T08:11:14.064 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:11:14.064 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:11:14.065 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4' 2024-09-18T08:11:14.163 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid 779349ca-7593-11ef-bceb-c7b262605968 -- ceph osd unset noscrub 2024-09-18T08:11:14.166 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6 2024-09-18T08:11:14.189 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:11:14.190 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:11:14.190 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6' 2024-09-18T08:11:14.291 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0 2024-09-18T08:11:14.313 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:11:14.314 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:11:14.314 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0' 2024-09-18T08:11:14.338 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:11:14 smithi050 ceph-mon[32371]: pgmap v403: 158 pgs: 13 creating+peering, 145 active+clean; 583 KiB data, 2.5 GiB used, 713 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-09-18T08:11:14.339 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:11:14 smithi050 ceph-mon[32371]: from='client.? 172.21.15.7:0/1739580982' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "unique_pool_2", "app": "rados", "yes_i_really_mean_it": true}]': finished 2024-09-18T08:11:14.339 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:11:14 smithi050 ceph-mon[32371]: osdmap e236: 8 total, 8 up, 8 in 2024-09-18T08:11:14.415 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4 2024-09-18T08:11:14.438 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:11:14.439 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:11:14.439 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4' 2024-09-18T08:11:14.443 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2024-09-18T08:11:14.443 INFO:teuthology.orchestra.run.smithi050.stdout: "ops": [], 2024-09-18T08:11:14.443 INFO:teuthology.orchestra.run.smithi050.stdout: "num_ops": 0 2024-09-18T08:11:14.443 INFO:teuthology.orchestra.run.smithi050.stdout:} 2024-09-18T08:11:14.480 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:11:14 smithi007 ceph-mon[29459]: pgmap v403: 158 pgs: 13 creating+peering, 145 active+clean; 583 KiB data, 2.5 GiB used, 713 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-09-18T08:11:14.481 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:11:14 smithi007 ceph-mon[29459]: from='client.? 172.21.15.7:0/1739580982' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "unique_pool_2", "app": "rados", "yes_i_really_mean_it": true}]': finished 2024-09-18T08:11:14.481 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:11:14 smithi007 ceph-mon[29459]: osdmap e236: 8 total, 8 up, 8 in 2024-09-18T08:11:14.481 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:11:14 smithi007 ceph-mon[39249]: pgmap v403: 158 pgs: 13 creating+peering, 145 active+clean; 583 KiB data, 2.5 GiB used, 713 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-09-18T08:11:14.481 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:11:14 smithi007 ceph-mon[39249]: from='client.? 172.21.15.7:0/1739580982' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "unique_pool_2", "app": "rados", "yes_i_really_mean_it": true}]': finished 2024-09-18T08:11:14.481 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:11:14 smithi007 ceph-mon[39249]: osdmap e236: 8 total, 8 up, 8 in 2024-09-18T08:11:14.540 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6 2024-09-18T08:11:14.563 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:11:14.564 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:11:14.565 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6' 2024-09-18T08:11:14.641 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-09-18T08:11:14.642 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-09-18T08:11:14.642 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid 779349ca-7593-11ef-bceb-c7b262605968 -- ceph tell osd.5 'trim stale osdmaps' 2024-09-18T08:11:14.666 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1 2024-09-18T08:11:14.681 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/779349ca-7593-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T08:11:14.707 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:11:14.707 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:11:14.708 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1' 2024-09-18T08:11:14.809 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7 2024-09-18T08:11:14.832 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:11:14.832 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:11:14.833 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7' 2024-09-18T08:11:14.933 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6 2024-09-18T08:11:14.956 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:11:14.957 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:11:14.957 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6' 2024-09-18T08:11:15.058 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0 2024-09-18T08:11:15.063 DEBUG:teuthology.orchestra.run.smithi050:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid 779349ca-7593-11ef-bceb-c7b262605968 -- ceph daemon osd.5 dump_blocked_ops 2024-09-18T08:11:15.081 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:11:15.081 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:11:15.082 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0' 2024-09-18T08:11:15.097 DEBUG:teuthology.orchestra.run.smithi007:> /bin/sh -c 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados --no-log-to-stderr --name client.0 -b 65536 --object-size 65536 -p unique_pool_2 bench 90 write' 2024-09-18T08:11:15.124 INFO:tasks.radosbench:joining radosbench (timing out after 3000s) 2024-09-18T08:11:15.125 INFO:teuthology.orchestra.run:waiting for 3000 2024-09-18T08:11:15.171 INFO:teuthology.orchestra.run.smithi007.stdout:osd.0: {} 2024-09-18T08:11:15.172 INFO:teuthology.orchestra.run.smithi007.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-18T08:11:15.181 INFO:tasks.radosbench.radosbench.0.smithi007.stdout:hints = 1 2024-09-18T08:11:15.181 INFO:tasks.radosbench.radosbench.0.smithi007.stdout:Maintaining 16 concurrent writes of 65536 bytes to objects of size 65536 for up to 90 seconds or 0 objects 2024-09-18T08:11:15.181 INFO:tasks.radosbench.radosbench.0.smithi007.stdout:Object prefix: benchmark_data_smithi007_124019 2024-09-18T08:11:15.181 INFO:tasks.radosbench.radosbench.0.smithi007.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-09-18T08:11:15.182 INFO:tasks.radosbench.radosbench.0.smithi007.stdout: 0 0 0 0 0 0 - 0 2024-09-18T08:11:15.183 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3 2024-09-18T08:11:15.206 INFO:teuthology.orchestra.run.smithi007.stdout:osd.1: {} 2024-09-18T08:11:15.206 INFO:teuthology.orchestra.run.smithi007.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-18T08:11:15.237 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:11:15.238 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:11:15.238 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3' 2024-09-18T08:11:15.250 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/779349ca-7593-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T08:11:15.250 INFO:teuthology.orchestra.run.smithi007.stdout:osd.2: {} 2024-09-18T08:11:15.251 INFO:teuthology.orchestra.run.smithi007.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-18T08:11:15.259 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/779349ca-7593-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T08:11:15.291 INFO:teuthology.orchestra.run.smithi007.stdout:osd.3: {} 2024-09-18T08:11:15.291 INFO:teuthology.orchestra.run.smithi007.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-18T08:11:15.332 INFO:teuthology.orchestra.run.smithi007.stdout:osd.4: {} 2024-09-18T08:11:15.332 INFO:teuthology.orchestra.run.smithi007.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-18T08:11:15.340 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1 2024-09-18T08:11:15.371 INFO:teuthology.orchestra.run.smithi007.stdout:osd.5: {} 2024-09-18T08:11:15.371 INFO:teuthology.orchestra.run.smithi007.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-18T08:11:15.383 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:11:15.384 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:11:15.384 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1' 2024-09-18T08:11:15.401 INFO:teuthology.orchestra.run.smithi050.stderr:Inferring config /var/lib/ceph/779349ca-7593-11ef-bceb-c7b262605968/mon.b/config 2024-09-18T08:11:15.410 INFO:teuthology.orchestra.run.smithi007.stdout:osd.6: {} 2024-09-18T08:11:15.410 INFO:teuthology.orchestra.run.smithi007.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-18T08:11:15.445 INFO:teuthology.orchestra.run.smithi007.stdout:osd.7: {} 2024-09-18T08:11:15.446 INFO:teuthology.orchestra.run.smithi007.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-18T08:11:15.460 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:11:15 smithi007 ceph-mon[29459]: from='client.? 172.21.15.7:0/129435508' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-18T08:11:15.460 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:11:15 smithi007 ceph-mon[39249]: from='client.? 172.21.15.7:0/129435508' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-18T08:11:15.485 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2 2024-09-18T08:11:15.526 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:11:15.526 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:11:15.527 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2' 2024-09-18T08:11:15.627 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1 2024-09-18T08:11:15.656 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:11:15 smithi050 ceph-mon[32371]: from='client.? 172.21.15.7:0/129435508' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-18T08:11:15.671 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:11:15.672 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:11:15.672 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1' 2024-09-18T08:11:15.774 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3 2024-09-18T08:11:15.811 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:11:15.812 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:11:15.813 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3' 2024-09-18T08:11:15.913 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0 2024-09-18T08:11:15.951 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:11:15.952 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:11:15.952 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0' 2024-09-18T08:11:16.053 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2 2024-09-18T08:11:16.089 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:11:16.089 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:11:16.090 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2' 2024-09-18T08:11:16.142 INFO:teuthology.orchestra.run.smithi050.stdout:ERROR: (22) Invalid argument 2024-09-18T08:11:16.142 INFO:teuthology.orchestra.run.smithi050.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-18T08:11:16.182 INFO:tasks.radosbench.radosbench.0.smithi007.stdout: 1 16 2998 2982 186.364 186.375 0.00261355 0.00534847 2024-09-18T08:11:16.191 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5 2024-09-18T08:11:16.226 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:11:16.226 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:11:16.227 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5' 2024-09-18T08:11:16.328 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1 2024-09-18T08:11:16.367 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:11:16.368 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:11:16.368 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1' 2024-09-18T08:11:16.416 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:11:16 smithi007 ceph-mon[29459]: pgmap v405: 158 pgs: 158 active+clean; 583 KiB data, 2.5 GiB used, 713 GiB / 715 GiB avail; 856 B/s rd, 0 op/s 2024-09-18T08:11:16.416 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:11:16 smithi007 ceph-mon[29459]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-09-18T08:11:16.416 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:11:16 smithi007 ceph-mon[39249]: pgmap v405: 158 pgs: 158 active+clean; 583 KiB data, 2.5 GiB used, 713 GiB / 715 GiB avail; 856 B/s rd, 0 op/s 2024-09-18T08:11:16.417 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:11:16 smithi007 ceph-mon[39249]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-09-18T08:11:16.469 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6 2024-09-18T08:11:16.493 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:11:16.493 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:11:16.493 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6' 2024-09-18T08:11:16.594 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3 2024-09-18T08:11:16.647 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:11:16.648 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:11:16.648 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3' 2024-09-18T08:11:16.656 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:11:16 smithi050 ceph-mon[32371]: pgmap v405: 158 pgs: 158 active+clean; 583 KiB data, 2.5 GiB used, 713 GiB / 715 GiB avail; 856 B/s rd, 0 op/s 2024-09-18T08:11:16.656 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:11:16 smithi050 ceph-mon[32371]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-09-18T08:11:16.749 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1 2024-09-18T08:11:16.791 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:11:16.792 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:11:16.792 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1' 2024-09-18T08:11:16.818 INFO:teuthology.orchestra.run.smithi007.stderr:no valid command found; 10 closest matches: 2024-09-18T08:11:16.818 INFO:teuthology.orchestra.run.smithi007.stderr:0 2024-09-18T08:11:16.818 INFO:teuthology.orchestra.run.smithi007.stderr:1 2024-09-18T08:11:16.818 INFO:teuthology.orchestra.run.smithi007.stderr:2 2024-09-18T08:11:16.818 INFO:teuthology.orchestra.run.smithi007.stderr:abort 2024-09-18T08:11:16.818 INFO:teuthology.orchestra.run.smithi007.stderr:assert 2024-09-18T08:11:16.818 INFO:teuthology.orchestra.run.smithi007.stderr:bench [] [] [] [] 2024-09-18T08:11:16.818 INFO:teuthology.orchestra.run.smithi007.stderr:bluefs debug_inject_read_zeros 2024-09-18T08:11:16.819 INFO:teuthology.orchestra.run.smithi007.stderr:bluefs files list 2024-09-18T08:11:16.819 INFO:teuthology.orchestra.run.smithi007.stderr:bluefs stats 2024-09-18T08:11:16.819 INFO:teuthology.orchestra.run.smithi007.stderr:bluestore allocator dump block 2024-09-18T08:11:16.819 INFO:teuthology.orchestra.run.smithi007.stderr:Error EINVAL: invalid command 2024-09-18T08:11:16.868 DEBUG:teuthology.orchestra.run.smithi050:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid 779349ca-7593-11ef-bceb-c7b262605968 -- ceph daemon osd.5 dump_historic_ops 2024-09-18T08:11:16.893 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4 2024-09-18T08:11:16.949 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:11:16.950 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:11:16.950 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4' 2024-09-18T08:11:16.980 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Sep 18 08:11:16 smithi007 ceph-779349ca-7593-11ef-bceb-c7b262605968-mgr-y[29688]: ::ffff:172.21.15.50 - - [18/Sep/2024:08:11:16] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-09-18T08:11:17.051 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5 2024-09-18T08:11:17.085 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:11:17.085 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:11:17.086 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5' 2024-09-18T08:11:17.182 INFO:tasks.radosbench.radosbench.0.smithi007.stdout: 2 16 5981 5965 186.392 186.438 0.00488467 0.00535273 2024-09-18T08:11:17.186 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0 2024-09-18T08:11:17.225 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:11:17.226 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:11:17.226 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0' 2024-09-18T08:11:17.294 INFO:teuthology.orchestra.run.smithi050.stderr:Inferring config /var/lib/ceph/779349ca-7593-11ef-bceb-c7b262605968/mon.b/config 2024-09-18T08:11:17.317 INFO:teuthology.orchestra.run.smithi007.stderr:noscrub is unset 2024-09-18T08:11:17.327 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3 2024-09-18T08:11:17.365 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:11:17.365 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:11:17.366 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3' 2024-09-18T08:11:17.466 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7 2024-09-18T08:11:17.504 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:11:17.505 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:11:17.505 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7' 2024-09-18T08:11:17.580 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:11:17 smithi050 ceph-mon[32371]: from='client.? 172.21.15.7:0/717486146' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-18T08:11:17.581 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:11:17 smithi050 ceph-mon[32371]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-18T08:11:17.606 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0 2024-09-18T08:11:17.609 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:11:17 smithi007 ceph-mon[29459]: from='client.? 172.21.15.7:0/717486146' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-18T08:11:17.610 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:11:17 smithi007 ceph-mon[29459]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-18T08:11:17.610 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:11:17 smithi007 ceph-mon[39249]: from='client.? 172.21.15.7:0/717486146' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-18T08:11:17.610 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:11:17 smithi007 ceph-mon[39249]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-18T08:11:17.635 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:11:17.636 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:11:17.637 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0' 2024-09-18T08:11:17.648 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-09-18T08:11:17.649 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-09-18T08:11:17.649 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid 779349ca-7593-11ef-bceb-c7b262605968 -- ceph tell osd.7 'trim stale osdmaps' 2024-09-18T08:11:17.738 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0 2024-09-18T08:11:17.781 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:11:17.782 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:11:17.782 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0' 2024-09-18T08:11:17.883 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6 2024-09-18T08:11:17.901 INFO:teuthology.orchestra.run.smithi050.stdout:ERROR: (22) Invalid argument 2024-09-18T08:11:17.901 INFO:teuthology.orchestra.run.smithi050.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-18T08:11:17.919 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:11:17.920 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:11:17.920 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6' 2024-09-18T08:11:18.021 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3 2024-09-18T08:11:18.058 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:11:18.059 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:11:18.059 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3' 2024-09-18T08:11:18.160 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3 2024-09-18T08:11:18.182 INFO:tasks.radosbench.radosbench.0.smithi007.stdout: 3 16 8856 8840 184.149 179.688 0.00267721 0.00541897 2024-09-18T08:11:18.200 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:11:18.201 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:11:18.201 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3' 2024-09-18T08:11:18.222 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid 779349ca-7593-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-18T08:11:18.303 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3 2024-09-18T08:11:18.349 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:11:18.350 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:11:18.350 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3' 2024-09-18T08:11:18.451 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6 2024-09-18T08:11:18.487 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:11:18.487 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:11:18.488 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6' 2024-09-18T08:11:18.556 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:11:18 smithi050 ceph-mon[32371]: pgmap v406: 158 pgs: 158 active+clean; 583 KiB data, 2.5 GiB used, 713 GiB / 715 GiB avail; 733 B/s rd, 0 op/s 2024-09-18T08:11:18.556 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:11:18 smithi050 ceph-mon[32371]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-18T08:11:18.557 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:11:18 smithi050 ceph-mon[32371]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-18T08:11:18.557 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:11:18 smithi050 ceph-mon[32371]: osdmap e237: 8 total, 8 up, 8 in 2024-09-18T08:11:18.557 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:11:18 smithi050 ceph-mon[32371]: 5.17 scrub starts 2024-09-18T08:11:18.557 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:11:18 smithi050 ceph-mon[32371]: 5.17 scrub ok 2024-09-18T08:11:18.557 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:11:18 smithi050 ceph-mon[32371]: from='mgr.24470 172.21.15.7:0/2594390261' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-18T08:11:18.588 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2 2024-09-18T08:11:18.610 DEBUG:teuthology.orchestra.run.smithi050:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid 779349ca-7593-11ef-bceb-c7b262605968 -- ceph daemon osd.6 dump_ops_in_flight 2024-09-18T08:11:18.628 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:11:18.629 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:11:18.629 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2' 2024-09-18T08:11:18.730 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5 2024-09-18T08:11:18.731 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:11:18 smithi007 ceph-mon[29459]: pgmap v406: 158 pgs: 158 active+clean; 583 KiB data, 2.5 GiB used, 713 GiB / 715 GiB avail; 733 B/s rd, 0 op/s 2024-09-18T08:11:18.731 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:11:18 smithi007 ceph-mon[29459]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-18T08:11:18.731 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:11:18 smithi007 ceph-mon[29459]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-18T08:11:18.731 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:11:18 smithi007 ceph-mon[29459]: osdmap e237: 8 total, 8 up, 8 in 2024-09-18T08:11:18.731 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:11:18 smithi007 ceph-mon[29459]: 5.17 scrub starts 2024-09-18T08:11:18.731 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:11:18 smithi007 ceph-mon[29459]: 5.17 scrub ok 2024-09-18T08:11:18.731 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:11:18 smithi007 ceph-mon[29459]: from='mgr.24470 172.21.15.7:0/2594390261' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-18T08:11:18.731 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:11:18 smithi007 ceph-mon[39249]: pgmap v406: 158 pgs: 158 active+clean; 583 KiB data, 2.5 GiB used, 713 GiB / 715 GiB avail; 733 B/s rd, 0 op/s 2024-09-18T08:11:18.732 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:11:18 smithi007 ceph-mon[39249]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-18T08:11:18.732 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:11:18 smithi007 ceph-mon[39249]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-18T08:11:18.732 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:11:18 smithi007 ceph-mon[39249]: osdmap e237: 8 total, 8 up, 8 in 2024-09-18T08:11:18.732 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:11:18 smithi007 ceph-mon[39249]: 5.17 scrub starts 2024-09-18T08:11:18.732 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:11:18 smithi007 ceph-mon[39249]: 5.17 scrub ok 2024-09-18T08:11:18.732 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:11:18 smithi007 ceph-mon[39249]: from='mgr.24470 172.21.15.7:0/2594390261' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-18T08:11:18.769 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:11:18.770 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:11:18.770 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5' 2024-09-18T08:11:18.871 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4 2024-09-18T08:11:18.906 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:11:18.907 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:11:18.907 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4' 2024-09-18T08:11:18.945 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/779349ca-7593-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T08:11:19.007 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/779349ca-7593-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T08:11:19.008 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1 2024-09-18T08:11:19.024 INFO:teuthology.orchestra.run.smithi050.stderr:Inferring config /var/lib/ceph/779349ca-7593-11ef-bceb-c7b262605968/mon.b/config 2024-09-18T08:11:19.056 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:11:19.057 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:11:19.057 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1' 2024-09-18T08:11:19.158 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4 2024-09-18T08:11:19.182 INFO:tasks.radosbench.radosbench.0.smithi007.stdout: 4 15 11618 11603 181.267 172.688 0.00641198 0.00550859 2024-09-18T08:11:19.195 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:11:19.196 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:11:19.196 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4' 2024-09-18T08:11:19.297 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0 2024-09-18T08:11:19.324 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:11:19.325 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:11:19.325 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0' 2024-09-18T08:11:19.376 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:11:19 smithi007 ceph-mon[29459]: purged_snaps scrub starts 2024-09-18T08:11:19.377 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:11:19 smithi007 ceph-mon[29459]: purged_snaps scrub ok 2024-09-18T08:11:19.377 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:11:19 smithi007 ceph-mon[39249]: purged_snaps scrub starts 2024-09-18T08:11:19.377 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:11:19 smithi007 ceph-mon[39249]: purged_snaps scrub ok 2024-09-18T08:11:19.406 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:11:19 smithi050 ceph-mon[32371]: purged_snaps scrub starts 2024-09-18T08:11:19.406 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:11:19 smithi050 ceph-mon[32371]: purged_snaps scrub ok 2024-09-18T08:11:19.406 INFO:journalctl@ceph.iscsi.iscsi.a.smithi050.stdout:Sep 18 08:11:19 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-iscsi-iscsi-a[57782]: debug there is no tcmu-runner data available 2024-09-18T08:11:19.427 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3 2024-09-18T08:11:19.465 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:11:19.466 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:11:19.466 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3' 2024-09-18T08:11:19.567 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7 2024-09-18T08:11:19.600 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:11:19.600 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:11:19.601 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7' 2024-09-18T08:11:19.630 INFO:teuthology.orchestra.run.smithi050.stdout:ERROR: (22) Invalid argument 2024-09-18T08:11:19.630 INFO:teuthology.orchestra.run.smithi050.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-18T08:11:19.702 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6 2024-09-18T08:11:19.735 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:11:19.735 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:11:19.736 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6' 2024-09-18T08:11:19.837 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7 2024-09-18T08:11:19.874 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:11:19.874 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:11:19.875 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7' 2024-09-18T08:11:19.923 INFO:teuthology.orchestra.run.smithi007.stderr:no valid command found; 10 closest matches: 2024-09-18T08:11:19.924 INFO:teuthology.orchestra.run.smithi007.stderr:0 2024-09-18T08:11:19.924 INFO:teuthology.orchestra.run.smithi007.stderr:1 2024-09-18T08:11:19.924 INFO:teuthology.orchestra.run.smithi007.stderr:2 2024-09-18T08:11:19.924 INFO:teuthology.orchestra.run.smithi007.stderr:abort 2024-09-18T08:11:19.924 INFO:teuthology.orchestra.run.smithi007.stderr:assert 2024-09-18T08:11:19.924 INFO:teuthology.orchestra.run.smithi007.stderr:bench [] [] [] [] 2024-09-18T08:11:19.924 INFO:teuthology.orchestra.run.smithi007.stderr:bluefs debug_inject_read_zeros 2024-09-18T08:11:19.924 INFO:teuthology.orchestra.run.smithi007.stderr:bluefs files list 2024-09-18T08:11:19.924 INFO:teuthology.orchestra.run.smithi007.stderr:bluefs stats 2024-09-18T08:11:19.924 INFO:teuthology.orchestra.run.smithi007.stderr:bluestore allocator dump block 2024-09-18T08:11:19.924 INFO:teuthology.orchestra.run.smithi007.stderr:Error EINVAL: invalid command 2024-09-18T08:11:19.976 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3 2024-09-18T08:11:20.014 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:11:20.015 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:11:20.015 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3' 2024-09-18T08:11:20.116 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7 2024-09-18T08:11:20.150 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:11:20.150 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:11:20.151 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7' 2024-09-18T08:11:20.182 INFO:tasks.radosbench.radosbench.0.smithi007.stdout: 5 16 14102 14086 176.049 155.188 0.0047011 0.00567183 2024-09-18T08:11:20.251 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5 2024-09-18T08:11:20.285 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:11:20.285 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:11:20.286 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5' 2024-09-18T08:11:20.387 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7 2024-09-18T08:11:20.389 DEBUG:teuthology.orchestra.run.smithi050:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid 779349ca-7593-11ef-bceb-c7b262605968 -- ceph daemon osd.6 dump_blocked_ops 2024-09-18T08:11:20.436 INFO:teuthology.orchestra.run.smithi007.stdout:osd.0: {} 2024-09-18T08:11:20.436 INFO:teuthology.orchestra.run.smithi007.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-18T08:11:20.463 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:11:20.464 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:11:20.465 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7' 2024-09-18T08:11:20.475 INFO:teuthology.orchestra.run.smithi007.stdout:osd.1: {} 2024-09-18T08:11:20.475 INFO:teuthology.orchestra.run.smithi007.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-18T08:11:20.480 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:11:20 smithi007 ceph-mon[29459]: 6.e scrub starts 2024-09-18T08:11:20.481 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:11:20 smithi007 ceph-mon[29459]: 6.e scrub ok 2024-09-18T08:11:20.481 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:11:20 smithi007 ceph-mon[29459]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-18T08:11:20.481 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:11:20 smithi007 ceph-mon[29459]: purged_snaps scrub starts 2024-09-18T08:11:20.481 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:11:20 smithi007 ceph-mon[29459]: purged_snaps scrub ok 2024-09-18T08:11:20.481 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:11:20 smithi007 ceph-mon[29459]: pgmap v408: 158 pgs: 158 active+clean; 629 MiB data, 4.1 GiB used, 711 GiB / 715 GiB avail; 1.3 KiB/s rd, 79 MiB/s wr, 1.26k op/s 2024-09-18T08:11:20.482 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:11:20 smithi007 ceph-mon[29459]: osdmap e238: 8 total, 8 up, 8 in 2024-09-18T08:11:20.482 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:11:20 smithi007 ceph-mon[39249]: 6.e scrub starts 2024-09-18T08:11:20.482 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:11:20 smithi007 ceph-mon[39249]: 6.e scrub ok 2024-09-18T08:11:20.482 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:11:20 smithi007 ceph-mon[39249]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-18T08:11:20.482 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:11:20 smithi007 ceph-mon[39249]: purged_snaps scrub starts 2024-09-18T08:11:20.482 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:11:20 smithi007 ceph-mon[39249]: purged_snaps scrub ok 2024-09-18T08:11:20.482 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:11:20 smithi007 ceph-mon[39249]: pgmap v408: 158 pgs: 158 active+clean; 629 MiB data, 4.1 GiB used, 711 GiB / 715 GiB avail; 1.3 KiB/s rd, 79 MiB/s wr, 1.26k op/s 2024-09-18T08:11:20.483 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:11:20 smithi007 ceph-mon[39249]: osdmap e238: 8 total, 8 up, 8 in 2024-09-18T08:11:20.514 INFO:teuthology.orchestra.run.smithi007.stdout:osd.2: {} 2024-09-18T08:11:20.514 INFO:teuthology.orchestra.run.smithi007.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-18T08:11:20.555 INFO:teuthology.orchestra.run.smithi007.stdout:osd.3: {} 2024-09-18T08:11:20.556 INFO:teuthology.orchestra.run.smithi007.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-18T08:11:20.565 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6 2024-09-18T08:11:20.592 INFO:teuthology.orchestra.run.smithi007.stdout:osd.4: {} 2024-09-18T08:11:20.593 INFO:teuthology.orchestra.run.smithi007.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-18T08:11:20.601 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:11:20.602 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:11:20.603 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6' 2024-09-18T08:11:20.630 INFO:teuthology.orchestra.run.smithi007.stdout:osd.5: {} 2024-09-18T08:11:20.631 INFO:teuthology.orchestra.run.smithi007.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-18T08:11:20.656 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:11:20 smithi050 ceph-mon[32371]: 6.e scrub starts 2024-09-18T08:11:20.656 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:11:20 smithi050 ceph-mon[32371]: 6.e scrub ok 2024-09-18T08:11:20.656 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:11:20 smithi050 ceph-mon[32371]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-18T08:11:20.657 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:11:20 smithi050 ceph-mon[32371]: purged_snaps scrub starts 2024-09-18T08:11:20.657 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:11:20 smithi050 ceph-mon[32371]: purged_snaps scrub ok 2024-09-18T08:11:20.657 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:11:20 smithi050 ceph-mon[32371]: pgmap v408: 158 pgs: 158 active+clean; 629 MiB data, 4.1 GiB used, 711 GiB / 715 GiB avail; 1.3 KiB/s rd, 79 MiB/s wr, 1.26k op/s 2024-09-18T08:11:20.657 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:11:20 smithi050 ceph-mon[32371]: osdmap e238: 8 total, 8 up, 8 in 2024-09-18T08:11:20.666 INFO:teuthology.orchestra.run.smithi007.stdout:osd.6: {} 2024-09-18T08:11:20.667 INFO:teuthology.orchestra.run.smithi007.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-18T08:11:20.704 INFO:teuthology.orchestra.run.smithi007.stdout:osd.7: {} 2024-09-18T08:11:20.704 INFO:teuthology.orchestra.run.smithi007.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-18T08:11:20.704 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5 2024-09-18T08:11:20.744 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:11:20.745 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:11:20.745 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5' 2024-09-18T08:11:20.846 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5 2024-09-18T08:11:20.853 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid 779349ca-7593-11ef-bceb-c7b262605968 -- ceph osd unset nodeep-scrub 2024-09-18T08:11:20.872 INFO:teuthology.orchestra.run.smithi050.stderr:Inferring config /var/lib/ceph/779349ca-7593-11ef-bceb-c7b262605968/mon.b/config 2024-09-18T08:11:20.886 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:11:20.888 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:11:20.888 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5' 2024-09-18T08:11:20.989 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1 2024-09-18T08:11:21.030 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:11:21.031 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:11:21.031 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1' 2024-09-18T08:11:21.132 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4 2024-09-18T08:11:21.167 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:11:21.168 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:11:21.169 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4' 2024-09-18T08:11:21.183 INFO:tasks.radosbench.radosbench.0.smithi007.stdout: 6 16 16708 16692 173.85 162.875 0.00734837 0.00574468 2024-09-18T08:11:21.274 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0 2024-09-18T08:11:21.288 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-09-18T08:11:21.288 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-09-18T08:11:21.289 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid 779349ca-7593-11ef-bceb-c7b262605968 -- ceph tell osd.6 'trim stale osdmaps' 2024-09-18T08:11:21.304 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:11:21.304 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:11:21.305 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0' 2024-09-18T08:11:21.406 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6 2024-09-18T08:11:21.442 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:11:21.442 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:11:21.443 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6' 2024-09-18T08:11:21.544 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7 2024-09-18T08:11:21.582 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:11:21.583 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:11:21.583 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7' 2024-09-18T08:11:21.599 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:11:21 smithi007 ceph-mon[29459]: 5.11 scrub starts 2024-09-18T08:11:21.600 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:11:21 smithi007 ceph-mon[29459]: 5.11 scrub ok 2024-09-18T08:11:21.600 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:11:21 smithi007 ceph-mon[29459]: 3.7 scrub starts 2024-09-18T08:11:21.600 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:11:21 smithi007 ceph-mon[29459]: 3.7 scrub ok 2024-09-18T08:11:21.600 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:11:21 smithi007 ceph-mon[29459]: from='client.? 172.21.15.7:0/3539993162' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-18T08:11:21.600 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:11:21 smithi007 ceph-mon[39249]: 5.11 scrub starts 2024-09-18T08:11:21.600 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:11:21 smithi007 ceph-mon[39249]: 5.11 scrub ok 2024-09-18T08:11:21.600 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:11:21 smithi007 ceph-mon[39249]: 3.7 scrub starts 2024-09-18T08:11:21.600 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:11:21 smithi007 ceph-mon[39249]: 3.7 scrub ok 2024-09-18T08:11:21.600 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:11:21 smithi007 ceph-mon[39249]: from='client.? 172.21.15.7:0/3539993162' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-18T08:11:21.609 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:11:21 smithi050 ceph-mon[32371]: 5.11 scrub starts 2024-09-18T08:11:21.609 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:11:21 smithi050 ceph-mon[32371]: 5.11 scrub ok 2024-09-18T08:11:21.609 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:11:21 smithi050 ceph-mon[32371]: 3.7 scrub starts 2024-09-18T08:11:21.609 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:11:21 smithi050 ceph-mon[32371]: 3.7 scrub ok 2024-09-18T08:11:21.609 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:11:21 smithi050 ceph-mon[32371]: from='client.? 172.21.15.7:0/3539993162' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-18T08:11:21.650 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2024-09-18T08:11:21.651 INFO:teuthology.orchestra.run.smithi050.stdout: "ops": [], 2024-09-18T08:11:21.651 INFO:teuthology.orchestra.run.smithi050.stdout: "complaint_time": 30, 2024-09-18T08:11:21.651 INFO:teuthology.orchestra.run.smithi050.stdout: "num_blocked_ops": 0 2024-09-18T08:11:21.651 INFO:teuthology.orchestra.run.smithi050.stdout:} 2024-09-18T08:11:21.685 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2 2024-09-18T08:11:21.728 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:11:21.729 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:11:21.729 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2' 2024-09-18T08:11:21.830 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4 2024-09-18T08:11:21.866 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:11:21.867 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:11:21.867 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4' 2024-09-18T08:11:21.968 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2 2024-09-18T08:11:22.007 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:11:22.007 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:11:22.007 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2' 2024-09-18T08:11:22.108 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0 2024-09-18T08:11:22.136 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/779349ca-7593-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T08:11:22.173 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:11:22.174 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:11:22.174 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0' 2024-09-18T08:11:22.178 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/779349ca-7593-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T08:11:22.183 INFO:tasks.radosbench.radosbench.0.smithi007.stdout: 7 16 19114 19098 170.473 150.375 0.00581039 0.00585816 2024-09-18T08:11:22.275 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4 2024-09-18T08:11:22.307 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:11:22.307 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:11:22.308 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4' 2024-09-18T08:11:22.409 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1 2024-09-18T08:11:22.417 DEBUG:teuthology.orchestra.run.smithi050:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid 779349ca-7593-11ef-bceb-c7b262605968 -- ceph daemon osd.6 dump_historic_ops 2024-09-18T08:11:22.448 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:11:22.449 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:11:22.449 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1' 2024-09-18T08:11:22.549 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1 2024-09-18T08:11:22.556 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:11:22 smithi007 ceph-mon[29459]: 6.3 scrub starts 2024-09-18T08:11:22.556 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:11:22 smithi007 ceph-mon[29459]: 6.3 scrub ok 2024-09-18T08:11:22.556 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:11:22 smithi007 ceph-mon[29459]: pgmap v410: 158 pgs: 158 active+clean; 629 MiB data, 4.1 GiB used, 711 GiB / 715 GiB avail; 1.3 KiB/s rd, 79 MiB/s wr, 1.26k op/s 2024-09-18T08:11:22.556 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:11:22 smithi007 ceph-mon[29459]: 6.b scrub starts 2024-09-18T08:11:22.556 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:11:22 smithi007 ceph-mon[29459]: 6.b scrub ok 2024-09-18T08:11:22.557 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:11:22 smithi007 ceph-mon[39249]: 6.3 scrub starts 2024-09-18T08:11:22.557 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:11:22 smithi007 ceph-mon[39249]: 6.3 scrub ok 2024-09-18T08:11:22.557 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:11:22 smithi007 ceph-mon[39249]: pgmap v410: 158 pgs: 158 active+clean; 629 MiB data, 4.1 GiB used, 711 GiB / 715 GiB avail; 1.3 KiB/s rd, 79 MiB/s wr, 1.26k op/s 2024-09-18T08:11:22.557 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:11:22 smithi007 ceph-mon[39249]: 6.b scrub starts 2024-09-18T08:11:22.557 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:11:22 smithi007 ceph-mon[39249]: 6.b scrub ok 2024-09-18T08:11:22.591 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:11:22.592 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:11:22.592 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1' 2024-09-18T08:11:22.656 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:11:22 smithi050 ceph-mon[32371]: 6.3 scrub starts 2024-09-18T08:11:22.656 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:11:22 smithi050 ceph-mon[32371]: 6.3 scrub ok 2024-09-18T08:11:22.656 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:11:22 smithi050 ceph-mon[32371]: pgmap v410: 158 pgs: 158 active+clean; 629 MiB data, 4.1 GiB used, 711 GiB / 715 GiB avail; 1.3 KiB/s rd, 79 MiB/s wr, 1.26k op/s 2024-09-18T08:11:22.656 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:11:22 smithi050 ceph-mon[32371]: 6.b scrub starts 2024-09-18T08:11:22.656 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:11:22 smithi050 ceph-mon[32371]: 6.b scrub ok 2024-09-18T08:11:22.693 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0 2024-09-18T08:11:22.738 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:11:22.738 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:11:22.739 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0' 2024-09-18T08:11:22.804 INFO:teuthology.orchestra.run.smithi050.stderr:Inferring config /var/lib/ceph/779349ca-7593-11ef-bceb-c7b262605968/mon.b/config 2024-09-18T08:11:22.839 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4 2024-09-18T08:11:22.877 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:11:22.878 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:11:22.878 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4' 2024-09-18T08:11:22.979 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7 2024-09-18T08:11:23.012 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:11:23.013 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:11:23.013 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7' 2024-09-18T08:11:23.114 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4 2024-09-18T08:11:23.143 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:11:23.144 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:11:23.144 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4' 2024-09-18T08:11:23.184 INFO:tasks.radosbench.radosbench.0.smithi007.stdout: 8 16 21258 21242 165.913 134 0.00716201 0.00602107 2024-09-18T08:11:23.245 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6 2024-09-18T08:11:23.282 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:11:23.282 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:11:23.283 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6' 2024-09-18T08:11:23.384 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6 2024-09-18T08:11:23.420 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2024-09-18T08:11:23.420 INFO:teuthology.orchestra.run.smithi050.stdout: "size": 20, 2024-09-18T08:11:23.420 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 600, 2024-09-18T08:11:23.420 INFO:teuthology.orchestra.run.smithi050.stdout: "ops": [ 2024-09-18T08:11:23.421 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:11:23.421 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.14805.0:38000 7.7 7:e509d896:::benchmark_data_smithi007_73155_object37999:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e94)", 2024-09-18T08:11:23.421 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-09-18T08:04:23.049009+0000", 2024-09-18T08:11:23.421 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 420.36613716199997, 2024-09-18T08:11:23.421 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.027565702000000001, 2024-09-18T08:11:23.421 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-09-18T08:11:23.421 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-09-18T08:11:23.421 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T08:11:23.421 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2024-09-18T08:11:23.421 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.14805", 2024-09-18T08:11:23.421 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "172.21.15.7:0/640320436", 2024-09-18T08:11:23.421 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 38000 2024-09-18T08:11:23.421 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:11:23.422 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-09-18T08:11:23.422 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:11:23.422 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-09-18T08:11:23.422 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:23.049009+0000", 2024-09-18T08:11:23.422 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-18T08:11:23.422 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:11:23.422 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:11:23.422 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-09-18T08:11:23.422 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:23.049009+0000", 2024-09-18T08:11:23.422 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-18T08:11:23.422 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:11:23.422 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:11:23.422 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-09-18T08:11:23.423 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:23.049010+0000", 2024-09-18T08:11:23.423 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.11e-06 2024-09-18T08:11:23.423 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:11:23.423 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:11:23.423 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-09-18T08:11:23.423 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:23.068284+0000", 2024-09-18T08:11:23.423 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.019274264999999999 2024-09-18T08:11:23.423 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:11:23.423 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:11:23.423 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-09-18T08:11:23.423 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:23.068323+0000", 2024-09-18T08:11:23.423 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 3.8167999999999998e-05 2024-09-18T08:11:23.423 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:11:23.424 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:11:23.424 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-09-18T08:11:23.424 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:23.068329+0000", 2024-09-18T08:11:23.424 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 6.331e-06 2024-09-18T08:11:23.424 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:11:23.424 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:11:23.424 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-18T08:11:23.424 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:23.068399+0000", 2024-09-18T08:11:23.424 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 7.0055999999999995e-05 2024-09-18T08:11:23.424 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:11:23.424 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:11:23.424 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-09-18T08:11:23.424 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:23.068503+0000", 2024-09-18T08:11:23.425 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000104206 2024-09-18T08:11:23.425 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:11:23.425 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:11:23.425 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "waiting for subops from 0,3", 2024-09-18T08:11:23.425 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:23.068640+0000", 2024-09-18T08:11:23.425 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00013645000000000001 2024-09-18T08:11:23.425 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:11:23.425 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:11:23.425 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "op_commit", 2024-09-18T08:11:23.425 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:23.073270+0000", 2024-09-18T08:11:23.425 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0046298950000000002 2024-09-18T08:11:23.425 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:11:23.425 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:11:23.426 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:11:23.426 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:23.073311+0000", 2024-09-18T08:11:23.426 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4.1406999999999998e-05 2024-09-18T08:11:23.426 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:11:23.426 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:11:23.426 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:11:23.426 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:23.076521+0000", 2024-09-18T08:11:23.426 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.003209862 2024-09-18T08:11:23.426 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:11:23.426 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:11:23.426 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "commit_sent", 2024-09-18T08:11:23.426 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:23.076552+0000", 2024-09-18T08:11:23.427 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 3.1698e-05 2024-09-18T08:11:23.427 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:11:23.427 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:11:23.427 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-09-18T08:11:23.427 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:23.076575+0000", 2024-09-18T08:11:23.427 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 2.2254000000000001e-05 2024-09-18T08:11:23.427 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-18T08:11:23.427 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-09-18T08:11:23.427 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-18T08:11:23.427 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:11:23.427 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:11:23.427 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.14805.0:39064 7.7 7:ef833039:::benchmark_data_smithi007_73155_object39063:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e95)", 2024-09-18T08:11:23.427 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-09-18T08:04:23.658960+0000", 2024-09-18T08:11:23.428 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 419.75618588999998, 2024-09-18T08:11:23.428 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.029075624000000001, 2024-09-18T08:11:23.428 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-09-18T08:11:23.428 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-09-18T08:11:23.428 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T08:11:23.428 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2024-09-18T08:11:23.428 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.14805", 2024-09-18T08:11:23.428 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "172.21.15.7:0/640320436", 2024-09-18T08:11:23.428 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 39064 2024-09-18T08:11:23.428 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:11:23.428 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-09-18T08:11:23.428 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:11:23.428 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-09-18T08:11:23.429 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:23.658960+0000", 2024-09-18T08:11:23.429 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-18T08:11:23.429 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:11:23.429 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:11:23.429 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-09-18T08:11:23.429 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:23.658960+0000", 2024-09-18T08:11:23.429 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-18T08:11:23.429 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:11:23.429 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:11:23.429 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-09-18T08:11:23.429 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:23.658961+0000", 2024-09-18T08:11:23.429 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 6.7899999999999998e-07 2024-09-18T08:11:23.429 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:11:23.430 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:11:23.430 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-09-18T08:11:23.430 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:23.658991+0000", 2024-09-18T08:11:23.430 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 3.0168999999999999e-05 2024-09-18T08:11:23.430 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:11:23.430 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:11:23.430 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-09-18T08:11:23.430 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:23.659001+0000", 2024-09-18T08:11:23.430 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 9.6220000000000004e-06 2024-09-18T08:11:23.430 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:11:23.430 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:11:23.430 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-09-18T08:11:23.430 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:23.659010+0000", 2024-09-18T08:11:23.431 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 8.9160000000000007e-06 2024-09-18T08:11:23.431 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:11:23.431 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:11:23.431 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-18T08:11:23.431 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:23.661923+0000", 2024-09-18T08:11:23.431 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.002913279 2024-09-18T08:11:23.431 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:11:23.431 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:11:23.431 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-09-18T08:11:23.431 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:23.662030+0000", 2024-09-18T08:11:23.431 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00010665700000000001 2024-09-18T08:11:23.431 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:11:23.431 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:11:23.432 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "waiting for subops from 0,3", 2024-09-18T08:11:23.432 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:23.662167+0000", 2024-09-18T08:11:23.432 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00013728300000000001 2024-09-18T08:11:23.432 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:11:23.432 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:11:23.432 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "op_commit", 2024-09-18T08:11:23.432 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:23.663255+0000", 2024-09-18T08:11:23.432 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0010883900000000001 2024-09-18T08:11:23.432 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:11:23.432 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:11:23.432 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:11:23.432 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:23.671279+0000", 2024-09-18T08:11:23.433 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0080233749999999993 2024-09-18T08:11:23.433 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:11:23.433 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:11:23.433 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:11:23.433 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:23.687965+0000", 2024-09-18T08:11:23.433 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.016686590000000001 2024-09-18T08:11:23.433 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:11:23.433 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:11:23.433 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "commit_sent", 2024-09-18T08:11:23.433 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:23.688005+0000", 2024-09-18T08:11:23.433 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 3.9604000000000002e-05 2024-09-18T08:11:23.433 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:11:23.433 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:11:23.434 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-09-18T08:11:23.434 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:23.688036+0000", 2024-09-18T08:11:23.434 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 3.1059999999999997e-05 2024-09-18T08:11:23.434 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-18T08:11:23.434 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-09-18T08:11:23.434 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-18T08:11:23.434 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:11:23.434 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:11:23.434 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.14805.0:39805 7.7 7:e9cd36b3:::benchmark_data_smithi007_73155_object39804:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e95)", 2024-09-18T08:11:23.434 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-09-18T08:04:24.061655+0000", 2024-09-18T08:11:23.434 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 419.35349120900003, 2024-09-18T08:11:23.434 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.036883131, 2024-09-18T08:11:23.434 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-09-18T08:11:23.434 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-09-18T08:11:23.435 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T08:11:23.435 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2024-09-18T08:11:23.435 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.14805", 2024-09-18T08:11:23.435 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "172.21.15.7:0/640320436", 2024-09-18T08:11:23.435 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 39805 2024-09-18T08:11:23.435 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:11:23.436 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-09-18T08:11:23.436 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:11:23.436 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-09-18T08:11:23.436 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:24.061655+0000", 2024-09-18T08:11:23.436 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-18T08:11:23.436 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:11:23.436 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:11:23.436 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-09-18T08:11:23.436 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:24.061655+0000", 2024-09-18T08:11:23.436 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-18T08:11:23.436 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:11:23.436 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:11:23.436 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-09-18T08:11:23.437 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:24.061656+0000", 2024-09-18T08:11:23.437 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.4449999999999999e-06 2024-09-18T08:11:23.437 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:11:23.437 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:11:23.437 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-09-18T08:11:23.437 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:24.061686+0000", 2024-09-18T08:11:23.437 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 2.9984e-05 2024-09-18T08:11:23.437 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:11:23.437 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:11:23.437 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-09-18T08:11:23.437 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:24.061699+0000", 2024-09-18T08:11:23.437 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.2850000000000001e-05 2024-09-18T08:11:23.437 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:11:23.438 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:11:23.438 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-09-18T08:11:23.438 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:24.061706+0000", 2024-09-18T08:11:23.438 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 6.8560000000000001e-06 2024-09-18T08:11:23.438 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:11:23.438 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:11:23.438 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-18T08:11:23.438 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:24.061755+0000", 2024-09-18T08:11:23.438 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4.9001000000000001e-05 2024-09-18T08:11:23.438 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:11:23.438 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:11:23.438 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-09-18T08:11:23.439 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:24.061896+0000", 2024-09-18T08:11:23.439 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00014040499999999999 2024-09-18T08:11:23.439 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:11:23.439 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:11:23.439 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "waiting for subops from 0,3", 2024-09-18T08:11:23.439 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:24.062006+0000", 2024-09-18T08:11:23.439 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000110813 2024-09-18T08:11:23.439 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:11:23.439 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:11:23.439 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:11:23.439 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:24.065677+0000", 2024-09-18T08:11:23.439 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0036702800000000002 2024-09-18T08:11:23.440 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:11:23.440 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:11:23.440 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "op_commit", 2024-09-18T08:11:23.440 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:24.065694+0000", 2024-09-18T08:11:23.440 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.7876e-05 2024-09-18T08:11:23.440 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:11:23.440 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:11:23.440 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:11:23.440 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:24.098357+0000", 2024-09-18T08:11:23.440 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.032662140999999999 2024-09-18T08:11:23.440 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:11:23.440 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:11:23.440 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "commit_sent", 2024-09-18T08:11:23.441 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:24.098447+0000", 2024-09-18T08:11:23.441 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 9.0724999999999996e-05 2024-09-18T08:11:23.441 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:11:23.441 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:11:23.441 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-09-18T08:11:23.441 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:24.098538+0000", 2024-09-18T08:11:23.441 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 9.0754999999999994e-05 2024-09-18T08:11:23.441 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-18T08:11:23.441 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-09-18T08:11:23.441 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-18T08:11:23.441 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:11:23.441 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:11:23.441 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.14805.0:39807 7.d 7:bf1e27dc:::benchmark_data_smithi007_73155_object39806:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e95)", 2024-09-18T08:11:23.442 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-09-18T08:04:24.061827+0000", 2024-09-18T08:11:23.442 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 419.353318914, 2024-09-18T08:11:23.442 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.037358672000000002, 2024-09-18T08:11:23.442 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-09-18T08:11:23.442 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-09-18T08:11:23.442 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T08:11:23.442 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2024-09-18T08:11:23.442 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.14805", 2024-09-18T08:11:23.442 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "172.21.15.7:0/640320436", 2024-09-18T08:11:23.442 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 39807 2024-09-18T08:11:23.442 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:11:23.442 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-09-18T08:11:23.443 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:11:23.443 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-09-18T08:11:23.443 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:24.061827+0000", 2024-09-18T08:11:23.443 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-18T08:11:23.443 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:11:23.443 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:11:23.443 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-09-18T08:11:23.443 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:24.061827+0000", 2024-09-18T08:11:23.443 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-18T08:11:23.443 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:11:23.443 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:11:23.443 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-09-18T08:11:23.443 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:24.061828+0000", 2024-09-18T08:11:23.444 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 7.9899999999999999e-07 2024-09-18T08:11:23.444 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:11:23.444 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:11:23.444 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-09-18T08:11:23.444 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:24.061888+0000", 2024-09-18T08:11:23.444 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 5.9497000000000003e-05 2024-09-18T08:11:23.444 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:11:23.444 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:11:23.444 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-09-18T08:11:23.444 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:24.061904+0000", 2024-09-18T08:11:23.444 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.6393999999999999e-05 2024-09-18T08:11:23.444 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:11:23.444 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:11:23.444 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-09-18T08:11:23.445 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:24.061912+0000", 2024-09-18T08:11:23.445 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 8.0909999999999993e-06 2024-09-18T08:11:23.445 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:11:23.445 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:11:23.445 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-18T08:11:23.445 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:24.061968+0000", 2024-09-18T08:11:23.445 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 5.5543e-05 2024-09-18T08:11:23.445 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:11:23.445 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:11:23.445 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-09-18T08:11:23.445 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:24.062061+0000", 2024-09-18T08:11:23.445 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 9.3558000000000001e-05 2024-09-18T08:11:23.445 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:11:23.445 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:11:23.446 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "waiting for subops from 0,1", 2024-09-18T08:11:23.446 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:24.062163+0000", 2024-09-18T08:11:23.446 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000101903 2024-09-18T08:11:23.446 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:11:23.446 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:11:23.446 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "op_commit", 2024-09-18T08:11:23.446 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:24.064018+0000", 2024-09-18T08:11:23.446 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.001855071 2024-09-18T08:11:23.446 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:11:23.446 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:11:23.446 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:11:23.446 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:24.064283+0000", 2024-09-18T08:11:23.446 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000265173 2024-09-18T08:11:23.446 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:11:23.447 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:11:23.447 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:11:23.447 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:24.099043+0000", 2024-09-18T08:11:23.447 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.034760159999999998 2024-09-18T08:11:23.447 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:11:23.447 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:11:23.447 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "commit_sent", 2024-09-18T08:11:23.447 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:24.099100+0000", 2024-09-18T08:11:23.447 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 5.6147000000000003e-05 2024-09-18T08:11:23.447 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:11:23.447 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:11:23.447 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-09-18T08:11:23.448 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:24.099186+0000", 2024-09-18T08:11:23.448 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 8.6335999999999995e-05 2024-09-18T08:11:23.448 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-18T08:11:23.448 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-09-18T08:11:23.448 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-18T08:11:23.448 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:11:23.448 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:11:23.448 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.14805.0:39808 7.d 7:b98cab8f:::benchmark_data_smithi007_73155_object39807:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e95)", 2024-09-18T08:11:23.448 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-09-18T08:04:24.067964+0000", 2024-09-18T08:11:23.448 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 419.34718247699999, 2024-09-18T08:11:23.448 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.034258180999999999, 2024-09-18T08:11:23.448 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-09-18T08:11:23.449 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-09-18T08:11:23.449 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T08:11:23.449 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2024-09-18T08:11:23.449 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.14805", 2024-09-18T08:11:23.449 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "172.21.15.7:0/640320436", 2024-09-18T08:11:23.449 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 39808 2024-09-18T08:11:23.449 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:11:23.449 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-09-18T08:11:23.449 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:11:23.449 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-09-18T08:11:23.449 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:24.067964+0000", 2024-09-18T08:11:23.449 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-18T08:11:23.449 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:11:23.450 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:11:23.450 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-09-18T08:11:23.450 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:24.067964+0000", 2024-09-18T08:11:23.450 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-18T08:11:23.450 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:11:23.450 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:11:23.451 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:11:23.451 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-09-18T08:11:23.451 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:24.067965+0000", 2024-09-18T08:11:23.451 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.4750000000000001e-06 2024-09-18T08:11:23.451 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:11:23.451 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:11:23.451 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-09-18T08:11:23.452 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:24.067992+0000", 2024-09-18T08:11:23.452 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 2.6760000000000001e-05 2024-09-18T08:11:23.452 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:11:23.452 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:11:23.452 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-09-18T08:11:23.452 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:24.068004+0000", 2024-09-18T08:11:23.452 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.2456000000000001e-05 2024-09-18T08:11:23.452 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:11:23.452 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:11:23.452 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-09-18T08:11:23.452 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:24.068012+0000", 2024-09-18T08:11:23.452 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 8.0779999999999996e-06 2024-09-18T08:11:23.452 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:11:23.453 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:11:23.453 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-18T08:11:23.453 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:24.068072+0000", 2024-09-18T08:11:23.453 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 5.9299000000000003e-05 2024-09-18T08:11:23.453 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:11:23.453 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:11:23.453 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-09-18T08:11:23.453 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:24.068164+0000", 2024-09-18T08:11:23.453 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 9.2053999999999997e-05 2024-09-18T08:11:23.453 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:11:23.453 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:11:23.453 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "waiting for subops from 0,1", 2024-09-18T08:11:23.453 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:24.096945+0000", 2024-09-18T08:11:23.454 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.028781128 2024-09-18T08:11:23.454 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:11:23.454 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:11:23.454 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "op_commit", 2024-09-18T08:11:23.454 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:24.099254+0000", 2024-09-18T08:11:23.454 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.002309218 2024-09-18T08:11:23.454 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:11:23.454 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:11:23.454 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:11:23.454 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:24.100291+0000", 2024-09-18T08:11:23.454 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0010365280000000001 2024-09-18T08:11:23.454 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:11:23.454 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:11:23.455 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:11:23.455 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:24.102125+0000", 2024-09-18T08:11:23.455 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0018346599999999999 2024-09-18T08:11:23.455 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:11:23.455 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:11:23.455 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "commit_sent", 2024-09-18T08:11:23.455 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:24.102184+0000", 2024-09-18T08:11:23.455 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 5.8965000000000002e-05 2024-09-18T08:11:23.455 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:11:23.455 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:11:23.455 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-09-18T08:11:23.455 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:24.102222+0000", 2024-09-18T08:11:23.455 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 3.756e-05 2024-09-18T08:11:23.456 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-18T08:11:23.456 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-09-18T08:11:23.456 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-18T08:11:23.456 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:11:23.456 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:11:23.456 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_repop_reply(client.14805.0:39807 7.d e95/83 ondisk, result = 0)", 2024-09-18T08:11:23.456 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-09-18T08:04:24.069321+0000", 2024-09-18T08:11:23.456 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 419.34582511299999, 2024-09-18T08:11:23.456 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.029872724999999999, 2024-09-18T08:11:23.456 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-09-18T08:11:23.456 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-09-18T08:11:23.456 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "started", 2024-09-18T08:11:23.456 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-09-18T08:11:23.457 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:11:23.457 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-09-18T08:11:23.457 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:24.069321+0000", 2024-09-18T08:11:23.457 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-18T08:11:23.457 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:11:23.457 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:11:23.457 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-09-18T08:11:23.457 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:24.069321+0000", 2024-09-18T08:11:23.457 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-18T08:11:23.457 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:11:23.457 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:11:23.458 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-09-18T08:11:23.458 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:24.069322+0000", 2024-09-18T08:11:23.458 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 8.3799999999999996e-07 2024-09-18T08:11:23.458 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:11:23.458 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:11:23.458 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-09-18T08:11:23.458 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:24.069327+0000", 2024-09-18T08:11:23.458 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4.8679999999999998e-06 2024-09-18T08:11:23.458 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:11:23.458 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:11:23.458 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-09-18T08:11:23.458 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:24.098180+0000", 2024-09-18T08:11:23.458 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.028853407000000001 2024-09-18T08:11:23.459 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:11:23.459 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:11:23.459 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-09-18T08:11:23.459 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:24.098189+0000", 2024-09-18T08:11:23.459 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 8.8850000000000005e-06 2024-09-18T08:11:23.459 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:11:23.459 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:11:23.459 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-18T08:11:23.459 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:24.099029+0000", 2024-09-18T08:11:23.459 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00084036500000000004 2024-09-18T08:11:23.459 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:11:23.459 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:11:23.459 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-09-18T08:11:23.460 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:24.099037+0000", 2024-09-18T08:11:23.460 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 8.0150000000000003e-06 2024-09-18T08:11:23.460 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:11:23.460 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:11:23.460 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-09-18T08:11:23.460 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:24.099194+0000", 2024-09-18T08:11:23.460 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00015634700000000001 2024-09-18T08:11:23.460 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-18T08:11:23.460 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-09-18T08:11:23.460 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-18T08:11:23.460 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:11:23.460 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:11:23.460 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.14805.0:39817 7.c 7:3826d3a8:::benchmark_data_smithi007_73155_object39816:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e96)", 2024-09-18T08:11:23.461 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-09-18T08:04:24.077849+0000", 2024-09-18T08:11:23.461 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 419.337297165, 2024-09-18T08:11:23.461 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.028025705000000001, 2024-09-18T08:11:23.461 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-09-18T08:11:23.461 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-09-18T08:11:23.461 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T08:11:23.461 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2024-09-18T08:11:23.461 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.14805", 2024-09-18T08:11:23.461 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "172.21.15.7:0/640320436", 2024-09-18T08:11:23.461 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 39817 2024-09-18T08:11:23.461 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:11:23.461 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-09-18T08:11:23.461 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:11:23.462 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-09-18T08:11:23.462 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:24.077849+0000", 2024-09-18T08:11:23.462 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-18T08:11:23.462 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:11:23.462 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:11:23.462 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-09-18T08:11:23.462 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:24.077849+0000", 2024-09-18T08:11:23.462 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-18T08:11:23.462 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:11:23.462 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:11:23.462 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-09-18T08:11:23.462 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:24.077851+0000", 2024-09-18T08:11:23.462 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 2.322e-06 2024-09-18T08:11:23.462 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:11:23.463 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:11:23.463 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-09-18T08:11:23.463 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:24.077879+0000", 2024-09-18T08:11:23.463 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 2.7596000000000001e-05 2024-09-18T08:11:23.463 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:11:23.463 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:11:23.463 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-09-18T08:11:23.463 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:24.097805+0000", 2024-09-18T08:11:23.463 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.019925552999999999 2024-09-18T08:11:23.463 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:11:23.463 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:11:23.463 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-09-18T08:11:23.463 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:24.097820+0000", 2024-09-18T08:11:23.464 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.5705999999999998e-05 2024-09-18T08:11:23.464 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:11:23.464 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:11:23.464 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-18T08:11:23.464 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:24.097984+0000", 2024-09-18T08:11:23.464 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00016333499999999999 2024-09-18T08:11:23.464 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:11:23.464 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:11:23.464 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "op must wait for map", 2024-09-18T08:11:23.464 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:24.097990+0000", 2024-09-18T08:11:23.464 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 6.81e-06 2024-09-18T08:11:23.464 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:11:23.465 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:11:23.465 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-18T08:11:23.465 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:24.102003+0000", 2024-09-18T08:11:23.465 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0040126160000000001 2024-09-18T08:11:23.465 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:11:23.465 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:11:23.465 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-09-18T08:11:23.465 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:24.102155+0000", 2024-09-18T08:11:23.465 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00015247500000000001 2024-09-18T08:11:23.465 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:11:23.465 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:11:23.465 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "waiting for subops from 3,7", 2024-09-18T08:11:23.465 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:24.102305+0000", 2024-09-18T08:11:23.465 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000149872 2024-09-18T08:11:23.466 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:11:23.466 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:11:23.466 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "op_commit", 2024-09-18T08:11:23.466 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:24.105200+0000", 2024-09-18T08:11:23.466 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:11:23.467 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6' 2024-09-18T08:11:23.467 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0028950730000000002 2024-09-18T08:11:23.467 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:11:23.467 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:11:23.467 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:11:23.467 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:24.105248+0000", 2024-09-18T08:11:23.468 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4.7787999999999997e-05 2024-09-18T08:11:23.468 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:11:23.468 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:11:23.468 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:11:23.468 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:24.105772+0000", 2024-09-18T08:11:23.468 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00052425699999999998 2024-09-18T08:11:23.468 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:11:23.468 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:11:23.468 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "commit_sent", 2024-09-18T08:11:23.468 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:24.105826+0000", 2024-09-18T08:11:23.468 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 5.393e-05 2024-09-18T08:11:23.468 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:11:23.468 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:11:23.469 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-09-18T08:11:23.469 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:24.105875+0000", 2024-09-18T08:11:23.469 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4.8371999999999997e-05 2024-09-18T08:11:23.469 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-18T08:11:23.469 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-09-18T08:11:23.469 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-18T08:11:23.469 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:11:23.469 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:11:23.469 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.14805.0:43273 7.c 7:38821c7c:::benchmark_data_smithi007_73155_object43272:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e96)", 2024-09-18T08:11:23.469 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-09-18T08:04:26.167716+0000", 2024-09-18T08:11:23.469 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 417.24743023600001, 2024-09-18T08:11:23.469 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.027026471999999999, 2024-09-18T08:11:23.469 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-09-18T08:11:23.470 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-09-18T08:11:23.470 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T08:11:23.470 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2024-09-18T08:11:23.470 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.14805", 2024-09-18T08:11:23.470 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "172.21.15.7:0/640320436", 2024-09-18T08:11:23.470 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 43273 2024-09-18T08:11:23.470 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:11:23.470 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-09-18T08:11:23.470 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:11:23.470 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-09-18T08:11:23.470 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:26.167716+0000", 2024-09-18T08:11:23.470 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-18T08:11:23.470 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:11:23.470 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:11:23.471 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-09-18T08:11:23.471 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:26.167716+0000", 2024-09-18T08:11:23.471 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-18T08:11:23.471 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:11:23.471 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:11:23.471 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-09-18T08:11:23.471 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:26.167717+0000", 2024-09-18T08:11:23.471 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.3629999999999999e-06 2024-09-18T08:11:23.471 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:11:23.471 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:11:23.471 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-09-18T08:11:23.471 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:26.187242+0000", 2024-09-18T08:11:23.471 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.019525094999999999 2024-09-18T08:11:23.471 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:11:23.471 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:11:23.472 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-09-18T08:11:23.472 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:26.187255+0000", 2024-09-18T08:11:23.472 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.2669e-05 2024-09-18T08:11:23.472 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:11:23.472 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:11:23.472 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-09-18T08:11:23.472 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:26.187264+0000", 2024-09-18T08:11:23.472 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 9.3770000000000002e-06 2024-09-18T08:11:23.472 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:11:23.472 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:11:23.472 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-18T08:11:23.472 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:26.187321+0000", 2024-09-18T08:11:23.472 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 5.6379999999999999e-05 2024-09-18T08:11:23.473 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:11:23.473 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:11:23.473 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-09-18T08:11:23.473 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:26.187444+0000", 2024-09-18T08:11:23.473 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00012314699999999999 2024-09-18T08:11:23.473 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:11:23.473 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:11:23.473 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "waiting for subops from 3,7", 2024-09-18T08:11:23.473 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:26.187575+0000", 2024-09-18T08:11:23.473 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00013138700000000001 2024-09-18T08:11:23.473 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:11:23.474 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:11:23.474 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "op_commit", 2024-09-18T08:11:23.474 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:26.189195+0000", 2024-09-18T08:11:23.474 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0016195770000000001 2024-09-18T08:11:23.474 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:11:23.474 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:11:23.474 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:11:23.474 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:26.189327+0000", 2024-09-18T08:11:23.474 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00013181099999999999 2024-09-18T08:11:23.474 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:11:23.474 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:11:23.474 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:11:23.474 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:26.194627+0000", 2024-09-18T08:11:23.475 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0052998799999999999 2024-09-18T08:11:23.475 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:11:23.475 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:11:23.475 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "commit_sent", 2024-09-18T08:11:23.475 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:26.194693+0000", 2024-09-18T08:11:23.475 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 6.6748999999999994e-05 2024-09-18T08:11:23.475 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:11:23.475 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:11:23.475 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-09-18T08:11:23.475 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:26.194742+0000", 2024-09-18T08:11:23.475 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4.9036999999999998e-05 2024-09-18T08:11:23.475 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-18T08:11:23.475 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-09-18T08:11:23.476 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-18T08:11:23.476 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:11:23.476 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:11:23.476 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.14805.0:43281 7.c 7:35a814bc:::benchmark_data_smithi007_73155_object43280:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e96)", 2024-09-18T08:11:23.476 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-09-18T08:04:26.187285+0000", 2024-09-18T08:11:23.476 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 417.22786104800002, 2024-09-18T08:11:23.476 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.026194973999999999, 2024-09-18T08:11:23.476 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-09-18T08:11:23.476 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-09-18T08:11:23.476 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T08:11:23.476 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2024-09-18T08:11:23.476 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.14805", 2024-09-18T08:11:23.477 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "172.21.15.7:0/640320436", 2024-09-18T08:11:23.477 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 43281 2024-09-18T08:11:23.477 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:11:23.477 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-09-18T08:11:23.477 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:11:23.477 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-09-18T08:11:23.477 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:26.187285+0000", 2024-09-18T08:11:23.477 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-18T08:11:23.477 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:11:23.477 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:11:23.477 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-09-18T08:11:23.477 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:26.187285+0000", 2024-09-18T08:11:23.478 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-18T08:11:23.478 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:11:23.478 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:11:23.478 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-09-18T08:11:23.478 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:26.187286+0000", 2024-09-18T08:11:23.478 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.2190000000000001e-06 2024-09-18T08:11:23.478 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:11:23.478 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:11:23.478 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-09-18T08:11:23.478 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:26.206523+0000", 2024-09-18T08:11:23.478 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.019236797999999999 2024-09-18T08:11:23.478 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:11:23.478 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:11:23.479 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-09-18T08:11:23.479 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:26.206550+0000", 2024-09-18T08:11:23.479 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 2.6509999999999999e-05 2024-09-18T08:11:23.479 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:11:23.479 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:11:23.479 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-09-18T08:11:23.479 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:26.206567+0000", 2024-09-18T08:11:23.479 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.7401000000000002e-05 2024-09-18T08:11:23.479 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:11:23.479 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:11:23.479 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-18T08:11:23.479 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:26.206657+0000", 2024-09-18T08:11:23.479 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 8.9485000000000001e-05 2024-09-18T08:11:23.480 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:11:23.480 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:11:23.480 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-09-18T08:11:23.480 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:26.206738+0000", 2024-09-18T08:11:23.480 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 8.1077999999999999e-05 2024-09-18T08:11:23.480 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:11:23.480 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:11:23.480 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "waiting for subops from 3,7", 2024-09-18T08:11:23.480 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:26.206832+0000", 2024-09-18T08:11:23.480 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 9.4160000000000006e-05 2024-09-18T08:11:23.480 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:11:23.481 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:11:23.481 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:11:23.481 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:26.213326+0000", 2024-09-18T08:11:23.481 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0064943099999999997 2024-09-18T08:11:23.481 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:11:23.481 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:11:23.481 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "op_commit", 2024-09-18T08:11:23.481 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:26.213342+0000", 2024-09-18T08:11:23.481 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.5415000000000001e-05 2024-09-18T08:11:23.481 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:11:23.481 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:11:23.481 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:11:23.481 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:26.213393+0000", 2024-09-18T08:11:23.482 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 5.1038999999999997e-05 2024-09-18T08:11:23.482 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:11:23.482 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:11:23.482 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "commit_sent", 2024-09-18T08:11:23.482 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:26.213443+0000", 2024-09-18T08:11:23.482 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4.9938999999999997e-05 2024-09-18T08:11:23.482 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:11:23.482 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:11:23.482 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-09-18T08:11:23.482 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:26.213480+0000", 2024-09-18T08:11:23.482 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 3.7620000000000002e-05 2024-09-18T08:11:23.482 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-18T08:11:23.483 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-09-18T08:11:23.483 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-18T08:11:23.483 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:11:23.483 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:11:23.483 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.14805.0:43731 7.d 7:b3fc8faf:::benchmark_data_smithi007_73155_object43730:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e97)", 2024-09-18T08:11:23.483 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-09-18T08:04:26.432079+0000", 2024-09-18T08:11:23.483 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 416.98306689100002, 2024-09-18T08:11:23.483 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.027497293999999999, 2024-09-18T08:11:23.483 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-09-18T08:11:23.483 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-09-18T08:11:23.483 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T08:11:23.483 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2024-09-18T08:11:23.483 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.14805", 2024-09-18T08:11:23.484 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "172.21.15.7:0/640320436", 2024-09-18T08:11:23.484 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 43731 2024-09-18T08:11:23.484 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:11:23.484 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-09-18T08:11:23.484 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:11:23.484 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-09-18T08:11:23.484 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:26.432079+0000", 2024-09-18T08:11:23.484 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-18T08:11:23.484 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:11:23.484 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:11:23.484 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-09-18T08:11:23.484 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:26.432079+0000", 2024-09-18T08:11:23.484 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-18T08:11:23.485 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:11:23.485 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:11:23.485 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-09-18T08:11:23.485 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:26.432080+0000", 2024-09-18T08:11:23.485 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.128e-06 2024-09-18T08:11:23.485 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:11:23.485 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:11:23.485 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-09-18T08:11:23.485 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:26.432153+0000", 2024-09-18T08:11:23.485 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 7.2521999999999994e-05 2024-09-18T08:11:23.485 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:11:23.485 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:11:23.486 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-09-18T08:11:23.486 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:26.432163+0000", 2024-09-18T08:11:23.486 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.0115999999999999e-05 2024-09-18T08:11:23.486 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:11:23.486 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:11:23.486 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-09-18T08:11:23.486 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:26.432170+0000", 2024-09-18T08:11:23.486 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 7.0729999999999999e-06 2024-09-18T08:11:23.486 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:11:23.486 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:11:23.486 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-18T08:11:23.486 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:26.432225+0000", 2024-09-18T08:11:23.486 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 5.4463999999999999e-05 2024-09-18T08:11:23.486 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:11:23.487 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:11:23.487 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-09-18T08:11:23.487 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:26.432342+0000", 2024-09-18T08:11:23.487 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000117118 2024-09-18T08:11:23.487 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:11:23.487 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:11:23.487 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "waiting for subops from 0,1", 2024-09-18T08:11:23.487 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:26.432467+0000", 2024-09-18T08:11:23.487 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00012505700000000001 2024-09-18T08:11:23.487 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:11:23.487 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:11:23.487 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "op_commit", 2024-09-18T08:11:23.487 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:26.433881+0000", 2024-09-18T08:11:23.488 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.001413812 2024-09-18T08:11:23.488 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:11:23.488 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:11:23.488 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:11:23.488 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:26.434589+0000", 2024-09-18T08:11:23.488 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00070883600000000001 2024-09-18T08:11:23.488 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:11:23.488 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:11:23.488 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:11:23.488 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:26.459469+0000", 2024-09-18T08:11:23.488 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.024879951000000001 2024-09-18T08:11:23.488 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:11:23.488 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:11:23.489 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "commit_sent", 2024-09-18T08:11:23.489 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:26.459527+0000", 2024-09-18T08:11:23.489 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 5.7126999999999997e-05 2024-09-18T08:11:23.489 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:11:23.489 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:11:23.489 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-09-18T08:11:23.489 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:26.459577+0000", 2024-09-18T08:11:23.489 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 5.0090000000000003e-05 2024-09-18T08:11:23.490 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-18T08:11:23.490 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-09-18T08:11:23.490 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-18T08:11:23.490 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:11:23.490 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:11:23.490 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.14805.0:63861 7.d 7:bf91474b:::benchmark_data_smithi007_73155_object63860:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e100)", 2024-09-18T08:11:23.490 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-09-18T08:04:41.125989+0000", 2024-09-18T08:11:23.490 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 402.28915759900002, 2024-09-18T08:11:23.490 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.031566575999999999, 2024-09-18T08:11:23.490 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-09-18T08:11:23.490 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-09-18T08:11:23.490 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T08:11:23.491 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2024-09-18T08:11:23.491 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.14805", 2024-09-18T08:11:23.491 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "172.21.15.7:0/640320436", 2024-09-18T08:11:23.491 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 63861 2024-09-18T08:11:23.491 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:11:23.491 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-09-18T08:11:23.491 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:11:23.491 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-09-18T08:11:23.491 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:41.125989+0000", 2024-09-18T08:11:23.491 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-18T08:11:23.491 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:11:23.491 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:11:23.491 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-09-18T08:11:23.492 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:41.125989+0000", 2024-09-18T08:11:23.492 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-18T08:11:23.492 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:11:23.492 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:11:23.492 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-09-18T08:11:23.492 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:41.125990+0000", 2024-09-18T08:11:23.492 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.6109999999999999e-06 2024-09-18T08:11:23.492 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:11:23.492 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:11:23.492 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-09-18T08:11:23.492 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:41.126017+0000", 2024-09-18T08:11:23.492 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 2.6594999999999999e-05 2024-09-18T08:11:23.493 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:11:23.493 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:11:23.493 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-09-18T08:11:23.493 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:41.126035+0000", 2024-09-18T08:11:23.493 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.7842999999999998e-05 2024-09-18T08:11:23.493 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:11:23.493 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:11:23.493 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-09-18T08:11:23.493 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:41.126043+0000", 2024-09-18T08:11:23.493 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 8.8489999999999995e-06 2024-09-18T08:11:23.493 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:11:23.493 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:11:23.493 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-18T08:11:23.493 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:41.126104+0000", 2024-09-18T08:11:23.494 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 6.0448999999999997e-05 2024-09-18T08:11:23.494 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:11:23.494 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:11:23.494 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-09-18T08:11:23.494 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:41.126225+0000", 2024-09-18T08:11:23.494 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000121101 2024-09-18T08:11:23.494 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:11:23.494 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:11:23.494 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "waiting for subops from 0,1", 2024-09-18T08:11:23.494 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:41.126366+0000", 2024-09-18T08:11:23.494 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00014059899999999999 2024-09-18T08:11:23.494 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:11:23.495 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:11:23.495 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "op_commit", 2024-09-18T08:11:23.495 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:41.129750+0000", 2024-09-18T08:11:23.495 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.003383863 2024-09-18T08:11:23.495 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:11:23.495 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:11:23.495 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:11:23.495 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:41.140266+0000", 2024-09-18T08:11:23.495 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.010516266999999999 2024-09-18T08:11:23.495 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:11:23.495 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:11:23.495 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:11:23.495 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:41.157441+0000", 2024-09-18T08:11:23.495 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.017174891000000001 2024-09-18T08:11:23.495 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:11:23.496 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:11:23.496 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "commit_sent", 2024-09-18T08:11:23.496 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:41.157509+0000", 2024-09-18T08:11:23.496 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 6.8627999999999994e-05 2024-09-18T08:11:23.496 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:11:23.496 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:11:23.496 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-09-18T08:11:23.496 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:41.157555+0000", 2024-09-18T08:11:23.496 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4.5880000000000001e-05 2024-09-18T08:11:23.496 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-18T08:11:23.496 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-09-18T08:11:23.496 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-18T08:11:23.496 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:11:23.497 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:11:23.497 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.14805.0:63862 7.d 7:b1f318bc:::benchmark_data_smithi007_73155_object63861:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e100)", 2024-09-18T08:11:23.497 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-09-18T08:04:41.126934+0000", 2024-09-18T08:11:23.497 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 402.28821246199999, 2024-09-18T08:11:23.497 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.030729600999999999, 2024-09-18T08:11:23.497 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-09-18T08:11:23.497 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-09-18T08:11:23.497 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T08:11:23.497 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2024-09-18T08:11:23.497 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.14805", 2024-09-18T08:11:23.497 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "172.21.15.7:0/640320436", 2024-09-18T08:11:23.497 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 63862 2024-09-18T08:11:23.497 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:11:23.498 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-09-18T08:11:23.498 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:11:23.498 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-09-18T08:11:23.498 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:41.126934+0000", 2024-09-18T08:11:23.498 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-18T08:11:23.498 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:11:23.498 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:11:23.498 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-09-18T08:11:23.498 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:41.126934+0000", 2024-09-18T08:11:23.498 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-18T08:11:23.498 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:11:23.498 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:11:23.499 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-09-18T08:11:23.499 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:41.126935+0000", 2024-09-18T08:11:23.499 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 7.9599999999999998e-07 2024-09-18T08:11:23.499 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:11:23.499 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:11:23.499 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-09-18T08:11:23.499 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:41.126989+0000", 2024-09-18T08:11:23.499 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 5.4783e-05 2024-09-18T08:11:23.499 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:11:23.499 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:11:23.499 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-09-18T08:11:23.499 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:41.126999+0000", 2024-09-18T08:11:23.499 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 9.8889999999999997e-06 2024-09-18T08:11:23.499 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:11:23.500 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:11:23.500 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-09-18T08:11:23.500 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:41.127007+0000", 2024-09-18T08:11:23.500 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 7.4399999999999999e-06 2024-09-18T08:11:23.500 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:11:23.500 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:11:23.500 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-18T08:11:23.500 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:41.128171+0000", 2024-09-18T08:11:23.500 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0011642600000000001 2024-09-18T08:11:23.500 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:11:23.500 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:11:23.500 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-09-18T08:11:23.501 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:41.128287+0000", 2024-09-18T08:11:23.501 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000116508 2024-09-18T08:11:23.501 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:11:23.501 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:11:23.501 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "waiting for subops from 0,1", 2024-09-18T08:11:23.501 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:41.128375+0000", 2024-09-18T08:11:23.501 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 8.7300999999999996e-05 2024-09-18T08:11:23.501 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:11:23.501 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:11:23.501 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "op_commit", 2024-09-18T08:11:23.501 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:41.130919+0000", 2024-09-18T08:11:23.502 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0025447450000000002 2024-09-18T08:11:23.502 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:11:23.502 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:11:23.502 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:11:23.502 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:41.140334+0000", 2024-09-18T08:11:23.502 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0094146549999999992 2024-09-18T08:11:23.502 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:11:23.502 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:11:23.502 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:11:23.502 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:41.157602+0000", 2024-09-18T08:11:23.502 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.017267966999999999 2024-09-18T08:11:23.502 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:11:23.503 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:11:23.503 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "commit_sent", 2024-09-18T08:11:23.503 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:41.157636+0000", 2024-09-18T08:11:23.503 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 3.4214000000000001e-05 2024-09-18T08:11:23.503 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:11:23.503 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:11:23.503 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-09-18T08:11:23.503 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:41.157663+0000", 2024-09-18T08:11:23.503 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 2.7042999999999998e-05 2024-09-18T08:11:23.503 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-18T08:11:23.503 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-09-18T08:11:23.503 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-18T08:11:23.503 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:11:23.504 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:11:23.504 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.14805.0:63864 7.d 7:bb7b1761:::benchmark_data_smithi007_73155_object63863:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e100)", 2024-09-18T08:11:23.504 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-09-18T08:04:41.127581+0000", 2024-09-18T08:11:23.504 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 402.28756499899998, 2024-09-18T08:11:23.504 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.030231638000000002, 2024-09-18T08:11:23.504 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-09-18T08:11:23.504 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-09-18T08:11:23.504 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T08:11:23.504 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2024-09-18T08:11:23.504 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.14805", 2024-09-18T08:11:23.504 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "172.21.15.7:0/640320436", 2024-09-18T08:11:23.504 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 63864 2024-09-18T08:11:23.505 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:11:23.505 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-09-18T08:11:23.505 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:11:23.505 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-09-18T08:11:23.505 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:41.127581+0000", 2024-09-18T08:11:23.505 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-18T08:11:23.505 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:11:23.505 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:11:23.505 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-09-18T08:11:23.505 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:41.127581+0000", 2024-09-18T08:11:23.505 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-18T08:11:23.505 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:11:23.505 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:11:23.506 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-09-18T08:11:23.506 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:41.127582+0000", 2024-09-18T08:11:23.506 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 7.8999999999999995e-07 2024-09-18T08:11:23.506 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:11:23.506 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:11:23.506 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-09-18T08:11:23.506 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:41.127648+0000", 2024-09-18T08:11:23.506 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 6.5618999999999996e-05 2024-09-18T08:11:23.506 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:11:23.506 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:11:23.506 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-09-18T08:11:23.506 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:41.127658+0000", 2024-09-18T08:11:23.506 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.0192e-05 2024-09-18T08:11:23.506 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:11:23.507 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:11:23.507 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-09-18T08:11:23.507 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:41.127663+0000", 2024-09-18T08:11:23.507 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 5.2800000000000003e-06 2024-09-18T08:11:23.507 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:11:23.507 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:11:23.507 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-18T08:11:23.507 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:41.129783+0000", 2024-09-18T08:11:23.507 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0021199740000000002 2024-09-18T08:11:23.507 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:11:23.507 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:11:23.507 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-09-18T08:11:23.507 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:41.129885+0000", 2024-09-18T08:11:23.508 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000101987 2024-09-18T08:11:23.508 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:11:23.508 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:11:23.508 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "waiting for subops from 0,1", 2024-09-18T08:11:23.508 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:41.129980+0000", 2024-09-18T08:11:23.508 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 9.4480000000000003e-05 2024-09-18T08:11:23.508 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:11:23.508 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:11:23.508 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "op_commit", 2024-09-18T08:11:23.508 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:41.131035+0000", 2024-09-18T08:11:23.508 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.001055884 2024-09-18T08:11:23.508 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:11:23.508 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:11:23.508 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:11:23.509 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:41.140376+0000", 2024-09-18T08:11:23.509 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0093401439999999999 2024-09-18T08:11:23.509 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:11:23.509 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:11:23.509 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:11:23.509 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:41.157713+0000", 2024-09-18T08:11:23.509 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.017337676999999999 2024-09-18T08:11:23.509 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:11:23.509 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:11:23.509 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "commit_sent", 2024-09-18T08:11:23.509 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:41.157771+0000", 2024-09-18T08:11:23.509 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 5.7674999999999999e-05 2024-09-18T08:11:23.509 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:11:23.510 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:11:23.510 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-09-18T08:11:23.510 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:41.157813+0000", 2024-09-18T08:11:23.510 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4.1936e-05 2024-09-18T08:11:23.510 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-18T08:11:23.510 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-09-18T08:11:23.510 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-18T08:11:23.510 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:11:23.510 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:11:23.510 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.14805.0:63883 7.7 7:ebafa8b8:::benchmark_data_smithi007_73155_object63882:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e100)", 2024-09-18T08:11:23.510 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-09-18T08:04:41.154209+0000", 2024-09-18T08:11:23.510 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 402.26093680700001, 2024-09-18T08:11:23.510 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.042338318999999999, 2024-09-18T08:11:23.510 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-09-18T08:11:23.511 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-09-18T08:11:23.511 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T08:11:23.511 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2024-09-18T08:11:23.511 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.14805", 2024-09-18T08:11:23.511 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "172.21.15.7:0/640320436", 2024-09-18T08:11:23.511 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 63883 2024-09-18T08:11:23.511 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:11:23.511 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-09-18T08:11:23.512 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:11:23 smithi007 ceph-mon[29459]: 3.c scrub starts 2024-09-18T08:11:23.512 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:11:23 smithi007 ceph-mon[29459]: 3.c scrub ok 2024-09-18T08:11:23.512 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:11:23 smithi007 ceph-mon[29459]: 4.11 scrub starts 2024-09-18T08:11:23.512 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:11:23 smithi007 ceph-mon[29459]: 4.11 scrub ok 2024-09-18T08:11:23.512 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:11:23 smithi007 ceph-mon[29459]: 5.d scrub starts 2024-09-18T08:11:23.512 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:11:23 smithi007 ceph-mon[29459]: 5.d scrub ok 2024-09-18T08:11:23.513 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:11:23.513 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-09-18T08:11:23.513 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:41.154209+0000", 2024-09-18T08:11:23.513 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-18T08:11:23.513 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:11:23.513 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:11:23.513 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-09-18T08:11:23.513 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:41.154209+0000", 2024-09-18T08:11:23.513 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-18T08:11:23.513 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:11:23.513 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:11:23.513 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-09-18T08:11:23.513 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:41.154211+0000", 2024-09-18T08:11:23.514 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.2109999999999999e-06 2024-09-18T08:11:23.514 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:11:23.514 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:11:23.514 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-09-18T08:11:23.514 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:41.154259+0000", 2024-09-18T08:11:23.514 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4.8412000000000003e-05 2024-09-18T08:11:23.514 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:11:23.514 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:11:23.514 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-09-18T08:11:23.514 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:41.154270+0000", 2024-09-18T08:11:23.514 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.0515e-05 2024-09-18T08:11:23.514 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:11:23.514 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:11:23.515 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-09-18T08:11:23.515 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:41.154275+0000", 2024-09-18T08:11:23.515 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 5.4469999999999997e-06 2024-09-18T08:11:23.515 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:11:23.515 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:11:23.515 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-18T08:11:23.515 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:41.154323+0000", 2024-09-18T08:11:23.515 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4.8297999999999997e-05 2024-09-18T08:11:23.515 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:11:23.515 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:11:23.515 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-09-18T08:11:23.515 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:41.154456+0000", 2024-09-18T08:11:23.516 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000132593 2024-09-18T08:11:23.516 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:11:23.516 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:11:23.516 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "waiting for subops from 0,3", 2024-09-18T08:11:23.516 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:41.154615+0000", 2024-09-18T08:11:23.516 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000159049 2024-09-18T08:11:23.516 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:11:23.516 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:11:23.516 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "op_commit", 2024-09-18T08:11:23.516 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:41.156649+0000", 2024-09-18T08:11:23.516 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0020336709999999999 2024-09-18T08:11:23.516 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:11:23.516 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:11:23.517 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:11:23.517 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:41.165163+0000", 2024-09-18T08:11:23.517 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0085141100000000001 2024-09-18T08:11:23.517 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:11:23.517 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:11:23.517 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:11:23.517 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:41.196483+0000", 2024-09-18T08:11:23.517 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.031320300000000002 2024-09-18T08:11:23.517 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:11:23.517 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:11:23.517 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "commit_sent", 2024-09-18T08:11:23.517 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:41.196519+0000", 2024-09-18T08:11:23.517 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 3.6109999999999998e-05 2024-09-18T08:11:23.518 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:11:23.518 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:11:23.518 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-09-18T08:11:23.518 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:41.196548+0000", 2024-09-18T08:11:23.518 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 2.8603e-05 2024-09-18T08:11:23.518 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-18T08:11:23.518 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-09-18T08:11:23.518 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-18T08:11:23.518 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:11:23.518 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:11:23.518 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.14805.0:63908 7.d 7:bd06bac1:::benchmark_data_smithi007_73155_object63907:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e100)", 2024-09-18T08:11:23.518 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-09-18T08:04:41.183375+0000", 2024-09-18T08:11:23.518 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 402.23177104500002, 2024-09-18T08:11:23.519 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.033452764000000003, 2024-09-18T08:11:23.519 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-09-18T08:11:23.519 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-09-18T08:11:23.519 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T08:11:23.519 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2024-09-18T08:11:23.519 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.14805", 2024-09-18T08:11:23.519 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "172.21.15.7:0/640320436", 2024-09-18T08:11:23.519 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 63908 2024-09-18T08:11:23.519 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:11:23.519 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-09-18T08:11:23.519 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:11:23.519 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-09-18T08:11:23.519 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:41.183375+0000", 2024-09-18T08:11:23.520 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-18T08:11:23.520 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:11:23.520 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:11:23.520 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-09-18T08:11:23.520 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:41.183375+0000", 2024-09-18T08:11:23.520 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-18T08:11:23.520 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:11:23.520 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:11:23.520 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-09-18T08:11:23.520 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:41.183376+0000", 2024-09-18T08:11:23.520 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.226e-06 2024-09-18T08:11:23.520 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:11:23.520 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:11:23.521 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-09-18T08:11:23.521 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:41.183403+0000", 2024-09-18T08:11:23.521 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 2.6404e-05 2024-09-18T08:11:23.521 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:11:23.521 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:11:23.521 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-09-18T08:11:23.521 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:41.183417+0000", 2024-09-18T08:11:23.521 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.4399000000000001e-05 2024-09-18T08:11:23.521 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:11:23.521 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:11:23.521 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-09-18T08:11:23.521 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:41.183425+0000", 2024-09-18T08:11:23.521 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 8.1459999999999996e-06 2024-09-18T08:11:23.522 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:11:23.522 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:11:23.522 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-18T08:11:23.522 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:41.183485+0000", 2024-09-18T08:11:23.522 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 5.9673999999999998e-05 2024-09-18T08:11:23.522 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:11:23.522 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:11:23.522 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-09-18T08:11:23.522 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:41.183587+0000", 2024-09-18T08:11:23.522 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000101611 2024-09-18T08:11:23.522 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:11:23.522 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:11:23.522 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "waiting for subops from 0,1", 2024-09-18T08:11:23.523 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:41.183686+0000", 2024-09-18T08:11:23.523 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 9.9823999999999998e-05 2024-09-18T08:11:23.523 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:11:23.523 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:11:23.523 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "op_commit", 2024-09-18T08:11:23.523 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:41.185240+0000", 2024-09-18T08:11:23.523 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0015539340000000001 2024-09-18T08:11:23.523 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:11:23.523 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:11:23.523 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:11:23.523 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:41.191524+0000", 2024-09-18T08:11:23.523 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0062835690000000001 2024-09-18T08:11:23.523 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:11:23.523 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:11:23.524 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:11:23.524 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:41.216746+0000", 2024-09-18T08:11:23.524 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.025222364000000001 2024-09-18T08:11:23.524 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:11:23.524 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:11:23.524 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "commit_sent", 2024-09-18T08:11:23.524 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:41.216782+0000", 2024-09-18T08:11:23.524 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 3.6069999999999999e-05 2024-09-18T08:11:23.524 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:11:23.524 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:11:23.524 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-09-18T08:11:23.524 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:41.216828+0000", 2024-09-18T08:11:23.525 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4.5543000000000001e-05 2024-09-18T08:11:23.525 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-18T08:11:23.525 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-09-18T08:11:23.525 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-18T08:11:23.525 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:11:23.525 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:11:23.525 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.14805.0:63960 7.c 7:30cc2060:::benchmark_data_smithi007_73155_object63959:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e100)", 2024-09-18T08:11:23.525 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-09-18T08:04:41.252452+0000", 2024-09-18T08:11:23.525 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 402.16269431900002, 2024-09-18T08:11:23.525 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.027634619999999999, 2024-09-18T08:11:23.525 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-09-18T08:11:23.525 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-09-18T08:11:23.525 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T08:11:23.526 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2024-09-18T08:11:23.526 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.14805", 2024-09-18T08:11:23.526 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "172.21.15.7:0/640320436", 2024-09-18T08:11:23.526 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 63960 2024-09-18T08:11:23.526 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:11:23.526 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-09-18T08:11:23.526 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:11:23.526 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-09-18T08:11:23.526 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:41.252452+0000", 2024-09-18T08:11:23.526 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-18T08:11:23.526 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:11:23.526 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:11:23.526 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-09-18T08:11:23.526 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:41.252452+0000", 2024-09-18T08:11:23.527 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-18T08:11:23.527 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:11:23.527 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:11:23.527 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-09-18T08:11:23.527 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:41.252453+0000", 2024-09-18T08:11:23.527 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.063e-06 2024-09-18T08:11:23.527 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:11:23.527 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:11:23.527 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:11:23 smithi007 ceph-mon[39249]: 3.c scrub starts 2024-09-18T08:11:23.528 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:11:23 smithi007 ceph-mon[39249]: 3.c scrub ok 2024-09-18T08:11:23.528 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:11:23 smithi007 ceph-mon[39249]: 4.11 scrub starts 2024-09-18T08:11:23.528 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:11:23 smithi007 ceph-mon[39249]: 4.11 scrub ok 2024-09-18T08:11:23.528 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:11:23 smithi007 ceph-mon[39249]: 5.d scrub starts 2024-09-18T08:11:23.528 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:11:23 smithi007 ceph-mon[39249]: 5.d scrub ok 2024-09-18T08:11:23.528 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:11:23 smithi050 ceph-mon[32371]: 3.c scrub starts 2024-09-18T08:11:23.528 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:11:23 smithi050 ceph-mon[32371]: 3.c scrub ok 2024-09-18T08:11:23.528 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:11:23 smithi050 ceph-mon[32371]: 4.11 scrub starts 2024-09-18T08:11:23.528 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:11:23 smithi050 ceph-mon[32371]: 4.11 scrub ok 2024-09-18T08:11:23.528 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:11:23 smithi050 ceph-mon[32371]: 5.d scrub starts 2024-09-18T08:11:23.528 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:11:23 smithi050 ceph-mon[32371]: 5.d scrub ok 2024-09-18T08:11:23.529 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-09-18T08:11:23.529 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:41.252503+0000", 2024-09-18T08:11:23.529 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 5.024e-05 2024-09-18T08:11:23.529 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:11:23.529 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:11:23.529 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-09-18T08:11:23.529 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:41.252513+0000", 2024-09-18T08:11:23.529 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.0137e-05 2024-09-18T08:11:23.529 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:11:23.529 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:11:23.530 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-09-18T08:11:23.532 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:41.252519+0000", 2024-09-18T08:11:23.532 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 5.2630000000000003e-06 2024-09-18T08:11:23.532 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:11:23.532 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:11:23.532 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-18T08:11:23.533 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:41.252564+0000", 2024-09-18T08:11:23.533 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4.5528999999999997e-05 2024-09-18T08:11:23.533 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:11:23.533 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:11:23.533 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-09-18T08:11:23.533 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:41.252645+0000", 2024-09-18T08:11:23.533 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 8.1118000000000005e-05 2024-09-18T08:11:23.533 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:11:23.533 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:11:23.533 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "waiting for subops from 3,7", 2024-09-18T08:11:23.533 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:41.252726+0000", 2024-09-18T08:11:23.533 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 8.0912000000000001e-05 2024-09-18T08:11:23.533 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:11:23.534 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:11:23.534 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "op_commit", 2024-09-18T08:11:23.534 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:41.254200+0000", 2024-09-18T08:11:23.534 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0014738010000000001 2024-09-18T08:11:23.534 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:11:23.534 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:11:23.534 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:11:23.534 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:41.254525+0000", 2024-09-18T08:11:23.534 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00032529599999999998 2024-09-18T08:11:23.534 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:11:23.534 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:11:23.534 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:11:23.534 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:41.280019+0000", 2024-09-18T08:11:23.534 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.025493512999999999 2024-09-18T08:11:23.535 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:11:23.535 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:11:23.535 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "commit_sent", 2024-09-18T08:11:23.535 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:41.280057+0000", 2024-09-18T08:11:23.535 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 3.8287000000000001e-05 2024-09-18T08:11:23.535 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:11:23.535 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:11:23.535 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-09-18T08:11:23.535 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:41.280086+0000", 2024-09-18T08:11:23.535 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 2.9461000000000001e-05 2024-09-18T08:11:23.535 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-18T08:11:23.535 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-09-18T08:11:23.535 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-18T08:11:23.536 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:11:23.536 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:11:23.536 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.14805.0:65051 7.d 7:bcc5331c:::benchmark_data_smithi007_73155_object65050:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e100)", 2024-09-18T08:11:23.536 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-09-18T08:04:42.227551+0000", 2024-09-18T08:11:23.536 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 401.187595558, 2024-09-18T08:11:23.536 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.057084449000000002, 2024-09-18T08:11:23.536 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-09-18T08:11:23.536 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-09-18T08:11:23.536 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "waiting for sub ops", 2024-09-18T08:11:23.536 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2024-09-18T08:11:23.536 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.14805", 2024-09-18T08:11:23.536 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "172.21.15.7:0/640320436", 2024-09-18T08:11:23.537 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 65051 2024-09-18T08:11:23.537 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:11:23.537 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-09-18T08:11:23.537 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:11:23.537 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-09-18T08:11:23.537 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:42.227551+0000", 2024-09-18T08:11:23.537 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-18T08:11:23.537 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:11:23.537 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:11:23.537 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-09-18T08:11:23.537 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:42.227551+0000", 2024-09-18T08:11:23.537 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-18T08:11:23.538 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:11:23.538 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:11:23.538 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-09-18T08:11:23.538 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:42.227552+0000", 2024-09-18T08:11:23.538 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.5319999999999999e-06 2024-09-18T08:11:23.538 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:11:23.538 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:11:23.538 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-09-18T08:11:23.538 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:42.227606+0000", 2024-09-18T08:11:23.538 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 5.4290999999999999e-05 2024-09-18T08:11:23.538 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:11:23.538 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:11:23.538 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-09-18T08:11:23.539 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:42.227619+0000", 2024-09-18T08:11:23.539 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.2937e-05 2024-09-18T08:11:23.539 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:11:23.539 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:11:23.539 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-09-18T08:11:23.539 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:42.227627+0000", 2024-09-18T08:11:23.539 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 7.9340000000000006e-06 2024-09-18T08:11:23.539 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:11:23.539 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:11:23.539 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-18T08:11:23.539 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:42.227681+0000", 2024-09-18T08:11:23.539 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 5.3771999999999999e-05 2024-09-18T08:11:23.539 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:11:23.539 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:11:23.540 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-09-18T08:11:23.540 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:42.227809+0000", 2024-09-18T08:11:23.540 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00012750399999999999 2024-09-18T08:11:23.540 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:11:23.540 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:11:23.540 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "waiting for subops from 0,1", 2024-09-18T08:11:23.540 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:42.227961+0000", 2024-09-18T08:11:23.540 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00015280599999999999 2024-09-18T08:11:23.540 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:11:23.540 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:11:23.540 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "op_commit", 2024-09-18T08:11:23.540 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:42.229971+0000", 2024-09-18T08:11:23.540 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0020099179999999999 2024-09-18T08:11:23.540 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:11:23.541 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:11:23.541 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:11:23.541 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:42.232646+0000", 2024-09-18T08:11:23.541 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0026748359999999999 2024-09-18T08:11:23.541 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:11:23.541 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:11:23.541 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-09-18T08:11:23.541 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:42.284635+0000", 2024-09-18T08:11:23.541 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.051988919000000001 2024-09-18T08:11:23.541 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-18T08:11:23.541 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-09-18T08:11:23.541 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-18T08:11:23.541 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:11:23.541 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:11:23.542 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.14805.0:65057 7.d 7:bffbe46c:::benchmark_data_smithi007_73155_object65056:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e100)", 2024-09-18T08:11:23.542 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-09-18T08:04:42.229929+0000", 2024-09-18T08:11:23.542 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 401.185216862, 2024-09-18T08:11:23.542 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.054704183000000003, 2024-09-18T08:11:23.542 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-09-18T08:11:23.542 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-09-18T08:11:23.542 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "waiting for sub ops", 2024-09-18T08:11:23.542 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2024-09-18T08:11:23.542 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.14805", 2024-09-18T08:11:23.542 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "172.21.15.7:0/640320436", 2024-09-18T08:11:23.542 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 65057 2024-09-18T08:11:23.542 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:11:23.542 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-09-18T08:11:23.542 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:11:23.543 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-09-18T08:11:23.543 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:42.229929+0000", 2024-09-18T08:11:23.543 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-18T08:11:23.543 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:11:23.543 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:11:23.543 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-09-18T08:11:23.543 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:42.229929+0000", 2024-09-18T08:11:23.543 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-18T08:11:23.543 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:11:23.543 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:11:23.543 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-09-18T08:11:23.543 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:42.229930+0000", 2024-09-18T08:11:23.543 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.0219999999999999e-06 2024-09-18T08:11:23.544 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:11:23.544 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:11:23.544 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-09-18T08:11:23.544 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:42.229973+0000", 2024-09-18T08:11:23.544 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4.2904000000000001e-05 2024-09-18T08:11:23.544 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:11:23.544 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:11:23.544 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-09-18T08:11:23.544 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:42.229987+0000", 2024-09-18T08:11:23.544 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.3405e-05 2024-09-18T08:11:23.544 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:11:23.544 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:11:23.544 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-09-18T08:11:23.545 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:42.229996+0000", 2024-09-18T08:11:23.545 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 9.6299999999999993e-06 2024-09-18T08:11:23.545 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:11:23.545 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:11:23.545 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-18T08:11:23.545 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:42.230048+0000", 2024-09-18T08:11:23.545 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 5.1839000000000003e-05 2024-09-18T08:11:23.545 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:11:23.545 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:11:23.545 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-09-18T08:11:23.545 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:42.230146+0000", 2024-09-18T08:11:23.545 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 9.8101000000000001e-05 2024-09-18T08:11:23.546 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:11:23.546 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:11:23.546 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "waiting for subops from 0,1", 2024-09-18T08:11:23.546 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:42.230237+0000", 2024-09-18T08:11:23.546 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 9.0634e-05 2024-09-18T08:11:23.546 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:11:23.546 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:11:23.546 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "op_commit", 2024-09-18T08:11:23.546 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:42.231752+0000", 2024-09-18T08:11:23.546 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.001515548 2024-09-18T08:11:23.546 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:11:23.546 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:11:23.546 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:11:23.547 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:42.232681+0000", 2024-09-18T08:11:23.547 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00092884 2024-09-18T08:11:23.547 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:11:23.547 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:11:23.547 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-09-18T08:11:23.547 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:42.284634+0000", 2024-09-18T08:11:23.547 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.05195226 2024-09-18T08:11:23.547 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-18T08:11:23.547 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-09-18T08:11:23.547 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-18T08:11:23.547 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:11:23.547 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:11:23.548 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.14805.0:65062 7.d 7:b1e686a2:::benchmark_data_smithi007_73155_object65061:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e100)", 2024-09-18T08:11:23.548 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-09-18T08:04:42.236979+0000", 2024-09-18T08:11:23.548 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 401.17816754699999, 2024-09-18T08:11:23.548 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.047653305999999999, 2024-09-18T08:11:23.548 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-09-18T08:11:23.548 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-09-18T08:11:23.548 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "waiting for sub ops", 2024-09-18T08:11:23.548 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2024-09-18T08:11:23.548 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.14805", 2024-09-18T08:11:23.548 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "172.21.15.7:0/640320436", 2024-09-18T08:11:23.548 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 65062 2024-09-18T08:11:23.548 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:11:23.549 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-09-18T08:11:23.549 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:11:23.549 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-09-18T08:11:23.549 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:42.236979+0000", 2024-09-18T08:11:23.549 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-18T08:11:23.549 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:11:23.549 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:11:23.549 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-09-18T08:11:23.549 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:42.236979+0000", 2024-09-18T08:11:23.549 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-18T08:11:23.549 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:11:23.549 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:11:23.549 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-09-18T08:11:23.550 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:42.236980+0000", 2024-09-18T08:11:23.550 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.124e-06 2024-09-18T08:11:23.550 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:11:23.550 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:11:23.550 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-09-18T08:11:23.550 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:42.237029+0000", 2024-09-18T08:11:23.550 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4.9450999999999998e-05 2024-09-18T08:11:23.550 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:11:23.550 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:11:23.550 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-09-18T08:11:23.550 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:42.237041+0000", 2024-09-18T08:11:23.550 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.1418e-05 2024-09-18T08:11:23.550 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:11:23.551 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:11:23.551 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-09-18T08:11:23.551 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:42.237049+0000", 2024-09-18T08:11:23.551 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 8.0059999999999992e-06 2024-09-18T08:11:23.551 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:11:23.551 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:11:23.551 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-18T08:11:23.551 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:42.237104+0000", 2024-09-18T08:11:23.551 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 5.4870999999999997e-05 2024-09-18T08:11:23.551 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:11:23.551 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:11:23.551 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-09-18T08:11:23.552 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:42.237258+0000", 2024-09-18T08:11:23.552 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000154857 2024-09-18T08:11:23.552 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:11:23.552 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:11:23.552 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "waiting for subops from 0,1", 2024-09-18T08:11:23.552 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:42.237413+0000", 2024-09-18T08:11:23.552 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000154764 2024-09-18T08:11:23.552 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:11:23.552 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:11:23.552 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "op_commit", 2024-09-18T08:11:23.552 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:42.240054+0000", 2024-09-18T08:11:23.552 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0026404549999999999 2024-09-18T08:11:23.552 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:11:23.552 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:11:23.553 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-09-18T08:11:23.553 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:42.284632+0000", 2024-09-18T08:11:23.553 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.044578359999999997 2024-09-18T08:11:23.553 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-18T08:11:23.553 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-09-18T08:11:23.553 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-18T08:11:23.553 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:11:23.553 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:11:23.553 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.14805.0:65063 7.d 7:b037f784:::benchmark_data_smithi007_73155_object65062:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e100)", 2024-09-18T08:11:23.553 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-09-18T08:04:42.237116+0000", 2024-09-18T08:11:23.553 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 401.17803008099997, 2024-09-18T08:11:23.553 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.047512190000000003, 2024-09-18T08:11:23.553 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-09-18T08:11:23.554 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-09-18T08:11:23.554 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "waiting for sub ops", 2024-09-18T08:11:23.554 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2024-09-18T08:11:23.554 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.14805", 2024-09-18T08:11:23.554 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "172.21.15.7:0/640320436", 2024-09-18T08:11:23.554 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 65063 2024-09-18T08:11:23.554 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:11:23.554 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-09-18T08:11:23.554 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:11:23.554 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-09-18T08:11:23.554 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:42.237116+0000", 2024-09-18T08:11:23.554 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-18T08:11:23.554 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:11:23.554 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:11:23.555 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-09-18T08:11:23.555 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:42.237116+0000", 2024-09-18T08:11:23.555 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-18T08:11:23.555 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:11:23.555 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:11:23.555 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-09-18T08:11:23.555 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:42.237117+0000", 2024-09-18T08:11:23.555 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 6.7000000000000004e-07 2024-09-18T08:11:23.555 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:11:23.555 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:11:23.555 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-09-18T08:11:23.555 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:42.237190+0000", 2024-09-18T08:11:23.555 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 7.2972000000000005e-05 2024-09-18T08:11:23.555 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:11:23.556 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:11:23.556 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-09-18T08:11:23.556 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:42.237197+0000", 2024-09-18T08:11:23.556 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 7.2729999999999997e-06 2024-09-18T08:11:23.556 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:11:23.556 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:11:23.556 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-09-18T08:11:23.556 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:42.237203+0000", 2024-09-18T08:11:23.556 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 5.5289999999999999e-06 2024-09-18T08:11:23.556 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:11:23.556 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:11:23.556 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-18T08:11:23.556 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:42.238960+0000", 2024-09-18T08:11:23.557 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.001757759 2024-09-18T08:11:23.557 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:11:23.557 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:11:23.557 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-09-18T08:11:23.557 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:42.239038+0000", 2024-09-18T08:11:23.557 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 7.8054999999999997e-05 2024-09-18T08:11:23.557 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:11:23.557 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:11:23.557 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "waiting for subops from 0,1", 2024-09-18T08:11:23.557 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:42.239122+0000", 2024-09-18T08:11:23.557 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 8.3963999999999998e-05 2024-09-18T08:11:23.557 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:11:23.557 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:11:23.557 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "op_commit", 2024-09-18T08:11:23.558 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:42.240157+0000", 2024-09-18T08:11:23.558 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.001034372 2024-09-18T08:11:23.558 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:11:23.558 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:11:23.558 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-09-18T08:11:23.558 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:42.284628+0000", 2024-09-18T08:11:23.558 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.044471596000000002 2024-09-18T08:11:23.558 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-18T08:11:23.558 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-09-18T08:11:23.558 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-18T08:11:23.558 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-18T08:11:23.558 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-09-18T08:11:23.558 INFO:teuthology.orchestra.run.smithi050.stdout:} 2024-09-18T08:11:23.568 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6 2024-09-18T08:11:23.603 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:11:23.603 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:11:23.604 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6' 2024-09-18T08:11:23.705 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6 2024-09-18T08:11:23.740 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:11:23.741 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:11:23.741 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6' 2024-09-18T08:11:23.770 INFO:teuthology.orchestra.run.smithi007.stderr:no valid command found; 10 closest matches: 2024-09-18T08:11:23.770 INFO:teuthology.orchestra.run.smithi007.stderr:0 2024-09-18T08:11:23.771 INFO:teuthology.orchestra.run.smithi007.stderr:1 2024-09-18T08:11:23.771 INFO:teuthology.orchestra.run.smithi007.stderr:2 2024-09-18T08:11:23.771 INFO:teuthology.orchestra.run.smithi007.stderr:abort 2024-09-18T08:11:23.771 INFO:teuthology.orchestra.run.smithi007.stderr:assert 2024-09-18T08:11:23.771 INFO:teuthology.orchestra.run.smithi007.stderr:bench [] [] [] [] 2024-09-18T08:11:23.772 INFO:teuthology.orchestra.run.smithi007.stderr:bluefs debug_inject_read_zeros 2024-09-18T08:11:23.772 INFO:teuthology.orchestra.run.smithi007.stderr:bluefs files list 2024-09-18T08:11:23.772 INFO:teuthology.orchestra.run.smithi007.stderr:bluefs stats 2024-09-18T08:11:23.772 INFO:teuthology.orchestra.run.smithi007.stderr:bluestore allocator dump block 2024-09-18T08:11:23.772 INFO:teuthology.orchestra.run.smithi007.stderr:Error EINVAL: invalid command 2024-09-18T08:11:23.842 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7 2024-09-18T08:11:23.870 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:11:23.871 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:11:23.872 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7' 2024-09-18T08:11:23.972 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0 2024-09-18T08:11:24.009 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:11:24.010 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:11:24.010 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0' 2024-09-18T08:11:24.078 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid 779349ca-7593-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-18T08:11:24.111 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5 2024-09-18T08:11:24.145 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:11:24.148 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:11:24.149 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5' 2024-09-18T08:11:24.184 INFO:tasks.radosbench.radosbench.0.smithi007.stdout: 9 16 23464 23448 162.786 137.875 0.00768825 0.00613791 2024-09-18T08:11:24.250 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4 2024-09-18T08:11:24.285 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:11:24.286 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:11:24.286 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4' 2024-09-18T08:11:24.359 INFO:teuthology.orchestra.run.smithi007.stderr:nodeep-scrub is unset 2024-09-18T08:11:24.383 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:11:24 smithi007 ceph-mon[29459]: 4.13 scrub starts 2024-09-18T08:11:24.383 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:11:24 smithi007 ceph-mon[29459]: 4.13 scrub ok 2024-09-18T08:11:24.383 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:11:24 smithi007 ceph-mon[29459]: 6.a scrub starts 2024-09-18T08:11:24.383 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:11:24 smithi007 ceph-mon[29459]: 6.a scrub ok 2024-09-18T08:11:24.384 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:11:24 smithi007 ceph-mon[29459]: pgmap v411: 158 pgs: 158 active+clean; 773 MiB data, 4.9 GiB used, 710 GiB / 715 GiB avail; 639 B/s rd, 96 MiB/s wr, 1.54k op/s 2024-09-18T08:11:24.384 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:11:24 smithi007 ceph-mon[29459]: from='client.? 172.21.15.7:0/1867345373' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-18T08:11:24.384 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:11:24 smithi007 ceph-mon[29459]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-18T08:11:24.384 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:11:24 smithi007 ceph-mon[39249]: 4.13 scrub starts 2024-09-18T08:11:24.384 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:11:24 smithi007 ceph-mon[39249]: 4.13 scrub ok 2024-09-18T08:11:24.384 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:11:24 smithi007 ceph-mon[39249]: 6.a scrub starts 2024-09-18T08:11:24.384 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:11:24 smithi007 ceph-mon[39249]: 6.a scrub ok 2024-09-18T08:11:24.384 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:11:24 smithi007 ceph-mon[39249]: pgmap v411: 158 pgs: 158 active+clean; 773 MiB data, 4.9 GiB used, 710 GiB / 715 GiB avail; 639 B/s rd, 96 MiB/s wr, 1.54k op/s 2024-09-18T08:11:24.384 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:11:24 smithi007 ceph-mon[39249]: from='client.? 172.21.15.7:0/1867345373' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-18T08:11:24.384 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:11:24 smithi007 ceph-mon[39249]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-18T08:11:24.387 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1 2024-09-18T08:11:24.428 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:11:24.429 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:11:24.429 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1' 2024-09-18T08:11:24.510 DEBUG:teuthology.orchestra.run.smithi050:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid 779349ca-7593-11ef-bceb-c7b262605968 -- ceph daemon osd.7 dump_ops_in_flight 2024-09-18T08:11:24.530 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6 2024-09-18T08:11:24.593 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:11:24.594 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:11:24.594 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6' 2024-09-18T08:11:24.656 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:11:24 smithi050 ceph-mon[32371]: 4.13 scrub starts 2024-09-18T08:11:24.656 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:11:24 smithi050 ceph-mon[32371]: 4.13 scrub ok 2024-09-18T08:11:24.656 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:11:24 smithi050 ceph-mon[32371]: 6.a scrub starts 2024-09-18T08:11:24.656 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:11:24 smithi050 ceph-mon[32371]: 6.a scrub ok 2024-09-18T08:11:24.656 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:11:24 smithi050 ceph-mon[32371]: pgmap v411: 158 pgs: 158 active+clean; 773 MiB data, 4.9 GiB used, 710 GiB / 715 GiB avail; 639 B/s rd, 96 MiB/s wr, 1.54k op/s 2024-09-18T08:11:24.656 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:11:24 smithi050 ceph-mon[32371]: from='client.? 172.21.15.7:0/1867345373' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-18T08:11:24.656 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:11:24 smithi050 ceph-mon[32371]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-18T08:11:24.695 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4 2024-09-18T08:11:24.735 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:11:24.736 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:11:24.736 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4' 2024-09-18T08:11:24.837 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6 2024-09-18T08:11:24.878 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:11:24.878 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:11:24.879 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6' 2024-09-18T08:11:24.948 INFO:teuthology.orchestra.run.smithi050.stderr:Inferring config /var/lib/ceph/779349ca-7593-11ef-bceb-c7b262605968/mon.b/config 2024-09-18T08:11:24.980 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5 2024-09-18T08:11:25.021 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:11:25.021 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:11:25.022 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5' 2024-09-18T08:11:25.123 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2 2024-09-18T08:11:25.159 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:11:25.160 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:11:25.160 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2' 2024-09-18T08:11:25.184 INFO:tasks.radosbench.radosbench.0.smithi007.stdout: 10 16 25743 25727 160.75 142.438 0.00615807 0.0062152 2024-09-18T08:11:25.261 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4 2024-09-18T08:11:25.297 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:11:25.298 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:11:25.298 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4' 2024-09-18T08:11:25.399 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1 2024-09-18T08:11:25.451 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:11:25.452 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:11:25.452 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1' 2024-09-18T08:11:25.553 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7 2024-09-18T08:11:25.589 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:11:25.589 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:11:25.590 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7' 2024-09-18T08:11:25.655 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:11:25 smithi050 ceph-mon[32371]: 4.2 scrub starts 2024-09-18T08:11:25.656 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:11:25 smithi050 ceph-mon[32371]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-18T08:11:25.656 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:11:25 smithi050 ceph-mon[32371]: Cluster is now healthy 2024-09-18T08:11:25.656 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:11:25 smithi050 ceph-mon[32371]: 4.2 scrub ok 2024-09-18T08:11:25.656 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:11:25 smithi050 ceph-mon[32371]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-18T08:11:25.656 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:11:25 smithi050 ceph-mon[32371]: osdmap e239: 8 total, 8 up, 8 in 2024-09-18T08:11:25.690 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5 2024-09-18T08:11:25.717 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:11:25.718 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:11:25.718 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5' 2024-09-18T08:11:25.730 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:11:25 smithi007 ceph-mon[29459]: 4.2 scrub starts 2024-09-18T08:11:25.731 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:11:25 smithi007 ceph-mon[29459]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-18T08:11:25.731 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:11:25 smithi007 ceph-mon[29459]: Cluster is now healthy 2024-09-18T08:11:25.731 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:11:25 smithi007 ceph-mon[29459]: 4.2 scrub ok 2024-09-18T08:11:25.731 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:11:25 smithi007 ceph-mon[29459]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-18T08:11:25.731 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:11:25 smithi007 ceph-mon[29459]: osdmap e239: 8 total, 8 up, 8 in 2024-09-18T08:11:25.731 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:11:25 smithi007 ceph-mon[39249]: 4.2 scrub starts 2024-09-18T08:11:25.731 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:11:25 smithi007 ceph-mon[39249]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-18T08:11:25.731 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:11:25 smithi007 ceph-mon[39249]: Cluster is now healthy 2024-09-18T08:11:25.731 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:11:25 smithi007 ceph-mon[39249]: 4.2 scrub ok 2024-09-18T08:11:25.731 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:11:25 smithi007 ceph-mon[39249]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-18T08:11:25.732 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:11:25 smithi007 ceph-mon[39249]: osdmap e239: 8 total, 8 up, 8 in 2024-09-18T08:11:25.811 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2024-09-18T08:11:25.812 INFO:teuthology.orchestra.run.smithi050.stdout: "ops": [ 2024-09-18T08:11:25.812 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:11:25.812 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.15582.0:26974 9.d 9:b21990f0:::benchmark_data_smithi007_124019_object26973:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536 in=65536b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e239)", 2024-09-18T08:11:25.812 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-09-18T08:11:25.804061+0000", 2024-09-18T08:11:25.812 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 0.004223292, 2024-09-18T08:11:25.812 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0042546010000000002, 2024-09-18T08:11:25.812 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-09-18T08:11:25.812 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-09-18T08:11:25.812 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "waiting for sub ops", 2024-09-18T08:11:25.812 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2024-09-18T08:11:25.813 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.15582", 2024-09-18T08:11:25.813 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "172.21.15.7:0/3598605839", 2024-09-18T08:11:25.813 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 26974 2024-09-18T08:11:25.813 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:11:25.813 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-09-18T08:11:25.813 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:11:25.813 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-09-18T08:11:25.813 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:11:25.804061+0000", 2024-09-18T08:11:25.814 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-18T08:11:25.814 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:11:25.814 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:11:25.814 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-09-18T08:11:25.814 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:11:25.804061+0000", 2024-09-18T08:11:25.814 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-18T08:11:25.814 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:11:25.814 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:11:25.814 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-09-18T08:11:25.814 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:11:25.804062+0000", 2024-09-18T08:11:25.814 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.1689999999999999e-06 2024-09-18T08:11:25.814 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:11:25.814 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:11:25.814 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-09-18T08:11:25.815 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:11:25.804097+0000", 2024-09-18T08:11:25.815 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 3.5479999999999999e-05 2024-09-18T08:11:25.815 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:11:25.815 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:11:25.815 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-09-18T08:11:25.815 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:11:25.804106+0000", 2024-09-18T08:11:25.815 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 8.5080000000000005e-06 2024-09-18T08:11:25.815 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:11:25.815 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:11:25.815 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-09-18T08:11:25.815 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:11:25.804112+0000", 2024-09-18T08:11:25.815 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 6.2450000000000003e-06 2024-09-18T08:11:25.816 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:11:25.816 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:11:25.816 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-18T08:11:25.816 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:11:25.804171+0000", 2024-09-18T08:11:25.816 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 5.8940000000000002e-05 2024-09-18T08:11:25.816 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:11:25.816 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:11:25.816 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-09-18T08:11:25.816 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:11:25.804287+0000", 2024-09-18T08:11:25.816 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000116364 2024-09-18T08:11:25.816 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:11:25.816 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:11:25.816 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "waiting for subops from 1,5", 2024-09-18T08:11:25.817 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:11:25.804398+0000", 2024-09-18T08:11:25.817 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000110519 2024-09-18T08:11:25.817 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:11:25.817 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:11:25.817 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "op_commit", 2024-09-18T08:11:25.817 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:11:25.805626+0000", 2024-09-18T08:11:25.817 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0012284170000000001 2024-09-18T08:11:25.817 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:11:25.817 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:11:25.817 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:11:25.817 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:11:25.808261+0000", 2024-09-18T08:11:25.817 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.002635105 2024-09-18T08:11:25.818 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-18T08:11:25.818 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-09-18T08:11:25.818 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-18T08:11:25.818 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:11:25.818 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:11:25.818 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_repop(client.15582.0:26972 9.1 e239/232 9:86af81b5:::benchmark_data_smithi007_124019_object26971:head v 239'1698, mlcod=239'1696)", 2024-09-18T08:11:25.818 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-09-18T08:11:25.805041+0000", 2024-09-18T08:11:25.818 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 0.0032425589999999999, 2024-09-18T08:11:25.818 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0033579769999999998, 2024-09-18T08:11:25.818 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-09-18T08:11:25.819 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-09-18T08:11:25.819 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "started", 2024-09-18T08:11:25.819 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-09-18T08:11:25.819 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:11:25.819 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-09-18T08:11:25.819 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:11:25.805041+0000", 2024-09-18T08:11:25.819 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-18T08:11:25.819 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:11:25.819 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:11:25.819 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-09-18T08:11:25.820 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:11:25.805041+0000", 2024-09-18T08:11:25.820 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-18T08:11:25.820 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:11:25.820 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:11:25.820 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-09-18T08:11:25.820 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:11:25.805042+0000", 2024-09-18T08:11:25.820 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 3.6300000000000001e-07 2024-09-18T08:11:25.820 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:11:25.820 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:11:25.820 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-09-18T08:11:25.820 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:11:25.805068+0000", 2024-09-18T08:11:25.820 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 2.6106e-05 2024-09-18T08:11:25.821 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:11:25.821 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:11:25.821 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-09-18T08:11:25.821 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:11:25.805074+0000", 2024-09-18T08:11:25.821 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 5.8499999999999999e-06 2024-09-18T08:11:25.821 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:11:25.821 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:11:25.821 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-09-18T08:11:25.821 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:11:25.805078+0000", 2024-09-18T08:11:25.821 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4.5530000000000003e-06 2024-09-18T08:11:25.821 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:11:25.822 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:11:25.822 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-18T08:11:25.822 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:11:25.805425+0000", 2024-09-18T08:11:25.822 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00034729900000000002 2024-09-18T08:11:25.822 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:11:25.822 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:11:25.822 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-09-18T08:11:25.822 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:11:25.805454+0000", 2024-09-18T08:11:25.822 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 2.8507e-05 2024-09-18T08:11:25.822 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-18T08:11:25.822 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-09-18T08:11:25.822 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-18T08:11:25.822 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:11:25.822 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:11:25.823 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_repop(client.15582.0:26973 9.1 e239/232 9:890871c1:::benchmark_data_smithi007_124019_object26972:head v 239'1699, mlcod=239'1696)", 2024-09-18T08:11:25.823 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-09-18T08:11:25.806410+0000", 2024-09-18T08:11:25.823 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 0.0018733910000000001, 2024-09-18T08:11:25.823 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0020593500000000002, 2024-09-18T08:11:25.823 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-09-18T08:11:25.823 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-09-18T08:11:25.823 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "started", 2024-09-18T08:11:25.823 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-09-18T08:11:25.823 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:11:25.823 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-09-18T08:11:25.823 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:11:25.806410+0000", 2024-09-18T08:11:25.823 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-18T08:11:25.823 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:11:25.824 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:11:25.824 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-09-18T08:11:25.824 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:11:25.806410+0000", 2024-09-18T08:11:25.824 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-18T08:11:25.824 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:11:25.824 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:11:25.824 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-09-18T08:11:25.824 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:11:25.806411+0000", 2024-09-18T08:11:25.824 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 8.5700000000000001e-07 2024-09-18T08:11:25.824 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:11:25.824 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:11:25.824 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-09-18T08:11:25.825 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:11:25.806441+0000", 2024-09-18T08:11:25.825 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 2.9612e-05 2024-09-18T08:11:25.825 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:11:25.825 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:11:25.825 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-09-18T08:11:25.825 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:11:25.806451+0000", 2024-09-18T08:11:25.825 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 9.628e-06 2024-09-18T08:11:25.825 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:11:25.825 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:11:25.825 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-09-18T08:11:25.825 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:11:25.806457+0000", 2024-09-18T08:11:25.825 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 6.2910000000000004e-06 2024-09-18T08:11:25.825 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:11:25.826 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:11:25.826 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-18T08:11:25.826 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:11:25.807982+0000", 2024-09-18T08:11:25.826 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.001525261 2024-09-18T08:11:25.826 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:11:25.826 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:11:25.826 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-09-18T08:11:25.826 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:11:25.808001+0000", 2024-09-18T08:11:25.826 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.9089e-05 2024-09-18T08:11:25.826 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-18T08:11:25.826 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-09-18T08:11:25.826 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-18T08:11:25.826 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:11:25.826 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:11:25.827 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_repop(client.15582.0:26981 9.7 e239/232 9:e4b797d6:::benchmark_data_smithi007_124019_object26980:head v 239'1671, mlcod=239'1669)", 2024-09-18T08:11:25.827 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-09-18T08:11:25.806904+0000", 2024-09-18T08:11:25.827 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 0.0013799870000000001, 2024-09-18T08:11:25.827 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.001615867, 2024-09-18T08:11:25.827 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-09-18T08:11:25.827 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-09-18T08:11:25.827 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "started", 2024-09-18T08:11:25.827 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-09-18T08:11:25.827 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:11:25.827 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-09-18T08:11:25.828 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:11:25.806904+0000", 2024-09-18T08:11:25.828 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-18T08:11:25.828 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:11:25.828 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:11:25.828 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-09-18T08:11:25.828 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:11:25.806904+0000", 2024-09-18T08:11:25.828 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-18T08:11:25.828 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:11:25.828 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:11:25.828 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-09-18T08:11:25.828 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:11:25.806905+0000", 2024-09-18T08:11:25.828 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 8.3900000000000004e-07 2024-09-18T08:11:25.829 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:11:25.829 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:11:25.829 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-09-18T08:11:25.829 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:11:25.806926+0000", 2024-09-18T08:11:25.829 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 2.1571000000000001e-05 2024-09-18T08:11:25.829 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:11:25.829 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:11:25.829 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-09-18T08:11:25.829 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:11:25.806951+0000", 2024-09-18T08:11:25.829 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 2.5057000000000001e-05 2024-09-18T08:11:25.829 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:11:25.830 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:11:25.830 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-09-18T08:11:25.830 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:11:25.806958+0000", 2024-09-18T08:11:25.830 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 6.793e-06 2024-09-18T08:11:25.830 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:11:25.830 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:11:25.830 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-18T08:11:25.830 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:11:25.807033+0000", 2024-09-18T08:11:25.830 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 7.4901999999999995e-05 2024-09-18T08:11:25.830 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:11:25.830 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:11:25.830 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-09-18T08:11:25.831 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:11:25.807052+0000", 2024-09-18T08:11:25.831 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.8566000000000001e-05 2024-09-18T08:11:25.831 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-18T08:11:25.831 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-09-18T08:11:25.831 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-18T08:11:25.831 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:11:25.831 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:11:25.831 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_repop(client.15582.0:26975 9.5 e239/232 9:a6ee24c1:::benchmark_data_smithi007_124019_object26974:head v 239'1723, mlcod=239'1720)", 2024-09-18T08:11:25.831 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-09-18T08:11:25.807806+0000", 2024-09-18T08:11:25.831 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 0.00047823800000000002, 2024-09-18T08:11:25.831 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00076555500000000001, 2024-09-18T08:11:25.831 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-09-18T08:11:25.831 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-09-18T08:11:25.832 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "started", 2024-09-18T08:11:25.832 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-09-18T08:11:25.832 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:11:25.832 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-09-18T08:11:25.832 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:11:25.807806+0000", 2024-09-18T08:11:25.832 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-18T08:11:25.832 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:11:25.832 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:11:25.832 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-09-18T08:11:25.832 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:11:25.807806+0000", 2024-09-18T08:11:25.832 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-18T08:11:25.833 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:11:25.833 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:11:25.833 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-09-18T08:11:25.833 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:11:25.807806+0000", 2024-09-18T08:11:25.833 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 6.1200000000000003e-07 2024-09-18T08:11:25.833 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:11:25.834 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:11:25.834 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-09-18T08:11:25.834 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:11:25.807841+0000", 2024-09-18T08:11:25.834 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 3.4518999999999999e-05 2024-09-18T08:11:25.834 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:11:25.834 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:11:25.834 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-09-18T08:11:25.835 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:11:25.807855+0000", 2024-09-18T08:11:25.835 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.4572999999999999e-05 2024-09-18T08:11:25.835 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:11:25.835 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:11:25.835 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-09-18T08:11:25.835 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:11:25.807862+0000", 2024-09-18T08:11:25.835 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 7.0940000000000002e-06 2024-09-18T08:11:25.835 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:11:25.835 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:11:25.835 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-18T08:11:25.836 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:11:25.807942+0000", 2024-09-18T08:11:25.836 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 7.9942000000000001e-05 2024-09-18T08:11:25.836 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:11:25.836 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:11:25.836 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-09-18T08:11:25.836 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:11:25.807960+0000", 2024-09-18T08:11:25.836 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.8099000000000001e-05 2024-09-18T08:11:25.836 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-18T08:11:25.836 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-09-18T08:11:25.836 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-18T08:11:25.836 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:11:25.836 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:11:25.837 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_repop_reply(client.15582.0:26974 9.d e239/232)", 2024-09-18T08:11:25.837 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-09-18T08:11:25.808569+0000", 2024-09-18T08:11:25.837 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 4294967295.9997149, 2024-09-18T08:11:25.837 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 5.2601e-05, 2024-09-18T08:11:25.837 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-09-18T08:11:25.837 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-09-18T08:11:25.837 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "queued for pg", 2024-09-18T08:11:25.837 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-09-18T08:11:25.837 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:11:25.837 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-09-18T08:11:25.837 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:11:25.808569+0000", 2024-09-18T08:11:25.837 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-18T08:11:25.838 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:11:25.838 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:11:25.838 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-09-18T08:11:25.838 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:11:25.808569+0000", 2024-09-18T08:11:25.838 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-18T08:11:25.838 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:11:25.838 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:11:25.838 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-09-18T08:11:25.838 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:11:25.808570+0000", 2024-09-18T08:11:25.838 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 8.2099999999999995e-07 2024-09-18T08:11:25.838 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:11:25.839 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:11:25.839 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-09-18T08:11:25.839 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:11:25.808574+0000", 2024-09-18T08:11:25.839 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4.578e-06 2024-09-18T08:11:25.839 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:11:25.839 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:11:25.839 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-09-18T08:11:25.839 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:11:25.808584+0000", 2024-09-18T08:11:25.839 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 9.7210000000000006e-06 2024-09-18T08:11:25.839 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:11:25.839 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:11:25.839 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-09-18T08:11:25.839 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:11:25.808590+0000", 2024-09-18T08:11:25.840 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 6.3679999999999998e-06 2024-09-18T08:11:25.840 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-18T08:11:25.840 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-09-18T08:11:25.840 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-18T08:11:25.840 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-18T08:11:25.840 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2024-09-18T08:11:25.840 INFO:teuthology.orchestra.run.smithi050.stdout: "num_ops": 6 2024-09-18T08:11:25.840 INFO:teuthology.orchestra.run.smithi050.stdout:} 2024-09-18T08:11:25.840 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1 2024-09-18T08:11:25.879 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:11:25.880 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:11:25.880 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1' 2024-09-18T08:11:25.981 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0 2024-09-18T08:11:26.023 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:11:26.024 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:11:26.025 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0' 2024-09-18T08:11:26.125 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2 2024-09-18T08:11:26.161 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:11:26.162 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:11:26.163 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2' 2024-09-18T08:11:26.185 INFO:tasks.radosbench.radosbench.0.smithi007.stdout: 11 16 27860 27844 158.165 132.312 0.0051694 0.00631664 2024-09-18T08:11:26.264 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6 2024-09-18T08:11:26.300 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:11:26.301 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:11:26.301 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6' 2024-09-18T08:11:26.402 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5 2024-09-18T08:11:26.438 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:11:26.438 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:11:26.439 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5' 2024-09-18T08:11:26.539 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2 2024-09-18T08:11:26.582 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:11:26.582 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:11:26.583 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2' 2024-09-18T08:11:26.655 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:11:26 smithi050 ceph-mon[32371]: pgmap v413: 158 pgs: 158 active+clean; 1.4 GiB data, 6.4 GiB used, 709 GiB / 715 GiB avail; 1.3 KiB/s rd, 174 MiB/s wr, 2.79k op/s 2024-09-18T08:11:26.656 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:11:26 smithi050 ceph-mon[32371]: 3.10 scrub starts 2024-09-18T08:11:26.656 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:11:26 smithi050 ceph-mon[32371]: 3.10 scrub ok 2024-09-18T08:11:26.656 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:11:26 smithi050 ceph-mon[32371]: 5.12 scrub starts 2024-09-18T08:11:26.683 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5 2024-09-18T08:11:26.698 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:11:26 smithi007 ceph-mon[39249]: pgmap v413: 158 pgs: 158 active+clean; 1.4 GiB data, 6.4 GiB used, 709 GiB / 715 GiB avail; 1.3 KiB/s rd, 174 MiB/s wr, 2.79k op/s 2024-09-18T08:11:26.699 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:11:26 smithi007 ceph-mon[39249]: 3.10 scrub starts 2024-09-18T08:11:26.699 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:11:26 smithi007 ceph-mon[39249]: 3.10 scrub ok 2024-09-18T08:11:26.699 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:11:26 smithi007 ceph-mon[39249]: 5.12 scrub starts 2024-09-18T08:11:26.699 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:11:26 smithi007 ceph-mon[29459]: pgmap v413: 158 pgs: 158 active+clean; 1.4 GiB data, 6.4 GiB used, 709 GiB / 715 GiB avail; 1.3 KiB/s rd, 174 MiB/s wr, 2.79k op/s 2024-09-18T08:11:26.699 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:11:26 smithi007 ceph-mon[29459]: 3.10 scrub starts 2024-09-18T08:11:26.699 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:11:26 smithi007 ceph-mon[29459]: 3.10 scrub ok 2024-09-18T08:11:26.699 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:11:26 smithi007 ceph-mon[29459]: 5.12 scrub starts 2024-09-18T08:11:26.720 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:11:26.721 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:11:26.721 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5' 2024-09-18T08:11:26.822 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7 2024-09-18T08:11:26.846 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:11:26.847 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:11:26.847 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7' 2024-09-18T08:11:26.948 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7 2024-09-18T08:11:26.980 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Sep 18 08:11:26 smithi007 ceph-779349ca-7593-11ef-bceb-c7b262605968-mgr-y[29688]: ::ffff:172.21.15.50 - - [18/Sep/2024:08:11:26] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-09-18T08:11:26.982 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:11:26.983 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:11:26.984 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7' 2024-09-18T08:11:27.085 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2 2024-09-18T08:11:27.124 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:11:27.124 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:11:27.125 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2' 2024-09-18T08:11:27.185 INFO:tasks.radosbench.radosbench.0.smithi007.stdout: 12 15 29804 29789 155.114 121.562 0.00656152 0.00644191 2024-09-18T08:11:27.226 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5 2024-09-18T08:11:27.260 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:11:27.260 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:11:27.261 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5' 2024-09-18T08:11:27.362 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4 2024-09-18T08:11:27.396 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:11:27.396 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:11:27.396 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4' 2024-09-18T08:11:27.497 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7 2024-09-18T08:11:27.533 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:11:27.534 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:11:27.534 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7' 2024-09-18T08:11:27.635 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7 2024-09-18T08:11:27.655 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:11:27 smithi050 ceph-mon[32371]: 5.12 scrub ok 2024-09-18T08:11:27.670 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:11:27.670 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:11:27.670 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7' 2024-09-18T08:11:27.730 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:11:27 smithi007 ceph-mon[29459]: 5.12 scrub ok 2024-09-18T08:11:27.730 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:11:27 smithi007 ceph-mon[39249]: 5.12 scrub ok 2024-09-18T08:11:27.771 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0 2024-09-18T08:11:27.801 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-09-18T08:11:27.802 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-09-18T08:11:27.802 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid 779349ca-7593-11ef-bceb-c7b262605968 -- ceph tell osd.4 'trim stale osdmaps' 2024-09-18T08:11:27.812 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:11:27.813 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:11:27.814 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0' 2024-09-18T08:11:27.914 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5 2024-09-18T08:11:27.940 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:11:27.941 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:11:27.941 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5' 2024-09-18T08:11:28.042 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2 2024-09-18T08:11:28.081 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:11:28.082 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:11:28.082 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2' 2024-09-18T08:11:28.107 DEBUG:teuthology.orchestra.run.smithi050:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid 779349ca-7593-11ef-bceb-c7b262605968 -- ceph daemon osd.7 dump_blocked_ops 2024-09-18T08:11:28.183 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2 2024-09-18T08:11:28.185 INFO:tasks.radosbench.radosbench.0.smithi007.stdout: 13 16 31854 31838 153.031 128.062 0.00698411 0.00652956 2024-09-18T08:11:28.225 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:11:28.226 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:11:28.226 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2' 2024-09-18T08:11:28.327 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5 2024-09-18T08:11:28.373 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:11:28.373 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:11:28.374 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5' 2024-09-18T08:11:28.475 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3 2024-09-18T08:11:28.506 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/779349ca-7593-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T08:11:28.527 INFO:teuthology.orchestra.run.smithi050.stderr:Inferring config /var/lib/ceph/779349ca-7593-11ef-bceb-c7b262605968/mon.b/config 2024-09-18T08:11:28.529 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:11:28.530 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:11:28.530 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3' 2024-09-18T08:11:28.631 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6 2024-09-18T08:11:28.641 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:11:28 smithi007 ceph-mon[29459]: 6.15 scrub starts 2024-09-18T08:11:28.642 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:11:28 smithi007 ceph-mon[29459]: 6.15 scrub ok 2024-09-18T08:11:28.642 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:11:28 smithi007 ceph-mon[29459]: pgmap v414: 158 pgs: 158 active+clean; 1.4 GiB data, 6.4 GiB used, 709 GiB / 715 GiB avail; 639 B/s rd, 94 MiB/s wr, 1.51k op/s 2024-09-18T08:11:28.642 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:11:28 smithi007 ceph-mon[29459]: 4.9 scrub starts 2024-09-18T08:11:28.642 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:11:28 smithi007 ceph-mon[29459]: 4.9 scrub ok 2024-09-18T08:11:28.642 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:11:28 smithi007 ceph-mon[39249]: 6.15 scrub starts 2024-09-18T08:11:28.642 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:11:28 smithi007 ceph-mon[39249]: 6.15 scrub ok 2024-09-18T08:11:28.642 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:11:28 smithi007 ceph-mon[39249]: pgmap v414: 158 pgs: 158 active+clean; 1.4 GiB data, 6.4 GiB used, 709 GiB / 715 GiB avail; 639 B/s rd, 94 MiB/s wr, 1.51k op/s 2024-09-18T08:11:28.642 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:11:28 smithi007 ceph-mon[39249]: 4.9 scrub starts 2024-09-18T08:11:28.642 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:11:28 smithi007 ceph-mon[39249]: 4.9 scrub ok 2024-09-18T08:11:28.653 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:11:28.654 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:11:28.654 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6' 2024-09-18T08:11:28.655 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:11:28 smithi050 ceph-mon[32371]: 6.15 scrub starts 2024-09-18T08:11:28.655 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:11:28 smithi050 ceph-mon[32371]: 6.15 scrub ok 2024-09-18T08:11:28.656 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:11:28 smithi050 ceph-mon[32371]: pgmap v414: 158 pgs: 158 active+clean; 1.4 GiB data, 6.4 GiB used, 709 GiB / 715 GiB avail; 639 B/s rd, 94 MiB/s wr, 1.51k op/s 2024-09-18T08:11:28.656 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:11:28 smithi050 ceph-mon[32371]: 4.9 scrub starts 2024-09-18T08:11:28.656 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:11:28 smithi050 ceph-mon[32371]: 4.9 scrub ok 2024-09-18T08:11:28.756 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1 2024-09-18T08:11:28.793 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:11:28.794 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:11:28.794 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1' 2024-09-18T08:11:28.894 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6 2024-09-18T08:11:28.927 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:11:28.928 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:11:28.929 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6' 2024-09-18T08:11:29.029 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4 2024-09-18T08:11:29.063 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:11:29.064 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:11:29.065 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4' 2024-09-18T08:11:29.165 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6 2024-09-18T08:11:29.185 INFO:tasks.radosbench.radosbench.0.smithi007.stdout: 14 16 33622 33606 149.993 110.5 0.00630902 0.00666074 2024-09-18T08:11:29.197 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:11:29.198 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:11:29.198 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6' 2024-09-18T08:11:29.299 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7 2024-09-18T08:11:29.335 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:11:29.335 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:11:29.336 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7' 2024-09-18T08:11:29.437 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2 2024-09-18T08:11:29.467 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2024-09-18T08:11:29.467 INFO:teuthology.orchestra.run.smithi050.stdout: "ops": [], 2024-09-18T08:11:29.468 INFO:teuthology.orchestra.run.smithi050.stdout: "complaint_time": 30, 2024-09-18T08:11:29.468 INFO:teuthology.orchestra.run.smithi050.stdout: "num_blocked_ops": 0 2024-09-18T08:11:29.468 INFO:teuthology.orchestra.run.smithi050.stdout:} 2024-09-18T08:11:29.468 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:11:29 smithi050 ceph-mon[32371]: 5.f scrub starts 2024-09-18T08:11:29.468 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:11:29 smithi050 ceph-mon[32371]: 5.f scrub ok 2024-09-18T08:11:29.468 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:11:29 smithi050 ceph-mon[32371]: 6.1 scrub starts 2024-09-18T08:11:29.468 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:11:29 smithi050 ceph-mon[32371]: 6.1 scrub ok 2024-09-18T08:11:29.469 INFO:journalctl@ceph.iscsi.iscsi.a.smithi050.stdout:Sep 18 08:11:29 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-iscsi-iscsi-a[57782]: debug there is no tcmu-runner data available 2024-09-18T08:11:29.474 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:11:29.475 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:11:29.475 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2' 2024-09-18T08:11:29.576 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1 2024-09-18T08:11:29.614 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:11:29.615 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:11:29.616 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1' 2024-09-18T08:11:29.716 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6 2024-09-18T08:11:29.730 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:11:29 smithi007 ceph-mon[29459]: 5.f scrub starts 2024-09-18T08:11:29.730 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:11:29 smithi007 ceph-mon[29459]: 5.f scrub ok 2024-09-18T08:11:29.730 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:11:29 smithi007 ceph-mon[29459]: 6.1 scrub starts 2024-09-18T08:11:29.730 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:11:29 smithi007 ceph-mon[29459]: 6.1 scrub ok 2024-09-18T08:11:29.731 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:11:29 smithi007 ceph-mon[39249]: 5.f scrub starts 2024-09-18T08:11:29.731 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:11:29 smithi007 ceph-mon[39249]: 5.f scrub ok 2024-09-18T08:11:29.731 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:11:29 smithi007 ceph-mon[39249]: 6.1 scrub starts 2024-09-18T08:11:29.731 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:11:29 smithi007 ceph-mon[39249]: 6.1 scrub ok 2024-09-18T08:11:29.754 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:11:29.755 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:11:29.755 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6' 2024-09-18T08:11:29.856 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3 2024-09-18T08:11:29.895 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:11:29.896 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:11:29.896 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3' 2024-09-18T08:11:29.997 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5 2024-09-18T08:11:30.035 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:11:30.035 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:11:30.036 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5' 2024-09-18T08:11:30.137 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4 2024-09-18T08:11:30.184 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:11:30.185 INFO:tasks.radosbench.radosbench.0.smithi007.stdout: 15 16 35498 35482 147.81 117.25 0.00729002 0.00675957 2024-09-18T08:11:30.185 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:11:30.186 DEBUG:teuthology.orchestra.run.smithi050:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid 779349ca-7593-11ef-bceb-c7b262605968 -- ceph daemon osd.7 dump_historic_ops 2024-09-18T08:11:30.186 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4' 2024-09-18T08:11:30.287 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3 2024-09-18T08:11:30.323 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:11:30.324 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:11:30.324 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3' 2024-09-18T08:11:30.425 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4 2024-09-18T08:11:30.455 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:11:30.455 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:11:30.455 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4' 2024-09-18T08:11:30.494 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid 779349ca-7593-11ef-bceb-c7b262605968 -- ceph osd set noscrub 2024-09-18T08:11:30.556 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7 2024-09-18T08:11:30.599 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:11:30.600 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:11:30.600 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7' 2024-09-18T08:11:30.617 INFO:teuthology.orchestra.run.smithi050.stderr:Inferring config /var/lib/ceph/779349ca-7593-11ef-bceb-c7b262605968/mon.b/config 2024-09-18T08:11:30.635 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:11:30 smithi007 ceph-mon[29459]: 1.a scrub starts 2024-09-18T08:11:30.635 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:11:30 smithi007 ceph-mon[29459]: 1.a scrub ok 2024-09-18T08:11:30.635 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:11:30 smithi007 ceph-mon[29459]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-18T08:11:30.635 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:11:30 smithi007 ceph-mon[29459]: pgmap v415: 158 pgs: 158 active+clean; 2.0 GiB data, 8.4 GiB used, 707 GiB / 715 GiB avail; 1.0 KiB/s rd, 142 MiB/s wr, 2.27k op/s 2024-09-18T08:11:30.635 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:11:30 smithi007 ceph-mon[29459]: 6.13 scrub starts 2024-09-18T08:11:30.635 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:11:30 smithi007 ceph-mon[29459]: 6.13 scrub ok 2024-09-18T08:11:30.636 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:11:30 smithi007 ceph-mon[39249]: 1.a scrub starts 2024-09-18T08:11:30.636 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:11:30 smithi007 ceph-mon[39249]: 1.a scrub ok 2024-09-18T08:11:30.636 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:11:30 smithi007 ceph-mon[39249]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-18T08:11:30.636 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:11:30 smithi007 ceph-mon[39249]: pgmap v415: 158 pgs: 158 active+clean; 2.0 GiB data, 8.4 GiB used, 707 GiB / 715 GiB avail; 1.0 KiB/s rd, 142 MiB/s wr, 2.27k op/s 2024-09-18T08:11:30.636 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:11:30 smithi007 ceph-mon[39249]: 6.13 scrub starts 2024-09-18T08:11:30.636 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:11:30 smithi007 ceph-mon[39249]: 6.13 scrub ok 2024-09-18T08:11:30.656 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:11:30 smithi050 ceph-mon[32371]: 1.a scrub starts 2024-09-18T08:11:30.656 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:11:30 smithi050 ceph-mon[32371]: 1.a scrub ok 2024-09-18T08:11:30.656 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:11:30 smithi050 ceph-mon[32371]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-18T08:11:30.656 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:11:30 smithi050 ceph-mon[32371]: pgmap v415: 158 pgs: 158 active+clean; 2.0 GiB data, 8.4 GiB used, 707 GiB / 715 GiB avail; 1.0 KiB/s rd, 142 MiB/s wr, 2.27k op/s 2024-09-18T08:11:30.656 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:11:30 smithi050 ceph-mon[32371]: 6.13 scrub starts 2024-09-18T08:11:30.656 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:11:30 smithi050 ceph-mon[32371]: 6.13 scrub ok 2024-09-18T08:11:30.701 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6 2024-09-18T08:11:30.723 INFO:teuthology.orchestra.run.smithi007.stdout:osd.0: {} 2024-09-18T08:11:30.724 INFO:teuthology.orchestra.run.smithi007.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-18T08:11:30.729 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:11:30.730 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:11:30.730 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6' 2024-09-18T08:11:30.738 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/779349ca-7593-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T08:11:30.773 INFO:teuthology.orchestra.run.smithi007.stdout:osd.1: {} 2024-09-18T08:11:30.773 INFO:teuthology.orchestra.run.smithi007.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-18T08:11:30.808 INFO:teuthology.orchestra.run.smithi007.stdout:osd.2: {} 2024-09-18T08:11:30.808 INFO:teuthology.orchestra.run.smithi007.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-18T08:11:30.831 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3 2024-09-18T08:11:30.852 INFO:teuthology.orchestra.run.smithi007.stdout:osd.3: {} 2024-09-18T08:11:30.853 INFO:teuthology.orchestra.run.smithi007.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-18T08:11:30.877 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:11:30.878 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:11:30.878 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3' 2024-09-18T08:11:30.888 INFO:teuthology.orchestra.run.smithi007.stdout:osd.4: {} 2024-09-18T08:11:30.888 INFO:teuthology.orchestra.run.smithi007.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-18T08:11:30.928 INFO:teuthology.orchestra.run.smithi007.stdout:osd.5: {} 2024-09-18T08:11:30.928 INFO:teuthology.orchestra.run.smithi007.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-18T08:11:30.955 INFO:teuthology.orchestra.run.smithi007.stdout:osd.6: {} 2024-09-18T08:11:30.955 INFO:teuthology.orchestra.run.smithi007.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-18T08:11:30.976 INFO:teuthology.orchestra.run.smithi007.stdout:osd.7: {} 2024-09-18T08:11:30.976 INFO:teuthology.orchestra.run.smithi007.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-18T08:11:30.979 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1 2024-09-18T08:11:31.008 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:11:31.009 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:11:31.009 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1' 2024-09-18T08:11:31.110 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1 2024-09-18T08:11:31.138 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:11:31.139 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:11:31.139 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1' 2024-09-18T08:11:31.185 INFO:tasks.radosbench.radosbench.0.smithi007.stdout: 16 16 37071 37055 144.716 98.3125 0.00621795 0.00690517 2024-09-18T08:11:31.241 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0 2024-09-18T08:11:31.274 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:11:31.275 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:11:31.275 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0' 2024-09-18T08:11:31.376 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5 2024-09-18T08:11:31.411 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:11:31.412 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:11:31.412 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5' 2024-09-18T08:11:31.513 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4 2024-09-18T08:11:31.548 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:11:31.548 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:11:31.549 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4' 2024-09-18T08:11:31.606 INFO:teuthology.orchestra.run.smithi007.stderr:no valid command found; 10 closest matches: 2024-09-18T08:11:31.606 INFO:teuthology.orchestra.run.smithi007.stderr:0 2024-09-18T08:11:31.606 INFO:teuthology.orchestra.run.smithi007.stderr:1 2024-09-18T08:11:31.606 INFO:teuthology.orchestra.run.smithi007.stderr:2 2024-09-18T08:11:31.606 INFO:teuthology.orchestra.run.smithi007.stderr:abort 2024-09-18T08:11:31.606 INFO:teuthology.orchestra.run.smithi007.stderr:assert 2024-09-18T08:11:31.606 INFO:teuthology.orchestra.run.smithi007.stderr:bench [] [] [] [] 2024-09-18T08:11:31.607 INFO:teuthology.orchestra.run.smithi007.stderr:bluefs debug_inject_read_zeros 2024-09-18T08:11:31.607 INFO:teuthology.orchestra.run.smithi007.stderr:bluefs files list 2024-09-18T08:11:31.607 INFO:teuthology.orchestra.run.smithi007.stderr:bluefs stats 2024-09-18T08:11:31.607 INFO:teuthology.orchestra.run.smithi007.stderr:bluestore allocator dump block 2024-09-18T08:11:31.607 INFO:teuthology.orchestra.run.smithi007.stderr:Error EINVAL: invalid command 2024-09-18T08:11:31.649 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3 2024-09-18T08:11:31.655 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:11:31 smithi050 ceph-mon[32371]: 1.1 scrub starts 2024-09-18T08:11:31.656 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:11:31 smithi050 ceph-mon[32371]: 1.1 scrub ok 2024-09-18T08:11:31.656 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:11:31 smithi050 ceph-mon[32371]: 5.13 scrub starts 2024-09-18T08:11:31.656 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:11:31 smithi050 ceph-mon[32371]: 5.13 scrub ok 2024-09-18T08:11:31.656 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:11:31 smithi050 ceph-mon[32371]: from='client.? 172.21.15.7:0/2862642418' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-18T08:11:31.682 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:11:31.683 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:11:31.683 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3' 2024-09-18T08:11:31.712 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:11:31 smithi007 ceph-mon[29459]: 1.1 scrub starts 2024-09-18T08:11:31.712 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:11:31 smithi007 ceph-mon[29459]: 1.1 scrub ok 2024-09-18T08:11:31.713 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:11:31 smithi007 ceph-mon[29459]: 5.13 scrub starts 2024-09-18T08:11:31.713 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:11:31 smithi007 ceph-mon[29459]: 5.13 scrub ok 2024-09-18T08:11:31.713 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:11:31 smithi007 ceph-mon[29459]: from='client.? 172.21.15.7:0/2862642418' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-18T08:11:31.714 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:11:31 smithi007 ceph-mon[39249]: 1.1 scrub starts 2024-09-18T08:11:31.714 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:11:31 smithi007 ceph-mon[39249]: 1.1 scrub ok 2024-09-18T08:11:31.714 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:11:31 smithi007 ceph-mon[39249]: 5.13 scrub starts 2024-09-18T08:11:31.714 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:11:31 smithi007 ceph-mon[39249]: 5.13 scrub ok 2024-09-18T08:11:31.714 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:11:31 smithi007 ceph-mon[39249]: from='client.? 172.21.15.7:0/2862642418' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-18T08:11:31.763 INFO:teuthology.orchestra.run.smithi050.stdout:ERROR: (22) Invalid argument 2024-09-18T08:11:31.763 INFO:teuthology.orchestra.run.smithi050.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-18T08:11:31.784 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5 2024-09-18T08:11:31.827 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:11:31.828 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:11:31.828 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5' 2024-09-18T08:11:31.929 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1 2024-09-18T08:11:31.967 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:11:31.968 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:11:31.969 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1' 2024-09-18T08:11:32.069 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7 2024-09-18T08:11:32.107 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:11:32.107 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:11:32.108 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7' 2024-09-18T08:11:32.185 INFO:tasks.radosbench.radosbench.0.smithi007.stdout: 17 16 39014 38998 143.346 121.438 0.00869605 0.00697166 2024-09-18T08:11:32.209 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2 2024-09-18T08:11:32.246 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:11:32.247 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:11:32.247 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2' 2024-09-18T08:11:32.348 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5 2024-09-18T08:11:32.387 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:11:32.387 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:11:32.388 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5' 2024-09-18T08:11:32.489 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0 2024-09-18T08:11:32.526 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:11:32.527 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:11:32.527 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0' 2024-09-18T08:11:32.533 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-09-18T08:11:32.534 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-09-18T08:11:32.535 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid 779349ca-7593-11ef-bceb-c7b262605968 -- ceph daemon osd.0 dump_ops_in_flight 2024-09-18T08:11:32.628 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5 2024-09-18T08:11:32.656 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:11:32 smithi050 ceph-mon[32371]: 5.1a scrub starts 2024-09-18T08:11:32.656 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:11:32 smithi050 ceph-mon[32371]: 5.1a scrub ok 2024-09-18T08:11:32.656 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:11:32 smithi050 ceph-mon[32371]: pgmap v416: 158 pgs: 158 active+clean; 2.0 GiB data, 8.4 GiB used, 707 GiB / 715 GiB avail; 1023 B/s rd, 141 MiB/s wr, 2.26k op/s 2024-09-18T08:11:32.656 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:11:32 smithi050 ceph-mon[32371]: 3.0 scrub starts 2024-09-18T08:11:32.656 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:11:32 smithi050 ceph-mon[32371]: 3.0 scrub ok 2024-09-18T08:11:32.656 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:11:32 smithi050 ceph-mon[32371]: 6.8 scrub starts 2024-09-18T08:11:32.656 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:11:32 smithi050 ceph-mon[32371]: 6.8 scrub ok 2024-09-18T08:11:32.662 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:11:32.663 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:11:32.663 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5' 2024-09-18T08:11:32.730 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:11:32 smithi007 ceph-mon[29459]: 5.1a scrub starts 2024-09-18T08:11:32.730 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:11:32 smithi007 ceph-mon[29459]: 5.1a scrub ok 2024-09-18T08:11:32.730 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:11:32 smithi007 ceph-mon[29459]: pgmap v416: 158 pgs: 158 active+clean; 2.0 GiB data, 8.4 GiB used, 707 GiB / 715 GiB avail; 1023 B/s rd, 141 MiB/s wr, 2.26k op/s 2024-09-18T08:11:32.730 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:11:32 smithi007 ceph-mon[29459]: 3.0 scrub starts 2024-09-18T08:11:32.731 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:11:32 smithi007 ceph-mon[29459]: 3.0 scrub ok 2024-09-18T08:11:32.731 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:11:32 smithi007 ceph-mon[29459]: 6.8 scrub starts 2024-09-18T08:11:32.731 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:11:32 smithi007 ceph-mon[29459]: 6.8 scrub ok 2024-09-18T08:11:32.731 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:11:32 smithi007 ceph-mon[39249]: 5.1a scrub starts 2024-09-18T08:11:32.731 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:11:32 smithi007 ceph-mon[39249]: 5.1a scrub ok 2024-09-18T08:11:32.731 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:11:32 smithi007 ceph-mon[39249]: pgmap v416: 158 pgs: 158 active+clean; 2.0 GiB data, 8.4 GiB used, 707 GiB / 715 GiB avail; 1023 B/s rd, 141 MiB/s wr, 2.26k op/s 2024-09-18T08:11:32.731 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:11:32 smithi007 ceph-mon[39249]: 3.0 scrub starts 2024-09-18T08:11:32.731 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:11:32 smithi007 ceph-mon[39249]: 3.0 scrub ok 2024-09-18T08:11:32.731 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:11:32 smithi007 ceph-mon[39249]: 6.8 scrub starts 2024-09-18T08:11:32.731 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:11:32 smithi007 ceph-mon[39249]: 6.8 scrub ok 2024-09-18T08:11:32.764 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5 2024-09-18T08:11:32.797 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:11:32.798 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:11:32.798 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5' 2024-09-18T08:11:32.899 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1 2024-09-18T08:11:32.941 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:11:32.941 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:11:32.942 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1' 2024-09-18T08:11:33.043 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2 2024-09-18T08:11:33.079 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:11:33.080 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:11:33.080 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2' 2024-09-18T08:11:33.181 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6 2024-09-18T08:11:33.185 INFO:tasks.radosbench.radosbench.0.smithi007.stdout: 18 16 40947 40931 142.094 120.812 0.00747269 0.00703138 2024-09-18T08:11:33.211 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:11:33.212 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:11:33.212 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6' 2024-09-18T08:11:33.313 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7 2024-09-18T08:11:33.348 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:11:33.348 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:11:33.349 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7' 2024-09-18T08:11:33.450 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4 2024-09-18T08:11:33.483 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:11:33.483 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:11:33.484 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4' 2024-09-18T08:11:33.585 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4 2024-09-18T08:11:33.623 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:11:33.623 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:11:33.624 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4' 2024-09-18T08:11:33.709 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:11:33 smithi007 ceph-mon[29459]: 4.1d scrub starts 2024-09-18T08:11:33.710 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:11:33 smithi007 ceph-mon[29459]: 4.1d scrub ok 2024-09-18T08:11:33.710 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:11:33 smithi007 ceph-mon[29459]: purged_snaps scrub starts 2024-09-18T08:11:33.710 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:11:33 smithi007 ceph-mon[29459]: purged_snaps scrub ok 2024-09-18T08:11:33.710 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:11:33 smithi007 ceph-mon[29459]: 5.4 scrub starts 2024-09-18T08:11:33.710 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:11:33 smithi007 ceph-mon[29459]: 5.4 scrub ok 2024-09-18T08:11:33.710 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:11:33 smithi007 ceph-mon[29459]: from='mgr.24470 172.21.15.7:0/2594390261' entity='mgr.y' 2024-09-18T08:11:33.710 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:11:33 smithi007 ceph-mon[29459]: from='mgr.24470 172.21.15.7:0/2594390261' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-18T08:11:33.710 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:11:33 smithi007 ceph-mon[39249]: 4.1d scrub starts 2024-09-18T08:11:33.710 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:11:33 smithi007 ceph-mon[39249]: 4.1d scrub ok 2024-09-18T08:11:33.711 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:11:33 smithi007 ceph-mon[39249]: purged_snaps scrub starts 2024-09-18T08:11:33.711 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:11:33 smithi007 ceph-mon[39249]: purged_snaps scrub ok 2024-09-18T08:11:33.711 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:11:33 smithi007 ceph-mon[39249]: 5.4 scrub starts 2024-09-18T08:11:33.711 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:11:33 smithi007 ceph-mon[39249]: 5.4 scrub ok 2024-09-18T08:11:33.711 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:11:33 smithi007 ceph-mon[39249]: from='mgr.24470 172.21.15.7:0/2594390261' entity='mgr.y' 2024-09-18T08:11:33.711 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:11:33 smithi007 ceph-mon[39249]: from='mgr.24470 172.21.15.7:0/2594390261' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-18T08:11:33.724 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0 2024-09-18T08:11:33.732 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/779349ca-7593-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T08:11:33.779 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:11:33.780 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:11:33.780 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0' 2024-09-18T08:11:33.881 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0 2024-09-18T08:11:33.905 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:11:33 smithi050 ceph-mon[32371]: 4.1d scrub starts 2024-09-18T08:11:33.906 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:11:33 smithi050 ceph-mon[32371]: 4.1d scrub ok 2024-09-18T08:11:33.906 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:11:33 smithi050 ceph-mon[32371]: purged_snaps scrub starts 2024-09-18T08:11:33.906 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:11:33 smithi050 ceph-mon[32371]: purged_snaps scrub ok 2024-09-18T08:11:33.906 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:11:33 smithi050 ceph-mon[32371]: 5.4 scrub starts 2024-09-18T08:11:33.906 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:11:33 smithi050 ceph-mon[32371]: 5.4 scrub ok 2024-09-18T08:11:33.906 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:11:33 smithi050 ceph-mon[32371]: from='mgr.24470 172.21.15.7:0/2594390261' entity='mgr.y' 2024-09-18T08:11:33.906 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:11:33 smithi050 ceph-mon[32371]: from='mgr.24470 172.21.15.7:0/2594390261' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-18T08:11:33.926 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:11:33.926 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:11:33.927 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0' 2024-09-18T08:11:34.027 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4 2024-09-18T08:11:34.062 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:11:34.063 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:11:34.063 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4' 2024-09-18T08:11:34.164 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1 2024-09-18T08:11:34.185 INFO:tasks.radosbench.radosbench.0.smithi007.stdout: 19 15 42609 42594 140.086 103.938 0.00564637 0.00713404 2024-09-18T08:11:34.205 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:11:34.206 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:11:34.206 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1' 2024-09-18T08:11:34.307 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6 2024-09-18T08:11:34.340 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:11:34.340 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:11:34.341 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6' 2024-09-18T08:11:34.442 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4 2024-09-18T08:11:34.473 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:11:34.474 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:11:34.474 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4' 2024-09-18T08:11:34.478 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/779349ca-7593-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T08:11:34.575 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1 2024-09-18T08:11:34.626 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:11:34.626 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:11:34.626 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1' 2024-09-18T08:11:34.669 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:11:34 smithi007 ceph-mon[29459]: pgmap v417: 158 pgs: 158 active+clean; 2.1 GiB data, 9.0 GiB used, 706 GiB / 715 GiB avail; 1023 B/s rd, 138 MiB/s wr, 2.20k op/s 2024-09-18T08:11:34.669 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:11:34 smithi007 ceph-mon[29459]: osdmap e240: 8 total, 8 up, 8 in 2024-09-18T08:11:34.670 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:11:34 smithi007 ceph-mon[39249]: pgmap v417: 158 pgs: 158 active+clean; 2.1 GiB data, 9.0 GiB used, 706 GiB / 715 GiB avail; 1023 B/s rd, 138 MiB/s wr, 2.20k op/s 2024-09-18T08:11:34.670 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:11:34 smithi007 ceph-mon[39249]: osdmap e240: 8 total, 8 up, 8 in 2024-09-18T08:11:34.727 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2 2024-09-18T08:11:34.767 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:11:34.767 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:11:34.768 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2' 2024-09-18T08:11:34.868 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2 2024-09-18T08:11:34.904 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:11:34.905 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:11:34.905 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2' 2024-09-18T08:11:34.905 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:11:34 smithi050 ceph-mon[32371]: pgmap v417: 158 pgs: 158 active+clean; 2.1 GiB data, 9.0 GiB used, 706 GiB / 715 GiB avail; 1023 B/s rd, 138 MiB/s wr, 2.20k op/s 2024-09-18T08:11:34.906 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:11:34 smithi050 ceph-mon[32371]: osdmap e240: 8 total, 8 up, 8 in 2024-09-18T08:11:35.006 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1 2024-09-18T08:11:35.049 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:11:35.050 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:11:35.050 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1' 2024-09-18T08:11:35.151 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7 2024-09-18T08:11:35.153 INFO:teuthology.orchestra.run.smithi007.stdout:ERROR: (22) Invalid argument 2024-09-18T08:11:35.153 INFO:teuthology.orchestra.run.smithi007.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-18T08:11:35.181 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:11:35.182 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:11:35.182 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7' 2024-09-18T08:11:35.185 INFO:tasks.radosbench.radosbench.0.smithi007.stdout:2024-09-18T08:11:35.184730+0000 min lat: 0.00114659 max lat: 0.108791 avg lat: 0.00722152 2024-09-18T08:11:35.185 INFO:tasks.radosbench.radosbench.0.smithi007.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-09-18T08:11:35.185 INFO:tasks.radosbench.radosbench.0.smithi007.stdout: 20 16 44303 44287 138.372 105.812 0.0138175 0.00722152 2024-09-18T08:11:35.283 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2 2024-09-18T08:11:35.322 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:11:35.323 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:11:35.323 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2' 2024-09-18T08:11:35.417 INFO:teuthology.orchestra.run.smithi007.stderr:noscrub is set 2024-09-18T08:11:35.423 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7 2024-09-18T08:11:35.454 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:11:35.454 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:11:35.455 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7' 2024-09-18T08:11:35.555 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3 2024-09-18T08:11:35.595 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:11:35.596 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:11:35.597 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3' 2024-09-18T08:11:35.697 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5 2024-09-18T08:11:35.712 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid 779349ca-7593-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-18T08:11:35.730 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:11:35 smithi007 ceph-mon[29459]: 6.1e scrub starts 2024-09-18T08:11:35.730 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:11:35 smithi007 ceph-mon[29459]: 6.1e scrub ok 2024-09-18T08:11:35.730 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:11:35 smithi007 ceph-mon[29459]: from='client.? 172.21.15.7:0/3788089803' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-18T08:11:35.730 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:11:35 smithi007 ceph-mon[29459]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-18T08:11:35.731 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:11:35 smithi007 ceph-mon[39249]: 6.1e scrub starts 2024-09-18T08:11:35.731 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:11:35 smithi007 ceph-mon[39249]: 6.1e scrub ok 2024-09-18T08:11:35.731 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:11:35 smithi007 ceph-mon[39249]: from='client.? 172.21.15.7:0/3788089803' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-18T08:11:35.731 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:11:35 smithi007 ceph-mon[39249]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-18T08:11:35.732 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:11:35.733 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:11:35.734 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5' 2024-09-18T08:11:35.834 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3 2024-09-18T08:11:35.878 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:11:35.879 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:11:35.879 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3' 2024-09-18T08:11:35.906 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:11:35 smithi050 ceph-mon[32371]: 6.1e scrub starts 2024-09-18T08:11:35.906 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:11:35 smithi050 ceph-mon[32371]: 6.1e scrub ok 2024-09-18T08:11:35.906 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:11:35 smithi050 ceph-mon[32371]: from='client.? 172.21.15.7:0/3788089803' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-18T08:11:35.906 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:11:35 smithi050 ceph-mon[32371]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-18T08:11:35.980 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6 2024-09-18T08:11:36.015 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:11:36.015 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:11:36.016 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6' 2024-09-18T08:11:36.117 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7 2024-09-18T08:11:36.152 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:11:36.153 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:11:36.153 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7' 2024-09-18T08:11:36.186 INFO:tasks.radosbench.radosbench.0.smithi007.stdout: 21 16 46063 46047 137.015 110 0.0189264 0.00729286 2024-09-18T08:11:36.203 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid 779349ca-7593-11ef-bceb-c7b262605968 -- ceph daemon osd.0 dump_blocked_ops 2024-09-18T08:11:36.254 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4 2024-09-18T08:11:36.287 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:11:36.288 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:11:36.289 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4' 2024-09-18T08:11:36.389 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3 2024-09-18T08:11:36.427 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:11:36 smithi007 ceph-mon[29459]: 1.0 scrub starts 2024-09-18T08:11:36.427 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:11:36 smithi007 ceph-mon[29459]: 1.0 scrub ok 2024-09-18T08:11:36.433 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:11:36.433 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:11:36.434 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3' 2024-09-18T08:11:36.535 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0 2024-09-18T08:11:36.582 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:11:36.583 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:11:36.583 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0' 2024-09-18T08:11:36.684 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3 2024-09-18T08:11:36.700 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:11:36 smithi007 ceph-mon[29459]: pgmap v419: 158 pgs: 158 active+clean; 2.5 GiB data, 10 GiB used, 705 GiB / 715 GiB avail; 1023 B/s rd, 122 MiB/s wr, 1.95k op/s 2024-09-18T08:11:36.700 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:11:36 smithi007 ceph-mon[29459]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-18T08:11:36.700 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:11:36 smithi007 ceph-mon[29459]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-18T08:11:36.700 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:11:36 smithi007 ceph-mon[29459]: osdmap e241: 8 total, 8 up, 8 in 2024-09-18T08:11:36.700 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Sep 18 08:11:36 smithi007 ceph-779349ca-7593-11ef-bceb-c7b262605968-mgr-y[29688]: ::ffff:172.21.15.50 - - [18/Sep/2024:08:11:36] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-09-18T08:11:36.700 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:11:36 smithi007 ceph-mon[39249]: 1.0 scrub starts 2024-09-18T08:11:36.700 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:11:36 smithi007 ceph-mon[39249]: 1.0 scrub ok 2024-09-18T08:11:36.701 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:11:36 smithi007 ceph-mon[39249]: pgmap v419: 158 pgs: 158 active+clean; 2.5 GiB data, 10 GiB used, 705 GiB / 715 GiB avail; 1023 B/s rd, 122 MiB/s wr, 1.95k op/s 2024-09-18T08:11:36.701 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:11:36 smithi007 ceph-mon[39249]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-18T08:11:36.701 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:11:36 smithi007 ceph-mon[39249]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-18T08:11:36.701 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:11:36 smithi007 ceph-mon[39249]: osdmap e241: 8 total, 8 up, 8 in 2024-09-18T08:11:36.729 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:11:36.729 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:11:36.730 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3' 2024-09-18T08:11:36.827 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/779349ca-7593-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T08:11:36.830 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3 2024-09-18T08:11:36.853 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/779349ca-7593-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T08:11:36.889 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:11:36.890 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:11:36.890 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3' 2024-09-18T08:11:36.905 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:11:36 smithi050 ceph-mon[32371]: 1.0 scrub starts 2024-09-18T08:11:36.906 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:11:36 smithi050 ceph-mon[32371]: 1.0 scrub ok 2024-09-18T08:11:36.906 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:11:36 smithi050 ceph-mon[32371]: pgmap v419: 158 pgs: 158 active+clean; 2.5 GiB data, 10 GiB used, 705 GiB / 715 GiB avail; 1023 B/s rd, 122 MiB/s wr, 1.95k op/s 2024-09-18T08:11:36.906 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:11:36 smithi050 ceph-mon[32371]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-18T08:11:36.906 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:11:36 smithi050 ceph-mon[32371]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-18T08:11:36.906 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:11:36 smithi050 ceph-mon[32371]: osdmap e241: 8 total, 8 up, 8 in 2024-09-18T08:11:36.992 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4 2024-09-18T08:11:37.029 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:11:37.030 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:11:37.030 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4' 2024-09-18T08:11:37.131 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6 2024-09-18T08:11:37.162 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:11:37.162 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:11:37.163 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6' 2024-09-18T08:11:37.186 INFO:tasks.radosbench.radosbench.0.smithi007.stdout: 22 16 47710 47694 135.465 102.938 0.00203114 0.00736817 2024-09-18T08:11:37.264 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0 2024-09-18T08:11:37.302 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:11:37.303 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:11:37.303 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0' 2024-09-18T08:11:37.404 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3 2024-09-18T08:11:37.447 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:11:37.448 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:11:37.448 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3' 2024-09-18T08:11:37.535 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 5, 7, 6, 4] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-09-18T08:11:37.535 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-18T08:11:37.535 INFO:tasks.thrashosds.thrasher:Killing osd 1, live_osds are [0, 1, 2, 3, 4, 5, 6, 7] 2024-09-18T08:11:37.535 INFO:tasks.cephadm.osd.1:Stopping osd.1... 2024-09-18T08:11:37.536 DEBUG:teuthology.orchestra.run.smithi007:> sudo systemctl stop ceph-779349ca-7593-11ef-bceb-c7b262605968@osd.1 2024-09-18T08:11:37.549 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5 2024-09-18T08:11:37.582 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:11:37.583 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:11:37.583 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5' 2024-09-18T08:11:37.599 INFO:teuthology.orchestra.run.smithi007.stdout:osd.0: {} 2024-09-18T08:11:37.599 INFO:teuthology.orchestra.run.smithi007.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-18T08:11:37.641 INFO:teuthology.orchestra.run.smithi007.stdout:osd.1: {} 2024-09-18T08:11:37.642 INFO:teuthology.orchestra.run.smithi007.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-18T08:11:37.684 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3 2024-09-18T08:11:37.689 INFO:teuthology.orchestra.run.smithi007.stdout:osd.2: {} 2024-09-18T08:11:37.689 INFO:teuthology.orchestra.run.smithi007.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-18T08:11:37.726 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:11:37.726 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:11:37.727 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3' 2024-09-18T08:11:37.729 INFO:teuthology.orchestra.run.smithi007.stdout:osd.3: {} 2024-09-18T08:11:37.729 INFO:teuthology.orchestra.run.smithi007.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-18T08:11:37.759 INFO:teuthology.orchestra.run.smithi007.stdout:osd.4: {} 2024-09-18T08:11:37.759 INFO:teuthology.orchestra.run.smithi007.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-18T08:11:37.798 INFO:teuthology.orchestra.run.smithi007.stdout:osd.5: {} 2024-09-18T08:11:37.799 INFO:teuthology.orchestra.run.smithi007.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-18T08:11:37.827 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6 2024-09-18T08:11:37.835 INFO:teuthology.orchestra.run.smithi007.stdout:osd.6: {} 2024-09-18T08:11:37.836 INFO:teuthology.orchestra.run.smithi007.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-18T08:11:37.836 INFO:journalctl@ceph.osd.1.smithi007.stdout:Sep 18 08:11:37 smithi007 systemd[1]: Stopping Ceph osd.1 for 779349ca-7593-11ef-bceb-c7b262605968... 2024-09-18T08:11:37.856 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:11:37.857 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:11:37.857 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6' 2024-09-18T08:11:37.874 INFO:teuthology.orchestra.run.smithi007.stdout:osd.7: {} 2024-09-18T08:11:37.874 INFO:teuthology.orchestra.run.smithi007.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-18T08:11:37.940 INFO:teuthology.orchestra.run.smithi007.stdout:{ 2024-09-18T08:11:37.940 INFO:teuthology.orchestra.run.smithi007.stdout: "ops": [], 2024-09-18T08:11:37.940 INFO:teuthology.orchestra.run.smithi007.stdout: "complaint_time": 30, 2024-09-18T08:11:37.940 INFO:teuthology.orchestra.run.smithi007.stdout: "num_blocked_ops": 0 2024-09-18T08:11:37.940 INFO:teuthology.orchestra.run.smithi007.stdout:} 2024-09-18T08:11:37.958 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3 2024-09-18T08:11:37.998 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:11:37.999 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:11:37.999 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3' 2024-09-18T08:11:38.100 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6 2024-09-18T08:11:38.132 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:11:38.133 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:11:38.133 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6' 2024-09-18T08:11:38.186 INFO:tasks.radosbench.radosbench.0.smithi007.stdout: 23 16 49155 49139 133.502 90.3125 0.00626612 0.00748465 2024-09-18T08:11:38.234 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7 2024-09-18T08:11:38.267 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:11:38.268 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:11:38.268 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7' 2024-09-18T08:11:38.369 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0 2024-09-18T08:11:38.408 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:11:38.409 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:11:38.410 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0' 2024-09-18T08:11:38.480 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:11:38 smithi007 ceph-mon[29459]: pgmap v421: 158 pgs: 158 active+clean; 2.5 GiB data, 10 GiB used, 705 GiB / 715 GiB avail; 639 B/s rd, 70 MiB/s wr, 1.12k op/s 2024-09-18T08:11:38.480 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:11:38 smithi007 ceph-mon[29459]: from='client.? 172.21.15.7:0/219641422' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-18T08:11:38.481 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:11:38 smithi007 ceph-mon[39249]: pgmap v421: 158 pgs: 158 active+clean; 2.5 GiB data, 10 GiB used, 705 GiB / 715 GiB avail; 639 B/s rd, 70 MiB/s wr, 1.12k op/s 2024-09-18T08:11:38.481 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:11:38 smithi007 ceph-mon[39249]: from='client.? 172.21.15.7:0/219641422' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-18T08:11:38.510 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2 2024-09-18T08:11:38.549 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:11:38.550 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:11:38.550 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2' 2024-09-18T08:11:38.651 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3 2024-09-18T08:11:38.691 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:11:38.692 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:11:38.693 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3' 2024-09-18T08:11:38.793 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3 2024-09-18T08:11:38.803 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid 779349ca-7593-11ef-bceb-c7b262605968 -- ceph osd set nodeep-scrub 2024-09-18T08:11:38.835 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:11:38.836 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:11:38.836 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3' 2024-09-18T08:11:38.905 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:11:38 smithi050 ceph-mon[32371]: pgmap v421: 158 pgs: 158 active+clean; 2.5 GiB data, 10 GiB used, 705 GiB / 715 GiB avail; 639 B/s rd, 70 MiB/s wr, 1.12k op/s 2024-09-18T08:11:38.906 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:11:38 smithi050 ceph-mon[32371]: from='client.? 172.21.15.7:0/219641422' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-18T08:11:38.937 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3 2024-09-18T08:11:38.977 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:11:38.978 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:11:38.978 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3' 2024-09-18T08:11:39.079 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4 2024-09-18T08:11:39.113 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:11:39.113 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:11:39.113 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4' 2024-09-18T08:11:39.186 INFO:tasks.radosbench.radosbench.0.smithi007.stdout: 24 16 50856 50840 132.369 106.312 0.00375635 0.00754903 2024-09-18T08:11:39.214 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6 2024-09-18T08:11:39.241 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:11:39.242 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:11:39.242 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6' 2024-09-18T08:11:39.343 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6 2024-09-18T08:11:39.376 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:11:39.377 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:11:39.378 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6' 2024-09-18T08:11:39.478 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3 2024-09-18T08:11:39.502 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:11:39.502 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:11:39.503 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3' 2024-09-18T08:11:39.604 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2 2024-09-18T08:11:39.644 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:11:39.645 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:11:39.645 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2' 2024-09-18T08:11:39.655 INFO:journalctl@ceph.iscsi.iscsi.a.smithi050.stdout:Sep 18 08:11:39 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-iscsi-iscsi-a[57782]: debug there is no tcmu-runner data available 2024-09-18T08:11:39.746 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3 2024-09-18T08:11:39.788 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:11:39.788 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:11:39.789 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3' 2024-09-18T08:11:39.890 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0 2024-09-18T08:11:39.933 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:11:39.934 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:11:39.934 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0' 2024-09-18T08:11:39.981 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid 779349ca-7593-11ef-bceb-c7b262605968 -- ceph daemon osd.0 dump_historic_ops 2024-09-18T08:11:40.035 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0 2024-09-18T08:11:40.081 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:11:40.081 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:11:40.081 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0' 2024-09-18T08:11:40.183 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7 2024-09-18T08:11:40.190 INFO:tasks.radosbench.radosbench.0.smithi007.stdout: 25 16 52205 52189 130.437 84.3125 0.0204762 0.00766079 2024-09-18T08:11:40.211 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:11:40.212 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:11:40.212 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7' 2024-09-18T08:11:40.313 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0 2024-09-18T08:11:40.370 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/779349ca-7593-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T08:11:40.375 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:11:40.376 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:11:40.376 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0' 2024-09-18T08:11:40.477 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5 2024-09-18T08:11:40.508 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:11:40.509 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:11:40.509 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5' 2024-09-18T08:11:40.550 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:11:40 smithi007 ceph-mon[29459]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-18T08:11:40.550 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:11:40 smithi007 ceph-mon[29459]: pgmap v422: 158 pgs: 158 active+clean; 3.1 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 1.2 KiB/s rd, 135 MiB/s wr, 2.16k op/s 2024-09-18T08:11:40.551 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:11:40 smithi007 ceph-mon[29459]: osdmap e242: 8 total, 8 up, 8 in 2024-09-18T08:11:40.551 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:11:40 smithi007 ceph-mon[29459]: purged_snaps scrub starts 2024-09-18T08:11:40.551 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:11:40 smithi007 ceph-mon[29459]: purged_snaps scrub ok 2024-09-18T08:11:40.551 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:11:40 smithi007 ceph-mon[39249]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-18T08:11:40.551 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:11:40 smithi007 ceph-mon[39249]: pgmap v422: 158 pgs: 158 active+clean; 3.1 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 1.2 KiB/s rd, 135 MiB/s wr, 2.16k op/s 2024-09-18T08:11:40.551 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:11:40 smithi007 ceph-mon[39249]: osdmap e242: 8 total, 8 up, 8 in 2024-09-18T08:11:40.551 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:11:40 smithi007 ceph-mon[39249]: purged_snaps scrub starts 2024-09-18T08:11:40.551 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:11:40 smithi007 ceph-mon[39249]: purged_snaps scrub ok 2024-09-18T08:11:40.551 INFO:journalctl@ceph.osd.1.smithi007.stdout:Sep 18 08:11:40 smithi007 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd-1[47960]: 2024-09-18T08:11:40.331+0000 7fcd0226e640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.1 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-09-18T08:11:40.552 INFO:journalctl@ceph.osd.1.smithi007.stdout:Sep 18 08:11:40 smithi007 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd-1[47960]: 2024-09-18T08:11:40.331+0000 7fcd0226e640 -1 osd.1 243 *** Got signal Terminated *** 2024-09-18T08:11:40.552 INFO:journalctl@ceph.osd.1.smithi007.stdout:Sep 18 08:11:40 smithi007 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd-1[47960]: 2024-09-18T08:11:40.331+0000 7fcd0226e640 -1 osd.1 243 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-09-18T08:11:40.610 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0 2024-09-18T08:11:40.651 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:11:40.652 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:11:40.652 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0' 2024-09-18T08:11:40.656 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:11:40 smithi050 ceph-mon[32371]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-18T08:11:40.656 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:11:40 smithi050 ceph-mon[32371]: pgmap v422: 158 pgs: 158 active+clean; 3.1 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 1.2 KiB/s rd, 135 MiB/s wr, 2.16k op/s 2024-09-18T08:11:40.656 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:11:40 smithi050 ceph-mon[32371]: osdmap e242: 8 total, 8 up, 8 in 2024-09-18T08:11:40.656 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:11:40 smithi050 ceph-mon[32371]: purged_snaps scrub starts 2024-09-18T08:11:40.656 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:11:40 smithi050 ceph-mon[32371]: purged_snaps scrub ok 2024-09-18T08:11:40.753 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5 2024-09-18T08:11:40.777 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:11:40.778 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:11:40.778 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5' 2024-09-18T08:11:40.879 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7 2024-09-18T08:11:40.910 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:11:40.910 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:11:40.911 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7' 2024-09-18T08:11:41.012 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4 2024-09-18T08:11:41.034 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:11:41.035 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:11:41.035 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4' 2024-09-18T08:11:41.136 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4 2024-09-18T08:11:41.168 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:11:41.169 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:11:41.169 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4' 2024-09-18T08:11:41.188 INFO:tasks.radosbench.radosbench.0.smithi007.stdout: 26 16 53406 53390 128.307 75.0625 0.0266311 0.00778835 2024-09-18T08:11:41.270 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5 2024-09-18T08:11:41.298 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:11:41.299 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:11:41.299 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5' 2024-09-18T08:11:41.313 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/779349ca-7593-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T08:11:41.400 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5 2024-09-18T08:11:41.422 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:11:41.423 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:11:41.423 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5' 2024-09-18T08:11:41.524 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7 2024-09-18T08:11:41.548 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:11:41.549 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:11:41.549 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7' 2024-09-18T08:11:41.585 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:11:41 smithi007 ceph-mon[29459]: osdmap e243: 8 total, 8 up, 8 in 2024-09-18T08:11:41.585 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:11:41 smithi007 ceph-mon[29459]: osd.1 marked itself down and dead 2024-09-18T08:11:41.586 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:11:41 smithi007 ceph-mon[39249]: osdmap e243: 8 total, 8 up, 8 in 2024-09-18T08:11:41.586 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:11:41 smithi007 ceph-mon[39249]: osd.1 marked itself down and dead 2024-09-18T08:11:41.650 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5 2024-09-18T08:11:41.655 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:11:41 smithi050 ceph-mon[32371]: osdmap e243: 8 total, 8 up, 8 in 2024-09-18T08:11:41.656 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:11:41 smithi050 ceph-mon[32371]: osd.1 marked itself down and dead 2024-09-18T08:11:41.673 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:11:41.673 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:11:41.673 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5' 2024-09-18T08:11:41.774 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0 2024-09-18T08:11:41.799 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:11:41.800 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:11:41.800 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0' 2024-09-18T08:11:41.816 INFO:teuthology.orchestra.run.smithi007.stdout:{ 2024-09-18T08:11:41.816 INFO:teuthology.orchestra.run.smithi007.stdout: "size": 20, 2024-09-18T08:11:41.816 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 600, 2024-09-18T08:11:41.816 INFO:teuthology.orchestra.run.smithi007.stdout: "ops": [ 2024-09-18T08:11:41.816 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:11:41.816 INFO:teuthology.orchestra.run.smithi007.stdout: "description": "osd_op(client.14805.0:65085 7.7 7:e643a565:::benchmark_data_smithi007_73155_object65084:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e101)", 2024-09-18T08:11:41.816 INFO:teuthology.orchestra.run.smithi007.stdout: "initiated_at": "2024-09-18T08:04:42.325315+0000", 2024-09-18T08:11:41.816 INFO:teuthology.orchestra.run.smithi007.stdout: "age": 419.468577262, 2024-09-18T08:11:41.817 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.99419469699999996, 2024-09-18T08:11:41.817 INFO:teuthology.orchestra.run.smithi007.stdout: "continuous": false, 2024-09-18T08:11:41.817 INFO:teuthology.orchestra.run.smithi007.stdout: "type_data": { 2024-09-18T08:11:41.817 INFO:teuthology.orchestra.run.smithi007.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T08:11:41.817 INFO:teuthology.orchestra.run.smithi007.stdout: "client_info": { 2024-09-18T08:11:41.817 INFO:teuthology.orchestra.run.smithi007.stdout: "client": "client.14805", 2024-09-18T08:11:41.817 INFO:teuthology.orchestra.run.smithi007.stdout: "client_addr": "172.21.15.7:0/640320436", 2024-09-18T08:11:41.817 INFO:teuthology.orchestra.run.smithi007.stdout: "tid": 65085 2024-09-18T08:11:41.817 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:11:41.817 INFO:teuthology.orchestra.run.smithi007.stdout: "events": [ 2024-09-18T08:11:41.817 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:11:41.817 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "initiated", 2024-09-18T08:11:41.817 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:04:42.325315+0000", 2024-09-18T08:11:41.818 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-09-18T08:11:41.818 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:11:41.818 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:11:41.818 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "header_read", 2024-09-18T08:11:41.818 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:04:42.325315+0000", 2024-09-18T08:11:41.818 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-09-18T08:11:41.818 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:11:41.818 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:11:41.818 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "throttled", 2024-09-18T08:11:41.818 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:04:42.325316+0000", 2024-09-18T08:11:41.818 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.621e-06 2024-09-18T08:11:41.818 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:11:41.818 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:11:41.819 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "all_read", 2024-09-18T08:11:41.819 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:04:42.325346+0000", 2024-09-18T08:11:41.819 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 2.9215999999999999e-05 2024-09-18T08:11:41.819 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:11:41.819 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:11:41.819 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "dispatched", 2024-09-18T08:11:41.819 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:04:42.325438+0000", 2024-09-18T08:11:41.819 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 9.2672999999999999e-05 2024-09-18T08:11:41.819 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:11:41.819 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:11:41.819 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "queued_for_pg", 2024-09-18T08:11:41.819 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:04:42.325447+0000", 2024-09-18T08:11:41.819 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 8.6989999999999992e-06 2024-09-18T08:11:41.820 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:11:41.820 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:11:41.820 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-09-18T08:11:41.820 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:04:42.325518+0000", 2024-09-18T08:11:41.820 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 7.0924000000000001e-05 2024-09-18T08:11:41.820 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:11:41.820 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:11:41.820 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "waiting for peered", 2024-09-18T08:11:41.820 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:04:42.325529+0000", 2024-09-18T08:11:41.820 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.0774999999999999e-05 2024-09-18T08:11:41.820 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:11:41.820 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:11:41.820 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-09-18T08:11:41.821 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:04:43.305082+0000", 2024-09-18T08:11:41.821 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.97955334900000002 2024-09-18T08:11:41.821 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:11:41.821 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:11:41.821 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "started", 2024-09-18T08:11:41.821 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:04:43.306711+0000", 2024-09-18T08:11:41.821 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.0016285259999999999 2024-09-18T08:11:41.821 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:11:41.821 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:11:41.821 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "waiting for subops from 3,6", 2024-09-18T08:11:41.821 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:04:43.306878+0000", 2024-09-18T08:11:41.821 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.00016743300000000001 2024-09-18T08:11:41.821 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:11:41.822 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:11:41.822 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:11:41.822 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:04:43.312523+0000", 2024-09-18T08:11:41.822 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.0056453880000000003 2024-09-18T08:11:41.822 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:11:41.822 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:11:41.822 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:11:41.822 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:04:43.319352+0000", 2024-09-18T08:11:41.822 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.0068285100000000003 2024-09-18T08:11:41.822 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:11:41.822 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:11:41.823 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "op_commit", 2024-09-18T08:11:41.823 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:04:43.319365+0000", 2024-09-18T08:11:41.823 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.2870999999999999e-05 2024-09-18T08:11:41.823 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:11:41.823 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:11:41.823 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "commit_sent", 2024-09-18T08:11:41.823 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:04:43.319431+0000", 2024-09-18T08:11:41.823 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 6.6431000000000001e-05 2024-09-18T08:11:41.823 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:11:41.823 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:11:41.823 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "done", 2024-09-18T08:11:41.824 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:04:43.319509+0000", 2024-09-18T08:11:41.824 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 7.8281000000000005e-05 2024-09-18T08:11:41.824 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-09-18T08:11:41.824 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-09-18T08:11:41.824 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-09-18T08:11:41.824 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:11:41.824 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:11:41.824 INFO:teuthology.orchestra.run.smithi007.stdout: "description": "osd_op(client.14805.0:65088 7.7 7:ec575e01:::benchmark_data_smithi007_73155_object65087:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e101)", 2024-09-18T08:11:41.824 INFO:teuthology.orchestra.run.smithi007.stdout: "initiated_at": "2024-09-18T08:04:42.325472+0000", 2024-09-18T08:11:41.824 INFO:teuthology.orchestra.run.smithi007.stdout: "age": 419.46841987900001, 2024-09-18T08:11:41.824 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.99450635799999998, 2024-09-18T08:11:41.824 INFO:teuthology.orchestra.run.smithi007.stdout: "continuous": false, 2024-09-18T08:11:41.824 INFO:teuthology.orchestra.run.smithi007.stdout: "type_data": { 2024-09-18T08:11:41.825 INFO:teuthology.orchestra.run.smithi007.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T08:11:41.825 INFO:teuthology.orchestra.run.smithi007.stdout: "client_info": { 2024-09-18T08:11:41.825 INFO:teuthology.orchestra.run.smithi007.stdout: "client": "client.14805", 2024-09-18T08:11:41.825 INFO:teuthology.orchestra.run.smithi007.stdout: "client_addr": "172.21.15.7:0/640320436", 2024-09-18T08:11:41.825 INFO:teuthology.orchestra.run.smithi007.stdout: "tid": 65088 2024-09-18T08:11:41.825 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:11:41.825 INFO:teuthology.orchestra.run.smithi007.stdout: "events": [ 2024-09-18T08:11:41.825 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:11:41.825 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "initiated", 2024-09-18T08:11:41.825 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:04:42.325472+0000", 2024-09-18T08:11:41.825 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-09-18T08:11:41.825 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:11:41.825 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:11:41.825 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "header_read", 2024-09-18T08:11:41.826 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:04:42.325472+0000", 2024-09-18T08:11:41.826 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-09-18T08:11:41.826 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:11:41.826 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:11:41.826 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "throttled", 2024-09-18T08:11:41.826 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:04:42.325473+0000", 2024-09-18T08:11:41.826 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 6.5700000000000002e-07 2024-09-18T08:11:41.826 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:11:41.826 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:11:41.826 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "all_read", 2024-09-18T08:11:41.826 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:04:42.325501+0000", 2024-09-18T08:11:41.826 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 2.7662999999999999e-05 2024-09-18T08:11:41.826 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:11:41.827 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:11:41.827 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "dispatched", 2024-09-18T08:11:41.827 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:04:42.325508+0000", 2024-09-18T08:11:41.827 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 7.0509999999999999e-06 2024-09-18T08:11:41.827 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:11:41.827 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:11:41.827 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "queued_for_pg", 2024-09-18T08:11:41.827 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:04:42.325513+0000", 2024-09-18T08:11:41.827 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 5.8760000000000002e-06 2024-09-18T08:11:41.827 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:11:41.827 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:11:41.827 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-09-18T08:11:41.827 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:04:42.325558+0000", 2024-09-18T08:11:41.827 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 4.4384000000000001e-05 2024-09-18T08:11:41.827 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:11:41.828 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:11:41.828 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "waiting for peered", 2024-09-18T08:11:41.828 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:04:42.325570+0000", 2024-09-18T08:11:41.828 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.1719000000000001e-05 2024-09-18T08:11:41.828 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:11:41.828 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:11:41.828 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-09-18T08:11:41.828 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:04:43.309576+0000", 2024-09-18T08:11:41.828 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.98400614799999997 2024-09-18T08:11:41.828 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:11:41.828 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:11:41.828 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "started", 2024-09-18T08:11:41.828 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:04:43.309740+0000", 2024-09-18T08:11:41.829 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.000164229 2024-09-18T08:11:41.829 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:11:41.829 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:11:41.829 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "waiting for subops from 3,6", 2024-09-18T08:11:41.829 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:04:43.310047+0000", 2024-09-18T08:11:41.829 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.00030731600000000002 2024-09-18T08:11:41.829 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:11:41.829 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:11:41.829 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:11:41.829 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:04:43.312568+0000", 2024-09-18T08:11:41.829 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.0025208069999999999 2024-09-18T08:11:41.830 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:11:41.830 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:11:41.830 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "op_commit", 2024-09-18T08:11:41.830 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:04:43.319825+0000", 2024-09-18T08:11:41.830 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.0072565499999999996 2024-09-18T08:11:41.830 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:11:41.830 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:11:41.830 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:11:41.830 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:04:43.319883+0000", 2024-09-18T08:11:41.830 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 5.8579000000000003e-05 2024-09-18T08:11:41.830 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:11:41.830 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:11:41.830 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "commit_sent", 2024-09-18T08:11:41.831 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:04:43.319940+0000", 2024-09-18T08:11:41.831 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 5.6436e-05 2024-09-18T08:11:41.831 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:11:41.831 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:11:41.831 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "done", 2024-09-18T08:11:41.831 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:04:43.319979+0000", 2024-09-18T08:11:41.831 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 3.8943000000000003e-05 2024-09-18T08:11:41.831 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-09-18T08:11:41.831 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-09-18T08:11:41.831 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-09-18T08:11:41.831 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:11:41.831 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:11:41.831 INFO:teuthology.orchestra.run.smithi007.stdout: "description": "osd_op(client.14805.0:65089 7.7 7:effcefb5:::benchmark_data_smithi007_73155_object65088:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e101)", 2024-09-18T08:11:41.831 INFO:teuthology.orchestra.run.smithi007.stdout: "initiated_at": "2024-09-18T08:04:42.325980+0000", 2024-09-18T08:11:41.832 INFO:teuthology.orchestra.run.smithi007.stdout: "age": 419.46791171900003, 2024-09-18T08:11:41.832 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.99416539400000004, 2024-09-18T08:11:41.832 INFO:teuthology.orchestra.run.smithi007.stdout: "continuous": false, 2024-09-18T08:11:41.832 INFO:teuthology.orchestra.run.smithi007.stdout: "type_data": { 2024-09-18T08:11:41.832 INFO:teuthology.orchestra.run.smithi007.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T08:11:41.832 INFO:teuthology.orchestra.run.smithi007.stdout: "client_info": { 2024-09-18T08:11:41.832 INFO:teuthology.orchestra.run.smithi007.stdout: "client": "client.14805", 2024-09-18T08:11:41.832 INFO:teuthology.orchestra.run.smithi007.stdout: "client_addr": "172.21.15.7:0/640320436", 2024-09-18T08:11:41.832 INFO:teuthology.orchestra.run.smithi007.stdout: "tid": 65089 2024-09-18T08:11:41.832 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:11:41.832 INFO:teuthology.orchestra.run.smithi007.stdout: "events": [ 2024-09-18T08:11:41.832 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:11:41.832 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "initiated", 2024-09-18T08:11:41.833 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:04:42.325980+0000", 2024-09-18T08:11:41.833 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-09-18T08:11:41.833 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:11:41.833 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:11:41.833 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "header_read", 2024-09-18T08:11:41.833 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:04:42.325980+0000", 2024-09-18T08:11:41.833 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-09-18T08:11:41.833 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:11:41.833 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:11:41.834 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "throttled", 2024-09-18T08:11:41.834 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:04:42.325981+0000", 2024-09-18T08:11:41.834 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.026e-06 2024-09-18T08:11:41.834 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:11:41.834 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:11:41.834 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "all_read", 2024-09-18T08:11:41.834 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:04:42.326003+0000", 2024-09-18T08:11:41.834 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 2.2048999999999998e-05 2024-09-18T08:11:41.834 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:11:41.834 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:11:41.834 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "dispatched", 2024-09-18T08:11:41.834 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:04:42.326014+0000", 2024-09-18T08:11:41.834 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.0560000000000001e-05 2024-09-18T08:11:41.834 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:11:41.835 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:11:41.835 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "queued_for_pg", 2024-09-18T08:11:41.835 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:04:42.326021+0000", 2024-09-18T08:11:41.835 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 6.9990000000000002e-06 2024-09-18T08:11:41.835 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:11:41.835 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:11:41.835 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-09-18T08:11:41.835 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:04:42.326162+0000", 2024-09-18T08:11:41.835 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.000141447 2024-09-18T08:11:41.835 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:11:41.835 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:11:41.835 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "waiting for peered", 2024-09-18T08:11:41.835 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:04:42.326172+0000", 2024-09-18T08:11:41.836 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 9.2459999999999992e-06 2024-09-18T08:11:41.836 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:11:41.836 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:11:41.836 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-09-18T08:11:41.836 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:04:43.312600+0000", 2024-09-18T08:11:41.836 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.98642846200000001 2024-09-18T08:11:41.836 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:11:41.836 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:11:41.836 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "started", 2024-09-18T08:11:41.836 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:04:43.312707+0000", 2024-09-18T08:11:41.836 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.000107225 2024-09-18T08:11:41.836 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:11:41.836 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:11:41.837 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "waiting for subops from 3,6", 2024-09-18T08:11:41.837 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:04:43.312837+0000", 2024-09-18T08:11:41.837 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.00013003900000000001 2024-09-18T08:11:41.837 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:11:41.837 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:11:41.837 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:11:41.837 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:04:43.319777+0000", 2024-09-18T08:11:41.837 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.0069396969999999999 2024-09-18T08:11:41.837 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:11:41.837 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:11:41.837 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "op_commit", 2024-09-18T08:11:41.837 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:04:43.319830+0000", 2024-09-18T08:11:41.837 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 5.2747000000000002e-05 2024-09-18T08:11:41.838 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:11:41.838 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:11:41.838 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:11:41.838 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:04:43.320062+0000", 2024-09-18T08:11:41.838 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.00023171800000000001 2024-09-18T08:11:41.838 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:11:41.838 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:11:41.838 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "commit_sent", 2024-09-18T08:11:41.838 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:04:43.320107+0000", 2024-09-18T08:11:41.838 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 4.5578999999999998e-05 2024-09-18T08:11:41.838 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:11:41.838 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:11:41.838 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "done", 2024-09-18T08:11:41.838 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:04:43.320146+0000", 2024-09-18T08:11:41.839 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 3.8600000000000003e-05 2024-09-18T08:11:41.839 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-09-18T08:11:41.839 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-09-18T08:11:41.839 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-09-18T08:11:41.839 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:11:41.839 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:11:41.839 INFO:teuthology.orchestra.run.smithi007.stdout: "description": "osd_op(client.14805.0:65105 7.7 7:e54b6e92:::benchmark_data_smithi007_73155_object65104:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e101)", 2024-09-18T08:11:41.839 INFO:teuthology.orchestra.run.smithi007.stdout: "initiated_at": "2024-09-18T08:04:42.341572+0000", 2024-09-18T08:11:41.839 INFO:teuthology.orchestra.run.smithi007.stdout: "age": 419.45231982500002, 2024-09-18T08:11:41.839 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.99348088999999995, 2024-09-18T08:11:41.839 INFO:teuthology.orchestra.run.smithi007.stdout: "continuous": false, 2024-09-18T08:11:41.839 INFO:teuthology.orchestra.run.smithi007.stdout: "type_data": { 2024-09-18T08:11:41.839 INFO:teuthology.orchestra.run.smithi007.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T08:11:41.839 INFO:teuthology.orchestra.run.smithi007.stdout: "client_info": { 2024-09-18T08:11:41.840 INFO:teuthology.orchestra.run.smithi007.stdout: "client": "client.14805", 2024-09-18T08:11:41.840 INFO:teuthology.orchestra.run.smithi007.stdout: "client_addr": "172.21.15.7:0/640320436", 2024-09-18T08:11:41.840 INFO:teuthology.orchestra.run.smithi007.stdout: "tid": 65105 2024-09-18T08:11:41.840 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:11:41.840 INFO:teuthology.orchestra.run.smithi007.stdout: "events": [ 2024-09-18T08:11:41.840 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:11:41.840 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "initiated", 2024-09-18T08:11:41.840 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:04:42.341572+0000", 2024-09-18T08:11:41.840 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-09-18T08:11:41.840 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:11:41.840 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:11:41.840 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "header_read", 2024-09-18T08:11:41.840 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:04:42.341572+0000", 2024-09-18T08:11:41.840 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-09-18T08:11:41.841 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:11:41.841 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:11:41.841 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "throttled", 2024-09-18T08:11:41.841 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:04:42.341574+0000", 2024-09-18T08:11:41.841 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.6080000000000001e-06 2024-09-18T08:11:41.841 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:11:41.841 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:11:41.841 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "all_read", 2024-09-18T08:11:41.841 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:04:42.341597+0000", 2024-09-18T08:11:41.841 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 2.2691999999999999e-05 2024-09-18T08:11:41.841 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:11:41.841 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:11:41.841 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "dispatched", 2024-09-18T08:11:41.842 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:04:42.341612+0000", 2024-09-18T08:11:41.842 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.5775000000000001e-05 2024-09-18T08:11:41.842 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:11:41.842 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:11:41.842 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "queued_for_pg", 2024-09-18T08:11:41.842 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:04:42.341622+0000", 2024-09-18T08:11:41.842 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 9.329e-06 2024-09-18T08:11:41.842 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:11:41.842 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:11:41.842 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-09-18T08:11:41.842 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:04:42.341694+0000", 2024-09-18T08:11:41.842 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 7.2757999999999997e-05 2024-09-18T08:11:41.842 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:11:41.842 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:11:41.842 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "waiting for peered", 2024-09-18T08:11:41.843 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:04:42.341706+0000", 2024-09-18T08:11:41.843 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.2075999999999999e-05 2024-09-18T08:11:41.843 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:11:41.843 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:11:41.843 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-09-18T08:11:41.843 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:04:43.315804+0000", 2024-09-18T08:11:41.843 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.97409722899999995 2024-09-18T08:11:41.843 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:11:41.843 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:11:41.843 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "started", 2024-09-18T08:11:41.843 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:04:43.315915+0000", 2024-09-18T08:11:41.843 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.000111437 2024-09-18T08:11:41.843 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:11:41.844 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:11:41.844 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "waiting for subops from 3,6", 2024-09-18T08:11:41.844 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:04:43.316035+0000", 2024-09-18T08:11:41.844 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.00011955000000000001 2024-09-18T08:11:41.844 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:11:41.844 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:11:41.844 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "op_commit", 2024-09-18T08:11:41.844 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:04:43.319834+0000", 2024-09-18T08:11:41.844 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.0037993900000000001 2024-09-18T08:11:41.844 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:11:41.844 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:11:41.844 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:11:41.844 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:04:43.320021+0000", 2024-09-18T08:11:41.844 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.00018648499999999999 2024-09-18T08:11:41.845 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:11:41.845 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:11:41.845 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:11:41.845 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:04:43.334952+0000", 2024-09-18T08:11:41.845 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.014931275000000001 2024-09-18T08:11:41.845 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:11:41.845 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:11:41.845 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "commit_sent", 2024-09-18T08:11:41.845 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:04:43.335007+0000", 2024-09-18T08:11:41.846 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 5.4849000000000003e-05 2024-09-18T08:11:41.846 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:11:41.846 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:11:41.846 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "done", 2024-09-18T08:11:41.846 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:04:43.335053+0000", 2024-09-18T08:11:41.846 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 4.6437000000000003e-05 2024-09-18T08:11:41.846 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-09-18T08:11:41.846 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-09-18T08:11:41.846 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-09-18T08:11:41.846 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:11:41.846 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:11:41.846 INFO:teuthology.orchestra.run.smithi007.stdout: "description": "osd_op(client.14805.0:65108 7.7 7:ef77f963:::benchmark_data_smithi007_73155_object65107:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e101)", 2024-09-18T08:11:41.846 INFO:teuthology.orchestra.run.smithi007.stdout: "initiated_at": "2024-09-18T08:04:42.343962+0000", 2024-09-18T08:11:41.847 INFO:teuthology.orchestra.run.smithi007.stdout: "age": 419.44993039899998, 2024-09-18T08:11:41.847 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.99132680500000003, 2024-09-18T08:11:41.847 INFO:teuthology.orchestra.run.smithi007.stdout: "continuous": false, 2024-09-18T08:11:41.847 INFO:teuthology.orchestra.run.smithi007.stdout: "type_data": { 2024-09-18T08:11:41.847 INFO:teuthology.orchestra.run.smithi007.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T08:11:41.847 INFO:teuthology.orchestra.run.smithi007.stdout: "client_info": { 2024-09-18T08:11:41.847 INFO:teuthology.orchestra.run.smithi007.stdout: "client": "client.14805", 2024-09-18T08:11:41.847 INFO:teuthology.orchestra.run.smithi007.stdout: "client_addr": "172.21.15.7:0/640320436", 2024-09-18T08:11:41.847 INFO:teuthology.orchestra.run.smithi007.stdout: "tid": 65108 2024-09-18T08:11:41.847 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:11:41.847 INFO:teuthology.orchestra.run.smithi007.stdout: "events": [ 2024-09-18T08:11:41.847 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:11:41.847 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "initiated", 2024-09-18T08:11:41.847 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:04:42.343962+0000", 2024-09-18T08:11:41.848 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-09-18T08:11:41.848 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:11:41.848 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:11:41.848 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "header_read", 2024-09-18T08:11:41.848 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:04:42.343962+0000", 2024-09-18T08:11:41.848 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-09-18T08:11:41.848 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:11:41.848 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:11:41.848 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "throttled", 2024-09-18T08:11:41.848 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:04:42.343963+0000", 2024-09-18T08:11:41.848 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.531e-06 2024-09-18T08:11:41.848 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:11:41.848 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:11:41.848 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "all_read", 2024-09-18T08:11:41.849 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:04:42.343989+0000", 2024-09-18T08:11:41.849 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 2.5735999999999999e-05 2024-09-18T08:11:41.849 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:11:41.849 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:11:41.849 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "dispatched", 2024-09-18T08:11:41.849 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:04:42.344001+0000", 2024-09-18T08:11:41.849 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.219e-05 2024-09-18T08:11:41.849 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:11:41.849 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:11:41.849 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "queued_for_pg", 2024-09-18T08:11:41.849 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:04:42.344009+0000", 2024-09-18T08:11:41.849 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 7.6420000000000007e-06 2024-09-18T08:11:41.849 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:11:41.850 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:11:41.850 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-09-18T08:11:41.850 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:04:42.344058+0000", 2024-09-18T08:11:41.850 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 4.9076999999999998e-05 2024-09-18T08:11:41.850 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:11:41.850 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:11:41.850 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "waiting for peered", 2024-09-18T08:11:41.850 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:04:42.344064+0000", 2024-09-18T08:11:41.850 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 6.1240000000000002e-06 2024-09-18T08:11:41.850 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:11:41.850 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:11:41.850 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-09-18T08:11:41.850 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:04:43.320175+0000", 2024-09-18T08:11:41.851 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.97611146900000001 2024-09-18T08:11:41.851 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:11:41.851 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:11:41.851 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "started", 2024-09-18T08:11:41.851 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:04:43.320286+0000", 2024-09-18T08:11:41.851 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.000110221 2024-09-18T08:11:41.851 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:11:41.851 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:11:41.851 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "waiting for subops from 3,6", 2024-09-18T08:11:41.851 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:04:43.320416+0000", 2024-09-18T08:11:41.851 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.00013077100000000001 2024-09-18T08:11:41.851 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:11:41.851 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:11:41.852 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:11:41.852 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:04:43.325654+0000", 2024-09-18T08:11:41.852 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.0052372850000000004 2024-09-18T08:11:41.852 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:11:41.852 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:11:41.852 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "op_commit", 2024-09-18T08:11:41.852 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:04:43.332014+0000", 2024-09-18T08:11:41.852 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.0063605989999999998 2024-09-18T08:11:41.852 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:11:41.852 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:11:41.852 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:11:41.852 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:04:43.335151+0000", 2024-09-18T08:11:41.852 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.0031363960000000001 2024-09-18T08:11:41.852 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:11:41.853 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:11:41.853 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "commit_sent", 2024-09-18T08:11:41.853 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:04:43.335241+0000", 2024-09-18T08:11:41.853 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 9.0564999999999998e-05 2024-09-18T08:11:41.853 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:11:41.853 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:11:41.853 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "done", 2024-09-18T08:11:41.853 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:04:43.335288+0000", 2024-09-18T08:11:41.853 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 4.7199e-05 2024-09-18T08:11:41.853 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-09-18T08:11:41.854 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-09-18T08:11:41.854 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-09-18T08:11:41.854 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:11:41.854 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:11:41.854 INFO:teuthology.orchestra.run.smithi007.stdout: "description": "osd_op(client.14805.0:65121 7.7 7:e3b4a34b:::benchmark_data_smithi007_73155_object65120:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e101)", 2024-09-18T08:11:41.854 INFO:teuthology.orchestra.run.smithi007.stdout: "initiated_at": "2024-09-18T08:04:42.359562+0000", 2024-09-18T08:11:41.854 INFO:teuthology.orchestra.run.smithi007.stdout: "age": 419.43432980599999, 2024-09-18T08:11:41.854 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.97975838599999998, 2024-09-18T08:11:41.854 INFO:teuthology.orchestra.run.smithi007.stdout: "continuous": false, 2024-09-18T08:11:41.854 INFO:teuthology.orchestra.run.smithi007.stdout: "type_data": { 2024-09-18T08:11:41.854 INFO:teuthology.orchestra.run.smithi007.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T08:11:41.854 INFO:teuthology.orchestra.run.smithi007.stdout: "client_info": { 2024-09-18T08:11:41.854 INFO:teuthology.orchestra.run.smithi007.stdout: "client": "client.14805", 2024-09-18T08:11:41.854 INFO:teuthology.orchestra.run.smithi007.stdout: "client_addr": "172.21.15.7:0/640320436", 2024-09-18T08:11:41.855 INFO:teuthology.orchestra.run.smithi007.stdout: "tid": 65121 2024-09-18T08:11:41.855 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:11:41.855 INFO:teuthology.orchestra.run.smithi007.stdout: "events": [ 2024-09-18T08:11:41.855 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:11:41.855 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "initiated", 2024-09-18T08:11:41.855 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:04:42.359562+0000", 2024-09-18T08:11:41.855 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-09-18T08:11:41.855 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:11:41.855 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:11:41.855 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "header_read", 2024-09-18T08:11:41.855 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:04:42.359562+0000", 2024-09-18T08:11:41.855 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-09-18T08:11:41.855 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:11:41.856 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:11:41.856 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "throttled", 2024-09-18T08:11:41.856 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:04:42.359563+0000", 2024-09-18T08:11:41.856 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 8.6700000000000002e-07 2024-09-18T08:11:41.856 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:11:41.856 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:11:41.856 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "all_read", 2024-09-18T08:11:41.856 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:04:42.359587+0000", 2024-09-18T08:11:41.856 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 2.3782999999999999e-05 2024-09-18T08:11:41.856 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:11:41.856 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:11:41.857 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "dispatched", 2024-09-18T08:11:41.857 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:04:42.359597+0000", 2024-09-18T08:11:41.857 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 9.9660000000000008e-06 2024-09-18T08:11:41.857 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:11:41.857 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:11:41.857 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "queued_for_pg", 2024-09-18T08:11:41.857 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:04:42.359606+0000", 2024-09-18T08:11:41.857 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 9.4120000000000007e-06 2024-09-18T08:11:41.857 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:11:41.857 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:11:41.857 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-09-18T08:11:41.857 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:04:42.359663+0000", 2024-09-18T08:11:41.857 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 5.7113e-05 2024-09-18T08:11:41.858 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:11:41.858 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:11:41.858 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "waiting for peered", 2024-09-18T08:11:41.858 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:04:42.359671+0000", 2024-09-18T08:11:41.858 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 7.5159999999999997e-06 2024-09-18T08:11:41.858 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:11:41.858 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:11:41.858 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-09-18T08:11:41.858 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:04:43.322602+0000", 2024-09-18T08:11:41.858 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.96293140099999996 2024-09-18T08:11:41.858 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:11:41.858 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:11:41.858 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "started", 2024-09-18T08:11:41.858 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:04:43.322715+0000", 2024-09-18T08:11:41.859 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.000113027 2024-09-18T08:11:41.859 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:11:41.859 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:11:41.859 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "waiting for subops from 3,6", 2024-09-18T08:11:41.859 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:04:43.322871+0000", 2024-09-18T08:11:41.859 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.00015569299999999999 2024-09-18T08:11:41.859 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:11:41.859 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:11:41.859 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:11:41.859 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:04:43.327357+0000", 2024-09-18T08:11:41.859 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.0044856920000000003 2024-09-18T08:11:41.859 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:11:41.859 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:11:41.859 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "op_commit", 2024-09-18T08:11:41.860 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:04:43.332022+0000", 2024-09-18T08:11:41.860 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.0046651119999999999 2024-09-18T08:11:41.860 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:11:41.860 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:11:41.860 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:11:41.860 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:04:43.339232+0000", 2024-09-18T08:11:41.860 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.0072099140000000004 2024-09-18T08:11:41.860 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:11:41.860 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:11:41.860 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "commit_sent", 2024-09-18T08:11:41.860 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:04:43.339284+0000", 2024-09-18T08:11:41.860 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 5.1975999999999998e-05 2024-09-18T08:11:41.860 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:11:41.861 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:11:41.861 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "done", 2024-09-18T08:11:41.861 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:04:43.339321+0000", 2024-09-18T08:11:41.861 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 3.6913999999999999e-05 2024-09-18T08:11:41.861 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-09-18T08:11:41.861 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-09-18T08:11:41.861 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-09-18T08:11:41.861 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:11:41.861 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:11:41.861 INFO:teuthology.orchestra.run.smithi007.stdout: "description": "osd_op(client.15216.0:58041 8.5 8:af180ac5:::benchmark_data_smithi007_101583_object58040:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e191)", 2024-09-18T08:11:41.861 INFO:teuthology.orchestra.run.smithi007.stdout: "initiated_at": "2024-09-18T08:08:39.201157+0000", 2024-09-18T08:11:41.861 INFO:teuthology.orchestra.run.smithi007.stdout: "age": 182.59273473600001, 2024-09-18T08:11:41.861 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.18427327299999999, 2024-09-18T08:11:41.862 INFO:teuthology.orchestra.run.smithi007.stdout: "continuous": false, 2024-09-18T08:11:41.862 INFO:teuthology.orchestra.run.smithi007.stdout: "type_data": { 2024-09-18T08:11:41.862 INFO:teuthology.orchestra.run.smithi007.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T08:11:41.862 INFO:teuthology.orchestra.run.smithi007.stdout: "client_info": { 2024-09-18T08:11:41.862 INFO:teuthology.orchestra.run.smithi007.stdout: "client": "client.15216", 2024-09-18T08:11:41.862 INFO:teuthology.orchestra.run.smithi007.stdout: "client_addr": "172.21.15.7:0/2596041177", 2024-09-18T08:11:41.862 INFO:teuthology.orchestra.run.smithi007.stdout: "tid": 58041 2024-09-18T08:11:41.862 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:11:41.862 INFO:teuthology.orchestra.run.smithi007.stdout: "events": [ 2024-09-18T08:11:41.862 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:11:41.862 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "initiated", 2024-09-18T08:11:41.862 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:39.201157+0000", 2024-09-18T08:11:41.862 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-09-18T08:11:41.862 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:11:41.863 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:11:41.863 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "header_read", 2024-09-18T08:11:41.863 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:39.201157+0000", 2024-09-18T08:11:41.863 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-09-18T08:11:41.863 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:11:41.863 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:11:41.863 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "throttled", 2024-09-18T08:11:41.863 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:39.201158+0000", 2024-09-18T08:11:41.863 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.018e-06 2024-09-18T08:11:41.863 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:11:41.863 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:11:41.863 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "all_read", 2024-09-18T08:11:41.863 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:39.201181+0000", 2024-09-18T08:11:41.863 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 2.2750000000000001e-05 2024-09-18T08:11:41.864 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:11:41.864 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:11:41.864 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "dispatched", 2024-09-18T08:11:41.864 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:39.201188+0000", 2024-09-18T08:11:41.864 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 6.9079999999999998e-06 2024-09-18T08:11:41.864 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:11:41.864 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:11:41.864 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "queued_for_pg", 2024-09-18T08:11:41.864 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:39.201194+0000", 2024-09-18T08:11:41.864 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 5.6749999999999999e-06 2024-09-18T08:11:41.864 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:11:41.864 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:11:41.865 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-09-18T08:11:41.865 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:39.207391+0000", 2024-09-18T08:11:41.865 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.0061970630000000001 2024-09-18T08:11:41.865 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:11:41.865 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:11:41.865 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "started", 2024-09-18T08:11:41.865 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:39.207540+0000", 2024-09-18T08:11:41.865 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.00014970900000000001 2024-09-18T08:11:41.865 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:11:41.865 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:11:41.865 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "waiting for subops from 4,6,7", 2024-09-18T08:11:41.865 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:39.207688+0000", 2024-09-18T08:11:41.865 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.00014783799999999999 2024-09-18T08:11:41.865 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:11:41.866 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:11:41.866 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "op_commit", 2024-09-18T08:11:41.866 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:39.211104+0000", 2024-09-18T08:11:41.866 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.0034154139999999999 2024-09-18T08:11:41.866 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:11:41.866 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:11:41.866 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:11:41.866 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:39.216640+0000", 2024-09-18T08:11:41.866 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.0055363890000000001 2024-09-18T08:11:41.866 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:11:41.866 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:11:41.866 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:11:41.866 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:39.216689+0000", 2024-09-18T08:11:41.866 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 4.905e-05 2024-09-18T08:11:41.867 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:11:41.867 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:11:41.867 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:11:41.867 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:39.383822+0000", 2024-09-18T08:11:41.867 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.167132486 2024-09-18T08:11:41.867 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:11:41.867 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:11:41.867 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "commit_sent", 2024-09-18T08:11:41.867 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:39.383895+0000", 2024-09-18T08:11:41.867 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 7.3714000000000006e-05 2024-09-18T08:11:41.867 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:11:41.867 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:11:41.867 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "done", 2024-09-18T08:11:41.867 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:39.385431+0000", 2024-09-18T08:11:41.868 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.001535259 2024-09-18T08:11:41.868 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-09-18T08:11:41.868 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-09-18T08:11:41.868 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-09-18T08:11:41.868 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:11:41.868 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:11:41.868 INFO:teuthology.orchestra.run.smithi007.stdout: "description": "osd_op(client.15216.0:58051 8.2 8:468f9313:::benchmark_data_smithi007_101583_object58050:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e191)", 2024-09-18T08:11:41.868 INFO:teuthology.orchestra.run.smithi007.stdout: "initiated_at": "2024-09-18T08:08:39.206997+0000", 2024-09-18T08:11:41.868 INFO:teuthology.orchestra.run.smithi007.stdout: "age": 182.58689512800001, 2024-09-18T08:11:41.868 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.19563271199999999, 2024-09-18T08:11:41.868 INFO:teuthology.orchestra.run.smithi007.stdout: "continuous": false, 2024-09-18T08:11:41.868 INFO:teuthology.orchestra.run.smithi007.stdout: "type_data": { 2024-09-18T08:11:41.868 INFO:teuthology.orchestra.run.smithi007.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T08:11:41.869 INFO:teuthology.orchestra.run.smithi007.stdout: "client_info": { 2024-09-18T08:11:41.869 INFO:teuthology.orchestra.run.smithi007.stdout: "client": "client.15216", 2024-09-18T08:11:41.869 INFO:teuthology.orchestra.run.smithi007.stdout: "client_addr": "172.21.15.7:0/2596041177", 2024-09-18T08:11:41.869 INFO:teuthology.orchestra.run.smithi007.stdout: "tid": 58051 2024-09-18T08:11:41.869 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:11:41.869 INFO:teuthology.orchestra.run.smithi007.stdout: "events": [ 2024-09-18T08:11:41.869 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:11:41.869 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "initiated", 2024-09-18T08:11:41.869 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:39.206997+0000", 2024-09-18T08:11:41.869 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-09-18T08:11:41.869 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:11:41.869 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:11:41.869 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "header_read", 2024-09-18T08:11:41.869 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:39.206997+0000", 2024-09-18T08:11:41.870 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-09-18T08:11:41.870 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:11:41.870 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:11:41.870 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "throttled", 2024-09-18T08:11:41.870 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:39.207000+0000", 2024-09-18T08:11:41.870 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 2.6390000000000001e-06 2024-09-18T08:11:41.870 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:11:41.870 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:11:41.870 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "all_read", 2024-09-18T08:11:41.870 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:39.207025+0000", 2024-09-18T08:11:41.870 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 2.5724999999999998e-05 2024-09-18T08:11:41.870 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:11:41.870 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:11:41.870 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "dispatched", 2024-09-18T08:11:41.871 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:39.207038+0000", 2024-09-18T08:11:41.871 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.2537999999999999e-05 2024-09-18T08:11:41.871 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:11:41.871 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:11:41.871 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "queued_for_pg", 2024-09-18T08:11:41.871 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:39.207046+0000", 2024-09-18T08:11:41.871 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 7.9589999999999995e-06 2024-09-18T08:11:41.871 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:11:41.871 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:11:41.871 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-09-18T08:11:41.871 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:39.208412+0000", 2024-09-18T08:11:41.871 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.0013659200000000001 2024-09-18T08:11:41.871 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:11:41.871 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:11:41.872 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "started", 2024-09-18T08:11:41.872 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:39.208538+0000", 2024-09-18T08:11:41.872 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.00012663600000000001 2024-09-18T08:11:41.872 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:11:41.872 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:11:41.872 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "waiting for subops from 4,5", 2024-09-18T08:11:41.872 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:39.208690+0000", 2024-09-18T08:11:41.872 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.00015148900000000001 2024-09-18T08:11:41.872 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:11:41.872 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:11:41.872 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:11:41.872 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:39.383233+0000", 2024-09-18T08:11:41.872 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.17454354 2024-09-18T08:11:41.873 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:11:41.873 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:11:41.873 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:11:41.873 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:39.383283+0000", 2024-09-18T08:11:41.873 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 4.9959000000000001e-05 2024-09-18T08:11:41.873 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:11:41.873 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:11:41.873 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "op_commit", 2024-09-18T08:11:41.873 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:39.400147+0000", 2024-09-18T08:11:41.873 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.016863856999999999 2024-09-18T08:11:41.874 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:11:41.874 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:11:41.874 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "commit_sent", 2024-09-18T08:11:41.874 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:39.400207+0000", 2024-09-18T08:11:41.874 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 5.9825999999999999e-05 2024-09-18T08:11:41.874 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:11:41.874 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:11:41.874 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "done", 2024-09-18T08:11:41.874 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:39.402630+0000", 2024-09-18T08:11:41.874 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.002422624 2024-09-18T08:11:41.874 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-09-18T08:11:41.874 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-09-18T08:11:41.874 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-09-18T08:11:41.875 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:11:41.875 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:11:41.875 INFO:teuthology.orchestra.run.smithi007.stdout: "description": "osd_op(client.15216.0:58052 8.6 8:6a6ca2d5:::benchmark_data_smithi007_101583_object58051:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e191)", 2024-09-18T08:11:41.875 INFO:teuthology.orchestra.run.smithi007.stdout: "initiated_at": "2024-09-18T08:08:39.207662+0000", 2024-09-18T08:11:41.875 INFO:teuthology.orchestra.run.smithi007.stdout: "age": 182.58622978299999, 2024-09-18T08:11:41.875 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.17395756900000001, 2024-09-18T08:11:41.875 INFO:teuthology.orchestra.run.smithi007.stdout: "continuous": false, 2024-09-18T08:11:41.875 INFO:teuthology.orchestra.run.smithi007.stdout: "type_data": { 2024-09-18T08:11:41.875 INFO:teuthology.orchestra.run.smithi007.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T08:11:41.875 INFO:teuthology.orchestra.run.smithi007.stdout: "client_info": { 2024-09-18T08:11:41.875 INFO:teuthology.orchestra.run.smithi007.stdout: "client": "client.15216", 2024-09-18T08:11:41.875 INFO:teuthology.orchestra.run.smithi007.stdout: "client_addr": "172.21.15.7:0/2596041177", 2024-09-18T08:11:41.875 INFO:teuthology.orchestra.run.smithi007.stdout: "tid": 58052 2024-09-18T08:11:41.875 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:11:41.876 INFO:teuthology.orchestra.run.smithi007.stdout: "events": [ 2024-09-18T08:11:41.876 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:11:41.876 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "initiated", 2024-09-18T08:11:41.876 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:39.207662+0000", 2024-09-18T08:11:41.876 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-09-18T08:11:41.876 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:11:41.876 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:11:41.876 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "header_read", 2024-09-18T08:11:41.876 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:39.207662+0000", 2024-09-18T08:11:41.876 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-09-18T08:11:41.876 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:11:41.876 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:11:41.876 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "throttled", 2024-09-18T08:11:41.877 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:39.207663+0000", 2024-09-18T08:11:41.877 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.189e-06 2024-09-18T08:11:41.877 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:11:41.877 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:11:41.877 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "all_read", 2024-09-18T08:11:41.877 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:39.207690+0000", 2024-09-18T08:11:41.877 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 2.6410999999999998e-05 2024-09-18T08:11:41.877 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:11:41.877 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:11:41.877 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "dispatched", 2024-09-18T08:11:41.877 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:39.207701+0000", 2024-09-18T08:11:41.877 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.1365999999999999e-05 2024-09-18T08:11:41.877 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:11:41.877 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:11:41.878 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "queued_for_pg", 2024-09-18T08:11:41.878 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:39.207708+0000", 2024-09-18T08:11:41.878 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 7.1999999999999997e-06 2024-09-18T08:11:41.878 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:11:41.878 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:11:41.878 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-09-18T08:11:41.878 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:39.207758+0000", 2024-09-18T08:11:41.878 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 4.9830999999999998e-05 2024-09-18T08:11:41.878 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:11:41.878 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:11:41.878 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "started", 2024-09-18T08:11:41.878 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:39.207894+0000", 2024-09-18T08:11:41.878 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.00013613000000000001 2024-09-18T08:11:41.878 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:11:41.879 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:11:41.879 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "waiting for subops from 6,7", 2024-09-18T08:11:41.879 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:39.208016+0000", 2024-09-18T08:11:41.879 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.000121795 2024-09-18T08:11:41.879 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:11:41.879 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:11:41.879 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "op_commit", 2024-09-18T08:11:41.879 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:39.212363+0000", 2024-09-18T08:11:41.879 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.0043472609999999998 2024-09-18T08:11:41.879 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:11:41.879 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:11:41.879 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:11:41.879 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:39.213435+0000", 2024-09-18T08:11:41.879 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.001071203 2024-09-18T08:11:41.880 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:11:41.880 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:11:41.880 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:11:41.880 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:39.216765+0000", 2024-09-18T08:11:41.880 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.003330038 2024-09-18T08:11:41.880 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:11:41.880 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:11:41.880 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "commit_sent", 2024-09-18T08:11:41.880 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:39.381577+0000", 2024-09-18T08:11:41.880 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.16481205900000001 2024-09-18T08:11:41.880 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:11:41.880 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:11:41.880 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "done", 2024-09-18T08:11:41.881 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:39.381620+0000", 2024-09-18T08:11:41.881 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 4.3086e-05 2024-09-18T08:11:41.881 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-09-18T08:11:41.881 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-09-18T08:11:41.881 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-09-18T08:11:41.881 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:11:41.881 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:11:41.881 INFO:teuthology.orchestra.run.smithi007.stdout: "description": "MOSDPGPushReply(8.2 191/182 [PushReplyOp(8:4d3513c0:::benchmark_data_smithi007_101583_object4464:head)])", 2024-09-18T08:11:41.881 INFO:teuthology.orchestra.run.smithi007.stdout: "initiated_at": "2024-09-18T08:08:39.210406+0000", 2024-09-18T08:11:41.881 INFO:teuthology.orchestra.run.smithi007.stdout: "age": 182.58348572200001, 2024-09-18T08:11:41.881 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.17277838100000001, 2024-09-18T08:11:41.881 INFO:teuthology.orchestra.run.smithi007.stdout: "continuous": false, 2024-09-18T08:11:41.881 INFO:teuthology.orchestra.run.smithi007.stdout: "type_data": { 2024-09-18T08:11:41.882 INFO:teuthology.orchestra.run.smithi007.stdout: "flag_point": "reached pg", 2024-09-18T08:11:41.882 INFO:teuthology.orchestra.run.smithi007.stdout: "events": [ 2024-09-18T08:11:41.882 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:11:41.882 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "initiated", 2024-09-18T08:11:41.882 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:39.210406+0000", 2024-09-18T08:11:41.882 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-09-18T08:11:41.882 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:11:41.882 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:11:41.882 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "header_read", 2024-09-18T08:11:41.882 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:39.210406+0000", 2024-09-18T08:11:41.882 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-09-18T08:11:41.882 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:11:41.882 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:11:41.882 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "throttled", 2024-09-18T08:11:41.883 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:39.210407+0000", 2024-09-18T08:11:41.883 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 7.3099999999999997e-07 2024-09-18T08:11:41.883 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:11:41.883 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:11:41.883 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "all_read", 2024-09-18T08:11:41.883 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:39.210412+0000", 2024-09-18T08:11:41.883 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 5.2499999999999997e-06 2024-09-18T08:11:41.883 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:11:41.883 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:11:41.883 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "dispatched", 2024-09-18T08:11:41.883 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:39.210421+0000", 2024-09-18T08:11:41.883 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 8.6780000000000006e-06 2024-09-18T08:11:41.883 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:11:41.883 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:11:41.884 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "queued_for_pg", 2024-09-18T08:11:41.884 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:39.210428+0000", 2024-09-18T08:11:41.884 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 7.3459999999999996e-06 2024-09-18T08:11:41.884 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:11:41.884 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:11:41.884 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-09-18T08:11:41.884 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:39.215674+0000", 2024-09-18T08:11:41.884 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.005245792 2024-09-18T08:11:41.884 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:11:41.884 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:11:41.884 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "done", 2024-09-18T08:11:41.884 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:39.383185+0000", 2024-09-18T08:11:41.884 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.16751058399999999 2024-09-18T08:11:41.884 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-09-18T08:11:41.885 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-09-18T08:11:41.885 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-09-18T08:11:41.885 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:11:41.885 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:11:41.885 INFO:teuthology.orchestra.run.smithi007.stdout: "description": "osd_op(client.15216.0:58055 8.c 8:30c437a7:::benchmark_data_smithi007_101583_object58054:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e191)", 2024-09-18T08:11:41.885 INFO:teuthology.orchestra.run.smithi007.stdout: "initiated_at": "2024-09-18T08:08:39.212412+0000", 2024-09-18T08:11:41.885 INFO:teuthology.orchestra.run.smithi007.stdout: "age": 182.58148046299999, 2024-09-18T08:11:41.885 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.168680411, 2024-09-18T08:11:41.885 INFO:teuthology.orchestra.run.smithi007.stdout: "continuous": false, 2024-09-18T08:11:41.885 INFO:teuthology.orchestra.run.smithi007.stdout: "type_data": { 2024-09-18T08:11:41.885 INFO:teuthology.orchestra.run.smithi007.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T08:11:41.885 INFO:teuthology.orchestra.run.smithi007.stdout: "client_info": { 2024-09-18T08:11:41.885 INFO:teuthology.orchestra.run.smithi007.stdout: "client": "client.15216", 2024-09-18T08:11:41.885 INFO:teuthology.orchestra.run.smithi007.stdout: "client_addr": "172.21.15.7:0/2596041177", 2024-09-18T08:11:41.886 INFO:teuthology.orchestra.run.smithi007.stdout: "tid": 58055 2024-09-18T08:11:41.886 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:11:41.886 INFO:teuthology.orchestra.run.smithi007.stdout: "events": [ 2024-09-18T08:11:41.886 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:11:41.886 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "initiated", 2024-09-18T08:11:41.886 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:39.212412+0000", 2024-09-18T08:11:41.886 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-09-18T08:11:41.886 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:11:41.886 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:11:41.886 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "header_read", 2024-09-18T08:11:41.886 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:39.212412+0000", 2024-09-18T08:11:41.886 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-09-18T08:11:41.886 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:11:41.886 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:11:41.887 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "throttled", 2024-09-18T08:11:41.887 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:39.212413+0000", 2024-09-18T08:11:41.887 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.5099999999999999e-06 2024-09-18T08:11:41.887 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:11:41.887 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:11:41.887 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "all_read", 2024-09-18T08:11:41.887 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:39.212438+0000", 2024-09-18T08:11:41.887 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 2.4754000000000001e-05 2024-09-18T08:11:41.887 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:11:41.887 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:11:41.887 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "dispatched", 2024-09-18T08:11:41.887 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:39.212450+0000", 2024-09-18T08:11:41.887 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.1688999999999999e-05 2024-09-18T08:11:41.887 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:11:41.888 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:11:41.888 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "queued_for_pg", 2024-09-18T08:11:41.888 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:39.212458+0000", 2024-09-18T08:11:41.888 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 8.3620000000000006e-06 2024-09-18T08:11:41.888 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:11:41.888 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:11:41.888 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-09-18T08:11:41.888 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:39.212519+0000", 2024-09-18T08:11:41.888 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 6.1494000000000004e-05 2024-09-18T08:11:41.888 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:11:41.888 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:11:41.888 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "started", 2024-09-18T08:11:41.888 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:39.212682+0000", 2024-09-18T08:11:41.889 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.000162273 2024-09-18T08:11:41.889 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:11:41.889 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:11:41.889 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "waiting for subops from 3,7", 2024-09-18T08:11:41.889 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:39.213614+0000", 2024-09-18T08:11:41.889 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.00093235599999999996 2024-09-18T08:11:41.889 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:11:41.889 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:11:41.889 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "op_commit", 2024-09-18T08:11:41.889 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:39.216741+0000", 2024-09-18T08:11:41.889 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.0031266319999999998 2024-09-18T08:11:41.889 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:11:41.889 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:11:41.889 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:11:41.890 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:39.380839+0000", 2024-09-18T08:11:41.890 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.16409826299999999 2024-09-18T08:11:41.890 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:11:41.890 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:11:41.890 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:11:41.890 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:39.380975+0000", 2024-09-18T08:11:41.890 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.00013578799999999999 2024-09-18T08:11:41.890 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:11:41.890 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:11:41.890 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "commit_sent", 2024-09-18T08:11:41.890 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:39.381034+0000", 2024-09-18T08:11:41.890 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 5.9293999999999998e-05 2024-09-18T08:11:41.890 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:11:41.891 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:11:41.891 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "done", 2024-09-18T08:11:41.891 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:39.381092+0000", 2024-09-18T08:11:41.891 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 5.7995999999999998e-05 2024-09-18T08:11:41.891 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-09-18T08:11:41.891 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-09-18T08:11:41.891 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-09-18T08:11:41.891 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:11:41.891 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:11:41.891 INFO:teuthology.orchestra.run.smithi007.stdout: "description": "osd_op(client.15216.0:58057 8.e 8:74910ecc:::benchmark_data_smithi007_101583_object58056:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e191)", 2024-09-18T08:11:41.891 INFO:teuthology.orchestra.run.smithi007.stdout: "initiated_at": "2024-09-18T08:08:39.213320+0000", 2024-09-18T08:11:41.891 INFO:teuthology.orchestra.run.smithi007.stdout: "age": 182.58057204599999, 2024-09-18T08:11:41.891 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.177310562, 2024-09-18T08:11:41.891 INFO:teuthology.orchestra.run.smithi007.stdout: "continuous": false, 2024-09-18T08:11:41.892 INFO:teuthology.orchestra.run.smithi007.stdout: "type_data": { 2024-09-18T08:11:41.892 INFO:teuthology.orchestra.run.smithi007.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T08:11:41.892 INFO:teuthology.orchestra.run.smithi007.stdout: "client_info": { 2024-09-18T08:11:41.892 INFO:teuthology.orchestra.run.smithi007.stdout: "client": "client.15216", 2024-09-18T08:11:41.892 INFO:teuthology.orchestra.run.smithi007.stdout: "client_addr": "172.21.15.7:0/2596041177", 2024-09-18T08:11:41.892 INFO:teuthology.orchestra.run.smithi007.stdout: "tid": 58057 2024-09-18T08:11:41.892 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:11:41.892 INFO:teuthology.orchestra.run.smithi007.stdout: "events": [ 2024-09-18T08:11:41.892 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:11:41.892 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "initiated", 2024-09-18T08:11:41.892 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:39.213320+0000", 2024-09-18T08:11:41.892 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-09-18T08:11:41.892 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:11:41.893 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:11:41.893 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "header_read", 2024-09-18T08:11:41.893 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:39.213320+0000", 2024-09-18T08:11:41.893 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-09-18T08:11:41.893 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:11:41.893 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:11:41.893 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "throttled", 2024-09-18T08:11:41.893 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:39.213321+0000", 2024-09-18T08:11:41.893 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 8.2399999999999997e-07 2024-09-18T08:11:41.893 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:11:41.893 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:11:41.893 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "all_read", 2024-09-18T08:11:41.893 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:39.213340+0000", 2024-09-18T08:11:41.893 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.9609000000000001e-05 2024-09-18T08:11:41.894 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:11:41.894 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:11:41.894 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "dispatched", 2024-09-18T08:11:41.894 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:39.213350+0000", 2024-09-18T08:11:41.894 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 9.859e-06 2024-09-18T08:11:41.894 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:11:41.894 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:11:41.894 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "queued_for_pg", 2024-09-18T08:11:41.894 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:39.213357+0000", 2024-09-18T08:11:41.894 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 6.737e-06 2024-09-18T08:11:41.894 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:11:41.894 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:11:41.894 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-09-18T08:11:41.895 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:39.213470+0000", 2024-09-18T08:11:41.895 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.000112483 2024-09-18T08:11:41.895 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:11:41.895 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:11:41.895 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "started", 2024-09-18T08:11:41.895 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:39.213614+0000", 2024-09-18T08:11:41.895 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.00014436399999999999 2024-09-18T08:11:41.895 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:11:41.895 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:11:41.895 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "waiting for subops from 2,4", 2024-09-18T08:11:41.895 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:39.213759+0000", 2024-09-18T08:11:41.895 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.00014517100000000001 2024-09-18T08:11:41.895 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:11:41.895 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:11:41.896 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:11:41.896 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:39.383459+0000", 2024-09-18T08:11:41.896 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.16970006200000001 2024-09-18T08:11:41.896 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:11:41.896 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:11:41.896 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:11:41.896 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:39.383815+0000", 2024-09-18T08:11:41.896 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.00035547699999999998 2024-09-18T08:11:41.896 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:11:41.896 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:11:41.896 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "op_commit", 2024-09-18T08:11:41.896 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:39.388873+0000", 2024-09-18T08:11:41.896 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.0050581080000000004 2024-09-18T08:11:41.897 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:11:41.897 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:11:41.897 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "commit_sent", 2024-09-18T08:11:41.897 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:39.388923+0000", 2024-09-18T08:11:41.897 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 5.0120999999999997e-05 2024-09-18T08:11:41.897 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:11:41.897 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:11:41.897 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "done", 2024-09-18T08:11:41.897 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:39.390631+0000", 2024-09-18T08:11:41.897 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.0017077469999999999 2024-09-18T08:11:41.897 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-09-18T08:11:41.897 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-09-18T08:11:41.897 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-09-18T08:11:41.897 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:11:41.898 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:11:41.898 INFO:teuthology.orchestra.run.smithi007.stdout: "description": "osd_repop_reply(client.15216.0:58051 8.2 e191/182 ondisk, result = 0)", 2024-09-18T08:11:41.898 INFO:teuthology.orchestra.run.smithi007.stdout: "initiated_at": "2024-09-18T08:08:39.213393+0000", 2024-09-18T08:11:41.898 INFO:teuthology.orchestra.run.smithi007.stdout: "age": 182.580499524, 2024-09-18T08:11:41.898 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.16985014000000001, 2024-09-18T08:11:41.898 INFO:teuthology.orchestra.run.smithi007.stdout: "continuous": false, 2024-09-18T08:11:41.898 INFO:teuthology.orchestra.run.smithi007.stdout: "type_data": { 2024-09-18T08:11:41.898 INFO:teuthology.orchestra.run.smithi007.stdout: "flag_point": "started", 2024-09-18T08:11:41.898 INFO:teuthology.orchestra.run.smithi007.stdout: "events": [ 2024-09-18T08:11:41.898 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:11:41.898 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "initiated", 2024-09-18T08:11:41.898 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:39.213393+0000", 2024-09-18T08:11:41.898 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-09-18T08:11:41.898 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:11:41.899 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:11:41.899 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "header_read", 2024-09-18T08:11:41.899 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:39.213393+0000", 2024-09-18T08:11:41.899 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-09-18T08:11:41.899 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:11:41.899 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:11:41.899 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "throttled", 2024-09-18T08:11:41.899 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:39.213393+0000", 2024-09-18T08:11:41.899 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 8.1699999999999997e-07 2024-09-18T08:11:41.899 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:11:41.899 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:11:41.899 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "all_read", 2024-09-18T08:11:41.899 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:39.213399+0000", 2024-09-18T08:11:41.899 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 5.823e-06 2024-09-18T08:11:41.899 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:11:41.900 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:11:41.900 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "dispatched", 2024-09-18T08:11:41.900 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:39.213410+0000", 2024-09-18T08:11:41.900 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.0406e-05 2024-09-18T08:11:41.900 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:11:41.900 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:11:41.900 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "queued_for_pg", 2024-09-18T08:11:41.900 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:39.213417+0000", 2024-09-18T08:11:41.900 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 7.041e-06 2024-09-18T08:11:41.900 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:11:41.900 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:11:41.900 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-09-18T08:11:41.900 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:39.383216+0000", 2024-09-18T08:11:41.900 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.169799749 2024-09-18T08:11:41.901 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:11:41.901 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:11:41.901 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "started", 2024-09-18T08:11:41.901 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:39.383226+0000", 2024-09-18T08:11:41.901 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 9.6250000000000002e-06 2024-09-18T08:11:41.901 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:11:41.901 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:11:41.901 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "done", 2024-09-18T08:11:41.901 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:39.383243+0000", 2024-09-18T08:11:41.901 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.6679000000000001e-05 2024-09-18T08:11:41.901 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-09-18T08:11:41.901 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-09-18T08:11:41.901 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-09-18T08:11:41.901 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:11:41.902 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:11:41.902 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6 2024-09-18T08:11:41.903 INFO:teuthology.orchestra.run.smithi007.stdout: "description": "osd_op(client.15216.0:58060 8.6 8:62cd67bd:::benchmark_data_smithi007_101583_object58059:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e191)", 2024-09-18T08:11:41.903 INFO:teuthology.orchestra.run.smithi007.stdout: "initiated_at": "2024-09-18T08:08:39.213567+0000", 2024-09-18T08:11:41.903 INFO:teuthology.orchestra.run.smithi007.stdout: "age": 182.580324808, 2024-09-18T08:11:41.904 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.17373263999999999, 2024-09-18T08:11:41.904 INFO:teuthology.orchestra.run.smithi007.stdout: "continuous": false, 2024-09-18T08:11:41.904 INFO:teuthology.orchestra.run.smithi007.stdout: "type_data": { 2024-09-18T08:11:41.904 INFO:teuthology.orchestra.run.smithi007.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T08:11:41.904 INFO:teuthology.orchestra.run.smithi007.stdout: "client_info": { 2024-09-18T08:11:41.904 INFO:teuthology.orchestra.run.smithi007.stdout: "client": "client.15216", 2024-09-18T08:11:41.904 INFO:teuthology.orchestra.run.smithi007.stdout: "client_addr": "172.21.15.7:0/2596041177", 2024-09-18T08:11:41.904 INFO:teuthology.orchestra.run.smithi007.stdout: "tid": 58060 2024-09-18T08:11:41.904 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:11:41.904 INFO:teuthology.orchestra.run.smithi007.stdout: "events": [ 2024-09-18T08:11:41.904 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:11:41.904 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "initiated", 2024-09-18T08:11:41.905 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:39.213567+0000", 2024-09-18T08:11:41.905 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-09-18T08:11:41.905 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:11:41.905 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:11:41.905 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "header_read", 2024-09-18T08:11:41.905 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:39.213567+0000", 2024-09-18T08:11:41.905 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-09-18T08:11:41.905 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:11:41.905 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:11:41.905 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "throttled", 2024-09-18T08:11:41.905 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:39.213568+0000", 2024-09-18T08:11:41.905 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 9.540000000000001e-07 2024-09-18T08:11:41.905 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:11:41.906 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:11:41.906 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "all_read", 2024-09-18T08:11:41.906 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:39.213591+0000", 2024-09-18T08:11:41.906 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 2.3087000000000001e-05 2024-09-18T08:11:41.906 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:11:41.906 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:11:41.906 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "dispatched", 2024-09-18T08:11:41.906 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:39.213601+0000", 2024-09-18T08:11:41.906 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.008e-05 2024-09-18T08:11:41.906 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:11:41.906 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:11:41.906 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "queued_for_pg", 2024-09-18T08:11:41.906 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:39.213610+0000", 2024-09-18T08:11:41.906 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 8.5019999999999992e-06 2024-09-18T08:11:41.907 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:11:41.907 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:11:41.907 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-09-18T08:11:41.907 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:39.213663+0000", 2024-09-18T08:11:41.907 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 5.3270000000000003e-05 2024-09-18T08:11:41.907 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:11:41.907 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:11:41.907 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "started", 2024-09-18T08:11:41.907 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:39.213773+0000", 2024-09-18T08:11:41.907 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.000109832 2024-09-18T08:11:41.907 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:11:41.907 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:11:41.907 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "waiting for subops from 6,7", 2024-09-18T08:11:41.907 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:39.213893+0000", 2024-09-18T08:11:41.908 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.000120071 2024-09-18T08:11:41.908 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:11:41.908 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:11:41.908 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:11:41.908 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:39.383386+0000", 2024-09-18T08:11:41.908 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.16949255799999999 2024-09-18T08:11:41.908 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:11:41.908 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:11:41.908 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:11:41.908 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:39.383419+0000", 2024-09-18T08:11:41.908 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 3.3513999999999998e-05 2024-09-18T08:11:41.908 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:11:41.908 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:11:41.908 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "op_commit", 2024-09-18T08:11:41.909 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:39.387208+0000", 2024-09-18T08:11:41.909 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.0037891399999999999 2024-09-18T08:11:41.909 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:11:41.909 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:11:41.909 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "commit_sent", 2024-09-18T08:11:41.909 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:39.387257+0000", 2024-09-18T08:11:41.909 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 4.8931999999999998e-05 2024-09-18T08:11:41.909 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:11:41.909 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:11:41.909 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "done", 2024-09-18T08:11:41.909 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:39.387300+0000", 2024-09-18T08:11:41.909 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 4.2700000000000001e-05 2024-09-18T08:11:41.909 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-09-18T08:11:41.910 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-09-18T08:11:41.910 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-09-18T08:11:41.910 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:11:41.910 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:11:41.910 INFO:teuthology.orchestra.run.smithi007.stdout: "description": "osd_repop_reply(client.15216.0:58051 8.2 e191/182 ondisk, result = 0)", 2024-09-18T08:11:41.910 INFO:teuthology.orchestra.run.smithi007.stdout: "initiated_at": "2024-09-18T08:08:39.216750+0000", 2024-09-18T08:11:41.910 INFO:teuthology.orchestra.run.smithi007.stdout: "age": 182.577141982, 2024-09-18T08:11:41.910 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.16654195399999999, 2024-09-18T08:11:41.910 INFO:teuthology.orchestra.run.smithi007.stdout: "continuous": false, 2024-09-18T08:11:41.910 INFO:teuthology.orchestra.run.smithi007.stdout: "type_data": { 2024-09-18T08:11:41.910 INFO:teuthology.orchestra.run.smithi007.stdout: "flag_point": "started", 2024-09-18T08:11:41.910 INFO:teuthology.orchestra.run.smithi007.stdout: "events": [ 2024-09-18T08:11:41.910 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:11:41.911 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "initiated", 2024-09-18T08:11:41.911 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:39.216750+0000", 2024-09-18T08:11:41.911 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-09-18T08:11:41.911 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:11:41.911 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:11:41.911 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "header_read", 2024-09-18T08:11:41.911 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:39.216750+0000", 2024-09-18T08:11:41.911 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-09-18T08:11:41.911 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:11:41.911 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:11:41.911 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "throttled", 2024-09-18T08:11:41.911 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:39.216751+0000", 2024-09-18T08:11:41.911 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 6.1399999999999997e-07 2024-09-18T08:11:41.911 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:11:41.912 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:11:41.912 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "all_read", 2024-09-18T08:11:41.912 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:39.216755+0000", 2024-09-18T08:11:41.912 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 3.8279999999999996e-06 2024-09-18T08:11:41.912 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:11:41.912 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:11:41.912 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "dispatched", 2024-09-18T08:11:41.912 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:39.216762+0000", 2024-09-18T08:11:41.912 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 7.2980000000000003e-06 2024-09-18T08:11:41.912 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:11:41.912 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:11:41.912 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "queued_for_pg", 2024-09-18T08:11:41.912 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:39.216768+0000", 2024-09-18T08:11:41.913 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 6.1140000000000002e-06 2024-09-18T08:11:41.913 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:11:41.913 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:11:41.913 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-09-18T08:11:41.913 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:39.383268+0000", 2024-09-18T08:11:41.913 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.16650020400000001 2024-09-18T08:11:41.913 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:11:41.913 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:11:41.913 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "started", 2024-09-18T08:11:41.913 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:39.383277+0000", 2024-09-18T08:11:41.913 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 8.5920000000000001e-06 2024-09-18T08:11:41.913 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:11:41.914 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:11:41.914 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "done", 2024-09-18T08:11:41.914 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:39.383292+0000", 2024-09-18T08:11:41.914 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.5304000000000001e-05 2024-09-18T08:11:41.914 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-09-18T08:11:41.914 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-09-18T08:11:41.914 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-09-18T08:11:41.914 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:11:41.914 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:11:41.914 INFO:teuthology.orchestra.run.smithi007.stdout: "description": "osd_repop_reply(client.15216.0:58057 8.e e191/182 ondisk, result = 0)", 2024-09-18T08:11:41.914 INFO:teuthology.orchestra.run.smithi007.stdout: "initiated_at": "2024-09-18T08:08:39.216910+0000", 2024-09-18T08:11:41.914 INFO:teuthology.orchestra.run.smithi007.stdout: "age": 182.57698217999999, 2024-09-18T08:11:41.914 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.16655629999999999, 2024-09-18T08:11:41.914 INFO:teuthology.orchestra.run.smithi007.stdout: "continuous": false, 2024-09-18T08:11:41.915 INFO:teuthology.orchestra.run.smithi007.stdout: "type_data": { 2024-09-18T08:11:41.915 INFO:teuthology.orchestra.run.smithi007.stdout: "flag_point": "started", 2024-09-18T08:11:41.915 INFO:teuthology.orchestra.run.smithi007.stdout: "events": [ 2024-09-18T08:11:41.915 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:11:41.915 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "initiated", 2024-09-18T08:11:41.915 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:39.216910+0000", 2024-09-18T08:11:41.915 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-09-18T08:11:41.915 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:11:41.915 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:11:41.915 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "header_read", 2024-09-18T08:11:41.915 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:39.216910+0000", 2024-09-18T08:11:41.915 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-09-18T08:11:41.915 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:11:41.915 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:11:41.916 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "throttled", 2024-09-18T08:11:41.916 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:39.216911+0000", 2024-09-18T08:11:41.916 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 8.1999999999999998e-07 2024-09-18T08:11:41.916 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:11:41.916 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:11:41.916 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "all_read", 2024-09-18T08:11:41.916 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:39.216915+0000", 2024-09-18T08:11:41.916 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 4.724e-06 2024-09-18T08:11:41.916 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:11:41.916 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:11:41.916 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "dispatched", 2024-09-18T08:11:41.916 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:39.380587+0000", 2024-09-18T08:11:41.916 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.16367198799999999 2024-09-18T08:11:41.917 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:11:41.917 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:11:41.917 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "queued_for_pg", 2024-09-18T08:11:41.917 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:39.380599+0000", 2024-09-18T08:11:41.917 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.1219999999999999e-05 2024-09-18T08:11:41.917 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:11:41.917 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:11:41.917 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-09-18T08:11:41.917 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:39.383446+0000", 2024-09-18T08:11:41.917 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.002847839 2024-09-18T08:11:41.917 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:11:41.917 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:11:41.917 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "started", 2024-09-18T08:11:41.917 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:39.383453+0000", 2024-09-18T08:11:41.918 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 6.9909999999999996e-06 2024-09-18T08:11:41.918 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:11:41.918 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:11:41.918 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "done", 2024-09-18T08:11:41.918 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:39.383466+0000", 2024-09-18T08:11:41.918 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.2717999999999999e-05 2024-09-18T08:11:41.918 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-09-18T08:11:41.918 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-09-18T08:11:41.918 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-09-18T08:11:41.918 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:11:41.918 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:11:41.918 INFO:teuthology.orchestra.run.smithi007.stdout: "description": "osd_op(client.15216.0:58062 8.2 8:4f34fc9d:::benchmark_data_smithi007_101583_object58061:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e191)", 2024-09-18T08:11:41.918 INFO:teuthology.orchestra.run.smithi007.stdout: "initiated_at": "2024-09-18T08:08:39.218220+0000", 2024-09-18T08:11:41.919 INFO:teuthology.orchestra.run.smithi007.stdout: "age": 182.575672129, 2024-09-18T08:11:41.919 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.19349875799999999, 2024-09-18T08:11:41.919 INFO:teuthology.orchestra.run.smithi007.stdout: "continuous": false, 2024-09-18T08:11:41.919 INFO:teuthology.orchestra.run.smithi007.stdout: "type_data": { 2024-09-18T08:11:41.919 INFO:teuthology.orchestra.run.smithi007.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T08:11:41.919 INFO:teuthology.orchestra.run.smithi007.stdout: "client_info": { 2024-09-18T08:11:41.919 INFO:teuthology.orchestra.run.smithi007.stdout: "client": "client.15216", 2024-09-18T08:11:41.919 INFO:teuthology.orchestra.run.smithi007.stdout: "client_addr": "172.21.15.7:0/2596041177", 2024-09-18T08:11:41.919 INFO:teuthology.orchestra.run.smithi007.stdout: "tid": 58062 2024-09-18T08:11:41.919 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:11:41.919 INFO:teuthology.orchestra.run.smithi007.stdout: "events": [ 2024-09-18T08:11:41.919 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:11:41.919 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "initiated", 2024-09-18T08:11:41.920 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:39.218220+0000", 2024-09-18T08:11:41.920 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-09-18T08:11:41.920 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:11:41.920 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:11:41.920 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "header_read", 2024-09-18T08:11:41.920 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:39.218220+0000", 2024-09-18T08:11:41.920 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-09-18T08:11:41.920 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:11:41.920 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:11:41.920 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "throttled", 2024-09-18T08:11:41.920 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:39.218222+0000", 2024-09-18T08:11:41.920 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 2.0260000000000002e-06 2024-09-18T08:11:41.920 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:11:41.920 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:11:41.921 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "all_read", 2024-09-18T08:11:41.921 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:39.218248+0000", 2024-09-18T08:11:41.921 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 2.5814000000000001e-05 2024-09-18T08:11:41.921 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:11:41.921 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:11:41.921 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "dispatched", 2024-09-18T08:11:41.921 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:39.380566+0000", 2024-09-18T08:11:41.921 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.16231814899999999 2024-09-18T08:11:41.921 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:11:41.921 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:11:41.921 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "queued_for_pg", 2024-09-18T08:11:41.921 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:39.380587+0000", 2024-09-18T08:11:41.921 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 2.1452999999999999e-05 2024-09-18T08:11:41.922 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:11:41.922 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:11:41.922 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-09-18T08:11:41.922 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:39.391842+0000", 2024-09-18T08:11:41.922 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.011254846000000001 2024-09-18T08:11:41.922 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:11:41.922 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:11:41.922 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "started", 2024-09-18T08:11:41.922 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:39.392236+0000", 2024-09-18T08:11:41.922 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.00039376100000000001 2024-09-18T08:11:41.922 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:11:41.922 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:11:41.922 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "waiting for subops from 4,5", 2024-09-18T08:11:41.923 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:39.392404+0000", 2024-09-18T08:11:41.923 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.00016846299999999999 2024-09-18T08:11:41.923 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:11:41.923 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:11:41.923 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "op_commit", 2024-09-18T08:11:41.923 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:39.408713+0000", 2024-09-18T08:11:41.923 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.016308458000000001 2024-09-18T08:11:41.923 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:11:41.923 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:11:41.923 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:11:41.923 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:39.408785+0000", 2024-09-18T08:11:41.923 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 7.2548999999999999e-05 2024-09-18T08:11:41.923 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:11:41.924 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:11:41.924 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:11:41.924 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:39.408899+0000", 2024-09-18T08:11:41.924 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.000114028 2024-09-18T08:11:41.924 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:11:41.924 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:11:41.924 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "commit_sent", 2024-09-18T08:11:41.924 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:39.409049+0000", 2024-09-18T08:11:41.924 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.00014910300000000001 2024-09-18T08:11:41.924 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:11:41.924 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:11:41.924 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "done", 2024-09-18T08:11:41.924 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:39.411719+0000", 2024-09-18T08:11:41.925 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.002670108 2024-09-18T08:11:41.925 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-09-18T08:11:41.925 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-09-18T08:11:41.925 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-09-18T08:11:41.925 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:11:41.925 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:11:41.925 INFO:teuthology.orchestra.run.smithi007.stdout: "description": "osd_op(client.15216.0:58285 8.2 8:4c635a5b:::benchmark_data_smithi007_101583_object58284:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e191)", 2024-09-18T08:11:41.925 INFO:teuthology.orchestra.run.smithi007.stdout: "initiated_at": "2024-09-18T08:08:39.561086+0000", 2024-09-18T08:11:41.925 INFO:teuthology.orchestra.run.smithi007.stdout: "age": 182.232805821, 2024-09-18T08:11:41.925 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.176084982, 2024-09-18T08:11:41.925 INFO:teuthology.orchestra.run.smithi007.stdout: "continuous": false, 2024-09-18T08:11:41.925 INFO:teuthology.orchestra.run.smithi007.stdout: "type_data": { 2024-09-18T08:11:41.925 INFO:teuthology.orchestra.run.smithi007.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T08:11:41.926 INFO:teuthology.orchestra.run.smithi007.stdout: "client_info": { 2024-09-18T08:11:41.926 INFO:teuthology.orchestra.run.smithi007.stdout: "client": "client.15216", 2024-09-18T08:11:41.926 INFO:teuthology.orchestra.run.smithi007.stdout: "client_addr": "172.21.15.7:0/2596041177", 2024-09-18T08:11:41.926 INFO:teuthology.orchestra.run.smithi007.stdout: "tid": 58285 2024-09-18T08:11:41.926 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:11:41.926 INFO:teuthology.orchestra.run.smithi007.stdout: "events": [ 2024-09-18T08:11:41.926 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:11:41.926 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "initiated", 2024-09-18T08:11:41.926 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:39.561086+0000", 2024-09-18T08:11:41.926 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-09-18T08:11:41.926 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:11:41.926 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:11:41.926 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "header_read", 2024-09-18T08:11:41.927 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:39.561086+0000", 2024-09-18T08:11:41.927 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-09-18T08:11:41.927 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:11:41.927 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:11:41.927 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "throttled", 2024-09-18T08:11:41.927 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:39.561088+0000", 2024-09-18T08:11:41.927 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.4640000000000001e-06 2024-09-18T08:11:41.927 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:11:41.927 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:11:41.927 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "all_read", 2024-09-18T08:11:41.927 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:39.561113+0000", 2024-09-18T08:11:41.927 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 2.5381999999999999e-05 2024-09-18T08:11:41.927 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:11:41.927 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:11:41.928 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "dispatched", 2024-09-18T08:11:41.928 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:39.561124+0000", 2024-09-18T08:11:41.928 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.1097999999999999e-05 2024-09-18T08:11:41.928 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:11:41.928 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:11:41.928 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "queued_for_pg", 2024-09-18T08:11:41.928 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:39.561131+0000", 2024-09-18T08:11:41.928 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 7.2710000000000003e-06 2024-09-18T08:11:41.928 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:11:41.928 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:11:41.928 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-09-18T08:11:41.928 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:39.572658+0000", 2024-09-18T08:11:41.928 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.011526586 2024-09-18T08:11:41.928 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:11:41.929 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:11:41.929 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "started", 2024-09-18T08:11:41.929 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:39.572791+0000", 2024-09-18T08:11:41.929 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.00013306799999999999 2024-09-18T08:11:41.929 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:11:41.929 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:11:41.929 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "waiting for subops from 4,5", 2024-09-18T08:11:41.929 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:39.573341+0000", 2024-09-18T08:11:41.929 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.00054952400000000002 2024-09-18T08:11:41.929 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:11:41.929 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:11:41.929 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:11:41.929 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:39.583006+0000", 2024-09-18T08:11:41.930 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.0096658060000000007 2024-09-18T08:11:41.930 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:11:41.930 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:11:41.930 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:11:41.930 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:39.583051+0000", 2024-09-18T08:11:41.930 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 4.4255999999999998e-05 2024-09-18T08:11:41.930 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:11:41.930 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:11:41.930 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "op_commit", 2024-09-18T08:11:41.930 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:39.734703+0000", 2024-09-18T08:11:41.930 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.151652711 2024-09-18T08:11:41.930 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:11:41.930 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:11:41.930 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "commit_sent", 2024-09-18T08:11:41.931 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:39.735334+0000", 2024-09-18T08:11:41.931 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.00063036799999999997 2024-09-18T08:11:41.931 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:11:41.931 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:11:41.931 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "done", 2024-09-18T08:11:41.931 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:39.737171+0000", 2024-09-18T08:11:41.931 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.001837448 2024-09-18T08:11:41.931 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-09-18T08:11:41.931 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-09-18T08:11:41.931 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-09-18T08:11:41.931 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:11:41.931 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:11:41.931 INFO:teuthology.orchestra.run.smithi007.stdout: "description": "osd_op(client.15216.0:58290 8.2 8:4e5d59fe:::benchmark_data_smithi007_101583_object58289:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e191)", 2024-09-18T08:11:41.931 INFO:teuthology.orchestra.run.smithi007.stdout: "initiated_at": "2024-09-18T08:08:39.562276+0000", 2024-09-18T08:11:41.932 INFO:teuthology.orchestra.run.smithi007.stdout: "age": 182.23161642100001, 2024-09-18T08:11:41.932 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.17848797499999999, 2024-09-18T08:11:41.932 INFO:teuthology.orchestra.run.smithi007.stdout: "continuous": false, 2024-09-18T08:11:41.932 INFO:teuthology.orchestra.run.smithi007.stdout: "type_data": { 2024-09-18T08:11:41.932 INFO:teuthology.orchestra.run.smithi007.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T08:11:41.932 INFO:teuthology.orchestra.run.smithi007.stdout: "client_info": { 2024-09-18T08:11:41.932 INFO:teuthology.orchestra.run.smithi007.stdout: "client": "client.15216", 2024-09-18T08:11:41.932 INFO:teuthology.orchestra.run.smithi007.stdout: "client_addr": "172.21.15.7:0/2596041177", 2024-09-18T08:11:41.932 INFO:teuthology.orchestra.run.smithi007.stdout: "tid": 58290 2024-09-18T08:11:41.932 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:11:41.932 INFO:teuthology.orchestra.run.smithi007.stdout: "events": [ 2024-09-18T08:11:41.932 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:11:41.932 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "initiated", 2024-09-18T08:11:41.933 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:39.562276+0000", 2024-09-18T08:11:41.933 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-09-18T08:11:41.933 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:11:41.933 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:11:41.933 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "header_read", 2024-09-18T08:11:41.933 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:39.562276+0000", 2024-09-18T08:11:41.933 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-09-18T08:11:41.933 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:11:41.933 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:11:41.933 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "throttled", 2024-09-18T08:11:41.933 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:39.562277+0000", 2024-09-18T08:11:41.933 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.452e-06 2024-09-18T08:11:41.934 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:11:41.934 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:11:41.934 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "all_read", 2024-09-18T08:11:41.934 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:39.562298+0000", 2024-09-18T08:11:41.934 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 2.0826e-05 2024-09-18T08:11:41.934 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:11:41.934 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:11:41.934 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "dispatched", 2024-09-18T08:11:41.934 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:39.562314+0000", 2024-09-18T08:11:41.934 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.573e-05 2024-09-18T08:11:41.934 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:11:41.934 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:11:41.934 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "queued_for_pg", 2024-09-18T08:11:41.934 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:39.562321+0000", 2024-09-18T08:11:41.935 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 7.3520000000000001e-06 2024-09-18T08:11:41.935 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:11:41.935 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:11:41.935 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-09-18T08:11:41.935 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:39.575844+0000", 2024-09-18T08:11:41.935 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.013522798000000001 2024-09-18T08:11:41.935 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:11:41.935 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:11:41.935 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "started", 2024-09-18T08:11:41.935 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:39.576013+0000", 2024-09-18T08:11:41.935 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.000169555 2024-09-18T08:11:41.935 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:11:41.935 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:11:41.936 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "waiting for subops from 4,5", 2024-09-18T08:11:41.936 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:39.576163+0000", 2024-09-18T08:11:41.936 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.000149977 2024-09-18T08:11:41.936 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:11:41.936 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:11:41.936 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:11:41.936 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:39.583090+0000", 2024-09-18T08:11:41.936 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.0069266340000000001 2024-09-18T08:11:41.936 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:11:41.936 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:11:41.936 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:11:41.936 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:39.583129+0000", 2024-09-18T08:11:41.936 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 3.8769000000000001e-05 2024-09-18T08:11:41.937 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:11:41.937 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:11:41.937 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "op_commit", 2024-09-18T08:11:41.937 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:39.739472+0000", 2024-09-18T08:11:41.937 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.15634289800000001 2024-09-18T08:11:41.937 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:11:41.937 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:11:41.937 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "commit_sent", 2024-09-18T08:11:41.937 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:39.739541+0000", 2024-09-18T08:11:41.937 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 6.9728999999999996e-05 2024-09-18T08:11:41.937 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:11:41.937 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:11:41.937 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "done", 2024-09-18T08:11:41.937 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:39.740764+0000", 2024-09-18T08:11:41.938 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.001222255 2024-09-18T08:11:41.938 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-09-18T08:11:41.938 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-09-18T08:11:41.938 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-09-18T08:11:41.938 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:11:41.938 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:11:41.938 INFO:teuthology.orchestra.run.smithi007.stdout: "description": "MOSDPGPushReply(8.2 191/182 [PushReplyOp(8:4ab5a84b:::benchmark_data_smithi007_101583_object4602:head)])", 2024-09-18T08:11:41.938 INFO:teuthology.orchestra.run.smithi007.stdout: "initiated_at": "2024-09-18T08:08:39.568444+0000", 2024-09-18T08:11:41.938 INFO:teuthology.orchestra.run.smithi007.stdout: "age": 182.22544797800001, 2024-09-18T08:11:41.938 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.168734621, 2024-09-18T08:11:41.938 INFO:teuthology.orchestra.run.smithi007.stdout: "continuous": false, 2024-09-18T08:11:41.938 INFO:teuthology.orchestra.run.smithi007.stdout: "type_data": { 2024-09-18T08:11:41.938 INFO:teuthology.orchestra.run.smithi007.stdout: "flag_point": "reached pg", 2024-09-18T08:11:41.939 INFO:teuthology.orchestra.run.smithi007.stdout: "events": [ 2024-09-18T08:11:41.939 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:11:41.939 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "initiated", 2024-09-18T08:11:41.939 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:39.568444+0000", 2024-09-18T08:11:41.939 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-09-18T08:11:41.939 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:11:41.939 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:11:41.939 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "header_read", 2024-09-18T08:11:41.939 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:39.568444+0000", 2024-09-18T08:11:41.939 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-09-18T08:11:41.939 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:11:41.939 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:11:41.939 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "throttled", 2024-09-18T08:11:41.939 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:39.568445+0000", 2024-09-18T08:11:41.940 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 5.13e-07 2024-09-18T08:11:41.940 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:11:41.940 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:11:41.940 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "all_read", 2024-09-18T08:11:41.940 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:39.568450+0000", 2024-09-18T08:11:41.940 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 5.0209999999999999e-06 2024-09-18T08:11:41.940 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:11:41.940 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:11:41.940 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "dispatched", 2024-09-18T08:11:41.940 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:39.568457+0000", 2024-09-18T08:11:41.940 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 7.841e-06 2024-09-18T08:11:41.940 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:11:41.940 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:11:41.940 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "queued_for_pg", 2024-09-18T08:11:41.941 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:39.568465+0000", 2024-09-18T08:11:41.941 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 7.7060000000000004e-06 2024-09-18T08:11:41.941 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:11:41.941 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:11:41.941 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-09-18T08:11:41.941 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:39.579487+0000", 2024-09-18T08:11:41.941 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.011021665 2024-09-18T08:11:41.941 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:11:41.941 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:11:41.941 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "done", 2024-09-18T08:11:41.941 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:39.737179+0000", 2024-09-18T08:11:41.941 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.15769187500000001 2024-09-18T08:11:41.941 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-09-18T08:11:41.941 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-09-18T08:11:41.942 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-09-18T08:11:41.942 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-09-18T08:11:41.942 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-09-18T08:11:41.942 INFO:teuthology.orchestra.run.smithi007.stdout:} 2024-09-18T08:11:41.943 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:11:41.943 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:11:41.943 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6' 2024-09-18T08:11:42.044 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3 2024-09-18T08:11:42.068 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:11:42.068 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:11:42.069 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3' 2024-09-18T08:11:42.141 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid 779349ca-7593-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-18T08:11:42.170 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4 2024-09-18T08:11:42.189 INFO:tasks.radosbench.radosbench.0.smithi007.stdout: 27 16 53579 53563 123.956 10.8125 0.038062 0.0078041 2024-09-18T08:11:42.192 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:11:42.193 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:11:42.193 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4' 2024-09-18T08:11:42.294 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7 2024-09-18T08:11:42.317 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:11:42.318 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:11:42.318 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7' 2024-09-18T08:11:42.328 INFO:teuthology.orchestra.run.smithi007.stderr:nodeep-scrub is set 2024-09-18T08:11:42.419 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5 2024-09-18T08:11:42.457 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:11:42.458 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:11:42.458 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5' 2024-09-18T08:11:42.480 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:11:42 smithi007 ceph-mon[29459]: pgmap v425: 158 pgs: 158 active+clean; 3.1 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 853 B/s rd, 87 MiB/s wr, 1.39k op/s 2024-09-18T08:11:42.481 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:11:42 smithi007 ceph-mon[29459]: Health check failed: 1 osds down (OSD_DOWN) 2024-09-18T08:11:42.481 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:11:42 smithi007 ceph-mon[29459]: osdmap e244: 8 total, 7 up, 8 in 2024-09-18T08:11:42.481 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:11:42 smithi007 ceph-mon[29459]: from='client.? 172.21.15.7:0/1252779293' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-18T08:11:42.481 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:11:42 smithi007 ceph-mon[29459]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-18T08:11:42.481 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:11:42 smithi007 ceph-mon[39249]: pgmap v425: 158 pgs: 158 active+clean; 3.1 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 853 B/s rd, 87 MiB/s wr, 1.39k op/s 2024-09-18T08:11:42.481 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:11:42 smithi007 ceph-mon[39249]: Health check failed: 1 osds down (OSD_DOWN) 2024-09-18T08:11:42.481 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:11:42 smithi007 ceph-mon[39249]: osdmap e244: 8 total, 7 up, 8 in 2024-09-18T08:11:42.481 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:11:42 smithi007 ceph-mon[39249]: from='client.? 172.21.15.7:0/1252779293' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-18T08:11:42.481 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:11:42 smithi007 ceph-mon[39249]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-18T08:11:42.482 INFO:journalctl@ceph.osd.1.smithi007.stdout:Sep 18 08:11:42 smithi007 podman[126810]: 2024-09-18 08:11:42.158315911 +0000 UTC m=+1.999227876 container died 264f9cbfeda9b1e1a713940fdeee330fe0dd494b7b1e5569645bc1e026bc0228 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:1f5b5a3734f665ef333266f39143ac3ef1eb1951b21258f2307e10332e2d7ac9, name=ceph-779349ca-7593-11ef-bceb-c7b262605968-osd-1, io.buildah.version=1.37.2, GIT_CLEAN=True, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-guits-main-2024-09-17-1213-e7fb7b5, ceph=True, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, org.label-schema.build-date=20240731, org.label-schema.name=CentOS Stream 9 Base Image) 2024-09-18T08:11:42.559 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6 2024-09-18T08:11:42.598 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:11:42.598 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:11:42.598 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6' 2024-09-18T08:11:42.656 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:11:42 smithi050 ceph-mon[32371]: pgmap v425: 158 pgs: 158 active+clean; 3.1 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 853 B/s rd, 87 MiB/s wr, 1.39k op/s 2024-09-18T08:11:42.656 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:11:42 smithi050 ceph-mon[32371]: Health check failed: 1 osds down (OSD_DOWN) 2024-09-18T08:11:42.656 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:11:42 smithi050 ceph-mon[32371]: osdmap e244: 8 total, 7 up, 8 in 2024-09-18T08:11:42.656 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:11:42 smithi050 ceph-mon[32371]: from='client.? 172.21.15.7:0/1252779293' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-18T08:11:42.656 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:11:42 smithi050 ceph-mon[32371]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-18T08:11:42.699 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7 2024-09-18T08:11:42.740 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:11:42.741 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:11:42.741 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7' 2024-09-18T08:11:42.842 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0 2024-09-18T08:11:42.877 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:11:42.878 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:11:42.879 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0' 2024-09-18T08:11:42.980 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3 2024-09-18T08:11:43.014 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:11:43.016 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:11:43.016 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3' 2024-09-18T08:11:43.117 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6 2024-09-18T08:11:43.162 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:11:43.162 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:11:43.163 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6' 2024-09-18T08:11:43.189 INFO:tasks.radosbench.radosbench.0.smithi007.stdout: 28 16 55081 55065 122.882 93.875 0.00532322 0.00813268 2024-09-18T08:11:43.230 INFO:journalctl@ceph.osd.1.smithi007.stdout:Sep 18 08:11:42 smithi007 podman[126810]: 2024-09-18 08:11:42.955008185 +0000 UTC m=+2.795920153 container cleanup 264f9cbfeda9b1e1a713940fdeee330fe0dd494b7b1e5569645bc1e026bc0228 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:1f5b5a3734f665ef333266f39143ac3ef1eb1951b21258f2307e10332e2d7ac9, name=ceph-779349ca-7593-11ef-bceb-c7b262605968-osd-1, org.label-schema.build-date=20240731, io.buildah.version=1.37.2, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.license=GPLv2, RELEASE=wip-guits-main-2024-09-17-1213-e7fb7b5, ceph=True) 2024-09-18T08:11:43.230 INFO:journalctl@ceph.osd.1.smithi007.stdout:Sep 18 08:11:42 smithi007 bash[126810]: ceph-779349ca-7593-11ef-bceb-c7b262605968-osd-1 2024-09-18T08:11:43.230 INFO:journalctl@ceph.osd.1.smithi007.stdout:Sep 18 08:11:43 smithi007 podman[127003]: 2024-09-18 08:11:43.167683999 +0000 UTC m=+1.236658982 container remove 264f9cbfeda9b1e1a713940fdeee330fe0dd494b7b1e5569645bc1e026bc0228 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:1f5b5a3734f665ef333266f39143ac3ef1eb1951b21258f2307e10332e2d7ac9, name=ceph-779349ca-7593-11ef-bceb-c7b262605968-osd-1, io.buildah.version=1.37.2, CEPH_POINT_RELEASE=, RELEASE=wip-guits-main-2024-09-17-1213-e7fb7b5, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, ceph=True, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 9 Base Image, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, org.label-schema.build-date=20240731, maintainer=Guillaume Abrioux , GIT_CLEAN=True) 2024-09-18T08:11:43.263 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5 2024-09-18T08:11:43.303 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:11:43.304 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:11:43.304 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5' 2024-09-18T08:11:43.405 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4 2024-09-18T08:11:43.441 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:11:43.442 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:11:43.442 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4' 2024-09-18T08:11:43.543 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4 2024-09-18T08:11:43.591 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:11:43.592 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:11:43.592 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4' 2024-09-18T08:11:43.655 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:11:43 smithi050 ceph-mon[32371]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-18T08:11:43.656 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:11:43 smithi050 ceph-mon[32371]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-18T08:11:43.656 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:11:43 smithi050 ceph-mon[32371]: osdmap e245: 8 total, 7 up, 8 in 2024-09-18T08:11:43.693 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6 2024-09-18T08:11:43.730 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:11:43 smithi007 ceph-mon[29459]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-18T08:11:43.730 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:11:43 smithi007 ceph-mon[29459]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-18T08:11:43.730 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:11:43 smithi007 ceph-mon[29459]: osdmap e245: 8 total, 7 up, 8 in 2024-09-18T08:11:43.730 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:11:43 smithi007 ceph-mon[39249]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-18T08:11:43.730 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:11:43 smithi007 ceph-mon[39249]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-18T08:11:43.731 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:11:43 smithi007 ceph-mon[39249]: osdmap e245: 8 total, 7 up, 8 in 2024-09-18T08:11:43.733 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:11:43.734 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:11:43.734 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6' 2024-09-18T08:11:43.835 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5 2024-09-18T08:11:43.871 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:11:43.871 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:11:43.872 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5' 2024-09-18T08:11:43.972 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4 2024-09-18T08:11:44.010 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:11:44.010 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:11:44.011 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4' 2024-09-18T08:11:44.112 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6 2024-09-18T08:11:44.152 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:11:44.153 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:11:44.153 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6' 2024-09-18T08:11:44.189 INFO:tasks.radosbench.radosbench.0.smithi007.stdout: 29 16 56701 56685 122.136 101.25 0.00764449 0.00818224 2024-09-18T08:11:44.254 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2 2024-09-18T08:11:44.290 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:11:44.291 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:11:44.291 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2' 2024-09-18T08:11:44.392 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5 2024-09-18T08:11:44.431 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:11:44.431 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:11:44.432 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5' 2024-09-18T08:11:44.473 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid 779349ca-7593-11ef-bceb-c7b262605968 -- ceph daemon osd.2 dump_ops_in_flight 2024-09-18T08:11:44.533 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6 2024-09-18T08:11:44.536 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/779349ca-7593-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T08:11:44.575 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:11:44.575 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:11:44.576 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6' 2024-09-18T08:11:44.656 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:11:44 smithi050 ceph-mon[32371]: pgmap v428: 158 pgs: 10 active+undersized, 14 stale+active+clean, 10 active+undersized+degraded, 124 active+clean; 3.1 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 5.0 KiB/s rd, 22 MiB/s wr, 364 op/s; 13885/154800 objects degraded (8.970%); 29 KiB/s, 0 objects/s recovering 2024-09-18T08:11:44.656 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:11:44 smithi050 ceph-mon[32371]: Health check failed: Degraded data redundancy: 13885/154800 objects degraded (8.970%), 10 pgs degraded (PG_DEGRADED) 2024-09-18T08:11:44.656 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:11:44 smithi050 ceph-mon[32371]: osdmap e246: 8 total, 7 up, 8 in 2024-09-18T08:11:44.674 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:11:44 smithi007 ceph-mon[29459]: pgmap v428: 158 pgs: 10 active+undersized, 14 stale+active+clean, 10 active+undersized+degraded, 124 active+clean; 3.1 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 5.0 KiB/s rd, 22 MiB/s wr, 364 op/s; 13885/154800 objects degraded (8.970%); 29 KiB/s, 0 objects/s recovering 2024-09-18T08:11:44.674 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:11:44 smithi007 ceph-mon[29459]: Health check failed: Degraded data redundancy: 13885/154800 objects degraded (8.970%), 10 pgs degraded (PG_DEGRADED) 2024-09-18T08:11:44.674 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:11:44 smithi007 ceph-mon[29459]: osdmap e246: 8 total, 7 up, 8 in 2024-09-18T08:11:44.674 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:11:44 smithi007 ceph-mon[39249]: pgmap v428: 158 pgs: 10 active+undersized, 14 stale+active+clean, 10 active+undersized+degraded, 124 active+clean; 3.1 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 5.0 KiB/s rd, 22 MiB/s wr, 364 op/s; 13885/154800 objects degraded (8.970%); 29 KiB/s, 0 objects/s recovering 2024-09-18T08:11:44.675 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:11:44 smithi007 ceph-mon[39249]: Health check failed: Degraded data redundancy: 13885/154800 objects degraded (8.970%), 10 pgs degraded (PG_DEGRADED) 2024-09-18T08:11:44.675 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:11:44 smithi007 ceph-mon[39249]: osdmap e246: 8 total, 7 up, 8 in 2024-09-18T08:11:44.677 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4 2024-09-18T08:11:44.714 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:11:44.714 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:11:44.715 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4' 2024-09-18T08:11:44.816 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4 2024-09-18T08:11:44.851 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:11:44.851 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:11:44.852 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4' 2024-09-18T08:11:44.952 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3 2024-09-18T08:11:44.993 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:11:44.994 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:11:44.994 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3' 2024-09-18T08:11:45.095 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0 2024-09-18T08:11:45.131 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:11:45.132 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:11:45.133 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0' 2024-09-18T08:11:45.189 INFO:tasks.radosbench.radosbench.0.smithi007.stdout: 30 16 58305 58289 121.406 100.25 0.00534437 0.00823208 2024-09-18T08:11:45.234 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0 2024-09-18T08:11:45.259 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:11:45.260 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:11:45.260 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0' 2024-09-18T08:11:45.361 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7 2024-09-18T08:11:45.397 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:11:45.397 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:11:45.397 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7' 2024-09-18T08:11:45.498 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5 2024-09-18T08:11:45.535 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:11:45.535 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:11:45.536 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5' 2024-09-18T08:11:45.636 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3 2024-09-18T08:11:45.655 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:11:45 smithi050 ceph-mon[32371]: from='client.? 172.21.15.7:0/1075979948' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-18T08:11:45.659 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:11:45.659 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:11:45.660 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3' 2024-09-18T08:11:45.730 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:11:45 smithi007 ceph-mon[29459]: from='client.? 172.21.15.7:0/1075979948' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-18T08:11:45.730 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:11:45 smithi007 ceph-mon[39249]: from='client.? 172.21.15.7:0/1075979948' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-18T08:11:45.761 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2 2024-09-18T08:11:45.795 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:11:45.796 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:11:45.797 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2' 2024-09-18T08:11:45.897 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5 2024-09-18T08:11:45.937 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:11:45.938 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:11:45.938 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5' 2024-09-18T08:11:46.017 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid 779349ca-7593-11ef-bceb-c7b262605968 -- ceph osd unset noscrub 2024-09-18T08:11:46.038 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6 2024-09-18T08:11:46.044 INFO:teuthology.orchestra.run.smithi007.stdout:osd.0: {} 2024-09-18T08:11:46.044 INFO:teuthology.orchestra.run.smithi007.stdout:osd.2: {} 2024-09-18T08:11:46.044 INFO:teuthology.orchestra.run.smithi007.stdout:osd.3: {} 2024-09-18T08:11:46.044 INFO:teuthology.orchestra.run.smithi007.stdout:osd.4: {} 2024-09-18T08:11:46.044 INFO:teuthology.orchestra.run.smithi007.stdout:osd.5: {} 2024-09-18T08:11:46.044 INFO:teuthology.orchestra.run.smithi007.stdout:osd.6: {} 2024-09-18T08:11:46.044 INFO:teuthology.orchestra.run.smithi007.stdout:osd.7: {} 2024-09-18T08:11:46.044 INFO:teuthology.orchestra.run.smithi007.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-18T08:11:46.045 INFO:teuthology.orchestra.run.smithi007.stderr:Error ENXIO: problem getting command descriptions from osd.1 2024-09-18T08:11:46.045 INFO:teuthology.orchestra.run.smithi007.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-18T08:11:46.045 INFO:teuthology.orchestra.run.smithi007.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-18T08:11:46.045 INFO:teuthology.orchestra.run.smithi007.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-18T08:11:46.045 INFO:teuthology.orchestra.run.smithi007.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-18T08:11:46.045 INFO:teuthology.orchestra.run.smithi007.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-18T08:11:46.045 INFO:teuthology.orchestra.run.smithi007.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-18T08:11:46.082 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:11:46.083 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:11:46.083 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6' 2024-09-18T08:11:46.184 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7 2024-09-18T08:11:46.189 INFO:tasks.radosbench.radosbench.0.smithi007.stdout: 31 16 60042 60026 120.992 108.562 0.00563909 0.00825983 2024-09-18T08:11:46.227 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:11:46.228 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:11:46.228 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7' 2024-09-18T08:11:46.329 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0 2024-09-18T08:11:46.389 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:11:46.390 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:11:46.390 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0' 2024-09-18T08:11:46.491 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6 2024-09-18T08:11:46.531 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:11:46.532 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:11:46.532 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6' 2024-09-18T08:11:46.633 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5 2024-09-18T08:11:46.657 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:11:46 smithi050 ceph-mon[32371]: pgmap v430: 158 pgs: 30 active+undersized, 34 active+undersized+degraded, 94 active+clean; 3.4 GiB data, 13 GiB used, 703 GiB / 715 GiB avail; 73 MiB/s wr, 1.16k op/s; 31350/167781 objects degraded (18.685%); 23 KiB/s, 0 objects/s recovering 2024-09-18T08:11:46.677 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:11:46.678 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:11:46.678 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5' 2024-09-18T08:11:46.696 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:11:46 smithi007 ceph-mon[29459]: pgmap v430: 158 pgs: 30 active+undersized, 34 active+undersized+degraded, 94 active+clean; 3.4 GiB data, 13 GiB used, 703 GiB / 715 GiB avail; 73 MiB/s wr, 1.16k op/s; 31350/167781 objects degraded (18.685%); 23 KiB/s, 0 objects/s recovering 2024-09-18T08:11:46.697 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:11:46 smithi007 ceph-mon[39249]: pgmap v430: 158 pgs: 30 active+undersized, 34 active+undersized+degraded, 94 active+clean; 3.4 GiB data, 13 GiB used, 703 GiB / 715 GiB avail; 73 MiB/s wr, 1.16k op/s; 31350/167781 objects degraded (18.685%); 23 KiB/s, 0 objects/s recovering 2024-09-18T08:11:46.778 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3 2024-09-18T08:11:46.813 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:11:46.814 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:11:46.814 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3' 2024-09-18T08:11:46.915 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3 2024-09-18T08:11:46.937 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:11:46.938 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:11:46.938 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3' 2024-09-18T08:11:46.980 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Sep 18 08:11:46 smithi007 ceph-779349ca-7593-11ef-bceb-c7b262605968-mgr-y[29688]: ::ffff:172.21.15.50 - - [18/Sep/2024:08:11:46] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-09-18T08:11:47.039 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6 2024-09-18T08:11:47.076 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:11:47.077 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:11:47.077 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6' 2024-09-18T08:11:47.178 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7 2024-09-18T08:11:47.189 INFO:tasks.radosbench.radosbench.0.smithi007.stdout: 32 16 61689 61673 120.427 102.938 0.010154 0.00829883 2024-09-18T08:11:47.219 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:11:47.220 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:11:47.220 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7' 2024-09-18T08:11:47.321 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2 2024-09-18T08:11:47.354 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:11:47.354 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:11:47.355 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2' 2024-09-18T08:11:47.456 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0 2024-09-18T08:11:47.490 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:11:47.491 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:11:47.491 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0' 2024-09-18T08:11:47.592 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2 2024-09-18T08:11:47.613 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:11:47.614 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:11:47.614 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2' 2024-09-18T08:11:47.715 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6 2024-09-18T08:11:47.761 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:11:47.761 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:11:47.762 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6' 2024-09-18T08:11:47.862 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6 2024-09-18T08:11:47.901 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:11:47.902 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:11:47.903 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6' 2024-09-18T08:11:48.003 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5 2024-09-18T08:11:48.038 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:11:48.038 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:11:48.039 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5' 2024-09-18T08:11:48.047 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/779349ca-7593-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T08:11:48.058 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-09-18T08:11:48.058 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-09-18T08:11:48.112 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/779349ca-7593-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T08:11:48.139 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5 2024-09-18T08:11:48.179 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:11:48.180 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:11:48.180 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5' 2024-09-18T08:11:48.189 INFO:tasks.radosbench.radosbench.0.smithi007.stdout: 33 16 63312 63296 119.852 101.438 0.0085076 0.00833862 2024-09-18T08:11:48.281 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0 2024-09-18T08:11:48.312 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:11:48.313 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:11:48.314 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0' 2024-09-18T08:11:48.415 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3 2024-09-18T08:11:48.442 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:11:48.443 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:11:48.443 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3' 2024-09-18T08:11:48.544 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2 2024-09-18T08:11:48.567 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:11:48.568 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:11:48.568 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2' 2024-09-18T08:11:48.597 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:11:48 smithi007 ceph-mon[39249]: pgmap v431: 158 pgs: 30 active+undersized, 34 active+undersized+degraded, 94 active+clean; 3.4 GiB data, 13 GiB used, 703 GiB / 715 GiB avail; 60 MiB/s wr, 962 op/s; 31350/167781 objects degraded (18.685%); 19 KiB/s, 0 objects/s recovering 2024-09-18T08:11:48.598 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:11:48 smithi007 ceph-mon[29459]: pgmap v431: 158 pgs: 30 active+undersized, 34 active+undersized+degraded, 94 active+clean; 3.4 GiB data, 13 GiB used, 703 GiB / 715 GiB avail; 60 MiB/s wr, 962 op/s; 31350/167781 objects degraded (18.685%); 19 KiB/s, 0 objects/s recovering 2024-09-18T08:11:48.656 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:11:48 smithi050 ceph-mon[32371]: pgmap v431: 158 pgs: 30 active+undersized, 34 active+undersized+degraded, 94 active+clean; 3.4 GiB data, 13 GiB used, 703 GiB / 715 GiB avail; 60 MiB/s wr, 962 op/s; 31350/167781 objects degraded (18.685%); 19 KiB/s, 0 objects/s recovering 2024-09-18T08:11:48.669 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7 2024-09-18T08:11:48.717 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:11:48.717 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:11:48.718 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7' 2024-09-18T08:11:48.818 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5 2024-09-18T08:11:48.852 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:11:48.853 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:11:48.853 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5' 2024-09-18T08:11:48.954 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2 2024-09-18T08:11:49.002 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:11:49.003 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:11:49.003 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2' 2024-09-18T08:11:49.104 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4 2024-09-18T08:11:49.145 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:11:49.146 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:11:49.146 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4' 2024-09-18T08:11:49.189 INFO:teuthology.orchestra.run.smithi007.stdout:ERROR: (22) Invalid argument 2024-09-18T08:11:49.189 INFO:teuthology.orchestra.run.smithi007.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-18T08:11:49.189 INFO:tasks.radosbench.radosbench.0.smithi007.stdout: 34 16 64576 64560 118.65 79 0.00784792 0.00842366 2024-09-18T08:11:49.247 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2 2024-09-18T08:11:49.278 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:11:49.279 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:11:49.279 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2' 2024-09-18T08:11:49.380 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5 2024-09-18T08:11:49.420 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:11:49.420 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:11:49.421 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5' 2024-09-18T08:11:49.480 INFO:journalctl@ceph.osd.1.smithi007.stdout:Sep 18 08:11:49 smithi007 podman[127388]: 2024-09-18 08:11:49.189400094 +0000 UTC m=+2.828555864 container died 85d29fe772450dac157886895aeef69f64f94ace23b308edcfc36a047f0465b8 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e, name=competent_benz, RELEASE=wip-guits-main-2024-09-17-1213-e7fb7b5, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240731, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.vendor=CentOS, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, io.buildah.version=1.37.2, ceph=True, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-09-18T08:11:49.481 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:11:49 smithi007 ceph-mon[29459]: from='mgr.24470 172.21.15.7:0/2594390261' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-18T08:11:49.481 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:11:49 smithi007 ceph-mon[29459]: from='mgr.24470 172.21.15.7:0/2594390261' entity='mgr.y' 2024-09-18T08:11:49.481 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:11:49 smithi007 ceph-mon[29459]: from='mgr.24470 172.21.15.7:0/2594390261' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-18T08:11:49.481 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:11:49 smithi007 ceph-mon[29459]: from='mgr.24470 172.21.15.7:0/2594390261' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-18T08:11:49.481 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:11:49 smithi007 ceph-mon[29459]: from='mgr.24470 172.21.15.7:0/2594390261' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-18T08:11:49.481 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:11:49 smithi007 ceph-mon[29459]: from='mgr.24470 172.21.15.7:0/2594390261' entity='mgr.y' 2024-09-18T08:11:49.481 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:11:49 smithi007 ceph-mon[29459]: Health check update: Degraded data redundancy: 31350/167781 objects degraded (18.685%), 34 pgs degraded (PG_DEGRADED) 2024-09-18T08:11:49.482 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:11:49 smithi007 ceph-mon[29459]: osdmap e247: 8 total, 7 up, 8 in 2024-09-18T08:11:49.482 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:11:49 smithi007 ceph-mon[29459]: from='client.? 172.21.15.7:0/4054511733' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-18T08:11:49.482 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:11:49 smithi007 ceph-mon[39249]: from='mgr.24470 172.21.15.7:0/2594390261' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-18T08:11:49.482 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:11:49 smithi007 ceph-mon[39249]: from='mgr.24470 172.21.15.7:0/2594390261' entity='mgr.y' 2024-09-18T08:11:49.482 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:11:49 smithi007 ceph-mon[39249]: from='mgr.24470 172.21.15.7:0/2594390261' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-18T08:11:49.482 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:11:49 smithi007 ceph-mon[39249]: from='mgr.24470 172.21.15.7:0/2594390261' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-18T08:11:49.482 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:11:49 smithi007 ceph-mon[39249]: from='mgr.24470 172.21.15.7:0/2594390261' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-18T08:11:49.482 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:11:49 smithi007 ceph-mon[39249]: from='mgr.24470 172.21.15.7:0/2594390261' entity='mgr.y' 2024-09-18T08:11:49.482 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:11:49 smithi007 ceph-mon[39249]: Health check update: Degraded data redundancy: 31350/167781 objects degraded (18.685%), 34 pgs degraded (PG_DEGRADED) 2024-09-18T08:11:49.482 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:11:49 smithi007 ceph-mon[39249]: osdmap e247: 8 total, 7 up, 8 in 2024-09-18T08:11:49.482 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:11:49 smithi007 ceph-mon[39249]: from='client.? 172.21.15.7:0/4054511733' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-18T08:11:49.522 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4 2024-09-18T08:11:49.560 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:11:49.561 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:11:49.562 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4' 2024-09-18T08:11:49.656 INFO:journalctl@ceph.iscsi.iscsi.a.smithi050.stdout:Sep 18 08:11:49 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-iscsi-iscsi-a[57782]: debug there is no tcmu-runner data available 2024-09-18T08:11:49.656 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:11:49 smithi050 ceph-mon[32371]: from='mgr.24470 172.21.15.7:0/2594390261' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-18T08:11:49.656 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:11:49 smithi050 ceph-mon[32371]: from='mgr.24470 172.21.15.7:0/2594390261' entity='mgr.y' 2024-09-18T08:11:49.656 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:11:49 smithi050 ceph-mon[32371]: from='mgr.24470 172.21.15.7:0/2594390261' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-18T08:11:49.656 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:11:49 smithi050 ceph-mon[32371]: from='mgr.24470 172.21.15.7:0/2594390261' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-18T08:11:49.656 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:11:49 smithi050 ceph-mon[32371]: from='mgr.24470 172.21.15.7:0/2594390261' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-18T08:11:49.656 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:11:49 smithi050 ceph-mon[32371]: from='mgr.24470 172.21.15.7:0/2594390261' entity='mgr.y' 2024-09-18T08:11:49.657 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:11:49 smithi050 ceph-mon[32371]: Health check update: Degraded data redundancy: 31350/167781 objects degraded (18.685%), 34 pgs degraded (PG_DEGRADED) 2024-09-18T08:11:49.657 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:11:49 smithi050 ceph-mon[32371]: osdmap e247: 8 total, 7 up, 8 in 2024-09-18T08:11:49.657 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:11:49 smithi050 ceph-mon[32371]: from='client.? 172.21.15.7:0/4054511733' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-18T08:11:49.662 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2 2024-09-18T08:11:49.700 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:11:49.700 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:11:49.700 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2' 2024-09-18T08:11:49.801 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6 2024-09-18T08:11:49.839 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:11:49.839 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:11:49.840 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6' 2024-09-18T08:11:49.941 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3 2024-09-18T08:11:49.971 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:11:49.972 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:11:49.972 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3' 2024-09-18T08:11:50.059 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid 779349ca-7593-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-18T08:11:50.073 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5 2024-09-18T08:11:50.110 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:11:50.110 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:11:50.111 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5' 2024-09-18T08:11:50.189 INFO:tasks.radosbench.radosbench.0.smithi007.stdout: 35 16 66177 66161 118.118 100.062 0.00790071 0.00846041 2024-09-18T08:11:50.211 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7 2024-09-18T08:11:50.250 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:11:50.251 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:11:50.251 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7' 2024-09-18T08:11:50.291 INFO:teuthology.orchestra.run.smithi007.stderr:noscrub is unset 2024-09-18T08:11:50.352 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3 2024-09-18T08:11:50.384 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:11:50.385 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:11:50.385 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3' 2024-09-18T08:11:50.466 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid 779349ca-7593-11ef-bceb-c7b262605968 -- ceph daemon osd.2 dump_blocked_ops 2024-09-18T08:11:50.486 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5 2024-09-18T08:11:50.523 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:11:50.524 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:11:50.525 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5' 2024-09-18T08:11:50.626 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5 2024-09-18T08:11:50.655 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:11:50 smithi050 ceph-mon[32371]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-18T08:11:50.656 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:11:50 smithi050 ceph-mon[32371]: pgmap v432: 158 pgs: 30 active+undersized, 34 active+undersized+degraded, 94 active+clean; 3.9 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 109 MiB/s wr, 1.74k op/s; 36005/191964 objects degraded (18.756%); 15 KiB/s, 0 objects/s recovering 2024-09-18T08:11:50.656 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:11:50 smithi050 ceph-mon[32371]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-18T08:11:50.656 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:11:50 smithi050 ceph-mon[32371]: from='client.? 172.21.15.7:0/4054511733' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-18T08:11:50.656 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:11:50 smithi050 ceph-mon[32371]: osdmap e248: 8 total, 7 up, 8 in 2024-09-18T08:11:50.662 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:11:50.663 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:11:50.663 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5' 2024-09-18T08:11:50.701 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:11:50 smithi007 ceph-mon[29459]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-18T08:11:50.701 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:11:50 smithi007 ceph-mon[29459]: pgmap v432: 158 pgs: 30 active+undersized, 34 active+undersized+degraded, 94 active+clean; 3.9 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 109 MiB/s wr, 1.74k op/s; 36005/191964 objects degraded (18.756%); 15 KiB/s, 0 objects/s recovering 2024-09-18T08:11:50.701 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:11:50 smithi007 ceph-mon[29459]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-18T08:11:50.701 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:11:50 smithi007 ceph-mon[29459]: from='client.? 172.21.15.7:0/4054511733' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-18T08:11:50.701 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:11:50 smithi007 ceph-mon[29459]: osdmap e248: 8 total, 7 up, 8 in 2024-09-18T08:11:50.703 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:11:50 smithi007 ceph-mon[39249]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-18T08:11:50.703 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:11:50 smithi007 ceph-mon[39249]: pgmap v432: 158 pgs: 30 active+undersized, 34 active+undersized+degraded, 94 active+clean; 3.9 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 109 MiB/s wr, 1.74k op/s; 36005/191964 objects degraded (18.756%); 15 KiB/s, 0 objects/s recovering 2024-09-18T08:11:50.703 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:11:50 smithi007 ceph-mon[39249]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-18T08:11:50.704 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:11:50 smithi007 ceph-mon[39249]: from='client.? 172.21.15.7:0/4054511733' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-18T08:11:50.704 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:11:50 smithi007 ceph-mon[39249]: osdmap e248: 8 total, 7 up, 8 in 2024-09-18T08:11:50.764 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6 2024-09-18T08:11:50.800 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:11:50.800 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:11:50.801 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6' 2024-09-18T08:11:50.901 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4 2024-09-18T08:11:50.941 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:11:50.942 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:11:50.942 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4' 2024-09-18T08:11:51.043 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6 2024-09-18T08:11:51.082 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:11:51.082 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:11:51.083 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6' 2024-09-18T08:11:51.184 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0 2024-09-18T08:11:51.190 INFO:tasks.radosbench.radosbench.0.smithi007.stdout: 36 16 67798 67782 117.65 101.312 0.00924115 0.00849508 2024-09-18T08:11:51.226 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:11:51.227 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:11:51.227 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0' 2024-09-18T08:11:51.244 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/779349ca-7593-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T08:11:51.328 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6 2024-09-18T08:11:51.367 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:11:51.368 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:11:51.368 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6' 2024-09-18T08:11:51.469 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0 2024-09-18T08:11:51.500 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:11:51.501 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:11:51.502 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0' 2024-09-18T08:11:51.603 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5 2024-09-18T08:11:51.639 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:11:51.639 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:11:51.640 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5' 2024-09-18T08:11:51.655 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:11:51 smithi050 ceph-mon[32371]: purged_snaps scrub starts 2024-09-18T08:11:51.656 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:11:51 smithi050 ceph-mon[32371]: purged_snaps scrub ok 2024-09-18T08:11:51.730 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:11:51 smithi007 ceph-mon[29459]: purged_snaps scrub starts 2024-09-18T08:11:51.730 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:11:51 smithi007 ceph-mon[29459]: purged_snaps scrub ok 2024-09-18T08:11:51.731 INFO:journalctl@ceph.osd.1.smithi007.stdout:Sep 18 08:11:51 smithi007 podman[127990]: 2024-09-18 08:11:51.381617585 +0000 UTC m=+0.096115381 image pull b22e1e37ddf1ca6e22136cd6540a9ab61e7cafb8bfbc7a91a2f20f7f9941bd9a quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:1f5b5a3734f665ef333266f39143ac3ef1eb1951b21258f2307e10332e2d7ac9 2024-09-18T08:11:51.731 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:11:51 smithi007 ceph-mon[39249]: purged_snaps scrub starts 2024-09-18T08:11:51.731 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:11:51 smithi007 ceph-mon[39249]: purged_snaps scrub ok 2024-09-18T08:11:51.740 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3 2024-09-18T08:11:51.776 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:11:51.777 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:11:51.777 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3' 2024-09-18T08:11:51.878 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7 2024-09-18T08:11:51.913 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:11:51.914 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:11:51.915 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7' 2024-09-18T08:11:52.016 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3 2024-09-18T08:11:52.050 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:11:52.050 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:11:52.051 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3' 2024-09-18T08:11:52.151 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3 2024-09-18T08:11:52.183 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:11:52.184 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:11:52.184 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3' 2024-09-18T08:11:52.190 INFO:tasks.radosbench.radosbench.0.smithi007.stdout: 37 16 69404 69388 117.183 100.375 0.00522077 0.0085288 2024-09-18T08:11:52.286 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5 2024-09-18T08:11:52.323 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:11:52.324 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:11:52.324 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5' 2024-09-18T08:11:52.425 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3 2024-09-18T08:11:52.457 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:11:52.458 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:11:52.458 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3' 2024-09-18T08:11:52.480 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:11:52 smithi007 ceph-mon[29459]: pgmap v435: 158 pgs: 30 active+undersized, 34 active+undersized+degraded, 94 active+clean; 3.9 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 731 B/s rd, 72 MiB/s wr, 1.15k op/s; 36005/191964 objects degraded (18.756%) 2024-09-18T08:11:52.480 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:11:52 smithi007 ceph-mon[29459]: osdmap e249: 8 total, 7 up, 8 in 2024-09-18T08:11:52.480 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:11:52 smithi007 ceph-mon[29459]: 1.2 scrub starts 2024-09-18T08:11:52.480 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:11:52 smithi007 ceph-mon[29459]: 1.2 scrub ok 2024-09-18T08:11:52.481 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:11:52 smithi007 ceph-mon[39249]: pgmap v435: 158 pgs: 30 active+undersized, 34 active+undersized+degraded, 94 active+clean; 3.9 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 731 B/s rd, 72 MiB/s wr, 1.15k op/s; 36005/191964 objects degraded (18.756%) 2024-09-18T08:11:52.481 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:11:52 smithi007 ceph-mon[39249]: osdmap e249: 8 total, 7 up, 8 in 2024-09-18T08:11:52.481 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:11:52 smithi007 ceph-mon[39249]: 1.2 scrub starts 2024-09-18T08:11:52.481 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:11:52 smithi007 ceph-mon[39249]: 1.2 scrub ok 2024-09-18T08:11:52.559 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3 2024-09-18T08:11:52.590 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:11:52.590 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:11:52.591 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3' 2024-09-18T08:11:52.656 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:11:52 smithi050 ceph-mon[32371]: pgmap v435: 158 pgs: 30 active+undersized, 34 active+undersized+degraded, 94 active+clean; 3.9 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 731 B/s rd, 72 MiB/s wr, 1.15k op/s; 36005/191964 objects degraded (18.756%) 2024-09-18T08:11:52.656 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:11:52 smithi050 ceph-mon[32371]: osdmap e249: 8 total, 7 up, 8 in 2024-09-18T08:11:52.656 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:11:52 smithi050 ceph-mon[32371]: 1.2 scrub starts 2024-09-18T08:11:52.656 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:11:52 smithi050 ceph-mon[32371]: 1.2 scrub ok 2024-09-18T08:11:52.692 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2 2024-09-18T08:11:52.731 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:11:52.732 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:11:52.733 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2' 2024-09-18T08:11:52.833 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3 2024-09-18T08:11:52.871 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:11:52.872 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:11:52.872 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3' 2024-09-18T08:11:52.973 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0 2024-09-18T08:11:52.980 INFO:journalctl@ceph.osd.1.smithi007.stdout:Sep 18 08:11:52 smithi007 podman[127990]: 2024-09-18 08:11:52.84694236 +0000 UTC m=+1.561440128 container create 7f39ef31c9d3508fd840bafc00076f3895ae7ffe4a2a270501c069a6720d390c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:1f5b5a3734f665ef333266f39143ac3ef1eb1951b21258f2307e10332e2d7ac9, name=ceph-779349ca-7593-11ef-bceb-c7b262605968-osd-1-deactivate, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, org.label-schema.name=CentOS Stream 9 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, RELEASE=wip-guits-main-2024-09-17-1213-e7fb7b5, io.buildah.version=1.37.2, org.label-schema.license=GPLv2, org.label-schema.build-date=20240731, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, ceph=True) 2024-09-18T08:11:53.011 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:11:53.012 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:11:53.012 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0' 2024-09-18T08:11:53.113 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6 2024-09-18T08:11:53.149 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:11:53.150 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:11:53.150 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6' 2024-09-18T08:11:53.174 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid 779349ca-7593-11ef-bceb-c7b262605968 -- ceph osd unset nodeep-scrub 2024-09-18T08:11:53.190 INFO:tasks.radosbench.radosbench.0.smithi007.stdout: 38 16 70992 70976 116.711 99.25 0.00523499 0.00856269 2024-09-18T08:11:53.240 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/779349ca-7593-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T08:11:53.250 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5 2024-09-18T08:11:53.285 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:11:53.286 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:11:53.286 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5' 2024-09-18T08:11:53.296 INFO:teuthology.orchestra.run.smithi007.stdout:osd.0: {} 2024-09-18T08:11:53.296 INFO:teuthology.orchestra.run.smithi007.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-18T08:11:53.296 INFO:teuthology.orchestra.run.smithi007.stderr:Error ENXIO: problem getting command descriptions from osd.1 2024-09-18T08:11:53.332 INFO:teuthology.orchestra.run.smithi007.stdout:osd.2: {} 2024-09-18T08:11:53.332 INFO:teuthology.orchestra.run.smithi007.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-18T08:11:53.368 INFO:teuthology.orchestra.run.smithi007.stdout:osd.3: {} 2024-09-18T08:11:53.369 INFO:teuthology.orchestra.run.smithi007.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-18T08:11:53.387 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4 2024-09-18T08:11:53.402 INFO:teuthology.orchestra.run.smithi007.stdout:osd.4: {} 2024-09-18T08:11:53.402 INFO:teuthology.orchestra.run.smithi007.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-18T08:11:53.421 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:11:53.422 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:11:53.422 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4' 2024-09-18T08:11:53.441 INFO:teuthology.orchestra.run.smithi007.stdout:osd.5: {} 2024-09-18T08:11:53.441 INFO:teuthology.orchestra.run.smithi007.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-18T08:11:53.480 INFO:teuthology.orchestra.run.smithi007.stdout:osd.6: {} 2024-09-18T08:11:53.481 INFO:teuthology.orchestra.run.smithi007.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-18T08:11:53.520 INFO:teuthology.orchestra.run.smithi007.stdout:osd.7: {} 2024-09-18T08:11:53.520 INFO:teuthology.orchestra.run.smithi007.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-18T08:11:53.523 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4 2024-09-18T08:11:53.554 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:11:53.555 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:11:53.555 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4' 2024-09-18T08:11:53.614 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:11:53 smithi007 ceph-mon[29459]: from='client.? 172.21.15.7:0/4136601237' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-18T08:11:53.614 INFO:journalctl@ceph.osd.1.smithi007.stdout:Sep 18 08:11:53 smithi007 podman[127990]: 2024-09-18 08:11:53.360725315 +0000 UTC m=+2.075223114 container init 7f39ef31c9d3508fd840bafc00076f3895ae7ffe4a2a270501c069a6720d390c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:1f5b5a3734f665ef333266f39143ac3ef1eb1951b21258f2307e10332e2d7ac9, name=ceph-779349ca-7593-11ef-bceb-c7b262605968-osd-1-deactivate, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, GIT_CLEAN=True, ceph=True, org.label-schema.build-date=20240731, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, RELEASE=wip-guits-main-2024-09-17-1213-e7fb7b5, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.license=GPLv2, io.buildah.version=1.37.2, org.label-schema.vendor=CentOS) 2024-09-18T08:11:53.614 INFO:journalctl@ceph.osd.1.smithi007.stdout:Sep 18 08:11:53 smithi007 podman[127990]: 2024-09-18 08:11:53.369578746 +0000 UTC m=+2.084076511 container start 7f39ef31c9d3508fd840bafc00076f3895ae7ffe4a2a270501c069a6720d390c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:1f5b5a3734f665ef333266f39143ac3ef1eb1951b21258f2307e10332e2d7ac9, name=ceph-779349ca-7593-11ef-bceb-c7b262605968-osd-1-deactivate, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, RELEASE=wip-guits-main-2024-09-17-1213-e7fb7b5, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, ceph=True, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_CLEAN=True, io.buildah.version=1.37.2, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.license=GPLv2, org.label-schema.build-date=20240731, GIT_BRANCH=HEAD) 2024-09-18T08:11:53.615 INFO:journalctl@ceph.osd.1.smithi007.stdout:Sep 18 08:11:53 smithi007 podman[127990]: 2024-09-18 08:11:53.460767364 +0000 UTC m=+2.175265131 container attach 7f39ef31c9d3508fd840bafc00076f3895ae7ffe4a2a270501c069a6720d390c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:1f5b5a3734f665ef333266f39143ac3ef1eb1951b21258f2307e10332e2d7ac9, name=ceph-779349ca-7593-11ef-bceb-c7b262605968-osd-1-deactivate, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.build-date=20240731, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-guits-main-2024-09-17-1213-e7fb7b5, io.buildah.version=1.37.2, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , GIT_CLEAN=True, ceph=True) 2024-09-18T08:11:53.615 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:11:53 smithi007 ceph-mon[39249]: from='client.? 172.21.15.7:0/4136601237' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-18T08:11:53.655 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5 2024-09-18T08:11:53.697 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:11:53.697 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:11:53.698 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5' 2024-09-18T08:11:53.799 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4 2024-09-18T08:11:53.834 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:11:53.835 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:11:53.835 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4' 2024-09-18T08:11:53.837 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:11:53 smithi050 ceph-mon[32371]: from='client.? 172.21.15.7:0/4136601237' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-18T08:11:53.936 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4 2024-09-18T08:11:53.971 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:11:53.971 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:11:53.972 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4' 2024-09-18T08:11:53.980 INFO:journalctl@ceph.osd.1.smithi007.stdout:Sep 18 08:11:53 smithi007 podman[127990]: 2024-09-18 08:11:53.783486434 +0000 UTC m=+2.497984210 container died 7f39ef31c9d3508fd840bafc00076f3895ae7ffe4a2a270501c069a6720d390c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:1f5b5a3734f665ef333266f39143ac3ef1eb1951b21258f2307e10332e2d7ac9, name=ceph-779349ca-7593-11ef-bceb-c7b262605968-osd-1-deactivate, GIT_BRANCH=HEAD, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240731, RELEASE=wip-guits-main-2024-09-17-1213-e7fb7b5, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, org.label-schema.name=CentOS Stream 9 Base Image, io.buildah.version=1.37.2, org.label-schema.vendor=CentOS, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-09-18T08:11:54.072 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4 2024-09-18T08:11:54.107 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:11:54.108 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:11:54.108 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4' 2024-09-18T08:11:54.190 INFO:tasks.radosbench.radosbench.0.smithi007.stdout: 39 16 72508 72492 116.148 94.75 0.00412449 0.00860473 2024-09-18T08:11:54.210 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6 2024-09-18T08:11:54.246 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:11:54.246 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:11:54.247 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6' 2024-09-18T08:11:54.350 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4 2024-09-18T08:11:54.391 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:11:54.391 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:11:54.392 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4' 2024-09-18T08:11:54.480 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:11:54 smithi007 ceph-mon[29459]: pgmap v437: 158 pgs: 30 active+undersized, 34 active+undersized+degraded, 94 active+clean; 4.0 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 852 B/s rd, 94 MiB/s wr, 1.51k op/s; 37009/194976 objects degraded (18.981%) 2024-09-18T08:11:54.480 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:11:54 smithi007 ceph-mon[29459]: 4.1 scrub starts 2024-09-18T08:11:54.480 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:11:54 smithi007 ceph-mon[29459]: 4.1 scrub ok 2024-09-18T08:11:54.480 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:11:54 smithi007 ceph-mon[29459]: Health check update: Degraded data redundancy: 37009/194976 objects degraded (18.981%), 34 pgs degraded (PG_DEGRADED) 2024-09-18T08:11:54.481 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:11:54 smithi007 ceph-mon[39249]: pgmap v437: 158 pgs: 30 active+undersized, 34 active+undersized+degraded, 94 active+clean; 4.0 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 852 B/s rd, 94 MiB/s wr, 1.51k op/s; 37009/194976 objects degraded (18.981%) 2024-09-18T08:11:54.481 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:11:54 smithi007 ceph-mon[39249]: 4.1 scrub starts 2024-09-18T08:11:54.481 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:11:54 smithi007 ceph-mon[39249]: 4.1 scrub ok 2024-09-18T08:11:54.481 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:11:54 smithi007 ceph-mon[39249]: Health check update: Degraded data redundancy: 37009/194976 objects degraded (18.981%), 34 pgs degraded (PG_DEGRADED) 2024-09-18T08:11:54.492 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2 2024-09-18T08:11:54.527 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:11:54.527 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:11:54.527 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2' 2024-09-18T08:11:54.628 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3 2024-09-18T08:11:54.663 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:11:54.663 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:11:54.664 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3' 2024-09-18T08:11:54.688 DEBUG:teuthology.orchestra.run.smithi007:> sudo pkill -f 'journalctl -f -n 0 -u ceph-779349ca-7593-11ef-bceb-c7b262605968@osd.1.service' 2024-09-18T08:11:54.764 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6 2024-09-18T08:11:54.786 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:11:54 smithi050 ceph-mon[32371]: pgmap v437: 158 pgs: 30 active+undersized, 34 active+undersized+degraded, 94 active+clean; 4.0 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 852 B/s rd, 94 MiB/s wr, 1.51k op/s; 37009/194976 objects degraded (18.981%) 2024-09-18T08:11:54.786 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:11:54 smithi050 ceph-mon[32371]: 4.1 scrub starts 2024-09-18T08:11:54.786 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:11:54 smithi050 ceph-mon[32371]: 4.1 scrub ok 2024-09-18T08:11:54.786 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:11:54 smithi050 ceph-mon[32371]: Health check update: Degraded data redundancy: 37009/194976 objects degraded (18.981%), 34 pgs degraded (PG_DEGRADED) 2024-09-18T08:11:54.808 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:11:54.808 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:11:54.809 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6' 2024-09-18T08:11:54.810 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-09-18T08:11:54.810 INFO:tasks.cephadm.osd.1:Stopped osd.1 2024-09-18T08:11:54.810 INFO:tasks.thrashosds.thrasher:Testing ceph-objectstore-tool on down osd.1 2024-09-18T08:11:54.811 INFO:tasks.thrashosds.thrasher:remote for osd 1 is ubuntu@smithi007.front.sepia.ceph.com 2024-09-18T08:11:54.811 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell -n osd.1 --fsid 779349ca-7593-11ef-bceb-c7b262605968 -- ceph-objectstore-tool --err-to-stderr --no-mon-config '--log-file=/var/log/ceph/objectstore_tool.$pid.log' --data-path /var/lib/ceph/osd/ceph-1 --journal-path /var/lib/ceph/osd/ceph-1/journal --op list-pgs 2024-09-18T08:11:54.909 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3 2024-09-18T08:11:54.949 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:11:54.950 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:11:54.950 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3' 2024-09-18T08:11:55.051 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6 2024-09-18T08:11:55.090 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:11:55.090 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:11:55.091 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6' 2024-09-18T08:11:55.186 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/779349ca-7593-11ef-bceb-c7b262605968/osd.1/config 2024-09-18T08:11:55.191 INFO:tasks.radosbench.radosbench.0.smithi007.stdout:2024-09-18T08:11:55.190427+0000 min lat: 0.00114659 max lat: 1.04338 avg lat: 0.00863797 2024-09-18T08:11:55.191 INFO:tasks.radosbench.radosbench.0.smithi007.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-09-18T08:11:55.191 INFO:tasks.radosbench.radosbench.0.smithi007.stdout: 40 16 74085 74069 115.706 98.5625 0.0159529 0.00863797 2024-09-18T08:11:55.191 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3 2024-09-18T08:11:55.229 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-09-18T08:11:55.230 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-09-18T08:11:55.243 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/779349ca-7593-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T08:11:55.250 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:11:55.251 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:11:55.251 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3' 2024-09-18T08:11:55.352 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6 2024-09-18T08:11:55.387 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:11:55.388 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:11:55.388 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6' 2024-09-18T08:11:55.489 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0 2024-09-18T08:11:55.527 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:11:55.528 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:11:55.528 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0' 2024-09-18T08:11:55.629 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7 2024-09-18T08:11:55.665 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:11:55.666 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:11:55.666 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7' 2024-09-18T08:11:55.701 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:11:55 smithi007 ceph-mon[29459]: 1.5 scrub starts 2024-09-18T08:11:55.701 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:11:55 smithi007 ceph-mon[29459]: 1.5 scrub ok 2024-09-18T08:11:55.701 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:11:55 smithi007 ceph-mon[39249]: 1.5 scrub starts 2024-09-18T08:11:55.701 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:11:55 smithi007 ceph-mon[39249]: 1.5 scrub ok 2024-09-18T08:11:55.767 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7 2024-09-18T08:11:55.809 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:11:55.810 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:11:55.810 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7' 2024-09-18T08:11:55.905 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:11:55 smithi050 ceph-mon[32371]: 1.5 scrub starts 2024-09-18T08:11:55.906 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:11:55 smithi050 ceph-mon[32371]: 1.5 scrub ok 2024-09-18T08:11:55.911 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0 2024-09-18T08:11:55.942 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:11:55.943 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:11:55.943 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0' 2024-09-18T08:11:56.044 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2 2024-09-18T08:11:56.074 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:11:56.075 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:11:56.075 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2' 2024-09-18T08:11:56.176 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3 2024-09-18T08:11:56.191 INFO:tasks.radosbench.radosbench.0.smithi007.stdout: 41 16 75676 75660 115.309 99.4375 0.00683296 0.00866687 2024-09-18T08:11:56.210 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:11:56.210 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:11:56.211 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3' 2024-09-18T08:11:56.312 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6 2024-09-18T08:11:56.348 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:11:56.349 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:11:56.349 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6' 2024-09-18T08:11:56.450 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6 2024-09-18T08:11:56.487 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:11:56.487 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:11:56.487 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6' 2024-09-18T08:11:56.588 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3 2024-09-18T08:11:56.620 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:11:56.621 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:11:56.621 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3' 2024-09-18T08:11:56.680 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:11:56 smithi007 ceph-mon[29459]: pgmap v438: 158 pgs: 30 active+undersized, 34 active+undersized+degraded, 94 active+clean; 4.4 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 852 B/s rd, 81 MiB/s wr, 1.30k op/s; 40338/215427 objects degraded (18.725%) 2024-09-18T08:11:56.680 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:11:56 smithi007 ceph-mon[39249]: pgmap v438: 158 pgs: 30 active+undersized, 34 active+undersized+degraded, 94 active+clean; 4.4 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 852 B/s rd, 81 MiB/s wr, 1.30k op/s; 40338/215427 objects degraded (18.725%) 2024-09-18T08:11:56.724 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5 2024-09-18T08:11:56.767 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:11:56.768 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:11:56.769 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5' 2024-09-18T08:11:56.869 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0 2024-09-18T08:11:56.899 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:11:56.900 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:11:56.900 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0' 2024-09-18T08:11:56.905 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:11:56 smithi050 ceph-mon[32371]: pgmap v438: 158 pgs: 30 active+undersized, 34 active+undersized+degraded, 94 active+clean; 4.4 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 852 B/s rd, 81 MiB/s wr, 1.30k op/s; 40338/215427 objects degraded (18.725%) 2024-09-18T08:11:56.980 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Sep 18 08:11:56 smithi007 ceph-779349ca-7593-11ef-bceb-c7b262605968-mgr-y[29688]: ::ffff:172.21.15.50 - - [18/Sep/2024:08:11:56] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-09-18T08:11:57.001 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4 2024-09-18T08:11:57.039 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:11:57.040 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:11:57.040 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4' 2024-09-18T08:11:57.141 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5 2024-09-18T08:11:57.177 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:11:57.178 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:11:57.178 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5' 2024-09-18T08:11:57.191 INFO:tasks.radosbench.radosbench.0.smithi007.stdout: 42 16 77205 77189 114.839 95.5625 0.00772897 0.00870297 2024-09-18T08:11:57.230 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid 779349ca-7593-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-18T08:11:57.249 INFO:teuthology.orchestra.run.smithi007.stdout:{ 2024-09-18T08:11:57.249 INFO:teuthology.orchestra.run.smithi007.stdout: "ops": [], 2024-09-18T08:11:57.249 INFO:teuthology.orchestra.run.smithi007.stdout: "complaint_time": 30, 2024-09-18T08:11:57.249 INFO:teuthology.orchestra.run.smithi007.stdout: "num_blocked_ops": 0 2024-09-18T08:11:57.249 INFO:teuthology.orchestra.run.smithi007.stdout:} 2024-09-18T08:11:57.279 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2 2024-09-18T08:11:57.339 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:11:57.340 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:11:57.340 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2' 2024-09-18T08:11:57.383 INFO:teuthology.orchestra.run.smithi007.stderr:Creating an OSD daemon form without an OSD FSID value 2024-09-18T08:11:57.441 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0 2024-09-18T08:11:57.481 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:11:57.482 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:11:57.482 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0' 2024-09-18T08:11:57.583 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7 2024-09-18T08:11:57.621 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:11:57.621 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:11:57.621 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7' 2024-09-18T08:11:57.722 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3 2024-09-18T08:11:57.730 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:11:57 smithi007 ceph-mon[29459]: purged_snaps scrub starts 2024-09-18T08:11:57.730 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:11:57 smithi007 ceph-mon[29459]: purged_snaps scrub ok 2024-09-18T08:11:57.730 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:11:57 smithi007 ceph-mon[29459]: osdmap e250: 8 total, 7 up, 8 in 2024-09-18T08:11:57.731 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:11:57 smithi007 ceph-mon[39249]: purged_snaps scrub starts 2024-09-18T08:11:57.731 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:11:57 smithi007 ceph-mon[39249]: purged_snaps scrub ok 2024-09-18T08:11:57.731 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:11:57 smithi007 ceph-mon[39249]: osdmap e250: 8 total, 7 up, 8 in 2024-09-18T08:11:57.753 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:11:57.754 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:11:57.754 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3' 2024-09-18T08:11:57.855 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5 2024-09-18T08:11:57.896 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:11:57.897 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:11:57.898 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5' 2024-09-18T08:11:57.905 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:11:57 smithi050 ceph-mon[32371]: purged_snaps scrub starts 2024-09-18T08:11:57.906 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:11:57 smithi050 ceph-mon[32371]: purged_snaps scrub ok 2024-09-18T08:11:57.906 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:11:57 smithi050 ceph-mon[32371]: osdmap e250: 8 total, 7 up, 8 in 2024-09-18T08:11:57.994 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid 779349ca-7593-11ef-bceb-c7b262605968 -- ceph daemon osd.2 dump_historic_ops 2024-09-18T08:11:57.999 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7 2024-09-18T08:11:58.038 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:11:58.039 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:11:58.039 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7' 2024-09-18T08:11:58.140 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5 2024-09-18T08:11:58.178 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:11:58.178 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:11:58.179 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5' 2024-09-18T08:11:58.192 INFO:tasks.radosbench.radosbench.0.smithi007.stdout: 43 15 78745 78730 114.408 96.3125 0.00435405 0.00873611 2024-09-18T08:11:58.279 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4 2024-09-18T08:11:58.321 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:11:58.322 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:11:58.322 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4' 2024-09-18T08:11:58.423 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6 2024-09-18T08:11:58.467 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:11:58.467 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:11:58.468 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6' 2024-09-18T08:11:58.481 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:11:58 smithi007 ceph-mon[29459]: pgmap v440: 158 pgs: 30 active+undersized, 34 active+undersized+degraded, 94 active+clean; 4.4 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 733 B/s rd, 70 MiB/s wr, 1.12k op/s; 40338/215427 objects degraded (18.725%) 2024-09-18T08:11:58.481 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:11:58 smithi007 ceph-mon[39249]: pgmap v440: 158 pgs: 30 active+undersized, 34 active+undersized+degraded, 94 active+clean; 4.4 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 733 B/s rd, 70 MiB/s wr, 1.12k op/s; 40338/215427 objects degraded (18.725%) 2024-09-18T08:11:58.569 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5 2024-09-18T08:11:58.602 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:11:58.603 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:11:58.603 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5' 2024-09-18T08:11:58.704 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7 2024-09-18T08:11:58.740 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:11:58.741 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:11:58.741 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7' 2024-09-18T08:11:58.842 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2 2024-09-18T08:11:58.882 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:11:58.882 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:11:58.883 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2' 2024-09-18T08:11:58.905 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:11:58 smithi050 ceph-mon[32371]: pgmap v440: 158 pgs: 30 active+undersized, 34 active+undersized+degraded, 94 active+clean; 4.4 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 733 B/s rd, 70 MiB/s wr, 1.12k op/s; 40338/215427 objects degraded (18.725%) 2024-09-18T08:11:58.984 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3 2024-09-18T08:11:59.035 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:11:59.036 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:11:59.036 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3' 2024-09-18T08:11:59.075 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/779349ca-7593-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T08:11:59.079 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/779349ca-7593-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T08:11:59.137 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0 2024-09-18T08:11:59.187 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:11:59.188 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:11:59.188 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0' 2024-09-18T08:11:59.193 INFO:tasks.radosbench.radosbench.0.smithi007.stdout: 44 16 80214 80198 113.89 91.75 0.012646 0.00877567 2024-09-18T08:11:59.289 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3 2024-09-18T08:11:59.343 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:11:59.343 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:11:59.343 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3' 2024-09-18T08:11:59.444 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3 2024-09-18T08:11:59.470 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:11:59 smithi007 ceph-mon[29459]: 6.f scrub starts 2024-09-18T08:11:59.470 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:11:59 smithi007 ceph-mon[29459]: 6.f scrub ok 2024-09-18T08:11:59.470 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:11:59 smithi007 ceph-mon[29459]: Health check update: Degraded data redundancy: 40338/215427 objects degraded (18.725%), 34 pgs degraded (PG_DEGRADED) 2024-09-18T08:11:59.470 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:11:59 smithi007 ceph-mon[29459]: osdmap e251: 8 total, 7 up, 8 in 2024-09-18T08:11:59.470 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:11:59 smithi007 ceph-mon[29459]: from='client.? 172.21.15.7:0/2168209908' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-18T08:11:59.470 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:11:59 smithi007 ceph-mon[29459]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-18T08:11:59.470 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:11:59 smithi007 ceph-mon[39249]: 6.f scrub starts 2024-09-18T08:11:59.470 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:11:59 smithi007 ceph-mon[39249]: 6.f scrub ok 2024-09-18T08:11:59.470 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:11:59 smithi007 ceph-mon[39249]: Health check update: Degraded data redundancy: 40338/215427 objects degraded (18.725%), 34 pgs degraded (PG_DEGRADED) 2024-09-18T08:11:59.470 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:11:59 smithi007 ceph-mon[39249]: osdmap e251: 8 total, 7 up, 8 in 2024-09-18T08:11:59.470 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:11:59 smithi007 ceph-mon[39249]: from='client.? 172.21.15.7:0/2168209908' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-18T08:11:59.471 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:11:59 smithi007 ceph-mon[39249]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-18T08:11:59.496 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:11:59.497 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:11:59.497 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3' 2024-09-18T08:11:59.598 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2 2024-09-18T08:11:59.642 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:11:59.642 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:11:59.643 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2' 2024-09-18T08:11:59.656 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:11:59 smithi050 ceph-mon[32371]: 6.f scrub starts 2024-09-18T08:11:59.656 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:11:59 smithi050 ceph-mon[32371]: 6.f scrub ok 2024-09-18T08:11:59.656 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:11:59 smithi050 ceph-mon[32371]: Health check update: Degraded data redundancy: 40338/215427 objects degraded (18.725%), 34 pgs degraded (PG_DEGRADED) 2024-09-18T08:11:59.656 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:11:59 smithi050 ceph-mon[32371]: osdmap e251: 8 total, 7 up, 8 in 2024-09-18T08:11:59.656 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:11:59 smithi050 ceph-mon[32371]: from='client.? 172.21.15.7:0/2168209908' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-18T08:11:59.656 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:11:59 smithi050 ceph-mon[32371]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-18T08:11:59.656 INFO:journalctl@ceph.iscsi.iscsi.a.smithi050.stdout:Sep 18 08:11:59 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-iscsi-iscsi-a[57782]: debug there is no tcmu-runner data available 2024-09-18T08:11:59.744 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2 2024-09-18T08:11:59.783 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:11:59.783 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:11:59.784 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2' 2024-09-18T08:11:59.885 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0 2024-09-18T08:11:59.920 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:11:59.921 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:11:59.921 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0' 2024-09-18T08:11:59.977 INFO:teuthology.orchestra.run.smithi007.stdout:{ 2024-09-18T08:11:59.977 INFO:teuthology.orchestra.run.smithi007.stdout: "size": 20, 2024-09-18T08:11:59.977 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 600, 2024-09-18T08:11:59.977 INFO:teuthology.orchestra.run.smithi007.stdout: "ops": [ 2024-09-18T08:11:59.977 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:11:59.977 INFO:teuthology.orchestra.run.smithi007.stdout: "description": "osd_op(client.14805.0:66030 7.e 7:7b16ec01:::benchmark_data_smithi007_73155_object66029:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e102)", 2024-09-18T08:11:59.977 INFO:teuthology.orchestra.run.smithi007.stdout: "initiated_at": "2024-09-18T08:04:43.960797+0000", 2024-09-18T08:11:59.977 INFO:teuthology.orchestra.run.smithi007.stdout: "age": 435.80244965700001, 2024-09-18T08:11:59.978 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.089093715000000004, 2024-09-18T08:11:59.978 INFO:teuthology.orchestra.run.smithi007.stdout: "continuous": false, 2024-09-18T08:11:59.978 INFO:teuthology.orchestra.run.smithi007.stdout: "type_data": { 2024-09-18T08:11:59.978 INFO:teuthology.orchestra.run.smithi007.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T08:11:59.978 INFO:teuthology.orchestra.run.smithi007.stdout: "client_info": { 2024-09-18T08:11:59.978 INFO:teuthology.orchestra.run.smithi007.stdout: "client": "client.14805", 2024-09-18T08:11:59.978 INFO:teuthology.orchestra.run.smithi007.stdout: "client_addr": "172.21.15.7:0/640320436", 2024-09-18T08:11:59.978 INFO:teuthology.orchestra.run.smithi007.stdout: "tid": 66030 2024-09-18T08:11:59.978 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:11:59.978 INFO:teuthology.orchestra.run.smithi007.stdout: "events": [ 2024-09-18T08:11:59.978 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:11:59.978 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "initiated", 2024-09-18T08:11:59.978 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:04:43.960797+0000", 2024-09-18T08:11:59.979 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-09-18T08:11:59.979 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:11:59.979 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:11:59.979 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "header_read", 2024-09-18T08:11:59.979 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:04:43.960797+0000", 2024-09-18T08:11:59.979 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-09-18T08:11:59.979 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:11:59.979 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:11:59.979 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "throttled", 2024-09-18T08:11:59.979 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:04:43.960798+0000", 2024-09-18T08:11:59.979 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.1909999999999999e-06 2024-09-18T08:11:59.979 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:11:59.979 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:11:59.979 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "all_read", 2024-09-18T08:11:59.980 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:04:43.960822+0000", 2024-09-18T08:11:59.980 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 2.3785999999999999e-05 2024-09-18T08:11:59.980 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:11:59.980 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:11:59.980 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "dispatched", 2024-09-18T08:11:59.980 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:04:43.960830+0000", 2024-09-18T08:11:59.980 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 8.2400000000000007e-06 2024-09-18T08:11:59.980 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:11:59.980 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:11:59.980 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "queued_for_pg", 2024-09-18T08:11:59.980 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:04:43.960837+0000", 2024-09-18T08:11:59.980 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 6.7109999999999998e-06 2024-09-18T08:11:59.981 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:11:59.981 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:11:59.981 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-09-18T08:11:59.981 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:04:43.964072+0000", 2024-09-18T08:11:59.981 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.0032352380000000001 2024-09-18T08:11:59.981 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:11:59.981 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:11:59.981 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "started", 2024-09-18T08:11:59.981 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:04:43.964179+0000", 2024-09-18T08:11:59.981 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.000107524 2024-09-18T08:11:59.981 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:11:59.981 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:11:59.981 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "waiting for subops from 0,5", 2024-09-18T08:11:59.981 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:04:43.964697+0000", 2024-09-18T08:11:59.982 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.000517097 2024-09-18T08:11:59.982 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:11:59.982 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:11:59.982 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:11:59.982 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:04:43.970563+0000", 2024-09-18T08:11:59.982 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.0058661649999999996 2024-09-18T08:11:59.982 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:11:59.982 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:11:59.982 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "op_commit", 2024-09-18T08:11:59.982 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:04:43.970665+0000", 2024-09-18T08:11:59.982 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.000101888 2024-09-18T08:11:59.982 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:11:59.983 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:11:59.983 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:11:59.983 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:04:44.049154+0000", 2024-09-18T08:11:59.983 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.078489797 2024-09-18T08:11:59.983 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:11:59.983 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:11:59.983 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "commit_sent", 2024-09-18T08:11:59.983 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:04:44.049385+0000", 2024-09-18T08:11:59.983 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.00023076499999999999 2024-09-18T08:11:59.983 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:11:59.983 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:11:59.983 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "done", 2024-09-18T08:11:59.983 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:04:44.049890+0000", 2024-09-18T08:11:59.984 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.00050531299999999996 2024-09-18T08:11:59.984 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-09-18T08:11:59.984 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-09-18T08:11:59.984 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-09-18T08:11:59.984 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:11:59.984 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:11:59.984 INFO:teuthology.orchestra.run.smithi007.stdout: "description": "osd_op(client.14805.0:94181 7.e 7:72880d48:::benchmark_data_smithi007_73155_object94180:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e108)", 2024-09-18T08:11:59.984 INFO:teuthology.orchestra.run.smithi007.stdout: "initiated_at": "2024-09-18T08:05:09.961032+0000", 2024-09-18T08:11:59.984 INFO:teuthology.orchestra.run.smithi007.stdout: "age": 409.80221416000001, 2024-09-18T08:11:59.984 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.103403914, 2024-09-18T08:11:59.984 INFO:teuthology.orchestra.run.smithi007.stdout: "continuous": false, 2024-09-18T08:11:59.984 INFO:teuthology.orchestra.run.smithi007.stdout: "type_data": { 2024-09-18T08:11:59.984 INFO:teuthology.orchestra.run.smithi007.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T08:11:59.985 INFO:teuthology.orchestra.run.smithi007.stdout: "client_info": { 2024-09-18T08:11:59.985 INFO:teuthology.orchestra.run.smithi007.stdout: "client": "client.14805", 2024-09-18T08:11:59.985 INFO:teuthology.orchestra.run.smithi007.stdout: "client_addr": "172.21.15.7:0/640320436", 2024-09-18T08:11:59.985 INFO:teuthology.orchestra.run.smithi007.stdout: "tid": 94181 2024-09-18T08:11:59.985 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:11:59.985 INFO:teuthology.orchestra.run.smithi007.stdout: "events": [ 2024-09-18T08:11:59.985 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:11:59.985 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "initiated", 2024-09-18T08:11:59.985 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:05:09.961032+0000", 2024-09-18T08:11:59.985 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-09-18T08:11:59.986 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:11:59.986 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:11:59.986 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "header_read", 2024-09-18T08:11:59.986 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:05:09.961032+0000", 2024-09-18T08:11:59.986 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-09-18T08:11:59.986 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:11:59.986 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:11:59.986 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "throttled", 2024-09-18T08:11:59.986 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:05:09.961034+0000", 2024-09-18T08:11:59.986 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.342e-06 2024-09-18T08:11:59.986 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:11:59.986 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:11:59.986 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "all_read", 2024-09-18T08:11:59.986 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:05:09.961055+0000", 2024-09-18T08:11:59.987 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 2.1114000000000001e-05 2024-09-18T08:11:59.987 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:11:59.987 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:11:59.987 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "dispatched", 2024-09-18T08:11:59.987 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:05:09.961066+0000", 2024-09-18T08:11:59.987 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.1185999999999999e-05 2024-09-18T08:11:59.987 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:11:59.987 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:11:59.987 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "queued_for_pg", 2024-09-18T08:11:59.987 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:05:09.961073+0000", 2024-09-18T08:11:59.987 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 6.9630000000000001e-06 2024-09-18T08:11:59.987 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:11:59.987 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:11:59.987 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-09-18T08:11:59.987 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:05:09.961131+0000", 2024-09-18T08:11:59.988 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 5.7666e-05 2024-09-18T08:11:59.988 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:11:59.988 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:11:59.988 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "started", 2024-09-18T08:11:59.988 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:05:09.961241+0000", 2024-09-18T08:11:59.988 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.000110321 2024-09-18T08:11:59.988 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:11:59.988 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:11:59.988 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "waiting for subops from 0,5", 2024-09-18T08:11:59.988 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:05:09.961364+0000", 2024-09-18T08:11:59.988 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.00012286400000000001 2024-09-18T08:11:59.988 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:11:59.988 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:11:59.989 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:11:59.989 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:05:09.965873+0000", 2024-09-18T08:11:59.989 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.0045095930000000001 2024-09-18T08:11:59.989 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:11:59.989 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:11:59.989 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "op_commit", 2024-09-18T08:11:59.989 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:05:09.965897+0000", 2024-09-18T08:11:59.989 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 2.4077999999999999e-05 2024-09-18T08:11:59.989 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:11:59.989 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:11:59.989 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:11:59.989 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:05:10.064128+0000", 2024-09-18T08:11:59.989 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.098230856000000005 2024-09-18T08:11:59.989 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:11:59.990 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:11:59.990 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "commit_sent", 2024-09-18T08:11:59.990 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:05:10.064388+0000", 2024-09-18T08:11:59.990 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.00025948099999999999 2024-09-18T08:11:59.990 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:11:59.990 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:11:59.990 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "done", 2024-09-18T08:11:59.990 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:05:10.064436+0000", 2024-09-18T08:11:59.990 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 4.8449999999999999e-05 2024-09-18T08:11:59.990 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-09-18T08:11:59.990 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-09-18T08:11:59.990 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-09-18T08:11:59.990 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:11:59.990 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:11:59.991 INFO:teuthology.orchestra.run.smithi007.stdout: "description": "osd_op(client.14805.0:96235 7.e 7:70c09d0d:::benchmark_data_smithi007_73155_object96234:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e109)", 2024-09-18T08:11:59.991 INFO:teuthology.orchestra.run.smithi007.stdout: "initiated_at": "2024-09-18T08:05:12.494260+0000", 2024-09-18T08:11:59.991 INFO:teuthology.orchestra.run.smithi007.stdout: "age": 407.26898667699999, 2024-09-18T08:11:59.991 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.089927492999999997, 2024-09-18T08:11:59.991 INFO:teuthology.orchestra.run.smithi007.stdout: "continuous": false, 2024-09-18T08:11:59.991 INFO:teuthology.orchestra.run.smithi007.stdout: "type_data": { 2024-09-18T08:11:59.991 INFO:teuthology.orchestra.run.smithi007.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T08:11:59.991 INFO:teuthology.orchestra.run.smithi007.stdout: "client_info": { 2024-09-18T08:11:59.991 INFO:teuthology.orchestra.run.smithi007.stdout: "client": "client.14805", 2024-09-18T08:11:59.991 INFO:teuthology.orchestra.run.smithi007.stdout: "client_addr": "172.21.15.7:0/640320436", 2024-09-18T08:11:59.991 INFO:teuthology.orchestra.run.smithi007.stdout: "tid": 96235 2024-09-18T08:11:59.991 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:11:59.991 INFO:teuthology.orchestra.run.smithi007.stdout: "events": [ 2024-09-18T08:11:59.992 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:11:59.992 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "initiated", 2024-09-18T08:11:59.992 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:05:12.494260+0000", 2024-09-18T08:11:59.992 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-09-18T08:11:59.992 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:11:59.992 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:11:59.992 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "header_read", 2024-09-18T08:11:59.992 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:05:12.494260+0000", 2024-09-18T08:11:59.992 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-09-18T08:11:59.992 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:11:59.992 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:11:59.992 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "throttled", 2024-09-18T08:11:59.992 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:05:12.494351+0000", 2024-09-18T08:11:59.993 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 9.0839000000000002e-05 2024-09-18T08:11:59.993 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:11:59.993 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:11:59.993 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "all_read", 2024-09-18T08:11:59.993 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:05:12.494375+0000", 2024-09-18T08:11:59.993 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 2.4921000000000001e-05 2024-09-18T08:11:59.993 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:11:59.993 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:11:59.993 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "dispatched", 2024-09-18T08:11:59.993 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:05:12.494388+0000", 2024-09-18T08:11:59.993 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.2415000000000001e-05 2024-09-18T08:11:59.993 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:11:59.993 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:11:59.993 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "queued_for_pg", 2024-09-18T08:11:59.994 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:05:12.494396+0000", 2024-09-18T08:11:59.994 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 8.0770000000000008e-06 2024-09-18T08:11:59.994 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:11:59.994 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:11:59.994 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-09-18T08:11:59.994 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:05:12.500910+0000", 2024-09-18T08:11:59.994 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.0065145089999999999 2024-09-18T08:11:59.994 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:11:59.994 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:11:59.994 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "started", 2024-09-18T08:11:59.994 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:05:12.501024+0000", 2024-09-18T08:11:59.994 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.000113992 2024-09-18T08:11:59.994 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:11:59.994 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:11:59.995 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "waiting for subops from 0,5", 2024-09-18T08:11:59.995 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:05:12.501165+0000", 2024-09-18T08:11:59.995 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.000140347 2024-09-18T08:11:59.995 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:11:59.995 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:11:59.995 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:11:59.995 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:05:12.531672+0000", 2024-09-18T08:11:59.995 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.030506945000000001 2024-09-18T08:11:59.995 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:11:59.995 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:11:59.995 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "op_commit", 2024-09-18T08:11:59.995 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:05:12.531683+0000", 2024-09-18T08:11:59.995 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.1092e-05 2024-09-18T08:11:59.996 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:11:59.996 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:11:59.996 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:11:59.996 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:05:12.583879+0000", 2024-09-18T08:11:59.996 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.052195966000000003 2024-09-18T08:11:59.996 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:11:59.996 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:11:59.996 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "commit_sent", 2024-09-18T08:11:59.996 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:05:12.584048+0000", 2024-09-18T08:11:59.996 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.00016902 2024-09-18T08:11:59.996 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:11:59.996 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:11:59.996 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "done", 2024-09-18T08:11:59.997 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:05:12.584187+0000", 2024-09-18T08:11:59.997 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.00013936999999999999 2024-09-18T08:11:59.997 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-09-18T08:11:59.997 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-09-18T08:11:59.997 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-09-18T08:11:59.997 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:11:59.997 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:11:59.997 INFO:teuthology.orchestra.run.smithi007.stdout: "description": "osd_op(client.14805.0:96694 7.e 7:77dc6698:::benchmark_data_smithi007_73155_object96693:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e109)", 2024-09-18T08:11:59.997 INFO:teuthology.orchestra.run.smithi007.stdout: "initiated_at": "2024-09-18T08:05:13.227222+0000", 2024-09-18T08:11:59.997 INFO:teuthology.orchestra.run.smithi007.stdout: "age": 406.53602457699998, 2024-09-18T08:11:59.997 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.092647683999999994, 2024-09-18T08:11:59.997 INFO:teuthology.orchestra.run.smithi007.stdout: "continuous": false, 2024-09-18T08:11:59.997 INFO:teuthology.orchestra.run.smithi007.stdout: "type_data": { 2024-09-18T08:11:59.997 INFO:teuthology.orchestra.run.smithi007.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T08:11:59.998 INFO:teuthology.orchestra.run.smithi007.stdout: "client_info": { 2024-09-18T08:11:59.998 INFO:teuthology.orchestra.run.smithi007.stdout: "client": "client.14805", 2024-09-18T08:11:59.998 INFO:teuthology.orchestra.run.smithi007.stdout: "client_addr": "172.21.15.7:0/640320436", 2024-09-18T08:11:59.998 INFO:teuthology.orchestra.run.smithi007.stdout: "tid": 96694 2024-09-18T08:11:59.998 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:11:59.998 INFO:teuthology.orchestra.run.smithi007.stdout: "events": [ 2024-09-18T08:11:59.998 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:11:59.998 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "initiated", 2024-09-18T08:11:59.998 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:05:13.227222+0000", 2024-09-18T08:11:59.998 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-09-18T08:11:59.998 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:11:59.998 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:11:59.998 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "header_read", 2024-09-18T08:11:59.998 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:05:13.227222+0000", 2024-09-18T08:11:59.999 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-09-18T08:11:59.999 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:11:59.999 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:11:59.999 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "throttled", 2024-09-18T08:11:59.999 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:05:13.227224+0000", 2024-09-18T08:11:59.999 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 2.243e-06 2024-09-18T08:12:00.000 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:12:00.000 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:12:00.000 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "all_read", 2024-09-18T08:12:00.000 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:05:13.227249+0000", 2024-09-18T08:12:00.000 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 2.4820999999999998e-05 2024-09-18T08:12:00.000 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:12:00.001 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:12:00.001 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "dispatched", 2024-09-18T08:12:00.001 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:05:13.227262+0000", 2024-09-18T08:12:00.001 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.2918999999999999e-05 2024-09-18T08:12:00.001 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:12:00.001 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:12:00.001 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "queued_for_pg", 2024-09-18T08:12:00.002 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:05:13.227275+0000", 2024-09-18T08:12:00.002 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.3607e-05 2024-09-18T08:12:00.002 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:12:00.002 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:12:00.002 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-09-18T08:12:00.002 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:05:13.239760+0000", 2024-09-18T08:12:00.003 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.012484419 2024-09-18T08:12:00.003 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:12:00.003 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:12:00.003 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "started", 2024-09-18T08:12:00.003 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:05:13.239892+0000", 2024-09-18T08:12:00.003 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.00013202900000000001 2024-09-18T08:12:00.003 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:12:00.003 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:12:00.003 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "waiting for subops from 0,5", 2024-09-18T08:12:00.004 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:05:13.240049+0000", 2024-09-18T08:12:00.004 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.00015725599999999999 2024-09-18T08:12:00.004 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:12:00.004 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:12:00.004 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "op_commit", 2024-09-18T08:12:00.004 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:05:13.247290+0000", 2024-09-18T08:12:00.005 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.00724072 2024-09-18T08:12:00.005 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:12:00.005 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:12:00.005 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:12:00.005 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:05:13.247559+0000", 2024-09-18T08:12:00.005 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.00026900699999999998 2024-09-18T08:12:00.006 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:12:00.006 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:12:00.006 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:12:00.006 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:05:13.319747+0000", 2024-09-18T08:12:00.006 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.072187789000000002 2024-09-18T08:12:00.006 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:12:00.006 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:12:00.007 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "commit_sent", 2024-09-18T08:12:00.007 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:05:13.319810+0000", 2024-09-18T08:12:00.007 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 6.3139000000000007e-05 2024-09-18T08:12:00.007 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:12:00.007 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:12:00.007 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "done", 2024-09-18T08:12:00.007 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:05:13.319869+0000", 2024-09-18T08:12:00.007 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 5.9735000000000003e-05 2024-09-18T08:12:00.007 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-09-18T08:12:00.007 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-09-18T08:12:00.008 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-09-18T08:12:00.008 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:12:00.008 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:12:00.008 INFO:teuthology.orchestra.run.smithi007.stdout: "description": "osd_op(client.14805.0:96719 7.e 7:7f123a7e:::benchmark_data_smithi007_73155_object96718:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e109)", 2024-09-18T08:12:00.008 INFO:teuthology.orchestra.run.smithi007.stdout: "initiated_at": "2024-09-18T08:05:13.293471+0000", 2024-09-18T08:12:00.008 INFO:teuthology.orchestra.run.smithi007.stdout: "age": 406.46977579000003, 2024-09-18T08:12:00.008 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.091559383999999994, 2024-09-18T08:12:00.008 INFO:teuthology.orchestra.run.smithi007.stdout: "continuous": false, 2024-09-18T08:12:00.008 INFO:teuthology.orchestra.run.smithi007.stdout: "type_data": { 2024-09-18T08:12:00.008 INFO:teuthology.orchestra.run.smithi007.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T08:12:00.008 INFO:teuthology.orchestra.run.smithi007.stdout: "client_info": { 2024-09-18T08:12:00.008 INFO:teuthology.orchestra.run.smithi007.stdout: "client": "client.14805", 2024-09-18T08:12:00.008 INFO:teuthology.orchestra.run.smithi007.stdout: "client_addr": "172.21.15.7:0/640320436", 2024-09-18T08:12:00.009 INFO:teuthology.orchestra.run.smithi007.stdout: "tid": 96719 2024-09-18T08:12:00.009 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:12:00.009 INFO:teuthology.orchestra.run.smithi007.stdout: "events": [ 2024-09-18T08:12:00.009 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:12:00.009 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "initiated", 2024-09-18T08:12:00.009 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:05:13.293471+0000", 2024-09-18T08:12:00.009 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-09-18T08:12:00.009 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:12:00.009 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:12:00.009 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "header_read", 2024-09-18T08:12:00.009 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:05:13.293471+0000", 2024-09-18T08:12:00.009 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-09-18T08:12:00.009 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:12:00.009 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:12:00.010 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "throttled", 2024-09-18T08:12:00.010 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:05:13.293472+0000", 2024-09-18T08:12:00.010 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.482e-06 2024-09-18T08:12:00.010 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:12:00.010 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:12:00.010 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "all_read", 2024-09-18T08:12:00.010 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:05:13.293496+0000", 2024-09-18T08:12:00.010 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 2.3708000000000001e-05 2024-09-18T08:12:00.010 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:12:00.010 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:12:00.010 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "dispatched", 2024-09-18T08:12:00.010 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:05:13.293506+0000", 2024-09-18T08:12:00.010 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.0706e-05 2024-09-18T08:12:00.011 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:12:00.011 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:12:00.011 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "queued_for_pg", 2024-09-18T08:12:00.011 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:05:13.293514+0000", 2024-09-18T08:12:00.011 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 7.4449999999999999e-06 2024-09-18T08:12:00.011 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:12:00.011 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:12:00.011 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-09-18T08:12:00.011 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:05:13.320155+0000", 2024-09-18T08:12:00.011 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.026641436000000001 2024-09-18T08:12:00.011 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:12:00.011 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:12:00.011 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "started", 2024-09-18T08:12:00.011 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:05:13.320303+0000", 2024-09-18T08:12:00.012 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.00014768800000000001 2024-09-18T08:12:00.012 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:12:00.012 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:12:00.012 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "waiting for subops from 0,5", 2024-09-18T08:12:00.012 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:05:13.320454+0000", 2024-09-18T08:12:00.012 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.000151328 2024-09-18T08:12:00.012 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:12:00.012 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:12:00.012 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:12:00.012 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:05:13.326669+0000", 2024-09-18T08:12:00.012 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.0062146090000000003 2024-09-18T08:12:00.012 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:12:00.013 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:12:00.013 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "op_commit", 2024-09-18T08:12:00.013 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:05:13.326681+0000", 2024-09-18T08:12:00.013 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.2160999999999999e-05 2024-09-18T08:12:00.013 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:12:00.013 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:12:00.013 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:12:00.013 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:05:13.384909+0000", 2024-09-18T08:12:00.013 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.058227999000000003 2024-09-18T08:12:00.016 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:12:00.016 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:12:00.016 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "commit_sent", 2024-09-18T08:12:00.016 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:05:13.384968+0000", 2024-09-18T08:12:00.016 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 5.8479e-05 2024-09-18T08:12:00.016 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:12:00.016 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:12:00.017 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "done", 2024-09-18T08:12:00.017 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:05:13.385030+0000", 2024-09-18T08:12:00.017 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 6.2342999999999996e-05 2024-09-18T08:12:00.017 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-09-18T08:12:00.017 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-09-18T08:12:00.017 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-09-18T08:12:00.017 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:12:00.017 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:12:00.017 INFO:teuthology.orchestra.run.smithi007.stdout: "description": "osd_op(client.14805.0:141891 7.e 7:7d0cb9e1:::benchmark_data_smithi007_73155_object23600:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e127)", 2024-09-18T08:12:00.017 INFO:teuthology.orchestra.run.smithi007.stdout: "initiated_at": "2024-09-18T08:05:59.091427+0000", 2024-09-18T08:12:00.017 INFO:teuthology.orchestra.run.smithi007.stdout: "age": 360.67181914600002, 2024-09-18T08:12:00.017 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.090109581999999994, 2024-09-18T08:12:00.017 INFO:teuthology.orchestra.run.smithi007.stdout: "continuous": false, 2024-09-18T08:12:00.018 INFO:teuthology.orchestra.run.smithi007.stdout: "type_data": { 2024-09-18T08:12:00.018 INFO:teuthology.orchestra.run.smithi007.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T08:12:00.018 INFO:teuthology.orchestra.run.smithi007.stdout: "client_info": { 2024-09-18T08:12:00.018 INFO:teuthology.orchestra.run.smithi007.stdout: "client": "client.14805", 2024-09-18T08:12:00.018 INFO:teuthology.orchestra.run.smithi007.stdout: "client_addr": "172.21.15.7:0/640320436", 2024-09-18T08:12:00.018 INFO:teuthology.orchestra.run.smithi007.stdout: "tid": 141891 2024-09-18T08:12:00.018 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:12:00.018 INFO:teuthology.orchestra.run.smithi007.stdout: "events": [ 2024-09-18T08:12:00.018 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:12:00.018 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "initiated", 2024-09-18T08:12:00.018 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:05:59.091427+0000", 2024-09-18T08:12:00.019 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-09-18T08:12:00.019 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:12:00.019 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:12:00.019 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "header_read", 2024-09-18T08:12:00.019 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:05:59.091427+0000", 2024-09-18T08:12:00.019 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-09-18T08:12:00.019 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:12:00.019 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:12:00.019 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "throttled", 2024-09-18T08:12:00.019 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:05:59.091429+0000", 2024-09-18T08:12:00.019 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.719e-06 2024-09-18T08:12:00.019 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:12:00.019 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:12:00.019 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "all_read", 2024-09-18T08:12:00.020 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:05:59.091435+0000", 2024-09-18T08:12:00.020 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 5.8850000000000004e-06 2024-09-18T08:12:00.020 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:12:00.020 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:12:00.020 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "dispatched", 2024-09-18T08:12:00.020 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:05:59.091447+0000", 2024-09-18T08:12:00.020 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.2007e-05 2024-09-18T08:12:00.020 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:12:00.020 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:12:00.020 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "queued_for_pg", 2024-09-18T08:12:00.020 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:05:59.091455+0000", 2024-09-18T08:12:00.020 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 8.5490000000000006e-06 2024-09-18T08:12:00.020 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:12:00.021 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:12:00.021 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-09-18T08:12:00.021 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:05:59.091549+0000", 2024-09-18T08:12:00.021 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 9.3411000000000004e-05 2024-09-18T08:12:00.021 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:12:00.021 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:12:00.021 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "started", 2024-09-18T08:12:00.021 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:05:59.091625+0000", 2024-09-18T08:12:00.021 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 7.5745000000000001e-05 2024-09-18T08:12:00.021 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:12:00.021 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:12:00.021 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "waiting for subops from 0,5", 2024-09-18T08:12:00.022 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:05:59.091687+0000", 2024-09-18T08:12:00.022 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 6.2831999999999997e-05 2024-09-18T08:12:00.022 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:12:00.022 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:12:00.022 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "op_commit", 2024-09-18T08:12:00.022 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:05:59.094073+0000", 2024-09-18T08:12:00.022 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.0023852959999999999 2024-09-18T08:12:00.022 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:12:00.022 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:12:00.022 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:12:00.022 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:05:59.094439+0000", 2024-09-18T08:12:00.022 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.00036632800000000002 2024-09-18T08:12:00.022 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:12:00.022 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:12:00.022 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:12:00.023 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:05:59.181431+0000", 2024-09-18T08:12:00.023 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.086991857000000006 2024-09-18T08:12:00.023 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:12:00.023 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:12:00.023 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "commit_sent", 2024-09-18T08:12:00.023 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:05:59.181489+0000", 2024-09-18T08:12:00.023 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 5.8477999999999998e-05 2024-09-18T08:12:00.023 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:12:00.023 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:12:00.023 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "done", 2024-09-18T08:12:00.023 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:05:59.181537+0000", 2024-09-18T08:12:00.023 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 4.7475000000000002e-05 2024-09-18T08:12:00.023 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-09-18T08:12:00.024 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-09-18T08:12:00.024 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-09-18T08:12:00.024 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:12:00.024 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:12:00.024 INFO:teuthology.orchestra.run.smithi007.stdout: "description": "osd_op(client.14805.0:147830 7.e 7:7137fe31:::benchmark_data_smithi007_73155_object29539:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e127)", 2024-09-18T08:12:00.024 INFO:teuthology.orchestra.run.smithi007.stdout: "initiated_at": "2024-09-18T08:06:04.696707+0000", 2024-09-18T08:12:00.024 INFO:teuthology.orchestra.run.smithi007.stdout: "age": 355.06653922200002, 2024-09-18T08:12:00.024 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.093481047999999997, 2024-09-18T08:12:00.024 INFO:teuthology.orchestra.run.smithi007.stdout: "continuous": false, 2024-09-18T08:12:00.024 INFO:teuthology.orchestra.run.smithi007.stdout: "type_data": { 2024-09-18T08:12:00.024 INFO:teuthology.orchestra.run.smithi007.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T08:12:00.024 INFO:teuthology.orchestra.run.smithi007.stdout: "client_info": { 2024-09-18T08:12:00.024 INFO:teuthology.orchestra.run.smithi007.stdout: "client": "client.14805", 2024-09-18T08:12:00.025 INFO:teuthology.orchestra.run.smithi007.stdout: "client_addr": "172.21.15.7:0/640320436", 2024-09-18T08:12:00.025 INFO:teuthology.orchestra.run.smithi007.stdout: "tid": 147830 2024-09-18T08:12:00.025 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:12:00.025 INFO:teuthology.orchestra.run.smithi007.stdout: "events": [ 2024-09-18T08:12:00.025 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:12:00.025 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "initiated", 2024-09-18T08:12:00.025 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:06:04.696707+0000", 2024-09-18T08:12:00.025 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-09-18T08:12:00.025 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:12:00.025 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:12:00.025 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "header_read", 2024-09-18T08:12:00.025 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:06:04.696707+0000", 2024-09-18T08:12:00.025 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-09-18T08:12:00.025 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:12:00.026 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:12:00.026 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "throttled", 2024-09-18T08:12:00.026 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:06:04.696709+0000", 2024-09-18T08:12:00.026 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.4610000000000001e-06 2024-09-18T08:12:00.026 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:12:00.026 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:12:00.026 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "all_read", 2024-09-18T08:12:00.026 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:06:04.696714+0000", 2024-09-18T08:12:00.026 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 5.4929999999999998e-06 2024-09-18T08:12:00.026 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:12:00.026 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:12:00.026 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "dispatched", 2024-09-18T08:12:00.026 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:06:04.696726+0000", 2024-09-18T08:12:00.027 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.1405e-05 2024-09-18T08:12:00.027 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:12:00.027 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:12:00.027 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "queued_for_pg", 2024-09-18T08:12:00.027 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:06:04.696733+0000", 2024-09-18T08:12:00.027 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 7.5440000000000001e-06 2024-09-18T08:12:00.027 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:12:00.027 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:12:00.027 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-09-18T08:12:00.027 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:06:04.696834+0000", 2024-09-18T08:12:00.027 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.00010042799999999999 2024-09-18T08:12:00.027 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:12:00.028 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:12:00.028 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "started", 2024-09-18T08:12:00.028 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:06:04.696964+0000", 2024-09-18T08:12:00.028 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.00013050599999999999 2024-09-18T08:12:00.028 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:12:00.028 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:12:00.028 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "waiting for subops from 0,5", 2024-09-18T08:12:00.028 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:06:04.697079+0000", 2024-09-18T08:12:00.028 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.000114769 2024-09-18T08:12:00.028 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:12:00.028 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:12:00.028 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:12:00.029 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:06:04.703144+0000", 2024-09-18T08:12:00.029 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.0060647729999999999 2024-09-18T08:12:00.029 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:12:00.029 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:12:00.029 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "op_commit", 2024-09-18T08:12:00.029 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:06:04.703157+0000", 2024-09-18T08:12:00.029 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.3434e-05 2024-09-18T08:12:00.029 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:12:00.029 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:12:00.030 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:12:00.030 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:06:04.789746+0000", 2024-09-18T08:12:00.030 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.086589049000000001 2024-09-18T08:12:00.030 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:12:00.030 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:12:00.030 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "commit_sent", 2024-09-18T08:12:00.030 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:06:04.789946+0000", 2024-09-18T08:12:00.031 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.000199996 2024-09-18T08:12:00.031 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:12:00.031 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:12:00.031 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "done", 2024-09-18T08:12:00.031 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:06:04.790188+0000", 2024-09-18T08:12:00.031 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.00024219000000000001 2024-09-18T08:12:00.031 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-09-18T08:12:00.031 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-09-18T08:12:00.031 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-09-18T08:12:00.031 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:12:00.031 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:12:00.031 INFO:teuthology.orchestra.run.smithi007.stdout: "description": "osd_op(client.14805.0:147846 7.e 7:77e37675:::benchmark_data_smithi007_73155_object29555:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e127)", 2024-09-18T08:12:00.031 INFO:teuthology.orchestra.run.smithi007.stdout: "initiated_at": "2024-09-18T08:06:04.708478+0000", 2024-09-18T08:12:00.032 INFO:teuthology.orchestra.run.smithi007.stdout: "age": 355.05476803699997, 2024-09-18T08:12:00.032 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.089318937000000001, 2024-09-18T08:12:00.032 INFO:teuthology.orchestra.run.smithi007.stdout: "continuous": false, 2024-09-18T08:12:00.032 INFO:teuthology.orchestra.run.smithi007.stdout: "type_data": { 2024-09-18T08:12:00.032 INFO:teuthology.orchestra.run.smithi007.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T08:12:00.032 INFO:teuthology.orchestra.run.smithi007.stdout: "client_info": { 2024-09-18T08:12:00.032 INFO:teuthology.orchestra.run.smithi007.stdout: "client": "client.14805", 2024-09-18T08:12:00.032 INFO:teuthology.orchestra.run.smithi007.stdout: "client_addr": "172.21.15.7:0/640320436", 2024-09-18T08:12:00.032 INFO:teuthology.orchestra.run.smithi007.stdout: "tid": 147846 2024-09-18T08:12:00.032 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:12:00.032 INFO:teuthology.orchestra.run.smithi007.stdout: "events": [ 2024-09-18T08:12:00.032 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:12:00.033 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "initiated", 2024-09-18T08:12:00.033 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:06:04.708478+0000", 2024-09-18T08:12:00.033 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-09-18T08:12:00.033 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:12:00.033 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:12:00.033 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "header_read", 2024-09-18T08:12:00.033 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:06:04.708478+0000", 2024-09-18T08:12:00.033 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-09-18T08:12:00.033 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:12:00.033 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:12:00.033 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "throttled", 2024-09-18T08:12:00.033 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:06:04.708480+0000", 2024-09-18T08:12:00.033 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.3799999999999999e-06 2024-09-18T08:12:00.034 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:12:00.034 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:12:00.034 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "all_read", 2024-09-18T08:12:00.034 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:06:04.708485+0000", 2024-09-18T08:12:00.034 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 5.5140000000000001e-06 2024-09-18T08:12:00.034 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:12:00.034 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:12:00.034 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "dispatched", 2024-09-18T08:12:00.034 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:06:04.708496+0000", 2024-09-18T08:12:00.034 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.0468e-05 2024-09-18T08:12:00.034 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:12:00.034 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:12:00.034 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "queued_for_pg", 2024-09-18T08:12:00.035 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:06:04.708503+0000", 2024-09-18T08:12:00.035 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 6.9709999999999998e-06 2024-09-18T08:12:00.035 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:12:00.035 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:12:00.035 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-09-18T08:12:00.035 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:06:04.708559+0000", 2024-09-18T08:12:00.035 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 5.5967999999999997e-05 2024-09-18T08:12:00.035 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:12:00.035 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:12:00.035 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "started", 2024-09-18T08:12:00.035 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:06:04.708664+0000", 2024-09-18T08:12:00.035 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.000104956 2024-09-18T08:12:00.036 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:12:00.036 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:12:00.036 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "waiting for subops from 0,5", 2024-09-18T08:12:00.036 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:06:04.708767+0000", 2024-09-18T08:12:00.036 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.000102889 2024-09-18T08:12:00.036 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:12:00.036 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:12:00.036 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:12:00.036 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:06:04.781177+0000", 2024-09-18T08:12:00.037 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.072410522000000005 2024-09-18T08:12:00.037 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:12:00.037 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:12:00.037 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "op_commit", 2024-09-18T08:12:00.037 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:06:04.782476+0000", 2024-09-18T08:12:00.037 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.0012993150000000001 2024-09-18T08:12:00.037 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:12:00.037 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:12:00.037 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:12:00.037 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:06:04.797423+0000", 2024-09-18T08:12:00.037 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.014946213999999999 2024-09-18T08:12:00.037 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:12:00.037 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:12:00.038 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "commit_sent", 2024-09-18T08:12:00.038 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:06:04.797637+0000", 2024-09-18T08:12:00.038 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.000214353 2024-09-18T08:12:00.038 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:12:00.038 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:12:00.038 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "done", 2024-09-18T08:12:00.038 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:06:04.797797+0000", 2024-09-18T08:12:00.038 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.00016038700000000001 2024-09-18T08:12:00.038 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-09-18T08:12:00.038 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-09-18T08:12:00.038 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-09-18T08:12:00.038 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:12:00.038 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:12:00.039 INFO:teuthology.orchestra.run.smithi007.stdout: "description": "osd_op(client.14805.0:147849 7.e 7:7e6f39e7:::benchmark_data_smithi007_73155_object29558:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e127)", 2024-09-18T08:12:00.039 INFO:teuthology.orchestra.run.smithi007.stdout: "initiated_at": "2024-09-18T08:06:04.708908+0000", 2024-09-18T08:12:00.039 INFO:teuthology.orchestra.run.smithi007.stdout: "age": 355.05433875300002, 2024-09-18T08:12:00.039 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.089203497000000007, 2024-09-18T08:12:00.039 INFO:teuthology.orchestra.run.smithi007.stdout: "continuous": false, 2024-09-18T08:12:00.039 INFO:teuthology.orchestra.run.smithi007.stdout: "type_data": { 2024-09-18T08:12:00.039 INFO:teuthology.orchestra.run.smithi007.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T08:12:00.039 INFO:teuthology.orchestra.run.smithi007.stdout: "client_info": { 2024-09-18T08:12:00.039 INFO:teuthology.orchestra.run.smithi007.stdout: "client": "client.14805", 2024-09-18T08:12:00.039 INFO:teuthology.orchestra.run.smithi007.stdout: "client_addr": "172.21.15.7:0/640320436", 2024-09-18T08:12:00.039 INFO:teuthology.orchestra.run.smithi007.stdout: "tid": 147849 2024-09-18T08:12:00.039 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:12:00.039 INFO:teuthology.orchestra.run.smithi007.stdout: "events": [ 2024-09-18T08:12:00.040 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:12:00.040 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "initiated", 2024-09-18T08:12:00.040 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:06:04.708908+0000", 2024-09-18T08:12:00.040 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-09-18T08:12:00.040 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:12:00.040 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:12:00.040 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "header_read", 2024-09-18T08:12:00.040 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:06:04.708908+0000", 2024-09-18T08:12:00.040 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-09-18T08:12:00.040 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:12:00.040 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:12:00.040 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "throttled", 2024-09-18T08:12:00.040 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:06:04.708908+0000", 2024-09-18T08:12:00.041 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 7.6199999999999997e-07 2024-09-18T08:12:00.041 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:12:00.041 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:12:00.041 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "all_read", 2024-09-18T08:12:00.041 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:06:04.708913+0000", 2024-09-18T08:12:00.041 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 4.1049999999999997e-06 2024-09-18T08:12:00.041 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:12:00.041 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:12:00.041 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "dispatched", 2024-09-18T08:12:00.041 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:06:04.708922+0000", 2024-09-18T08:12:00.041 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 9.2930000000000006e-06 2024-09-18T08:12:00.041 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:12:00.041 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:12:00.042 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "queued_for_pg", 2024-09-18T08:12:00.042 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:06:04.708928+0000", 2024-09-18T08:12:00.042 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 6.4919999999999998e-06 2024-09-18T08:12:00.042 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:12:00.042 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:12:00.042 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-09-18T08:12:00.042 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:06:04.712299+0000", 2024-09-18T08:12:00.042 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.0033709510000000001 2024-09-18T08:12:00.042 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:12:00.042 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:12:00.042 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "started", 2024-09-18T08:12:00.042 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:06:04.712407+0000", 2024-09-18T08:12:00.042 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.000108229 2024-09-18T08:12:00.043 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:12:00.043 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:12:00.043 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "waiting for subops from 0,5", 2024-09-18T08:12:00.043 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:06:04.712519+0000", 2024-09-18T08:12:00.043 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.00011200399999999999 2024-09-18T08:12:00.043 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:12:00.043 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:12:00.043 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:12:00.043 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:06:04.782460+0000", 2024-09-18T08:12:00.043 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.069940536999999997 2024-09-18T08:12:00.043 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:12:00.043 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:12:00.043 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "op_commit", 2024-09-18T08:12:00.044 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:06:04.782500+0000", 2024-09-18T08:12:00.044 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 4.0463000000000002e-05 2024-09-18T08:12:00.044 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:12:00.044 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:12:00.044 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:12:00.044 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:06:04.797935+0000", 2024-09-18T08:12:00.044 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.015434188 2024-09-18T08:12:00.044 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:12:00.044 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:12:00.044 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "commit_sent", 2024-09-18T08:12:00.044 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:06:04.798073+0000", 2024-09-18T08:12:00.044 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.00013849000000000001 2024-09-18T08:12:00.044 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:12:00.045 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:12:00.045 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "done", 2024-09-18T08:12:00.045 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:06:04.798111+0000", 2024-09-18T08:12:00.045 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 3.7982999999999999e-05 2024-09-18T08:12:00.045 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-09-18T08:12:00.045 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-09-18T08:12:00.045 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-09-18T08:12:00.045 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:12:00.045 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:12:00.045 INFO:teuthology.orchestra.run.smithi007.stdout: "description": "osd_op(client.14805.0:177709 7.e 7:780b878d:::benchmark_data_smithi007_73155_object59418:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e139)", 2024-09-18T08:12:00.045 INFO:teuthology.orchestra.run.smithi007.stdout: "initiated_at": "2024-09-18T08:06:38.569829+0000", 2024-09-18T08:12:00.045 INFO:teuthology.orchestra.run.smithi007.stdout: "age": 321.19341720699998, 2024-09-18T08:12:00.045 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.111834402, 2024-09-18T08:12:00.046 INFO:teuthology.orchestra.run.smithi007.stdout: "continuous": false, 2024-09-18T08:12:00.046 INFO:teuthology.orchestra.run.smithi007.stdout: "type_data": { 2024-09-18T08:12:00.046 INFO:teuthology.orchestra.run.smithi007.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T08:12:00.046 INFO:teuthology.orchestra.run.smithi007.stdout: "client_info": { 2024-09-18T08:12:00.046 INFO:teuthology.orchestra.run.smithi007.stdout: "client": "client.14805", 2024-09-18T08:12:00.046 INFO:teuthology.orchestra.run.smithi007.stdout: "client_addr": "172.21.15.7:0/640320436", 2024-09-18T08:12:00.046 INFO:teuthology.orchestra.run.smithi007.stdout: "tid": 177709 2024-09-18T08:12:00.046 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:12:00.046 INFO:teuthology.orchestra.run.smithi007.stdout: "events": [ 2024-09-18T08:12:00.046 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:12:00.046 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "initiated", 2024-09-18T08:12:00.046 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:06:38.569829+0000", 2024-09-18T08:12:00.046 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-09-18T08:12:00.046 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:12:00.047 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:12:00.047 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "header_read", 2024-09-18T08:12:00.047 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:06:38.569829+0000", 2024-09-18T08:12:00.047 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-09-18T08:12:00.047 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:12:00.047 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:12:00.047 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "throttled", 2024-09-18T08:12:00.047 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:06:38.569831+0000", 2024-09-18T08:12:00.047 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.451e-06 2024-09-18T08:12:00.047 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:12:00.047 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:12:00.047 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "all_read", 2024-09-18T08:12:00.047 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:06:38.569836+0000", 2024-09-18T08:12:00.048 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 5.4269999999999999e-06 2024-09-18T08:12:00.048 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:12:00.048 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:12:00.048 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "dispatched", 2024-09-18T08:12:00.048 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:06:38.569852+0000", 2024-09-18T08:12:00.048 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.6375000000000002e-05 2024-09-18T08:12:00.048 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:12:00.048 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:12:00.048 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "queued_for_pg", 2024-09-18T08:12:00.048 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:06:38.569861+0000", 2024-09-18T08:12:00.048 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 8.2749999999999995e-06 2024-09-18T08:12:00.048 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:12:00.048 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:12:00.049 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-09-18T08:12:00.049 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:06:38.571353+0000", 2024-09-18T08:12:00.049 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.0014927650000000001 2024-09-18T08:12:00.049 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:12:00.049 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:12:00.049 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "started", 2024-09-18T08:12:00.049 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:06:38.571454+0000", 2024-09-18T08:12:00.049 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.000100065 2024-09-18T08:12:00.049 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:12:00.049 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:12:00.049 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "waiting for subops from 0,5", 2024-09-18T08:12:00.049 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:06:38.571551+0000", 2024-09-18T08:12:00.049 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 9.7884e-05 2024-09-18T08:12:00.050 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:12:00.050 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:12:00.050 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:12:00.050 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:06:38.576046+0000", 2024-09-18T08:12:00.050 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.0044943379999999996 2024-09-18T08:12:00.050 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:12:00.050 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:12:00.050 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "op_commit", 2024-09-18T08:12:00.050 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:06:38.576188+0000", 2024-09-18T08:12:00.050 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.00014268500000000001 2024-09-18T08:12:00.050 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:12:00.050 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:12:00.050 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:12:00.050 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:06:38.681566+0000", 2024-09-18T08:12:00.051 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.10537708 2024-09-18T08:12:00.051 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:12:00.051 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:12:00.051 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "commit_sent", 2024-09-18T08:12:00.051 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:06:38.681619+0000", 2024-09-18T08:12:00.051 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 5.3919999999999999e-05 2024-09-18T08:12:00.051 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:12:00.051 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:12:00.051 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "done", 2024-09-18T08:12:00.051 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:06:38.681664+0000", 2024-09-18T08:12:00.051 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 4.4137000000000001e-05 2024-09-18T08:12:00.051 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-09-18T08:12:00.051 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-09-18T08:12:00.052 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-09-18T08:12:00.052 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:12:00.052 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:12:00.052 INFO:teuthology.orchestra.run.smithi007.stdout: "description": "osd_op(client.14805.0:204554 7.e 7:75af683c:::benchmark_data_smithi007_73155_object86263:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e148)", 2024-09-18T08:12:00.052 INFO:teuthology.orchestra.run.smithi007.stdout: "initiated_at": "2024-09-18T08:07:06.550254+0000", 2024-09-18T08:12:00.052 INFO:teuthology.orchestra.run.smithi007.stdout: "age": 293.21299218000001, 2024-09-18T08:12:00.052 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.090495268000000004, 2024-09-18T08:12:00.052 INFO:teuthology.orchestra.run.smithi007.stdout: "continuous": false, 2024-09-18T08:12:00.052 INFO:teuthology.orchestra.run.smithi007.stdout: "type_data": { 2024-09-18T08:12:00.052 INFO:teuthology.orchestra.run.smithi007.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T08:12:00.052 INFO:teuthology.orchestra.run.smithi007.stdout: "client_info": { 2024-09-18T08:12:00.052 INFO:teuthology.orchestra.run.smithi007.stdout: "client": "client.14805", 2024-09-18T08:12:00.052 INFO:teuthology.orchestra.run.smithi007.stdout: "client_addr": "172.21.15.7:0/640320436", 2024-09-18T08:12:00.052 INFO:teuthology.orchestra.run.smithi007.stdout: "tid": 204554 2024-09-18T08:12:00.053 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:12:00.053 INFO:teuthology.orchestra.run.smithi007.stdout: "events": [ 2024-09-18T08:12:00.053 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:12:00.053 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "initiated", 2024-09-18T08:12:00.053 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:07:06.550254+0000", 2024-09-18T08:12:00.053 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-09-18T08:12:00.053 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:12:00.053 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:12:00.053 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "header_read", 2024-09-18T08:12:00.053 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:07:06.550254+0000", 2024-09-18T08:12:00.053 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-09-18T08:12:00.053 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:12:00.053 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:12:00.054 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "throttled", 2024-09-18T08:12:00.054 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:07:06.550255+0000", 2024-09-18T08:12:00.054 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.04e-06 2024-09-18T08:12:00.054 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:12:00.054 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:12:00.054 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "all_read", 2024-09-18T08:12:00.054 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:07:06.550260+0000", 2024-09-18T08:12:00.054 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 4.4939999999999997e-06 2024-09-18T08:12:00.054 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:12:00.054 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:12:00.054 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "dispatched", 2024-09-18T08:12:00.054 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:07:06.550272+0000", 2024-09-18T08:12:00.054 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.2672e-05 2024-09-18T08:12:00.054 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:12:00.055 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:12:00.055 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "queued_for_pg", 2024-09-18T08:12:00.055 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:07:06.550279+0000", 2024-09-18T08:12:00.055 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 7.0099999999999998e-06 2024-09-18T08:12:00.055 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:12:00.055 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:12:00.055 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-09-18T08:12:00.055 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:07:06.553248+0000", 2024-09-18T08:12:00.055 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.0029683999999999999 2024-09-18T08:12:00.055 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:12:00.055 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:12:00.055 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "started", 2024-09-18T08:12:00.055 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:07:06.553346+0000", 2024-09-18T08:12:00.056 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 9.8618999999999999e-05 2024-09-18T08:12:00.056 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:12:00.056 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:12:00.056 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "waiting for subops from 0,5", 2024-09-18T08:12:00.056 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:07:06.553450+0000", 2024-09-18T08:12:00.056 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.00010311599999999999 2024-09-18T08:12:00.056 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:12:00.056 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:12:00.056 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "op_commit", 2024-09-18T08:12:00.056 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:07:06.560605+0000", 2024-09-18T08:12:00.056 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.0071556229999999998 2024-09-18T08:12:00.056 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:12:00.056 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:12:00.056 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:12:00.057 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:07:06.565908+0000", 2024-09-18T08:12:00.057 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.0053023480000000001 2024-09-18T08:12:00.057 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:12:00.057 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:12:00.057 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:12:00.057 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:07:06.640602+0000", 2024-09-18T08:12:00.057 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.074694521 2024-09-18T08:12:00.057 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:12:00.057 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:12:00.057 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "commit_sent", 2024-09-18T08:12:00.057 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:07:06.640706+0000", 2024-09-18T08:12:00.057 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.00010401800000000001 2024-09-18T08:12:00.057 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:12:00.058 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:12:00.058 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "done", 2024-09-18T08:12:00.058 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:07:06.640749+0000", 2024-09-18T08:12:00.058 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 4.3407e-05 2024-09-18T08:12:00.058 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-09-18T08:12:00.058 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-09-18T08:12:00.058 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-09-18T08:12:00.058 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:12:00.058 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:12:00.058 INFO:teuthology.orchestra.run.smithi007.stdout: "description": "osd_op(client.14805.0:204569 7.5 7:a57af9cd:::benchmark_data_smithi007_73155_object86278:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e148)", 2024-09-18T08:12:00.058 INFO:teuthology.orchestra.run.smithi007.stdout: "initiated_at": "2024-09-18T08:07:06.563553+0000", 2024-09-18T08:12:00.058 INFO:teuthology.orchestra.run.smithi007.stdout: "age": 293.19969309700002, 2024-09-18T08:12:00.058 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.089587238, 2024-09-18T08:12:00.059 INFO:teuthology.orchestra.run.smithi007.stdout: "continuous": false, 2024-09-18T08:12:00.059 INFO:teuthology.orchestra.run.smithi007.stdout: "type_data": { 2024-09-18T08:12:00.059 INFO:teuthology.orchestra.run.smithi007.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T08:12:00.059 INFO:teuthology.orchestra.run.smithi007.stdout: "client_info": { 2024-09-18T08:12:00.059 INFO:teuthology.orchestra.run.smithi007.stdout: "client": "client.14805", 2024-09-18T08:12:00.059 INFO:teuthology.orchestra.run.smithi007.stdout: "client_addr": "172.21.15.7:0/640320436", 2024-09-18T08:12:00.059 INFO:teuthology.orchestra.run.smithi007.stdout: "tid": 204569 2024-09-18T08:12:00.059 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:12:00.059 INFO:teuthology.orchestra.run.smithi007.stdout: "events": [ 2024-09-18T08:12:00.059 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:12:00.059 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "initiated", 2024-09-18T08:12:00.059 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:07:06.563553+0000", 2024-09-18T08:12:00.059 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-09-18T08:12:00.060 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:12:00.060 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:12:00.060 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "header_read", 2024-09-18T08:12:00.060 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:07:06.563553+0000", 2024-09-18T08:12:00.060 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-09-18T08:12:00.060 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:12:00.060 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:12:00.060 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "throttled", 2024-09-18T08:12:00.060 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:07:06.563554+0000", 2024-09-18T08:12:00.060 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 4.8100000000000003e-07 2024-09-18T08:12:00.060 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:12:00.060 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:12:00.060 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "all_read", 2024-09-18T08:12:00.060 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:07:06.563557+0000", 2024-09-18T08:12:00.061 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 2.7839999999999999e-06 2024-09-18T08:12:00.061 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:12:00.061 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:12:00.061 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "dispatched", 2024-09-18T08:12:00.061 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:07:06.563562+0000", 2024-09-18T08:12:00.061 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 5.7289999999999997e-06 2024-09-18T08:12:00.061 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:12:00.061 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:12:00.061 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "queued_for_pg", 2024-09-18T08:12:00.061 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:07:06.563567+0000", 2024-09-18T08:12:00.061 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 4.9949999999999996e-06 2024-09-18T08:12:00.061 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:12:00.061 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:12:00.061 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-09-18T08:12:00.062 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:07:06.563655+0000", 2024-09-18T08:12:00.062 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 8.7689999999999996e-05 2024-09-18T08:12:00.062 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:12:00.062 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:12:00.062 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "started", 2024-09-18T08:12:00.062 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:07:06.563758+0000", 2024-09-18T08:12:00.062 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.000102538 2024-09-18T08:12:00.062 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:12:00.062 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:12:00.062 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "waiting for subops from 0,7", 2024-09-18T08:12:00.062 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:07:06.563861+0000", 2024-09-18T08:12:00.062 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.00010385799999999999 2024-09-18T08:12:00.063 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:12:00.063 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:12:00.063 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:12:00.063 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:07:06.570416+0000", 2024-09-18T08:12:00.063 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.0065544449999999999 2024-09-18T08:12:00.063 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:12:00.063 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:12:00.063 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "op_commit", 2024-09-18T08:12:00.063 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:07:06.570428+0000", 2024-09-18T08:12:00.063 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.2537999999999999e-05 2024-09-18T08:12:00.063 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:12:00.063 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:12:00.063 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:12:00.063 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:07:06.649777+0000", 2024-09-18T08:12:00.064 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.079348518000000007 2024-09-18T08:12:00.064 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:12:00.064 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:12:00.064 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "commit_sent", 2024-09-18T08:12:00.064 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:07:06.653092+0000", 2024-09-18T08:12:00.064 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.0033149939999999999 2024-09-18T08:12:00.064 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:12:00.064 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:12:00.064 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "done", 2024-09-18T08:12:00.064 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:07:06.653141+0000", 2024-09-18T08:12:00.064 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 4.8668000000000002e-05 2024-09-18T08:12:00.064 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-09-18T08:12:00.064 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-09-18T08:12:00.065 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-09-18T08:12:00.065 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:12:00.065 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:12:00.065 INFO:teuthology.orchestra.run.smithi007.stdout: "description": "osd_op(client.14805.0:204572 7.5 7:a6f477fa:::benchmark_data_smithi007_73155_object86281:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e148)", 2024-09-18T08:12:00.065 INFO:teuthology.orchestra.run.smithi007.stdout: "initiated_at": "2024-09-18T08:07:06.565647+0000", 2024-09-18T08:12:00.065 INFO:teuthology.orchestra.run.smithi007.stdout: "age": 293.19759982599999, 2024-09-18T08:12:00.065 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.087897958999999998, 2024-09-18T08:12:00.065 INFO:teuthology.orchestra.run.smithi007.stdout: "continuous": false, 2024-09-18T08:12:00.065 INFO:teuthology.orchestra.run.smithi007.stdout: "type_data": { 2024-09-18T08:12:00.065 INFO:teuthology.orchestra.run.smithi007.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T08:12:00.065 INFO:teuthology.orchestra.run.smithi007.stdout: "client_info": { 2024-09-18T08:12:00.065 INFO:teuthology.orchestra.run.smithi007.stdout: "client": "client.14805", 2024-09-18T08:12:00.065 INFO:teuthology.orchestra.run.smithi007.stdout: "client_addr": "172.21.15.7:0/640320436", 2024-09-18T08:12:00.066 INFO:teuthology.orchestra.run.smithi007.stdout: "tid": 204572 2024-09-18T08:12:00.066 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:12:00.066 INFO:teuthology.orchestra.run.smithi007.stdout: "events": [ 2024-09-18T08:12:00.066 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:12:00.066 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "initiated", 2024-09-18T08:12:00.066 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:07:06.565647+0000", 2024-09-18T08:12:00.066 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-09-18T08:12:00.066 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:12:00.066 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:12:00.066 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "header_read", 2024-09-18T08:12:00.066 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:07:06.565647+0000", 2024-09-18T08:12:00.066 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-09-18T08:12:00.066 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:12:00.066 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:12:00.067 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "throttled", 2024-09-18T08:12:00.067 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:07:06.565647+0000", 2024-09-18T08:12:00.067 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 7.0800000000000004e-07 2024-09-18T08:12:00.067 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:12:00.067 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:12:00.067 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "all_read", 2024-09-18T08:12:00.067 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:07:06.565651+0000", 2024-09-18T08:12:00.067 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 3.3299999999999999e-06 2024-09-18T08:12:00.067 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:12:00.067 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:12:00.067 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "dispatched", 2024-09-18T08:12:00.067 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:07:06.565659+0000", 2024-09-18T08:12:00.067 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 8.7250000000000003e-06 2024-09-18T08:12:00.067 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:12:00.068 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:12:00.068 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "queued_for_pg", 2024-09-18T08:12:00.068 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:07:06.565665+0000", 2024-09-18T08:12:00.068 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 5.7259999999999999e-06 2024-09-18T08:12:00.068 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:12:00.068 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:12:00.068 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-09-18T08:12:00.068 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:07:06.567127+0000", 2024-09-18T08:12:00.068 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.001461604 2024-09-18T08:12:00.068 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:12:00.068 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:12:00.068 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "started", 2024-09-18T08:12:00.068 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:07:06.567226+0000", 2024-09-18T08:12:00.069 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 9.9807000000000002e-05 2024-09-18T08:12:00.069 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:12:00.069 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:12:00.069 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "waiting for subops from 0,7", 2024-09-18T08:12:00.069 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:07:06.567324+0000", 2024-09-18T08:12:00.069 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 9.7994999999999999e-05 2024-09-18T08:12:00.069 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:12:00.069 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:12:00.069 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:12:00.069 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:07:06.570467+0000", 2024-09-18T08:12:00.069 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.0031425300000000001 2024-09-18T08:12:00.069 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:12:00.069 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:12:00.070 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "op_commit", 2024-09-18T08:12:00.070 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:07:06.570486+0000", 2024-09-18T08:12:00.070 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.906e-05 2024-09-18T08:12:00.070 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:12:00.070 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:12:00.070 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:12:00.070 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:07:06.653186+0000", 2024-09-18T08:12:00.070 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.082700138000000006 2024-09-18T08:12:00.070 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:12:00.070 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:12:00.070 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "commit_sent", 2024-09-18T08:12:00.070 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:07:06.653506+0000", 2024-09-18T08:12:00.070 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.00031962000000000001 2024-09-18T08:12:00.071 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:12:00.071 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:12:00.071 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "done", 2024-09-18T08:12:00.071 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:07:06.653545+0000", 2024-09-18T08:12:00.071 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 3.8716e-05 2024-09-18T08:12:00.071 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-09-18T08:12:00.071 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-09-18T08:12:00.071 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-09-18T08:12:00.071 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:12:00.071 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:12:00.072 INFO:teuthology.orchestra.run.smithi007.stdout: "description": "osd_op(client.14805.0:223793 7.5 7:a37824ff:::benchmark_data_smithi007_73155_object105502:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e154)", 2024-09-18T08:12:00.072 INFO:teuthology.orchestra.run.smithi007.stdout: "initiated_at": "2024-09-18T08:07:26.183612+0000", 2024-09-18T08:12:00.072 INFO:teuthology.orchestra.run.smithi007.stdout: "age": 273.57963462100003, 2024-09-18T08:12:00.072 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.104886888, 2024-09-18T08:12:00.072 INFO:teuthology.orchestra.run.smithi007.stdout: "continuous": false, 2024-09-18T08:12:00.072 INFO:teuthology.orchestra.run.smithi007.stdout: "type_data": { 2024-09-18T08:12:00.072 INFO:teuthology.orchestra.run.smithi007.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T08:12:00.072 INFO:teuthology.orchestra.run.smithi007.stdout: "client_info": { 2024-09-18T08:12:00.072 INFO:teuthology.orchestra.run.smithi007.stdout: "client": "client.14805", 2024-09-18T08:12:00.072 INFO:teuthology.orchestra.run.smithi007.stdout: "client_addr": "172.21.15.7:0/640320436", 2024-09-18T08:12:00.072 INFO:teuthology.orchestra.run.smithi007.stdout: "tid": 223793 2024-09-18T08:12:00.072 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:12:00.073 INFO:teuthology.orchestra.run.smithi007.stdout: "events": [ 2024-09-18T08:12:00.073 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:12:00.073 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "initiated", 2024-09-18T08:12:00.073 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:07:26.183612+0000", 2024-09-18T08:12:00.073 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-09-18T08:12:00.073 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:12:00.073 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:12:00.073 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "header_read", 2024-09-18T08:12:00.073 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:07:26.183612+0000", 2024-09-18T08:12:00.073 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-09-18T08:12:00.073 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:12:00.073 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:12:00.073 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "throttled", 2024-09-18T08:12:00.074 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:07:26.183613+0000", 2024-09-18T08:12:00.074 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.3030000000000001e-06 2024-09-18T08:12:00.074 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:12:00.074 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:12:00.074 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "all_read", 2024-09-18T08:12:00.074 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:07:26.183618+0000", 2024-09-18T08:12:00.074 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 4.7870000000000001e-06 2024-09-18T08:12:00.074 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:12:00.074 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:12:00.074 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "dispatched", 2024-09-18T08:12:00.074 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:07:26.183629+0000", 2024-09-18T08:12:00.074 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.1100999999999999e-05 2024-09-18T08:12:00.074 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:12:00.075 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:12:00.075 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "queued_for_pg", 2024-09-18T08:12:00.075 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:07:26.183636+0000", 2024-09-18T08:12:00.075 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 7.3980000000000002e-06 2024-09-18T08:12:00.075 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:12:00.075 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:12:00.075 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-09-18T08:12:00.075 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:07:26.183957+0000", 2024-09-18T08:12:00.075 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.00032060999999999998 2024-09-18T08:12:00.075 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:12:00.075 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:12:00.075 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "started", 2024-09-18T08:12:00.076 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:07:26.184232+0000", 2024-09-18T08:12:00.076 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.00027534499999999997 2024-09-18T08:12:00.076 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:12:00.076 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:12:00.076 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "waiting for subops from 0,7", 2024-09-18T08:12:00.076 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:07:26.184350+0000", 2024-09-18T08:12:00.076 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.00011803600000000001 2024-09-18T08:12:00.076 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:12:00.076 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:12:00.076 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:12:00.076 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:07:26.235645+0000", 2024-09-18T08:12:00.076 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.051294868 2024-09-18T08:12:00.076 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:12:00.076 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:12:00.077 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "op_commit", 2024-09-18T08:12:00.077 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:07:26.235664+0000", 2024-09-18T08:12:00.077 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.9236e-05 2024-09-18T08:12:00.077 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:12:00.077 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:12:00.077 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:12:00.077 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:07:26.288261+0000", 2024-09-18T08:12:00.077 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.052596716000000002 2024-09-18T08:12:00.077 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:12:00.077 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:12:00.077 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "commit_sent", 2024-09-18T08:12:00.077 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:07:26.288452+0000", 2024-09-18T08:12:00.077 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.00019043700000000001 2024-09-18T08:12:00.078 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:12:00.078 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:12:00.078 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "done", 2024-09-18T08:12:00.078 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:07:26.288499+0000", 2024-09-18T08:12:00.078 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 4.7051000000000001e-05 2024-09-18T08:12:00.078 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-09-18T08:12:00.078 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-09-18T08:12:00.078 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-09-18T08:12:00.078 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:12:00.078 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:12:00.078 INFO:teuthology.orchestra.run.smithi007.stdout: "description": "osd_op(client.14805.0:223799 7.e 7:7f447ed9:::benchmark_data_smithi007_73155_object105508:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e154)", 2024-09-18T08:12:00.078 INFO:teuthology.orchestra.run.smithi007.stdout: "initiated_at": "2024-09-18T08:07:26.184566+0000", 2024-09-18T08:12:00.078 INFO:teuthology.orchestra.run.smithi007.stdout: "age": 273.57868065500003, 2024-09-18T08:12:00.079 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.114658156, 2024-09-18T08:12:00.079 INFO:teuthology.orchestra.run.smithi007.stdout: "continuous": false, 2024-09-18T08:12:00.079 INFO:teuthology.orchestra.run.smithi007.stdout: "type_data": { 2024-09-18T08:12:00.079 INFO:teuthology.orchestra.run.smithi007.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T08:12:00.079 INFO:teuthology.orchestra.run.smithi007.stdout: "client_info": { 2024-09-18T08:12:00.079 INFO:teuthology.orchestra.run.smithi007.stdout: "client": "client.14805", 2024-09-18T08:12:00.079 INFO:teuthology.orchestra.run.smithi007.stdout: "client_addr": "172.21.15.7:0/640320436", 2024-09-18T08:12:00.079 INFO:teuthology.orchestra.run.smithi007.stdout: "tid": 223799 2024-09-18T08:12:00.079 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:12:00.079 INFO:teuthology.orchestra.run.smithi007.stdout: "events": [ 2024-09-18T08:12:00.079 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:12:00.079 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "initiated", 2024-09-18T08:12:00.079 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:07:26.184566+0000", 2024-09-18T08:12:00.080 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-09-18T08:12:00.080 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:12:00.080 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:12:00.080 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "header_read", 2024-09-18T08:12:00.080 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:07:26.184566+0000", 2024-09-18T08:12:00.080 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-09-18T08:12:00.080 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:12:00.080 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:12:00.080 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "throttled", 2024-09-18T08:12:00.080 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:07:26.184593+0000", 2024-09-18T08:12:00.080 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 2.6831000000000001e-05 2024-09-18T08:12:00.080 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:12:00.080 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:12:00.080 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "all_read", 2024-09-18T08:12:00.081 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:07:26.184597+0000", 2024-09-18T08:12:00.081 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 4.8119999999999998e-06 2024-09-18T08:12:00.081 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:12:00.081 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:12:00.081 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "dispatched", 2024-09-18T08:12:00.081 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:07:26.184607+0000", 2024-09-18T08:12:00.081 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 9.9559999999999992e-06 2024-09-18T08:12:00.081 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:12:00.081 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:12:00.081 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "queued_for_pg", 2024-09-18T08:12:00.081 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:07:26.184614+0000", 2024-09-18T08:12:00.081 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 6.7730000000000002e-06 2024-09-18T08:12:00.081 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:12:00.082 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:12:00.082 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-09-18T08:12:00.082 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:07:26.184669+0000", 2024-09-18T08:12:00.082 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 5.4719000000000002e-05 2024-09-18T08:12:00.082 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:12:00.082 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:12:00.082 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "started", 2024-09-18T08:12:00.082 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:07:26.184764+0000", 2024-09-18T08:12:00.082 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 9.5529e-05 2024-09-18T08:12:00.082 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:12:00.082 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:12:00.082 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "waiting for subops from 0,5", 2024-09-18T08:12:00.082 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:07:26.184872+0000", 2024-09-18T08:12:00.082 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.000108115 2024-09-18T08:12:00.083 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:12:00.083 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:12:00.083 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:12:00.083 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:07:26.235645+0000", 2024-09-18T08:12:00.083 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.050772920999999999 2024-09-18T08:12:00.083 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:12:00.083 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:12:00.083 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "op_commit", 2024-09-18T08:12:00.083 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:07:26.235665+0000", 2024-09-18T08:12:00.083 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.9585999999999998e-05 2024-09-18T08:12:00.083 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:12:00.083 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:12:00.083 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:12:00.083 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:07:26.298883+0000", 2024-09-18T08:12:00.084 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.063217624 2024-09-18T08:12:00.084 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:12:00.084 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:12:00.084 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "commit_sent", 2024-09-18T08:12:00.084 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:07:26.299173+0000", 2024-09-18T08:12:00.084 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.00029084500000000002 2024-09-18T08:12:00.084 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:12:00.084 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:12:00.084 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "done", 2024-09-18T08:12:00.084 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:07:26.299224+0000", 2024-09-18T08:12:00.084 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 5.0445000000000002e-05 2024-09-18T08:12:00.084 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-09-18T08:12:00.084 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-09-18T08:12:00.084 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-09-18T08:12:00.085 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:12:00.085 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:12:00.085 INFO:teuthology.orchestra.run.smithi007.stdout: "description": "osd_op(client.15582.0:53557 9.0 9:0cf6806d:::benchmark_data_smithi007_124019_object53556:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e244)", 2024-09-18T08:12:00.085 INFO:teuthology.orchestra.run.smithi007.stdout: "initiated_at": "2024-09-18T08:11:41.343691+0000", 2024-09-18T08:12:00.085 INFO:teuthology.orchestra.run.smithi007.stdout: "age": 18.419555614, 2024-09-18T08:12:00.085 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.98520017699999995, 2024-09-18T08:12:00.085 INFO:teuthology.orchestra.run.smithi007.stdout: "continuous": false, 2024-09-18T08:12:00.085 INFO:teuthology.orchestra.run.smithi007.stdout: "type_data": { 2024-09-18T08:12:00.085 INFO:teuthology.orchestra.run.smithi007.stdout: "flag_point": "reached pg", 2024-09-18T08:12:00.085 INFO:teuthology.orchestra.run.smithi007.stdout: "client_info": { 2024-09-18T08:12:00.085 INFO:teuthology.orchestra.run.smithi007.stdout: "client": "client.15582", 2024-09-18T08:12:00.085 INFO:teuthology.orchestra.run.smithi007.stdout: "client_addr": "172.21.15.7:0/3598605839", 2024-09-18T08:12:00.085 INFO:teuthology.orchestra.run.smithi007.stdout: "tid": 53557 2024-09-18T08:12:00.085 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:12:00.086 INFO:teuthology.orchestra.run.smithi007.stdout: "events": [ 2024-09-18T08:12:00.086 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:12:00.086 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "initiated", 2024-09-18T08:12:00.086 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:11:41.343691+0000", 2024-09-18T08:12:00.086 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-09-18T08:12:00.086 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:12:00.086 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:12:00.086 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "header_read", 2024-09-18T08:12:00.086 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:11:41.343691+0000", 2024-09-18T08:12:00.086 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-09-18T08:12:00.086 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:12:00.086 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:12:00.087 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "throttled", 2024-09-18T08:12:00.087 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:11:41.343693+0000", 2024-09-18T08:12:00.087 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.804e-06 2024-09-18T08:12:00.087 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:12:00.087 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:12:00.087 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "all_read", 2024-09-18T08:12:00.087 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:11:41.343759+0000", 2024-09-18T08:12:00.087 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 6.6369000000000001e-05 2024-09-18T08:12:00.087 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:12:00.087 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:12:00.087 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "dispatched", 2024-09-18T08:12:00.087 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:11:41.343774+0000", 2024-09-18T08:12:00.087 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.5489999999999999e-05 2024-09-18T08:12:00.087 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:12:00.088 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:12:00.088 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "queued_for_pg", 2024-09-18T08:12:00.088 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:11:41.343785+0000", 2024-09-18T08:12:00.088 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.078e-05 2024-09-18T08:12:00.088 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:12:00.088 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:12:00.088 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-09-18T08:12:00.088 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:11:41.344215+0000", 2024-09-18T08:12:00.088 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.00042940799999999999 2024-09-18T08:12:00.088 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:12:00.088 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:12:00.088 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "waiting for peered", 2024-09-18T08:12:00.088 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:11:41.344258+0000", 2024-09-18T08:12:00.089 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 4.3460000000000001e-05 2024-09-18T08:12:00.089 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:12:00.089 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:12:00.089 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-09-18T08:12:00.089 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:11:42.326886+0000", 2024-09-18T08:12:00.089 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.98262749800000004 2024-09-18T08:12:00.089 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:12:00.089 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:12:00.089 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "done", 2024-09-18T08:12:00.089 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:11:42.328891+0000", 2024-09-18T08:12:00.089 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.002005368 2024-09-18T08:12:00.089 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-09-18T08:12:00.089 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-09-18T08:12:00.089 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-09-18T08:12:00.089 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:12:00.090 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:12:00.090 INFO:teuthology.orchestra.run.smithi007.stdout: "description": "osd_op(client.15582.0:53566 9.0 9:09a05dd0:::benchmark_data_smithi007_124019_object53565:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e244)", 2024-09-18T08:12:00.090 INFO:teuthology.orchestra.run.smithi007.stdout: "initiated_at": "2024-09-18T08:11:41.343807+0000", 2024-09-18T08:12:00.090 INFO:teuthology.orchestra.run.smithi007.stdout: "age": 18.419439786000002, 2024-09-18T08:12:00.090 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.007200514, 2024-09-18T08:12:00.090 INFO:teuthology.orchestra.run.smithi007.stdout: "continuous": false, 2024-09-18T08:12:00.090 INFO:teuthology.orchestra.run.smithi007.stdout: "type_data": { 2024-09-18T08:12:00.090 INFO:teuthology.orchestra.run.smithi007.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T08:12:00.090 INFO:teuthology.orchestra.run.smithi007.stdout: "client_info": { 2024-09-18T08:12:00.090 INFO:teuthology.orchestra.run.smithi007.stdout: "client": "client.15582", 2024-09-18T08:12:00.090 INFO:teuthology.orchestra.run.smithi007.stdout: "client_addr": "172.21.15.7:0/3598605839", 2024-09-18T08:12:00.090 INFO:teuthology.orchestra.run.smithi007.stdout: "tid": 53566 2024-09-18T08:12:00.090 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:12:00.091 INFO:teuthology.orchestra.run.smithi007.stdout: "events": [ 2024-09-18T08:12:00.091 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:12:00.091 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "initiated", 2024-09-18T08:12:00.091 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:11:41.343807+0000", 2024-09-18T08:12:00.091 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-09-18T08:12:00.091 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:12:00.091 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:12:00.091 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "header_read", 2024-09-18T08:12:00.091 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:11:41.343807+0000", 2024-09-18T08:12:00.091 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-09-18T08:12:00.091 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:12:00.091 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:12:00.091 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "throttled", 2024-09-18T08:12:00.091 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:11:41.343807+0000", 2024-09-18T08:12:00.092 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 5.3000000000000001e-07 2024-09-18T08:12:00.092 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:12:00.092 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:12:00.092 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "all_read", 2024-09-18T08:12:00.092 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:11:41.343826+0000", 2024-09-18T08:12:00.092 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.9338999999999999e-05 2024-09-18T08:12:00.092 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:12:00.092 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:12:00.092 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "dispatched", 2024-09-18T08:12:00.092 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:11:41.343833+0000", 2024-09-18T08:12:00.092 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 6.7710000000000001e-06 2024-09-18T08:12:00.092 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:12:00.092 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:12:00.092 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "queued_for_pg", 2024-09-18T08:12:00.093 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:11:41.343839+0000", 2024-09-18T08:12:00.093 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 5.4269999999999999e-06 2024-09-18T08:12:00.093 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:12:00.093 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:12:00.093 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-09-18T08:12:00.093 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:11:41.344332+0000", 2024-09-18T08:12:00.093 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.00049369799999999999 2024-09-18T08:12:00.093 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:12:00.093 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:12:00.093 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "waiting for peered", 2024-09-18T08:12:00.093 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:11:41.344339+0000", 2024-09-18T08:12:00.093 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 6.8589999999999999e-06 2024-09-18T08:12:00.093 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:12:00.094 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:12:00.094 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-09-18T08:12:00.094 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:11:42.328960+0000", 2024-09-18T08:12:00.094 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.98462126599999999 2024-09-18T08:12:00.094 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:12:00.094 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:12:00.094 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "started", 2024-09-18T08:12:00.094 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:11:42.329310+0000", 2024-09-18T08:12:00.094 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.00034983799999999999 2024-09-18T08:12:00.094 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:12:00.094 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:12:00.094 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "waiting for subops from 6", 2024-09-18T08:12:00.094 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:11:42.330445+0000", 2024-09-18T08:12:00.095 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.0011349859999999999 2024-09-18T08:12:00.095 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:12:00.095 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:12:00.095 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "op_commit", 2024-09-18T08:12:00.095 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:11:42.339902+0000", 2024-09-18T08:12:00.095 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.0094567869999999995 2024-09-18T08:12:00.095 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:12:00.095 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:12:00.095 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:12:00.095 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:11:42.350663+0000", 2024-09-18T08:12:00.095 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.010761299 2024-09-18T08:12:00.095 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:12:00.095 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:12:00.096 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "commit_sent", 2024-09-18T08:12:00.096 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:11:42.350832+0000", 2024-09-18T08:12:00.096 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.00016828399999999999 2024-09-18T08:12:00.096 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:12:00.096 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:12:00.096 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "done", 2024-09-18T08:12:00.096 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:11:42.351007+0000", 2024-09-18T08:12:00.096 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.00017542999999999999 2024-09-18T08:12:00.096 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-09-18T08:12:00.096 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-09-18T08:12:00.096 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-09-18T08:12:00.096 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:12:00.096 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:12:00.097 INFO:teuthology.orchestra.run.smithi007.stdout: "description": "osd_op(client.15582.0:53575 9.0 9:0f4fbab7:::benchmark_data_smithi007_124019_object53574:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e244)", 2024-09-18T08:12:00.097 INFO:teuthology.orchestra.run.smithi007.stdout: "initiated_at": "2024-09-18T08:11:41.343854+0000", 2024-09-18T08:12:00.097 INFO:teuthology.orchestra.run.smithi007.stdout: "age": 18.419392343999998, 2024-09-18T08:12:00.097 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.0074121789999999, 2024-09-18T08:12:00.097 INFO:teuthology.orchestra.run.smithi007.stdout: "continuous": false, 2024-09-18T08:12:00.097 INFO:teuthology.orchestra.run.smithi007.stdout: "type_data": { 2024-09-18T08:12:00.097 INFO:teuthology.orchestra.run.smithi007.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T08:12:00.097 INFO:teuthology.orchestra.run.smithi007.stdout: "client_info": { 2024-09-18T08:12:00.097 INFO:teuthology.orchestra.run.smithi007.stdout: "client": "client.15582", 2024-09-18T08:12:00.097 INFO:teuthology.orchestra.run.smithi007.stdout: "client_addr": "172.21.15.7:0/3598605839", 2024-09-18T08:12:00.097 INFO:teuthology.orchestra.run.smithi007.stdout: "tid": 53575 2024-09-18T08:12:00.097 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:12:00.097 INFO:teuthology.orchestra.run.smithi007.stdout: "events": [ 2024-09-18T08:12:00.098 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:12:00.098 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "initiated", 2024-09-18T08:12:00.098 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:11:41.343854+0000", 2024-09-18T08:12:00.098 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-09-18T08:12:00.098 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:12:00.098 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:12:00.098 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "header_read", 2024-09-18T08:12:00.098 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:11:41.343854+0000", 2024-09-18T08:12:00.098 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-09-18T08:12:00.098 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:12:00.098 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:12:00.098 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "throttled", 2024-09-18T08:12:00.098 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:11:41.343855+0000", 2024-09-18T08:12:00.099 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 5.5000000000000003e-07 2024-09-18T08:12:00.099 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:12:00.099 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:12:00.099 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "all_read", 2024-09-18T08:12:00.099 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:11:41.343878+0000", 2024-09-18T08:12:00.099 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 2.3740999999999999e-05 2024-09-18T08:12:00.099 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:12:00.099 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:12:00.099 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "dispatched", 2024-09-18T08:12:00.099 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:11:41.343886+0000", 2024-09-18T08:12:00.099 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 7.8809999999999996e-06 2024-09-18T08:12:00.099 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:12:00.099 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:12:00.100 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "queued_for_pg", 2024-09-18T08:12:00.100 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:11:41.343892+0000", 2024-09-18T08:12:00.100 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 5.8200000000000002e-06 2024-09-18T08:12:00.100 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:12:00.100 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:12:00.100 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-09-18T08:12:00.100 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:11:41.344821+0000", 2024-09-18T08:12:00.100 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.000928874 2024-09-18T08:12:00.100 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:12:00.100 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:12:00.100 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6 2024-09-18T08:12:00.101 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "waiting for peered", 2024-09-18T08:12:00.102 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:11:41.344861+0000", 2024-09-18T08:12:00.102 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 3.9919999999999997e-05 2024-09-18T08:12:00.102 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:12:00.102 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:12:00.102 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-09-18T08:12:00.102 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:11:42.334161+0000", 2024-09-18T08:12:00.102 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.98930047799999998 2024-09-18T08:12:00.102 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:12:00.102 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:12:00.102 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "started", 2024-09-18T08:12:00.102 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:11:42.334487+0000", 2024-09-18T08:12:00.102 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.00032586300000000002 2024-09-18T08:12:00.103 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:12:00.103 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:12:00.103 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "waiting for subops from 6", 2024-09-18T08:12:00.103 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:11:42.334655+0000", 2024-09-18T08:12:00.103 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.000167419 2024-09-18T08:12:00.103 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:12:00.103 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:12:00.103 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "op_commit", 2024-09-18T08:12:00.103 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:11:42.346631+0000", 2024-09-18T08:12:00.103 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.011976496 2024-09-18T08:12:00.103 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:12:00.103 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:12:00.103 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:12:00.104 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:11:42.351115+0000", 2024-09-18T08:12:00.104 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.0044843590000000003 2024-09-18T08:12:00.104 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:12:00.104 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:12:00.104 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "commit_sent", 2024-09-18T08:12:00.104 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:11:42.351173+0000", 2024-09-18T08:12:00.104 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 5.7061000000000001e-05 2024-09-18T08:12:00.104 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:12:00.104 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:12:00.104 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "done", 2024-09-18T08:12:00.104 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:11:42.351266+0000", 2024-09-18T08:12:00.104 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 9.3716999999999997e-05 2024-09-18T08:12:00.104 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-09-18T08:12:00.104 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-09-18T08:12:00.105 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-09-18T08:12:00.105 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:12:00.105 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:12:00.105 INFO:teuthology.orchestra.run.smithi007.stdout: "description": "osd_op(client.15582.0:53576 9.0 9:0f9d6982:::benchmark_data_smithi007_124019_object53575:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e244)", 2024-09-18T08:12:00.105 INFO:teuthology.orchestra.run.smithi007.stdout: "initiated_at": "2024-09-18T08:11:41.343901+0000", 2024-09-18T08:12:00.105 INFO:teuthology.orchestra.run.smithi007.stdout: "age": 18.419345123999999, 2024-09-18T08:12:00.105 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.01002054, 2024-09-18T08:12:00.105 INFO:teuthology.orchestra.run.smithi007.stdout: "continuous": false, 2024-09-18T08:12:00.105 INFO:teuthology.orchestra.run.smithi007.stdout: "type_data": { 2024-09-18T08:12:00.105 INFO:teuthology.orchestra.run.smithi007.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T08:12:00.105 INFO:teuthology.orchestra.run.smithi007.stdout: "client_info": { 2024-09-18T08:12:00.105 INFO:teuthology.orchestra.run.smithi007.stdout: "client": "client.15582", 2024-09-18T08:12:00.105 INFO:teuthology.orchestra.run.smithi007.stdout: "client_addr": "172.21.15.7:0/3598605839", 2024-09-18T08:12:00.105 INFO:teuthology.orchestra.run.smithi007.stdout: "tid": 53576 2024-09-18T08:12:00.106 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:12:00.106 INFO:teuthology.orchestra.run.smithi007.stdout: "events": [ 2024-09-18T08:12:00.106 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:12:00.106 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "initiated", 2024-09-18T08:12:00.106 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:11:41.343901+0000", 2024-09-18T08:12:00.106 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-09-18T08:12:00.106 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:12:00.106 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:12:00.106 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "header_read", 2024-09-18T08:12:00.106 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:11:41.343901+0000", 2024-09-18T08:12:00.106 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-09-18T08:12:00.106 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:12:00.106 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:12:00.107 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "throttled", 2024-09-18T08:12:00.107 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:11:41.343902+0000", 2024-09-18T08:12:00.107 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 4.7100000000000002e-07 2024-09-18T08:12:00.107 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:12:00.107 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:12:00.107 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "all_read", 2024-09-18T08:12:00.107 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:11:41.343923+0000", 2024-09-18T08:12:00.107 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 2.1009000000000002e-05 2024-09-18T08:12:00.107 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:12:00.107 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:12:00.107 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "dispatched", 2024-09-18T08:12:00.107 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:11:41.343930+0000", 2024-09-18T08:12:00.107 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 7.0040000000000002e-06 2024-09-18T08:12:00.107 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:12:00.107 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:12:00.108 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "queued_for_pg", 2024-09-18T08:12:00.108 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:11:41.343935+0000", 2024-09-18T08:12:00.108 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 5.0549999999999999e-06 2024-09-18T08:12:00.108 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:12:00.108 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:12:00.108 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-09-18T08:12:00.108 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:11:41.345163+0000", 2024-09-18T08:12:00.108 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.0012281200000000001 2024-09-18T08:12:00.108 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:12:00.108 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:12:00.108 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "waiting for peered", 2024-09-18T08:12:00.108 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:11:41.345169+0000", 2024-09-18T08:12:00.108 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 5.6999999999999996e-06 2024-09-18T08:12:00.108 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:12:00.109 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:12:00.109 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-09-18T08:12:00.109 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:11:42.336827+0000", 2024-09-18T08:12:00.109 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.99165844599999997 2024-09-18T08:12:00.109 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:12:00.109 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:12:00.109 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "started", 2024-09-18T08:12:00.109 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:11:42.336957+0000", 2024-09-18T08:12:00.109 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.00013009 2024-09-18T08:12:00.109 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:12:00.109 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:12:00.109 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "waiting for subops from 6", 2024-09-18T08:12:00.109 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:11:42.337097+0000", 2024-09-18T08:12:00.110 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.00014021899999999999 2024-09-18T08:12:00.110 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:12:00.110 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:12:00.110 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "op_commit", 2024-09-18T08:12:00.110 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:11:42.346639+0000", 2024-09-18T08:12:00.110 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.0095413370000000004 2024-09-18T08:12:00.110 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:12:00.110 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:12:00.110 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:12:00.110 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:11:42.353750+0000", 2024-09-18T08:12:00.110 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.0071108530000000003 2024-09-18T08:12:00.110 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:12:00.110 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:12:00.110 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "commit_sent", 2024-09-18T08:12:00.111 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:11:42.353813+0000", 2024-09-18T08:12:00.111 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 6.3721999999999997e-05 2024-09-18T08:12:00.111 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:12:00.111 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:12:00.111 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "done", 2024-09-18T08:12:00.111 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:11:42.353922+0000", 2024-09-18T08:12:00.111 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.000108514 2024-09-18T08:12:00.111 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-09-18T08:12:00.111 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-09-18T08:12:00.111 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-09-18T08:12:00.111 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:12:00.111 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:12:00.111 INFO:teuthology.orchestra.run.smithi007.stdout: "description": "osd_op(client.24356.0:2055 4.4 4:22d26bf9:::data_loggenerations_metadata:head [watch reconnect cookie 94700888688768 gen 2] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e244)", 2024-09-18T08:12:00.111 INFO:teuthology.orchestra.run.smithi007.stdout: "initiated_at": "2024-09-18T08:11:41.923730+0000", 2024-09-18T08:12:00.111 INFO:teuthology.orchestra.run.smithi007.stdout: "age": 17.839516757999998, 2024-09-18T08:12:00.112 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.42490792300000002, 2024-09-18T08:12:00.112 INFO:teuthology.orchestra.run.smithi007.stdout: "continuous": false, 2024-09-18T08:12:00.112 INFO:teuthology.orchestra.run.smithi007.stdout: "type_data": { 2024-09-18T08:12:00.112 INFO:teuthology.orchestra.run.smithi007.stdout: "flag_point": "started", 2024-09-18T08:12:00.112 INFO:teuthology.orchestra.run.smithi007.stdout: "client_info": { 2024-09-18T08:12:00.112 INFO:teuthology.orchestra.run.smithi007.stdout: "client": "client.24356", 2024-09-18T08:12:00.112 INFO:teuthology.orchestra.run.smithi007.stdout: "client_addr": "172.21.15.7:0/219168083", 2024-09-18T08:12:00.112 INFO:teuthology.orchestra.run.smithi007.stdout: "tid": 2055 2024-09-18T08:12:00.112 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:12:00.112 INFO:teuthology.orchestra.run.smithi007.stdout: "events": [ 2024-09-18T08:12:00.112 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:12:00.112 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "initiated", 2024-09-18T08:12:00.112 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:11:41.923730+0000", 2024-09-18T08:12:00.112 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-09-18T08:12:00.112 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:12:00.113 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:12:00.113 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "header_read", 2024-09-18T08:12:00.113 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:11:41.923730+0000", 2024-09-18T08:12:00.113 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-09-18T08:12:00.113 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:12:00.113 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:12:00.113 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "throttled", 2024-09-18T08:12:00.113 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:11:41.923732+0000", 2024-09-18T08:12:00.113 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 2.5170000000000002e-06 2024-09-18T08:12:00.113 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:12:00.113 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:12:00.113 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "all_read", 2024-09-18T08:12:00.113 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:11:41.923744+0000", 2024-09-18T08:12:00.113 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.1895999999999999e-05 2024-09-18T08:12:00.114 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:12:00.114 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:12:00.114 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "dispatched", 2024-09-18T08:12:00.114 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:11:41.923762+0000", 2024-09-18T08:12:00.114 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.7476e-05 2024-09-18T08:12:00.114 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:12:00.114 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:12:00.114 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "queued_for_pg", 2024-09-18T08:12:00.114 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:11:41.923774+0000", 2024-09-18T08:12:00.114 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.2072000000000001e-05 2024-09-18T08:12:00.114 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:12:00.114 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:12:00.114 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-09-18T08:12:00.114 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:11:41.923863+0000", 2024-09-18T08:12:00.115 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 8.9174999999999999e-05 2024-09-18T08:12:00.115 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:12:00.115 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:12:00.115 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "waiting for peered", 2024-09-18T08:12:00.115 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:11:41.923875+0000", 2024-09-18T08:12:00.115 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.2544000000000001e-05 2024-09-18T08:12:00.115 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:12:00.115 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:12:00.115 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-09-18T08:12:00.115 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:11:42.331723+0000", 2024-09-18T08:12:00.115 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.40784754899999998 2024-09-18T08:12:00.115 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:12:00.115 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:12:00.115 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "started", 2024-09-18T08:12:00.115 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:11:42.331911+0000", 2024-09-18T08:12:00.115 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.00018769699999999999 2024-09-18T08:12:00.116 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:12:00.116 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:12:00.116 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "done", 2024-09-18T08:12:00.116 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:11:42.348638+0000", 2024-09-18T08:12:00.116 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.016726997 2024-09-18T08:12:00.116 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-09-18T08:12:00.116 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-09-18T08:12:00.116 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-09-18T08:12:00.116 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-09-18T08:12:00.116 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-09-18T08:12:00.116 INFO:teuthology.orchestra.run.smithi007.stdout:} 2024-09-18T08:12:00.137 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:12:00.137 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:12:00.138 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6' 2024-09-18T08:12:00.193 INFO:tasks.radosbench.radosbench.0.smithi007.stdout: 45 16 81623 81607 113.316 88.0625 0.00658454 0.00882018 2024-09-18T08:12:00.238 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7 2024-09-18T08:12:00.278 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:12:00.278 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:12:00.279 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7' 2024-09-18T08:12:00.306 INFO:teuthology.orchestra.run.smithi007.stderr:nodeep-scrub is unset 2024-09-18T08:12:00.379 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4 2024-09-18T08:12:00.419 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:12:00.420 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:12:00.420 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4' 2024-09-18T08:12:00.521 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5 2024-09-18T08:12:00.557 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:12:00.557 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:12:00.558 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5' 2024-09-18T08:12:00.659 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4 2024-09-18T08:12:00.670 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:12:00 smithi007 ceph-mon[29459]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-18T08:12:00.670 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:12:00 smithi007 ceph-mon[29459]: pgmap v441: 158 pgs: 30 active+undersized, 34 active+undersized+degraded, 94 active+clean; 4.8 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 1.2 KiB/s rd, 122 MiB/s wr, 1.95k op/s; 44774/238704 objects degraded (18.757%) 2024-09-18T08:12:00.671 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:12:00 smithi007 ceph-mon[29459]: 3.9 scrub starts 2024-09-18T08:12:00.671 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:12:00 smithi007 ceph-mon[29459]: 3.9 scrub ok 2024-09-18T08:12:00.671 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:12:00 smithi007 ceph-mon[29459]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-18T08:12:00.671 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:12:00 smithi007 ceph-mon[29459]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-18T08:12:00.671 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:12:00 smithi007 ceph-mon[29459]: osdmap e252: 8 total, 7 up, 8 in 2024-09-18T08:12:00.671 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:12:00 smithi007 ceph-mon[39249]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-18T08:12:00.671 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:12:00 smithi007 ceph-mon[39249]: pgmap v441: 158 pgs: 30 active+undersized, 34 active+undersized+degraded, 94 active+clean; 4.8 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 1.2 KiB/s rd, 122 MiB/s wr, 1.95k op/s; 44774/238704 objects degraded (18.757%) 2024-09-18T08:12:00.671 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:12:00 smithi007 ceph-mon[39249]: 3.9 scrub starts 2024-09-18T08:12:00.671 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:12:00 smithi007 ceph-mon[39249]: 3.9 scrub ok 2024-09-18T08:12:00.671 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:12:00 smithi007 ceph-mon[39249]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-18T08:12:00.672 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:12:00 smithi007 ceph-mon[39249]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-18T08:12:00.672 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:12:00 smithi007 ceph-mon[39249]: osdmap e252: 8 total, 7 up, 8 in 2024-09-18T08:12:00.686 INFO:teuthology.orchestra.run.smithi007.stdout:osd.0: {} 2024-09-18T08:12:00.686 INFO:teuthology.orchestra.run.smithi007.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-18T08:12:00.691 INFO:teuthology.orchestra.run.smithi007.stderr:Error ENXIO: problem getting command descriptions from osd.1 2024-09-18T08:12:00.694 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:12:00.695 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:12:00.695 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4' 2024-09-18T08:12:00.722 INFO:teuthology.orchestra.run.smithi007.stdout:osd.2: {} 2024-09-18T08:12:00.722 INFO:teuthology.orchestra.run.smithi007.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-18T08:12:00.761 INFO:teuthology.orchestra.run.smithi007.stdout:osd.3: {} 2024-09-18T08:12:00.762 INFO:teuthology.orchestra.run.smithi007.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-18T08:12:00.796 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2 2024-09-18T08:12:00.799 INFO:teuthology.orchestra.run.smithi007.stdout:osd.4: {} 2024-09-18T08:12:00.800 INFO:teuthology.orchestra.run.smithi007.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-18T08:12:00.838 INFO:teuthology.orchestra.run.smithi007.stdout:osd.5: {} 2024-09-18T08:12:00.839 INFO:teuthology.orchestra.run.smithi007.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-18T08:12:00.839 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:12:00.840 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:12:00.840 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2' 2024-09-18T08:12:00.871 INFO:teuthology.orchestra.run.smithi007.stdout:osd.6: {} 2024-09-18T08:12:00.872 INFO:teuthology.orchestra.run.smithi007.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-18T08:12:00.904 INFO:teuthology.orchestra.run.smithi007.stdout:osd.7: {} 2024-09-18T08:12:00.904 INFO:teuthology.orchestra.run.smithi007.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-18T08:12:00.905 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:12:00 smithi050 ceph-mon[32371]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-18T08:12:00.906 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:12:00 smithi050 ceph-mon[32371]: pgmap v441: 158 pgs: 30 active+undersized, 34 active+undersized+degraded, 94 active+clean; 4.8 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 1.2 KiB/s rd, 122 MiB/s wr, 1.95k op/s; 44774/238704 objects degraded (18.757%) 2024-09-18T08:12:00.906 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:12:00 smithi050 ceph-mon[32371]: 3.9 scrub starts 2024-09-18T08:12:00.906 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:12:00 smithi050 ceph-mon[32371]: 3.9 scrub ok 2024-09-18T08:12:00.906 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:12:00 smithi050 ceph-mon[32371]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-18T08:12:00.906 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:12:00 smithi050 ceph-mon[32371]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-18T08:12:00.906 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:12:00 smithi050 ceph-mon[32371]: osdmap e252: 8 total, 7 up, 8 in 2024-09-18T08:12:00.941 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4 2024-09-18T08:12:00.978 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:12:00.978 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:12:00.978 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4' 2024-09-18T08:12:01.079 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0 2024-09-18T08:12:01.106 INFO:teuthology.orchestra.run.smithi007.stdout:9.f 2024-09-18T08:12:01.106 INFO:teuthology.orchestra.run.smithi007.stdout:9.e 2024-09-18T08:12:01.106 INFO:teuthology.orchestra.run.smithi007.stdout:9.a 2024-09-18T08:12:01.106 INFO:teuthology.orchestra.run.smithi007.stdout:9.6 2024-09-18T08:12:01.106 INFO:teuthology.orchestra.run.smithi007.stdout:4.b 2024-09-18T08:12:01.107 INFO:teuthology.orchestra.run.smithi007.stdout:4.1f 2024-09-18T08:12:01.107 INFO:teuthology.orchestra.run.smithi007.stdout:4.18 2024-09-18T08:12:01.107 INFO:teuthology.orchestra.run.smithi007.stdout:6.19 2024-09-18T08:12:01.107 INFO:teuthology.orchestra.run.smithi007.stdout:4.6 2024-09-18T08:12:01.107 INFO:teuthology.orchestra.run.smithi007.stdout:4.17 2024-09-18T08:12:01.107 INFO:teuthology.orchestra.run.smithi007.stdout:5.0 2024-09-18T08:12:01.109 INFO:teuthology.orchestra.run.smithi007.stdout:4.a 2024-09-18T08:12:01.109 INFO:teuthology.orchestra.run.smithi007.stdout:4.13 2024-09-18T08:12:01.110 INFO:teuthology.orchestra.run.smithi007.stdout:4.12 2024-09-18T08:12:01.110 INFO:teuthology.orchestra.run.smithi007.stdout:3.6 2024-09-18T08:12:01.110 INFO:teuthology.orchestra.run.smithi007.stdout:3.e 2024-09-18T08:12:01.110 INFO:teuthology.orchestra.run.smithi007.stdout:1.3 2024-09-18T08:12:01.110 INFO:teuthology.orchestra.run.smithi007.stdout:4.e 2024-09-18T08:12:01.110 INFO:teuthology.orchestra.run.smithi007.stdout:4.1c 2024-09-18T08:12:01.110 INFO:teuthology.orchestra.run.smithi007.stdout:3.19 2024-09-18T08:12:01.110 INFO:teuthology.orchestra.run.smithi007.stdout:6.10 2024-09-18T08:12:01.110 INFO:teuthology.orchestra.run.smithi007.stdout:4.2 2024-09-18T08:12:01.110 INFO:teuthology.orchestra.run.smithi007.stdout:4.d 2024-09-18T08:12:01.110 INFO:teuthology.orchestra.run.smithi007.stdout:4.1d 2024-09-18T08:12:01.110 INFO:teuthology.orchestra.run.smithi007.stdout:9.3 2024-09-18T08:12:01.110 INFO:teuthology.orchestra.run.smithi007.stdout:3.18 2024-09-18T08:12:01.110 INFO:teuthology.orchestra.run.smithi007.stdout:4.4 2024-09-18T08:12:01.111 INFO:teuthology.orchestra.run.smithi007.stdout:3.16 2024-09-18T08:12:01.111 INFO:teuthology.orchestra.run.smithi007.stdout:9.d 2024-09-18T08:12:01.111 INFO:teuthology.orchestra.run.smithi007.stdout:3.8 2024-09-18T08:12:01.111 INFO:teuthology.orchestra.run.smithi007.stdout:3.1a 2024-09-18T08:12:01.111 INFO:teuthology.orchestra.run.smithi007.stdout:5.19 2024-09-18T08:12:01.111 INFO:teuthology.orchestra.run.smithi007.stdout:5.5 2024-09-18T08:12:01.112 INFO:teuthology.orchestra.run.smithi007.stdout:3.1c 2024-09-18T08:12:01.112 INFO:teuthology.orchestra.run.smithi007.stdout:3.4 2024-09-18T08:12:01.112 INFO:teuthology.orchestra.run.smithi007.stdout:5.16 2024-09-18T08:12:01.112 INFO:teuthology.orchestra.run.smithi007.stdout:2.0 2024-09-18T08:12:01.112 INFO:teuthology.orchestra.run.smithi007.stdout:5.13 2024-09-18T08:12:01.112 INFO:teuthology.orchestra.run.smithi007.stdout:4.1b 2024-09-18T08:12:01.112 INFO:teuthology.orchestra.run.smithi007.stdout:3.a 2024-09-18T08:12:01.112 INFO:teuthology.orchestra.run.smithi007.stdout:5.8 2024-09-18T08:12:01.112 INFO:teuthology.orchestra.run.smithi007.stdout:4.f 2024-09-18T08:12:01.112 INFO:teuthology.orchestra.run.smithi007.stdout:5.12 2024-09-18T08:12:01.112 INFO:teuthology.orchestra.run.smithi007.stdout:4.10 2024-09-18T08:12:01.112 INFO:teuthology.orchestra.run.smithi007.stdout:6.e 2024-09-18T08:12:01.113 INFO:teuthology.orchestra.run.smithi007.stdout:5.15 2024-09-18T08:12:01.113 INFO:teuthology.orchestra.run.smithi007.stdout:5.1a 2024-09-18T08:12:01.113 INFO:teuthology.orchestra.run.smithi007.stdout:5.7 2024-09-18T08:12:01.113 INFO:teuthology.orchestra.run.smithi007.stdout:5.17 2024-09-18T08:12:01.113 INFO:teuthology.orchestra.run.smithi007.stdout:5.18 2024-09-18T08:12:01.113 INFO:teuthology.orchestra.run.smithi007.stdout:5.1d 2024-09-18T08:12:01.113 INFO:teuthology.orchestra.run.smithi007.stdout:5.c 2024-09-18T08:12:01.113 INFO:teuthology.orchestra.run.smithi007.stdout:2.2 2024-09-18T08:12:01.113 INFO:teuthology.orchestra.run.smithi007.stdout:6.18 2024-09-18T08:12:01.113 INFO:teuthology.orchestra.run.smithi007.stdout:6.1 2024-09-18T08:12:01.114 INFO:teuthology.orchestra.run.smithi007.stdout:6.1a 2024-09-18T08:12:01.114 INFO:teuthology.orchestra.run.smithi007.stdout:9.0 2024-09-18T08:12:01.114 INFO:teuthology.orchestra.run.smithi007.stdout:6.1d 2024-09-18T08:12:01.114 INFO:teuthology.orchestra.run.smithi007.stdout:9.2 2024-09-18T08:12:01.114 INFO:teuthology.orchestra.run.smithi007.stdout:6.4 2024-09-18T08:12:01.115 INFO:teuthology.orchestra.run.smithi007.stdout:6.b 2024-09-18T08:12:01.115 INFO:teuthology.orchestra.run.smithi007.stdout:6.d 2024-09-18T08:12:01.115 INFO:teuthology.orchestra.run.smithi007.stdout:3.0 2024-09-18T08:12:01.115 INFO:teuthology.orchestra.run.smithi007.stdout:9.4 2024-09-18T08:12:01.118 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:12:01.119 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:12:01.119 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0' 2024-09-18T08:12:01.193 INFO:tasks.radosbench.radosbench.0.smithi007.stdout: 46 16 83088 83072 112.843 91.5625 0.0110208 0.00885744 2024-09-18T08:12:01.220 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4 2024-09-18T08:12:01.256 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:12:01.257 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:12:01.257 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4' 2024-09-18T08:12:01.358 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5 2024-09-18T08:12:01.396 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:12:01.396 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:12:01.397 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5' 2024-09-18T08:12:01.498 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4 2024-09-18T08:12:01.536 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:12:01.537 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:12:01.537 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4' 2024-09-18T08:12:01.638 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0 2024-09-18T08:12:01.674 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:12:01.675 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:12:01.675 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0' 2024-09-18T08:12:01.730 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:12:01 smithi007 ceph-mon[29459]: 4.1a scrub starts 2024-09-18T08:12:01.730 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:12:01 smithi007 ceph-mon[29459]: 4.1a scrub ok 2024-09-18T08:12:01.730 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:12:01 smithi007 ceph-mon[29459]: from='client.? 172.21.15.7:0/3297877927' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-18T08:12:01.731 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:12:01 smithi007 ceph-mon[39249]: 4.1a scrub starts 2024-09-18T08:12:01.731 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:12:01 smithi007 ceph-mon[39249]: 4.1a scrub ok 2024-09-18T08:12:01.731 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:12:01 smithi007 ceph-mon[39249]: from='client.? 172.21.15.7:0/3297877927' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-18T08:12:01.777 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5 2024-09-18T08:12:01.814 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:12:01.814 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:12:01.815 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5' 2024-09-18T08:12:01.905 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:12:01 smithi050 ceph-mon[32371]: 4.1a scrub starts 2024-09-18T08:12:01.906 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:12:01 smithi050 ceph-mon[32371]: 4.1a scrub ok 2024-09-18T08:12:01.906 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:12:01 smithi050 ceph-mon[32371]: from='client.? 172.21.15.7:0/3297877927' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-18T08:12:01.915 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2 2024-09-18T08:12:01.941 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:12:01.942 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:12:01.942 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2' 2024-09-18T08:12:02.043 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7 2024-09-18T08:12:02.080 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:12:02.080 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:12:02.080 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7' 2024-09-18T08:12:02.181 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5 2024-09-18T08:12:02.193 INFO:tasks.radosbench.radosbench.0.smithi007.stdout: 47 16 84590 84574 112.439 93.875 0.00733764 0.00888876 2024-09-18T08:12:02.218 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:12:02.219 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:12:02.219 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5' 2024-09-18T08:12:02.320 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4 2024-09-18T08:12:02.361 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:12:02.362 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:12:02.362 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4' 2024-09-18T08:12:02.364 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid 779349ca-7593-11ef-bceb-c7b262605968 -- ceph daemon osd.3 dump_ops_in_flight 2024-09-18T08:12:02.463 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3 2024-09-18T08:12:02.498 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:12:02.499 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:12:02.500 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3' 2024-09-18T08:12:02.600 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2 2024-09-18T08:12:02.622 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:12:02.623 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:12:02.623 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2' 2024-09-18T08:12:02.724 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0 2024-09-18T08:12:02.730 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:12:02 smithi007 ceph-mon[29459]: 1.8 scrub starts 2024-09-18T08:12:02.730 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:12:02 smithi007 ceph-mon[29459]: 1.8 scrub ok 2024-09-18T08:12:02.730 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:12:02 smithi007 ceph-mon[29459]: 5.14 scrub starts 2024-09-18T08:12:02.730 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:12:02 smithi007 ceph-mon[29459]: 5.14 scrub ok 2024-09-18T08:12:02.730 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:12:02 smithi007 ceph-mon[29459]: pgmap v444: 158 pgs: 30 active+undersized, 34 active+undersized+degraded, 94 active+clean; 4.8 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 852 B/s rd, 81 MiB/s wr, 1.29k op/s; 44774/238704 objects degraded (18.757%) 2024-09-18T08:12:02.731 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:12:02 smithi007 ceph-mon[39249]: 1.8 scrub starts 2024-09-18T08:12:02.731 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:12:02 smithi007 ceph-mon[39249]: 1.8 scrub ok 2024-09-18T08:12:02.731 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:12:02 smithi007 ceph-mon[39249]: 5.14 scrub starts 2024-09-18T08:12:02.731 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:12:02 smithi007 ceph-mon[39249]: 5.14 scrub ok 2024-09-18T08:12:02.731 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:12:02 smithi007 ceph-mon[39249]: pgmap v444: 158 pgs: 30 active+undersized, 34 active+undersized+degraded, 94 active+clean; 4.8 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 852 B/s rd, 81 MiB/s wr, 1.29k op/s; 44774/238704 objects degraded (18.757%) 2024-09-18T08:12:02.753 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:12:02.754 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:12:02.755 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0' 2024-09-18T08:12:02.855 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3 2024-09-18T08:12:02.888 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:12:02.889 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:12:02.889 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3' 2024-09-18T08:12:02.905 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:12:02 smithi050 ceph-mon[32371]: 1.8 scrub starts 2024-09-18T08:12:02.906 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:12:02 smithi050 ceph-mon[32371]: 1.8 scrub ok 2024-09-18T08:12:02.906 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:12:02 smithi050 ceph-mon[32371]: 5.14 scrub starts 2024-09-18T08:12:02.906 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:12:02 smithi050 ceph-mon[32371]: 5.14 scrub ok 2024-09-18T08:12:02.906 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:12:02 smithi050 ceph-mon[32371]: pgmap v444: 158 pgs: 30 active+undersized, 34 active+undersized+degraded, 94 active+clean; 4.8 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 852 B/s rd, 81 MiB/s wr, 1.29k op/s; 44774/238704 objects degraded (18.757%) 2024-09-18T08:12:02.989 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4 2024-09-18T08:12:03.030 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:12:03.030 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:12:03.031 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4' 2024-09-18T08:12:03.131 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2 2024-09-18T08:12:03.164 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:12:03.165 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:12:03.165 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2' 2024-09-18T08:12:03.193 INFO:tasks.radosbench.radosbench.0.smithi007.stdout: 48 16 86057 86041 112.007 91.6875 0.00505198 0.00892316 2024-09-18T08:12:03.266 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2 2024-09-18T08:12:03.301 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:12:03.302 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:12:03.302 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2' 2024-09-18T08:12:03.403 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0 2024-09-18T08:12:03.434 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:12:03.434 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:12:03.434 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0' 2024-09-18T08:12:03.535 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5 2024-09-18T08:12:03.565 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:12:03.566 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:12:03.566 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5' 2024-09-18T08:12:03.637 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid 779349ca-7593-11ef-bceb-c7b262605968 -- ceph osd set noscrub 2024-09-18T08:12:03.667 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4 2024-09-18T08:12:03.706 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:12:03.706 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:12:03.707 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4' 2024-09-18T08:12:03.730 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:12:03 smithi007 ceph-mon[29459]: 6.2 scrub starts 2024-09-18T08:12:03.730 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:12:03 smithi007 ceph-mon[29459]: 6.2 scrub ok 2024-09-18T08:12:03.730 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:12:03 smithi007 ceph-mon[29459]: from='mgr.24470 172.21.15.7:0/2594390261' entity='mgr.y' 2024-09-18T08:12:03.730 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:12:03 smithi007 ceph-mon[29459]: from='mgr.24470 172.21.15.7:0/2594390261' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-18T08:12:03.731 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:12:03 smithi007 ceph-mon[39249]: 6.2 scrub starts 2024-09-18T08:12:03.731 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:12:03 smithi007 ceph-mon[39249]: 6.2 scrub ok 2024-09-18T08:12:03.731 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:12:03 smithi007 ceph-mon[39249]: from='mgr.24470 172.21.15.7:0/2594390261' entity='mgr.y' 2024-09-18T08:12:03.731 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:12:03 smithi007 ceph-mon[39249]: from='mgr.24470 172.21.15.7:0/2594390261' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-18T08:12:03.807 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5 2024-09-18T08:12:03.829 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:12:03.830 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:12:03.831 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5' 2024-09-18T08:12:03.905 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:12:03 smithi050 ceph-mon[32371]: 6.2 scrub starts 2024-09-18T08:12:03.905 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:12:03 smithi050 ceph-mon[32371]: 6.2 scrub ok 2024-09-18T08:12:03.906 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:12:03 smithi050 ceph-mon[32371]: from='mgr.24470 172.21.15.7:0/2594390261' entity='mgr.y' 2024-09-18T08:12:03.906 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:12:03 smithi050 ceph-mon[32371]: from='mgr.24470 172.21.15.7:0/2594390261' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-18T08:12:03.931 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0 2024-09-18T08:12:03.966 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:12:03.967 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:12:03.967 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0' 2024-09-18T08:12:04.068 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4 2024-09-18T08:12:04.109 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:12:04.110 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:12:04.110 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4' 2024-09-18T08:12:04.193 INFO:tasks.radosbench.radosbench.0.smithi007.stdout: 49 16 87310 87294 111.319 78.3125 0.0102008 0.00897777 2024-09-18T08:12:04.210 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2 2024-09-18T08:12:04.240 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:12:04.241 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:12:04.241 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2' 2024-09-18T08:12:04.342 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4 2024-09-18T08:12:04.379 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:12:04.380 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:12:04.380 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4' 2024-09-18T08:12:04.481 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6 2024-09-18T08:12:04.524 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:12:04.524 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:12:04.524 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6' 2024-09-18T08:12:04.602 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-09-18T08:12:04.603 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-09-18T08:12:04.626 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6 2024-09-18T08:12:04.662 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:12:04.662 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:12:04.663 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6' 2024-09-18T08:12:04.730 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:12:04 smithi007 ceph-mon[29459]: 3.1e scrub starts 2024-09-18T08:12:04.730 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:12:04 smithi007 ceph-mon[29459]: 3.1e scrub ok 2024-09-18T08:12:04.730 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:12:04 smithi007 ceph-mon[29459]: pgmap v445: 158 pgs: 30 active+undersized, 34 active+undersized+degraded, 94 active+clean; 4.9 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 748 B/s rd, 79 MiB/s wr, 1.27k op/s; 45690/241452 objects degraded (18.923%) 2024-09-18T08:12:04.730 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:12:04 smithi007 ceph-mon[29459]: 3.11 scrub starts 2024-09-18T08:12:04.731 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:12:04 smithi007 ceph-mon[29459]: 3.11 scrub ok 2024-09-18T08:12:04.731 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:12:04 smithi007 ceph-mon[29459]: 2.1 scrub starts 2024-09-18T08:12:04.731 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:12:04 smithi007 ceph-mon[29459]: 2.1 scrub ok 2024-09-18T08:12:04.731 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:12:04 smithi007 ceph-mon[29459]: Health check update: Degraded data redundancy: 45690/241452 objects degraded (18.923%), 34 pgs degraded (PG_DEGRADED) 2024-09-18T08:12:04.731 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:12:04 smithi007 ceph-mon[29459]: osdmap e253: 8 total, 7 up, 8 in 2024-09-18T08:12:04.731 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:12:04 smithi007 ceph-mon[39249]: 3.1e scrub starts 2024-09-18T08:12:04.731 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:12:04 smithi007 ceph-mon[39249]: 3.1e scrub ok 2024-09-18T08:12:04.731 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:12:04 smithi007 ceph-mon[39249]: pgmap v445: 158 pgs: 30 active+undersized, 34 active+undersized+degraded, 94 active+clean; 4.9 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 748 B/s rd, 79 MiB/s wr, 1.27k op/s; 45690/241452 objects degraded (18.923%) 2024-09-18T08:12:04.731 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:12:04 smithi007 ceph-mon[39249]: 3.11 scrub starts 2024-09-18T08:12:04.731 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:12:04 smithi007 ceph-mon[39249]: 3.11 scrub ok 2024-09-18T08:12:04.731 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:12:04 smithi007 ceph-mon[39249]: 2.1 scrub starts 2024-09-18T08:12:04.732 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:12:04 smithi007 ceph-mon[39249]: 2.1 scrub ok 2024-09-18T08:12:04.732 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:12:04 smithi007 ceph-mon[39249]: Health check update: Degraded data redundancy: 45690/241452 objects degraded (18.923%), 34 pgs degraded (PG_DEGRADED) 2024-09-18T08:12:04.732 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:12:04 smithi007 ceph-mon[39249]: osdmap e253: 8 total, 7 up, 8 in 2024-09-18T08:12:04.763 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0 2024-09-18T08:12:04.796 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:12:04.797 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:12:04.797 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0' 2024-09-18T08:12:04.898 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0 2024-09-18T08:12:04.905 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:12:04 smithi050 ceph-mon[32371]: 3.1e scrub starts 2024-09-18T08:12:04.906 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:12:04 smithi050 ceph-mon[32371]: 3.1e scrub ok 2024-09-18T08:12:04.906 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:12:04 smithi050 ceph-mon[32371]: pgmap v445: 158 pgs: 30 active+undersized, 34 active+undersized+degraded, 94 active+clean; 4.9 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 748 B/s rd, 79 MiB/s wr, 1.27k op/s; 45690/241452 objects degraded (18.923%) 2024-09-18T08:12:04.906 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:12:04 smithi050 ceph-mon[32371]: 3.11 scrub starts 2024-09-18T08:12:04.906 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:12:04 smithi050 ceph-mon[32371]: 3.11 scrub ok 2024-09-18T08:12:04.906 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:12:04 smithi050 ceph-mon[32371]: 2.1 scrub starts 2024-09-18T08:12:04.906 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:12:04 smithi050 ceph-mon[32371]: 2.1 scrub ok 2024-09-18T08:12:04.906 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:12:04 smithi050 ceph-mon[32371]: Health check update: Degraded data redundancy: 45690/241452 objects degraded (18.923%), 34 pgs degraded (PG_DEGRADED) 2024-09-18T08:12:04.906 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:12:04 smithi050 ceph-mon[32371]: osdmap e253: 8 total, 7 up, 8 in 2024-09-18T08:12:04.931 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:12:04.932 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:12:04.932 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0' 2024-09-18T08:12:05.033 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4 2024-09-18T08:12:05.075 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:12:05.075 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:12:05.076 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4' 2024-09-18T08:12:05.177 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2 2024-09-18T08:12:05.193 INFO:tasks.radosbench.radosbench.0.smithi007.stdout: 50 16 88831 88815 110.994 95.0625 0.00657191 0.00900428 2024-09-18T08:12:05.208 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:12:05.209 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:12:05.209 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2' 2024-09-18T08:12:05.245 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell -n osd.1 --fsid 779349ca-7593-11ef-bceb-c7b262605968 -- ceph-objectstore-tool --err-to-stderr --no-mon-config '--log-file=/var/log/ceph/objectstore_tool.$pid.log' --data-path /var/lib/ceph/osd/ceph-1 --journal-path /var/lib/ceph/osd/ceph-1/journal --op export --pgid 4.f --file /var/log/ceph/exp.4.f.1 2024-09-18T08:12:05.310 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6 2024-09-18T08:12:05.325 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/779349ca-7593-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T08:12:05.348 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:12:05.348 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:12:05.348 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6' 2024-09-18T08:12:05.360 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/779349ca-7593-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T08:12:05.449 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6 2024-09-18T08:12:05.488 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:12:05.489 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:12:05.489 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6' 2024-09-18T08:12:05.590 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3 2024-09-18T08:12:05.625 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:12:05.626 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:12:05.626 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3' 2024-09-18T08:12:05.727 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4 2024-09-18T08:12:05.730 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:12:05 smithi007 ceph-mon[29459]: 4.7 scrub starts 2024-09-18T08:12:05.730 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:12:05 smithi007 ceph-mon[29459]: 4.7 scrub ok 2024-09-18T08:12:05.731 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:12:05 smithi007 ceph-mon[39249]: 4.7 scrub starts 2024-09-18T08:12:05.731 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:12:05 smithi007 ceph-mon[39249]: 4.7 scrub ok 2024-09-18T08:12:05.770 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:12:05.770 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:12:05.771 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4' 2024-09-18T08:12:05.872 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5 2024-09-18T08:12:05.904 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:12:05.905 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:12:05.906 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5' 2024-09-18T08:12:05.906 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:12:05 smithi050 ceph-mon[32371]: 4.7 scrub starts 2024-09-18T08:12:05.906 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:12:05 smithi050 ceph-mon[32371]: 4.7 scrub ok 2024-09-18T08:12:06.007 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3 2024-09-18T08:12:06.039 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:12:06.040 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:12:06.040 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3' 2024-09-18T08:12:06.141 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7 2024-09-18T08:12:06.184 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:12:06.185 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:12:06.185 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7' 2024-09-18T08:12:06.193 INFO:tasks.radosbench.radosbench.0.smithi007.stdout: 51 16 90308 90292 110.627 92.3125 0.00812492 0.00903446 2024-09-18T08:12:06.198 INFO:teuthology.orchestra.run.smithi007.stdout:ERROR: (22) Invalid argument 2024-09-18T08:12:06.198 INFO:teuthology.orchestra.run.smithi007.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-18T08:12:06.286 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0 2024-09-18T08:12:06.316 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:12:06.317 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:12:06.317 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0' 2024-09-18T08:12:06.418 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0 2024-09-18T08:12:06.446 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:12:06.447 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:12:06.447 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0' 2024-09-18T08:12:06.548 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6 2024-09-18T08:12:06.587 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:12:06.588 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:12:06.588 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6' 2024-09-18T08:12:06.603 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid 779349ca-7593-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-18T08:12:06.689 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2 2024-09-18T08:12:06.729 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:12:06.730 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:12:06.730 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2' 2024-09-18T08:12:06.831 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7 2024-09-18T08:12:06.866 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:12:06.867 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:12:06.867 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7' 2024-09-18T08:12:06.905 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:12:06 smithi050 ceph-mon[32371]: 6.5 scrub starts 2024-09-18T08:12:06.906 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:12:06 smithi050 ceph-mon[32371]: 6.5 scrub ok 2024-09-18T08:12:06.906 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:12:06 smithi050 ceph-mon[32371]: pgmap v447: 158 pgs: 30 active+undersized, 34 active+undersized+degraded, 94 active+clean; 5.3 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 853 B/s rd, 75 MiB/s wr, 1.20k op/s; 48905/260355 objects degraded (18.784%) 2024-09-18T08:12:06.906 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:12:06 smithi050 ceph-mon[32371]: 3.1 scrub starts 2024-09-18T08:12:06.906 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:12:06 smithi050 ceph-mon[32371]: 3.1 scrub ok 2024-09-18T08:12:06.911 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:12:06 smithi007 ceph-mon[29459]: 6.5 scrub starts 2024-09-18T08:12:06.911 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:12:06 smithi007 ceph-mon[29459]: 6.5 scrub ok 2024-09-18T08:12:06.911 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:12:06 smithi007 ceph-mon[29459]: pgmap v447: 158 pgs: 30 active+undersized, 34 active+undersized+degraded, 94 active+clean; 5.3 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 853 B/s rd, 75 MiB/s wr, 1.20k op/s; 48905/260355 objects degraded (18.784%) 2024-09-18T08:12:06.911 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:12:06 smithi007 ceph-mon[29459]: 3.1 scrub starts 2024-09-18T08:12:06.912 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:12:06 smithi007 ceph-mon[29459]: 3.1 scrub ok 2024-09-18T08:12:06.912 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Sep 18 08:12:06 smithi007 ceph-779349ca-7593-11ef-bceb-c7b262605968-mgr-y[29688]: ::ffff:172.21.15.50 - - [18/Sep/2024:08:12:06] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-09-18T08:12:06.912 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:12:06 smithi007 ceph-mon[39249]: 6.5 scrub starts 2024-09-18T08:12:06.912 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:12:06 smithi007 ceph-mon[39249]: 6.5 scrub ok 2024-09-18T08:12:06.912 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:12:06 smithi007 ceph-mon[39249]: pgmap v447: 158 pgs: 30 active+undersized, 34 active+undersized+degraded, 94 active+clean; 5.3 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 853 B/s rd, 75 MiB/s wr, 1.20k op/s; 48905/260355 objects degraded (18.784%) 2024-09-18T08:12:06.912 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:12:06 smithi007 ceph-mon[39249]: 3.1 scrub starts 2024-09-18T08:12:06.912 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:12:06 smithi007 ceph-mon[39249]: 3.1 scrub ok 2024-09-18T08:12:06.940 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/779349ca-7593-11ef-bceb-c7b262605968/osd.1/config 2024-09-18T08:12:06.969 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid 779349ca-7593-11ef-bceb-c7b262605968 -- ceph daemon osd.3 dump_blocked_ops 2024-09-18T08:12:06.969 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6 2024-09-18T08:12:07.012 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:12:07.013 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:12:07.013 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6' 2024-09-18T08:12:07.114 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5 2024-09-18T08:12:07.152 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:12:07.152 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:12:07.153 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5' 2024-09-18T08:12:07.193 INFO:tasks.radosbench.radosbench.0.smithi007.stdout: 52 16 91794 91778 110.286 92.875 0.00799374 0.00906241 2024-09-18T08:12:07.253 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7 2024-09-18T08:12:07.287 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:12:07.288 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:12:07.288 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7' 2024-09-18T08:12:07.389 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4 2024-09-18T08:12:07.429 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:12:07.430 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:12:07.430 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4' 2024-09-18T08:12:07.531 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7 2024-09-18T08:12:07.553 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:12:07.553 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:12:07.554 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7' 2024-09-18T08:12:07.654 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7 2024-09-18T08:12:07.689 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:12:07.689 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:12:07.690 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7' 2024-09-18T08:12:07.791 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0 2024-09-18T08:12:07.826 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:12:07.827 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:12:07.827 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0' 2024-09-18T08:12:07.905 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:12:07 smithi050 ceph-mon[32371]: 6.7 scrub starts 2024-09-18T08:12:07.906 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:12:07 smithi050 ceph-mon[32371]: 6.7 scrub ok 2024-09-18T08:12:07.906 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:12:07 smithi050 ceph-mon[32371]: 6.6 scrub starts 2024-09-18T08:12:07.906 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:12:07 smithi050 ceph-mon[32371]: 6.6 scrub ok 2024-09-18T08:12:07.906 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:12:07 smithi050 ceph-mon[32371]: 6.17 scrub starts 2024-09-18T08:12:07.906 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:12:07 smithi050 ceph-mon[32371]: 6.17 scrub ok 2024-09-18T08:12:07.909 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:12:07 smithi007 ceph-mon[29459]: 6.7 scrub starts 2024-09-18T08:12:07.909 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:12:07 smithi007 ceph-mon[29459]: 6.7 scrub ok 2024-09-18T08:12:07.909 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:12:07 smithi007 ceph-mon[29459]: 6.6 scrub starts 2024-09-18T08:12:07.909 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:12:07 smithi007 ceph-mon[29459]: 6.6 scrub ok 2024-09-18T08:12:07.909 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:12:07 smithi007 ceph-mon[29459]: 6.17 scrub starts 2024-09-18T08:12:07.909 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:12:07 smithi007 ceph-mon[29459]: 6.17 scrub ok 2024-09-18T08:12:07.911 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:12:07 smithi007 ceph-mon[39249]: 6.7 scrub starts 2024-09-18T08:12:07.911 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:12:07 smithi007 ceph-mon[39249]: 6.7 scrub ok 2024-09-18T08:12:07.911 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:12:07 smithi007 ceph-mon[39249]: 6.6 scrub starts 2024-09-18T08:12:07.911 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:12:07 smithi007 ceph-mon[39249]: 6.6 scrub ok 2024-09-18T08:12:07.911 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:12:07 smithi007 ceph-mon[39249]: 6.17 scrub starts 2024-09-18T08:12:07.911 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:12:07 smithi007 ceph-mon[39249]: 6.17 scrub ok 2024-09-18T08:12:07.929 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6 2024-09-18T08:12:07.970 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:12:07.970 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:12:07.971 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6' 2024-09-18T08:12:08.071 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3 2024-09-18T08:12:08.106 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:12:08.107 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:12:08.107 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3' 2024-09-18T08:12:08.193 INFO:tasks.radosbench.radosbench.0.smithi007.stdout: 53 15 93239 93224 109.91 90.375 0.0103688 0.00909371 2024-09-18T08:12:08.209 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2 2024-09-18T08:12:08.240 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:12:08.241 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:12:08.241 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2' 2024-09-18T08:12:08.342 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6 2024-09-18T08:12:08.382 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:12:08.383 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:12:08.383 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6' 2024-09-18T08:12:08.484 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5 2024-09-18T08:12:08.522 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:12:08.523 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:12:08.523 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5' 2024-09-18T08:12:08.624 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2 2024-09-18T08:12:08.658 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:12:08.659 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:12:08.659 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2' 2024-09-18T08:12:08.760 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0 2024-09-18T08:12:08.793 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:12:08.793 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:12:08.794 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0' 2024-09-18T08:12:08.808 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:12:08 smithi007 ceph-mon[29459]: 3.15 scrub starts 2024-09-18T08:12:08.808 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:12:08 smithi007 ceph-mon[29459]: 3.15 scrub ok 2024-09-18T08:12:08.808 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:12:08 smithi007 ceph-mon[29459]: 3.5 scrub starts 2024-09-18T08:12:08.809 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:12:08 smithi007 ceph-mon[29459]: 3.5 scrub ok 2024-09-18T08:12:08.809 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:12:08 smithi007 ceph-mon[29459]: pgmap v448: 158 pgs: 30 active+undersized, 34 active+undersized+degraded, 94 active+clean; 5.3 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 640 B/s rd, 56 MiB/s wr, 903 op/s; 48905/260355 objects degraded (18.784%) 2024-09-18T08:12:08.809 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:12:08 smithi007 ceph-mon[39249]: 3.15 scrub starts 2024-09-18T08:12:08.809 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:12:08 smithi007 ceph-mon[39249]: 3.15 scrub ok 2024-09-18T08:12:08.809 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:12:08 smithi007 ceph-mon[39249]: 3.5 scrub starts 2024-09-18T08:12:08.809 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:12:08 smithi007 ceph-mon[39249]: 3.5 scrub ok 2024-09-18T08:12:08.809 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:12:08 smithi007 ceph-mon[39249]: pgmap v448: 158 pgs: 30 active+undersized, 34 active+undersized+degraded, 94 active+clean; 5.3 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 640 B/s rd, 56 MiB/s wr, 903 op/s; 48905/260355 objects degraded (18.784%) 2024-09-18T08:12:08.875 INFO:teuthology.orchestra.run.smithi007.stderr:Creating an OSD daemon form without an OSD FSID value 2024-09-18T08:12:08.894 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0 2024-09-18T08:12:08.905 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:12:08 smithi050 ceph-mon[32371]: 3.15 scrub starts 2024-09-18T08:12:08.906 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:12:08 smithi050 ceph-mon[32371]: 3.15 scrub ok 2024-09-18T08:12:08.906 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:12:08 smithi050 ceph-mon[32371]: 3.5 scrub starts 2024-09-18T08:12:08.906 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:12:08 smithi050 ceph-mon[32371]: 3.5 scrub ok 2024-09-18T08:12:08.906 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:12:08 smithi050 ceph-mon[32371]: pgmap v448: 158 pgs: 30 active+undersized, 34 active+undersized+degraded, 94 active+clean; 5.3 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 640 B/s rd, 56 MiB/s wr, 903 op/s; 48905/260355 objects degraded (18.784%) 2024-09-18T08:12:08.930 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:12:08.931 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:12:08.932 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0' 2024-09-18T08:12:09.032 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4 2024-09-18T08:12:09.075 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:12:09.075 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:12:09.076 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4' 2024-09-18T08:12:09.111 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/779349ca-7593-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T08:12:09.177 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7 2024-09-18T08:12:09.193 INFO:tasks.radosbench.radosbench.0.smithi007.stdout: 54 16 94735 94719 109.605 93.4375 0.0151394 0.009119 2024-09-18T08:12:09.220 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:12:09.221 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:12:09.221 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7' 2024-09-18T08:12:09.322 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0 2024-09-18T08:12:09.362 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:12:09.362 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:12:09.363 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0' 2024-09-18T08:12:09.464 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7 2024-09-18T08:12:09.500 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:12:09.501 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:12:09.501 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7' 2024-09-18T08:12:09.502 INFO:journalctl@ceph.iscsi.iscsi.a.smithi050.stdout:Sep 18 08:12:09 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-iscsi-iscsi-a[57782]: debug there is no tcmu-runner data available 2024-09-18T08:12:09.602 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0 2024-09-18T08:12:09.640 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:12:09.641 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:12:09.642 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0' 2024-09-18T08:12:09.730 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:12:09 smithi007 ceph-mon[29459]: Health check update: Degraded data redundancy: 48905/260355 objects degraded (18.784%), 34 pgs degraded (PG_DEGRADED) 2024-09-18T08:12:09.731 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:12:09 smithi007 ceph-mon[39249]: Health check update: Degraded data redundancy: 48905/260355 objects degraded (18.784%), 34 pgs degraded (PG_DEGRADED) 2024-09-18T08:12:09.742 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0 2024-09-18T08:12:09.799 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:12:09.800 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:12:09.800 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0' 2024-09-18T08:12:09.901 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4 2024-09-18T08:12:09.907 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:12:09 smithi050 ceph-mon[32371]: Health check update: Degraded data redundancy: 48905/260355 objects degraded (18.784%), 34 pgs degraded (PG_DEGRADED) 2024-09-18T08:12:09.946 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:12:09.947 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:12:09.947 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4' 2024-09-18T08:12:10.048 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3 2024-09-18T08:12:10.083 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:12:10.084 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:12:10.084 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3' 2024-09-18T08:12:10.186 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7 2024-09-18T08:12:10.196 INFO:tasks.radosbench.radosbench.0.smithi007.stdout: 55 16 96151 96135 109.218 88.5 0.00733314 0.00915098 2024-09-18T08:12:10.227 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:12:10.227 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:12:10.228 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7' 2024-09-18T08:12:10.329 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7 2024-09-18T08:12:10.362 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:12:10.363 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:12:10.363 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7' 2024-09-18T08:12:10.378 INFO:teuthology.orchestra.run.smithi007.stdout:osd.0: {} 2024-09-18T08:12:10.378 INFO:teuthology.orchestra.run.smithi007.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-18T08:12:10.378 INFO:teuthology.orchestra.run.smithi007.stderr:Error ENXIO: problem getting command descriptions from osd.1 2024-09-18T08:12:10.397 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/779349ca-7593-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T08:12:10.412 INFO:teuthology.orchestra.run.smithi007.stdout:osd.2: {} 2024-09-18T08:12:10.412 INFO:teuthology.orchestra.run.smithi007.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-18T08:12:10.453 INFO:teuthology.orchestra.run.smithi007.stdout:osd.3: {} 2024-09-18T08:12:10.453 INFO:teuthology.orchestra.run.smithi007.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-18T08:12:10.464 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6 2024-09-18T08:12:10.492 INFO:teuthology.orchestra.run.smithi007.stdout:osd.4: {} 2024-09-18T08:12:10.492 INFO:teuthology.orchestra.run.smithi007.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-18T08:12:10.498 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:12:10.499 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:12:10.499 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6' 2024-09-18T08:12:10.526 INFO:teuthology.orchestra.run.smithi007.stdout:osd.5: {} 2024-09-18T08:12:10.526 INFO:teuthology.orchestra.run.smithi007.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-18T08:12:10.530 INFO:teuthology.orchestra.run.smithi007.stderr:noscrub is set 2024-09-18T08:12:10.550 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:12:10 smithi007 ceph-mon[29459]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-18T08:12:10.551 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:12:10 smithi007 ceph-mon[29459]: pgmap v449: 158 pgs: 30 active+undersized, 34 active+undersized+degraded, 94 active+clean; 5.7 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 1.1 KiB/s rd, 101 MiB/s wr, 1.62k op/s; 52927/282204 objects degraded (18.755%) 2024-09-18T08:12:10.551 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:12:10 smithi007 ceph-mon[29459]: from='client.? 172.21.15.7:0/3676008317' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-18T08:12:10.551 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:12:10 smithi007 ceph-mon[29459]: from='client.? 172.21.15.7:0/526191093' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-18T08:12:10.551 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:12:10 smithi007 ceph-mon[39249]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-18T08:12:10.551 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:12:10 smithi007 ceph-mon[39249]: pgmap v449: 158 pgs: 30 active+undersized, 34 active+undersized+degraded, 94 active+clean; 5.7 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 1.1 KiB/s rd, 101 MiB/s wr, 1.62k op/s; 52927/282204 objects degraded (18.755%) 2024-09-18T08:12:10.551 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:12:10 smithi007 ceph-mon[39249]: from='client.? 172.21.15.7:0/3676008317' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-18T08:12:10.551 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:12:10 smithi007 ceph-mon[39249]: from='client.? 172.21.15.7:0/526191093' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-18T08:12:10.567 INFO:teuthology.orchestra.run.smithi007.stdout:osd.6: {} 2024-09-18T08:12:10.568 INFO:teuthology.orchestra.run.smithi007.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-18T08:12:10.600 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2 2024-09-18T08:12:10.623 INFO:teuthology.orchestra.run.smithi007.stdout:osd.7: {} 2024-09-18T08:12:10.623 INFO:teuthology.orchestra.run.smithi007.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-18T08:12:10.645 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:12:10.645 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:12:10.646 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2' 2024-09-18T08:12:10.747 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0 2024-09-18T08:12:10.786 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:12:10.786 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:12:10.787 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0' 2024-09-18T08:12:10.887 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5 2024-09-18T08:12:10.907 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:12:10 smithi050 ceph-mon[32371]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-18T08:12:10.907 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:12:10 smithi050 ceph-mon[32371]: pgmap v449: 158 pgs: 30 active+undersized, 34 active+undersized+degraded, 94 active+clean; 5.7 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 1.1 KiB/s rd, 101 MiB/s wr, 1.62k op/s; 52927/282204 objects degraded (18.755%) 2024-09-18T08:12:10.908 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:12:10 smithi050 ceph-mon[32371]: from='client.? 172.21.15.7:0/3676008317' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-18T08:12:10.908 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:12:10 smithi050 ceph-mon[32371]: from='client.? 172.21.15.7:0/526191093' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-18T08:12:10.925 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:12:10.926 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:12:10.926 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5' 2024-09-18T08:12:11.027 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6 2024-09-18T08:12:11.061 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:12:11.062 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:12:11.063 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6' 2024-09-18T08:12:11.163 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4 2024-09-18T08:12:11.195 INFO:tasks.radosbench.radosbench.0.smithi007.stdout: 56 16 97423 97407 108.688 79.5 0.0161928 0.00919555 2024-09-18T08:12:11.199 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:12:11.199 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:12:11.199 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4' 2024-09-18T08:12:11.300 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6 2024-09-18T08:12:11.346 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:12:11.347 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:12:11.347 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6' 2024-09-18T08:12:11.448 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6 2024-09-18T08:12:11.491 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:12:11.491 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:12:11.492 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6' 2024-09-18T08:12:11.593 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2 2024-09-18T08:12:11.635 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:12:11.636 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:12:11.637 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2' 2024-09-18T08:12:11.728 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:12:11 smithi007 ceph-mon[29459]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-18T08:12:11.728 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:12:11 smithi007 ceph-mon[29459]: from='client.? 172.21.15.7:0/3676008317' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-18T08:12:11.728 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:12:11 smithi007 ceph-mon[29459]: osdmap e254: 8 total, 7 up, 8 in 2024-09-18T08:12:11.728 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:12:11 smithi007 ceph-mon[39249]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-18T08:12:11.728 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:12:11 smithi007 ceph-mon[39249]: from='client.? 172.21.15.7:0/3676008317' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-18T08:12:11.728 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:12:11 smithi007 ceph-mon[39249]: osdmap e254: 8 total, 7 up, 8 in 2024-09-18T08:12:11.738 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7 2024-09-18T08:12:11.776 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:12:11.776 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:12:11.777 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7' 2024-09-18T08:12:11.878 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2 2024-09-18T08:12:11.899 INFO:teuthology.orchestra.run.smithi007.stdout:{ 2024-09-18T08:12:11.899 INFO:teuthology.orchestra.run.smithi007.stdout: "ops": [], 2024-09-18T08:12:11.899 INFO:teuthology.orchestra.run.smithi007.stdout: "complaint_time": 30, 2024-09-18T08:12:11.899 INFO:teuthology.orchestra.run.smithi007.stdout: "num_blocked_ops": 0 2024-09-18T08:12:11.899 INFO:teuthology.orchestra.run.smithi007.stdout:} 2024-09-18T08:12:11.905 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:12:11 smithi050 ceph-mon[32371]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-18T08:12:11.906 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:12:11 smithi050 ceph-mon[32371]: from='client.? 172.21.15.7:0/3676008317' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-18T08:12:11.906 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:12:11 smithi050 ceph-mon[32371]: osdmap e254: 8 total, 7 up, 8 in 2024-09-18T08:12:11.924 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:12:11.924 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:12:11.925 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2' 2024-09-18T08:12:12.026 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7 2024-09-18T08:12:12.069 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:12:12.069 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:12:12.070 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7' 2024-09-18T08:12:12.171 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3 2024-09-18T08:12:12.195 INFO:tasks.radosbench.radosbench.0.smithi007.stdout: 57 16 98847 98831 108.342 89 0.0228591 0.00922543 2024-09-18T08:12:12.208 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:12:12.208 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:12:12.208 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3' 2024-09-18T08:12:12.309 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0 2024-09-18T08:12:12.342 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:12:12.343 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:12:12.343 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0' 2024-09-18T08:12:12.396 INFO:teuthology.orchestra.run.smithi007.stderr:Exporting 4.f info 4.f( v 69'15 (0'0,69'15] local-lis/les=181/182 n=7 ec=63/63 lis/c=181/181 les/c/f=182/182/0 sis=181) 2024-09-18T08:12:12.397 INFO:teuthology.orchestra.run.smithi007.stderr:Read #4:f04f5452:::obj_delete_at_hint.0000000124:head# 2024-09-18T08:12:12.397 INFO:teuthology.orchestra.run.smithi007.stderr:Read #4:f1983ba0:::obj_delete_at_hint.0000000001:head# 2024-09-18T08:12:12.397 INFO:teuthology.orchestra.run.smithi007.stderr:Read #4:f24cb4a8:gc::gc.31:head# 2024-09-18T08:12:12.397 INFO:teuthology.orchestra.run.smithi007.stderr:Read #4:f2c84a15:::obj_delete_at_hint.0000000126:head# 2024-09-18T08:12:12.397 INFO:teuthology.orchestra.run.smithi007.stderr:Read #4:f3db1134:::obj_delete_at_hint.0000000055:head# 2024-09-18T08:12:12.397 INFO:teuthology.orchestra.run.smithi007.stderr:Read #4:f47f4d4d:::obj_delete_at_hint.0000000061:head# 2024-09-18T08:12:12.398 INFO:teuthology.orchestra.run.smithi007.stderr:Read #4:f6792a7a:::obj_delete_at_hint.0000000097:head# 2024-09-18T08:12:12.398 INFO:teuthology.orchestra.run.smithi007.stderr:Export successful 2024-09-18T08:12:12.444 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2 2024-09-18T08:12:12.476 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:12:12.477 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:12:12.477 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2' 2024-09-18T08:12:12.578 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2 2024-09-18T08:12:12.611 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:12:12.612 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:12:12.612 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2' 2024-09-18T08:12:12.693 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:12:12 smithi007 ceph-mon[29459]: pgmap v451: 158 pgs: 30 active+undersized, 34 active+undersized+degraded, 94 active+clean; 5.7 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 1.2 KiB/s rd, 106 MiB/s wr, 1.70k op/s; 52927/282204 objects degraded (18.755%) 2024-09-18T08:12:12.694 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:12:12 smithi007 ceph-mon[39249]: pgmap v451: 158 pgs: 30 active+undersized, 34 active+undersized+degraded, 94 active+clean; 5.7 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 1.2 KiB/s rd, 106 MiB/s wr, 1.70k op/s; 52927/282204 objects degraded (18.755%) 2024-09-18T08:12:12.713 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0 2024-09-18T08:12:12.747 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:12:12.748 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:12:12.748 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0' 2024-09-18T08:12:12.807 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:12:12 smithi050 ceph-mon[32371]: pgmap v451: 158 pgs: 30 active+undersized, 34 active+undersized+degraded, 94 active+clean; 5.7 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 1.2 KiB/s rd, 106 MiB/s wr, 1.70k op/s; 52927/282204 objects degraded (18.755%) 2024-09-18T08:12:12.849 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4 2024-09-18T08:12:12.885 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-09-18T08:12:12.885 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-09-18T08:12:12.890 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:12:12.891 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:12:12.892 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4' 2024-09-18T08:12:12.993 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6 2024-09-18T08:12:13.037 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:12:13.038 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:12:13.038 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6' 2024-09-18T08:12:13.139 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6 2024-09-18T08:12:13.181 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:12:13.181 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:12:13.182 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6' 2024-09-18T08:12:13.195 INFO:tasks.radosbench.radosbench.0.smithi007.stdout: 58 16 100288 100272 108.027 90.0625 0.0240963 0.00925191 2024-09-18T08:12:13.282 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7 2024-09-18T08:12:13.323 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:12:13.323 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:12:13.324 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7' 2024-09-18T08:12:13.425 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0 2024-09-18T08:12:13.456 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:12:13.457 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:12:13.457 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0' 2024-09-18T08:12:13.558 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6 2024-09-18T08:12:13.593 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid 779349ca-7593-11ef-bceb-c7b262605968 -- ceph daemon osd.3 dump_historic_ops 2024-09-18T08:12:13.603 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:12:13.603 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:12:13.603 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6' 2024-09-18T08:12:13.704 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4 2024-09-18T08:12:13.746 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:12:13.746 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:12:13.746 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4' 2024-09-18T08:12:13.847 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3 2024-09-18T08:12:13.884 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:12:13.884 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:12:13.885 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3' 2024-09-18T08:12:13.986 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2 2024-09-18T08:12:14.014 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:12:14.014 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:12:14.015 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2' 2024-09-18T08:12:14.116 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4 2024-09-18T08:12:14.159 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:12:14.159 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:12:14.160 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4' 2024-09-18T08:12:14.195 INFO:tasks.radosbench.radosbench.0.smithi007.stdout: 59 16 101467 101451 107.445 73.6875 0.0042414 0.00930261 2024-09-18T08:12:14.261 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4 2024-09-18T08:12:14.300 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:12:14.301 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:12:14.301 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4' 2024-09-18T08:12:14.402 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0 2024-09-18T08:12:14.405 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid 779349ca-7593-11ef-bceb-c7b262605968 -- ceph osd set nodeep-scrub 2024-09-18T08:12:14.434 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:12:14.435 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:12:14.436 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0' 2024-09-18T08:12:14.537 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0 2024-09-18T08:12:14.574 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:12:14.575 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:12:14.576 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0' 2024-09-18T08:12:14.676 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0 2024-09-18T08:12:14.702 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:12:14.703 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:12:14.703 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0' 2024-09-18T08:12:14.730 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:12:14 smithi007 ceph-mon[39249]: pgmap v452: 158 pgs: 30 active+undersized, 34 active+undersized+degraded, 94 active+clean; 5.8 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 568 B/s rd, 56 MiB/s wr, 903 op/s; 53773/284742 objects degraded (18.885%) 2024-09-18T08:12:14.730 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:12:14 smithi007 ceph-mon[39249]: Health check update: Degraded data redundancy: 53773/284742 objects degraded (18.885%), 34 pgs degraded (PG_DEGRADED) 2024-09-18T08:12:14.730 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:12:14 smithi007 ceph-mon[39249]: osdmap e255: 8 total, 7 up, 8 in 2024-09-18T08:12:14.731 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:12:14 smithi007 ceph-mon[29459]: pgmap v452: 158 pgs: 30 active+undersized, 34 active+undersized+degraded, 94 active+clean; 5.8 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 568 B/s rd, 56 MiB/s wr, 903 op/s; 53773/284742 objects degraded (18.885%) 2024-09-18T08:12:14.731 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:12:14 smithi007 ceph-mon[29459]: Health check update: Degraded data redundancy: 53773/284742 objects degraded (18.885%), 34 pgs degraded (PG_DEGRADED) 2024-09-18T08:12:14.731 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:12:14 smithi007 ceph-mon[29459]: osdmap e255: 8 total, 7 up, 8 in 2024-09-18T08:12:14.804 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7 2024-09-18T08:12:14.846 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:12:14.847 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:12:14.847 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7' 2024-09-18T08:12:14.865 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:12:14 smithi050 ceph-mon[32371]: pgmap v452: 158 pgs: 30 active+undersized, 34 active+undersized+degraded, 94 active+clean; 5.8 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 568 B/s rd, 56 MiB/s wr, 903 op/s; 53773/284742 objects degraded (18.885%) 2024-09-18T08:12:14.865 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:12:14 smithi050 ceph-mon[32371]: Health check update: Degraded data redundancy: 53773/284742 objects degraded (18.885%), 34 pgs degraded (PG_DEGRADED) 2024-09-18T08:12:14.865 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:12:14 smithi050 ceph-mon[32371]: osdmap e255: 8 total, 7 up, 8 in 2024-09-18T08:12:14.887 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid 779349ca-7593-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-18T08:12:14.949 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4 2024-09-18T08:12:14.987 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:12:14.987 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:12:14.987 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4' 2024-09-18T08:12:15.088 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3 2024-09-18T08:12:15.125 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:12:15.126 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:12:15.126 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3' 2024-09-18T08:12:15.195 INFO:tasks.radosbench.radosbench.0.smithi007.stdout:2024-09-18T08:12:15.194610+0000 min lat: 0.00114659 max lat: 1.04338 avg lat: 0.00934062 2024-09-18T08:12:15.195 INFO:tasks.radosbench.radosbench.0.smithi007.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-09-18T08:12:15.196 INFO:tasks.radosbench.radosbench.0.smithi007.stdout: 60 16 102766 102750 107.007 81.1875 0.00831398 0.00934062 2024-09-18T08:12:15.227 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6 2024-09-18T08:12:15.265 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:12:15.266 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:12:15.266 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6' 2024-09-18T08:12:15.367 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0 2024-09-18T08:12:15.401 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:12:15.402 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:12:15.403 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0' 2024-09-18T08:12:15.504 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7 2024-09-18T08:12:15.545 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:12:15.545 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:12:15.546 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7' 2024-09-18T08:12:15.581 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell -n osd.1 --fsid 779349ca-7593-11ef-bceb-c7b262605968 -- ceph-objectstore-tool --err-to-stderr --no-mon-config '--log-file=/var/log/ceph/objectstore_tool.$pid.log' --data-path /var/lib/ceph/osd/ceph-1 --journal-path /var/lib/ceph/osd/ceph-1/journal --force --op remove --pgid 4.f 2024-09-18T08:12:15.646 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4 2024-09-18T08:12:15.680 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:12:15.681 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:12:15.681 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4' 2024-09-18T08:12:15.761 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/779349ca-7593-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T08:12:15.782 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6 2024-09-18T08:12:15.790 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/779349ca-7593-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T08:12:15.797 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/779349ca-7593-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T08:12:15.818 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:12:15.819 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:12:15.819 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6' 2024-09-18T08:12:15.920 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6 2024-09-18T08:12:15.960 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:12:15.961 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:12:15.961 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6' 2024-09-18T08:12:16.062 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2 2024-09-18T08:12:16.069 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/779349ca-7593-11ef-bceb-c7b262605968/osd.1/config 2024-09-18T08:12:16.119 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:12:16.119 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:12:16.119 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2' 2024-09-18T08:12:16.195 INFO:tasks.radosbench.radosbench.0.smithi007.stdout: 61 16 104181 104165 106.703 88.4375 0.0100936 0.00936742 2024-09-18T08:12:16.221 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4 2024-09-18T08:12:16.259 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:12:16.260 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:12:16.260 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4' 2024-09-18T08:12:16.361 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6 2024-09-18T08:12:16.401 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:12:16.402 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:12:16.402 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6' 2024-09-18T08:12:16.503 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5 2024-09-18T08:12:16.539 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:12:16.540 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:12:16.540 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5' 2024-09-18T08:12:16.641 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7 2024-09-18T08:12:16.680 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:12:16.680 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:12:16.681 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7' 2024-09-18T08:12:16.699 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:12:16 smithi007 ceph-mon[29459]: pgmap v454: 158 pgs: 30 active+undersized, 34 active+undersized+degraded, 94 active+clean; 6.2 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 1.2 KiB/s rd, 111 MiB/s wr, 1.78k op/s; 56849/302976 objects degraded (18.764%) 2024-09-18T08:12:16.700 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:12:16 smithi007 ceph-mon[39249]: pgmap v454: 158 pgs: 30 active+undersized, 34 active+undersized+degraded, 94 active+clean; 6.2 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 1.2 KiB/s rd, 111 MiB/s wr, 1.78k op/s; 56849/302976 objects degraded (18.764%) 2024-09-18T08:12:16.781 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5 2024-09-18T08:12:16.816 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:12:16.816 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:12:16.817 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5' 2024-09-18T08:12:16.859 INFO:teuthology.orchestra.run.smithi007.stdout:{ 2024-09-18T08:12:16.859 INFO:teuthology.orchestra.run.smithi007.stdout: "size": 20, 2024-09-18T08:12:16.859 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 600, 2024-09-18T08:12:16.859 INFO:teuthology.orchestra.run.smithi007.stdout: "ops": [ 2024-09-18T08:12:16.859 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:12:16.859 INFO:teuthology.orchestra.run.smithi007.stdout: "description": "osd_op(client.15216.0:22606 8.f 8:f10856c4:::benchmark_data_smithi007_101583_object22605:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e174)", 2024-09-18T08:12:16.859 INFO:teuthology.orchestra.run.smithi007.stdout: "initiated_at": "2024-09-18T08:08:08.436554+0000", 2024-09-18T08:12:16.859 INFO:teuthology.orchestra.run.smithi007.stdout: "age": 248.41753324199999, 2024-09-18T08:12:16.859 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.768179902, 2024-09-18T08:12:16.859 INFO:teuthology.orchestra.run.smithi007.stdout: "continuous": false, 2024-09-18T08:12:16.860 INFO:teuthology.orchestra.run.smithi007.stdout: "type_data": { 2024-09-18T08:12:16.860 INFO:teuthology.orchestra.run.smithi007.stdout: "flag_point": "reached pg", 2024-09-18T08:12:16.860 INFO:teuthology.orchestra.run.smithi007.stdout: "client_info": { 2024-09-18T08:12:16.860 INFO:teuthology.orchestra.run.smithi007.stdout: "client": "client.15216", 2024-09-18T08:12:16.860 INFO:teuthology.orchestra.run.smithi007.stdout: "client_addr": "172.21.15.7:0/2596041177", 2024-09-18T08:12:16.860 INFO:teuthology.orchestra.run.smithi007.stdout: "tid": 22606 2024-09-18T08:12:16.860 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:12:16.860 INFO:teuthology.orchestra.run.smithi007.stdout: "events": [ 2024-09-18T08:12:16.860 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:12:16.861 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "initiated", 2024-09-18T08:12:16.861 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:08.436554+0000", 2024-09-18T08:12:16.861 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-09-18T08:12:16.861 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:12:16.861 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:12:16.861 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "header_read", 2024-09-18T08:12:16.861 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:08.436554+0000", 2024-09-18T08:12:16.861 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-09-18T08:12:16.861 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:12:16.861 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:12:16.861 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "throttled", 2024-09-18T08:12:16.861 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:08.436555+0000", 2024-09-18T08:12:16.861 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 7.8299999999999996e-07 2024-09-18T08:12:16.862 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:12:16.862 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:12:16.862 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "all_read", 2024-09-18T08:12:16.862 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:08.436576+0000", 2024-09-18T08:12:16.862 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 2.1214999999999999e-05 2024-09-18T08:12:16.862 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:12:16.862 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:12:16.862 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "dispatched", 2024-09-18T08:12:16.862 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:08.436586+0000", 2024-09-18T08:12:16.862 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.0088999999999999e-05 2024-09-18T08:12:16.862 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:12:16.862 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:12:16.862 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "queued_for_pg", 2024-09-18T08:12:16.863 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:08.436593+0000", 2024-09-18T08:12:16.863 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 7.3379999999999999e-06 2024-09-18T08:12:16.863 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:12:16.863 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:12:16.863 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-09-18T08:12:16.863 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:08.436642+0000", 2024-09-18T08:12:16.863 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 4.8928000000000003e-05 2024-09-18T08:12:16.863 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:12:16.863 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:12:16.863 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "started", 2024-09-18T08:12:16.863 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:08.436744+0000", 2024-09-18T08:12:16.863 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.000101778 2024-09-18T08:12:16.864 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:12:16.864 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:12:16.864 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "waiting for subops from 1,2", 2024-09-18T08:12:16.864 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:08.436890+0000", 2024-09-18T08:12:16.864 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.00014579099999999999 2024-09-18T08:12:16.864 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:12:16.864 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:12:16.864 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "op_commit", 2024-09-18T08:12:16.864 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:08.449655+0000", 2024-09-18T08:12:16.864 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.012765334 2024-09-18T08:12:16.864 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:12:16.864 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:12:16.864 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-09-18T08:12:16.865 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:08.460251+0000", 2024-09-18T08:12:16.865 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.010595635000000001 2024-09-18T08:12:16.865 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:12:16.865 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:12:16.865 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "waiting for peered", 2024-09-18T08:12:16.865 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:08.460264+0000", 2024-09-18T08:12:16.865 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.2921e-05 2024-09-18T08:12:16.865 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:12:16.865 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:12:16.865 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-09-18T08:12:16.865 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:09.152996+0000", 2024-09-18T08:12:16.865 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.69273260999999997 2024-09-18T08:12:16.866 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:12:16.866 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:12:16.866 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "waiting for peered", 2024-09-18T08:12:16.866 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:09.153008+0000", 2024-09-18T08:12:16.866 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.1459e-05 2024-09-18T08:12:16.866 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:12:16.866 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:12:16.866 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-09-18T08:12:16.866 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:10.204625+0000", 2024-09-18T08:12:16.866 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.0516172269999999 2024-09-18T08:12:16.866 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:12:16.866 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:12:16.867 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "done", 2024-09-18T08:12:16.867 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:10.204734+0000", 2024-09-18T08:12:16.867 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.000108794 2024-09-18T08:12:16.867 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-09-18T08:12:16.867 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-09-18T08:12:16.867 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-09-18T08:12:16.867 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:12:16.867 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:12:16.867 INFO:teuthology.orchestra.run.smithi007.stdout: "description": "osd_op(client.15216.0:22608 8.b 8:d92bd5a2:::benchmark_data_smithi007_101583_object22607:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e174)", 2024-09-18T08:12:16.867 INFO:teuthology.orchestra.run.smithi007.stdout: "initiated_at": "2024-09-18T08:08:08.436747+0000", 2024-09-18T08:12:16.868 INFO:teuthology.orchestra.run.smithi007.stdout: "age": 248.41733961099999, 2024-09-18T08:12:16.868 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.8187816670000001, 2024-09-18T08:12:16.868 INFO:teuthology.orchestra.run.smithi007.stdout: "continuous": false, 2024-09-18T08:12:16.868 INFO:teuthology.orchestra.run.smithi007.stdout: "type_data": { 2024-09-18T08:12:16.868 INFO:teuthology.orchestra.run.smithi007.stdout: "flag_point": "reached pg", 2024-09-18T08:12:16.868 INFO:teuthology.orchestra.run.smithi007.stdout: "client_info": { 2024-09-18T08:12:16.868 INFO:teuthology.orchestra.run.smithi007.stdout: "client": "client.15216", 2024-09-18T08:12:16.869 INFO:teuthology.orchestra.run.smithi007.stdout: "client_addr": "172.21.15.7:0/2596041177", 2024-09-18T08:12:16.869 INFO:teuthology.orchestra.run.smithi007.stdout: "tid": 22608 2024-09-18T08:12:16.869 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:12:16.869 INFO:teuthology.orchestra.run.smithi007.stdout: "events": [ 2024-09-18T08:12:16.869 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:12:16.869 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "initiated", 2024-09-18T08:12:16.869 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:08.436747+0000", 2024-09-18T08:12:16.869 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-09-18T08:12:16.869 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:12:16.869 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:12:16.869 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "header_read", 2024-09-18T08:12:16.869 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:08.436747+0000", 2024-09-18T08:12:16.870 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-09-18T08:12:16.870 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:12:16.870 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:12:16.870 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "throttled", 2024-09-18T08:12:16.870 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:08.436748+0000", 2024-09-18T08:12:16.870 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 9.2299999999999999e-07 2024-09-18T08:12:16.870 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:12:16.870 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:12:16.870 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "all_read", 2024-09-18T08:12:16.870 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:08.436769+0000", 2024-09-18T08:12:16.870 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 2.0392999999999999e-05 2024-09-18T08:12:16.870 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:12:16.870 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:12:16.871 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "dispatched", 2024-09-18T08:12:16.871 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:08.436778+0000", 2024-09-18T08:12:16.871 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 8.8689999999999993e-06 2024-09-18T08:12:16.871 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:12:16.871 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:12:16.871 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "queued_for_pg", 2024-09-18T08:12:16.871 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:08.436785+0000", 2024-09-18T08:12:16.871 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 7.199e-06 2024-09-18T08:12:16.871 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:12:16.871 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:12:16.871 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-09-18T08:12:16.871 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:08.437535+0000", 2024-09-18T08:12:16.871 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.00075027000000000004 2024-09-18T08:12:16.871 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:12:16.872 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:12:16.872 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "started", 2024-09-18T08:12:16.872 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:08.438489+0000", 2024-09-18T08:12:16.872 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.00095404400000000003 2024-09-18T08:12:16.872 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:12:16.872 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:12:16.872 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "waiting for subops from 1,2", 2024-09-18T08:12:16.872 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:08.438742+0000", 2024-09-18T08:12:16.872 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.00025305599999999998 2024-09-18T08:12:16.872 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:12:16.872 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:12:16.872 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "op_commit", 2024-09-18T08:12:16.872 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:08.440394+0000", 2024-09-18T08:12:16.873 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.001652096 2024-09-18T08:12:16.873 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:12:16.873 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:12:16.873 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:12:16.873 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:08.449571+0000", 2024-09-18T08:12:16.873 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.0091767580000000001 2024-09-18T08:12:16.873 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:12:16.873 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:12:16.873 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-09-18T08:12:16.873 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:08.455097+0000", 2024-09-18T08:12:16.873 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.0055257730000000003 2024-09-18T08:12:16.873 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:12:16.873 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:12:16.874 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "waiting for peered", 2024-09-18T08:12:16.874 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:08.455115+0000", 2024-09-18T08:12:16.874 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.8501000000000001e-05 2024-09-18T08:12:16.874 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:12:16.874 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:12:16.874 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-09-18T08:12:16.874 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:09.152107+0000", 2024-09-18T08:12:16.874 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.69699121100000005 2024-09-18T08:12:16.874 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:12:16.874 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:12:16.874 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "waiting for peered", 2024-09-18T08:12:16.875 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:09.152128+0000", 2024-09-18T08:12:16.875 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 2.1557e-05 2024-09-18T08:12:16.875 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:12:16.876 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:12:16.876 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-09-18T08:12:16.876 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:10.255451+0000", 2024-09-18T08:12:16.876 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.1033232070000001 2024-09-18T08:12:16.876 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:12:16.876 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:12:16.876 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "done", 2024-09-18T08:12:16.876 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:10.255529+0000", 2024-09-18T08:12:16.876 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 7.7810000000000002e-05 2024-09-18T08:12:16.876 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-09-18T08:12:16.876 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-09-18T08:12:16.876 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-09-18T08:12:16.877 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:12:16.877 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:12:16.877 INFO:teuthology.orchestra.run.smithi007.stdout: "description": "osd_op(client.15216.0:22614 8.f 8:f07b5883:::benchmark_data_smithi007_101583_object22613:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e174)", 2024-09-18T08:12:16.877 INFO:teuthology.orchestra.run.smithi007.stdout: "initiated_at": "2024-09-18T08:08:08.439788+0000", 2024-09-18T08:12:16.877 INFO:teuthology.orchestra.run.smithi007.stdout: "age": 248.414299548, 2024-09-18T08:12:16.877 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.7882368309999999, 2024-09-18T08:12:16.877 INFO:teuthology.orchestra.run.smithi007.stdout: "continuous": false, 2024-09-18T08:12:16.877 INFO:teuthology.orchestra.run.smithi007.stdout: "type_data": { 2024-09-18T08:12:16.877 INFO:teuthology.orchestra.run.smithi007.stdout: "flag_point": "reached pg", 2024-09-18T08:12:16.877 INFO:teuthology.orchestra.run.smithi007.stdout: "client_info": { 2024-09-18T08:12:16.878 INFO:teuthology.orchestra.run.smithi007.stdout: "client": "client.15216", 2024-09-18T08:12:16.878 INFO:teuthology.orchestra.run.smithi007.stdout: "client_addr": "172.21.15.7:0/2596041177", 2024-09-18T08:12:16.878 INFO:teuthology.orchestra.run.smithi007.stdout: "tid": 22614 2024-09-18T08:12:16.878 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:12:16.878 INFO:teuthology.orchestra.run.smithi007.stdout: "events": [ 2024-09-18T08:12:16.878 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:12:16.878 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "initiated", 2024-09-18T08:12:16.878 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:08.439788+0000", 2024-09-18T08:12:16.878 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-09-18T08:12:16.878 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:12:16.878 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:12:16.878 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "header_read", 2024-09-18T08:12:16.878 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:08.439788+0000", 2024-09-18T08:12:16.879 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-09-18T08:12:16.879 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:12:16.879 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:12:16.879 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "throttled", 2024-09-18T08:12:16.879 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:08.439789+0000", 2024-09-18T08:12:16.879 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.5009999999999999e-06 2024-09-18T08:12:16.879 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:12:16.879 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:12:16.879 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "all_read", 2024-09-18T08:12:16.879 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:08.439813+0000", 2024-09-18T08:12:16.879 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 2.442e-05 2024-09-18T08:12:16.879 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:12:16.879 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:12:16.880 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "dispatched", 2024-09-18T08:12:16.880 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:08.439827+0000", 2024-09-18T08:12:16.880 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.3540000000000001e-05 2024-09-18T08:12:16.880 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:12:16.880 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:12:16.880 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "queued_for_pg", 2024-09-18T08:12:16.880 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:08.439836+0000", 2024-09-18T08:12:16.880 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 8.7630000000000006e-06 2024-09-18T08:12:16.880 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:12:16.880 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:12:16.880 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-09-18T08:12:16.880 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:08.445915+0000", 2024-09-18T08:12:16.880 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.0060795210000000001 2024-09-18T08:12:16.881 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:12:16.881 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:12:16.881 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "started", 2024-09-18T08:12:16.881 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:08.446054+0000", 2024-09-18T08:12:16.881 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.00013859699999999999 2024-09-18T08:12:16.881 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:12:16.881 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:12:16.881 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "waiting for subops from 1,2", 2024-09-18T08:12:16.881 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:08.448436+0000", 2024-09-18T08:12:16.881 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.0023821939999999998 2024-09-18T08:12:16.881 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:12:16.881 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:12:16.881 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "op_commit", 2024-09-18T08:12:16.882 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:08.449664+0000", 2024-09-18T08:12:16.882 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.0012275000000000001 2024-09-18T08:12:16.882 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:12:16.882 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:12:16.882 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-09-18T08:12:16.882 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:08.460300+0000", 2024-09-18T08:12:16.882 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.010636487999999999 2024-09-18T08:12:16.882 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:12:16.882 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:12:16.883 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "waiting for peered", 2024-09-18T08:12:16.883 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:08.460310+0000", 2024-09-18T08:12:16.883 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 9.5240000000000006e-06 2024-09-18T08:12:16.883 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:12:16.883 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:12:16.883 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-09-18T08:12:16.883 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:09.153052+0000", 2024-09-18T08:12:16.883 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.69274252300000005 2024-09-18T08:12:16.883 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:12:16.883 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:12:16.883 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "waiting for peered", 2024-09-18T08:12:16.883 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:09.153062+0000", 2024-09-18T08:12:16.883 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 9.7569999999999999e-06 2024-09-18T08:12:16.884 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:12:16.884 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:12:16.884 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-09-18T08:12:16.884 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:10.220005+0000", 2024-09-18T08:12:16.884 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.0669431330000001 2024-09-18T08:12:16.884 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:12:16.884 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:12:16.884 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "waiting for degraded object", 2024-09-18T08:12:16.884 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:10.220517+0000", 2024-09-18T08:12:16.884 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.00051243100000000004 2024-09-18T08:12:16.884 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:12:16.885 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:12:16.885 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-09-18T08:12:16.885 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:10.227921+0000", 2024-09-18T08:12:16.885 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.0074033850000000002 2024-09-18T08:12:16.885 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:12:16.885 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:12:16.885 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "done", 2024-09-18T08:12:16.885 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:10.228024+0000", 2024-09-18T08:12:16.885 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.000103554 2024-09-18T08:12:16.885 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-09-18T08:12:16.885 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-09-18T08:12:16.885 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-09-18T08:12:16.886 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:12:16.886 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:12:16.886 INFO:teuthology.orchestra.run.smithi007.stdout: "description": "osd_op(client.15216.0:22617 8.b 8:dab1c1d5:::benchmark_data_smithi007_101583_object22616:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e174)", 2024-09-18T08:12:16.886 INFO:teuthology.orchestra.run.smithi007.stdout: "initiated_at": "2024-09-18T08:08:08.447023+0000", 2024-09-18T08:12:16.886 INFO:teuthology.orchestra.run.smithi007.stdout: "age": 248.407063736, 2024-09-18T08:12:16.886 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.8206128700000002, 2024-09-18T08:12:16.886 INFO:teuthology.orchestra.run.smithi007.stdout: "continuous": false, 2024-09-18T08:12:16.886 INFO:teuthology.orchestra.run.smithi007.stdout: "type_data": { 2024-09-18T08:12:16.886 INFO:teuthology.orchestra.run.smithi007.stdout: "flag_point": "reached pg", 2024-09-18T08:12:16.886 INFO:teuthology.orchestra.run.smithi007.stdout: "client_info": { 2024-09-18T08:12:16.886 INFO:teuthology.orchestra.run.smithi007.stdout: "client": "client.15216", 2024-09-18T08:12:16.886 INFO:teuthology.orchestra.run.smithi007.stdout: "client_addr": "172.21.15.7:0/2596041177", 2024-09-18T08:12:16.886 INFO:teuthology.orchestra.run.smithi007.stdout: "tid": 22617 2024-09-18T08:12:16.887 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:12:16.887 INFO:teuthology.orchestra.run.smithi007.stdout: "events": [ 2024-09-18T08:12:16.887 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:12:16.887 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "initiated", 2024-09-18T08:12:16.887 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:08.447023+0000", 2024-09-18T08:12:16.887 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-09-18T08:12:16.887 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:12:16.887 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:12:16.887 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "header_read", 2024-09-18T08:12:16.887 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:08.447023+0000", 2024-09-18T08:12:16.887 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-09-18T08:12:16.887 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:12:16.887 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:12:16.887 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "throttled", 2024-09-18T08:12:16.888 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:08.447025+0000", 2024-09-18T08:12:16.888 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.685e-06 2024-09-18T08:12:16.888 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:12:16.888 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:12:16.888 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "all_read", 2024-09-18T08:12:16.888 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:08.447053+0000", 2024-09-18T08:12:16.888 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 2.7953000000000002e-05 2024-09-18T08:12:16.888 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:12:16.888 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:12:16.888 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "dispatched", 2024-09-18T08:12:16.888 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:08.447070+0000", 2024-09-18T08:12:16.888 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.6650000000000002e-05 2024-09-18T08:12:16.888 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:12:16.889 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:12:16.889 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "queued_for_pg", 2024-09-18T08:12:16.889 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:08.447080+0000", 2024-09-18T08:12:16.889 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.0148e-05 2024-09-18T08:12:16.889 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:12:16.889 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:12:16.889 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-09-18T08:12:16.889 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:08.449604+0000", 2024-09-18T08:12:16.889 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.0025244640000000001 2024-09-18T08:12:16.889 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:12:16.889 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:12:16.889 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "started", 2024-09-18T08:12:16.889 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:08.449753+0000", 2024-09-18T08:12:16.890 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.00014878600000000001 2024-09-18T08:12:16.890 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:12:16.890 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:12:16.890 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "waiting for subops from 1,2", 2024-09-18T08:12:16.890 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:08.449904+0000", 2024-09-18T08:12:16.890 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.00015076299999999999 2024-09-18T08:12:16.890 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:12:16.890 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:12:16.890 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-09-18T08:12:16.890 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:08.455203+0000", 2024-09-18T08:12:16.890 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.0052987950000000002 2024-09-18T08:12:16.890 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:12:16.890 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:12:16.890 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "waiting for peered", 2024-09-18T08:12:16.891 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:08.455211+0000", 2024-09-18T08:12:16.891 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 8.8179999999999993e-06 2024-09-18T08:12:16.891 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:12:16.891 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:12:16.891 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-09-18T08:12:16.891 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:09.152202+0000", 2024-09-18T08:12:16.891 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.69699049000000002 2024-09-18T08:12:16.891 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:12:16.891 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:12:16.891 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "waiting for peered", 2024-09-18T08:12:16.891 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:09.152212+0000", 2024-09-18T08:12:16.892 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.0251000000000001e-05 2024-09-18T08:12:16.892 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:12:16.892 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:12:16.892 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-09-18T08:12:16.892 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:10.256406+0000", 2024-09-18T08:12:16.892 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.104194101 2024-09-18T08:12:16.892 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:12:16.892 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:12:16.892 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "waiting for degraded object", 2024-09-18T08:12:16.892 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:10.256961+0000", 2024-09-18T08:12:16.893 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.00055520899999999998 2024-09-18T08:12:16.893 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:12:16.893 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:12:16.893 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-09-18T08:12:16.893 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:10.267550+0000", 2024-09-18T08:12:16.893 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.010588640999999999 2024-09-18T08:12:16.893 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:12:16.893 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:12:16.893 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "done", 2024-09-18T08:12:16.893 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:10.267636+0000", 2024-09-18T08:12:16.893 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 8.6116e-05 2024-09-18T08:12:16.894 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-09-18T08:12:16.894 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-09-18T08:12:16.894 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-09-18T08:12:16.894 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:12:16.894 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:12:16.894 INFO:teuthology.orchestra.run.smithi007.stdout: "description": "osd_op(client.15216.0:22630 8.b 8:da804186:::benchmark_data_smithi007_101583_object22629:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e175)", 2024-09-18T08:12:16.894 INFO:teuthology.orchestra.run.smithi007.stdout: "initiated_at": "2024-09-18T08:08:08.467388+0000", 2024-09-18T08:12:16.894 INFO:teuthology.orchestra.run.smithi007.stdout: "age": 248.386698989, 2024-09-18T08:12:16.894 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.8127456770000001, 2024-09-18T08:12:16.894 INFO:teuthology.orchestra.run.smithi007.stdout: "continuous": false, 2024-09-18T08:12:16.894 INFO:teuthology.orchestra.run.smithi007.stdout: "type_data": { 2024-09-18T08:12:16.894 INFO:teuthology.orchestra.run.smithi007.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T08:12:16.894 INFO:teuthology.orchestra.run.smithi007.stdout: "client_info": { 2024-09-18T08:12:16.895 INFO:teuthology.orchestra.run.smithi007.stdout: "client": "client.15216", 2024-09-18T08:12:16.895 INFO:teuthology.orchestra.run.smithi007.stdout: "client_addr": "172.21.15.7:0/2596041177", 2024-09-18T08:12:16.895 INFO:teuthology.orchestra.run.smithi007.stdout: "tid": 22630 2024-09-18T08:12:16.895 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:12:16.895 INFO:teuthology.orchestra.run.smithi007.stdout: "events": [ 2024-09-18T08:12:16.895 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:12:16.895 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "initiated", 2024-09-18T08:12:16.895 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:08.467388+0000", 2024-09-18T08:12:16.895 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-09-18T08:12:16.895 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:12:16.895 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:12:16.895 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "header_read", 2024-09-18T08:12:16.895 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:08.467388+0000", 2024-09-18T08:12:16.895 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-09-18T08:12:16.896 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:12:16.896 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:12:16.896 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "throttled", 2024-09-18T08:12:16.896 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:08.467389+0000", 2024-09-18T08:12:16.896 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.373e-06 2024-09-18T08:12:16.896 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:12:16.896 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:12:16.896 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "all_read", 2024-09-18T08:12:16.896 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:08.467447+0000", 2024-09-18T08:12:16.896 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 5.7238999999999999e-05 2024-09-18T08:12:16.896 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:12:16.896 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:12:16.896 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "dispatched", 2024-09-18T08:12:16.897 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:08.467493+0000", 2024-09-18T08:12:16.897 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 4.6755000000000002e-05 2024-09-18T08:12:16.897 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:12:16.897 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:12:16.897 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "queued_for_pg", 2024-09-18T08:12:16.897 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:08.467513+0000", 2024-09-18T08:12:16.897 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.9873000000000001e-05 2024-09-18T08:12:16.897 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:12:16.897 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:12:16.897 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-09-18T08:12:16.897 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:08.467970+0000", 2024-09-18T08:12:16.897 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.00045696899999999999 2024-09-18T08:12:16.898 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:12:16.898 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:12:16.898 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "waiting for peered", 2024-09-18T08:12:16.898 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:08.467979+0000", 2024-09-18T08:12:16.898 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 8.6440000000000006e-06 2024-09-18T08:12:16.898 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:12:16.898 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:12:16.898 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-09-18T08:12:16.898 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:09.152276+0000", 2024-09-18T08:12:16.898 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.68429694500000005 2024-09-18T08:12:16.898 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:12:16.898 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:12:16.898 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "waiting for peered", 2024-09-18T08:12:16.899 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:09.152284+0000", 2024-09-18T08:12:16.899 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 7.7049999999999999e-06 2024-09-18T08:12:16.899 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:12:16.899 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:12:16.899 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-09-18T08:12:16.899 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:10.257004+0000", 2024-09-18T08:12:16.899 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.104719961 2024-09-18T08:12:16.899 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:12:16.899 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:12:16.899 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "started", 2024-09-18T08:12:16.899 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:10.258354+0000", 2024-09-18T08:12:16.899 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.001350097 2024-09-18T08:12:16.899 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:12:16.899 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:12:16.900 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "waiting for subops from 1,7", 2024-09-18T08:12:16.900 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:10.258951+0000", 2024-09-18T08:12:16.900 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.00059747000000000001 2024-09-18T08:12:16.900 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:12:16.900 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:12:16.900 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:12:16.900 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:10.267720+0000", 2024-09-18T08:12:16.900 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.0087689169999999993 2024-09-18T08:12:16.900 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:12:16.901 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:12:16.901 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:12:16.901 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:10.267767+0000", 2024-09-18T08:12:16.901 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 4.6616000000000002e-05 2024-09-18T08:12:16.901 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:12:16.901 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:12:16.901 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "op_commit", 2024-09-18T08:12:16.901 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:10.278730+0000", 2024-09-18T08:12:16.901 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.010963818 2024-09-18T08:12:16.901 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:12:16.901 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:12:16.901 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "commit_sent", 2024-09-18T08:12:16.902 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:10.278929+0000", 2024-09-18T08:12:16.902 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.000198638 2024-09-18T08:12:16.902 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:12:16.902 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:12:16.902 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "done", 2024-09-18T08:12:16.902 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:10.280134+0000", 2024-09-18T08:12:16.902 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.001204657 2024-09-18T08:12:16.902 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-09-18T08:12:16.902 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-09-18T08:12:16.902 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-09-18T08:12:16.902 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:12:16.902 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:12:16.902 INFO:teuthology.orchestra.run.smithi007.stdout: "description": "osd_op(client.15216.0:22665 8.f 8:f8e596fa:::benchmark_data_smithi007_101583_object22664:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e175)", 2024-09-18T08:12:16.902 INFO:teuthology.orchestra.run.smithi007.stdout: "initiated_at": "2024-09-18T08:08:08.484581+0000", 2024-09-18T08:12:16.903 INFO:teuthology.orchestra.run.smithi007.stdout: "age": 248.36950606900001, 2024-09-18T08:12:16.903 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.7443272359999999, 2024-09-18T08:12:16.903 INFO:teuthology.orchestra.run.smithi007.stdout: "continuous": false, 2024-09-18T08:12:16.903 INFO:teuthology.orchestra.run.smithi007.stdout: "type_data": { 2024-09-18T08:12:16.903 INFO:teuthology.orchestra.run.smithi007.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T08:12:16.903 INFO:teuthology.orchestra.run.smithi007.stdout: "client_info": { 2024-09-18T08:12:16.903 INFO:teuthology.orchestra.run.smithi007.stdout: "client": "client.15216", 2024-09-18T08:12:16.903 INFO:teuthology.orchestra.run.smithi007.stdout: "client_addr": "172.21.15.7:0/2596041177", 2024-09-18T08:12:16.903 INFO:teuthology.orchestra.run.smithi007.stdout: "tid": 22665 2024-09-18T08:12:16.903 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:12:16.903 INFO:teuthology.orchestra.run.smithi007.stdout: "events": [ 2024-09-18T08:12:16.903 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:12:16.903 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "initiated", 2024-09-18T08:12:16.903 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:08.484581+0000", 2024-09-18T08:12:16.904 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-09-18T08:12:16.904 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:12:16.904 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:12:16.904 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "header_read", 2024-09-18T08:12:16.904 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:08.484581+0000", 2024-09-18T08:12:16.904 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-09-18T08:12:16.904 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:12:16.904 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:12:16.904 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "throttled", 2024-09-18T08:12:16.904 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:08.484583+0000", 2024-09-18T08:12:16.904 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.9250000000000002e-06 2024-09-18T08:12:16.904 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:12:16.904 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:12:16.905 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "all_read", 2024-09-18T08:12:16.905 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:08.484606+0000", 2024-09-18T08:12:16.905 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 2.2965000000000001e-05 2024-09-18T08:12:16.905 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:12:16.905 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:12:16.905 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "dispatched", 2024-09-18T08:12:16.905 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:08.484617+0000", 2024-09-18T08:12:16.905 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.1240000000000001e-05 2024-09-18T08:12:16.905 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:12:16.905 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:12:16.905 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "queued_for_pg", 2024-09-18T08:12:16.906 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:08.484625+0000", 2024-09-18T08:12:16.906 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 7.5020000000000003e-06 2024-09-18T08:12:16.906 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:12:16.906 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:12:16.906 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-09-18T08:12:16.906 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:08.484862+0000", 2024-09-18T08:12:16.907 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.00023732500000000001 2024-09-18T08:12:16.907 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:12:16.907 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:12:16.907 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "waiting for peered", 2024-09-18T08:12:16.907 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:08.484872+0000", 2024-09-18T08:12:16.907 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.0231000000000001e-05 2024-09-18T08:12:16.908 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:12:16.908 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:12:16.908 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-09-18T08:12:16.908 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:09.153091+0000", 2024-09-18T08:12:16.909 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.66821881900000002 2024-09-18T08:12:16.909 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:12:16.909 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:12:16.909 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "waiting for peered", 2024-09-18T08:12:16.909 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:09.153100+0000", 2024-09-18T08:12:16.909 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 8.7560000000000005e-06 2024-09-18T08:12:16.910 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:12:16.910 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:12:16.910 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-09-18T08:12:16.910 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:10.220552+0000", 2024-09-18T08:12:16.910 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.0674517699999999 2024-09-18T08:12:16.910 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:12:16.910 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:12:16.910 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "started", 2024-09-18T08:12:16.911 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:10.220667+0000", 2024-09-18T08:12:16.911 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.000115671 2024-09-18T08:12:16.911 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:12:16.911 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:12:16.911 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "waiting for subops from 1,6", 2024-09-18T08:12:16.911 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:10.220826+0000", 2024-09-18T08:12:16.912 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.00015917700000000001 2024-09-18T08:12:16.912 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:12:16.912 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:12:16.912 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "op_commit", 2024-09-18T08:12:16.912 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:10.224779+0000", 2024-09-18T08:12:16.912 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.0039521119999999998 2024-09-18T08:12:16.913 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:12:16.913 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:12:16.913 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:12:16.913 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:10.228102+0000", 2024-09-18T08:12:16.913 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.0033237230000000002 2024-09-18T08:12:16.913 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:12:16.914 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:12:16.914 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:12:16.914 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:10.228153+0000", 2024-09-18T08:12:16.914 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 5.0705000000000003e-05 2024-09-18T08:12:16.914 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:12:16.914 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:12:16.914 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "commit_sent", 2024-09-18T08:12:16.914 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:10.228212+0000", 2024-09-18T08:12:16.915 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 5.9404999999999998e-05 2024-09-18T08:12:16.915 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:12:16.915 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:12:16.915 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "done", 2024-09-18T08:12:16.915 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:10.228908+0000", 2024-09-18T08:12:16.915 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.00069591 2024-09-18T08:12:16.915 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-09-18T08:12:16.915 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-09-18T08:12:16.915 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-09-18T08:12:16.915 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:12:16.915 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:12:16.915 INFO:teuthology.orchestra.run.smithi007.stdout: "description": "osd_op(client.15216.0:22668 8.f 8:f26239dc:::benchmark_data_smithi007_101583_object22667:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e175)", 2024-09-18T08:12:16.915 INFO:teuthology.orchestra.run.smithi007.stdout: "initiated_at": "2024-09-18T08:08:08.486691+0000", 2024-09-18T08:12:16.916 INFO:teuthology.orchestra.run.smithi007.stdout: "age": 248.36739601100001, 2024-09-18T08:12:16.916 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.7435772489999999, 2024-09-18T08:12:16.916 INFO:teuthology.orchestra.run.smithi007.stdout: "continuous": false, 2024-09-18T08:12:16.916 INFO:teuthology.orchestra.run.smithi007.stdout: "type_data": { 2024-09-18T08:12:16.916 INFO:teuthology.orchestra.run.smithi007.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T08:12:16.916 INFO:teuthology.orchestra.run.smithi007.stdout: "client_info": { 2024-09-18T08:12:16.916 INFO:teuthology.orchestra.run.smithi007.stdout: "client": "client.15216", 2024-09-18T08:12:16.916 INFO:teuthology.orchestra.run.smithi007.stdout: "client_addr": "172.21.15.7:0/2596041177", 2024-09-18T08:12:16.916 INFO:teuthology.orchestra.run.smithi007.stdout: "tid": 22668 2024-09-18T08:12:16.916 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:12:16.916 INFO:teuthology.orchestra.run.smithi007.stdout: "events": [ 2024-09-18T08:12:16.916 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:12:16.916 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "initiated", 2024-09-18T08:12:16.917 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:08.486691+0000", 2024-09-18T08:12:16.917 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-09-18T08:12:16.917 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:12:16.917 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:12:16.917 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "header_read", 2024-09-18T08:12:16.917 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:08.486691+0000", 2024-09-18T08:12:16.917 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-09-18T08:12:16.917 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:12:16.917 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:12:16.917 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "throttled", 2024-09-18T08:12:16.917 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:08.486692+0000", 2024-09-18T08:12:16.917 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.3030000000000001e-06 2024-09-18T08:12:16.918 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:12:16.918 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:12:16.918 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "all_read", 2024-09-18T08:12:16.918 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:08.486716+0000", 2024-09-18T08:12:16.918 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 2.3164999999999999e-05 2024-09-18T08:12:16.918 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:12:16.918 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:12:16.918 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "dispatched", 2024-09-18T08:12:16.918 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:08.486728+0000", 2024-09-18T08:12:16.918 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.2648e-05 2024-09-18T08:12:16.918 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:12:16.918 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:12:16.918 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "queued_for_pg", 2024-09-18T08:12:16.919 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:08.486736+0000", 2024-09-18T08:12:16.919 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 7.8779999999999998e-06 2024-09-18T08:12:16.919 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:12:16.919 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:12:16.919 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-09-18T08:12:16.919 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:08.486921+0000", 2024-09-18T08:12:16.919 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.00018532100000000001 2024-09-18T08:12:16.919 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:12:16.919 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:12:16.919 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "waiting for peered", 2024-09-18T08:12:16.919 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:08.486930+0000", 2024-09-18T08:12:16.920 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 8.6300000000000004e-06 2024-09-18T08:12:16.920 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:12:16.920 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:12:16.920 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-09-18T08:12:16.920 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:09.153126+0000", 2024-09-18T08:12:16.920 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.66619636299999996 2024-09-18T08:12:16.920 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:12:16.920 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:12:16.920 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "waiting for peered", 2024-09-18T08:12:16.920 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:09.153280+0000", 2024-09-18T08:12:16.920 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.00015323800000000001 2024-09-18T08:12:16.920 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:12:16.921 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:12:16.921 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-09-18T08:12:16.921 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:10.223256+0000", 2024-09-18T08:12:16.921 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.069976313 2024-09-18T08:12:16.921 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:12:16.921 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:12:16.921 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "started", 2024-09-18T08:12:16.921 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:10.223428+0000", 2024-09-18T08:12:16.921 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.00017207299999999999 2024-09-18T08:12:16.921 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:12:16.921 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:12:16.921 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "waiting for subops from 1,6", 2024-09-18T08:12:16.921 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:10.223632+0000", 2024-09-18T08:12:16.922 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.000204289 2024-09-18T08:12:16.922 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:12:16.922 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:12:16.922 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "op_commit", 2024-09-18T08:12:16.922 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:10.228023+0000", 2024-09-18T08:12:16.922 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.0043910140000000004 2024-09-18T08:12:16.922 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:12:16.922 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:12:16.922 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:12:16.922 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:10.228966+0000", 2024-09-18T08:12:16.922 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.00094256400000000001 2024-09-18T08:12:16.922 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:12:16.922 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:12:16.923 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:12:16.923 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:10.229018+0000", 2024-09-18T08:12:16.923 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 5.2089000000000002e-05 2024-09-18T08:12:16.923 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:12:16.923 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:12:16.923 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "commit_sent", 2024-09-18T08:12:16.923 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:10.229075+0000", 2024-09-18T08:12:16.923 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 5.7079e-05 2024-09-18T08:12:16.923 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:12:16.923 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:12:16.923 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "done", 2024-09-18T08:12:16.923 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:10.230268+0000", 2024-09-18T08:12:16.923 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.0011932819999999999 2024-09-18T08:12:16.923 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-09-18T08:12:16.924 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-09-18T08:12:16.924 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-09-18T08:12:16.924 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:12:16.924 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:12:16.924 INFO:teuthology.orchestra.run.smithi007.stdout: "description": "osd_op(client.15216.0:22675 8.f 8:f41edfb7:::benchmark_data_smithi007_101583_object22674:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e175)", 2024-09-18T08:12:16.924 INFO:teuthology.orchestra.run.smithi007.stdout: "initiated_at": "2024-09-18T08:08:08.490781+0000", 2024-09-18T08:12:16.924 INFO:teuthology.orchestra.run.smithi007.stdout: "age": 248.363306198, 2024-09-18T08:12:16.924 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.74874595, 2024-09-18T08:12:16.924 INFO:teuthology.orchestra.run.smithi007.stdout: "continuous": false, 2024-09-18T08:12:16.924 INFO:teuthology.orchestra.run.smithi007.stdout: "type_data": { 2024-09-18T08:12:16.924 INFO:teuthology.orchestra.run.smithi007.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T08:12:16.924 INFO:teuthology.orchestra.run.smithi007.stdout: "client_info": { 2024-09-18T08:12:16.924 INFO:teuthology.orchestra.run.smithi007.stdout: "client": "client.15216", 2024-09-18T08:12:16.925 INFO:teuthology.orchestra.run.smithi007.stdout: "client_addr": "172.21.15.7:0/2596041177", 2024-09-18T08:12:16.925 INFO:teuthology.orchestra.run.smithi007.stdout: "tid": 22675 2024-09-18T08:12:16.925 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:12:16.925 INFO:teuthology.orchestra.run.smithi007.stdout: "events": [ 2024-09-18T08:12:16.925 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:12:16.925 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "initiated", 2024-09-18T08:12:16.925 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:08.490781+0000", 2024-09-18T08:12:16.925 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-09-18T08:12:16.925 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:12:16.925 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:12:16.925 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "header_read", 2024-09-18T08:12:16.926 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5 2024-09-18T08:12:16.928 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:12:16 smithi050 ceph-mon[32371]: pgmap v454: 158 pgs: 30 active+undersized, 34 active+undersized+degraded, 94 active+clean; 6.2 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 1.2 KiB/s rd, 111 MiB/s wr, 1.78k op/s; 56849/302976 objects degraded (18.764%) 2024-09-18T08:12:16.929 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:08.490781+0000", 2024-09-18T08:12:16.929 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-09-18T08:12:16.929 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:12:16.929 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:12:16.929 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "throttled", 2024-09-18T08:12:16.930 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:08.490783+0000", 2024-09-18T08:12:16.930 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.708e-06 2024-09-18T08:12:16.930 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:12:16.930 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:12:16.930 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "all_read", 2024-09-18T08:12:16.930 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:08.490807+0000", 2024-09-18T08:12:16.931 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 2.4300000000000001e-05 2024-09-18T08:12:16.931 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:12:16.931 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:12:16.931 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "dispatched", 2024-09-18T08:12:16.931 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:08.490819+0000", 2024-09-18T08:12:16.932 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.2468e-05 2024-09-18T08:12:16.932 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:12:16.932 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:12:16.932 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "queued_for_pg", 2024-09-18T08:12:16.932 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:08.490829+0000", 2024-09-18T08:12:16.932 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 9.2320000000000007e-06 2024-09-18T08:12:16.932 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:12:16.932 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:12:16.932 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-09-18T08:12:16.932 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:08.490917+0000", 2024-09-18T08:12:16.932 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 8.8350000000000006e-05 2024-09-18T08:12:16.933 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:12:16.933 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:12:16.933 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "waiting for peered", 2024-09-18T08:12:16.933 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:08.490928+0000", 2024-09-18T08:12:16.933 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.0848e-05 2024-09-18T08:12:16.933 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:12:16.933 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:12:16.933 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-09-18T08:12:16.933 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:09.153316+0000", 2024-09-18T08:12:16.933 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.66238866600000001 2024-09-18T08:12:16.934 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:12:16.934 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:12:16.934 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "waiting for peered", 2024-09-18T08:12:16.934 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:09.153325+0000", 2024-09-18T08:12:16.934 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 8.1089999999999998e-06 2024-09-18T08:12:16.934 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:12:16.934 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:12:16.934 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-09-18T08:12:16.934 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:10.230312+0000", 2024-09-18T08:12:16.934 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.0769875980000001 2024-09-18T08:12:16.934 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:12:16.934 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:12:16.934 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "started", 2024-09-18T08:12:16.935 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:10.230436+0000", 2024-09-18T08:12:16.935 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.000123824 2024-09-18T08:12:16.935 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:12:16.935 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:12:16.935 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "waiting for subops from 1,6", 2024-09-18T08:12:16.935 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:10.230594+0000", 2024-09-18T08:12:16.935 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.00015772099999999999 2024-09-18T08:12:16.935 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:12:16.935 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:12:16.935 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "op_commit", 2024-09-18T08:12:16.935 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:10.234562+0000", 2024-09-18T08:12:16.935 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.0039686460000000002 2024-09-18T08:12:16.935 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:12:16.935 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:12:16.936 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:12:16.936 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:10.234625+0000", 2024-09-18T08:12:16.936 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 6.2675999999999994e-05 2024-09-18T08:12:16.936 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:12:16.936 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:12:16.936 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:12:16.936 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:10.238138+0000", 2024-09-18T08:12:16.936 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.0035124710000000001 2024-09-18T08:12:16.936 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:12:16.936 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:12:16.936 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "commit_sent", 2024-09-18T08:12:16.937 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:10.238230+0000", 2024-09-18T08:12:16.937 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 9.2304000000000003e-05 2024-09-18T08:12:16.937 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:12:16.937 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:12:16.937 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "done", 2024-09-18T08:12:16.937 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:10.239527+0000", 2024-09-18T08:12:16.937 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.0012970290000000001 2024-09-18T08:12:16.937 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-09-18T08:12:16.937 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-09-18T08:12:16.937 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-09-18T08:12:16.938 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:12:16.938 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:12:16.938 INFO:teuthology.orchestra.run.smithi007.stdout: "description": "osd_op(client.15216.0:22685 8.b 8:d2c6b3c8:::benchmark_data_smithi007_101583_object22684:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e175)", 2024-09-18T08:12:16.938 INFO:teuthology.orchestra.run.smithi007.stdout: "initiated_at": "2024-09-18T08:08:08.499266+0000", 2024-09-18T08:12:16.938 INFO:teuthology.orchestra.run.smithi007.stdout: "age": 248.35482122400001, 2024-09-18T08:12:16.938 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.781783984, 2024-09-18T08:12:16.938 INFO:teuthology.orchestra.run.smithi007.stdout: "continuous": false, 2024-09-18T08:12:16.938 INFO:teuthology.orchestra.run.smithi007.stdout: "type_data": { 2024-09-18T08:12:16.938 INFO:teuthology.orchestra.run.smithi007.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T08:12:16.938 INFO:teuthology.orchestra.run.smithi007.stdout: "client_info": { 2024-09-18T08:12:16.938 INFO:teuthology.orchestra.run.smithi007.stdout: "client": "client.15216", 2024-09-18T08:12:16.938 INFO:teuthology.orchestra.run.smithi007.stdout: "client_addr": "172.21.15.7:0/2596041177", 2024-09-18T08:12:16.938 INFO:teuthology.orchestra.run.smithi007.stdout: "tid": 22685 2024-09-18T08:12:16.939 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:12:16.939 INFO:teuthology.orchestra.run.smithi007.stdout: "events": [ 2024-09-18T08:12:16.939 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:12:16.939 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "initiated", 2024-09-18T08:12:16.939 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:08.499266+0000", 2024-09-18T08:12:16.939 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-09-18T08:12:16.939 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:12:16.939 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:12:16.939 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "header_read", 2024-09-18T08:12:16.939 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:08.499266+0000", 2024-09-18T08:12:16.940 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-09-18T08:12:16.940 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:12:16.940 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:12:16.940 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "throttled", 2024-09-18T08:12:16.940 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:08.499268+0000", 2024-09-18T08:12:16.940 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.7060000000000001e-06 2024-09-18T08:12:16.940 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:12:16.940 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:12:16.940 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "all_read", 2024-09-18T08:12:16.940 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:08.499297+0000", 2024-09-18T08:12:16.940 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 2.9298e-05 2024-09-18T08:12:16.941 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:12:16.941 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:12:16.941 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "dispatched", 2024-09-18T08:12:16.941 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:08.499318+0000", 2024-09-18T08:12:16.941 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 2.0758999999999999e-05 2024-09-18T08:12:16.941 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:12:16.941 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:12:16.941 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "queued_for_pg", 2024-09-18T08:12:16.941 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:08.499325+0000", 2024-09-18T08:12:16.941 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 7.8599999999999993e-06 2024-09-18T08:12:16.941 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:12:16.941 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:12:16.941 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-09-18T08:12:16.941 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:08.499379+0000", 2024-09-18T08:12:16.942 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 5.3257999999999997e-05 2024-09-18T08:12:16.942 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:12:16.942 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:12:16.942 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "waiting for peered", 2024-09-18T08:12:16.942 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:08.499391+0000", 2024-09-18T08:12:16.942 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.2483000000000001e-05 2024-09-18T08:12:16.942 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:12:16.942 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:12:16.942 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-09-18T08:12:16.942 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:09.152309+0000", 2024-09-18T08:12:16.942 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.65291789700000002 2024-09-18T08:12:16.942 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:12:16.942 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:12:16.943 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "waiting for peered", 2024-09-18T08:12:16.943 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:09.152317+0000", 2024-09-18T08:12:16.943 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 7.4830000000000002e-06 2024-09-18T08:12:16.943 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:12:16.943 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:12:16.943 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-09-18T08:12:16.943 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:10.260397+0000", 2024-09-18T08:12:16.943 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.108080043 2024-09-18T08:12:16.943 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:12:16.943 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:12:16.943 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "started", 2024-09-18T08:12:16.943 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:10.260553+0000", 2024-09-18T08:12:16.943 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.000156594 2024-09-18T08:12:16.943 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:12:16.944 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:12:16.944 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "waiting for subops from 1,7", 2024-09-18T08:12:16.944 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:10.260761+0000", 2024-09-18T08:12:16.944 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.00020734200000000001 2024-09-18T08:12:16.944 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:12:16.944 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:12:16.944 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:12:16.944 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:10.267815+0000", 2024-09-18T08:12:16.944 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.007053948 2024-09-18T08:12:16.944 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:12:16.944 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:12:16.944 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:12:16.945 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:10.267878+0000", 2024-09-18T08:12:16.945 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 6.3346e-05 2024-09-18T08:12:16.945 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:12:16.945 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:12:16.945 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "op_commit", 2024-09-18T08:12:16.945 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:10.280193+0000", 2024-09-18T08:12:16.945 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.012314782999999999 2024-09-18T08:12:16.945 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:12:16.945 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:12:16.945 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "commit_sent", 2024-09-18T08:12:16.945 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:10.280222+0000", 2024-09-18T08:12:16.945 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 2.9224e-05 2024-09-18T08:12:16.945 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:12:16.945 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:12:16.946 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "done", 2024-09-18T08:12:16.946 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:10.281050+0000", 2024-09-18T08:12:16.946 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.00082795999999999998 2024-09-18T08:12:16.946 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-09-18T08:12:16.946 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-09-18T08:12:16.946 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-09-18T08:12:16.946 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:12:16.946 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:12:16.946 INFO:teuthology.orchestra.run.smithi007.stdout: "description": "osd_op(client.15216.0:22689 8.f 8:f532acf4:::benchmark_data_smithi007_101583_object22688:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e175)", 2024-09-18T08:12:16.946 INFO:teuthology.orchestra.run.smithi007.stdout: "initiated_at": "2024-09-18T08:08:08.501798+0000", 2024-09-18T08:12:16.946 INFO:teuthology.orchestra.run.smithi007.stdout: "age": 248.35228920899999, 2024-09-18T08:12:16.946 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.7388306359999999, 2024-09-18T08:12:16.947 INFO:teuthology.orchestra.run.smithi007.stdout: "continuous": false, 2024-09-18T08:12:16.947 INFO:teuthology.orchestra.run.smithi007.stdout: "type_data": { 2024-09-18T08:12:16.947 INFO:teuthology.orchestra.run.smithi007.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T08:12:16.947 INFO:teuthology.orchestra.run.smithi007.stdout: "client_info": { 2024-09-18T08:12:16.947 INFO:teuthology.orchestra.run.smithi007.stdout: "client": "client.15216", 2024-09-18T08:12:16.947 INFO:teuthology.orchestra.run.smithi007.stdout: "client_addr": "172.21.15.7:0/2596041177", 2024-09-18T08:12:16.947 INFO:teuthology.orchestra.run.smithi007.stdout: "tid": 22689 2024-09-18T08:12:16.947 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:12:16.947 INFO:teuthology.orchestra.run.smithi007.stdout: "events": [ 2024-09-18T08:12:16.947 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:12:16.948 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "initiated", 2024-09-18T08:12:16.948 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:08.501798+0000", 2024-09-18T08:12:16.948 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-09-18T08:12:16.948 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:12:16.948 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:12:16.948 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "header_read", 2024-09-18T08:12:16.948 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:08.501798+0000", 2024-09-18T08:12:16.948 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-09-18T08:12:16.948 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:12:16.948 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:12:16.948 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "throttled", 2024-09-18T08:12:16.949 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:08.501798+0000", 2024-09-18T08:12:16.949 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 5.9400000000000005e-07 2024-09-18T08:12:16.949 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:12:16.949 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:12:16.949 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "all_read", 2024-09-18T08:12:16.949 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:08.501835+0000", 2024-09-18T08:12:16.949 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 3.6974000000000002e-05 2024-09-18T08:12:16.949 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:12:16.949 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:12:16.949 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "dispatched", 2024-09-18T08:12:16.949 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:08.501841+0000", 2024-09-18T08:12:16.950 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 5.2909999999999998e-06 2024-09-18T08:12:16.950 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:12:16.950 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:12:16.950 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "queued_for_pg", 2024-09-18T08:12:16.950 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:08.501844+0000", 2024-09-18T08:12:16.950 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 3.7330000000000001e-06 2024-09-18T08:12:16.950 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:12:16.951 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:12:16.951 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-09-18T08:12:16.951 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:08.501904+0000", 2024-09-18T08:12:16.952 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 5.9861000000000002e-05 2024-09-18T08:12:16.952 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:12:16.952 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:12:16.952 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "waiting for peered", 2024-09-18T08:12:16.952 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:08.501917+0000", 2024-09-18T08:12:16.953 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.2181000000000001e-05 2024-09-18T08:12:16.953 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:12:16.953 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:12:16.953 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-09-18T08:12:16.953 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:09.153352+0000", 2024-09-18T08:12:16.953 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.65143504100000005 2024-09-18T08:12:16.953 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:12:16.953 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:12:16.953 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "waiting for peered", 2024-09-18T08:12:16.953 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:09.153359+0000", 2024-09-18T08:12:16.953 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 7.4760000000000001e-06 2024-09-18T08:12:16.954 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:12:16.954 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:12:16.954 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-09-18T08:12:16.954 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:10.232890+0000", 2024-09-18T08:12:16.954 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.0795311249999999 2024-09-18T08:12:16.954 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:12:16.954 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:12:16.954 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "started", 2024-09-18T08:12:16.954 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:10.233035+0000", 2024-09-18T08:12:16.954 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.000145282 2024-09-18T08:12:16.954 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:12:16.954 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:12:16.954 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "waiting for subops from 1,6", 2024-09-18T08:12:16.955 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:10.233242+0000", 2024-09-18T08:12:16.955 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.000206181 2024-09-18T08:12:16.955 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:12:16.955 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:12:16.955 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:12:16.955 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:10.238074+0000", 2024-09-18T08:12:16.955 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.0048323649999999999 2024-09-18T08:12:16.955 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:12:16.955 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:12:16.955 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "op_commit", 2024-09-18T08:12:16.955 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:10.238090+0000", 2024-09-18T08:12:16.955 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.5934e-05 2024-09-18T08:12:16.955 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:12:16.956 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:12:16.956 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:12:16.956 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:10.239669+0000", 2024-09-18T08:12:16.956 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.0015793039999999999 2024-09-18T08:12:16.956 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:12:16.956 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:12:16.956 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "commit_sent", 2024-09-18T08:12:16.956 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:10.239733+0000", 2024-09-18T08:12:16.956 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 6.3404999999999993e-05 2024-09-18T08:12:16.956 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:12:16.956 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:12:16.956 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "done", 2024-09-18T08:12:16.956 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:10.240629+0000", 2024-09-18T08:12:16.957 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.00089588899999999997 2024-09-18T08:12:16.957 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-09-18T08:12:16.957 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-09-18T08:12:16.957 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-09-18T08:12:16.957 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:12:16.957 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:12:16.957 INFO:teuthology.orchestra.run.smithi007.stdout: "description": "osd_op(client.15216.0:69187 8.f 8:f51333c2:::benchmark_data_smithi007_101583_object69186:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e198)", 2024-09-18T08:12:16.957 INFO:teuthology.orchestra.run.smithi007.stdout: "initiated_at": "2024-09-18T08:08:51.699105+0000", 2024-09-18T08:12:16.957 INFO:teuthology.orchestra.run.smithi007.stdout: "age": 205.15498171300001, 2024-09-18T08:12:16.957 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 2.1312144279999998, 2024-09-18T08:12:16.957 INFO:teuthology.orchestra.run.smithi007.stdout: "continuous": false, 2024-09-18T08:12:16.957 INFO:teuthology.orchestra.run.smithi007.stdout: "type_data": { 2024-09-18T08:12:16.957 INFO:teuthology.orchestra.run.smithi007.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T08:12:16.957 INFO:teuthology.orchestra.run.smithi007.stdout: "client_info": { 2024-09-18T08:12:16.958 INFO:teuthology.orchestra.run.smithi007.stdout: "client": "client.15216", 2024-09-18T08:12:16.958 INFO:teuthology.orchestra.run.smithi007.stdout: "client_addr": "172.21.15.7:0/2596041177", 2024-09-18T08:12:16.958 INFO:teuthology.orchestra.run.smithi007.stdout: "tid": 69187 2024-09-18T08:12:16.958 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:12:16.958 INFO:teuthology.orchestra.run.smithi007.stdout: "events": [ 2024-09-18T08:12:16.958 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:12:16.958 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "initiated", 2024-09-18T08:12:16.958 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:51.699105+0000", 2024-09-18T08:12:16.958 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-09-18T08:12:16.958 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:12:16.958 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:12:16.958 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "header_read", 2024-09-18T08:12:16.958 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:51.699105+0000", 2024-09-18T08:12:16.959 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-09-18T08:12:16.959 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:12:16.959 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:12:16.959 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "throttled", 2024-09-18T08:12:16.959 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:51.699107+0000", 2024-09-18T08:12:16.959 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.5149999999999999e-06 2024-09-18T08:12:16.959 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:12:16.959 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:12:16.959 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "all_read", 2024-09-18T08:12:16.959 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:51.699133+0000", 2024-09-18T08:12:16.959 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 2.5752999999999999e-05 2024-09-18T08:12:16.959 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:12:16.959 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:12:16.959 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "dispatched", 2024-09-18T08:12:16.960 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:51.699141+0000", 2024-09-18T08:12:16.960 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 8.2709999999999992e-06 2024-09-18T08:12:16.960 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:12:16.960 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:12:16.960 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "queued_for_pg", 2024-09-18T08:12:16.960 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:51.699163+0000", 2024-09-18T08:12:16.960 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 2.2432000000000001e-05 2024-09-18T08:12:16.960 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:12:16.960 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:12:16.960 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-09-18T08:12:16.960 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:51.699402+0000", 2024-09-18T08:12:16.960 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.000239046 2024-09-18T08:12:16.960 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:12:16.960 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:12:16.961 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "waiting for peered", 2024-09-18T08:12:16.961 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:51.699413+0000", 2024-09-18T08:12:16.961 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.0387000000000001e-05 2024-09-18T08:12:16.961 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:12:16.961 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:12:16.961 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-09-18T08:12:16.961 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:52.696975+0000", 2024-09-18T08:12:16.961 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.99756218100000005 2024-09-18T08:12:16.961 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:12:16.961 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:12:16.961 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "waiting for peered", 2024-09-18T08:12:16.961 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:52.697002+0000", 2024-09-18T08:12:16.961 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 2.7464e-05 2024-09-18T08:12:16.962 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:12:16.962 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:12:16.962 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-09-18T08:12:16.962 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:53.824414+0000", 2024-09-18T08:12:16.962 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.1274112789999999 2024-09-18T08:12:16.962 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:12:16.962 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:12:16.962 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "started", 2024-09-18T08:12:16.962 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:53.824573+0000", 2024-09-18T08:12:16.962 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.00015908400000000001 2024-09-18T08:12:16.962 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:12:16.962 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:12:16.962 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "waiting for subops from 1,2,4", 2024-09-18T08:12:16.962 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:53.824756+0000", 2024-09-18T08:12:16.963 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.00018345199999999999 2024-09-18T08:12:16.963 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:12:16.963 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:12:16.963 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:12:16.963 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:53.827276+0000", 2024-09-18T08:12:16.963 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.002519444 2024-09-18T08:12:16.963 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:12:16.963 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:12:16.963 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:12:16.963 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:53.827356+0000", 2024-09-18T08:12:16.963 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 8.0029000000000002e-05 2024-09-18T08:12:16.963 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:12:16.963 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:12:16.963 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "op_commit", 2024-09-18T08:12:16.964 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:53.827376+0000", 2024-09-18T08:12:16.964 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 2.0250000000000001e-05 2024-09-18T08:12:16.964 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:12:16.964 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:12:16.964 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:12:16.964 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:53.830181+0000", 2024-09-18T08:12:16.964 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.0028047469999999998 2024-09-18T08:12:16.964 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:12:16.964 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:12:16.964 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "commit_sent", 2024-09-18T08:12:16.964 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:53.830246+0000", 2024-09-18T08:12:16.964 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 6.5140000000000003e-05 2024-09-18T08:12:16.964 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:12:16.964 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:12:16.965 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "done", 2024-09-18T08:12:16.965 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:53.830320+0000", 2024-09-18T08:12:16.965 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 7.3954000000000004e-05 2024-09-18T08:12:16.965 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-09-18T08:12:16.965 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-09-18T08:12:16.965 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-09-18T08:12:16.965 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:12:16.965 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:12:16.965 INFO:teuthology.orchestra.run.smithi007.stdout: "description": "osd_op(client.15216.0:69195 8.f 8:fafe41e6:::benchmark_data_smithi007_101583_object69194:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e198)", 2024-09-18T08:12:16.965 INFO:teuthology.orchestra.run.smithi007.stdout: "initiated_at": "2024-09-18T08:08:51.711966+0000", 2024-09-18T08:12:16.965 INFO:teuthology.orchestra.run.smithi007.stdout: "age": 205.14212095100001, 2024-09-18T08:12:16.965 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 2.1187014670000002, 2024-09-18T08:12:16.965 INFO:teuthology.orchestra.run.smithi007.stdout: "continuous": false, 2024-09-18T08:12:16.966 INFO:teuthology.orchestra.run.smithi007.stdout: "type_data": { 2024-09-18T08:12:16.966 INFO:teuthology.orchestra.run.smithi007.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T08:12:16.966 INFO:teuthology.orchestra.run.smithi007.stdout: "client_info": { 2024-09-18T08:12:16.966 INFO:teuthology.orchestra.run.smithi007.stdout: "client": "client.15216", 2024-09-18T08:12:16.966 INFO:teuthology.orchestra.run.smithi007.stdout: "client_addr": "172.21.15.7:0/2596041177", 2024-09-18T08:12:16.966 INFO:teuthology.orchestra.run.smithi007.stdout: "tid": 69195 2024-09-18T08:12:16.966 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:12:16.966 INFO:teuthology.orchestra.run.smithi007.stdout: "events": [ 2024-09-18T08:12:16.966 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:12:16.966 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "initiated", 2024-09-18T08:12:16.966 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:51.711966+0000", 2024-09-18T08:12:16.966 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-09-18T08:12:16.966 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:12:16.966 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:12:16.967 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "header_read", 2024-09-18T08:12:16.967 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:51.711966+0000", 2024-09-18T08:12:16.967 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-09-18T08:12:16.967 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:12:16.967 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:12:16.967 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "throttled", 2024-09-18T08:12:16.967 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:51.711968+0000", 2024-09-18T08:12:16.967 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 2.013e-06 2024-09-18T08:12:16.967 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:12:16.967 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:12:16.967 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "all_read", 2024-09-18T08:12:16.967 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:51.711997+0000", 2024-09-18T08:12:16.967 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 2.8561999999999999e-05 2024-09-18T08:12:16.968 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:12:16.968 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:12:16.968 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "dispatched", 2024-09-18T08:12:16.968 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:51.712012+0000", 2024-09-18T08:12:16.968 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.5184000000000001e-05 2024-09-18T08:12:16.968 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:12:16.968 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:12:16.968 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "queued_for_pg", 2024-09-18T08:12:16.968 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:51.712022+0000", 2024-09-18T08:12:16.968 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 9.6530000000000006e-06 2024-09-18T08:12:16.968 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:12:16.968 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:12:16.968 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-09-18T08:12:16.968 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:51.712088+0000", 2024-09-18T08:12:16.969 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 6.6921000000000005e-05 2024-09-18T08:12:16.969 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:12:16.969 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:12:16.969 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "waiting for peered", 2024-09-18T08:12:16.969 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:51.712101+0000", 2024-09-18T08:12:16.969 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.2768e-05 2024-09-18T08:12:16.969 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:12:16.969 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:12:16.969 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-09-18T08:12:16.969 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:52.697039+0000", 2024-09-18T08:12:16.969 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.98493815299999998 2024-09-18T08:12:16.969 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:12:16.969 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:12:16.969 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "waiting for peered", 2024-09-18T08:12:16.969 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:52.697052+0000", 2024-09-18T08:12:16.970 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.2475e-05 2024-09-18T08:12:16.970 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:12:16.970 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:12:16.970 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-09-18T08:12:16.970 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:53.827400+0000", 2024-09-18T08:12:16.970 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.1303482419999999 2024-09-18T08:12:16.970 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:12:16.970 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:12:16.970 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "started", 2024-09-18T08:12:16.970 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:53.827528+0000", 2024-09-18T08:12:16.970 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.000128009 2024-09-18T08:12:16.970 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:12:16.970 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:12:16.970 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "waiting for subops from 1,2,4", 2024-09-18T08:12:16.971 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:53.827701+0000", 2024-09-18T08:12:16.971 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.00017307100000000001 2024-09-18T08:12:16.971 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:12:16.971 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:12:16.971 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:12:16.971 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:53.830376+0000", 2024-09-18T08:12:16.971 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.002675144 2024-09-18T08:12:16.971 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:12:16.971 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:12:16.971 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:12:16.971 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:53.830419+0000", 2024-09-18T08:12:16.971 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 4.3087000000000003e-05 2024-09-18T08:12:16.971 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:12:16.972 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:12:16.972 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "op_commit", 2024-09-18T08:12:16.972 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:53.830444+0000", 2024-09-18T08:12:16.972 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 2.4579e-05 2024-09-18T08:12:16.972 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:12:16.972 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:12:16.972 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:12:16.972 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:53.830485+0000", 2024-09-18T08:12:16.972 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 4.1236999999999998e-05 2024-09-18T08:12:16.972 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:12:16.972 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:12:16.972 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "commit_sent", 2024-09-18T08:12:16.972 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:53.830541+0000", 2024-09-18T08:12:16.972 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 5.6075000000000001e-05 2024-09-18T08:12:16.973 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:12:16.973 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:12:16.973 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "done", 2024-09-18T08:12:16.973 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:53.830668+0000", 2024-09-18T08:12:16.973 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.00012629399999999999 2024-09-18T08:12:16.973 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-09-18T08:12:16.973 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-09-18T08:12:16.973 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-09-18T08:12:16.973 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:12:16.973 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:12:16.973 INFO:teuthology.orchestra.run.smithi007.stdout: "description": "osd_op(client.15216.0:69207 8.f 8:f63411d3:::benchmark_data_smithi007_101583_object69206:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e198)", 2024-09-18T08:12:16.973 INFO:teuthology.orchestra.run.smithi007.stdout: "initiated_at": "2024-09-18T08:08:51.719518+0000", 2024-09-18T08:12:16.973 INFO:teuthology.orchestra.run.smithi007.stdout: "age": 205.134568591, 2024-09-18T08:12:16.974 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 2.1171781319999998, 2024-09-18T08:12:16.974 INFO:teuthology.orchestra.run.smithi007.stdout: "continuous": false, 2024-09-18T08:12:16.974 INFO:teuthology.orchestra.run.smithi007.stdout: "type_data": { 2024-09-18T08:12:16.974 INFO:teuthology.orchestra.run.smithi007.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T08:12:16.974 INFO:teuthology.orchestra.run.smithi007.stdout: "client_info": { 2024-09-18T08:12:16.974 INFO:teuthology.orchestra.run.smithi007.stdout: "client": "client.15216", 2024-09-18T08:12:16.974 INFO:teuthology.orchestra.run.smithi007.stdout: "client_addr": "172.21.15.7:0/2596041177", 2024-09-18T08:12:16.974 INFO:teuthology.orchestra.run.smithi007.stdout: "tid": 69207 2024-09-18T08:12:16.974 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:12:16.974 INFO:teuthology.orchestra.run.smithi007.stdout: "events": [ 2024-09-18T08:12:16.974 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:12:16.974 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "initiated", 2024-09-18T08:12:16.974 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:51.719518+0000", 2024-09-18T08:12:16.974 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-09-18T08:12:16.975 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:12:16.975 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:12:16.975 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "header_read", 2024-09-18T08:12:16.975 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:51.719518+0000", 2024-09-18T08:12:16.975 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-09-18T08:12:16.975 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:12:16.975 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:12:16.975 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "throttled", 2024-09-18T08:12:16.975 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:51.719519+0000", 2024-09-18T08:12:16.975 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 5.7999999999999995e-07 2024-09-18T08:12:16.975 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:12:16.975 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:12:16.976 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "all_read", 2024-09-18T08:12:16.976 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:51.719540+0000", 2024-09-18T08:12:16.976 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 2.1097000000000001e-05 2024-09-18T08:12:16.976 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:12:16.976 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:12:16.976 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "dispatched", 2024-09-18T08:12:16.976 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:51.719547+0000", 2024-09-18T08:12:16.976 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 6.4810000000000003e-06 2024-09-18T08:12:16.976 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:12:16.976 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:12:16.976 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "queued_for_pg", 2024-09-18T08:12:16.976 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:51.719552+0000", 2024-09-18T08:12:16.976 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 5.2100000000000001e-06 2024-09-18T08:12:16.977 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:12:16.977 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:12:16.977 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-09-18T08:12:16.977 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:51.719676+0000", 2024-09-18T08:12:16.977 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.00012415899999999999 2024-09-18T08:12:16.977 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:12:16.977 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:12:16.977 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "waiting for peered", 2024-09-18T08:12:16.977 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:51.719686+0000", 2024-09-18T08:12:16.977 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 9.5000000000000005e-06 2024-09-18T08:12:16.977 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:12:16.978 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:12:16.978 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-09-18T08:12:16.978 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:52.697757+0000", 2024-09-18T08:12:16.978 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.97807110100000005 2024-09-18T08:12:16.978 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:12:16.978 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:12:16.978 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "waiting for peered", 2024-09-18T08:12:16.978 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:52.697767+0000", 2024-09-18T08:12:16.978 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 9.8810000000000008e-06 2024-09-18T08:12:16.978 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:12:16.978 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:12:16.978 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-09-18T08:12:16.979 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:53.830718+0000", 2024-09-18T08:12:16.979 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.1329519370000001 2024-09-18T08:12:16.979 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:12:16.979 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:12:16.979 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "started", 2024-09-18T08:12:16.979 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:53.830870+0000", 2024-09-18T08:12:16.979 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.00015120300000000001 2024-09-18T08:12:16.979 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:12:16.979 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:12:16.979 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "waiting for subops from 1,2,4", 2024-09-18T08:12:16.979 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:53.831043+0000", 2024-09-18T08:12:16.979 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.00017385499999999999 2024-09-18T08:12:16.979 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:12:16.980 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:12:16.980 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "op_commit", 2024-09-18T08:12:16.980 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:53.836372+0000", 2024-09-18T08:12:16.980 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.0053287689999999997 2024-09-18T08:12:16.980 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:12:16.980 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:12:16.980 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:12:16.980 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:53.836420+0000", 2024-09-18T08:12:16.980 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 4.7490000000000001e-05 2024-09-18T08:12:16.980 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:12:16.980 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:12:16.980 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:12:16.980 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:53.836464+0000", 2024-09-18T08:12:16.981 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 4.4254e-05 2024-09-18T08:12:16.981 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:12:16.981 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:12:16.981 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:12:16.981 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:53.836505+0000", 2024-09-18T08:12:16.981 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 4.1414e-05 2024-09-18T08:12:16.981 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:12:16.981 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:12:16.981 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "commit_sent", 2024-09-18T08:12:16.981 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:53.836566+0000", 2024-09-18T08:12:16.981 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 6.0538999999999998e-05 2024-09-18T08:12:16.981 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:12:16.981 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:12:16.981 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "done", 2024-09-18T08:12:16.982 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:53.836697+0000", 2024-09-18T08:12:16.982 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.000130662 2024-09-18T08:12:16.982 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-09-18T08:12:16.982 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-09-18T08:12:16.982 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-09-18T08:12:16.982 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:12:16.982 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:12:16.982 INFO:teuthology.orchestra.run.smithi007.stdout: "description": "osd_op(client.15216.0:69218 8.f 8:fd5c3396:::benchmark_data_smithi007_101583_object69217:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e198)", 2024-09-18T08:12:16.982 INFO:teuthology.orchestra.run.smithi007.stdout: "initiated_at": "2024-09-18T08:08:51.731702+0000", 2024-09-18T08:12:16.982 INFO:teuthology.orchestra.run.smithi007.stdout: "age": 205.12238533499999, 2024-09-18T08:12:16.982 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 2.1112667140000001, 2024-09-18T08:12:16.982 INFO:teuthology.orchestra.run.smithi007.stdout: "continuous": false, 2024-09-18T08:12:16.982 INFO:teuthology.orchestra.run.smithi007.stdout: "type_data": { 2024-09-18T08:12:16.982 INFO:teuthology.orchestra.run.smithi007.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T08:12:16.982 INFO:teuthology.orchestra.run.smithi007.stdout: "client_info": { 2024-09-18T08:12:16.983 INFO:teuthology.orchestra.run.smithi007.stdout: "client": "client.15216", 2024-09-18T08:12:16.983 INFO:teuthology.orchestra.run.smithi007.stdout: "client_addr": "172.21.15.7:0/2596041177", 2024-09-18T08:12:16.983 INFO:teuthology.orchestra.run.smithi007.stdout: "tid": 69218 2024-09-18T08:12:16.983 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:12:16.983 INFO:teuthology.orchestra.run.smithi007.stdout: "events": [ 2024-09-18T08:12:16.983 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:12:16.983 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "initiated", 2024-09-18T08:12:16.983 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:51.731702+0000", 2024-09-18T08:12:16.983 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-09-18T08:12:16.983 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:12:16.983 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:12:16.983 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "header_read", 2024-09-18T08:12:16.983 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:51.731702+0000", 2024-09-18T08:12:16.984 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-09-18T08:12:16.984 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:12:16.984 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:12:16.984 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "throttled", 2024-09-18T08:12:16.984 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:51.731703+0000", 2024-09-18T08:12:16.984 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.265e-06 2024-09-18T08:12:16.984 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:12:16.984 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:12:16.984 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "all_read", 2024-09-18T08:12:16.984 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:51.731771+0000", 2024-09-18T08:12:16.984 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 6.7945000000000001e-05 2024-09-18T08:12:16.984 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:12:16.984 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:12:16.984 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "dispatched", 2024-09-18T08:12:16.985 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:51.731781+0000", 2024-09-18T08:12:16.985 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 9.8030000000000008e-06 2024-09-18T08:12:16.985 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:12:16.985 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:12:16.985 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "queued_for_pg", 2024-09-18T08:12:16.985 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:51.731788+0000", 2024-09-18T08:12:16.985 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 7.1620000000000002e-06 2024-09-18T08:12:16.985 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:12:16.985 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:12:16.985 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-09-18T08:12:16.985 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:51.731861+0000", 2024-09-18T08:12:16.985 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 7.2669999999999994e-05 2024-09-18T08:12:16.985 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:12:16.985 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:12:16.985 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "waiting for peered", 2024-09-18T08:12:16.986 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:51.731875+0000", 2024-09-18T08:12:16.986 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.4209999999999999e-05 2024-09-18T08:12:16.986 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:12:16.986 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:12:16.986 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-09-18T08:12:16.986 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:52.697807+0000", 2024-09-18T08:12:16.986 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.96593184200000004 2024-09-18T08:12:16.986 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:12:16.986 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:12:16.986 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "waiting for peered", 2024-09-18T08:12:16.986 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:52.697815+0000", 2024-09-18T08:12:16.986 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 8.4239999999999993e-06 2024-09-18T08:12:16.986 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:12:16.986 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:12:16.987 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-09-18T08:12:16.987 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:53.833616+0000", 2024-09-18T08:12:16.987 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.1358008079999999 2024-09-18T08:12:16.987 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:12:16.987 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:12:16.987 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "started", 2024-09-18T08:12:16.987 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:53.833763+0000", 2024-09-18T08:12:16.987 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.000146663 2024-09-18T08:12:16.987 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:12:16.987 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:12:16.987 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "waiting for subops from 1,2,4", 2024-09-18T08:12:16.987 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:53.833911+0000", 2024-09-18T08:12:16.987 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.00014877900000000001 2024-09-18T08:12:16.987 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:12:16.988 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:12:16.988 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:12:16.988 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:53.836740+0000", 2024-09-18T08:12:16.988 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.002828882 2024-09-18T08:12:16.988 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:12:16.988 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:12:16.988 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "op_commit", 2024-09-18T08:12:16.988 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:53.836756+0000", 2024-09-18T08:12:16.988 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.6195999999999999e-05 2024-09-18T08:12:16.988 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:12:16.988 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:12:16.988 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:12:16.988 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:53.842695+0000", 2024-09-18T08:12:16.988 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.0059383209999999999 2024-09-18T08:12:16.989 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:12:16.989 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:12:16.989 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:12:16.989 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:53.842746+0000", 2024-09-18T08:12:16.989 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 5.0966e-05 2024-09-18T08:12:16.989 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:12:16.989 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:12:16.989 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "commit_sent", 2024-09-18T08:12:16.989 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:53.842817+0000", 2024-09-18T08:12:16.989 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 7.1437000000000001e-05 2024-09-18T08:12:16.990 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:12:16.990 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:12:16.990 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "done", 2024-09-18T08:12:16.990 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:53.842968+0000", 2024-09-18T08:12:16.990 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.00015134100000000001 2024-09-18T08:12:16.990 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-09-18T08:12:16.990 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-09-18T08:12:16.990 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-09-18T08:12:16.990 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:12:16.990 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:12:16.990 INFO:teuthology.orchestra.run.smithi007.stdout: "description": "osd_op(client.15216.0:69239 8.f 8:f2d8399e:::benchmark_data_smithi007_101583_object69238:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e198)", 2024-09-18T08:12:16.990 INFO:teuthology.orchestra.run.smithi007.stdout: "initiated_at": "2024-09-18T08:08:51.752739+0000", 2024-09-18T08:12:16.990 INFO:teuthology.orchestra.run.smithi007.stdout: "age": 205.101347804, 2024-09-18T08:12:16.991 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 2.0962772900000002, 2024-09-18T08:12:16.991 INFO:teuthology.orchestra.run.smithi007.stdout: "continuous": false, 2024-09-18T08:12:16.991 INFO:teuthology.orchestra.run.smithi007.stdout: "type_data": { 2024-09-18T08:12:16.991 INFO:teuthology.orchestra.run.smithi007.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T08:12:16.991 INFO:teuthology.orchestra.run.smithi007.stdout: "client_info": { 2024-09-18T08:12:16.991 INFO:teuthology.orchestra.run.smithi007.stdout: "client": "client.15216", 2024-09-18T08:12:16.991 INFO:teuthology.orchestra.run.smithi007.stdout: "client_addr": "172.21.15.7:0/2596041177", 2024-09-18T08:12:16.991 INFO:teuthology.orchestra.run.smithi007.stdout: "tid": 69239 2024-09-18T08:12:16.991 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:12:16.991 INFO:teuthology.orchestra.run.smithi007.stdout: "events": [ 2024-09-18T08:12:16.991 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:12:16.991 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "initiated", 2024-09-18T08:12:16.991 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:51.752739+0000", 2024-09-18T08:12:16.991 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-09-18T08:12:16.992 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:12:16.992 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:12:16.992 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "header_read", 2024-09-18T08:12:16.992 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:51.752739+0000", 2024-09-18T08:12:16.992 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-09-18T08:12:16.992 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:12:16.992 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:12:16.992 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "throttled", 2024-09-18T08:12:16.992 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:51.752741+0000", 2024-09-18T08:12:16.992 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.829e-06 2024-09-18T08:12:16.992 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:12:16.992 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:12:16.992 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "all_read", 2024-09-18T08:12:16.992 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:51.752766+0000", 2024-09-18T08:12:16.993 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 2.4842e-05 2024-09-18T08:12:16.993 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:12:16.993 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:12:16.993 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "dispatched", 2024-09-18T08:12:16.993 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:51.752777+0000", 2024-09-18T08:12:16.993 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.1395e-05 2024-09-18T08:12:16.993 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:12:16.993 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:12:16.993 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "queued_for_pg", 2024-09-18T08:12:16.993 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:51.752785+0000", 2024-09-18T08:12:16.993 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 7.401e-06 2024-09-18T08:12:16.993 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:12:16.994 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:12:16.994 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-09-18T08:12:16.994 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:51.753101+0000", 2024-09-18T08:12:16.994 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.00031663600000000002 2024-09-18T08:12:16.994 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:12:16.994 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:12:16.994 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "waiting for peered", 2024-09-18T08:12:16.994 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:51.753112+0000", 2024-09-18T08:12:16.994 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.0825000000000001e-05 2024-09-18T08:12:16.994 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:12:16.994 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:12:16.994 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-09-18T08:12:16.994 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:52.697883+0000", 2024-09-18T08:12:16.995 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.94477077700000001 2024-09-18T08:12:16.995 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:12:16.995 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:12:16.995 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "waiting for peered", 2024-09-18T08:12:16.995 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:52.697893+0000", 2024-09-18T08:12:16.995 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 9.5820000000000008e-06 2024-09-18T08:12:16.995 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:12:16.995 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:12:16.995 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-09-18T08:12:16.995 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:53.836781+0000", 2024-09-18T08:12:16.995 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.1388879329999999 2024-09-18T08:12:16.995 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:12:16.995 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:12:16.996 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "started", 2024-09-18T08:12:16.996 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:53.836912+0000", 2024-09-18T08:12:16.996 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.00013171799999999999 2024-09-18T08:12:16.996 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:12:16.996 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:12:16.996 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "waiting for subops from 1,2,4", 2024-09-18T08:12:16.996 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:53.837063+0000", 2024-09-18T08:12:16.996 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.00015040799999999999 2024-09-18T08:12:16.996 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:12:16.996 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:12:16.996 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:12:16.996 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:53.843015+0000", 2024-09-18T08:12:16.996 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.0059519250000000003 2024-09-18T08:12:16.997 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:12:16.997 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:12:16.997 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:12:16.997 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:53.843063+0000", 2024-09-18T08:12:16.997 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 4.8199999999999999e-05 2024-09-18T08:12:16.997 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:12:16.997 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:12:16.997 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "op_commit", 2024-09-18T08:12:16.997 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:53.848550+0000", 2024-09-18T08:12:16.997 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.0054868929999999996 2024-09-18T08:12:16.997 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:12:16.997 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:12:16.997 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:12:16.998 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:53.848663+0000", 2024-09-18T08:12:16.998 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.000113195 2024-09-18T08:12:16.998 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:12:16.998 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:12:16.998 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "commit_sent", 2024-09-18T08:12:16.998 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:53.848860+0000", 2024-09-18T08:12:16.998 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.00019676000000000001 2024-09-18T08:12:16.998 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:12:16.998 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:12:16.998 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "done", 2024-09-18T08:12:16.998 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:53.849017+0000", 2024-09-18T08:12:16.999 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.000156971 2024-09-18T08:12:16.999 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-09-18T08:12:16.999 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-09-18T08:12:16.999 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-09-18T08:12:16.999 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:12:16.999 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:12:16.999 INFO:teuthology.orchestra.run.smithi007.stdout: "description": "osd_op(client.15216.0:69262 8.f 8:f2d117f8:::benchmark_data_smithi007_101583_object69261:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e198)", 2024-09-18T08:12:16.999 INFO:teuthology.orchestra.run.smithi007.stdout: "initiated_at": "2024-09-18T08:08:51.776735+0000", 2024-09-18T08:12:16.999 INFO:teuthology.orchestra.run.smithi007.stdout: "age": 205.07735216399999, 2024-09-18T08:12:16.999 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 2.0727967970000001, 2024-09-18T08:12:16.999 INFO:teuthology.orchestra.run.smithi007.stdout: "continuous": false, 2024-09-18T08:12:16.999 INFO:teuthology.orchestra.run.smithi007.stdout: "type_data": { 2024-09-18T08:12:16.999 INFO:teuthology.orchestra.run.smithi007.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T08:12:16.999 INFO:teuthology.orchestra.run.smithi007.stdout: "client_info": { 2024-09-18T08:12:17.000 INFO:teuthology.orchestra.run.smithi007.stdout: "client": "client.15216", 2024-09-18T08:12:17.000 INFO:teuthology.orchestra.run.smithi007.stdout: "client_addr": "172.21.15.7:0/2596041177", 2024-09-18T08:12:17.000 INFO:teuthology.orchestra.run.smithi007.stdout: "tid": 69262 2024-09-18T08:12:17.000 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:12:17.000 INFO:teuthology.orchestra.run.smithi007.stdout: "events": [ 2024-09-18T08:12:17.000 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:12:17.000 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "initiated", 2024-09-18T08:12:17.000 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:51.776735+0000", 2024-09-18T08:12:17.000 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-09-18T08:12:17.000 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:12:17.000 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:12:17.000 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "header_read", 2024-09-18T08:12:17.001 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:51.776735+0000", 2024-09-18T08:12:17.001 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-09-18T08:12:17.001 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:12:17.001 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:12:17.001 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "throttled", 2024-09-18T08:12:17.001 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:51.776737+0000", 2024-09-18T08:12:17.001 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 2.0990000000000001e-06 2024-09-18T08:12:17.001 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:12:17.001 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:12:17.001 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "all_read", 2024-09-18T08:12:17.001 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:51.776765+0000", 2024-09-18T08:12:17.001 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 2.7562000000000001e-05 2024-09-18T08:12:17.001 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:12:17.001 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:12:17.002 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "dispatched", 2024-09-18T08:12:17.002 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:51.776785+0000", 2024-09-18T08:12:17.002 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 2.0136000000000002e-05 2024-09-18T08:12:17.002 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:12:17.002 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:12:17.002 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "queued_for_pg", 2024-09-18T08:12:17.002 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:51.776794+0000", 2024-09-18T08:12:17.002 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 9.3109999999999995e-06 2024-09-18T08:12:17.002 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:12:17.002 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:12:17.002 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-09-18T08:12:17.002 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:51.776887+0000", 2024-09-18T08:12:17.002 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 9.3382999999999997e-05 2024-09-18T08:12:17.002 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:12:17.003 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:12:17.003 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "waiting for peered", 2024-09-18T08:12:17.003 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:51.776898+0000", 2024-09-18T08:12:17.003 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.0278999999999999e-05 2024-09-18T08:12:17.003 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:12:17.003 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:12:17.003 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-09-18T08:12:17.003 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:52.697921+0000", 2024-09-18T08:12:17.003 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.92102297700000002 2024-09-18T08:12:17.003 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:12:17.003 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:12:17.003 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "waiting for peered", 2024-09-18T08:12:17.003 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:52.697929+0000", 2024-09-18T08:12:17.003 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 8.0709999999999995e-06 2024-09-18T08:12:17.004 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:12:17.004 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:12:17.004 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-09-18T08:12:17.004 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:53.839602+0000", 2024-09-18T08:12:17.004 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.1416730690000001 2024-09-18T08:12:17.004 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:12:17.004 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:12:17.004 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "started", 2024-09-18T08:12:17.004 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:53.839748+0000", 2024-09-18T08:12:17.004 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.00014631299999999999 2024-09-18T08:12:17.004 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:12:17.004 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:12:17.004 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "waiting for subops from 1,2,4", 2024-09-18T08:12:17.004 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:53.839927+0000", 2024-09-18T08:12:17.004 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.00017868800000000001 2024-09-18T08:12:17.005 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:12:17.006 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:12:17.006 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:12:17.006 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:53.843113+0000", 2024-09-18T08:12:17.006 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.003186595 2024-09-18T08:12:17.006 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:12:17.006 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:12:17.006 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:12:17.007 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:53.848599+0000", 2024-09-18T08:12:17.007 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.0054851049999999997 2024-09-18T08:12:17.007 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:12:17.007 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:12:17.007 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "op_commit", 2024-09-18T08:12:17.007 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:53.848616+0000", 2024-09-18T08:12:17.007 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.785e-05 2024-09-18T08:12:17.007 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:12:17.007 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:12:17.007 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:12:17.007 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:53.849125+0000", 2024-09-18T08:12:17.007 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.000508579 2024-09-18T08:12:17.008 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:12:17.008 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:12:17.008 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "commit_sent", 2024-09-18T08:12:17.008 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:53.849311+0000", 2024-09-18T08:12:17.008 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.00018621299999999999 2024-09-18T08:12:17.008 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:12:17.008 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:12:17.008 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "done", 2024-09-18T08:12:17.008 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:53.849532+0000", 2024-09-18T08:12:17.008 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.00022056699999999999 2024-09-18T08:12:17.008 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-09-18T08:12:17.008 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-09-18T08:12:17.008 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-09-18T08:12:17.009 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:12:17.009 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:12:17.009 INFO:teuthology.orchestra.run.smithi007.stdout: "description": "osd_op(client.15216.0:69340 8.f 8:ffd37843:::benchmark_data_smithi007_101583_object69339:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e198)", 2024-09-18T08:12:17.009 INFO:teuthology.orchestra.run.smithi007.stdout: "initiated_at": "2024-09-18T08:08:51.891119+0000", 2024-09-18T08:12:17.009 INFO:teuthology.orchestra.run.smithi007.stdout: "age": 204.96296843799999, 2024-09-18T08:12:17.009 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.9666553580000001, 2024-09-18T08:12:17.009 INFO:teuthology.orchestra.run.smithi007.stdout: "continuous": false, 2024-09-18T08:12:17.009 INFO:teuthology.orchestra.run.smithi007.stdout: "type_data": { 2024-09-18T08:12:17.010 INFO:teuthology.orchestra.run.smithi007.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T08:12:17.010 INFO:teuthology.orchestra.run.smithi007.stdout: "client_info": { 2024-09-18T08:12:17.010 INFO:teuthology.orchestra.run.smithi007.stdout: "client": "client.15216", 2024-09-18T08:12:17.010 INFO:teuthology.orchestra.run.smithi007.stdout: "client_addr": "172.21.15.7:0/2596041177", 2024-09-18T08:12:17.010 INFO:teuthology.orchestra.run.smithi007.stdout: "tid": 69340 2024-09-18T08:12:17.010 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:12:17.010 INFO:teuthology.orchestra.run.smithi007.stdout: "events": [ 2024-09-18T08:12:17.010 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:12:17.010 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "initiated", 2024-09-18T08:12:17.010 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:51.891119+0000", 2024-09-18T08:12:17.010 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-09-18T08:12:17.010 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:12:17.011 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:12:17.011 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "header_read", 2024-09-18T08:12:17.011 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:51.891119+0000", 2024-09-18T08:12:17.011 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-09-18T08:12:17.011 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:12:17.011 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:12:17.011 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "throttled", 2024-09-18T08:12:17.011 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:51.891120+0000", 2024-09-18T08:12:17.011 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.6649999999999999e-06 2024-09-18T08:12:17.011 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:12:17.011 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:12:17.011 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "all_read", 2024-09-18T08:12:17.011 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:51.891145+0000", 2024-09-18T08:12:17.011 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 2.4229000000000001e-05 2024-09-18T08:12:17.012 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:12:17.012 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:12:17.012 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "dispatched", 2024-09-18T08:12:17.012 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:51.891156+0000", 2024-09-18T08:12:17.012 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.1479e-05 2024-09-18T08:12:17.012 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:12:17.012 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:12:17.012 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "queued_for_pg", 2024-09-18T08:12:17.012 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:51.891163+0000", 2024-09-18T08:12:17.012 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 6.4989999999999999e-06 2024-09-18T08:12:17.012 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:12:17.012 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:12:17.012 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-09-18T08:12:17.012 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:51.891224+0000", 2024-09-18T08:12:17.012 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 6.1450000000000003e-05 2024-09-18T08:12:17.013 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:12:17.013 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:12:17.013 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "waiting for peered", 2024-09-18T08:12:17.013 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:51.891236+0000", 2024-09-18T08:12:17.013 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.205e-05 2024-09-18T08:12:17.013 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:12:17.013 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:12:17.013 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-09-18T08:12:17.013 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:52.697963+0000", 2024-09-18T08:12:17.013 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.80672717599999999 2024-09-18T08:12:17.013 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:12:17.013 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:12:17.013 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "waiting for peered", 2024-09-18T08:12:17.014 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:52.697971+0000", 2024-09-18T08:12:17.014 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 7.9000000000000006e-06 2024-09-18T08:12:17.014 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:12:17.014 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:12:17.014 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-09-18T08:12:17.014 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:53.843149+0000", 2024-09-18T08:12:17.014 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.1451775449999999 2024-09-18T08:12:17.014 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:12:17.014 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:12:17.014 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "started", 2024-09-18T08:12:17.014 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:53.843325+0000", 2024-09-18T08:12:17.014 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.000176059 2024-09-18T08:12:17.014 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:12:17.014 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:12:17.015 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "waiting for subops from 1,2,4", 2024-09-18T08:12:17.015 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:53.843462+0000", 2024-09-18T08:12:17.015 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.000137446 2024-09-18T08:12:17.015 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:12:17.015 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:12:17.015 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "op_commit", 2024-09-18T08:12:17.015 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:53.848623+0000", 2024-09-18T08:12:17.015 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.0051606589999999997 2024-09-18T08:12:17.015 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:12:17.015 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:12:17.015 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:12:17.015 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:53.849075+0000", 2024-09-18T08:12:17.015 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.00045226899999999999 2024-09-18T08:12:17.015 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:12:17.015 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:12:17.016 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:12:17.016 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:53.849640+0000", 2024-09-18T08:12:17.016 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.00056498900000000001 2024-09-18T08:12:17.016 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:12:17.016 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:12:17.016 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:12:17.016 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:53.857586+0000", 2024-09-18T08:12:17.016 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.0079462319999999993 2024-09-18T08:12:17.016 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:12:17.016 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:12:17.016 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "commit_sent", 2024-09-18T08:12:17.016 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:53.857652+0000", 2024-09-18T08:12:17.016 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 6.5831e-05 2024-09-18T08:12:17.016 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:12:17.017 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:12:17.017 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "done", 2024-09-18T08:12:17.017 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:53.857774+0000", 2024-09-18T08:12:17.017 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.00012188000000000001 2024-09-18T08:12:17.017 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-09-18T08:12:17.017 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-09-18T08:12:17.017 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-09-18T08:12:17.017 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:12:17.017 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:12:17.017 INFO:teuthology.orchestra.run.smithi007.stdout: "description": "osd_op(client.15216.0:69360 8.f 8:fee0f726:::benchmark_data_smithi007_101583_object69359:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e198)", 2024-09-18T08:12:17.017 INFO:teuthology.orchestra.run.smithi007.stdout: "initiated_at": "2024-09-18T08:08:51.959160+0000", 2024-09-18T08:12:17.017 INFO:teuthology.orchestra.run.smithi007.stdout: "age": 204.894927424, 2024-09-18T08:12:17.017 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.8990693009999999, 2024-09-18T08:12:17.018 INFO:teuthology.orchestra.run.smithi007.stdout: "continuous": false, 2024-09-18T08:12:17.018 INFO:teuthology.orchestra.run.smithi007.stdout: "type_data": { 2024-09-18T08:12:17.018 INFO:teuthology.orchestra.run.smithi007.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T08:12:17.018 INFO:teuthology.orchestra.run.smithi007.stdout: "client_info": { 2024-09-18T08:12:17.018 INFO:teuthology.orchestra.run.smithi007.stdout: "client": "client.15216", 2024-09-18T08:12:17.018 INFO:teuthology.orchestra.run.smithi007.stdout: "client_addr": "172.21.15.7:0/2596041177", 2024-09-18T08:12:17.018 INFO:teuthology.orchestra.run.smithi007.stdout: "tid": 69360 2024-09-18T08:12:17.018 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:12:17.018 INFO:teuthology.orchestra.run.smithi007.stdout: "events": [ 2024-09-18T08:12:17.018 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:12:17.018 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "initiated", 2024-09-18T08:12:17.018 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:51.959160+0000", 2024-09-18T08:12:17.018 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-09-18T08:12:17.018 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:12:17.018 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:12:17.018 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "header_read", 2024-09-18T08:12:17.019 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:51.959160+0000", 2024-09-18T08:12:17.019 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-09-18T08:12:17.019 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:12:17.019 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:12:17.019 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "throttled", 2024-09-18T08:12:17.019 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:51.959162+0000", 2024-09-18T08:12:17.019 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 2.1069999999999999e-06 2024-09-18T08:12:17.019 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:12:17.019 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:12:17.019 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "all_read", 2024-09-18T08:12:17.019 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:51.959187+0000", 2024-09-18T08:12:17.019 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 2.4902e-05 2024-09-18T08:12:17.019 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:12:17.019 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:12:17.019 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "dispatched", 2024-09-18T08:12:17.019 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:51.959207+0000", 2024-09-18T08:12:17.020 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 2.0051000000000002e-05 2024-09-18T08:12:17.020 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:12:17.020 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:12:17.020 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "queued_for_pg", 2024-09-18T08:12:17.020 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:51.959215+0000", 2024-09-18T08:12:17.020 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 8.7679999999999997e-06 2024-09-18T08:12:17.020 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:12:17.020 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:12:17.020 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-09-18T08:12:17.020 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:51.959299+0000", 2024-09-18T08:12:17.020 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 8.3129999999999999e-05 2024-09-18T08:12:17.020 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:12:17.021 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:12:17.021 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "waiting for peered", 2024-09-18T08:12:17.021 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:51.959313+0000", 2024-09-18T08:12:17.021 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.3967e-05 2024-09-18T08:12:17.021 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:12:17.021 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:12:17.021 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-09-18T08:12:17.021 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:52.698965+0000", 2024-09-18T08:12:17.021 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.73965287599999996 2024-09-18T08:12:17.021 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:12:17.021 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:12:17.021 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "waiting for peered", 2024-09-18T08:12:17.024 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:52.698974+0000", 2024-09-18T08:12:17.024 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 8.1929999999999993e-06 2024-09-18T08:12:17.024 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:12:17.024 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:12:17.024 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-09-18T08:12:17.024 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:53.845783+0000", 2024-09-18T08:12:17.024 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.1468095030000001 2024-09-18T08:12:17.024 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:12:17.025 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:12:17.025 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "started", 2024-09-18T08:12:17.025 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:53.845945+0000", 2024-09-18T08:12:17.025 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.00016167500000000001 2024-09-18T08:12:17.025 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:12:17.025 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:12:17.025 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "waiting for subops from 1,2,4", 2024-09-18T08:12:17.025 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:53.846104+0000", 2024-09-18T08:12:17.025 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.00015877600000000001 2024-09-18T08:12:17.025 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:12:17.025 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:12:17.025 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "op_commit", 2024-09-18T08:12:17.025 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:53.849087+0000", 2024-09-18T08:12:17.026 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.0029835420000000001 2024-09-18T08:12:17.026 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:12:17.026 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:12:17.026 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:12:17.026 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:53.849594+0000", 2024-09-18T08:12:17.026 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.00050686399999999999 2024-09-18T08:12:17.026 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:12:17.026 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:12:17.026 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:12:17.026 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:53.857524+0000", 2024-09-18T08:12:17.026 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.0079296000000000002 2024-09-18T08:12:17.026 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:12:17.026 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:12:17.026 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:12:17.027 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:53.857905+0000", 2024-09-18T08:12:17.027 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.000381107 2024-09-18T08:12:17.027 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:12:17.027 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:12:17.027 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "commit_sent", 2024-09-18T08:12:17.027 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:53.858073+0000", 2024-09-18T08:12:17.027 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.00016861900000000001 2024-09-18T08:12:17.027 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:12:17.027 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:12:17.027 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "done", 2024-09-18T08:12:17.027 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:53.858229+0000", 2024-09-18T08:12:17.027 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.00015562099999999999 2024-09-18T08:12:17.027 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-09-18T08:12:17.028 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-09-18T08:12:17.028 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-09-18T08:12:17.028 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:12:17.028 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:12:17.028 INFO:teuthology.orchestra.run.smithi007.stdout: "description": "osd_op(client.15216.0:100478 8.f 8:fbdbf1d3:::benchmark_data_smithi007_101583_object100477:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e210)", 2024-09-18T08:12:17.028 INFO:teuthology.orchestra.run.smithi007.stdout: "initiated_at": "2024-09-18T08:09:27.339441+0000", 2024-09-18T08:12:17.028 INFO:teuthology.orchestra.run.smithi007.stdout: "age": 169.51464592900001, 2024-09-18T08:12:17.028 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.0272729549999999, 2024-09-18T08:12:17.028 INFO:teuthology.orchestra.run.smithi007.stdout: "continuous": false, 2024-09-18T08:12:17.028 INFO:teuthology.orchestra.run.smithi007.stdout: "type_data": { 2024-09-18T08:12:17.028 INFO:teuthology.orchestra.run.smithi007.stdout: "flag_point": "reached pg", 2024-09-18T08:12:17.028 INFO:teuthology.orchestra.run.smithi007.stdout: "client_info": { 2024-09-18T08:12:17.028 INFO:teuthology.orchestra.run.smithi007.stdout: "client": "client.15216", 2024-09-18T08:12:17.029 INFO:teuthology.orchestra.run.smithi007.stdout: "client_addr": "172.21.15.7:0/2596041177", 2024-09-18T08:12:17.029 INFO:teuthology.orchestra.run.smithi007.stdout: "tid": 100478 2024-09-18T08:12:17.029 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:12:17.029 INFO:teuthology.orchestra.run.smithi007.stdout: "events": [ 2024-09-18T08:12:17.029 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:12:17.029 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "initiated", 2024-09-18T08:12:17.029 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:09:27.339441+0000", 2024-09-18T08:12:17.029 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-09-18T08:12:17.029 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:12:17.029 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:12:17.029 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "header_read", 2024-09-18T08:12:17.029 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:09:27.339441+0000", 2024-09-18T08:12:17.029 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-09-18T08:12:17.029 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:12:17.030 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:12:17.030 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "throttled", 2024-09-18T08:12:17.030 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:09:27.339442+0000", 2024-09-18T08:12:17.030 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 8.9800000000000002e-07 2024-09-18T08:12:17.030 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:12:17.030 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:12:17.030 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "all_read", 2024-09-18T08:12:17.030 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:09:27.339463+0000", 2024-09-18T08:12:17.030 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 2.0744999999999999e-05 2024-09-18T08:12:17.030 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:12:17.030 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:12:17.030 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "dispatched", 2024-09-18T08:12:17.030 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:09:27.339477+0000", 2024-09-18T08:12:17.031 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.4656e-05 2024-09-18T08:12:17.031 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:12:17.031 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:12:17.031 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "queued_for_pg", 2024-09-18T08:12:17.031 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:09:27.339485+0000", 2024-09-18T08:12:17.031 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 7.2699999999999999e-06 2024-09-18T08:12:17.031 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:12:17.031 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:12:17.031 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-09-18T08:12:17.031 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:09:27.339537+0000", 2024-09-18T08:12:17.031 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 5.2635e-05 2024-09-18T08:12:17.031 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:12:17.031 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:12:17.032 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "started", 2024-09-18T08:12:17.032 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:09:27.339659+0000", 2024-09-18T08:12:17.032 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.000121951 2024-09-18T08:12:17.032 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:12:17.032 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:12:17.032 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "waiting for subops from 1,2,4", 2024-09-18T08:12:17.032 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:09:27.339805+0000", 2024-09-18T08:12:17.032 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.00014521799999999999 2024-09-18T08:12:17.032 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:12:17.032 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:12:17.032 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "op_commit", 2024-09-18T08:12:17.032 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:09:27.343945+0000", 2024-09-18T08:12:17.032 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.0041406120000000001 2024-09-18T08:12:17.033 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:12:17.033 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:12:17.033 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:12:17.033 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:09:27.345369+0000", 2024-09-18T08:12:17.033 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.0014239299999999999 2024-09-18T08:12:17.033 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:12:17.033 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:12:17.033 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:12:17.033 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:09:27.345414+0000", 2024-09-18T08:12:17.033 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 4.4471999999999997e-05 2024-09-18T08:12:17.033 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:12:17.033 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:12:17.033 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-09-18T08:12:17.033 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:09:27.362113+0000", 2024-09-18T08:12:17.034 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.016699421999999998 2024-09-18T08:12:17.034 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:12:17.034 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:12:17.034 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "waiting for peered", 2024-09-18T08:12:17.034 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:09:27.362124+0000", 2024-09-18T08:12:17.034 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.1268e-05 2024-09-18T08:12:17.034 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:12:17.034 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:12:17.034 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-09-18T08:12:17.034 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:09:28.366617+0000", 2024-09-18T08:12:17.034 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.0044926059999999 2024-09-18T08:12:17.034 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:12:17.034 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:12:17.034 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "done", 2024-09-18T08:12:17.034 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:09:28.366714+0000", 2024-09-18T08:12:17.035 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 9.7272e-05 2024-09-18T08:12:17.035 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-09-18T08:12:17.035 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-09-18T08:12:17.035 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-09-18T08:12:17.035 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:12:17.035 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:12:17.035 INFO:teuthology.orchestra.run.smithi007.stdout: "description": "osd_op(client.15216.0:100489 8.f 8:f94e46ee:::benchmark_data_smithi007_101583_object100488:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e211)", 2024-09-18T08:12:17.035 INFO:teuthology.orchestra.run.smithi007.stdout: "initiated_at": "2024-09-18T08:09:27.351893+0000", 2024-09-18T08:12:17.035 INFO:teuthology.orchestra.run.smithi007.stdout: "age": 169.50219361500001, 2024-09-18T08:12:17.036 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.0213833750000001, 2024-09-18T08:12:17.036 INFO:teuthology.orchestra.run.smithi007.stdout: "continuous": false, 2024-09-18T08:12:17.036 INFO:teuthology.orchestra.run.smithi007.stdout: "type_data": { 2024-09-18T08:12:17.036 INFO:teuthology.orchestra.run.smithi007.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T08:12:17.036 INFO:teuthology.orchestra.run.smithi007.stdout: "client_info": { 2024-09-18T08:12:17.036 INFO:teuthology.orchestra.run.smithi007.stdout: "client": "client.15216", 2024-09-18T08:12:17.036 INFO:teuthology.orchestra.run.smithi007.stdout: "client_addr": "172.21.15.7:0/2596041177", 2024-09-18T08:12:17.036 INFO:teuthology.orchestra.run.smithi007.stdout: "tid": 100489 2024-09-18T08:12:17.036 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:12:17.036 INFO:teuthology.orchestra.run.smithi007.stdout: "events": [ 2024-09-18T08:12:17.036 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:12:17.036 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "initiated", 2024-09-18T08:12:17.036 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:09:27.351893+0000", 2024-09-18T08:12:17.037 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-09-18T08:12:17.037 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:12:17.037 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:12:17.037 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "header_read", 2024-09-18T08:12:17.037 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Sep 18 08:12:16 smithi007 ceph-779349ca-7593-11ef-bceb-c7b262605968-mgr-y[29688]: ::ffff:172.21.15.50 - - [18/Sep/2024:08:12:16] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-09-18T08:12:17.037 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:12:17.038 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:09:27.351893+0000", 2024-09-18T08:12:17.038 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-09-18T08:12:17.038 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:12:17.038 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:12:17.039 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "throttled", 2024-09-18T08:12:17.039 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:09:27.351894+0000", 2024-09-18T08:12:17.039 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 5.5300000000000004e-07 2024-09-18T08:12:17.039 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:12:17.039 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:12:17.039 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "all_read", 2024-09-18T08:12:17.039 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:09:27.351919+0000", 2024-09-18T08:12:17.039 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 2.4837999999999998e-05 2024-09-18T08:12:17.039 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:12:17.039 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:12:17.040 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "dispatched", 2024-09-18T08:12:17.040 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:09:27.351926+0000", 2024-09-18T08:12:17.040 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 7.277e-06 2024-09-18T08:12:17.040 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:12:17.040 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:12:17.040 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "queued_for_pg", 2024-09-18T08:12:17.040 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:09:27.351933+0000", 2024-09-18T08:12:17.040 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 6.4810000000000003e-06 2024-09-18T08:12:17.040 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:12:17.040 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:12:17.040 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-09-18T08:12:17.041 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:09:27.352062+0000", 2024-09-18T08:12:17.041 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.00012982900000000001 2024-09-18T08:12:17.041 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:12:17.041 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:12:17.041 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "op must wait for map", 2024-09-18T08:12:17.041 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:09:27.352069+0000", 2024-09-18T08:12:17.041 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 6.8739999999999997e-06 2024-09-18T08:12:17.041 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:12:17.041 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:12:17.041 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-09-18T08:12:17.041 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:09:27.368007+0000", 2024-09-18T08:12:17.041 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.015938009999999999 2024-09-18T08:12:17.041 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:12:17.041 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:12:17.042 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "waiting for peered", 2024-09-18T08:12:17.042 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:09:27.368018+0000", 2024-09-18T08:12:17.042 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.0781000000000001e-05 2024-09-18T08:12:17.042 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:12:17.042 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:12:17.042 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-09-18T08:12:17.042 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:09:28.367234+0000", 2024-09-18T08:12:17.042 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.99921584100000005 2024-09-18T08:12:17.042 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:12:17.042 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:12:17.042 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "started", 2024-09-18T08:12:17.042 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:09:28.367328+0000", 2024-09-18T08:12:17.042 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 9.4083e-05 2024-09-18T08:12:17.042 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:12:17.042 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:12:17.043 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "waiting for subops from 1,2", 2024-09-18T08:12:17.043 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:09:28.367448+0000", 2024-09-18T08:12:17.043 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.00012029800000000001 2024-09-18T08:12:17.043 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:12:17.043 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:12:17.043 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:12:17.043 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:09:28.373055+0000", 2024-09-18T08:12:17.043 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.0056071460000000004 2024-09-18T08:12:17.043 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:12:17.043 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:12:17.043 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:12:17.043 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:09:28.373134+0000", 2024-09-18T08:12:17.043 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 7.8740999999999997e-05 2024-09-18T08:12:17.043 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:12:17.043 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:12:17.044 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "op_commit", 2024-09-18T08:12:17.044 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:09:28.373213+0000", 2024-09-18T08:12:17.044 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 7.894e-05 2024-09-18T08:12:17.044 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:12:17.044 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:12:17.044 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "commit_sent", 2024-09-18T08:12:17.044 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:09:28.373255+0000", 2024-09-18T08:12:17.044 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 4.1718000000000003e-05 2024-09-18T08:12:17.044 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:12:17.044 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:12:17.044 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "done", 2024-09-18T08:12:17.044 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:09:28.373277+0000", 2024-09-18T08:12:17.044 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 2.1965000000000001e-05 2024-09-18T08:12:17.044 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-09-18T08:12:17.044 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-09-18T08:12:17.045 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-09-18T08:12:17.045 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-09-18T08:12:17.045 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-09-18T08:12:17.045 INFO:teuthology.orchestra.run.smithi007.stdout:} 2024-09-18T08:12:17.045 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:12:17.045 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5' 2024-09-18T08:12:17.146 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7 2024-09-18T08:12:17.171 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:12:17.172 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:12:17.172 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7' 2024-09-18T08:12:17.195 INFO:tasks.radosbench.radosbench.0.smithi007.stdout: 62 16 105544 105528 106.356 85.1875 0.00299047 0.00939391 2024-09-18T08:12:17.273 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3 2024-09-18T08:12:17.307 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:12:17.308 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:12:17.308 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3' 2024-09-18T08:12:17.409 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7 2024-09-18T08:12:17.442 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:12:17.444 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:12:17.444 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7' 2024-09-18T08:12:17.545 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2 2024-09-18T08:12:17.552 INFO:teuthology.orchestra.run.smithi007.stderr:nodeep-scrub is set 2024-09-18T08:12:17.586 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:12:17.587 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:12:17.587 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2' 2024-09-18T08:12:17.688 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6 2024-09-18T08:12:17.729 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:12:17.730 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:12:17.730 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6' 2024-09-18T08:12:17.816 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:12:17 smithi007 ceph-mon[29459]: from='client.? 172.21.15.7:0/1941453808' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-18T08:12:17.816 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:12:17 smithi007 ceph-mon[39249]: from='client.? 172.21.15.7:0/1941453808' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-18T08:12:17.832 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7 2024-09-18T08:12:17.844 INFO:teuthology.orchestra.run.smithi007.stdout:osd.0: {} 2024-09-18T08:12:17.845 INFO:teuthology.orchestra.run.smithi007.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-18T08:12:17.845 INFO:teuthology.orchestra.run.smithi007.stderr:Error ENXIO: problem getting command descriptions from osd.1 2024-09-18T08:12:17.845 INFO:teuthology.orchestra.run.smithi007.stdout:osd.2: {} 2024-09-18T08:12:17.845 INFO:teuthology.orchestra.run.smithi007.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-18T08:12:17.871 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:12:17.872 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:12:17.872 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7' 2024-09-18T08:12:17.876 INFO:teuthology.orchestra.run.smithi007.stdout:osd.3: {} 2024-09-18T08:12:17.876 INFO:teuthology.orchestra.run.smithi007.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-18T08:12:17.905 INFO:teuthology.orchestra.run.smithi007.stdout:osd.4: {} 2024-09-18T08:12:17.905 INFO:teuthology.orchestra.run.smithi007.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-18T08:12:17.906 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:12:17 smithi050 ceph-mon[32371]: from='client.? 172.21.15.7:0/1941453808' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-18T08:12:17.937 INFO:teuthology.orchestra.run.smithi007.stdout:osd.5: {} 2024-09-18T08:12:17.938 INFO:teuthology.orchestra.run.smithi007.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-18T08:12:17.967 INFO:teuthology.orchestra.run.smithi007.stdout:osd.6: {} 2024-09-18T08:12:17.968 INFO:teuthology.orchestra.run.smithi007.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-18T08:12:17.973 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6 2024-09-18T08:12:17.997 INFO:teuthology.orchestra.run.smithi007.stdout:osd.7: {} 2024-09-18T08:12:17.997 INFO:teuthology.orchestra.run.smithi007.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-18T08:12:18.008 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:12:18.009 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:12:18.010 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6' 2024-09-18T08:12:18.110 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6 2024-09-18T08:12:18.148 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:12:18.148 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:12:18.149 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6' 2024-09-18T08:12:18.195 INFO:tasks.radosbench.radosbench.0.smithi007.stdout: 63 16 106998 106982 106.11 90.875 0.0089924 0.00941997 2024-09-18T08:12:18.252 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3 2024-09-18T08:12:18.284 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:12:18.285 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:12:18.286 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3' 2024-09-18T08:12:18.386 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7 2024-09-18T08:12:18.427 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:12:18.428 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:12:18.428 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7' 2024-09-18T08:12:18.529 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6 2024-09-18T08:12:18.569 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:12:18.570 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:12:18.570 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6' 2024-09-18T08:12:18.671 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0 2024-09-18T08:12:18.707 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:12:18.708 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:12:18.708 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0' 2024-09-18T08:12:18.783 DEBUG:teuthology.orchestra.run.smithi050:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid 779349ca-7593-11ef-bceb-c7b262605968 -- ceph daemon osd.4 dump_ops_in_flight 2024-09-18T08:12:18.809 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5 2024-09-18T08:12:18.825 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:12:18 smithi050 ceph-mon[32371]: pgmap v455: 158 pgs: 30 active+undersized, 34 active+undersized+degraded, 94 active+clean; 6.2 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 639 B/s rd, 54 MiB/s wr, 865 op/s; 56849/302976 objects degraded (18.764%) 2024-09-18T08:12:18.825 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:12:18 smithi050 ceph-mon[32371]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-18T08:12:18.825 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:12:18 smithi050 ceph-mon[32371]: from='client.? 172.21.15.7:0/1941453808' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-18T08:12:18.825 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:12:18 smithi050 ceph-mon[32371]: osdmap e256: 8 total, 7 up, 8 in 2024-09-18T08:12:18.826 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:12:18 smithi050 ceph-mon[32371]: from='client.? 172.21.15.7:0/1895266109' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-18T08:12:18.826 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:12:18 smithi050 ceph-mon[32371]: from='mgr.24470 172.21.15.7:0/2594390261' entity='mgr.y' 2024-09-18T08:12:18.826 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:12:18 smithi050 ceph-mon[32371]: from='mgr.24470 172.21.15.7:0/2594390261' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-18T08:12:18.867 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:12:18.867 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:12:18.868 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5' 2024-09-18T08:12:18.868 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:12:18 smithi007 ceph-mon[29459]: pgmap v455: 158 pgs: 30 active+undersized, 34 active+undersized+degraded, 94 active+clean; 6.2 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 639 B/s rd, 54 MiB/s wr, 865 op/s; 56849/302976 objects degraded (18.764%) 2024-09-18T08:12:18.868 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:12:18 smithi007 ceph-mon[29459]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-18T08:12:18.869 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:12:18 smithi007 ceph-mon[29459]: from='client.? 172.21.15.7:0/1941453808' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-18T08:12:18.869 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:12:18 smithi007 ceph-mon[29459]: osdmap e256: 8 total, 7 up, 8 in 2024-09-18T08:12:18.869 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:12:18 smithi007 ceph-mon[29459]: from='client.? 172.21.15.7:0/1895266109' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-18T08:12:18.869 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:12:18 smithi007 ceph-mon[29459]: from='mgr.24470 172.21.15.7:0/2594390261' entity='mgr.y' 2024-09-18T08:12:18.869 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:12:18 smithi007 ceph-mon[29459]: from='mgr.24470 172.21.15.7:0/2594390261' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-18T08:12:18.869 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:12:18 smithi007 ceph-mon[39249]: pgmap v455: 158 pgs: 30 active+undersized, 34 active+undersized+degraded, 94 active+clean; 6.2 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 639 B/s rd, 54 MiB/s wr, 865 op/s; 56849/302976 objects degraded (18.764%) 2024-09-18T08:12:18.869 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:12:18 smithi007 ceph-mon[39249]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-18T08:12:18.869 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:12:18 smithi007 ceph-mon[39249]: from='client.? 172.21.15.7:0/1941453808' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-18T08:12:18.869 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:12:18 smithi007 ceph-mon[39249]: osdmap e256: 8 total, 7 up, 8 in 2024-09-18T08:12:18.869 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:12:18 smithi007 ceph-mon[39249]: from='client.? 172.21.15.7:0/1895266109' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-18T08:12:18.869 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:12:18 smithi007 ceph-mon[39249]: from='mgr.24470 172.21.15.7:0/2594390261' entity='mgr.y' 2024-09-18T08:12:18.870 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:12:18 smithi007 ceph-mon[39249]: from='mgr.24470 172.21.15.7:0/2594390261' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-18T08:12:18.968 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4 2024-09-18T08:12:19.027 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:12:19.028 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:12:19.028 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4' 2024-09-18T08:12:19.129 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3 2024-09-18T08:12:19.157 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:12:19.158 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:12:19.158 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3' 2024-09-18T08:12:19.195 INFO:tasks.radosbench.radosbench.0.smithi007.stdout: 64 16 108393 108377 105.814 87.1875 0.0171434 0.00944569 2024-09-18T08:12:19.254 INFO:teuthology.orchestra.run.smithi050.stderr:Inferring config /var/lib/ceph/779349ca-7593-11ef-bceb-c7b262605968/mon.b/config 2024-09-18T08:12:19.259 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7 2024-09-18T08:12:19.318 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:12:19.318 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:12:19.319 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7' 2024-09-18T08:12:19.420 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6 2024-09-18T08:12:19.459 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:12:19.460 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:12:19.460 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6' 2024-09-18T08:12:19.561 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6 2024-09-18T08:12:19.598 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:12:19.599 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:12:19.599 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6' 2024-09-18T08:12:19.631 INFO:journalctl@ceph.iscsi.iscsi.a.smithi050.stdout:Sep 18 08:12:19 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-iscsi-iscsi-a[57782]: debug there is no tcmu-runner data available 2024-09-18T08:12:19.700 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2 2024-09-18T08:12:19.729 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:12:19.729 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:12:19.730 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2' 2024-09-18T08:12:19.830 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4 2024-09-18T08:12:19.873 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:12:19.874 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:12:19.875 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4' 2024-09-18T08:12:19.956 INFO:teuthology.orchestra.run.smithi050.stdout:ERROR: (22) Invalid argument 2024-09-18T08:12:19.956 INFO:teuthology.orchestra.run.smithi050.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-18T08:12:19.975 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4 2024-09-18T08:12:19.999 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:12:19.999 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:12:20.000 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4' 2024-09-18T08:12:20.101 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5 2024-09-18T08:12:20.143 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:12:20.144 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:12:20.144 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5' 2024-09-18T08:12:20.196 INFO:tasks.radosbench.radosbench.0.smithi007.stdout: 65 15 109570 109555 105.319 73.625 0.0113223 0.00949027 2024-09-18T08:12:20.245 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0 2024-09-18T08:12:20.277 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:12:20.278 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:12:20.278 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0' 2024-09-18T08:12:20.379 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5 2024-09-18T08:12:20.418 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:12:20.419 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:12:20.419 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5' 2024-09-18T08:12:20.520 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4 2024-09-18T08:12:20.559 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:12:20.559 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:12:20.560 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4' 2024-09-18T08:12:20.656 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:12:20 smithi050 ceph-mon[32371]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-18T08:12:20.656 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:12:20 smithi050 ceph-mon[32371]: pgmap v457: 158 pgs: 30 active+undersized, 34 active+undersized+degraded, 94 active+clean; 6.6 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 1.2 KiB/s rd, 107 MiB/s wr, 1.72k op/s; 60662/323472 objects degraded (18.753%) 2024-09-18T08:12:20.656 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:12:20 smithi050 ceph-mon[32371]: Health check update: Degraded data redundancy: 56849/302976 objects degraded (18.764%), 34 pgs degraded (PG_DEGRADED) 2024-09-18T08:12:20.656 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:12:20 smithi050 ceph-mon[32371]: osdmap e257: 8 total, 7 up, 8 in 2024-09-18T08:12:20.656 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:12:20 smithi050 ceph-mon[32371]: purged_snaps scrub starts 2024-09-18T08:12:20.656 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:12:20 smithi050 ceph-mon[32371]: purged_snaps scrub ok 2024-09-18T08:12:20.660 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0 2024-09-18T08:12:20.685 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:12:20.686 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:12:20.686 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0' 2024-09-18T08:12:20.730 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:12:20 smithi007 ceph-mon[29459]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-18T08:12:20.730 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:12:20 smithi007 ceph-mon[29459]: pgmap v457: 158 pgs: 30 active+undersized, 34 active+undersized+degraded, 94 active+clean; 6.6 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 1.2 KiB/s rd, 107 MiB/s wr, 1.72k op/s; 60662/323472 objects degraded (18.753%) 2024-09-18T08:12:20.730 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:12:20 smithi007 ceph-mon[29459]: Health check update: Degraded data redundancy: 56849/302976 objects degraded (18.764%), 34 pgs degraded (PG_DEGRADED) 2024-09-18T08:12:20.730 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:12:20 smithi007 ceph-mon[29459]: osdmap e257: 8 total, 7 up, 8 in 2024-09-18T08:12:20.731 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:12:20 smithi007 ceph-mon[29459]: purged_snaps scrub starts 2024-09-18T08:12:20.731 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:12:20 smithi007 ceph-mon[29459]: purged_snaps scrub ok 2024-09-18T08:12:20.731 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:12:20 smithi007 ceph-mon[39249]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-18T08:12:20.731 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:12:20 smithi007 ceph-mon[39249]: pgmap v457: 158 pgs: 30 active+undersized, 34 active+undersized+degraded, 94 active+clean; 6.6 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 1.2 KiB/s rd, 107 MiB/s wr, 1.72k op/s; 60662/323472 objects degraded (18.753%) 2024-09-18T08:12:20.731 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:12:20 smithi007 ceph-mon[39249]: Health check update: Degraded data redundancy: 56849/302976 objects degraded (18.764%), 34 pgs degraded (PG_DEGRADED) 2024-09-18T08:12:20.731 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:12:20 smithi007 ceph-mon[39249]: osdmap e257: 8 total, 7 up, 8 in 2024-09-18T08:12:20.731 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:12:20 smithi007 ceph-mon[39249]: purged_snaps scrub starts 2024-09-18T08:12:20.731 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:12:20 smithi007 ceph-mon[39249]: purged_snaps scrub ok 2024-09-18T08:12:20.787 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0 2024-09-18T08:12:20.816 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:12:20.816 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:12:20.817 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0' 2024-09-18T08:12:20.834 DEBUG:teuthology.orchestra.run.smithi050:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid 779349ca-7593-11ef-bceb-c7b262605968 -- ceph daemon osd.4 dump_blocked_ops 2024-09-18T08:12:20.918 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6 2024-09-18T08:12:20.950 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:12:20.951 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:12:20.951 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6' 2024-09-18T08:12:21.052 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7 2024-09-18T08:12:21.099 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:12:21.100 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:12:21.100 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7' 2024-09-18T08:12:21.196 INFO:tasks.radosbench.radosbench.0.smithi007.stdout: 66 16 110869 110853 104.952 81.125 0.0194732 0.00952377 2024-09-18T08:12:21.202 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2 2024-09-18T08:12:21.233 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:12:21.234 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:12:21.234 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2' 2024-09-18T08:12:21.293 INFO:teuthology.orchestra.run.smithi050.stderr:Inferring config /var/lib/ceph/779349ca-7593-11ef-bceb-c7b262605968/mon.b/config 2024-09-18T08:12:21.335 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7 2024-09-18T08:12:21.387 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:12:21.388 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:12:21.389 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7' 2024-09-18T08:12:21.489 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2 2024-09-18T08:12:21.521 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:12:21.522 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:12:21.522 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2' 2024-09-18T08:12:21.623 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6 2024-09-18T08:12:21.657 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:12:21.658 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:12:21.659 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6' 2024-09-18T08:12:21.745 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:12:21 smithi050 ceph-mon[32371]: osdmap e258: 8 total, 7 up, 8 in 2024-09-18T08:12:21.760 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6 2024-09-18T08:12:21.800 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:12:21.801 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:12:21.801 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6' 2024-09-18T08:12:21.902 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2 2024-09-18T08:12:21.921 INFO:teuthology.orchestra.run.smithi050.stdout:ERROR: (22) Invalid argument 2024-09-18T08:12:21.921 INFO:teuthology.orchestra.run.smithi050.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-18T08:12:21.930 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:12:21.931 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:12:21.931 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2' 2024-09-18T08:12:21.980 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:12:21 smithi007 ceph-mon[29459]: osdmap e258: 8 total, 7 up, 8 in 2024-09-18T08:12:21.980 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:12:21 smithi007 ceph-mon[39249]: osdmap e258: 8 total, 7 up, 8 in 2024-09-18T08:12:22.032 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5 2024-09-18T08:12:22.073 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:12:22.074 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:12:22.074 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5' 2024-09-18T08:12:22.119 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-09-18T08:12:22.119 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-09-18T08:12:22.175 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2 2024-09-18T08:12:22.196 INFO:tasks.radosbench.radosbench.0.smithi007.stdout: 67 16 112227 112211 104.652 84.875 0.0115465 0.00955091 2024-09-18T08:12:22.208 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:12:22.209 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:12:22.209 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2' 2024-09-18T08:12:22.310 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2 2024-09-18T08:12:22.344 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:12:22.348 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:12:22.348 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2' 2024-09-18T08:12:22.449 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6 2024-09-18T08:12:22.491 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:12:22.492 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:12:22.492 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6' 2024-09-18T08:12:22.592 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3 2024-09-18T08:12:22.625 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:12:22.626 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:12:22.626 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3' 2024-09-18T08:12:22.705 DEBUG:teuthology.orchestra.run.smithi050:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid 779349ca-7593-11ef-bceb-c7b262605968 -- ceph daemon osd.4 dump_historic_ops 2024-09-18T08:12:22.727 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4 2024-09-18T08:12:22.746 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:12:22 smithi050 ceph-mon[32371]: pgmap v460: 158 pgs: 30 active+undersized, 34 active+undersized+degraded, 94 active+clean; 6.6 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 853 B/s rd, 71 MiB/s wr, 1.14k op/s; 60662/323472 objects degraded (18.753%) 2024-09-18T08:12:22.799 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:12:22.800 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:12:22.800 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4' 2024-09-18T08:12:22.901 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0 2024-09-18T08:12:22.932 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:12:22.933 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:12:22.934 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0' 2024-09-18T08:12:22.980 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:12:22 smithi007 ceph-mon[29459]: pgmap v460: 158 pgs: 30 active+undersized, 34 active+undersized+degraded, 94 active+clean; 6.6 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 853 B/s rd, 71 MiB/s wr, 1.14k op/s; 60662/323472 objects degraded (18.753%) 2024-09-18T08:12:22.980 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:12:22 smithi007 ceph-mon[39249]: pgmap v460: 158 pgs: 30 active+undersized, 34 active+undersized+degraded, 94 active+clean; 6.6 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 853 B/s rd, 71 MiB/s wr, 1.14k op/s; 60662/323472 objects degraded (18.753%) 2024-09-18T08:12:23.035 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0 2024-09-18T08:12:23.066 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:12:23.067 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:12:23.067 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0' 2024-09-18T08:12:23.132 INFO:teuthology.orchestra.run.smithi007.stderr:Creating an OSD daemon form without an OSD FSID value 2024-09-18T08:12:23.166 INFO:teuthology.orchestra.run.smithi050.stderr:Inferring config /var/lib/ceph/779349ca-7593-11ef-bceb-c7b262605968/mon.b/config 2024-09-18T08:12:23.168 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3 2024-09-18T08:12:23.196 INFO:tasks.radosbench.radosbench.0.smithi007.stdout: 68 16 113653 113637 104.424 89.125 0.014933 0.00957156 2024-09-18T08:12:23.202 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:12:23.203 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:12:23.203 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3' 2024-09-18T08:12:23.304 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5 2024-09-18T08:12:23.345 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:12:23.346 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:12:23.346 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5' 2024-09-18T08:12:23.447 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4 2024-09-18T08:12:23.482 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:12:23.483 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:12:23.484 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4' 2024-09-18T08:12:23.585 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3 2024-09-18T08:12:23.617 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:12:23.618 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:12:23.619 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3' 2024-09-18T08:12:23.719 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5 2024-09-18T08:12:23.761 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:12:23.762 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:12:23.763 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5' 2024-09-18T08:12:23.791 INFO:teuthology.orchestra.run.smithi050.stdout:ERROR: (22) Invalid argument 2024-09-18T08:12:23.791 INFO:teuthology.orchestra.run.smithi050.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-18T08:12:23.864 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4 2024-09-18T08:12:23.901 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:12:23.901 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:12:23.902 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4' 2024-09-18T08:12:24.003 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3 2024-09-18T08:12:24.036 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:12:24.037 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:12:24.038 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3' 2024-09-18T08:12:24.121 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid 779349ca-7593-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-18T08:12:24.139 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7 2024-09-18T08:12:24.177 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:12:24.181 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:12:24.182 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7' 2024-09-18T08:12:24.196 INFO:tasks.radosbench.radosbench.0.smithi007.stdout: 69 15 114979 114964 104.113 82.9375 0.01521 0.00960067 2024-09-18T08:12:24.282 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5 2024-09-18T08:12:24.317 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:12:24.318 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:12:24.318 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5' 2024-09-18T08:12:24.419 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5 2024-09-18T08:12:24.460 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:12:24.460 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:12:24.461 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5' 2024-09-18T08:12:24.491 DEBUG:teuthology.orchestra.run.smithi050:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid 779349ca-7593-11ef-bceb-c7b262605968 -- ceph daemon osd.5 dump_ops_in_flight 2024-09-18T08:12:24.546 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/779349ca-7593-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T08:12:24.562 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3 2024-09-18T08:12:24.578 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid 779349ca-7593-11ef-bceb-c7b262605968 -- ceph osd unset noscrub 2024-09-18T08:12:24.608 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:12:24.608 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:12:24.609 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3' 2024-09-18T08:12:24.710 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2 2024-09-18T08:12:24.753 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:12:24.754 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:12:24.754 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2' 2024-09-18T08:12:24.855 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4 2024-09-18T08:12:24.864 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:12:24 smithi007 ceph-mon[29459]: pgmap v461: 158 pgs: 30 active+undersized, 34 active+undersized+degraded, 94 active+clean; 6.6 GiB data, 21 GiB used, 695 GiB / 715 GiB avail; 853 B/s rd, 80 MiB/s wr, 1.28k op/s; 61501/325989 objects degraded (18.866%) 2024-09-18T08:12:24.865 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:12:24 smithi007 ceph-mon[29459]: Health check update: Degraded data redundancy: 61501/325989 objects degraded (18.866%), 34 pgs degraded (PG_DEGRADED) 2024-09-18T08:12:24.865 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:12:24 smithi007 ceph-mon[29459]: osdmap e259: 8 total, 7 up, 8 in 2024-09-18T08:12:24.865 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:12:24 smithi007 ceph-mon[39249]: pgmap v461: 158 pgs: 30 active+undersized, 34 active+undersized+degraded, 94 active+clean; 6.6 GiB data, 21 GiB used, 695 GiB / 715 GiB avail; 853 B/s rd, 80 MiB/s wr, 1.28k op/s; 61501/325989 objects degraded (18.866%) 2024-09-18T08:12:24.865 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:12:24 smithi007 ceph-mon[39249]: Health check update: Degraded data redundancy: 61501/325989 objects degraded (18.866%), 34 pgs degraded (PG_DEGRADED) 2024-09-18T08:12:24.865 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:12:24 smithi007 ceph-mon[39249]: osdmap e259: 8 total, 7 up, 8 in 2024-09-18T08:12:24.906 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:12:24 smithi050 ceph-mon[32371]: pgmap v461: 158 pgs: 30 active+undersized, 34 active+undersized+degraded, 94 active+clean; 6.6 GiB data, 21 GiB used, 695 GiB / 715 GiB avail; 853 B/s rd, 80 MiB/s wr, 1.28k op/s; 61501/325989 objects degraded (18.866%) 2024-09-18T08:12:24.907 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:12:24 smithi050 ceph-mon[32371]: Health check update: Degraded data redundancy: 61501/325989 objects degraded (18.866%), 34 pgs degraded (PG_DEGRADED) 2024-09-18T08:12:24.907 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:12:24 smithi050 ceph-mon[32371]: osdmap e259: 8 total, 7 up, 8 in 2024-09-18T08:12:24.910 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:12:24.911 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:12:24.911 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4' 2024-09-18T08:12:24.961 INFO:teuthology.orchestra.run.smithi050.stderr:Inferring config /var/lib/ceph/779349ca-7593-11ef-bceb-c7b262605968/mon.b/config 2024-09-18T08:12:25.013 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5 2024-09-18T08:12:25.060 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:12:25.061 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:12:25.061 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5' 2024-09-18T08:12:25.162 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3 2024-09-18T08:12:25.196 INFO:tasks.radosbench.radosbench.0.smithi007.stdout: 70 16 116287 116271 103.792 81.6875 0.0139698 0.00962967 2024-09-18T08:12:25.205 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:12:25.209 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:12:25.209 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3' 2024-09-18T08:12:25.304 INFO:teuthology.orchestra.run.smithi007.stdout:osd.0: {} 2024-09-18T08:12:25.304 INFO:teuthology.orchestra.run.smithi007.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-18T08:12:25.305 INFO:teuthology.orchestra.run.smithi007.stderr:Error ENXIO: problem getting command descriptions from osd.1 2024-09-18T08:12:25.310 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7 2024-09-18T08:12:25.335 INFO:teuthology.orchestra.run.smithi007.stdout:osd.2: {} 2024-09-18T08:12:25.335 INFO:teuthology.orchestra.run.smithi007.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-18T08:12:25.346 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:12:25.346 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:12:25.347 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7' 2024-09-18T08:12:25.367 INFO:teuthology.orchestra.run.smithi007.stdout:osd.3: {} 2024-09-18T08:12:25.367 INFO:teuthology.orchestra.run.smithi007.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-18T08:12:25.413 INFO:teuthology.orchestra.run.smithi007.stdout:osd.4: {} 2024-09-18T08:12:25.413 INFO:teuthology.orchestra.run.smithi007.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-18T08:12:25.448 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0 2024-09-18T08:12:25.451 INFO:teuthology.orchestra.run.smithi007.stdout:osd.5: {} 2024-09-18T08:12:25.452 INFO:teuthology.orchestra.run.smithi007.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-18T08:12:25.466 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/779349ca-7593-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T08:12:25.489 INFO:teuthology.orchestra.run.smithi007.stdout:osd.6: {} 2024-09-18T08:12:25.489 INFO:teuthology.orchestra.run.smithi007.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-18T08:12:25.495 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:12:25.496 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:12:25.496 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0' 2024-09-18T08:12:25.527 INFO:teuthology.orchestra.run.smithi007.stdout:osd.7: {} 2024-09-18T08:12:25.527 INFO:teuthology.orchestra.run.smithi007.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-18T08:12:25.579 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2024-09-18T08:12:25.579 INFO:teuthology.orchestra.run.smithi050.stdout: "ops": [ 2024-09-18T08:12:25.579 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:12:25.579 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.15582.0:116745 9.7 9:e9186345:::benchmark_data_smithi007_124019_object116744:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536 in=65536b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e259)", 2024-09-18T08:12:25.579 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-09-18T08:12:25.566011+0000", 2024-09-18T08:12:25.580 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 0.0099990389999999995, 2024-09-18T08:12:25.580 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.010031897, 2024-09-18T08:12:25.580 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-09-18T08:12:25.580 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-09-18T08:12:25.580 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "waiting for sub ops", 2024-09-18T08:12:25.580 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2024-09-18T08:12:25.580 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.15582", 2024-09-18T08:12:25.580 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "172.21.15.7:0/3598605839", 2024-09-18T08:12:25.580 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 116745 2024-09-18T08:12:25.580 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:12:25.580 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-09-18T08:12:25.580 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:12:25.580 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-09-18T08:12:25.580 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:12:25.566011+0000", 2024-09-18T08:12:25.581 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-18T08:12:25.581 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:12:25.581 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:12:25.581 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-09-18T08:12:25.581 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:12:25.566011+0000", 2024-09-18T08:12:25.581 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-18T08:12:25.581 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:12:25.581 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:12:25.581 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-09-18T08:12:25.581 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:12:25.566012+0000", 2024-09-18T08:12:25.581 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 7.4900000000000005e-07 2024-09-18T08:12:25.581 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:12:25.582 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:12:25.582 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-09-18T08:12:25.582 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:12:25.566042+0000", 2024-09-18T08:12:25.582 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 2.9697e-05 2024-09-18T08:12:25.582 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:12:25.582 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:12:25.582 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-09-18T08:12:25.582 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:12:25.566050+0000", 2024-09-18T08:12:25.582 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 8.1920000000000005e-06 2024-09-18T08:12:25.582 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:12:25.582 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:12:25.582 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-09-18T08:12:25.582 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:12:25.566057+0000", 2024-09-18T08:12:25.582 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 7.0809999999999996e-06 2024-09-18T08:12:25.583 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:12:25.583 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:12:25.583 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-18T08:12:25.583 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:12:25.569622+0000", 2024-09-18T08:12:25.583 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0035649789999999998 2024-09-18T08:12:25.583 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:12:25.583 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:12:25.583 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-09-18T08:12:25.583 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:12:25.569745+0000", 2024-09-18T08:12:25.583 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00012311399999999999 2024-09-18T08:12:25.583 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:12:25.583 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:12:25.583 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "waiting for subops from 3,7", 2024-09-18T08:12:25.584 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:12:25.569883+0000", 2024-09-18T08:12:25.584 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00013820999999999999 2024-09-18T08:12:25.584 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-18T08:12:25.584 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-09-18T08:12:25.584 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-18T08:12:25.584 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:12:25.584 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:12:25.584 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.15582.0:116748 9.f 9:f5c105bd:::benchmark_data_smithi007_124019_object116747:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536 in=65536b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e259)", 2024-09-18T08:12:25.584 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-09-18T08:12:25.566648+0000", 2024-09-18T08:12:25.584 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 0.0093625600000000007, 2024-09-18T08:12:25.584 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.009478986, 2024-09-18T08:12:25.585 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-09-18T08:12:25.585 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-09-18T08:12:25.585 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "waiting for sub ops", 2024-09-18T08:12:25.585 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2024-09-18T08:12:25.585 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.15582", 2024-09-18T08:12:25.585 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "172.21.15.7:0/3598605839", 2024-09-18T08:12:25.585 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 116748 2024-09-18T08:12:25.585 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:12:25.585 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-09-18T08:12:25.585 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:12:25.585 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-09-18T08:12:25.585 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:12:25.566648+0000", 2024-09-18T08:12:25.586 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-18T08:12:25.586 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:12:25.586 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:12:25.586 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-09-18T08:12:25.586 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:12:25.566648+0000", 2024-09-18T08:12:25.586 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-18T08:12:25.586 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:12:25.586 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:12:25.586 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-09-18T08:12:25.586 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:12:25.566649+0000", 2024-09-18T08:12:25.586 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 9.7499999999999998e-07 2024-09-18T08:12:25.586 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:12:25.586 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:12:25.587 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-09-18T08:12:25.587 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:12:25.566679+0000", 2024-09-18T08:12:25.587 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 3.0719999999999997e-05 2024-09-18T08:12:25.587 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:12:25.587 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:12:25.587 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-09-18T08:12:25.587 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:12:25.566689+0000", 2024-09-18T08:12:25.587 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 9.9210000000000004e-06 2024-09-18T08:12:25.587 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:12:25.587 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:12:25.587 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-09-18T08:12:25.587 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:12:25.566697+0000", 2024-09-18T08:12:25.587 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 7.7880000000000007e-06 2024-09-18T08:12:25.587 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:12:25.588 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:12:25.588 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-18T08:12:25.588 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:12:25.571122+0000", 2024-09-18T08:12:25.588 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0044249220000000004 2024-09-18T08:12:25.588 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:12:25.588 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:12:25.588 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-09-18T08:12:25.588 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:12:25.571242+0000", 2024-09-18T08:12:25.588 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000120011 2024-09-18T08:12:25.588 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:12:25.588 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:12:25.588 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "waiting for subops from 3", 2024-09-18T08:12:25.588 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:12:25.571384+0000", 2024-09-18T08:12:25.589 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000141965 2024-09-18T08:12:25.589 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-18T08:12:25.589 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-09-18T08:12:25.589 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-18T08:12:25.589 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:12:25.589 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:12:25.589 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.15582.0:116749 9.a 9:53d9765e:::benchmark_data_smithi007_124019_object116748:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536 in=65536b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e259)", 2024-09-18T08:12:25.589 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-09-18T08:12:25.566709+0000", 2024-09-18T08:12:25.589 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 0.0093014039999999992, 2024-09-18T08:12:25.589 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0094908300000000004, 2024-09-18T08:12:25.590 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-09-18T08:12:25.590 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-09-18T08:12:25.590 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "waiting for sub ops", 2024-09-18T08:12:25.590 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2024-09-18T08:12:25.590 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.15582", 2024-09-18T08:12:25.590 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "172.21.15.7:0/3598605839", 2024-09-18T08:12:25.590 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 116749 2024-09-18T08:12:25.590 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:12:25.590 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-09-18T08:12:25.590 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:12:25.590 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-09-18T08:12:25.590 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:12:25.566709+0000", 2024-09-18T08:12:25.590 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-18T08:12:25.590 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:12:25.591 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:12:25.591 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-09-18T08:12:25.591 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:12:25.566709+0000", 2024-09-18T08:12:25.591 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-18T08:12:25.591 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:12:25.591 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:12:25.591 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-09-18T08:12:25.591 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:12:25.566709+0000", 2024-09-18T08:12:25.591 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 5.7800000000000001e-07 2024-09-18T08:12:25.591 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:12:25.591 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:12:25.591 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-09-18T08:12:25.591 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:12:25.566753+0000", 2024-09-18T08:12:25.591 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4.3229000000000002e-05 2024-09-18T08:12:25.592 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:12:25.592 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:12:25.592 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-09-18T08:12:25.592 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:12:25.566760+0000", 2024-09-18T08:12:25.592 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 7.4830000000000002e-06 2024-09-18T08:12:25.592 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:12:25.592 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:12:25.592 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-09-18T08:12:25.592 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:12:25.566767+0000", 2024-09-18T08:12:25.592 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 6.4860000000000002e-06 2024-09-18T08:12:25.592 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:12:25.592 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:12:25.592 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-18T08:12:25.593 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:12:25.572462+0000", 2024-09-18T08:12:25.593 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00569521 2024-09-18T08:12:25.593 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:12:25.593 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:12:25.593 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-09-18T08:12:25.593 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:12:25.572612+0000", 2024-09-18T08:12:25.593 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000150025 2024-09-18T08:12:25.593 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:12:25.593 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:12:25.593 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "waiting for subops from 7", 2024-09-18T08:12:25.593 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:12:25.572843+0000", 2024-09-18T08:12:25.593 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00023079400000000001 2024-09-18T08:12:25.593 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-18T08:12:25.593 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-09-18T08:12:25.594 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-18T08:12:25.594 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:12:25.594 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:12:25.594 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.15582.0:116754 9.f 9.75bb969f (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e259)", 2024-09-18T08:12:25.594 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-09-18T08:12:25.571798+0000", 2024-09-18T08:12:25.594 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 0.0042124850000000002, 2024-09-18T08:12:25.594 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0044683429999999996, 2024-09-18T08:12:25.595 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-09-18T08:12:25.595 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-09-18T08:12:25.595 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "queued for pg", 2024-09-18T08:12:25.595 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2024-09-18T08:12:25.595 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.15582", 2024-09-18T08:12:25.595 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "172.21.15.7:0/3598605839", 2024-09-18T08:12:25.595 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 116754 2024-09-18T08:12:25.595 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:12:25.595 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-09-18T08:12:25.595 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:12:25.595 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-09-18T08:12:25.595 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:12:25.571798+0000", 2024-09-18T08:12:25.595 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-18T08:12:25.595 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:12:25.595 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:12:25.596 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-09-18T08:12:25.596 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:12:25.571798+0000", 2024-09-18T08:12:25.596 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-18T08:12:25.596 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:12:25.596 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:12:25.596 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-09-18T08:12:25.596 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:12:25.571799+0000", 2024-09-18T08:12:25.596 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.02e-06 2024-09-18T08:12:25.596 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:12:25.596 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:12:25.596 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-09-18T08:12:25.596 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:12:25.571841+0000", 2024-09-18T08:12:25.596 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4.1869000000000002e-05 2024-09-18T08:12:25.597 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:12:25.597 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:12:25.597 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-09-18T08:12:25.597 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:12:25.571851+0000", 2024-09-18T08:12:25.597 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.0407e-05 2024-09-18T08:12:25.597 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:12:25.597 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:12:25.597 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-09-18T08:12:25.597 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:12:25.571858+0000", 2024-09-18T08:12:25.597 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 7.1040000000000001e-06 2024-09-18T08:12:25.597 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-18T08:12:25.597 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-09-18T08:12:25.597 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-18T08:12:25.598 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:12:25.598 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:12:25.598 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_repop_reply(client.15582.0:116737 9.a e259/244)", 2024-09-18T08:12:25.598 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-09-18T08:12:25.573027+0000", 2024-09-18T08:12:25.598 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 0.0029828300000000001, 2024-09-18T08:12:25.598 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0032846020000000002, 2024-09-18T08:12:25.598 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-09-18T08:12:25.598 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-09-18T08:12:25.598 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "queued for pg", 2024-09-18T08:12:25.598 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-09-18T08:12:25.598 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:12:25.598 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-09-18T08:12:25.598 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:12:25.573027+0000", 2024-09-18T08:12:25.599 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-18T08:12:25.599 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:12:25.599 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:12:25.599 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-09-18T08:12:25.599 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:12:25.573027+0000", 2024-09-18T08:12:25.599 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-18T08:12:25.599 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:12:25.599 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:12:25.599 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-09-18T08:12:25.599 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:12:25.573028+0000", 2024-09-18T08:12:25.599 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 3.7899999999999999e-07 2024-09-18T08:12:25.599 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:12:25.599 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:12:25.599 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-09-18T08:12:25.600 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:12:25.573030+0000", 2024-09-18T08:12:25.600 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 2.2879999999999999e-06 2024-09-18T08:12:25.600 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:12:25.600 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:12:25.600 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-09-18T08:12:25.600 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:12:25.573035+0000", 2024-09-18T08:12:25.600 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 5.1900000000000003e-06 2024-09-18T08:12:25.600 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:12:25.600 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:12:25.600 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-09-18T08:12:25.600 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:12:25.573040+0000", 2024-09-18T08:12:25.600 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4.3579999999999996e-06 2024-09-18T08:12:25.600 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-18T08:12:25.600 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-09-18T08:12:25.601 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-18T08:12:25.601 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:12:25.601 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:12:25.601 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_repop(client.15582.0:116756 9.b e259/232 9:d21821ba:::benchmark_data_smithi007_124019_object116755:head v 259'7202, mlcod=259'7200)", 2024-09-18T08:12:25.601 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-09-18T08:12:25.573937+0000", 2024-09-18T08:12:25.601 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 0.0020731460000000001, 2024-09-18T08:12:25.601 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0024187129999999999, 2024-09-18T08:12:25.601 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-09-18T08:12:25.601 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-09-18T08:12:25.601 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "started", 2024-09-18T08:12:25.601 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-09-18T08:12:25.601 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:12:25.601 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-09-18T08:12:25.602 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:12:25.573937+0000", 2024-09-18T08:12:25.602 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-18T08:12:25.602 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:12:25.602 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:12:25.602 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-09-18T08:12:25.602 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:12:25.573937+0000", 2024-09-18T08:12:25.602 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-18T08:12:25.602 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:12:25.602 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:12:25.602 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-09-18T08:12:25.602 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:12:25.573950+0000", 2024-09-18T08:12:25.602 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.2492e-05 2024-09-18T08:12:25.602 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:12:25.602 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:12:25.603 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-09-18T08:12:25.603 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:12:25.574120+0000", 2024-09-18T08:12:25.603 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000170755 2024-09-18T08:12:25.603 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:12:25.603 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:12:25.603 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-09-18T08:12:25.603 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:12:25.574135+0000", 2024-09-18T08:12:25.603 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.4311000000000001e-05 2024-09-18T08:12:25.603 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:12:25.603 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:12:25.603 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-09-18T08:12:25.603 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:12:25.574141+0000", 2024-09-18T08:12:25.603 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 6.6810000000000001e-06 2024-09-18T08:12:25.604 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:12:25.604 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:12:25.604 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-18T08:12:25.604 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:12:25.574203+0000", 2024-09-18T08:12:25.604 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 6.2038000000000005e-05 2024-09-18T08:12:25.604 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:12:25.604 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:12:25.604 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-09-18T08:12:25.604 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:12:25.574221+0000", 2024-09-18T08:12:25.604 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.7810999999999999e-05 2024-09-18T08:12:25.604 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-18T08:12:25.604 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-09-18T08:12:25.604 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-18T08:12:25.605 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:12:25.605 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:12:25.605 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.15582.0:116758 9.a 9.2b04de4a (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e259)", 2024-09-18T08:12:25.605 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-09-18T08:12:25.575729+0000", 2024-09-18T08:12:25.605 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 0.000280893, 2024-09-18T08:12:25.605 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00067910599999999996, 2024-09-18T08:12:25.605 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-09-18T08:12:25.605 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-09-18T08:12:25.605 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "queued for pg", 2024-09-18T08:12:25.605 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2024-09-18T08:12:25.605 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.15582", 2024-09-18T08:12:25.605 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "172.21.15.7:0/3598605839", 2024-09-18T08:12:25.605 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 116758 2024-09-18T08:12:25.605 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:12:25.606 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-09-18T08:12:25.606 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:12:25.606 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-09-18T08:12:25.606 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:12:25.575729+0000", 2024-09-18T08:12:25.606 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-18T08:12:25.606 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:12:25.606 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:12:25.606 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-09-18T08:12:25.606 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:12:25.575729+0000", 2024-09-18T08:12:25.606 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-18T08:12:25.606 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:12:25.606 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:12:25.606 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-09-18T08:12:25.606 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:12:25.575731+0000", 2024-09-18T08:12:25.606 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.764e-06 2024-09-18T08:12:25.607 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:12:25.607 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:12:25.607 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-09-18T08:12:25.607 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:12:25.575783+0000", 2024-09-18T08:12:25.607 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 5.147e-05 2024-09-18T08:12:25.607 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:12:25.607 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:12:25.607 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-09-18T08:12:25.607 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:12:25.575793+0000", 2024-09-18T08:12:25.607 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.0621999999999999e-05 2024-09-18T08:12:25.607 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:12:25.607 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:12:25.607 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-09-18T08:12:25.607 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:12:25.575801+0000", 2024-09-18T08:12:25.607 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 7.8150000000000006e-06 2024-09-18T08:12:25.608 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-18T08:12:25.608 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-09-18T08:12:25.608 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-18T08:12:25.608 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:12:25.608 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:12:25.608 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_repop_reply(client.15582.0:116748 9.f e259/244)", 2024-09-18T08:12:25.608 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-09-18T08:12:25.575997+0000", 2024-09-18T08:12:25.608 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 1.3159999999999999e-05, 2024-09-18T08:12:25.608 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00046047900000000002, 2024-09-18T08:12:25.608 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-09-18T08:12:25.608 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-09-18T08:12:25.608 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "queued for pg", 2024-09-18T08:12:25.608 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-09-18T08:12:25.608 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:12:25.609 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-09-18T08:12:25.609 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:12:25.575997+0000", 2024-09-18T08:12:25.609 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-18T08:12:25.609 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:12:25.609 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:12:25.609 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-09-18T08:12:25.609 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:12:25.575997+0000", 2024-09-18T08:12:25.609 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-18T08:12:25.609 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:12:25.609 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:12:25.609 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-09-18T08:12:25.609 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:12:25.575997+0000", 2024-09-18T08:12:25.609 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 2.8900000000000001e-07 2024-09-18T08:12:25.610 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:12:25.610 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:12:25.610 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-09-18T08:12:25.610 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:12:25.575999+0000", 2024-09-18T08:12:25.610 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.962e-06 2024-09-18T08:12:25.610 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:12:25.610 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:12:25.610 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-09-18T08:12:25.610 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:12:25.576004+0000", 2024-09-18T08:12:25.610 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4.8579999999999999e-06 2024-09-18T08:12:25.610 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:12:25.610 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:12:25.610 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-09-18T08:12:25.610 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:12:25.576008+0000", 2024-09-18T08:12:25.611 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4.1300000000000003e-06 2024-09-18T08:12:25.611 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-18T08:12:25.611 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-09-18T08:12:25.611 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-18T08:12:25.611 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:12:25.611 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:12:25.611 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_repop(client.15582.0:116757 9.d e259/244 9:b268c261:::benchmark_data_smithi007_124019_object116756:head v 259'7404, mlcod=259'7401)", 2024-09-18T08:12:25.611 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-09-18T08:12:25.576041+0000", 2024-09-18T08:12:25.611 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 4294967295.9999695, 2024-09-18T08:12:25.611 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00046129, 2024-09-18T08:12:25.611 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-09-18T08:12:25.611 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-09-18T08:12:25.611 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "started", 2024-09-18T08:12:25.612 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-09-18T08:12:25.612 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:12:25.612 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-09-18T08:12:25.612 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:12:25.576041+0000", 2024-09-18T08:12:25.612 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-18T08:12:25.612 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:12:25.612 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:12:25.612 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-09-18T08:12:25.612 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:12:25.576041+0000", 2024-09-18T08:12:25.612 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-18T08:12:25.612 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:12:25.612 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:12:25.612 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-09-18T08:12:25.613 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:12:25.576041+0000", 2024-09-18T08:12:25.613 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4.3099999999999998e-07 2024-09-18T08:12:25.613 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:12:25.613 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:12:25.613 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-09-18T08:12:25.613 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:12:25.576064+0000", 2024-09-18T08:12:25.613 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 2.3084000000000001e-05 2024-09-18T08:12:25.613 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:12:25.613 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:12:25.613 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-09-18T08:12:25.613 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:12:25.576071+0000", 2024-09-18T08:12:25.613 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 7.2760000000000003e-06 2024-09-18T08:12:25.613 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:12:25.614 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:12:25.614 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-09-18T08:12:25.614 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:12:25.576077+0000", 2024-09-18T08:12:25.614 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 5.4650000000000002e-06 2024-09-18T08:12:25.614 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:12:25.614 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:12:25.614 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-18T08:12:25.614 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:12:25.576234+0000", 2024-09-18T08:12:25.614 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00015738699999999999 2024-09-18T08:12:25.614 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:12:25.614 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:12:25.614 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-09-18T08:12:25.614 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:12:25.576256+0000", 2024-09-18T08:12:25.615 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 2.1342999999999999e-05 2024-09-18T08:12:25.615 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-18T08:12:25.615 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-09-18T08:12:25.615 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-18T08:12:25.615 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:12:25.615 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:12:25.615 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.15582.0:116759 9.6 9.3ce82266 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e259)", 2024-09-18T08:12:25.615 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-09-18T08:12:25.576138+0000", 2024-09-18T08:12:25.616 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 4294967295.9998717, 2024-09-18T08:12:25.616 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000411137, 2024-09-18T08:12:25.616 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-09-18T08:12:25.616 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-09-18T08:12:25.616 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "queued for pg", 2024-09-18T08:12:25.616 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2024-09-18T08:12:25.616 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.15582", 2024-09-18T08:12:25.616 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "172.21.15.7:0/3598605839", 2024-09-18T08:12:25.616 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 116759 2024-09-18T08:12:25.616 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:12:25.616 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-09-18T08:12:25.616 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:12:25.616 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-09-18T08:12:25.617 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:12:25.576138+0000", 2024-09-18T08:12:25.617 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-18T08:12:25.617 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:12:25.617 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:12:25.617 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-09-18T08:12:25.617 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:12:25.576138+0000", 2024-09-18T08:12:25.617 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-18T08:12:25.617 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:12:25.617 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:12:25.617 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-09-18T08:12:25.617 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:12:25.576139+0000", 2024-09-18T08:12:25.617 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 8.5000000000000001e-07 2024-09-18T08:12:25.617 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:12:25.618 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:12:25.618 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-09-18T08:12:25.618 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:12:25.576169+0000", 2024-09-18T08:12:25.618 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 2.9285000000000002e-05 2024-09-18T08:12:25.618 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:12:25.618 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:12:25.618 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-09-18T08:12:25.618 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:12:25.576177+0000", 2024-09-18T08:12:25.618 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 8.8950000000000004e-06 2024-09-18T08:12:25.618 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:12:25.618 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:12:25.618 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-09-18T08:12:25.618 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:12:25.576184+0000", 2024-09-18T08:12:25.619 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 6.934e-06 2024-09-18T08:12:25.619 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-18T08:12:25.619 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-09-18T08:12:25.619 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-18T08:12:25.619 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:12:25.619 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:12:25.619 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.15582.0:116737 9.a 9:5a5e543b:::benchmark_data_smithi007_124019_object116736:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536 in=65536b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e259)", 2024-09-18T08:12:25.619 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-09-18T08:12:25.558647+0000", 2024-09-18T08:12:25.619 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 0.017363516999999998, 2024-09-18T08:12:25.619 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.017955358000000001, 2024-09-18T08:12:25.619 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-09-18T08:12:25.619 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-09-18T08:12:25.619 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "waiting for sub ops", 2024-09-18T08:12:25.620 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2024-09-18T08:12:25.620 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.15582", 2024-09-18T08:12:25.620 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "172.21.15.7:0/3598605839", 2024-09-18T08:12:25.620 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 116737 2024-09-18T08:12:25.620 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:12:25.620 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-09-18T08:12:25.620 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:12:25.620 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-09-18T08:12:25.620 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:12:25.558647+0000", 2024-09-18T08:12:25.620 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-18T08:12:25.621 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:12:25.621 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:12:25.621 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-09-18T08:12:25.621 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:12:25.558647+0000", 2024-09-18T08:12:25.621 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-18T08:12:25.621 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:12:25.621 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:12:25.621 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-09-18T08:12:25.621 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:12:25.558647+0000", 2024-09-18T08:12:25.621 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 5.7999999999999995e-07 2024-09-18T08:12:25.621 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:12:25.621 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:12:25.622 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-09-18T08:12:25.622 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:12:25.558680+0000", 2024-09-18T08:12:25.622 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 3.2305999999999999e-05 2024-09-18T08:12:25.622 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:12:25.622 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:12:25.622 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-09-18T08:12:25.622 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:12:25.558688+0000", 2024-09-18T08:12:25.622 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 8.3129999999999999e-06 2024-09-18T08:12:25.622 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:12:25.622 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:12:25.622 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-09-18T08:12:25.622 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:12:25.558694+0000", 2024-09-18T08:12:25.622 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 6.4189999999999999e-06 2024-09-18T08:12:25.623 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:12:25.623 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:12:25.623 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-18T08:12:25.623 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:12:25.561618+0000", 2024-09-18T08:12:25.623 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.002923859 2024-09-18T08:12:25.623 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:12:25.623 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:12:25.623 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-09-18T08:12:25.623 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:12:25.561748+0000", 2024-09-18T08:12:25.623 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000129445 2024-09-18T08:12:25.623 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:12:25.623 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:12:25.624 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "waiting for subops from 7", 2024-09-18T08:12:25.624 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:12:25.561959+0000", 2024-09-18T08:12:25.624 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00021134100000000001 2024-09-18T08:12:25.624 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-18T08:12:25.624 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-09-18T08:12:25.624 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-18T08:12:25.624 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-18T08:12:25.624 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2024-09-18T08:12:25.624 INFO:teuthology.orchestra.run.smithi050.stdout: "num_ops": 11 2024-09-18T08:12:25.624 INFO:teuthology.orchestra.run.smithi050.stdout:} 2024-09-18T08:12:25.624 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6 2024-09-18T08:12:25.663 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:12:25.664 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:12:25.664 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6' 2024-09-18T08:12:25.730 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:12:25 smithi007 ceph-mon[29459]: from='client.? 172.21.15.7:0/3979894899' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-18T08:12:25.730 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:12:25 smithi007 ceph-mon[39249]: from='client.? 172.21.15.7:0/3979894899' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-18T08:12:25.765 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5 2024-09-18T08:12:25.801 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:12:25.802 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:12:25.802 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5' 2024-09-18T08:12:25.903 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0 2024-09-18T08:12:25.905 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:12:25 smithi050 ceph-mon[32371]: from='client.? 172.21.15.7:0/3979894899' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-18T08:12:25.939 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:12:25.939 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:12:25.940 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0' 2024-09-18T08:12:26.041 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7 2024-09-18T08:12:26.078 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:12:26.079 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:12:26.079 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7' 2024-09-18T08:12:26.180 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7 2024-09-18T08:12:26.196 INFO:tasks.radosbench.radosbench.0.smithi007.stdout: 71 16 117502 117486 103.4 75.9375 0.0184969 0.00966655 2024-09-18T08:12:26.217 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:12:26.218 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:12:26.218 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7' 2024-09-18T08:12:26.319 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6 2024-09-18T08:12:26.355 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:12:26.355 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:12:26.356 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6' 2024-09-18T08:12:26.457 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0 2024-09-18T08:12:26.491 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:12:26.491 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:12:26.492 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0' 2024-09-18T08:12:26.526 DEBUG:teuthology.orchestra.run.smithi050:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid 779349ca-7593-11ef-bceb-c7b262605968 -- ceph daemon osd.5 dump_blocked_ops 2024-09-18T08:12:26.593 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0 2024-09-18T08:12:26.627 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:12:26.628 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:12:26.628 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0' 2024-09-18T08:12:26.729 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0 2024-09-18T08:12:26.751 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:12:26.752 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:12:26.752 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0' 2024-09-18T08:12:26.853 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6 2024-09-18T08:12:26.902 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:12:26.903 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:12:26.903 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6' 2024-09-18T08:12:26.905 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:12:26 smithi050 ceph-mon[32371]: pgmap v463: 158 pgs: 30 active+undersized, 34 active+undersized+degraded, 94 active+clean; 7.0 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 853 B/s rd, 69 MiB/s wr, 1.10k op/s; 64456/343254 objects degraded (18.778%) 2024-09-18T08:12:26.972 INFO:teuthology.orchestra.run.smithi050.stderr:Inferring config /var/lib/ceph/779349ca-7593-11ef-bceb-c7b262605968/mon.b/config 2024-09-18T08:12:26.980 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:12:26 smithi007 ceph-mon[29459]: pgmap v463: 158 pgs: 30 active+undersized, 34 active+undersized+degraded, 94 active+clean; 7.0 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 853 B/s rd, 69 MiB/s wr, 1.10k op/s; 64456/343254 objects degraded (18.778%) 2024-09-18T08:12:26.980 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:12:26 smithi007 ceph-mon[39249]: pgmap v463: 158 pgs: 30 active+undersized, 34 active+undersized+degraded, 94 active+clean; 7.0 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 853 B/s rd, 69 MiB/s wr, 1.10k op/s; 64456/343254 objects degraded (18.778%) 2024-09-18T08:12:26.980 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Sep 18 08:12:26 smithi007 ceph-779349ca-7593-11ef-bceb-c7b262605968-mgr-y[29688]: ::ffff:172.21.15.50 - - [18/Sep/2024:08:12:26] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-09-18T08:12:27.004 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6 2024-09-18T08:12:27.026 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:12:27.026 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:12:27.027 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6' 2024-09-18T08:12:27.127 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3 2024-09-18T08:12:27.164 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:12:27.164 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:12:27.165 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3' 2024-09-18T08:12:27.196 INFO:tasks.radosbench.radosbench.0.smithi007.stdout: 72 15 118704 118689 103.008 75.1875 0.0134511 0.00970367 2024-09-18T08:12:27.266 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7 2024-09-18T08:12:27.300 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:12:27.301 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:12:27.301 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7' 2024-09-18T08:12:27.402 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5 2024-09-18T08:12:27.445 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:12:27.446 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:12:27.447 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5' 2024-09-18T08:12:27.547 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4 2024-09-18T08:12:27.575 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2024-09-18T08:12:27.575 INFO:teuthology.orchestra.run.smithi050.stdout: "ops": [], 2024-09-18T08:12:27.575 INFO:teuthology.orchestra.run.smithi050.stdout: "complaint_time": 30, 2024-09-18T08:12:27.576 INFO:teuthology.orchestra.run.smithi050.stdout: "num_blocked_ops": 0 2024-09-18T08:12:27.576 INFO:teuthology.orchestra.run.smithi050.stdout:} 2024-09-18T08:12:27.583 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:12:27.584 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:12:27.584 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4' 2024-09-18T08:12:27.685 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7 2024-09-18T08:12:27.715 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:12:27.716 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:12:27.716 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7' 2024-09-18T08:12:27.732 INFO:teuthology.orchestra.run.smithi007.stdout: marking collection for removal 2024-09-18T08:12:27.732 INFO:teuthology.orchestra.run.smithi007.stdout:setting '_remove' omap key 2024-09-18T08:12:27.732 INFO:teuthology.orchestra.run.smithi007.stdout:finish_remove_pgs 4.f_head removing 4.f 2024-09-18T08:12:27.735 INFO:teuthology.orchestra.run.smithi007.stdout:Remove successful 2024-09-18T08:12:27.817 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7 2024-09-18T08:12:27.854 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:12:27.855 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:12:27.855 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7' 2024-09-18T08:12:27.956 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7 2024-09-18T08:12:28.001 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:12:28.002 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:12:28.002 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7' 2024-09-18T08:12:28.103 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5 2024-09-18T08:12:28.138 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:12:28.138 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:12:28.139 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5' 2024-09-18T08:12:28.196 INFO:tasks.radosbench.radosbench.0.smithi007.stdout: 73 16 120046 120030 102.745 83.8125 0.00744889 0.00972812 2024-09-18T08:12:28.240 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7 2024-09-18T08:12:28.279 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:12:28.280 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:12:28.280 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7' 2024-09-18T08:12:28.381 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6 2024-09-18T08:12:28.383 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-09-18T08:12:28.384 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-09-18T08:12:28.392 DEBUG:teuthology.orchestra.run.smithi050:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid 779349ca-7593-11ef-bceb-c7b262605968 -- ceph daemon osd.5 dump_historic_ops 2024-09-18T08:12:28.418 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:12:28.419 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:12:28.419 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6' 2024-09-18T08:12:28.521 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3 2024-09-18T08:12:28.550 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:12:28.550 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:12:28.551 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3' 2024-09-18T08:12:28.593 INFO:teuthology.orchestra.run.smithi007.stderr:noscrub is unset 2024-09-18T08:12:28.652 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3 2024-09-18T08:12:28.655 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:12:28 smithi050 ceph-mon[32371]: pgmap v464: 158 pgs: 30 active+undersized, 34 active+undersized+degraded, 94 active+clean; 7.0 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 640 B/s rd, 52 MiB/s wr, 825 op/s; 64456/343254 objects degraded (18.778%) 2024-09-18T08:12:28.655 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:12:28 smithi050 ceph-mon[32371]: from='client.? 172.21.15.7:0/2558416185' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-18T08:12:28.683 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:12:28.684 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:12:28.684 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3' 2024-09-18T08:12:28.785 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7 2024-09-18T08:12:28.834 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:12:28.834 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:12:28.835 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7' 2024-09-18T08:12:28.876 INFO:teuthology.orchestra.run.smithi050.stderr:Inferring config /var/lib/ceph/779349ca-7593-11ef-bceb-c7b262605968/mon.b/config 2024-09-18T08:12:28.936 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7 2024-09-18T08:12:28.973 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:12:28.973 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:12:28.974 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7' 2024-09-18T08:12:28.980 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:12:28 smithi007 ceph-mon[29459]: pgmap v464: 158 pgs: 30 active+undersized, 34 active+undersized+degraded, 94 active+clean; 7.0 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 640 B/s rd, 52 MiB/s wr, 825 op/s; 64456/343254 objects degraded (18.778%) 2024-09-18T08:12:28.980 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:12:28 smithi007 ceph-mon[29459]: from='client.? 172.21.15.7:0/2558416185' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-18T08:12:28.980 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:12:28 smithi007 ceph-mon[39249]: pgmap v464: 158 pgs: 30 active+undersized, 34 active+undersized+degraded, 94 active+clean; 7.0 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 640 B/s rd, 52 MiB/s wr, 825 op/s; 64456/343254 objects degraded (18.778%) 2024-09-18T08:12:28.980 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:12:28 smithi007 ceph-mon[39249]: from='client.? 172.21.15.7:0/2558416185' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-18T08:12:29.045 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell -n osd.1 --fsid 779349ca-7593-11ef-bceb-c7b262605968 -- ceph-objectstore-tool --err-to-stderr --data-path /var/lib/ceph/osd/ceph-1 --journal-path /var/lib/ceph/osd/ceph-1/journal '--log-file=/var/log/ceph/objectstore_tool.$pid.log' --op import --file /var/log/ceph/exp.4.f.1 2024-09-18T08:12:29.075 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6 2024-09-18T08:12:29.118 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:12:29.118 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:12:29.118 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6' 2024-09-18T08:12:29.196 INFO:tasks.radosbench.radosbench.0.smithi007.stdout: 74 16 121382 121366 102.485 83.5 0.0112381 0.00975309 2024-09-18T08:12:29.219 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0 2024-09-18T08:12:29.255 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:12:29.256 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:12:29.256 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0' 2024-09-18T08:12:29.357 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2 2024-09-18T08:12:29.390 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:12:29.392 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:12:29.392 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2' 2024-09-18T08:12:29.493 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3 2024-09-18T08:12:29.528 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:12:29.529 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:12:29.530 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3' 2024-09-18T08:12:29.530 INFO:journalctl@ceph.iscsi.iscsi.a.smithi050.stdout:Sep 18 08:12:29 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-iscsi-iscsi-a[57782]: debug there is no tcmu-runner data available 2024-09-18T08:12:29.630 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3 2024-09-18T08:12:29.661 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:12:29.661 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:12:29.662 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3' 2024-09-18T08:12:29.700 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2024-09-18T08:12:29.700 INFO:teuthology.orchestra.run.smithi050.stdout: "size": 20, 2024-09-18T08:12:29.700 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 600, 2024-09-18T08:12:29.700 INFO:teuthology.orchestra.run.smithi050.stdout: "ops": [ 2024-09-18T08:12:29.700 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:12:29.701 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.15216.0:22610 8.4 8:2876ee9f:::benchmark_data_smithi007_101583_object22609:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e174)", 2024-09-18T08:12:29.701 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-09-18T08:08:08.437530+0000", 2024-09-18T08:12:29.701 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 261.25758557799998, 2024-09-18T08:12:29.701 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.7699730649999998, 2024-09-18T08:12:29.701 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-09-18T08:12:29.701 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-09-18T08:12:29.702 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "reached pg", 2024-09-18T08:12:29.702 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2024-09-18T08:12:29.702 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.15216", 2024-09-18T08:12:29.702 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "172.21.15.7:0/2596041177", 2024-09-18T08:12:29.702 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 22610 2024-09-18T08:12:29.702 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:12:29.703 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-09-18T08:12:29.703 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:12:29.703 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-09-18T08:12:29.703 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:08:08.437530+0000", 2024-09-18T08:12:29.703 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-18T08:12:29.703 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:12:29.704 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:12:29.704 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-09-18T08:12:29.704 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:08:08.437530+0000", 2024-09-18T08:12:29.704 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-18T08:12:29.704 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:12:29.704 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:12:29.705 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-09-18T08:12:29.705 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:08:08.437531+0000", 2024-09-18T08:12:29.705 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.0890000000000001e-06 2024-09-18T08:12:29.705 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:12:29.705 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:12:29.705 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-09-18T08:12:29.706 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:08:08.437611+0000", 2024-09-18T08:12:29.706 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 7.9339999999999996e-05 2024-09-18T08:12:29.706 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:12:29.706 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:12:29.706 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-09-18T08:12:29.706 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:08:08.437623+0000", 2024-09-18T08:12:29.707 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.2442999999999999e-05 2024-09-18T08:12:29.707 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:12:29.707 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:12:29.707 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-09-18T08:12:29.707 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:08:08.437631+0000", 2024-09-18T08:12:29.707 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 8.0420000000000003e-06 2024-09-18T08:12:29.708 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:12:29.708 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:12:29.708 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-18T08:12:29.708 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:08:08.437684+0000", 2024-09-18T08:12:29.708 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 5.3430000000000002e-05 2024-09-18T08:12:29.708 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:12:29.709 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:12:29.709 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-09-18T08:12:29.709 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:08:08.437818+0000", 2024-09-18T08:12:29.709 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00013359400000000001 2024-09-18T08:12:29.709 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:12:29.710 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:12:29.710 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "waiting for subops from 2,6", 2024-09-18T08:12:29.710 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:08:08.438886+0000", 2024-09-18T08:12:29.710 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0010681 2024-09-18T08:12:29.710 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:12:29.710 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:12:29.711 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "op_commit", 2024-09-18T08:12:29.711 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:08:08.442178+0000", 2024-09-18T08:12:29.711 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0032919889999999999 2024-09-18T08:12:29.711 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:12:29.711 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:12:29.711 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-18T08:12:29.712 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:08:08.447653+0000", 2024-09-18T08:12:29.712 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0054751269999999998 2024-09-18T08:12:29.712 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:12:29.712 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:12:29.712 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "waiting for peered", 2024-09-18T08:12:29.713 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:08:08.447695+0000", 2024-09-18T08:12:29.713 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4.1869999999999997e-05 2024-09-18T08:12:29.713 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:12:29.713 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:12:29.714 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-18T08:12:29.714 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:08:09.153169+0000", 2024-09-18T08:12:29.714 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.70547423099999995 2024-09-18T08:12:29.714 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:12:29.714 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:12:29.714 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "waiting for peered", 2024-09-18T08:12:29.714 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:08:09.153246+0000", 2024-09-18T08:12:29.715 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 7.6285000000000006e-05 2024-09-18T08:12:29.715 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:12:29.715 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:12:29.715 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-18T08:12:29.715 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:08:10.207423+0000", 2024-09-18T08:12:29.715 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.0541769270000001 2024-09-18T08:12:29.716 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:12:29.716 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:12:29.716 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-09-18T08:12:29.716 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:08:10.207503+0000", 2024-09-18T08:12:29.716 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 8.0598000000000003e-05 2024-09-18T08:12:29.716 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-18T08:12:29.716 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-09-18T08:12:29.716 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-18T08:12:29.716 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:12:29.716 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:12:29.717 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.15216.0:22651 8.4 8:2e29b373:::benchmark_data_smithi007_101583_object22650:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e175)", 2024-09-18T08:12:29.717 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-09-18T08:08:08.478683+0000", 2024-09-18T08:12:29.717 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 261.21643312100002, 2024-09-18T08:12:29.717 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.7382695699999999, 2024-09-18T08:12:29.717 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-09-18T08:12:29.717 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-09-18T08:12:29.717 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T08:12:29.717 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2024-09-18T08:12:29.717 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.15216", 2024-09-18T08:12:29.717 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "172.21.15.7:0/2596041177", 2024-09-18T08:12:29.717 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 22651 2024-09-18T08:12:29.717 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:12:29.717 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-09-18T08:12:29.718 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:12:29.718 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-09-18T08:12:29.718 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:08:08.478683+0000", 2024-09-18T08:12:29.718 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-18T08:12:29.718 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:12:29.718 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:12:29.718 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-09-18T08:12:29.718 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:08:08.478683+0000", 2024-09-18T08:12:29.718 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-18T08:12:29.718 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:12:29.718 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:12:29.718 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-09-18T08:12:29.718 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:08:08.478683+0000", 2024-09-18T08:12:29.719 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 6.3099999999999997e-07 2024-09-18T08:12:29.719 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:12:29.719 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:12:29.719 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-09-18T08:12:29.719 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:08:08.478752+0000", 2024-09-18T08:12:29.719 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 6.9239999999999994e-05 2024-09-18T08:12:29.719 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:12:29.719 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:12:29.719 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-09-18T08:12:29.719 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:08:08.478761+0000", 2024-09-18T08:12:29.719 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 8.7979999999999995e-06 2024-09-18T08:12:29.719 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:12:29.719 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:12:29.720 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-09-18T08:12:29.720 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:08:08.478768+0000", 2024-09-18T08:12:29.720 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 6.7680000000000003e-06 2024-09-18T08:12:29.720 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:12:29.720 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:12:29.720 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-18T08:12:29.720 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:08:08.478835+0000", 2024-09-18T08:12:29.720 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 6.6914000000000003e-05 2024-09-18T08:12:29.720 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:12:29.720 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:12:29.720 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "waiting for peered", 2024-09-18T08:12:29.720 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:08:08.478845+0000", 2024-09-18T08:12:29.720 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 9.9969999999999993e-06 2024-09-18T08:12:29.720 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:12:29.721 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:12:29.721 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-18T08:12:29.721 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:08:09.155047+0000", 2024-09-18T08:12:29.721 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.67620250999999998 2024-09-18T08:12:29.721 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:12:29.721 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:12:29.721 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "waiting for peered", 2024-09-18T08:12:29.721 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:08:09.155057+0000", 2024-09-18T08:12:29.721 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 9.2739999999999996e-06 2024-09-18T08:12:29.721 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:12:29.721 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:12:29.721 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-18T08:12:29.721 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:08:10.208122+0000", 2024-09-18T08:12:29.721 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.0530651179999999 2024-09-18T08:12:29.722 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:12:29.722 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:12:29.722 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-09-18T08:12:29.722 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:08:10.208293+0000", 2024-09-18T08:12:29.722 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00017156699999999999 2024-09-18T08:12:29.722 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:12:29.722 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:12:29.722 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "waiting for subops from 3,6", 2024-09-18T08:12:29.722 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:08:10.208594+0000", 2024-09-18T08:12:29.722 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00030062200000000001 2024-09-18T08:12:29.722 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:12:29.722 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:12:29.722 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:12:29.722 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:08:10.213477+0000", 2024-09-18T08:12:29.723 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0048826920000000001 2024-09-18T08:12:29.723 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:12:29.723 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:12:29.723 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "op_commit", 2024-09-18T08:12:29.724 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:12:29 smithi007 ceph-mon[29459]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-18T08:12:29.724 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:12:29 smithi007 ceph-mon[29459]: from='client.? 172.21.15.7:0/2558416185' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-18T08:12:29.724 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:12:29 smithi007 ceph-mon[29459]: osdmap e260: 8 total, 7 up, 8 in 2024-09-18T08:12:29.724 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:12:29 smithi007 ceph-mon[29459]: 1.6 scrub starts 2024-09-18T08:12:29.724 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:12:29 smithi007 ceph-mon[29459]: 1.6 scrub ok 2024-09-18T08:12:29.724 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:12:29 smithi007 ceph-mon[29459]: Health check update: Degraded data redundancy: 64456/343254 objects degraded (18.778%), 34 pgs degraded (PG_DEGRADED) 2024-09-18T08:12:29.724 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:12:29 smithi007 ceph-mon[29459]: osdmap e261: 8 total, 7 up, 8 in 2024-09-18T08:12:29.725 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:12:29 smithi007 ceph-mon[39249]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-18T08:12:29.725 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:12:29 smithi007 ceph-mon[39249]: from='client.? 172.21.15.7:0/2558416185' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-18T08:12:29.725 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:12:29 smithi007 ceph-mon[39249]: osdmap e260: 8 total, 7 up, 8 in 2024-09-18T08:12:29.725 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:12:29 smithi007 ceph-mon[39249]: 1.6 scrub starts 2024-09-18T08:12:29.725 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:12:29 smithi007 ceph-mon[39249]: 1.6 scrub ok 2024-09-18T08:12:29.725 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:12:29 smithi007 ceph-mon[39249]: Health check update: Degraded data redundancy: 64456/343254 objects degraded (18.778%), 34 pgs degraded (PG_DEGRADED) 2024-09-18T08:12:29.725 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:12:29 smithi007 ceph-mon[39249]: osdmap e261: 8 total, 7 up, 8 in 2024-09-18T08:12:29.725 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:08:10.213660+0000", 2024-09-18T08:12:29.725 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00018310599999999999 2024-09-18T08:12:29.725 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:12:29.725 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:12:29.726 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:12:29.726 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:08:10.215742+0000", 2024-09-18T08:12:29.726 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0020817399999999999 2024-09-18T08:12:29.726 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:12:29.726 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:12:29.726 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "commit_sent", 2024-09-18T08:12:29.726 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:08:10.215809+0000", 2024-09-18T08:12:29.726 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 6.7220999999999999e-05 2024-09-18T08:12:29.726 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:12:29.726 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:12:29.726 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-09-18T08:12:29.726 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:08:10.216952+0000", 2024-09-18T08:12:29.726 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0011433719999999999 2024-09-18T08:12:29.726 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-18T08:12:29.727 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-09-18T08:12:29.727 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-18T08:12:29.727 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:12:29.727 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:12:29.727 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.15216.0:22653 8.4 8:28542a20:::benchmark_data_smithi007_101583_object22652:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e175)", 2024-09-18T08:12:29.727 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-09-18T08:08:08.478788+0000", 2024-09-18T08:12:29.727 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 261.21632806000002, 2024-09-18T08:12:29.727 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.7390986750000001, 2024-09-18T08:12:29.727 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-09-18T08:12:29.727 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-09-18T08:12:29.727 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T08:12:29.727 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2024-09-18T08:12:29.727 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.15216", 2024-09-18T08:12:29.728 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "172.21.15.7:0/2596041177", 2024-09-18T08:12:29.728 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 22653 2024-09-18T08:12:29.728 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:12:29.728 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-09-18T08:12:29.728 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:12:29.728 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-09-18T08:12:29.728 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:08:08.478788+0000", 2024-09-18T08:12:29.728 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-18T08:12:29.728 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:12:29.728 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:12:29.728 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-09-18T08:12:29.728 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:08:08.478788+0000", 2024-09-18T08:12:29.728 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-18T08:12:29.728 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:12:29.729 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:12:29.729 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-09-18T08:12:29.729 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:08:08.478788+0000", 2024-09-18T08:12:29.729 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 8.2600000000000001e-07 2024-09-18T08:12:29.729 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:12:29.729 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:12:29.729 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-09-18T08:12:29.729 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:08:08.478822+0000", 2024-09-18T08:12:29.729 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 3.3343999999999998e-05 2024-09-18T08:12:29.729 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:12:29.729 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:12:29.729 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-09-18T08:12:29.729 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:08:08.478830+0000", 2024-09-18T08:12:29.730 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 8.4549999999999995e-06 2024-09-18T08:12:29.730 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:12:29.730 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:12:29.730 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-09-18T08:12:29.730 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:08:08.478837+0000", 2024-09-18T08:12:29.730 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 6.3199999999999996e-06 2024-09-18T08:12:29.730 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:12:29.730 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:12:29.731 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-18T08:12:29.731 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:08:08.478877+0000", 2024-09-18T08:12:29.731 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4.0157999999999997e-05 2024-09-18T08:12:29.731 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:12:29.731 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:12:29.731 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "waiting for peered", 2024-09-18T08:12:29.731 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:08:08.478882+0000", 2024-09-18T08:12:29.731 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 5.0520000000000001e-06 2024-09-18T08:12:29.731 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:12:29.732 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:12:29.732 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-18T08:12:29.732 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:08:09.155087+0000", 2024-09-18T08:12:29.732 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.67620514899999995 2024-09-18T08:12:29.732 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:12:29.732 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:12:29.732 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "waiting for peered", 2024-09-18T08:12:29.732 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:08:09.155095+0000", 2024-09-18T08:12:29.733 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 8.4039999999999995e-06 2024-09-18T08:12:29.733 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:12:29.733 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:12:29.733 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-18T08:12:29.733 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:08:10.209614+0000", 2024-09-18T08:12:29.733 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.0545187579999999 2024-09-18T08:12:29.733 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:12:29.733 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:12:29.733 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-09-18T08:12:29.733 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:08:10.209770+0000", 2024-09-18T08:12:29.733 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00015591699999999999 2024-09-18T08:12:29.734 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:12:29.734 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:12:29.734 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "waiting for subops from 3,6", 2024-09-18T08:12:29.734 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:08:10.209987+0000", 2024-09-18T08:12:29.734 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000216565 2024-09-18T08:12:29.734 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:12:29.734 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:12:29.734 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:12:29.734 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:08:10.213551+0000", 2024-09-18T08:12:29.734 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0035644280000000001 2024-09-18T08:12:29.734 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:12:29.734 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:12:29.735 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "op_commit", 2024-09-18T08:12:29.735 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:08:10.213670+0000", 2024-09-18T08:12:29.735 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00011882700000000001 2024-09-18T08:12:29.735 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:12:29.735 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:12:29.735 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:12:29.735 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:08:10.217201+0000", 2024-09-18T08:12:29.735 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.003531307 2024-09-18T08:12:29.735 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:12:29.735 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:12:29.735 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "commit_sent", 2024-09-18T08:12:29.735 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:08:10.217269+0000", 2024-09-18T08:12:29.735 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 6.7539999999999994e-05 2024-09-18T08:12:29.736 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:12:29.736 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:12:29.736 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-09-18T08:12:29.736 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:08:10.217886+0000", 2024-09-18T08:12:29.736 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00061762500000000001 2024-09-18T08:12:29.736 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-18T08:12:29.736 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-09-18T08:12:29.736 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-18T08:12:29.736 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:12:29.736 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:12:29.736 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.15216.0:22670 8.4 8:2c79527e:::benchmark_data_smithi007_101583_object22669:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e175)", 2024-09-18T08:12:29.736 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-09-18T08:08:08.488620+0000", 2024-09-18T08:12:29.736 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 261.2064962, 2024-09-18T08:12:29.737 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.7300631360000001, 2024-09-18T08:12:29.737 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-09-18T08:12:29.737 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-09-18T08:12:29.737 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T08:12:29.737 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2024-09-18T08:12:29.737 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.15216", 2024-09-18T08:12:29.737 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "172.21.15.7:0/2596041177", 2024-09-18T08:12:29.737 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 22670 2024-09-18T08:12:29.737 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:12:29.737 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-09-18T08:12:29.737 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:12:29.737 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-09-18T08:12:29.738 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:08:08.488620+0000", 2024-09-18T08:12:29.738 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-18T08:12:29.738 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:12:29.738 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:12:29.738 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-09-18T08:12:29.738 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:08:08.488620+0000", 2024-09-18T08:12:29.738 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-18T08:12:29.738 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:12:29.738 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:12:29.738 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-09-18T08:12:29.738 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:08:08.488621+0000", 2024-09-18T08:12:29.738 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.3820000000000001e-06 2024-09-18T08:12:29.738 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:12:29.738 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:12:29.739 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-09-18T08:12:29.739 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:08:08.488653+0000", 2024-09-18T08:12:29.739 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 3.1826999999999999e-05 2024-09-18T08:12:29.739 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:12:29.739 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:12:29.739 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-09-18T08:12:29.739 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:08:08.488667+0000", 2024-09-18T08:12:29.739 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.4042e-05 2024-09-18T08:12:29.739 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:12:29.739 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:12:29.739 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-09-18T08:12:29.739 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:08:08.488674+0000", 2024-09-18T08:12:29.740 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 7.1570000000000003e-06 2024-09-18T08:12:29.740 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:12:29.740 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:12:29.740 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-18T08:12:29.740 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:08:08.489170+0000", 2024-09-18T08:12:29.740 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00049649999999999998 2024-09-18T08:12:29.740 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:12:29.740 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:12:29.740 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "waiting for peered", 2024-09-18T08:12:29.740 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:08:08.489181+0000", 2024-09-18T08:12:29.740 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.0991e-05 2024-09-18T08:12:29.740 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:12:29.740 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:12:29.740 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-18T08:12:29.741 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:08:09.155124+0000", 2024-09-18T08:12:29.741 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.66594271500000002 2024-09-18T08:12:29.741 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:12:29.741 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:12:29.741 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "waiting for peered", 2024-09-18T08:12:29.741 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:08:09.155132+0000", 2024-09-18T08:12:29.741 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 7.8509999999999999e-06 2024-09-18T08:12:29.741 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:12:29.741 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:12:29.741 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-18T08:12:29.741 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:08:10.212427+0000", 2024-09-18T08:12:29.741 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.0572947880000001 2024-09-18T08:12:29.741 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:12:29.741 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:12:29.742 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-09-18T08:12:29.742 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:08:10.212575+0000", 2024-09-18T08:12:29.742 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00014837199999999999 2024-09-18T08:12:29.742 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:12:29.742 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:12:29.742 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "waiting for subops from 3,6", 2024-09-18T08:12:29.742 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:08:10.212772+0000", 2024-09-18T08:12:29.742 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00019651400000000001 2024-09-18T08:12:29.742 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:12:29.743 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:12:29.743 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "op_commit", 2024-09-18T08:12:29.743 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:08:10.214660+0000", 2024-09-18T08:12:29.743 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.001887854 2024-09-18T08:12:29.743 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:12:29.743 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:12:29.743 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:12:29.743 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:08:10.217118+0000", 2024-09-18T08:12:29.743 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0024580309999999998 2024-09-18T08:12:29.743 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:12:29.743 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:12:29.743 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:12:29.744 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:08:10.218023+0000", 2024-09-18T08:12:29.744 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00090551 2024-09-18T08:12:29.744 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:12:29.744 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:12:29.744 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "commit_sent", 2024-09-18T08:12:29.744 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:08:10.218067+0000", 2024-09-18T08:12:29.744 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4.3547999999999997e-05 2024-09-18T08:12:29.744 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:12:29.744 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:12:29.745 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-09-18T08:12:29.745 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:08:10.218683+0000", 2024-09-18T08:12:29.745 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00061605399999999999 2024-09-18T08:12:29.745 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-18T08:12:29.745 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-09-18T08:12:29.745 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-18T08:12:29.745 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:12:29.745 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:12:29.745 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.15216.0:22690 8.4 8:29940999:::benchmark_data_smithi007_101583_object22689:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e175)", 2024-09-18T08:12:29.745 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-09-18T08:08:08.501955+0000", 2024-09-18T08:12:29.745 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 261.19316105799999, 2024-09-18T08:12:29.745 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.717490398, 2024-09-18T08:12:29.746 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-09-18T08:12:29.746 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-09-18T08:12:29.746 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T08:12:29.746 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2024-09-18T08:12:29.746 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.15216", 2024-09-18T08:12:29.746 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "172.21.15.7:0/2596041177", 2024-09-18T08:12:29.746 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 22690 2024-09-18T08:12:29.746 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:12:29.746 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-09-18T08:12:29.746 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:12:29.746 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-09-18T08:12:29.746 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:08:08.501955+0000", 2024-09-18T08:12:29.746 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-18T08:12:29.747 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:12:29.747 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:12:29.747 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-09-18T08:12:29.747 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:08:08.501955+0000", 2024-09-18T08:12:29.747 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-18T08:12:29.747 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:12:29.747 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:12:29.747 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-09-18T08:12:29.747 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:08:08.501956+0000", 2024-09-18T08:12:29.747 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.4190000000000001e-06 2024-09-18T08:12:29.747 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:12:29.747 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:12:29.748 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-09-18T08:12:29.748 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:08:08.502052+0000", 2024-09-18T08:12:29.748 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 9.5538999999999994e-05 2024-09-18T08:12:29.748 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:12:29.748 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:12:29.748 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-09-18T08:12:29.748 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:08:08.502067+0000", 2024-09-18T08:12:29.748 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.5014999999999999e-05 2024-09-18T08:12:29.748 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:12:29.748 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:12:29.748 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-09-18T08:12:29.748 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:08:08.502075+0000", 2024-09-18T08:12:29.748 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 8.0029999999999994e-06 2024-09-18T08:12:29.749 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:12:29.749 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:12:29.749 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-18T08:12:29.749 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:08:08.502136+0000", 2024-09-18T08:12:29.749 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 6.1029999999999997e-05 2024-09-18T08:12:29.749 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:12:29.749 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:12:29.749 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "waiting for peered", 2024-09-18T08:12:29.749 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:08:08.502142+0000", 2024-09-18T08:12:29.749 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 6.5749999999999997e-06 2024-09-18T08:12:29.749 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:12:29.749 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:12:29.750 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-18T08:12:29.750 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:08:09.155157+0000", 2024-09-18T08:12:29.750 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.65301457600000001 2024-09-18T08:12:29.750 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:12:29.750 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:12:29.750 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "waiting for peered", 2024-09-18T08:12:29.750 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:08:09.155165+0000", 2024-09-18T08:12:29.750 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 7.7889999999999995e-06 2024-09-18T08:12:29.750 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:12:29.750 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:12:29.750 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-18T08:12:29.750 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:08:10.213708+0000", 2024-09-18T08:12:29.750 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.0585432850000001 2024-09-18T08:12:29.751 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:12:29.751 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:12:29.751 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-09-18T08:12:29.751 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:08:10.213852+0000", 2024-09-18T08:12:29.751 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00014445899999999999 2024-09-18T08:12:29.751 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:12:29.751 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:12:29.751 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "waiting for subops from 3,6", 2024-09-18T08:12:29.751 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:08:10.214051+0000", 2024-09-18T08:12:29.751 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000198758 2024-09-18T08:12:29.751 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:12:29.751 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:12:29.751 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "op_commit", 2024-09-18T08:12:29.751 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:08:10.217140+0000", 2024-09-18T08:12:29.752 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0030884929999999999 2024-09-18T08:12:29.752 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:12:29.752 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:12:29.752 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:12:29.752 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:08:10.217957+0000", 2024-09-18T08:12:29.752 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00081764700000000001 2024-09-18T08:12:29.752 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:12:29.752 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:12:29.752 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:12:29.752 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:08:10.218885+0000", 2024-09-18T08:12:29.753 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00092746099999999998 2024-09-18T08:12:29.753 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:12:29.753 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:12:29.753 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "commit_sent", 2024-09-18T08:12:29.753 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:08:10.218945+0000", 2024-09-18T08:12:29.753 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 6.0149000000000003e-05 2024-09-18T08:12:29.753 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:12:29.753 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:12:29.753 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-09-18T08:12:29.753 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:08:10.219445+0000", 2024-09-18T08:12:29.754 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00050020000000000002 2024-09-18T08:12:29.754 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-18T08:12:29.754 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-09-18T08:12:29.754 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-18T08:12:29.754 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:12:29.754 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:12:29.754 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.15216.0:22694 8.4 8:285f3aa9:::benchmark_data_smithi007_101583_object22693:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e175)", 2024-09-18T08:12:29.754 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-09-18T08:08:08.509228+0000", 2024-09-18T08:12:29.754 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 261.18588815499999, 2024-09-18T08:12:29.754 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.710799596, 2024-09-18T08:12:29.754 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-09-18T08:12:29.754 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-09-18T08:12:29.754 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T08:12:29.754 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2024-09-18T08:12:29.755 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.15216", 2024-09-18T08:12:29.755 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "172.21.15.7:0/2596041177", 2024-09-18T08:12:29.755 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 22694 2024-09-18T08:12:29.755 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:12:29.755 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-09-18T08:12:29.755 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:12:29.755 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-09-18T08:12:29.755 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:08:08.509228+0000", 2024-09-18T08:12:29.755 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-18T08:12:29.755 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:12:29.755 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:12:29.755 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-09-18T08:12:29.755 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:08:08.509228+0000", 2024-09-18T08:12:29.756 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-18T08:12:29.756 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:12:29.756 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:12:29.756 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-09-18T08:12:29.756 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:08:08.509229+0000", 2024-09-18T08:12:29.756 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.6640000000000001e-06 2024-09-18T08:12:29.756 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:12:29.756 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:12:29.756 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-09-18T08:12:29.756 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:08:08.509266+0000", 2024-09-18T08:12:29.756 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 3.6563000000000002e-05 2024-09-18T08:12:29.756 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:12:29.756 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:12:29.756 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-09-18T08:12:29.757 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:08:08.509280+0000", 2024-09-18T08:12:29.757 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.4272e-05 2024-09-18T08:12:29.757 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:12:29.757 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:12:29.757 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-09-18T08:12:29.757 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:08:08.509288+0000", 2024-09-18T08:12:29.757 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 8.439e-06 2024-09-18T08:12:29.757 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:12:29.757 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:12:29.757 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-18T08:12:29.758 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:08:08.509344+0000", 2024-09-18T08:12:29.758 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 5.5538000000000003e-05 2024-09-18T08:12:29.758 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:12:29.758 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:12:29.758 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "waiting for peered", 2024-09-18T08:12:29.758 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:08:08.509352+0000", 2024-09-18T08:12:29.758 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 7.5329999999999997e-06 2024-09-18T08:12:29.758 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:12:29.758 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:12:29.758 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-18T08:12:29.758 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:08:09.155189+0000", 2024-09-18T08:12:29.758 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.64583694300000005 2024-09-18T08:12:29.758 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:12:29.759 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:12:29.759 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "waiting for peered", 2024-09-18T08:12:29.759 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:08:09.155196+0000", 2024-09-18T08:12:29.759 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 7.6760000000000007e-06 2024-09-18T08:12:29.759 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:12:29.759 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:12:29.759 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-18T08:12:29.759 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:08:10.214718+0000", 2024-09-18T08:12:29.759 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.059522045 2024-09-18T08:12:29.759 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:12:29.759 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:12:29.759 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-09-18T08:12:29.759 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:08:10.214859+0000", 2024-09-18T08:12:29.760 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000140922 2024-09-18T08:12:29.760 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:12:29.760 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:12:29.760 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "waiting for subops from 3,6", 2024-09-18T08:12:29.760 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:08:10.215042+0000", 2024-09-18T08:12:29.760 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000183141 2024-09-18T08:12:29.760 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:12:29.760 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:12:29.760 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "op_commit", 2024-09-18T08:12:29.760 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:08:10.218698+0000", 2024-09-18T08:12:29.760 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.003655845 2024-09-18T08:12:29.760 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:12:29.760 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:12:29.760 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:12:29.761 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:08:10.218758+0000", 2024-09-18T08:12:29.761 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 5.9463999999999999e-05 2024-09-18T08:12:29.761 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:12:29.761 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:12:29.761 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:12:29.761 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:08:10.219503+0000", 2024-09-18T08:12:29.761 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00074575399999999995 2024-09-18T08:12:29.761 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:12:29.761 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:12:29.761 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "commit_sent", 2024-09-18T08:12:29.761 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:08:10.219563+0000", 2024-09-18T08:12:29.761 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 5.9209999999999997e-05 2024-09-18T08:12:29.761 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:12:29.762 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:12:29.762 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-09-18T08:12:29.762 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:08:10.220027+0000", 2024-09-18T08:12:29.762 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00046458699999999998 2024-09-18T08:12:29.762 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-18T08:12:29.762 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-09-18T08:12:29.762 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-18T08:12:29.762 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:12:29.762 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:12:29.762 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.15216.0:69189 8.4 8:298d5120:::benchmark_data_smithi007_101583_object69188:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e198)", 2024-09-18T08:12:29.762 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-09-18T08:08:51.700999+0000", 2024-09-18T08:12:29.762 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 217.994116488, 2024-09-18T08:12:29.762 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 2.1566947330000001, 2024-09-18T08:12:29.763 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-09-18T08:12:29.763 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-09-18T08:12:29.763 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T08:12:29.763 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2024-09-18T08:12:29.763 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.15216", 2024-09-18T08:12:29.763 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "172.21.15.7:0/2596041177", 2024-09-18T08:12:29.763 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 69189 2024-09-18T08:12:29.763 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:12:29.763 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-09-18T08:12:29.763 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:12:29.764 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-09-18T08:12:29.764 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:08:51.700999+0000", 2024-09-18T08:12:29.764 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-18T08:12:29.764 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:12:29.764 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:12:29.764 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-09-18T08:12:29.764 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:08:51.700999+0000", 2024-09-18T08:12:29.764 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-18T08:12:29.764 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:12:29.764 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:12:29.764 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-09-18T08:12:29.764 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:08:51.701001+0000", 2024-09-18T08:12:29.764 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.9889999999999999e-06 2024-09-18T08:12:29.765 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:12:29.765 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:12:29.765 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-09-18T08:12:29.765 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:08:51.701054+0000", 2024-09-18T08:12:29.765 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 5.3019999999999997e-05 2024-09-18T08:12:29.765 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:12:29.765 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:12:29.765 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-09-18T08:12:29.765 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:08:51.701066+0000", 2024-09-18T08:12:29.765 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.204e-05 2024-09-18T08:12:29.765 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:12:29.765 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:12:29.766 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-09-18T08:12:29.766 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:08:51.701073+0000", 2024-09-18T08:12:29.766 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 6.5060000000000001e-06 2024-09-18T08:12:29.766 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:12:29.766 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:12:29.766 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-18T08:12:29.766 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:08:51.701115+0000", 2024-09-18T08:12:29.766 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4.2219999999999999e-05 2024-09-18T08:12:29.766 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:12:29.766 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:12:29.766 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "waiting for peered", 2024-09-18T08:12:29.767 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:08:51.701123+0000", 2024-09-18T08:12:29.767 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 8.3790000000000006e-06 2024-09-18T08:12:29.767 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:12:29.767 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:12:29.767 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-18T08:12:29.767 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:08:52.682893+0000", 2024-09-18T08:12:29.767 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.981770001 2024-09-18T08:12:29.767 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:12:29.767 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:12:29.767 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "waiting for peered", 2024-09-18T08:12:29.767 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:08:52.682917+0000", 2024-09-18T08:12:29.767 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 2.3329999999999999e-05 2024-09-18T08:12:29.767 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:12:29.767 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:12:29.768 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-18T08:12:29.768 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:08:53.834777+0000", 2024-09-18T08:12:29.768 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.151860171 2024-09-18T08:12:29.768 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:12:29.768 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:12:29.768 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-09-18T08:12:29.768 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:08:53.834944+0000", 2024-09-18T08:12:29.768 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00016724700000000001 2024-09-18T08:12:29.768 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:12:29.768 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:12:29.768 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "waiting for subops from 2,4", 2024-09-18T08:12:29.768 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:08:53.835122+0000", 2024-09-18T08:12:29.769 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00017796399999999999 2024-09-18T08:12:29.769 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:12:29.769 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:12:29.769 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:12:29.769 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:08:53.840827+0000", 2024-09-18T08:12:29.769 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0057047570000000004 2024-09-18T08:12:29.769 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:12:29.769 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:12:29.769 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "op_commit", 2024-09-18T08:12:29.769 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:08:53.840849+0000", 2024-09-18T08:12:29.769 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 2.1985e-05 2024-09-18T08:12:29.769 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:12:29.769 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:12:29.770 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:12:29.770 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:08:53.855347+0000", 2024-09-18T08:12:29.770 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.014498603000000001 2024-09-18T08:12:29.770 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:12:29.770 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:12:29.770 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "commit_sent", 2024-09-18T08:12:29.770 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:08:53.855403+0000", 2024-09-18T08:12:29.770 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 5.5977999999999998e-05 2024-09-18T08:12:29.770 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:12:29.770 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:12:29.770 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-09-18T08:12:29.770 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:08:53.857694+0000", 2024-09-18T08:12:29.770 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0022905429999999999 2024-09-18T08:12:29.770 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-18T08:12:29.771 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-09-18T08:12:29.771 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-18T08:12:29.771 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:12:29.771 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:12:29.771 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.15216.0:69191 8.4 8:2175794a:::benchmark_data_smithi007_101583_object69190:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e198)", 2024-09-18T08:12:29.771 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-09-18T08:08:51.705131+0000", 2024-09-18T08:12:29.771 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 217.98998499999999, 2024-09-18T08:12:29.771 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 2.1552133659999999, 2024-09-18T08:12:29.771 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0 2024-09-18T08:12:29.773 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-09-18T08:12:29.774 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-09-18T08:12:29.774 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T08:12:29.774 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2024-09-18T08:12:29.774 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.15216", 2024-09-18T08:12:29.774 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "172.21.15.7:0/2596041177", 2024-09-18T08:12:29.774 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 69191 2024-09-18T08:12:29.774 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:12:29.774 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-09-18T08:12:29.774 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:12:29.774 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-09-18T08:12:29.774 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:08:51.705131+0000", 2024-09-18T08:12:29.774 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-18T08:12:29.775 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:12:29.775 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:12:29.775 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-09-18T08:12:29.775 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:08:51.705131+0000", 2024-09-18T08:12:29.775 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-18T08:12:29.775 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:12:29.775 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:12:29.775 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-09-18T08:12:29.775 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:08:51.705138+0000", 2024-09-18T08:12:29.775 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 7.4560000000000003e-06 2024-09-18T08:12:29.775 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:12:29.775 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:12:29.775 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-09-18T08:12:29.775 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:08:51.705169+0000", 2024-09-18T08:12:29.776 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 3.1266999999999997e-05 2024-09-18T08:12:29.776 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:12:29.776 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:12:29.776 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-09-18T08:12:29.776 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:08:51.705181+0000", 2024-09-18T08:12:29.776 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.1610000000000001e-05 2024-09-18T08:12:29.776 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:12:29.776 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:12:29.776 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-09-18T08:12:29.776 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:08:51.705187+0000", 2024-09-18T08:12:29.776 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 5.5639999999999996e-06 2024-09-18T08:12:29.776 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:12:29.776 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:12:29.776 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-18T08:12:29.776 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:08:51.705248+0000", 2024-09-18T08:12:29.777 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 6.1258000000000002e-05 2024-09-18T08:12:29.777 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:12:29.777 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:12:29.777 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "waiting for peered", 2024-09-18T08:12:29.777 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:08:51.705260+0000", 2024-09-18T08:12:29.777 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.2305e-05 2024-09-18T08:12:29.777 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:12:29.777 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:12:29.777 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-18T08:12:29.777 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:08:52.683701+0000", 2024-09-18T08:12:29.777 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.97844060799999999 2024-09-18T08:12:29.777 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:12:29.778 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:12:29.778 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "waiting for peered", 2024-09-18T08:12:29.778 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:08:52.683710+0000", 2024-09-18T08:12:29.778 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 9.6760000000000002e-06 2024-09-18T08:12:29.778 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:12:29.778 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:12:29.778 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-18T08:12:29.778 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:08:53.838687+0000", 2024-09-18T08:12:29.778 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.1549762699999999 2024-09-18T08:12:29.778 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:12:29.778 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:12:29.778 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-09-18T08:12:29.778 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:08:53.838837+0000", 2024-09-18T08:12:29.779 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00015000300000000001 2024-09-18T08:12:29.779 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:12:29.779 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:12:29.779 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "waiting for subops from 2,4", 2024-09-18T08:12:29.779 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:08:53.838962+0000", 2024-09-18T08:12:29.779 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00012499200000000001 2024-09-18T08:12:29.779 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:12:29.779 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:12:29.779 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "op_commit", 2024-09-18T08:12:29.779 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:08:53.843446+0000", 2024-09-18T08:12:29.779 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0044846399999999998 2024-09-18T08:12:29.779 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:12:29.779 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:12:29.779 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:12:29.780 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:08:53.843632+0000", 2024-09-18T08:12:29.780 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00018546 2024-09-18T08:12:29.780 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:12:29.780 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:12:29.780 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:12:29.780 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:08:53.858138+0000", 2024-09-18T08:12:29.780 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.014506204 2024-09-18T08:12:29.780 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:12:29.780 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:12:29.780 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "commit_sent", 2024-09-18T08:12:29.780 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:08:53.858201+0000", 2024-09-18T08:12:29.780 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 6.2593000000000002e-05 2024-09-18T08:12:29.780 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:12:29.780 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:12:29.781 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-09-18T08:12:29.781 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:08:53.860344+0000", 2024-09-18T08:12:29.781 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0021434599999999998 2024-09-18T08:12:29.781 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-18T08:12:29.781 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-09-18T08:12:29.781 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-18T08:12:29.781 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:12:29.781 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:12:29.781 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.15216.0:69225 8.4 8:23f8fd92:::benchmark_data_smithi007_101583_object69224:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e198)", 2024-09-18T08:12:29.781 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-09-18T08:08:51.739781+0000", 2024-09-18T08:12:29.781 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 217.955334319, 2024-09-18T08:12:29.781 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 2.127987402, 2024-09-18T08:12:29.781 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-09-18T08:12:29.782 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-09-18T08:12:29.782 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T08:12:29.782 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2024-09-18T08:12:29.782 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.15216", 2024-09-18T08:12:29.782 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "172.21.15.7:0/2596041177", 2024-09-18T08:12:29.782 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 69225 2024-09-18T08:12:29.782 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:12:29.782 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-09-18T08:12:29.782 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:12:29.782 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-09-18T08:12:29.782 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:08:51.739781+0000", 2024-09-18T08:12:29.782 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-18T08:12:29.782 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:12:29.782 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:12:29.783 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-09-18T08:12:29.783 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:08:51.739781+0000", 2024-09-18T08:12:29.783 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-18T08:12:29.783 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:12:29.783 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:12:29.783 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-09-18T08:12:29.783 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:08:51.739782+0000", 2024-09-18T08:12:29.783 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.0550000000000001e-06 2024-09-18T08:12:29.783 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:12:29.783 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:12:29.783 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-09-18T08:12:29.783 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:08:51.739840+0000", 2024-09-18T08:12:29.783 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 5.7058999999999997e-05 2024-09-18T08:12:29.784 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:12:29.784 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:12:29.784 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-09-18T08:12:29.784 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:08:51.739850+0000", 2024-09-18T08:12:29.784 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.0926e-05 2024-09-18T08:12:29.784 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:12:29.784 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:12:29.784 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-09-18T08:12:29.785 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:08:51.739856+0000", 2024-09-18T08:12:29.785 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 5.5079999999999996e-06 2024-09-18T08:12:29.785 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:12:29.785 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:12:29.785 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-18T08:12:29.785 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:08:51.739923+0000", 2024-09-18T08:12:29.785 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 6.7543e-05 2024-09-18T08:12:29.785 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:12:29.785 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:12:29.785 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "waiting for peered", 2024-09-18T08:12:29.785 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:08:51.739936+0000", 2024-09-18T08:12:29.785 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.2027e-05 2024-09-18T08:12:29.785 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:12:29.785 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:12:29.785 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-18T08:12:29.786 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:08:52.683740+0000", 2024-09-18T08:12:29.786 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.94380415500000003 2024-09-18T08:12:29.786 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:12:29.786 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:12:29.786 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "waiting for peered", 2024-09-18T08:12:29.786 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:08:52.683748+0000", 2024-09-18T08:12:29.786 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 7.8390000000000007e-06 2024-09-18T08:12:29.786 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:12:29.786 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:12:29.786 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-18T08:12:29.786 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:08:53.840880+0000", 2024-09-18T08:12:29.786 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.157132941 2024-09-18T08:12:29.786 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:12:29.786 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:12:29.786 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-09-18T08:12:29.786 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:08:53.841039+0000", 2024-09-18T08:12:29.787 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00015902099999999999 2024-09-18T08:12:29.787 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:12:29.787 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:12:29.787 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "waiting for subops from 2,4", 2024-09-18T08:12:29.787 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:08:53.841222+0000", 2024-09-18T08:12:29.787 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000182215 2024-09-18T08:12:29.787 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:12:29.787 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:12:29.787 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:12:29.787 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:08:53.848840+0000", 2024-09-18T08:12:29.787 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0076179430000000003 2024-09-18T08:12:29.787 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:12:29.787 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:12:29.787 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "op_commit", 2024-09-18T08:12:29.787 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:08:53.848901+0000", 2024-09-18T08:12:29.787 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 6.1829999999999996e-05 2024-09-18T08:12:29.788 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:12:29.788 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:12:29.788 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:12:29.788 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:08:53.866059+0000", 2024-09-18T08:12:29.788 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.017157466999999999 2024-09-18T08:12:29.788 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:12:29.788 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:12:29.788 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "commit_sent", 2024-09-18T08:12:29.788 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:08:53.866105+0000", 2024-09-18T08:12:29.788 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4.6094000000000003e-05 2024-09-18T08:12:29.788 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:12:29.788 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:12:29.788 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-09-18T08:12:29.788 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:08:53.867769+0000", 2024-09-18T08:12:29.789 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.001663779 2024-09-18T08:12:29.789 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-18T08:12:29.789 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-09-18T08:12:29.789 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-18T08:12:29.789 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:12:29.789 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:12:29.789 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.15216.0:69249 8.4 8:21d6a20b:::benchmark_data_smithi007_101583_object69248:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e198)", 2024-09-18T08:12:29.789 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-09-18T08:08:51.761740+0000", 2024-09-18T08:12:29.789 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 217.93337568600001, 2024-09-18T08:12:29.789 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 2.1074109980000002, 2024-09-18T08:12:29.790 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-09-18T08:12:29.790 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-09-18T08:12:29.790 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T08:12:29.790 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2024-09-18T08:12:29.790 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.15216", 2024-09-18T08:12:29.790 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "172.21.15.7:0/2596041177", 2024-09-18T08:12:29.790 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 69249 2024-09-18T08:12:29.790 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:12:29.790 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-09-18T08:12:29.790 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:12:29.790 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-09-18T08:12:29.790 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:08:51.761740+0000", 2024-09-18T08:12:29.790 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-18T08:12:29.790 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:12:29.791 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:12:29.791 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-09-18T08:12:29.791 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:08:51.761740+0000", 2024-09-18T08:12:29.791 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-18T08:12:29.791 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:12:29.791 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:12:29.791 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-09-18T08:12:29.791 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:08:51.761741+0000", 2024-09-18T08:12:29.791 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.051e-06 2024-09-18T08:12:29.791 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:12:29.791 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:12:29.791 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-09-18T08:12:29.791 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:08:51.761813+0000", 2024-09-18T08:12:29.791 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 7.1748000000000005e-05 2024-09-18T08:12:29.792 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:12:29.792 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:12:29.792 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-09-18T08:12:29.792 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:08:51.761822+0000", 2024-09-18T08:12:29.792 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 9.6520000000000001e-06 2024-09-18T08:12:29.792 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:12:29.792 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:12:29.792 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-09-18T08:12:29.792 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:08:51.761828+0000", 2024-09-18T08:12:29.792 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 5.305e-06 2024-09-18T08:12:29.792 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:12:29.792 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:12:29.793 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-18T08:12:29.793 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:08:51.761893+0000", 2024-09-18T08:12:29.793 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 6.5613999999999999e-05 2024-09-18T08:12:29.793 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:12:29.793 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:12:29.793 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "waiting for peered", 2024-09-18T08:12:29.793 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:08:51.761919+0000", 2024-09-18T08:12:29.793 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 2.567e-05 2024-09-18T08:12:29.793 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:12:29.793 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:12:29.793 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-18T08:12:29.793 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:08:52.683789+0000", 2024-09-18T08:12:29.793 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.92186995000000005 2024-09-18T08:12:29.794 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:12:29.794 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:12:29.794 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "waiting for peered", 2024-09-18T08:12:29.794 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:08:52.683797+0000", 2024-09-18T08:12:29.794 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 7.9079999999999995e-06 2024-09-18T08:12:29.794 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:12:29.794 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:12:29.794 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-18T08:12:29.795 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:08:53.843803+0000", 2024-09-18T08:12:29.795 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.1600056759999999 2024-09-18T08:12:29.795 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:12:29.795 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:12:29.795 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-09-18T08:12:29.795 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:08:53.843954+0000", 2024-09-18T08:12:29.795 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000151037 2024-09-18T08:12:29.795 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:12:29.795 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:12:29.795 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "waiting for subops from 2,4", 2024-09-18T08:12:29.796 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:08:53.844132+0000", 2024-09-18T08:12:29.796 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00017874000000000001 2024-09-18T08:12:29.796 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:12:29.796 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:12:29.796 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:12:29.796 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:08:53.848886+0000", 2024-09-18T08:12:29.796 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0047536779999999999 2024-09-18T08:12:29.796 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:12:29.796 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:12:29.796 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "op_commit", 2024-09-18T08:12:29.796 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:08:53.848906+0000", 2024-09-18T08:12:29.796 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.9758999999999998e-05 2024-09-18T08:12:29.797 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:12:29.797 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:12:29.797 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:12:29.797 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:08:53.867840+0000", 2024-09-18T08:12:29.797 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.018934359000000001 2024-09-18T08:12:29.797 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:12:29.797 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:12:29.797 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "commit_sent", 2024-09-18T08:12:29.797 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:08:53.867882+0000", 2024-09-18T08:12:29.797 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4.1526000000000002e-05 2024-09-18T08:12:29.797 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:12:29.798 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:12:29.798 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-09-18T08:12:29.798 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:08:53.869151+0000", 2024-09-18T08:12:29.798 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.001269325 2024-09-18T08:12:29.798 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-18T08:12:29.798 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-09-18T08:12:29.798 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-18T08:12:29.798 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:12:29.798 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:12:29.798 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.15216.0:69269 8.4 8:2819c4dc:::benchmark_data_smithi007_101583_object69268:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e198)", 2024-09-18T08:12:29.798 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-09-18T08:08:51.782860+0000", 2024-09-18T08:12:29.798 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 217.91225589699999, 2024-09-18T08:12:29.798 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 2.0970977999999998, 2024-09-18T08:12:29.799 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-09-18T08:12:29.799 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-09-18T08:12:29.799 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T08:12:29.799 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2024-09-18T08:12:29.799 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.15216", 2024-09-18T08:12:29.799 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "172.21.15.7:0/2596041177", 2024-09-18T08:12:29.799 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 69269 2024-09-18T08:12:29.799 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:12:29.799 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-09-18T08:12:29.799 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:12:29.799 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-09-18T08:12:29.799 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:08:51.782860+0000", 2024-09-18T08:12:29.799 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-18T08:12:29.800 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:12:29.800 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:12:29.800 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-09-18T08:12:29.800 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:08:51.782860+0000", 2024-09-18T08:12:29.800 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-18T08:12:29.800 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:12:29.800 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:12:29.800 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-09-18T08:12:29.800 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:08:51.782861+0000", 2024-09-18T08:12:29.800 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.3039999999999999e-06 2024-09-18T08:12:29.800 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:12:29.800 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:12:29.800 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-09-18T08:12:29.801 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:08:51.782932+0000", 2024-09-18T08:12:29.801 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 7.1074000000000004e-05 2024-09-18T08:12:29.801 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:12:29.801 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:12:29.801 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-09-18T08:12:29.801 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:08:51.782944+0000", 2024-09-18T08:12:29.801 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.2164000000000001e-05 2024-09-18T08:12:29.801 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:12:29.801 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:12:29.801 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-09-18T08:12:29.801 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:08:51.782952+0000", 2024-09-18T08:12:29.801 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 7.1559999999999998e-06 2024-09-18T08:12:29.802 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:12:29.802 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:12:29.802 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-18T08:12:29.802 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:08:51.783016+0000", 2024-09-18T08:12:29.802 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 6.4747999999999997e-05 2024-09-18T08:12:29.802 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:12:29.802 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:12:29.802 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "waiting for peered", 2024-09-18T08:12:29.802 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:08:51.783028+0000", 2024-09-18T08:12:29.802 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.1463e-05 2024-09-18T08:12:29.802 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:12:29.802 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:12:29.802 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-18T08:12:29.802 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:08:52.683825+0000", 2024-09-18T08:12:29.803 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.90079731799999996 2024-09-18T08:12:29.803 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:12:29.803 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:12:29.803 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "waiting for peered", 2024-09-18T08:12:29.803 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:08:52.683833+0000", 2024-09-18T08:12:29.803 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 7.763e-06 2024-09-18T08:12:29.803 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:12:29.803 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:12:29.803 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-18T08:12:29.803 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:08:53.846992+0000", 2024-09-18T08:12:29.803 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.163159319 2024-09-18T08:12:29.803 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:12:29.803 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:12:29.804 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-09-18T08:12:29.804 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:08:53.847095+0000", 2024-09-18T08:12:29.804 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000102392 2024-09-18T08:12:29.804 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:12:29.804 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:12:29.804 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "waiting for subops from 2,4", 2024-09-18T08:12:29.804 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:08:53.847204+0000", 2024-09-18T08:12:29.804 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00010955000000000001 2024-09-18T08:12:29.804 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:12:29.804 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:12:29.804 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:12:29.804 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:08:53.853079+0000", 2024-09-18T08:12:29.804 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0058749609999999997 2024-09-18T08:12:29.805 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:12:29.805 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:12:29.805 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "op_commit", 2024-09-18T08:12:29.805 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:08:53.853167+0000", 2024-09-18T08:12:29.805 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 8.7645000000000005e-05 2024-09-18T08:12:29.805 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:12:29.805 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:12:29.805 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:12:29.805 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:08:53.878127+0000", 2024-09-18T08:12:29.805 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.024960336 2024-09-18T08:12:29.806 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:12:29.806 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:12:29.806 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "commit_sent", 2024-09-18T08:12:29.806 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:08:53.878167+0000", 2024-09-18T08:12:29.806 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 3.9728999999999999e-05 2024-09-18T08:12:29.806 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:12:29.806 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:12:29.806 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-09-18T08:12:29.806 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:08:53.879958+0000", 2024-09-18T08:12:29.806 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0017908780000000001 2024-09-18T08:12:29.806 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-18T08:12:29.806 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-09-18T08:12:29.806 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-18T08:12:29.807 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:12:29.807 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:12:29.807 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.15216.0:69283 8.4 8:2031b74c:::benchmark_data_smithi007_101583_object69282:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e198)", 2024-09-18T08:12:29.807 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-09-18T08:08:51.806433+0000", 2024-09-18T08:12:29.807 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 217.888682929, 2024-09-18T08:12:29.807 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 2.086781915, 2024-09-18T08:12:29.807 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-09-18T08:12:29.807 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-09-18T08:12:29.807 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T08:12:29.807 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2024-09-18T08:12:29.807 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.15216", 2024-09-18T08:12:29.807 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "172.21.15.7:0/2596041177", 2024-09-18T08:12:29.807 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 69283 2024-09-18T08:12:29.807 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:12:29.808 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-09-18T08:12:29.808 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:12:29.808 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-09-18T08:12:29.808 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:08:51.806433+0000", 2024-09-18T08:12:29.808 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-18T08:12:29.808 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:12:29.808 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:12:29.808 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-09-18T08:12:29.808 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:08:51.806433+0000", 2024-09-18T08:12:29.808 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-18T08:12:29.808 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:12:29.808 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:12:29.808 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-09-18T08:12:29.808 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:08:51.806434+0000", 2024-09-18T08:12:29.809 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.341e-06 2024-09-18T08:12:29.809 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:12:29.809 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:12:29.809 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-09-18T08:12:29.809 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:08:51.806474+0000", 2024-09-18T08:12:29.809 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 3.9456999999999998e-05 2024-09-18T08:12:29.809 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:12:29.809 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:12:29.809 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-09-18T08:12:29.809 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:08:51.806486+0000", 2024-09-18T08:12:29.809 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.2619999999999999e-05 2024-09-18T08:12:29.809 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:12:29.809 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:12:29.810 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-09-18T08:12:29.810 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:08:51.806494+0000", 2024-09-18T08:12:29.810 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 7.3019999999999997e-06 2024-09-18T08:12:29.810 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:12:29.810 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:12:29.810 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-18T08:12:29.810 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:08:51.806557+0000", 2024-09-18T08:12:29.810 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 6.3452000000000002e-05 2024-09-18T08:12:29.810 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:12:29.810 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:12:29.810 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "waiting for peered", 2024-09-18T08:12:29.810 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:08:51.806568+0000", 2024-09-18T08:12:29.810 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.0671e-05 2024-09-18T08:12:29.810 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:12:29.810 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:12:29.811 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-18T08:12:29.811 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:08:52.683860+0000", 2024-09-18T08:12:29.811 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.87729208700000005 2024-09-18T08:12:29.811 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:12:29.811 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:12:29.811 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "waiting for peered", 2024-09-18T08:12:29.811 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:08:52.683868+0000", 2024-09-18T08:12:29.811 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 7.8050000000000007e-06 2024-09-18T08:12:29.811 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:12:29.811 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:12:29.811 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-18T08:12:29.812 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:08:53.848952+0000", 2024-09-18T08:12:29.812 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.1650841270000001 2024-09-18T08:12:29.812 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:12:29.812 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:12:29.812 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-09-18T08:12:29.812 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:08:53.849101+0000", 2024-09-18T08:12:29.812 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00014899999999999999 2024-09-18T08:12:29.812 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:12:29.812 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:12:29.812 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "waiting for subops from 2,4", 2024-09-18T08:12:29.812 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:08:53.849292+0000", 2024-09-18T08:12:29.812 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000191349 2024-09-18T08:12:29.812 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:12:29.812 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:12:29.813 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:12:29.813 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:08:53.853148+0000", 2024-09-18T08:12:29.813 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0038562010000000001 2024-09-18T08:12:29.813 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:12:29.813 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:12:29.813 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "op_commit", 2024-09-18T08:12:29.813 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:08:53.853173+0000", 2024-09-18T08:12:29.813 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 2.4482e-05 2024-09-18T08:12:29.813 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:12:29.814 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:12:29.814 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:12:29.814 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:08:53.891653+0000", 2024-09-18T08:12:29.814 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.038480225 2024-09-18T08:12:29.814 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:12:29.814 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:12:29.814 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "commit_sent", 2024-09-18T08:12:29.814 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:08:53.891697+0000", 2024-09-18T08:12:29.814 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4.4171000000000001e-05 2024-09-18T08:12:29.814 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:12:29.814 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:12:29.814 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-09-18T08:12:29.814 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:08:53.893215+0000", 2024-09-18T08:12:29.814 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0015176250000000001 2024-09-18T08:12:29.814 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-18T08:12:29.815 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-09-18T08:12:29.815 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-18T08:12:29.815 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:12:29.815 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:12:29.815 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.15216.0:69323 8.4 8:2053bc8e:::benchmark_data_smithi007_101583_object69322:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e198)", 2024-09-18T08:12:29.815 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-09-18T08:08:51.863425+0000", 2024-09-18T08:12:29.815 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 217.83169100500001, 2024-09-18T08:12:29.815 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 2.0310337920000001, 2024-09-18T08:12:29.815 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-09-18T08:12:29.815 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-09-18T08:12:29.815 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T08:12:29.816 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2024-09-18T08:12:29.816 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.15216", 2024-09-18T08:12:29.816 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "172.21.15.7:0/2596041177", 2024-09-18T08:12:29.816 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 69323 2024-09-18T08:12:29.816 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:12:29.816 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-09-18T08:12:29.816 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:12:29.816 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-09-18T08:12:29.816 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:08:51.863425+0000", 2024-09-18T08:12:29.816 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-18T08:12:29.816 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:12:29.816 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:12:29.816 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-09-18T08:12:29.816 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:08:51.863425+0000", 2024-09-18T08:12:29.817 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-18T08:12:29.817 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:12:29.817 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:12:29.817 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-09-18T08:12:29.817 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:08:51.863427+0000", 2024-09-18T08:12:29.817 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.846e-06 2024-09-18T08:12:29.817 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:12:29.817 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:12:29.817 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-09-18T08:12:29.817 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:08:51.863472+0000", 2024-09-18T08:12:29.817 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4.5639000000000001e-05 2024-09-18T08:12:29.817 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:12:29.817 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:12:29.817 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-09-18T08:12:29.818 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:08:51.863487+0000", 2024-09-18T08:12:29.818 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.4688999999999999e-05 2024-09-18T08:12:29.818 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:12:29.818 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:12:29.818 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-09-18T08:12:29.818 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:08:51.863495+0000", 2024-09-18T08:12:29.818 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 7.6850000000000001e-06 2024-09-18T08:12:29.818 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:12:29.818 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:12:29.818 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-18T08:12:29.818 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:08:51.863561+0000", 2024-09-18T08:12:29.818 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 6.6688999999999998e-05 2024-09-18T08:12:29.818 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:12:29.818 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:12:29.818 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "waiting for peered", 2024-09-18T08:12:29.819 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:08:51.863573+0000", 2024-09-18T08:12:29.819 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.1663e-05 2024-09-18T08:12:29.819 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:12:29.819 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:12:29.819 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-18T08:12:29.819 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:08:52.683896+0000", 2024-09-18T08:12:29.819 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.82032282700000003 2024-09-18T08:12:29.819 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:12:29.819 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:12:29.819 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "waiting for peered", 2024-09-18T08:12:29.819 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:08:52.683910+0000", 2024-09-18T08:12:29.819 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.4576000000000001e-05 2024-09-18T08:12:29.819 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:12:29.820 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:12:29.820 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-18T08:12:29.820 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:08:53.851037+0000", 2024-09-18T08:12:29.820 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.1671269989999999 2024-09-18T08:12:29.820 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:12:29.820 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:12:29.820 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-09-18T08:12:29.820 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:08:53.851121+0000", 2024-09-18T08:12:29.820 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 8.3844999999999994e-05 2024-09-18T08:12:29.820 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:12:29.820 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:12:29.820 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "waiting for subops from 2,4", 2024-09-18T08:12:29.820 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:08:53.851225+0000", 2024-09-18T08:12:29.820 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000103806 2024-09-18T08:12:29.821 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:12:29.821 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:12:29.821 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:12:29.821 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:08:53.860401+0000", 2024-09-18T08:12:29.821 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0091763390000000004 2024-09-18T08:12:29.821 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:12:29.821 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:12:29.821 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "op_commit", 2024-09-18T08:12:29.821 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:08:53.860538+0000", 2024-09-18T08:12:29.821 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00013657 2024-09-18T08:12:29.821 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:12:29.821 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:12:29.821 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:12:29.822 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:08:53.893287+0000", 2024-09-18T08:12:29.822 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.032748691000000003 2024-09-18T08:12:29.822 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:12:29.822 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:12:29.822 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "commit_sent", 2024-09-18T08:12:29.822 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:08:53.893357+0000", 2024-09-18T08:12:29.822 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 7.0751000000000001e-05 2024-09-18T08:12:29.822 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:12:29.822 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:12:29.822 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-09-18T08:12:29.822 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:08:53.894459+0000", 2024-09-18T08:12:29.822 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0011011770000000001 2024-09-18T08:12:29.822 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-18T08:12:29.822 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-09-18T08:12:29.823 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-18T08:12:29.823 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:12:29.823 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:12:29.823 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.15216.0:69346 8.4 8:28700287:::benchmark_data_smithi007_101583_object69345:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e198)", 2024-09-18T08:12:29.823 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-09-18T08:08:51.900473+0000", 2024-09-18T08:12:29.823 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 217.79464262400001, 2024-09-18T08:12:29.823 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 2.013468198, 2024-09-18T08:12:29.823 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-09-18T08:12:29.823 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-09-18T08:12:29.823 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T08:12:29.823 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2024-09-18T08:12:29.823 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.15216", 2024-09-18T08:12:29.823 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "172.21.15.7:0/2596041177", 2024-09-18T08:12:29.823 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 69346 2024-09-18T08:12:29.824 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:12:29.824 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-09-18T08:12:29.824 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:12:29.824 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-09-18T08:12:29.824 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:08:51.900473+0000", 2024-09-18T08:12:29.824 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-18T08:12:29.824 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:12:29.824 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:12:29.824 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-09-18T08:12:29.824 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:08:51.900473+0000", 2024-09-18T08:12:29.824 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-18T08:12:29.824 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:12:29.824 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:12:29.825 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-09-18T08:12:29.825 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:08:51.900474+0000", 2024-09-18T08:12:29.825 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.3060000000000001e-06 2024-09-18T08:12:29.825 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:12:29.825 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:12:29.825 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-09-18T08:12:29.825 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:08:51.900534+0000", 2024-09-18T08:12:29.825 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 5.9429000000000003e-05 2024-09-18T08:12:29.825 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:12:29.825 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:12:29.826 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-09-18T08:12:29.826 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:08:51.900547+0000", 2024-09-18T08:12:29.826 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.2716e-05 2024-09-18T08:12:29.826 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:12:29.826 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:12:29.826 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-09-18T08:12:29.826 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:08:51.900555+0000", 2024-09-18T08:12:29.826 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 8.0390000000000005e-06 2024-09-18T08:12:29.826 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:12:29.826 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:12:29.827 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-18T08:12:29.827 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:08:51.900615+0000", 2024-09-18T08:12:29.827 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 6.0646000000000002e-05 2024-09-18T08:12:29.827 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:12:29.827 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:12:29.827 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "waiting for peered", 2024-09-18T08:12:29.827 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:08:51.900621+0000", 2024-09-18T08:12:29.827 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 6.1820000000000003e-06 2024-09-18T08:12:29.827 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:12:29.827 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:12:29.827 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-18T08:12:29.827 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:08:52.683937+0000", 2024-09-18T08:12:29.827 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.78331511700000001 2024-09-18T08:12:29.827 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:12:29.828 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:12:29.828 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "waiting for peered", 2024-09-18T08:12:29.828 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:08:52.683945+0000", 2024-09-18T08:12:29.828 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 8.0630000000000006e-06 2024-09-18T08:12:29.828 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:12:29.828 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:12:29.828 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-18T08:12:29.828 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:08:53.853192+0000", 2024-09-18T08:12:29.828 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.169247726 2024-09-18T08:12:29.828 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:12:29.828 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:12:29.828 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-09-18T08:12:29.828 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:08:53.853272+0000", 2024-09-18T08:12:29.828 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 8.0107999999999999e-05 2024-09-18T08:12:29.829 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:12:29.829 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:12:29.829 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "waiting for subops from 2,4", 2024-09-18T08:12:29.829 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:08:53.853369+0000", 2024-09-18T08:12:29.829 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 9.6391000000000006e-05 2024-09-18T08:12:29.829 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:12:29.829 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:12:29.829 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:12:29.829 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:08:53.860443+0000", 2024-09-18T08:12:29.829 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0070746749999999999 2024-09-18T08:12:29.829 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:12:29.829 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:12:29.829 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "op_commit", 2024-09-18T08:12:29.829 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:08:53.860549+0000", 2024-09-18T08:12:29.830 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000105679 2024-09-18T08:12:29.830 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:12:29.830 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:12:29.830 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:12:29.830 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:08:53.912224+0000", 2024-09-18T08:12:29.830 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.051674987999999998 2024-09-18T08:12:29.830 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:12:29.830 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:12:29.830 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "commit_sent", 2024-09-18T08:12:29.830 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:08:53.912271+0000", 2024-09-18T08:12:29.830 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4.6724000000000002e-05 2024-09-18T08:12:29.830 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:12:29.830 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:12:29.831 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-09-18T08:12:29.831 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:08:53.913941+0000", 2024-09-18T08:12:29.831 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0016704090000000001 2024-09-18T08:12:29.831 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-18T08:12:29.831 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-09-18T08:12:29.831 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-18T08:12:29.831 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:12:29.831 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:12:29.831 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.15216.0:151197 8.4 8:28e40b7f:::benchmark_data_smithi007_101583_object50445:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e218)", 2024-09-18T08:12:29.832 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-09-18T08:10:14.236495+0000", 2024-09-18T08:12:29.832 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 135.45862108, 2024-09-18T08:12:29.832 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.027586009, 2024-09-18T08:12:29.832 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-09-18T08:12:29.832 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-09-18T08:12:29.832 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "reached pg", 2024-09-18T08:12:29.833 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2024-09-18T08:12:29.833 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.15216", 2024-09-18T08:12:29.833 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "172.21.15.7:0/2596041177", 2024-09-18T08:12:29.835 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/779349ca-7593-11ef-bceb-c7b262605968/osd.1/config 2024-09-18T08:12:29.835 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:12:29 smithi050 ceph-mon[32371]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-18T08:12:29.835 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:12:29 smithi050 ceph-mon[32371]: from='client.? 172.21.15.7:0/2558416185' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-18T08:12:29.835 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:12:29 smithi050 ceph-mon[32371]: osdmap e260: 8 total, 7 up, 8 in 2024-09-18T08:12:29.835 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:12:29 smithi050 ceph-mon[32371]: 1.6 scrub starts 2024-09-18T08:12:29.835 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:12:29 smithi050 ceph-mon[32371]: 1.6 scrub ok 2024-09-18T08:12:29.836 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:12:29 smithi050 ceph-mon[32371]: Health check update: Degraded data redundancy: 64456/343254 objects degraded (18.778%), 34 pgs degraded (PG_DEGRADED) 2024-09-18T08:12:29.836 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:12:29 smithi050 ceph-mon[32371]: osdmap e261: 8 total, 7 up, 8 in 2024-09-18T08:12:29.836 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 151197 2024-09-18T08:12:29.837 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:12:29.837 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-09-18T08:12:29.837 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:12:29.837 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-09-18T08:12:29.837 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:10:14.236495+0000", 2024-09-18T08:12:29.838 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-18T08:12:29.838 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:12:29.838 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:12:29.838 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-09-18T08:12:29.838 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:10:14.236495+0000", 2024-09-18T08:12:29.838 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-18T08:12:29.838 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:12:29.839 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:12:29.839 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-09-18T08:12:29.839 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:10:14.236495+0000", 2024-09-18T08:12:29.839 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4.3099999999999998e-07 2024-09-18T08:12:29.839 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:12:29.839 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:12:29.840 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-09-18T08:12:29.840 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:10:14.236497+0000", 2024-09-18T08:12:29.840 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 2.057e-06 2024-09-18T08:12:29.840 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:12:29.840 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:12:29.840 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-09-18T08:12:29.840 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:10:14.236503+0000", 2024-09-18T08:12:29.841 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 5.3870000000000003e-06 2024-09-18T08:12:29.841 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:12:29.841 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:12:29.841 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-09-18T08:12:29.841 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:10:14.236508+0000", 2024-09-18T08:12:29.841 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 5.2190000000000003e-06 2024-09-18T08:12:29.841 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:12:29.841 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:12:29.841 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-18T08:12:29.841 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:10:14.236572+0000", 2024-09-18T08:12:29.841 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 6.3966000000000004e-05 2024-09-18T08:12:29.841 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:12:29.842 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:12:29.842 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-09-18T08:12:29.842 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:10:14.236702+0000", 2024-09-18T08:12:29.842 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00013004299999999999 2024-09-18T08:12:29.842 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:12:29.842 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:12:29.842 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "waiting for subops from 2,4", 2024-09-18T08:12:29.842 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:10:14.236830+0000", 2024-09-18T08:12:29.842 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00012811100000000001 2024-09-18T08:12:29.842 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:12:29.842 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:12:29.842 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:12:29.842 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:10:14.241959+0000", 2024-09-18T08:12:29.842 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.005129471 2024-09-18T08:12:29.843 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:12:29.843 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:12:29.843 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "op_commit", 2024-09-18T08:12:29.843 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:10:14.242315+0000", 2024-09-18T08:12:29.843 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00035617899999999998 2024-09-18T08:12:29.843 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:12:29.843 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:12:29.843 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-18T08:12:29.843 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:10:14.249236+0000", 2024-09-18T08:12:29.843 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0069204399999999999 2024-09-18T08:12:29.843 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:12:29.843 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:12:29.843 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "waiting for peered", 2024-09-18T08:12:29.844 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:10:14.249256+0000", 2024-09-18T08:12:29.844 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.9888999999999999e-05 2024-09-18T08:12:29.844 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:12:29.844 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:12:29.844 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-18T08:12:29.844 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:10:15.264014+0000", 2024-09-18T08:12:29.844 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.014757718 2024-09-18T08:12:29.844 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:12:29.844 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:12:29.844 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-09-18T08:12:29.844 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:10:15.264081+0000", 2024-09-18T08:12:29.844 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 6.7098e-05 2024-09-18T08:12:29.845 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-18T08:12:29.845 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-09-18T08:12:29.845 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-18T08:12:29.845 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:12:29.845 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:12:29.845 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.15216.0:151204 8.4 8:22e9648e:::benchmark_data_smithi007_101583_object50452:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e219)", 2024-09-18T08:12:29.845 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-09-18T08:10:14.251817+0000", 2024-09-18T08:12:29.845 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 135.44329911099999, 2024-09-18T08:12:29.845 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.0215490739999999, 2024-09-18T08:12:29.845 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-09-18T08:12:29.845 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-09-18T08:12:29.845 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T08:12:29.846 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2024-09-18T08:12:29.846 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.15216", 2024-09-18T08:12:29.846 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "172.21.15.7:0/2596041177", 2024-09-18T08:12:29.846 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 151204 2024-09-18T08:12:29.846 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:12:29.846 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-09-18T08:12:29.846 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:12:29.846 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-09-18T08:12:29.846 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:10:14.251817+0000", 2024-09-18T08:12:29.846 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-18T08:12:29.846 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:12:29.846 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:12:29.846 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-09-18T08:12:29.847 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:10:14.251817+0000", 2024-09-18T08:12:29.847 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-18T08:12:29.847 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:12:29.847 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:12:29.847 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-09-18T08:12:29.847 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:10:14.251818+0000", 2024-09-18T08:12:29.847 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.46e-06 2024-09-18T08:12:29.847 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:12:29.847 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:12:29.847 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-09-18T08:12:29.848 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:10:14.251823+0000", 2024-09-18T08:12:29.848 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 5.4190000000000001e-06 2024-09-18T08:12:29.848 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:12:29.848 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:12:29.848 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-09-18T08:12:29.848 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:10:14.251838+0000", 2024-09-18T08:12:29.848 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.4647000000000001e-05 2024-09-18T08:12:29.848 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:12:29.848 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:12:29.848 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-09-18T08:12:29.848 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:10:14.251846+0000", 2024-09-18T08:12:29.848 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 7.8699999999999992e-06 2024-09-18T08:12:29.848 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:12:29.848 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:12:29.849 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-18T08:12:29.849 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:10:14.251908+0000", 2024-09-18T08:12:29.849 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 6.2402000000000003e-05 2024-09-18T08:12:29.849 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:12:29.849 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:12:29.849 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "waiting for peered", 2024-09-18T08:12:29.849 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:10:14.251918+0000", 2024-09-18T08:12:29.849 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 9.3659999999999998e-06 2024-09-18T08:12:29.849 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:12:29.849 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:12:29.849 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-18T08:12:29.849 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:10:15.264536+0000", 2024-09-18T08:12:29.849 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.012618697 2024-09-18T08:12:29.850 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:12:29.850 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:12:29.850 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-09-18T08:12:29.850 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:10:15.264615+0000", 2024-09-18T08:12:29.850 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 7.9005000000000007e-05 2024-09-18T08:12:29.850 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:12:29.850 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:12:29.850 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "waiting for subops from 2,4", 2024-09-18T08:12:29.850 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:10:15.264722+0000", 2024-09-18T08:12:29.850 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000106829 2024-09-18T08:12:29.850 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:12:29.850 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:12:29.850 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:12:29.851 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:10:15.269903+0000", 2024-09-18T08:12:29.851 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0051810099999999998 2024-09-18T08:12:29.851 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:12:29.851 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:12:29.851 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:12:29.851 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:10:15.269953+0000", 2024-09-18T08:12:29.851 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4.9539000000000001e-05 2024-09-18T08:12:29.851 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:12:29.851 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:12:29.851 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "op_commit", 2024-09-18T08:12:29.851 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:10:15.273300+0000", 2024-09-18T08:12:29.851 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.003347222 2024-09-18T08:12:29.851 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:12:29.852 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:12:29.852 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "commit_sent", 2024-09-18T08:12:29.852 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:10:15.273336+0000", 2024-09-18T08:12:29.852 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 3.5663000000000001e-05 2024-09-18T08:12:29.852 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:12:29.852 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:12:29.852 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-09-18T08:12:29.852 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:10:15.273366+0000", 2024-09-18T08:12:29.853 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 2.9944999999999999e-05 2024-09-18T08:12:29.853 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-18T08:12:29.853 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-09-18T08:12:29.853 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-18T08:12:29.853 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:12:29.853 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:12:29.854 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.15216.0:151205 8.4 8:2047b51c:::benchmark_data_smithi007_101583_object50453:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e219)", 2024-09-18T08:12:29.854 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-09-18T08:10:14.252763+0000", 2024-09-18T08:12:29.854 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 135.44235223300001, 2024-09-18T08:12:29.854 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.020725949, 2024-09-18T08:12:29.854 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-09-18T08:12:29.854 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-09-18T08:12:29.854 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T08:12:29.854 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2024-09-18T08:12:29.854 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.15216", 2024-09-18T08:12:29.854 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "172.21.15.7:0/2596041177", 2024-09-18T08:12:29.855 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 151205 2024-09-18T08:12:29.855 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:12:29.855 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-09-18T08:12:29.855 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:12:29.855 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-09-18T08:12:29.855 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:10:14.252763+0000", 2024-09-18T08:12:29.855 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-18T08:12:29.855 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:12:29.855 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:12:29.855 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-09-18T08:12:29.855 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:10:14.252763+0000", 2024-09-18T08:12:29.855 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-18T08:12:29.856 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:12:29.856 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:12:29.856 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-09-18T08:12:29.856 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:10:14.252764+0000", 2024-09-18T08:12:29.856 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 9.3600000000000002e-07 2024-09-18T08:12:29.856 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:12:29.856 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:12:29.856 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-09-18T08:12:29.856 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:10:14.252772+0000", 2024-09-18T08:12:29.856 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 7.7239999999999992e-06 2024-09-18T08:12:29.856 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:12:29.856 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:12:29.856 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-09-18T08:12:29.857 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:10:14.252786+0000", 2024-09-18T08:12:29.857 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.4249e-05 2024-09-18T08:12:29.857 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:12:29.857 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:12:29.857 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-09-18T08:12:29.857 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:10:14.252794+0000", 2024-09-18T08:12:29.857 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 7.2679999999999997e-06 2024-09-18T08:12:29.857 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:12:29.857 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:12:29.857 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-18T08:12:29.857 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:10:14.253208+0000", 2024-09-18T08:12:29.858 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000414793 2024-09-18T08:12:29.858 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:12:29.858 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:12:29.858 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "waiting for peered", 2024-09-18T08:12:29.858 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:10:14.253218+0000", 2024-09-18T08:12:29.858 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 9.8670000000000006e-06 2024-09-18T08:12:29.858 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:12:29.858 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:12:29.858 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-18T08:12:29.858 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:10:15.267590+0000", 2024-09-18T08:12:29.858 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.0143720919999999 2024-09-18T08:12:29.858 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:12:29.859 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:12:29.859 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-09-18T08:12:29.859 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:10:15.267713+0000", 2024-09-18T08:12:29.859 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00012211200000000001 2024-09-18T08:12:29.859 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:12:29.859 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:12:29.859 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "waiting for subops from 2,4", 2024-09-18T08:12:29.859 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:10:15.267851+0000", 2024-09-18T08:12:29.859 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00013890500000000001 2024-09-18T08:12:29.859 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:12:29.859 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:12:29.859 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:12:29.859 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:10:15.273292+0000", 2024-09-18T08:12:29.859 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0054400849999999999 2024-09-18T08:12:29.859 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:12:29.860 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:12:29.860 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "op_commit", 2024-09-18T08:12:29.860 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:10:15.273373+0000", 2024-09-18T08:12:29.860 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 8.1631000000000005e-05 2024-09-18T08:12:29.860 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:12:29.860 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:12:29.860 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:12:29.860 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:10:15.273402+0000", 2024-09-18T08:12:29.860 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 2.9297000000000001e-05 2024-09-18T08:12:29.860 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:12:29.860 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:12:29.860 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "commit_sent", 2024-09-18T08:12:29.860 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:10:15.273448+0000", 2024-09-18T08:12:29.860 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4.5079e-05 2024-09-18T08:12:29.861 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:12:29.861 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:12:29.861 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-09-18T08:12:29.861 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:10:15.273489+0000", 2024-09-18T08:12:29.861 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4.1910999999999999e-05 2024-09-18T08:12:29.861 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-18T08:12:29.861 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-09-18T08:12:29.861 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-18T08:12:29.861 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:12:29.861 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:12:29.861 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.15582.0:53570 9.6 9:66e3b7ee:::benchmark_data_smithi007_124019_object53569:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e243)", 2024-09-18T08:12:29.861 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-09-18T08:11:41.318092+0000", 2024-09-18T08:12:29.861 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 48.377024136999999, 2024-09-18T08:12:29.861 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.026594762, 2024-09-18T08:12:29.862 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-09-18T08:12:29.862 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-09-18T08:12:29.862 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "reached pg", 2024-09-18T08:12:29.862 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2024-09-18T08:12:29.862 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.15582", 2024-09-18T08:12:29.862 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "172.21.15.7:0/3598605839", 2024-09-18T08:12:29.862 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 53570 2024-09-18T08:12:29.862 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:12:29.862 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-09-18T08:12:29.862 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:12:29.862 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-09-18T08:12:29.862 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:11:41.318092+0000", 2024-09-18T08:12:29.862 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-18T08:12:29.862 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:12:29.862 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:12:29.863 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-09-18T08:12:29.863 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:11:41.318092+0000", 2024-09-18T08:12:29.863 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-18T08:12:29.863 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:12:29.863 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:12:29.863 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-09-18T08:12:29.863 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:11:41.318094+0000", 2024-09-18T08:12:29.863 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 2.0829999999999998e-06 2024-09-18T08:12:29.863 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:12:29.863 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:12:29.863 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-09-18T08:12:29.863 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:11:41.318130+0000", 2024-09-18T08:12:29.863 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 3.6049e-05 2024-09-18T08:12:29.863 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:12:29.863 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:12:29.864 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-09-18T08:12:29.864 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:11:41.318150+0000", 2024-09-18T08:12:29.864 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 2.0237999999999998e-05 2024-09-18T08:12:29.864 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:12:29.864 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:12:29.864 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-09-18T08:12:29.864 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:11:41.318160+0000", 2024-09-18T08:12:29.864 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 9.7059999999999999e-06 2024-09-18T08:12:29.864 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:12:29.864 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:12:29.864 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-18T08:12:29.864 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:11:41.318221+0000", 2024-09-18T08:12:29.864 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 6.1245000000000001e-05 2024-09-18T08:12:29.865 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:12:29.865 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:12:29.865 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-09-18T08:12:29.865 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:11:41.318360+0000", 2024-09-18T08:12:29.865 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00013884600000000001 2024-09-18T08:12:29.865 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:12:29.865 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:12:29.865 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "waiting for subops from 1,2", 2024-09-18T08:12:29.865 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:11:41.318511+0000", 2024-09-18T08:12:29.865 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00015077800000000001 2024-09-18T08:12:29.865 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:12:29.865 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:12:29.865 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "op_commit", 2024-09-18T08:12:29.865 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:11:41.325101+0000", 2024-09-18T08:12:29.866 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0065902720000000003 2024-09-18T08:12:29.866 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:12:29.866 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:12:29.866 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-18T08:12:29.866 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:11:41.334372+0000", 2024-09-18T08:12:29.866 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0092712360000000004 2024-09-18T08:12:29.866 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:12:29.866 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:12:29.866 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "waiting for peered", 2024-09-18T08:12:29.866 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:11:41.334377+0000", 2024-09-18T08:12:29.866 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 5.4689999999999996e-06 2024-09-18T08:12:29.867 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:12:29.867 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:12:29.867 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-18T08:12:29.867 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:11:42.344620+0000", 2024-09-18T08:12:29.867 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.010242721 2024-09-18T08:12:29.867 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:12:29.867 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:12:29.867 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-09-18T08:12:29.867 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:11:42.344686+0000", 2024-09-18T08:12:29.867 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 6.6118999999999995e-05 2024-09-18T08:12:29.867 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-18T08:12:29.868 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-09-18T08:12:29.868 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-18T08:12:29.868 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:12:29.868 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:12:29.868 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.15582.0:53571 9.f 9:f864f7c7:::benchmark_data_smithi007_124019_object53570:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e243)", 2024-09-18T08:12:29.868 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-09-18T08:11:41.318182+0000", 2024-09-18T08:12:29.868 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 48.376933418, 2024-09-18T08:12:29.868 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.0233721120000001, 2024-09-18T08:12:29.868 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-09-18T08:12:29.868 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-09-18T08:12:29.868 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "reached pg", 2024-09-18T08:12:29.868 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2024-09-18T08:12:29.868 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.15582", 2024-09-18T08:12:29.868 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "172.21.15.7:0/3598605839", 2024-09-18T08:12:29.868 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 53571 2024-09-18T08:12:29.869 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:12:29.869 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-09-18T08:12:29.869 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:12:29.869 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-09-18T08:12:29.869 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:11:41.318182+0000", 2024-09-18T08:12:29.869 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-18T08:12:29.869 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:12:29.869 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:12:29.869 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-09-18T08:12:29.869 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:11:41.318182+0000", 2024-09-18T08:12:29.869 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-18T08:12:29.869 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:12:29.869 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:12:29.870 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-09-18T08:12:29.870 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:11:41.318183+0000", 2024-09-18T08:12:29.870 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 6.1999999999999999e-07 2024-09-18T08:12:29.870 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:12:29.870 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:12:29.870 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-09-18T08:12:29.870 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:11:41.318216+0000", 2024-09-18T08:12:29.870 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 3.3243e-05 2024-09-18T08:12:29.870 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:12:29.870 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:12:29.870 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-09-18T08:12:29.870 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:11:41.318225+0000", 2024-09-18T08:12:29.870 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 8.5320000000000006e-06 2024-09-18T08:12:29.870 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:12:29.871 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:12:29.871 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-09-18T08:12:29.871 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:11:41.318231+0000", 2024-09-18T08:12:29.871 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 6.8040000000000004e-06 2024-09-18T08:12:29.871 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:12:29.871 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:12:29.871 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-18T08:12:29.871 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:11:41.319825+0000", 2024-09-18T08:12:29.871 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0015931490000000001 2024-09-18T08:12:29.871 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:12:29.871 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:12:29.871 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-09-18T08:12:29.871 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:11:41.319972+0000", 2024-09-18T08:12:29.871 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00014711999999999999 2024-09-18T08:12:29.872 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:12:29.872 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:12:29.872 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "waiting for subops from 1,3", 2024-09-18T08:12:29.872 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:11:41.320112+0000", 2024-09-18T08:12:29.872 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000140319 2024-09-18T08:12:29.872 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:12:29.872 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:12:29.872 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "op_commit", 2024-09-18T08:12:29.872 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:11:41.326432+0000", 2024-09-18T08:12:29.872 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0063201799999999999 2024-09-18T08:12:29.872 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:12:29.872 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:12:29.872 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-18T08:12:29.872 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:11:41.334005+0000", 2024-09-18T08:12:29.873 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0075727659999999999 2024-09-18T08:12:29.873 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:12:29.873 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:12:29.873 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "waiting for peered", 2024-09-18T08:12:29.873 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:11:41.334018+0000", 2024-09-18T08:12:29.873 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.2728000000000001e-05 2024-09-18T08:12:29.873 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:12:29.873 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:12:29.873 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-18T08:12:29.873 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:11:42.340988+0000", 2024-09-18T08:12:29.873 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.0069699830000001 2024-09-18T08:12:29.873 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:12:29.873 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:12:29.873 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-09-18T08:12:29.874 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:11:42.341554+0000", 2024-09-18T08:12:29.874 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00056666799999999999 2024-09-18T08:12:29.874 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-18T08:12:29.874 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-09-18T08:12:29.874 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-18T08:12:29.874 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:12:29.874 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:12:29.874 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.15582.0:53572 9.f 9:f97a7dfc:::benchmark_data_smithi007_124019_object53571:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e243)", 2024-09-18T08:12:29.874 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-09-18T08:11:41.318250+0000", 2024-09-18T08:12:29.874 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 48.376866165999999, 2024-09-18T08:12:29.874 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.0237908229999999, 2024-09-18T08:12:29.874 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-09-18T08:12:29.874 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-09-18T08:12:29.874 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "reached pg", 2024-09-18T08:12:29.875 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2024-09-18T08:12:29.875 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.15582", 2024-09-18T08:12:29.875 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "172.21.15.7:0/3598605839", 2024-09-18T08:12:29.875 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 53572 2024-09-18T08:12:29.875 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:12:29.875 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-09-18T08:12:29.875 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:12:29.875 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-09-18T08:12:29.875 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:11:41.318250+0000", 2024-09-18T08:12:29.875 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-18T08:12:29.875 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:12:29.875 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:12:29.875 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-09-18T08:12:29.875 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:11:41.318250+0000", 2024-09-18T08:12:29.875 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-18T08:12:29.876 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:12:29.876 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:12:29.876 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-09-18T08:12:29.876 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:11:41.318250+0000", 2024-09-18T08:12:29.876 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 7.92e-07 2024-09-18T08:12:29.876 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:12:29.876 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:12:29.876 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-09-18T08:12:29.876 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:11:41.318283+0000", 2024-09-18T08:12:29.876 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 3.2858999999999998e-05 2024-09-18T08:12:29.876 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:12:29.876 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:12:29.876 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-09-18T08:12:29.876 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:11:41.318291+0000", 2024-09-18T08:12:29.877 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 7.5780000000000001e-06 2024-09-18T08:12:29.877 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:12:29.877 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:12:29.877 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-09-18T08:12:29.877 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:11:41.318297+0000", 2024-09-18T08:12:29.877 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 6.3770000000000001e-06 2024-09-18T08:12:29.877 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:12:29.877 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:12:29.877 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-18T08:12:29.877 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:11:41.322223+0000", 2024-09-18T08:12:29.878 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0039256059999999999 2024-09-18T08:12:29.878 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:12:29.878 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:12:29.878 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-09-18T08:12:29.878 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:11:41.322364+0000", 2024-09-18T08:12:29.878 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00014106599999999999 2024-09-18T08:12:29.878 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:12:29.878 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:12:29.878 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "waiting for subops from 1,3", 2024-09-18T08:12:29.878 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:11:41.322457+0000", 2024-09-18T08:12:29.878 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 9.3479000000000004e-05 2024-09-18T08:12:29.878 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:12:29.878 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:12:29.879 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "op_commit", 2024-09-18T08:12:29.879 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:11:41.326613+0000", 2024-09-18T08:12:29.879 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0041556989999999997 2024-09-18T08:12:29.879 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:12:29.879 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:12:29.879 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-18T08:12:29.879 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:11:41.334061+0000", 2024-09-18T08:12:29.879 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0074481039999999997 2024-09-18T08:12:29.879 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:12:29.879 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:12:29.879 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "waiting for peered", 2024-09-18T08:12:29.879 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:11:41.334070+0000", 2024-09-18T08:12:29.879 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 8.5879999999999998e-06 2024-09-18T08:12:29.880 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:12:29.880 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:12:29.880 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-18T08:12:29.880 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:11:42.341976+0000", 2024-09-18T08:12:29.880 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.007905845 2024-09-18T08:12:29.880 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:12:29.880 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:12:29.880 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-09-18T08:12:29.880 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:11:42.342040+0000", 2024-09-18T08:12:29.880 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 6.4830000000000001e-05 2024-09-18T08:12:29.880 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-18T08:12:29.880 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-09-18T08:12:29.881 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-18T08:12:29.881 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-18T08:12:29.881 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-09-18T08:12:29.881 INFO:teuthology.orchestra.run.smithi050.stdout:} 2024-09-18T08:12:29.883 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:12:29.883 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:12:29.883 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0' 2024-09-18T08:12:29.984 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7 2024-09-18T08:12:30.034 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:12:30.035 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:12:30.035 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7' 2024-09-18T08:12:30.136 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6 2024-09-18T08:12:30.172 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:12:30.173 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:12:30.173 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6' 2024-09-18T08:12:30.196 INFO:tasks.radosbench.radosbench.0.smithi007.stdout: 75 16 122702 122686 102.218 82.5 0.0172672 0.00977847 2024-09-18T08:12:30.274 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4 2024-09-18T08:12:30.315 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:12:30.316 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:12:30.316 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4' 2024-09-18T08:12:30.384 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid 779349ca-7593-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-18T08:12:30.417 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5 2024-09-18T08:12:30.473 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:12:30.473 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:12:30.473 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5' 2024-09-18T08:12:30.574 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5 2024-09-18T08:12:30.612 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:12:30.613 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:12:30.613 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5' 2024-09-18T08:12:30.714 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7 2024-09-18T08:12:30.756 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:12:30.756 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:12:30.757 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7' 2024-09-18T08:12:30.857 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0 2024-09-18T08:12:30.887 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:12:30.887 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:12:30.888 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0' 2024-09-18T08:12:30.906 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:12:30 smithi050 ceph-mon[32371]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-18T08:12:30.906 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:12:30 smithi050 ceph-mon[32371]: pgmap v466: 158 pgs: 30 active+undersized, 34 active+undersized+degraded, 94 active+clean; 7.4 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 1.2 KiB/s rd, 102 MiB/s wr, 1.63k op/s; 68070/362502 objects degraded (18.778%) 2024-09-18T08:12:30.921 DEBUG:teuthology.orchestra.run.smithi050:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid 779349ca-7593-11ef-bceb-c7b262605968 -- ceph daemon osd.6 dump_ops_in_flight 2024-09-18T08:12:30.980 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:12:30 smithi007 ceph-mon[29459]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-18T08:12:30.980 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:12:30 smithi007 ceph-mon[29459]: pgmap v466: 158 pgs: 30 active+undersized, 34 active+undersized+degraded, 94 active+clean; 7.4 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 1.2 KiB/s rd, 102 MiB/s wr, 1.63k op/s; 68070/362502 objects degraded (18.778%) 2024-09-18T08:12:30.980 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:12:30 smithi007 ceph-mon[39249]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-18T08:12:30.981 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:12:30 smithi007 ceph-mon[39249]: pgmap v466: 158 pgs: 30 active+undersized, 34 active+undersized+degraded, 94 active+clean; 7.4 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 1.2 KiB/s rd, 102 MiB/s wr, 1.63k op/s; 68070/362502 objects degraded (18.778%) 2024-09-18T08:12:30.989 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7 2024-09-18T08:12:31.030 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:12:31.031 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:12:31.031 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7' 2024-09-18T08:12:31.074 INFO:teuthology.orchestra.run.smithi007.stderr:Creating an OSD daemon form without an OSD FSID value 2024-09-18T08:12:31.132 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3 2024-09-18T08:12:31.159 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:12:31.160 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:12:31.160 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3' 2024-09-18T08:12:31.197 INFO:tasks.radosbench.radosbench.0.smithi007.stdout: 76 16 124080 124064 102.006 86.125 0.00483845 0.00979876 2024-09-18T08:12:31.261 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2 2024-09-18T08:12:31.290 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:12:31.291 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:12:31.291 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2' 2024-09-18T08:12:31.362 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/779349ca-7593-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T08:12:31.383 INFO:teuthology.orchestra.run.smithi050.stderr:Inferring config /var/lib/ceph/779349ca-7593-11ef-bceb-c7b262605968/mon.b/config 2024-09-18T08:12:31.392 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7 2024-09-18T08:12:31.418 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:12:31.419 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:12:31.419 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7' 2024-09-18T08:12:31.520 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3 2024-09-18T08:12:31.560 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:12:31.561 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:12:31.561 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3' 2024-09-18T08:12:31.662 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0 2024-09-18T08:12:31.690 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:12:31.691 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:12:31.691 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0' 2024-09-18T08:12:31.792 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6 2024-09-18T08:12:31.812 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:12:31 smithi050 ceph-mon[32371]: 5.d scrub starts 2024-09-18T08:12:31.812 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:12:31 smithi050 ceph-mon[32371]: 5.d scrub ok 2024-09-18T08:12:31.812 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:12:31 smithi050 ceph-mon[32371]: 5.1 scrub starts 2024-09-18T08:12:31.812 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:12:31 smithi050 ceph-mon[32371]: 5.1 scrub ok 2024-09-18T08:12:31.812 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:12:31 smithi050 ceph-mon[32371]: purged_snaps scrub starts 2024-09-18T08:12:31.812 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:12:31 smithi050 ceph-mon[32371]: purged_snaps scrub ok 2024-09-18T08:12:31.834 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid 779349ca-7593-11ef-bceb-c7b262605968 -- ceph osd unset nodeep-scrub 2024-09-18T08:12:31.838 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:12:31.839 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:12:31.839 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6' 2024-09-18T08:12:31.884 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:12:31 smithi007 ceph-mon[29459]: 5.d scrub starts 2024-09-18T08:12:31.884 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:12:31 smithi007 ceph-mon[29459]: 5.d scrub ok 2024-09-18T08:12:31.884 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:12:31 smithi007 ceph-mon[29459]: 5.1 scrub starts 2024-09-18T08:12:31.885 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:12:31 smithi007 ceph-mon[29459]: 5.1 scrub ok 2024-09-18T08:12:31.885 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:12:31 smithi007 ceph-mon[29459]: purged_snaps scrub starts 2024-09-18T08:12:31.885 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:12:31 smithi007 ceph-mon[29459]: purged_snaps scrub ok 2024-09-18T08:12:31.885 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:12:31 smithi007 ceph-mon[39249]: 5.d scrub starts 2024-09-18T08:12:31.885 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:12:31 smithi007 ceph-mon[39249]: 5.d scrub ok 2024-09-18T08:12:31.885 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:12:31 smithi007 ceph-mon[39249]: 5.1 scrub starts 2024-09-18T08:12:31.885 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:12:31 smithi007 ceph-mon[39249]: 5.1 scrub ok 2024-09-18T08:12:31.885 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:12:31 smithi007 ceph-mon[39249]: purged_snaps scrub starts 2024-09-18T08:12:31.885 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:12:31 smithi007 ceph-mon[39249]: purged_snaps scrub ok 2024-09-18T08:12:31.940 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0 2024-09-18T08:12:31.977 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:12:31.977 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:12:31.978 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0' 2024-09-18T08:12:32.076 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2024-09-18T08:12:32.076 INFO:teuthology.orchestra.run.smithi050.stdout: "ops": [], 2024-09-18T08:12:32.076 INFO:teuthology.orchestra.run.smithi050.stdout: "num_ops": 0 2024-09-18T08:12:32.076 INFO:teuthology.orchestra.run.smithi050.stdout:} 2024-09-18T08:12:32.079 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7 2024-09-18T08:12:32.111 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:12:32.112 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:12:32.112 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7' 2024-09-18T08:12:32.197 INFO:tasks.radosbench.radosbench.0.smithi007.stdout: 77 16 125370 125354 101.729 80.625 0.00854993 0.00982586 2024-09-18T08:12:32.212 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7 2024-09-18T08:12:32.250 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:12:32.251 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:12:32.251 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7' 2024-09-18T08:12:32.352 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0 2024-09-18T08:12:32.369 INFO:teuthology.orchestra.run.smithi007.stdout:osd.0: {} 2024-09-18T08:12:32.369 INFO:teuthology.orchestra.run.smithi007.stdout:osd.2: {} 2024-09-18T08:12:32.369 INFO:teuthology.orchestra.run.smithi007.stdout:osd.3: {} 2024-09-18T08:12:32.369 INFO:teuthology.orchestra.run.smithi007.stdout:osd.4: {} 2024-09-18T08:12:32.369 INFO:teuthology.orchestra.run.smithi007.stdout:osd.5: {} 2024-09-18T08:12:32.369 INFO:teuthology.orchestra.run.smithi007.stdout:osd.6: {} 2024-09-18T08:12:32.369 INFO:teuthology.orchestra.run.smithi007.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-18T08:12:32.370 INFO:teuthology.orchestra.run.smithi007.stderr:Error ENXIO: problem getting command descriptions from osd.1 2024-09-18T08:12:32.370 INFO:teuthology.orchestra.run.smithi007.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-18T08:12:32.370 INFO:teuthology.orchestra.run.smithi007.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-18T08:12:32.370 INFO:teuthology.orchestra.run.smithi007.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-18T08:12:32.370 INFO:teuthology.orchestra.run.smithi007.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-18T08:12:32.370 INFO:teuthology.orchestra.run.smithi007.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-18T08:12:32.393 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:12:32.394 INFO:teuthology.orchestra.run.smithi007.stdout:osd.7: {} 2024-09-18T08:12:32.394 INFO:teuthology.orchestra.run.smithi007.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-18T08:12:32.395 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:12:32.395 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0' 2024-09-18T08:12:32.402 INFO:teuthology.orchestra.run.smithi007.stdout:get_pg_num_history pg_num_history pg_num_history(e243 pg_nums {1={152=11},2={60=3},3={61=32},4={63=32},5={65=32},6={67=32},8={165=16},9={232=16}} deleted_pools 231,8) 2024-09-18T08:12:32.402 INFO:teuthology.orchestra.run.smithi007.stdout:Importing pgid 4.f 2024-09-18T08:12:32.402 INFO:teuthology.orchestra.run.smithi007.stdout:Write #4:f04f5452:::obj_delete_at_hint.0000000124:head# 2024-09-18T08:12:32.402 INFO:teuthology.orchestra.run.smithi007.stdout:snapset 0=[]:{} 2024-09-18T08:12:32.403 INFO:teuthology.orchestra.run.smithi007.stdout:Write #4:f1983ba0:::obj_delete_at_hint.0000000001:head# 2024-09-18T08:12:32.403 INFO:teuthology.orchestra.run.smithi007.stdout:snapset 0=[]:{} 2024-09-18T08:12:32.403 INFO:teuthology.orchestra.run.smithi007.stdout:Write #4:f24cb4a8:gc::gc.31:head# 2024-09-18T08:12:32.403 INFO:teuthology.orchestra.run.smithi007.stdout:snapset 0=[]:{} 2024-09-18T08:12:32.403 INFO:teuthology.orchestra.run.smithi007.stdout:Write #4:f2c84a15:::obj_delete_at_hint.0000000126:head# 2024-09-18T08:12:32.403 INFO:teuthology.orchestra.run.smithi007.stdout:snapset 0=[]:{} 2024-09-18T08:12:32.404 INFO:teuthology.orchestra.run.smithi007.stdout:Write #4:f3db1134:::obj_delete_at_hint.0000000055:head# 2024-09-18T08:12:32.404 INFO:teuthology.orchestra.run.smithi007.stdout:snapset 0=[]:{} 2024-09-18T08:12:32.404 INFO:teuthology.orchestra.run.smithi007.stdout:Write #4:f47f4d4d:::obj_delete_at_hint.0000000061:head# 2024-09-18T08:12:32.404 INFO:teuthology.orchestra.run.smithi007.stdout:snapset 0=[]:{} 2024-09-18T08:12:32.405 INFO:teuthology.orchestra.run.smithi007.stdout:Write #4:f6792a7a:::obj_delete_at_hint.0000000097:head# 2024-09-18T08:12:32.405 INFO:teuthology.orchestra.run.smithi007.stdout:snapset 0=[]:{} 2024-09-18T08:12:32.405 INFO:teuthology.orchestra.run.smithi007.stdout:write_pg epoch 226 info 4.f( v 69'15 (0'0,69'15] local-lis/les=181/182 n=7 ec=63/63 lis/c=181/181 les/c/f=182/182/0 sis=181) 2024-09-18T08:12:32.405 INFO:teuthology.orchestra.run.smithi007.stdout:Import successful 2024-09-18T08:12:32.496 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6 2024-09-18T08:12:32.533 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:12:32.534 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:12:32.535 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6' 2024-09-18T08:12:32.635 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5 2024-09-18T08:12:32.671 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:12:32.672 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:12:32.672 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5' 2024-09-18T08:12:32.773 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6 2024-09-18T08:12:32.811 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:12:32.812 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:12:32.812 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6' 2024-09-18T08:12:32.906 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:12:32 smithi050 ceph-mon[32371]: 3.d scrub starts 2024-09-18T08:12:32.906 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:12:32 smithi050 ceph-mon[32371]: 3.d scrub ok 2024-09-18T08:12:32.906 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:12:32 smithi050 ceph-mon[32371]: pgmap v468: 158 pgs: 30 active+undersized, 34 active+undersized+degraded, 94 active+clean; 7.4 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 732 B/s rd, 57 MiB/s wr, 918 op/s; 68070/362502 objects degraded (18.778%) 2024-09-18T08:12:32.906 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:12:32 smithi050 ceph-mon[32371]: osdmap e262: 8 total, 7 up, 8 in 2024-09-18T08:12:32.906 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:12:32 smithi050 ceph-mon[32371]: 5.1e scrub starts 2024-09-18T08:12:32.906 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:12:32 smithi050 ceph-mon[32371]: 5.1e scrub ok 2024-09-18T08:12:32.906 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:12:32 smithi050 ceph-mon[32371]: from='client.? 172.21.15.7:0/3364142718' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-18T08:12:32.913 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6 2024-09-18T08:12:32.922 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:12:32 smithi007 ceph-mon[39249]: 3.d scrub starts 2024-09-18T08:12:32.922 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:12:32 smithi007 ceph-mon[39249]: 3.d scrub ok 2024-09-18T08:12:32.923 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:12:32 smithi007 ceph-mon[39249]: pgmap v468: 158 pgs: 30 active+undersized, 34 active+undersized+degraded, 94 active+clean; 7.4 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 732 B/s rd, 57 MiB/s wr, 918 op/s; 68070/362502 objects degraded (18.778%) 2024-09-18T08:12:32.923 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:12:32 smithi007 ceph-mon[39249]: osdmap e262: 8 total, 7 up, 8 in 2024-09-18T08:12:32.923 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:12:32 smithi007 ceph-mon[39249]: 5.1e scrub starts 2024-09-18T08:12:32.923 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:12:32 smithi007 ceph-mon[39249]: 5.1e scrub ok 2024-09-18T08:12:32.923 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:12:32 smithi007 ceph-mon[39249]: from='client.? 172.21.15.7:0/3364142718' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-18T08:12:32.923 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:12:32 smithi007 ceph-mon[29459]: 3.d scrub starts 2024-09-18T08:12:32.923 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:12:32 smithi007 ceph-mon[29459]: 3.d scrub ok 2024-09-18T08:12:32.923 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:12:32 smithi007 ceph-mon[29459]: pgmap v468: 158 pgs: 30 active+undersized, 34 active+undersized+degraded, 94 active+clean; 7.4 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 732 B/s rd, 57 MiB/s wr, 918 op/s; 68070/362502 objects degraded (18.778%) 2024-09-18T08:12:32.923 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:12:32 smithi007 ceph-mon[29459]: osdmap e262: 8 total, 7 up, 8 in 2024-09-18T08:12:32.923 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:12:32 smithi007 ceph-mon[29459]: 5.1e scrub starts 2024-09-18T08:12:32.923 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:12:32 smithi007 ceph-mon[29459]: 5.1e scrub ok 2024-09-18T08:12:32.924 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:12:32 smithi007 ceph-mon[29459]: from='client.? 172.21.15.7:0/3364142718' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-18T08:12:32.949 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:12:32.950 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:12:32.950 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6' 2024-09-18T08:12:33.031 DEBUG:teuthology.orchestra.run.smithi050:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid 779349ca-7593-11ef-bceb-c7b262605968 -- ceph daemon osd.6 dump_blocked_ops 2024-09-18T08:12:33.050 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2 2024-09-18T08:12:33.088 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:12:33.089 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:12:33.089 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2' 2024-09-18T08:12:33.190 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3 2024-09-18T08:12:33.197 INFO:tasks.radosbench.radosbench.0.smithi007.stdout: 78 15 126704 126689 101.494 83.4375 0.0583852 0.00984243 2024-09-18T08:12:33.222 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:12:33.223 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:12:33.224 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3' 2024-09-18T08:12:33.325 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5 2024-09-18T08:12:33.358 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:12:33.358 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:12:33.359 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5' 2024-09-18T08:12:33.442 INFO:teuthology.orchestra.run.smithi050.stderr:Inferring config /var/lib/ceph/779349ca-7593-11ef-bceb-c7b262605968/mon.b/config 2024-09-18T08:12:33.460 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0 2024-09-18T08:12:33.492 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:12:33.492 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:12:33.493 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0' 2024-09-18T08:12:33.594 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5 2024-09-18T08:12:33.634 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:12:33.634 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:12:33.634 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5' 2024-09-18T08:12:33.735 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0 2024-09-18T08:12:33.766 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:12:33.766 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:12:33.767 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0' 2024-09-18T08:12:33.867 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2 2024-09-18T08:12:33.879 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:12:33 smithi007 ceph-mon[29459]: from='mgr.24470 172.21.15.7:0/2594390261' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-18T08:12:33.879 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:12:33 smithi007 ceph-mon[39249]: from='mgr.24470 172.21.15.7:0/2594390261' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-18T08:12:33.899 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:12:33 smithi050 ceph-mon[32371]: from='mgr.24470 172.21.15.7:0/2594390261' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-18T08:12:33.900 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:12:33.901 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:12:33.901 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2' 2024-09-18T08:12:34.002 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0 2024-09-18T08:12:34.033 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:12:34.033 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:12:34.034 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0' 2024-09-18T08:12:34.135 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2 2024-09-18T08:12:34.156 INFO:teuthology.orchestra.run.smithi050.stdout:ERROR: (22) Invalid argument 2024-09-18T08:12:34.156 INFO:teuthology.orchestra.run.smithi050.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-18T08:12:34.163 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:12:34.163 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:12:34.164 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2' 2024-09-18T08:12:34.197 INFO:tasks.radosbench.radosbench.0.smithi007.stdout: 79 16 128050 128034 101.273 84.0625 0.0110288 0.00987006 2024-09-18T08:12:34.264 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6 2024-09-18T08:12:34.308 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:12:34.309 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:12:34.309 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6' 2024-09-18T08:12:34.356 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-09-18T08:12:34.357 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-09-18T08:12:34.410 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5 2024-09-18T08:12:34.446 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:12:34.446 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:12:34.447 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5' 2024-09-18T08:12:34.548 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3 2024-09-18T08:12:34.579 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:12:34.580 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:12:34.580 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3' 2024-09-18T08:12:34.681 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7 2024-09-18T08:12:34.720 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:12:34.720 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:12:34.721 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7' 2024-09-18T08:12:34.773 DEBUG:teuthology.orchestra.run.smithi007:> sudo rm -f /var/log/ceph/779349ca-7593-11ef-bceb-c7b262605968/exp.4.f.1 2024-09-18T08:12:34.809 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/779349ca-7593-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T08:12:34.821 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0 2024-09-18T08:12:34.857 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:12:34.858 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:12:34.858 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0' 2024-09-18T08:12:34.906 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:12:34 smithi050 ceph-mon[32371]: pgmap v470: 158 pgs: 30 active+undersized, 34 active+undersized+degraded, 94 active+clean; 7.4 GiB data, 23 GiB used, 693 GiB / 715 GiB avail; 853 B/s rd, 75 MiB/s wr, 1.21k op/s; 68886/364950 objects degraded (18.875%) 2024-09-18T08:12:34.906 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:12:34 smithi050 ceph-mon[32371]: Health check update: Degraded data redundancy: 68886/364950 objects degraded (18.875%), 34 pgs degraded (PG_DEGRADED) 2024-09-18T08:12:34.906 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:12:34 smithi050 ceph-mon[32371]: osdmap e263: 8 total, 7 up, 8 in 2024-09-18T08:12:34.934 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:12:34 smithi007 ceph-mon[29459]: pgmap v470: 158 pgs: 30 active+undersized, 34 active+undersized+degraded, 94 active+clean; 7.4 GiB data, 23 GiB used, 693 GiB / 715 GiB avail; 853 B/s rd, 75 MiB/s wr, 1.21k op/s; 68886/364950 objects degraded (18.875%) 2024-09-18T08:12:34.934 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:12:34 smithi007 ceph-mon[29459]: Health check update: Degraded data redundancy: 68886/364950 objects degraded (18.875%), 34 pgs degraded (PG_DEGRADED) 2024-09-18T08:12:34.935 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:12:34 smithi007 ceph-mon[29459]: osdmap e263: 8 total, 7 up, 8 in 2024-09-18T08:12:34.935 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:12:34 smithi007 ceph-mon[39249]: pgmap v470: 158 pgs: 30 active+undersized, 34 active+undersized+degraded, 94 active+clean; 7.4 GiB data, 23 GiB used, 693 GiB / 715 GiB avail; 853 B/s rd, 75 MiB/s wr, 1.21k op/s; 68886/364950 objects degraded (18.875%) 2024-09-18T08:12:34.935 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:12:34 smithi007 ceph-mon[39249]: Health check update: Degraded data redundancy: 68886/364950 objects degraded (18.875%), 34 pgs degraded (PG_DEGRADED) 2024-09-18T08:12:34.935 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:12:34 smithi007 ceph-mon[39249]: osdmap e263: 8 total, 7 up, 8 in 2024-09-18T08:12:34.959 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2 2024-09-18T08:12:34.982 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:12:34.983 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:12:34.983 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2' 2024-09-18T08:12:35.084 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4 2024-09-18T08:12:35.112 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:12:35.113 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:12:35.113 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4' 2024-09-18T08:12:35.197 INFO:tasks.radosbench.radosbench.0.smithi007.stdout:2024-09-18T08:12:35.196372+0000 min lat: 0.00114659 max lat: 1.04338 avg lat: 0.00991098 2024-09-18T08:12:35.197 INFO:tasks.radosbench.radosbench.0.smithi007.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-09-18T08:12:35.197 INFO:tasks.radosbench.radosbench.0.smithi007.stdout: 80 16 129103 129087 100.83 65.8125 0.00320073 0.00991098 2024-09-18T08:12:35.214 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0 2024-09-18T08:12:35.244 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:12:35.245 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:12:35.245 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0' 2024-09-18T08:12:35.346 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7 2024-09-18T08:12:35.384 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:12:35.385 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:12:35.385 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7' 2024-09-18T08:12:35.486 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0 2024-09-18T08:12:35.521 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:12:35.521 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:12:35.521 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0' 2024-09-18T08:12:35.622 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3 2024-09-18T08:12:35.665 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:12:35.666 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:12:35.666 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3' 2024-09-18T08:12:35.666 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:12:35 smithi007 ceph-mon[39249]: 1.7 scrub starts 2024-09-18T08:12:35.667 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:12:35 smithi007 ceph-mon[39249]: 1.7 scrub ok 2024-09-18T08:12:35.767 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3 2024-09-18T08:12:35.798 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:12:35.798 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:12:35.798 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3' 2024-09-18T08:12:35.899 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7 2024-09-18T08:12:35.940 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:12:35.941 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:12:35.941 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7' 2024-09-18T08:12:35.980 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:12:35 smithi007 ceph-mon[29459]: 1.7 scrub starts 2024-09-18T08:12:35.980 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:12:35 smithi007 ceph-mon[29459]: 1.7 scrub ok 2024-09-18T08:12:36.042 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6 2024-09-18T08:12:36.080 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:12:36.081 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:12:36.082 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6' 2024-09-18T08:12:36.155 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:12:35 smithi050 ceph-mon[32371]: 1.7 scrub starts 2024-09-18T08:12:36.156 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:12:35 smithi050 ceph-mon[32371]: 1.7 scrub ok 2024-09-18T08:12:36.183 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7 2024-09-18T08:12:36.197 INFO:tasks.radosbench.radosbench.0.smithi007.stdout: 81 16 130411 130395 100.594 81.75 0.00510469 0.00993657 2024-09-18T08:12:36.216 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:12:36.217 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:12:36.217 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7' 2024-09-18T08:12:36.318 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5 2024-09-18T08:12:36.357 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:12:36.357 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid 779349ca-7593-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-18T08:12:36.358 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:12:36.358 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5' 2024-09-18T08:12:36.459 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7 2024-09-18T08:12:36.495 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:12:36.496 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:12:36.496 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7' 2024-09-18T08:12:36.597 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3 2024-09-18T08:12:36.631 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:12:36.632 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:12:36.632 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3' 2024-09-18T08:12:36.674 INFO:teuthology.orchestra.run.smithi007.stderr:nodeep-scrub is unset 2024-09-18T08:12:36.733 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2 2024-09-18T08:12:36.768 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:12:36.769 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:12:36.769 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2' 2024-09-18T08:12:36.870 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3 2024-09-18T08:12:36.904 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:12:36.905 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:12:36.905 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3' 2024-09-18T08:12:36.937 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Sep 18 08:12:36 smithi007 ceph-779349ca-7593-11ef-bceb-c7b262605968-mgr-y[29688]: ::ffff:172.21.15.50 - - [18/Sep/2024:08:12:36] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-09-18T08:12:36.937 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:12:36 smithi007 ceph-mon[39249]: pgmap v472: 158 pgs: 30 active+undersized, 34 active+undersized+degraded, 94 active+clean; 7.8 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 853 B/s rd, 69 MiB/s wr, 1.11k op/s; 71809/382449 objects degraded (18.776%) 2024-09-18T08:12:36.937 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:12:36 smithi007 ceph-mon[39249]: from='client.? 172.21.15.7:0/3915973810' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-18T08:12:36.937 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:12:36 smithi007 ceph-mon[39249]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-18T08:12:36.938 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:12:36 smithi007 ceph-mon[29459]: pgmap v472: 158 pgs: 30 active+undersized, 34 active+undersized+degraded, 94 active+clean; 7.8 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 853 B/s rd, 69 MiB/s wr, 1.11k op/s; 71809/382449 objects degraded (18.776%) 2024-09-18T08:12:36.938 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:12:36 smithi007 ceph-mon[29459]: from='client.? 172.21.15.7:0/3915973810' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-18T08:12:36.938 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:12:36 smithi007 ceph-mon[29459]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-18T08:12:37.007 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2 2024-09-18T08:12:37.029 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:12:37.030 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:12:37.030 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2' 2024-09-18T08:12:37.131 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7 2024-09-18T08:12:37.156 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:12:36 smithi050 ceph-mon[32371]: pgmap v472: 158 pgs: 30 active+undersized, 34 active+undersized+degraded, 94 active+clean; 7.8 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 853 B/s rd, 69 MiB/s wr, 1.11k op/s; 71809/382449 objects degraded (18.776%) 2024-09-18T08:12:37.156 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:12:36 smithi050 ceph-mon[32371]: from='client.? 172.21.15.7:0/3915973810' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-18T08:12:37.156 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:12:36 smithi050 ceph-mon[32371]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-18T08:12:37.168 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:12:37.169 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:12:37.169 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7' 2024-09-18T08:12:37.197 INFO:tasks.radosbench.radosbench.0.smithi007.stdout: 82 16 131801 131785 100.427 86.875 0.0138424 0.00995298 2024-09-18T08:12:37.270 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2 2024-09-18T08:12:37.304 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:12:37.304 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:12:37.305 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2' 2024-09-18T08:12:37.406 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4 2024-09-18T08:12:37.446 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:12:37.447 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:12:37.447 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4' 2024-09-18T08:12:37.462 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/779349ca-7593-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T08:12:37.485 DEBUG:teuthology.orchestra.run.smithi050:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid 779349ca-7593-11ef-bceb-c7b262605968 -- ceph daemon osd.6 dump_historic_ops 2024-09-18T08:12:37.548 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4 2024-09-18T08:12:37.614 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:12:37.615 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:12:37.615 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4' 2024-09-18T08:12:37.717 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6 2024-09-18T08:12:37.730 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:12:37 smithi007 ceph-mon[29459]: purged_snaps scrub starts 2024-09-18T08:12:37.731 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:12:37 smithi007 ceph-mon[29459]: purged_snaps scrub ok 2024-09-18T08:12:37.731 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:12:37 smithi007 ceph-mon[29459]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-18T08:12:37.731 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:12:37 smithi007 ceph-mon[29459]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-18T08:12:37.731 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:12:37 smithi007 ceph-mon[29459]: osdmap e264: 8 total, 7 up, 8 in 2024-09-18T08:12:37.731 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:12:37 smithi007 ceph-mon[29459]: pgmap v474: 158 pgs: 30 active+undersized, 34 active+undersized+degraded, 94 active+clean; 7.8 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 853 B/s rd, 69 MiB/s wr, 1.11k op/s; 71809/382449 objects degraded (18.776%) 2024-09-18T08:12:37.732 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:12:37 smithi007 ceph-mon[39249]: purged_snaps scrub starts 2024-09-18T08:12:37.732 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:12:37 smithi007 ceph-mon[39249]: purged_snaps scrub ok 2024-09-18T08:12:37.732 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:12:37 smithi007 ceph-mon[39249]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-18T08:12:37.732 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:12:37 smithi007 ceph-mon[39249]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-18T08:12:37.732 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:12:37 smithi007 ceph-mon[39249]: osdmap e264: 8 total, 7 up, 8 in 2024-09-18T08:12:37.733 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:12:37 smithi007 ceph-mon[39249]: pgmap v474: 158 pgs: 30 active+undersized, 34 active+undersized+degraded, 94 active+clean; 7.8 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 853 B/s rd, 69 MiB/s wr, 1.11k op/s; 71809/382449 objects degraded (18.776%) 2024-09-18T08:12:37.765 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:12:37.766 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:12:37.766 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6' 2024-09-18T08:12:37.867 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4 2024-09-18T08:12:37.900 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:12:37.901 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:12:37.901 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4' 2024-09-18T08:12:38.002 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4 2024-09-18T08:12:38.049 INFO:teuthology.orchestra.run.smithi050.stderr:Inferring config /var/lib/ceph/779349ca-7593-11ef-bceb-c7b262605968/mon.b/config 2024-09-18T08:12:38.053 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:12:38.054 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:12:38.055 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4' 2024-09-18T08:12:38.064 INFO:teuthology.orchestra.run.smithi007.stdout:osd.0: {} 2024-09-18T08:12:38.064 INFO:teuthology.orchestra.run.smithi007.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-18T08:12:38.068 INFO:teuthology.orchestra.run.smithi007.stderr:Error ENXIO: problem getting command descriptions from osd.1 2024-09-18T08:12:38.102 INFO:teuthology.orchestra.run.smithi007.stdout:osd.2: {} 2024-09-18T08:12:38.102 INFO:teuthology.orchestra.run.smithi007.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-18T08:12:38.137 INFO:teuthology.orchestra.run.smithi007.stdout:osd.3: {} 2024-09-18T08:12:38.137 INFO:teuthology.orchestra.run.smithi007.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-18T08:12:38.155 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2 2024-09-18T08:12:38.156 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:12:37 smithi050 ceph-mon[32371]: purged_snaps scrub starts 2024-09-18T08:12:38.156 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:12:37 smithi050 ceph-mon[32371]: purged_snaps scrub ok 2024-09-18T08:12:38.156 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:12:37 smithi050 ceph-mon[32371]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-18T08:12:38.157 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:12:37 smithi050 ceph-mon[32371]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-18T08:12:38.157 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:12:37 smithi050 ceph-mon[32371]: osdmap e264: 8 total, 7 up, 8 in 2024-09-18T08:12:38.157 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:12:37 smithi050 ceph-mon[32371]: pgmap v474: 158 pgs: 30 active+undersized, 34 active+undersized+degraded, 94 active+clean; 7.8 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 853 B/s rd, 69 MiB/s wr, 1.11k op/s; 71809/382449 objects degraded (18.776%) 2024-09-18T08:12:38.167 INFO:teuthology.orchestra.run.smithi007.stdout:osd.4: {} 2024-09-18T08:12:38.167 INFO:teuthology.orchestra.run.smithi007.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-18T08:12:38.197 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:12:38.197 INFO:tasks.radosbench.radosbench.0.smithi007.stdout: 83 16 133086 133070 100.184 80.3125 0.00698586 0.00997685 2024-09-18T08:12:38.198 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:12:38.198 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2' 2024-09-18T08:12:38.205 INFO:teuthology.orchestra.run.smithi007.stdout:osd.5: {} 2024-09-18T08:12:38.205 INFO:teuthology.orchestra.run.smithi007.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-18T08:12:38.238 INFO:teuthology.orchestra.run.smithi007.stdout:osd.6: {} 2024-09-18T08:12:38.238 INFO:teuthology.orchestra.run.smithi007.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-18T08:12:38.269 INFO:teuthology.orchestra.run.smithi007.stdout:osd.7: {} 2024-09-18T08:12:38.269 INFO:teuthology.orchestra.run.smithi007.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-18T08:12:38.299 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5 2024-09-18T08:12:38.335 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:12:38.336 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:12:38.336 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5' 2024-09-18T08:12:38.437 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6 2024-09-18T08:12:38.473 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:12:38.474 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:12:38.475 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6' 2024-09-18T08:12:38.575 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0 2024-09-18T08:12:38.616 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:12:38.617 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:12:38.618 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0' 2024-09-18T08:12:38.718 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2 2024-09-18T08:12:38.753 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:12:38.753 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:12:38.754 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2' 2024-09-18T08:12:38.787 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:12:38 smithi050 ceph-mon[32371]: 4.0 scrub starts 2024-09-18T08:12:38.788 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:12:38 smithi050 ceph-mon[32371]: 4.0 scrub ok 2024-09-18T08:12:38.788 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:12:38 smithi050 ceph-mon[32371]: 5.2 scrub starts 2024-09-18T08:12:38.788 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:12:38 smithi050 ceph-mon[32371]: 5.2 scrub ok 2024-09-18T08:12:38.788 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:12:38 smithi050 ceph-mon[32371]: from='client.? 172.21.15.7:0/3428594654' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-18T08:12:38.854 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0 2024-09-18T08:12:38.859 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:12:38 smithi007 ceph-mon[29459]: 4.0 scrub starts 2024-09-18T08:12:38.859 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:12:38 smithi007 ceph-mon[29459]: 4.0 scrub ok 2024-09-18T08:12:38.859 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:12:38 smithi007 ceph-mon[29459]: 5.2 scrub starts 2024-09-18T08:12:38.859 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:12:38 smithi007 ceph-mon[29459]: 5.2 scrub ok 2024-09-18T08:12:38.859 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:12:38 smithi007 ceph-mon[29459]: from='client.? 172.21.15.7:0/3428594654' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-18T08:12:38.860 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:12:38 smithi007 ceph-mon[39249]: 4.0 scrub starts 2024-09-18T08:12:38.860 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:12:38 smithi007 ceph-mon[39249]: 4.0 scrub ok 2024-09-18T08:12:38.860 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:12:38 smithi007 ceph-mon[39249]: 5.2 scrub starts 2024-09-18T08:12:38.860 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:12:38 smithi007 ceph-mon[39249]: 5.2 scrub ok 2024-09-18T08:12:38.860 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:12:38 smithi007 ceph-mon[39249]: from='client.? 172.21.15.7:0/3428594654' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-18T08:12:38.891 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:12:38.892 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:12:38.892 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0' 2024-09-18T08:12:38.972 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2024-09-18T08:12:38.972 INFO:teuthology.orchestra.run.smithi050.stdout: "size": 20, 2024-09-18T08:12:38.972 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 600, 2024-09-18T08:12:38.972 INFO:teuthology.orchestra.run.smithi050.stdout: "ops": [ 2024-09-18T08:12:38.973 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:12:38.973 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.14805.0:38000 7.7 7:e509d896:::benchmark_data_smithi007_73155_object37999:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e94)", 2024-09-18T08:12:38.973 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-09-18T08:04:23.049009+0000", 2024-09-18T08:12:38.973 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 495.908789313, 2024-09-18T08:12:38.973 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.027565702000000001, 2024-09-18T08:12:38.973 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-09-18T08:12:38.973 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-09-18T08:12:38.973 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T08:12:38.973 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2024-09-18T08:12:38.973 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.14805", 2024-09-18T08:12:38.974 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "172.21.15.7:0/640320436", 2024-09-18T08:12:38.974 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 38000 2024-09-18T08:12:38.974 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:12:38.974 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-09-18T08:12:38.974 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:12:38.974 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-09-18T08:12:38.974 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:23.049009+0000", 2024-09-18T08:12:38.974 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-18T08:12:38.974 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:12:38.974 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:12:38.975 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-09-18T08:12:38.975 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:23.049009+0000", 2024-09-18T08:12:38.975 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-18T08:12:38.975 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:12:38.975 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:12:38.975 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-09-18T08:12:38.975 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:23.049010+0000", 2024-09-18T08:12:38.975 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.11e-06 2024-09-18T08:12:38.975 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:12:38.975 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:12:38.976 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-09-18T08:12:38.976 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:23.068284+0000", 2024-09-18T08:12:38.976 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.019274264999999999 2024-09-18T08:12:38.976 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:12:38.976 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:12:38.976 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-09-18T08:12:38.976 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:23.068323+0000", 2024-09-18T08:12:38.976 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 3.8167999999999998e-05 2024-09-18T08:12:38.976 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:12:38.976 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:12:38.976 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-09-18T08:12:38.977 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:23.068329+0000", 2024-09-18T08:12:38.977 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 6.331e-06 2024-09-18T08:12:38.977 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:12:38.977 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:12:38.977 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-18T08:12:38.977 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:23.068399+0000", 2024-09-18T08:12:38.977 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 7.0055999999999995e-05 2024-09-18T08:12:38.977 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:12:38.977 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:12:38.977 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-09-18T08:12:38.977 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:23.068503+0000", 2024-09-18T08:12:38.977 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000104206 2024-09-18T08:12:38.978 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:12:38.978 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:12:38.978 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "waiting for subops from 0,3", 2024-09-18T08:12:38.978 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:23.068640+0000", 2024-09-18T08:12:38.978 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00013645000000000001 2024-09-18T08:12:38.978 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:12:38.978 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:12:38.978 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "op_commit", 2024-09-18T08:12:38.978 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:23.073270+0000", 2024-09-18T08:12:38.978 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0046298950000000002 2024-09-18T08:12:38.978 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:12:38.978 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:12:38.979 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:12:38.979 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:23.073311+0000", 2024-09-18T08:12:38.979 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4.1406999999999998e-05 2024-09-18T08:12:38.979 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:12:38.979 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:12:38.979 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:12:38.979 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:23.076521+0000", 2024-09-18T08:12:38.979 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.003209862 2024-09-18T08:12:38.979 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:12:38.979 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:12:38.979 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "commit_sent", 2024-09-18T08:12:38.979 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:23.076552+0000", 2024-09-18T08:12:38.980 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 3.1698e-05 2024-09-18T08:12:38.980 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:12:38.980 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:12:38.980 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-09-18T08:12:38.980 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:23.076575+0000", 2024-09-18T08:12:38.980 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 2.2254000000000001e-05 2024-09-18T08:12:38.980 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-18T08:12:38.980 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-09-18T08:12:38.980 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-18T08:12:38.981 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:12:38.981 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:12:38.981 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.14805.0:39064 7.7 7:ef833039:::benchmark_data_smithi007_73155_object39063:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e95)", 2024-09-18T08:12:38.981 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-09-18T08:04:23.658960+0000", 2024-09-18T08:12:38.981 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 495.29883804100001, 2024-09-18T08:12:38.981 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.029075624000000001, 2024-09-18T08:12:38.981 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-09-18T08:12:38.981 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-09-18T08:12:38.981 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T08:12:38.981 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2024-09-18T08:12:38.981 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.14805", 2024-09-18T08:12:38.982 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "172.21.15.7:0/640320436", 2024-09-18T08:12:38.982 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 39064 2024-09-18T08:12:38.982 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:12:38.982 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-09-18T08:12:38.982 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:12:38.982 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-09-18T08:12:38.982 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:23.658960+0000", 2024-09-18T08:12:38.982 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-18T08:12:38.982 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:12:38.982 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:12:38.982 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-09-18T08:12:38.983 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:23.658960+0000", 2024-09-18T08:12:38.983 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-18T08:12:38.983 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:12:38.983 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:12:38.983 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-09-18T08:12:38.983 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:23.658961+0000", 2024-09-18T08:12:38.983 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 6.7899999999999998e-07 2024-09-18T08:12:38.983 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:12:38.983 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:12:38.983 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-09-18T08:12:38.983 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:23.658991+0000", 2024-09-18T08:12:38.983 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 3.0168999999999999e-05 2024-09-18T08:12:38.984 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:12:38.984 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:12:38.984 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-09-18T08:12:38.984 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:23.659001+0000", 2024-09-18T08:12:38.984 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 9.6220000000000004e-06 2024-09-18T08:12:38.984 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:12:38.984 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:12:38.984 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-09-18T08:12:38.984 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:23.659010+0000", 2024-09-18T08:12:38.984 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 8.9160000000000007e-06 2024-09-18T08:12:38.984 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:12:38.984 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:12:38.985 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-18T08:12:38.985 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:23.661923+0000", 2024-09-18T08:12:38.985 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.002913279 2024-09-18T08:12:38.985 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:12:38.985 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:12:38.985 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-09-18T08:12:38.985 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:23.662030+0000", 2024-09-18T08:12:38.985 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00010665700000000001 2024-09-18T08:12:38.985 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:12:38.985 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:12:38.985 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "waiting for subops from 0,3", 2024-09-18T08:12:38.985 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:23.662167+0000", 2024-09-18T08:12:38.985 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00013728300000000001 2024-09-18T08:12:38.985 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:12:38.986 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:12:38.986 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "op_commit", 2024-09-18T08:12:38.986 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:23.663255+0000", 2024-09-18T08:12:38.986 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0010883900000000001 2024-09-18T08:12:38.986 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:12:38.986 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:12:38.986 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:12:38.986 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:23.671279+0000", 2024-09-18T08:12:38.986 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0080233749999999993 2024-09-18T08:12:38.986 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:12:38.986 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:12:38.986 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:12:38.986 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:23.687965+0000", 2024-09-18T08:12:38.986 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.016686590000000001 2024-09-18T08:12:38.987 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:12:38.987 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:12:38.987 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "commit_sent", 2024-09-18T08:12:38.987 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:23.688005+0000", 2024-09-18T08:12:38.987 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 3.9604000000000002e-05 2024-09-18T08:12:38.987 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:12:38.987 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:12:38.987 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-09-18T08:12:38.987 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:23.688036+0000", 2024-09-18T08:12:38.987 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 3.1059999999999997e-05 2024-09-18T08:12:38.987 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-18T08:12:38.987 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-09-18T08:12:38.987 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-18T08:12:38.988 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:12:38.988 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:12:38.988 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.14805.0:39805 7.7 7:e9cd36b3:::benchmark_data_smithi007_73155_object39804:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e95)", 2024-09-18T08:12:38.988 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-09-18T08:04:24.061655+0000", 2024-09-18T08:12:38.988 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 494.89614336, 2024-09-18T08:12:38.988 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.036883131, 2024-09-18T08:12:38.988 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-09-18T08:12:38.988 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-09-18T08:12:38.988 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T08:12:38.988 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2024-09-18T08:12:38.988 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.14805", 2024-09-18T08:12:38.989 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "172.21.15.7:0/640320436", 2024-09-18T08:12:38.989 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 39805 2024-09-18T08:12:38.989 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:12:38.989 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-09-18T08:12:38.989 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:12:38.989 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-09-18T08:12:38.989 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:24.061655+0000", 2024-09-18T08:12:38.989 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-18T08:12:38.989 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:12:38.989 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:12:38.989 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-09-18T08:12:38.990 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:24.061655+0000", 2024-09-18T08:12:38.990 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-18T08:12:38.990 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:12:38.990 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:12:38.990 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-09-18T08:12:38.990 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:24.061656+0000", 2024-09-18T08:12:38.990 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.4449999999999999e-06 2024-09-18T08:12:38.990 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:12:38.990 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:12:38.990 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-09-18T08:12:38.991 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:24.061686+0000", 2024-09-18T08:12:38.991 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 2.9984e-05 2024-09-18T08:12:38.991 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:12:38.991 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:12:38.991 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-09-18T08:12:38.991 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:24.061699+0000", 2024-09-18T08:12:38.991 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.2850000000000001e-05 2024-09-18T08:12:38.991 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:12:38.991 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:12:38.991 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-09-18T08:12:38.991 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:24.061706+0000", 2024-09-18T08:12:38.991 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 6.8560000000000001e-06 2024-09-18T08:12:38.991 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:12:38.992 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:12:38.992 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-18T08:12:38.992 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:24.061755+0000", 2024-09-18T08:12:38.992 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4.9001000000000001e-05 2024-09-18T08:12:38.992 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:12:38.992 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:12:38.992 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-09-18T08:12:38.992 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:24.061896+0000", 2024-09-18T08:12:38.992 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00014040499999999999 2024-09-18T08:12:38.993 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:12:38.993 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:12:38.993 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "waiting for subops from 0,3", 2024-09-18T08:12:38.993 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:24.062006+0000", 2024-09-18T08:12:38.993 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000110813 2024-09-18T08:12:38.993 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:12:38.993 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:12:38.993 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:12:38.993 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:24.065677+0000", 2024-09-18T08:12:38.993 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0036702800000000002 2024-09-18T08:12:38.993 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:12:38.994 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:12:38.994 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "op_commit", 2024-09-18T08:12:38.994 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:24.065694+0000", 2024-09-18T08:12:38.994 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.7876e-05 2024-09-18T08:12:38.994 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:12:38.994 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:12:38.994 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:12:38.994 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:24.098357+0000", 2024-09-18T08:12:38.994 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.032662140999999999 2024-09-18T08:12:38.994 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:12:38.994 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:12:38.994 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "commit_sent", 2024-09-18T08:12:38.995 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:24.098447+0000", 2024-09-18T08:12:38.995 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 9.0724999999999996e-05 2024-09-18T08:12:38.995 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:12:38.995 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:12:38.995 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-09-18T08:12:38.995 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:24.098538+0000", 2024-09-18T08:12:38.995 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 9.0754999999999994e-05 2024-09-18T08:12:38.995 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-18T08:12:38.995 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-09-18T08:12:38.995 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-18T08:12:38.995 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:12:38.995 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:12:38.995 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.14805.0:39807 7.d 7:bf1e27dc:::benchmark_data_smithi007_73155_object39806:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e95)", 2024-09-18T08:12:38.996 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-09-18T08:04:24.061827+0000", 2024-09-18T08:12:38.996 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 494.89597106500003, 2024-09-18T08:12:38.996 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.037358672000000002, 2024-09-18T08:12:38.996 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-09-18T08:12:38.996 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-09-18T08:12:38.996 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T08:12:38.996 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2024-09-18T08:12:38.996 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.14805", 2024-09-18T08:12:38.996 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "172.21.15.7:0/640320436", 2024-09-18T08:12:38.996 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 39807 2024-09-18T08:12:38.996 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:12:38.997 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-09-18T08:12:38.997 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:12:38.997 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-09-18T08:12:38.997 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:24.061827+0000", 2024-09-18T08:12:38.997 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-18T08:12:38.997 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:12:38.997 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:12:38.997 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-09-18T08:12:38.997 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:24.061827+0000", 2024-09-18T08:12:38.997 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-18T08:12:38.997 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:12:38.997 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:12:38.998 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-09-18T08:12:38.998 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:24.061828+0000", 2024-09-18T08:12:38.998 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 7.9899999999999999e-07 2024-09-18T08:12:38.998 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:12:38.998 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:12:38.998 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-09-18T08:12:38.998 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:24.061888+0000", 2024-09-18T08:12:38.998 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 5.9497000000000003e-05 2024-09-18T08:12:38.998 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:12:38.998 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:12:38.998 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-09-18T08:12:38.998 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:24.061904+0000", 2024-09-18T08:12:38.998 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.6393999999999999e-05 2024-09-18T08:12:38.999 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:12:38.999 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:12:38.999 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-09-18T08:12:38.999 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:24.061912+0000", 2024-09-18T08:12:38.999 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 8.0909999999999993e-06 2024-09-18T08:12:38.999 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:12:38.999 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:12:38.999 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-18T08:12:38.999 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:24.061968+0000", 2024-09-18T08:12:38.999 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 5.5543e-05 2024-09-18T08:12:38.999 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:12:38.999 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:12:38.999 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-09-18T08:12:39.000 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:24.062061+0000", 2024-09-18T08:12:39.000 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 9.3558000000000001e-05 2024-09-18T08:12:39.000 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:12:39.000 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:12:39.000 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "waiting for subops from 0,1", 2024-09-18T08:12:39.000 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:24.062163+0000", 2024-09-18T08:12:39.000 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000101903 2024-09-18T08:12:39.000 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:12:39.000 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:12:39.000 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "op_commit", 2024-09-18T08:12:39.000 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:24.064018+0000", 2024-09-18T08:12:39.000 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.001855071 2024-09-18T08:12:39.001 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:12:39.001 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:12:39.001 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:12:39.001 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:24.064283+0000", 2024-09-18T08:12:39.001 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000265173 2024-09-18T08:12:39.001 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:12:39.001 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:12:39.001 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:12:39.001 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:24.099043+0000", 2024-09-18T08:12:39.001 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.034760159999999998 2024-09-18T08:12:39.001 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:12:39.001 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:12:39.001 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "commit_sent", 2024-09-18T08:12:39.001 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:24.099100+0000", 2024-09-18T08:12:39.002 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 5.6147000000000003e-05 2024-09-18T08:12:39.002 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:12:39.002 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:12:39.002 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-09-18T08:12:39.002 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:24.099186+0000", 2024-09-18T08:12:39.002 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 8.6335999999999995e-05 2024-09-18T08:12:39.002 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-18T08:12:39.002 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-09-18T08:12:39.002 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-18T08:12:39.002 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:12:39.002 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:12:39.003 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.14805.0:39808 7.d 7:b98cab8f:::benchmark_data_smithi007_73155_object39807:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e95)", 2024-09-18T08:12:39.003 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-09-18T08:04:24.067964+0000", 2024-09-18T08:12:39.003 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 494.88983462800002, 2024-09-18T08:12:39.003 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.034258180999999999, 2024-09-18T08:12:39.003 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-09-18T08:12:39.003 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-09-18T08:12:39.003 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T08:12:39.003 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2024-09-18T08:12:39.003 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.14805", 2024-09-18T08:12:39.003 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "172.21.15.7:0/640320436", 2024-09-18T08:12:39.003 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 39808 2024-09-18T08:12:39.003 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:12:39.004 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-09-18T08:12:39.004 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:12:39.004 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-09-18T08:12:39.004 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:24.067964+0000", 2024-09-18T08:12:39.004 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-18T08:12:39.004 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:12:39.004 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:12:39.004 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-09-18T08:12:39.004 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:24.067964+0000", 2024-09-18T08:12:39.004 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-18T08:12:39.004 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:12:39.004 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:12:39.004 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-09-18T08:12:39.004 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:24.067965+0000", 2024-09-18T08:12:39.005 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.4750000000000001e-06 2024-09-18T08:12:39.005 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:12:39.005 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:12:39.005 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-09-18T08:12:39.005 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:24.067992+0000", 2024-09-18T08:12:39.005 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 2.6760000000000001e-05 2024-09-18T08:12:39.005 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:12:39.005 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:12:39.005 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-09-18T08:12:39.005 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:24.068004+0000", 2024-09-18T08:12:39.005 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.2456000000000001e-05 2024-09-18T08:12:39.005 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:12:39.005 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:12:39.006 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-09-18T08:12:39.006 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:24.068012+0000", 2024-09-18T08:12:39.006 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 8.0779999999999996e-06 2024-09-18T08:12:39.006 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:12:39.006 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:12:39.006 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-18T08:12:39.006 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:24.068072+0000", 2024-09-18T08:12:39.006 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 5.9299000000000003e-05 2024-09-18T08:12:39.006 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:12:39.006 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:12:39.006 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-09-18T08:12:39.006 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:24.068164+0000", 2024-09-18T08:12:39.006 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 9.2053999999999997e-05 2024-09-18T08:12:39.006 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:12:39.007 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:12:39.007 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "waiting for subops from 0,1", 2024-09-18T08:12:39.007 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:24.096945+0000", 2024-09-18T08:12:39.007 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.028781128 2024-09-18T08:12:39.007 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:12:39.007 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:12:39.007 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "op_commit", 2024-09-18T08:12:39.007 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:24.099254+0000", 2024-09-18T08:12:39.007 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.002309218 2024-09-18T08:12:39.007 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:12:39.007 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:12:39.007 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:12:39.007 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:24.100291+0000", 2024-09-18T08:12:39.007 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0010365280000000001 2024-09-18T08:12:39.008 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:12:39.008 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:12:39.008 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:12:39.008 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:24.102125+0000", 2024-09-18T08:12:39.008 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0018346599999999999 2024-09-18T08:12:39.008 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:12:39.008 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:12:39.008 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "commit_sent", 2024-09-18T08:12:39.008 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:24.102184+0000", 2024-09-18T08:12:39.008 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 5.8965000000000002e-05 2024-09-18T08:12:39.008 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:12:39.008 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:12:39.008 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-09-18T08:12:39.008 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:24.102222+0000", 2024-09-18T08:12:39.009 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 3.756e-05 2024-09-18T08:12:39.009 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-18T08:12:39.009 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-09-18T08:12:39.009 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-18T08:12:39.009 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:12:39.009 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:12:39.009 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_repop_reply(client.14805.0:39807 7.d e95/83 ondisk, result = 0)", 2024-09-18T08:12:39.009 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-09-18T08:04:24.069321+0000", 2024-09-18T08:12:39.009 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 494.88847726400002, 2024-09-18T08:12:39.009 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.029872724999999999, 2024-09-18T08:12:39.009 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-09-18T08:12:39.009 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-09-18T08:12:39.009 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "started", 2024-09-18T08:12:39.009 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-09-18T08:12:39.010 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:12:39.010 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-09-18T08:12:39.010 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:24.069321+0000", 2024-09-18T08:12:39.010 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-18T08:12:39.010 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:12:39.010 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:12:39.010 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-09-18T08:12:39.010 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:24.069321+0000", 2024-09-18T08:12:39.010 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-18T08:12:39.010 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:12:39.010 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:12:39.010 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-09-18T08:12:39.010 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:24.069322+0000", 2024-09-18T08:12:39.011 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 8.3799999999999996e-07 2024-09-18T08:12:39.011 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:12:39.011 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:12:39.011 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-09-18T08:12:39.011 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:24.069327+0000", 2024-09-18T08:12:39.011 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4.8679999999999998e-06 2024-09-18T08:12:39.011 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:12:39.011 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:12:39.011 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-09-18T08:12:39.011 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:24.098180+0000", 2024-09-18T08:12:39.011 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.028853407000000001 2024-09-18T08:12:39.011 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:12:39.011 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:12:39.011 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-09-18T08:12:39.012 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:24.098189+0000", 2024-09-18T08:12:39.012 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 8.8850000000000005e-06 2024-09-18T08:12:39.012 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:12:39.012 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:12:39.012 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-18T08:12:39.012 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:24.099029+0000", 2024-09-18T08:12:39.012 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00084036500000000004 2024-09-18T08:12:39.012 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:12:39.012 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:12:39.012 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-09-18T08:12:39.012 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:24.099037+0000", 2024-09-18T08:12:39.012 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 8.0150000000000003e-06 2024-09-18T08:12:39.012 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:12:39.012 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:12:39.013 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-09-18T08:12:39.013 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:24.099194+0000", 2024-09-18T08:12:39.013 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00015634700000000001 2024-09-18T08:12:39.013 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-18T08:12:39.013 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-09-18T08:12:39.013 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-18T08:12:39.013 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:12:39.013 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:12:39.013 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.14805.0:39817 7.c 7:3826d3a8:::benchmark_data_smithi007_73155_object39816:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e96)", 2024-09-18T08:12:39.013 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-09-18T08:04:24.077849+0000", 2024-09-18T08:12:39.013 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 494.87994931600002, 2024-09-18T08:12:39.013 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.028025705000000001, 2024-09-18T08:12:39.013 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-09-18T08:12:39.014 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-09-18T08:12:39.014 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T08:12:39.014 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2024-09-18T08:12:39.014 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.14805", 2024-09-18T08:12:39.014 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "172.21.15.7:0/640320436", 2024-09-18T08:12:39.014 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 39817 2024-09-18T08:12:39.014 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:12:39.014 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-09-18T08:12:39.014 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:12:39.014 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-09-18T08:12:39.014 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:24.077849+0000", 2024-09-18T08:12:39.014 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-18T08:12:39.014 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:12:39.014 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:12:39.014 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-09-18T08:12:39.015 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:24.077849+0000", 2024-09-18T08:12:39.015 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-18T08:12:39.015 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:12:39.015 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:12:39.015 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-09-18T08:12:39.015 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:24.077851+0000", 2024-09-18T08:12:39.015 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 2.322e-06 2024-09-18T08:12:39.015 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:12:39.015 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:12:39.015 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-09-18T08:12:39.015 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:24.077879+0000", 2024-09-18T08:12:39.015 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 2.7596000000000001e-05 2024-09-18T08:12:39.015 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:12:39.015 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:12:39.016 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-09-18T08:12:39.016 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:24.097805+0000", 2024-09-18T08:12:39.016 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.019925552999999999 2024-09-18T08:12:39.016 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:12:39.016 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:12:39.016 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-09-18T08:12:39.016 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:24.097820+0000", 2024-09-18T08:12:39.016 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.5705999999999998e-05 2024-09-18T08:12:39.016 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:12:39.016 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:12:39.016 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-18T08:12:39.016 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:24.097984+0000", 2024-09-18T08:12:39.016 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00016333499999999999 2024-09-18T08:12:39.017 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:12:39.017 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:12:39.017 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "op must wait for map", 2024-09-18T08:12:39.017 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:24.097990+0000", 2024-09-18T08:12:39.017 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 6.81e-06 2024-09-18T08:12:39.017 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:12:39.017 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:12:39.017 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-18T08:12:39.017 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:24.102003+0000", 2024-09-18T08:12:39.017 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0040126160000000001 2024-09-18T08:12:39.017 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:12:39.018 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:12:39.018 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-09-18T08:12:39.018 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:24.102155+0000", 2024-09-18T08:12:39.018 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00015247500000000001 2024-09-18T08:12:39.018 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:12:39.018 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:12:39.018 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "waiting for subops from 3,7", 2024-09-18T08:12:39.018 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:24.102305+0000", 2024-09-18T08:12:39.018 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000149872 2024-09-18T08:12:39.018 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:12:39.018 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:12:39.018 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "op_commit", 2024-09-18T08:12:39.018 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:24.105200+0000", 2024-09-18T08:12:39.019 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3 2024-09-18T08:12:39.020 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0028950730000000002 2024-09-18T08:12:39.020 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:12:39.020 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:12:39.020 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:12:39.021 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:24.105248+0000", 2024-09-18T08:12:39.021 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4.7787999999999997e-05 2024-09-18T08:12:39.021 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:12:39.021 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:12:39.021 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:12:39.021 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:24.105772+0000", 2024-09-18T08:12:39.021 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00052425699999999998 2024-09-18T08:12:39.021 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:12:39.021 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:12:39.021 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "commit_sent", 2024-09-18T08:12:39.021 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:24.105826+0000", 2024-09-18T08:12:39.021 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 5.393e-05 2024-09-18T08:12:39.021 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:12:39.022 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:12:39.022 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-09-18T08:12:39.022 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:24.105875+0000", 2024-09-18T08:12:39.022 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4.8371999999999997e-05 2024-09-18T08:12:39.022 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-18T08:12:39.022 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-09-18T08:12:39.022 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-18T08:12:39.022 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:12:39.022 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:12:39.022 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.14805.0:43731 7.d 7:b3fc8faf:::benchmark_data_smithi007_73155_object43730:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e97)", 2024-09-18T08:12:39.022 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-09-18T08:04:26.432079+0000", 2024-09-18T08:12:39.022 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 492.52571904199999, 2024-09-18T08:12:39.022 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.027497293999999999, 2024-09-18T08:12:39.023 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-09-18T08:12:39.023 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-09-18T08:12:39.023 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T08:12:39.023 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2024-09-18T08:12:39.023 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.14805", 2024-09-18T08:12:39.023 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "172.21.15.7:0/640320436", 2024-09-18T08:12:39.023 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 43731 2024-09-18T08:12:39.023 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:12:39.023 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-09-18T08:12:39.023 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:12:39.023 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-09-18T08:12:39.023 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:26.432079+0000", 2024-09-18T08:12:39.024 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-18T08:12:39.024 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:12:39.024 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:12:39.024 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-09-18T08:12:39.024 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:26.432079+0000", 2024-09-18T08:12:39.024 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-18T08:12:39.024 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:12:39.024 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:12:39.024 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-09-18T08:12:39.024 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:26.432080+0000", 2024-09-18T08:12:39.024 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.128e-06 2024-09-18T08:12:39.024 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:12:39.024 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:12:39.025 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-09-18T08:12:39.025 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:26.432153+0000", 2024-09-18T08:12:39.025 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 7.2521999999999994e-05 2024-09-18T08:12:39.025 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:12:39.025 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:12:39.025 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-09-18T08:12:39.025 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:26.432163+0000", 2024-09-18T08:12:39.025 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.0115999999999999e-05 2024-09-18T08:12:39.025 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:12:39.025 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:12:39.025 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-09-18T08:12:39.025 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:26.432170+0000", 2024-09-18T08:12:39.025 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 7.0729999999999999e-06 2024-09-18T08:12:39.025 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:12:39.026 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:12:39.026 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-18T08:12:39.026 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:26.432225+0000", 2024-09-18T08:12:39.026 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 5.4463999999999999e-05 2024-09-18T08:12:39.026 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:12:39.026 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:12:39.026 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-09-18T08:12:39.026 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:26.432342+0000", 2024-09-18T08:12:39.026 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000117118 2024-09-18T08:12:39.026 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:12:39.026 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:12:39.026 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "waiting for subops from 0,1", 2024-09-18T08:12:39.026 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:26.432467+0000", 2024-09-18T08:12:39.027 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00012505700000000001 2024-09-18T08:12:39.027 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:12:39.027 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:12:39.027 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "op_commit", 2024-09-18T08:12:39.027 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:26.433881+0000", 2024-09-18T08:12:39.027 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.001413812 2024-09-18T08:12:39.027 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:12:39.027 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:12:39.027 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:12:39.027 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:26.434589+0000", 2024-09-18T08:12:39.027 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00070883600000000001 2024-09-18T08:12:39.027 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:12:39.028 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:12:39.028 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:12:39.028 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:26.459469+0000", 2024-09-18T08:12:39.028 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.024879951000000001 2024-09-18T08:12:39.028 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:12:39.028 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:12:39.028 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "commit_sent", 2024-09-18T08:12:39.028 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:26.459527+0000", 2024-09-18T08:12:39.028 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 5.7126999999999997e-05 2024-09-18T08:12:39.028 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:12:39.028 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:12:39.028 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-09-18T08:12:39.029 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:26.459577+0000", 2024-09-18T08:12:39.029 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 5.0090000000000003e-05 2024-09-18T08:12:39.029 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-18T08:12:39.029 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-09-18T08:12:39.029 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-18T08:12:39.029 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:12:39.029 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:12:39.029 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.14805.0:63861 7.d 7:bf91474b:::benchmark_data_smithi007_73155_object63860:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e100)", 2024-09-18T08:12:39.029 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-09-18T08:04:41.125989+0000", 2024-09-18T08:12:39.029 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 477.83180974999999, 2024-09-18T08:12:39.029 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.031566575999999999, 2024-09-18T08:12:39.029 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-09-18T08:12:39.029 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-09-18T08:12:39.030 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T08:12:39.030 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2024-09-18T08:12:39.030 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.14805", 2024-09-18T08:12:39.030 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "172.21.15.7:0/640320436", 2024-09-18T08:12:39.030 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 63861 2024-09-18T08:12:39.030 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:12:39.030 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-09-18T08:12:39.030 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:12:39.030 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-09-18T08:12:39.030 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:41.125989+0000", 2024-09-18T08:12:39.030 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-18T08:12:39.030 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:12:39.030 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:12:39.030 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-09-18T08:12:39.031 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:41.125989+0000", 2024-09-18T08:12:39.031 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-18T08:12:39.031 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:12:39.031 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:12:39.031 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-09-18T08:12:39.031 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:41.125990+0000", 2024-09-18T08:12:39.031 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.6109999999999999e-06 2024-09-18T08:12:39.031 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:12:39.031 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:12:39.031 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-09-18T08:12:39.031 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:41.126017+0000", 2024-09-18T08:12:39.031 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 2.6594999999999999e-05 2024-09-18T08:12:39.031 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:12:39.031 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:12:39.032 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-09-18T08:12:39.032 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:41.126035+0000", 2024-09-18T08:12:39.032 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.7842999999999998e-05 2024-09-18T08:12:39.032 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:12:39.032 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:12:39.032 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-09-18T08:12:39.032 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:41.126043+0000", 2024-09-18T08:12:39.032 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 8.8489999999999995e-06 2024-09-18T08:12:39.032 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:12:39.032 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:12:39.032 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-18T08:12:39.032 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:41.126104+0000", 2024-09-18T08:12:39.032 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 6.0448999999999997e-05 2024-09-18T08:12:39.033 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:12:39.033 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:12:39.033 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-09-18T08:12:39.033 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:41.126225+0000", 2024-09-18T08:12:39.033 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000121101 2024-09-18T08:12:39.033 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:12:39.033 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:12:39.033 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "waiting for subops from 0,1", 2024-09-18T08:12:39.033 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:41.126366+0000", 2024-09-18T08:12:39.033 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00014059899999999999 2024-09-18T08:12:39.033 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:12:39.033 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:12:39.033 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "op_commit", 2024-09-18T08:12:39.034 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:41.129750+0000", 2024-09-18T08:12:39.034 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.003383863 2024-09-18T08:12:39.034 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:12:39.034 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:12:39.034 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:12:39.034 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:41.140266+0000", 2024-09-18T08:12:39.034 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.010516266999999999 2024-09-18T08:12:39.034 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:12:39.034 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:12:39.034 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:12:39.034 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:41.157441+0000", 2024-09-18T08:12:39.034 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.017174891000000001 2024-09-18T08:12:39.034 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:12:39.034 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:12:39.035 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "commit_sent", 2024-09-18T08:12:39.035 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:41.157509+0000", 2024-09-18T08:12:39.035 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 6.8627999999999994e-05 2024-09-18T08:12:39.035 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:12:39.035 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:12:39.035 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-09-18T08:12:39.035 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:41.157555+0000", 2024-09-18T08:12:39.035 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4.5880000000000001e-05 2024-09-18T08:12:39.035 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-18T08:12:39.035 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-09-18T08:12:39.035 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-18T08:12:39.035 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:12:39.035 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:12:39.036 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.14805.0:63862 7.d 7:b1f318bc:::benchmark_data_smithi007_73155_object63861:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e100)", 2024-09-18T08:12:39.036 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-09-18T08:04:41.126934+0000", 2024-09-18T08:12:39.036 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 477.83086461300002, 2024-09-18T08:12:39.036 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.030729600999999999, 2024-09-18T08:12:39.036 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-09-18T08:12:39.036 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-09-18T08:12:39.036 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T08:12:39.036 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2024-09-18T08:12:39.036 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.14805", 2024-09-18T08:12:39.036 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "172.21.15.7:0/640320436", 2024-09-18T08:12:39.036 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 63862 2024-09-18T08:12:39.036 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:12:39.037 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-09-18T08:12:39.037 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:12:39.037 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-09-18T08:12:39.037 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:41.126934+0000", 2024-09-18T08:12:39.037 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-18T08:12:39.037 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:12:39.037 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:12:39.037 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-09-18T08:12:39.037 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:41.126934+0000", 2024-09-18T08:12:39.037 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-18T08:12:39.037 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:12:39.037 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:12:39.037 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-09-18T08:12:39.038 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:41.126935+0000", 2024-09-18T08:12:39.038 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 7.9599999999999998e-07 2024-09-18T08:12:39.038 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:12:39.038 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:12:39.038 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-09-18T08:12:39.038 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:41.126989+0000", 2024-09-18T08:12:39.038 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 5.4783e-05 2024-09-18T08:12:39.038 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:12:39.038 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:12:39.038 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-09-18T08:12:39.038 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:41.126999+0000", 2024-09-18T08:12:39.038 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 9.8889999999999997e-06 2024-09-18T08:12:39.038 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:12:39.038 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:12:39.039 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-09-18T08:12:39.039 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:41.127007+0000", 2024-09-18T08:12:39.039 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 7.4399999999999999e-06 2024-09-18T08:12:39.039 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:12:39.039 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:12:39.039 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-18T08:12:39.039 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:41.128171+0000", 2024-09-18T08:12:39.039 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0011642600000000001 2024-09-18T08:12:39.039 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:12:39.039 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:12:39.039 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-09-18T08:12:39.039 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:41.128287+0000", 2024-09-18T08:12:39.039 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000116508 2024-09-18T08:12:39.040 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:12:39.040 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:12:39.040 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "waiting for subops from 0,1", 2024-09-18T08:12:39.040 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:41.128375+0000", 2024-09-18T08:12:39.040 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 8.7300999999999996e-05 2024-09-18T08:12:39.040 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:12:39.040 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:12:39.040 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "op_commit", 2024-09-18T08:12:39.040 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:41.130919+0000", 2024-09-18T08:12:39.040 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0025447450000000002 2024-09-18T08:12:39.040 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:12:39.040 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:12:39.040 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:12:39.040 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:41.140334+0000", 2024-09-18T08:12:39.041 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0094146549999999992 2024-09-18T08:12:39.041 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:12:39.041 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:12:39.041 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:12:39.041 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:41.157602+0000", 2024-09-18T08:12:39.041 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.017267966999999999 2024-09-18T08:12:39.041 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:12:39.041 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:12:39.041 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "commit_sent", 2024-09-18T08:12:39.041 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:41.157636+0000", 2024-09-18T08:12:39.041 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 3.4214000000000001e-05 2024-09-18T08:12:39.041 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:12:39.041 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:12:39.042 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-09-18T08:12:39.042 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:41.157663+0000", 2024-09-18T08:12:39.042 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 2.7042999999999998e-05 2024-09-18T08:12:39.042 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-18T08:12:39.042 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-09-18T08:12:39.042 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-18T08:12:39.042 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:12:39.042 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:12:39.042 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.14805.0:63864 7.d 7:bb7b1761:::benchmark_data_smithi007_73155_object63863:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e100)", 2024-09-18T08:12:39.042 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-09-18T08:04:41.127581+0000", 2024-09-18T08:12:39.042 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 477.83021715000001, 2024-09-18T08:12:39.042 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.030231638000000002, 2024-09-18T08:12:39.042 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-09-18T08:12:39.043 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-09-18T08:12:39.043 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T08:12:39.043 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2024-09-18T08:12:39.043 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.14805", 2024-09-18T08:12:39.043 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "172.21.15.7:0/640320436", 2024-09-18T08:12:39.043 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 63864 2024-09-18T08:12:39.043 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:12:39.043 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-09-18T08:12:39.043 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:12:39.043 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-09-18T08:12:39.043 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:41.127581+0000", 2024-09-18T08:12:39.043 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-18T08:12:39.043 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:12:39.043 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:12:39.044 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-09-18T08:12:39.044 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:41.127581+0000", 2024-09-18T08:12:39.044 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-18T08:12:39.044 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:12:39.044 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:12:39.044 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-09-18T08:12:39.044 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:41.127582+0000", 2024-09-18T08:12:39.044 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 7.8999999999999995e-07 2024-09-18T08:12:39.044 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:12:39.044 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:12:39.045 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-09-18T08:12:39.045 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:41.127648+0000", 2024-09-18T08:12:39.045 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 6.5618999999999996e-05 2024-09-18T08:12:39.045 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:12:39.045 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:12:39.045 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-09-18T08:12:39.045 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:41.127658+0000", 2024-09-18T08:12:39.045 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.0192e-05 2024-09-18T08:12:39.045 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:12:39.045 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:12:39.045 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-09-18T08:12:39.045 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:41.127663+0000", 2024-09-18T08:12:39.045 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 5.2800000000000003e-06 2024-09-18T08:12:39.046 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:12:39.046 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:12:39.046 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-18T08:12:39.046 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:41.129783+0000", 2024-09-18T08:12:39.046 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0021199740000000002 2024-09-18T08:12:39.046 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:12:39.046 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:12:39.046 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-09-18T08:12:39.046 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:41.129885+0000", 2024-09-18T08:12:39.046 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000101987 2024-09-18T08:12:39.046 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:12:39.046 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:12:39.046 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "waiting for subops from 0,1", 2024-09-18T08:12:39.046 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:41.129980+0000", 2024-09-18T08:12:39.047 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 9.4480000000000003e-05 2024-09-18T08:12:39.047 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:12:39.047 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:12:39.047 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "op_commit", 2024-09-18T08:12:39.047 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:41.131035+0000", 2024-09-18T08:12:39.047 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.001055884 2024-09-18T08:12:39.047 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:12:39.047 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:12:39.047 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:12:39.047 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:41.140376+0000", 2024-09-18T08:12:39.047 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0093401439999999999 2024-09-18T08:12:39.047 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:12:39.047 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:12:39.047 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:12:39.048 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:41.157713+0000", 2024-09-18T08:12:39.048 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.017337676999999999 2024-09-18T08:12:39.048 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:12:39.048 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:12:39.048 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "commit_sent", 2024-09-18T08:12:39.048 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:41.157771+0000", 2024-09-18T08:12:39.048 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 5.7674999999999999e-05 2024-09-18T08:12:39.048 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:12:39.048 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:12:39.048 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-09-18T08:12:39.048 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:41.157813+0000", 2024-09-18T08:12:39.048 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4.1936e-05 2024-09-18T08:12:39.048 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-18T08:12:39.048 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-09-18T08:12:39.048 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-18T08:12:39.049 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:12:39.049 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:12:39.049 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.14805.0:63883 7.7 7:ebafa8b8:::benchmark_data_smithi007_73155_object63882:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e100)", 2024-09-18T08:12:39.049 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-09-18T08:04:41.154209+0000", 2024-09-18T08:12:39.049 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 477.80358895799998, 2024-09-18T08:12:39.049 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.042338318999999999, 2024-09-18T08:12:39.049 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-09-18T08:12:39.049 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-09-18T08:12:39.049 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T08:12:39.049 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2024-09-18T08:12:39.049 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.14805", 2024-09-18T08:12:39.049 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "172.21.15.7:0/640320436", 2024-09-18T08:12:39.049 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 63883 2024-09-18T08:12:39.050 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:12:39.050 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-09-18T08:12:39.050 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:12:39.050 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-09-18T08:12:39.050 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:41.154209+0000", 2024-09-18T08:12:39.050 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-18T08:12:39.050 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:12:39.050 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:12:39.050 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-09-18T08:12:39.050 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:41.154209+0000", 2024-09-18T08:12:39.050 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-18T08:12:39.050 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:12:39.050 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:12:39.051 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-09-18T08:12:39.051 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:41.154211+0000", 2024-09-18T08:12:39.051 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.2109999999999999e-06 2024-09-18T08:12:39.051 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:12:39.051 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:12:39.051 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-09-18T08:12:39.051 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:41.154259+0000", 2024-09-18T08:12:39.051 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4.8412000000000003e-05 2024-09-18T08:12:39.051 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:12:39.051 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:12:39.051 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-09-18T08:12:39.051 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:41.154270+0000", 2024-09-18T08:12:39.051 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.0515e-05 2024-09-18T08:12:39.051 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:12:39.051 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:12:39.052 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-09-18T08:12:39.052 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:41.154275+0000", 2024-09-18T08:12:39.052 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 5.4469999999999997e-06 2024-09-18T08:12:39.052 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:12:39.052 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:12:39.052 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-18T08:12:39.052 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:41.154323+0000", 2024-09-18T08:12:39.052 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4.8297999999999997e-05 2024-09-18T08:12:39.052 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:12:39.052 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:12:39.052 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-09-18T08:12:39.052 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:41.154456+0000", 2024-09-18T08:12:39.052 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000132593 2024-09-18T08:12:39.052 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:12:39.053 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:12:39.053 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "waiting for subops from 0,3", 2024-09-18T08:12:39.053 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:41.154615+0000", 2024-09-18T08:12:39.053 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000159049 2024-09-18T08:12:39.053 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:12:39.053 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:12:39.053 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "op_commit", 2024-09-18T08:12:39.053 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:41.156649+0000", 2024-09-18T08:12:39.053 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0020336709999999999 2024-09-18T08:12:39.053 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:12:39.053 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:12:39.053 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:12:39.053 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:41.165163+0000", 2024-09-18T08:12:39.054 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0085141100000000001 2024-09-18T08:12:39.054 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:12:39.054 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:12:39.054 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:12:39.054 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:41.196483+0000", 2024-09-18T08:12:39.054 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.031320300000000002 2024-09-18T08:12:39.054 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:12:39.054 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:12:39.054 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "commit_sent", 2024-09-18T08:12:39.054 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:41.196519+0000", 2024-09-18T08:12:39.054 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 3.6109999999999998e-05 2024-09-18T08:12:39.054 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:12:39.054 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:12:39.055 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-09-18T08:12:39.055 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:41.196548+0000", 2024-09-18T08:12:39.055 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 2.8603e-05 2024-09-18T08:12:39.055 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-18T08:12:39.055 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-09-18T08:12:39.055 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-18T08:12:39.055 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:12:39.055 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:12:39.055 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.14805.0:63908 7.d 7:bd06bac1:::benchmark_data_smithi007_73155_object63907:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e100)", 2024-09-18T08:12:39.055 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-09-18T08:04:41.183375+0000", 2024-09-18T08:12:39.055 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 477.77442319599999, 2024-09-18T08:12:39.055 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.033452764000000003, 2024-09-18T08:12:39.055 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-09-18T08:12:39.056 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-09-18T08:12:39.056 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T08:12:39.056 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2024-09-18T08:12:39.056 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.14805", 2024-09-18T08:12:39.056 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "172.21.15.7:0/640320436", 2024-09-18T08:12:39.056 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 63908 2024-09-18T08:12:39.056 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:12:39.056 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-09-18T08:12:39.056 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:12:39.056 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-09-18T08:12:39.056 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:41.183375+0000", 2024-09-18T08:12:39.056 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-18T08:12:39.057 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:12:39.057 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:12:39.057 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-09-18T08:12:39.057 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:41.183375+0000", 2024-09-18T08:12:39.057 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-18T08:12:39.057 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:12:39.057 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:12:39.057 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-09-18T08:12:39.057 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:41.183376+0000", 2024-09-18T08:12:39.057 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.226e-06 2024-09-18T08:12:39.057 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:12:39.057 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:12:39.057 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-09-18T08:12:39.058 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:41.183403+0000", 2024-09-18T08:12:39.058 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 2.6404e-05 2024-09-18T08:12:39.058 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:12:39.058 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:12:39.058 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-09-18T08:12:39.058 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:41.183417+0000", 2024-09-18T08:12:39.058 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.4399000000000001e-05 2024-09-18T08:12:39.058 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:12:39.058 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:12:39.058 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-09-18T08:12:39.058 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:41.183425+0000", 2024-09-18T08:12:39.058 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 8.1459999999999996e-06 2024-09-18T08:12:39.058 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:12:39.059 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:12:39.059 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-18T08:12:39.059 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:41.183485+0000", 2024-09-18T08:12:39.059 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 5.9673999999999998e-05 2024-09-18T08:12:39.059 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:12:39.059 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:12:39.059 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-09-18T08:12:39.059 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:41.183587+0000", 2024-09-18T08:12:39.059 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000101611 2024-09-18T08:12:39.059 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:12:39.059 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:12:39.059 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "waiting for subops from 0,1", 2024-09-18T08:12:39.060 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:41.183686+0000", 2024-09-18T08:12:39.060 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 9.9823999999999998e-05 2024-09-18T08:12:39.060 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:12:39.060 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:12:39.060 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "op_commit", 2024-09-18T08:12:39.060 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:41.185240+0000", 2024-09-18T08:12:39.060 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0015539340000000001 2024-09-18T08:12:39.060 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:12:39.060 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:12:39.060 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:12:39.060 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:41.191524+0000", 2024-09-18T08:12:39.060 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0062835690000000001 2024-09-18T08:12:39.061 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:12:39.061 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:12:39.061 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:12:39.061 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:41.216746+0000", 2024-09-18T08:12:39.061 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.025222364000000001 2024-09-18T08:12:39.061 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:12:39.061 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:12:39.061 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "commit_sent", 2024-09-18T08:12:39.061 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:41.216782+0000", 2024-09-18T08:12:39.061 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 3.6069999999999999e-05 2024-09-18T08:12:39.061 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:12:39.061 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:12:39.061 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-09-18T08:12:39.062 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:41.216828+0000", 2024-09-18T08:12:39.062 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4.5543000000000001e-05 2024-09-18T08:12:39.062 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-18T08:12:39.062 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-09-18T08:12:39.062 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-18T08:12:39.062 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:12:39.062 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:12:39.062 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.14805.0:63960 7.c 7:30cc2060:::benchmark_data_smithi007_73155_object63959:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e100)", 2024-09-18T08:12:39.062 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-09-18T08:04:41.252452+0000", 2024-09-18T08:12:39.062 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 477.70534646999999, 2024-09-18T08:12:39.062 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.027634619999999999, 2024-09-18T08:12:39.062 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-09-18T08:12:39.063 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-09-18T08:12:39.063 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T08:12:39.063 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2024-09-18T08:12:39.063 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.14805", 2024-09-18T08:12:39.063 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "172.21.15.7:0/640320436", 2024-09-18T08:12:39.063 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 63960 2024-09-18T08:12:39.063 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:12:39.063 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-09-18T08:12:39.063 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:12:39.063 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-09-18T08:12:39.063 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:41.252452+0000", 2024-09-18T08:12:39.063 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-18T08:12:39.063 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:12:39.063 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:12:39.064 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-09-18T08:12:39.064 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:41.252452+0000", 2024-09-18T08:12:39.064 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-18T08:12:39.064 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:12:39.064 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:12:39.064 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-09-18T08:12:39.064 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:41.252453+0000", 2024-09-18T08:12:39.064 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.063e-06 2024-09-18T08:12:39.064 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:12:39.064 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:12:39.064 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-09-18T08:12:39.064 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:41.252503+0000", 2024-09-18T08:12:39.064 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 5.024e-05 2024-09-18T08:12:39.064 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:12:39.065 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:12:39.065 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-09-18T08:12:39.065 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:41.252513+0000", 2024-09-18T08:12:39.065 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.0137e-05 2024-09-18T08:12:39.065 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:12:39.065 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:12:39.065 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-09-18T08:12:39.065 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:41.252519+0000", 2024-09-18T08:12:39.065 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 5.2630000000000003e-06 2024-09-18T08:12:39.065 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:12:39.065 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:12:39.065 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-18T08:12:39.065 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:41.252564+0000", 2024-09-18T08:12:39.066 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4.5528999999999997e-05 2024-09-18T08:12:39.066 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:12:39.066 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:12:39.066 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-09-18T08:12:39.066 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:41.252645+0000", 2024-09-18T08:12:39.066 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 8.1118000000000005e-05 2024-09-18T08:12:39.066 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:12:39.066 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:12:39.066 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "waiting for subops from 3,7", 2024-09-18T08:12:39.066 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:41.252726+0000", 2024-09-18T08:12:39.066 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 8.0912000000000001e-05 2024-09-18T08:12:39.066 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:12:39.066 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:12:39.066 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "op_commit", 2024-09-18T08:12:39.066 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:41.254200+0000", 2024-09-18T08:12:39.067 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0014738010000000001 2024-09-18T08:12:39.067 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:12:39.067 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:12:39.067 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:12:39.067 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:41.254525+0000", 2024-09-18T08:12:39.067 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00032529599999999998 2024-09-18T08:12:39.067 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:12:39.067 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:12:39.067 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:12:39.067 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:41.280019+0000", 2024-09-18T08:12:39.067 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.025493512999999999 2024-09-18T08:12:39.067 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:12:39.068 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:12:39.068 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "commit_sent", 2024-09-18T08:12:39.068 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:41.280057+0000", 2024-09-18T08:12:39.068 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 3.8287000000000001e-05 2024-09-18T08:12:39.068 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:12:39.068 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:12:39.068 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-09-18T08:12:39.068 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:41.280086+0000", 2024-09-18T08:12:39.068 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 2.9461000000000001e-05 2024-09-18T08:12:39.068 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-18T08:12:39.068 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-09-18T08:12:39.068 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-18T08:12:39.068 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:12:39.068 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:12:39.069 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.14805.0:65051 7.d 7:bcc5331c:::benchmark_data_smithi007_73155_object65050:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e100)", 2024-09-18T08:12:39.069 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-09-18T08:04:42.227551+0000", 2024-09-18T08:12:39.069 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 476.73024770900003, 2024-09-18T08:12:39.069 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.057084449000000002, 2024-09-18T08:12:39.069 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-09-18T08:12:39.069 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-09-18T08:12:39.069 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "waiting for sub ops", 2024-09-18T08:12:39.069 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2024-09-18T08:12:39.069 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.14805", 2024-09-18T08:12:39.069 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "172.21.15.7:0/640320436", 2024-09-18T08:12:39.069 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 65051 2024-09-18T08:12:39.069 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:12:39.069 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-09-18T08:12:39.069 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:12:39.070 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-09-18T08:12:39.070 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:42.227551+0000", 2024-09-18T08:12:39.070 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-18T08:12:39.070 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:12:39.070 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:12:39.070 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-09-18T08:12:39.070 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:42.227551+0000", 2024-09-18T08:12:39.070 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-18T08:12:39.070 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:12:39.070 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:12:39.070 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-09-18T08:12:39.070 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:42.227552+0000", 2024-09-18T08:12:39.070 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.5319999999999999e-06 2024-09-18T08:12:39.070 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:12:39.071 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:12:39.071 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-09-18T08:12:39.071 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:42.227606+0000", 2024-09-18T08:12:39.071 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 5.4290999999999999e-05 2024-09-18T08:12:39.071 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:12:39.071 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:12:39.071 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-09-18T08:12:39.071 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:42.227619+0000", 2024-09-18T08:12:39.071 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.2937e-05 2024-09-18T08:12:39.071 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:12:39.071 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:12:39.071 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-09-18T08:12:39.071 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:42.227627+0000", 2024-09-18T08:12:39.071 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 7.9340000000000006e-06 2024-09-18T08:12:39.071 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:12:39.072 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:12:39.072 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-18T08:12:39.072 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:42.227681+0000", 2024-09-18T08:12:39.072 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 5.3771999999999999e-05 2024-09-18T08:12:39.072 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:12:39.072 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:12:39.072 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-09-18T08:12:39.072 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:42.227809+0000", 2024-09-18T08:12:39.072 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00012750399999999999 2024-09-18T08:12:39.072 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:12:39.072 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:12:39.072 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "waiting for subops from 0,1", 2024-09-18T08:12:39.072 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:42.227961+0000", 2024-09-18T08:12:39.072 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00015280599999999999 2024-09-18T08:12:39.072 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:12:39.073 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:12:39.073 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "op_commit", 2024-09-18T08:12:39.073 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:42.229971+0000", 2024-09-18T08:12:39.073 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0020099179999999999 2024-09-18T08:12:39.073 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:12:39.073 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:12:39.073 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:12:39.073 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:42.232646+0000", 2024-09-18T08:12:39.073 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0026748359999999999 2024-09-18T08:12:39.073 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:12:39.073 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:12:39.073 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-09-18T08:12:39.073 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:42.284635+0000", 2024-09-18T08:12:39.073 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.051988919000000001 2024-09-18T08:12:39.073 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-18T08:12:39.074 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-09-18T08:12:39.074 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-18T08:12:39.074 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:12:39.074 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:12:39.074 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.14805.0:65057 7.d 7:bffbe46c:::benchmark_data_smithi007_73155_object65056:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e100)", 2024-09-18T08:12:39.074 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-09-18T08:04:42.229929+0000", 2024-09-18T08:12:39.074 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 476.72786901299997, 2024-09-18T08:12:39.074 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.054704183000000003, 2024-09-18T08:12:39.074 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-09-18T08:12:39.074 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-09-18T08:12:39.074 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "waiting for sub ops", 2024-09-18T08:12:39.074 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2024-09-18T08:12:39.074 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.14805", 2024-09-18T08:12:39.074 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "172.21.15.7:0/640320436", 2024-09-18T08:12:39.075 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 65057 2024-09-18T08:12:39.075 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:12:39.075 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-09-18T08:12:39.075 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:12:39.075 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-09-18T08:12:39.075 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:42.229929+0000", 2024-09-18T08:12:39.075 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-18T08:12:39.075 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:12:39.075 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:12:39.075 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-09-18T08:12:39.075 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:42.229929+0000", 2024-09-18T08:12:39.075 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-18T08:12:39.075 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:12:39.075 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:12:39.075 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-09-18T08:12:39.076 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:42.229930+0000", 2024-09-18T08:12:39.076 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.0219999999999999e-06 2024-09-18T08:12:39.076 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:12:39.076 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:12:39.076 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-09-18T08:12:39.076 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:42.229973+0000", 2024-09-18T08:12:39.076 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4.2904000000000001e-05 2024-09-18T08:12:39.076 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:12:39.076 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:12:39.076 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-09-18T08:12:39.076 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:42.229987+0000", 2024-09-18T08:12:39.076 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.3405e-05 2024-09-18T08:12:39.076 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:12:39.076 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:12:39.077 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-09-18T08:12:39.078 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:42.229996+0000", 2024-09-18T08:12:39.078 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 9.6299999999999993e-06 2024-09-18T08:12:39.078 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:12:39.078 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:12:39.078 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-18T08:12:39.078 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:42.230048+0000", 2024-09-18T08:12:39.078 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 5.1839000000000003e-05 2024-09-18T08:12:39.078 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:12:39.078 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:12:39.079 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-09-18T08:12:39.079 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:42.230146+0000", 2024-09-18T08:12:39.079 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 9.8101000000000001e-05 2024-09-18T08:12:39.079 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:12:39.079 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:12:39.079 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "waiting for subops from 0,1", 2024-09-18T08:12:39.079 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:42.230237+0000", 2024-09-18T08:12:39.079 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 9.0634e-05 2024-09-18T08:12:39.079 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:12:39.079 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:12:39.079 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "op_commit", 2024-09-18T08:12:39.079 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:42.231752+0000", 2024-09-18T08:12:39.079 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.001515548 2024-09-18T08:12:39.079 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:12:39.079 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:12:39.080 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:12:39.080 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:42.232681+0000", 2024-09-18T08:12:39.080 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00092884 2024-09-18T08:12:39.080 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:12:39.080 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:12:39.080 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-09-18T08:12:39.080 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:42.284634+0000", 2024-09-18T08:12:39.080 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.05195226 2024-09-18T08:12:39.080 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-18T08:12:39.080 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-09-18T08:12:39.080 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-18T08:12:39.080 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:12:39.080 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:12:39.081 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.14805.0:65062 7.d 7:b1e686a2:::benchmark_data_smithi007_73155_object65061:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e100)", 2024-09-18T08:12:39.081 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-09-18T08:04:42.236979+0000", 2024-09-18T08:12:39.081 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 476.72081969800001, 2024-09-18T08:12:39.081 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.047653305999999999, 2024-09-18T08:12:39.081 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-09-18T08:12:39.081 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-09-18T08:12:39.081 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "waiting for sub ops", 2024-09-18T08:12:39.081 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2024-09-18T08:12:39.081 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.14805", 2024-09-18T08:12:39.081 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "172.21.15.7:0/640320436", 2024-09-18T08:12:39.081 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 65062 2024-09-18T08:12:39.081 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:12:39.081 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-09-18T08:12:39.081 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:12:39.082 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-09-18T08:12:39.082 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:42.236979+0000", 2024-09-18T08:12:39.082 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-18T08:12:39.082 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:12:39.082 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:12:39.082 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-09-18T08:12:39.082 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:42.236979+0000", 2024-09-18T08:12:39.082 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-18T08:12:39.082 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:12:39.082 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:12:39.082 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-09-18T08:12:39.082 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:42.236980+0000", 2024-09-18T08:12:39.082 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.124e-06 2024-09-18T08:12:39.082 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:12:39.083 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:12:39.083 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-09-18T08:12:39.083 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:42.237029+0000", 2024-09-18T08:12:39.083 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4.9450999999999998e-05 2024-09-18T08:12:39.083 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:12:39.083 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:12:39.083 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-09-18T08:12:39.083 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:42.237041+0000", 2024-09-18T08:12:39.083 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.1418e-05 2024-09-18T08:12:39.083 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:12:39.083 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:12:39.083 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-09-18T08:12:39.083 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:42.237049+0000", 2024-09-18T08:12:39.083 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 8.0059999999999992e-06 2024-09-18T08:12:39.084 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:12:39.084 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:12:39.084 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-18T08:12:39.084 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:42.237104+0000", 2024-09-18T08:12:39.084 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 5.4870999999999997e-05 2024-09-18T08:12:39.084 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:12:39.084 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:12:39.084 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-09-18T08:12:39.084 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:42.237258+0000", 2024-09-18T08:12:39.084 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000154857 2024-09-18T08:12:39.084 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:12:39.084 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:12:39.084 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "waiting for subops from 0,1", 2024-09-18T08:12:39.084 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:42.237413+0000", 2024-09-18T08:12:39.084 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000154764 2024-09-18T08:12:39.085 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:12:39.085 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:12:39.085 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "op_commit", 2024-09-18T08:12:39.085 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:42.240054+0000", 2024-09-18T08:12:39.085 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0026404549999999999 2024-09-18T08:12:39.085 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:12:39.085 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:12:39.085 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-09-18T08:12:39.085 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:42.284632+0000", 2024-09-18T08:12:39.085 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.044578359999999997 2024-09-18T08:12:39.085 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-18T08:12:39.085 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-09-18T08:12:39.085 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-18T08:12:39.085 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:12:39.085 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:12:39.086 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.14805.0:65063 7.d 7:b037f784:::benchmark_data_smithi007_73155_object65062:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e100)", 2024-09-18T08:12:39.086 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-09-18T08:04:42.237116+0000", 2024-09-18T08:12:39.086 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 476.720682232, 2024-09-18T08:12:39.086 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.047512190000000003, 2024-09-18T08:12:39.086 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-09-18T08:12:39.086 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-09-18T08:12:39.086 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "waiting for sub ops", 2024-09-18T08:12:39.086 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2024-09-18T08:12:39.086 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.14805", 2024-09-18T08:12:39.086 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "172.21.15.7:0/640320436", 2024-09-18T08:12:39.086 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 65063 2024-09-18T08:12:39.087 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:12:39.087 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-09-18T08:12:39.087 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:12:39.087 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-09-18T08:12:39.087 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:42.237116+0000", 2024-09-18T08:12:39.087 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-18T08:12:39.087 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:12:39.087 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:12:39.087 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-09-18T08:12:39.087 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:42.237116+0000", 2024-09-18T08:12:39.087 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-18T08:12:39.087 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:12:39.087 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:12:39.087 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-09-18T08:12:39.088 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:42.237117+0000", 2024-09-18T08:12:39.088 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 6.7000000000000004e-07 2024-09-18T08:12:39.088 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:12:39.088 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:12:39.088 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-09-18T08:12:39.088 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:42.237190+0000", 2024-09-18T08:12:39.088 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 7.2972000000000005e-05 2024-09-18T08:12:39.088 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:12:39.088 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:12:39.088 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-09-18T08:12:39.088 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:42.237197+0000", 2024-09-18T08:12:39.088 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 7.2729999999999997e-06 2024-09-18T08:12:39.088 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:12:39.089 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:12:39.089 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-09-18T08:12:39.089 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:42.237203+0000", 2024-09-18T08:12:39.089 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 5.5289999999999999e-06 2024-09-18T08:12:39.089 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:12:39.089 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:12:39.089 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-18T08:12:39.089 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:42.238960+0000", 2024-09-18T08:12:39.089 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.001757759 2024-09-18T08:12:39.089 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:12:39.089 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:12:39.089 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-09-18T08:12:39.089 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:42.239038+0000", 2024-09-18T08:12:39.090 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 7.8054999999999997e-05 2024-09-18T08:12:39.090 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:12:39.090 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:12:39.090 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "waiting for subops from 0,1", 2024-09-18T08:12:39.090 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:42.239122+0000", 2024-09-18T08:12:39.090 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 8.3963999999999998e-05 2024-09-18T08:12:39.090 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:12:39.090 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:12:39.090 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "op_commit", 2024-09-18T08:12:39.090 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:42.240157+0000", 2024-09-18T08:12:39.090 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.001034372 2024-09-18T08:12:39.091 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:12:39.091 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:12:39.091 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-09-18T08:12:39.091 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:42.284628+0000", 2024-09-18T08:12:39.091 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.044471596000000002 2024-09-18T08:12:39.091 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-18T08:12:39.091 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-09-18T08:12:39.091 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-18T08:12:39.091 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:12:39.091 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:12:39.091 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_repop(client.15582.0:123972 9.0 e261/244 9:0e6c2bd0:::benchmark_data_smithi007_124019_object123971:head v 261'7682, mlcod=261'7678)", 2024-09-18T08:12:39.091 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-09-18T08:12:31.108179+0000", 2024-09-18T08:12:39.091 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 7.8496197199999997, 2024-09-18T08:12:39.091 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.029088182000000001, 2024-09-18T08:12:39.092 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-09-18T08:12:39.092 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-09-18T08:12:39.092 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T08:12:39.092 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-09-18T08:12:39.092 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:12:39.093 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-09-18T08:12:39.093 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:12:31.108179+0000", 2024-09-18T08:12:39.093 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-18T08:12:39.093 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:12:39.093 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:12:39.093 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-09-18T08:12:39.093 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:12:31.108179+0000", 2024-09-18T08:12:39.093 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-18T08:12:39.093 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:12:39.093 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:12:39.093 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-09-18T08:12:39.093 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:12:31.108183+0000", 2024-09-18T08:12:39.094 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4.0029999999999996e-06 2024-09-18T08:12:39.094 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:12:39.094 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:12:39.094 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-09-18T08:12:39.094 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:12:31.108232+0000", 2024-09-18T08:12:39.094 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4.9462000000000002e-05 2024-09-18T08:12:39.094 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:12:39.094 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:12:39.094 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-09-18T08:12:39.094 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:12:31.108247+0000", 2024-09-18T08:12:39.094 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.5350999999999999e-05 2024-09-18T08:12:39.094 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:12:39.094 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:12:39.094 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-09-18T08:12:39.095 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:12:31.108255+0000", 2024-09-18T08:12:39.095 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 7.9449999999999993e-06 2024-09-18T08:12:39.095 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:12:39.095 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:12:39.095 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-18T08:12:39.095 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:12:31.109933+0000", 2024-09-18T08:12:39.095 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.001677869 2024-09-18T08:12:39.095 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:12:39.095 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:12:39.095 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-09-18T08:12:39.095 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:12:31.109957+0000", 2024-09-18T08:12:39.095 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 2.3374e-05 2024-09-18T08:12:39.096 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:12:39.096 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:12:39.096 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "commit_sent", 2024-09-18T08:12:39.096 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:12:31.137170+0000", 2024-09-18T08:12:39.096 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.027213580000000001 2024-09-18T08:12:39.096 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:12:39.096 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:12:39.096 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-09-18T08:12:39.096 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:12:31.137267+0000", 2024-09-18T08:12:39.096 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 9.6597999999999999e-05 2024-09-18T08:12:39.096 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-18T08:12:39.096 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-09-18T08:12:39.096 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-18T08:12:39.097 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:12:39.097 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:12:39.097 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_repop(client.15582.0:123975 9.0 e261/244 9:030e6d7f:::benchmark_data_smithi007_124019_object123974:head v 261'7683, mlcod=261'7680)", 2024-09-18T08:12:39.097 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-09-18T08:12:31.113187+0000", 2024-09-18T08:12:39.097 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 7.8446110630000003, 2024-09-18T08:12:39.097 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.027941135999999998, 2024-09-18T08:12:39.097 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-09-18T08:12:39.097 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-09-18T08:12:39.097 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T08:12:39.097 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-09-18T08:12:39.097 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:12:39.097 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-09-18T08:12:39.097 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:12:31.113187+0000", 2024-09-18T08:12:39.098 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-18T08:12:39.098 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:12:39.098 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:12:39.098 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-09-18T08:12:39.098 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:12:31.113187+0000", 2024-09-18T08:12:39.098 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-18T08:12:39.098 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:12:39.098 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:12:39.098 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-09-18T08:12:39.098 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:12:31.113189+0000", 2024-09-18T08:12:39.098 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.4500000000000001e-06 2024-09-18T08:12:39.098 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:12:39.098 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:12:39.098 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-09-18T08:12:39.099 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:12:31.113223+0000", 2024-09-18T08:12:39.099 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 3.4588000000000002e-05 2024-09-18T08:12:39.099 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:12:39.099 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:12:39.099 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-09-18T08:12:39.099 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:12:31.113240+0000", 2024-09-18T08:12:39.099 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.6580999999999999e-05 2024-09-18T08:12:39.099 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:12:39.099 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:12:39.099 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-09-18T08:12:39.099 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:12:31.113248+0000", 2024-09-18T08:12:39.099 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 8.0560000000000005e-06 2024-09-18T08:12:39.099 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:12:39.099 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:12:39.099 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-18T08:12:39.100 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:12:31.113470+0000", 2024-09-18T08:12:39.100 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00022196100000000001 2024-09-18T08:12:39.100 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:12:39.100 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:12:39.100 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-09-18T08:12:39.100 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:12:31.113490+0000", 2024-09-18T08:12:39.100 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 2.0377000000000002e-05 2024-09-18T08:12:39.100 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:12:39.100 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:12:39.100 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "commit_sent", 2024-09-18T08:12:39.100 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:12:31.140806+0000", 2024-09-18T08:12:39.100 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.027315715000000001 2024-09-18T08:12:39.100 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:12:39.101 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:12:39.101 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-09-18T08:12:39.101 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:12:31.141128+0000", 2024-09-18T08:12:39.101 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00032240799999999999 2024-09-18T08:12:39.101 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-18T08:12:39.101 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-09-18T08:12:39.101 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-18T08:12:39.101 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-18T08:12:39.101 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-09-18T08:12:39.101 INFO:teuthology.orchestra.run.smithi050.stdout:} 2024-09-18T08:12:39.102 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:12:39.102 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:12:39.102 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3' 2024-09-18T08:12:39.197 INFO:tasks.radosbench.radosbench.0.smithi007.stdout: 84 16 134394 134378 99.9648 81.75 0.0232016 0.00999913 2024-09-18T08:12:39.203 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3 2024-09-18T08:12:39.243 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:12:39.243 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:12:39.244 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3' 2024-09-18T08:12:39.345 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5 2024-09-18T08:12:39.383 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:12:39.384 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:12:39.385 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5' 2024-09-18T08:12:39.405 INFO:journalctl@ceph.iscsi.iscsi.a.smithi050.stdout:Sep 18 08:12:39 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-iscsi-iscsi-a[57782]: debug there is no tcmu-runner data available 2024-09-18T08:12:39.486 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7 2024-09-18T08:12:39.502 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid 779349ca-7593-11ef-bceb-c7b262605968 -- ceph osd set noscrub 2024-09-18T08:12:39.522 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:12:39.523 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:12:39.523 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7' 2024-09-18T08:12:39.624 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5 2024-09-18T08:12:39.631 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-09-18T08:12:39.632 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-09-18T08:12:39.656 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:12:39.657 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:12:39.657 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5' 2024-09-18T08:12:39.758 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6 2024-09-18T08:12:39.797 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:12:39.798 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:12:39.798 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6' 2024-09-18T08:12:39.805 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 5, 7, 6, 4] out_osds: [] dead_osds: [1] live_osds: [0, 2, 3, 4, 5, 6, 7] 2024-09-18T08:12:39.805 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-18T08:12:39.805 INFO:tasks.thrashosds.thrasher:inject_pause on osd.6 2024-09-18T08:12:39.805 INFO:tasks.thrashosds.thrasher:Testing filestore_inject_stall pause injection for duration 3 2024-09-18T08:12:39.805 INFO:tasks.thrashosds.thrasher:Checking after 0, should_be_down=False 2024-09-18T08:12:39.806 DEBUG:teuthology.orchestra.run.smithi050:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid 779349ca-7593-11ef-bceb-c7b262605968 -- ceph daemon osd.6 config set filestore_inject_stall 3 2024-09-18T08:12:39.875 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:12:39 smithi050 ceph-mon[32371]: 1.4 scrub starts 2024-09-18T08:12:39.875 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:12:39 smithi050 ceph-mon[32371]: 1.4 scrub ok 2024-09-18T08:12:39.875 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:12:39 smithi050 ceph-mon[32371]: 6.9 scrub starts 2024-09-18T08:12:39.876 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:12:39 smithi050 ceph-mon[32371]: 6.9 scrub ok 2024-09-18T08:12:39.876 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:12:39 smithi050 ceph-mon[32371]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-18T08:12:39.876 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:12:39 smithi050 ceph-mon[32371]: pgmap v475: 158 pgs: 30 active+undersized, 34 active+undersized+degraded, 94 active+clean; 8.2 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 1.3 KiB/s rd, 107 MiB/s wr, 1.71k op/s; 75467/401763 objects degraded (18.784%) 2024-09-18T08:12:39.876 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:12:39 smithi050 ceph-mon[32371]: Health check update: Degraded data redundancy: 71809/382449 objects degraded (18.776%), 34 pgs degraded (PG_DEGRADED) 2024-09-18T08:12:39.876 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:12:39 smithi050 ceph-mon[32371]: osdmap e265: 8 total, 7 up, 8 in 2024-09-18T08:12:39.900 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0 2024-09-18T08:12:39.943 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:12:39.943 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/779349ca-7593-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T08:12:39.943 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:12:39.944 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0' 2024-09-18T08:12:39.980 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:12:39 smithi007 ceph-mon[29459]: 1.4 scrub starts 2024-09-18T08:12:39.980 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:12:39 smithi007 ceph-mon[29459]: 1.4 scrub ok 2024-09-18T08:12:39.981 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:12:39 smithi007 ceph-mon[29459]: 6.9 scrub starts 2024-09-18T08:12:39.981 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:12:39 smithi007 ceph-mon[29459]: 6.9 scrub ok 2024-09-18T08:12:39.981 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:12:39 smithi007 ceph-mon[29459]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-18T08:12:39.981 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:12:39 smithi007 ceph-mon[29459]: pgmap v475: 158 pgs: 30 active+undersized, 34 active+undersized+degraded, 94 active+clean; 8.2 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 1.3 KiB/s rd, 107 MiB/s wr, 1.71k op/s; 75467/401763 objects degraded (18.784%) 2024-09-18T08:12:39.981 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:12:39 smithi007 ceph-mon[29459]: Health check update: Degraded data redundancy: 71809/382449 objects degraded (18.776%), 34 pgs degraded (PG_DEGRADED) 2024-09-18T08:12:39.981 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:12:39 smithi007 ceph-mon[29459]: osdmap e265: 8 total, 7 up, 8 in 2024-09-18T08:12:39.981 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:12:39 smithi007 ceph-mon[39249]: 1.4 scrub starts 2024-09-18T08:12:39.982 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:12:39 smithi007 ceph-mon[39249]: 1.4 scrub ok 2024-09-18T08:12:39.982 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:12:39 smithi007 ceph-mon[39249]: 6.9 scrub starts 2024-09-18T08:12:39.982 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:12:39 smithi007 ceph-mon[39249]: 6.9 scrub ok 2024-09-18T08:12:39.982 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:12:39 smithi007 ceph-mon[39249]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-18T08:12:39.982 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:12:39 smithi007 ceph-mon[39249]: pgmap v475: 158 pgs: 30 active+undersized, 34 active+undersized+degraded, 94 active+clean; 8.2 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 1.3 KiB/s rd, 107 MiB/s wr, 1.71k op/s; 75467/401763 objects degraded (18.784%) 2024-09-18T08:12:39.982 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:12:39 smithi007 ceph-mon[39249]: Health check update: Degraded data redundancy: 71809/382449 objects degraded (18.776%), 34 pgs degraded (PG_DEGRADED) 2024-09-18T08:12:39.982 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:12:39 smithi007 ceph-mon[39249]: osdmap e265: 8 total, 7 up, 8 in 2024-09-18T08:12:40.046 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2 2024-09-18T08:12:40.076 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:12:40.076 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:12:40.077 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2' 2024-09-18T08:12:40.177 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6 2024-09-18T08:12:40.197 INFO:tasks.radosbench.radosbench.0.smithi007.stdout: 85 16 135735 135719 99.7747 83.8125 0.00737109 0.0100183 2024-09-18T08:12:40.233 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:12:40.234 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:12:40.234 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6' 2024-09-18T08:12:40.267 DEBUG:teuthology.orchestra.run.smithi050:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid 779349ca-7593-11ef-bceb-c7b262605968 -- ceph daemon osd.7 dump_ops_in_flight 2024-09-18T08:12:40.336 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4 2024-09-18T08:12:40.339 INFO:teuthology.orchestra.run.smithi050.stderr:Inferring config /var/lib/ceph/779349ca-7593-11ef-bceb-c7b262605968/mon.b/config 2024-09-18T08:12:40.361 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:12:40.361 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:12:40.362 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4' 2024-09-18T08:12:40.462 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0 2024-09-18T08:12:40.494 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:12:40.495 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:12:40.495 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0' 2024-09-18T08:12:40.596 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5 2024-09-18T08:12:40.636 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:12:40.636 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:12:40.637 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5' 2024-09-18T08:12:40.738 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0 2024-09-18T08:12:40.773 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:12:40.774 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:12:40.774 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0' 2024-09-18T08:12:40.875 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2 2024-09-18T08:12:40.897 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:12:40.898 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:12:40.898 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2' 2024-09-18T08:12:40.905 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:12:40 smithi050 ceph-mon[32371]: 6.1b scrub starts 2024-09-18T08:12:40.905 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:12:40 smithi050 ceph-mon[32371]: 6.1b scrub ok 2024-09-18T08:12:40.980 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:12:40 smithi007 ceph-mon[29459]: 6.1b scrub starts 2024-09-18T08:12:40.980 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:12:40 smithi007 ceph-mon[29459]: 6.1b scrub ok 2024-09-18T08:12:40.980 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:12:40 smithi007 ceph-mon[39249]: 6.1b scrub starts 2024-09-18T08:12:40.981 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:12:40 smithi007 ceph-mon[39249]: 6.1b scrub ok 2024-09-18T08:12:40.999 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0 2024-09-18T08:12:41.032 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:12:41.033 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:12:41.033 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0' 2024-09-18T08:12:41.134 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6 2024-09-18T08:12:41.169 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:12:41.170 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:12:41.171 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6' 2024-09-18T08:12:41.197 INFO:tasks.radosbench.radosbench.0.smithi007.stdout: 86 16 136772 136756 99.3681 64.8125 0.00763444 0.010059 2024-09-18T08:12:41.271 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3 2024-09-18T08:12:41.300 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:12:41.301 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:12:41.301 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3' 2024-09-18T08:12:41.402 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4 2024-09-18T08:12:41.442 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:12:41.442 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:12:41.443 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4' 2024-09-18T08:12:41.544 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2 2024-09-18T08:12:41.573 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:12:41.574 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:12:41.574 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2' 2024-09-18T08:12:41.633 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid 779349ca-7593-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-18T08:12:41.676 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7 2024-09-18T08:12:41.718 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:12:41.719 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:12:41.719 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7' 2024-09-18T08:12:41.724 INFO:teuthology.orchestra.run.smithi007.stderr:noscrub is set 2024-09-18T08:12:41.820 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3 2024-09-18T08:12:41.856 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:12:41.857 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:12:41.857 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3' 2024-09-18T08:12:41.894 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:12:41 smithi050 ceph-mon[32371]: 4.14 scrub starts 2024-09-18T08:12:41.895 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:12:41 smithi050 ceph-mon[32371]: 4.14 scrub ok 2024-09-18T08:12:41.895 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:12:41 smithi050 ceph-mon[32371]: 9.7 scrub starts 2024-09-18T08:12:41.895 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:12:41 smithi050 ceph-mon[32371]: 3.1b scrub starts 2024-09-18T08:12:41.895 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:12:41 smithi050 ceph-mon[32371]: 3.1b scrub ok 2024-09-18T08:12:41.895 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:12:41 smithi050 ceph-mon[32371]: from='client.? 172.21.15.7:0/3683688661' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-18T08:12:41.895 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:12:41 smithi050 ceph-mon[32371]: pgmap v477: 158 pgs: 30 active+undersized, 34 active+undersized+degraded, 94 active+clean; 8.2 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 732 B/s rd, 58 MiB/s wr, 921 op/s; 75467/401763 objects degraded (18.784%) 2024-09-18T08:12:41.958 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4 2024-09-18T08:12:41.980 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:12:41 smithi007 ceph-mon[29459]: 4.14 scrub starts 2024-09-18T08:12:41.980 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:12:41 smithi007 ceph-mon[29459]: 4.14 scrub ok 2024-09-18T08:12:41.980 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:12:41 smithi007 ceph-mon[29459]: 9.7 scrub starts 2024-09-18T08:12:41.980 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:12:41 smithi007 ceph-mon[29459]: 3.1b scrub starts 2024-09-18T08:12:41.980 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:12:41 smithi007 ceph-mon[29459]: 3.1b scrub ok 2024-09-18T08:12:41.980 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:12:41 smithi007 ceph-mon[29459]: from='client.? 172.21.15.7:0/3683688661' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-18T08:12:41.981 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:12:41 smithi007 ceph-mon[29459]: pgmap v477: 158 pgs: 30 active+undersized, 34 active+undersized+degraded, 94 active+clean; 8.2 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 732 B/s rd, 58 MiB/s wr, 921 op/s; 75467/401763 objects degraded (18.784%) 2024-09-18T08:12:41.981 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:12:41 smithi007 ceph-mon[39249]: 4.14 scrub starts 2024-09-18T08:12:41.981 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:12:41 smithi007 ceph-mon[39249]: 4.14 scrub ok 2024-09-18T08:12:41.981 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:12:41 smithi007 ceph-mon[39249]: 9.7 scrub starts 2024-09-18T08:12:41.981 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:12:41 smithi007 ceph-mon[39249]: 3.1b scrub starts 2024-09-18T08:12:41.981 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:12:41 smithi007 ceph-mon[39249]: 3.1b scrub ok 2024-09-18T08:12:41.981 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:12:41 smithi007 ceph-mon[39249]: from='client.? 172.21.15.7:0/3683688661' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-18T08:12:41.981 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:12:41 smithi007 ceph-mon[39249]: pgmap v477: 158 pgs: 30 active+undersized, 34 active+undersized+degraded, 94 active+clean; 8.2 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 732 B/s rd, 58 MiB/s wr, 921 op/s; 75467/401763 objects degraded (18.784%) 2024-09-18T08:12:42.008 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:12:42.008 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:12:42.008 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4' 2024-09-18T08:12:42.109 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6 2024-09-18T08:12:42.144 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:12:42.145 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:12:42.145 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6' 2024-09-18T08:12:42.176 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2024-09-18T08:12:42.176 INFO:teuthology.orchestra.run.smithi050.stdout: "success": "" 2024-09-18T08:12:42.176 INFO:teuthology.orchestra.run.smithi050.stdout:} 2024-09-18T08:12:42.197 INFO:tasks.radosbench.radosbench.0.smithi007.stdout: 87 16 138047 138031 99.1418 79.6875 0.00498912 0.0100821 2024-09-18T08:12:42.246 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2 2024-09-18T08:12:42.284 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:12:42.284 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:12:42.285 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2' 2024-09-18T08:12:42.386 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3 2024-09-18T08:12:42.394 INFO:teuthology.orchestra.run.smithi050.stderr:Inferring config /var/lib/ceph/779349ca-7593-11ef-bceb-c7b262605968/mon.b/config 2024-09-18T08:12:42.422 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:12:42.422 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:12:42.423 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3' 2024-09-18T08:12:42.524 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2 2024-09-18T08:12:42.559 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:12:42.560 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:12:42.560 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2' 2024-09-18T08:12:42.661 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3 2024-09-18T08:12:42.691 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:12:42.692 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:12:42.692 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3' 2024-09-18T08:12:42.793 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3 2024-09-18T08:12:42.823 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:12:42.823 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:12:42.824 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3' 2024-09-18T08:12:42.925 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6 2024-09-18T08:12:42.963 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:12:42.964 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:12:42.964 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6' 2024-09-18T08:12:42.979 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:12:42 smithi007 ceph-mon[29459]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-18T08:12:42.979 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:12:42 smithi007 ceph-mon[29459]: from='client.? 172.21.15.7:0/3683688661' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-18T08:12:42.979 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:12:42 smithi007 ceph-mon[29459]: osdmap e266: 8 total, 7 up, 8 in 2024-09-18T08:12:42.979 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:12:42 smithi007 ceph-mon[29459]: purged_snaps scrub starts 2024-09-18T08:12:42.979 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:12:42 smithi007 ceph-mon[29459]: purged_snaps scrub ok 2024-09-18T08:12:42.980 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:12:42 smithi007 ceph-mon[39249]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-18T08:12:42.980 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:12:42 smithi007 ceph-mon[39249]: from='client.? 172.21.15.7:0/3683688661' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-18T08:12:42.980 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:12:42 smithi007 ceph-mon[39249]: osdmap e266: 8 total, 7 up, 8 in 2024-09-18T08:12:42.980 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:12:42 smithi007 ceph-mon[39249]: purged_snaps scrub starts 2024-09-18T08:12:42.980 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:12:42 smithi007 ceph-mon[39249]: purged_snaps scrub ok 2024-09-18T08:12:42.993 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/779349ca-7593-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T08:12:43.013 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:12:42 smithi050 ceph-mon[32371]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-18T08:12:43.013 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:12:42 smithi050 ceph-mon[32371]: from='client.? 172.21.15.7:0/3683688661' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-18T08:12:43.013 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:12:42 smithi050 ceph-mon[32371]: osdmap e266: 8 total, 7 up, 8 in 2024-09-18T08:12:43.013 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:12:42 smithi050 ceph-mon[32371]: purged_snaps scrub starts 2024-09-18T08:12:43.013 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:12:42 smithi050 ceph-mon[32371]: purged_snaps scrub ok 2024-09-18T08:12:43.065 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6 2024-09-18T08:12:43.105 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:12:43.105 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:12:43.106 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6' 2024-09-18T08:12:43.198 INFO:tasks.radosbench.radosbench.0.smithi007.stdout: 88 16 139442 139426 99.0059 87.1875 0.00294348 0.0100958 2024-09-18T08:12:43.206 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7 2024-09-18T08:12:43.267 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:12:43.267 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:12:43.268 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7' 2024-09-18T08:12:43.369 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2 2024-09-18T08:12:43.396 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:12:43.397 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:12:43.397 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2' 2024-09-18T08:12:43.459 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2024-09-18T08:12:43.460 INFO:teuthology.orchestra.run.smithi050.stdout: "ops": [ 2024-09-18T08:12:43.460 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:12:43.460 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.15582.0:139750 9.d 9:b51dff61:::benchmark_data_smithi007_124019_object139749:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536 in=65536b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e267)", 2024-09-18T08:12:43.460 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-09-18T08:12:43.440401+0000", 2024-09-18T08:12:43.460 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 0.016257655999999999, 2024-09-18T08:12:43.460 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.016289390000000001, 2024-09-18T08:12:43.460 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-09-18T08:12:43.460 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-09-18T08:12:43.460 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "waiting for sub ops", 2024-09-18T08:12:43.460 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2024-09-18T08:12:43.460 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.15582", 2024-09-18T08:12:43.461 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "172.21.15.7:0/3598605839", 2024-09-18T08:12:43.461 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 139750 2024-09-18T08:12:43.461 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:12:43.461 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-09-18T08:12:43.461 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:12:43.461 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-09-18T08:12:43.461 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:12:43.440401+0000", 2024-09-18T08:12:43.461 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-18T08:12:43.461 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:12:43.461 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:12:43.461 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-09-18T08:12:43.461 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:12:43.440401+0000", 2024-09-18T08:12:43.462 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-18T08:12:43.462 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:12:43.462 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:12:43.462 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-09-18T08:12:43.462 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:12:43.440401+0000", 2024-09-18T08:12:43.462 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 8.6600000000000005e-07 2024-09-18T08:12:43.462 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:12:43.462 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:12:43.462 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-09-18T08:12:43.462 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:12:43.440467+0000", 2024-09-18T08:12:43.462 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 6.5502999999999999e-05 2024-09-18T08:12:43.462 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:12:43.462 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:12:43.463 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-09-18T08:12:43.463 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:12:43.440478+0000", 2024-09-18T08:12:43.463 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.1395e-05 2024-09-18T08:12:43.463 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:12:43.463 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:12:43.463 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-09-18T08:12:43.463 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:12:43.440486+0000", 2024-09-18T08:12:43.463 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 7.7840000000000004e-06 2024-09-18T08:12:43.463 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:12:43.463 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:12:43.463 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-18T08:12:43.463 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:12:43.440560+0000", 2024-09-18T08:12:43.463 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 7.4300000000000004e-05 2024-09-18T08:12:43.463 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:12:43.464 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:12:43.464 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-09-18T08:12:43.464 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:12:43.440702+0000", 2024-09-18T08:12:43.464 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000141972 2024-09-18T08:12:43.464 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:12:43.464 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:12:43.464 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "waiting for subops from 5", 2024-09-18T08:12:43.464 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:12:43.441047+0000", 2024-09-18T08:12:43.464 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000345087 2024-09-18T08:12:43.464 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:12:43.464 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:12:43.464 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "op_commit", 2024-09-18T08:12:43.464 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:12:43.455835+0000", 2024-09-18T08:12:43.465 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.014787915 2024-09-18T08:12:43.465 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-18T08:12:43.465 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-09-18T08:12:43.465 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-18T08:12:43.465 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:12:43.465 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:12:43.465 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.15582.0:139751 9.d 9:ba9bc3b2:::benchmark_data_smithi007_124019_object139750:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536 in=65536b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e267)", 2024-09-18T08:12:43.465 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-09-18T08:12:43.440828+0000", 2024-09-18T08:12:43.465 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 0.015830489, 2024-09-18T08:12:43.465 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.015941047, 2024-09-18T08:12:43.465 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-09-18T08:12:43.465 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-09-18T08:12:43.465 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "waiting for sub ops", 2024-09-18T08:12:43.466 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2024-09-18T08:12:43.466 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.15582", 2024-09-18T08:12:43.466 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "172.21.15.7:0/3598605839", 2024-09-18T08:12:43.466 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 139751 2024-09-18T08:12:43.466 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:12:43.466 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-09-18T08:12:43.466 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:12:43.466 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-09-18T08:12:43.466 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:12:43.440828+0000", 2024-09-18T08:12:43.466 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-18T08:12:43.466 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:12:43.466 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:12:43.466 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-09-18T08:12:43.467 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:12:43.440828+0000", 2024-09-18T08:12:43.467 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-18T08:12:43.467 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:12:43.467 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:12:43.467 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-09-18T08:12:43.467 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:12:43.440829+0000", 2024-09-18T08:12:43.467 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 8.2099999999999995e-07 2024-09-18T08:12:43.467 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:12:43.467 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:12:43.467 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-09-18T08:12:43.467 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:12:43.440909+0000", 2024-09-18T08:12:43.467 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 8.0649999999999995e-05 2024-09-18T08:12:43.467 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:12:43.468 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:12:43.468 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-09-18T08:12:43.468 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:12:43.440919+0000", 2024-09-18T08:12:43.468 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.0197000000000001e-05 2024-09-18T08:12:43.468 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:12:43.468 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:12:43.468 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-09-18T08:12:43.468 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:12:43.440927+0000", 2024-09-18T08:12:43.468 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 7.1690000000000003e-06 2024-09-18T08:12:43.468 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:12:43.468 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:12:43.468 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-18T08:12:43.468 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:12:43.443859+0000", 2024-09-18T08:12:43.469 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0029329069999999998 2024-09-18T08:12:43.469 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:12:43.469 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:12:43.469 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-09-18T08:12:43.469 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:12:43.443997+0000", 2024-09-18T08:12:43.469 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00013745900000000001 2024-09-18T08:12:43.469 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:12:43.469 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:12:43.469 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "waiting for subops from 5", 2024-09-18T08:12:43.469 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:12:43.444144+0000", 2024-09-18T08:12:43.469 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00014735 2024-09-18T08:12:43.469 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:12:43.469 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:12:43.470 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "op_commit", 2024-09-18T08:12:43.470 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:12:43.455843+0000", 2024-09-18T08:12:43.470 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.011698734000000001 2024-09-18T08:12:43.470 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-18T08:12:43.470 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-09-18T08:12:43.470 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-18T08:12:43.470 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:12:43.470 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:12:43.470 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.15582.0:139754 9.d 9:bcc2ef81:::benchmark_data_smithi007_124019_object139753:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536 in=65536b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e267)", 2024-09-18T08:12:43.470 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-09-18T08:12:43.441336+0000", 2024-09-18T08:12:43.470 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 0.015322400999999999, 2024-09-18T08:12:43.470 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.01550093, 2024-09-18T08:12:43.470 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-09-18T08:12:43.471 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-09-18T08:12:43.471 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "waiting for sub ops", 2024-09-18T08:12:43.471 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2024-09-18T08:12:43.471 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.15582", 2024-09-18T08:12:43.471 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "172.21.15.7:0/3598605839", 2024-09-18T08:12:43.471 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 139754 2024-09-18T08:12:43.471 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:12:43.471 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-09-18T08:12:43.471 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:12:43.471 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-09-18T08:12:43.471 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:12:43.441336+0000", 2024-09-18T08:12:43.471 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-18T08:12:43.471 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:12:43.471 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:12:43.472 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-09-18T08:12:43.472 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:12:43.441336+0000", 2024-09-18T08:12:43.472 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-18T08:12:43.472 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:12:43.472 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:12:43.472 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-09-18T08:12:43.472 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:12:43.441337+0000", 2024-09-18T08:12:43.472 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 8.4200000000000005e-07 2024-09-18T08:12:43.472 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:12:43.472 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:12:43.472 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-09-18T08:12:43.472 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:12:43.441457+0000", 2024-09-18T08:12:43.472 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000120361 2024-09-18T08:12:43.473 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:12:43.473 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:12:43.473 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-09-18T08:12:43.473 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:12:43.441468+0000", 2024-09-18T08:12:43.473 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.1423999999999999e-05 2024-09-18T08:12:43.473 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:12:43.473 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:12:43.473 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-09-18T08:12:43.473 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:12:43.441476+0000", 2024-09-18T08:12:43.473 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 7.6129999999999998e-06 2024-09-18T08:12:43.473 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:12:43.473 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:12:43.473 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-18T08:12:43.474 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:12:43.446614+0000", 2024-09-18T08:12:43.474 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0051381990000000004 2024-09-18T08:12:43.474 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:12:43.474 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:12:43.474 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-09-18T08:12:43.474 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:12:43.446743+0000", 2024-09-18T08:12:43.475 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00012906199999999999 2024-09-18T08:12:43.475 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:12:43.475 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:12:43.475 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "waiting for subops from 5", 2024-09-18T08:12:43.475 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:12:43.446883+0000", 2024-09-18T08:12:43.475 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00014000199999999999 2024-09-18T08:12:43.475 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:12:43.475 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:12:43.475 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "op_commit", 2024-09-18T08:12:43.475 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:12:43.455946+0000", 2024-09-18T08:12:43.475 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0090622729999999992 2024-09-18T08:12:43.475 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-18T08:12:43.475 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-09-18T08:12:43.476 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-18T08:12:43.476 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:12:43.476 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:12:43.476 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.15582.0:139756 9.d 9:b0e5b58b:::benchmark_data_smithi007_124019_object139755:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536 in=65536b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e267)", 2024-09-18T08:12:43.476 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-09-18T08:12:43.441936+0000", 2024-09-18T08:12:43.476 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 0.014722224000000001, 2024-09-18T08:12:43.476 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.014976384000000001, 2024-09-18T08:12:43.476 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-09-18T08:12:43.476 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-09-18T08:12:43.476 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "waiting for sub ops", 2024-09-18T08:12:43.476 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2024-09-18T08:12:43.476 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.15582", 2024-09-18T08:12:43.476 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "172.21.15.7:0/3598605839", 2024-09-18T08:12:43.477 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 139756 2024-09-18T08:12:43.477 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:12:43.477 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-09-18T08:12:43.477 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:12:43.477 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-09-18T08:12:43.477 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:12:43.441936+0000", 2024-09-18T08:12:43.477 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-18T08:12:43.477 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:12:43.477 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:12:43.477 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-09-18T08:12:43.477 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:12:43.441936+0000", 2024-09-18T08:12:43.477 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-18T08:12:43.477 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:12:43.478 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:12:43.478 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-09-18T08:12:43.478 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:12:43.441937+0000", 2024-09-18T08:12:43.478 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 7.9999999999999996e-07 2024-09-18T08:12:43.478 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:12:43.478 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:12:43.478 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-09-18T08:12:43.478 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:12:43.442079+0000", 2024-09-18T08:12:43.478 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000141953 2024-09-18T08:12:43.478 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:12:43.478 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:12:43.478 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-09-18T08:12:43.479 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:12:43.442090+0000", 2024-09-18T08:12:43.479 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.1055e-05 2024-09-18T08:12:43.479 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:12:43.479 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:12:43.479 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-09-18T08:12:43.479 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:12:43.442097+0000", 2024-09-18T08:12:43.479 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 7.588e-06 2024-09-18T08:12:43.479 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:12:43.479 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:12:43.479 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-18T08:12:43.479 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:12:43.451628+0000", 2024-09-18T08:12:43.479 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0095308879999999995 2024-09-18T08:12:43.479 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:12:43.480 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:12:43.480 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-09-18T08:12:43.480 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:12:43.451761+0000", 2024-09-18T08:12:43.480 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00013275200000000001 2024-09-18T08:12:43.480 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:12:43.480 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:12:43.480 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "waiting for subops from 5", 2024-09-18T08:12:43.480 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:12:43.452658+0000", 2024-09-18T08:12:43.480 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00089673300000000004 2024-09-18T08:12:43.480 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:12:43.480 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:12:43.480 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "op_commit", 2024-09-18T08:12:43.480 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:12:43.455951+0000", 2024-09-18T08:12:43.480 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0032936950000000001 2024-09-18T08:12:43.481 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-18T08:12:43.481 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-09-18T08:12:43.481 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-18T08:12:43.481 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:12:43.481 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:12:43.481 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_repop(client.15582.0:139761 9.e e267/244 9:7b4e05f6:::benchmark_data_smithi007_124019_object139760:head v 267'8669, mlcod=267'8666)", 2024-09-18T08:12:43.481 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-09-18T08:12:43.455780+0000", 2024-09-18T08:12:43.481 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 0.00087800699999999998, 2024-09-18T08:12:43.481 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00120141, 2024-09-18T08:12:43.481 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-09-18T08:12:43.481 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-09-18T08:12:43.481 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "started", 2024-09-18T08:12:43.481 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-09-18T08:12:43.482 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:12:43.482 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-09-18T08:12:43.482 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:12:43.455780+0000", 2024-09-18T08:12:43.482 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-18T08:12:43.482 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:12:43.482 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:12:43.482 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-09-18T08:12:43.482 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:12:43.455780+0000", 2024-09-18T08:12:43.482 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-18T08:12:43.482 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:12:43.482 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:12:43.482 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-09-18T08:12:43.482 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:12:43.455781+0000", 2024-09-18T08:12:43.482 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 8.1399999999999996e-07 2024-09-18T08:12:43.483 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:12:43.483 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:12:43.483 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-09-18T08:12:43.483 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:12:43.455809+0000", 2024-09-18T08:12:43.483 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 2.7651999999999999e-05 2024-09-18T08:12:43.483 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:12:43.483 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:12:43.483 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-09-18T08:12:43.483 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:12:43.455832+0000", 2024-09-18T08:12:43.483 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 2.3465999999999999e-05 2024-09-18T08:12:43.483 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:12:43.483 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:12:43.483 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-09-18T08:12:43.483 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:12:43.455840+0000", 2024-09-18T08:12:43.484 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 8.0420000000000003e-06 2024-09-18T08:12:43.484 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:12:43.484 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:12:43.484 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-18T08:12:43.484 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:12:43.455925+0000", 2024-09-18T08:12:43.484 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 8.4492000000000004e-05 2024-09-18T08:12:43.484 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:12:43.484 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:12:43.484 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-09-18T08:12:43.484 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:12:43.456022+0000", 2024-09-18T08:12:43.484 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 9.7213000000000006e-05 2024-09-18T08:12:43.484 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-18T08:12:43.484 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-09-18T08:12:43.484 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-18T08:12:43.485 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:12:43.485 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:12:43.485 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_repop(client.15582.0:139770 9.a e267/244 9:52ca6e90:::benchmark_data_smithi007_124019_object139769:head v 267'8726, mlcod=267'8723)", 2024-09-18T08:12:43.485 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-09-18T08:12:43.456498+0000", 2024-09-18T08:12:43.485 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 0.00015987099999999999, 2024-09-18T08:12:43.485 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00053877700000000005, 2024-09-18T08:12:43.485 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-09-18T08:12:43.485 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-09-18T08:12:43.485 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "started", 2024-09-18T08:12:43.485 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-09-18T08:12:43.485 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:12:43.485 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-09-18T08:12:43.485 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:12:43.456498+0000", 2024-09-18T08:12:43.485 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-18T08:12:43.486 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:12:43.486 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:12:43.486 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-09-18T08:12:43.486 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:12:43.456498+0000", 2024-09-18T08:12:43.486 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-18T08:12:43.486 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:12:43.486 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:12:43.486 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-09-18T08:12:43.486 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:12:43.456499+0000", 2024-09-18T08:12:43.486 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 6.4899999999999995e-07 2024-09-18T08:12:43.486 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:12:43.486 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:12:43.486 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-09-18T08:12:43.486 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:12:43.456523+0000", 2024-09-18T08:12:43.487 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 2.4202e-05 2024-09-18T08:12:43.487 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:12:43.487 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:12:43.487 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-09-18T08:12:43.487 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:12:43.456535+0000", 2024-09-18T08:12:43.487 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.1426e-05 2024-09-18T08:12:43.487 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:12:43.487 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:12:43.487 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-09-18T08:12:43.487 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:12:43.456542+0000", 2024-09-18T08:12:43.487 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 7.0160000000000003e-06 2024-09-18T08:12:43.487 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:12:43.487 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:12:43.487 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-18T08:12:43.488 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:12:43.456591+0000", 2024-09-18T08:12:43.488 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4.9125999999999997e-05 2024-09-18T08:12:43.488 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:12:43.488 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:12:43.488 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-09-18T08:12:43.488 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:12:43.456611+0000", 2024-09-18T08:12:43.488 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 2.0630000000000001e-05 2024-09-18T08:12:43.488 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-18T08:12:43.488 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-09-18T08:12:43.488 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-18T08:12:43.488 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:12:43.488 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:12:43.488 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_repop_reply(client.15582.0:139750 9.d e267/244)", 2024-09-18T08:12:43.488 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-09-18T08:12:43.456911+0000", 2024-09-18T08:12:43.489 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 4294967295.9997473, 2024-09-18T08:12:43.489 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00018142899999999999, 2024-09-18T08:12:43.489 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-09-18T08:12:43.489 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-09-18T08:12:43.489 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "queued for pg", 2024-09-18T08:12:43.489 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-09-18T08:12:43.490 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:12:43.490 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-09-18T08:12:43.490 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:12:43.456911+0000", 2024-09-18T08:12:43.490 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-18T08:12:43.490 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:12:43.490 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:12:43.490 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-09-18T08:12:43.491 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:12:43.456911+0000", 2024-09-18T08:12:43.491 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-18T08:12:43.491 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:12:43.491 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:12:43.491 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-09-18T08:12:43.491 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:12:43.456912+0000", 2024-09-18T08:12:43.492 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 5.82e-07 2024-09-18T08:12:43.492 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:12:43.492 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:12:43.492 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-09-18T08:12:43.492 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:12:43.456915+0000", 2024-09-18T08:12:43.492 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 3.9790000000000004e-06 2024-09-18T08:12:43.493 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:12:43.493 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:12:43.493 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-09-18T08:12:43.493 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:12:43.456928+0000", 2024-09-18T08:12:43.493 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.2961e-05 2024-09-18T08:12:43.493 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:12:43.494 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:12:43.494 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-09-18T08:12:43.494 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:12:43.456935+0000", 2024-09-18T08:12:43.494 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 6.3470000000000004e-06 2024-09-18T08:12:43.494 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-18T08:12:43.494 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-09-18T08:12:43.495 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-18T08:12:43.495 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-18T08:12:43.495 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2024-09-18T08:12:43.495 INFO:teuthology.orchestra.run.smithi050.stdout: "num_ops": 7 2024-09-18T08:12:43.495 INFO:teuthology.orchestra.run.smithi050.stdout:} 2024-09-18T08:12:43.498 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2 2024-09-18T08:12:43.528 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:12:43.528 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:12:43.529 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2' 2024-09-18T08:12:43.553 INFO:teuthology.orchestra.run.smithi007.stdout:osd.0: {} 2024-09-18T08:12:43.554 INFO:teuthology.orchestra.run.smithi007.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-18T08:12:43.554 INFO:teuthology.orchestra.run.smithi007.stderr:Error ENXIO: problem getting command descriptions from osd.1 2024-09-18T08:12:43.585 INFO:teuthology.orchestra.run.smithi007.stdout:osd.2: {} 2024-09-18T08:12:43.586 INFO:teuthology.orchestra.run.smithi007.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-18T08:12:43.613 INFO:teuthology.orchestra.run.smithi007.stdout:osd.3: {} 2024-09-18T08:12:43.613 INFO:teuthology.orchestra.run.smithi007.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-18T08:12:43.630 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4 2024-09-18T08:12:43.643 INFO:teuthology.orchestra.run.smithi007.stdout:osd.4: {} 2024-09-18T08:12:43.643 INFO:teuthology.orchestra.run.smithi007.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-18T08:12:43.669 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:12:43.670 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:12:43.670 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4' 2024-09-18T08:12:43.674 INFO:teuthology.orchestra.run.smithi007.stdout:osd.5: {} 2024-09-18T08:12:43.674 INFO:teuthology.orchestra.run.smithi007.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-18T08:12:43.703 INFO:teuthology.orchestra.run.smithi007.stdout:osd.6: {} 2024-09-18T08:12:43.704 INFO:teuthology.orchestra.run.smithi007.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-18T08:12:43.733 INFO:teuthology.orchestra.run.smithi007.stdout:osd.7: {} 2024-09-18T08:12:43.734 INFO:teuthology.orchestra.run.smithi007.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-18T08:12:43.771 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3 2024-09-18T08:12:43.803 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:12:43.804 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:12:43.804 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3' 2024-09-18T08:12:43.905 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0 2024-09-18T08:12:43.938 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:12:43.939 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:12:43.939 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0' 2024-09-18T08:12:43.980 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:12:43 smithi007 ceph-mon[29459]: osdmap e267: 8 total, 7 up, 8 in 2024-09-18T08:12:43.980 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:12:43 smithi007 ceph-mon[29459]: pgmap v480: 158 pgs: 30 active+undersized, 34 active+undersized+degraded, 94 active+clean; 8.2 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 853 B/s rd, 75 MiB/s wr, 1.19k op/s; 76194/403944 objects degraded (18.863%) 2024-09-18T08:12:43.980 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:12:43 smithi007 ceph-mon[29459]: from='client.? 172.21.15.7:0/1365004002' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-18T08:12:43.981 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:12:43 smithi007 ceph-mon[39249]: osdmap e267: 8 total, 7 up, 8 in 2024-09-18T08:12:43.981 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:12:43 smithi007 ceph-mon[39249]: pgmap v480: 158 pgs: 30 active+undersized, 34 active+undersized+degraded, 94 active+clean; 8.2 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 853 B/s rd, 75 MiB/s wr, 1.19k op/s; 76194/403944 objects degraded (18.863%) 2024-09-18T08:12:43.981 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:12:43 smithi007 ceph-mon[39249]: from='client.? 172.21.15.7:0/1365004002' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-18T08:12:44.040 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7 2024-09-18T08:12:44.054 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:12:43 smithi050 ceph-mon[32371]: osdmap e267: 8 total, 7 up, 8 in 2024-09-18T08:12:44.054 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:12:43 smithi050 ceph-mon[32371]: pgmap v480: 158 pgs: 30 active+undersized, 34 active+undersized+degraded, 94 active+clean; 8.2 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 853 B/s rd, 75 MiB/s wr, 1.19k op/s; 76194/403944 objects degraded (18.863%) 2024-09-18T08:12:44.055 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:12:43 smithi050 ceph-mon[32371]: from='client.? 172.21.15.7:0/1365004002' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-18T08:12:44.081 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:12:44.082 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:12:44.083 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7' 2024-09-18T08:12:44.183 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7 2024-09-18T08:12:44.198 INFO:tasks.radosbench.radosbench.0.smithi007.stdout: 89 16 140777 140761 98.831 83.4375 0.0156732 0.0101135 2024-09-18T08:12:44.225 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:12:44.226 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:12:44.227 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7' 2024-09-18T08:12:44.228 DEBUG:teuthology.orchestra.run.smithi050:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid 779349ca-7593-11ef-bceb-c7b262605968 -- ceph daemon osd.7 dump_blocked_ops 2024-09-18T08:12:44.328 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6 2024-09-18T08:12:44.374 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:12:44.375 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:12:44.375 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6' 2024-09-18T08:12:44.476 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3 2024-09-18T08:12:44.504 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:12:44.504 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:12:44.505 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3' 2024-09-18T08:12:44.606 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5 2024-09-18T08:12:44.665 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:12:44.665 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:12:44.665 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5' 2024-09-18T08:12:44.753 INFO:teuthology.orchestra.run.smithi050.stderr:Inferring config /var/lib/ceph/779349ca-7593-11ef-bceb-c7b262605968/mon.b/config 2024-09-18T08:12:44.766 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4 2024-09-18T08:12:44.827 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:12:44.828 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:12:44.828 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4' 2024-09-18T08:12:44.929 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5 2024-09-18T08:12:44.943 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:12:44 smithi007 ceph-mon[29459]: Health check update: Degraded data redundancy: 76194/403944 objects degraded (18.863%), 34 pgs degraded, 64 pgs undersized (PG_DEGRADED) 2024-09-18T08:12:44.943 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:12:44 smithi007 ceph-mon[39249]: Health check update: Degraded data redundancy: 76194/403944 objects degraded (18.863%), 34 pgs degraded, 64 pgs undersized (PG_DEGRADED) 2024-09-18T08:12:44.970 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:12:44.970 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:12:44.971 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5' 2024-09-18T08:12:44.984 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-09-18T08:12:44.985 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-09-18T08:12:45.035 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid 779349ca-7593-11ef-bceb-c7b262605968 -- ceph osd set nodeep-scrub 2024-09-18T08:12:45.046 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:12:44 smithi050 ceph-mon[32371]: Health check update: Degraded data redundancy: 76194/403944 objects degraded (18.863%), 34 pgs degraded, 64 pgs undersized (PG_DEGRADED) 2024-09-18T08:12:45.072 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7 2024-09-18T08:12:45.108 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:12:45.108 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:12:45.109 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7' 2024-09-18T08:12:45.198 INFO:tasks.radosbench.radosbench.0.smithi007.stdout:Total time run: 90.0062 2024-09-18T08:12:45.198 INFO:tasks.radosbench.radosbench.0.smithi007.stdout:Total writes made: 141882 2024-09-18T08:12:45.198 INFO:tasks.radosbench.radosbench.0.smithi007.stdout:Write size: 65536 2024-09-18T08:12:45.198 INFO:tasks.radosbench.radosbench.0.smithi007.stdout:Object size: 65536 2024-09-18T08:12:45.198 INFO:tasks.radosbench.radosbench.0.smithi007.stdout:Bandwidth (MB/sec): 98.5224 2024-09-18T08:12:45.199 INFO:tasks.radosbench.radosbench.0.smithi007.stdout:Stddev Bandwidth: 27.6278 2024-09-18T08:12:45.199 INFO:tasks.radosbench.radosbench.0.smithi007.stdout:Max bandwidth (MB/sec): 186.438 2024-09-18T08:12:45.199 INFO:tasks.radosbench.radosbench.0.smithi007.stdout:Min bandwidth (MB/sec): 10.8125 2024-09-18T08:12:45.199 INFO:tasks.radosbench.radosbench.0.smithi007.stdout:Average IOPS: 1576 2024-09-18T08:12:45.199 INFO:tasks.radosbench.radosbench.0.smithi007.stdout:Stddev IOPS: 442.045 2024-09-18T08:12:45.199 INFO:tasks.radosbench.radosbench.0.smithi007.stdout:Max IOPS: 2983 2024-09-18T08:12:45.199 INFO:tasks.radosbench.radosbench.0.smithi007.stdout:Min IOPS: 173 2024-09-18T08:12:45.199 INFO:tasks.radosbench.radosbench.0.smithi007.stdout:Average Latency(s): 0.0101461 2024-09-18T08:12:45.199 INFO:tasks.radosbench.radosbench.0.smithi007.stdout:Stddev Latency(s): 0.013039 2024-09-18T08:12:45.199 INFO:tasks.radosbench.radosbench.0.smithi007.stdout:Max latency(s): 1.04338 2024-09-18T08:12:45.199 INFO:tasks.radosbench.radosbench.0.smithi007.stdout:Min latency(s): 0.00114659 2024-09-18T08:12:45.202 INFO:tasks.radosbench.radosbench.0.smithi007.stdout:Cleaning up (deleting benchmark objects) 2024-09-18T08:12:45.209 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3 2024-09-18T08:12:45.240 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:12:45.241 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:12:45.241 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3' 2024-09-18T08:12:45.342 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2 2024-09-18T08:12:45.375 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:12:45.376 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:12:45.377 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2' 2024-09-18T08:12:45.377 INFO:teuthology.orchestra.run.smithi050.stdout:ERROR: (22) Invalid argument 2024-09-18T08:12:45.378 INFO:teuthology.orchestra.run.smithi050.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-18T08:12:45.386 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/779349ca-7593-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T08:12:45.478 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6 2024-09-18T08:12:45.517 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:12:45.518 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:12:45.518 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6' 2024-09-18T08:12:45.619 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2 2024-09-18T08:12:45.650 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:12:45.651 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:12:45.651 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2' 2024-09-18T08:12:45.752 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3 2024-09-18T08:12:45.785 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:12:45.786 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:12:45.787 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3' 2024-09-18T08:12:45.812 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:12:45 smithi007 ceph-mon[29459]: pgmap v481: 158 pgs: 30 active+undersized, 34 active+undersized+degraded, 94 active+clean; 8.5 GiB data, 25 GiB used, 690 GiB / 715 GiB avail; 852 B/s rd, 66 MiB/s wr, 1.06k op/s; 78992/420816 objects degraded (18.771%) 2024-09-18T08:12:45.812 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:12:45 smithi007 ceph-mon[39249]: pgmap v481: 158 pgs: 30 active+undersized, 34 active+undersized+degraded, 94 active+clean; 8.5 GiB data, 25 GiB used, 690 GiB / 715 GiB avail; 852 B/s rd, 66 MiB/s wr, 1.06k op/s; 78992/420816 objects degraded (18.771%) 2024-09-18T08:12:45.888 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6 2024-09-18T08:12:45.928 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:12:45.928 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:12:45.929 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6' 2024-09-18T08:12:46.030 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0 2024-09-18T08:12:46.062 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:12:46.062 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:12:46.063 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0' 2024-09-18T08:12:46.085 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:12:45 smithi050 ceph-mon[32371]: pgmap v481: 158 pgs: 30 active+undersized, 34 active+undersized+degraded, 94 active+clean; 8.5 GiB data, 25 GiB used, 690 GiB / 715 GiB avail; 852 B/s rd, 66 MiB/s wr, 1.06k op/s; 78992/420816 objects degraded (18.771%) 2024-09-18T08:12:46.134 DEBUG:teuthology.orchestra.run.smithi050:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid 779349ca-7593-11ef-bceb-c7b262605968 -- ceph daemon osd.7 dump_historic_ops 2024-09-18T08:12:46.164 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5 2024-09-18T08:12:46.216 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:12:46.217 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:12:46.218 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5' 2024-09-18T08:12:46.318 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5 2024-09-18T08:12:46.361 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:12:46.362 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:12:46.362 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5' 2024-09-18T08:12:46.463 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6 2024-09-18T08:12:46.520 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:12:46.520 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:12:46.521 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6' 2024-09-18T08:12:46.570 INFO:teuthology.orchestra.run.smithi050.stderr:Inferring config /var/lib/ceph/779349ca-7593-11ef-bceb-c7b262605968/mon.b/config 2024-09-18T08:12:46.621 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4 2024-09-18T08:12:46.665 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:12:46.666 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:12:46.666 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4' 2024-09-18T08:12:46.767 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7 2024-09-18T08:12:46.768 INFO:teuthology.orchestra.run.smithi007.stderr:nodeep-scrub is set 2024-09-18T08:12:46.809 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:12:46.809 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:12:46.810 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7' 2024-09-18T08:12:46.910 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0 2024-09-18T08:12:46.941 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:12:46.942 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:12:46.942 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0' 2024-09-18T08:12:46.980 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Sep 18 08:12:46 smithi007 ceph-779349ca-7593-11ef-bceb-c7b262605968-mgr-y[29688]: ::ffff:172.21.15.50 - - [18/Sep/2024:08:12:46] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-09-18T08:12:46.980 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:12:46 smithi007 ceph-mon[29459]: from='client.? 172.21.15.7:0/4195250091' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-18T08:12:46.980 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:12:46 smithi007 ceph-mon[29459]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-18T08:12:46.981 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:12:46 smithi007 ceph-mon[39249]: from='client.? 172.21.15.7:0/4195250091' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-18T08:12:46.981 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:12:46 smithi007 ceph-mon[39249]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-18T08:12:46.986 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid 779349ca-7593-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-18T08:12:47.004 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:12:46 smithi050 ceph-mon[32371]: from='client.? 172.21.15.7:0/4195250091' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-18T08:12:47.004 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:12:46 smithi050 ceph-mon[32371]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-18T08:12:47.043 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0 2024-09-18T08:12:47.080 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:12:47.080 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:12:47.080 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0' 2024-09-18T08:12:47.181 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6 2024-09-18T08:12:47.219 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:12:47.220 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:12:47.220 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6' 2024-09-18T08:12:47.319 INFO:teuthology.orchestra.run.smithi050.stdout:ERROR: (22) Invalid argument 2024-09-18T08:12:47.320 INFO:teuthology.orchestra.run.smithi050.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-18T08:12:47.321 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6 2024-09-18T08:12:47.360 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:12:47.361 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:12:47.361 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6' 2024-09-18T08:12:47.462 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0 2024-09-18T08:12:47.493 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:12:47.493 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:12:47.494 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0' 2024-09-18T08:12:47.594 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3 2024-09-18T08:12:47.622 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:12:47.623 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:12:47.623 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3' 2024-09-18T08:12:47.724 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7 2024-09-18T08:12:47.768 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:12:47.769 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:12:47.769 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7' 2024-09-18T08:12:47.871 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2 2024-09-18T08:12:47.894 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/779349ca-7593-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T08:12:47.910 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:12:47.911 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:12:47.911 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2' 2024-09-18T08:12:47.961 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 5, 7, 6, 4] out_osds: [] dead_osds: [1] live_osds: [0, 2, 3, 4, 5, 6, 7] 2024-09-18T08:12:47.961 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-18T08:12:47.961 INFO:tasks.thrashosds.thrasher:primary_affinity 2024-09-18T08:12:47.961 INFO:tasks.thrashosds.thrasher:Setting osd 4 primary_affinity to 1.000000 2024-09-18T08:12:47.961 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid 779349ca-7593-11ef-bceb-c7b262605968 -- ceph osd primary-affinity 4 1 2024-09-18T08:12:48.012 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7 2024-09-18T08:12:48.013 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:12:47 smithi050 ceph-mon[32371]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-18T08:12:48.013 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:12:47 smithi050 ceph-mon[32371]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-18T08:12:48.013 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:12:47 smithi050 ceph-mon[32371]: osdmap e268: 8 total, 7 up, 8 in 2024-09-18T08:12:48.013 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:12:47 smithi050 ceph-mon[32371]: pgmap v483: 158 pgs: 30 active+undersized, 34 active+undersized+degraded, 94 active+clean; 8.5 GiB data, 25 GiB used, 690 GiB / 715 GiB avail; 852 B/s rd, 66 MiB/s wr, 1.06k op/s; 78992/420816 objects degraded (18.771%) 2024-09-18T08:12:48.027 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:12:47 smithi007 ceph-mon[29459]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-18T08:12:48.027 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:12:47 smithi007 ceph-mon[29459]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-18T08:12:48.027 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:12:47 smithi007 ceph-mon[29459]: osdmap e268: 8 total, 7 up, 8 in 2024-09-18T08:12:48.027 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:12:47 smithi007 ceph-mon[29459]: pgmap v483: 158 pgs: 30 active+undersized, 34 active+undersized+degraded, 94 active+clean; 8.5 GiB data, 25 GiB used, 690 GiB / 715 GiB avail; 852 B/s rd, 66 MiB/s wr, 1.06k op/s; 78992/420816 objects degraded (18.771%) 2024-09-18T08:12:48.027 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:12:47 smithi007 ceph-mon[39249]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-18T08:12:48.027 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:12:47 smithi007 ceph-mon[39249]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-18T08:12:48.027 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:12:47 smithi007 ceph-mon[39249]: osdmap e268: 8 total, 7 up, 8 in 2024-09-18T08:12:48.028 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:12:47 smithi007 ceph-mon[39249]: pgmap v483: 158 pgs: 30 active+undersized, 34 active+undersized+degraded, 94 active+clean; 8.5 GiB data, 25 GiB used, 690 GiB / 715 GiB avail; 852 B/s rd, 66 MiB/s wr, 1.06k op/s; 78992/420816 objects degraded (18.771%) 2024-09-18T08:12:48.041 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid 779349ca-7593-11ef-bceb-c7b262605968 -- ceph daemon osd.0 dump_ops_in_flight 2024-09-18T08:12:48.061 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:12:48.061 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:12:48.062 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7' 2024-09-18T08:12:48.163 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7 2024-09-18T08:12:48.197 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:12:48.198 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:12:48.198 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7' 2024-09-18T08:12:48.299 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5 2024-09-18T08:12:48.337 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:12:48.337 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:12:48.338 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5' 2024-09-18T08:12:48.438 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7 2024-09-18T08:12:48.468 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:12:48.468 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:12:48.468 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7' 2024-09-18T08:12:48.569 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6 2024-09-18T08:12:48.602 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:12:48.603 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:12:48.603 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6' 2024-09-18T08:12:48.704 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6 2024-09-18T08:12:48.739 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:12:48.740 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:12:48.740 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6' 2024-09-18T08:12:48.841 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2 2024-09-18T08:12:48.877 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:12:48.877 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:12:48.878 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2' 2024-09-18T08:12:48.978 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:12:48 smithi007 ceph-mon[29459]: from='mgr.24470 172.21.15.7:0/2594390261' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-18T08:12:48.978 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:12:48 smithi007 ceph-mon[39249]: from='mgr.24470 172.21.15.7:0/2594390261' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-18T08:12:48.979 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7 2024-09-18T08:12:49.021 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:12:49.022 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:12:49.022 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7' 2024-09-18T08:12:49.123 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2 2024-09-18T08:12:49.136 INFO:teuthology.orchestra.run.smithi007.stdout:osd.0: {} 2024-09-18T08:12:49.136 INFO:teuthology.orchestra.run.smithi007.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-18T08:12:49.140 INFO:teuthology.orchestra.run.smithi007.stderr:Error ENXIO: problem getting command descriptions from osd.1 2024-09-18T08:12:49.155 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:12:48 smithi050 ceph-mon[32371]: from='mgr.24470 172.21.15.7:0/2594390261' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-18T08:12:49.168 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:12:49.169 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:12:49.169 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2' 2024-09-18T08:12:49.180 INFO:teuthology.orchestra.run.smithi007.stdout:osd.2: {} 2024-09-18T08:12:49.181 INFO:teuthology.orchestra.run.smithi007.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-18T08:12:49.205 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/779349ca-7593-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T08:12:49.215 INFO:teuthology.orchestra.run.smithi007.stdout:osd.3: {} 2024-09-18T08:12:49.215 INFO:teuthology.orchestra.run.smithi007.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-18T08:12:49.252 INFO:teuthology.orchestra.run.smithi007.stdout:osd.4: {} 2024-09-18T08:12:49.252 INFO:teuthology.orchestra.run.smithi007.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-18T08:12:49.270 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2 2024-09-18T08:12:49.290 INFO:teuthology.orchestra.run.smithi007.stdout:osd.5: {} 2024-09-18T08:12:49.290 INFO:teuthology.orchestra.run.smithi007.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-18T08:12:49.326 INFO:teuthology.orchestra.run.smithi007.stdout:osd.6: {} 2024-09-18T08:12:49.326 INFO:teuthology.orchestra.run.smithi007.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-18T08:12:49.335 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:12:49.335 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:12:49.335 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2' 2024-09-18T08:12:49.363 INFO:teuthology.orchestra.run.smithi007.stdout:osd.7: {} 2024-09-18T08:12:49.363 INFO:teuthology.orchestra.run.smithi007.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-18T08:12:49.437 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0 2024-09-18T08:12:49.468 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:12:49.469 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:12:49.469 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0' 2024-09-18T08:12:49.572 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6 2024-09-18T08:12:49.598 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:12:49.599 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:12:49.600 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6' 2024-09-18T08:12:49.656 INFO:journalctl@ceph.iscsi.iscsi.a.smithi050.stdout:Sep 18 08:12:49 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-iscsi-iscsi-a[57782]: debug there is no tcmu-runner data available 2024-09-18T08:12:49.701 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7 2024-09-18T08:12:49.736 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:12:49.737 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:12:49.737 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7' 2024-09-18T08:12:49.803 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid 779349ca-7593-11ef-bceb-c7b262605968 -- ceph osd unset noscrub 2024-09-18T08:12:49.838 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6 2024-09-18T08:12:49.878 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:12:49.878 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:12:49.879 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6' 2024-09-18T08:12:49.980 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5 2024-09-18T08:12:50.023 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:12:50.023 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:12:50.024 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5' 2024-09-18T08:12:50.035 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:12:49 smithi007 ceph-mon[39249]: from='client.? 172.21.15.7:0/2265414013' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-18T08:12:50.035 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:12:49 smithi007 ceph-mon[39249]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-18T08:12:50.035 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:12:49 smithi007 ceph-mon[39249]: pgmap v484: 158 pgs: 30 active+undersized, 34 active+undersized+degraded, 94 active+clean; 8.4 GiB data, 25 GiB used, 690 GiB / 715 GiB avail; 1.5 KiB/s rd, 67 MiB/s wr, 1.72k op/s; 77560/411789 objects degraded (18.835%) 2024-09-18T08:12:50.035 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:12:49 smithi007 ceph-mon[39249]: Health check update: Degraded data redundancy: 78992/420816 objects degraded (18.771%), 34 pgs degraded, 64 pgs undersized (PG_DEGRADED) 2024-09-18T08:12:50.035 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:12:49 smithi007 ceph-mon[39249]: osdmap e269: 8 total, 7 up, 8 in 2024-09-18T08:12:50.036 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:12:49 smithi007 ceph-mon[29459]: from='client.? 172.21.15.7:0/2265414013' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-18T08:12:50.036 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:12:49 smithi007 ceph-mon[29459]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-18T08:12:50.036 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:12:49 smithi007 ceph-mon[29459]: pgmap v484: 158 pgs: 30 active+undersized, 34 active+undersized+degraded, 94 active+clean; 8.4 GiB data, 25 GiB used, 690 GiB / 715 GiB avail; 1.5 KiB/s rd, 67 MiB/s wr, 1.72k op/s; 77560/411789 objects degraded (18.835%) 2024-09-18T08:12:50.036 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:12:49 smithi007 ceph-mon[29459]: Health check update: Degraded data redundancy: 78992/420816 objects degraded (18.771%), 34 pgs degraded, 64 pgs undersized (PG_DEGRADED) 2024-09-18T08:12:50.036 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:12:49 smithi007 ceph-mon[29459]: osdmap e269: 8 total, 7 up, 8 in 2024-09-18T08:12:50.125 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7 2024-09-18T08:12:50.157 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:12:49 smithi050 ceph-mon[32371]: from='client.? 172.21.15.7:0/2265414013' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-18T08:12:50.158 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:12:49 smithi050 ceph-mon[32371]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-18T08:12:50.158 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:12:49 smithi050 ceph-mon[32371]: pgmap v484: 158 pgs: 30 active+undersized, 34 active+undersized+degraded, 94 active+clean; 8.4 GiB data, 25 GiB used, 690 GiB / 715 GiB avail; 1.5 KiB/s rd, 67 MiB/s wr, 1.72k op/s; 77560/411789 objects degraded (18.835%) 2024-09-18T08:12:50.158 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:12:49 smithi050 ceph-mon[32371]: Health check update: Degraded data redundancy: 78992/420816 objects degraded (18.771%), 34 pgs degraded, 64 pgs undersized (PG_DEGRADED) 2024-09-18T08:12:50.158 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:12:49 smithi050 ceph-mon[32371]: osdmap e269: 8 total, 7 up, 8 in 2024-09-18T08:12:50.175 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:12:50.176 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:12:50.176 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7' 2024-09-18T08:12:50.277 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0 2024-09-18T08:12:50.306 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:12:50.307 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:12:50.308 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0' 2024-09-18T08:12:50.408 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3 2024-09-18T08:12:50.441 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:12:50.442 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:12:50.442 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3' 2024-09-18T08:12:50.522 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-09-18T08:12:50.523 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-09-18T08:12:50.544 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4 2024-09-18T08:12:50.580 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:12:50.580 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:12:50.580 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4' 2024-09-18T08:12:50.681 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0 2024-09-18T08:12:50.685 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/779349ca-7593-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T08:12:50.730 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:12:50.731 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:12:50.731 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0' 2024-09-18T08:12:50.738 INFO:teuthology.orchestra.run.smithi007.stdout:{ 2024-09-18T08:12:50.738 INFO:teuthology.orchestra.run.smithi007.stdout: "ops": [ 2024-09-18T08:12:50.738 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:12:50.738 INFO:teuthology.orchestra.run.smithi007.stdout: "description": "osd_repop_reply(client.15582.0:149675 9.5 e269/232 ondisk, result = 0)", 2024-09-18T08:12:50.738 INFO:teuthology.orchestra.run.smithi007.stdout: "initiated_at": "2024-09-18T08:12:50.735477+0000", 2024-09-18T08:12:50.738 INFO:teuthology.orchestra.run.smithi007.stdout: "age": 0.00026197200000000002, 2024-09-18T08:12:50.738 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.00028452399999999997, 2024-09-18T08:12:50.738 INFO:teuthology.orchestra.run.smithi007.stdout: "continuous": false, 2024-09-18T08:12:50.739 INFO:teuthology.orchestra.run.smithi007.stdout: "type_data": { 2024-09-18T08:12:50.739 INFO:teuthology.orchestra.run.smithi007.stdout: "flag_point": "started", 2024-09-18T08:12:50.739 INFO:teuthology.orchestra.run.smithi007.stdout: "events": [ 2024-09-18T08:12:50.739 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:12:50.739 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "initiated", 2024-09-18T08:12:50.739 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:12:50.735477+0000", 2024-09-18T08:12:50.739 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-09-18T08:12:50.739 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:12:50.739 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:12:50.739 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "header_read", 2024-09-18T08:12:50.739 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:12:50.735477+0000", 2024-09-18T08:12:50.739 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-09-18T08:12:50.739 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:12:50.740 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:12:50.740 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "throttled", 2024-09-18T08:12:50.740 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:12:50.735477+0000", 2024-09-18T08:12:50.740 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 2.7599999999999998e-07 2024-09-18T08:12:50.740 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:12:50.740 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:12:50.740 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "all_read", 2024-09-18T08:12:50.740 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:12:50.735480+0000", 2024-09-18T08:12:50.740 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 2.1600000000000001e-06 2024-09-18T08:12:50.740 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:12:50.741 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:12:50.741 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "dispatched", 2024-09-18T08:12:50.741 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:12:50.735484+0000", 2024-09-18T08:12:50.741 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 4.7310000000000001e-06 2024-09-18T08:12:50.741 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:12:50.741 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:12:50.741 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "queued_for_pg", 2024-09-18T08:12:50.741 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:12:50.735488+0000", 2024-09-18T08:12:50.741 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 3.9580000000000001e-06 2024-09-18T08:12:50.741 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:12:50.741 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:12:50.741 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-09-18T08:12:50.742 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:12:50.735709+0000", 2024-09-18T08:12:50.742 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.00022066499999999999 2024-09-18T08:12:50.742 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:12:50.742 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:12:50.742 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "started", 2024-09-18T08:12:50.742 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:12:50.735715+0000", 2024-09-18T08:12:50.742 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 5.8780000000000003e-06 2024-09-18T08:12:50.742 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-09-18T08:12:50.742 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-09-18T08:12:50.742 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-09-18T08:12:50.742 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:12:50.743 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:12:50.743 INFO:teuthology.orchestra.run.smithi007.stdout: "description": "osd_op(client.15582.0:149672 9.b 9:d95500dc:::benchmark_data_smithi007_124019_object7787:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e269)", 2024-09-18T08:12:50.743 INFO:teuthology.orchestra.run.smithi007.stdout: "initiated_at": "2024-09-18T08:12:50.721663+0000", 2024-09-18T08:12:50.743 INFO:teuthology.orchestra.run.smithi007.stdout: "age": 0.014075756, 2024-09-18T08:12:50.743 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.014174915, 2024-09-18T08:12:50.743 INFO:teuthology.orchestra.run.smithi007.stdout: "continuous": false, 2024-09-18T08:12:50.743 INFO:teuthology.orchestra.run.smithi007.stdout: "type_data": { 2024-09-18T08:12:50.743 INFO:teuthology.orchestra.run.smithi007.stdout: "flag_point": "waiting for sub ops", 2024-09-18T08:12:50.743 INFO:teuthology.orchestra.run.smithi007.stdout: "client_info": { 2024-09-18T08:12:50.743 INFO:teuthology.orchestra.run.smithi007.stdout: "client": "client.15582", 2024-09-18T08:12:50.743 INFO:teuthology.orchestra.run.smithi007.stdout: "client_addr": "172.21.15.7:0/3598605839", 2024-09-18T08:12:50.743 INFO:teuthology.orchestra.run.smithi007.stdout: "tid": 149672 2024-09-18T08:12:50.743 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:12:50.743 INFO:teuthology.orchestra.run.smithi007.stdout: "events": [ 2024-09-18T08:12:50.744 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:12:50.744 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "initiated", 2024-09-18T08:12:50.744 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:12:50.721663+0000", 2024-09-18T08:12:50.744 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-09-18T08:12:50.744 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:12:50.744 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:12:50.744 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "header_read", 2024-09-18T08:12:50.744 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:12:50.721663+0000", 2024-09-18T08:12:50.744 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-09-18T08:12:50.744 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:12:50.744 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:12:50.744 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "throttled", 2024-09-18T08:12:50.744 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:12:50.721664+0000", 2024-09-18T08:12:50.745 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 3.5100000000000001e-07 2024-09-18T08:12:50.745 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:12:50.745 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:12:50.745 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "all_read", 2024-09-18T08:12:50.745 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:12:50.721666+0000", 2024-09-18T08:12:50.745 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.9259999999999999e-06 2024-09-18T08:12:50.745 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:12:50.745 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:12:50.745 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "dispatched", 2024-09-18T08:12:50.745 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:12:50.721670+0000", 2024-09-18T08:12:50.746 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 4.4039999999999997e-06 2024-09-18T08:12:50.746 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:12:50.746 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:12:50.746 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "queued_for_pg", 2024-09-18T08:12:50.746 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:12:50.721674+0000", 2024-09-18T08:12:50.746 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 4.2100000000000003e-06 2024-09-18T08:12:50.746 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:12:50.746 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:12:50.746 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-09-18T08:12:50.746 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:12:50.723351+0000", 2024-09-18T08:12:50.746 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.0016763959999999999 2024-09-18T08:12:50.746 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:12:50.746 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:12:50.746 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "started", 2024-09-18T08:12:50.747 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:12:50.723481+0000", 2024-09-18T08:12:50.747 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.000130887 2024-09-18T08:12:50.747 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:12:50.747 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:12:50.747 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "waiting for subops from 5,6", 2024-09-18T08:12:50.747 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:12:50.723595+0000", 2024-09-18T08:12:50.747 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.000113529 2024-09-18T08:12:50.747 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:12:50.747 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:12:50.748 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "op_commit", 2024-09-18T08:12:50.748 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:12:50.727172+0000", 2024-09-18T08:12:50.748 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.003576606 2024-09-18T08:12:50.748 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:12:50.748 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:12:50.749 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:12:50.749 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:12:50.735574+0000", 2024-09-18T08:12:50.749 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.0084020039999999994 2024-09-18T08:12:50.749 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-09-18T08:12:50.749 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-09-18T08:12:50.750 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-09-18T08:12:50.750 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:12:50.750 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:12:50.750 INFO:teuthology.orchestra.run.smithi007.stdout: "description": "osd_op(client.15582.0:149683 9.5 9:a13712a6:::benchmark_data_smithi007_124019_object7798:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e269)", 2024-09-18T08:12:50.750 INFO:teuthology.orchestra.run.smithi007.stdout: "initiated_at": "2024-09-18T08:12:50.730460+0000", 2024-09-18T08:12:50.751 INFO:teuthology.orchestra.run.smithi007.stdout: "age": 0.0052788749999999997, 2024-09-18T08:12:50.751 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.0054601579999999997, 2024-09-18T08:12:50.751 INFO:teuthology.orchestra.run.smithi007.stdout: "continuous": false, 2024-09-18T08:12:50.751 INFO:teuthology.orchestra.run.smithi007.stdout: "type_data": { 2024-09-18T08:12:50.751 INFO:teuthology.orchestra.run.smithi007.stdout: "flag_point": "waiting for sub ops", 2024-09-18T08:12:50.751 INFO:teuthology.orchestra.run.smithi007.stdout: "client_info": { 2024-09-18T08:12:50.752 INFO:teuthology.orchestra.run.smithi007.stdout: "client": "client.15582", 2024-09-18T08:12:50.752 INFO:teuthology.orchestra.run.smithi007.stdout: "client_addr": "172.21.15.7:0/3598605839", 2024-09-18T08:12:50.752 INFO:teuthology.orchestra.run.smithi007.stdout: "tid": 149683 2024-09-18T08:12:50.752 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:12:50.752 INFO:teuthology.orchestra.run.smithi007.stdout: "events": [ 2024-09-18T08:12:50.753 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:12:50.753 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "initiated", 2024-09-18T08:12:50.753 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:12:50.730460+0000", 2024-09-18T08:12:50.753 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-09-18T08:12:50.753 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:12:50.754 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:12:50.754 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "header_read", 2024-09-18T08:12:50.754 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:12:50.730460+0000", 2024-09-18T08:12:50.754 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-09-18T08:12:50.754 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:12:50.754 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:12:50.755 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "throttled", 2024-09-18T08:12:50.755 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:12:50.730462+0000", 2024-09-18T08:12:50.755 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.8279999999999999e-06 2024-09-18T08:12:50.755 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:12:50.755 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:12:50.755 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "all_read", 2024-09-18T08:12:50.756 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:12:50.730468+0000", 2024-09-18T08:12:50.756 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 6.4489999999999996e-06 2024-09-18T08:12:50.756 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:12:50.756 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:12:50.756 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "dispatched", 2024-09-18T08:12:50.756 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:12:50.730485+0000", 2024-09-18T08:12:50.757 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.6265000000000001e-05 2024-09-18T08:12:50.757 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:12:50.757 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:12:50.757 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "queued_for_pg", 2024-09-18T08:12:50.757 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:12:50.730495+0000", 2024-09-18T08:12:50.758 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 9.7860000000000008e-06 2024-09-18T08:12:50.758 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:12:50.758 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:12:50.758 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-09-18T08:12:50.758 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:12:50.730613+0000", 2024-09-18T08:12:50.759 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.000117995 2024-09-18T08:12:50.759 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:12:50.759 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:12:50.759 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "started", 2024-09-18T08:12:50.759 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:12:50.730731+0000", 2024-09-18T08:12:50.759 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.00011843199999999999 2024-09-18T08:12:50.760 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:12:50.760 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:12:50.760 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "waiting for subops from 5,7", 2024-09-18T08:12:50.760 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:12:50.731278+0000", 2024-09-18T08:12:50.760 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.000546562 2024-09-18T08:12:50.760 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:12:50.760 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:12:50.760 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "op_commit", 2024-09-18T08:12:50.760 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:12:50.734653+0000", 2024-09-18T08:12:50.760 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.0033750830000000001 2024-09-18T08:12:50.760 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-09-18T08:12:50.760 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-09-18T08:12:50.761 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-09-18T08:12:50.761 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:12:50.761 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:12:50.761 INFO:teuthology.orchestra.run.smithi007.stdout: "description": "osd_op(client.15582.0:149684 9.9 9:9eec2a70:::benchmark_data_smithi007_124019_object7799:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e269)", 2024-09-18T08:12:50.761 INFO:teuthology.orchestra.run.smithi007.stdout: "initiated_at": "2024-09-18T08:12:50.730516+0000", 2024-09-18T08:12:50.762 INFO:teuthology.orchestra.run.smithi007.stdout: "age": 0.0052231159999999999, 2024-09-18T08:12:50.762 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.0054772800000000002, 2024-09-18T08:12:50.762 INFO:teuthology.orchestra.run.smithi007.stdout: "continuous": false, 2024-09-18T08:12:50.762 INFO:teuthology.orchestra.run.smithi007.stdout: "type_data": { 2024-09-18T08:12:50.762 INFO:teuthology.orchestra.run.smithi007.stdout: "flag_point": "waiting for sub ops", 2024-09-18T08:12:50.762 INFO:teuthology.orchestra.run.smithi007.stdout: "client_info": { 2024-09-18T08:12:50.762 INFO:teuthology.orchestra.run.smithi007.stdout: "client": "client.15582", 2024-09-18T08:12:50.762 INFO:teuthology.orchestra.run.smithi007.stdout: "client_addr": "172.21.15.7:0/3598605839", 2024-09-18T08:12:50.762 INFO:teuthology.orchestra.run.smithi007.stdout: "tid": 149684 2024-09-18T08:12:50.762 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:12:50.762 INFO:teuthology.orchestra.run.smithi007.stdout: "events": [ 2024-09-18T08:12:50.763 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:12:50.763 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "initiated", 2024-09-18T08:12:50.763 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:12:50.730516+0000", 2024-09-18T08:12:50.763 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-09-18T08:12:50.763 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:12:50.763 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:12:50.763 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "header_read", 2024-09-18T08:12:50.763 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:12:50.730516+0000", 2024-09-18T08:12:50.763 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-09-18T08:12:50.763 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:12:50.763 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:12:50.763 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "throttled", 2024-09-18T08:12:50.763 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:12:50.730517+0000", 2024-09-18T08:12:50.764 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 5.9599999999999999e-07 2024-09-18T08:12:50.764 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:12:50.764 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:12:50.764 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "all_read", 2024-09-18T08:12:50.764 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:12:50.730520+0000", 2024-09-18T08:12:50.764 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 3.1439999999999999e-06 2024-09-18T08:12:50.764 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:12:50.764 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:12:50.764 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "dispatched", 2024-09-18T08:12:50.764 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:12:50.730526+0000", 2024-09-18T08:12:50.764 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 6.7909999999999999e-06 2024-09-18T08:12:50.764 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:12:50.764 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:12:50.765 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "queued_for_pg", 2024-09-18T08:12:50.765 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:12:50.730532+0000", 2024-09-18T08:12:50.765 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 5.9259999999999997e-06 2024-09-18T08:12:50.765 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:12:50.765 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:12:50.765 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-09-18T08:12:50.765 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:12:50.730592+0000", 2024-09-18T08:12:50.765 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 5.9249999999999997e-05 2024-09-18T08:12:50.765 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:12:50.765 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:12:50.766 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "started", 2024-09-18T08:12:50.766 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:12:50.730713+0000", 2024-09-18T08:12:50.766 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.000121789 2024-09-18T08:12:50.766 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:12:50.766 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:12:50.766 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "waiting for subops from 5,7", 2024-09-18T08:12:50.766 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:12:50.730829+0000", 2024-09-18T08:12:50.766 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.000115831 2024-09-18T08:12:50.766 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:12:50.766 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:12:50.766 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "op_commit", 2024-09-18T08:12:50.766 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:12:50.733559+0000", 2024-09-18T08:12:50.766 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.0027294369999999999 2024-09-18T08:12:50.767 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-09-18T08:12:50.767 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-09-18T08:12:50.767 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-09-18T08:12:50.767 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:12:50.767 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:12:50.767 INFO:teuthology.orchestra.run.smithi007.stdout: "description": "osd_repop(client.15582.0:149688 9.8 e269/232 9:16d4f509:::benchmark_data_smithi007_124019_object7803:head v 269'9413, mlcod=269'9409)", 2024-09-18T08:12:50.767 INFO:teuthology.orchestra.run.smithi007.stdout: "initiated_at": "2024-09-18T08:12:50.734939+0000", 2024-09-18T08:12:50.767 INFO:teuthology.orchestra.run.smithi007.stdout: "age": 0.00080029000000000005, 2024-09-18T08:12:50.767 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.001128006, 2024-09-18T08:12:50.767 INFO:teuthology.orchestra.run.smithi007.stdout: "continuous": false, 2024-09-18T08:12:50.767 INFO:teuthology.orchestra.run.smithi007.stdout: "type_data": { 2024-09-18T08:12:50.767 INFO:teuthology.orchestra.run.smithi007.stdout: "flag_point": "started", 2024-09-18T08:12:50.767 INFO:teuthology.orchestra.run.smithi007.stdout: "events": [ 2024-09-18T08:12:50.768 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:12:50.768 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "initiated", 2024-09-18T08:12:50.768 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:12:50.734939+0000", 2024-09-18T08:12:50.768 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-09-18T08:12:50.768 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:12:50.768 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:12:50.768 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "header_read", 2024-09-18T08:12:50.768 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:12:50.734939+0000", 2024-09-18T08:12:50.768 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-09-18T08:12:50.769 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:12:50.769 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:12:50.769 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "throttled", 2024-09-18T08:12:50.769 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:12:50.734940+0000", 2024-09-18T08:12:50.769 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 8.47e-07 2024-09-18T08:12:50.769 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:12:50.769 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:12:50.769 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "all_read", 2024-09-18T08:12:50.769 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:12:50.734946+0000", 2024-09-18T08:12:50.769 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 5.9989999999999997e-06 2024-09-18T08:12:50.770 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:12:50.770 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:12:50.770 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "dispatched", 2024-09-18T08:12:50.770 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:12:50.734958+0000", 2024-09-18T08:12:50.770 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.222e-05 2024-09-18T08:12:50.770 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:12:50.770 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:12:50.770 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "queued_for_pg", 2024-09-18T08:12:50.770 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:12:50.734964+0000", 2024-09-18T08:12:50.770 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 6.3620000000000002e-06 2024-09-18T08:12:50.770 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:12:50.770 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:12:50.770 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-09-18T08:12:50.771 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:12:50.735013+0000", 2024-09-18T08:12:50.771 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 4.8958000000000001e-05 2024-09-18T08:12:50.771 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:12:50.771 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:12:50.771 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "started", 2024-09-18T08:12:50.771 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:12:50.735032+0000", 2024-09-18T08:12:50.771 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.8382000000000001e-05 2024-09-18T08:12:50.771 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-09-18T08:12:50.771 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-09-18T08:12:50.771 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-09-18T08:12:50.771 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-09-18T08:12:50.771 INFO:teuthology.orchestra.run.smithi007.stdout: ], 2024-09-18T08:12:50.771 INFO:teuthology.orchestra.run.smithi007.stdout: "num_ops": 5 2024-09-18T08:12:50.772 INFO:teuthology.orchestra.run.smithi007.stdout:} 2024-09-18T08:12:50.832 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6 2024-09-18T08:12:50.871 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:12:50.871 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:12:50.872 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6' 2024-09-18T08:12:50.973 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4 2024-09-18T08:12:51.006 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:12:51.007 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:12:51.007 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4' 2024-09-18T08:12:51.077 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:12:50 smithi007 ceph-mon[29459]: from='mgr.24470 172.21.15.7:0/2594390261' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-18T08:12:51.077 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:12:50 smithi007 ceph-mon[39249]: from='mgr.24470 172.21.15.7:0/2594390261' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-18T08:12:51.107 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3 2024-09-18T08:12:51.143 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:12:51.144 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:12:51.145 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3' 2024-09-18T08:12:51.155 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:12:50 smithi050 ceph-mon[32371]: from='mgr.24470 172.21.15.7:0/2594390261' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-18T08:12:51.246 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4 2024-09-18T08:12:51.283 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:12:51.283 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:12:51.284 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4' 2024-09-18T08:12:51.385 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5 2024-09-18T08:12:51.421 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:12:51.422 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:12:51.422 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5' 2024-09-18T08:12:51.523 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5 2024-09-18T08:12:51.559 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:12:51.560 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:12:51.560 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5' 2024-09-18T08:12:51.614 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid 779349ca-7593-11ef-bceb-c7b262605968 -- ceph daemon osd.0 dump_blocked_ops 2024-09-18T08:12:51.661 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3 2024-09-18T08:12:51.705 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:12:51.706 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:12:51.706 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3' 2024-09-18T08:12:51.807 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3 2024-09-18T08:12:51.838 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:12:51 smithi007 ceph-mon[29459]: from='mgr.24470 172.21.15.7:0/2594390261' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-18T08:12:51.838 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:12:51 smithi007 ceph-mon[29459]: from='mgr.24470 172.21.15.7:0/2594390261' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-18T08:12:51.838 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:12:51 smithi007 ceph-mon[29459]: from='mgr.24470 172.21.15.7:0/2594390261' entity='mgr.y' 2024-09-18T08:12:51.838 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:12:51 smithi007 ceph-mon[29459]: pgmap v486: 158 pgs: 30 active+undersized, 34 active+undersized+degraded, 94 active+clean; 8.4 GiB data, 25 GiB used, 690 GiB / 715 GiB avail; 1.4 KiB/s rd, 58 MiB/s wr, 1.53k op/s; 77560/411789 objects degraded (18.835%) 2024-09-18T08:12:51.839 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:12:51 smithi007 ceph-mon[39249]: from='mgr.24470 172.21.15.7:0/2594390261' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-18T08:12:51.839 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:12:51 smithi007 ceph-mon[39249]: from='mgr.24470 172.21.15.7:0/2594390261' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-18T08:12:51.839 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:12:51 smithi007 ceph-mon[39249]: from='mgr.24470 172.21.15.7:0/2594390261' entity='mgr.y' 2024-09-18T08:12:51.839 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:12:51 smithi007 ceph-mon[39249]: pgmap v486: 158 pgs: 30 active+undersized, 34 active+undersized+degraded, 94 active+clean; 8.4 GiB data, 25 GiB used, 690 GiB / 715 GiB avail; 1.4 KiB/s rd, 58 MiB/s wr, 1.53k op/s; 77560/411789 objects degraded (18.835%) 2024-09-18T08:12:51.857 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:12:51.858 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:12:51.858 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3' 2024-09-18T08:12:51.959 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2 2024-09-18T08:12:52.000 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:12:52.001 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:12:52.001 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2' 2024-09-18T08:12:52.084 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/779349ca-7593-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T08:12:52.103 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4 2024-09-18T08:12:52.138 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:12:52.139 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:12:52.139 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4' 2024-09-18T08:12:52.155 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:12:51 smithi050 ceph-mon[32371]: from='mgr.24470 172.21.15.7:0/2594390261' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-18T08:12:52.155 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:12:51 smithi050 ceph-mon[32371]: from='mgr.24470 172.21.15.7:0/2594390261' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-18T08:12:52.156 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:12:51 smithi050 ceph-mon[32371]: from='mgr.24470 172.21.15.7:0/2594390261' entity='mgr.y' 2024-09-18T08:12:52.156 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:12:51 smithi050 ceph-mon[32371]: pgmap v486: 158 pgs: 30 active+undersized, 34 active+undersized+degraded, 94 active+clean; 8.4 GiB data, 25 GiB used, 690 GiB / 715 GiB avail; 1.4 KiB/s rd, 58 MiB/s wr, 1.53k op/s; 77560/411789 objects degraded (18.835%) 2024-09-18T08:12:52.240 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7 2024-09-18T08:12:52.277 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:12:52.278 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:12:52.278 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7' 2024-09-18T08:12:52.379 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5 2024-09-18T08:12:52.412 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:12:52.413 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:12:52.413 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5' 2024-09-18T08:12:52.514 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2 2024-09-18T08:12:52.524 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid 779349ca-7593-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-18T08:12:52.544 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:12:52.545 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:12:52.545 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2' 2024-09-18T08:12:52.646 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4 2024-09-18T08:12:52.682 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:12:52.682 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:12:52.683 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4' 2024-09-18T08:12:52.784 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3 2024-09-18T08:12:52.810 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/779349ca-7593-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T08:12:52.847 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:12:52.848 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:12:52.848 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3' 2024-09-18T08:12:52.949 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5 2024-09-18T08:12:52.978 INFO:teuthology.orchestra.run.smithi007.stderr:set osd.4 primary-affinity to 1 (8655362) 2024-09-18T08:12:52.989 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:12:52.990 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:12:52.990 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5' 2024-09-18T08:12:53.091 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7 2024-09-18T08:12:53.130 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:12:53.131 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:12:53.131 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7' 2024-09-18T08:12:53.230 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:12:52 smithi007 ceph-mon[29459]: from='client.? 172.21.15.7:0/468970323' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 4, "weight": 1.0} : dispatch 2024-09-18T08:12:53.231 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:12:52 smithi007 ceph-mon[39249]: from='client.? 172.21.15.7:0/468970323' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 4, "weight": 1.0} : dispatch 2024-09-18T08:12:53.232 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2 2024-09-18T08:12:53.266 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:12:53.267 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:12:53.267 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2' 2024-09-18T08:12:53.368 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7 2024-09-18T08:12:53.405 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:12:52 smithi050 ceph-mon[32371]: from='client.? 172.21.15.7:0/468970323' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 4, "weight": 1.0} : dispatch 2024-09-18T08:12:53.407 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:12:53.408 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:12:53.408 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7' 2024-09-18T08:12:53.509 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5 2024-09-18T08:12:53.551 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:12:53.552 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:12:53.553 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5' 2024-09-18T08:12:53.653 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3 2024-09-18T08:12:53.684 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:12:53.684 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:12:53.685 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3' 2024-09-18T08:12:53.785 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2 2024-09-18T08:12:53.820 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:12:53.821 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:12:53.821 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2' 2024-09-18T08:12:53.922 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5 2024-09-18T08:12:53.967 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:12:53.967 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:12:53.968 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5' 2024-09-18T08:12:54.002 INFO:teuthology.orchestra.run.smithi007.stderr:noscrub is unset 2024-09-18T08:12:54.002 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:12:53 smithi007 ceph-mon[29459]: from='client.? 172.21.15.7:0/468970323' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 4, "weight": 1.0}]': finished 2024-09-18T08:12:54.002 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:12:53 smithi007 ceph-mon[29459]: osdmap e270: 8 total, 7 up, 8 in 2024-09-18T08:12:54.002 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:12:53 smithi007 ceph-mon[29459]: from='client.? 172.21.15.7:0/4165220054' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-18T08:12:54.002 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:12:53 smithi007 ceph-mon[29459]: pgmap v488: 158 pgs: 30 active+undersized, 34 active+undersized+degraded, 94 active+clean; 8.4 GiB data, 25 GiB used, 690 GiB / 715 GiB avail; 940 B/s rd, 17 MiB/s wr, 1.01k op/s; 77560/411789 objects degraded (18.835%) 2024-09-18T08:12:54.003 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:12:53 smithi007 ceph-mon[39249]: from='client.? 172.21.15.7:0/468970323' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 4, "weight": 1.0}]': finished 2024-09-18T08:12:54.003 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:12:53 smithi007 ceph-mon[39249]: osdmap e270: 8 total, 7 up, 8 in 2024-09-18T08:12:54.003 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:12:53 smithi007 ceph-mon[39249]: from='client.? 172.21.15.7:0/4165220054' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-18T08:12:54.003 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:12:53 smithi007 ceph-mon[39249]: pgmap v488: 158 pgs: 30 active+undersized, 34 active+undersized+degraded, 94 active+clean; 8.4 GiB data, 25 GiB used, 690 GiB / 715 GiB avail; 940 B/s rd, 17 MiB/s wr, 1.01k op/s; 77560/411789 objects degraded (18.835%) 2024-09-18T08:12:54.068 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2 2024-09-18T08:12:54.100 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:12:54.101 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:12:54.101 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2' 2024-09-18T08:12:54.202 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0 2024-09-18T08:12:54.242 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:12:54.242 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:12:54.243 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0' 2024-09-18T08:12:54.344 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4 2024-09-18T08:12:54.381 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:12:54.381 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:12:54.382 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4' 2024-09-18T08:12:54.405 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:12:53 smithi050 ceph-mon[32371]: from='client.? 172.21.15.7:0/468970323' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 4, "weight": 1.0}]': finished 2024-09-18T08:12:54.406 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:12:53 smithi050 ceph-mon[32371]: osdmap e270: 8 total, 7 up, 8 in 2024-09-18T08:12:54.406 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:12:53 smithi050 ceph-mon[32371]: from='client.? 172.21.15.7:0/4165220054' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-18T08:12:54.406 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:12:53 smithi050 ceph-mon[32371]: pgmap v488: 158 pgs: 30 active+undersized, 34 active+undersized+degraded, 94 active+clean; 8.4 GiB data, 25 GiB used, 690 GiB / 715 GiB avail; 940 B/s rd, 17 MiB/s wr, 1.01k op/s; 77560/411789 objects degraded (18.835%) 2024-09-18T08:12:54.483 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6 2024-09-18T08:12:54.517 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:12:54.518 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:12:54.519 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6' 2024-09-18T08:12:54.619 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2 2024-09-18T08:12:54.653 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:12:54.654 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:12:54.654 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2' 2024-09-18T08:12:54.756 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3 2024-09-18T08:12:54.784 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:12:54.785 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:12:54.785 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3' 2024-09-18T08:12:54.886 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7 2024-09-18T08:12:54.928 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:12:54.929 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:12:54.929 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7' 2024-09-18T08:12:55.030 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4 2024-09-18T08:12:55.072 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:12:55.072 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:12:55.072 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4' 2024-09-18T08:12:55.173 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3 2024-09-18T08:12:55.202 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:12:55.203 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:12:55.203 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3' 2024-09-18T08:12:55.230 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:12:54 smithi007 ceph-mon[29459]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-18T08:12:55.230 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:12:54 smithi007 ceph-mon[29459]: from='client.? 172.21.15.7:0/4165220054' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-18T08:12:55.230 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:12:54 smithi007 ceph-mon[29459]: osdmap e271: 8 total, 7 up, 8 in 2024-09-18T08:12:55.230 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:12:54 smithi007 ceph-mon[29459]: Health check update: Degraded data redundancy: 77560/411789 objects degraded (18.835%), 34 pgs degraded, 64 pgs undersized (PG_DEGRADED) 2024-09-18T08:12:55.230 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:12:54 smithi007 ceph-mon[39249]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-18T08:12:55.231 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:12:54 smithi007 ceph-mon[39249]: from='client.? 172.21.15.7:0/4165220054' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-18T08:12:55.231 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:12:54 smithi007 ceph-mon[39249]: osdmap e271: 8 total, 7 up, 8 in 2024-09-18T08:12:55.231 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:12:54 smithi007 ceph-mon[39249]: Health check update: Degraded data redundancy: 77560/411789 objects degraded (18.835%), 34 pgs degraded, 64 pgs undersized (PG_DEGRADED) 2024-09-18T08:12:55.304 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5 2024-09-18T08:12:55.346 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:12:55.347 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:12:55.347 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5' 2024-09-18T08:12:55.347 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:12:54 smithi050 ceph-mon[32371]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-18T08:12:55.347 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:12:54 smithi050 ceph-mon[32371]: from='client.? 172.21.15.7:0/4165220054' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-18T08:12:55.347 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:12:54 smithi050 ceph-mon[32371]: osdmap e271: 8 total, 7 up, 8 in 2024-09-18T08:12:55.347 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:12:54 smithi050 ceph-mon[32371]: Health check update: Degraded data redundancy: 77560/411789 objects degraded (18.835%), 34 pgs degraded, 64 pgs undersized (PG_DEGRADED) 2024-09-18T08:12:55.366 INFO:teuthology.orchestra.run.smithi007.stdout:{ 2024-09-18T08:12:55.366 INFO:teuthology.orchestra.run.smithi007.stdout: "ops": [], 2024-09-18T08:12:55.366 INFO:teuthology.orchestra.run.smithi007.stdout: "complaint_time": 30, 2024-09-18T08:12:55.366 INFO:teuthology.orchestra.run.smithi007.stdout: "num_blocked_ops": 0 2024-09-18T08:12:55.366 INFO:teuthology.orchestra.run.smithi007.stdout:} 2024-09-18T08:12:55.448 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6 2024-09-18T08:12:55.488 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:12:55.489 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:12:55.489 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6' 2024-09-18T08:12:55.590 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2 2024-09-18T08:12:55.626 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:12:55.627 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:12:55.627 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2' 2024-09-18T08:12:55.728 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3 2024-09-18T08:12:55.757 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:12:55.758 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:12:55.759 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3' 2024-09-18T08:12:55.859 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3 2024-09-18T08:12:55.891 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:12:55.891 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:12:55.892 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3' 2024-09-18T08:12:55.992 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5 2024-09-18T08:12:56.030 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:12:56.031 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:12:56.032 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5' 2024-09-18T08:12:56.127 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/779349ca-7593-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T08:12:56.132 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3 2024-09-18T08:12:56.152 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid 779349ca-7593-11ef-bceb-c7b262605968 -- ceph daemon osd.0 dump_historic_ops 2024-09-18T08:12:56.170 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:12:56.171 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:12:56.171 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3' 2024-09-18T08:12:56.261 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:12:56 smithi007 ceph-mon[29459]: 6.15 scrub starts 2024-09-18T08:12:56.262 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:12:56 smithi007 ceph-mon[29459]: 6.15 scrub ok 2024-09-18T08:12:56.262 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:12:56 smithi007 ceph-mon[29459]: pgmap v490: 158 pgs: 30 active+undersized, 34 active+undersized+degraded, 94 active+clean; 7.9 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 852 B/s rd, 0 B/s wr, 1.23k op/s; 73418/389568 objects degraded (18.846%) 2024-09-18T08:12:56.262 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:12:56 smithi007 ceph-mon[39249]: 6.15 scrub starts 2024-09-18T08:12:56.262 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:12:56 smithi007 ceph-mon[39249]: 6.15 scrub ok 2024-09-18T08:12:56.262 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:12:56 smithi007 ceph-mon[39249]: pgmap v490: 158 pgs: 30 active+undersized, 34 active+undersized+degraded, 94 active+clean; 7.9 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 852 B/s rd, 0 B/s wr, 1.23k op/s; 73418/389568 objects degraded (18.846%) 2024-09-18T08:12:56.263 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:12:56 smithi050 ceph-mon[32371]: 6.15 scrub starts 2024-09-18T08:12:56.263 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:12:56 smithi050 ceph-mon[32371]: 6.15 scrub ok 2024-09-18T08:12:56.263 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:12:56 smithi050 ceph-mon[32371]: pgmap v490: 158 pgs: 30 active+undersized, 34 active+undersized+degraded, 94 active+clean; 7.9 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 852 B/s rd, 0 B/s wr, 1.23k op/s; 73418/389568 objects degraded (18.846%) 2024-09-18T08:12:56.271 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0 2024-09-18T08:12:56.309 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:12:56.310 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:12:56.311 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0' 2024-09-18T08:12:56.412 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4 2024-09-18T08:12:56.448 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:12:56.449 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:12:56.449 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4' 2024-09-18T08:12:56.550 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5 2024-09-18T08:12:56.586 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:12:56.587 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:12:56.587 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5' 2024-09-18T08:12:56.688 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6 2024-09-18T08:12:56.725 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:12:56.726 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:12:56.726 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6' 2024-09-18T08:12:56.745 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/779349ca-7593-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T08:12:56.748 INFO:teuthology.orchestra.run.smithi007.stdout:osd.0: {} 2024-09-18T08:12:56.749 INFO:teuthology.orchestra.run.smithi007.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-18T08:12:56.749 INFO:teuthology.orchestra.run.smithi007.stderr:Error ENXIO: problem getting command descriptions from osd.1 2024-09-18T08:12:56.782 INFO:teuthology.orchestra.run.smithi007.stdout:osd.2: {} 2024-09-18T08:12:56.782 INFO:teuthology.orchestra.run.smithi007.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-18T08:12:56.813 INFO:teuthology.orchestra.run.smithi007.stdout:osd.3: {} 2024-09-18T08:12:56.813 INFO:teuthology.orchestra.run.smithi007.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-18T08:12:56.828 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5 2024-09-18T08:12:56.844 INFO:teuthology.orchestra.run.smithi007.stdout:osd.4: {} 2024-09-18T08:12:56.844 INFO:teuthology.orchestra.run.smithi007.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-18T08:12:56.844 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Sep 18 08:12:56 smithi007 ceph-779349ca-7593-11ef-bceb-c7b262605968-mgr-y[29688]: ::ffff:172.21.15.50 - - [18/Sep/2024:08:12:56] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-09-18T08:12:56.865 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:12:56.866 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:12:56.866 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5' 2024-09-18T08:12:56.878 INFO:teuthology.orchestra.run.smithi007.stdout:osd.5: {} 2024-09-18T08:12:56.878 INFO:teuthology.orchestra.run.smithi007.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-18T08:12:56.908 INFO:teuthology.orchestra.run.smithi007.stdout:osd.6: {} 2024-09-18T08:12:56.908 INFO:teuthology.orchestra.run.smithi007.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-18T08:12:56.942 INFO:teuthology.orchestra.run.smithi007.stdout:osd.7: {} 2024-09-18T08:12:56.942 INFO:teuthology.orchestra.run.smithi007.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-18T08:12:56.966 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3 2024-09-18T08:12:57.003 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:12:57.004 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:12:57.004 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3' 2024-09-18T08:12:57.105 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6 2024-09-18T08:12:57.132 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:12:57.133 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:12:57.133 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6' 2024-09-18T08:12:57.230 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:12:57 smithi007 ceph-mon[29459]: from='client.? 172.21.15.7:0/4290750117' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-18T08:12:57.231 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:12:57 smithi007 ceph-mon[39249]: from='client.? 172.21.15.7:0/4290750117' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-18T08:12:57.234 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4 2024-09-18T08:12:57.254 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid 779349ca-7593-11ef-bceb-c7b262605968 -- ceph osd unset nodeep-scrub 2024-09-18T08:12:57.269 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:12:57.269 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:12:57.270 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4' 2024-09-18T08:12:57.370 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6 2024-09-18T08:12:57.405 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:12:57 smithi050 ceph-mon[32371]: from='client.? 172.21.15.7:0/4290750117' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-18T08:12:57.407 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:12:57.408 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:12:57.408 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6' 2024-09-18T08:12:57.509 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4 2024-09-18T08:12:57.541 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:12:57.542 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:12:57.542 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4' 2024-09-18T08:12:57.552 INFO:teuthology.orchestra.run.smithi007.stdout:ERROR: (22) Invalid argument 2024-09-18T08:12:57.552 INFO:teuthology.orchestra.run.smithi007.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-18T08:12:57.643 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5 2024-09-18T08:12:57.681 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:12:57.682 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:12:57.683 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5' 2024-09-18T08:12:57.783 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3 2024-09-18T08:12:57.816 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:12:57.816 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:12:57.817 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3' 2024-09-18T08:12:57.918 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3 2024-09-18T08:12:57.953 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:12:57.956 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:12:57.957 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3' 2024-09-18T08:12:58.057 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3 2024-09-18T08:12:58.068 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-09-18T08:12:58.068 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-09-18T08:12:58.090 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:12:58.091 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:12:58.091 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3' 2024-09-18T08:12:58.192 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0 2024-09-18T08:12:58.222 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:12:58.223 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:12:58.223 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0' 2024-09-18T08:12:58.304 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:12:58 smithi007 ceph-mon[29459]: 5.4 scrub starts 2024-09-18T08:12:58.304 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:12:58 smithi007 ceph-mon[29459]: 5.4 scrub ok 2024-09-18T08:12:58.304 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:12:58 smithi007 ceph-mon[29459]: 5.1b scrub starts 2024-09-18T08:12:58.304 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:12:58 smithi007 ceph-mon[29459]: 5.1b scrub ok 2024-09-18T08:12:58.304 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:12:58 smithi007 ceph-mon[29459]: pgmap v491: 158 pgs: 30 active+undersized, 34 active+undersized+degraded, 94 active+clean; 7.9 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 640 B/s rd, 0 B/s wr, 927 op/s; 73418/389568 objects degraded (18.846%) 2024-09-18T08:12:58.305 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:12:58 smithi007 ceph-mon[39249]: 5.4 scrub starts 2024-09-18T08:12:58.305 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:12:58 smithi007 ceph-mon[39249]: 5.4 scrub ok 2024-09-18T08:12:58.305 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:12:58 smithi007 ceph-mon[39249]: 5.1b scrub starts 2024-09-18T08:12:58.305 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:12:58 smithi007 ceph-mon[39249]: 5.1b scrub ok 2024-09-18T08:12:58.305 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:12:58 smithi007 ceph-mon[39249]: pgmap v491: 158 pgs: 30 active+undersized, 34 active+undersized+degraded, 94 active+clean; 7.9 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 640 B/s rd, 0 B/s wr, 927 op/s; 73418/389568 objects degraded (18.846%) 2024-09-18T08:12:58.324 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2 2024-09-18T08:12:58.366 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:12:58.367 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:12:58.367 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2' 2024-09-18T08:12:58.406 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:12:58 smithi050 ceph-mon[32371]: 5.4 scrub starts 2024-09-18T08:12:58.406 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:12:58 smithi050 ceph-mon[32371]: 5.4 scrub ok 2024-09-18T08:12:58.406 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:12:58 smithi050 ceph-mon[32371]: 5.1b scrub starts 2024-09-18T08:12:58.406 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:12:58 smithi050 ceph-mon[32371]: 5.1b scrub ok 2024-09-18T08:12:58.406 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:12:58 smithi050 ceph-mon[32371]: pgmap v491: 158 pgs: 30 active+undersized, 34 active+undersized+degraded, 94 active+clean; 7.9 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 640 B/s rd, 0 B/s wr, 927 op/s; 73418/389568 objects degraded (18.846%) 2024-09-18T08:12:58.468 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4 2024-09-18T08:12:58.501 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:12:58.501 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:12:58.502 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4' 2024-09-18T08:12:58.603 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5 2024-09-18T08:12:58.644 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:12:58.644 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:12:58.645 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5' 2024-09-18T08:12:58.746 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7 2024-09-18T08:12:58.781 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:12:58.782 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:12:58.782 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7' 2024-09-18T08:12:58.795 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 5, 7, 6, 4] out_osds: [] dead_osds: [1] live_osds: [0, 2, 3, 4, 5, 6, 7] 2024-09-18T08:12:58.795 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-18T08:12:58.796 INFO:tasks.thrashosds.thrasher:inject_pause on osd.3 2024-09-18T08:12:58.796 INFO:tasks.thrashosds.thrasher:Testing filestore_inject_stall pause injection for duration 3 2024-09-18T08:12:58.796 INFO:tasks.thrashosds.thrasher:Checking after 0, should_be_down=False 2024-09-18T08:12:58.796 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid 779349ca-7593-11ef-bceb-c7b262605968 -- ceph daemon osd.3 config set filestore_inject_stall 3 2024-09-18T08:12:58.882 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7 2024-09-18T08:12:58.884 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid 779349ca-7593-11ef-bceb-c7b262605968 -- ceph daemon osd.2 dump_ops_in_flight 2024-09-18T08:12:58.919 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:12:58.919 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:12:58.919 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7' 2024-09-18T08:12:58.925 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/779349ca-7593-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T08:12:59.020 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7 2024-09-18T08:12:59.052 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:12:59.053 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:12:59.054 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7' 2024-09-18T08:12:59.098 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:12:59 smithi007 ceph-mon[29459]: 4.15 scrub starts 2024-09-18T08:12:59.099 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:12:59 smithi007 ceph-mon[29459]: 4.15 scrub ok 2024-09-18T08:12:59.099 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:12:59 smithi007 ceph-mon[39249]: 4.15 scrub starts 2024-09-18T08:12:59.099 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:12:59 smithi007 ceph-mon[39249]: 4.15 scrub ok 2024-09-18T08:12:59.155 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6 2024-09-18T08:12:59.187 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:12:59.187 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:12:59.188 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6' 2024-09-18T08:12:59.289 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2 2024-09-18T08:12:59.299 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:12:59 smithi050 ceph-mon[32371]: 4.15 scrub starts 2024-09-18T08:12:59.300 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:12:59 smithi050 ceph-mon[32371]: 4.15 scrub ok 2024-09-18T08:12:59.334 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:12:59.335 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:12:59.335 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2' 2024-09-18T08:12:59.436 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0 2024-09-18T08:12:59.469 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:12:59.470 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:12:59.470 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0' 2024-09-18T08:12:59.571 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4 2024-09-18T08:12:59.608 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:12:59.609 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:12:59.609 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4' 2024-09-18T08:12:59.655 INFO:journalctl@ceph.iscsi.iscsi.a.smithi050.stdout:Sep 18 08:12:59 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-iscsi-iscsi-a[57782]: debug there is no tcmu-runner data available 2024-09-18T08:12:59.710 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3 2024-09-18T08:12:59.732 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:12:59.733 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:12:59.733 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3' 2024-09-18T08:12:59.834 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4 2024-09-18T08:12:59.868 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:12:59.868 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:12:59.869 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4' 2024-09-18T08:12:59.969 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3 2024-09-18T08:13:00.001 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:13:00.002 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:13:00.003 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3' 2024-09-18T08:13:00.069 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid 779349ca-7593-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-18T08:13:00.103 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4 2024-09-18T08:13:00.145 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:13:00.146 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:13:00.146 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4' 2024-09-18T08:13:00.247 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6 2024-09-18T08:13:00.282 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:13:00.283 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:13:00.283 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6' 2024-09-18T08:13:00.319 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:13:00 smithi007 ceph-mon[29459]: pgmap v492: 158 pgs: 30 active+undersized, 34 active+undersized+degraded, 94 active+clean; 7.4 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 55 KiB/s rd, 0 B/s wr, 1.97k op/s; 69070/366567 objects degraded (18.842%) 2024-09-18T08:13:00.319 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:13:00 smithi007 ceph-mon[29459]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-18T08:13:00.319 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:13:00 smithi007 ceph-mon[29459]: Health check update: Degraded data redundancy: 73418/389568 objects degraded (18.846%), 34 pgs degraded, 64 pgs undersized (PG_DEGRADED) 2024-09-18T08:13:00.320 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:13:00 smithi007 ceph-mon[39249]: pgmap v492: 158 pgs: 30 active+undersized, 34 active+undersized+degraded, 94 active+clean; 7.4 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 55 KiB/s rd, 0 B/s wr, 1.97k op/s; 69070/366567 objects degraded (18.842%) 2024-09-18T08:13:00.320 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:13:00 smithi007 ceph-mon[39249]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-18T08:13:00.320 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:13:00 smithi007 ceph-mon[39249]: Health check update: Degraded data redundancy: 73418/389568 objects degraded (18.846%), 34 pgs degraded, 64 pgs undersized (PG_DEGRADED) 2024-09-18T08:13:00.384 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0 2024-09-18T08:13:00.405 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:13:00 smithi050 ceph-mon[32371]: pgmap v492: 158 pgs: 30 active+undersized, 34 active+undersized+degraded, 94 active+clean; 7.4 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 55 KiB/s rd, 0 B/s wr, 1.97k op/s; 69070/366567 objects degraded (18.842%) 2024-09-18T08:13:00.405 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:13:00 smithi050 ceph-mon[32371]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-18T08:13:00.405 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:13:00 smithi050 ceph-mon[32371]: Health check update: Degraded data redundancy: 73418/389568 objects degraded (18.846%), 34 pgs degraded, 64 pgs undersized (PG_DEGRADED) 2024-09-18T08:13:00.407 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:13:00.407 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:13:00.407 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0' 2024-09-18T08:13:00.508 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5 2024-09-18T08:13:00.548 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:13:00.548 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:13:00.549 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5' 2024-09-18T08:13:00.649 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6 2024-09-18T08:13:00.688 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:13:00.689 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:13:00.689 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6' 2024-09-18T08:13:00.790 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2 2024-09-18T08:13:00.812 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:13:00.812 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:13:00.813 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2' 2024-09-18T08:13:00.914 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6 2024-09-18T08:13:00.955 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:13:00.956 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:13:00.956 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6' 2024-09-18T08:13:01.057 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7 2024-09-18T08:13:01.087 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:13:01.088 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:13:01.088 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7' 2024-09-18T08:13:01.189 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7 2024-09-18T08:13:01.227 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:13:01.227 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:13:01.228 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7' 2024-09-18T08:13:01.328 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3 2024-09-18T08:13:01.359 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:13:01.360 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:13:01.360 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3' 2024-09-18T08:13:01.406 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:13:01 smithi050 ceph-mon[32371]: 5.1f scrub starts 2024-09-18T08:13:01.406 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:13:01 smithi050 ceph-mon[32371]: 5.1f scrub ok 2024-09-18T08:13:01.406 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:13:01 smithi050 ceph-mon[32371]: from='client.? 172.21.15.7:0/1463138860' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-18T08:13:01.452 INFO:teuthology.orchestra.run.smithi007.stderr:nodeep-scrub is unset 2024-09-18T08:13:01.453 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:13:01 smithi007 ceph-mon[29459]: 5.1f scrub starts 2024-09-18T08:13:01.453 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:13:01 smithi007 ceph-mon[29459]: 5.1f scrub ok 2024-09-18T08:13:01.454 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:13:01 smithi007 ceph-mon[29459]: from='client.? 172.21.15.7:0/1463138860' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-18T08:13:01.454 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:13:01 smithi007 ceph-mon[39249]: 5.1f scrub starts 2024-09-18T08:13:01.454 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:13:01 smithi007 ceph-mon[39249]: 5.1f scrub ok 2024-09-18T08:13:01.454 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:13:01 smithi007 ceph-mon[39249]: from='client.? 172.21.15.7:0/1463138860' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-18T08:13:01.461 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2 2024-09-18T08:13:01.495 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:13:01.495 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:13:01.496 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2' 2024-09-18T08:13:01.596 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7 2024-09-18T08:13:01.628 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:13:01.629 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:13:01.629 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7' 2024-09-18T08:13:01.730 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0 2024-09-18T08:13:01.751 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:13:01.752 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:13:01.752 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0' 2024-09-18T08:13:01.853 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5 2024-09-18T08:13:01.886 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:13:01.887 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:13:01.888 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5' 2024-09-18T08:13:01.988 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0 2024-09-18T08:13:02.019 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:13:02.020 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:13:02.020 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0' 2024-09-18T08:13:02.121 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2 2024-09-18T08:13:02.153 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:13:02.154 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:13:02.155 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2' 2024-09-18T08:13:02.256 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0 2024-09-18T08:13:02.288 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:13:02.289 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:13:02.289 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0' 2024-09-18T08:13:02.390 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6 2024-09-18T08:13:02.406 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:13:02 smithi050 ceph-mon[32371]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-18T08:13:02.406 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:13:02 smithi050 ceph-mon[32371]: from='client.? 172.21.15.7:0/1463138860' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-18T08:13:02.407 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:13:02 smithi050 ceph-mon[32371]: osdmap e272: 8 total, 7 up, 8 in 2024-09-18T08:13:02.407 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:13:02 smithi050 ceph-mon[32371]: purged_snaps scrub starts 2024-09-18T08:13:02.407 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:13:02 smithi050 ceph-mon[32371]: purged_snaps scrub ok 2024-09-18T08:13:02.407 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:13:02 smithi050 ceph-mon[32371]: pgmap v494: 158 pgs: 30 active+undersized, 34 active+undersized+degraded, 94 active+clean; 7.4 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 55 KiB/s rd, 0 B/s wr, 1.97k op/s; 69070/366567 objects degraded (18.842%) 2024-09-18T08:13:02.429 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:13:02.430 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:13:02.430 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6' 2024-09-18T08:13:02.480 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:13:02 smithi007 ceph-mon[29459]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-18T08:13:02.480 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:13:02 smithi007 ceph-mon[29459]: from='client.? 172.21.15.7:0/1463138860' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-18T08:13:02.480 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:13:02 smithi007 ceph-mon[29459]: osdmap e272: 8 total, 7 up, 8 in 2024-09-18T08:13:02.481 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:13:02 smithi007 ceph-mon[29459]: purged_snaps scrub starts 2024-09-18T08:13:02.481 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:13:02 smithi007 ceph-mon[29459]: purged_snaps scrub ok 2024-09-18T08:13:02.481 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:13:02 smithi007 ceph-mon[29459]: pgmap v494: 158 pgs: 30 active+undersized, 34 active+undersized+degraded, 94 active+clean; 7.4 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 55 KiB/s rd, 0 B/s wr, 1.97k op/s; 69070/366567 objects degraded (18.842%) 2024-09-18T08:13:02.481 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:13:02 smithi007 ceph-mon[39249]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-18T08:13:02.481 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:13:02 smithi007 ceph-mon[39249]: from='client.? 172.21.15.7:0/1463138860' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-18T08:13:02.481 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:13:02 smithi007 ceph-mon[39249]: osdmap e272: 8 total, 7 up, 8 in 2024-09-18T08:13:02.481 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:13:02 smithi007 ceph-mon[39249]: purged_snaps scrub starts 2024-09-18T08:13:02.481 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:13:02 smithi007 ceph-mon[39249]: purged_snaps scrub ok 2024-09-18T08:13:02.482 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:13:02 smithi007 ceph-mon[39249]: pgmap v494: 158 pgs: 30 active+undersized, 34 active+undersized+degraded, 94 active+clean; 7.4 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 55 KiB/s rd, 0 B/s wr, 1.97k op/s; 69070/366567 objects degraded (18.842%) 2024-09-18T08:13:02.531 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3 2024-09-18T08:13:02.562 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:13:02.563 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:13:02.564 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3' 2024-09-18T08:13:02.656 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/779349ca-7593-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T08:13:02.664 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5 2024-09-18T08:13:02.701 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:13:02.702 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:13:02.702 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5' 2024-09-18T08:13:02.803 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0 2024-09-18T08:13:02.833 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:13:02.834 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:13:02.834 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0' 2024-09-18T08:13:02.935 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7 2024-09-18T08:13:02.970 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:13:02.971 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:13:02.971 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7' 2024-09-18T08:13:03.072 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5 2024-09-18T08:13:03.108 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:13:03.109 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:13:03.109 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5' 2024-09-18T08:13:03.186 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/779349ca-7593-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T08:13:03.209 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2 2024-09-18T08:13:03.250 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:13:03.251 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:13:03.251 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2' 2024-09-18T08:13:03.352 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6 2024-09-18T08:13:03.393 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:13:03.394 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:13:03.394 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6' 2024-09-18T08:13:03.405 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:13:03 smithi050 ceph-mon[32371]: osdmap e273: 8 total, 7 up, 8 in 2024-09-18T08:13:03.434 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:13:03 smithi007 ceph-mon[29459]: osdmap e273: 8 total, 7 up, 8 in 2024-09-18T08:13:03.434 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:13:03 smithi007 ceph-mon[39249]: osdmap e273: 8 total, 7 up, 8 in 2024-09-18T08:13:03.495 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3 2024-09-18T08:13:03.542 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:13:03.543 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:13:03.543 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3' 2024-09-18T08:13:03.644 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0 2024-09-18T08:13:03.682 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:13:03.683 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:13:03.684 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0' 2024-09-18T08:13:03.692 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/779349ca-7593-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T08:13:03.738 INFO:teuthology.orchestra.run.smithi007.stdout:ERROR: (22) Invalid argument 2024-09-18T08:13:03.738 INFO:teuthology.orchestra.run.smithi007.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-18T08:13:03.785 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5 2024-09-18T08:13:03.820 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:13:03.821 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:13:03.821 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5' 2024-09-18T08:13:03.922 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7 2024-09-18T08:13:03.958 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:13:03.959 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:13:03.959 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7' 2024-09-18T08:13:04.060 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3 2024-09-18T08:13:04.097 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:13:04.097 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:13:04.098 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3' 2024-09-18T08:13:04.199 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5 2024-09-18T08:13:04.234 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:13:04.234 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:13:04.235 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5' 2024-09-18T08:13:04.335 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3 2024-09-18T08:13:04.372 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:13:04.373 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:13:04.373 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3' 2024-09-18T08:13:04.474 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7 2024-09-18T08:13:04.513 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:13:04.514 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:13:04.514 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7' 2024-09-18T08:13:04.615 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0 2024-09-18T08:13:04.649 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:13:04.649 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:13:04.650 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0' 2024-09-18T08:13:04.748 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:13:04 smithi007 ceph-mon[29459]: pgmap v496: 158 pgs: 30 active+undersized, 34 active+undersized+degraded, 94 active+clean; 7.4 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 54 KiB/s rd, 0 B/s wr, 1.05k op/s; 69070/366567 objects degraded (18.842%) 2024-09-18T08:13:04.748 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:13:04 smithi007 ceph-mon[29459]: from='mgr.24470 172.21.15.7:0/2594390261' entity='mgr.y' 2024-09-18T08:13:04.748 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:13:04 smithi007 ceph-mon[29459]: from='mgr.24470 172.21.15.7:0/2594390261' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-18T08:13:04.748 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:13:04 smithi007 ceph-mon[29459]: Health check update: Degraded data redundancy: 69070/366567 objects degraded (18.842%), 34 pgs degraded, 64 pgs undersized (PG_DEGRADED) 2024-09-18T08:13:04.748 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:13:04 smithi007 ceph-mon[39249]: pgmap v496: 158 pgs: 30 active+undersized, 34 active+undersized+degraded, 94 active+clean; 7.4 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 54 KiB/s rd, 0 B/s wr, 1.05k op/s; 69070/366567 objects degraded (18.842%) 2024-09-18T08:13:04.748 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:13:04 smithi007 ceph-mon[39249]: from='mgr.24470 172.21.15.7:0/2594390261' entity='mgr.y' 2024-09-18T08:13:04.749 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:13:04 smithi007 ceph-mon[39249]: from='mgr.24470 172.21.15.7:0/2594390261' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-18T08:13:04.749 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:13:04 smithi007 ceph-mon[39249]: Health check update: Degraded data redundancy: 69070/366567 objects degraded (18.842%), 34 pgs degraded, 64 pgs undersized (PG_DEGRADED) 2024-09-18T08:13:04.750 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid 779349ca-7593-11ef-bceb-c7b262605968 -- ceph daemon osd.2 dump_blocked_ops 2024-09-18T08:13:04.750 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6 2024-09-18T08:13:04.785 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:13:04.786 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:13:04.786 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6' 2024-09-18T08:13:04.887 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5 2024-09-18T08:13:04.906 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:13:04 smithi050 ceph-mon[32371]: pgmap v496: 158 pgs: 30 active+undersized, 34 active+undersized+degraded, 94 active+clean; 7.4 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 54 KiB/s rd, 0 B/s wr, 1.05k op/s; 69070/366567 objects degraded (18.842%) 2024-09-18T08:13:04.906 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:13:04 smithi050 ceph-mon[32371]: from='mgr.24470 172.21.15.7:0/2594390261' entity='mgr.y' 2024-09-18T08:13:04.906 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:13:04 smithi050 ceph-mon[32371]: from='mgr.24470 172.21.15.7:0/2594390261' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-18T08:13:04.906 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:13:04 smithi050 ceph-mon[32371]: Health check update: Degraded data redundancy: 69070/366567 objects degraded (18.842%), 34 pgs degraded, 64 pgs undersized (PG_DEGRADED) 2024-09-18T08:13:04.923 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:13:04.924 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:13:04.924 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5' 2024-09-18T08:13:05.025 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2 2024-09-18T08:13:05.058 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:13:05.059 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:13:05.059 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2' 2024-09-18T08:13:05.160 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6 2024-09-18T08:13:05.197 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:13:05.198 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:13:05.198 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6' 2024-09-18T08:13:05.247 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid 779349ca-7593-11ef-bceb-c7b262605968 -- ceph osd set noscrub 2024-09-18T08:13:05.299 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5 2024-09-18T08:13:05.333 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:13:05.334 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:13:05.335 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5' 2024-09-18T08:13:05.436 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2 2024-09-18T08:13:05.478 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:13:05.478 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:13:05.478 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2' 2024-09-18T08:13:05.498 INFO:teuthology.orchestra.run.smithi007.stdout:osd.0: {} 2024-09-18T08:13:05.499 INFO:teuthology.orchestra.run.smithi007.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-18T08:13:05.500 INFO:teuthology.orchestra.run.smithi007.stderr:Error ENXIO: problem getting command descriptions from osd.1 2024-09-18T08:13:05.540 INFO:teuthology.orchestra.run.smithi007.stdout:osd.2: {} 2024-09-18T08:13:05.540 INFO:teuthology.orchestra.run.smithi007.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-18T08:13:05.579 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0 2024-09-18T08:13:05.580 INFO:teuthology.orchestra.run.smithi007.stdout:osd.3: {} 2024-09-18T08:13:05.581 INFO:teuthology.orchestra.run.smithi007.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-18T08:13:05.611 INFO:teuthology.orchestra.run.smithi007.stdout:osd.4: {} 2024-09-18T08:13:05.612 INFO:teuthology.orchestra.run.smithi007.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-18T08:13:05.622 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:13:05.623 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:13:05.623 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0' 2024-09-18T08:13:05.652 INFO:teuthology.orchestra.run.smithi007.stdout:osd.5: {} 2024-09-18T08:13:05.652 INFO:teuthology.orchestra.run.smithi007.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-18T08:13:05.692 INFO:teuthology.orchestra.run.smithi007.stdout:osd.6: {} 2024-09-18T08:13:05.692 INFO:teuthology.orchestra.run.smithi007.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-18T08:13:05.724 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3 2024-09-18T08:13:05.733 INFO:teuthology.orchestra.run.smithi007.stdout:{ 2024-09-18T08:13:05.733 INFO:teuthology.orchestra.run.smithi007.stdout: "success": "" 2024-09-18T08:13:05.733 INFO:teuthology.orchestra.run.smithi007.stdout:} 2024-09-18T08:13:05.741 INFO:teuthology.orchestra.run.smithi007.stdout:osd.7: {} 2024-09-18T08:13:05.741 INFO:teuthology.orchestra.run.smithi007.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-18T08:13:05.781 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:13:05.782 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:13:05.783 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3' 2024-09-18T08:13:05.883 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2 2024-09-18T08:13:05.906 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:13:05 smithi050 ceph-mon[32371]: purged_snaps scrub starts 2024-09-18T08:13:05.906 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:13:05 smithi050 ceph-mon[32371]: purged_snaps scrub ok 2024-09-18T08:13:05.906 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:13:05 smithi050 ceph-mon[32371]: 9.c scrub starts 2024-09-18T08:13:05.906 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:13:05 smithi050 ceph-mon[32371]: osdmap e274: 8 total, 7 up, 8 in 2024-09-18T08:13:05.906 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:13:05 smithi050 ceph-mon[32371]: from='client.? 172.21.15.7:0/3418924080' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-18T08:13:05.917 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:13:05.917 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:13:05.918 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2' 2024-09-18T08:13:05.979 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:13:05 smithi007 ceph-mon[29459]: purged_snaps scrub starts 2024-09-18T08:13:05.980 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:13:05 smithi007 ceph-mon[29459]: purged_snaps scrub ok 2024-09-18T08:13:05.980 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:13:05 smithi007 ceph-mon[29459]: 9.c scrub starts 2024-09-18T08:13:05.980 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:13:05 smithi007 ceph-mon[29459]: osdmap e274: 8 total, 7 up, 8 in 2024-09-18T08:13:05.980 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:13:05 smithi007 ceph-mon[29459]: from='client.? 172.21.15.7:0/3418924080' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-18T08:13:05.980 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:13:05 smithi007 ceph-mon[39249]: purged_snaps scrub starts 2024-09-18T08:13:05.981 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:13:05 smithi007 ceph-mon[39249]: purged_snaps scrub ok 2024-09-18T08:13:05.981 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:13:05 smithi007 ceph-mon[39249]: 9.c scrub starts 2024-09-18T08:13:05.981 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:13:05 smithi007 ceph-mon[39249]: osdmap e274: 8 total, 7 up, 8 in 2024-09-18T08:13:05.981 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:13:05 smithi007 ceph-mon[39249]: from='client.? 172.21.15.7:0/3418924080' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-18T08:13:06.019 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4 2024-09-18T08:13:06.055 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:13:06.055 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:13:06.056 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4' 2024-09-18T08:13:06.157 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4 2024-09-18T08:13:06.194 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:13:06.195 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:13:06.195 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4' 2024-09-18T08:13:06.296 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2 2024-09-18T08:13:06.325 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:13:06.326 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:13:06.326 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2' 2024-09-18T08:13:06.427 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5 2024-09-18T08:13:06.448 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-09-18T08:13:06.449 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-09-18T08:13:06.467 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:13:06.467 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:13:06.468 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5' 2024-09-18T08:13:06.568 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0 2024-09-18T08:13:06.598 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:13:06.601 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:13:06.601 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0' 2024-09-18T08:13:06.610 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/779349ca-7593-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T08:13:06.696 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:13:06 smithi007 ceph-mon[29459]: pgmap v498: 158 pgs: 1 active+clean+scrubbing, 30 active+undersized, 34 active+undersized+degraded, 93 active+clean; 7.0 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 852 B/s rd, 0 B/s wr, 1.15k op/s; 65202/345912 objects degraded (18.849%) 2024-09-18T08:13:06.697 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:13:06 smithi007 ceph-mon[29459]: 9.9 scrub starts 2024-09-18T08:13:06.697 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:13:06 smithi007 ceph-mon[39249]: pgmap v498: 158 pgs: 1 active+clean+scrubbing, 30 active+undersized, 34 active+undersized+degraded, 93 active+clean; 7.0 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 852 B/s rd, 0 B/s wr, 1.15k op/s; 65202/345912 objects degraded (18.849%) 2024-09-18T08:13:06.697 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:13:06 smithi007 ceph-mon[39249]: 9.9 scrub starts 2024-09-18T08:13:06.702 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5 2024-09-18T08:13:06.740 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:13:06.741 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:13:06.744 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5' 2024-09-18T08:13:06.845 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5 2024-09-18T08:13:06.882 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:13:06.882 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:13:06.883 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5' 2024-09-18T08:13:06.905 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:13:06 smithi050 ceph-mon[32371]: pgmap v498: 158 pgs: 1 active+clean+scrubbing, 30 active+undersized, 34 active+undersized+degraded, 93 active+clean; 7.0 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 852 B/s rd, 0 B/s wr, 1.15k op/s; 65202/345912 objects degraded (18.849%) 2024-09-18T08:13:06.905 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:13:06 smithi050 ceph-mon[32371]: 9.9 scrub starts 2024-09-18T08:13:06.980 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Sep 18 08:13:06 smithi007 ceph-779349ca-7593-11ef-bceb-c7b262605968-mgr-y[29688]: ::ffff:172.21.15.50 - - [18/Sep/2024:08:13:06] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-09-18T08:13:06.983 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0 2024-09-18T08:13:07.019 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:13:07.020 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:13:07.020 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0' 2024-09-18T08:13:07.100 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/779349ca-7593-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T08:13:07.121 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2 2024-09-18T08:13:07.155 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:13:07.156 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:13:07.156 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2' 2024-09-18T08:13:07.257 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4 2024-09-18T08:13:07.298 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:13:07.299 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:13:07.299 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4' 2024-09-18T08:13:07.400 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7 2024-09-18T08:13:07.432 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:13:07.433 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:13:07.433 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7' 2024-09-18T08:13:07.534 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7 2024-09-18T08:13:07.570 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:13:07.570 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:13:07.570 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7' 2024-09-18T08:13:07.623 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:13:07 smithi007 ceph-mon[29459]: 9.c scrub ok 2024-09-18T08:13:07.623 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:13:07 smithi007 ceph-mon[29459]: 3.10 scrub starts 2024-09-18T08:13:07.623 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:13:07 smithi007 ceph-mon[29459]: 3.10 scrub ok 2024-09-18T08:13:07.623 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:13:07 smithi007 ceph-mon[39249]: 9.c scrub ok 2024-09-18T08:13:07.624 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:13:07 smithi007 ceph-mon[39249]: 3.10 scrub starts 2024-09-18T08:13:07.624 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:13:07 smithi007 ceph-mon[39249]: 3.10 scrub ok 2024-09-18T08:13:07.671 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3 2024-09-18T08:13:07.683 INFO:teuthology.orchestra.run.smithi007.stdout:{ 2024-09-18T08:13:07.683 INFO:teuthology.orchestra.run.smithi007.stdout: "ops": [], 2024-09-18T08:13:07.683 INFO:teuthology.orchestra.run.smithi007.stdout: "complaint_time": 30, 2024-09-18T08:13:07.683 INFO:teuthology.orchestra.run.smithi007.stdout: "num_blocked_ops": 0 2024-09-18T08:13:07.683 INFO:teuthology.orchestra.run.smithi007.stdout:} 2024-09-18T08:13:07.709 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:13:07.710 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:13:07.710 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3' 2024-09-18T08:13:07.811 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2 2024-09-18T08:13:07.843 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:13:07.843 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:13:07.844 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2' 2024-09-18T08:13:07.906 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:13:07 smithi050 ceph-mon[32371]: 9.c scrub ok 2024-09-18T08:13:07.906 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:13:07 smithi050 ceph-mon[32371]: 3.10 scrub starts 2024-09-18T08:13:07.906 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:13:07 smithi050 ceph-mon[32371]: 3.10 scrub ok 2024-09-18T08:13:07.944 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3 2024-09-18T08:13:07.975 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:13:07.976 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:13:07.976 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3' 2024-09-18T08:13:08.076 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2 2024-09-18T08:13:08.106 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:13:08.107 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:13:08.107 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2' 2024-09-18T08:13:08.208 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0 2024-09-18T08:13:08.238 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:13:08.239 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:13:08.239 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0' 2024-09-18T08:13:08.340 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5 2024-09-18T08:13:08.377 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:13:08.378 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:13:08.378 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5' 2024-09-18T08:13:08.450 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid 779349ca-7593-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-18T08:13:08.479 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4 2024-09-18T08:13:08.514 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:13:08.514 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:13:08.515 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4' 2024-09-18T08:13:08.615 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5 2024-09-18T08:13:08.645 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:13:08 smithi007 ceph-mon[29459]: pgmap v499: 158 pgs: 1 active+clean+scrubbing, 30 active+undersized, 34 active+undersized+degraded, 93 active+clean; 7.0 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 820 B/s rd, 0 B/s wr, 1.10k op/s; 65202/345912 objects degraded (18.849%) 2024-09-18T08:13:08.646 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:13:08 smithi007 ceph-mon[29459]: 9.9 scrub ok 2024-09-18T08:13:08.646 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:13:08 smithi007 ceph-mon[39249]: pgmap v499: 158 pgs: 1 active+clean+scrubbing, 30 active+undersized, 34 active+undersized+degraded, 93 active+clean; 7.0 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 820 B/s rd, 0 B/s wr, 1.10k op/s; 65202/345912 objects degraded (18.849%) 2024-09-18T08:13:08.646 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:13:08 smithi007 ceph-mon[39249]: 9.9 scrub ok 2024-09-18T08:13:08.654 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:13:08.655 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:13:08.655 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5' 2024-09-18T08:13:08.756 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3 2024-09-18T08:13:08.789 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:13:08.789 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:13:08.789 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3' 2024-09-18T08:13:08.890 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5 2024-09-18T08:13:08.907 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:13:08 smithi050 ceph-mon[32371]: pgmap v499: 158 pgs: 1 active+clean+scrubbing, 30 active+undersized, 34 active+undersized+degraded, 93 active+clean; 7.0 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 820 B/s rd, 0 B/s wr, 1.10k op/s; 65202/345912 objects degraded (18.849%) 2024-09-18T08:13:08.907 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:13:08 smithi050 ceph-mon[32371]: 9.9 scrub ok 2024-09-18T08:13:08.930 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:13:08.930 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:13:08.930 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5' 2024-09-18T08:13:08.956 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid 779349ca-7593-11ef-bceb-c7b262605968 -- ceph daemon osd.2 dump_historic_ops 2024-09-18T08:13:09.031 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3 2024-09-18T08:13:09.033 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/779349ca-7593-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T08:13:09.090 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:13:09.091 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:13:09.091 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3' 2024-09-18T08:13:09.192 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4 2024-09-18T08:13:09.223 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:13:09.223 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:13:09.223 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4' 2024-09-18T08:13:09.312 INFO:teuthology.orchestra.run.smithi007.stderr:noscrub is set 2024-09-18T08:13:09.324 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6 2024-09-18T08:13:09.357 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:13:09.358 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:13:09.358 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6' 2024-09-18T08:13:09.459 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0 2024-09-18T08:13:09.493 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:13:09.494 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:13:09.494 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0' 2024-09-18T08:13:09.595 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6 2024-09-18T08:13:09.633 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:13:09.633 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:13:09.634 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6' 2024-09-18T08:13:09.655 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:13:09 smithi050 ceph-mon[32371]: from='client.? 172.21.15.7:0/2038378000' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-18T08:13:09.655 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:13:09 smithi050 ceph-mon[32371]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-18T08:13:09.655 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:13:09 smithi050 ceph-mon[32371]: Health check update: Degraded data redundancy: 65202/345912 objects degraded (18.849%), 34 pgs degraded, 64 pgs undersized (PG_DEGRADED) 2024-09-18T08:13:09.656 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:13:09 smithi050 ceph-mon[32371]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-18T08:13:09.656 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:13:09 smithi050 ceph-mon[32371]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-18T08:13:09.656 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:13:09 smithi050 ceph-mon[32371]: osdmap e275: 8 total, 7 up, 8 in 2024-09-18T08:13:09.656 INFO:journalctl@ceph.iscsi.iscsi.a.smithi050.stdout:Sep 18 08:13:09 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-iscsi-iscsi-a[57782]: debug there is no tcmu-runner data available 2024-09-18T08:13:09.730 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:13:09 smithi007 ceph-mon[29459]: from='client.? 172.21.15.7:0/2038378000' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-18T08:13:09.730 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:13:09 smithi007 ceph-mon[29459]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-18T08:13:09.730 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:13:09 smithi007 ceph-mon[29459]: Health check update: Degraded data redundancy: 65202/345912 objects degraded (18.849%), 34 pgs degraded, 64 pgs undersized (PG_DEGRADED) 2024-09-18T08:13:09.730 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:13:09 smithi007 ceph-mon[29459]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-18T08:13:09.730 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:13:09 smithi007 ceph-mon[29459]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-18T08:13:09.730 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:13:09 smithi007 ceph-mon[29459]: osdmap e275: 8 total, 7 up, 8 in 2024-09-18T08:13:09.731 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:13:09 smithi007 ceph-mon[39249]: from='client.? 172.21.15.7:0/2038378000' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-18T08:13:09.731 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:13:09 smithi007 ceph-mon[39249]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-18T08:13:09.731 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:13:09 smithi007 ceph-mon[39249]: Health check update: Degraded data redundancy: 65202/345912 objects degraded (18.849%), 34 pgs degraded, 64 pgs undersized (PG_DEGRADED) 2024-09-18T08:13:09.731 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:13:09 smithi007 ceph-mon[39249]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-18T08:13:09.731 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:13:09 smithi007 ceph-mon[39249]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-18T08:13:09.731 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:13:09 smithi007 ceph-mon[39249]: osdmap e275: 8 total, 7 up, 8 in 2024-09-18T08:13:09.734 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5 2024-09-18T08:13:09.766 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:13:09.766 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:13:09.766 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5' 2024-09-18T08:13:09.841 INFO:teuthology.orchestra.run.smithi007.stdout:osd.0: {} 2024-09-18T08:13:09.841 INFO:teuthology.orchestra.run.smithi007.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-18T08:13:09.843 INFO:teuthology.orchestra.run.smithi007.stderr:Error ENXIO: problem getting command descriptions from osd.1 2024-09-18T08:13:09.867 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0 2024-09-18T08:13:09.885 INFO:teuthology.orchestra.run.smithi007.stdout:osd.2: {} 2024-09-18T08:13:09.886 INFO:teuthology.orchestra.run.smithi007.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-18T08:13:09.918 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:13:09.919 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:13:09.919 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0' 2024-09-18T08:13:09.921 INFO:teuthology.orchestra.run.smithi007.stdout:osd.3: {} 2024-09-18T08:13:09.922 INFO:teuthology.orchestra.run.smithi007.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-18T08:13:09.952 INFO:teuthology.orchestra.run.smithi007.stdout:osd.4: {} 2024-09-18T08:13:09.952 INFO:teuthology.orchestra.run.smithi007.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-18T08:13:09.987 INFO:teuthology.orchestra.run.smithi007.stdout:osd.5: {} 2024-09-18T08:13:09.988 INFO:teuthology.orchestra.run.smithi007.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-18T08:13:10.020 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0 2024-09-18T08:13:10.024 INFO:teuthology.orchestra.run.smithi007.stdout:osd.6: {} 2024-09-18T08:13:10.024 INFO:teuthology.orchestra.run.smithi007.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-18T08:13:10.056 INFO:teuthology.orchestra.run.smithi007.stdout:osd.7: {} 2024-09-18T08:13:10.057 INFO:teuthology.orchestra.run.smithi007.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-18T08:13:10.059 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:13:10.059 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:13:10.060 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0' 2024-09-18T08:13:10.160 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5 2024-09-18T08:13:10.197 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:13:10.197 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:13:10.198 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5' 2024-09-18T08:13:10.299 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5 2024-09-18T08:13:10.333 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:13:10.334 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:13:10.335 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5' 2024-09-18T08:13:10.435 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6 2024-09-18T08:13:10.481 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:13:10.482 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:13:10.482 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6' 2024-09-18T08:13:10.583 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3 2024-09-18T08:13:10.620 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:13:10.621 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:13:10.621 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3' 2024-09-18T08:13:10.722 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2 2024-09-18T08:13:10.730 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:13:10 smithi007 ceph-mon[29459]: pgmap v500: 158 pgs: 30 active+undersized, 34 active+undersized+degraded, 94 active+clean; 6.6 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.76k op/s; 61165/324216 objects degraded (18.866%) 2024-09-18T08:13:10.730 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:13:10 smithi007 ceph-mon[29459]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-18T08:13:10.730 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:13:10 smithi007 ceph-mon[29459]: from='client.? 172.21.15.7:0/3022238136' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-18T08:13:10.730 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:13:10 smithi007 ceph-mon[29459]: purged_snaps scrub starts 2024-09-18T08:13:10.730 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:13:10 smithi007 ceph-mon[29459]: purged_snaps scrub ok 2024-09-18T08:13:10.731 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:13:10 smithi007 ceph-mon[39249]: pgmap v500: 158 pgs: 30 active+undersized, 34 active+undersized+degraded, 94 active+clean; 6.6 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.76k op/s; 61165/324216 objects degraded (18.866%) 2024-09-18T08:13:10.731 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:13:10 smithi007 ceph-mon[39249]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-18T08:13:10.731 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:13:10 smithi007 ceph-mon[39249]: from='client.? 172.21.15.7:0/3022238136' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-18T08:13:10.731 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:13:10 smithi007 ceph-mon[39249]: purged_snaps scrub starts 2024-09-18T08:13:10.731 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:13:10 smithi007 ceph-mon[39249]: purged_snaps scrub ok 2024-09-18T08:13:10.758 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:13:10.759 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:13:10.759 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2' 2024-09-18T08:13:10.860 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5 2024-09-18T08:13:10.888 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:13:10.889 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:13:10.889 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5' 2024-09-18T08:13:10.905 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:13:10 smithi050 ceph-mon[32371]: pgmap v500: 158 pgs: 30 active+undersized, 34 active+undersized+degraded, 94 active+clean; 6.6 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.76k op/s; 61165/324216 objects degraded (18.866%) 2024-09-18T08:13:10.906 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:13:10 smithi050 ceph-mon[32371]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-18T08:13:10.906 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:13:10 smithi050 ceph-mon[32371]: from='client.? 172.21.15.7:0/3022238136' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-18T08:13:10.906 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:13:10 smithi050 ceph-mon[32371]: purged_snaps scrub starts 2024-09-18T08:13:10.906 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:13:10 smithi050 ceph-mon[32371]: purged_snaps scrub ok 2024-09-18T08:13:10.990 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0 2024-09-18T08:13:11.026 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:13:11.027 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:13:11.027 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0' 2024-09-18T08:13:11.128 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4 2024-09-18T08:13:11.166 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:13:11.166 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:13:11.167 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4' 2024-09-18T08:13:11.267 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5 2024-09-18T08:13:11.307 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:13:11.307 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:13:11.308 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5' 2024-09-18T08:13:11.409 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3 2024-09-18T08:13:11.437 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:13:11.438 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:13:11.438 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3' 2024-09-18T08:13:11.539 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3 2024-09-18T08:13:11.572 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:13:11.573 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:13:11.573 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3' 2024-09-18T08:13:11.674 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4 2024-09-18T08:13:11.714 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:13:11.715 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:13:11.715 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4' 2024-09-18T08:13:11.816 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3 2024-09-18T08:13:11.846 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:13:11.847 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:13:11.847 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3' 2024-09-18T08:13:11.905 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:13:11 smithi050 ceph-mon[32371]: osdmap e276: 8 total, 7 up, 8 in 2024-09-18T08:13:11.948 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0 2024-09-18T08:13:11.980 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:13:11 smithi007 ceph-mon[29459]: osdmap e276: 8 total, 7 up, 8 in 2024-09-18T08:13:11.980 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:13:11 smithi007 ceph-mon[39249]: osdmap e276: 8 total, 7 up, 8 in 2024-09-18T08:13:11.982 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:13:11.983 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:13:11.983 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0' 2024-09-18T08:13:12.035 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-09-18T08:13:12.036 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-09-18T08:13:12.063 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 5, 7, 6, 4] out_osds: [] dead_osds: [1] live_osds: [0, 2, 3, 4, 5, 6, 7] 2024-09-18T08:13:12.063 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-18T08:13:12.063 INFO:tasks.thrashosds.thrasher:thrash_pg_upmap 2024-09-18T08:13:12.063 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid 779349ca-7593-11ef-bceb-c7b262605968 -- ceph osd dump -f json-pretty 2024-09-18T08:13:12.076 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/779349ca-7593-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T08:13:12.083 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5 2024-09-18T08:13:12.120 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:13:12.121 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:13:12.121 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5' 2024-09-18T08:13:12.222 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2 2024-09-18T08:13:12.257 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:13:12.258 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:13:12.258 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2' 2024-09-18T08:13:12.359 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7 2024-09-18T08:13:12.360 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid 779349ca-7593-11ef-bceb-c7b262605968 -- ceph osd set nodeep-scrub 2024-09-18T08:13:12.396 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:13:12.397 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:13:12.397 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7' 2024-09-18T08:13:12.498 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7 2024-09-18T08:13:12.538 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:13:12.539 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:13:12.539 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7' 2024-09-18T08:13:12.640 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0 2024-09-18T08:13:12.692 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:13:12.692 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:13:12.692 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0' 2024-09-18T08:13:12.718 INFO:teuthology.orchestra.run.smithi007.stdout:ERROR: (22) Invalid argument 2024-09-18T08:13:12.718 INFO:teuthology.orchestra.run.smithi007.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-18T08:13:12.729 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/779349ca-7593-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T08:13:12.793 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2 2024-09-18T08:13:12.807 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:13:12 smithi050 ceph-mon[32371]: pgmap v503: 158 pgs: 30 active+undersized, 34 active+undersized+degraded, 94 active+clean; 6.6 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 751 B/s rd, 0 B/s wr, 1.06k op/s; 61165/324216 objects degraded (18.866%) 2024-09-18T08:13:12.822 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:13:12.823 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:13:12.823 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2' 2024-09-18T08:13:12.924 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5 2024-09-18T08:13:12.965 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:13:12.965 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:13:12.966 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5' 2024-09-18T08:13:12.980 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:13:12 smithi007 ceph-mon[29459]: pgmap v503: 158 pgs: 30 active+undersized, 34 active+undersized+degraded, 94 active+clean; 6.6 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 751 B/s rd, 0 B/s wr, 1.06k op/s; 61165/324216 objects degraded (18.866%) 2024-09-18T08:13:12.980 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:13:12 smithi007 ceph-mon[39249]: pgmap v503: 158 pgs: 30 active+undersized, 34 active+undersized+degraded, 94 active+clean; 6.6 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 751 B/s rd, 0 B/s wr, 1.06k op/s; 61165/324216 objects degraded (18.866%) 2024-09-18T08:13:13.067 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2 2024-09-18T08:13:13.101 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:13:13.101 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:13:13.102 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2' 2024-09-18T08:13:13.203 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5 2024-09-18T08:13:13.249 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:13:13.249 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:13:13.250 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5' 2024-09-18T08:13:13.351 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0 2024-09-18T08:13:13.382 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:13:13.382 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:13:13.383 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0' 2024-09-18T08:13:13.431 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid 779349ca-7593-11ef-bceb-c7b262605968 -- ceph daemon osd.3 dump_ops_in_flight 2024-09-18T08:13:13.483 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0 2024-09-18T08:13:13.514 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:13:13.515 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:13:13.516 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0' 2024-09-18T08:13:13.616 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4 2024-09-18T08:13:13.664 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:13:13.665 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:13:13.665 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4' 2024-09-18T08:13:13.766 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4 2024-09-18T08:13:13.812 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:13:13.813 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:13:13.813 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4' 2024-09-18T08:13:13.914 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4 2024-09-18T08:13:13.954 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:13:13.955 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:13:13.956 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4' 2024-09-18T08:13:13.978 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/779349ca-7593-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T08:13:14.037 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid 779349ca-7593-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-18T08:13:14.056 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3 2024-09-18T08:13:14.117 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:13:14.118 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:13:14.118 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3' 2024-09-18T08:13:14.219 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6 2024-09-18T08:13:14.256 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:13:14.256 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:13:14.257 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6' 2024-09-18T08:13:14.286 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-09-18T08:13:14.287 INFO:teuthology.orchestra.run.smithi007.stdout:{ 2024-09-18T08:13:14.287 INFO:teuthology.orchestra.run.smithi007.stdout: "epoch": 276, 2024-09-18T08:13:14.287 INFO:teuthology.orchestra.run.smithi007.stdout: "fsid": "779349ca-7593-11ef-bceb-c7b262605968", 2024-09-18T08:13:14.287 INFO:teuthology.orchestra.run.smithi007.stdout: "created": "2024-09-18T07:57:09.295710+0000", 2024-09-18T08:13:14.287 INFO:teuthology.orchestra.run.smithi007.stdout: "modified": "2024-09-18T08:13:10.525594+0000", 2024-09-18T08:13:14.287 INFO:teuthology.orchestra.run.smithi007.stdout: "last_up_change": "2024-09-18T08:11:41.300261+0000", 2024-09-18T08:13:14.287 INFO:teuthology.orchestra.run.smithi007.stdout: "last_in_change": "2024-09-18T08:08:17.483129+0000", 2024-09-18T08:13:14.287 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": "noscrub,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit", 2024-09-18T08:13:14.287 INFO:teuthology.orchestra.run.smithi007.stdout: "flags_num": 5801984, 2024-09-18T08:13:14.287 INFO:teuthology.orchestra.run.smithi007.stdout: "flags_set": [ 2024-09-18T08:13:14.287 INFO:teuthology.orchestra.run.smithi007.stdout: "noscrub", 2024-09-18T08:13:14.287 INFO:teuthology.orchestra.run.smithi007.stdout: "pglog_hardlimit", 2024-09-18T08:13:14.287 INFO:teuthology.orchestra.run.smithi007.stdout: "purged_snapdirs", 2024-09-18T08:13:14.288 INFO:teuthology.orchestra.run.smithi007.stdout: "recovery_deletes", 2024-09-18T08:13:14.288 INFO:teuthology.orchestra.run.smithi007.stdout: "sortbitwise" 2024-09-18T08:13:14.288 INFO:teuthology.orchestra.run.smithi007.stdout: ], 2024-09-18T08:13:14.288 INFO:teuthology.orchestra.run.smithi007.stdout: "crush_version": 18, 2024-09-18T08:13:14.288 INFO:teuthology.orchestra.run.smithi007.stdout: "full_ratio": 0.94999998807907104, 2024-09-18T08:13:14.288 INFO:teuthology.orchestra.run.smithi007.stdout: "backfillfull_ratio": 0.89999997615814209, 2024-09-18T08:13:14.288 INFO:teuthology.orchestra.run.smithi007.stdout: "nearfull_ratio": 0.85000002384185791, 2024-09-18T08:13:14.288 INFO:teuthology.orchestra.run.smithi007.stdout: "cluster_snapshot": "", 2024-09-18T08:13:14.288 INFO:teuthology.orchestra.run.smithi007.stdout: "pool_max": 9, 2024-09-18T08:13:14.288 INFO:teuthology.orchestra.run.smithi007.stdout: "max_osd": 8, 2024-09-18T08:13:14.288 INFO:teuthology.orchestra.run.smithi007.stdout: "require_min_compat_client": "luminous", 2024-09-18T08:13:14.288 INFO:teuthology.orchestra.run.smithi007.stdout: "min_compat_client": "luminous", 2024-09-18T08:13:14.288 INFO:teuthology.orchestra.run.smithi007.stdout: "require_osd_release": "squid", 2024-09-18T08:13:14.289 INFO:teuthology.orchestra.run.smithi007.stdout: "allow_crimson": false, 2024-09-18T08:13:14.289 INFO:teuthology.orchestra.run.smithi007.stdout: "pools": [ 2024-09-18T08:13:14.289 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:13:14.289 INFO:teuthology.orchestra.run.smithi007.stdout: "pool": 1, 2024-09-18T08:13:14.289 INFO:teuthology.orchestra.run.smithi007.stdout: "pool_name": ".mgr", 2024-09-18T08:13:14.289 INFO:teuthology.orchestra.run.smithi007.stdout: "create_time": "2024-09-18T08:00:11.802627+0000", 2024-09-18T08:13:14.289 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 1, 2024-09-18T08:13:14.289 INFO:teuthology.orchestra.run.smithi007.stdout: "flags_names": "hashpspool", 2024-09-18T08:13:14.289 INFO:teuthology.orchestra.run.smithi007.stdout: "type": 1, 2024-09-18T08:13:14.289 INFO:teuthology.orchestra.run.smithi007.stdout: "size": 3, 2024-09-18T08:13:14.289 INFO:teuthology.orchestra.run.smithi007.stdout: "min_size": 2, 2024-09-18T08:13:14.289 INFO:teuthology.orchestra.run.smithi007.stdout: "crush_rule": 0, 2024-09-18T08:13:14.289 INFO:teuthology.orchestra.run.smithi007.stdout: "peering_crush_bucket_count": 0, 2024-09-18T08:13:14.290 INFO:teuthology.orchestra.run.smithi007.stdout: "peering_crush_bucket_target": 0, 2024-09-18T08:13:14.290 INFO:teuthology.orchestra.run.smithi007.stdout: "peering_crush_bucket_barrier": 0, 2024-09-18T08:13:14.290 INFO:teuthology.orchestra.run.smithi007.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-09-18T08:13:14.290 INFO:teuthology.orchestra.run.smithi007.stdout: "is_stretch_pool": false, 2024-09-18T08:13:14.290 INFO:teuthology.orchestra.run.smithi007.stdout: "object_hash": 2, 2024-09-18T08:13:14.290 INFO:teuthology.orchestra.run.smithi007.stdout: "pg_autoscale_mode": "off", 2024-09-18T08:13:14.290 INFO:teuthology.orchestra.run.smithi007.stdout: "pg_num": 11, 2024-09-18T08:13:14.290 INFO:teuthology.orchestra.run.smithi007.stdout: "pg_placement_num": 11, 2024-09-18T08:13:14.290 INFO:teuthology.orchestra.run.smithi007.stdout: "pg_placement_num_target": 11, 2024-09-18T08:13:14.291 INFO:teuthology.orchestra.run.smithi007.stdout: "pg_num_target": 11, 2024-09-18T08:13:14.291 INFO:teuthology.orchestra.run.smithi007.stdout: "pg_num_pending": 11, 2024-09-18T08:13:14.291 INFO:teuthology.orchestra.run.smithi007.stdout: "last_pg_merge_meta": { 2024-09-18T08:13:14.291 INFO:teuthology.orchestra.run.smithi007.stdout: "source_pgid": "0.0", 2024-09-18T08:13:14.291 INFO:teuthology.orchestra.run.smithi007.stdout: "ready_epoch": 0, 2024-09-18T08:13:14.291 INFO:teuthology.orchestra.run.smithi007.stdout: "last_epoch_started": 0, 2024-09-18T08:13:14.291 INFO:teuthology.orchestra.run.smithi007.stdout: "last_epoch_clean": 0, 2024-09-18T08:13:14.291 INFO:teuthology.orchestra.run.smithi007.stdout: "source_version": "0'0", 2024-09-18T08:13:14.291 INFO:teuthology.orchestra.run.smithi007.stdout: "target_version": "0'0" 2024-09-18T08:13:14.291 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:13:14.291 INFO:teuthology.orchestra.run.smithi007.stdout: "last_change": "154", 2024-09-18T08:13:14.291 INFO:teuthology.orchestra.run.smithi007.stdout: "last_force_op_resend": "0", 2024-09-18T08:13:14.292 INFO:teuthology.orchestra.run.smithi007.stdout: "last_force_op_resend_prenautilus": "0", 2024-09-18T08:13:14.292 INFO:teuthology.orchestra.run.smithi007.stdout: "last_force_op_resend_preluminous": "152", 2024-09-18T08:13:14.292 INFO:teuthology.orchestra.run.smithi007.stdout: "auid": 0, 2024-09-18T08:13:14.292 INFO:teuthology.orchestra.run.smithi007.stdout: "snap_mode": "selfmanaged", 2024-09-18T08:13:14.292 INFO:teuthology.orchestra.run.smithi007.stdout: "snap_seq": 0, 2024-09-18T08:13:14.292 INFO:teuthology.orchestra.run.smithi007.stdout: "snap_epoch": 0, 2024-09-18T08:13:14.292 INFO:teuthology.orchestra.run.smithi007.stdout: "pool_snaps": [], 2024-09-18T08:13:14.292 INFO:teuthology.orchestra.run.smithi007.stdout: "removed_snaps": "[]", 2024-09-18T08:13:14.292 INFO:teuthology.orchestra.run.smithi007.stdout: "quota_max_bytes": 0, 2024-09-18T08:13:14.292 INFO:teuthology.orchestra.run.smithi007.stdout: "quota_max_objects": 0, 2024-09-18T08:13:14.292 INFO:teuthology.orchestra.run.smithi007.stdout: "tiers": [], 2024-09-18T08:13:14.292 INFO:teuthology.orchestra.run.smithi007.stdout: "tier_of": -1, 2024-09-18T08:13:14.292 INFO:teuthology.orchestra.run.smithi007.stdout: "read_tier": -1, 2024-09-18T08:13:14.293 INFO:teuthology.orchestra.run.smithi007.stdout: "write_tier": -1, 2024-09-18T08:13:14.293 INFO:teuthology.orchestra.run.smithi007.stdout: "cache_mode": "none", 2024-09-18T08:13:14.293 INFO:teuthology.orchestra.run.smithi007.stdout: "target_max_bytes": 0, 2024-09-18T08:13:14.293 INFO:teuthology.orchestra.run.smithi007.stdout: "target_max_objects": 0, 2024-09-18T08:13:14.293 INFO:teuthology.orchestra.run.smithi007.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-09-18T08:13:14.293 INFO:teuthology.orchestra.run.smithi007.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-09-18T08:13:14.293 INFO:teuthology.orchestra.run.smithi007.stdout: "cache_target_full_ratio_micro": 800000, 2024-09-18T08:13:14.293 INFO:teuthology.orchestra.run.smithi007.stdout: "cache_min_flush_age": 0, 2024-09-18T08:13:14.293 INFO:teuthology.orchestra.run.smithi007.stdout: "cache_min_evict_age": 0, 2024-09-18T08:13:14.293 INFO:teuthology.orchestra.run.smithi007.stdout: "erasure_code_profile": "", 2024-09-18T08:13:14.293 INFO:teuthology.orchestra.run.smithi007.stdout: "hit_set_params": { 2024-09-18T08:13:14.293 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "none" 2024-09-18T08:13:14.294 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:13:14.294 INFO:teuthology.orchestra.run.smithi007.stdout: "hit_set_period": 0, 2024-09-18T08:13:14.294 INFO:teuthology.orchestra.run.smithi007.stdout: "hit_set_count": 0, 2024-09-18T08:13:14.294 INFO:teuthology.orchestra.run.smithi007.stdout: "use_gmt_hitset": true, 2024-09-18T08:13:14.294 INFO:teuthology.orchestra.run.smithi007.stdout: "min_read_recency_for_promote": 0, 2024-09-18T08:13:14.294 INFO:teuthology.orchestra.run.smithi007.stdout: "min_write_recency_for_promote": 0, 2024-09-18T08:13:14.294 INFO:teuthology.orchestra.run.smithi007.stdout: "hit_set_grade_decay_rate": 0, 2024-09-18T08:13:14.294 INFO:teuthology.orchestra.run.smithi007.stdout: "hit_set_search_last_n": 0, 2024-09-18T08:13:14.294 INFO:teuthology.orchestra.run.smithi007.stdout: "grade_table": [], 2024-09-18T08:13:14.294 INFO:teuthology.orchestra.run.smithi007.stdout: "stripe_width": 0, 2024-09-18T08:13:14.294 INFO:teuthology.orchestra.run.smithi007.stdout: "expected_num_objects": 0, 2024-09-18T08:13:14.294 INFO:teuthology.orchestra.run.smithi007.stdout: "fast_read": false, 2024-09-18T08:13:14.295 INFO:teuthology.orchestra.run.smithi007.stdout: "options": { 2024-09-18T08:13:14.295 INFO:teuthology.orchestra.run.smithi007.stdout: "pg_num_max": 32, 2024-09-18T08:13:14.295 INFO:teuthology.orchestra.run.smithi007.stdout: "pg_num_min": 1 2024-09-18T08:13:14.295 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:13:14.295 INFO:teuthology.orchestra.run.smithi007.stdout: "application_metadata": { 2024-09-18T08:13:14.295 INFO:teuthology.orchestra.run.smithi007.stdout: "mgr": {} 2024-09-18T08:13:14.295 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:13:14.295 INFO:teuthology.orchestra.run.smithi007.stdout: "read_balance": { 2024-09-18T08:13:14.295 INFO:teuthology.orchestra.run.smithi007.stdout: "score_type": "Fair distribution", 2024-09-18T08:13:14.295 INFO:teuthology.orchestra.run.smithi007.stdout: "score_acting": 1.9099999666213989, 2024-09-18T08:13:14.295 INFO:teuthology.orchestra.run.smithi007.stdout: "score_stable": 1.9099999666213989, 2024-09-18T08:13:14.295 INFO:teuthology.orchestra.run.smithi007.stdout: "optimal_score": 1, 2024-09-18T08:13:14.295 INFO:teuthology.orchestra.run.smithi007.stdout: "raw_score_acting": 1.9099999666213989, 2024-09-18T08:13:14.295 INFO:teuthology.orchestra.run.smithi007.stdout: "raw_score_stable": 1.9099999666213989, 2024-09-18T08:13:14.296 INFO:teuthology.orchestra.run.smithi007.stdout: "primary_affinity_weighted": 0.84128189086914062, 2024-09-18T08:13:14.296 INFO:teuthology.orchestra.run.smithi007.stdout: "average_primary_affinity": 0.95999997854232788, 2024-09-18T08:13:14.296 INFO:teuthology.orchestra.run.smithi007.stdout: "average_primary_affinity_weighted": 0.8399999737739563 2024-09-18T08:13:14.296 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-09-18T08:13:14.296 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:13:14.296 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:13:14.296 INFO:teuthology.orchestra.run.smithi007.stdout: "pool": 2, 2024-09-18T08:13:14.296 INFO:teuthology.orchestra.run.smithi007.stdout: "pool_name": "datapool", 2024-09-18T08:13:14.296 INFO:teuthology.orchestra.run.smithi007.stdout: "create_time": "2024-09-18T08:02:45.374099+0000", 2024-09-18T08:13:14.296 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 8193, 2024-09-18T08:13:14.296 INFO:teuthology.orchestra.run.smithi007.stdout: "flags_names": "hashpspool,selfmanaged_snaps", 2024-09-18T08:13:14.296 INFO:teuthology.orchestra.run.smithi007.stdout: "type": 1, 2024-09-18T08:13:14.297 INFO:teuthology.orchestra.run.smithi007.stdout: "size": 3, 2024-09-18T08:13:14.297 INFO:teuthology.orchestra.run.smithi007.stdout: "min_size": 2, 2024-09-18T08:13:14.297 INFO:teuthology.orchestra.run.smithi007.stdout: "crush_rule": 0, 2024-09-18T08:13:14.297 INFO:teuthology.orchestra.run.smithi007.stdout: "peering_crush_bucket_count": 0, 2024-09-18T08:13:14.297 INFO:teuthology.orchestra.run.smithi007.stdout: "peering_crush_bucket_target": 0, 2024-09-18T08:13:14.297 INFO:teuthology.orchestra.run.smithi007.stdout: "peering_crush_bucket_barrier": 0, 2024-09-18T08:13:14.297 INFO:teuthology.orchestra.run.smithi007.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-09-18T08:13:14.297 INFO:teuthology.orchestra.run.smithi007.stdout: "is_stretch_pool": false, 2024-09-18T08:13:14.297 INFO:teuthology.orchestra.run.smithi007.stdout: "object_hash": 2, 2024-09-18T08:13:14.297 INFO:teuthology.orchestra.run.smithi007.stdout: "pg_autoscale_mode": "off", 2024-09-18T08:13:14.297 INFO:teuthology.orchestra.run.smithi007.stdout: "pg_num": 3, 2024-09-18T08:13:14.297 INFO:teuthology.orchestra.run.smithi007.stdout: "pg_placement_num": 3, 2024-09-18T08:13:14.297 INFO:teuthology.orchestra.run.smithi007.stdout: "pg_placement_num_target": 3, 2024-09-18T08:13:14.297 INFO:teuthology.orchestra.run.smithi007.stdout: "pg_num_target": 3, 2024-09-18T08:13:14.298 INFO:teuthology.orchestra.run.smithi007.stdout: "pg_num_pending": 3, 2024-09-18T08:13:14.298 INFO:teuthology.orchestra.run.smithi007.stdout: "last_pg_merge_meta": { 2024-09-18T08:13:14.298 INFO:teuthology.orchestra.run.smithi007.stdout: "source_pgid": "0.0", 2024-09-18T08:13:14.298 INFO:teuthology.orchestra.run.smithi007.stdout: "ready_epoch": 0, 2024-09-18T08:13:14.298 INFO:teuthology.orchestra.run.smithi007.stdout: "last_epoch_started": 0, 2024-09-18T08:13:14.298 INFO:teuthology.orchestra.run.smithi007.stdout: "last_epoch_clean": 0, 2024-09-18T08:13:14.298 INFO:teuthology.orchestra.run.smithi007.stdout: "source_version": "0'0", 2024-09-18T08:13:14.298 INFO:teuthology.orchestra.run.smithi007.stdout: "target_version": "0'0" 2024-09-18T08:13:14.298 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:13:14.298 INFO:teuthology.orchestra.run.smithi007.stdout: "last_change": "64", 2024-09-18T08:13:14.298 INFO:teuthology.orchestra.run.smithi007.stdout: "last_force_op_resend": "0", 2024-09-18T08:13:14.298 INFO:teuthology.orchestra.run.smithi007.stdout: "last_force_op_resend_prenautilus": "0", 2024-09-18T08:13:14.298 INFO:teuthology.orchestra.run.smithi007.stdout: "last_force_op_resend_preluminous": "0", 2024-09-18T08:13:14.299 INFO:teuthology.orchestra.run.smithi007.stdout: "auid": 0, 2024-09-18T08:13:14.299 INFO:teuthology.orchestra.run.smithi007.stdout: "snap_mode": "selfmanaged", 2024-09-18T08:13:14.299 INFO:teuthology.orchestra.run.smithi007.stdout: "snap_seq": 2, 2024-09-18T08:13:14.299 INFO:teuthology.orchestra.run.smithi007.stdout: "snap_epoch": 64, 2024-09-18T08:13:14.299 INFO:teuthology.orchestra.run.smithi007.stdout: "pool_snaps": [], 2024-09-18T08:13:14.299 INFO:teuthology.orchestra.run.smithi007.stdout: "removed_snaps": "[]", 2024-09-18T08:13:14.299 INFO:teuthology.orchestra.run.smithi007.stdout: "quota_max_bytes": 0, 2024-09-18T08:13:14.299 INFO:teuthology.orchestra.run.smithi007.stdout: "quota_max_objects": 0, 2024-09-18T08:13:14.299 INFO:teuthology.orchestra.run.smithi007.stdout: "tiers": [], 2024-09-18T08:13:14.299 INFO:teuthology.orchestra.run.smithi007.stdout: "tier_of": -1, 2024-09-18T08:13:14.299 INFO:teuthology.orchestra.run.smithi007.stdout: "read_tier": -1, 2024-09-18T08:13:14.299 INFO:teuthology.orchestra.run.smithi007.stdout: "write_tier": -1, 2024-09-18T08:13:14.299 INFO:teuthology.orchestra.run.smithi007.stdout: "cache_mode": "none", 2024-09-18T08:13:14.300 INFO:teuthology.orchestra.run.smithi007.stdout: "target_max_bytes": 0, 2024-09-18T08:13:14.300 INFO:teuthology.orchestra.run.smithi007.stdout: "target_max_objects": 0, 2024-09-18T08:13:14.300 INFO:teuthology.orchestra.run.smithi007.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-09-18T08:13:14.300 INFO:teuthology.orchestra.run.smithi007.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-09-18T08:13:14.300 INFO:teuthology.orchestra.run.smithi007.stdout: "cache_target_full_ratio_micro": 800000, 2024-09-18T08:13:14.300 INFO:teuthology.orchestra.run.smithi007.stdout: "cache_min_flush_age": 0, 2024-09-18T08:13:14.300 INFO:teuthology.orchestra.run.smithi007.stdout: "cache_min_evict_age": 0, 2024-09-18T08:13:14.300 INFO:teuthology.orchestra.run.smithi007.stdout: "erasure_code_profile": "", 2024-09-18T08:13:14.300 INFO:teuthology.orchestra.run.smithi007.stdout: "hit_set_params": { 2024-09-18T08:13:14.300 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "none" 2024-09-18T08:13:14.300 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:13:14.300 INFO:teuthology.orchestra.run.smithi007.stdout: "hit_set_period": 0, 2024-09-18T08:13:14.300 INFO:teuthology.orchestra.run.smithi007.stdout: "hit_set_count": 0, 2024-09-18T08:13:14.300 INFO:teuthology.orchestra.run.smithi007.stdout: "use_gmt_hitset": true, 2024-09-18T08:13:14.300 INFO:teuthology.orchestra.run.smithi007.stdout: "min_read_recency_for_promote": 0, 2024-09-18T08:13:14.301 INFO:teuthology.orchestra.run.smithi007.stdout: "min_write_recency_for_promote": 0, 2024-09-18T08:13:14.301 INFO:teuthology.orchestra.run.smithi007.stdout: "hit_set_grade_decay_rate": 0, 2024-09-18T08:13:14.301 INFO:teuthology.orchestra.run.smithi007.stdout: "hit_set_search_last_n": 0, 2024-09-18T08:13:14.301 INFO:teuthology.orchestra.run.smithi007.stdout: "grade_table": [], 2024-09-18T08:13:14.301 INFO:teuthology.orchestra.run.smithi007.stdout: "stripe_width": 0, 2024-09-18T08:13:14.301 INFO:teuthology.orchestra.run.smithi007.stdout: "expected_num_objects": 0, 2024-09-18T08:13:14.301 INFO:teuthology.orchestra.run.smithi007.stdout: "fast_read": false, 2024-09-18T08:13:14.301 INFO:teuthology.orchestra.run.smithi007.stdout: "options": {}, 2024-09-18T08:13:14.301 INFO:teuthology.orchestra.run.smithi007.stdout: "application_metadata": { 2024-09-18T08:13:14.301 INFO:teuthology.orchestra.run.smithi007.stdout: "rbd": {} 2024-09-18T08:13:14.301 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:13:14.301 INFO:teuthology.orchestra.run.smithi007.stdout: "read_balance": { 2024-09-18T08:13:14.301 INFO:teuthology.orchestra.run.smithi007.stdout: "score_type": "Fair distribution", 2024-09-18T08:13:14.302 INFO:teuthology.orchestra.run.smithi007.stdout: "score_acting": 2.3299999237060547, 2024-09-18T08:13:14.302 INFO:teuthology.orchestra.run.smithi007.stdout: "score_stable": 2.3299999237060547, 2024-09-18T08:13:14.302 INFO:teuthology.orchestra.run.smithi007.stdout: "optimal_score": 0.86000001430511475, 2024-09-18T08:13:14.302 INFO:teuthology.orchestra.run.smithi007.stdout: "raw_score_acting": 2, 2024-09-18T08:13:14.302 INFO:teuthology.orchestra.run.smithi007.stdout: "raw_score_stable": 2, 2024-09-18T08:13:14.302 INFO:teuthology.orchestra.run.smithi007.stdout: "primary_affinity_weighted": 0.84128189086914062, 2024-09-18T08:13:14.302 INFO:teuthology.orchestra.run.smithi007.stdout: "average_primary_affinity": 0.95999997854232788, 2024-09-18T08:13:14.302 INFO:teuthology.orchestra.run.smithi007.stdout: "average_primary_affinity_weighted": 0.8399999737739563 2024-09-18T08:13:14.302 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-09-18T08:13:14.302 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:13:14.302 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:13:14.302 INFO:teuthology.orchestra.run.smithi007.stdout: "pool": 3, 2024-09-18T08:13:14.302 INFO:teuthology.orchestra.run.smithi007.stdout: "pool_name": ".rgw.root", 2024-09-18T08:13:14.303 INFO:teuthology.orchestra.run.smithi007.stdout: "create_time": "2024-09-18T08:02:46.319947+0000", 2024-09-18T08:13:14.304 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 1, 2024-09-18T08:13:14.304 INFO:teuthology.orchestra.run.smithi007.stdout: "flags_names": "hashpspool", 2024-09-18T08:13:14.304 INFO:teuthology.orchestra.run.smithi007.stdout: "type": 1, 2024-09-18T08:13:14.305 INFO:teuthology.orchestra.run.smithi007.stdout: "size": 3, 2024-09-18T08:13:14.305 INFO:teuthology.orchestra.run.smithi007.stdout: "min_size": 2, 2024-09-18T08:13:14.305 INFO:teuthology.orchestra.run.smithi007.stdout: "crush_rule": 0, 2024-09-18T08:13:14.305 INFO:teuthology.orchestra.run.smithi007.stdout: "peering_crush_bucket_count": 0, 2024-09-18T08:13:14.305 INFO:teuthology.orchestra.run.smithi007.stdout: "peering_crush_bucket_target": 0, 2024-09-18T08:13:14.305 INFO:teuthology.orchestra.run.smithi007.stdout: "peering_crush_bucket_barrier": 0, 2024-09-18T08:13:14.305 INFO:teuthology.orchestra.run.smithi007.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-09-18T08:13:14.305 INFO:teuthology.orchestra.run.smithi007.stdout: "is_stretch_pool": false, 2024-09-18T08:13:14.305 INFO:teuthology.orchestra.run.smithi007.stdout: "object_hash": 2, 2024-09-18T08:13:14.305 INFO:teuthology.orchestra.run.smithi007.stdout: "pg_autoscale_mode": "off", 2024-09-18T08:13:14.305 INFO:teuthology.orchestra.run.smithi007.stdout: "pg_num": 32, 2024-09-18T08:13:14.305 INFO:teuthology.orchestra.run.smithi007.stdout: "pg_placement_num": 32, 2024-09-18T08:13:14.305 INFO:teuthology.orchestra.run.smithi007.stdout: "pg_placement_num_target": 32, 2024-09-18T08:13:14.306 INFO:teuthology.orchestra.run.smithi007.stdout: "pg_num_target": 32, 2024-09-18T08:13:14.306 INFO:teuthology.orchestra.run.smithi007.stdout: "pg_num_pending": 32, 2024-09-18T08:13:14.306 INFO:teuthology.orchestra.run.smithi007.stdout: "last_pg_merge_meta": { 2024-09-18T08:13:14.306 INFO:teuthology.orchestra.run.smithi007.stdout: "source_pgid": "0.0", 2024-09-18T08:13:14.306 INFO:teuthology.orchestra.run.smithi007.stdout: "ready_epoch": 0, 2024-09-18T08:13:14.306 INFO:teuthology.orchestra.run.smithi007.stdout: "last_epoch_started": 0, 2024-09-18T08:13:14.306 INFO:teuthology.orchestra.run.smithi007.stdout: "last_epoch_clean": 0, 2024-09-18T08:13:14.306 INFO:teuthology.orchestra.run.smithi007.stdout: "source_version": "0'0", 2024-09-18T08:13:14.306 INFO:teuthology.orchestra.run.smithi007.stdout: "target_version": "0'0" 2024-09-18T08:13:14.306 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:13:14.306 INFO:teuthology.orchestra.run.smithi007.stdout: "last_change": "63", 2024-09-18T08:13:14.306 INFO:teuthology.orchestra.run.smithi007.stdout: "last_force_op_resend": "0", 2024-09-18T08:13:14.306 INFO:teuthology.orchestra.run.smithi007.stdout: "last_force_op_resend_prenautilus": "0", 2024-09-18T08:13:14.307 INFO:teuthology.orchestra.run.smithi007.stdout: "last_force_op_resend_preluminous": "0", 2024-09-18T08:13:14.307 INFO:teuthology.orchestra.run.smithi007.stdout: "auid": 0, 2024-09-18T08:13:14.307 INFO:teuthology.orchestra.run.smithi007.stdout: "snap_mode": "selfmanaged", 2024-09-18T08:13:14.307 INFO:teuthology.orchestra.run.smithi007.stdout: "snap_seq": 0, 2024-09-18T08:13:14.307 INFO:teuthology.orchestra.run.smithi007.stdout: "snap_epoch": 0, 2024-09-18T08:13:14.307 INFO:teuthology.orchestra.run.smithi007.stdout: "pool_snaps": [], 2024-09-18T08:13:14.307 INFO:teuthology.orchestra.run.smithi007.stdout: "removed_snaps": "[]", 2024-09-18T08:13:14.307 INFO:teuthology.orchestra.run.smithi007.stdout: "quota_max_bytes": 0, 2024-09-18T08:13:14.307 INFO:teuthology.orchestra.run.smithi007.stdout: "quota_max_objects": 0, 2024-09-18T08:13:14.307 INFO:teuthology.orchestra.run.smithi007.stdout: "tiers": [], 2024-09-18T08:13:14.307 INFO:teuthology.orchestra.run.smithi007.stdout: "tier_of": -1, 2024-09-18T08:13:14.307 INFO:teuthology.orchestra.run.smithi007.stdout: "read_tier": -1, 2024-09-18T08:13:14.307 INFO:teuthology.orchestra.run.smithi007.stdout: "write_tier": -1, 2024-09-18T08:13:14.308 INFO:teuthology.orchestra.run.smithi007.stdout: "cache_mode": "none", 2024-09-18T08:13:14.308 INFO:teuthology.orchestra.run.smithi007.stdout: "target_max_bytes": 0, 2024-09-18T08:13:14.308 INFO:teuthology.orchestra.run.smithi007.stdout: "target_max_objects": 0, 2024-09-18T08:13:14.308 INFO:teuthology.orchestra.run.smithi007.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-09-18T08:13:14.308 INFO:teuthology.orchestra.run.smithi007.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-09-18T08:13:14.308 INFO:teuthology.orchestra.run.smithi007.stdout: "cache_target_full_ratio_micro": 800000, 2024-09-18T08:13:14.308 INFO:teuthology.orchestra.run.smithi007.stdout: "cache_min_flush_age": 0, 2024-09-18T08:13:14.308 INFO:teuthology.orchestra.run.smithi007.stdout: "cache_min_evict_age": 0, 2024-09-18T08:13:14.308 INFO:teuthology.orchestra.run.smithi007.stdout: "erasure_code_profile": "", 2024-09-18T08:13:14.308 INFO:teuthology.orchestra.run.smithi007.stdout: "hit_set_params": { 2024-09-18T08:13:14.308 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "none" 2024-09-18T08:13:14.308 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:13:14.308 INFO:teuthology.orchestra.run.smithi007.stdout: "hit_set_period": 0, 2024-09-18T08:13:14.309 INFO:teuthology.orchestra.run.smithi007.stdout: "hit_set_count": 0, 2024-09-18T08:13:14.309 INFO:teuthology.orchestra.run.smithi007.stdout: "use_gmt_hitset": true, 2024-09-18T08:13:14.309 INFO:teuthology.orchestra.run.smithi007.stdout: "min_read_recency_for_promote": 0, 2024-09-18T08:13:14.309 INFO:teuthology.orchestra.run.smithi007.stdout: "min_write_recency_for_promote": 0, 2024-09-18T08:13:14.309 INFO:teuthology.orchestra.run.smithi007.stdout: "hit_set_grade_decay_rate": 0, 2024-09-18T08:13:14.309 INFO:teuthology.orchestra.run.smithi007.stdout: "hit_set_search_last_n": 0, 2024-09-18T08:13:14.309 INFO:teuthology.orchestra.run.smithi007.stdout: "grade_table": [], 2024-09-18T08:13:14.309 INFO:teuthology.orchestra.run.smithi007.stdout: "stripe_width": 0, 2024-09-18T08:13:14.309 INFO:teuthology.orchestra.run.smithi007.stdout: "expected_num_objects": 0, 2024-09-18T08:13:14.309 INFO:teuthology.orchestra.run.smithi007.stdout: "fast_read": false, 2024-09-18T08:13:14.309 INFO:teuthology.orchestra.run.smithi007.stdout: "options": {}, 2024-09-18T08:13:14.309 INFO:teuthology.orchestra.run.smithi007.stdout: "application_metadata": { 2024-09-18T08:13:14.309 INFO:teuthology.orchestra.run.smithi007.stdout: "rgw": {} 2024-09-18T08:13:14.310 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:13:14.310 INFO:teuthology.orchestra.run.smithi007.stdout: "read_balance": { 2024-09-18T08:13:14.310 INFO:teuthology.orchestra.run.smithi007.stdout: "score_type": "Fair distribution", 2024-09-18T08:13:14.310 INFO:teuthology.orchestra.run.smithi007.stdout: "score_acting": 1.5299999713897705, 2024-09-18T08:13:14.310 INFO:teuthology.orchestra.run.smithi007.stdout: "score_stable": 1.5299999713897705, 2024-09-18T08:13:14.310 INFO:teuthology.orchestra.run.smithi007.stdout: "optimal_score": 1, 2024-09-18T08:13:14.310 INFO:teuthology.orchestra.run.smithi007.stdout: "raw_score_acting": 1.5299999713897705, 2024-09-18T08:13:14.310 INFO:teuthology.orchestra.run.smithi007.stdout: "raw_score_stable": 1.5299999713897705, 2024-09-18T08:13:14.310 INFO:teuthology.orchestra.run.smithi007.stdout: "primary_affinity_weighted": 0.84128189086914062, 2024-09-18T08:13:14.310 INFO:teuthology.orchestra.run.smithi007.stdout: "average_primary_affinity": 0.95999997854232788, 2024-09-18T08:13:14.310 INFO:teuthology.orchestra.run.smithi007.stdout: "average_primary_affinity_weighted": 0.8399999737739563 2024-09-18T08:13:14.310 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-09-18T08:13:14.310 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:13:14.311 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:13:14.311 INFO:teuthology.orchestra.run.smithi007.stdout: "pool": 4, 2024-09-18T08:13:14.311 INFO:teuthology.orchestra.run.smithi007.stdout: "pool_name": "default.rgw.log", 2024-09-18T08:13:14.311 INFO:teuthology.orchestra.run.smithi007.stdout: "create_time": "2024-09-18T08:02:47.948552+0000", 2024-09-18T08:13:14.311 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 1, 2024-09-18T08:13:14.311 INFO:teuthology.orchestra.run.smithi007.stdout: "flags_names": "hashpspool", 2024-09-18T08:13:14.311 INFO:teuthology.orchestra.run.smithi007.stdout: "type": 1, 2024-09-18T08:13:14.311 INFO:teuthology.orchestra.run.smithi007.stdout: "size": 3, 2024-09-18T08:13:14.311 INFO:teuthology.orchestra.run.smithi007.stdout: "min_size": 2, 2024-09-18T08:13:14.311 INFO:teuthology.orchestra.run.smithi007.stdout: "crush_rule": 0, 2024-09-18T08:13:14.311 INFO:teuthology.orchestra.run.smithi007.stdout: "peering_crush_bucket_count": 0, 2024-09-18T08:13:14.311 INFO:teuthology.orchestra.run.smithi007.stdout: "peering_crush_bucket_target": 0, 2024-09-18T08:13:14.311 INFO:teuthology.orchestra.run.smithi007.stdout: "peering_crush_bucket_barrier": 0, 2024-09-18T08:13:14.312 INFO:teuthology.orchestra.run.smithi007.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-09-18T08:13:14.312 INFO:teuthology.orchestra.run.smithi007.stdout: "is_stretch_pool": false, 2024-09-18T08:13:14.312 INFO:teuthology.orchestra.run.smithi007.stdout: "object_hash": 2, 2024-09-18T08:13:14.312 INFO:teuthology.orchestra.run.smithi007.stdout: "pg_autoscale_mode": "off", 2024-09-18T08:13:14.312 INFO:teuthology.orchestra.run.smithi007.stdout: "pg_num": 32, 2024-09-18T08:13:14.312 INFO:teuthology.orchestra.run.smithi007.stdout: "pg_placement_num": 32, 2024-09-18T08:13:14.312 INFO:teuthology.orchestra.run.smithi007.stdout: "pg_placement_num_target": 32, 2024-09-18T08:13:14.312 INFO:teuthology.orchestra.run.smithi007.stdout: "pg_num_target": 32, 2024-09-18T08:13:14.312 INFO:teuthology.orchestra.run.smithi007.stdout: "pg_num_pending": 32, 2024-09-18T08:13:14.312 INFO:teuthology.orchestra.run.smithi007.stdout: "last_pg_merge_meta": { 2024-09-18T08:13:14.312 INFO:teuthology.orchestra.run.smithi007.stdout: "source_pgid": "0.0", 2024-09-18T08:13:14.312 INFO:teuthology.orchestra.run.smithi007.stdout: "ready_epoch": 0, 2024-09-18T08:13:14.312 INFO:teuthology.orchestra.run.smithi007.stdout: "last_epoch_started": 0, 2024-09-18T08:13:14.313 INFO:teuthology.orchestra.run.smithi007.stdout: "last_epoch_clean": 0, 2024-09-18T08:13:14.313 INFO:teuthology.orchestra.run.smithi007.stdout: "source_version": "0'0", 2024-09-18T08:13:14.313 INFO:teuthology.orchestra.run.smithi007.stdout: "target_version": "0'0" 2024-09-18T08:13:14.313 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:13:14.313 INFO:teuthology.orchestra.run.smithi007.stdout: "last_change": "65", 2024-09-18T08:13:14.313 INFO:teuthology.orchestra.run.smithi007.stdout: "last_force_op_resend": "0", 2024-09-18T08:13:14.313 INFO:teuthology.orchestra.run.smithi007.stdout: "last_force_op_resend_prenautilus": "0", 2024-09-18T08:13:14.313 INFO:teuthology.orchestra.run.smithi007.stdout: "last_force_op_resend_preluminous": "0", 2024-09-18T08:13:14.313 INFO:teuthology.orchestra.run.smithi007.stdout: "auid": 0, 2024-09-18T08:13:14.313 INFO:teuthology.orchestra.run.smithi007.stdout: "snap_mode": "selfmanaged", 2024-09-18T08:13:14.313 INFO:teuthology.orchestra.run.smithi007.stdout: "snap_seq": 0, 2024-09-18T08:13:14.313 INFO:teuthology.orchestra.run.smithi007.stdout: "snap_epoch": 0, 2024-09-18T08:13:14.313 INFO:teuthology.orchestra.run.smithi007.stdout: "pool_snaps": [], 2024-09-18T08:13:14.314 INFO:teuthology.orchestra.run.smithi007.stdout: "removed_snaps": "[]", 2024-09-18T08:13:14.314 INFO:teuthology.orchestra.run.smithi007.stdout: "quota_max_bytes": 0, 2024-09-18T08:13:14.314 INFO:teuthology.orchestra.run.smithi007.stdout: "quota_max_objects": 0, 2024-09-18T08:13:14.314 INFO:teuthology.orchestra.run.smithi007.stdout: "tiers": [], 2024-09-18T08:13:14.314 INFO:teuthology.orchestra.run.smithi007.stdout: "tier_of": -1, 2024-09-18T08:13:14.314 INFO:teuthology.orchestra.run.smithi007.stdout: "read_tier": -1, 2024-09-18T08:13:14.314 INFO:teuthology.orchestra.run.smithi007.stdout: "write_tier": -1, 2024-09-18T08:13:14.314 INFO:teuthology.orchestra.run.smithi007.stdout: "cache_mode": "none", 2024-09-18T08:13:14.314 INFO:teuthology.orchestra.run.smithi007.stdout: "target_max_bytes": 0, 2024-09-18T08:13:14.314 INFO:teuthology.orchestra.run.smithi007.stdout: "target_max_objects": 0, 2024-09-18T08:13:14.314 INFO:teuthology.orchestra.run.smithi007.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-09-18T08:13:14.314 INFO:teuthology.orchestra.run.smithi007.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-09-18T08:13:14.314 INFO:teuthology.orchestra.run.smithi007.stdout: "cache_target_full_ratio_micro": 800000, 2024-09-18T08:13:14.315 INFO:teuthology.orchestra.run.smithi007.stdout: "cache_min_flush_age": 0, 2024-09-18T08:13:14.315 INFO:teuthology.orchestra.run.smithi007.stdout: "cache_min_evict_age": 0, 2024-09-18T08:13:14.315 INFO:teuthology.orchestra.run.smithi007.stdout: "erasure_code_profile": "", 2024-09-18T08:13:14.315 INFO:teuthology.orchestra.run.smithi007.stdout: "hit_set_params": { 2024-09-18T08:13:14.315 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "none" 2024-09-18T08:13:14.315 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:13:14.315 INFO:teuthology.orchestra.run.smithi007.stdout: "hit_set_period": 0, 2024-09-18T08:13:14.315 INFO:teuthology.orchestra.run.smithi007.stdout: "hit_set_count": 0, 2024-09-18T08:13:14.315 INFO:teuthology.orchestra.run.smithi007.stdout: "use_gmt_hitset": true, 2024-09-18T08:13:14.315 INFO:teuthology.orchestra.run.smithi007.stdout: "min_read_recency_for_promote": 0, 2024-09-18T08:13:14.315 INFO:teuthology.orchestra.run.smithi007.stdout: "min_write_recency_for_promote": 0, 2024-09-18T08:13:14.315 INFO:teuthology.orchestra.run.smithi007.stdout: "hit_set_grade_decay_rate": 0, 2024-09-18T08:13:14.315 INFO:teuthology.orchestra.run.smithi007.stdout: "hit_set_search_last_n": 0, 2024-09-18T08:13:14.316 INFO:teuthology.orchestra.run.smithi007.stdout: "grade_table": [], 2024-09-18T08:13:14.316 INFO:teuthology.orchestra.run.smithi007.stdout: "stripe_width": 0, 2024-09-18T08:13:14.316 INFO:teuthology.orchestra.run.smithi007.stdout: "expected_num_objects": 0, 2024-09-18T08:13:14.316 INFO:teuthology.orchestra.run.smithi007.stdout: "fast_read": false, 2024-09-18T08:13:14.316 INFO:teuthology.orchestra.run.smithi007.stdout: "options": {}, 2024-09-18T08:13:14.316 INFO:teuthology.orchestra.run.smithi007.stdout: "application_metadata": { 2024-09-18T08:13:14.316 INFO:teuthology.orchestra.run.smithi007.stdout: "rgw": {} 2024-09-18T08:13:14.316 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:13:14.316 INFO:teuthology.orchestra.run.smithi007.stdout: "read_balance": { 2024-09-18T08:13:14.316 INFO:teuthology.orchestra.run.smithi007.stdout: "score_type": "Fair distribution", 2024-09-18T08:13:14.316 INFO:teuthology.orchestra.run.smithi007.stdout: "score_acting": 9.6099996566772461, 2024-09-18T08:13:14.316 INFO:teuthology.orchestra.run.smithi007.stdout: "score_stable": 2.190000057220459, 2024-09-18T08:13:14.316 INFO:teuthology.orchestra.run.smithi007.stdout: "optimal_score": 1, 2024-09-18T08:13:14.317 INFO:teuthology.orchestra.run.smithi007.stdout: "raw_score_acting": 9.6099996566772461, 2024-09-18T08:13:14.317 INFO:teuthology.orchestra.run.smithi007.stdout: "raw_score_stable": 2.190000057220459, 2024-09-18T08:13:14.317 INFO:teuthology.orchestra.run.smithi007.stdout: "primary_affinity_weighted": 0.84128189086914062, 2024-09-18T08:13:14.317 INFO:teuthology.orchestra.run.smithi007.stdout: "average_primary_affinity": 0.95999997854232788, 2024-09-18T08:13:14.317 INFO:teuthology.orchestra.run.smithi007.stdout: "average_primary_affinity_weighted": 0.8399999737739563, 2024-09-18T08:13:14.317 INFO:teuthology.orchestra.run.smithi007.stdout: "error_message": "pool 4 has acting primaries on OSD(s) with primary affinity 0, read balance score is not accurate" 2024-09-18T08:13:14.318 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-09-18T08:13:14.318 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:13:14.319 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:13:14.319 INFO:teuthology.orchestra.run.smithi007.stdout: "pool": 5, 2024-09-18T08:13:14.319 INFO:teuthology.orchestra.run.smithi007.stdout: "pool_name": "default.rgw.control", 2024-09-18T08:13:14.319 INFO:teuthology.orchestra.run.smithi007.stdout: "create_time": "2024-09-18T08:02:49.904302+0000", 2024-09-18T08:13:14.319 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 1, 2024-09-18T08:13:14.319 INFO:teuthology.orchestra.run.smithi007.stdout: "flags_names": "hashpspool", 2024-09-18T08:13:14.319 INFO:teuthology.orchestra.run.smithi007.stdout: "type": 1, 2024-09-18T08:13:14.319 INFO:teuthology.orchestra.run.smithi007.stdout: "size": 3, 2024-09-18T08:13:14.319 INFO:teuthology.orchestra.run.smithi007.stdout: "min_size": 2, 2024-09-18T08:13:14.319 INFO:teuthology.orchestra.run.smithi007.stdout: "crush_rule": 0, 2024-09-18T08:13:14.319 INFO:teuthology.orchestra.run.smithi007.stdout: "peering_crush_bucket_count": 0, 2024-09-18T08:13:14.319 INFO:teuthology.orchestra.run.smithi007.stdout: "peering_crush_bucket_target": 0, 2024-09-18T08:13:14.319 INFO:teuthology.orchestra.run.smithi007.stdout: "peering_crush_bucket_barrier": 0, 2024-09-18T08:13:14.320 INFO:teuthology.orchestra.run.smithi007.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-09-18T08:13:14.321 INFO:teuthology.orchestra.run.smithi007.stdout: "is_stretch_pool": false, 2024-09-18T08:13:14.321 INFO:teuthology.orchestra.run.smithi007.stdout: "object_hash": 2, 2024-09-18T08:13:14.321 INFO:teuthology.orchestra.run.smithi007.stdout: "pg_autoscale_mode": "off", 2024-09-18T08:13:14.321 INFO:teuthology.orchestra.run.smithi007.stdout: "pg_num": 32, 2024-09-18T08:13:14.321 INFO:teuthology.orchestra.run.smithi007.stdout: "pg_placement_num": 32, 2024-09-18T08:13:14.321 INFO:teuthology.orchestra.run.smithi007.stdout: "pg_placement_num_target": 32, 2024-09-18T08:13:14.321 INFO:teuthology.orchestra.run.smithi007.stdout: "pg_num_target": 32, 2024-09-18T08:13:14.321 INFO:teuthology.orchestra.run.smithi007.stdout: "pg_num_pending": 32, 2024-09-18T08:13:14.321 INFO:teuthology.orchestra.run.smithi007.stdout: "last_pg_merge_meta": { 2024-09-18T08:13:14.321 INFO:teuthology.orchestra.run.smithi007.stdout: "source_pgid": "0.0", 2024-09-18T08:13:14.321 INFO:teuthology.orchestra.run.smithi007.stdout: "ready_epoch": 0, 2024-09-18T08:13:14.322 INFO:teuthology.orchestra.run.smithi007.stdout: "last_epoch_started": 0, 2024-09-18T08:13:14.322 INFO:teuthology.orchestra.run.smithi007.stdout: "last_epoch_clean": 0, 2024-09-18T08:13:14.322 INFO:teuthology.orchestra.run.smithi007.stdout: "source_version": "0'0", 2024-09-18T08:13:14.322 INFO:teuthology.orchestra.run.smithi007.stdout: "target_version": "0'0" 2024-09-18T08:13:14.322 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:13:14.322 INFO:teuthology.orchestra.run.smithi007.stdout: "last_change": "67", 2024-09-18T08:13:14.322 INFO:teuthology.orchestra.run.smithi007.stdout: "last_force_op_resend": "0", 2024-09-18T08:13:14.322 INFO:teuthology.orchestra.run.smithi007.stdout: "last_force_op_resend_prenautilus": "0", 2024-09-18T08:13:14.322 INFO:teuthology.orchestra.run.smithi007.stdout: "last_force_op_resend_preluminous": "0", 2024-09-18T08:13:14.322 INFO:teuthology.orchestra.run.smithi007.stdout: "auid": 0, 2024-09-18T08:13:14.322 INFO:teuthology.orchestra.run.smithi007.stdout: "snap_mode": "selfmanaged", 2024-09-18T08:13:14.322 INFO:teuthology.orchestra.run.smithi007.stdout: "snap_seq": 0, 2024-09-18T08:13:14.323 INFO:teuthology.orchestra.run.smithi007.stdout: "snap_epoch": 0, 2024-09-18T08:13:14.323 INFO:teuthology.orchestra.run.smithi007.stdout: "pool_snaps": [], 2024-09-18T08:13:14.323 INFO:teuthology.orchestra.run.smithi007.stdout: "removed_snaps": "[]", 2024-09-18T08:13:14.323 INFO:teuthology.orchestra.run.smithi007.stdout: "quota_max_bytes": 0, 2024-09-18T08:13:14.323 INFO:teuthology.orchestra.run.smithi007.stdout: "quota_max_objects": 0, 2024-09-18T08:13:14.323 INFO:teuthology.orchestra.run.smithi007.stdout: "tiers": [], 2024-09-18T08:13:14.323 INFO:teuthology.orchestra.run.smithi007.stdout: "tier_of": -1, 2024-09-18T08:13:14.323 INFO:teuthology.orchestra.run.smithi007.stdout: "read_tier": -1, 2024-09-18T08:13:14.323 INFO:teuthology.orchestra.run.smithi007.stdout: "write_tier": -1, 2024-09-18T08:13:14.323 INFO:teuthology.orchestra.run.smithi007.stdout: "cache_mode": "none", 2024-09-18T08:13:14.323 INFO:teuthology.orchestra.run.smithi007.stdout: "target_max_bytes": 0, 2024-09-18T08:13:14.323 INFO:teuthology.orchestra.run.smithi007.stdout: "target_max_objects": 0, 2024-09-18T08:13:14.323 INFO:teuthology.orchestra.run.smithi007.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-09-18T08:13:14.324 INFO:teuthology.orchestra.run.smithi007.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-09-18T08:13:14.324 INFO:teuthology.orchestra.run.smithi007.stdout: "cache_target_full_ratio_micro": 800000, 2024-09-18T08:13:14.324 INFO:teuthology.orchestra.run.smithi007.stdout: "cache_min_flush_age": 0, 2024-09-18T08:13:14.324 INFO:teuthology.orchestra.run.smithi007.stdout: "cache_min_evict_age": 0, 2024-09-18T08:13:14.324 INFO:teuthology.orchestra.run.smithi007.stdout: "erasure_code_profile": "", 2024-09-18T08:13:14.324 INFO:teuthology.orchestra.run.smithi007.stdout: "hit_set_params": { 2024-09-18T08:13:14.324 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "none" 2024-09-18T08:13:14.324 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:13:14.324 INFO:teuthology.orchestra.run.smithi007.stdout: "hit_set_period": 0, 2024-09-18T08:13:14.324 INFO:teuthology.orchestra.run.smithi007.stdout: "hit_set_count": 0, 2024-09-18T08:13:14.324 INFO:teuthology.orchestra.run.smithi007.stdout: "use_gmt_hitset": true, 2024-09-18T08:13:14.324 INFO:teuthology.orchestra.run.smithi007.stdout: "min_read_recency_for_promote": 0, 2024-09-18T08:13:14.324 INFO:teuthology.orchestra.run.smithi007.stdout: "min_write_recency_for_promote": 0, 2024-09-18T08:13:14.325 INFO:teuthology.orchestra.run.smithi007.stdout: "hit_set_grade_decay_rate": 0, 2024-09-18T08:13:14.325 INFO:teuthology.orchestra.run.smithi007.stdout: "hit_set_search_last_n": 0, 2024-09-18T08:13:14.325 INFO:teuthology.orchestra.run.smithi007.stdout: "grade_table": [], 2024-09-18T08:13:14.325 INFO:teuthology.orchestra.run.smithi007.stdout: "stripe_width": 0, 2024-09-18T08:13:14.325 INFO:teuthology.orchestra.run.smithi007.stdout: "expected_num_objects": 0, 2024-09-18T08:13:14.325 INFO:teuthology.orchestra.run.smithi007.stdout: "fast_read": false, 2024-09-18T08:13:14.325 INFO:teuthology.orchestra.run.smithi007.stdout: "options": {}, 2024-09-18T08:13:14.325 INFO:teuthology.orchestra.run.smithi007.stdout: "application_metadata": { 2024-09-18T08:13:14.325 INFO:teuthology.orchestra.run.smithi007.stdout: "rgw": {} 2024-09-18T08:13:14.325 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:13:14.326 INFO:teuthology.orchestra.run.smithi007.stdout: "read_balance": { 2024-09-18T08:13:14.326 INFO:teuthology.orchestra.run.smithi007.stdout: "score_type": "Fair distribution", 2024-09-18T08:13:14.326 INFO:teuthology.orchestra.run.smithi007.stdout: "score_acting": 1.75, 2024-09-18T08:13:14.326 INFO:teuthology.orchestra.run.smithi007.stdout: "score_stable": 1.75, 2024-09-18T08:13:14.326 INFO:teuthology.orchestra.run.smithi007.stdout: "optimal_score": 1, 2024-09-18T08:13:14.326 INFO:teuthology.orchestra.run.smithi007.stdout: "raw_score_acting": 1.75, 2024-09-18T08:13:14.326 INFO:teuthology.orchestra.run.smithi007.stdout: "raw_score_stable": 1.75, 2024-09-18T08:13:14.326 INFO:teuthology.orchestra.run.smithi007.stdout: "primary_affinity_weighted": 0.84128189086914062, 2024-09-18T08:13:14.326 INFO:teuthology.orchestra.run.smithi007.stdout: "average_primary_affinity": 0.95999997854232788, 2024-09-18T08:13:14.327 INFO:teuthology.orchestra.run.smithi007.stdout: "average_primary_affinity_weighted": 0.8399999737739563 2024-09-18T08:13:14.327 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-09-18T08:13:14.327 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:13:14.327 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:13:14.327 INFO:teuthology.orchestra.run.smithi007.stdout: "pool": 6, 2024-09-18T08:13:14.327 INFO:teuthology.orchestra.run.smithi007.stdout: "pool_name": "default.rgw.meta", 2024-09-18T08:13:14.327 INFO:teuthology.orchestra.run.smithi007.stdout: "create_time": "2024-09-18T08:02:51.971465+0000", 2024-09-18T08:13:14.327 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 1, 2024-09-18T08:13:14.327 INFO:teuthology.orchestra.run.smithi007.stdout: "flags_names": "hashpspool", 2024-09-18T08:13:14.327 INFO:teuthology.orchestra.run.smithi007.stdout: "type": 1, 2024-09-18T08:13:14.327 INFO:teuthology.orchestra.run.smithi007.stdout: "size": 3, 2024-09-18T08:13:14.328 INFO:teuthology.orchestra.run.smithi007.stdout: "min_size": 2, 2024-09-18T08:13:14.328 INFO:teuthology.orchestra.run.smithi007.stdout: "crush_rule": 0, 2024-09-18T08:13:14.328 INFO:teuthology.orchestra.run.smithi007.stdout: "peering_crush_bucket_count": 0, 2024-09-18T08:13:14.328 INFO:teuthology.orchestra.run.smithi007.stdout: "peering_crush_bucket_target": 0, 2024-09-18T08:13:14.328 INFO:teuthology.orchestra.run.smithi007.stdout: "peering_crush_bucket_barrier": 0, 2024-09-18T08:13:14.328 INFO:teuthology.orchestra.run.smithi007.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-09-18T08:13:14.328 INFO:teuthology.orchestra.run.smithi007.stdout: "is_stretch_pool": false, 2024-09-18T08:13:14.328 INFO:teuthology.orchestra.run.smithi007.stdout: "object_hash": 2, 2024-09-18T08:13:14.328 INFO:teuthology.orchestra.run.smithi007.stdout: "pg_autoscale_mode": "off", 2024-09-18T08:13:14.328 INFO:teuthology.orchestra.run.smithi007.stdout: "pg_num": 32, 2024-09-18T08:13:14.328 INFO:teuthology.orchestra.run.smithi007.stdout: "pg_placement_num": 32, 2024-09-18T08:13:14.328 INFO:teuthology.orchestra.run.smithi007.stdout: "pg_placement_num_target": 32, 2024-09-18T08:13:14.329 INFO:teuthology.orchestra.run.smithi007.stdout: "pg_num_target": 32, 2024-09-18T08:13:14.329 INFO:teuthology.orchestra.run.smithi007.stdout: "pg_num_pending": 32, 2024-09-18T08:13:14.329 INFO:teuthology.orchestra.run.smithi007.stdout: "last_pg_merge_meta": { 2024-09-18T08:13:14.329 INFO:teuthology.orchestra.run.smithi007.stdout: "source_pgid": "0.0", 2024-09-18T08:13:14.329 INFO:teuthology.orchestra.run.smithi007.stdout: "ready_epoch": 0, 2024-09-18T08:13:14.329 INFO:teuthology.orchestra.run.smithi007.stdout: "last_epoch_started": 0, 2024-09-18T08:13:14.329 INFO:teuthology.orchestra.run.smithi007.stdout: "last_epoch_clean": 0, 2024-09-18T08:13:14.329 INFO:teuthology.orchestra.run.smithi007.stdout: "source_version": "0'0", 2024-09-18T08:13:14.329 INFO:teuthology.orchestra.run.smithi007.stdout: "target_version": "0'0" 2024-09-18T08:13:14.329 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:13:14.329 INFO:teuthology.orchestra.run.smithi007.stdout: "last_change": "70", 2024-09-18T08:13:14.329 INFO:teuthology.orchestra.run.smithi007.stdout: "last_force_op_resend": "0", 2024-09-18T08:13:14.329 INFO:teuthology.orchestra.run.smithi007.stdout: "last_force_op_resend_prenautilus": "0", 2024-09-18T08:13:14.330 INFO:teuthology.orchestra.run.smithi007.stdout: "last_force_op_resend_preluminous": "0", 2024-09-18T08:13:14.330 INFO:teuthology.orchestra.run.smithi007.stdout: "auid": 0, 2024-09-18T08:13:14.330 INFO:teuthology.orchestra.run.smithi007.stdout: "snap_mode": "selfmanaged", 2024-09-18T08:13:14.330 INFO:teuthology.orchestra.run.smithi007.stdout: "snap_seq": 0, 2024-09-18T08:13:14.330 INFO:teuthology.orchestra.run.smithi007.stdout: "snap_epoch": 0, 2024-09-18T08:13:14.330 INFO:teuthology.orchestra.run.smithi007.stdout: "pool_snaps": [], 2024-09-18T08:13:14.330 INFO:teuthology.orchestra.run.smithi007.stdout: "removed_snaps": "[]", 2024-09-18T08:13:14.330 INFO:teuthology.orchestra.run.smithi007.stdout: "quota_max_bytes": 0, 2024-09-18T08:13:14.330 INFO:teuthology.orchestra.run.smithi007.stdout: "quota_max_objects": 0, 2024-09-18T08:13:14.330 INFO:teuthology.orchestra.run.smithi007.stdout: "tiers": [], 2024-09-18T08:13:14.330 INFO:teuthology.orchestra.run.smithi007.stdout: "tier_of": -1, 2024-09-18T08:13:14.330 INFO:teuthology.orchestra.run.smithi007.stdout: "read_tier": -1, 2024-09-18T08:13:14.330 INFO:teuthology.orchestra.run.smithi007.stdout: "write_tier": -1, 2024-09-18T08:13:14.331 INFO:teuthology.orchestra.run.smithi007.stdout: "cache_mode": "none", 2024-09-18T08:13:14.331 INFO:teuthology.orchestra.run.smithi007.stdout: "target_max_bytes": 0, 2024-09-18T08:13:14.331 INFO:teuthology.orchestra.run.smithi007.stdout: "target_max_objects": 0, 2024-09-18T08:13:14.331 INFO:teuthology.orchestra.run.smithi007.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-09-18T08:13:14.331 INFO:teuthology.orchestra.run.smithi007.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-09-18T08:13:14.331 INFO:teuthology.orchestra.run.smithi007.stdout: "cache_target_full_ratio_micro": 800000, 2024-09-18T08:13:14.331 INFO:teuthology.orchestra.run.smithi007.stdout: "cache_min_flush_age": 0, 2024-09-18T08:13:14.331 INFO:teuthology.orchestra.run.smithi007.stdout: "cache_min_evict_age": 0, 2024-09-18T08:13:14.331 INFO:teuthology.orchestra.run.smithi007.stdout: "erasure_code_profile": "", 2024-09-18T08:13:14.331 INFO:teuthology.orchestra.run.smithi007.stdout: "hit_set_params": { 2024-09-18T08:13:14.331 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "none" 2024-09-18T08:13:14.331 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:13:14.331 INFO:teuthology.orchestra.run.smithi007.stdout: "hit_set_period": 0, 2024-09-18T08:13:14.332 INFO:teuthology.orchestra.run.smithi007.stdout: "hit_set_count": 0, 2024-09-18T08:13:14.332 INFO:teuthology.orchestra.run.smithi007.stdout: "use_gmt_hitset": true, 2024-09-18T08:13:14.332 INFO:teuthology.orchestra.run.smithi007.stdout: "min_read_recency_for_promote": 0, 2024-09-18T08:13:14.332 INFO:teuthology.orchestra.run.smithi007.stdout: "min_write_recency_for_promote": 0, 2024-09-18T08:13:14.332 INFO:teuthology.orchestra.run.smithi007.stdout: "hit_set_grade_decay_rate": 0, 2024-09-18T08:13:14.332 INFO:teuthology.orchestra.run.smithi007.stdout: "hit_set_search_last_n": 0, 2024-09-18T08:13:14.332 INFO:teuthology.orchestra.run.smithi007.stdout: "grade_table": [], 2024-09-18T08:13:14.332 INFO:teuthology.orchestra.run.smithi007.stdout: "stripe_width": 0, 2024-09-18T08:13:14.332 INFO:teuthology.orchestra.run.smithi007.stdout: "expected_num_objects": 0, 2024-09-18T08:13:14.332 INFO:teuthology.orchestra.run.smithi007.stdout: "fast_read": false, 2024-09-18T08:13:14.332 INFO:teuthology.orchestra.run.smithi007.stdout: "options": { 2024-09-18T08:13:14.332 INFO:teuthology.orchestra.run.smithi007.stdout: "pg_autoscale_bias": 4 2024-09-18T08:13:14.333 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:13:14.333 INFO:teuthology.orchestra.run.smithi007.stdout: "application_metadata": { 2024-09-18T08:13:14.333 INFO:teuthology.orchestra.run.smithi007.stdout: "rgw": {} 2024-09-18T08:13:14.333 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:13:14.333 INFO:teuthology.orchestra.run.smithi007.stdout: "read_balance": { 2024-09-18T08:13:14.333 INFO:teuthology.orchestra.run.smithi007.stdout: "score_type": "Fair distribution", 2024-09-18T08:13:14.333 INFO:teuthology.orchestra.run.smithi007.stdout: "score_acting": 1.5299999713897705, 2024-09-18T08:13:14.333 INFO:teuthology.orchestra.run.smithi007.stdout: "score_stable": 1.5299999713897705, 2024-09-18T08:13:14.333 INFO:teuthology.orchestra.run.smithi007.stdout: "optimal_score": 1, 2024-09-18T08:13:14.333 INFO:teuthology.orchestra.run.smithi007.stdout: "raw_score_acting": 1.5299999713897705, 2024-09-18T08:13:14.333 INFO:teuthology.orchestra.run.smithi007.stdout: "raw_score_stable": 1.5299999713897705, 2024-09-18T08:13:14.333 INFO:teuthology.orchestra.run.smithi007.stdout: "primary_affinity_weighted": 0.84128189086914062, 2024-09-18T08:13:14.334 INFO:teuthology.orchestra.run.smithi007.stdout: "average_primary_affinity": 0.95999997854232788, 2024-09-18T08:13:14.334 INFO:teuthology.orchestra.run.smithi007.stdout: "average_primary_affinity_weighted": 0.8399999737739563 2024-09-18T08:13:14.334 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-09-18T08:13:14.334 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:13:14.334 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:13:14.334 INFO:teuthology.orchestra.run.smithi007.stdout: "pool": 9, 2024-09-18T08:13:14.334 INFO:teuthology.orchestra.run.smithi007.stdout: "pool_name": "unique_pool_2", 2024-09-18T08:13:14.334 INFO:teuthology.orchestra.run.smithi007.stdout: "create_time": "2024-09-18T08:11:08.258268+0000", 2024-09-18T08:13:14.334 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 1, 2024-09-18T08:13:14.334 INFO:teuthology.orchestra.run.smithi007.stdout: "flags_names": "hashpspool", 2024-09-18T08:13:14.334 INFO:teuthology.orchestra.run.smithi007.stdout: "type": 1, 2024-09-18T08:13:14.334 INFO:teuthology.orchestra.run.smithi007.stdout: "size": 3, 2024-09-18T08:13:14.334 INFO:teuthology.orchestra.run.smithi007.stdout: "min_size": 2, 2024-09-18T08:13:14.335 INFO:teuthology.orchestra.run.smithi007.stdout: "crush_rule": 0, 2024-09-18T08:13:14.335 INFO:teuthology.orchestra.run.smithi007.stdout: "peering_crush_bucket_count": 0, 2024-09-18T08:13:14.335 INFO:teuthology.orchestra.run.smithi007.stdout: "peering_crush_bucket_target": 0, 2024-09-18T08:13:14.335 INFO:teuthology.orchestra.run.smithi007.stdout: "peering_crush_bucket_barrier": 0, 2024-09-18T08:13:14.335 INFO:teuthology.orchestra.run.smithi007.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-09-18T08:13:14.335 INFO:teuthology.orchestra.run.smithi007.stdout: "is_stretch_pool": false, 2024-09-18T08:13:14.335 INFO:teuthology.orchestra.run.smithi007.stdout: "object_hash": 2, 2024-09-18T08:13:14.335 INFO:teuthology.orchestra.run.smithi007.stdout: "pg_autoscale_mode": "off", 2024-09-18T08:13:14.335 INFO:teuthology.orchestra.run.smithi007.stdout: "pg_num": 16, 2024-09-18T08:13:14.335 INFO:teuthology.orchestra.run.smithi007.stdout: "pg_placement_num": 16, 2024-09-18T08:13:14.335 INFO:teuthology.orchestra.run.smithi007.stdout: "pg_placement_num_target": 16, 2024-09-18T08:13:14.335 INFO:teuthology.orchestra.run.smithi007.stdout: "pg_num_target": 16, 2024-09-18T08:13:14.335 INFO:teuthology.orchestra.run.smithi007.stdout: "pg_num_pending": 16, 2024-09-18T08:13:14.336 INFO:teuthology.orchestra.run.smithi007.stdout: "last_pg_merge_meta": { 2024-09-18T08:13:14.336 INFO:teuthology.orchestra.run.smithi007.stdout: "source_pgid": "0.0", 2024-09-18T08:13:14.336 INFO:teuthology.orchestra.run.smithi007.stdout: "ready_epoch": 0, 2024-09-18T08:13:14.336 INFO:teuthology.orchestra.run.smithi007.stdout: "last_epoch_started": 0, 2024-09-18T08:13:14.336 INFO:teuthology.orchestra.run.smithi007.stdout: "last_epoch_clean": 0, 2024-09-18T08:13:14.336 INFO:teuthology.orchestra.run.smithi007.stdout: "source_version": "0'0", 2024-09-18T08:13:14.336 INFO:teuthology.orchestra.run.smithi007.stdout: "target_version": "0'0" 2024-09-18T08:13:14.336 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:13:14.336 INFO:teuthology.orchestra.run.smithi007.stdout: "last_change": "236", 2024-09-18T08:13:14.336 INFO:teuthology.orchestra.run.smithi007.stdout: "last_force_op_resend": "0", 2024-09-18T08:13:14.336 INFO:teuthology.orchestra.run.smithi007.stdout: "last_force_op_resend_prenautilus": "0", 2024-09-18T08:13:14.336 INFO:teuthology.orchestra.run.smithi007.stdout: "last_force_op_resend_preluminous": "0", 2024-09-18T08:13:14.336 INFO:teuthology.orchestra.run.smithi007.stdout: "auid": 0, 2024-09-18T08:13:14.337 INFO:teuthology.orchestra.run.smithi007.stdout: "snap_mode": "selfmanaged", 2024-09-18T08:13:14.337 INFO:teuthology.orchestra.run.smithi007.stdout: "snap_seq": 0, 2024-09-18T08:13:14.337 INFO:teuthology.orchestra.run.smithi007.stdout: "snap_epoch": 0, 2024-09-18T08:13:14.337 INFO:teuthology.orchestra.run.smithi007.stdout: "pool_snaps": [], 2024-09-18T08:13:14.337 INFO:teuthology.orchestra.run.smithi007.stdout: "removed_snaps": "[]", 2024-09-18T08:13:14.337 INFO:teuthology.orchestra.run.smithi007.stdout: "quota_max_bytes": 0, 2024-09-18T08:13:14.337 INFO:teuthology.orchestra.run.smithi007.stdout: "quota_max_objects": 0, 2024-09-18T08:13:14.337 INFO:teuthology.orchestra.run.smithi007.stdout: "tiers": [], 2024-09-18T08:13:14.337 INFO:teuthology.orchestra.run.smithi007.stdout: "tier_of": -1, 2024-09-18T08:13:14.337 INFO:teuthology.orchestra.run.smithi007.stdout: "read_tier": -1, 2024-09-18T08:13:14.337 INFO:teuthology.orchestra.run.smithi007.stdout: "write_tier": -1, 2024-09-18T08:13:14.337 INFO:teuthology.orchestra.run.smithi007.stdout: "cache_mode": "none", 2024-09-18T08:13:14.337 INFO:teuthology.orchestra.run.smithi007.stdout: "target_max_bytes": 0, 2024-09-18T08:13:14.338 INFO:teuthology.orchestra.run.smithi007.stdout: "target_max_objects": 0, 2024-09-18T08:13:14.338 INFO:teuthology.orchestra.run.smithi007.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-09-18T08:13:14.338 INFO:teuthology.orchestra.run.smithi007.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-09-18T08:13:14.338 INFO:teuthology.orchestra.run.smithi007.stdout: "cache_target_full_ratio_micro": 800000, 2024-09-18T08:13:14.338 INFO:teuthology.orchestra.run.smithi007.stdout: "cache_min_flush_age": 0, 2024-09-18T08:13:14.338 INFO:teuthology.orchestra.run.smithi007.stdout: "cache_min_evict_age": 0, 2024-09-18T08:13:14.338 INFO:teuthology.orchestra.run.smithi007.stdout: "erasure_code_profile": "", 2024-09-18T08:13:14.338 INFO:teuthology.orchestra.run.smithi007.stdout: "hit_set_params": { 2024-09-18T08:13:14.338 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "none" 2024-09-18T08:13:14.338 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:13:14.338 INFO:teuthology.orchestra.run.smithi007.stdout: "hit_set_period": 0, 2024-09-18T08:13:14.338 INFO:teuthology.orchestra.run.smithi007.stdout: "hit_set_count": 0, 2024-09-18T08:13:14.338 INFO:teuthology.orchestra.run.smithi007.stdout: "use_gmt_hitset": true, 2024-09-18T08:13:14.338 INFO:teuthology.orchestra.run.smithi007.stdout: "min_read_recency_for_promote": 0, 2024-09-18T08:13:14.339 INFO:teuthology.orchestra.run.smithi007.stdout: "min_write_recency_for_promote": 0, 2024-09-18T08:13:14.339 INFO:teuthology.orchestra.run.smithi007.stdout: "hit_set_grade_decay_rate": 0, 2024-09-18T08:13:14.339 INFO:teuthology.orchestra.run.smithi007.stdout: "hit_set_search_last_n": 0, 2024-09-18T08:13:14.339 INFO:teuthology.orchestra.run.smithi007.stdout: "grade_table": [], 2024-09-18T08:13:14.339 INFO:teuthology.orchestra.run.smithi007.stdout: "stripe_width": 0, 2024-09-18T08:13:14.339 INFO:teuthology.orchestra.run.smithi007.stdout: "expected_num_objects": 0, 2024-09-18T08:13:14.339 INFO:teuthology.orchestra.run.smithi007.stdout: "fast_read": false, 2024-09-18T08:13:14.339 INFO:teuthology.orchestra.run.smithi007.stdout: "options": {}, 2024-09-18T08:13:14.339 INFO:teuthology.orchestra.run.smithi007.stdout: "application_metadata": { 2024-09-18T08:13:14.339 INFO:teuthology.orchestra.run.smithi007.stdout: "rados": {} 2024-09-18T08:13:14.339 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:13:14.339 INFO:teuthology.orchestra.run.smithi007.stdout: "read_balance": { 2024-09-18T08:13:14.339 INFO:teuthology.orchestra.run.smithi007.stdout: "score_type": "Fair distribution", 2024-09-18T08:13:14.340 INFO:teuthology.orchestra.run.smithi007.stdout: "score_acting": 1.75, 2024-09-18T08:13:14.340 INFO:teuthology.orchestra.run.smithi007.stdout: "score_stable": 1.75, 2024-09-18T08:13:14.340 INFO:teuthology.orchestra.run.smithi007.stdout: "optimal_score": 1, 2024-09-18T08:13:14.340 INFO:teuthology.orchestra.run.smithi007.stdout: "raw_score_acting": 1.75, 2024-09-18T08:13:14.340 INFO:teuthology.orchestra.run.smithi007.stdout: "raw_score_stable": 1.75, 2024-09-18T08:13:14.340 INFO:teuthology.orchestra.run.smithi007.stdout: "primary_affinity_weighted": 0.84128189086914062, 2024-09-18T08:13:14.340 INFO:teuthology.orchestra.run.smithi007.stdout: "average_primary_affinity": 0.95999997854232788, 2024-09-18T08:13:14.340 INFO:teuthology.orchestra.run.smithi007.stdout: "average_primary_affinity_weighted": 0.8399999737739563 2024-09-18T08:13:14.340 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-09-18T08:13:14.340 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-09-18T08:13:14.340 INFO:teuthology.orchestra.run.smithi007.stdout: ], 2024-09-18T08:13:14.340 INFO:teuthology.orchestra.run.smithi007.stdout: "osds": [ 2024-09-18T08:13:14.340 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:13:14.341 INFO:teuthology.orchestra.run.smithi007.stdout: "osd": 0, 2024-09-18T08:13:14.341 INFO:teuthology.orchestra.run.smithi007.stdout: "uuid": "ae8f60df-72ac-46fb-91a1-f132b851c876", 2024-09-18T08:13:14.341 INFO:teuthology.orchestra.run.smithi007.stdout: "up": 1, 2024-09-18T08:13:14.341 INFO:teuthology.orchestra.run.smithi007.stdout: "in": 1, 2024-09-18T08:13:14.341 INFO:teuthology.orchestra.run.smithi007.stdout: "weight": 1, 2024-09-18T08:13:14.341 INFO:teuthology.orchestra.run.smithi007.stdout: "primary_affinity": 1, 2024-09-18T08:13:14.341 INFO:teuthology.orchestra.run.smithi007.stdout: "last_clean_begin": 0, 2024-09-18T08:13:14.341 INFO:teuthology.orchestra.run.smithi007.stdout: "last_clean_end": 0, 2024-09-18T08:13:14.341 INFO:teuthology.orchestra.run.smithi007.stdout: "up_from": 8, 2024-09-18T08:13:14.341 INFO:teuthology.orchestra.run.smithi007.stdout: "up_thru": 244, 2024-09-18T08:13:14.341 INFO:teuthology.orchestra.run.smithi007.stdout: "down_at": 0, 2024-09-18T08:13:14.341 INFO:teuthology.orchestra.run.smithi007.stdout: "lost_at": 0, 2024-09-18T08:13:14.341 INFO:teuthology.orchestra.run.smithi007.stdout: "public_addrs": { 2024-09-18T08:13:14.341 INFO:teuthology.orchestra.run.smithi007.stdout: "addrvec": [ 2024-09-18T08:13:14.342 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:13:14.342 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "v2", 2024-09-18T08:13:14.342 INFO:teuthology.orchestra.run.smithi007.stdout: "addr": "172.21.15.7:6801", 2024-09-18T08:13:14.342 INFO:teuthology.orchestra.run.smithi007.stdout: "nonce": 2896425794 2024-09-18T08:13:14.342 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-09-18T08:13:14.342 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-09-18T08:13:14.342 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:13:14.342 INFO:teuthology.orchestra.run.smithi007.stdout: "cluster_addrs": { 2024-09-18T08:13:14.342 INFO:teuthology.orchestra.run.smithi007.stdout: "addrvec": [ 2024-09-18T08:13:14.342 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:13:14.342 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "v2", 2024-09-18T08:13:14.342 INFO:teuthology.orchestra.run.smithi007.stdout: "addr": "172.21.15.7:6802", 2024-09-18T08:13:14.342 INFO:teuthology.orchestra.run.smithi007.stdout: "nonce": 2896425794 2024-09-18T08:13:14.343 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-09-18T08:13:14.343 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-09-18T08:13:14.343 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:13:14.343 INFO:teuthology.orchestra.run.smithi007.stdout: "heartbeat_back_addrs": { 2024-09-18T08:13:14.343 INFO:teuthology.orchestra.run.smithi007.stdout: "addrvec": [ 2024-09-18T08:13:14.343 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:13:14.343 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "v2", 2024-09-18T08:13:14.343 INFO:teuthology.orchestra.run.smithi007.stdout: "addr": "172.21.15.7:6804", 2024-09-18T08:13:14.343 INFO:teuthology.orchestra.run.smithi007.stdout: "nonce": 2896425794 2024-09-18T08:13:14.343 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-09-18T08:13:14.343 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-09-18T08:13:14.343 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:13:14.343 INFO:teuthology.orchestra.run.smithi007.stdout: "heartbeat_front_addrs": { 2024-09-18T08:13:14.344 INFO:teuthology.orchestra.run.smithi007.stdout: "addrvec": [ 2024-09-18T08:13:14.344 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:13:14.344 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "v2", 2024-09-18T08:13:14.344 INFO:teuthology.orchestra.run.smithi007.stdout: "addr": "172.21.15.7:6803", 2024-09-18T08:13:14.344 INFO:teuthology.orchestra.run.smithi007.stdout: "nonce": 2896425794 2024-09-18T08:13:14.344 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-09-18T08:13:14.344 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-09-18T08:13:14.345 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:13:14.345 INFO:teuthology.orchestra.run.smithi007.stdout: "public_addr": "172.21.15.7:6801/2896425794", 2024-09-18T08:13:14.345 INFO:teuthology.orchestra.run.smithi007.stdout: "cluster_addr": "172.21.15.7:6802/2896425794", 2024-09-18T08:13:14.345 INFO:teuthology.orchestra.run.smithi007.stdout: "heartbeat_back_addr": "172.21.15.7:6804/2896425794", 2024-09-18T08:13:14.345 INFO:teuthology.orchestra.run.smithi007.stdout: "heartbeat_front_addr": "172.21.15.7:6803/2896425794", 2024-09-18T08:13:14.345 INFO:teuthology.orchestra.run.smithi007.stdout: "state": [ 2024-09-18T08:13:14.345 INFO:teuthology.orchestra.run.smithi007.stdout: "exists", 2024-09-18T08:13:14.345 INFO:teuthology.orchestra.run.smithi007.stdout: "up" 2024-09-18T08:13:14.345 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-09-18T08:13:14.345 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:13:14.345 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:13:14.345 INFO:teuthology.orchestra.run.smithi007.stdout: "osd": 1, 2024-09-18T08:13:14.345 INFO:teuthology.orchestra.run.smithi007.stdout: "uuid": "be28b70c-1f73-494f-ad4b-cb7ec5311999", 2024-09-18T08:13:14.346 INFO:teuthology.orchestra.run.smithi007.stdout: "up": 0, 2024-09-18T08:13:14.346 INFO:teuthology.orchestra.run.smithi007.stdout: "in": 1, 2024-09-18T08:13:14.346 INFO:teuthology.orchestra.run.smithi007.stdout: "weight": 1, 2024-09-18T08:13:14.346 INFO:teuthology.orchestra.run.smithi007.stdout: "primary_affinity": 1, 2024-09-18T08:13:14.346 INFO:teuthology.orchestra.run.smithi007.stdout: "last_clean_begin": 0, 2024-09-18T08:13:14.346 INFO:teuthology.orchestra.run.smithi007.stdout: "last_clean_end": 0, 2024-09-18T08:13:14.346 INFO:teuthology.orchestra.run.smithi007.stdout: "up_from": 13, 2024-09-18T08:13:14.346 INFO:teuthology.orchestra.run.smithi007.stdout: "up_thru": 232, 2024-09-18T08:13:14.346 INFO:teuthology.orchestra.run.smithi007.stdout: "down_at": 244, 2024-09-18T08:13:14.346 INFO:teuthology.orchestra.run.smithi007.stdout: "lost_at": 0, 2024-09-18T08:13:14.346 INFO:teuthology.orchestra.run.smithi007.stdout: "public_addrs": { 2024-09-18T08:13:14.346 INFO:teuthology.orchestra.run.smithi007.stdout: "addrvec": [ 2024-09-18T08:13:14.346 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:13:14.347 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "v2", 2024-09-18T08:13:14.347 INFO:teuthology.orchestra.run.smithi007.stdout: "addr": "172.21.15.7:6805", 2024-09-18T08:13:14.347 INFO:teuthology.orchestra.run.smithi007.stdout: "nonce": 1971397549 2024-09-18T08:13:14.347 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-09-18T08:13:14.347 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-09-18T08:13:14.347 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:13:14.347 INFO:teuthology.orchestra.run.smithi007.stdout: "cluster_addrs": { 2024-09-18T08:13:14.347 INFO:teuthology.orchestra.run.smithi007.stdout: "addrvec": [ 2024-09-18T08:13:14.347 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:13:14.347 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "v2", 2024-09-18T08:13:14.347 INFO:teuthology.orchestra.run.smithi007.stdout: "addr": "172.21.15.7:6806", 2024-09-18T08:13:14.347 INFO:teuthology.orchestra.run.smithi007.stdout: "nonce": 1971397549 2024-09-18T08:13:14.347 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-09-18T08:13:14.348 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-09-18T08:13:14.348 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:13:14.348 INFO:teuthology.orchestra.run.smithi007.stdout: "heartbeat_back_addrs": { 2024-09-18T08:13:14.348 INFO:teuthology.orchestra.run.smithi007.stdout: "addrvec": [ 2024-09-18T08:13:14.348 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:13:14.348 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "v2", 2024-09-18T08:13:14.348 INFO:teuthology.orchestra.run.smithi007.stdout: "addr": "172.21.15.7:6808", 2024-09-18T08:13:14.348 INFO:teuthology.orchestra.run.smithi007.stdout: "nonce": 1971397549 2024-09-18T08:13:14.348 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-09-18T08:13:14.348 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-09-18T08:13:14.348 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:13:14.348 INFO:teuthology.orchestra.run.smithi007.stdout: "heartbeat_front_addrs": { 2024-09-18T08:13:14.348 INFO:teuthology.orchestra.run.smithi007.stdout: "addrvec": [ 2024-09-18T08:13:14.349 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:13:14.349 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "v2", 2024-09-18T08:13:14.349 INFO:teuthology.orchestra.run.smithi007.stdout: "addr": "172.21.15.7:6807", 2024-09-18T08:13:14.349 INFO:teuthology.orchestra.run.smithi007.stdout: "nonce": 1971397549 2024-09-18T08:13:14.349 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-09-18T08:13:14.349 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-09-18T08:13:14.349 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:13:14.349 INFO:teuthology.orchestra.run.smithi007.stdout: "public_addr": "172.21.15.7:6805/1971397549", 2024-09-18T08:13:14.349 INFO:teuthology.orchestra.run.smithi007.stdout: "cluster_addr": "172.21.15.7:6806/1971397549", 2024-09-18T08:13:14.350 INFO:teuthology.orchestra.run.smithi007.stdout: "heartbeat_back_addr": "172.21.15.7:6808/1971397549", 2024-09-18T08:13:14.350 INFO:teuthology.orchestra.run.smithi007.stdout: "heartbeat_front_addr": "172.21.15.7:6807/1971397549", 2024-09-18T08:13:14.350 INFO:teuthology.orchestra.run.smithi007.stdout: "state": [ 2024-09-18T08:13:14.350 INFO:teuthology.orchestra.run.smithi007.stdout: "exists" 2024-09-18T08:13:14.350 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-09-18T08:13:14.350 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:13:14.350 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:13:14.350 INFO:teuthology.orchestra.run.smithi007.stdout: "osd": 2, 2024-09-18T08:13:14.350 INFO:teuthology.orchestra.run.smithi007.stdout: "uuid": "cfa4f083-c7fd-4743-be65-84a3040fe6dc", 2024-09-18T08:13:14.350 INFO:teuthology.orchestra.run.smithi007.stdout: "up": 1, 2024-09-18T08:13:14.350 INFO:teuthology.orchestra.run.smithi007.stdout: "in": 1, 2024-09-18T08:13:14.350 INFO:teuthology.orchestra.run.smithi007.stdout: "weight": 1, 2024-09-18T08:13:14.350 INFO:teuthology.orchestra.run.smithi007.stdout: "primary_affinity": 1, 2024-09-18T08:13:14.351 INFO:teuthology.orchestra.run.smithi007.stdout: "last_clean_begin": 0, 2024-09-18T08:13:14.351 INFO:teuthology.orchestra.run.smithi007.stdout: "last_clean_end": 0, 2024-09-18T08:13:14.351 INFO:teuthology.orchestra.run.smithi007.stdout: "up_from": 18, 2024-09-18T08:13:14.351 INFO:teuthology.orchestra.run.smithi007.stdout: "up_thru": 244, 2024-09-18T08:13:14.351 INFO:teuthology.orchestra.run.smithi007.stdout: "down_at": 0, 2024-09-18T08:13:14.351 INFO:teuthology.orchestra.run.smithi007.stdout: "lost_at": 0, 2024-09-18T08:13:14.351 INFO:teuthology.orchestra.run.smithi007.stdout: "public_addrs": { 2024-09-18T08:13:14.351 INFO:teuthology.orchestra.run.smithi007.stdout: "addrvec": [ 2024-09-18T08:13:14.351 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:13:14.351 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "v2", 2024-09-18T08:13:14.351 INFO:teuthology.orchestra.run.smithi007.stdout: "addr": "172.21.15.7:6809", 2024-09-18T08:13:14.351 INFO:teuthology.orchestra.run.smithi007.stdout: "nonce": 556142559 2024-09-18T08:13:14.351 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-09-18T08:13:14.352 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-09-18T08:13:14.352 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:13:14.352 INFO:teuthology.orchestra.run.smithi007.stdout: "cluster_addrs": { 2024-09-18T08:13:14.352 INFO:teuthology.orchestra.run.smithi007.stdout: "addrvec": [ 2024-09-18T08:13:14.352 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:13:14.352 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "v2", 2024-09-18T08:13:14.352 INFO:teuthology.orchestra.run.smithi007.stdout: "addr": "172.21.15.7:6810", 2024-09-18T08:13:14.352 INFO:teuthology.orchestra.run.smithi007.stdout: "nonce": 556142559 2024-09-18T08:13:14.352 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-09-18T08:13:14.352 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-09-18T08:13:14.352 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:13:14.352 INFO:teuthology.orchestra.run.smithi007.stdout: "heartbeat_back_addrs": { 2024-09-18T08:13:14.352 INFO:teuthology.orchestra.run.smithi007.stdout: "addrvec": [ 2024-09-18T08:13:14.353 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:13:14.353 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "v2", 2024-09-18T08:13:14.353 INFO:teuthology.orchestra.run.smithi007.stdout: "addr": "172.21.15.7:6812", 2024-09-18T08:13:14.353 INFO:teuthology.orchestra.run.smithi007.stdout: "nonce": 556142559 2024-09-18T08:13:14.353 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-09-18T08:13:14.353 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-09-18T08:13:14.353 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:13:14.353 INFO:teuthology.orchestra.run.smithi007.stdout: "heartbeat_front_addrs": { 2024-09-18T08:13:14.353 INFO:teuthology.orchestra.run.smithi007.stdout: "addrvec": [ 2024-09-18T08:13:14.353 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:13:14.353 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "v2", 2024-09-18T08:13:14.353 INFO:teuthology.orchestra.run.smithi007.stdout: "addr": "172.21.15.7:6811", 2024-09-18T08:13:14.353 INFO:teuthology.orchestra.run.smithi007.stdout: "nonce": 556142559 2024-09-18T08:13:14.354 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-09-18T08:13:14.354 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-09-18T08:13:14.354 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:13:14.354 INFO:teuthology.orchestra.run.smithi007.stdout: "public_addr": "172.21.15.7:6809/556142559", 2024-09-18T08:13:14.354 INFO:teuthology.orchestra.run.smithi007.stdout: "cluster_addr": "172.21.15.7:6810/556142559", 2024-09-18T08:13:14.354 INFO:teuthology.orchestra.run.smithi007.stdout: "heartbeat_back_addr": "172.21.15.7:6812/556142559", 2024-09-18T08:13:14.354 INFO:teuthology.orchestra.run.smithi007.stdout: "heartbeat_front_addr": "172.21.15.7:6811/556142559", 2024-09-18T08:13:14.354 INFO:teuthology.orchestra.run.smithi007.stdout: "state": [ 2024-09-18T08:13:14.354 INFO:teuthology.orchestra.run.smithi007.stdout: "exists", 2024-09-18T08:13:14.354 INFO:teuthology.orchestra.run.smithi007.stdout: "up" 2024-09-18T08:13:14.354 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-09-18T08:13:14.354 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:13:14.354 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:13:14.355 INFO:teuthology.orchestra.run.smithi007.stdout: "osd": 3, 2024-09-18T08:13:14.355 INFO:teuthology.orchestra.run.smithi007.stdout: "uuid": "11ecd7e1-a751-4b75-95f5-00b32d2a5bc4", 2024-09-18T08:13:14.355 INFO:teuthology.orchestra.run.smithi007.stdout: "up": 1, 2024-09-18T08:13:14.355 INFO:teuthology.orchestra.run.smithi007.stdout: "in": 1, 2024-09-18T08:13:14.355 INFO:teuthology.orchestra.run.smithi007.stdout: "weight": 1, 2024-09-18T08:13:14.355 INFO:teuthology.orchestra.run.smithi007.stdout: "primary_affinity": 1, 2024-09-18T08:13:14.355 INFO:teuthology.orchestra.run.smithi007.stdout: "last_clean_begin": 0, 2024-09-18T08:13:14.355 INFO:teuthology.orchestra.run.smithi007.stdout: "last_clean_end": 0, 2024-09-18T08:13:14.355 INFO:teuthology.orchestra.run.smithi007.stdout: "up_from": 26, 2024-09-18T08:13:14.355 INFO:teuthology.orchestra.run.smithi007.stdout: "up_thru": 244, 2024-09-18T08:13:14.355 INFO:teuthology.orchestra.run.smithi007.stdout: "down_at": 0, 2024-09-18T08:13:14.355 INFO:teuthology.orchestra.run.smithi007.stdout: "lost_at": 0, 2024-09-18T08:13:14.355 INFO:teuthology.orchestra.run.smithi007.stdout: "public_addrs": { 2024-09-18T08:13:14.355 INFO:teuthology.orchestra.run.smithi007.stdout: "addrvec": [ 2024-09-18T08:13:14.356 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:13:14.356 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "v2", 2024-09-18T08:13:14.356 INFO:teuthology.orchestra.run.smithi007.stdout: "addr": "172.21.15.7:6813", 2024-09-18T08:13:14.356 INFO:teuthology.orchestra.run.smithi007.stdout: "nonce": 163587772 2024-09-18T08:13:14.356 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-09-18T08:13:14.356 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-09-18T08:13:14.356 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:13:14.356 INFO:teuthology.orchestra.run.smithi007.stdout: "cluster_addrs": { 2024-09-18T08:13:14.356 INFO:teuthology.orchestra.run.smithi007.stdout: "addrvec": [ 2024-09-18T08:13:14.356 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:13:14.356 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "v2", 2024-09-18T08:13:14.356 INFO:teuthology.orchestra.run.smithi007.stdout: "addr": "172.21.15.7:6814", 2024-09-18T08:13:14.356 INFO:teuthology.orchestra.run.smithi007.stdout: "nonce": 163587772 2024-09-18T08:13:14.357 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-09-18T08:13:14.357 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-09-18T08:13:14.357 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:13:14.357 INFO:teuthology.orchestra.run.smithi007.stdout: "heartbeat_back_addrs": { 2024-09-18T08:13:14.357 INFO:teuthology.orchestra.run.smithi007.stdout: "addrvec": [ 2024-09-18T08:13:14.357 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:13:14.357 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "v2", 2024-09-18T08:13:14.357 INFO:teuthology.orchestra.run.smithi007.stdout: "addr": "172.21.15.7:6816", 2024-09-18T08:13:14.357 INFO:teuthology.orchestra.run.smithi007.stdout: "nonce": 163587772 2024-09-18T08:13:14.357 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-09-18T08:13:14.357 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-09-18T08:13:14.357 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:13:14.357 INFO:teuthology.orchestra.run.smithi007.stdout: "heartbeat_front_addrs": { 2024-09-18T08:13:14.358 INFO:teuthology.orchestra.run.smithi007.stdout: "addrvec": [ 2024-09-18T08:13:14.358 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:13:14.358 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "v2", 2024-09-18T08:13:14.358 INFO:teuthology.orchestra.run.smithi007.stdout: "addr": "172.21.15.7:6815", 2024-09-18T08:13:14.358 INFO:teuthology.orchestra.run.smithi007.stdout: "nonce": 163587772 2024-09-18T08:13:14.358 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-09-18T08:13:14.358 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-09-18T08:13:14.358 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:13:14.358 INFO:teuthology.orchestra.run.smithi007.stdout: "public_addr": "172.21.15.7:6813/163587772", 2024-09-18T08:13:14.358 INFO:teuthology.orchestra.run.smithi007.stdout: "cluster_addr": "172.21.15.7:6814/163587772", 2024-09-18T08:13:14.358 INFO:teuthology.orchestra.run.smithi007.stdout: "heartbeat_back_addr": "172.21.15.7:6816/163587772", 2024-09-18T08:13:14.358 INFO:teuthology.orchestra.run.smithi007.stdout: "heartbeat_front_addr": "172.21.15.7:6815/163587772", 2024-09-18T08:13:14.358 INFO:teuthology.orchestra.run.smithi007.stdout: "state": [ 2024-09-18T08:13:14.358 INFO:teuthology.orchestra.run.smithi007.stdout: "exists", 2024-09-18T08:13:14.359 INFO:teuthology.orchestra.run.smithi007.stdout: "up" 2024-09-18T08:13:14.359 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-09-18T08:13:14.359 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:13:14.359 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:13:14.359 INFO:teuthology.orchestra.run.smithi007.stdout: "osd": 4, 2024-09-18T08:13:14.359 INFO:teuthology.orchestra.run.smithi007.stdout: "uuid": "98992b34-cfa1-45bf-97a1-7d557739252f", 2024-09-18T08:13:14.359 INFO:teuthology.orchestra.run.smithi007.stdout: "up": 1, 2024-09-18T08:13:14.359 INFO:teuthology.orchestra.run.smithi007.stdout: "in": 1, 2024-09-18T08:13:14.359 INFO:teuthology.orchestra.run.smithi007.stdout: "weight": 1, 2024-09-18T08:13:14.359 INFO:teuthology.orchestra.run.smithi007.stdout: "primary_affinity": 1, 2024-09-18T08:13:14.359 INFO:teuthology.orchestra.run.smithi007.stdout: "last_clean_begin": 0, 2024-09-18T08:13:14.359 INFO:teuthology.orchestra.run.smithi007.stdout: "last_clean_end": 0, 2024-09-18T08:13:14.359 INFO:teuthology.orchestra.run.smithi007.stdout: "up_from": 32, 2024-09-18T08:13:14.359 INFO:teuthology.orchestra.run.smithi007.stdout: "up_thru": 244, 2024-09-18T08:13:14.360 INFO:teuthology.orchestra.run.smithi007.stdout: "down_at": 0, 2024-09-18T08:13:14.360 INFO:teuthology.orchestra.run.smithi007.stdout: "lost_at": 0, 2024-09-18T08:13:14.360 INFO:teuthology.orchestra.run.smithi007.stdout: "public_addrs": { 2024-09-18T08:13:14.360 INFO:teuthology.orchestra.run.smithi007.stdout: "addrvec": [ 2024-09-18T08:13:14.360 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:13:14.360 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "v2", 2024-09-18T08:13:14.360 INFO:teuthology.orchestra.run.smithi007.stdout: "addr": "172.21.15.50:6800", 2024-09-18T08:13:14.360 INFO:teuthology.orchestra.run.smithi007.stdout: "nonce": 3662119805 2024-09-18T08:13:14.360 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-09-18T08:13:14.360 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-09-18T08:13:14.360 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:13:14.360 INFO:teuthology.orchestra.run.smithi007.stdout: "cluster_addrs": { 2024-09-18T08:13:14.360 INFO:teuthology.orchestra.run.smithi007.stdout: "addrvec": [ 2024-09-18T08:13:14.361 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:13:14.361 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "v2", 2024-09-18T08:13:14.361 INFO:teuthology.orchestra.run.smithi007.stdout: "addr": "172.21.15.50:6801", 2024-09-18T08:13:14.361 INFO:teuthology.orchestra.run.smithi007.stdout: "nonce": 3662119805 2024-09-18T08:13:14.361 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-09-18T08:13:14.361 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-09-18T08:13:14.361 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:13:14.361 INFO:teuthology.orchestra.run.smithi007.stdout: "heartbeat_back_addrs": { 2024-09-18T08:13:14.361 INFO:teuthology.orchestra.run.smithi007.stdout: "addrvec": [ 2024-09-18T08:13:14.361 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:13:14.361 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "v2", 2024-09-18T08:13:14.361 INFO:teuthology.orchestra.run.smithi007.stdout: "addr": "172.21.15.50:6803", 2024-09-18T08:13:14.361 INFO:teuthology.orchestra.run.smithi007.stdout: "nonce": 3662119805 2024-09-18T08:13:14.361 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-09-18T08:13:14.362 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-09-18T08:13:14.362 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:13:14.362 INFO:teuthology.orchestra.run.smithi007.stdout: "heartbeat_front_addrs": { 2024-09-18T08:13:14.362 INFO:teuthology.orchestra.run.smithi007.stdout: "addrvec": [ 2024-09-18T08:13:14.362 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:13:14.362 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "v2", 2024-09-18T08:13:14.362 INFO:teuthology.orchestra.run.smithi007.stdout: "addr": "172.21.15.50:6802", 2024-09-18T08:13:14.362 INFO:teuthology.orchestra.run.smithi007.stdout: "nonce": 3662119805 2024-09-18T08:13:14.362 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-09-18T08:13:14.362 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-09-18T08:13:14.362 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:13:14.362 INFO:teuthology.orchestra.run.smithi007.stdout: "public_addr": "172.21.15.50:6800/3662119805", 2024-09-18T08:13:14.362 INFO:teuthology.orchestra.run.smithi007.stdout: "cluster_addr": "172.21.15.50:6801/3662119805", 2024-09-18T08:13:14.363 INFO:teuthology.orchestra.run.smithi007.stdout: "heartbeat_back_addr": "172.21.15.50:6803/3662119805", 2024-09-18T08:13:14.363 INFO:teuthology.orchestra.run.smithi007.stdout: "heartbeat_front_addr": "172.21.15.50:6802/3662119805", 2024-09-18T08:13:14.363 INFO:teuthology.orchestra.run.smithi007.stdout: "state": [ 2024-09-18T08:13:14.363 INFO:teuthology.orchestra.run.smithi007.stdout: "exists", 2024-09-18T08:13:14.363 INFO:teuthology.orchestra.run.smithi007.stdout: "up" 2024-09-18T08:13:14.363 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-09-18T08:13:14.363 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:13:14.363 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:13:14.363 INFO:teuthology.orchestra.run.smithi007.stdout: "osd": 5, 2024-09-18T08:13:14.363 INFO:teuthology.orchestra.run.smithi007.stdout: "uuid": "7f290005-ebca-467b-87e4-02ead1063610", 2024-09-18T08:13:14.363 INFO:teuthology.orchestra.run.smithi007.stdout: "up": 1, 2024-09-18T08:13:14.363 INFO:teuthology.orchestra.run.smithi007.stdout: "in": 1, 2024-09-18T08:13:14.363 INFO:teuthology.orchestra.run.smithi007.stdout: "weight": 1, 2024-09-18T08:13:14.364 INFO:teuthology.orchestra.run.smithi007.stdout: "primary_affinity": 1, 2024-09-18T08:13:14.364 INFO:teuthology.orchestra.run.smithi007.stdout: "last_clean_begin": 0, 2024-09-18T08:13:14.364 INFO:teuthology.orchestra.run.smithi007.stdout: "last_clean_end": 0, 2024-09-18T08:13:14.364 INFO:teuthology.orchestra.run.smithi007.stdout: "up_from": 39, 2024-09-18T08:13:14.364 INFO:teuthology.orchestra.run.smithi007.stdout: "up_thru": 244, 2024-09-18T08:13:14.364 INFO:teuthology.orchestra.run.smithi007.stdout: "down_at": 0, 2024-09-18T08:13:14.364 INFO:teuthology.orchestra.run.smithi007.stdout: "lost_at": 0, 2024-09-18T08:13:14.364 INFO:teuthology.orchestra.run.smithi007.stdout: "public_addrs": { 2024-09-18T08:13:14.364 INFO:teuthology.orchestra.run.smithi007.stdout: "addrvec": [ 2024-09-18T08:13:14.364 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:13:14.364 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "v2", 2024-09-18T08:13:14.365 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7 2024-09-18T08:13:14.366 INFO:teuthology.orchestra.run.smithi007.stdout: "addr": "172.21.15.50:6804", 2024-09-18T08:13:14.366 INFO:teuthology.orchestra.run.smithi007.stdout: "nonce": 1199797954 2024-09-18T08:13:14.366 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-09-18T08:13:14.366 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-09-18T08:13:14.366 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:13:14.366 INFO:teuthology.orchestra.run.smithi007.stdout: "cluster_addrs": { 2024-09-18T08:13:14.366 INFO:teuthology.orchestra.run.smithi007.stdout: "addrvec": [ 2024-09-18T08:13:14.366 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:13:14.366 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "v2", 2024-09-18T08:13:14.366 INFO:teuthology.orchestra.run.smithi007.stdout: "addr": "172.21.15.50:6805", 2024-09-18T08:13:14.366 INFO:teuthology.orchestra.run.smithi007.stdout: "nonce": 1199797954 2024-09-18T08:13:14.367 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-09-18T08:13:14.367 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-09-18T08:13:14.367 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:13:14.367 INFO:teuthology.orchestra.run.smithi007.stdout: "heartbeat_back_addrs": { 2024-09-18T08:13:14.367 INFO:teuthology.orchestra.run.smithi007.stdout: "addrvec": [ 2024-09-18T08:13:14.367 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:13:14.367 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "v2", 2024-09-18T08:13:14.367 INFO:teuthology.orchestra.run.smithi007.stdout: "addr": "172.21.15.50:6807", 2024-09-18T08:13:14.367 INFO:teuthology.orchestra.run.smithi007.stdout: "nonce": 1199797954 2024-09-18T08:13:14.367 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-09-18T08:13:14.367 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-09-18T08:13:14.367 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:13:14.367 INFO:teuthology.orchestra.run.smithi007.stdout: "heartbeat_front_addrs": { 2024-09-18T08:13:14.367 INFO:teuthology.orchestra.run.smithi007.stdout: "addrvec": [ 2024-09-18T08:13:14.368 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:13:14.368 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "v2", 2024-09-18T08:13:14.368 INFO:teuthology.orchestra.run.smithi007.stdout: "addr": "172.21.15.50:6806", 2024-09-18T08:13:14.368 INFO:teuthology.orchestra.run.smithi007.stdout: "nonce": 1199797954 2024-09-18T08:13:14.368 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-09-18T08:13:14.368 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-09-18T08:13:14.368 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:13:14.368 INFO:teuthology.orchestra.run.smithi007.stdout: "public_addr": "172.21.15.50:6804/1199797954", 2024-09-18T08:13:14.368 INFO:teuthology.orchestra.run.smithi007.stdout: "cluster_addr": "172.21.15.50:6805/1199797954", 2024-09-18T08:13:14.368 INFO:teuthology.orchestra.run.smithi007.stdout: "heartbeat_back_addr": "172.21.15.50:6807/1199797954", 2024-09-18T08:13:14.368 INFO:teuthology.orchestra.run.smithi007.stdout: "heartbeat_front_addr": "172.21.15.50:6806/1199797954", 2024-09-18T08:13:14.368 INFO:teuthology.orchestra.run.smithi007.stdout: "state": [ 2024-09-18T08:13:14.369 INFO:teuthology.orchestra.run.smithi007.stdout: "exists", 2024-09-18T08:13:14.369 INFO:teuthology.orchestra.run.smithi007.stdout: "up" 2024-09-18T08:13:14.369 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-09-18T08:13:14.369 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:13:14.369 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:13:14.369 INFO:teuthology.orchestra.run.smithi007.stdout: "osd": 6, 2024-09-18T08:13:14.369 INFO:teuthology.orchestra.run.smithi007.stdout: "uuid": "e07d846c-ae42-4fea-aa5c-3a25835d18a6", 2024-09-18T08:13:14.369 INFO:teuthology.orchestra.run.smithi007.stdout: "up": 1, 2024-09-18T08:13:14.369 INFO:teuthology.orchestra.run.smithi007.stdout: "in": 1, 2024-09-18T08:13:14.369 INFO:teuthology.orchestra.run.smithi007.stdout: "weight": 1, 2024-09-18T08:13:14.369 INFO:teuthology.orchestra.run.smithi007.stdout: "primary_affinity": 0, 2024-09-18T08:13:14.369 INFO:teuthology.orchestra.run.smithi007.stdout: "last_clean_begin": 0, 2024-09-18T08:13:14.369 INFO:teuthology.orchestra.run.smithi007.stdout: "last_clean_end": 0, 2024-09-18T08:13:14.369 INFO:teuthology.orchestra.run.smithi007.stdout: "up_from": 47, 2024-09-18T08:13:14.370 INFO:teuthology.orchestra.run.smithi007.stdout: "up_thru": 244, 2024-09-18T08:13:14.370 INFO:teuthology.orchestra.run.smithi007.stdout: "down_at": 0, 2024-09-18T08:13:14.370 INFO:teuthology.orchestra.run.smithi007.stdout: "lost_at": 0, 2024-09-18T08:13:14.370 INFO:teuthology.orchestra.run.smithi007.stdout: "public_addrs": { 2024-09-18T08:13:14.370 INFO:teuthology.orchestra.run.smithi007.stdout: "addrvec": [ 2024-09-18T08:13:14.370 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:13:14.370 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "v2", 2024-09-18T08:13:14.370 INFO:teuthology.orchestra.run.smithi007.stdout: "addr": "172.21.15.50:6808", 2024-09-18T08:13:14.370 INFO:teuthology.orchestra.run.smithi007.stdout: "nonce": 255960422 2024-09-18T08:13:14.370 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-09-18T08:13:14.370 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-09-18T08:13:14.370 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:13:14.370 INFO:teuthology.orchestra.run.smithi007.stdout: "cluster_addrs": { 2024-09-18T08:13:14.371 INFO:teuthology.orchestra.run.smithi007.stdout: "addrvec": [ 2024-09-18T08:13:14.371 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:13:14.371 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "v2", 2024-09-18T08:13:14.371 INFO:teuthology.orchestra.run.smithi007.stdout: "addr": "172.21.15.50:6809", 2024-09-18T08:13:14.371 INFO:teuthology.orchestra.run.smithi007.stdout: "nonce": 255960422 2024-09-18T08:13:14.371 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-09-18T08:13:14.371 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-09-18T08:13:14.371 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:13:14.371 INFO:teuthology.orchestra.run.smithi007.stdout: "heartbeat_back_addrs": { 2024-09-18T08:13:14.371 INFO:teuthology.orchestra.run.smithi007.stdout: "addrvec": [ 2024-09-18T08:13:14.371 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:13:14.371 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "v2", 2024-09-18T08:13:14.371 INFO:teuthology.orchestra.run.smithi007.stdout: "addr": "172.21.15.50:6811", 2024-09-18T08:13:14.371 INFO:teuthology.orchestra.run.smithi007.stdout: "nonce": 255960422 2024-09-18T08:13:14.372 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-09-18T08:13:14.372 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-09-18T08:13:14.372 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:13:14.372 INFO:teuthology.orchestra.run.smithi007.stdout: "heartbeat_front_addrs": { 2024-09-18T08:13:14.372 INFO:teuthology.orchestra.run.smithi007.stdout: "addrvec": [ 2024-09-18T08:13:14.372 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:13:14.372 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "v2", 2024-09-18T08:13:14.372 INFO:teuthology.orchestra.run.smithi007.stdout: "addr": "172.21.15.50:6810", 2024-09-18T08:13:14.372 INFO:teuthology.orchestra.run.smithi007.stdout: "nonce": 255960422 2024-09-18T08:13:14.372 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-09-18T08:13:14.372 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-09-18T08:13:14.372 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:13:14.372 INFO:teuthology.orchestra.run.smithi007.stdout: "public_addr": "172.21.15.50:6808/255960422", 2024-09-18T08:13:14.373 INFO:teuthology.orchestra.run.smithi007.stdout: "cluster_addr": "172.21.15.50:6809/255960422", 2024-09-18T08:13:14.373 INFO:teuthology.orchestra.run.smithi007.stdout: "heartbeat_back_addr": "172.21.15.50:6811/255960422", 2024-09-18T08:13:14.373 INFO:teuthology.orchestra.run.smithi007.stdout: "heartbeat_front_addr": "172.21.15.50:6810/255960422", 2024-09-18T08:13:14.373 INFO:teuthology.orchestra.run.smithi007.stdout: "state": [ 2024-09-18T08:13:14.373 INFO:teuthology.orchestra.run.smithi007.stdout: "exists", 2024-09-18T08:13:14.373 INFO:teuthology.orchestra.run.smithi007.stdout: "up" 2024-09-18T08:13:14.373 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-09-18T08:13:14.373 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:13:14.373 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:13:14.373 INFO:teuthology.orchestra.run.smithi007.stdout: "osd": 7, 2024-09-18T08:13:14.373 INFO:teuthology.orchestra.run.smithi007.stdout: "uuid": "c9dac3b7-d651-489d-b96e-5b10fa32d9af", 2024-09-18T08:13:14.373 INFO:teuthology.orchestra.run.smithi007.stdout: "up": 1, 2024-09-18T08:13:14.373 INFO:teuthology.orchestra.run.smithi007.stdout: "in": 1, 2024-09-18T08:13:14.374 INFO:teuthology.orchestra.run.smithi007.stdout: "weight": 1, 2024-09-18T08:13:14.374 INFO:teuthology.orchestra.run.smithi007.stdout: "primary_affinity": 0.730255126953125, 2024-09-18T08:13:14.374 INFO:teuthology.orchestra.run.smithi007.stdout: "last_clean_begin": 0, 2024-09-18T08:13:14.374 INFO:teuthology.orchestra.run.smithi007.stdout: "last_clean_end": 0, 2024-09-18T08:13:14.374 INFO:teuthology.orchestra.run.smithi007.stdout: "up_from": 56, 2024-09-18T08:13:14.374 INFO:teuthology.orchestra.run.smithi007.stdout: "up_thru": 244, 2024-09-18T08:13:14.374 INFO:teuthology.orchestra.run.smithi007.stdout: "down_at": 0, 2024-09-18T08:13:14.374 INFO:teuthology.orchestra.run.smithi007.stdout: "lost_at": 0, 2024-09-18T08:13:14.374 INFO:teuthology.orchestra.run.smithi007.stdout: "public_addrs": { 2024-09-18T08:13:14.374 INFO:teuthology.orchestra.run.smithi007.stdout: "addrvec": [ 2024-09-18T08:13:14.374 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:13:14.374 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "v2", 2024-09-18T08:13:14.374 INFO:teuthology.orchestra.run.smithi007.stdout: "addr": "172.21.15.50:6812", 2024-09-18T08:13:14.375 INFO:teuthology.orchestra.run.smithi007.stdout: "nonce": 874923904 2024-09-18T08:13:14.375 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-09-18T08:13:14.375 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-09-18T08:13:14.375 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:13:14.375 INFO:teuthology.orchestra.run.smithi007.stdout: "cluster_addrs": { 2024-09-18T08:13:14.375 INFO:teuthology.orchestra.run.smithi007.stdout: "addrvec": [ 2024-09-18T08:13:14.375 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:13:14.375 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "v2", 2024-09-18T08:13:14.375 INFO:teuthology.orchestra.run.smithi007.stdout: "addr": "172.21.15.50:6813", 2024-09-18T08:13:14.375 INFO:teuthology.orchestra.run.smithi007.stdout: "nonce": 874923904 2024-09-18T08:13:14.375 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-09-18T08:13:14.375 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-09-18T08:13:14.375 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:13:14.375 INFO:teuthology.orchestra.run.smithi007.stdout: "heartbeat_back_addrs": { 2024-09-18T08:13:14.376 INFO:teuthology.orchestra.run.smithi007.stdout: "addrvec": [ 2024-09-18T08:13:14.376 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:13:14.376 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "v2", 2024-09-18T08:13:14.376 INFO:teuthology.orchestra.run.smithi007.stdout: "addr": "172.21.15.50:6815", 2024-09-18T08:13:14.376 INFO:teuthology.orchestra.run.smithi007.stdout: "nonce": 874923904 2024-09-18T08:13:14.376 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-09-18T08:13:14.376 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-09-18T08:13:14.376 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:13:14.376 INFO:teuthology.orchestra.run.smithi007.stdout: "heartbeat_front_addrs": { 2024-09-18T08:13:14.376 INFO:teuthology.orchestra.run.smithi007.stdout: "addrvec": [ 2024-09-18T08:13:14.376 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:13:14.376 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "v2", 2024-09-18T08:13:14.377 INFO:teuthology.orchestra.run.smithi007.stdout: "addr": "172.21.15.50:6814", 2024-09-18T08:13:14.377 INFO:teuthology.orchestra.run.smithi007.stdout: "nonce": 874923904 2024-09-18T08:13:14.377 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-09-18T08:13:14.377 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-09-18T08:13:14.377 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:13:14.377 INFO:teuthology.orchestra.run.smithi007.stdout: "public_addr": "172.21.15.50:6812/874923904", 2024-09-18T08:13:14.377 INFO:teuthology.orchestra.run.smithi007.stdout: "cluster_addr": "172.21.15.50:6813/874923904", 2024-09-18T08:13:14.377 INFO:teuthology.orchestra.run.smithi007.stdout: "heartbeat_back_addr": "172.21.15.50:6815/874923904", 2024-09-18T08:13:14.377 INFO:teuthology.orchestra.run.smithi007.stdout: "heartbeat_front_addr": "172.21.15.50:6814/874923904", 2024-09-18T08:13:14.377 INFO:teuthology.orchestra.run.smithi007.stdout: "state": [ 2024-09-18T08:13:14.377 INFO:teuthology.orchestra.run.smithi007.stdout: "exists", 2024-09-18T08:13:14.377 INFO:teuthology.orchestra.run.smithi007.stdout: "up" 2024-09-18T08:13:14.377 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-09-18T08:13:14.378 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-09-18T08:13:14.378 INFO:teuthology.orchestra.run.smithi007.stdout: ], 2024-09-18T08:13:14.378 INFO:teuthology.orchestra.run.smithi007.stdout: "osd_xinfo": [ 2024-09-18T08:13:14.378 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:13:14.378 INFO:teuthology.orchestra.run.smithi007.stdout: "osd": 0, 2024-09-18T08:13:14.378 INFO:teuthology.orchestra.run.smithi007.stdout: "down_stamp": "0.000000", 2024-09-18T08:13:14.378 INFO:teuthology.orchestra.run.smithi007.stdout: "laggy_probability": 0, 2024-09-18T08:13:14.378 INFO:teuthology.orchestra.run.smithi007.stdout: "laggy_interval": 0, 2024-09-18T08:13:14.378 INFO:teuthology.orchestra.run.smithi007.stdout: "features": 4540719139924082687, 2024-09-18T08:13:14.378 INFO:teuthology.orchestra.run.smithi007.stdout: "old_weight": 0, 2024-09-18T08:13:14.378 INFO:teuthology.orchestra.run.smithi007.stdout: "last_purged_snaps_scrub": "2024-09-18T08:12:36.340648+0000", 2024-09-18T08:13:14.378 INFO:teuthology.orchestra.run.smithi007.stdout: "dead_epoch": 0 2024-09-18T08:13:14.378 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:13:14.378 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:13:14.379 INFO:teuthology.orchestra.run.smithi007.stdout: "osd": 1, 2024-09-18T08:13:14.379 INFO:teuthology.orchestra.run.smithi007.stdout: "down_stamp": "0.000000", 2024-09-18T08:13:14.379 INFO:teuthology.orchestra.run.smithi007.stdout: "laggy_probability": 0, 2024-09-18T08:13:14.379 INFO:teuthology.orchestra.run.smithi007.stdout: "laggy_interval": 0, 2024-09-18T08:13:14.379 INFO:teuthology.orchestra.run.smithi007.stdout: "features": 4540719139924082687, 2024-09-18T08:13:14.379 INFO:teuthology.orchestra.run.smithi007.stdout: "old_weight": 0, 2024-09-18T08:13:14.379 INFO:teuthology.orchestra.run.smithi007.stdout: "last_purged_snaps_scrub": "2024-09-18T08:10:56.667957+0000", 2024-09-18T08:13:14.379 INFO:teuthology.orchestra.run.smithi007.stdout: "dead_epoch": 243 2024-09-18T08:13:14.379 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:13:14.379 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:13:14.379 INFO:teuthology.orchestra.run.smithi007.stdout: "osd": 2, 2024-09-18T08:13:14.379 INFO:teuthology.orchestra.run.smithi007.stdout: "down_stamp": "0.000000", 2024-09-18T08:13:14.379 INFO:teuthology.orchestra.run.smithi007.stdout: "laggy_probability": 0, 2024-09-18T08:13:14.379 INFO:teuthology.orchestra.run.smithi007.stdout: "laggy_interval": 0, 2024-09-18T08:13:14.380 INFO:teuthology.orchestra.run.smithi007.stdout: "features": 4540719139924082687, 2024-09-18T08:13:14.380 INFO:teuthology.orchestra.run.smithi007.stdout: "old_weight": 0, 2024-09-18T08:13:14.380 INFO:teuthology.orchestra.run.smithi007.stdout: "last_purged_snaps_scrub": "2024-09-18T08:12:30.885757+0000", 2024-09-18T08:13:14.380 INFO:teuthology.orchestra.run.smithi007.stdout: "dead_epoch": 0 2024-09-18T08:13:14.380 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:13:14.380 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:13:14.380 INFO:teuthology.orchestra.run.smithi007.stdout: "osd": 3, 2024-09-18T08:13:14.380 INFO:teuthology.orchestra.run.smithi007.stdout: "down_stamp": "0.000000", 2024-09-18T08:13:14.380 INFO:teuthology.orchestra.run.smithi007.stdout: "laggy_probability": 0, 2024-09-18T08:13:14.380 INFO:teuthology.orchestra.run.smithi007.stdout: "laggy_interval": 0, 2024-09-18T08:13:14.380 INFO:teuthology.orchestra.run.smithi007.stdout: "features": 4540719139924082687, 2024-09-18T08:13:14.380 INFO:teuthology.orchestra.run.smithi007.stdout: "old_weight": 0, 2024-09-18T08:13:14.381 INFO:teuthology.orchestra.run.smithi007.stdout: "last_purged_snaps_scrub": "2024-09-18T08:13:01.289441+0000", 2024-09-18T08:13:14.381 INFO:teuthology.orchestra.run.smithi007.stdout: "dead_epoch": 0 2024-09-18T08:13:14.381 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:13:14.381 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:13:14.381 INFO:teuthology.orchestra.run.smithi007.stdout: "osd": 4, 2024-09-18T08:13:14.381 INFO:teuthology.orchestra.run.smithi007.stdout: "down_stamp": "0.000000", 2024-09-18T08:13:14.381 INFO:teuthology.orchestra.run.smithi007.stdout: "laggy_probability": 0, 2024-09-18T08:13:14.381 INFO:teuthology.orchestra.run.smithi007.stdout: "laggy_interval": 0, 2024-09-18T08:13:14.381 INFO:teuthology.orchestra.run.smithi007.stdout: "features": 4540719139924082687, 2024-09-18T08:13:14.381 INFO:teuthology.orchestra.run.smithi007.stdout: "old_weight": 0, 2024-09-18T08:13:14.381 INFO:teuthology.orchestra.run.smithi007.stdout: "last_purged_snaps_scrub": "2024-09-18T08:13:10.281294+0000", 2024-09-18T08:13:14.381 INFO:teuthology.orchestra.run.smithi007.stdout: "dead_epoch": 0 2024-09-18T08:13:14.381 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:13:14.382 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:13:14.382 INFO:teuthology.orchestra.run.smithi007.stdout: "osd": 5, 2024-09-18T08:13:14.382 INFO:teuthology.orchestra.run.smithi007.stdout: "down_stamp": "0.000000", 2024-09-18T08:13:14.382 INFO:teuthology.orchestra.run.smithi007.stdout: "laggy_probability": 0, 2024-09-18T08:13:14.382 INFO:teuthology.orchestra.run.smithi007.stdout: "laggy_interval": 0, 2024-09-18T08:13:14.382 INFO:teuthology.orchestra.run.smithi007.stdout: "features": 4540719139924082687, 2024-09-18T08:13:14.382 INFO:teuthology.orchestra.run.smithi007.stdout: "old_weight": 0, 2024-09-18T08:13:14.382 INFO:teuthology.orchestra.run.smithi007.stdout: "last_purged_snaps_scrub": "2024-09-18T08:12:19.708001+0000", 2024-09-18T08:13:14.382 INFO:teuthology.orchestra.run.smithi007.stdout: "dead_epoch": 0 2024-09-18T08:13:14.382 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:13:14.382 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:13:14.382 INFO:teuthology.orchestra.run.smithi007.stdout: "osd": 6, 2024-09-18T08:13:14.382 INFO:teuthology.orchestra.run.smithi007.stdout: "down_stamp": "0.000000", 2024-09-18T08:13:14.382 INFO:teuthology.orchestra.run.smithi007.stdout: "laggy_probability": 0, 2024-09-18T08:13:14.383 INFO:teuthology.orchestra.run.smithi007.stdout: "laggy_interval": 0, 2024-09-18T08:13:14.383 INFO:teuthology.orchestra.run.smithi007.stdout: "features": 4540719139924082687, 2024-09-18T08:13:14.383 INFO:teuthology.orchestra.run.smithi007.stdout: "old_weight": 0, 2024-09-18T08:13:14.383 INFO:teuthology.orchestra.run.smithi007.stdout: "last_purged_snaps_scrub": "2024-09-18T08:13:03.684975+0000", 2024-09-18T08:13:14.383 INFO:teuthology.orchestra.run.smithi007.stdout: "dead_epoch": 0 2024-09-18T08:13:14.383 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:13:14.383 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:13:14.383 INFO:teuthology.orchestra.run.smithi007.stdout: "osd": 7, 2024-09-18T08:13:14.383 INFO:teuthology.orchestra.run.smithi007.stdout: "down_stamp": "0.000000", 2024-09-18T08:13:14.383 INFO:teuthology.orchestra.run.smithi007.stdout: "laggy_probability": 0, 2024-09-18T08:13:14.383 INFO:teuthology.orchestra.run.smithi007.stdout: "laggy_interval": 0, 2024-09-18T08:13:14.383 INFO:teuthology.orchestra.run.smithi007.stdout: "features": 4540719139924082687, 2024-09-18T08:13:14.383 INFO:teuthology.orchestra.run.smithi007.stdout: "old_weight": 0, 2024-09-18T08:13:14.383 INFO:teuthology.orchestra.run.smithi007.stdout: "last_purged_snaps_scrub": "2024-09-18T08:12:42.349589+0000", 2024-09-18T08:13:14.384 INFO:teuthology.orchestra.run.smithi007.stdout: "dead_epoch": 0 2024-09-18T08:13:14.384 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-09-18T08:13:14.384 INFO:teuthology.orchestra.run.smithi007.stdout: ], 2024-09-18T08:13:14.384 INFO:teuthology.orchestra.run.smithi007.stdout: "pg_upmap": [], 2024-09-18T08:13:14.384 INFO:teuthology.orchestra.run.smithi007.stdout: "pg_upmap_items": [ 2024-09-18T08:13:14.384 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:13:14.384 INFO:teuthology.orchestra.run.smithi007.stdout: "pgid": "4.7", 2024-09-18T08:13:14.384 INFO:teuthology.orchestra.run.smithi007.stdout: "mappings": [ 2024-09-18T08:13:14.384 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:13:14.384 INFO:teuthology.orchestra.run.smithi007.stdout: "from": 1, 2024-09-18T08:13:14.384 INFO:teuthology.orchestra.run.smithi007.stdout: "to": 2 2024-09-18T08:13:14.384 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-09-18T08:13:14.384 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-09-18T08:13:14.385 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:13:14.385 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:13:14.385 INFO:teuthology.orchestra.run.smithi007.stdout: "pgid": "4.9", 2024-09-18T08:13:14.385 INFO:teuthology.orchestra.run.smithi007.stdout: "mappings": [ 2024-09-18T08:13:14.385 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:13:14.385 INFO:teuthology.orchestra.run.smithi007.stdout: "from": 1, 2024-09-18T08:13:14.385 INFO:teuthology.orchestra.run.smithi007.stdout: "to": 5 2024-09-18T08:13:14.385 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-09-18T08:13:14.385 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-09-18T08:13:14.385 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:13:14.385 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:13:14.385 INFO:teuthology.orchestra.run.smithi007.stdout: "pgid": "4.14", 2024-09-18T08:13:14.385 INFO:teuthology.orchestra.run.smithi007.stdout: "mappings": [ 2024-09-18T08:13:14.385 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:13:14.386 INFO:teuthology.orchestra.run.smithi007.stdout: "from": 1, 2024-09-18T08:13:14.386 INFO:teuthology.orchestra.run.smithi007.stdout: "to": 2 2024-09-18T08:13:14.386 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-09-18T08:13:14.386 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-09-18T08:13:14.386 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-09-18T08:13:14.386 INFO:teuthology.orchestra.run.smithi007.stdout: ], 2024-09-18T08:13:14.386 INFO:teuthology.orchestra.run.smithi007.stdout: "pg_upmap_primaries": [], 2024-09-18T08:13:14.386 INFO:teuthology.orchestra.run.smithi007.stdout: "pg_temp": [], 2024-09-18T08:13:14.386 INFO:teuthology.orchestra.run.smithi007.stdout: "primary_temp": [], 2024-09-18T08:13:14.386 INFO:teuthology.orchestra.run.smithi007.stdout: "blocklist": { 2024-09-18T08:13:14.386 INFO:teuthology.orchestra.run.smithi007.stdout: "172.21.15.7:0/398367658": "2024-09-19T08:03:17.052903+0000", 2024-09-18T08:13:14.386 INFO:teuthology.orchestra.run.smithi007.stdout: "172.21.15.7:6800/2918496291": "2024-09-19T07:57:31.015269+0000", 2024-09-18T08:13:14.386 INFO:teuthology.orchestra.run.smithi007.stdout: "172.21.15.7:0/2514354773": "2024-09-19T07:57:31.015269+0000", 2024-09-18T08:13:14.387 INFO:teuthology.orchestra.run.smithi007.stdout: "172.21.15.7:0/3924911335": "2024-09-19T07:57:31.015269+0000", 2024-09-18T08:13:14.387 INFO:teuthology.orchestra.run.smithi007.stdout: "172.21.15.7:0/1668598084": "2024-09-19T07:57:57.346149+0000", 2024-09-18T08:13:14.387 INFO:teuthology.orchestra.run.smithi007.stdout: "172.21.15.7:0/4272027292": "2024-09-19T07:57:31.015269+0000", 2024-09-18T08:13:14.387 INFO:teuthology.orchestra.run.smithi007.stdout: "172.21.15.7:6800/1013061071": "2024-09-19T07:57:57.346149+0000", 2024-09-18T08:13:14.387 INFO:teuthology.orchestra.run.smithi007.stdout: "172.21.15.7:6800/1491061076": "2024-09-19T08:03:17.052903+0000", 2024-09-18T08:13:14.387 INFO:teuthology.orchestra.run.smithi007.stdout: "172.21.15.7:0/533727535": "2024-09-19T07:57:57.346149+0000", 2024-09-18T08:13:14.387 INFO:teuthology.orchestra.run.smithi007.stdout: "172.21.15.7:0/1605217610": "2024-09-19T08:03:17.052903+0000", 2024-09-18T08:13:14.387 INFO:teuthology.orchestra.run.smithi007.stdout: "172.21.15.7:0/1722968065": "2024-09-19T08:03:17.052903+0000", 2024-09-18T08:13:14.387 INFO:teuthology.orchestra.run.smithi007.stdout: "172.21.15.7:0/1277549846": "2024-09-19T07:57:57.346149+0000", 2024-09-18T08:13:14.387 INFO:teuthology.orchestra.run.smithi007.stdout: "172.21.15.7:0/232452890": "2024-09-19T08:03:17.052903+0000", 2024-09-18T08:13:14.387 INFO:teuthology.orchestra.run.smithi007.stdout: "172.21.15.7:0/3881670470": "2024-09-19T08:03:17.052903+0000" 2024-09-18T08:13:14.387 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:13:14.387 INFO:teuthology.orchestra.run.smithi007.stdout: "range_blocklist": {}, 2024-09-18T08:13:14.387 INFO:teuthology.orchestra.run.smithi007.stdout: "erasure_code_profiles": { 2024-09-18T08:13:14.388 INFO:teuthology.orchestra.run.smithi007.stdout: "default": { 2024-09-18T08:13:14.388 INFO:teuthology.orchestra.run.smithi007.stdout: "crush-failure-domain": "osd", 2024-09-18T08:13:14.388 INFO:teuthology.orchestra.run.smithi007.stdout: "k": "2", 2024-09-18T08:13:14.388 INFO:teuthology.orchestra.run.smithi007.stdout: "m": "1", 2024-09-18T08:13:14.388 INFO:teuthology.orchestra.run.smithi007.stdout: "plugin": "jerasure", 2024-09-18T08:13:14.388 INFO:teuthology.orchestra.run.smithi007.stdout: "technique": "reed_sol_van" 2024-09-18T08:13:14.388 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-09-18T08:13:14.388 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:13:14.388 INFO:teuthology.orchestra.run.smithi007.stdout: "removed_snaps_queue": [], 2024-09-18T08:13:14.388 INFO:teuthology.orchestra.run.smithi007.stdout: "new_removed_snaps": [], 2024-09-18T08:13:14.388 INFO:teuthology.orchestra.run.smithi007.stdout: "new_purged_snaps": [], 2024-09-18T08:13:14.388 INFO:teuthology.orchestra.run.smithi007.stdout: "crush_node_flags": {}, 2024-09-18T08:13:14.388 INFO:teuthology.orchestra.run.smithi007.stdout: "device_class_flags": {}, 2024-09-18T08:13:14.389 INFO:teuthology.orchestra.run.smithi007.stdout: "stretch_mode": { 2024-09-18T08:13:14.389 INFO:teuthology.orchestra.run.smithi007.stdout: "stretch_mode_enabled": false, 2024-09-18T08:13:14.389 INFO:teuthology.orchestra.run.smithi007.stdout: "stretch_bucket_count": 0, 2024-09-18T08:13:14.389 INFO:teuthology.orchestra.run.smithi007.stdout: "degraded_stretch_mode": 0, 2024-09-18T08:13:14.389 INFO:teuthology.orchestra.run.smithi007.stdout: "recovering_stretch_mode": 0, 2024-09-18T08:13:14.389 INFO:teuthology.orchestra.run.smithi007.stdout: "stretch_mode_bucket": 0 2024-09-18T08:13:14.389 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-09-18T08:13:14.389 INFO:teuthology.orchestra.run.smithi007.stdout:} 2024-09-18T08:13:14.402 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:13:14.402 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:13:14.403 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7' 2024-09-18T08:13:14.504 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4 2024-09-18T08:13:14.548 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:13:14.549 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:13:14.550 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4' 2024-09-18T08:13:14.651 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7 2024-09-18T08:13:14.687 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:13:14.688 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:13:14.689 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7' 2024-09-18T08:13:14.724 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:13:14 smithi007 ceph-mon[29459]: pgmap v504: 158 pgs: 30 active+undersized, 34 active+undersized+degraded, 94 active+clean; 6.6 GiB data, 21 GiB used, 695 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 904 op/s; 61165/324216 objects degraded (18.866%) 2024-09-18T08:13:14.724 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:13:14 smithi007 ceph-mon[29459]: from='client.? 172.21.15.7:0/813771709' entity='client.admin' cmd={"prefix": "osd dump", "format": "json-pretty"} : dispatch 2024-09-18T08:13:14.724 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:13:14 smithi007 ceph-mon[29459]: Health check update: Degraded data redundancy: 61165/324216 objects degraded (18.866%), 34 pgs degraded, 64 pgs undersized (PG_DEGRADED) 2024-09-18T08:13:14.724 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:13:14 smithi007 ceph-mon[29459]: osdmap e277: 8 total, 7 up, 8 in 2024-09-18T08:13:14.724 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:13:14 smithi007 ceph-mon[39249]: pgmap v504: 158 pgs: 30 active+undersized, 34 active+undersized+degraded, 94 active+clean; 6.6 GiB data, 21 GiB used, 695 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 904 op/s; 61165/324216 objects degraded (18.866%) 2024-09-18T08:13:14.724 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:13:14 smithi007 ceph-mon[39249]: from='client.? 172.21.15.7:0/813771709' entity='client.admin' cmd={"prefix": "osd dump", "format": "json-pretty"} : dispatch 2024-09-18T08:13:14.725 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:13:14 smithi007 ceph-mon[39249]: Health check update: Degraded data redundancy: 61165/324216 objects degraded (18.866%), 34 pgs degraded, 64 pgs undersized (PG_DEGRADED) 2024-09-18T08:13:14.725 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:13:14 smithi007 ceph-mon[39249]: osdmap e277: 8 total, 7 up, 8 in 2024-09-18T08:13:14.789 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6 2024-09-18T08:13:14.826 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:13:14.827 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:13:14.827 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6' 2024-09-18T08:13:14.906 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:13:14 smithi050 ceph-mon[32371]: pgmap v504: 158 pgs: 30 active+undersized, 34 active+undersized+degraded, 94 active+clean; 6.6 GiB data, 21 GiB used, 695 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 904 op/s; 61165/324216 objects degraded (18.866%) 2024-09-18T08:13:14.906 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:13:14 smithi050 ceph-mon[32371]: from='client.? 172.21.15.7:0/813771709' entity='client.admin' cmd={"prefix": "osd dump", "format": "json-pretty"} : dispatch 2024-09-18T08:13:14.906 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:13:14 smithi050 ceph-mon[32371]: Health check update: Degraded data redundancy: 61165/324216 objects degraded (18.866%), 34 pgs degraded, 64 pgs undersized (PG_DEGRADED) 2024-09-18T08:13:14.906 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:13:14 smithi050 ceph-mon[32371]: osdmap e277: 8 total, 7 up, 8 in 2024-09-18T08:13:14.928 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4 2024-09-18T08:13:14.967 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:13:14.968 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:13:14.969 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4' 2024-09-18T08:13:14.991 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/779349ca-7593-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T08:13:15.070 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5 2024-09-18T08:13:15.106 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:13:15.107 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:13:15.107 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5' 2024-09-18T08:13:15.208 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0 2024-09-18T08:13:15.239 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:13:15.239 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:13:15.240 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0' 2024-09-18T08:13:15.341 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2 2024-09-18T08:13:15.370 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:13:15.371 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:13:15.372 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2' 2024-09-18T08:13:15.472 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0 2024-09-18T08:13:15.511 INFO:tasks.thrashosds.thrasher:j is {'epoch': 276, 'fsid': '779349ca-7593-11ef-bceb-c7b262605968', 'created': '2024-09-18T07:57:09.295710+0000', 'modified': '2024-09-18T08:13:10.525594+0000', 'last_up_change': '2024-09-18T08:11:41.300261+0000', 'last_in_change': '2024-09-18T08:08:17.483129+0000', 'flags': 'noscrub,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit', 'flags_num': 5801984, 'flags_set': ['noscrub', 'pglog_hardlimit', 'purged_snapdirs', 'recovery_deletes', 'sortbitwise'], 'crush_version': 18, 'full_ratio': 0.949999988079071, 'backfillfull_ratio': 0.8999999761581421, 'nearfull_ratio': 0.8500000238418579, 'cluster_snapshot': '', 'pool_max': 9, 'max_osd': 8, 'require_min_compat_client': 'luminous', 'min_compat_client': 'luminous', 'require_osd_release': 'squid', 'allow_crimson': False, 'pools': [{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-09-18T08:00:11.802627+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'is_stretch_pool': False, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 11, 'pg_placement_num': 11, 'pg_placement_num_target': 11, 'pg_num_target': 11, 'pg_num_pending': 11, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '154', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '152', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.909999966621399, 'score_stable': 1.909999966621399, 'optimal_score': 1, 'raw_score_acting': 1.909999966621399, 'raw_score_stable': 1.909999966621399, 'primary_affinity_weighted': 0.8412818908691406, 'average_primary_affinity': 0.9599999785423279, 'average_primary_affinity_weighted': 0.8399999737739563}}, {'pool': 2, 'pool_name': 'datapool', 'create_time': '2024-09-18T08:02:45.374099+0000', 'flags': 8193, 'flags_names': 'hashpspool,selfmanaged_snaps', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'is_stretch_pool': False, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 3, 'pg_placement_num': 3, 'pg_placement_num_target': 3, 'pg_num_target': 3, 'pg_num_pending': 3, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '64', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 2, 'snap_epoch': 64, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.3299999237060547, 'score_stable': 2.3299999237060547, 'optimal_score': 0.8600000143051147, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 0.8412818908691406, 'average_primary_affinity': 0.9599999785423279, 'average_primary_affinity_weighted': 0.8399999737739563}}, {'pool': 3, 'pool_name': '.rgw.root', 'create_time': '2024-09-18T08:02:46.319947+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'is_stretch_pool': False, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '63', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rgw': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5299999713897705, 'score_stable': 1.5299999713897705, 'optimal_score': 1, 'raw_score_acting': 1.5299999713897705, 'raw_score_stable': 1.5299999713897705, 'primary_affinity_weighted': 0.8412818908691406, 'average_primary_affinity': 0.9599999785423279, 'average_primary_affinity_weighted': 0.8399999737739563}}, {'pool': 4, 'pool_name': 'default.rgw.log', 'create_time': '2024-09-18T08:02:47.948552+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'is_stretch_pool': False, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '65', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rgw': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 9.609999656677246, 'score_stable': 2.190000057220459, 'optimal_score': 1, 'raw_score_acting': 9.609999656677246, 'raw_score_stable': 2.190000057220459, 'primary_affinity_weighted': 0.8412818908691406, 'average_primary_affinity': 0.9599999785423279, 'average_primary_affinity_weighted': 0.8399999737739563, 'error_message': 'pool 4 has acting primaries on OSD(s) with primary affinity 0, read balance score is not accurate'}}, {'pool': 5, 'pool_name': 'default.rgw.control', 'create_time': '2024-09-18T08:02:49.904302+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'is_stretch_pool': False, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '67', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rgw': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.75, 'score_stable': 1.75, 'optimal_score': 1, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 0.8412818908691406, 'average_primary_affinity': 0.9599999785423279, 'average_primary_affinity_weighted': 0.8399999737739563}}, {'pool': 6, 'pool_name': 'default.rgw.meta', 'create_time': '2024-09-18T08:02:51.971465+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'is_stretch_pool': False, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '70', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4}, 'application_metadata': {'rgw': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5299999713897705, 'score_stable': 1.5299999713897705, 'optimal_score': 1, 'raw_score_acting': 1.5299999713897705, 'raw_score_stable': 1.5299999713897705, 'primary_affinity_weighted': 0.8412818908691406, 'average_primary_affinity': 0.9599999785423279, 'average_primary_affinity_weighted': 0.8399999737739563}}, {'pool': 9, 'pool_name': 'unique_pool_2', 'create_time': '2024-09-18T08:11:08.258268+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'is_stretch_pool': False, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 16, 'pg_placement_num': 16, 'pg_placement_num_target': 16, 'pg_num_target': 16, 'pg_num_pending': 16, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '236', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rados': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.75, 'score_stable': 1.75, 'optimal_score': 1, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 0.8412818908691406, 'average_primary_affinity': 0.9599999785423279, 'average_primary_affinity_weighted': 0.8399999737739563}}], 'osds': [{'osd': 0, 'uuid': 'ae8f60df-72ac-46fb-91a1-f132b851c876', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 8, 'up_thru': 244, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.7:6801', 'nonce': 2896425794}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.7:6802', 'nonce': 2896425794}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.7:6804', 'nonce': 2896425794}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.7:6803', 'nonce': 2896425794}]}, 'public_addr': '172.21.15.7:6801/2896425794', 'cluster_addr': '172.21.15.7:6802/2896425794', 'heartbeat_back_addr': '172.21.15.7:6804/2896425794', 'heartbeat_front_addr': '172.21.15.7:6803/2896425794', 'state': ['exists', 'up']}, {'osd': 1, 'uuid': 'be28b70c-1f73-494f-ad4b-cb7ec5311999', 'up': 0, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 13, 'up_thru': 232, 'down_at': 244, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.7:6805', 'nonce': 1971397549}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.7:6806', 'nonce': 1971397549}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.7:6808', 'nonce': 1971397549}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.7:6807', 'nonce': 1971397549}]}, 'public_addr': '172.21.15.7:6805/1971397549', 'cluster_addr': '172.21.15.7:6806/1971397549', 'heartbeat_back_addr': '172.21.15.7:6808/1971397549', 'heartbeat_front_addr': '172.21.15.7:6807/1971397549', 'state': ['exists']}, {'osd': 2, 'uuid': 'cfa4f083-c7fd-4743-be65-84a3040fe6dc', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 18, 'up_thru': 244, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.7:6809', 'nonce': 556142559}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.7:6810', 'nonce': 556142559}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.7:6812', 'nonce': 556142559}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.7:6811', 'nonce': 556142559}]}, 'public_addr': '172.21.15.7:6809/556142559', 'cluster_addr': '172.21.15.7:6810/556142559', 'heartbeat_back_addr': '172.21.15.7:6812/556142559', 'heartbeat_front_addr': '172.21.15.7:6811/556142559', 'state': ['exists', 'up']}, {'osd': 3, 'uuid': '11ecd7e1-a751-4b75-95f5-00b32d2a5bc4', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 26, 'up_thru': 244, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.7:6813', 'nonce': 163587772}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.7:6814', 'nonce': 163587772}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.7:6816', 'nonce': 163587772}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.7:6815', 'nonce': 163587772}]}, 'public_addr': '172.21.15.7:6813/163587772', 'cluster_addr': '172.21.15.7:6814/163587772', 'heartbeat_back_addr': '172.21.15.7:6816/163587772', 'heartbeat_front_addr': '172.21.15.7:6815/163587772', 'state': ['exists', 'up']}, {'osd': 4, 'uuid': '98992b34-cfa1-45bf-97a1-7d557739252f', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 32, 'up_thru': 244, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.50:6800', 'nonce': 3662119805}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.50:6801', 'nonce': 3662119805}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.50:6803', 'nonce': 3662119805}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.50:6802', 'nonce': 3662119805}]}, 'public_addr': '172.21.15.50:6800/3662119805', 'cluster_addr': '172.21.15.50:6801/3662119805', 'heartbeat_back_addr': '172.21.15.50:6803/3662119805', 'heartbeat_front_addr': '172.21.15.50:6802/3662119805', 'state': ['exists', 'up']}, {'osd': 5, 'uuid': '7f290005-ebca-467b-87e4-02ead1063610', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 39, 'up_thru': 244, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.50:6804', 'nonce': 1199797954}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.50:6805', 'nonce': 1199797954}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.50:6807', 'nonce': 1199797954}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.50:6806', 'nonce': 1199797954}]}, 'public_addr': '172.21.15.50:6804/1199797954', 'cluster_addr': '172.21.15.50:6805/1199797954', 'heartbeat_back_addr': '172.21.15.50:6807/1199797954', 'heartbeat_front_addr': '172.21.15.50:6806/1199797954', 'state': ['exists', 'up']}, {'osd': 6, 'uuid': 'e07d846c-ae42-4fea-aa5c-3a25835d18a6', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 0, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 47, 'up_thru': 244, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.50:6808', 'nonce': 255960422}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.50:6809', 'nonce': 255960422}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.50:6811', 'nonce': 255960422}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.50:6810', 'nonce': 255960422}]}, 'public_addr': '172.21.15.50:6808/255960422', 'cluster_addr': '172.21.15.50:6809/255960422', 'heartbeat_back_addr': '172.21.15.50:6811/255960422', 'heartbeat_front_addr': '172.21.15.50:6810/255960422', 'state': ['exists', 'up']}, {'osd': 7, 'uuid': 'c9dac3b7-d651-489d-b96e-5b10fa32d9af', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 0.730255126953125, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 56, 'up_thru': 244, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.50:6812', 'nonce': 874923904}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.50:6813', 'nonce': 874923904}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.50:6815', 'nonce': 874923904}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.50:6814', 'nonce': 874923904}]}, 'public_addr': '172.21.15.50:6812/874923904', 'cluster_addr': '172.21.15.50:6813/874923904', 'heartbeat_back_addr': '172.21.15.50:6815/874923904', 'heartbeat_front_addr': '172.21.15.50:6814/874923904', 'state': ['exists', 'up']}], 'osd_xinfo': [{'osd': 0, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540719139924082687, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-18T08:12:36.340648+0000', 'dead_epoch': 0}, {'osd': 1, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540719139924082687, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-18T08:10:56.667957+0000', 'dead_epoch': 243}, {'osd': 2, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540719139924082687, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-18T08:12:30.885757+0000', 'dead_epoch': 0}, {'osd': 3, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540719139924082687, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-18T08:13:01.289441+0000', 'dead_epoch': 0}, {'osd': 4, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540719139924082687, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-18T08:13:10.281294+0000', 'dead_epoch': 0}, {'osd': 5, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540719139924082687, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-18T08:12:19.708001+0000', 'dead_epoch': 0}, {'osd': 6, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540719139924082687, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-18T08:13:03.684975+0000', 'dead_epoch': 0}, {'osd': 7, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540719139924082687, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-18T08:12:42.349589+0000', 'dead_epoch': 0}], 'pg_upmap': [], 'pg_upmap_items': [{'pgid': '4.7', 'mappings': [{'from': 1, 'to': 2}]}, {'pgid': '4.9', 'mappings': [{'from': 1, 'to': 5}]}, {'pgid': '4.14', 'mappings': [{'from': 1, 'to': 2}]}], 'pg_upmap_primaries': [], 'pg_temp': [], 'primary_temp': [], 'blocklist': {'172.21.15.7:0/398367658': '2024-09-19T08:03:17.052903+0000', '172.21.15.7:6800/2918496291': '2024-09-19T07:57:31.015269+0000', '172.21.15.7:0/2514354773': '2024-09-19T07:57:31.015269+0000', '172.21.15.7:0/3924911335': '2024-09-19T07:57:31.015269+0000', '172.21.15.7:0/1668598084': '2024-09-19T07:57:57.346149+0000', '172.21.15.7:0/4272027292': '2024-09-19T07:57:31.015269+0000', '172.21.15.7:6800/1013061071': '2024-09-19T07:57:57.346149+0000', '172.21.15.7:6800/1491061076': '2024-09-19T08:03:17.052903+0000', '172.21.15.7:0/533727535': '2024-09-19T07:57:57.346149+0000', '172.21.15.7:0/1605217610': '2024-09-19T08:03:17.052903+0000', '172.21.15.7:0/1722968065': '2024-09-19T08:03:17.052903+0000', '172.21.15.7:0/1277549846': '2024-09-19T07:57:57.346149+0000', '172.21.15.7:0/232452890': '2024-09-19T08:03:17.052903+0000', '172.21.15.7:0/3881670470': '2024-09-19T08:03:17.052903+0000'}, 'range_blocklist': {}, 'erasure_code_profiles': {'default': {'crush-failure-domain': 'osd', 'k': '2', 'm': '1', 'plugin': 'jerasure', 'technique': 'reed_sol_van'}}, 'removed_snaps_queue': [], 'new_removed_snaps': [], 'new_purged_snaps': [], 'crush_node_flags': {}, 'device_class_flags': {}, 'stretch_mode': {'stretch_mode_enabled': False, 'stretch_bucket_count': 0, 'degraded_stretch_mode': 0, 'recovering_stretch_mode': 0, 'stretch_mode_bucket': 0}} 2024-09-18T08:13:15.511 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid 779349ca-7593-11ef-bceb-c7b262605968 -- ceph pg dump --format=json 2024-09-18T08:13:15.512 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:13:15.513 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:13:15.513 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0' 2024-09-18T08:13:15.560 INFO:teuthology.orchestra.run.smithi007.stderr:nodeep-scrub is set 2024-09-18T08:13:15.614 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4 2024-09-18T08:13:15.654 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:13:15.655 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:13:15.655 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4' 2024-09-18T08:13:15.730 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:13:15 smithi007 ceph-mon[29459]: from='client.? 172.21.15.7:0/2695463532' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-18T08:13:15.730 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:13:15 smithi007 ceph-mon[39249]: from='client.? 172.21.15.7:0/2695463532' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-18T08:13:15.756 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2 2024-09-18T08:13:15.810 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:13:15.811 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:13:15.812 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2' 2024-09-18T08:13:15.905 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:13:15 smithi050 ceph-mon[32371]: from='client.? 172.21.15.7:0/2695463532' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-18T08:13:15.912 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3 2024-09-18T08:13:15.951 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:13:15.952 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:13:15.953 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3' 2024-09-18T08:13:16.053 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4 2024-09-18T08:13:16.091 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:13:16.092 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:13:16.092 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4' 2024-09-18T08:13:16.193 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4 2024-09-18T08:13:16.228 INFO:teuthology.orchestra.run.smithi007.stdout:ERROR: (22) Invalid argument 2024-09-18T08:13:16.228 INFO:teuthology.orchestra.run.smithi007.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-18T08:13:16.231 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:13:16.232 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:13:16.232 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4' 2024-09-18T08:13:16.333 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0 2024-09-18T08:13:16.365 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:13:16.365 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:13:16.366 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0' 2024-09-18T08:13:16.466 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7 2024-09-18T08:13:16.500 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:13:16.501 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:13:16.502 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7' 2024-09-18T08:13:16.602 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3 2024-09-18T08:13:16.634 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:13:16.635 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:13:16.635 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3' 2024-09-18T08:13:16.736 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4 2024-09-18T08:13:16.759 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:13:16.759 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:13:16.759 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4' 2024-09-18T08:13:16.860 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6 2024-09-18T08:13:16.898 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:13:16.899 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:13:16.899 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6' 2024-09-18T08:13:16.905 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:13:16 smithi050 ceph-mon[32371]: pgmap v506: 158 pgs: 30 active+undersized, 34 active+undersized+degraded, 94 active+clean; 6.1 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 852 B/s rd, 0 B/s wr, 1.21k op/s; 57071/302514 objects degraded (18.866%) 2024-09-18T08:13:16.905 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:13:16 smithi050 ceph-mon[32371]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-18T08:13:16.905 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:13:16 smithi050 ceph-mon[32371]: from='client.? 172.21.15.7:0/2695463532' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-18T08:13:16.905 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:13:16 smithi050 ceph-mon[32371]: osdmap e278: 8 total, 7 up, 8 in 2024-09-18T08:13:16.980 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Sep 18 08:13:16 smithi007 ceph-779349ca-7593-11ef-bceb-c7b262605968-mgr-y[29688]: ::ffff:172.21.15.50 - - [18/Sep/2024:08:13:16] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-09-18T08:13:16.980 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:13:16 smithi007 ceph-mon[29459]: pgmap v506: 158 pgs: 30 active+undersized, 34 active+undersized+degraded, 94 active+clean; 6.1 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 852 B/s rd, 0 B/s wr, 1.21k op/s; 57071/302514 objects degraded (18.866%) 2024-09-18T08:13:16.981 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:13:16 smithi007 ceph-mon[29459]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-18T08:13:16.981 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:13:16 smithi007 ceph-mon[29459]: from='client.? 172.21.15.7:0/2695463532' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-18T08:13:16.981 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:13:16 smithi007 ceph-mon[29459]: osdmap e278: 8 total, 7 up, 8 in 2024-09-18T08:13:16.981 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:13:16 smithi007 ceph-mon[39249]: pgmap v506: 158 pgs: 30 active+undersized, 34 active+undersized+degraded, 94 active+clean; 6.1 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 852 B/s rd, 0 B/s wr, 1.21k op/s; 57071/302514 objects degraded (18.866%) 2024-09-18T08:13:16.981 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:13:16 smithi007 ceph-mon[39249]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-18T08:13:16.981 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:13:16 smithi007 ceph-mon[39249]: from='client.? 172.21.15.7:0/2695463532' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-18T08:13:16.981 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:13:16 smithi007 ceph-mon[39249]: osdmap e278: 8 total, 7 up, 8 in 2024-09-18T08:13:17.001 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2 2024-09-18T08:13:17.030 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:13:17.031 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:13:17.032 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2' 2024-09-18T08:13:17.132 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5 2024-09-18T08:13:17.165 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:13:17.165 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:13:17.166 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5' 2024-09-18T08:13:17.266 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2 2024-09-18T08:13:17.297 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:13:17.297 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:13:17.298 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2' 2024-09-18T08:13:17.398 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2 2024-09-18T08:13:17.430 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:13:17.431 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:13:17.431 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2' 2024-09-18T08:13:17.532 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5 2024-09-18T08:13:17.569 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:13:17.570 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:13:17.571 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5' 2024-09-18T08:13:17.671 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7 2024-09-18T08:13:17.711 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:13:17.712 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:13:17.712 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7' 2024-09-18T08:13:17.813 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4 2024-09-18T08:13:17.854 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:13:17.854 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:13:17.854 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4' 2024-09-18T08:13:17.955 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4 2024-09-18T08:13:17.993 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:13:17.994 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:13:17.994 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4' 2024-09-18T08:13:18.095 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3 2024-09-18T08:13:18.126 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:13:18.127 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:13:18.127 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3' 2024-09-18T08:13:18.228 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4 2024-09-18T08:13:18.265 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:13:18.266 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:13:18.266 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4' 2024-09-18T08:13:18.367 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6 2024-09-18T08:13:18.404 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:13:18.405 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:13:18.405 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6' 2024-09-18T08:13:18.506 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3 2024-09-18T08:13:18.539 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:13:18.539 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:13:18.540 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3' 2024-09-18T08:13:18.641 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3 2024-09-18T08:13:18.676 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:13:18.677 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:13:18.678 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3' 2024-09-18T08:13:18.752 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/779349ca-7593-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T08:13:18.779 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0 2024-09-18T08:13:18.812 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:13:18.813 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:13:18.814 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0' 2024-09-18T08:13:18.840 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:13:18 smithi007 ceph-mon[29459]: pgmap v508: 158 pgs: 30 active+undersized, 34 active+undersized+degraded, 94 active+clean; 6.1 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 756 B/s rd, 0 B/s wr, 1.07k op/s; 57071/302514 objects degraded (18.866%) 2024-09-18T08:13:18.840 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:13:18 smithi007 ceph-mon[29459]: from='mgr.24470 172.21.15.7:0/2594390261' entity='mgr.y' 2024-09-18T08:13:18.840 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:13:18 smithi007 ceph-mon[29459]: from='mgr.24470 172.21.15.7:0/2594390261' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-18T08:13:18.840 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:13:18 smithi007 ceph-mon[39249]: pgmap v508: 158 pgs: 30 active+undersized, 34 active+undersized+degraded, 94 active+clean; 6.1 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 756 B/s rd, 0 B/s wr, 1.07k op/s; 57071/302514 objects degraded (18.866%) 2024-09-18T08:13:18.840 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:13:18 smithi007 ceph-mon[39249]: from='mgr.24470 172.21.15.7:0/2594390261' entity='mgr.y' 2024-09-18T08:13:18.841 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:13:18 smithi007 ceph-mon[39249]: from='mgr.24470 172.21.15.7:0/2594390261' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-18T08:13:18.906 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:13:18 smithi050 ceph-mon[32371]: pgmap v508: 158 pgs: 30 active+undersized, 34 active+undersized+degraded, 94 active+clean; 6.1 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 756 B/s rd, 0 B/s wr, 1.07k op/s; 57071/302514 objects degraded (18.866%) 2024-09-18T08:13:18.906 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:13:18 smithi050 ceph-mon[32371]: from='mgr.24470 172.21.15.7:0/2594390261' entity='mgr.y' 2024-09-18T08:13:18.906 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:13:18 smithi050 ceph-mon[32371]: from='mgr.24470 172.21.15.7:0/2594390261' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-18T08:13:18.915 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3 2024-09-18T08:13:18.945 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:13:18.947 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:13:18.948 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3' 2024-09-18T08:13:19.048 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5 2024-09-18T08:13:19.081 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:13:19.082 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:13:19.083 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5' 2024-09-18T08:13:19.183 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3 2024-09-18T08:13:19.216 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid 779349ca-7593-11ef-bceb-c7b262605968 -- ceph daemon osd.3 dump_blocked_ops 2024-09-18T08:13:19.238 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:13:19.239 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:13:19.240 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3' 2024-09-18T08:13:19.341 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3 2024-09-18T08:13:19.389 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:13:19.389 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:13:19.390 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3' 2024-09-18T08:13:19.491 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4 2024-09-18T08:13:19.527 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:13:19.527 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:13:19.528 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4' 2024-09-18T08:13:19.587 INFO:journalctl@ceph.iscsi.iscsi.a.smithi050.stdout:Sep 18 08:13:19 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-iscsi-iscsi-a[57782]: debug there is no tcmu-runner data available 2024-09-18T08:13:19.629 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4 2024-09-18T08:13:19.668 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:13:19.669 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:13:19.669 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4' 2024-09-18T08:13:19.770 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4 2024-09-18T08:13:19.790 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/779349ca-7593-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T08:13:19.810 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:13:19.811 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:13:19.811 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4' 2024-09-18T08:13:19.821 INFO:teuthology.orchestra.run.smithi007.stdout:osd.0: {} 2024-09-18T08:13:19.821 INFO:teuthology.orchestra.run.smithi007.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-18T08:13:19.824 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:13:19 smithi007 ceph-mon[29459]: Health check update: Degraded data redundancy: 57071/302514 objects degraded (18.866%), 34 pgs degraded, 64 pgs undersized (PG_DEGRADED) 2024-09-18T08:13:19.824 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:13:19 smithi007 ceph-mon[29459]: osdmap e279: 8 total, 7 up, 8 in 2024-09-18T08:13:19.824 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:13:19 smithi007 ceph-mon[39249]: Health check update: Degraded data redundancy: 57071/302514 objects degraded (18.866%), 34 pgs degraded, 64 pgs undersized (PG_DEGRADED) 2024-09-18T08:13:19.824 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:13:19 smithi007 ceph-mon[39249]: osdmap e279: 8 total, 7 up, 8 in 2024-09-18T08:13:19.826 INFO:teuthology.orchestra.run.smithi007.stderr:Error ENXIO: problem getting command descriptions from osd.1 2024-09-18T08:13:19.869 INFO:teuthology.orchestra.run.smithi007.stdout:osd.2: {} 2024-09-18T08:13:19.870 INFO:teuthology.orchestra.run.smithi007.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-18T08:13:19.901 INFO:teuthology.orchestra.run.smithi007.stdout:osd.3: {} 2024-09-18T08:13:19.901 INFO:teuthology.orchestra.run.smithi007.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-18T08:13:19.905 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:13:19 smithi050 ceph-mon[32371]: Health check update: Degraded data redundancy: 57071/302514 objects degraded (18.866%), 34 pgs degraded, 64 pgs undersized (PG_DEGRADED) 2024-09-18T08:13:19.905 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:13:19 smithi050 ceph-mon[32371]: osdmap e279: 8 total, 7 up, 8 in 2024-09-18T08:13:19.912 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6 2024-09-18T08:13:19.930 INFO:teuthology.orchestra.run.smithi007.stdout:osd.4: {} 2024-09-18T08:13:19.930 INFO:teuthology.orchestra.run.smithi007.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-18T08:13:19.955 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:13:19.955 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:13:19.956 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6' 2024-09-18T08:13:19.959 INFO:teuthology.orchestra.run.smithi007.stdout:osd.5: {} 2024-09-18T08:13:19.959 INFO:teuthology.orchestra.run.smithi007.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-18T08:13:19.995 INFO:teuthology.orchestra.run.smithi007.stdout:osd.6: {} 2024-09-18T08:13:19.995 INFO:teuthology.orchestra.run.smithi007.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-18T08:13:20.027 INFO:teuthology.orchestra.run.smithi007.stdout:osd.7: {} 2024-09-18T08:13:20.027 INFO:teuthology.orchestra.run.smithi007.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-18T08:13:20.056 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7 2024-09-18T08:13:20.091 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:13:20.091 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:13:20.092 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7' 2024-09-18T08:13:20.193 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2 2024-09-18T08:13:20.230 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:13:20.230 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:13:20.230 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2' 2024-09-18T08:13:20.331 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3 2024-09-18T08:13:20.362 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:13:20.363 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:13:20.363 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3' 2024-09-18T08:13:20.464 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4 2024-09-18T08:13:20.501 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:13:20.502 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:13:20.502 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4' 2024-09-18T08:13:20.603 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6 2024-09-18T08:13:20.649 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:13:20.650 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:13:20.650 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6' 2024-09-18T08:13:20.707 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid 779349ca-7593-11ef-bceb-c7b262605968 -- ceph osd unset noscrub 2024-09-18T08:13:20.730 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:13:20 smithi007 ceph-mon[29459]: pgmap v509: 158 pgs: 30 active+undersized, 34 active+undersized+degraded, 94 active+clean; 5.7 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.80k op/s; 53030/280920 objects degraded (18.877%) 2024-09-18T08:13:20.730 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:13:20 smithi007 ceph-mon[29459]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-18T08:13:20.730 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:13:20 smithi007 ceph-mon[29459]: from='client.? 172.21.15.7:0/2122318243' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-18T08:13:20.730 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:13:20 smithi007 ceph-mon[39249]: pgmap v509: 158 pgs: 30 active+undersized, 34 active+undersized+degraded, 94 active+clean; 5.7 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.80k op/s; 53030/280920 objects degraded (18.877%) 2024-09-18T08:13:20.730 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:13:20 smithi007 ceph-mon[39249]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-18T08:13:20.731 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:13:20 smithi007 ceph-mon[39249]: from='client.? 172.21.15.7:0/2122318243' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-18T08:13:20.751 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4 2024-09-18T08:13:20.790 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:13:20.791 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:13:20.791 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4' 2024-09-18T08:13:20.892 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5 2024-09-18T08:13:20.905 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:13:20 smithi050 ceph-mon[32371]: pgmap v509: 158 pgs: 30 active+undersized, 34 active+undersized+degraded, 94 active+clean; 5.7 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.80k op/s; 53030/280920 objects degraded (18.877%) 2024-09-18T08:13:20.905 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:13:20 smithi050 ceph-mon[32371]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-18T08:13:20.905 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:13:20 smithi050 ceph-mon[32371]: from='client.? 172.21.15.7:0/2122318243' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-18T08:13:20.913 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-09-18T08:13:20.916 INFO:teuthology.orchestra.run.smithi007.stderr:dumped all 2024-09-18T08:13:20.928 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:13:20.929 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:13:20.929 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5' 2024-09-18T08:13:21.030 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6 2024-09-18T08:13:21.067 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:13:21.067 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:13:21.067 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6' 2024-09-18T08:13:21.168 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6 2024-09-18T08:13:21.169 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-09-18T08:13:21.170 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-09-18T08:13:21.204 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:13:21.205 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:13:21.205 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6' 2024-09-18T08:13:21.306 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5 2024-09-18T08:13:21.341 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:13:21.342 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:13:21.342 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5' 2024-09-18T08:13:21.443 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3 2024-09-18T08:13:21.475 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:13:21.476 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:13:21.476 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3' 2024-09-18T08:13:21.577 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0 2024-09-18T08:13:21.608 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:13:21.609 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:13:21.609 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0' 2024-09-18T08:13:21.710 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0 2024-09-18T08:13:21.739 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:13:21.740 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:13:21.741 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0' 2024-09-18T08:13:21.841 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0 2024-09-18T08:13:21.853 INFO:teuthology.orchestra.run.smithi007.stdout:{"pg_ready":true,"pg_map":{"version":510,"stamp":"2024-09-18T08:13:19.312016+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":6124334041,"num_objects":93640,"num_object_clones":0,"num_object_copies":280920,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":53030,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":93640,"num_whiteouts":0,"num_read":1248,"num_read_kb":1010,"num_write":191058,"num_write_kb":9080642,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":204,"num_bytes_recovered":125726,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":81382,"ondisk_log_size":81382,"up":410,"acting":410,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":474,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":19666760,"kb_used_data":16462664,"kb_used_omap":12,"kb_used_meta":3203571,"kb_avail":730294456,"statfs":{"total":767960285184,"available":747821522944,"internally_reserved":0,"allocated":16857767936,"data_stored":16852906323,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12750,"internal_metadata":3280457266},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-945815552,"num_objects":-14432,"num_object_clones":0,"num_object_copies":-43296,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-8135,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-14432,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":14432,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.011894"},"pg_stats":[{"pgid":"3.1f","version":"0'0","reported_seq":486,"reported_epoch":278,"state":"active+clean","last_fresh":"2024-09-18T08:13:15.564770+0000","last_change":"2024-09-18T08:10:15.850596+0000","last_active":"2024-09-18T08:13:15.564770+0000","last_peered":"2024-09-18T08:13:15.564770+0000","last_clean":"2024-09-18T08:13:15.564770+0000","last_became_active":"2024-09-18T08:08:52.697499+0000","last_became_peered":"2024-09-18T08:08:52.697499+0000","last_unstale":"2024-09-18T08:13:15.564770+0000","last_undegraded":"2024-09-18T08:13:15.564770+0000","last_fullsized":"2024-09-18T08:13:15.564770+0000","mapping_epoch":198,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":199,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:10:15.850515+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:46.860092+0000","last_clean_scrub_stamp":"2024-09-18T08:10:15.850515+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:13:20.679917+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"271'15","reported_seq":237,"reported_epoch":278,"state":"active+undersized+degraded","last_fresh":"2024-09-18T08:13:15.564614+0000","last_change":"2024-09-18T08:11:42.323251+0000","last_active":"2024-09-18T08:13:15.564614+0000","last_peered":"2024-09-18T08:13:15.564614+0000","last_clean":"2024-09-18T08:11:40.306776+0000","last_became_active":"2024-09-18T08:11:42.323251+0000","last_became_peered":"2024-09-18T08:11:42.323251+0000","last_unstale":"2024-09-18T08:13:15.564614+0000","last_undegraded":"2024-09-18T08:11:42.318308+0000","last_fullsized":"2024-09-18T08:11:42.318179+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":182,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'9","last_scrub_stamp":"2024-09-18T08:10:19.271241+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_clean_scrub_stamp":"2024-09-18T08:10:19.271241+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":25,"num_read_kb":19,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":["4","6"],"object_location_counts":[{"shards":"4,6","objects":4}],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":76,"reported_epoch":278,"state":"active+undersized","last_fresh":"2024-09-18T08:13:15.571457+0000","last_change":"2024-09-18T08:11:42.331640+0000","last_active":"2024-09-18T08:13:15.571457+0000","last_peered":"2024-09-18T08:13:15.571457+0000","last_clean":"2024-09-18T08:02:50.886830+0000","last_became_active":"2024-09-18T08:11:42.331640+0000","last_became_peered":"2024-09-18T08:11:42.331640+0000","last_unstale":"2024-09-18T08:13:15.571457+0000","last_undegraded":"2024-09-18T08:13:15.571457+0000","last_fullsized":"2024-09-18T08:11:42.326296+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:10:15.603289+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:50.886830+0000","last_clean_scrub_stamp":"2024-09-18T08:10:15.603289+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:14:13.131588+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":["5","7"],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":210,"reported_epoch":278,"state":"active+undersized","last_fresh":"2024-09-18T08:13:15.566136+0000","last_change":"2024-09-18T08:11:42.333493+0000","last_active":"2024-09-18T08:13:15.566136+0000","last_peered":"2024-09-18T08:13:15.566136+0000","last_clean":"2024-09-18T08:11:40.307445+0000","last_became_active":"2024-09-18T08:11:42.333493+0000","last_became_peered":"2024-09-18T08:11:42.333493+0000","last_unstale":"2024-09-18T08:13:15.566136+0000","last_undegraded":"2024-09-18T08:13:15.566136+0000","last_fullsized":"2024-09-18T08:11:42.319356+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":182,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:07:08.451865+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_clean_scrub_stamp":"2024-09-18T08:07:08.451865+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:13:52.472014+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":["4","5"],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":482,"reported_epoch":278,"state":"active+clean","last_fresh":"2024-09-18T08:13:15.570264+0000","last_change":"2024-09-18T08:12:38.915577+0000","last_active":"2024-09-18T08:13:15.570264+0000","last_peered":"2024-09-18T08:13:15.570264+0000","last_clean":"2024-09-18T08:13:15.570264+0000","last_became_active":"2024-09-18T08:06:24.128511+0000","last_became_peered":"2024-09-18T08:06:24.128511+0000","last_unstale":"2024-09-18T08:13:15.570264+0000","last_undegraded":"2024-09-18T08:13:15.570264+0000","last_fullsized":"2024-09-18T08:13:15.570264+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:12:38.915501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_clean_scrub_stamp":"2024-09-18T08:12:38.915501+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:13:39.193844+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":495,"reported_epoch":278,"state":"active+clean","last_fresh":"2024-09-18T08:13:15.573233+0000","last_change":"2024-09-18T08:12:03.127882+0000","last_active":"2024-09-18T08:13:15.573233+0000","last_peered":"2024-09-18T08:13:15.573233+0000","last_clean":"2024-09-18T08:13:15.573233+0000","last_became_active":"2024-09-18T08:06:24.126194+0000","last_became_peered":"2024-09-18T08:06:24.126194+0000","last_unstale":"2024-09-18T08:13:15.573233+0000","last_undegraded":"2024-09-18T08:13:15.573233+0000","last_fullsized":"2024-09-18T08:13:15.573233+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:12:03.127810+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:46.860092+0000","last_clean_scrub_stamp":"2024-09-18T08:12:03.127810+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:13:26.206963+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"271'27","reported_seq":495,"reported_epoch":278,"state":"active+clean","last_fresh":"2024-09-18T08:13:15.570941+0000","last_change":"2024-09-18T08:09:32.575836+0000","last_active":"2024-09-18T08:13:15.570941+0000","last_peered":"2024-09-18T08:13:15.570941+0000","last_clean":"2024-09-18T08:13:15.570941+0000","last_became_active":"2024-09-18T08:02:49.886106+0000","last_became_peered":"2024-09-18T08:02:49.886106+0000","last_unstale":"2024-09-18T08:13:15.570941+0000","last_undegraded":"2024-09-18T08:13:15.570941+0000","last_fullsized":"2024-09-18T08:13:15.570941+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'15","last_scrub_stamp":"2024-09-18T08:09:32.575741+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_clean_scrub_stamp":"2024-09-18T08:09:32.575741+0000","objects_scrubbed":7,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:13:21.875835+0000","scrub_duration":40,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":44,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":210,"reported_epoch":278,"state":"active+undersized","last_fresh":"2024-09-18T08:13:15.564614+0000","last_change":"2024-09-18T08:11:42.323423+0000","last_active":"2024-09-18T08:13:15.564614+0000","last_peered":"2024-09-18T08:13:15.564614+0000","last_clean":"2024-09-18T08:11:40.306782+0000","last_became_active":"2024-09-18T08:11:42.323423+0000","last_became_peered":"2024-09-18T08:11:42.323423+0000","last_unstale":"2024-09-18T08:13:15.564614+0000","last_undegraded":"2024-09-18T08:13:15.564614+0000","last_fullsized":"2024-09-18T08:11:42.318286+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":182,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:08:09.142188+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:50.886830+0000","last_clean_scrub_stamp":"2024-09-18T08:08:09.142188+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:13:57.407915+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":["4","6"],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":499,"reported_epoch":278,"state":"active+clean","last_fresh":"2024-09-18T08:13:15.572707+0000","last_change":"2024-09-18T08:10:51.226034+0000","last_active":"2024-09-18T08:13:15.572707+0000","last_peered":"2024-09-18T08:13:15.572707+0000","last_clean":"2024-09-18T08:13:15.572707+0000","last_became_active":"2024-09-18T08:08:18.548203+0000","last_became_peered":"2024-09-18T08:08:18.548203+0000","last_unstale":"2024-09-18T08:13:15.572707+0000","last_undegraded":"2024-09-18T08:13:15.572707+0000","last_fullsized":"2024-09-18T08:13:15.572707+0000","mapping_epoch":181,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":182,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:10:51.225974+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:46.860092+0000","last_clean_scrub_stamp":"2024-09-18T08:10:51.225974+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:14:14.183168+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"271'15","reported_seq":241,"reported_epoch":278,"state":"active+clean","last_fresh":"2024-09-18T08:13:15.566367+0000","last_change":"2024-09-18T08:12:00.466295+0000","last_active":"2024-09-18T08:13:15.566367+0000","last_peered":"2024-09-18T08:13:15.566367+0000","last_clean":"2024-09-18T08:13:15.566367+0000","last_became_active":"2024-09-18T08:08:18.550960+0000","last_became_peered":"2024-09-18T08:08:18.550960+0000","last_unstale":"2024-09-18T08:13:15.566367+0000","last_undegraded":"2024-09-18T08:13:15.566367+0000","last_fullsized":"2024-09-18T08:13:15.566367+0000","mapping_epoch":181,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":182,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'9","last_scrub_stamp":"2024-09-18T08:12:00.466202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_clean_scrub_stamp":"2024-09-18T08:12:00.466202+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":25,"num_read_kb":19,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":463,"reported_epoch":278,"state":"active+clean","last_fresh":"2024-09-18T08:13:15.571030+0000","last_change":"2024-09-18T08:12:57.218495+0000","last_active":"2024-09-18T08:13:15.571030+0000","last_peered":"2024-09-18T08:13:15.571030+0000","last_clean":"2024-09-18T08:13:15.571030+0000","last_became_active":"2024-09-18T08:02:51.907296+0000","last_became_peered":"2024-09-18T08:02:51.907296+0000","last_unstale":"2024-09-18T08:13:15.571030+0000","last_undegraded":"2024-09-18T08:13:15.571030+0000","last_fullsized":"2024-09-18T08:13:15.571030+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:12:57.218286+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:50.886830+0000","last_clean_scrub_stamp":"2024-09-18T08:12:57.218286+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:14:06.566496+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":456,"reported_epoch":278,"state":"active+undersized","last_fresh":"2024-09-18T08:13:15.566052+0000","last_change":"2024-09-18T08:11:42.330079+0000","last_active":"2024-09-18T08:13:15.566052+0000","last_peered":"2024-09-18T08:13:15.566052+0000","last_clean":"2024-09-18T08:11:40.308148+0000","last_became_active":"2024-09-18T08:11:42.330079+0000","last_became_peered":"2024-09-18T08:11:42.330079+0000","last_unstale":"2024-09-18T08:13:15.566052+0000","last_undegraded":"2024-09-18T08:13:15.566052+0000","last_fullsized":"2024-09-18T08:11:42.319204+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:08:14.645993+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_clean_scrub_stamp":"2024-09-18T08:08:14.645993+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:13:55.773992+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":["0","7"],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":473,"reported_epoch":278,"state":"active+undersized","last_fresh":"2024-09-18T08:13:15.568986+0000","last_change":"2024-09-18T08:11:42.339943+0000","last_active":"2024-09-18T08:13:15.568986+0000","last_peered":"2024-09-18T08:13:15.568986+0000","last_clean":"2024-09-18T08:11:40.302512+0000","last_became_active":"2024-09-18T08:11:42.339943+0000","last_became_peered":"2024-09-18T08:11:42.339943+0000","last_unstale":"2024-09-18T08:13:15.568986+0000","last_undegraded":"2024-09-18T08:13:15.568986+0000","last_fullsized":"2024-09-18T08:11:42.322341+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":182,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:04:32.849119+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:46.860092+0000","last_clean_scrub_stamp":"2024-09-18T08:04:32.849119+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:13:48.261776+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":["5","4"],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1b","version":"271'7","reported_seq":227,"reported_epoch":278,"state":"active+undersized+degraded","last_fresh":"2024-09-18T08:13:15.565740+0000","last_change":"2024-09-18T08:11:42.329404+0000","last_active":"2024-09-18T08:13:15.565740+0000","last_peered":"2024-09-18T08:13:15.565740+0000","last_clean":"2024-09-18T08:11:40.307173+0000","last_became_active":"2024-09-18T08:11:42.329404+0000","last_became_peered":"2024-09-18T08:11:42.329404+0000","last_unstale":"2024-09-18T08:13:15.565740+0000","last_undegraded":"2024-09-18T08:11:42.319890+0000","last_fullsized":"2024-09-18T08:11:42.319791+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":182,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'5","last_scrub_stamp":"2024-09-18T08:10:52.400411+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_clean_scrub_stamp":"2024-09-18T08:10:52.400411+0000","objects_scrubbed":2,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:14:06.467689+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":13,"num_read_kb":9,"num_write":8,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":["4","3"],"object_location_counts":[{"shards":"3,4","objects":2}],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":505,"reported_epoch":278,"state":"active+undersized","last_fresh":"2024-09-18T08:13:15.568727+0000","last_change":"2024-09-18T08:11:42.329293+0000","last_active":"2024-09-18T08:13:15.568727+0000","last_peered":"2024-09-18T08:13:15.568727+0000","last_clean":"2024-09-18T08:11:40.308737+0000","last_became_active":"2024-09-18T08:11:42.329293+0000","last_became_peered":"2024-09-18T08:11:42.329293+0000","last_unstale":"2024-09-18T08:13:15.568727+0000","last_undegraded":"2024-09-18T08:13:15.568727+0000","last_fullsized":"2024-09-18T08:11:42.320879+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":182,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:11:30.837341+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:50.886830+0000","last_clean_scrub_stamp":"2024-09-18T08:11:30.837341+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:13:50.677084+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":["7","4"],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":464,"reported_epoch":278,"state":"active+undersized","last_fresh":"2024-09-18T08:13:15.572998+0000","last_change":"2024-09-18T08:11:42.332383+0000","last_active":"2024-09-18T08:13:15.572998+0000","last_peered":"2024-09-18T08:13:15.572998+0000","last_clean":"2024-09-18T08:11:40.302158+0000","last_became_active":"2024-09-18T08:11:42.332383+0000","last_became_peered":"2024-09-18T08:11:42.332383+0000","last_unstale":"2024-09-18T08:13:15.572998+0000","last_undegraded":"2024-09-18T08:13:15.572998+0000","last_fullsized":"2024-09-18T08:11:42.326605+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:11:01.108648+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_clean_scrub_stamp":"2024-09-18T08:11:01.108648+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:13:27.191986+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":["5","3"],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":214,"reported_epoch":278,"state":"active+clean","last_fresh":"2024-09-18T08:13:15.567498+0000","last_change":"2024-09-18T08:11:34.625641+0000","last_active":"2024-09-18T08:13:15.567498+0000","last_peered":"2024-09-18T08:13:15.567498+0000","last_clean":"2024-09-18T08:13:15.567498+0000","last_became_active":"2024-09-18T08:08:18.539604+0000","last_became_peered":"2024-09-18T08:08:18.539604+0000","last_unstale":"2024-09-18T08:13:15.567498+0000","last_undegraded":"2024-09-18T08:13:15.567498+0000","last_fullsized":"2024-09-18T08:13:15.567498+0000","mapping_epoch":181,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":182,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:11:34.625572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_clean_scrub_stamp":"2024-09-18T08:11:34.625572+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":485,"reported_epoch":278,"state":"active+clean","last_fresh":"2024-09-18T08:13:15.565206+0000","last_change":"2024-09-18T08:12:40.711594+0000","last_active":"2024-09-18T08:13:15.565206+0000","last_peered":"2024-09-18T08:13:15.565206+0000","last_clean":"2024-09-18T08:13:15.565206+0000","last_became_active":"2024-09-18T08:08:18.523611+0000","last_became_peered":"2024-09-18T08:08:18.523611+0000","last_unstale":"2024-09-18T08:13:15.565206+0000","last_undegraded":"2024-09-18T08:13:15.565206+0000","last_fullsized":"2024-09-18T08:13:15.565206+0000","mapping_epoch":181,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":182,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:12:40.711507+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:46.860092+0000","last_clean_scrub_stamp":"2024-09-18T08:12:40.711507+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:13:48.813540+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1c","version":"271'27","reported_seq":209,"reported_epoch":278,"state":"active+undersized+degraded","last_fresh":"2024-09-18T08:13:15.565262+0000","last_change":"2024-09-18T08:11:42.329080+0000","last_active":"2024-09-18T08:13:15.565262+0000","last_peered":"2024-09-18T08:13:15.565262+0000","last_clean":"2024-09-18T08:11:40.306909+0000","last_became_active":"2024-09-18T08:11:42.329080+0000","last_became_peered":"2024-09-18T08:11:42.329080+0000","last_unstale":"2024-09-18T08:13:15.565262+0000","last_undegraded":"2024-09-18T08:11:42.319732+0000","last_fullsized":"2024-09-18T08:11:42.319638+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":199,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'15","last_scrub_stamp":"2024-09-18T08:07:39.530340+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_clean_scrub_stamp":"2024-09-18T08:07:39.530340+0000","objects_scrubbed":7,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:13:47.287389+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":7,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":44,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":7}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":349,"reported_epoch":278,"state":"active+undersized","last_fresh":"2024-09-18T08:13:15.564987+0000","last_change":"2024-09-18T08:11:42.330489+0000","last_active":"2024-09-18T08:13:15.564987+0000","last_peered":"2024-09-18T08:13:15.564987+0000","last_clean":"2024-09-18T08:08:15.505698+0000","last_became_active":"2024-09-18T08:11:42.330489+0000","last_became_peered":"2024-09-18T08:11:42.330489+0000","last_unstale":"2024-09-18T08:13:15.564987+0000","last_undegraded":"2024-09-18T08:13:15.564987+0000","last_fullsized":"2024-09-18T08:11:42.318950+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":182,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:10:59.645112+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:50.886830+0000","last_clean_scrub_stamp":"2024-09-18T08:10:59.645112+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:14:05.464503+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":["4","0"],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":467,"reported_epoch":278,"state":"active+clean","last_fresh":"2024-09-18T08:13:15.573343+0000","last_change":"2024-09-18T08:09:36.592348+0000","last_active":"2024-09-18T08:13:15.573343+0000","last_peered":"2024-09-18T08:13:15.573343+0000","last_clean":"2024-09-18T08:13:15.573343+0000","last_became_active":"2024-09-18T08:06:24.128659+0000","last_became_peered":"2024-09-18T08:06:24.128659+0000","last_unstale":"2024-09-18T08:13:15.573343+0000","last_undegraded":"2024-09-18T08:13:15.573343+0000","last_fullsized":"2024-09-18T08:13:15.573343+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:09:36.592281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_clean_scrub_stamp":"2024-09-18T08:09:36.592281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:13:20.906385+0000","scrub_duration":16,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":210,"reported_epoch":278,"state":"active+undersized","last_fresh":"2024-09-18T08:13:15.565908+0000","last_change":"2024-09-18T08:11:42.335948+0000","last_active":"2024-09-18T08:13:15.565908+0000","last_peered":"2024-09-18T08:13:15.565908+0000","last_clean":"2024-09-18T08:11:40.306989+0000","last_became_active":"2024-09-18T08:11:42.335948+0000","last_became_peered":"2024-09-18T08:11:42.335948+0000","last_unstale":"2024-09-18T08:13:15.565908+0000","last_undegraded":"2024-09-18T08:13:15.565908+0000","last_fullsized":"2024-09-18T08:11:42.319106+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":182,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:07:13.543722+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:46.860092+0000","last_clean_scrub_stamp":"2024-09-18T08:07:13.543722+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:13:54.429777+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":["4","2"],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"271'18","reported_seq":550,"reported_epoch":278,"state":"active+undersized+degraded","last_fresh":"2024-09-18T08:13:15.572876+0000","last_change":"2024-09-18T08:11:42.329686+0000","last_active":"2024-09-18T08:13:15.572876+0000","last_peered":"2024-09-18T08:13:15.572876+0000","last_clean":"2024-09-18T08:11:40.313323+0000","last_became_active":"2024-09-18T08:11:42.329686+0000","last_became_peered":"2024-09-18T08:11:42.329686+0000","last_unstale":"2024-09-18T08:13:15.572876+0000","last_undegraded":"2024-09-18T08:11:42.324380+0000","last_fullsized":"2024-09-18T08:11:42.324261+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":182,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'12","last_scrub_stamp":"2024-09-18T08:11:32.304472+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_clean_scrub_stamp":"2024-09-18T08:11:32.304472+0000","objects_scrubbed":5,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":5,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":34,"num_read_kb":25,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":["3","4"],"object_location_counts":[{"shards":"3,4","objects":5}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":214,"reported_epoch":278,"state":"active+clean","last_fresh":"2024-09-18T08:13:15.566123+0000","last_change":"2024-09-18T08:10:46.303912+0000","last_active":"2024-09-18T08:13:15.566123+0000","last_peered":"2024-09-18T08:13:15.566123+0000","last_clean":"2024-09-18T08:13:15.566123+0000","last_became_active":"2024-09-18T08:08:18.531438+0000","last_became_peered":"2024-09-18T08:08:18.531438+0000","last_unstale":"2024-09-18T08:13:15.566123+0000","last_undegraded":"2024-09-18T08:13:15.566123+0000","last_fullsized":"2024-09-18T08:13:15.566123+0000","mapping_epoch":181,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":182,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:10:46.303848+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:50.886830+0000","last_clean_scrub_stamp":"2024-09-18T08:10:46.303848+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1c","version":"69'1","reported_seq":302,"reported_epoch":278,"state":"active+clean","last_fresh":"2024-09-18T08:13:15.568669+0000","last_change":"2024-09-18T08:10:46.212697+0000","last_active":"2024-09-18T08:13:15.568669+0000","last_peered":"2024-09-18T08:13:15.568669+0000","last_clean":"2024-09-18T08:13:15.568669+0000","last_became_active":"2024-09-18T08:06:46.141661+0000","last_became_peered":"2024-09-18T08:06:46.141661+0000","last_unstale":"2024-09-18T08:13:15.568669+0000","last_undegraded":"2024-09-18T08:13:15.568669+0000","last_fullsized":"2024-09-18T08:13:15.568669+0000","mapping_epoch":143,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":144,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'1","last_scrub_stamp":"2024-09-18T08:10:46.212626+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_clean_scrub_stamp":"2024-09-18T08:10:46.212626+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":403,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,2],"acting":[5,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"62'1","reported_seq":86,"reported_epoch":278,"state":"active+undersized+degraded","last_fresh":"2024-09-18T08:13:15.570674+0000","last_change":"2024-09-18T08:11:42.335870+0000","last_active":"2024-09-18T08:13:15.570674+0000","last_peered":"2024-09-18T08:13:15.570674+0000","last_clean":"2024-09-18T08:02:47.882026+0000","last_became_active":"2024-09-18T08:11:42.335870+0000","last_became_peered":"2024-09-18T08:11:42.335870+0000","last_unstale":"2024-09-18T08:13:15.570674+0000","last_undegraded":"2024-09-18T08:11:42.323433+0000","last_fullsized":"2024-09-18T08:11:42.323288+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":182,"parent":"0.0","parent_split_bits":0,"last_scrub":"62'1","last_scrub_stamp":"2024-09-18T08:11:00.634190+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:46.860092+0000","last_clean_scrub_stamp":"2024-09-18T08:11:00.634190+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:13:59.006000+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":["3","4"],"object_location_counts":[{"shards":"3,4","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"271'20","reported_seq":482,"reported_epoch":278,"state":"active+clean","last_fresh":"2024-09-18T08:13:15.564710+0000","last_change":"2024-09-18T08:10:19.932544+0000","last_active":"2024-09-18T08:13:15.564710+0000","last_peered":"2024-09-18T08:13:15.564710+0000","last_clean":"2024-09-18T08:13:15.564710+0000","last_became_active":"2024-09-18T08:02:49.892024+0000","last_became_peered":"2024-09-18T08:02:49.892024+0000","last_unstale":"2024-09-18T08:13:15.564710+0000","last_undegraded":"2024-09-18T08:13:15.564710+0000","last_fullsized":"2024-09-18T08:13:15.564710+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'10","last_scrub_stamp":"2024-09-18T08:10:19.932465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_clean_scrub_stamp":"2024-09-18T08:10:19.932465+0000","objects_scrubbed":5,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:13:22.749697+0000","scrub_duration":19,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":25,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1f","version":"182'13","reported_seq":279,"reported_epoch":278,"state":"active+clean","last_fresh":"2024-09-18T08:13:15.567583+0000","last_change":"2024-09-18T08:12:59.403122+0000","last_active":"2024-09-18T08:13:15.567583+0000","last_peered":"2024-09-18T08:13:15.567583+0000","last_clean":"2024-09-18T08:13:15.567583+0000","last_became_active":"2024-09-18T08:08:18.543095+0000","last_became_peered":"2024-09-18T08:08:18.543095+0000","last_unstale":"2024-09-18T08:13:15.567583+0000","last_undegraded":"2024-09-18T08:13:15.567583+0000","last_fullsized":"2024-09-18T08:13:15.567583+0000","mapping_epoch":181,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":182,"parent":"0.0","parent_split_bits":0,"last_scrub":"182'13","last_scrub_stamp":"2024-09-18T08:12:59.403054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:50.886830+0000","last_clean_scrub_stamp":"2024-09-18T08:12:59.403054+0000","objects_scrubbed":1,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:14:04.748540+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,7],"acting":[4,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.2","version":"278'11982","reported_seq":12090,"reported_epoch":278,"state":"active+undersized+degraded","last_fresh":"2024-09-18T08:13:19.049875+0000","last_change":"2024-09-18T08:11:42.343258+0000","last_active":"2024-09-18T08:13:19.049875+0000","last_peered":"2024-09-18T08:13:19.049875+0000","last_clean":"2024-09-18T08:11:41.306257+0000","last_became_active":"2024-09-18T08:11:42.343258+0000","last_became_peered":"2024-09-18T08:11:42.343258+0000","last_unstale":"2024-09-18T08:13:19.049875+0000","last_undegraded":"2024-09-18T08:11:42.323950+0000","last_fullsized":"2024-09-18T08:11:42.323830+0000","mapping_epoch":244,"log_start":"261'7600","ondisk_log_start":"261'7600","created":232,"last_epoch_clean":233,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:11:08.262286+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:11:08.262286+0000","last_clean_scrub_stamp":"2024-09-18T08:11:08.262286+0000","objects_scrubbed":0,"log_size":4382,"log_dups_size":0,"ondisk_log_size":4382,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:13:57.965537+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":386662400,"num_objects":5900,"num_object_clones":0,"num_object_copies":17700,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":5900,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5900,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11982,"num_write_kb":572224,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["3","2"],"object_location_counts":[{"shards":"2,3","objects":5900}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"74'79","reported_seq":284,"reported_epoch":278,"state":"active+clean","last_fresh":"2024-09-18T08:13:15.568605+0000","last_change":"2024-09-18T08:11:29.068419+0000","last_active":"2024-09-18T08:13:15.568605+0000","last_peered":"2024-09-18T08:13:15.568605+0000","last_clean":"2024-09-18T08:13:15.568605+0000","last_became_active":"2024-09-18T08:07:26.394584+0000","last_became_peered":"2024-09-18T08:07:26.394584+0000","last_unstale":"2024-09-18T08:13:15.568605+0000","last_undegraded":"2024-09-18T08:13:15.568605+0000","last_fullsized":"2024-09-18T08:13:15.568605+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":152,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":4,"last_scrub":"74'79","last_scrub_stamp":"2024-09-18T08:11:29.068355+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:00:11.893742+0000","last_clean_scrub_stamp":"2024-09-18T08:11:29.068355+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,6],"acting":[5,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.f","version":"271'27","reported_seq":139,"reported_epoch":278,"state":"active+undersized+degraded","last_fresh":"2024-09-18T08:13:15.573591+0000","last_change":"2024-09-18T08:11:42.336910+0000","last_active":"2024-09-18T08:13:15.573591+0000","last_peered":"2024-09-18T08:13:15.573591+0000","last_clean":"2024-09-18T08:02:54.411508+0000","last_became_active":"2024-09-18T08:11:42.336910+0000","last_became_peered":"2024-09-18T08:11:42.336910+0000","last_unstale":"2024-09-18T08:13:15.573591+0000","last_undegraded":"2024-09-18T08:11:42.326875+0000","last_fullsized":"2024-09-18T08:11:42.326758+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":182,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'15","last_scrub_stamp":"2024-09-18T08:10:41.548188+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_clean_scrub_stamp":"2024-09-18T08:10:41.548188+0000","objects_scrubbed":0,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:13:45.030554+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":7,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":43,"num_read_kb":34,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":["3","4"],"object_location_counts":[{"shards":"3,4","objects":7}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":468,"reported_epoch":278,"state":"active+undersized","last_fresh":"2024-09-18T08:13:15.571076+0000","last_change":"2024-09-18T08:11:42.338122+0000","last_active":"2024-09-18T08:13:15.571076+0000","last_peered":"2024-09-18T08:13:15.571076+0000","last_clean":"2024-09-18T08:11:40.314538+0000","last_became_active":"2024-09-18T08:11:42.338122+0000","last_became_peered":"2024-09-18T08:11:42.338122+0000","last_unstale":"2024-09-18T08:13:15.571076+0000","last_undegraded":"2024-09-18T08:13:15.571076+0000","last_fullsized":"2024-09-18T08:11:42.325100+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:08:18.389273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:46.860092+0000","last_clean_scrub_stamp":"2024-09-18T08:08:18.389273+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:13:52.020423+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":["3","7"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.e","version":"182'12","reported_seq":279,"reported_epoch":278,"state":"active+clean","last_fresh":"2024-09-18T08:13:15.567447+0000","last_change":"2024-09-18T08:10:56.414273+0000","last_active":"2024-09-18T08:13:15.567447+0000","last_peered":"2024-09-18T08:13:15.567447+0000","last_clean":"2024-09-18T08:13:15.567447+0000","last_became_active":"2024-09-18T08:08:18.525147+0000","last_became_peered":"2024-09-18T08:08:18.525147+0000","last_unstale":"2024-09-18T08:13:15.567447+0000","last_undegraded":"2024-09-18T08:13:15.567447+0000","last_fullsized":"2024-09-18T08:13:15.567447+0000","mapping_epoch":181,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":182,"parent":"0.0","parent_split_bits":0,"last_scrub":"182'12","last_scrub_stamp":"2024-09-18T08:10:56.414189+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:50.886830+0000","last_clean_scrub_stamp":"2024-09-18T08:10:56.414189+0000","objects_scrubbed":1,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:13:24.638095+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":464,"reported_epoch":278,"state":"active+undersized","last_fresh":"2024-09-18T08:13:15.569660+0000","last_change":"2024-09-18T08:11:42.330265+0000","last_active":"2024-09-18T08:13:15.569660+0000","last_peered":"2024-09-18T08:13:15.569660+0000","last_clean":"2024-09-18T08:11:40.302096+0000","last_became_active":"2024-09-18T08:11:42.330265+0000","last_became_peered":"2024-09-18T08:11:42.330265+0000","last_unstale":"2024-09-18T08:13:15.569660+0000","last_undegraded":"2024-09-18T08:13:15.569660+0000","last_fullsized":"2024-09-18T08:11:42.324373+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:07:19.714430+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_clean_scrub_stamp":"2024-09-18T08:07:19.714430+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:14:04.243722+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":["5","0"],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.d","version":"278'11923","reported_seq":12031,"reported_epoch":278,"state":"active+undersized+degraded","last_fresh":"2024-09-18T08:13:18.372194+0000","last_change":"2024-09-18T08:11:42.340820+0000","last_active":"2024-09-18T08:13:18.372194+0000","last_peered":"2024-09-18T08:13:18.372194+0000","last_clean":"2024-09-18T08:11:41.317368+0000","last_became_active":"2024-09-18T08:11:42.340820+0000","last_became_peered":"2024-09-18T08:11:42.340820+0000","last_unstale":"2024-09-18T08:13:18.372194+0000","last_undegraded":"2024-09-18T08:11:42.327808+0000","last_fullsized":"2024-09-18T08:11:42.327694+0000","mapping_epoch":244,"log_start":"259'7300","ondisk_log_start":"259'7300","created":232,"last_epoch_clean":233,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:11:08.262286+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:11:08.262286+0000","last_clean_scrub_stamp":"2024-09-18T08:11:08.262286+0000","objects_scrubbed":0,"log_size":4623,"log_dups_size":0,"ondisk_log_size":4623,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:13:29.850751+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":392167448,"num_objects":5985,"num_object_clones":0,"num_object_copies":17955,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":5985,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5985,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":11923,"num_write_kb":572993,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":["7","5"],"object_location_counts":[{"shards":"5,7","objects":5985}],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.5","version":"74'79","reported_seq":229,"reported_epoch":278,"state":"active+clean","last_fresh":"2024-09-18T08:13:15.566751+0000","last_change":"2024-09-18T08:11:54.503697+0000","last_active":"2024-09-18T08:13:15.566751+0000","last_peered":"2024-09-18T08:13:15.566751+0000","last_clean":"2024-09-18T08:13:15.566751+0000","last_became_active":"2024-09-18T08:08:52.690344+0000","last_became_peered":"2024-09-18T08:08:52.690344+0000","last_unstale":"2024-09-18T08:13:15.566751+0000","last_undegraded":"2024-09-18T08:13:15.566751+0000","last_fullsized":"2024-09-18T08:13:15.566751+0000","mapping_epoch":198,"log_start":"0'0","ondisk_log_start":"0'0","created":152,"last_epoch_clean":199,"parent":"0.0","parent_split_bits":3,"last_scrub":"74'79","last_scrub_stamp":"2024-09-18T08:11:54.503603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:00:11.893742+0000","last_clean_scrub_stamp":"2024-09-18T08:11:54.503603+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,6],"acting":[4,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.0","version":"271'30","reported_seq":527,"reported_epoch":278,"state":"active+clean","last_fresh":"2024-09-18T08:13:15.570628+0000","last_change":"2024-09-18T08:12:36.956399+0000","last_active":"2024-09-18T08:13:15.570628+0000","last_peered":"2024-09-18T08:13:15.570628+0000","last_clean":"2024-09-18T08:13:15.570628+0000","last_became_active":"2024-09-18T08:02:49.886818+0000","last_became_peered":"2024-09-18T08:02:49.886818+0000","last_unstale":"2024-09-18T08:13:15.570628+0000","last_undegraded":"2024-09-18T08:13:15.570628+0000","last_fullsized":"2024-09-18T08:13:15.570628+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'18","last_scrub_stamp":"2024-09-18T08:12:36.956258+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_clean_scrub_stamp":"2024-09-18T08:12:36.956258+0000","objects_scrubbed":8,"log_size":30,"log_dups_size":0,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:13:41.247230+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":52,"num_read_kb":40,"num_write":32,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":488,"reported_epoch":278,"state":"active+clean","last_fresh":"2024-09-18T08:13:15.578430+0000","last_change":"2024-09-18T08:11:20.379525+0000","last_active":"2024-09-18T08:13:15.578430+0000","last_peered":"2024-09-18T08:13:15.578430+0000","last_clean":"2024-09-18T08:13:15.578430+0000","last_became_active":"2024-09-18T08:02:47.880225+0000","last_became_peered":"2024-09-18T08:02:47.880225+0000","last_unstale":"2024-09-18T08:13:15.578430+0000","last_undegraded":"2024-09-18T08:13:15.578430+0000","last_fullsized":"2024-09-18T08:13:15.578430+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:11:20.379467+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:46.860092+0000","last_clean_scrub_stamp":"2024-09-18T08:11:20.379467+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":222,"reported_epoch":278,"state":"active+clean","last_fresh":"2024-09-18T08:13:15.566361+0000","last_change":"2024-09-18T08:12:30.732619+0000","last_active":"2024-09-18T08:13:15.566361+0000","last_peered":"2024-09-18T08:13:15.566361+0000","last_clean":"2024-09-18T08:13:15.566361+0000","last_became_active":"2024-09-18T08:08:18.530341+0000","last_became_peered":"2024-09-18T08:08:18.530341+0000","last_unstale":"2024-09-18T08:13:15.566361+0000","last_undegraded":"2024-09-18T08:13:15.566361+0000","last_fullsized":"2024-09-18T08:13:15.566361+0000","mapping_epoch":181,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":182,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:12:30.732539+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:50.886830+0000","last_clean_scrub_stamp":"2024-09-18T08:12:30.732539+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:13:50.343431+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":222,"reported_epoch":278,"state":"active+clean","last_fresh":"2024-09-18T08:13:15.566230+0000","last_change":"2024-09-18T08:12:02.473229+0000","last_active":"2024-09-18T08:13:15.566230+0000","last_peered":"2024-09-18T08:13:15.566230+0000","last_clean":"2024-09-18T08:13:15.566230+0000","last_became_active":"2024-09-18T08:08:18.543694+0000","last_became_peered":"2024-09-18T08:08:18.543694+0000","last_unstale":"2024-09-18T08:13:15.566230+0000","last_undegraded":"2024-09-18T08:13:15.566230+0000","last_fullsized":"2024-09-18T08:13:15.566230+0000","mapping_epoch":181,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":182,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:12:02.473168+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_clean_scrub_stamp":"2024-09-18T08:12:02.473168+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:13:19.522644+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.c","version":"278'11885","reported_seq":12069,"reported_epoch":278,"state":"active+clean","last_fresh":"2024-09-18T08:13:19.046845+0000","last_change":"2024-09-18T08:13:06.086787+0000","last_active":"2024-09-18T08:13:19.046845+0000","last_peered":"2024-09-18T08:13:19.046845+0000","last_clean":"2024-09-18T08:13:19.046845+0000","last_became_active":"2024-09-18T08:11:09.293176+0000","last_became_peered":"2024-09-18T08:11:09.293176+0000","last_unstale":"2024-09-18T08:13:19.046845+0000","last_undegraded":"2024-09-18T08:13:19.046845+0000","last_fullsized":"2024-09-18T08:13:19.046845+0000","mapping_epoch":232,"log_start":"259'7500","ondisk_log_start":"259'7500","created":232,"last_epoch_clean":233,"parent":"0.0","parent_split_bits":0,"last_scrub":"274'10686","last_scrub_stamp":"2024-09-18T08:13:06.086640+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:11:08.262286+0000","last_clean_scrub_stamp":"2024-09-18T08:13:06.086640+0000","objects_scrubbed":7100,"log_size":4385,"log_dups_size":0,"ondisk_log_size":4385,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:14:25.235882+0000","scrub_duration":2162,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":381091840,"num_objects":5815,"num_object_clones":0,"num_object_copies":17445,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5815,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11885,"num_write_kb":566400,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.4","version":"74'79","reported_seq":298,"reported_epoch":278,"state":"active+clean","last_fresh":"2024-09-18T08:13:15.570421+0000","last_change":"2024-09-18T08:12:37.920428+0000","last_active":"2024-09-18T08:13:15.570421+0000","last_peered":"2024-09-18T08:13:15.570421+0000","last_clean":"2024-09-18T08:13:15.570421+0000","last_became_active":"2024-09-18T08:07:26.398930+0000","last_became_peered":"2024-09-18T08:07:26.398930+0000","last_unstale":"2024-09-18T08:13:15.570421+0000","last_undegraded":"2024-09-18T08:13:15.570421+0000","last_fullsized":"2024-09-18T08:13:15.570421+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":152,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":3,"last_scrub":"74'79","last_scrub_stamp":"2024-09-18T08:12:37.920342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:00:11.893742+0000","last_clean_scrub_stamp":"2024-09-18T08:12:37.920342+0000","objects_scrubbed":1,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:13:38.011227+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":327680,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,7],"acting":[3,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"271'28","reported_seq":251,"reported_epoch":278,"state":"active+clean","last_fresh":"2024-09-18T08:13:15.566181+0000","last_change":"2024-09-18T08:11:53.524126+0000","last_active":"2024-09-18T08:13:15.566181+0000","last_peered":"2024-09-18T08:13:15.566181+0000","last_clean":"2024-09-18T08:13:15.566181+0000","last_became_active":"2024-09-18T08:08:18.549671+0000","last_became_peered":"2024-09-18T08:08:18.549671+0000","last_unstale":"2024-09-18T08:13:15.566181+0000","last_undegraded":"2024-09-18T08:13:15.566181+0000","last_fullsized":"2024-09-18T08:13:15.566181+0000","mapping_epoch":181,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":182,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'14","last_scrub_stamp":"2024-09-18T08:11:53.524045+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_clean_scrub_stamp":"2024-09-18T08:11:53.524045+0000","objects_scrubbed":7,"log_size":28,"log_dups_size":0,"ondisk_log_size":28,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:13:17.577878+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":41,"num_read_kb":34,"num_write":28,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"62'2","reported_seq":500,"reported_epoch":278,"state":"active+undersized+degraded","last_fresh":"2024-09-18T08:13:15.565121+0000","last_change":"2024-09-18T08:11:42.338771+0000","last_active":"2024-09-18T08:13:15.565121+0000","last_peered":"2024-09-18T08:13:15.565121+0000","last_clean":"2024-09-18T08:11:40.307484+0000","last_became_active":"2024-09-18T08:11:42.338771+0000","last_became_peered":"2024-09-18T08:11:42.338771+0000","last_unstale":"2024-09-18T08:13:15.565121+0000","last_undegraded":"2024-09-18T08:11:42.320507+0000","last_fullsized":"2024-09-18T08:11:42.320405+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":182,"parent":"0.0","parent_split_bits":0,"last_scrub":"62'2","last_scrub_stamp":"2024-09-18T08:06:43.607152+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:46.860092+0000","last_clean_scrub_stamp":"2024-09-18T08:06:43.607152+0000","objects_scrubbed":2,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:13:46.675999+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1039,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":2078,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":["0","4"],"object_location_counts":[{"shards":"0,4","objects":2}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.0","version":"245'11","reported_seq":619,"reported_epoch":278,"state":"active+undersized+degraded","last_fresh":"2024-09-18T08:13:15.570302+0000","last_change":"2024-09-18T08:11:42.339963+0000","last_active":"2024-09-18T08:13:15.570302+0000","last_peered":"2024-09-18T08:13:15.570302+0000","last_clean":"2024-09-18T08:11:40.313426+0000","last_became_active":"2024-09-18T08:11:42.339963+0000","last_became_peered":"2024-09-18T08:11:42.339963+0000","last_unstale":"2024-09-18T08:13:15.570302+0000","last_undegraded":"2024-09-18T08:11:42.326291+0000","last_fullsized":"2024-09-18T08:11:42.326182+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":182,"parent":"0.0","parent_split_bits":0,"last_scrub":"182'10","last_scrub_stamp":"2024-09-18T08:09:51.467349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:50.886830+0000","last_clean_scrub_stamp":"2024-09-18T08:09:51.467349+0000","objects_scrubbed":1,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:13:56.035212+0000","scrub_duration":19,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":["3","4"],"object_location_counts":[{"shards":"3,4","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":490,"reported_epoch":278,"state":"active+clean","last_fresh":"2024-09-18T08:13:15.569819+0000","last_change":"2024-09-18T08:11:20.750758+0000","last_active":"2024-09-18T08:13:15.569819+0000","last_peered":"2024-09-18T08:13:15.569819+0000","last_clean":"2024-09-18T08:13:15.569819+0000","last_became_active":"2024-09-18T08:06:24.140303+0000","last_became_peered":"2024-09-18T08:06:24.140303+0000","last_unstale":"2024-09-18T08:13:15.569819+0000","last_undegraded":"2024-09-18T08:13:15.569819+0000","last_fullsized":"2024-09-18T08:13:15.569819+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:11:20.750690+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_clean_scrub_stamp":"2024-09-18T08:11:20.750690+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:13:27.856504+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.f","version":"278'11931","reported_seq":12036,"reported_epoch":278,"state":"active+undersized+degraded","last_fresh":"2024-09-18T08:13:18.495069+0000","last_change":"2024-09-18T08:11:42.337301+0000","last_active":"2024-09-18T08:13:18.495069+0000","last_peered":"2024-09-18T08:13:18.495069+0000","last_clean":"2024-09-18T08:11:41.300139+0000","last_became_active":"2024-09-18T08:11:42.337301+0000","last_became_peered":"2024-09-18T08:11:42.337301+0000","last_unstale":"2024-09-18T08:13:18.495069+0000","last_undegraded":"2024-09-18T08:11:42.332576+0000","last_fullsized":"2024-09-18T08:11:42.332455+0000","mapping_epoch":244,"log_start":"258'7000","ondisk_log_start":"258'7000","created":232,"last_epoch_clean":233,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:11:08.262286+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:11:08.262286+0000","last_clean_scrub_stamp":"2024-09-18T08:11:08.262286+0000","objects_scrubbed":0,"log_size":4931,"log_dups_size":0,"ondisk_log_size":4931,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":384237568,"num_objects":5863,"num_object_clones":0,"num_object_copies":17589,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":5863,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5863,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11931,"num_write_kb":569408,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":["5","3"],"object_location_counts":[{"shards":"3,5","objects":5863}],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.7","version":"74'79","reported_seq":293,"reported_epoch":278,"state":"active+clean","last_fresh":"2024-09-18T08:13:15.577989+0000","last_change":"2024-09-18T08:12:33.846970+0000","last_active":"2024-09-18T08:13:15.577989+0000","last_peered":"2024-09-18T08:13:15.577989+0000","last_clean":"2024-09-18T08:13:15.577989+0000","last_became_active":"2024-09-18T08:07:26.398289+0000","last_became_peered":"2024-09-18T08:07:26.398289+0000","last_unstale":"2024-09-18T08:13:15.577989+0000","last_undegraded":"2024-09-18T08:13:15.577989+0000","last_fullsized":"2024-09-18T08:13:15.577989+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":152,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":3,"last_scrub":"74'79","last_scrub_stamp":"2024-09-18T08:12:33.846913+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:00:11.893742+0000","last_clean_scrub_stamp":"2024-09-18T08:12:33.846913+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:13:51.842141+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,7],"acting":[3,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.2","version":"271'20","reported_seq":126,"reported_epoch":278,"state":"active+undersized+degraded","last_fresh":"2024-09-18T08:13:15.569090+0000","last_change":"2024-09-18T08:11:42.341545+0000","last_active":"2024-09-18T08:13:15.569090+0000","last_peered":"2024-09-18T08:13:15.569090+0000","last_clean":"2024-09-18T08:02:54.382625+0000","last_became_active":"2024-09-18T08:11:42.341545+0000","last_became_peered":"2024-09-18T08:11:42.341545+0000","last_unstale":"2024-09-18T08:13:15.569090+0000","last_undegraded":"2024-09-18T08:11:42.323463+0000","last_fullsized":"2024-09-18T08:11:42.323350+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":182,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'10","last_scrub_stamp":"2024-09-18T08:11:24.347134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_clean_scrub_stamp":"2024-09-18T08:11:24.347134+0000","objects_scrubbed":0,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":5,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":29,"num_read_kb":24,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":["5","4"],"object_location_counts":[{"shards":"4,5","objects":5}],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":472,"reported_epoch":278,"state":"active+clean","last_fresh":"2024-09-18T08:13:15.572795+0000","last_change":"2024-09-18T08:12:06.876766+0000","last_active":"2024-09-18T08:13:15.572795+0000","last_peered":"2024-09-18T08:13:15.572795+0000","last_clean":"2024-09-18T08:13:15.572795+0000","last_became_active":"2024-09-18T08:02:47.874727+0000","last_became_peered":"2024-09-18T08:02:47.874727+0000","last_unstale":"2024-09-18T08:13:15.572795+0000","last_undegraded":"2024-09-18T08:13:15.572795+0000","last_fullsized":"2024-09-18T08:13:15.572795+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:12:06.876541+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:46.860092+0000","last_clean_scrub_stamp":"2024-09-18T08:12:06.876541+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.3","version":"138'10","reported_seq":614,"reported_epoch":278,"state":"active+clean","last_fresh":"2024-09-18T08:13:18.053350+0000","last_change":"2024-09-18T08:07:46.695714+0000","last_active":"2024-09-18T08:13:18.053350+0000","last_peered":"2024-09-18T08:13:18.053350+0000","last_clean":"2024-09-18T08:13:18.053350+0000","last_became_active":"2024-09-18T08:06:24.134194+0000","last_became_peered":"2024-09-18T08:06:24.134194+0000","last_unstale":"2024-09-18T08:13:18.053350+0000","last_undegraded":"2024-09-18T08:13:18.053350+0000","last_fullsized":"2024-09-18T08:13:18.053350+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"138'10","last_scrub_stamp":"2024-09-18T08:07:46.695656+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:50.886830+0000","last_clean_scrub_stamp":"2024-09-18T08:07:46.695656+0000","objects_scrubbed":1,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:13:50.682156+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":483,"reported_epoch":278,"state":"active+clean","last_fresh":"2024-09-18T08:13:15.566159+0000","last_change":"2024-09-18T08:09:46.989395+0000","last_active":"2024-09-18T08:13:15.566159+0000","last_peered":"2024-09-18T08:13:15.566159+0000","last_clean":"2024-09-18T08:13:15.566159+0000","last_became_active":"2024-09-18T08:08:52.700080+0000","last_became_peered":"2024-09-18T08:08:52.700080+0000","last_unstale":"2024-09-18T08:13:15.566159+0000","last_undegraded":"2024-09-18T08:13:15.566159+0000","last_fullsized":"2024-09-18T08:13:15.566159+0000","mapping_epoch":198,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":199,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:09:46.989329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_clean_scrub_stamp":"2024-09-18T08:09:46.989329+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.e","version":"278'11712","reported_seq":11818,"reported_epoch":278,"state":"active+undersized+degraded","last_fresh":"2024-09-18T08:13:18.305123+0000","last_change":"2024-09-18T08:11:42.329475+0000","last_active":"2024-09-18T08:13:18.305123+0000","last_peered":"2024-09-18T08:13:18.305123+0000","last_clean":"2024-09-18T08:11:41.287488+0000","last_became_active":"2024-09-18T08:11:42.329475+0000","last_became_peered":"2024-09-18T08:11:42.329475+0000","last_unstale":"2024-09-18T08:13:18.305123+0000","last_undegraded":"2024-09-18T08:11:42.322141+0000","last_fullsized":"2024-09-18T08:11:42.321960+0000","mapping_epoch":244,"log_start":"258'7000","ondisk_log_start":"258'7000","created":232,"last_epoch_clean":233,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:11:08.262286+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:11:08.262286+0000","last_clean_scrub_stamp":"2024-09-18T08:11:08.262286+0000","objects_scrubbed":0,"log_size":4712,"log_dups_size":0,"ondisk_log_size":4712,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:14:08.457475+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":386269184,"num_objects":5894,"num_object_clones":0,"num_object_copies":17682,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":5894,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5894,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11712,"num_write_kb":563392,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":["4","7"],"object_location_counts":[{"shards":"4,7","objects":5894}],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.6","version":"74'79","reported_seq":373,"reported_epoch":278,"state":"active+clean","last_fresh":"2024-09-18T08:13:15.564368+0000","last_change":"2024-09-18T08:12:28.659146+0000","last_active":"2024-09-18T08:13:15.564368+0000","last_peered":"2024-09-18T08:13:15.564368+0000","last_clean":"2024-09-18T08:13:15.564368+0000","last_became_active":"2024-09-18T08:07:26.426532+0000","last_became_peered":"2024-09-18T08:07:26.426532+0000","last_unstale":"2024-09-18T08:13:15.564368+0000","last_undegraded":"2024-09-18T08:13:15.564368+0000","last_fullsized":"2024-09-18T08:13:15.564368+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":152,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":3,"last_scrub":"74'79","last_scrub_stamp":"2024-09-18T08:12:28.659059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:00:11.893742+0000","last_clean_scrub_stamp":"2024-09-18T08:12:28.659059+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:13:46.488791+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,3],"acting":[0,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.3","version":"271'29","reported_seq":511,"reported_epoch":278,"state":"active+clean","last_fresh":"2024-09-18T08:13:15.564898+0000","last_change":"2024-09-18T08:10:21.940118+0000","last_active":"2024-09-18T08:13:15.564898+0000","last_peered":"2024-09-18T08:13:15.564898+0000","last_clean":"2024-09-18T08:13:15.564898+0000","last_became_active":"2024-09-18T08:02:49.890318+0000","last_became_peered":"2024-09-18T08:02:49.890318+0000","last_unstale":"2024-09-18T08:13:15.564898+0000","last_undegraded":"2024-09-18T08:13:15.564898+0000","last_fullsized":"2024-09-18T08:13:15.564898+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'19","last_scrub_stamp":"2024-09-18T08:10:21.940027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_clean_scrub_stamp":"2024-09-18T08:10:21.940027+0000","objects_scrubbed":8,"log_size":29,"log_dups_size":0,"ondisk_log_size":29,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:13:26.679243+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":54,"num_read_kb":40,"num_write":32,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.4","version":"62'1","reported_seq":385,"reported_epoch":278,"state":"active+undersized+degraded","last_fresh":"2024-09-18T08:13:15.565061+0000","last_change":"2024-09-18T08:11:42.328236+0000","last_active":"2024-09-18T08:13:15.565061+0000","last_peered":"2024-09-18T08:13:15.565061+0000","last_clean":"2024-09-18T08:08:50.690355+0000","last_became_active":"2024-09-18T08:11:42.328236+0000","last_became_peered":"2024-09-18T08:11:42.328236+0000","last_unstale":"2024-09-18T08:13:15.565061+0000","last_undegraded":"2024-09-18T08:11:42.319415+0000","last_fullsized":"2024-09-18T08:11:42.319312+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":199,"parent":"0.0","parent_split_bits":0,"last_scrub":"62'1","last_scrub_stamp":"2024-09-18T08:10:07.593004+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:46.860092+0000","last_clean_scrub_stamp":"2024-09-18T08:10:07.593004+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:13:55.289663+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":436,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":436,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":["2","5"],"object_location_counts":[{"shards":"2,5","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":390,"reported_epoch":278,"state":"active+clean","last_fresh":"2024-09-18T08:13:15.564538+0000","last_change":"2024-09-18T08:12:37.732331+0000","last_active":"2024-09-18T08:13:15.564538+0000","last_peered":"2024-09-18T08:13:15.564538+0000","last_clean":"2024-09-18T08:13:15.564538+0000","last_became_active":"2024-09-18T08:06:24.133262+0000","last_became_peered":"2024-09-18T08:06:24.133262+0000","last_unstale":"2024-09-18T08:13:15.564538+0000","last_undegraded":"2024-09-18T08:13:15.564538+0000","last_fullsized":"2024-09-18T08:13:15.564538+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:12:37.732260+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:50.886830+0000","last_clean_scrub_stamp":"2024-09-18T08:12:37.732260+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:14:02.173227+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":393,"reported_epoch":278,"state":"active+undersized","last_fresh":"2024-09-18T08:13:15.562543+0000","last_change":"2024-09-18T08:11:42.332856+0000","last_active":"2024-09-18T08:13:15.562543+0000","last_peered":"2024-09-18T08:13:15.562543+0000","last_clean":"2024-09-18T08:08:50.690188+0000","last_became_active":"2024-09-18T08:11:42.332856+0000","last_became_peered":"2024-09-18T08:11:42.332856+0000","last_unstale":"2024-09-18T08:13:15.562543+0000","last_undegraded":"2024-09-18T08:13:15.562543+0000","last_fullsized":"2024-09-18T08:11:42.318131+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":199,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:11:28.346395+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_clean_scrub_stamp":"2024-09-18T08:11:28.346395+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:13:53.343534+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":["2","6"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.9","version":"278'11932","reported_seq":12114,"reported_epoch":278,"state":"active+clean","last_fresh":"2024-09-18T08:13:19.051699+0000","last_change":"2024-09-18T08:13:08.016091+0000","last_active":"2024-09-18T08:13:19.051699+0000","last_peered":"2024-09-18T08:13:19.051699+0000","last_clean":"2024-09-18T08:13:19.051699+0000","last_became_active":"2024-09-18T08:11:09.292238+0000","last_became_peered":"2024-09-18T08:11:09.292238+0000","last_unstale":"2024-09-18T08:13:19.051699+0000","last_undegraded":"2024-09-18T08:13:19.051699+0000","last_fullsized":"2024-09-18T08:13:19.051699+0000","mapping_epoch":232,"log_start":"254'6200","ondisk_log_start":"254'6200","created":232,"last_epoch_clean":233,"parent":"0.0","parent_split_bits":0,"last_scrub":"274'10944","last_scrub_stamp":"2024-09-18T08:13:08.015855+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:11:08.262286+0000","last_clean_scrub_stamp":"2024-09-18T08:13:08.015855+0000","objects_scrubbed":6827,"log_size":5732,"log_dups_size":0,"ondisk_log_size":5732,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:14:26.287586+0000","scrub_duration":2172,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":376438784,"num_objects":5744,"num_object_clones":0,"num_object_copies":17232,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5744,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11932,"num_write_kb":565632,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,5],"acting":[0,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"74'79","reported_seq":284,"reported_epoch":278,"state":"active+clean","last_fresh":"2024-09-18T08:13:15.571297+0000","last_change":"2024-09-18T08:11:30.036947+0000","last_active":"2024-09-18T08:13:15.571297+0000","last_peered":"2024-09-18T08:13:15.571297+0000","last_clean":"2024-09-18T08:13:15.571297+0000","last_became_active":"2024-09-18T08:07:26.400643+0000","last_became_peered":"2024-09-18T08:07:26.400643+0000","last_unstale":"2024-09-18T08:13:15.571297+0000","last_undegraded":"2024-09-18T08:13:15.571297+0000","last_fullsized":"2024-09-18T08:13:15.571297+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":152,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":4,"last_scrub":"74'79","last_scrub_stamp":"2024-09-18T08:11:30.036868+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:00:11.893742+0000","last_clean_scrub_stamp":"2024-09-18T08:11:30.036868+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,2],"acting":[5,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.4","version":"271'41","reported_seq":185,"reported_epoch":278,"state":"active+undersized+degraded","last_fresh":"2024-09-18T08:13:15.565970+0000","last_change":"2024-09-18T08:11:42.328675+0000","last_active":"2024-09-18T08:13:15.565970+0000","last_peered":"2024-09-18T08:13:15.565970+0000","last_clean":"2024-09-18T08:02:54.384549+0000","last_became_active":"2024-09-18T08:11:42.328675+0000","last_became_peered":"2024-09-18T08:11:42.328675+0000","last_unstale":"2024-09-18T08:13:15.565970+0000","last_undegraded":"2024-09-18T08:11:42.320375+0000","last_fullsized":"2024-09-18T08:11:42.320272+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'28","last_scrub_stamp":"2024-09-18T08:10:46.497874+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_clean_scrub_stamp":"2024-09-18T08:10:46.497874+0000","objects_scrubbed":0,"log_size":41,"log_dups_size":0,"ondisk_log_size":41,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:14:14.418386+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":10,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":65,"num_read_kb":50,"num_write":38,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":10}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":206,"reported_epoch":278,"state":"active+clean","last_fresh":"2024-09-18T08:13:15.565835+0000","last_change":"2024-09-18T08:08:18.543579+0000","last_active":"2024-09-18T08:13:15.565835+0000","last_peered":"2024-09-18T08:13:15.565835+0000","last_clean":"2024-09-18T08:13:15.565835+0000","last_became_active":"2024-09-18T08:08:18.538852+0000","last_became_peered":"2024-09-18T08:08:18.538852+0000","last_unstale":"2024-09-18T08:13:15.565835+0000","last_undegraded":"2024-09-18T08:13:15.565835+0000","last_fullsized":"2024-09-18T08:13:15.565835+0000","mapping_epoch":181,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":182,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:05:39.490141+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:46.860092+0000","last_clean_scrub_stamp":"2024-09-18T08:05:39.490141+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:13:48.505870+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"64'2","reported_seq":500,"reported_epoch":278,"state":"active+undersized+degraded","last_fresh":"2024-09-18T08:13:15.569328+0000","last_change":"2024-09-18T08:11:42.338718+0000","last_active":"2024-09-18T08:13:15.569328+0000","last_peered":"2024-09-18T08:13:15.569328+0000","last_clean":"2024-09-18T08:11:40.302328+0000","last_became_active":"2024-09-18T08:11:42.338718+0000","last_became_peered":"2024-09-18T08:11:42.338718+0000","last_unstale":"2024-09-18T08:13:15.569328+0000","last_undegraded":"2024-09-18T08:11:42.331457+0000","last_fullsized":"2024-09-18T08:11:42.330826+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"64'2","last_scrub_stamp":"2024-09-18T08:05:44.802942+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:45.855979+0000","last_clean_scrub_stamp":"2024-09-18T08:05:44.802942+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:13:59.231831+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.001348349,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":38,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":["5","6"],"object_location_counts":[{"shards":"5,6","objects":1}],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":473,"reported_epoch":278,"state":"active+undersized","last_fresh":"2024-09-18T08:13:15.565267+0000","last_change":"2024-09-18T08:11:42.334063+0000","last_active":"2024-09-18T08:13:15.565267+0000","last_peered":"2024-09-18T08:13:15.565267+0000","last_clean":"2024-09-18T08:11:40.307605+0000","last_became_active":"2024-09-18T08:11:42.334063+0000","last_became_peered":"2024-09-18T08:11:42.334063+0000","last_unstale":"2024-09-18T08:13:15.565267+0000","last_undegraded":"2024-09-18T08:13:15.565267+0000","last_fullsized":"2024-09-18T08:11:42.319418+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":182,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:10:53.880606+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:50.886830+0000","last_clean_scrub_stamp":"2024-09-18T08:10:53.880606+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:13:43.745796+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":["0","4"],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":475,"reported_epoch":278,"state":"active+clean","last_fresh":"2024-09-18T08:13:15.572965+0000","last_change":"2024-09-18T08:12:06.185241+0000","last_active":"2024-09-18T08:13:15.572965+0000","last_peered":"2024-09-18T08:13:15.572965+0000","last_clean":"2024-09-18T08:13:15.572965+0000","last_became_active":"2024-09-18T08:08:18.542678+0000","last_became_peered":"2024-09-18T08:08:18.542678+0000","last_unstale":"2024-09-18T08:13:15.572965+0000","last_undegraded":"2024-09-18T08:13:15.572965+0000","last_fullsized":"2024-09-18T08:13:15.572965+0000","mapping_epoch":181,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":182,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:12:06.185172+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_clean_scrub_stamp":"2024-09-18T08:12:06.185172+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:13:24.623291+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.a","version":"278'11855","reported_seq":11963,"reported_epoch":278,"state":"active+undersized+degraded","last_fresh":"2024-09-18T08:13:18.494214+0000","last_change":"2024-09-18T08:11:42.337494+0000","last_active":"2024-09-18T08:13:18.494214+0000","last_peered":"2024-09-18T08:13:18.494214+0000","last_clean":"2024-09-18T08:11:41.323914+0000","last_became_active":"2024-09-18T08:11:42.337494+0000","last_became_peered":"2024-09-18T08:11:42.337494+0000","last_unstale":"2024-09-18T08:13:18.494214+0000","last_undegraded":"2024-09-18T08:11:42.329944+0000","last_fullsized":"2024-09-18T08:11:42.329820+0000","mapping_epoch":244,"log_start":"258'6900","ondisk_log_start":"258'6900","created":232,"last_epoch_clean":233,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:11:08.262286+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:11:08.262286+0000","last_clean_scrub_stamp":"2024-09-18T08:11:08.262286+0000","objects_scrubbed":0,"log_size":4955,"log_dups_size":0,"ondisk_log_size":4955,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":382402560,"num_objects":5835,"num_object_clones":0,"num_object_copies":17505,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":5835,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5835,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11855,"num_write_kb":566080,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":["5","7"],"object_location_counts":[{"shards":"5,7","objects":5835}],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.2","version":"74'79","reported_seq":386,"reported_epoch":278,"state":"active+clean","last_fresh":"2024-09-18T08:13:15.564363+0000","last_change":"2024-09-18T08:11:51.733475+0000","last_active":"2024-09-18T08:13:15.564363+0000","last_peered":"2024-09-18T08:13:15.564363+0000","last_clean":"2024-09-18T08:13:15.564363+0000","last_became_active":"2024-09-18T08:08:18.522977+0000","last_became_peered":"2024-09-18T08:08:18.522977+0000","last_unstale":"2024-09-18T08:13:15.564363+0000","last_undegraded":"2024-09-18T08:13:15.564363+0000","last_fullsized":"2024-09-18T08:13:15.564363+0000","mapping_epoch":181,"log_start":"0'0","ondisk_log_start":"0'0","created":152,"last_epoch_clean":182,"parent":"0.0","parent_split_bits":4,"last_scrub":"74'79","last_scrub_stamp":"2024-09-18T08:11:51.733401+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:00:11.893742+0000","last_clean_scrub_stamp":"2024-09-18T08:11:51.733401+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:13:16.439237+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,4],"acting":[0,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.7","version":"271'17","reported_seq":420,"reported_epoch":278,"state":"active+clean","last_fresh":"2024-09-18T08:13:15.566650+0000","last_change":"2024-09-18T08:12:04.519313+0000","last_active":"2024-09-18T08:13:15.566650+0000","last_peered":"2024-09-18T08:13:15.566650+0000","last_clean":"2024-09-18T08:13:15.566650+0000","last_became_active":"2024-09-18T08:04:18.144925+0000","last_became_peered":"2024-09-18T08:04:18.144925+0000","last_unstale":"2024-09-18T08:13:15.566650+0000","last_undegraded":"2024-09-18T08:13:15.566650+0000","last_fullsized":"2024-09-18T08:13:15.566650+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'13","last_scrub_stamp":"2024-09-18T08:12:04.519232+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_clean_scrub_stamp":"2024-09-18T08:12:04.519232+0000","objects_scrubbed":5,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:13:29.319792+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":35,"num_read_kb":24,"num_write":20,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,0],"acting":[2,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":251,"reported_epoch":278,"state":"active+undersized","last_fresh":"2024-09-18T08:13:15.563070+0000","last_change":"2024-09-18T08:11:42.322313+0000","last_active":"2024-09-18T08:13:15.563070+0000","last_peered":"2024-09-18T08:13:15.563070+0000","last_clean":"2024-09-18T08:06:21.513250+0000","last_became_active":"2024-09-18T08:11:42.322313+0000","last_became_peered":"2024-09-18T08:11:42.322313+0000","last_unstale":"2024-09-18T08:13:15.563070+0000","last_undegraded":"2024-09-18T08:13:15.563070+0000","last_fullsized":"2024-09-18T08:11:42.318707+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:11:31.367525+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:46.860092+0000","last_clean_scrub_stamp":"2024-09-18T08:11:31.367525+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:13:54.335823+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":["2","6"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1","version":"62'1","reported_seq":482,"reported_epoch":278,"state":"active+clean","last_fresh":"2024-09-18T08:13:15.564265+0000","last_change":"2024-09-18T08:12:03.497218+0000","last_active":"2024-09-18T08:13:15.564265+0000","last_peered":"2024-09-18T08:13:15.564265+0000","last_clean":"2024-09-18T08:13:15.564265+0000","last_became_active":"2024-09-18T08:02:46.868175+0000","last_became_peered":"2024-09-18T08:02:46.868175+0000","last_unstale":"2024-09-18T08:13:15.564265+0000","last_undegraded":"2024-09-18T08:13:15.564265+0000","last_fullsized":"2024-09-18T08:13:15.564265+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"62'1","last_scrub_stamp":"2024-09-18T08:12:03.497148+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:45.855979+0000","last_clean_scrub_stamp":"2024-09-18T08:12:03.497148+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.00066785499999999997,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":172,"reported_epoch":278,"state":"active+clean","last_fresh":"2024-09-18T08:13:15.565874+0000","last_change":"2024-09-18T08:08:52.690656+0000","last_active":"2024-09-18T08:13:15.565874+0000","last_peered":"2024-09-18T08:13:15.565874+0000","last_clean":"2024-09-18T08:13:15.565874+0000","last_became_active":"2024-09-18T08:08:52.690286+0000","last_became_peered":"2024-09-18T08:08:52.690286+0000","last_unstale":"2024-09-18T08:13:15.565874+0000","last_undegraded":"2024-09-18T08:13:15.565874+0000","last_fullsized":"2024-09-18T08:13:15.565874+0000","mapping_epoch":198,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":199,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:07:08.806748+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:50.886830+0000","last_clean_scrub_stamp":"2024-09-18T08:07:08.806748+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:13:45.259864+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":482,"reported_epoch":278,"state":"active+clean","last_fresh":"2024-09-18T08:13:15.572065+0000","last_change":"2024-09-18T08:12:04.859323+0000","last_active":"2024-09-18T08:13:15.572065+0000","last_peered":"2024-09-18T08:13:15.572065+0000","last_clean":"2024-09-18T08:13:15.572065+0000","last_became_active":"2024-09-18T08:06:24.145382+0000","last_became_peered":"2024-09-18T08:06:24.145382+0000","last_unstale":"2024-09-18T08:13:15.572065+0000","last_undegraded":"2024-09-18T08:13:15.572065+0000","last_fullsized":"2024-09-18T08:13:15.572065+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:12:04.859145+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_clean_scrub_stamp":"2024-09-18T08:12:04.859145+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:13:29.331505+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.b","version":"278'11785","reported_seq":11888,"reported_epoch":278,"state":"active+clean","last_fresh":"2024-09-18T08:13:19.044945+0000","last_change":"2024-09-18T08:11:09.489007+0000","last_active":"2024-09-18T08:13:19.044945+0000","last_peered":"2024-09-18T08:13:19.044945+0000","last_clean":"2024-09-18T08:13:19.044945+0000","last_became_active":"2024-09-18T08:11:09.488677+0000","last_became_peered":"2024-09-18T08:11:09.488677+0000","last_unstale":"2024-09-18T08:13:19.044945+0000","last_undegraded":"2024-09-18T08:13:19.044945+0000","last_fullsized":"2024-09-18T08:13:19.044945+0000","mapping_epoch":232,"log_start":"254'6100","ondisk_log_start":"254'6100","created":232,"last_epoch_clean":233,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:11:08.262286+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:11:08.262286+0000","last_clean_scrub_stamp":"2024-09-18T08:11:08.262286+0000","objects_scrubbed":0,"log_size":5685,"log_dups_size":0,"ondisk_log_size":5685,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:13:41.748081+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":377815040,"num_objects":5765,"num_object_clones":0,"num_object_copies":17295,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5765,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11785,"num_write_kb":561600,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"74'79","reported_seq":154,"reported_epoch":278,"state":"active+undersized","last_fresh":"2024-09-18T08:13:15.565107+0000","last_change":"2024-09-18T08:11:42.324310+0000","last_active":"2024-09-18T08:13:15.565107+0000","last_peered":"2024-09-18T08:13:15.565107+0000","last_clean":"2024-09-18T08:08:15.515052+0000","last_became_active":"2024-09-18T08:11:42.324310+0000","last_became_peered":"2024-09-18T08:11:42.324310+0000","last_unstale":"2024-09-18T08:13:15.565107+0000","last_undegraded":"2024-09-18T08:13:15.565107+0000","last_fullsized":"2024-09-18T08:11:42.318664+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":152,"last_epoch_clean":182,"parent":"0.0","parent_split_bits":3,"last_scrub":"74'79","last_scrub_stamp":"2024-09-18T08:09:41.651729+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:00:11.893742+0000","last_clean_scrub_stamp":"2024-09-18T08:09:41.651729+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"user requested scrub scheduled @ 2024-09-18T08:13:43.631429+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":53670,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":10,"num_read_kb":20,"num_write":11,"num_write_kb":148,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":53670,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":["4","6"],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.6","version":"271'24","reported_seq":549,"reported_epoch":278,"state":"active+undersized+degraded","last_fresh":"2024-09-18T08:13:15.564804+0000","last_change":"2024-09-18T08:11:42.342654+0000","last_active":"2024-09-18T08:13:15.564804+0000","last_peered":"2024-09-18T08:13:15.564804+0000","last_clean":"2024-09-18T08:11:40.307291+0000","last_became_active":"2024-09-18T08:11:42.342654+0000","last_became_peered":"2024-09-18T08:11:42.342654+0000","last_unstale":"2024-09-18T08:13:15.564804+0000","last_undegraded":"2024-09-18T08:11:42.320107+0000","last_fullsized":"2024-09-18T08:11:42.320001+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":200,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'12","last_scrub_stamp":"2024-09-18T08:10:58.837531+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_clean_scrub_stamp":"2024-09-18T08:10:58.837531+0000","objects_scrubbed":6,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:13:47.699391+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":6,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":36,"num_read_kb":30,"num_write":24,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0,2","objects":6}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":494,"reported_epoch":278,"state":"active+clean","last_fresh":"2024-09-18T08:13:15.564244+0000","last_change":"2024-09-18T08:12:05.794780+0000","last_active":"2024-09-18T08:13:15.564244+0000","last_peered":"2024-09-18T08:13:15.564244+0000","last_clean":"2024-09-18T08:13:15.564244+0000","last_became_active":"2024-09-18T08:08:18.523135+0000","last_became_peered":"2024-09-18T08:08:18.523135+0000","last_unstale":"2024-09-18T08:13:15.564244+0000","last_undegraded":"2024-09-18T08:13:15.564244+0000","last_fullsized":"2024-09-18T08:13:15.564244+0000","mapping_epoch":181,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":182,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:12:05.794714+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:46.860092+0000","last_clean_scrub_stamp":"2024-09-18T08:12:05.794714+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:13:34.497459+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"70'5","reported_seq":216,"reported_epoch":278,"state":"active+undersized+degraded","last_fresh":"2024-09-18T08:13:17.508168+0000","last_change":"2024-09-18T08:11:42.330514+0000","last_active":"2024-09-18T08:13:17.508168+0000","last_peered":"2024-09-18T08:13:17.508168+0000","last_clean":"2024-09-18T08:02:58.641213+0000","last_became_active":"2024-09-18T08:11:42.330514+0000","last_became_peered":"2024-09-18T08:11:42.330514+0000","last_unstale":"2024-09-18T08:13:17.508168+0000","last_undegraded":"2024-09-18T08:11:42.319867+0000","last_fullsized":"2024-09-18T08:11:42.319754+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":144,"parent":"0.0","parent_split_bits":0,"last_scrub":"70'5","last_scrub_stamp":"2024-09-18T08:10:44.388400+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:45.855979+0000","last_clean_scrub_stamp":"2024-09-18T08:10:44.388400+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:13:34.709050+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000504055,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":103,"num_read_kb":98,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":["0","7"],"object_location_counts":[{"shards":"0,7","objects":1}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":467,"reported_epoch":278,"state":"active+undersized","last_fresh":"2024-09-18T08:13:15.573812+0000","last_change":"2024-09-18T08:11:42.333954+0000","last_active":"2024-09-18T08:13:15.573812+0000","last_peered":"2024-09-18T08:13:15.573812+0000","last_clean":"2024-09-18T08:11:40.304599+0000","last_became_active":"2024-09-18T08:11:42.333954+0000","last_became_peered":"2024-09-18T08:11:42.333954+0000","last_unstale":"2024-09-18T08:13:15.573812+0000","last_undegraded":"2024-09-18T08:13:15.573812+0000","last_fullsized":"2024-09-18T08:11:42.326679+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:07:20.767841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:50.886830+0000","last_clean_scrub_stamp":"2024-09-18T08:07:20.767841+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:14:02.229317+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":["5","0"],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":76,"reported_epoch":278,"state":"active+undersized","last_fresh":"2024-09-18T08:13:15.569150+0000","last_change":"2024-09-18T08:11:42.341422+0000","last_active":"2024-09-18T08:13:15.569150+0000","last_peered":"2024-09-18T08:13:15.569150+0000","last_clean":"2024-09-18T08:02:52.890475+0000","last_became_active":"2024-09-18T08:11:42.341422+0000","last_became_peered":"2024-09-18T08:11:42.341422+0000","last_unstale":"2024-09-18T08:13:15.569150+0000","last_undegraded":"2024-09-18T08:13:15.569150+0000","last_fullsized":"2024-09-18T08:11:42.325862+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:09:49.702055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_clean_scrub_stamp":"2024-09-18T08:09:49.702055+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:14:07.222323+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":["5","3"],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.8","version":"278'11918","reported_seq":12022,"reported_epoch":278,"state":"active+clean","last_fresh":"2024-09-18T08:13:19.054918+0000","last_change":"2024-09-18T08:11:09.291403+0000","last_active":"2024-09-18T08:13:19.054918+0000","last_peered":"2024-09-18T08:13:19.054918+0000","last_clean":"2024-09-18T08:13:19.054918+0000","last_became_active":"2024-09-18T08:11:09.291038+0000","last_became_peered":"2024-09-18T08:11:09.291038+0000","last_unstale":"2024-09-18T08:13:19.054918+0000","last_undegraded":"2024-09-18T08:13:19.054918+0000","last_fullsized":"2024-09-18T08:13:19.054918+0000","mapping_epoch":232,"log_start":"259'7500","ondisk_log_start":"259'7500","created":232,"last_epoch_clean":233,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:11:08.262286+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:11:08.262286+0000","last_clean_scrub_stamp":"2024-09-18T08:11:08.262286+0000","objects_scrubbed":0,"log_size":4418,"log_dups_size":0,"ondisk_log_size":4418,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:13:46.024564+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":387317760,"num_objects":5910,"num_object_clones":0,"num_object_copies":17730,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5910,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11918,"num_write_kb":570496,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.5","version":"271'33","reported_seq":311,"reported_epoch":278,"state":"active+clean","last_fresh":"2024-09-18T08:13:15.566895+0000","last_change":"2024-09-18T08:10:50.350728+0000","last_active":"2024-09-18T08:13:15.566895+0000","last_peered":"2024-09-18T08:13:15.566895+0000","last_clean":"2024-09-18T08:13:15.566895+0000","last_became_active":"2024-09-18T08:08:18.547555+0000","last_became_peered":"2024-09-18T08:08:18.547555+0000","last_unstale":"2024-09-18T08:13:15.566895+0000","last_undegraded":"2024-09-18T08:13:15.566895+0000","last_fullsized":"2024-09-18T08:13:15.566895+0000","mapping_epoch":181,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":182,"parent":"0.0","parent_split_bits":0,"last_scrub":"182'21","last_scrub_stamp":"2024-09-18T08:10:50.350670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_clean_scrub_stamp":"2024-09-18T08:10:50.350670+0000","objects_scrubbed":8,"log_size":33,"log_dups_size":0,"ondisk_log_size":33,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:13:22.617174+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":42,"num_read_kb":32,"num_write":25,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":154,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,7],"acting":[4,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":479,"reported_epoch":278,"state":"active+clean","last_fresh":"2024-09-18T08:13:15.574685+0000","last_change":"2024-09-18T08:09:45.572278+0000","last_active":"2024-09-18T08:13:15.574685+0000","last_peered":"2024-09-18T08:13:15.574685+0000","last_clean":"2024-09-18T08:13:15.574685+0000","last_became_active":"2024-09-18T08:06:24.133425+0000","last_became_peered":"2024-09-18T08:06:24.133425+0000","last_unstale":"2024-09-18T08:13:15.574685+0000","last_undegraded":"2024-09-18T08:13:15.574685+0000","last_fullsized":"2024-09-18T08:13:15.574685+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:09:45.572201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:46.860092+0000","last_clean_scrub_stamp":"2024-09-18T08:09:45.572201+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:13:25.853472+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"74'79","reported_seq":568,"reported_epoch":278,"state":"active+clean","last_fresh":"2024-09-18T08:13:15.569079+0000","last_change":"2024-09-18T08:11:34.834467+0000","last_active":"2024-09-18T08:13:15.569079+0000","last_peered":"2024-09-18T08:13:15.569079+0000","last_clean":"2024-09-18T08:13:15.569079+0000","last_became_active":"2024-09-18T08:07:24.376931+0000","last_became_peered":"2024-09-18T08:07:24.376931+0000","last_unstale":"2024-09-18T08:13:15.569079+0000","last_undegraded":"2024-09-18T08:13:15.569079+0000","last_fullsized":"2024-09-18T08:13:15.569079+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":153,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'79","last_scrub_stamp":"2024-09-18T08:11:34.834393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:00:11.893742+0000","last_clean_scrub_stamp":"2024-09-18T08:11:34.834393+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:13:44.792838+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":308,"reported_epoch":278,"state":"active+clean","last_fresh":"2024-09-18T08:13:15.565607+0000","last_change":"2024-09-18T08:12:56.304598+0000","last_active":"2024-09-18T08:13:15.565607+0000","last_peered":"2024-09-18T08:13:15.565607+0000","last_clean":"2024-09-18T08:13:15.565607+0000","last_became_active":"2024-09-18T08:06:46.141046+0000","last_became_peered":"2024-09-18T08:06:46.141046+0000","last_unstale":"2024-09-18T08:13:15.565607+0000","last_undegraded":"2024-09-18T08:13:15.565607+0000","last_fullsized":"2024-09-18T08:13:15.565607+0000","mapping_epoch":143,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":144,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:12:56.304527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:50.886830+0000","last_clean_scrub_stamp":"2024-09-18T08:12:56.304527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:14:25.953150+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":474,"reported_epoch":278,"state":"active+clean","last_fresh":"2024-09-18T08:13:15.573544+0000","last_change":"2024-09-18T08:12:05.886343+0000","last_active":"2024-09-18T08:13:15.573544+0000","last_peered":"2024-09-18T08:13:15.573544+0000","last_clean":"2024-09-18T08:13:15.573544+0000","last_became_active":"2024-09-18T08:08:18.548233+0000","last_became_peered":"2024-09-18T08:08:18.548233+0000","last_unstale":"2024-09-18T08:13:15.573544+0000","last_undegraded":"2024-09-18T08:13:15.573544+0000","last_fullsized":"2024-09-18T08:13:15.573544+0000","mapping_epoch":181,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":182,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:12:05.886281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_clean_scrub_stamp":"2024-09-18T08:12:05.886281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:13:23.913083+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.3","version":"278'12108","reported_seq":12213,"reported_epoch":278,"state":"active+undersized+degraded","last_fresh":"2024-09-18T08:13:18.377812+0000","last_change":"2024-09-18T08:11:42.332945+0000","last_active":"2024-09-18T08:13:18.377812+0000","last_peered":"2024-09-18T08:13:18.377812+0000","last_clean":"2024-09-18T08:11:41.283527+0000","last_became_active":"2024-09-18T08:11:42.332945+0000","last_became_peered":"2024-09-18T08:11:42.332945+0000","last_unstale":"2024-09-18T08:13:18.377812+0000","last_undegraded":"2024-09-18T08:11:42.322363+0000","last_fullsized":"2024-09-18T08:11:42.322217+0000","mapping_epoch":244,"log_start":"259'7500","ondisk_log_start":"259'7500","created":232,"last_epoch_clean":233,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:11:08.262286+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:11:08.262286+0000","last_clean_scrub_stamp":"2024-09-18T08:11:08.262286+0000","objects_scrubbed":0,"log_size":4608,"log_dups_size":0,"ondisk_log_size":4608,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":394133504,"num_objects":6014,"num_object_clones":0,"num_object_copies":18042,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":6014,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6014,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12108,"num_write_kb":579904,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":["7","3"],"object_location_counts":[{"shards":"3,7","objects":6014}],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.e","version":"271'19","reported_seq":242,"reported_epoch":278,"state":"active+undersized+degraded","last_fresh":"2024-09-18T08:13:15.567215+0000","last_change":"2024-09-18T08:11:42.337720+0000","last_active":"2024-09-18T08:13:15.567215+0000","last_peered":"2024-09-18T08:13:15.567215+0000","last_clean":"2024-09-18T08:11:40.307924+0000","last_became_active":"2024-09-18T08:11:42.337720+0000","last_became_peered":"2024-09-18T08:11:42.337720+0000","last_unstale":"2024-09-18T08:13:15.567215+0000","last_undegraded":"2024-09-18T08:11:42.327549+0000","last_fullsized":"2024-09-18T08:11:42.327437+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":182,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'11","last_scrub_stamp":"2024-09-18T08:10:53.406574+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_clean_scrub_stamp":"2024-09-18T08:10:53.406574+0000","objects_scrubbed":5,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:13:33.725845+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":5,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":31,"num_read_kb":24,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":["4","6"],"object_location_counts":[{"shards":"4,6","objects":5}],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":222,"reported_epoch":278,"state":"active+clean","last_fresh":"2024-09-18T08:13:15.566522+0000","last_change":"2024-09-18T08:11:59.458276+0000","last_active":"2024-09-18T08:13:15.566522+0000","last_peered":"2024-09-18T08:13:15.566522+0000","last_clean":"2024-09-18T08:13:15.566522+0000","last_became_active":"2024-09-18T08:08:18.524229+0000","last_became_peered":"2024-09-18T08:08:18.524229+0000","last_unstale":"2024-09-18T08:13:15.566522+0000","last_undegraded":"2024-09-18T08:13:15.566522+0000","last_fullsized":"2024-09-18T08:13:15.566522+0000","mapping_epoch":181,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":182,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:11:59.458232+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:46.860092+0000","last_clean_scrub_stamp":"2024-09-18T08:11:59.458232+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:13:15.966196+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":491,"reported_epoch":278,"state":"active+clean","last_fresh":"2024-09-18T08:13:15.573314+0000","last_change":"2024-09-18T08:11:28.031016+0000","last_active":"2024-09-18T08:13:15.573314+0000","last_peered":"2024-09-18T08:13:15.573314+0000","last_clean":"2024-09-18T08:13:15.573314+0000","last_became_active":"2024-09-18T08:08:18.547105+0000","last_became_peered":"2024-09-18T08:08:18.547105+0000","last_unstale":"2024-09-18T08:13:15.573314+0000","last_undegraded":"2024-09-18T08:13:15.573314+0000","last_fullsized":"2024-09-18T08:13:15.573314+0000","mapping_epoch":181,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":182,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:11:28.030965+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:50.886830+0000","last_clean_scrub_stamp":"2024-09-18T08:11:28.030965+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":466,"reported_epoch":278,"state":"active+clean","last_fresh":"2024-09-18T08:13:15.570530+0000","last_change":"2024-09-18T08:07:09.197912+0000","last_active":"2024-09-18T08:13:15.570530+0000","last_peered":"2024-09-18T08:13:15.570530+0000","last_clean":"2024-09-18T08:13:15.570530+0000","last_became_active":"2024-09-18T08:06:24.130266+0000","last_became_peered":"2024-09-18T08:06:24.130266+0000","last_unstale":"2024-09-18T08:13:15.570530+0000","last_undegraded":"2024-09-18T08:13:15.570530+0000","last_fullsized":"2024-09-18T08:13:15.570530+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:07:09.197841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_clean_scrub_stamp":"2024-09-18T08:07:09.197841+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:14:13.754695+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.0","version":"278'11833","reported_seq":11939,"reported_epoch":278,"state":"active+undersized+degraded","last_fresh":"2024-09-18T08:13:18.765946+0000","last_change":"2024-09-18T08:11:42.323264+0000","last_active":"2024-09-18T08:13:18.765946+0000","last_peered":"2024-09-18T08:13:18.765946+0000","last_clean":"2024-09-18T08:11:41.291289+0000","last_became_active":"2024-09-18T08:11:42.323264+0000","last_became_peered":"2024-09-18T08:11:42.323264+0000","last_unstale":"2024-09-18T08:13:18.765946+0000","last_undegraded":"2024-09-18T08:11:42.320910+0000","last_fullsized":"2024-09-18T08:11:42.317956+0000","mapping_epoch":244,"log_start":"252'5300","ondisk_log_start":"252'5300","created":232,"last_epoch_clean":233,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:11:08.262286+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:11:08.262286+0000","last_clean_scrub_stamp":"2024-09-18T08:11:08.262286+0000","objects_scrubbed":0,"log_size":6533,"log_dups_size":0,"ondisk_log_size":6533,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:13:21.608334+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":379518976,"num_objects":5791,"num_object_clones":0,"num_object_copies":17373,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":5791,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5791,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11833,"num_write_kb":563968,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":["2","6"],"object_location_counts":[{"shards":"2,6","objects":5791}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.8","version":"74'79","reported_seq":291,"reported_epoch":278,"state":"active+clean","last_fresh":"2024-09-18T08:13:15.572751+0000","last_change":"2024-09-18T08:12:00.980995+0000","last_active":"2024-09-18T08:13:15.572751+0000","last_peered":"2024-09-18T08:13:15.572751+0000","last_clean":"2024-09-18T08:13:15.572751+0000","last_became_active":"2024-09-18T08:07:26.407298+0000","last_became_peered":"2024-09-18T08:07:26.407298+0000","last_unstale":"2024-09-18T08:13:15.572751+0000","last_undegraded":"2024-09-18T08:13:15.572751+0000","last_fullsized":"2024-09-18T08:13:15.572751+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":152,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":4,"last_scrub":"74'79","last_scrub_stamp":"2024-09-18T08:12:00.980917+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:00:11.893742+0000","last_clean_scrub_stamp":"2024-09-18T08:12:00.980917+0000","objects_scrubbed":1,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:13:27.761718+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262688,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,3],"acting":[5,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.d","version":"271'31","reported_seq":249,"reported_epoch":278,"state":"active+undersized+degraded","last_fresh":"2024-09-18T08:13:15.566971+0000","last_change":"2024-09-18T08:11:42.336457+0000","last_active":"2024-09-18T08:13:15.566971+0000","last_peered":"2024-09-18T08:13:15.566971+0000","last_clean":"2024-09-18T08:11:40.307800+0000","last_became_active":"2024-09-18T08:11:42.336457+0000","last_became_peered":"2024-09-18T08:11:42.336457+0000","last_unstale":"2024-09-18T08:13:15.566971+0000","last_undegraded":"2024-09-18T08:11:42.322722+0000","last_fullsized":"2024-09-18T08:11:42.322622+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":182,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'17","last_scrub_stamp":"2024-09-18T08:08:09.671248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_clean_scrub_stamp":"2024-09-18T08:08:09.671248+0000","objects_scrubbed":0,"log_size":31,"log_dups_size":0,"ondisk_log_size":31,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:14:02.406292+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":8,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":49,"num_read_kb":39,"num_write":32,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":["4","2"],"object_location_counts":[{"shards":"2,4","objects":8}],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":218,"reported_epoch":278,"state":"active+undersized","last_fresh":"2024-09-18T08:13:15.565641+0000","last_change":"2024-09-18T08:11:42.323936+0000","last_active":"2024-09-18T08:13:15.565641+0000","last_peered":"2024-09-18T08:13:15.565641+0000","last_clean":"2024-09-18T08:11:40.306980+0000","last_became_active":"2024-09-18T08:11:42.323936+0000","last_became_peered":"2024-09-18T08:11:42.323936+0000","last_unstale":"2024-09-18T08:13:15.565641+0000","last_undegraded":"2024-09-18T08:13:15.565641+0000","last_fullsized":"2024-09-18T08:11:42.319023+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":182,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:10:22.243148+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:46.860092+0000","last_clean_scrub_stamp":"2024-09-18T08:10:22.243148+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:14:04.461992+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":["4","6"],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":341,"reported_epoch":278,"state":"active+undersized","last_fresh":"2024-09-18T08:13:15.565576+0000","last_change":"2024-09-18T08:11:42.331646+0000","last_active":"2024-09-18T08:13:15.565576+0000","last_peered":"2024-09-18T08:13:15.565576+0000","last_clean":"2024-09-18T08:08:15.506244+0000","last_became_active":"2024-09-18T08:11:42.331646+0000","last_became_peered":"2024-09-18T08:11:42.331646+0000","last_unstale":"2024-09-18T08:13:15.565576+0000","last_undegraded":"2024-09-18T08:13:15.565576+0000","last_fullsized":"2024-09-18T08:11:42.318456+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":182,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:09:06.610879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:50.886830+0000","last_clean_scrub_stamp":"2024-09-18T08:09:06.610879+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:14:03.450879+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":["4","0"],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":187,"reported_epoch":278,"state":"active+clean","last_fresh":"2024-09-18T08:13:15.566554+0000","last_change":"2024-09-18T08:11:58.657536+0000","last_active":"2024-09-18T08:13:15.566554+0000","last_peered":"2024-09-18T08:13:15.566554+0000","last_clean":"2024-09-18T08:13:15.566554+0000","last_became_active":"2024-09-18T08:08:52.696300+0000","last_became_peered":"2024-09-18T08:08:52.696300+0000","last_unstale":"2024-09-18T08:13:15.566554+0000","last_undegraded":"2024-09-18T08:13:15.566554+0000","last_fullsized":"2024-09-18T08:13:15.566554+0000","mapping_epoch":198,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":199,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:11:58.657473+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_clean_scrub_stamp":"2024-09-18T08:11:58.657473+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:13:15.987566+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.1","version":"278'11964","reported_seq":12068,"reported_epoch":278,"state":"active+clean","last_fresh":"2024-09-18T08:13:19.043708+0000","last_change":"2024-09-18T08:11:09.489038+0000","last_active":"2024-09-18T08:13:19.043708+0000","last_peered":"2024-09-18T08:13:19.043708+0000","last_clean":"2024-09-18T08:13:19.043708+0000","last_became_active":"2024-09-18T08:11:09.488718+0000","last_became_peered":"2024-09-18T08:11:09.488718+0000","last_unstale":"2024-09-18T08:13:19.043708+0000","last_undegraded":"2024-09-18T08:13:19.043708+0000","last_fullsized":"2024-09-18T08:13:19.043708+0000","mapping_epoch":232,"log_start":"260'7600","ondisk_log_start":"260'7600","created":232,"last_epoch_clean":233,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:11:08.262286+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:11:08.262286+0000","last_clean_scrub_stamp":"2024-09-18T08:11:08.262286+0000","objects_scrubbed":0,"log_size":4364,"log_dups_size":0,"ondisk_log_size":4364,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:14:14.716160+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":381026304,"num_objects":5814,"num_object_clones":0,"num_object_copies":17442,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5814,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11964,"num_write_kb":568896,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,7],"acting":[3,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.9","version":"74'79","reported_seq":581,"reported_epoch":278,"state":"active+clean","last_fresh":"2024-09-18T08:13:15.572089+0000","last_change":"2024-09-18T08:09:27.676247+0000","last_active":"2024-09-18T08:13:15.572089+0000","last_peered":"2024-09-18T08:13:15.572089+0000","last_clean":"2024-09-18T08:13:15.572089+0000","last_became_active":"2024-09-18T08:08:52.691492+0000","last_became_peered":"2024-09-18T08:08:52.691492+0000","last_unstale":"2024-09-18T08:13:15.572089+0000","last_undegraded":"2024-09-18T08:13:15.572089+0000","last_fullsized":"2024-09-18T08:13:15.572089+0000","mapping_epoch":198,"log_start":"0'0","ondisk_log_start":"0'0","created":152,"last_epoch_clean":199,"parent":"0.0","parent_split_bits":4,"last_scrub":"74'79","last_scrub_stamp":"2024-09-18T08:09:27.676169+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:00:11.893742+0000","last_clean_scrub_stamp":"2024-09-18T08:09:27.676169+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:13:43.830751+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,2],"acting":[7,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.c","version":"271'20","reported_seq":233,"reported_epoch":278,"state":"active+clean","last_fresh":"2024-09-18T08:13:15.565945+0000","last_change":"2024-09-18T08:08:18.642452+0000","last_active":"2024-09-18T08:13:15.565945+0000","last_peered":"2024-09-18T08:13:15.565945+0000","last_clean":"2024-09-18T08:13:15.565945+0000","last_became_active":"2024-09-18T08:08:18.622980+0000","last_became_peered":"2024-09-18T08:08:18.622980+0000","last_unstale":"2024-09-18T08:13:15.565945+0000","last_undegraded":"2024-09-18T08:13:15.565945+0000","last_fullsized":"2024-09-18T08:13:15.565945+0000","mapping_epoch":181,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":182,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'10","last_scrub_stamp":"2024-09-18T08:05:41.474813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_clean_scrub_stamp":"2024-09-18T08:05:41.474813+0000","objects_scrubbed":0,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:13:45.565099+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":29,"num_read_kb":24,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":486,"reported_epoch":278,"state":"active+clean","last_fresh":"2024-09-18T08:13:15.570897+0000","last_change":"2024-09-18T08:10:57.455046+0000","last_active":"2024-09-18T08:13:15.570897+0000","last_peered":"2024-09-18T08:13:15.570897+0000","last_clean":"2024-09-18T08:13:15.570897+0000","last_became_active":"2024-09-18T08:08:18.542424+0000","last_became_peered":"2024-09-18T08:08:18.542424+0000","last_unstale":"2024-09-18T08:13:15.570897+0000","last_undegraded":"2024-09-18T08:13:15.570897+0000","last_fullsized":"2024-09-18T08:13:15.570897+0000","mapping_epoch":181,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":182,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:10:57.455000+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:46.860092+0000","last_clean_scrub_stamp":"2024-09-18T08:10:57.455000+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:13:50.076444+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.d","version":"69'8","reported_seq":607,"reported_epoch":278,"state":"active+clean","last_fresh":"2024-09-18T08:13:15.566298+0000","last_change":"2024-09-18T08:12:30.371968+0000","last_active":"2024-09-18T08:13:15.566298+0000","last_peered":"2024-09-18T08:13:15.566298+0000","last_clean":"2024-09-18T08:13:15.566298+0000","last_became_active":"2024-09-18T08:02:51.909011+0000","last_became_peered":"2024-09-18T08:02:51.909011+0000","last_unstale":"2024-09-18T08:13:15.566298+0000","last_undegraded":"2024-09-18T08:13:15.566298+0000","last_fullsized":"2024-09-18T08:13:15.566298+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'8","last_scrub_stamp":"2024-09-18T08:12:30.371904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:50.886830+0000","last_clean_scrub_stamp":"2024-09-18T08:12:30.371904+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:13:46.486491+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":225,"reported_epoch":278,"state":"active+undersized","last_fresh":"2024-09-18T08:13:15.567567+0000","last_change":"2024-09-18T08:11:42.337415+0000","last_active":"2024-09-18T08:13:15.567567+0000","last_peered":"2024-09-18T08:13:15.567567+0000","last_clean":"2024-09-18T08:11:40.308045+0000","last_became_active":"2024-09-18T08:11:42.337415+0000","last_became_peered":"2024-09-18T08:11:42.337415+0000","last_unstale":"2024-09-18T08:13:15.567567+0000","last_undegraded":"2024-09-18T08:13:15.567567+0000","last_fullsized":"2024-09-18T08:11:42.322962+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":199,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:11:18.475967+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_clean_scrub_stamp":"2024-09-18T08:11:18.475967+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":["4","2"],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.6","version":"278'11891","reported_seq":11998,"reported_epoch":278,"state":"active+undersized+degraded","last_fresh":"2024-09-18T08:13:18.500193+0000","last_change":"2024-09-18T08:11:42.339935+0000","last_active":"2024-09-18T08:13:18.500193+0000","last_peered":"2024-09-18T08:13:18.500193+0000","last_clean":"2024-09-18T08:11:41.290923+0000","last_became_active":"2024-09-18T08:11:42.339935+0000","last_became_peered":"2024-09-18T08:11:42.339935+0000","last_unstale":"2024-09-18T08:13:18.500193+0000","last_undegraded":"2024-09-18T08:11:42.325251+0000","last_fullsized":"2024-09-18T08:11:42.325122+0000","mapping_epoch":244,"log_start":"258'6900","ondisk_log_start":"258'6900","created":232,"last_epoch_clean":233,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:11:08.262286+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:11:08.262286+0000","last_clean_scrub_stamp":"2024-09-18T08:11:08.262286+0000","objects_scrubbed":0,"log_size":4991,"log_dups_size":0,"ondisk_log_size":4991,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:13:35.300193+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":382402560,"num_objects":5835,"num_object_clones":0,"num_object_copies":17505,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":5835,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5835,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11891,"num_write_kb":567232,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":["5","2"],"object_location_counts":[{"shards":"2,5","objects":5835}],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.b","version":"271'15","reported_seq":530,"reported_epoch":278,"state":"active+undersized+degraded","last_fresh":"2024-09-18T08:13:15.564740+0000","last_change":"2024-09-18T08:11:42.333863+0000","last_active":"2024-09-18T08:13:15.564740+0000","last_peered":"2024-09-18T08:13:15.564740+0000","last_clean":"2024-09-18T08:11:40.306474+0000","last_became_active":"2024-09-18T08:11:42.333863+0000","last_became_peered":"2024-09-18T08:11:42.333863+0000","last_unstale":"2024-09-18T08:13:15.564740+0000","last_undegraded":"2024-09-18T08:11:42.319070+0000","last_fullsized":"2024-09-18T08:11:42.318966+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":182,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'9","last_scrub_stamp":"2024-09-18T08:10:47.928596+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_clean_scrub_stamp":"2024-09-18T08:10:47.928596+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:13:27.656524+0000","scrub_duration":16,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":["0","4"],"object_location_counts":[{"shards":"0,4","objects":4}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":485,"reported_epoch":278,"state":"active+clean","last_fresh":"2024-09-18T08:13:15.569316+0000","last_change":"2024-09-18T08:11:22.076505+0000","last_active":"2024-09-18T08:13:15.569316+0000","last_peered":"2024-09-18T08:13:15.569316+0000","last_clean":"2024-09-18T08:13:15.569316+0000","last_became_active":"2024-09-18T08:06:24.135005+0000","last_became_peered":"2024-09-18T08:06:24.135005+0000","last_unstale":"2024-09-18T08:13:15.569316+0000","last_undegraded":"2024-09-18T08:13:15.569316+0000","last_fullsized":"2024-09-18T08:13:15.569316+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:11:22.076452+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:46.860092+0000","last_clean_scrub_stamp":"2024-09-18T08:11:22.076452+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":478,"reported_epoch":278,"state":"active+clean","last_fresh":"2024-09-18T08:13:15.564673+0000","last_change":"2024-09-18T08:08:18.509651+0000","last_active":"2024-09-18T08:13:15.564673+0000","last_peered":"2024-09-18T08:13:15.564673+0000","last_clean":"2024-09-18T08:13:15.564673+0000","last_became_active":"2024-09-18T08:08:18.509247+0000","last_became_peered":"2024-09-18T08:08:18.509247+0000","last_unstale":"2024-09-18T08:13:15.564673+0000","last_undegraded":"2024-09-18T08:13:15.564673+0000","last_fullsized":"2024-09-18T08:13:15.564673+0000","mapping_epoch":181,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":182,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:07:10.728679+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:50.886830+0000","last_clean_scrub_stamp":"2024-09-18T08:07:10.728679+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:13:46.290405+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,3],"acting":[2,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":490,"reported_epoch":278,"state":"active+clean","last_fresh":"2024-09-18T08:13:15.564207+0000","last_change":"2024-09-18T08:12:38.703021+0000","last_active":"2024-09-18T08:13:15.564207+0000","last_peered":"2024-09-18T08:13:15.564207+0000","last_clean":"2024-09-18T08:13:15.564207+0000","last_became_active":"2024-09-18T08:08:52.697337+0000","last_became_peered":"2024-09-18T08:08:52.697337+0000","last_unstale":"2024-09-18T08:13:15.564207+0000","last_undegraded":"2024-09-18T08:13:15.564207+0000","last_fullsized":"2024-09-18T08:13:15.564207+0000","mapping_epoch":198,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":199,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:12:38.702798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_clean_scrub_stamp":"2024-09-18T08:12:38.702798+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:13:41.813263+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.7","version":"278'11759","reported_seq":11892,"reported_epoch":278,"state":"active+clean","last_fresh":"2024-09-18T08:13:18.499506+0000","last_change":"2024-09-18T08:12:41.750258+0000","last_active":"2024-09-18T08:13:18.499506+0000","last_peered":"2024-09-18T08:13:18.499506+0000","last_clean":"2024-09-18T08:13:18.499506+0000","last_became_active":"2024-09-18T08:11:09.288659+0000","last_became_peered":"2024-09-18T08:11:09.288659+0000","last_unstale":"2024-09-18T08:13:18.499506+0000","last_undegraded":"2024-09-18T08:13:18.499506+0000","last_fullsized":"2024-09-18T08:13:18.499506+0000","mapping_epoch":232,"log_start":"258'6800","ondisk_log_start":"258'6800","created":232,"last_epoch_clean":233,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:11:08.262286+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:11:08.262286+0000","last_clean_scrub_stamp":"2024-09-18T08:11:08.262286+0000","objects_scrubbed":2599,"log_size":4959,"log_dups_size":0,"ondisk_log_size":4959,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":373096448,"num_objects":5693,"num_object_clones":0,"num_object_copies":17079,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5693,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11759,"num_write_kb":558464,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.a","version":"271'35","reported_seq":258,"reported_epoch":278,"state":"active+undersized+degraded","last_fresh":"2024-09-18T08:13:15.569891+0000","last_change":"2024-09-18T08:11:42.325717+0000","last_active":"2024-09-18T08:13:15.569891+0000","last_peered":"2024-09-18T08:13:15.569891+0000","last_clean":"2024-09-18T08:06:21.509422+0000","last_became_active":"2024-09-18T08:11:42.325717+0000","last_became_peered":"2024-09-18T08:11:42.325717+0000","last_unstale":"2024-09-18T08:13:15.569891+0000","last_undegraded":"2024-09-18T08:11:42.322210+0000","last_fullsized":"2024-09-18T08:11:42.322097+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'19","last_scrub_stamp":"2024-09-18T08:10:19.603091+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_clean_scrub_stamp":"2024-09-18T08:10:19.603091+0000","objects_scrubbed":0,"log_size":35,"log_dups_size":0,"ondisk_log_size":35,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:13:45.051164+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":9,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":55,"num_read_kb":44,"num_write":36,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":["6","7"],"object_location_counts":[{"shards":"6,7","objects":9}],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":481,"reported_epoch":278,"state":"active+clean","last_fresh":"2024-09-18T08:13:15.572475+0000","last_change":"2024-09-18T08:12:31.205529+0000","last_active":"2024-09-18T08:13:15.572475+0000","last_peered":"2024-09-18T08:13:15.572475+0000","last_clean":"2024-09-18T08:13:15.572475+0000","last_became_active":"2024-09-18T08:06:46.171298+0000","last_became_peered":"2024-09-18T08:06:46.171298+0000","last_unstale":"2024-09-18T08:13:15.572475+0000","last_undegraded":"2024-09-18T08:13:15.572475+0000","last_fullsized":"2024-09-18T08:13:15.572475+0000","mapping_epoch":143,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":144,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:12:31.205468+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:46.860092+0000","last_clean_scrub_stamp":"2024-09-18T08:12:31.205468+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:13:32.806252+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":179,"reported_epoch":278,"state":"active+clean","last_fresh":"2024-09-18T08:13:15.564769+0000","last_change":"2024-09-18T08:10:41.682366+0000","last_active":"2024-09-18T08:13:15.564769+0000","last_peered":"2024-09-18T08:13:15.564769+0000","last_clean":"2024-09-18T08:13:15.564769+0000","last_became_active":"2024-09-18T08:08:52.697932+0000","last_became_peered":"2024-09-18T08:08:52.697932+0000","last_unstale":"2024-09-18T08:13:15.564769+0000","last_undegraded":"2024-09-18T08:13:15.564769+0000","last_fullsized":"2024-09-18T08:13:15.564769+0000","mapping_epoch":198,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":199,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:10:41.682297+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:50.886830+0000","last_clean_scrub_stamp":"2024-09-18T08:10:41.682297+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":300,"reported_epoch":278,"state":"active+clean","last_fresh":"2024-09-18T08:13:15.562844+0000","last_change":"2024-09-18T08:11:31.707207+0000","last_active":"2024-09-18T08:13:15.562844+0000","last_peered":"2024-09-18T08:13:15.562844+0000","last_clean":"2024-09-18T08:13:15.562844+0000","last_became_active":"2024-09-18T08:06:46.137872+0000","last_became_peered":"2024-09-18T08:06:46.137872+0000","last_unstale":"2024-09-18T08:13:15.562844+0000","last_undegraded":"2024-09-18T08:13:15.562844+0000","last_fullsized":"2024-09-18T08:13:15.562844+0000","mapping_epoch":143,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":144,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:11:31.707130+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_clean_scrub_stamp":"2024-09-18T08:11:31.707130+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:13:52.311573+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,3],"acting":[2,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.4","version":"278'11911","reported_seq":12019,"reported_epoch":278,"state":"active+undersized+degraded","last_fresh":"2024-09-18T08:13:18.287239+0000","last_change":"2024-09-18T08:11:42.349543+0000","last_active":"2024-09-18T08:13:18.287239+0000","last_peered":"2024-09-18T08:13:18.287239+0000","last_clean":"2024-09-18T08:11:41.314761+0000","last_became_active":"2024-09-18T08:11:42.329000+0000","last_became_peered":"2024-09-18T08:11:42.329000+0000","last_unstale":"2024-09-18T08:13:18.287239+0000","last_undegraded":"2024-09-18T08:11:42.318915+0000","last_fullsized":"2024-09-18T08:11:42.318397+0000","mapping_epoch":244,"log_start":"258'7200","ondisk_log_start":"258'7200","created":232,"last_epoch_clean":233,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:11:08.262286+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:11:08.262286+0000","last_clean_scrub_stamp":"2024-09-18T08:11:08.262286+0000","objects_scrubbed":0,"log_size":4711,"log_dups_size":0,"ondisk_log_size":4711,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":380960768,"num_objects":5813,"num_object_clones":0,"num_object_copies":17439,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":5813,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5813,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11911,"num_write_kb":567168,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":65536,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":["4","3"],"object_location_counts":[{"shards":"3,4","objects":5813}],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.9","version":"271'18","reported_seq":254,"reported_epoch":278,"state":"active+clean","last_fresh":"2024-09-18T08:13:15.566075+0000","last_change":"2024-09-18T08:11:27.607131+0000","last_active":"2024-09-18T08:13:15.566075+0000","last_peered":"2024-09-18T08:13:15.566075+0000","last_clean":"2024-09-18T08:13:15.566075+0000","last_became_active":"2024-09-18T08:10:18.362984+0000","last_became_peered":"2024-09-18T08:10:18.362984+0000","last_unstale":"2024-09-18T08:13:15.566075+0000","last_undegraded":"2024-09-18T08:13:15.566075+0000","last_fullsized":"2024-09-18T08:13:15.566075+0000","mapping_epoch":221,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":222,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'12","last_scrub_stamp":"2024-09-18T08:11:27.607049+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_clean_scrub_stamp":"2024-09-18T08:11:27.607049+0000","objects_scrubbed":5,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":33,"num_read_kb":24,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":660,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,3],"acting":[4,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":505,"reported_epoch":278,"state":"active+undersized","last_fresh":"2024-09-18T08:13:15.571778+0000","last_change":"2024-09-18T08:11:42.338535+0000","last_active":"2024-09-18T08:13:15.571778+0000","last_peered":"2024-09-18T08:13:15.571778+0000","last_clean":"2024-09-18T08:11:40.309174+0000","last_became_active":"2024-09-18T08:11:42.338535+0000","last_became_peered":"2024-09-18T08:11:42.338535+0000","last_unstale":"2024-09-18T08:13:15.571778+0000","last_undegraded":"2024-09-18T08:13:15.571778+0000","last_fullsized":"2024-09-18T08:11:42.324265+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":182,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:10:57.579577+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:46.860092+0000","last_clean_scrub_stamp":"2024-09-18T08:10:57.579577+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:13:22.840015+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":["7","4"],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":468,"reported_epoch":278,"state":"active+undersized","last_fresh":"2024-09-18T08:13:15.562908+0000","last_change":"2024-09-18T08:11:42.325714+0000","last_active":"2024-09-18T08:13:15.562908+0000","last_peered":"2024-09-18T08:13:15.562908+0000","last_clean":"2024-09-18T08:11:40.310260+0000","last_became_active":"2024-09-18T08:11:42.325714+0000","last_became_peered":"2024-09-18T08:11:42.325714+0000","last_unstale":"2024-09-18T08:13:15.562908+0000","last_undegraded":"2024-09-18T08:13:15.562908+0000","last_fullsized":"2024-09-18T08:11:42.318417+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:10:47.799791+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:50.886830+0000","last_clean_scrub_stamp":"2024-09-18T08:10:47.799791+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:13:17.603163+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":["2","0"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":473,"reported_epoch":278,"state":"active+undersized","last_fresh":"2024-09-18T08:13:15.570358+0000","last_change":"2024-09-18T08:11:42.331389+0000","last_active":"2024-09-18T08:13:15.570358+0000","last_peered":"2024-09-18T08:13:15.570358+0000","last_clean":"2024-09-18T08:11:40.314736+0000","last_became_active":"2024-09-18T08:11:42.331389+0000","last_became_peered":"2024-09-18T08:11:42.331389+0000","last_unstale":"2024-09-18T08:13:15.570358+0000","last_undegraded":"2024-09-18T08:13:15.570358+0000","last_fullsized":"2024-09-18T08:11:42.324129+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:11:21.372405+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_clean_scrub_stamp":"2024-09-18T08:11:21.372405+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":["3","7"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.5","version":"278'11936","reported_seq":12039,"reported_epoch":278,"state":"active+clean","last_fresh":"2024-09-18T08:13:19.026493+0000","last_change":"2024-09-18T08:11:09.292021+0000","last_active":"2024-09-18T08:13:19.026493+0000","last_peered":"2024-09-18T08:13:19.026493+0000","last_clean":"2024-09-18T08:13:19.026493+0000","last_became_active":"2024-09-18T08:11:09.291723+0000","last_became_peered":"2024-09-18T08:11:09.291723+0000","last_unstale":"2024-09-18T08:13:19.026493+0000","last_undegraded":"2024-09-18T08:13:19.026493+0000","last_fullsized":"2024-09-18T08:13:19.026493+0000","mapping_epoch":232,"log_start":"254'6200","ondisk_log_start":"254'6200","created":232,"last_epoch_clean":233,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:11:08.262286+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:11:08.262286+0000","last_clean_scrub_stamp":"2024-09-18T08:11:08.262286+0000","objects_scrubbed":0,"log_size":5736,"log_dups_size":0,"ondisk_log_size":5736,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:13:53.751312+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":378142720,"num_objects":5770,"num_object_clones":0,"num_object_copies":17310,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5770,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11936,"num_write_kb":566592,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.8","version":"271'27","reported_seq":540,"reported_epoch":278,"state":"active+clean","last_fresh":"2024-09-18T08:13:15.572849+0000","last_change":"2024-09-18T08:06:24.162625+0000","last_active":"2024-09-18T08:13:15.572849+0000","last_peered":"2024-09-18T08:13:15.572849+0000","last_clean":"2024-09-18T08:13:15.572849+0000","last_became_active":"2024-09-18T08:06:24.138259+0000","last_became_peered":"2024-09-18T08:06:24.138259+0000","last_unstale":"2024-09-18T08:13:15.572849+0000","last_undegraded":"2024-09-18T08:13:15.572849+0000","last_fullsized":"2024-09-18T08:13:15.572849+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'15","last_scrub_stamp":"2024-09-18T08:04:26.859044+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_clean_scrub_stamp":"2024-09-18T08:04:26.859044+0000","objects_scrubbed":7,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:13:46.333773+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":44,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.f","version":"62'2","reported_seq":517,"reported_epoch":278,"state":"active+clean","last_fresh":"2024-09-18T08:13:15.571504+0000","last_change":"2024-09-18T08:10:53.580490+0000","last_active":"2024-09-18T08:13:15.571504+0000","last_peered":"2024-09-18T08:13:15.571504+0000","last_clean":"2024-09-18T08:13:15.571504+0000","last_became_active":"2024-09-18T08:08:18.524593+0000","last_became_peered":"2024-09-18T08:08:18.524593+0000","last_unstale":"2024-09-18T08:13:15.571504+0000","last_undegraded":"2024-09-18T08:13:15.571504+0000","last_fullsized":"2024-09-18T08:13:15.571504+0000","mapping_epoch":181,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":182,"parent":"0.0","parent_split_bits":0,"last_scrub":"62'2","last_scrub_stamp":"2024-09-18T08:10:53.580444+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:46.860092+0000","last_clean_scrub_stamp":"2024-09-18T08:10:53.580444+0000","objects_scrubbed":2,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:13:15.817988+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":4,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":184,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.9","version":"182'12","reported_seq":654,"reported_epoch":278,"state":"active+clean","last_fresh":"2024-09-18T08:13:15.571975+0000","last_change":"2024-09-18T08:09:46.742515+0000","last_active":"2024-09-18T08:13:15.571975+0000","last_peered":"2024-09-18T08:13:15.571975+0000","last_clean":"2024-09-18T08:13:15.571975+0000","last_became_active":"2024-09-18T08:08:18.524858+0000","last_became_peered":"2024-09-18T08:08:18.524858+0000","last_unstale":"2024-09-18T08:13:15.571975+0000","last_undegraded":"2024-09-18T08:13:15.571975+0000","last_fullsized":"2024-09-18T08:13:15.571975+0000","mapping_epoch":181,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":182,"parent":"0.0","parent_split_bits":0,"last_scrub":"182'12","last_scrub_stamp":"2024-09-18T08:09:46.742442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:50.886830+0000","last_clean_scrub_stamp":"2024-09-18T08:09:46.742442+0000","objects_scrubbed":1,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:14:14.803415+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":473,"reported_epoch":278,"state":"active+clean","last_fresh":"2024-09-18T08:13:15.568777+0000","last_change":"2024-09-18T08:11:23.050828+0000","last_active":"2024-09-18T08:13:15.568777+0000","last_peered":"2024-09-18T08:13:15.568777+0000","last_clean":"2024-09-18T08:13:15.568777+0000","last_became_active":"2024-09-18T08:06:24.133119+0000","last_became_peered":"2024-09-18T08:06:24.133119+0000","last_unstale":"2024-09-18T08:13:15.568777+0000","last_undegraded":"2024-09-18T08:13:15.568777+0000","last_fullsized":"2024-09-18T08:13:15.568777+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:11:23.050755+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_clean_scrub_stamp":"2024-09-18T08:11:23.050755+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":397,"reported_epoch":278,"state":"active+clean","last_fresh":"2024-09-18T08:13:15.566267+0000","last_change":"2024-09-18T08:13:06.860401+0000","last_active":"2024-09-18T08:13:15.566267+0000","last_peered":"2024-09-18T08:13:15.566267+0000","last_clean":"2024-09-18T08:13:15.566267+0000","last_became_active":"2024-09-18T08:06:24.133644+0000","last_became_peered":"2024-09-18T08:06:24.133644+0000","last_unstale":"2024-09-18T08:13:15.566267+0000","last_undegraded":"2024-09-18T08:13:15.566267+0000","last_fullsized":"2024-09-18T08:13:15.566267+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:13:06.860320+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:46.860092+0000","last_clean_scrub_stamp":"2024-09-18T08:13:06.860320+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:14:07.559037+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.17","version":"271'12","reported_seq":498,"reported_epoch":278,"state":"active+undersized+degraded","last_fresh":"2024-09-18T08:13:15.578171+0000","last_change":"2024-09-18T08:11:42.331376+0000","last_active":"2024-09-18T08:13:15.578171+0000","last_peered":"2024-09-18T08:13:15.578171+0000","last_clean":"2024-09-18T08:11:40.317526+0000","last_became_active":"2024-09-18T08:11:42.331376+0000","last_became_peered":"2024-09-18T08:11:42.331376+0000","last_unstale":"2024-09-18T08:13:15.578171+0000","last_undegraded":"2024-09-18T08:11:42.327618+0000","last_fullsized":"2024-09-18T08:11:42.327485+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'6","last_scrub_stamp":"2024-09-18T08:10:15.515914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_clean_scrub_stamp":"2024-09-18T08:10:15.515914+0000","objects_scrubbed":3,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:13:57.014260+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":18,"num_read_kb":15,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":["3","7"],"object_location_counts":[{"shards":"3,7","objects":3}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":467,"reported_epoch":278,"state":"active+undersized","last_fresh":"2024-09-18T08:13:15.570874+0000","last_change":"2024-09-18T08:11:42.331518+0000","last_active":"2024-09-18T08:13:15.570874+0000","last_peered":"2024-09-18T08:13:15.570874+0000","last_clean":"2024-09-18T08:11:40.302611+0000","last_became_active":"2024-09-18T08:11:42.331518+0000","last_became_peered":"2024-09-18T08:11:42.331518+0000","last_unstale":"2024-09-18T08:13:15.570874+0000","last_undegraded":"2024-09-18T08:13:15.570874+0000","last_fullsized":"2024-09-18T08:11:42.328308+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:10:49.169375+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:50.886830+0000","last_clean_scrub_stamp":"2024-09-18T08:10:49.169375+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:13:34.218350+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":["5","3"],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":504,"reported_epoch":278,"state":"active+clean","last_fresh":"2024-09-18T08:13:15.569081+0000","last_change":"2024-09-18T08:12:54.661635+0000","last_active":"2024-09-18T08:13:15.569081+0000","last_peered":"2024-09-18T08:13:15.569081+0000","last_clean":"2024-09-18T08:13:15.569081+0000","last_became_active":"2024-09-18T08:08:18.525260+0000","last_became_peered":"2024-09-18T08:08:18.525260+0000","last_unstale":"2024-09-18T08:13:15.569081+0000","last_undegraded":"2024-09-18T08:13:15.569081+0000","last_fullsized":"2024-09-18T08:13:15.569081+0000","mapping_epoch":181,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":182,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:12:54.661578+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_clean_scrub_stamp":"2024-09-18T08:12:54.661578+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:14:04.552010+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"271'15","reported_seq":492,"reported_epoch":278,"state":"active+clean","last_fresh":"2024-09-18T08:13:15.565024+0000","last_change":"2024-09-18T08:09:44.116395+0000","last_active":"2024-09-18T08:13:15.565024+0000","last_peered":"2024-09-18T08:13:15.565024+0000","last_clean":"2024-09-18T08:13:15.565024+0000","last_became_active":"2024-09-18T08:02:49.891567+0000","last_became_peered":"2024-09-18T08:02:49.891567+0000","last_unstale":"2024-09-18T08:13:15.565024+0000","last_undegraded":"2024-09-18T08:13:15.565024+0000","last_fullsized":"2024-09-18T08:13:15.565024+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'9","last_scrub_stamp":"2024-09-18T08:09:44.116308+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_clean_scrub_stamp":"2024-09-18T08:09:44.116308+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:13:19.711352+0000","scrub_duration":43,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":222,"reported_epoch":278,"state":"active+clean","last_fresh":"2024-09-18T08:13:15.566277+0000","last_change":"2024-09-18T08:12:03.433488+0000","last_active":"2024-09-18T08:13:15.566277+0000","last_peered":"2024-09-18T08:13:15.566277+0000","last_clean":"2024-09-18T08:13:15.566277+0000","last_became_active":"2024-09-18T08:08:18.545953+0000","last_became_peered":"2024-09-18T08:08:18.545953+0000","last_unstale":"2024-09-18T08:13:15.566277+0000","last_undegraded":"2024-09-18T08:13:15.566277+0000","last_fullsized":"2024-09-18T08:13:15.566277+0000","mapping_epoch":181,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":182,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:12:03.433407+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:46.860092+0000","last_clean_scrub_stamp":"2024-09-18T08:12:03.433407+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":468,"reported_epoch":278,"state":"active+undersized","last_fresh":"2024-09-18T08:13:15.573434+0000","last_change":"2024-09-18T08:11:42.328676+0000","last_active":"2024-09-18T08:13:15.573434+0000","last_peered":"2024-09-18T08:13:15.573434+0000","last_clean":"2024-09-18T08:11:40.317184+0000","last_became_active":"2024-09-18T08:11:42.328676+0000","last_became_peered":"2024-09-18T08:11:42.328676+0000","last_unstale":"2024-09-18T08:13:15.573434+0000","last_undegraded":"2024-09-18T08:13:15.573434+0000","last_fullsized":"2024-09-18T08:11:42.326031+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:11:17.424701+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:50.886830+0000","last_clean_scrub_stamp":"2024-09-18T08:11:17.424701+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:14:01.930131+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":["3","7"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":475,"reported_epoch":278,"state":"active+clean","last_fresh":"2024-09-18T08:13:15.565516+0000","last_change":"2024-09-18T08:10:42.731922+0000","last_active":"2024-09-18T08:13:15.565516+0000","last_peered":"2024-09-18T08:13:15.565516+0000","last_clean":"2024-09-18T08:13:15.565516+0000","last_became_active":"2024-09-18T08:08:18.511019+0000","last_became_peered":"2024-09-18T08:08:18.511019+0000","last_unstale":"2024-09-18T08:13:15.565516+0000","last_undegraded":"2024-09-18T08:13:15.565516+0000","last_fullsized":"2024-09-18T08:13:15.565516+0000","mapping_epoch":181,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":182,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:10:42.731845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_clean_scrub_stamp":"2024-09-18T08:10:42.731845+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:13:20.564381+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.15","version":"271'15","reported_seq":492,"reported_epoch":278,"state":"active+clean","last_fresh":"2024-09-18T08:13:15.572613+0000","last_change":"2024-09-18T08:12:58.225479+0000","last_active":"2024-09-18T08:13:15.572613+0000","last_peered":"2024-09-18T08:13:15.572613+0000","last_clean":"2024-09-18T08:13:15.572613+0000","last_became_active":"2024-09-18T08:02:49.890497+0000","last_became_peered":"2024-09-18T08:02:49.890497+0000","last_unstale":"2024-09-18T08:13:15.572613+0000","last_undegraded":"2024-09-18T08:13:15.572613+0000","last_fullsized":"2024-09-18T08:13:15.572613+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"271'15","last_scrub_stamp":"2024-09-18T08:12:58.225295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_clean_scrub_stamp":"2024-09-18T08:12:58.225295+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:14:10.026769+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":479,"reported_epoch":278,"state":"active+clean","last_fresh":"2024-09-18T08:13:15.564695+0000","last_change":"2024-09-18T08:09:46.035501+0000","last_active":"2024-09-18T08:13:15.564695+0000","last_peered":"2024-09-18T08:13:15.564695+0000","last_clean":"2024-09-18T08:13:15.564695+0000","last_became_active":"2024-09-18T08:02:47.875983+0000","last_became_peered":"2024-09-18T08:02:47.875983+0000","last_unstale":"2024-09-18T08:13:15.564695+0000","last_undegraded":"2024-09-18T08:13:15.564695+0000","last_fullsized":"2024-09-18T08:13:15.564695+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:09:46.035332+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:46.860092+0000","last_clean_scrub_stamp":"2024-09-18T08:09:46.035332+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:14:14.822467+0000","scrub_duration":18,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.14","version":"199'12","reported_seq":631,"reported_epoch":278,"state":"active+clean","last_fresh":"2024-09-18T08:13:15.570679+0000","last_change":"2024-09-18T08:12:01.112383+0000","last_active":"2024-09-18T08:13:15.570679+0000","last_peered":"2024-09-18T08:13:15.570679+0000","last_clean":"2024-09-18T08:13:15.570679+0000","last_became_active":"2024-09-18T08:08:52.696201+0000","last_became_peered":"2024-09-18T08:08:52.696201+0000","last_unstale":"2024-09-18T08:13:15.570679+0000","last_undegraded":"2024-09-18T08:13:15.570679+0000","last_fullsized":"2024-09-18T08:13:15.570679+0000","mapping_epoch":198,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":199,"parent":"0.0","parent_split_bits":0,"last_scrub":"199'12","last_scrub_stamp":"2024-09-18T08:12:01.112299+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:50.886830+0000","last_clean_scrub_stamp":"2024-09-18T08:12:01.112299+0000","objects_scrubbed":1,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:13:21.584055+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":214,"reported_epoch":278,"state":"active+clean","last_fresh":"2024-09-18T08:13:15.567448+0000","last_change":"2024-09-18T08:12:06.521847+0000","last_active":"2024-09-18T08:13:15.567448+0000","last_peered":"2024-09-18T08:13:15.567448+0000","last_clean":"2024-09-18T08:13:15.567448+0000","last_became_active":"2024-09-18T08:08:18.522483+0000","last_became_peered":"2024-09-18T08:08:18.522483+0000","last_unstale":"2024-09-18T08:13:15.567448+0000","last_undegraded":"2024-09-18T08:13:15.567448+0000","last_fullsized":"2024-09-18T08:13:15.567448+0000","mapping_epoch":181,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":182,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:12:06.521774+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_clean_scrub_stamp":"2024-09-18T08:12:06.521774+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.14","version":"271'20","reported_seq":517,"reported_epoch":278,"state":"active+clean","last_fresh":"2024-09-18T08:13:15.572631+0000","last_change":"2024-09-18T08:12:39.946516+0000","last_active":"2024-09-18T08:13:15.572631+0000","last_peered":"2024-09-18T08:13:15.572631+0000","last_clean":"2024-09-18T08:13:15.572631+0000","last_became_active":"2024-09-18T08:04:18.150968+0000","last_became_peered":"2024-09-18T08:04:18.150968+0000","last_unstale":"2024-09-18T08:13:15.572631+0000","last_undegraded":"2024-09-18T08:13:15.572631+0000","last_fullsized":"2024-09-18T08:13:15.572631+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'10","last_scrub_stamp":"2024-09-18T08:12:39.946462+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_clean_scrub_stamp":"2024-09-18T08:12:39.946462+0000","objects_scrubbed":5,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:13:55.625608+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":25,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,7],"acting":[3,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":493,"reported_epoch":278,"state":"active+clean","last_fresh":"2024-09-18T08:13:15.570084+0000","last_change":"2024-09-18T08:10:50.584750+0000","last_active":"2024-09-18T08:13:15.570084+0000","last_peered":"2024-09-18T08:13:15.570084+0000","last_clean":"2024-09-18T08:13:15.570084+0000","last_became_active":"2024-09-18T08:08:18.524218+0000","last_became_peered":"2024-09-18T08:08:18.524218+0000","last_unstale":"2024-09-18T08:13:15.570084+0000","last_undegraded":"2024-09-18T08:13:15.570084+0000","last_fullsized":"2024-09-18T08:13:15.570084+0000","mapping_epoch":181,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":182,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:10:50.584652+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:46.860092+0000","last_clean_scrub_stamp":"2024-09-18T08:10:50.584652+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:13:25.880092+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.15","version":"245'9","reported_seq":611,"reported_epoch":278,"state":"active+undersized+degraded","last_fresh":"2024-09-18T08:13:15.572380+0000","last_change":"2024-09-18T08:11:42.329955+0000","last_active":"2024-09-18T08:13:15.572380+0000","last_peered":"2024-09-18T08:13:15.572380+0000","last_clean":"2024-09-18T08:11:40.302227+0000","last_became_active":"2024-09-18T08:11:42.329955+0000","last_became_peered":"2024-09-18T08:11:42.329955+0000","last_unstale":"2024-09-18T08:13:15.572380+0000","last_undegraded":"2024-09-18T08:11:42.324112+0000","last_fullsized":"2024-09-18T08:11:42.324002+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'8","last_scrub_stamp":"2024-09-18T08:10:54.148986+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:50.886830+0000","last_clean_scrub_stamp":"2024-09-18T08:10:54.148986+0000","objects_scrubbed":1,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:13:38.308095+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":["5","0"],"object_location_counts":[{"shards":"0,5","objects":1}],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":460,"reported_epoch":278,"state":"active+clean","last_fresh":"2024-09-18T08:13:15.564469+0000","last_change":"2024-09-18T08:09:42.047337+0000","last_active":"2024-09-18T08:13:15.564469+0000","last_peered":"2024-09-18T08:13:15.564469+0000","last_clean":"2024-09-18T08:13:15.564469+0000","last_became_active":"2024-09-18T08:02:53.912112+0000","last_became_peered":"2024-09-18T08:02:53.912112+0000","last_unstale":"2024-09-18T08:13:15.564469+0000","last_undegraded":"2024-09-18T08:13:15.564469+0000","last_fullsized":"2024-09-18T08:13:15.564469+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:09:42.047270+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_clean_scrub_stamp":"2024-09-18T08:09:42.047270+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:13:15.697719+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.13","version":"271'19","reported_seq":242,"reported_epoch":278,"state":"active+undersized+degraded","last_fresh":"2024-09-18T08:13:15.565444+0000","last_change":"2024-09-18T08:11:42.324665+0000","last_active":"2024-09-18T08:13:15.565444+0000","last_peered":"2024-09-18T08:13:15.565444+0000","last_clean":"2024-09-18T08:11:40.307120+0000","last_became_active":"2024-09-18T08:11:42.324665+0000","last_became_peered":"2024-09-18T08:11:42.324665+0000","last_unstale":"2024-09-18T08:13:15.565444+0000","last_undegraded":"2024-09-18T08:11:42.319891+0000","last_fullsized":"2024-09-18T08:11:42.319779+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":182,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'11","last_scrub_stamp":"2024-09-18T08:11:22.565415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_clean_scrub_stamp":"2024-09-18T08:11:22.565415+0000","objects_scrubbed":5,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":5,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":31,"num_read_kb":24,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":["4","6"],"object_location_counts":[{"shards":"4,6","objects":5}],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":214,"reported_epoch":278,"state":"active+clean","last_fresh":"2024-09-18T08:13:15.566173+0000","last_change":"2024-09-18T08:10:58.402364+0000","last_active":"2024-09-18T08:13:15.566173+0000","last_peered":"2024-09-18T08:13:15.566173+0000","last_clean":"2024-09-18T08:13:15.566173+0000","last_became_active":"2024-09-18T08:08:18.622893+0000","last_became_peered":"2024-09-18T08:08:18.622893+0000","last_unstale":"2024-09-18T08:13:15.566173+0000","last_undegraded":"2024-09-18T08:13:15.566173+0000","last_fullsized":"2024-09-18T08:13:15.566173+0000","mapping_epoch":181,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":182,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:10:58.402308+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:46.860092+0000","last_clean_scrub_stamp":"2024-09-18T08:10:58.402308+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:13:15.643515+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":76,"reported_epoch":278,"state":"active+undersized","last_fresh":"2024-09-18T08:13:15.568926+0000","last_change":"2024-09-18T08:11:42.332003+0000","last_active":"2024-09-18T08:13:15.568926+0000","last_peered":"2024-09-18T08:13:15.568926+0000","last_clean":"2024-09-18T08:02:50.886830+0000","last_became_active":"2024-09-18T08:11:42.332003+0000","last_became_peered":"2024-09-18T08:11:42.332003+0000","last_unstale":"2024-09-18T08:13:15.568926+0000","last_undegraded":"2024-09-18T08:13:15.568926+0000","last_fullsized":"2024-09-18T08:11:42.330392+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:11:26.316948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:50.886830+0000","last_clean_scrub_stamp":"2024-09-18T08:11:26.316948+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":["5","3"],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":469,"reported_epoch":278,"state":"active+clean","last_fresh":"2024-09-18T08:13:15.570191+0000","last_change":"2024-09-18T08:10:20.525295+0000","last_active":"2024-09-18T08:13:15.570191+0000","last_peered":"2024-09-18T08:13:15.570191+0000","last_clean":"2024-09-18T08:13:15.570191+0000","last_became_active":"2024-09-18T08:02:53.911429+0000","last_became_peered":"2024-09-18T08:02:53.911429+0000","last_unstale":"2024-09-18T08:13:15.570191+0000","last_undegraded":"2024-09-18T08:13:15.570191+0000","last_fullsized":"2024-09-18T08:13:15.570191+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:10:20.525222+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_clean_scrub_stamp":"2024-09-18T08:10:20.525222+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:13:27.769169+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.12","version":"271'15","reported_seq":121,"reported_epoch":278,"state":"active+undersized+degraded","last_fresh":"2024-09-18T08:13:15.564767+0000","last_change":"2024-09-18T08:11:42.329218+0000","last_active":"2024-09-18T08:13:15.564767+0000","last_peered":"2024-09-18T08:13:15.564767+0000","last_clean":"2024-09-18T08:02:54.300830+0000","last_became_active":"2024-09-18T08:11:42.329218+0000","last_became_peered":"2024-09-18T08:11:42.329218+0000","last_unstale":"2024-09-18T08:13:15.564767+0000","last_undegraded":"2024-09-18T08:11:42.318592+0000","last_fullsized":"2024-09-18T08:11:42.318478+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'9","last_scrub_stamp":"2024-09-18T08:10:06.582804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_clean_scrub_stamp":"2024-09-18T08:10:06.582804+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:13:59.326333+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":25,"num_read_kb":19,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":["2","6"],"object_location_counts":[{"shards":"2,6","objects":4}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":501,"reported_epoch":278,"state":"active+clean","last_fresh":"2024-09-18T08:13:15.572458+0000","last_change":"2024-09-18T08:12:06.859635+0000","last_active":"2024-09-18T08:13:15.572458+0000","last_peered":"2024-09-18T08:13:15.572458+0000","last_clean":"2024-09-18T08:13:15.572458+0000","last_became_active":"2024-09-18T08:08:18.525527+0000","last_became_peered":"2024-09-18T08:08:18.525527+0000","last_unstale":"2024-09-18T08:13:15.572458+0000","last_undegraded":"2024-09-18T08:13:15.572458+0000","last_fullsized":"2024-09-18T08:13:15.572458+0000","mapping_epoch":181,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":182,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:12:06.854036+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:46.860092+0000","last_clean_scrub_stamp":"2024-09-18T08:12:06.854036+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:13:23.581731+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":468,"reported_epoch":278,"state":"active+undersized","last_fresh":"2024-09-18T08:13:15.570552+0000","last_change":"2024-09-18T08:11:42.328480+0000","last_active":"2024-09-18T08:13:15.570552+0000","last_peered":"2024-09-18T08:13:15.570552+0000","last_clean":"2024-09-18T08:11:40.314825+0000","last_became_active":"2024-09-18T08:11:42.328480+0000","last_became_peered":"2024-09-18T08:11:42.328480+0000","last_unstale":"2024-09-18T08:13:15.570552+0000","last_undegraded":"2024-09-18T08:13:15.570552+0000","last_fullsized":"2024-09-18T08:11:42.323648+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:11:30.302179+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:50.886830+0000","last_clean_scrub_stamp":"2024-09-18T08:11:30.302179+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":["3","0"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.10","version":"69'1","reported_seq":452,"reported_epoch":278,"state":"active+undersized+degraded","last_fresh":"2024-09-18T08:13:15.565833+0000","last_change":"2024-09-18T08:11:42.328627+0000","last_active":"2024-09-18T08:13:15.565833+0000","last_peered":"2024-09-18T08:13:15.565833+0000","last_clean":"2024-09-18T08:11:40.307990+0000","last_became_active":"2024-09-18T08:11:42.328627+0000","last_became_peered":"2024-09-18T08:11:42.328627+0000","last_unstale":"2024-09-18T08:13:15.565833+0000","last_undegraded":"2024-09-18T08:11:42.318971+0000","last_fullsized":"2024-09-18T08:11:42.318795+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'1","last_scrub_stamp":"2024-09-18T08:10:17.878965+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_clean_scrub_stamp":"2024-09-18T08:10:17.878965+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:13:57.798003+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":["0","5"],"object_location_counts":[{"shards":"0,5","objects":1}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.11","version":"271'19","reported_seq":553,"reported_epoch":278,"state":"active+clean","last_fresh":"2024-09-18T08:13:15.570517+0000","last_change":"2024-09-18T08:11:22.332386+0000","last_active":"2024-09-18T08:13:15.570517+0000","last_peered":"2024-09-18T08:13:15.570517+0000","last_clean":"2024-09-18T08:13:15.570517+0000","last_became_active":"2024-09-18T08:06:24.132416+0000","last_became_peered":"2024-09-18T08:06:24.132416+0000","last_unstale":"2024-09-18T08:13:15.570517+0000","last_undegraded":"2024-09-18T08:13:15.570517+0000","last_fullsized":"2024-09-18T08:13:15.570517+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'11","last_scrub_stamp":"2024-09-18T08:11:22.332279+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_clean_scrub_stamp":"2024-09-18T08:11:22.332279+0000","objects_scrubbed":5,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:14:11.722874+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":32,"num_read_kb":25,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":475,"reported_epoch":278,"state":"active+undersized","last_fresh":"2024-09-18T08:13:15.573435+0000","last_change":"2024-09-18T08:11:42.334948+0000","last_active":"2024-09-18T08:13:15.573435+0000","last_peered":"2024-09-18T08:13:15.573435+0000","last_clean":"2024-09-18T08:11:40.302665+0000","last_became_active":"2024-09-18T08:11:42.334948+0000","last_became_peered":"2024-09-18T08:11:42.334948+0000","last_unstale":"2024-09-18T08:13:15.573435+0000","last_undegraded":"2024-09-18T08:13:15.573435+0000","last_fullsized":"2024-09-18T08:11:42.330896+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:10:53.145315+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:46.860092+0000","last_clean_scrub_stamp":"2024-09-18T08:10:53.145315+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:13:26.203508+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":["5","7"],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":507,"reported_epoch":278,"state":"active+clean","last_fresh":"2024-09-18T08:13:15.568671+0000","last_change":"2024-09-18T08:10:52.577620+0000","last_active":"2024-09-18T08:13:15.568671+0000","last_peered":"2024-09-18T08:13:15.568671+0000","last_clean":"2024-09-18T08:13:15.568671+0000","last_became_active":"2024-09-18T08:08:18.524966+0000","last_became_peered":"2024-09-18T08:08:18.524966+0000","last_unstale":"2024-09-18T08:13:15.568671+0000","last_undegraded":"2024-09-18T08:13:15.568671+0000","last_fullsized":"2024-09-18T08:13:15.568671+0000","mapping_epoch":181,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":182,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:10:52.577568+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:50.886830+0000","last_clean_scrub_stamp":"2024-09-18T08:10:52.577568+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:13:24.877607+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":474,"reported_epoch":278,"state":"active+clean","last_fresh":"2024-09-18T08:13:15.570791+0000","last_change":"2024-09-18T08:11:29.344444+0000","last_active":"2024-09-18T08:13:15.570791+0000","last_peered":"2024-09-18T08:13:15.570791+0000","last_clean":"2024-09-18T08:13:15.570791+0000","last_became_active":"2024-09-18T08:06:24.128582+0000","last_became_peered":"2024-09-18T08:06:24.128582+0000","last_unstale":"2024-09-18T08:13:15.570791+0000","last_undegraded":"2024-09-18T08:13:15.570791+0000","last_fullsized":"2024-09-18T08:13:15.570791+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:11:29.344382+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_clean_scrub_stamp":"2024-09-18T08:11:29.344382+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.10","version":"271'8","reported_seq":510,"reported_epoch":278,"state":"active+undersized+degraded","last_fresh":"2024-09-18T08:13:15.570432+0000","last_change":"2024-09-18T08:11:42.340283+0000","last_active":"2024-09-18T08:13:15.570432+0000","last_peered":"2024-09-18T08:13:15.570432+0000","last_clean":"2024-09-18T08:11:40.313525+0000","last_became_active":"2024-09-18T08:11:42.340283+0000","last_became_peered":"2024-09-18T08:11:42.340283+0000","last_unstale":"2024-09-18T08:13:15.570432+0000","last_undegraded":"2024-09-18T08:11:42.331866+0000","last_fullsized":"2024-09-18T08:11:42.325406+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'4","last_scrub_stamp":"2024-09-18T08:10:18.542984+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_clean_scrub_stamp":"2024-09-18T08:10:18.542984+0000","objects_scrubbed":2,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:13:32.840472+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":12,"num_read_kb":10,"num_write":8,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":["3","6"],"object_location_counts":[{"shards":"3,6","objects":2}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":479,"reported_epoch":278,"state":"active+clean","last_fresh":"2024-09-18T08:13:15.564879+0000","last_change":"2024-09-18T08:10:13.820205+0000","last_active":"2024-09-18T08:13:15.564879+0000","last_peered":"2024-09-18T08:13:15.564879+0000","last_clean":"2024-09-18T08:13:15.564879+0000","last_became_active":"2024-09-18T08:02:47.877616+0000","last_became_peered":"2024-09-18T08:02:47.877616+0000","last_unstale":"2024-09-18T08:13:15.564879+0000","last_undegraded":"2024-09-18T08:13:15.564879+0000","last_fullsized":"2024-09-18T08:13:15.564879+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:10:13.820142+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:46.860092+0000","last_clean_scrub_stamp":"2024-09-18T08:10:13.820142+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:13:21.715485+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":214,"reported_epoch":278,"state":"active+clean","last_fresh":"2024-09-18T08:13:15.566441+0000","last_change":"2024-09-18T08:11:19.524308+0000","last_active":"2024-09-18T08:13:15.566441+0000","last_peered":"2024-09-18T08:13:15.566441+0000","last_clean":"2024-09-18T08:13:15.566441+0000","last_became_active":"2024-09-18T08:08:18.545361+0000","last_became_peered":"2024-09-18T08:08:18.545361+0000","last_unstale":"2024-09-18T08:13:15.566441+0000","last_undegraded":"2024-09-18T08:13:15.566441+0000","last_fullsized":"2024-09-18T08:13:15.566441+0000","mapping_epoch":181,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":182,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:11:19.524235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:50.886830+0000","last_clean_scrub_stamp":"2024-09-18T08:11:19.524235+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,7],"acting":[4,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":482,"reported_epoch":278,"state":"active+clean","last_fresh":"2024-09-18T08:13:15.568637+0000","last_change":"2024-09-18T08:09:34.607443+0000","last_active":"2024-09-18T08:13:15.568637+0000","last_peered":"2024-09-18T08:13:15.568637+0000","last_clean":"2024-09-18T08:13:15.568637+0000","last_became_active":"2024-09-18T08:08:18.525552+0000","last_became_peered":"2024-09-18T08:08:18.525552+0000","last_unstale":"2024-09-18T08:13:15.568637+0000","last_undegraded":"2024-09-18T08:13:15.568637+0000","last_fullsized":"2024-09-18T08:13:15.568637+0000","mapping_epoch":181,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":182,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:09:34.607374+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_clean_scrub_stamp":"2024-09-18T08:09:34.607374+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:13:49.631974+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,4],"acting":[7,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":322,"reported_epoch":278,"state":"active+undersized","last_fresh":"2024-09-18T08:13:15.572283+0000","last_change":"2024-09-18T08:11:42.341016+0000","last_active":"2024-09-18T08:13:15.572283+0000","last_peered":"2024-09-18T08:13:15.572283+0000","last_clean":"2024-09-18T08:08:15.505600+0000","last_became_active":"2024-09-18T08:11:42.341016+0000","last_became_peered":"2024-09-18T08:11:42.341016+0000","last_unstale":"2024-09-18T08:13:15.572283+0000","last_undegraded":"2024-09-18T08:13:15.572283+0000","last_fullsized":"2024-09-18T08:11:42.325794+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":182,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:10:52.641537+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_clean_scrub_stamp":"2024-09-18T08:10:52.641537+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:13:25.169621+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":["5","4"],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":468,"reported_epoch":278,"state":"active+undersized","last_fresh":"2024-09-18T08:13:15.570889+0000","last_change":"2024-09-18T08:11:42.342099+0000","last_active":"2024-09-18T08:13:15.570889+0000","last_peered":"2024-09-18T08:13:15.570889+0000","last_clean":"2024-09-18T08:11:40.314449+0000","last_became_active":"2024-09-18T08:11:42.342099+0000","last_became_peered":"2024-09-18T08:11:42.342099+0000","last_unstale":"2024-09-18T08:13:15.570889+0000","last_undegraded":"2024-09-18T08:13:15.570889+0000","last_fullsized":"2024-09-18T08:11:42.325904+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:07:17.277894+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:46.860092+0000","last_clean_scrub_stamp":"2024-09-18T08:07:17.277894+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:13:48.047234+0000","scrub_duration":16,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":["3","0"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"271'19","reported_seq":477,"reported_epoch":278,"state":"active+undersized+degraded","last_fresh":"2024-09-18T08:13:15.573627+0000","last_change":"2024-09-18T08:11:42.334941+0000","last_active":"2024-09-18T08:13:15.573627+0000","last_peered":"2024-09-18T08:13:15.573627+0000","last_clean":"2024-09-18T08:11:40.304598+0000","last_became_active":"2024-09-18T08:11:42.334941+0000","last_became_peered":"2024-09-18T08:11:42.334941+0000","last_unstale":"2024-09-18T08:13:15.573627+0000","last_undegraded":"2024-09-18T08:11:42.328266+0000","last_fullsized":"2024-09-18T08:11:42.327013+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'11","last_scrub_stamp":"2024-09-18T08:10:50.208543+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_clean_scrub_stamp":"2024-09-18T08:10:50.208543+0000","objects_scrubbed":5,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:13:36.350714+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":5,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":31,"num_read_kb":24,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":["5","6"],"object_location_counts":[{"shards":"5,6","objects":5}],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":486,"reported_epoch":278,"state":"active+clean","last_fresh":"2024-09-18T08:13:15.564621+0000","last_change":"2024-09-18T08:12:31.664510+0000","last_active":"2024-09-18T08:13:15.564621+0000","last_peered":"2024-09-18T08:13:15.564621+0000","last_clean":"2024-09-18T08:13:15.564621+0000","last_became_active":"2024-09-18T08:08:52.700595+0000","last_became_peered":"2024-09-18T08:08:52.700595+0000","last_unstale":"2024-09-18T08:13:15.564621+0000","last_undegraded":"2024-09-18T08:13:15.564621+0000","last_fullsized":"2024-09-18T08:13:15.564621+0000","mapping_epoch":198,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":199,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:12:31.664456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:50.886830+0000","last_clean_scrub_stamp":"2024-09-18T08:12:31.664456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:13:44.437533+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":9,"num_pg":16,"stat_sum":{"num_bytes":6123683864,"num_objects":93441,"num_object_clones":0,"num_object_copies":280323,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":52930,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":93441,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":190325,"num_write_kb":9080449,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":65536,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":16796098560,"data_stored":16796090416,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":79725,"ondisk_log_size":79725,"up":39,"acting":39,"num_store_stats":8},{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":416,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":86,"acting":86,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":87,"ondisk_log_size":87,"up":82,"acting":82,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":178,"num_object_clones":0,"num_object_copies":534,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":91,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":178,"num_whiteouts":0,"num_read":1110,"num_read_kb":867,"num_write":703,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":178,"num_bytes_recovered":3784,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":417792,"data_stored":11106,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":685,"ondisk_log_size":685,"up":79,"acting":79,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":1613,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":24,"num_read_kb":24,"num_write":10,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":2698,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":4839,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":6,"ondisk_log_size":6,"up":85,"acting":85,"num_store_stats":8},{"poolid":2,"num_pg":3,"stat_sum":{"num_bytes":408,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":103,"num_read_kb":98,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":38,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":8,"ondisk_log_size":8,"up":7,"acting":7,"num_store_stats":7},{"poolid":1,"num_pg":11,"stat_sum":{"num_bytes":644038,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":10,"num_read_kb":20,"num_write":11,"num_write_kb":148,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":53670,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":869,"ondisk_log_size":869,"up":32,"acting":32,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":56,"seq":240518168706,"num_pgs":66,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":3833844,"kb_used_data":3410852,"kb_used_omap":1,"kb_used_meta":422654,"kb_avail":89911308,"statfs":{"total":95995035648,"available":92069179392,"internally_reserved":0,"allocated":3492712448,"data_stored":3492084127,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":432798154},"hb_peers":[0,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":47,"seq":201863463048,"num_pgs":51,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1474972,"kb_used_data":1128652,"kb_used_omap":1,"kb_used_meta":346302,"kb_avail":92270180,"statfs":{"total":95995035648,"available":94484664320,"internally_reserved":0,"allocated":1155739648,"data_stored":1155134794,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":354613702},"hb_peers":[0,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":39,"seq":167503724686,"num_pgs":61,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":3821048,"kb_used_data":3384488,"kb_used_omap":1,"kb_used_meta":436542,"kb_avail":89924104,"statfs":{"total":95995035648,"available":92082282496,"internally_reserved":0,"allocated":3465715712,"data_stored":3465083782,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":447019462},"hb_peers":[0,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":32,"seq":137438953619,"num_pgs":64,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1507964,"kb_used_data":1139436,"kb_used_omap":1,"kb_used_meta":368510,"kb_avail":92237188,"statfs":{"total":95995035648,"available":94450880512,"internally_reserved":0,"allocated":1166782464,"data_stored":1166146440,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":377354693},"hb_peers":[0,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":26,"seq":111669149850,"num_pgs":69,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":3517892,"kb_used_data":2995188,"kb_used_omap":1,"kb_used_meta":522622,"kb_avail":90227260,"statfs":{"total":95995035648,"available":92392714240,"internally_reserved":0,"allocated":3067072512,"data_stored":3066428133,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":535165384},"hb_peers":[0,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411488,"num_pgs":46,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1548844,"kb_used_data":1130332,"kb_used_omap":1,"kb_used_meta":418494,"kb_avail":92196308,"statfs":{"total":95995035648,"available":94409019392,"internally_reserved":0,"allocated":1157459968,"data_stored":1156840188,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":428538309},"hb_peers":[0,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574992,"num_pgs":64,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2106904,"kb_used_data":1782792,"kb_used_omap":1,"kb_used_meta":324094,"kb_avail":91638248,"statfs":{"total":95995035648,"available":93837565952,"internally_reserved":0,"allocated":1825579008,"data_stored":1825118505,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":331872709},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738539,"num_pgs":53,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1855292,"kb_used_data":1490924,"kb_used_omap":1,"kb_used_meta":364350,"kb_avail":91889860,"statfs":{"total":95995035648,"available":94095216640,"internally_reserved":0,"allocated":1526706176,"data_stored":1526070354,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":373094853},"hb_peers":[2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":266240,"data_stored":262688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1131,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1521,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":436,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":46,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1177,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":436,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":69632,"data_stored":1788,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1128,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":1898,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1474,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1210,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1034,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1254,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":403,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":416,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":403,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":1519648768,"data_stored":1519648768,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":1860698112,"data_stored":1860698112,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":1148846080,"data_stored":1148846080,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":3052208128,"data_stored":3052208128,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":1153826816,"data_stored":1153826816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":3447263232,"data_stored":3447259160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":1151008768,"data_stored":1151008768,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":3462598656,"data_stored":3462594584,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T08:13:21.860 INFO:tasks.thrashosds.thrasher:Setting 6.3 to [5, 3, 2] 2024-09-18T08:13:21.860 INFO:tasks.thrashosds.thrasher:cmd ['osd', 'pg-upmap', '6.3', '5', '3', '2'] 2024-09-18T08:13:21.860 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid 779349ca-7593-11ef-bceb-c7b262605968 -- ceph osd pg-upmap 6.3 5 3 2 2024-09-18T08:13:21.874 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/779349ca-7593-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T08:13:21.883 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:13:21.883 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:13:21.883 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0' 2024-09-18T08:13:21.984 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4 2024-09-18T08:13:22.018 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:13:22.019 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:13:22.019 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4' 2024-09-18T08:13:22.120 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5 2024-09-18T08:13:22.167 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:13:22.168 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:13:22.168 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5' 2024-09-18T08:13:22.269 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7 2024-09-18T08:13:22.306 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:13:22.306 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:13:22.307 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7' 2024-09-18T08:13:22.408 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5 2024-09-18T08:13:22.439 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:13:22.440 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:13:22.441 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5' 2024-09-18T08:13:22.462 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/779349ca-7593-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T08:13:22.525 INFO:teuthology.orchestra.run.smithi007.stdout:{ 2024-09-18T08:13:22.525 INFO:teuthology.orchestra.run.smithi007.stdout: "ops": [], 2024-09-18T08:13:22.525 INFO:teuthology.orchestra.run.smithi007.stdout: "complaint_time": 30, 2024-09-18T08:13:22.525 INFO:teuthology.orchestra.run.smithi007.stdout: "num_blocked_ops": 0 2024-09-18T08:13:22.525 INFO:teuthology.orchestra.run.smithi007.stdout:} 2024-09-18T08:13:22.534 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/779349ca-7593-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T08:13:22.541 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2 2024-09-18T08:13:22.585 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:13:22.585 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:13:22.586 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2' 2024-09-18T08:13:22.614 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:13:22 smithi007 ceph-mon[29459]: from='client.25471 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-18T08:13:22.614 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:13:22 smithi007 ceph-mon[29459]: pgmap v511: 158 pgs: 30 active+undersized, 34 active+undersized+degraded, 94 active+clean; 5.7 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 732 B/s rd, 0 B/s wr, 1.03k op/s; 53030/280920 objects degraded (18.877%) 2024-09-18T08:13:22.614 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:13:22 smithi007 ceph-mon[39249]: from='client.25471 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-18T08:13:22.615 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:13:22 smithi007 ceph-mon[39249]: pgmap v511: 158 pgs: 30 active+undersized, 34 active+undersized+degraded, 94 active+clean; 5.7 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 732 B/s rd, 0 B/s wr, 1.03k op/s; 53030/280920 objects degraded (18.877%) 2024-09-18T08:13:22.686 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4 2024-09-18T08:13:22.725 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:13:22.726 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:13:22.727 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4' 2024-09-18T08:13:22.827 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7 2024-09-18T08:13:22.870 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:13:22.870 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:13:22.871 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7' 2024-09-18T08:13:22.905 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:13:22 smithi050 ceph-mon[32371]: from='client.25471 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-18T08:13:22.905 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:13:22 smithi050 ceph-mon[32371]: pgmap v511: 158 pgs: 30 active+undersized, 34 active+undersized+degraded, 94 active+clean; 5.7 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 732 B/s rd, 0 B/s wr, 1.03k op/s; 53030/280920 objects degraded (18.877%) 2024-09-18T08:13:22.972 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0 2024-09-18T08:13:23.004 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:13:23.005 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:13:23.005 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0' 2024-09-18T08:13:23.106 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3 2024-09-18T08:13:23.137 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:13:23.138 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:13:23.138 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3' 2024-09-18T08:13:23.170 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid 779349ca-7593-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-18T08:13:23.239 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4 2024-09-18T08:13:23.276 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:13:23.277 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:13:23.277 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4' 2024-09-18T08:13:23.340 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid 779349ca-7593-11ef-bceb-c7b262605968 -- ceph daemon osd.3 dump_historic_ops 2024-09-18T08:13:23.378 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5 2024-09-18T08:13:23.415 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:13:23.416 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:13:23.416 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5' 2024-09-18T08:13:23.517 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3 2024-09-18T08:13:23.553 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:13:23.554 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:13:23.554 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3' 2024-09-18T08:13:23.655 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2 2024-09-18T08:13:23.688 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:13:23.689 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:13:23.689 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2' 2024-09-18T08:13:23.790 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3 2024-09-18T08:13:23.819 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:13:23.819 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:13:23.820 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3' 2024-09-18T08:13:23.920 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6 2024-09-18T08:13:23.959 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:13:23.959 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:13:23.960 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6' 2024-09-18T08:13:24.060 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7 2024-09-18T08:13:24.097 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:13:24.097 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:13:24.098 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7' 2024-09-18T08:13:24.199 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5 2024-09-18T08:13:24.233 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:13:24.234 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:13:24.234 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5' 2024-09-18T08:13:24.335 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7 2024-09-18T08:13:24.374 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:13:24.375 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:13:24.375 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7' 2024-09-18T08:13:24.476 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2 2024-09-18T08:13:24.509 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:13:24.510 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:13:24.510 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2' 2024-09-18T08:13:24.611 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3 2024-09-18T08:13:24.641 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:13:24.642 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:13:24.642 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3' 2024-09-18T08:13:24.743 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0 2024-09-18T08:13:24.775 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:13:24.776 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:13:24.776 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0' 2024-09-18T08:13:24.877 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5 2024-09-18T08:13:24.905 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:13:24 smithi050 ceph-mon[32371]: pgmap v512: 158 pgs: 30 active+undersized, 34 active+undersized+degraded, 94 active+clean; 5.7 GiB data, 19 GiB used, 697 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 900 op/s; 53030/280920 objects degraded (18.877%) 2024-09-18T08:13:24.906 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:13:24 smithi050 ceph-mon[32371]: Health check update: Degraded data redundancy: 53030/280920 objects degraded (18.877%), 34 pgs degraded, 64 pgs undersized (PG_DEGRADED) 2024-09-18T08:13:24.913 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:13:24.914 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:13:24.914 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5' 2024-09-18T08:13:24.980 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:13:24 smithi007 ceph-mon[39249]: pgmap v512: 158 pgs: 30 active+undersized, 34 active+undersized+degraded, 94 active+clean; 5.7 GiB data, 19 GiB used, 697 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 900 op/s; 53030/280920 objects degraded (18.877%) 2024-09-18T08:13:24.980 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:13:24 smithi007 ceph-mon[39249]: Health check update: Degraded data redundancy: 53030/280920 objects degraded (18.877%), 34 pgs degraded, 64 pgs undersized (PG_DEGRADED) 2024-09-18T08:13:24.980 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:13:24 smithi007 ceph-mon[29459]: pgmap v512: 158 pgs: 30 active+undersized, 34 active+undersized+degraded, 94 active+clean; 5.7 GiB data, 19 GiB used, 697 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 900 op/s; 53030/280920 objects degraded (18.877%) 2024-09-18T08:13:24.981 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:13:24 smithi007 ceph-mon[29459]: Health check update: Degraded data redundancy: 53030/280920 objects degraded (18.877%), 34 pgs degraded, 64 pgs undersized (PG_DEGRADED) 2024-09-18T08:13:25.015 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6 2024-09-18T08:13:25.058 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:13:25.059 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:13:25.059 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6' 2024-09-18T08:13:25.160 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7 2024-09-18T08:13:25.195 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:13:25.195 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:13:25.196 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7' 2024-09-18T08:13:25.297 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6 2024-09-18T08:13:25.327 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:13:25.328 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:13:25.328 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6' 2024-09-18T08:13:25.429 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3 2024-09-18T08:13:25.461 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:13:25.461 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:13:25.462 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3' 2024-09-18T08:13:25.563 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3 2024-09-18T08:13:25.596 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:13:25.596 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:13:25.596 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3' 2024-09-18T08:13:25.697 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7 2024-09-18T08:13:25.736 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:13:25.737 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:13:25.737 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7' 2024-09-18T08:13:25.838 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0 2024-09-18T08:13:25.863 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:13:25.863 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:13:25.864 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0' 2024-09-18T08:13:25.964 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3 2024-09-18T08:13:25.987 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:13:25.987 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:13:25.988 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3' 2024-09-18T08:13:26.089 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0 2024-09-18T08:13:26.121 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:13:26.122 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:13:26.123 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0' 2024-09-18T08:13:26.223 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5 2024-09-18T08:13:26.257 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:13:26.258 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:13:26.258 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5' 2024-09-18T08:13:26.359 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2 2024-09-18T08:13:26.399 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:13:26.400 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:13:26.400 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2' 2024-09-18T08:13:26.501 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6 2024-09-18T08:13:26.536 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:13:26.537 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:13:26.538 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6' 2024-09-18T08:13:26.639 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2 2024-09-18T08:13:26.670 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:13:26.671 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:13:26.671 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2' 2024-09-18T08:13:26.772 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6 2024-09-18T08:13:26.806 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:13:26.807 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:13:26.807 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6' 2024-09-18T08:13:26.905 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:13:26 smithi050 ceph-mon[32371]: pgmap v513: 158 pgs: 30 active+undersized, 34 active+undersized+degraded, 94 active+clean; 5.2 GiB data, 18 GiB used, 698 GiB / 715 GiB avail; 1.0 KiB/s rd, 0 B/s wr, 1.51k op/s; 48760/258474 objects degraded (18.865%) 2024-09-18T08:13:26.909 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6 2024-09-18T08:13:26.932 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:13:26 smithi007 ceph-mon[29459]: pgmap v513: 158 pgs: 30 active+undersized, 34 active+undersized+degraded, 94 active+clean; 5.2 GiB data, 18 GiB used, 698 GiB / 715 GiB avail; 1.0 KiB/s rd, 0 B/s wr, 1.51k op/s; 48760/258474 objects degraded (18.865%) 2024-09-18T08:13:26.932 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Sep 18 08:13:26 smithi007 ceph-779349ca-7593-11ef-bceb-c7b262605968-mgr-y[29688]: ::ffff:172.21.15.50 - - [18/Sep/2024:08:13:26] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-09-18T08:13:26.932 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:13:26 smithi007 ceph-mon[39249]: pgmap v513: 158 pgs: 30 active+undersized, 34 active+undersized+degraded, 94 active+clean; 5.2 GiB data, 18 GiB used, 698 GiB / 715 GiB avail; 1.0 KiB/s rd, 0 B/s wr, 1.51k op/s; 48760/258474 objects degraded (18.865%) 2024-09-18T08:13:26.943 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:13:26.944 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:13:26.945 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6' 2024-09-18T08:13:27.045 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0 2024-09-18T08:13:27.095 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:13:27.096 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:13:27.096 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0' 2024-09-18T08:13:27.109 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/779349ca-7593-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T08:13:27.118 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/779349ca-7593-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T08:13:27.197 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4 2024-09-18T08:13:27.232 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:13:27.233 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:13:27.233 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4' 2024-09-18T08:13:27.334 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4 2024-09-18T08:13:27.378 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:13:27.379 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:13:27.379 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4' 2024-09-18T08:13:27.480 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7 2024-09-18T08:13:27.518 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:13:27.518 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:13:27.519 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7' 2024-09-18T08:13:27.617 INFO:teuthology.orchestra.run.smithi007.stderr:noscrub is unset 2024-09-18T08:13:27.617 INFO:teuthology.orchestra.run.smithi007.stderr:set 6.3 pg_upmap mapping to [5,3,2] 2024-09-18T08:13:27.619 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4 2024-09-18T08:13:27.654 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:13:27.654 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:13:27.655 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4' 2024-09-18T08:13:27.755 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4 2024-09-18T08:13:27.785 INFO:teuthology.orchestra.run.smithi007.stdout:{ 2024-09-18T08:13:27.786 INFO:teuthology.orchestra.run.smithi007.stdout: "size": 20, 2024-09-18T08:13:27.786 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 600, 2024-09-18T08:13:27.786 INFO:teuthology.orchestra.run.smithi007.stdout: "ops": [ 2024-09-18T08:13:27.786 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:13:27.786 INFO:teuthology.orchestra.run.smithi007.stdout: "description": "osd_op(client.15216.0:22606 8.f 8:f10856c4:::benchmark_data_smithi007_101583_object22605:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e174)", 2024-09-18T08:13:27.786 INFO:teuthology.orchestra.run.smithi007.stdout: "initiated_at": "2024-09-18T08:08:08.436554+0000", 2024-09-18T08:13:27.786 INFO:teuthology.orchestra.run.smithi007.stdout: "age": 319.342840056, 2024-09-18T08:13:27.786 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.768179902, 2024-09-18T08:13:27.786 INFO:teuthology.orchestra.run.smithi007.stdout: "continuous": false, 2024-09-18T08:13:27.786 INFO:teuthology.orchestra.run.smithi007.stdout: "type_data": { 2024-09-18T08:13:27.786 INFO:teuthology.orchestra.run.smithi007.stdout: "flag_point": "reached pg", 2024-09-18T08:13:27.786 INFO:teuthology.orchestra.run.smithi007.stdout: "client_info": { 2024-09-18T08:13:27.787 INFO:teuthology.orchestra.run.smithi007.stdout: "client": "client.15216", 2024-09-18T08:13:27.787 INFO:teuthology.orchestra.run.smithi007.stdout: "client_addr": "172.21.15.7:0/2596041177", 2024-09-18T08:13:27.787 INFO:teuthology.orchestra.run.smithi007.stdout: "tid": 22606 2024-09-18T08:13:27.787 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:13:27.787 INFO:teuthology.orchestra.run.smithi007.stdout: "events": [ 2024-09-18T08:13:27.787 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:13:27.787 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "initiated", 2024-09-18T08:13:27.787 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:08.436554+0000", 2024-09-18T08:13:27.787 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-09-18T08:13:27.787 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:13:27.788 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:13:27.788 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "header_read", 2024-09-18T08:13:27.788 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:08.436554+0000", 2024-09-18T08:13:27.788 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-09-18T08:13:27.788 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:13:27.788 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:13:27.788 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "throttled", 2024-09-18T08:13:27.788 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:08.436555+0000", 2024-09-18T08:13:27.788 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 7.8299999999999996e-07 2024-09-18T08:13:27.788 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:13:27.788 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:13:27.789 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "all_read", 2024-09-18T08:13:27.789 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:08.436576+0000", 2024-09-18T08:13:27.789 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 2.1214999999999999e-05 2024-09-18T08:13:27.789 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:13:27.789 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:13:27.789 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "dispatched", 2024-09-18T08:13:27.789 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:08.436586+0000", 2024-09-18T08:13:27.789 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.0088999999999999e-05 2024-09-18T08:13:27.789 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:13:27.789 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:13:27.789 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "queued_for_pg", 2024-09-18T08:13:27.789 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:08.436593+0000", 2024-09-18T08:13:27.789 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 7.3379999999999999e-06 2024-09-18T08:13:27.790 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:13:27.790 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:13:27.790 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-09-18T08:13:27.790 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:08.436642+0000", 2024-09-18T08:13:27.790 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 4.8928000000000003e-05 2024-09-18T08:13:27.790 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:13:27.790 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:13:27.790 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "started", 2024-09-18T08:13:27.790 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:08.436744+0000", 2024-09-18T08:13:27.790 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.000101778 2024-09-18T08:13:27.790 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:13:27.790 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:13:27.790 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "waiting for subops from 1,2", 2024-09-18T08:13:27.790 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:08.436890+0000", 2024-09-18T08:13:27.791 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.00014579099999999999 2024-09-18T08:13:27.791 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:13:27.791 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:13:27.791 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "op_commit", 2024-09-18T08:13:27.791 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:08.449655+0000", 2024-09-18T08:13:27.791 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.012765334 2024-09-18T08:13:27.791 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:13:27.791 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:13:27.791 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-09-18T08:13:27.791 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:08.460251+0000", 2024-09-18T08:13:27.791 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.010595635000000001 2024-09-18T08:13:27.791 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:13:27.791 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:13:27.792 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "waiting for peered", 2024-09-18T08:13:27.792 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:08.460264+0000", 2024-09-18T08:13:27.792 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.2921e-05 2024-09-18T08:13:27.792 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:13:27.792 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:13:27.792 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-09-18T08:13:27.792 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:09.152996+0000", 2024-09-18T08:13:27.792 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.69273260999999997 2024-09-18T08:13:27.792 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:13:27.792 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:13:27.792 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "waiting for peered", 2024-09-18T08:13:27.792 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:09.153008+0000", 2024-09-18T08:13:27.792 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.1459e-05 2024-09-18T08:13:27.793 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:13:27.793 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:13:27.793 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-09-18T08:13:27.793 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:10.204625+0000", 2024-09-18T08:13:27.793 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.0516172269999999 2024-09-18T08:13:27.793 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:13:27.793 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:13:27.793 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "done", 2024-09-18T08:13:27.793 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:10.204734+0000", 2024-09-18T08:13:27.793 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.000108794 2024-09-18T08:13:27.793 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-09-18T08:13:27.793 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-09-18T08:13:27.793 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-09-18T08:13:27.793 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:13:27.794 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:13:27.794 INFO:teuthology.orchestra.run.smithi007.stdout: "description": "osd_op(client.15216.0:22608 8.b 8:d92bd5a2:::benchmark_data_smithi007_101583_object22607:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e174)", 2024-09-18T08:13:27.794 INFO:teuthology.orchestra.run.smithi007.stdout: "initiated_at": "2024-09-18T08:08:08.436747+0000", 2024-09-18T08:13:27.794 INFO:teuthology.orchestra.run.smithi007.stdout: "age": 319.342646425, 2024-09-18T08:13:27.794 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.8187816670000001, 2024-09-18T08:13:27.794 INFO:teuthology.orchestra.run.smithi007.stdout: "continuous": false, 2024-09-18T08:13:27.794 INFO:teuthology.orchestra.run.smithi007.stdout: "type_data": { 2024-09-18T08:13:27.794 INFO:teuthology.orchestra.run.smithi007.stdout: "flag_point": "reached pg", 2024-09-18T08:13:27.794 INFO:teuthology.orchestra.run.smithi007.stdout: "client_info": { 2024-09-18T08:13:27.794 INFO:teuthology.orchestra.run.smithi007.stdout: "client": "client.15216", 2024-09-18T08:13:27.794 INFO:teuthology.orchestra.run.smithi007.stdout: "client_addr": "172.21.15.7:0/2596041177", 2024-09-18T08:13:27.794 INFO:teuthology.orchestra.run.smithi007.stdout: "tid": 22608 2024-09-18T08:13:27.794 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:13:27.795 INFO:teuthology.orchestra.run.smithi007.stdout: "events": [ 2024-09-18T08:13:27.795 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:13:27.795 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "initiated", 2024-09-18T08:13:27.795 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:08.436747+0000", 2024-09-18T08:13:27.795 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-09-18T08:13:27.795 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:13:27.795 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:13:27.795 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "header_read", 2024-09-18T08:13:27.795 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:08.436747+0000", 2024-09-18T08:13:27.795 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-09-18T08:13:27.795 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:13:27.795 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:13:27.795 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "throttled", 2024-09-18T08:13:27.795 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:08.436748+0000", 2024-09-18T08:13:27.796 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 9.2299999999999999e-07 2024-09-18T08:13:27.796 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:13:27.796 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:13:27.796 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "all_read", 2024-09-18T08:13:27.796 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:08.436769+0000", 2024-09-18T08:13:27.796 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 2.0392999999999999e-05 2024-09-18T08:13:27.796 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:13:27.796 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:13:27.796 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "dispatched", 2024-09-18T08:13:27.796 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:08.436778+0000", 2024-09-18T08:13:27.796 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 8.8689999999999993e-06 2024-09-18T08:13:27.796 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:13:27.796 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:13:27.796 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "queued_for_pg", 2024-09-18T08:13:27.797 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:08.436785+0000", 2024-09-18T08:13:27.797 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 7.199e-06 2024-09-18T08:13:27.797 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:13:27.797 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:13:27.797 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-09-18T08:13:27.797 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:08.437535+0000", 2024-09-18T08:13:27.797 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.00075027000000000004 2024-09-18T08:13:27.797 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:13:27.797 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:13:27.797 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "started", 2024-09-18T08:13:27.797 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:08.438489+0000", 2024-09-18T08:13:27.797 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.00095404400000000003 2024-09-18T08:13:27.797 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:13:27.798 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:13:27.798 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "waiting for subops from 1,2", 2024-09-18T08:13:27.798 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:08.438742+0000", 2024-09-18T08:13:27.798 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.00025305599999999998 2024-09-18T08:13:27.798 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:13:27.798 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:13:27.798 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "op_commit", 2024-09-18T08:13:27.798 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:08.440394+0000", 2024-09-18T08:13:27.798 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.001652096 2024-09-18T08:13:27.798 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:13:27.798 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:13:27.798 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:13:27.798 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:08.449571+0000", 2024-09-18T08:13:27.799 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.0091767580000000001 2024-09-18T08:13:27.799 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:13:27.799 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:13:27.799 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-09-18T08:13:27.799 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:08.455097+0000", 2024-09-18T08:13:27.799 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.0055257730000000003 2024-09-18T08:13:27.799 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:13:27.799 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:13:27.799 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "waiting for peered", 2024-09-18T08:13:27.799 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:08.455115+0000", 2024-09-18T08:13:27.799 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.8501000000000001e-05 2024-09-18T08:13:27.799 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:13:27.799 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:13:27.799 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-09-18T08:13:27.800 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:09.152107+0000", 2024-09-18T08:13:27.800 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.69699121100000005 2024-09-18T08:13:27.800 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:13:27.800 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:13:27.800 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "waiting for peered", 2024-09-18T08:13:27.800 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:09.152128+0000", 2024-09-18T08:13:27.800 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 2.1557e-05 2024-09-18T08:13:27.800 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:13:27.800 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:13:27.800 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-09-18T08:13:27.800 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:10.255451+0000", 2024-09-18T08:13:27.800 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.1033232070000001 2024-09-18T08:13:27.800 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:13:27.800 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:13:27.801 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "done", 2024-09-18T08:13:27.801 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:10.255529+0000", 2024-09-18T08:13:27.801 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 7.7810000000000002e-05 2024-09-18T08:13:27.801 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-09-18T08:13:27.801 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-09-18T08:13:27.801 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-09-18T08:13:27.801 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:13:27.801 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:13:27.801 INFO:teuthology.orchestra.run.smithi007.stdout: "description": "osd_op(client.15216.0:22614 8.f 8:f07b5883:::benchmark_data_smithi007_101583_object22613:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e174)", 2024-09-18T08:13:27.801 INFO:teuthology.orchestra.run.smithi007.stdout: "initiated_at": "2024-09-18T08:08:08.439788+0000", 2024-09-18T08:13:27.801 INFO:teuthology.orchestra.run.smithi007.stdout: "age": 319.33960636199998, 2024-09-18T08:13:27.801 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.7882368309999999, 2024-09-18T08:13:27.801 INFO:teuthology.orchestra.run.smithi007.stdout: "continuous": false, 2024-09-18T08:13:27.801 INFO:teuthology.orchestra.run.smithi007.stdout: "type_data": { 2024-09-18T08:13:27.801 INFO:teuthology.orchestra.run.smithi007.stdout: "flag_point": "reached pg", 2024-09-18T08:13:27.802 INFO:teuthology.orchestra.run.smithi007.stdout: "client_info": { 2024-09-18T08:13:27.802 INFO:teuthology.orchestra.run.smithi007.stdout: "client": "client.15216", 2024-09-18T08:13:27.802 INFO:teuthology.orchestra.run.smithi007.stdout: "client_addr": "172.21.15.7:0/2596041177", 2024-09-18T08:13:27.802 INFO:teuthology.orchestra.run.smithi007.stdout: "tid": 22614 2024-09-18T08:13:27.802 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:13:27.802 INFO:teuthology.orchestra.run.smithi007.stdout: "events": [ 2024-09-18T08:13:27.802 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:13:27.802 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "initiated", 2024-09-18T08:13:27.802 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:08.439788+0000", 2024-09-18T08:13:27.802 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-09-18T08:13:27.802 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:13:27.802 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:13:27.802 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "header_read", 2024-09-18T08:13:27.802 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:08.439788+0000", 2024-09-18T08:13:27.802 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-09-18T08:13:27.803 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:13:27.803 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:13:27.803 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "throttled", 2024-09-18T08:13:27.803 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:08.439789+0000", 2024-09-18T08:13:27.803 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.5009999999999999e-06 2024-09-18T08:13:27.803 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:13:27.803 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:13:27.803 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "all_read", 2024-09-18T08:13:27.803 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:08.439813+0000", 2024-09-18T08:13:27.803 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 2.442e-05 2024-09-18T08:13:27.803 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:13:27.803 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:13:27.803 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "dispatched", 2024-09-18T08:13:27.803 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:08.439827+0000", 2024-09-18T08:13:27.803 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.3540000000000001e-05 2024-09-18T08:13:27.803 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:13:27.804 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:13:27.804 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "queued_for_pg", 2024-09-18T08:13:27.804 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:08.439836+0000", 2024-09-18T08:13:27.804 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 8.7630000000000006e-06 2024-09-18T08:13:27.804 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:13:27.804 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:13:27.804 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-09-18T08:13:27.804 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:08.445915+0000", 2024-09-18T08:13:27.804 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.0060795210000000001 2024-09-18T08:13:27.804 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:13:27.804 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:13:27.804 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "started", 2024-09-18T08:13:27.804 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:08.446054+0000", 2024-09-18T08:13:27.804 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.00013859699999999999 2024-09-18T08:13:27.804 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:13:27.805 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:13:27.805 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "waiting for subops from 1,2", 2024-09-18T08:13:27.805 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:08.448436+0000", 2024-09-18T08:13:27.805 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.0023821939999999998 2024-09-18T08:13:27.805 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:13:27.805 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:13:27.805 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "op_commit", 2024-09-18T08:13:27.805 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:08.449664+0000", 2024-09-18T08:13:27.805 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.0012275000000000001 2024-09-18T08:13:27.805 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:13:27.805 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:13:27.805 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-09-18T08:13:27.805 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:08.460300+0000", 2024-09-18T08:13:27.805 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.010636487999999999 2024-09-18T08:13:27.806 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:13:27.806 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:13:27.806 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "waiting for peered", 2024-09-18T08:13:27.806 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:08.460310+0000", 2024-09-18T08:13:27.806 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 9.5240000000000006e-06 2024-09-18T08:13:27.806 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:13:27.806 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:13:27.806 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-09-18T08:13:27.806 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:09.153052+0000", 2024-09-18T08:13:27.806 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.69274252300000005 2024-09-18T08:13:27.806 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:13:27.806 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:13:27.806 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "waiting for peered", 2024-09-18T08:13:27.806 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:09.153062+0000", 2024-09-18T08:13:27.806 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 9.7569999999999999e-06 2024-09-18T08:13:27.807 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:13:27.807 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:13:27.807 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-09-18T08:13:27.807 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:10.220005+0000", 2024-09-18T08:13:27.807 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.0669431330000001 2024-09-18T08:13:27.807 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:13:27.807 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:13:27.807 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "waiting for degraded object", 2024-09-18T08:13:27.807 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:10.220517+0000", 2024-09-18T08:13:27.807 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.00051243100000000004 2024-09-18T08:13:27.807 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:13:27.807 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:13:27.807 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-09-18T08:13:27.807 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:10.227921+0000", 2024-09-18T08:13:27.807 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.0074033850000000002 2024-09-18T08:13:27.808 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:13:27.808 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:13:27.808 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "done", 2024-09-18T08:13:27.808 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:10.228024+0000", 2024-09-18T08:13:27.808 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.000103554 2024-09-18T08:13:27.808 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-09-18T08:13:27.808 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-09-18T08:13:27.808 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-09-18T08:13:27.808 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:13:27.808 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:13:27.808 INFO:teuthology.orchestra.run.smithi007.stdout: "description": "osd_op(client.15216.0:22617 8.b 8:dab1c1d5:::benchmark_data_smithi007_101583_object22616:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e174)", 2024-09-18T08:13:27.808 INFO:teuthology.orchestra.run.smithi007.stdout: "initiated_at": "2024-09-18T08:08:08.447023+0000", 2024-09-18T08:13:27.808 INFO:teuthology.orchestra.run.smithi007.stdout: "age": 319.33237055000001, 2024-09-18T08:13:27.808 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.8206128700000002, 2024-09-18T08:13:27.809 INFO:teuthology.orchestra.run.smithi007.stdout: "continuous": false, 2024-09-18T08:13:27.809 INFO:teuthology.orchestra.run.smithi007.stdout: "type_data": { 2024-09-18T08:13:27.809 INFO:teuthology.orchestra.run.smithi007.stdout: "flag_point": "reached pg", 2024-09-18T08:13:27.809 INFO:teuthology.orchestra.run.smithi007.stdout: "client_info": { 2024-09-18T08:13:27.809 INFO:teuthology.orchestra.run.smithi007.stdout: "client": "client.15216", 2024-09-18T08:13:27.809 INFO:teuthology.orchestra.run.smithi007.stdout: "client_addr": "172.21.15.7:0/2596041177", 2024-09-18T08:13:27.809 INFO:teuthology.orchestra.run.smithi007.stdout: "tid": 22617 2024-09-18T08:13:27.809 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:13:27.809 INFO:teuthology.orchestra.run.smithi007.stdout: "events": [ 2024-09-18T08:13:27.809 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:13:27.809 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "initiated", 2024-09-18T08:13:27.809 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:08.447023+0000", 2024-09-18T08:13:27.809 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-09-18T08:13:27.809 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:13:27.810 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:13:27.810 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "header_read", 2024-09-18T08:13:27.810 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:08.447023+0000", 2024-09-18T08:13:27.810 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-09-18T08:13:27.810 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:13:27.810 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:13:27.810 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "throttled", 2024-09-18T08:13:27.810 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:08.447025+0000", 2024-09-18T08:13:27.810 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.685e-06 2024-09-18T08:13:27.810 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:13:27.810 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:13:27.810 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "all_read", 2024-09-18T08:13:27.810 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:08.447053+0000", 2024-09-18T08:13:27.810 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 2.7953000000000002e-05 2024-09-18T08:13:27.810 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:13:27.810 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:13:27.811 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "dispatched", 2024-09-18T08:13:27.811 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:08.447070+0000", 2024-09-18T08:13:27.811 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.6650000000000002e-05 2024-09-18T08:13:27.811 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:13:27.811 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:13:27.811 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "queued_for_pg", 2024-09-18T08:13:27.811 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:08.447080+0000", 2024-09-18T08:13:27.811 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.0148e-05 2024-09-18T08:13:27.811 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:13:27.811 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:13:27.811 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-09-18T08:13:27.811 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:08.449604+0000", 2024-09-18T08:13:27.811 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.0025244640000000001 2024-09-18T08:13:27.811 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:13:27.811 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:13:27.812 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "started", 2024-09-18T08:13:27.812 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:08.449753+0000", 2024-09-18T08:13:27.812 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.00014878600000000001 2024-09-18T08:13:27.812 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:13:27.812 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:13:27.812 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "waiting for subops from 1,2", 2024-09-18T08:13:27.812 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:08.449904+0000", 2024-09-18T08:13:27.812 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.00015076299999999999 2024-09-18T08:13:27.812 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:13:27.812 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:13:27.812 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-09-18T08:13:27.812 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:08.455203+0000", 2024-09-18T08:13:27.812 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.0052987950000000002 2024-09-18T08:13:27.812 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:13:27.812 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:13:27.813 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "waiting for peered", 2024-09-18T08:13:27.813 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:08.455211+0000", 2024-09-18T08:13:27.814 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 8.8179999999999993e-06 2024-09-18T08:13:27.814 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:13:27.814 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:13:27.814 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-09-18T08:13:27.814 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:09.152202+0000", 2024-09-18T08:13:27.814 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.69699049000000002 2024-09-18T08:13:27.814 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:13:27.814 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:13:27.814 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "waiting for peered", 2024-09-18T08:13:27.814 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:09.152212+0000", 2024-09-18T08:13:27.814 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.0251000000000001e-05 2024-09-18T08:13:27.815 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:13:27.815 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:13:27.815 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-09-18T08:13:27.815 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:10.256406+0000", 2024-09-18T08:13:27.815 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.104194101 2024-09-18T08:13:27.815 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:13:27.815 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:13:27.815 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "waiting for degraded object", 2024-09-18T08:13:27.815 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:10.256961+0000", 2024-09-18T08:13:27.815 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.00055520899999999998 2024-09-18T08:13:27.815 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:13:27.815 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:13:27.815 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-09-18T08:13:27.815 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:10.267550+0000", 2024-09-18T08:13:27.816 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.010588640999999999 2024-09-18T08:13:27.816 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:13:27.816 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:13:27.816 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "done", 2024-09-18T08:13:27.816 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:10.267636+0000", 2024-09-18T08:13:27.816 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 8.6116e-05 2024-09-18T08:13:27.816 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-09-18T08:13:27.816 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-09-18T08:13:27.816 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-09-18T08:13:27.816 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:13:27.816 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:13:27.816 INFO:teuthology.orchestra.run.smithi007.stdout: "description": "osd_op(client.15216.0:22630 8.b 8:da804186:::benchmark_data_smithi007_101583_object22629:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e175)", 2024-09-18T08:13:27.816 INFO:teuthology.orchestra.run.smithi007.stdout: "initiated_at": "2024-09-18T08:08:08.467388+0000", 2024-09-18T08:13:27.817 INFO:teuthology.orchestra.run.smithi007.stdout: "age": 319.31200580299998, 2024-09-18T08:13:27.817 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.8127456770000001, 2024-09-18T08:13:27.817 INFO:teuthology.orchestra.run.smithi007.stdout: "continuous": false, 2024-09-18T08:13:27.817 INFO:teuthology.orchestra.run.smithi007.stdout: "type_data": { 2024-09-18T08:13:27.817 INFO:teuthology.orchestra.run.smithi007.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T08:13:27.817 INFO:teuthology.orchestra.run.smithi007.stdout: "client_info": { 2024-09-18T08:13:27.817 INFO:teuthology.orchestra.run.smithi007.stdout: "client": "client.15216", 2024-09-18T08:13:27.817 INFO:teuthology.orchestra.run.smithi007.stdout: "client_addr": "172.21.15.7:0/2596041177", 2024-09-18T08:13:27.817 INFO:teuthology.orchestra.run.smithi007.stdout: "tid": 22630 2024-09-18T08:13:27.817 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:13:27.817 INFO:teuthology.orchestra.run.smithi007.stdout: "events": [ 2024-09-18T08:13:27.817 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:13:27.817 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "initiated", 2024-09-18T08:13:27.817 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:08.467388+0000", 2024-09-18T08:13:27.818 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-09-18T08:13:27.818 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:13:27.818 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:13:27.818 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "header_read", 2024-09-18T08:13:27.818 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:08.467388+0000", 2024-09-18T08:13:27.818 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-09-18T08:13:27.818 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:13:27.818 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:13:27.818 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "throttled", 2024-09-18T08:13:27.818 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:08.467389+0000", 2024-09-18T08:13:27.818 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.373e-06 2024-09-18T08:13:27.818 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:13:27.818 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:13:27.818 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "all_read", 2024-09-18T08:13:27.818 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:08.467447+0000", 2024-09-18T08:13:27.819 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 5.7238999999999999e-05 2024-09-18T08:13:27.819 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:13:27.819 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:13:27.819 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "dispatched", 2024-09-18T08:13:27.819 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:08.467493+0000", 2024-09-18T08:13:27.819 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 4.6755000000000002e-05 2024-09-18T08:13:27.819 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:13:27.819 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:13:27.819 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "queued_for_pg", 2024-09-18T08:13:27.819 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:08.467513+0000", 2024-09-18T08:13:27.819 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.9873000000000001e-05 2024-09-18T08:13:27.819 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:13:27.819 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:13:27.819 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-09-18T08:13:27.820 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:08.467970+0000", 2024-09-18T08:13:27.820 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.00045696899999999999 2024-09-18T08:13:27.820 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:13:27.820 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:13:27.820 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "waiting for peered", 2024-09-18T08:13:27.820 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:08.467979+0000", 2024-09-18T08:13:27.820 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 8.6440000000000006e-06 2024-09-18T08:13:27.820 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:13:27.820 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:13:27.820 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-09-18T08:13:27.820 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:09.152276+0000", 2024-09-18T08:13:27.820 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.68429694500000005 2024-09-18T08:13:27.820 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:13:27.820 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:13:27.820 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "waiting for peered", 2024-09-18T08:13:27.821 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:09.152284+0000", 2024-09-18T08:13:27.821 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 7.7049999999999999e-06 2024-09-18T08:13:27.821 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:13:27.821 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:13:27.821 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-09-18T08:13:27.821 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:10.257004+0000", 2024-09-18T08:13:27.821 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.104719961 2024-09-18T08:13:27.821 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:13:27.821 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:13:27.821 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "started", 2024-09-18T08:13:27.821 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:10.258354+0000", 2024-09-18T08:13:27.821 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.001350097 2024-09-18T08:13:27.821 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:13:27.821 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:13:27.821 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "waiting for subops from 1,7", 2024-09-18T08:13:27.822 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:10.258951+0000", 2024-09-18T08:13:27.822 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.00059747000000000001 2024-09-18T08:13:27.822 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:13:27.822 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:13:27.822 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:13:27.822 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:10.267720+0000", 2024-09-18T08:13:27.822 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.0087689169999999993 2024-09-18T08:13:27.822 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:13:27.822 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:13:27.822 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:13:27.822 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:10.267767+0000", 2024-09-18T08:13:27.822 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 4.6616000000000002e-05 2024-09-18T08:13:27.822 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:13:27.822 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:13:27.822 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "op_commit", 2024-09-18T08:13:27.822 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:10.278730+0000", 2024-09-18T08:13:27.823 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.010963818 2024-09-18T08:13:27.823 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:13:27.823 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:13:27.823 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "commit_sent", 2024-09-18T08:13:27.823 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:10.278929+0000", 2024-09-18T08:13:27.823 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.000198638 2024-09-18T08:13:27.823 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:13:27.823 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:13:27.823 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "done", 2024-09-18T08:13:27.823 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:10.280134+0000", 2024-09-18T08:13:27.823 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.001204657 2024-09-18T08:13:27.823 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-09-18T08:13:27.823 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-09-18T08:13:27.823 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-09-18T08:13:27.823 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:13:27.824 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:13:27.824 INFO:teuthology.orchestra.run.smithi007.stdout: "description": "osd_op(client.15216.0:22665 8.f 8:f8e596fa:::benchmark_data_smithi007_101583_object22664:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e175)", 2024-09-18T08:13:27.824 INFO:teuthology.orchestra.run.smithi007.stdout: "initiated_at": "2024-09-18T08:08:08.484581+0000", 2024-09-18T08:13:27.824 INFO:teuthology.orchestra.run.smithi007.stdout: "age": 319.29481288300002, 2024-09-18T08:13:27.824 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.7443272359999999, 2024-09-18T08:13:27.824 INFO:teuthology.orchestra.run.smithi007.stdout: "continuous": false, 2024-09-18T08:13:27.824 INFO:teuthology.orchestra.run.smithi007.stdout: "type_data": { 2024-09-18T08:13:27.824 INFO:teuthology.orchestra.run.smithi007.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T08:13:27.824 INFO:teuthology.orchestra.run.smithi007.stdout: "client_info": { 2024-09-18T08:13:27.824 INFO:teuthology.orchestra.run.smithi007.stdout: "client": "client.15216", 2024-09-18T08:13:27.824 INFO:teuthology.orchestra.run.smithi007.stdout: "client_addr": "172.21.15.7:0/2596041177", 2024-09-18T08:13:27.824 INFO:teuthology.orchestra.run.smithi007.stdout: "tid": 22665 2024-09-18T08:13:27.824 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:13:27.824 INFO:teuthology.orchestra.run.smithi007.stdout: "events": [ 2024-09-18T08:13:27.824 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:13:27.825 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "initiated", 2024-09-18T08:13:27.825 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:08.484581+0000", 2024-09-18T08:13:27.825 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-09-18T08:13:27.825 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:13:27.825 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:13:27.825 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "header_read", 2024-09-18T08:13:27.825 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:08.484581+0000", 2024-09-18T08:13:27.825 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-09-18T08:13:27.825 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:13:27.825 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:13:27.825 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "throttled", 2024-09-18T08:13:27.825 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:08.484583+0000", 2024-09-18T08:13:27.825 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.9250000000000002e-06 2024-09-18T08:13:27.826 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:13:27.826 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:13:27.826 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "all_read", 2024-09-18T08:13:27.826 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:08.484606+0000", 2024-09-18T08:13:27.826 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 2.2965000000000001e-05 2024-09-18T08:13:27.826 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:13:27.826 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:13:27.826 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "dispatched", 2024-09-18T08:13:27.826 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:08.484617+0000", 2024-09-18T08:13:27.826 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.1240000000000001e-05 2024-09-18T08:13:27.826 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:13:27.826 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:13:27.826 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "queued_for_pg", 2024-09-18T08:13:27.826 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:08.484625+0000", 2024-09-18T08:13:27.826 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 7.5020000000000003e-06 2024-09-18T08:13:27.827 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:13:27.827 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:13:27.827 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-09-18T08:13:27.827 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:08.484862+0000", 2024-09-18T08:13:27.827 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.00023732500000000001 2024-09-18T08:13:27.827 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:13:27.827 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:13:27.827 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "waiting for peered", 2024-09-18T08:13:27.827 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:08.484872+0000", 2024-09-18T08:13:27.827 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.0231000000000001e-05 2024-09-18T08:13:27.827 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:13:27.827 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:13:27.827 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-09-18T08:13:27.827 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:09.153091+0000", 2024-09-18T08:13:27.828 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.66821881900000002 2024-09-18T08:13:27.828 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:13:27.828 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:13:27.828 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "waiting for peered", 2024-09-18T08:13:27.828 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:09.153100+0000", 2024-09-18T08:13:27.828 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 8.7560000000000005e-06 2024-09-18T08:13:27.828 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:13:27.828 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:13:27.828 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-09-18T08:13:27.828 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:10.220552+0000", 2024-09-18T08:13:27.828 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.0674517699999999 2024-09-18T08:13:27.828 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:13:27.828 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:13:27.829 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "started", 2024-09-18T08:13:27.829 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:10.220667+0000", 2024-09-18T08:13:27.829 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.000115671 2024-09-18T08:13:27.829 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:13:27.829 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:13:27.829 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "waiting for subops from 1,6", 2024-09-18T08:13:27.829 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:10.220826+0000", 2024-09-18T08:13:27.829 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.00015917700000000001 2024-09-18T08:13:27.829 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:13:27.829 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:13:27.829 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "op_commit", 2024-09-18T08:13:27.829 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:10.224779+0000", 2024-09-18T08:13:27.829 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.0039521119999999998 2024-09-18T08:13:27.829 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:13:27.830 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:13:27.830 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:13:27.830 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:10.228102+0000", 2024-09-18T08:13:27.830 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.0033237230000000002 2024-09-18T08:13:27.830 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:13:27.830 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:13:27.830 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:13:27.830 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:10.228153+0000", 2024-09-18T08:13:27.830 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 5.0705000000000003e-05 2024-09-18T08:13:27.830 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:13:27.830 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:13:27.830 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "commit_sent", 2024-09-18T08:13:27.830 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:10.228212+0000", 2024-09-18T08:13:27.830 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 5.9404999999999998e-05 2024-09-18T08:13:27.830 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:13:27.831 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:13:27.831 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "done", 2024-09-18T08:13:27.831 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:10.228908+0000", 2024-09-18T08:13:27.831 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.00069591 2024-09-18T08:13:27.831 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-09-18T08:13:27.831 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-09-18T08:13:27.831 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-09-18T08:13:27.831 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:13:27.831 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:13:27.831 INFO:teuthology.orchestra.run.smithi007.stdout: "description": "osd_op(client.15216.0:22668 8.f 8:f26239dc:::benchmark_data_smithi007_101583_object22667:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e175)", 2024-09-18T08:13:27.831 INFO:teuthology.orchestra.run.smithi007.stdout: "initiated_at": "2024-09-18T08:08:08.486691+0000", 2024-09-18T08:13:27.831 INFO:teuthology.orchestra.run.smithi007.stdout: "age": 319.29270282499999, 2024-09-18T08:13:27.831 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.7435772489999999, 2024-09-18T08:13:27.831 INFO:teuthology.orchestra.run.smithi007.stdout: "continuous": false, 2024-09-18T08:13:27.831 INFO:teuthology.orchestra.run.smithi007.stdout: "type_data": { 2024-09-18T08:13:27.832 INFO:teuthology.orchestra.run.smithi007.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T08:13:27.832 INFO:teuthology.orchestra.run.smithi007.stdout: "client_info": { 2024-09-18T08:13:27.832 INFO:teuthology.orchestra.run.smithi007.stdout: "client": "client.15216", 2024-09-18T08:13:27.832 INFO:teuthology.orchestra.run.smithi007.stdout: "client_addr": "172.21.15.7:0/2596041177", 2024-09-18T08:13:27.832 INFO:teuthology.orchestra.run.smithi007.stdout: "tid": 22668 2024-09-18T08:13:27.832 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:13:27.832 INFO:teuthology.orchestra.run.smithi007.stdout: "events": [ 2024-09-18T08:13:27.832 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:13:27.832 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "initiated", 2024-09-18T08:13:27.832 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:08.486691+0000", 2024-09-18T08:13:27.832 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-09-18T08:13:27.832 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:13:27.832 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:13:27.832 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "header_read", 2024-09-18T08:13:27.832 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:08.486691+0000", 2024-09-18T08:13:27.832 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-09-18T08:13:27.833 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:13:27.833 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:13:27.833 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "throttled", 2024-09-18T08:13:27.833 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:08.486692+0000", 2024-09-18T08:13:27.833 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.3030000000000001e-06 2024-09-18T08:13:27.833 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:13:27.833 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:13:27.833 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "all_read", 2024-09-18T08:13:27.833 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:08.486716+0000", 2024-09-18T08:13:27.833 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 2.3164999999999999e-05 2024-09-18T08:13:27.833 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:13:27.833 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:13:27.833 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "dispatched", 2024-09-18T08:13:27.833 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:08.486728+0000", 2024-09-18T08:13:27.834 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.2648e-05 2024-09-18T08:13:27.834 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:13:27.834 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:13:27.834 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "queued_for_pg", 2024-09-18T08:13:27.834 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:08.486736+0000", 2024-09-18T08:13:27.834 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 7.8779999999999998e-06 2024-09-18T08:13:27.834 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:13:27.834 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:13:27.834 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-09-18T08:13:27.834 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:08.486921+0000", 2024-09-18T08:13:27.834 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.00018532100000000001 2024-09-18T08:13:27.834 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:13:27.834 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:13:27.834 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "waiting for peered", 2024-09-18T08:13:27.835 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:08.486930+0000", 2024-09-18T08:13:27.835 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 8.6300000000000004e-06 2024-09-18T08:13:27.835 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:13:27.835 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:13:27.835 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-09-18T08:13:27.835 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:09.153126+0000", 2024-09-18T08:13:27.835 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.66619636299999996 2024-09-18T08:13:27.835 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:13:27.835 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:13:27.835 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "waiting for peered", 2024-09-18T08:13:27.835 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:09.153280+0000", 2024-09-18T08:13:27.835 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.00015323800000000001 2024-09-18T08:13:27.835 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:13:27.836 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:13:27.836 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-09-18T08:13:27.836 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:10.223256+0000", 2024-09-18T08:13:27.836 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.069976313 2024-09-18T08:13:27.836 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:13:27.836 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:13:27.836 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "started", 2024-09-18T08:13:27.836 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:10.223428+0000", 2024-09-18T08:13:27.836 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.00017207299999999999 2024-09-18T08:13:27.836 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:13:27.836 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:13:27.836 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "waiting for subops from 1,6", 2024-09-18T08:13:27.836 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:10.223632+0000", 2024-09-18T08:13:27.836 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.000204289 2024-09-18T08:13:27.837 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:13:27.837 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:13:27.837 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "op_commit", 2024-09-18T08:13:27.837 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:10.228023+0000", 2024-09-18T08:13:27.837 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.0043910140000000004 2024-09-18T08:13:27.837 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:13:27.837 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:13:27.837 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:13:27.837 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:10.228966+0000", 2024-09-18T08:13:27.837 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.00094256400000000001 2024-09-18T08:13:27.837 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:13:27.837 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:13:27.837 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:13:27.838 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:10.229018+0000", 2024-09-18T08:13:27.838 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 5.2089000000000002e-05 2024-09-18T08:13:27.838 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:13:27.838 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:13:27.838 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "commit_sent", 2024-09-18T08:13:27.838 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:10.229075+0000", 2024-09-18T08:13:27.838 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 5.7079e-05 2024-09-18T08:13:27.838 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:13:27.838 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:13:27.838 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "done", 2024-09-18T08:13:27.838 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:10.230268+0000", 2024-09-18T08:13:27.838 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.0011932819999999999 2024-09-18T08:13:27.838 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-09-18T08:13:27.839 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-09-18T08:13:27.839 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-09-18T08:13:27.839 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:13:27.839 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:13:27.839 INFO:teuthology.orchestra.run.smithi007.stdout: "description": "osd_op(client.15216.0:22675 8.f 8:f41edfb7:::benchmark_data_smithi007_101583_object22674:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e175)", 2024-09-18T08:13:27.839 INFO:teuthology.orchestra.run.smithi007.stdout: "initiated_at": "2024-09-18T08:08:08.490781+0000", 2024-09-18T08:13:27.839 INFO:teuthology.orchestra.run.smithi007.stdout: "age": 319.28861301199998, 2024-09-18T08:13:27.839 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.74874595, 2024-09-18T08:13:27.839 INFO:teuthology.orchestra.run.smithi007.stdout: "continuous": false, 2024-09-18T08:13:27.839 INFO:teuthology.orchestra.run.smithi007.stdout: "type_data": { 2024-09-18T08:13:27.839 INFO:teuthology.orchestra.run.smithi007.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T08:13:27.839 INFO:teuthology.orchestra.run.smithi007.stdout: "client_info": { 2024-09-18T08:13:27.839 INFO:teuthology.orchestra.run.smithi007.stdout: "client": "client.15216", 2024-09-18T08:13:27.839 INFO:teuthology.orchestra.run.smithi007.stdout: "client_addr": "172.21.15.7:0/2596041177", 2024-09-18T08:13:27.840 INFO:teuthology.orchestra.run.smithi007.stdout: "tid": 22675 2024-09-18T08:13:27.840 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:13:27.840 INFO:teuthology.orchestra.run.smithi007.stdout: "events": [ 2024-09-18T08:13:27.840 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:13:27.840 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "initiated", 2024-09-18T08:13:27.840 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:08.490781+0000", 2024-09-18T08:13:27.840 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-09-18T08:13:27.840 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:13:27.840 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:13:27.840 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "header_read", 2024-09-18T08:13:27.840 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:08.490781+0000", 2024-09-18T08:13:27.840 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-09-18T08:13:27.841 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:13:27.841 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:13:27.841 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "throttled", 2024-09-18T08:13:27.841 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:08.490783+0000", 2024-09-18T08:13:27.841 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.708e-06 2024-09-18T08:13:27.841 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:13:27.841 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:13:27.841 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "all_read", 2024-09-18T08:13:27.841 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:08.490807+0000", 2024-09-18T08:13:27.841 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 2.4300000000000001e-05 2024-09-18T08:13:27.841 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:13:27.841 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:13:27.841 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "dispatched", 2024-09-18T08:13:27.841 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:08.490819+0000", 2024-09-18T08:13:27.842 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.2468e-05 2024-09-18T08:13:27.842 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:13:27.842 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:13:27.842 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "queued_for_pg", 2024-09-18T08:13:27.842 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:08.490829+0000", 2024-09-18T08:13:27.842 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 9.2320000000000007e-06 2024-09-18T08:13:27.842 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:13:27.842 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:13:27.842 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-09-18T08:13:27.842 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:08.490917+0000", 2024-09-18T08:13:27.842 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 8.8350000000000006e-05 2024-09-18T08:13:27.842 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:13:27.842 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:13:27.843 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "waiting for peered", 2024-09-18T08:13:27.843 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:08.490928+0000", 2024-09-18T08:13:27.843 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.0848e-05 2024-09-18T08:13:27.843 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:13:27.843 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:13:27.843 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-09-18T08:13:27.843 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:09.153316+0000", 2024-09-18T08:13:27.843 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.66238866600000001 2024-09-18T08:13:27.843 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:13:27.843 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:13:27.843 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "waiting for peered", 2024-09-18T08:13:27.843 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:09.153325+0000", 2024-09-18T08:13:27.843 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 8.1089999999999998e-06 2024-09-18T08:13:27.843 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:13:27.844 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:13:27.844 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-09-18T08:13:27.844 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:10.230312+0000", 2024-09-18T08:13:27.844 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.0769875980000001 2024-09-18T08:13:27.844 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:13:27.844 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:13:27.844 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "started", 2024-09-18T08:13:27.844 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:10.230436+0000", 2024-09-18T08:13:27.844 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.000123824 2024-09-18T08:13:27.844 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:13:27.844 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:13:27.844 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "waiting for subops from 1,6", 2024-09-18T08:13:27.844 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:10.230594+0000", 2024-09-18T08:13:27.844 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.00015772099999999999 2024-09-18T08:13:27.845 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:13:27.845 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:13:27.845 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "op_commit", 2024-09-18T08:13:27.845 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:10.234562+0000", 2024-09-18T08:13:27.845 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.0039686460000000002 2024-09-18T08:13:27.845 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:13:27.845 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:13:27.845 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:13:27.845 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:10.234625+0000", 2024-09-18T08:13:27.845 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 6.2675999999999994e-05 2024-09-18T08:13:27.845 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:13:27.845 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:13:27.845 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:13:27.845 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:10.238138+0000", 2024-09-18T08:13:27.846 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.0035124710000000001 2024-09-18T08:13:27.846 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:13:27.846 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:13:27.846 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "commit_sent", 2024-09-18T08:13:27.846 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:10.238230+0000", 2024-09-18T08:13:27.846 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 9.2304000000000003e-05 2024-09-18T08:13:27.846 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:13:27.846 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:13:27.846 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "done", 2024-09-18T08:13:27.846 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:10.239527+0000", 2024-09-18T08:13:27.846 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.0012970290000000001 2024-09-18T08:13:27.846 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-09-18T08:13:27.846 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-09-18T08:13:27.846 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-09-18T08:13:27.846 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:13:27.847 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:13:27.847 INFO:teuthology.orchestra.run.smithi007.stdout: "description": "osd_op(client.15216.0:22685 8.b 8:d2c6b3c8:::benchmark_data_smithi007_101583_object22684:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e175)", 2024-09-18T08:13:27.847 INFO:teuthology.orchestra.run.smithi007.stdout: "initiated_at": "2024-09-18T08:08:08.499266+0000", 2024-09-18T08:13:27.847 INFO:teuthology.orchestra.run.smithi007.stdout: "age": 319.28012803799999, 2024-09-18T08:13:27.847 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.781783984, 2024-09-18T08:13:27.847 INFO:teuthology.orchestra.run.smithi007.stdout: "continuous": false, 2024-09-18T08:13:27.847 INFO:teuthology.orchestra.run.smithi007.stdout: "type_data": { 2024-09-18T08:13:27.847 INFO:teuthology.orchestra.run.smithi007.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T08:13:27.847 INFO:teuthology.orchestra.run.smithi007.stdout: "client_info": { 2024-09-18T08:13:27.847 INFO:teuthology.orchestra.run.smithi007.stdout: "client": "client.15216", 2024-09-18T08:13:27.847 INFO:teuthology.orchestra.run.smithi007.stdout: "client_addr": "172.21.15.7:0/2596041177", 2024-09-18T08:13:27.847 INFO:teuthology.orchestra.run.smithi007.stdout: "tid": 22685 2024-09-18T08:13:27.847 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:13:27.848 INFO:teuthology.orchestra.run.smithi007.stdout: "events": [ 2024-09-18T08:13:27.848 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:13:27.848 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "initiated", 2024-09-18T08:13:27.848 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:08.499266+0000", 2024-09-18T08:13:27.848 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-09-18T08:13:27.848 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:13:27.848 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:13:27.848 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "header_read", 2024-09-18T08:13:27.848 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:08.499266+0000", 2024-09-18T08:13:27.848 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-09-18T08:13:27.848 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:13:27.848 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:13:27.848 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "throttled", 2024-09-18T08:13:27.849 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:08.499268+0000", 2024-09-18T08:13:27.849 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.7060000000000001e-06 2024-09-18T08:13:27.849 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:13:27.849 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:13:27.849 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "all_read", 2024-09-18T08:13:27.849 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:08.499297+0000", 2024-09-18T08:13:27.849 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 2.9298e-05 2024-09-18T08:13:27.849 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:13:27.849 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:13:27.849 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "dispatched", 2024-09-18T08:13:27.849 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:08.499318+0000", 2024-09-18T08:13:27.849 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 2.0758999999999999e-05 2024-09-18T08:13:27.849 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:13:27.850 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:13:27.850 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "queued_for_pg", 2024-09-18T08:13:27.850 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:08.499325+0000", 2024-09-18T08:13:27.850 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 7.8599999999999993e-06 2024-09-18T08:13:27.850 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:13:27.850 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:13:27.850 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-09-18T08:13:27.850 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:08.499379+0000", 2024-09-18T08:13:27.850 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 5.3257999999999997e-05 2024-09-18T08:13:27.850 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:13:27.850 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:13:27.850 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "waiting for peered", 2024-09-18T08:13:27.850 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:08.499391+0000", 2024-09-18T08:13:27.850 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.2483000000000001e-05 2024-09-18T08:13:27.851 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:13:27.851 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:13:27.851 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-09-18T08:13:27.851 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:09.152309+0000", 2024-09-18T08:13:27.851 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.65291789700000002 2024-09-18T08:13:27.851 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:13:27.851 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:13:27.851 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "waiting for peered", 2024-09-18T08:13:27.851 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:09.152317+0000", 2024-09-18T08:13:27.851 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 7.4830000000000002e-06 2024-09-18T08:13:27.851 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:13:27.851 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:13:27.851 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-09-18T08:13:27.851 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:10.260397+0000", 2024-09-18T08:13:27.851 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.108080043 2024-09-18T08:13:27.852 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:13:27.852 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:13:27.852 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "started", 2024-09-18T08:13:27.852 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:10.260553+0000", 2024-09-18T08:13:27.852 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.000156594 2024-09-18T08:13:27.852 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:13:27.852 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:13:27.852 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "waiting for subops from 1,7", 2024-09-18T08:13:27.852 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:10.260761+0000", 2024-09-18T08:13:27.852 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.00020734200000000001 2024-09-18T08:13:27.852 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:13:27.852 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:13:27.852 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:13:27.853 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:10.267815+0000", 2024-09-18T08:13:27.853 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.007053948 2024-09-18T08:13:27.853 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:13:27.853 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:13:27.853 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:13:27.853 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:10.267878+0000", 2024-09-18T08:13:27.853 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 6.3346e-05 2024-09-18T08:13:27.853 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:13:27.853 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:13:27.853 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "op_commit", 2024-09-18T08:13:27.853 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:10.280193+0000", 2024-09-18T08:13:27.853 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.012314782999999999 2024-09-18T08:13:27.853 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:13:27.853 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:13:27.854 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "commit_sent", 2024-09-18T08:13:27.854 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:10.280222+0000", 2024-09-18T08:13:27.854 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 2.9224e-05 2024-09-18T08:13:27.854 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:13:27.854 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:13:27.854 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "done", 2024-09-18T08:13:27.854 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:10.281050+0000", 2024-09-18T08:13:27.854 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:13:27.855 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:13:27.855 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4' 2024-09-18T08:13:27.856 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.00082795999999999998 2024-09-18T08:13:27.856 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-09-18T08:13:27.856 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-09-18T08:13:27.856 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-09-18T08:13:27.856 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:13:27.856 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:13:27.856 INFO:teuthology.orchestra.run.smithi007.stdout: "description": "osd_op(client.15216.0:22689 8.f 8:f532acf4:::benchmark_data_smithi007_101583_object22688:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e175)", 2024-09-18T08:13:27.856 INFO:teuthology.orchestra.run.smithi007.stdout: "initiated_at": "2024-09-18T08:08:08.501798+0000", 2024-09-18T08:13:27.856 INFO:teuthology.orchestra.run.smithi007.stdout: "age": 319.277596023, 2024-09-18T08:13:27.856 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.7388306359999999, 2024-09-18T08:13:27.857 INFO:teuthology.orchestra.run.smithi007.stdout: "continuous": false, 2024-09-18T08:13:27.857 INFO:teuthology.orchestra.run.smithi007.stdout: "type_data": { 2024-09-18T08:13:27.857 INFO:teuthology.orchestra.run.smithi007.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T08:13:27.857 INFO:teuthology.orchestra.run.smithi007.stdout: "client_info": { 2024-09-18T08:13:27.857 INFO:teuthology.orchestra.run.smithi007.stdout: "client": "client.15216", 2024-09-18T08:13:27.857 INFO:teuthology.orchestra.run.smithi007.stdout: "client_addr": "172.21.15.7:0/2596041177", 2024-09-18T08:13:27.857 INFO:teuthology.orchestra.run.smithi007.stdout: "tid": 22689 2024-09-18T08:13:27.857 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:13:27.857 INFO:teuthology.orchestra.run.smithi007.stdout: "events": [ 2024-09-18T08:13:27.857 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:13:27.857 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "initiated", 2024-09-18T08:13:27.857 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:08.501798+0000", 2024-09-18T08:13:27.857 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-09-18T08:13:27.858 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:13:27.858 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:13:27.858 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "header_read", 2024-09-18T08:13:27.858 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:08.501798+0000", 2024-09-18T08:13:27.858 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-09-18T08:13:27.858 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:13:27.858 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:13:27.858 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "throttled", 2024-09-18T08:13:27.858 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:08.501798+0000", 2024-09-18T08:13:27.858 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 5.9400000000000005e-07 2024-09-18T08:13:27.858 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:13:27.858 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:13:27.858 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "all_read", 2024-09-18T08:13:27.858 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:08.501835+0000", 2024-09-18T08:13:27.859 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 3.6974000000000002e-05 2024-09-18T08:13:27.859 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:13:27.859 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:13:27.859 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "dispatched", 2024-09-18T08:13:27.859 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:08.501841+0000", 2024-09-18T08:13:27.859 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 5.2909999999999998e-06 2024-09-18T08:13:27.859 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:13:27.859 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:13:27.859 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "queued_for_pg", 2024-09-18T08:13:27.859 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:08.501844+0000", 2024-09-18T08:13:27.859 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 3.7330000000000001e-06 2024-09-18T08:13:27.859 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:13:27.859 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:13:27.859 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-09-18T08:13:27.860 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:08.501904+0000", 2024-09-18T08:13:27.860 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 5.9861000000000002e-05 2024-09-18T08:13:27.860 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:13:27.860 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:13:27.860 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "waiting for peered", 2024-09-18T08:13:27.860 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:08.501917+0000", 2024-09-18T08:13:27.860 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.2181000000000001e-05 2024-09-18T08:13:27.860 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:13:27.860 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:13:27.860 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-09-18T08:13:27.860 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:09.153352+0000", 2024-09-18T08:13:27.860 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.65143504100000005 2024-09-18T08:13:27.860 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:13:27.860 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:13:27.861 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "waiting for peered", 2024-09-18T08:13:27.861 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:09.153359+0000", 2024-09-18T08:13:27.861 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 7.4760000000000001e-06 2024-09-18T08:13:27.861 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:13:27.861 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:13:27.861 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-09-18T08:13:27.861 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:10.232890+0000", 2024-09-18T08:13:27.861 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.0795311249999999 2024-09-18T08:13:27.861 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:13:27.861 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:13:27.861 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "started", 2024-09-18T08:13:27.861 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:10.233035+0000", 2024-09-18T08:13:27.861 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.000145282 2024-09-18T08:13:27.862 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:13:27.862 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:13:27.862 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "waiting for subops from 1,6", 2024-09-18T08:13:27.862 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:10.233242+0000", 2024-09-18T08:13:27.862 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.000206181 2024-09-18T08:13:27.862 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:13:27.862 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:13:27.862 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:13:27.862 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:10.238074+0000", 2024-09-18T08:13:27.862 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.0048323649999999999 2024-09-18T08:13:27.862 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:13:27.862 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:13:27.862 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "op_commit", 2024-09-18T08:13:27.862 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:10.238090+0000", 2024-09-18T08:13:27.862 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.5934e-05 2024-09-18T08:13:27.863 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:13:27.863 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:13:27.863 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:13:27.863 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:10.239669+0000", 2024-09-18T08:13:27.863 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.0015793039999999999 2024-09-18T08:13:27.863 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:13:27.863 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:13:27.863 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "commit_sent", 2024-09-18T08:13:27.863 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:10.239733+0000", 2024-09-18T08:13:27.864 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 6.3404999999999993e-05 2024-09-18T08:13:27.864 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:13:27.864 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:13:27.864 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "done", 2024-09-18T08:13:27.864 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:10.240629+0000", 2024-09-18T08:13:27.864 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.00089588899999999997 2024-09-18T08:13:27.864 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-09-18T08:13:27.864 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-09-18T08:13:27.864 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-09-18T08:13:27.864 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:13:27.864 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:13:27.864 INFO:teuthology.orchestra.run.smithi007.stdout: "description": "osd_op(client.15216.0:69187 8.f 8:f51333c2:::benchmark_data_smithi007_101583_object69186:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e198)", 2024-09-18T08:13:27.864 INFO:teuthology.orchestra.run.smithi007.stdout: "initiated_at": "2024-09-18T08:08:51.699105+0000", 2024-09-18T08:13:27.865 INFO:teuthology.orchestra.run.smithi007.stdout: "age": 276.08028852699999, 2024-09-18T08:13:27.865 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 2.1312144279999998, 2024-09-18T08:13:27.865 INFO:teuthology.orchestra.run.smithi007.stdout: "continuous": false, 2024-09-18T08:13:27.865 INFO:teuthology.orchestra.run.smithi007.stdout: "type_data": { 2024-09-18T08:13:27.865 INFO:teuthology.orchestra.run.smithi007.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T08:13:27.865 INFO:teuthology.orchestra.run.smithi007.stdout: "client_info": { 2024-09-18T08:13:27.865 INFO:teuthology.orchestra.run.smithi007.stdout: "client": "client.15216", 2024-09-18T08:13:27.865 INFO:teuthology.orchestra.run.smithi007.stdout: "client_addr": "172.21.15.7:0/2596041177", 2024-09-18T08:13:27.865 INFO:teuthology.orchestra.run.smithi007.stdout: "tid": 69187 2024-09-18T08:13:27.865 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:13:27.865 INFO:teuthology.orchestra.run.smithi007.stdout: "events": [ 2024-09-18T08:13:27.865 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:13:27.865 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "initiated", 2024-09-18T08:13:27.865 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:51.699105+0000", 2024-09-18T08:13:27.866 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-09-18T08:13:27.866 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:13:27.866 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:13:27.866 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "header_read", 2024-09-18T08:13:27.866 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:51.699105+0000", 2024-09-18T08:13:27.866 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-09-18T08:13:27.866 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:13:27.866 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:13:27.866 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "throttled", 2024-09-18T08:13:27.866 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:51.699107+0000", 2024-09-18T08:13:27.866 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.5149999999999999e-06 2024-09-18T08:13:27.866 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:13:27.866 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:13:27.866 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "all_read", 2024-09-18T08:13:27.867 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:51.699133+0000", 2024-09-18T08:13:27.867 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 2.5752999999999999e-05 2024-09-18T08:13:27.867 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:13:27.867 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:13:27.867 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "dispatched", 2024-09-18T08:13:27.867 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:51.699141+0000", 2024-09-18T08:13:27.867 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 8.2709999999999992e-06 2024-09-18T08:13:27.867 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:13:27.867 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:13:27.867 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "queued_for_pg", 2024-09-18T08:13:27.867 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:51.699163+0000", 2024-09-18T08:13:27.867 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 2.2432000000000001e-05 2024-09-18T08:13:27.867 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:13:27.868 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:13:27.868 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-09-18T08:13:27.868 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:51.699402+0000", 2024-09-18T08:13:27.868 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.000239046 2024-09-18T08:13:27.868 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:13:27.868 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:13:27.868 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "waiting for peered", 2024-09-18T08:13:27.868 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:51.699413+0000", 2024-09-18T08:13:27.868 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.0387000000000001e-05 2024-09-18T08:13:27.868 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:13:27.868 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:13:27.868 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-09-18T08:13:27.868 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:52.696975+0000", 2024-09-18T08:13:27.868 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.99756218100000005 2024-09-18T08:13:27.869 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:13:27.869 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:13:27.869 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "waiting for peered", 2024-09-18T08:13:27.869 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:52.697002+0000", 2024-09-18T08:13:27.869 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 2.7464e-05 2024-09-18T08:13:27.869 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:13:27.869 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:13:27.869 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-09-18T08:13:27.869 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:53.824414+0000", 2024-09-18T08:13:27.869 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.1274112789999999 2024-09-18T08:13:27.869 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:13:27.869 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:13:27.869 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "started", 2024-09-18T08:13:27.869 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:53.824573+0000", 2024-09-18T08:13:27.870 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.00015908400000000001 2024-09-18T08:13:27.870 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:13:27.870 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:13:27.870 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "waiting for subops from 1,2,4", 2024-09-18T08:13:27.870 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:53.824756+0000", 2024-09-18T08:13:27.870 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.00018345199999999999 2024-09-18T08:13:27.870 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:13:27.870 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:13:27.870 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:13:27.870 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:53.827276+0000", 2024-09-18T08:13:27.870 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.002519444 2024-09-18T08:13:27.870 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:13:27.871 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:13:27.871 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:13:27.871 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:53.827356+0000", 2024-09-18T08:13:27.871 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 8.0029000000000002e-05 2024-09-18T08:13:27.871 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:13:27.871 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:13:27.871 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "op_commit", 2024-09-18T08:13:27.871 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:53.827376+0000", 2024-09-18T08:13:27.871 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 2.0250000000000001e-05 2024-09-18T08:13:27.871 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:13:27.871 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:13:27.871 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:13:27.871 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:53.830181+0000", 2024-09-18T08:13:27.871 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.0028047469999999998 2024-09-18T08:13:27.871 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:13:27.872 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:13:27.872 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "commit_sent", 2024-09-18T08:13:27.872 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:53.830246+0000", 2024-09-18T08:13:27.872 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 6.5140000000000003e-05 2024-09-18T08:13:27.872 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:13:27.872 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:13:27.872 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "done", 2024-09-18T08:13:27.872 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:53.830320+0000", 2024-09-18T08:13:27.872 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 7.3954000000000004e-05 2024-09-18T08:13:27.872 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-09-18T08:13:27.872 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-09-18T08:13:27.872 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-09-18T08:13:27.872 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:13:27.872 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:13:27.872 INFO:teuthology.orchestra.run.smithi007.stdout: "description": "osd_op(client.15216.0:69195 8.f 8:fafe41e6:::benchmark_data_smithi007_101583_object69194:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e198)", 2024-09-18T08:13:27.873 INFO:teuthology.orchestra.run.smithi007.stdout: "initiated_at": "2024-09-18T08:08:51.711966+0000", 2024-09-18T08:13:27.873 INFO:teuthology.orchestra.run.smithi007.stdout: "age": 276.06742776499999, 2024-09-18T08:13:27.873 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 2.1187014670000002, 2024-09-18T08:13:27.873 INFO:teuthology.orchestra.run.smithi007.stdout: "continuous": false, 2024-09-18T08:13:27.873 INFO:teuthology.orchestra.run.smithi007.stdout: "type_data": { 2024-09-18T08:13:27.873 INFO:teuthology.orchestra.run.smithi007.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T08:13:27.873 INFO:teuthology.orchestra.run.smithi007.stdout: "client_info": { 2024-09-18T08:13:27.873 INFO:teuthology.orchestra.run.smithi007.stdout: "client": "client.15216", 2024-09-18T08:13:27.873 INFO:teuthology.orchestra.run.smithi007.stdout: "client_addr": "172.21.15.7:0/2596041177", 2024-09-18T08:13:27.873 INFO:teuthology.orchestra.run.smithi007.stdout: "tid": 69195 2024-09-18T08:13:27.873 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:13:27.873 INFO:teuthology.orchestra.run.smithi007.stdout: "events": [ 2024-09-18T08:13:27.873 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:13:27.873 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "initiated", 2024-09-18T08:13:27.874 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:51.711966+0000", 2024-09-18T08:13:27.874 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-09-18T08:13:27.874 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:13:27.874 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:13:27.874 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "header_read", 2024-09-18T08:13:27.874 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:51.711966+0000", 2024-09-18T08:13:27.874 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-09-18T08:13:27.874 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:13:27.874 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:13:27.874 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "throttled", 2024-09-18T08:13:27.874 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:51.711968+0000", 2024-09-18T08:13:27.874 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 2.013e-06 2024-09-18T08:13:27.874 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:13:27.874 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:13:27.874 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "all_read", 2024-09-18T08:13:27.875 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:51.711997+0000", 2024-09-18T08:13:27.875 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 2.8561999999999999e-05 2024-09-18T08:13:27.875 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:13:27.875 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:13:27.875 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "dispatched", 2024-09-18T08:13:27.875 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:51.712012+0000", 2024-09-18T08:13:27.875 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.5184000000000001e-05 2024-09-18T08:13:27.875 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:13:27.875 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:13:27.875 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "queued_for_pg", 2024-09-18T08:13:27.875 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:51.712022+0000", 2024-09-18T08:13:27.875 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 9.6530000000000006e-06 2024-09-18T08:13:27.875 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:13:27.875 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:13:27.875 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-09-18T08:13:27.876 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:51.712088+0000", 2024-09-18T08:13:27.876 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 6.6921000000000005e-05 2024-09-18T08:13:27.876 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:13:27.876 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:13:27.876 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "waiting for peered", 2024-09-18T08:13:27.876 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:51.712101+0000", 2024-09-18T08:13:27.876 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.2768e-05 2024-09-18T08:13:27.876 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:13:27.876 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:13:27.876 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-09-18T08:13:27.876 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:52.697039+0000", 2024-09-18T08:13:27.876 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.98493815299999998 2024-09-18T08:13:27.876 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:13:27.876 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:13:27.877 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "waiting for peered", 2024-09-18T08:13:27.877 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:52.697052+0000", 2024-09-18T08:13:27.877 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.2475e-05 2024-09-18T08:13:27.877 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:13:27.877 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:13:27.877 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-09-18T08:13:27.877 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:53.827400+0000", 2024-09-18T08:13:27.877 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.1303482419999999 2024-09-18T08:13:27.877 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:13:27.877 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:13:27.877 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "started", 2024-09-18T08:13:27.877 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:53.827528+0000", 2024-09-18T08:13:27.877 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.000128009 2024-09-18T08:13:27.877 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:13:27.878 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:13:27.878 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "waiting for subops from 1,2,4", 2024-09-18T08:13:27.878 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:53.827701+0000", 2024-09-18T08:13:27.878 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.00017307100000000001 2024-09-18T08:13:27.878 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:13:27.878 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:13:27.878 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:13:27.878 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:53.830376+0000", 2024-09-18T08:13:27.878 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.002675144 2024-09-18T08:13:27.878 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:13:27.878 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:13:27.878 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:13:27.878 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:53.830419+0000", 2024-09-18T08:13:27.878 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 4.3087000000000003e-05 2024-09-18T08:13:27.879 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:13:27.879 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:13:27.879 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "op_commit", 2024-09-18T08:13:27.879 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:53.830444+0000", 2024-09-18T08:13:27.879 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 2.4579e-05 2024-09-18T08:13:27.879 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:13:27.879 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:13:27.879 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:13:27.879 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:53.830485+0000", 2024-09-18T08:13:27.879 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 4.1236999999999998e-05 2024-09-18T08:13:27.879 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:13:27.879 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:13:27.879 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "commit_sent", 2024-09-18T08:13:27.879 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:53.830541+0000", 2024-09-18T08:13:27.880 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 5.6075000000000001e-05 2024-09-18T08:13:27.880 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:13:27.880 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:13:27.880 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "done", 2024-09-18T08:13:27.880 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:53.830668+0000", 2024-09-18T08:13:27.880 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.00012629399999999999 2024-09-18T08:13:27.880 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-09-18T08:13:27.880 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-09-18T08:13:27.880 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-09-18T08:13:27.880 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:13:27.880 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:13:27.880 INFO:teuthology.orchestra.run.smithi007.stdout: "description": "osd_op(client.15216.0:69207 8.f 8:f63411d3:::benchmark_data_smithi007_101583_object69206:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e198)", 2024-09-18T08:13:27.880 INFO:teuthology.orchestra.run.smithi007.stdout: "initiated_at": "2024-09-18T08:08:51.719518+0000", 2024-09-18T08:13:27.880 INFO:teuthology.orchestra.run.smithi007.stdout: "age": 276.05987540500001, 2024-09-18T08:13:27.880 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 2.1171781319999998, 2024-09-18T08:13:27.881 INFO:teuthology.orchestra.run.smithi007.stdout: "continuous": false, 2024-09-18T08:13:27.881 INFO:teuthology.orchestra.run.smithi007.stdout: "type_data": { 2024-09-18T08:13:27.881 INFO:teuthology.orchestra.run.smithi007.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T08:13:27.881 INFO:teuthology.orchestra.run.smithi007.stdout: "client_info": { 2024-09-18T08:13:27.881 INFO:teuthology.orchestra.run.smithi007.stdout: "client": "client.15216", 2024-09-18T08:13:27.881 INFO:teuthology.orchestra.run.smithi007.stdout: "client_addr": "172.21.15.7:0/2596041177", 2024-09-18T08:13:27.881 INFO:teuthology.orchestra.run.smithi007.stdout: "tid": 69207 2024-09-18T08:13:27.881 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:13:27.881 INFO:teuthology.orchestra.run.smithi007.stdout: "events": [ 2024-09-18T08:13:27.881 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:13:27.881 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "initiated", 2024-09-18T08:13:27.881 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:51.719518+0000", 2024-09-18T08:13:27.881 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-09-18T08:13:27.882 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:13:27.882 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:13:27.882 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "header_read", 2024-09-18T08:13:27.882 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:51.719518+0000", 2024-09-18T08:13:27.882 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-09-18T08:13:27.882 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:13:27.882 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:13:27.882 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "throttled", 2024-09-18T08:13:27.882 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:51.719519+0000", 2024-09-18T08:13:27.882 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 5.7999999999999995e-07 2024-09-18T08:13:27.882 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:13:27.882 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:13:27.882 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "all_read", 2024-09-18T08:13:27.882 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:51.719540+0000", 2024-09-18T08:13:27.883 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 2.1097000000000001e-05 2024-09-18T08:13:27.883 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:13:27.883 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:13:27.883 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "dispatched", 2024-09-18T08:13:27.883 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:51.719547+0000", 2024-09-18T08:13:27.883 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 6.4810000000000003e-06 2024-09-18T08:13:27.883 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:13:27.883 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:13:27.883 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "queued_for_pg", 2024-09-18T08:13:27.883 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:51.719552+0000", 2024-09-18T08:13:27.883 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 5.2100000000000001e-06 2024-09-18T08:13:27.883 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:13:27.883 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:13:27.883 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-09-18T08:13:27.883 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:51.719676+0000", 2024-09-18T08:13:27.884 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.00012415899999999999 2024-09-18T08:13:27.884 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:13:27.884 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:13:27.884 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "waiting for peered", 2024-09-18T08:13:27.884 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:51.719686+0000", 2024-09-18T08:13:27.884 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 9.5000000000000005e-06 2024-09-18T08:13:27.884 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:13:27.884 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:13:27.884 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-09-18T08:13:27.884 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:52.697757+0000", 2024-09-18T08:13:27.884 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.97807110100000005 2024-09-18T08:13:27.884 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:13:27.884 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:13:27.884 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "waiting for peered", 2024-09-18T08:13:27.884 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:52.697767+0000", 2024-09-18T08:13:27.885 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 9.8810000000000008e-06 2024-09-18T08:13:27.885 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:13:27.885 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:13:27.885 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-09-18T08:13:27.885 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:53.830718+0000", 2024-09-18T08:13:27.885 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.1329519370000001 2024-09-18T08:13:27.885 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:13:27.885 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:13:27.885 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "started", 2024-09-18T08:13:27.885 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:53.830870+0000", 2024-09-18T08:13:27.885 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.00015120300000000001 2024-09-18T08:13:27.885 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:13:27.885 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:13:27.886 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "waiting for subops from 1,2,4", 2024-09-18T08:13:27.886 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:53.831043+0000", 2024-09-18T08:13:27.886 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.00017385499999999999 2024-09-18T08:13:27.886 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:13:27.886 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:13:27.886 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "op_commit", 2024-09-18T08:13:27.886 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:53.836372+0000", 2024-09-18T08:13:27.886 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.0053287689999999997 2024-09-18T08:13:27.886 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:13:27.886 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:13:27.886 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:13:27.886 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:53.836420+0000", 2024-09-18T08:13:27.886 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 4.7490000000000001e-05 2024-09-18T08:13:27.886 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:13:27.886 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:13:27.887 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:13:27.887 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:53.836464+0000", 2024-09-18T08:13:27.887 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 4.4254e-05 2024-09-18T08:13:27.887 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:13:27.887 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:13:27.887 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:13:27.887 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:53.836505+0000", 2024-09-18T08:13:27.887 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 4.1414e-05 2024-09-18T08:13:27.887 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:13:27.887 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:13:27.887 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "commit_sent", 2024-09-18T08:13:27.887 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:53.836566+0000", 2024-09-18T08:13:27.887 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 6.0538999999999998e-05 2024-09-18T08:13:27.888 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:13:27.888 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:13:27.888 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "done", 2024-09-18T08:13:27.888 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:53.836697+0000", 2024-09-18T08:13:27.888 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.000130662 2024-09-18T08:13:27.888 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-09-18T08:13:27.888 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-09-18T08:13:27.888 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-09-18T08:13:27.888 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:13:27.888 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:13:27.888 INFO:teuthology.orchestra.run.smithi007.stdout: "description": "osd_op(client.15216.0:69218 8.f 8:fd5c3396:::benchmark_data_smithi007_101583_object69217:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e198)", 2024-09-18T08:13:27.888 INFO:teuthology.orchestra.run.smithi007.stdout: "initiated_at": "2024-09-18T08:08:51.731702+0000", 2024-09-18T08:13:27.888 INFO:teuthology.orchestra.run.smithi007.stdout: "age": 276.047692149, 2024-09-18T08:13:27.888 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 2.1112667140000001, 2024-09-18T08:13:27.889 INFO:teuthology.orchestra.run.smithi007.stdout: "continuous": false, 2024-09-18T08:13:27.889 INFO:teuthology.orchestra.run.smithi007.stdout: "type_data": { 2024-09-18T08:13:27.889 INFO:teuthology.orchestra.run.smithi007.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T08:13:27.889 INFO:teuthology.orchestra.run.smithi007.stdout: "client_info": { 2024-09-18T08:13:27.889 INFO:teuthology.orchestra.run.smithi007.stdout: "client": "client.15216", 2024-09-18T08:13:27.889 INFO:teuthology.orchestra.run.smithi007.stdout: "client_addr": "172.21.15.7:0/2596041177", 2024-09-18T08:13:27.889 INFO:teuthology.orchestra.run.smithi007.stdout: "tid": 69218 2024-09-18T08:13:27.889 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:13:27.889 INFO:teuthology.orchestra.run.smithi007.stdout: "events": [ 2024-09-18T08:13:27.889 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:13:27.889 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "initiated", 2024-09-18T08:13:27.889 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:51.731702+0000", 2024-09-18T08:13:27.889 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-09-18T08:13:27.889 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:13:27.890 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:13:27.890 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "header_read", 2024-09-18T08:13:27.890 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:51.731702+0000", 2024-09-18T08:13:27.890 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-09-18T08:13:27.890 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:13:27.890 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:13:27.890 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "throttled", 2024-09-18T08:13:27.890 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:51.731703+0000", 2024-09-18T08:13:27.890 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.265e-06 2024-09-18T08:13:27.890 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:13:27.890 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:13:27.890 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "all_read", 2024-09-18T08:13:27.890 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:51.731771+0000", 2024-09-18T08:13:27.890 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 6.7945000000000001e-05 2024-09-18T08:13:27.891 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:13:27.891 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:13:27.891 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "dispatched", 2024-09-18T08:13:27.891 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:51.731781+0000", 2024-09-18T08:13:27.891 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 9.8030000000000008e-06 2024-09-18T08:13:27.891 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:13:27.891 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:13:27.891 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "queued_for_pg", 2024-09-18T08:13:27.891 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:51.731788+0000", 2024-09-18T08:13:27.891 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 7.1620000000000002e-06 2024-09-18T08:13:27.891 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:13:27.891 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:13:27.891 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-09-18T08:13:27.891 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:51.731861+0000", 2024-09-18T08:13:27.892 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 7.2669999999999994e-05 2024-09-18T08:13:27.892 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:13:27.892 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:13:27.892 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "waiting for peered", 2024-09-18T08:13:27.892 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:51.731875+0000", 2024-09-18T08:13:27.892 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.4209999999999999e-05 2024-09-18T08:13:27.892 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:13:27.892 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:13:27.892 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-09-18T08:13:27.892 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:52.697807+0000", 2024-09-18T08:13:27.892 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.96593184200000004 2024-09-18T08:13:27.892 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:13:27.892 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:13:27.893 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "waiting for peered", 2024-09-18T08:13:27.893 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:52.697815+0000", 2024-09-18T08:13:27.893 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 8.4239999999999993e-06 2024-09-18T08:13:27.893 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:13:27.893 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:13:27.893 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-09-18T08:13:27.893 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:53.833616+0000", 2024-09-18T08:13:27.893 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.1358008079999999 2024-09-18T08:13:27.893 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:13:27.893 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:13:27.893 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "started", 2024-09-18T08:13:27.893 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:53.833763+0000", 2024-09-18T08:13:27.893 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.000146663 2024-09-18T08:13:27.894 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:13:27.894 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:13:27.894 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "waiting for subops from 1,2,4", 2024-09-18T08:13:27.894 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:53.833911+0000", 2024-09-18T08:13:27.894 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.00014877900000000001 2024-09-18T08:13:27.894 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:13:27.894 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:13:27.894 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:13:27.894 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:53.836740+0000", 2024-09-18T08:13:27.894 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.002828882 2024-09-18T08:13:27.894 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:13:27.894 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:13:27.894 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "op_commit", 2024-09-18T08:13:27.894 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:53.836756+0000", 2024-09-18T08:13:27.894 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.6195999999999999e-05 2024-09-18T08:13:27.895 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:13:27.895 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:13:27.895 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:13:27.895 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:53.842695+0000", 2024-09-18T08:13:27.895 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.0059383209999999999 2024-09-18T08:13:27.895 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:13:27.895 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:13:27.895 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:13:27.895 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:53.842746+0000", 2024-09-18T08:13:27.895 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 5.0966e-05 2024-09-18T08:13:27.895 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:13:27.895 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:13:27.895 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "commit_sent", 2024-09-18T08:13:27.895 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:53.842817+0000", 2024-09-18T08:13:27.896 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 7.1437000000000001e-05 2024-09-18T08:13:27.896 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:13:27.896 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:13:27.896 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "done", 2024-09-18T08:13:27.896 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:53.842968+0000", 2024-09-18T08:13:27.896 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.00015134100000000001 2024-09-18T08:13:27.896 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-09-18T08:13:27.896 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-09-18T08:13:27.896 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-09-18T08:13:27.896 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:13:27.896 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:13:27.897 INFO:teuthology.orchestra.run.smithi007.stdout: "description": "osd_op(client.15216.0:69239 8.f 8:f2d8399e:::benchmark_data_smithi007_101583_object69238:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e198)", 2024-09-18T08:13:27.897 INFO:teuthology.orchestra.run.smithi007.stdout: "initiated_at": "2024-09-18T08:08:51.752739+0000", 2024-09-18T08:13:27.897 INFO:teuthology.orchestra.run.smithi007.stdout: "age": 276.02665461800001, 2024-09-18T08:13:27.897 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 2.0962772900000002, 2024-09-18T08:13:27.897 INFO:teuthology.orchestra.run.smithi007.stdout: "continuous": false, 2024-09-18T08:13:27.897 INFO:teuthology.orchestra.run.smithi007.stdout: "type_data": { 2024-09-18T08:13:27.897 INFO:teuthology.orchestra.run.smithi007.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T08:13:27.897 INFO:teuthology.orchestra.run.smithi007.stdout: "client_info": { 2024-09-18T08:13:27.898 INFO:teuthology.orchestra.run.smithi007.stdout: "client": "client.15216", 2024-09-18T08:13:27.898 INFO:teuthology.orchestra.run.smithi007.stdout: "client_addr": "172.21.15.7:0/2596041177", 2024-09-18T08:13:27.898 INFO:teuthology.orchestra.run.smithi007.stdout: "tid": 69239 2024-09-18T08:13:27.898 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:13:27.898 INFO:teuthology.orchestra.run.smithi007.stdout: "events": [ 2024-09-18T08:13:27.898 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:13:27.898 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "initiated", 2024-09-18T08:13:27.898 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:51.752739+0000", 2024-09-18T08:13:27.898 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-09-18T08:13:27.898 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:13:27.898 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:13:27.898 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "header_read", 2024-09-18T08:13:27.898 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:51.752739+0000", 2024-09-18T08:13:27.898 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-09-18T08:13:27.899 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:13:27.899 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:13:27.899 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "throttled", 2024-09-18T08:13:27.899 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:51.752741+0000", 2024-09-18T08:13:27.899 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.829e-06 2024-09-18T08:13:27.899 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:13:27.899 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:13:27.899 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "all_read", 2024-09-18T08:13:27.899 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:51.752766+0000", 2024-09-18T08:13:27.899 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 2.4842e-05 2024-09-18T08:13:27.899 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:13:27.899 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:13:27.899 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "dispatched", 2024-09-18T08:13:27.900 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:51.752777+0000", 2024-09-18T08:13:27.900 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.1395e-05 2024-09-18T08:13:27.900 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:13:27.900 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:13:27.900 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "queued_for_pg", 2024-09-18T08:13:27.900 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:51.752785+0000", 2024-09-18T08:13:27.900 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 7.401e-06 2024-09-18T08:13:27.900 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:13:27.900 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:13:27.900 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-09-18T08:13:27.900 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:51.753101+0000", 2024-09-18T08:13:27.900 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.00031663600000000002 2024-09-18T08:13:27.900 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:13:27.900 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:13:27.901 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "waiting for peered", 2024-09-18T08:13:27.901 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:51.753112+0000", 2024-09-18T08:13:27.901 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.0825000000000001e-05 2024-09-18T08:13:27.901 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:13:27.901 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:13:27.901 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-09-18T08:13:27.901 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:52.697883+0000", 2024-09-18T08:13:27.901 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.94477077700000001 2024-09-18T08:13:27.901 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:13:27.901 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:13:27.901 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "waiting for peered", 2024-09-18T08:13:27.901 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:52.697893+0000", 2024-09-18T08:13:27.901 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 9.5820000000000008e-06 2024-09-18T08:13:27.902 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:13:27.902 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:13:27.902 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-09-18T08:13:27.902 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:53.836781+0000", 2024-09-18T08:13:27.902 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.1388879329999999 2024-09-18T08:13:27.902 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:13:27.902 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:13:27.902 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "started", 2024-09-18T08:13:27.902 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:53.836912+0000", 2024-09-18T08:13:27.902 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.00013171799999999999 2024-09-18T08:13:27.902 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:13:27.902 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:13:27.902 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "waiting for subops from 1,2,4", 2024-09-18T08:13:27.902 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:53.837063+0000", 2024-09-18T08:13:27.902 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.00015040799999999999 2024-09-18T08:13:27.903 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:13:27.903 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:13:27.903 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:13:27.903 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:53.843015+0000", 2024-09-18T08:13:27.903 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.0059519250000000003 2024-09-18T08:13:27.903 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:13:27.903 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:13:27.903 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:13:27.903 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:53.843063+0000", 2024-09-18T08:13:27.903 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 4.8199999999999999e-05 2024-09-18T08:13:27.903 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:13:27.903 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:13:27.903 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "op_commit", 2024-09-18T08:13:27.903 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:53.848550+0000", 2024-09-18T08:13:27.903 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.0054868929999999996 2024-09-18T08:13:27.904 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:13:27.904 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:13:27.904 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:13:27.904 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:53.848663+0000", 2024-09-18T08:13:27.904 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.000113195 2024-09-18T08:13:27.904 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:13:27.904 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:13:27.904 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "commit_sent", 2024-09-18T08:13:27.904 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:53.848860+0000", 2024-09-18T08:13:27.904 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.00019676000000000001 2024-09-18T08:13:27.904 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:13:27.904 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:13:27.904 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "done", 2024-09-18T08:13:27.904 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:53.849017+0000", 2024-09-18T08:13:27.905 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.000156971 2024-09-18T08:13:27.905 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-09-18T08:13:27.905 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-09-18T08:13:27.905 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-09-18T08:13:27.905 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:13:27.905 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:13:27.905 INFO:teuthology.orchestra.run.smithi007.stdout: "description": "osd_op(client.15216.0:69262 8.f 8:f2d117f8:::benchmark_data_smithi007_101583_object69261:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e198)", 2024-09-18T08:13:27.905 INFO:teuthology.orchestra.run.smithi007.stdout: "initiated_at": "2024-09-18T08:08:51.776735+0000", 2024-09-18T08:13:27.905 INFO:teuthology.orchestra.run.smithi007.stdout: "age": 276.002658978, 2024-09-18T08:13:27.905 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 2.0727967970000001, 2024-09-18T08:13:27.905 INFO:teuthology.orchestra.run.smithi007.stdout: "continuous": false, 2024-09-18T08:13:27.905 INFO:teuthology.orchestra.run.smithi007.stdout: "type_data": { 2024-09-18T08:13:27.905 INFO:teuthology.orchestra.run.smithi007.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T08:13:27.905 INFO:teuthology.orchestra.run.smithi007.stdout: "client_info": { 2024-09-18T08:13:27.906 INFO:teuthology.orchestra.run.smithi007.stdout: "client": "client.15216", 2024-09-18T08:13:27.906 INFO:teuthology.orchestra.run.smithi007.stdout: "client_addr": "172.21.15.7:0/2596041177", 2024-09-18T08:13:27.906 INFO:teuthology.orchestra.run.smithi007.stdout: "tid": 69262 2024-09-18T08:13:27.906 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:13:27.906 INFO:teuthology.orchestra.run.smithi007.stdout: "events": [ 2024-09-18T08:13:27.906 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:13:27.906 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "initiated", 2024-09-18T08:13:27.906 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:51.776735+0000", 2024-09-18T08:13:27.906 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-09-18T08:13:27.906 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:13:27.906 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:13:27.906 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "header_read", 2024-09-18T08:13:27.906 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:51.776735+0000", 2024-09-18T08:13:27.906 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-09-18T08:13:27.907 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:13:27.907 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:13:27.907 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "throttled", 2024-09-18T08:13:27.907 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:51.776737+0000", 2024-09-18T08:13:27.907 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 2.0990000000000001e-06 2024-09-18T08:13:27.907 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:13:27.907 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:13:27.907 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "all_read", 2024-09-18T08:13:27.907 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:51.776765+0000", 2024-09-18T08:13:27.907 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 2.7562000000000001e-05 2024-09-18T08:13:27.907 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:13:27.907 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:13:27.907 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "dispatched", 2024-09-18T08:13:27.907 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:51.776785+0000", 2024-09-18T08:13:27.908 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 2.0136000000000002e-05 2024-09-18T08:13:27.908 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:13:27.908 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:13:27.908 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "queued_for_pg", 2024-09-18T08:13:27.908 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:51.776794+0000", 2024-09-18T08:13:27.908 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 9.3109999999999995e-06 2024-09-18T08:13:27.908 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:13:27.908 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:13:27.908 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-09-18T08:13:27.908 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:51.776887+0000", 2024-09-18T08:13:27.908 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 9.3382999999999997e-05 2024-09-18T08:13:27.908 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:13:27.908 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:13:27.908 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "waiting for peered", 2024-09-18T08:13:27.909 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:51.776898+0000", 2024-09-18T08:13:27.909 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.0278999999999999e-05 2024-09-18T08:13:27.909 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:13:27.909 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:13:27.909 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-09-18T08:13:27.909 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:52.697921+0000", 2024-09-18T08:13:27.909 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.92102297700000002 2024-09-18T08:13:27.909 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:13:27.909 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:13:27.909 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "waiting for peered", 2024-09-18T08:13:27.909 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:52.697929+0000", 2024-09-18T08:13:27.909 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 8.0709999999999995e-06 2024-09-18T08:13:27.909 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:13:27.909 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:13:27.910 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-09-18T08:13:27.910 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:53.839602+0000", 2024-09-18T08:13:27.910 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.1416730690000001 2024-09-18T08:13:27.910 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:13:27.910 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:13:27.910 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "started", 2024-09-18T08:13:27.910 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:53.839748+0000", 2024-09-18T08:13:27.910 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.00014631299999999999 2024-09-18T08:13:27.910 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:13:27.910 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:13:27.910 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "waiting for subops from 1,2,4", 2024-09-18T08:13:27.910 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:53.839927+0000", 2024-09-18T08:13:27.910 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.00017868800000000001 2024-09-18T08:13:27.911 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:13:27 smithi007 ceph-mon[29459]: from='client.? 172.21.15.7:0/1855414533' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-18T08:13:27.911 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:13:27 smithi007 ceph-mon[29459]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-18T08:13:27.911 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:13:27 smithi007 ceph-mon[29459]: from='client.? 172.21.15.7:0/976374297' entity='client.admin' cmd={"prefix": "osd pg-upmap", "pgid": "6.3", "id": [5, 3, 2]} : dispatch 2024-09-18T08:13:27.911 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:13:27 smithi007 ceph-mon[29459]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pg-upmap", "pgid": "6.3", "id": [5, 3, 2]} : dispatch 2024-09-18T08:13:27.911 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:13:27 smithi007 ceph-mon[39249]: from='client.? 172.21.15.7:0/1855414533' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-18T08:13:27.911 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:13:27 smithi007 ceph-mon[39249]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-18T08:13:27.911 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:13:27 smithi007 ceph-mon[39249]: from='client.? 172.21.15.7:0/976374297' entity='client.admin' cmd={"prefix": "osd pg-upmap", "pgid": "6.3", "id": [5, 3, 2]} : dispatch 2024-09-18T08:13:27.911 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:13:27 smithi007 ceph-mon[39249]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pg-upmap", "pgid": "6.3", "id": [5, 3, 2]} : dispatch 2024-09-18T08:13:27.912 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:13:27.912 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:13:27.912 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:13:27.912 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:53.843113+0000", 2024-09-18T08:13:27.912 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.003186595 2024-09-18T08:13:27.912 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:13:27.912 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:13:27.913 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:13:27.913 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:53.848599+0000", 2024-09-18T08:13:27.913 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.0054851049999999997 2024-09-18T08:13:27.913 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:13:27.913 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:13:27.913 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "op_commit", 2024-09-18T08:13:27.913 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:53.848616+0000", 2024-09-18T08:13:27.913 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.785e-05 2024-09-18T08:13:27.913 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:13:27.913 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:13:27.913 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:13:27.913 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:53.849125+0000", 2024-09-18T08:13:27.913 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.000508579 2024-09-18T08:13:27.914 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:13:27.914 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:13:27.914 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "commit_sent", 2024-09-18T08:13:27.914 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:53.849311+0000", 2024-09-18T08:13:27.914 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.00018621299999999999 2024-09-18T08:13:27.914 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:13:27.914 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:13:27.914 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "done", 2024-09-18T08:13:27.914 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:53.849532+0000", 2024-09-18T08:13:27.914 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.00022056699999999999 2024-09-18T08:13:27.914 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-09-18T08:13:27.914 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-09-18T08:13:27.914 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-09-18T08:13:27.914 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:13:27.915 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:13:27.915 INFO:teuthology.orchestra.run.smithi007.stdout: "description": "osd_op(client.15216.0:69340 8.f 8:ffd37843:::benchmark_data_smithi007_101583_object69339:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e198)", 2024-09-18T08:13:27.915 INFO:teuthology.orchestra.run.smithi007.stdout: "initiated_at": "2024-09-18T08:08:51.891119+0000", 2024-09-18T08:13:27.915 INFO:teuthology.orchestra.run.smithi007.stdout: "age": 275.88827525200003, 2024-09-18T08:13:27.915 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.9666553580000001, 2024-09-18T08:13:27.915 INFO:teuthology.orchestra.run.smithi007.stdout: "continuous": false, 2024-09-18T08:13:27.915 INFO:teuthology.orchestra.run.smithi007.stdout: "type_data": { 2024-09-18T08:13:27.915 INFO:teuthology.orchestra.run.smithi007.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T08:13:27.915 INFO:teuthology.orchestra.run.smithi007.stdout: "client_info": { 2024-09-18T08:13:27.915 INFO:teuthology.orchestra.run.smithi007.stdout: "client": "client.15216", 2024-09-18T08:13:27.915 INFO:teuthology.orchestra.run.smithi007.stdout: "client_addr": "172.21.15.7:0/2596041177", 2024-09-18T08:13:27.915 INFO:teuthology.orchestra.run.smithi007.stdout: "tid": 69340 2024-09-18T08:13:27.915 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:13:27.916 INFO:teuthology.orchestra.run.smithi007.stdout: "events": [ 2024-09-18T08:13:27.916 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:13:27.916 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "initiated", 2024-09-18T08:13:27.916 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:51.891119+0000", 2024-09-18T08:13:27.916 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-09-18T08:13:27.916 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:13:27.916 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:13:27.916 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "header_read", 2024-09-18T08:13:27.916 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:51.891119+0000", 2024-09-18T08:13:27.916 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-09-18T08:13:27.916 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:13:27.916 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:13:27.916 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "throttled", 2024-09-18T08:13:27.916 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:51.891120+0000", 2024-09-18T08:13:27.917 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.6649999999999999e-06 2024-09-18T08:13:27.917 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:13:27.917 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:13:27.917 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "all_read", 2024-09-18T08:13:27.917 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:51.891145+0000", 2024-09-18T08:13:27.917 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 2.4229000000000001e-05 2024-09-18T08:13:27.917 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:13:27.917 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:13:27.917 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "dispatched", 2024-09-18T08:13:27.917 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:51.891156+0000", 2024-09-18T08:13:27.917 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.1479e-05 2024-09-18T08:13:27.917 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:13:27.917 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:13:27.917 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "queued_for_pg", 2024-09-18T08:13:27.918 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:51.891163+0000", 2024-09-18T08:13:27.918 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 6.4989999999999999e-06 2024-09-18T08:13:27.918 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:13:27.918 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:13:27.918 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-09-18T08:13:27.918 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:51.891224+0000", 2024-09-18T08:13:27.918 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 6.1450000000000003e-05 2024-09-18T08:13:27.918 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:13:27.918 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:13:27.918 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "waiting for peered", 2024-09-18T08:13:27.918 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:51.891236+0000", 2024-09-18T08:13:27.918 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.205e-05 2024-09-18T08:13:27.918 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:13:27.918 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:13:27.919 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-09-18T08:13:27.919 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:52.697963+0000", 2024-09-18T08:13:27.919 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.80672717599999999 2024-09-18T08:13:27.919 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:13:27.919 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:13:27.919 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "waiting for peered", 2024-09-18T08:13:27.919 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:52.697971+0000", 2024-09-18T08:13:27.919 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 7.9000000000000006e-06 2024-09-18T08:13:27.919 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:13:27.919 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:13:27.919 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-09-18T08:13:27.919 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:53.843149+0000", 2024-09-18T08:13:27.919 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.1451775449999999 2024-09-18T08:13:27.919 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:13:27.919 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:13:27.920 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "started", 2024-09-18T08:13:27.920 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:53.843325+0000", 2024-09-18T08:13:27.920 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.000176059 2024-09-18T08:13:27.920 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:13:27.920 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:13:27.920 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "waiting for subops from 1,2,4", 2024-09-18T08:13:27.920 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:53.843462+0000", 2024-09-18T08:13:27.920 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.000137446 2024-09-18T08:13:27.920 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:13:27.920 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:13:27.920 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "op_commit", 2024-09-18T08:13:27.920 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:53.848623+0000", 2024-09-18T08:13:27.920 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.0051606589999999997 2024-09-18T08:13:27.921 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:13:27.921 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:13:27.921 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:13:27.921 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:53.849075+0000", 2024-09-18T08:13:27.921 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.00045226899999999999 2024-09-18T08:13:27.921 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:13:27.921 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:13:27.921 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:13:27.921 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:53.849640+0000", 2024-09-18T08:13:27.921 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.00056498900000000001 2024-09-18T08:13:27.921 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:13:27.921 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:13:27.921 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:13:27.922 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:53.857586+0000", 2024-09-18T08:13:27.922 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.0079462319999999993 2024-09-18T08:13:27.922 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:13:27.922 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:13:27.922 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "commit_sent", 2024-09-18T08:13:27.922 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:53.857652+0000", 2024-09-18T08:13:27.922 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 6.5831e-05 2024-09-18T08:13:27.922 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:13:27.922 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:13:27.922 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "done", 2024-09-18T08:13:27.922 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:53.857774+0000", 2024-09-18T08:13:27.922 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.00012188000000000001 2024-09-18T08:13:27.922 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-09-18T08:13:27.922 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-09-18T08:13:27.923 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-09-18T08:13:27.923 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:13:27.923 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:13:27.923 INFO:teuthology.orchestra.run.smithi007.stdout: "description": "osd_op(client.15216.0:69360 8.f 8:fee0f726:::benchmark_data_smithi007_101583_object69359:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e198)", 2024-09-18T08:13:27.923 INFO:teuthology.orchestra.run.smithi007.stdout: "initiated_at": "2024-09-18T08:08:51.959160+0000", 2024-09-18T08:13:27.923 INFO:teuthology.orchestra.run.smithi007.stdout: "age": 275.82023423800001, 2024-09-18T08:13:27.923 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.8990693009999999, 2024-09-18T08:13:27.923 INFO:teuthology.orchestra.run.smithi007.stdout: "continuous": false, 2024-09-18T08:13:27.923 INFO:teuthology.orchestra.run.smithi007.stdout: "type_data": { 2024-09-18T08:13:27.923 INFO:teuthology.orchestra.run.smithi007.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T08:13:27.923 INFO:teuthology.orchestra.run.smithi007.stdout: "client_info": { 2024-09-18T08:13:27.923 INFO:teuthology.orchestra.run.smithi007.stdout: "client": "client.15216", 2024-09-18T08:13:27.923 INFO:teuthology.orchestra.run.smithi007.stdout: "client_addr": "172.21.15.7:0/2596041177", 2024-09-18T08:13:27.923 INFO:teuthology.orchestra.run.smithi007.stdout: "tid": 69360 2024-09-18T08:13:27.924 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:13:27.924 INFO:teuthology.orchestra.run.smithi007.stdout: "events": [ 2024-09-18T08:13:27.924 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:13:27.924 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "initiated", 2024-09-18T08:13:27.924 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:51.959160+0000", 2024-09-18T08:13:27.924 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-09-18T08:13:27.924 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:13:27.924 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:13:27.924 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "header_read", 2024-09-18T08:13:27.924 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:51.959160+0000", 2024-09-18T08:13:27.924 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-09-18T08:13:27.924 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:13:27.924 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:13:27.924 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "throttled", 2024-09-18T08:13:27.925 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:51.959162+0000", 2024-09-18T08:13:27.925 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 2.1069999999999999e-06 2024-09-18T08:13:27.925 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:13:27.925 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:13:27.925 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "all_read", 2024-09-18T08:13:27.925 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:51.959187+0000", 2024-09-18T08:13:27.925 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 2.4902e-05 2024-09-18T08:13:27.925 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:13:27.925 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:13:27.925 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "dispatched", 2024-09-18T08:13:27.925 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:51.959207+0000", 2024-09-18T08:13:27.925 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 2.0051000000000002e-05 2024-09-18T08:13:27.925 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:13:27.925 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:13:27.926 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "queued_for_pg", 2024-09-18T08:13:27.926 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:51.959215+0000", 2024-09-18T08:13:27.926 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 8.7679999999999997e-06 2024-09-18T08:13:27.926 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:13:27.926 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:13:27.926 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-09-18T08:13:27.926 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:51.959299+0000", 2024-09-18T08:13:27.926 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 8.3129999999999999e-05 2024-09-18T08:13:27.926 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:13:27.926 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:13:27.926 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "waiting for peered", 2024-09-18T08:13:27.926 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:51.959313+0000", 2024-09-18T08:13:27.926 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.3967e-05 2024-09-18T08:13:27.927 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:13:27.927 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:13:27.927 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-09-18T08:13:27.927 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:52.698965+0000", 2024-09-18T08:13:27.927 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.73965287599999996 2024-09-18T08:13:27.927 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:13:27.927 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:13:27.927 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "waiting for peered", 2024-09-18T08:13:27.927 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:52.698974+0000", 2024-09-18T08:13:27.927 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 8.1929999999999993e-06 2024-09-18T08:13:27.927 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:13:27.927 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:13:27.927 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-09-18T08:13:27.927 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:53.845783+0000", 2024-09-18T08:13:27.928 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.1468095030000001 2024-09-18T08:13:27.928 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:13:27.928 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:13:27.928 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "started", 2024-09-18T08:13:27.928 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:53.845945+0000", 2024-09-18T08:13:27.928 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.00016167500000000001 2024-09-18T08:13:27.928 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:13:27.928 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:13:27.928 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "waiting for subops from 1,2,4", 2024-09-18T08:13:27.928 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:53.846104+0000", 2024-09-18T08:13:27.928 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.00015877600000000001 2024-09-18T08:13:27.928 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:13:27.929 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:13:27.929 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "op_commit", 2024-09-18T08:13:27.929 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:53.849087+0000", 2024-09-18T08:13:27.929 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.0029835420000000001 2024-09-18T08:13:27.929 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:13:27.929 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:13:27.929 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:13:27.929 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:53.849594+0000", 2024-09-18T08:13:27.929 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.00050686399999999999 2024-09-18T08:13:27.929 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:13:27.929 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:13:27.929 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:13:27.929 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:53.857524+0000", 2024-09-18T08:13:27.929 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.0079296000000000002 2024-09-18T08:13:27.930 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:13:27.930 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:13:27.930 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:13:27.930 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:53.857905+0000", 2024-09-18T08:13:27.930 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.000381107 2024-09-18T08:13:27.930 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:13:27.930 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:13:27.930 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "commit_sent", 2024-09-18T08:13:27.930 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:53.858073+0000", 2024-09-18T08:13:27.930 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.00016861900000000001 2024-09-18T08:13:27.930 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:13:27.930 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:13:27.930 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "done", 2024-09-18T08:13:27.930 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:53.858229+0000", 2024-09-18T08:13:27.930 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.00015562099999999999 2024-09-18T08:13:27.931 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-09-18T08:13:27.931 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-09-18T08:13:27.931 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-09-18T08:13:27.931 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:13:27.931 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:13:27.931 INFO:teuthology.orchestra.run.smithi007.stdout: "description": "osd_op(client.15216.0:100478 8.f 8:fbdbf1d3:::benchmark_data_smithi007_101583_object100477:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e210)", 2024-09-18T08:13:27.931 INFO:teuthology.orchestra.run.smithi007.stdout: "initiated_at": "2024-09-18T08:09:27.339441+0000", 2024-09-18T08:13:27.931 INFO:teuthology.orchestra.run.smithi007.stdout: "age": 240.43995274299999, 2024-09-18T08:13:27.931 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.0272729549999999, 2024-09-18T08:13:27.931 INFO:teuthology.orchestra.run.smithi007.stdout: "continuous": false, 2024-09-18T08:13:27.931 INFO:teuthology.orchestra.run.smithi007.stdout: "type_data": { 2024-09-18T08:13:27.931 INFO:teuthology.orchestra.run.smithi007.stdout: "flag_point": "reached pg", 2024-09-18T08:13:27.931 INFO:teuthology.orchestra.run.smithi007.stdout: "client_info": { 2024-09-18T08:13:27.931 INFO:teuthology.orchestra.run.smithi007.stdout: "client": "client.15216", 2024-09-18T08:13:27.932 INFO:teuthology.orchestra.run.smithi007.stdout: "client_addr": "172.21.15.7:0/2596041177", 2024-09-18T08:13:27.932 INFO:teuthology.orchestra.run.smithi007.stdout: "tid": 100478 2024-09-18T08:13:27.932 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:13:27.932 INFO:teuthology.orchestra.run.smithi007.stdout: "events": [ 2024-09-18T08:13:27.932 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:13:27.932 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "initiated", 2024-09-18T08:13:27.932 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:09:27.339441+0000", 2024-09-18T08:13:27.932 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-09-18T08:13:27.932 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:13:27.932 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:13:27.932 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "header_read", 2024-09-18T08:13:27.932 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:09:27.339441+0000", 2024-09-18T08:13:27.932 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-09-18T08:13:27.933 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:13:27.933 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:13:27.933 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "throttled", 2024-09-18T08:13:27.933 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:09:27.339442+0000", 2024-09-18T08:13:27.933 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 8.9800000000000002e-07 2024-09-18T08:13:27.933 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:13:27.933 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:13:27.933 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "all_read", 2024-09-18T08:13:27.933 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:09:27.339463+0000", 2024-09-18T08:13:27.933 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 2.0744999999999999e-05 2024-09-18T08:13:27.933 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:13:27.933 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:13:27.933 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "dispatched", 2024-09-18T08:13:27.933 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:09:27.339477+0000", 2024-09-18T08:13:27.934 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.4656e-05 2024-09-18T08:13:27.934 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:13:27.934 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:13:27.934 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "queued_for_pg", 2024-09-18T08:13:27.934 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:09:27.339485+0000", 2024-09-18T08:13:27.934 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 7.2699999999999999e-06 2024-09-18T08:13:27.934 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:13:27.934 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:13:27.934 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-09-18T08:13:27.934 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:09:27.339537+0000", 2024-09-18T08:13:27.934 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 5.2635e-05 2024-09-18T08:13:27.934 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:13:27.934 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:13:27.934 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "started", 2024-09-18T08:13:27.935 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:09:27.339659+0000", 2024-09-18T08:13:27.935 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.000121951 2024-09-18T08:13:27.935 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:13:27.935 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:13:27.935 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "waiting for subops from 1,2,4", 2024-09-18T08:13:27.935 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:09:27.339805+0000", 2024-09-18T08:13:27.935 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.00014521799999999999 2024-09-18T08:13:27.935 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:13:27.935 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:13:27.935 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "op_commit", 2024-09-18T08:13:27.935 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:09:27.343945+0000", 2024-09-18T08:13:27.935 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.0041406120000000001 2024-09-18T08:13:27.935 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:13:27.936 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:13:27.936 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:13:27.936 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:09:27.345369+0000", 2024-09-18T08:13:27.936 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.0014239299999999999 2024-09-18T08:13:27.936 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:13:27.936 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:13:27.936 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:13:27.936 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:09:27.345414+0000", 2024-09-18T08:13:27.936 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 4.4471999999999997e-05 2024-09-18T08:13:27.936 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:13:27.936 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:13:27.936 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-09-18T08:13:27.936 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:09:27.362113+0000", 2024-09-18T08:13:27.936 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.016699421999999998 2024-09-18T08:13:27.937 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:13:27.937 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:13:27.937 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "waiting for peered", 2024-09-18T08:13:27.937 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:09:27.362124+0000", 2024-09-18T08:13:27.937 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.1268e-05 2024-09-18T08:13:27.937 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:13:27.937 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:13:27.937 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-09-18T08:13:27.937 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:09:28.366617+0000", 2024-09-18T08:13:27.937 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.0044926059999999 2024-09-18T08:13:27.937 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:13:27.937 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:13:27.937 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "done", 2024-09-18T08:13:27.938 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:09:28.366714+0000", 2024-09-18T08:13:27.938 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 9.7272e-05 2024-09-18T08:13:27.938 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-09-18T08:13:27.938 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-09-18T08:13:27.938 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-09-18T08:13:27.938 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:13:27.938 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:13:27.938 INFO:teuthology.orchestra.run.smithi007.stdout: "description": "osd_op(client.15216.0:100489 8.f 8:f94e46ee:::benchmark_data_smithi007_101583_object100488:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e211)", 2024-09-18T08:13:27.938 INFO:teuthology.orchestra.run.smithi007.stdout: "initiated_at": "2024-09-18T08:09:27.351893+0000", 2024-09-18T08:13:27.938 INFO:teuthology.orchestra.run.smithi007.stdout: "age": 240.42750042899999, 2024-09-18T08:13:27.938 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.0213833750000001, 2024-09-18T08:13:27.938 INFO:teuthology.orchestra.run.smithi007.stdout: "continuous": false, 2024-09-18T08:13:27.938 INFO:teuthology.orchestra.run.smithi007.stdout: "type_data": { 2024-09-18T08:13:27.939 INFO:teuthology.orchestra.run.smithi007.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T08:13:27.939 INFO:teuthology.orchestra.run.smithi007.stdout: "client_info": { 2024-09-18T08:13:27.939 INFO:teuthology.orchestra.run.smithi007.stdout: "client": "client.15216", 2024-09-18T08:13:27.939 INFO:teuthology.orchestra.run.smithi007.stdout: "client_addr": "172.21.15.7:0/2596041177", 2024-09-18T08:13:27.939 INFO:teuthology.orchestra.run.smithi007.stdout: "tid": 100489 2024-09-18T08:13:27.939 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:13:27.939 INFO:teuthology.orchestra.run.smithi007.stdout: "events": [ 2024-09-18T08:13:27.939 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:13:27.939 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "initiated", 2024-09-18T08:13:27.939 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:09:27.351893+0000", 2024-09-18T08:13:27.939 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-09-18T08:13:27.939 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:13:27.939 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:13:27.939 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "header_read", 2024-09-18T08:13:27.940 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:09:27.351893+0000", 2024-09-18T08:13:27.940 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-09-18T08:13:27.940 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:13:27.940 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:13:27.940 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "throttled", 2024-09-18T08:13:27.940 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:09:27.351894+0000", 2024-09-18T08:13:27.940 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 5.5300000000000004e-07 2024-09-18T08:13:27.940 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:13:27.940 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:13:27.940 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "all_read", 2024-09-18T08:13:27.940 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:09:27.351919+0000", 2024-09-18T08:13:27.940 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 2.4837999999999998e-05 2024-09-18T08:13:27.940 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:13:27.941 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:13:27.941 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "dispatched", 2024-09-18T08:13:27.941 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:09:27.351926+0000", 2024-09-18T08:13:27.941 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 7.277e-06 2024-09-18T08:13:27.941 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:13:27.941 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:13:27.941 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "queued_for_pg", 2024-09-18T08:13:27.941 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:09:27.351933+0000", 2024-09-18T08:13:27.941 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 6.4810000000000003e-06 2024-09-18T08:13:27.941 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:13:27.941 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:13:27.941 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-09-18T08:13:27.941 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:09:27.352062+0000", 2024-09-18T08:13:27.941 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.00012982900000000001 2024-09-18T08:13:27.942 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:13:27.942 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:13:27.942 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "op must wait for map", 2024-09-18T08:13:27.942 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:09:27.352069+0000", 2024-09-18T08:13:27.942 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 6.8739999999999997e-06 2024-09-18T08:13:27.942 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:13:27.942 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:13:27.942 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-09-18T08:13:27.942 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:09:27.368007+0000", 2024-09-18T08:13:27.942 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.015938009999999999 2024-09-18T08:13:27.942 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:13:27.942 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:13:27.942 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "waiting for peered", 2024-09-18T08:13:27.942 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:09:27.368018+0000", 2024-09-18T08:13:27.943 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.0781000000000001e-05 2024-09-18T08:13:27.943 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:13:27.943 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:13:27.943 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-09-18T08:13:27.943 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:09:28.367234+0000", 2024-09-18T08:13:27.943 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.99921584100000005 2024-09-18T08:13:27.943 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:13:27.943 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:13:27.943 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "started", 2024-09-18T08:13:27.943 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:09:28.367328+0000", 2024-09-18T08:13:27.943 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 9.4083e-05 2024-09-18T08:13:27.943 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:13:27.943 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:13:27.944 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "waiting for subops from 1,2", 2024-09-18T08:13:27.944 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:09:28.367448+0000", 2024-09-18T08:13:27.944 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.00012029800000000001 2024-09-18T08:13:27.944 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:13:27.944 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:13:27.944 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:13:27.944 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:09:28.373055+0000", 2024-09-18T08:13:27.944 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.0056071460000000004 2024-09-18T08:13:27.944 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:13:27.944 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:13:27.944 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:13:27.944 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:09:28.373134+0000", 2024-09-18T08:13:27.944 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 7.8740999999999997e-05 2024-09-18T08:13:27.944 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:13:27.945 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:13:27.945 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "op_commit", 2024-09-18T08:13:27.945 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:09:28.373213+0000", 2024-09-18T08:13:27.945 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 7.894e-05 2024-09-18T08:13:27.945 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:13:27.945 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:13:27.945 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "commit_sent", 2024-09-18T08:13:27.945 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:09:28.373255+0000", 2024-09-18T08:13:27.945 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 4.1718000000000003e-05 2024-09-18T08:13:27.945 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:13:27.945 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:13:27.945 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "done", 2024-09-18T08:13:27.945 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:09:28.373277+0000", 2024-09-18T08:13:27.945 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 2.1965000000000001e-05 2024-09-18T08:13:27.946 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-09-18T08:13:27.946 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-09-18T08:13:27.946 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-09-18T08:13:27.946 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-09-18T08:13:27.946 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-09-18T08:13:27.946 INFO:teuthology.orchestra.run.smithi007.stdout:} 2024-09-18T08:13:27.946 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:13:27 smithi050 ceph-mon[32371]: from='client.? 172.21.15.7:0/1855414533' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-18T08:13:27.946 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:13:27 smithi050 ceph-mon[32371]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-18T08:13:27.947 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:13:27 smithi050 ceph-mon[32371]: from='client.? 172.21.15.7:0/976374297' entity='client.admin' cmd={"prefix": "osd pg-upmap", "pgid": "6.3", "id": [5, 3, 2]} : dispatch 2024-09-18T08:13:27.947 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:13:27 smithi050 ceph-mon[32371]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pg-upmap", "pgid": "6.3", "id": [5, 3, 2]} : dispatch 2024-09-18T08:13:27.956 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6 2024-09-18T08:13:27.995 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:13:27.995 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:13:27.995 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6' 2024-09-18T08:13:28.096 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6 2024-09-18T08:13:28.137 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:13:28.137 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:13:28.137 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6' 2024-09-18T08:13:28.238 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5 2024-09-18T08:13:28.276 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:13:28.277 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:13:28.277 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5' 2024-09-18T08:13:28.378 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3 2024-09-18T08:13:28.400 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:13:28.401 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:13:28.401 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3' 2024-09-18T08:13:28.502 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4 2024-09-18T08:13:28.536 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:13:28.537 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:13:28.537 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4' 2024-09-18T08:13:28.638 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0 2024-09-18T08:13:28.667 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:13:28.667 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:13:28.668 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0' 2024-09-18T08:13:28.769 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6 2024-09-18T08:13:28.812 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:13:28.813 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:13:28.813 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6' 2024-09-18T08:13:28.905 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:13:28 smithi050 ceph-mon[32371]: pgmap v514: 158 pgs: 30 active+undersized, 34 active+undersized+degraded, 94 active+clean; 5.2 GiB data, 18 GiB used, 698 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.47k op/s; 48760/258474 objects degraded (18.865%) 2024-09-18T08:13:28.906 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:13:28 smithi050 ceph-mon[32371]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-18T08:13:28.906 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:13:28 smithi050 ceph-mon[32371]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-18T08:13:28.906 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:13:28 smithi050 ceph-mon[32371]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pg-upmap", "pgid": "6.3", "id": [5, 3, 2]}]': finished 2024-09-18T08:13:28.906 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:13:28 smithi050 ceph-mon[32371]: osdmap e280: 8 total, 7 up, 8 in 2024-09-18T08:13:28.914 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7 2024-09-18T08:13:28.947 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:13:28.948 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:13:28.948 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7' 2024-09-18T08:13:28.980 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:13:28 smithi007 ceph-mon[29459]: pgmap v514: 158 pgs: 30 active+undersized, 34 active+undersized+degraded, 94 active+clean; 5.2 GiB data, 18 GiB used, 698 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.47k op/s; 48760/258474 objects degraded (18.865%) 2024-09-18T08:13:28.980 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:13:28 smithi007 ceph-mon[29459]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-18T08:13:28.980 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:13:28 smithi007 ceph-mon[29459]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-18T08:13:28.980 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:13:28 smithi007 ceph-mon[29459]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pg-upmap", "pgid": "6.3", "id": [5, 3, 2]}]': finished 2024-09-18T08:13:28.981 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:13:28 smithi007 ceph-mon[29459]: osdmap e280: 8 total, 7 up, 8 in 2024-09-18T08:13:28.981 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:13:28 smithi007 ceph-mon[39249]: pgmap v514: 158 pgs: 30 active+undersized, 34 active+undersized+degraded, 94 active+clean; 5.2 GiB data, 18 GiB used, 698 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.47k op/s; 48760/258474 objects degraded (18.865%) 2024-09-18T08:13:28.981 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:13:28 smithi007 ceph-mon[39249]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-18T08:13:28.981 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:13:28 smithi007 ceph-mon[39249]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-18T08:13:28.981 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:13:28 smithi007 ceph-mon[39249]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pg-upmap", "pgid": "6.3", "id": [5, 3, 2]}]': finished 2024-09-18T08:13:28.981 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:13:28 smithi007 ceph-mon[39249]: osdmap e280: 8 total, 7 up, 8 in 2024-09-18T08:13:29.049 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0 2024-09-18T08:13:29.079 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:13:29.080 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:13:29.080 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0' 2024-09-18T08:13:29.181 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7 2024-09-18T08:13:29.218 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:13:29.219 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:13:29.219 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7' 2024-09-18T08:13:29.320 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3 2024-09-18T08:13:29.361 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:13:29.361 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:13:29.362 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3' 2024-09-18T08:13:29.462 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6 2024-09-18T08:13:29.502 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:13:29.503 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:13:29.503 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6' 2024-09-18T08:13:29.604 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2 2024-09-18T08:13:29.622 INFO:journalctl@ceph.iscsi.iscsi.a.smithi050.stdout:Sep 18 08:13:29 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-iscsi-iscsi-a[57782]: debug there is no tcmu-runner data available 2024-09-18T08:13:29.636 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:13:29.637 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:13:29.637 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2' 2024-09-18T08:13:29.730 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:13:29 smithi007 ceph-mon[29459]: 4.5 scrub starts 2024-09-18T08:13:29.730 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:13:29 smithi007 ceph-mon[29459]: osdmap e281: 8 total, 7 up, 8 in 2024-09-18T08:13:29.730 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:13:29 smithi007 ceph-mon[29459]: 4.5 scrub ok 2024-09-18T08:13:29.730 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:13:29 smithi007 ceph-mon[29459]: 6.11 scrub starts 2024-09-18T08:13:29.730 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:13:29 smithi007 ceph-mon[29459]: 6.11 scrub ok 2024-09-18T08:13:29.730 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:13:29 smithi007 ceph-mon[29459]: Health check update: Degraded data redundancy: 48760/258474 objects degraded (18.865%), 34 pgs degraded, 64 pgs undersized (PG_DEGRADED) 2024-09-18T08:13:29.731 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:13:29 smithi007 ceph-mon[39249]: 4.5 scrub starts 2024-09-18T08:13:29.731 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:13:29 smithi007 ceph-mon[39249]: osdmap e281: 8 total, 7 up, 8 in 2024-09-18T08:13:29.731 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:13:29 smithi007 ceph-mon[39249]: 4.5 scrub ok 2024-09-18T08:13:29.731 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:13:29 smithi007 ceph-mon[39249]: 6.11 scrub starts 2024-09-18T08:13:29.731 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:13:29 smithi007 ceph-mon[39249]: 6.11 scrub ok 2024-09-18T08:13:29.731 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:13:29 smithi007 ceph-mon[39249]: Health check update: Degraded data redundancy: 48760/258474 objects degraded (18.865%), 34 pgs degraded, 64 pgs undersized (PG_DEGRADED) 2024-09-18T08:13:29.738 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3 2024-09-18T08:13:29.769 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:13:29.770 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:13:29.770 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3' 2024-09-18T08:13:29.871 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6 2024-09-18T08:13:29.898 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:13:29.899 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:13:29.899 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6' 2024-09-18T08:13:29.906 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:13:29 smithi050 ceph-mon[32371]: 4.5 scrub starts 2024-09-18T08:13:29.906 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:13:29 smithi050 ceph-mon[32371]: osdmap e281: 8 total, 7 up, 8 in 2024-09-18T08:13:29.906 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:13:29 smithi050 ceph-mon[32371]: 4.5 scrub ok 2024-09-18T08:13:29.906 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:13:29 smithi050 ceph-mon[32371]: 6.11 scrub starts 2024-09-18T08:13:29.906 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:13:29 smithi050 ceph-mon[32371]: 6.11 scrub ok 2024-09-18T08:13:29.906 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:13:29 smithi050 ceph-mon[32371]: Health check update: Degraded data redundancy: 48760/258474 objects degraded (18.865%), 34 pgs degraded, 64 pgs undersized (PG_DEGRADED) 2024-09-18T08:13:30.000 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6 2024-09-18T08:13:30.035 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:13:30.036 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:13:30.036 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6' 2024-09-18T08:13:30.137 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7 2024-09-18T08:13:30.176 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:13:30.176 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:13:30.177 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7' 2024-09-18T08:13:30.278 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0 2024-09-18T08:13:30.310 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:13:30.311 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:13:30.312 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0' 2024-09-18T08:13:30.412 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4 2024-09-18T08:13:30.423 DEBUG:teuthology.orchestra.run.smithi050:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid 779349ca-7593-11ef-bceb-c7b262605968 -- ceph daemon osd.4 dump_ops_in_flight 2024-09-18T08:13:30.448 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:13:30.449 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:13:30.450 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4' 2024-09-18T08:13:30.550 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3 2024-09-18T08:13:30.581 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:13:30.582 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:13:30.582 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3' 2024-09-18T08:13:30.683 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0 2024-09-18T08:13:30.709 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:13:30 smithi007 ceph-mon[39249]: pgmap v517: 158 pgs: 1 peering, 30 active+undersized, 34 active+undersized+degraded, 93 active+clean; 4.8 GiB data, 17 GiB used, 699 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.83k op/s; 44660/236889 objects degraded (18.853%) 2024-09-18T08:13:30.709 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:13:30 smithi007 ceph-mon[39249]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-18T08:13:30.709 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:13:30 smithi007 ceph-mon[39249]: Health check failed: Reduced data availability: 1 pg inactive, 1 pg peering (PG_AVAILABILITY) 2024-09-18T08:13:30.709 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:13:30 smithi007 ceph-mon[29459]: pgmap v517: 158 pgs: 1 peering, 30 active+undersized, 34 active+undersized+degraded, 93 active+clean; 4.8 GiB data, 17 GiB used, 699 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.83k op/s; 44660/236889 objects degraded (18.853%) 2024-09-18T08:13:30.709 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:13:30 smithi007 ceph-mon[29459]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-18T08:13:30.710 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:13:30 smithi007 ceph-mon[29459]: Health check failed: Reduced data availability: 1 pg inactive, 1 pg peering (PG_AVAILABILITY) 2024-09-18T08:13:30.717 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:13:30.717 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:13:30.718 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0' 2024-09-18T08:13:30.819 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2 2024-09-18T08:13:30.849 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:13:30.850 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:13:30.850 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2' 2024-09-18T08:13:30.906 INFO:teuthology.orchestra.run.smithi050.stderr:Inferring config /var/lib/ceph/779349ca-7593-11ef-bceb-c7b262605968/mon.b/config 2024-09-18T08:13:30.906 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:13:30 smithi050 ceph-mon[32371]: pgmap v517: 158 pgs: 1 peering, 30 active+undersized, 34 active+undersized+degraded, 93 active+clean; 4.8 GiB data, 17 GiB used, 699 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.83k op/s; 44660/236889 objects degraded (18.853%) 2024-09-18T08:13:30.906 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:13:30 smithi050 ceph-mon[32371]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-18T08:13:30.906 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:13:30 smithi050 ceph-mon[32371]: Health check failed: Reduced data availability: 1 pg inactive, 1 pg peering (PG_AVAILABILITY) 2024-09-18T08:13:30.951 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5 2024-09-18T08:13:30.992 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:13:30.992 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:13:30.993 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5' 2024-09-18T08:13:30.994 INFO:teuthology.orchestra.run.smithi007.stdout:osd.0: {} 2024-09-18T08:13:30.994 INFO:teuthology.orchestra.run.smithi007.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-18T08:13:30.994 INFO:teuthology.orchestra.run.smithi007.stderr:Error ENXIO: problem getting command descriptions from osd.1 2024-09-18T08:13:31.019 INFO:teuthology.orchestra.run.smithi007.stdout:osd.2: {} 2024-09-18T08:13:31.020 INFO:teuthology.orchestra.run.smithi007.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-18T08:13:31.044 INFO:teuthology.orchestra.run.smithi007.stdout:osd.3: {} 2024-09-18T08:13:31.044 INFO:teuthology.orchestra.run.smithi007.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-18T08:13:31.075 INFO:teuthology.orchestra.run.smithi007.stdout:osd.4: {} 2024-09-18T08:13:31.075 INFO:teuthology.orchestra.run.smithi007.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-18T08:13:31.094 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2 2024-09-18T08:13:31.114 INFO:teuthology.orchestra.run.smithi007.stdout:osd.5: {} 2024-09-18T08:13:31.115 INFO:teuthology.orchestra.run.smithi007.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-18T08:13:31.133 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:13:31.134 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:13:31.134 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2' 2024-09-18T08:13:31.152 INFO:teuthology.orchestra.run.smithi007.stdout:osd.6: {} 2024-09-18T08:13:31.153 INFO:teuthology.orchestra.run.smithi007.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-18T08:13:31.179 INFO:teuthology.orchestra.run.smithi007.stdout:osd.7: {} 2024-09-18T08:13:31.180 INFO:teuthology.orchestra.run.smithi007.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-18T08:13:31.235 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4 2024-09-18T08:13:31.271 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:13:31.272 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:13:31.272 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4' 2024-09-18T08:13:31.373 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6 2024-09-18T08:13:31.414 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:13:31.415 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:13:31.415 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6' 2024-09-18T08:13:31.516 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7 2024-09-18T08:13:31.558 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:13:31.558 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:13:31.559 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7' 2024-09-18T08:13:31.660 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5 2024-09-18T08:13:31.699 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:13:31.700 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:13:31.700 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5' 2024-09-18T08:13:31.798 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:13:31 smithi050 ceph-mon[32371]: 4.7 scrub starts 2024-09-18T08:13:31.798 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:13:31 smithi050 ceph-mon[32371]: 4.7 scrub ok 2024-09-18T08:13:31.798 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:13:31 smithi050 ceph-mon[32371]: 4.1e scrub starts 2024-09-18T08:13:31.798 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:13:31 smithi050 ceph-mon[32371]: 4.1e scrub ok 2024-09-18T08:13:31.798 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:13:31 smithi050 ceph-mon[32371]: from='client.? 172.21.15.7:0/4280010098' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-18T08:13:31.801 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0 2024-09-18T08:13:31.830 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:13:31.831 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:13:31.831 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0' 2024-09-18T08:13:31.863 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:13:31 smithi007 ceph-mon[39249]: 4.7 scrub starts 2024-09-18T08:13:31.886 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:13:31 smithi007 ceph-mon[39249]: 4.7 scrub ok 2024-09-18T08:13:31.886 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:13:31 smithi007 ceph-mon[39249]: 4.1e scrub starts 2024-09-18T08:13:31.886 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:13:31 smithi007 ceph-mon[39249]: 4.1e scrub ok 2024-09-18T08:13:31.886 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:13:31 smithi007 ceph-mon[39249]: from='client.? 172.21.15.7:0/4280010098' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-18T08:13:31.886 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:13:31 smithi007 ceph-mon[29459]: 4.7 scrub starts 2024-09-18T08:13:31.886 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:13:31 smithi007 ceph-mon[29459]: 4.7 scrub ok 2024-09-18T08:13:31.886 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:13:31 smithi007 ceph-mon[29459]: 4.1e scrub starts 2024-09-18T08:13:31.887 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:13:31 smithi007 ceph-mon[29459]: 4.1e scrub ok 2024-09-18T08:13:31.887 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:13:31 smithi007 ceph-mon[29459]: from='client.? 172.21.15.7:0/4280010098' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-18T08:13:31.887 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-09-18T08:13:31.887 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-09-18T08:13:31.901 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid 779349ca-7593-11ef-bceb-c7b262605968 -- ceph osd unset nodeep-scrub 2024-09-18T08:13:31.933 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6 2024-09-18T08:13:31.972 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:13:31.973 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:13:31.973 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6' 2024-09-18T08:13:32.074 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3 2024-09-18T08:13:32.108 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:13:32.109 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:13:32.109 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3' 2024-09-18T08:13:32.209 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6 2024-09-18T08:13:32.253 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:13:32.254 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:13:32.254 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6' 2024-09-18T08:13:32.279 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/779349ca-7593-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T08:13:32.354 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2 2024-09-18T08:13:32.380 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:13:32.381 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:13:32.381 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2' 2024-09-18T08:13:32.482 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3 2024-09-18T08:13:32.511 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:13:32.512 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:13:32.512 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3' 2024-09-18T08:13:32.613 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6 2024-09-18T08:13:32.647 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:13:32.647 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:13:32.648 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6' 2024-09-18T08:13:32.665 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:13:32 smithi007 ceph-mon[29459]: purged_snaps scrub starts 2024-09-18T08:13:32.665 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:13:32 smithi007 ceph-mon[29459]: purged_snaps scrub ok 2024-09-18T08:13:32.665 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:13:32 smithi007 ceph-mon[29459]: pgmap v518: 158 pgs: 1 peering, 30 active+undersized, 34 active+undersized+degraded, 93 active+clean; 4.8 GiB data, 17 GiB used, 699 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.83k op/s; 44660/236889 objects degraded (18.853%) 2024-09-18T08:13:32.665 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:13:32 smithi007 ceph-mon[29459]: 5.e scrub starts 2024-09-18T08:13:32.665 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:13:32 smithi007 ceph-mon[29459]: 5.e scrub ok 2024-09-18T08:13:32.665 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:13:32 smithi007 ceph-mon[29459]: osdmap e282: 8 total, 7 up, 8 in 2024-09-18T08:13:32.665 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:13:32 smithi007 ceph-mon[29459]: 5.14 scrub starts 2024-09-18T08:13:32.665 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:13:32 smithi007 ceph-mon[29459]: 5.14 scrub ok 2024-09-18T08:13:32.665 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:13:32 smithi007 ceph-mon[39249]: purged_snaps scrub starts 2024-09-18T08:13:32.666 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:13:32 smithi007 ceph-mon[39249]: purged_snaps scrub ok 2024-09-18T08:13:32.666 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:13:32 smithi007 ceph-mon[39249]: pgmap v518: 158 pgs: 1 peering, 30 active+undersized, 34 active+undersized+degraded, 93 active+clean; 4.8 GiB data, 17 GiB used, 699 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.83k op/s; 44660/236889 objects degraded (18.853%) 2024-09-18T08:13:32.666 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:13:32 smithi007 ceph-mon[39249]: 5.e scrub starts 2024-09-18T08:13:32.666 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:13:32 smithi007 ceph-mon[39249]: 5.e scrub ok 2024-09-18T08:13:32.666 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:13:32 smithi007 ceph-mon[39249]: osdmap e282: 8 total, 7 up, 8 in 2024-09-18T08:13:32.666 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:13:32 smithi007 ceph-mon[39249]: 5.14 scrub starts 2024-09-18T08:13:32.666 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:13:32 smithi007 ceph-mon[39249]: 5.14 scrub ok 2024-09-18T08:13:32.749 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3 2024-09-18T08:13:32.781 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:13:32.782 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:13:32.782 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3' 2024-09-18T08:13:32.831 INFO:teuthology.orchestra.run.smithi050.stdout:ERROR: (22) Invalid argument 2024-09-18T08:13:32.831 INFO:teuthology.orchestra.run.smithi050.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-18T08:13:32.883 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2 2024-09-18T08:13:32.905 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:13:32 smithi050 ceph-mon[32371]: purged_snaps scrub starts 2024-09-18T08:13:32.905 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:13:32 smithi050 ceph-mon[32371]: purged_snaps scrub ok 2024-09-18T08:13:32.905 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:13:32 smithi050 ceph-mon[32371]: pgmap v518: 158 pgs: 1 peering, 30 active+undersized, 34 active+undersized+degraded, 93 active+clean; 4.8 GiB data, 17 GiB used, 699 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.83k op/s; 44660/236889 objects degraded (18.853%) 2024-09-18T08:13:32.906 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:13:32 smithi050 ceph-mon[32371]: 5.e scrub starts 2024-09-18T08:13:32.906 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:13:32 smithi050 ceph-mon[32371]: 5.e scrub ok 2024-09-18T08:13:32.906 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:13:32 smithi050 ceph-mon[32371]: osdmap e282: 8 total, 7 up, 8 in 2024-09-18T08:13:32.906 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:13:32 smithi050 ceph-mon[32371]: 5.14 scrub starts 2024-09-18T08:13:32.906 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:13:32 smithi050 ceph-mon[32371]: 5.14 scrub ok 2024-09-18T08:13:32.912 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:13:32.913 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:13:32.913 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2' 2024-09-18T08:13:33.014 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4 2024-09-18T08:13:33.050 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:13:33.050 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:13:33.051 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4' 2024-09-18T08:13:33.151 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6 2024-09-18T08:13:33.191 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:13:33.191 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:13:33.192 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6' 2024-09-18T08:13:33.292 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4 2024-09-18T08:13:33.333 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:13:33.334 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:13:33.334 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4' 2024-09-18T08:13:33.435 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0 2024-09-18T08:13:33.462 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:13:33.462 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:13:33.463 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0' 2024-09-18T08:13:33.563 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5 2024-09-18T08:13:33.601 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:13:33.602 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:13:33.602 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5' 2024-09-18T08:13:33.652 INFO:teuthology.orchestra.run.smithi007.stderr:nodeep-scrub is unset 2024-09-18T08:13:33.655 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:13:33 smithi050 ceph-mon[32371]: 5.11 scrub starts 2024-09-18T08:13:33.655 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:13:33 smithi050 ceph-mon[32371]: 5.11 scrub ok 2024-09-18T08:13:33.656 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:13:33 smithi050 ceph-mon[32371]: from='client.? 172.21.15.7:0/2878450783' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-18T08:13:33.656 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:13:33 smithi050 ceph-mon[32371]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-18T08:13:33.656 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:13:33 smithi050 ceph-mon[32371]: from='mgr.24470 172.21.15.7:0/2594390261' entity='mgr.y' 2024-09-18T08:13:33.656 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:13:33 smithi050 ceph-mon[32371]: from='mgr.24470 172.21.15.7:0/2594390261' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-18T08:13:33.703 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5 2024-09-18T08:13:33.729 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:13:33.729 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:13:33.730 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5' 2024-09-18T08:13:33.830 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2 2024-09-18T08:13:33.861 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:13:33.861 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:13:33.862 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2' 2024-09-18T08:13:33.875 DEBUG:teuthology.orchestra.run.smithi050:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid 779349ca-7593-11ef-bceb-c7b262605968 -- ceph daemon osd.4 dump_blocked_ops 2024-09-18T08:13:33.889 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid 779349ca-7593-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-18T08:13:33.931 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:13:33 smithi007 ceph-mon[39249]: 5.11 scrub starts 2024-09-18T08:13:33.931 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:13:33 smithi007 ceph-mon[39249]: 5.11 scrub ok 2024-09-18T08:13:33.931 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:13:33 smithi007 ceph-mon[39249]: from='client.? 172.21.15.7:0/2878450783' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-18T08:13:33.931 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:13:33 smithi007 ceph-mon[39249]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-18T08:13:33.931 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:13:33 smithi007 ceph-mon[39249]: from='mgr.24470 172.21.15.7:0/2594390261' entity='mgr.y' 2024-09-18T08:13:33.931 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:13:33 smithi007 ceph-mon[39249]: from='mgr.24470 172.21.15.7:0/2594390261' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-18T08:13:33.932 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:13:33 smithi007 ceph-mon[29459]: 5.11 scrub starts 2024-09-18T08:13:33.932 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:13:33 smithi007 ceph-mon[29459]: 5.11 scrub ok 2024-09-18T08:13:33.932 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:13:33 smithi007 ceph-mon[29459]: from='client.? 172.21.15.7:0/2878450783' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-18T08:13:33.932 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:13:33 smithi007 ceph-mon[29459]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-18T08:13:33.932 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:13:33 smithi007 ceph-mon[29459]: from='mgr.24470 172.21.15.7:0/2594390261' entity='mgr.y' 2024-09-18T08:13:33.932 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:13:33 smithi007 ceph-mon[29459]: from='mgr.24470 172.21.15.7:0/2594390261' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-18T08:13:33.962 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4 2024-09-18T08:13:34.007 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:13:34.008 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:13:34.009 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4' 2024-09-18T08:13:34.109 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2 2024-09-18T08:13:34.142 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:13:34.143 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:13:34.143 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2' 2024-09-18T08:13:34.244 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4 2024-09-18T08:13:34.303 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:13:34.304 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:13:34.304 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4' 2024-09-18T08:13:34.338 INFO:teuthology.orchestra.run.smithi050.stderr:Inferring config /var/lib/ceph/779349ca-7593-11ef-bceb-c7b262605968/mon.b/config 2024-09-18T08:13:34.405 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4 2024-09-18T08:13:34.438 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 5, 7, 6, 4] out_osds: [] dead_osds: [1] live_osds: [0, 2, 3, 4, 5, 6, 7] 2024-09-18T08:13:34.439 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-18T08:13:34.439 INFO:tasks.thrashosds.thrasher:fixing pg num pool .mgr 2024-09-18T08:13:34.439 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid 779349ca-7593-11ef-bceb-c7b262605968 -- ceph pg dump --format=json 2024-09-18T08:13:34.445 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:13:34.446 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:13:34.446 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4' 2024-09-18T08:13:34.547 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7 2024-09-18T08:13:34.569 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:13:34.570 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:13:34.570 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7' 2024-09-18T08:13:34.671 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5 2024-09-18T08:13:34.701 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:13:34.702 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:13:34.702 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5' 2024-09-18T08:13:34.730 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:13:34 smithi007 ceph-mon[29459]: 3.13 scrub starts 2024-09-18T08:13:34.730 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:13:34 smithi007 ceph-mon[29459]: 3.13 scrub ok 2024-09-18T08:13:34.730 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:13:34 smithi007 ceph-mon[29459]: 6.3 scrub starts 2024-09-18T08:13:34.730 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:13:34 smithi007 ceph-mon[29459]: 6.3 scrub ok 2024-09-18T08:13:34.730 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:13:34 smithi007 ceph-mon[29459]: pgmap v520: 158 pgs: 1 peering, 30 active+undersized, 34 active+undersized+degraded, 93 active+clean; 4.8 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 852 B/s rd, 0 B/s wr, 1.20k op/s; 44660/236889 objects degraded (18.853%) 2024-09-18T08:13:34.730 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:13:34 smithi007 ceph-mon[29459]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-18T08:13:34.731 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:13:34 smithi007 ceph-mon[29459]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-18T08:13:34.731 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:13:34 smithi007 ceph-mon[29459]: osdmap e283: 8 total, 7 up, 8 in 2024-09-18T08:13:34.731 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:13:34 smithi007 ceph-mon[29459]: 4.3 scrub starts 2024-09-18T08:13:34.731 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:13:34 smithi007 ceph-mon[29459]: 4.3 scrub ok 2024-09-18T08:13:34.731 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:13:34 smithi007 ceph-mon[29459]: Health check update: Degraded data redundancy: 44660/236889 objects degraded (18.853%), 34 pgs degraded, 64 pgs undersized (PG_DEGRADED) 2024-09-18T08:13:34.731 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:13:34 smithi007 ceph-mon[39249]: 3.13 scrub starts 2024-09-18T08:13:34.731 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:13:34 smithi007 ceph-mon[39249]: 3.13 scrub ok 2024-09-18T08:13:34.731 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:13:34 smithi007 ceph-mon[39249]: 6.3 scrub starts 2024-09-18T08:13:34.731 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:13:34 smithi007 ceph-mon[39249]: 6.3 scrub ok 2024-09-18T08:13:34.731 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:13:34 smithi007 ceph-mon[39249]: pgmap v520: 158 pgs: 1 peering, 30 active+undersized, 34 active+undersized+degraded, 93 active+clean; 4.8 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 852 B/s rd, 0 B/s wr, 1.20k op/s; 44660/236889 objects degraded (18.853%) 2024-09-18T08:13:34.731 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:13:34 smithi007 ceph-mon[39249]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-18T08:13:34.732 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:13:34 smithi007 ceph-mon[39249]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-18T08:13:34.732 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:13:34 smithi007 ceph-mon[39249]: osdmap e283: 8 total, 7 up, 8 in 2024-09-18T08:13:34.732 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:13:34 smithi007 ceph-mon[39249]: 4.3 scrub starts 2024-09-18T08:13:34.732 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:13:34 smithi007 ceph-mon[39249]: 4.3 scrub ok 2024-09-18T08:13:34.732 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:13:34 smithi007 ceph-mon[39249]: Health check update: Degraded data redundancy: 44660/236889 objects degraded (18.853%), 34 pgs degraded, 64 pgs undersized (PG_DEGRADED) 2024-09-18T08:13:34.802 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6 2024-09-18T08:13:34.843 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:13:34.843 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:13:34.843 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6' 2024-09-18T08:13:34.905 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:13:34 smithi050 ceph-mon[32371]: 3.13 scrub starts 2024-09-18T08:13:34.905 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:13:34 smithi050 ceph-mon[32371]: 3.13 scrub ok 2024-09-18T08:13:34.905 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:13:34 smithi050 ceph-mon[32371]: 6.3 scrub starts 2024-09-18T08:13:34.906 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:13:34 smithi050 ceph-mon[32371]: 6.3 scrub ok 2024-09-18T08:13:34.906 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:13:34 smithi050 ceph-mon[32371]: pgmap v520: 158 pgs: 1 peering, 30 active+undersized, 34 active+undersized+degraded, 93 active+clean; 4.8 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 852 B/s rd, 0 B/s wr, 1.20k op/s; 44660/236889 objects degraded (18.853%) 2024-09-18T08:13:34.906 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:13:34 smithi050 ceph-mon[32371]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-18T08:13:34.906 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:13:34 smithi050 ceph-mon[32371]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-18T08:13:34.906 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:13:34 smithi050 ceph-mon[32371]: osdmap e283: 8 total, 7 up, 8 in 2024-09-18T08:13:34.906 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:13:34 smithi050 ceph-mon[32371]: 4.3 scrub starts 2024-09-18T08:13:34.906 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:13:34 smithi050 ceph-mon[32371]: 4.3 scrub ok 2024-09-18T08:13:34.906 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:13:34 smithi050 ceph-mon[32371]: Health check update: Degraded data redundancy: 44660/236889 objects degraded (18.853%), 34 pgs degraded, 64 pgs undersized (PG_DEGRADED) 2024-09-18T08:13:34.945 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3 2024-09-18T08:13:34.997 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:13:34.997 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:13:34.998 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3' 2024-09-18T08:13:35.099 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4 2024-09-18T08:13:35.134 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:13:35.134 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:13:35.135 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4' 2024-09-18T08:13:35.236 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6 2024-09-18T08:13:35.271 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:13:35.271 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:13:35.272 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6' 2024-09-18T08:13:35.372 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7 2024-09-18T08:13:35.408 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:13:35.408 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:13:35.409 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7' 2024-09-18T08:13:35.510 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5 2024-09-18T08:13:35.546 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:13:35.547 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:13:35.547 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5' 2024-09-18T08:13:35.549 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/779349ca-7593-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T08:13:35.603 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/779349ca-7593-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T08:13:35.648 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2 2024-09-18T08:13:35.680 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:13:35.681 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:13:35.681 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2' 2024-09-18T08:13:35.782 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7 2024-09-18T08:13:35.807 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:13:35.807 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:13:35.808 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7' 2024-09-18T08:13:35.850 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:13:35 smithi007 ceph-mon[29459]: 5.f scrub starts 2024-09-18T08:13:35.850 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:13:35 smithi007 ceph-mon[29459]: 5.f scrub ok 2024-09-18T08:13:35.850 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:13:35 smithi007 ceph-mon[39249]: 5.f scrub starts 2024-09-18T08:13:35.851 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:13:35 smithi007 ceph-mon[39249]: 5.f scrub ok 2024-09-18T08:13:35.905 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:13:35 smithi050 ceph-mon[32371]: 5.f scrub starts 2024-09-18T08:13:35.905 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:13:35 smithi050 ceph-mon[32371]: 5.f scrub ok 2024-09-18T08:13:35.909 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2 2024-09-18T08:13:35.937 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:13:35.937 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:13:35.938 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2' 2024-09-18T08:13:36.038 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5 2024-09-18T08:13:36.072 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:13:36.073 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:13:36.074 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5' 2024-09-18T08:13:36.159 INFO:teuthology.orchestra.run.smithi007.stdout:osd.0: {} 2024-09-18T08:13:36.159 INFO:teuthology.orchestra.run.smithi007.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-18T08:13:36.159 INFO:teuthology.orchestra.run.smithi007.stderr:Error ENXIO: problem getting command descriptions from osd.1 2024-09-18T08:13:36.175 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5 2024-09-18T08:13:36.182 INFO:teuthology.orchestra.run.smithi007.stdout:osd.2: {} 2024-09-18T08:13:36.182 INFO:teuthology.orchestra.run.smithi007.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-18T08:13:36.207 INFO:teuthology.orchestra.run.smithi007.stdout:osd.3: {} 2024-09-18T08:13:36.208 INFO:teuthology.orchestra.run.smithi007.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-18T08:13:36.213 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:13:36.214 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:13:36.214 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5' 2024-09-18T08:13:36.229 INFO:teuthology.orchestra.run.smithi007.stdout:osd.4: {} 2024-09-18T08:13:36.229 INFO:teuthology.orchestra.run.smithi007.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-18T08:13:36.256 INFO:teuthology.orchestra.run.smithi007.stdout:osd.5: {} 2024-09-18T08:13:36.257 INFO:teuthology.orchestra.run.smithi007.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-18T08:13:36.284 INFO:teuthology.orchestra.run.smithi007.stdout:osd.6: {} 2024-09-18T08:13:36.284 INFO:teuthology.orchestra.run.smithi007.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-18T08:13:36.310 INFO:teuthology.orchestra.run.smithi007.stdout:osd.7: {} 2024-09-18T08:13:36.311 INFO:teuthology.orchestra.run.smithi007.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-18T08:13:36.315 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5 2024-09-18T08:13:36.358 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:13:36.358 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:13:36.358 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5' 2024-09-18T08:13:36.459 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7 2024-09-18T08:13:36.493 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2024-09-18T08:13:36.493 INFO:teuthology.orchestra.run.smithi050.stdout: "ops": [], 2024-09-18T08:13:36.493 INFO:teuthology.orchestra.run.smithi050.stdout: "complaint_time": 30, 2024-09-18T08:13:36.493 INFO:teuthology.orchestra.run.smithi050.stdout: "num_blocked_ops": 0 2024-09-18T08:13:36.493 INFO:teuthology.orchestra.run.smithi050.stdout:} 2024-09-18T08:13:36.508 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:13:36.509 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:13:36.510 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7' 2024-09-18T08:13:36.610 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6 2024-09-18T08:13:36.648 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:13:36.649 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:13:36.649 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6' 2024-09-18T08:13:36.750 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5 2024-09-18T08:13:36.789 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:13:36.789 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:13:36.790 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5' 2024-09-18T08:13:36.890 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2 2024-09-18T08:13:36.900 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:13:36 smithi050 ceph-mon[32371]: 3.15 scrub starts 2024-09-18T08:13:36.900 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:13:36 smithi050 ceph-mon[32371]: 3.15 scrub ok 2024-09-18T08:13:36.900 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:13:36 smithi050 ceph-mon[32371]: pgmap v522: 158 pgs: 30 active+undersized, 34 active+undersized+degraded, 94 active+clean; 4.4 GiB data, 16 GiB used, 700 GiB / 715 GiB avail; 1.5 KiB/s rd, 0 B/s wr, 1.57k op/s; 40816/216258 objects degraded (18.874%) 2024-09-18T08:13:36.900 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:13:36 smithi050 ceph-mon[32371]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg inactive, 1 pg peering) 2024-09-18T08:13:36.900 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:13:36 smithi050 ceph-mon[32371]: from='client.? 172.21.15.7:0/1647932138' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-18T08:13:36.922 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:13:36.922 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:13:36.922 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2' 2024-09-18T08:13:36.980 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:13:36 smithi007 ceph-mon[29459]: 3.15 scrub starts 2024-09-18T08:13:36.980 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:13:36 smithi007 ceph-mon[29459]: 3.15 scrub ok 2024-09-18T08:13:36.980 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:13:36 smithi007 ceph-mon[29459]: pgmap v522: 158 pgs: 30 active+undersized, 34 active+undersized+degraded, 94 active+clean; 4.4 GiB data, 16 GiB used, 700 GiB / 715 GiB avail; 1.5 KiB/s rd, 0 B/s wr, 1.57k op/s; 40816/216258 objects degraded (18.874%) 2024-09-18T08:13:36.981 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:13:36 smithi007 ceph-mon[29459]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg inactive, 1 pg peering) 2024-09-18T08:13:36.981 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:13:36 smithi007 ceph-mon[29459]: from='client.? 172.21.15.7:0/1647932138' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-18T08:13:36.981 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Sep 18 08:13:36 smithi007 ceph-779349ca-7593-11ef-bceb-c7b262605968-mgr-y[29688]: ::ffff:172.21.15.50 - - [18/Sep/2024:08:13:36] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-09-18T08:13:36.981 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:13:36 smithi007 ceph-mon[39249]: 3.15 scrub starts 2024-09-18T08:13:36.981 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:13:36 smithi007 ceph-mon[39249]: 3.15 scrub ok 2024-09-18T08:13:36.981 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:13:36 smithi007 ceph-mon[39249]: pgmap v522: 158 pgs: 30 active+undersized, 34 active+undersized+degraded, 94 active+clean; 4.4 GiB data, 16 GiB used, 700 GiB / 715 GiB avail; 1.5 KiB/s rd, 0 B/s wr, 1.57k op/s; 40816/216258 objects degraded (18.874%) 2024-09-18T08:13:36.981 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:13:36 smithi007 ceph-mon[39249]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg inactive, 1 pg peering) 2024-09-18T08:13:36.981 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:13:36 smithi007 ceph-mon[39249]: from='client.? 172.21.15.7:0/1647932138' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-18T08:13:37.023 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6 2024-09-18T08:13:37.058 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:13:37.059 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:13:37.059 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6' 2024-09-18T08:13:37.160 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6 2024-09-18T08:13:37.197 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:13:37.198 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:13:37.198 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6' 2024-09-18T08:13:37.299 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2 2024-09-18T08:13:37.330 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:13:37.331 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:13:37.331 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2' 2024-09-18T08:13:37.385 DEBUG:teuthology.orchestra.run.smithi050:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid 779349ca-7593-11ef-bceb-c7b262605968 -- ceph daemon osd.4 dump_historic_ops 2024-09-18T08:13:37.433 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5 2024-09-18T08:13:37.486 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:13:37.487 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:13:37.487 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5' 2024-09-18T08:13:37.561 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid 779349ca-7593-11ef-bceb-c7b262605968 -- ceph osd set noscrub 2024-09-18T08:13:37.588 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3 2024-09-18T08:13:37.637 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:13:37.638 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:13:37.638 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3' 2024-09-18T08:13:37.739 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3 2024-09-18T08:13:37.780 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:13:37.781 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:13:37.781 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3' 2024-09-18T08:13:37.809 INFO:teuthology.orchestra.run.smithi050.stderr:Inferring config /var/lib/ceph/779349ca-7593-11ef-bceb-c7b262605968/mon.b/config 2024-09-18T08:13:37.882 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4 2024-09-18T08:13:37.920 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:13:37.921 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:13:37.921 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4' 2024-09-18T08:13:37.931 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:13:37 smithi050 ceph-mon[32371]: 6.6 scrub starts 2024-09-18T08:13:37.931 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:13:37 smithi050 ceph-mon[32371]: 6.6 scrub ok 2024-09-18T08:13:37.931 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:13:37 smithi050 ceph-mon[32371]: 6.5 scrub starts 2024-09-18T08:13:37.931 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:13:37 smithi050 ceph-mon[32371]: 6.5 scrub ok 2024-09-18T08:13:37.980 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:13:37 smithi007 ceph-mon[29459]: 6.6 scrub starts 2024-09-18T08:13:37.980 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:13:37 smithi007 ceph-mon[29459]: 6.6 scrub ok 2024-09-18T08:13:37.980 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:13:37 smithi007 ceph-mon[29459]: 6.5 scrub starts 2024-09-18T08:13:37.980 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:13:37 smithi007 ceph-mon[29459]: 6.5 scrub ok 2024-09-18T08:13:37.981 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:13:37 smithi007 ceph-mon[39249]: 6.6 scrub starts 2024-09-18T08:13:37.981 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:13:37 smithi007 ceph-mon[39249]: 6.6 scrub ok 2024-09-18T08:13:37.981 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:13:37 smithi007 ceph-mon[39249]: 6.5 scrub starts 2024-09-18T08:13:37.981 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:13:37 smithi007 ceph-mon[39249]: 6.5 scrub ok 2024-09-18T08:13:38.022 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5 2024-09-18T08:13:38.061 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:13:38.062 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:13:38.062 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5' 2024-09-18T08:13:38.162 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4 2024-09-18T08:13:38.204 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:13:38.205 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:13:38.205 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4' 2024-09-18T08:13:38.306 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4 2024-09-18T08:13:38.356 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:13:38.356 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:13:38.357 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4' 2024-09-18T08:13:38.407 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2024-09-18T08:13:38.407 INFO:teuthology.orchestra.run.smithi050.stdout: "size": 20, 2024-09-18T08:13:38.407 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 600, 2024-09-18T08:13:38.407 INFO:teuthology.orchestra.run.smithi050.stdout: "ops": [ 2024-09-18T08:13:38.407 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:13:38.407 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.15216.0:86225 8.5 8:ae8df458:::benchmark_data_smithi007_101583_object86224:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e205)", 2024-09-18T08:13:38.407 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-09-18T08:09:12.178973+0000", 2024-09-18T08:13:38.407 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 266.22126200600002, 2024-09-18T08:13:38.407 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.99102768600000002, 2024-09-18T08:13:38.408 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-09-18T08:13:38.408 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-09-18T08:13:38.408 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "reached pg", 2024-09-18T08:13:38.408 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2024-09-18T08:13:38.408 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.15216", 2024-09-18T08:13:38.408 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "172.21.15.7:0/2596041177", 2024-09-18T08:13:38.408 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 86225 2024-09-18T08:13:38.408 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:13:38.408 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-09-18T08:13:38.408 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:13:38.408 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-09-18T08:13:38.408 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:12.178973+0000", 2024-09-18T08:13:38.408 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-18T08:13:38.409 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:13:38.409 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:13:38.409 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-09-18T08:13:38.409 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:12.178973+0000", 2024-09-18T08:13:38.409 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-18T08:13:38.409 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:13:38.409 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:13:38.409 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-09-18T08:13:38.409 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:12.178975+0000", 2024-09-18T08:13:38.409 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.3969999999999999e-06 2024-09-18T08:13:38.409 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:13:38.409 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:13:38.409 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-09-18T08:13:38.410 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:12.179022+0000", 2024-09-18T08:13:38.410 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4.7580000000000002e-05 2024-09-18T08:13:38.410 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:13:38.410 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:13:38.410 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-09-18T08:13:38.410 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:12.179034+0000", 2024-09-18T08:13:38.410 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.1477e-05 2024-09-18T08:13:38.410 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:13:38.410 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:13:38.410 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-09-18T08:13:38.410 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:12.179040+0000", 2024-09-18T08:13:38.410 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 5.7529999999999998e-06 2024-09-18T08:13:38.411 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:13:38.411 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:13:38.411 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-18T08:13:38.411 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:12.179092+0000", 2024-09-18T08:13:38.411 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 5.2840000000000002e-05 2024-09-18T08:13:38.411 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:13:38.411 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:13:38.411 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "waiting for peered", 2024-09-18T08:13:38.411 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:12.179102+0000", 2024-09-18T08:13:38.411 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 9.5210000000000008e-06 2024-09-18T08:13:38.411 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:13:38.411 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:13:38.411 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-18T08:13:38.411 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:13.169945+0000", 2024-09-18T08:13:38.412 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.99084323600000002 2024-09-18T08:13:38.412 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:13:38.412 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:13:38.412 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-09-18T08:13:38.412 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:13.170001+0000", 2024-09-18T08:13:38.412 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 5.5881999999999998e-05 2024-09-18T08:13:38.412 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-18T08:13:38.412 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-09-18T08:13:38.412 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-18T08:13:38.412 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:13:38.412 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:13:38.412 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.15216.0:86263 8.5 8:af593c7a:::benchmark_data_smithi007_101583_object86262:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e205)", 2024-09-18T08:13:38.412 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-09-18T08:09:12.204249+0000", 2024-09-18T08:13:38.412 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 266.19598600099999, 2024-09-18T08:13:38.413 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.98012944999999996, 2024-09-18T08:13:38.413 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-09-18T08:13:38.413 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-09-18T08:13:38.413 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T08:13:38.413 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2024-09-18T08:13:38.413 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.15216", 2024-09-18T08:13:38.413 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "172.21.15.7:0/2596041177", 2024-09-18T08:13:38.413 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 86263 2024-09-18T08:13:38.413 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:13:38.413 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-09-18T08:13:38.413 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:13:38.413 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-09-18T08:13:38.413 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:12.204249+0000", 2024-09-18T08:13:38.414 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-18T08:13:38.414 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:13:38.414 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:13:38.414 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-09-18T08:13:38.414 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:12.204249+0000", 2024-09-18T08:13:38.414 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-18T08:13:38.414 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:13:38.414 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:13:38.414 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-09-18T08:13:38.414 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:12.204251+0000", 2024-09-18T08:13:38.414 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.635e-06 2024-09-18T08:13:38.414 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:13:38.414 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:13:38.415 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-09-18T08:13:38.415 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:12.204299+0000", 2024-09-18T08:13:38.415 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4.8026999999999999e-05 2024-09-18T08:13:38.415 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:13:38.415 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:13:38.415 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-09-18T08:13:38.415 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:12.204313+0000", 2024-09-18T08:13:38.415 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.4066000000000001e-05 2024-09-18T08:13:38.415 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:13:38.415 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:13:38.415 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-09-18T08:13:38.415 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:12.204321+0000", 2024-09-18T08:13:38.416 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 8.1839999999999999e-06 2024-09-18T08:13:38.416 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:13:38.416 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:13:38.416 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-18T08:13:38.416 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:12.204392+0000", 2024-09-18T08:13:38.416 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 7.0340999999999996e-05 2024-09-18T08:13:38.416 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:13:38.416 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:13:38.416 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "waiting for peered", 2024-09-18T08:13:38.416 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:12.204403+0000", 2024-09-18T08:13:38.416 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.1654000000000001e-05 2024-09-18T08:13:38.416 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:13:38.416 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:13:38.417 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-18T08:13:38.417 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:13.170414+0000", 2024-09-18T08:13:38.417 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.96601117000000003 2024-09-18T08:13:38.417 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:13:38.417 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:13:38.417 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-09-18T08:13:38.417 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:13.170546+0000", 2024-09-18T08:13:38.417 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00013120100000000001 2024-09-18T08:13:38.417 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:13:38.417 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:13:38.417 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "waiting for subops from 6,7", 2024-09-18T08:13:38.417 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:13.170965+0000", 2024-09-18T08:13:38.417 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00041976200000000001 2024-09-18T08:13:38.418 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:13:38.418 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:13:38.418 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:13:38.418 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:13.181803+0000", 2024-09-18T08:13:38.418 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.010837167999999999 2024-09-18T08:13:38.418 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:13:38.418 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:13:38.418 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:13:38.418 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:13.181864+0000", 2024-09-18T08:13:38.418 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 6.1857000000000001e-05 2024-09-18T08:13:38.418 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:13:38.418 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:13:38.418 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "op_commit", 2024-09-18T08:13:38.419 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:13.184283+0000", 2024-09-18T08:13:38.419 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0024184340000000001 2024-09-18T08:13:38.419 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:13:38.419 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:13:38.419 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "commit_sent", 2024-09-18T08:13:38.419 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:13.184339+0000", 2024-09-18T08:13:38.419 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 5.6378999999999997e-05 2024-09-18T08:13:38.419 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:13:38.419 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:13:38.419 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-09-18T08:13:38.419 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:13.184379+0000", 2024-09-18T08:13:38.419 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 3.9572e-05 2024-09-18T08:13:38.419 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-18T08:13:38.420 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-09-18T08:13:38.420 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-18T08:13:38.420 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:13:38.420 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:13:38.420 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.15216.0:86267 8.5 8:a135397b:::benchmark_data_smithi007_101583_object86266:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e205)", 2024-09-18T08:13:38.420 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-09-18T08:09:12.207725+0000", 2024-09-18T08:13:38.420 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 266.19250995099998, 2024-09-18T08:13:38.420 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.97676783599999994, 2024-09-18T08:13:38.420 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-09-18T08:13:38.420 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-09-18T08:13:38.420 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T08:13:38.420 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2024-09-18T08:13:38.420 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.15216", 2024-09-18T08:13:38.421 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "172.21.15.7:0/2596041177", 2024-09-18T08:13:38.421 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 86267 2024-09-18T08:13:38.421 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:13:38.421 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-09-18T08:13:38.421 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:13:38.421 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-09-18T08:13:38.421 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:12.207725+0000", 2024-09-18T08:13:38.421 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-18T08:13:38.421 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:13:38.421 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:13:38.421 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-09-18T08:13:38.421 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:12.207725+0000", 2024-09-18T08:13:38.421 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-18T08:13:38.422 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:13:38.422 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:13:38.422 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-09-18T08:13:38.422 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:12.207727+0000", 2024-09-18T08:13:38.422 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.79e-06 2024-09-18T08:13:38.422 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:13:38.422 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:13:38.422 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-09-18T08:13:38.422 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:12.207765+0000", 2024-09-18T08:13:38.422 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 3.7864000000000002e-05 2024-09-18T08:13:38.422 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:13:38.422 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:13:38.423 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-09-18T08:13:38.423 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:12.207783+0000", 2024-09-18T08:13:38.423 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.7779999999999999e-05 2024-09-18T08:13:38.423 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:13:38.423 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:13:38.423 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-09-18T08:13:38.423 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:12.207791+0000", 2024-09-18T08:13:38.423 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 7.9729999999999997e-06 2024-09-18T08:13:38.423 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:13:38.423 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:13:38.423 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-18T08:13:38.423 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:12.207872+0000", 2024-09-18T08:13:38.423 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 8.1318999999999998e-05 2024-09-18T08:13:38.423 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:13:38.424 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:13:38.424 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "waiting for peered", 2024-09-18T08:13:38.424 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:12.207905+0000", 2024-09-18T08:13:38.424 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 3.2694000000000002e-05 2024-09-18T08:13:38.424 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:13:38.424 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:13:38.424 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-18T08:13:38.424 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:13.179332+0000", 2024-09-18T08:13:38.424 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.97142748800000001 2024-09-18T08:13:38.424 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:13:38.424 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:13:38.424 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-09-18T08:13:38.424 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:13.179470+0000", 2024-09-18T08:13:38.424 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00013747400000000001 2024-09-18T08:13:38.425 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:13:38.425 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:13:38.425 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "waiting for subops from 6,7", 2024-09-18T08:13:38.425 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:13.179634+0000", 2024-09-18T08:13:38.425 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00016417600000000001 2024-09-18T08:13:38.425 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:13:38.425 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:13:38.425 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:13:38.425 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:13.184267+0000", 2024-09-18T08:13:38.425 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0046332079999999998 2024-09-18T08:13:38.425 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:13:38.425 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:13:38.425 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "op_commit", 2024-09-18T08:13:38.426 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:13.184386+0000", 2024-09-18T08:13:38.426 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000118993 2024-09-18T08:13:38.426 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:13:38.426 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:13:38.426 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:13:38.426 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:13.184437+0000", 2024-09-18T08:13:38.426 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 5.0988e-05 2024-09-18T08:13:38.426 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:13:38.426 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:13:38.426 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "commit_sent", 2024-09-18T08:13:38.426 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:13.184469+0000", 2024-09-18T08:13:38.426 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 3.1603000000000002e-05 2024-09-18T08:13:38.426 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:13:38.426 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:13:38.427 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-09-18T08:13:38.427 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:13.184493+0000", 2024-09-18T08:13:38.427 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 2.4485999999999999e-05 2024-09-18T08:13:38.427 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-18T08:13:38.427 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-09-18T08:13:38.427 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-18T08:13:38.427 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:13:38.427 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:13:38.427 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.15216.0:86268 8.5 8:aa7096fb:::benchmark_data_smithi007_101583_object86267:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e205)", 2024-09-18T08:13:38.427 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-09-18T08:09:12.208946+0000", 2024-09-18T08:13:38.427 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 266.19128934399998, 2024-09-18T08:13:38.427 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.980827699, 2024-09-18T08:13:38.427 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-09-18T08:13:38.428 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-09-18T08:13:38.428 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T08:13:38.428 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2024-09-18T08:13:38.428 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.15216", 2024-09-18T08:13:38.428 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "172.21.15.7:0/2596041177", 2024-09-18T08:13:38.428 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 86268 2024-09-18T08:13:38.428 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:13:38.428 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-09-18T08:13:38.428 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:13:38.428 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-09-18T08:13:38.428 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:12.208946+0000", 2024-09-18T08:13:38.428 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-18T08:13:38.428 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:13:38.428 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:13:38.429 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-09-18T08:13:38.429 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:12.208946+0000", 2024-09-18T08:13:38.429 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-18T08:13:38.429 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:13:38.429 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:13:38.429 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-09-18T08:13:38.429 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:12.208947+0000", 2024-09-18T08:13:38.429 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.201e-06 2024-09-18T08:13:38.429 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:13:38.429 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:13:38.429 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-09-18T08:13:38.429 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:12.209036+0000", 2024-09-18T08:13:38.429 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 8.8865999999999999e-05 2024-09-18T08:13:38.429 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:13:38.430 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:13:38.430 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-09-18T08:13:38.430 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:12.209049+0000", 2024-09-18T08:13:38.430 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.2795e-05 2024-09-18T08:13:38.430 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:13:38.430 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:13:38.430 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-09-18T08:13:38.430 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:12.209056+0000", 2024-09-18T08:13:38.430 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 7.1160000000000001e-06 2024-09-18T08:13:38.430 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:13:38.430 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:13:38.430 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-18T08:13:38.430 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:12.209116+0000", 2024-09-18T08:13:38.431 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 5.9517e-05 2024-09-18T08:13:38.431 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:13:38.431 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:13:38.431 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "waiting for peered", 2024-09-18T08:13:38.431 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:12.209125+0000", 2024-09-18T08:13:38.431 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 9.6630000000000005e-06 2024-09-18T08:13:38.431 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:13:38.431 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:13:38.431 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-18T08:13:38.431 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:13.182004+0000", 2024-09-18T08:13:38.431 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.97287885799999996 2024-09-18T08:13:38.431 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:13:38.431 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:13:38.432 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-09-18T08:13:38.432 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:13.182091+0000", 2024-09-18T08:13:38.432 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 8.6907e-05 2024-09-18T08:13:38.432 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:13:38.432 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:13:38.432 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "waiting for subops from 6,7", 2024-09-18T08:13:38.432 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:13.182183+0000", 2024-09-18T08:13:38.432 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 9.2131000000000003e-05 2024-09-18T08:13:38.432 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:13:38.432 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:13:38.432 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "op_commit", 2024-09-18T08:13:38.432 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:13.187041+0000", 2024-09-18T08:13:38.433 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0048579210000000003 2024-09-18T08:13:38.433 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:13:38.433 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:13:38.433 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:13:38.433 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:13.189581+0000", 2024-09-18T08:13:38.433 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.002539707 2024-09-18T08:13:38.433 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:13:38.433 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:13:38.433 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:13:38.433 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:13.189646+0000", 2024-09-18T08:13:38.433 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 6.5699999999999998e-05 2024-09-18T08:13:38.433 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:13:38.433 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:13:38.434 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "commit_sent", 2024-09-18T08:13:38.434 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:13.189714+0000", 2024-09-18T08:13:38.434 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 6.7108999999999997e-05 2024-09-18T08:13:38.434 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:13:38.434 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:13:38.434 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-09-18T08:13:38.434 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:13.189774+0000", 2024-09-18T08:13:38.434 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 6.0207999999999997e-05 2024-09-18T08:13:38.434 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-18T08:13:38.434 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-09-18T08:13:38.434 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-18T08:13:38.434 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:13:38.434 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:13:38.435 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.15216.0:86303 8.5 8:aebc4459:::benchmark_data_smithi007_101583_object86302:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e205)", 2024-09-18T08:13:38.435 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-09-18T08:09:12.250052+0000", 2024-09-18T08:13:38.435 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 266.15018378799999, 2024-09-18T08:13:38.435 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.94992247699999999, 2024-09-18T08:13:38.435 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-09-18T08:13:38.435 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-09-18T08:13:38.435 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T08:13:38.435 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2024-09-18T08:13:38.436 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.15216", 2024-09-18T08:13:38.436 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "172.21.15.7:0/2596041177", 2024-09-18T08:13:38.436 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 86303 2024-09-18T08:13:38.437 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:13:38.437 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-09-18T08:13:38.437 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:13:38.437 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-09-18T08:13:38.437 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:12.250052+0000", 2024-09-18T08:13:38.437 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-18T08:13:38.437 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:13:38.437 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:13:38.438 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-09-18T08:13:38.438 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:12.250052+0000", 2024-09-18T08:13:38.438 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-18T08:13:38.438 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:13:38.438 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:13:38.438 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-09-18T08:13:38.438 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:12.250053+0000", 2024-09-18T08:13:38.438 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.6130000000000001e-06 2024-09-18T08:13:38.439 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:13:38.439 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:13:38.439 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-09-18T08:13:38.439 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:12.250090+0000", 2024-09-18T08:13:38.439 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 3.6307999999999999e-05 2024-09-18T08:13:38.439 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:13:38.439 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:13:38.439 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-09-18T08:13:38.439 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:12.250102+0000", 2024-09-18T08:13:38.439 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.2624e-05 2024-09-18T08:13:38.439 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:13:38.440 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:13:38.440 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-09-18T08:13:38.440 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:12.250109+0000", 2024-09-18T08:13:38.440 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 6.426e-06 2024-09-18T08:13:38.440 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:13:38.440 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:13:38.440 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-18T08:13:38.440 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:12.250173+0000", 2024-09-18T08:13:38.440 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 6.3966000000000004e-05 2024-09-18T08:13:38.440 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:13:38.440 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:13:38.440 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "waiting for peered", 2024-09-18T08:13:38.440 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:12.250184+0000", 2024-09-18T08:13:38.441 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.1401e-05 2024-09-18T08:13:38.441 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:13:38.441 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:13:38.441 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-18T08:13:38.441 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:13.184533+0000", 2024-09-18T08:13:38.441 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.93434866299999997 2024-09-18T08:13:38.441 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:13:38.441 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:13:38.441 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-09-18T08:13:38.441 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:13.184634+0000", 2024-09-18T08:13:38.441 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000101139 2024-09-18T08:13:38.441 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:13:38.441 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:13:38.442 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "waiting for subops from 6,7", 2024-09-18T08:13:38.442 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:13.184793+0000", 2024-09-18T08:13:38.442 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00015932000000000001 2024-09-18T08:13:38.442 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:13:38.442 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:13:38.442 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:13:38.442 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:13.194839+0000", 2024-09-18T08:13:38.442 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.010046144 2024-09-18T08:13:38.442 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:13:38.442 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:13:38.442 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:13:38.442 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:13.194928+0000", 2024-09-18T08:13:38.442 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 8.9037999999999997e-05 2024-09-18T08:13:38.443 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:13:38.443 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:13:38.443 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "op_commit", 2024-09-18T08:13:38.443 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:13.199865+0000", 2024-09-18T08:13:38.443 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0049365879999999996 2024-09-18T08:13:38.443 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:13:38.443 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:13:38.443 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "commit_sent", 2024-09-18T08:13:38.443 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:13.199924+0000", 2024-09-18T08:13:38.443 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 5.9383999999999999e-05 2024-09-18T08:13:38.443 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:13:38.443 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:13:38.443 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-09-18T08:13:38.444 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:13.199974+0000", 2024-09-18T08:13:38.444 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4.9863e-05 2024-09-18T08:13:38.444 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-18T08:13:38.444 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-09-18T08:13:38.444 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-18T08:13:38.444 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:13:38.444 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:13:38.444 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.15216.0:86344 8.5 8:ac157b17:::benchmark_data_smithi007_101583_object86343:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e205)", 2024-09-18T08:13:38.444 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-09-18T08:09:12.295743+0000", 2024-09-18T08:13:38.444 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 266.10449210799999, 2024-09-18T08:13:38.444 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.92374953400000004, 2024-09-18T08:13:38.444 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-09-18T08:13:38.445 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-09-18T08:13:38.445 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T08:13:38.445 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2024-09-18T08:13:38.445 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.15216", 2024-09-18T08:13:38.445 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "172.21.15.7:0/2596041177", 2024-09-18T08:13:38.445 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 86344 2024-09-18T08:13:38.445 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:13:38.445 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-09-18T08:13:38.445 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:13:38.445 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-09-18T08:13:38.445 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:12.295743+0000", 2024-09-18T08:13:38.445 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-18T08:13:38.445 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:13:38.445 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:13:38.446 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-09-18T08:13:38.446 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:12.295743+0000", 2024-09-18T08:13:38.446 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-18T08:13:38.446 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:13:38.446 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:13:38.446 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-09-18T08:13:38.446 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:12.295745+0000", 2024-09-18T08:13:38.446 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.57e-06 2024-09-18T08:13:38.446 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:13:38.446 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:13:38.446 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-09-18T08:13:38.446 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:12.295795+0000", 2024-09-18T08:13:38.446 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 5.0062999999999998e-05 2024-09-18T08:13:38.446 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:13:38.446 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:13:38.447 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-09-18T08:13:38.447 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:12.295807+0000", 2024-09-18T08:13:38.447 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.1578e-05 2024-09-18T08:13:38.447 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:13:38.447 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:13:38.447 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-09-18T08:13:38.447 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:12.295812+0000", 2024-09-18T08:13:38.447 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 5.8930000000000002e-06 2024-09-18T08:13:38.447 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:13:38.447 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:13:38.447 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-18T08:13:38.447 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:12.295875+0000", 2024-09-18T08:13:38.447 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 6.2168000000000005e-05 2024-09-18T08:13:38.447 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:13:38.448 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:13:38.448 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "waiting for peered", 2024-09-18T08:13:38.448 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:12.295883+0000", 2024-09-18T08:13:38.448 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 7.9969999999999998e-06 2024-09-18T08:13:38.448 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:13:38.448 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:13:38.448 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-18T08:13:38.448 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:13.187072+0000", 2024-09-18T08:13:38.448 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.89118984899999998 2024-09-18T08:13:38.448 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:13:38.448 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:13:38.448 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-09-18T08:13:38.448 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:13.187193+0000", 2024-09-18T08:13:38.449 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00012062599999999999 2024-09-18T08:13:38.449 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:13:38.449 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:13:38.449 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "waiting for subops from 6,7", 2024-09-18T08:13:38.449 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:13.187351+0000", 2024-09-18T08:13:38.449 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00015781500000000001 2024-09-18T08:13:38.449 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:13:38.449 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:13:38.449 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:13:38.449 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:13.194877+0000", 2024-09-18T08:13:38.449 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0075262139999999998 2024-09-18T08:13:38.449 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:13:38.449 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:13:38.450 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:13:38.450 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:13.195031+0000", 2024-09-18T08:13:38.450 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00015384799999999999 2024-09-18T08:13:38.450 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:13:38.450 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:13:38.450 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "op_commit", 2024-09-18T08:13:38.450 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:13.219386+0000", 2024-09-18T08:13:38.450 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.024355226000000001 2024-09-18T08:13:38.450 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:13:38.450 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:13:38.450 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "commit_sent", 2024-09-18T08:13:38.450 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:13.219440+0000", 2024-09-18T08:13:38.450 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 5.3786999999999998e-05 2024-09-18T08:13:38.450 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:13:38.451 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:13:38.451 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-09-18T08:13:38.451 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:13.219493+0000", 2024-09-18T08:13:38.451 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 5.2899999999999998e-05 2024-09-18T08:13:38.451 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-18T08:13:38.451 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-09-18T08:13:38.451 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-18T08:13:38.451 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:13:38.451 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:13:38.451 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.15216.0:86350 8.5 8:a14e8599:::benchmark_data_smithi007_101583_object86349:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e205)", 2024-09-18T08:13:38.451 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-09-18T08:09:12.297534+0000", 2024-09-18T08:13:38.451 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 266.10270162400002, 2024-09-18T08:13:38.451 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.922055505, 2024-09-18T08:13:38.451 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-09-18T08:13:38.452 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-09-18T08:13:38.452 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T08:13:38.452 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2024-09-18T08:13:38.452 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.15216", 2024-09-18T08:13:38.452 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "172.21.15.7:0/2596041177", 2024-09-18T08:13:38.452 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 86350 2024-09-18T08:13:38.452 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:13:38.452 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-09-18T08:13:38.452 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:13:38.452 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-09-18T08:13:38.452 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:12.297534+0000", 2024-09-18T08:13:38.452 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-18T08:13:38.452 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:13:38.452 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:13:38.453 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-09-18T08:13:38.453 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:12.297534+0000", 2024-09-18T08:13:38.453 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-18T08:13:38.453 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:13:38.453 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:13:38.453 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-09-18T08:13:38.453 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:12.297534+0000", 2024-09-18T08:13:38.453 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 7.0800000000000004e-07 2024-09-18T08:13:38.453 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:13:38.453 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:13:38.453 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-09-18T08:13:38.453 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:12.297606+0000", 2024-09-18T08:13:38.453 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 7.1075000000000006e-05 2024-09-18T08:13:38.454 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:13:38.454 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:13:38.454 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-09-18T08:13:38.454 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:12.297613+0000", 2024-09-18T08:13:38.454 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 7.6260000000000003e-06 2024-09-18T08:13:38.454 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:13:38.454 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:13:38.454 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-09-18T08:13:38.454 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:12.297618+0000", 2024-09-18T08:13:38.454 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4.8550000000000001e-06 2024-09-18T08:13:38.454 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:13:38.454 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:13:38.454 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-18T08:13:38.455 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:12.297660+0000", 2024-09-18T08:13:38.455 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4.1968000000000002e-05 2024-09-18T08:13:38.455 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:13:38.455 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:13:38.455 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "waiting for peered", 2024-09-18T08:13:38.455 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:12.297666+0000", 2024-09-18T08:13:38.455 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 5.806e-06 2024-09-18T08:13:38.455 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:13:38.455 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:13:38.455 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-18T08:13:38.455 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:13.189816+0000", 2024-09-18T08:13:38.455 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.89215055300000001 2024-09-18T08:13:38.455 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:13:38.455 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:13:38.456 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-09-18T08:13:38.456 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:13.190014+0000", 2024-09-18T08:13:38.456 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000197894 2024-09-18T08:13:38.456 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:13:38.456 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:13:38.456 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "waiting for subops from 6,7", 2024-09-18T08:13:38.456 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:13.190197+0000", 2024-09-18T08:13:38.456 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00018230399999999999 2024-09-18T08:13:38.456 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:13:38.456 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:13:38.456 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:13:38.456 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:13.194999+0000", 2024-09-18T08:13:38.456 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0048023019999999996 2024-09-18T08:13:38.456 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:13:38.457 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:13:38.457 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:13:38.457 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:13.197606+0000", 2024-09-18T08:13:38.457 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0026074890000000002 2024-09-18T08:13:38.457 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:13:38.457 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:13:38.457 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "op_commit", 2024-09-18T08:13:38.457 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:13.219502+0000", 2024-09-18T08:13:38.457 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.02189514 2024-09-18T08:13:38.457 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:13:38.457 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:13:38.457 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "commit_sent", 2024-09-18T08:13:38.457 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:13.219546+0000", 2024-09-18T08:13:38.458 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4.4574999999999999e-05 2024-09-18T08:13:38.458 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:13:38.458 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:13:38.458 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-09-18T08:13:38.458 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:13.219589+0000", 2024-09-18T08:13:38.458 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4.3210000000000001e-05 2024-09-18T08:13:38.458 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-18T08:13:38.458 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-09-18T08:13:38.458 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-18T08:13:38.458 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:13:38.458 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:13:38.458 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.15216.0:86366 8.5 8:a8c4536d:::benchmark_data_smithi007_101583_object86365:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e205)", 2024-09-18T08:13:38.458 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-09-18T08:09:12.318202+0000", 2024-09-18T08:13:38.459 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 266.08203294800001, 2024-09-18T08:13:38.459 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.90148185700000005, 2024-09-18T08:13:38.459 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-09-18T08:13:38.459 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-09-18T08:13:38.459 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T08:13:38.459 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2024-09-18T08:13:38.459 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.15216", 2024-09-18T08:13:38.459 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "172.21.15.7:0/2596041177", 2024-09-18T08:13:38.459 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 86366 2024-09-18T08:13:38.459 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:13:38.459 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-09-18T08:13:38.459 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:13:38.459 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-09-18T08:13:38.459 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:12.318202+0000", 2024-09-18T08:13:38.460 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-18T08:13:38.460 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:13:38.460 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:13:38.460 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-09-18T08:13:38.460 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:12.318202+0000", 2024-09-18T08:13:38.460 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-18T08:13:38.460 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:13:38.460 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:13:38.460 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-09-18T08:13:38.460 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:12.318205+0000", 2024-09-18T08:13:38.460 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 2.074e-06 2024-09-18T08:13:38.460 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:13:38.460 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:13:38.461 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-09-18T08:13:38.461 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:12.318242+0000", 2024-09-18T08:13:38.461 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 3.6996999999999998e-05 2024-09-18T08:13:38.461 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:13:38.461 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:13:38.461 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-09-18T08:13:38.461 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:12.318258+0000", 2024-09-18T08:13:38.461 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.6382e-05 2024-09-18T08:13:38.461 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:13:38.461 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:13:38.461 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-09-18T08:13:38.461 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:12.318266+0000", 2024-09-18T08:13:38.462 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 7.9820000000000008e-06 2024-09-18T08:13:38.462 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:13:38.462 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:13:38.462 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-18T08:13:38.462 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:12.318402+0000", 2024-09-18T08:13:38.462 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000136477 2024-09-18T08:13:38.462 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:13:38.462 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:13:38.462 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "waiting for peered", 2024-09-18T08:13:38.462 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:12.318414+0000", 2024-09-18T08:13:38.462 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.2096000000000001e-05 2024-09-18T08:13:38.462 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:13:38.462 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:13:38.463 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-18T08:13:38.463 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:13.192024+0000", 2024-09-18T08:13:38.463 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.87360952300000005 2024-09-18T08:13:38.463 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:13:38.463 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:13:38.463 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-09-18T08:13:38.463 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:13.192155+0000", 2024-09-18T08:13:38.463 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00013099499999999999 2024-09-18T08:13:38.463 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:13:38.463 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:13:38.463 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "waiting for subops from 6,7", 2024-09-18T08:13:38.463 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:13.192313+0000", 2024-09-18T08:13:38.463 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00015802500000000001 2024-09-18T08:13:38.464 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:13:38.464 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:13:38.464 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:13:38.464 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:13.195063+0000", 2024-09-18T08:13:38.464 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0027500279999999998 2024-09-18T08:13:38.464 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:13:38.464 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:13:38.464 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:13:38.464 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:13.200043+0000", 2024-09-18T08:13:38.464 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0049800929999999997 2024-09-18T08:13:38.464 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:13:38.464 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:13:38.464 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "op_commit", 2024-09-18T08:13:38.464 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:13.219598+0000", 2024-09-18T08:13:38.465 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.019554768 2024-09-18T08:13:38.465 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:13:38.465 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:13:38.465 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "commit_sent", 2024-09-18T08:13:38.465 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:13.219647+0000", 2024-09-18T08:13:38.465 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4.9327999999999999e-05 2024-09-18T08:13:38.465 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:13:38.465 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:13:38.465 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-09-18T08:13:38.465 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:13.219684+0000", 2024-09-18T08:13:38.465 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 3.7088999999999997e-05 2024-09-18T08:13:38.465 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-18T08:13:38.465 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-09-18T08:13:38.465 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-18T08:13:38.466 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:13:38.466 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:13:38.466 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.15216.0:86390 8.5 8:a8c7b3fd:::benchmark_data_smithi007_101583_object86389:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e205)", 2024-09-18T08:13:38.466 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-09-18T08:09:12.356640+0000", 2024-09-18T08:13:38.466 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 266.04359564700002, 2024-09-18T08:13:38.466 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.86366743700000004, 2024-09-18T08:13:38.466 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-09-18T08:13:38.466 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-09-18T08:13:38.466 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T08:13:38.466 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2024-09-18T08:13:38.466 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.15216", 2024-09-18T08:13:38.466 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "172.21.15.7:0/2596041177", 2024-09-18T08:13:38.466 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 86390 2024-09-18T08:13:38.466 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:13:38.467 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-09-18T08:13:38.467 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:13:38.467 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-09-18T08:13:38.467 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:12.356640+0000", 2024-09-18T08:13:38.467 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-18T08:13:38.467 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:13:38.467 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:13:38.467 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-09-18T08:13:38.467 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:12.356640+0000", 2024-09-18T08:13:38.467 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-18T08:13:38.467 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:13:38.467 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:13:38.467 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-09-18T08:13:38.467 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:12.356641+0000", 2024-09-18T08:13:38.468 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.5430000000000001e-06 2024-09-18T08:13:38.468 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:13:38.468 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:13:38.468 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-09-18T08:13:38.468 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:12.356684+0000", 2024-09-18T08:13:38.468 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4.2222999999999999e-05 2024-09-18T08:13:38.468 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:13:38.468 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:13:38.468 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-09-18T08:13:38.468 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:12.356696+0000", 2024-09-18T08:13:38.468 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.2663000000000001e-05 2024-09-18T08:13:38.468 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:13:38.468 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:13:38.469 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-09-18T08:13:38.469 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:12.356703+0000", 2024-09-18T08:13:38.469 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 6.6710000000000002e-06 2024-09-18T08:13:38.469 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:13:38.469 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:13:38.469 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-18T08:13:38.469 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:12.356759+0000", 2024-09-18T08:13:38.469 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 5.6382000000000003e-05 2024-09-18T08:13:38.469 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:13:38.469 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:13:38.469 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "waiting for peered", 2024-09-18T08:13:38.469 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:12.356767+0000", 2024-09-18T08:13:38.469 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 7.5190000000000003e-06 2024-09-18T08:13:38.470 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:13:38.470 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:13:38.470 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-18T08:13:38.470 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:13.195090+0000", 2024-09-18T08:13:38.470 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.83832288600000004 2024-09-18T08:13:38.470 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:13:38.470 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:13:38.470 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-09-18T08:13:38.470 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:13.195221+0000", 2024-09-18T08:13:38.470 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00013146699999999999 2024-09-18T08:13:38.470 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:13:38.470 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:13:38.470 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "waiting for subops from 6,7", 2024-09-18T08:13:38.471 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:13.195370+0000", 2024-09-18T08:13:38.471 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000148598 2024-09-18T08:13:38.471 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:13:38.471 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:13:38.471 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:13:38.471 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:13.199845+0000", 2024-09-18T08:13:38.471 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0044749999999999998 2024-09-18T08:13:38.471 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:13:38.471 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:13:38.471 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:13:38.471 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:13.207294+0000", 2024-09-18T08:13:38.471 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0074489309999999998 2024-09-18T08:13:38.471 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:13:38.471 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:13:38.472 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "op_commit", 2024-09-18T08:13:38.472 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:13.220218+0000", 2024-09-18T08:13:38.472 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.012924715 2024-09-18T08:13:38.472 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:13:38.472 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:13:38.472 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "commit_sent", 2024-09-18T08:13:38.472 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:13.220272+0000", 2024-09-18T08:13:38.472 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 5.3748000000000001e-05 2024-09-18T08:13:38.472 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:13:38.472 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:13:38.472 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-09-18T08:13:38.472 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:13.220307+0000", 2024-09-18T08:13:38.472 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 3.5091e-05 2024-09-18T08:13:38.473 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-18T08:13:38.473 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-09-18T08:13:38.473 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-18T08:13:38.473 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:13:38.473 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:13:38.473 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.15216.0:86394 8.5 8:a810cd9d:::benchmark_data_smithi007_101583_object86393:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e205)", 2024-09-18T08:13:38.473 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-09-18T08:09:12.360967+0000", 2024-09-18T08:13:38.473 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 266.03926806999999, 2024-09-18T08:13:38.473 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.85950780100000002, 2024-09-18T08:13:38.473 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-09-18T08:13:38.473 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-09-18T08:13:38.473 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T08:13:38.473 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2024-09-18T08:13:38.473 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.15216", 2024-09-18T08:13:38.474 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "172.21.15.7:0/2596041177", 2024-09-18T08:13:38.474 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 86394 2024-09-18T08:13:38.474 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:13:38.474 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-09-18T08:13:38.474 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:13:38.474 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-09-18T08:13:38.474 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:12.360967+0000", 2024-09-18T08:13:38.474 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-18T08:13:38.474 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:13:38.474 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:13:38.474 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-09-18T08:13:38.474 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:12.360967+0000", 2024-09-18T08:13:38.474 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-18T08:13:38.475 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:13:38.475 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:13:38.475 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-09-18T08:13:38.475 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:12.360969+0000", 2024-09-18T08:13:38.475 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.8500000000000001e-06 2024-09-18T08:13:38.475 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:13:38.475 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:13:38.475 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-09-18T08:13:38.475 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:12.361013+0000", 2024-09-18T08:13:38.475 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4.3693000000000003e-05 2024-09-18T08:13:38.475 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:13:38.475 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:13:38.475 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-09-18T08:13:38.476 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:12.361031+0000", 2024-09-18T08:13:38.476 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.8340000000000001e-05 2024-09-18T08:13:38.476 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:13:38.476 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:13:38.476 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-09-18T08:13:38.476 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:12.361040+0000", 2024-09-18T08:13:38.476 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 8.5320000000000006e-06 2024-09-18T08:13:38.476 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:13:38.476 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:13:38.476 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-18T08:13:38.476 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:12.361108+0000", 2024-09-18T08:13:38.476 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 6.7879000000000005e-05 2024-09-18T08:13:38.477 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:13:38.477 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:13:38.477 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "waiting for peered", 2024-09-18T08:13:38.477 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:12.361119+0000", 2024-09-18T08:13:38.477 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.1163e-05 2024-09-18T08:13:38.477 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:13:38.477 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:13:38.477 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-18T08:13:38.477 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:13.197641+0000", 2024-09-18T08:13:38.477 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.83652230299999997 2024-09-18T08:13:38.477 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:13:38.477 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:13:38.477 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-09-18T08:13:38.477 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:13.197776+0000", 2024-09-18T08:13:38.478 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00013499000000000001 2024-09-18T08:13:38.478 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:13:38.478 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:13:38.478 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "waiting for subops from 6,7", 2024-09-18T08:13:38.478 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:13.197954+0000", 2024-09-18T08:13:38.478 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00017745200000000001 2024-09-18T08:13:38.478 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:13:38.478 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:13:38.478 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:13:38.478 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:13.207329+0000", 2024-09-18T08:13:38.478 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0093754460000000008 2024-09-18T08:13:38.478 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:13:38.478 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:13:38.478 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:13:38.479 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:13.207363+0000", 2024-09-18T08:13:38.479 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 3.3535999999999999e-05 2024-09-18T08:13:38.479 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:13:38.479 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:13:38.479 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "op_commit", 2024-09-18T08:13:38.479 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:13.220323+0000", 2024-09-18T08:13:38.479 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.012960121999999999 2024-09-18T08:13:38.479 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:13:38.479 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:13:38.479 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "commit_sent", 2024-09-18T08:13:38.479 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:13.220445+0000", 2024-09-18T08:13:38.479 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00012217099999999999 2024-09-18T08:13:38.479 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:13:38.479 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:13:38.480 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-09-18T08:13:38.480 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:13.220475+0000", 2024-09-18T08:13:38.480 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0 2024-09-18T08:13:38.481 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 3.0324e-05 2024-09-18T08:13:38.481 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-18T08:13:38.481 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-09-18T08:13:38.481 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-18T08:13:38.481 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:13:38.481 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:13:38.481 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.15216.0:86399 8.5 8:a535189e:::benchmark_data_smithi007_101583_object86398:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e205)", 2024-09-18T08:13:38.482 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-09-18T08:09:12.364040+0000", 2024-09-18T08:13:38.482 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 266.03619500600001, 2024-09-18T08:13:38.482 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.85654366599999998, 2024-09-18T08:13:38.482 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-09-18T08:13:38.482 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-09-18T08:13:38.482 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T08:13:38.482 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2024-09-18T08:13:38.482 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.15216", 2024-09-18T08:13:38.482 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "172.21.15.7:0/2596041177", 2024-09-18T08:13:38.482 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 86399 2024-09-18T08:13:38.482 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:13:38.482 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-09-18T08:13:38.482 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:13:38.483 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-09-18T08:13:38.483 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:12.364040+0000", 2024-09-18T08:13:38.483 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-18T08:13:38.483 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:13:38.483 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:13:38.483 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-09-18T08:13:38.483 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:12.364040+0000", 2024-09-18T08:13:38.483 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-18T08:13:38.483 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:13:38.483 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:13:38.483 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-09-18T08:13:38.483 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:12.364042+0000", 2024-09-18T08:13:38.484 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.407e-06 2024-09-18T08:13:38.484 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:13:38.484 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:13:38.484 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-09-18T08:13:38.484 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:12.364174+0000", 2024-09-18T08:13:38.484 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000132545 2024-09-18T08:13:38.484 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:13:38.484 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:13:38.484 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-09-18T08:13:38.484 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:12.364186+0000", 2024-09-18T08:13:38.484 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.1912e-05 2024-09-18T08:13:38.484 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:13:38.485 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:13:38.485 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-09-18T08:13:38.485 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:12.364192+0000", 2024-09-18T08:13:38.485 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 5.4600000000000002e-06 2024-09-18T08:13:38.485 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:13:38.485 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:13:38.485 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-18T08:13:38.485 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:12.364257+0000", 2024-09-18T08:13:38.485 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 6.5606999999999997e-05 2024-09-18T08:13:38.485 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:13:38.485 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:13:38.485 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "waiting for peered", 2024-09-18T08:13:38.485 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:12.364271+0000", 2024-09-18T08:13:38.486 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.3433e-05 2024-09-18T08:13:38.486 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:13:38.486 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:13:38.486 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-18T08:13:38.486 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:13.200082+0000", 2024-09-18T08:13:38.486 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.83581135500000003 2024-09-18T08:13:38.486 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:13:38.486 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:13:38.486 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-09-18T08:13:38.486 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:13.200201+0000", 2024-09-18T08:13:38.486 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00011913099999999999 2024-09-18T08:13:38.486 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:13:38.486 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:13:38.487 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "waiting for subops from 6,7", 2024-09-18T08:13:38.487 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:13.200360+0000", 2024-09-18T08:13:38.487 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000159173 2024-09-18T08:13:38.487 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:13:38.487 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:13:38.487 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:13:38.487 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:13.207395+0000", 2024-09-18T08:13:38.487 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0070341090000000002 2024-09-18T08:13:38.487 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:13:38.487 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:13:38.487 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:13:38.487 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:13.207427+0000", 2024-09-18T08:13:38.487 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 3.2138000000000003e-05 2024-09-18T08:13:38.488 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:13:38.488 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:13:38.488 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "op_commit", 2024-09-18T08:13:38.488 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:13.220524+0000", 2024-09-18T08:13:38.488 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.013097698 2024-09-18T08:13:38.488 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:13:38.488 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:13:38.488 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "commit_sent", 2024-09-18T08:13:38.488 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:13.220557+0000", 2024-09-18T08:13:38.488 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 3.2682999999999999e-05 2024-09-18T08:13:38.488 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:13:38.488 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:13:38.489 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-09-18T08:13:38.489 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:13.220584+0000", 2024-09-18T08:13:38.489 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 2.7015000000000001e-05 2024-09-18T08:13:38.489 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-18T08:13:38.489 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-09-18T08:13:38.489 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-18T08:13:38.489 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:13:38.489 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:13:38.489 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.15216.0:86429 8.5 8:a7f1f687:::benchmark_data_smithi007_101583_object86428:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e205)", 2024-09-18T08:13:38.489 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-09-18T08:09:12.416915+0000", 2024-09-18T08:13:38.489 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 265.98332023699999, 2024-09-18T08:13:38.489 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.80373175900000005, 2024-09-18T08:13:38.490 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-09-18T08:13:38.490 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-09-18T08:13:38.490 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T08:13:38.490 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2024-09-18T08:13:38.490 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.15216", 2024-09-18T08:13:38.490 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "172.21.15.7:0/2596041177", 2024-09-18T08:13:38.490 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 86429 2024-09-18T08:13:38.490 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:13:38.490 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-09-18T08:13:38.490 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:13:38.490 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-09-18T08:13:38.490 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:12.416915+0000", 2024-09-18T08:13:38.490 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-18T08:13:38.491 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:13:38.491 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:13:38.491 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-09-18T08:13:38.491 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:12.416915+0000", 2024-09-18T08:13:38.491 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-18T08:13:38.491 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:13:38.491 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:13:38.491 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-09-18T08:13:38.491 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:12.416917+0000", 2024-09-18T08:13:38.491 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.7150000000000001e-06 2024-09-18T08:13:38.491 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:13:38.491 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:13:38.491 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-09-18T08:13:38.491 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:12.416952+0000", 2024-09-18T08:13:38.492 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 3.4860000000000002e-05 2024-09-18T08:13:38.492 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:13:38.492 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:13:38.492 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-09-18T08:13:38.492 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:12.416966+0000", 2024-09-18T08:13:38.492 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.3884e-05 2024-09-18T08:13:38.492 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:13:38.492 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:13:38.492 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-09-18T08:13:38.492 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:12.416973+0000", 2024-09-18T08:13:38.492 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 7.5809999999999999e-06 2024-09-18T08:13:38.492 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:13:38.492 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:13:38.493 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-18T08:13:38.493 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:12.417031+0000", 2024-09-18T08:13:38.493 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 5.7544999999999999e-05 2024-09-18T08:13:38.493 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:13:38.493 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:13:38.493 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "waiting for peered", 2024-09-18T08:13:38.493 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:12.417039+0000", 2024-09-18T08:13:38.493 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 8.5369999999999997e-06 2024-09-18T08:13:38.493 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:13:38.493 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:13:38.493 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-18T08:13:38.493 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:13.202704+0000", 2024-09-18T08:13:38.493 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.78566510099999998 2024-09-18T08:13:38.494 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:13:38.494 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:13:38.494 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-09-18T08:13:38.494 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:13.202825+0000", 2024-09-18T08:13:38.494 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00012023600000000001 2024-09-18T08:13:38.494 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:13:38.494 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:13:38.494 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "waiting for subops from 6,7", 2024-09-18T08:13:38.494 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:13.202989+0000", 2024-09-18T08:13:38.494 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00016451000000000001 2024-09-18T08:13:38.494 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:13:38.494 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:13:38.494 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:13:38.494 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:13.207464+0000", 2024-09-18T08:13:38.495 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0044748909999999999 2024-09-18T08:13:38.495 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:13:38.495 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:13:38.495 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:13:38.495 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:13.214034+0000", 2024-09-18T08:13:38.495 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0065698229999999998 2024-09-18T08:13:38.495 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:13:38.495 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:13:38.495 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "op_commit", 2024-09-18T08:13:38.495 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:13.220590+0000", 2024-09-18T08:13:38.495 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0065556909999999998 2024-09-18T08:13:38.495 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:13:38.495 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:13:38.496 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "commit_sent", 2024-09-18T08:13:38.496 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:13.220620+0000", 2024-09-18T08:13:38.496 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 3.0429999999999998e-05 2024-09-18T08:13:38.496 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:13:38.496 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:13:38.496 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-09-18T08:13:38.496 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:13.220647+0000", 2024-09-18T08:13:38.496 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 2.6954999999999999e-05 2024-09-18T08:13:38.496 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-18T08:13:38.496 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-09-18T08:13:38.496 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-18T08:13:38.496 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:13:38.496 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:13:38.497 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.15216.0:86446 8.5 8:a0654d9b:::benchmark_data_smithi007_101583_object86445:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e205)", 2024-09-18T08:13:38.497 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-09-18T08:09:12.450214+0000", 2024-09-18T08:13:38.497 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 265.95002113800001, 2024-09-18T08:13:38.497 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.77050818399999998, 2024-09-18T08:13:38.497 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-09-18T08:13:38.497 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-09-18T08:13:38.497 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T08:13:38.497 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2024-09-18T08:13:38.497 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.15216", 2024-09-18T08:13:38.497 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "172.21.15.7:0/2596041177", 2024-09-18T08:13:38.497 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 86446 2024-09-18T08:13:38.498 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:13:38.498 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-09-18T08:13:38.498 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:13:38.498 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-09-18T08:13:38.498 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:12.450214+0000", 2024-09-18T08:13:38.498 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-18T08:13:38.498 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:13:38.498 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:13:38.498 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-09-18T08:13:38.498 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:12.450214+0000", 2024-09-18T08:13:38.498 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-18T08:13:38.498 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:13:38.498 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:13:38.499 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-09-18T08:13:38.499 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:12.450216+0000", 2024-09-18T08:13:38.499 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.8050000000000001e-06 2024-09-18T08:13:38.499 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:13:38.499 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:13:38.499 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-09-18T08:13:38.499 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:12.450274+0000", 2024-09-18T08:13:38.499 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 5.7865999999999998e-05 2024-09-18T08:13:38.499 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:13:38.499 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:13:38.499 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-09-18T08:13:38.499 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:12.450290+0000", 2024-09-18T08:13:38.499 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.5817000000000001e-05 2024-09-18T08:13:38.500 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:13:38.500 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:13:38.500 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-09-18T08:13:38.500 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:12.450299+0000", 2024-09-18T08:13:38.500 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 9.3759999999999997e-06 2024-09-18T08:13:38.500 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:13:38.500 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:13:38.500 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-18T08:13:38.500 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:12.450369+0000", 2024-09-18T08:13:38.500 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 6.9474000000000006e-05 2024-09-18T08:13:38.500 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:13:38.500 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:13:38.501 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "waiting for peered", 2024-09-18T08:13:38.501 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:12.450380+0000", 2024-09-18T08:13:38.501 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.168e-05 2024-09-18T08:13:38.501 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:13:38.501 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:13:38.501 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-18T08:13:38.501 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:13.207495+0000", 2024-09-18T08:13:38.501 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.75711499500000001 2024-09-18T08:13:38.501 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:13:38.501 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:13:38.501 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-09-18T08:13:38.501 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:13.207620+0000", 2024-09-18T08:13:38.502 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00012430199999999999 2024-09-18T08:13:38.502 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:13:38.502 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:13:38.502 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "waiting for subops from 6,7", 2024-09-18T08:13:38.502 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:13.207781+0000", 2024-09-18T08:13:38.502 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00016101099999999999 2024-09-18T08:13:38.502 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:13:38.502 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:13:38.502 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:13:38.502 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:13.214075+0000", 2024-09-18T08:13:38.502 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0062940419999999997 2024-09-18T08:13:38.502 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:13:38.503 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:13:38.503 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:13:38.503 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:13.214116+0000", 2024-09-18T08:13:38.503 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4.1517000000000002e-05 2024-09-18T08:13:38.503 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:13:38.503 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:13:38.503 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "op_commit", 2024-09-18T08:13:38.503 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:13.220658+0000", 2024-09-18T08:13:38.503 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0065420809999999999 2024-09-18T08:13:38.503 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:13:38.503 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:13:38.503 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "commit_sent", 2024-09-18T08:13:38.503 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:13.220695+0000", 2024-09-18T08:13:38.504 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 3.6896e-05 2024-09-18T08:13:38.504 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:13:38.504 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:13:38.504 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-09-18T08:13:38.504 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:13.220722+0000", 2024-09-18T08:13:38.504 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 2.7322e-05 2024-09-18T08:13:38.504 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-18T08:13:38.504 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-09-18T08:13:38.504 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-18T08:13:38.504 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:13:38.504 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:13:38.504 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.15216.0:86465 8.5 8:a7ecc445:::benchmark_data_smithi007_101583_object86464:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e205)", 2024-09-18T08:13:38.505 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-09-18T08:09:12.495165+0000", 2024-09-18T08:13:38.505 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 265.90507012799998, 2024-09-18T08:13:38.505 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.72562491600000001, 2024-09-18T08:13:38.505 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-09-18T08:13:38.505 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-09-18T08:13:38.505 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T08:13:38.505 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2024-09-18T08:13:38.505 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.15216", 2024-09-18T08:13:38.505 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "172.21.15.7:0/2596041177", 2024-09-18T08:13:38.505 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 86465 2024-09-18T08:13:38.505 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:13:38.505 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-09-18T08:13:38.505 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:13:38.506 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-09-18T08:13:38.506 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:12.495165+0000", 2024-09-18T08:13:38.506 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-18T08:13:38.506 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:13:38.506 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:13:38.506 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-09-18T08:13:38.506 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:12.495165+0000", 2024-09-18T08:13:38.506 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-18T08:13:38.506 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:13:38.506 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:13:38.506 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-09-18T08:13:38.506 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:12.495167+0000", 2024-09-18T08:13:38.506 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.9089999999999998e-06 2024-09-18T08:13:38.507 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:13:38.507 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:13:38.507 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-09-18T08:13:38.507 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:12.495284+0000", 2024-09-18T08:13:38.507 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000117228 2024-09-18T08:13:38.507 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:13:38.507 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:13:38.507 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-09-18T08:13:38.507 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:12.495301+0000", 2024-09-18T08:13:38.507 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.7065e-05 2024-09-18T08:13:38.507 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:13:38.507 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:13:38.508 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-09-18T08:13:38.508 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:12.495312+0000", 2024-09-18T08:13:38.508 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.0132999999999999e-05 2024-09-18T08:13:38.508 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:13:38.508 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:13:38.508 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-18T08:13:38.508 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:12.495388+0000", 2024-09-18T08:13:38.508 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 7.6512000000000002e-05 2024-09-18T08:13:38.508 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:13:38.508 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:13:38.508 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "waiting for peered", 2024-09-18T08:13:38.508 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:12.495402+0000", 2024-09-18T08:13:38.508 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.3376e-05 2024-09-18T08:13:38.508 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:13:38.509 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:13:38.509 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-18T08:13:38.509 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:13.211447+0000", 2024-09-18T08:13:38.509 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.71604507500000003 2024-09-18T08:13:38.509 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:13:38.509 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:13:38.509 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-09-18T08:13:38.509 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:13.211727+0000", 2024-09-18T08:13:38.509 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00028030299999999998 2024-09-18T08:13:38.509 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:13:38.509 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:13:38.509 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "waiting for subops from 6,7", 2024-09-18T08:13:38.509 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:13.211872+0000", 2024-09-18T08:13:38.510 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00014542799999999999 2024-09-18T08:13:38.510 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:13:38.510 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:13:38.510 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:13:38.510 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:13.219374+0000", 2024-09-18T08:13:38.510 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0075019090000000002 2024-09-18T08:13:38.510 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:13:38.510 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:13:38.510 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:13:38.510 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:13.220515+0000", 2024-09-18T08:13:38.510 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0011411839999999999 2024-09-18T08:13:38.510 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:13:38.511 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:13:38.511 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "op_commit", 2024-09-18T08:13:38.511 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:13.220727+0000", 2024-09-18T08:13:38.511 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00021189200000000001 2024-09-18T08:13:38.511 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:13:38.511 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:13:38.511 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "commit_sent", 2024-09-18T08:13:38.511 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:13.220765+0000", 2024-09-18T08:13:38.511 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 3.7319e-05 2024-09-18T08:13:38.511 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:13:38.511 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:13:38.511 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-09-18T08:13:38.512 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:13.220790+0000", 2024-09-18T08:13:38.512 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 2.5582999999999999e-05 2024-09-18T08:13:38.512 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-18T08:13:38.512 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-09-18T08:13:38.512 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-18T08:13:38.512 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:13:38.512 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:13:38.512 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.15216.0:86475 8.5 8:a4f19050:::benchmark_data_smithi007_101583_object86474:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e205)", 2024-09-18T08:13:38.512 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-09-18T08:09:12.523844+0000", 2024-09-18T08:13:38.512 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 265.87639180999997, 2024-09-18T08:13:38.512 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.69708282899999996, 2024-09-18T08:13:38.512 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-09-18T08:13:38.512 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-09-18T08:13:38.513 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T08:13:38.513 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2024-09-18T08:13:38.513 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.15216", 2024-09-18T08:13:38.513 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "172.21.15.7:0/2596041177", 2024-09-18T08:13:38.513 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 86475 2024-09-18T08:13:38.513 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:13:38.513 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-09-18T08:13:38.513 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:13:38.513 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-09-18T08:13:38.513 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:12.523844+0000", 2024-09-18T08:13:38.513 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-18T08:13:38.513 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:13:38.513 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:13:38.514 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-09-18T08:13:38.514 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:12.523844+0000", 2024-09-18T08:13:38.514 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-18T08:13:38.514 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:13:38.514 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:13:38.514 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-09-18T08:13:38.514 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:12.523846+0000", 2024-09-18T08:13:38.514 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 2.0320000000000002e-06 2024-09-18T08:13:38.514 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:13:38.514 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:13:38.514 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-09-18T08:13:38.514 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:12.523886+0000", 2024-09-18T08:13:38.514 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4.0763000000000003e-05 2024-09-18T08:13:38.515 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:13:38.515 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:13:38.515 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-09-18T08:13:38.515 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:12.523911+0000", 2024-09-18T08:13:38.515 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 2.4202e-05 2024-09-18T08:13:38.515 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:13:38.515 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:13:38.515 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-09-18T08:13:38.515 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:12.523920+0000", 2024-09-18T08:13:38.515 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 8.9330000000000007e-06 2024-09-18T08:13:38.515 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:13:38.515 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:13:38.515 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-18T08:13:38.516 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:12.523990+0000", 2024-09-18T08:13:38.516 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 7.0505000000000003e-05 2024-09-18T08:13:38.516 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:13:38.516 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:13:38.516 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "waiting for peered", 2024-09-18T08:13:38.516 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:12.524004+0000", 2024-09-18T08:13:38.516 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.3607e-05 2024-09-18T08:13:38.516 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:13:38.516 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:13:38.516 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-18T08:13:38.516 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:13.214163+0000", 2024-09-18T08:13:38.516 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.69015934999999995 2024-09-18T08:13:38.517 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:13:38.517 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:13:38.517 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-09-18T08:13:38.517 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:13.214291+0000", 2024-09-18T08:13:38.517 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00012824399999999999 2024-09-18T08:13:38.517 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:13:38.517 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:13:38.517 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "waiting for subops from 6,7", 2024-09-18T08:13:38.517 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:13.214455+0000", 2024-09-18T08:13:38.517 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00016422499999999999 2024-09-18T08:13:38.517 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:13:38.517 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:13:38.517 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "op_commit", 2024-09-18T08:13:38.518 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:13.220795+0000", 2024-09-18T08:13:38.518 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0063397560000000002 2024-09-18T08:13:38.518 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:13:38.518 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:13:38.518 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:13:38.518 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:13.220825+0000", 2024-09-18T08:13:38.518 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 2.976e-05 2024-09-18T08:13:38.518 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:13:38.518 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:13:38.518 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:13:38.518 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:13.220855+0000", 2024-09-18T08:13:38.518 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 3.0043999999999999e-05 2024-09-18T08:13:38.518 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:13:38.519 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:13:38.519 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "commit_sent", 2024-09-18T08:13:38.519 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:13.220889+0000", 2024-09-18T08:13:38.519 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 3.4251000000000001e-05 2024-09-18T08:13:38.519 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:13:38.519 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:13:38.519 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-09-18T08:13:38.519 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:13.220926+0000", 2024-09-18T08:13:38.519 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 3.7156999999999997e-05 2024-09-18T08:13:38.519 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-18T08:13:38.519 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-09-18T08:13:38.519 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-18T08:13:38.519 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:13:38.520 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:13:38.520 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.15216.0:86499 8.5 8:ab3bde63:::benchmark_data_smithi007_101583_object86498:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e205)", 2024-09-18T08:13:38.520 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-09-18T08:09:12.639820+0000", 2024-09-18T08:13:38.520 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 265.76041502300001, 2024-09-18T08:13:38.520 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.59097304500000003, 2024-09-18T08:13:38.520 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-09-18T08:13:38.520 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-09-18T08:13:38.520 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T08:13:38.520 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2024-09-18T08:13:38.520 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.15216", 2024-09-18T08:13:38.520 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "172.21.15.7:0/2596041177", 2024-09-18T08:13:38.520 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 86499 2024-09-18T08:13:38.521 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:13:38.521 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-09-18T08:13:38.521 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:13:38.521 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-09-18T08:13:38.521 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:12.639820+0000", 2024-09-18T08:13:38.521 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-18T08:13:38.521 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:13:38.521 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:13:38.521 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-09-18T08:13:38.521 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:12.639820+0000", 2024-09-18T08:13:38.521 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-18T08:13:38.521 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:13:38.521 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:13:38.522 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-09-18T08:13:38.522 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:12.639822+0000", 2024-09-18T08:13:38.522 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.4440000000000001e-06 2024-09-18T08:13:38.522 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:13:38.522 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:13:38.522 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-09-18T08:13:38.522 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:12.639872+0000", 2024-09-18T08:13:38.522 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 5.0296000000000001e-05 2024-09-18T08:13:38.522 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:13:38.522 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:13:38.522 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-09-18T08:13:38.522 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:12.639885+0000", 2024-09-18T08:13:38.522 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.3173000000000001e-05 2024-09-18T08:13:38.522 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:13:38.523 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:13:38.523 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-09-18T08:13:38.523 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:12.639894+0000", 2024-09-18T08:13:38.523 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 8.5310000000000001e-06 2024-09-18T08:13:38.523 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:13:38.523 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:13:38.523 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-18T08:13:38.523 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:12.639972+0000", 2024-09-18T08:13:38.523 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 7.8307999999999996e-05 2024-09-18T08:13:38.523 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:13:38.523 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:13:38.523 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "waiting for peered", 2024-09-18T08:13:38.523 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:12.639980+0000", 2024-09-18T08:13:38.524 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 8.0669999999999992e-06 2024-09-18T08:13:38.524 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:13:38.524 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:13:38.524 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-18T08:13:38.524 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:13.216786+0000", 2024-09-18T08:13:38.524 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.57680564199999995 2024-09-18T08:13:38.524 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:13:38.524 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:13:38.524 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-09-18T08:13:38.524 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:13.216933+0000", 2024-09-18T08:13:38.524 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00014732999999999999 2024-09-18T08:13:38.524 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:13:38.525 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:13:38.525 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "waiting for subops from 6,7", 2024-09-18T08:13:38.525 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:13.217061+0000", 2024-09-18T08:13:38.525 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000128042 2024-09-18T08:13:38.525 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:13:38.525 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:13:38.525 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:13:38.525 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:13.220967+0000", 2024-09-18T08:13:38.527 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0039059939999999999 2024-09-18T08:13:38.527 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:13:38.527 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:13:38.527 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "op_commit", 2024-09-18T08:13:38.527 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:13.228425+0000", 2024-09-18T08:13:38.527 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.007457623 2024-09-18T08:13:38.527 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:13:38.527 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:13:38.527 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:13:38.528 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:13.230681+0000", 2024-09-18T08:13:38.528 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0022558280000000001 2024-09-18T08:13:38.528 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:13:38.528 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:13:38.528 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "commit_sent", 2024-09-18T08:13:38.528 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:13.230747+0000", 2024-09-18T08:13:38.528 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 6.5881000000000002e-05 2024-09-18T08:13:38.528 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:13:38.528 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:13:38.528 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-09-18T08:13:38.528 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:13.230793+0000", 2024-09-18T08:13:38.529 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4.6885999999999998e-05 2024-09-18T08:13:38.529 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-18T08:13:38.529 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-09-18T08:13:38.529 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-18T08:13:38.529 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:13:38.529 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:13:38.529 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.15582.0:53562 9.e 9:7f5cd26d:::benchmark_data_smithi007_124019_object53561:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e243)", 2024-09-18T08:13:38.529 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-09-18T08:11:41.305003+0000", 2024-09-18T08:13:38.529 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 117.095232713, 2024-09-18T08:13:38.529 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.032254158, 2024-09-18T08:13:38.529 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-09-18T08:13:38.529 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-09-18T08:13:38.529 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "reached pg", 2024-09-18T08:13:38.530 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2024-09-18T08:13:38.530 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.15582", 2024-09-18T08:13:38.530 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "172.21.15.7:0/3598605839", 2024-09-18T08:13:38.530 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 53562 2024-09-18T08:13:38.530 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:13:38.530 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-09-18T08:13:38.530 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:13:38.530 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-09-18T08:13:38.530 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:11:41.305003+0000", 2024-09-18T08:13:38.530 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-18T08:13:38.530 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:13:38.530 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:13:38.530 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-09-18T08:13:38.530 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:11:41.305003+0000", 2024-09-18T08:13:38.531 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-18T08:13:38.531 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:13:38.531 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:13:38.531 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-09-18T08:13:38.531 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:11:41.305004+0000", 2024-09-18T08:13:38.531 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.6750000000000001e-06 2024-09-18T08:13:38.531 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:13:38.531 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:13:38.531 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-09-18T08:13:38.531 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:11:41.305059+0000", 2024-09-18T08:13:38.531 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 5.4181999999999997e-05 2024-09-18T08:13:38.531 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:13:38.531 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:13:38.532 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-09-18T08:13:38.532 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:11:41.305074+0000", 2024-09-18T08:13:38.532 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.5379999999999998e-05 2024-09-18T08:13:38.532 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:13:38.532 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:13:38.532 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-09-18T08:13:38.532 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:11:41.305083+0000", 2024-09-18T08:13:38.532 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 8.9549999999999998e-06 2024-09-18T08:13:38.532 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:13:38.532 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:13:38.532 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-18T08:13:38.532 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:11:41.305153+0000", 2024-09-18T08:13:38.532 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 6.9703999999999995e-05 2024-09-18T08:13:38.533 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:13:38.533 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:13:38.533 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-09-18T08:13:38.533 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:11:41.305277+0000", 2024-09-18T08:13:38.533 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0001247 2024-09-18T08:13:38.533 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:13:38.533 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:13:38.533 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "waiting for subops from 1,7", 2024-09-18T08:13:38.533 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:11:41.305414+0000", 2024-09-18T08:13:38.533 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00013647900000000001 2024-09-18T08:13:38.533 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:13:38.533 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:13:38.534 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "op_commit", 2024-09-18T08:13:38.534 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:11:41.307315+0000", 2024-09-18T08:13:38.534 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.001901511 2024-09-18T08:13:38.534 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:13:38.534 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:13:38.534 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:13:38.534 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:11:41.307516+0000", 2024-09-18T08:13:38.534 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00020116900000000001 2024-09-18T08:13:38.534 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:13:38.534 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:13:38.534 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-18T08:13:38.534 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:11:41.323519+0000", 2024-09-18T08:13:38.534 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.016002283999999999 2024-09-18T08:13:38.535 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:13:38.535 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:13:38.535 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "waiting for peered", 2024-09-18T08:13:38.535 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:11:41.323530+0000", 2024-09-18T08:13:38.535 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.1729000000000001e-05 2024-09-18T08:13:38.535 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:13:38.535 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:13:38.535 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-18T08:13:38.536 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:11:42.337165+0000", 2024-09-18T08:13:38.536 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.0136343050000001 2024-09-18T08:13:38.536 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:13:38.536 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:13:38.536 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-09-18T08:13:38.536 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:11:42.337257+0000", 2024-09-18T08:13:38.537 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 9.2084999999999997e-05 2024-09-18T08:13:38.537 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-18T08:13:38.537 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-09-18T08:13:38.537 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-18T08:13:38.537 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:13:38.537 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:13:38.537 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.15582.0:53574 9.e 9:7c49e58c:::benchmark_data_smithi007_124019_object53573:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e243)", 2024-09-18T08:13:38.537 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-09-18T08:11:41.323835+0000", 2024-09-18T08:13:38.537 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 117.07640033, 2024-09-18T08:13:38.537 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.0202460840000001, 2024-09-18T08:13:38.537 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-09-18T08:13:38.537 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-09-18T08:13:38.538 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T08:13:38.538 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2024-09-18T08:13:38.538 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.15582", 2024-09-18T08:13:38.538 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "172.21.15.7:0/3598605839", 2024-09-18T08:13:38.538 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 53574 2024-09-18T08:13:38.538 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:13:38.538 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-09-18T08:13:38.538 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:13:38.538 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-09-18T08:13:38.538 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:11:41.323835+0000", 2024-09-18T08:13:38.538 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-18T08:13:38.538 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:13:38.539 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:13:38.539 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-09-18T08:13:38.539 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:11:41.323835+0000", 2024-09-18T08:13:38.539 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-18T08:13:38.539 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:13:38.539 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:13:38.539 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-09-18T08:13:38.539 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:11:41.323837+0000", 2024-09-18T08:13:38.539 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.8470000000000001e-06 2024-09-18T08:13:38.540 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:13:38.540 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:13:38.540 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-09-18T08:13:38.540 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:11:41.323872+0000", 2024-09-18T08:13:38.540 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 3.4660999999999999e-05 2024-09-18T08:13:38.540 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:13:38.540 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:13:38.540 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-09-18T08:13:38.540 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:11:41.323886+0000", 2024-09-18T08:13:38.540 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.4779e-05 2024-09-18T08:13:38.540 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:13:38.540 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:13:38.540 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-09-18T08:13:38.541 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:11:41.323895+0000", 2024-09-18T08:13:38.541 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 8.4630000000000001e-06 2024-09-18T08:13:38.541 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:13:38.541 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:13:38.541 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-18T08:13:38.541 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:11:41.323994+0000", 2024-09-18T08:13:38.541 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 9.9152000000000002e-05 2024-09-18T08:13:38.541 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:13:38.541 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:13:38.541 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "waiting for peered", 2024-09-18T08:13:38.541 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:11:41.324002+0000", 2024-09-18T08:13:38.541 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 8.4039999999999995e-06 2024-09-18T08:13:38.541 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:13:38.542 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:13:38.542 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-18T08:13:38.542 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:11:42.337953+0000", 2024-09-18T08:13:38.542 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.013950753 2024-09-18T08:13:38.542 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:13:38.542 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:13:38.542 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-09-18T08:13:38.542 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:11:42.338094+0000", 2024-09-18T08:13:38.542 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000140371 2024-09-18T08:13:38.542 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:13:38.542 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:13:38.542 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "waiting for subops from 7", 2024-09-18T08:13:38.542 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:11:42.338248+0000", 2024-09-18T08:13:38.543 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000154723 2024-09-18T08:13:38.543 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:13:38.543 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:13:38.543 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:13:38.543 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:11:42.343852+0000", 2024-09-18T08:13:38.543 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0056041909999999997 2024-09-18T08:13:38.543 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:13:38.543 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:13:38.543 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "op_commit", 2024-09-18T08:13:38.543 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:11:42.343917+0000", 2024-09-18T08:13:38.543 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 6.4090000000000005e-05 2024-09-18T08:13:38.543 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:13:38.543 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:13:38.544 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "commit_sent", 2024-09-18T08:13:38.544 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:11:42.343980+0000", 2024-09-18T08:13:38.544 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 6.3769999999999994e-05 2024-09-18T08:13:38.544 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:13:38.544 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:13:38.544 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-09-18T08:13:38.544 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:11:42.344081+0000", 2024-09-18T08:13:38.544 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00010088 2024-09-18T08:13:38.544 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-18T08:13:38.544 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-09-18T08:13:38.544 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-18T08:13:38.544 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:13:38.545 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:13:38.545 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.15582.0:53552 9.4 9:2d548aaa:::benchmark_data_smithi007_124019_object53551:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e244)", 2024-09-18T08:13:38.545 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-09-18T08:11:41.324333+0000", 2024-09-18T08:13:38.545 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 117.075902906, 2024-09-18T08:13:38.545 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.0111114809999999, 2024-09-18T08:13:38.545 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-09-18T08:13:38.545 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-09-18T08:13:38.545 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "reached pg", 2024-09-18T08:13:38.545 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2024-09-18T08:13:38.545 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.15582", 2024-09-18T08:13:38.545 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "172.21.15.7:0/3598605839", 2024-09-18T08:13:38.545 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 53552 2024-09-18T08:13:38.545 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:13:38.546 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-09-18T08:13:38.546 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:13:38.546 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-09-18T08:13:38.546 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:11:41.324333+0000", 2024-09-18T08:13:38.546 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-18T08:13:38.546 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:13:38.546 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:13:38.546 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-09-18T08:13:38.546 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:11:41.324333+0000", 2024-09-18T08:13:38.546 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-18T08:13:38.546 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:13:38.546 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:13:38.546 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-09-18T08:13:38.547 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:11:41.324334+0000", 2024-09-18T08:13:38.547 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.217e-06 2024-09-18T08:13:38.547 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:13:38.547 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:13:38.547 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-09-18T08:13:38.547 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:11:41.324369+0000", 2024-09-18T08:13:38.547 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 3.4805999999999998e-05 2024-09-18T08:13:38.547 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:13:38.547 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:13:38.547 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-09-18T08:13:38.547 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:11:41.324382+0000", 2024-09-18T08:13:38.547 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.3739e-05 2024-09-18T08:13:38.547 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:13:38.548 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:13:38.548 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-09-18T08:13:38.548 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:11:41.324390+0000", 2024-09-18T08:13:38.548 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 7.3989999999999998e-06 2024-09-18T08:13:38.548 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:13:38.548 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:13:38.548 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-18T08:13:38.548 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:11:41.324445+0000", 2024-09-18T08:13:38.548 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 5.5535999999999998e-05 2024-09-18T08:13:38.548 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:13:38.548 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:13:38.548 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "waiting for peered", 2024-09-18T08:13:38.549 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:11:41.324453+0000", 2024-09-18T08:13:38.549 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 8.2139999999999996e-06 2024-09-18T08:13:38.549 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:13:38.549 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:13:38.549 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-18T08:13:38.549 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:11:42.335357+0000", 2024-09-18T08:13:38.549 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.010903675 2024-09-18T08:13:38.549 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:13:38.549 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:13:38.549 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-09-18T08:13:38.549 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:11:42.335444+0000", 2024-09-18T08:13:38.549 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 8.6895000000000001e-05 2024-09-18T08:13:38.549 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-18T08:13:38.550 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-09-18T08:13:38.550 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-18T08:13:38.550 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:13:38.550 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:13:38.550 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.15582.0:53563 9.4 9:2254eea2:::benchmark_data_smithi007_124019_object53562:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e244)", 2024-09-18T08:13:38.550 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-09-18T08:11:41.324409+0000", 2024-09-18T08:13:38.550 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 117.075826228, 2024-09-18T08:13:38.550 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.022272439, 2024-09-18T08:13:38.550 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-09-18T08:13:38.550 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-09-18T08:13:38.550 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "reached pg", 2024-09-18T08:13:38.550 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2024-09-18T08:13:38.551 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.15582", 2024-09-18T08:13:38.551 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "172.21.15.7:0/3598605839", 2024-09-18T08:13:38.551 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 53563 2024-09-18T08:13:38.551 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:13:38.551 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-09-18T08:13:38.551 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:13:38.551 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-09-18T08:13:38.551 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:11:41.324409+0000", 2024-09-18T08:13:38.551 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-18T08:13:38.551 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:13:38.551 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:13:38.551 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-09-18T08:13:38.551 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:11:41.324409+0000", 2024-09-18T08:13:38.552 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-18T08:13:38.552 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:13:38.552 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:13:38.552 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-09-18T08:13:38.552 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:11:41.324410+0000", 2024-09-18T08:13:38.552 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 5.8400000000000004e-07 2024-09-18T08:13:38.552 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:13:38.552 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:13:38.552 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-09-18T08:13:38.552 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:11:41.324438+0000", 2024-09-18T08:13:38.552 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 2.8438000000000001e-05 2024-09-18T08:13:38.552 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:13:38.552 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:13:38.553 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-09-18T08:13:38.553 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:11:41.324446+0000", 2024-09-18T08:13:38.553 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 8.0020000000000006e-06 2024-09-18T08:13:38.553 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:13:38.553 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:13:38.553 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-09-18T08:13:38.553 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:11:41.324453+0000", 2024-09-18T08:13:38.553 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 6.4910000000000002e-06 2024-09-18T08:13:38.553 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:13:38.553 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:13:38.553 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-18T08:13:38.553 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:11:41.324565+0000", 2024-09-18T08:13:38.553 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000111983 2024-09-18T08:13:38.554 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:13:38.554 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:13:38.554 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "waiting for peered", 2024-09-18T08:13:38.554 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:11:41.324572+0000", 2024-09-18T08:13:38.554 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 7.8080000000000005e-06 2024-09-18T08:13:38.554 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:13:38.554 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:13:38.554 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-18T08:13:38.554 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:11:42.335472+0000", 2024-09-18T08:13:38.554 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.010899996 2024-09-18T08:13:38.554 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:13:38.554 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:13:38.554 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "waiting for missing object", 2024-09-18T08:13:38.555 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:11:42.335604+0000", 2024-09-18T08:13:38.555 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00013134999999999999 2024-09-18T08:13:38.555 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:13:38.555 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:13:38.555 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-18T08:13:38.555 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:11:42.346593+0000", 2024-09-18T08:13:38.555 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.01098872 2024-09-18T08:13:38.555 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:13:38.555 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:13:38.555 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-09-18T08:13:38.555 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:11:42.346682+0000", 2024-09-18T08:13:38.555 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 8.9067000000000006e-05 2024-09-18T08:13:38.555 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-18T08:13:38.556 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-09-18T08:13:38.556 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-18T08:13:38.556 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-18T08:13:38.556 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-09-18T08:13:38.556 INFO:teuthology.orchestra.run.smithi050.stdout:} 2024-09-18T08:13:38.560 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-09-18T08:13:38.562 INFO:teuthology.orchestra.run.smithi007.stderr:dumped all 2024-09-18T08:13:38.564 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:13:38.564 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:13:38.565 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0' 2024-09-18T08:13:38.665 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4 2024-09-18T08:13:38.703 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:13:38.703 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:13:38.704 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4' 2024-09-18T08:13:38.805 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3 2024-09-18T08:13:38.838 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:13:38.838 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:13:38.839 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3' 2024-09-18T08:13:38.906 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:13:38 smithi050 ceph-mon[32371]: pgmap v523: 158 pgs: 30 active+undersized, 34 active+undersized+degraded, 94 active+clean; 4.4 GiB data, 16 GiB used, 700 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 859 op/s; 40816/216258 objects degraded (18.874%) 2024-09-18T08:13:38.906 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:13:38 smithi050 ceph-mon[32371]: 3.1e scrub starts 2024-09-18T08:13:38.906 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:13:38 smithi050 ceph-mon[32371]: 3.1e scrub ok 2024-09-18T08:13:38.939 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4 2024-09-18T08:13:38.979 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:13:38.979 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:13:38.980 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4' 2024-09-18T08:13:38.980 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:13:38 smithi007 ceph-mon[29459]: pgmap v523: 158 pgs: 30 active+undersized, 34 active+undersized+degraded, 94 active+clean; 4.4 GiB data, 16 GiB used, 700 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 859 op/s; 40816/216258 objects degraded (18.874%) 2024-09-18T08:13:38.980 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:13:38 smithi007 ceph-mon[29459]: 3.1e scrub starts 2024-09-18T08:13:38.980 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:13:38 smithi007 ceph-mon[29459]: 3.1e scrub ok 2024-09-18T08:13:38.981 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:13:38 smithi007 ceph-mon[39249]: pgmap v523: 158 pgs: 30 active+undersized, 34 active+undersized+degraded, 94 active+clean; 4.4 GiB data, 16 GiB used, 700 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 859 op/s; 40816/216258 objects degraded (18.874%) 2024-09-18T08:13:38.981 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:13:38 smithi007 ceph-mon[39249]: 3.1e scrub starts 2024-09-18T08:13:38.981 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:13:38 smithi007 ceph-mon[39249]: 3.1e scrub ok 2024-09-18T08:13:39.080 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0 2024-09-18T08:13:39.116 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:13:39.117 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:13:39.118 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0' 2024-09-18T08:13:39.218 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7 2024-09-18T08:13:39.251 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:13:39.251 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:13:39.251 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7' 2024-09-18T08:13:39.352 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2 2024-09-18T08:13:39.384 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:13:39.385 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:13:39.385 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2' 2024-09-18T08:13:39.486 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5 2024-09-18T08:13:39.529 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:13:39.530 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:13:39.530 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5' 2024-09-18T08:13:39.631 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4 2024-09-18T08:13:39.656 INFO:journalctl@ceph.iscsi.iscsi.a.smithi050.stdout:Sep 18 08:13:39 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-iscsi-iscsi-a[57782]: debug there is no tcmu-runner data available 2024-09-18T08:13:39.668 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:13:39.669 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:13:39.669 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4' 2024-09-18T08:13:39.770 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7 2024-09-18T08:13:39.777 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-09-18T08:13:39.778 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-09-18T08:13:39.807 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:13:39.808 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:13:39.808 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7' 2024-09-18T08:13:39.896 DEBUG:teuthology.orchestra.run.smithi050:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid 779349ca-7593-11ef-bceb-c7b262605968 -- ceph daemon osd.5 dump_ops_in_flight 2024-09-18T08:13:39.909 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6 2024-09-18T08:13:39.938 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:13:39 smithi050 ceph-mon[32371]: from='client.25625 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-18T08:13:39.938 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:13:39 smithi050 ceph-mon[32371]: Health check update: Degraded data redundancy: 40816/216258 objects degraded (18.874%), 34 pgs degraded, 64 pgs undersized (PG_DEGRADED) 2024-09-18T08:13:39.974 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:13:39 smithi007 ceph-mon[29459]: from='client.25625 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-18T08:13:39.974 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:13:39 smithi007 ceph-mon[29459]: Health check update: Degraded data redundancy: 40816/216258 objects degraded (18.874%), 34 pgs degraded, 64 pgs undersized (PG_DEGRADED) 2024-09-18T08:13:39.974 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:13:39 smithi007 ceph-mon[39249]: from='client.25625 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-18T08:13:39.974 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:13:39 smithi007 ceph-mon[39249]: Health check update: Degraded data redundancy: 40816/216258 objects degraded (18.874%), 34 pgs degraded, 64 pgs undersized (PG_DEGRADED) 2024-09-18T08:13:39.979 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:13:39.979 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:13:39.979 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6' 2024-09-18T08:13:40.080 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4 2024-09-18T08:13:40.120 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:13:40.121 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:13:40.121 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4' 2024-09-18T08:13:40.222 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5 2024-09-18T08:13:40.276 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:13:40.277 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:13:40.277 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5' 2024-09-18T08:13:40.337 INFO:teuthology.orchestra.run.smithi007.stdout:{"pg_ready":true,"pg_map":{"version":523,"stamp":"2024-09-18T08:13:37.311048+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":4711771097,"num_objects":72086,"num_object_clones":0,"num_object_copies":216258,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":40816,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":72086,"num_whiteouts":0,"num_read":1263,"num_read_kb":1025,"num_write":212612,"num_write_kb":9080642,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":204,"num_bytes_recovered":125726,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":81333,"ondisk_log_size":81333,"up":410,"acting":410,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":474,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":16275840,"kb_used_data":13105392,"kb_used_omap":12,"kb_used_meta":3170291,"kb_avail":733685376,"statfs":{"total":767960285184,"available":751293825024,"internally_reserved":0,"allocated":13419921408,"data_stored":13414898231,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12750,"internal_metadata":3246378546},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,1],"upper_bound":16},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-450691072,"num_objects":-6877,"num_object_clones":0,"num_object_copies":-20631,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-3844,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-6877,"num_whiteouts":0,"num_read":5,"num_read_kb":5,"num_write":6877,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.003959"},"pg_stats":[{"pgid":"3.1f","version":"0'0","reported_seq":496,"reported_epoch":283,"state":"active+clean","last_fresh":"2024-09-18T08:13:33.657255+0000","last_change":"2024-09-18T08:10:15.850596+0000","last_active":"2024-09-18T08:13:33.657255+0000","last_peered":"2024-09-18T08:13:33.657255+0000","last_clean":"2024-09-18T08:13:33.657255+0000","last_became_active":"2024-09-18T08:08:52.697499+0000","last_became_peered":"2024-09-18T08:08:52.697499+0000","last_unstale":"2024-09-18T08:13:33.657255+0000","last_undegraded":"2024-09-18T08:13:33.657255+0000","last_fullsized":"2024-09-18T08:13:33.657255+0000","mapping_epoch":198,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":199,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:10:15.850515+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:46.860092+0000","last_clean_scrub_stamp":"2024-09-18T08:10:15.850515+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:14:24.842877+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"271'15","reported_seq":245,"reported_epoch":282,"state":"active+undersized+degraded","last_fresh":"2024-09-18T08:13:31.646790+0000","last_change":"2024-09-18T08:11:42.323251+0000","last_active":"2024-09-18T08:13:31.646790+0000","last_peered":"2024-09-18T08:13:31.646790+0000","last_clean":"2024-09-18T08:11:40.306776+0000","last_became_active":"2024-09-18T08:11:42.323251+0000","last_became_peered":"2024-09-18T08:11:42.323251+0000","last_unstale":"2024-09-18T08:13:31.646790+0000","last_undegraded":"2024-09-18T08:11:42.318308+0000","last_fullsized":"2024-09-18T08:11:42.318179+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":182,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'9","last_scrub_stamp":"2024-09-18T08:10:19.271241+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_clean_scrub_stamp":"2024-09-18T08:10:19.271241+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:14:17.601010+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":25,"num_read_kb":19,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":["4","6"],"object_location_counts":[{"shards":"4,6","objects":4}],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":84,"reported_epoch":282,"state":"active+undersized","last_fresh":"2024-09-18T08:13:31.650067+0000","last_change":"2024-09-18T08:11:42.331640+0000","last_active":"2024-09-18T08:13:31.650067+0000","last_peered":"2024-09-18T08:13:31.650067+0000","last_clean":"2024-09-18T08:02:50.886830+0000","last_became_active":"2024-09-18T08:11:42.331640+0000","last_became_peered":"2024-09-18T08:11:42.331640+0000","last_unstale":"2024-09-18T08:13:31.650067+0000","last_undegraded":"2024-09-18T08:13:31.650067+0000","last_fullsized":"2024-09-18T08:11:42.326296+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:10:15.603289+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:50.886830+0000","last_clean_scrub_stamp":"2024-09-18T08:10:15.603289+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:14:13.131588+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":["5","7"],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":218,"reported_epoch":282,"state":"active+undersized","last_fresh":"2024-09-18T08:13:31.647888+0000","last_change":"2024-09-18T08:11:42.333493+0000","last_active":"2024-09-18T08:13:31.647888+0000","last_peered":"2024-09-18T08:13:31.647888+0000","last_clean":"2024-09-18T08:11:40.307445+0000","last_became_active":"2024-09-18T08:11:42.333493+0000","last_became_peered":"2024-09-18T08:11:42.333493+0000","last_unstale":"2024-09-18T08:13:31.647888+0000","last_undegraded":"2024-09-18T08:13:31.647888+0000","last_fullsized":"2024-09-18T08:11:42.319356+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":182,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:07:08.451865+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_clean_scrub_stamp":"2024-09-18T08:07:08.451865+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:13:52.472014+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":["4","5"],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":492,"reported_epoch":283,"state":"active+clean","last_fresh":"2024-09-18T08:13:33.657526+0000","last_change":"2024-09-18T08:12:38.915577+0000","last_active":"2024-09-18T08:13:33.657526+0000","last_peered":"2024-09-18T08:13:33.657526+0000","last_clean":"2024-09-18T08:13:33.657526+0000","last_became_active":"2024-09-18T08:06:24.128511+0000","last_became_peered":"2024-09-18T08:06:24.128511+0000","last_unstale":"2024-09-18T08:13:33.657526+0000","last_undegraded":"2024-09-18T08:13:33.657526+0000","last_fullsized":"2024-09-18T08:13:33.657526+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:12:38.915501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_clean_scrub_stamp":"2024-09-18T08:12:38.915501+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:13:39.193844+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":505,"reported_epoch":283,"state":"active+clean","last_fresh":"2024-09-18T08:13:33.651904+0000","last_change":"2024-09-18T08:12:03.127882+0000","last_active":"2024-09-18T08:13:33.651904+0000","last_peered":"2024-09-18T08:13:33.651904+0000","last_clean":"2024-09-18T08:13:33.651904+0000","last_became_active":"2024-09-18T08:06:24.126194+0000","last_became_peered":"2024-09-18T08:06:24.126194+0000","last_unstale":"2024-09-18T08:13:33.651904+0000","last_undegraded":"2024-09-18T08:13:33.651904+0000","last_fullsized":"2024-09-18T08:13:33.651904+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:12:03.127810+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:46.860092+0000","last_clean_scrub_stamp":"2024-09-18T08:12:03.127810+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"271'27","reported_seq":505,"reported_epoch":283,"state":"active+clean","last_fresh":"2024-09-18T08:13:33.652553+0000","last_change":"2024-09-18T08:09:32.575836+0000","last_active":"2024-09-18T08:13:33.652553+0000","last_peered":"2024-09-18T08:13:33.652553+0000","last_clean":"2024-09-18T08:13:33.652553+0000","last_became_active":"2024-09-18T08:02:49.886106+0000","last_became_peered":"2024-09-18T08:02:49.886106+0000","last_unstale":"2024-09-18T08:13:33.652553+0000","last_undegraded":"2024-09-18T08:13:33.652553+0000","last_fullsized":"2024-09-18T08:13:33.652553+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'15","last_scrub_stamp":"2024-09-18T08:09:32.575741+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_clean_scrub_stamp":"2024-09-18T08:09:32.575741+0000","objects_scrubbed":7,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:14:22.687956+0000","scrub_duration":40,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":44,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":218,"reported_epoch":282,"state":"active+undersized","last_fresh":"2024-09-18T08:13:31.647000+0000","last_change":"2024-09-18T08:11:42.323423+0000","last_active":"2024-09-18T08:13:31.647000+0000","last_peered":"2024-09-18T08:13:31.647000+0000","last_clean":"2024-09-18T08:11:40.306782+0000","last_became_active":"2024-09-18T08:11:42.323423+0000","last_became_peered":"2024-09-18T08:11:42.323423+0000","last_unstale":"2024-09-18T08:13:31.647000+0000","last_undegraded":"2024-09-18T08:13:31.647000+0000","last_fullsized":"2024-09-18T08:11:42.318286+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":182,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:08:09.142188+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:50.886830+0000","last_clean_scrub_stamp":"2024-09-18T08:08:09.142188+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:13:57.407915+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":["4","6"],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":507,"reported_epoch":282,"state":"active+clean","last_fresh":"2024-09-18T08:13:31.644762+0000","last_change":"2024-09-18T08:10:51.226034+0000","last_active":"2024-09-18T08:13:31.644762+0000","last_peered":"2024-09-18T08:13:31.644762+0000","last_clean":"2024-09-18T08:13:31.644762+0000","last_became_active":"2024-09-18T08:08:18.548203+0000","last_became_peered":"2024-09-18T08:08:18.548203+0000","last_unstale":"2024-09-18T08:13:31.644762+0000","last_undegraded":"2024-09-18T08:13:31.644762+0000","last_fullsized":"2024-09-18T08:13:31.644762+0000","mapping_epoch":181,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":182,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:10:51.225974+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:46.860092+0000","last_clean_scrub_stamp":"2024-09-18T08:10:51.225974+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:14:14.183168+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"271'15","reported_seq":249,"reported_epoch":282,"state":"active+clean","last_fresh":"2024-09-18T08:13:31.648042+0000","last_change":"2024-09-18T08:12:00.466295+0000","last_active":"2024-09-18T08:13:31.648042+0000","last_peered":"2024-09-18T08:13:31.648042+0000","last_clean":"2024-09-18T08:13:31.648042+0000","last_became_active":"2024-09-18T08:08:18.550960+0000","last_became_peered":"2024-09-18T08:08:18.550960+0000","last_unstale":"2024-09-18T08:13:31.648042+0000","last_undegraded":"2024-09-18T08:13:31.648042+0000","last_fullsized":"2024-09-18T08:13:31.648042+0000","mapping_epoch":181,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":182,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'9","last_scrub_stamp":"2024-09-18T08:12:00.466202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_clean_scrub_stamp":"2024-09-18T08:12:00.466202+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":25,"num_read_kb":19,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":471,"reported_epoch":282,"state":"active+clean","last_fresh":"2024-09-18T08:13:31.643001+0000","last_change":"2024-09-18T08:12:57.218495+0000","last_active":"2024-09-18T08:13:31.643001+0000","last_peered":"2024-09-18T08:13:31.643001+0000","last_clean":"2024-09-18T08:13:31.643001+0000","last_became_active":"2024-09-18T08:02:51.907296+0000","last_became_peered":"2024-09-18T08:02:51.907296+0000","last_unstale":"2024-09-18T08:13:31.643001+0000","last_undegraded":"2024-09-18T08:13:31.643001+0000","last_fullsized":"2024-09-18T08:13:31.643001+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:12:57.218286+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:50.886830+0000","last_clean_scrub_stamp":"2024-09-18T08:12:57.218286+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:14:06.566496+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":466,"reported_epoch":283,"state":"active+undersized","last_fresh":"2024-09-18T08:13:33.656130+0000","last_change":"2024-09-18T08:11:42.330079+0000","last_active":"2024-09-18T08:13:33.656130+0000","last_peered":"2024-09-18T08:13:33.656130+0000","last_clean":"2024-09-18T08:11:40.308148+0000","last_became_active":"2024-09-18T08:11:42.330079+0000","last_became_peered":"2024-09-18T08:11:42.330079+0000","last_unstale":"2024-09-18T08:13:33.656130+0000","last_undegraded":"2024-09-18T08:13:33.656130+0000","last_fullsized":"2024-09-18T08:11:42.319204+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:08:14.645993+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_clean_scrub_stamp":"2024-09-18T08:08:14.645993+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:13:55.773992+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":["0","7"],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":481,"reported_epoch":282,"state":"active+undersized","last_fresh":"2024-09-18T08:13:31.643416+0000","last_change":"2024-09-18T08:11:42.339943+0000","last_active":"2024-09-18T08:13:31.643416+0000","last_peered":"2024-09-18T08:13:31.643416+0000","last_clean":"2024-09-18T08:11:40.302512+0000","last_became_active":"2024-09-18T08:11:42.339943+0000","last_became_peered":"2024-09-18T08:11:42.339943+0000","last_unstale":"2024-09-18T08:13:31.643416+0000","last_undegraded":"2024-09-18T08:13:31.643416+0000","last_fullsized":"2024-09-18T08:11:42.322341+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":182,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:04:32.849119+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:46.860092+0000","last_clean_scrub_stamp":"2024-09-18T08:04:32.849119+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:13:48.261776+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":["5","4"],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1b","version":"271'7","reported_seq":235,"reported_epoch":282,"state":"active+undersized+degraded","last_fresh":"2024-09-18T08:13:31.648778+0000","last_change":"2024-09-18T08:11:42.329404+0000","last_active":"2024-09-18T08:13:31.648778+0000","last_peered":"2024-09-18T08:13:31.648778+0000","last_clean":"2024-09-18T08:11:40.307173+0000","last_became_active":"2024-09-18T08:11:42.329404+0000","last_became_peered":"2024-09-18T08:11:42.329404+0000","last_unstale":"2024-09-18T08:13:31.648778+0000","last_undegraded":"2024-09-18T08:11:42.319890+0000","last_fullsized":"2024-09-18T08:11:42.319791+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":182,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'5","last_scrub_stamp":"2024-09-18T08:10:52.400411+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_clean_scrub_stamp":"2024-09-18T08:10:52.400411+0000","objects_scrubbed":2,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:14:06.467689+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":13,"num_read_kb":9,"num_write":8,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":["4","3"],"object_location_counts":[{"shards":"3,4","objects":2}],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":513,"reported_epoch":282,"state":"active+undersized","last_fresh":"2024-09-18T08:13:31.647999+0000","last_change":"2024-09-18T08:11:42.329293+0000","last_active":"2024-09-18T08:13:31.647999+0000","last_peered":"2024-09-18T08:13:31.647999+0000","last_clean":"2024-09-18T08:11:40.308737+0000","last_became_active":"2024-09-18T08:11:42.329293+0000","last_became_peered":"2024-09-18T08:11:42.329293+0000","last_unstale":"2024-09-18T08:13:31.647999+0000","last_undegraded":"2024-09-18T08:13:31.647999+0000","last_fullsized":"2024-09-18T08:11:42.320879+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":182,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:11:30.837341+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:50.886830+0000","last_clean_scrub_stamp":"2024-09-18T08:11:30.837341+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:13:50.677084+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":["7","4"],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":472,"reported_epoch":282,"state":"active+undersized","last_fresh":"2024-09-18T08:13:31.650238+0000","last_change":"2024-09-18T08:11:42.332383+0000","last_active":"2024-09-18T08:13:31.650238+0000","last_peered":"2024-09-18T08:13:31.650238+0000","last_clean":"2024-09-18T08:11:40.302158+0000","last_became_active":"2024-09-18T08:11:42.332383+0000","last_became_peered":"2024-09-18T08:11:42.332383+0000","last_unstale":"2024-09-18T08:13:31.650238+0000","last_undegraded":"2024-09-18T08:13:31.650238+0000","last_fullsized":"2024-09-18T08:11:42.326605+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:11:01.108648+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_clean_scrub_stamp":"2024-09-18T08:11:01.108648+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:14:30.163932+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":["5","3"],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":222,"reported_epoch":282,"state":"active+clean","last_fresh":"2024-09-18T08:13:31.651701+0000","last_change":"2024-09-18T08:11:34.625641+0000","last_active":"2024-09-18T08:13:31.651701+0000","last_peered":"2024-09-18T08:13:31.651701+0000","last_clean":"2024-09-18T08:13:31.651701+0000","last_became_active":"2024-09-18T08:08:18.539604+0000","last_became_peered":"2024-09-18T08:08:18.539604+0000","last_unstale":"2024-09-18T08:13:31.651701+0000","last_undegraded":"2024-09-18T08:13:31.651701+0000","last_fullsized":"2024-09-18T08:13:31.651701+0000","mapping_epoch":181,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":182,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:11:34.625572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_clean_scrub_stamp":"2024-09-18T08:11:34.625572+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":495,"reported_epoch":283,"state":"active+clean","last_fresh":"2024-09-18T08:13:33.657046+0000","last_change":"2024-09-18T08:12:40.711594+0000","last_active":"2024-09-18T08:13:33.657046+0000","last_peered":"2024-09-18T08:13:33.657046+0000","last_clean":"2024-09-18T08:13:33.657046+0000","last_became_active":"2024-09-18T08:08:18.523611+0000","last_became_peered":"2024-09-18T08:08:18.523611+0000","last_unstale":"2024-09-18T08:13:33.657046+0000","last_undegraded":"2024-09-18T08:13:33.657046+0000","last_fullsized":"2024-09-18T08:13:33.657046+0000","mapping_epoch":181,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":182,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:12:40.711507+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:46.860092+0000","last_clean_scrub_stamp":"2024-09-18T08:12:40.711507+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:13:48.813540+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1c","version":"271'27","reported_seq":219,"reported_epoch":283,"state":"active+undersized+degraded","last_fresh":"2024-09-18T08:13:33.651616+0000","last_change":"2024-09-18T08:11:42.329080+0000","last_active":"2024-09-18T08:13:33.651616+0000","last_peered":"2024-09-18T08:13:33.651616+0000","last_clean":"2024-09-18T08:11:40.306909+0000","last_became_active":"2024-09-18T08:11:42.329080+0000","last_became_peered":"2024-09-18T08:11:42.329080+0000","last_unstale":"2024-09-18T08:13:33.651616+0000","last_undegraded":"2024-09-18T08:11:42.319732+0000","last_fullsized":"2024-09-18T08:11:42.319638+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":199,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'15","last_scrub_stamp":"2024-09-18T08:07:39.530340+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_clean_scrub_stamp":"2024-09-18T08:07:39.530340+0000","objects_scrubbed":7,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:13:47.287389+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":7,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":44,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":7}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":357,"reported_epoch":282,"state":"active+undersized","last_fresh":"2024-09-18T08:13:31.649121+0000","last_change":"2024-09-18T08:11:42.330489+0000","last_active":"2024-09-18T08:13:31.649121+0000","last_peered":"2024-09-18T08:13:31.649121+0000","last_clean":"2024-09-18T08:08:15.505698+0000","last_became_active":"2024-09-18T08:11:42.330489+0000","last_became_peered":"2024-09-18T08:11:42.330489+0000","last_unstale":"2024-09-18T08:13:31.649121+0000","last_undegraded":"2024-09-18T08:13:31.649121+0000","last_fullsized":"2024-09-18T08:11:42.318950+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":182,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:10:59.645112+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:50.886830+0000","last_clean_scrub_stamp":"2024-09-18T08:10:59.645112+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:14:05.464503+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":["4","0"],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":477,"reported_epoch":283,"state":"active+clean","last_fresh":"2024-09-18T08:13:33.651451+0000","last_change":"2024-09-18T08:09:36.592348+0000","last_active":"2024-09-18T08:13:33.651451+0000","last_peered":"2024-09-18T08:13:33.651451+0000","last_clean":"2024-09-18T08:13:33.651451+0000","last_became_active":"2024-09-18T08:06:24.128659+0000","last_became_peered":"2024-09-18T08:06:24.128659+0000","last_unstale":"2024-09-18T08:13:33.651451+0000","last_undegraded":"2024-09-18T08:13:33.651451+0000","last_fullsized":"2024-09-18T08:13:33.651451+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:09:36.592281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_clean_scrub_stamp":"2024-09-18T08:09:36.592281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:14:21.721636+0000","scrub_duration":16,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":218,"reported_epoch":282,"state":"active+undersized","last_fresh":"2024-09-18T08:13:31.647675+0000","last_change":"2024-09-18T08:11:42.335948+0000","last_active":"2024-09-18T08:13:31.647675+0000","last_peered":"2024-09-18T08:13:31.647675+0000","last_clean":"2024-09-18T08:11:40.306989+0000","last_became_active":"2024-09-18T08:11:42.335948+0000","last_became_peered":"2024-09-18T08:11:42.335948+0000","last_unstale":"2024-09-18T08:13:31.647675+0000","last_undegraded":"2024-09-18T08:13:31.647675+0000","last_fullsized":"2024-09-18T08:11:42.319106+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":182,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:07:13.543722+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:46.860092+0000","last_clean_scrub_stamp":"2024-09-18T08:07:13.543722+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:13:54.429777+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":["4","2"],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"271'18","reported_seq":560,"reported_epoch":283,"state":"active+undersized+degraded","last_fresh":"2024-09-18T08:13:33.652151+0000","last_change":"2024-09-18T08:11:42.329686+0000","last_active":"2024-09-18T08:13:33.652151+0000","last_peered":"2024-09-18T08:13:33.652151+0000","last_clean":"2024-09-18T08:11:40.313323+0000","last_became_active":"2024-09-18T08:11:42.329686+0000","last_became_peered":"2024-09-18T08:11:42.329686+0000","last_unstale":"2024-09-18T08:13:33.652151+0000","last_undegraded":"2024-09-18T08:11:42.324380+0000","last_fullsized":"2024-09-18T08:11:42.324261+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":182,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'12","last_scrub_stamp":"2024-09-18T08:11:32.304472+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_clean_scrub_stamp":"2024-09-18T08:11:32.304472+0000","objects_scrubbed":5,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:14:27.707068+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":5,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":34,"num_read_kb":25,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":["3","4"],"object_location_counts":[{"shards":"3,4","objects":5}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":222,"reported_epoch":282,"state":"active+clean","last_fresh":"2024-09-18T08:13:31.649258+0000","last_change":"2024-09-18T08:10:46.303912+0000","last_active":"2024-09-18T08:13:31.649258+0000","last_peered":"2024-09-18T08:13:31.649258+0000","last_clean":"2024-09-18T08:13:31.649258+0000","last_became_active":"2024-09-18T08:08:18.531438+0000","last_became_peered":"2024-09-18T08:08:18.531438+0000","last_unstale":"2024-09-18T08:13:31.649258+0000","last_undegraded":"2024-09-18T08:13:31.649258+0000","last_fullsized":"2024-09-18T08:13:31.649258+0000","mapping_epoch":181,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":182,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:10:46.303848+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:50.886830+0000","last_clean_scrub_stamp":"2024-09-18T08:10:46.303848+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:14:19.606868+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1c","version":"69'1","reported_seq":310,"reported_epoch":282,"state":"active+clean","last_fresh":"2024-09-18T08:13:31.643038+0000","last_change":"2024-09-18T08:10:46.212697+0000","last_active":"2024-09-18T08:13:31.643038+0000","last_peered":"2024-09-18T08:13:31.643038+0000","last_clean":"2024-09-18T08:13:31.643038+0000","last_became_active":"2024-09-18T08:06:46.141661+0000","last_became_peered":"2024-09-18T08:06:46.141661+0000","last_unstale":"2024-09-18T08:13:31.643038+0000","last_undegraded":"2024-09-18T08:13:31.643038+0000","last_fullsized":"2024-09-18T08:13:31.643038+0000","mapping_epoch":143,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":144,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'1","last_scrub_stamp":"2024-09-18T08:10:46.212626+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_clean_scrub_stamp":"2024-09-18T08:10:46.212626+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:14:18.188204+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":403,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,2],"acting":[5,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"62'1","reported_seq":96,"reported_epoch":283,"state":"active+undersized+degraded","last_fresh":"2024-09-18T08:13:33.651664+0000","last_change":"2024-09-18T08:11:42.335870+0000","last_active":"2024-09-18T08:13:33.651664+0000","last_peered":"2024-09-18T08:13:33.651664+0000","last_clean":"2024-09-18T08:02:47.882026+0000","last_became_active":"2024-09-18T08:11:42.335870+0000","last_became_peered":"2024-09-18T08:11:42.335870+0000","last_unstale":"2024-09-18T08:13:33.651664+0000","last_undegraded":"2024-09-18T08:11:42.323433+0000","last_fullsized":"2024-09-18T08:11:42.323288+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":182,"parent":"0.0","parent_split_bits":0,"last_scrub":"62'1","last_scrub_stamp":"2024-09-18T08:11:00.634190+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:46.860092+0000","last_clean_scrub_stamp":"2024-09-18T08:11:00.634190+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:13:59.006000+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":["3","4"],"object_location_counts":[{"shards":"3,4","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"271'20","reported_seq":500,"reported_epoch":283,"state":"active+clean","last_fresh":"2024-09-18T08:13:33.656878+0000","last_change":"2024-09-18T08:13:30.833792+0000","last_active":"2024-09-18T08:13:33.656878+0000","last_peered":"2024-09-18T08:13:33.656878+0000","last_clean":"2024-09-18T08:13:33.656878+0000","last_became_active":"2024-09-18T08:02:49.892024+0000","last_became_peered":"2024-09-18T08:02:49.892024+0000","last_unstale":"2024-09-18T08:13:33.656878+0000","last_undegraded":"2024-09-18T08:13:33.656878+0000","last_fullsized":"2024-09-18T08:13:33.656878+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"271'20","last_scrub_stamp":"2024-09-18T08:13:30.833726+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_clean_scrub_stamp":"2024-09-18T08:13:30.833726+0000","objects_scrubbed":5,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:14:42.922788+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":25,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1f","version":"182'13","reported_seq":290,"reported_epoch":282,"state":"active+clean","last_fresh":"2024-09-18T08:13:31.650813+0000","last_change":"2024-09-18T08:12:59.403122+0000","last_active":"2024-09-18T08:13:31.650813+0000","last_peered":"2024-09-18T08:13:31.650813+0000","last_clean":"2024-09-18T08:13:31.650813+0000","last_became_active":"2024-09-18T08:08:18.543095+0000","last_became_peered":"2024-09-18T08:08:18.543095+0000","last_unstale":"2024-09-18T08:13:31.650813+0000","last_undegraded":"2024-09-18T08:13:31.650813+0000","last_fullsized":"2024-09-18T08:13:31.650813+0000","mapping_epoch":181,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":182,"parent":"0.0","parent_split_bits":0,"last_scrub":"182'13","last_scrub_stamp":"2024-09-18T08:12:59.403054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:50.886830+0000","last_clean_scrub_stamp":"2024-09-18T08:12:59.403054+0000","objects_scrubbed":1,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:14:04.748540+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,7],"acting":[4,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.2","version":"283'13337","reported_seq":13455,"reported_epoch":283,"state":"active+undersized+degraded","last_fresh":"2024-09-18T08:13:34.074987+0000","last_change":"2024-09-18T08:11:42.343258+0000","last_active":"2024-09-18T08:13:34.074987+0000","last_peered":"2024-09-18T08:13:34.074987+0000","last_clean":"2024-09-18T08:11:41.306257+0000","last_became_active":"2024-09-18T08:11:42.343258+0000","last_became_peered":"2024-09-18T08:11:42.343258+0000","last_unstale":"2024-09-18T08:13:34.074987+0000","last_undegraded":"2024-09-18T08:11:42.323950+0000","last_fullsized":"2024-09-18T08:11:42.323830+0000","mapping_epoch":244,"log_start":"267'9000","ondisk_log_start":"267'9000","created":232,"last_epoch_clean":233,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:11:08.262286+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:11:08.262286+0000","last_clean_scrub_stamp":"2024-09-18T08:11:08.262286+0000","objects_scrubbed":0,"log_size":4337,"log_dups_size":0,"ondisk_log_size":4337,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:13:57.965537+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":297861120,"num_objects":4545,"num_object_clones":0,"num_object_copies":13635,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4545,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4545,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13337,"num_write_kb":572224,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["3","2"],"object_location_counts":[{"shards":"2,3","objects":4545}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"74'79","reported_seq":292,"reported_epoch":282,"state":"active+clean","last_fresh":"2024-09-18T08:13:31.645730+0000","last_change":"2024-09-18T08:11:29.068419+0000","last_active":"2024-09-18T08:13:31.645730+0000","last_peered":"2024-09-18T08:13:31.645730+0000","last_clean":"2024-09-18T08:13:31.645730+0000","last_became_active":"2024-09-18T08:07:26.394584+0000","last_became_peered":"2024-09-18T08:07:26.394584+0000","last_unstale":"2024-09-18T08:13:31.645730+0000","last_undegraded":"2024-09-18T08:13:31.645730+0000","last_fullsized":"2024-09-18T08:13:31.645730+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":152,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":4,"last_scrub":"74'79","last_scrub_stamp":"2024-09-18T08:11:29.068355+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:00:11.893742+0000","last_clean_scrub_stamp":"2024-09-18T08:11:29.068355+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,6],"acting":[5,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.f","version":"271'27","reported_seq":149,"reported_epoch":283,"state":"active+undersized+degraded","last_fresh":"2024-09-18T08:13:33.651825+0000","last_change":"2024-09-18T08:11:42.336910+0000","last_active":"2024-09-18T08:13:33.651825+0000","last_peered":"2024-09-18T08:13:33.651825+0000","last_clean":"2024-09-18T08:02:54.411508+0000","last_became_active":"2024-09-18T08:11:42.336910+0000","last_became_peered":"2024-09-18T08:11:42.336910+0000","last_unstale":"2024-09-18T08:13:33.651825+0000","last_undegraded":"2024-09-18T08:11:42.326875+0000","last_fullsized":"2024-09-18T08:11:42.326758+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":182,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'15","last_scrub_stamp":"2024-09-18T08:10:41.548188+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_clean_scrub_stamp":"2024-09-18T08:10:41.548188+0000","objects_scrubbed":0,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:13:45.030554+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":7,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":43,"num_read_kb":34,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":["3","4"],"object_location_counts":[{"shards":"3,4","objects":7}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":478,"reported_epoch":283,"state":"active+undersized","last_fresh":"2024-09-18T08:13:33.654547+0000","last_change":"2024-09-18T08:11:42.338122+0000","last_active":"2024-09-18T08:13:33.654547+0000","last_peered":"2024-09-18T08:13:33.654547+0000","last_clean":"2024-09-18T08:11:40.314538+0000","last_became_active":"2024-09-18T08:11:42.338122+0000","last_became_peered":"2024-09-18T08:11:42.338122+0000","last_unstale":"2024-09-18T08:13:33.654547+0000","last_undegraded":"2024-09-18T08:13:33.654547+0000","last_fullsized":"2024-09-18T08:11:42.325100+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:08:18.389273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:46.860092+0000","last_clean_scrub_stamp":"2024-09-18T08:08:18.389273+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:13:52.020423+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":["3","7"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.e","version":"182'12","reported_seq":298,"reported_epoch":282,"state":"active+clean","last_fresh":"2024-09-18T08:13:31.651657+0000","last_change":"2024-09-18T08:13:31.620633+0000","last_active":"2024-09-18T08:13:31.651657+0000","last_peered":"2024-09-18T08:13:31.651657+0000","last_clean":"2024-09-18T08:13:31.651657+0000","last_became_active":"2024-09-18T08:08:18.525147+0000","last_became_peered":"2024-09-18T08:08:18.525147+0000","last_unstale":"2024-09-18T08:13:31.651657+0000","last_undegraded":"2024-09-18T08:13:31.651657+0000","last_fullsized":"2024-09-18T08:13:31.651657+0000","mapping_epoch":181,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":182,"parent":"0.0","parent_split_bits":0,"last_scrub":"182'12","last_scrub_stamp":"2024-09-18T08:13:31.620205+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:50.886830+0000","last_clean_scrub_stamp":"2024-09-18T08:13:31.620205+0000","objects_scrubbed":1,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:14:56.820672+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":472,"reported_epoch":282,"state":"active+undersized","last_fresh":"2024-09-18T08:13:31.643719+0000","last_change":"2024-09-18T08:11:42.330265+0000","last_active":"2024-09-18T08:13:31.643719+0000","last_peered":"2024-09-18T08:13:31.643719+0000","last_clean":"2024-09-18T08:11:40.302096+0000","last_became_active":"2024-09-18T08:11:42.330265+0000","last_became_peered":"2024-09-18T08:11:42.330265+0000","last_unstale":"2024-09-18T08:13:31.643719+0000","last_undegraded":"2024-09-18T08:13:31.643719+0000","last_fullsized":"2024-09-18T08:11:42.324373+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:07:19.714430+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_clean_scrub_stamp":"2024-09-18T08:07:19.714430+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:14:04.243722+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":["5","0"],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.d","version":"282'13284","reported_seq":13399,"reported_epoch":282,"state":"active+undersized+degraded","last_fresh":"2024-09-18T08:13:33.428691+0000","last_change":"2024-09-18T08:11:42.340820+0000","last_active":"2024-09-18T08:13:33.428691+0000","last_peered":"2024-09-18T08:13:33.428691+0000","last_clean":"2024-09-18T08:11:41.317368+0000","last_became_active":"2024-09-18T08:11:42.340820+0000","last_became_peered":"2024-09-18T08:11:42.340820+0000","last_unstale":"2024-09-18T08:13:33.428691+0000","last_undegraded":"2024-09-18T08:11:42.327808+0000","last_fullsized":"2024-09-18T08:11:42.327694+0000","mapping_epoch":244,"log_start":"265'8600","ondisk_log_start":"265'8600","created":232,"last_epoch_clean":233,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:11:08.262286+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:11:08.262286+0000","last_clean_scrub_stamp":"2024-09-18T08:11:08.262286+0000","objects_scrubbed":0,"log_size":4684,"log_dups_size":0,"ondisk_log_size":4684,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":302972952,"num_objects":4624,"num_object_clones":0,"num_object_copies":13872,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4624,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4624,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":13284,"num_write_kb":572993,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":["7","5"],"object_location_counts":[{"shards":"5,7","objects":4624}],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.5","version":"74'79","reported_seq":237,"reported_epoch":282,"state":"active+clean","last_fresh":"2024-09-18T08:13:31.648493+0000","last_change":"2024-09-18T08:11:54.503697+0000","last_active":"2024-09-18T08:13:31.648493+0000","last_peered":"2024-09-18T08:13:31.648493+0000","last_clean":"2024-09-18T08:13:31.648493+0000","last_became_active":"2024-09-18T08:08:52.690344+0000","last_became_peered":"2024-09-18T08:08:52.690344+0000","last_unstale":"2024-09-18T08:13:31.648493+0000","last_undegraded":"2024-09-18T08:13:31.648493+0000","last_fullsized":"2024-09-18T08:13:31.648493+0000","mapping_epoch":198,"log_start":"0'0","ondisk_log_start":"0'0","created":152,"last_epoch_clean":199,"parent":"0.0","parent_split_bits":3,"last_scrub":"74'79","last_scrub_stamp":"2024-09-18T08:11:54.503603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:00:11.893742+0000","last_clean_scrub_stamp":"2024-09-18T08:11:54.503603+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,6],"acting":[4,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.0","version":"271'30","reported_seq":537,"reported_epoch":283,"state":"active+clean","last_fresh":"2024-09-18T08:13:33.654311+0000","last_change":"2024-09-18T08:12:36.956399+0000","last_active":"2024-09-18T08:13:33.654311+0000","last_peered":"2024-09-18T08:13:33.654311+0000","last_clean":"2024-09-18T08:13:33.654311+0000","last_became_active":"2024-09-18T08:02:49.886818+0000","last_became_peered":"2024-09-18T08:02:49.886818+0000","last_unstale":"2024-09-18T08:13:33.654311+0000","last_undegraded":"2024-09-18T08:13:33.654311+0000","last_fullsized":"2024-09-18T08:13:33.654311+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'18","last_scrub_stamp":"2024-09-18T08:12:36.956258+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_clean_scrub_stamp":"2024-09-18T08:12:36.956258+0000","objects_scrubbed":8,"log_size":30,"log_dups_size":0,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:13:41.247230+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":52,"num_read_kb":40,"num_write":32,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":498,"reported_epoch":283,"state":"active+clean","last_fresh":"2024-09-18T08:13:33.652656+0000","last_change":"2024-09-18T08:11:20.379525+0000","last_active":"2024-09-18T08:13:33.652656+0000","last_peered":"2024-09-18T08:13:33.652656+0000","last_clean":"2024-09-18T08:13:33.652656+0000","last_became_active":"2024-09-18T08:02:47.880225+0000","last_became_peered":"2024-09-18T08:02:47.880225+0000","last_unstale":"2024-09-18T08:13:33.652656+0000","last_undegraded":"2024-09-18T08:13:33.652656+0000","last_fullsized":"2024-09-18T08:13:33.652656+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:11:20.379467+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:46.860092+0000","last_clean_scrub_stamp":"2024-09-18T08:11:20.379467+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:14:25.698313+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":230,"reported_epoch":282,"state":"active+clean","last_fresh":"2024-09-18T08:13:31.647680+0000","last_change":"2024-09-18T08:12:30.732619+0000","last_active":"2024-09-18T08:13:31.647680+0000","last_peered":"2024-09-18T08:13:31.647680+0000","last_clean":"2024-09-18T08:13:31.647680+0000","last_became_active":"2024-09-18T08:08:18.530341+0000","last_became_peered":"2024-09-18T08:08:18.530341+0000","last_unstale":"2024-09-18T08:13:31.647680+0000","last_undegraded":"2024-09-18T08:13:31.647680+0000","last_fullsized":"2024-09-18T08:13:31.647680+0000","mapping_epoch":181,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":182,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:12:30.732539+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:50.886830+0000","last_clean_scrub_stamp":"2024-09-18T08:12:30.732539+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:13:50.343431+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":230,"reported_epoch":282,"state":"active+clean","last_fresh":"2024-09-18T08:13:31.647951+0000","last_change":"2024-09-18T08:12:02.473229+0000","last_active":"2024-09-18T08:13:31.647951+0000","last_peered":"2024-09-18T08:13:31.647951+0000","last_clean":"2024-09-18T08:13:31.647951+0000","last_became_active":"2024-09-18T08:08:18.543694+0000","last_became_peered":"2024-09-18T08:08:18.543694+0000","last_unstale":"2024-09-18T08:13:31.647951+0000","last_undegraded":"2024-09-18T08:13:31.647951+0000","last_fullsized":"2024-09-18T08:13:31.647951+0000","mapping_epoch":181,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":182,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:12:02.473168+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_clean_scrub_stamp":"2024-09-18T08:12:02.473168+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.c","version":"283'13211","reported_seq":13406,"reported_epoch":283,"state":"active+clean","last_fresh":"2024-09-18T08:13:34.059311+0000","last_change":"2024-09-18T08:13:06.086787+0000","last_active":"2024-09-18T08:13:34.059311+0000","last_peered":"2024-09-18T08:13:34.059311+0000","last_clean":"2024-09-18T08:13:34.059311+0000","last_became_active":"2024-09-18T08:11:09.293176+0000","last_became_peered":"2024-09-18T08:11:09.293176+0000","last_unstale":"2024-09-18T08:13:34.059311+0000","last_undegraded":"2024-09-18T08:13:34.059311+0000","last_fullsized":"2024-09-18T08:13:34.059311+0000","mapping_epoch":232,"log_start":"267'8860","ondisk_log_start":"267'8860","created":232,"last_epoch_clean":233,"parent":"0.0","parent_split_bits":0,"last_scrub":"274'10686","last_scrub_stamp":"2024-09-18T08:13:06.086640+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:11:08.262286+0000","last_clean_scrub_stamp":"2024-09-18T08:13:06.086640+0000","objects_scrubbed":7100,"log_size":4351,"log_dups_size":0,"ondisk_log_size":4351,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:14:25.235882+0000","scrub_duration":2162,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":294191104,"num_objects":4489,"num_object_clones":0,"num_object_copies":13467,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4489,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13211,"num_write_kb":566400,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.4","version":"74'79","reported_seq":308,"reported_epoch":283,"state":"active+clean","last_fresh":"2024-09-18T08:13:33.653181+0000","last_change":"2024-09-18T08:12:37.920428+0000","last_active":"2024-09-18T08:13:33.653181+0000","last_peered":"2024-09-18T08:13:33.653181+0000","last_clean":"2024-09-18T08:13:33.653181+0000","last_became_active":"2024-09-18T08:07:26.398930+0000","last_became_peered":"2024-09-18T08:07:26.398930+0000","last_unstale":"2024-09-18T08:13:33.653181+0000","last_undegraded":"2024-09-18T08:13:33.653181+0000","last_fullsized":"2024-09-18T08:13:33.653181+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":152,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":3,"last_scrub":"74'79","last_scrub_stamp":"2024-09-18T08:12:37.920342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:00:11.893742+0000","last_clean_scrub_stamp":"2024-09-18T08:12:37.920342+0000","objects_scrubbed":1,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:13:38.011227+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":327680,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,7],"acting":[3,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"271'28","reported_seq":259,"reported_epoch":282,"state":"active+clean","last_fresh":"2024-09-18T08:13:31.647490+0000","last_change":"2024-09-18T08:11:53.524126+0000","last_active":"2024-09-18T08:13:31.647490+0000","last_peered":"2024-09-18T08:13:31.647490+0000","last_clean":"2024-09-18T08:13:31.647490+0000","last_became_active":"2024-09-18T08:08:18.549671+0000","last_became_peered":"2024-09-18T08:08:18.549671+0000","last_unstale":"2024-09-18T08:13:31.647490+0000","last_undegraded":"2024-09-18T08:13:31.647490+0000","last_fullsized":"2024-09-18T08:13:31.647490+0000","mapping_epoch":181,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":182,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'14","last_scrub_stamp":"2024-09-18T08:11:53.524045+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_clean_scrub_stamp":"2024-09-18T08:11:53.524045+0000","objects_scrubbed":7,"log_size":28,"log_dups_size":0,"ondisk_log_size":28,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":41,"num_read_kb":34,"num_write":28,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"62'2","reported_seq":510,"reported_epoch":283,"state":"active+undersized+degraded","last_fresh":"2024-09-18T08:13:33.657079+0000","last_change":"2024-09-18T08:11:42.338771+0000","last_active":"2024-09-18T08:13:33.657079+0000","last_peered":"2024-09-18T08:13:33.657079+0000","last_clean":"2024-09-18T08:11:40.307484+0000","last_became_active":"2024-09-18T08:11:42.338771+0000","last_became_peered":"2024-09-18T08:11:42.338771+0000","last_unstale":"2024-09-18T08:13:33.657079+0000","last_undegraded":"2024-09-18T08:11:42.320507+0000","last_fullsized":"2024-09-18T08:11:42.320405+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":182,"parent":"0.0","parent_split_bits":0,"last_scrub":"62'2","last_scrub_stamp":"2024-09-18T08:06:43.607152+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:46.860092+0000","last_clean_scrub_stamp":"2024-09-18T08:06:43.607152+0000","objects_scrubbed":2,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:13:46.675999+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1039,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":2078,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":["0","4"],"object_location_counts":[{"shards":"0,4","objects":2}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.0","version":"245'11","reported_seq":632,"reported_epoch":283,"state":"active+undersized+degraded","last_fresh":"2024-09-18T08:13:33.654089+0000","last_change":"2024-09-18T08:11:42.339963+0000","last_active":"2024-09-18T08:13:33.654089+0000","last_peered":"2024-09-18T08:13:33.654089+0000","last_clean":"2024-09-18T08:11:40.313426+0000","last_became_active":"2024-09-18T08:11:42.339963+0000","last_became_peered":"2024-09-18T08:11:42.339963+0000","last_unstale":"2024-09-18T08:13:33.654089+0000","last_undegraded":"2024-09-18T08:11:42.326291+0000","last_fullsized":"2024-09-18T08:11:42.326182+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":182,"parent":"0.0","parent_split_bits":0,"last_scrub":"182'10","last_scrub_stamp":"2024-09-18T08:09:51.467349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:50.886830+0000","last_clean_scrub_stamp":"2024-09-18T08:09:51.467349+0000","objects_scrubbed":1,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:13:56.035212+0000","scrub_duration":19,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":["3","4"],"object_location_counts":[{"shards":"3,4","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":22,"reported_epoch":282,"state":"active+clean","last_fresh":"2024-09-18T08:13:33.139096+0000","last_change":"2024-09-18T08:13:33.139096+0000","last_active":"2024-09-18T08:13:33.139096+0000","last_peered":"2024-09-18T08:13:33.139096+0000","last_clean":"2024-09-18T08:13:33.139096+0000","last_became_active":"2024-09-18T08:13:28.629930+0000","last_became_peered":"2024-09-18T08:13:28.629930+0000","last_unstale":"2024-09-18T08:13:33.139096+0000","last_undegraded":"2024-09-18T08:13:33.139096+0000","last_fullsized":"2024-09-18T08:13:33.139096+0000","mapping_epoch":280,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":281,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:13:33.138734+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_clean_scrub_stamp":"2024-09-18T08:13:33.138734+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:14:43.987341+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.f","version":"282'13239","reported_seq":13353,"reported_epoch":282,"state":"active+undersized+degraded","last_fresh":"2024-09-18T08:13:33.508934+0000","last_change":"2024-09-18T08:11:42.337301+0000","last_active":"2024-09-18T08:13:33.508934+0000","last_peered":"2024-09-18T08:13:33.508934+0000","last_clean":"2024-09-18T08:11:41.300139+0000","last_became_active":"2024-09-18T08:11:42.337301+0000","last_became_peered":"2024-09-18T08:11:42.337301+0000","last_unstale":"2024-09-18T08:13:33.508934+0000","last_undegraded":"2024-09-18T08:11:42.332576+0000","last_fullsized":"2024-09-18T08:11:42.332455+0000","mapping_epoch":244,"log_start":"264'8342","ondisk_log_start":"264'8342","created":232,"last_epoch_clean":233,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:11:08.262286+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:11:08.262286+0000","last_clean_scrub_stamp":"2024-09-18T08:11:08.262286+0000","objects_scrubbed":0,"log_size":4897,"log_dups_size":0,"ondisk_log_size":4897,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:14:21.188664+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":298516480,"num_objects":4555,"num_object_clones":0,"num_object_copies":13665,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4555,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4555,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13239,"num_write_kb":569408,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":["5","3"],"object_location_counts":[{"shards":"3,5","objects":4555}],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.7","version":"74'79","reported_seq":303,"reported_epoch":283,"state":"active+clean","last_fresh":"2024-09-18T08:13:33.652093+0000","last_change":"2024-09-18T08:12:33.846970+0000","last_active":"2024-09-18T08:13:33.652093+0000","last_peered":"2024-09-18T08:13:33.652093+0000","last_clean":"2024-09-18T08:13:33.652093+0000","last_became_active":"2024-09-18T08:07:26.398289+0000","last_became_peered":"2024-09-18T08:07:26.398289+0000","last_unstale":"2024-09-18T08:13:33.652093+0000","last_undegraded":"2024-09-18T08:13:33.652093+0000","last_fullsized":"2024-09-18T08:13:33.652093+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":152,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":3,"last_scrub":"74'79","last_scrub_stamp":"2024-09-18T08:12:33.846913+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:00:11.893742+0000","last_clean_scrub_stamp":"2024-09-18T08:12:33.846913+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:13:51.842141+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,7],"acting":[3,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.2","version":"271'20","reported_seq":134,"reported_epoch":282,"state":"active+undersized+degraded","last_fresh":"2024-09-18T08:13:31.646235+0000","last_change":"2024-09-18T08:11:42.341545+0000","last_active":"2024-09-18T08:13:31.646235+0000","last_peered":"2024-09-18T08:13:31.646235+0000","last_clean":"2024-09-18T08:02:54.382625+0000","last_became_active":"2024-09-18T08:11:42.341545+0000","last_became_peered":"2024-09-18T08:11:42.341545+0000","last_unstale":"2024-09-18T08:13:31.646235+0000","last_undegraded":"2024-09-18T08:11:42.323463+0000","last_fullsized":"2024-09-18T08:11:42.323350+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":182,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'10","last_scrub_stamp":"2024-09-18T08:11:24.347134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_clean_scrub_stamp":"2024-09-18T08:11:24.347134+0000","objects_scrubbed":0,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":5,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":29,"num_read_kb":24,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":["5","4"],"object_location_counts":[{"shards":"4,5","objects":5}],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":480,"reported_epoch":282,"state":"active+clean","last_fresh":"2024-09-18T08:13:31.644850+0000","last_change":"2024-09-18T08:12:06.876766+0000","last_active":"2024-09-18T08:13:31.644850+0000","last_peered":"2024-09-18T08:13:31.644850+0000","last_clean":"2024-09-18T08:13:31.644850+0000","last_became_active":"2024-09-18T08:02:47.874727+0000","last_became_peered":"2024-09-18T08:02:47.874727+0000","last_unstale":"2024-09-18T08:13:31.644850+0000","last_undegraded":"2024-09-18T08:13:31.644850+0000","last_fullsized":"2024-09-18T08:13:31.644850+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:12:06.876541+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:46.860092+0000","last_clean_scrub_stamp":"2024-09-18T08:12:06.876541+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.3","version":"138'10","reported_seq":627,"reported_epoch":283,"state":"active+clean","last_fresh":"2024-09-18T08:13:33.656733+0000","last_change":"2024-09-18T08:07:46.695714+0000","last_active":"2024-09-18T08:13:33.656733+0000","last_peered":"2024-09-18T08:13:33.656733+0000","last_clean":"2024-09-18T08:13:33.656733+0000","last_became_active":"2024-09-18T08:06:24.134194+0000","last_became_peered":"2024-09-18T08:06:24.134194+0000","last_unstale":"2024-09-18T08:13:33.656733+0000","last_undegraded":"2024-09-18T08:13:33.656733+0000","last_fullsized":"2024-09-18T08:13:33.656733+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"138'10","last_scrub_stamp":"2024-09-18T08:07:46.695656+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:50.886830+0000","last_clean_scrub_stamp":"2024-09-18T08:07:46.695656+0000","objects_scrubbed":1,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:13:50.682156+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":493,"reported_epoch":283,"state":"active+clean","last_fresh":"2024-09-18T08:13:33.656297+0000","last_change":"2024-09-18T08:09:46.989395+0000","last_active":"2024-09-18T08:13:33.656297+0000","last_peered":"2024-09-18T08:13:33.656297+0000","last_clean":"2024-09-18T08:13:33.656297+0000","last_became_active":"2024-09-18T08:08:52.700080+0000","last_became_peered":"2024-09-18T08:08:52.700080+0000","last_unstale":"2024-09-18T08:13:33.656297+0000","last_undegraded":"2024-09-18T08:13:33.656297+0000","last_fullsized":"2024-09-18T08:13:33.656297+0000","mapping_epoch":198,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":199,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:09:46.989329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_clean_scrub_stamp":"2024-09-18T08:09:46.989329+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:14:16.802519+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.e","version":"282'13163","reported_seq":13278,"reported_epoch":282,"state":"active+undersized+degraded","last_fresh":"2024-09-18T08:13:33.322150+0000","last_change":"2024-09-18T08:11:42.329475+0000","last_active":"2024-09-18T08:13:33.322150+0000","last_peered":"2024-09-18T08:13:33.322150+0000","last_clean":"2024-09-18T08:11:41.287488+0000","last_became_active":"2024-09-18T08:11:42.329475+0000","last_became_peered":"2024-09-18T08:11:42.329475+0000","last_unstale":"2024-09-18T08:13:33.322150+0000","last_undegraded":"2024-09-18T08:11:42.322141+0000","last_fullsized":"2024-09-18T08:11:42.321960+0000","mapping_epoch":244,"log_start":"265'8400","ondisk_log_start":"265'8400","created":232,"last_epoch_clean":233,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:11:08.262286+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:11:08.262286+0000","last_clean_scrub_stamp":"2024-09-18T08:11:08.262286+0000","objects_scrubbed":0,"log_size":4763,"log_dups_size":0,"ondisk_log_size":4763,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:14:08.457475+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":291176448,"num_objects":4443,"num_object_clones":0,"num_object_copies":13329,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4443,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4443,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13163,"num_write_kb":563392,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":["4","7"],"object_location_counts":[{"shards":"4,7","objects":4443}],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.6","version":"74'79","reported_seq":383,"reported_epoch":283,"state":"active+clean","last_fresh":"2024-09-18T08:13:33.656553+0000","last_change":"2024-09-18T08:12:28.659146+0000","last_active":"2024-09-18T08:13:33.656553+0000","last_peered":"2024-09-18T08:13:33.656553+0000","last_clean":"2024-09-18T08:13:33.656553+0000","last_became_active":"2024-09-18T08:07:26.426532+0000","last_became_peered":"2024-09-18T08:07:26.426532+0000","last_unstale":"2024-09-18T08:13:33.656553+0000","last_undegraded":"2024-09-18T08:13:33.656553+0000","last_fullsized":"2024-09-18T08:13:33.656553+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":152,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":3,"last_scrub":"74'79","last_scrub_stamp":"2024-09-18T08:12:28.659059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:00:11.893742+0000","last_clean_scrub_stamp":"2024-09-18T08:12:28.659059+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:13:46.488791+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,3],"acting":[0,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.3","version":"271'29","reported_seq":529,"reported_epoch":283,"state":"active+clean","last_fresh":"2024-09-18T08:13:33.815055+0000","last_change":"2024-09-18T08:13:33.815055+0000","last_active":"2024-09-18T08:13:33.815055+0000","last_peered":"2024-09-18T08:13:33.815055+0000","last_clean":"2024-09-18T08:13:33.815055+0000","last_became_active":"2024-09-18T08:02:49.890318+0000","last_became_peered":"2024-09-18T08:02:49.890318+0000","last_unstale":"2024-09-18T08:13:33.815055+0000","last_undegraded":"2024-09-18T08:13:33.815055+0000","last_fullsized":"2024-09-18T08:13:33.815055+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"271'29","last_scrub_stamp":"2024-09-18T08:13:33.814992+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_clean_scrub_stamp":"2024-09-18T08:13:33.814992+0000","objects_scrubbed":8,"log_size":29,"log_dups_size":0,"ondisk_log_size":29,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:14:48.599147+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":54,"num_read_kb":40,"num_write":32,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.4","version":"62'1","reported_seq":395,"reported_epoch":283,"state":"active+undersized+degraded","last_fresh":"2024-09-18T08:13:33.651427+0000","last_change":"2024-09-18T08:11:42.328236+0000","last_active":"2024-09-18T08:13:33.651427+0000","last_peered":"2024-09-18T08:13:33.651427+0000","last_clean":"2024-09-18T08:08:50.690355+0000","last_became_active":"2024-09-18T08:11:42.328236+0000","last_became_peered":"2024-09-18T08:11:42.328236+0000","last_unstale":"2024-09-18T08:13:33.651427+0000","last_undegraded":"2024-09-18T08:11:42.319415+0000","last_fullsized":"2024-09-18T08:11:42.319312+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":199,"parent":"0.0","parent_split_bits":0,"last_scrub":"62'1","last_scrub_stamp":"2024-09-18T08:10:07.593004+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:46.860092+0000","last_clean_scrub_stamp":"2024-09-18T08:10:07.593004+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:13:55.289663+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":436,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":436,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":["2","5"],"object_location_counts":[{"shards":"2,5","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":400,"reported_epoch":283,"state":"active+clean","last_fresh":"2024-09-18T08:13:33.656352+0000","last_change":"2024-09-18T08:12:37.732331+0000","last_active":"2024-09-18T08:13:33.656352+0000","last_peered":"2024-09-18T08:13:33.656352+0000","last_clean":"2024-09-18T08:13:33.656352+0000","last_became_active":"2024-09-18T08:06:24.133262+0000","last_became_peered":"2024-09-18T08:06:24.133262+0000","last_unstale":"2024-09-18T08:13:33.656352+0000","last_undegraded":"2024-09-18T08:13:33.656352+0000","last_fullsized":"2024-09-18T08:13:33.656352+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:12:37.732260+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:50.886830+0000","last_clean_scrub_stamp":"2024-09-18T08:12:37.732260+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:14:02.173227+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":403,"reported_epoch":283,"state":"active+undersized","last_fresh":"2024-09-18T08:13:33.649311+0000","last_change":"2024-09-18T08:11:42.332856+0000","last_active":"2024-09-18T08:13:33.649311+0000","last_peered":"2024-09-18T08:13:33.649311+0000","last_clean":"2024-09-18T08:08:50.690188+0000","last_became_active":"2024-09-18T08:11:42.332856+0000","last_became_peered":"2024-09-18T08:11:42.332856+0000","last_unstale":"2024-09-18T08:13:33.649311+0000","last_undegraded":"2024-09-18T08:13:33.649311+0000","last_fullsized":"2024-09-18T08:11:42.318131+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":199,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:11:28.346395+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_clean_scrub_stamp":"2024-09-18T08:11:28.346395+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:13:53.343534+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":["2","6"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.9","version":"283'13274","reported_seq":13466,"reported_epoch":283,"state":"active+clean","last_fresh":"2024-09-18T08:13:34.024058+0000","last_change":"2024-09-18T08:13:08.016091+0000","last_active":"2024-09-18T08:13:34.024058+0000","last_peered":"2024-09-18T08:13:34.024058+0000","last_clean":"2024-09-18T08:13:34.024058+0000","last_became_active":"2024-09-18T08:11:09.292238+0000","last_became_peered":"2024-09-18T08:11:09.292238+0000","last_unstale":"2024-09-18T08:13:34.024058+0000","last_undegraded":"2024-09-18T08:13:34.024058+0000","last_fullsized":"2024-09-18T08:13:34.024058+0000","mapping_epoch":232,"log_start":"261'7600","ondisk_log_start":"261'7600","created":232,"last_epoch_clean":233,"parent":"0.0","parent_split_bits":0,"last_scrub":"274'10944","last_scrub_stamp":"2024-09-18T08:13:08.015855+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:11:08.262286+0000","last_clean_scrub_stamp":"2024-09-18T08:13:08.015855+0000","objects_scrubbed":6827,"log_size":5674,"log_dups_size":0,"ondisk_log_size":5674,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:14:26.287586+0000","scrub_duration":2172,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":288489472,"num_objects":4402,"num_object_clones":0,"num_object_copies":13206,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4402,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13274,"num_write_kb":565632,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,5],"acting":[0,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"74'79","reported_seq":292,"reported_epoch":282,"state":"active+clean","last_fresh":"2024-09-18T08:13:31.649903+0000","last_change":"2024-09-18T08:11:30.036947+0000","last_active":"2024-09-18T08:13:31.649903+0000","last_peered":"2024-09-18T08:13:31.649903+0000","last_clean":"2024-09-18T08:13:31.649903+0000","last_became_active":"2024-09-18T08:07:26.400643+0000","last_became_peered":"2024-09-18T08:07:26.400643+0000","last_unstale":"2024-09-18T08:13:31.649903+0000","last_undegraded":"2024-09-18T08:13:31.649903+0000","last_fullsized":"2024-09-18T08:13:31.649903+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":152,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":4,"last_scrub":"74'79","last_scrub_stamp":"2024-09-18T08:11:30.036868+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:00:11.893742+0000","last_clean_scrub_stamp":"2024-09-18T08:11:30.036868+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,2],"acting":[5,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.4","version":"271'41","reported_seq":198,"reported_epoch":283,"state":"active+undersized+degraded","last_fresh":"2024-09-18T08:13:33.652111+0000","last_change":"2024-09-18T08:11:42.328675+0000","last_active":"2024-09-18T08:13:33.652111+0000","last_peered":"2024-09-18T08:13:33.652111+0000","last_clean":"2024-09-18T08:02:54.384549+0000","last_became_active":"2024-09-18T08:11:42.328675+0000","last_became_peered":"2024-09-18T08:11:42.328675+0000","last_unstale":"2024-09-18T08:13:33.652111+0000","last_undegraded":"2024-09-18T08:11:42.320375+0000","last_fullsized":"2024-09-18T08:11:42.320272+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'28","last_scrub_stamp":"2024-09-18T08:10:46.497874+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_clean_scrub_stamp":"2024-09-18T08:10:46.497874+0000","objects_scrubbed":0,"log_size":41,"log_dups_size":0,"ondisk_log_size":41,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:14:14.418386+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":10,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":65,"num_read_kb":50,"num_write":38,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":10}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":214,"reported_epoch":282,"state":"active+clean","last_fresh":"2024-09-18T08:13:31.648965+0000","last_change":"2024-09-18T08:08:18.543579+0000","last_active":"2024-09-18T08:13:31.648965+0000","last_peered":"2024-09-18T08:13:31.648965+0000","last_clean":"2024-09-18T08:13:31.648965+0000","last_became_active":"2024-09-18T08:08:18.538852+0000","last_became_peered":"2024-09-18T08:08:18.538852+0000","last_unstale":"2024-09-18T08:13:31.648965+0000","last_undegraded":"2024-09-18T08:13:31.648965+0000","last_fullsized":"2024-09-18T08:13:31.648965+0000","mapping_epoch":181,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":182,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:05:39.490141+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:46.860092+0000","last_clean_scrub_stamp":"2024-09-18T08:05:39.490141+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:13:48.505870+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"64'2","reported_seq":508,"reported_epoch":282,"state":"active+undersized+degraded","last_fresh":"2024-09-18T08:13:31.646466+0000","last_change":"2024-09-18T08:11:42.338718+0000","last_active":"2024-09-18T08:13:31.646466+0000","last_peered":"2024-09-18T08:13:31.646466+0000","last_clean":"2024-09-18T08:11:40.302328+0000","last_became_active":"2024-09-18T08:11:42.338718+0000","last_became_peered":"2024-09-18T08:11:42.338718+0000","last_unstale":"2024-09-18T08:13:31.646466+0000","last_undegraded":"2024-09-18T08:11:42.331457+0000","last_fullsized":"2024-09-18T08:11:42.330826+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"64'2","last_scrub_stamp":"2024-09-18T08:05:44.802942+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:45.855979+0000","last_clean_scrub_stamp":"2024-09-18T08:05:44.802942+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:13:59.231831+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.001348349,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":38,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":["5","6"],"object_location_counts":[{"shards":"5,6","objects":1}],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":484,"reported_epoch":283,"state":"active+undersized","last_fresh":"2024-09-18T08:13:33.656635+0000","last_change":"2024-09-18T08:11:42.334063+0000","last_active":"2024-09-18T08:13:33.656635+0000","last_peered":"2024-09-18T08:13:33.656635+0000","last_clean":"2024-09-18T08:11:40.307605+0000","last_became_active":"2024-09-18T08:11:42.334063+0000","last_became_peered":"2024-09-18T08:11:42.334063+0000","last_unstale":"2024-09-18T08:13:33.656635+0000","last_undegraded":"2024-09-18T08:13:33.656635+0000","last_fullsized":"2024-09-18T08:11:42.319418+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":182,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:10:53.880606+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:50.886830+0000","last_clean_scrub_stamp":"2024-09-18T08:10:53.880606+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:13:43.745796+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":["0","4"],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":485,"reported_epoch":283,"state":"active+clean","last_fresh":"2024-09-18T08:13:33.651635+0000","last_change":"2024-09-18T08:12:06.185241+0000","last_active":"2024-09-18T08:13:33.651635+0000","last_peered":"2024-09-18T08:13:33.651635+0000","last_clean":"2024-09-18T08:13:33.651635+0000","last_became_active":"2024-09-18T08:08:18.542678+0000","last_became_peered":"2024-09-18T08:08:18.542678+0000","last_unstale":"2024-09-18T08:13:33.651635+0000","last_undegraded":"2024-09-18T08:13:33.651635+0000","last_fullsized":"2024-09-18T08:13:33.651635+0000","mapping_epoch":181,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":182,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:12:06.185172+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_clean_scrub_stamp":"2024-09-18T08:12:06.185172+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.a","version":"282'13159","reported_seq":13275,"reported_epoch":282,"state":"active+undersized+degraded","last_fresh":"2024-09-18T08:13:33.488392+0000","last_change":"2024-09-18T08:11:42.337494+0000","last_active":"2024-09-18T08:13:33.488392+0000","last_peered":"2024-09-18T08:13:33.488392+0000","last_clean":"2024-09-18T08:11:41.323914+0000","last_became_active":"2024-09-18T08:11:42.337494+0000","last_became_peered":"2024-09-18T08:11:42.337494+0000","last_unstale":"2024-09-18T08:13:33.488392+0000","last_undegraded":"2024-09-18T08:11:42.329944+0000","last_fullsized":"2024-09-18T08:11:42.329820+0000","mapping_epoch":244,"log_start":"264'8300","ondisk_log_start":"264'8300","created":232,"last_epoch_clean":233,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:11:08.262286+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:11:08.262286+0000","last_clean_scrub_stamp":"2024-09-18T08:11:08.262286+0000","objects_scrubbed":0,"log_size":4859,"log_dups_size":0,"ondisk_log_size":4859,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":296943616,"num_objects":4531,"num_object_clones":0,"num_object_copies":13593,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4531,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4531,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13159,"num_write_kb":566080,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":["5","7"],"object_location_counts":[{"shards":"5,7","objects":4531}],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.2","version":"74'79","reported_seq":396,"reported_epoch":283,"state":"active+clean","last_fresh":"2024-09-18T08:13:33.656155+0000","last_change":"2024-09-18T08:11:51.733475+0000","last_active":"2024-09-18T08:13:33.656155+0000","last_peered":"2024-09-18T08:13:33.656155+0000","last_clean":"2024-09-18T08:13:33.656155+0000","last_became_active":"2024-09-18T08:08:18.522977+0000","last_became_peered":"2024-09-18T08:08:18.522977+0000","last_unstale":"2024-09-18T08:13:33.656155+0000","last_undegraded":"2024-09-18T08:13:33.656155+0000","last_fullsized":"2024-09-18T08:13:33.656155+0000","mapping_epoch":181,"log_start":"0'0","ondisk_log_start":"0'0","created":152,"last_epoch_clean":182,"parent":"0.0","parent_split_bits":4,"last_scrub":"74'79","last_scrub_stamp":"2024-09-18T08:11:51.733401+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:00:11.893742+0000","last_clean_scrub_stamp":"2024-09-18T08:11:51.733401+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,4],"acting":[0,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.7","version":"271'17","reported_seq":438,"reported_epoch":283,"state":"active+clean","last_fresh":"2024-09-18T08:13:33.650065+0000","last_change":"2024-09-18T08:13:30.296958+0000","last_active":"2024-09-18T08:13:33.650065+0000","last_peered":"2024-09-18T08:13:33.650065+0000","last_clean":"2024-09-18T08:13:33.650065+0000","last_became_active":"2024-09-18T08:04:18.144925+0000","last_became_peered":"2024-09-18T08:04:18.144925+0000","last_unstale":"2024-09-18T08:13:33.650065+0000","last_undegraded":"2024-09-18T08:13:33.650065+0000","last_fullsized":"2024-09-18T08:13:33.650065+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"271'17","last_scrub_stamp":"2024-09-18T08:13:30.296881+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_clean_scrub_stamp":"2024-09-18T08:13:30.296881+0000","objects_scrubbed":5,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:14:48.825965+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":35,"num_read_kb":24,"num_write":20,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,0],"acting":[2,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":261,"reported_epoch":283,"state":"active+undersized","last_fresh":"2024-09-18T08:13:33.650000+0000","last_change":"2024-09-18T08:11:42.322313+0000","last_active":"2024-09-18T08:13:33.650000+0000","last_peered":"2024-09-18T08:13:33.650000+0000","last_clean":"2024-09-18T08:06:21.513250+0000","last_became_active":"2024-09-18T08:11:42.322313+0000","last_became_peered":"2024-09-18T08:11:42.322313+0000","last_unstale":"2024-09-18T08:13:33.650000+0000","last_undegraded":"2024-09-18T08:13:33.650000+0000","last_fullsized":"2024-09-18T08:11:42.318707+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:11:31.367525+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:46.860092+0000","last_clean_scrub_stamp":"2024-09-18T08:11:31.367525+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:13:54.335823+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":["2","6"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1","version":"62'1","reported_seq":492,"reported_epoch":283,"state":"active+clean","last_fresh":"2024-09-18T08:13:33.649725+0000","last_change":"2024-09-18T08:12:03.497218+0000","last_active":"2024-09-18T08:13:33.649725+0000","last_peered":"2024-09-18T08:13:33.649725+0000","last_clean":"2024-09-18T08:13:33.649725+0000","last_became_active":"2024-09-18T08:02:46.868175+0000","last_became_peered":"2024-09-18T08:02:46.868175+0000","last_unstale":"2024-09-18T08:13:33.649725+0000","last_undegraded":"2024-09-18T08:13:33.649725+0000","last_fullsized":"2024-09-18T08:13:33.649725+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"62'1","last_scrub_stamp":"2024-09-18T08:12:03.497148+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:45.855979+0000","last_clean_scrub_stamp":"2024-09-18T08:12:03.497148+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:14:25.351946+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.00066785499999999997,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":182,"reported_epoch":283,"state":"active+clean","last_fresh":"2024-09-18T08:13:33.650331+0000","last_change":"2024-09-18T08:08:52.690656+0000","last_active":"2024-09-18T08:13:33.650331+0000","last_peered":"2024-09-18T08:13:33.650331+0000","last_clean":"2024-09-18T08:13:33.650331+0000","last_became_active":"2024-09-18T08:08:52.690286+0000","last_became_peered":"2024-09-18T08:08:52.690286+0000","last_unstale":"2024-09-18T08:13:33.650331+0000","last_undegraded":"2024-09-18T08:13:33.650331+0000","last_fullsized":"2024-09-18T08:13:33.650331+0000","mapping_epoch":198,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":199,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:07:08.806748+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:50.886830+0000","last_clean_scrub_stamp":"2024-09-18T08:07:08.806748+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:13:45.259864+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":491,"reported_epoch":282,"state":"active+clean","last_fresh":"2024-09-18T08:13:31.649280+0000","last_change":"2024-09-18T08:12:04.859323+0000","last_active":"2024-09-18T08:13:31.649280+0000","last_peered":"2024-09-18T08:13:31.649280+0000","last_clean":"2024-09-18T08:13:31.649280+0000","last_became_active":"2024-09-18T08:06:24.145382+0000","last_became_peered":"2024-09-18T08:06:24.145382+0000","last_unstale":"2024-09-18T08:13:31.649280+0000","last_undegraded":"2024-09-18T08:13:31.649280+0000","last_fullsized":"2024-09-18T08:13:31.649280+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:12:04.859145+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_clean_scrub_stamp":"2024-09-18T08:12:04.859145+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.b","version":"283'13118","reported_seq":13232,"reported_epoch":283,"state":"active+clean","last_fresh":"2024-09-18T08:13:34.001978+0000","last_change":"2024-09-18T08:11:09.489007+0000","last_active":"2024-09-18T08:13:34.001978+0000","last_peered":"2024-09-18T08:13:34.001978+0000","last_clean":"2024-09-18T08:13:34.001978+0000","last_became_active":"2024-09-18T08:11:09.488677+0000","last_became_peered":"2024-09-18T08:11:09.488677+0000","last_unstale":"2024-09-18T08:13:34.001978+0000","last_undegraded":"2024-09-18T08:13:34.001978+0000","last_fullsized":"2024-09-18T08:13:34.001978+0000","mapping_epoch":232,"log_start":"259'7400","ondisk_log_start":"259'7400","created":232,"last_epoch_clean":233,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:11:08.262286+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:11:08.262286+0000","last_clean_scrub_stamp":"2024-09-18T08:11:08.262286+0000","objects_scrubbed":0,"log_size":5718,"log_dups_size":0,"ondisk_log_size":5718,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:13:41.748081+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":290455552,"num_objects":4432,"num_object_clones":0,"num_object_copies":13296,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4432,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13118,"num_write_kb":561600,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"74'79","reported_seq":162,"reported_epoch":282,"state":"active+undersized","last_fresh":"2024-09-18T08:13:31.648278+0000","last_change":"2024-09-18T08:11:42.324310+0000","last_active":"2024-09-18T08:13:31.648278+0000","last_peered":"2024-09-18T08:13:31.648278+0000","last_clean":"2024-09-18T08:08:15.515052+0000","last_became_active":"2024-09-18T08:11:42.324310+0000","last_became_peered":"2024-09-18T08:11:42.324310+0000","last_unstale":"2024-09-18T08:13:31.648278+0000","last_undegraded":"2024-09-18T08:13:31.648278+0000","last_fullsized":"2024-09-18T08:11:42.318664+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":152,"last_epoch_clean":182,"parent":"0.0","parent_split_bits":3,"last_scrub":"74'79","last_scrub_stamp":"2024-09-18T08:09:41.651729+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:00:11.893742+0000","last_clean_scrub_stamp":"2024-09-18T08:09:41.651729+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"user requested scrub scheduled @ 2024-09-18T08:13:43.631429+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":53670,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":10,"num_read_kb":20,"num_write":11,"num_write_kb":148,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":53670,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":["4","6"],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.6","version":"271'24","reported_seq":559,"reported_epoch":283,"state":"active+undersized+degraded","last_fresh":"2024-09-18T08:13:33.656927+0000","last_change":"2024-09-18T08:11:42.342654+0000","last_active":"2024-09-18T08:13:33.656927+0000","last_peered":"2024-09-18T08:13:33.656927+0000","last_clean":"2024-09-18T08:11:40.307291+0000","last_became_active":"2024-09-18T08:11:42.342654+0000","last_became_peered":"2024-09-18T08:11:42.342654+0000","last_unstale":"2024-09-18T08:13:33.656927+0000","last_undegraded":"2024-09-18T08:11:42.320107+0000","last_fullsized":"2024-09-18T08:11:42.320001+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":200,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'12","last_scrub_stamp":"2024-09-18T08:10:58.837531+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_clean_scrub_stamp":"2024-09-18T08:10:58.837531+0000","objects_scrubbed":6,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:13:47.699391+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":6,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":36,"num_read_kb":30,"num_write":24,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0,2","objects":6}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":504,"reported_epoch":283,"state":"active+clean","last_fresh":"2024-09-18T08:13:33.656251+0000","last_change":"2024-09-18T08:12:05.794780+0000","last_active":"2024-09-18T08:13:33.656251+0000","last_peered":"2024-09-18T08:13:33.656251+0000","last_clean":"2024-09-18T08:13:33.656251+0000","last_became_active":"2024-09-18T08:08:18.523135+0000","last_became_peered":"2024-09-18T08:08:18.523135+0000","last_unstale":"2024-09-18T08:13:33.656251+0000","last_undegraded":"2024-09-18T08:13:33.656251+0000","last_fullsized":"2024-09-18T08:13:33.656251+0000","mapping_epoch":181,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":182,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:12:05.794714+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:46.860092+0000","last_clean_scrub_stamp":"2024-09-18T08:12:05.794714+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:13:34.497459+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"70'5","reported_seq":241,"reported_epoch":283,"state":"active+undersized+degraded","last_fresh":"2024-09-18T08:13:33.656663+0000","last_change":"2024-09-18T08:11:42.330514+0000","last_active":"2024-09-18T08:13:33.656663+0000","last_peered":"2024-09-18T08:13:33.656663+0000","last_clean":"2024-09-18T08:02:58.641213+0000","last_became_active":"2024-09-18T08:11:42.330514+0000","last_became_peered":"2024-09-18T08:11:42.330514+0000","last_unstale":"2024-09-18T08:13:33.656663+0000","last_undegraded":"2024-09-18T08:11:42.319867+0000","last_fullsized":"2024-09-18T08:11:42.319754+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":144,"parent":"0.0","parent_split_bits":0,"last_scrub":"70'5","last_scrub_stamp":"2024-09-18T08:10:44.388400+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:45.855979+0000","last_clean_scrub_stamp":"2024-09-18T08:10:44.388400+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:13:34.709050+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000504055,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":118,"num_read_kb":113,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":["0","7"],"object_location_counts":[{"shards":"0,7","objects":1}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":475,"reported_epoch":282,"state":"active+undersized","last_fresh":"2024-09-18T08:13:31.643660+0000","last_change":"2024-09-18T08:11:42.333954+0000","last_active":"2024-09-18T08:13:31.643660+0000","last_peered":"2024-09-18T08:13:31.643660+0000","last_clean":"2024-09-18T08:11:40.304599+0000","last_became_active":"2024-09-18T08:11:42.333954+0000","last_became_peered":"2024-09-18T08:11:42.333954+0000","last_unstale":"2024-09-18T08:13:31.643660+0000","last_undegraded":"2024-09-18T08:13:31.643660+0000","last_fullsized":"2024-09-18T08:11:42.326679+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:07:20.767841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:50.886830+0000","last_clean_scrub_stamp":"2024-09-18T08:07:20.767841+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:14:02.229317+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":["5","0"],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":84,"reported_epoch":282,"state":"active+undersized","last_fresh":"2024-09-18T08:13:31.643599+0000","last_change":"2024-09-18T08:11:42.341422+0000","last_active":"2024-09-18T08:13:31.643599+0000","last_peered":"2024-09-18T08:13:31.643599+0000","last_clean":"2024-09-18T08:02:52.890475+0000","last_became_active":"2024-09-18T08:11:42.341422+0000","last_became_peered":"2024-09-18T08:11:42.341422+0000","last_unstale":"2024-09-18T08:13:31.643599+0000","last_undegraded":"2024-09-18T08:13:31.643599+0000","last_fullsized":"2024-09-18T08:11:42.325862+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:09:49.702055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_clean_scrub_stamp":"2024-09-18T08:09:49.702055+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:14:07.222323+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":["5","3"],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.8","version":"283'13315","reported_seq":13428,"reported_epoch":283,"state":"active+clean","last_fresh":"2024-09-18T08:13:34.072219+0000","last_change":"2024-09-18T08:11:09.291403+0000","last_active":"2024-09-18T08:13:34.072219+0000","last_peered":"2024-09-18T08:13:34.072219+0000","last_clean":"2024-09-18T08:13:34.072219+0000","last_became_active":"2024-09-18T08:11:09.291038+0000","last_became_peered":"2024-09-18T08:11:09.291038+0000","last_unstale":"2024-09-18T08:13:34.072219+0000","last_undegraded":"2024-09-18T08:13:34.072219+0000","last_fullsized":"2024-09-18T08:13:34.072219+0000","mapping_epoch":232,"log_start":"267'8951","ondisk_log_start":"267'8951","created":232,"last_epoch_clean":233,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:11:08.262286+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:11:08.262286+0000","last_clean_scrub_stamp":"2024-09-18T08:11:08.262286+0000","objects_scrubbed":0,"log_size":4364,"log_dups_size":0,"ondisk_log_size":4364,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:13:46.024564+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":295763968,"num_objects":4513,"num_object_clones":0,"num_object_copies":13539,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4513,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13315,"num_write_kb":570496,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.5","version":"271'33","reported_seq":330,"reported_epoch":282,"state":"active+clean","last_fresh":"2024-09-18T08:13:31.648687+0000","last_change":"2024-09-18T08:13:28.627088+0000","last_active":"2024-09-18T08:13:31.648687+0000","last_peered":"2024-09-18T08:13:31.648687+0000","last_clean":"2024-09-18T08:13:31.648687+0000","last_became_active":"2024-09-18T08:08:18.547555+0000","last_became_peered":"2024-09-18T08:08:18.547555+0000","last_unstale":"2024-09-18T08:13:31.648687+0000","last_undegraded":"2024-09-18T08:13:31.648687+0000","last_fullsized":"2024-09-18T08:13:31.648687+0000","mapping_epoch":181,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":182,"parent":"0.0","parent_split_bits":0,"last_scrub":"271'33","last_scrub_stamp":"2024-09-18T08:13:28.626987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_clean_scrub_stamp":"2024-09-18T08:13:28.626987+0000","objects_scrubbed":8,"log_size":33,"log_dups_size":0,"ondisk_log_size":33,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:14:38.571943+0000","scrub_duration":24,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":42,"num_read_kb":32,"num_write":25,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":154,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,7],"acting":[4,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":489,"reported_epoch":283,"state":"active+clean","last_fresh":"2024-09-18T08:13:33.654168+0000","last_change":"2024-09-18T08:09:45.572278+0000","last_active":"2024-09-18T08:13:33.654168+0000","last_peered":"2024-09-18T08:13:33.654168+0000","last_clean":"2024-09-18T08:13:33.654168+0000","last_became_active":"2024-09-18T08:06:24.133425+0000","last_became_peered":"2024-09-18T08:06:24.133425+0000","last_unstale":"2024-09-18T08:13:33.654168+0000","last_undegraded":"2024-09-18T08:13:33.654168+0000","last_fullsized":"2024-09-18T08:13:33.654168+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:09:45.572201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:46.860092+0000","last_clean_scrub_stamp":"2024-09-18T08:09:45.572201+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:14:26.723239+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"74'79","reported_seq":576,"reported_epoch":282,"state":"active+clean","last_fresh":"2024-09-18T08:13:31.644463+0000","last_change":"2024-09-18T08:11:34.834467+0000","last_active":"2024-09-18T08:13:31.644463+0000","last_peered":"2024-09-18T08:13:31.644463+0000","last_clean":"2024-09-18T08:13:31.644463+0000","last_became_active":"2024-09-18T08:07:24.376931+0000","last_became_peered":"2024-09-18T08:07:24.376931+0000","last_unstale":"2024-09-18T08:13:31.644463+0000","last_undegraded":"2024-09-18T08:13:31.644463+0000","last_fullsized":"2024-09-18T08:13:31.644463+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":153,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'79","last_scrub_stamp":"2024-09-18T08:11:34.834393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:00:11.893742+0000","last_clean_scrub_stamp":"2024-09-18T08:11:34.834393+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:13:44.792838+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":318,"reported_epoch":283,"state":"active+clean","last_fresh":"2024-09-18T08:13:33.651935+0000","last_change":"2024-09-18T08:12:56.304598+0000","last_active":"2024-09-18T08:13:33.651935+0000","last_peered":"2024-09-18T08:13:33.651935+0000","last_clean":"2024-09-18T08:13:33.651935+0000","last_became_active":"2024-09-18T08:06:46.141046+0000","last_became_peered":"2024-09-18T08:06:46.141046+0000","last_unstale":"2024-09-18T08:13:33.651935+0000","last_undegraded":"2024-09-18T08:13:33.651935+0000","last_fullsized":"2024-09-18T08:13:33.651935+0000","mapping_epoch":143,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":144,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:12:56.304527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:50.886830+0000","last_clean_scrub_stamp":"2024-09-18T08:12:56.304527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:14:25.953150+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":482,"reported_epoch":282,"state":"active+clean","last_fresh":"2024-09-18T08:13:31.643563+0000","last_change":"2024-09-18T08:12:05.886343+0000","last_active":"2024-09-18T08:13:31.643563+0000","last_peered":"2024-09-18T08:13:31.643563+0000","last_clean":"2024-09-18T08:13:31.643563+0000","last_became_active":"2024-09-18T08:08:18.548233+0000","last_became_peered":"2024-09-18T08:08:18.548233+0000","last_unstale":"2024-09-18T08:13:31.643563+0000","last_undegraded":"2024-09-18T08:13:31.643563+0000","last_fullsized":"2024-09-18T08:13:31.643563+0000","mapping_epoch":181,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":182,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:12:05.886281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_clean_scrub_stamp":"2024-09-18T08:12:05.886281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.3","version":"282'13498","reported_seq":13612,"reported_epoch":282,"state":"active+undersized+degraded","last_fresh":"2024-09-18T08:13:33.426417+0000","last_change":"2024-09-18T08:11:42.332945+0000","last_active":"2024-09-18T08:13:33.426417+0000","last_peered":"2024-09-18T08:13:33.426417+0000","last_clean":"2024-09-18T08:11:41.283527+0000","last_became_active":"2024-09-18T08:11:42.332945+0000","last_became_peered":"2024-09-18T08:11:42.332945+0000","last_unstale":"2024-09-18T08:13:33.426417+0000","last_undegraded":"2024-09-18T08:11:42.322363+0000","last_fullsized":"2024-09-18T08:11:42.322217+0000","mapping_epoch":244,"log_start":"266'8800","ondisk_log_start":"266'8800","created":232,"last_epoch_clean":233,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:11:08.262286+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:11:08.262286+0000","last_clean_scrub_stamp":"2024-09-18T08:11:08.262286+0000","objects_scrubbed":0,"log_size":4698,"log_dups_size":0,"ondisk_log_size":4698,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:14:23.852876+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":303038464,"num_objects":4624,"num_object_clones":0,"num_object_copies":13872,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4624,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4624,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13498,"num_write_kb":579904,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":["7","3"],"object_location_counts":[{"shards":"3,7","objects":4624}],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.e","version":"271'19","reported_seq":250,"reported_epoch":282,"state":"active+undersized+degraded","last_fresh":"2024-09-18T08:13:31.651415+0000","last_change":"2024-09-18T08:11:42.337720+0000","last_active":"2024-09-18T08:13:31.651415+0000","last_peered":"2024-09-18T08:13:31.651415+0000","last_clean":"2024-09-18T08:11:40.307924+0000","last_became_active":"2024-09-18T08:11:42.337720+0000","last_became_peered":"2024-09-18T08:11:42.337720+0000","last_unstale":"2024-09-18T08:13:31.651415+0000","last_undegraded":"2024-09-18T08:11:42.327549+0000","last_fullsized":"2024-09-18T08:11:42.327437+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":182,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'11","last_scrub_stamp":"2024-09-18T08:10:53.406574+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_clean_scrub_stamp":"2024-09-18T08:10:53.406574+0000","objects_scrubbed":5,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:13:33.725845+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":5,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":31,"num_read_kb":24,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":["4","6"],"object_location_counts":[{"shards":"4,6","objects":5}],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":230,"reported_epoch":282,"state":"active+clean","last_fresh":"2024-09-18T08:13:31.647840+0000","last_change":"2024-09-18T08:11:59.458276+0000","last_active":"2024-09-18T08:13:31.647840+0000","last_peered":"2024-09-18T08:13:31.647840+0000","last_clean":"2024-09-18T08:13:31.647840+0000","last_became_active":"2024-09-18T08:08:18.524229+0000","last_became_peered":"2024-09-18T08:08:18.524229+0000","last_unstale":"2024-09-18T08:13:31.647840+0000","last_undegraded":"2024-09-18T08:13:31.647840+0000","last_fullsized":"2024-09-18T08:13:31.647840+0000","mapping_epoch":181,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":182,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:11:59.458232+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:46.860092+0000","last_clean_scrub_stamp":"2024-09-18T08:11:59.458232+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":499,"reported_epoch":282,"state":"active+clean","last_fresh":"2024-09-18T08:13:31.643383+0000","last_change":"2024-09-18T08:11:28.031016+0000","last_active":"2024-09-18T08:13:31.643383+0000","last_peered":"2024-09-18T08:13:31.643383+0000","last_clean":"2024-09-18T08:13:31.643383+0000","last_became_active":"2024-09-18T08:08:18.547105+0000","last_became_peered":"2024-09-18T08:08:18.547105+0000","last_unstale":"2024-09-18T08:13:31.643383+0000","last_undegraded":"2024-09-18T08:13:31.643383+0000","last_fullsized":"2024-09-18T08:13:31.643383+0000","mapping_epoch":181,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":182,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:11:28.030965+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:50.886830+0000","last_clean_scrub_stamp":"2024-09-18T08:11:28.030965+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":476,"reported_epoch":283,"state":"active+clean","last_fresh":"2024-09-18T08:13:33.653622+0000","last_change":"2024-09-18T08:07:09.197912+0000","last_active":"2024-09-18T08:13:33.653622+0000","last_peered":"2024-09-18T08:13:33.653622+0000","last_clean":"2024-09-18T08:13:33.653622+0000","last_became_active":"2024-09-18T08:06:24.130266+0000","last_became_peered":"2024-09-18T08:06:24.130266+0000","last_unstale":"2024-09-18T08:13:33.653622+0000","last_undegraded":"2024-09-18T08:13:33.653622+0000","last_fullsized":"2024-09-18T08:13:33.653622+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:07:09.197841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_clean_scrub_stamp":"2024-09-18T08:07:09.197841+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:14:13.754695+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.0","version":"283'13186","reported_seq":13302,"reported_epoch":283,"state":"active+undersized+degraded","last_fresh":"2024-09-18T08:13:33.776350+0000","last_change":"2024-09-18T08:11:42.323264+0000","last_active":"2024-09-18T08:13:33.776350+0000","last_peered":"2024-09-18T08:13:33.776350+0000","last_clean":"2024-09-18T08:11:41.291289+0000","last_became_active":"2024-09-18T08:11:42.323264+0000","last_became_peered":"2024-09-18T08:11:42.323264+0000","last_unstale":"2024-09-18T08:13:33.776350+0000","last_undegraded":"2024-09-18T08:11:42.320910+0000","last_fullsized":"2024-09-18T08:11:42.317956+0000","mapping_epoch":244,"log_start":"255'6600","ondisk_log_start":"255'6600","created":232,"last_epoch_clean":233,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:11:08.262286+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:11:08.262286+0000","last_clean_scrub_stamp":"2024-09-18T08:11:08.262286+0000","objects_scrubbed":0,"log_size":6586,"log_dups_size":0,"ondisk_log_size":6586,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:14:22.337245+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":290848768,"num_objects":4438,"num_object_clones":0,"num_object_copies":13314,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4438,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4438,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13186,"num_write_kb":563968,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":["2","6"],"object_location_counts":[{"shards":"2,6","objects":4438}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.8","version":"74'79","reported_seq":299,"reported_epoch":282,"state":"active+clean","last_fresh":"2024-09-18T08:13:31.644110+0000","last_change":"2024-09-18T08:12:00.980995+0000","last_active":"2024-09-18T08:13:31.644110+0000","last_peered":"2024-09-18T08:13:31.644110+0000","last_clean":"2024-09-18T08:13:31.644110+0000","last_became_active":"2024-09-18T08:07:26.407298+0000","last_became_peered":"2024-09-18T08:07:26.407298+0000","last_unstale":"2024-09-18T08:13:31.644110+0000","last_undegraded":"2024-09-18T08:13:31.644110+0000","last_fullsized":"2024-09-18T08:13:31.644110+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":152,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":4,"last_scrub":"74'79","last_scrub_stamp":"2024-09-18T08:12:00.980917+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:00:11.893742+0000","last_clean_scrub_stamp":"2024-09-18T08:12:00.980917+0000","objects_scrubbed":1,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262688,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,3],"acting":[5,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.d","version":"271'31","reported_seq":257,"reported_epoch":282,"state":"active+undersized+degraded","last_fresh":"2024-09-18T08:13:31.648838+0000","last_change":"2024-09-18T08:11:42.336457+0000","last_active":"2024-09-18T08:13:31.648838+0000","last_peered":"2024-09-18T08:13:31.648838+0000","last_clean":"2024-09-18T08:11:40.307800+0000","last_became_active":"2024-09-18T08:11:42.336457+0000","last_became_peered":"2024-09-18T08:11:42.336457+0000","last_unstale":"2024-09-18T08:13:31.648838+0000","last_undegraded":"2024-09-18T08:11:42.322722+0000","last_fullsized":"2024-09-18T08:11:42.322622+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":182,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'17","last_scrub_stamp":"2024-09-18T08:08:09.671248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_clean_scrub_stamp":"2024-09-18T08:08:09.671248+0000","objects_scrubbed":0,"log_size":31,"log_dups_size":0,"ondisk_log_size":31,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:14:02.406292+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":8,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":49,"num_read_kb":39,"num_write":32,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":["4","2"],"object_location_counts":[{"shards":"2,4","objects":8}],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":226,"reported_epoch":282,"state":"active+undersized","last_fresh":"2024-09-18T08:13:31.647503+0000","last_change":"2024-09-18T08:11:42.323936+0000","last_active":"2024-09-18T08:13:31.647503+0000","last_peered":"2024-09-18T08:13:31.647503+0000","last_clean":"2024-09-18T08:11:40.306980+0000","last_became_active":"2024-09-18T08:11:42.323936+0000","last_became_peered":"2024-09-18T08:11:42.323936+0000","last_unstale":"2024-09-18T08:13:31.647503+0000","last_undegraded":"2024-09-18T08:13:31.647503+0000","last_fullsized":"2024-09-18T08:11:42.319023+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":182,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:10:22.243148+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:46.860092+0000","last_clean_scrub_stamp":"2024-09-18T08:10:22.243148+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:14:04.461992+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":["4","6"],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":349,"reported_epoch":282,"state":"active+undersized","last_fresh":"2024-09-18T08:13:31.646887+0000","last_change":"2024-09-18T08:11:42.331646+0000","last_active":"2024-09-18T08:13:31.646887+0000","last_peered":"2024-09-18T08:13:31.646887+0000","last_clean":"2024-09-18T08:08:15.506244+0000","last_became_active":"2024-09-18T08:11:42.331646+0000","last_became_peered":"2024-09-18T08:11:42.331646+0000","last_unstale":"2024-09-18T08:13:31.646887+0000","last_undegraded":"2024-09-18T08:13:31.646887+0000","last_fullsized":"2024-09-18T08:11:42.318456+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":182,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:09:06.610879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:50.886830+0000","last_clean_scrub_stamp":"2024-09-18T08:09:06.610879+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:14:03.450879+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":["4","0"],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":198,"reported_epoch":283,"state":"active+clean","last_fresh":"2024-09-18T08:13:33.649950+0000","last_change":"2024-09-18T08:11:58.657536+0000","last_active":"2024-09-18T08:13:33.649950+0000","last_peered":"2024-09-18T08:13:33.649950+0000","last_clean":"2024-09-18T08:13:33.649950+0000","last_became_active":"2024-09-18T08:08:52.696300+0000","last_became_peered":"2024-09-18T08:08:52.696300+0000","last_unstale":"2024-09-18T08:13:33.649950+0000","last_undegraded":"2024-09-18T08:13:33.649950+0000","last_fullsized":"2024-09-18T08:13:33.649950+0000","mapping_epoch":198,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":199,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:11:58.657473+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_clean_scrub_stamp":"2024-09-18T08:11:58.657473+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:14:26.327515+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.1","version":"283'13324","reported_seq":13438,"reported_epoch":283,"state":"active+clean","last_fresh":"2024-09-18T08:13:34.072013+0000","last_change":"2024-09-18T08:11:09.489038+0000","last_active":"2024-09-18T08:13:34.072013+0000","last_peered":"2024-09-18T08:13:34.072013+0000","last_clean":"2024-09-18T08:13:34.072013+0000","last_became_active":"2024-09-18T08:11:09.488718+0000","last_became_peered":"2024-09-18T08:11:09.488718+0000","last_unstale":"2024-09-18T08:13:34.072013+0000","last_undegraded":"2024-09-18T08:13:34.072013+0000","last_fullsized":"2024-09-18T08:13:34.072013+0000","mapping_epoch":232,"log_start":"267'8950","ondisk_log_start":"267'8950","created":232,"last_epoch_clean":233,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:11:08.262286+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:11:08.262286+0000","last_clean_scrub_stamp":"2024-09-18T08:11:08.262286+0000","objects_scrubbed":0,"log_size":4374,"log_dups_size":0,"ondisk_log_size":4374,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:14:14.716160+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":291897344,"num_objects":4454,"num_object_clones":0,"num_object_copies":13362,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4454,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13324,"num_write_kb":568896,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,7],"acting":[3,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.9","version":"74'79","reported_seq":589,"reported_epoch":282,"state":"active+clean","last_fresh":"2024-09-18T08:13:31.650047+0000","last_change":"2024-09-18T08:09:27.676247+0000","last_active":"2024-09-18T08:13:31.650047+0000","last_peered":"2024-09-18T08:13:31.650047+0000","last_clean":"2024-09-18T08:13:31.650047+0000","last_became_active":"2024-09-18T08:08:52.691492+0000","last_became_peered":"2024-09-18T08:08:52.691492+0000","last_unstale":"2024-09-18T08:13:31.650047+0000","last_undegraded":"2024-09-18T08:13:31.650047+0000","last_fullsized":"2024-09-18T08:13:31.650047+0000","mapping_epoch":198,"log_start":"0'0","ondisk_log_start":"0'0","created":152,"last_epoch_clean":199,"parent":"0.0","parent_split_bits":4,"last_scrub":"74'79","last_scrub_stamp":"2024-09-18T08:09:27.676169+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:00:11.893742+0000","last_clean_scrub_stamp":"2024-09-18T08:09:27.676169+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:13:43.830751+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,2],"acting":[7,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.c","version":"271'20","reported_seq":241,"reported_epoch":282,"state":"active+clean","last_fresh":"2024-09-18T08:13:31.649164+0000","last_change":"2024-09-18T08:08:18.642452+0000","last_active":"2024-09-18T08:13:31.649164+0000","last_peered":"2024-09-18T08:13:31.649164+0000","last_clean":"2024-09-18T08:13:31.649164+0000","last_became_active":"2024-09-18T08:08:18.622980+0000","last_became_peered":"2024-09-18T08:08:18.622980+0000","last_unstale":"2024-09-18T08:13:31.649164+0000","last_undegraded":"2024-09-18T08:13:31.649164+0000","last_fullsized":"2024-09-18T08:13:31.649164+0000","mapping_epoch":181,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":182,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'10","last_scrub_stamp":"2024-09-18T08:05:41.474813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_clean_scrub_stamp":"2024-09-18T08:05:41.474813+0000","objects_scrubbed":0,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:13:45.565099+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":29,"num_read_kb":24,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":496,"reported_epoch":283,"state":"active+clean","last_fresh":"2024-09-18T08:13:33.658064+0000","last_change":"2024-09-18T08:10:57.455046+0000","last_active":"2024-09-18T08:13:33.658064+0000","last_peered":"2024-09-18T08:13:33.658064+0000","last_clean":"2024-09-18T08:13:33.658064+0000","last_became_active":"2024-09-18T08:08:18.542424+0000","last_became_peered":"2024-09-18T08:08:18.542424+0000","last_unstale":"2024-09-18T08:13:33.658064+0000","last_undegraded":"2024-09-18T08:13:33.658064+0000","last_fullsized":"2024-09-18T08:13:33.658064+0000","mapping_epoch":181,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":182,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:10:57.455000+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:46.860092+0000","last_clean_scrub_stamp":"2024-09-18T08:10:57.455000+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:13:50.076444+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.d","version":"69'8","reported_seq":620,"reported_epoch":283,"state":"active+clean","last_fresh":"2024-09-18T08:13:33.649484+0000","last_change":"2024-09-18T08:12:30.371968+0000","last_active":"2024-09-18T08:13:33.649484+0000","last_peered":"2024-09-18T08:13:33.649484+0000","last_clean":"2024-09-18T08:13:33.649484+0000","last_became_active":"2024-09-18T08:02:51.909011+0000","last_became_peered":"2024-09-18T08:02:51.909011+0000","last_unstale":"2024-09-18T08:13:33.649484+0000","last_undegraded":"2024-09-18T08:13:33.649484+0000","last_fullsized":"2024-09-18T08:13:33.649484+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'8","last_scrub_stamp":"2024-09-18T08:12:30.371904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:50.886830+0000","last_clean_scrub_stamp":"2024-09-18T08:12:30.371904+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:13:46.486491+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":233,"reported_epoch":282,"state":"active+undersized","last_fresh":"2024-09-18T08:13:31.651782+0000","last_change":"2024-09-18T08:11:42.337415+0000","last_active":"2024-09-18T08:13:31.651782+0000","last_peered":"2024-09-18T08:13:31.651782+0000","last_clean":"2024-09-18T08:11:40.308045+0000","last_became_active":"2024-09-18T08:11:42.337415+0000","last_became_peered":"2024-09-18T08:11:42.337415+0000","last_unstale":"2024-09-18T08:13:31.651782+0000","last_undegraded":"2024-09-18T08:13:31.651782+0000","last_fullsized":"2024-09-18T08:11:42.322962+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":199,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:11:18.475967+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_clean_scrub_stamp":"2024-09-18T08:11:18.475967+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":["4","2"],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.6","version":"282'13232","reported_seq":13347,"reported_epoch":282,"state":"active+undersized+degraded","last_fresh":"2024-09-18T08:13:33.489607+0000","last_change":"2024-09-18T08:11:42.339935+0000","last_active":"2024-09-18T08:13:33.489607+0000","last_peered":"2024-09-18T08:13:33.489607+0000","last_clean":"2024-09-18T08:11:41.290923+0000","last_became_active":"2024-09-18T08:11:42.339935+0000","last_became_peered":"2024-09-18T08:11:42.339935+0000","last_unstale":"2024-09-18T08:13:33.489607+0000","last_undegraded":"2024-09-18T08:11:42.325251+0000","last_fullsized":"2024-09-18T08:11:42.325122+0000","mapping_epoch":244,"log_start":"264'8300","ondisk_log_start":"264'8300","created":232,"last_epoch_clean":233,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:11:08.262286+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:11:08.262286+0000","last_clean_scrub_stamp":"2024-09-18T08:11:08.262286+0000","objects_scrubbed":0,"log_size":4932,"log_dups_size":0,"ondisk_log_size":4932,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:13:35.300193+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":294518784,"num_objects":4494,"num_object_clones":0,"num_object_copies":13482,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4494,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4494,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13232,"num_write_kb":567232,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":["5","2"],"object_location_counts":[{"shards":"2,5","objects":4494}],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.b","version":"271'15","reported_seq":540,"reported_epoch":283,"state":"active+undersized+degraded","last_fresh":"2024-09-18T08:13:33.656618+0000","last_change":"2024-09-18T08:11:42.333863+0000","last_active":"2024-09-18T08:13:33.656618+0000","last_peered":"2024-09-18T08:13:33.656618+0000","last_clean":"2024-09-18T08:11:40.306474+0000","last_became_active":"2024-09-18T08:11:42.333863+0000","last_became_peered":"2024-09-18T08:11:42.333863+0000","last_unstale":"2024-09-18T08:13:33.656618+0000","last_undegraded":"2024-09-18T08:11:42.319070+0000","last_fullsized":"2024-09-18T08:11:42.318966+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":182,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'9","last_scrub_stamp":"2024-09-18T08:10:47.928596+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_clean_scrub_stamp":"2024-09-18T08:10:47.928596+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":16,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":["0","4"],"object_location_counts":[{"shards":"0,4","objects":4}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":493,"reported_epoch":282,"state":"active+clean","last_fresh":"2024-09-18T08:13:31.643726+0000","last_change":"2024-09-18T08:11:22.076505+0000","last_active":"2024-09-18T08:13:31.643726+0000","last_peered":"2024-09-18T08:13:31.643726+0000","last_clean":"2024-09-18T08:13:31.643726+0000","last_became_active":"2024-09-18T08:06:24.135005+0000","last_became_peered":"2024-09-18T08:06:24.135005+0000","last_unstale":"2024-09-18T08:13:31.643726+0000","last_undegraded":"2024-09-18T08:13:31.643726+0000","last_fullsized":"2024-09-18T08:13:31.643726+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:11:22.076452+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:46.860092+0000","last_clean_scrub_stamp":"2024-09-18T08:11:22.076452+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:14:22.230059+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":488,"reported_epoch":283,"state":"active+clean","last_fresh":"2024-09-18T08:13:33.649726+0000","last_change":"2024-09-18T08:08:18.509651+0000","last_active":"2024-09-18T08:13:33.649726+0000","last_peered":"2024-09-18T08:13:33.649726+0000","last_clean":"2024-09-18T08:13:33.649726+0000","last_became_active":"2024-09-18T08:08:18.509247+0000","last_became_peered":"2024-09-18T08:08:18.509247+0000","last_unstale":"2024-09-18T08:13:33.649726+0000","last_undegraded":"2024-09-18T08:13:33.649726+0000","last_fullsized":"2024-09-18T08:13:33.649726+0000","mapping_epoch":181,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":182,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:07:10.728679+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:50.886830+0000","last_clean_scrub_stamp":"2024-09-18T08:07:10.728679+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:13:46.290405+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,3],"acting":[2,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":500,"reported_epoch":283,"state":"active+clean","last_fresh":"2024-09-18T08:13:33.656149+0000","last_change":"2024-09-18T08:12:38.703021+0000","last_active":"2024-09-18T08:13:33.656149+0000","last_peered":"2024-09-18T08:13:33.656149+0000","last_clean":"2024-09-18T08:13:33.656149+0000","last_became_active":"2024-09-18T08:08:52.697337+0000","last_became_peered":"2024-09-18T08:08:52.697337+0000","last_unstale":"2024-09-18T08:13:33.656149+0000","last_undegraded":"2024-09-18T08:13:33.656149+0000","last_fullsized":"2024-09-18T08:13:33.656149+0000","mapping_epoch":198,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":199,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:12:38.702798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_clean_scrub_stamp":"2024-09-18T08:12:38.702798+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:13:41.813263+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.7","version":"282'13027","reported_seq":13168,"reported_epoch":282,"state":"active+clean","last_fresh":"2024-09-18T08:13:33.507260+0000","last_change":"2024-09-18T08:12:41.750258+0000","last_active":"2024-09-18T08:13:33.507260+0000","last_peered":"2024-09-18T08:13:33.507260+0000","last_clean":"2024-09-18T08:13:33.507260+0000","last_became_active":"2024-09-18T08:11:09.288659+0000","last_became_peered":"2024-09-18T08:11:09.288659+0000","last_unstale":"2024-09-18T08:13:33.507260+0000","last_undegraded":"2024-09-18T08:13:33.507260+0000","last_fullsized":"2024-09-18T08:13:33.507260+0000","mapping_epoch":232,"log_start":"264'8100","ondisk_log_start":"264'8100","created":232,"last_epoch_clean":233,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:11:08.262286+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:11:08.262286+0000","last_clean_scrub_stamp":"2024-09-18T08:11:08.262286+0000","objects_scrubbed":2599,"log_size":4927,"log_dups_size":0,"ondisk_log_size":4927,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:14:20.208367+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":289996800,"num_objects":4425,"num_object_clones":0,"num_object_copies":13275,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4425,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13027,"num_write_kb":558464,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.a","version":"271'35","reported_seq":266,"reported_epoch":282,"state":"active+undersized+degraded","last_fresh":"2024-09-18T08:13:31.650200+0000","last_change":"2024-09-18T08:11:42.325717+0000","last_active":"2024-09-18T08:13:31.650200+0000","last_peered":"2024-09-18T08:13:31.650200+0000","last_clean":"2024-09-18T08:06:21.509422+0000","last_became_active":"2024-09-18T08:11:42.325717+0000","last_became_peered":"2024-09-18T08:11:42.325717+0000","last_unstale":"2024-09-18T08:13:31.650200+0000","last_undegraded":"2024-09-18T08:11:42.322210+0000","last_fullsized":"2024-09-18T08:11:42.322097+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'19","last_scrub_stamp":"2024-09-18T08:10:19.603091+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_clean_scrub_stamp":"2024-09-18T08:10:19.603091+0000","objects_scrubbed":0,"log_size":35,"log_dups_size":0,"ondisk_log_size":35,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:13:45.051164+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":9,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":55,"num_read_kb":44,"num_write":36,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":["6","7"],"object_location_counts":[{"shards":"6,7","objects":9}],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":490,"reported_epoch":282,"state":"active+clean","last_fresh":"2024-09-18T08:13:31.644541+0000","last_change":"2024-09-18T08:12:31.205529+0000","last_active":"2024-09-18T08:13:31.644541+0000","last_peered":"2024-09-18T08:13:31.644541+0000","last_clean":"2024-09-18T08:13:31.644541+0000","last_became_active":"2024-09-18T08:06:46.171298+0000","last_became_peered":"2024-09-18T08:06:46.171298+0000","last_unstale":"2024-09-18T08:13:31.644541+0000","last_undegraded":"2024-09-18T08:13:31.644541+0000","last_fullsized":"2024-09-18T08:13:31.644541+0000","mapping_epoch":143,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":144,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:12:31.205468+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:46.860092+0000","last_clean_scrub_stamp":"2024-09-18T08:12:31.205468+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":190,"reported_epoch":283,"state":"active+clean","last_fresh":"2024-09-18T08:13:33.649406+0000","last_change":"2024-09-18T08:10:41.682366+0000","last_active":"2024-09-18T08:13:33.649406+0000","last_peered":"2024-09-18T08:13:33.649406+0000","last_clean":"2024-09-18T08:13:33.649406+0000","last_became_active":"2024-09-18T08:08:52.697932+0000","last_became_peered":"2024-09-18T08:08:52.697932+0000","last_unstale":"2024-09-18T08:13:33.649406+0000","last_undegraded":"2024-09-18T08:13:33.649406+0000","last_fullsized":"2024-09-18T08:13:33.649406+0000","mapping_epoch":198,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":199,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:10:41.682297+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:50.886830+0000","last_clean_scrub_stamp":"2024-09-18T08:10:41.682297+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:14:19.387756+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":310,"reported_epoch":283,"state":"active+clean","last_fresh":"2024-09-18T08:13:33.649456+0000","last_change":"2024-09-18T08:11:31.707207+0000","last_active":"2024-09-18T08:13:33.649456+0000","last_peered":"2024-09-18T08:13:33.649456+0000","last_clean":"2024-09-18T08:13:33.649456+0000","last_became_active":"2024-09-18T08:06:46.137872+0000","last_became_peered":"2024-09-18T08:06:46.137872+0000","last_unstale":"2024-09-18T08:13:33.649456+0000","last_undegraded":"2024-09-18T08:13:33.649456+0000","last_fullsized":"2024-09-18T08:13:33.649456+0000","mapping_epoch":143,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":144,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:11:31.707130+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_clean_scrub_stamp":"2024-09-18T08:11:31.707130+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:13:52.311573+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,3],"acting":[2,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.4","version":"282'13262","reported_seq":13378,"reported_epoch":282,"state":"active+undersized+degraded","last_fresh":"2024-09-18T08:13:33.332926+0000","last_change":"2024-09-18T08:11:42.349543+0000","last_active":"2024-09-18T08:13:33.332926+0000","last_peered":"2024-09-18T08:13:33.332926+0000","last_clean":"2024-09-18T08:11:41.314761+0000","last_became_active":"2024-09-18T08:11:42.329000+0000","last_became_peered":"2024-09-18T08:11:42.329000+0000","last_unstale":"2024-09-18T08:13:33.332926+0000","last_undegraded":"2024-09-18T08:11:42.318915+0000","last_fullsized":"2024-09-18T08:11:42.318397+0000","mapping_epoch":244,"log_start":"265'8500","ondisk_log_start":"265'8500","created":232,"last_epoch_clean":233,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:11:08.262286+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:11:08.262286+0000","last_clean_scrub_stamp":"2024-09-18T08:11:08.262286+0000","objects_scrubbed":0,"log_size":4762,"log_dups_size":0,"ondisk_log_size":4762,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":292421632,"num_objects":4462,"num_object_clones":0,"num_object_copies":13386,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4462,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4462,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13262,"num_write_kb":567168,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":65536,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":["4","3"],"object_location_counts":[{"shards":"3,4","objects":4462}],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.9","version":"271'18","reported_seq":262,"reported_epoch":282,"state":"active+clean","last_fresh":"2024-09-18T08:13:31.647391+0000","last_change":"2024-09-18T08:11:27.607131+0000","last_active":"2024-09-18T08:13:31.647391+0000","last_peered":"2024-09-18T08:13:31.647391+0000","last_clean":"2024-09-18T08:13:31.647391+0000","last_became_active":"2024-09-18T08:10:18.362984+0000","last_became_peered":"2024-09-18T08:10:18.362984+0000","last_unstale":"2024-09-18T08:13:31.647391+0000","last_undegraded":"2024-09-18T08:13:31.647391+0000","last_fullsized":"2024-09-18T08:13:31.647391+0000","mapping_epoch":221,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":222,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'12","last_scrub_stamp":"2024-09-18T08:11:27.607049+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_clean_scrub_stamp":"2024-09-18T08:11:27.607049+0000","objects_scrubbed":5,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":33,"num_read_kb":24,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":660,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,3],"acting":[4,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":513,"reported_epoch":282,"state":"active+undersized","last_fresh":"2024-09-18T08:13:31.647345+0000","last_change":"2024-09-18T08:11:42.338535+0000","last_active":"2024-09-18T08:13:31.647345+0000","last_peered":"2024-09-18T08:13:31.647345+0000","last_clean":"2024-09-18T08:11:40.309174+0000","last_became_active":"2024-09-18T08:11:42.338535+0000","last_became_peered":"2024-09-18T08:11:42.338535+0000","last_unstale":"2024-09-18T08:13:31.647345+0000","last_undegraded":"2024-09-18T08:13:31.647345+0000","last_fullsized":"2024-09-18T08:11:42.324265+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":182,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:10:57.579577+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:46.860092+0000","last_clean_scrub_stamp":"2024-09-18T08:10:57.579577+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:14:24.872704+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":["7","4"],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":478,"reported_epoch":283,"state":"active+undersized","last_fresh":"2024-09-18T08:13:33.649834+0000","last_change":"2024-09-18T08:11:42.325714+0000","last_active":"2024-09-18T08:13:33.649834+0000","last_peered":"2024-09-18T08:13:33.649834+0000","last_clean":"2024-09-18T08:11:40.310260+0000","last_became_active":"2024-09-18T08:11:42.325714+0000","last_became_peered":"2024-09-18T08:11:42.325714+0000","last_unstale":"2024-09-18T08:13:33.649834+0000","last_undegraded":"2024-09-18T08:13:33.649834+0000","last_fullsized":"2024-09-18T08:11:42.318417+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:10:47.799791+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:50.886830+0000","last_clean_scrub_stamp":"2024-09-18T08:10:47.799791+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:14:20.377540+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":["2","0"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":483,"reported_epoch":283,"state":"active+undersized","last_fresh":"2024-09-18T08:13:33.658164+0000","last_change":"2024-09-18T08:11:42.331389+0000","last_active":"2024-09-18T08:13:33.658164+0000","last_peered":"2024-09-18T08:13:33.658164+0000","last_clean":"2024-09-18T08:11:40.314736+0000","last_became_active":"2024-09-18T08:11:42.331389+0000","last_became_peered":"2024-09-18T08:11:42.331389+0000","last_unstale":"2024-09-18T08:13:33.658164+0000","last_undegraded":"2024-09-18T08:13:33.658164+0000","last_fullsized":"2024-09-18T08:11:42.324129+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:11:21.372405+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_clean_scrub_stamp":"2024-09-18T08:11:21.372405+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:14:30.719269+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":["3","7"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.5","version":"283'13250","reported_seq":13364,"reported_epoch":283,"state":"active+clean","last_fresh":"2024-09-18T08:13:34.045421+0000","last_change":"2024-09-18T08:11:09.292021+0000","last_active":"2024-09-18T08:13:34.045421+0000","last_peered":"2024-09-18T08:13:34.045421+0000","last_clean":"2024-09-18T08:13:34.045421+0000","last_became_active":"2024-09-18T08:11:09.291723+0000","last_became_peered":"2024-09-18T08:11:09.291723+0000","last_unstale":"2024-09-18T08:13:34.045421+0000","last_undegraded":"2024-09-18T08:13:34.045421+0000","last_fullsized":"2024-09-18T08:13:34.045421+0000","mapping_epoch":232,"log_start":"259'7500","ondisk_log_start":"259'7500","created":232,"last_epoch_clean":233,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:11:08.262286+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:11:08.262286+0000","last_clean_scrub_stamp":"2024-09-18T08:11:08.262286+0000","objects_scrubbed":0,"log_size":5750,"log_dups_size":0,"ondisk_log_size":5750,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:13:53.751312+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":292028416,"num_objects":4456,"num_object_clones":0,"num_object_copies":13368,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4456,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13250,"num_write_kb":566592,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.8","version":"271'27","reported_seq":548,"reported_epoch":282,"state":"active+clean","last_fresh":"2024-09-18T08:13:31.644275+0000","last_change":"2024-09-18T08:06:24.162625+0000","last_active":"2024-09-18T08:13:31.644275+0000","last_peered":"2024-09-18T08:13:31.644275+0000","last_clean":"2024-09-18T08:13:31.644275+0000","last_became_active":"2024-09-18T08:06:24.138259+0000","last_became_peered":"2024-09-18T08:06:24.138259+0000","last_unstale":"2024-09-18T08:13:31.644275+0000","last_undegraded":"2024-09-18T08:13:31.644275+0000","last_fullsized":"2024-09-18T08:13:31.644275+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'15","last_scrub_stamp":"2024-09-18T08:04:26.859044+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_clean_scrub_stamp":"2024-09-18T08:04:26.859044+0000","objects_scrubbed":7,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:13:46.333773+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":44,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.f","version":"62'2","reported_seq":525,"reported_epoch":282,"state":"active+clean","last_fresh":"2024-09-18T08:13:31.649576+0000","last_change":"2024-09-18T08:10:53.580490+0000","last_active":"2024-09-18T08:13:31.649576+0000","last_peered":"2024-09-18T08:13:31.649576+0000","last_clean":"2024-09-18T08:13:31.649576+0000","last_became_active":"2024-09-18T08:08:18.524593+0000","last_became_peered":"2024-09-18T08:08:18.524593+0000","last_unstale":"2024-09-18T08:13:31.649576+0000","last_undegraded":"2024-09-18T08:13:31.649576+0000","last_fullsized":"2024-09-18T08:13:31.649576+0000","mapping_epoch":181,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":182,"parent":"0.0","parent_split_bits":0,"last_scrub":"62'2","last_scrub_stamp":"2024-09-18T08:10:53.580444+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:46.860092+0000","last_clean_scrub_stamp":"2024-09-18T08:10:53.580444+0000","objects_scrubbed":2,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:14:18.794780+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":4,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":184,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.9","version":"182'12","reported_seq":665,"reported_epoch":282,"state":"active+clean","last_fresh":"2024-09-18T08:13:31.649944+0000","last_change":"2024-09-18T08:09:46.742515+0000","last_active":"2024-09-18T08:13:31.649944+0000","last_peered":"2024-09-18T08:13:31.649944+0000","last_clean":"2024-09-18T08:13:31.649944+0000","last_became_active":"2024-09-18T08:08:18.524858+0000","last_became_peered":"2024-09-18T08:08:18.524858+0000","last_unstale":"2024-09-18T08:13:31.649944+0000","last_undegraded":"2024-09-18T08:13:31.649944+0000","last_fullsized":"2024-09-18T08:13:31.649944+0000","mapping_epoch":181,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":182,"parent":"0.0","parent_split_bits":0,"last_scrub":"182'12","last_scrub_stamp":"2024-09-18T08:09:46.742442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:50.886830+0000","last_clean_scrub_stamp":"2024-09-18T08:09:46.742442+0000","objects_scrubbed":1,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:14:14.803415+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":481,"reported_epoch":282,"state":"active+clean","last_fresh":"2024-09-18T08:13:31.645875+0000","last_change":"2024-09-18T08:11:23.050828+0000","last_active":"2024-09-18T08:13:31.645875+0000","last_peered":"2024-09-18T08:13:31.645875+0000","last_clean":"2024-09-18T08:13:31.645875+0000","last_became_active":"2024-09-18T08:06:24.133119+0000","last_became_peered":"2024-09-18T08:06:24.133119+0000","last_unstale":"2024-09-18T08:13:31.645875+0000","last_undegraded":"2024-09-18T08:13:31.645875+0000","last_fullsized":"2024-09-18T08:13:31.645875+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:11:23.050755+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_clean_scrub_stamp":"2024-09-18T08:11:23.050755+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:14:24.216573+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":407,"reported_epoch":283,"state":"active+clean","last_fresh":"2024-09-18T08:13:33.656393+0000","last_change":"2024-09-18T08:13:06.860401+0000","last_active":"2024-09-18T08:13:33.656393+0000","last_peered":"2024-09-18T08:13:33.656393+0000","last_clean":"2024-09-18T08:13:33.656393+0000","last_became_active":"2024-09-18T08:06:24.133644+0000","last_became_peered":"2024-09-18T08:06:24.133644+0000","last_unstale":"2024-09-18T08:13:33.656393+0000","last_undegraded":"2024-09-18T08:13:33.656393+0000","last_fullsized":"2024-09-18T08:13:33.656393+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:13:06.860320+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:46.860092+0000","last_clean_scrub_stamp":"2024-09-18T08:13:06.860320+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:14:07.559037+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.17","version":"271'12","reported_seq":508,"reported_epoch":283,"state":"active+undersized+degraded","last_fresh":"2024-09-18T08:13:33.652192+0000","last_change":"2024-09-18T08:11:42.331376+0000","last_active":"2024-09-18T08:13:33.652192+0000","last_peered":"2024-09-18T08:13:33.652192+0000","last_clean":"2024-09-18T08:11:40.317526+0000","last_became_active":"2024-09-18T08:11:42.331376+0000","last_became_peered":"2024-09-18T08:11:42.331376+0000","last_unstale":"2024-09-18T08:13:33.652192+0000","last_undegraded":"2024-09-18T08:11:42.327618+0000","last_fullsized":"2024-09-18T08:11:42.327485+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'6","last_scrub_stamp":"2024-09-18T08:10:15.515914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_clean_scrub_stamp":"2024-09-18T08:10:15.515914+0000","objects_scrubbed":3,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:13:57.014260+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":18,"num_read_kb":15,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":["3","7"],"object_location_counts":[{"shards":"3,7","objects":3}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":475,"reported_epoch":282,"state":"active+undersized","last_fresh":"2024-09-18T08:13:31.644567+0000","last_change":"2024-09-18T08:11:42.331518+0000","last_active":"2024-09-18T08:13:31.644567+0000","last_peered":"2024-09-18T08:13:31.644567+0000","last_clean":"2024-09-18T08:11:40.302611+0000","last_became_active":"2024-09-18T08:11:42.331518+0000","last_became_peered":"2024-09-18T08:11:42.331518+0000","last_unstale":"2024-09-18T08:13:31.644567+0000","last_undegraded":"2024-09-18T08:13:31.644567+0000","last_fullsized":"2024-09-18T08:11:42.328308+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:10:49.169375+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:50.886830+0000","last_clean_scrub_stamp":"2024-09-18T08:10:49.169375+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:13:34.218350+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":["5","3"],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":512,"reported_epoch":282,"state":"active+clean","last_fresh":"2024-09-18T08:13:31.649093+0000","last_change":"2024-09-18T08:12:54.661635+0000","last_active":"2024-09-18T08:13:31.649093+0000","last_peered":"2024-09-18T08:13:31.649093+0000","last_clean":"2024-09-18T08:13:31.649093+0000","last_became_active":"2024-09-18T08:08:18.525260+0000","last_became_peered":"2024-09-18T08:08:18.525260+0000","last_unstale":"2024-09-18T08:13:31.649093+0000","last_undegraded":"2024-09-18T08:13:31.649093+0000","last_fullsized":"2024-09-18T08:13:31.649093+0000","mapping_epoch":181,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":182,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:12:54.661578+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_clean_scrub_stamp":"2024-09-18T08:12:54.661578+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:14:04.552010+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"271'15","reported_seq":502,"reported_epoch":283,"state":"active+clean","last_fresh":"2024-09-18T08:13:33.656980+0000","last_change":"2024-09-18T08:09:44.116395+0000","last_active":"2024-09-18T08:13:33.656980+0000","last_peered":"2024-09-18T08:13:33.656980+0000","last_clean":"2024-09-18T08:13:33.656980+0000","last_became_active":"2024-09-18T08:02:49.891567+0000","last_became_peered":"2024-09-18T08:02:49.891567+0000","last_unstale":"2024-09-18T08:13:33.656980+0000","last_undegraded":"2024-09-18T08:13:33.656980+0000","last_fullsized":"2024-09-18T08:13:33.656980+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'9","last_scrub_stamp":"2024-09-18T08:09:44.116308+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_clean_scrub_stamp":"2024-09-18T08:09:44.116308+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:14:20.869035+0000","scrub_duration":43,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":230,"reported_epoch":282,"state":"active+clean","last_fresh":"2024-09-18T08:13:31.647597+0000","last_change":"2024-09-18T08:12:03.433488+0000","last_active":"2024-09-18T08:13:31.647597+0000","last_peered":"2024-09-18T08:13:31.647597+0000","last_clean":"2024-09-18T08:13:31.647597+0000","last_became_active":"2024-09-18T08:08:18.545953+0000","last_became_peered":"2024-09-18T08:08:18.545953+0000","last_unstale":"2024-09-18T08:13:31.647597+0000","last_undegraded":"2024-09-18T08:13:31.647597+0000","last_fullsized":"2024-09-18T08:13:31.647597+0000","mapping_epoch":181,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":182,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:12:03.433407+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:46.860092+0000","last_clean_scrub_stamp":"2024-09-18T08:12:03.433407+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":478,"reported_epoch":283,"state":"active+undersized","last_fresh":"2024-09-18T08:13:33.651645+0000","last_change":"2024-09-18T08:11:42.328676+0000","last_active":"2024-09-18T08:13:33.651645+0000","last_peered":"2024-09-18T08:13:33.651645+0000","last_clean":"2024-09-18T08:11:40.317184+0000","last_became_active":"2024-09-18T08:11:42.328676+0000","last_became_peered":"2024-09-18T08:11:42.328676+0000","last_unstale":"2024-09-18T08:13:33.651645+0000","last_undegraded":"2024-09-18T08:13:33.651645+0000","last_fullsized":"2024-09-18T08:11:42.326031+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:11:17.424701+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:50.886830+0000","last_clean_scrub_stamp":"2024-09-18T08:11:17.424701+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:14:01.930131+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":["3","7"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":485,"reported_epoch":283,"state":"active+clean","last_fresh":"2024-09-18T08:13:33.651843+0000","last_change":"2024-09-18T08:10:42.731922+0000","last_active":"2024-09-18T08:13:33.651843+0000","last_peered":"2024-09-18T08:13:33.651843+0000","last_clean":"2024-09-18T08:13:33.651843+0000","last_became_active":"2024-09-18T08:08:18.511019+0000","last_became_peered":"2024-09-18T08:08:18.511019+0000","last_unstale":"2024-09-18T08:13:33.651843+0000","last_undegraded":"2024-09-18T08:13:33.651843+0000","last_fullsized":"2024-09-18T08:13:33.651843+0000","mapping_epoch":181,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":182,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:10:42.731845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_clean_scrub_stamp":"2024-09-18T08:10:42.731845+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:14:21.375913+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.15","version":"271'15","reported_seq":500,"reported_epoch":282,"state":"active+clean","last_fresh":"2024-09-18T08:13:31.644663+0000","last_change":"2024-09-18T08:12:58.225479+0000","last_active":"2024-09-18T08:13:31.644663+0000","last_peered":"2024-09-18T08:13:31.644663+0000","last_clean":"2024-09-18T08:13:31.644663+0000","last_became_active":"2024-09-18T08:02:49.890497+0000","last_became_peered":"2024-09-18T08:02:49.890497+0000","last_unstale":"2024-09-18T08:13:31.644663+0000","last_undegraded":"2024-09-18T08:13:31.644663+0000","last_fullsized":"2024-09-18T08:13:31.644663+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"271'15","last_scrub_stamp":"2024-09-18T08:12:58.225295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_clean_scrub_stamp":"2024-09-18T08:12:58.225295+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:14:10.026769+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":489,"reported_epoch":283,"state":"active+clean","last_fresh":"2024-09-18T08:13:33.656459+0000","last_change":"2024-09-18T08:09:46.035501+0000","last_active":"2024-09-18T08:13:33.656459+0000","last_peered":"2024-09-18T08:13:33.656459+0000","last_clean":"2024-09-18T08:13:33.656459+0000","last_became_active":"2024-09-18T08:02:47.875983+0000","last_became_peered":"2024-09-18T08:02:47.875983+0000","last_unstale":"2024-09-18T08:13:33.656459+0000","last_undegraded":"2024-09-18T08:13:33.656459+0000","last_fullsized":"2024-09-18T08:13:33.656459+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:09:46.035332+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:46.860092+0000","last_clean_scrub_stamp":"2024-09-18T08:09:46.035332+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:14:14.822467+0000","scrub_duration":18,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.14","version":"199'12","reported_seq":652,"reported_epoch":283,"state":"active+clean","last_fresh":"2024-09-18T08:13:33.653724+0000","last_change":"2024-09-18T08:13:31.695412+0000","last_active":"2024-09-18T08:13:33.653724+0000","last_peered":"2024-09-18T08:13:33.653724+0000","last_clean":"2024-09-18T08:13:33.653724+0000","last_became_active":"2024-09-18T08:08:52.696201+0000","last_became_peered":"2024-09-18T08:08:52.696201+0000","last_unstale":"2024-09-18T08:13:33.653724+0000","last_undegraded":"2024-09-18T08:13:33.653724+0000","last_fullsized":"2024-09-18T08:13:33.653724+0000","mapping_epoch":198,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":199,"parent":"0.0","parent_split_bits":0,"last_scrub":"199'12","last_scrub_stamp":"2024-09-18T08:13:31.695354+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:50.886830+0000","last_clean_scrub_stamp":"2024-09-18T08:13:31.695354+0000","objects_scrubbed":1,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:14:47.423207+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":222,"reported_epoch":282,"state":"active+clean","last_fresh":"2024-09-18T08:13:31.650691+0000","last_change":"2024-09-18T08:12:06.521847+0000","last_active":"2024-09-18T08:13:31.650691+0000","last_peered":"2024-09-18T08:13:31.650691+0000","last_clean":"2024-09-18T08:13:31.650691+0000","last_became_active":"2024-09-18T08:08:18.522483+0000","last_became_peered":"2024-09-18T08:08:18.522483+0000","last_unstale":"2024-09-18T08:13:31.650691+0000","last_undegraded":"2024-09-18T08:13:31.650691+0000","last_fullsized":"2024-09-18T08:13:31.650691+0000","mapping_epoch":181,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":182,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:12:06.521774+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_clean_scrub_stamp":"2024-09-18T08:12:06.521774+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.14","version":"271'20","reported_seq":527,"reported_epoch":283,"state":"active+clean","last_fresh":"2024-09-18T08:13:33.654126+0000","last_change":"2024-09-18T08:12:39.946516+0000","last_active":"2024-09-18T08:13:33.654126+0000","last_peered":"2024-09-18T08:13:33.654126+0000","last_clean":"2024-09-18T08:13:33.654126+0000","last_became_active":"2024-09-18T08:04:18.150968+0000","last_became_peered":"2024-09-18T08:04:18.150968+0000","last_unstale":"2024-09-18T08:13:33.654126+0000","last_undegraded":"2024-09-18T08:13:33.654126+0000","last_fullsized":"2024-09-18T08:13:33.654126+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'10","last_scrub_stamp":"2024-09-18T08:12:39.946462+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_clean_scrub_stamp":"2024-09-18T08:12:39.946462+0000","objects_scrubbed":5,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:13:55.625608+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":25,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,7],"acting":[3,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":509,"reported_epoch":282,"state":"active+clean","last_fresh":"2024-09-18T08:13:32.934010+0000","last_change":"2024-09-18T08:13:32.934010+0000","last_active":"2024-09-18T08:13:32.934010+0000","last_peered":"2024-09-18T08:13:32.934010+0000","last_clean":"2024-09-18T08:13:32.934010+0000","last_became_active":"2024-09-18T08:08:18.524218+0000","last_became_peered":"2024-09-18T08:08:18.524218+0000","last_unstale":"2024-09-18T08:13:32.934010+0000","last_undegraded":"2024-09-18T08:13:32.934010+0000","last_fullsized":"2024-09-18T08:13:32.934010+0000","mapping_epoch":181,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":182,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:13:32.933940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:46.860092+0000","last_clean_scrub_stamp":"2024-09-18T08:13:32.933940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:14:56.760035+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.15","version":"245'9","reported_seq":622,"reported_epoch":282,"state":"active+undersized+degraded","last_fresh":"2024-09-18T08:13:31.644123+0000","last_change":"2024-09-18T08:11:42.329955+0000","last_active":"2024-09-18T08:13:31.644123+0000","last_peered":"2024-09-18T08:13:31.644123+0000","last_clean":"2024-09-18T08:11:40.302227+0000","last_became_active":"2024-09-18T08:11:42.329955+0000","last_became_peered":"2024-09-18T08:11:42.329955+0000","last_unstale":"2024-09-18T08:13:31.644123+0000","last_undegraded":"2024-09-18T08:11:42.324112+0000","last_fullsized":"2024-09-18T08:11:42.324002+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'8","last_scrub_stamp":"2024-09-18T08:10:54.148986+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:50.886830+0000","last_clean_scrub_stamp":"2024-09-18T08:10:54.148986+0000","objects_scrubbed":1,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:13:38.308095+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":["5","0"],"object_location_counts":[{"shards":"0,5","objects":1}],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":470,"reported_epoch":283,"state":"active+clean","last_fresh":"2024-09-18T08:13:33.656688+0000","last_change":"2024-09-18T08:09:42.047337+0000","last_active":"2024-09-18T08:13:33.656688+0000","last_peered":"2024-09-18T08:13:33.656688+0000","last_clean":"2024-09-18T08:13:33.656688+0000","last_became_active":"2024-09-18T08:02:53.912112+0000","last_became_peered":"2024-09-18T08:02:53.912112+0000","last_unstale":"2024-09-18T08:13:33.656688+0000","last_undegraded":"2024-09-18T08:13:33.656688+0000","last_fullsized":"2024-09-18T08:13:33.656688+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:09:42.047270+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_clean_scrub_stamp":"2024-09-18T08:09:42.047270+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:14:17.848703+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.13","version":"271'19","reported_seq":250,"reported_epoch":282,"state":"active+undersized+degraded","last_fresh":"2024-09-18T08:13:31.648584+0000","last_change":"2024-09-18T08:11:42.324665+0000","last_active":"2024-09-18T08:13:31.648584+0000","last_peered":"2024-09-18T08:13:31.648584+0000","last_clean":"2024-09-18T08:11:40.307120+0000","last_became_active":"2024-09-18T08:11:42.324665+0000","last_became_peered":"2024-09-18T08:11:42.324665+0000","last_unstale":"2024-09-18T08:13:31.648584+0000","last_undegraded":"2024-09-18T08:11:42.319891+0000","last_fullsized":"2024-09-18T08:11:42.319779+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":182,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'11","last_scrub_stamp":"2024-09-18T08:11:22.565415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_clean_scrub_stamp":"2024-09-18T08:11:22.565415+0000","objects_scrubbed":5,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":5,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":31,"num_read_kb":24,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":["4","6"],"object_location_counts":[{"shards":"4,6","objects":5}],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":222,"reported_epoch":282,"state":"active+clean","last_fresh":"2024-09-18T08:13:31.649400+0000","last_change":"2024-09-18T08:10:58.402364+0000","last_active":"2024-09-18T08:13:31.649400+0000","last_peered":"2024-09-18T08:13:31.649400+0000","last_clean":"2024-09-18T08:13:31.649400+0000","last_became_active":"2024-09-18T08:08:18.622893+0000","last_became_peered":"2024-09-18T08:08:18.622893+0000","last_unstale":"2024-09-18T08:13:31.649400+0000","last_undegraded":"2024-09-18T08:13:31.649400+0000","last_fullsized":"2024-09-18T08:13:31.649400+0000","mapping_epoch":181,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":182,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:10:58.402308+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:46.860092+0000","last_clean_scrub_stamp":"2024-09-18T08:10:58.402308+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:14:18.606023+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":84,"reported_epoch":282,"state":"active+undersized","last_fresh":"2024-09-18T08:13:31.646019+0000","last_change":"2024-09-18T08:11:42.332003+0000","last_active":"2024-09-18T08:13:31.646019+0000","last_peered":"2024-09-18T08:13:31.646019+0000","last_clean":"2024-09-18T08:02:50.886830+0000","last_became_active":"2024-09-18T08:11:42.332003+0000","last_became_peered":"2024-09-18T08:11:42.332003+0000","last_unstale":"2024-09-18T08:13:31.646019+0000","last_undegraded":"2024-09-18T08:13:31.646019+0000","last_fullsized":"2024-09-18T08:11:42.330392+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:11:26.316948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:50.886830+0000","last_clean_scrub_stamp":"2024-09-18T08:11:26.316948+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":["5","3"],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":487,"reported_epoch":283,"state":"active+clean","last_fresh":"2024-09-18T08:13:33.651217+0000","last_change":"2024-09-18T08:13:28.680281+0000","last_active":"2024-09-18T08:13:33.651217+0000","last_peered":"2024-09-18T08:13:33.651217+0000","last_clean":"2024-09-18T08:13:33.651217+0000","last_became_active":"2024-09-18T08:02:53.911429+0000","last_became_peered":"2024-09-18T08:02:53.911429+0000","last_unstale":"2024-09-18T08:13:33.651217+0000","last_undegraded":"2024-09-18T08:13:33.651217+0000","last_fullsized":"2024-09-18T08:13:33.651217+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:13:28.680237+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_clean_scrub_stamp":"2024-09-18T08:13:28.680237+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:14:29.890119+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.12","version":"271'15","reported_seq":131,"reported_epoch":283,"state":"active+undersized+degraded","last_fresh":"2024-09-18T08:13:33.650278+0000","last_change":"2024-09-18T08:11:42.329218+0000","last_active":"2024-09-18T08:13:33.650278+0000","last_peered":"2024-09-18T08:13:33.650278+0000","last_clean":"2024-09-18T08:02:54.300830+0000","last_became_active":"2024-09-18T08:11:42.329218+0000","last_became_peered":"2024-09-18T08:11:42.329218+0000","last_unstale":"2024-09-18T08:13:33.650278+0000","last_undegraded":"2024-09-18T08:11:42.318592+0000","last_fullsized":"2024-09-18T08:11:42.318478+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'9","last_scrub_stamp":"2024-09-18T08:10:06.582804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_clean_scrub_stamp":"2024-09-18T08:10:06.582804+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:13:59.326333+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":25,"num_read_kb":19,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":["2","6"],"object_location_counts":[{"shards":"2,6","objects":4}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":509,"reported_epoch":282,"state":"active+clean","last_fresh":"2024-09-18T08:13:31.649760+0000","last_change":"2024-09-18T08:12:06.859635+0000","last_active":"2024-09-18T08:13:31.649760+0000","last_peered":"2024-09-18T08:13:31.649760+0000","last_clean":"2024-09-18T08:13:31.649760+0000","last_became_active":"2024-09-18T08:08:18.525527+0000","last_became_peered":"2024-09-18T08:08:18.525527+0000","last_unstale":"2024-09-18T08:13:31.649760+0000","last_undegraded":"2024-09-18T08:13:31.649760+0000","last_fullsized":"2024-09-18T08:13:31.649760+0000","mapping_epoch":181,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":182,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:12:06.854036+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:46.860092+0000","last_clean_scrub_stamp":"2024-09-18T08:12:06.854036+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":478,"reported_epoch":283,"state":"active+undersized","last_fresh":"2024-09-18T08:13:33.657694+0000","last_change":"2024-09-18T08:11:42.328480+0000","last_active":"2024-09-18T08:13:33.657694+0000","last_peered":"2024-09-18T08:13:33.657694+0000","last_clean":"2024-09-18T08:11:40.314825+0000","last_became_active":"2024-09-18T08:11:42.328480+0000","last_became_peered":"2024-09-18T08:11:42.328480+0000","last_unstale":"2024-09-18T08:13:33.657694+0000","last_undegraded":"2024-09-18T08:13:33.657694+0000","last_fullsized":"2024-09-18T08:11:42.323648+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:11:30.302179+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:50.886830+0000","last_clean_scrub_stamp":"2024-09-18T08:11:30.302179+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:14:16.797973+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":["3","0"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.10","version":"69'1","reported_seq":462,"reported_epoch":283,"state":"active+undersized+degraded","last_fresh":"2024-09-18T08:13:33.655868+0000","last_change":"2024-09-18T08:11:42.328627+0000","last_active":"2024-09-18T08:13:33.655868+0000","last_peered":"2024-09-18T08:13:33.655868+0000","last_clean":"2024-09-18T08:11:40.307990+0000","last_became_active":"2024-09-18T08:11:42.328627+0000","last_became_peered":"2024-09-18T08:11:42.328627+0000","last_unstale":"2024-09-18T08:13:33.655868+0000","last_undegraded":"2024-09-18T08:11:42.318971+0000","last_fullsized":"2024-09-18T08:11:42.318795+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'1","last_scrub_stamp":"2024-09-18T08:10:17.878965+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_clean_scrub_stamp":"2024-09-18T08:10:17.878965+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:13:57.798003+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":["0","5"],"object_location_counts":[{"shards":"0,5","objects":1}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.11","version":"271'19","reported_seq":563,"reported_epoch":283,"state":"active+clean","last_fresh":"2024-09-18T08:13:33.651519+0000","last_change":"2024-09-18T08:11:22.332386+0000","last_active":"2024-09-18T08:13:33.651519+0000","last_peered":"2024-09-18T08:13:33.651519+0000","last_clean":"2024-09-18T08:13:33.651519+0000","last_became_active":"2024-09-18T08:06:24.132416+0000","last_became_peered":"2024-09-18T08:06:24.132416+0000","last_unstale":"2024-09-18T08:13:33.651519+0000","last_undegraded":"2024-09-18T08:13:33.651519+0000","last_fullsized":"2024-09-18T08:13:33.651519+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'11","last_scrub_stamp":"2024-09-18T08:11:22.332279+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_clean_scrub_stamp":"2024-09-18T08:11:22.332279+0000","objects_scrubbed":5,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:14:11.722874+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":32,"num_read_kb":25,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":483,"reported_epoch":282,"state":"active+undersized","last_fresh":"2024-09-18T08:13:31.645530+0000","last_change":"2024-09-18T08:11:42.334948+0000","last_active":"2024-09-18T08:13:31.645530+0000","last_peered":"2024-09-18T08:13:31.645530+0000","last_clean":"2024-09-18T08:11:40.302665+0000","last_became_active":"2024-09-18T08:11:42.334948+0000","last_became_peered":"2024-09-18T08:11:42.334948+0000","last_unstale":"2024-09-18T08:13:31.645530+0000","last_undegraded":"2024-09-18T08:13:31.645530+0000","last_fullsized":"2024-09-18T08:11:42.330896+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:10:53.145315+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:46.860092+0000","last_clean_scrub_stamp":"2024-09-18T08:10:53.145315+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:14:28.127387+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":["5","7"],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":515,"reported_epoch":282,"state":"active+clean","last_fresh":"2024-09-18T08:13:31.644022+0000","last_change":"2024-09-18T08:10:52.577620+0000","last_active":"2024-09-18T08:13:31.644022+0000","last_peered":"2024-09-18T08:13:31.644022+0000","last_clean":"2024-09-18T08:13:31.644022+0000","last_became_active":"2024-09-18T08:08:18.524966+0000","last_became_peered":"2024-09-18T08:08:18.524966+0000","last_unstale":"2024-09-18T08:13:31.644022+0000","last_undegraded":"2024-09-18T08:13:31.644022+0000","last_fullsized":"2024-09-18T08:13:31.644022+0000","mapping_epoch":181,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":182,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:10:52.577568+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:50.886830+0000","last_clean_scrub_stamp":"2024-09-18T08:10:52.577568+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:14:25.852306+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":484,"reported_epoch":283,"state":"active+clean","last_fresh":"2024-09-18T08:13:33.657982+0000","last_change":"2024-09-18T08:11:29.344444+0000","last_active":"2024-09-18T08:13:33.657982+0000","last_peered":"2024-09-18T08:13:33.657982+0000","last_clean":"2024-09-18T08:13:33.657982+0000","last_became_active":"2024-09-18T08:06:24.128582+0000","last_became_peered":"2024-09-18T08:06:24.128582+0000","last_unstale":"2024-09-18T08:13:33.657982+0000","last_undegraded":"2024-09-18T08:13:33.657982+0000","last_fullsized":"2024-09-18T08:13:33.657982+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:11:29.344382+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_clean_scrub_stamp":"2024-09-18T08:11:29.344382+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:14:24.734652+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.10","version":"271'8","reported_seq":520,"reported_epoch":283,"state":"active+undersized+degraded","last_fresh":"2024-09-18T08:13:33.654210+0000","last_change":"2024-09-18T08:11:42.340283+0000","last_active":"2024-09-18T08:13:33.654210+0000","last_peered":"2024-09-18T08:13:33.654210+0000","last_clean":"2024-09-18T08:11:40.313525+0000","last_became_active":"2024-09-18T08:11:42.340283+0000","last_became_peered":"2024-09-18T08:11:42.340283+0000","last_unstale":"2024-09-18T08:13:33.654210+0000","last_undegraded":"2024-09-18T08:11:42.331866+0000","last_fullsized":"2024-09-18T08:11:42.325406+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'4","last_scrub_stamp":"2024-09-18T08:10:18.542984+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_clean_scrub_stamp":"2024-09-18T08:10:18.542984+0000","objects_scrubbed":2,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":12,"num_read_kb":10,"num_write":8,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":["3","6"],"object_location_counts":[{"shards":"3,6","objects":2}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":489,"reported_epoch":283,"state":"active+clean","last_fresh":"2024-09-18T08:13:33.657265+0000","last_change":"2024-09-18T08:10:13.820205+0000","last_active":"2024-09-18T08:13:33.657265+0000","last_peered":"2024-09-18T08:13:33.657265+0000","last_clean":"2024-09-18T08:13:33.657265+0000","last_became_active":"2024-09-18T08:02:47.877616+0000","last_became_peered":"2024-09-18T08:02:47.877616+0000","last_unstale":"2024-09-18T08:13:33.657265+0000","last_undegraded":"2024-09-18T08:13:33.657265+0000","last_fullsized":"2024-09-18T08:13:33.657265+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:10:13.820142+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:46.860092+0000","last_clean_scrub_stamp":"2024-09-18T08:10:13.820142+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:14:25.821232+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":230,"reported_epoch":282,"state":"active+clean","last_fresh":"2024-09-18T08:13:32.640757+0000","last_change":"2024-09-18T08:13:32.640757+0000","last_active":"2024-09-18T08:13:32.640757+0000","last_peered":"2024-09-18T08:13:32.640757+0000","last_clean":"2024-09-18T08:13:32.640757+0000","last_became_active":"2024-09-18T08:08:18.545361+0000","last_became_peered":"2024-09-18T08:08:18.545361+0000","last_unstale":"2024-09-18T08:13:32.640757+0000","last_undegraded":"2024-09-18T08:13:32.640757+0000","last_fullsized":"2024-09-18T08:13:32.640757+0000","mapping_epoch":181,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":182,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:13:32.640669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:50.886830+0000","last_clean_scrub_stamp":"2024-09-18T08:13:32.640669+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:14:49.886506+0000","scrub_duration":20,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,7],"acting":[4,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":490,"reported_epoch":282,"state":"active+clean","last_fresh":"2024-09-18T08:13:31.645170+0000","last_change":"2024-09-18T08:09:34.607443+0000","last_active":"2024-09-18T08:13:31.645170+0000","last_peered":"2024-09-18T08:13:31.645170+0000","last_clean":"2024-09-18T08:13:31.645170+0000","last_became_active":"2024-09-18T08:08:18.525552+0000","last_became_peered":"2024-09-18T08:08:18.525552+0000","last_unstale":"2024-09-18T08:13:31.645170+0000","last_undegraded":"2024-09-18T08:13:31.645170+0000","last_fullsized":"2024-09-18T08:13:31.645170+0000","mapping_epoch":181,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":182,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:09:34.607374+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_clean_scrub_stamp":"2024-09-18T08:09:34.607374+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:13:49.631974+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,4],"acting":[7,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":330,"reported_epoch":282,"state":"active+undersized","last_fresh":"2024-09-18T08:13:31.644228+0000","last_change":"2024-09-18T08:11:42.341016+0000","last_active":"2024-09-18T08:13:31.644228+0000","last_peered":"2024-09-18T08:13:31.644228+0000","last_clean":"2024-09-18T08:08:15.505600+0000","last_became_active":"2024-09-18T08:11:42.341016+0000","last_became_peered":"2024-09-18T08:11:42.341016+0000","last_unstale":"2024-09-18T08:13:31.644228+0000","last_undegraded":"2024-09-18T08:13:31.644228+0000","last_fullsized":"2024-09-18T08:11:42.325794+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":182,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:10:52.641537+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_clean_scrub_stamp":"2024-09-18T08:10:52.641537+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:14:26.159423+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":["5","4"],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":478,"reported_epoch":283,"state":"active+undersized","last_fresh":"2024-09-18T08:13:33.654501+0000","last_change":"2024-09-18T08:11:42.342099+0000","last_active":"2024-09-18T08:13:33.654501+0000","last_peered":"2024-09-18T08:13:33.654501+0000","last_clean":"2024-09-18T08:11:40.314449+0000","last_became_active":"2024-09-18T08:11:42.342099+0000","last_became_peered":"2024-09-18T08:11:42.342099+0000","last_unstale":"2024-09-18T08:13:33.654501+0000","last_undegraded":"2024-09-18T08:13:33.654501+0000","last_fullsized":"2024-09-18T08:11:42.325904+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:07:17.277894+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:46.860092+0000","last_clean_scrub_stamp":"2024-09-18T08:07:17.277894+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:13:48.047234+0000","scrub_duration":16,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":["3","0"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"271'19","reported_seq":485,"reported_epoch":282,"state":"active+undersized+degraded","last_fresh":"2024-09-18T08:13:31.643584+0000","last_change":"2024-09-18T08:11:42.334941+0000","last_active":"2024-09-18T08:13:31.643584+0000","last_peered":"2024-09-18T08:13:31.643584+0000","last_clean":"2024-09-18T08:11:40.304598+0000","last_became_active":"2024-09-18T08:11:42.334941+0000","last_became_peered":"2024-09-18T08:11:42.334941+0000","last_unstale":"2024-09-18T08:13:31.643584+0000","last_undegraded":"2024-09-18T08:11:42.328266+0000","last_fullsized":"2024-09-18T08:11:42.327013+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'11","last_scrub_stamp":"2024-09-18T08:10:50.208543+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_clean_scrub_stamp":"2024-09-18T08:10:50.208543+0000","objects_scrubbed":5,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:13:36.350714+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":5,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":31,"num_read_kb":24,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":["5","6"],"object_location_counts":[{"shards":"5,6","objects":5}],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":496,"reported_epoch":283,"state":"active+clean","last_fresh":"2024-09-18T08:13:33.656757+0000","last_change":"2024-09-18T08:12:31.664510+0000","last_active":"2024-09-18T08:13:33.656757+0000","last_peered":"2024-09-18T08:13:33.656757+0000","last_clean":"2024-09-18T08:13:33.656757+0000","last_became_active":"2024-09-18T08:08:52.700595+0000","last_became_peered":"2024-09-18T08:08:52.700595+0000","last_unstale":"2024-09-18T08:13:33.656757+0000","last_undegraded":"2024-09-18T08:13:33.656757+0000","last_fullsized":"2024-09-18T08:13:33.656757+0000","mapping_epoch":198,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":199,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:12:31.664456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:50.886830+0000","last_clean_scrub_stamp":"2024-09-18T08:12:31.664456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:13:44.437533+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":9,"num_pg":16,"stat_sum":{"num_bytes":4711120920,"num_objects":71887,"num_object_clones":0,"num_object_copies":215661,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":40716,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":71887,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":211879,"num_write_kb":9080449,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":65536,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":13354213376,"data_stored":13354205232,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":79676,"ondisk_log_size":79676,"up":39,"acting":39,"num_store_stats":8},{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":416,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":86,"acting":86,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":87,"ondisk_log_size":87,"up":82,"acting":82,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":178,"num_object_clones":0,"num_object_copies":534,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":91,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":178,"num_whiteouts":0,"num_read":1110,"num_read_kb":867,"num_write":703,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":178,"num_bytes_recovered":3784,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":417792,"data_stored":11106,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":685,"ondisk_log_size":685,"up":79,"acting":79,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":1613,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":24,"num_read_kb":24,"num_write":10,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":2698,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":4839,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":6,"ondisk_log_size":6,"up":85,"acting":85,"num_store_stats":8},{"poolid":2,"num_pg":3,"stat_sum":{"num_bytes":408,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":118,"num_read_kb":113,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":38,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":8,"ondisk_log_size":8,"up":7,"acting":7,"num_store_stats":7},{"poolid":1,"num_pg":11,"stat_sum":{"num_bytes":644038,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":10,"num_read_kb":20,"num_write":11,"num_write_kb":148,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":53670,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":869,"ondisk_log_size":869,"up":32,"acting":32,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":56,"seq":240518168709,"num_pgs":65,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":3044004,"kb_used_data":2629652,"kb_used_omap":1,"kb_used_meta":414334,"kb_avail":90701148,"statfs":{"total":95995035648,"available":92877975552,"internally_reserved":0,"allocated":2692763648,"data_stored":2692113789,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":424278474},"hb_peers":[0,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":6,"up_from":47,"seq":201863463051,"num_pgs":50,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1213840,"kb_used_data":867516,"kb_used_omap":1,"kb_used_meta":346302,"kb_avail":92531312,"statfs":{"total":95995035648,"available":94752063488,"internally_reserved":0,"allocated":888336384,"data_stored":887709992,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":354613702},"hb_peers":[0,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":39,"seq":167503724689,"num_pgs":62,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":3047720,"kb_used_data":2619480,"kb_used_omap":1,"kb_used_meta":428222,"kb_avail":90697432,"statfs":{"total":95995035648,"available":92874170368,"internally_reserved":0,"allocated":2682347520,"data_stored":2681694052,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":438499782},"hb_peers":[0,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,1],"upper_bound":16},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":4,"up_from":32,"seq":137438953622,"num_pgs":64,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1241900,"kb_used_data":873372,"kb_used_omap":1,"kb_used_meta":368510,"kb_avail":92503252,"statfs":{"total":95995035648,"available":94723330048,"internally_reserved":0,"allocated":894332928,"data_stored":893675366,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":377354693},"hb_peers":[0,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":26,"seq":111669149853,"num_pgs":70,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2829316,"kb_used_data":2314992,"kb_used_omap":1,"kb_used_meta":514302,"kb_avail":90915836,"statfs":{"total":95995035648,"available":93097816064,"internally_reserved":0,"allocated":2370551808,"data_stored":2369880492,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":526645704},"hb_peers":[0,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411491,"num_pgs":46,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1289692,"kb_used_data":871180,"kb_used_omap":1,"kb_used_meta":418494,"kb_avail":92455460,"statfs":{"total":95995035648,"available":94674391040,"internally_reserved":0,"allocated":892088320,"data_stored":891447002,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":428538309},"hb_peers":[0,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574992,"num_pgs":64,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2106904,"kb_used_data":1782792,"kb_used_omap":1,"kb_used_meta":324094,"kb_avail":91638248,"statfs":{"total":95995035648,"available":93837565952,"internally_reserved":0,"allocated":1825579008,"data_stored":1825118505,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":331872709},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738542,"num_pgs":53,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1502464,"kb_used_data":1146408,"kb_used_omap":1,"kb_used_meta":356030,"kb_avail":92242688,"statfs":{"total":95995035648,"available":94456512512,"internally_reserved":0,"allocated":1173921792,"data_stored":1173259033,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":364575173},"hb_peers":[2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":266240,"data_stored":262688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1131,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1521,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":436,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":46,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1177,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":436,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":69632,"data_stored":1788,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1128,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":1898,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1474,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1210,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1034,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1254,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":403,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":416,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":403,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":1166868480,"data_stored":1166868480,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":1860698112,"data_stored":1860698112,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":883621888,"data_stored":883621888,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":2351104000,"data_stored":2351104000,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":881655808,"data_stored":881655808,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":2660831232,"data_stored":2660827160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":884539392,"data_stored":884539392,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":2664894464,"data_stored":2664890392,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T08:13:40.345 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid 779349ca-7593-11ef-bceb-c7b262605968 -- ceph osd pool set .mgr pgp_num 11 2024-09-18T08:13:40.346 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/779349ca-7593-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T08:13:40.357 INFO:teuthology.orchestra.run.smithi050.stderr:Inferring config /var/lib/ceph/779349ca-7593-11ef-bceb-c7b262605968/mon.b/config 2024-09-18T08:13:40.378 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7 2024-09-18T08:13:40.427 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:13:40.428 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:13:40.428 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7' 2024-09-18T08:13:40.529 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3 2024-09-18T08:13:40.565 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:13:40.565 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:13:40.566 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3' 2024-09-18T08:13:40.667 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4 2024-09-18T08:13:40.705 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:13:40.706 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:13:40.706 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4' 2024-09-18T08:13:40.807 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0 2024-09-18T08:13:40.823 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:13:40 smithi050 ceph-mon[32371]: pgmap v524: 158 pgs: 30 active+undersized, 34 active+undersized+degraded, 94 active+clean; 4.0 GiB data, 15 GiB used, 701 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.64k op/s; 37257/197457 objects degraded (18.868%) 2024-09-18T08:13:40.823 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:13:40 smithi050 ceph-mon[32371]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-18T08:13:40.840 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:13:40.841 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:13:40.842 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0' 2024-09-18T08:13:40.922 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:13:40 smithi007 ceph-mon[29459]: pgmap v524: 158 pgs: 30 active+undersized, 34 active+undersized+degraded, 94 active+clean; 4.0 GiB data, 15 GiB used, 701 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.64k op/s; 37257/197457 objects degraded (18.868%) 2024-09-18T08:13:40.922 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:13:40 smithi007 ceph-mon[29459]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-18T08:13:40.922 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:13:40 smithi007 ceph-mon[39249]: pgmap v524: 158 pgs: 30 active+undersized, 34 active+undersized+degraded, 94 active+clean; 4.0 GiB data, 15 GiB used, 701 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.64k op/s; 37257/197457 objects degraded (18.868%) 2024-09-18T08:13:40.923 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:13:40 smithi007 ceph-mon[39249]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-18T08:13:40.943 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6 2024-09-18T08:13:40.981 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:13:40.982 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:13:40.982 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6' 2024-09-18T08:13:40.996 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2024-09-18T08:13:40.997 INFO:teuthology.orchestra.run.smithi050.stdout: "ops": [ 2024-09-18T08:13:40.997 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:13:40.997 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.15582.0:221230 9.7 9:e826871b:::benchmark_data_smithi007_124019_object79345:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e283)", 2024-09-18T08:13:40.997 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-09-18T08:13:40.984256+0000", 2024-09-18T08:13:40.997 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 0.0097849189999999996, 2024-09-18T08:13:40.997 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0098137429999999998, 2024-09-18T08:13:40.997 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-09-18T08:13:40.997 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-09-18T08:13:40.997 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "waiting for sub ops", 2024-09-18T08:13:40.997 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2024-09-18T08:13:40.997 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.15582", 2024-09-18T08:13:40.997 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "172.21.15.7:0/3598605839", 2024-09-18T08:13:40.997 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 221230 2024-09-18T08:13:40.998 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:13:40.998 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-09-18T08:13:40.998 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:13:40.998 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-09-18T08:13:40.998 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:13:40.984256+0000", 2024-09-18T08:13:40.998 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-18T08:13:40.998 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:13:40.998 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:13:40.998 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-09-18T08:13:40.998 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:13:40.984256+0000", 2024-09-18T08:13:40.998 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-18T08:13:40.998 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:13:40.998 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:13:40.999 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-09-18T08:13:40.999 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:13:40.984257+0000", 2024-09-18T08:13:40.999 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.5400000000000001e-06 2024-09-18T08:13:40.999 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:13:40.999 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:13:40.999 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-09-18T08:13:40.999 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:13:40.984262+0000", 2024-09-18T08:13:40.999 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4.9280000000000001e-06 2024-09-18T08:13:40.999 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:13:40.999 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:13:40.999 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-09-18T08:13:40.999 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:13:40.984277+0000", 2024-09-18T08:13:40.999 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.4749999999999999e-05 2024-09-18T08:13:41.000 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:13:41.000 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:13:41.000 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-09-18T08:13:41.000 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:13:40.984284+0000", 2024-09-18T08:13:41.000 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 7.1579999999999999e-06 2024-09-18T08:13:41.000 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:13:41.000 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:13:41.000 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-18T08:13:41.000 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:13:40.984380+0000", 2024-09-18T08:13:41.000 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 9.5575000000000006e-05 2024-09-18T08:13:41.000 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:13:41.000 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:13:41.000 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-09-18T08:13:41.001 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:13:40.984486+0000", 2024-09-18T08:13:41.001 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000106809 2024-09-18T08:13:41.001 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:13:41.001 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:13:41.001 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "waiting for subops from 3,7", 2024-09-18T08:13:41.001 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:13:40.984593+0000", 2024-09-18T08:13:41.001 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00010691700000000001 2024-09-18T08:13:41.001 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:13:41.001 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:13:41.001 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:13:41.001 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:13:40.990949+0000", 2024-09-18T08:13:41.001 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.006355477 2024-09-18T08:13:41.001 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:13:41.001 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:13:41.002 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:13:41.002 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:13:40.993889+0000", 2024-09-18T08:13:41.002 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0029404510000000002 2024-09-18T08:13:41.002 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-18T08:13:41.002 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-09-18T08:13:41.002 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-18T08:13:41.002 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:13:41.002 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:13:41.002 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.15582.0:221234 9.f 9:ff0eb223:::benchmark_data_smithi007_124019_object79349:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e283)", 2024-09-18T08:13:41.002 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-09-18T08:13:40.984866+0000", 2024-09-18T08:13:41.002 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 0.0091741270000000007, 2024-09-18T08:13:41.002 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0092906889999999995, 2024-09-18T08:13:41.002 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-09-18T08:13:41.002 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-09-18T08:13:41.003 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "waiting for sub ops", 2024-09-18T08:13:41.003 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2024-09-18T08:13:41.003 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.15582", 2024-09-18T08:13:41.003 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "172.21.15.7:0/3598605839", 2024-09-18T08:13:41.003 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 221234 2024-09-18T08:13:41.003 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:13:41.003 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-09-18T08:13:41.003 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:13:41.003 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-09-18T08:13:41.003 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:13:40.984866+0000", 2024-09-18T08:13:41.003 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-18T08:13:41.003 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:13:41.003 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:13:41.003 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-09-18T08:13:41.003 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:13:40.984866+0000", 2024-09-18T08:13:41.004 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-18T08:13:41.004 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:13:41.004 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:13:41.004 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-09-18T08:13:41.004 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:13:40.984867+0000", 2024-09-18T08:13:41.004 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 6.9500000000000002e-07 2024-09-18T08:13:41.004 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:13:41.004 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:13:41.004 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-09-18T08:13:41.004 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:13:40.984871+0000", 2024-09-18T08:13:41.004 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4.2769999999999999e-06 2024-09-18T08:13:41.004 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:13:41.005 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:13:41.005 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-09-18T08:13:41.005 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:13:40.984883+0000", 2024-09-18T08:13:41.005 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.1260000000000001e-05 2024-09-18T08:13:41.005 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:13:41.005 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:13:41.005 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-09-18T08:13:41.005 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:13:40.984890+0000", 2024-09-18T08:13:41.005 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 7.0720000000000002e-06 2024-09-18T08:13:41.005 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:13:41.005 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:13:41.005 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-18T08:13:41.005 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:13:40.985051+0000", 2024-09-18T08:13:41.005 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00016163700000000001 2024-09-18T08:13:41.006 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:13:41.006 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:13:41.006 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-09-18T08:13:41.006 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:13:40.985160+0000", 2024-09-18T08:13:41.006 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000108255 2024-09-18T08:13:41.006 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:13:41.006 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:13:41.006 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "waiting for subops from 3", 2024-09-18T08:13:41.006 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:13:40.985268+0000", 2024-09-18T08:13:41.006 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00010872 2024-09-18T08:13:41.006 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-18T08:13:41.006 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-09-18T08:13:41.006 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-18T08:13:41.006 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:13:41.007 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:13:41.007 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.15582.0:221235 9.7 9:eb6c03a2:::benchmark_data_smithi007_124019_object79350:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e283)", 2024-09-18T08:13:41.007 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-09-18T08:13:40.985945+0000", 2024-09-18T08:13:41.007 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 0.0080951010000000004, 2024-09-18T08:13:41.007 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0082809340000000002, 2024-09-18T08:13:41.007 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-09-18T08:13:41.007 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-09-18T08:13:41.007 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "waiting for sub ops", 2024-09-18T08:13:41.007 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2024-09-18T08:13:41.007 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.15582", 2024-09-18T08:13:41.007 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "172.21.15.7:0/3598605839", 2024-09-18T08:13:41.007 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 221235 2024-09-18T08:13:41.007 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:13:41.007 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-09-18T08:13:41.008 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:13:41.008 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-09-18T08:13:41.008 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:13:40.985945+0000", 2024-09-18T08:13:41.008 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-18T08:13:41.008 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:13:41.008 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:13:41.008 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-09-18T08:13:41.008 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:13:40.985945+0000", 2024-09-18T08:13:41.008 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-18T08:13:41.008 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:13:41.008 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:13:41.008 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-09-18T08:13:41.008 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:13:40.985947+0000", 2024-09-18T08:13:41.008 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.0839999999999999e-06 2024-09-18T08:13:41.008 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:13:41.009 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:13:41.009 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-09-18T08:13:41.009 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:13:40.985951+0000", 2024-09-18T08:13:41.009 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4.6210000000000003e-06 2024-09-18T08:13:41.009 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:13:41.009 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:13:41.009 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-09-18T08:13:41.009 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:13:40.985962+0000", 2024-09-18T08:13:41.009 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.1095e-05 2024-09-18T08:13:41.009 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:13:41.009 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:13:41.009 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-09-18T08:13:41.009 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:13:40.985970+0000", 2024-09-18T08:13:41.010 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 7.3529999999999997e-06 2024-09-18T08:13:41.010 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:13:41.010 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:13:41.010 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-18T08:13:41.010 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:13:40.990981+0000", 2024-09-18T08:13:41.010 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0050109999999999998 2024-09-18T08:13:41.010 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:13:41.010 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:13:41.010 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-09-18T08:13:41.010 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:13:40.991101+0000", 2024-09-18T08:13:41.010 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000119973 2024-09-18T08:13:41.010 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:13:41.010 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:13:41.011 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/779349ca-7593-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T08:13:41.011 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "waiting for subops from 3,7", 2024-09-18T08:13:41.011 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:13:40.991250+0000", 2024-09-18T08:13:41.011 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000149307 2024-09-18T08:13:41.011 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-18T08:13:41.011 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-09-18T08:13:41.012 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-18T08:13:41.012 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:13:41.012 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:13:41.012 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_repop(client.15582.0:221236 9.5 e283/232 9:a5fcf677:::benchmark_data_smithi007_124019_object79351:head v 283'13795, mlcod=283'13793)", 2024-09-18T08:13:41.012 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-09-18T08:13:40.988498+0000", 2024-09-18T08:13:41.012 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 0.0055422409999999998, 2024-09-18T08:13:41.012 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0057950479999999997, 2024-09-18T08:13:41.012 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-09-18T08:13:41.012 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-09-18T08:13:41.012 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "started", 2024-09-18T08:13:41.012 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-09-18T08:13:41.012 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:13:41.012 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-09-18T08:13:41.013 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:13:40.988498+0000", 2024-09-18T08:13:41.013 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-18T08:13:41.013 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:13:41.013 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:13:41.013 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-09-18T08:13:41.013 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:13:40.988498+0000", 2024-09-18T08:13:41.013 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-18T08:13:41.013 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:13:41.013 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:13:41.013 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-09-18T08:13:41.013 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:13:40.988499+0000", 2024-09-18T08:13:41.013 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 6.7400000000000003e-07 2024-09-18T08:13:41.013 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:13:41.013 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:13:41.013 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-09-18T08:13:41.014 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:13:40.988506+0000", 2024-09-18T08:13:41.014 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 7.4599999999999997e-06 2024-09-18T08:13:41.014 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:13:41.014 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:13:41.014 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-09-18T08:13:41.014 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:13:40.988519+0000", 2024-09-18T08:13:41.014 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.2323e-05 2024-09-18T08:13:41.014 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:13:41.014 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:13:41.014 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-09-18T08:13:41.014 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:13:40.988526+0000", 2024-09-18T08:13:41.014 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 6.9349999999999997e-06 2024-09-18T08:13:41.014 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:13:41.014 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:13:41.015 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-18T08:13:41.015 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:13:40.988623+0000", 2024-09-18T08:13:41.015 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 9.7264999999999998e-05 2024-09-18T08:13:41.015 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:13:41.015 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:13:41.015 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-09-18T08:13:41.015 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:13:40.988642+0000", 2024-09-18T08:13:41.015 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.9358999999999999e-05 2024-09-18T08:13:41.015 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-18T08:13:41.015 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-09-18T08:13:41.015 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-18T08:13:41.015 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:13:41.015 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:13:41.016 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.15582.0:221238 9.6 9:62202f18:::benchmark_data_smithi007_124019_object79353:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e283)", 2024-09-18T08:13:41.016 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-09-18T08:13:40.989564+0000", 2024-09-18T08:13:41.016 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 0.0044767569999999996, 2024-09-18T08:13:41.016 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0047961710000000001, 2024-09-18T08:13:41.016 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-09-18T08:13:41.016 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-09-18T08:13:41.016 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "waiting for sub ops", 2024-09-18T08:13:41.016 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2024-09-18T08:13:41.016 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.15582", 2024-09-18T08:13:41.016 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "172.21.15.7:0/3598605839", 2024-09-18T08:13:41.016 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 221238 2024-09-18T08:13:41.016 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:13:41.017 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-09-18T08:13:41.017 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:13:41.017 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-09-18T08:13:41.017 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:13:40.989564+0000", 2024-09-18T08:13:41.017 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-18T08:13:41.017 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:13:41.017 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:13:41.017 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-09-18T08:13:41.017 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:13:40.989564+0000", 2024-09-18T08:13:41.017 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-18T08:13:41.017 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:13:41.017 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:13:41.017 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-09-18T08:13:41.017 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:13:40.989565+0000", 2024-09-18T08:13:41.017 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.5e-06 2024-09-18T08:13:41.018 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:13:41.018 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:13:41.018 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-09-18T08:13:41.018 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:13:40.989570+0000", 2024-09-18T08:13:41.018 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4.9760000000000003e-06 2024-09-18T08:13:41.018 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:13:41.018 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:13:41.018 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-09-18T08:13:41.018 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:13:40.989583+0000", 2024-09-18T08:13:41.018 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.2612000000000001e-05 2024-09-18T08:13:41.018 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:13:41.018 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:13:41.018 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-09-18T08:13:41.018 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:13:40.989590+0000", 2024-09-18T08:13:41.019 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 7.4290000000000004e-06 2024-09-18T08:13:41.019 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:13:41.019 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:13:41.019 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-18T08:13:41.019 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:13:40.989647+0000", 2024-09-18T08:13:41.019 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 5.6372000000000002e-05 2024-09-18T08:13:41.019 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:13:41.019 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:13:41.019 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-09-18T08:13:41.019 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:13:40.989772+0000", 2024-09-18T08:13:41.019 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000124843 2024-09-18T08:13:41.019 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:13:41.019 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:13:41.019 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "waiting for subops from 2", 2024-09-18T08:13:41.019 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:13:40.989915+0000", 2024-09-18T08:13:41.020 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00014373200000000001 2024-09-18T08:13:41.020 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-18T08:13:41.020 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-09-18T08:13:41.020 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-18T08:13:41.020 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:13:41.020 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:13:41.020 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_repop(client.15582.0:221241 9.5 e283/232)", 2024-09-18T08:13:41.020 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-09-18T08:13:40.992589+0000", 2024-09-18T08:13:41.020 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 0.0014515660000000001, 2024-09-18T08:13:41.020 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.001833671, 2024-09-18T08:13:41.020 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-09-18T08:13:41.020 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-09-18T08:13:41.020 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "queued for pg", 2024-09-18T08:13:41.020 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-09-18T08:13:41.021 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:13:41.021 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-09-18T08:13:41.021 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:13:40.992589+0000", 2024-09-18T08:13:41.021 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-18T08:13:41.021 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:13:41.021 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:13:41.021 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-09-18T08:13:41.021 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:13:40.992589+0000", 2024-09-18T08:13:41.021 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-18T08:13:41.021 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:13:41.021 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:13:41.021 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-09-18T08:13:41.021 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:13:40.992593+0000", 2024-09-18T08:13:41.021 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 3.811e-06 2024-09-18T08:13:41.021 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:13:41.022 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:13:41.022 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-09-18T08:13:41.022 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:13:40.992597+0000", 2024-09-18T08:13:41.022 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4.4580000000000003e-06 2024-09-18T08:13:41.022 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:13:41.022 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:13:41.022 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-09-18T08:13:41.022 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:13:40.992606+0000", 2024-09-18T08:13:41.022 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 8.4139999999999994e-06 2024-09-18T08:13:41.022 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:13:41.022 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:13:41.022 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-09-18T08:13:41.022 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:13:40.992612+0000", 2024-09-18T08:13:41.022 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 5.8109999999999999e-06 2024-09-18T08:13:41.022 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-18T08:13:41.023 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-09-18T08:13:41.023 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-18T08:13:41.023 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:13:41.023 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:13:41.023 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.15582.0:221243 9.7 9:eafc12ec:::benchmark_data_smithi007_124019_object79358:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e283)", 2024-09-18T08:13:41.023 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-09-18T08:13:40.992920+0000", 2024-09-18T08:13:41.023 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 0.001120465, 2024-09-18T08:13:41.023 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.001547142, 2024-09-18T08:13:41.023 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-09-18T08:13:41.023 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-09-18T08:13:41.023 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "waiting for sub ops", 2024-09-18T08:13:41.023 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2024-09-18T08:13:41.023 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.15582", 2024-09-18T08:13:41.023 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "172.21.15.7:0/3598605839", 2024-09-18T08:13:41.023 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 221243 2024-09-18T08:13:41.023 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:13:41.024 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-09-18T08:13:41.024 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:13:41.024 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-09-18T08:13:41.024 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:13:40.992920+0000", 2024-09-18T08:13:41.024 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-18T08:13:41.024 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:13:41.024 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:13:41.024 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-09-18T08:13:41.024 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:13:40.992920+0000", 2024-09-18T08:13:41.024 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-18T08:13:41.024 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:13:41.024 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:13:41.024 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-09-18T08:13:41.024 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:13:40.992921+0000", 2024-09-18T08:13:41.025 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.35e-06 2024-09-18T08:13:41.025 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:13:41.025 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:13:41.025 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-09-18T08:13:41.025 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:13:40.992925+0000", 2024-09-18T08:13:41.025 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4.0260000000000001e-06 2024-09-18T08:13:41.025 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:13:41.025 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:13:41.025 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-09-18T08:13:41.025 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:13:40.992939+0000", 2024-09-18T08:13:41.025 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.3266e-05 2024-09-18T08:13:41.025 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:13:41.025 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:13:41.025 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-09-18T08:13:41.026 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:13:40.992946+0000", 2024-09-18T08:13:41.026 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 7.3200000000000002e-06 2024-09-18T08:13:41.026 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:13:41.026 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:13:41.026 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-18T08:13:41.026 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:13:40.994211+0000", 2024-09-18T08:13:41.026 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0012652239999999999 2024-09-18T08:13:41.026 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:13:41.026 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:13:41.026 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-09-18T08:13:41.026 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:13:40.994321+0000", 2024-09-18T08:13:41.026 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00011014999999999999 2024-09-18T08:13:41.026 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:13:41.027 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:13:41.027 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "waiting for subops from 3,7", 2024-09-18T08:13:41.027 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:13:40.994408+0000", 2024-09-18T08:13:41.027 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 8.6656999999999994e-05 2024-09-18T08:13:41.027 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-18T08:13:41.027 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-09-18T08:13:41.027 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-18T08:13:41.027 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:13:41.027 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:13:41.027 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_repop(client.15582.0:221242 9.d e283/244 9:b2afae73:::benchmark_data_smithi007_124019_object79357:head v 283'13928, mlcod=283'13926)", 2024-09-18T08:13:41.027 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-09-18T08:13:40.994062+0000", 2024-09-18T08:13:41.027 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 4294967295.999979, 2024-09-18T08:13:41.027 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00047982100000000001, 2024-09-18T08:13:41.027 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-09-18T08:13:41.028 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-09-18T08:13:41.028 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "started", 2024-09-18T08:13:41.028 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-09-18T08:13:41.028 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:13:41.028 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-09-18T08:13:41.028 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:13:40.994062+0000", 2024-09-18T08:13:41.028 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-18T08:13:41.028 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:13:41.028 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:13:41.028 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-09-18T08:13:41.028 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:13:40.994062+0000", 2024-09-18T08:13:41.028 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-18T08:13:41.029 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:13:41.029 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:13:41.029 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-09-18T08:13:41.029 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:13:40.994062+0000", 2024-09-18T08:13:41.029 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 7.7100000000000001e-07 2024-09-18T08:13:41.029 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:13:41.029 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:13:41.029 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-09-18T08:13:41.029 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:13:40.994068+0000", 2024-09-18T08:13:41.029 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 5.8080000000000001e-06 2024-09-18T08:13:41.029 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:13:41.029 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:13:41.029 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-09-18T08:13:41.030 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:13:40.994078+0000", 2024-09-18T08:13:41.030 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 9.5969999999999998e-06 2024-09-18T08:13:41.030 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:13:41.030 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:13:41.030 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-09-18T08:13:41.030 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:13:40.994084+0000", 2024-09-18T08:13:41.030 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 6.3940000000000001e-06 2024-09-18T08:13:41.030 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:13:41.030 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:13:41.030 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-18T08:13:41.030 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:13:40.994472+0000", 2024-09-18T08:13:41.030 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00038737100000000002 2024-09-18T08:13:41.030 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:13:41.030 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:13:41.031 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-09-18T08:13:41.031 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:13:40.994491+0000", 2024-09-18T08:13:41.031 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.9164000000000002e-05 2024-09-18T08:13:41.031 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-18T08:13:41.031 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-09-18T08:13:41.031 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-18T08:13:41.031 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:13:41.031 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:13:41.031 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_repop_reply(client.15582.0:221235 9.7 e283/232)", 2024-09-18T08:13:41.031 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-09-18T08:13:40.994248+0000", 2024-09-18T08:13:41.031 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 4294967295.9997921, 2024-09-18T08:13:41.031 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00034620899999999999, 2024-09-18T08:13:41.031 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-09-18T08:13:41.031 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-09-18T08:13:41.031 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "queued for pg", 2024-09-18T08:13:41.032 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-09-18T08:13:41.032 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:13:41.032 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-09-18T08:13:41.032 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:13:40.994248+0000", 2024-09-18T08:13:41.032 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-18T08:13:41.032 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:13:41.032 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:13:41.032 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-09-18T08:13:41.032 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:13:40.994248+0000", 2024-09-18T08:13:41.032 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-18T08:13:41.032 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:13:41.032 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:13:41.032 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-09-18T08:13:41.033 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:13:40.994249+0000", 2024-09-18T08:13:41.033 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 5.6300000000000005e-07 2024-09-18T08:13:41.033 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:13:41.033 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:13:41.033 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-09-18T08:13:41.033 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:13:40.994253+0000", 2024-09-18T08:13:41.033 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 3.8070000000000001e-06 2024-09-18T08:13:41.033 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:13:41.033 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:13:41.033 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-09-18T08:13:41.033 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:13:40.994261+0000", 2024-09-18T08:13:41.033 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 8.5660000000000007e-06 2024-09-18T08:13:41.033 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:13:41.033 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:13:41.034 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-09-18T08:13:41.034 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:13:40.994267+0000", 2024-09-18T08:13:41.034 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 5.9410000000000004e-06 2024-09-18T08:13:41.034 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-18T08:13:41.034 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-09-18T08:13:41.034 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-18T08:13:41.034 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:13:41.034 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:13:41.034 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_repop_reply(client.15582.0:221234 9.f e283/244)", 2024-09-18T08:13:41.034 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-09-18T08:13:40.994350+0000", 2024-09-18T08:13:41.034 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 4294967295.9996905, 2024-09-18T08:13:41.034 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00028892399999999997, 2024-09-18T08:13:41.034 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-09-18T08:13:41.034 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-09-18T08:13:41.035 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "queued for pg", 2024-09-18T08:13:41.035 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-09-18T08:13:41.035 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:13:41.035 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-09-18T08:13:41.035 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:13:40.994350+0000", 2024-09-18T08:13:41.035 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-18T08:13:41.035 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:13:41.035 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:13:41.035 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-09-18T08:13:41.035 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:13:40.994350+0000", 2024-09-18T08:13:41.035 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-18T08:13:41.035 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:13:41.035 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:13:41.035 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-09-18T08:13:41.035 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:13:40.994351+0000", 2024-09-18T08:13:41.036 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4.9299999999999998e-07 2024-09-18T08:13:41.036 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:13:41.036 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:13:41.036 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-09-18T08:13:41.036 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:13:40.994354+0000", 2024-09-18T08:13:41.036 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 3.0900000000000001e-06 2024-09-18T08:13:41.036 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:13:41.036 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:13:41.036 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-09-18T08:13:41.036 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:13:40.994361+0000", 2024-09-18T08:13:41.036 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 7.0400000000000004e-06 2024-09-18T08:13:41.036 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:13:41.036 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:13:41.036 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-09-18T08:13:41.036 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:13:40.994366+0000", 2024-09-18T08:13:41.037 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 5.5230000000000003e-06 2024-09-18T08:13:41.037 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-18T08:13:41.037 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-09-18T08:13:41.037 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-18T08:13:41.037 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-18T08:13:41.037 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2024-09-18T08:13:41.037 INFO:teuthology.orchestra.run.smithi050.stdout: "num_ops": 10 2024-09-18T08:13:41.037 INFO:teuthology.orchestra.run.smithi050.stdout:} 2024-09-18T08:13:41.082 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0 2024-09-18T08:13:41.117 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:13:41.118 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:13:41.118 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0' 2024-09-18T08:13:41.219 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4 2024-09-18T08:13:41.253 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:13:41.253 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:13:41.254 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4' 2024-09-18T08:13:41.354 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0 2024-09-18T08:13:41.386 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:13:41.386 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:13:41.387 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0' 2024-09-18T08:13:41.488 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2 2024-09-18T08:13:41.521 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:13:41.522 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:13:41.522 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2' 2024-09-18T08:13:41.623 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0 2024-09-18T08:13:41.659 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:13:41.660 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:13:41.660 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0' 2024-09-18T08:13:41.681 INFO:teuthology.orchestra.run.smithi007.stderr:noscrub is set 2024-09-18T08:13:41.761 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2 2024-09-18T08:13:41.779 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid 779349ca-7593-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-18T08:13:41.796 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:13:41.797 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:13:41.797 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2' 2024-09-18T08:13:41.834 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:13:41 smithi007 ceph-mon[39249]: from='client.? 172.21.15.7:0/2758673785' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-18T08:13:41.835 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:13:41 smithi007 ceph-mon[29459]: from='client.? 172.21.15.7:0/2758673785' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-18T08:13:41.898 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7 2024-09-18T08:13:41.907 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:13:41 smithi050 ceph-mon[32371]: from='client.? 172.21.15.7:0/2758673785' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-18T08:13:41.938 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:13:41.938 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:13:41.939 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7' 2024-09-18T08:13:41.963 DEBUG:teuthology.orchestra.run.smithi050:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid 779349ca-7593-11ef-bceb-c7b262605968 -- ceph daemon osd.5 dump_blocked_ops 2024-09-18T08:13:42.040 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5 2024-09-18T08:13:42.090 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:13:42.090 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:13:42.091 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5' 2024-09-18T08:13:42.191 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0 2024-09-18T08:13:42.219 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:13:42.219 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:13:42.220 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0' 2024-09-18T08:13:42.321 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4 2024-09-18T08:13:42.377 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:13:42.378 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:13:42.378 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4' 2024-09-18T08:13:42.448 INFO:teuthology.orchestra.run.smithi050.stderr:Inferring config /var/lib/ceph/779349ca-7593-11ef-bceb-c7b262605968/mon.b/config 2024-09-18T08:13:42.479 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3 2024-09-18T08:13:42.515 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:13:42.516 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:13:42.516 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3' 2024-09-18T08:13:42.617 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7 2024-09-18T08:13:42.625 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/779349ca-7593-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T08:13:42.661 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:13:42.662 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:13:42.662 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7' 2024-09-18T08:13:42.683 INFO:teuthology.orchestra.run.smithi007.stderr:set pool 1 pgp_num to 11 2024-09-18T08:13:42.730 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:13:42 smithi007 ceph-mon[29459]: pgmap v525: 158 pgs: 30 active+undersized, 34 active+undersized+degraded, 94 active+clean; 4.0 GiB data, 15 GiB used, 701 GiB / 715 GiB avail; 1.0 KiB/s rd, 0 B/s wr, 1.36k op/s; 37257/197457 objects degraded (18.868%) 2024-09-18T08:13:42.730 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:13:42 smithi007 ceph-mon[29459]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-18T08:13:42.730 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:13:42 smithi007 ceph-mon[29459]: from='client.? 172.21.15.7:0/2758673785' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-18T08:13:42.730 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:13:42 smithi007 ceph-mon[29459]: osdmap e284: 8 total, 7 up, 8 in 2024-09-18T08:13:42.731 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:13:42 smithi007 ceph-mon[29459]: from='client.? 172.21.15.7:0/1065906969' entity='client.admin' cmd={"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num", "val": "11"} : dispatch 2024-09-18T08:13:42.731 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:13:42 smithi007 ceph-mon[39249]: pgmap v525: 158 pgs: 30 active+undersized, 34 active+undersized+degraded, 94 active+clean; 4.0 GiB data, 15 GiB used, 701 GiB / 715 GiB avail; 1.0 KiB/s rd, 0 B/s wr, 1.36k op/s; 37257/197457 objects degraded (18.868%) 2024-09-18T08:13:42.731 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:13:42 smithi007 ceph-mon[39249]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-18T08:13:42.731 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:13:42 smithi007 ceph-mon[39249]: from='client.? 172.21.15.7:0/2758673785' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-18T08:13:42.731 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:13:42 smithi007 ceph-mon[39249]: osdmap e284: 8 total, 7 up, 8 in 2024-09-18T08:13:42.731 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:13:42 smithi007 ceph-mon[39249]: from='client.? 172.21.15.7:0/1065906969' entity='client.admin' cmd={"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num", "val": "11"} : dispatch 2024-09-18T08:13:42.763 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5 2024-09-18T08:13:42.804 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:13:42.804 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:13:42.804 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5' 2024-09-18T08:13:42.905 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0 2024-09-18T08:13:42.922 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:13:42 smithi050 ceph-mon[32371]: pgmap v525: 158 pgs: 30 active+undersized, 34 active+undersized+degraded, 94 active+clean; 4.0 GiB data, 15 GiB used, 701 GiB / 715 GiB avail; 1.0 KiB/s rd, 0 B/s wr, 1.36k op/s; 37257/197457 objects degraded (18.868%) 2024-09-18T08:13:42.923 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:13:42 smithi050 ceph-mon[32371]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-18T08:13:42.923 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:13:42 smithi050 ceph-mon[32371]: from='client.? 172.21.15.7:0/2758673785' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-18T08:13:42.923 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:13:42 smithi050 ceph-mon[32371]: osdmap e284: 8 total, 7 up, 8 in 2024-09-18T08:13:42.923 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:13:42 smithi050 ceph-mon[32371]: from='client.? 172.21.15.7:0/1065906969' entity='client.admin' cmd={"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num", "val": "11"} : dispatch 2024-09-18T08:13:42.945 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:13:42.945 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:13:42.945 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0' 2024-09-18T08:13:43.046 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2 2024-09-18T08:13:43.076 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:13:43.076 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:13:43.077 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2' 2024-09-18T08:13:43.084 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2024-09-18T08:13:43.085 INFO:teuthology.orchestra.run.smithi050.stdout: "ops": [], 2024-09-18T08:13:43.085 INFO:teuthology.orchestra.run.smithi050.stdout: "complaint_time": 30, 2024-09-18T08:13:43.085 INFO:teuthology.orchestra.run.smithi050.stdout: "num_blocked_ops": 0 2024-09-18T08:13:43.085 INFO:teuthology.orchestra.run.smithi050.stdout:} 2024-09-18T08:13:43.177 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6 2024-09-18T08:13:43.216 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:13:43.217 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:13:43.217 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6' 2024-09-18T08:13:43.318 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2 2024-09-18T08:13:43.345 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:13:43.346 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:13:43.348 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2' 2024-09-18T08:13:43.449 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4 2024-09-18T08:13:43.491 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:13:43.491 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:13:43.492 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4' 2024-09-18T08:13:43.593 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0 2024-09-18T08:13:43.628 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:13:43.629 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:13:43.629 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0' 2024-09-18T08:13:43.730 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2 2024-09-18T08:13:43.759 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:13:43.759 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:13:43.760 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2' 2024-09-18T08:13:43.829 DEBUG:teuthology.orchestra.run.smithi050:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid 779349ca-7593-11ef-bceb-c7b262605968 -- ceph daemon osd.5 dump_historic_ops 2024-09-18T08:13:43.841 INFO:teuthology.orchestra.run.smithi007.stdout:osd.0: {} 2024-09-18T08:13:43.841 INFO:teuthology.orchestra.run.smithi007.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-18T08:13:43.842 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:13:43 smithi007 ceph-mon[29459]: from='client.? 172.21.15.7:0/1065906969' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num", "val": "11"}]': finished 2024-09-18T08:13:43.842 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:13:43 smithi007 ceph-mon[29459]: osdmap e285: 8 total, 7 up, 8 in 2024-09-18T08:13:43.842 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:13:43 smithi007 ceph-mon[29459]: pgmap v528: 158 pgs: 30 active+undersized, 34 active+undersized+degraded, 94 active+clean; 4.0 GiB data, 15 GiB used, 701 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 783 op/s; 37257/197457 objects degraded (18.868%) 2024-09-18T08:13:43.842 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:13:43 smithi007 ceph-mon[39249]: from='client.? 172.21.15.7:0/1065906969' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num", "val": "11"}]': finished 2024-09-18T08:13:43.842 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:13:43 smithi007 ceph-mon[39249]: osdmap e285: 8 total, 7 up, 8 in 2024-09-18T08:13:43.842 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:13:43 smithi007 ceph-mon[39249]: pgmap v528: 158 pgs: 30 active+undersized, 34 active+undersized+degraded, 94 active+clean; 4.0 GiB data, 15 GiB used, 701 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 783 op/s; 37257/197457 objects degraded (18.868%) 2024-09-18T08:13:43.842 INFO:teuthology.orchestra.run.smithi007.stderr:Error ENXIO: problem getting command descriptions from osd.1 2024-09-18T08:13:43.861 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5 2024-09-18T08:13:43.867 INFO:teuthology.orchestra.run.smithi007.stdout:osd.2: {} 2024-09-18T08:13:43.868 INFO:teuthology.orchestra.run.smithi007.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-18T08:13:43.888 INFO:teuthology.orchestra.run.smithi007.stdout:osd.3: {} 2024-09-18T08:13:43.889 INFO:teuthology.orchestra.run.smithi007.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-18T08:13:43.906 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:13:43 smithi050 ceph-mon[32371]: from='client.? 172.21.15.7:0/1065906969' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num", "val": "11"}]': finished 2024-09-18T08:13:43.906 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:13:43 smithi050 ceph-mon[32371]: osdmap e285: 8 total, 7 up, 8 in 2024-09-18T08:13:43.906 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:13:43 smithi050 ceph-mon[32371]: pgmap v528: 158 pgs: 30 active+undersized, 34 active+undersized+degraded, 94 active+clean; 4.0 GiB data, 15 GiB used, 701 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 783 op/s; 37257/197457 objects degraded (18.868%) 2024-09-18T08:13:43.917 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:13:43.917 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:13:43.918 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5' 2024-09-18T08:13:43.924 INFO:teuthology.orchestra.run.smithi007.stdout:osd.4: {} 2024-09-18T08:13:43.924 INFO:teuthology.orchestra.run.smithi007.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-18T08:13:43.957 INFO:teuthology.orchestra.run.smithi007.stdout:osd.5: {} 2024-09-18T08:13:43.957 INFO:teuthology.orchestra.run.smithi007.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-18T08:13:43.983 INFO:teuthology.orchestra.run.smithi007.stdout:osd.6: {} 2024-09-18T08:13:43.983 INFO:teuthology.orchestra.run.smithi007.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-18T08:13:44.012 INFO:teuthology.orchestra.run.smithi007.stdout:osd.7: {} 2024-09-18T08:13:44.012 INFO:teuthology.orchestra.run.smithi007.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-18T08:13:44.019 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0 2024-09-18T08:13:44.049 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:13:44.050 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:13:44.050 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0' 2024-09-18T08:13:44.151 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6 2024-09-18T08:13:44.206 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:13:44.207 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:13:44.207 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6' 2024-09-18T08:13:44.283 INFO:teuthology.orchestra.run.smithi050.stderr:Inferring config /var/lib/ceph/779349ca-7593-11ef-bceb-c7b262605968/mon.b/config 2024-09-18T08:13:44.308 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7 2024-09-18T08:13:44.363 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:13:44.363 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:13:44.364 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7' 2024-09-18T08:13:44.465 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7 2024-09-18T08:13:44.515 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:13:44.515 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid 779349ca-7593-11ef-bceb-c7b262605968 -- ceph osd set nodeep-scrub 2024-09-18T08:13:44.516 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:13:44.516 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7' 2024-09-18T08:13:44.617 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2 2024-09-18T08:13:44.650 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:13:44.651 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:13:44.652 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2' 2024-09-18T08:13:44.752 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5 2024-09-18T08:13:44.792 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:13:44.793 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:13:44.793 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5' 2024-09-18T08:13:44.894 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3 2024-09-18T08:13:44.920 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:13:44.921 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:13:44.921 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3' 2024-09-18T08:13:44.943 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:13:44 smithi007 ceph-mon[29459]: from='client.? 172.21.15.7:0/4059227488' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-18T08:13:44.943 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:13:44 smithi007 ceph-mon[29459]: Health check update: Degraded data redundancy: 37257/197457 objects degraded (18.868%), 34 pgs degraded, 64 pgs undersized (PG_DEGRADED) 2024-09-18T08:13:44.943 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:13:44 smithi007 ceph-mon[39249]: from='client.? 172.21.15.7:0/4059227488' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-18T08:13:44.944 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:13:44 smithi007 ceph-mon[39249]: Health check update: Degraded data redundancy: 37257/197457 objects degraded (18.868%), 34 pgs degraded, 64 pgs undersized (PG_DEGRADED) 2024-09-18T08:13:44.986 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-09-18T08:13:44.987 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-09-18T08:13:45.022 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0 2024-09-18T08:13:45.069 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:13:45.070 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:13:45.070 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0' 2024-09-18T08:13:45.091 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/779349ca-7593-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T08:13:45.156 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:13:44 smithi050 ceph-mon[32371]: from='client.? 172.21.15.7:0/4059227488' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-18T08:13:45.156 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:13:44 smithi050 ceph-mon[32371]: Health check update: Degraded data redundancy: 37257/197457 objects degraded (18.868%), 34 pgs degraded, 64 pgs undersized (PG_DEGRADED) 2024-09-18T08:13:45.171 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7 2024-09-18T08:13:45.211 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:13:45.212 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:13:45.212 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7' 2024-09-18T08:13:45.313 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7 2024-09-18T08:13:45.351 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:13:45.351 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:13:45.351 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7' 2024-09-18T08:13:45.452 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6 2024-09-18T08:13:45.496 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:13:45.497 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:13:45.497 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6' 2024-09-18T08:13:45.598 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7 2024-09-18T08:13:45.621 INFO:teuthology.orchestra.run.smithi050.stdout:ERROR: (22) Invalid argument 2024-09-18T08:13:45.621 INFO:teuthology.orchestra.run.smithi050.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-18T08:13:45.640 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:13:45.641 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:13:45.641 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7' 2024-09-18T08:13:45.742 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5 2024-09-18T08:13:45.782 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:13:45.782 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:13:45.782 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5' 2024-09-18T08:13:45.883 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7 2024-09-18T08:13:45.905 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:13:45 smithi050 ceph-mon[32371]: pgmap v529: 158 pgs: 30 active+undersized, 34 active+undersized+degraded, 94 active+clean; 3.6 GiB data, 14 GiB used, 702 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.64k op/s; 33332/176841 objects degraded (18.849%) 2024-09-18T08:13:45.921 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:13:45.922 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:13:45.922 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7' 2024-09-18T08:13:45.980 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:13:45 smithi007 ceph-mon[29459]: pgmap v529: 158 pgs: 30 active+undersized, 34 active+undersized+degraded, 94 active+clean; 3.6 GiB data, 14 GiB used, 702 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.64k op/s; 33332/176841 objects degraded (18.849%) 2024-09-18T08:13:45.980 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:13:45 smithi007 ceph-mon[39249]: pgmap v529: 158 pgs: 30 active+undersized, 34 active+undersized+degraded, 94 active+clean; 3.6 GiB data, 14 GiB used, 702 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.64k op/s; 33332/176841 objects degraded (18.849%) 2024-09-18T08:13:46.023 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7 2024-09-18T08:13:46.062 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:13:46.062 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:13:46.063 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7' 2024-09-18T08:13:46.164 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7 2024-09-18T08:13:46.202 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:13:46.203 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:13:46.203 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7' 2024-09-18T08:13:46.304 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3 2024-09-18T08:13:46.327 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:13:46.328 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:13:46.328 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3' 2024-09-18T08:13:46.429 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7 2024-09-18T08:13:46.463 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:13:46.464 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:13:46.465 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7' 2024-09-18T08:13:46.565 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3 2024-09-18T08:13:46.595 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:13:46.595 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:13:46.596 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3' 2024-09-18T08:13:46.697 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3 2024-09-18T08:13:46.720 INFO:teuthology.orchestra.run.smithi007.stderr:nodeep-scrub is set 2024-09-18T08:13:46.723 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:13:46.724 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:13:46.724 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3' 2024-09-18T08:13:46.825 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7 2024-09-18T08:13:46.866 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:13:46.867 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:13:46.868 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7' 2024-09-18T08:13:46.969 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0 2024-09-18T08:13:46.980 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:13:46 smithi007 ceph-mon[29459]: purged_snaps scrub starts 2024-09-18T08:13:46.980 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:13:46 smithi007 ceph-mon[29459]: purged_snaps scrub ok 2024-09-18T08:13:46.980 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:13:46 smithi007 ceph-mon[29459]: from='client.? 172.21.15.7:0/3338350931' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-18T08:13:46.980 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Sep 18 08:13:46 smithi007 ceph-779349ca-7593-11ef-bceb-c7b262605968-mgr-y[29688]: ::ffff:172.21.15.50 - - [18/Sep/2024:08:13:46] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-09-18T08:13:46.981 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:13:46 smithi007 ceph-mon[39249]: purged_snaps scrub starts 2024-09-18T08:13:46.981 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:13:46 smithi007 ceph-mon[39249]: purged_snaps scrub ok 2024-09-18T08:13:46.981 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:13:46 smithi007 ceph-mon[39249]: from='client.? 172.21.15.7:0/3338350931' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-18T08:13:46.988 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid 779349ca-7593-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-18T08:13:47.004 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:13:47.004 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:13:47.004 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0' 2024-09-18T08:13:47.105 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2 2024-09-18T08:13:47.141 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:13:47.142 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:13:47.142 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2' 2024-09-18T08:13:47.156 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:13:46 smithi050 ceph-mon[32371]: purged_snaps scrub starts 2024-09-18T08:13:47.156 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:13:46 smithi050 ceph-mon[32371]: purged_snaps scrub ok 2024-09-18T08:13:47.156 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:13:46 smithi050 ceph-mon[32371]: from='client.? 172.21.15.7:0/3338350931' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-18T08:13:47.242 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6 2024-09-18T08:13:47.283 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:13:47.284 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:13:47.284 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6' 2024-09-18T08:13:47.385 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6 2024-09-18T08:13:47.419 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:13:47.419 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:13:47.420 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6' 2024-09-18T08:13:47.493 DEBUG:teuthology.orchestra.run.smithi050:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid 779349ca-7593-11ef-bceb-c7b262605968 -- ceph daemon osd.6 dump_ops_in_flight 2024-09-18T08:13:47.520 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3 2024-09-18T08:13:47.555 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:13:47.555 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:13:47.556 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3' 2024-09-18T08:13:47.557 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/779349ca-7593-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T08:13:47.657 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2 2024-09-18T08:13:47.685 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:13:47.685 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:13:47.685 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2' 2024-09-18T08:13:47.786 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7 2024-09-18T08:13:47.846 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:13:47.847 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:13:47.847 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7' 2024-09-18T08:13:47.948 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7 2024-09-18T08:13:47.958 INFO:teuthology.orchestra.run.smithi050.stderr:Inferring config /var/lib/ceph/779349ca-7593-11ef-bceb-c7b262605968/mon.b/config 2024-09-18T08:13:47.980 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:13:47 smithi007 ceph-mon[29459]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-18T08:13:47.980 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:13:47 smithi007 ceph-mon[29459]: from='client.? 172.21.15.7:0/3338350931' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-18T08:13:47.980 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:13:47 smithi007 ceph-mon[29459]: osdmap e286: 8 total, 7 up, 8 in 2024-09-18T08:13:47.981 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:13:47 smithi007 ceph-mon[29459]: pgmap v531: 158 pgs: 30 active+undersized, 34 active+undersized+degraded, 94 active+clean; 3.6 GiB data, 14 GiB used, 702 GiB / 715 GiB avail; 852 B/s rd, 0 B/s wr, 1.15k op/s; 33332/176841 objects degraded (18.849%) 2024-09-18T08:13:47.981 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:13:47 smithi007 ceph-mon[39249]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-18T08:13:47.981 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:13:47 smithi007 ceph-mon[39249]: from='client.? 172.21.15.7:0/3338350931' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-18T08:13:47.981 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:13:47 smithi007 ceph-mon[39249]: osdmap e286: 8 total, 7 up, 8 in 2024-09-18T08:13:47.981 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:13:47 smithi007 ceph-mon[39249]: pgmap v531: 158 pgs: 30 active+undersized, 34 active+undersized+degraded, 94 active+clean; 3.6 GiB data, 14 GiB used, 702 GiB / 715 GiB avail; 852 B/s rd, 0 B/s wr, 1.15k op/s; 33332/176841 objects degraded (18.849%) 2024-09-18T08:13:48.014 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:13:48.014 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:13:48.015 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7' 2024-09-18T08:13:48.096 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:13:47 smithi050 ceph-mon[32371]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-18T08:13:48.096 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:13:47 smithi050 ceph-mon[32371]: from='client.? 172.21.15.7:0/3338350931' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-18T08:13:48.096 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:13:47 smithi050 ceph-mon[32371]: osdmap e286: 8 total, 7 up, 8 in 2024-09-18T08:13:48.097 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:13:47 smithi050 ceph-mon[32371]: pgmap v531: 158 pgs: 30 active+undersized, 34 active+undersized+degraded, 94 active+clean; 3.6 GiB data, 14 GiB used, 702 GiB / 715 GiB avail; 852 B/s rd, 0 B/s wr, 1.15k op/s; 33332/176841 objects degraded (18.849%) 2024-09-18T08:13:48.116 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4 2024-09-18T08:13:48.164 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:13:48.164 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:13:48.165 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4' 2024-09-18T08:13:48.167 INFO:teuthology.orchestra.run.smithi007.stdout:osd.0: {} 2024-09-18T08:13:48.167 INFO:teuthology.orchestra.run.smithi007.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-18T08:13:48.167 INFO:teuthology.orchestra.run.smithi007.stderr:Error ENXIO: problem getting command descriptions from osd.1 2024-09-18T08:13:48.190 INFO:teuthology.orchestra.run.smithi007.stdout:osd.2: {} 2024-09-18T08:13:48.190 INFO:teuthology.orchestra.run.smithi007.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-18T08:13:48.216 INFO:teuthology.orchestra.run.smithi007.stdout:osd.3: {} 2024-09-18T08:13:48.217 INFO:teuthology.orchestra.run.smithi007.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-18T08:13:48.247 INFO:teuthology.orchestra.run.smithi007.stdout:osd.4: {} 2024-09-18T08:13:48.248 INFO:teuthology.orchestra.run.smithi007.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-18T08:13:48.265 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3 2024-09-18T08:13:48.280 INFO:teuthology.orchestra.run.smithi007.stdout:osd.5: {} 2024-09-18T08:13:48.280 INFO:teuthology.orchestra.run.smithi007.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-18T08:13:48.295 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:13:48.296 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:13:48.296 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3' 2024-09-18T08:13:48.315 INFO:teuthology.orchestra.run.smithi007.stdout:osd.6: {} 2024-09-18T08:13:48.316 INFO:teuthology.orchestra.run.smithi007.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-18T08:13:48.343 INFO:teuthology.orchestra.run.smithi007.stdout:osd.7: {} 2024-09-18T08:13:48.343 INFO:teuthology.orchestra.run.smithi007.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-18T08:13:48.397 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6 2024-09-18T08:13:48.445 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:13:48.446 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:13:48.446 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6' 2024-09-18T08:13:48.547 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0 2024-09-18T08:13:48.581 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:13:48.582 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:13:48.582 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0' 2024-09-18T08:13:48.587 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2024-09-18T08:13:48.587 INFO:teuthology.orchestra.run.smithi050.stdout: "ops": [], 2024-09-18T08:13:48.587 INFO:teuthology.orchestra.run.smithi050.stdout: "num_ops": 0 2024-09-18T08:13:48.587 INFO:teuthology.orchestra.run.smithi050.stdout:} 2024-09-18T08:13:48.626 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 5, 7, 6, 4] out_osds: [] dead_osds: [1] live_osds: [0, 2, 3, 4, 5, 6, 7] 2024-09-18T08:13:48.626 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-18T08:13:48.626 INFO:tasks.thrashosds.thrasher:primary_affinity 2024-09-18T08:13:48.626 INFO:tasks.thrashosds.thrasher:Setting osd 7 primary_affinity to 0.474176 2024-09-18T08:13:48.626 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid 779349ca-7593-11ef-bceb-c7b262605968 -- ceph osd primary-affinity 7 0.47417594159104537 2024-09-18T08:13:48.683 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0 2024-09-18T08:13:48.716 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:13:48.717 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:13:48.717 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0' 2024-09-18T08:13:48.818 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4 2024-09-18T08:13:48.858 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:13:48.859 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:13:48.860 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4' 2024-09-18T08:13:48.961 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4 2024-09-18T08:13:48.980 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:13:48 smithi007 ceph-mon[29459]: from='client.? 172.21.15.7:0/1310398174' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-18T08:13:48.980 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:13:48 smithi007 ceph-mon[29459]: from='mgr.24470 172.21.15.7:0/2594390261' entity='mgr.y' 2024-09-18T08:13:48.980 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:13:48 smithi007 ceph-mon[29459]: from='mgr.24470 172.21.15.7:0/2594390261' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-18T08:13:48.981 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:13:48 smithi007 ceph-mon[39249]: from='client.? 172.21.15.7:0/1310398174' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-18T08:13:48.981 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:13:48 smithi007 ceph-mon[39249]: from='mgr.24470 172.21.15.7:0/2594390261' entity='mgr.y' 2024-09-18T08:13:48.981 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:13:48 smithi007 ceph-mon[39249]: from='mgr.24470 172.21.15.7:0/2594390261' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-18T08:13:48.998 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:13:48.999 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:13:48.999 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4' 2024-09-18T08:13:49.077 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-09-18T08:13:49.077 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-09-18T08:13:49.100 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2 2024-09-18T08:13:49.144 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:13:49.145 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:13:49.145 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2' 2024-09-18T08:13:49.155 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:13:48 smithi050 ceph-mon[32371]: from='client.? 172.21.15.7:0/1310398174' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-18T08:13:49.155 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:13:48 smithi050 ceph-mon[32371]: from='mgr.24470 172.21.15.7:0/2594390261' entity='mgr.y' 2024-09-18T08:13:49.155 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:13:48 smithi050 ceph-mon[32371]: from='mgr.24470 172.21.15.7:0/2594390261' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-18T08:13:49.160 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/779349ca-7593-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T08:13:49.246 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5 2024-09-18T08:13:49.294 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:13:49.295 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:13:49.295 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5' 2024-09-18T08:13:49.396 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2 2024-09-18T08:13:49.429 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:13:49.429 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:13:49.430 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2' 2024-09-18T08:13:49.467 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid 779349ca-7593-11ef-bceb-c7b262605968 -- ceph osd unset noscrub 2024-09-18T08:13:49.531 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0 2024-09-18T08:13:49.569 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:13:49.570 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:13:49.570 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0' 2024-09-18T08:13:49.656 INFO:journalctl@ceph.iscsi.iscsi.a.smithi050.stdout:Sep 18 08:13:49 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-iscsi-iscsi-a[57782]: debug there is no tcmu-runner data available 2024-09-18T08:13:49.670 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6 2024-09-18T08:13:49.711 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:13:49.712 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:13:49.712 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6' 2024-09-18T08:13:49.754 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:13:49 smithi050 ceph-mon[32371]: Health check update: Degraded data redundancy: 33332/176841 objects degraded (18.849%), 34 pgs degraded, 64 pgs undersized (PG_DEGRADED) 2024-09-18T08:13:49.754 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:13:49 smithi050 ceph-mon[32371]: pgmap v532: 158 pgs: 30 active+undersized, 34 active+undersized+degraded, 94 active+clean; 3.2 GiB data, 13 GiB used, 703 GiB / 715 GiB avail; 1.3 KiB/s rd, 0 B/s wr, 1.81k op/s; 29452/156102 objects degraded (18.867%) 2024-09-18T08:13:49.754 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:13:49 smithi050 ceph-mon[32371]: osdmap e287: 8 total, 7 up, 8 in 2024-09-18T08:13:49.754 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:13:49 smithi050 ceph-mon[32371]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-18T08:13:49.803 DEBUG:teuthology.orchestra.run.smithi050:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid 779349ca-7593-11ef-bceb-c7b262605968 -- ceph daemon osd.6 dump_blocked_ops 2024-09-18T08:13:49.813 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7 2024-09-18T08:13:49.897 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/779349ca-7593-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T08:13:49.901 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:13:49.902 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:13:49.902 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7' 2024-09-18T08:13:49.940 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:13:49 smithi007 ceph-mon[29459]: Health check update: Degraded data redundancy: 33332/176841 objects degraded (18.849%), 34 pgs degraded, 64 pgs undersized (PG_DEGRADED) 2024-09-18T08:13:49.940 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:13:49 smithi007 ceph-mon[29459]: pgmap v532: 158 pgs: 30 active+undersized, 34 active+undersized+degraded, 94 active+clean; 3.2 GiB data, 13 GiB used, 703 GiB / 715 GiB avail; 1.3 KiB/s rd, 0 B/s wr, 1.81k op/s; 29452/156102 objects degraded (18.867%) 2024-09-18T08:13:49.940 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:13:49 smithi007 ceph-mon[29459]: osdmap e287: 8 total, 7 up, 8 in 2024-09-18T08:13:49.940 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:13:49 smithi007 ceph-mon[29459]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-18T08:13:49.940 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:13:49 smithi007 ceph-mon[39249]: Health check update: Degraded data redundancy: 33332/176841 objects degraded (18.849%), 34 pgs degraded, 64 pgs undersized (PG_DEGRADED) 2024-09-18T08:13:49.940 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:13:49 smithi007 ceph-mon[39249]: pgmap v532: 158 pgs: 30 active+undersized, 34 active+undersized+degraded, 94 active+clean; 3.2 GiB data, 13 GiB used, 703 GiB / 715 GiB avail; 1.3 KiB/s rd, 0 B/s wr, 1.81k op/s; 29452/156102 objects degraded (18.867%) 2024-09-18T08:13:49.940 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:13:49 smithi007 ceph-mon[39249]: osdmap e287: 8 total, 7 up, 8 in 2024-09-18T08:13:49.940 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:13:49 smithi007 ceph-mon[39249]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-18T08:13:50.003 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2 2024-09-18T08:13:50.040 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:13:50.040 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:13:50.041 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2' 2024-09-18T08:13:50.142 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4 2024-09-18T08:13:50.205 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:13:50.206 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:13:50.206 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4' 2024-09-18T08:13:50.244 INFO:teuthology.orchestra.run.smithi050.stderr:Inferring config /var/lib/ceph/779349ca-7593-11ef-bceb-c7b262605968/mon.b/config 2024-09-18T08:13:50.307 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0 2024-09-18T08:13:50.343 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:13:50.344 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:13:50.345 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0' 2024-09-18T08:13:50.445 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5 2024-09-18T08:13:50.481 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:13:50.482 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:13:50.482 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5' 2024-09-18T08:13:50.583 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6 2024-09-18T08:13:50.624 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:13:50.625 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:13:50.625 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6' 2024-09-18T08:13:50.726 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4 2024-09-18T08:13:50.732 INFO:teuthology.orchestra.run.smithi007.stderr:set osd.7 primary-affinity to 0.474176 (8310752) 2024-09-18T08:13:50.775 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:13:50.776 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:13:50.776 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4' 2024-09-18T08:13:50.777 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:13:50 smithi007 ceph-mon[29459]: from='client.? 172.21.15.7:0/1200088615' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 7, "weight": 0.47417594159104537} : dispatch 2024-09-18T08:13:50.777 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:13:50 smithi007 ceph-mon[39249]: from='client.? 172.21.15.7:0/1200088615' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 7, "weight": 0.47417594159104537} : dispatch 2024-09-18T08:13:50.869 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2024-09-18T08:13:50.869 INFO:teuthology.orchestra.run.smithi050.stdout: "ops": [], 2024-09-18T08:13:50.869 INFO:teuthology.orchestra.run.smithi050.stdout: "complaint_time": 30, 2024-09-18T08:13:50.869 INFO:teuthology.orchestra.run.smithi050.stdout: "num_blocked_ops": 0 2024-09-18T08:13:50.869 INFO:teuthology.orchestra.run.smithi050.stdout:} 2024-09-18T08:13:50.877 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7 2024-09-18T08:13:50.912 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:13:50.912 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:13:50.913 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7' 2024-09-18T08:13:51.013 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7 2024-09-18T08:13:51.036 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:13:51.036 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:13:51.037 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7' 2024-09-18T08:13:51.078 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid 779349ca-7593-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-18T08:13:51.137 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7 2024-09-18T08:13:51.155 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:13:50 smithi050 ceph-mon[32371]: from='client.? 172.21.15.7:0/1200088615' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 7, "weight": 0.47417594159104537} : dispatch 2024-09-18T08:13:51.160 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:13:51.161 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:13:51.161 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7' 2024-09-18T08:13:51.262 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5 2024-09-18T08:13:51.284 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:13:51.285 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:13:51.285 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5' 2024-09-18T08:13:51.386 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3 2024-09-18T08:13:51.408 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:13:51.409 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:13:51.409 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3' 2024-09-18T08:13:51.510 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5 2024-09-18T08:13:51.533 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:13:51.534 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:13:51.534 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5' 2024-09-18T08:13:51.573 DEBUG:teuthology.orchestra.run.smithi050:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid 779349ca-7593-11ef-bceb-c7b262605968 -- ceph daemon osd.6 dump_historic_ops 2024-09-18T08:13:51.635 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5 2024-09-18T08:13:51.678 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:13:51.679 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:13:51.679 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5' 2024-09-18T08:13:51.733 INFO:teuthology.orchestra.run.smithi007.stderr:noscrub is unset 2024-09-18T08:13:51.780 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3 2024-09-18T08:13:51.808 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:13:51.809 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:13:51.809 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3' 2024-09-18T08:13:51.910 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4 2024-09-18T08:13:51.954 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:13:51.955 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:13:51.955 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4' 2024-09-18T08:13:51.974 INFO:teuthology.orchestra.run.smithi050.stderr:Inferring config /var/lib/ceph/779349ca-7593-11ef-bceb-c7b262605968/mon.b/config 2024-09-18T08:13:51.980 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:13:51 smithi007 ceph-mon[29459]: from='client.? 172.21.15.7:0/1200088615' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 7, "weight": 0.47417594159104537}]': finished 2024-09-18T08:13:51.980 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:13:51 smithi007 ceph-mon[29459]: osdmap e288: 8 total, 7 up, 8 in 2024-09-18T08:13:51.980 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:13:51 smithi007 ceph-mon[29459]: from='client.? 172.21.15.7:0/3121730880' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-18T08:13:51.980 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:13:51 smithi007 ceph-mon[29459]: pgmap v535: 158 pgs: 30 active+undersized, 34 active+undersized+degraded, 94 active+clean; 3.2 GiB data, 13 GiB used, 703 GiB / 715 GiB avail; 852 B/s rd, 0 B/s wr, 1.15k op/s; 29452/156102 objects degraded (18.867%) 2024-09-18T08:13:51.980 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:13:51 smithi007 ceph-mon[29459]: from='mgr.24470 172.21.15.7:0/2594390261' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-18T08:13:51.980 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:13:51 smithi007 ceph-mon[29459]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-18T08:13:51.980 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:13:51 smithi007 ceph-mon[29459]: from='client.? 172.21.15.7:0/3121730880' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-18T08:13:51.981 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:13:51 smithi007 ceph-mon[29459]: osdmap e289: 8 total, 7 up, 8 in 2024-09-18T08:13:51.981 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:13:51 smithi007 ceph-mon[39249]: from='client.? 172.21.15.7:0/1200088615' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 7, "weight": 0.47417594159104537}]': finished 2024-09-18T08:13:51.981 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:13:51 smithi007 ceph-mon[39249]: osdmap e288: 8 total, 7 up, 8 in 2024-09-18T08:13:51.981 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:13:51 smithi007 ceph-mon[39249]: from='client.? 172.21.15.7:0/3121730880' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-18T08:13:51.981 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:13:51 smithi007 ceph-mon[39249]: pgmap v535: 158 pgs: 30 active+undersized, 34 active+undersized+degraded, 94 active+clean; 3.2 GiB data, 13 GiB used, 703 GiB / 715 GiB avail; 852 B/s rd, 0 B/s wr, 1.15k op/s; 29452/156102 objects degraded (18.867%) 2024-09-18T08:13:51.981 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:13:51 smithi007 ceph-mon[39249]: from='mgr.24470 172.21.15.7:0/2594390261' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-18T08:13:51.981 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:13:51 smithi007 ceph-mon[39249]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-18T08:13:51.981 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:13:51 smithi007 ceph-mon[39249]: from='client.? 172.21.15.7:0/3121730880' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-18T08:13:51.981 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:13:51 smithi007 ceph-mon[39249]: osdmap e289: 8 total, 7 up, 8 in 2024-09-18T08:13:52.056 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7 2024-09-18T08:13:52.106 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:13:52.107 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:13:52.107 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7' 2024-09-18T08:13:52.109 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:13:51 smithi050 ceph-mon[32371]: from='client.? 172.21.15.7:0/1200088615' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 7, "weight": 0.47417594159104537}]': finished 2024-09-18T08:13:52.109 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:13:51 smithi050 ceph-mon[32371]: osdmap e288: 8 total, 7 up, 8 in 2024-09-18T08:13:52.109 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:13:51 smithi050 ceph-mon[32371]: from='client.? 172.21.15.7:0/3121730880' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-18T08:13:52.109 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:13:51 smithi050 ceph-mon[32371]: pgmap v535: 158 pgs: 30 active+undersized, 34 active+undersized+degraded, 94 active+clean; 3.2 GiB data, 13 GiB used, 703 GiB / 715 GiB avail; 852 B/s rd, 0 B/s wr, 1.15k op/s; 29452/156102 objects degraded (18.867%) 2024-09-18T08:13:52.110 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:13:51 smithi050 ceph-mon[32371]: from='mgr.24470 172.21.15.7:0/2594390261' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-18T08:13:52.110 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:13:51 smithi050 ceph-mon[32371]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-18T08:13:52.110 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:13:51 smithi050 ceph-mon[32371]: from='client.? 172.21.15.7:0/3121730880' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-18T08:13:52.110 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:13:51 smithi050 ceph-mon[32371]: osdmap e289: 8 total, 7 up, 8 in 2024-09-18T08:13:52.208 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5 2024-09-18T08:13:52.248 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:13:52.249 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:13:52.249 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5' 2024-09-18T08:13:52.350 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4 2024-09-18T08:13:52.461 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:13:52.462 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:13:52.462 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4' 2024-09-18T08:13:52.563 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0 2024-09-18T08:13:52.595 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:13:52.596 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:13:52.596 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0' 2024-09-18T08:13:52.617 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2024-09-18T08:13:52.617 INFO:teuthology.orchestra.run.smithi050.stdout: "size": 20, 2024-09-18T08:13:52.618 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 600, 2024-09-18T08:13:52.618 INFO:teuthology.orchestra.run.smithi050.stdout: "ops": [ 2024-09-18T08:13:52.618 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:13:52.618 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.14805.0:38000 7.7 7:e509d896:::benchmark_data_smithi007_73155_object37999:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e94)", 2024-09-18T08:13:52.618 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-09-18T08:04:23.049009+0000", 2024-09-18T08:13:52.618 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 569.56392226000003, 2024-09-18T08:13:52.618 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.027565702000000001, 2024-09-18T08:13:52.618 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-09-18T08:13:52.618 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-09-18T08:13:52.618 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T08:13:52.618 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2024-09-18T08:13:52.618 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.14805", 2024-09-18T08:13:52.619 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "172.21.15.7:0/640320436", 2024-09-18T08:13:52.619 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 38000 2024-09-18T08:13:52.619 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:13:52.619 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-09-18T08:13:52.619 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:13:52.619 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-09-18T08:13:52.619 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:23.049009+0000", 2024-09-18T08:13:52.619 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-18T08:13:52.619 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:13:52.619 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:13:52.619 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-09-18T08:13:52.619 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:23.049009+0000", 2024-09-18T08:13:52.619 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-18T08:13:52.620 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:13:52.620 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:13:52.620 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-09-18T08:13:52.620 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:23.049010+0000", 2024-09-18T08:13:52.620 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.11e-06 2024-09-18T08:13:52.620 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:13:52.620 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:13:52.620 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-09-18T08:13:52.620 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:23.068284+0000", 2024-09-18T08:13:52.621 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.019274264999999999 2024-09-18T08:13:52.621 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:13:52.621 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:13:52.621 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-09-18T08:13:52.621 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:23.068323+0000", 2024-09-18T08:13:52.621 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 3.8167999999999998e-05 2024-09-18T08:13:52.621 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:13:52.621 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:13:52.621 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-09-18T08:13:52.621 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:23.068329+0000", 2024-09-18T08:13:52.621 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 6.331e-06 2024-09-18T08:13:52.621 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:13:52.622 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:13:52.622 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-18T08:13:52.622 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:23.068399+0000", 2024-09-18T08:13:52.622 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 7.0055999999999995e-05 2024-09-18T08:13:52.622 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:13:52.622 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:13:52.622 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-09-18T08:13:52.622 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:23.068503+0000", 2024-09-18T08:13:52.622 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000104206 2024-09-18T08:13:52.622 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:13:52.622 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:13:52.622 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "waiting for subops from 0,3", 2024-09-18T08:13:52.622 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:23.068640+0000", 2024-09-18T08:13:52.623 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00013645000000000001 2024-09-18T08:13:52.623 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:13:52.623 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:13:52.623 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "op_commit", 2024-09-18T08:13:52.623 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:23.073270+0000", 2024-09-18T08:13:52.623 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0046298950000000002 2024-09-18T08:13:52.623 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:13:52.623 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:13:52.623 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:13:52.623 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:23.073311+0000", 2024-09-18T08:13:52.623 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4.1406999999999998e-05 2024-09-18T08:13:52.623 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:13:52.623 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:13:52.623 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:13:52.624 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:23.076521+0000", 2024-09-18T08:13:52.624 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.003209862 2024-09-18T08:13:52.624 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:13:52.624 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:13:52.624 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "commit_sent", 2024-09-18T08:13:52.624 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:23.076552+0000", 2024-09-18T08:13:52.624 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 3.1698e-05 2024-09-18T08:13:52.624 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:13:52.624 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:13:52.624 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-09-18T08:13:52.624 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:23.076575+0000", 2024-09-18T08:13:52.624 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 2.2254000000000001e-05 2024-09-18T08:13:52.624 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-18T08:13:52.625 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-09-18T08:13:52.625 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-18T08:13:52.625 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:13:52.625 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:13:52.625 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.14805.0:39064 7.7 7:ef833039:::benchmark_data_smithi007_73155_object39063:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e95)", 2024-09-18T08:13:52.625 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-09-18T08:04:23.658960+0000", 2024-09-18T08:13:52.625 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 568.95397098800004, 2024-09-18T08:13:52.625 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.029075624000000001, 2024-09-18T08:13:52.625 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-09-18T08:13:52.625 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-09-18T08:13:52.625 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T08:13:52.625 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2024-09-18T08:13:52.625 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.14805", 2024-09-18T08:13:52.626 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "172.21.15.7:0/640320436", 2024-09-18T08:13:52.626 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 39064 2024-09-18T08:13:52.626 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:13:52.626 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-09-18T08:13:52.626 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:13:52.626 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-09-18T08:13:52.626 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:23.658960+0000", 2024-09-18T08:13:52.626 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-18T08:13:52.626 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:13:52.626 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:13:52.626 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-09-18T08:13:52.626 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:23.658960+0000", 2024-09-18T08:13:52.626 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-18T08:13:52.627 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:13:52.627 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:13:52.627 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-09-18T08:13:52.627 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:23.658961+0000", 2024-09-18T08:13:52.627 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 6.7899999999999998e-07 2024-09-18T08:13:52.627 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:13:52.627 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:13:52.627 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-09-18T08:13:52.627 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:23.658991+0000", 2024-09-18T08:13:52.627 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 3.0168999999999999e-05 2024-09-18T08:13:52.627 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:13:52.628 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:13:52.628 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-09-18T08:13:52.628 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:23.659001+0000", 2024-09-18T08:13:52.628 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 9.6220000000000004e-06 2024-09-18T08:13:52.628 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:13:52.628 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:13:52.629 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-09-18T08:13:52.629 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:23.659010+0000", 2024-09-18T08:13:52.629 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 8.9160000000000007e-06 2024-09-18T08:13:52.629 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:13:52.629 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:13:52.629 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-18T08:13:52.629 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:23.661923+0000", 2024-09-18T08:13:52.629 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.002913279 2024-09-18T08:13:52.629 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:13:52.629 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:13:52.629 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-09-18T08:13:52.629 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:23.662030+0000", 2024-09-18T08:13:52.629 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00010665700000000001 2024-09-18T08:13:52.629 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:13:52.630 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:13:52.630 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "waiting for subops from 0,3", 2024-09-18T08:13:52.630 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:23.662167+0000", 2024-09-18T08:13:52.630 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00013728300000000001 2024-09-18T08:13:52.630 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:13:52.630 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:13:52.630 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "op_commit", 2024-09-18T08:13:52.630 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:23.663255+0000", 2024-09-18T08:13:52.630 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0010883900000000001 2024-09-18T08:13:52.630 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:13:52.630 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:13:52.630 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:13:52.630 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:23.671279+0000", 2024-09-18T08:13:52.630 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0080233749999999993 2024-09-18T08:13:52.630 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:13:52.631 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:13:52.631 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:13:52.631 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:23.687965+0000", 2024-09-18T08:13:52.631 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.016686590000000001 2024-09-18T08:13:52.631 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:13:52.631 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:13:52.631 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "commit_sent", 2024-09-18T08:13:52.631 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:23.688005+0000", 2024-09-18T08:13:52.631 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 3.9604000000000002e-05 2024-09-18T08:13:52.631 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:13:52.631 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:13:52.631 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-09-18T08:13:52.631 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:23.688036+0000", 2024-09-18T08:13:52.632 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 3.1059999999999997e-05 2024-09-18T08:13:52.632 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-18T08:13:52.632 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-09-18T08:13:52.632 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-18T08:13:52.632 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:13:52.632 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:13:52.632 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.14805.0:39805 7.7 7:e9cd36b3:::benchmark_data_smithi007_73155_object39804:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e95)", 2024-09-18T08:13:52.632 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-09-18T08:04:24.061655+0000", 2024-09-18T08:13:52.632 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 568.55127630699997, 2024-09-18T08:13:52.633 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.036883131, 2024-09-18T08:13:52.633 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-09-18T08:13:52.633 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-09-18T08:13:52.633 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T08:13:52.633 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2024-09-18T08:13:52.633 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.14805", 2024-09-18T08:13:52.633 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "172.21.15.7:0/640320436", 2024-09-18T08:13:52.633 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 39805 2024-09-18T08:13:52.633 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:13:52.633 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-09-18T08:13:52.633 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:13:52.634 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-09-18T08:13:52.634 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:24.061655+0000", 2024-09-18T08:13:52.634 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-18T08:13:52.634 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:13:52.634 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:13:52.634 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-09-18T08:13:52.634 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:24.061655+0000", 2024-09-18T08:13:52.634 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-18T08:13:52.634 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:13:52.634 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:13:52.634 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-09-18T08:13:52.634 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:24.061656+0000", 2024-09-18T08:13:52.634 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.4449999999999999e-06 2024-09-18T08:13:52.635 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:13:52.635 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:13:52.635 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-09-18T08:13:52.635 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:24.061686+0000", 2024-09-18T08:13:52.635 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 2.9984e-05 2024-09-18T08:13:52.635 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:13:52.635 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:13:52.635 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-09-18T08:13:52.635 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:24.061699+0000", 2024-09-18T08:13:52.635 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.2850000000000001e-05 2024-09-18T08:13:52.635 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:13:52.635 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:13:52.635 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-09-18T08:13:52.635 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:24.061706+0000", 2024-09-18T08:13:52.636 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 6.8560000000000001e-06 2024-09-18T08:13:52.636 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:13:52.636 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:13:52.636 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-18T08:13:52.636 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:24.061755+0000", 2024-09-18T08:13:52.636 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4.9001000000000001e-05 2024-09-18T08:13:52.636 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:13:52.636 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:13:52.636 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-09-18T08:13:52.636 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:24.061896+0000", 2024-09-18T08:13:52.636 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00014040499999999999 2024-09-18T08:13:52.636 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:13:52.637 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:13:52.637 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "waiting for subops from 0,3", 2024-09-18T08:13:52.637 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:24.062006+0000", 2024-09-18T08:13:52.637 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000110813 2024-09-18T08:13:52.637 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:13:52.637 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:13:52.637 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:13:52.637 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:24.065677+0000", 2024-09-18T08:13:52.637 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0036702800000000002 2024-09-18T08:13:52.637 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:13:52.637 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:13:52.637 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "op_commit", 2024-09-18T08:13:52.637 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:24.065694+0000", 2024-09-18T08:13:52.637 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.7876e-05 2024-09-18T08:13:52.638 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:13:52.638 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:13:52.638 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:13:52.638 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:24.098357+0000", 2024-09-18T08:13:52.638 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.032662140999999999 2024-09-18T08:13:52.638 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:13:52.638 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:13:52.638 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "commit_sent", 2024-09-18T08:13:52.638 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:24.098447+0000", 2024-09-18T08:13:52.638 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 9.0724999999999996e-05 2024-09-18T08:13:52.638 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:13:52.638 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:13:52.638 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-09-18T08:13:52.638 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:24.098538+0000", 2024-09-18T08:13:52.638 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 9.0754999999999994e-05 2024-09-18T08:13:52.639 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-18T08:13:52.639 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-09-18T08:13:52.639 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-18T08:13:52.639 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:13:52.639 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:13:52.639 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.14805.0:39807 7.d 7:bf1e27dc:::benchmark_data_smithi007_73155_object39806:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e95)", 2024-09-18T08:13:52.639 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-09-18T08:04:24.061827+0000", 2024-09-18T08:13:52.639 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 568.551104012, 2024-09-18T08:13:52.639 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.037358672000000002, 2024-09-18T08:13:52.639 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-09-18T08:13:52.639 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-09-18T08:13:52.639 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T08:13:52.639 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2024-09-18T08:13:52.640 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.14805", 2024-09-18T08:13:52.640 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "172.21.15.7:0/640320436", 2024-09-18T08:13:52.640 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 39807 2024-09-18T08:13:52.640 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:13:52.640 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-09-18T08:13:52.640 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:13:52.640 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-09-18T08:13:52.640 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:24.061827+0000", 2024-09-18T08:13:52.640 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-18T08:13:52.640 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:13:52.640 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:13:52.640 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-09-18T08:13:52.640 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:24.061827+0000", 2024-09-18T08:13:52.640 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-18T08:13:52.641 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:13:52.641 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:13:52.641 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-09-18T08:13:52.641 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:24.061828+0000", 2024-09-18T08:13:52.641 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 7.9899999999999999e-07 2024-09-18T08:13:52.641 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:13:52.641 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:13:52.641 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-09-18T08:13:52.641 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:24.061888+0000", 2024-09-18T08:13:52.641 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 5.9497000000000003e-05 2024-09-18T08:13:52.641 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:13:52.641 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:13:52.641 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-09-18T08:13:52.642 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:24.061904+0000", 2024-09-18T08:13:52.642 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.6393999999999999e-05 2024-09-18T08:13:52.642 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:13:52.642 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:13:52.642 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-09-18T08:13:52.642 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:24.061912+0000", 2024-09-18T08:13:52.642 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 8.0909999999999993e-06 2024-09-18T08:13:52.642 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:13:52.642 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:13:52.642 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-18T08:13:52.642 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:24.061968+0000", 2024-09-18T08:13:52.642 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 5.5543e-05 2024-09-18T08:13:52.642 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:13:52.643 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:13:52.643 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-09-18T08:13:52.643 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:24.062061+0000", 2024-09-18T08:13:52.643 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 9.3558000000000001e-05 2024-09-18T08:13:52.643 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:13:52.643 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:13:52.643 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "waiting for subops from 0,1", 2024-09-18T08:13:52.643 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:24.062163+0000", 2024-09-18T08:13:52.643 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000101903 2024-09-18T08:13:52.643 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:13:52.643 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:13:52.643 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "op_commit", 2024-09-18T08:13:52.643 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:24.064018+0000", 2024-09-18T08:13:52.643 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.001855071 2024-09-18T08:13:52.644 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:13:52.644 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:13:52.644 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:13:52.644 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:24.064283+0000", 2024-09-18T08:13:52.644 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000265173 2024-09-18T08:13:52.644 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:13:52.644 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:13:52.644 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:13:52.644 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:24.099043+0000", 2024-09-18T08:13:52.644 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.034760159999999998 2024-09-18T08:13:52.644 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:13:52.644 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:13:52.644 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "commit_sent", 2024-09-18T08:13:52.645 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:24.099100+0000", 2024-09-18T08:13:52.645 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 5.6147000000000003e-05 2024-09-18T08:13:52.645 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:13:52.645 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:13:52.645 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-09-18T08:13:52.645 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:24.099186+0000", 2024-09-18T08:13:52.645 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 8.6335999999999995e-05 2024-09-18T08:13:52.645 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-18T08:13:52.645 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-09-18T08:13:52.645 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-18T08:13:52.646 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:13:52.646 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:13:52.646 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.14805.0:39808 7.d 7:b98cab8f:::benchmark_data_smithi007_73155_object39807:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e95)", 2024-09-18T08:13:52.646 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-09-18T08:04:24.067964+0000", 2024-09-18T08:13:52.646 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 568.54496757499999, 2024-09-18T08:13:52.646 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.034258180999999999, 2024-09-18T08:13:52.646 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-09-18T08:13:52.646 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-09-18T08:13:52.646 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T08:13:52.646 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2024-09-18T08:13:52.646 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.14805", 2024-09-18T08:13:52.646 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "172.21.15.7:0/640320436", 2024-09-18T08:13:52.646 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 39808 2024-09-18T08:13:52.647 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:13:52.647 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-09-18T08:13:52.647 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:13:52.647 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-09-18T08:13:52.647 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:24.067964+0000", 2024-09-18T08:13:52.647 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-18T08:13:52.647 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:13:52.647 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:13:52.647 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-09-18T08:13:52.647 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:24.067964+0000", 2024-09-18T08:13:52.647 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-18T08:13:52.647 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:13:52.647 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:13:52.649 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-09-18T08:13:52.649 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:24.067965+0000", 2024-09-18T08:13:52.649 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.4750000000000001e-06 2024-09-18T08:13:52.649 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:13:52.649 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:13:52.649 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-09-18T08:13:52.649 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:24.067992+0000", 2024-09-18T08:13:52.649 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 2.6760000000000001e-05 2024-09-18T08:13:52.650 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:13:52.650 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:13:52.650 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-09-18T08:13:52.650 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:24.068004+0000", 2024-09-18T08:13:52.650 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.2456000000000001e-05 2024-09-18T08:13:52.650 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:13:52.650 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:13:52.650 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-09-18T08:13:52.650 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:24.068012+0000", 2024-09-18T08:13:52.650 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 8.0779999999999996e-06 2024-09-18T08:13:52.650 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:13:52.650 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:13:52.651 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-18T08:13:52.651 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:24.068072+0000", 2024-09-18T08:13:52.651 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 5.9299000000000003e-05 2024-09-18T08:13:52.651 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:13:52.651 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:13:52.651 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-09-18T08:13:52.651 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:24.068164+0000", 2024-09-18T08:13:52.651 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 9.2053999999999997e-05 2024-09-18T08:13:52.651 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:13:52.651 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:13:52.651 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "waiting for subops from 0,1", 2024-09-18T08:13:52.651 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:24.096945+0000", 2024-09-18T08:13:52.651 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.028781128 2024-09-18T08:13:52.652 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:13:52.652 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:13:52.652 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "op_commit", 2024-09-18T08:13:52.652 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:24.099254+0000", 2024-09-18T08:13:52.652 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.002309218 2024-09-18T08:13:52.652 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:13:52.652 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:13:52.652 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:13:52.652 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:24.100291+0000", 2024-09-18T08:13:52.652 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0010365280000000001 2024-09-18T08:13:52.652 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:13:52.652 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:13:52.652 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:13:52.652 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:24.102125+0000", 2024-09-18T08:13:52.653 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0018346599999999999 2024-09-18T08:13:52.653 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:13:52.653 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:13:52.653 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "commit_sent", 2024-09-18T08:13:52.653 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:24.102184+0000", 2024-09-18T08:13:52.653 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 5.8965000000000002e-05 2024-09-18T08:13:52.653 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:13:52.653 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:13:52.653 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-09-18T08:13:52.653 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:24.102222+0000", 2024-09-18T08:13:52.653 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 3.756e-05 2024-09-18T08:13:52.653 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-18T08:13:52.653 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-09-18T08:13:52.654 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-18T08:13:52.654 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:13:52.654 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:13:52.654 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_repop_reply(client.14805.0:39807 7.d e95/83 ondisk, result = 0)", 2024-09-18T08:13:52.654 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-09-18T08:04:24.069321+0000", 2024-09-18T08:13:52.654 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 568.54361021099999, 2024-09-18T08:13:52.654 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.029872724999999999, 2024-09-18T08:13:52.654 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-09-18T08:13:52.654 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-09-18T08:13:52.654 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "started", 2024-09-18T08:13:52.654 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-09-18T08:13:52.654 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:13:52.654 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-09-18T08:13:52.654 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:24.069321+0000", 2024-09-18T08:13:52.655 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-18T08:13:52.655 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:13:52.655 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:13:52.655 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-09-18T08:13:52.655 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:24.069321+0000", 2024-09-18T08:13:52.655 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-18T08:13:52.655 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:13:52.655 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:13:52.655 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-09-18T08:13:52.655 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:24.069322+0000", 2024-09-18T08:13:52.655 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 8.3799999999999996e-07 2024-09-18T08:13:52.656 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:13:52.656 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:13:52.656 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-09-18T08:13:52.656 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:24.069327+0000", 2024-09-18T08:13:52.656 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4.8679999999999998e-06 2024-09-18T08:13:52.656 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:13:52.656 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:13:52.656 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-09-18T08:13:52.656 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:24.098180+0000", 2024-09-18T08:13:52.656 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.028853407000000001 2024-09-18T08:13:52.656 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:13:52.656 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:13:52.656 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-09-18T08:13:52.656 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:24.098189+0000", 2024-09-18T08:13:52.657 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 8.8850000000000005e-06 2024-09-18T08:13:52.657 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:13:52.657 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:13:52.657 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-18T08:13:52.657 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:24.099029+0000", 2024-09-18T08:13:52.657 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00084036500000000004 2024-09-18T08:13:52.657 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:13:52.657 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:13:52.657 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-09-18T08:13:52.657 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:24.099037+0000", 2024-09-18T08:13:52.657 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 8.0150000000000003e-06 2024-09-18T08:13:52.657 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:13:52.657 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:13:52.658 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-09-18T08:13:52.658 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:24.099194+0000", 2024-09-18T08:13:52.658 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00015634700000000001 2024-09-18T08:13:52.658 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-18T08:13:52.658 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-09-18T08:13:52.658 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-18T08:13:52.658 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:13:52.658 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:13:52.658 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.14805.0:39817 7.c 7:3826d3a8:::benchmark_data_smithi007_73155_object39816:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e96)", 2024-09-18T08:13:52.658 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-09-18T08:04:24.077849+0000", 2024-09-18T08:13:52.658 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 568.53508226300005, 2024-09-18T08:13:52.658 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.028025705000000001, 2024-09-18T08:13:52.658 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-09-18T08:13:52.658 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-09-18T08:13:52.659 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T08:13:52.659 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2024-09-18T08:13:52.659 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.14805", 2024-09-18T08:13:52.659 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "172.21.15.7:0/640320436", 2024-09-18T08:13:52.659 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 39817 2024-09-18T08:13:52.659 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:13:52.659 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-09-18T08:13:52.659 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:13:52.659 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-09-18T08:13:52.659 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:24.077849+0000", 2024-09-18T08:13:52.659 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-18T08:13:52.659 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:13:52.659 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:13:52.659 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-09-18T08:13:52.660 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:24.077849+0000", 2024-09-18T08:13:52.660 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-18T08:13:52.660 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:13:52.660 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:13:52.660 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-09-18T08:13:52.660 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:24.077851+0000", 2024-09-18T08:13:52.660 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 2.322e-06 2024-09-18T08:13:52.660 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:13:52.660 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:13:52.660 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-09-18T08:13:52.660 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:24.077879+0000", 2024-09-18T08:13:52.660 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 2.7596000000000001e-05 2024-09-18T08:13:52.660 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:13:52.660 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:13:52.661 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-09-18T08:13:52.661 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:24.097805+0000", 2024-09-18T08:13:52.661 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.019925552999999999 2024-09-18T08:13:52.661 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:13:52.661 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:13:52.661 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-09-18T08:13:52.661 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:24.097820+0000", 2024-09-18T08:13:52.661 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.5705999999999998e-05 2024-09-18T08:13:52.661 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:13:52.661 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:13:52.661 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-18T08:13:52.661 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:24.097984+0000", 2024-09-18T08:13:52.661 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00016333499999999999 2024-09-18T08:13:52.662 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:13:52.662 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:13:52.662 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "op must wait for map", 2024-09-18T08:13:52.662 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:24.097990+0000", 2024-09-18T08:13:52.662 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 6.81e-06 2024-09-18T08:13:52.662 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:13:52.662 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:13:52.662 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-18T08:13:52.662 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:24.102003+0000", 2024-09-18T08:13:52.662 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0040126160000000001 2024-09-18T08:13:52.662 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:13:52.662 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:13:52.662 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-09-18T08:13:52.662 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:24.102155+0000", 2024-09-18T08:13:52.663 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00015247500000000001 2024-09-18T08:13:52.663 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:13:52.663 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:13:52.663 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "waiting for subops from 3,7", 2024-09-18T08:13:52.663 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:24.102305+0000", 2024-09-18T08:13:52.663 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000149872 2024-09-18T08:13:52.663 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:13:52.663 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:13:52.663 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "op_commit", 2024-09-18T08:13:52.663 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:24.105200+0000", 2024-09-18T08:13:52.663 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0028950730000000002 2024-09-18T08:13:52.663 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:13:52.663 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:13:52.663 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:13:52.664 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:24.105248+0000", 2024-09-18T08:13:52.664 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4.7787999999999997e-05 2024-09-18T08:13:52.664 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:13:52.664 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:13:52.664 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:13:52.664 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:24.105772+0000", 2024-09-18T08:13:52.664 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00052425699999999998 2024-09-18T08:13:52.664 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:13:52.664 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:13:52.664 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "commit_sent", 2024-09-18T08:13:52.664 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:24.105826+0000", 2024-09-18T08:13:52.664 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 5.393e-05 2024-09-18T08:13:52.664 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:13:52.664 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:13:52.665 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-09-18T08:13:52.665 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:24.105875+0000", 2024-09-18T08:13:52.665 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4.8371999999999997e-05 2024-09-18T08:13:52.665 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-18T08:13:52.665 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-09-18T08:13:52.665 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-18T08:13:52.665 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:13:52.665 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:13:52.665 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.14805.0:43731 7.d 7:b3fc8faf:::benchmark_data_smithi007_73155_object43730:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e97)", 2024-09-18T08:13:52.665 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-09-18T08:04:26.432079+0000", 2024-09-18T08:13:52.665 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 566.18085198899996, 2024-09-18T08:13:52.665 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.027497293999999999, 2024-09-18T08:13:52.665 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-09-18T08:13:52.665 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-09-18T08:13:52.666 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T08:13:52.666 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2024-09-18T08:13:52.666 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.14805", 2024-09-18T08:13:52.666 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "172.21.15.7:0/640320436", 2024-09-18T08:13:52.666 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 43731 2024-09-18T08:13:52.666 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:13:52.666 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-09-18T08:13:52.666 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:13:52.666 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-09-18T08:13:52.666 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:26.432079+0000", 2024-09-18T08:13:52.666 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-18T08:13:52.666 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:13:52.666 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:13:52.667 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-09-18T08:13:52.667 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:26.432079+0000", 2024-09-18T08:13:52.667 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-18T08:13:52.667 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:13:52.667 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:13:52.667 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-09-18T08:13:52.667 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:26.432080+0000", 2024-09-18T08:13:52.667 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.128e-06 2024-09-18T08:13:52.667 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:13:52.667 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:13:52.667 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-09-18T08:13:52.667 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:26.432153+0000", 2024-09-18T08:13:52.667 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 7.2521999999999994e-05 2024-09-18T08:13:52.667 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:13:52.668 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:13:52.668 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-09-18T08:13:52.668 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:26.432163+0000", 2024-09-18T08:13:52.668 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.0115999999999999e-05 2024-09-18T08:13:52.668 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:13:52.668 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:13:52.668 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-09-18T08:13:52.668 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:26.432170+0000", 2024-09-18T08:13:52.668 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 7.0729999999999999e-06 2024-09-18T08:13:52.668 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:13:52.668 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:13:52.668 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-18T08:13:52.668 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:26.432225+0000", 2024-09-18T08:13:52.668 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 5.4463999999999999e-05 2024-09-18T08:13:52.669 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:13:52.669 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:13:52.669 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-09-18T08:13:52.669 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:26.432342+0000", 2024-09-18T08:13:52.669 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000117118 2024-09-18T08:13:52.669 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:13:52.669 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:13:52.669 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "waiting for subops from 0,1", 2024-09-18T08:13:52.669 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:26.432467+0000", 2024-09-18T08:13:52.669 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00012505700000000001 2024-09-18T08:13:52.669 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:13:52.669 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:13:52.669 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "op_commit", 2024-09-18T08:13:52.670 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:26.433881+0000", 2024-09-18T08:13:52.670 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.001413812 2024-09-18T08:13:52.670 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:13:52.670 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:13:52.670 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:13:52.670 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:26.434589+0000", 2024-09-18T08:13:52.670 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00070883600000000001 2024-09-18T08:13:52.670 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:13:52.670 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:13:52.670 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:13:52.670 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:26.459469+0000", 2024-09-18T08:13:52.670 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.024879951000000001 2024-09-18T08:13:52.670 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:13:52.670 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:13:52.671 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "commit_sent", 2024-09-18T08:13:52.671 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:26.459527+0000", 2024-09-18T08:13:52.671 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 5.7126999999999997e-05 2024-09-18T08:13:52.671 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:13:52.671 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:13:52.671 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-09-18T08:13:52.671 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:26.459577+0000", 2024-09-18T08:13:52.671 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 5.0090000000000003e-05 2024-09-18T08:13:52.671 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-18T08:13:52.671 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-09-18T08:13:52.671 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-18T08:13:52.671 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:13:52.671 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:13:52.672 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.14805.0:63861 7.d 7:bf91474b:::benchmark_data_smithi007_73155_object63860:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e100)", 2024-09-18T08:13:52.672 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-09-18T08:04:41.125989+0000", 2024-09-18T08:13:52.672 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 551.48694269700002, 2024-09-18T08:13:52.672 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.031566575999999999, 2024-09-18T08:13:52.672 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-09-18T08:13:52.672 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-09-18T08:13:52.672 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T08:13:52.672 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2024-09-18T08:13:52.672 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.14805", 2024-09-18T08:13:52.672 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "172.21.15.7:0/640320436", 2024-09-18T08:13:52.672 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 63861 2024-09-18T08:13:52.672 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:13:52.672 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-09-18T08:13:52.672 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:13:52.673 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-09-18T08:13:52.673 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:41.125989+0000", 2024-09-18T08:13:52.673 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-18T08:13:52.673 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:13:52.673 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:13:52.673 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-09-18T08:13:52.673 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:41.125989+0000", 2024-09-18T08:13:52.673 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-18T08:13:52.673 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:13:52.673 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:13:52.673 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-09-18T08:13:52.673 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:41.125990+0000", 2024-09-18T08:13:52.673 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.6109999999999999e-06 2024-09-18T08:13:52.673 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:13:52.674 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:13:52.674 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-09-18T08:13:52.674 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:41.126017+0000", 2024-09-18T08:13:52.674 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 2.6594999999999999e-05 2024-09-18T08:13:52.674 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:13:52.674 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:13:52.674 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-09-18T08:13:52.674 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:41.126035+0000", 2024-09-18T08:13:52.674 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.7842999999999998e-05 2024-09-18T08:13:52.674 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:13:52.674 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:13:52.674 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-09-18T08:13:52.674 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:41.126043+0000", 2024-09-18T08:13:52.675 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 8.8489999999999995e-06 2024-09-18T08:13:52.675 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:13:52.675 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:13:52.675 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-18T08:13:52.675 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:41.126104+0000", 2024-09-18T08:13:52.675 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 6.0448999999999997e-05 2024-09-18T08:13:52.675 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:13:52.675 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:13:52.675 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-09-18T08:13:52.675 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:41.126225+0000", 2024-09-18T08:13:52.675 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000121101 2024-09-18T08:13:52.675 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:13:52.676 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:13:52.676 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "waiting for subops from 0,1", 2024-09-18T08:13:52.676 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:41.126366+0000", 2024-09-18T08:13:52.676 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00014059899999999999 2024-09-18T08:13:52.676 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:13:52.676 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:13:52.676 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "op_commit", 2024-09-18T08:13:52.676 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:41.129750+0000", 2024-09-18T08:13:52.676 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.003383863 2024-09-18T08:13:52.676 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:13:52.676 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:13:52.676 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:13:52.676 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:41.140266+0000", 2024-09-18T08:13:52.676 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.010516266999999999 2024-09-18T08:13:52.677 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:13:52.677 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:13:52.677 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:13:52.677 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:41.157441+0000", 2024-09-18T08:13:52.677 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.017174891000000001 2024-09-18T08:13:52.677 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:13:52.677 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:13:52.677 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "commit_sent", 2024-09-18T08:13:52.677 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:41.157509+0000", 2024-09-18T08:13:52.677 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 6.8627999999999994e-05 2024-09-18T08:13:52.677 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:13:52.677 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:13:52.678 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-09-18T08:13:52.678 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:41.157555+0000", 2024-09-18T08:13:52.678 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4.5880000000000001e-05 2024-09-18T08:13:52.678 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-18T08:13:52.678 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-09-18T08:13:52.678 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-18T08:13:52.678 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:13:52.678 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:13:52.678 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.14805.0:63862 7.d 7:b1f318bc:::benchmark_data_smithi007_73155_object63861:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e100)", 2024-09-18T08:13:52.678 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-09-18T08:04:41.126934+0000", 2024-09-18T08:13:52.678 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 551.48599755999999, 2024-09-18T08:13:52.678 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.030729600999999999, 2024-09-18T08:13:52.678 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-09-18T08:13:52.678 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-09-18T08:13:52.679 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T08:13:52.679 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2024-09-18T08:13:52.679 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.14805", 2024-09-18T08:13:52.679 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "172.21.15.7:0/640320436", 2024-09-18T08:13:52.679 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 63862 2024-09-18T08:13:52.679 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:13:52.679 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-09-18T08:13:52.679 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:13:52.679 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-09-18T08:13:52.679 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:41.126934+0000", 2024-09-18T08:13:52.679 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-18T08:13:52.679 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:13:52.679 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:13:52.680 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-09-18T08:13:52.680 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:41.126934+0000", 2024-09-18T08:13:52.680 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-18T08:13:52.680 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:13:52.680 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:13:52.680 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-09-18T08:13:52.680 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:41.126935+0000", 2024-09-18T08:13:52.680 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 7.9599999999999998e-07 2024-09-18T08:13:52.680 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:13:52.680 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:13:52.680 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-09-18T08:13:52.680 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:41.126989+0000", 2024-09-18T08:13:52.680 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 5.4783e-05 2024-09-18T08:13:52.680 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:13:52.681 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:13:52.681 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-09-18T08:13:52.681 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:41.126999+0000", 2024-09-18T08:13:52.681 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 9.8889999999999997e-06 2024-09-18T08:13:52.681 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:13:52.681 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:13:52.681 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-09-18T08:13:52.681 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:41.127007+0000", 2024-09-18T08:13:52.681 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 7.4399999999999999e-06 2024-09-18T08:13:52.681 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:13:52.681 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:13:52.681 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-18T08:13:52.681 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:41.128171+0000", 2024-09-18T08:13:52.681 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0011642600000000001 2024-09-18T08:13:52.681 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:13:52.682 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:13:52.682 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-09-18T08:13:52.682 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:41.128287+0000", 2024-09-18T08:13:52.682 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000116508 2024-09-18T08:13:52.682 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:13:52.682 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:13:52.682 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "waiting for subops from 0,1", 2024-09-18T08:13:52.682 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:41.128375+0000", 2024-09-18T08:13:52.682 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 8.7300999999999996e-05 2024-09-18T08:13:52.682 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:13:52.682 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:13:52.682 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "op_commit", 2024-09-18T08:13:52.682 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:41.130919+0000", 2024-09-18T08:13:52.683 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0025447450000000002 2024-09-18T08:13:52.683 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:13:52.683 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:13:52.683 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:13:52.683 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:41.140334+0000", 2024-09-18T08:13:52.683 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0094146549999999992 2024-09-18T08:13:52.683 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:13:52.683 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:13:52.683 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:13:52.683 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:41.157602+0000", 2024-09-18T08:13:52.683 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.017267966999999999 2024-09-18T08:13:52.683 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:13:52.683 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:13:52.684 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "commit_sent", 2024-09-18T08:13:52.684 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:41.157636+0000", 2024-09-18T08:13:52.684 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 3.4214000000000001e-05 2024-09-18T08:13:52.684 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:13:52.684 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:13:52.684 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-09-18T08:13:52.684 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:41.157663+0000", 2024-09-18T08:13:52.684 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 2.7042999999999998e-05 2024-09-18T08:13:52.684 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-18T08:13:52.684 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-09-18T08:13:52.684 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-18T08:13:52.684 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:13:52.684 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:13:52.685 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.14805.0:63864 7.d 7:bb7b1761:::benchmark_data_smithi007_73155_object63863:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e100)", 2024-09-18T08:13:52.685 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-09-18T08:04:41.127581+0000", 2024-09-18T08:13:52.685 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 551.48535009700004, 2024-09-18T08:13:52.685 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.030231638000000002, 2024-09-18T08:13:52.685 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-09-18T08:13:52.685 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-09-18T08:13:52.685 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T08:13:52.685 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2024-09-18T08:13:52.685 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.14805", 2024-09-18T08:13:52.685 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "172.21.15.7:0/640320436", 2024-09-18T08:13:52.685 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 63864 2024-09-18T08:13:52.685 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:13:52.685 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-09-18T08:13:52.686 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:13:52.686 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-09-18T08:13:52.686 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:41.127581+0000", 2024-09-18T08:13:52.686 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-18T08:13:52.686 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:13:52.686 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:13:52.686 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-09-18T08:13:52.686 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:41.127581+0000", 2024-09-18T08:13:52.686 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-18T08:13:52.686 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:13:52.686 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:13:52.686 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-09-18T08:13:52.686 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:41.127582+0000", 2024-09-18T08:13:52.686 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 7.8999999999999995e-07 2024-09-18T08:13:52.687 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:13:52.687 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:13:52.687 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-09-18T08:13:52.687 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:41.127648+0000", 2024-09-18T08:13:52.687 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 6.5618999999999996e-05 2024-09-18T08:13:52.687 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:13:52.687 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:13:52.687 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-09-18T08:13:52.687 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:41.127658+0000", 2024-09-18T08:13:52.687 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.0192e-05 2024-09-18T08:13:52.687 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:13:52.687 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:13:52.687 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-09-18T08:13:52.687 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:41.127663+0000", 2024-09-18T08:13:52.688 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 5.2800000000000003e-06 2024-09-18T08:13:52.688 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:13:52.688 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:13:52.688 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-18T08:13:52.688 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:41.129783+0000", 2024-09-18T08:13:52.688 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0021199740000000002 2024-09-18T08:13:52.688 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:13:52.688 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:13:52.688 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-09-18T08:13:52.688 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:41.129885+0000", 2024-09-18T08:13:52.688 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000101987 2024-09-18T08:13:52.688 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:13:52.688 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:13:52.689 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "waiting for subops from 0,1", 2024-09-18T08:13:52.689 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:41.129980+0000", 2024-09-18T08:13:52.689 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 9.4480000000000003e-05 2024-09-18T08:13:52.689 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:13:52.689 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:13:52.689 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "op_commit", 2024-09-18T08:13:52.689 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:41.131035+0000", 2024-09-18T08:13:52.689 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.001055884 2024-09-18T08:13:52.689 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:13:52.690 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:13:52.690 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:13:52.690 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:41.140376+0000", 2024-09-18T08:13:52.690 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0093401439999999999 2024-09-18T08:13:52.690 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:13:52.690 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:13:52.691 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:13:52.691 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:41.157713+0000", 2024-09-18T08:13:52.691 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.017337676999999999 2024-09-18T08:13:52.691 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:13:52.691 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:13:52.691 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "commit_sent", 2024-09-18T08:13:52.691 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:41.157771+0000", 2024-09-18T08:13:52.691 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 5.7674999999999999e-05 2024-09-18T08:13:52.691 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:13:52.691 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:13:52.691 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-09-18T08:13:52.691 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:41.157813+0000", 2024-09-18T08:13:52.692 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4.1936e-05 2024-09-18T08:13:52.692 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-18T08:13:52.692 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-09-18T08:13:52.692 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-18T08:13:52.692 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:13:52.692 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:13:52.693 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.14805.0:63883 7.7 7:ebafa8b8:::benchmark_data_smithi007_73155_object63882:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e100)", 2024-09-18T08:13:52.693 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-09-18T08:04:41.154209+0000", 2024-09-18T08:13:52.693 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 551.45872190499995, 2024-09-18T08:13:52.693 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.042338318999999999, 2024-09-18T08:13:52.693 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-09-18T08:13:52.694 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-09-18T08:13:52.694 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T08:13:52.694 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2024-09-18T08:13:52.694 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.14805", 2024-09-18T08:13:52.694 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "172.21.15.7:0/640320436", 2024-09-18T08:13:52.694 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 63883 2024-09-18T08:13:52.694 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:13:52.694 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-09-18T08:13:52.694 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:13:52.694 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-09-18T08:13:52.694 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:41.154209+0000", 2024-09-18T08:13:52.694 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-18T08:13:52.694 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:13:52.695 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:13:52.695 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-09-18T08:13:52.695 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:41.154209+0000", 2024-09-18T08:13:52.695 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-18T08:13:52.695 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:13:52.695 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:13:52.695 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-09-18T08:13:52.695 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:41.154211+0000", 2024-09-18T08:13:52.695 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.2109999999999999e-06 2024-09-18T08:13:52.696 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:13:52.696 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:13:52.696 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-09-18T08:13:52.696 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:41.154259+0000", 2024-09-18T08:13:52.696 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4.8412000000000003e-05 2024-09-18T08:13:52.696 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:13:52.696 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:13:52.696 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-09-18T08:13:52.696 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:41.154270+0000", 2024-09-18T08:13:52.696 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.0515e-05 2024-09-18T08:13:52.696 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:13:52.696 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:13:52.696 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-09-18T08:13:52.697 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:41.154275+0000", 2024-09-18T08:13:52.697 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 5.4469999999999997e-06 2024-09-18T08:13:52.697 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:13:52.697 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:13:52.697 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-18T08:13:52.697 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:41.154323+0000", 2024-09-18T08:13:52.697 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4.8297999999999997e-05 2024-09-18T08:13:52.697 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:13:52.697 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:13:52.698 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-09-18T08:13:52.698 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:41.154456+0000", 2024-09-18T08:13:52.698 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000132593 2024-09-18T08:13:52.698 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:13:52.698 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:13:52.698 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "waiting for subops from 0,3", 2024-09-18T08:13:52.698 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:41.154615+0000", 2024-09-18T08:13:52.698 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000159049 2024-09-18T08:13:52.698 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:13:52.698 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:13:52.698 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "op_commit", 2024-09-18T08:13:52.698 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:41.156649+0000", 2024-09-18T08:13:52.698 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0020336709999999999 2024-09-18T08:13:52.699 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:13:52.699 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:13:52.699 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:13:52.699 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:41.165163+0000", 2024-09-18T08:13:52.699 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0085141100000000001 2024-09-18T08:13:52.699 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:13:52.699 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:13:52.699 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:13:52.699 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:41.196483+0000", 2024-09-18T08:13:52.699 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.031320300000000002 2024-09-18T08:13:52.699 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:13:52.699 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:13:52.699 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "commit_sent", 2024-09-18T08:13:52.700 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:41.196519+0000", 2024-09-18T08:13:52.700 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 3.6109999999999998e-05 2024-09-18T08:13:52.700 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:13:52.700 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:13:52.700 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-09-18T08:13:52.700 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:41.196548+0000", 2024-09-18T08:13:52.700 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 2.8603e-05 2024-09-18T08:13:52.700 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-18T08:13:52.700 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-09-18T08:13:52.700 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-18T08:13:52.701 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:13:52.701 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:13:52.701 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.14805.0:63908 7.d 7:bd06bac1:::benchmark_data_smithi007_73155_object63907:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e100)", 2024-09-18T08:13:52.701 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-09-18T08:04:41.183375+0000", 2024-09-18T08:13:52.701 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 551.42955614300001, 2024-09-18T08:13:52.701 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.033452764000000003, 2024-09-18T08:13:52.701 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-09-18T08:13:52.701 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-09-18T08:13:52.701 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T08:13:52.701 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2024-09-18T08:13:52.701 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.14805", 2024-09-18T08:13:52.701 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "172.21.15.7:0/640320436", 2024-09-18T08:13:52.701 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 63908 2024-09-18T08:13:52.701 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:13:52.702 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-09-18T08:13:52.702 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:13:52.702 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-09-18T08:13:52.702 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:41.183375+0000", 2024-09-18T08:13:52.702 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-18T08:13:52.702 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:13:52.702 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:13:52.702 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-09-18T08:13:52.702 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:41.183375+0000", 2024-09-18T08:13:52.702 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-18T08:13:52.702 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:13:52.702 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:13:52.702 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-09-18T08:13:52.702 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:41.183376+0000", 2024-09-18T08:13:52.703 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.226e-06 2024-09-18T08:13:52.703 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:13:52.703 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:13:52.703 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-09-18T08:13:52.703 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:41.183403+0000", 2024-09-18T08:13:52.703 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 2.6404e-05 2024-09-18T08:13:52.703 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:13:52.703 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:13:52.703 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-09-18T08:13:52.703 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:41.183417+0000", 2024-09-18T08:13:52.703 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.4399000000000001e-05 2024-09-18T08:13:52.703 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:13:52.703 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:13:52.704 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-09-18T08:13:52.704 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:41.183425+0000", 2024-09-18T08:13:52.704 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 8.1459999999999996e-06 2024-09-18T08:13:52.704 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:13:52.704 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:13:52.704 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-18T08:13:52.704 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:41.183485+0000", 2024-09-18T08:13:52.704 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 5.9673999999999998e-05 2024-09-18T08:13:52.704 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:13:52.704 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:13:52.704 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-09-18T08:13:52.704 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:41.183587+0000", 2024-09-18T08:13:52.705 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000101611 2024-09-18T08:13:52.705 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:13:52.705 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:13:52.705 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "waiting for subops from 0,1", 2024-09-18T08:13:52.705 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:41.183686+0000", 2024-09-18T08:13:52.705 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 9.9823999999999998e-05 2024-09-18T08:13:52.705 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:13:52.705 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:13:52.705 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "op_commit", 2024-09-18T08:13:52.705 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:41.185240+0000", 2024-09-18T08:13:52.705 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0015539340000000001 2024-09-18T08:13:52.706 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:13:52.706 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:13:52.706 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:13:52.706 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:41.191524+0000", 2024-09-18T08:13:52.706 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0062835690000000001 2024-09-18T08:13:52.706 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:13:52.706 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:13:52.706 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:13:52.706 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:41.216746+0000", 2024-09-18T08:13:52.706 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.025222364000000001 2024-09-18T08:13:52.706 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:13:52.706 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:13:52.706 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "commit_sent", 2024-09-18T08:13:52.707 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:41.216782+0000", 2024-09-18T08:13:52.707 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 3.6069999999999999e-05 2024-09-18T08:13:52.707 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:13:52.707 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:13:52.707 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-09-18T08:13:52.707 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:41.216828+0000", 2024-09-18T08:13:52.707 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4.5543000000000001e-05 2024-09-18T08:13:52.707 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-18T08:13:52.707 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-09-18T08:13:52.707 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-18T08:13:52.707 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:13:52.708 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:13:52.708 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.14805.0:63960 7.c 7:30cc2060:::benchmark_data_smithi007_73155_object63959:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e100)", 2024-09-18T08:13:52.708 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-09-18T08:04:41.252452+0000", 2024-09-18T08:13:52.708 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 551.36047941699996, 2024-09-18T08:13:52.708 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.027634619999999999, 2024-09-18T08:13:52.708 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-09-18T08:13:52.708 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-09-18T08:13:52.708 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T08:13:52.708 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2024-09-18T08:13:52.708 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.14805", 2024-09-18T08:13:52.708 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "172.21.15.7:0/640320436", 2024-09-18T08:13:52.709 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 63960 2024-09-18T08:13:52.709 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:13:52.709 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-09-18T08:13:52.709 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:13:52.709 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-09-18T08:13:52.709 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:41.252452+0000", 2024-09-18T08:13:52.709 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-18T08:13:52.709 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:13:52.710 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:13:52.710 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-09-18T08:13:52.710 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:41.252452+0000", 2024-09-18T08:13:52.710 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-18T08:13:52.710 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:13:52.710 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:13:52.710 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-09-18T08:13:52.710 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:41.252453+0000", 2024-09-18T08:13:52.710 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.063e-06 2024-09-18T08:13:52.710 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:13:52.710 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:13:52.711 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-09-18T08:13:52.711 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:41.252503+0000", 2024-09-18T08:13:52.711 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 5.024e-05 2024-09-18T08:13:52.711 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:13:52.711 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:13:52.711 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-09-18T08:13:52.711 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:41.252513+0000", 2024-09-18T08:13:52.711 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.0137e-05 2024-09-18T08:13:52.711 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:13:52.711 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:13:52.711 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-09-18T08:13:52.711 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:41.252519+0000", 2024-09-18T08:13:52.711 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 5.2630000000000003e-06 2024-09-18T08:13:52.712 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:13:52.712 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:13:52.712 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-18T08:13:52.712 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:41.252564+0000", 2024-09-18T08:13:52.712 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4.5528999999999997e-05 2024-09-18T08:13:52.712 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:13:52.712 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:13:52.712 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-09-18T08:13:52.712 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:41.252645+0000", 2024-09-18T08:13:52.712 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 8.1118000000000005e-05 2024-09-18T08:13:52.712 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:13:52.712 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:13:52.712 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "waiting for subops from 3,7", 2024-09-18T08:13:52.713 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:41.252726+0000", 2024-09-18T08:13:52.713 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 8.0912000000000001e-05 2024-09-18T08:13:52.713 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:13:52.713 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:13:52.713 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "op_commit", 2024-09-18T08:13:52.713 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:41.254200+0000", 2024-09-18T08:13:52.713 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0014738010000000001 2024-09-18T08:13:52.713 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:13:52.713 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:13:52.713 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:13:52.713 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:41.254525+0000", 2024-09-18T08:13:52.713 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00032529599999999998 2024-09-18T08:13:52.713 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:13:52.714 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:13:52.714 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:13:52.714 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:41.280019+0000", 2024-09-18T08:13:52.714 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.025493512999999999 2024-09-18T08:13:52.714 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:13:52.714 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:13:52.714 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "commit_sent", 2024-09-18T08:13:52.714 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:41.280057+0000", 2024-09-18T08:13:52.714 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 3.8287000000000001e-05 2024-09-18T08:13:52.714 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:13:52.714 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:13:52.714 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-09-18T08:13:52.714 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:41.280086+0000", 2024-09-18T08:13:52.714 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 2.9461000000000001e-05 2024-09-18T08:13:52.715 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-18T08:13:52.715 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-09-18T08:13:52.715 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-18T08:13:52.715 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:13:52.715 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:13:52.715 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.14805.0:65051 7.d 7:bcc5331c:::benchmark_data_smithi007_73155_object65050:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e100)", 2024-09-18T08:13:52.715 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-09-18T08:04:42.227551+0000", 2024-09-18T08:13:52.715 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 550.38538065600005, 2024-09-18T08:13:52.715 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.057084449000000002, 2024-09-18T08:13:52.715 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-09-18T08:13:52.715 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-09-18T08:13:52.715 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "waiting for sub ops", 2024-09-18T08:13:52.715 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2024-09-18T08:13:52.716 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.14805", 2024-09-18T08:13:52.716 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "172.21.15.7:0/640320436", 2024-09-18T08:13:52.716 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 65051 2024-09-18T08:13:52.716 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:13:52.716 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-09-18T08:13:52.716 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:13:52.716 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-09-18T08:13:52.716 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:42.227551+0000", 2024-09-18T08:13:52.716 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-18T08:13:52.716 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:13:52.716 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:13:52.716 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-09-18T08:13:52.717 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:42.227551+0000", 2024-09-18T08:13:52.717 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-18T08:13:52.717 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:13:52.717 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:13:52.717 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-09-18T08:13:52.717 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:42.227552+0000", 2024-09-18T08:13:52.717 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.5319999999999999e-06 2024-09-18T08:13:52.717 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:13:52.717 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:13:52.717 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-09-18T08:13:52.717 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:42.227606+0000", 2024-09-18T08:13:52.717 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 5.4290999999999999e-05 2024-09-18T08:13:52.717 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:13:52.718 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:13:52.718 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-09-18T08:13:52.718 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:42.227619+0000", 2024-09-18T08:13:52.718 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.2937e-05 2024-09-18T08:13:52.718 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:13:52.718 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:13:52.718 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-09-18T08:13:52.718 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:42.227627+0000", 2024-09-18T08:13:52.718 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 7.9340000000000006e-06 2024-09-18T08:13:52.718 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:13:52.718 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:13:52.718 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-18T08:13:52.719 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:42.227681+0000", 2024-09-18T08:13:52.719 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 5.3771999999999999e-05 2024-09-18T08:13:52.719 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:13:52.719 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:13:52.719 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-09-18T08:13:52.719 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:42.227809+0000", 2024-09-18T08:13:52.719 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00012750399999999999 2024-09-18T08:13:52.719 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:13:52.719 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:13:52.719 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "waiting for subops from 0,1", 2024-09-18T08:13:52.719 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:42.227961+0000", 2024-09-18T08:13:52.719 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00015280599999999999 2024-09-18T08:13:52.719 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:13:52.720 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:13:52.720 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "op_commit", 2024-09-18T08:13:52.720 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:42.229971+0000", 2024-09-18T08:13:52.720 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0020099179999999999 2024-09-18T08:13:52.720 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:13:52.720 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:13:52.720 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:13:52.720 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:42.232646+0000", 2024-09-18T08:13:52.720 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0026748359999999999 2024-09-18T08:13:52.720 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:13:52.720 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:13:52.720 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-09-18T08:13:52.720 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:42.284635+0000", 2024-09-18T08:13:52.720 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.051988919000000001 2024-09-18T08:13:52.721 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-18T08:13:52.721 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-09-18T08:13:52.721 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-18T08:13:52.721 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:13:52.721 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:13:52.721 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.14805.0:65057 7.d 7:bffbe46c:::benchmark_data_smithi007_73155_object65056:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e100)", 2024-09-18T08:13:52.721 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-09-18T08:04:42.229929+0000", 2024-09-18T08:13:52.721 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 550.38300196, 2024-09-18T08:13:52.721 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.054704183000000003, 2024-09-18T08:13:52.721 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-09-18T08:13:52.721 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-09-18T08:13:52.721 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "waiting for sub ops", 2024-09-18T08:13:52.721 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2024-09-18T08:13:52.722 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.14805", 2024-09-18T08:13:52.722 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "172.21.15.7:0/640320436", 2024-09-18T08:13:52.722 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 65057 2024-09-18T08:13:52.722 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:13:52.722 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-09-18T08:13:52.722 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:13:52.722 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-09-18T08:13:52.722 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:42.229929+0000", 2024-09-18T08:13:52.722 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-18T08:13:52.722 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:13:52.722 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:13:52.722 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-09-18T08:13:52.722 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:42.229929+0000", 2024-09-18T08:13:52.722 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-18T08:13:52.723 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:13:52.723 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:13:52.723 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-09-18T08:13:52.723 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:42.229930+0000", 2024-09-18T08:13:52.723 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.0219999999999999e-06 2024-09-18T08:13:52.723 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:13:52.723 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:13:52.723 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-09-18T08:13:52.723 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:42.229973+0000", 2024-09-18T08:13:52.723 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4.2904000000000001e-05 2024-09-18T08:13:52.723 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:13:52.723 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:13:52.723 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-09-18T08:13:52.724 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:42.229987+0000", 2024-09-18T08:13:52.724 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.3405e-05 2024-09-18T08:13:52.724 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:13:52.724 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:13:52.724 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-09-18T08:13:52.724 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:42.229996+0000", 2024-09-18T08:13:52.724 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 9.6299999999999993e-06 2024-09-18T08:13:52.724 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:13:52.724 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:13:52.724 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-18T08:13:52.724 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:42.230048+0000", 2024-09-18T08:13:52.724 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 5.1839000000000003e-05 2024-09-18T08:13:52.724 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:13:52.725 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:13:52.725 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-09-18T08:13:52.725 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:42.230146+0000", 2024-09-18T08:13:52.725 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 9.8101000000000001e-05 2024-09-18T08:13:52.725 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:13:52.725 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:13:52.725 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "waiting for subops from 0,1", 2024-09-18T08:13:52.725 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:42.230237+0000", 2024-09-18T08:13:52.725 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 9.0634e-05 2024-09-18T08:13:52.725 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:13:52.725 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:13:52.725 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "op_commit", 2024-09-18T08:13:52.725 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:42.231752+0000", 2024-09-18T08:13:52.726 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.001515548 2024-09-18T08:13:52.726 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:13:52.726 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:13:52.726 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:13:52.726 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:42.232681+0000", 2024-09-18T08:13:52.726 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00092884 2024-09-18T08:13:52.726 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:13:52.726 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:13:52.726 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-09-18T08:13:52.726 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:42.284634+0000", 2024-09-18T08:13:52.726 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.05195226 2024-09-18T08:13:52.726 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-18T08:13:52.726 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-09-18T08:13:52.727 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-18T08:13:52.727 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:13:52.727 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:13:52.727 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.14805.0:65062 7.d 7:b1e686a2:::benchmark_data_smithi007_73155_object65061:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e100)", 2024-09-18T08:13:52.727 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-09-18T08:04:42.236979+0000", 2024-09-18T08:13:52.727 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 550.37595264499998, 2024-09-18T08:13:52.727 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.047653305999999999, 2024-09-18T08:13:52.727 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-09-18T08:13:52.727 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-09-18T08:13:52.727 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "waiting for sub ops", 2024-09-18T08:13:52.727 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2024-09-18T08:13:52.727 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.14805", 2024-09-18T08:13:52.727 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "172.21.15.7:0/640320436", 2024-09-18T08:13:52.727 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 65062 2024-09-18T08:13:52.728 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:13:52.728 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-09-18T08:13:52.728 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:13:52.728 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-09-18T08:13:52.728 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:42.236979+0000", 2024-09-18T08:13:52.728 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-18T08:13:52.728 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:13:52.728 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:13:52.728 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-09-18T08:13:52.728 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:42.236979+0000", 2024-09-18T08:13:52.728 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-18T08:13:52.728 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:13:52.729 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:13:52.729 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-09-18T08:13:52.729 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:42.236980+0000", 2024-09-18T08:13:52.729 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.124e-06 2024-09-18T08:13:52.729 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:13:52.729 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:13:52.729 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-09-18T08:13:52.729 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:42.237029+0000", 2024-09-18T08:13:52.729 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4.9450999999999998e-05 2024-09-18T08:13:52.729 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:13:52.730 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:13:52.730 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-09-18T08:13:52.730 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:42.237041+0000", 2024-09-18T08:13:52.730 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.1418e-05 2024-09-18T08:13:52.730 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:13:52.730 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:13:52.730 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-09-18T08:13:52.730 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:42.237049+0000", 2024-09-18T08:13:52.730 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 8.0059999999999992e-06 2024-09-18T08:13:52.730 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:13:52.730 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:13:52.730 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-18T08:13:52.730 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:42.237104+0000", 2024-09-18T08:13:52.730 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 5.4870999999999997e-05 2024-09-18T08:13:52.731 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:13:52.731 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:13:52.731 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-09-18T08:13:52.731 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:42.237258+0000", 2024-09-18T08:13:52.731 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000154857 2024-09-18T08:13:52.731 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:13:52.731 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:13:52.731 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "waiting for subops from 0,1", 2024-09-18T08:13:52.731 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:42.237413+0000", 2024-09-18T08:13:52.731 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000154764 2024-09-18T08:13:52.731 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:13:52.731 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:13:52.731 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "op_commit", 2024-09-18T08:13:52.732 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:42.240054+0000", 2024-09-18T08:13:52.732 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0026404549999999999 2024-09-18T08:13:52.732 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:13:52.732 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:13:52.732 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-09-18T08:13:52.732 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:42.284632+0000", 2024-09-18T08:13:52.732 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.044578359999999997 2024-09-18T08:13:52.732 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-18T08:13:52.732 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-09-18T08:13:52.732 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-18T08:13:52.733 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:13:52.733 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:13:52.733 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.14805.0:65063 7.d 7:b037f784:::benchmark_data_smithi007_73155_object65062:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e100)", 2024-09-18T08:13:52.733 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-09-18T08:04:42.237116+0000", 2024-09-18T08:13:52.733 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 550.37581517900003, 2024-09-18T08:13:52.733 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.047512190000000003, 2024-09-18T08:13:52.733 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-09-18T08:13:52.733 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-09-18T08:13:52.733 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "waiting for sub ops", 2024-09-18T08:13:52.733 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2024-09-18T08:13:52.733 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.14805", 2024-09-18T08:13:52.733 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "172.21.15.7:0/640320436", 2024-09-18T08:13:52.734 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 65063 2024-09-18T08:13:52.734 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:13:52.734 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-09-18T08:13:52.734 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:13:52.734 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-09-18T08:13:52.734 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:42.237116+0000", 2024-09-18T08:13:52.734 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-18T08:13:52.734 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:13:52.734 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:13:52.734 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-09-18T08:13:52.734 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:42.237116+0000", 2024-09-18T08:13:52.734 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-18T08:13:52.734 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:13:52.735 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:13:52.735 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-09-18T08:13:52.735 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:42.237117+0000", 2024-09-18T08:13:52.735 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 6.7000000000000004e-07 2024-09-18T08:13:52.735 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:13:52.735 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:13:52.735 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-09-18T08:13:52.735 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:42.237190+0000", 2024-09-18T08:13:52.735 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 7.2972000000000005e-05 2024-09-18T08:13:52.735 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:13:52.735 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:13:52.735 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-09-18T08:13:52.735 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:42.237197+0000", 2024-09-18T08:13:52.736 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 7.2729999999999997e-06 2024-09-18T08:13:52.736 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:13:52.736 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:13:52.736 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-09-18T08:13:52.736 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:42.237203+0000", 2024-09-18T08:13:52.736 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 5.5289999999999999e-06 2024-09-18T08:13:52.736 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:13:52.736 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:13:52.736 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-18T08:13:52.736 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:42.238960+0000", 2024-09-18T08:13:52.736 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.001757759 2024-09-18T08:13:52.736 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:13:52.736 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:13:52.737 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-09-18T08:13:52.737 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:42.239038+0000", 2024-09-18T08:13:52.737 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 7.8054999999999997e-05 2024-09-18T08:13:52.737 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:13:52.737 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:13:52.737 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "waiting for subops from 0,1", 2024-09-18T08:13:52.737 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:42.239122+0000", 2024-09-18T08:13:52.737 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 8.3963999999999998e-05 2024-09-18T08:13:52.737 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:13:52.737 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:13:52.737 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "op_commit", 2024-09-18T08:13:52.737 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:42.240157+0000", 2024-09-18T08:13:52.737 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.001034372 2024-09-18T08:13:52.738 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:13:52.738 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:13:52.738 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-09-18T08:13:52.738 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:04:42.284628+0000", 2024-09-18T08:13:52.738 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.044471596000000002 2024-09-18T08:13:52.738 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-18T08:13:52.738 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-09-18T08:13:52.738 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-18T08:13:52.738 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:13:52.738 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:13:52.739 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_repop(client.15582.0:123972 9.0 e261/244 9:0e6c2bd0:::benchmark_data_smithi007_124019_object123971:head v 261'7682, mlcod=261'7678)", 2024-09-18T08:13:52.739 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-09-18T08:12:31.108179+0000", 2024-09-18T08:13:52.739 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 81.504752667000005, 2024-09-18T08:13:52.739 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.029088182000000001, 2024-09-18T08:13:52.739 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-09-18T08:13:52.739 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-09-18T08:13:52.739 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T08:13:52.739 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-09-18T08:13:52.739 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:13:52.739 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0 2024-09-18T08:13:52.740 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-09-18T08:13:52.740 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:12:31.108179+0000", 2024-09-18T08:13:52.740 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-18T08:13:52.740 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:13:52.740 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:13:52.741 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-09-18T08:13:52.741 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:12:31.108179+0000", 2024-09-18T08:13:52.741 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-18T08:13:52.741 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:13:52.741 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:13:52.741 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-09-18T08:13:52.741 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:12:31.108183+0000", 2024-09-18T08:13:52.741 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4.0029999999999996e-06 2024-09-18T08:13:52.741 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:13:52.741 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:13:52.741 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-09-18T08:13:52.741 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:12:31.108232+0000", 2024-09-18T08:13:52.741 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4.9462000000000002e-05 2024-09-18T08:13:52.742 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:13:52.742 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:13:52.742 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-09-18T08:13:52.742 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:12:31.108247+0000", 2024-09-18T08:13:52.742 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.5350999999999999e-05 2024-09-18T08:13:52.742 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:13:52.742 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:13:52.742 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-09-18T08:13:52.742 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:12:31.108255+0000", 2024-09-18T08:13:52.742 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 7.9449999999999993e-06 2024-09-18T08:13:52.742 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:13:52.742 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:13:52.742 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-18T08:13:52.742 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:12:31.109933+0000", 2024-09-18T08:13:52.743 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.001677869 2024-09-18T08:13:52.743 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:13:52.743 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:13:52.743 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-09-18T08:13:52.743 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:12:31.109957+0000", 2024-09-18T08:13:52.743 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 2.3374e-05 2024-09-18T08:13:52.743 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:13:52.743 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:13:52.743 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "commit_sent", 2024-09-18T08:13:52.743 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:12:31.137170+0000", 2024-09-18T08:13:52.743 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.027213580000000001 2024-09-18T08:13:52.743 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:13:52.743 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:13:52.743 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-09-18T08:13:52.744 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:12:31.137267+0000", 2024-09-18T08:13:52.744 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 9.6597999999999999e-05 2024-09-18T08:13:52.744 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-18T08:13:52.744 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-09-18T08:13:52.744 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-18T08:13:52.744 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:13:52.744 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:13:52.744 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_repop(client.15582.0:123975 9.0 e261/244 9:030e6d7f:::benchmark_data_smithi007_124019_object123974:head v 261'7683, mlcod=261'7680)", 2024-09-18T08:13:52.744 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-09-18T08:12:31.113187+0000", 2024-09-18T08:13:52.744 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 81.499744010000001, 2024-09-18T08:13:52.744 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.027941135999999998, 2024-09-18T08:13:52.744 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-09-18T08:13:52.744 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-09-18T08:13:52.745 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T08:13:52.745 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-09-18T08:13:52.745 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:13:52.745 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-09-18T08:13:52.745 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:12:31.113187+0000", 2024-09-18T08:13:52.745 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-18T08:13:52.745 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:13:52.745 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:13:52.745 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-09-18T08:13:52.745 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:12:31.113187+0000", 2024-09-18T08:13:52.745 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-18T08:13:52.745 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:13:52.745 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:13:52.746 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-09-18T08:13:52.746 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:12:31.113189+0000", 2024-09-18T08:13:52.746 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.4500000000000001e-06 2024-09-18T08:13:52.746 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:13:52.746 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:13:52.746 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-09-18T08:13:52.746 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:12:31.113223+0000", 2024-09-18T08:13:52.746 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 3.4588000000000002e-05 2024-09-18T08:13:52.746 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:13:52.746 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:13:52.746 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-09-18T08:13:52.746 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:12:31.113240+0000", 2024-09-18T08:13:52.746 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.6580999999999999e-05 2024-09-18T08:13:52.746 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:13:52.747 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:13:52.747 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-09-18T08:13:52.747 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:12:31.113248+0000", 2024-09-18T08:13:52.747 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 8.0560000000000005e-06 2024-09-18T08:13:52.747 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:13:52.747 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:13:52.747 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-18T08:13:52.747 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:12:31.113470+0000", 2024-09-18T08:13:52.747 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00022196100000000001 2024-09-18T08:13:52.747 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:13:52.747 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:13:52.747 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-09-18T08:13:52.747 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:12:31.113490+0000", 2024-09-18T08:13:52.748 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 2.0377000000000002e-05 2024-09-18T08:13:52.748 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:13:52.748 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:13:52.748 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "commit_sent", 2024-09-18T08:13:52.748 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:12:31.140806+0000", 2024-09-18T08:13:52.748 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.027315715000000001 2024-09-18T08:13:52.748 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:13:52.748 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:13:52.748 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-09-18T08:13:52.748 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:12:31.141128+0000", 2024-09-18T08:13:52.748 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00032240799999999999 2024-09-18T08:13:52.748 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-18T08:13:52.749 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-09-18T08:13:52.749 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-18T08:13:52.749 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-18T08:13:52.749 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-09-18T08:13:52.749 INFO:teuthology.orchestra.run.smithi050.stdout:} 2024-09-18T08:13:52.788 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:13:52.788 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:13:52.789 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0' 2024-09-18T08:13:52.889 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0 2024-09-18T08:13:52.920 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:13:52.921 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:13:52.921 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0' 2024-09-18T08:13:52.980 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:13:52 smithi007 ceph-mon[29459]: 1.0 scrub starts 2024-09-18T08:13:52.981 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:13:52 smithi007 ceph-mon[29459]: 1.0 scrub ok 2024-09-18T08:13:52.981 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:13:52 smithi007 ceph-mon[29459]: from='mgr.24470 172.21.15.7:0/2594390261' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"} : dispatch 2024-09-18T08:13:52.981 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:13:52 smithi007 ceph-mon[29459]: from='mgr.24470 172.21.15.7:0/2594390261' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"} : dispatch 2024-09-18T08:13:52.981 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:13:52 smithi007 ceph-mon[29459]: from='mgr.24470 172.21.15.7:0/2594390261' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"} : dispatch 2024-09-18T08:13:52.981 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:13:52 smithi007 ceph-mon[29459]: from='mgr.24470 172.21.15.7:0/2594390261' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"} : dispatch 2024-09-18T08:13:52.981 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:13:52 smithi007 ceph-mon[29459]: from='mgr.24470 172.21.15.7:0/2594390261' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"} : dispatch 2024-09-18T08:13:52.981 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:13:52 smithi007 ceph-mon[29459]: from='mgr.24470 172.21.15.7:0/2594390261' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"} : dispatch 2024-09-18T08:13:52.981 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:13:52 smithi007 ceph-mon[29459]: from='mgr.24470 172.21.15.7:0/2594390261' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"} : dispatch 2024-09-18T08:13:52.981 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:13:52 smithi007 ceph-mon[29459]: from='mgr.24470 172.21.15.7:0/2594390261' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"} : dispatch 2024-09-18T08:13:52.981 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:13:52 smithi007 ceph-mon[29459]: Adjusting osd_memory_target on smithi050 to 3249M 2024-09-18T08:13:52.981 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:13:52 smithi007 ceph-mon[29459]: Adjusting osd_memory_target on smithi007 to 3249M 2024-09-18T08:13:52.981 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:13:52 smithi007 ceph-mon[29459]: from='mgr.24470 172.21.15.7:0/2594390261' entity='mgr.y' 2024-09-18T08:13:52.982 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:13:52 smithi007 ceph-mon[29459]: from='mgr.24470 172.21.15.7:0/2594390261' entity='mgr.y' 2024-09-18T08:13:52.982 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:13:52 smithi007 ceph-mon[29459]: from='mgr.24470 172.21.15.7:0/2594390261' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-18T08:13:52.982 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:13:52 smithi007 ceph-mon[29459]: from='mgr.24470 172.21.15.7:0/2594390261' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-18T08:13:52.982 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:13:52 smithi007 ceph-mon[29459]: from='mgr.24470 172.21.15.7:0/2594390261' entity='mgr.y' 2024-09-18T08:13:52.982 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:13:52 smithi007 ceph-mon[39249]: 1.0 scrub starts 2024-09-18T08:13:52.982 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:13:52 smithi007 ceph-mon[39249]: 1.0 scrub ok 2024-09-18T08:13:52.982 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:13:52 smithi007 ceph-mon[39249]: from='mgr.24470 172.21.15.7:0/2594390261' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"} : dispatch 2024-09-18T08:13:52.982 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:13:52 smithi007 ceph-mon[39249]: from='mgr.24470 172.21.15.7:0/2594390261' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"} : dispatch 2024-09-18T08:13:52.982 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:13:52 smithi007 ceph-mon[39249]: from='mgr.24470 172.21.15.7:0/2594390261' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"} : dispatch 2024-09-18T08:13:52.982 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:13:52 smithi007 ceph-mon[39249]: from='mgr.24470 172.21.15.7:0/2594390261' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"} : dispatch 2024-09-18T08:13:52.983 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:13:52 smithi007 ceph-mon[39249]: from='mgr.24470 172.21.15.7:0/2594390261' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"} : dispatch 2024-09-18T08:13:52.983 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:13:52 smithi007 ceph-mon[39249]: from='mgr.24470 172.21.15.7:0/2594390261' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"} : dispatch 2024-09-18T08:13:52.983 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:13:52 smithi007 ceph-mon[39249]: from='mgr.24470 172.21.15.7:0/2594390261' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"} : dispatch 2024-09-18T08:13:52.983 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:13:52 smithi007 ceph-mon[39249]: from='mgr.24470 172.21.15.7:0/2594390261' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"} : dispatch 2024-09-18T08:13:52.983 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:13:52 smithi007 ceph-mon[39249]: Adjusting osd_memory_target on smithi050 to 3249M 2024-09-18T08:13:52.983 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:13:52 smithi007 ceph-mon[39249]: Adjusting osd_memory_target on smithi007 to 3249M 2024-09-18T08:13:52.983 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:13:52 smithi007 ceph-mon[39249]: from='mgr.24470 172.21.15.7:0/2594390261' entity='mgr.y' 2024-09-18T08:13:52.983 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:13:52 smithi007 ceph-mon[39249]: from='mgr.24470 172.21.15.7:0/2594390261' entity='mgr.y' 2024-09-18T08:13:52.983 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:13:52 smithi007 ceph-mon[39249]: from='mgr.24470 172.21.15.7:0/2594390261' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-18T08:13:52.983 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:13:52 smithi007 ceph-mon[39249]: from='mgr.24470 172.21.15.7:0/2594390261' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-18T08:13:52.983 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:13:52 smithi007 ceph-mon[39249]: from='mgr.24470 172.21.15.7:0/2594390261' entity='mgr.y' 2024-09-18T08:13:53.023 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2 2024-09-18T08:13:53.052 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:13:53.053 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:13:53.053 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2' 2024-09-18T08:13:53.154 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4 2024-09-18T08:13:53.155 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:13:52 smithi050 ceph-mon[32371]: 1.0 scrub starts 2024-09-18T08:13:53.155 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:13:52 smithi050 ceph-mon[32371]: 1.0 scrub ok 2024-09-18T08:13:53.156 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:13:52 smithi050 ceph-mon[32371]: from='mgr.24470 172.21.15.7:0/2594390261' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"} : dispatch 2024-09-18T08:13:53.156 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:13:52 smithi050 ceph-mon[32371]: from='mgr.24470 172.21.15.7:0/2594390261' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"} : dispatch 2024-09-18T08:13:53.156 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:13:52 smithi050 ceph-mon[32371]: from='mgr.24470 172.21.15.7:0/2594390261' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"} : dispatch 2024-09-18T08:13:53.156 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:13:52 smithi050 ceph-mon[32371]: from='mgr.24470 172.21.15.7:0/2594390261' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"} : dispatch 2024-09-18T08:13:53.156 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:13:52 smithi050 ceph-mon[32371]: from='mgr.24470 172.21.15.7:0/2594390261' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"} : dispatch 2024-09-18T08:13:53.156 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:13:52 smithi050 ceph-mon[32371]: from='mgr.24470 172.21.15.7:0/2594390261' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"} : dispatch 2024-09-18T08:13:53.156 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:13:52 smithi050 ceph-mon[32371]: from='mgr.24470 172.21.15.7:0/2594390261' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"} : dispatch 2024-09-18T08:13:53.156 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:13:52 smithi050 ceph-mon[32371]: from='mgr.24470 172.21.15.7:0/2594390261' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"} : dispatch 2024-09-18T08:13:53.156 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:13:52 smithi050 ceph-mon[32371]: Adjusting osd_memory_target on smithi050 to 3249M 2024-09-18T08:13:53.156 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:13:52 smithi050 ceph-mon[32371]: Adjusting osd_memory_target on smithi007 to 3249M 2024-09-18T08:13:53.156 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:13:52 smithi050 ceph-mon[32371]: from='mgr.24470 172.21.15.7:0/2594390261' entity='mgr.y' 2024-09-18T08:13:53.156 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:13:52 smithi050 ceph-mon[32371]: from='mgr.24470 172.21.15.7:0/2594390261' entity='mgr.y' 2024-09-18T08:13:53.157 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:13:52 smithi050 ceph-mon[32371]: from='mgr.24470 172.21.15.7:0/2594390261' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-18T08:13:53.157 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:13:52 smithi050 ceph-mon[32371]: from='mgr.24470 172.21.15.7:0/2594390261' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-18T08:13:53.157 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:13:52 smithi050 ceph-mon[32371]: from='mgr.24470 172.21.15.7:0/2594390261' entity='mgr.y' 2024-09-18T08:13:53.171 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/779349ca-7593-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T08:13:53.204 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:13:53.204 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:13:53.205 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4' 2024-09-18T08:13:53.306 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4 2024-09-18T08:13:53.348 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:13:53.348 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:13:53.349 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4' 2024-09-18T08:13:53.450 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2 2024-09-18T08:13:53.485 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:13:53.485 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:13:53.485 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2' 2024-09-18T08:13:53.586 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5 2024-09-18T08:13:53.622 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:13:53.623 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:13:53.623 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5' 2024-09-18T08:13:53.724 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7 2024-09-18T08:13:53.769 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:13:53.769 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:13:53.770 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7' 2024-09-18T08:13:53.794 INFO:teuthology.orchestra.run.smithi007.stdout:osd.0: {} 2024-09-18T08:13:53.795 INFO:teuthology.orchestra.run.smithi007.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-18T08:13:53.795 INFO:teuthology.orchestra.run.smithi007.stderr:Error ENXIO: problem getting command descriptions from osd.1 2024-09-18T08:13:53.818 INFO:teuthology.orchestra.run.smithi007.stdout:osd.2: {} 2024-09-18T08:13:53.819 INFO:teuthology.orchestra.run.smithi007.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-18T08:13:53.841 INFO:teuthology.orchestra.run.smithi007.stdout:osd.3: {} 2024-09-18T08:13:53.841 INFO:teuthology.orchestra.run.smithi007.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-18T08:13:53.843 DEBUG:teuthology.orchestra.run.smithi050:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid 779349ca-7593-11ef-bceb-c7b262605968 -- ceph daemon osd.7 dump_ops_in_flight 2024-09-18T08:13:53.870 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2 2024-09-18T08:13:53.871 INFO:teuthology.orchestra.run.smithi007.stdout:osd.4: {} 2024-09-18T08:13:53.872 INFO:teuthology.orchestra.run.smithi007.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-18T08:13:53.901 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:13:53.901 INFO:teuthology.orchestra.run.smithi007.stdout:osd.5: {} 2024-09-18T08:13:53.901 INFO:teuthology.orchestra.run.smithi007.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-18T08:13:53.902 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:13:53.902 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2' 2024-09-18T08:13:53.929 INFO:teuthology.orchestra.run.smithi007.stdout:osd.6: {} 2024-09-18T08:13:53.929 INFO:teuthology.orchestra.run.smithi007.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-18T08:13:53.962 INFO:teuthology.orchestra.run.smithi007.stdout:osd.7: {} 2024-09-18T08:13:53.962 INFO:teuthology.orchestra.run.smithi007.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-18T08:13:54.003 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2 2024-09-18T08:13:54.031 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:13:54.032 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:13:54.033 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2' 2024-09-18T08:13:54.133 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7 2024-09-18T08:13:54.156 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:13:53 smithi050 ceph-mon[32371]: 1.9 scrub starts 2024-09-18T08:13:54.156 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:13:53 smithi050 ceph-mon[32371]: 1.9 scrub ok 2024-09-18T08:13:54.156 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:13:53 smithi050 ceph-mon[32371]: 3.3 scrub starts 2024-09-18T08:13:54.156 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:13:53 smithi050 ceph-mon[32371]: 3.3 scrub ok 2024-09-18T08:13:54.157 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:13:53 smithi050 ceph-mon[32371]: pgmap v537: 158 pgs: 30 active+undersized, 34 active+undersized+degraded, 94 active+clean; 3.2 GiB data, 13 GiB used, 703 GiB / 715 GiB avail; 852 B/s rd, 0 B/s wr, 1.15k op/s; 29452/156102 objects degraded (18.867%) 2024-09-18T08:13:54.182 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:13:54.182 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:13:54.183 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7' 2024-09-18T08:13:54.230 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:13:53 smithi007 ceph-mon[29459]: 1.9 scrub starts 2024-09-18T08:13:54.230 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:13:53 smithi007 ceph-mon[29459]: 1.9 scrub ok 2024-09-18T08:13:54.230 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:13:53 smithi007 ceph-mon[29459]: 3.3 scrub starts 2024-09-18T08:13:54.230 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:13:53 smithi007 ceph-mon[29459]: 3.3 scrub ok 2024-09-18T08:13:54.231 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:13:53 smithi007 ceph-mon[29459]: pgmap v537: 158 pgs: 30 active+undersized, 34 active+undersized+degraded, 94 active+clean; 3.2 GiB data, 13 GiB used, 703 GiB / 715 GiB avail; 852 B/s rd, 0 B/s wr, 1.15k op/s; 29452/156102 objects degraded (18.867%) 2024-09-18T08:13:54.231 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:13:53 smithi007 ceph-mon[39249]: 1.9 scrub starts 2024-09-18T08:13:54.231 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:13:53 smithi007 ceph-mon[39249]: 1.9 scrub ok 2024-09-18T08:13:54.231 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:13:53 smithi007 ceph-mon[39249]: 3.3 scrub starts 2024-09-18T08:13:54.231 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:13:53 smithi007 ceph-mon[39249]: 3.3 scrub ok 2024-09-18T08:13:54.231 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:13:53 smithi007 ceph-mon[39249]: pgmap v537: 158 pgs: 30 active+undersized, 34 active+undersized+degraded, 94 active+clean; 3.2 GiB data, 13 GiB used, 703 GiB / 715 GiB avail; 852 B/s rd, 0 B/s wr, 1.15k op/s; 29452/156102 objects degraded (18.867%) 2024-09-18T08:13:54.283 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4 2024-09-18T08:13:54.300 INFO:teuthology.orchestra.run.smithi050.stderr:Inferring config /var/lib/ceph/779349ca-7593-11ef-bceb-c7b262605968/mon.b/config 2024-09-18T08:13:54.342 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:13:54.342 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:13:54.343 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4' 2024-09-18T08:13:54.444 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0 2024-09-18T08:13:54.474 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:13:54.474 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:13:54.475 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0' 2024-09-18T08:13:54.575 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5 2024-09-18T08:13:54.613 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:13:54.614 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:13:54.614 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5' 2024-09-18T08:13:54.675 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-09-18T08:13:54.676 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-09-18T08:13:54.694 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid 779349ca-7593-11ef-bceb-c7b262605968 -- ceph osd unset nodeep-scrub 2024-09-18T08:13:54.715 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6 2024-09-18T08:13:54.746 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:13:54.747 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:13:54.747 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6' 2024-09-18T08:13:54.848 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0 2024-09-18T08:13:54.881 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:13:54.882 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:13:54.882 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0' 2024-09-18T08:13:54.980 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:13:54 smithi007 ceph-mon[29459]: from='client.? 172.21.15.7:0/2119706196' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-18T08:13:54.980 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:13:54 smithi007 ceph-mon[29459]: Health check update: Degraded data redundancy: 29452/156102 objects degraded (18.867%), 34 pgs degraded, 64 pgs undersized (PG_DEGRADED) 2024-09-18T08:13:54.980 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:13:54 smithi007 ceph-mon[39249]: from='client.? 172.21.15.7:0/2119706196' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-18T08:13:54.980 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:13:54 smithi007 ceph-mon[39249]: Health check update: Degraded data redundancy: 29452/156102 objects degraded (18.867%), 34 pgs degraded, 64 pgs undersized (PG_DEGRADED) 2024-09-18T08:13:54.983 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7 2024-09-18T08:13:55.019 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:13:55.019 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:13:55.020 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7' 2024-09-18T08:13:55.039 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/779349ca-7593-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T08:13:55.120 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0 2024-09-18T08:13:55.148 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:13:55.149 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:13:55.149 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0' 2024-09-18T08:13:55.155 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:13:54 smithi050 ceph-mon[32371]: from='client.? 172.21.15.7:0/2119706196' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-18T08:13:55.155 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:13:54 smithi050 ceph-mon[32371]: Health check update: Degraded data redundancy: 29452/156102 objects degraded (18.867%), 34 pgs degraded, 64 pgs undersized (PG_DEGRADED) 2024-09-18T08:13:55.250 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2 2024-09-18T08:13:55.283 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:13:55.283 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:13:55.284 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2' 2024-09-18T08:13:55.385 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2 2024-09-18T08:13:55.420 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:13:55.420 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:13:55.421 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2' 2024-09-18T08:13:55.521 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0 2024-09-18T08:13:55.560 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:13:55.561 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:13:55.562 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0' 2024-09-18T08:13:55.662 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2 2024-09-18T08:13:55.689 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:13:55.690 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:13:55.690 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2' 2024-09-18T08:13:55.791 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0 2024-09-18T08:13:55.823 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:13:55.824 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:13:55.824 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0' 2024-09-18T08:13:55.925 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3 2024-09-18T08:13:55.955 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:13:55.955 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:13:55.956 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3' 2024-09-18T08:13:56.057 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6 2024-09-18T08:13:56.091 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:13:56.092 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:13:56.092 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6' 2024-09-18T08:13:56.155 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:13:55 smithi050 ceph-mon[32371]: pgmap v538: 158 pgs: 30 active+undersized, 34 active+undersized+degraded, 94 active+clean; 2.8 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 852 B/s rd, 0 B/s wr, 957 op/s; 26202/138867 objects degraded (18.868%) 2024-09-18T08:13:56.193 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2 2024-09-18T08:13:56.223 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:13:56.223 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:13:56.224 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2' 2024-09-18T08:13:56.230 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:13:55 smithi007 ceph-mon[29459]: pgmap v538: 158 pgs: 30 active+undersized, 34 active+undersized+degraded, 94 active+clean; 2.8 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 852 B/s rd, 0 B/s wr, 957 op/s; 26202/138867 objects degraded (18.868%) 2024-09-18T08:13:56.230 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:13:55 smithi007 ceph-mon[39249]: pgmap v538: 158 pgs: 30 active+undersized, 34 active+undersized+degraded, 94 active+clean; 2.8 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 852 B/s rd, 0 B/s wr, 957 op/s; 26202/138867 objects degraded (18.868%) 2024-09-18T08:13:56.325 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5 2024-09-18T08:13:56.361 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:13:56.362 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:13:56.362 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5' 2024-09-18T08:13:56.463 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2 2024-09-18T08:13:56.492 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:13:56.493 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:13:56.494 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2' 2024-09-18T08:13:56.594 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6 2024-09-18T08:13:56.616 INFO:teuthology.orchestra.run.smithi050.stdout:ERROR: (22) Invalid argument 2024-09-18T08:13:56.616 INFO:teuthology.orchestra.run.smithi050.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-18T08:13:56.633 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:13:56.634 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:13:56.634 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6' 2024-09-18T08:13:56.677 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid 779349ca-7593-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-18T08:13:56.695 INFO:journalctl@ceph.grafana.a.smithi050.stdout:Sep 18 08:13:56 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-grafana-a[60403]: logger=cleanup t=2024-09-18T08:13:56.520809021Z level=info msg="Completed cleanup jobs" duration=4.589206ms 2024-09-18T08:13:56.695 INFO:journalctl@ceph.grafana.a.smithi050.stdout:Sep 18 08:13:56 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-grafana-a[60403]: logger=sqlstore.transactions t=2024-09-18T08:13:56.52724548Z level=info msg="Database locked, sleeping then retrying" error="database is locked" retry=0 code="database is locked" 2024-09-18T08:13:56.695 INFO:journalctl@ceph.grafana.a.smithi050.stdout:Sep 18 08:13:56 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-grafana-a[60403]: logger=plugins.update.checker t=2024-09-18T08:13:56.668101874Z level=info msg="Update check succeeded" duration=75.174433ms 2024-09-18T08:13:56.735 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5 2024-09-18T08:13:56.771 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:13:56.772 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:13:56.773 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5' 2024-09-18T08:13:56.779 INFO:teuthology.orchestra.run.smithi007.stderr:nodeep-scrub is unset 2024-09-18T08:13:56.873 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5 2024-09-18T08:13:56.910 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:13:56.911 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:13:56.911 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5' 2024-09-18T08:13:56.980 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Sep 18 08:13:56 smithi007 ceph-779349ca-7593-11ef-bceb-c7b262605968-mgr-y[29688]: ::ffff:172.21.15.50 - - [18/Sep/2024:08:13:56] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-09-18T08:13:56.981 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:13:56 smithi007 ceph-mon[29459]: 1.1 scrub starts 2024-09-18T08:13:56.981 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:13:56 smithi007 ceph-mon[29459]: 1.1 scrub ok 2024-09-18T08:13:56.981 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:13:56 smithi007 ceph-mon[29459]: purged_snaps scrub starts 2024-09-18T08:13:56.981 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:13:56 smithi007 ceph-mon[29459]: purged_snaps scrub ok 2024-09-18T08:13:56.981 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:13:56 smithi007 ceph-mon[29459]: osdmap e290: 8 total, 7 up, 8 in 2024-09-18T08:13:56.981 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:13:56 smithi007 ceph-mon[29459]: from='client.? 172.21.15.7:0/3881307331' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-18T08:13:56.981 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:13:56 smithi007 ceph-mon[29459]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-18T08:13:56.981 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:13:56 smithi007 ceph-mon[39249]: 1.1 scrub starts 2024-09-18T08:13:56.981 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:13:56 smithi007 ceph-mon[39249]: 1.1 scrub ok 2024-09-18T08:13:56.982 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:13:56 smithi007 ceph-mon[39249]: purged_snaps scrub starts 2024-09-18T08:13:56.982 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:13:56 smithi007 ceph-mon[39249]: purged_snaps scrub ok 2024-09-18T08:13:56.982 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:13:56 smithi007 ceph-mon[39249]: osdmap e290: 8 total, 7 up, 8 in 2024-09-18T08:13:56.982 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:13:56 smithi007 ceph-mon[39249]: from='client.? 172.21.15.7:0/3881307331' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-18T08:13:56.982 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:13:56 smithi007 ceph-mon[39249]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-18T08:13:56.998 INFO:journalctl@ceph.grafana.a.smithi050.stdout:Sep 18 08:13:56 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-grafana-a[60403]: logger=grafana.update.checker t=2024-09-18T08:13:56.693975579Z level=info msg="Update check succeeded" duration=95.713316ms 2024-09-18T08:13:56.999 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:13:56 smithi050 ceph-mon[32371]: 1.1 scrub starts 2024-09-18T08:13:56.999 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:13:56 smithi050 ceph-mon[32371]: 1.1 scrub ok 2024-09-18T08:13:56.999 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:13:56 smithi050 ceph-mon[32371]: purged_snaps scrub starts 2024-09-18T08:13:56.999 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:13:56 smithi050 ceph-mon[32371]: purged_snaps scrub ok 2024-09-18T08:13:56.999 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:13:56 smithi050 ceph-mon[32371]: osdmap e290: 8 total, 7 up, 8 in 2024-09-18T08:13:56.999 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:13:56 smithi050 ceph-mon[32371]: from='client.? 172.21.15.7:0/3881307331' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-18T08:13:56.999 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:13:56 smithi050 ceph-mon[32371]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-18T08:13:57.013 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4 2024-09-18T08:13:57.047 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:13:57.048 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:13:57.049 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4' 2024-09-18T08:13:57.149 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3 2024-09-18T08:13:57.177 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:13:57.178 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:13:57.178 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3' 2024-09-18T08:13:57.279 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6 2024-09-18T08:13:57.317 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:13:57.318 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:13:57.318 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6' 2024-09-18T08:13:57.365 DEBUG:teuthology.orchestra.run.smithi050:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid 779349ca-7593-11ef-bceb-c7b262605968 -- ceph daemon osd.7 dump_blocked_ops 2024-09-18T08:13:57.420 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2 2024-09-18T08:13:57.447 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:13:57.448 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:13:57.449 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2' 2024-09-18T08:13:57.549 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4 2024-09-18T08:13:57.602 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:13:57.602 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:13:57.602 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4' 2024-09-18T08:13:57.703 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4 2024-09-18T08:13:57.754 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:13:57.754 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:13:57.755 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4' 2024-09-18T08:13:57.825 INFO:teuthology.orchestra.run.smithi050.stderr:Inferring config /var/lib/ceph/779349ca-7593-11ef-bceb-c7b262605968/mon.b/config 2024-09-18T08:13:57.856 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5 2024-09-18T08:13:57.895 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:13:57.895 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:13:57.895 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5' 2024-09-18T08:13:57.996 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7 2024-09-18T08:13:58.033 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:13:58.034 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:13:58.034 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7' 2024-09-18T08:13:58.116 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:13:57 smithi050 ceph-mon[32371]: 6.12 scrub starts 2024-09-18T08:13:58.116 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:13:57 smithi050 ceph-mon[32371]: 6.12 scrub ok 2024-09-18T08:13:58.116 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:13:57 smithi050 ceph-mon[32371]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-18T08:13:58.116 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:13:57 smithi050 ceph-mon[32371]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-18T08:13:58.116 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:13:57 smithi050 ceph-mon[32371]: osdmap e291: 8 total, 7 up, 8 in 2024-09-18T08:13:58.116 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:13:57 smithi050 ceph-mon[32371]: pgmap v541: 158 pgs: 30 active+undersized, 34 active+undersized+degraded, 94 active+clean; 2.8 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 852 B/s rd, 0 B/s wr, 957 op/s; 26202/138867 objects degraded (18.868%) 2024-09-18T08:13:58.135 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5 2024-09-18T08:13:58.172 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:13:58.173 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:13:58.173 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5' 2024-09-18T08:13:58.191 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 5, 7, 6, 4] out_osds: [] dead_osds: [1] live_osds: [0, 2, 3, 4, 5, 6, 7] 2024-09-18T08:13:58.191 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-18T08:13:58.191 INFO:tasks.thrashosds.thrasher:Removing osd 7, in_osds are: [0, 1, 2, 3, 5, 7, 6, 4] 2024-09-18T08:13:58.191 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid 779349ca-7593-11ef-bceb-c7b262605968 -- ceph osd out 7 2024-09-18T08:13:58.218 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:13:57 smithi007 ceph-mon[29459]: 6.12 scrub starts 2024-09-18T08:13:58.219 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:13:57 smithi007 ceph-mon[29459]: 6.12 scrub ok 2024-09-18T08:13:58.219 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:13:57 smithi007 ceph-mon[29459]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-18T08:13:58.219 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:13:57 smithi007 ceph-mon[29459]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-18T08:13:58.219 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:13:57 smithi007 ceph-mon[29459]: osdmap e291: 8 total, 7 up, 8 in 2024-09-18T08:13:58.219 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:13:57 smithi007 ceph-mon[29459]: pgmap v541: 158 pgs: 30 active+undersized, 34 active+undersized+degraded, 94 active+clean; 2.8 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 852 B/s rd, 0 B/s wr, 957 op/s; 26202/138867 objects degraded (18.868%) 2024-09-18T08:13:58.219 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:13:57 smithi007 ceph-mon[39249]: 6.12 scrub starts 2024-09-18T08:13:58.219 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:13:57 smithi007 ceph-mon[39249]: 6.12 scrub ok 2024-09-18T08:13:58.219 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:13:57 smithi007 ceph-mon[39249]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-18T08:13:58.219 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:13:57 smithi007 ceph-mon[39249]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-18T08:13:58.220 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:13:57 smithi007 ceph-mon[39249]: osdmap e291: 8 total, 7 up, 8 in 2024-09-18T08:13:58.220 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:13:57 smithi007 ceph-mon[39249]: pgmap v541: 158 pgs: 30 active+undersized, 34 active+undersized+degraded, 94 active+clean; 2.8 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 852 B/s rd, 0 B/s wr, 957 op/s; 26202/138867 objects degraded (18.868%) 2024-09-18T08:13:58.274 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4 2024-09-18T08:13:58.315 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:13:58.315 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:13:58.316 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4' 2024-09-18T08:13:58.416 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6 2024-09-18T08:13:58.439 INFO:teuthology.orchestra.run.smithi050.stdout:ERROR: (22) Invalid argument 2024-09-18T08:13:58.440 INFO:teuthology.orchestra.run.smithi050.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-18T08:13:58.458 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:13:58.458 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:13:58.459 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6' 2024-09-18T08:13:58.559 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2 2024-09-18T08:13:58.591 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:13:58.591 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:13:58.592 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2' 2024-09-18T08:13:58.693 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4 2024-09-18T08:13:58.719 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:13:58.719 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:13:58.719 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4' 2024-09-18T08:13:58.820 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5 2024-09-18T08:13:58.859 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:13:58.860 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:13:58.860 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5' 2024-09-18T08:13:58.961 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2 2024-09-18T08:13:58.995 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:13:58.996 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:13:58.996 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2' 2024-09-18T08:13:59.097 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0 2024-09-18T08:13:59.128 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:13:59.128 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:13:59.128 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0' 2024-09-18T08:13:59.229 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6 2024-09-18T08:13:59.258 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:13:59.258 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:13:59.258 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6' 2024-09-18T08:13:59.277 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:13:59 smithi050 ceph-mon[32371]: 9.5 scrub starts 2024-09-18T08:13:59.321 DEBUG:teuthology.orchestra.run.smithi050:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid 779349ca-7593-11ef-bceb-c7b262605968 -- ceph daemon osd.7 dump_historic_ops 2024-09-18T08:13:59.360 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0 2024-09-18T08:13:59.393 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:13:59.394 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:13:59.394 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0' 2024-09-18T08:13:59.480 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:13:59 smithi007 ceph-mon[29459]: 9.5 scrub starts 2024-09-18T08:13:59.480 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:13:59 smithi007 ceph-mon[39249]: 9.5 scrub starts 2024-09-18T08:13:59.495 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4 2024-09-18T08:13:59.535 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:13:59.536 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:13:59.536 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4' 2024-09-18T08:13:59.637 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6 2024-09-18T08:13:59.655 INFO:journalctl@ceph.iscsi.iscsi.a.smithi050.stdout:Sep 18 08:13:59 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-iscsi-iscsi-a[57782]: debug there is no tcmu-runner data available 2024-09-18T08:13:59.687 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:13:59.688 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:13:59.688 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6' 2024-09-18T08:13:59.783 INFO:teuthology.orchestra.run.smithi050.stderr:Inferring config /var/lib/ceph/779349ca-7593-11ef-bceb-c7b262605968/mon.b/config 2024-09-18T08:13:59.789 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5 2024-09-18T08:13:59.818 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:13:59.819 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:13:59.819 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5' 2024-09-18T08:13:59.920 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7 2024-09-18T08:13:59.966 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:13:59.967 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:13:59.967 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7' 2024-09-18T08:14:00.068 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0 2024-09-18T08:14:00.096 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:14:00.097 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:14:00.097 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0' 2024-09-18T08:14:00.198 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2 2024-09-18T08:14:00.230 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:14:00 smithi007 ceph-mon[29459]: Health check update: Degraded data redundancy: 26202/138867 objects degraded (18.868%), 34 pgs degraded, 63 pgs undersized (PG_DEGRADED) 2024-09-18T08:14:00.231 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:14:00 smithi007 ceph-mon[29459]: pgmap v542: 158 pgs: 1 active+clean+scrubbing, 30 active+undersized, 34 active+undersized+degraded, 93 active+clean; 2.4 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 1.3 KiB/s rd, 0 B/s wr, 1.63k op/s; 22476/118998 objects degraded (18.888%) 2024-09-18T08:14:00.231 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:14:00 smithi007 ceph-mon[29459]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-18T08:14:00.231 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:14:00 smithi007 ceph-mon[29459]: 9.5 scrub ok 2024-09-18T08:14:00.231 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:14:00 smithi007 ceph-mon[39249]: Health check update: Degraded data redundancy: 26202/138867 objects degraded (18.868%), 34 pgs degraded, 63 pgs undersized (PG_DEGRADED) 2024-09-18T08:14:00.231 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:14:00 smithi007 ceph-mon[39249]: pgmap v542: 158 pgs: 1 active+clean+scrubbing, 30 active+undersized, 34 active+undersized+degraded, 93 active+clean; 2.4 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 1.3 KiB/s rd, 0 B/s wr, 1.63k op/s; 22476/118998 objects degraded (18.888%) 2024-09-18T08:14:00.231 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:14:00 smithi007 ceph-mon[39249]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-18T08:14:00.231 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:14:00 smithi007 ceph-mon[39249]: 9.5 scrub ok 2024-09-18T08:14:00.232 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:14:00.232 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:14:00.233 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2' 2024-09-18T08:14:00.334 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3 2024-09-18T08:14:00.361 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:14:00.362 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:14:00.362 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3' 2024-09-18T08:14:00.420 INFO:teuthology.orchestra.run.smithi050.stdout:ERROR: (22) Invalid argument 2024-09-18T08:14:00.420 INFO:teuthology.orchestra.run.smithi050.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-18T08:14:00.463 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4 2024-09-18T08:14:00.484 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:14:00 smithi050 ceph-mon[32371]: Health check update: Degraded data redundancy: 26202/138867 objects degraded (18.868%), 34 pgs degraded, 63 pgs undersized (PG_DEGRADED) 2024-09-18T08:14:00.484 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:14:00 smithi050 ceph-mon[32371]: pgmap v542: 158 pgs: 1 active+clean+scrubbing, 30 active+undersized, 34 active+undersized+degraded, 93 active+clean; 2.4 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 1.3 KiB/s rd, 0 B/s wr, 1.63k op/s; 22476/118998 objects degraded (18.888%) 2024-09-18T08:14:00.484 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:14:00 smithi050 ceph-mon[32371]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-18T08:14:00.484 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:14:00 smithi050 ceph-mon[32371]: 9.5 scrub ok 2024-09-18T08:14:00.505 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:14:00.505 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:14:00.505 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4' 2024-09-18T08:14:00.606 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0 2024-09-18T08:14:00.640 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:14:00.641 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:14:00.641 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0' 2024-09-18T08:14:00.742 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2 2024-09-18T08:14:00.775 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:14:00.775 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:14:00.776 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2' 2024-09-18T08:14:00.877 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2 2024-09-18T08:14:00.909 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:14:00.909 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:14:00.910 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2' 2024-09-18T08:14:01.011 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5 2024-09-18T08:14:01.048 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:14:01.048 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:14:01.049 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5' 2024-09-18T08:14:01.149 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0 2024-09-18T08:14:01.153 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid 779349ca-7593-11ef-bceb-c7b262605968 -- ceph daemon osd.0 dump_ops_in_flight 2024-09-18T08:14:01.176 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:14:01.177 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:14:01.178 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0' 2024-09-18T08:14:01.278 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7 2024-09-18T08:14:01.316 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:14:01.317 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:14:01.317 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7' 2024-09-18T08:14:01.418 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4 2024-09-18T08:14:01.452 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:14:01.453 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:14:01.453 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4' 2024-09-18T08:14:01.554 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4 2024-09-18T08:14:01.593 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:14:01.593 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:14:01.593 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4' 2024-09-18T08:14:01.694 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2 2024-09-18T08:14:01.730 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:14:01.731 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:14:01.731 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2' 2024-09-18T08:14:01.832 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6 2024-09-18T08:14:01.843 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/779349ca-7593-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T08:14:01.863 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:14:01.864 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:14:01.864 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6' 2024-09-18T08:14:01.965 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2 2024-09-18T08:14:01.994 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:14:01.995 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:14:01.995 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2' 2024-09-18T08:14:02.096 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4 2024-09-18T08:14:02.136 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:14:02.136 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:14:02.137 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4' 2024-09-18T08:14:02.156 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/779349ca-7593-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T08:14:02.158 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/779349ca-7593-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T08:14:02.237 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0 2024-09-18T08:14:02.269 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:14:02.269 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:14:02.269 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0' 2024-09-18T08:14:02.370 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5 2024-09-18T08:14:02.405 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:14:02.405 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:14:02.405 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5' 2024-09-18T08:14:02.496 INFO:teuthology.orchestra.run.smithi007.stdout:ERROR: (22) Invalid argument 2024-09-18T08:14:02.497 INFO:teuthology.orchestra.run.smithi007.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-18T08:14:02.506 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7 2024-09-18T08:14:02.545 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:14:02.545 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:14:02.546 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7' 2024-09-18T08:14:02.584 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:14:02 smithi007 ceph-mon[29459]: pgmap v543: 158 pgs: 1 active+clean+scrubbing, 30 active+undersized, 34 active+undersized+degraded, 93 active+clean; 2.4 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.55k op/s; 22476/118998 objects degraded (18.888%) 2024-09-18T08:14:02.584 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:14:02 smithi007 ceph-mon[29459]: 1.2 scrub starts 2024-09-18T08:14:02.584 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:14:02 smithi007 ceph-mon[29459]: 1.2 scrub ok 2024-09-18T08:14:02.584 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:14:02 smithi007 ceph-mon[39249]: pgmap v543: 158 pgs: 1 active+clean+scrubbing, 30 active+undersized, 34 active+undersized+degraded, 93 active+clean; 2.4 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.55k op/s; 22476/118998 objects degraded (18.888%) 2024-09-18T08:14:02.584 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:14:02 smithi007 ceph-mon[39249]: 1.2 scrub starts 2024-09-18T08:14:02.584 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:14:02 smithi007 ceph-mon[39249]: 1.2 scrub ok 2024-09-18T08:14:02.647 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7 2024-09-18T08:14:02.656 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:14:02 smithi050 ceph-mon[32371]: pgmap v543: 158 pgs: 1 active+clean+scrubbing, 30 active+undersized, 34 active+undersized+degraded, 93 active+clean; 2.4 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.55k op/s; 22476/118998 objects degraded (18.888%) 2024-09-18T08:14:02.656 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:14:02 smithi050 ceph-mon[32371]: 1.2 scrub starts 2024-09-18T08:14:02.656 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:14:02 smithi050 ceph-mon[32371]: 1.2 scrub ok 2024-09-18T08:14:02.683 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:14:02.684 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:14:02.685 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7' 2024-09-18T08:14:02.785 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4 2024-09-18T08:14:02.819 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:14:02.819 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:14:02.820 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4' 2024-09-18T08:14:02.921 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4 2024-09-18T08:14:02.956 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:14:02.956 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:14:02.957 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4' 2024-09-18T08:14:03.057 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7 2024-09-18T08:14:03.091 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:14:03.091 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:14:03.092 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7' 2024-09-18T08:14:03.193 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6 2024-09-18T08:14:03.231 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:14:03.232 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:14:03.232 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6' 2024-09-18T08:14:03.333 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4 2024-09-18T08:14:03.368 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:14:03.369 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:14:03.369 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4' 2024-09-18T08:14:03.470 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5 2024-09-18T08:14:03.511 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:14:03.511 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:14:03.512 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5' 2024-09-18T08:14:03.575 INFO:teuthology.orchestra.run.smithi007.stdout:osd.0: {} 2024-09-18T08:14:03.575 INFO:teuthology.orchestra.run.smithi007.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-18T08:14:03.575 INFO:teuthology.orchestra.run.smithi007.stderr:Error ENXIO: problem getting command descriptions from osd.1 2024-09-18T08:14:03.576 INFO:teuthology.orchestra.run.smithi007.stdout:osd.2: {} 2024-09-18T08:14:03.577 INFO:teuthology.orchestra.run.smithi007.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-18T08:14:03.604 INFO:teuthology.orchestra.run.smithi007.stdout:osd.3: {} 2024-09-18T08:14:03.604 INFO:teuthology.orchestra.run.smithi007.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-18T08:14:03.612 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0 2024-09-18T08:14:03.636 INFO:teuthology.orchestra.run.smithi007.stdout:osd.4: {} 2024-09-18T08:14:03.637 INFO:teuthology.orchestra.run.smithi007.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-18T08:14:03.637 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:14:03 smithi007 ceph-mon[29459]: 1.6 scrub starts 2024-09-18T08:14:03.637 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:14:03 smithi007 ceph-mon[29459]: 1.6 scrub ok 2024-09-18T08:14:03.637 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:14:03 smithi007 ceph-mon[39249]: 1.6 scrub starts 2024-09-18T08:14:03.638 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:14:03 smithi007 ceph-mon[39249]: 1.6 scrub ok 2024-09-18T08:14:03.651 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:14:03.651 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:14:03.651 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0' 2024-09-18T08:14:03.655 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:14:03 smithi050 ceph-mon[32371]: 1.6 scrub starts 2024-09-18T08:14:03.655 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:14:03 smithi050 ceph-mon[32371]: 1.6 scrub ok 2024-09-18T08:14:03.672 INFO:teuthology.orchestra.run.smithi007.stdout:osd.5: {} 2024-09-18T08:14:03.672 INFO:teuthology.orchestra.run.smithi007.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-18T08:14:03.704 INFO:teuthology.orchestra.run.smithi007.stdout:osd.6: {} 2024-09-18T08:14:03.704 INFO:teuthology.orchestra.run.smithi007.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-18T08:14:03.731 INFO:teuthology.orchestra.run.smithi007.stdout:osd.7: {} 2024-09-18T08:14:03.731 INFO:teuthology.orchestra.run.smithi007.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-18T08:14:03.752 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4 2024-09-18T08:14:03.786 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:14:03.787 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:14:03.787 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4' 2024-09-18T08:14:03.801 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid 779349ca-7593-11ef-bceb-c7b262605968 -- ceph osd set noscrub 2024-09-18T08:14:03.888 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2 2024-09-18T08:14:03.927 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:14:03.927 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:14:03.928 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2' 2024-09-18T08:14:04.028 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5 2024-09-18T08:14:04.062 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:14:04.063 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:14:04.063 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5' 2024-09-18T08:14:04.164 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0 2024-09-18T08:14:04.204 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:14:04.205 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:14:04.205 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0' 2024-09-18T08:14:04.228 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:14:03 smithi007 ceph-779349ca-7593-11ef-bceb-c7b262605968-mon-a[29455]: 2024-09-18T08:14:03.970+0000 7fcd2b595640 -1 mon.a@0(leader).osd e291 definitely_dead 0 2024-09-18T08:14:04.230 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid 779349ca-7593-11ef-bceb-c7b262605968 -- ceph daemon osd.0 dump_blocked_ops 2024-09-18T08:14:04.306 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4 2024-09-18T08:14:04.340 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:14:04.341 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:14:04.341 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4' 2024-09-18T08:14:04.442 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3 2024-09-18T08:14:04.478 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:14:04.479 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:14:04.480 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3' 2024-09-18T08:14:04.481 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:14:04 smithi007 ceph-mon[29459]: pgmap v544: 158 pgs: 1 active+clean+scrubbing, 30 active+undersized, 34 active+undersized+degraded, 93 active+clean; 2.4 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 828 op/s; 22476/118998 objects degraded (18.888%) 2024-09-18T08:14:04.481 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:14:04 smithi007 ceph-mon[29459]: from='client.? 172.21.15.7:0/1019738648' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-18T08:14:04.481 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:14:04 smithi007 ceph-mon[29459]: from='mgr.24470 172.21.15.7:0/2594390261' entity='mgr.y' 2024-09-18T08:14:04.481 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:14:04 smithi007 ceph-mon[29459]: from='mgr.24470 172.21.15.7:0/2594390261' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-18T08:14:04.481 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:14:04 smithi007 ceph-mon[29459]: from='client.? 172.21.15.7:0/1260985188' entity='client.admin' cmd={"prefix": "osd out", "ids": ["7"]} : dispatch 2024-09-18T08:14:04.482 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:14:04 smithi007 ceph-mon[29459]: Client client.admin marked osd.7 out, while it was still marked up 2024-09-18T08:14:04.482 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:14:04 smithi007 ceph-mon[29459]: Health check update: Degraded data redundancy: 22476/118998 objects degraded (18.888%), 34 pgs degraded, 63 pgs undersized (PG_DEGRADED) 2024-09-18T08:14:04.482 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:14:04 smithi007 ceph-mon[39249]: pgmap v544: 158 pgs: 1 active+clean+scrubbing, 30 active+undersized, 34 active+undersized+degraded, 93 active+clean; 2.4 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 828 op/s; 22476/118998 objects degraded (18.888%) 2024-09-18T08:14:04.482 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:14:04 smithi007 ceph-mon[39249]: from='client.? 172.21.15.7:0/1019738648' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-18T08:14:04.482 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:14:04 smithi007 ceph-mon[39249]: from='mgr.24470 172.21.15.7:0/2594390261' entity='mgr.y' 2024-09-18T08:14:04.482 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:14:04 smithi007 ceph-mon[39249]: from='mgr.24470 172.21.15.7:0/2594390261' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-18T08:14:04.482 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:14:04 smithi007 ceph-mon[39249]: from='client.? 172.21.15.7:0/1260985188' entity='client.admin' cmd={"prefix": "osd out", "ids": ["7"]} : dispatch 2024-09-18T08:14:04.483 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:14:04 smithi007 ceph-mon[39249]: Client client.admin marked osd.7 out, while it was still marked up 2024-09-18T08:14:04.483 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:14:04 smithi007 ceph-mon[39249]: Health check update: Degraded data redundancy: 22476/118998 objects degraded (18.888%), 34 pgs degraded, 63 pgs undersized (PG_DEGRADED) 2024-09-18T08:14:04.562 INFO:teuthology.orchestra.run.smithi007.stderr:marked out osd.7. 2024-09-18T08:14:04.580 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2 2024-09-18T08:14:04.613 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:14:04.614 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:14:04.616 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2' 2024-09-18T08:14:04.656 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:14:04 smithi050 ceph-mon[32371]: pgmap v544: 158 pgs: 1 active+clean+scrubbing, 30 active+undersized, 34 active+undersized+degraded, 93 active+clean; 2.4 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 828 op/s; 22476/118998 objects degraded (18.888%) 2024-09-18T08:14:04.656 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:14:04 smithi050 ceph-mon[32371]: from='client.? 172.21.15.7:0/1019738648' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-18T08:14:04.656 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:14:04 smithi050 ceph-mon[32371]: from='mgr.24470 172.21.15.7:0/2594390261' entity='mgr.y' 2024-09-18T08:14:04.656 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:14:04 smithi050 ceph-mon[32371]: from='mgr.24470 172.21.15.7:0/2594390261' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-18T08:14:04.656 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:14:04 smithi050 ceph-mon[32371]: from='client.? 172.21.15.7:0/1260985188' entity='client.admin' cmd={"prefix": "osd out", "ids": ["7"]} : dispatch 2024-09-18T08:14:04.656 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:14:04 smithi050 ceph-mon[32371]: Client client.admin marked osd.7 out, while it was still marked up 2024-09-18T08:14:04.656 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:14:04 smithi050 ceph-mon[32371]: Health check update: Degraded data redundancy: 22476/118998 objects degraded (18.888%), 34 pgs degraded, 63 pgs undersized (PG_DEGRADED) 2024-09-18T08:14:04.717 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6 2024-09-18T08:14:04.740 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:14:04.740 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:14:04.741 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6' 2024-09-18T08:14:04.842 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6 2024-09-18T08:14:04.864 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:14:04.865 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:14:04.865 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6' 2024-09-18T08:14:04.966 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7 2024-09-18T08:14:04.989 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:14:04.989 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:14:04.990 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7' 2024-09-18T08:14:05.083 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-09-18T08:14:05.084 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-09-18T08:14:05.090 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0 2024-09-18T08:14:05.113 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:14:05.114 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:14:05.114 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0' 2024-09-18T08:14:05.215 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2 2024-09-18T08:14:05.238 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:14:05.239 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:14:05.240 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2' 2024-09-18T08:14:05.340 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2 2024-09-18T08:14:05.365 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:14:05.366 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:14:05.366 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2' 2024-09-18T08:14:05.394 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/779349ca-7593-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T08:14:05.428 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/779349ca-7593-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T08:14:05.467 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5 2024-09-18T08:14:05.494 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:14:05.494 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:14:05.495 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5' 2024-09-18T08:14:05.595 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5 2024-09-18T08:14:05.626 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:14:05.627 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:14:05.627 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5' 2024-09-18T08:14:05.668 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:14:05 smithi007 ceph-mon[29459]: from='client.? 172.21.15.7:0/1260985188' entity='client.admin' cmd='[{"prefix": "osd out", "ids": ["7"]}]': finished 2024-09-18T08:14:05.668 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:14:05 smithi007 ceph-mon[29459]: osdmap e292: 8 total, 7 up, 7 in 2024-09-18T08:14:05.668 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:14:05 smithi007 ceph-mon[29459]: 3.b scrub starts 2024-09-18T08:14:05.668 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:14:05 smithi007 ceph-mon[29459]: 3.b scrub ok 2024-09-18T08:14:05.669 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:14:05 smithi007 ceph-mon[39249]: from='client.? 172.21.15.7:0/1260985188' entity='client.admin' cmd='[{"prefix": "osd out", "ids": ["7"]}]': finished 2024-09-18T08:14:05.669 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:14:05 smithi007 ceph-mon[39249]: osdmap e292: 8 total, 7 up, 7 in 2024-09-18T08:14:05.669 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:14:05 smithi007 ceph-mon[39249]: 3.b scrub starts 2024-09-18T08:14:05.669 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:14:05 smithi007 ceph-mon[39249]: 3.b scrub ok 2024-09-18T08:14:05.729 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4 2024-09-18T08:14:05.753 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:14:05.754 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:14:05.754 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4' 2024-09-18T08:14:05.855 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3 2024-09-18T08:14:05.878 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:14:05.878 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:14:05.879 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3' 2024-09-18T08:14:05.905 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:14:05 smithi050 ceph-mon[32371]: from='client.? 172.21.15.7:0/1260985188' entity='client.admin' cmd='[{"prefix": "osd out", "ids": ["7"]}]': finished 2024-09-18T08:14:05.906 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:14:05 smithi050 ceph-mon[32371]: osdmap e292: 8 total, 7 up, 7 in 2024-09-18T08:14:05.906 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:14:05 smithi050 ceph-mon[32371]: 3.b scrub starts 2024-09-18T08:14:05.906 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:14:05 smithi050 ceph-mon[32371]: 3.b scrub ok 2024-09-18T08:14:05.979 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3 2024-09-18T08:14:06.001 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:14:06.002 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:14:06.002 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3' 2024-09-18T08:14:06.103 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2 2024-09-18T08:14:06.125 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:14:06.126 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:14:06.126 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2' 2024-09-18T08:14:06.177 INFO:teuthology.orchestra.run.smithi007.stdout:{ 2024-09-18T08:14:06.178 INFO:teuthology.orchestra.run.smithi007.stdout: "ops": [], 2024-09-18T08:14:06.178 INFO:teuthology.orchestra.run.smithi007.stdout: "complaint_time": 30, 2024-09-18T08:14:06.178 INFO:teuthology.orchestra.run.smithi007.stdout: "num_blocked_ops": 0 2024-09-18T08:14:06.178 INFO:teuthology.orchestra.run.smithi007.stdout:} 2024-09-18T08:14:06.228 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5 2024-09-18T08:14:06.256 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:14:06.257 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:14:06.257 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5' 2024-09-18T08:14:06.358 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3 2024-09-18T08:14:06.380 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:14:06.380 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:14:06.381 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3' 2024-09-18T08:14:06.481 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3 2024-09-18T08:14:06.504 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:14:06.504 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:14:06.505 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3' 2024-09-18T08:14:06.605 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4 2024-09-18T08:14:06.639 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:14:06.639 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:14:06.640 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4' 2024-09-18T08:14:06.741 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3 2024-09-18T08:14:06.765 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:14:06.766 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:14:06.766 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3' 2024-09-18T08:14:06.867 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0 2024-09-18T08:14:06.893 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:14:06.894 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:14:06.894 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0' 2024-09-18T08:14:06.905 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:14:06 smithi050 ceph-mon[32371]: pgmap v546: 158 pgs: 30 active+undersized, 34 active+undersized+degraded, 94 active+clean; 2.0 GiB data, 8.2 GiB used, 618 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.59k op/s; 18546/98061 objects degraded (18.913%) 2024-09-18T08:14:06.905 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:14:06 smithi050 ceph-mon[32371]: osdmap e293: 8 total, 7 up, 7 in 2024-09-18T08:14:06.905 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:14:06 smithi050 ceph-mon[32371]: from='client.? 172.21.15.7:0/2967588325' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-18T08:14:06.980 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:14:06 smithi007 ceph-mon[29459]: pgmap v546: 158 pgs: 30 active+undersized, 34 active+undersized+degraded, 94 active+clean; 2.0 GiB data, 8.2 GiB used, 618 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.59k op/s; 18546/98061 objects degraded (18.913%) 2024-09-18T08:14:06.980 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:14:06 smithi007 ceph-mon[29459]: osdmap e293: 8 total, 7 up, 7 in 2024-09-18T08:14:06.980 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:14:06 smithi007 ceph-mon[29459]: from='client.? 172.21.15.7:0/2967588325' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-18T08:14:06.980 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Sep 18 08:14:06 smithi007 ceph-779349ca-7593-11ef-bceb-c7b262605968-mgr-y[29688]: ::ffff:172.21.15.50 - - [18/Sep/2024:08:14:06] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-09-18T08:14:06.980 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:14:06 smithi007 ceph-mon[39249]: pgmap v546: 158 pgs: 30 active+undersized, 34 active+undersized+degraded, 94 active+clean; 2.0 GiB data, 8.2 GiB used, 618 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.59k op/s; 18546/98061 objects degraded (18.913%) 2024-09-18T08:14:06.981 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:14:06 smithi007 ceph-mon[39249]: osdmap e293: 8 total, 7 up, 7 in 2024-09-18T08:14:06.981 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:14:06 smithi007 ceph-mon[39249]: from='client.? 172.21.15.7:0/2967588325' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-18T08:14:06.995 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5 2024-09-18T08:14:07.025 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:14:07.025 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:14:07.026 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5' 2024-09-18T08:14:07.085 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid 779349ca-7593-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-18T08:14:07.126 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0 2024-09-18T08:14:07.179 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:14:07.179 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:14:07.180 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0' 2024-09-18T08:14:07.281 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7 2024-09-18T08:14:07.321 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:14:07.321 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:14:07.322 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7' 2024-09-18T08:14:07.423 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5 2024-09-18T08:14:07.463 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:14:07.464 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:14:07.464 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5' 2024-09-18T08:14:07.565 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5 2024-09-18T08:14:07.603 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:14:07.603 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:14:07.604 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5' 2024-09-18T08:14:07.705 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4 2024-09-18T08:14:07.740 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:14:07.741 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:14:07.741 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4' 2024-09-18T08:14:07.842 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0 2024-09-18T08:14:07.877 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:14:07.878 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:14:07.878 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0' 2024-09-18T08:14:07.906 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:14:07 smithi050 ceph-mon[32371]: purged_snaps scrub starts 2024-09-18T08:14:07.906 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:14:07 smithi050 ceph-mon[32371]: purged_snaps scrub ok 2024-09-18T08:14:07.906 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:14:07 smithi050 ceph-mon[32371]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-18T08:14:07.906 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:14:07 smithi050 ceph-mon[32371]: from='client.? 172.21.15.7:0/2967588325' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-18T08:14:07.906 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:14:07 smithi050 ceph-mon[32371]: 9.c starting backfill to osd.0 from (0'0,0'0] MAX to 291'15734 2024-09-18T08:14:07.906 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:14:07 smithi050 ceph-mon[32371]: 9.3 starting backfill to osd.5 from (0'0,0'0] MAX to 291'16149 2024-09-18T08:14:07.906 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:14:07 smithi050 ceph-mon[32371]: osdmap e294: 8 total, 7 up, 7 in 2024-09-18T08:14:07.906 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:14:07 smithi050 ceph-mon[32371]: 9.e starting backfill to osd.0 from (0'0,0'0] MAX to 291'15728 2024-09-18T08:14:07.906 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:14:07 smithi050 ceph-mon[32371]: 9.1 starting backfill to osd.4 from (0'0,0'0] MAX to 291'15851 2024-09-18T08:14:07.906 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:14:07 smithi050 ceph-mon[32371]: 9.a starting backfill to osd.3 from (0'0,0'0] MAX to 291'15680 2024-09-18T08:14:07.906 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:14:07 smithi050 ceph-mon[32371]: 9.9 starting backfill to osd.3 from (0'0,0'0] MAX to 291'15776 2024-09-18T08:14:07.907 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:14:07 smithi050 ceph-mon[32371]: 9.d starting backfill to osd.4 from (0'0,0'0] MAX to 291'15945 2024-09-18T08:14:07.907 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:14:07 smithi050 ceph-mon[32371]: 9.7 starting backfill to osd.2 from (0'0,0'0] MAX to 291'15488 2024-09-18T08:14:07.979 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5 2024-09-18T08:14:07.980 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:14:07 smithi007 ceph-mon[29459]: purged_snaps scrub starts 2024-09-18T08:14:07.980 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:14:07 smithi007 ceph-mon[29459]: purged_snaps scrub ok 2024-09-18T08:14:07.980 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:14:07 smithi007 ceph-mon[29459]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-18T08:14:07.980 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:14:07 smithi007 ceph-mon[29459]: from='client.? 172.21.15.7:0/2967588325' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-18T08:14:07.980 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:14:07 smithi007 ceph-mon[29459]: 9.c starting backfill to osd.0 from (0'0,0'0] MAX to 291'15734 2024-09-18T08:14:07.980 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:14:07 smithi007 ceph-mon[29459]: 9.3 starting backfill to osd.5 from (0'0,0'0] MAX to 291'16149 2024-09-18T08:14:07.981 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:14:07 smithi007 ceph-mon[29459]: osdmap e294: 8 total, 7 up, 7 in 2024-09-18T08:14:07.981 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:14:07 smithi007 ceph-mon[29459]: 9.e starting backfill to osd.0 from (0'0,0'0] MAX to 291'15728 2024-09-18T08:14:07.981 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:14:07 smithi007 ceph-mon[29459]: 9.1 starting backfill to osd.4 from (0'0,0'0] MAX to 291'15851 2024-09-18T08:14:07.981 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:14:07 smithi007 ceph-mon[29459]: 9.a starting backfill to osd.3 from (0'0,0'0] MAX to 291'15680 2024-09-18T08:14:07.981 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:14:07 smithi007 ceph-mon[29459]: 9.9 starting backfill to osd.3 from (0'0,0'0] MAX to 291'15776 2024-09-18T08:14:07.981 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:14:07 smithi007 ceph-mon[29459]: 9.d starting backfill to osd.4 from (0'0,0'0] MAX to 291'15945 2024-09-18T08:14:07.981 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:14:07 smithi007 ceph-mon[29459]: 9.7 starting backfill to osd.2 from (0'0,0'0] MAX to 291'15488 2024-09-18T08:14:07.981 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:14:07 smithi007 ceph-mon[39249]: purged_snaps scrub starts 2024-09-18T08:14:07.981 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:14:07 smithi007 ceph-mon[39249]: purged_snaps scrub ok 2024-09-18T08:14:07.981 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:14:07 smithi007 ceph-mon[39249]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-18T08:14:07.981 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:14:07 smithi007 ceph-mon[39249]: from='client.? 172.21.15.7:0/2967588325' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-18T08:14:07.982 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:14:07 smithi007 ceph-mon[39249]: 9.c starting backfill to osd.0 from (0'0,0'0] MAX to 291'15734 2024-09-18T08:14:07.982 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:14:07 smithi007 ceph-mon[39249]: 9.3 starting backfill to osd.5 from (0'0,0'0] MAX to 291'16149 2024-09-18T08:14:07.982 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:14:07 smithi007 ceph-mon[39249]: osdmap e294: 8 total, 7 up, 7 in 2024-09-18T08:14:07.982 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:14:07 smithi007 ceph-mon[39249]: 9.e starting backfill to osd.0 from (0'0,0'0] MAX to 291'15728 2024-09-18T08:14:07.982 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:14:07 smithi007 ceph-mon[39249]: 9.1 starting backfill to osd.4 from (0'0,0'0] MAX to 291'15851 2024-09-18T08:14:07.982 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:14:07 smithi007 ceph-mon[39249]: 9.a starting backfill to osd.3 from (0'0,0'0] MAX to 291'15680 2024-09-18T08:14:07.982 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:14:07 smithi007 ceph-mon[39249]: 9.9 starting backfill to osd.3 from (0'0,0'0] MAX to 291'15776 2024-09-18T08:14:07.982 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:14:07 smithi007 ceph-mon[39249]: 9.d starting backfill to osd.4 from (0'0,0'0] MAX to 291'15945 2024-09-18T08:14:07.982 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:14:07 smithi007 ceph-mon[39249]: 9.7 starting backfill to osd.2 from (0'0,0'0] MAX to 291'15488 2024-09-18T08:14:08.021 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:14:08.022 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:14:08.022 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5' 2024-09-18T08:14:08.123 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2 2024-09-18T08:14:08.161 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:14:08.162 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:14:08.162 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2' 2024-09-18T08:14:08.263 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3 2024-09-18T08:14:08.299 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:14:08.300 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:14:08.301 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3' 2024-09-18T08:14:08.401 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2 2024-09-18T08:14:08.442 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:14:08.443 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:14:08.443 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2' 2024-09-18T08:14:08.544 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7 2024-09-18T08:14:08.577 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:14:08.580 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:14:08.581 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7' 2024-09-18T08:14:08.681 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4 2024-09-18T08:14:08.724 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:14:08.725 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:14:08.725 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4' 2024-09-18T08:14:08.767 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid 779349ca-7593-11ef-bceb-c7b262605968 -- ceph daemon osd.0 dump_historic_ops 2024-09-18T08:14:08.826 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2 2024-09-18T08:14:08.882 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:14:08.883 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:14:08.883 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2' 2024-09-18T08:14:08.905 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:14:08 smithi050 ceph-mon[32371]: pgmap v549: 158 pgs: 30 active+undersized, 34 active+undersized+degraded, 94 active+clean; 2.0 GiB data, 8.2 GiB used, 618 GiB / 626 GiB avail; 853 B/s rd, 0 B/s wr, 1.16k op/s; 18546/98061 objects degraded (18.913%) 2024-09-18T08:14:08.980 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:14:08 smithi007 ceph-mon[29459]: pgmap v549: 158 pgs: 30 active+undersized, 34 active+undersized+degraded, 94 active+clean; 2.0 GiB data, 8.2 GiB used, 618 GiB / 626 GiB avail; 853 B/s rd, 0 B/s wr, 1.16k op/s; 18546/98061 objects degraded (18.913%) 2024-09-18T08:14:08.982 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:14:08 smithi007 ceph-mon[39249]: pgmap v549: 158 pgs: 30 active+undersized, 34 active+undersized+degraded, 94 active+clean; 2.0 GiB data, 8.2 GiB used, 618 GiB / 626 GiB avail; 853 B/s rd, 0 B/s wr, 1.16k op/s; 18546/98061 objects degraded (18.913%) 2024-09-18T08:14:08.984 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4 2024-09-18T08:14:09.011 INFO:teuthology.orchestra.run.smithi007.stderr:noscrub is set 2024-09-18T08:14:09.020 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:14:09.021 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:14:09.021 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4' 2024-09-18T08:14:09.122 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4 2024-09-18T08:14:09.158 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:14:09.159 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:14:09.159 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4' 2024-09-18T08:14:09.261 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5 2024-09-18T08:14:09.297 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:14:09.298 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:14:09.299 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5' 2024-09-18T08:14:09.400 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4 2024-09-18T08:14:09.437 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:14:09.438 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:14:09.438 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4' 2024-09-18T08:14:09.539 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3 2024-09-18T08:14:09.578 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:14:09.579 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:14:09.579 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3' 2024-09-18T08:14:09.655 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:14:09 smithi050 ceph-mon[32371]: Health check update: Degraded data redundancy: 18546/98061 objects degraded (18.913%), 34 pgs degraded, 63 pgs undersized (PG_DEGRADED) 2024-09-18T08:14:09.655 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:14:09 smithi050 ceph-mon[32371]: osdmap e295: 8 total, 7 up, 7 in 2024-09-18T08:14:09.656 INFO:journalctl@ceph.iscsi.iscsi.a.smithi050.stdout:Sep 18 08:14:09 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-iscsi-iscsi-a[57782]: debug there is no tcmu-runner data available 2024-09-18T08:14:09.680 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0 2024-09-18T08:14:09.718 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:14:09.719 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:14:09.720 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0' 2024-09-18T08:14:09.820 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6 2024-09-18T08:14:09.863 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:14:09.863 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:14:09.864 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6' 2024-09-18T08:14:09.918 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/779349ca-7593-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T08:14:09.964 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/779349ca-7593-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T08:14:09.964 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3 2024-09-18T08:14:09.983 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:14:09 smithi007 ceph-mon[29459]: Health check update: Degraded data redundancy: 18546/98061 objects degraded (18.913%), 34 pgs degraded, 63 pgs undersized (PG_DEGRADED) 2024-09-18T08:14:09.984 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:14:09 smithi007 ceph-mon[29459]: osdmap e295: 8 total, 7 up, 7 in 2024-09-18T08:14:09.984 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:14:09 smithi007 ceph-mon[39249]: Health check update: Degraded data redundancy: 18546/98061 objects degraded (18.913%), 34 pgs degraded, 63 pgs undersized (PG_DEGRADED) 2024-09-18T08:14:09.984 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:14:09 smithi007 ceph-mon[39249]: osdmap e295: 8 total, 7 up, 7 in 2024-09-18T08:14:10.019 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:14:10.019 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:14:10.019 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3' 2024-09-18T08:14:10.120 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7 2024-09-18T08:14:10.157 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:14:10.158 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:14:10.158 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7' 2024-09-18T08:14:10.259 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6 2024-09-18T08:14:10.294 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:14:10.294 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:14:10.295 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6' 2024-09-18T08:14:10.377 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 5, 6, 4] out_osds: [7] dead_osds: [1] live_osds: [0, 2, 3, 4, 5, 6, 7] 2024-09-18T08:14:10.378 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-18T08:14:10.378 INFO:tasks.thrashosds.thrasher:Shrinking pool .mgr 2024-09-18T08:14:10.378 INFO:tasks.cephadm.ceph_manager.ceph:contract_pool .mgr by 10 min 8 2024-09-18T08:14:10.378 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid 779349ca-7593-11ef-bceb-c7b262605968 -- ceph pg dump --format=json 2024-09-18T08:14:10.395 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7 2024-09-18T08:14:10.432 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:14:10.433 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:14:10.433 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7' 2024-09-18T08:14:10.534 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3 2024-09-18T08:14:10.575 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:14:10.576 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:14:10.576 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3' 2024-09-18T08:14:10.672 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:14:10 smithi007 ceph-mon[29459]: pgmap v550: 158 pgs: 4 active+undersized+degraded+remapped+backfilling, 3 active+remapped+backfilling, 25 active+clean+remapped, 22 active+undersized, 1 active+remapped+backfill_toofull, 27 active+undersized+degraded, 76 active+clean; 1.8 GiB data, 7.9 GiB used, 618 GiB / 626 GiB avail; 341 B/s rd, 0 B/s wr, 1.75k op/s; 15160/87525 objects degraded (17.321%); 15367/87525 objects misplaced (17.557%); 25 MiB/s, 403 objects/s recovering 2024-09-18T08:14:10.673 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:14:10 smithi007 ceph-mon[29459]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-18T08:14:10.673 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:14:10 smithi007 ceph-mon[29459]: Health check failed: Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull (PG_BACKFILL_FULL) 2024-09-18T08:14:10.673 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:14:10 smithi007 ceph-mon[39249]: pgmap v550: 158 pgs: 4 active+undersized+degraded+remapped+backfilling, 3 active+remapped+backfilling, 25 active+clean+remapped, 22 active+undersized, 1 active+remapped+backfill_toofull, 27 active+undersized+degraded, 76 active+clean; 1.8 GiB data, 7.9 GiB used, 618 GiB / 626 GiB avail; 341 B/s rd, 0 B/s wr, 1.75k op/s; 15160/87525 objects degraded (17.321%); 15367/87525 objects misplaced (17.557%); 25 MiB/s, 403 objects/s recovering 2024-09-18T08:14:10.673 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:14:10 smithi007 ceph-mon[39249]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-18T08:14:10.673 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:14:10 smithi007 ceph-mon[39249]: Health check failed: Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull (PG_BACKFILL_FULL) 2024-09-18T08:14:10.677 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3 2024-09-18T08:14:10.731 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:14:10.732 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:14:10.732 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3' 2024-09-18T08:14:10.833 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2 2024-09-18T08:14:10.852 INFO:teuthology.orchestra.run.smithi007.stdout:ERROR: (22) Invalid argument 2024-09-18T08:14:10.852 INFO:teuthology.orchestra.run.smithi007.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-18T08:14:10.888 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:14:10.889 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:14:10.889 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2' 2024-09-18T08:14:10.900 INFO:teuthology.orchestra.run.smithi007.stdout:osd.0: {} 2024-09-18T08:14:10.900 INFO:teuthology.orchestra.run.smithi007.stdout:osd.2: {} 2024-09-18T08:14:10.900 INFO:teuthology.orchestra.run.smithi007.stdout:osd.3: {} 2024-09-18T08:14:10.901 INFO:teuthology.orchestra.run.smithi007.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-18T08:14:10.901 INFO:teuthology.orchestra.run.smithi007.stderr:Error ENXIO: problem getting command descriptions from osd.1 2024-09-18T08:14:10.901 INFO:teuthology.orchestra.run.smithi007.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-18T08:14:10.901 INFO:teuthology.orchestra.run.smithi007.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-18T08:14:10.905 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:14:10 smithi050 ceph-mon[32371]: pgmap v550: 158 pgs: 4 active+undersized+degraded+remapped+backfilling, 3 active+remapped+backfilling, 25 active+clean+remapped, 22 active+undersized, 1 active+remapped+backfill_toofull, 27 active+undersized+degraded, 76 active+clean; 1.8 GiB data, 7.9 GiB used, 618 GiB / 626 GiB avail; 341 B/s rd, 0 B/s wr, 1.75k op/s; 15160/87525 objects degraded (17.321%); 15367/87525 objects misplaced (17.557%); 25 MiB/s, 403 objects/s recovering 2024-09-18T08:14:10.906 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:14:10 smithi050 ceph-mon[32371]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-18T08:14:10.906 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:14:10 smithi050 ceph-mon[32371]: Health check failed: Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull (PG_BACKFILL_FULL) 2024-09-18T08:14:10.909 INFO:teuthology.orchestra.run.smithi007.stdout:osd.4: {} 2024-09-18T08:14:10.909 INFO:teuthology.orchestra.run.smithi007.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-18T08:14:10.947 INFO:teuthology.orchestra.run.smithi007.stdout:osd.5: {} 2024-09-18T08:14:10.947 INFO:teuthology.orchestra.run.smithi007.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-18T08:14:10.980 INFO:teuthology.orchestra.run.smithi007.stdout:osd.6: {} 2024-09-18T08:14:10.981 INFO:teuthology.orchestra.run.smithi007.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-18T08:14:10.990 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6 2024-09-18T08:14:11.015 INFO:teuthology.orchestra.run.smithi007.stdout:osd.7: {} 2024-09-18T08:14:11.015 INFO:teuthology.orchestra.run.smithi007.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-18T08:14:11.031 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:14:11.031 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:14:11.032 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6' 2024-09-18T08:14:11.133 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2 2024-09-18T08:14:11.173 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:14:11.173 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:14:11.173 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2' 2024-09-18T08:14:11.274 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7 2024-09-18T08:14:11.312 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:14:11.313 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:14:11.314 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7' 2024-09-18T08:14:11.414 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4 2024-09-18T08:14:11.453 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:14:11.454 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:14:11.454 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4' 2024-09-18T08:14:11.555 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7 2024-09-18T08:14:11.594 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:14:11.594 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:14:11.595 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7' 2024-09-18T08:14:11.696 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4 2024-09-18T08:14:11.731 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:14:11.731 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:14:11.732 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4' 2024-09-18T08:14:11.823 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid 779349ca-7593-11ef-bceb-c7b262605968 -- ceph osd set nodeep-scrub 2024-09-18T08:14:11.833 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3 2024-09-18T08:14:11.846 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:14:11 smithi007 ceph-mon[29459]: from='client.? 172.21.15.7:0/2394460826' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-18T08:14:11.846 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:14:11 smithi007 ceph-mon[39249]: from='client.? 172.21.15.7:0/2394460826' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-18T08:14:11.888 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:14:11.889 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:14:11.889 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3' 2024-09-18T08:14:11.905 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:14:11 smithi050 ceph-mon[32371]: from='client.? 172.21.15.7:0/2394460826' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-18T08:14:11.990 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2 2024-09-18T08:14:12.026 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:14:12.027 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:14:12.028 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2' 2024-09-18T08:14:12.129 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3 2024-09-18T08:14:12.171 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:14:12.172 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:14:12.172 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3' 2024-09-18T08:14:12.273 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7 2024-09-18T08:14:12.311 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:14:12.311 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:14:12.312 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7' 2024-09-18T08:14:12.413 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6 2024-09-18T08:14:12.451 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:14:12.452 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:14:12.452 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6' 2024-09-18T08:14:12.553 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6 2024-09-18T08:14:12.593 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:14:12.593 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:14:12.594 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6' 2024-09-18T08:14:12.694 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7 2024-09-18T08:14:12.719 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:14:12.719 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:14:12.720 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7' 2024-09-18T08:14:12.821 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6 2024-09-18T08:14:12.865 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:14:12.866 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:14:12.866 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6' 2024-09-18T08:14:12.905 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:14:12 smithi050 ceph-mon[32371]: pgmap v552: 158 pgs: 4 active+undersized+degraded+remapped+backfilling, 3 active+remapped+backfilling, 25 active+clean+remapped, 22 active+undersized, 1 active+remapped+backfill_toofull, 27 active+undersized+degraded, 76 active+clean; 1.8 GiB data, 7.9 GiB used, 618 GiB / 626 GiB avail; 0 B/s wr, 585 op/s; 15160/87525 objects degraded (17.321%); 15367/87525 objects misplaced (17.557%); 25 MiB/s, 403 objects/s recovering 2024-09-18T08:14:12.967 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0 2024-09-18T08:14:12.980 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:14:12 smithi007 ceph-mon[29459]: pgmap v552: 158 pgs: 4 active+undersized+degraded+remapped+backfilling, 3 active+remapped+backfilling, 25 active+clean+remapped, 22 active+undersized, 1 active+remapped+backfill_toofull, 27 active+undersized+degraded, 76 active+clean; 1.8 GiB data, 7.9 GiB used, 618 GiB / 626 GiB avail; 0 B/s wr, 585 op/s; 15160/87525 objects degraded (17.321%); 15367/87525 objects misplaced (17.557%); 25 MiB/s, 403 objects/s recovering 2024-09-18T08:14:12.980 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:14:12 smithi007 ceph-mon[39249]: pgmap v552: 158 pgs: 4 active+undersized+degraded+remapped+backfilling, 3 active+remapped+backfilling, 25 active+clean+remapped, 22 active+undersized, 1 active+remapped+backfill_toofull, 27 active+undersized+degraded, 76 active+clean; 1.8 GiB data, 7.9 GiB used, 618 GiB / 626 GiB avail; 0 B/s wr, 585 op/s; 15160/87525 objects degraded (17.321%); 15367/87525 objects misplaced (17.557%); 25 MiB/s, 403 objects/s recovering 2024-09-18T08:14:13.002 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:14:13.003 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:14:13.003 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0' 2024-09-18T08:14:13.105 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7 2024-09-18T08:14:13.149 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:14:13.150 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:14:13.151 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7' 2024-09-18T08:14:13.252 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0 2024-09-18T08:14:13.288 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:14:13.289 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:14:13.290 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0' 2024-09-18T08:14:13.390 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4 2024-09-18T08:14:13.429 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:14:13.430 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:14:13.430 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4' 2024-09-18T08:14:13.523 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-09-18T08:14:13.524 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-09-18T08:14:13.531 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7 2024-09-18T08:14:13.568 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:14:13.569 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:14:13.570 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7' 2024-09-18T08:14:13.670 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4 2024-09-18T08:14:13.716 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:14:13.717 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:14:13.717 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4' 2024-09-18T08:14:13.818 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2 2024-09-18T08:14:13.851 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:14:13.852 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:14:13.852 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2' 2024-09-18T08:14:13.953 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5 2024-09-18T08:14:13.987 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:14:13.988 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:14:13.988 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5' 2024-09-18T08:14:14.030 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid 779349ca-7593-11ef-bceb-c7b262605968 -- ceph daemon osd.2 dump_ops_in_flight 2024-09-18T08:14:14.047 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/779349ca-7593-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T08:14:14.052 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/779349ca-7593-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T08:14:14.088 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6 2024-09-18T08:14:14.124 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:14:14.125 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:14:14.125 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6' 2024-09-18T08:14:14.226 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6 2024-09-18T08:14:14.266 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:14:14.266 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:14:14.267 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6' 2024-09-18T08:14:14.368 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6 2024-09-18T08:14:14.408 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:14:14.409 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:14:14.409 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6' 2024-09-18T08:14:14.510 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7 2024-09-18T08:14:14.544 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:14:14.545 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:14:14.545 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7' 2024-09-18T08:14:14.646 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2 2024-09-18T08:14:14.688 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:14:14.688 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:14:14.689 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2' 2024-09-18T08:14:14.789 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5 2024-09-18T08:14:14.830 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:14:14 smithi007 ceph-mon[29459]: pgmap v553: 158 pgs: 4 active+undersized+degraded+remapped+backfilling, 3 active+remapped+backfilling, 25 active+clean+remapped, 22 active+undersized, 1 active+remapped+backfill_toofull, 27 active+undersized+degraded, 76 active+clean; 1.8 GiB data, 7.9 GiB used, 618 GiB / 626 GiB avail; 0 B/s wr, 452 op/s; 15160/87525 objects degraded (17.321%); 15367/87525 objects misplaced (17.557%); 19 MiB/s, 311 objects/s recovering 2024-09-18T08:14:14.830 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:14:14 smithi007 ceph-mon[29459]: Health check update: Degraded data redundancy: 15160/87525 objects degraded (17.321%), 31 pgs degraded, 49 pgs undersized (PG_DEGRADED) 2024-09-18T08:14:14.830 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:14:14 smithi007 ceph-mon[39249]: pgmap v553: 158 pgs: 4 active+undersized+degraded+remapped+backfilling, 3 active+remapped+backfilling, 25 active+clean+remapped, 22 active+undersized, 1 active+remapped+backfill_toofull, 27 active+undersized+degraded, 76 active+clean; 1.8 GiB data, 7.9 GiB used, 618 GiB / 626 GiB avail; 0 B/s wr, 452 op/s; 15160/87525 objects degraded (17.321%); 15367/87525 objects misplaced (17.557%); 19 MiB/s, 311 objects/s recovering 2024-09-18T08:14:14.830 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:14:14 smithi007 ceph-mon[39249]: Health check update: Degraded data redundancy: 15160/87525 objects degraded (17.321%), 31 pgs degraded, 49 pgs undersized (PG_DEGRADED) 2024-09-18T08:14:14.831 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:14:14.831 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:14:14.832 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5' 2024-09-18T08:14:14.891 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:14:14 smithi050 ceph-mon[32371]: pgmap v553: 158 pgs: 4 active+undersized+degraded+remapped+backfilling, 3 active+remapped+backfilling, 25 active+clean+remapped, 22 active+undersized, 1 active+remapped+backfill_toofull, 27 active+undersized+degraded, 76 active+clean; 1.8 GiB data, 7.9 GiB used, 618 GiB / 626 GiB avail; 0 B/s wr, 452 op/s; 15160/87525 objects degraded (17.321%); 15367/87525 objects misplaced (17.557%); 19 MiB/s, 311 objects/s recovering 2024-09-18T08:14:14.891 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:14:14 smithi050 ceph-mon[32371]: Health check update: Degraded data redundancy: 15160/87525 objects degraded (17.321%), 31 pgs degraded, 49 pgs undersized (PG_DEGRADED) 2024-09-18T08:14:14.933 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6 2024-09-18T08:14:14.966 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:14:14.966 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:14:14.967 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6' 2024-09-18T08:14:15.068 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7 2024-09-18T08:14:15.098 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:14:15.098 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:14:15.098 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7' 2024-09-18T08:14:15.199 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7 2024-09-18T08:14:15.227 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:14:15.227 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:14:15.228 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7' 2024-09-18T08:14:15.328 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7 2024-09-18T08:14:15.357 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:14:15.357 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:14:15.358 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7' 2024-09-18T08:14:15.460 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3 2024-09-18T08:14:15.493 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:14:15.494 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:14:15.495 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3' 2024-09-18T08:14:15.524 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid 779349ca-7593-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-18T08:14:15.596 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7 2024-09-18T08:14:15.622 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:14:15.622 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:14:15.623 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7' 2024-09-18T08:14:15.658 INFO:teuthology.orchestra.run.smithi007.stderr:nodeep-scrub is set 2024-09-18T08:14:15.724 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0 2024-09-18T08:14:15.761 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:14:15.761 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:14:15.762 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0' 2024-09-18T08:14:15.863 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6 2024-09-18T08:14:15.887 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:14:15.888 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:14:15.888 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6' 2024-09-18T08:14:15.905 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:14:15 smithi050 ceph-mon[32371]: from='client.? 172.21.15.7:0/2025457663' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-18T08:14:15.905 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:14:15 smithi050 ceph-mon[32371]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-18T08:14:15.980 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:14:15 smithi007 ceph-mon[29459]: from='client.? 172.21.15.7:0/2025457663' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-18T08:14:15.980 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:14:15 smithi007 ceph-mon[29459]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-18T08:14:15.980 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:14:15 smithi007 ceph-mon[39249]: from='client.? 172.21.15.7:0/2025457663' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-18T08:14:15.981 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:14:15 smithi007 ceph-mon[39249]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-18T08:14:15.990 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3 2024-09-18T08:14:16.016 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:14:16.016 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:14:16.017 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3' 2024-09-18T08:14:16.118 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4 2024-09-18T08:14:16.142 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:14:16.143 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:14:16.143 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4' 2024-09-18T08:14:16.244 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4 2024-09-18T08:14:16.268 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:14:16.269 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:14:16.269 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4' 2024-09-18T08:14:16.295 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-09-18T08:14:16.297 INFO:teuthology.orchestra.run.smithi007.stderr:dumped all 2024-09-18T08:14:16.370 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5 2024-09-18T08:14:16.392 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:14:16.392 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:14:16.393 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5' 2024-09-18T08:14:16.493 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7 2024-09-18T08:14:16.516 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:14:16.517 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:14:16.517 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7' 2024-09-18T08:14:16.618 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2 2024-09-18T08:14:16.643 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:14:16.644 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:14:16.644 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2' 2024-09-18T08:14:16.745 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5 2024-09-18T08:14:16.767 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:14:16.768 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:14:16.768 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5' 2024-09-18T08:14:16.869 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2 2024-09-18T08:14:16.884 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Sep 18 08:14:16 smithi007 ceph-779349ca-7593-11ef-bceb-c7b262605968-mgr-y[29688]: ::ffff:172.21.15.50 - - [18/Sep/2024:08:14:16] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-09-18T08:14:16.884 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:14:16 smithi007 ceph-mon[29459]: from='client.15894 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-18T08:14:16.885 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:14:16 smithi007 ceph-mon[29459]: pgmap v554: 158 pgs: 4 active+undersized+degraded+remapped+backfilling, 3 active+remapped+backfilling, 25 active+clean+remapped, 22 active+undersized, 1 active+remapped+backfill_toofull, 27 active+undersized+degraded, 76 active+clean; 1.4 GiB data, 7.5 GiB used, 618 GiB / 626 GiB avail; 233 B/s rd, 0 B/s wr, 1.07k op/s; 9554/70002 objects degraded (13.648%); 10096/70002 objects misplaced (14.422%); 54 MiB/s, 875 objects/s recovering 2024-09-18T08:14:16.885 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:14:16 smithi007 ceph-mon[29459]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-18T08:14:16.885 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:14:16 smithi007 ceph-mon[29459]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-18T08:14:16.885 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:14:16 smithi007 ceph-mon[29459]: osdmap e296: 8 total, 7 up, 7 in 2024-09-18T08:14:16.885 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:14:16 smithi007 ceph-mon[39249]: from='client.15894 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-18T08:14:16.885 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:14:16 smithi007 ceph-mon[39249]: pgmap v554: 158 pgs: 4 active+undersized+degraded+remapped+backfilling, 3 active+remapped+backfilling, 25 active+clean+remapped, 22 active+undersized, 1 active+remapped+backfill_toofull, 27 active+undersized+degraded, 76 active+clean; 1.4 GiB data, 7.5 GiB used, 618 GiB / 626 GiB avail; 233 B/s rd, 0 B/s wr, 1.07k op/s; 9554/70002 objects degraded (13.648%); 10096/70002 objects misplaced (14.422%); 54 MiB/s, 875 objects/s recovering 2024-09-18T08:14:16.888 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:14:16 smithi007 ceph-mon[39249]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-18T08:14:16.888 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:14:16 smithi007 ceph-mon[39249]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-18T08:14:16.888 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:14:16 smithi007 ceph-mon[39249]: osdmap e296: 8 total, 7 up, 7 in 2024-09-18T08:14:16.894 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:14:16.895 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:14:16.895 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2' 2024-09-18T08:14:16.905 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:14:16 smithi050 ceph-mon[32371]: from='client.15894 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-18T08:14:16.905 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:14:16 smithi050 ceph-mon[32371]: pgmap v554: 158 pgs: 4 active+undersized+degraded+remapped+backfilling, 3 active+remapped+backfilling, 25 active+clean+remapped, 22 active+undersized, 1 active+remapped+backfill_toofull, 27 active+undersized+degraded, 76 active+clean; 1.4 GiB data, 7.5 GiB used, 618 GiB / 626 GiB avail; 233 B/s rd, 0 B/s wr, 1.07k op/s; 9554/70002 objects degraded (13.648%); 10096/70002 objects misplaced (14.422%); 54 MiB/s, 875 objects/s recovering 2024-09-18T08:14:16.905 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:14:16 smithi050 ceph-mon[32371]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-18T08:14:16.906 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:14:16 smithi050 ceph-mon[32371]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-18T08:14:16.906 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:14:16 smithi050 ceph-mon[32371]: osdmap e296: 8 total, 7 up, 7 in 2024-09-18T08:14:16.996 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3 2024-09-18T08:14:17.021 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:14:17.021 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:14:17.022 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3' 2024-09-18T08:14:17.122 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2 2024-09-18T08:14:17.148 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:14:17.149 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:14:17.150 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2' 2024-09-18T08:14:17.250 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0 2024-09-18T08:14:17.277 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:14:17.278 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:14:17.279 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0' 2024-09-18T08:14:17.379 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2 2024-09-18T08:14:17.407 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:14:17.407 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:14:17.408 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2' 2024-09-18T08:14:17.508 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0 2024-09-18T08:14:17.534 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:14:17.535 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:14:17.535 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0' 2024-09-18T08:14:17.636 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3 2024-09-18T08:14:17.665 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:14:17.666 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:14:17.666 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3' 2024-09-18T08:14:17.767 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0 2024-09-18T08:14:17.795 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:14:17.796 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:14:17.796 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0' 2024-09-18T08:14:17.897 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7 2024-09-18T08:14:17.905 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:14:17 smithi050 ceph-mon[32371]: osdmap e297: 8 total, 7 up, 7 in 2024-09-18T08:14:17.905 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:14:17 smithi050 ceph-mon[32371]: osdmap e298: 8 total, 7 up, 7 in 2024-09-18T08:14:17.935 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:14:18.075 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:14:17 smithi007 ceph-mon[29459]: osdmap e297: 8 total, 7 up, 7 in 2024-09-18T08:14:18.075 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:14:17 smithi007 ceph-mon[29459]: osdmap e298: 8 total, 7 up, 7 in 2024-09-18T08:14:18.075 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:14:17 smithi007 ceph-mon[39249]: osdmap e297: 8 total, 7 up, 7 in 2024-09-18T08:14:18.075 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:14:17 smithi007 ceph-mon[39249]: osdmap e298: 8 total, 7 up, 7 in 2024-09-18T08:14:18.076 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:14:18.076 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7' 2024-09-18T08:14:18.177 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0 2024-09-18T08:14:18.218 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:14:18.218 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:14:18.219 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0' 2024-09-18T08:14:18.320 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7 2024-09-18T08:14:18.358 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:14:18.359 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:14:18.359 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7' 2024-09-18T08:14:18.460 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4 2024-09-18T08:14:18.495 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:14:18.496 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:14:18.496 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4' 2024-09-18T08:14:18.597 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0 2024-09-18T08:14:18.621 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:14:18.621 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:14:18.622 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0' 2024-09-18T08:14:18.725 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7 2024-09-18T08:14:18.765 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:14:18.766 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:14:18.766 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7' 2024-09-18T08:14:18.867 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0 2024-09-18T08:14:18.902 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:14:18.902 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:14:18.903 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0' 2024-09-18T08:14:18.905 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:14:18 smithi050 ceph-mon[32371]: pgmap v557: 158 pgs: 4 active+undersized+degraded+remapped+backfilling, 3 active+remapped+backfilling, 25 active+clean+remapped, 22 active+undersized, 1 active+remapped+backfill_toofull, 27 active+undersized+degraded, 76 active+clean; 1.4 GiB data, 7.5 GiB used, 618 GiB / 626 GiB avail; 639 B/s rd, 0 B/s wr, 730 op/s; 9554/70002 objects degraded (13.648%); 10096/70002 objects misplaced (14.422%); 41 MiB/s, 656 objects/s recovering 2024-09-18T08:14:18.905 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:14:18 smithi050 ceph-mon[32371]: from='mgr.24470 172.21.15.7:0/2594390261' entity='mgr.y' 2024-09-18T08:14:18.905 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:14:18 smithi050 ceph-mon[32371]: from='mgr.24470 172.21.15.7:0/2594390261' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-18T08:14:18.977 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:14:18 smithi007 ceph-mon[29459]: pgmap v557: 158 pgs: 4 active+undersized+degraded+remapped+backfilling, 3 active+remapped+backfilling, 25 active+clean+remapped, 22 active+undersized, 1 active+remapped+backfill_toofull, 27 active+undersized+degraded, 76 active+clean; 1.4 GiB data, 7.5 GiB used, 618 GiB / 626 GiB avail; 639 B/s rd, 0 B/s wr, 730 op/s; 9554/70002 objects degraded (13.648%); 10096/70002 objects misplaced (14.422%); 41 MiB/s, 656 objects/s recovering 2024-09-18T08:14:18.977 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:14:18 smithi007 ceph-mon[29459]: from='mgr.24470 172.21.15.7:0/2594390261' entity='mgr.y' 2024-09-18T08:14:18.977 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:14:18 smithi007 ceph-mon[29459]: from='mgr.24470 172.21.15.7:0/2594390261' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-18T08:14:18.977 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:14:18 smithi007 ceph-mon[39249]: pgmap v557: 158 pgs: 4 active+undersized+degraded+remapped+backfilling, 3 active+remapped+backfilling, 25 active+clean+remapped, 22 active+undersized, 1 active+remapped+backfill_toofull, 27 active+undersized+degraded, 76 active+clean; 1.4 GiB data, 7.5 GiB used, 618 GiB / 626 GiB avail; 639 B/s rd, 0 B/s wr, 730 op/s; 9554/70002 objects degraded (13.648%); 10096/70002 objects misplaced (14.422%); 41 MiB/s, 656 objects/s recovering 2024-09-18T08:14:18.978 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:14:18 smithi007 ceph-mon[39249]: from='mgr.24470 172.21.15.7:0/2594390261' entity='mgr.y' 2024-09-18T08:14:18.978 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:14:18 smithi007 ceph-mon[39249]: from='mgr.24470 172.21.15.7:0/2594390261' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-18T08:14:19.003 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0 2024-09-18T08:14:19.042 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:14:19.043 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:14:19.043 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0' 2024-09-18T08:14:19.144 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6 2024-09-18T08:14:19.177 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:14:19.178 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:14:19.179 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6' 2024-09-18T08:14:19.280 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2 2024-09-18T08:14:19.315 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:14:19.316 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:14:19.316 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2' 2024-09-18T08:14:19.417 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3 2024-09-18T08:14:19.454 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:14:19.454 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:14:19.454 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3' 2024-09-18T08:14:19.549 INFO:teuthology.orchestra.run.smithi007.stdout:{"pg_ready":true,"pg_map":{"version":553,"stamp":"2024-09-18T08:14:13.327813+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":1899555801,"num_objects":29175,"num_object_clones":0,"num_object_copies":87525,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":15160,"num_objects_misplaced":15367,"num_objects_unfound":0,"num_objects_dirty":29175,"num_whiteouts":0,"num_read":1290,"num_read_kb":1052,"num_write":255523,"num_write_kb":9080642,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2625,"num_bytes_recovered":156299418,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":70761,"ondisk_log_size":70761,"up":396,"acting":421,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":445,"num_osds":7,"num_per_pool_osds":7,"num_per_pool_omap_osds":7,"kb":656216064,"kb_used":8327232,"kb_used_data":5443204,"kb_used_omap":10,"kb_used_meta":2883701,"kb_avail":647888832,"statfs":{"total":671965249536,"available":663438163968,"internally_reserved":0,"allocated":5573840896,"data_stored":5569090020,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":11160,"internal_metadata":2952909928},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,5,4,6,1,0,0,0,1,1,0,5],"upper_bound":4096},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-230162432,"num_objects":-3512,"num_object_clones":0,"num_object_copies":-10536,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-3386,"num_objects_misplaced":15367,"num_objects_unfound":0,"num_objects_dirty":-3512,"num_whiteouts":0,"num_read":-3,"num_read_kb":-3,"num_write":3512,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2421,"num_bytes_recovered":156173692,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"7.763581"},"pg_stats":[{"pgid":"3.1f","version":"0'0","reported_seq":518,"reported_epoch":294,"state":"active+clean","last_fresh":"2024-09-18T08:14:06.578522+0000","last_change":"2024-09-18T08:10:15.850596+0000","last_active":"2024-09-18T08:14:06.578522+0000","last_peered":"2024-09-18T08:14:06.578522+0000","last_clean":"2024-09-18T08:14:06.578522+0000","last_became_active":"2024-09-18T08:08:52.697499+0000","last_became_peered":"2024-09-18T08:08:52.697499+0000","last_unstale":"2024-09-18T08:14:06.578522+0000","last_undegraded":"2024-09-18T08:14:06.578522+0000","last_fullsized":"2024-09-18T08:14:06.578522+0000","mapping_epoch":198,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":199,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:10:15.850515+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:46.860092+0000","last_clean_scrub_stamp":"2024-09-18T08:10:15.850515+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:14:24.842877+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"271'15","reported_seq":269,"reported_epoch":294,"state":"active+undersized+degraded","last_fresh":"2024-09-18T08:14:06.570915+0000","last_change":"2024-09-18T08:11:42.323251+0000","last_active":"2024-09-18T08:14:06.570915+0000","last_peered":"2024-09-18T08:14:06.570915+0000","last_clean":"2024-09-18T08:11:40.306776+0000","last_became_active":"2024-09-18T08:11:42.323251+0000","last_became_peered":"2024-09-18T08:11:42.323251+0000","last_unstale":"2024-09-18T08:14:06.570915+0000","last_undegraded":"2024-09-18T08:11:42.318308+0000","last_fullsized":"2024-09-18T08:11:42.318179+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":182,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'9","last_scrub_stamp":"2024-09-18T08:10:19.271241+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_clean_scrub_stamp":"2024-09-18T08:10:19.271241+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:14:17.601010+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":25,"num_read_kb":19,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":["4","6"],"object_location_counts":[{"shards":"4,6","objects":4}],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":115,"reported_epoch":294,"state":"active+clean+remapped","last_fresh":"2024-09-18T08:14:06.589333+0000","last_change":"2024-09-18T08:14:06.587052+0000","last_active":"2024-09-18T08:14:06.589333+0000","last_peered":"2024-09-18T08:14:06.589333+0000","last_clean":"2024-09-18T08:14:06.589333+0000","last_became_active":"2024-09-18T08:14:06.586588+0000","last_became_peered":"2024-09-18T08:14:06.586588+0000","last_unstale":"2024-09-18T08:14:06.589333+0000","last_undegraded":"2024-09-18T08:14:06.589333+0000","last_fullsized":"2024-09-18T08:14:06.589333+0000","mapping_epoch":293,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":294,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:10:15.603289+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:50.886830+0000","last_clean_scrub_stamp":"2024-09-18T08:10:15.603289+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2,7],"avail_no_missing":["5","2","7"],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":242,"reported_epoch":294,"state":"active+undersized","last_fresh":"2024-09-18T08:14:06.571829+0000","last_change":"2024-09-18T08:11:42.333493+0000","last_active":"2024-09-18T08:14:06.571829+0000","last_peered":"2024-09-18T08:14:06.571829+0000","last_clean":"2024-09-18T08:11:40.307445+0000","last_became_active":"2024-09-18T08:11:42.333493+0000","last_became_peered":"2024-09-18T08:11:42.333493+0000","last_unstale":"2024-09-18T08:14:06.571829+0000","last_undegraded":"2024-09-18T08:14:06.571829+0000","last_fullsized":"2024-09-18T08:11:42.319356+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":182,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:07:08.451865+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_clean_scrub_stamp":"2024-09-18T08:07:08.451865+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:14:53.706027+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":["4","5"],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":520,"reported_epoch":294,"state":"active+clean+remapped","last_fresh":"2024-09-18T08:14:06.599896+0000","last_change":"2024-09-18T08:14:06.599749+0000","last_active":"2024-09-18T08:14:06.599896+0000","last_peered":"2024-09-18T08:14:06.599896+0000","last_clean":"2024-09-18T08:14:06.599896+0000","last_became_active":"2024-09-18T08:14:06.599307+0000","last_became_peered":"2024-09-18T08:14:06.599307+0000","last_unstale":"2024-09-18T08:14:06.599896+0000","last_undegraded":"2024-09-18T08:14:06.599896+0000","last_fullsized":"2024-09-18T08:14:06.599896+0000","mapping_epoch":293,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":294,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:12:38.915501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_clean_scrub_stamp":"2024-09-18T08:12:38.915501+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6,7],"avail_no_missing":["3","6","7"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":535,"reported_epoch":294,"state":"active+clean","last_fresh":"2024-09-18T08:14:06.574341+0000","last_change":"2024-09-18T08:13:37.627262+0000","last_active":"2024-09-18T08:14:06.574341+0000","last_peered":"2024-09-18T08:14:06.574341+0000","last_clean":"2024-09-18T08:14:06.574341+0000","last_became_active":"2024-09-18T08:06:24.126194+0000","last_became_peered":"2024-09-18T08:06:24.126194+0000","last_unstale":"2024-09-18T08:14:06.574341+0000","last_undegraded":"2024-09-18T08:14:06.574341+0000","last_fullsized":"2024-09-18T08:14:06.574341+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:13:37.627195+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:46.860092+0000","last_clean_scrub_stamp":"2024-09-18T08:13:37.627195+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:14:53.988477+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"271'27","reported_seq":527,"reported_epoch":294,"state":"active+clean","last_fresh":"2024-09-18T08:14:06.578404+0000","last_change":"2024-09-18T08:09:32.575836+0000","last_active":"2024-09-18T08:14:06.578404+0000","last_peered":"2024-09-18T08:14:06.578404+0000","last_clean":"2024-09-18T08:14:06.578404+0000","last_became_active":"2024-09-18T08:02:49.886106+0000","last_became_peered":"2024-09-18T08:02:49.886106+0000","last_unstale":"2024-09-18T08:14:06.578404+0000","last_undegraded":"2024-09-18T08:14:06.578404+0000","last_fullsized":"2024-09-18T08:14:06.578404+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'15","last_scrub_stamp":"2024-09-18T08:09:32.575741+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_clean_scrub_stamp":"2024-09-18T08:09:32.575741+0000","objects_scrubbed":7,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:14:22.687956+0000","scrub_duration":40,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":44,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":242,"reported_epoch":294,"state":"active+undersized","last_fresh":"2024-09-18T08:14:06.571050+0000","last_change":"2024-09-18T08:11:42.323423+0000","last_active":"2024-09-18T08:14:06.571050+0000","last_peered":"2024-09-18T08:14:06.571050+0000","last_clean":"2024-09-18T08:11:40.306782+0000","last_became_active":"2024-09-18T08:11:42.323423+0000","last_became_peered":"2024-09-18T08:11:42.323423+0000","last_unstale":"2024-09-18T08:14:06.571050+0000","last_undegraded":"2024-09-18T08:14:06.571050+0000","last_fullsized":"2024-09-18T08:11:42.318286+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":182,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:08:09.142188+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:50.886830+0000","last_clean_scrub_stamp":"2024-09-18T08:08:09.142188+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":["4","6"],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":531,"reported_epoch":294,"state":"active+clean","last_fresh":"2024-09-18T08:14:06.583910+0000","last_change":"2024-09-18T08:10:51.226034+0000","last_active":"2024-09-18T08:14:06.583910+0000","last_peered":"2024-09-18T08:14:06.583910+0000","last_clean":"2024-09-18T08:14:06.583910+0000","last_became_active":"2024-09-18T08:08:18.548203+0000","last_became_peered":"2024-09-18T08:08:18.548203+0000","last_unstale":"2024-09-18T08:14:06.583910+0000","last_undegraded":"2024-09-18T08:14:06.583910+0000","last_fullsized":"2024-09-18T08:14:06.583910+0000","mapping_epoch":181,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":182,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:10:51.225974+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:46.860092+0000","last_clean_scrub_stamp":"2024-09-18T08:10:51.225974+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:14:14.183168+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"271'15","reported_seq":273,"reported_epoch":294,"state":"active+clean","last_fresh":"2024-09-18T08:14:06.573198+0000","last_change":"2024-09-18T08:12:00.466295+0000","last_active":"2024-09-18T08:14:06.573198+0000","last_peered":"2024-09-18T08:14:06.573198+0000","last_clean":"2024-09-18T08:14:06.573198+0000","last_became_active":"2024-09-18T08:08:18.550960+0000","last_became_peered":"2024-09-18T08:08:18.550960+0000","last_unstale":"2024-09-18T08:14:06.573198+0000","last_undegraded":"2024-09-18T08:14:06.573198+0000","last_fullsized":"2024-09-18T08:14:06.573198+0000","mapping_epoch":181,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":182,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'9","last_scrub_stamp":"2024-09-18T08:12:00.466202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_clean_scrub_stamp":"2024-09-18T08:12:00.466202+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":25,"num_read_kb":19,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":502,"reported_epoch":294,"state":"active+clean","last_fresh":"2024-09-18T08:14:06.579309+0000","last_change":"2024-09-18T08:14:05.597183+0000","last_active":"2024-09-18T08:14:06.579309+0000","last_peered":"2024-09-18T08:14:06.579309+0000","last_clean":"2024-09-18T08:14:06.579309+0000","last_became_active":"2024-09-18T08:14:05.594118+0000","last_became_peered":"2024-09-18T08:14:05.594118+0000","last_unstale":"2024-09-18T08:14:06.579309+0000","last_undegraded":"2024-09-18T08:14:06.579309+0000","last_fullsized":"2024-09-18T08:14:06.579309+0000","mapping_epoch":292,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":293,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:12:57.218286+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:50.886830+0000","last_clean_scrub_stamp":"2024-09-18T08:12:57.218286+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:14:16.682756+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,4],"acting":[5,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":494,"reported_epoch":294,"state":"active+clean+remapped","last_fresh":"2024-09-18T08:14:06.588898+0000","last_change":"2024-09-18T08:14:06.588755+0000","last_active":"2024-09-18T08:14:06.588898+0000","last_peered":"2024-09-18T08:14:06.588898+0000","last_clean":"2024-09-18T08:14:06.588898+0000","last_became_active":"2024-09-18T08:14:06.587687+0000","last_became_peered":"2024-09-18T08:14:06.587687+0000","last_unstale":"2024-09-18T08:14:06.588898+0000","last_undegraded":"2024-09-18T08:14:06.588898+0000","last_fullsized":"2024-09-18T08:14:06.588898+0000","mapping_epoch":293,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":294,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:08:14.645993+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_clean_scrub_stamp":"2024-09-18T08:08:14.645993+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2,7],"avail_no_missing":["0","2","7"],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":505,"reported_epoch":294,"state":"active+undersized","last_fresh":"2024-09-18T08:14:06.579393+0000","last_change":"2024-09-18T08:11:42.339943+0000","last_active":"2024-09-18T08:14:06.579393+0000","last_peered":"2024-09-18T08:14:06.579393+0000","last_clean":"2024-09-18T08:11:40.302512+0000","last_became_active":"2024-09-18T08:11:42.339943+0000","last_became_peered":"2024-09-18T08:11:42.339943+0000","last_unstale":"2024-09-18T08:14:06.579393+0000","last_undegraded":"2024-09-18T08:14:06.579393+0000","last_fullsized":"2024-09-18T08:11:42.322341+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":182,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:04:32.849119+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:46.860092+0000","last_clean_scrub_stamp":"2024-09-18T08:04:32.849119+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:14:49.114040+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":["5","4"],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1b","version":"271'7","reported_seq":259,"reported_epoch":294,"state":"active+undersized+degraded","last_fresh":"2024-09-18T08:14:06.572402+0000","last_change":"2024-09-18T08:11:42.329404+0000","last_active":"2024-09-18T08:14:06.572402+0000","last_peered":"2024-09-18T08:14:06.572402+0000","last_clean":"2024-09-18T08:11:40.307173+0000","last_became_active":"2024-09-18T08:11:42.329404+0000","last_became_peered":"2024-09-18T08:11:42.329404+0000","last_unstale":"2024-09-18T08:14:06.572402+0000","last_undegraded":"2024-09-18T08:11:42.319890+0000","last_fullsized":"2024-09-18T08:11:42.319791+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":182,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'5","last_scrub_stamp":"2024-09-18T08:10:52.400411+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_clean_scrub_stamp":"2024-09-18T08:10:52.400411+0000","objects_scrubbed":2,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":13,"num_read_kb":9,"num_write":8,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":["4","3"],"object_location_counts":[{"shards":"3,4","objects":2}],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":445,"reported_epoch":294,"state":"active+clean+remapped","last_fresh":"2024-09-18T08:14:06.586145+0000","last_change":"2024-09-18T08:14:06.585465+0000","last_active":"2024-09-18T08:14:06.586145+0000","last_peered":"2024-09-18T08:14:06.586145+0000","last_clean":"2024-09-18T08:14:06.586145+0000","last_became_active":"2024-09-18T08:14:06.583168+0000","last_became_peered":"2024-09-18T08:14:06.583168+0000","last_unstale":"2024-09-18T08:14:06.586145+0000","last_undegraded":"2024-09-18T08:14:06.586145+0000","last_fullsized":"2024-09-18T08:14:06.586145+0000","mapping_epoch":293,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":294,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:11:30.837341+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:50.886830+0000","last_clean_scrub_stamp":"2024-09-18T08:11:30.837341+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0,7],"avail_no_missing":["4","0","7"],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":496,"reported_epoch":294,"state":"active+undersized","last_fresh":"2024-09-18T08:14:06.578295+0000","last_change":"2024-09-18T08:11:42.332383+0000","last_active":"2024-09-18T08:14:06.578295+0000","last_peered":"2024-09-18T08:14:06.578295+0000","last_clean":"2024-09-18T08:11:40.302158+0000","last_became_active":"2024-09-18T08:11:42.332383+0000","last_became_peered":"2024-09-18T08:11:42.332383+0000","last_unstale":"2024-09-18T08:14:06.578295+0000","last_undegraded":"2024-09-18T08:14:06.578295+0000","last_fullsized":"2024-09-18T08:11:42.326605+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:11:01.108648+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_clean_scrub_stamp":"2024-09-18T08:11:01.108648+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:14:30.163932+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":["5","3"],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":246,"reported_epoch":294,"state":"active+clean","last_fresh":"2024-09-18T08:14:06.576077+0000","last_change":"2024-09-18T08:11:34.625641+0000","last_active":"2024-09-18T08:14:06.576077+0000","last_peered":"2024-09-18T08:14:06.576077+0000","last_clean":"2024-09-18T08:14:06.576077+0000","last_became_active":"2024-09-18T08:08:18.539604+0000","last_became_peered":"2024-09-18T08:08:18.539604+0000","last_unstale":"2024-09-18T08:14:06.576077+0000","last_undegraded":"2024-09-18T08:14:06.576077+0000","last_fullsized":"2024-09-18T08:14:06.576077+0000","mapping_epoch":181,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":182,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:11:34.625572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_clean_scrub_stamp":"2024-09-18T08:11:34.625572+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":523,"reported_epoch":294,"state":"active+clean+remapped","last_fresh":"2024-09-18T08:14:06.591459+0000","last_change":"2024-09-18T08:14:06.591177+0000","last_active":"2024-09-18T08:14:06.591459+0000","last_peered":"2024-09-18T08:14:06.591459+0000","last_clean":"2024-09-18T08:14:06.591459+0000","last_became_active":"2024-09-18T08:14:06.589757+0000","last_became_peered":"2024-09-18T08:14:06.589757+0000","last_unstale":"2024-09-18T08:14:06.591459+0000","last_undegraded":"2024-09-18T08:14:06.591459+0000","last_fullsized":"2024-09-18T08:14:06.591459+0000","mapping_epoch":293,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":294,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:12:40.711507+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:46.860092+0000","last_clean_scrub_stamp":"2024-09-18T08:12:40.711507+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4,7],"avail_no_missing":["0","4","7"],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1c","version":"271'27","reported_seq":242,"reported_epoch":294,"state":"active+undersized+degraded","last_fresh":"2024-09-18T08:14:06.573299+0000","last_change":"2024-09-18T08:11:42.329080+0000","last_active":"2024-09-18T08:14:06.573299+0000","last_peered":"2024-09-18T08:14:06.573299+0000","last_clean":"2024-09-18T08:11:40.306909+0000","last_became_active":"2024-09-18T08:11:42.329080+0000","last_became_peered":"2024-09-18T08:11:42.329080+0000","last_unstale":"2024-09-18T08:14:06.573299+0000","last_undegraded":"2024-09-18T08:11:42.319732+0000","last_fullsized":"2024-09-18T08:11:42.319638+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":199,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'15","last_scrub_stamp":"2024-09-18T08:07:39.530340+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_clean_scrub_stamp":"2024-09-18T08:07:39.530340+0000","objects_scrubbed":7,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:14:53.523616+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":7,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":44,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":7}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":381,"reported_epoch":294,"state":"active+undersized","last_fresh":"2024-09-18T08:14:06.572318+0000","last_change":"2024-09-18T08:11:42.330489+0000","last_active":"2024-09-18T08:14:06.572318+0000","last_peered":"2024-09-18T08:14:06.572318+0000","last_clean":"2024-09-18T08:08:15.505698+0000","last_became_active":"2024-09-18T08:11:42.330489+0000","last_became_peered":"2024-09-18T08:11:42.330489+0000","last_unstale":"2024-09-18T08:14:06.572318+0000","last_undegraded":"2024-09-18T08:14:06.572318+0000","last_fullsized":"2024-09-18T08:11:42.318950+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":182,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:10:59.645112+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:50.886830+0000","last_clean_scrub_stamp":"2024-09-18T08:10:59.645112+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":["4","0"],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":499,"reported_epoch":294,"state":"active+clean","last_fresh":"2024-09-18T08:14:06.579280+0000","last_change":"2024-09-18T08:09:36.592348+0000","last_active":"2024-09-18T08:14:06.579280+0000","last_peered":"2024-09-18T08:14:06.579280+0000","last_clean":"2024-09-18T08:14:06.579280+0000","last_became_active":"2024-09-18T08:06:24.128659+0000","last_became_peered":"2024-09-18T08:06:24.128659+0000","last_unstale":"2024-09-18T08:14:06.579280+0000","last_undegraded":"2024-09-18T08:14:06.579280+0000","last_fullsized":"2024-09-18T08:14:06.579280+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:09:36.592281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_clean_scrub_stamp":"2024-09-18T08:09:36.592281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:14:21.721636+0000","scrub_duration":16,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":242,"reported_epoch":294,"state":"active+undersized","last_fresh":"2024-09-18T08:14:06.571624+0000","last_change":"2024-09-18T08:11:42.335948+0000","last_active":"2024-09-18T08:14:06.571624+0000","last_peered":"2024-09-18T08:14:06.571624+0000","last_clean":"2024-09-18T08:11:40.306989+0000","last_became_active":"2024-09-18T08:11:42.335948+0000","last_became_peered":"2024-09-18T08:11:42.335948+0000","last_unstale":"2024-09-18T08:14:06.571624+0000","last_undegraded":"2024-09-18T08:14:06.571624+0000","last_fullsized":"2024-09-18T08:11:42.319106+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":182,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:07:13.543722+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:46.860092+0000","last_clean_scrub_stamp":"2024-09-18T08:07:13.543722+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":["4","2"],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"271'18","reported_seq":582,"reported_epoch":294,"state":"active+undersized+degraded","last_fresh":"2024-09-18T08:14:06.575610+0000","last_change":"2024-09-18T08:11:42.329686+0000","last_active":"2024-09-18T08:14:06.575610+0000","last_peered":"2024-09-18T08:14:06.575610+0000","last_clean":"2024-09-18T08:11:40.313323+0000","last_became_active":"2024-09-18T08:11:42.329686+0000","last_became_peered":"2024-09-18T08:11:42.329686+0000","last_unstale":"2024-09-18T08:14:06.575610+0000","last_undegraded":"2024-09-18T08:11:42.324380+0000","last_fullsized":"2024-09-18T08:11:42.324261+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":182,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'12","last_scrub_stamp":"2024-09-18T08:11:32.304472+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_clean_scrub_stamp":"2024-09-18T08:11:32.304472+0000","objects_scrubbed":5,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:14:27.707068+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":5,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":34,"num_read_kb":25,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":["3","4"],"object_location_counts":[{"shards":"3,4","objects":5}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":246,"reported_epoch":294,"state":"active+clean","last_fresh":"2024-09-18T08:14:06.573676+0000","last_change":"2024-09-18T08:10:46.303912+0000","last_active":"2024-09-18T08:14:06.573676+0000","last_peered":"2024-09-18T08:14:06.573676+0000","last_clean":"2024-09-18T08:14:06.573676+0000","last_became_active":"2024-09-18T08:08:18.531438+0000","last_became_peered":"2024-09-18T08:08:18.531438+0000","last_unstale":"2024-09-18T08:14:06.573676+0000","last_undegraded":"2024-09-18T08:14:06.573676+0000","last_fullsized":"2024-09-18T08:14:06.573676+0000","mapping_epoch":181,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":182,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:10:46.303848+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:50.886830+0000","last_clean_scrub_stamp":"2024-09-18T08:10:46.303848+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:14:19.606868+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1c","version":"69'1","reported_seq":345,"reported_epoch":294,"state":"active+clean","last_fresh":"2024-09-18T08:14:06.579017+0000","last_change":"2024-09-18T08:14:05.604359+0000","last_active":"2024-09-18T08:14:06.579017+0000","last_peered":"2024-09-18T08:14:06.579017+0000","last_clean":"2024-09-18T08:14:06.579017+0000","last_became_active":"2024-09-18T08:14:05.593353+0000","last_became_peered":"2024-09-18T08:14:05.593353+0000","last_unstale":"2024-09-18T08:14:06.579017+0000","last_undegraded":"2024-09-18T08:14:06.579017+0000","last_fullsized":"2024-09-18T08:14:06.579017+0000","mapping_epoch":292,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":293,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'1","last_scrub_stamp":"2024-09-18T08:10:46.212626+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_clean_scrub_stamp":"2024-09-18T08:10:46.212626+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":403,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":403,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,3],"acting":[5,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"62'1","reported_seq":119,"reported_epoch":294,"state":"active+undersized+degraded","last_fresh":"2024-09-18T08:14:06.578127+0000","last_change":"2024-09-18T08:11:42.335870+0000","last_active":"2024-09-18T08:14:06.578127+0000","last_peered":"2024-09-18T08:14:06.578127+0000","last_clean":"2024-09-18T08:02:47.882026+0000","last_became_active":"2024-09-18T08:11:42.335870+0000","last_became_peered":"2024-09-18T08:11:42.335870+0000","last_unstale":"2024-09-18T08:14:06.578127+0000","last_undegraded":"2024-09-18T08:11:42.323433+0000","last_fullsized":"2024-09-18T08:11:42.323288+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":182,"parent":"0.0","parent_split_bits":0,"last_scrub":"62'1","last_scrub_stamp":"2024-09-18T08:11:00.634190+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:46.860092+0000","last_clean_scrub_stamp":"2024-09-18T08:11:00.634190+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":["3","4"],"object_location_counts":[{"shards":"3,4","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"271'20","reported_seq":528,"reported_epoch":294,"state":"active+clean+remapped","last_fresh":"2024-09-18T08:14:06.583564+0000","last_change":"2024-09-18T08:14:06.583408+0000","last_active":"2024-09-18T08:14:06.583564+0000","last_peered":"2024-09-18T08:14:06.583564+0000","last_clean":"2024-09-18T08:14:06.583564+0000","last_became_active":"2024-09-18T08:14:06.581032+0000","last_became_peered":"2024-09-18T08:14:06.581032+0000","last_unstale":"2024-09-18T08:14:06.583564+0000","last_undegraded":"2024-09-18T08:14:06.583564+0000","last_fullsized":"2024-09-18T08:14:06.583564+0000","mapping_epoch":293,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":294,"parent":"0.0","parent_split_bits":0,"last_scrub":"271'20","last_scrub_stamp":"2024-09-18T08:13:30.833726+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_clean_scrub_stamp":"2024-09-18T08:13:30.833726+0000","objects_scrubbed":5,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:14:56.429751+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":5,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":25,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3,7],"avail_no_missing":["0","3","7"],"object_location_counts":[{"shards":"0,3,7","objects":5}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1f","version":"293'14","reported_seq":333,"reported_epoch":294,"state":"active+clean","last_fresh":"2024-09-18T08:14:06.572878+0000","last_change":"2024-09-18T08:14:05.592694+0000","last_active":"2024-09-18T08:14:06.572878+0000","last_peered":"2024-09-18T08:14:06.572878+0000","last_clean":"2024-09-18T08:14:06.572878+0000","last_became_active":"2024-09-18T08:14:05.581783+0000","last_became_peered":"2024-09-18T08:14:05.581783+0000","last_unstale":"2024-09-18T08:14:06.572878+0000","last_undegraded":"2024-09-18T08:14:06.572878+0000","last_fullsized":"2024-09-18T08:14:06.572878+0000","mapping_epoch":292,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":293,"parent":"0.0","parent_split_bits":0,"last_scrub":"182'13","last_scrub_stamp":"2024-09-18T08:12:59.403054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:50.886830+0000","last_clean_scrub_stamp":"2024-09-18T08:12:59.403054+0000","objects_scrubbed":1,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:14:15.119686+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,2],"acting":[4,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.2","version":"294'15998","reported_seq":16136,"reported_epoch":294,"state":"active+undersized+degraded","last_fresh":"2024-09-18T08:14:09.083348+0000","last_change":"2024-09-18T08:11:42.343258+0000","last_active":"2024-09-18T08:14:09.083348+0000","last_peered":"2024-09-18T08:14:09.083348+0000","last_clean":"2024-09-18T08:11:41.306257+0000","last_became_active":"2024-09-18T08:11:42.343258+0000","last_became_peered":"2024-09-18T08:11:42.343258+0000","last_unstale":"2024-09-18T08:14:09.083348+0000","last_undegraded":"2024-09-18T08:11:42.323950+0000","last_fullsized":"2024-09-18T08:11:42.323830+0000","mapping_epoch":244,"log_start":"279'12080","ondisk_log_start":"279'12080","created":232,"last_epoch_clean":233,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:11:08.262286+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:11:08.262286+0000","last_clean_scrub_stamp":"2024-09-18T08:11:08.262286+0000","objects_scrubbed":0,"log_size":3918,"log_dups_size":0,"ondisk_log_size":3918,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":123469824,"num_objects":1884,"num_object_clones":0,"num_object_copies":5652,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1884,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1884,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15998,"num_write_kb":572224,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["3","2"],"object_location_counts":[{"shards":"2,3","objects":1884}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"74'79","reported_seq":318,"reported_epoch":294,"state":"active+clean","last_fresh":"2024-09-18T08:14:06.578687+0000","last_change":"2024-09-18T08:11:29.068419+0000","last_active":"2024-09-18T08:14:06.578687+0000","last_peered":"2024-09-18T08:14:06.578687+0000","last_clean":"2024-09-18T08:14:06.578687+0000","last_became_active":"2024-09-18T08:07:26.394584+0000","last_became_peered":"2024-09-18T08:07:26.394584+0000","last_unstale":"2024-09-18T08:14:06.578687+0000","last_undegraded":"2024-09-18T08:14:06.578687+0000","last_fullsized":"2024-09-18T08:14:06.578687+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":152,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":4,"last_scrub":"74'79","last_scrub_stamp":"2024-09-18T08:11:29.068355+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:00:11.893742+0000","last_clean_scrub_stamp":"2024-09-18T08:11:29.068355+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,6],"acting":[5,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.f","version":"271'27","reported_seq":172,"reported_epoch":294,"state":"active+undersized+degraded","last_fresh":"2024-09-18T08:14:06.581996+0000","last_change":"2024-09-18T08:11:42.336910+0000","last_active":"2024-09-18T08:14:06.581996+0000","last_peered":"2024-09-18T08:14:06.581996+0000","last_clean":"2024-09-18T08:02:54.411508+0000","last_became_active":"2024-09-18T08:11:42.336910+0000","last_became_peered":"2024-09-18T08:11:42.336910+0000","last_unstale":"2024-09-18T08:14:06.581996+0000","last_undegraded":"2024-09-18T08:11:42.326875+0000","last_fullsized":"2024-09-18T08:11:42.326758+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":182,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'15","last_scrub_stamp":"2024-09-18T08:10:41.548188+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_clean_scrub_stamp":"2024-09-18T08:10:41.548188+0000","objects_scrubbed":0,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:14:45.535046+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":7,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":43,"num_read_kb":34,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":["3","4"],"object_location_counts":[{"shards":"3,4","objects":7}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":507,"reported_epoch":294,"state":"active+clean+remapped","last_fresh":"2024-09-18T08:14:06.585107+0000","last_change":"2024-09-18T08:14:06.584971+0000","last_active":"2024-09-18T08:14:06.585107+0000","last_peered":"2024-09-18T08:14:06.585107+0000","last_clean":"2024-09-18T08:14:06.585107+0000","last_became_active":"2024-09-18T08:14:06.584447+0000","last_became_peered":"2024-09-18T08:14:06.584447+0000","last_unstale":"2024-09-18T08:14:06.585107+0000","last_undegraded":"2024-09-18T08:14:06.585107+0000","last_fullsized":"2024-09-18T08:14:06.585107+0000","mapping_epoch":293,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":294,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:08:18.389273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:46.860092+0000","last_clean_scrub_stamp":"2024-09-18T08:08:18.389273+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4,7],"avail_no_missing":["3","4","7"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.e","version":"182'12","reported_seq":329,"reported_epoch":294,"state":"active+clean","last_fresh":"2024-09-18T08:14:06.575789+0000","last_change":"2024-09-18T08:13:31.620633+0000","last_active":"2024-09-18T08:14:06.575789+0000","last_peered":"2024-09-18T08:14:06.575789+0000","last_clean":"2024-09-18T08:14:06.575789+0000","last_became_active":"2024-09-18T08:08:18.525147+0000","last_became_peered":"2024-09-18T08:08:18.525147+0000","last_unstale":"2024-09-18T08:14:06.575789+0000","last_undegraded":"2024-09-18T08:14:06.575789+0000","last_fullsized":"2024-09-18T08:14:06.575789+0000","mapping_epoch":181,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":182,"parent":"0.0","parent_split_bits":0,"last_scrub":"182'12","last_scrub_stamp":"2024-09-18T08:13:31.620205+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:50.886830+0000","last_clean_scrub_stamp":"2024-09-18T08:13:31.620205+0000","objects_scrubbed":1,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:14:56.820672+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":496,"reported_epoch":294,"state":"active+undersized","last_fresh":"2024-09-18T08:14:06.581379+0000","last_change":"2024-09-18T08:11:42.330265+0000","last_active":"2024-09-18T08:14:06.581379+0000","last_peered":"2024-09-18T08:14:06.581379+0000","last_clean":"2024-09-18T08:11:40.302096+0000","last_became_active":"2024-09-18T08:11:42.330265+0000","last_became_peered":"2024-09-18T08:11:42.330265+0000","last_unstale":"2024-09-18T08:14:06.581379+0000","last_undegraded":"2024-09-18T08:14:06.581379+0000","last_fullsized":"2024-09-18T08:11:42.324373+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:07:19.714430+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_clean_scrub_stamp":"2024-09-18T08:07:19.714430+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":["5","0"],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.d","version":"294'16069","reported_seq":17120,"reported_epoch":294,"state":"active+undersized+degraded+remapped+backfilling","last_fresh":"2024-09-18T08:14:08.595280+0000","last_change":"2024-09-18T08:14:06.738199+0000","last_active":"2024-09-18T08:14:08.595280+0000","last_peered":"2024-09-18T08:14:08.595280+0000","last_clean":"2024-09-18T08:11:41.317368+0000","last_became_active":"2024-09-18T08:14:06.692114+0000","last_became_peered":"2024-09-18T08:14:06.692114+0000","last_unstale":"2024-09-18T08:14:08.595280+0000","last_undegraded":"2024-09-18T08:14:06.611557+0000","last_fullsized":"2024-09-18T08:14:06.582329+0000","mapping_epoch":293,"log_start":"278'11698","ondisk_log_start":"278'11698","created":232,"last_epoch_clean":233,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:11:08.262286+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:11:08.262286+0000","last_clean_scrub_stamp":"2024-09-18T08:11:08.262286+0000","objects_scrubbed":0,"log_size":4371,"log_dups_size":0,"ondisk_log_size":4371,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":120455192,"num_objects":1839,"num_object_clones":0,"num_object_copies":5517,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1412,"num_objects_misplaced":1839,"num_objects_unfound":0,"num_objects_dirty":1839,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":16069,"num_write_kb":572993,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":451,"num_bytes_recovered":29556736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,7],"avail_no_missing":["5","7"],"object_location_counts":[{"shards":"5,7","objects":1839}],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.5","version":"74'79","reported_seq":262,"reported_epoch":294,"state":"active+clean","last_fresh":"2024-09-18T08:14:06.572793+0000","last_change":"2024-09-18T08:11:54.503697+0000","last_active":"2024-09-18T08:14:06.572793+0000","last_peered":"2024-09-18T08:14:06.572793+0000","last_clean":"2024-09-18T08:14:06.572793+0000","last_became_active":"2024-09-18T08:08:52.690344+0000","last_became_peered":"2024-09-18T08:08:52.690344+0000","last_unstale":"2024-09-18T08:14:06.572793+0000","last_undegraded":"2024-09-18T08:14:06.572793+0000","last_fullsized":"2024-09-18T08:14:06.572793+0000","mapping_epoch":198,"log_start":"0'0","ondisk_log_start":"0'0","created":152,"last_epoch_clean":199,"parent":"0.0","parent_split_bits":3,"last_scrub":"74'79","last_scrub_stamp":"2024-09-18T08:11:54.503603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:00:11.893742+0000","last_clean_scrub_stamp":"2024-09-18T08:11:54.503603+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,6],"acting":[4,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.0","version":"271'30","reported_seq":565,"reported_epoch":294,"state":"active+clean+remapped","last_fresh":"2024-09-18T08:14:06.590109+0000","last_change":"2024-09-18T08:14:06.589953+0000","last_active":"2024-09-18T08:14:06.590109+0000","last_peered":"2024-09-18T08:14:06.590109+0000","last_clean":"2024-09-18T08:14:06.590109+0000","last_became_active":"2024-09-18T08:14:06.589484+0000","last_became_peered":"2024-09-18T08:14:06.589484+0000","last_unstale":"2024-09-18T08:14:06.590109+0000","last_undegraded":"2024-09-18T08:14:06.590109+0000","last_fullsized":"2024-09-18T08:14:06.590109+0000","mapping_epoch":293,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":294,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'18","last_scrub_stamp":"2024-09-18T08:12:36.956258+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_clean_scrub_stamp":"2024-09-18T08:12:36.956258+0000","objects_scrubbed":8,"log_size":30,"log_dups_size":0,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":8,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":52,"num_read_kb":40,"num_write":32,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0,7],"avail_no_missing":["3","0","7"],"object_location_counts":[{"shards":"0,3,7","objects":8}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":528,"reported_epoch":294,"state":"active+clean","last_fresh":"2024-09-18T08:14:06.584797+0000","last_change":"2024-09-18T08:14:05.595200+0000","last_active":"2024-09-18T08:14:06.584797+0000","last_peered":"2024-09-18T08:14:06.584797+0000","last_clean":"2024-09-18T08:14:06.584797+0000","last_became_active":"2024-09-18T08:14:05.594825+0000","last_became_peered":"2024-09-18T08:14:05.594825+0000","last_unstale":"2024-09-18T08:14:06.584797+0000","last_undegraded":"2024-09-18T08:14:06.584797+0000","last_fullsized":"2024-09-18T08:14:06.584797+0000","mapping_epoch":292,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":293,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:11:20.379467+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:46.860092+0000","last_clean_scrub_stamp":"2024-09-18T08:11:20.379467+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":261,"reported_epoch":294,"state":"active+clean","last_fresh":"2024-09-18T08:14:06.571838+0000","last_change":"2024-09-18T08:14:05.588953+0000","last_active":"2024-09-18T08:14:06.571838+0000","last_peered":"2024-09-18T08:14:06.571838+0000","last_clean":"2024-09-18T08:14:06.571838+0000","last_became_active":"2024-09-18T08:14:05.588085+0000","last_became_peered":"2024-09-18T08:14:05.588085+0000","last_unstale":"2024-09-18T08:14:06.571838+0000","last_undegraded":"2024-09-18T08:14:06.571838+0000","last_fullsized":"2024-09-18T08:14:06.571838+0000","mapping_epoch":292,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":293,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:12:30.732539+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:50.886830+0000","last_clean_scrub_stamp":"2024-09-18T08:12:30.732539+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":254,"reported_epoch":294,"state":"active+clean","last_fresh":"2024-09-18T08:14:06.572483+0000","last_change":"2024-09-18T08:12:02.473229+0000","last_active":"2024-09-18T08:14:06.572483+0000","last_peered":"2024-09-18T08:14:06.572483+0000","last_clean":"2024-09-18T08:14:06.572483+0000","last_became_active":"2024-09-18T08:08:18.543694+0000","last_became_peered":"2024-09-18T08:08:18.543694+0000","last_unstale":"2024-09-18T08:14:06.572483+0000","last_undegraded":"2024-09-18T08:14:06.572483+0000","last_fullsized":"2024-09-18T08:14:06.572483+0000","mapping_epoch":181,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":182,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:12:02.473168+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_clean_scrub_stamp":"2024-09-18T08:12:02.473168+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.c","version":"294'15910","reported_seq":16715,"reported_epoch":294,"state":"active+remapped+backfilling","last_fresh":"2024-09-18T08:14:09.091855+0000","last_change":"2024-09-18T08:14:07.047488+0000","last_active":"2024-09-18T08:14:09.091855+0000","last_peered":"2024-09-18T08:14:09.091855+0000","last_clean":"2024-09-18T08:14:04.559906+0000","last_became_active":"2024-09-18T08:14:06.682077+0000","last_became_peered":"2024-09-18T08:14:06.682077+0000","last_unstale":"2024-09-18T08:14:09.091855+0000","last_undegraded":"2024-09-18T08:14:09.091855+0000","last_fullsized":"2024-09-18T08:14:09.091855+0000","mapping_epoch":293,"log_start":"279'11988","ondisk_log_start":"279'11988","created":232,"last_epoch_clean":233,"parent":"0.0","parent_split_bits":0,"last_scrub":"274'10686","last_scrub_stamp":"2024-09-18T08:13:06.086640+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:11:08.262286+0000","last_clean_scrub_stamp":"2024-09-18T08:13:06.086640+0000","objects_scrubbed":7100,"log_size":3922,"log_dups_size":0,"ondisk_log_size":3922,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"no scrub is scheduled","scrub_duration":2162,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117309440,"num_objects":1790,"num_object_clones":0,"num_object_copies":5370,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1508,"num_objects_unfound":0,"num_objects_dirty":1790,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15910,"num_write_kb":566400,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":292,"num_bytes_recovered":19136512,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,0],"acting":[3,4,7],"avail_no_missing":["3","4","7"],"object_location_counts":[{"shards":"3,4,7","objects":1790}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.4","version":"74'79","reported_seq":343,"reported_epoch":294,"state":"active+clean","last_fresh":"2024-09-18T08:14:06.573698+0000","last_change":"2024-09-18T08:14:05.602732+0000","last_active":"2024-09-18T08:14:06.573698+0000","last_peered":"2024-09-18T08:14:06.573698+0000","last_clean":"2024-09-18T08:14:06.573698+0000","last_became_active":"2024-09-18T08:14:05.594796+0000","last_became_peered":"2024-09-18T08:14:05.594796+0000","last_unstale":"2024-09-18T08:14:06.573698+0000","last_undegraded":"2024-09-18T08:14:06.573698+0000","last_fullsized":"2024-09-18T08:14:06.573698+0000","mapping_epoch":292,"log_start":"0'0","ondisk_log_start":"0'0","created":152,"last_epoch_clean":293,"parent":"0.0","parent_split_bits":3,"last_scrub":"74'79","last_scrub_stamp":"2024-09-18T08:12:37.920342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:00:11.893742+0000","last_clean_scrub_stamp":"2024-09-18T08:12:37.920342+0000","objects_scrubbed":1,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":327680,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":327680,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"271'28","reported_seq":283,"reported_epoch":294,"state":"active+clean","last_fresh":"2024-09-18T08:14:06.571545+0000","last_change":"2024-09-18T08:11:53.524126+0000","last_active":"2024-09-18T08:14:06.571545+0000","last_peered":"2024-09-18T08:14:06.571545+0000","last_clean":"2024-09-18T08:14:06.571545+0000","last_became_active":"2024-09-18T08:08:18.549671+0000","last_became_peered":"2024-09-18T08:08:18.549671+0000","last_unstale":"2024-09-18T08:14:06.571545+0000","last_undegraded":"2024-09-18T08:14:06.571545+0000","last_fullsized":"2024-09-18T08:14:06.571545+0000","mapping_epoch":181,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":182,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'14","last_scrub_stamp":"2024-09-18T08:11:53.524045+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_clean_scrub_stamp":"2024-09-18T08:11:53.524045+0000","objects_scrubbed":7,"log_size":28,"log_dups_size":0,"ondisk_log_size":28,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":41,"num_read_kb":34,"num_write":28,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"62'2","reported_seq":533,"reported_epoch":294,"state":"active+undersized+degraded","last_fresh":"2024-09-18T08:14:06.579967+0000","last_change":"2024-09-18T08:11:42.338771+0000","last_active":"2024-09-18T08:14:06.579967+0000","last_peered":"2024-09-18T08:14:06.579967+0000","last_clean":"2024-09-18T08:11:40.307484+0000","last_became_active":"2024-09-18T08:11:42.338771+0000","last_became_peered":"2024-09-18T08:11:42.338771+0000","last_unstale":"2024-09-18T08:14:06.579967+0000","last_undegraded":"2024-09-18T08:11:42.320507+0000","last_fullsized":"2024-09-18T08:11:42.320405+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":182,"parent":"0.0","parent_split_bits":0,"last_scrub":"62'2","last_scrub_stamp":"2024-09-18T08:06:43.607152+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:46.860092+0000","last_clean_scrub_stamp":"2024-09-18T08:06:43.607152+0000","objects_scrubbed":2,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:14:48.752332+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1039,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":2078,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":["0","4"],"object_location_counts":[{"shards":"0,4","objects":2}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.0","version":"245'11","reported_seq":661,"reported_epoch":294,"state":"active+undersized+degraded","last_fresh":"2024-09-18T08:14:06.570885+0000","last_change":"2024-09-18T08:11:42.339963+0000","last_active":"2024-09-18T08:14:06.570885+0000","last_peered":"2024-09-18T08:14:06.570885+0000","last_clean":"2024-09-18T08:11:40.313426+0000","last_became_active":"2024-09-18T08:11:42.339963+0000","last_became_peered":"2024-09-18T08:11:42.339963+0000","last_unstale":"2024-09-18T08:14:06.570885+0000","last_undegraded":"2024-09-18T08:11:42.326291+0000","last_fullsized":"2024-09-18T08:11:42.326182+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":182,"parent":"0.0","parent_split_bits":0,"last_scrub":"182'10","last_scrub_stamp":"2024-09-18T08:09:51.467349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:50.886830+0000","last_clean_scrub_stamp":"2024-09-18T08:09:51.467349+0000","objects_scrubbed":1,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:14:58.502525+0000","scrub_duration":19,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":["3","4"],"object_location_counts":[{"shards":"3,4","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":46,"reported_epoch":294,"state":"active+clean","last_fresh":"2024-09-18T08:14:06.578999+0000","last_change":"2024-09-18T08:13:33.139096+0000","last_active":"2024-09-18T08:14:06.578999+0000","last_peered":"2024-09-18T08:14:06.578999+0000","last_clean":"2024-09-18T08:14:06.578999+0000","last_became_active":"2024-09-18T08:13:28.629930+0000","last_became_peered":"2024-09-18T08:13:28.629930+0000","last_unstale":"2024-09-18T08:14:06.578999+0000","last_undegraded":"2024-09-18T08:14:06.578999+0000","last_fullsized":"2024-09-18T08:14:06.578999+0000","mapping_epoch":280,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":281,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:13:33.138734+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_clean_scrub_stamp":"2024-09-18T08:13:33.138734+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:14:43.987341+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.f","version":"294'16002","reported_seq":16140,"reported_epoch":294,"state":"active+undersized+degraded","last_fresh":"2024-09-18T08:14:08.590208+0000","last_change":"2024-09-18T08:11:42.337301+0000","last_active":"2024-09-18T08:14:08.590208+0000","last_peered":"2024-09-18T08:14:08.590208+0000","last_clean":"2024-09-18T08:11:41.300139+0000","last_became_active":"2024-09-18T08:11:42.337301+0000","last_became_peered":"2024-09-18T08:11:42.337301+0000","last_unstale":"2024-09-18T08:14:08.590208+0000","last_undegraded":"2024-09-18T08:11:42.332576+0000","last_fullsized":"2024-09-18T08:11:42.332455+0000","mapping_epoch":244,"log_start":"277'11602","ondisk_log_start":"277'11602","created":232,"last_epoch_clean":233,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:11:08.262286+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:11:08.262286+0000","last_clean_scrub_stamp":"2024-09-18T08:11:08.262286+0000","objects_scrubbed":0,"log_size":4400,"log_dups_size":0,"ondisk_log_size":4400,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:14:21.188664+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117440512,"num_objects":1792,"num_object_clones":0,"num_object_copies":5376,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1792,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1792,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16002,"num_write_kb":569408,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":["5","3"],"object_location_counts":[{"shards":"3,5","objects":1792}],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.7","version":"74'79","reported_seq":334,"reported_epoch":294,"state":"active+clean","last_fresh":"2024-09-18T08:14:06.583769+0000","last_change":"2024-09-18T08:14:05.599570+0000","last_active":"2024-09-18T08:14:06.583769+0000","last_peered":"2024-09-18T08:14:06.583769+0000","last_clean":"2024-09-18T08:14:06.583769+0000","last_became_active":"2024-09-18T08:14:05.599346+0000","last_became_peered":"2024-09-18T08:14:05.599346+0000","last_unstale":"2024-09-18T08:14:06.583769+0000","last_undegraded":"2024-09-18T08:14:06.583769+0000","last_fullsized":"2024-09-18T08:14:06.583769+0000","mapping_epoch":292,"log_start":"0'0","ondisk_log_start":"0'0","created":152,"last_epoch_clean":293,"parent":"0.0","parent_split_bits":3,"last_scrub":"74'79","last_scrub_stamp":"2024-09-18T08:12:33.846913+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:00:11.893742+0000","last_clean_scrub_stamp":"2024-09-18T08:12:33.846913+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.2","version":"271'20","reported_seq":159,"reported_epoch":294,"state":"active+undersized+degraded","last_fresh":"2024-09-18T08:14:06.581804+0000","last_change":"2024-09-18T08:11:42.341545+0000","last_active":"2024-09-18T08:14:06.581804+0000","last_peered":"2024-09-18T08:14:06.581804+0000","last_clean":"2024-09-18T08:02:54.382625+0000","last_became_active":"2024-09-18T08:11:42.341545+0000","last_became_peered":"2024-09-18T08:11:42.341545+0000","last_unstale":"2024-09-18T08:14:06.581804+0000","last_undegraded":"2024-09-18T08:11:42.323463+0000","last_fullsized":"2024-09-18T08:11:42.323350+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":182,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'10","last_scrub_stamp":"2024-09-18T08:11:24.347134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_clean_scrub_stamp":"2024-09-18T08:11:24.347134+0000","objects_scrubbed":0,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":5,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":29,"num_read_kb":24,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":["5","4"],"object_location_counts":[{"shards":"4,5","objects":5}],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":504,"reported_epoch":294,"state":"active+clean","last_fresh":"2024-09-18T08:14:06.584104+0000","last_change":"2024-09-18T08:12:06.876766+0000","last_active":"2024-09-18T08:14:06.584104+0000","last_peered":"2024-09-18T08:14:06.584104+0000","last_clean":"2024-09-18T08:14:06.584104+0000","last_became_active":"2024-09-18T08:02:47.874727+0000","last_became_peered":"2024-09-18T08:02:47.874727+0000","last_unstale":"2024-09-18T08:14:06.584104+0000","last_undegraded":"2024-09-18T08:14:06.584104+0000","last_fullsized":"2024-09-18T08:14:06.584104+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:12:06.876541+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:46.860092+0000","last_clean_scrub_stamp":"2024-09-18T08:12:06.876541+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.3","version":"138'10","reported_seq":656,"reported_epoch":294,"state":"active+clean","last_fresh":"2024-09-18T08:14:06.580349+0000","last_change":"2024-09-18T08:07:46.695714+0000","last_active":"2024-09-18T08:14:06.580349+0000","last_peered":"2024-09-18T08:14:06.580349+0000","last_clean":"2024-09-18T08:14:06.580349+0000","last_became_active":"2024-09-18T08:06:24.134194+0000","last_became_peered":"2024-09-18T08:06:24.134194+0000","last_unstale":"2024-09-18T08:14:06.580349+0000","last_undegraded":"2024-09-18T08:14:06.580349+0000","last_fullsized":"2024-09-18T08:14:06.580349+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"138'10","last_scrub_stamp":"2024-09-18T08:07:46.695656+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:50.886830+0000","last_clean_scrub_stamp":"2024-09-18T08:07:46.695656+0000","objects_scrubbed":1,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:14:50.804104+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":515,"reported_epoch":294,"state":"active+clean","last_fresh":"2024-09-18T08:14:06.577720+0000","last_change":"2024-09-18T08:09:46.989395+0000","last_active":"2024-09-18T08:14:06.577720+0000","last_peered":"2024-09-18T08:14:06.577720+0000","last_clean":"2024-09-18T08:14:06.577720+0000","last_became_active":"2024-09-18T08:08:52.700080+0000","last_became_peered":"2024-09-18T08:08:52.700080+0000","last_unstale":"2024-09-18T08:14:06.577720+0000","last_undegraded":"2024-09-18T08:14:06.577720+0000","last_fullsized":"2024-09-18T08:14:06.577720+0000","mapping_epoch":198,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":199,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:09:46.989329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_clean_scrub_stamp":"2024-09-18T08:09:46.989329+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:14:16.802519+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.e","version":"294'15831","reported_seq":16519,"reported_epoch":294,"state":"active+undersized+degraded+remapped+backfilling","last_fresh":"2024-09-18T08:14:08.373925+0000","last_change":"2024-09-18T08:14:06.720229+0000","last_active":"2024-09-18T08:14:08.373925+0000","last_peered":"2024-09-18T08:14:08.373925+0000","last_clean":"2024-09-18T08:11:41.287488+0000","last_became_active":"2024-09-18T08:14:06.696534+0000","last_became_peered":"2024-09-18T08:14:06.696534+0000","last_unstale":"2024-09-18T08:14:08.373925+0000","last_undegraded":"2024-09-18T08:14:06.591476+0000","last_fullsized":"2024-09-18T08:14:06.572600+0000","mapping_epoch":293,"log_start":"278'11603","ondisk_log_start":"278'11603","created":232,"last_epoch_clean":233,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:11:08.262286+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:11:08.262286+0000","last_clean_scrub_stamp":"2024-09-18T08:11:08.262286+0000","objects_scrubbed":0,"log_size":4228,"log_dups_size":0,"ondisk_log_size":4228,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":116326400,"num_objects":1775,"num_object_clones":0,"num_object_copies":5325,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1512,"num_objects_misplaced":1775,"num_objects_unfound":0,"num_objects_dirty":1775,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15831,"num_write_kb":563392,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":272,"num_bytes_recovered":17825792,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,7],"avail_no_missing":["4","7"],"object_location_counts":[{"shards":"4,7","objects":1775}],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.6","version":"74'79","reported_seq":415,"reported_epoch":294,"state":"active+clean","last_fresh":"2024-09-18T08:14:06.577476+0000","last_change":"2024-09-18T08:14:02.837236+0000","last_active":"2024-09-18T08:14:06.577476+0000","last_peered":"2024-09-18T08:14:06.577476+0000","last_clean":"2024-09-18T08:14:06.577476+0000","last_became_active":"2024-09-18T08:07:26.426532+0000","last_became_peered":"2024-09-18T08:07:26.426532+0000","last_unstale":"2024-09-18T08:14:06.577476+0000","last_undegraded":"2024-09-18T08:14:06.577476+0000","last_fullsized":"2024-09-18T08:14:06.577476+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":152,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":3,"last_scrub":"74'79","last_scrub_stamp":"2024-09-18T08:14:02.837167+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:00:11.893742+0000","last_clean_scrub_stamp":"2024-09-18T08:14:02.837167+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:15:32.552339+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,3],"acting":[0,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.3","version":"271'29","reported_seq":557,"reported_epoch":294,"state":"active+clean+remapped","last_fresh":"2024-09-18T08:14:06.592395+0000","last_change":"2024-09-18T08:14:06.591251+0000","last_active":"2024-09-18T08:14:06.592395+0000","last_peered":"2024-09-18T08:14:06.592395+0000","last_clean":"2024-09-18T08:14:06.592395+0000","last_became_active":"2024-09-18T08:14:06.589563+0000","last_became_peered":"2024-09-18T08:14:06.589563+0000","last_unstale":"2024-09-18T08:14:06.592395+0000","last_undegraded":"2024-09-18T08:14:06.592395+0000","last_fullsized":"2024-09-18T08:14:06.592395+0000","mapping_epoch":293,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":294,"parent":"0.0","parent_split_bits":0,"last_scrub":"271'29","last_scrub_stamp":"2024-09-18T08:13:33.814992+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_clean_scrub_stamp":"2024-09-18T08:13:33.814992+0000","objects_scrubbed":8,"log_size":29,"log_dups_size":0,"ondisk_log_size":29,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:14:47.981166+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":8,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":54,"num_read_kb":40,"num_write":32,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5,7],"avail_no_missing":["0","5","7"],"object_location_counts":[{"shards":"0,5,7","objects":8}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.4","version":"62'1","reported_seq":418,"reported_epoch":294,"state":"active+undersized+degraded","last_fresh":"2024-09-18T08:14:06.572873+0000","last_change":"2024-09-18T08:11:42.328236+0000","last_active":"2024-09-18T08:14:06.572873+0000","last_peered":"2024-09-18T08:14:06.572873+0000","last_clean":"2024-09-18T08:08:50.690355+0000","last_became_active":"2024-09-18T08:11:42.328236+0000","last_became_peered":"2024-09-18T08:11:42.328236+0000","last_unstale":"2024-09-18T08:14:06.572873+0000","last_undegraded":"2024-09-18T08:11:42.319415+0000","last_fullsized":"2024-09-18T08:11:42.319312+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":199,"parent":"0.0","parent_split_bits":0,"last_scrub":"62'1","last_scrub_stamp":"2024-09-18T08:10:07.593004+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:46.860092+0000","last_clean_scrub_stamp":"2024-09-18T08:10:07.593004+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:15:02.614234+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":436,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":436,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":["2","5"],"object_location_counts":[{"shards":"2,5","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":422,"reported_epoch":294,"state":"active+clean","last_fresh":"2024-09-18T08:14:06.577844+0000","last_change":"2024-09-18T08:12:37.732331+0000","last_active":"2024-09-18T08:14:06.577844+0000","last_peered":"2024-09-18T08:14:06.577844+0000","last_clean":"2024-09-18T08:14:06.577844+0000","last_became_active":"2024-09-18T08:06:24.133262+0000","last_became_peered":"2024-09-18T08:06:24.133262+0000","last_unstale":"2024-09-18T08:14:06.577844+0000","last_undegraded":"2024-09-18T08:14:06.577844+0000","last_fullsized":"2024-09-18T08:14:06.577844+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:12:37.732260+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:50.886830+0000","last_clean_scrub_stamp":"2024-09-18T08:12:37.732260+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":426,"reported_epoch":294,"state":"active+undersized","last_fresh":"2024-09-18T08:14:06.571105+0000","last_change":"2024-09-18T08:11:42.332856+0000","last_active":"2024-09-18T08:14:06.571105+0000","last_peered":"2024-09-18T08:14:06.571105+0000","last_clean":"2024-09-18T08:08:50.690188+0000","last_became_active":"2024-09-18T08:11:42.332856+0000","last_became_peered":"2024-09-18T08:11:42.332856+0000","last_unstale":"2024-09-18T08:14:06.571105+0000","last_undegraded":"2024-09-18T08:14:06.571105+0000","last_fullsized":"2024-09-18T08:11:42.318131+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":199,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:11:28.346395+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_clean_scrub_stamp":"2024-09-18T08:11:28.346395+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":["2","6"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.9","version":"294'15945","reported_seq":16635,"reported_epoch":294,"state":"active+remapped+backfilling","last_fresh":"2024-09-18T08:14:09.067731+0000","last_change":"2024-09-18T08:14:07.121119+0000","last_active":"2024-09-18T08:14:09.067731+0000","last_peered":"2024-09-18T08:14:09.067731+0000","last_clean":"2024-09-18T08:14:04.565261+0000","last_became_active":"2024-09-18T08:14:07.034609+0000","last_became_peered":"2024-09-18T08:14:07.034609+0000","last_unstale":"2024-09-18T08:14:09.067731+0000","last_undegraded":"2024-09-18T08:14:09.067731+0000","last_fullsized":"2024-09-18T08:14:09.067731+0000","mapping_epoch":293,"log_start":"276'11399","ondisk_log_start":"276'11399","created":232,"last_epoch_clean":233,"parent":"0.0","parent_split_bits":0,"last_scrub":"274'10944","last_scrub_stamp":"2024-09-18T08:13:08.015855+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:11:08.262286+0000","last_clean_scrub_stamp":"2024-09-18T08:13:08.015855+0000","objects_scrubbed":6827,"log_size":4546,"log_dups_size":0,"ondisk_log_size":4546,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"no scrub is scheduled","scrub_duration":2172,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":113442816,"num_objects":1731,"num_object_clones":0,"num_object_copies":5193,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1506,"num_objects_unfound":0,"num_objects_dirty":1731,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15945,"num_write_kb":565632,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":235,"num_bytes_recovered":15400960,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,7],"avail_no_missing":["0","5","7"],"object_location_counts":[{"shards":"0,5,7","objects":1731}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"74'79","reported_seq":326,"reported_epoch":294,"state":"active+clean","last_fresh":"2024-09-18T08:14:06.577571+0000","last_change":"2024-09-18T08:13:55.196810+0000","last_active":"2024-09-18T08:14:06.577571+0000","last_peered":"2024-09-18T08:14:06.577571+0000","last_clean":"2024-09-18T08:14:06.577571+0000","last_became_active":"2024-09-18T08:07:26.400643+0000","last_became_peered":"2024-09-18T08:07:26.400643+0000","last_unstale":"2024-09-18T08:14:06.577571+0000","last_undegraded":"2024-09-18T08:14:06.577571+0000","last_fullsized":"2024-09-18T08:14:06.577571+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":152,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":4,"last_scrub":"74'79","last_scrub_stamp":"2024-09-18T08:13:55.194411+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:00:11.893742+0000","last_clean_scrub_stamp":"2024-09-18T08:13:55.194411+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:15:03.286645+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,2],"acting":[5,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.4","version":"271'41","reported_seq":228,"reported_epoch":294,"state":"active+undersized+degraded","last_fresh":"2024-09-18T08:14:06.574035+0000","last_change":"2024-09-18T08:11:42.328675+0000","last_active":"2024-09-18T08:14:06.574035+0000","last_peered":"2024-09-18T08:14:06.574035+0000","last_clean":"2024-09-18T08:02:54.384549+0000","last_became_active":"2024-09-18T08:11:42.328675+0000","last_became_peered":"2024-09-18T08:11:42.328675+0000","last_unstale":"2024-09-18T08:14:06.574035+0000","last_undegraded":"2024-09-18T08:11:42.320375+0000","last_fullsized":"2024-09-18T08:11:42.320272+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'28","last_scrub_stamp":"2024-09-18T08:10:46.497874+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_clean_scrub_stamp":"2024-09-18T08:10:46.497874+0000","objects_scrubbed":0,"log_size":41,"log_dups_size":0,"ondisk_log_size":41,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:14:14.418386+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":10,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":65,"num_read_kb":50,"num_write":38,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":10}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":246,"reported_epoch":294,"state":"active+clean","last_fresh":"2024-09-18T08:14:06.572608+0000","last_change":"2024-09-18T08:13:52.760213+0000","last_active":"2024-09-18T08:14:06.572608+0000","last_peered":"2024-09-18T08:14:06.572608+0000","last_clean":"2024-09-18T08:14:06.572608+0000","last_became_active":"2024-09-18T08:08:18.538852+0000","last_became_peered":"2024-09-18T08:08:18.538852+0000","last_unstale":"2024-09-18T08:14:06.572608+0000","last_undegraded":"2024-09-18T08:14:06.572608+0000","last_fullsized":"2024-09-18T08:14:06.572608+0000","mapping_epoch":181,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":182,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:13:52.760146+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:46.860092+0000","last_clean_scrub_stamp":"2024-09-18T08:13:52.760146+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:14:54.775611+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"64'2","reported_seq":532,"reported_epoch":294,"state":"active+undersized+degraded","last_fresh":"2024-09-18T08:14:06.592172+0000","last_change":"2024-09-18T08:11:42.338718+0000","last_active":"2024-09-18T08:14:06.592172+0000","last_peered":"2024-09-18T08:14:06.592172+0000","last_clean":"2024-09-18T08:11:40.302328+0000","last_became_active":"2024-09-18T08:11:42.338718+0000","last_became_peered":"2024-09-18T08:11:42.338718+0000","last_unstale":"2024-09-18T08:14:06.592172+0000","last_undegraded":"2024-09-18T08:11:42.331457+0000","last_fullsized":"2024-09-18T08:11:42.330826+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"64'2","last_scrub_stamp":"2024-09-18T08:05:44.802942+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:45.855979+0000","last_clean_scrub_stamp":"2024-09-18T08:05:44.802942+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:15:02.376480+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.001348349,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":38,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":["5","6"],"object_location_counts":[{"shards":"5,6","objects":1}],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":506,"reported_epoch":294,"state":"active+undersized","last_fresh":"2024-09-18T08:14:06.582465+0000","last_change":"2024-09-18T08:11:42.334063+0000","last_active":"2024-09-18T08:14:06.582465+0000","last_peered":"2024-09-18T08:14:06.582465+0000","last_clean":"2024-09-18T08:11:40.307605+0000","last_became_active":"2024-09-18T08:11:42.334063+0000","last_became_peered":"2024-09-18T08:11:42.334063+0000","last_unstale":"2024-09-18T08:14:06.582465+0000","last_undegraded":"2024-09-18T08:14:06.582465+0000","last_fullsized":"2024-09-18T08:11:42.319418+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":182,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:10:53.880606+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:50.886830+0000","last_clean_scrub_stamp":"2024-09-18T08:10:53.880606+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:14:44.773598+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":["0","4"],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":523,"reported_epoch":294,"state":"active+clean","last_fresh":"2024-09-18T08:14:06.573960+0000","last_change":"2024-09-18T08:14:05.578066+0000","last_active":"2024-09-18T08:14:06.573960+0000","last_peered":"2024-09-18T08:14:06.573960+0000","last_clean":"2024-09-18T08:14:06.573960+0000","last_became_active":"2024-09-18T08:14:05.577377+0000","last_became_peered":"2024-09-18T08:14:05.577377+0000","last_unstale":"2024-09-18T08:14:06.573960+0000","last_undegraded":"2024-09-18T08:14:06.573960+0000","last_fullsized":"2024-09-18T08:14:06.573960+0000","mapping_epoch":292,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":293,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:13:36.653164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_clean_scrub_stamp":"2024-09-18T08:13:36.653164+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:14:41.556848+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,6],"acting":[3,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.a","version":"294'15825","reported_seq":16540,"reported_epoch":294,"state":"active+undersized+degraded+remapped+backfilling","last_fresh":"2024-09-18T08:14:08.593151+0000","last_change":"2024-09-18T08:14:06.804156+0000","last_active":"2024-09-18T08:14:08.593151+0000","last_peered":"2024-09-18T08:14:08.593151+0000","last_clean":"2024-09-18T08:11:41.323914+0000","last_became_active":"2024-09-18T08:14:06.766821+0000","last_became_peered":"2024-09-18T08:14:06.766821+0000","last_unstale":"2024-09-18T08:14:08.593151+0000","last_undegraded":"2024-09-18T08:14:06.606070+0000","last_fullsized":"2024-09-18T08:14:06.577365+0000","mapping_epoch":293,"log_start":"276'11433","ondisk_log_start":"276'11433","created":232,"last_epoch_clean":233,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:11:08.262286+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:11:08.262286+0000","last_clean_scrub_stamp":"2024-09-18T08:11:08.262286+0000","objects_scrubbed":0,"log_size":4392,"log_dups_size":0,"ondisk_log_size":4392,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":122224640,"num_objects":1865,"num_object_clones":0,"num_object_copies":5595,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1596,"num_objects_misplaced":1865,"num_objects_unfound":0,"num_objects_dirty":1865,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15825,"num_write_kb":566080,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":285,"num_bytes_recovered":18677760,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,7],"avail_no_missing":["5","7"],"object_location_counts":[{"shards":"5,7","objects":1865}],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.2","version":"74'79","reported_seq":435,"reported_epoch":294,"state":"active+clean","last_fresh":"2024-09-18T08:14:06.577675+0000","last_change":"2024-09-18T08:14:05.586022+0000","last_active":"2024-09-18T08:14:06.577675+0000","last_peered":"2024-09-18T08:14:06.577675+0000","last_clean":"2024-09-18T08:14:06.577675+0000","last_became_active":"2024-09-18T08:14:05.585608+0000","last_became_peered":"2024-09-18T08:14:05.585608+0000","last_unstale":"2024-09-18T08:14:06.577675+0000","last_undegraded":"2024-09-18T08:14:06.577675+0000","last_fullsized":"2024-09-18T08:14:06.577675+0000","mapping_epoch":292,"log_start":"0'0","ondisk_log_start":"0'0","created":152,"last_epoch_clean":293,"parent":"0.0","parent_split_bits":4,"last_scrub":"74'79","last_scrub_stamp":"2024-09-18T08:14:01.846338+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:00:11.893742+0000","last_clean_scrub_stamp":"2024-09-18T08:14:01.846338+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:15:31.647562+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,2],"acting":[0,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.7","version":"271'17","reported_seq":460,"reported_epoch":294,"state":"active+clean","last_fresh":"2024-09-18T08:14:06.572221+0000","last_change":"2024-09-18T08:13:30.296958+0000","last_active":"2024-09-18T08:14:06.572221+0000","last_peered":"2024-09-18T08:14:06.572221+0000","last_clean":"2024-09-18T08:14:06.572221+0000","last_became_active":"2024-09-18T08:04:18.144925+0000","last_became_peered":"2024-09-18T08:04:18.144925+0000","last_unstale":"2024-09-18T08:14:06.572221+0000","last_undegraded":"2024-09-18T08:14:06.572221+0000","last_fullsized":"2024-09-18T08:14:06.572221+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"271'17","last_scrub_stamp":"2024-09-18T08:13:30.296881+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_clean_scrub_stamp":"2024-09-18T08:13:30.296881+0000","objects_scrubbed":5,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:14:48.825965+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":35,"num_read_kb":24,"num_write":20,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,0],"acting":[2,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":284,"reported_epoch":294,"state":"active+undersized","last_fresh":"2024-09-18T08:14:06.572357+0000","last_change":"2024-09-18T08:11:42.322313+0000","last_active":"2024-09-18T08:14:06.572357+0000","last_peered":"2024-09-18T08:14:06.572357+0000","last_clean":"2024-09-18T08:06:21.513250+0000","last_became_active":"2024-09-18T08:11:42.322313+0000","last_became_peered":"2024-09-18T08:11:42.322313+0000","last_unstale":"2024-09-18T08:14:06.572357+0000","last_undegraded":"2024-09-18T08:14:06.572357+0000","last_fullsized":"2024-09-18T08:11:42.318707+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:11:31.367525+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:46.860092+0000","last_clean_scrub_stamp":"2024-09-18T08:11:31.367525+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":["2","6"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1","version":"62'1","reported_seq":514,"reported_epoch":294,"state":"active+clean","last_fresh":"2024-09-18T08:14:06.571625+0000","last_change":"2024-09-18T08:12:03.497218+0000","last_active":"2024-09-18T08:14:06.571625+0000","last_peered":"2024-09-18T08:14:06.571625+0000","last_clean":"2024-09-18T08:14:06.571625+0000","last_became_active":"2024-09-18T08:02:46.868175+0000","last_became_peered":"2024-09-18T08:02:46.868175+0000","last_unstale":"2024-09-18T08:14:06.571625+0000","last_undegraded":"2024-09-18T08:14:06.571625+0000","last_fullsized":"2024-09-18T08:14:06.571625+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"62'1","last_scrub_stamp":"2024-09-18T08:12:03.497148+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:45.855979+0000","last_clean_scrub_stamp":"2024-09-18T08:12:03.497148+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:14:25.351946+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.00066785499999999997,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":211,"reported_epoch":294,"state":"active+clean","last_fresh":"2024-09-18T08:14:06.575280+0000","last_change":"2024-09-18T08:14:05.593996+0000","last_active":"2024-09-18T08:14:06.575280+0000","last_peered":"2024-09-18T08:14:06.575280+0000","last_clean":"2024-09-18T08:14:06.575280+0000","last_became_active":"2024-09-18T08:14:05.593636+0000","last_became_peered":"2024-09-18T08:14:05.593636+0000","last_unstale":"2024-09-18T08:14:06.575280+0000","last_undegraded":"2024-09-18T08:14:06.575280+0000","last_fullsized":"2024-09-18T08:14:06.575280+0000","mapping_epoch":292,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":293,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:07:08.806748+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:50.886830+0000","last_clean_scrub_stamp":"2024-09-18T08:07:08.806748+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,4],"acting":[2,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":194,"reported_epoch":294,"state":"active+clean","last_fresh":"2024-09-18T08:14:06.574559+0000","last_change":"2024-09-18T08:14:05.578356+0000","last_active":"2024-09-18T08:14:06.574559+0000","last_peered":"2024-09-18T08:14:06.574559+0000","last_clean":"2024-09-18T08:14:06.574559+0000","last_became_active":"2024-09-18T08:14:05.577516+0000","last_became_peered":"2024-09-18T08:14:05.577516+0000","last_unstale":"2024-09-18T08:14:06.574559+0000","last_undegraded":"2024-09-18T08:14:06.574559+0000","last_fullsized":"2024-09-18T08:14:06.574559+0000","mapping_epoch":292,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":293,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:13:36.978582+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_clean_scrub_stamp":"2024-09-18T08:13:36.978582+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:14:52.123847+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.b","version":"294'15759","reported_seq":15895,"reported_epoch":294,"state":"active+clean","last_fresh":"2024-09-18T08:14:09.079799+0000","last_change":"2024-09-18T08:11:09.489007+0000","last_active":"2024-09-18T08:14:09.079799+0000","last_peered":"2024-09-18T08:14:09.079799+0000","last_clean":"2024-09-18T08:14:09.079799+0000","last_became_active":"2024-09-18T08:11:09.488677+0000","last_became_peered":"2024-09-18T08:11:09.488677+0000","last_unstale":"2024-09-18T08:14:09.079799+0000","last_undegraded":"2024-09-18T08:14:09.079799+0000","last_fullsized":"2024-09-18T08:14:09.079799+0000","mapping_epoch":232,"log_start":"276'11229","ondisk_log_start":"276'11229","created":232,"last_epoch_clean":233,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:11:08.262286+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:11:08.262286+0000","last_clean_scrub_stamp":"2024-09-18T08:11:08.262286+0000","objects_scrubbed":0,"log_size":4530,"log_dups_size":0,"ondisk_log_size":4530,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:14:43.811553+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117374976,"num_objects":1791,"num_object_clones":0,"num_object_copies":5373,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1791,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15759,"num_write_kb":561600,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"74'79","reported_seq":188,"reported_epoch":294,"state":"active+undersized","last_fresh":"2024-09-18T08:14:06.571627+0000","last_change":"2024-09-18T08:11:42.324310+0000","last_active":"2024-09-18T08:14:06.571627+0000","last_peered":"2024-09-18T08:14:06.571627+0000","last_clean":"2024-09-18T08:08:15.515052+0000","last_became_active":"2024-09-18T08:11:42.324310+0000","last_became_peered":"2024-09-18T08:11:42.324310+0000","last_unstale":"2024-09-18T08:14:06.571627+0000","last_undegraded":"2024-09-18T08:14:06.571627+0000","last_fullsized":"2024-09-18T08:11:42.318664+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":152,"last_epoch_clean":182,"parent":"0.0","parent_split_bits":3,"last_scrub":"74'79","last_scrub_stamp":"2024-09-18T08:09:41.651729+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:00:11.893742+0000","last_clean_scrub_stamp":"2024-09-18T08:09:41.651729+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"user requested scrub scheduled @ 2024-09-18T08:14:43.628779+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":53670,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":10,"num_read_kb":20,"num_write":11,"num_write_kb":148,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":53670,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":["4","6"],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.6","version":"271'24","reported_seq":582,"reported_epoch":294,"state":"active+undersized+degraded","last_fresh":"2024-09-18T08:14:06.578456+0000","last_change":"2024-09-18T08:11:42.342654+0000","last_active":"2024-09-18T08:14:06.578456+0000","last_peered":"2024-09-18T08:14:06.578456+0000","last_clean":"2024-09-18T08:11:40.307291+0000","last_became_active":"2024-09-18T08:11:42.342654+0000","last_became_peered":"2024-09-18T08:11:42.342654+0000","last_unstale":"2024-09-18T08:14:06.578456+0000","last_undegraded":"2024-09-18T08:11:42.320107+0000","last_fullsized":"2024-09-18T08:11:42.320001+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":200,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'12","last_scrub_stamp":"2024-09-18T08:10:58.837531+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_clean_scrub_stamp":"2024-09-18T08:10:58.837531+0000","objects_scrubbed":6,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:14:54.835181+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":6,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":36,"num_read_kb":30,"num_write":24,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0,2","objects":6}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":526,"reported_epoch":294,"state":"active+clean","last_fresh":"2024-09-18T08:14:06.582678+0000","last_change":"2024-09-18T08:12:05.794780+0000","last_active":"2024-09-18T08:14:06.582678+0000","last_peered":"2024-09-18T08:14:06.582678+0000","last_clean":"2024-09-18T08:14:06.582678+0000","last_became_active":"2024-09-18T08:08:18.523135+0000","last_became_peered":"2024-09-18T08:08:18.523135+0000","last_unstale":"2024-09-18T08:14:06.582678+0000","last_undegraded":"2024-09-18T08:14:06.582678+0000","last_fullsized":"2024-09-18T08:14:06.582678+0000","mapping_epoch":181,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":182,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:12:05.794714+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:46.860092+0000","last_clean_scrub_stamp":"2024-09-18T08:12:05.794714+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"70'5","reported_seq":307,"reported_epoch":294,"state":"active+clean+remapped","last_fresh":"2024-09-18T08:14:07.593607+0000","last_change":"2024-09-18T08:14:06.596114+0000","last_active":"2024-09-18T08:14:07.593607+0000","last_peered":"2024-09-18T08:14:07.593607+0000","last_clean":"2024-09-18T08:14:07.593607+0000","last_became_active":"2024-09-18T08:14:06.588753+0000","last_became_peered":"2024-09-18T08:14:06.588753+0000","last_unstale":"2024-09-18T08:14:07.593607+0000","last_undegraded":"2024-09-18T08:14:07.593607+0000","last_fullsized":"2024-09-18T08:14:07.593607+0000","mapping_epoch":293,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":294,"parent":"0.0","parent_split_bits":0,"last_scrub":"70'5","last_scrub_stamp":"2024-09-18T08:10:44.388400+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:45.855979+0000","last_clean_scrub_stamp":"2024-09-18T08:10:44.388400+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000504055,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":145,"num_read_kb":140,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":389,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3,7],"avail_no_missing":["0","3","7"],"object_location_counts":[{"shards":"0,3,7","objects":1}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":500,"reported_epoch":294,"state":"active+undersized","last_fresh":"2024-09-18T08:14:06.610984+0000","last_change":"2024-09-18T08:11:42.333954+0000","last_active":"2024-09-18T08:14:06.610984+0000","last_peered":"2024-09-18T08:14:06.610984+0000","last_clean":"2024-09-18T08:11:40.304599+0000","last_became_active":"2024-09-18T08:11:42.333954+0000","last_became_peered":"2024-09-18T08:11:42.333954+0000","last_unstale":"2024-09-18T08:14:06.610984+0000","last_undegraded":"2024-09-18T08:14:06.610984+0000","last_fullsized":"2024-09-18T08:11:42.326679+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:07:20.767841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:50.886830+0000","last_clean_scrub_stamp":"2024-09-18T08:07:20.767841+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:15:03.417204+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":["5","0"],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":109,"reported_epoch":294,"state":"active+undersized","last_fresh":"2024-09-18T08:14:06.579552+0000","last_change":"2024-09-18T08:11:42.341422+0000","last_active":"2024-09-18T08:14:06.579552+0000","last_peered":"2024-09-18T08:14:06.579552+0000","last_clean":"2024-09-18T08:02:52.890475+0000","last_became_active":"2024-09-18T08:11:42.341422+0000","last_became_peered":"2024-09-18T08:11:42.341422+0000","last_unstale":"2024-09-18T08:14:06.579552+0000","last_undegraded":"2024-09-18T08:14:06.579552+0000","last_fullsized":"2024-09-18T08:11:42.325862+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:09:49.702055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_clean_scrub_stamp":"2024-09-18T08:09:49.702055+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:14:07.222323+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":["5","3"],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.8","version":"294'16020","reported_seq":16156,"reported_epoch":294,"state":"active+clean","last_fresh":"2024-09-18T08:14:09.080682+0000","last_change":"2024-09-18T08:11:09.291403+0000","last_active":"2024-09-18T08:14:09.080682+0000","last_peered":"2024-09-18T08:14:09.080682+0000","last_clean":"2024-09-18T08:14:09.080682+0000","last_became_active":"2024-09-18T08:11:09.291038+0000","last_became_peered":"2024-09-18T08:11:09.291038+0000","last_unstale":"2024-09-18T08:14:09.080682+0000","last_undegraded":"2024-09-18T08:14:09.080682+0000","last_fullsized":"2024-09-18T08:14:09.080682+0000","mapping_epoch":232,"log_start":"279'12104","ondisk_log_start":"279'12104","created":232,"last_epoch_clean":233,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:11:08.262286+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:11:08.262286+0000","last_clean_scrub_stamp":"2024-09-18T08:11:08.262286+0000","objects_scrubbed":0,"log_size":3916,"log_dups_size":0,"ondisk_log_size":3916,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:14:47.488844+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":118489088,"num_objects":1808,"num_object_clones":0,"num_object_copies":5424,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1808,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16020,"num_write_kb":570496,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.5","version":"293'34","reported_seq":387,"reported_epoch":294,"state":"active+clean","last_fresh":"2024-09-18T08:14:06.573105+0000","last_change":"2024-09-18T08:14:05.604747+0000","last_active":"2024-09-18T08:14:06.573105+0000","last_peered":"2024-09-18T08:14:06.573105+0000","last_clean":"2024-09-18T08:14:06.573105+0000","last_became_active":"2024-09-18T08:14:05.585190+0000","last_became_peered":"2024-09-18T08:14:05.585190+0000","last_unstale":"2024-09-18T08:14:06.573105+0000","last_undegraded":"2024-09-18T08:14:06.573105+0000","last_fullsized":"2024-09-18T08:14:06.573105+0000","mapping_epoch":292,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":293,"parent":"0.0","parent_split_bits":0,"last_scrub":"271'33","last_scrub_stamp":"2024-09-18T08:13:28.626987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_clean_scrub_stamp":"2024-09-18T08:13:28.626987+0000","objects_scrubbed":8,"log_size":34,"log_dups_size":0,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:14:46.287420+0000","scrub_duration":24,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":42,"num_read_kb":32,"num_write":25,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":308,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,0],"acting":[4,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":511,"reported_epoch":294,"state":"active+clean","last_fresh":"2024-09-18T08:14:06.572311+0000","last_change":"2024-09-18T08:09:45.572278+0000","last_active":"2024-09-18T08:14:06.572311+0000","last_peered":"2024-09-18T08:14:06.572311+0000","last_clean":"2024-09-18T08:14:06.572311+0000","last_became_active":"2024-09-18T08:06:24.133425+0000","last_became_peered":"2024-09-18T08:06:24.133425+0000","last_unstale":"2024-09-18T08:14:06.572311+0000","last_undegraded":"2024-09-18T08:14:06.572311+0000","last_fullsized":"2024-09-18T08:14:06.572311+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:09:45.572201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:46.860092+0000","last_clean_scrub_stamp":"2024-09-18T08:09:45.572201+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:14:26.723239+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"74'79","reported_seq":117,"reported_epoch":294,"state":"active+clean+remapped","last_fresh":"2024-09-18T08:14:06.589678+0000","last_change":"2024-09-18T08:14:06.589234+0000","last_active":"2024-09-18T08:14:06.589678+0000","last_peered":"2024-09-18T08:14:06.589678+0000","last_clean":"2024-09-18T08:14:06.589678+0000","last_became_active":"2024-09-18T08:14:06.588319+0000","last_became_peered":"2024-09-18T08:14:06.588319+0000","last_unstale":"2024-09-18T08:14:06.589678+0000","last_undegraded":"2024-09-18T08:14:06.589678+0000","last_fullsized":"2024-09-18T08:14:06.589678+0000","mapping_epoch":293,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":294,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'79","last_scrub_stamp":"2024-09-18T08:13:51.810296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:00:11.893742+0000","last_clean_scrub_stamp":"2024-09-18T08:13:51.810296+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:15:07.919718+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6,7],"avail_no_missing":["0","6","7"],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":347,"reported_epoch":294,"state":"active+clean","last_fresh":"2024-09-18T08:14:06.573721+0000","last_change":"2024-09-18T08:14:05.584317+0000","last_active":"2024-09-18T08:14:06.573721+0000","last_peered":"2024-09-18T08:14:06.573721+0000","last_clean":"2024-09-18T08:14:06.573721+0000","last_became_active":"2024-09-18T08:14:05.583946+0000","last_became_peered":"2024-09-18T08:14:05.583946+0000","last_unstale":"2024-09-18T08:14:06.573721+0000","last_undegraded":"2024-09-18T08:14:06.573721+0000","last_fullsized":"2024-09-18T08:14:06.573721+0000","mapping_epoch":292,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":293,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:12:56.304527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:50.886830+0000","last_clean_scrub_stamp":"2024-09-18T08:12:56.304527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:14:08.783188+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,0],"acting":[2,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":507,"reported_epoch":294,"state":"active+clean","last_fresh":"2024-09-18T08:14:06.610839+0000","last_change":"2024-09-18T08:12:05.886343+0000","last_active":"2024-09-18T08:14:06.610839+0000","last_peered":"2024-09-18T08:14:06.610839+0000","last_clean":"2024-09-18T08:14:06.610839+0000","last_became_active":"2024-09-18T08:08:18.548233+0000","last_became_peered":"2024-09-18T08:08:18.548233+0000","last_unstale":"2024-09-18T08:14:06.610839+0000","last_undegraded":"2024-09-18T08:14:06.610839+0000","last_fullsized":"2024-09-18T08:14:06.610839+0000","mapping_epoch":181,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":182,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:12:05.886281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_clean_scrub_stamp":"2024-09-18T08:12:05.886281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.3","version":"294'16310","reported_seq":17214,"reported_epoch":294,"state":"active+undersized+degraded+remapped+backfilling","last_fresh":"2024-09-18T08:14:09.099232+0000","last_change":"2024-09-18T08:14:06.831081+0000","last_active":"2024-09-18T08:14:09.099232+0000","last_peered":"2024-09-18T08:14:09.099232+0000","last_clean":"2024-09-18T08:11:41.283527+0000","last_became_active":"2024-09-18T08:14:06.696641+0000","last_became_peered":"2024-09-18T08:14:06.696641+0000","last_unstale":"2024-09-18T08:14:09.099232+0000","last_undegraded":"2024-09-18T08:14:06.598678+0000","last_fullsized":"2024-09-18T08:14:06.572081+0000","mapping_epoch":293,"log_start":"279'12403","ondisk_log_start":"279'12403","created":232,"last_epoch_clean":233,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:11:08.262286+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:11:08.262286+0000","last_clean_scrub_stamp":"2024-09-18T08:11:08.262286+0000","objects_scrubbed":0,"log_size":3907,"log_dups_size":0,"ondisk_log_size":3907,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":118751232,"num_objects":1812,"num_object_clones":0,"num_object_copies":5436,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1453,"num_objects_misplaced":1812,"num_objects_unfound":0,"num_objects_dirty":1812,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16310,"num_write_kb":579904,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":380,"num_bytes_recovered":24838144,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,7],"avail_no_missing":["3","7"],"object_location_counts":[{"shards":"3,7","objects":1812}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.e","version":"271'19","reported_seq":274,"reported_epoch":294,"state":"active+undersized+degraded","last_fresh":"2024-09-18T08:14:06.574596+0000","last_change":"2024-09-18T08:11:42.337720+0000","last_active":"2024-09-18T08:14:06.574596+0000","last_peered":"2024-09-18T08:14:06.574596+0000","last_clean":"2024-09-18T08:11:40.307924+0000","last_became_active":"2024-09-18T08:11:42.337720+0000","last_became_peered":"2024-09-18T08:11:42.337720+0000","last_unstale":"2024-09-18T08:14:06.574596+0000","last_undegraded":"2024-09-18T08:11:42.327549+0000","last_fullsized":"2024-09-18T08:11:42.327437+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":182,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'11","last_scrub_stamp":"2024-09-18T08:10:53.406574+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_clean_scrub_stamp":"2024-09-18T08:10:53.406574+0000","objects_scrubbed":5,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:14:34.671796+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":5,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":31,"num_read_kb":24,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":["4","6"],"object_location_counts":[{"shards":"4,6","objects":5}],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":260,"reported_epoch":294,"state":"active+clean+remapped","last_fresh":"2024-09-18T08:14:06.588009+0000","last_change":"2024-09-18T08:14:06.587860+0000","last_active":"2024-09-18T08:14:06.588009+0000","last_peered":"2024-09-18T08:14:06.588009+0000","last_clean":"2024-09-18T08:14:06.588009+0000","last_became_active":"2024-09-18T08:14:06.586627+0000","last_became_peered":"2024-09-18T08:14:06.586627+0000","last_unstale":"2024-09-18T08:14:06.588009+0000","last_undegraded":"2024-09-18T08:14:06.588009+0000","last_fullsized":"2024-09-18T08:14:06.588009+0000","mapping_epoch":293,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":294,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:11:59.458232+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:46.860092+0000","last_clean_scrub_stamp":"2024-09-18T08:11:59.458232+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2,7],"avail_no_missing":["4","2","7"],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":532,"reported_epoch":294,"state":"active+clean","last_fresh":"2024-09-18T08:14:06.610734+0000","last_change":"2024-09-18T08:13:34.187365+0000","last_active":"2024-09-18T08:14:06.610734+0000","last_peered":"2024-09-18T08:14:06.610734+0000","last_clean":"2024-09-18T08:14:06.610734+0000","last_became_active":"2024-09-18T08:08:18.547105+0000","last_became_peered":"2024-09-18T08:08:18.547105+0000","last_unstale":"2024-09-18T08:14:06.610734+0000","last_undegraded":"2024-09-18T08:14:06.610734+0000","last_fullsized":"2024-09-18T08:14:06.610734+0000","mapping_epoch":181,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":182,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:13:34.187158+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:50.886830+0000","last_clean_scrub_stamp":"2024-09-18T08:13:34.187158+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:14:39.225982+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":498,"reported_epoch":294,"state":"active+clean","last_fresh":"2024-09-18T08:14:06.573803+0000","last_change":"2024-09-18T08:07:09.197912+0000","last_active":"2024-09-18T08:14:06.573803+0000","last_peered":"2024-09-18T08:14:06.573803+0000","last_clean":"2024-09-18T08:14:06.573803+0000","last_became_active":"2024-09-18T08:06:24.130266+0000","last_became_peered":"2024-09-18T08:06:24.130266+0000","last_unstale":"2024-09-18T08:14:06.573803+0000","last_undegraded":"2024-09-18T08:14:06.573803+0000","last_fullsized":"2024-09-18T08:14:06.573803+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:07:09.197841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_clean_scrub_stamp":"2024-09-18T08:07:09.197841+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:14:13.754695+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.0","version":"294'15851","reported_seq":15990,"reported_epoch":294,"state":"active+undersized+degraded","last_fresh":"2024-09-18T08:14:08.786490+0000","last_change":"2024-09-18T08:11:42.323264+0000","last_active":"2024-09-18T08:14:08.786490+0000","last_peered":"2024-09-18T08:14:08.786490+0000","last_clean":"2024-09-18T08:11:41.291289+0000","last_became_active":"2024-09-18T08:11:42.323264+0000","last_became_peered":"2024-09-18T08:11:42.323264+0000","last_unstale":"2024-09-18T08:14:08.786490+0000","last_undegraded":"2024-09-18T08:11:42.320910+0000","last_fullsized":"2024-09-18T08:11:42.317956+0000","mapping_epoch":244,"log_start":"273'10376","ondisk_log_start":"273'10376","created":232,"last_epoch_clean":233,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:11:08.262286+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:11:08.262286+0000","last_clean_scrub_stamp":"2024-09-18T08:11:08.262286+0000","objects_scrubbed":0,"log_size":5475,"log_dups_size":0,"ondisk_log_size":5475,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:14:22.337245+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":116195328,"num_objects":1773,"num_object_clones":0,"num_object_copies":5319,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1773,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1773,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15851,"num_write_kb":563968,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":["2","6"],"object_location_counts":[{"shards":"2,6","objects":1773}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.8","version":"74'79","reported_seq":325,"reported_epoch":294,"state":"active+clean","last_fresh":"2024-09-18T08:14:06.576497+0000","last_change":"2024-09-18T08:12:00.980995+0000","last_active":"2024-09-18T08:14:06.576497+0000","last_peered":"2024-09-18T08:14:06.576497+0000","last_clean":"2024-09-18T08:14:06.576497+0000","last_became_active":"2024-09-18T08:07:26.407298+0000","last_became_peered":"2024-09-18T08:07:26.407298+0000","last_unstale":"2024-09-18T08:14:06.576497+0000","last_undegraded":"2024-09-18T08:14:06.576497+0000","last_fullsized":"2024-09-18T08:14:06.576497+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":152,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":4,"last_scrub":"74'79","last_scrub_stamp":"2024-09-18T08:12:00.980917+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:00:11.893742+0000","last_clean_scrub_stamp":"2024-09-18T08:12:00.980917+0000","objects_scrubbed":1,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262688,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,3],"acting":[5,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.d","version":"271'31","reported_seq":281,"reported_epoch":294,"state":"active+undersized+degraded","last_fresh":"2024-09-18T08:14:06.573329+0000","last_change":"2024-09-18T08:11:42.336457+0000","last_active":"2024-09-18T08:14:06.573329+0000","last_peered":"2024-09-18T08:14:06.573329+0000","last_clean":"2024-09-18T08:11:40.307800+0000","last_became_active":"2024-09-18T08:11:42.336457+0000","last_became_peered":"2024-09-18T08:11:42.336457+0000","last_unstale":"2024-09-18T08:14:06.573329+0000","last_undegraded":"2024-09-18T08:11:42.322722+0000","last_fullsized":"2024-09-18T08:11:42.322622+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":182,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'17","last_scrub_stamp":"2024-09-18T08:08:09.671248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_clean_scrub_stamp":"2024-09-18T08:08:09.671248+0000","objects_scrubbed":0,"log_size":31,"log_dups_size":0,"ondisk_log_size":31,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":8,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":49,"num_read_kb":39,"num_write":32,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":["4","2"],"object_location_counts":[{"shards":"2,4","objects":8}],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":250,"reported_epoch":294,"state":"active+undersized","last_fresh":"2024-09-18T08:14:06.571505+0000","last_change":"2024-09-18T08:11:42.323936+0000","last_active":"2024-09-18T08:14:06.571505+0000","last_peered":"2024-09-18T08:14:06.571505+0000","last_clean":"2024-09-18T08:11:40.306980+0000","last_became_active":"2024-09-18T08:11:42.323936+0000","last_became_peered":"2024-09-18T08:11:42.323936+0000","last_unstale":"2024-09-18T08:14:06.571505+0000","last_undegraded":"2024-09-18T08:14:06.571505+0000","last_fullsized":"2024-09-18T08:11:42.319023+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":182,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:10:22.243148+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:46.860092+0000","last_clean_scrub_stamp":"2024-09-18T08:10:22.243148+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":["4","6"],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":374,"reported_epoch":294,"state":"active+undersized","last_fresh":"2024-09-18T08:14:06.572841+0000","last_change":"2024-09-18T08:11:42.331646+0000","last_active":"2024-09-18T08:14:06.572841+0000","last_peered":"2024-09-18T08:14:06.572841+0000","last_clean":"2024-09-18T08:08:15.506244+0000","last_became_active":"2024-09-18T08:11:42.331646+0000","last_became_peered":"2024-09-18T08:11:42.331646+0000","last_unstale":"2024-09-18T08:14:06.572841+0000","last_undegraded":"2024-09-18T08:14:06.572841+0000","last_fullsized":"2024-09-18T08:11:42.318456+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":182,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:09:06.610879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:50.886830+0000","last_clean_scrub_stamp":"2024-09-18T08:09:06.610879+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":["4","0"],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":220,"reported_epoch":294,"state":"active+clean","last_fresh":"2024-09-18T08:14:06.572121+0000","last_change":"2024-09-18T08:11:58.657536+0000","last_active":"2024-09-18T08:14:06.572121+0000","last_peered":"2024-09-18T08:14:06.572121+0000","last_clean":"2024-09-18T08:14:06.572121+0000","last_became_active":"2024-09-18T08:08:52.696300+0000","last_became_peered":"2024-09-18T08:08:52.696300+0000","last_unstale":"2024-09-18T08:14:06.572121+0000","last_undegraded":"2024-09-18T08:14:06.572121+0000","last_fullsized":"2024-09-18T08:14:06.572121+0000","mapping_epoch":198,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":199,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:11:58.657473+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_clean_scrub_stamp":"2024-09-18T08:11:58.657473+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:14:26.327515+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.1","version":"294'15987","reported_seq":17056,"reported_epoch":294,"state":"active+remapped+backfilling","last_fresh":"2024-09-18T08:14:09.099877+0000","last_change":"2024-09-18T08:14:06.818560+0000","last_active":"2024-09-18T08:14:09.099877+0000","last_peered":"2024-09-18T08:14:09.099877+0000","last_clean":"2024-09-18T08:14:04.564805+0000","last_became_active":"2024-09-18T08:14:06.696020+0000","last_became_peered":"2024-09-18T08:14:06.696020+0000","last_unstale":"2024-09-18T08:14:09.099877+0000","last_undegraded":"2024-09-18T08:14:09.099877+0000","last_fullsized":"2024-09-18T08:14:09.099877+0000","mapping_epoch":293,"log_start":"279'12105","ondisk_log_start":"279'12105","created":232,"last_epoch_clean":233,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:11:08.262286+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:11:08.262286+0000","last_clean_scrub_stamp":"2024-09-18T08:11:08.262286+0000","objects_scrubbed":0,"log_size":3882,"log_dups_size":0,"ondisk_log_size":3882,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117374976,"num_objects":1791,"num_object_clones":0,"num_object_copies":5373,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1345,"num_objects_unfound":0,"num_objects_dirty":1791,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15987,"num_write_kb":568896,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":464,"num_bytes_recovered":30408704,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,4],"acting":[3,6,7],"avail_no_missing":["3","6","7"],"object_location_counts":[{"shards":"3,6,7","objects":1791}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.9","version":"74'79","reported_seq":325,"reported_epoch":294,"state":"active+clean","last_fresh":"2024-09-18T08:14:06.575599+0000","last_change":"2024-09-18T08:14:05.595777+0000","last_active":"2024-09-18T08:14:06.575599+0000","last_peered":"2024-09-18T08:14:06.575599+0000","last_clean":"2024-09-18T08:14:06.575599+0000","last_became_active":"2024-09-18T08:14:05.594854+0000","last_became_peered":"2024-09-18T08:14:05.594854+0000","last_unstale":"2024-09-18T08:14:06.575599+0000","last_undegraded":"2024-09-18T08:14:06.575599+0000","last_fullsized":"2024-09-18T08:14:06.575599+0000","mapping_epoch":292,"log_start":"0'0","ondisk_log_start":"0'0","created":152,"last_epoch_clean":293,"parent":"0.0","parent_split_bits":4,"last_scrub":"74'79","last_scrub_stamp":"2024-09-18T08:13:52.510099+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:00:11.893742+0000","last_clean_scrub_stamp":"2024-09-18T08:13:52.510099+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:14:54.465659+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.c","version":"271'20","reported_seq":265,"reported_epoch":294,"state":"active+clean","last_fresh":"2024-09-18T08:14:06.573597+0000","last_change":"2024-09-18T08:08:18.642452+0000","last_active":"2024-09-18T08:14:06.573597+0000","last_peered":"2024-09-18T08:14:06.573597+0000","last_clean":"2024-09-18T08:14:06.573597+0000","last_became_active":"2024-09-18T08:08:18.622980+0000","last_became_peered":"2024-09-18T08:08:18.622980+0000","last_unstale":"2024-09-18T08:14:06.573597+0000","last_undegraded":"2024-09-18T08:14:06.573597+0000","last_fullsized":"2024-09-18T08:14:06.573597+0000","mapping_epoch":181,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":182,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'10","last_scrub_stamp":"2024-09-18T08:05:41.474813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_clean_scrub_stamp":"2024-09-18T08:05:41.474813+0000","objects_scrubbed":0,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:14:47.660985+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":29,"num_read_kb":24,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":527,"reported_epoch":294,"state":"active+clean","last_fresh":"2024-09-18T08:14:06.575698+0000","last_change":"2024-09-18T08:14:04.626151+0000","last_active":"2024-09-18T08:14:06.575698+0000","last_peered":"2024-09-18T08:14:06.575698+0000","last_clean":"2024-09-18T08:14:06.575698+0000","last_became_active":"2024-09-18T08:08:18.542424+0000","last_became_peered":"2024-09-18T08:08:18.542424+0000","last_unstale":"2024-09-18T08:14:06.575698+0000","last_undegraded":"2024-09-18T08:14:06.575698+0000","last_fullsized":"2024-09-18T08:14:06.575698+0000","mapping_epoch":181,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":182,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:14:04.626103+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:46.860092+0000","last_clean_scrub_stamp":"2024-09-18T08:14:04.626103+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:15:13.477322+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.d","version":"293'9","reported_seq":662,"reported_epoch":294,"state":"active+clean","last_fresh":"2024-09-18T08:14:08.063386+0000","last_change":"2024-09-18T08:14:05.605124+0000","last_active":"2024-09-18T08:14:08.063386+0000","last_peered":"2024-09-18T08:14:08.063386+0000","last_clean":"2024-09-18T08:14:08.063386+0000","last_became_active":"2024-09-18T08:14:05.593693+0000","last_became_peered":"2024-09-18T08:14:05.593693+0000","last_unstale":"2024-09-18T08:14:08.063386+0000","last_undegraded":"2024-09-18T08:14:08.063386+0000","last_fullsized":"2024-09-18T08:14:08.063386+0000","mapping_epoch":292,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":293,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'8","last_scrub_stamp":"2024-09-18T08:12:30.371904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:50.886830+0000","last_clean_scrub_stamp":"2024-09-18T08:12:30.371904+0000","objects_scrubbed":1,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,0],"acting":[2,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":257,"reported_epoch":294,"state":"active+undersized","last_fresh":"2024-09-18T08:14:06.576197+0000","last_change":"2024-09-18T08:11:42.337415+0000","last_active":"2024-09-18T08:14:06.576197+0000","last_peered":"2024-09-18T08:14:06.576197+0000","last_clean":"2024-09-18T08:11:40.308045+0000","last_became_active":"2024-09-18T08:11:42.337415+0000","last_became_peered":"2024-09-18T08:11:42.337415+0000","last_unstale":"2024-09-18T08:14:06.576197+0000","last_undegraded":"2024-09-18T08:14:06.576197+0000","last_fullsized":"2024-09-18T08:11:42.322962+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":199,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:11:18.475967+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_clean_scrub_stamp":"2024-09-18T08:11:18.475967+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:14:35.681036+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":["4","2"],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.6","version":"294'15860","reported_seq":15999,"reported_epoch":294,"state":"active+undersized+degraded","last_fresh":"2024-09-18T08:14:08.587921+0000","last_change":"2024-09-18T08:11:42.339935+0000","last_active":"2024-09-18T08:14:08.587921+0000","last_peered":"2024-09-18T08:14:08.587921+0000","last_clean":"2024-09-18T08:11:41.290923+0000","last_became_active":"2024-09-18T08:11:42.339935+0000","last_became_peered":"2024-09-18T08:11:42.339935+0000","last_unstale":"2024-09-18T08:14:08.587921+0000","last_undegraded":"2024-09-18T08:11:42.325251+0000","last_fullsized":"2024-09-18T08:11:42.325122+0000","mapping_epoch":244,"log_start":"276'11488","ondisk_log_start":"276'11488","created":232,"last_epoch_clean":233,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:11:08.262286+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:11:08.262286+0000","last_clean_scrub_stamp":"2024-09-18T08:11:08.262286+0000","objects_scrubbed":0,"log_size":4372,"log_dups_size":0,"ondisk_log_size":4372,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:14:42.147293+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":122290176,"num_objects":1866,"num_object_clones":0,"num_object_copies":5598,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1866,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1866,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15860,"num_write_kb":567232,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":["5","2"],"object_location_counts":[{"shards":"2,5","objects":1866}],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.b","version":"271'15","reported_seq":563,"reported_epoch":294,"state":"active+undersized+degraded","last_fresh":"2024-09-18T08:14:06.580524+0000","last_change":"2024-09-18T08:11:42.333863+0000","last_active":"2024-09-18T08:14:06.580524+0000","last_peered":"2024-09-18T08:14:06.580524+0000","last_clean":"2024-09-18T08:11:40.306474+0000","last_became_active":"2024-09-18T08:11:42.333863+0000","last_became_peered":"2024-09-18T08:11:42.333863+0000","last_unstale":"2024-09-18T08:14:06.580524+0000","last_undegraded":"2024-09-18T08:11:42.319070+0000","last_fullsized":"2024-09-18T08:11:42.318966+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":182,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'9","last_scrub_stamp":"2024-09-18T08:10:47.928596+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_clean_scrub_stamp":"2024-09-18T08:10:47.928596+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:14:38.795239+0000","scrub_duration":16,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":["0","4"],"object_location_counts":[{"shards":"0,4","objects":4}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":518,"reported_epoch":294,"state":"active+clean","last_fresh":"2024-09-18T08:14:06.579594+0000","last_change":"2024-09-18T08:11:22.076505+0000","last_active":"2024-09-18T08:14:06.579594+0000","last_peered":"2024-09-18T08:14:06.579594+0000","last_clean":"2024-09-18T08:14:06.579594+0000","last_became_active":"2024-09-18T08:06:24.135005+0000","last_became_peered":"2024-09-18T08:06:24.135005+0000","last_unstale":"2024-09-18T08:14:06.579594+0000","last_undegraded":"2024-09-18T08:14:06.579594+0000","last_fullsized":"2024-09-18T08:14:06.579594+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:11:22.076452+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:46.860092+0000","last_clean_scrub_stamp":"2024-09-18T08:11:22.076452+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:14:22.230059+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":511,"reported_epoch":294,"state":"active+clean","last_fresh":"2024-09-18T08:14:06.572905+0000","last_change":"2024-09-18T08:08:18.509651+0000","last_active":"2024-09-18T08:14:06.572905+0000","last_peered":"2024-09-18T08:14:06.572905+0000","last_clean":"2024-09-18T08:14:06.572905+0000","last_became_active":"2024-09-18T08:08:18.509247+0000","last_became_peered":"2024-09-18T08:08:18.509247+0000","last_unstale":"2024-09-18T08:14:06.572905+0000","last_undegraded":"2024-09-18T08:14:06.572905+0000","last_fullsized":"2024-09-18T08:14:06.572905+0000","mapping_epoch":181,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":182,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:07:10.728679+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:50.886830+0000","last_clean_scrub_stamp":"2024-09-18T08:07:10.728679+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:14:51.569187+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,3],"acting":[2,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":530,"reported_epoch":294,"state":"active+clean","last_fresh":"2024-09-18T08:14:06.582567+0000","last_change":"2024-09-18T08:14:05.585049+0000","last_active":"2024-09-18T08:14:06.582567+0000","last_peered":"2024-09-18T08:14:06.582567+0000","last_clean":"2024-09-18T08:14:06.582567+0000","last_became_active":"2024-09-18T08:14:05.583887+0000","last_became_peered":"2024-09-18T08:14:05.583887+0000","last_unstale":"2024-09-18T08:14:06.582567+0000","last_undegraded":"2024-09-18T08:14:06.582567+0000","last_fullsized":"2024-09-18T08:14:06.582567+0000","mapping_epoch":292,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":293,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:12:38.702798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_clean_scrub_stamp":"2024-09-18T08:12:38.702798+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,3],"acting":[0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.7","version":"294'15597","reported_seq":15767,"reported_epoch":294,"state":"active+remapped+backfill_toofull","last_fresh":"2024-09-18T08:14:08.597943+0000","last_change":"2024-09-18T08:14:06.733359+0000","last_active":"2024-09-18T08:14:08.597943+0000","last_peered":"2024-09-18T08:14:08.597943+0000","last_clean":"2024-09-18T08:14:04.566476+0000","last_became_active":"2024-09-18T08:14:06.702584+0000","last_became_peered":"2024-09-18T08:14:06.702584+0000","last_unstale":"2024-09-18T08:14:08.597943+0000","last_undegraded":"2024-09-18T08:14:08.597943+0000","last_fullsized":"2024-09-18T08:14:08.597943+0000","mapping_epoch":293,"log_start":"276'11241","ondisk_log_start":"276'11241","created":232,"last_epoch_clean":233,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:11:08.262286+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:11:08.262286+0000","last_clean_scrub_stamp":"2024-09-18T08:11:08.262286+0000","objects_scrubbed":2599,"log_size":4356,"log_dups_size":0,"ondisk_log_size":4356,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":121569280,"num_objects":1855,"num_object_clones":0,"num_object_copies":5565,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1855,"num_objects_unfound":0,"num_objects_dirty":1855,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15597,"num_write_kb":558464,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,7],"avail_no_missing":["5","3","7"],"object_location_counts":[{"shards":"3,5,7","objects":1855}],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.a","version":"271'35","reported_seq":33,"reported_epoch":294,"state":"active+clean+remapped","last_fresh":"2024-09-18T08:14:06.629038+0000","last_change":"2024-09-18T08:14:06.628451+0000","last_active":"2024-09-18T08:14:06.629038+0000","last_peered":"2024-09-18T08:14:06.629038+0000","last_clean":"2024-09-18T08:14:06.629038+0000","last_became_active":"2024-09-18T08:14:06.608845+0000","last_became_peered":"2024-09-18T08:14:06.608845+0000","last_unstale":"2024-09-18T08:14:06.629038+0000","last_undegraded":"2024-09-18T08:14:06.629038+0000","last_fullsized":"2024-09-18T08:14:06.629038+0000","mapping_epoch":293,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":294,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'19","last_scrub_stamp":"2024-09-18T08:10:19.603091+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_clean_scrub_stamp":"2024-09-18T08:10:19.603091+0000","objects_scrubbed":0,"log_size":35,"log_dups_size":0,"ondisk_log_size":35,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":9,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":55,"num_read_kb":44,"num_write":36,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6,7],"avail_no_missing":["5","6","7"],"object_location_counts":[{"shards":"5,6,7","objects":9}],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":521,"reported_epoch":294,"state":"active+clean","last_fresh":"2024-09-18T08:14:06.582181+0000","last_change":"2024-09-18T08:14:05.605090+0000","last_active":"2024-09-18T08:14:06.582181+0000","last_peered":"2024-09-18T08:14:06.582181+0000","last_clean":"2024-09-18T08:14:06.582181+0000","last_became_active":"2024-09-18T08:14:05.604198+0000","last_became_peered":"2024-09-18T08:14:05.604198+0000","last_unstale":"2024-09-18T08:14:06.582181+0000","last_undegraded":"2024-09-18T08:14:06.582181+0000","last_fullsized":"2024-09-18T08:14:06.582181+0000","mapping_epoch":292,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":293,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:12:31.205468+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:46.860092+0000","last_clean_scrub_stamp":"2024-09-18T08:12:31.205468+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,3],"acting":[5,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":212,"reported_epoch":294,"state":"active+clean","last_fresh":"2024-09-18T08:14:06.571142+0000","last_change":"2024-09-18T08:10:41.682366+0000","last_active":"2024-09-18T08:14:06.571142+0000","last_peered":"2024-09-18T08:14:06.571142+0000","last_clean":"2024-09-18T08:14:06.571142+0000","last_became_active":"2024-09-18T08:08:52.697932+0000","last_became_peered":"2024-09-18T08:08:52.697932+0000","last_unstale":"2024-09-18T08:14:06.571142+0000","last_undegraded":"2024-09-18T08:14:06.571142+0000","last_fullsized":"2024-09-18T08:14:06.571142+0000","mapping_epoch":198,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":199,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:10:41.682297+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:50.886830+0000","last_clean_scrub_stamp":"2024-09-18T08:10:41.682297+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:14:19.387756+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":339,"reported_epoch":294,"state":"active+clean","last_fresh":"2024-09-18T08:14:06.572087+0000","last_change":"2024-09-18T08:14:05.578772+0000","last_active":"2024-09-18T08:14:06.572087+0000","last_peered":"2024-09-18T08:14:06.572087+0000","last_clean":"2024-09-18T08:14:06.572087+0000","last_became_active":"2024-09-18T08:14:05.578053+0000","last_became_peered":"2024-09-18T08:14:05.578053+0000","last_unstale":"2024-09-18T08:14:06.572087+0000","last_undegraded":"2024-09-18T08:14:06.572087+0000","last_fullsized":"2024-09-18T08:14:06.572087+0000","mapping_epoch":292,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":293,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:11:31.707130+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_clean_scrub_stamp":"2024-09-18T08:11:31.707130+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.4","version":"294'15939","reported_seq":16080,"reported_epoch":294,"state":"active+undersized+degraded","last_fresh":"2024-09-18T08:14:08.356434+0000","last_change":"2024-09-18T08:11:42.349543+0000","last_active":"2024-09-18T08:14:08.356434+0000","last_peered":"2024-09-18T08:14:08.356434+0000","last_clean":"2024-09-18T08:11:41.314761+0000","last_became_active":"2024-09-18T08:11:42.329000+0000","last_became_peered":"2024-09-18T08:11:42.329000+0000","last_unstale":"2024-09-18T08:14:08.356434+0000","last_undegraded":"2024-09-18T08:11:42.318915+0000","last_fullsized":"2024-09-18T08:11:42.318397+0000","mapping_epoch":244,"log_start":"277'11614","ondisk_log_start":"277'11614","created":232,"last_epoch_clean":233,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:11:08.262286+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:11:08.262286+0000","last_clean_scrub_stamp":"2024-09-18T08:11:08.262286+0000","objects_scrubbed":0,"log_size":4325,"log_dups_size":0,"ondisk_log_size":4325,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:14:37.598862+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":116981760,"num_objects":1785,"num_object_clones":0,"num_object_copies":5355,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1785,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1785,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15939,"num_write_kb":567168,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":65536,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":["4","3"],"object_location_counts":[{"shards":"3,4","objects":1785}],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.9","version":"271'18","reported_seq":286,"reported_epoch":294,"state":"active+clean","last_fresh":"2024-09-18T08:14:06.571173+0000","last_change":"2024-09-18T08:11:27.607131+0000","last_active":"2024-09-18T08:14:06.571173+0000","last_peered":"2024-09-18T08:14:06.571173+0000","last_clean":"2024-09-18T08:14:06.571173+0000","last_became_active":"2024-09-18T08:10:18.362984+0000","last_became_peered":"2024-09-18T08:10:18.362984+0000","last_unstale":"2024-09-18T08:14:06.571173+0000","last_undegraded":"2024-09-18T08:14:06.571173+0000","last_fullsized":"2024-09-18T08:14:06.571173+0000","mapping_epoch":221,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":222,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'12","last_scrub_stamp":"2024-09-18T08:11:27.607049+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_clean_scrub_stamp":"2024-09-18T08:11:27.607049+0000","objects_scrubbed":5,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":33,"num_read_kb":24,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":660,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,3],"acting":[4,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":445,"reported_epoch":294,"state":"active+clean+remapped","last_fresh":"2024-09-18T08:14:06.593245+0000","last_change":"2024-09-18T08:14:06.592515+0000","last_active":"2024-09-18T08:14:06.593245+0000","last_peered":"2024-09-18T08:14:06.593245+0000","last_clean":"2024-09-18T08:14:06.593245+0000","last_became_active":"2024-09-18T08:14:06.591911+0000","last_became_peered":"2024-09-18T08:14:06.591911+0000","last_unstale":"2024-09-18T08:14:06.593245+0000","last_undegraded":"2024-09-18T08:14:06.593245+0000","last_fullsized":"2024-09-18T08:14:06.593245+0000","mapping_epoch":293,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":294,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:10:57.579577+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:46.860092+0000","last_clean_scrub_stamp":"2024-09-18T08:10:57.579577+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0,7],"avail_no_missing":["4","0","7"],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":501,"reported_epoch":294,"state":"active+undersized","last_fresh":"2024-09-18T08:14:06.572188+0000","last_change":"2024-09-18T08:11:42.325714+0000","last_active":"2024-09-18T08:14:06.572188+0000","last_peered":"2024-09-18T08:14:06.572188+0000","last_clean":"2024-09-18T08:11:40.310260+0000","last_became_active":"2024-09-18T08:11:42.325714+0000","last_became_peered":"2024-09-18T08:11:42.325714+0000","last_unstale":"2024-09-18T08:14:06.572188+0000","last_undegraded":"2024-09-18T08:14:06.572188+0000","last_fullsized":"2024-09-18T08:11:42.318417+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:10:47.799791+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:50.886830+0000","last_clean_scrub_stamp":"2024-09-18T08:10:47.799791+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:14:20.377540+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":["2","0"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":512,"reported_epoch":294,"state":"active+clean+remapped","last_fresh":"2024-09-18T08:14:06.601319+0000","last_change":"2024-09-18T08:14:06.600683+0000","last_active":"2024-09-18T08:14:06.601319+0000","last_peered":"2024-09-18T08:14:06.601319+0000","last_clean":"2024-09-18T08:14:06.601319+0000","last_became_active":"2024-09-18T08:14:06.599711+0000","last_became_peered":"2024-09-18T08:14:06.599711+0000","last_unstale":"2024-09-18T08:14:06.601319+0000","last_undegraded":"2024-09-18T08:14:06.601319+0000","last_fullsized":"2024-09-18T08:14:06.601319+0000","mapping_epoch":293,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":294,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:11:21.372405+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_clean_scrub_stamp":"2024-09-18T08:11:21.372405+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6,7],"avail_no_missing":["3","6","7"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.5","version":"294'15887","reported_seq":16064,"reported_epoch":294,"state":"active+clean+remapped","last_fresh":"2024-09-18T08:14:09.049872+0000","last_change":"2024-09-18T08:14:06.672588+0000","last_active":"2024-09-18T08:14:09.049872+0000","last_peered":"2024-09-18T08:14:09.049872+0000","last_clean":"2024-09-18T08:14:09.049872+0000","last_became_active":"2024-09-18T08:14:06.612591+0000","last_became_peered":"2024-09-18T08:14:06.612591+0000","last_unstale":"2024-09-18T08:14:09.049872+0000","last_undegraded":"2024-09-18T08:14:09.049872+0000","last_fullsized":"2024-09-18T08:14:09.049872+0000","mapping_epoch":293,"log_start":"276'11330","ondisk_log_start":"276'11330","created":232,"last_epoch_clean":294,"parent":"0.0","parent_split_bits":0,"last_scrub":"291'15284","last_scrub_stamp":"2024-09-18T08:13:59.720022+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:11:08.262286+0000","last_clean_scrub_stamp":"2024-09-18T08:13:59.720022+0000","objects_scrubbed":2451,"log_size":4557,"log_dups_size":0,"ondisk_log_size":4557,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:15:04.349850+0000","scrub_duration":876,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":119209984,"num_objects":1819,"num_object_clones":0,"num_object_copies":5457,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1819,"num_objects_unfound":0,"num_objects_dirty":1819,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15887,"num_write_kb":566592,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5,7],"avail_no_missing":["0","5","7"],"object_location_counts":[{"shards":"0,5,7","objects":1819}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.8","version":"271'27","reported_seq":596,"reported_epoch":294,"state":"active+clean","last_fresh":"2024-09-18T08:14:06.576712+0000","last_change":"2024-09-18T08:14:05.614629+0000","last_active":"2024-09-18T08:14:06.576712+0000","last_peered":"2024-09-18T08:14:06.576712+0000","last_clean":"2024-09-18T08:14:06.576712+0000","last_became_active":"2024-09-18T08:14:05.597891+0000","last_became_peered":"2024-09-18T08:14:05.597891+0000","last_unstale":"2024-09-18T08:14:06.576712+0000","last_undegraded":"2024-09-18T08:14:06.576712+0000","last_fullsized":"2024-09-18T08:14:06.576712+0000","mapping_epoch":292,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":293,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'15","last_scrub_stamp":"2024-09-18T08:04:26.859044+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_clean_scrub_stamp":"2024-09-18T08:04:26.859044+0000","objects_scrubbed":7,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":44,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,4],"acting":[5,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.f","version":"62'2","reported_seq":313,"reported_epoch":294,"state":"active+clean+remapped","last_fresh":"2024-09-18T08:14:06.584611+0000","last_change":"2024-09-18T08:14:06.583629+0000","last_active":"2024-09-18T08:14:06.584611+0000","last_peered":"2024-09-18T08:14:06.584611+0000","last_clean":"2024-09-18T08:14:06.584611+0000","last_became_active":"2024-09-18T08:14:06.583000+0000","last_became_peered":"2024-09-18T08:14:06.583000+0000","last_unstale":"2024-09-18T08:14:06.584611+0000","last_undegraded":"2024-09-18T08:14:06.584611+0000","last_fullsized":"2024-09-18T08:14:06.584611+0000","mapping_epoch":293,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":294,"parent":"0.0","parent_split_bits":0,"last_scrub":"62'2","last_scrub_stamp":"2024-09-18T08:10:53.580444+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:46.860092+0000","last_clean_scrub_stamp":"2024-09-18T08:10:53.580444+0000","objects_scrubbed":2,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":4,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":92,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0,7],"avail_no_missing":["4","0","7"],"object_location_counts":[{"shards":"0,4,7","objects":2}],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.9","version":"294'14","reported_seq":394,"reported_epoch":294,"state":"active+clean+remapped","last_fresh":"2024-09-18T08:14:08.068861+0000","last_change":"2024-09-18T08:14:06.586247+0000","last_active":"2024-09-18T08:14:08.068861+0000","last_peered":"2024-09-18T08:14:08.068861+0000","last_clean":"2024-09-18T08:14:08.068861+0000","last_became_active":"2024-09-18T08:14:06.585737+0000","last_became_peered":"2024-09-18T08:14:06.585737+0000","last_unstale":"2024-09-18T08:14:08.068861+0000","last_undegraded":"2024-09-18T08:14:08.068861+0000","last_fullsized":"2024-09-18T08:14:08.068861+0000","mapping_epoch":293,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":294,"parent":"0.0","parent_split_bits":0,"last_scrub":"182'12","last_scrub_stamp":"2024-09-18T08:09:46.742442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:50.886830+0000","last_clean_scrub_stamp":"2024-09-18T08:09:46.742442+0000","objects_scrubbed":1,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6,7],"avail_no_missing":["4","6","7"],"object_location_counts":[{"shards":"4,6,7","objects":1}],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":506,"reported_epoch":294,"state":"active+clean","last_fresh":"2024-09-18T08:14:06.580386+0000","last_change":"2024-09-18T08:11:23.050828+0000","last_active":"2024-09-18T08:14:06.580386+0000","last_peered":"2024-09-18T08:14:06.580386+0000","last_clean":"2024-09-18T08:14:06.580386+0000","last_became_active":"2024-09-18T08:06:24.133119+0000","last_became_peered":"2024-09-18T08:06:24.133119+0000","last_unstale":"2024-09-18T08:14:06.580386+0000","last_undegraded":"2024-09-18T08:14:06.580386+0000","last_fullsized":"2024-09-18T08:14:06.580386+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:11:23.050755+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_clean_scrub_stamp":"2024-09-18T08:11:23.050755+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:14:24.216573+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":430,"reported_epoch":294,"state":"active+clean","last_fresh":"2024-09-18T08:14:06.577819+0000","last_change":"2024-09-18T08:13:06.860401+0000","last_active":"2024-09-18T08:14:06.577819+0000","last_peered":"2024-09-18T08:14:06.577819+0000","last_clean":"2024-09-18T08:14:06.577819+0000","last_became_active":"2024-09-18T08:06:24.133644+0000","last_became_peered":"2024-09-18T08:06:24.133644+0000","last_unstale":"2024-09-18T08:14:06.577819+0000","last_undegraded":"2024-09-18T08:14:06.577819+0000","last_fullsized":"2024-09-18T08:14:06.577819+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:13:06.860320+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:46.860092+0000","last_clean_scrub_stamp":"2024-09-18T08:13:06.860320+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:14:07.559037+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.17","version":"271'12","reported_seq":544,"reported_epoch":294,"state":"active+clean+remapped","last_fresh":"2024-09-18T08:14:06.601492+0000","last_change":"2024-09-18T08:14:06.601345+0000","last_active":"2024-09-18T08:14:06.601492+0000","last_peered":"2024-09-18T08:14:06.601492+0000","last_clean":"2024-09-18T08:14:06.601492+0000","last_became_active":"2024-09-18T08:14:06.590751+0000","last_became_peered":"2024-09-18T08:14:06.590751+0000","last_unstale":"2024-09-18T08:14:06.601492+0000","last_undegraded":"2024-09-18T08:14:06.601492+0000","last_fullsized":"2024-09-18T08:14:06.601492+0000","mapping_epoch":293,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":294,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'6","last_scrub_stamp":"2024-09-18T08:10:15.515914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_clean_scrub_stamp":"2024-09-18T08:10:15.515914+0000","objects_scrubbed":3,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":18,"num_read_kb":15,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2,7],"avail_no_missing":["3","2","7"],"object_location_counts":[{"shards":"2,3,7","objects":3}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":499,"reported_epoch":294,"state":"active+undersized","last_fresh":"2024-09-18T08:14:06.584136+0000","last_change":"2024-09-18T08:11:42.331518+0000","last_active":"2024-09-18T08:14:06.584136+0000","last_peered":"2024-09-18T08:14:06.584136+0000","last_clean":"2024-09-18T08:11:40.302611+0000","last_became_active":"2024-09-18T08:11:42.331518+0000","last_became_peered":"2024-09-18T08:11:42.331518+0000","last_unstale":"2024-09-18T08:14:06.584136+0000","last_undegraded":"2024-09-18T08:14:06.584136+0000","last_fullsized":"2024-09-18T08:11:42.328308+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:10:49.169375+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:50.886830+0000","last_clean_scrub_stamp":"2024-09-18T08:10:49.169375+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:14:39.180443+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":["5","3"],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":299,"reported_epoch":294,"state":"active+clean","last_fresh":"2024-09-18T08:14:06.572603+0000","last_change":"2024-09-18T08:14:05.596154+0000","last_active":"2024-09-18T08:14:06.572603+0000","last_peered":"2024-09-18T08:14:06.572603+0000","last_clean":"2024-09-18T08:14:06.572603+0000","last_became_active":"2024-09-18T08:14:05.595218+0000","last_became_peered":"2024-09-18T08:14:05.595218+0000","last_unstale":"2024-09-18T08:14:06.572603+0000","last_undegraded":"2024-09-18T08:14:06.572603+0000","last_fullsized":"2024-09-18T08:14:06.572603+0000","mapping_epoch":292,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":293,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:12:54.661578+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_clean_scrub_stamp":"2024-09-18T08:12:54.661578+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:14:15.269642+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.16","version":"271'15","reported_seq":541,"reported_epoch":294,"state":"active+clean","last_fresh":"2024-09-18T08:14:06.578715+0000","last_change":"2024-09-18T08:14:05.588589+0000","last_active":"2024-09-18T08:14:06.578715+0000","last_peered":"2024-09-18T08:14:06.578715+0000","last_clean":"2024-09-18T08:14:06.578715+0000","last_became_active":"2024-09-18T08:14:05.578519+0000","last_became_peered":"2024-09-18T08:14:05.578519+0000","last_unstale":"2024-09-18T08:14:06.578715+0000","last_undegraded":"2024-09-18T08:14:06.578715+0000","last_fullsized":"2024-09-18T08:14:06.578715+0000","mapping_epoch":292,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":293,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'9","last_scrub_stamp":"2024-09-18T08:09:44.116308+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_clean_scrub_stamp":"2024-09-18T08:09:44.116308+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":43,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":261,"reported_epoch":294,"state":"active+clean","last_fresh":"2024-09-18T08:14:06.571713+0000","last_change":"2024-09-18T08:14:05.583783+0000","last_active":"2024-09-18T08:14:06.571713+0000","last_peered":"2024-09-18T08:14:06.571713+0000","last_clean":"2024-09-18T08:14:06.571713+0000","last_became_active":"2024-09-18T08:14:05.583435+0000","last_became_peered":"2024-09-18T08:14:05.583435+0000","last_unstale":"2024-09-18T08:14:06.571713+0000","last_undegraded":"2024-09-18T08:14:06.571713+0000","last_fullsized":"2024-09-18T08:14:06.571713+0000","mapping_epoch":292,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":293,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:12:03.433407+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:46.860092+0000","last_clean_scrub_stamp":"2024-09-18T08:12:03.433407+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,0],"acting":[4,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":506,"reported_epoch":294,"state":"active+clean+remapped","last_fresh":"2024-09-18T08:14:06.590203+0000","last_change":"2024-09-18T08:14:06.589074+0000","last_active":"2024-09-18T08:14:06.590203+0000","last_peered":"2024-09-18T08:14:06.590203+0000","last_clean":"2024-09-18T08:14:06.590203+0000","last_became_active":"2024-09-18T08:14:06.588521+0000","last_became_peered":"2024-09-18T08:14:06.588521+0000","last_unstale":"2024-09-18T08:14:06.590203+0000","last_undegraded":"2024-09-18T08:14:06.590203+0000","last_fullsized":"2024-09-18T08:14:06.590203+0000","mapping_epoch":293,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":294,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:11:17.424701+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:50.886830+0000","last_clean_scrub_stamp":"2024-09-18T08:11:17.424701+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0,7],"avail_no_missing":["3","0","7"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":514,"reported_epoch":294,"state":"active+clean","last_fresh":"2024-09-18T08:14:06.573628+0000","last_change":"2024-09-18T08:14:05.582569+0000","last_active":"2024-09-18T08:14:06.573628+0000","last_peered":"2024-09-18T08:14:06.573628+0000","last_clean":"2024-09-18T08:14:06.573628+0000","last_became_active":"2024-09-18T08:14:05.582226+0000","last_became_peered":"2024-09-18T08:14:05.582226+0000","last_unstale":"2024-09-18T08:14:06.573628+0000","last_undegraded":"2024-09-18T08:14:06.573628+0000","last_fullsized":"2024-09-18T08:14:06.573628+0000","mapping_epoch":292,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":293,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:10:42.731845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_clean_scrub_stamp":"2024-09-18T08:10:42.731845+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,0],"acting":[2,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.15","version":"271'15","reported_seq":541,"reported_epoch":294,"state":"active+clean","last_fresh":"2024-09-18T08:14:06.583782+0000","last_change":"2024-09-18T08:14:05.612424+0000","last_active":"2024-09-18T08:14:06.583782+0000","last_peered":"2024-09-18T08:14:06.583782+0000","last_clean":"2024-09-18T08:14:06.583782+0000","last_became_active":"2024-09-18T08:14:05.603660+0000","last_became_peered":"2024-09-18T08:14:05.603660+0000","last_unstale":"2024-09-18T08:14:06.583782+0000","last_undegraded":"2024-09-18T08:14:06.583782+0000","last_fullsized":"2024-09-18T08:14:06.583782+0000","mapping_epoch":292,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":293,"parent":"0.0","parent_split_bits":0,"last_scrub":"271'15","last_scrub_stamp":"2024-09-18T08:12:58.225295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_clean_scrub_stamp":"2024-09-18T08:12:58.225295+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":517,"reported_epoch":294,"state":"active+clean+remapped","last_fresh":"2024-09-18T08:14:06.583263+0000","last_change":"2024-09-18T08:14:06.583119+0000","last_active":"2024-09-18T08:14:06.583263+0000","last_peered":"2024-09-18T08:14:06.583263+0000","last_clean":"2024-09-18T08:14:06.583263+0000","last_became_active":"2024-09-18T08:14:06.582708+0000","last_became_peered":"2024-09-18T08:14:06.582708+0000","last_unstale":"2024-09-18T08:14:06.583263+0000","last_undegraded":"2024-09-18T08:14:06.583263+0000","last_fullsized":"2024-09-18T08:14:06.583263+0000","mapping_epoch":293,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":294,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:09:46.035332+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:46.860092+0000","last_clean_scrub_stamp":"2024-09-18T08:09:46.035332+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":18,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3,7],"avail_no_missing":["0","3","7"],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.14","version":"294'14","reported_seq":689,"reported_epoch":294,"state":"active+clean+remapped","last_fresh":"2024-09-18T08:14:06.623060+0000","last_change":"2024-09-18T08:14:06.605581+0000","last_active":"2024-09-18T08:14:06.623060+0000","last_peered":"2024-09-18T08:14:06.623060+0000","last_clean":"2024-09-18T08:14:06.623060+0000","last_became_active":"2024-09-18T08:14:06.605082+0000","last_became_peered":"2024-09-18T08:14:06.605082+0000","last_unstale":"2024-09-18T08:14:06.623060+0000","last_undegraded":"2024-09-18T08:14:06.623060+0000","last_fullsized":"2024-09-18T08:14:06.623060+0000","mapping_epoch":293,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":294,"parent":"0.0","parent_split_bits":0,"last_scrub":"199'12","last_scrub_stamp":"2024-09-18T08:13:31.695354+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:50.886830+0000","last_clean_scrub_stamp":"2024-09-18T08:13:31.695354+0000","objects_scrubbed":1,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:15:01.521230+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2,7],"avail_no_missing":["3","2","7"],"object_location_counts":[{"shards":"2,3,7","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":246,"reported_epoch":294,"state":"active+clean","last_fresh":"2024-09-18T08:14:06.572640+0000","last_change":"2024-09-18T08:12:06.521847+0000","last_active":"2024-09-18T08:14:06.572640+0000","last_peered":"2024-09-18T08:14:06.572640+0000","last_clean":"2024-09-18T08:14:06.572640+0000","last_became_active":"2024-09-18T08:08:18.522483+0000","last_became_peered":"2024-09-18T08:08:18.522483+0000","last_unstale":"2024-09-18T08:14:06.572640+0000","last_undegraded":"2024-09-18T08:14:06.572640+0000","last_fullsized":"2024-09-18T08:14:06.572640+0000","mapping_epoch":181,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":182,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:12:06.521774+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_clean_scrub_stamp":"2024-09-18T08:12:06.521774+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.14","version":"271'20","reported_seq":555,"reported_epoch":294,"state":"active+clean+remapped","last_fresh":"2024-09-18T08:14:06.606962+0000","last_change":"2024-09-18T08:14:06.606798+0000","last_active":"2024-09-18T08:14:06.606962+0000","last_peered":"2024-09-18T08:14:06.606962+0000","last_clean":"2024-09-18T08:14:06.606962+0000","last_became_active":"2024-09-18T08:14:06.605206+0000","last_became_peered":"2024-09-18T08:14:06.605206+0000","last_unstale":"2024-09-18T08:14:06.606962+0000","last_undegraded":"2024-09-18T08:14:06.606962+0000","last_fullsized":"2024-09-18T08:14:06.606962+0000","mapping_epoch":293,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":294,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'10","last_scrub_stamp":"2024-09-18T08:12:39.946462+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_clean_scrub_stamp":"2024-09-18T08:12:39.946462+0000","objects_scrubbed":5,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":5,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":25,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2,7],"avail_no_missing":["3","2","7"],"object_location_counts":[{"shards":"2,3,7","objects":5}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":288,"reported_epoch":294,"state":"active+clean","last_fresh":"2024-09-18T08:14:06.572079+0000","last_change":"2024-09-18T08:14:05.588122+0000","last_active":"2024-09-18T08:14:06.572079+0000","last_peered":"2024-09-18T08:14:06.572079+0000","last_clean":"2024-09-18T08:14:06.572079+0000","last_became_active":"2024-09-18T08:14:05.587338+0000","last_became_peered":"2024-09-18T08:14:05.587338+0000","last_unstale":"2024-09-18T08:14:06.572079+0000","last_undegraded":"2024-09-18T08:14:06.572079+0000","last_fullsized":"2024-09-18T08:14:06.572079+0000","mapping_epoch":292,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":293,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:13:32.933940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:46.860092+0000","last_clean_scrub_stamp":"2024-09-18T08:13:32.933940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:14:58.839481+0000","scrub_duration":21,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,6],"acting":[4,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.15","version":"245'9","reported_seq":653,"reported_epoch":294,"state":"active+undersized+degraded","last_fresh":"2024-09-18T08:14:06.581809+0000","last_change":"2024-09-18T08:11:42.329955+0000","last_active":"2024-09-18T08:14:06.581809+0000","last_peered":"2024-09-18T08:14:06.581809+0000","last_clean":"2024-09-18T08:11:40.302227+0000","last_became_active":"2024-09-18T08:11:42.329955+0000","last_became_peered":"2024-09-18T08:11:42.329955+0000","last_unstale":"2024-09-18T08:14:06.581809+0000","last_undegraded":"2024-09-18T08:11:42.324112+0000","last_fullsized":"2024-09-18T08:11:42.324002+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'8","last_scrub_stamp":"2024-09-18T08:10:54.148986+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:50.886830+0000","last_clean_scrub_stamp":"2024-09-18T08:10:54.148986+0000","objects_scrubbed":1,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:14:53.115683+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":["5","0"],"object_location_counts":[{"shards":"0,5","objects":1}],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":499,"reported_epoch":294,"state":"active+clean","last_fresh":"2024-09-18T08:14:06.578252+0000","last_change":"2024-09-18T08:14:05.590487+0000","last_active":"2024-09-18T08:14:06.578252+0000","last_peered":"2024-09-18T08:14:06.578252+0000","last_clean":"2024-09-18T08:14:06.578252+0000","last_became_active":"2024-09-18T08:14:05.590147+0000","last_became_peered":"2024-09-18T08:14:05.590147+0000","last_unstale":"2024-09-18T08:14:06.578252+0000","last_undegraded":"2024-09-18T08:14:06.578252+0000","last_fullsized":"2024-09-18T08:14:06.578252+0000","mapping_epoch":292,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":293,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:09:42.047270+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_clean_scrub_stamp":"2024-09-18T08:09:42.047270+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.13","version":"271'19","reported_seq":274,"reported_epoch":294,"state":"active+undersized+degraded","last_fresh":"2024-09-18T08:14:06.572147+0000","last_change":"2024-09-18T08:11:42.324665+0000","last_active":"2024-09-18T08:14:06.572147+0000","last_peered":"2024-09-18T08:14:06.572147+0000","last_clean":"2024-09-18T08:11:40.307120+0000","last_became_active":"2024-09-18T08:11:42.324665+0000","last_became_peered":"2024-09-18T08:11:42.324665+0000","last_unstale":"2024-09-18T08:14:06.572147+0000","last_undegraded":"2024-09-18T08:11:42.319891+0000","last_fullsized":"2024-09-18T08:11:42.319779+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":182,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'11","last_scrub_stamp":"2024-09-18T08:11:22.565415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_clean_scrub_stamp":"2024-09-18T08:11:22.565415+0000","objects_scrubbed":5,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:14:40.574453+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":5,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":31,"num_read_kb":24,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":["4","6"],"object_location_counts":[{"shards":"4,6","objects":5}],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":253,"reported_epoch":294,"state":"active+clean","last_fresh":"2024-09-18T08:14:06.573724+0000","last_change":"2024-09-18T08:14:05.582592+0000","last_active":"2024-09-18T08:14:06.573724+0000","last_peered":"2024-09-18T08:14:06.573724+0000","last_clean":"2024-09-18T08:14:06.573724+0000","last_became_active":"2024-09-18T08:14:05.581338+0000","last_became_peered":"2024-09-18T08:14:05.581338+0000","last_unstale":"2024-09-18T08:14:06.573724+0000","last_undegraded":"2024-09-18T08:14:06.573724+0000","last_fullsized":"2024-09-18T08:14:06.573724+0000","mapping_epoch":292,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":293,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:10:58.402308+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:46.860092+0000","last_clean_scrub_stamp":"2024-09-18T08:10:58.402308+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,2],"acting":[4,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":109,"reported_epoch":294,"state":"active+undersized","last_fresh":"2024-09-18T08:14:06.580729+0000","last_change":"2024-09-18T08:11:42.332003+0000","last_active":"2024-09-18T08:14:06.580729+0000","last_peered":"2024-09-18T08:14:06.580729+0000","last_clean":"2024-09-18T08:02:50.886830+0000","last_became_active":"2024-09-18T08:11:42.332003+0000","last_became_peered":"2024-09-18T08:11:42.332003+0000","last_unstale":"2024-09-18T08:14:06.580729+0000","last_undegraded":"2024-09-18T08:14:06.580729+0000","last_fullsized":"2024-09-18T08:11:42.330392+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:11:26.316948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:50.886830+0000","last_clean_scrub_stamp":"2024-09-18T08:11:26.316948+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":["5","3"],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":509,"reported_epoch":294,"state":"active+clean","last_fresh":"2024-09-18T08:14:06.573120+0000","last_change":"2024-09-18T08:13:28.680281+0000","last_active":"2024-09-18T08:14:06.573120+0000","last_peered":"2024-09-18T08:14:06.573120+0000","last_clean":"2024-09-18T08:14:06.573120+0000","last_became_active":"2024-09-18T08:02:53.911429+0000","last_became_peered":"2024-09-18T08:02:53.911429+0000","last_unstale":"2024-09-18T08:14:06.573120+0000","last_undegraded":"2024-09-18T08:14:06.573120+0000","last_fullsized":"2024-09-18T08:14:06.573120+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:13:28.680237+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_clean_scrub_stamp":"2024-09-18T08:13:28.680237+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:14:29.890119+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.12","version":"271'15","reported_seq":154,"reported_epoch":294,"state":"active+undersized+degraded","last_fresh":"2024-09-18T08:14:06.572990+0000","last_change":"2024-09-18T08:11:42.329218+0000","last_active":"2024-09-18T08:14:06.572990+0000","last_peered":"2024-09-18T08:14:06.572990+0000","last_clean":"2024-09-18T08:02:54.300830+0000","last_became_active":"2024-09-18T08:11:42.329218+0000","last_became_peered":"2024-09-18T08:11:42.329218+0000","last_unstale":"2024-09-18T08:14:06.572990+0000","last_undegraded":"2024-09-18T08:11:42.318592+0000","last_fullsized":"2024-09-18T08:11:42.318478+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'9","last_scrub_stamp":"2024-09-18T08:10:06.582804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_clean_scrub_stamp":"2024-09-18T08:10:06.582804+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:15:03.644094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":25,"num_read_kb":19,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":["2","6"],"object_location_counts":[{"shards":"2,6","objects":4}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":303,"reported_epoch":294,"state":"active+clean","last_fresh":"2024-09-18T08:14:06.575844+0000","last_change":"2024-09-18T08:14:05.581553+0000","last_active":"2024-09-18T08:14:06.575844+0000","last_peered":"2024-09-18T08:14:06.575844+0000","last_clean":"2024-09-18T08:14:06.575844+0000","last_became_active":"2024-09-18T08:14:05.581038+0000","last_became_peered":"2024-09-18T08:14:05.581038+0000","last_unstale":"2024-09-18T08:14:06.575844+0000","last_undegraded":"2024-09-18T08:14:06.575844+0000","last_fullsized":"2024-09-18T08:14:06.575844+0000","mapping_epoch":292,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":293,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:13:35.000383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:46.860092+0000","last_clean_scrub_stamp":"2024-09-18T08:13:35.000383+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:14:35.482327+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,0],"acting":[3,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":500,"reported_epoch":294,"state":"active+undersized","last_fresh":"2024-09-18T08:14:06.574299+0000","last_change":"2024-09-18T08:11:42.328480+0000","last_active":"2024-09-18T08:14:06.574299+0000","last_peered":"2024-09-18T08:14:06.574299+0000","last_clean":"2024-09-18T08:11:40.314825+0000","last_became_active":"2024-09-18T08:11:42.328480+0000","last_became_peered":"2024-09-18T08:11:42.328480+0000","last_unstale":"2024-09-18T08:14:06.574299+0000","last_undegraded":"2024-09-18T08:14:06.574299+0000","last_fullsized":"2024-09-18T08:11:42.323648+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:11:30.302179+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:50.886830+0000","last_clean_scrub_stamp":"2024-09-18T08:11:30.302179+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:14:16.797973+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":["3","0"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.10","version":"69'1","reported_seq":484,"reported_epoch":294,"state":"active+undersized+degraded","last_fresh":"2024-09-18T08:14:06.577215+0000","last_change":"2024-09-18T08:11:42.328627+0000","last_active":"2024-09-18T08:14:06.577215+0000","last_peered":"2024-09-18T08:14:06.577215+0000","last_clean":"2024-09-18T08:11:40.307990+0000","last_became_active":"2024-09-18T08:11:42.328627+0000","last_became_peered":"2024-09-18T08:11:42.328627+0000","last_unstale":"2024-09-18T08:14:06.577215+0000","last_undegraded":"2024-09-18T08:11:42.318971+0000","last_fullsized":"2024-09-18T08:11:42.318795+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'1","last_scrub_stamp":"2024-09-18T08:10:17.878965+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_clean_scrub_stamp":"2024-09-18T08:10:17.878965+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:14:57.836367+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":["0","5"],"object_location_counts":[{"shards":"0,5","objects":1}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.11","version":"271'19","reported_seq":604,"reported_epoch":294,"state":"active+clean","last_fresh":"2024-09-18T08:14:06.575731+0000","last_change":"2024-09-18T08:14:05.610486+0000","last_active":"2024-09-18T08:14:06.575731+0000","last_peered":"2024-09-18T08:14:06.575731+0000","last_clean":"2024-09-18T08:14:06.575731+0000","last_became_active":"2024-09-18T08:14:05.594777+0000","last_became_peered":"2024-09-18T08:14:05.594777+0000","last_unstale":"2024-09-18T08:14:06.575731+0000","last_undegraded":"2024-09-18T08:14:06.575731+0000","last_fullsized":"2024-09-18T08:14:06.575731+0000","mapping_epoch":292,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":293,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'11","last_scrub_stamp":"2024-09-18T08:11:22.332279+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_clean_scrub_stamp":"2024-09-18T08:11:22.332279+0000","objects_scrubbed":5,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":32,"num_read_kb":25,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":513,"reported_epoch":294,"state":"active+clean+remapped","last_fresh":"2024-09-18T08:14:06.590334+0000","last_change":"2024-09-18T08:14:06.590187+0000","last_active":"2024-09-18T08:14:06.590334+0000","last_peered":"2024-09-18T08:14:06.590334+0000","last_clean":"2024-09-18T08:14:06.590334+0000","last_became_active":"2024-09-18T08:14:06.589661+0000","last_became_peered":"2024-09-18T08:14:06.589661+0000","last_unstale":"2024-09-18T08:14:06.590334+0000","last_undegraded":"2024-09-18T08:14:06.590334+0000","last_fullsized":"2024-09-18T08:14:06.590334+0000","mapping_epoch":293,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":294,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:10:53.145315+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:46.860092+0000","last_clean_scrub_stamp":"2024-09-18T08:10:53.145315+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3,7],"avail_no_missing":["5","3","7"],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":302,"reported_epoch":294,"state":"active+clean","last_fresh":"2024-09-18T08:14:06.570977+0000","last_change":"2024-09-18T08:14:05.586140+0000","last_active":"2024-09-18T08:14:06.570977+0000","last_peered":"2024-09-18T08:14:06.570977+0000","last_clean":"2024-09-18T08:14:06.570977+0000","last_became_active":"2024-09-18T08:14:05.585776+0000","last_became_peered":"2024-09-18T08:14:05.585776+0000","last_unstale":"2024-09-18T08:14:06.570977+0000","last_undegraded":"2024-09-18T08:14:06.570977+0000","last_fullsized":"2024-09-18T08:14:06.570977+0000","mapping_epoch":292,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":293,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:10:52.577568+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:50.886830+0000","last_clean_scrub_stamp":"2024-09-18T08:10:52.577568+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,0],"acting":[4,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":506,"reported_epoch":294,"state":"active+clean","last_fresh":"2024-09-18T08:14:06.575582+0000","last_change":"2024-09-18T08:11:29.344444+0000","last_active":"2024-09-18T08:14:06.575582+0000","last_peered":"2024-09-18T08:14:06.575582+0000","last_clean":"2024-09-18T08:14:06.575582+0000","last_became_active":"2024-09-18T08:06:24.128582+0000","last_became_peered":"2024-09-18T08:06:24.128582+0000","last_unstale":"2024-09-18T08:14:06.575582+0000","last_undegraded":"2024-09-18T08:14:06.575582+0000","last_fullsized":"2024-09-18T08:14:06.575582+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:11:29.344382+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_clean_scrub_stamp":"2024-09-18T08:11:29.344382+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:14:24.734652+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.10","version":"271'8","reported_seq":542,"reported_epoch":294,"state":"active+undersized+degraded","last_fresh":"2024-09-18T08:14:06.570984+0000","last_change":"2024-09-18T08:11:42.340283+0000","last_active":"2024-09-18T08:14:06.570984+0000","last_peered":"2024-09-18T08:14:06.570984+0000","last_clean":"2024-09-18T08:11:40.313525+0000","last_became_active":"2024-09-18T08:11:42.340283+0000","last_became_peered":"2024-09-18T08:11:42.340283+0000","last_unstale":"2024-09-18T08:14:06.570984+0000","last_undegraded":"2024-09-18T08:11:42.331866+0000","last_fullsized":"2024-09-18T08:11:42.325406+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'4","last_scrub_stamp":"2024-09-18T08:10:18.542984+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_clean_scrub_stamp":"2024-09-18T08:10:18.542984+0000","objects_scrubbed":2,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:14:34.684257+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":12,"num_read_kb":10,"num_write":8,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":["3","6"],"object_location_counts":[{"shards":"3,6","objects":2}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":511,"reported_epoch":294,"state":"active+clean","last_fresh":"2024-09-18T08:14:06.578610+0000","last_change":"2024-09-18T08:10:13.820205+0000","last_active":"2024-09-18T08:14:06.578610+0000","last_peered":"2024-09-18T08:14:06.578610+0000","last_clean":"2024-09-18T08:14:06.578610+0000","last_became_active":"2024-09-18T08:02:47.877616+0000","last_became_peered":"2024-09-18T08:02:47.877616+0000","last_unstale":"2024-09-18T08:14:06.578610+0000","last_undegraded":"2024-09-18T08:14:06.578610+0000","last_fullsized":"2024-09-18T08:14:06.578610+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:10:13.820142+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:46.860092+0000","last_clean_scrub_stamp":"2024-09-18T08:10:13.820142+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:14:25.821232+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":261,"reported_epoch":294,"state":"active+clean","last_fresh":"2024-09-18T08:14:06.572073+0000","last_change":"2024-09-18T08:14:05.589315+0000","last_active":"2024-09-18T08:14:06.572073+0000","last_peered":"2024-09-18T08:14:06.572073+0000","last_clean":"2024-09-18T08:14:06.572073+0000","last_became_active":"2024-09-18T08:14:05.588551+0000","last_became_peered":"2024-09-18T08:14:05.588551+0000","last_unstale":"2024-09-18T08:14:06.572073+0000","last_undegraded":"2024-09-18T08:14:06.572073+0000","last_fullsized":"2024-09-18T08:14:06.572073+0000","mapping_epoch":292,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":293,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:13:32.640669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:50.886830+0000","last_clean_scrub_stamp":"2024-09-18T08:13:32.640669+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:14:34.935709+0000","scrub_duration":20,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,0],"acting":[4,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":308,"reported_epoch":294,"state":"active+clean","last_fresh":"2024-09-18T08:14:06.572761+0000","last_change":"2024-09-18T08:14:05.582585+0000","last_active":"2024-09-18T08:14:06.572761+0000","last_peered":"2024-09-18T08:14:06.572761+0000","last_clean":"2024-09-18T08:14:06.572761+0000","last_became_active":"2024-09-18T08:14:05.582250+0000","last_became_peered":"2024-09-18T08:14:05.582250+0000","last_unstale":"2024-09-18T08:14:06.572761+0000","last_undegraded":"2024-09-18T08:14:06.572761+0000","last_fullsized":"2024-09-18T08:14:06.572761+0000","mapping_epoch":292,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":293,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:13:56.525167+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_clean_scrub_stamp":"2024-09-18T08:13:56.525167+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:15:15.184232+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,3],"acting":[2,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":354,"reported_epoch":294,"state":"active+undersized","last_fresh":"2024-09-18T08:14:06.581962+0000","last_change":"2024-09-18T08:11:42.341016+0000","last_active":"2024-09-18T08:14:06.581962+0000","last_peered":"2024-09-18T08:14:06.581962+0000","last_clean":"2024-09-18T08:08:15.505600+0000","last_became_active":"2024-09-18T08:11:42.341016+0000","last_became_peered":"2024-09-18T08:11:42.341016+0000","last_unstale":"2024-09-18T08:14:06.581962+0000","last_undegraded":"2024-09-18T08:14:06.581962+0000","last_fullsized":"2024-09-18T08:11:42.325794+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":182,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:10:52.641537+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_clean_scrub_stamp":"2024-09-18T08:10:52.641537+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:14:26.159423+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":["5","4"],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":500,"reported_epoch":294,"state":"active+undersized","last_fresh":"2024-09-18T08:14:06.571319+0000","last_change":"2024-09-18T08:11:42.342099+0000","last_active":"2024-09-18T08:14:06.571319+0000","last_peered":"2024-09-18T08:14:06.571319+0000","last_clean":"2024-09-18T08:11:40.314449+0000","last_became_active":"2024-09-18T08:11:42.342099+0000","last_became_peered":"2024-09-18T08:11:42.342099+0000","last_unstale":"2024-09-18T08:14:06.571319+0000","last_undegraded":"2024-09-18T08:14:06.571319+0000","last_fullsized":"2024-09-18T08:11:42.325904+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:07:17.277894+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:46.860092+0000","last_clean_scrub_stamp":"2024-09-18T08:07:17.277894+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:14:50.443986+0000","scrub_duration":16,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":["3","0"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"271'19","reported_seq":509,"reported_epoch":294,"state":"active+undersized+degraded","last_fresh":"2024-09-18T08:14:06.610834+0000","last_change":"2024-09-18T08:11:42.334941+0000","last_active":"2024-09-18T08:14:06.610834+0000","last_peered":"2024-09-18T08:14:06.610834+0000","last_clean":"2024-09-18T08:11:40.304598+0000","last_became_active":"2024-09-18T08:11:42.334941+0000","last_became_peered":"2024-09-18T08:11:42.334941+0000","last_unstale":"2024-09-18T08:14:06.610834+0000","last_undegraded":"2024-09-18T08:11:42.328266+0000","last_fullsized":"2024-09-18T08:11:42.327013+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'11","last_scrub_stamp":"2024-09-18T08:10:50.208543+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_clean_scrub_stamp":"2024-09-18T08:10:50.208543+0000","objects_scrubbed":5,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:14:45.109285+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":5,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":31,"num_read_kb":24,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":["5","6"],"object_location_counts":[{"shards":"5,6","objects":5}],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":524,"reported_epoch":294,"state":"active+clean+remapped","last_fresh":"2024-09-18T08:14:06.587241+0000","last_change":"2024-09-18T08:14:06.587083+0000","last_active":"2024-09-18T08:14:06.587241+0000","last_peered":"2024-09-18T08:14:06.587241+0000","last_clean":"2024-09-18T08:14:06.587241+0000","last_became_active":"2024-09-18T08:14:06.582599+0000","last_became_peered":"2024-09-18T08:14:06.582599+0000","last_unstale":"2024-09-18T08:14:06.587241+0000","last_undegraded":"2024-09-18T08:14:06.587241+0000","last_fullsized":"2024-09-18T08:14:06.587241+0000","mapping_epoch":293,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":294,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:12:31.664456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:50.886830+0000","last_clean_scrub_stamp":"2024-09-18T08:12:31.664456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2,7],"avail_no_missing":["0","2","7"],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":9,"num_pg":16,"stat_sum":{"num_bytes":1898905624,"num_objects":28976,"num_object_clones":0,"num_object_copies":86928,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":15073,"num_objects_misplaced":15324,"num_objects_unfound":0,"num_objects_dirty":28976,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":254790,"num_write_kb":9080449,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2380,"num_bytes_recovered":155910144,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":6655778816,"data_stored":6655770672,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":69097,"ondisk_log_size":69097,"up":38,"acting":39,"num_store_stats":8},{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":416,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":403,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":85,"acting":88,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":93,"ondisk_log_size":93,"up":79,"acting":85,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":178,"num_object_clones":0,"num_object_copies":534,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":79,"num_objects_misplaced":38,"num_objects_unfound":0,"num_objects_dirty":178,"num_whiteouts":0,"num_read":1110,"num_read_kb":867,"num_write":703,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":218,"num_bytes_recovered":4488,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":421888,"data_stored":11216,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":686,"ondisk_log_size":686,"up":75,"acting":81,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":1613,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":24,"num_read_kb":24,"num_write":10,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":2606,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":4839,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":6,"ondisk_log_size":6,"up":81,"acting":88,"num_store_stats":8},{"poolid":2,"num_pg":3,"stat_sum":{"num_bytes":408,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":145,"num_read_kb":140,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":427,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1613,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":8,"ondisk_log_size":8,"up":7,"acting":8,"num_store_stats":7},{"poolid":1,"num_pg":11,"stat_sum":{"num_bytes":644038,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":10,"num_read_kb":20,"num_write":11,"num_write_kb":148,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":381350,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":869,"ondisk_log_size":869,"up":31,"acting":32,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":0,"seq":240518168716,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":47,"seq":201863463058,"num_pgs":56,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":703072,"kb_used_data":356752,"kb_used_omap":1,"kb_used_meta":346302,"kb_avail":93042080,"statfs":{"total":95995035648,"available":95275089920,"internally_reserved":0,"allocated":365314048,"data_stored":364624156,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":354613702},"hb_peers":[0,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,1],"upper_bound":4},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":39,"seq":167503724696,"num_pgs":68,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1549060,"kb_used_data":1084592,"kb_used_omap":1,"kb_used_meta":464318,"kb_avail":92196092,"statfs":{"total":95995035648,"available":94408798208,"internally_reserved":0,"allocated":1110622208,"data_stored":1109901254,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":475462086},"hb_peers":[0,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,0,0,0,0,0,5],"upper_bound":4096},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":32,"seq":137438953629,"num_pgs":71,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":760260,"kb_used_data":390320,"kb_used_omap":1,"kb_used_meta":369918,"kb_avail":92984892,"statfs":{"total":95995035648,"available":95216529408,"internally_reserved":0,"allocated":399687680,"data_stored":398966618,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":378796485},"hb_peers":[0,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,1,0,1],"upper_bound":16},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":26,"seq":111669149860,"num_pgs":78,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1525004,"kb_used_data":970364,"kb_used_omap":1,"kb_used_meta":554558,"kb_avail":92220148,"statfs":{"total":95995035648,"available":94433431552,"internally_reserved":0,"allocated":993652736,"data_stored":992919905,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":567867848},"hb_peers":[0,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,3,3,5,1],"upper_bound":32},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411498,"num_pgs":55,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":793844,"kb_used_data":357280,"kb_used_omap":1,"kb_used_meta":436542,"kb_avail":92951308,"statfs":{"total":95995035648,"available":95182139392,"internally_reserved":0,"allocated":365854720,"data_stored":365149902,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":447019461},"hb_peers":[0,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,1],"upper_bound":8},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574992,"num_pgs":64,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2106904,"kb_used_data":1782792,"kb_used_omap":1,"kb_used_meta":324094,"kb_avail":91638248,"statfs":{"total":95995035648,"available":93837565952,"internally_reserved":0,"allocated":1825579008,"data_stored":1825118505,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":331872709},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738549,"num_pgs":53,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":889088,"kb_used_data":501104,"kb_used_omap":1,"kb_used_meta":387966,"kb_avail":92856064,"statfs":{"total":95995035648,"available":95084609536,"internally_reserved":0,"allocated":513130496,"data_stored":512409680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":397277637},"hb_peers":[2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,0,0,1,1],"upper_bound":1024},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":266240,"data_stored":262688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1131,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1521,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":436,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":46,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1177,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":436,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1474,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":69632,"data_stored":1788,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1238,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":1898,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":1584,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1430,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":1144,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":660,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":403,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":403,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":416,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":511574016,"data_stored":511574016,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":1860698112,"data_stored":1860698112,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":362872832,"data_stored":362872832,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":978124800,"data_stored":978124800,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":402849792,"data_stored":402849792,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":1099894784,"data_stored":1099890712,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":359399424,"data_stored":359399424,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":1080365056,"data_stored":1080360984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T08:14:19.557 INFO:tasks.cephadm.ceph_manager.ceph:would drop below min_pgs, proj 1, currently 11 2024-09-18T08:14:19.558 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0 2024-09-18T08:14:19.624 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:14:19.625 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:14:19.625 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0' 2024-09-18T08:14:19.655 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:14:19 smithi050 ceph-mon[32371]: Health check update: Degraded data redundancy: 9554/70002 objects degraded (13.648%), 31 pgs degraded, 49 pgs undersized (PG_DEGRADED) 2024-09-18T08:14:19.656 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:14:19 smithi050 ceph-mon[32371]: osdmap e299: 8 total, 7 up, 7 in 2024-09-18T08:14:19.656 INFO:journalctl@ceph.iscsi.iscsi.a.smithi050.stdout:Sep 18 08:14:19 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-iscsi-iscsi-a[57782]: debug there is no tcmu-runner data available 2024-09-18T08:14:19.727 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2 2024-09-18T08:14:19.763 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:14:19.764 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:14:19.764 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2' 2024-09-18T08:14:19.865 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3 2024-09-18T08:14:19.901 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:14:19.902 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:14:19.902 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3' 2024-09-18T08:14:19.930 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:14:19 smithi007 ceph-mon[29459]: Health check update: Degraded data redundancy: 9554/70002 objects degraded (13.648%), 31 pgs degraded, 49 pgs undersized (PG_DEGRADED) 2024-09-18T08:14:19.930 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:14:19 smithi007 ceph-mon[29459]: osdmap e299: 8 total, 7 up, 7 in 2024-09-18T08:14:19.931 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:14:19 smithi007 ceph-mon[39249]: Health check update: Degraded data redundancy: 9554/70002 objects degraded (13.648%), 31 pgs degraded, 49 pgs undersized (PG_DEGRADED) 2024-09-18T08:14:19.931 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:14:19 smithi007 ceph-mon[39249]: osdmap e299: 8 total, 7 up, 7 in 2024-09-18T08:14:20.004 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4 2024-09-18T08:14:20.041 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:14:20.042 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/779349ca-7593-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T08:14:20.042 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:14:20.043 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4' 2024-09-18T08:14:20.111 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/779349ca-7593-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T08:14:20.144 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6 2024-09-18T08:14:20.184 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:14:20.185 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:14:20.185 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6' 2024-09-18T08:14:20.286 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6 2024-09-18T08:14:20.323 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:14:20.324 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:14:20.324 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6' 2024-09-18T08:14:20.425 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4 2024-09-18T08:14:20.451 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:14:20.452 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:14:20.452 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4' 2024-09-18T08:14:20.553 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3 2024-09-18T08:14:20.588 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:14:20.589 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:14:20.589 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3' 2024-09-18T08:14:20.690 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4 2024-09-18T08:14:20.726 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:14:20.727 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:14:20.727 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4' 2024-09-18T08:14:20.744 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:14:20 smithi007 ceph-mon[29459]: pgmap v560: 158 pgs: 29 active+clean+remapped, 22 active+undersized, 1 active+remapped+backfill_toofull, 27 active+undersized+degraded, 79 active+clean; 1.2 GiB data, 7.4 GiB used, 618 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 867 op/s; 6319/59607 objects degraded (10.601%); 7430/59607 objects misplaced (12.465%); 52 MiB/s, 832 objects/s recovering 2024-09-18T08:14:20.744 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:14:20 smithi007 ceph-mon[29459]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-18T08:14:20.745 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:14:20 smithi007 ceph-mon[39249]: pgmap v560: 158 pgs: 29 active+clean+remapped, 22 active+undersized, 1 active+remapped+backfill_toofull, 27 active+undersized+degraded, 79 active+clean; 1.2 GiB data, 7.4 GiB used, 618 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 867 op/s; 6319/59607 objects degraded (10.601%); 7430/59607 objects misplaced (12.465%); 52 MiB/s, 832 objects/s recovering 2024-09-18T08:14:20.745 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:14:20 smithi007 ceph-mon[39249]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-18T08:14:20.828 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7 2024-09-18T08:14:20.861 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:14:20.862 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:14:20.863 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7' 2024-09-18T08:14:20.905 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:14:20 smithi050 ceph-mon[32371]: pgmap v560: 158 pgs: 29 active+clean+remapped, 22 active+undersized, 1 active+remapped+backfill_toofull, 27 active+undersized+degraded, 79 active+clean; 1.2 GiB data, 7.4 GiB used, 618 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 867 op/s; 6319/59607 objects degraded (10.601%); 7430/59607 objects misplaced (12.465%); 52 MiB/s, 832 objects/s recovering 2024-09-18T08:14:20.905 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:14:20 smithi050 ceph-mon[32371]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-18T08:14:20.964 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6 2024-09-18T08:14:21.003 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:14:21.003 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:14:21.004 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6' 2024-09-18T08:14:21.025 INFO:teuthology.orchestra.run.smithi007.stdout:osd.0: {} 2024-09-18T08:14:21.026 INFO:teuthology.orchestra.run.smithi007.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-18T08:14:21.026 INFO:teuthology.orchestra.run.smithi007.stderr:Error ENXIO: problem getting command descriptions from osd.1 2024-09-18T08:14:21.057 INFO:teuthology.orchestra.run.smithi007.stdout:osd.2: {} 2024-09-18T08:14:21.057 INFO:teuthology.orchestra.run.smithi007.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-18T08:14:21.094 INFO:teuthology.orchestra.run.smithi007.stdout:osd.3: {} 2024-09-18T08:14:21.095 INFO:teuthology.orchestra.run.smithi007.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-18T08:14:21.105 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5 2024-09-18T08:14:21.128 INFO:teuthology.orchestra.run.smithi007.stdout:osd.4: {} 2024-09-18T08:14:21.128 INFO:teuthology.orchestra.run.smithi007.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-18T08:14:21.142 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:14:21.143 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:14:21.143 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5' 2024-09-18T08:14:21.158 INFO:teuthology.orchestra.run.smithi007.stdout:osd.5: {} 2024-09-18T08:14:21.158 INFO:teuthology.orchestra.run.smithi007.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-18T08:14:21.187 INFO:teuthology.orchestra.run.smithi007.stdout:osd.6: {} 2024-09-18T08:14:21.187 INFO:teuthology.orchestra.run.smithi007.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-18T08:14:21.220 INFO:teuthology.orchestra.run.smithi007.stdout:osd.7: {} 2024-09-18T08:14:21.220 INFO:teuthology.orchestra.run.smithi007.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-18T08:14:21.244 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5 2024-09-18T08:14:21.276 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:14:21.277 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:14:21.277 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5' 2024-09-18T08:14:21.378 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6 2024-09-18T08:14:21.411 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:14:21.412 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:14:21.413 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6' 2024-09-18T08:14:21.513 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6 2024-09-18T08:14:21.543 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:14:21.544 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:14:21.545 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6' 2024-09-18T08:14:21.645 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7 2024-09-18T08:14:21.681 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:14:21.682 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:14:21.683 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7' 2024-09-18T08:14:21.730 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:14:21 smithi007 ceph-mon[29459]: from='client.? 172.21.15.7:0/4206153711' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-18T08:14:21.730 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:14:21 smithi007 ceph-mon[39249]: from='client.? 172.21.15.7:0/4206153711' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-18T08:14:21.783 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3 2024-09-18T08:14:21.824 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:14:21.824 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:14:21.825 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3' 2024-09-18T08:14:21.905 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:14:21 smithi050 ceph-mon[32371]: from='client.? 172.21.15.7:0/4206153711' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-18T08:14:21.926 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4 2024-09-18T08:14:21.961 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:14:21.962 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:14:21.963 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4' 2024-09-18T08:14:21.978 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid 779349ca-7593-11ef-bceb-c7b262605968 -- ceph osd unset noscrub 2024-09-18T08:14:22.063 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2 2024-09-18T08:14:22.114 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:14:22.114 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:14:22.115 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2' 2024-09-18T08:14:22.215 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7 2024-09-18T08:14:22.248 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:14:22.249 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:14:22.249 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7' 2024-09-18T08:14:22.350 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4 2024-09-18T08:14:22.386 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:14:22.387 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:14:22.387 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4' 2024-09-18T08:14:22.488 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3 2024-09-18T08:14:22.512 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:14:22.513 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:14:22.514 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3' 2024-09-18T08:14:22.593 INFO:teuthology.orchestra.run.smithi007.stdout:{ 2024-09-18T08:14:22.593 INFO:teuthology.orchestra.run.smithi007.stdout: "ops": [ 2024-09-18T08:14:22.593 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:14:22.593 INFO:teuthology.orchestra.run.smithi007.stdout: "description": "osd_repop(client.15582.0:268206 9.6 e299/244 9:66c32ea6:::benchmark_data_smithi007_124019_object126321:head v 299'16744, mlcod=299'16741)", 2024-09-18T08:14:22.593 INFO:teuthology.orchestra.run.smithi007.stdout: "initiated_at": "2024-09-18T08:14:21.418896+0000", 2024-09-18T08:14:22.593 INFO:teuthology.orchestra.run.smithi007.stdout: "age": 0.0030727609999999998, 2024-09-18T08:14:22.593 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.0030993769999999999, 2024-09-18T08:14:22.593 INFO:teuthology.orchestra.run.smithi007.stdout: "continuous": false, 2024-09-18T08:14:22.593 INFO:teuthology.orchestra.run.smithi007.stdout: "type_data": { 2024-09-18T08:14:22.594 INFO:teuthology.orchestra.run.smithi007.stdout: "flag_point": "started", 2024-09-18T08:14:22.594 INFO:teuthology.orchestra.run.smithi007.stdout: "events": [ 2024-09-18T08:14:22.594 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:14:22.594 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "initiated", 2024-09-18T08:14:22.594 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:14:21.418896+0000", 2024-09-18T08:14:22.594 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-09-18T08:14:22.594 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:14:22.594 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:14:22.594 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "header_read", 2024-09-18T08:14:22.594 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:14:21.418896+0000", 2024-09-18T08:14:22.594 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-09-18T08:14:22.594 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:14:22.595 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:14:22.595 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "throttled", 2024-09-18T08:14:22.595 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:14:21.418896+0000", 2024-09-18T08:14:22.595 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 7.9599999999999998e-07 2024-09-18T08:14:22.595 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:14:22.595 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:14:22.595 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "all_read", 2024-09-18T08:14:22.595 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:14:21.418905+0000", 2024-09-18T08:14:22.595 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 8.276e-06 2024-09-18T08:14:22.595 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:14:22.595 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:14:22.595 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "dispatched", 2024-09-18T08:14:22.595 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:14:21.418915+0000", 2024-09-18T08:14:22.596 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.0013999999999999e-05 2024-09-18T08:14:22.596 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:14:22.596 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:14:22.596 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "queued_for_pg", 2024-09-18T08:14:22.596 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:14:21.418921+0000", 2024-09-18T08:14:22.596 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 6.285e-06 2024-09-18T08:14:22.596 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:14:22.596 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:14:22.596 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-09-18T08:14:22.596 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:14:21.418985+0000", 2024-09-18T08:14:22.596 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 6.3669000000000003e-05 2024-09-18T08:14:22.596 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:14:22.597 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:14:22.597 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "started", 2024-09-18T08:14:22.597 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:14:21.419005+0000", 2024-09-18T08:14:22.597 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 2.0565e-05 2024-09-18T08:14:22.597 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-09-18T08:14:22.597 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-09-18T08:14:22.597 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-09-18T08:14:22.597 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:14:22.597 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:14:22.597 INFO:teuthology.orchestra.run.smithi007.stdout: "description": "osd_repop(client.15582.0:268208 9.2 e299/244 9:40d5178d:::benchmark_data_smithi007_124019_object126323:head v 299'16858, mlcod=299'16856)", 2024-09-18T08:14:22.597 INFO:teuthology.orchestra.run.smithi007.stdout: "initiated_at": "2024-09-18T08:14:21.419972+0000", 2024-09-18T08:14:22.597 INFO:teuthology.orchestra.run.smithi007.stdout: "age": 0.0019961860000000001, 2024-09-18T08:14:22.597 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.0020776169999999999, 2024-09-18T08:14:22.598 INFO:teuthology.orchestra.run.smithi007.stdout: "continuous": false, 2024-09-18T08:14:22.598 INFO:teuthology.orchestra.run.smithi007.stdout: "type_data": { 2024-09-18T08:14:22.598 INFO:teuthology.orchestra.run.smithi007.stdout: "flag_point": "started", 2024-09-18T08:14:22.598 INFO:teuthology.orchestra.run.smithi007.stdout: "events": [ 2024-09-18T08:14:22.598 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:14:22.598 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "initiated", 2024-09-18T08:14:22.598 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:14:21.419972+0000", 2024-09-18T08:14:22.598 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-09-18T08:14:22.598 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:14:22.598 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:14:22.598 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "header_read", 2024-09-18T08:14:22.598 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:14:21.419972+0000", 2024-09-18T08:14:22.599 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-09-18T08:14:22.599 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:14:22.599 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:14:22.599 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "throttled", 2024-09-18T08:14:22.599 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:14:21.419973+0000", 2024-09-18T08:14:22.599 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.1880000000000001e-06 2024-09-18T08:14:22.599 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:14:22.599 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:14:22.599 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "all_read", 2024-09-18T08:14:22.599 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:14:21.419980+0000", 2024-09-18T08:14:22.599 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 6.3840000000000002e-06 2024-09-18T08:14:22.599 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:14:22.599 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:14:22.600 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "dispatched", 2024-09-18T08:14:22.600 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:14:21.420003+0000", 2024-09-18T08:14:22.600 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 2.3567e-05 2024-09-18T08:14:22.600 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:14:22.600 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:14:22.600 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "queued_for_pg", 2024-09-18T08:14:22.600 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:14:21.420011+0000", 2024-09-18T08:14:22.600 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 7.3329999999999999e-06 2024-09-18T08:14:22.600 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:14:22.600 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:14:22.600 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-09-18T08:14:22.600 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:14:21.420133+0000", 2024-09-18T08:14:22.600 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.00012238799999999999 2024-09-18T08:14:22.600 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:14:22.601 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:14:22.601 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "started", 2024-09-18T08:14:22.601 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:14:21.420153+0000", 2024-09-18T08:14:22.601 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 2.0098999999999998e-05 2024-09-18T08:14:22.601 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-09-18T08:14:22.601 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-09-18T08:14:22.601 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-09-18T08:14:22.601 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:14:22.601 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:14:22.601 INFO:teuthology.orchestra.run.smithi007.stdout: "description": "osd_op(client.15582.0:268211 9.0 9:05751824:::benchmark_data_smithi007_124019_object126326:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e299)", 2024-09-18T08:14:22.601 INFO:teuthology.orchestra.run.smithi007.stdout: "initiated_at": "2024-09-18T08:14:21.420586+0000", 2024-09-18T08:14:22.601 INFO:teuthology.orchestra.run.smithi007.stdout: "age": 0.0013823609999999999, 2024-09-18T08:14:22.601 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.00151677, 2024-09-18T08:14:22.602 INFO:teuthology.orchestra.run.smithi007.stdout: "continuous": false, 2024-09-18T08:14:22.602 INFO:teuthology.orchestra.run.smithi007.stdout: "type_data": { 2024-09-18T08:14:22.602 INFO:teuthology.orchestra.run.smithi007.stdout: "flag_point": "waiting for sub ops", 2024-09-18T08:14:22.602 INFO:teuthology.orchestra.run.smithi007.stdout: "client_info": { 2024-09-18T08:14:22.602 INFO:teuthology.orchestra.run.smithi007.stdout: "client": "client.15582", 2024-09-18T08:14:22.602 INFO:teuthology.orchestra.run.smithi007.stdout: "client_addr": "172.21.15.7:0/3598605839", 2024-09-18T08:14:22.602 INFO:teuthology.orchestra.run.smithi007.stdout: "tid": 268211 2024-09-18T08:14:22.602 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:14:22.602 INFO:teuthology.orchestra.run.smithi007.stdout: "events": [ 2024-09-18T08:14:22.602 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:14:22.602 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "initiated", 2024-09-18T08:14:22.602 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:14:21.420586+0000", 2024-09-18T08:14:22.602 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-09-18T08:14:22.602 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:14:22.603 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:14:22.603 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "header_read", 2024-09-18T08:14:22.603 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:14:21.420586+0000", 2024-09-18T08:14:22.603 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-09-18T08:14:22.603 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:14:22.603 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:14:22.603 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "throttled", 2024-09-18T08:14:22.603 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:14:21.420587+0000", 2024-09-18T08:14:22.604 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.1370000000000001e-06 2024-09-18T08:14:22.604 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:14:22.604 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:14:22.604 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "all_read", 2024-09-18T08:14:22.604 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:14:21.420591+0000", 2024-09-18T08:14:22.604 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 4.3900000000000003e-06 2024-09-18T08:14:22.604 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:14:22.604 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:14:22.604 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "dispatched", 2024-09-18T08:14:22.604 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:14:21.420599+0000", 2024-09-18T08:14:22.605 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 7.3039999999999999e-06 2024-09-18T08:14:22.605 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:14:22.605 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:14:22.605 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "queued_for_pg", 2024-09-18T08:14:22.605 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:14:21.420605+0000", 2024-09-18T08:14:22.605 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 6.5579999999999997e-06 2024-09-18T08:14:22.605 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:14:22.605 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:14:22.605 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-09-18T08:14:22.605 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:14:21.420720+0000", 2024-09-18T08:14:22.605 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.000115173 2024-09-18T08:14:22.606 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:14:22.606 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:14:22.606 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "started", 2024-09-18T08:14:22.606 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:14:21.420820+0000", 2024-09-18T08:14:22.606 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 9.9543000000000006e-05 2024-09-18T08:14:22.606 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:14:22.606 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:14:22.606 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "waiting for subops from 6", 2024-09-18T08:14:22.606 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:14:21.420932+0000", 2024-09-18T08:14:22.606 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.000112076 2024-09-18T08:14:22.606 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-09-18T08:14:22.606 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-09-18T08:14:22.607 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-09-18T08:14:22.607 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:14:22.607 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:14:22.607 INFO:teuthology.orchestra.run.smithi007.stdout: "description": "osd_repop(client.15582.0:268213 9.7 e299/293)", 2024-09-18T08:14:22.607 INFO:teuthology.orchestra.run.smithi007.stdout: "initiated_at": "2024-09-18T08:14:21.421766+0000", 2024-09-18T08:14:22.607 INFO:teuthology.orchestra.run.smithi007.stdout: "age": 0.00020193000000000001, 2024-09-18T08:14:22.607 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.00039418, 2024-09-18T08:14:22.607 INFO:teuthology.orchestra.run.smithi007.stdout: "continuous": false, 2024-09-18T08:14:22.607 INFO:teuthology.orchestra.run.smithi007.stdout: "type_data": { 2024-09-18T08:14:22.607 INFO:teuthology.orchestra.run.smithi007.stdout: "flag_point": "queued for pg", 2024-09-18T08:14:22.607 INFO:teuthology.orchestra.run.smithi007.stdout: "events": [ 2024-09-18T08:14:22.607 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:14:22.607 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "initiated", 2024-09-18T08:14:22.608 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:14:21.421766+0000", 2024-09-18T08:14:22.608 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-09-18T08:14:22.608 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:14:22.608 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:14:22.608 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "header_read", 2024-09-18T08:14:22.608 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:14:21.421766+0000", 2024-09-18T08:14:22.608 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-09-18T08:14:22.608 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:14:22.608 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:14:22.608 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "throttled", 2024-09-18T08:14:22.608 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:14:21.421767+0000", 2024-09-18T08:14:22.608 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 8.1900000000000001e-07 2024-09-18T08:14:22.608 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:14:22.609 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:14:22.609 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "all_read", 2024-09-18T08:14:22.609 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:14:21.421773+0000", 2024-09-18T08:14:22.609 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 5.9819999999999996e-06 2024-09-18T08:14:22.609 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:14:22.609 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:14:22.609 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "dispatched", 2024-09-18T08:14:22.609 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:14:21.421809+0000", 2024-09-18T08:14:22.609 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 3.5366e-05 2024-09-18T08:14:22.609 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:14:22.609 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:14:22.609 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "queued_for_pg", 2024-09-18T08:14:22.609 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:14:21.421827+0000", 2024-09-18T08:14:22.610 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.7992999999999999e-05 2024-09-18T08:14:22.610 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-09-18T08:14:22.610 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-09-18T08:14:22.610 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-09-18T08:14:22.610 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-09-18T08:14:22.610 INFO:teuthology.orchestra.run.smithi007.stdout: ], 2024-09-18T08:14:22.610 INFO:teuthology.orchestra.run.smithi007.stdout: "num_ops": 4 2024-09-18T08:14:22.610 INFO:teuthology.orchestra.run.smithi007.stdout:} 2024-09-18T08:14:22.616 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0 2024-09-18T08:14:22.651 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:14:22.651 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:14:22.652 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0' 2024-09-18T08:14:22.753 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0 2024-09-18T08:14:22.785 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:14:22.786 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:14:22.786 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0' 2024-09-18T08:14:22.887 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6 2024-09-18T08:14:22.897 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:14:22 smithi007 ceph-mon[29459]: pgmap v561: 158 pgs: 29 active+clean+remapped, 22 active+undersized, 1 active+remapped+backfill_toofull, 27 active+undersized+degraded, 79 active+clean; 1.2 GiB data, 7.4 GiB used, 618 GiB / 626 GiB avail; 894 B/s rd, 0 B/s wr, 605 op/s; 6319/59607 objects degraded (10.601%); 7430/59607 objects misplaced (12.465%); 36 MiB/s, 581 objects/s recovering 2024-09-18T08:14:22.897 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:14:22 smithi007 ceph-mon[39249]: pgmap v561: 158 pgs: 29 active+clean+remapped, 22 active+undersized, 1 active+remapped+backfill_toofull, 27 active+undersized+degraded, 79 active+clean; 1.2 GiB data, 7.4 GiB used, 618 GiB / 626 GiB avail; 894 B/s rd, 0 B/s wr, 605 op/s; 6319/59607 objects degraded (10.601%); 7430/59607 objects misplaced (12.465%); 36 MiB/s, 581 objects/s recovering 2024-09-18T08:14:22.905 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:14:22 smithi050 ceph-mon[32371]: pgmap v561: 158 pgs: 29 active+clean+remapped, 22 active+undersized, 1 active+remapped+backfill_toofull, 27 active+undersized+degraded, 79 active+clean; 1.2 GiB data, 7.4 GiB used, 618 GiB / 626 GiB avail; 894 B/s rd, 0 B/s wr, 605 op/s; 6319/59607 objects degraded (10.601%); 7430/59607 objects misplaced (12.465%); 36 MiB/s, 581 objects/s recovering 2024-09-18T08:14:22.928 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:14:22.928 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:14:22.929 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6' 2024-09-18T08:14:23.030 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0 2024-09-18T08:14:23.062 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:14:23.063 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:14:23.063 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0' 2024-09-18T08:14:23.164 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6 2024-09-18T08:14:23.213 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:14:23.214 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:14:23.214 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6' 2024-09-18T08:14:23.315 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7 2024-09-18T08:14:23.354 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:14:23.355 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:14:23.355 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7' 2024-09-18T08:14:23.391 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid 779349ca-7593-11ef-bceb-c7b262605968 -- ceph daemon osd.2 dump_blocked_ops 2024-09-18T08:14:23.456 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0 2024-09-18T08:14:23.498 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:14:23.499 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:14:23.499 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0' 2024-09-18T08:14:23.600 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6 2024-09-18T08:14:23.636 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:14:23.636 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:14:23.637 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6' 2024-09-18T08:14:23.741 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0 2024-09-18T08:14:23.766 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:14:23.767 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:14:23.767 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0' 2024-09-18T08:14:23.868 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5 2024-09-18T08:14:23.877 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-09-18T08:14:23.878 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-09-18T08:14:23.890 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/779349ca-7593-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T08:14:23.902 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:14:23.903 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:14:23.903 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5' 2024-09-18T08:14:24.004 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4 2024-09-18T08:14:24.039 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:14:24.040 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:14:24.040 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4' 2024-09-18T08:14:24.141 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2 2024-09-18T08:14:24.190 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:14:24.191 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:14:24.192 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2' 2024-09-18T08:14:24.293 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2 2024-09-18T08:14:24.330 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:14:24.331 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:14:24.331 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2' 2024-09-18T08:14:24.430 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/779349ca-7593-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T08:14:24.432 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2 2024-09-18T08:14:24.473 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:14:24.474 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:14:24.474 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2' 2024-09-18T08:14:24.558 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 5, 6, 4] out_osds: [7] dead_osds: [1] live_osds: [0, 2, 3, 4, 5, 6, 7] 2024-09-18T08:14:24.558 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-18T08:14:24.559 INFO:tasks.thrashosds.thrasher:Reviving osd 1 2024-09-18T08:14:24.559 INFO:tasks.cephadm.osd.1:Restarting osd.1 (starting--it wasn't running)... 2024-09-18T08:14:24.559 DEBUG:teuthology.orchestra.run.smithi007:osd.1> sudo journalctl -f -n 0 -u ceph-779349ca-7593-11ef-bceb-c7b262605968@osd.1.service 2024-09-18T08:14:24.563 DEBUG:teuthology.orchestra.run.smithi007:> sudo systemctl start ceph-779349ca-7593-11ef-bceb-c7b262605968@osd.1 2024-09-18T08:14:24.576 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3 2024-09-18T08:14:24.652 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:14:24.652 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:14:24.653 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3' 2024-09-18T08:14:24.754 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6 2024-09-18T08:14:24.787 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:14:24.787 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:14:24.788 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6' 2024-09-18T08:14:24.819 INFO:journalctl@ceph.osd.1.smithi007.stdout:Sep 18 08:14:24 smithi007 systemd[1]: Starting Ceph osd.1 for 779349ca-7593-11ef-bceb-c7b262605968... 2024-09-18T08:14:24.820 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:14:24 smithi007 ceph-mon[29459]: pgmap v562: 158 pgs: 29 active+clean+remapped, 22 active+undersized, 1 active+remapped+backfill_toofull, 27 active+undersized+degraded, 79 active+clean; 1.2 GiB data, 7.4 GiB used, 618 GiB / 626 GiB avail; 761 B/s rd, 0 B/s wr, 515 op/s; 6319/59607 objects degraded (10.601%); 7430/59607 objects misplaced (12.465%); 31 MiB/s, 495 objects/s recovering 2024-09-18T08:14:24.820 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:14:24 smithi007 ceph-mon[29459]: Health check update: Degraded data redundancy: 6319/59607 objects degraded (10.601%), 27 pgs degraded, 49 pgs undersized (PG_DEGRADED) 2024-09-18T08:14:24.820 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:14:24 smithi007 ceph-mon[39249]: pgmap v562: 158 pgs: 29 active+clean+remapped, 22 active+undersized, 1 active+remapped+backfill_toofull, 27 active+undersized+degraded, 79 active+clean; 1.2 GiB data, 7.4 GiB used, 618 GiB / 626 GiB avail; 761 B/s rd, 0 B/s wr, 515 op/s; 6319/59607 objects degraded (10.601%); 7430/59607 objects misplaced (12.465%); 31 MiB/s, 495 objects/s recovering 2024-09-18T08:14:24.820 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:14:24 smithi007 ceph-mon[39249]: Health check update: Degraded data redundancy: 6319/59607 objects degraded (10.601%), 27 pgs degraded, 49 pgs undersized (PG_DEGRADED) 2024-09-18T08:14:24.888 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0 2024-09-18T08:14:24.905 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:14:24 smithi050 ceph-mon[32371]: pgmap v562: 158 pgs: 29 active+clean+remapped, 22 active+undersized, 1 active+remapped+backfill_toofull, 27 active+undersized+degraded, 79 active+clean; 1.2 GiB data, 7.4 GiB used, 618 GiB / 626 GiB avail; 761 B/s rd, 0 B/s wr, 515 op/s; 6319/59607 objects degraded (10.601%); 7430/59607 objects misplaced (12.465%); 31 MiB/s, 495 objects/s recovering 2024-09-18T08:14:24.905 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:14:24 smithi050 ceph-mon[32371]: Health check update: Degraded data redundancy: 6319/59607 objects degraded (10.601%), 27 pgs degraded, 49 pgs undersized (PG_DEGRADED) 2024-09-18T08:14:24.924 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:14:24.925 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:14:24.925 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0' 2024-09-18T08:14:25.026 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0 2024-09-18T08:14:25.068 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:14:25.069 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:14:25.069 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0' 2024-09-18T08:14:25.095 INFO:teuthology.orchestra.run.smithi007.stdout:{ 2024-09-18T08:14:25.095 INFO:teuthology.orchestra.run.smithi007.stdout: "ops": [], 2024-09-18T08:14:25.096 INFO:teuthology.orchestra.run.smithi007.stdout: "complaint_time": 30, 2024-09-18T08:14:25.096 INFO:teuthology.orchestra.run.smithi007.stdout: "num_blocked_ops": 0 2024-09-18T08:14:25.096 INFO:teuthology.orchestra.run.smithi007.stdout:} 2024-09-18T08:14:25.170 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4 2024-09-18T08:14:25.205 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:14:25.205 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:14:25.206 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4' 2024-09-18T08:14:25.306 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3 2024-09-18T08:14:25.349 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:14:25.350 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:14:25.351 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3' 2024-09-18T08:14:25.452 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4 2024-09-18T08:14:25.492 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:14:25.493 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:14:25.493 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4' 2024-09-18T08:14:25.594 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4 2024-09-18T08:14:25.631 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:14:25.631 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:14:25.632 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4' 2024-09-18T08:14:25.691 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:14:25 smithi007 ceph-mon[29459]: from='client.? 172.21.15.7:0/1146366760' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-18T08:14:25.692 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:14:25 smithi007 ceph-mon[29459]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-18T08:14:25.692 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:14:25 smithi007 ceph-mon[39249]: from='client.? 172.21.15.7:0/1146366760' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-18T08:14:25.692 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:14:25 smithi007 ceph-mon[39249]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-18T08:14:25.692 INFO:journalctl@ceph.osd.1.smithi007.stdout:Sep 18 08:14:25 smithi007 podman[145016]: 2024-09-18 08:14:25.324902957 +0000 UTC m=+0.228059764 image pull b22e1e37ddf1ca6e22136cd6540a9ab61e7cafb8bfbc7a91a2f20f7f9941bd9a quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:1f5b5a3734f665ef333266f39143ac3ef1eb1951b21258f2307e10332e2d7ac9 2024-09-18T08:14:25.692 INFO:journalctl@ceph.osd.1.smithi007.stdout:Sep 18 08:14:25 smithi007 podman[145016]: 2024-09-18 08:14:25.465312525 +0000 UTC m=+0.368469314 container create a4b7a8aad7c38eb69924bd12fba0c381f262b3409da796c57cb1752196b3f65e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:1f5b5a3734f665ef333266f39143ac3ef1eb1951b21258f2307e10332e2d7ac9, name=ceph-779349ca-7593-11ef-bceb-c7b262605968-osd-1-activate, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , io.buildah.version=1.37.2, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, RELEASE=wip-guits-main-2024-09-17-1213-e7fb7b5, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.build-date=20240731, ceph=True, GIT_CLEAN=True) 2024-09-18T08:14:25.693 INFO:teuthology.orchestra.run.smithi007.stderr:noscrub is unset 2024-09-18T08:14:25.732 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6 2024-09-18T08:14:25.775 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:14:25.775 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:14:25.776 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6' 2024-09-18T08:14:25.876 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4 2024-09-18T08:14:25.879 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid 779349ca-7593-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-18T08:14:25.911 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:14:25.912 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:14:25.912 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4' 2024-09-18T08:14:26.013 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0 2024-09-18T08:14:26.056 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:14:26.057 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:14:26.057 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0' 2024-09-18T08:14:26.155 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:14:25 smithi050 ceph-mon[32371]: from='client.? 172.21.15.7:0/1146366760' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-18T08:14:26.155 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:14:25 smithi050 ceph-mon[32371]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-18T08:14:26.158 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3 2024-09-18T08:14:26.203 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:14:26.204 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:14:26.205 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3' 2024-09-18T08:14:26.212 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid 779349ca-7593-11ef-bceb-c7b262605968 -- ceph daemon osd.2 dump_historic_ops 2024-09-18T08:14:26.305 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2 2024-09-18T08:14:26.356 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:14:26.357 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:14:26.357 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2' 2024-09-18T08:14:26.458 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2 2024-09-18T08:14:26.481 INFO:journalctl@ceph.osd.1.smithi007.stdout:Sep 18 08:14:26 smithi007 podman[145016]: 2024-09-18 08:14:26.382609448 +0000 UTC m=+1.285766254 container init a4b7a8aad7c38eb69924bd12fba0c381f262b3409da796c57cb1752196b3f65e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:1f5b5a3734f665ef333266f39143ac3ef1eb1951b21258f2307e10332e2d7ac9, name=ceph-779349ca-7593-11ef-bceb-c7b262605968-osd-1-activate, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, RELEASE=wip-guits-main-2024-09-17-1213-e7fb7b5, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.build-date=20240731, GIT_CLEAN=True, io.buildah.version=1.37.2) 2024-09-18T08:14:26.482 INFO:journalctl@ceph.osd.1.smithi007.stdout:Sep 18 08:14:26 smithi007 podman[145016]: 2024-09-18 08:14:26.393042776 +0000 UTC m=+1.296199574 container start a4b7a8aad7c38eb69924bd12fba0c381f262b3409da796c57cb1752196b3f65e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:1f5b5a3734f665ef333266f39143ac3ef1eb1951b21258f2307e10332e2d7ac9, name=ceph-779349ca-7593-11ef-bceb-c7b262605968-osd-1-activate, org.label-schema.license=GPLv2, io.buildah.version=1.37.2, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, ceph=True, RELEASE=wip-guits-main-2024-09-17-1213-e7fb7b5, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.build-date=20240731, maintainer=Guillaume Abrioux , GIT_CLEAN=True, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0) 2024-09-18T08:14:26.513 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:14:26.514 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:14:26.514 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2' 2024-09-18T08:14:26.614 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5 2024-09-18T08:14:26.646 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:14:26.647 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:14:26.647 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5' 2024-09-18T08:14:26.748 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3 2024-09-18T08:14:26.783 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:14:26.784 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:14:26.784 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3' 2024-09-18T08:14:26.885 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0 2024-09-18T08:14:26.923 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:14:26.923 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:14:26.923 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0' 2024-09-18T08:14:26.980 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Sep 18 08:14:26 smithi007 ceph-779349ca-7593-11ef-bceb-c7b262605968-mgr-y[29688]: ::ffff:172.21.15.50 - - [18/Sep/2024:08:14:26] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-09-18T08:14:26.980 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:14:26 smithi007 ceph-mon[39249]: pgmap v563: 158 pgs: 29 active+clean+remapped, 22 active+undersized, 1 active+remapped+backfill_toofull, 27 active+undersized+degraded, 79 active+clean; 739 MiB data, 6.3 GiB used, 620 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.41k op/s; 3859/36054 objects degraded (10.703%); 4517/36054 objects misplaced (12.528%); 26 MiB/s, 415 objects/s recovering 2024-09-18T08:14:26.980 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:14:26 smithi007 ceph-mon[39249]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-18T08:14:26.980 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:14:26 smithi007 ceph-mon[39249]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-18T08:14:26.981 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:14:26 smithi007 ceph-mon[39249]: osdmap e300: 8 total, 7 up, 7 in 2024-09-18T08:14:26.981 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:14:26 smithi007 ceph-mon[39249]: 6.0 scrub starts 2024-09-18T08:14:26.981 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:14:26 smithi007 ceph-mon[39249]: 6.0 scrub ok 2024-09-18T08:14:26.981 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:14:26 smithi007 ceph-mon[29459]: pgmap v563: 158 pgs: 29 active+clean+remapped, 22 active+undersized, 1 active+remapped+backfill_toofull, 27 active+undersized+degraded, 79 active+clean; 739 MiB data, 6.3 GiB used, 620 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.41k op/s; 3859/36054 objects degraded (10.703%); 4517/36054 objects misplaced (12.528%); 26 MiB/s, 415 objects/s recovering 2024-09-18T08:14:26.981 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:14:26 smithi007 ceph-mon[29459]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-18T08:14:26.981 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:14:26 smithi007 ceph-mon[29459]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-18T08:14:26.981 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:14:26 smithi007 ceph-mon[29459]: osdmap e300: 8 total, 7 up, 7 in 2024-09-18T08:14:26.981 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:14:26 smithi007 ceph-mon[29459]: 6.0 scrub starts 2024-09-18T08:14:26.981 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:14:26 smithi007 ceph-mon[29459]: 6.0 scrub ok 2024-09-18T08:14:26.982 INFO:journalctl@ceph.osd.1.smithi007.stdout:Sep 18 08:14:26 smithi007 podman[145016]: 2024-09-18 08:14:26.599539693 +0000 UTC m=+1.502696483 container attach a4b7a8aad7c38eb69924bd12fba0c381f262b3409da796c57cb1752196b3f65e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:1f5b5a3734f665ef333266f39143ac3ef1eb1951b21258f2307e10332e2d7ac9, name=ceph-779349ca-7593-11ef-bceb-c7b262605968-osd-1-activate, io.buildah.version=1.37.2, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, ceph=True, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240731, GIT_CLEAN=True, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.schema-version=1.0, RELEASE=wip-guits-main-2024-09-17-1213-e7fb7b5) 2024-09-18T08:14:26.982 INFO:journalctl@ceph.osd.1.smithi007.stdout:Sep 18 08:14:26 smithi007 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd-1-activate[145061]: Running command: /usr/bin/ceph-authtool --gen-print-key 2024-09-18T08:14:26.982 INFO:journalctl@ceph.osd.1.smithi007.stdout:Sep 18 08:14:26 smithi007 bash[145016]: Running command: /usr/bin/ceph-authtool --gen-print-key 2024-09-18T08:14:26.982 INFO:journalctl@ceph.osd.1.smithi007.stdout:Sep 18 08:14:26 smithi007 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd-1-activate[145061]: Running command: /usr/bin/ceph-authtool --gen-print-key 2024-09-18T08:14:26.982 INFO:journalctl@ceph.osd.1.smithi007.stdout:Sep 18 08:14:26 smithi007 bash[145016]: Running command: /usr/bin/ceph-authtool --gen-print-key 2024-09-18T08:14:27.025 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7 2024-09-18T08:14:27.060 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:14:27.061 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:14:27.061 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7' 2024-09-18T08:14:27.155 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:14:26 smithi050 ceph-mon[32371]: pgmap v563: 158 pgs: 29 active+clean+remapped, 22 active+undersized, 1 active+remapped+backfill_toofull, 27 active+undersized+degraded, 79 active+clean; 739 MiB data, 6.3 GiB used, 620 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.41k op/s; 3859/36054 objects degraded (10.703%); 4517/36054 objects misplaced (12.528%); 26 MiB/s, 415 objects/s recovering 2024-09-18T08:14:27.156 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:14:26 smithi050 ceph-mon[32371]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-18T08:14:27.156 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:14:26 smithi050 ceph-mon[32371]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-18T08:14:27.156 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:14:26 smithi050 ceph-mon[32371]: osdmap e300: 8 total, 7 up, 7 in 2024-09-18T08:14:27.156 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:14:26 smithi050 ceph-mon[32371]: 6.0 scrub starts 2024-09-18T08:14:27.156 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:14:26 smithi050 ceph-mon[32371]: 6.0 scrub ok 2024-09-18T08:14:27.162 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2 2024-09-18T08:14:27.198 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:14:27.199 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:14:27.199 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2' 2024-09-18T08:14:27.300 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3 2024-09-18T08:14:27.338 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:14:27.340 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:14:27.340 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3' 2024-09-18T08:14:27.441 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6 2024-09-18T08:14:27.463 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:14:27.464 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:14:27.464 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6' 2024-09-18T08:14:27.565 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5 2024-09-18T08:14:27.595 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:14:27.596 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:14:27.596 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5' 2024-09-18T08:14:27.608 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/779349ca-7593-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T08:14:27.625 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/779349ca-7593-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T08:14:27.697 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4 2024-09-18T08:14:27.732 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:14:27.733 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:14:27.734 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4' 2024-09-18T08:14:27.753 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:14:27 smithi007 ceph-mon[39249]: 4.19 scrub starts 2024-09-18T08:14:27.753 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:14:27 smithi007 ceph-mon[39249]: 4.19 scrub ok 2024-09-18T08:14:27.753 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:14:27 smithi007 ceph-mon[39249]: purged_snaps scrub starts 2024-09-18T08:14:27.753 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:14:27 smithi007 ceph-mon[39249]: purged_snaps scrub ok 2024-09-18T08:14:27.753 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:14:27 smithi007 ceph-mon[39249]: osdmap e301: 8 total, 7 up, 7 in 2024-09-18T08:14:27.753 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:14:27 smithi007 ceph-mon[39249]: pgmap v566: 158 pgs: 29 active+clean+remapped, 22 active+undersized, 1 active+remapped+backfill_toofull, 27 active+undersized+degraded, 79 active+clean; 739 MiB data, 6.3 GiB used, 620 GiB / 626 GiB avail; 639 B/s rd, 0 B/s wr, 981 op/s; 3859/36054 objects degraded (10.703%); 4517/36054 objects misplaced (12.528%) 2024-09-18T08:14:27.754 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:14:27 smithi007 ceph-mon[29459]: 4.19 scrub starts 2024-09-18T08:14:27.754 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:14:27 smithi007 ceph-mon[29459]: 4.19 scrub ok 2024-09-18T08:14:27.754 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:14:27 smithi007 ceph-mon[29459]: purged_snaps scrub starts 2024-09-18T08:14:27.754 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:14:27 smithi007 ceph-mon[29459]: purged_snaps scrub ok 2024-09-18T08:14:27.754 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:14:27 smithi007 ceph-mon[29459]: osdmap e301: 8 total, 7 up, 7 in 2024-09-18T08:14:27.754 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:14:27 smithi007 ceph-mon[29459]: pgmap v566: 158 pgs: 29 active+clean+remapped, 22 active+undersized, 1 active+remapped+backfill_toofull, 27 active+undersized+degraded, 79 active+clean; 739 MiB data, 6.3 GiB used, 620 GiB / 626 GiB avail; 639 B/s rd, 0 B/s wr, 981 op/s; 3859/36054 objects degraded (10.703%); 4517/36054 objects misplaced (12.528%) 2024-09-18T08:14:27.835 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6 2024-09-18T08:14:27.867 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:14:27.867 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:14:27.868 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6' 2024-09-18T08:14:27.969 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4 2024-09-18T08:14:28.013 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:14:28.014 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:14:28.014 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4' 2024-09-18T08:14:28.115 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6 2024-09-18T08:14:28.150 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:14:28.151 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:14:28.152 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6' 2024-09-18T08:14:28.155 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:14:27 smithi050 ceph-mon[32371]: 4.19 scrub starts 2024-09-18T08:14:28.155 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:14:27 smithi050 ceph-mon[32371]: 4.19 scrub ok 2024-09-18T08:14:28.155 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:14:27 smithi050 ceph-mon[32371]: purged_snaps scrub starts 2024-09-18T08:14:28.155 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:14:27 smithi050 ceph-mon[32371]: purged_snaps scrub ok 2024-09-18T08:14:28.156 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:14:27 smithi050 ceph-mon[32371]: osdmap e301: 8 total, 7 up, 7 in 2024-09-18T08:14:28.156 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:14:27 smithi050 ceph-mon[32371]: pgmap v566: 158 pgs: 29 active+clean+remapped, 22 active+undersized, 1 active+remapped+backfill_toofull, 27 active+undersized+degraded, 79 active+clean; 739 MiB data, 6.3 GiB used, 620 GiB / 626 GiB avail; 639 B/s rd, 0 B/s wr, 981 op/s; 3859/36054 objects degraded (10.703%); 4517/36054 objects misplaced (12.528%) 2024-09-18T08:14:28.252 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2 2024-09-18T08:14:28.288 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:14:28.289 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:14:28.289 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2' 2024-09-18T08:14:28.390 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0 2024-09-18T08:14:28.431 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:14:28.431 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:14:28.432 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0' 2024-09-18T08:14:28.533 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0 2024-09-18T08:14:28.571 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:14:28.572 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:14:28.572 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0' 2024-09-18T08:14:28.673 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0 2024-09-18T08:14:28.694 INFO:journalctl@ceph.osd.1.smithi007.stdout:Sep 18 08:14:28 smithi007 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd-1-activate[145061]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-1 2024-09-18T08:14:28.695 INFO:journalctl@ceph.osd.1.smithi007.stdout:Sep 18 08:14:28 smithi007 bash[145016]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-1 2024-09-18T08:14:28.695 INFO:journalctl@ceph.osd.1.smithi007.stdout:Sep 18 08:14:28 smithi007 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd-1-activate[145061]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-1 --no-mon-config --dev /dev/mapper/vg_nvme-lv_3 2024-09-18T08:14:28.695 INFO:journalctl@ceph.osd.1.smithi007.stdout:Sep 18 08:14:28 smithi007 bash[145016]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-1 --no-mon-config --dev /dev/mapper/vg_nvme-lv_3 2024-09-18T08:14:28.705 INFO:teuthology.orchestra.run.smithi007.stdout:{ 2024-09-18T08:14:28.705 INFO:teuthology.orchestra.run.smithi007.stdout: "size": 20, 2024-09-18T08:14:28.705 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 600, 2024-09-18T08:14:28.705 INFO:teuthology.orchestra.run.smithi007.stdout: "ops": [ 2024-09-18T08:14:28.705 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:14:28.705 INFO:teuthology.orchestra.run.smithi007.stdout: "description": "osd_op(client.14805.0:66030 7.e 7:7b16ec01:::benchmark_data_smithi007_73155_object66029:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e102)", 2024-09-18T08:14:28.706 INFO:teuthology.orchestra.run.smithi007.stdout: "initiated_at": "2024-09-18T08:04:43.960797+0000", 2024-09-18T08:14:28.706 INFO:teuthology.orchestra.run.smithi007.stdout: "age": 584.73888464499998, 2024-09-18T08:14:28.706 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.089093715000000004, 2024-09-18T08:14:28.706 INFO:teuthology.orchestra.run.smithi007.stdout: "continuous": false, 2024-09-18T08:14:28.706 INFO:teuthology.orchestra.run.smithi007.stdout: "type_data": { 2024-09-18T08:14:28.706 INFO:teuthology.orchestra.run.smithi007.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T08:14:28.706 INFO:teuthology.orchestra.run.smithi007.stdout: "client_info": { 2024-09-18T08:14:28.706 INFO:teuthology.orchestra.run.smithi007.stdout: "client": "client.14805", 2024-09-18T08:14:28.706 INFO:teuthology.orchestra.run.smithi007.stdout: "client_addr": "172.21.15.7:0/640320436", 2024-09-18T08:14:28.706 INFO:teuthology.orchestra.run.smithi007.stdout: "tid": 66030 2024-09-18T08:14:28.706 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:14:28.706 INFO:teuthology.orchestra.run.smithi007.stdout: "events": [ 2024-09-18T08:14:28.706 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:14:28.707 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "initiated", 2024-09-18T08:14:28.707 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:04:43.960797+0000", 2024-09-18T08:14:28.707 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-09-18T08:14:28.707 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:14:28.707 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:14:28.707 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "header_read", 2024-09-18T08:14:28.707 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:04:43.960797+0000", 2024-09-18T08:14:28.707 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-09-18T08:14:28.707 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:14:28.707 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:14:28.707 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "throttled", 2024-09-18T08:14:28.707 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:04:43.960798+0000", 2024-09-18T08:14:28.707 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.1909999999999999e-06 2024-09-18T08:14:28.708 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:14:28.708 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:14:28.708 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "all_read", 2024-09-18T08:14:28.708 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:04:43.960822+0000", 2024-09-18T08:14:28.708 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 2.3785999999999999e-05 2024-09-18T08:14:28.708 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:14:28.708 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:14:28.708 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "dispatched", 2024-09-18T08:14:28.708 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:04:43.960830+0000", 2024-09-18T08:14:28.708 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 8.2400000000000007e-06 2024-09-18T08:14:28.708 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:14:28.708 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:14:28.708 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "queued_for_pg", 2024-09-18T08:14:28.708 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:04:43.960837+0000", 2024-09-18T08:14:28.709 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 6.7109999999999998e-06 2024-09-18T08:14:28.709 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:14:28.709 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:14:28.709 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-09-18T08:14:28.709 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:04:43.964072+0000", 2024-09-18T08:14:28.709 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.0032352380000000001 2024-09-18T08:14:28.709 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:14:28.709 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:14:28.709 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "started", 2024-09-18T08:14:28.709 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:04:43.964179+0000", 2024-09-18T08:14:28.709 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.000107524 2024-09-18T08:14:28.709 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:14:28.709 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:14:28.710 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "waiting for subops from 0,5", 2024-09-18T08:14:28.710 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:04:43.964697+0000", 2024-09-18T08:14:28.710 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.000517097 2024-09-18T08:14:28.710 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:14:28.710 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:14:28.710 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:14:28.710 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:04:43.970563+0000", 2024-09-18T08:14:28.710 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.0058661649999999996 2024-09-18T08:14:28.710 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:14:28.710 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:14:28.710 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "op_commit", 2024-09-18T08:14:28.710 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:04:43.970665+0000", 2024-09-18T08:14:28.710 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.000101888 2024-09-18T08:14:28.710 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:14:28.711 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:14:28.711 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:14:28.711 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:04:44.049154+0000", 2024-09-18T08:14:28.711 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.078489797 2024-09-18T08:14:28.711 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:14:28.711 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:14:28.711 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "commit_sent", 2024-09-18T08:14:28.711 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:04:44.049385+0000", 2024-09-18T08:14:28.711 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.00023076499999999999 2024-09-18T08:14:28.711 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:14:28.711 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:14:28.711 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "done", 2024-09-18T08:14:28.711 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:04:44.049890+0000", 2024-09-18T08:14:28.712 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.00050531299999999996 2024-09-18T08:14:28.712 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-09-18T08:14:28.712 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-09-18T08:14:28.712 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-09-18T08:14:28.712 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:14:28.712 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:14:28.712 INFO:teuthology.orchestra.run.smithi007.stdout: "description": "osd_op(client.14805.0:94181 7.e 7:72880d48:::benchmark_data_smithi007_73155_object94180:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e108)", 2024-09-18T08:14:28.712 INFO:teuthology.orchestra.run.smithi007.stdout: "initiated_at": "2024-09-18T08:05:09.961032+0000", 2024-09-18T08:14:28.712 INFO:teuthology.orchestra.run.smithi007.stdout: "age": 558.73864914800004, 2024-09-18T08:14:28.712 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.103403914, 2024-09-18T08:14:28.712 INFO:teuthology.orchestra.run.smithi007.stdout: "continuous": false, 2024-09-18T08:14:28.712 INFO:teuthology.orchestra.run.smithi007.stdout: "type_data": { 2024-09-18T08:14:28.712 INFO:teuthology.orchestra.run.smithi007.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T08:14:28.713 INFO:teuthology.orchestra.run.smithi007.stdout: "client_info": { 2024-09-18T08:14:28.713 INFO:teuthology.orchestra.run.smithi007.stdout: "client": "client.14805", 2024-09-18T08:14:28.713 INFO:teuthology.orchestra.run.smithi007.stdout: "client_addr": "172.21.15.7:0/640320436", 2024-09-18T08:14:28.713 INFO:teuthology.orchestra.run.smithi007.stdout: "tid": 94181 2024-09-18T08:14:28.713 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:14:28.713 INFO:teuthology.orchestra.run.smithi007.stdout: "events": [ 2024-09-18T08:14:28.713 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:14:28.713 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "initiated", 2024-09-18T08:14:28.713 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:05:09.961032+0000", 2024-09-18T08:14:28.713 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-09-18T08:14:28.713 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:14:28.713 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:14:28.713 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "header_read", 2024-09-18T08:14:28.713 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:05:09.961032+0000", 2024-09-18T08:14:28.714 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-09-18T08:14:28.714 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:14:28.714 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:14:28.714 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "throttled", 2024-09-18T08:14:28.714 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:05:09.961034+0000", 2024-09-18T08:14:28.714 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.342e-06 2024-09-18T08:14:28.714 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:14:28.714 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:14:28.714 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "all_read", 2024-09-18T08:14:28.714 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:05:09.961055+0000", 2024-09-18T08:14:28.714 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 2.1114000000000001e-05 2024-09-18T08:14:28.714 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:14:28.714 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:14:28.714 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "dispatched", 2024-09-18T08:14:28.715 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:05:09.961066+0000", 2024-09-18T08:14:28.715 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.1185999999999999e-05 2024-09-18T08:14:28.715 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:14:28.715 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:14:28.715 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "queued_for_pg", 2024-09-18T08:14:28.715 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:05:09.961073+0000", 2024-09-18T08:14:28.715 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 6.9630000000000001e-06 2024-09-18T08:14:28.715 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:14:28.715 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:14:28.715 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-09-18T08:14:28.715 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:05:09.961131+0000", 2024-09-18T08:14:28.715 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 5.7666e-05 2024-09-18T08:14:28.715 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:14:28.715 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:14:28.716 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "started", 2024-09-18T08:14:28.716 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:05:09.961241+0000", 2024-09-18T08:14:28.716 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.000110321 2024-09-18T08:14:28.716 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:14:28.716 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:14:28.716 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "waiting for subops from 0,5", 2024-09-18T08:14:28.716 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:05:09.961364+0000", 2024-09-18T08:14:28.716 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.00012286400000000001 2024-09-18T08:14:28.716 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:14:28.716 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:14:28.716 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:14:28.716 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:05:09.965873+0000", 2024-09-18T08:14:28.716 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.0045095930000000001 2024-09-18T08:14:28.717 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:14:28.717 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:14:28.717 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "op_commit", 2024-09-18T08:14:28.717 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:05:09.965897+0000", 2024-09-18T08:14:28.717 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 2.4077999999999999e-05 2024-09-18T08:14:28.717 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:14:28.717 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:14:28.717 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:14:28.717 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:05:10.064128+0000", 2024-09-18T08:14:28.717 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.098230856000000005 2024-09-18T08:14:28.717 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:14:28.717 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:14:28.717 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "commit_sent", 2024-09-18T08:14:28.717 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:05:10.064388+0000", 2024-09-18T08:14:28.718 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.00025948099999999999 2024-09-18T08:14:28.718 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:14:28.718 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:14:28.718 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "done", 2024-09-18T08:14:28.718 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:05:10.064436+0000", 2024-09-18T08:14:28.718 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 4.8449999999999999e-05 2024-09-18T08:14:28.718 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-09-18T08:14:28.718 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-09-18T08:14:28.718 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-09-18T08:14:28.718 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:14:28.718 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:14:28.718 INFO:teuthology.orchestra.run.smithi007.stdout: "description": "osd_op(client.14805.0:96235 7.e 7:70c09d0d:::benchmark_data_smithi007_73155_object96234:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e109)", 2024-09-18T08:14:28.718 INFO:teuthology.orchestra.run.smithi007.stdout: "initiated_at": "2024-09-18T08:05:12.494260+0000", 2024-09-18T08:14:28.719 INFO:teuthology.orchestra.run.smithi007.stdout: "age": 556.20542166500002, 2024-09-18T08:14:28.719 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.089927492999999997, 2024-09-18T08:14:28.719 INFO:teuthology.orchestra.run.smithi007.stdout: "continuous": false, 2024-09-18T08:14:28.719 INFO:teuthology.orchestra.run.smithi007.stdout: "type_data": { 2024-09-18T08:14:28.719 INFO:teuthology.orchestra.run.smithi007.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T08:14:28.719 INFO:teuthology.orchestra.run.smithi007.stdout: "client_info": { 2024-09-18T08:14:28.719 INFO:teuthology.orchestra.run.smithi007.stdout: "client": "client.14805", 2024-09-18T08:14:28.719 INFO:teuthology.orchestra.run.smithi007.stdout: "client_addr": "172.21.15.7:0/640320436", 2024-09-18T08:14:28.719 INFO:teuthology.orchestra.run.smithi007.stdout: "tid": 96235 2024-09-18T08:14:28.719 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:14:28.719 INFO:teuthology.orchestra.run.smithi007.stdout: "events": [ 2024-09-18T08:14:28.719 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:14:28.719 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "initiated", 2024-09-18T08:14:28.719 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:05:12.494260+0000", 2024-09-18T08:14:28.720 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-09-18T08:14:28.720 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:14:28.720 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:14:28.720 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "header_read", 2024-09-18T08:14:28.720 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:05:12.494260+0000", 2024-09-18T08:14:28.720 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-09-18T08:14:28.720 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:14:28.720 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:14:28.720 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "throttled", 2024-09-18T08:14:28.720 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:05:12.494351+0000", 2024-09-18T08:14:28.720 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 9.0839000000000002e-05 2024-09-18T08:14:28.720 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:14:28.720 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:14:28.721 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "all_read", 2024-09-18T08:14:28.721 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:05:12.494375+0000", 2024-09-18T08:14:28.721 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 2.4921000000000001e-05 2024-09-18T08:14:28.721 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:14:28.721 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:14:28.721 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "dispatched", 2024-09-18T08:14:28.721 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:05:12.494388+0000", 2024-09-18T08:14:28.721 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.2415000000000001e-05 2024-09-18T08:14:28.721 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:14:28.721 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:14:28.721 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "queued_for_pg", 2024-09-18T08:14:28.721 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:05:12.494396+0000", 2024-09-18T08:14:28.721 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 8.0770000000000008e-06 2024-09-18T08:14:28.721 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:14:28.722 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:14:28.722 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-09-18T08:14:28.722 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:05:12.500910+0000", 2024-09-18T08:14:28.722 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.0065145089999999999 2024-09-18T08:14:28.722 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:14:28.722 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:14:28.722 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "started", 2024-09-18T08:14:28.722 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:05:12.501024+0000", 2024-09-18T08:14:28.722 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.000113992 2024-09-18T08:14:28.722 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:14:28.722 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:14:28.722 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "waiting for subops from 0,5", 2024-09-18T08:14:28.722 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:05:12.501165+0000", 2024-09-18T08:14:28.723 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.000140347 2024-09-18T08:14:28.723 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:14:28.723 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:14:28.723 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:14:28.723 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:05:12.531672+0000", 2024-09-18T08:14:28.723 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.030506945000000001 2024-09-18T08:14:28.723 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:14:28.723 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:14:28.723 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "op_commit", 2024-09-18T08:14:28.723 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:05:12.531683+0000", 2024-09-18T08:14:28.723 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.1092e-05 2024-09-18T08:14:28.723 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:14:28.723 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:14:28.723 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:14:28.724 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:05:12.583879+0000", 2024-09-18T08:14:28.724 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.052195966000000003 2024-09-18T08:14:28.724 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:14:28.724 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:14:28.724 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "commit_sent", 2024-09-18T08:14:28.724 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:05:12.584048+0000", 2024-09-18T08:14:28.724 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.00016902 2024-09-18T08:14:28.724 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:14:28.724 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:14:28.724 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "done", 2024-09-18T08:14:28.724 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:05:12.584187+0000", 2024-09-18T08:14:28.724 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.00013936999999999999 2024-09-18T08:14:28.724 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-09-18T08:14:28.724 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-09-18T08:14:28.725 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-09-18T08:14:28.725 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:14:28.725 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:14:28.725 INFO:teuthology.orchestra.run.smithi007.stdout: "description": "osd_op(client.14805.0:96694 7.e 7:77dc6698:::benchmark_data_smithi007_73155_object96693:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e109)", 2024-09-18T08:14:28.725 INFO:teuthology.orchestra.run.smithi007.stdout: "initiated_at": "2024-09-18T08:05:13.227222+0000", 2024-09-18T08:14:28.725 INFO:teuthology.orchestra.run.smithi007.stdout: "age": 555.47245956500001, 2024-09-18T08:14:28.725 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.092647683999999994, 2024-09-18T08:14:28.725 INFO:teuthology.orchestra.run.smithi007.stdout: "continuous": false, 2024-09-18T08:14:28.725 INFO:teuthology.orchestra.run.smithi007.stdout: "type_data": { 2024-09-18T08:14:28.725 INFO:teuthology.orchestra.run.smithi007.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T08:14:28.725 INFO:teuthology.orchestra.run.smithi007.stdout: "client_info": { 2024-09-18T08:14:28.725 INFO:teuthology.orchestra.run.smithi007.stdout: "client": "client.14805", 2024-09-18T08:14:28.725 INFO:teuthology.orchestra.run.smithi007.stdout: "client_addr": "172.21.15.7:0/640320436", 2024-09-18T08:14:28.726 INFO:teuthology.orchestra.run.smithi007.stdout: "tid": 96694 2024-09-18T08:14:28.726 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:14:28.726 INFO:teuthology.orchestra.run.smithi007.stdout: "events": [ 2024-09-18T08:14:28.726 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:14:28.726 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "initiated", 2024-09-18T08:14:28.726 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:05:13.227222+0000", 2024-09-18T08:14:28.726 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-09-18T08:14:28.726 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:14:28.726 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:14:28.726 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "header_read", 2024-09-18T08:14:28.726 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:05:13.227222+0000", 2024-09-18T08:14:28.726 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-09-18T08:14:28.726 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:14:28.727 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:14:28.727 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "throttled", 2024-09-18T08:14:28.727 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:05:13.227224+0000", 2024-09-18T08:14:28.727 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 2.243e-06 2024-09-18T08:14:28.727 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:14:28.727 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:14:28.727 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "all_read", 2024-09-18T08:14:28.727 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:05:13.227249+0000", 2024-09-18T08:14:28.727 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 2.4820999999999998e-05 2024-09-18T08:14:28.727 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:14:28.727 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:14:28.727 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "dispatched", 2024-09-18T08:14:28.727 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:05:13.227262+0000", 2024-09-18T08:14:28.727 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.2918999999999999e-05 2024-09-18T08:14:28.728 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:14:28.728 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:14:28.728 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "queued_for_pg", 2024-09-18T08:14:28.728 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:05:13.227275+0000", 2024-09-18T08:14:28.728 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.3607e-05 2024-09-18T08:14:28.728 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:14:28.728 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:14:28.728 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-09-18T08:14:28.728 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:05:13.239760+0000", 2024-09-18T08:14:28.728 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.012484419 2024-09-18T08:14:28.728 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:14:28.728 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:14:28.728 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "started", 2024-09-18T08:14:28.728 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:05:13.239892+0000", 2024-09-18T08:14:28.729 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.00013202900000000001 2024-09-18T08:14:28.729 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:14:28.729 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:14:28.729 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "waiting for subops from 0,5", 2024-09-18T08:14:28.729 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:05:13.240049+0000", 2024-09-18T08:14:28.729 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.00015725599999999999 2024-09-18T08:14:28.729 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:14:28.729 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:14:28.729 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "op_commit", 2024-09-18T08:14:28.729 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:05:13.247290+0000", 2024-09-18T08:14:28.729 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.00724072 2024-09-18T08:14:28.729 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:14:28.729 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:14:28.730 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:14:28.730 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:05:13.247559+0000", 2024-09-18T08:14:28.730 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.00026900699999999998 2024-09-18T08:14:28.730 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:14:28.730 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:14:28.730 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:14:28.730 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:05:13.319747+0000", 2024-09-18T08:14:28.730 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.072187789000000002 2024-09-18T08:14:28.730 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:14:28.730 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:14:28.730 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "commit_sent", 2024-09-18T08:14:28.730 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:05:13.319810+0000", 2024-09-18T08:14:28.730 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 6.3139000000000007e-05 2024-09-18T08:14:28.730 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:14:28.731 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:14:28.731 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "done", 2024-09-18T08:14:28.731 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:05:13.319869+0000", 2024-09-18T08:14:28.731 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 5.9735000000000003e-05 2024-09-18T08:14:28.731 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-09-18T08:14:28.731 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-09-18T08:14:28.731 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-09-18T08:14:28.731 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:14:28.731 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:14:28.731 INFO:teuthology.orchestra.run.smithi007.stdout: "description": "osd_op(client.14805.0:96719 7.e 7:7f123a7e:::benchmark_data_smithi007_73155_object96718:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e109)", 2024-09-18T08:14:28.731 INFO:teuthology.orchestra.run.smithi007.stdout: "initiated_at": "2024-09-18T08:05:13.293471+0000", 2024-09-18T08:14:28.731 INFO:teuthology.orchestra.run.smithi007.stdout: "age": 555.40621077799995, 2024-09-18T08:14:28.731 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.091559383999999994, 2024-09-18T08:14:28.731 INFO:teuthology.orchestra.run.smithi007.stdout: "continuous": false, 2024-09-18T08:14:28.732 INFO:teuthology.orchestra.run.smithi007.stdout: "type_data": { 2024-09-18T08:14:28.732 INFO:teuthology.orchestra.run.smithi007.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T08:14:28.732 INFO:teuthology.orchestra.run.smithi007.stdout: "client_info": { 2024-09-18T08:14:28.732 INFO:teuthology.orchestra.run.smithi007.stdout: "client": "client.14805", 2024-09-18T08:14:28.732 INFO:teuthology.orchestra.run.smithi007.stdout: "client_addr": "172.21.15.7:0/640320436", 2024-09-18T08:14:28.732 INFO:teuthology.orchestra.run.smithi007.stdout: "tid": 96719 2024-09-18T08:14:28.732 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:14:28.732 INFO:teuthology.orchestra.run.smithi007.stdout: "events": [ 2024-09-18T08:14:28.732 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:14:28.732 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "initiated", 2024-09-18T08:14:28.732 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:05:13.293471+0000", 2024-09-18T08:14:28.732 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-09-18T08:14:28.732 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:14:28.732 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:14:28.733 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "header_read", 2024-09-18T08:14:28.733 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:05:13.293471+0000", 2024-09-18T08:14:28.733 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-09-18T08:14:28.733 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:14:28.733 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:14:28.733 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "throttled", 2024-09-18T08:14:28.733 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:05:13.293472+0000", 2024-09-18T08:14:28.734 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.482e-06 2024-09-18T08:14:28.734 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:14:28.734 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:14:28.734 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "all_read", 2024-09-18T08:14:28.734 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:05:13.293496+0000", 2024-09-18T08:14:28.734 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 2.3708000000000001e-05 2024-09-18T08:14:28.734 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:14:28.734 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:14:28.734 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "dispatched", 2024-09-18T08:14:28.734 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:05:13.293506+0000", 2024-09-18T08:14:28.734 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.0706e-05 2024-09-18T08:14:28.734 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:14:28.734 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:14:28.735 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "queued_for_pg", 2024-09-18T08:14:28.735 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:05:13.293514+0000", 2024-09-18T08:14:28.735 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 7.4449999999999999e-06 2024-09-18T08:14:28.735 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:14:28.735 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:14:28.735 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-09-18T08:14:28.735 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:05:13.320155+0000", 2024-09-18T08:14:28.735 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.026641436000000001 2024-09-18T08:14:28.735 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:14:28.735 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:14:28.735 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "started", 2024-09-18T08:14:28.735 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:05:13.320303+0000", 2024-09-18T08:14:28.736 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.00014768800000000001 2024-09-18T08:14:28.736 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:14:28.736 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:14:28.736 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "waiting for subops from 0,5", 2024-09-18T08:14:28.736 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:05:13.320454+0000", 2024-09-18T08:14:28.736 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.000151328 2024-09-18T08:14:28.736 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:14:28.736 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:14:28.736 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:14:28.736 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:05:13.326669+0000", 2024-09-18T08:14:28.736 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.0062146090000000003 2024-09-18T08:14:28.736 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:14:28.736 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:14:28.736 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "op_commit", 2024-09-18T08:14:28.737 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:05:13.326681+0000", 2024-09-18T08:14:28.737 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.2160999999999999e-05 2024-09-18T08:14:28.737 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:14:28.737 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:14:28.737 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:14:28.737 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:05:13.384909+0000", 2024-09-18T08:14:28.737 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.058227999000000003 2024-09-18T08:14:28.738 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:14:28.738 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:14:28.738 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "commit_sent", 2024-09-18T08:14:28.738 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:05:13.384968+0000", 2024-09-18T08:14:28.738 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 5.8479e-05 2024-09-18T08:14:28.738 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:14:28.738 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:14:28.738 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "done", 2024-09-18T08:14:28.739 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:05:13.385030+0000", 2024-09-18T08:14:28.739 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 6.2342999999999996e-05 2024-09-18T08:14:28.739 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-09-18T08:14:28.739 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-09-18T08:14:28.739 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-09-18T08:14:28.739 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:14:28.739 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:14:28.739 INFO:teuthology.orchestra.run.smithi007.stdout: "description": "osd_op(client.14805.0:141891 7.e 7:7d0cb9e1:::benchmark_data_smithi007_73155_object23600:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e127)", 2024-09-18T08:14:28.739 INFO:teuthology.orchestra.run.smithi007.stdout: "initiated_at": "2024-09-18T08:05:59.091427+0000", 2024-09-18T08:14:28.739 INFO:teuthology.orchestra.run.smithi007.stdout: "age": 509.60825413399999, 2024-09-18T08:14:28.739 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.090109581999999994, 2024-09-18T08:14:28.739 INFO:teuthology.orchestra.run.smithi007.stdout: "continuous": false, 2024-09-18T08:14:28.739 INFO:teuthology.orchestra.run.smithi007.stdout: "type_data": { 2024-09-18T08:14:28.740 INFO:teuthology.orchestra.run.smithi007.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T08:14:28.740 INFO:teuthology.orchestra.run.smithi007.stdout: "client_info": { 2024-09-18T08:14:28.740 INFO:teuthology.orchestra.run.smithi007.stdout: "client": "client.14805", 2024-09-18T08:14:28.740 INFO:teuthology.orchestra.run.smithi007.stdout: "client_addr": "172.21.15.7:0/640320436", 2024-09-18T08:14:28.740 INFO:teuthology.orchestra.run.smithi007.stdout: "tid": 141891 2024-09-18T08:14:28.740 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:14:28.740 INFO:teuthology.orchestra.run.smithi007.stdout: "events": [ 2024-09-18T08:14:28.740 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:14:28.740 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "initiated", 2024-09-18T08:14:28.740 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:05:59.091427+0000", 2024-09-18T08:14:28.740 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-09-18T08:14:28.740 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:14:28.740 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:14:28.741 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "header_read", 2024-09-18T08:14:28.741 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:05:59.091427+0000", 2024-09-18T08:14:28.741 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-09-18T08:14:28.741 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:14:28.741 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:14:28.741 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "throttled", 2024-09-18T08:14:28.741 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:05:59.091429+0000", 2024-09-18T08:14:28.741 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.719e-06 2024-09-18T08:14:28.741 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:14:28.741 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:14:28.741 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "all_read", 2024-09-18T08:14:28.741 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:05:59.091435+0000", 2024-09-18T08:14:28.741 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 5.8850000000000004e-06 2024-09-18T08:14:28.741 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:14:28.742 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:14:28.742 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "dispatched", 2024-09-18T08:14:28.742 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:05:59.091447+0000", 2024-09-18T08:14:28.742 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.2007e-05 2024-09-18T08:14:28.742 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:14:28.742 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:14:28.742 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "queued_for_pg", 2024-09-18T08:14:28.742 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:05:59.091455+0000", 2024-09-18T08:14:28.742 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 8.5490000000000006e-06 2024-09-18T08:14:28.742 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:14:28.742 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:14:28.742 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-09-18T08:14:28.742 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:05:59.091549+0000", 2024-09-18T08:14:28.742 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 9.3411000000000004e-05 2024-09-18T08:14:28.743 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:14:28.743 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:14:28.743 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "started", 2024-09-18T08:14:28.743 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:05:59.091625+0000", 2024-09-18T08:14:28.743 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 7.5745000000000001e-05 2024-09-18T08:14:28.743 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:14:28.743 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:14:28.743 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "waiting for subops from 0,5", 2024-09-18T08:14:28.743 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:05:59.091687+0000", 2024-09-18T08:14:28.743 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 6.2831999999999997e-05 2024-09-18T08:14:28.743 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:14:28.743 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:14:28.743 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "op_commit", 2024-09-18T08:14:28.743 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:05:59.094073+0000", 2024-09-18T08:14:28.743 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.0023852959999999999 2024-09-18T08:14:28.744 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:14:28.744 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:14:28.744 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:14:28.744 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:05:59.094439+0000", 2024-09-18T08:14:28.744 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.00036632800000000002 2024-09-18T08:14:28.744 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:14:28.744 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:14:28.744 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:14:28.744 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:05:59.181431+0000", 2024-09-18T08:14:28.744 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.086991857000000006 2024-09-18T08:14:28.744 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:14:28.744 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:14:28.744 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "commit_sent", 2024-09-18T08:14:28.745 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:05:59.181489+0000", 2024-09-18T08:14:28.745 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 5.8477999999999998e-05 2024-09-18T08:14:28.745 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:14:28.745 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:14:28.745 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "done", 2024-09-18T08:14:28.745 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:05:59.181537+0000", 2024-09-18T08:14:28.745 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 4.7475000000000002e-05 2024-09-18T08:14:28.745 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-09-18T08:14:28.745 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-09-18T08:14:28.745 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-09-18T08:14:28.746 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:14:28.746 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:14:28.746 INFO:teuthology.orchestra.run.smithi007.stdout: "description": "osd_op(client.14805.0:147830 7.e 7:7137fe31:::benchmark_data_smithi007_73155_object29539:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e127)", 2024-09-18T08:14:28.746 INFO:teuthology.orchestra.run.smithi007.stdout: "initiated_at": "2024-09-18T08:06:04.696707+0000", 2024-09-18T08:14:28.746 INFO:teuthology.orchestra.run.smithi007.stdout: "age": 504.00297420999999, 2024-09-18T08:14:28.746 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.093481047999999997, 2024-09-18T08:14:28.746 INFO:teuthology.orchestra.run.smithi007.stdout: "continuous": false, 2024-09-18T08:14:28.746 INFO:teuthology.orchestra.run.smithi007.stdout: "type_data": { 2024-09-18T08:14:28.746 INFO:teuthology.orchestra.run.smithi007.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T08:14:28.746 INFO:teuthology.orchestra.run.smithi007.stdout: "client_info": { 2024-09-18T08:14:28.746 INFO:teuthology.orchestra.run.smithi007.stdout: "client": "client.14805", 2024-09-18T08:14:28.746 INFO:teuthology.orchestra.run.smithi007.stdout: "client_addr": "172.21.15.7:0/640320436", 2024-09-18T08:14:28.746 INFO:teuthology.orchestra.run.smithi007.stdout: "tid": 147830 2024-09-18T08:14:28.746 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:14:28.747 INFO:teuthology.orchestra.run.smithi007.stdout: "events": [ 2024-09-18T08:14:28.747 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:14:28.747 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "initiated", 2024-09-18T08:14:28.747 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:06:04.696707+0000", 2024-09-18T08:14:28.747 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-09-18T08:14:28.747 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:14:28.747 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:14:28.747 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "header_read", 2024-09-18T08:14:28.747 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:06:04.696707+0000", 2024-09-18T08:14:28.747 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-09-18T08:14:28.747 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:14:28.747 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:14:28.747 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "throttled", 2024-09-18T08:14:28.747 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:06:04.696709+0000", 2024-09-18T08:14:28.748 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.4610000000000001e-06 2024-09-18T08:14:28.748 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:14:28.748 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:14:28.748 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "all_read", 2024-09-18T08:14:28.748 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:06:04.696714+0000", 2024-09-18T08:14:28.748 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 5.4929999999999998e-06 2024-09-18T08:14:28.748 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:14:28.748 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:14:28.748 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "dispatched", 2024-09-18T08:14:28.748 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:06:04.696726+0000", 2024-09-18T08:14:28.748 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.1405e-05 2024-09-18T08:14:28.748 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:14:28.748 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:14:28.748 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "queued_for_pg", 2024-09-18T08:14:28.749 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:14:28.749 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:14:28.750 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0' 2024-09-18T08:14:28.750 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:06:04.696733+0000", 2024-09-18T08:14:28.751 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 7.5440000000000001e-06 2024-09-18T08:14:28.751 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:14:28.751 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:14:28.752 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-09-18T08:14:28.752 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:06:04.696834+0000", 2024-09-18T08:14:28.752 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.00010042799999999999 2024-09-18T08:14:28.752 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:14:28.752 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:14:28.752 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "started", 2024-09-18T08:14:28.752 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:06:04.696964+0000", 2024-09-18T08:14:28.752 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.00013050599999999999 2024-09-18T08:14:28.752 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:14:28.752 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:14:28.752 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "waiting for subops from 0,5", 2024-09-18T08:14:28.752 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:06:04.697079+0000", 2024-09-18T08:14:28.752 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.000114769 2024-09-18T08:14:28.753 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:14:28.753 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:14:28.753 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:14:28.753 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:06:04.703144+0000", 2024-09-18T08:14:28.753 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.0060647729999999999 2024-09-18T08:14:28.753 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:14:28.753 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:14:28.753 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "op_commit", 2024-09-18T08:14:28.753 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:06:04.703157+0000", 2024-09-18T08:14:28.753 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.3434e-05 2024-09-18T08:14:28.753 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:14:28.753 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:14:28.753 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:14:28.754 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:06:04.789746+0000", 2024-09-18T08:14:28.754 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.086589049000000001 2024-09-18T08:14:28.754 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:14:28.754 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:14:28.754 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "commit_sent", 2024-09-18T08:14:28.754 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:06:04.789946+0000", 2024-09-18T08:14:28.754 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.000199996 2024-09-18T08:14:28.754 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:14:28.754 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:14:28.754 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "done", 2024-09-18T08:14:28.754 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:06:04.790188+0000", 2024-09-18T08:14:28.754 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.00024219000000000001 2024-09-18T08:14:28.754 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-09-18T08:14:28.754 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-09-18T08:14:28.755 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-09-18T08:14:28.755 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:14:28.755 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:14:28.755 INFO:teuthology.orchestra.run.smithi007.stdout: "description": "osd_op(client.14805.0:147846 7.e 7:77e37675:::benchmark_data_smithi007_73155_object29555:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e127)", 2024-09-18T08:14:28.755 INFO:teuthology.orchestra.run.smithi007.stdout: "initiated_at": "2024-09-18T08:06:04.708478+0000", 2024-09-18T08:14:28.755 INFO:teuthology.orchestra.run.smithi007.stdout: "age": 503.991203025, 2024-09-18T08:14:28.755 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.089318937000000001, 2024-09-18T08:14:28.755 INFO:teuthology.orchestra.run.smithi007.stdout: "continuous": false, 2024-09-18T08:14:28.755 INFO:teuthology.orchestra.run.smithi007.stdout: "type_data": { 2024-09-18T08:14:28.755 INFO:teuthology.orchestra.run.smithi007.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T08:14:28.755 INFO:teuthology.orchestra.run.smithi007.stdout: "client_info": { 2024-09-18T08:14:28.755 INFO:teuthology.orchestra.run.smithi007.stdout: "client": "client.14805", 2024-09-18T08:14:28.755 INFO:teuthology.orchestra.run.smithi007.stdout: "client_addr": "172.21.15.7:0/640320436", 2024-09-18T08:14:28.755 INFO:teuthology.orchestra.run.smithi007.stdout: "tid": 147846 2024-09-18T08:14:28.756 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:14:28.756 INFO:teuthology.orchestra.run.smithi007.stdout: "events": [ 2024-09-18T08:14:28.756 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:14:28.756 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "initiated", 2024-09-18T08:14:28.756 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:06:04.708478+0000", 2024-09-18T08:14:28.756 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-09-18T08:14:28.756 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:14:28.756 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:14:28.756 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "header_read", 2024-09-18T08:14:28.756 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:06:04.708478+0000", 2024-09-18T08:14:28.756 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-09-18T08:14:28.756 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:14:28.756 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:14:28.757 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "throttled", 2024-09-18T08:14:28.757 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:06:04.708480+0000", 2024-09-18T08:14:28.757 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.3799999999999999e-06 2024-09-18T08:14:28.757 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:14:28.757 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:14:28.757 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "all_read", 2024-09-18T08:14:28.757 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:06:04.708485+0000", 2024-09-18T08:14:28.757 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 5.5140000000000001e-06 2024-09-18T08:14:28.757 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:14:28.757 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:14:28.757 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "dispatched", 2024-09-18T08:14:28.757 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:06:04.708496+0000", 2024-09-18T08:14:28.757 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.0468e-05 2024-09-18T08:14:28.758 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:14:28.758 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:14:28.758 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "queued_for_pg", 2024-09-18T08:14:28.758 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:06:04.708503+0000", 2024-09-18T08:14:28.758 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 6.9709999999999998e-06 2024-09-18T08:14:28.758 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:14:28.758 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:14:28.758 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-09-18T08:14:28.758 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:06:04.708559+0000", 2024-09-18T08:14:28.758 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 5.5967999999999997e-05 2024-09-18T08:14:28.758 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:14:28.758 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:14:28.758 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "started", 2024-09-18T08:14:28.758 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:06:04.708664+0000", 2024-09-18T08:14:28.759 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.000104956 2024-09-18T08:14:28.759 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:14:28.759 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:14:28.759 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "waiting for subops from 0,5", 2024-09-18T08:14:28.759 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:06:04.708767+0000", 2024-09-18T08:14:28.759 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.000102889 2024-09-18T08:14:28.759 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:14:28.759 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:14:28.759 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:14:28.759 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:06:04.781177+0000", 2024-09-18T08:14:28.759 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.072410522000000005 2024-09-18T08:14:28.759 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:14:28.759 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:14:28.760 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "op_commit", 2024-09-18T08:14:28.760 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:06:04.782476+0000", 2024-09-18T08:14:28.760 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.0012993150000000001 2024-09-18T08:14:28.760 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:14:28.760 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:14:28.760 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:14:28.760 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:06:04.797423+0000", 2024-09-18T08:14:28.760 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.014946213999999999 2024-09-18T08:14:28.760 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:14:28.760 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:14:28.760 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "commit_sent", 2024-09-18T08:14:28.760 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:06:04.797637+0000", 2024-09-18T08:14:28.760 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.000214353 2024-09-18T08:14:28.760 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:14:28.761 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:14:28.761 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "done", 2024-09-18T08:14:28.761 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:06:04.797797+0000", 2024-09-18T08:14:28.761 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.00016038700000000001 2024-09-18T08:14:28.761 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-09-18T08:14:28.761 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-09-18T08:14:28.761 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-09-18T08:14:28.761 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:14:28.761 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:14:28.761 INFO:teuthology.orchestra.run.smithi007.stdout: "description": "osd_op(client.14805.0:147849 7.e 7:7e6f39e7:::benchmark_data_smithi007_73155_object29558:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e127)", 2024-09-18T08:14:28.761 INFO:teuthology.orchestra.run.smithi007.stdout: "initiated_at": "2024-09-18T08:06:04.708908+0000", 2024-09-18T08:14:28.761 INFO:teuthology.orchestra.run.smithi007.stdout: "age": 503.990773741, 2024-09-18T08:14:28.761 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.089203497000000007, 2024-09-18T08:14:28.761 INFO:teuthology.orchestra.run.smithi007.stdout: "continuous": false, 2024-09-18T08:14:28.762 INFO:teuthology.orchestra.run.smithi007.stdout: "type_data": { 2024-09-18T08:14:28.762 INFO:teuthology.orchestra.run.smithi007.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T08:14:28.762 INFO:teuthology.orchestra.run.smithi007.stdout: "client_info": { 2024-09-18T08:14:28.762 INFO:teuthology.orchestra.run.smithi007.stdout: "client": "client.14805", 2024-09-18T08:14:28.762 INFO:teuthology.orchestra.run.smithi007.stdout: "client_addr": "172.21.15.7:0/640320436", 2024-09-18T08:14:28.762 INFO:teuthology.orchestra.run.smithi007.stdout: "tid": 147849 2024-09-18T08:14:28.762 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:14:28.762 INFO:teuthology.orchestra.run.smithi007.stdout: "events": [ 2024-09-18T08:14:28.762 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:14:28.762 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "initiated", 2024-09-18T08:14:28.762 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:06:04.708908+0000", 2024-09-18T08:14:28.762 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-09-18T08:14:28.762 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:14:28.762 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:14:28.762 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "header_read", 2024-09-18T08:14:28.763 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:06:04.708908+0000", 2024-09-18T08:14:28.763 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-09-18T08:14:28.763 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:14:28.763 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:14:28.763 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "throttled", 2024-09-18T08:14:28.763 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:06:04.708908+0000", 2024-09-18T08:14:28.763 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 7.6199999999999997e-07 2024-09-18T08:14:28.763 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:14:28.763 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:14:28.763 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "all_read", 2024-09-18T08:14:28.763 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:06:04.708913+0000", 2024-09-18T08:14:28.763 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 4.1049999999999997e-06 2024-09-18T08:14:28.763 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:14:28.763 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:14:28.764 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "dispatched", 2024-09-18T08:14:28.764 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:06:04.708922+0000", 2024-09-18T08:14:28.764 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 9.2930000000000006e-06 2024-09-18T08:14:28.764 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:14:28.764 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:14:28.764 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "queued_for_pg", 2024-09-18T08:14:28.764 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:06:04.708928+0000", 2024-09-18T08:14:28.764 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 6.4919999999999998e-06 2024-09-18T08:14:28.764 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:14:28.764 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:14:28.764 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-09-18T08:14:28.764 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:06:04.712299+0000", 2024-09-18T08:14:28.764 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.0033709510000000001 2024-09-18T08:14:28.764 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:14:28.765 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:14:28.765 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "started", 2024-09-18T08:14:28.765 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:06:04.712407+0000", 2024-09-18T08:14:28.765 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.000108229 2024-09-18T08:14:28.765 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:14:28.765 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:14:28.765 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "waiting for subops from 0,5", 2024-09-18T08:14:28.765 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:06:04.712519+0000", 2024-09-18T08:14:28.765 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.00011200399999999999 2024-09-18T08:14:28.765 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:14:28.765 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:14:28.765 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:14:28.765 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:06:04.782460+0000", 2024-09-18T08:14:28.766 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.069940536999999997 2024-09-18T08:14:28.766 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:14:28.766 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:14:28.766 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "op_commit", 2024-09-18T08:14:28.766 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:06:04.782500+0000", 2024-09-18T08:14:28.766 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 4.0463000000000002e-05 2024-09-18T08:14:28.766 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:14:28.766 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:14:28.766 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:14:28.766 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:06:04.797935+0000", 2024-09-18T08:14:28.766 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.015434188 2024-09-18T08:14:28.766 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:14:28.766 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:14:28.766 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "commit_sent", 2024-09-18T08:14:28.767 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:06:04.798073+0000", 2024-09-18T08:14:28.767 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.00013849000000000001 2024-09-18T08:14:28.767 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:14:28.767 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:14:28.767 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "done", 2024-09-18T08:14:28.767 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:06:04.798111+0000", 2024-09-18T08:14:28.767 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 3.7982999999999999e-05 2024-09-18T08:14:28.767 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-09-18T08:14:28.767 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-09-18T08:14:28.767 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-09-18T08:14:28.767 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:14:28.767 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:14:28.767 INFO:teuthology.orchestra.run.smithi007.stdout: "description": "osd_op(client.14805.0:177709 7.e 7:780b878d:::benchmark_data_smithi007_73155_object59418:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e139)", 2024-09-18T08:14:28.768 INFO:teuthology.orchestra.run.smithi007.stdout: "initiated_at": "2024-09-18T08:06:38.569829+0000", 2024-09-18T08:14:28.768 INFO:teuthology.orchestra.run.smithi007.stdout: "age": 470.12985219500001, 2024-09-18T08:14:28.768 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.111834402, 2024-09-18T08:14:28.768 INFO:teuthology.orchestra.run.smithi007.stdout: "continuous": false, 2024-09-18T08:14:28.768 INFO:teuthology.orchestra.run.smithi007.stdout: "type_data": { 2024-09-18T08:14:28.768 INFO:teuthology.orchestra.run.smithi007.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T08:14:28.768 INFO:teuthology.orchestra.run.smithi007.stdout: "client_info": { 2024-09-18T08:14:28.768 INFO:teuthology.orchestra.run.smithi007.stdout: "client": "client.14805", 2024-09-18T08:14:28.768 INFO:teuthology.orchestra.run.smithi007.stdout: "client_addr": "172.21.15.7:0/640320436", 2024-09-18T08:14:28.768 INFO:teuthology.orchestra.run.smithi007.stdout: "tid": 177709 2024-09-18T08:14:28.768 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:14:28.768 INFO:teuthology.orchestra.run.smithi007.stdout: "events": [ 2024-09-18T08:14:28.768 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:14:28.769 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "initiated", 2024-09-18T08:14:28.769 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:06:38.569829+0000", 2024-09-18T08:14:28.769 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-09-18T08:14:28.769 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:14:28.769 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:14:28.769 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "header_read", 2024-09-18T08:14:28.769 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:06:38.569829+0000", 2024-09-18T08:14:28.769 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-09-18T08:14:28.769 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:14:28.769 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:14:28.769 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "throttled", 2024-09-18T08:14:28.769 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:06:38.569831+0000", 2024-09-18T08:14:28.769 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.451e-06 2024-09-18T08:14:28.770 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:14:28.770 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:14:28.770 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "all_read", 2024-09-18T08:14:28.770 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:06:38.569836+0000", 2024-09-18T08:14:28.770 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 5.4269999999999999e-06 2024-09-18T08:14:28.770 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:14:28.770 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:14:28.770 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "dispatched", 2024-09-18T08:14:28.770 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:06:38.569852+0000", 2024-09-18T08:14:28.770 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.6375000000000002e-05 2024-09-18T08:14:28.770 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:14:28.770 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:14:28.770 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "queued_for_pg", 2024-09-18T08:14:28.770 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:06:38.569861+0000", 2024-09-18T08:14:28.771 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 8.2749999999999995e-06 2024-09-18T08:14:28.771 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:14:28.771 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:14:28.771 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-09-18T08:14:28.771 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:06:38.571353+0000", 2024-09-18T08:14:28.771 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.0014927650000000001 2024-09-18T08:14:28.771 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:14:28.771 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:14:28.771 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "started", 2024-09-18T08:14:28.771 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:06:38.571454+0000", 2024-09-18T08:14:28.771 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.000100065 2024-09-18T08:14:28.771 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:14:28.771 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:14:28.772 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "waiting for subops from 0,5", 2024-09-18T08:14:28.772 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:06:38.571551+0000", 2024-09-18T08:14:28.772 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 9.7884e-05 2024-09-18T08:14:28.772 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:14:28.772 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:14:28.772 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:14:28.772 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:06:38.576046+0000", 2024-09-18T08:14:28.772 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.0044943379999999996 2024-09-18T08:14:28.772 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:14:28.772 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:14:28.772 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "op_commit", 2024-09-18T08:14:28.772 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:06:38.576188+0000", 2024-09-18T08:14:28.772 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.00014268500000000001 2024-09-18T08:14:28.772 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:14:28.773 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:14:28.773 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:14:28.773 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:06:38.681566+0000", 2024-09-18T08:14:28.773 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.10537708 2024-09-18T08:14:28.773 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:14:28.773 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:14:28.773 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "commit_sent", 2024-09-18T08:14:28.773 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:06:38.681619+0000", 2024-09-18T08:14:28.773 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 5.3919999999999999e-05 2024-09-18T08:14:28.773 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:14:28.773 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:14:28.773 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "done", 2024-09-18T08:14:28.773 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:06:38.681664+0000", 2024-09-18T08:14:28.774 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 4.4137000000000001e-05 2024-09-18T08:14:28.774 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-09-18T08:14:28.774 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-09-18T08:14:28.774 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-09-18T08:14:28.774 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:14:28.774 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:14:28.774 INFO:teuthology.orchestra.run.smithi007.stdout: "description": "osd_op(client.14805.0:204554 7.e 7:75af683c:::benchmark_data_smithi007_73155_object86263:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e148)", 2024-09-18T08:14:28.774 INFO:teuthology.orchestra.run.smithi007.stdout: "initiated_at": "2024-09-18T08:07:06.550254+0000", 2024-09-18T08:14:28.774 INFO:teuthology.orchestra.run.smithi007.stdout: "age": 442.14942716799999, 2024-09-18T08:14:28.774 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.090495268000000004, 2024-09-18T08:14:28.774 INFO:teuthology.orchestra.run.smithi007.stdout: "continuous": false, 2024-09-18T08:14:28.774 INFO:teuthology.orchestra.run.smithi007.stdout: "type_data": { 2024-09-18T08:14:28.774 INFO:teuthology.orchestra.run.smithi007.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T08:14:28.774 INFO:teuthology.orchestra.run.smithi007.stdout: "client_info": { 2024-09-18T08:14:28.775 INFO:teuthology.orchestra.run.smithi007.stdout: "client": "client.14805", 2024-09-18T08:14:28.775 INFO:teuthology.orchestra.run.smithi007.stdout: "client_addr": "172.21.15.7:0/640320436", 2024-09-18T08:14:28.775 INFO:teuthology.orchestra.run.smithi007.stdout: "tid": 204554 2024-09-18T08:14:28.775 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:14:28.775 INFO:teuthology.orchestra.run.smithi007.stdout: "events": [ 2024-09-18T08:14:28.775 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:14:28.775 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "initiated", 2024-09-18T08:14:28.775 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:07:06.550254+0000", 2024-09-18T08:14:28.775 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-09-18T08:14:28.775 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:14:28.775 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:14:28.775 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "header_read", 2024-09-18T08:14:28.775 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:07:06.550254+0000", 2024-09-18T08:14:28.775 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-09-18T08:14:28.776 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:14:28.776 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:14:28.776 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "throttled", 2024-09-18T08:14:28.776 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:07:06.550255+0000", 2024-09-18T08:14:28.776 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.04e-06 2024-09-18T08:14:28.776 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:14:28.776 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:14:28.776 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "all_read", 2024-09-18T08:14:28.776 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:07:06.550260+0000", 2024-09-18T08:14:28.776 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 4.4939999999999997e-06 2024-09-18T08:14:28.776 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:14:28.776 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:14:28.776 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "dispatched", 2024-09-18T08:14:28.777 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:07:06.550272+0000", 2024-09-18T08:14:28.777 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.2672e-05 2024-09-18T08:14:28.777 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:14:28.777 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:14:28.777 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "queued_for_pg", 2024-09-18T08:14:28.777 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:07:06.550279+0000", 2024-09-18T08:14:28.777 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 7.0099999999999998e-06 2024-09-18T08:14:28.777 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:14:28.777 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:14:28.777 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-09-18T08:14:28.777 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:07:06.553248+0000", 2024-09-18T08:14:28.777 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.0029683999999999999 2024-09-18T08:14:28.777 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:14:28.777 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:14:28.778 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "started", 2024-09-18T08:14:28.778 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:07:06.553346+0000", 2024-09-18T08:14:28.778 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 9.8618999999999999e-05 2024-09-18T08:14:28.778 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:14:28.778 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:14:28.778 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "waiting for subops from 0,5", 2024-09-18T08:14:28.778 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:07:06.553450+0000", 2024-09-18T08:14:28.778 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.00010311599999999999 2024-09-18T08:14:28.778 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:14:28.779 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:14:28.779 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "op_commit", 2024-09-18T08:14:28.779 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:07:06.560605+0000", 2024-09-18T08:14:28.779 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.0071556229999999998 2024-09-18T08:14:28.779 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:14:28.779 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:14:28.779 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:14:28.779 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:07:06.565908+0000", 2024-09-18T08:14:28.779 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.0053023480000000001 2024-09-18T08:14:28.779 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:14:28.779 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:14:28.779 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:14:28.779 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:07:06.640602+0000", 2024-09-18T08:14:28.780 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.074694521 2024-09-18T08:14:28.780 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:14:28.780 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:14:28.781 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "commit_sent", 2024-09-18T08:14:28.781 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:07:06.640706+0000", 2024-09-18T08:14:28.781 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.00010401800000000001 2024-09-18T08:14:28.781 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:14:28.781 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:14:28.781 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "done", 2024-09-18T08:14:28.781 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:07:06.640749+0000", 2024-09-18T08:14:28.781 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 4.3407e-05 2024-09-18T08:14:28.781 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-09-18T08:14:28.781 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-09-18T08:14:28.781 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-09-18T08:14:28.782 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:14:28.782 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:14:28.782 INFO:teuthology.orchestra.run.smithi007.stdout: "description": "osd_op(client.14805.0:204569 7.5 7:a57af9cd:::benchmark_data_smithi007_73155_object86278:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e148)", 2024-09-18T08:14:28.782 INFO:teuthology.orchestra.run.smithi007.stdout: "initiated_at": "2024-09-18T08:07:06.563553+0000", 2024-09-18T08:14:28.782 INFO:teuthology.orchestra.run.smithi007.stdout: "age": 442.136128085, 2024-09-18T08:14:28.782 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.089587238, 2024-09-18T08:14:28.782 INFO:teuthology.orchestra.run.smithi007.stdout: "continuous": false, 2024-09-18T08:14:28.782 INFO:teuthology.orchestra.run.smithi007.stdout: "type_data": { 2024-09-18T08:14:28.782 INFO:teuthology.orchestra.run.smithi007.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T08:14:28.782 INFO:teuthology.orchestra.run.smithi007.stdout: "client_info": { 2024-09-18T08:14:28.782 INFO:teuthology.orchestra.run.smithi007.stdout: "client": "client.14805", 2024-09-18T08:14:28.782 INFO:teuthology.orchestra.run.smithi007.stdout: "client_addr": "172.21.15.7:0/640320436", 2024-09-18T08:14:28.782 INFO:teuthology.orchestra.run.smithi007.stdout: "tid": 204569 2024-09-18T08:14:28.783 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:14:28.783 INFO:teuthology.orchestra.run.smithi007.stdout: "events": [ 2024-09-18T08:14:28.783 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:14:28.783 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "initiated", 2024-09-18T08:14:28.783 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:07:06.563553+0000", 2024-09-18T08:14:28.783 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-09-18T08:14:28.783 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:14:28.783 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:14:28.783 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "header_read", 2024-09-18T08:14:28.783 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:07:06.563553+0000", 2024-09-18T08:14:28.784 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-09-18T08:14:28.784 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:14:28.784 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:14:28.784 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "throttled", 2024-09-18T08:14:28.784 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:07:06.563554+0000", 2024-09-18T08:14:28.784 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 4.8100000000000003e-07 2024-09-18T08:14:28.784 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:14:28.784 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:14:28.784 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "all_read", 2024-09-18T08:14:28.784 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:07:06.563557+0000", 2024-09-18T08:14:28.784 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 2.7839999999999999e-06 2024-09-18T08:14:28.784 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:14:28.784 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:14:28.785 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "dispatched", 2024-09-18T08:14:28.785 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:07:06.563562+0000", 2024-09-18T08:14:28.785 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 5.7289999999999997e-06 2024-09-18T08:14:28.785 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:14:28.785 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:14:28.785 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "queued_for_pg", 2024-09-18T08:14:28.785 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:07:06.563567+0000", 2024-09-18T08:14:28.785 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 4.9949999999999996e-06 2024-09-18T08:14:28.785 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:14:28.785 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:14:28.785 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-09-18T08:14:28.785 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:07:06.563655+0000", 2024-09-18T08:14:28.785 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 8.7689999999999996e-05 2024-09-18T08:14:28.786 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:14:28.786 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:14:28.786 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "started", 2024-09-18T08:14:28.786 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:07:06.563758+0000", 2024-09-18T08:14:28.786 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.000102538 2024-09-18T08:14:28.786 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:14:28.786 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:14:28.786 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "waiting for subops from 0,7", 2024-09-18T08:14:28.786 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:07:06.563861+0000", 2024-09-18T08:14:28.786 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.00010385799999999999 2024-09-18T08:14:28.786 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:14:28.786 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:14:28.786 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:14:28.787 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:07:06.570416+0000", 2024-09-18T08:14:28.787 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.0065544449999999999 2024-09-18T08:14:28.787 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:14:28.787 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:14:28.787 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "op_commit", 2024-09-18T08:14:28.787 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:07:06.570428+0000", 2024-09-18T08:14:28.787 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.2537999999999999e-05 2024-09-18T08:14:28.787 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:14:28.787 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:14:28.787 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:14:28.788 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:07:06.649777+0000", 2024-09-18T08:14:28.788 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.079348518000000007 2024-09-18T08:14:28.788 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:14:28.788 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:14:28.788 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "commit_sent", 2024-09-18T08:14:28.788 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:07:06.653092+0000", 2024-09-18T08:14:28.788 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.0033149939999999999 2024-09-18T08:14:28.788 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:14:28.788 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:14:28.788 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "done", 2024-09-18T08:14:28.788 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:07:06.653141+0000", 2024-09-18T08:14:28.788 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 4.8668000000000002e-05 2024-09-18T08:14:28.789 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-09-18T08:14:28.789 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-09-18T08:14:28.789 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-09-18T08:14:28.789 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:14:28.789 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:14:28.789 INFO:teuthology.orchestra.run.smithi007.stdout: "description": "osd_op(client.14805.0:223793 7.5 7:a37824ff:::benchmark_data_smithi007_73155_object105502:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e154)", 2024-09-18T08:14:28.789 INFO:teuthology.orchestra.run.smithi007.stdout: "initiated_at": "2024-09-18T08:07:26.183612+0000", 2024-09-18T08:14:28.789 INFO:teuthology.orchestra.run.smithi007.stdout: "age": 422.516069609, 2024-09-18T08:14:28.789 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.104886888, 2024-09-18T08:14:28.789 INFO:teuthology.orchestra.run.smithi007.stdout: "continuous": false, 2024-09-18T08:14:28.789 INFO:teuthology.orchestra.run.smithi007.stdout: "type_data": { 2024-09-18T08:14:28.789 INFO:teuthology.orchestra.run.smithi007.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T08:14:28.789 INFO:teuthology.orchestra.run.smithi007.stdout: "client_info": { 2024-09-18T08:14:28.790 INFO:teuthology.orchestra.run.smithi007.stdout: "client": "client.14805", 2024-09-18T08:14:28.790 INFO:teuthology.orchestra.run.smithi007.stdout: "client_addr": "172.21.15.7:0/640320436", 2024-09-18T08:14:28.790 INFO:teuthology.orchestra.run.smithi007.stdout: "tid": 223793 2024-09-18T08:14:28.790 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:14:28.790 INFO:teuthology.orchestra.run.smithi007.stdout: "events": [ 2024-09-18T08:14:28.790 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:14:28.790 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "initiated", 2024-09-18T08:14:28.790 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:07:26.183612+0000", 2024-09-18T08:14:28.790 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-09-18T08:14:28.790 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:14:28.790 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:14:28.790 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "header_read", 2024-09-18T08:14:28.790 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:07:26.183612+0000", 2024-09-18T08:14:28.791 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-09-18T08:14:28.791 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:14:28.791 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:14:28.791 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "throttled", 2024-09-18T08:14:28.791 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:07:26.183613+0000", 2024-09-18T08:14:28.791 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.3030000000000001e-06 2024-09-18T08:14:28.791 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:14:28.791 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:14:28.791 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "all_read", 2024-09-18T08:14:28.791 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:07:26.183618+0000", 2024-09-18T08:14:28.791 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 4.7870000000000001e-06 2024-09-18T08:14:28.791 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:14:28.791 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:14:28.792 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "dispatched", 2024-09-18T08:14:28.792 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:07:26.183629+0000", 2024-09-18T08:14:28.792 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.1100999999999999e-05 2024-09-18T08:14:28.792 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:14:28.792 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:14:28.792 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "queued_for_pg", 2024-09-18T08:14:28.792 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:07:26.183636+0000", 2024-09-18T08:14:28.792 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 7.3980000000000002e-06 2024-09-18T08:14:28.792 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:14:28.792 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:14:28.792 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-09-18T08:14:28.792 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:07:26.183957+0000", 2024-09-18T08:14:28.792 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.00032060999999999998 2024-09-18T08:14:28.793 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:14:28.793 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:14:28.793 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "started", 2024-09-18T08:14:28.793 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:07:26.184232+0000", 2024-09-18T08:14:28.793 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.00027534499999999997 2024-09-18T08:14:28.793 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:14:28.793 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:14:28.793 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "waiting for subops from 0,7", 2024-09-18T08:14:28.793 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:07:26.184350+0000", 2024-09-18T08:14:28.793 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.00011803600000000001 2024-09-18T08:14:28.793 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:14:28.793 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:14:28.793 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:14:28.794 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:07:26.235645+0000", 2024-09-18T08:14:28.794 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.051294868 2024-09-18T08:14:28.794 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:14:28.794 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:14:28.794 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "op_commit", 2024-09-18T08:14:28.794 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:07:26.235664+0000", 2024-09-18T08:14:28.794 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.9236e-05 2024-09-18T08:14:28.794 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:14:28.794 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:14:28.794 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:14:28.794 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:07:26.288261+0000", 2024-09-18T08:14:28.794 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.052596716000000002 2024-09-18T08:14:28.794 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:14:28.795 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:14:28.795 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "commit_sent", 2024-09-18T08:14:28.795 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:07:26.288452+0000", 2024-09-18T08:14:28.795 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.00019043700000000001 2024-09-18T08:14:28.795 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:14:28.795 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:14:28.795 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "done", 2024-09-18T08:14:28.795 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:07:26.288499+0000", 2024-09-18T08:14:28.795 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 4.7051000000000001e-05 2024-09-18T08:14:28.795 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-09-18T08:14:28.795 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-09-18T08:14:28.795 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-09-18T08:14:28.795 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:14:28.796 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:14:28.796 INFO:teuthology.orchestra.run.smithi007.stdout: "description": "osd_op(client.14805.0:223799 7.e 7:7f447ed9:::benchmark_data_smithi007_73155_object105508:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e154)", 2024-09-18T08:14:28.796 INFO:teuthology.orchestra.run.smithi007.stdout: "initiated_at": "2024-09-18T08:07:26.184566+0000", 2024-09-18T08:14:28.796 INFO:teuthology.orchestra.run.smithi007.stdout: "age": 422.515115643, 2024-09-18T08:14:28.796 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.114658156, 2024-09-18T08:14:28.796 INFO:teuthology.orchestra.run.smithi007.stdout: "continuous": false, 2024-09-18T08:14:28.796 INFO:teuthology.orchestra.run.smithi007.stdout: "type_data": { 2024-09-18T08:14:28.796 INFO:teuthology.orchestra.run.smithi007.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T08:14:28.796 INFO:teuthology.orchestra.run.smithi007.stdout: "client_info": { 2024-09-18T08:14:28.796 INFO:teuthology.orchestra.run.smithi007.stdout: "client": "client.14805", 2024-09-18T08:14:28.796 INFO:teuthology.orchestra.run.smithi007.stdout: "client_addr": "172.21.15.7:0/640320436", 2024-09-18T08:14:28.796 INFO:teuthology.orchestra.run.smithi007.stdout: "tid": 223799 2024-09-18T08:14:28.796 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:14:28.797 INFO:teuthology.orchestra.run.smithi007.stdout: "events": [ 2024-09-18T08:14:28.797 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:14:28.797 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "initiated", 2024-09-18T08:14:28.797 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:07:26.184566+0000", 2024-09-18T08:14:28.797 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-09-18T08:14:28.797 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:14:28.797 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:14:28.797 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "header_read", 2024-09-18T08:14:28.797 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:07:26.184566+0000", 2024-09-18T08:14:28.797 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-09-18T08:14:28.797 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:14:28.797 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:14:28.797 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "throttled", 2024-09-18T08:14:28.798 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:07:26.184593+0000", 2024-09-18T08:14:28.798 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 2.6831000000000001e-05 2024-09-18T08:14:28.798 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:14:28.798 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:14:28.798 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "all_read", 2024-09-18T08:14:28.798 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:07:26.184597+0000", 2024-09-18T08:14:28.798 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 4.8119999999999998e-06 2024-09-18T08:14:28.798 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:14:28.798 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:14:28.798 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "dispatched", 2024-09-18T08:14:28.798 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:07:26.184607+0000", 2024-09-18T08:14:28.798 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 9.9559999999999992e-06 2024-09-18T08:14:28.798 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:14:28.799 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:14:28.799 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "queued_for_pg", 2024-09-18T08:14:28.799 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:07:26.184614+0000", 2024-09-18T08:14:28.799 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 6.7730000000000002e-06 2024-09-18T08:14:28.799 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:14:28.799 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:14:28.799 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-09-18T08:14:28.799 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:07:26.184669+0000", 2024-09-18T08:14:28.799 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 5.4719000000000002e-05 2024-09-18T08:14:28.799 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:14:28.799 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:14:28.799 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "started", 2024-09-18T08:14:28.799 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:07:26.184764+0000", 2024-09-18T08:14:28.800 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 9.5529e-05 2024-09-18T08:14:28.800 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:14:28.800 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:14:28.800 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "waiting for subops from 0,5", 2024-09-18T08:14:28.800 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:07:26.184872+0000", 2024-09-18T08:14:28.800 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.000108115 2024-09-18T08:14:28.800 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:14:28.800 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:14:28.800 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:14:28.800 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:07:26.235645+0000", 2024-09-18T08:14:28.800 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.050772920999999999 2024-09-18T08:14:28.800 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:14:28.801 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:14:28.801 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "op_commit", 2024-09-18T08:14:28.801 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:07:26.235665+0000", 2024-09-18T08:14:28.801 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.9585999999999998e-05 2024-09-18T08:14:28.801 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:14:28.801 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:14:28.801 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:14:28.801 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:07:26.298883+0000", 2024-09-18T08:14:28.801 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.063217624 2024-09-18T08:14:28.801 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:14:28.801 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:14:28.801 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "commit_sent", 2024-09-18T08:14:28.801 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:07:26.299173+0000", 2024-09-18T08:14:28.801 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.00029084500000000002 2024-09-18T08:14:28.802 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:14:28.802 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:14:28.802 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "done", 2024-09-18T08:14:28.802 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:07:26.299224+0000", 2024-09-18T08:14:28.802 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 5.0445000000000002e-05 2024-09-18T08:14:28.802 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-09-18T08:14:28.802 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-09-18T08:14:28.802 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-09-18T08:14:28.802 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:14:28.802 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:14:28.802 INFO:teuthology.orchestra.run.smithi007.stdout: "description": "osd_op(client.15582.0:53557 9.0 9:0cf6806d:::benchmark_data_smithi007_124019_object53556:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e244)", 2024-09-18T08:14:28.802 INFO:teuthology.orchestra.run.smithi007.stdout: "initiated_at": "2024-09-18T08:11:41.343691+0000", 2024-09-18T08:14:28.803 INFO:teuthology.orchestra.run.smithi007.stdout: "age": 167.35599060199999, 2024-09-18T08:14:28.803 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.98520017699999995, 2024-09-18T08:14:28.803 INFO:teuthology.orchestra.run.smithi007.stdout: "continuous": false, 2024-09-18T08:14:28.803 INFO:teuthology.orchestra.run.smithi007.stdout: "type_data": { 2024-09-18T08:14:28.803 INFO:teuthology.orchestra.run.smithi007.stdout: "flag_point": "reached pg", 2024-09-18T08:14:28.803 INFO:teuthology.orchestra.run.smithi007.stdout: "client_info": { 2024-09-18T08:14:28.803 INFO:teuthology.orchestra.run.smithi007.stdout: "client": "client.15582", 2024-09-18T08:14:28.803 INFO:teuthology.orchestra.run.smithi007.stdout: "client_addr": "172.21.15.7:0/3598605839", 2024-09-18T08:14:28.803 INFO:teuthology.orchestra.run.smithi007.stdout: "tid": 53557 2024-09-18T08:14:28.803 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:14:28.803 INFO:teuthology.orchestra.run.smithi007.stdout: "events": [ 2024-09-18T08:14:28.803 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:14:28.803 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "initiated", 2024-09-18T08:14:28.804 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:11:41.343691+0000", 2024-09-18T08:14:28.804 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-09-18T08:14:28.804 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:14:28.804 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:14:28.804 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "header_read", 2024-09-18T08:14:28.804 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:11:41.343691+0000", 2024-09-18T08:14:28.804 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-09-18T08:14:28.804 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:14:28.804 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:14:28.804 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "throttled", 2024-09-18T08:14:28.804 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:11:41.343693+0000", 2024-09-18T08:14:28.804 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.804e-06 2024-09-18T08:14:28.804 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:14:28.805 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:14:28.805 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "all_read", 2024-09-18T08:14:28.805 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:11:41.343759+0000", 2024-09-18T08:14:28.805 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 6.6369000000000001e-05 2024-09-18T08:14:28.805 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:14:28.805 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:14:28.805 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "dispatched", 2024-09-18T08:14:28.805 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:11:41.343774+0000", 2024-09-18T08:14:28.805 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.5489999999999999e-05 2024-09-18T08:14:28.805 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:14:28.805 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:14:28.805 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "queued_for_pg", 2024-09-18T08:14:28.805 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:11:41.343785+0000", 2024-09-18T08:14:28.805 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.078e-05 2024-09-18T08:14:28.806 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:14:28.806 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:14:28.806 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-09-18T08:14:28.806 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:11:41.344215+0000", 2024-09-18T08:14:28.806 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.00042940799999999999 2024-09-18T08:14:28.806 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:14:28.806 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:14:28.806 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "waiting for peered", 2024-09-18T08:14:28.806 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:11:41.344258+0000", 2024-09-18T08:14:28.806 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 4.3460000000000001e-05 2024-09-18T08:14:28.806 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:14:28.806 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:14:28.806 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-09-18T08:14:28.807 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:11:42.326886+0000", 2024-09-18T08:14:28.807 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.98262749800000004 2024-09-18T08:14:28.807 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:14:28.807 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:14:28.807 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "done", 2024-09-18T08:14:28.807 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:11:42.328891+0000", 2024-09-18T08:14:28.807 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.002005368 2024-09-18T08:14:28.807 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-09-18T08:14:28.807 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-09-18T08:14:28.807 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-09-18T08:14:28.807 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:14:28.807 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:14:28.807 INFO:teuthology.orchestra.run.smithi007.stdout: "description": "osd_op(client.15582.0:53566 9.0 9:09a05dd0:::benchmark_data_smithi007_124019_object53565:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e244)", 2024-09-18T08:14:28.808 INFO:teuthology.orchestra.run.smithi007.stdout: "initiated_at": "2024-09-18T08:11:41.343807+0000", 2024-09-18T08:14:28.808 INFO:teuthology.orchestra.run.smithi007.stdout: "age": 167.355874774, 2024-09-18T08:14:28.808 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.007200514, 2024-09-18T08:14:28.808 INFO:teuthology.orchestra.run.smithi007.stdout: "continuous": false, 2024-09-18T08:14:28.808 INFO:teuthology.orchestra.run.smithi007.stdout: "type_data": { 2024-09-18T08:14:28.808 INFO:teuthology.orchestra.run.smithi007.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T08:14:28.808 INFO:teuthology.orchestra.run.smithi007.stdout: "client_info": { 2024-09-18T08:14:28.808 INFO:teuthology.orchestra.run.smithi007.stdout: "client": "client.15582", 2024-09-18T08:14:28.808 INFO:teuthology.orchestra.run.smithi007.stdout: "client_addr": "172.21.15.7:0/3598605839", 2024-09-18T08:14:28.808 INFO:teuthology.orchestra.run.smithi007.stdout: "tid": 53566 2024-09-18T08:14:28.808 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:14:28.808 INFO:teuthology.orchestra.run.smithi007.stdout: "events": [ 2024-09-18T08:14:28.808 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:14:28.809 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "initiated", 2024-09-18T08:14:28.809 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:11:41.343807+0000", 2024-09-18T08:14:28.809 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-09-18T08:14:28.809 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:14:28.809 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:14:28.809 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "header_read", 2024-09-18T08:14:28.809 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:11:41.343807+0000", 2024-09-18T08:14:28.809 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-09-18T08:14:28.809 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:14:28.809 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:14:28.809 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "throttled", 2024-09-18T08:14:28.809 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:11:41.343807+0000", 2024-09-18T08:14:28.809 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 5.3000000000000001e-07 2024-09-18T08:14:28.810 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:14:28.810 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:14:28.810 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "all_read", 2024-09-18T08:14:28.810 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:11:41.343826+0000", 2024-09-18T08:14:28.810 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.9338999999999999e-05 2024-09-18T08:14:28.810 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:14:28.810 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:14:28.810 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "dispatched", 2024-09-18T08:14:28.810 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:11:41.343833+0000", 2024-09-18T08:14:28.810 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 6.7710000000000001e-06 2024-09-18T08:14:28.810 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:14:28.810 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:14:28.811 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "queued_for_pg", 2024-09-18T08:14:28.811 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:11:41.343839+0000", 2024-09-18T08:14:28.811 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 5.4269999999999999e-06 2024-09-18T08:14:28.811 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:14:28.812 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:14:28.812 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-09-18T08:14:28.812 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:11:41.344332+0000", 2024-09-18T08:14:28.812 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.00049369799999999999 2024-09-18T08:14:28.812 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:14:28.812 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:14:28.812 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "waiting for peered", 2024-09-18T08:14:28.812 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:11:41.344339+0000", 2024-09-18T08:14:28.812 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 6.8589999999999999e-06 2024-09-18T08:14:28.812 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:14:28.812 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:14:28.812 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-09-18T08:14:28.813 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:11:42.328960+0000", 2024-09-18T08:14:28.813 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.98462126599999999 2024-09-18T08:14:28.813 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:14:28.813 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:14:28.813 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "started", 2024-09-18T08:14:28.813 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:11:42.329310+0000", 2024-09-18T08:14:28.813 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.00034983799999999999 2024-09-18T08:14:28.813 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:14:28.813 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:14:28.813 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "waiting for subops from 6", 2024-09-18T08:14:28.813 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:11:42.330445+0000", 2024-09-18T08:14:28.813 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.0011349859999999999 2024-09-18T08:14:28.813 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:14:28.814 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:14:28.814 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "op_commit", 2024-09-18T08:14:28.814 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:11:42.339902+0000", 2024-09-18T08:14:28.814 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.0094567869999999995 2024-09-18T08:14:28.814 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:14:28.814 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:14:28.814 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:14:28.814 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:11:42.350663+0000", 2024-09-18T08:14:28.814 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.010761299 2024-09-18T08:14:28.814 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:14:28.814 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:14:28.814 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "commit_sent", 2024-09-18T08:14:28.814 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:11:42.350832+0000", 2024-09-18T08:14:28.815 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.00016828399999999999 2024-09-18T08:14:28.815 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:14:28.815 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:14:28.815 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "done", 2024-09-18T08:14:28.815 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:11:42.351007+0000", 2024-09-18T08:14:28.815 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.00017542999999999999 2024-09-18T08:14:28.815 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-09-18T08:14:28.815 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-09-18T08:14:28.815 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-09-18T08:14:28.815 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:14:28.815 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:14:28.815 INFO:teuthology.orchestra.run.smithi007.stdout: "description": "osd_op(client.15582.0:53575 9.0 9:0f4fbab7:::benchmark_data_smithi007_124019_object53574:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e244)", 2024-09-18T08:14:28.815 INFO:teuthology.orchestra.run.smithi007.stdout: "initiated_at": "2024-09-18T08:11:41.343854+0000", 2024-09-18T08:14:28.815 INFO:teuthology.orchestra.run.smithi007.stdout: "age": 167.35582733199999, 2024-09-18T08:14:28.816 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.0074121789999999, 2024-09-18T08:14:28.816 INFO:teuthology.orchestra.run.smithi007.stdout: "continuous": false, 2024-09-18T08:14:28.816 INFO:teuthology.orchestra.run.smithi007.stdout: "type_data": { 2024-09-18T08:14:28.816 INFO:teuthology.orchestra.run.smithi007.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T08:14:28.816 INFO:teuthology.orchestra.run.smithi007.stdout: "client_info": { 2024-09-18T08:14:28.816 INFO:teuthology.orchestra.run.smithi007.stdout: "client": "client.15582", 2024-09-18T08:14:28.816 INFO:teuthology.orchestra.run.smithi007.stdout: "client_addr": "172.21.15.7:0/3598605839", 2024-09-18T08:14:28.816 INFO:teuthology.orchestra.run.smithi007.stdout: "tid": 53575 2024-09-18T08:14:28.816 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:14:28.816 INFO:teuthology.orchestra.run.smithi007.stdout: "events": [ 2024-09-18T08:14:28.816 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:14:28.816 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "initiated", 2024-09-18T08:14:28.817 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:11:41.343854+0000", 2024-09-18T08:14:28.817 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-09-18T08:14:28.817 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:14:28.817 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:14:28.817 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "header_read", 2024-09-18T08:14:28.817 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:11:41.343854+0000", 2024-09-18T08:14:28.817 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-09-18T08:14:28.817 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:14:28.817 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:14:28.817 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "throttled", 2024-09-18T08:14:28.817 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:11:41.343855+0000", 2024-09-18T08:14:28.817 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 5.5000000000000003e-07 2024-09-18T08:14:28.817 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:14:28.818 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:14:28.818 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "all_read", 2024-09-18T08:14:28.818 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:11:41.343878+0000", 2024-09-18T08:14:28.818 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 2.3740999999999999e-05 2024-09-18T08:14:28.818 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:14:28.818 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:14:28.818 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "dispatched", 2024-09-18T08:14:28.818 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:11:41.343886+0000", 2024-09-18T08:14:28.818 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 7.8809999999999996e-06 2024-09-18T08:14:28.818 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:14:28.818 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:14:28.818 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "queued_for_pg", 2024-09-18T08:14:28.818 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:11:41.343892+0000", 2024-09-18T08:14:28.818 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 5.8200000000000002e-06 2024-09-18T08:14:28.819 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:14:28.819 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:14:28.819 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-09-18T08:14:28.819 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:11:41.344821+0000", 2024-09-18T08:14:28.819 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.000928874 2024-09-18T08:14:28.819 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:14:28.819 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:14:28.819 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "waiting for peered", 2024-09-18T08:14:28.819 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:11:41.344861+0000", 2024-09-18T08:14:28.820 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 3.9919999999999997e-05 2024-09-18T08:14:28.820 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:14:28.820 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:14:28.820 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-09-18T08:14:28.820 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:11:42.334161+0000", 2024-09-18T08:14:28.820 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.98930047799999998 2024-09-18T08:14:28.820 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:14:28.820 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:14:28.820 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "started", 2024-09-18T08:14:28.820 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:11:42.334487+0000", 2024-09-18T08:14:28.820 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.00032586300000000002 2024-09-18T08:14:28.820 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:14:28.821 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:14:28.821 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "waiting for subops from 6", 2024-09-18T08:14:28.821 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:11:42.334655+0000", 2024-09-18T08:14:28.821 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.000167419 2024-09-18T08:14:28.821 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:14:28.821 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:14:28.821 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "op_commit", 2024-09-18T08:14:28.821 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:11:42.346631+0000", 2024-09-18T08:14:28.821 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.011976496 2024-09-18T08:14:28.821 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:14:28.821 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:14:28.821 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:14:28.821 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:11:42.351115+0000", 2024-09-18T08:14:28.822 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.0044843590000000003 2024-09-18T08:14:28.822 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:14:28.822 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:14:28.822 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "commit_sent", 2024-09-18T08:14:28.822 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:11:42.351173+0000", 2024-09-18T08:14:28.822 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 5.7061000000000001e-05 2024-09-18T08:14:28.822 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:14:28.822 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:14:28.822 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "done", 2024-09-18T08:14:28.822 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:11:42.351266+0000", 2024-09-18T08:14:28.822 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 9.3716999999999997e-05 2024-09-18T08:14:28.822 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-09-18T08:14:28.822 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-09-18T08:14:28.823 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-09-18T08:14:28.823 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:14:28.823 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:14:28.823 INFO:teuthology.orchestra.run.smithi007.stdout: "description": "osd_op(client.15582.0:53576 9.0 9:0f9d6982:::benchmark_data_smithi007_124019_object53575:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e244)", 2024-09-18T08:14:28.823 INFO:teuthology.orchestra.run.smithi007.stdout: "initiated_at": "2024-09-18T08:11:41.343901+0000", 2024-09-18T08:14:28.823 INFO:teuthology.orchestra.run.smithi007.stdout: "age": 167.35578011199999, 2024-09-18T08:14:28.823 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.01002054, 2024-09-18T08:14:28.823 INFO:teuthology.orchestra.run.smithi007.stdout: "continuous": false, 2024-09-18T08:14:28.823 INFO:teuthology.orchestra.run.smithi007.stdout: "type_data": { 2024-09-18T08:14:28.823 INFO:teuthology.orchestra.run.smithi007.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T08:14:28.823 INFO:teuthology.orchestra.run.smithi007.stdout: "client_info": { 2024-09-18T08:14:28.823 INFO:teuthology.orchestra.run.smithi007.stdout: "client": "client.15582", 2024-09-18T08:14:28.823 INFO:teuthology.orchestra.run.smithi007.stdout: "client_addr": "172.21.15.7:0/3598605839", 2024-09-18T08:14:28.824 INFO:teuthology.orchestra.run.smithi007.stdout: "tid": 53576 2024-09-18T08:14:28.824 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:14:28.824 INFO:teuthology.orchestra.run.smithi007.stdout: "events": [ 2024-09-18T08:14:28.824 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:14:28.824 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "initiated", 2024-09-18T08:14:28.824 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:11:41.343901+0000", 2024-09-18T08:14:28.824 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-09-18T08:14:28.824 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:14:28.824 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:14:28.824 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "header_read", 2024-09-18T08:14:28.824 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:11:41.343901+0000", 2024-09-18T08:14:28.824 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-09-18T08:14:28.825 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:14:28.825 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:14:28.825 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "throttled", 2024-09-18T08:14:28.825 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:11:41.343902+0000", 2024-09-18T08:14:28.825 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 4.7100000000000002e-07 2024-09-18T08:14:28.825 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:14:28.825 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:14:28.825 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "all_read", 2024-09-18T08:14:28.825 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:11:41.343923+0000", 2024-09-18T08:14:28.825 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 2.1009000000000002e-05 2024-09-18T08:14:28.825 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:14:28.825 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:14:28.825 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "dispatched", 2024-09-18T08:14:28.826 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:11:41.343930+0000", 2024-09-18T08:14:28.826 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 7.0040000000000002e-06 2024-09-18T08:14:28.826 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:14:28.826 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:14:28.826 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "queued_for_pg", 2024-09-18T08:14:28.826 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:11:41.343935+0000", 2024-09-18T08:14:28.826 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 5.0549999999999999e-06 2024-09-18T08:14:28.826 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:14:28.826 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:14:28.826 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-09-18T08:14:28.826 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:11:41.345163+0000", 2024-09-18T08:14:28.826 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.0012281200000000001 2024-09-18T08:14:28.826 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:14:28.827 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:14:28.827 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "waiting for peered", 2024-09-18T08:14:28.827 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:11:41.345169+0000", 2024-09-18T08:14:28.827 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 5.6999999999999996e-06 2024-09-18T08:14:28.827 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:14:28.827 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:14:28.827 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-09-18T08:14:28.827 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:11:42.336827+0000", 2024-09-18T08:14:28.827 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.99165844599999997 2024-09-18T08:14:28.827 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:14:28.827 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:14:28.827 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "started", 2024-09-18T08:14:28.827 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:11:42.336957+0000", 2024-09-18T08:14:28.828 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.00013009 2024-09-18T08:14:28.828 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:14:28.828 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:14:28.828 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "waiting for subops from 6", 2024-09-18T08:14:28.828 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:11:42.337097+0000", 2024-09-18T08:14:28.828 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.00014021899999999999 2024-09-18T08:14:28.828 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:14:28.828 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:14:28.828 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "op_commit", 2024-09-18T08:14:28.828 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:11:42.346639+0000", 2024-09-18T08:14:28.828 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.0095413370000000004 2024-09-18T08:14:28.828 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:14:28.828 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:14:28.829 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:14:28.829 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:11:42.353750+0000", 2024-09-18T08:14:28.829 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.0071108530000000003 2024-09-18T08:14:28.829 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:14:28.829 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:14:28.829 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "commit_sent", 2024-09-18T08:14:28.829 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:11:42.353813+0000", 2024-09-18T08:14:28.829 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 6.3721999999999997e-05 2024-09-18T08:14:28.829 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:14:28.829 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:14:28.829 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "done", 2024-09-18T08:14:28.829 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:11:42.353922+0000", 2024-09-18T08:14:28.829 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.000108514 2024-09-18T08:14:28.830 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-09-18T08:14:28.830 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-09-18T08:14:28.830 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-09-18T08:14:28.830 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:14:28.830 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:14:28.830 INFO:teuthology.orchestra.run.smithi007.stdout: "description": "osd_op(client.24356.0:2055 4.4 4:22d26bf9:::data_loggenerations_metadata:head [watch reconnect cookie 94700888688768 gen 2] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e244)", 2024-09-18T08:14:28.830 INFO:teuthology.orchestra.run.smithi007.stdout: "initiated_at": "2024-09-18T08:11:41.923730+0000", 2024-09-18T08:14:28.830 INFO:teuthology.orchestra.run.smithi007.stdout: "age": 166.775951746, 2024-09-18T08:14:28.830 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.42490792300000002, 2024-09-18T08:14:28.830 INFO:teuthology.orchestra.run.smithi007.stdout: "continuous": false, 2024-09-18T08:14:28.830 INFO:teuthology.orchestra.run.smithi007.stdout: "type_data": { 2024-09-18T08:14:28.830 INFO:teuthology.orchestra.run.smithi007.stdout: "flag_point": "started", 2024-09-18T08:14:28.830 INFO:teuthology.orchestra.run.smithi007.stdout: "client_info": { 2024-09-18T08:14:28.831 INFO:teuthology.orchestra.run.smithi007.stdout: "client": "client.24356", 2024-09-18T08:14:28.831 INFO:teuthology.orchestra.run.smithi007.stdout: "client_addr": "172.21.15.7:0/219168083", 2024-09-18T08:14:28.831 INFO:teuthology.orchestra.run.smithi007.stdout: "tid": 2055 2024-09-18T08:14:28.831 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:14:28.831 INFO:teuthology.orchestra.run.smithi007.stdout: "events": [ 2024-09-18T08:14:28.831 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:14:28.831 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "initiated", 2024-09-18T08:14:28.831 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:11:41.923730+0000", 2024-09-18T08:14:28.831 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-09-18T08:14:28.831 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:14:28.831 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:14:28.831 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "header_read", 2024-09-18T08:14:28.831 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:11:41.923730+0000", 2024-09-18T08:14:28.832 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-09-18T08:14:28.832 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:14:28.832 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:14:28.832 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "throttled", 2024-09-18T08:14:28.832 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:11:41.923732+0000", 2024-09-18T08:14:28.832 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 2.5170000000000002e-06 2024-09-18T08:14:28.832 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:14:28.832 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:14:28.832 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "all_read", 2024-09-18T08:14:28.832 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:11:41.923744+0000", 2024-09-18T08:14:28.832 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.1895999999999999e-05 2024-09-18T08:14:28.832 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:14:28.832 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:14:28.833 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "dispatched", 2024-09-18T08:14:28.833 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:11:41.923762+0000", 2024-09-18T08:14:28.833 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.7476e-05 2024-09-18T08:14:28.833 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:14:28.833 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:14:28.833 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "queued_for_pg", 2024-09-18T08:14:28.833 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:11:41.923774+0000", 2024-09-18T08:14:28.833 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.2072000000000001e-05 2024-09-18T08:14:28.833 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:14:28.833 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:14:28.833 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-09-18T08:14:28.833 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:11:41.923863+0000", 2024-09-18T08:14:28.833 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 8.9174999999999999e-05 2024-09-18T08:14:28.834 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:14:28.834 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:14:28.834 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "waiting for peered", 2024-09-18T08:14:28.834 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:11:41.923875+0000", 2024-09-18T08:14:28.834 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.2544000000000001e-05 2024-09-18T08:14:28.834 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:14:28.834 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:14:28.834 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-09-18T08:14:28.834 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:11:42.331723+0000", 2024-09-18T08:14:28.834 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.40784754899999998 2024-09-18T08:14:28.834 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:14:28.834 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:14:28.834 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "started", 2024-09-18T08:14:28.835 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:11:42.331911+0000", 2024-09-18T08:14:28.835 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.00018769699999999999 2024-09-18T08:14:28.835 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:14:28.835 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:14:28.835 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "done", 2024-09-18T08:14:28.835 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:11:42.348638+0000", 2024-09-18T08:14:28.835 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.016726997 2024-09-18T08:14:28.835 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-09-18T08:14:28.835 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-09-18T08:14:28.835 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-09-18T08:14:28.835 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:14:28.835 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:14:28.835 INFO:teuthology.orchestra.run.smithi007.stdout: "description": "osd_op(client.24356.0:2782 5.d 5:b4812045:::notify.4:head [watch reconnect cookie 94700888684160 gen 1] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e292)", 2024-09-18T08:14:28.836 INFO:teuthology.orchestra.run.smithi007.stdout: "initiated_at": "2024-09-18T08:14:04.565391+0000", 2024-09-18T08:14:28.836 INFO:teuthology.orchestra.run.smithi007.stdout: "age": 24.134290131, 2024-09-18T08:14:28.836 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.0413668460000001, 2024-09-18T08:14:28.836 INFO:teuthology.orchestra.run.smithi007.stdout: "continuous": false, 2024-09-18T08:14:28.836 INFO:teuthology.orchestra.run.smithi007.stdout: "type_data": { 2024-09-18T08:14:28.836 INFO:teuthology.orchestra.run.smithi007.stdout: "flag_point": "started", 2024-09-18T08:14:28.836 INFO:teuthology.orchestra.run.smithi007.stdout: "client_info": { 2024-09-18T08:14:28.836 INFO:teuthology.orchestra.run.smithi007.stdout: "client": "client.24356", 2024-09-18T08:14:28.836 INFO:teuthology.orchestra.run.smithi007.stdout: "client_addr": "172.21.15.7:0/219168083", 2024-09-18T08:14:28.836 INFO:teuthology.orchestra.run.smithi007.stdout: "tid": 2782 2024-09-18T08:14:28.836 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:14:28.836 INFO:teuthology.orchestra.run.smithi007.stdout: "events": [ 2024-09-18T08:14:28.837 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:14:28.837 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "initiated", 2024-09-18T08:14:28.837 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:14:04.565391+0000", 2024-09-18T08:14:28.837 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-09-18T08:14:28.837 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:14:28.837 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:14:28.837 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "header_read", 2024-09-18T08:14:28.837 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:14:04.565391+0000", 2024-09-18T08:14:28.837 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-09-18T08:14:28.837 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:14:28.837 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:14:28.837 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "throttled", 2024-09-18T08:14:28.837 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:14:04.565393+0000", 2024-09-18T08:14:28.838 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.8199999999999999e-06 2024-09-18T08:14:28.838 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:14:28.838 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:14:28.838 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "all_read", 2024-09-18T08:14:28.838 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:14:04.565400+0000", 2024-09-18T08:14:28.838 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 6.968e-06 2024-09-18T08:14:28.838 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:14:28.838 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:14:28.838 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "dispatched", 2024-09-18T08:14:28.838 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:14:04.565413+0000", 2024-09-18T08:14:28.838 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.3003000000000001e-05 2024-09-18T08:14:28.838 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:14:28.838 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:14:28.839 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "queued_for_pg", 2024-09-18T08:14:28.839 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:14:04.565422+0000", 2024-09-18T08:14:28.839 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 9.3870000000000001e-06 2024-09-18T08:14:28.839 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:14:28.839 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:14:28.839 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-09-18T08:14:28.839 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:14:04.565500+0000", 2024-09-18T08:14:28.839 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 7.7395e-05 2024-09-18T08:14:28.839 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:14:28.839 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:14:28.839 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "op must wait for map", 2024-09-18T08:14:28.839 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:14:04.565506+0000", 2024-09-18T08:14:28.839 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 6.5180000000000001e-06 2024-09-18T08:14:28.840 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:14:28.840 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:14:28.840 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-09-18T08:14:28.840 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:14:04.576465+0000", 2024-09-18T08:14:28.840 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.010958183 2024-09-18T08:14:28.840 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:14:28.840 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:14:28.840 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "waiting for peered", 2024-09-18T08:14:28.840 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:14:04.576476+0000", 2024-09-18T08:14:28.840 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.1877e-05 2024-09-18T08:14:28.840 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:14:28.840 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:14:28.840 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-09-18T08:14:28.841 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:14:05.594014+0000", 2024-09-18T08:14:28.841 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.0175379360000001 2024-09-18T08:14:28.841 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:14:28.841 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:14:28.841 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "waiting for degraded object", 2024-09-18T08:14:28.841 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:14:05.594307+0000", 2024-09-18T08:14:28.841 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.00029306699999999999 2024-09-18T08:14:28.841 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:14:28.841 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:14:28.841 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-09-18T08:14:28.841 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:14:05.595721+0000", 2024-09-18T08:14:28.841 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.001413826 2024-09-18T08:14:28.841 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:14:28.842 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:14:28.842 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "started", 2024-09-18T08:14:28.842 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:14:05.595822+0000", 2024-09-18T08:14:28.842 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.000100814 2024-09-18T08:14:28.842 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:14:28.842 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:14:28.842 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "done", 2024-09-18T08:14:28.842 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:14:05.606758+0000", 2024-09-18T08:14:28.842 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.010936052 2024-09-18T08:14:28.842 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-09-18T08:14:28.842 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-09-18T08:14:28.842 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-09-18T08:14:28.842 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-09-18T08:14:28.842 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-09-18T08:14:28.843 INFO:teuthology.orchestra.run.smithi007.stdout:} 2024-09-18T08:14:28.851 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6 2024-09-18T08:14:28.888 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:14:28.889 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:14:28.889 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6' 2024-09-18T08:14:28.980 INFO:journalctl@ceph.osd.1.smithi007.stdout:Sep 18 08:14:28 smithi007 bash[145016]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_3 2024-09-18T08:14:28.980 INFO:journalctl@ceph.osd.1.smithi007.stdout:Sep 18 08:14:28 smithi007 bash[145016]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-2 2024-09-18T08:14:28.981 INFO:journalctl@ceph.osd.1.smithi007.stdout:Sep 18 08:14:28 smithi007 bash[145016]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-1/block 2024-09-18T08:14:28.981 INFO:journalctl@ceph.osd.1.smithi007.stdout:Sep 18 08:14:28 smithi007 bash[145016]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-1 2024-09-18T08:14:28.981 INFO:journalctl@ceph.osd.1.smithi007.stdout:Sep 18 08:14:28 smithi007 bash[145016]: --> ceph-volume raw activate successful for osd ID: 1 2024-09-18T08:14:28.981 INFO:journalctl@ceph.osd.1.smithi007.stdout:Sep 18 08:14:28 smithi007 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd-1-activate[145061]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_3 2024-09-18T08:14:28.981 INFO:journalctl@ceph.osd.1.smithi007.stdout:Sep 18 08:14:28 smithi007 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd-1-activate[145061]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-2 2024-09-18T08:14:28.981 INFO:journalctl@ceph.osd.1.smithi007.stdout:Sep 18 08:14:28 smithi007 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd-1-activate[145061]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-1/block 2024-09-18T08:14:28.981 INFO:journalctl@ceph.osd.1.smithi007.stdout:Sep 18 08:14:28 smithi007 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd-1-activate[145061]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-1 2024-09-18T08:14:28.981 INFO:journalctl@ceph.osd.1.smithi007.stdout:Sep 18 08:14:28 smithi007 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd-1-activate[145061]: --> ceph-volume raw activate successful for osd ID: 1 2024-09-18T08:14:28.981 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:14:28 smithi007 ceph-mon[29459]: 3.12 scrub starts 2024-09-18T08:14:28.981 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:14:28 smithi007 ceph-mon[29459]: 3.12 scrub ok 2024-09-18T08:14:28.982 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:14:28 smithi007 ceph-mon[39249]: 3.12 scrub starts 2024-09-18T08:14:28.982 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:14:28 smithi007 ceph-mon[39249]: 3.12 scrub ok 2024-09-18T08:14:28.990 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3 2024-09-18T08:14:29.027 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:14:29.028 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:14:29.028 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3' 2024-09-18T08:14:29.129 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7 2024-09-18T08:14:29.155 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:14:28 smithi050 ceph-mon[32371]: 3.12 scrub starts 2024-09-18T08:14:29.155 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:14:28 smithi050 ceph-mon[32371]: 3.12 scrub ok 2024-09-18T08:14:29.165 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:14:29.165 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:14:29.166 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7' 2024-09-18T08:14:29.266 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7 2024-09-18T08:14:29.303 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:14:29.303 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:14:29.304 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7' 2024-09-18T08:14:29.377 INFO:teuthology.orchestra.run.smithi007.stdout:osd.0: {} 2024-09-18T08:14:29.377 INFO:teuthology.orchestra.run.smithi007.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-18T08:14:29.378 INFO:teuthology.orchestra.run.smithi007.stderr:Error ENXIO: problem getting command descriptions from osd.1 2024-09-18T08:14:29.404 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3 2024-09-18T08:14:29.405 INFO:teuthology.orchestra.run.smithi007.stdout:osd.2: {} 2024-09-18T08:14:29.406 INFO:teuthology.orchestra.run.smithi007.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-18T08:14:29.442 INFO:teuthology.orchestra.run.smithi007.stdout:osd.3: {} 2024-09-18T08:14:29.443 INFO:teuthology.orchestra.run.smithi007.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-18T08:14:29.451 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:14:29.451 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:14:29.452 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3' 2024-09-18T08:14:29.480 INFO:teuthology.orchestra.run.smithi007.stdout:osd.4: {} 2024-09-18T08:14:29.480 INFO:teuthology.orchestra.run.smithi007.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-18T08:14:29.511 INFO:teuthology.orchestra.run.smithi007.stdout:osd.5: {} 2024-09-18T08:14:29.511 INFO:teuthology.orchestra.run.smithi007.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-18T08:14:29.528 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid 779349ca-7593-11ef-bceb-c7b262605968 -- ceph osd unset nodeep-scrub 2024-09-18T08:14:29.549 INFO:teuthology.orchestra.run.smithi007.stdout:osd.6: {} 2024-09-18T08:14:29.549 INFO:teuthology.orchestra.run.smithi007.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-18T08:14:29.552 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5 2024-09-18T08:14:29.580 INFO:teuthology.orchestra.run.smithi007.stdout:osd.7: {} 2024-09-18T08:14:29.581 INFO:teuthology.orchestra.run.smithi007.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-18T08:14:29.582 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:14:29.583 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:14:29.583 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5' 2024-09-18T08:14:29.655 INFO:journalctl@ceph.iscsi.iscsi.a.smithi050.stdout:Sep 18 08:14:29 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-iscsi-iscsi-a[57782]: debug there is no tcmu-runner data available 2024-09-18T08:14:29.684 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3 2024-09-18T08:14:29.697 INFO:journalctl@ceph.osd.1.smithi007.stdout:Sep 18 08:14:29 smithi007 podman[145016]: 2024-09-18 08:14:29.444125115 +0000 UTC m=+4.347281908 container died a4b7a8aad7c38eb69924bd12fba0c381f262b3409da796c57cb1752196b3f65e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:1f5b5a3734f665ef333266f39143ac3ef1eb1951b21258f2307e10332e2d7ac9, name=ceph-779349ca-7593-11ef-bceb-c7b262605968-osd-1-activate, RELEASE=wip-guits-main-2024-09-17-1213-e7fb7b5, org.label-schema.vendor=CentOS, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , io.buildah.version=1.37.2, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, org.label-schema.build-date=20240731, ceph=True, org.label-schema.name=CentOS Stream 9 Base Image, CEPH_POINT_RELEASE=) 2024-09-18T08:14:29.720 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:14:29.721 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:14:29.721 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3' 2024-09-18T08:14:29.822 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6 2024-09-18T08:14:29.859 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:14:29.860 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:14:29.860 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6' 2024-09-18T08:14:29.961 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5 2024-09-18T08:14:29.980 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:14:29 smithi007 ceph-mon[39249]: 5.9 scrub starts 2024-09-18T08:14:29.980 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:14:29 smithi007 ceph-mon[39249]: 5.9 scrub ok 2024-09-18T08:14:29.981 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:14:29 smithi007 ceph-mon[39249]: 6.8 scrub starts 2024-09-18T08:14:29.981 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:14:29 smithi007 ceph-mon[39249]: 6.8 scrub ok 2024-09-18T08:14:29.981 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:14:29 smithi007 ceph-mon[39249]: 3.1f scrub starts 2024-09-18T08:14:29.981 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:14:29 smithi007 ceph-mon[39249]: 3.1f scrub ok 2024-09-18T08:14:29.981 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:14:29 smithi007 ceph-mon[39249]: from='client.? 172.21.15.7:0/1748169751' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-18T08:14:29.981 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:14:29 smithi007 ceph-mon[39249]: Health check update: Degraded data redundancy: 3859/36054 objects degraded (10.703%), 27 pgs degraded, 49 pgs undersized (PG_DEGRADED) 2024-09-18T08:14:29.981 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:14:29 smithi007 ceph-mon[39249]: pgmap v567: 158 pgs: 29 active+clean+remapped, 22 active+undersized, 1 active+remapped+backfill_toofull, 27 active+undersized+degraded, 79 active+clean; 275 MiB data, 5.3 GiB used, 621 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.91k op/s; 1498/13752 objects degraded (10.893%); 1778/13752 objects misplaced (12.929%) 2024-09-18T08:14:29.981 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:14:29 smithi007 ceph-mon[39249]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-18T08:14:29.981 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:14:29 smithi007 ceph-mon[29459]: 5.9 scrub starts 2024-09-18T08:14:29.981 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:14:29 smithi007 ceph-mon[29459]: 5.9 scrub ok 2024-09-18T08:14:29.981 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:14:29 smithi007 ceph-mon[29459]: 6.8 scrub starts 2024-09-18T08:14:29.982 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:14:29 smithi007 ceph-mon[29459]: 6.8 scrub ok 2024-09-18T08:14:29.982 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:14:29 smithi007 ceph-mon[29459]: 3.1f scrub starts 2024-09-18T08:14:29.982 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:14:29 smithi007 ceph-mon[29459]: 3.1f scrub ok 2024-09-18T08:14:29.982 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:14:29 smithi007 ceph-mon[29459]: from='client.? 172.21.15.7:0/1748169751' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-18T08:14:29.982 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:14:29 smithi007 ceph-mon[29459]: Health check update: Degraded data redundancy: 3859/36054 objects degraded (10.703%), 27 pgs degraded, 49 pgs undersized (PG_DEGRADED) 2024-09-18T08:14:29.982 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:14:29 smithi007 ceph-mon[29459]: pgmap v567: 158 pgs: 29 active+clean+remapped, 22 active+undersized, 1 active+remapped+backfill_toofull, 27 active+undersized+degraded, 79 active+clean; 275 MiB data, 5.3 GiB used, 621 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.91k op/s; 1498/13752 objects degraded (10.893%); 1778/13752 objects misplaced (12.929%) 2024-09-18T08:14:29.982 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:14:29 smithi007 ceph-mon[29459]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-18T08:14:29.993 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:14:29.994 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:14:29.994 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5' 2024-09-18T08:14:30.095 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3 2024-09-18T08:14:30.127 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:14:30.128 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:14:30.128 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3' 2024-09-18T08:14:30.155 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:14:29 smithi050 ceph-mon[32371]: 5.9 scrub starts 2024-09-18T08:14:30.156 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:14:29 smithi050 ceph-mon[32371]: 5.9 scrub ok 2024-09-18T08:14:30.156 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:14:29 smithi050 ceph-mon[32371]: 6.8 scrub starts 2024-09-18T08:14:30.156 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:14:29 smithi050 ceph-mon[32371]: 6.8 scrub ok 2024-09-18T08:14:30.156 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:14:29 smithi050 ceph-mon[32371]: 3.1f scrub starts 2024-09-18T08:14:30.156 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:14:29 smithi050 ceph-mon[32371]: 3.1f scrub ok 2024-09-18T08:14:30.156 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:14:29 smithi050 ceph-mon[32371]: from='client.? 172.21.15.7:0/1748169751' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-18T08:14:30.156 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:14:29 smithi050 ceph-mon[32371]: Health check update: Degraded data redundancy: 3859/36054 objects degraded (10.703%), 27 pgs degraded, 49 pgs undersized (PG_DEGRADED) 2024-09-18T08:14:30.156 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:14:29 smithi050 ceph-mon[32371]: pgmap v567: 158 pgs: 29 active+clean+remapped, 22 active+undersized, 1 active+remapped+backfill_toofull, 27 active+undersized+degraded, 79 active+clean; 275 MiB data, 5.3 GiB used, 621 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.91k op/s; 1498/13752 objects degraded (10.893%); 1778/13752 objects misplaced (12.929%) 2024-09-18T08:14:30.156 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:14:29 smithi050 ceph-mon[32371]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-18T08:14:30.229 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3 2024-09-18T08:14:30.265 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:14:30.265 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:14:30.266 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3' 2024-09-18T08:14:30.366 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2 2024-09-18T08:14:30.401 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:14:30.401 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:14:30.402 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2' 2024-09-18T08:14:30.480 INFO:journalctl@ceph.osd.1.smithi007.stdout:Sep 18 08:14:30 smithi007 podman[145586]: 2024-09-18 08:14:30.284568831 +0000 UTC m=+1.454272610 container remove a4b7a8aad7c38eb69924bd12fba0c381f262b3409da796c57cb1752196b3f65e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:1f5b5a3734f665ef333266f39143ac3ef1eb1951b21258f2307e10332e2d7ac9, name=ceph-779349ca-7593-11ef-bceb-c7b262605968-osd-1-activate, RELEASE=wip-guits-main-2024-09-17-1213-e7fb7b5, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, ceph=True, GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240731, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, io.buildah.version=1.37.2) 2024-09-18T08:14:30.502 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6 2024-09-18T08:14:30.535 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:14:30.536 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:14:30.536 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6' 2024-09-18T08:14:30.637 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2 2024-09-18T08:14:30.669 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:14:30.670 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:14:30.670 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2' 2024-09-18T08:14:30.771 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3 2024-09-18T08:14:30.806 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:14:30.806 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:14:30.806 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3' 2024-09-18T08:14:30.907 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0 2024-09-18T08:14:30.942 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:14:30.943 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:14:30.943 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0' 2024-09-18T08:14:31.040 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-09-18T08:14:31.041 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-09-18T08:14:31.044 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4 2024-09-18T08:14:31.078 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:14:31.078 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:14:31.079 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4' 2024-09-18T08:14:31.180 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5 2024-09-18T08:14:31.219 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:14:31.219 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:14:31.220 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5' 2024-09-18T08:14:31.321 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7 2024-09-18T08:14:31.362 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:14:31.362 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:14:31.363 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7' 2024-09-18T08:14:31.464 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2 2024-09-18T08:14:31.500 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:14:31.501 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:14:31.501 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2' 2024-09-18T08:14:31.576 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid 779349ca-7593-11ef-bceb-c7b262605968 -- ceph daemon osd.3 dump_ops_in_flight 2024-09-18T08:14:31.602 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4 2024-09-18T08:14:31.616 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:14:31 smithi007 ceph-mon[39249]: purged_snaps scrub starts 2024-09-18T08:14:31.616 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:14:31 smithi007 ceph-mon[39249]: purged_snaps scrub ok 2024-09-18T08:14:31.616 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:14:31 smithi007 ceph-mon[39249]: osdmap e302: 8 total, 7 up, 7 in 2024-09-18T08:14:31.617 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:14:31 smithi007 ceph-mon[29459]: purged_snaps scrub starts 2024-09-18T08:14:31.617 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:14:31 smithi007 ceph-mon[29459]: purged_snaps scrub ok 2024-09-18T08:14:31.617 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:14:31 smithi007 ceph-mon[29459]: osdmap e302: 8 total, 7 up, 7 in 2024-09-18T08:14:31.631 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:14:31.632 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:14:31.632 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4' 2024-09-18T08:14:31.638 INFO:tasks.radosbench.radosbench.0.smithi007.stdout:Removed 141882 objects 2024-09-18T08:14:31.638 INFO:tasks.radosbench.radosbench.0.smithi007.stdout:Clean up completed and total clean up time :106.436 2024-09-18T08:14:31.655 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:14:31 smithi050 ceph-mon[32371]: purged_snaps scrub starts 2024-09-18T08:14:31.655 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:14:31 smithi050 ceph-mon[32371]: purged_snaps scrub ok 2024-09-18T08:14:31.655 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:14:31 smithi050 ceph-mon[32371]: osdmap e302: 8 total, 7 up, 7 in 2024-09-18T08:14:31.734 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2 2024-09-18T08:14:31.756 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:14:31.757 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:14:31.757 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2' 2024-09-18T08:14:31.858 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2 2024-09-18T08:14:31.882 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:14:31.883 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:14:31.883 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2' 2024-09-18T08:14:31.895 INFO:journalctl@ceph.osd.1.smithi007.stdout:Sep 18 08:14:31 smithi007 podman[145914]: 2024-09-18 08:14:31.653222882 +0000 UTC m=+0.020274180 image pull b22e1e37ddf1ca6e22136cd6540a9ab61e7cafb8bfbc7a91a2f20f7f9941bd9a quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:1f5b5a3734f665ef333266f39143ac3ef1eb1951b21258f2307e10332e2d7ac9 2024-09-18T08:14:31.896 INFO:journalctl@ceph.osd.1.smithi007.stdout:Sep 18 08:14:31 smithi007 podman[145914]: 2024-09-18 08:14:31.779915625 +0000 UTC m=+0.146966849 container create 30865df549477a986230448f6dfbc1fbfa574da663a76054079bb27922256cca (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:1f5b5a3734f665ef333266f39143ac3ef1eb1951b21258f2307e10332e2d7ac9, name=ceph-779349ca-7593-11ef-bceb-c7b262605968-osd-1, RELEASE=wip-guits-main-2024-09-17-1213-e7fb7b5, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.build-date=20240731, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, ceph=True, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, io.buildah.version=1.37.2, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 9 Base Image) 2024-09-18T08:14:31.984 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4 2024-09-18T08:14:32.008 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:14:32.009 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:14:32.009 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4' 2024-09-18T08:14:32.110 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2 2024-09-18T08:14:32.116 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid 779349ca-7593-11ef-bceb-c7b262605968 -- ceph daemon osd.1 dump_ops_in_flight 2024-09-18T08:14:32.117 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/779349ca-7593-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T08:14:32.136 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:14:32.136 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:14:32.137 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2' 2024-09-18T08:14:32.159 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/779349ca-7593-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T08:14:32.232 INFO:journalctl@ceph.osd.1.smithi007.stdout:Sep 18 08:14:32 smithi007 podman[145914]: 2024-09-18 08:14:32.01645412 +0000 UTC m=+0.383505357 container init 30865df549477a986230448f6dfbc1fbfa574da663a76054079bb27922256cca (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:1f5b5a3734f665ef333266f39143ac3ef1eb1951b21258f2307e10332e2d7ac9, name=ceph-779349ca-7593-11ef-bceb-c7b262605968-osd-1, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, org.label-schema.name=CentOS Stream 9 Base Image, RELEASE=wip-guits-main-2024-09-17-1213-e7fb7b5, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, io.buildah.version=1.37.2, org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240731) 2024-09-18T08:14:32.232 INFO:journalctl@ceph.osd.1.smithi007.stdout:Sep 18 08:14:32 smithi007 podman[145914]: 2024-09-18 08:14:32.020062981 +0000 UTC m=+0.387114215 container start 30865df549477a986230448f6dfbc1fbfa574da663a76054079bb27922256cca (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:1f5b5a3734f665ef333266f39143ac3ef1eb1951b21258f2307e10332e2d7ac9, name=ceph-779349ca-7593-11ef-bceb-c7b262605968-osd-1, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , RELEASE=wip-guits-main-2024-09-17-1213-e7fb7b5, ceph=True, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 9 Base Image, GIT_BRANCH=HEAD, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240731, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.37.2) 2024-09-18T08:14:32.232 INFO:journalctl@ceph.osd.1.smithi007.stdout:Sep 18 08:14:32 smithi007 ceph-osd[146007]: -- 172.21.15.7:0/2215805760 <== mon.0 v2:172.21.15.7:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x559054891a40 con 0x55905486fc00 2024-09-18T08:14:32.232 INFO:journalctl@ceph.osd.1.smithi007.stdout:Sep 18 08:14:32 smithi007 bash[145914]: 30865df549477a986230448f6dfbc1fbfa574da663a76054079bb27922256cca 2024-09-18T08:14:32.232 INFO:journalctl@ceph.osd.1.smithi007.stdout:Sep 18 08:14:32 smithi007 systemd[1]: Started Ceph osd.1 for 779349ca-7593-11ef-bceb-c7b262605968. 2024-09-18T08:14:32.238 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2 2024-09-18T08:14:32.260 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:14:32.260 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:14:32.261 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2' 2024-09-18T08:14:32.362 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7 2024-09-18T08:14:32.388 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:14:32.388 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:14:32.388 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7' 2024-09-18T08:14:32.489 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6 2024-09-18T08:14:32.511 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:14:32 smithi007 ceph-mon[29459]: purged_snaps scrub starts 2024-09-18T08:14:32.511 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:14:32 smithi007 ceph-mon[29459]: purged_snaps scrub ok 2024-09-18T08:14:32.511 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:14:32 smithi007 ceph-mon[29459]: pgmap v569: 158 pgs: 29 active+clean+remapped, 22 active+undersized, 1 active+remapped+backfill_toofull, 27 active+undersized+degraded, 79 active+clean; 275 MiB data, 5.3 GiB used, 621 GiB / 626 GiB avail; 852 B/s rd, 0 B/s wr, 1.24k op/s; 1498/13752 objects degraded (10.893%); 1778/13752 objects misplaced (12.929%) 2024-09-18T08:14:32.511 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:14:32 smithi007 ceph-mon[29459]: osdmap e303: 8 total, 7 up, 7 in 2024-09-18T08:14:32.511 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:14:32 smithi007 ceph-mon[29459]: 2.1 scrub starts 2024-09-18T08:14:32.511 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:14:32 smithi007 ceph-mon[29459]: 2.1 scrub ok 2024-09-18T08:14:32.512 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:14:32 smithi007 ceph-mon[39249]: purged_snaps scrub starts 2024-09-18T08:14:32.512 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:14:32 smithi007 ceph-mon[39249]: purged_snaps scrub ok 2024-09-18T08:14:32.512 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:14:32 smithi007 ceph-mon[39249]: pgmap v569: 158 pgs: 29 active+clean+remapped, 22 active+undersized, 1 active+remapped+backfill_toofull, 27 active+undersized+degraded, 79 active+clean; 275 MiB data, 5.3 GiB used, 621 GiB / 626 GiB avail; 852 B/s rd, 0 B/s wr, 1.24k op/s; 1498/13752 objects degraded (10.893%); 1778/13752 objects misplaced (12.929%) 2024-09-18T08:14:32.512 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:14:32 smithi007 ceph-mon[39249]: osdmap e303: 8 total, 7 up, 7 in 2024-09-18T08:14:32.512 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:14:32 smithi007 ceph-mon[39249]: 2.1 scrub starts 2024-09-18T08:14:32.512 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:14:32 smithi007 ceph-mon[39249]: 2.1 scrub ok 2024-09-18T08:14:32.514 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:14:32.515 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:14:32.515 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6' 2024-09-18T08:14:32.616 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5 2024-09-18T08:14:32.641 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:14:32.642 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:14:32.644 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5' 2024-09-18T08:14:32.655 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:14:32 smithi050 ceph-mon[32371]: purged_snaps scrub starts 2024-09-18T08:14:32.655 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:14:32 smithi050 ceph-mon[32371]: purged_snaps scrub ok 2024-09-18T08:14:32.656 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:14:32 smithi050 ceph-mon[32371]: pgmap v569: 158 pgs: 29 active+clean+remapped, 22 active+undersized, 1 active+remapped+backfill_toofull, 27 active+undersized+degraded, 79 active+clean; 275 MiB data, 5.3 GiB used, 621 GiB / 626 GiB avail; 852 B/s rd, 0 B/s wr, 1.24k op/s; 1498/13752 objects degraded (10.893%); 1778/13752 objects misplaced (12.929%) 2024-09-18T08:14:32.656 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:14:32 smithi050 ceph-mon[32371]: osdmap e303: 8 total, 7 up, 7 in 2024-09-18T08:14:32.656 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:14:32 smithi050 ceph-mon[32371]: 2.1 scrub starts 2024-09-18T08:14:32.656 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:14:32 smithi050 ceph-mon[32371]: 2.1 scrub ok 2024-09-18T08:14:32.745 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2 2024-09-18T08:14:32.769 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:14:32.770 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:14:32.770 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2' 2024-09-18T08:14:32.871 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5 2024-09-18T08:14:32.896 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:14:32.897 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:14:32.897 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5' 2024-09-18T08:14:32.998 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2 2024-09-18T08:14:33.021 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:14:33.021 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:14:33.022 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2' 2024-09-18T08:14:33.041 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid 779349ca-7593-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-18T08:14:33.122 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5 2024-09-18T08:14:33.149 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:14:33.150 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:14:33.151 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5' 2024-09-18T08:14:33.251 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4 2024-09-18T08:14:33.278 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:14:33.279 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:14:33.279 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4' 2024-09-18T08:14:33.379 INFO:teuthology.orchestra.run.smithi007.stderr:nodeep-scrub is unset 2024-09-18T08:14:33.380 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3 2024-09-18T08:14:33.406 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:14:33.407 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:14:33.408 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3' 2024-09-18T08:14:33.508 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4 2024-09-18T08:14:33.535 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:14:33.536 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:14:33.537 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4' 2024-09-18T08:14:33.638 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7 2024-09-18T08:14:33.655 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:14:33 smithi050 ceph-mon[32371]: 3.2 scrub starts 2024-09-18T08:14:33.655 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:14:33 smithi050 ceph-mon[32371]: 3.2 scrub ok 2024-09-18T08:14:33.655 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:14:33 smithi050 ceph-mon[32371]: from='client.? 172.21.15.7:0/505395823' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-18T08:14:33.655 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:14:33 smithi050 ceph-mon[32371]: 3.17 scrub starts 2024-09-18T08:14:33.656 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:14:33 smithi050 ceph-mon[32371]: 3.17 scrub ok 2024-09-18T08:14:33.665 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:14:33.665 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:14:33.666 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7' 2024-09-18T08:14:33.688 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:14:33 smithi007 ceph-mon[29459]: 3.2 scrub starts 2024-09-18T08:14:33.688 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:14:33 smithi007 ceph-mon[29459]: 3.2 scrub ok 2024-09-18T08:14:33.688 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:14:33 smithi007 ceph-mon[29459]: from='client.? 172.21.15.7:0/505395823' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-18T08:14:33.688 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:14:33 smithi007 ceph-mon[29459]: 3.17 scrub starts 2024-09-18T08:14:33.688 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:14:33 smithi007 ceph-mon[29459]: 3.17 scrub ok 2024-09-18T08:14:33.689 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:14:33 smithi007 ceph-mon[39249]: 3.2 scrub starts 2024-09-18T08:14:33.689 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:14:33 smithi007 ceph-mon[39249]: 3.2 scrub ok 2024-09-18T08:14:33.689 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:14:33 smithi007 ceph-mon[39249]: from='client.? 172.21.15.7:0/505395823' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-18T08:14:33.689 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:14:33 smithi007 ceph-mon[39249]: 3.17 scrub starts 2024-09-18T08:14:33.689 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:14:33 smithi007 ceph-mon[39249]: 3.17 scrub ok 2024-09-18T08:14:33.767 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2 2024-09-18T08:14:33.791 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:14:33.793 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:14:33.793 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2' 2024-09-18T08:14:33.895 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3 2024-09-18T08:14:33.918 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:14:33.919 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:14:33.919 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3' 2024-09-18T08:14:33.980 INFO:journalctl@ceph.osd.1.smithi007.stdout:Sep 18 08:14:33 smithi007 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd-1[145989]: 2024-09-18T08:14:33.651+0000 7f1cb68d9640 -1 Falling back to public interface 2024-09-18T08:14:34.020 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4 2024-09-18T08:14:34.047 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:14:34.048 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:14:34.049 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4' 2024-09-18T08:14:34.149 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7 2024-09-18T08:14:34.176 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:14:34.177 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:14:34.177 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7' 2024-09-18T08:14:34.264 INFO:teuthology.orchestra.run.smithi007.stdout:ERROR: (22) Invalid argument 2024-09-18T08:14:34.264 INFO:teuthology.orchestra.run.smithi007.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-18T08:14:34.277 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4 2024-09-18T08:14:34.304 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:14:34.305 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:14:34.305 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4' 2024-09-18T08:14:34.406 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0 2024-09-18T08:14:34.431 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:14:34.432 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:14:34.433 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0' 2024-09-18T08:14:34.533 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4 2024-09-18T08:14:34.561 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:14:34.562 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:14:34.562 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4' 2024-09-18T08:14:34.655 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:14:34 smithi050 ceph-mon[32371]: pgmap v571: 158 pgs: 29 active+clean+remapped, 22 active+undersized, 1 active+remapped+backfill_toofull, 27 active+undersized+degraded, 79 active+clean; 275 MiB data, 5.3 GiB used, 621 GiB / 626 GiB avail; 769 B/s rd, 0 B/s wr, 1.12k op/s; 1498/13752 objects degraded (10.893%); 1778/13752 objects misplaced (12.929%) 2024-09-18T08:14:34.655 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:14:34 smithi050 ceph-mon[32371]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-18T08:14:34.656 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:14:34 smithi050 ceph-mon[32371]: from='client.? 172.21.15.7:0/505395823' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-18T08:14:34.656 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:14:34 smithi050 ceph-mon[32371]: osdmap e304: 8 total, 7 up, 7 in 2024-09-18T08:14:34.656 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:14:34 smithi050 ceph-mon[32371]: from='mgr.24470 172.21.15.7:0/2594390261' entity='mgr.y' 2024-09-18T08:14:34.656 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:14:34 smithi050 ceph-mon[32371]: from='mgr.24470 172.21.15.7:0/2594390261' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-18T08:14:34.656 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:14:34 smithi050 ceph-mon[32371]: 4.17 scrub starts 2024-09-18T08:14:34.656 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:14:34 smithi050 ceph-mon[32371]: 4.17 scrub ok 2024-09-18T08:14:34.656 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:14:34 smithi050 ceph-mon[32371]: Health check update: Degraded data redundancy: 1498/13752 objects degraded (10.893%), 27 pgs degraded, 49 pgs undersized (PG_DEGRADED) 2024-09-18T08:14:34.656 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:14:34 smithi050 ceph-mon[32371]: osdmap e305: 8 total, 7 up, 7 in 2024-09-18T08:14:34.663 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5 2024-09-18T08:14:34.689 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:14:34.690 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:14:34.690 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5' 2024-09-18T08:14:34.730 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:14:34 smithi007 ceph-mon[29459]: pgmap v571: 158 pgs: 29 active+clean+remapped, 22 active+undersized, 1 active+remapped+backfill_toofull, 27 active+undersized+degraded, 79 active+clean; 275 MiB data, 5.3 GiB used, 621 GiB / 626 GiB avail; 769 B/s rd, 0 B/s wr, 1.12k op/s; 1498/13752 objects degraded (10.893%); 1778/13752 objects misplaced (12.929%) 2024-09-18T08:14:34.730 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:14:34 smithi007 ceph-mon[29459]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-18T08:14:34.730 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:14:34 smithi007 ceph-mon[29459]: from='client.? 172.21.15.7:0/505395823' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-18T08:14:34.731 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:14:34 smithi007 ceph-mon[29459]: osdmap e304: 8 total, 7 up, 7 in 2024-09-18T08:14:34.731 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:14:34 smithi007 ceph-mon[29459]: from='mgr.24470 172.21.15.7:0/2594390261' entity='mgr.y' 2024-09-18T08:14:34.731 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:14:34 smithi007 ceph-mon[29459]: from='mgr.24470 172.21.15.7:0/2594390261' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-18T08:14:34.731 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:14:34 smithi007 ceph-mon[29459]: 4.17 scrub starts 2024-09-18T08:14:34.731 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:14:34 smithi007 ceph-mon[29459]: 4.17 scrub ok 2024-09-18T08:14:34.731 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:14:34 smithi007 ceph-mon[29459]: Health check update: Degraded data redundancy: 1498/13752 objects degraded (10.893%), 27 pgs degraded, 49 pgs undersized (PG_DEGRADED) 2024-09-18T08:14:34.731 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:14:34 smithi007 ceph-mon[29459]: osdmap e305: 8 total, 7 up, 7 in 2024-09-18T08:14:34.731 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:14:34 smithi007 ceph-mon[39249]: pgmap v571: 158 pgs: 29 active+clean+remapped, 22 active+undersized, 1 active+remapped+backfill_toofull, 27 active+undersized+degraded, 79 active+clean; 275 MiB data, 5.3 GiB used, 621 GiB / 626 GiB avail; 769 B/s rd, 0 B/s wr, 1.12k op/s; 1498/13752 objects degraded (10.893%); 1778/13752 objects misplaced (12.929%) 2024-09-18T08:14:34.732 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:14:34 smithi007 ceph-mon[39249]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-18T08:14:34.732 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:14:34 smithi007 ceph-mon[39249]: from='client.? 172.21.15.7:0/505395823' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-18T08:14:34.732 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:14:34 smithi007 ceph-mon[39249]: osdmap e304: 8 total, 7 up, 7 in 2024-09-18T08:14:34.732 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:14:34 smithi007 ceph-mon[39249]: from='mgr.24470 172.21.15.7:0/2594390261' entity='mgr.y' 2024-09-18T08:14:34.732 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:14:34 smithi007 ceph-mon[39249]: from='mgr.24470 172.21.15.7:0/2594390261' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-18T08:14:34.732 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:14:34 smithi007 ceph-mon[39249]: 4.17 scrub starts 2024-09-18T08:14:34.732 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:14:34 smithi007 ceph-mon[39249]: 4.17 scrub ok 2024-09-18T08:14:34.732 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:14:34 smithi007 ceph-mon[39249]: Health check update: Degraded data redundancy: 1498/13752 objects degraded (10.893%), 27 pgs degraded, 49 pgs undersized (PG_DEGRADED) 2024-09-18T08:14:34.732 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:14:34 smithi007 ceph-mon[39249]: osdmap e305: 8 total, 7 up, 7 in 2024-09-18T08:14:34.791 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5 2024-09-18T08:14:34.817 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:14:34.818 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:14:34.818 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5' 2024-09-18T08:14:34.919 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3 2024-09-18T08:14:34.946 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:14:34.947 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:14:34.947 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3' 2024-09-18T08:14:35.048 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0 2024-09-18T08:14:35.074 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:14:35.075 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:14:35.075 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0' 2024-09-18T08:14:35.176 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4 2024-09-18T08:14:35.203 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:14:35.204 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:14:35.205 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4' 2024-09-18T08:14:35.305 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5 2024-09-18T08:14:35.332 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:14:35.332 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:14:35.333 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5' 2024-09-18T08:14:35.433 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7 2024-09-18T08:14:35.459 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:14:35.460 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:14:35.461 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7' 2024-09-18T08:14:35.562 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0 2024-09-18T08:14:35.587 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:14:35.588 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:14:35.588 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0' 2024-09-18T08:14:35.655 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:14:35 smithi050 ceph-mon[32371]: 2.0 scrub starts 2024-09-18T08:14:35.655 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:14:35 smithi050 ceph-mon[32371]: 2.0 scrub ok 2024-09-18T08:14:35.689 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6 2024-09-18T08:14:35.716 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:14:35.716 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:14:35.717 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6' 2024-09-18T08:14:35.730 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:14:35 smithi007 ceph-mon[29459]: 2.0 scrub starts 2024-09-18T08:14:35.730 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:14:35 smithi007 ceph-mon[29459]: 2.0 scrub ok 2024-09-18T08:14:35.730 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:14:35 smithi007 ceph-mon[39249]: 2.0 scrub starts 2024-09-18T08:14:35.730 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:14:35 smithi007 ceph-mon[39249]: 2.0 scrub ok 2024-09-18T08:14:35.818 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4 2024-09-18T08:14:35.843 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:14:35.844 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:14:35.844 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4' 2024-09-18T08:14:35.945 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0 2024-09-18T08:14:35.972 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:14:35.973 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:14:35.974 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0' 2024-09-18T08:14:36.075 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2 2024-09-18T08:14:36.102 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:14:36.103 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:14:36.103 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2' 2024-09-18T08:14:36.204 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0 2024-09-18T08:14:36.231 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:14:36.232 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:14:36.232 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0' 2024-09-18T08:14:36.333 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6 2024-09-18T08:14:36.360 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:14:36.360 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:14:36.361 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6' 2024-09-18T08:14:36.462 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5 2024-09-18T08:14:36.469 INFO:tasks.cephadm.ceph_manager.ceph:removing pool_name unique_pool_2 2024-09-18T08:14:36.469 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid 779349ca-7593-11ef-bceb-c7b262605968 -- ceph osd pool rm unique_pool_2 unique_pool_2 --yes-i-really-really-mean-it 2024-09-18T08:14:36.488 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:14:36.488 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:14:36.489 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5' 2024-09-18T08:14:36.590 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7 2024-09-18T08:14:36.616 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:14:36.616 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:14:36.617 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7' 2024-09-18T08:14:36.655 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:14:36 smithi050 ceph-mon[32371]: pgmap v574: 158 pgs: 29 active+clean+remapped, 25 active+undersized, 1 active+remapped+backfill_toofull, 24 active+undersized+degraded, 79 active+clean; 635 KiB data, 4.6 GiB used, 621 GiB / 626 GiB avail; 1.0 KiB/s rd, 0 B/s wr, 881 op/s; 87/597 objects degraded (14.573%); 43/597 objects misplaced (7.203%) 2024-09-18T08:14:36.655 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:14:36 smithi050 ceph-mon[32371]: 6.f scrub starts 2024-09-18T08:14:36.655 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:14:36 smithi050 ceph-mon[32371]: 6.f scrub ok 2024-09-18T08:14:36.695 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:14:36 smithi007 ceph-mon[29459]: pgmap v574: 158 pgs: 29 active+clean+remapped, 25 active+undersized, 1 active+remapped+backfill_toofull, 24 active+undersized+degraded, 79 active+clean; 635 KiB data, 4.6 GiB used, 621 GiB / 626 GiB avail; 1.0 KiB/s rd, 0 B/s wr, 881 op/s; 87/597 objects degraded (14.573%); 43/597 objects misplaced (7.203%) 2024-09-18T08:14:36.695 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:14:36 smithi007 ceph-mon[29459]: 6.f scrub starts 2024-09-18T08:14:36.695 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:14:36 smithi007 ceph-mon[29459]: 6.f scrub ok 2024-09-18T08:14:36.696 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:14:36 smithi007 ceph-mon[39249]: pgmap v574: 158 pgs: 29 active+clean+remapped, 25 active+undersized, 1 active+remapped+backfill_toofull, 24 active+undersized+degraded, 79 active+clean; 635 KiB data, 4.6 GiB used, 621 GiB / 626 GiB avail; 1.0 KiB/s rd, 0 B/s wr, 881 op/s; 87/597 objects degraded (14.573%); 43/597 objects misplaced (7.203%) 2024-09-18T08:14:36.696 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:14:36 smithi007 ceph-mon[39249]: 6.f scrub starts 2024-09-18T08:14:36.696 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:14:36 smithi007 ceph-mon[39249]: 6.f scrub ok 2024-09-18T08:14:36.717 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0 2024-09-18T08:14:36.741 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:14:36.742 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:14:36.742 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0' 2024-09-18T08:14:36.843 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3 2024-09-18T08:14:36.865 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:14:36.866 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:14:36.866 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3' 2024-09-18T08:14:36.967 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2 2024-09-18T08:14:36.968 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Sep 18 08:14:36 smithi007 ceph-779349ca-7593-11ef-bceb-c7b262605968-mgr-y[29688]: ::ffff:172.21.15.50 - - [18/Sep/2024:08:14:36] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-09-18T08:14:36.992 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:14:36.993 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:14:36.993 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2' 2024-09-18T08:14:37.002 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid 779349ca-7593-11ef-bceb-c7b262605968 -- ceph daemon osd.3 dump_blocked_ops 2024-09-18T08:14:37.094 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5 2024-09-18T08:14:37.121 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:14:37.121 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:14:37.122 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5' 2024-09-18T08:14:37.223 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4 2024-09-18T08:14:37.249 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:14:37.250 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:14:37.251 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4' 2024-09-18T08:14:37.352 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0 2024-09-18T08:14:37.375 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:14:37.375 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:14:37.376 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0' 2024-09-18T08:14:37.476 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7 2024-09-18T08:14:37.503 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:14:37.504 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:14:37.504 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7' 2024-09-18T08:14:37.605 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6 2024-09-18T08:14:37.613 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:14:37 smithi007 ceph-mon[39249]: 3.1 scrub starts 2024-09-18T08:14:37.613 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:14:37 smithi007 ceph-mon[39249]: 3.1 scrub ok 2024-09-18T08:14:37.613 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:14:37 smithi007 ceph-mon[39249]: from='osd.1 v2:172.21.15.7:6805/3300107791' entity='osd.1' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]} : dispatch 2024-09-18T08:14:37.613 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:14:37 smithi007 ceph-mon[39249]: from='osd.1 ' entity='osd.1' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]} : dispatch 2024-09-18T08:14:37.614 INFO:journalctl@ceph.osd.1.smithi007.stdout:Sep 18 08:14:37 smithi007 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd-1[145989]: 2024-09-18T08:14:37.246+0000 7f1cb68d9640 -1 osd.1 243 log_to_monitors true 2024-09-18T08:14:37.614 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:14:37 smithi007 ceph-mon[29459]: 3.1 scrub starts 2024-09-18T08:14:37.614 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:14:37 smithi007 ceph-mon[29459]: 3.1 scrub ok 2024-09-18T08:14:37.614 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:14:37 smithi007 ceph-mon[29459]: from='osd.1 v2:172.21.15.7:6805/3300107791' entity='osd.1' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]} : dispatch 2024-09-18T08:14:37.614 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:14:37 smithi007 ceph-mon[29459]: from='osd.1 ' entity='osd.1' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]} : dispatch 2024-09-18T08:14:37.631 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:14:37.632 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:14:37.632 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6' 2024-09-18T08:14:37.655 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:14:37 smithi050 ceph-mon[32371]: 3.1 scrub starts 2024-09-18T08:14:37.655 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:14:37 smithi050 ceph-mon[32371]: 3.1 scrub ok 2024-09-18T08:14:37.655 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:14:37 smithi050 ceph-mon[32371]: from='osd.1 v2:172.21.15.7:6805/3300107791' entity='osd.1' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]} : dispatch 2024-09-18T08:14:37.656 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:14:37 smithi050 ceph-mon[32371]: from='osd.1 ' entity='osd.1' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]} : dispatch 2024-09-18T08:14:37.733 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3 2024-09-18T08:14:37.734 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/779349ca-7593-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T08:14:37.734 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/779349ca-7593-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T08:14:37.737 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/779349ca-7593-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T08:14:37.770 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:14:37.771 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:14:37.771 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3' 2024-09-18T08:14:37.865 INFO:journalctl@ceph.osd.1.smithi007.stdout:Sep 18 08:14:37 smithi007 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd-1[145989]: 2024-09-18T08:14:37.807+0000 7f1caa67b640 -1 osd.1 306 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-18T08:14:37.873 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6 2024-09-18T08:14:37.900 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:14:37.901 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:14:37.901 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6' 2024-09-18T08:14:38.002 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7 2024-09-18T08:14:38.029 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:14:38.029 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:14:38.030 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7' 2024-09-18T08:14:38.130 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6 2024-09-18T08:14:38.157 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:14:38.157 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:14:38.158 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6' 2024-09-18T08:14:38.258 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5 2024-09-18T08:14:38.284 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:14:38.285 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:14:38.286 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5' 2024-09-18T08:14:38.387 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6 2024-09-18T08:14:38.424 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:14:38.425 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:14:38.426 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6' 2024-09-18T08:14:38.455 INFO:teuthology.orchestra.run.smithi007.stdout:osd.0: {} 2024-09-18T08:14:38.455 INFO:teuthology.orchestra.run.smithi007.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-18T08:14:38.471 INFO:teuthology.orchestra.run.smithi007.stdout:osd.1: {} 2024-09-18T08:14:38.472 INFO:teuthology.orchestra.run.smithi007.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-18T08:14:38.488 INFO:teuthology.orchestra.run.smithi007.stdout:osd.2: {} 2024-09-18T08:14:38.488 INFO:teuthology.orchestra.run.smithi007.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-18T08:14:38.504 INFO:teuthology.orchestra.run.smithi007.stdout:osd.3: {} 2024-09-18T08:14:38.504 INFO:teuthology.orchestra.run.smithi007.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-18T08:14:38.521 INFO:teuthology.orchestra.run.smithi007.stdout:osd.4: {} 2024-09-18T08:14:38.522 INFO:teuthology.orchestra.run.smithi007.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-18T08:14:38.526 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7 2024-09-18T08:14:38.538 INFO:teuthology.orchestra.run.smithi007.stdout:osd.5: {} 2024-09-18T08:14:38.538 INFO:teuthology.orchestra.run.smithi007.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-18T08:14:38.549 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:14:38.550 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:14:38.550 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7' 2024-09-18T08:14:38.555 INFO:teuthology.orchestra.run.smithi007.stdout:osd.6: {} 2024-09-18T08:14:38.556 INFO:teuthology.orchestra.run.smithi007.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-18T08:14:38.573 INFO:teuthology.orchestra.run.smithi007.stdout:osd.7: {} 2024-09-18T08:14:38.573 INFO:teuthology.orchestra.run.smithi007.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-18T08:14:38.651 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0 2024-09-18T08:14:38.655 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:14:38 smithi050 ceph-mon[32371]: pgmap v575: 158 pgs: 29 active+clean+remapped, 25 active+undersized, 1 active+remapped+backfill_toofull, 24 active+undersized+degraded, 79 active+clean; 635 KiB data, 4.6 GiB used, 621 GiB / 626 GiB avail; 853 B/s rd, 0 B/s wr, 731 op/s; 87/597 objects degraded (14.573%); 43/597 objects misplaced (7.203%) 2024-09-18T08:14:38.655 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:14:38 smithi050 ceph-mon[32371]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-09-18T08:14:38.655 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:14:38 smithi050 ceph-mon[32371]: osdmap e306: 8 total, 7 up, 7 in 2024-09-18T08:14:38.655 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:14:38 smithi050 ceph-mon[32371]: from='osd.1 v2:172.21.15.7:6805/3300107791' entity='osd.1' cmd={"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi007", "root=default"]} : dispatch 2024-09-18T08:14:38.655 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:14:38 smithi050 ceph-mon[32371]: from='osd.1 ' entity='osd.1' cmd={"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi007", "root=default"]} : dispatch 2024-09-18T08:14:38.656 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:14:38 smithi050 ceph-mon[32371]: from='osd.1 ' entity='osd.1' 2024-09-18T08:14:38.656 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:14:38 smithi050 ceph-mon[32371]: 5.d scrub starts 2024-09-18T08:14:38.656 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:14:38 smithi050 ceph-mon[32371]: 5.d scrub ok 2024-09-18T08:14:38.675 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:14:38.676 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:14:38.676 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0' 2024-09-18T08:14:38.730 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:14:38 smithi007 ceph-mon[39249]: pgmap v575: 158 pgs: 29 active+clean+remapped, 25 active+undersized, 1 active+remapped+backfill_toofull, 24 active+undersized+degraded, 79 active+clean; 635 KiB data, 4.6 GiB used, 621 GiB / 626 GiB avail; 853 B/s rd, 0 B/s wr, 731 op/s; 87/597 objects degraded (14.573%); 43/597 objects misplaced (7.203%) 2024-09-18T08:14:38.730 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:14:38 smithi007 ceph-mon[39249]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-09-18T08:14:38.731 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:14:38 smithi007 ceph-mon[39249]: osdmap e306: 8 total, 7 up, 7 in 2024-09-18T08:14:38.731 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:14:38 smithi007 ceph-mon[39249]: from='osd.1 v2:172.21.15.7:6805/3300107791' entity='osd.1' cmd={"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi007", "root=default"]} : dispatch 2024-09-18T08:14:38.731 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:14:38 smithi007 ceph-mon[39249]: from='osd.1 ' entity='osd.1' cmd={"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi007", "root=default"]} : dispatch 2024-09-18T08:14:38.731 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:14:38 smithi007 ceph-mon[39249]: from='osd.1 ' entity='osd.1' 2024-09-18T08:14:38.731 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:14:38 smithi007 ceph-mon[39249]: 5.d scrub starts 2024-09-18T08:14:38.731 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:14:38 smithi007 ceph-mon[39249]: 5.d scrub ok 2024-09-18T08:14:38.731 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:14:38 smithi007 ceph-mon[29459]: pgmap v575: 158 pgs: 29 active+clean+remapped, 25 active+undersized, 1 active+remapped+backfill_toofull, 24 active+undersized+degraded, 79 active+clean; 635 KiB data, 4.6 GiB used, 621 GiB / 626 GiB avail; 853 B/s rd, 0 B/s wr, 731 op/s; 87/597 objects degraded (14.573%); 43/597 objects misplaced (7.203%) 2024-09-18T08:14:38.731 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:14:38 smithi007 ceph-mon[29459]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-09-18T08:14:38.731 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:14:38 smithi007 ceph-mon[29459]: osdmap e306: 8 total, 7 up, 7 in 2024-09-18T08:14:38.731 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:14:38 smithi007 ceph-mon[29459]: from='osd.1 v2:172.21.15.7:6805/3300107791' entity='osd.1' cmd={"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi007", "root=default"]} : dispatch 2024-09-18T08:14:38.731 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:14:38 smithi007 ceph-mon[29459]: from='osd.1 ' entity='osd.1' cmd={"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi007", "root=default"]} : dispatch 2024-09-18T08:14:38.732 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:14:38 smithi007 ceph-mon[29459]: from='osd.1 ' entity='osd.1' 2024-09-18T08:14:38.732 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:14:38 smithi007 ceph-mon[29459]: 5.d scrub starts 2024-09-18T08:14:38.732 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:14:38 smithi007 ceph-mon[29459]: 5.d scrub ok 2024-09-18T08:14:38.777 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3 2024-09-18T08:14:38.800 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:14:38.801 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:14:38.801 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3' 2024-09-18T08:14:38.902 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3 2024-09-18T08:14:38.926 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:14:38.926 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:14:38.927 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3' 2024-09-18T08:14:39.027 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2 2024-09-18T08:14:39.049 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:14:39.050 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:14:39.050 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2' 2024-09-18T08:14:39.151 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7 2024-09-18T08:14:39.178 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:14:39.179 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:14:39.179 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7' 2024-09-18T08:14:39.280 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7 2024-09-18T08:14:39.306 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:14:39.307 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:14:39.307 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7' 2024-09-18T08:14:39.342 INFO:teuthology.orchestra.run.smithi007.stderr:pool 'unique_pool_2' removed 2024-09-18T08:14:39.408 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3 2024-09-18T08:14:39.462 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:14:39.463 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:14:39.463 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3' 2024-09-18T08:14:39.564 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7 2024-09-18T08:14:39.586 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:14:39.587 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:14:39.587 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7' 2024-09-18T08:14:39.647 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid 779349ca-7593-11ef-bceb-c7b262605968 -- ceph osd set noscrub 2024-09-18T08:14:39.655 INFO:journalctl@ceph.iscsi.iscsi.a.smithi050.stdout:Sep 18 08:14:39 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-iscsi-iscsi-a[57782]: debug there is no tcmu-runner data available 2024-09-18T08:14:39.656 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:14:39 smithi050 ceph-mon[32371]: 5.10 scrub starts 2024-09-18T08:14:39.656 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:14:39 smithi050 ceph-mon[32371]: 5.10 scrub ok 2024-09-18T08:14:39.656 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:14:39 smithi050 ceph-mon[32371]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-09-18T08:14:39.656 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:14:39 smithi050 ceph-mon[32371]: osd.1 v2:172.21.15.7:6805/3300107791 boot 2024-09-18T08:14:39.656 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:14:39 smithi050 ceph-mon[32371]: osdmap e307: 8 total, 8 up, 7 in 2024-09-18T08:14:39.656 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:14:39 smithi050 ceph-mon[32371]: from='mgr.24470 172.21.15.7:0/2594390261' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 1} : dispatch 2024-09-18T08:14:39.656 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:14:39 smithi050 ceph-mon[32371]: from='client.? 172.21.15.7:0/3555459202' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-18T08:14:39.656 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:14:39 smithi050 ceph-mon[32371]: 5.4 scrub starts 2024-09-18T08:14:39.656 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:14:39 smithi050 ceph-mon[32371]: 5.4 scrub ok 2024-09-18T08:14:39.656 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:14:39 smithi050 ceph-mon[32371]: 4.11 scrub starts 2024-09-18T08:14:39.657 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:14:39 smithi050 ceph-mon[32371]: 4.11 scrub ok 2024-09-18T08:14:39.657 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:14:39 smithi050 ceph-mon[32371]: from='client.? 172.21.15.7:0/546540442' entity='client.admin' cmd={"prefix": "osd pool rm", "pool": "unique_pool_2", "pool2": "unique_pool_2", "yes_i_really_really_mean_it": true} : dispatch 2024-09-18T08:14:39.657 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:14:39 smithi050 ceph-mon[32371]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool rm", "pool": "unique_pool_2", "pool2": "unique_pool_2", "yes_i_really_really_mean_it": true} : dispatch 2024-09-18T08:14:39.657 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:14:39 smithi050 ceph-mon[32371]: Health check update: Degraded data redundancy: 87/597 objects degraded (14.573%), 24 pgs degraded, 49 pgs undersized (PG_DEGRADED) 2024-09-18T08:14:39.657 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:14:39 smithi050 ceph-mon[32371]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "unique_pool_2", "pool2": "unique_pool_2", "yes_i_really_really_mean_it": true}]': finished 2024-09-18T08:14:39.657 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:14:39 smithi050 ceph-mon[32371]: osdmap e308: 8 total, 8 up, 7 in 2024-09-18T08:14:39.676 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:14:39 smithi007 ceph-mon[29459]: 5.10 scrub starts 2024-09-18T08:14:39.676 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:14:39 smithi007 ceph-mon[29459]: 5.10 scrub ok 2024-09-18T08:14:39.677 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:14:39 smithi007 ceph-mon[29459]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-09-18T08:14:39.677 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:14:39 smithi007 ceph-mon[29459]: osd.1 v2:172.21.15.7:6805/3300107791 boot 2024-09-18T08:14:39.677 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:14:39 smithi007 ceph-mon[29459]: osdmap e307: 8 total, 8 up, 7 in 2024-09-18T08:14:39.677 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:14:39 smithi007 ceph-mon[29459]: from='mgr.24470 172.21.15.7:0/2594390261' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 1} : dispatch 2024-09-18T08:14:39.677 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:14:39 smithi007 ceph-mon[29459]: from='client.? 172.21.15.7:0/3555459202' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-18T08:14:39.677 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:14:39 smithi007 ceph-mon[29459]: 5.4 scrub starts 2024-09-18T08:14:39.677 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:14:39 smithi007 ceph-mon[29459]: 5.4 scrub ok 2024-09-18T08:14:39.677 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:14:39 smithi007 ceph-mon[29459]: 4.11 scrub starts 2024-09-18T08:14:39.678 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:14:39 smithi007 ceph-mon[29459]: 4.11 scrub ok 2024-09-18T08:14:39.678 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:14:39 smithi007 ceph-mon[29459]: from='client.? 172.21.15.7:0/546540442' entity='client.admin' cmd={"prefix": "osd pool rm", "pool": "unique_pool_2", "pool2": "unique_pool_2", "yes_i_really_really_mean_it": true} : dispatch 2024-09-18T08:14:39.678 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:14:39 smithi007 ceph-mon[29459]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool rm", "pool": "unique_pool_2", "pool2": "unique_pool_2", "yes_i_really_really_mean_it": true} : dispatch 2024-09-18T08:14:39.678 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:14:39 smithi007 ceph-mon[29459]: Health check update: Degraded data redundancy: 87/597 objects degraded (14.573%), 24 pgs degraded, 49 pgs undersized (PG_DEGRADED) 2024-09-18T08:14:39.678 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:14:39 smithi007 ceph-mon[29459]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "unique_pool_2", "pool2": "unique_pool_2", "yes_i_really_really_mean_it": true}]': finished 2024-09-18T08:14:39.678 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:14:39 smithi007 ceph-mon[29459]: osdmap e308: 8 total, 8 up, 7 in 2024-09-18T08:14:39.678 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:14:39 smithi007 ceph-mon[39249]: 5.10 scrub starts 2024-09-18T08:14:39.679 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:14:39 smithi007 ceph-mon[39249]: 5.10 scrub ok 2024-09-18T08:14:39.679 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:14:39 smithi007 ceph-mon[39249]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-09-18T08:14:39.679 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:14:39 smithi007 ceph-mon[39249]: osd.1 v2:172.21.15.7:6805/3300107791 boot 2024-09-18T08:14:39.679 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:14:39 smithi007 ceph-mon[39249]: osdmap e307: 8 total, 8 up, 7 in 2024-09-18T08:14:39.679 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:14:39 smithi007 ceph-mon[39249]: from='mgr.24470 172.21.15.7:0/2594390261' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 1} : dispatch 2024-09-18T08:14:39.679 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:14:39 smithi007 ceph-mon[39249]: from='client.? 172.21.15.7:0/3555459202' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-18T08:14:39.679 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:14:39 smithi007 ceph-mon[39249]: 5.4 scrub starts 2024-09-18T08:14:39.679 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:14:39 smithi007 ceph-mon[39249]: 5.4 scrub ok 2024-09-18T08:14:39.679 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:14:39 smithi007 ceph-mon[39249]: 4.11 scrub starts 2024-09-18T08:14:39.679 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:14:39 smithi007 ceph-mon[39249]: 4.11 scrub ok 2024-09-18T08:14:39.679 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:14:39 smithi007 ceph-mon[39249]: from='client.? 172.21.15.7:0/546540442' entity='client.admin' cmd={"prefix": "osd pool rm", "pool": "unique_pool_2", "pool2": "unique_pool_2", "yes_i_really_really_mean_it": true} : dispatch 2024-09-18T08:14:39.679 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:14:39 smithi007 ceph-mon[39249]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool rm", "pool": "unique_pool_2", "pool2": "unique_pool_2", "yes_i_really_really_mean_it": true} : dispatch 2024-09-18T08:14:39.679 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:14:39 smithi007 ceph-mon[39249]: Health check update: Degraded data redundancy: 87/597 objects degraded (14.573%), 24 pgs degraded, 49 pgs undersized (PG_DEGRADED) 2024-09-18T08:14:39.680 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:14:39 smithi007 ceph-mon[39249]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "unique_pool_2", "pool2": "unique_pool_2", "yes_i_really_really_mean_it": true}]': finished 2024-09-18T08:14:39.680 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:14:39 smithi007 ceph-mon[39249]: osdmap e308: 8 total, 8 up, 7 in 2024-09-18T08:14:39.688 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4 2024-09-18T08:14:39.710 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:14:39.711 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:14:39.711 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4' 2024-09-18T08:14:39.812 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4 2024-09-18T08:14:39.834 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:14:39.835 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:14:39.835 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4' 2024-09-18T08:14:39.936 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4 2024-09-18T08:14:39.958 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:14:39.959 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:14:39.959 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4' 2024-09-18T08:14:40.057 INFO:teuthology.task.full_sequential:In full_sequential, running task radosbench... 2024-09-18T08:14:40.058 INFO:tasks.radosbench:Beginning radosbench... 2024-09-18T08:14:40.058 INFO:tasks.cephadm.ceph_manager.ceph:creating pool_name unique_pool_3 2024-09-18T08:14:40.058 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid 779349ca-7593-11ef-bceb-c7b262605968 -- ceph osd pool create unique_pool_3 16 2024-09-18T08:14:40.060 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0 2024-09-18T08:14:40.120 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:14:40.121 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:14:40.121 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0' 2024-09-18T08:14:40.222 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4 2024-09-18T08:14:40.246 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:14:40.246 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:14:40.247 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4' 2024-09-18T08:14:40.348 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3 2024-09-18T08:14:40.383 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:14:40.383 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:14:40.384 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3' 2024-09-18T08:14:40.484 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6 2024-09-18T08:14:40.508 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:14:40.508 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:14:40.509 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6' 2024-09-18T08:14:40.595 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:14:40 smithi007 ceph-mon[29459]: purged_snaps scrub starts 2024-09-18T08:14:40.595 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:14:40 smithi007 ceph-mon[29459]: purged_snaps scrub ok 2024-09-18T08:14:40.595 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:14:40 smithi007 ceph-mon[29459]: pgmap v579: 142 pgs: 20 peering, 19 remapped+peering, 5 active+clean+remapped, 13 active+undersized, 11 active+undersized+degraded, 74 active+clean; 635 KiB data, 4.5 GiB used, 621 GiB / 626 GiB avail; 408 B/s rd, 0 op/s; 52/597 objects degraded (8.710%); 3/597 objects misplaced (0.503%) 2024-09-18T08:14:40.595 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:14:40 smithi007 ceph-mon[29459]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-18T08:14:40.596 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:14:40 smithi007 ceph-mon[29459]: 1.3 scrub starts 2024-09-18T08:14:40.596 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:14:40 smithi007 ceph-mon[29459]: 1.3 scrub ok 2024-09-18T08:14:40.596 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:14:40 smithi007 ceph-mon[29459]: Health check cleared: PG_BACKFILL_FULL (was: Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull) 2024-09-18T08:14:40.596 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:14:40 smithi007 ceph-mon[29459]: osdmap e309: 8 total, 8 up, 7 in 2024-09-18T08:14:40.596 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:14:40 smithi007 ceph-mon[39249]: purged_snaps scrub starts 2024-09-18T08:14:40.596 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:14:40 smithi007 ceph-mon[39249]: purged_snaps scrub ok 2024-09-18T08:14:40.596 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:14:40 smithi007 ceph-mon[39249]: pgmap v579: 142 pgs: 20 peering, 19 remapped+peering, 5 active+clean+remapped, 13 active+undersized, 11 active+undersized+degraded, 74 active+clean; 635 KiB data, 4.5 GiB used, 621 GiB / 626 GiB avail; 408 B/s rd, 0 op/s; 52/597 objects degraded (8.710%); 3/597 objects misplaced (0.503%) 2024-09-18T08:14:40.596 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:14:40 smithi007 ceph-mon[39249]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-18T08:14:40.596 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:14:40 smithi007 ceph-mon[39249]: 1.3 scrub starts 2024-09-18T08:14:40.596 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:14:40 smithi007 ceph-mon[39249]: 1.3 scrub ok 2024-09-18T08:14:40.597 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:14:40 smithi007 ceph-mon[39249]: Health check cleared: PG_BACKFILL_FULL (was: Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull) 2024-09-18T08:14:40.597 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:14:40 smithi007 ceph-mon[39249]: osdmap e309: 8 total, 8 up, 7 in 2024-09-18T08:14:40.609 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3 2024-09-18T08:14:40.631 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:14:40.632 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:14:40.632 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3' 2024-09-18T08:14:40.655 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:14:40 smithi050 ceph-mon[32371]: purged_snaps scrub starts 2024-09-18T08:14:40.655 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:14:40 smithi050 ceph-mon[32371]: purged_snaps scrub ok 2024-09-18T08:14:40.655 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:14:40 smithi050 ceph-mon[32371]: pgmap v579: 142 pgs: 20 peering, 19 remapped+peering, 5 active+clean+remapped, 13 active+undersized, 11 active+undersized+degraded, 74 active+clean; 635 KiB data, 4.5 GiB used, 621 GiB / 626 GiB avail; 408 B/s rd, 0 op/s; 52/597 objects degraded (8.710%); 3/597 objects misplaced (0.503%) 2024-09-18T08:14:40.655 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:14:40 smithi050 ceph-mon[32371]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-18T08:14:40.656 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:14:40 smithi050 ceph-mon[32371]: 1.3 scrub starts 2024-09-18T08:14:40.656 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:14:40 smithi050 ceph-mon[32371]: 1.3 scrub ok 2024-09-18T08:14:40.656 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:14:40 smithi050 ceph-mon[32371]: Health check cleared: PG_BACKFILL_FULL (was: Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull) 2024-09-18T08:14:40.656 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:14:40 smithi050 ceph-mon[32371]: osdmap e309: 8 total, 8 up, 7 in 2024-09-18T08:14:40.703 INFO:teuthology.orchestra.run.smithi007.stdout:{ 2024-09-18T08:14:40.703 INFO:teuthology.orchestra.run.smithi007.stdout: "ops": [], 2024-09-18T08:14:40.703 INFO:teuthology.orchestra.run.smithi007.stdout: "num_ops": 0 2024-09-18T08:14:40.703 INFO:teuthology.orchestra.run.smithi007.stdout:} 2024-09-18T08:14:40.710 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/779349ca-7593-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T08:14:40.733 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0 2024-09-18T08:14:40.754 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:14:40.755 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:14:40.755 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0' 2024-09-18T08:14:40.856 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3 2024-09-18T08:14:40.878 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:14:40.879 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:14:40.879 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3' 2024-09-18T08:14:40.980 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3 2024-09-18T08:14:41.002 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:14:41.003 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:14:41.003 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3' 2024-09-18T08:14:41.104 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7 2024-09-18T08:14:41.131 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:14:41.131 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:14:41.132 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7' 2024-09-18T08:14:41.233 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3 2024-09-18T08:14:41.255 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:14:41.256 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:14:41.256 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3' 2024-09-18T08:14:41.357 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3 2024-09-18T08:14:41.379 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:14:41.380 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:14:41.380 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3' 2024-09-18T08:14:41.481 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7 2024-09-18T08:14:41.508 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:14:41.509 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:14:41.509 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7' 2024-09-18T08:14:41.610 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4 2024-09-18T08:14:41.636 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:14:41.637 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:14:41.637 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4' 2024-09-18T08:14:41.655 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:14:41 smithi050 ceph-mon[32371]: 4.1c scrub starts 2024-09-18T08:14:41.655 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:14:41 smithi050 ceph-mon[32371]: 4.1c scrub ok 2024-09-18T08:14:41.687 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:14:41 smithi007 ceph-mon[29459]: 4.1c scrub starts 2024-09-18T08:14:41.687 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:14:41 smithi007 ceph-mon[29459]: 4.1c scrub ok 2024-09-18T08:14:41.688 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:14:41 smithi007 ceph-mon[39249]: 4.1c scrub starts 2024-09-18T08:14:41.688 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:14:41 smithi007 ceph-mon[39249]: 4.1c scrub ok 2024-09-18T08:14:41.738 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3 2024-09-18T08:14:41.761 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:14:41.762 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:14:41.762 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3' 2024-09-18T08:14:41.863 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2 2024-09-18T08:14:41.886 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:14:41.886 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:14:41.887 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2' 2024-09-18T08:14:41.895 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/779349ca-7593-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T08:14:41.987 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0 2024-09-18T08:14:42.010 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:14:42.011 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:14:42.011 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0' 2024-09-18T08:14:42.112 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3 2024-09-18T08:14:42.135 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:14:42.135 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:14:42.136 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3' 2024-09-18T08:14:42.236 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2 2024-09-18T08:14:42.254 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid 779349ca-7593-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-18T08:14:42.259 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:14:42.260 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:14:42.260 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2' 2024-09-18T08:14:42.361 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7 2024-09-18T08:14:42.388 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:14:42.389 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:14:42.389 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7' 2024-09-18T08:14:42.395 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/779349ca-7593-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T08:14:42.409 INFO:teuthology.orchestra.run.smithi007.stderr:noscrub is set 2024-09-18T08:14:42.490 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3 2024-09-18T08:14:42.512 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:14:42.513 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:14:42.513 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3' 2024-09-18T08:14:42.614 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0 2024-09-18T08:14:42.637 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:14:42.637 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:14:42.638 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0' 2024-09-18T08:14:42.730 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:14:42 smithi007 ceph-mon[29459]: 3.1a scrub starts 2024-09-18T08:14:42.730 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:14:42 smithi007 ceph-mon[29459]: 3.1a scrub ok 2024-09-18T08:14:42.730 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:14:42 smithi007 ceph-mon[29459]: pgmap v581: 142 pgs: 20 peering, 19 remapped+peering, 5 active+clean+remapped, 13 active+undersized, 11 active+undersized+degraded, 74 active+clean; 635 KiB data, 4.5 GiB used, 621 GiB / 626 GiB avail; 511 B/s rd, 0 op/s; 52/597 objects degraded (8.710%); 3/597 objects misplaced (0.503%) 2024-09-18T08:14:42.731 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:14:42 smithi007 ceph-mon[29459]: 5.8 scrub starts 2024-09-18T08:14:42.731 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:14:42 smithi007 ceph-mon[29459]: 5.8 scrub ok 2024-09-18T08:14:42.731 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:14:42 smithi007 ceph-mon[29459]: from='client.? 172.21.15.7:0/4245210103' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-18T08:14:42.731 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:14:42 smithi007 ceph-mon[29459]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-18T08:14:42.731 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:14:42 smithi007 ceph-mon[29459]: 3.6 scrub starts 2024-09-18T08:14:42.731 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:14:42 smithi007 ceph-mon[29459]: 3.6 scrub ok 2024-09-18T08:14:42.731 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:14:42 smithi007 ceph-mon[29459]: 5.c scrub starts 2024-09-18T08:14:42.731 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:14:42 smithi007 ceph-mon[29459]: 5.c scrub ok 2024-09-18T08:14:42.731 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:14:42 smithi007 ceph-mon[39249]: 3.1a scrub starts 2024-09-18T08:14:42.731 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:14:42 smithi007 ceph-mon[39249]: 3.1a scrub ok 2024-09-18T08:14:42.732 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:14:42 smithi007 ceph-mon[39249]: pgmap v581: 142 pgs: 20 peering, 19 remapped+peering, 5 active+clean+remapped, 13 active+undersized, 11 active+undersized+degraded, 74 active+clean; 635 KiB data, 4.5 GiB used, 621 GiB / 626 GiB avail; 511 B/s rd, 0 op/s; 52/597 objects degraded (8.710%); 3/597 objects misplaced (0.503%) 2024-09-18T08:14:42.732 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:14:42 smithi007 ceph-mon[39249]: 5.8 scrub starts 2024-09-18T08:14:42.732 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:14:42 smithi007 ceph-mon[39249]: 5.8 scrub ok 2024-09-18T08:14:42.732 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:14:42 smithi007 ceph-mon[39249]: from='client.? 172.21.15.7:0/4245210103' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-18T08:14:42.732 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:14:42 smithi007 ceph-mon[39249]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-18T08:14:42.732 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:14:42 smithi007 ceph-mon[39249]: 3.6 scrub starts 2024-09-18T08:14:42.732 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:14:42 smithi007 ceph-mon[39249]: 3.6 scrub ok 2024-09-18T08:14:42.732 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:14:42 smithi007 ceph-mon[39249]: 5.c scrub starts 2024-09-18T08:14:42.732 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:14:42 smithi007 ceph-mon[39249]: 5.c scrub ok 2024-09-18T08:14:42.739 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2 2024-09-18T08:14:42.760 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:14:42.761 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:14:42.761 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2' 2024-09-18T08:14:42.840 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:14:42 smithi050 ceph-mon[32371]: 3.1a scrub starts 2024-09-18T08:14:42.841 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:14:42 smithi050 ceph-mon[32371]: 3.1a scrub ok 2024-09-18T08:14:42.841 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:14:42 smithi050 ceph-mon[32371]: pgmap v581: 142 pgs: 20 peering, 19 remapped+peering, 5 active+clean+remapped, 13 active+undersized, 11 active+undersized+degraded, 74 active+clean; 635 KiB data, 4.5 GiB used, 621 GiB / 626 GiB avail; 511 B/s rd, 0 op/s; 52/597 objects degraded (8.710%); 3/597 objects misplaced (0.503%) 2024-09-18T08:14:42.841 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:14:42 smithi050 ceph-mon[32371]: 5.8 scrub starts 2024-09-18T08:14:42.841 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:14:42 smithi050 ceph-mon[32371]: 5.8 scrub ok 2024-09-18T08:14:42.841 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:14:42 smithi050 ceph-mon[32371]: from='client.? 172.21.15.7:0/4245210103' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-18T08:14:42.841 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:14:42 smithi050 ceph-mon[32371]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-18T08:14:42.841 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:14:42 smithi050 ceph-mon[32371]: 3.6 scrub starts 2024-09-18T08:14:42.841 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:14:42 smithi050 ceph-mon[32371]: 3.6 scrub ok 2024-09-18T08:14:42.841 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:14:42 smithi050 ceph-mon[32371]: 5.c scrub starts 2024-09-18T08:14:42.841 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:14:42 smithi050 ceph-mon[32371]: 5.c scrub ok 2024-09-18T08:14:42.862 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5 2024-09-18T08:14:42.884 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:14:42.885 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:14:42.885 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5' 2024-09-18T08:14:42.986 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3 2024-09-18T08:14:43.008 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:14:43.009 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:14:43.009 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3' 2024-09-18T08:14:43.110 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2 2024-09-18T08:14:43.133 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:14:43.134 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:14:43.134 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2' 2024-09-18T08:14:43.235 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1 2024-09-18T08:14:43.258 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:14:43.259 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:14:43.259 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1' 2024-09-18T08:14:43.360 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3 2024-09-18T08:14:43.383 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:14:43.384 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:14:43.384 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3' 2024-09-18T08:14:43.420 INFO:teuthology.orchestra.run.smithi007.stderr:pool 'unique_pool_3' created 2024-09-18T08:14:43.485 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2 2024-09-18T08:14:43.508 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:14:43.509 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:14:43.509 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2' 2024-09-18T08:14:43.610 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1 2024-09-18T08:14:43.633 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:14:43.633 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:14:43.634 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1' 2024-09-18T08:14:43.730 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:14:43 smithi007 ceph-mon[29459]: 6.1a scrub starts 2024-09-18T08:14:43.730 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:14:43 smithi007 ceph-mon[29459]: 6.1a scrub ok 2024-09-18T08:14:43.730 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:14:43 smithi007 ceph-mon[29459]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-18T08:14:43.730 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:14:43 smithi007 ceph-mon[29459]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-18T08:14:43.731 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:14:43 smithi007 ceph-mon[29459]: osdmap e310: 8 total, 8 up, 7 in 2024-09-18T08:14:43.731 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:14:43 smithi007 ceph-mon[29459]: 3.1c scrub starts 2024-09-18T08:14:43.731 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:14:43 smithi007 ceph-mon[29459]: from='client.? 172.21.15.7:0/979532036' entity='client.admin' cmd={"prefix": "osd pool create", "pool": "unique_pool_3", "pg_num": 16} : dispatch 2024-09-18T08:14:43.731 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:14:43 smithi007 ceph-mon[39249]: 6.1a scrub starts 2024-09-18T08:14:43.731 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:14:43 smithi007 ceph-mon[39249]: 6.1a scrub ok 2024-09-18T08:14:43.731 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:14:43 smithi007 ceph-mon[39249]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-18T08:14:43.731 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:14:43 smithi007 ceph-mon[39249]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-18T08:14:43.731 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:14:43 smithi007 ceph-mon[39249]: osdmap e310: 8 total, 8 up, 7 in 2024-09-18T08:14:43.731 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:14:43 smithi007 ceph-mon[39249]: 3.1c scrub starts 2024-09-18T08:14:43.731 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:14:43 smithi007 ceph-mon[39249]: from='client.? 172.21.15.7:0/979532036' entity='client.admin' cmd={"prefix": "osd pool create", "pool": "unique_pool_3", "pg_num": 16} : dispatch 2024-09-18T08:14:43.735 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5 2024-09-18T08:14:43.757 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:14:43.757 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:14:43.758 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5' 2024-09-18T08:14:43.847 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:14:43 smithi050 ceph-mon[32371]: 6.1a scrub starts 2024-09-18T08:14:43.847 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:14:43 smithi050 ceph-mon[32371]: 6.1a scrub ok 2024-09-18T08:14:43.848 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:14:43 smithi050 ceph-mon[32371]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-18T08:14:43.848 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:14:43 smithi050 ceph-mon[32371]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-18T08:14:43.848 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:14:43 smithi050 ceph-mon[32371]: osdmap e310: 8 total, 8 up, 7 in 2024-09-18T08:14:43.848 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:14:43 smithi050 ceph-mon[32371]: 3.1c scrub starts 2024-09-18T08:14:43.848 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:14:43 smithi050 ceph-mon[32371]: from='client.? 172.21.15.7:0/979532036' entity='client.admin' cmd={"prefix": "osd pool create", "pool": "unique_pool_3", "pg_num": 16} : dispatch 2024-09-18T08:14:43.858 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3 2024-09-18T08:14:43.882 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:14:43.883 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:14:43.883 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3' 2024-09-18T08:14:43.984 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3 2024-09-18T08:14:44.008 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:14:44.008 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:14:44.008 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3' 2024-09-18T08:14:44.109 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6 2024-09-18T08:14:44.132 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:14:44.133 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:14:44.133 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6' 2024-09-18T08:14:44.229 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/779349ca-7593-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T08:14:44.234 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2 2024-09-18T08:14:44.259 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid 779349ca-7593-11ef-bceb-c7b262605968 -- ceph osd pool application enable unique_pool_3 rados --yes-i-really-mean-it || true 2024-09-18T08:14:44.259 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:14:44.260 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:14:44.261 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2' 2024-09-18T08:14:44.339 INFO:teuthology.orchestra.run.smithi007.stdout:{ 2024-09-18T08:14:44.339 INFO:teuthology.orchestra.run.smithi007.stdout: "ops": [], 2024-09-18T08:14:44.339 INFO:teuthology.orchestra.run.smithi007.stdout: "complaint_time": 30, 2024-09-18T08:14:44.339 INFO:teuthology.orchestra.run.smithi007.stdout: "num_blocked_ops": 0 2024-09-18T08:14:44.340 INFO:teuthology.orchestra.run.smithi007.stdout:} 2024-09-18T08:14:44.362 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5 2024-09-18T08:14:44.385 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:14:44.386 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:14:44.386 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5' 2024-09-18T08:14:44.480 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:14:44 smithi007 ceph-mon[39249]: pgmap v583: 142 pgs: 20 peering, 16 remapped+peering, 5 active+clean+remapped, 5 active+undersized, 5 active+undersized+degraded, 91 active+clean; 583 KiB data, 2.6 GiB used, 623 GiB / 626 GiB avail; 24/597 objects degraded (4.020%); 3/597 objects misplaced (0.503%) 2024-09-18T08:14:44.480 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:14:44 smithi007 ceph-mon[39249]: from='client.? 172.21.15.7:0/979532036' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "unique_pool_3", "pg_num": 16}]': finished 2024-09-18T08:14:44.480 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:14:44 smithi007 ceph-mon[39249]: osdmap e311: 8 total, 8 up, 7 in 2024-09-18T08:14:44.480 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:14:44 smithi007 ceph-mon[39249]: Health check update: Degraded data redundancy: 24/597 objects degraded (4.020%), 5 pgs degraded, 10 pgs undersized (PG_DEGRADED) 2024-09-18T08:14:44.480 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:14:44 smithi007 ceph-mon[29459]: pgmap v583: 142 pgs: 20 peering, 16 remapped+peering, 5 active+clean+remapped, 5 active+undersized, 5 active+undersized+degraded, 91 active+clean; 583 KiB data, 2.6 GiB used, 623 GiB / 626 GiB avail; 24/597 objects degraded (4.020%); 3/597 objects misplaced (0.503%) 2024-09-18T08:14:44.481 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:14:44 smithi007 ceph-mon[29459]: from='client.? 172.21.15.7:0/979532036' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "unique_pool_3", "pg_num": 16}]': finished 2024-09-18T08:14:44.481 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:14:44 smithi007 ceph-mon[29459]: osdmap e311: 8 total, 8 up, 7 in 2024-09-18T08:14:44.481 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:14:44 smithi007 ceph-mon[29459]: Health check update: Degraded data redundancy: 24/597 objects degraded (4.020%), 5 pgs degraded, 10 pgs undersized (PG_DEGRADED) 2024-09-18T08:14:44.487 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3 2024-09-18T08:14:44.510 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:14:44.510 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:14:44.511 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3' 2024-09-18T08:14:44.611 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2 2024-09-18T08:14:44.635 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:14:44.635 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:14:44.635 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2' 2024-09-18T08:14:44.736 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6 2024-09-18T08:14:44.762 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:14:44.762 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:14:44.763 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6' 2024-09-18T08:14:44.864 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3 2024-09-18T08:14:44.887 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:14:44.888 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:14:44.888 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3' 2024-09-18T08:14:44.905 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:14:44 smithi050 ceph-mon[32371]: pgmap v583: 142 pgs: 20 peering, 16 remapped+peering, 5 active+clean+remapped, 5 active+undersized, 5 active+undersized+degraded, 91 active+clean; 583 KiB data, 2.6 GiB used, 623 GiB / 626 GiB avail; 24/597 objects degraded (4.020%); 3/597 objects misplaced (0.503%) 2024-09-18T08:14:44.905 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:14:44 smithi050 ceph-mon[32371]: from='client.? 172.21.15.7:0/979532036' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "unique_pool_3", "pg_num": 16}]': finished 2024-09-18T08:14:44.906 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:14:44 smithi050 ceph-mon[32371]: osdmap e311: 8 total, 8 up, 7 in 2024-09-18T08:14:44.906 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:14:44 smithi050 ceph-mon[32371]: Health check update: Degraded data redundancy: 24/597 objects degraded (4.020%), 5 pgs degraded, 10 pgs undersized (PG_DEGRADED) 2024-09-18T08:14:44.989 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2 2024-09-18T08:14:45.015 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:14:45.016 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:14:45.016 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2' 2024-09-18T08:14:45.026 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/779349ca-7593-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T08:14:45.027 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid 779349ca-7593-11ef-bceb-c7b262605968 -- ceph daemon osd.3 dump_historic_ops 2024-09-18T08:14:45.112 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid 779349ca-7593-11ef-bceb-c7b262605968 -- ceph osd set nodeep-scrub 2024-09-18T08:14:45.117 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4 2024-09-18T08:14:45.144 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:14:45.145 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:14:45.145 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4' 2024-09-18T08:14:45.246 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1 2024-09-18T08:14:45.269 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:14:45.269 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:14:45.270 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1' 2024-09-18T08:14:45.370 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4 2024-09-18T08:14:45.398 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:14:45.398 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:14:45.398 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4' 2024-09-18T08:14:45.499 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2 2024-09-18T08:14:45.522 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:14:45.522 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:14:45.523 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2' 2024-09-18T08:14:45.546 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:14:45 smithi007 ceph-mon[29459]: osdmap e312: 8 total, 8 up, 7 in 2024-09-18T08:14:45.546 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:14:45 smithi007 ceph-mon[39249]: osdmap e312: 8 total, 8 up, 7 in 2024-09-18T08:14:45.624 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5 2024-09-18T08:14:45.650 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:14:45.650 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:14:45.651 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5' 2024-09-18T08:14:45.752 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7 2024-09-18T08:14:45.777 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:14:45.778 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:14:45.778 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7' 2024-09-18T08:14:45.879 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6 2024-09-18T08:14:45.905 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:14:45 smithi050 ceph-mon[32371]: osdmap e312: 8 total, 8 up, 7 in 2024-09-18T08:14:45.906 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:14:45.906 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:14:45.907 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6' 2024-09-18T08:14:45.977 INFO:teuthology.orchestra.run.smithi007.stdout:osd.0: {} 2024-09-18T08:14:45.978 INFO:teuthology.orchestra.run.smithi007.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-18T08:14:45.998 INFO:teuthology.orchestra.run.smithi007.stdout:osd.1: {} 2024-09-18T08:14:45.998 INFO:teuthology.orchestra.run.smithi007.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-18T08:14:46.007 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5 2024-09-18T08:14:46.017 INFO:teuthology.orchestra.run.smithi007.stdout:osd.2: {} 2024-09-18T08:14:46.017 INFO:teuthology.orchestra.run.smithi007.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-18T08:14:46.019 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/779349ca-7593-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T08:14:46.019 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/779349ca-7593-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T08:14:46.033 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:14:46.033 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:14:46.034 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5' 2024-09-18T08:14:46.043 INFO:teuthology.orchestra.run.smithi007.stdout:osd.3: {} 2024-09-18T08:14:46.044 INFO:teuthology.orchestra.run.smithi007.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-18T08:14:46.069 INFO:teuthology.orchestra.run.smithi007.stdout:osd.4: {} 2024-09-18T08:14:46.069 INFO:teuthology.orchestra.run.smithi007.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-18T08:14:46.086 INFO:teuthology.orchestra.run.smithi007.stdout:osd.5: {} 2024-09-18T08:14:46.086 INFO:teuthology.orchestra.run.smithi007.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-18T08:14:46.103 INFO:teuthology.orchestra.run.smithi007.stdout:osd.6: {} 2024-09-18T08:14:46.104 INFO:teuthology.orchestra.run.smithi007.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-18T08:14:46.120 INFO:teuthology.orchestra.run.smithi007.stdout:osd.7: {} 2024-09-18T08:14:46.120 INFO:teuthology.orchestra.run.smithi007.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-18T08:14:46.135 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0 2024-09-18T08:14:46.159 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:14:46.160 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:14:46.160 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0' 2024-09-18T08:14:46.261 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5 2024-09-18T08:14:46.289 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:14:46.289 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:14:46.290 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5' 2024-09-18T08:14:46.391 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1 2024-09-18T08:14:46.415 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:14:46.416 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:14:46.416 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1' 2024-09-18T08:14:46.441 INFO:teuthology.orchestra.run.smithi007.stderr:enabled application 'rados' on pool 'unique_pool_3' 2024-09-18T08:14:46.517 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5 2024-09-18T08:14:46.544 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:14:46.545 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:14:46.545 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5' 2024-09-18T08:14:46.646 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2 2024-09-18T08:14:46.670 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:14:46.671 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:14:46.671 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2' 2024-09-18T08:14:46.697 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:14:46 smithi007 ceph-mon[29459]: pgmap v586: 158 pgs: 9 creating+peering, 7 unknown, 142 active+clean; 583 KiB data, 2.7 GiB used, 623 GiB / 626 GiB avail; 0 B/s, 20 objects/s recovering 2024-09-18T08:14:46.697 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:14:46 smithi007 ceph-mon[29459]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-18T08:14:46.697 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:14:46 smithi007 ceph-mon[29459]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 24/597 objects degraded (4.020%), 5 pgs degraded, 10 pgs undersized) 2024-09-18T08:14:46.697 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:14:46 smithi007 ceph-mon[29459]: osdmap e313: 8 total, 8 up, 7 in 2024-09-18T08:14:46.697 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:14:46 smithi007 ceph-mon[29459]: from='client.? 172.21.15.7:0/3210142062' entity='client.admin' cmd={"prefix": "osd pool application enable", "pool": "unique_pool_3", "app": "rados", "yes_i_really_mean_it": true} : dispatch 2024-09-18T08:14:46.697 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:14:46 smithi007 ceph-mon[29459]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool application enable", "pool": "unique_pool_3", "app": "rados", "yes_i_really_mean_it": true} : dispatch 2024-09-18T08:14:46.697 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:14:46 smithi007 ceph-mon[29459]: from='client.? 172.21.15.7:0/2125431191' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-18T08:14:46.697 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:14:46 smithi007 ceph-mon[39249]: pgmap v586: 158 pgs: 9 creating+peering, 7 unknown, 142 active+clean; 583 KiB data, 2.7 GiB used, 623 GiB / 626 GiB avail; 0 B/s, 20 objects/s recovering 2024-09-18T08:14:46.698 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:14:46 smithi007 ceph-mon[39249]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-18T08:14:46.698 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:14:46 smithi007 ceph-mon[39249]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 24/597 objects degraded (4.020%), 5 pgs degraded, 10 pgs undersized) 2024-09-18T08:14:46.698 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:14:46 smithi007 ceph-mon[39249]: osdmap e313: 8 total, 8 up, 7 in 2024-09-18T08:14:46.698 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:14:46 smithi007 ceph-mon[39249]: from='client.? 172.21.15.7:0/3210142062' entity='client.admin' cmd={"prefix": "osd pool application enable", "pool": "unique_pool_3", "app": "rados", "yes_i_really_mean_it": true} : dispatch 2024-09-18T08:14:46.698 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:14:46 smithi007 ceph-mon[39249]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool application enable", "pool": "unique_pool_3", "app": "rados", "yes_i_really_mean_it": true} : dispatch 2024-09-18T08:14:46.698 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:14:46 smithi007 ceph-mon[39249]: from='client.? 172.21.15.7:0/2125431191' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-18T08:14:46.772 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7 2024-09-18T08:14:46.799 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:14:46.799 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:14:46.800 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7' 2024-09-18T08:14:46.901 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2 2024-09-18T08:14:46.905 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:14:46 smithi050 ceph-mon[32371]: pgmap v586: 158 pgs: 9 creating+peering, 7 unknown, 142 active+clean; 583 KiB data, 2.7 GiB used, 623 GiB / 626 GiB avail; 0 B/s, 20 objects/s recovering 2024-09-18T08:14:46.905 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:14:46 smithi050 ceph-mon[32371]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-18T08:14:46.905 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:14:46 smithi050 ceph-mon[32371]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 24/597 objects degraded (4.020%), 5 pgs degraded, 10 pgs undersized) 2024-09-18T08:14:46.905 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:14:46 smithi050 ceph-mon[32371]: osdmap e313: 8 total, 8 up, 7 in 2024-09-18T08:14:46.906 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:14:46 smithi050 ceph-mon[32371]: from='client.? 172.21.15.7:0/3210142062' entity='client.admin' cmd={"prefix": "osd pool application enable", "pool": "unique_pool_3", "app": "rados", "yes_i_really_mean_it": true} : dispatch 2024-09-18T08:14:46.906 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:14:46 smithi050 ceph-mon[32371]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool application enable", "pool": "unique_pool_3", "app": "rados", "yes_i_really_mean_it": true} : dispatch 2024-09-18T08:14:46.906 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:14:46 smithi050 ceph-mon[32371]: from='client.? 172.21.15.7:0/2125431191' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-18T08:14:46.922 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 5, 6, 4] out_osds: [7] dead_osds: [] live_osds: [0, 2, 3, 4, 5, 6, 7, 1] 2024-09-18T08:14:46.922 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-18T08:14:46.922 INFO:tasks.thrashosds.thrasher:Removing osd 5, in_osds are: [0, 1, 2, 3, 5, 6, 4] 2024-09-18T08:14:46.923 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid 779349ca-7593-11ef-bceb-c7b262605968 -- ceph osd out 5 2024-09-18T08:14:46.925 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:14:46.925 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:14:46.926 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2' 2024-09-18T08:14:46.948 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Sep 18 08:14:46 smithi007 ceph-779349ca-7593-11ef-bceb-c7b262605968-mgr-y[29688]: ::ffff:172.21.15.50 - - [18/Sep/2024:08:14:46] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-09-18T08:14:47.027 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4 2024-09-18T08:14:47.053 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:14:47.054 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:14:47.054 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4' 2024-09-18T08:14:47.155 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2 2024-09-18T08:14:47.178 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:14:47.179 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:14:47.179 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2' 2024-09-18T08:14:47.281 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2 2024-09-18T08:14:47.302 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:14:47.303 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:14:47.303 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2' 2024-09-18T08:14:47.355 INFO:teuthology.orchestra.run.smithi007.stdout:ERROR: (22) Invalid argument 2024-09-18T08:14:47.355 INFO:teuthology.orchestra.run.smithi007.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-18T08:14:47.404 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3 2024-09-18T08:14:47.426 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:14:47.427 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:14:47.427 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3' 2024-09-18T08:14:47.528 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5 2024-09-18T08:14:47.554 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:14:47.555 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:14:47.555 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5' 2024-09-18T08:14:47.561 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:14:47 smithi007 ceph-mon[39249]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "unique_pool_3", "app": "rados", "yes_i_really_mean_it": true}]': finished 2024-09-18T08:14:47.561 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:14:47 smithi007 ceph-mon[39249]: osdmap e314: 8 total, 8 up, 7 in 2024-09-18T08:14:47.561 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:14:47 smithi007 ceph-mon[29459]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "unique_pool_3", "app": "rados", "yes_i_really_mean_it": true}]': finished 2024-09-18T08:14:47.561 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:14:47 smithi007 ceph-mon[29459]: osdmap e314: 8 total, 8 up, 7 in 2024-09-18T08:14:47.656 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3 2024-09-18T08:14:47.679 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:14:47.679 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:14:47.680 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3' 2024-09-18T08:14:47.780 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1 2024-09-18T08:14:47.803 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:14:47.804 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:14:47.804 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1' 2024-09-18T08:14:47.905 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5 2024-09-18T08:14:47.906 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:14:47 smithi050 ceph-mon[32371]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "unique_pool_3", "app": "rados", "yes_i_really_mean_it": true}]': finished 2024-09-18T08:14:47.906 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:14:47 smithi050 ceph-mon[32371]: osdmap e314: 8 total, 8 up, 7 in 2024-09-18T08:14:47.932 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:14:47.933 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:14:47.933 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5' 2024-09-18T08:14:48.034 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0 2024-09-18T08:14:48.058 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:14:48.058 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:14:48.059 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0' 2024-09-18T08:14:48.159 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2 2024-09-18T08:14:48.184 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:14:48.185 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:14:48.185 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2' 2024-09-18T08:14:48.275 DEBUG:teuthology.orchestra.run.smithi007:> /bin/sh -c 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados --no-log-to-stderr --name client.0 -b 65536 --object-size 65536 -p unique_pool_3 bench 90 write' 2024-09-18T08:14:48.278 INFO:tasks.radosbench:joining radosbench (timing out after 3000s) 2024-09-18T08:14:48.278 INFO:teuthology.orchestra.run:waiting for 3000 2024-09-18T08:14:48.286 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4 2024-09-18T08:14:48.313 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:14:48.314 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:14:48.314 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4' 2024-09-18T08:14:48.333 INFO:tasks.radosbench.radosbench.0.smithi007.stdout:hints = 1 2024-09-18T08:14:48.333 INFO:tasks.radosbench.radosbench.0.smithi007.stdout:Maintaining 16 concurrent writes of 65536 bytes to objects of size 65536 for up to 90 seconds or 0 objects 2024-09-18T08:14:48.333 INFO:tasks.radosbench.radosbench.0.smithi007.stdout:Object prefix: benchmark_data_smithi007_148834 2024-09-18T08:14:48.333 INFO:tasks.radosbench.radosbench.0.smithi007.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-09-18T08:14:48.333 INFO:tasks.radosbench.radosbench.0.smithi007.stdout: 0 0 0 0 0 0 - 0 2024-09-18T08:14:48.415 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4 2024-09-18T08:14:48.451 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:14:48.452 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:14:48.453 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4' 2024-09-18T08:14:48.553 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1 2024-09-18T08:14:48.597 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:14:48.598 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:14:48 smithi007 ceph-mon[29459]: pgmap v589: 158 pgs: 9 creating+peering, 7 unknown, 142 active+clean; 583 KiB data, 2.7 GiB used, 623 GiB / 626 GiB avail; 160 B/s, 18 objects/s recovering 2024-09-18T08:14:48.598 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:14:48 smithi007 ceph-mon[29459]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-09-18T08:14:48.598 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:14:48 smithi007 ceph-mon[39249]: pgmap v589: 158 pgs: 9 creating+peering, 7 unknown, 142 active+clean; 583 KiB data, 2.7 GiB used, 623 GiB / 626 GiB avail; 160 B/s, 18 objects/s recovering 2024-09-18T08:14:48.598 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:14:48 smithi007 ceph-mon[39249]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-09-18T08:14:48.598 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:14:48.599 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1' 2024-09-18T08:14:48.700 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5 2024-09-18T08:14:48.732 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:14:48.732 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:14:48.733 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5' 2024-09-18T08:14:48.834 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5 2024-09-18T08:14:48.865 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:14:48.866 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:14:48.866 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5' 2024-09-18T08:14:48.905 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:14:48 smithi050 ceph-mon[32371]: pgmap v589: 158 pgs: 9 creating+peering, 7 unknown, 142 active+clean; 583 KiB data, 2.7 GiB used, 623 GiB / 626 GiB avail; 160 B/s, 18 objects/s recovering 2024-09-18T08:14:48.905 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:14:48 smithi050 ceph-mon[32371]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-09-18T08:14:48.967 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4 2024-09-18T08:14:49.000 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:14:49.001 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:14:49.001 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4' 2024-09-18T08:14:49.102 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1 2024-09-18T08:14:49.108 DEBUG:teuthology.orchestra.run.smithi050:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid 779349ca-7593-11ef-bceb-c7b262605968 -- ceph daemon osd.4 dump_ops_in_flight 2024-09-18T08:14:49.144 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:14:49.145 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:14:49.145 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1' 2024-09-18T08:14:49.188 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/779349ca-7593-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T08:14:49.245 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0 2024-09-18T08:14:49.293 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:14:49.293 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:14:49.293 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0' 2024-09-18T08:14:49.333 INFO:tasks.radosbench.radosbench.0.smithi007.stdout: 1 16 3194 3178 198.608 198.625 0.00488938 0.00501794 2024-09-18T08:14:49.344 INFO:teuthology.orchestra.run.smithi007.stderr:nodeep-scrub is set 2024-09-18T08:14:49.394 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0 2024-09-18T08:14:49.437 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:14:49.438 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:14:49.438 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0' 2024-09-18T08:14:49.494 INFO:teuthology.orchestra.run.smithi050.stderr:Inferring config /var/lib/ceph/779349ca-7593-11ef-bceb-c7b262605968/mon.b/config 2024-09-18T08:14:49.539 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1 2024-09-18T08:14:49.565 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:14:49.566 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:14:49.566 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1' 2024-09-18T08:14:49.667 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4 2024-09-18T08:14:49.702 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:14:49.703 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:14:49.703 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4' 2024-09-18T08:14:49.727 INFO:journalctl@ceph.iscsi.iscsi.a.smithi050.stdout:Sep 18 08:14:49 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-iscsi-iscsi-a[57782]: debug there is no tcmu-runner data available 2024-09-18T08:14:49.727 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:14:49 smithi050 ceph-mon[32371]: from='mgr.24470 172.21.15.7:0/2594390261' entity='mgr.y' 2024-09-18T08:14:49.727 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:14:49 smithi050 ceph-mon[32371]: from='mgr.24470 172.21.15.7:0/2594390261' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-18T08:14:49.727 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:14:49 smithi050 ceph-mon[32371]: from='client.? 172.21.15.7:0/195780802' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-18T08:14:49.727 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:14:49 smithi050 ceph-mon[32371]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-18T08:14:49.727 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:14:49 smithi050 ceph-mon[32371]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-18T08:14:49.727 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:14:49 smithi050 ceph-mon[32371]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-18T08:14:49.727 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:14:49 smithi050 ceph-mon[32371]: osdmap e315: 8 total, 8 up, 7 in 2024-09-18T08:14:49.804 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3 2024-09-18T08:14:49.829 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:14:49.829 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:14:49.829 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3' 2024-09-18T08:14:49.930 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0 2024-09-18T08:14:49.970 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:14:49.971 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:14:49.971 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0' 2024-09-18T08:14:49.980 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:14:49 smithi007 ceph-mon[29459]: from='mgr.24470 172.21.15.7:0/2594390261' entity='mgr.y' 2024-09-18T08:14:49.980 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:14:49 smithi007 ceph-mon[29459]: from='mgr.24470 172.21.15.7:0/2594390261' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-18T08:14:49.980 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:14:49 smithi007 ceph-mon[29459]: from='client.? 172.21.15.7:0/195780802' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-18T08:14:49.980 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:14:49 smithi007 ceph-mon[29459]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-18T08:14:49.980 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:14:49 smithi007 ceph-mon[29459]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-18T08:14:49.980 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:14:49 smithi007 ceph-mon[29459]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-18T08:14:49.980 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:14:49 smithi007 ceph-mon[29459]: osdmap e315: 8 total, 8 up, 7 in 2024-09-18T08:14:49.981 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:14:49 smithi007 ceph-mon[39249]: from='mgr.24470 172.21.15.7:0/2594390261' entity='mgr.y' 2024-09-18T08:14:49.981 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:14:49 smithi007 ceph-mon[39249]: from='mgr.24470 172.21.15.7:0/2594390261' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-18T08:14:49.981 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:14:49 smithi007 ceph-mon[39249]: from='client.? 172.21.15.7:0/195780802' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-18T08:14:49.981 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:14:49 smithi007 ceph-mon[39249]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-18T08:14:49.981 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:14:49 smithi007 ceph-mon[39249]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-18T08:14:49.981 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:14:49 smithi007 ceph-mon[39249]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-18T08:14:49.981 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:14:49 smithi007 ceph-mon[39249]: osdmap e315: 8 total, 8 up, 7 in 2024-09-18T08:14:50.072 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3 2024-09-18T08:14:50.109 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:14:50.109 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:14:50.110 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3' 2024-09-18T08:14:50.210 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2 2024-09-18T08:14:50.250 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:14:50.251 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:14:50.251 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2' 2024-09-18T08:14:50.328 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid 779349ca-7593-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-18T08:14:50.333 INFO:tasks.radosbench.radosbench.0.smithi007.stdout: 2 15 6001 5986 187.031 175.5 0.00525489 0.00532982 2024-09-18T08:14:50.352 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7 2024-09-18T08:14:50.383 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:14:50.383 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:14:50.384 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7' 2024-09-18T08:14:50.485 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1 2024-09-18T08:14:50.536 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:14:50.537 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:14:50.537 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1' 2024-09-18T08:14:50.638 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2 2024-09-18T08:14:50.678 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:14:50.679 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:14:50.679 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2' 2024-09-18T08:14:50.689 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:14:50 smithi050 ceph-mon[32371]: pgmap v591: 158 pgs: 158 active+clean; 73 MiB data, 2.8 GiB used, 623 GiB / 626 GiB avail; 1.0 KiB/s rd, 15 MiB/s wr, 234 op/s 2024-09-18T08:14:50.689 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:14:50 smithi050 ceph-mon[32371]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-18T08:14:50.732 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:14:50 smithi007 ceph-mon[39249]: pgmap v591: 158 pgs: 158 active+clean; 73 MiB data, 2.8 GiB used, 623 GiB / 626 GiB avail; 1.0 KiB/s rd, 15 MiB/s wr, 234 op/s 2024-09-18T08:14:50.732 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:14:50 smithi007 ceph-mon[39249]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-18T08:14:50.732 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:14:50 smithi007 ceph-mon[29459]: pgmap v591: 158 pgs: 158 active+clean; 73 MiB data, 2.8 GiB used, 623 GiB / 626 GiB avail; 1.0 KiB/s rd, 15 MiB/s wr, 234 op/s 2024-09-18T08:14:50.732 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:14:50 smithi007 ceph-mon[29459]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-18T08:14:50.740 INFO:teuthology.orchestra.run.smithi050.stdout:ERROR: (22) Invalid argument 2024-09-18T08:14:50.740 INFO:teuthology.orchestra.run.smithi050.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-18T08:14:50.780 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2 2024-09-18T08:14:50.805 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/779349ca-7593-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T08:14:50.831 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:14:50.832 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:14:50.833 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2' 2024-09-18T08:14:50.933 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3 2024-09-18T08:14:50.976 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:14:50.977 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:14:50.977 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3' 2024-09-18T08:14:51.078 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5 2024-09-18T08:14:51.112 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:14:51.113 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:14:51.113 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5' 2024-09-18T08:14:51.214 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3 2024-09-18T08:14:51.230 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:14:50 smithi007 ceph-779349ca-7593-11ef-bceb-c7b262605968-mon-a[29455]: 2024-09-18T08:14:50.909+0000 7fcd2b595640 -1 mon.a@0(leader).osd e315 definitely_dead 0 2024-09-18T08:14:51.260 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:14:51.260 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:14:51.261 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3' 2024-09-18T08:14:51.334 INFO:tasks.radosbench.radosbench.0.smithi007.stdout: 3 16 8640 8624 179.641 164.875 0.00558381 0.00555771 2024-09-18T08:14:51.361 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6 2024-09-18T08:14:51.392 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:14:51.393 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:14:51.393 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6' 2024-09-18T08:14:51.491 INFO:teuthology.orchestra.run.smithi007.stdout:osd.0: {} 2024-09-18T08:14:51.491 INFO:teuthology.orchestra.run.smithi007.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-18T08:14:51.494 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4 2024-09-18T08:14:51.530 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:14:51.531 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:14:51.531 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4' 2024-09-18T08:14:51.532 INFO:teuthology.orchestra.run.smithi007.stdout:osd.1: {} 2024-09-18T08:14:51.532 INFO:teuthology.orchestra.run.smithi007.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-18T08:14:51.554 DEBUG:teuthology.orchestra.run.smithi050:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid 779349ca-7593-11ef-bceb-c7b262605968 -- ceph daemon osd.4 dump_blocked_ops 2024-09-18T08:14:51.573 INFO:teuthology.orchestra.run.smithi007.stdout:osd.2: {} 2024-09-18T08:14:51.574 INFO:teuthology.orchestra.run.smithi007.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-18T08:14:51.612 INFO:teuthology.orchestra.run.smithi007.stdout:osd.3: {} 2024-09-18T08:14:51.613 INFO:teuthology.orchestra.run.smithi007.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-18T08:14:51.632 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5 2024-09-18T08:14:51.645 INFO:teuthology.orchestra.run.smithi007.stderr:marked out osd.5. 2024-09-18T08:14:51.658 INFO:teuthology.orchestra.run.smithi007.stdout:osd.4: {} 2024-09-18T08:14:51.658 INFO:teuthology.orchestra.run.smithi007.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-18T08:14:51.673 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:14:51.674 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:14:51.674 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5' 2024-09-18T08:14:51.688 INFO:teuthology.orchestra.run.smithi007.stdout:osd.5: {} 2024-09-18T08:14:51.688 INFO:teuthology.orchestra.run.smithi007.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-18T08:14:51.710 INFO:teuthology.orchestra.run.smithi007.stdout:osd.6: {} 2024-09-18T08:14:51.710 INFO:teuthology.orchestra.run.smithi007.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-18T08:14:51.728 INFO:teuthology.orchestra.run.smithi007.stdout:osd.7: {} 2024-09-18T08:14:51.728 INFO:teuthology.orchestra.run.smithi007.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-18T08:14:51.775 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3 2024-09-18T08:14:51.798 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:14:51.798 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:14:51.799 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3' 2024-09-18T08:14:51.875 INFO:teuthology.orchestra.run.smithi050.stderr:Inferring config /var/lib/ceph/779349ca-7593-11ef-bceb-c7b262605968/mon.b/config 2024-09-18T08:14:51.900 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2 2024-09-18T08:14:51.905 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:14:51 smithi050 ceph-mon[32371]: from='client.? 172.21.15.7:0/3422253412' entity='client.admin' cmd={"prefix": "osd out", "ids": ["5"]} : dispatch 2024-09-18T08:14:51.905 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:14:51 smithi050 ceph-mon[32371]: from='client.? ' entity='client.admin' cmd={"prefix": "osd out", "ids": ["5"]} : dispatch 2024-09-18T08:14:51.906 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:14:51 smithi050 ceph-mon[32371]: Client client.admin marked osd.5 out, while it was still marked up 2024-09-18T08:14:51.906 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:14:51 smithi050 ceph-mon[32371]: from='client.? 172.21.15.7:0/3258011348' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-18T08:14:51.922 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:14:51.922 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:14:51.923 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2' 2024-09-18T08:14:51.980 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:14:51 smithi007 ceph-mon[29459]: from='client.? 172.21.15.7:0/3422253412' entity='client.admin' cmd={"prefix": "osd out", "ids": ["5"]} : dispatch 2024-09-18T08:14:51.980 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:14:51 smithi007 ceph-mon[29459]: from='client.? ' entity='client.admin' cmd={"prefix": "osd out", "ids": ["5"]} : dispatch 2024-09-18T08:14:51.980 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:14:51 smithi007 ceph-mon[29459]: Client client.admin marked osd.5 out, while it was still marked up 2024-09-18T08:14:51.980 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:14:51 smithi007 ceph-mon[29459]: from='client.? 172.21.15.7:0/3258011348' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-18T08:14:51.980 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:14:51 smithi007 ceph-mon[39249]: from='client.? 172.21.15.7:0/3422253412' entity='client.admin' cmd={"prefix": "osd out", "ids": ["5"]} : dispatch 2024-09-18T08:14:51.980 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:14:51 smithi007 ceph-mon[39249]: from='client.? ' entity='client.admin' cmd={"prefix": "osd out", "ids": ["5"]} : dispatch 2024-09-18T08:14:51.980 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:14:51 smithi007 ceph-mon[39249]: Client client.admin marked osd.5 out, while it was still marked up 2024-09-18T08:14:51.981 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:14:51 smithi007 ceph-mon[39249]: from='client.? 172.21.15.7:0/3258011348' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-18T08:14:52.023 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3 2024-09-18T08:14:52.045 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:14:52.046 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:14:52.047 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3' 2024-09-18T08:14:52.147 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5 2024-09-18T08:14:52.185 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:14:52.185 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:14:52.186 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5' 2024-09-18T08:14:52.209 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid 779349ca-7593-11ef-bceb-c7b262605968 -- ceph osd unset noscrub 2024-09-18T08:14:52.286 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3 2024-09-18T08:14:52.308 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:14:52.309 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:14:52.310 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3' 2024-09-18T08:14:52.334 INFO:tasks.radosbench.radosbench.0.smithi007.stdout: 4 16 9501 9485 148.183 53.8125 0.00191923 0.00559973 2024-09-18T08:14:52.368 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2024-09-18T08:14:52.368 INFO:teuthology.orchestra.run.smithi050.stdout: "ops": [], 2024-09-18T08:14:52.368 INFO:teuthology.orchestra.run.smithi050.stdout: "complaint_time": 30, 2024-09-18T08:14:52.368 INFO:teuthology.orchestra.run.smithi050.stdout: "num_blocked_ops": 0 2024-09-18T08:14:52.368 INFO:teuthology.orchestra.run.smithi050.stdout:} 2024-09-18T08:14:52.411 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4 2024-09-18T08:14:52.434 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:14:52.435 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:14:52.435 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4' 2024-09-18T08:14:52.536 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4 2024-09-18T08:14:52.559 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:14:52.559 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:14:52.560 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4' 2024-09-18T08:14:52.661 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1 2024-09-18T08:14:52.710 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:14:52.711 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:14:52.711 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1' 2024-09-18T08:14:52.812 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3 2024-09-18T08:14:52.834 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:14:52.835 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:14:52.835 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3' 2024-09-18T08:14:52.905 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:14:52 smithi050 ceph-mon[32371]: pgmap v592: 158 pgs: 158 active+clean; 73 MiB data, 2.8 GiB used, 623 GiB / 626 GiB avail; 853 B/s rd, 12 MiB/s wr, 192 op/s 2024-09-18T08:14:52.906 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:14:52 smithi050 ceph-mon[32371]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd out", "ids": ["5"]}]': finished 2024-09-18T08:14:52.906 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:14:52 smithi050 ceph-mon[32371]: osdmap e316: 8 total, 8 up, 6 in 2024-09-18T08:14:52.936 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1 2024-09-18T08:14:52.958 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:14:52.958 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:14:52.959 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1' 2024-09-18T08:14:52.980 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:14:52 smithi007 ceph-mon[29459]: pgmap v592: 158 pgs: 158 active+clean; 73 MiB data, 2.8 GiB used, 623 GiB / 626 GiB avail; 853 B/s rd, 12 MiB/s wr, 192 op/s 2024-09-18T08:14:52.980 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:14:52 smithi007 ceph-mon[29459]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd out", "ids": ["5"]}]': finished 2024-09-18T08:14:52.980 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:14:52 smithi007 ceph-mon[29459]: osdmap e316: 8 total, 8 up, 6 in 2024-09-18T08:14:52.980 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:14:52 smithi007 ceph-mon[39249]: pgmap v592: 158 pgs: 158 active+clean; 73 MiB data, 2.8 GiB used, 623 GiB / 626 GiB avail; 853 B/s rd, 12 MiB/s wr, 192 op/s 2024-09-18T08:14:52.980 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:14:52 smithi007 ceph-mon[39249]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd out", "ids": ["5"]}]': finished 2024-09-18T08:14:52.980 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:14:52 smithi007 ceph-mon[39249]: osdmap e316: 8 total, 8 up, 6 in 2024-09-18T08:14:53.000 DEBUG:teuthology.orchestra.run.smithi050:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid 779349ca-7593-11ef-bceb-c7b262605968 -- ceph daemon osd.4 dump_historic_ops 2024-09-18T08:14:53.060 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5 2024-09-18T08:14:53.062 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/779349ca-7593-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T08:14:53.082 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:14:53.083 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:14:53.083 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5' 2024-09-18T08:14:53.184 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1 2024-09-18T08:14:53.207 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:14:53.207 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:14:53.207 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1' 2024-09-18T08:14:53.260 INFO:teuthology.orchestra.run.smithi050.stderr:Inferring config /var/lib/ceph/779349ca-7593-11ef-bceb-c7b262605968/mon.b/config 2024-09-18T08:14:53.308 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5 2024-09-18T08:14:53.332 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:14:53.332 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:14:53.332 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5' 2024-09-18T08:14:53.334 INFO:tasks.radosbench.radosbench.0.smithi007.stdout: 5 16 9501 9485 118.548 0 - 0.00559973 2024-09-18T08:14:53.433 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3 2024-09-18T08:14:53.456 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:14:53.456 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:14:53.457 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3' 2024-09-18T08:14:53.557 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6 2024-09-18T08:14:53.579 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:14:53.580 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:14:53.580 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6' 2024-09-18T08:14:53.681 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3 2024-09-18T08:14:53.721 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:14:53.721 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:14:53.721 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3' 2024-09-18T08:14:53.808 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2024-09-18T08:14:53.808 INFO:teuthology.orchestra.run.smithi050.stdout: "size": 20, 2024-09-18T08:14:53.808 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 600, 2024-09-18T08:14:53.809 INFO:teuthology.orchestra.run.smithi050.stdout: "ops": [ 2024-09-18T08:14:53.809 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:14:53.809 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.15216.0:86225 8.5 8:ae8df458:::benchmark_data_smithi007_101583_object86224:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e205)", 2024-09-18T08:14:53.809 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-09-18T08:09:12.178973+0000", 2024-09-18T08:14:53.809 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 341.626622591, 2024-09-18T08:14:53.809 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.99102768600000002, 2024-09-18T08:14:53.809 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-09-18T08:14:53.809 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-09-18T08:14:53.809 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "reached pg", 2024-09-18T08:14:53.809 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2024-09-18T08:14:53.809 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.15216", 2024-09-18T08:14:53.809 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "172.21.15.7:0/2596041177", 2024-09-18T08:14:53.810 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 86225 2024-09-18T08:14:53.810 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:14:53.810 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-09-18T08:14:53.810 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:14:53.810 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-09-18T08:14:53.810 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:12.178973+0000", 2024-09-18T08:14:53.810 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-18T08:14:53.810 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:14:53.810 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:14:53.810 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-09-18T08:14:53.810 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:12.178973+0000", 2024-09-18T08:14:53.810 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-18T08:14:53.810 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:14:53.811 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:14:53.811 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-09-18T08:14:53.811 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:12.178975+0000", 2024-09-18T08:14:53.811 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.3969999999999999e-06 2024-09-18T08:14:53.811 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:14:53.811 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:14:53.811 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-09-18T08:14:53.811 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:12.179022+0000", 2024-09-18T08:14:53.811 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4.7580000000000002e-05 2024-09-18T08:14:53.811 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:14:53.811 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:14:53.811 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-09-18T08:14:53.811 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:12.179034+0000", 2024-09-18T08:14:53.812 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.1477e-05 2024-09-18T08:14:53.812 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:14:53.812 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:14:53.812 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-09-18T08:14:53.812 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:12.179040+0000", 2024-09-18T08:14:53.812 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 5.7529999999999998e-06 2024-09-18T08:14:53.812 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:14:53.812 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:14:53.812 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-18T08:14:53.812 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:12.179092+0000", 2024-09-18T08:14:53.812 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 5.2840000000000002e-05 2024-09-18T08:14:53.812 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:14:53.813 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:14:53.813 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "waiting for peered", 2024-09-18T08:14:53.813 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:12.179102+0000", 2024-09-18T08:14:53.813 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 9.5210000000000008e-06 2024-09-18T08:14:53.813 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:14:53.813 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:14:53.813 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-18T08:14:53.813 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:13.169945+0000", 2024-09-18T08:14:53.813 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.99084323600000002 2024-09-18T08:14:53.813 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:14:53.813 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:14:53.813 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-09-18T08:14:53.813 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:13.170001+0000", 2024-09-18T08:14:53.814 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 5.5881999999999998e-05 2024-09-18T08:14:53.814 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-18T08:14:53.814 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-09-18T08:14:53.814 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-18T08:14:53.814 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:14:53.814 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:14:53.814 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.15216.0:86263 8.5 8:af593c7a:::benchmark_data_smithi007_101583_object86262:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e205)", 2024-09-18T08:14:53.814 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-09-18T08:09:12.204249+0000", 2024-09-18T08:14:53.814 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 341.60134658599998, 2024-09-18T08:14:53.814 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.98012944999999996, 2024-09-18T08:14:53.814 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-09-18T08:14:53.814 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-09-18T08:14:53.814 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T08:14:53.815 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2024-09-18T08:14:53.815 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.15216", 2024-09-18T08:14:53.815 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "172.21.15.7:0/2596041177", 2024-09-18T08:14:53.815 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 86263 2024-09-18T08:14:53.815 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:14:53.815 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-09-18T08:14:53.815 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:14:53.815 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-09-18T08:14:53.815 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:12.204249+0000", 2024-09-18T08:14:53.815 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-18T08:14:53.815 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:14:53.815 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:14:53.815 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-09-18T08:14:53.816 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:12.204249+0000", 2024-09-18T08:14:53.816 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-18T08:14:53.816 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:14:53.816 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:14:53.816 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-09-18T08:14:53.816 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:12.204251+0000", 2024-09-18T08:14:53.816 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.635e-06 2024-09-18T08:14:53.816 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:14:53.816 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:14:53.816 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-09-18T08:14:53.816 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:12.204299+0000", 2024-09-18T08:14:53.816 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4.8026999999999999e-05 2024-09-18T08:14:53.816 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:14:53.817 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:14:53.817 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-09-18T08:14:53.817 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:12.204313+0000", 2024-09-18T08:14:53.817 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.4066000000000001e-05 2024-09-18T08:14:53.817 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:14:53.817 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:14:53.817 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-09-18T08:14:53.817 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:12.204321+0000", 2024-09-18T08:14:53.817 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 8.1839999999999999e-06 2024-09-18T08:14:53.817 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:14:53.817 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:14:53.818 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-18T08:14:53.818 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:12.204392+0000", 2024-09-18T08:14:53.818 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 7.0340999999999996e-05 2024-09-18T08:14:53.818 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:14:53.818 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:14:53.818 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "waiting for peered", 2024-09-18T08:14:53.818 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:12.204403+0000", 2024-09-18T08:14:53.818 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.1654000000000001e-05 2024-09-18T08:14:53.818 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:14:53.818 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:14:53.818 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-18T08:14:53.818 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:13.170414+0000", 2024-09-18T08:14:53.818 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.96601117000000003 2024-09-18T08:14:53.819 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:14:53.819 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:14:53.819 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-09-18T08:14:53.819 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:13.170546+0000", 2024-09-18T08:14:53.819 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00013120100000000001 2024-09-18T08:14:53.819 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:14:53.819 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:14:53.819 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "waiting for subops from 6,7", 2024-09-18T08:14:53.819 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:13.170965+0000", 2024-09-18T08:14:53.819 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00041976200000000001 2024-09-18T08:14:53.819 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:14:53.819 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:14:53.819 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:14:53.819 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:13.181803+0000", 2024-09-18T08:14:53.820 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.010837167999999999 2024-09-18T08:14:53.820 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:14:53.820 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:14:53.820 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:14:53.820 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:13.181864+0000", 2024-09-18T08:14:53.820 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 6.1857000000000001e-05 2024-09-18T08:14:53.820 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:14:53.820 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:14:53.820 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "op_commit", 2024-09-18T08:14:53.820 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:13.184283+0000", 2024-09-18T08:14:53.820 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0024184340000000001 2024-09-18T08:14:53.820 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:14:53.820 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:14:53.820 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "commit_sent", 2024-09-18T08:14:53.821 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:13.184339+0000", 2024-09-18T08:14:53.821 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 5.6378999999999997e-05 2024-09-18T08:14:53.821 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:14:53.821 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:14:53.821 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-09-18T08:14:53.821 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:13.184379+0000", 2024-09-18T08:14:53.821 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 3.9572e-05 2024-09-18T08:14:53.821 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-18T08:14:53.821 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-09-18T08:14:53.821 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-18T08:14:53.821 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:14:53.821 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:14:53.821 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.15216.0:86267 8.5 8:a135397b:::benchmark_data_smithi007_101583_object86266:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e205)", 2024-09-18T08:14:53.821 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-09-18T08:09:12.207725+0000", 2024-09-18T08:14:53.822 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 341.59787053600002, 2024-09-18T08:14:53.822 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.97676783599999994, 2024-09-18T08:14:53.822 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-09-18T08:14:53.822 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-09-18T08:14:53.822 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T08:14:53.822 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2024-09-18T08:14:53.822 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.15216", 2024-09-18T08:14:53.822 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "172.21.15.7:0/2596041177", 2024-09-18T08:14:53.822 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 86267 2024-09-18T08:14:53.822 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:14:53.822 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-09-18T08:14:53.822 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:14:53.822 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-09-18T08:14:53.823 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:12.207725+0000", 2024-09-18T08:14:53.823 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-18T08:14:53.823 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:14:53.823 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5 2024-09-18T08:14:53.824 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:14:53.825 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-09-18T08:14:53.825 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:12.207725+0000", 2024-09-18T08:14:53.825 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-18T08:14:53.825 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:14:53.825 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:14:53.825 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-09-18T08:14:53.825 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:12.207727+0000", 2024-09-18T08:14:53.825 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.79e-06 2024-09-18T08:14:53.825 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:14:53.825 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:14:53.825 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-09-18T08:14:53.825 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:12.207765+0000", 2024-09-18T08:14:53.825 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 3.7864000000000002e-05 2024-09-18T08:14:53.826 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:14:53.826 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:14:53.826 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-09-18T08:14:53.826 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:12.207783+0000", 2024-09-18T08:14:53.826 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.7779999999999999e-05 2024-09-18T08:14:53.826 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:14:53.826 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:14:53.826 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-09-18T08:14:53.826 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:12.207791+0000", 2024-09-18T08:14:53.826 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 7.9729999999999997e-06 2024-09-18T08:14:53.827 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:14:53.827 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:14:53.827 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-18T08:14:53.827 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:12.207872+0000", 2024-09-18T08:14:53.827 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 8.1318999999999998e-05 2024-09-18T08:14:53.827 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:14:53.827 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:14:53.827 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "waiting for peered", 2024-09-18T08:14:53.827 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:12.207905+0000", 2024-09-18T08:14:53.827 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 3.2694000000000002e-05 2024-09-18T08:14:53.827 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:14:53.827 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:14:53.827 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-18T08:14:53.828 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:13.179332+0000", 2024-09-18T08:14:53.828 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.97142748800000001 2024-09-18T08:14:53.828 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:14:53.828 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:14:53.828 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-09-18T08:14:53.828 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:13.179470+0000", 2024-09-18T08:14:53.828 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00013747400000000001 2024-09-18T08:14:53.828 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:14:53.828 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:14:53.828 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "waiting for subops from 6,7", 2024-09-18T08:14:53.828 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:13.179634+0000", 2024-09-18T08:14:53.828 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00016417600000000001 2024-09-18T08:14:53.828 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:14:53.829 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:14:53.829 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:14:53.829 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:13.184267+0000", 2024-09-18T08:14:53.829 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0046332079999999998 2024-09-18T08:14:53.829 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:14:53.829 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:14:53.829 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "op_commit", 2024-09-18T08:14:53.829 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:13.184386+0000", 2024-09-18T08:14:53.829 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000118993 2024-09-18T08:14:53.829 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:14:53.829 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:14:53.829 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:14:53.829 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:13.184437+0000", 2024-09-18T08:14:53.830 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 5.0988e-05 2024-09-18T08:14:53.830 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:14:53.830 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:14:53.830 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "commit_sent", 2024-09-18T08:14:53.830 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:13.184469+0000", 2024-09-18T08:14:53.830 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 3.1603000000000002e-05 2024-09-18T08:14:53.830 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:14:53.830 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:14:53.830 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-09-18T08:14:53.830 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:13.184493+0000", 2024-09-18T08:14:53.830 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 2.4485999999999999e-05 2024-09-18T08:14:53.830 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-18T08:14:53.830 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-09-18T08:14:53.831 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-18T08:14:53.831 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:14:53.831 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:14:53.831 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.15216.0:86268 8.5 8:aa7096fb:::benchmark_data_smithi007_101583_object86267:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e205)", 2024-09-18T08:14:53.831 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-09-18T08:09:12.208946+0000", 2024-09-18T08:14:53.831 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 341.59664992900002, 2024-09-18T08:14:53.831 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.980827699, 2024-09-18T08:14:53.831 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-09-18T08:14:53.831 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-09-18T08:14:53.831 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T08:14:53.831 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2024-09-18T08:14:53.831 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.15216", 2024-09-18T08:14:53.832 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "172.21.15.7:0/2596041177", 2024-09-18T08:14:53.832 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 86268 2024-09-18T08:14:53.832 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:14:53.832 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-09-18T08:14:53.832 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:14:53.832 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-09-18T08:14:53.832 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:12.208946+0000", 2024-09-18T08:14:53.832 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-18T08:14:53.832 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:14:53.832 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:14:53.832 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-09-18T08:14:53.832 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:12.208946+0000", 2024-09-18T08:14:53.832 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-18T08:14:53.833 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:14:53.833 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:14:53.833 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-09-18T08:14:53.833 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:12.208947+0000", 2024-09-18T08:14:53.833 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.201e-06 2024-09-18T08:14:53.833 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:14:53.833 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:14:53.833 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-09-18T08:14:53.833 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:12.209036+0000", 2024-09-18T08:14:53.833 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 8.8865999999999999e-05 2024-09-18T08:14:53.833 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:14:53.833 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:14:53.833 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-09-18T08:14:53.834 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:12.209049+0000", 2024-09-18T08:14:53.834 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.2795e-05 2024-09-18T08:14:53.834 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:14:53.834 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:14:53.834 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-09-18T08:14:53.834 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:12.209056+0000", 2024-09-18T08:14:53.834 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 7.1160000000000001e-06 2024-09-18T08:14:53.834 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:14:53.834 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:14:53.834 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-18T08:14:53.834 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:12.209116+0000", 2024-09-18T08:14:53.834 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 5.9517e-05 2024-09-18T08:14:53.834 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:14:53.835 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:14:53.835 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "waiting for peered", 2024-09-18T08:14:53.835 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:12.209125+0000", 2024-09-18T08:14:53.835 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 9.6630000000000005e-06 2024-09-18T08:14:53.835 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:14:53.835 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:14:53.835 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-18T08:14:53.835 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:13.182004+0000", 2024-09-18T08:14:53.835 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.97287885799999996 2024-09-18T08:14:53.835 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:14:53.835 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:14:53.835 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-09-18T08:14:53.835 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:13.182091+0000", 2024-09-18T08:14:53.835 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 8.6907e-05 2024-09-18T08:14:53.836 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:14:53.836 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:14:53.836 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "waiting for subops from 6,7", 2024-09-18T08:14:53.836 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:13.182183+0000", 2024-09-18T08:14:53.836 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 9.2131000000000003e-05 2024-09-18T08:14:53.836 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:14:53.836 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:14:53.836 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "op_commit", 2024-09-18T08:14:53.836 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:13.187041+0000", 2024-09-18T08:14:53.836 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0048579210000000003 2024-09-18T08:14:53.836 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:14:53.836 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:14:53.836 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:14:53.836 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:13.189581+0000", 2024-09-18T08:14:53.837 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.002539707 2024-09-18T08:14:53.837 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:14:53.837 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:14:53.837 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:14:53.838 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:13.189646+0000", 2024-09-18T08:14:53.838 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 6.5699999999999998e-05 2024-09-18T08:14:53.838 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:14:53.838 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:14:53.838 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "commit_sent", 2024-09-18T08:14:53.838 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:13.189714+0000", 2024-09-18T08:14:53.838 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 6.7108999999999997e-05 2024-09-18T08:14:53.838 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:14:53.838 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:14:53.838 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-09-18T08:14:53.838 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:13.189774+0000", 2024-09-18T08:14:53.838 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 6.0207999999999997e-05 2024-09-18T08:14:53.839 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-18T08:14:53.839 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-09-18T08:14:53.839 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-18T08:14:53.839 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:14:53.839 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:14:53.839 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.15216.0:86303 8.5 8:aebc4459:::benchmark_data_smithi007_101583_object86302:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e205)", 2024-09-18T08:14:53.839 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-09-18T08:09:12.250052+0000", 2024-09-18T08:14:53.839 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 341.55554437299998, 2024-09-18T08:14:53.839 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.94992247699999999, 2024-09-18T08:14:53.839 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-09-18T08:14:53.839 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-09-18T08:14:53.839 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T08:14:53.839 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2024-09-18T08:14:53.840 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.15216", 2024-09-18T08:14:53.840 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "172.21.15.7:0/2596041177", 2024-09-18T08:14:53.840 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 86303 2024-09-18T08:14:53.840 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:14:53.840 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-09-18T08:14:53.840 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:14:53.840 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-09-18T08:14:53.840 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:12.250052+0000", 2024-09-18T08:14:53.840 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-18T08:14:53.840 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:14:53.840 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:14:53.840 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-09-18T08:14:53.840 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:12.250052+0000", 2024-09-18T08:14:53.841 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-18T08:14:53.841 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:14:53.841 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:14:53.841 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-09-18T08:14:53.841 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:12.250053+0000", 2024-09-18T08:14:53.841 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.6130000000000001e-06 2024-09-18T08:14:53.841 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:14:53.841 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:14:53.841 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-09-18T08:14:53.841 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:12.250090+0000", 2024-09-18T08:14:53.841 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 3.6307999999999999e-05 2024-09-18T08:14:53.841 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:14:53.841 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:14:53.842 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-09-18T08:14:53.842 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:12.250102+0000", 2024-09-18T08:14:53.842 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.2624e-05 2024-09-18T08:14:53.842 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:14:53.842 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:14:53.842 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-09-18T08:14:53.842 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:12.250109+0000", 2024-09-18T08:14:53.842 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 6.426e-06 2024-09-18T08:14:53.842 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:14:53.842 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:14:53.842 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-18T08:14:53.842 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:12.250173+0000", 2024-09-18T08:14:53.842 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 6.3966000000000004e-05 2024-09-18T08:14:53.843 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:14:53.843 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:14:53.843 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "waiting for peered", 2024-09-18T08:14:53.843 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:12.250184+0000", 2024-09-18T08:14:53.843 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.1401e-05 2024-09-18T08:14:53.843 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:14:53.843 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:14:53.843 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-18T08:14:53.843 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:13.184533+0000", 2024-09-18T08:14:53.843 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.93434866299999997 2024-09-18T08:14:53.843 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:14:53.843 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:14:53.843 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-09-18T08:14:53.844 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:13.184634+0000", 2024-09-18T08:14:53.844 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000101139 2024-09-18T08:14:53.844 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:14:53.844 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:14:53.844 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "waiting for subops from 6,7", 2024-09-18T08:14:53.844 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:13.184793+0000", 2024-09-18T08:14:53.844 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00015932000000000001 2024-09-18T08:14:53.844 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:14:53.844 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:14:53.844 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:14:53.844 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:13.194839+0000", 2024-09-18T08:14:53.844 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.010046144 2024-09-18T08:14:53.844 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:14:53.845 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:14:53.845 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:14:53.845 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:13.194928+0000", 2024-09-18T08:14:53.845 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 8.9037999999999997e-05 2024-09-18T08:14:53.845 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:14:53.845 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:14:53.845 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "op_commit", 2024-09-18T08:14:53.845 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:13.199865+0000", 2024-09-18T08:14:53.845 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0049365879999999996 2024-09-18T08:14:53.845 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:14:53.845 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:14:53.845 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "commit_sent", 2024-09-18T08:14:53.845 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:13.199924+0000", 2024-09-18T08:14:53.846 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 5.9383999999999999e-05 2024-09-18T08:14:53.846 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:14:53.846 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:14:53.846 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-09-18T08:14:53.846 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:13.199974+0000", 2024-09-18T08:14:53.846 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4.9863e-05 2024-09-18T08:14:53.846 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-18T08:14:53.846 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-09-18T08:14:53.846 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-18T08:14:53.846 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:14:53.846 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:14:53.846 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.15216.0:86344 8.5 8:ac157b17:::benchmark_data_smithi007_101583_object86343:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e205)", 2024-09-18T08:14:53.846 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-09-18T08:09:12.295743+0000", 2024-09-18T08:14:53.847 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 341.50985269300003, 2024-09-18T08:14:53.847 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.92374953400000004, 2024-09-18T08:14:53.847 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-09-18T08:14:53.847 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-09-18T08:14:53.847 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T08:14:53.847 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2024-09-18T08:14:53.847 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.15216", 2024-09-18T08:14:53.847 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "172.21.15.7:0/2596041177", 2024-09-18T08:14:53.847 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 86344 2024-09-18T08:14:53.847 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:14:53.847 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-09-18T08:14:53.847 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:14:53.847 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-09-18T08:14:53.848 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:12.295743+0000", 2024-09-18T08:14:53.848 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-18T08:14:53.848 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:14:53.848 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:14:53.848 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-09-18T08:14:53.848 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:12.295743+0000", 2024-09-18T08:14:53.848 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-18T08:14:53.848 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:14:53.848 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:14:53.848 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-09-18T08:14:53.848 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:12.295745+0000", 2024-09-18T08:14:53.848 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.57e-06 2024-09-18T08:14:53.848 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:14:53.849 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:14:53.849 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-09-18T08:14:53.849 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:12.295795+0000", 2024-09-18T08:14:53.849 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 5.0062999999999998e-05 2024-09-18T08:14:53.849 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:14:53.849 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:14:53.849 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-09-18T08:14:53.849 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:12.295807+0000", 2024-09-18T08:14:53.849 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.1578e-05 2024-09-18T08:14:53.849 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:14:53.849 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:14:53.849 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-09-18T08:14:53.849 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:12.295812+0000", 2024-09-18T08:14:53.850 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 5.8930000000000002e-06 2024-09-18T08:14:53.850 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:14:53.850 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:14:53.850 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-18T08:14:53.850 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:12.295875+0000", 2024-09-18T08:14:53.850 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 6.2168000000000005e-05 2024-09-18T08:14:53.850 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:14:53.850 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:14:53.850 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "waiting for peered", 2024-09-18T08:14:53.850 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:12.295883+0000", 2024-09-18T08:14:53.850 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 7.9969999999999998e-06 2024-09-18T08:14:53.850 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:14:53.850 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:14:53.850 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-18T08:14:53.851 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:13.187072+0000", 2024-09-18T08:14:53.851 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.89118984899999998 2024-09-18T08:14:53.851 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:14:53.851 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:14:53.851 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-09-18T08:14:53.851 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:13.187193+0000", 2024-09-18T08:14:53.851 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00012062599999999999 2024-09-18T08:14:53.851 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:14:53.851 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:14:53.851 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "waiting for subops from 6,7", 2024-09-18T08:14:53.851 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:13.187351+0000", 2024-09-18T08:14:53.851 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00015781500000000001 2024-09-18T08:14:53.851 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:14:53.852 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:14:53.852 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:14:53.852 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:13.194877+0000", 2024-09-18T08:14:53.852 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0075262139999999998 2024-09-18T08:14:53.852 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:14:53.852 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:14:53.852 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:14:53.852 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:13.195031+0000", 2024-09-18T08:14:53.852 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00015384799999999999 2024-09-18T08:14:53.852 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:14:53.852 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:14:53.852 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "op_commit", 2024-09-18T08:14:53.852 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:13.219386+0000", 2024-09-18T08:14:53.853 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.024355226000000001 2024-09-18T08:14:53.853 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:14:53.853 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:14:53.853 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "commit_sent", 2024-09-18T08:14:53.853 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:13.219440+0000", 2024-09-18T08:14:53.853 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 5.3786999999999998e-05 2024-09-18T08:14:53.853 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:14:53.853 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:14:53.853 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-09-18T08:14:53.853 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:13.219493+0000", 2024-09-18T08:14:53.854 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 5.2899999999999998e-05 2024-09-18T08:14:53.854 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-18T08:14:53.854 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-09-18T08:14:53.854 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-18T08:14:53.854 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:14:53.854 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:14:53.854 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.15216.0:86350 8.5 8:a14e8599:::benchmark_data_smithi007_101583_object86349:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e205)", 2024-09-18T08:14:53.854 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-09-18T08:09:12.297534+0000", 2024-09-18T08:14:53.854 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 341.508062209, 2024-09-18T08:14:53.854 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.922055505, 2024-09-18T08:14:53.854 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-09-18T08:14:53.854 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-09-18T08:14:53.854 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T08:14:53.855 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2024-09-18T08:14:53.855 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.15216", 2024-09-18T08:14:53.855 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "172.21.15.7:0/2596041177", 2024-09-18T08:14:53.855 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 86350 2024-09-18T08:14:53.855 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:14:53.855 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-09-18T08:14:53.855 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:14:53.855 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-09-18T08:14:53.855 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:12.297534+0000", 2024-09-18T08:14:53.856 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-18T08:14:53.856 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:14:53.856 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:14:53.856 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-09-18T08:14:53.856 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:12.297534+0000", 2024-09-18T08:14:53.856 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-18T08:14:53.856 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:14:53.856 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:14:53.856 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-09-18T08:14:53.856 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:12.297534+0000", 2024-09-18T08:14:53.856 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 7.0800000000000004e-07 2024-09-18T08:14:53.856 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:14:53.856 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:14:53.857 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-09-18T08:14:53.857 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:12.297606+0000", 2024-09-18T08:14:53.857 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 7.1075000000000006e-05 2024-09-18T08:14:53.857 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:14:53.857 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:14:53.857 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-09-18T08:14:53.857 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:12.297613+0000", 2024-09-18T08:14:53.857 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 7.6260000000000003e-06 2024-09-18T08:14:53.857 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:14:53.857 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:14:53.857 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-09-18T08:14:53.857 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:12.297618+0000", 2024-09-18T08:14:53.857 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4.8550000000000001e-06 2024-09-18T08:14:53.858 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:14:53.858 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:14:53.858 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-18T08:14:53.858 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:12.297660+0000", 2024-09-18T08:14:53.858 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4.1968000000000002e-05 2024-09-18T08:14:53.858 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:14:53.858 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:14:53.858 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "waiting for peered", 2024-09-18T08:14:53.858 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:12.297666+0000", 2024-09-18T08:14:53.858 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 5.806e-06 2024-09-18T08:14:53.858 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:14:53.858 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:14:53.858 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-18T08:14:53.859 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:13.189816+0000", 2024-09-18T08:14:53.859 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.89215055300000001 2024-09-18T08:14:53.859 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:14:53.859 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:14:53.859 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-09-18T08:14:53.859 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:13.190014+0000", 2024-09-18T08:14:53.859 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000197894 2024-09-18T08:14:53.859 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:14:53.859 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:14:53.859 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "waiting for subops from 6,7", 2024-09-18T08:14:53.859 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:13.190197+0000", 2024-09-18T08:14:53.859 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00018230399999999999 2024-09-18T08:14:53.859 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:14:53.860 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:14:53.860 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:14:53.860 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:13.194999+0000", 2024-09-18T08:14:53.860 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0048023019999999996 2024-09-18T08:14:53.860 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:14:53.860 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:14:53.861 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:14:53.861 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:13.197606+0000", 2024-09-18T08:14:53.861 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0026074890000000002 2024-09-18T08:14:53.861 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:14:53.861 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:14:53.861 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "op_commit", 2024-09-18T08:14:53.861 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:13.219502+0000", 2024-09-18T08:14:53.861 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.02189514 2024-09-18T08:14:53.861 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:14:53.861 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:14:53.861 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "commit_sent", 2024-09-18T08:14:53.861 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:13.219546+0000", 2024-09-18T08:14:53.862 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4.4574999999999999e-05 2024-09-18T08:14:53.862 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:14:53.862 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:14:53.862 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-09-18T08:14:53.862 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:13.219589+0000", 2024-09-18T08:14:53.862 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4.3210000000000001e-05 2024-09-18T08:14:53.862 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-18T08:14:53.862 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-09-18T08:14:53.862 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-18T08:14:53.862 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:14:53.862 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:14:53.862 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.15216.0:86366 8.5 8:a8c4536d:::benchmark_data_smithi007_101583_object86365:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e205)", 2024-09-18T08:14:53.862 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-09-18T08:09:12.318202+0000", 2024-09-18T08:14:53.863 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 341.48739353299999, 2024-09-18T08:14:53.863 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.90148185700000005, 2024-09-18T08:14:53.863 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-09-18T08:14:53.863 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-09-18T08:14:53.863 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T08:14:53.863 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2024-09-18T08:14:53.863 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.15216", 2024-09-18T08:14:53.863 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "172.21.15.7:0/2596041177", 2024-09-18T08:14:53.863 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 86366 2024-09-18T08:14:53.863 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:14:53.863 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-09-18T08:14:53.863 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:14:53.863 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-09-18T08:14:53.864 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:12.318202+0000", 2024-09-18T08:14:53.864 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-18T08:14:53.864 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:14:53.864 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:14:53.864 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-09-18T08:14:53.864 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:12.318202+0000", 2024-09-18T08:14:53.864 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-18T08:14:53.864 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:14:53.864 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:14:53.864 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-09-18T08:14:53.864 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:12.318205+0000", 2024-09-18T08:14:53.864 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 2.074e-06 2024-09-18T08:14:53.864 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:14:53.865 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:14:53.865 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-09-18T08:14:53.865 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:12.318242+0000", 2024-09-18T08:14:53.865 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 3.6996999999999998e-05 2024-09-18T08:14:53.865 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:14:53.865 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:14:53.865 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-09-18T08:14:53.865 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:12.318258+0000", 2024-09-18T08:14:53.865 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.6382e-05 2024-09-18T08:14:53.865 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:14:53.865 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:14:53.865 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-09-18T08:14:53.865 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:12.318266+0000", 2024-09-18T08:14:53.865 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 7.9820000000000008e-06 2024-09-18T08:14:53.866 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:14:53.866 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:14:53.866 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-18T08:14:53.866 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:12.318402+0000", 2024-09-18T08:14:53.866 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000136477 2024-09-18T08:14:53.866 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:14:53.866 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:14:53.866 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "waiting for peered", 2024-09-18T08:14:53.866 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:12.318414+0000", 2024-09-18T08:14:53.866 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.2096000000000001e-05 2024-09-18T08:14:53.866 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:14:53.866 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:14:53.866 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-18T08:14:53.866 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:13.192024+0000", 2024-09-18T08:14:53.867 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.87360952300000005 2024-09-18T08:14:53.867 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:14:53.867 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:14:53.867 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-09-18T08:14:53.867 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:13.192155+0000", 2024-09-18T08:14:53.867 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00013099499999999999 2024-09-18T08:14:53.867 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:14:53.867 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:14:53.867 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "waiting for subops from 6,7", 2024-09-18T08:14:53.867 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:13.192313+0000", 2024-09-18T08:14:53.867 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00015802500000000001 2024-09-18T08:14:53.867 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:14:53.867 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:14:53.868 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:14:53.868 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:13.195063+0000", 2024-09-18T08:14:53.868 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0027500279999999998 2024-09-18T08:14:53.868 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:14:53.868 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:14:53.868 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:14:53.868 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:13.200043+0000", 2024-09-18T08:14:53.868 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0049800929999999997 2024-09-18T08:14:53.868 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:14:53.868 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:14:53.868 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "op_commit", 2024-09-18T08:14:53.868 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:13.219598+0000", 2024-09-18T08:14:53.868 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.019554768 2024-09-18T08:14:53.869 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:14:53.869 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:14:53.869 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "commit_sent", 2024-09-18T08:14:53.869 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:13.219647+0000", 2024-09-18T08:14:53.869 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4.9327999999999999e-05 2024-09-18T08:14:53.869 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:14:53.869 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:14:53.869 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-09-18T08:14:53.869 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:13.219684+0000", 2024-09-18T08:14:53.869 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 3.7088999999999997e-05 2024-09-18T08:14:53.869 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-18T08:14:53.870 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-09-18T08:14:53.870 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-18T08:14:53.870 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:14:53.870 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:14:53.870 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.15216.0:86390 8.5 8:a8c7b3fd:::benchmark_data_smithi007_101583_object86389:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e205)", 2024-09-18T08:14:53.870 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-09-18T08:09:12.356640+0000", 2024-09-18T08:14:53.870 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 341.448956232, 2024-09-18T08:14:53.870 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.86366743700000004, 2024-09-18T08:14:53.870 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-09-18T08:14:53.870 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-09-18T08:14:53.870 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T08:14:53.870 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2024-09-18T08:14:53.870 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.15216", 2024-09-18T08:14:53.871 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "172.21.15.7:0/2596041177", 2024-09-18T08:14:53.871 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 86390 2024-09-18T08:14:53.871 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:14:53.871 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-09-18T08:14:53.871 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:14:53.871 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-09-18T08:14:53.871 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:12.356640+0000", 2024-09-18T08:14:53.871 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-18T08:14:53.871 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:14:53.871 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:14:53.871 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-09-18T08:14:53.871 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:12.356640+0000", 2024-09-18T08:14:53.871 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-18T08:14:53.871 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:14:53.872 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:14:53.872 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-09-18T08:14:53.872 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:12.356641+0000", 2024-09-18T08:14:53.872 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.5430000000000001e-06 2024-09-18T08:14:53.872 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:14:53.872 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:14:53.872 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-09-18T08:14:53.872 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:12.356684+0000", 2024-09-18T08:14:53.872 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4.2222999999999999e-05 2024-09-18T08:14:53.872 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:14:53.873 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:14:53.873 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-09-18T08:14:53.873 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:12.356696+0000", 2024-09-18T08:14:53.873 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.2663000000000001e-05 2024-09-18T08:14:53.873 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:14:53.873 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:14:53.873 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-09-18T08:14:53.873 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:12.356703+0000", 2024-09-18T08:14:53.873 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 6.6710000000000002e-06 2024-09-18T08:14:53.873 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:14:53.873 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:14:53.873 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-18T08:14:53.873 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:12.356759+0000", 2024-09-18T08:14:53.874 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 5.6382000000000003e-05 2024-09-18T08:14:53.874 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:14:53.874 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:14:53.874 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "waiting for peered", 2024-09-18T08:14:53.874 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:12.356767+0000", 2024-09-18T08:14:53.874 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 7.5190000000000003e-06 2024-09-18T08:14:53.875 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:14:53.875 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:14:53.875 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-18T08:14:53.875 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:13.195090+0000", 2024-09-18T08:14:53.875 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.83832288600000004 2024-09-18T08:14:53.875 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:14:53.875 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:14:53.875 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-09-18T08:14:53.875 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:13.195221+0000", 2024-09-18T08:14:53.875 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00013146699999999999 2024-09-18T08:14:53.875 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:14:53.876 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:14:53.876 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "waiting for subops from 6,7", 2024-09-18T08:14:53.876 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:13.195370+0000", 2024-09-18T08:14:53.876 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000148598 2024-09-18T08:14:53.876 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:14:53.876 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:14:53.876 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:14:53.876 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:13.199845+0000", 2024-09-18T08:14:53.876 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0044749999999999998 2024-09-18T08:14:53.876 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:14:53.876 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:14:53.876 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:14:53.877 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:13.207294+0000", 2024-09-18T08:14:53.877 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0074489309999999998 2024-09-18T08:14:53.877 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:14:53.877 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:14:53.877 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "op_commit", 2024-09-18T08:14:53.877 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:13.220218+0000", 2024-09-18T08:14:53.877 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.012924715 2024-09-18T08:14:53.877 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:14:53.877 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:14:53.877 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "commit_sent", 2024-09-18T08:14:53.877 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:13.220272+0000", 2024-09-18T08:14:53.877 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 5.3748000000000001e-05 2024-09-18T08:14:53.877 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:14:53.877 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:14:53.878 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-09-18T08:14:53.878 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:13.220307+0000", 2024-09-18T08:14:53.878 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 3.5091e-05 2024-09-18T08:14:53.878 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-18T08:14:53.878 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-09-18T08:14:53.878 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-18T08:14:53.878 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:14:53.878 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:14:53.878 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.15216.0:86394 8.5 8:a810cd9d:::benchmark_data_smithi007_101583_object86393:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e205)", 2024-09-18T08:14:53.878 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-09-18T08:09:12.360967+0000", 2024-09-18T08:14:53.879 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 341.44462865499997, 2024-09-18T08:14:53.879 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.85950780100000002, 2024-09-18T08:14:53.879 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-09-18T08:14:53.879 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-09-18T08:14:53.879 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T08:14:53.879 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2024-09-18T08:14:53.879 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.15216", 2024-09-18T08:14:53.879 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "172.21.15.7:0/2596041177", 2024-09-18T08:14:53.879 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 86394 2024-09-18T08:14:53.879 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:14:53.879 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-09-18T08:14:53.879 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:14:53.879 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-09-18T08:14:53.879 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:12.360967+0000", 2024-09-18T08:14:53.880 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-18T08:14:53.880 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:14:53.880 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:14:53.880 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-09-18T08:14:53.880 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:12.360967+0000", 2024-09-18T08:14:53.880 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-18T08:14:53.880 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:14:53.880 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:14:53.880 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-09-18T08:14:53.881 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:12.360969+0000", 2024-09-18T08:14:53.881 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.8500000000000001e-06 2024-09-18T08:14:53.881 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:14:53.881 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:14:53.881 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-09-18T08:14:53.881 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:12.361013+0000", 2024-09-18T08:14:53.881 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4.3693000000000003e-05 2024-09-18T08:14:53.881 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:14:53.881 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:14:53.881 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-09-18T08:14:53.881 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:12.361031+0000", 2024-09-18T08:14:53.881 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.8340000000000001e-05 2024-09-18T08:14:53.881 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:14:53.881 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:14:53.882 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-09-18T08:14:53.882 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:12.361040+0000", 2024-09-18T08:14:53.882 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 8.5320000000000006e-06 2024-09-18T08:14:53.882 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:14:53.882 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:14:53.882 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-18T08:14:53.882 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:12.361108+0000", 2024-09-18T08:14:53.882 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 6.7879000000000005e-05 2024-09-18T08:14:53.882 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:14:53.882 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:14:53.882 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "waiting for peered", 2024-09-18T08:14:53.882 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:12.361119+0000", 2024-09-18T08:14:53.882 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.1163e-05 2024-09-18T08:14:53.882 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:14:53.883 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:14:53.883 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-18T08:14:53.883 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:13.197641+0000", 2024-09-18T08:14:53.883 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.83652230299999997 2024-09-18T08:14:53.883 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:14:53.883 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:14:53.883 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-09-18T08:14:53.883 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:13.197776+0000", 2024-09-18T08:14:53.883 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00013499000000000001 2024-09-18T08:14:53.883 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:14:53.884 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:14:53.884 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "waiting for subops from 6,7", 2024-09-18T08:14:53.884 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:13.197954+0000", 2024-09-18T08:14:53.884 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00017745200000000001 2024-09-18T08:14:53.884 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:14:53.884 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:14:53.884 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:14:53.884 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:13.207329+0000", 2024-09-18T08:14:53.884 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0093754460000000008 2024-09-18T08:14:53.884 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:14:53.884 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:14:53.884 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:14:53.884 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:13.207363+0000", 2024-09-18T08:14:53.885 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 3.3535999999999999e-05 2024-09-18T08:14:53.885 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:14:53.885 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:14:53.885 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "op_commit", 2024-09-18T08:14:53.885 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:13.220323+0000", 2024-09-18T08:14:53.885 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.012960121999999999 2024-09-18T08:14:53.885 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:14:53.885 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:14:53.885 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "commit_sent", 2024-09-18T08:14:53.885 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:13.220445+0000", 2024-09-18T08:14:53.885 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00012217099999999999 2024-09-18T08:14:53.885 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:14:53.885 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:14:53.886 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-09-18T08:14:53.886 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:13.220475+0000", 2024-09-18T08:14:53.887 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 3.0324e-05 2024-09-18T08:14:53.887 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-18T08:14:53.887 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-09-18T08:14:53.888 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-18T08:14:53.888 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:14:53.888 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:14:53.888 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.15216.0:86399 8.5 8:a535189e:::benchmark_data_smithi007_101583_object86398:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e205)", 2024-09-18T08:14:53.888 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-09-18T08:09:12.364040+0000", 2024-09-18T08:14:53.888 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 341.441555591, 2024-09-18T08:14:53.888 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.85654366599999998, 2024-09-18T08:14:53.888 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-09-18T08:14:53.888 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-09-18T08:14:53.888 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T08:14:53.888 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2024-09-18T08:14:53.888 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.15216", 2024-09-18T08:14:53.888 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "172.21.15.7:0/2596041177", 2024-09-18T08:14:53.889 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 86399 2024-09-18T08:14:53.889 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:14:53.889 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-09-18T08:14:53.889 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:14:53.889 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-09-18T08:14:53.889 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:12.364040+0000", 2024-09-18T08:14:53.889 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-18T08:14:53.889 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:14:53.889 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:14:53.889 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-09-18T08:14:53.889 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:12.364040+0000", 2024-09-18T08:14:53.889 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-18T08:14:53.889 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:14:53.890 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:14:53.890 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-09-18T08:14:53.890 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:12.364042+0000", 2024-09-18T08:14:53.890 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.407e-06 2024-09-18T08:14:53.890 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:14:53.890 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:14:53.890 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-09-18T08:14:53.890 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:12.364174+0000", 2024-09-18T08:14:53.890 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000132545 2024-09-18T08:14:53.890 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:14:53.890 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:14:53.890 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-09-18T08:14:53.890 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:12.364186+0000", 2024-09-18T08:14:53.890 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.1912e-05 2024-09-18T08:14:53.891 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:14:53.891 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:14:53.891 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-09-18T08:14:53.891 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:12.364192+0000", 2024-09-18T08:14:53.891 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 5.4600000000000002e-06 2024-09-18T08:14:53.891 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:14:53.891 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:14:53.891 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-18T08:14:53.892 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:12.364257+0000", 2024-09-18T08:14:53.892 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 6.5606999999999997e-05 2024-09-18T08:14:53.892 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:14:53.892 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:14:53.892 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "waiting for peered", 2024-09-18T08:14:53.892 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:12.364271+0000", 2024-09-18T08:14:53.892 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.3433e-05 2024-09-18T08:14:53.892 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:14:53.892 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:14:53.892 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-18T08:14:53.892 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:13.200082+0000", 2024-09-18T08:14:53.893 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.83581135500000003 2024-09-18T08:14:53.893 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:14:53.893 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:14:53.893 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-09-18T08:14:53.893 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:13.200201+0000", 2024-09-18T08:14:53.893 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00011913099999999999 2024-09-18T08:14:53.893 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:14:53.893 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:14:53.893 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "waiting for subops from 6,7", 2024-09-18T08:14:53.893 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:13.200360+0000", 2024-09-18T08:14:53.893 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000159173 2024-09-18T08:14:53.893 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:14:53.893 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:14:53.894 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:14:53.894 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:13.207395+0000", 2024-09-18T08:14:53.894 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0070341090000000002 2024-09-18T08:14:53.894 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:14:53.894 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:14:53.894 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:14:53.894 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:13.207427+0000", 2024-09-18T08:14:53.894 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 3.2138000000000003e-05 2024-09-18T08:14:53.894 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:14:53.894 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:14:53.894 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "op_commit", 2024-09-18T08:14:53.894 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:13.220524+0000", 2024-09-18T08:14:53.894 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.013097698 2024-09-18T08:14:53.894 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:14:53.895 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:14:53.895 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "commit_sent", 2024-09-18T08:14:53.895 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:13.220557+0000", 2024-09-18T08:14:53.895 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 3.2682999999999999e-05 2024-09-18T08:14:53.895 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:14:53.895 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:14:53.895 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-09-18T08:14:53.895 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:13.220584+0000", 2024-09-18T08:14:53.895 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 2.7015000000000001e-05 2024-09-18T08:14:53.895 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-18T08:14:53.895 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-09-18T08:14:53.895 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-18T08:14:53.895 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:14:53.895 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:14:53.896 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.15216.0:86429 8.5 8:a7f1f687:::benchmark_data_smithi007_101583_object86428:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e205)", 2024-09-18T08:14:53.896 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-09-18T08:09:12.416915+0000", 2024-09-18T08:14:53.896 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 341.38868082200003, 2024-09-18T08:14:53.896 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.80373175900000005, 2024-09-18T08:14:53.896 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-09-18T08:14:53.896 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-09-18T08:14:53.896 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T08:14:53.896 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2024-09-18T08:14:53.896 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.15216", 2024-09-18T08:14:53.896 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "172.21.15.7:0/2596041177", 2024-09-18T08:14:53.896 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 86429 2024-09-18T08:14:53.896 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:14:53.896 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-09-18T08:14:53.897 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:14:53.897 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-09-18T08:14:53.897 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:12.416915+0000", 2024-09-18T08:14:53.897 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-18T08:14:53.897 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:14:53.897 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:14:53.897 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-09-18T08:14:53.897 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:12.416915+0000", 2024-09-18T08:14:53.897 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-18T08:14:53.897 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:14:53.897 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:14:53.897 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-09-18T08:14:53.897 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:12.416917+0000", 2024-09-18T08:14:53.897 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.7150000000000001e-06 2024-09-18T08:14:53.898 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:14:53.898 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:14:53.898 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-09-18T08:14:53.898 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:12.416952+0000", 2024-09-18T08:14:53.898 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 3.4860000000000002e-05 2024-09-18T08:14:53.898 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:14:53.898 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:14:53.898 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-09-18T08:14:53.898 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:12.416966+0000", 2024-09-18T08:14:53.898 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.3884e-05 2024-09-18T08:14:53.898 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:14:53.898 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:14:53.898 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-09-18T08:14:53.898 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:12.416973+0000", 2024-09-18T08:14:53.899 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 7.5809999999999999e-06 2024-09-18T08:14:53.899 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:14:53.899 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:14:53.899 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-18T08:14:53.899 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:12.417031+0000", 2024-09-18T08:14:53.899 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 5.7544999999999999e-05 2024-09-18T08:14:53.899 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:14:53.899 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:14:53.899 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "waiting for peered", 2024-09-18T08:14:53.899 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:12.417039+0000", 2024-09-18T08:14:53.899 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 8.5369999999999997e-06 2024-09-18T08:14:53.899 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:14:53.899 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:14:53.899 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-18T08:14:53.900 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:13.202704+0000", 2024-09-18T08:14:53.900 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.78566510099999998 2024-09-18T08:14:53.900 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:14:53.900 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:14:53.900 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-09-18T08:14:53.900 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:13.202825+0000", 2024-09-18T08:14:53.900 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00012023600000000001 2024-09-18T08:14:53.900 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:14:53.900 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:14:53.900 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "waiting for subops from 6,7", 2024-09-18T08:14:53.900 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:13.202989+0000", 2024-09-18T08:14:53.900 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00016451000000000001 2024-09-18T08:14:53.900 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:14:53.900 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:14:53.901 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:14:53.901 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:13.207464+0000", 2024-09-18T08:14:53.901 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0044748909999999999 2024-09-18T08:14:53.901 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:14:53.901 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:14:53.901 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:14:53.901 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:13.214034+0000", 2024-09-18T08:14:53.901 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0065698229999999998 2024-09-18T08:14:53.901 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:14:53.901 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:14:53.901 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "op_commit", 2024-09-18T08:14:53.901 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:13.220590+0000", 2024-09-18T08:14:53.901 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0065556909999999998 2024-09-18T08:14:53.901 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:14:53.902 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:14:53.902 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "commit_sent", 2024-09-18T08:14:53.902 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:13.220620+0000", 2024-09-18T08:14:53.902 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 3.0429999999999998e-05 2024-09-18T08:14:53.902 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:14:53.902 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:14:53.902 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-09-18T08:14:53.902 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:13.220647+0000", 2024-09-18T08:14:53.902 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 2.6954999999999999e-05 2024-09-18T08:14:53.902 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-18T08:14:53.902 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-09-18T08:14:53.902 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-18T08:14:53.902 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:14:53.903 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:14:53.903 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.15582.0:53562 9.e 9:7f5cd26d:::benchmark_data_smithi007_124019_object53561:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e243)", 2024-09-18T08:14:53.903 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-09-18T08:11:41.305003+0000", 2024-09-18T08:14:53.903 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 192.50059329800001, 2024-09-18T08:14:53.903 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.032254158, 2024-09-18T08:14:53.903 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-09-18T08:14:53.903 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-09-18T08:14:53.903 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "reached pg", 2024-09-18T08:14:53.903 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2024-09-18T08:14:53.903 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.15582", 2024-09-18T08:14:53.903 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "172.21.15.7:0/3598605839", 2024-09-18T08:14:53.903 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 53562 2024-09-18T08:14:53.903 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:14:53.903 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-09-18T08:14:53.904 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:14:53.904 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-09-18T08:14:53.904 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:11:41.305003+0000", 2024-09-18T08:14:53.904 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-18T08:14:53.904 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:14:53.904 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:14:53.904 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-09-18T08:14:53.904 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:11:41.305003+0000", 2024-09-18T08:14:53.904 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-18T08:14:53.904 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:14:53.904 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:14:53.904 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-09-18T08:14:53.904 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:11:41.305004+0000", 2024-09-18T08:14:53.905 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.6750000000000001e-06 2024-09-18T08:14:53.905 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:14:53.905 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:14:53.905 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-09-18T08:14:53.905 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:11:41.305059+0000", 2024-09-18T08:14:53.905 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 5.4181999999999997e-05 2024-09-18T08:14:53.905 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:14:53.905 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:14:53.905 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-09-18T08:14:53.905 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:11:41.305074+0000", 2024-09-18T08:14:53.905 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.5379999999999998e-05 2024-09-18T08:14:53.905 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:14:53.905 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:14:53.906 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-09-18T08:14:53.906 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:11:41.305083+0000", 2024-09-18T08:14:53.908 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 8.9549999999999998e-06 2024-09-18T08:14:53.908 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:14:53.908 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:14:53.908 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-18T08:14:53.908 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:11:41.305153+0000", 2024-09-18T08:14:53.908 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 6.9703999999999995e-05 2024-09-18T08:14:53.908 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:14:53.909 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:14:53.909 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-09-18T08:14:53.909 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:11:41.305277+0000", 2024-09-18T08:14:53.909 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0001247 2024-09-18T08:14:53.909 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:14:53.909 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:14:53.909 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "waiting for subops from 1,7", 2024-09-18T08:14:53.909 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:11:41.305414+0000", 2024-09-18T08:14:53.909 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00013647900000000001 2024-09-18T08:14:53.909 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:14:53.909 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:14:53.909 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "op_commit", 2024-09-18T08:14:53.909 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:11:41.307315+0000", 2024-09-18T08:14:53.909 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.001901511 2024-09-18T08:14:53.910 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:14:53.910 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:14:53.910 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:14:53.910 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:11:41.307516+0000", 2024-09-18T08:14:53.910 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00020116900000000001 2024-09-18T08:14:53.910 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:14:53.910 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:14:53.910 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-18T08:14:53.910 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:11:41.323519+0000", 2024-09-18T08:14:53.910 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.016002283999999999 2024-09-18T08:14:53.910 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:14:53.910 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:14:53.910 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "waiting for peered", 2024-09-18T08:14:53.911 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:11:41.323530+0000", 2024-09-18T08:14:53.911 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.1729000000000001e-05 2024-09-18T08:14:53.911 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:14:53.911 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:14:53.911 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-18T08:14:53.911 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:11:42.337165+0000", 2024-09-18T08:14:53.911 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.0136343050000001 2024-09-18T08:14:53.911 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:14:53.911 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:14:53.911 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-09-18T08:14:53.911 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:11:42.337257+0000", 2024-09-18T08:14:53.911 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 9.2084999999999997e-05 2024-09-18T08:14:53.911 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-18T08:14:53.911 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-09-18T08:14:53.912 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-18T08:14:53.912 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:14:53.912 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:14:53.912 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.15582.0:53574 9.e 9:7c49e58c:::benchmark_data_smithi007_124019_object53573:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e243)", 2024-09-18T08:14:53.912 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-09-18T08:11:41.323835+0000", 2024-09-18T08:14:53.912 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 192.481760915, 2024-09-18T08:14:53.912 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.0202460840000001, 2024-09-18T08:14:53.912 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-09-18T08:14:53.912 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-09-18T08:14:53.912 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T08:14:53.912 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2024-09-18T08:14:53.912 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.15582", 2024-09-18T08:14:53.912 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "172.21.15.7:0/3598605839", 2024-09-18T08:14:53.913 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 53574 2024-09-18T08:14:53.913 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:14:53.913 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-09-18T08:14:53.913 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:14:53.913 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-09-18T08:14:53.913 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:11:41.323835+0000", 2024-09-18T08:14:53.913 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-18T08:14:53.913 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:14:53.913 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:14:53.913 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-09-18T08:14:53.913 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:11:41.323835+0000", 2024-09-18T08:14:53.913 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-18T08:14:53.913 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:14:53.913 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:14:53.914 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-09-18T08:14:53.914 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:11:41.323837+0000", 2024-09-18T08:14:53.914 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.8470000000000001e-06 2024-09-18T08:14:53.914 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:14:53.914 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:14:53.914 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-09-18T08:14:53.914 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:11:41.323872+0000", 2024-09-18T08:14:53.914 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 3.4660999999999999e-05 2024-09-18T08:14:53.914 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:14:53.914 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:14:53.914 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-09-18T08:14:53.914 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:11:41.323886+0000", 2024-09-18T08:14:53.914 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.4779e-05 2024-09-18T08:14:53.914 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:14:53.915 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:14:53.915 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-09-18T08:14:53.915 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:11:41.323895+0000", 2024-09-18T08:14:53.915 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 8.4630000000000001e-06 2024-09-18T08:14:53.915 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:14:53.915 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:14:53.915 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-18T08:14:53.915 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:11:41.323994+0000", 2024-09-18T08:14:53.915 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 9.9152000000000002e-05 2024-09-18T08:14:53.915 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:14:53.915 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:14:53.915 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "waiting for peered", 2024-09-18T08:14:53.915 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:11:41.324002+0000", 2024-09-18T08:14:53.915 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 8.4039999999999995e-06 2024-09-18T08:14:53.916 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:14:53.916 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:14:53.916 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-18T08:14:53.916 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:11:42.337953+0000", 2024-09-18T08:14:53.916 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.013950753 2024-09-18T08:14:53.916 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:14:53.916 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:14:53.916 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-09-18T08:14:53.916 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:11:42.338094+0000", 2024-09-18T08:14:53.916 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000140371 2024-09-18T08:14:53.916 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:14:53.916 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:14:53.916 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "waiting for subops from 7", 2024-09-18T08:14:53.917 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:11:42.338248+0000", 2024-09-18T08:14:53.917 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000154723 2024-09-18T08:14:53.917 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:14:53.917 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:14:53.917 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:14:53.917 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:11:42.343852+0000", 2024-09-18T08:14:53.917 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0056041909999999997 2024-09-18T08:14:53.917 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:14:53.917 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:14:53.917 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "op_commit", 2024-09-18T08:14:53.917 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:11:42.343917+0000", 2024-09-18T08:14:53.917 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 6.4090000000000005e-05 2024-09-18T08:14:53.917 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:14:53.917 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:14:53.918 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "commit_sent", 2024-09-18T08:14:53.918 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:11:42.343980+0000", 2024-09-18T08:14:53.918 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 6.3769999999999994e-05 2024-09-18T08:14:53.918 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:14:53.918 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:14:53.918 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-09-18T08:14:53.918 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:11:42.344081+0000", 2024-09-18T08:14:53.918 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00010088 2024-09-18T08:14:53.918 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-18T08:14:53.918 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-09-18T08:14:53.918 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-18T08:14:53.918 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:14:53.918 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:14:53.919 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.15582.0:53552 9.4 9:2d548aaa:::benchmark_data_smithi007_124019_object53551:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e244)", 2024-09-18T08:14:53.919 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-09-18T08:11:41.324333+0000", 2024-09-18T08:14:53.919 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 192.48126349099999, 2024-09-18T08:14:53.919 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.0111114809999999, 2024-09-18T08:14:53.919 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-09-18T08:14:53.919 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-09-18T08:14:53.919 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "reached pg", 2024-09-18T08:14:53.919 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2024-09-18T08:14:53.919 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.15582", 2024-09-18T08:14:53.919 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "172.21.15.7:0/3598605839", 2024-09-18T08:14:53.919 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 53552 2024-09-18T08:14:53.919 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:14:53.919 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-09-18T08:14:53.919 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:14:53.920 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-09-18T08:14:53.920 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:11:41.324333+0000", 2024-09-18T08:14:53.920 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-18T08:14:53.920 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:14:53.920 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:14:53.920 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-09-18T08:14:53.920 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:11:41.324333+0000", 2024-09-18T08:14:53.920 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-18T08:14:53.920 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:14:53.920 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:14:53.920 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-09-18T08:14:53.920 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:11:41.324334+0000", 2024-09-18T08:14:53.920 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.217e-06 2024-09-18T08:14:53.921 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:14:53.921 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:14:53.921 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-09-18T08:14:53.921 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:11:41.324369+0000", 2024-09-18T08:14:53.921 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 3.4805999999999998e-05 2024-09-18T08:14:53.921 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:14:53.921 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:14:53.921 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-09-18T08:14:53.921 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:11:41.324382+0000", 2024-09-18T08:14:53.921 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.3739e-05 2024-09-18T08:14:53.921 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:14:53.921 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:14:53.921 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-09-18T08:14:53.921 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:11:41.324390+0000", 2024-09-18T08:14:53.922 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 7.3989999999999998e-06 2024-09-18T08:14:53.922 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:14:53.922 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:14:53.922 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-18T08:14:53.922 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:11:41.324445+0000", 2024-09-18T08:14:53.922 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 5.5535999999999998e-05 2024-09-18T08:14:53.922 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:14:53.922 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:14:53.922 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "waiting for peered", 2024-09-18T08:14:53.922 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:11:41.324453+0000", 2024-09-18T08:14:53.922 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 8.2139999999999996e-06 2024-09-18T08:14:53.922 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:14:53.922 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:14:53.923 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-18T08:14:53.923 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:11:42.335357+0000", 2024-09-18T08:14:53.923 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.010903675 2024-09-18T08:14:53.923 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:14:53.923 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:14:53.923 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-09-18T08:14:53.923 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:11:42.335444+0000", 2024-09-18T08:14:53.923 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 8.6895000000000001e-05 2024-09-18T08:14:53.923 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-18T08:14:53.923 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-09-18T08:14:53.923 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-18T08:14:53.923 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:14:53.923 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:14:53.923 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.15582.0:53563 9.4 9:2254eea2:::benchmark_data_smithi007_124019_object53562:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e244)", 2024-09-18T08:14:53.924 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-09-18T08:11:41.324409+0000", 2024-09-18T08:14:53.924 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 192.48118681299999, 2024-09-18T08:14:53.924 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.022272439, 2024-09-18T08:14:53.924 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-09-18T08:14:53.924 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-09-18T08:14:53.924 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "reached pg", 2024-09-18T08:14:53.924 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2024-09-18T08:14:53.924 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.15582", 2024-09-18T08:14:53.924 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "172.21.15.7:0/3598605839", 2024-09-18T08:14:53.924 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 53563 2024-09-18T08:14:53.924 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:14:53.924 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-09-18T08:14:53.924 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:14:53.925 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-09-18T08:14:53.925 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:11:41.324409+0000", 2024-09-18T08:14:53.925 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-18T08:14:53.925 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:14:53.925 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:14:53.925 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-09-18T08:14:53.925 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:11:41.324409+0000", 2024-09-18T08:14:53.925 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-18T08:14:53.925 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:14:53.925 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:14:53.925 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-09-18T08:14:53.925 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:11:41.324410+0000", 2024-09-18T08:14:53.925 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 5.8400000000000004e-07 2024-09-18T08:14:53.925 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:14:53.926 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:14:53.926 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-09-18T08:14:53.926 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:11:41.324438+0000", 2024-09-18T08:14:53.926 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 2.8438000000000001e-05 2024-09-18T08:14:53.926 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:14:53.926 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:14:53.926 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-09-18T08:14:53.926 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:11:41.324446+0000", 2024-09-18T08:14:53.926 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 8.0020000000000006e-06 2024-09-18T08:14:53.926 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:14:53.926 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:14:53.926 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-09-18T08:14:53.926 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:11:41.324453+0000", 2024-09-18T08:14:53.927 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 6.4910000000000002e-06 2024-09-18T08:14:53.927 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:14:53.927 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:14:53.927 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-18T08:14:53.927 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:11:41.324565+0000", 2024-09-18T08:14:53.927 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000111983 2024-09-18T08:14:53.927 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:14:53.927 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:14:53.927 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "waiting for peered", 2024-09-18T08:14:53.927 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:11:41.324572+0000", 2024-09-18T08:14:53.927 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 7.8080000000000005e-06 2024-09-18T08:14:53.927 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:14:53.927 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:14:53.927 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-18T08:14:53.928 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:11:42.335472+0000", 2024-09-18T08:14:53.928 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.010899996 2024-09-18T08:14:53.928 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:14:53.928 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:14:53.928 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "waiting for missing object", 2024-09-18T08:14:53.928 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:11:42.335604+0000", 2024-09-18T08:14:53.928 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00013134999999999999 2024-09-18T08:14:53.928 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:14:53.928 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:14:53.928 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-18T08:14:53.928 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:11:42.346593+0000", 2024-09-18T08:14:53.928 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.01098872 2024-09-18T08:14:53.928 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:14:53.929 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:14:53.929 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-09-18T08:14:53.929 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:11:42.346682+0000", 2024-09-18T08:14:53.929 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 8.9067000000000006e-05 2024-09-18T08:14:53.929 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-18T08:14:53.929 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-09-18T08:14:53.929 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-18T08:14:53.929 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:14:53.929 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:14:53.929 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.24356.0:2784 5.1f 5:f95f44c2:::notify.0:head [watch reconnect cookie 94700888679552 gen 6] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e292)", 2024-09-18T08:14:53.929 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-09-18T08:14:04.566002+0000", 2024-09-18T08:14:53.929 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 49.239594218000001, 2024-09-18T08:14:53.929 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.026328065, 2024-09-18T08:14:53.929 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-09-18T08:14:53.930 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-09-18T08:14:53.930 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "started", 2024-09-18T08:14:53.930 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2024-09-18T08:14:53.930 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.24356", 2024-09-18T08:14:53.930 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "172.21.15.7:0/219168083", 2024-09-18T08:14:53.930 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 2784 2024-09-18T08:14:53.930 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:14:53.930 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-09-18T08:14:53.930 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:14:53.930 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-09-18T08:14:53.930 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:14:04.566002+0000", 2024-09-18T08:14:53.930 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-18T08:14:53.930 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:14:53.930 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:14:53.931 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-09-18T08:14:53.931 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:14:04.566002+0000", 2024-09-18T08:14:53.931 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-18T08:14:53.931 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:14:53.931 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:14:53.931 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-09-18T08:14:53.931 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:14:04.566004+0000", 2024-09-18T08:14:53.931 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.9350000000000001e-06 2024-09-18T08:14:53.931 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:14:53.932 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:14:53.932 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-09-18T08:14:53.932 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:14:04.566009+0000", 2024-09-18T08:14:53.932 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 5.5589999999999996e-06 2024-09-18T08:14:53.932 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:14:53.932 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:14:53.932 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-09-18T08:14:53.932 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:14:04.566021+0000", 2024-09-18T08:14:53.932 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.1398e-05 2024-09-18T08:14:53.932 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:14:53.932 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:14:53.932 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-09-18T08:14:53.933 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:14:53.933 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:14:53.934 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5' 2024-09-18T08:14:53.934 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:14:04.566028+0000", 2024-09-18T08:14:53.934 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 7.8029999999999997e-06 2024-09-18T08:14:53.934 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:14:53.934 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:14:53.935 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-18T08:14:53.935 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:14:04.574060+0000", 2024-09-18T08:14:53.935 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0080314459999999994 2024-09-18T08:14:53.935 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:14:53.935 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:14:53.935 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "waiting for peered", 2024-09-18T08:14:53.935 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:14:04.574071+0000", 2024-09-18T08:14:53.935 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.0693000000000001e-05 2024-09-18T08:14:53.935 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:14:53.935 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:14:53.935 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-18T08:14:53.935 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:14:05.582227+0000", 2024-09-18T08:14:53.935 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.0081559760000001 2024-09-18T08:14:53.935 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:14:53.936 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:14:53.936 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "waiting for degraded object", 2024-09-18T08:14:53.936 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:14:05.582584+0000", 2024-09-18T08:14:53.936 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000357096 2024-09-18T08:14:53.936 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:14:53.936 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:14:53.936 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-18T08:14:53.936 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:14:05.585756+0000", 2024-09-18T08:14:53.936 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0031727320000000002 2024-09-18T08:14:53.936 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:14:53.936 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:14:53.936 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-09-18T08:14:53.936 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:14:05.585848+0000", 2024-09-18T08:14:53.937 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 9.1587000000000003e-05 2024-09-18T08:14:53.937 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:14:53.937 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:14:53.937 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-09-18T08:14:53.937 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:14:05.592330+0000", 2024-09-18T08:14:53.937 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00648184 2024-09-18T08:14:53.937 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-18T08:14:53.937 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-09-18T08:14:53.937 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-18T08:14:53.937 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:14:53.937 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:14:53.937 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.24356.0:2785 4.5 4:a60f9269:lc::lua_package_allowlist:head [watch reconnect cookie 94700889442560 gen 6] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e292)", 2024-09-18T08:14:53.937 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-09-18T08:14:04.566047+0000", 2024-09-18T08:14:53.938 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 49.239548982999999, 2024-09-18T08:14:53.938 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.027464521, 2024-09-18T08:14:53.938 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-09-18T08:14:53.938 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-09-18T08:14:53.938 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "started", 2024-09-18T08:14:53.938 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2024-09-18T08:14:53.938 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.24356", 2024-09-18T08:14:53.938 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "172.21.15.7:0/219168083", 2024-09-18T08:14:53.938 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 2785 2024-09-18T08:14:53.938 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:14:53.938 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-09-18T08:14:53.938 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:14:53.938 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-09-18T08:14:53.939 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:14:04.566047+0000", 2024-09-18T08:14:53.939 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-18T08:14:53.939 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:14:53.939 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:14:53.939 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-09-18T08:14:53.939 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:14:04.566047+0000", 2024-09-18T08:14:53.939 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-18T08:14:53.939 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:14:53.939 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:14:53.939 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-09-18T08:14:53.939 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:14:04.566048+0000", 2024-09-18T08:14:53.939 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 5.8500000000000001e-07 2024-09-18T08:14:53.939 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:14:53.939 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:14:53.940 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-09-18T08:14:53.940 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:14:04.566050+0000", 2024-09-18T08:14:53.940 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 2.8509999999999999e-06 2024-09-18T08:14:53.940 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:14:53.940 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:14:53.940 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-09-18T08:14:53.940 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:14:04.566057+0000", 2024-09-18T08:14:53.940 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 6.9369999999999998e-06 2024-09-18T08:14:53.940 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:14:53.940 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:14:53.940 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-09-18T08:14:53.940 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:14:04.566063+0000", 2024-09-18T08:14:53.940 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 5.6049999999999997e-06 2024-09-18T08:14:53.940 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:14:53.941 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:14:53.941 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-18T08:14:53.941 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:14:04.568597+0000", 2024-09-18T08:14:53.941 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0025344730000000002 2024-09-18T08:14:53.941 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:14:53.941 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:14:53.941 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "waiting for peered", 2024-09-18T08:14:53.941 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:14:04.568606+0000", 2024-09-18T08:14:53.941 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 8.0649999999999999e-06 2024-09-18T08:14:53.941 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:14:53.941 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:14:53.941 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-18T08:14:53.941 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:14:05.586207+0000", 2024-09-18T08:14:53.942 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.017601309 2024-09-18T08:14:53.942 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:14:53.942 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:14:53.942 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "waiting for degraded object", 2024-09-18T08:14:53.942 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:14:05.586551+0000", 2024-09-18T08:14:53.942 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00034398799999999999 2024-09-18T08:14:53.942 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:14:53.942 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:14:53.942 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-18T08:14:53.942 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:14:05.590223+0000", 2024-09-18T08:14:53.942 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0036724499999999998 2024-09-18T08:14:53.942 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:14:53.942 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:14:53.942 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-09-18T08:14:53.943 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:14:05.590327+0000", 2024-09-18T08:14:53.943 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000103261 2024-09-18T08:14:53.943 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:14:53.943 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:14:53.943 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-09-18T08:14:53.943 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:14:05.593512+0000", 2024-09-18T08:14:53.943 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0031849970000000002 2024-09-18T08:14:53.943 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-18T08:14:53.943 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-09-18T08:14:53.943 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-18T08:14:53.943 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:14:53.943 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:14:53.943 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.24356.0:2786 5.9 5:93e5b521:::notify.7:head [watch reconnect cookie 94700888687616 gen 5] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e292)", 2024-09-18T08:14:53.943 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-09-18T08:14:04.566075+0000", 2024-09-18T08:14:53.944 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 49.239521330999999, 2024-09-18T08:14:53.944 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 2.0238415459999999, 2024-09-18T08:14:53.944 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-09-18T08:14:53.944 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-09-18T08:14:53.944 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "started", 2024-09-18T08:14:53.944 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2024-09-18T08:14:53.944 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.24356", 2024-09-18T08:14:53.944 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "172.21.15.7:0/219168083", 2024-09-18T08:14:53.944 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 2786 2024-09-18T08:14:53.944 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:14:53.944 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-09-18T08:14:53.944 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:14:53.944 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-09-18T08:14:53.945 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:14:04.566075+0000", 2024-09-18T08:14:53.945 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-18T08:14:53.945 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:14:53.945 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:14:53.945 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-09-18T08:14:53.945 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:14:04.566075+0000", 2024-09-18T08:14:53.945 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-18T08:14:53.945 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:14:53.945 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:14:53.945 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-09-18T08:14:53.945 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:14:04.566075+0000", 2024-09-18T08:14:53.945 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4.6699999999999999e-07 2024-09-18T08:14:53.945 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:14:53.945 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:14:53.946 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-09-18T08:14:53.946 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:14:04.566077+0000", 2024-09-18T08:14:53.946 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 2.283e-06 2024-09-18T08:14:53.946 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:14:53.946 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:14:53.946 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-09-18T08:14:53.946 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:14:04.566083+0000", 2024-09-18T08:14:53.946 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 5.4739999999999996e-06 2024-09-18T08:14:53.946 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:14:53.946 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:14:53.946 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-09-18T08:14:53.946 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:14:04.566088+0000", 2024-09-18T08:14:53.946 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4.887e-06 2024-09-18T08:14:53.946 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:14:53.947 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:14:53.947 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-18T08:14:53.947 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:14:04.567450+0000", 2024-09-18T08:14:53.947 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.001362261 2024-09-18T08:14:53.947 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:14:53.947 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:14:53.947 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "waiting for peered", 2024-09-18T08:14:53.947 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:14:04.567459+0000", 2024-09-18T08:14:53.947 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 8.5110000000000003e-06 2024-09-18T08:14:53.947 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:14:53.947 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:14:53.947 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-18T08:14:53.947 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:14:05.568619+0000", 2024-09-18T08:14:53.947 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.001160869 2024-09-18T08:14:53.948 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:14:53.948 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:14:53.948 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "waiting for peered", 2024-09-18T08:14:53.948 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:14:05.568637+0000", 2024-09-18T08:14:53.948 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.7801000000000001e-05 2024-09-18T08:14:53.948 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:14:53.948 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:14:53.948 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-18T08:14:53.948 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:14:06.586669+0000", 2024-09-18T08:14:53.948 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.01803212 2024-09-18T08:14:53.948 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:14:53.948 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:14:53.948 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-09-18T08:14:53.949 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:14:06.586966+0000", 2024-09-18T08:14:53.949 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00029684900000000002 2024-09-18T08:14:53.949 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:14:53.949 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:14:53.949 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-09-18T08:14:53.949 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:14:06.589916+0000", 2024-09-18T08:14:53.949 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0029500239999999999 2024-09-18T08:14:53.949 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-18T08:14:53.949 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-09-18T08:14:53.949 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-18T08:14:53.949 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:14:53.949 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:14:53.949 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.24356.0:2788 5.9 5:93e5b521:::notify.7:head [watch reconnect cookie 94700888687616 gen 6] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e293)", 2024-09-18T08:14:53.949 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-09-18T08:14:05.569076+0000", 2024-09-18T08:14:53.950 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 48.236519792999999, 2024-09-18T08:14:53.950 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.021128322, 2024-09-18T08:14:53.950 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-09-18T08:14:53.950 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-09-18T08:14:53.950 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "started", 2024-09-18T08:14:53.950 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2024-09-18T08:14:53.950 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.24356", 2024-09-18T08:14:53.950 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "172.21.15.7:0/219168083", 2024-09-18T08:14:53.950 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 2788 2024-09-18T08:14:53.950 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:14:53.950 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-09-18T08:14:53.950 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:14:53.950 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-09-18T08:14:53.950 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:14:05.569076+0000", 2024-09-18T08:14:53.951 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-18T08:14:53.951 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:14:53.951 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:14:53.951 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-09-18T08:14:53.951 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:14:05.569076+0000", 2024-09-18T08:14:53.951 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-18T08:14:53.951 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:14:53.951 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:14:53.951 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-09-18T08:14:53.951 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:14:05.569078+0000", 2024-09-18T08:14:53.951 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 2.0250000000000001e-06 2024-09-18T08:14:53.951 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:14:53.951 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:14:53.951 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-09-18T08:14:53.952 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:14:05.569086+0000", 2024-09-18T08:14:53.952 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 7.9500000000000001e-06 2024-09-18T08:14:53.952 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:14:53.952 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:14:53.952 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-09-18T08:14:53.952 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:14:05.569099+0000", 2024-09-18T08:14:53.952 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.2755000000000001e-05 2024-09-18T08:14:53.952 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:14:53.952 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:14:53.952 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-09-18T08:14:53.952 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:14:05.569110+0000", 2024-09-18T08:14:53.952 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.1085e-05 2024-09-18T08:14:53.952 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:14:53.952 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:14:53.953 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-18T08:14:53.953 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:14:05.569176+0000", 2024-09-18T08:14:53.953 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 6.5499000000000004e-05 2024-09-18T08:14:53.953 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:14:53.953 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:14:53.953 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "waiting for peered", 2024-09-18T08:14:53.953 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:14:05.569187+0000", 2024-09-18T08:14:53.953 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.1537e-05 2024-09-18T08:14:53.953 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:14:53.953 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:14:53.953 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-18T08:14:53.953 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:14:06.588218+0000", 2024-09-18T08:14:53.953 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.0190308189999999 2024-09-18T08:14:53.953 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:14:53.954 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:14:53.954 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-09-18T08:14:53.954 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:14:06.588315+0000", 2024-09-18T08:14:53.954 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 9.7161e-05 2024-09-18T08:14:53.954 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:14:53.954 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:14:53.954 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-09-18T08:14:53.954 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:14:06.590205+0000", 2024-09-18T08:14:53.954 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0018894910000000001 2024-09-18T08:14:53.954 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-18T08:14:53.954 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-09-18T08:14:53.954 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-18T08:14:53.954 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-18T08:14:53.954 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-09-18T08:14:53.955 INFO:teuthology.orchestra.run.smithi050.stdout:} 2024-09-18T08:14:53.955 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:14:53 smithi050 ceph-mon[32371]: osdmap e317: 8 total, 8 up, 6 in 2024-09-18T08:14:53.955 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:14:53 smithi050 ceph-mon[32371]: from='mgr.24470 172.21.15.7:0/2594390261' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-18T08:14:53.955 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:14:53 smithi050 ceph-mon[32371]: 1.1 scrub starts 2024-09-18T08:14:53.955 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:14:53 smithi050 ceph-mon[32371]: from='mgr.24470 172.21.15.7:0/2594390261' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-18T08:14:53.955 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:14:53 smithi050 ceph-mon[32371]: from='mgr.24470 172.21.15.7:0/2594390261' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-18T08:14:53.955 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:14:53 smithi050 ceph-mon[32371]: from='mgr.24470 172.21.15.7:0/2594390261' entity='mgr.y' 2024-09-18T08:14:53.980 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:14:53 smithi007 ceph-mon[29459]: osdmap e317: 8 total, 8 up, 6 in 2024-09-18T08:14:53.980 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:14:53 smithi007 ceph-mon[29459]: from='mgr.24470 172.21.15.7:0/2594390261' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-18T08:14:53.980 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:14:53 smithi007 ceph-mon[29459]: 1.1 scrub starts 2024-09-18T08:14:53.980 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:14:53 smithi007 ceph-mon[29459]: from='mgr.24470 172.21.15.7:0/2594390261' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-18T08:14:53.980 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:14:53 smithi007 ceph-mon[29459]: from='mgr.24470 172.21.15.7:0/2594390261' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-18T08:14:53.981 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:14:53 smithi007 ceph-mon[29459]: from='mgr.24470 172.21.15.7:0/2594390261' entity='mgr.y' 2024-09-18T08:14:53.981 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:14:53 smithi007 ceph-mon[39249]: osdmap e317: 8 total, 8 up, 6 in 2024-09-18T08:14:53.981 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:14:53 smithi007 ceph-mon[39249]: from='mgr.24470 172.21.15.7:0/2594390261' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-18T08:14:53.981 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:14:53 smithi007 ceph-mon[39249]: 1.1 scrub starts 2024-09-18T08:14:53.981 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:14:53 smithi007 ceph-mon[39249]: from='mgr.24470 172.21.15.7:0/2594390261' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-18T08:14:53.981 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:14:53 smithi007 ceph-mon[39249]: from='mgr.24470 172.21.15.7:0/2594390261' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-18T08:14:53.981 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:14:53 smithi007 ceph-mon[39249]: from='mgr.24470 172.21.15.7:0/2594390261' entity='mgr.y' 2024-09-18T08:14:54.035 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3 2024-09-18T08:14:54.081 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:14:54.082 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:14:54.082 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3' 2024-09-18T08:14:54.183 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0 2024-09-18T08:14:54.225 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:14:54.226 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:14:54.226 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0' 2024-09-18T08:14:54.327 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5 2024-09-18T08:14:54.334 INFO:tasks.radosbench.radosbench.0.smithi007.stdout: 6 15 10495 10480 109.154 31.0938 0.00919662 0.00914792 2024-09-18T08:14:54.354 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:14:54.355 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:14:54.356 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5' 2024-09-18T08:14:54.357 INFO:teuthology.orchestra.run.smithi007.stderr:noscrub is unset 2024-09-18T08:14:54.456 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1 2024-09-18T08:14:54.497 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:14:54.499 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:14:54.499 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1' 2024-09-18T08:14:54.600 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6 2024-09-18T08:14:54.626 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:14:54.626 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:14:54.627 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6' 2024-09-18T08:14:54.728 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6 2024-09-18T08:14:54.755 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:14:54.756 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:14:54.757 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6' 2024-09-18T08:14:54.857 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4 2024-09-18T08:14:54.883 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:14:54 smithi050 ceph-mon[32371]: pgmap v595: 158 pgs: 14 peering, 144 active+clean; 186 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 1.7 KiB/s rd, 31 MiB/s wr, 495 op/s 2024-09-18T08:14:54.883 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:14:54 smithi050 ceph-mon[32371]: Health check failed: Reduced data availability: 1 pg peering (PG_AVAILABILITY) 2024-09-18T08:14:54.883 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:14:54 smithi050 ceph-mon[32371]: osdmap e318: 8 total, 8 up, 6 in 2024-09-18T08:14:54.883 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:14:54 smithi050 ceph-mon[32371]: 1.a scrub starts 2024-09-18T08:14:54.883 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:14:54 smithi050 ceph-mon[32371]: from='client.? 172.21.15.7:0/394089255' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-18T08:14:54.883 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:14:54 smithi050 ceph-mon[32371]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-18T08:14:54.883 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:14:54 smithi050 ceph-mon[32371]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-18T08:14:54.883 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:14:54 smithi050 ceph-mon[32371]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-18T08:14:54.883 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:14:54 smithi050 ceph-mon[32371]: osdmap e319: 8 total, 8 up, 6 in 2024-09-18T08:14:54.886 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:14:54.886 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:14:54.887 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4' 2024-09-18T08:14:54.920 DEBUG:teuthology.orchestra.run.smithi050:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid 779349ca-7593-11ef-bceb-c7b262605968 -- ceph daemon osd.5 dump_ops_in_flight 2024-09-18T08:14:54.980 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:14:54 smithi007 ceph-mon[29459]: pgmap v595: 158 pgs: 14 peering, 144 active+clean; 186 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 1.7 KiB/s rd, 31 MiB/s wr, 495 op/s 2024-09-18T08:14:54.981 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:14:54 smithi007 ceph-mon[29459]: Health check failed: Reduced data availability: 1 pg peering (PG_AVAILABILITY) 2024-09-18T08:14:54.981 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:14:54 smithi007 ceph-mon[29459]: osdmap e318: 8 total, 8 up, 6 in 2024-09-18T08:14:54.981 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:14:54 smithi007 ceph-mon[29459]: 1.a scrub starts 2024-09-18T08:14:54.981 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:14:54 smithi007 ceph-mon[29459]: from='client.? 172.21.15.7:0/394089255' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-18T08:14:54.981 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:14:54 smithi007 ceph-mon[29459]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-18T08:14:54.981 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:14:54 smithi007 ceph-mon[29459]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-18T08:14:54.981 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:14:54 smithi007 ceph-mon[29459]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-18T08:14:54.981 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:14:54 smithi007 ceph-mon[29459]: osdmap e319: 8 total, 8 up, 6 in 2024-09-18T08:14:54.981 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:14:54 smithi007 ceph-mon[39249]: pgmap v595: 158 pgs: 14 peering, 144 active+clean; 186 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 1.7 KiB/s rd, 31 MiB/s wr, 495 op/s 2024-09-18T08:14:54.981 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:14:54 smithi007 ceph-mon[39249]: Health check failed: Reduced data availability: 1 pg peering (PG_AVAILABILITY) 2024-09-18T08:14:54.982 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:14:54 smithi007 ceph-mon[39249]: osdmap e318: 8 total, 8 up, 6 in 2024-09-18T08:14:54.982 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:14:54 smithi007 ceph-mon[39249]: 1.a scrub starts 2024-09-18T08:14:54.982 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:14:54 smithi007 ceph-mon[39249]: from='client.? 172.21.15.7:0/394089255' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-18T08:14:54.982 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:14:54 smithi007 ceph-mon[39249]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-18T08:14:54.982 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:14:54 smithi007 ceph-mon[39249]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-18T08:14:54.982 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:14:54 smithi007 ceph-mon[39249]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-18T08:14:54.982 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:14:54 smithi007 ceph-mon[39249]: osdmap e319: 8 total, 8 up, 6 in 2024-09-18T08:14:54.987 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1 2024-09-18T08:14:55.009 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:14:55.009 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:14:55.010 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1' 2024-09-18T08:14:55.010 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid 779349ca-7593-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-18T08:14:55.111 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1 2024-09-18T08:14:55.133 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:14:55.134 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:14:55.134 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1' 2024-09-18T08:14:55.231 INFO:teuthology.orchestra.run.smithi050.stderr:Inferring config /var/lib/ceph/779349ca-7593-11ef-bceb-c7b262605968/mon.b/config 2024-09-18T08:14:55.235 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2 2024-09-18T08:14:55.261 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:14:55.262 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:14:55.263 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2' 2024-09-18T08:14:55.335 INFO:tasks.radosbench.radosbench.0.smithi007.stdout: 7 16 11783 11767 105.034 80.4375 0.00613806 0.00951038 2024-09-18T08:14:55.364 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5 2024-09-18T08:14:55.392 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:14:55.393 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:14:55.393 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5' 2024-09-18T08:14:55.494 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2 2024-09-18T08:14:55.542 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:14:55.542 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:14:55.542 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2' 2024-09-18T08:14:55.643 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6 2024-09-18T08:14:55.668 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:14:55.669 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:14:55.669 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6' 2024-09-18T08:14:55.770 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6 2024-09-18T08:14:55.800 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:14:55.800 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:14:55.801 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6' 2024-09-18T08:14:55.902 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2 2024-09-18T08:14:55.924 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:14:55.925 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:14:55.925 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2' 2024-09-18T08:14:55.980 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:14:55 smithi007 ceph-mon[29459]: 1.8 scrub starts 2024-09-18T08:14:55.980 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:14:55 smithi007 ceph-mon[29459]: 1.8 scrub ok 2024-09-18T08:14:55.980 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:14:55 smithi007 ceph-mon[39249]: 1.8 scrub starts 2024-09-18T08:14:55.980 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:14:55 smithi007 ceph-mon[39249]: 1.8 scrub ok 2024-09-18T08:14:56.026 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4 2024-09-18T08:14:56.054 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:14:56.055 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:14:56.055 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4' 2024-09-18T08:14:56.143 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:14:55 smithi050 ceph-mon[32371]: 1.8 scrub starts 2024-09-18T08:14:56.143 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:14:55 smithi050 ceph-mon[32371]: 1.8 scrub ok 2024-09-18T08:14:56.156 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1 2024-09-18T08:14:56.199 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:14:56.200 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:14:56.200 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1' 2024-09-18T08:14:56.301 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0 2024-09-18T08:14:56.335 INFO:tasks.radosbench.radosbench.0.smithi007.stdout: 8 16 13229 13213 103.201 90.375 0.00704572 0.00967282 2024-09-18T08:14:56.337 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:14:56.338 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:14:56.338 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0' 2024-09-18T08:14:56.439 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6 2024-09-18T08:14:56.461 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:14:56.462 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:14:56.463 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6' 2024-09-18T08:14:56.564 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7 2024-09-18T08:14:56.589 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:14:56.590 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:14:56.590 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7' 2024-09-18T08:14:56.691 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4 2024-09-18T08:14:56.719 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:14:56.719 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:14:56.720 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4' 2024-09-18T08:14:56.787 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/779349ca-7593-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T08:14:56.820 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1 2024-09-18T08:14:56.864 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:14:56.865 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:14:56.866 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1' 2024-09-18T08:14:56.933 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:14:56 smithi007 ceph-mon[29459]: purged_snaps scrub starts 2024-09-18T08:14:56.933 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:14:56 smithi007 ceph-mon[29459]: purged_snaps scrub ok 2024-09-18T08:14:56.933 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:14:56 smithi007 ceph-mon[29459]: pgmap v598: 158 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+degraded+remapped, 3 active+recovering+undersized+remapped, 14 peering, 138 active+clean; 611 MiB data, 3.7 GiB used, 533 GiB / 536 GiB avail; 16 KiB/s rd, 135 MiB/s wr, 2.17k op/s; 586/29886 objects degraded (1.961%); 2858/29886 objects misplaced (9.563%); 2.3 MiB/s, 40 objects/s recovering 2024-09-18T08:14:56.933 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:14:56 smithi007 ceph-mon[29459]: Health check failed: Degraded data redundancy: 586/29886 objects degraded (1.961%), 3 pgs degraded (PG_DEGRADED) 2024-09-18T08:14:56.933 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:14:56 smithi007 ceph-mon[29459]: osdmap e320: 8 total, 8 up, 6 in 2024-09-18T08:14:56.934 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Sep 18 08:14:56 smithi007 ceph-779349ca-7593-11ef-bceb-c7b262605968-mgr-y[29688]: ::ffff:172.21.15.50 - - [18/Sep/2024:08:14:56] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-09-18T08:14:56.934 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:14:56 smithi007 ceph-mon[39249]: purged_snaps scrub starts 2024-09-18T08:14:56.934 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:14:56 smithi007 ceph-mon[39249]: purged_snaps scrub ok 2024-09-18T08:14:56.934 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:14:56 smithi007 ceph-mon[39249]: pgmap v598: 158 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+degraded+remapped, 3 active+recovering+undersized+remapped, 14 peering, 138 active+clean; 611 MiB data, 3.7 GiB used, 533 GiB / 536 GiB avail; 16 KiB/s rd, 135 MiB/s wr, 2.17k op/s; 586/29886 objects degraded (1.961%); 2858/29886 objects misplaced (9.563%); 2.3 MiB/s, 40 objects/s recovering 2024-09-18T08:14:56.934 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:14:56 smithi007 ceph-mon[39249]: Health check failed: Degraded data redundancy: 586/29886 objects degraded (1.961%), 3 pgs degraded (PG_DEGRADED) 2024-09-18T08:14:56.934 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:14:56 smithi007 ceph-mon[39249]: osdmap e320: 8 total, 8 up, 6 in 2024-09-18T08:14:56.967 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2 2024-09-18T08:14:56.996 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:14:56.997 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:14:56.997 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2' 2024-09-18T08:14:57.098 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0 2024-09-18T08:14:57.121 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:14:57.121 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:14:57.122 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0' 2024-09-18T08:14:57.138 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:14:56 smithi050 ceph-mon[32371]: purged_snaps scrub starts 2024-09-18T08:14:57.139 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:14:56 smithi050 ceph-mon[32371]: purged_snaps scrub ok 2024-09-18T08:14:57.139 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:14:56 smithi050 ceph-mon[32371]: pgmap v598: 158 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+degraded+remapped, 3 active+recovering+undersized+remapped, 14 peering, 138 active+clean; 611 MiB data, 3.7 GiB used, 533 GiB / 536 GiB avail; 16 KiB/s rd, 135 MiB/s wr, 2.17k op/s; 586/29886 objects degraded (1.961%); 2858/29886 objects misplaced (9.563%); 2.3 MiB/s, 40 objects/s recovering 2024-09-18T08:14:57.139 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:14:56 smithi050 ceph-mon[32371]: Health check failed: Degraded data redundancy: 586/29886 objects degraded (1.961%), 3 pgs degraded (PG_DEGRADED) 2024-09-18T08:14:57.139 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:14:56 smithi050 ceph-mon[32371]: osdmap e320: 8 total, 8 up, 6 in 2024-09-18T08:14:57.222 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2 2024-09-18T08:14:57.254 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:14:57.255 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:14:57.255 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2' 2024-09-18T08:14:57.335 INFO:tasks.radosbench.radosbench.0.smithi007.stdout: 9 16 14396 14380 99.8383 72.9375 0.0101963 0.0100072 2024-09-18T08:14:57.356 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1 2024-09-18T08:14:57.380 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 6, 4] out_osds: [7, 5] dead_osds: [] live_osds: [0, 2, 3, 4, 5, 6, 7, 1] 2024-09-18T08:14:57.380 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-18T08:14:57.380 INFO:tasks.thrashosds.thrasher:thrash_pg_upmap_items 2024-09-18T08:14:57.380 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid 779349ca-7593-11ef-bceb-c7b262605968 -- ceph osd dump -f json-pretty 2024-09-18T08:14:57.401 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:14:57.402 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:14:57.403 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1' 2024-09-18T08:14:57.504 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6 2024-09-18T08:14:57.532 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:14:57.533 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:14:57.533 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6' 2024-09-18T08:14:57.634 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4 2024-09-18T08:14:57.659 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:14:57.659 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:14:57.660 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4' 2024-09-18T08:14:57.730 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:14:57 smithi007 ceph-mon[29459]: purged_snaps scrub starts 2024-09-18T08:14:57.730 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:14:57 smithi007 ceph-mon[29459]: purged_snaps scrub ok 2024-09-18T08:14:57.732 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:14:57 smithi007 ceph-mon[39249]: purged_snaps scrub starts 2024-09-18T08:14:57.732 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:14:57 smithi007 ceph-mon[39249]: purged_snaps scrub ok 2024-09-18T08:14:57.760 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7 2024-09-18T08:14:57.789 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:14:57.790 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:14:57.790 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7' 2024-09-18T08:14:57.891 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7 2024-09-18T08:14:57.921 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:14:57.921 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:14:57.922 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7' 2024-09-18T08:14:58.023 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1 2024-09-18T08:14:58.069 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:14:58.069 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:14:58.070 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1' 2024-09-18T08:14:58.122 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:14:57 smithi050 ceph-mon[32371]: purged_snaps scrub starts 2024-09-18T08:14:58.122 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:14:57 smithi050 ceph-mon[32371]: purged_snaps scrub ok 2024-09-18T08:14:58.170 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0 2024-09-18T08:14:58.213 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:14:58.214 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:14:58.215 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0' 2024-09-18T08:14:58.262 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2024-09-18T08:14:58.262 INFO:teuthology.orchestra.run.smithi050.stdout: "ops": [], 2024-09-18T08:14:58.262 INFO:teuthology.orchestra.run.smithi050.stdout: "num_ops": 0 2024-09-18T08:14:58.262 INFO:teuthology.orchestra.run.smithi050.stdout:} 2024-09-18T08:14:58.315 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7 2024-09-18T08:14:58.335 INFO:tasks.radosbench.radosbench.0.smithi007.stdout: 10 16 15932 15916 99.4536 96 0.0143878 0.0100475 2024-09-18T08:14:58.346 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:14:58.347 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:14:58.347 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7' 2024-09-18T08:14:58.448 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3 2024-09-18T08:14:58.487 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:14:58.487 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:14:58.488 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3' 2024-09-18T08:14:58.588 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5 2024-09-18T08:14:58.616 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:14:58.617 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:14:58.617 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5' 2024-09-18T08:14:58.627 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid 779349ca-7593-11ef-bceb-c7b262605968 -- ceph osd unset nodeep-scrub 2024-09-18T08:14:58.718 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2 2024-09-18T08:14:58.761 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:14:58.762 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:14:58.763 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2' 2024-09-18T08:14:58.864 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5 2024-09-18T08:14:58.888 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:14:58.889 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:14:58.889 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5' 2024-09-18T08:14:58.905 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:14:58 smithi050 ceph-mon[32371]: pgmap v600: 158 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+degraded+remapped, 3 active+recovering+undersized+remapped, 14 peering, 138 active+clean; 611 MiB data, 3.7 GiB used, 533 GiB / 536 GiB avail; 13 KiB/s rd, 90 MiB/s wr, 1.46k op/s; 586/29886 objects degraded (1.961%); 2858/29886 objects misplaced (9.563%); 1.9 MiB/s, 36 objects/s recovering 2024-09-18T08:14:58.905 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:14:58 smithi050 ceph-mon[32371]: osdmap e321: 8 total, 8 up, 6 in 2024-09-18T08:14:58.980 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:14:58 smithi007 ceph-mon[29459]: pgmap v600: 158 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+degraded+remapped, 3 active+recovering+undersized+remapped, 14 peering, 138 active+clean; 611 MiB data, 3.7 GiB used, 533 GiB / 536 GiB avail; 13 KiB/s rd, 90 MiB/s wr, 1.46k op/s; 586/29886 objects degraded (1.961%); 2858/29886 objects misplaced (9.563%); 1.9 MiB/s, 36 objects/s recovering 2024-09-18T08:14:58.980 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:14:58 smithi007 ceph-mon[29459]: osdmap e321: 8 total, 8 up, 6 in 2024-09-18T08:14:58.981 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:14:58 smithi007 ceph-mon[39249]: pgmap v600: 158 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+degraded+remapped, 3 active+recovering+undersized+remapped, 14 peering, 138 active+clean; 611 MiB data, 3.7 GiB used, 533 GiB / 536 GiB avail; 13 KiB/s rd, 90 MiB/s wr, 1.46k op/s; 586/29886 objects degraded (1.961%); 2858/29886 objects misplaced (9.563%); 1.9 MiB/s, 36 objects/s recovering 2024-09-18T08:14:58.981 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:14:58 smithi007 ceph-mon[39249]: osdmap e321: 8 total, 8 up, 6 in 2024-09-18T08:14:58.990 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7 2024-09-18T08:14:59.021 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:14:59.022 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:14:59.022 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7' 2024-09-18T08:14:59.060 DEBUG:teuthology.orchestra.run.smithi050:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid 779349ca-7593-11ef-bceb-c7b262605968 -- ceph daemon osd.5 dump_blocked_ops 2024-09-18T08:14:59.123 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6 2024-09-18T08:14:59.171 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:14:59.172 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:14:59.172 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6' 2024-09-18T08:14:59.273 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6 2024-09-18T08:14:59.297 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:14:59.297 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:14:59.297 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6' 2024-09-18T08:14:59.336 INFO:tasks.radosbench.radosbench.0.smithi007.stdout: 11 16 16650 16634 94.4921 44.875 0.0024454 0.0100363 2024-09-18T08:14:59.362 INFO:teuthology.orchestra.run.smithi050.stderr:Inferring config /var/lib/ceph/779349ca-7593-11ef-bceb-c7b262605968/mon.b/config 2024-09-18T08:14:59.398 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6 2024-09-18T08:14:59.422 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:14:59.423 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:14:59.423 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6' 2024-09-18T08:14:59.524 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0 2024-09-18T08:14:59.546 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:14:59.547 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:14:59.547 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0' 2024-09-18T08:14:59.648 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2 2024-09-18T08:14:59.673 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:14:59.674 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:14:59.674 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2' 2024-09-18T08:14:59.732 INFO:journalctl@ceph.iscsi.iscsi.a.smithi050.stdout:Sep 18 08:14:59 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-iscsi-iscsi-a[57782]: debug there is no tcmu-runner data available 2024-09-18T08:14:59.732 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:14:59 smithi050 ceph-mon[32371]: osdmap e322: 8 total, 8 up, 6 in 2024-09-18T08:14:59.732 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:14:59 smithi050 ceph-mon[32371]: 5.3 scrub starts 2024-09-18T08:14:59.732 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:14:59 smithi050 ceph-mon[32371]: 5.3 scrub ok 2024-09-18T08:14:59.732 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:14:59 smithi050 ceph-mon[32371]: osdmap e323: 8 total, 8 up, 6 in 2024-09-18T08:14:59.775 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4 2024-09-18T08:14:59.798 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:14:59.798 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:14:59.798 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4' 2024-09-18T08:14:59.899 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7 2024-09-18T08:14:59.922 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:14:59.922 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:14:59.923 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7' 2024-09-18T08:14:59.971 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2024-09-18T08:14:59.971 INFO:teuthology.orchestra.run.smithi050.stdout: "ops": [], 2024-09-18T08:14:59.971 INFO:teuthology.orchestra.run.smithi050.stdout: "complaint_time": 30, 2024-09-18T08:14:59.971 INFO:teuthology.orchestra.run.smithi050.stdout: "num_blocked_ops": 0 2024-09-18T08:14:59.971 INFO:teuthology.orchestra.run.smithi050.stdout:} 2024-09-18T08:14:59.980 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:14:59 smithi007 ceph-mon[29459]: osdmap e322: 8 total, 8 up, 6 in 2024-09-18T08:14:59.980 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:14:59 smithi007 ceph-mon[29459]: 5.3 scrub starts 2024-09-18T08:14:59.980 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:14:59 smithi007 ceph-mon[29459]: 5.3 scrub ok 2024-09-18T08:14:59.980 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:14:59 smithi007 ceph-mon[29459]: osdmap e323: 8 total, 8 up, 6 in 2024-09-18T08:14:59.980 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:14:59 smithi007 ceph-mon[39249]: osdmap e322: 8 total, 8 up, 6 in 2024-09-18T08:14:59.980 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:14:59 smithi007 ceph-mon[39249]: 5.3 scrub starts 2024-09-18T08:14:59.980 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:14:59 smithi007 ceph-mon[39249]: 5.3 scrub ok 2024-09-18T08:14:59.980 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:14:59 smithi007 ceph-mon[39249]: osdmap e323: 8 total, 8 up, 6 in 2024-09-18T08:15:00.024 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0 2024-09-18T08:15:00.046 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:15:00.048 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:15:00.048 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0' 2024-09-18T08:15:00.149 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3 2024-09-18T08:15:00.172 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:15:00.173 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:15:00.173 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3' 2024-09-18T08:15:00.274 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1 2024-09-18T08:15:00.297 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:15:00.297 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:15:00.298 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1' 2024-09-18T08:15:00.336 INFO:tasks.radosbench.radosbench.0.smithi007.stdout: 12 16 16737 16721 87.0717 5.4375 0.00185322 0.0105687 2024-09-18T08:15:00.398 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4 2024-09-18T08:15:00.428 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:15:00.428 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:15:00.429 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4' 2024-09-18T08:15:00.530 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3 2024-09-18T08:15:00.552 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:15:00.553 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:15:00.553 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3' 2024-09-18T08:15:00.654 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5 2024-09-18T08:15:00.677 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:15:00.678 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:15:00.678 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5' 2024-09-18T08:15:00.779 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1 2024-09-18T08:15:00.801 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:15:00.802 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:15:00.802 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1' 2024-09-18T08:15:00.903 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0 2024-09-18T08:15:00.905 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:15:00 smithi050 ceph-mon[32371]: pgmap v604: 158 pgs: 1 peering, 4 active+undersized+remapped, 1 active+recovering+undersized+remapped, 152 active+clean; 1005 MiB data, 4.9 GiB used, 531 GiB / 536 GiB avail; 99 MiB/s wr, 1.57k op/s; 565/48816 objects misplaced (1.157%); 45 MiB/s, 724 objects/s recovering 2024-09-18T08:15:00.905 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:15:00 smithi050 ceph-mon[32371]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-18T08:15:00.906 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:15:00 smithi050 ceph-mon[32371]: 6.d scrub starts 2024-09-18T08:15:00.906 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:15:00 smithi050 ceph-mon[32371]: 6.d scrub ok 2024-09-18T08:15:00.906 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:15:00 smithi050 ceph-mon[32371]: 6.10 scrub starts 2024-09-18T08:15:00.906 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:15:00 smithi050 ceph-mon[32371]: 6.10 scrub ok 2024-09-18T08:15:00.906 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:15:00 smithi050 ceph-mon[32371]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg peering) 2024-09-18T08:15:00.906 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:15:00 smithi050 ceph-mon[32371]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 586/29886 objects degraded (1.961%), 3 pgs degraded) 2024-09-18T08:15:00.906 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:15:00 smithi050 ceph-mon[32371]: osdmap e324: 8 total, 8 up, 6 in 2024-09-18T08:15:00.925 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:15:00.926 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:15:00.926 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0' 2024-09-18T08:15:00.980 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:15:00 smithi007 ceph-mon[29459]: pgmap v604: 158 pgs: 1 peering, 4 active+undersized+remapped, 1 active+recovering+undersized+remapped, 152 active+clean; 1005 MiB data, 4.9 GiB used, 531 GiB / 536 GiB avail; 99 MiB/s wr, 1.57k op/s; 565/48816 objects misplaced (1.157%); 45 MiB/s, 724 objects/s recovering 2024-09-18T08:15:00.980 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:15:00 smithi007 ceph-mon[29459]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-18T08:15:00.981 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:15:00 smithi007 ceph-mon[29459]: 6.d scrub starts 2024-09-18T08:15:00.981 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:15:00 smithi007 ceph-mon[29459]: 6.d scrub ok 2024-09-18T08:15:00.981 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:15:00 smithi007 ceph-mon[29459]: 6.10 scrub starts 2024-09-18T08:15:00.981 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:15:00 smithi007 ceph-mon[29459]: 6.10 scrub ok 2024-09-18T08:15:00.981 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:15:00 smithi007 ceph-mon[29459]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg peering) 2024-09-18T08:15:00.981 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:15:00 smithi007 ceph-mon[29459]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 586/29886 objects degraded (1.961%), 3 pgs degraded) 2024-09-18T08:15:00.981 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:15:00 smithi007 ceph-mon[29459]: osdmap e324: 8 total, 8 up, 6 in 2024-09-18T08:15:00.981 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:15:00 smithi007 ceph-mon[39249]: pgmap v604: 158 pgs: 1 peering, 4 active+undersized+remapped, 1 active+recovering+undersized+remapped, 152 active+clean; 1005 MiB data, 4.9 GiB used, 531 GiB / 536 GiB avail; 99 MiB/s wr, 1.57k op/s; 565/48816 objects misplaced (1.157%); 45 MiB/s, 724 objects/s recovering 2024-09-18T08:15:00.981 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:15:00 smithi007 ceph-mon[39249]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-18T08:15:00.981 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:15:00 smithi007 ceph-mon[39249]: 6.d scrub starts 2024-09-18T08:15:00.981 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:15:00 smithi007 ceph-mon[39249]: 6.d scrub ok 2024-09-18T08:15:00.982 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:15:00 smithi007 ceph-mon[39249]: 6.10 scrub starts 2024-09-18T08:15:00.982 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:15:00 smithi007 ceph-mon[39249]: 6.10 scrub ok 2024-09-18T08:15:00.982 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:15:00 smithi007 ceph-mon[39249]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg peering) 2024-09-18T08:15:00.982 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:15:00 smithi007 ceph-mon[39249]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 586/29886 objects degraded (1.961%), 3 pgs degraded) 2024-09-18T08:15:00.982 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:15:00 smithi007 ceph-mon[39249]: osdmap e324: 8 total, 8 up, 6 in 2024-09-18T08:15:00.983 DEBUG:teuthology.orchestra.run.smithi050:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid 779349ca-7593-11ef-bceb-c7b262605968 -- ceph daemon osd.5 dump_historic_ops 2024-09-18T08:15:01.027 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4 2024-09-18T08:15:01.050 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:15:01.051 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:15:01.051 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4' 2024-09-18T08:15:01.152 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4 2024-09-18T08:15:01.175 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:15:01.176 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:15:01.176 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4' 2024-09-18T08:15:01.238 INFO:teuthology.orchestra.run.smithi050.stderr:Inferring config /var/lib/ceph/779349ca-7593-11ef-bceb-c7b262605968/mon.b/config 2024-09-18T08:15:01.277 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4 2024-09-18T08:15:01.304 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:15:01.304 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:15:01.304 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4' 2024-09-18T08:15:01.336 INFO:tasks.radosbench.radosbench.0.smithi007.stdout: 13 16 17057 17041 81.9127 20 0.00148642 0.0113624 2024-09-18T08:15:01.405 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7 2024-09-18T08:15:01.435 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:15:01.436 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:15:01.436 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7' 2024-09-18T08:15:01.537 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2 2024-09-18T08:15:01.562 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:15:01.563 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:15:01.563 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2' 2024-09-18T08:15:01.664 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4 2024-09-18T08:15:01.699 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:15:01.699 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:15:01.700 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4' 2024-09-18T08:15:01.801 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5 2024-09-18T08:15:01.819 INFO:teuthology.orchestra.run.smithi050.stdout:ERROR: (22) Invalid argument 2024-09-18T08:15:01.819 INFO:teuthology.orchestra.run.smithi050.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-18T08:15:01.830 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:15:01.831 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:15:01.832 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5' 2024-09-18T08:15:01.932 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1 2024-09-18T08:15:01.971 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:15:01.971 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:15:01.972 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1' 2024-09-18T08:15:01.980 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:15:01 smithi007 ceph-mon[29459]: 4.8 scrub starts 2024-09-18T08:15:01.980 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:15:01 smithi007 ceph-mon[29459]: 4.8 scrub ok 2024-09-18T08:15:01.980 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:15:01 smithi007 ceph-mon[29459]: 5.6 scrub starts 2024-09-18T08:15:01.980 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:15:01 smithi007 ceph-mon[29459]: 5.6 scrub ok 2024-09-18T08:15:01.980 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:15:01 smithi007 ceph-mon[29459]: from='client.? 172.21.15.7:0/434951464' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-18T08:15:01.981 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:15:01 smithi007 ceph-mon[29459]: osdmap e325: 8 total, 8 up, 6 in 2024-09-18T08:15:01.981 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:15:01 smithi007 ceph-mon[39249]: 4.8 scrub starts 2024-09-18T08:15:01.981 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:15:01 smithi007 ceph-mon[39249]: 4.8 scrub ok 2024-09-18T08:15:01.981 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:15:01 smithi007 ceph-mon[39249]: 5.6 scrub starts 2024-09-18T08:15:01.981 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:15:01 smithi007 ceph-mon[39249]: 5.6 scrub ok 2024-09-18T08:15:01.981 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:15:01 smithi007 ceph-mon[39249]: from='client.? 172.21.15.7:0/434951464' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-18T08:15:01.981 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:15:01 smithi007 ceph-mon[39249]: osdmap e325: 8 total, 8 up, 6 in 2024-09-18T08:15:02.073 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3 2024-09-18T08:15:02.111 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:15:02.111 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:15:02.112 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3' 2024-09-18T08:15:02.155 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:15:01 smithi050 ceph-mon[32371]: 4.8 scrub starts 2024-09-18T08:15:02.155 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:15:01 smithi050 ceph-mon[32371]: 4.8 scrub ok 2024-09-18T08:15:02.156 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:15:01 smithi050 ceph-mon[32371]: 5.6 scrub starts 2024-09-18T08:15:02.156 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:15:01 smithi050 ceph-mon[32371]: 5.6 scrub ok 2024-09-18T08:15:02.156 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:15:01 smithi050 ceph-mon[32371]: from='client.? 172.21.15.7:0/434951464' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-18T08:15:02.156 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:15:01 smithi050 ceph-mon[32371]: osdmap e325: 8 total, 8 up, 6 in 2024-09-18T08:15:02.212 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2 2024-09-18T08:15:02.254 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:15:02.255 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:15:02.255 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2' 2024-09-18T08:15:02.336 INFO:tasks.radosbench.radosbench.0.smithi007.stdout: 14 16 18951 18935 84.5162 118.375 0.00536245 0.0118254 2024-09-18T08:15:02.356 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4 2024-09-18T08:15:02.388 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:15:02.389 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:15:02.389 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4' 2024-09-18T08:15:02.490 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0 2024-09-18T08:15:02.534 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:15:02.534 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:15:02.535 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0' 2024-09-18T08:15:02.547 INFO:teuthology.orchestra.run.smithi007.stdout:osd.0: {} 2024-09-18T08:15:02.547 INFO:teuthology.orchestra.run.smithi007.stdout:osd.1: {} 2024-09-18T08:15:02.547 INFO:teuthology.orchestra.run.smithi007.stdout:osd.2: {} 2024-09-18T08:15:02.547 INFO:teuthology.orchestra.run.smithi007.stdout:osd.3: {} 2024-09-18T08:15:02.547 INFO:teuthology.orchestra.run.smithi007.stdout:osd.4: {} 2024-09-18T08:15:02.547 INFO:teuthology.orchestra.run.smithi007.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-18T08:15:02.547 INFO:teuthology.orchestra.run.smithi007.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-18T08:15:02.547 INFO:teuthology.orchestra.run.smithi007.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-18T08:15:02.547 INFO:teuthology.orchestra.run.smithi007.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-18T08:15:02.548 INFO:teuthology.orchestra.run.smithi007.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-18T08:15:02.548 INFO:teuthology.orchestra.run.smithi007.stdout:osd.5: {} 2024-09-18T08:15:02.548 INFO:teuthology.orchestra.run.smithi007.stdout:osd.6: {} 2024-09-18T08:15:02.548 INFO:teuthology.orchestra.run.smithi007.stdout:osd.7: {} 2024-09-18T08:15:02.548 INFO:teuthology.orchestra.run.smithi007.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-18T08:15:02.548 INFO:teuthology.orchestra.run.smithi007.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-18T08:15:02.548 INFO:teuthology.orchestra.run.smithi007.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-18T08:15:02.635 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6 2024-09-18T08:15:02.652 DEBUG:teuthology.orchestra.run.smithi050:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid 779349ca-7593-11ef-bceb-c7b262605968 -- ceph daemon osd.6 dump_ops_in_flight 2024-09-18T08:15:02.657 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:15:02.658 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:15:02.659 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6' 2024-09-18T08:15:02.760 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5 2024-09-18T08:15:02.783 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:15:02.784 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:15:02.784 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5' 2024-09-18T08:15:02.885 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5 2024-09-18T08:15:02.906 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:15:02 smithi050 ceph-mon[32371]: pgmap v606: 158 pgs: 1 peering, 4 active+undersized+remapped, 1 active+recovering+undersized+remapped, 152 active+clean; 1005 MiB data, 4.9 GiB used, 531 GiB / 536 GiB avail; 99 MiB/s wr, 1.57k op/s; 565/48816 objects misplaced (1.157%); 45 MiB/s, 724 objects/s recovering 2024-09-18T08:15:02.906 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:15:02 smithi050 ceph-mon[32371]: 3.1c scrub starts 2024-09-18T08:15:02.906 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:15:02 smithi050 ceph-mon[32371]: 3.1c scrub ok 2024-09-18T08:15:02.906 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:15:02 smithi050 ceph-mon[32371]: 5.7 scrub starts 2024-09-18T08:15:02.906 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:15:02 smithi050 ceph-mon[32371]: 5.7 scrub ok 2024-09-18T08:15:02.922 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:15:02.923 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:15:02.923 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5' 2024-09-18T08:15:02.947 INFO:teuthology.orchestra.run.smithi050.stderr:Inferring config /var/lib/ceph/779349ca-7593-11ef-bceb-c7b262605968/mon.b/config 2024-09-18T08:15:02.980 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:15:02 smithi007 ceph-mon[29459]: pgmap v606: 158 pgs: 1 peering, 4 active+undersized+remapped, 1 active+recovering+undersized+remapped, 152 active+clean; 1005 MiB data, 4.9 GiB used, 531 GiB / 536 GiB avail; 99 MiB/s wr, 1.57k op/s; 565/48816 objects misplaced (1.157%); 45 MiB/s, 724 objects/s recovering 2024-09-18T08:15:02.980 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:15:02 smithi007 ceph-mon[29459]: 3.1c scrub starts 2024-09-18T08:15:02.980 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:15:02 smithi007 ceph-mon[29459]: 3.1c scrub ok 2024-09-18T08:15:02.980 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:15:02 smithi007 ceph-mon[29459]: 5.7 scrub starts 2024-09-18T08:15:02.981 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:15:02 smithi007 ceph-mon[29459]: 5.7 scrub ok 2024-09-18T08:15:02.981 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:15:02 smithi007 ceph-mon[39249]: pgmap v606: 158 pgs: 1 peering, 4 active+undersized+remapped, 1 active+recovering+undersized+remapped, 152 active+clean; 1005 MiB data, 4.9 GiB used, 531 GiB / 536 GiB avail; 99 MiB/s wr, 1.57k op/s; 565/48816 objects misplaced (1.157%); 45 MiB/s, 724 objects/s recovering 2024-09-18T08:15:02.981 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:15:02 smithi007 ceph-mon[39249]: 3.1c scrub starts 2024-09-18T08:15:02.981 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:15:02 smithi007 ceph-mon[39249]: 3.1c scrub ok 2024-09-18T08:15:02.981 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:15:02 smithi007 ceph-mon[39249]: 5.7 scrub starts 2024-09-18T08:15:02.981 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:15:02 smithi007 ceph-mon[39249]: 5.7 scrub ok 2024-09-18T08:15:03.024 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0 2024-09-18T08:15:03.065 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:15:03.066 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:15:03.067 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0' 2024-09-18T08:15:03.168 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3 2024-09-18T08:15:03.206 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:15:03.207 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:15:03.207 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3' 2024-09-18T08:15:03.308 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1 2024-09-18T08:15:03.332 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:15:03.332 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:15:03.333 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1' 2024-09-18T08:15:03.336 INFO:tasks.radosbench.radosbench.0.smithi007.stdout: 15 16 20436 20420 85.0684 92.8125 0.00182275 0.0117193 2024-09-18T08:15:03.434 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7 2024-09-18T08:15:03.471 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:15:03.472 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:15:03.472 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7' 2024-09-18T08:15:03.573 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6 2024-09-18T08:15:03.577 INFO:teuthology.orchestra.run.smithi050.stdout:ERROR: (22) Invalid argument 2024-09-18T08:15:03.578 INFO:teuthology.orchestra.run.smithi050.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-18T08:15:03.598 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:15:03.599 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:15:03.600 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6' 2024-09-18T08:15:03.700 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7 2024-09-18T08:15:03.729 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:15:03.730 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:15:03.730 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7' 2024-09-18T08:15:03.831 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4 2024-09-18T08:15:03.856 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:15:03.857 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:15:03.858 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4' 2024-09-18T08:15:03.959 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1 2024-09-18T08:15:03.980 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:15:03 smithi007 ceph-mon[29459]: 6.c scrub starts 2024-09-18T08:15:03.980 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:15:03 smithi007 ceph-mon[29459]: 6.c scrub ok 2024-09-18T08:15:03.980 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:15:03 smithi007 ceph-mon[29459]: from='mgr.24470 172.21.15.7:0/2594390261' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-18T08:15:03.980 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:15:03 smithi007 ceph-mon[39249]: 6.c scrub starts 2024-09-18T08:15:03.980 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:15:03 smithi007 ceph-mon[39249]: 6.c scrub ok 2024-09-18T08:15:03.981 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:15:03 smithi007 ceph-mon[39249]: from='mgr.24470 172.21.15.7:0/2594390261' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-18T08:15:03.983 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:15:03.985 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:15:03.985 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1' 2024-09-18T08:15:04.086 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5 2024-09-18T08:15:04.115 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:15:04.116 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:15:04.116 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5' 2024-09-18T08:15:04.155 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:15:03 smithi050 ceph-mon[32371]: 6.c scrub starts 2024-09-18T08:15:04.155 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:15:03 smithi050 ceph-mon[32371]: 6.c scrub ok 2024-09-18T08:15:04.156 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:15:03 smithi050 ceph-mon[32371]: from='mgr.24470 172.21.15.7:0/2594390261' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-18T08:15:04.217 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2 2024-09-18T08:15:04.242 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:15:04.243 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:15:04.243 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2' 2024-09-18T08:15:04.336 INFO:tasks.radosbench.radosbench.0.smithi007.stdout: 16 16 21565 21549 84.161 70.5625 0.0015353 0.0118299 2024-09-18T08:15:04.345 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1 2024-09-18T08:15:04.383 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:15:04.385 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:15:04.386 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1' 2024-09-18T08:15:04.487 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0 2024-09-18T08:15:04.525 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:15:04.526 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:15:04.526 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0' 2024-09-18T08:15:04.627 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4 2024-09-18T08:15:04.658 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:15:04.659 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:15:04.659 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4' 2024-09-18T08:15:04.760 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2 2024-09-18T08:15:04.799 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:15:04.800 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:15:04.800 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2' 2024-09-18T08:15:04.829 DEBUG:teuthology.orchestra.run.smithi050:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid 779349ca-7593-11ef-bceb-c7b262605968 -- ceph daemon osd.6 dump_blocked_ops 2024-09-18T08:15:04.901 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4 2024-09-18T08:15:04.929 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:15:04.930 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:15:04.930 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4' 2024-09-18T08:15:04.980 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:15:04 smithi007 ceph-mon[39249]: pgmap v608: 158 pgs: 1 peering, 4 active+undersized+remapped, 1 active+recovering+undersized+remapped, 152 active+clean; 1.0 GiB data, 5.1 GiB used, 531 GiB / 536 GiB avail; 63 MiB/s wr, 1k op/s; 565/51492 objects misplaced (1.097%); 38 MiB/s, 615 objects/s recovering 2024-09-18T08:15:04.980 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:15:04 smithi007 ceph-mon[39249]: 4.d scrub starts 2024-09-18T08:15:04.981 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:15:04 smithi007 ceph-mon[39249]: 4.d scrub ok 2024-09-18T08:15:04.981 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:15:04 smithi007 ceph-mon[39249]: 5.5 scrub starts 2024-09-18T08:15:04.981 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:15:04 smithi007 ceph-mon[39249]: 5.5 scrub ok 2024-09-18T08:15:04.981 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:15:04 smithi007 ceph-mon[29459]: pgmap v608: 158 pgs: 1 peering, 4 active+undersized+remapped, 1 active+recovering+undersized+remapped, 152 active+clean; 1.0 GiB data, 5.1 GiB used, 531 GiB / 536 GiB avail; 63 MiB/s wr, 1k op/s; 565/51492 objects misplaced (1.097%); 38 MiB/s, 615 objects/s recovering 2024-09-18T08:15:04.981 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:15:04 smithi007 ceph-mon[29459]: 4.d scrub starts 2024-09-18T08:15:04.981 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:15:04 smithi007 ceph-mon[29459]: 4.d scrub ok 2024-09-18T08:15:04.981 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:15:04 smithi007 ceph-mon[29459]: 5.5 scrub starts 2024-09-18T08:15:04.981 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:15:04 smithi007 ceph-mon[29459]: 5.5 scrub ok 2024-09-18T08:15:05.031 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2 2024-09-18T08:15:05.080 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:15:05.081 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:15:05.081 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2' 2024-09-18T08:15:05.155 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:15:04 smithi050 ceph-mon[32371]: pgmap v608: 158 pgs: 1 peering, 4 active+undersized+remapped, 1 active+recovering+undersized+remapped, 152 active+clean; 1.0 GiB data, 5.1 GiB used, 531 GiB / 536 GiB avail; 63 MiB/s wr, 1k op/s; 565/51492 objects misplaced (1.097%); 38 MiB/s, 615 objects/s recovering 2024-09-18T08:15:05.155 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:15:04 smithi050 ceph-mon[32371]: 4.d scrub starts 2024-09-18T08:15:05.155 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:15:04 smithi050 ceph-mon[32371]: 4.d scrub ok 2024-09-18T08:15:05.156 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:15:04 smithi050 ceph-mon[32371]: 5.5 scrub starts 2024-09-18T08:15:05.156 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:15:04 smithi050 ceph-mon[32371]: 5.5 scrub ok 2024-09-18T08:15:05.179 INFO:teuthology.orchestra.run.smithi050.stderr:Inferring config /var/lib/ceph/779349ca-7593-11ef-bceb-c7b262605968/mon.b/config 2024-09-18T08:15:05.182 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4 2024-09-18T08:15:05.216 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:15:05.217 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:15:05.217 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4' 2024-09-18T08:15:05.318 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1 2024-09-18T08:15:05.337 INFO:tasks.radosbench.radosbench.0.smithi007.stdout: 17 16 23638 23622 86.8284 129.562 0.00520459 0.0115109 2024-09-18T08:15:05.356 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:15:05.357 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:15:05.357 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1' 2024-09-18T08:15:05.458 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6 2024-09-18T08:15:05.490 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:15:05.490 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:15:05.491 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6' 2024-09-18T08:15:05.592 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7 2024-09-18T08:15:05.616 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:15:05.617 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:15:05.617 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7' 2024-09-18T08:15:05.674 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/779349ca-7593-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T08:15:05.696 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/779349ca-7593-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T08:15:05.718 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5 2024-09-18T08:15:05.749 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:15:05.749 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:15:05.750 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5' 2024-09-18T08:15:05.761 INFO:teuthology.orchestra.run.smithi050.stdout:ERROR: (22) Invalid argument 2024-09-18T08:15:05.761 INFO:teuthology.orchestra.run.smithi050.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-18T08:15:05.843 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:15:05 smithi007 ceph-mon[29459]: 4.18 scrub starts 2024-09-18T08:15:05.843 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:15:05 smithi007 ceph-mon[29459]: 4.18 scrub ok 2024-09-18T08:15:05.844 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:15:05 smithi007 ceph-mon[29459]: pgmap v609: 158 pgs: 158 active+clean; 1.3 GiB data, 5.9 GiB used, 531 GiB / 536 GiB avail; 852 B/s rd, 48 MiB/s wr, 766 op/s; 5.9 MiB/s, 94 objects/s recovering 2024-09-18T08:15:05.844 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:15:05 smithi007 ceph-mon[39249]: 4.18 scrub starts 2024-09-18T08:15:05.845 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:15:05 smithi007 ceph-mon[39249]: 4.18 scrub ok 2024-09-18T08:15:05.845 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:15:05 smithi007 ceph-mon[39249]: pgmap v609: 158 pgs: 158 active+clean; 1.3 GiB data, 5.9 GiB used, 531 GiB / 536 GiB avail; 852 B/s rd, 48 MiB/s wr, 766 op/s; 5.9 MiB/s, 94 objects/s recovering 2024-09-18T08:15:05.851 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7 2024-09-18T08:15:05.880 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:15:05.880 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:15:05.881 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7' 2024-09-18T08:15:05.905 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:15:05 smithi050 ceph-mon[32371]: 4.18 scrub starts 2024-09-18T08:15:05.905 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:15:05 smithi050 ceph-mon[32371]: 4.18 scrub ok 2024-09-18T08:15:05.905 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:15:05 smithi050 ceph-mon[32371]: pgmap v609: 158 pgs: 158 active+clean; 1.3 GiB data, 5.9 GiB used, 531 GiB / 536 GiB avail; 852 B/s rd, 48 MiB/s wr, 766 op/s; 5.9 MiB/s, 94 objects/s recovering 2024-09-18T08:15:05.982 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2 2024-09-18T08:15:06.027 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:15:06.028 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:15:06.028 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2' 2024-09-18T08:15:06.129 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7 2024-09-18T08:15:06.159 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:15:06.159 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:15:06.160 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7' 2024-09-18T08:15:06.261 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5 2024-09-18T08:15:06.287 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:15:06.288 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:15:06.288 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5' 2024-09-18T08:15:06.337 INFO:tasks.radosbench.radosbench.0.smithi007.stdout: 18 16 25617 25601 88.8754 123.688 0.0026008 0.0112372 2024-09-18T08:15:06.389 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2 2024-09-18T08:15:06.430 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:15:06.432 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:15:06.432 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2' 2024-09-18T08:15:06.533 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5 2024-09-18T08:15:06.560 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:15:06.561 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:15:06.561 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5' 2024-09-18T08:15:06.662 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4 2024-09-18T08:15:06.689 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:15:06.690 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:15:06.690 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4' 2024-09-18T08:15:06.791 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1 2024-09-18T08:15:06.829 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:15:06.830 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:15:06.830 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1' 2024-09-18T08:15:06.931 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5 2024-09-18T08:15:06.958 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:15:06.959 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:15:06.959 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5' 2024-09-18T08:15:06.980 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:15:06 smithi007 ceph-mon[29459]: 6.1f scrub starts 2024-09-18T08:15:06.980 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:15:06 smithi007 ceph-mon[29459]: 6.1f scrub ok 2024-09-18T08:15:06.981 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Sep 18 08:15:06 smithi007 ceph-779349ca-7593-11ef-bceb-c7b262605968-mgr-y[29688]: ::ffff:172.21.15.50 - - [18/Sep/2024:08:15:06] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-09-18T08:15:06.981 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:15:06 smithi007 ceph-mon[39249]: 6.1f scrub starts 2024-09-18T08:15:06.981 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:15:06 smithi007 ceph-mon[39249]: 6.1f scrub ok 2024-09-18T08:15:06.986 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:15:06 smithi050 ceph-mon[32371]: 6.1f scrub starts 2024-09-18T08:15:06.986 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:15:06 smithi050 ceph-mon[32371]: 6.1f scrub ok 2024-09-18T08:15:07.025 DEBUG:teuthology.orchestra.run.smithi050:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid 779349ca-7593-11ef-bceb-c7b262605968 -- ceph daemon osd.6 dump_historic_ops 2024-09-18T08:15:07.060 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0 2024-09-18T08:15:07.099 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:15:07.100 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:15:07.100 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0' 2024-09-18T08:15:07.171 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-09-18T08:15:07.171 INFO:teuthology.orchestra.run.smithi007.stdout:{ 2024-09-18T08:15:07.172 INFO:teuthology.orchestra.run.smithi007.stdout: "epoch": 325, 2024-09-18T08:15:07.172 INFO:teuthology.orchestra.run.smithi007.stdout: "fsid": "779349ca-7593-11ef-bceb-c7b262605968", 2024-09-18T08:15:07.172 INFO:teuthology.orchestra.run.smithi007.stdout: "created": "2024-09-18T07:57:09.295710+0000", 2024-09-18T08:15:07.172 INFO:teuthology.orchestra.run.smithi007.stdout: "modified": "2024-09-18T08:15:01.347778+0000", 2024-09-18T08:15:07.172 INFO:teuthology.orchestra.run.smithi007.stdout: "last_up_change": "2024-09-18T08:14:38.391465+0000", 2024-09-18T08:15:07.172 INFO:teuthology.orchestra.run.smithi007.stdout: "last_in_change": "2024-09-18T08:14:51.619626+0000", 2024-09-18T08:15:07.172 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": "nodeep-scrub,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit", 2024-09-18T08:15:07.172 INFO:teuthology.orchestra.run.smithi007.stdout: "flags_num": 5804032, 2024-09-18T08:15:07.172 INFO:teuthology.orchestra.run.smithi007.stdout: "flags_set": [ 2024-09-18T08:15:07.172 INFO:teuthology.orchestra.run.smithi007.stdout: "nodeep-scrub", 2024-09-18T08:15:07.172 INFO:teuthology.orchestra.run.smithi007.stdout: "pglog_hardlimit", 2024-09-18T08:15:07.172 INFO:teuthology.orchestra.run.smithi007.stdout: "purged_snapdirs", 2024-09-18T08:15:07.172 INFO:teuthology.orchestra.run.smithi007.stdout: "recovery_deletes", 2024-09-18T08:15:07.173 INFO:teuthology.orchestra.run.smithi007.stdout: "sortbitwise" 2024-09-18T08:15:07.173 INFO:teuthology.orchestra.run.smithi007.stdout: ], 2024-09-18T08:15:07.173 INFO:teuthology.orchestra.run.smithi007.stdout: "crush_version": 19, 2024-09-18T08:15:07.173 INFO:teuthology.orchestra.run.smithi007.stdout: "full_ratio": 0.94999998807907104, 2024-09-18T08:15:07.173 INFO:teuthology.orchestra.run.smithi007.stdout: "backfillfull_ratio": 0.89999997615814209, 2024-09-18T08:15:07.173 INFO:teuthology.orchestra.run.smithi007.stdout: "nearfull_ratio": 0.85000002384185791, 2024-09-18T08:15:07.173 INFO:teuthology.orchestra.run.smithi007.stdout: "cluster_snapshot": "", 2024-09-18T08:15:07.173 INFO:teuthology.orchestra.run.smithi007.stdout: "pool_max": 10, 2024-09-18T08:15:07.173 INFO:teuthology.orchestra.run.smithi007.stdout: "max_osd": 8, 2024-09-18T08:15:07.173 INFO:teuthology.orchestra.run.smithi007.stdout: "require_min_compat_client": "luminous", 2024-09-18T08:15:07.173 INFO:teuthology.orchestra.run.smithi007.stdout: "min_compat_client": "luminous", 2024-09-18T08:15:07.173 INFO:teuthology.orchestra.run.smithi007.stdout: "require_osd_release": "squid", 2024-09-18T08:15:07.173 INFO:teuthology.orchestra.run.smithi007.stdout: "allow_crimson": false, 2024-09-18T08:15:07.174 INFO:teuthology.orchestra.run.smithi007.stdout: "pools": [ 2024-09-18T08:15:07.174 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:15:07.174 INFO:teuthology.orchestra.run.smithi007.stdout: "pool": 1, 2024-09-18T08:15:07.174 INFO:teuthology.orchestra.run.smithi007.stdout: "pool_name": ".mgr", 2024-09-18T08:15:07.174 INFO:teuthology.orchestra.run.smithi007.stdout: "create_time": "2024-09-18T08:00:11.802627+0000", 2024-09-18T08:15:07.174 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 1, 2024-09-18T08:15:07.174 INFO:teuthology.orchestra.run.smithi007.stdout: "flags_names": "hashpspool", 2024-09-18T08:15:07.174 INFO:teuthology.orchestra.run.smithi007.stdout: "type": 1, 2024-09-18T08:15:07.174 INFO:teuthology.orchestra.run.smithi007.stdout: "size": 3, 2024-09-18T08:15:07.174 INFO:teuthology.orchestra.run.smithi007.stdout: "min_size": 2, 2024-09-18T08:15:07.175 INFO:teuthology.orchestra.run.smithi007.stdout: "crush_rule": 0, 2024-09-18T08:15:07.175 INFO:teuthology.orchestra.run.smithi007.stdout: "peering_crush_bucket_count": 0, 2024-09-18T08:15:07.175 INFO:teuthology.orchestra.run.smithi007.stdout: "peering_crush_bucket_target": 0, 2024-09-18T08:15:07.175 INFO:teuthology.orchestra.run.smithi007.stdout: "peering_crush_bucket_barrier": 0, 2024-09-18T08:15:07.175 INFO:teuthology.orchestra.run.smithi007.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-09-18T08:15:07.175 INFO:teuthology.orchestra.run.smithi007.stdout: "is_stretch_pool": false, 2024-09-18T08:15:07.175 INFO:teuthology.orchestra.run.smithi007.stdout: "object_hash": 2, 2024-09-18T08:15:07.175 INFO:teuthology.orchestra.run.smithi007.stdout: "pg_autoscale_mode": "off", 2024-09-18T08:15:07.175 INFO:teuthology.orchestra.run.smithi007.stdout: "pg_num": 11, 2024-09-18T08:15:07.175 INFO:teuthology.orchestra.run.smithi007.stdout: "pg_placement_num": 11, 2024-09-18T08:15:07.175 INFO:teuthology.orchestra.run.smithi007.stdout: "pg_placement_num_target": 11, 2024-09-18T08:15:07.175 INFO:teuthology.orchestra.run.smithi007.stdout: "pg_num_target": 11, 2024-09-18T08:15:07.175 INFO:teuthology.orchestra.run.smithi007.stdout: "pg_num_pending": 11, 2024-09-18T08:15:07.176 INFO:teuthology.orchestra.run.smithi007.stdout: "last_pg_merge_meta": { 2024-09-18T08:15:07.176 INFO:teuthology.orchestra.run.smithi007.stdout: "source_pgid": "0.0", 2024-09-18T08:15:07.176 INFO:teuthology.orchestra.run.smithi007.stdout: "ready_epoch": 0, 2024-09-18T08:15:07.176 INFO:teuthology.orchestra.run.smithi007.stdout: "last_epoch_started": 0, 2024-09-18T08:15:07.176 INFO:teuthology.orchestra.run.smithi007.stdout: "last_epoch_clean": 0, 2024-09-18T08:15:07.176 INFO:teuthology.orchestra.run.smithi007.stdout: "source_version": "0'0", 2024-09-18T08:15:07.176 INFO:teuthology.orchestra.run.smithi007.stdout: "target_version": "0'0" 2024-09-18T08:15:07.176 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:15:07.176 INFO:teuthology.orchestra.run.smithi007.stdout: "last_change": "285", 2024-09-18T08:15:07.176 INFO:teuthology.orchestra.run.smithi007.stdout: "last_force_op_resend": "0", 2024-09-18T08:15:07.176 INFO:teuthology.orchestra.run.smithi007.stdout: "last_force_op_resend_prenautilus": "0", 2024-09-18T08:15:07.177 INFO:teuthology.orchestra.run.smithi007.stdout: "last_force_op_resend_preluminous": "152", 2024-09-18T08:15:07.177 INFO:teuthology.orchestra.run.smithi007.stdout: "auid": 0, 2024-09-18T08:15:07.177 INFO:teuthology.orchestra.run.smithi007.stdout: "snap_mode": "selfmanaged", 2024-09-18T08:15:07.177 INFO:teuthology.orchestra.run.smithi007.stdout: "snap_seq": 0, 2024-09-18T08:15:07.177 INFO:teuthology.orchestra.run.smithi007.stdout: "snap_epoch": 0, 2024-09-18T08:15:07.177 INFO:teuthology.orchestra.run.smithi007.stdout: "pool_snaps": [], 2024-09-18T08:15:07.177 INFO:teuthology.orchestra.run.smithi007.stdout: "removed_snaps": "[]", 2024-09-18T08:15:07.177 INFO:teuthology.orchestra.run.smithi007.stdout: "quota_max_bytes": 0, 2024-09-18T08:15:07.177 INFO:teuthology.orchestra.run.smithi007.stdout: "quota_max_objects": 0, 2024-09-18T08:15:07.177 INFO:teuthology.orchestra.run.smithi007.stdout: "tiers": [], 2024-09-18T08:15:07.177 INFO:teuthology.orchestra.run.smithi007.stdout: "tier_of": -1, 2024-09-18T08:15:07.177 INFO:teuthology.orchestra.run.smithi007.stdout: "read_tier": -1, 2024-09-18T08:15:07.177 INFO:teuthology.orchestra.run.smithi007.stdout: "write_tier": -1, 2024-09-18T08:15:07.177 INFO:teuthology.orchestra.run.smithi007.stdout: "cache_mode": "none", 2024-09-18T08:15:07.178 INFO:teuthology.orchestra.run.smithi007.stdout: "target_max_bytes": 0, 2024-09-18T08:15:07.178 INFO:teuthology.orchestra.run.smithi007.stdout: "target_max_objects": 0, 2024-09-18T08:15:07.178 INFO:teuthology.orchestra.run.smithi007.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-09-18T08:15:07.178 INFO:teuthology.orchestra.run.smithi007.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-09-18T08:15:07.178 INFO:teuthology.orchestra.run.smithi007.stdout: "cache_target_full_ratio_micro": 800000, 2024-09-18T08:15:07.178 INFO:teuthology.orchestra.run.smithi007.stdout: "cache_min_flush_age": 0, 2024-09-18T08:15:07.178 INFO:teuthology.orchestra.run.smithi007.stdout: "cache_min_evict_age": 0, 2024-09-18T08:15:07.178 INFO:teuthology.orchestra.run.smithi007.stdout: "erasure_code_profile": "", 2024-09-18T08:15:07.178 INFO:teuthology.orchestra.run.smithi007.stdout: "hit_set_params": { 2024-09-18T08:15:07.178 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "none" 2024-09-18T08:15:07.178 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:15:07.178 INFO:teuthology.orchestra.run.smithi007.stdout: "hit_set_period": 0, 2024-09-18T08:15:07.178 INFO:teuthology.orchestra.run.smithi007.stdout: "hit_set_count": 0, 2024-09-18T08:15:07.179 INFO:teuthology.orchestra.run.smithi007.stdout: "use_gmt_hitset": true, 2024-09-18T08:15:07.179 INFO:teuthology.orchestra.run.smithi007.stdout: "min_read_recency_for_promote": 0, 2024-09-18T08:15:07.179 INFO:teuthology.orchestra.run.smithi007.stdout: "min_write_recency_for_promote": 0, 2024-09-18T08:15:07.179 INFO:teuthology.orchestra.run.smithi007.stdout: "hit_set_grade_decay_rate": 0, 2024-09-18T08:15:07.179 INFO:teuthology.orchestra.run.smithi007.stdout: "hit_set_search_last_n": 0, 2024-09-18T08:15:07.179 INFO:teuthology.orchestra.run.smithi007.stdout: "grade_table": [], 2024-09-18T08:15:07.179 INFO:teuthology.orchestra.run.smithi007.stdout: "stripe_width": 0, 2024-09-18T08:15:07.179 INFO:teuthology.orchestra.run.smithi007.stdout: "expected_num_objects": 0, 2024-09-18T08:15:07.179 INFO:teuthology.orchestra.run.smithi007.stdout: "fast_read": false, 2024-09-18T08:15:07.179 INFO:teuthology.orchestra.run.smithi007.stdout: "options": { 2024-09-18T08:15:07.179 INFO:teuthology.orchestra.run.smithi007.stdout: "pg_num_max": 32, 2024-09-18T08:15:07.179 INFO:teuthology.orchestra.run.smithi007.stdout: "pg_num_min": 1 2024-09-18T08:15:07.179 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:15:07.180 INFO:teuthology.orchestra.run.smithi007.stdout: "application_metadata": { 2024-09-18T08:15:07.180 INFO:teuthology.orchestra.run.smithi007.stdout: "mgr": {} 2024-09-18T08:15:07.180 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:15:07.180 INFO:teuthology.orchestra.run.smithi007.stdout: "read_balance": { 2024-09-18T08:15:07.180 INFO:teuthology.orchestra.run.smithi007.stdout: "score_type": "Fair distribution", 2024-09-18T08:15:07.180 INFO:teuthology.orchestra.run.smithi007.stdout: "score_acting": 2.5299999713897705, 2024-09-18T08:15:07.180 INFO:teuthology.orchestra.run.smithi007.stdout: "score_stable": 2.5299999713897705, 2024-09-18T08:15:07.180 INFO:teuthology.orchestra.run.smithi007.stdout: "optimal_score": 0.86000001430511475, 2024-09-18T08:15:07.181 INFO:teuthology.orchestra.run.smithi007.stdout: "raw_score_acting": 2.1800000667572021, 2024-09-18T08:15:07.181 INFO:teuthology.orchestra.run.smithi007.stdout: "raw_score_stable": 2.1800000667572021, 2024-09-18T08:15:07.181 INFO:teuthology.orchestra.run.smithi007.stdout: "primary_affinity_weighted": 0.80927085876464844, 2024-09-18T08:15:07.181 INFO:teuthology.orchestra.run.smithi007.stdout: "average_primary_affinity": 0.92000001668930054, 2024-09-18T08:15:07.181 INFO:teuthology.orchestra.run.smithi007.stdout: "average_primary_affinity_weighted": 0.81000000238418579 2024-09-18T08:15:07.181 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-09-18T08:15:07.181 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:15:07.181 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:15:07.181 INFO:teuthology.orchestra.run.smithi007.stdout: "pool": 2, 2024-09-18T08:15:07.181 INFO:teuthology.orchestra.run.smithi007.stdout: "pool_name": "datapool", 2024-09-18T08:15:07.181 INFO:teuthology.orchestra.run.smithi007.stdout: "create_time": "2024-09-18T08:02:45.374099+0000", 2024-09-18T08:15:07.182 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 8193, 2024-09-18T08:15:07.182 INFO:teuthology.orchestra.run.smithi007.stdout: "flags_names": "hashpspool,selfmanaged_snaps", 2024-09-18T08:15:07.182 INFO:teuthology.orchestra.run.smithi007.stdout: "type": 1, 2024-09-18T08:15:07.182 INFO:teuthology.orchestra.run.smithi007.stdout: "size": 3, 2024-09-18T08:15:07.182 INFO:teuthology.orchestra.run.smithi007.stdout: "min_size": 2, 2024-09-18T08:15:07.182 INFO:teuthology.orchestra.run.smithi007.stdout: "crush_rule": 0, 2024-09-18T08:15:07.182 INFO:teuthology.orchestra.run.smithi007.stdout: "peering_crush_bucket_count": 0, 2024-09-18T08:15:07.182 INFO:teuthology.orchestra.run.smithi007.stdout: "peering_crush_bucket_target": 0, 2024-09-18T08:15:07.182 INFO:teuthology.orchestra.run.smithi007.stdout: "peering_crush_bucket_barrier": 0, 2024-09-18T08:15:07.183 INFO:teuthology.orchestra.run.smithi007.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-09-18T08:15:07.183 INFO:teuthology.orchestra.run.smithi007.stdout: "is_stretch_pool": false, 2024-09-18T08:15:07.183 INFO:teuthology.orchestra.run.smithi007.stdout: "object_hash": 2, 2024-09-18T08:15:07.183 INFO:teuthology.orchestra.run.smithi007.stdout: "pg_autoscale_mode": "off", 2024-09-18T08:15:07.183 INFO:teuthology.orchestra.run.smithi007.stdout: "pg_num": 3, 2024-09-18T08:15:07.183 INFO:teuthology.orchestra.run.smithi007.stdout: "pg_placement_num": 3, 2024-09-18T08:15:07.183 INFO:teuthology.orchestra.run.smithi007.stdout: "pg_placement_num_target": 3, 2024-09-18T08:15:07.183 INFO:teuthology.orchestra.run.smithi007.stdout: "pg_num_target": 3, 2024-09-18T08:15:07.183 INFO:teuthology.orchestra.run.smithi007.stdout: "pg_num_pending": 3, 2024-09-18T08:15:07.183 INFO:teuthology.orchestra.run.smithi007.stdout: "last_pg_merge_meta": { 2024-09-18T08:15:07.183 INFO:teuthology.orchestra.run.smithi007.stdout: "source_pgid": "0.0", 2024-09-18T08:15:07.183 INFO:teuthology.orchestra.run.smithi007.stdout: "ready_epoch": 0, 2024-09-18T08:15:07.183 INFO:teuthology.orchestra.run.smithi007.stdout: "last_epoch_started": 0, 2024-09-18T08:15:07.183 INFO:teuthology.orchestra.run.smithi007.stdout: "last_epoch_clean": 0, 2024-09-18T08:15:07.184 INFO:teuthology.orchestra.run.smithi007.stdout: "source_version": "0'0", 2024-09-18T08:15:07.184 INFO:teuthology.orchestra.run.smithi007.stdout: "target_version": "0'0" 2024-09-18T08:15:07.184 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:15:07.184 INFO:teuthology.orchestra.run.smithi007.stdout: "last_change": "64", 2024-09-18T08:15:07.184 INFO:teuthology.orchestra.run.smithi007.stdout: "last_force_op_resend": "0", 2024-09-18T08:15:07.184 INFO:teuthology.orchestra.run.smithi007.stdout: "last_force_op_resend_prenautilus": "0", 2024-09-18T08:15:07.184 INFO:teuthology.orchestra.run.smithi007.stdout: "last_force_op_resend_preluminous": "0", 2024-09-18T08:15:07.184 INFO:teuthology.orchestra.run.smithi007.stdout: "auid": 0, 2024-09-18T08:15:07.184 INFO:teuthology.orchestra.run.smithi007.stdout: "snap_mode": "selfmanaged", 2024-09-18T08:15:07.184 INFO:teuthology.orchestra.run.smithi007.stdout: "snap_seq": 2, 2024-09-18T08:15:07.184 INFO:teuthology.orchestra.run.smithi007.stdout: "snap_epoch": 64, 2024-09-18T08:15:07.184 INFO:teuthology.orchestra.run.smithi007.stdout: "pool_snaps": [], 2024-09-18T08:15:07.184 INFO:teuthology.orchestra.run.smithi007.stdout: "removed_snaps": "[]", 2024-09-18T08:15:07.185 INFO:teuthology.orchestra.run.smithi007.stdout: "quota_max_bytes": 0, 2024-09-18T08:15:07.185 INFO:teuthology.orchestra.run.smithi007.stdout: "quota_max_objects": 0, 2024-09-18T08:15:07.185 INFO:teuthology.orchestra.run.smithi007.stdout: "tiers": [], 2024-09-18T08:15:07.185 INFO:teuthology.orchestra.run.smithi007.stdout: "tier_of": -1, 2024-09-18T08:15:07.185 INFO:teuthology.orchestra.run.smithi007.stdout: "read_tier": -1, 2024-09-18T08:15:07.185 INFO:teuthology.orchestra.run.smithi007.stdout: "write_tier": -1, 2024-09-18T08:15:07.185 INFO:teuthology.orchestra.run.smithi007.stdout: "cache_mode": "none", 2024-09-18T08:15:07.185 INFO:teuthology.orchestra.run.smithi007.stdout: "target_max_bytes": 0, 2024-09-18T08:15:07.185 INFO:teuthology.orchestra.run.smithi007.stdout: "target_max_objects": 0, 2024-09-18T08:15:07.185 INFO:teuthology.orchestra.run.smithi007.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-09-18T08:15:07.186 INFO:teuthology.orchestra.run.smithi007.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-09-18T08:15:07.186 INFO:teuthology.orchestra.run.smithi007.stdout: "cache_target_full_ratio_micro": 800000, 2024-09-18T08:15:07.186 INFO:teuthology.orchestra.run.smithi007.stdout: "cache_min_flush_age": 0, 2024-09-18T08:15:07.186 INFO:teuthology.orchestra.run.smithi007.stdout: "cache_min_evict_age": 0, 2024-09-18T08:15:07.186 INFO:teuthology.orchestra.run.smithi007.stdout: "erasure_code_profile": "", 2024-09-18T08:15:07.186 INFO:teuthology.orchestra.run.smithi007.stdout: "hit_set_params": { 2024-09-18T08:15:07.186 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "none" 2024-09-18T08:15:07.186 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:15:07.186 INFO:teuthology.orchestra.run.smithi007.stdout: "hit_set_period": 0, 2024-09-18T08:15:07.186 INFO:teuthology.orchestra.run.smithi007.stdout: "hit_set_count": 0, 2024-09-18T08:15:07.186 INFO:teuthology.orchestra.run.smithi007.stdout: "use_gmt_hitset": true, 2024-09-18T08:15:07.186 INFO:teuthology.orchestra.run.smithi007.stdout: "min_read_recency_for_promote": 0, 2024-09-18T08:15:07.186 INFO:teuthology.orchestra.run.smithi007.stdout: "min_write_recency_for_promote": 0, 2024-09-18T08:15:07.187 INFO:teuthology.orchestra.run.smithi007.stdout: "hit_set_grade_decay_rate": 0, 2024-09-18T08:15:07.187 INFO:teuthology.orchestra.run.smithi007.stdout: "hit_set_search_last_n": 0, 2024-09-18T08:15:07.187 INFO:teuthology.orchestra.run.smithi007.stdout: "grade_table": [], 2024-09-18T08:15:07.187 INFO:teuthology.orchestra.run.smithi007.stdout: "stripe_width": 0, 2024-09-18T08:15:07.187 INFO:teuthology.orchestra.run.smithi007.stdout: "expected_num_objects": 0, 2024-09-18T08:15:07.187 INFO:teuthology.orchestra.run.smithi007.stdout: "fast_read": false, 2024-09-18T08:15:07.187 INFO:teuthology.orchestra.run.smithi007.stdout: "options": {}, 2024-09-18T08:15:07.187 INFO:teuthology.orchestra.run.smithi007.stdout: "application_metadata": { 2024-09-18T08:15:07.187 INFO:teuthology.orchestra.run.smithi007.stdout: "rbd": {} 2024-09-18T08:15:07.187 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:15:07.188 INFO:teuthology.orchestra.run.smithi007.stdout: "read_balance": { 2024-09-18T08:15:07.188 INFO:teuthology.orchestra.run.smithi007.stdout: "score_type": "Fair distribution", 2024-09-18T08:15:07.188 INFO:teuthology.orchestra.run.smithi007.stdout: "score_acting": 4.690000057220459, 2024-09-18T08:15:07.188 INFO:teuthology.orchestra.run.smithi007.stdout: "score_stable": 4.690000057220459, 2024-09-18T08:15:07.188 INFO:teuthology.orchestra.run.smithi007.stdout: "optimal_score": 0.70999997854232788, 2024-09-18T08:15:07.188 INFO:teuthology.orchestra.run.smithi007.stdout: "raw_score_acting": 3.3299999237060547, 2024-09-18T08:15:07.188 INFO:teuthology.orchestra.run.smithi007.stdout: "raw_score_stable": 3.3299999237060547, 2024-09-18T08:15:07.188 INFO:teuthology.orchestra.run.smithi007.stdout: "primary_affinity_weighted": 0.80927085876464844, 2024-09-18T08:15:07.188 INFO:teuthology.orchestra.run.smithi007.stdout: "average_primary_affinity": 0.92000001668930054, 2024-09-18T08:15:07.188 INFO:teuthology.orchestra.run.smithi007.stdout: "average_primary_affinity_weighted": 0.81000000238418579 2024-09-18T08:15:07.188 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-09-18T08:15:07.188 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:15:07.189 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:15:07.189 INFO:teuthology.orchestra.run.smithi007.stdout: "pool": 3, 2024-09-18T08:15:07.189 INFO:teuthology.orchestra.run.smithi007.stdout: "pool_name": ".rgw.root", 2024-09-18T08:15:07.189 INFO:teuthology.orchestra.run.smithi007.stdout: "create_time": "2024-09-18T08:02:46.319947+0000", 2024-09-18T08:15:07.189 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 1, 2024-09-18T08:15:07.189 INFO:teuthology.orchestra.run.smithi007.stdout: "flags_names": "hashpspool", 2024-09-18T08:15:07.189 INFO:teuthology.orchestra.run.smithi007.stdout: "type": 1, 2024-09-18T08:15:07.189 INFO:teuthology.orchestra.run.smithi007.stdout: "size": 3, 2024-09-18T08:15:07.189 INFO:teuthology.orchestra.run.smithi007.stdout: "min_size": 2, 2024-09-18T08:15:07.189 INFO:teuthology.orchestra.run.smithi007.stdout: "crush_rule": 0, 2024-09-18T08:15:07.189 INFO:teuthology.orchestra.run.smithi007.stdout: "peering_crush_bucket_count": 0, 2024-09-18T08:15:07.190 INFO:teuthology.orchestra.run.smithi007.stdout: "peering_crush_bucket_target": 0, 2024-09-18T08:15:07.190 INFO:teuthology.orchestra.run.smithi007.stdout: "peering_crush_bucket_barrier": 0, 2024-09-18T08:15:07.190 INFO:teuthology.orchestra.run.smithi007.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-09-18T08:15:07.190 INFO:teuthology.orchestra.run.smithi007.stdout: "is_stretch_pool": false, 2024-09-18T08:15:07.190 INFO:teuthology.orchestra.run.smithi007.stdout: "object_hash": 2, 2024-09-18T08:15:07.190 INFO:teuthology.orchestra.run.smithi007.stdout: "pg_autoscale_mode": "off", 2024-09-18T08:15:07.190 INFO:teuthology.orchestra.run.smithi007.stdout: "pg_num": 32, 2024-09-18T08:15:07.190 INFO:teuthology.orchestra.run.smithi007.stdout: "pg_placement_num": 32, 2024-09-18T08:15:07.190 INFO:teuthology.orchestra.run.smithi007.stdout: "pg_placement_num_target": 32, 2024-09-18T08:15:07.190 INFO:teuthology.orchestra.run.smithi007.stdout: "pg_num_target": 32, 2024-09-18T08:15:07.190 INFO:teuthology.orchestra.run.smithi007.stdout: "pg_num_pending": 32, 2024-09-18T08:15:07.190 INFO:teuthology.orchestra.run.smithi007.stdout: "last_pg_merge_meta": { 2024-09-18T08:15:07.190 INFO:teuthology.orchestra.run.smithi007.stdout: "source_pgid": "0.0", 2024-09-18T08:15:07.191 INFO:teuthology.orchestra.run.smithi007.stdout: "ready_epoch": 0, 2024-09-18T08:15:07.191 INFO:teuthology.orchestra.run.smithi007.stdout: "last_epoch_started": 0, 2024-09-18T08:15:07.191 INFO:teuthology.orchestra.run.smithi007.stdout: "last_epoch_clean": 0, 2024-09-18T08:15:07.191 INFO:teuthology.orchestra.run.smithi007.stdout: "source_version": "0'0", 2024-09-18T08:15:07.191 INFO:teuthology.orchestra.run.smithi007.stdout: "target_version": "0'0" 2024-09-18T08:15:07.191 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:15:07.191 INFO:teuthology.orchestra.run.smithi007.stdout: "last_change": "63", 2024-09-18T08:15:07.191 INFO:teuthology.orchestra.run.smithi007.stdout: "last_force_op_resend": "0", 2024-09-18T08:15:07.191 INFO:teuthology.orchestra.run.smithi007.stdout: "last_force_op_resend_prenautilus": "0", 2024-09-18T08:15:07.191 INFO:teuthology.orchestra.run.smithi007.stdout: "last_force_op_resend_preluminous": "0", 2024-09-18T08:15:07.191 INFO:teuthology.orchestra.run.smithi007.stdout: "auid": 0, 2024-09-18T08:15:07.191 INFO:teuthology.orchestra.run.smithi007.stdout: "snap_mode": "selfmanaged", 2024-09-18T08:15:07.191 INFO:teuthology.orchestra.run.smithi007.stdout: "snap_seq": 0, 2024-09-18T08:15:07.192 INFO:teuthology.orchestra.run.smithi007.stdout: "snap_epoch": 0, 2024-09-18T08:15:07.192 INFO:teuthology.orchestra.run.smithi007.stdout: "pool_snaps": [], 2024-09-18T08:15:07.192 INFO:teuthology.orchestra.run.smithi007.stdout: "removed_snaps": "[]", 2024-09-18T08:15:07.192 INFO:teuthology.orchestra.run.smithi007.stdout: "quota_max_bytes": 0, 2024-09-18T08:15:07.192 INFO:teuthology.orchestra.run.smithi007.stdout: "quota_max_objects": 0, 2024-09-18T08:15:07.192 INFO:teuthology.orchestra.run.smithi007.stdout: "tiers": [], 2024-09-18T08:15:07.192 INFO:teuthology.orchestra.run.smithi007.stdout: "tier_of": -1, 2024-09-18T08:15:07.192 INFO:teuthology.orchestra.run.smithi007.stdout: "read_tier": -1, 2024-09-18T08:15:07.192 INFO:teuthology.orchestra.run.smithi007.stdout: "write_tier": -1, 2024-09-18T08:15:07.192 INFO:teuthology.orchestra.run.smithi007.stdout: "cache_mode": "none", 2024-09-18T08:15:07.192 INFO:teuthology.orchestra.run.smithi007.stdout: "target_max_bytes": 0, 2024-09-18T08:15:07.192 INFO:teuthology.orchestra.run.smithi007.stdout: "target_max_objects": 0, 2024-09-18T08:15:07.193 INFO:teuthology.orchestra.run.smithi007.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-09-18T08:15:07.193 INFO:teuthology.orchestra.run.smithi007.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-09-18T08:15:07.193 INFO:teuthology.orchestra.run.smithi007.stdout: "cache_target_full_ratio_micro": 800000, 2024-09-18T08:15:07.193 INFO:teuthology.orchestra.run.smithi007.stdout: "cache_min_flush_age": 0, 2024-09-18T08:15:07.193 INFO:teuthology.orchestra.run.smithi007.stdout: "cache_min_evict_age": 0, 2024-09-18T08:15:07.193 INFO:teuthology.orchestra.run.smithi007.stdout: "erasure_code_profile": "", 2024-09-18T08:15:07.193 INFO:teuthology.orchestra.run.smithi007.stdout: "hit_set_params": { 2024-09-18T08:15:07.193 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "none" 2024-09-18T08:15:07.193 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:15:07.193 INFO:teuthology.orchestra.run.smithi007.stdout: "hit_set_period": 0, 2024-09-18T08:15:07.193 INFO:teuthology.orchestra.run.smithi007.stdout: "hit_set_count": 0, 2024-09-18T08:15:07.193 INFO:teuthology.orchestra.run.smithi007.stdout: "use_gmt_hitset": true, 2024-09-18T08:15:07.194 INFO:teuthology.orchestra.run.smithi007.stdout: "min_read_recency_for_promote": 0, 2024-09-18T08:15:07.194 INFO:teuthology.orchestra.run.smithi007.stdout: "min_write_recency_for_promote": 0, 2024-09-18T08:15:07.194 INFO:teuthology.orchestra.run.smithi007.stdout: "hit_set_grade_decay_rate": 0, 2024-09-18T08:15:07.194 INFO:teuthology.orchestra.run.smithi007.stdout: "hit_set_search_last_n": 0, 2024-09-18T08:15:07.194 INFO:teuthology.orchestra.run.smithi007.stdout: "grade_table": [], 2024-09-18T08:15:07.194 INFO:teuthology.orchestra.run.smithi007.stdout: "stripe_width": 0, 2024-09-18T08:15:07.194 INFO:teuthology.orchestra.run.smithi007.stdout: "expected_num_objects": 0, 2024-09-18T08:15:07.194 INFO:teuthology.orchestra.run.smithi007.stdout: "fast_read": false, 2024-09-18T08:15:07.194 INFO:teuthology.orchestra.run.smithi007.stdout: "options": {}, 2024-09-18T08:15:07.194 INFO:teuthology.orchestra.run.smithi007.stdout: "application_metadata": { 2024-09-18T08:15:07.194 INFO:teuthology.orchestra.run.smithi007.stdout: "rgw": {} 2024-09-18T08:15:07.194 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:15:07.194 INFO:teuthology.orchestra.run.smithi007.stdout: "read_balance": { 2024-09-18T08:15:07.194 INFO:teuthology.orchestra.run.smithi007.stdout: "score_type": "Fair distribution", 2024-09-18T08:15:07.195 INFO:teuthology.orchestra.run.smithi007.stdout: "score_acting": 2.4000000953674316, 2024-09-18T08:15:07.196 INFO:teuthology.orchestra.run.smithi007.stdout: "score_stable": 2.4000000953674316, 2024-09-18T08:15:07.196 INFO:teuthology.orchestra.run.smithi007.stdout: "optimal_score": 0.86000001430511475, 2024-09-18T08:15:07.196 INFO:teuthology.orchestra.run.smithi007.stdout: "raw_score_acting": 2.059999942779541, 2024-09-18T08:15:07.196 INFO:teuthology.orchestra.run.smithi007.stdout: "raw_score_stable": 2.059999942779541, 2024-09-18T08:15:07.197 INFO:teuthology.orchestra.run.smithi007.stdout: "primary_affinity_weighted": 0.80927085876464844, 2024-09-18T08:15:07.197 INFO:teuthology.orchestra.run.smithi007.stdout: "average_primary_affinity": 0.92000001668930054, 2024-09-18T08:15:07.197 INFO:teuthology.orchestra.run.smithi007.stdout: "average_primary_affinity_weighted": 0.81000000238418579 2024-09-18T08:15:07.197 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-09-18T08:15:07.197 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:15:07.197 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:15:07.197 INFO:teuthology.orchestra.run.smithi007.stdout: "pool": 4, 2024-09-18T08:15:07.197 INFO:teuthology.orchestra.run.smithi007.stdout: "pool_name": "default.rgw.log", 2024-09-18T08:15:07.197 INFO:teuthology.orchestra.run.smithi007.stdout: "create_time": "2024-09-18T08:02:47.948552+0000", 2024-09-18T08:15:07.197 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 1, 2024-09-18T08:15:07.197 INFO:teuthology.orchestra.run.smithi007.stdout: "flags_names": "hashpspool", 2024-09-18T08:15:07.197 INFO:teuthology.orchestra.run.smithi007.stdout: "type": 1, 2024-09-18T08:15:07.197 INFO:teuthology.orchestra.run.smithi007.stdout: "size": 3, 2024-09-18T08:15:07.198 INFO:teuthology.orchestra.run.smithi007.stdout: "min_size": 2, 2024-09-18T08:15:07.198 INFO:teuthology.orchestra.run.smithi007.stdout: "crush_rule": 0, 2024-09-18T08:15:07.198 INFO:teuthology.orchestra.run.smithi007.stdout: "peering_crush_bucket_count": 0, 2024-09-18T08:15:07.198 INFO:teuthology.orchestra.run.smithi007.stdout: "peering_crush_bucket_target": 0, 2024-09-18T08:15:07.198 INFO:teuthology.orchestra.run.smithi007.stdout: "peering_crush_bucket_barrier": 0, 2024-09-18T08:15:07.198 INFO:teuthology.orchestra.run.smithi007.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-09-18T08:15:07.198 INFO:teuthology.orchestra.run.smithi007.stdout: "is_stretch_pool": false, 2024-09-18T08:15:07.198 INFO:teuthology.orchestra.run.smithi007.stdout: "object_hash": 2, 2024-09-18T08:15:07.198 INFO:teuthology.orchestra.run.smithi007.stdout: "pg_autoscale_mode": "off", 2024-09-18T08:15:07.198 INFO:teuthology.orchestra.run.smithi007.stdout: "pg_num": 32, 2024-09-18T08:15:07.199 INFO:teuthology.orchestra.run.smithi007.stdout: "pg_placement_num": 32, 2024-09-18T08:15:07.199 INFO:teuthology.orchestra.run.smithi007.stdout: "pg_placement_num_target": 32, 2024-09-18T08:15:07.199 INFO:teuthology.orchestra.run.smithi007.stdout: "pg_num_target": 32, 2024-09-18T08:15:07.199 INFO:teuthology.orchestra.run.smithi007.stdout: "pg_num_pending": 32, 2024-09-18T08:15:07.199 INFO:teuthology.orchestra.run.smithi007.stdout: "last_pg_merge_meta": { 2024-09-18T08:15:07.199 INFO:teuthology.orchestra.run.smithi007.stdout: "source_pgid": "0.0", 2024-09-18T08:15:07.199 INFO:teuthology.orchestra.run.smithi007.stdout: "ready_epoch": 0, 2024-09-18T08:15:07.199 INFO:teuthology.orchestra.run.smithi007.stdout: "last_epoch_started": 0, 2024-09-18T08:15:07.199 INFO:teuthology.orchestra.run.smithi007.stdout: "last_epoch_clean": 0, 2024-09-18T08:15:07.199 INFO:teuthology.orchestra.run.smithi007.stdout: "source_version": "0'0", 2024-09-18T08:15:07.199 INFO:teuthology.orchestra.run.smithi007.stdout: "target_version": "0'0" 2024-09-18T08:15:07.199 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:15:07.199 INFO:teuthology.orchestra.run.smithi007.stdout: "last_change": "65", 2024-09-18T08:15:07.200 INFO:teuthology.orchestra.run.smithi007.stdout: "last_force_op_resend": "0", 2024-09-18T08:15:07.200 INFO:teuthology.orchestra.run.smithi007.stdout: "last_force_op_resend_prenautilus": "0", 2024-09-18T08:15:07.200 INFO:teuthology.orchestra.run.smithi007.stdout: "last_force_op_resend_preluminous": "0", 2024-09-18T08:15:07.200 INFO:teuthology.orchestra.run.smithi007.stdout: "auid": 0, 2024-09-18T08:15:07.200 INFO:teuthology.orchestra.run.smithi007.stdout: "snap_mode": "selfmanaged", 2024-09-18T08:15:07.200 INFO:teuthology.orchestra.run.smithi007.stdout: "snap_seq": 0, 2024-09-18T08:15:07.200 INFO:teuthology.orchestra.run.smithi007.stdout: "snap_epoch": 0, 2024-09-18T08:15:07.200 INFO:teuthology.orchestra.run.smithi007.stdout: "pool_snaps": [], 2024-09-18T08:15:07.200 INFO:teuthology.orchestra.run.smithi007.stdout: "removed_snaps": "[]", 2024-09-18T08:15:07.200 INFO:teuthology.orchestra.run.smithi007.stdout: "quota_max_bytes": 0, 2024-09-18T08:15:07.201 INFO:teuthology.orchestra.run.smithi007.stdout: "quota_max_objects": 0, 2024-09-18T08:15:07.201 INFO:teuthology.orchestra.run.smithi007.stdout: "tiers": [], 2024-09-18T08:15:07.201 INFO:teuthology.orchestra.run.smithi007.stdout: "tier_of": -1, 2024-09-18T08:15:07.201 INFO:teuthology.orchestra.run.smithi007.stdout: "read_tier": -1, 2024-09-18T08:15:07.201 INFO:teuthology.orchestra.run.smithi007.stdout: "write_tier": -1, 2024-09-18T08:15:07.201 INFO:teuthology.orchestra.run.smithi007.stdout: "cache_mode": "none", 2024-09-18T08:15:07.201 INFO:teuthology.orchestra.run.smithi007.stdout: "target_max_bytes": 0, 2024-09-18T08:15:07.201 INFO:teuthology.orchestra.run.smithi007.stdout: "target_max_objects": 0, 2024-09-18T08:15:07.201 INFO:teuthology.orchestra.run.smithi007.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-09-18T08:15:07.201 INFO:teuthology.orchestra.run.smithi007.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-09-18T08:15:07.201 INFO:teuthology.orchestra.run.smithi007.stdout: "cache_target_full_ratio_micro": 800000, 2024-09-18T08:15:07.201 INFO:teuthology.orchestra.run.smithi007.stdout: "cache_min_flush_age": 0, 2024-09-18T08:15:07.201 INFO:teuthology.orchestra.run.smithi007.stdout: "cache_min_evict_age": 0, 2024-09-18T08:15:07.202 INFO:teuthology.orchestra.run.smithi007.stdout: "erasure_code_profile": "", 2024-09-18T08:15:07.202 INFO:teuthology.orchestra.run.smithi007.stdout: "hit_set_params": { 2024-09-18T08:15:07.202 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "none" 2024-09-18T08:15:07.202 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:15:07.202 INFO:teuthology.orchestra.run.smithi007.stdout: "hit_set_period": 0, 2024-09-18T08:15:07.202 INFO:teuthology.orchestra.run.smithi007.stdout: "hit_set_count": 0, 2024-09-18T08:15:07.202 INFO:teuthology.orchestra.run.smithi007.stdout: "use_gmt_hitset": true, 2024-09-18T08:15:07.202 INFO:teuthology.orchestra.run.smithi007.stdout: "min_read_recency_for_promote": 0, 2024-09-18T08:15:07.202 INFO:teuthology.orchestra.run.smithi007.stdout: "min_write_recency_for_promote": 0, 2024-09-18T08:15:07.202 INFO:teuthology.orchestra.run.smithi007.stdout: "hit_set_grade_decay_rate": 0, 2024-09-18T08:15:07.202 INFO:teuthology.orchestra.run.smithi007.stdout: "hit_set_search_last_n": 0, 2024-09-18T08:15:07.202 INFO:teuthology.orchestra.run.smithi007.stdout: "grade_table": [], 2024-09-18T08:15:07.202 INFO:teuthology.orchestra.run.smithi007.stdout: "stripe_width": 0, 2024-09-18T08:15:07.203 INFO:teuthology.orchestra.run.smithi007.stdout: "expected_num_objects": 0, 2024-09-18T08:15:07.203 INFO:teuthology.orchestra.run.smithi007.stdout: "fast_read": false, 2024-09-18T08:15:07.203 INFO:teuthology.orchestra.run.smithi007.stdout: "options": {}, 2024-09-18T08:15:07.203 INFO:teuthology.orchestra.run.smithi007.stdout: "application_metadata": { 2024-09-18T08:15:07.203 INFO:teuthology.orchestra.run.smithi007.stdout: "rgw": {} 2024-09-18T08:15:07.203 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:15:07.203 INFO:teuthology.orchestra.run.smithi007.stdout: "read_balance": { 2024-09-18T08:15:07.203 INFO:teuthology.orchestra.run.smithi007.stdout: "score_type": "Fair distribution", 2024-09-18T08:15:07.203 INFO:teuthology.orchestra.run.smithi007.stdout: "score_acting": 2.4000000953674316, 2024-09-18T08:15:07.203 INFO:teuthology.orchestra.run.smithi007.stdout: "score_stable": 2.4000000953674316, 2024-09-18T08:15:07.203 INFO:teuthology.orchestra.run.smithi007.stdout: "optimal_score": 0.86000001430511475, 2024-09-18T08:15:07.203 INFO:teuthology.orchestra.run.smithi007.stdout: "raw_score_acting": 2.059999942779541, 2024-09-18T08:15:07.203 INFO:teuthology.orchestra.run.smithi007.stdout: "raw_score_stable": 2.059999942779541, 2024-09-18T08:15:07.203 INFO:teuthology.orchestra.run.smithi007.stdout: "primary_affinity_weighted": 0.80927085876464844, 2024-09-18T08:15:07.204 INFO:teuthology.orchestra.run.smithi007.stdout: "average_primary_affinity": 0.92000001668930054, 2024-09-18T08:15:07.204 INFO:teuthology.orchestra.run.smithi007.stdout: "average_primary_affinity_weighted": 0.81000000238418579 2024-09-18T08:15:07.204 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-09-18T08:15:07.204 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:15:07.204 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:15:07.204 INFO:teuthology.orchestra.run.smithi007.stdout: "pool": 5, 2024-09-18T08:15:07.204 INFO:teuthology.orchestra.run.smithi007.stdout: "pool_name": "default.rgw.control", 2024-09-18T08:15:07.204 INFO:teuthology.orchestra.run.smithi007.stdout: "create_time": "2024-09-18T08:02:49.904302+0000", 2024-09-18T08:15:07.204 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 1, 2024-09-18T08:15:07.204 INFO:teuthology.orchestra.run.smithi007.stdout: "flags_names": "hashpspool", 2024-09-18T08:15:07.204 INFO:teuthology.orchestra.run.smithi007.stdout: "type": 1, 2024-09-18T08:15:07.204 INFO:teuthology.orchestra.run.smithi007.stdout: "size": 3, 2024-09-18T08:15:07.204 INFO:teuthology.orchestra.run.smithi007.stdout: "min_size": 2, 2024-09-18T08:15:07.205 INFO:teuthology.orchestra.run.smithi007.stdout: "crush_rule": 0, 2024-09-18T08:15:07.205 INFO:teuthology.orchestra.run.smithi007.stdout: "peering_crush_bucket_count": 0, 2024-09-18T08:15:07.205 INFO:teuthology.orchestra.run.smithi007.stdout: "peering_crush_bucket_target": 0, 2024-09-18T08:15:07.205 INFO:teuthology.orchestra.run.smithi007.stdout: "peering_crush_bucket_barrier": 0, 2024-09-18T08:15:07.205 INFO:teuthology.orchestra.run.smithi007.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-09-18T08:15:07.205 INFO:teuthology.orchestra.run.smithi007.stdout: "is_stretch_pool": false, 2024-09-18T08:15:07.205 INFO:teuthology.orchestra.run.smithi007.stdout: "object_hash": 2, 2024-09-18T08:15:07.205 INFO:teuthology.orchestra.run.smithi007.stdout: "pg_autoscale_mode": "off", 2024-09-18T08:15:07.205 INFO:teuthology.orchestra.run.smithi007.stdout: "pg_num": 32, 2024-09-18T08:15:07.205 INFO:teuthology.orchestra.run.smithi007.stdout: "pg_placement_num": 32, 2024-09-18T08:15:07.205 INFO:teuthology.orchestra.run.smithi007.stdout: "pg_placement_num_target": 32, 2024-09-18T08:15:07.205 INFO:teuthology.orchestra.run.smithi007.stdout: "pg_num_target": 32, 2024-09-18T08:15:07.205 INFO:teuthology.orchestra.run.smithi007.stdout: "pg_num_pending": 32, 2024-09-18T08:15:07.206 INFO:teuthology.orchestra.run.smithi007.stdout: "last_pg_merge_meta": { 2024-09-18T08:15:07.206 INFO:teuthology.orchestra.run.smithi007.stdout: "source_pgid": "0.0", 2024-09-18T08:15:07.206 INFO:teuthology.orchestra.run.smithi007.stdout: "ready_epoch": 0, 2024-09-18T08:15:07.206 INFO:teuthology.orchestra.run.smithi007.stdout: "last_epoch_started": 0, 2024-09-18T08:15:07.206 INFO:teuthology.orchestra.run.smithi007.stdout: "last_epoch_clean": 0, 2024-09-18T08:15:07.206 INFO:teuthology.orchestra.run.smithi007.stdout: "source_version": "0'0", 2024-09-18T08:15:07.206 INFO:teuthology.orchestra.run.smithi007.stdout: "target_version": "0'0" 2024-09-18T08:15:07.206 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:15:07.206 INFO:teuthology.orchestra.run.smithi007.stdout: "last_change": "67", 2024-09-18T08:15:07.206 INFO:teuthology.orchestra.run.smithi007.stdout: "last_force_op_resend": "0", 2024-09-18T08:15:07.206 INFO:teuthology.orchestra.run.smithi007.stdout: "last_force_op_resend_prenautilus": "0", 2024-09-18T08:15:07.206 INFO:teuthology.orchestra.run.smithi007.stdout: "last_force_op_resend_preluminous": "0", 2024-09-18T08:15:07.207 INFO:teuthology.orchestra.run.smithi007.stdout: "auid": 0, 2024-09-18T08:15:07.207 INFO:teuthology.orchestra.run.smithi007.stdout: "snap_mode": "selfmanaged", 2024-09-18T08:15:07.207 INFO:teuthology.orchestra.run.smithi007.stdout: "snap_seq": 0, 2024-09-18T08:15:07.207 INFO:teuthology.orchestra.run.smithi007.stdout: "snap_epoch": 0, 2024-09-18T08:15:07.207 INFO:teuthology.orchestra.run.smithi007.stdout: "pool_snaps": [], 2024-09-18T08:15:07.207 INFO:teuthology.orchestra.run.smithi007.stdout: "removed_snaps": "[]", 2024-09-18T08:15:07.207 INFO:teuthology.orchestra.run.smithi007.stdout: "quota_max_bytes": 0, 2024-09-18T08:15:07.207 INFO:teuthology.orchestra.run.smithi007.stdout: "quota_max_objects": 0, 2024-09-18T08:15:07.207 INFO:teuthology.orchestra.run.smithi007.stdout: "tiers": [], 2024-09-18T08:15:07.207 INFO:teuthology.orchestra.run.smithi007.stdout: "tier_of": -1, 2024-09-18T08:15:07.207 INFO:teuthology.orchestra.run.smithi007.stdout: "read_tier": -1, 2024-09-18T08:15:07.208 INFO:teuthology.orchestra.run.smithi007.stdout: "write_tier": -1, 2024-09-18T08:15:07.208 INFO:teuthology.orchestra.run.smithi007.stdout: "cache_mode": "none", 2024-09-18T08:15:07.208 INFO:teuthology.orchestra.run.smithi007.stdout: "target_max_bytes": 0, 2024-09-18T08:15:07.208 INFO:teuthology.orchestra.run.smithi007.stdout: "target_max_objects": 0, 2024-09-18T08:15:07.208 INFO:teuthology.orchestra.run.smithi007.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-09-18T08:15:07.208 INFO:teuthology.orchestra.run.smithi007.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-09-18T08:15:07.208 INFO:teuthology.orchestra.run.smithi007.stdout: "cache_target_full_ratio_micro": 800000, 2024-09-18T08:15:07.208 INFO:teuthology.orchestra.run.smithi007.stdout: "cache_min_flush_age": 0, 2024-09-18T08:15:07.208 INFO:teuthology.orchestra.run.smithi007.stdout: "cache_min_evict_age": 0, 2024-09-18T08:15:07.208 INFO:teuthology.orchestra.run.smithi007.stdout: "erasure_code_profile": "", 2024-09-18T08:15:07.208 INFO:teuthology.orchestra.run.smithi007.stdout: "hit_set_params": { 2024-09-18T08:15:07.208 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "none" 2024-09-18T08:15:07.209 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:15:07.209 INFO:teuthology.orchestra.run.smithi007.stdout: "hit_set_period": 0, 2024-09-18T08:15:07.209 INFO:teuthology.orchestra.run.smithi007.stdout: "hit_set_count": 0, 2024-09-18T08:15:07.209 INFO:teuthology.orchestra.run.smithi007.stdout: "use_gmt_hitset": true, 2024-09-18T08:15:07.209 INFO:teuthology.orchestra.run.smithi007.stdout: "min_read_recency_for_promote": 0, 2024-09-18T08:15:07.209 INFO:teuthology.orchestra.run.smithi007.stdout: "min_write_recency_for_promote": 0, 2024-09-18T08:15:07.209 INFO:teuthology.orchestra.run.smithi007.stdout: "hit_set_grade_decay_rate": 0, 2024-09-18T08:15:07.209 INFO:teuthology.orchestra.run.smithi007.stdout: "hit_set_search_last_n": 0, 2024-09-18T08:15:07.209 INFO:teuthology.orchestra.run.smithi007.stdout: "grade_table": [], 2024-09-18T08:15:07.209 INFO:teuthology.orchestra.run.smithi007.stdout: "stripe_width": 0, 2024-09-18T08:15:07.209 INFO:teuthology.orchestra.run.smithi007.stdout: "expected_num_objects": 0, 2024-09-18T08:15:07.209 INFO:teuthology.orchestra.run.smithi007.stdout: "fast_read": false, 2024-09-18T08:15:07.209 INFO:teuthology.orchestra.run.smithi007.stdout: "options": {}, 2024-09-18T08:15:07.209 INFO:teuthology.orchestra.run.smithi007.stdout: "application_metadata": { 2024-09-18T08:15:07.210 INFO:teuthology.orchestra.run.smithi007.stdout: "rgw": {} 2024-09-18T08:15:07.210 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:15:07.210 INFO:teuthology.orchestra.run.smithi007.stdout: "read_balance": { 2024-09-18T08:15:07.210 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1 2024-09-18T08:15:07.211 INFO:teuthology.orchestra.run.smithi007.stdout: "score_type": "Fair distribution", 2024-09-18T08:15:07.211 INFO:teuthology.orchestra.run.smithi007.stdout: "score_acting": 2.190000057220459, 2024-09-18T08:15:07.211 INFO:teuthology.orchestra.run.smithi007.stdout: "score_stable": 2.190000057220459, 2024-09-18T08:15:07.211 INFO:teuthology.orchestra.run.smithi007.stdout: "optimal_score": 0.86000001430511475, 2024-09-18T08:15:07.211 INFO:teuthology.orchestra.run.smithi007.stdout: "raw_score_acting": 1.8799999952316284, 2024-09-18T08:15:07.211 INFO:teuthology.orchestra.run.smithi007.stdout: "raw_score_stable": 1.8799999952316284, 2024-09-18T08:15:07.211 INFO:teuthology.orchestra.run.smithi007.stdout: "primary_affinity_weighted": 0.80927085876464844, 2024-09-18T08:15:07.212 INFO:teuthology.orchestra.run.smithi007.stdout: "average_primary_affinity": 0.92000001668930054, 2024-09-18T08:15:07.212 INFO:teuthology.orchestra.run.smithi007.stdout: "average_primary_affinity_weighted": 0.81000000238418579 2024-09-18T08:15:07.212 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-09-18T08:15:07.212 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:15:07.212 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:15:07.212 INFO:teuthology.orchestra.run.smithi007.stdout: "pool": 6, 2024-09-18T08:15:07.212 INFO:teuthology.orchestra.run.smithi007.stdout: "pool_name": "default.rgw.meta", 2024-09-18T08:15:07.212 INFO:teuthology.orchestra.run.smithi007.stdout: "create_time": "2024-09-18T08:02:51.971465+0000", 2024-09-18T08:15:07.213 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 1, 2024-09-18T08:15:07.213 INFO:teuthology.orchestra.run.smithi007.stdout: "flags_names": "hashpspool", 2024-09-18T08:15:07.213 INFO:teuthology.orchestra.run.smithi007.stdout: "type": 1, 2024-09-18T08:15:07.213 INFO:teuthology.orchestra.run.smithi007.stdout: "size": 3, 2024-09-18T08:15:07.213 INFO:teuthology.orchestra.run.smithi007.stdout: "min_size": 2, 2024-09-18T08:15:07.213 INFO:teuthology.orchestra.run.smithi007.stdout: "crush_rule": 0, 2024-09-18T08:15:07.213 INFO:teuthology.orchestra.run.smithi007.stdout: "peering_crush_bucket_count": 0, 2024-09-18T08:15:07.213 INFO:teuthology.orchestra.run.smithi007.stdout: "peering_crush_bucket_target": 0, 2024-09-18T08:15:07.213 INFO:teuthology.orchestra.run.smithi007.stdout: "peering_crush_bucket_barrier": 0, 2024-09-18T08:15:07.213 INFO:teuthology.orchestra.run.smithi007.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-09-18T08:15:07.213 INFO:teuthology.orchestra.run.smithi007.stdout: "is_stretch_pool": false, 2024-09-18T08:15:07.213 INFO:teuthology.orchestra.run.smithi007.stdout: "object_hash": 2, 2024-09-18T08:15:07.213 INFO:teuthology.orchestra.run.smithi007.stdout: "pg_autoscale_mode": "off", 2024-09-18T08:15:07.214 INFO:teuthology.orchestra.run.smithi007.stdout: "pg_num": 32, 2024-09-18T08:15:07.214 INFO:teuthology.orchestra.run.smithi007.stdout: "pg_placement_num": 32, 2024-09-18T08:15:07.214 INFO:teuthology.orchestra.run.smithi007.stdout: "pg_placement_num_target": 32, 2024-09-18T08:15:07.214 INFO:teuthology.orchestra.run.smithi007.stdout: "pg_num_target": 32, 2024-09-18T08:15:07.214 INFO:teuthology.orchestra.run.smithi007.stdout: "pg_num_pending": 32, 2024-09-18T08:15:07.214 INFO:teuthology.orchestra.run.smithi007.stdout: "last_pg_merge_meta": { 2024-09-18T08:15:07.214 INFO:teuthology.orchestra.run.smithi007.stdout: "source_pgid": "0.0", 2024-09-18T08:15:07.214 INFO:teuthology.orchestra.run.smithi007.stdout: "ready_epoch": 0, 2024-09-18T08:15:07.214 INFO:teuthology.orchestra.run.smithi007.stdout: "last_epoch_started": 0, 2024-09-18T08:15:07.214 INFO:teuthology.orchestra.run.smithi007.stdout: "last_epoch_clean": 0, 2024-09-18T08:15:07.215 INFO:teuthology.orchestra.run.smithi007.stdout: "source_version": "0'0", 2024-09-18T08:15:07.215 INFO:teuthology.orchestra.run.smithi007.stdout: "target_version": "0'0" 2024-09-18T08:15:07.215 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:15:07.215 INFO:teuthology.orchestra.run.smithi007.stdout: "last_change": "70", 2024-09-18T08:15:07.215 INFO:teuthology.orchestra.run.smithi007.stdout: "last_force_op_resend": "0", 2024-09-18T08:15:07.215 INFO:teuthology.orchestra.run.smithi007.stdout: "last_force_op_resend_prenautilus": "0", 2024-09-18T08:15:07.215 INFO:teuthology.orchestra.run.smithi007.stdout: "last_force_op_resend_preluminous": "0", 2024-09-18T08:15:07.215 INFO:teuthology.orchestra.run.smithi007.stdout: "auid": 0, 2024-09-18T08:15:07.215 INFO:teuthology.orchestra.run.smithi007.stdout: "snap_mode": "selfmanaged", 2024-09-18T08:15:07.215 INFO:teuthology.orchestra.run.smithi007.stdout: "snap_seq": 0, 2024-09-18T08:15:07.215 INFO:teuthology.orchestra.run.smithi007.stdout: "snap_epoch": 0, 2024-09-18T08:15:07.215 INFO:teuthology.orchestra.run.smithi007.stdout: "pool_snaps": [], 2024-09-18T08:15:07.215 INFO:teuthology.orchestra.run.smithi007.stdout: "removed_snaps": "[]", 2024-09-18T08:15:07.215 INFO:teuthology.orchestra.run.smithi007.stdout: "quota_max_bytes": 0, 2024-09-18T08:15:07.216 INFO:teuthology.orchestra.run.smithi007.stdout: "quota_max_objects": 0, 2024-09-18T08:15:07.216 INFO:teuthology.orchestra.run.smithi007.stdout: "tiers": [], 2024-09-18T08:15:07.216 INFO:teuthology.orchestra.run.smithi007.stdout: "tier_of": -1, 2024-09-18T08:15:07.216 INFO:teuthology.orchestra.run.smithi007.stdout: "read_tier": -1, 2024-09-18T08:15:07.216 INFO:teuthology.orchestra.run.smithi007.stdout: "write_tier": -1, 2024-09-18T08:15:07.216 INFO:teuthology.orchestra.run.smithi007.stdout: "cache_mode": "none", 2024-09-18T08:15:07.216 INFO:teuthology.orchestra.run.smithi007.stdout: "target_max_bytes": 0, 2024-09-18T08:15:07.216 INFO:teuthology.orchestra.run.smithi007.stdout: "target_max_objects": 0, 2024-09-18T08:15:07.216 INFO:teuthology.orchestra.run.smithi007.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-09-18T08:15:07.216 INFO:teuthology.orchestra.run.smithi007.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-09-18T08:15:07.216 INFO:teuthology.orchestra.run.smithi007.stdout: "cache_target_full_ratio_micro": 800000, 2024-09-18T08:15:07.217 INFO:teuthology.orchestra.run.smithi007.stdout: "cache_min_flush_age": 0, 2024-09-18T08:15:07.217 INFO:teuthology.orchestra.run.smithi007.stdout: "cache_min_evict_age": 0, 2024-09-18T08:15:07.217 INFO:teuthology.orchestra.run.smithi007.stdout: "erasure_code_profile": "", 2024-09-18T08:15:07.217 INFO:teuthology.orchestra.run.smithi007.stdout: "hit_set_params": { 2024-09-18T08:15:07.217 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "none" 2024-09-18T08:15:07.217 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:15:07.217 INFO:teuthology.orchestra.run.smithi007.stdout: "hit_set_period": 0, 2024-09-18T08:15:07.217 INFO:teuthology.orchestra.run.smithi007.stdout: "hit_set_count": 0, 2024-09-18T08:15:07.217 INFO:teuthology.orchestra.run.smithi007.stdout: "use_gmt_hitset": true, 2024-09-18T08:15:07.218 INFO:teuthology.orchestra.run.smithi007.stdout: "min_read_recency_for_promote": 0, 2024-09-18T08:15:07.218 INFO:teuthology.orchestra.run.smithi007.stdout: "min_write_recency_for_promote": 0, 2024-09-18T08:15:07.218 INFO:teuthology.orchestra.run.smithi007.stdout: "hit_set_grade_decay_rate": 0, 2024-09-18T08:15:07.218 INFO:teuthology.orchestra.run.smithi007.stdout: "hit_set_search_last_n": 0, 2024-09-18T08:15:07.218 INFO:teuthology.orchestra.run.smithi007.stdout: "grade_table": [], 2024-09-18T08:15:07.218 INFO:teuthology.orchestra.run.smithi007.stdout: "stripe_width": 0, 2024-09-18T08:15:07.218 INFO:teuthology.orchestra.run.smithi007.stdout: "expected_num_objects": 0, 2024-09-18T08:15:07.218 INFO:teuthology.orchestra.run.smithi007.stdout: "fast_read": false, 2024-09-18T08:15:07.218 INFO:teuthology.orchestra.run.smithi007.stdout: "options": { 2024-09-18T08:15:07.218 INFO:teuthology.orchestra.run.smithi007.stdout: "pg_autoscale_bias": 4 2024-09-18T08:15:07.218 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:15:07.218 INFO:teuthology.orchestra.run.smithi007.stdout: "application_metadata": { 2024-09-18T08:15:07.219 INFO:teuthology.orchestra.run.smithi007.stdout: "rgw": {} 2024-09-18T08:15:07.219 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:15:07.219 INFO:teuthology.orchestra.run.smithi007.stdout: "read_balance": { 2024-09-18T08:15:07.219 INFO:teuthology.orchestra.run.smithi007.stdout: "score_type": "Fair distribution", 2024-09-18T08:15:07.219 INFO:teuthology.orchestra.run.smithi007.stdout: "score_acting": 1.9700000286102295, 2024-09-18T08:15:07.219 INFO:teuthology.orchestra.run.smithi007.stdout: "score_stable": 1.9700000286102295, 2024-09-18T08:15:07.219 INFO:teuthology.orchestra.run.smithi007.stdout: "optimal_score": 0.86000001430511475, 2024-09-18T08:15:07.219 INFO:teuthology.orchestra.run.smithi007.stdout: "raw_score_acting": 1.690000057220459, 2024-09-18T08:15:07.220 INFO:teuthology.orchestra.run.smithi007.stdout: "raw_score_stable": 1.690000057220459, 2024-09-18T08:15:07.220 INFO:teuthology.orchestra.run.smithi007.stdout: "primary_affinity_weighted": 0.80927085876464844, 2024-09-18T08:15:07.220 INFO:teuthology.orchestra.run.smithi007.stdout: "average_primary_affinity": 0.92000001668930054, 2024-09-18T08:15:07.220 INFO:teuthology.orchestra.run.smithi007.stdout: "average_primary_affinity_weighted": 0.81000000238418579 2024-09-18T08:15:07.220 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-09-18T08:15:07.220 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:15:07.220 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:15:07.220 INFO:teuthology.orchestra.run.smithi007.stdout: "pool": 10, 2024-09-18T08:15:07.220 INFO:teuthology.orchestra.run.smithi007.stdout: "pool_name": "unique_pool_3", 2024-09-18T08:15:07.220 INFO:teuthology.orchestra.run.smithi007.stdout: "create_time": "2024-09-18T08:14:42.583623+0000", 2024-09-18T08:15:07.220 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 1, 2024-09-18T08:15:07.221 INFO:teuthology.orchestra.run.smithi007.stdout: "flags_names": "hashpspool", 2024-09-18T08:15:07.221 INFO:teuthology.orchestra.run.smithi007.stdout: "type": 1, 2024-09-18T08:15:07.221 INFO:teuthology.orchestra.run.smithi007.stdout: "size": 3, 2024-09-18T08:15:07.221 INFO:teuthology.orchestra.run.smithi007.stdout: "min_size": 2, 2024-09-18T08:15:07.221 INFO:teuthology.orchestra.run.smithi007.stdout: "crush_rule": 0, 2024-09-18T08:15:07.221 INFO:teuthology.orchestra.run.smithi007.stdout: "peering_crush_bucket_count": 0, 2024-09-18T08:15:07.221 INFO:teuthology.orchestra.run.smithi007.stdout: "peering_crush_bucket_target": 0, 2024-09-18T08:15:07.221 INFO:teuthology.orchestra.run.smithi007.stdout: "peering_crush_bucket_barrier": 0, 2024-09-18T08:15:07.221 INFO:teuthology.orchestra.run.smithi007.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-09-18T08:15:07.221 INFO:teuthology.orchestra.run.smithi007.stdout: "is_stretch_pool": false, 2024-09-18T08:15:07.222 INFO:teuthology.orchestra.run.smithi007.stdout: "object_hash": 2, 2024-09-18T08:15:07.222 INFO:teuthology.orchestra.run.smithi007.stdout: "pg_autoscale_mode": "off", 2024-09-18T08:15:07.222 INFO:teuthology.orchestra.run.smithi007.stdout: "pg_num": 16, 2024-09-18T08:15:07.222 INFO:teuthology.orchestra.run.smithi007.stdout: "pg_placement_num": 16, 2024-09-18T08:15:07.222 INFO:teuthology.orchestra.run.smithi007.stdout: "pg_placement_num_target": 16, 2024-09-18T08:15:07.222 INFO:teuthology.orchestra.run.smithi007.stdout: "pg_num_target": 16, 2024-09-18T08:15:07.222 INFO:teuthology.orchestra.run.smithi007.stdout: "pg_num_pending": 16, 2024-09-18T08:15:07.222 INFO:teuthology.orchestra.run.smithi007.stdout: "last_pg_merge_meta": { 2024-09-18T08:15:07.222 INFO:teuthology.orchestra.run.smithi007.stdout: "source_pgid": "0.0", 2024-09-18T08:15:07.222 INFO:teuthology.orchestra.run.smithi007.stdout: "ready_epoch": 0, 2024-09-18T08:15:07.222 INFO:teuthology.orchestra.run.smithi007.stdout: "last_epoch_started": 0, 2024-09-18T08:15:07.223 INFO:teuthology.orchestra.run.smithi007.stdout: "last_epoch_clean": 0, 2024-09-18T08:15:07.223 INFO:teuthology.orchestra.run.smithi007.stdout: "source_version": "0'0", 2024-09-18T08:15:07.223 INFO:teuthology.orchestra.run.smithi007.stdout: "target_version": "0'0" 2024-09-18T08:15:07.223 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:15:07.223 INFO:teuthology.orchestra.run.smithi007.stdout: "last_change": "314", 2024-09-18T08:15:07.223 INFO:teuthology.orchestra.run.smithi007.stdout: "last_force_op_resend": "0", 2024-09-18T08:15:07.223 INFO:teuthology.orchestra.run.smithi007.stdout: "last_force_op_resend_prenautilus": "0", 2024-09-18T08:15:07.223 INFO:teuthology.orchestra.run.smithi007.stdout: "last_force_op_resend_preluminous": "0", 2024-09-18T08:15:07.223 INFO:teuthology.orchestra.run.smithi007.stdout: "auid": 0, 2024-09-18T08:15:07.223 INFO:teuthology.orchestra.run.smithi007.stdout: "snap_mode": "selfmanaged", 2024-09-18T08:15:07.223 INFO:teuthology.orchestra.run.smithi007.stdout: "snap_seq": 0, 2024-09-18T08:15:07.223 INFO:teuthology.orchestra.run.smithi007.stdout: "snap_epoch": 0, 2024-09-18T08:15:07.224 INFO:teuthology.orchestra.run.smithi007.stdout: "pool_snaps": [], 2024-09-18T08:15:07.224 INFO:teuthology.orchestra.run.smithi007.stdout: "removed_snaps": "[]", 2024-09-18T08:15:07.224 INFO:teuthology.orchestra.run.smithi007.stdout: "quota_max_bytes": 0, 2024-09-18T08:15:07.224 INFO:teuthology.orchestra.run.smithi007.stdout: "quota_max_objects": 0, 2024-09-18T08:15:07.224 INFO:teuthology.orchestra.run.smithi007.stdout: "tiers": [], 2024-09-18T08:15:07.224 INFO:teuthology.orchestra.run.smithi007.stdout: "tier_of": -1, 2024-09-18T08:15:07.224 INFO:teuthology.orchestra.run.smithi007.stdout: "read_tier": -1, 2024-09-18T08:15:07.224 INFO:teuthology.orchestra.run.smithi007.stdout: "write_tier": -1, 2024-09-18T08:15:07.224 INFO:teuthology.orchestra.run.smithi007.stdout: "cache_mode": "none", 2024-09-18T08:15:07.224 INFO:teuthology.orchestra.run.smithi007.stdout: "target_max_bytes": 0, 2024-09-18T08:15:07.224 INFO:teuthology.orchestra.run.smithi007.stdout: "target_max_objects": 0, 2024-09-18T08:15:07.224 INFO:teuthology.orchestra.run.smithi007.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-09-18T08:15:07.224 INFO:teuthology.orchestra.run.smithi007.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-09-18T08:15:07.224 INFO:teuthology.orchestra.run.smithi007.stdout: "cache_target_full_ratio_micro": 800000, 2024-09-18T08:15:07.225 INFO:teuthology.orchestra.run.smithi007.stdout: "cache_min_flush_age": 0, 2024-09-18T08:15:07.225 INFO:teuthology.orchestra.run.smithi007.stdout: "cache_min_evict_age": 0, 2024-09-18T08:15:07.225 INFO:teuthology.orchestra.run.smithi007.stdout: "erasure_code_profile": "", 2024-09-18T08:15:07.225 INFO:teuthology.orchestra.run.smithi007.stdout: "hit_set_params": { 2024-09-18T08:15:07.225 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "none" 2024-09-18T08:15:07.225 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:15:07.225 INFO:teuthology.orchestra.run.smithi007.stdout: "hit_set_period": 0, 2024-09-18T08:15:07.225 INFO:teuthology.orchestra.run.smithi007.stdout: "hit_set_count": 0, 2024-09-18T08:15:07.225 INFO:teuthology.orchestra.run.smithi007.stdout: "use_gmt_hitset": true, 2024-09-18T08:15:07.225 INFO:teuthology.orchestra.run.smithi007.stdout: "min_read_recency_for_promote": 0, 2024-09-18T08:15:07.225 INFO:teuthology.orchestra.run.smithi007.stdout: "min_write_recency_for_promote": 0, 2024-09-18T08:15:07.225 INFO:teuthology.orchestra.run.smithi007.stdout: "hit_set_grade_decay_rate": 0, 2024-09-18T08:15:07.225 INFO:teuthology.orchestra.run.smithi007.stdout: "hit_set_search_last_n": 0, 2024-09-18T08:15:07.225 INFO:teuthology.orchestra.run.smithi007.stdout: "grade_table": [], 2024-09-18T08:15:07.226 INFO:teuthology.orchestra.run.smithi007.stdout: "stripe_width": 0, 2024-09-18T08:15:07.226 INFO:teuthology.orchestra.run.smithi007.stdout: "expected_num_objects": 0, 2024-09-18T08:15:07.226 INFO:teuthology.orchestra.run.smithi007.stdout: "fast_read": false, 2024-09-18T08:15:07.226 INFO:teuthology.orchestra.run.smithi007.stdout: "options": {}, 2024-09-18T08:15:07.226 INFO:teuthology.orchestra.run.smithi007.stdout: "application_metadata": { 2024-09-18T08:15:07.226 INFO:teuthology.orchestra.run.smithi007.stdout: "rados": {} 2024-09-18T08:15:07.226 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:15:07.226 INFO:teuthology.orchestra.run.smithi007.stdout: "read_balance": { 2024-09-18T08:15:07.226 INFO:teuthology.orchestra.run.smithi007.stdout: "score_type": "Fair distribution", 2024-09-18T08:15:07.226 INFO:teuthology.orchestra.run.smithi007.stdout: "score_acting": 1.7400000095367432, 2024-09-18T08:15:07.226 INFO:teuthology.orchestra.run.smithi007.stdout: "score_stable": 1.7400000095367432, 2024-09-18T08:15:07.226 INFO:teuthology.orchestra.run.smithi007.stdout: "optimal_score": 0.86000001430511475, 2024-09-18T08:15:07.226 INFO:teuthology.orchestra.run.smithi007.stdout: "raw_score_acting": 1.5, 2024-09-18T08:15:07.227 INFO:teuthology.orchestra.run.smithi007.stdout: "raw_score_stable": 1.5, 2024-09-18T08:15:07.227 INFO:teuthology.orchestra.run.smithi007.stdout: "primary_affinity_weighted": 0.80927085876464844, 2024-09-18T08:15:07.227 INFO:teuthology.orchestra.run.smithi007.stdout: "average_primary_affinity": 0.92000001668930054, 2024-09-18T08:15:07.227 INFO:teuthology.orchestra.run.smithi007.stdout: "average_primary_affinity_weighted": 0.81000000238418579 2024-09-18T08:15:07.227 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-09-18T08:15:07.227 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-09-18T08:15:07.227 INFO:teuthology.orchestra.run.smithi007.stdout: ], 2024-09-18T08:15:07.227 INFO:teuthology.orchestra.run.smithi007.stdout: "osds": [ 2024-09-18T08:15:07.227 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:15:07.227 INFO:teuthology.orchestra.run.smithi007.stdout: "osd": 0, 2024-09-18T08:15:07.227 INFO:teuthology.orchestra.run.smithi007.stdout: "uuid": "ae8f60df-72ac-46fb-91a1-f132b851c876", 2024-09-18T08:15:07.227 INFO:teuthology.orchestra.run.smithi007.stdout: "up": 1, 2024-09-18T08:15:07.228 INFO:teuthology.orchestra.run.smithi007.stdout: "in": 1, 2024-09-18T08:15:07.228 INFO:teuthology.orchestra.run.smithi007.stdout: "weight": 1, 2024-09-18T08:15:07.228 INFO:teuthology.orchestra.run.smithi007.stdout: "primary_affinity": 1, 2024-09-18T08:15:07.228 INFO:teuthology.orchestra.run.smithi007.stdout: "last_clean_begin": 0, 2024-09-18T08:15:07.228 INFO:teuthology.orchestra.run.smithi007.stdout: "last_clean_end": 0, 2024-09-18T08:15:07.228 INFO:teuthology.orchestra.run.smithi007.stdout: "up_from": 8, 2024-09-18T08:15:07.228 INFO:teuthology.orchestra.run.smithi007.stdout: "up_thru": 323, 2024-09-18T08:15:07.228 INFO:teuthology.orchestra.run.smithi007.stdout: "down_at": 0, 2024-09-18T08:15:07.228 INFO:teuthology.orchestra.run.smithi007.stdout: "lost_at": 0, 2024-09-18T08:15:07.228 INFO:teuthology.orchestra.run.smithi007.stdout: "public_addrs": { 2024-09-18T08:15:07.228 INFO:teuthology.orchestra.run.smithi007.stdout: "addrvec": [ 2024-09-18T08:15:07.229 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:15:07.229 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "v2", 2024-09-18T08:15:07.229 INFO:teuthology.orchestra.run.smithi007.stdout: "addr": "172.21.15.7:6801", 2024-09-18T08:15:07.229 INFO:teuthology.orchestra.run.smithi007.stdout: "nonce": 2896425794 2024-09-18T08:15:07.229 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-09-18T08:15:07.229 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-09-18T08:15:07.229 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:15:07.229 INFO:teuthology.orchestra.run.smithi007.stdout: "cluster_addrs": { 2024-09-18T08:15:07.229 INFO:teuthology.orchestra.run.smithi007.stdout: "addrvec": [ 2024-09-18T08:15:07.229 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:15:07.229 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "v2", 2024-09-18T08:15:07.229 INFO:teuthology.orchestra.run.smithi007.stdout: "addr": "172.21.15.7:6802", 2024-09-18T08:15:07.229 INFO:teuthology.orchestra.run.smithi007.stdout: "nonce": 2896425794 2024-09-18T08:15:07.230 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-09-18T08:15:07.230 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-09-18T08:15:07.230 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:15:07.230 INFO:teuthology.orchestra.run.smithi007.stdout: "heartbeat_back_addrs": { 2024-09-18T08:15:07.230 INFO:teuthology.orchestra.run.smithi007.stdout: "addrvec": [ 2024-09-18T08:15:07.230 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:15:07.230 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "v2", 2024-09-18T08:15:07.230 INFO:teuthology.orchestra.run.smithi007.stdout: "addr": "172.21.15.7:6804", 2024-09-18T08:15:07.230 INFO:teuthology.orchestra.run.smithi007.stdout: "nonce": 2896425794 2024-09-18T08:15:07.230 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-09-18T08:15:07.230 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-09-18T08:15:07.230 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:15:07.230 INFO:teuthology.orchestra.run.smithi007.stdout: "heartbeat_front_addrs": { 2024-09-18T08:15:07.230 INFO:teuthology.orchestra.run.smithi007.stdout: "addrvec": [ 2024-09-18T08:15:07.231 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:15:07.231 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "v2", 2024-09-18T08:15:07.231 INFO:teuthology.orchestra.run.smithi007.stdout: "addr": "172.21.15.7:6803", 2024-09-18T08:15:07.231 INFO:teuthology.orchestra.run.smithi007.stdout: "nonce": 2896425794 2024-09-18T08:15:07.231 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-09-18T08:15:07.231 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-09-18T08:15:07.231 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:15:07.231 INFO:teuthology.orchestra.run.smithi007.stdout: "public_addr": "172.21.15.7:6801/2896425794", 2024-09-18T08:15:07.231 INFO:teuthology.orchestra.run.smithi007.stdout: "cluster_addr": "172.21.15.7:6802/2896425794", 2024-09-18T08:15:07.231 INFO:teuthology.orchestra.run.smithi007.stdout: "heartbeat_back_addr": "172.21.15.7:6804/2896425794", 2024-09-18T08:15:07.231 INFO:teuthology.orchestra.run.smithi007.stdout: "heartbeat_front_addr": "172.21.15.7:6803/2896425794", 2024-09-18T08:15:07.231 INFO:teuthology.orchestra.run.smithi007.stdout: "state": [ 2024-09-18T08:15:07.231 INFO:teuthology.orchestra.run.smithi007.stdout: "exists", 2024-09-18T08:15:07.231 INFO:teuthology.orchestra.run.smithi007.stdout: "up" 2024-09-18T08:15:07.232 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-09-18T08:15:07.232 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:15:07.232 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:15:07.232 INFO:teuthology.orchestra.run.smithi007.stdout: "osd": 1, 2024-09-18T08:15:07.232 INFO:teuthology.orchestra.run.smithi007.stdout: "uuid": "be28b70c-1f73-494f-ad4b-cb7ec5311999", 2024-09-18T08:15:07.232 INFO:teuthology.orchestra.run.smithi007.stdout: "up": 1, 2024-09-18T08:15:07.232 INFO:teuthology.orchestra.run.smithi007.stdout: "in": 1, 2024-09-18T08:15:07.232 INFO:teuthology.orchestra.run.smithi007.stdout: "weight": 1, 2024-09-18T08:15:07.232 INFO:teuthology.orchestra.run.smithi007.stdout: "primary_affinity": 1, 2024-09-18T08:15:07.232 INFO:teuthology.orchestra.run.smithi007.stdout: "last_clean_begin": 13, 2024-09-18T08:15:07.232 INFO:teuthology.orchestra.run.smithi007.stdout: "last_clean_end": 243, 2024-09-18T08:15:07.232 INFO:teuthology.orchestra.run.smithi007.stdout: "up_from": 307, 2024-09-18T08:15:07.232 INFO:teuthology.orchestra.run.smithi007.stdout: "up_thru": 316, 2024-09-18T08:15:07.233 INFO:teuthology.orchestra.run.smithi007.stdout: "down_at": 244, 2024-09-18T08:15:07.233 INFO:teuthology.orchestra.run.smithi007.stdout: "lost_at": 0, 2024-09-18T08:15:07.233 INFO:teuthology.orchestra.run.smithi007.stdout: "public_addrs": { 2024-09-18T08:15:07.233 INFO:teuthology.orchestra.run.smithi007.stdout: "addrvec": [ 2024-09-18T08:15:07.233 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:15:07.233 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "v2", 2024-09-18T08:15:07.233 INFO:teuthology.orchestra.run.smithi007.stdout: "addr": "172.21.15.7:6805", 2024-09-18T08:15:07.233 INFO:teuthology.orchestra.run.smithi007.stdout: "nonce": 3300107791 2024-09-18T08:15:07.233 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-09-18T08:15:07.233 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-09-18T08:15:07.233 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:15:07.233 INFO:teuthology.orchestra.run.smithi007.stdout: "cluster_addrs": { 2024-09-18T08:15:07.233 INFO:teuthology.orchestra.run.smithi007.stdout: "addrvec": [ 2024-09-18T08:15:07.234 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:15:07.234 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "v2", 2024-09-18T08:15:07.234 INFO:teuthology.orchestra.run.smithi007.stdout: "addr": "172.21.15.7:6806", 2024-09-18T08:15:07.234 INFO:teuthology.orchestra.run.smithi007.stdout: "nonce": 3300107791 2024-09-18T08:15:07.234 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-09-18T08:15:07.234 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-09-18T08:15:07.234 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:15:07.234 INFO:teuthology.orchestra.run.smithi007.stdout: "heartbeat_back_addrs": { 2024-09-18T08:15:07.234 INFO:teuthology.orchestra.run.smithi007.stdout: "addrvec": [ 2024-09-18T08:15:07.234 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:15:07.234 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "v2", 2024-09-18T08:15:07.234 INFO:teuthology.orchestra.run.smithi007.stdout: "addr": "172.21.15.7:6808", 2024-09-18T08:15:07.234 INFO:teuthology.orchestra.run.smithi007.stdout: "nonce": 3300107791 2024-09-18T08:15:07.234 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-09-18T08:15:07.235 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-09-18T08:15:07.235 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:15:07.235 INFO:teuthology.orchestra.run.smithi007.stdout: "heartbeat_front_addrs": { 2024-09-18T08:15:07.235 INFO:teuthology.orchestra.run.smithi007.stdout: "addrvec": [ 2024-09-18T08:15:07.235 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:15:07.235 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "v2", 2024-09-18T08:15:07.235 INFO:teuthology.orchestra.run.smithi007.stdout: "addr": "172.21.15.7:6807", 2024-09-18T08:15:07.235 INFO:teuthology.orchestra.run.smithi007.stdout: "nonce": 3300107791 2024-09-18T08:15:07.235 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-09-18T08:15:07.235 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-09-18T08:15:07.235 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:15:07.235 INFO:teuthology.orchestra.run.smithi007.stdout: "public_addr": "172.21.15.7:6805/3300107791", 2024-09-18T08:15:07.235 INFO:teuthology.orchestra.run.smithi007.stdout: "cluster_addr": "172.21.15.7:6806/3300107791", 2024-09-18T08:15:07.236 INFO:teuthology.orchestra.run.smithi007.stdout: "heartbeat_back_addr": "172.21.15.7:6808/3300107791", 2024-09-18T08:15:07.236 INFO:teuthology.orchestra.run.smithi007.stdout: "heartbeat_front_addr": "172.21.15.7:6807/3300107791", 2024-09-18T08:15:07.236 INFO:teuthology.orchestra.run.smithi007.stdout: "state": [ 2024-09-18T08:15:07.236 INFO:teuthology.orchestra.run.smithi007.stdout: "exists", 2024-09-18T08:15:07.236 INFO:teuthology.orchestra.run.smithi007.stdout: "up" 2024-09-18T08:15:07.236 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-09-18T08:15:07.236 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:15:07.236 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:15:07.236 INFO:teuthology.orchestra.run.smithi007.stdout: "osd": 2, 2024-09-18T08:15:07.236 INFO:teuthology.orchestra.run.smithi007.stdout: "uuid": "cfa4f083-c7fd-4743-be65-84a3040fe6dc", 2024-09-18T08:15:07.236 INFO:teuthology.orchestra.run.smithi007.stdout: "up": 1, 2024-09-18T08:15:07.236 INFO:teuthology.orchestra.run.smithi007.stdout: "in": 1, 2024-09-18T08:15:07.236 INFO:teuthology.orchestra.run.smithi007.stdout: "weight": 1, 2024-09-18T08:15:07.236 INFO:teuthology.orchestra.run.smithi007.stdout: "primary_affinity": 1, 2024-09-18T08:15:07.237 INFO:teuthology.orchestra.run.smithi007.stdout: "last_clean_begin": 0, 2024-09-18T08:15:07.237 INFO:teuthology.orchestra.run.smithi007.stdout: "last_clean_end": 0, 2024-09-18T08:15:07.237 INFO:teuthology.orchestra.run.smithi007.stdout: "up_from": 18, 2024-09-18T08:15:07.237 INFO:teuthology.orchestra.run.smithi007.stdout: "up_thru": 324, 2024-09-18T08:15:07.237 INFO:teuthology.orchestra.run.smithi007.stdout: "down_at": 0, 2024-09-18T08:15:07.237 INFO:teuthology.orchestra.run.smithi007.stdout: "lost_at": 0, 2024-09-18T08:15:07.237 INFO:teuthology.orchestra.run.smithi007.stdout: "public_addrs": { 2024-09-18T08:15:07.237 INFO:teuthology.orchestra.run.smithi007.stdout: "addrvec": [ 2024-09-18T08:15:07.237 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:15:07.237 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "v2", 2024-09-18T08:15:07.237 INFO:teuthology.orchestra.run.smithi007.stdout: "addr": "172.21.15.7:6809", 2024-09-18T08:15:07.237 INFO:teuthology.orchestra.run.smithi007.stdout: "nonce": 556142559 2024-09-18T08:15:07.237 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-09-18T08:15:07.237 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-09-18T08:15:07.238 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:15:07.238 INFO:teuthology.orchestra.run.smithi007.stdout: "cluster_addrs": { 2024-09-18T08:15:07.238 INFO:teuthology.orchestra.run.smithi007.stdout: "addrvec": [ 2024-09-18T08:15:07.238 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:15:07.238 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "v2", 2024-09-18T08:15:07.238 INFO:teuthology.orchestra.run.smithi007.stdout: "addr": "172.21.15.7:6810", 2024-09-18T08:15:07.238 INFO:teuthology.orchestra.run.smithi007.stdout: "nonce": 556142559 2024-09-18T08:15:07.238 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-09-18T08:15:07.238 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-09-18T08:15:07.239 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:15:07.239 INFO:teuthology.orchestra.run.smithi007.stdout: "heartbeat_back_addrs": { 2024-09-18T08:15:07.239 INFO:teuthology.orchestra.run.smithi007.stdout: "addrvec": [ 2024-09-18T08:15:07.239 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:15:07.239 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "v2", 2024-09-18T08:15:07.239 INFO:teuthology.orchestra.run.smithi007.stdout: "addr": "172.21.15.7:6812", 2024-09-18T08:15:07.239 INFO:teuthology.orchestra.run.smithi007.stdout: "nonce": 556142559 2024-09-18T08:15:07.239 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-09-18T08:15:07.239 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-09-18T08:15:07.239 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:15:07.239 INFO:teuthology.orchestra.run.smithi007.stdout: "heartbeat_front_addrs": { 2024-09-18T08:15:07.239 INFO:teuthology.orchestra.run.smithi007.stdout: "addrvec": [ 2024-09-18T08:15:07.239 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:15:07.240 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "v2", 2024-09-18T08:15:07.240 INFO:teuthology.orchestra.run.smithi007.stdout: "addr": "172.21.15.7:6811", 2024-09-18T08:15:07.240 INFO:teuthology.orchestra.run.smithi007.stdout: "nonce": 556142559 2024-09-18T08:15:07.240 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-09-18T08:15:07.240 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-09-18T08:15:07.240 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:15:07.240 INFO:teuthology.orchestra.run.smithi007.stdout: "public_addr": "172.21.15.7:6809/556142559", 2024-09-18T08:15:07.240 INFO:teuthology.orchestra.run.smithi007.stdout: "cluster_addr": "172.21.15.7:6810/556142559", 2024-09-18T08:15:07.240 INFO:teuthology.orchestra.run.smithi007.stdout: "heartbeat_back_addr": "172.21.15.7:6812/556142559", 2024-09-18T08:15:07.240 INFO:teuthology.orchestra.run.smithi007.stdout: "heartbeat_front_addr": "172.21.15.7:6811/556142559", 2024-09-18T08:15:07.240 INFO:teuthology.orchestra.run.smithi007.stdout: "state": [ 2024-09-18T08:15:07.240 INFO:teuthology.orchestra.run.smithi007.stdout: "exists", 2024-09-18T08:15:07.241 INFO:teuthology.orchestra.run.smithi007.stdout: "up" 2024-09-18T08:15:07.241 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-09-18T08:15:07.241 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:15:07.241 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:15:07.241 INFO:teuthology.orchestra.run.smithi007.stdout: "osd": 3, 2024-09-18T08:15:07.241 INFO:teuthology.orchestra.run.smithi007.stdout: "uuid": "11ecd7e1-a751-4b75-95f5-00b32d2a5bc4", 2024-09-18T08:15:07.241 INFO:teuthology.orchestra.run.smithi007.stdout: "up": 1, 2024-09-18T08:15:07.241 INFO:teuthology.orchestra.run.smithi007.stdout: "in": 1, 2024-09-18T08:15:07.241 INFO:teuthology.orchestra.run.smithi007.stdout: "weight": 1, 2024-09-18T08:15:07.241 INFO:teuthology.orchestra.run.smithi007.stdout: "primary_affinity": 1, 2024-09-18T08:15:07.241 INFO:teuthology.orchestra.run.smithi007.stdout: "last_clean_begin": 0, 2024-09-18T08:15:07.241 INFO:teuthology.orchestra.run.smithi007.stdout: "last_clean_end": 0, 2024-09-18T08:15:07.241 INFO:teuthology.orchestra.run.smithi007.stdout: "up_from": 26, 2024-09-18T08:15:07.241 INFO:teuthology.orchestra.run.smithi007.stdout: "up_thru": 322, 2024-09-18T08:15:07.242 INFO:teuthology.orchestra.run.smithi007.stdout: "down_at": 0, 2024-09-18T08:15:07.242 INFO:teuthology.orchestra.run.smithi007.stdout: "lost_at": 0, 2024-09-18T08:15:07.242 INFO:teuthology.orchestra.run.smithi007.stdout: "public_addrs": { 2024-09-18T08:15:07.242 INFO:teuthology.orchestra.run.smithi007.stdout: "addrvec": [ 2024-09-18T08:15:07.242 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:15:07.242 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "v2", 2024-09-18T08:15:07.242 INFO:teuthology.orchestra.run.smithi007.stdout: "addr": "172.21.15.7:6813", 2024-09-18T08:15:07.242 INFO:teuthology.orchestra.run.smithi007.stdout: "nonce": 163587772 2024-09-18T08:15:07.242 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-09-18T08:15:07.242 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-09-18T08:15:07.242 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:15:07.242 INFO:teuthology.orchestra.run.smithi007.stdout: "cluster_addrs": { 2024-09-18T08:15:07.242 INFO:teuthology.orchestra.run.smithi007.stdout: "addrvec": [ 2024-09-18T08:15:07.242 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:15:07.243 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "v2", 2024-09-18T08:15:07.243 INFO:teuthology.orchestra.run.smithi007.stdout: "addr": "172.21.15.7:6814", 2024-09-18T08:15:07.243 INFO:teuthology.orchestra.run.smithi007.stdout: "nonce": 163587772 2024-09-18T08:15:07.243 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-09-18T08:15:07.243 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-09-18T08:15:07.243 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:15:07.243 INFO:teuthology.orchestra.run.smithi007.stdout: "heartbeat_back_addrs": { 2024-09-18T08:15:07.243 INFO:teuthology.orchestra.run.smithi007.stdout: "addrvec": [ 2024-09-18T08:15:07.243 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:15:07.243 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "v2", 2024-09-18T08:15:07.243 INFO:teuthology.orchestra.run.smithi007.stdout: "addr": "172.21.15.7:6816", 2024-09-18T08:15:07.243 INFO:teuthology.orchestra.run.smithi007.stdout: "nonce": 163587772 2024-09-18T08:15:07.243 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-09-18T08:15:07.243 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-09-18T08:15:07.244 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:15:07.244 INFO:teuthology.orchestra.run.smithi007.stdout: "heartbeat_front_addrs": { 2024-09-18T08:15:07.244 INFO:teuthology.orchestra.run.smithi007.stdout: "addrvec": [ 2024-09-18T08:15:07.244 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:15:07.244 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "v2", 2024-09-18T08:15:07.244 INFO:teuthology.orchestra.run.smithi007.stdout: "addr": "172.21.15.7:6815", 2024-09-18T08:15:07.244 INFO:teuthology.orchestra.run.smithi007.stdout: "nonce": 163587772 2024-09-18T08:15:07.244 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-09-18T08:15:07.244 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-09-18T08:15:07.244 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:15:07.245 INFO:teuthology.orchestra.run.smithi007.stdout: "public_addr": "172.21.15.7:6813/163587772", 2024-09-18T08:15:07.245 INFO:teuthology.orchestra.run.smithi007.stdout: "cluster_addr": "172.21.15.7:6814/163587772", 2024-09-18T08:15:07.245 INFO:teuthology.orchestra.run.smithi007.stdout: "heartbeat_back_addr": "172.21.15.7:6816/163587772", 2024-09-18T08:15:07.245 INFO:teuthology.orchestra.run.smithi007.stdout: "heartbeat_front_addr": "172.21.15.7:6815/163587772", 2024-09-18T08:15:07.245 INFO:teuthology.orchestra.run.smithi007.stdout: "state": [ 2024-09-18T08:15:07.245 INFO:teuthology.orchestra.run.smithi007.stdout: "exists", 2024-09-18T08:15:07.245 INFO:teuthology.orchestra.run.smithi007.stdout: "up" 2024-09-18T08:15:07.245 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-09-18T08:15:07.245 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:15:07.245 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:15:07.245 INFO:teuthology.orchestra.run.smithi007.stdout: "osd": 4, 2024-09-18T08:15:07.245 INFO:teuthology.orchestra.run.smithi007.stdout: "uuid": "98992b34-cfa1-45bf-97a1-7d557739252f", 2024-09-18T08:15:07.245 INFO:teuthology.orchestra.run.smithi007.stdout: "up": 1, 2024-09-18T08:15:07.245 INFO:teuthology.orchestra.run.smithi007.stdout: "in": 1, 2024-09-18T08:15:07.246 INFO:teuthology.orchestra.run.smithi007.stdout: "weight": 1, 2024-09-18T08:15:07.246 INFO:teuthology.orchestra.run.smithi007.stdout: "primary_affinity": 1, 2024-09-18T08:15:07.246 INFO:teuthology.orchestra.run.smithi007.stdout: "last_clean_begin": 0, 2024-09-18T08:15:07.246 INFO:teuthology.orchestra.run.smithi007.stdout: "last_clean_end": 0, 2024-09-18T08:15:07.246 INFO:teuthology.orchestra.run.smithi007.stdout: "up_from": 32, 2024-09-18T08:15:07.246 INFO:teuthology.orchestra.run.smithi007.stdout: "up_thru": 316, 2024-09-18T08:15:07.246 INFO:teuthology.orchestra.run.smithi007.stdout: "down_at": 0, 2024-09-18T08:15:07.246 INFO:teuthology.orchestra.run.smithi007.stdout: "lost_at": 0, 2024-09-18T08:15:07.246 INFO:teuthology.orchestra.run.smithi007.stdout: "public_addrs": { 2024-09-18T08:15:07.246 INFO:teuthology.orchestra.run.smithi007.stdout: "addrvec": [ 2024-09-18T08:15:07.246 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:15:07.246 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "v2", 2024-09-18T08:15:07.246 INFO:teuthology.orchestra.run.smithi007.stdout: "addr": "172.21.15.50:6800", 2024-09-18T08:15:07.246 INFO:teuthology.orchestra.run.smithi007.stdout: "nonce": 3662119805 2024-09-18T08:15:07.247 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-09-18T08:15:07.247 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-09-18T08:15:07.247 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:15:07.247 INFO:teuthology.orchestra.run.smithi007.stdout: "cluster_addrs": { 2024-09-18T08:15:07.247 INFO:teuthology.orchestra.run.smithi007.stdout: "addrvec": [ 2024-09-18T08:15:07.247 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:15:07.247 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "v2", 2024-09-18T08:15:07.247 INFO:teuthology.orchestra.run.smithi007.stdout: "addr": "172.21.15.50:6801", 2024-09-18T08:15:07.247 INFO:teuthology.orchestra.run.smithi007.stdout: "nonce": 3662119805 2024-09-18T08:15:07.247 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-09-18T08:15:07.247 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-09-18T08:15:07.247 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:15:07.247 INFO:teuthology.orchestra.run.smithi007.stdout: "heartbeat_back_addrs": { 2024-09-18T08:15:07.247 INFO:teuthology.orchestra.run.smithi007.stdout: "addrvec": [ 2024-09-18T08:15:07.248 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:15:07.248 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "v2", 2024-09-18T08:15:07.248 INFO:teuthology.orchestra.run.smithi007.stdout: "addr": "172.21.15.50:6803", 2024-09-18T08:15:07.248 INFO:teuthology.orchestra.run.smithi007.stdout: "nonce": 3662119805 2024-09-18T08:15:07.248 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-09-18T08:15:07.248 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-09-18T08:15:07.248 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:15:07.248 INFO:teuthology.orchestra.run.smithi007.stdout: "heartbeat_front_addrs": { 2024-09-18T08:15:07.248 INFO:teuthology.orchestra.run.smithi007.stdout: "addrvec": [ 2024-09-18T08:15:07.248 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:15:07.249 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "v2", 2024-09-18T08:15:07.249 INFO:teuthology.orchestra.run.smithi007.stdout: "addr": "172.21.15.50:6802", 2024-09-18T08:15:07.249 INFO:teuthology.orchestra.run.smithi007.stdout: "nonce": 3662119805 2024-09-18T08:15:07.249 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-09-18T08:15:07.249 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-09-18T08:15:07.249 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:15:07.249 INFO:teuthology.orchestra.run.smithi007.stdout: "public_addr": "172.21.15.50:6800/3662119805", 2024-09-18T08:15:07.249 INFO:teuthology.orchestra.run.smithi007.stdout: "cluster_addr": "172.21.15.50:6801/3662119805", 2024-09-18T08:15:07.249 INFO:teuthology.orchestra.run.smithi007.stdout: "heartbeat_back_addr": "172.21.15.50:6803/3662119805", 2024-09-18T08:15:07.249 INFO:teuthology.orchestra.run.smithi007.stdout: "heartbeat_front_addr": "172.21.15.50:6802/3662119805", 2024-09-18T08:15:07.249 INFO:teuthology.orchestra.run.smithi007.stdout: "state": [ 2024-09-18T08:15:07.249 INFO:teuthology.orchestra.run.smithi007.stdout: "exists", 2024-09-18T08:15:07.249 INFO:teuthology.orchestra.run.smithi007.stdout: "up" 2024-09-18T08:15:07.250 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-09-18T08:15:07.250 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:15:07.250 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:15:07.250 INFO:teuthology.orchestra.run.smithi007.stdout: "osd": 5, 2024-09-18T08:15:07.250 INFO:teuthology.orchestra.run.smithi007.stdout: "uuid": "7f290005-ebca-467b-87e4-02ead1063610", 2024-09-18T08:15:07.250 INFO:teuthology.orchestra.run.smithi007.stdout: "up": 1, 2024-09-18T08:15:07.250 INFO:teuthology.orchestra.run.smithi007.stdout: "in": 0, 2024-09-18T08:15:07.250 INFO:teuthology.orchestra.run.smithi007.stdout: "weight": 0, 2024-09-18T08:15:07.250 INFO:teuthology.orchestra.run.smithi007.stdout: "primary_affinity": 1, 2024-09-18T08:15:07.250 INFO:teuthology.orchestra.run.smithi007.stdout: "last_clean_begin": 0, 2024-09-18T08:15:07.250 INFO:teuthology.orchestra.run.smithi007.stdout: "last_clean_end": 0, 2024-09-18T08:15:07.250 INFO:teuthology.orchestra.run.smithi007.stdout: "up_from": 39, 2024-09-18T08:15:07.250 INFO:teuthology.orchestra.run.smithi007.stdout: "up_thru": 311, 2024-09-18T08:15:07.250 INFO:teuthology.orchestra.run.smithi007.stdout: "down_at": 0, 2024-09-18T08:15:07.251 INFO:teuthology.orchestra.run.smithi007.stdout: "lost_at": 0, 2024-09-18T08:15:07.251 INFO:teuthology.orchestra.run.smithi007.stdout: "public_addrs": { 2024-09-18T08:15:07.251 INFO:teuthology.orchestra.run.smithi007.stdout: "addrvec": [ 2024-09-18T08:15:07.251 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:15:07.252 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "v2", 2024-09-18T08:15:07.252 INFO:teuthology.orchestra.run.smithi007.stdout: "addr": "172.21.15.50:6804", 2024-09-18T08:15:07.252 INFO:teuthology.orchestra.run.smithi007.stdout: "nonce": 1199797954 2024-09-18T08:15:07.252 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-09-18T08:15:07.252 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-09-18T08:15:07.252 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:15:07.252 INFO:teuthology.orchestra.run.smithi007.stdout: "cluster_addrs": { 2024-09-18T08:15:07.252 INFO:teuthology.orchestra.run.smithi007.stdout: "addrvec": [ 2024-09-18T08:15:07.252 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:15:07.252 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "v2", 2024-09-18T08:15:07.252 INFO:teuthology.orchestra.run.smithi007.stdout: "addr": "172.21.15.50:6805", 2024-09-18T08:15:07.252 INFO:teuthology.orchestra.run.smithi007.stdout: "nonce": 1199797954 2024-09-18T08:15:07.252 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-09-18T08:15:07.253 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-09-18T08:15:07.253 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:15:07.253 INFO:teuthology.orchestra.run.smithi007.stdout: "heartbeat_back_addrs": { 2024-09-18T08:15:07.253 INFO:teuthology.orchestra.run.smithi007.stdout: "addrvec": [ 2024-09-18T08:15:07.253 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:15:07.253 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "v2", 2024-09-18T08:15:07.253 INFO:teuthology.orchestra.run.smithi007.stdout: "addr": "172.21.15.50:6807", 2024-09-18T08:15:07.253 INFO:teuthology.orchestra.run.smithi007.stdout: "nonce": 1199797954 2024-09-18T08:15:07.253 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-09-18T08:15:07.253 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-09-18T08:15:07.253 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:15:07.253 INFO:teuthology.orchestra.run.smithi007.stdout: "heartbeat_front_addrs": { 2024-09-18T08:15:07.253 INFO:teuthology.orchestra.run.smithi007.stdout: "addrvec": [ 2024-09-18T08:15:07.254 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:15:07.254 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "v2", 2024-09-18T08:15:07.254 INFO:teuthology.orchestra.run.smithi007.stdout: "addr": "172.21.15.50:6806", 2024-09-18T08:15:07.254 INFO:teuthology.orchestra.run.smithi007.stdout: "nonce": 1199797954 2024-09-18T08:15:07.254 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-09-18T08:15:07.254 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-09-18T08:15:07.254 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:15:07.254 INFO:teuthology.orchestra.run.smithi007.stdout: "public_addr": "172.21.15.50:6804/1199797954", 2024-09-18T08:15:07.254 INFO:teuthology.orchestra.run.smithi007.stdout: "cluster_addr": "172.21.15.50:6805/1199797954", 2024-09-18T08:15:07.254 INFO:teuthology.orchestra.run.smithi007.stdout: "heartbeat_back_addr": "172.21.15.50:6807/1199797954", 2024-09-18T08:15:07.254 INFO:teuthology.orchestra.run.smithi007.stdout: "heartbeat_front_addr": "172.21.15.50:6806/1199797954", 2024-09-18T08:15:07.255 INFO:teuthology.orchestra.run.smithi007.stdout: "state": [ 2024-09-18T08:15:07.255 INFO:teuthology.orchestra.run.smithi007.stdout: "exists", 2024-09-18T08:15:07.255 INFO:teuthology.orchestra.run.smithi007.stdout: "up" 2024-09-18T08:15:07.255 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-09-18T08:15:07.255 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:15:07.255 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:15:07.255 INFO:teuthology.orchestra.run.smithi007.stdout: "osd": 6, 2024-09-18T08:15:07.255 INFO:teuthology.orchestra.run.smithi007.stdout: "uuid": "e07d846c-ae42-4fea-aa5c-3a25835d18a6", 2024-09-18T08:15:07.255 INFO:teuthology.orchestra.run.smithi007.stdout: "up": 1, 2024-09-18T08:15:07.255 INFO:teuthology.orchestra.run.smithi007.stdout: "in": 1, 2024-09-18T08:15:07.256 INFO:teuthology.orchestra.run.smithi007.stdout: "weight": 1, 2024-09-18T08:15:07.256 INFO:teuthology.orchestra.run.smithi007.stdout: "primary_affinity": 0, 2024-09-18T08:15:07.256 INFO:teuthology.orchestra.run.smithi007.stdout: "last_clean_begin": 0, 2024-09-18T08:15:07.256 INFO:teuthology.orchestra.run.smithi007.stdout: "last_clean_end": 0, 2024-09-18T08:15:07.256 INFO:teuthology.orchestra.run.smithi007.stdout: "up_from": 47, 2024-09-18T08:15:07.256 INFO:teuthology.orchestra.run.smithi007.stdout: "up_thru": 244, 2024-09-18T08:15:07.256 INFO:teuthology.orchestra.run.smithi007.stdout: "down_at": 0, 2024-09-18T08:15:07.256 INFO:teuthology.orchestra.run.smithi007.stdout: "lost_at": 0, 2024-09-18T08:15:07.256 INFO:teuthology.orchestra.run.smithi007.stdout: "public_addrs": { 2024-09-18T08:15:07.256 INFO:teuthology.orchestra.run.smithi007.stdout: "addrvec": [ 2024-09-18T08:15:07.256 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:15:07.257 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "v2", 2024-09-18T08:15:07.257 INFO:teuthology.orchestra.run.smithi007.stdout: "addr": "172.21.15.50:6808", 2024-09-18T08:15:07.257 INFO:teuthology.orchestra.run.smithi007.stdout: "nonce": 255960422 2024-09-18T08:15:07.257 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-09-18T08:15:07.257 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-09-18T08:15:07.257 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:15:07.257 INFO:teuthology.orchestra.run.smithi007.stdout: "cluster_addrs": { 2024-09-18T08:15:07.257 INFO:teuthology.orchestra.run.smithi007.stdout: "addrvec": [ 2024-09-18T08:15:07.257 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:15:07.257 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "v2", 2024-09-18T08:15:07.257 INFO:teuthology.orchestra.run.smithi007.stdout: "addr": "172.21.15.50:6809", 2024-09-18T08:15:07.257 INFO:teuthology.orchestra.run.smithi007.stdout: "nonce": 255960422 2024-09-18T08:15:07.257 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-09-18T08:15:07.258 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-09-18T08:15:07.258 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:15:07.258 INFO:teuthology.orchestra.run.smithi007.stdout: "heartbeat_back_addrs": { 2024-09-18T08:15:07.258 INFO:teuthology.orchestra.run.smithi007.stdout: "addrvec": [ 2024-09-18T08:15:07.258 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:15:07.258 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "v2", 2024-09-18T08:15:07.258 INFO:teuthology.orchestra.run.smithi007.stdout: "addr": "172.21.15.50:6811", 2024-09-18T08:15:07.258 INFO:teuthology.orchestra.run.smithi007.stdout: "nonce": 255960422 2024-09-18T08:15:07.258 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-09-18T08:15:07.258 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-09-18T08:15:07.258 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:15:07.258 INFO:teuthology.orchestra.run.smithi007.stdout: "heartbeat_front_addrs": { 2024-09-18T08:15:07.258 INFO:teuthology.orchestra.run.smithi007.stdout: "addrvec": [ 2024-09-18T08:15:07.258 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:15:07.259 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "v2", 2024-09-18T08:15:07.259 INFO:teuthology.orchestra.run.smithi007.stdout: "addr": "172.21.15.50:6810", 2024-09-18T08:15:07.259 INFO:teuthology.orchestra.run.smithi007.stdout: "nonce": 255960422 2024-09-18T08:15:07.259 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-09-18T08:15:07.259 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-09-18T08:15:07.259 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:15:07.259 INFO:teuthology.orchestra.run.smithi007.stdout: "public_addr": "172.21.15.50:6808/255960422", 2024-09-18T08:15:07.259 INFO:teuthology.orchestra.run.smithi007.stdout: "cluster_addr": "172.21.15.50:6809/255960422", 2024-09-18T08:15:07.259 INFO:teuthology.orchestra.run.smithi007.stdout: "heartbeat_back_addr": "172.21.15.50:6811/255960422", 2024-09-18T08:15:07.259 INFO:teuthology.orchestra.run.smithi007.stdout: "heartbeat_front_addr": "172.21.15.50:6810/255960422", 2024-09-18T08:15:07.260 INFO:teuthology.orchestra.run.smithi007.stdout: "state": [ 2024-09-18T08:15:07.260 INFO:teuthology.orchestra.run.smithi007.stdout: "exists", 2024-09-18T08:15:07.260 INFO:teuthology.orchestra.run.smithi007.stdout: "up" 2024-09-18T08:15:07.260 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-09-18T08:15:07.260 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:15:07.260 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:15:07.260 INFO:teuthology.orchestra.run.smithi007.stdout: "osd": 7, 2024-09-18T08:15:07.260 INFO:teuthology.orchestra.run.smithi007.stdout: "uuid": "c9dac3b7-d651-489d-b96e-5b10fa32d9af", 2024-09-18T08:15:07.260 INFO:teuthology.orchestra.run.smithi007.stdout: "up": 1, 2024-09-18T08:15:07.260 INFO:teuthology.orchestra.run.smithi007.stdout: "in": 0, 2024-09-18T08:15:07.260 INFO:teuthology.orchestra.run.smithi007.stdout: "weight": 0, 2024-09-18T08:15:07.261 INFO:teuthology.orchestra.run.smithi007.stdout: "primary_affinity": 0.4741668701171875, 2024-09-18T08:15:07.261 INFO:teuthology.orchestra.run.smithi007.stdout: "last_clean_begin": 0, 2024-09-18T08:15:07.261 INFO:teuthology.orchestra.run.smithi007.stdout: "last_clean_end": 0, 2024-09-18T08:15:07.261 INFO:teuthology.orchestra.run.smithi007.stdout: "up_from": 56, 2024-09-18T08:15:07.261 INFO:teuthology.orchestra.run.smithi007.stdout: "up_thru": 244, 2024-09-18T08:15:07.261 INFO:teuthology.orchestra.run.smithi007.stdout: "down_at": 0, 2024-09-18T08:15:07.261 INFO:teuthology.orchestra.run.smithi007.stdout: "lost_at": 0, 2024-09-18T08:15:07.261 INFO:teuthology.orchestra.run.smithi007.stdout: "public_addrs": { 2024-09-18T08:15:07.261 INFO:teuthology.orchestra.run.smithi007.stdout: "addrvec": [ 2024-09-18T08:15:07.261 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:15:07.261 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "v2", 2024-09-18T08:15:07.261 INFO:teuthology.orchestra.run.smithi007.stdout: "addr": "172.21.15.50:6812", 2024-09-18T08:15:07.261 INFO:teuthology.orchestra.run.smithi007.stdout: "nonce": 874923904 2024-09-18T08:15:07.262 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-09-18T08:15:07.262 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-09-18T08:15:07.262 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:15:07.262 INFO:teuthology.orchestra.run.smithi007.stdout: "cluster_addrs": { 2024-09-18T08:15:07.262 INFO:teuthology.orchestra.run.smithi007.stdout: "addrvec": [ 2024-09-18T08:15:07.262 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:15:07.262 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "v2", 2024-09-18T08:15:07.262 INFO:teuthology.orchestra.run.smithi007.stdout: "addr": "172.21.15.50:6813", 2024-09-18T08:15:07.262 INFO:teuthology.orchestra.run.smithi007.stdout: "nonce": 874923904 2024-09-18T08:15:07.262 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-09-18T08:15:07.262 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-09-18T08:15:07.262 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:15:07.262 INFO:teuthology.orchestra.run.smithi007.stdout: "heartbeat_back_addrs": { 2024-09-18T08:15:07.263 INFO:teuthology.orchestra.run.smithi007.stdout: "addrvec": [ 2024-09-18T08:15:07.263 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:15:07.263 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "v2", 2024-09-18T08:15:07.263 INFO:teuthology.orchestra.run.smithi007.stdout: "addr": "172.21.15.50:6815", 2024-09-18T08:15:07.263 INFO:teuthology.orchestra.run.smithi007.stdout: "nonce": 874923904 2024-09-18T08:15:07.263 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-09-18T08:15:07.263 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-09-18T08:15:07.263 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:15:07.263 INFO:teuthology.orchestra.run.smithi007.stdout: "heartbeat_front_addrs": { 2024-09-18T08:15:07.263 INFO:teuthology.orchestra.run.smithi007.stdout: "addrvec": [ 2024-09-18T08:15:07.263 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:15:07.263 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "v2", 2024-09-18T08:15:07.264 INFO:teuthology.orchestra.run.smithi007.stdout: "addr": "172.21.15.50:6814", 2024-09-18T08:15:07.264 INFO:teuthology.orchestra.run.smithi007.stdout: "nonce": 874923904 2024-09-18T08:15:07.264 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-09-18T08:15:07.264 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-09-18T08:15:07.264 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:15:07.264 INFO:teuthology.orchestra.run.smithi007.stdout: "public_addr": "172.21.15.50:6812/874923904", 2024-09-18T08:15:07.264 INFO:teuthology.orchestra.run.smithi007.stdout: "cluster_addr": "172.21.15.50:6813/874923904", 2024-09-18T08:15:07.264 INFO:teuthology.orchestra.run.smithi007.stdout: "heartbeat_back_addr": "172.21.15.50:6815/874923904", 2024-09-18T08:15:07.264 INFO:teuthology.orchestra.run.smithi007.stdout: "heartbeat_front_addr": "172.21.15.50:6814/874923904", 2024-09-18T08:15:07.264 INFO:teuthology.orchestra.run.smithi007.stdout: "state": [ 2024-09-18T08:15:07.264 INFO:teuthology.orchestra.run.smithi007.stdout: "exists", 2024-09-18T08:15:07.264 INFO:teuthology.orchestra.run.smithi007.stdout: "up" 2024-09-18T08:15:07.264 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-09-18T08:15:07.265 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-09-18T08:15:07.265 INFO:teuthology.orchestra.run.smithi007.stdout: ], 2024-09-18T08:15:07.265 INFO:teuthology.orchestra.run.smithi007.stdout: "osd_xinfo": [ 2024-09-18T08:15:07.265 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:15:07.265 INFO:teuthology.orchestra.run.smithi007.stdout: "osd": 0, 2024-09-18T08:15:07.265 INFO:teuthology.orchestra.run.smithi007.stdout: "down_stamp": "0.000000", 2024-09-18T08:15:07.265 INFO:teuthology.orchestra.run.smithi007.stdout: "laggy_probability": 0, 2024-09-18T08:15:07.265 INFO:teuthology.orchestra.run.smithi007.stdout: "laggy_interval": 0, 2024-09-18T08:15:07.265 INFO:teuthology.orchestra.run.smithi007.stdout: "features": 4540719139924082687, 2024-09-18T08:15:07.265 INFO:teuthology.orchestra.run.smithi007.stdout: "old_weight": 0, 2024-09-18T08:15:07.265 INFO:teuthology.orchestra.run.smithi007.stdout: "last_purged_snaps_scrub": "2024-09-18T08:13:55.278942+0000", 2024-09-18T08:15:07.265 INFO:teuthology.orchestra.run.smithi007.stdout: "dead_epoch": 0 2024-09-18T08:15:07.265 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:15:07.265 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:15:07.266 INFO:teuthology.orchestra.run.smithi007.stdout: "osd": 1, 2024-09-18T08:15:07.266 INFO:teuthology.orchestra.run.smithi007.stdout: "down_stamp": "0.000000", 2024-09-18T08:15:07.266 INFO:teuthology.orchestra.run.smithi007.stdout: "laggy_probability": 0, 2024-09-18T08:15:07.266 INFO:teuthology.orchestra.run.smithi007.stdout: "laggy_interval": 0, 2024-09-18T08:15:07.266 INFO:teuthology.orchestra.run.smithi007.stdout: "features": 4540719139924082687, 2024-09-18T08:15:07.266 INFO:teuthology.orchestra.run.smithi007.stdout: "old_weight": 0, 2024-09-18T08:15:07.266 INFO:teuthology.orchestra.run.smithi007.stdout: "last_purged_snaps_scrub": "2024-09-18T08:14:38.200574+0000", 2024-09-18T08:15:07.266 INFO:teuthology.orchestra.run.smithi007.stdout: "dead_epoch": 243 2024-09-18T08:15:07.266 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:15:07.266 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:15:07.266 INFO:teuthology.orchestra.run.smithi007.stdout: "osd": 2, 2024-09-18T08:15:07.266 INFO:teuthology.orchestra.run.smithi007.stdout: "down_stamp": "0.000000", 2024-09-18T08:15:07.266 INFO:teuthology.orchestra.run.smithi007.stdout: "laggy_probability": 0, 2024-09-18T08:15:07.266 INFO:teuthology.orchestra.run.smithi007.stdout: "laggy_interval": 0, 2024-09-18T08:15:07.267 INFO:teuthology.orchestra.run.smithi007.stdout: "features": 4540719139924082687, 2024-09-18T08:15:07.267 INFO:teuthology.orchestra.run.smithi007.stdout: "old_weight": 0, 2024-09-18T08:15:07.267 INFO:teuthology.orchestra.run.smithi007.stdout: "last_purged_snaps_scrub": "2024-09-18T08:14:56.771953+0000", 2024-09-18T08:15:07.267 INFO:teuthology.orchestra.run.smithi007.stdout: "dead_epoch": 0 2024-09-18T08:15:07.267 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:15:07.267 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:15:07.267 INFO:teuthology.orchestra.run.smithi007.stdout: "osd": 3, 2024-09-18T08:15:07.267 INFO:teuthology.orchestra.run.smithi007.stdout: "down_stamp": "0.000000", 2024-09-18T08:15:07.267 INFO:teuthology.orchestra.run.smithi007.stdout: "laggy_probability": 0, 2024-09-18T08:15:07.267 INFO:teuthology.orchestra.run.smithi007.stdout: "laggy_interval": 0, 2024-09-18T08:15:07.267 INFO:teuthology.orchestra.run.smithi007.stdout: "features": 4540719139924082687, 2024-09-18T08:15:07.267 INFO:teuthology.orchestra.run.smithi007.stdout: "old_weight": 0, 2024-09-18T08:15:07.267 INFO:teuthology.orchestra.run.smithi007.stdout: "last_purged_snaps_scrub": "2024-09-18T08:14:31.312214+0000", 2024-09-18T08:15:07.268 INFO:teuthology.orchestra.run.smithi007.stdout: "dead_epoch": 0 2024-09-18T08:15:07.268 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:15:07.268 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:15:07.268 INFO:teuthology.orchestra.run.smithi007.stdout: "osd": 4, 2024-09-18T08:15:07.268 INFO:teuthology.orchestra.run.smithi007.stdout: "down_stamp": "0.000000", 2024-09-18T08:15:07.268 INFO:teuthology.orchestra.run.smithi007.stdout: "laggy_probability": 0, 2024-09-18T08:15:07.268 INFO:teuthology.orchestra.run.smithi007.stdout: "laggy_interval": 0, 2024-09-18T08:15:07.268 INFO:teuthology.orchestra.run.smithi007.stdout: "features": 4540719139924082687, 2024-09-18T08:15:07.268 INFO:teuthology.orchestra.run.smithi007.stdout: "old_weight": 0, 2024-09-18T08:15:07.268 INFO:teuthology.orchestra.run.smithi007.stdout: "last_purged_snaps_scrub": "2024-09-18T08:14:29.676529+0000", 2024-09-18T08:15:07.268 INFO:teuthology.orchestra.run.smithi007.stdout: "dead_epoch": 0 2024-09-18T08:15:07.268 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:15:07.268 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:15:07.269 INFO:teuthology.orchestra.run.smithi007.stdout: "osd": 5, 2024-09-18T08:15:07.269 INFO:teuthology.orchestra.run.smithi007.stdout: "down_stamp": "0.000000", 2024-09-18T08:15:07.269 INFO:teuthology.orchestra.run.smithi007.stdout: "laggy_probability": 0, 2024-09-18T08:15:07.269 INFO:teuthology.orchestra.run.smithi007.stdout: "laggy_interval": 0, 2024-09-18T08:15:07.269 INFO:teuthology.orchestra.run.smithi007.stdout: "features": 4540719139924082687, 2024-09-18T08:15:07.269 INFO:teuthology.orchestra.run.smithi007.stdout: "old_weight": 65536, 2024-09-18T08:15:07.269 INFO:teuthology.orchestra.run.smithi007.stdout: "last_purged_snaps_scrub": "2024-09-18T08:14:55.084309+0000", 2024-09-18T08:15:07.269 INFO:teuthology.orchestra.run.smithi007.stdout: "dead_epoch": 0 2024-09-18T08:15:07.269 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:15:07.269 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:15:07.270 INFO:teuthology.orchestra.run.smithi007.stdout: "osd": 6, 2024-09-18T08:15:07.270 INFO:teuthology.orchestra.run.smithi007.stdout: "down_stamp": "0.000000", 2024-09-18T08:15:07.270 INFO:teuthology.orchestra.run.smithi007.stdout: "laggy_probability": 0, 2024-09-18T08:15:07.270 INFO:teuthology.orchestra.run.smithi007.stdout: "laggy_interval": 0, 2024-09-18T08:15:07.270 INFO:teuthology.orchestra.run.smithi007.stdout: "features": 4540719139924082687, 2024-09-18T08:15:07.270 INFO:teuthology.orchestra.run.smithi007.stdout: "old_weight": 0, 2024-09-18T08:15:07.270 INFO:teuthology.orchestra.run.smithi007.stdout: "last_purged_snaps_scrub": "2024-09-18T08:14:26.613211+0000", 2024-09-18T08:15:07.270 INFO:teuthology.orchestra.run.smithi007.stdout: "dead_epoch": 0 2024-09-18T08:15:07.270 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:15:07.270 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:15:07.270 INFO:teuthology.orchestra.run.smithi007.stdout: "osd": 7, 2024-09-18T08:15:07.271 INFO:teuthology.orchestra.run.smithi007.stdout: "down_stamp": "0.000000", 2024-09-18T08:15:07.271 INFO:teuthology.orchestra.run.smithi007.stdout: "laggy_probability": 0, 2024-09-18T08:15:07.271 INFO:teuthology.orchestra.run.smithi007.stdout: "laggy_interval": 0, 2024-09-18T08:15:07.271 INFO:teuthology.orchestra.run.smithi007.stdout: "features": 4540719139924082687, 2024-09-18T08:15:07.271 INFO:teuthology.orchestra.run.smithi007.stdout: "old_weight": 65536, 2024-09-18T08:15:07.271 INFO:teuthology.orchestra.run.smithi007.stdout: "last_purged_snaps_scrub": "2024-09-18T08:14:06.213584+0000", 2024-09-18T08:15:07.271 INFO:teuthology.orchestra.run.smithi007.stdout: "dead_epoch": 0 2024-09-18T08:15:07.271 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-09-18T08:15:07.271 INFO:teuthology.orchestra.run.smithi007.stdout: ], 2024-09-18T08:15:07.271 INFO:teuthology.orchestra.run.smithi007.stdout: "pg_upmap": [ 2024-09-18T08:15:07.271 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:15:07.271 INFO:teuthology.orchestra.run.smithi007.stdout: "pgid": "6.3", 2024-09-18T08:15:07.271 INFO:teuthology.orchestra.run.smithi007.stdout: "osds": [ 2024-09-18T08:15:07.271 INFO:teuthology.orchestra.run.smithi007.stdout: 5, 2024-09-18T08:15:07.272 INFO:teuthology.orchestra.run.smithi007.stdout: 3, 2024-09-18T08:15:07.272 INFO:teuthology.orchestra.run.smithi007.stdout: 2 2024-09-18T08:15:07.272 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-09-18T08:15:07.272 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-09-18T08:15:07.272 INFO:teuthology.orchestra.run.smithi007.stdout: ], 2024-09-18T08:15:07.272 INFO:teuthology.orchestra.run.smithi007.stdout: "pg_upmap_items": [ 2024-09-18T08:15:07.272 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:15:07.272 INFO:teuthology.orchestra.run.smithi007.stdout: "pgid": "4.7", 2024-09-18T08:15:07.272 INFO:teuthology.orchestra.run.smithi007.stdout: "mappings": [ 2024-09-18T08:15:07.272 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:15:07.272 INFO:teuthology.orchestra.run.smithi007.stdout: "from": 1, 2024-09-18T08:15:07.272 INFO:teuthology.orchestra.run.smithi007.stdout: "to": 2 2024-09-18T08:15:07.272 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-09-18T08:15:07.273 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-09-18T08:15:07.273 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:15:07.273 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:15:07.273 INFO:teuthology.orchestra.run.smithi007.stdout: "pgid": "4.14", 2024-09-18T08:15:07.273 INFO:teuthology.orchestra.run.smithi007.stdout: "mappings": [ 2024-09-18T08:15:07.273 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:15:07.273 INFO:teuthology.orchestra.run.smithi007.stdout: "from": 1, 2024-09-18T08:15:07.273 INFO:teuthology.orchestra.run.smithi007.stdout: "to": 2 2024-09-18T08:15:07.273 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-09-18T08:15:07.273 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-09-18T08:15:07.273 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-09-18T08:15:07.273 INFO:teuthology.orchestra.run.smithi007.stdout: ], 2024-09-18T08:15:07.273 INFO:teuthology.orchestra.run.smithi007.stdout: "pg_upmap_primaries": [], 2024-09-18T08:15:07.273 INFO:teuthology.orchestra.run.smithi007.stdout: "pg_temp": [], 2024-09-18T08:15:07.274 INFO:teuthology.orchestra.run.smithi007.stdout: "primary_temp": [], 2024-09-18T08:15:07.274 INFO:teuthology.orchestra.run.smithi007.stdout: "blocklist": { 2024-09-18T08:15:07.274 INFO:teuthology.orchestra.run.smithi007.stdout: "172.21.15.7:0/398367658": "2024-09-19T08:03:17.052903+0000", 2024-09-18T08:15:07.274 INFO:teuthology.orchestra.run.smithi007.stdout: "172.21.15.7:6800/2918496291": "2024-09-19T07:57:31.015269+0000", 2024-09-18T08:15:07.274 INFO:teuthology.orchestra.run.smithi007.stdout: "172.21.15.7:0/2514354773": "2024-09-19T07:57:31.015269+0000", 2024-09-18T08:15:07.274 INFO:teuthology.orchestra.run.smithi007.stdout: "172.21.15.7:0/3924911335": "2024-09-19T07:57:31.015269+0000", 2024-09-18T08:15:07.274 INFO:teuthology.orchestra.run.smithi007.stdout: "172.21.15.7:0/1668598084": "2024-09-19T07:57:57.346149+0000", 2024-09-18T08:15:07.274 INFO:teuthology.orchestra.run.smithi007.stdout: "172.21.15.7:0/4272027292": "2024-09-19T07:57:31.015269+0000", 2024-09-18T08:15:07.274 INFO:teuthology.orchestra.run.smithi007.stdout: "172.21.15.7:6800/1013061071": "2024-09-19T07:57:57.346149+0000", 2024-09-18T08:15:07.274 INFO:teuthology.orchestra.run.smithi007.stdout: "172.21.15.7:6800/1491061076": "2024-09-19T08:03:17.052903+0000", 2024-09-18T08:15:07.274 INFO:teuthology.orchestra.run.smithi007.stdout: "172.21.15.7:0/533727535": "2024-09-19T07:57:57.346149+0000", 2024-09-18T08:15:07.274 INFO:teuthology.orchestra.run.smithi007.stdout: "172.21.15.7:0/1605217610": "2024-09-19T08:03:17.052903+0000", 2024-09-18T08:15:07.274 INFO:teuthology.orchestra.run.smithi007.stdout: "172.21.15.7:0/1722968065": "2024-09-19T08:03:17.052903+0000", 2024-09-18T08:15:07.275 INFO:teuthology.orchestra.run.smithi007.stdout: "172.21.15.7:0/1277549846": "2024-09-19T07:57:57.346149+0000", 2024-09-18T08:15:07.275 INFO:teuthology.orchestra.run.smithi007.stdout: "172.21.15.7:0/232452890": "2024-09-19T08:03:17.052903+0000", 2024-09-18T08:15:07.275 INFO:teuthology.orchestra.run.smithi007.stdout: "172.21.15.7:0/3881670470": "2024-09-19T08:03:17.052903+0000" 2024-09-18T08:15:07.275 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:15:07.275 INFO:teuthology.orchestra.run.smithi007.stdout: "range_blocklist": {}, 2024-09-18T08:15:07.275 INFO:teuthology.orchestra.run.smithi007.stdout: "erasure_code_profiles": { 2024-09-18T08:15:07.275 INFO:teuthology.orchestra.run.smithi007.stdout: "default": { 2024-09-18T08:15:07.275 INFO:teuthology.orchestra.run.smithi007.stdout: "crush-failure-domain": "osd", 2024-09-18T08:15:07.275 INFO:teuthology.orchestra.run.smithi007.stdout: "k": "2", 2024-09-18T08:15:07.275 INFO:teuthology.orchestra.run.smithi007.stdout: "m": "1", 2024-09-18T08:15:07.275 INFO:teuthology.orchestra.run.smithi007.stdout: "plugin": "jerasure", 2024-09-18T08:15:07.275 INFO:teuthology.orchestra.run.smithi007.stdout: "technique": "reed_sol_van" 2024-09-18T08:15:07.275 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-09-18T08:15:07.275 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:15:07.276 INFO:teuthology.orchestra.run.smithi007.stdout: "removed_snaps_queue": [], 2024-09-18T08:15:07.276 INFO:teuthology.orchestra.run.smithi007.stdout: "new_removed_snaps": [], 2024-09-18T08:15:07.276 INFO:teuthology.orchestra.run.smithi007.stdout: "new_purged_snaps": [], 2024-09-18T08:15:07.276 INFO:teuthology.orchestra.run.smithi007.stdout: "crush_node_flags": {}, 2024-09-18T08:15:07.276 INFO:teuthology.orchestra.run.smithi007.stdout: "device_class_flags": {}, 2024-09-18T08:15:07.276 INFO:teuthology.orchestra.run.smithi007.stdout: "stretch_mode": { 2024-09-18T08:15:07.276 INFO:teuthology.orchestra.run.smithi007.stdout: "stretch_mode_enabled": false, 2024-09-18T08:15:07.276 INFO:teuthology.orchestra.run.smithi007.stdout: "stretch_bucket_count": 0, 2024-09-18T08:15:07.276 INFO:teuthology.orchestra.run.smithi007.stdout: "degraded_stretch_mode": 0, 2024-09-18T08:15:07.276 INFO:teuthology.orchestra.run.smithi007.stdout: "recovering_stretch_mode": 0, 2024-09-18T08:15:07.276 INFO:teuthology.orchestra.run.smithi007.stdout: "stretch_mode_bucket": 0 2024-09-18T08:15:07.276 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-09-18T08:15:07.276 INFO:teuthology.orchestra.run.smithi007.stdout:} 2024-09-18T08:15:07.288 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:15:07.289 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:15:07.289 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1' 2024-09-18T08:15:07.337 INFO:tasks.radosbench.radosbench.0.smithi007.stdout: 19 16 27325 27309 89.8155 106.75 0.00458001 0.0111283 2024-09-18T08:15:07.391 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5 2024-09-18T08:15:07.403 INFO:teuthology.orchestra.run.smithi050.stderr:Inferring config /var/lib/ceph/779349ca-7593-11ef-bceb-c7b262605968/mon.b/config 2024-09-18T08:15:07.421 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:15:07.422 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:15:07.423 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5' 2024-09-18T08:15:07.523 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2 2024-09-18T08:15:07.565 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:15:07.565 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:15:07.566 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2' 2024-09-18T08:15:07.648 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid 779349ca-7593-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-18T08:15:07.667 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4 2024-09-18T08:15:07.696 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:15:07.697 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:15:07.697 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4' 2024-09-18T08:15:07.718 INFO:teuthology.orchestra.run.smithi007.stderr:nodeep-scrub is unset 2024-09-18T08:15:07.797 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6 2024-09-18T08:15:07.825 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:15:07.826 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:15:07.826 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6' 2024-09-18T08:15:07.849 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:15:07 smithi050 ceph-mon[32371]: 6.1c scrub starts 2024-09-18T08:15:07.849 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:15:07 smithi050 ceph-mon[32371]: 6.1c scrub ok 2024-09-18T08:15:07.849 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:15:07 smithi050 ceph-mon[32371]: from='client.? 172.21.15.7:0/205270132' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-18T08:15:07.849 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:15:07 smithi050 ceph-mon[32371]: from='client.? 172.21.15.7:0/431301539' entity='client.admin' cmd={"prefix": "osd dump", "format": "json-pretty"} : dispatch 2024-09-18T08:15:07.849 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:15:07 smithi050 ceph-mon[32371]: 6.4 scrub starts 2024-09-18T08:15:07.849 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:15:07 smithi050 ceph-mon[32371]: 6.4 scrub ok 2024-09-18T08:15:07.849 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:15:07 smithi050 ceph-mon[32371]: pgmap v610: 158 pgs: 158 active+clean; 1.3 GiB data, 5.9 GiB used, 531 GiB / 536 GiB avail; 639 B/s rd, 36 MiB/s wr, 575 op/s; 4.4 MiB/s, 70 objects/s recovering 2024-09-18T08:15:07.927 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4 2024-09-18T08:15:07.956 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:15:07.957 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:15:07.958 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4' 2024-09-18T08:15:07.981 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:15:07 smithi007 ceph-mon[29459]: 6.1c scrub starts 2024-09-18T08:15:07.981 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:15:07 smithi007 ceph-mon[29459]: 6.1c scrub ok 2024-09-18T08:15:07.981 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:15:07 smithi007 ceph-mon[29459]: from='client.? 172.21.15.7:0/205270132' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-18T08:15:07.981 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:15:07 smithi007 ceph-mon[29459]: from='client.? 172.21.15.7:0/431301539' entity='client.admin' cmd={"prefix": "osd dump", "format": "json-pretty"} : dispatch 2024-09-18T08:15:07.981 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:15:07 smithi007 ceph-mon[29459]: 6.4 scrub starts 2024-09-18T08:15:07.981 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:15:07 smithi007 ceph-mon[29459]: 6.4 scrub ok 2024-09-18T08:15:07.981 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:15:07 smithi007 ceph-mon[29459]: pgmap v610: 158 pgs: 158 active+clean; 1.3 GiB data, 5.9 GiB used, 531 GiB / 536 GiB avail; 639 B/s rd, 36 MiB/s wr, 575 op/s; 4.4 MiB/s, 70 objects/s recovering 2024-09-18T08:15:07.981 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:15:07 smithi007 ceph-mon[39249]: 6.1c scrub starts 2024-09-18T08:15:07.981 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:15:07 smithi007 ceph-mon[39249]: 6.1c scrub ok 2024-09-18T08:15:07.982 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:15:07 smithi007 ceph-mon[39249]: from='client.? 172.21.15.7:0/205270132' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-18T08:15:07.982 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:15:07 smithi007 ceph-mon[39249]: from='client.? 172.21.15.7:0/431301539' entity='client.admin' cmd={"prefix": "osd dump", "format": "json-pretty"} : dispatch 2024-09-18T08:15:07.982 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:15:07 smithi007 ceph-mon[39249]: 6.4 scrub starts 2024-09-18T08:15:07.982 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:15:07 smithi007 ceph-mon[39249]: 6.4 scrub ok 2024-09-18T08:15:07.982 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:15:07 smithi007 ceph-mon[39249]: pgmap v610: 158 pgs: 158 active+clean; 1.3 GiB data, 5.9 GiB used, 531 GiB / 536 GiB avail; 639 B/s rd, 36 MiB/s wr, 575 op/s; 4.4 MiB/s, 70 objects/s recovering 2024-09-18T08:15:07.986 INFO:teuthology.orchestra.run.smithi050.stdout:ERROR: (22) Invalid argument 2024-09-18T08:15:07.986 INFO:teuthology.orchestra.run.smithi050.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-18T08:15:08.059 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0 2024-09-18T08:15:08.101 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:15:08.102 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:15:08.102 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0' 2024-09-18T08:15:08.203 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2 2024-09-18T08:15:08.242 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:15:08.243 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:15:08.243 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2' 2024-09-18T08:15:08.337 INFO:tasks.radosbench.radosbench.0.smithi007.stdout:2024-09-18T08:15:08.336604+0000 min lat: 0.00117701 max lat: 2.07039 avg lat: 0.0109611 2024-09-18T08:15:08.337 INFO:tasks.radosbench.radosbench.0.smithi007.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-09-18T08:15:08.337 INFO:tasks.radosbench.radosbench.0.smithi007.stdout: 20 16 29197 29181 91.1741 117 0.00999335 0.0109611 2024-09-18T08:15:08.345 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6 2024-09-18T08:15:08.370 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:15:08.371 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:15:08.371 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6' 2024-09-18T08:15:08.386 INFO:tasks.thrashosds.thrasher:j is {'epoch': 325, 'fsid': '779349ca-7593-11ef-bceb-c7b262605968', 'created': '2024-09-18T07:57:09.295710+0000', 'modified': '2024-09-18T08:15:01.347778+0000', 'last_up_change': '2024-09-18T08:14:38.391465+0000', 'last_in_change': '2024-09-18T08:14:51.619626+0000', 'flags': 'nodeep-scrub,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit', 'flags_num': 5804032, 'flags_set': ['nodeep-scrub', 'pglog_hardlimit', 'purged_snapdirs', 'recovery_deletes', 'sortbitwise'], 'crush_version': 19, 'full_ratio': 0.949999988079071, 'backfillfull_ratio': 0.8999999761581421, 'nearfull_ratio': 0.8500000238418579, 'cluster_snapshot': '', 'pool_max': 10, 'max_osd': 8, 'require_min_compat_client': 'luminous', 'min_compat_client': 'luminous', 'require_osd_release': 'squid', 'allow_crimson': False, 'pools': [{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-09-18T08:00:11.802627+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'is_stretch_pool': False, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 11, 'pg_placement_num': 11, 'pg_placement_num_target': 11, 'pg_num_target': 11, 'pg_num_pending': 11, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '285', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '152', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.5299999713897705, 'score_stable': 2.5299999713897705, 'optimal_score': 0.8600000143051147, 'raw_score_acting': 2.180000066757202, 'raw_score_stable': 2.180000066757202, 'primary_affinity_weighted': 0.8092708587646484, 'average_primary_affinity': 0.9200000166893005, 'average_primary_affinity_weighted': 0.8100000023841858}}, {'pool': 2, 'pool_name': 'datapool', 'create_time': '2024-09-18T08:02:45.374099+0000', 'flags': 8193, 'flags_names': 'hashpspool,selfmanaged_snaps', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'is_stretch_pool': False, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 3, 'pg_placement_num': 3, 'pg_placement_num_target': 3, 'pg_num_target': 3, 'pg_num_pending': 3, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '64', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 2, 'snap_epoch': 64, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 4.690000057220459, 'score_stable': 4.690000057220459, 'optimal_score': 0.7099999785423279, 'raw_score_acting': 3.3299999237060547, 'raw_score_stable': 3.3299999237060547, 'primary_affinity_weighted': 0.8092708587646484, 'average_primary_affinity': 0.9200000166893005, 'average_primary_affinity_weighted': 0.8100000023841858}}, {'pool': 3, 'pool_name': '.rgw.root', 'create_time': '2024-09-18T08:02:46.319947+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'is_stretch_pool': False, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '63', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rgw': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.4000000953674316, 'score_stable': 2.4000000953674316, 'optimal_score': 0.8600000143051147, 'raw_score_acting': 2.059999942779541, 'raw_score_stable': 2.059999942779541, 'primary_affinity_weighted': 0.8092708587646484, 'average_primary_affinity': 0.9200000166893005, 'average_primary_affinity_weighted': 0.8100000023841858}}, {'pool': 4, 'pool_name': 'default.rgw.log', 'create_time': '2024-09-18T08:02:47.948552+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'is_stretch_pool': False, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '65', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rgw': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.4000000953674316, 'score_stable': 2.4000000953674316, 'optimal_score': 0.8600000143051147, 'raw_score_acting': 2.059999942779541, 'raw_score_stable': 2.059999942779541, 'primary_affinity_weighted': 0.8092708587646484, 'average_primary_affinity': 0.9200000166893005, 'average_primary_affinity_weighted': 0.8100000023841858}}, {'pool': 5, 'pool_name': 'default.rgw.control', 'create_time': '2024-09-18T08:02:49.904302+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'is_stretch_pool': False, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '67', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rgw': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.190000057220459, 'score_stable': 2.190000057220459, 'optimal_score': 0.8600000143051147, 'raw_score_acting': 1.8799999952316284, 'raw_score_stable': 1.8799999952316284, 'primary_affinity_weighted': 0.8092708587646484, 'average_primary_affinity': 0.9200000166893005, 'average_primary_affinity_weighted': 0.8100000023841858}}, {'pool': 6, 'pool_name': 'default.rgw.meta', 'create_time': '2024-09-18T08:02:51.971465+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'is_stretch_pool': False, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '70', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4}, 'application_metadata': {'rgw': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.9700000286102295, 'score_stable': 1.9700000286102295, 'optimal_score': 0.8600000143051147, 'raw_score_acting': 1.690000057220459, 'raw_score_stable': 1.690000057220459, 'primary_affinity_weighted': 0.8092708587646484, 'average_primary_affinity': 0.9200000166893005, 'average_primary_affinity_weighted': 0.8100000023841858}}, {'pool': 10, 'pool_name': 'unique_pool_3', 'create_time': '2024-09-18T08:14:42.583623+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'is_stretch_pool': False, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 16, 'pg_placement_num': 16, 'pg_placement_num_target': 16, 'pg_num_target': 16, 'pg_num_pending': 16, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '314', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rados': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.7400000095367432, 'score_stable': 1.7400000095367432, 'optimal_score': 0.8600000143051147, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.8092708587646484, 'average_primary_affinity': 0.9200000166893005, 'average_primary_affinity_weighted': 0.8100000023841858}}], 'osds': [{'osd': 0, 'uuid': 'ae8f60df-72ac-46fb-91a1-f132b851c876', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 8, 'up_thru': 323, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.7:6801', 'nonce': 2896425794}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.7:6802', 'nonce': 2896425794}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.7:6804', 'nonce': 2896425794}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.7:6803', 'nonce': 2896425794}]}, 'public_addr': '172.21.15.7:6801/2896425794', 'cluster_addr': '172.21.15.7:6802/2896425794', 'heartbeat_back_addr': '172.21.15.7:6804/2896425794', 'heartbeat_front_addr': '172.21.15.7:6803/2896425794', 'state': ['exists', 'up']}, {'osd': 1, 'uuid': 'be28b70c-1f73-494f-ad4b-cb7ec5311999', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 13, 'last_clean_end': 243, 'up_from': 307, 'up_thru': 316, 'down_at': 244, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.7:6805', 'nonce': 3300107791}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.7:6806', 'nonce': 3300107791}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.7:6808', 'nonce': 3300107791}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.7:6807', 'nonce': 3300107791}]}, 'public_addr': '172.21.15.7:6805/3300107791', 'cluster_addr': '172.21.15.7:6806/3300107791', 'heartbeat_back_addr': '172.21.15.7:6808/3300107791', 'heartbeat_front_addr': '172.21.15.7:6807/3300107791', 'state': ['exists', 'up']}, {'osd': 2, 'uuid': 'cfa4f083-c7fd-4743-be65-84a3040fe6dc', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 18, 'up_thru': 324, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.7:6809', 'nonce': 556142559}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.7:6810', 'nonce': 556142559}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.7:6812', 'nonce': 556142559}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.7:6811', 'nonce': 556142559}]}, 'public_addr': '172.21.15.7:6809/556142559', 'cluster_addr': '172.21.15.7:6810/556142559', 'heartbeat_back_addr': '172.21.15.7:6812/556142559', 'heartbeat_front_addr': '172.21.15.7:6811/556142559', 'state': ['exists', 'up']}, {'osd': 3, 'uuid': '11ecd7e1-a751-4b75-95f5-00b32d2a5bc4', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 26, 'up_thru': 322, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.7:6813', 'nonce': 163587772}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.7:6814', 'nonce': 163587772}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.7:6816', 'nonce': 163587772}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.7:6815', 'nonce': 163587772}]}, 'public_addr': '172.21.15.7:6813/163587772', 'cluster_addr': '172.21.15.7:6814/163587772', 'heartbeat_back_addr': '172.21.15.7:6816/163587772', 'heartbeat_front_addr': '172.21.15.7:6815/163587772', 'state': ['exists', 'up']}, {'osd': 4, 'uuid': '98992b34-cfa1-45bf-97a1-7d557739252f', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 32, 'up_thru': 316, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.50:6800', 'nonce': 3662119805}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.50:6801', 'nonce': 3662119805}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.50:6803', 'nonce': 3662119805}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.50:6802', 'nonce': 3662119805}]}, 'public_addr': '172.21.15.50:6800/3662119805', 'cluster_addr': '172.21.15.50:6801/3662119805', 'heartbeat_back_addr': '172.21.15.50:6803/3662119805', 'heartbeat_front_addr': '172.21.15.50:6802/3662119805', 'state': ['exists', 'up']}, {'osd': 5, 'uuid': '7f290005-ebca-467b-87e4-02ead1063610', 'up': 1, 'in': 0, 'weight': 0, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 39, 'up_thru': 311, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.50:6804', 'nonce': 1199797954}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.50:6805', 'nonce': 1199797954}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.50:6807', 'nonce': 1199797954}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.50:6806', 'nonce': 1199797954}]}, 'public_addr': '172.21.15.50:6804/1199797954', 'cluster_addr': '172.21.15.50:6805/1199797954', 'heartbeat_back_addr': '172.21.15.50:6807/1199797954', 'heartbeat_front_addr': '172.21.15.50:6806/1199797954', 'state': ['exists', 'up']}, {'osd': 6, 'uuid': 'e07d846c-ae42-4fea-aa5c-3a25835d18a6', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 0, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 47, 'up_thru': 244, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.50:6808', 'nonce': 255960422}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.50:6809', 'nonce': 255960422}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.50:6811', 'nonce': 255960422}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.50:6810', 'nonce': 255960422}]}, 'public_addr': '172.21.15.50:6808/255960422', 'cluster_addr': '172.21.15.50:6809/255960422', 'heartbeat_back_addr': '172.21.15.50:6811/255960422', 'heartbeat_front_addr': '172.21.15.50:6810/255960422', 'state': ['exists', 'up']}, {'osd': 7, 'uuid': 'c9dac3b7-d651-489d-b96e-5b10fa32d9af', 'up': 1, 'in': 0, 'weight': 0, 'primary_affinity': 0.4741668701171875, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 56, 'up_thru': 244, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.50:6812', 'nonce': 874923904}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.50:6813', 'nonce': 874923904}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.50:6815', 'nonce': 874923904}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.50:6814', 'nonce': 874923904}]}, 'public_addr': '172.21.15.50:6812/874923904', 'cluster_addr': '172.21.15.50:6813/874923904', 'heartbeat_back_addr': '172.21.15.50:6815/874923904', 'heartbeat_front_addr': '172.21.15.50:6814/874923904', 'state': ['exists', 'up']}], 'osd_xinfo': [{'osd': 0, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540719139924082687, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-18T08:13:55.278942+0000', 'dead_epoch': 0}, {'osd': 1, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540719139924082687, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-18T08:14:38.200574+0000', 'dead_epoch': 243}, {'osd': 2, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540719139924082687, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-18T08:14:56.771953+0000', 'dead_epoch': 0}, {'osd': 3, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540719139924082687, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-18T08:14:31.312214+0000', 'dead_epoch': 0}, {'osd': 4, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540719139924082687, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-18T08:14:29.676529+0000', 'dead_epoch': 0}, {'osd': 5, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540719139924082687, 'old_weight': 65536, 'last_purged_snaps_scrub': '2024-09-18T08:14:55.084309+0000', 'dead_epoch': 0}, {'osd': 6, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540719139924082687, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-18T08:14:26.613211+0000', 'dead_epoch': 0}, {'osd': 7, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540719139924082687, 'old_weight': 65536, 'last_purged_snaps_scrub': '2024-09-18T08:14:06.213584+0000', 'dead_epoch': 0}], 'pg_upmap': [{'pgid': '6.3', 'osds': [5, 3, 2]}], 'pg_upmap_items': [{'pgid': '4.7', 'mappings': [{'from': 1, 'to': 2}]}, {'pgid': '4.14', 'mappings': [{'from': 1, 'to': 2}]}], 'pg_upmap_primaries': [], 'pg_temp': [], 'primary_temp': [], 'blocklist': {'172.21.15.7:0/398367658': '2024-09-19T08:03:17.052903+0000', '172.21.15.7:6800/2918496291': '2024-09-19T07:57:31.015269+0000', '172.21.15.7:0/2514354773': '2024-09-19T07:57:31.015269+0000', '172.21.15.7:0/3924911335': '2024-09-19T07:57:31.015269+0000', '172.21.15.7:0/1668598084': '2024-09-19T07:57:57.346149+0000', '172.21.15.7:0/4272027292': '2024-09-19T07:57:31.015269+0000', '172.21.15.7:6800/1013061071': '2024-09-19T07:57:57.346149+0000', '172.21.15.7:6800/1491061076': '2024-09-19T08:03:17.052903+0000', '172.21.15.7:0/533727535': '2024-09-19T07:57:57.346149+0000', '172.21.15.7:0/1605217610': '2024-09-19T08:03:17.052903+0000', '172.21.15.7:0/1722968065': '2024-09-19T08:03:17.052903+0000', '172.21.15.7:0/1277549846': '2024-09-19T07:57:57.346149+0000', '172.21.15.7:0/232452890': '2024-09-19T08:03:17.052903+0000', '172.21.15.7:0/3881670470': '2024-09-19T08:03:17.052903+0000'}, 'range_blocklist': {}, 'erasure_code_profiles': {'default': {'crush-failure-domain': 'osd', 'k': '2', 'm': '1', 'plugin': 'jerasure', 'technique': 'reed_sol_van'}}, 'removed_snaps_queue': [], 'new_removed_snaps': [], 'new_purged_snaps': [], 'crush_node_flags': {}, 'device_class_flags': {}, 'stretch_mode': {'stretch_mode_enabled': False, 'stretch_bucket_count': 0, 'degraded_stretch_mode': 0, 'recovering_stretch_mode': 0, 'stretch_mode_bucket': 0}} 2024-09-18T08:15:08.386 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid 779349ca-7593-11ef-bceb-c7b262605968 -- ceph pg dump --format=json 2024-09-18T08:15:08.472 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6 2024-09-18T08:15:08.499 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:15:08.500 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:15:08.500 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6' 2024-09-18T08:15:08.601 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4 2024-09-18T08:15:08.627 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:15:08.628 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:15:08.629 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4' 2024-09-18T08:15:08.729 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4 2024-09-18T08:15:08.764 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:15:08.766 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:15:08.766 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4' 2024-09-18T08:15:08.782 DEBUG:teuthology.orchestra.run.smithi050:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid 779349ca-7593-11ef-bceb-c7b262605968 -- ceph daemon osd.7 dump_ops_in_flight 2024-09-18T08:15:08.852 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:15:08 smithi050 ceph-mon[32371]: purged_snaps scrub starts 2024-09-18T08:15:08.852 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:15:08 smithi050 ceph-mon[32371]: purged_snaps scrub ok 2024-09-18T08:15:08.852 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:15:08 smithi050 ceph-mon[32371]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-18T08:15:08.852 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:15:08 smithi050 ceph-mon[32371]: Cluster is now healthy 2024-09-18T08:15:08.853 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:15:08 smithi050 ceph-mon[32371]: from='client.? 172.21.15.7:0/205270132' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-18T08:15:08.853 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:15:08 smithi050 ceph-mon[32371]: osdmap e326: 8 total, 8 up, 6 in 2024-09-18T08:15:08.853 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:15:08 smithi050 ceph-mon[32371]: 5.16 scrub starts 2024-09-18T08:15:08.853 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:15:08 smithi050 ceph-mon[32371]: 5.16 scrub ok 2024-09-18T08:15:08.867 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2 2024-09-18T08:15:08.903 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:15:08.903 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:15:08.904 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2' 2024-09-18T08:15:08.980 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:15:08 smithi007 ceph-mon[29459]: purged_snaps scrub starts 2024-09-18T08:15:08.980 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:15:08 smithi007 ceph-mon[29459]: purged_snaps scrub ok 2024-09-18T08:15:08.980 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:15:08 smithi007 ceph-mon[29459]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-18T08:15:08.980 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:15:08 smithi007 ceph-mon[29459]: Cluster is now healthy 2024-09-18T08:15:08.981 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:15:08 smithi007 ceph-mon[29459]: from='client.? 172.21.15.7:0/205270132' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-18T08:15:08.981 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:15:08 smithi007 ceph-mon[29459]: osdmap e326: 8 total, 8 up, 6 in 2024-09-18T08:15:08.981 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:15:08 smithi007 ceph-mon[29459]: 5.16 scrub starts 2024-09-18T08:15:08.981 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:15:08 smithi007 ceph-mon[29459]: 5.16 scrub ok 2024-09-18T08:15:08.981 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:15:08 smithi007 ceph-mon[39249]: purged_snaps scrub starts 2024-09-18T08:15:08.981 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:15:08 smithi007 ceph-mon[39249]: purged_snaps scrub ok 2024-09-18T08:15:08.981 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:15:08 smithi007 ceph-mon[39249]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-18T08:15:08.981 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:15:08 smithi007 ceph-mon[39249]: Cluster is now healthy 2024-09-18T08:15:08.982 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:15:08 smithi007 ceph-mon[39249]: from='client.? 172.21.15.7:0/205270132' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-18T08:15:08.982 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:15:08 smithi007 ceph-mon[39249]: osdmap e326: 8 total, 8 up, 6 in 2024-09-18T08:15:08.982 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:15:08 smithi007 ceph-mon[39249]: 5.16 scrub starts 2024-09-18T08:15:08.982 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:15:08 smithi007 ceph-mon[39249]: 5.16 scrub ok 2024-09-18T08:15:09.004 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2 2024-09-18T08:15:09.047 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:15:09.047 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:15:09.048 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2' 2024-09-18T08:15:09.149 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3 2024-09-18T08:15:09.166 INFO:teuthology.orchestra.run.smithi050.stderr:Inferring config /var/lib/ceph/779349ca-7593-11ef-bceb-c7b262605968/mon.b/config 2024-09-18T08:15:09.189 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:15:09.189 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:15:09.190 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3' 2024-09-18T08:15:09.291 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0 2024-09-18T08:15:09.337 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:15:09.337 INFO:tasks.radosbench.radosbench.0.smithi007.stdout: 21 16 30922 30906 91.9658 107.812 0.00907883 0.0108663 2024-09-18T08:15:09.338 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:15:09.338 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0' 2024-09-18T08:15:09.439 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3 2024-09-18T08:15:09.479 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:15:09.480 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:15:09.481 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3' 2024-09-18T08:15:09.581 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1 2024-09-18T08:15:09.601 INFO:journalctl@ceph.iscsi.iscsi.a.smithi050.stdout:Sep 18 08:15:09 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-iscsi-iscsi-a[57782]: debug there is no tcmu-runner data available 2024-09-18T08:15:09.648 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:15:09.650 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:15:09.650 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1' 2024-09-18T08:15:09.679 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/779349ca-7593-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T08:15:09.728 INFO:teuthology.orchestra.run.smithi050.stdout:ERROR: (22) Invalid argument 2024-09-18T08:15:09.729 INFO:teuthology.orchestra.run.smithi050.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-18T08:15:09.736 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/779349ca-7593-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T08:15:09.751 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5 2024-09-18T08:15:09.782 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:15:09.782 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:15:09.783 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5' 2024-09-18T08:15:09.831 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:15:09 smithi007 ceph-mon[29459]: 3.7 scrub starts 2024-09-18T08:15:09.831 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:15:09 smithi007 ceph-mon[29459]: 3.7 scrub ok 2024-09-18T08:15:09.831 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:15:09 smithi007 ceph-mon[29459]: pgmap v613: 158 pgs: 158 active+clean; 1.8 GiB data, 7.6 GiB used, 529 GiB / 536 GiB avail; 1.2 KiB/s rd, 106 MiB/s wr, 1.69k op/s; 4.4 MiB/s, 70 objects/s recovering 2024-09-18T08:15:09.831 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:15:09 smithi007 ceph-mon[29459]: osdmap e327: 8 total, 8 up, 6 in 2024-09-18T08:15:09.832 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:15:09 smithi007 ceph-mon[29459]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-18T08:15:09.832 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:15:09 smithi007 ceph-mon[39249]: 3.7 scrub starts 2024-09-18T08:15:09.832 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:15:09 smithi007 ceph-mon[39249]: 3.7 scrub ok 2024-09-18T08:15:09.832 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:15:09 smithi007 ceph-mon[39249]: pgmap v613: 158 pgs: 158 active+clean; 1.8 GiB data, 7.6 GiB used, 529 GiB / 536 GiB avail; 1.2 KiB/s rd, 106 MiB/s wr, 1.69k op/s; 4.4 MiB/s, 70 objects/s recovering 2024-09-18T08:15:09.832 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:15:09 smithi007 ceph-mon[39249]: osdmap e327: 8 total, 8 up, 6 in 2024-09-18T08:15:09.832 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:15:09 smithi007 ceph-mon[39249]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-18T08:15:09.884 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5 2024-09-18T08:15:09.905 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:15:09 smithi050 ceph-mon[32371]: 3.7 scrub starts 2024-09-18T08:15:09.905 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:15:09 smithi050 ceph-mon[32371]: 3.7 scrub ok 2024-09-18T08:15:09.905 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:15:09 smithi050 ceph-mon[32371]: pgmap v613: 158 pgs: 158 active+clean; 1.8 GiB data, 7.6 GiB used, 529 GiB / 536 GiB avail; 1.2 KiB/s rd, 106 MiB/s wr, 1.69k op/s; 4.4 MiB/s, 70 objects/s recovering 2024-09-18T08:15:09.905 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:15:09 smithi050 ceph-mon[32371]: osdmap e327: 8 total, 8 up, 6 in 2024-09-18T08:15:09.906 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:15:09 smithi050 ceph-mon[32371]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-18T08:15:09.906 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:15:09.906 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:15:09.907 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5' 2024-09-18T08:15:10.007 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1 2024-09-18T08:15:10.044 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:15:10.044 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:15:10.045 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1' 2024-09-18T08:15:10.146 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7 2024-09-18T08:15:10.174 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:15:10.174 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:15:10.175 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7' 2024-09-18T08:15:10.275 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5 2024-09-18T08:15:10.303 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:15:10.304 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:15:10.305 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5' 2024-09-18T08:15:10.337 INFO:tasks.radosbench.radosbench.0.smithi007.stdout: 22 16 32594 32578 92.5351 104.5 0.00365907 0.0108006 2024-09-18T08:15:10.406 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2 2024-09-18T08:15:10.450 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:15:10.451 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:15:10.451 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2' 2024-09-18T08:15:10.491 DEBUG:teuthology.orchestra.run.smithi050:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid 779349ca-7593-11ef-bceb-c7b262605968 -- ceph daemon osd.7 dump_blocked_ops 2024-09-18T08:15:10.552 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5 2024-09-18T08:15:10.585 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:15:10.586 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:15:10.586 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5' 2024-09-18T08:15:10.687 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1 2024-09-18T08:15:10.726 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:15:10.726 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:15:10.727 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1' 2024-09-18T08:15:10.748 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:15:10 smithi007 ceph-mon[29459]: 3.a scrub starts 2024-09-18T08:15:10.748 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:15:10 smithi007 ceph-mon[29459]: 3.a scrub ok 2024-09-18T08:15:10.748 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:15:10 smithi007 ceph-mon[29459]: 4.6 scrub starts 2024-09-18T08:15:10.748 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:15:10 smithi007 ceph-mon[29459]: 4.6 scrub ok 2024-09-18T08:15:10.748 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:15:10 smithi007 ceph-mon[39249]: 3.a scrub starts 2024-09-18T08:15:10.748 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:15:10 smithi007 ceph-mon[39249]: 3.a scrub ok 2024-09-18T08:15:10.748 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:15:10 smithi007 ceph-mon[39249]: 4.6 scrub starts 2024-09-18T08:15:10.749 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:15:10 smithi007 ceph-mon[39249]: 4.6 scrub ok 2024-09-18T08:15:10.827 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3 2024-09-18T08:15:10.840 INFO:teuthology.orchestra.run.smithi050.stderr:Inferring config /var/lib/ceph/779349ca-7593-11ef-bceb-c7b262605968/mon.b/config 2024-09-18T08:15:10.869 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:15:10.869 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:15:10.870 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3' 2024-09-18T08:15:10.971 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1 2024-09-18T08:15:10.990 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:15:10 smithi050 ceph-mon[32371]: 3.a scrub starts 2024-09-18T08:15:10.990 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:15:10 smithi050 ceph-mon[32371]: 3.a scrub ok 2024-09-18T08:15:10.990 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:15:10 smithi050 ceph-mon[32371]: 4.6 scrub starts 2024-09-18T08:15:10.990 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:15:10 smithi050 ceph-mon[32371]: 4.6 scrub ok 2024-09-18T08:15:11.021 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:15:11.021 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:15:11.022 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1' 2024-09-18T08:15:11.094 INFO:teuthology.orchestra.run.smithi007.stdout:osd.0: {} 2024-09-18T08:15:11.094 INFO:teuthology.orchestra.run.smithi007.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-18T08:15:11.122 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5 2024-09-18T08:15:11.139 INFO:teuthology.orchestra.run.smithi007.stdout:osd.1: {} 2024-09-18T08:15:11.140 INFO:teuthology.orchestra.run.smithi007.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-18T08:15:11.159 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:15:11.160 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:15:11.160 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5' 2024-09-18T08:15:11.173 INFO:teuthology.orchestra.run.smithi007.stdout:osd.2: {} 2024-09-18T08:15:11.173 INFO:teuthology.orchestra.run.smithi007.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-18T08:15:11.193 INFO:teuthology.orchestra.run.smithi007.stdout:osd.3: {} 2024-09-18T08:15:11.193 INFO:teuthology.orchestra.run.smithi007.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-18T08:15:11.229 INFO:teuthology.orchestra.run.smithi007.stdout:osd.4: {} 2024-09-18T08:15:11.229 INFO:teuthology.orchestra.run.smithi007.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-18T08:15:11.261 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0 2024-09-18T08:15:11.265 INFO:teuthology.orchestra.run.smithi007.stdout:osd.5: {} 2024-09-18T08:15:11.265 INFO:teuthology.orchestra.run.smithi007.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-18T08:15:11.303 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:15:11.304 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:15:11.304 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0' 2024-09-18T08:15:11.305 INFO:teuthology.orchestra.run.smithi007.stdout:osd.6: {} 2024-09-18T08:15:11.305 INFO:teuthology.orchestra.run.smithi007.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-18T08:15:11.337 INFO:tasks.radosbench.radosbench.0.smithi007.stdout: 23 16 34253 34237 93.0196 103.688 0.00793374 0.0107423 2024-09-18T08:15:11.338 INFO:teuthology.orchestra.run.smithi007.stdout:osd.7: {} 2024-09-18T08:15:11.338 INFO:teuthology.orchestra.run.smithi007.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-18T08:15:11.393 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2024-09-18T08:15:11.393 INFO:teuthology.orchestra.run.smithi050.stdout: "ops": [], 2024-09-18T08:15:11.393 INFO:teuthology.orchestra.run.smithi050.stdout: "complaint_time": 30, 2024-09-18T08:15:11.393 INFO:teuthology.orchestra.run.smithi050.stdout: "num_blocked_ops": 0 2024-09-18T08:15:11.393 INFO:teuthology.orchestra.run.smithi050.stdout:} 2024-09-18T08:15:11.405 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6 2024-09-18T08:15:11.437 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:15:11.438 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:15:11.438 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6' 2024-09-18T08:15:11.539 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4 2024-09-18T08:15:11.565 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:15:11.566 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:15:11.566 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4' 2024-09-18T08:15:11.617 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid 779349ca-7593-11ef-bceb-c7b262605968 -- ceph osd set noscrub 2024-09-18T08:15:11.667 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0 2024-09-18T08:15:11.732 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:15:11.732 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:15:11.733 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0' 2024-09-18T08:15:11.833 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1 2024-09-18T08:15:11.883 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:15:11.884 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:15:11.884 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1' 2024-09-18T08:15:11.927 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:15:11 smithi007 ceph-mon[39249]: 5.b scrub starts 2024-09-18T08:15:11.927 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:15:11 smithi007 ceph-mon[39249]: 5.b scrub ok 2024-09-18T08:15:11.928 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:15:11 smithi007 ceph-mon[39249]: from='client.? 172.21.15.7:0/3220718208' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-18T08:15:11.928 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:15:11 smithi007 ceph-mon[39249]: pgmap v614: 158 pgs: 158 active+clean; 1.8 GiB data, 7.6 GiB used, 529 GiB / 536 GiB avail; 639 B/s rd, 99 MiB/s wr, 1.58k op/s; 4.4 MiB/s, 70 objects/s recovering 2024-09-18T08:15:11.928 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:15:11 smithi007 ceph-mon[29459]: 5.b scrub starts 2024-09-18T08:15:11.928 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:15:11 smithi007 ceph-mon[29459]: 5.b scrub ok 2024-09-18T08:15:11.928 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:15:11 smithi007 ceph-mon[29459]: from='client.? 172.21.15.7:0/3220718208' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-18T08:15:11.928 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:15:11 smithi007 ceph-mon[29459]: pgmap v614: 158 pgs: 158 active+clean; 1.8 GiB data, 7.6 GiB used, 529 GiB / 536 GiB avail; 639 B/s rd, 99 MiB/s wr, 1.58k op/s; 4.4 MiB/s, 70 objects/s recovering 2024-09-18T08:15:11.986 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6 2024-09-18T08:15:11.993 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-09-18T08:15:11.996 INFO:teuthology.orchestra.run.smithi007.stderr:dumped all 2024-09-18T08:15:11.997 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:15:11 smithi050 ceph-mon[32371]: 5.b scrub starts 2024-09-18T08:15:11.997 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:15:11 smithi050 ceph-mon[32371]: 5.b scrub ok 2024-09-18T08:15:11.997 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:15:11 smithi050 ceph-mon[32371]: from='client.? 172.21.15.7:0/3220718208' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-18T08:15:11.997 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:15:11 smithi050 ceph-mon[32371]: pgmap v614: 158 pgs: 158 active+clean; 1.8 GiB data, 7.6 GiB used, 529 GiB / 536 GiB avail; 639 B/s rd, 99 MiB/s wr, 1.58k op/s; 4.4 MiB/s, 70 objects/s recovering 2024-09-18T08:15:12.014 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:15:12.014 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:15:12.015 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6' 2024-09-18T08:15:12.073 DEBUG:teuthology.orchestra.run.smithi050:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid 779349ca-7593-11ef-bceb-c7b262605968 -- ceph daemon osd.7 dump_historic_ops 2024-09-18T08:15:12.116 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0 2024-09-18T08:15:12.159 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:15:12.160 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:15:12.160 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0' 2024-09-18T08:15:12.261 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1 2024-09-18T08:15:12.313 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:15:12.314 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:15:12.314 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1' 2024-09-18T08:15:12.338 INFO:tasks.radosbench.radosbench.0.smithi007.stdout: 24 16 35792 35776 93.148 96.1875 0.00606909 0.0107297 2024-09-18T08:15:12.416 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7 2024-09-18T08:15:12.426 INFO:teuthology.orchestra.run.smithi050.stderr:Inferring config /var/lib/ceph/779349ca-7593-11ef-bceb-c7b262605968/mon.b/config 2024-09-18T08:15:12.451 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:15:12.452 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:15:12.453 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7' 2024-09-18T08:15:12.554 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6 2024-09-18T08:15:12.583 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:15:12.584 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:15:12.584 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6' 2024-09-18T08:15:12.685 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0 2024-09-18T08:15:12.725 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:15:12.726 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:15:12.726 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0' 2024-09-18T08:15:12.827 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6 2024-09-18T08:15:12.836 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:15:12 smithi050 ceph-mon[32371]: from='client.25672 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-18T08:15:12.836 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:15:12 smithi050 ceph-mon[32371]: 5.19 scrub starts 2024-09-18T08:15:12.836 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:15:12 smithi050 ceph-mon[32371]: 5.19 scrub ok 2024-09-18T08:15:12.859 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:15:12.860 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:15:12.861 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6' 2024-09-18T08:15:12.961 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7 2024-09-18T08:15:12.965 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:15:12 smithi007 ceph-mon[29459]: from='client.25672 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-18T08:15:12.965 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:15:12 smithi007 ceph-mon[29459]: 5.19 scrub starts 2024-09-18T08:15:12.965 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:15:12 smithi007 ceph-mon[29459]: 5.19 scrub ok 2024-09-18T08:15:12.965 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:15:12 smithi007 ceph-mon[39249]: from='client.25672 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-18T08:15:12.966 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:15:12 smithi007 ceph-mon[39249]: 5.19 scrub starts 2024-09-18T08:15:12.966 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:15:12 smithi007 ceph-mon[39249]: 5.19 scrub ok 2024-09-18T08:15:12.982 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2024-09-18T08:15:12.982 INFO:teuthology.orchestra.run.smithi050.stdout: "size": 20, 2024-09-18T08:15:12.982 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 600, 2024-09-18T08:15:12.982 INFO:teuthology.orchestra.run.smithi050.stdout: "ops": [ 2024-09-18T08:15:12.982 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:15:12.982 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.14805.0:141882 7.5 7:a19796b6:::benchmark_data_smithi007_73155_object23591:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e127)", 2024-09-18T08:15:12.982 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-09-18T08:05:59.085169+0000", 2024-09-18T08:15:12.983 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 553.89396352599999, 2024-09-18T08:15:12.983 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.091421107000000001, 2024-09-18T08:15:12.983 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-09-18T08:15:12.983 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-09-18T08:15:12.983 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T08:15:12.983 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2024-09-18T08:15:12.983 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.14805", 2024-09-18T08:15:12.983 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "172.21.15.7:0/640320436", 2024-09-18T08:15:12.983 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 141882 2024-09-18T08:15:12.983 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:15:12.983 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-09-18T08:15:12.983 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:15:12.983 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-09-18T08:15:12.984 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:05:59.085169+0000", 2024-09-18T08:15:12.984 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-18T08:15:12.984 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:15:12.984 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:15:12.984 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-09-18T08:15:12.984 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:05:59.085169+0000", 2024-09-18T08:15:12.984 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-18T08:15:12.984 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:15:12.984 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:15:12.984 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-09-18T08:15:12.984 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:05:59.085171+0000", 2024-09-18T08:15:12.984 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.7179999999999999e-06 2024-09-18T08:15:12.984 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:15:12.985 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:15:12.985 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-09-18T08:15:12.985 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:05:59.085177+0000", 2024-09-18T08:15:12.985 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 6.1310000000000003e-06 2024-09-18T08:15:12.985 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:15:12.985 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:15:12.985 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-09-18T08:15:12.985 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:05:59.085190+0000", 2024-09-18T08:15:12.985 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.2651e-05 2024-09-18T08:15:12.985 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:15:12.985 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:15:12.986 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-09-18T08:15:12.986 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:05:59.085195+0000", 2024-09-18T08:15:12.986 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 5.6890000000000001e-06 2024-09-18T08:15:12.986 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:15:12.986 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:15:12.986 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-18T08:15:12.986 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:05:59.085256+0000", 2024-09-18T08:15:12.986 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 6.0435e-05 2024-09-18T08:15:12.986 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:15:12.986 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:15:12.986 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-09-18T08:15:12.986 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:05:59.085378+0000", 2024-09-18T08:15:12.987 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00012267699999999999 2024-09-18T08:15:12.987 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:15:12.987 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:15:12.987 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "waiting for subops from 0,2", 2024-09-18T08:15:12.987 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:05:59.085502+0000", 2024-09-18T08:15:12.987 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00012316599999999999 2024-09-18T08:15:12.987 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:15:12.987 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:15:12.987 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "op_commit", 2024-09-18T08:15:12.987 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:05:59.089418+0000", 2024-09-18T08:15:12.987 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.003916597 2024-09-18T08:15:12.987 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:15:12.987 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:15:12.988 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:15:12.988 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:05:59.091324+0000", 2024-09-18T08:15:12.988 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0019060909999999999 2024-09-18T08:15:12.988 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:15:12.988 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:15:12.988 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:15:12.988 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:05:59.176490+0000", 2024-09-18T08:15:12.988 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.085165442999999993 2024-09-18T08:15:12.988 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:15:12.988 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:15:12.988 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "commit_sent", 2024-09-18T08:15:12.989 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:05:59.176544+0000", 2024-09-18T08:15:12.989 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 5.3730000000000002e-05 2024-09-18T08:15:12.989 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:15:12.989 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:15:12.989 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-09-18T08:15:12.989 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:05:59.176590+0000", 2024-09-18T08:15:12.989 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4.6779000000000001e-05 2024-09-18T08:15:12.989 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-18T08:15:12.989 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-09-18T08:15:12.989 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-18T08:15:12.989 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:15:12.989 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:15:12.989 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.14805.0:141883 7.5 7:a296541a:::benchmark_data_smithi007_73155_object23592:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e127)", 2024-09-18T08:15:12.990 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-09-18T08:05:59.086449+0000", 2024-09-18T08:15:12.990 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 553.892683799, 2024-09-18T08:15:12.990 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.095639653000000005, 2024-09-18T08:15:12.990 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-09-18T08:15:12.990 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-09-18T08:15:12.990 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T08:15:12.990 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2024-09-18T08:15:12.990 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.14805", 2024-09-18T08:15:12.990 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "172.21.15.7:0/640320436", 2024-09-18T08:15:12.990 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 141883 2024-09-18T08:15:12.990 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:15:12.990 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-09-18T08:15:12.990 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:15:12.990 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-09-18T08:15:12.991 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:05:59.086449+0000", 2024-09-18T08:15:12.991 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-18T08:15:12.991 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:15:12.991 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:15:12.991 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-09-18T08:15:12.991 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:05:59.086449+0000", 2024-09-18T08:15:12.991 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-18T08:15:12.991 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:15:12.991 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:15:12.991 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-09-18T08:15:12.991 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:05:59.086450+0000", 2024-09-18T08:15:12.991 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 6.8999999999999996e-07 2024-09-18T08:15:12.991 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:15:12.991 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:15:12.992 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-09-18T08:15:12.992 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:05:59.086456+0000", 2024-09-18T08:15:12.992 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 6.5760000000000002e-06 2024-09-18T08:15:12.992 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:15:12.992 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:15:12.992 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-09-18T08:15:12.992 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:05:59.086466+0000", 2024-09-18T08:15:12.992 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 9.645e-06 2024-09-18T08:15:12.992 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:15:12.992 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:15:12.992 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-09-18T08:15:12.992 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:05:59.086471+0000", 2024-09-18T08:15:12.992 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 5.2950000000000001e-06 2024-09-18T08:15:12.993 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:15:12.993 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:15:12.993 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-18T08:15:12.993 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:05:59.087436+0000", 2024-09-18T08:15:12.993 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00096524000000000002 2024-09-18T08:15:12.993 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:15:12.993 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:15:12.993 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-09-18T08:15:12.993 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:05:59.087568+0000", 2024-09-18T08:15:12.993 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000131426 2024-09-18T08:15:12.993 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:15:12.993 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:15:12.994 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "waiting for subops from 0,2", 2024-09-18T08:15:12.994 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:05:59.087696+0000", 2024-09-18T08:15:12.994 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000128238 2024-09-18T08:15:12.994 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:15:12.994 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:15:12.994 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "op_commit", 2024-09-18T08:15:12.994 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:05:59.089499+0000", 2024-09-18T08:15:12.994 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.001803453 2024-09-18T08:15:12.994 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:15:12.994 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:15:12.994 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:15:12.994 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:05:59.091350+0000", 2024-09-18T08:15:12.994 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.001850361 2024-09-18T08:15:12.994 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:15:12.995 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:15:12.995 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:15:12.995 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:05:59.181978+0000", 2024-09-18T08:15:12.995 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.090627842 2024-09-18T08:15:12.995 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:15:12.995 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:15:12.995 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "commit_sent", 2024-09-18T08:15:12.995 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:05:59.182038+0000", 2024-09-18T08:15:12.995 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 6.0486999999999999e-05 2024-09-18T08:15:12.995 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:15:12.995 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:15:12.995 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-09-18T08:15:12.996 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:05:59.182089+0000", 2024-09-18T08:15:12.996 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 5.0399999999999999e-05 2024-09-18T08:15:12.996 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-18T08:15:12.996 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-09-18T08:15:12.996 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-18T08:15:12.996 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:15:12.996 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:15:12.996 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.14805.0:141892 7.5 7:ac271655:::benchmark_data_smithi007_73155_object23601:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e127)", 2024-09-18T08:15:12.996 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-09-18T08:05:59.091521+0000", 2024-09-18T08:15:12.996 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 553.88761168300005, 2024-09-18T08:15:12.996 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.10324135199999999, 2024-09-18T08:15:12.996 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-09-18T08:15:12.996 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-09-18T08:15:12.997 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T08:15:12.997 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2024-09-18T08:15:12.997 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.14805", 2024-09-18T08:15:12.997 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "172.21.15.7:0/640320436", 2024-09-18T08:15:12.997 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 141892 2024-09-18T08:15:12.997 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:15:12.998 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-09-18T08:15:12.998 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:15:12.998 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-09-18T08:15:12.998 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:05:59.091521+0000", 2024-09-18T08:15:12.998 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-18T08:15:12.998 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:15:12.998 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:15:12.998 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-09-18T08:15:12.998 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:05:59.091521+0000", 2024-09-18T08:15:12.998 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-18T08:15:12.998 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:15:12.998 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:15:12.999 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-09-18T08:15:12.999 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:05:59.091522+0000", 2024-09-18T08:15:12.999 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.4479999999999999e-06 2024-09-18T08:15:12.999 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:15:12.999 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:15:12.999 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-09-18T08:15:12.999 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:05:59.091527+0000", 2024-09-18T08:15:12.999 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4.0520000000000003e-06 2024-09-18T08:15:12.999 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:15:12.999 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:15:12.999 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-09-18T08:15:12.999 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:05:59.091538+0000", 2024-09-18T08:15:13.000 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.1173e-05 2024-09-18T08:15:13.000 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:15:13.000 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:15:13.000 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-09-18T08:15:13.000 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:05:59.091545+0000", 2024-09-18T08:15:13.000 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 7.0650000000000001e-06 2024-09-18T08:15:13.000 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:15:13.000 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:15:13.000 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-18T08:15:13.000 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:05:59.091589+0000", 2024-09-18T08:15:13.000 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4.4258999999999998e-05 2024-09-18T08:15:13.000 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:15:13.000 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:15:13.001 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-09-18T08:15:13.001 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:05:59.091689+0000", 2024-09-18T08:15:13.001 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 9.9739000000000002e-05 2024-09-18T08:15:13.001 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:15:13.001 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:15:13.001 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "waiting for subops from 0,2", 2024-09-18T08:15:13.001 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:05:59.091795+0000", 2024-09-18T08:15:13.001 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000106721 2024-09-18T08:15:13.001 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:15:13.001 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:15:13.001 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "op_commit", 2024-09-18T08:15:13.001 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:05:59.093876+0000", 2024-09-18T08:15:13.001 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0020808990000000002 2024-09-18T08:15:13.002 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:15:13.002 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:15:13.002 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:15:13.002 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:05:59.094173+0000", 2024-09-18T08:15:13.002 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000296569 2024-09-18T08:15:13.002 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:15:13.003 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:15:13.003 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:15:13.003 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:05:59.194693+0000", 2024-09-18T08:15:13.003 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.10051969600000001 2024-09-18T08:15:13.003 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:15:13.003 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:15:13.003 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "commit_sent", 2024-09-18T08:15:13.003 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:05:59.194729+0000", 2024-09-18T08:15:13.003 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 3.6541000000000001e-05 2024-09-18T08:15:13.003 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:15:13.003 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:15:13.003 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-09-18T08:15:13.004 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:05:59.194762+0000", 2024-09-18T08:15:13.004 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 3.3189999999999999e-05 2024-09-18T08:15:13.004 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-18T08:15:13.004 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-09-18T08:15:13.004 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-18T08:15:13.004 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:15:13.004 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:15:13.004 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.15216.0:20344 8.5 8:a9772924:::benchmark_data_smithi007_101583_object20343:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e174)", 2024-09-18T08:15:13.004 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-09-18T08:08:07.241675+0000", 2024-09-18T08:15:13.004 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 425.73745767000003, 2024-09-18T08:15:13.004 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.104822005, 2024-09-18T08:15:13.004 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-09-18T08:15:13.005 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-09-18T08:15:13.005 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T08:15:13.005 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2024-09-18T08:15:13.005 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.15216", 2024-09-18T08:15:13.005 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "172.21.15.7:0/2596041177", 2024-09-18T08:15:13.005 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 20344 2024-09-18T08:15:13.005 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:15:13.005 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-09-18T08:15:13.005 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:15:13.005 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-09-18T08:15:13.005 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:08:07.241675+0000", 2024-09-18T08:15:13.005 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-18T08:15:13.006 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:15:13.006 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:15:13.006 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-09-18T08:15:13.006 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:08:07.241675+0000", 2024-09-18T08:15:13.006 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-18T08:15:13.006 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:15:13.006 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:15:13.006 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-09-18T08:15:13.006 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:08:07.241676+0000", 2024-09-18T08:15:13.006 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.423e-06 2024-09-18T08:15:13.006 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:15:13.006 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:15:13.006 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-09-18T08:15:13.007 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:08:07.241706+0000", 2024-09-18T08:15:13.007 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 3.0035e-05 2024-09-18T08:15:13.007 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:15:13.007 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:15:13.007 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-09-18T08:15:13.007 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:08:07.241716+0000", 2024-09-18T08:15:13.007 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 9.7720000000000006e-06 2024-09-18T08:15:13.007 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:15:13.007 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:15:13.007 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-09-18T08:15:13.007 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:08:07.241723+0000", 2024-09-18T08:15:13.007 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 7.199e-06 2024-09-18T08:15:13.007 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:15:13.008 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:15:13.008 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-18T08:15:13.008 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:08:07.241863+0000", 2024-09-18T08:15:13.008 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000139068 2024-09-18T08:15:13.008 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:15:13.008 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:15:13.008 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-09-18T08:15:13.008 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:08:07.241991+0000", 2024-09-18T08:15:13.008 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000128677 2024-09-18T08:15:13.008 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:15:13.008 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:15:13.008 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "waiting for subops from 0,6", 2024-09-18T08:15:13.008 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:08:07.242170+0000", 2024-09-18T08:15:13.009 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000179125 2024-09-18T08:15:13.009 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:15:13.009 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:15:13.009 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "op_commit", 2024-09-18T08:15:13.009 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:08:07.243297+0000", 2024-09-18T08:15:13.009 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.001126305 2024-09-18T08:15:13.009 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:15:13.009 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:15:13.009 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:15:13.009 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:08:07.243395+0000", 2024-09-18T08:15:13.009 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 9.8490000000000001e-05 2024-09-18T08:15:13.009 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:15:13.009 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:15:13.010 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:15:13.010 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:08:07.346408+0000", 2024-09-18T08:15:13.010 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.103012803 2024-09-18T08:15:13.010 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:15:13.010 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:15:13.010 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "commit_sent", 2024-09-18T08:15:13.010 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:08:07.346446+0000", 2024-09-18T08:15:13.010 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 3.7908999999999999e-05 2024-09-18T08:15:13.010 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:15:13.010 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:15:13.010 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-09-18T08:15:13.010 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:08:07.346497+0000", 2024-09-18T08:15:13.010 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 5.1199000000000002e-05 2024-09-18T08:15:13.010 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-18T08:15:13.011 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-09-18T08:15:13.011 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-18T08:15:13.011 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:15:13.011 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:15:13.011 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.15582.0:53567 9.3 9:ce224b9e:::benchmark_data_smithi007_124019_object53566:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e243)", 2024-09-18T08:15:13.011 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-09-18T08:11:41.311629+0000", 2024-09-18T08:15:13.011 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 211.66750371699999, 2024-09-18T08:15:13.011 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.0246146169999999, 2024-09-18T08:15:13.012 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-09-18T08:15:13.012 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-09-18T08:15:13.012 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "reached pg", 2024-09-18T08:15:13.012 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2024-09-18T08:15:13.012 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.15582", 2024-09-18T08:15:13.012 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "172.21.15.7:0/3598605839", 2024-09-18T08:15:13.012 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 53567 2024-09-18T08:15:13.012 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:15:13.012 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-09-18T08:15:13.013 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:15:13.013 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-09-18T08:15:13.013 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:11:41.311629+0000", 2024-09-18T08:15:13.013 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-18T08:15:13.013 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:15:13.013 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:15:13.013 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-09-18T08:15:13.013 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:11:41.311629+0000", 2024-09-18T08:15:13.013 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-18T08:15:13.013 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:15:13.013 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:15:13.013 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-09-18T08:15:13.013 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:11:41.311631+0000", 2024-09-18T08:15:13.014 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.9929999999999998e-06 2024-09-18T08:15:13.014 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:15:13.014 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:15:13.014 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-09-18T08:15:13.014 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:11:41.311694+0000", 2024-09-18T08:15:13.014 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 6.3456999999999999e-05 2024-09-18T08:15:13.014 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:15:13.014 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:15:13.014 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-09-18T08:15:13.014 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:11:41.311711+0000", 2024-09-18T08:15:13.014 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.6816e-05 2024-09-18T08:15:13.014 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:15:13.014 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:15:13.015 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-09-18T08:15:13.015 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:11:41.311721+0000", 2024-09-18T08:15:13.015 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 9.7259999999999997e-06 2024-09-18T08:15:13.015 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:15:13.015 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:15:13.015 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-18T08:15:13.015 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:11:41.311783+0000", 2024-09-18T08:15:13.015 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 6.1737999999999997e-05 2024-09-18T08:15:13.015 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:15:13.015 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:15:13.015 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-09-18T08:15:13.015 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:11:41.311938+0000", 2024-09-18T08:15:13.015 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000155291 2024-09-18T08:15:13.016 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:15:13.016 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:15:13.016 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "waiting for subops from 1,3", 2024-09-18T08:15:13.016 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:11:41.312026+0000", 2024-09-18T08:15:13.016 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 8.8227999999999996e-05 2024-09-18T08:15:13.016 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:15:13.016 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:15:13.016 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "op_commit", 2024-09-18T08:15:13.016 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:11:41.313766+0000", 2024-09-18T08:15:13.016 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0017395819999999999 2024-09-18T08:15:13.016 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:15:13.016 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:15:13.016 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:15:13.017 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:11:41.320080+0000", 2024-09-18T08:15:13.017 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0063146590000000002 2024-09-18T08:15:13.017 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:15:13.017 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:15:13.017 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-18T08:15:13.017 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:11:41.329868+0000", 2024-09-18T08:15:13.017 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0097875719999999996 2024-09-18T08:15:13.017 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:15:13.017 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:15:13.017 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "waiting for peered", 2024-09-18T08:15:13.017 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:11:41.329881+0000", 2024-09-18T08:15:13.017 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.3154e-05 2024-09-18T08:15:13.017 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:15:13.018 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:15:13.018 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-18T08:15:13.018 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:11:42.336164+0000", 2024-09-18T08:15:13.018 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.0062827560000001 2024-09-18T08:15:13.018 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:15:13.018 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:15:13.018 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-09-18T08:15:13.018 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:11:42.336244+0000", 2024-09-18T08:15:13.018 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 7.9645000000000001e-05 2024-09-18T08:15:13.018 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-18T08:15:13.018 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-09-18T08:15:13.018 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-18T08:15:13.018 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:15:13.019 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:15:13.019 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.15582.0:53569 9.3 9:cedd1e96:::benchmark_data_smithi007_124019_object53568:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e243)", 2024-09-18T08:15:13.019 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-09-18T08:11:41.313768+0000", 2024-09-18T08:15:13.019 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 211.66536447799999, 2024-09-18T08:15:13.019 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.0232493, 2024-09-18T08:15:13.019 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-09-18T08:15:13.019 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-09-18T08:15:13.019 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "reached pg", 2024-09-18T08:15:13.019 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2024-09-18T08:15:13.019 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.15582", 2024-09-18T08:15:13.019 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "172.21.15.7:0/3598605839", 2024-09-18T08:15:13.019 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 53569 2024-09-18T08:15:13.019 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:15:13.019 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-09-18T08:15:13.020 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:15:13.020 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-09-18T08:15:13.020 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:11:41.313768+0000", 2024-09-18T08:15:13.020 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-18T08:15:13.020 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:15:13.020 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:15:13.020 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-09-18T08:15:13.020 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:11:41.313768+0000", 2024-09-18T08:15:13.020 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-18T08:15:13.020 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:15:13.020 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:15:13.020 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-09-18T08:15:13.020 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:11:41.313769+0000", 2024-09-18T08:15:13.021 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.2109999999999999e-06 2024-09-18T08:15:13.021 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:15:13.021 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:15:13.021 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-09-18T08:15:13.021 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:11:41.313829+0000", 2024-09-18T08:15:13.021 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 5.9518999999999997e-05 2024-09-18T08:15:13.021 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:15:13.021 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:15:13.021 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-09-18T08:15:13.021 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:11:41.313839+0000", 2024-09-18T08:15:13.021 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 9.7820000000000005e-06 2024-09-18T08:15:13.021 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:15:13.021 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:15:13.022 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-09-18T08:15:13.022 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:11:41.313846+0000", 2024-09-18T08:15:13.022 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 7.6410000000000002e-06 2024-09-18T08:15:13.022 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:15:13.022 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:15:13.022 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-18T08:15:13.022 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:11:41.313909+0000", 2024-09-18T08:15:13.022 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 6.3079999999999999e-05 2024-09-18T08:15:13.022 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:15:13.022 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:15:13.022 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-09-18T08:15:13.022 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:11:41.313999+0000", 2024-09-18T08:15:13.022 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 8.9446999999999999e-05 2024-09-18T08:15:13.022 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:15:13.023 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:15:13.023 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "waiting for subops from 1,3", 2024-09-18T08:15:13.023 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:11:41.314130+0000", 2024-09-18T08:15:13.023 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000130748 2024-09-18T08:15:13.023 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:15:13.023 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:15:13.023 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "op_commit", 2024-09-18T08:15:13.023 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:11:41.315241+0000", 2024-09-18T08:15:13.023 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.001111108 2024-09-18T08:15:13.023 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:15:13.023 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:15:13.023 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:15:13.023 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:11:41.320126+0000", 2024-09-18T08:15:13.023 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0048852110000000004 2024-09-18T08:15:13.024 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:15:13.024 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:15:13.024 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-18T08:15:13.024 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:11:41.329924+0000", 2024-09-18T08:15:13.024 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0097982309999999993 2024-09-18T08:15:13.024 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:15:13.024 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:15:13.024 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "waiting for peered", 2024-09-18T08:15:13.024 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:11:41.329932+0000", 2024-09-18T08:15:13.024 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 7.7670000000000003e-06 2024-09-18T08:15:13.024 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:15:13.024 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:15:13.024 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-18T08:15:13.025 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:11:42.336953+0000", 2024-09-18T08:15:13.025 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.007021433 2024-09-18T08:15:13.025 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:15:13.025 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:15:13.025 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-09-18T08:15:13.025 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:11:42.337018+0000", 2024-09-18T08:15:13.025 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 6.4121999999999994e-05 2024-09-18T08:15:13.025 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-18T08:15:13.025 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-09-18T08:15:13.025 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-18T08:15:13.025 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:15:13.025 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:15:13.025 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.15582.0:53578 9.d 9:bae76b75:::benchmark_data_smithi007_124019_object53577:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e244)", 2024-09-18T08:15:13.025 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-09-18T08:11:41.338704+0000", 2024-09-18T08:15:13.026 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 211.640429033, 2024-09-18T08:15:13.026 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.0150159940000001, 2024-09-18T08:15:13.026 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-09-18T08:15:13.026 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-09-18T08:15:13.026 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T08:15:13.026 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2024-09-18T08:15:13.026 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.15582", 2024-09-18T08:15:13.026 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "172.21.15.7:0/3598605839", 2024-09-18T08:15:13.026 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 53578 2024-09-18T08:15:13.026 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:15:13.026 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-09-18T08:15:13.026 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:15:13.026 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-09-18T08:15:13.026 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:11:41.338704+0000", 2024-09-18T08:15:13.027 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-18T08:15:13.027 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:15:13.027 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:15:13.027 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-09-18T08:15:13.027 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:11:41.338704+0000", 2024-09-18T08:15:13.027 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-18T08:15:13.027 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:15:13.027 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:15:13.027 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-09-18T08:15:13.027 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:11:41.338705+0000", 2024-09-18T08:15:13.027 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.339e-06 2024-09-18T08:15:13.027 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:15:13.027 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:15:13.027 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-09-18T08:15:13.028 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:11:41.338780+0000", 2024-09-18T08:15:13.028 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 7.5328999999999997e-05 2024-09-18T08:15:13.028 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:15:13.028 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:15:13.028 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:15:13.029 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:15:13.029 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7' 2024-09-18T08:15:13.030 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-09-18T08:15:13.030 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:11:41.338795+0000", 2024-09-18T08:15:13.030 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.4476e-05 2024-09-18T08:15:13.030 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:15:13.030 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:15:13.030 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-09-18T08:15:13.031 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:11:41.338816+0000", 2024-09-18T08:15:13.031 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 2.1205000000000001e-05 2024-09-18T08:15:13.031 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:15:13.031 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:15:13.031 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-18T08:15:13.031 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:11:41.338884+0000", 2024-09-18T08:15:13.031 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 6.8219999999999994e-05 2024-09-18T08:15:13.031 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:15:13.031 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:15:13.031 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "waiting for peered", 2024-09-18T08:15:13.031 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:11:41.338895+0000", 2024-09-18T08:15:13.031 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.0559e-05 2024-09-18T08:15:13.031 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:15:13.032 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:15:13.032 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-18T08:15:13.032 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:11:42.344107+0000", 2024-09-18T08:15:13.032 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.0052120229999999 2024-09-18T08:15:13.032 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:15:13.032 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:15:13.032 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-09-18T08:15:13.032 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:11:42.344239+0000", 2024-09-18T08:15:13.032 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000132655 2024-09-18T08:15:13.032 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:15:13.032 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:15:13.032 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "waiting for subops from 5", 2024-09-18T08:15:13.032 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:11:42.344389+0000", 2024-09-18T08:15:13.033 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00014924099999999999 2024-09-18T08:15:13.033 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:15:13.033 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:15:13.033 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:15:13.033 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:11:42.353327+0000", 2024-09-18T08:15:13.033 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0089384949999999994 2024-09-18T08:15:13.033 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:15:13.033 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:15:13.033 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "op_commit", 2024-09-18T08:15:13.033 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:11:42.353567+0000", 2024-09-18T08:15:13.033 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000240232 2024-09-18T08:15:13.033 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:15:13.033 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:15:13.034 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "commit_sent", 2024-09-18T08:15:13.034 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:11:42.353618+0000", 2024-09-18T08:15:13.034 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 5.0488999999999997e-05 2024-09-18T08:15:13.034 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:15:13.034 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:15:13.034 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-09-18T08:15:13.034 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:11:42.353720+0000", 2024-09-18T08:15:13.034 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00010173100000000001 2024-09-18T08:15:13.034 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-18T08:15:13.034 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-09-18T08:15:13.034 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-18T08:15:13.034 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:15:13.034 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:15:13.035 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.15582.0:53579 9.d 9:bbcadf3f:::benchmark_data_smithi007_124019_object53578:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e244)", 2024-09-18T08:15:13.035 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-09-18T08:11:41.355809+0000", 2024-09-18T08:15:13.035 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 211.623323236, 2024-09-18T08:15:13.035 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.99805365400000001, 2024-09-18T08:15:13.035 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-09-18T08:15:13.035 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-09-18T08:15:13.035 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T08:15:13.035 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2024-09-18T08:15:13.035 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.15582", 2024-09-18T08:15:13.035 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "172.21.15.7:0/3598605839", 2024-09-18T08:15:13.035 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 53579 2024-09-18T08:15:13.035 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:15:13.035 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-09-18T08:15:13.036 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:15:13.036 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-09-18T08:15:13.036 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:11:41.355809+0000", 2024-09-18T08:15:13.036 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-18T08:15:13.036 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:15:13.036 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:15:13.036 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-09-18T08:15:13.036 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:11:41.355809+0000", 2024-09-18T08:15:13.036 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-18T08:15:13.036 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:15:13.036 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:15:13.036 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-09-18T08:15:13.036 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:11:41.355811+0000", 2024-09-18T08:15:13.037 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.204e-06 2024-09-18T08:15:13.037 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:15:13.037 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:15:13.037 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-09-18T08:15:13.037 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:11:41.355868+0000", 2024-09-18T08:15:13.037 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 5.6900000000000001e-05 2024-09-18T08:15:13.037 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:15:13.037 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:15:13.037 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-09-18T08:15:13.037 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:11:41.355879+0000", 2024-09-18T08:15:13.037 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.1314e-05 2024-09-18T08:15:13.037 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:15:13.037 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:15:13.038 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-09-18T08:15:13.038 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:11:41.355885+0000", 2024-09-18T08:15:13.038 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 5.8420000000000001e-06 2024-09-18T08:15:13.038 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:15:13.038 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:15:13.038 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-18T08:15:13.038 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:11:41.355958+0000", 2024-09-18T08:15:13.038 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 7.3339999999999999e-05 2024-09-18T08:15:13.038 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:15:13.038 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:15:13.038 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "waiting for peered", 2024-09-18T08:15:13.038 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:11:41.355964+0000", 2024-09-18T08:15:13.038 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 5.6790000000000002e-06 2024-09-18T08:15:13.039 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:15:13.039 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:15:13.039 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-18T08:15:13.039 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:11:42.346633+0000", 2024-09-18T08:15:13.039 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.99066880899999998 2024-09-18T08:15:13.039 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:15:13.039 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:15:13.039 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-09-18T08:15:13.039 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:11:42.346957+0000", 2024-09-18T08:15:13.039 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00032403199999999998 2024-09-18T08:15:13.039 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:15:13.039 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:15:13.039 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "waiting for subops from 5", 2024-09-18T08:15:13.039 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:11:42.347119+0000", 2024-09-18T08:15:13.040 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000162812 2024-09-18T08:15:13.040 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:15:13.040 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:15:13.040 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:15:13.040 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:11:42.353508+0000", 2024-09-18T08:15:13.040 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0063887969999999999 2024-09-18T08:15:13.040 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:15:13.040 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:15:13.040 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "op_commit", 2024-09-18T08:15:13.040 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:11:42.353736+0000", 2024-09-18T08:15:13.040 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000228163 2024-09-18T08:15:13.040 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:15:13.040 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:15:13.041 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "commit_sent", 2024-09-18T08:15:13.041 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:11:42.353782+0000", 2024-09-18T08:15:13.041 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4.5800000000000002e-05 2024-09-18T08:15:13.041 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:15:13.041 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:15:13.041 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-09-18T08:15:13.041 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:11:42.353863+0000", 2024-09-18T08:15:13.041 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 8.0962000000000002e-05 2024-09-18T08:15:13.041 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-18T08:15:13.041 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-09-18T08:15:13.041 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-18T08:15:13.041 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:15:13.041 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:15:13.042 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.15582.0:118440 9.3 9:c1657e05:::benchmark_data_smithi007_124019_object118439:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e259)", 2024-09-18T08:15:13.042 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-09-18T08:12:26.956503+0000", 2024-09-18T08:15:13.042 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 166.02262937200001, 2024-09-18T08:15:13.042 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.088731318000000003, 2024-09-18T08:15:13.042 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-09-18T08:15:13.042 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-09-18T08:15:13.042 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T08:15:13.042 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2024-09-18T08:15:13.042 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.15582", 2024-09-18T08:15:13.042 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "172.21.15.7:0/3598605839", 2024-09-18T08:15:13.042 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 118440 2024-09-18T08:15:13.042 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:15:13.042 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-09-18T08:15:13.042 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:15:13.043 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-09-18T08:15:13.043 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:12:26.956503+0000", 2024-09-18T08:15:13.043 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-18T08:15:13.043 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:15:13.043 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:15:13.043 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-09-18T08:15:13.043 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:12:26.956503+0000", 2024-09-18T08:15:13.043 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-18T08:15:13.043 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:15:13.043 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:15:13.043 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-09-18T08:15:13.043 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:12:26.956505+0000", 2024-09-18T08:15:13.043 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.511e-06 2024-09-18T08:15:13.044 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:15:13.044 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:15:13.044 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-09-18T08:15:13.044 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:12:26.956541+0000", 2024-09-18T08:15:13.044 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 3.6275000000000001e-05 2024-09-18T08:15:13.044 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:15:13.044 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:15:13.044 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-09-18T08:15:13.044 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:12:26.956550+0000", 2024-09-18T08:15:13.044 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 9.0319999999999992e-06 2024-09-18T08:15:13.044 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:15:13.044 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:15:13.045 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-09-18T08:15:13.045 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:12:26.956558+0000", 2024-09-18T08:15:13.045 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 7.3819999999999998e-06 2024-09-18T08:15:13.045 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:15:13.045 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:15:13.045 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-18T08:15:13.045 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:12:26.956930+0000", 2024-09-18T08:15:13.045 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00037266799999999999 2024-09-18T08:15:13.045 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:15:13.045 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:15:13.045 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-09-18T08:15:13.045 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:12:26.957058+0000", 2024-09-18T08:15:13.046 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000127704 2024-09-18T08:15:13.046 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:15:13.046 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:15:13.046 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "waiting for subops from 3", 2024-09-18T08:15:13.046 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:12:26.957353+0000", 2024-09-18T08:15:13.046 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00029481599999999999 2024-09-18T08:15:13.046 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:15:13.046 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:15:13.046 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "op_commit", 2024-09-18T08:15:13.046 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:12:26.963508+0000", 2024-09-18T08:15:13.046 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0061552960000000002 2024-09-18T08:15:13.046 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:15:13.046 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:15:13.047 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:15:13.047 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:12:27.045102+0000", 2024-09-18T08:15:13.047 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.081594011999999994 2024-09-18T08:15:13.047 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:15:13.047 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:15:13.047 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "commit_sent", 2024-09-18T08:15:13.047 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:12:27.045152+0000", 2024-09-18T08:15:13.047 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 5.0000000000000002e-05 2024-09-18T08:15:13.047 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:15:13.047 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:15:13.047 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-09-18T08:15:13.047 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:12:27.045235+0000", 2024-09-18T08:15:13.047 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 8.2621999999999996e-05 2024-09-18T08:15:13.048 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-18T08:15:13.048 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-09-18T08:15:13.048 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-18T08:15:13.048 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:15:13.048 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:15:13.048 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.15582.0:118446 9.3 9:cfd3961a:::benchmark_data_smithi007_124019_object118445:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e259)", 2024-09-18T08:15:13.048 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-09-18T08:12:26.959401+0000", 2024-09-18T08:15:13.048 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 166.01973208699999, 2024-09-18T08:15:13.048 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.086763688000000005, 2024-09-18T08:15:13.048 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-09-18T08:15:13.048 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-09-18T08:15:13.048 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T08:15:13.049 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2024-09-18T08:15:13.049 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.15582", 2024-09-18T08:15:13.049 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "172.21.15.7:0/3598605839", 2024-09-18T08:15:13.049 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 118446 2024-09-18T08:15:13.049 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:15:13.049 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-09-18T08:15:13.049 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:15:13.049 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-09-18T08:15:13.049 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:12:26.959401+0000", 2024-09-18T08:15:13.049 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-18T08:15:13.049 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:15:13.049 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:15:13.049 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-09-18T08:15:13.050 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:12:26.959401+0000", 2024-09-18T08:15:13.050 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-18T08:15:13.050 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:15:13.050 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:15:13.050 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-09-18T08:15:13.050 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:12:26.959402+0000", 2024-09-18T08:15:13.050 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 9.7999999999999993e-07 2024-09-18T08:15:13.050 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:15:13.050 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:15:13.050 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-09-18T08:15:13.050 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:12:26.959487+0000", 2024-09-18T08:15:13.050 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 8.5572e-05 2024-09-18T08:15:13.050 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:15:13.051 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:15:13.051 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-09-18T08:15:13.051 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:12:26.959502+0000", 2024-09-18T08:15:13.051 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.4568e-05 2024-09-18T08:15:13.051 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:15:13.051 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:15:13.051 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-09-18T08:15:13.051 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:12:26.959508+0000", 2024-09-18T08:15:13.051 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 6.6370000000000002e-06 2024-09-18T08:15:13.051 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:15:13.051 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:15:13.051 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-18T08:15:13.052 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:12:26.960169+0000", 2024-09-18T08:15:13.052 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00066084800000000001 2024-09-18T08:15:13.052 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:15:13.052 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:15:13.052 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-09-18T08:15:13.052 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:12:26.960284+0000", 2024-09-18T08:15:13.052 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00011485 2024-09-18T08:15:13.052 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:15:13.052 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:15:13.052 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "waiting for subops from 3", 2024-09-18T08:15:13.052 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:12:26.960424+0000", 2024-09-18T08:15:13.052 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000139787 2024-09-18T08:15:13.052 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:15:13.053 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:15:13.053 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "op_commit", 2024-09-18T08:15:13.053 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:12:26.963516+0000", 2024-09-18T08:15:13.053 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0030923209999999999 2024-09-18T08:15:13.053 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:15:13.053 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:15:13.053 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:15:13.053 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:12:27.046011+0000", 2024-09-18T08:15:13.053 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.082495000999999998 2024-09-18T08:15:13.053 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:15:13.053 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:15:13.053 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "commit_sent", 2024-09-18T08:15:13.053 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:12:27.046075+0000", 2024-09-18T08:15:13.054 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 6.4066000000000006e-05 2024-09-18T08:15:13.054 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:15:13.054 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:15:13.054 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-09-18T08:15:13.054 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:12:27.046164+0000", 2024-09-18T08:15:13.054 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 8.9058e-05 2024-09-18T08:15:13.054 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-18T08:15:13.054 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-09-18T08:15:13.054 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-18T08:15:13.054 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:15:13.054 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:15:13.054 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.15582.0:197623 9.3 9:ce24c1ab:::benchmark_data_smithi007_124019_object55738:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e279)", 2024-09-18T08:15:13.054 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-09-18T08:13:23.563996+0000", 2024-09-18T08:15:13.055 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 109.415136695, 2024-09-18T08:15:13.055 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.093202403000000003, 2024-09-18T08:15:13.055 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-09-18T08:15:13.055 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-09-18T08:15:13.055 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T08:15:13.055 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2024-09-18T08:15:13.055 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.15582", 2024-09-18T08:15:13.055 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "172.21.15.7:0/3598605839", 2024-09-18T08:15:13.055 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 197623 2024-09-18T08:15:13.055 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:15:13.055 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-09-18T08:15:13.055 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:15:13.055 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-09-18T08:15:13.055 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:13:23.563996+0000", 2024-09-18T08:15:13.056 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-18T08:15:13.056 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:15:13.056 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:15:13.056 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-09-18T08:15:13.056 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:13:23.563996+0000", 2024-09-18T08:15:13.056 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-18T08:15:13.056 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:15:13.056 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:15:13.056 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-09-18T08:15:13.056 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:13:23.563997+0000", 2024-09-18T08:15:13.056 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.0759999999999999e-06 2024-09-18T08:15:13.056 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:15:13.056 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:15:13.056 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-09-18T08:15:13.057 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:13:23.564001+0000", 2024-09-18T08:15:13.057 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 3.9500000000000003e-06 2024-09-18T08:15:13.057 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:15:13.057 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:15:13.057 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-09-18T08:15:13.057 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:13:23.564009+0000", 2024-09-18T08:15:13.057 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 7.6750000000000002e-06 2024-09-18T08:15:13.057 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:15:13.057 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:15:13.057 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-09-18T08:15:13.057 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:13:23.564014+0000", 2024-09-18T08:15:13.057 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 5.118e-06 2024-09-18T08:15:13.057 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:15:13.058 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:15:13.058 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-18T08:15:13.058 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:13:23.564067+0000", 2024-09-18T08:15:13.058 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 5.2788999999999999e-05 2024-09-18T08:15:13.058 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:15:13.058 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:15:13.058 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-09-18T08:15:13.058 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:13:23.564191+0000", 2024-09-18T08:15:13.058 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000124802 2024-09-18T08:15:13.058 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:15:13.058 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:15:13.058 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "waiting for subops from 3", 2024-09-18T08:15:13.058 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:13:23.564293+0000", 2024-09-18T08:15:13.059 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00010120600000000001 2024-09-18T08:15:13.059 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:15:13.059 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:15:13.059 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "op_commit", 2024-09-18T08:15:13.059 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:13:23.566905+0000", 2024-09-18T08:15:13.059 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.002611876 2024-09-18T08:15:13.059 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:15:13.059 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:15:13.059 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:15:13.059 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:13:23.656985+0000", 2024-09-18T08:15:13.059 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.090080405000000002 2024-09-18T08:15:13.059 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:15:13.059 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:15:13.059 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "commit_sent", 2024-09-18T08:15:13.060 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:13:23.657067+0000", 2024-09-18T08:15:13.060 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 8.2237999999999994e-05 2024-09-18T08:15:13.060 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:15:13.060 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:15:13.060 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-09-18T08:15:13.060 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:13:23.657198+0000", 2024-09-18T08:15:13.060 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000131268 2024-09-18T08:15:13.060 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-18T08:15:13.060 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-09-18T08:15:13.060 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-18T08:15:13.060 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:15:13.060 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:15:13.060 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.15582.0:201003 9.3 9:cd2f601a:::benchmark_data_smithi007_124019_object59118:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e279)", 2024-09-18T08:15:13.061 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-09-18T08:13:25.902357+0000", 2024-09-18T08:15:13.061 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 107.076775927, 2024-09-18T08:15:13.061 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.122441563, 2024-09-18T08:15:13.061 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-09-18T08:15:13.061 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-09-18T08:15:13.061 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T08:15:13.061 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2024-09-18T08:15:13.061 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.15582", 2024-09-18T08:15:13.061 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "172.21.15.7:0/3598605839", 2024-09-18T08:15:13.061 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 201003 2024-09-18T08:15:13.061 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:15:13.061 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-09-18T08:15:13.061 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:15:13.062 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-09-18T08:15:13.062 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:13:25.902357+0000", 2024-09-18T08:15:13.062 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-18T08:15:13.062 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:15:13.062 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:15:13.062 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-09-18T08:15:13.062 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:13:25.902357+0000", 2024-09-18T08:15:13.062 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-18T08:15:13.062 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:15:13.062 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:15:13.062 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-09-18T08:15:13.062 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:13:25.902358+0000", 2024-09-18T08:15:13.062 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 9.2399999999999996e-07 2024-09-18T08:15:13.062 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:15:13.063 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:15:13.063 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-09-18T08:15:13.063 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:13:25.902362+0000", 2024-09-18T08:15:13.063 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4.5310000000000003e-06 2024-09-18T08:15:13.063 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:15:13.063 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:15:13.063 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-09-18T08:15:13.063 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:13:25.902376+0000", 2024-09-18T08:15:13.063 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.3281e-05 2024-09-18T08:15:13.063 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:15:13.063 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:15:13.063 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-09-18T08:15:13.063 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:13:25.902382+0000", 2024-09-18T08:15:13.064 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 6.985e-06 2024-09-18T08:15:13.064 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:15:13.064 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:15:13.064 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-18T08:15:13.064 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:13:25.904951+0000", 2024-09-18T08:15:13.064 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0025684979999999998 2024-09-18T08:15:13.064 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:15:13.064 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:15:13.064 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-09-18T08:15:13.064 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:13:25.905053+0000", 2024-09-18T08:15:13.064 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000102191 2024-09-18T08:15:13.064 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:15:13.064 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:15:13.065 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "waiting for subops from 3", 2024-09-18T08:15:13.065 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:13:25.905161+0000", 2024-09-18T08:15:13.065 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000107934 2024-09-18T08:15:13.065 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:15:13.065 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:15:13.065 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "op_commit", 2024-09-18T08:15:13.065 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:13:25.907844+0000", 2024-09-18T08:15:13.065 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0026833769999999998 2024-09-18T08:15:13.065 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:15:13.065 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:15:13.065 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:15:13.065 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:13:26.024646+0000", 2024-09-18T08:15:13.065 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.116801815 2024-09-18T08:15:13.066 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:15:13.066 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:15:13.066 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "commit_sent", 2024-09-18T08:15:13.066 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:13:26.024707+0000", 2024-09-18T08:15:13.066 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 6.0414000000000001e-05 2024-09-18T08:15:13.066 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:15:13.066 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:15:13.066 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-09-18T08:15:13.066 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:13:26.024798+0000", 2024-09-18T08:15:13.066 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 9.1613000000000005e-05 2024-09-18T08:15:13.066 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-18T08:15:13.066 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-09-18T08:15:13.066 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-18T08:15:13.067 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:15:13.067 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:15:13.067 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.15582.0:201029 9.3 9:ce30e03a:::benchmark_data_smithi007_124019_object59144:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e279)", 2024-09-18T08:15:13.067 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-09-18T08:13:25.921094+0000", 2024-09-18T08:15:13.067 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 107.058039194, 2024-09-18T08:15:13.067 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.10395275499999999, 2024-09-18T08:15:13.067 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-09-18T08:15:13.067 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-09-18T08:15:13.067 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T08:15:13.067 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2024-09-18T08:15:13.067 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.15582", 2024-09-18T08:15:13.067 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "172.21.15.7:0/3598605839", 2024-09-18T08:15:13.067 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 201029 2024-09-18T08:15:13.067 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:15:13.068 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-09-18T08:15:13.068 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:15:13.068 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-09-18T08:15:13.068 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:13:25.921094+0000", 2024-09-18T08:15:13.068 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-18T08:15:13.068 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:15:13.068 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:15:13.068 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-09-18T08:15:13.068 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:13:25.921094+0000", 2024-09-18T08:15:13.068 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-18T08:15:13.068 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:15:13.068 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:15:13.068 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-09-18T08:15:13.069 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:13:25.921094+0000", 2024-09-18T08:15:13.069 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 8.6400000000000001e-07 2024-09-18T08:15:13.069 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:15:13.069 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:15:13.069 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-09-18T08:15:13.069 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:13:25.921098+0000", 2024-09-18T08:15:13.069 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 3.5109999999999999e-06 2024-09-18T08:15:13.069 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:15:13.069 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:15:13.069 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-09-18T08:15:13.069 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:13:25.921106+0000", 2024-09-18T08:15:13.069 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 8.6049999999999998e-06 2024-09-18T08:15:13.069 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:15:13.069 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:15:13.070 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-09-18T08:15:13.070 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:13:25.921111+0000", 2024-09-18T08:15:13.070 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4.4000000000000002e-06 2024-09-18T08:15:13.070 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:15:13.070 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:15:13.070 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-18T08:15:13.070 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:13:25.921174+0000", 2024-09-18T08:15:13.070 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 6.3596999999999994e-05 2024-09-18T08:15:13.070 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:15:13.070 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:15:13.070 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-09-18T08:15:13.070 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:13:25.921291+0000", 2024-09-18T08:15:13.070 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000116626 2024-09-18T08:15:13.070 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:15:13.071 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:15:13.071 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "waiting for subops from 3", 2024-09-18T08:15:13.071 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:13:25.921423+0000", 2024-09-18T08:15:13.071 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00013227000000000001 2024-09-18T08:15:13.071 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:15:13.071 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:15:13.071 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "op_commit", 2024-09-18T08:15:13.071 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:13:25.923648+0000", 2024-09-18T08:15:13.071 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0022248189999999998 2024-09-18T08:15:13.071 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:15:13.071 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:15:13.071 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:15:13.071 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:13:26.024871+0000", 2024-09-18T08:15:13.071 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.101222511 2024-09-18T08:15:13.072 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:15:13.072 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:15:13.072 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "commit_sent", 2024-09-18T08:15:13.072 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:13:26.024961+0000", 2024-09-18T08:15:13.072 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 8.9897999999999999e-05 2024-09-18T08:15:13.072 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:15:13.072 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:15:13.072 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-09-18T08:15:13.072 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:13:26.025046+0000", 2024-09-18T08:15:13.072 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 8.5654000000000004e-05 2024-09-18T08:15:13.072 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-18T08:15:13.072 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-09-18T08:15:13.072 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-18T08:15:13.073 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:15:13.073 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:15:13.073 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.15582.0:204526 9.d 9:b5fdf812:::benchmark_data_smithi007_124019_object62641:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e280)", 2024-09-18T08:15:13.073 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-09-18T08:13:28.337128+0000", 2024-09-18T08:15:13.073 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 104.64200519400001, 2024-09-18T08:15:13.073 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.104726574, 2024-09-18T08:15:13.073 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-09-18T08:15:13.073 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-09-18T08:15:13.073 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T08:15:13.073 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2024-09-18T08:15:13.073 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.15582", 2024-09-18T08:15:13.073 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "172.21.15.7:0/3598605839", 2024-09-18T08:15:13.073 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 204526 2024-09-18T08:15:13.074 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:15:13.074 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-09-18T08:15:13.074 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:15:13.074 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-09-18T08:15:13.074 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:13:28.337128+0000", 2024-09-18T08:15:13.074 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-18T08:15:13.074 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:15:13.074 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:15:13.074 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-09-18T08:15:13.074 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:13:28.337128+0000", 2024-09-18T08:15:13.074 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-18T08:15:13.074 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:15:13.074 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:15:13.074 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-09-18T08:15:13.075 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:13:28.337129+0000", 2024-09-18T08:15:13.075 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.421e-06 2024-09-18T08:15:13.075 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:15:13.075 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:15:13.075 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-09-18T08:15:13.075 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:13:28.337134+0000", 2024-09-18T08:15:13.075 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 5.074e-06 2024-09-18T08:15:13.075 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:15:13.075 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:15:13.075 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-09-18T08:15:13.075 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:13:28.337144+0000", 2024-09-18T08:15:13.075 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.0056e-05 2024-09-18T08:15:13.075 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:15:13.076 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:15:13.076 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-09-18T08:15:13.076 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:13:28.337151+0000", 2024-09-18T08:15:13.076 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 6.8700000000000003e-06 2024-09-18T08:15:13.076 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:15:13.076 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:15:13.076 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-18T08:15:13.076 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:13:28.337197+0000", 2024-09-18T08:15:13.076 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4.6455000000000002e-05 2024-09-18T08:15:13.076 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:15:13.076 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:15:13.076 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-09-18T08:15:13.076 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:13:28.337309+0000", 2024-09-18T08:15:13.077 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000112069 2024-09-18T08:15:13.077 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:15:13.077 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:15:13.077 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "waiting for subops from 5", 2024-09-18T08:15:13.077 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:13:28.337431+0000", 2024-09-18T08:15:13.077 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000121668 2024-09-18T08:15:13.077 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:15:13.077 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:15:13.077 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "op_commit", 2024-09-18T08:15:13.077 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:13:28.340148+0000", 2024-09-18T08:15:13.077 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.002716979 2024-09-18T08:15:13.077 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:15:13.077 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:15:13.078 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:15:13.078 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:13:28.441704+0000", 2024-09-18T08:15:13.078 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.101555944 2024-09-18T08:15:13.078 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:15:13.078 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:15:13.078 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "commit_sent", 2024-09-18T08:15:13.078 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:13:28.441755+0000", 2024-09-18T08:15:13.078 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 5.0594999999999999e-05 2024-09-18T08:15:13.078 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:15:13.078 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:15:13.078 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-09-18T08:15:13.078 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:13:28.441854+0000", 2024-09-18T08:15:13.078 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 9.9443000000000003e-05 2024-09-18T08:15:13.078 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-18T08:15:13.079 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-09-18T08:15:13.079 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-18T08:15:13.079 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:15:13.079 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:15:13.079 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.15582.0:204533 9.d 9:bec31205:::benchmark_data_smithi007_124019_object62648:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e280)", 2024-09-18T08:15:13.079 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-09-18T08:13:28.341130+0000", 2024-09-18T08:15:13.079 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 104.63800306, 2024-09-18T08:15:13.079 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.100963845, 2024-09-18T08:15:13.079 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-09-18T08:15:13.079 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-09-18T08:15:13.079 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T08:15:13.079 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2024-09-18T08:15:13.079 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.15582", 2024-09-18T08:15:13.080 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "172.21.15.7:0/3598605839", 2024-09-18T08:15:13.080 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 204533 2024-09-18T08:15:13.080 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:15:13.080 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-09-18T08:15:13.080 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:15:13.080 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-09-18T08:15:13.080 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:13:28.341130+0000", 2024-09-18T08:15:13.080 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-18T08:15:13.080 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:15:13.080 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:15:13.080 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-09-18T08:15:13.080 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:13:28.341130+0000", 2024-09-18T08:15:13.080 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-18T08:15:13.081 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:15:13.081 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:15:13.081 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-09-18T08:15:13.081 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:13:28.341131+0000", 2024-09-18T08:15:13.081 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.2079999999999999e-06 2024-09-18T08:15:13.081 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:15:13.081 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:15:13.081 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-09-18T08:15:13.081 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:13:28.341136+0000", 2024-09-18T08:15:13.081 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4.7299999999999996e-06 2024-09-18T08:15:13.081 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:15:13.081 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:15:13.081 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-09-18T08:15:13.081 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:13:28.341145+0000", 2024-09-18T08:15:13.082 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 9.0010000000000007e-06 2024-09-18T08:15:13.082 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:15:13.082 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:15:13.082 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-09-18T08:15:13.082 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:13:28.341150+0000", 2024-09-18T08:15:13.082 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 5.5550000000000002e-06 2024-09-18T08:15:13.082 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:15:13.082 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:15:13.082 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-18T08:15:13.082 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:13:28.341189+0000", 2024-09-18T08:15:13.082 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 3.8544000000000002e-05 2024-09-18T08:15:13.082 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:15:13.082 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:15:13.083 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-09-18T08:15:13.083 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:13:28.341262+0000", 2024-09-18T08:15:13.083 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 7.3706000000000002e-05 2024-09-18T08:15:13.083 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:15:13.083 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:15:13.083 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "waiting for subops from 5", 2024-09-18T08:15:13.083 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:13:28.341335+0000", 2024-09-18T08:15:13.083 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 7.2220999999999999e-05 2024-09-18T08:15:13.083 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:15:13.083 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:15:13.083 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "op_commit", 2024-09-18T08:15:13.083 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:13:28.343163+0000", 2024-09-18T08:15:13.083 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0018283100000000001 2024-09-18T08:15:13.083 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:15:13.084 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:15:13.084 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:15:13.084 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:13:28.441919+0000", 2024-09-18T08:15:13.084 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.098756517000000002 2024-09-18T08:15:13.084 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:15:13.084 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:15:13.084 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "commit_sent", 2024-09-18T08:15:13.084 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:13:28.442005+0000", 2024-09-18T08:15:13.084 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 8.5185000000000005e-05 2024-09-18T08:15:13.084 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:15:13.084 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:15:13.084 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-09-18T08:15:13.084 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:13:28.442093+0000", 2024-09-18T08:15:13.084 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 8.8868000000000004e-05 2024-09-18T08:15:13.085 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-18T08:15:13.085 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-09-18T08:15:13.085 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-18T08:15:13.085 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:15:13.085 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:15:13.085 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.15582.0:204536 9.d 9:bfcc6a39:::benchmark_data_smithi007_124019_object62651:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e280)", 2024-09-18T08:15:13.085 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-09-18T08:13:28.344569+0000", 2024-09-18T08:15:13.085 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 104.634563456, 2024-09-18T08:15:13.085 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.097733613999999996, 2024-09-18T08:15:13.085 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-09-18T08:15:13.085 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-09-18T08:15:13.085 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T08:15:13.085 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2024-09-18T08:15:13.086 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.15582", 2024-09-18T08:15:13.086 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "172.21.15.7:0/3598605839", 2024-09-18T08:15:13.086 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 204536 2024-09-18T08:15:13.086 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:15:13.086 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-09-18T08:15:13.086 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:15:13.086 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-09-18T08:15:13.086 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:13:28.344569+0000", 2024-09-18T08:15:13.086 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-18T08:15:13.086 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:15:13.086 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:15:13.086 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-09-18T08:15:13.086 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:13:28.344569+0000", 2024-09-18T08:15:13.087 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-18T08:15:13.087 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:15:13.087 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:15:13.087 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-09-18T08:15:13.087 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:13:28.344571+0000", 2024-09-18T08:15:13.087 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.8899999999999999e-06 2024-09-18T08:15:13.087 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:15:13.087 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:15:13.087 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-09-18T08:15:13.087 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:13:28.344581+0000", 2024-09-18T08:15:13.087 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 9.9340000000000001e-06 2024-09-18T08:15:13.087 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:15:13.087 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:15:13.087 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-09-18T08:15:13.088 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:13:28.344604+0000", 2024-09-18T08:15:13.088 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 2.2501999999999999e-05 2024-09-18T08:15:13.088 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:15:13.088 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:15:13.088 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-09-18T08:15:13.088 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:13:28.344617+0000", 2024-09-18T08:15:13.088 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.3179999999999999e-05 2024-09-18T08:15:13.088 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:15:13.088 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:15:13.088 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-18T08:15:13.088 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:13:28.344737+0000", 2024-09-18T08:15:13.088 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000120184 2024-09-18T08:15:13.089 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:15:13.089 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:15:13.089 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-09-18T08:15:13.089 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:13:28.430332+0000", 2024-09-18T08:15:13.089 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.085595180000000007 2024-09-18T08:15:13.089 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:15:13.089 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:15:13.089 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "waiting for subops from 5", 2024-09-18T08:15:13.089 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:13:28.430450+0000", 2024-09-18T08:15:13.089 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00011794 2024-09-18T08:15:13.089 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:15:13.089 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:15:13.089 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "op_commit", 2024-09-18T08:15:13.089 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:13:28.434318+0000", 2024-09-18T08:15:13.090 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0038681639999999999 2024-09-18T08:15:13.090 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:15:13.090 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:15:13.090 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:15:13.090 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:13:28.442166+0000", 2024-09-18T08:15:13.090 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0078480779999999997 2024-09-18T08:15:13.090 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:15:13.090 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:15:13.090 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "commit_sent", 2024-09-18T08:15:13.090 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:13:28.442220+0000", 2024-09-18T08:15:13.090 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 5.3488e-05 2024-09-18T08:15:13.090 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:15:13.091 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:15:13.091 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-09-18T08:15:13.091 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:13:28.442303+0000", 2024-09-18T08:15:13.091 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 8.3073999999999998e-05 2024-09-18T08:15:13.091 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-18T08:15:13.091 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-09-18T08:15:13.091 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-18T08:15:13.091 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:15:13.091 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:15:13.091 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.15582.0:204537 9.3 9:cbf209e5:::benchmark_data_smithi007_124019_object62652:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e280)", 2024-09-18T08:15:13.091 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-09-18T08:13:28.344665+0000", 2024-09-18T08:15:13.091 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 104.634467474, 2024-09-18T08:15:13.091 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.089161554000000004, 2024-09-18T08:15:13.092 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-09-18T08:15:13.092 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-09-18T08:15:13.092 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T08:15:13.092 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2024-09-18T08:15:13.092 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.15582", 2024-09-18T08:15:13.092 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "172.21.15.7:0/3598605839", 2024-09-18T08:15:13.092 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 204537 2024-09-18T08:15:13.092 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:15:13.092 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-09-18T08:15:13.092 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:15:13.092 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-09-18T08:15:13.092 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:13:28.344665+0000", 2024-09-18T08:15:13.092 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-18T08:15:13.093 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:15:13.093 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:15:13.093 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-09-18T08:15:13.093 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:13:28.344665+0000", 2024-09-18T08:15:13.093 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-18T08:15:13.093 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:15:13.093 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:15:13.093 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-09-18T08:15:13.093 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:13:28.344667+0000", 2024-09-18T08:15:13.093 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.3030000000000001e-06 2024-09-18T08:15:13.093 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:15:13.093 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:15:13.094 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-09-18T08:15:13.094 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:13:28.344673+0000", 2024-09-18T08:15:13.094 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 6.3570000000000003e-06 2024-09-18T08:15:13.094 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:15:13.094 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:15:13.094 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-09-18T08:15:13.094 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:13:28.344686+0000", 2024-09-18T08:15:13.094 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.2911999999999999e-05 2024-09-18T08:15:13.094 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:15:13.094 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:15:13.094 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-09-18T08:15:13.094 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:13:28.344694+0000", 2024-09-18T08:15:13.094 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 8.4970000000000001e-06 2024-09-18T08:15:13.094 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:15:13.095 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:15:13.095 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-18T08:15:13.095 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:13:28.344796+0000", 2024-09-18T08:15:13.095 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000101418 2024-09-18T08:15:13.095 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:15:13.095 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:15:13.095 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-09-18T08:15:13.095 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:13:28.428130+0000", 2024-09-18T08:15:13.095 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.083334277999999998 2024-09-18T08:15:13.095 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:15:13.095 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:15:13.095 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "waiting for subops from 3", 2024-09-18T08:15:13.095 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:13:28.428251+0000", 2024-09-18T08:15:13.096 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000121437 2024-09-18T08:15:13.096 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:15:13.096 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:15:13.096 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "op_commit", 2024-09-18T08:15:13.096 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:13:28.430866+0000", 2024-09-18T08:15:13.096 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0026143619999999998 2024-09-18T08:15:13.096 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:15:13.096 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:15:13.096 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:15:13.096 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:13:28.433618+0000", 2024-09-18T08:15:13.096 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0027524519999999999 2024-09-18T08:15:13.096 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:15:13.096 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:15:13.097 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "commit_sent", 2024-09-18T08:15:13.097 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:13:28.433678+0000", 2024-09-18T08:15:13.097 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 5.9432000000000003e-05 2024-09-18T08:15:13.097 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:15:13.097 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:15:13.097 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-09-18T08:15:13.097 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:13:28.433827+0000", 2024-09-18T08:15:13.097 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00014910600000000001 2024-09-18T08:15:13.097 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-18T08:15:13.097 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-09-18T08:15:13.097 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-18T08:15:13.097 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:15:13.097 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:15:13.097 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.15582.0:215801 9.3 9:c417b2c8:::benchmark_data_smithi007_124019_object73916:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e283)", 2024-09-18T08:15:13.098 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-09-18T08:13:36.813982+0000", 2024-09-18T08:15:13.098 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 96.165150928000003, 2024-09-18T08:15:13.098 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.103025502, 2024-09-18T08:15:13.098 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-09-18T08:15:13.098 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-09-18T08:15:13.098 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T08:15:13.098 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2024-09-18T08:15:13.098 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.15582", 2024-09-18T08:15:13.098 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "172.21.15.7:0/3598605839", 2024-09-18T08:15:13.098 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 215801 2024-09-18T08:15:13.098 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:15:13.098 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-09-18T08:15:13.098 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:15:13.098 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-09-18T08:15:13.099 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:13:36.813982+0000", 2024-09-18T08:15:13.099 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-18T08:15:13.099 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:15:13.099 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:15:13.099 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-09-18T08:15:13.099 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:13:36.813982+0000", 2024-09-18T08:15:13.099 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-18T08:15:13.099 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:15:13.099 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:15:13.099 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-09-18T08:15:13.099 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:13:36.813983+0000", 2024-09-18T08:15:13.099 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.3799999999999999e-06 2024-09-18T08:15:13.099 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:15:13.099 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:15:13.100 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-09-18T08:15:13.100 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:13:36.813988+0000", 2024-09-18T08:15:13.100 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 5.1449999999999999e-06 2024-09-18T08:15:13.100 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:15:13.100 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:15:13.100 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-09-18T08:15:13.100 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:13:36.813999+0000", 2024-09-18T08:15:13.100 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.0278999999999999e-05 2024-09-18T08:15:13.100 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:15:13.100 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:15:13.100 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-09-18T08:15:13.100 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:13:36.814006+0000", 2024-09-18T08:15:13.100 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 7.3200000000000002e-06 2024-09-18T08:15:13.101 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:15:13.101 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:15:13.101 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-18T08:15:13.101 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:13:36.814067+0000", 2024-09-18T08:15:13.101 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 6.1179999999999994e-05 2024-09-18T08:15:13.101 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:15:13.101 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:15:13.101 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-09-18T08:15:13.101 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:13:36.814173+0000", 2024-09-18T08:15:13.101 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000105923 2024-09-18T08:15:13.101 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:15:13.101 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:15:13.101 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "waiting for subops from 3", 2024-09-18T08:15:13.102 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:13:36.814277+0000", 2024-09-18T08:15:13.102 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000104001 2024-09-18T08:15:13.102 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:15:13.102 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:15:13.102 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "op_commit", 2024-09-18T08:15:13.102 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:13:36.816739+0000", 2024-09-18T08:15:13.102 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0024617900000000002 2024-09-18T08:15:13.102 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:15:13.102 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:15:13.102 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:15:13.103 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:13:36.916863+0000", 2024-09-18T08:15:13.103 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.100124276 2024-09-18T08:15:13.104 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:15:13.104 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:15:13.104 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "commit_sent", 2024-09-18T08:15:13.104 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:13:36.916923+0000", 2024-09-18T08:15:13.104 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 5.9951000000000002e-05 2024-09-18T08:15:13.104 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:15:13.104 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:15:13.104 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-09-18T08:15:13.104 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:13:36.917007+0000", 2024-09-18T08:15:13.104 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 8.4257000000000003e-05 2024-09-18T08:15:13.104 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-18T08:15:13.104 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-09-18T08:15:13.105 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-18T08:15:13.105 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:15:13.105 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:15:13.105 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.15582.0:215807 9.3 9:c0cd06f9:::benchmark_data_smithi007_124019_object73922:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e283)", 2024-09-18T08:15:13.105 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-09-18T08:13:36.817961+0000", 2024-09-18T08:15:13.105 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 96.161171706000005, 2024-09-18T08:15:13.105 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.099281575999999996, 2024-09-18T08:15:13.105 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-09-18T08:15:13.105 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-09-18T08:15:13.105 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T08:15:13.105 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2024-09-18T08:15:13.105 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.15582", 2024-09-18T08:15:13.106 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "172.21.15.7:0/3598605839", 2024-09-18T08:15:13.106 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 215807 2024-09-18T08:15:13.106 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:15:13.106 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-09-18T08:15:13.106 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:15:13.106 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-09-18T08:15:13.106 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:13:36.817961+0000", 2024-09-18T08:15:13.106 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-18T08:15:13.106 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:15:13.106 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:15:13.106 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-09-18T08:15:13.106 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:13:36.817961+0000", 2024-09-18T08:15:13.106 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-18T08:15:13.106 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:15:13.107 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:15:13.107 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-09-18T08:15:13.107 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:13:36.817963+0000", 2024-09-18T08:15:13.107 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 2.2539999999999999e-06 2024-09-18T08:15:13.107 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:15:13.107 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:15:13.107 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-09-18T08:15:13.107 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:13:36.817967+0000", 2024-09-18T08:15:13.107 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 3.72e-06 2024-09-18T08:15:13.107 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:15:13.107 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:15:13.107 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-09-18T08:15:13.107 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:13:36.817980+0000", 2024-09-18T08:15:13.108 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.3345e-05 2024-09-18T08:15:13.108 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:15:13.108 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:15:13.108 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-09-18T08:15:13.108 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:13:36.817988+0000", 2024-09-18T08:15:13.108 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 7.9370000000000004e-06 2024-09-18T08:15:13.108 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:15:13.108 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:15:13.108 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-18T08:15:13.108 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:13:36.818041+0000", 2024-09-18T08:15:13.108 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 5.3081000000000002e-05 2024-09-18T08:15:13.108 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:15:13.109 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:15:13.109 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-09-18T08:15:13.109 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:13:36.818171+0000", 2024-09-18T08:15:13.109 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00012934399999999999 2024-09-18T08:15:13.109 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:15:13.109 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:15:13.109 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "waiting for subops from 3", 2024-09-18T08:15:13.109 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:13:36.818298+0000", 2024-09-18T08:15:13.109 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00012757900000000001 2024-09-18T08:15:13.109 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:15:13.109 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:15:13.109 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "op_commit", 2024-09-18T08:15:13.110 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:13:36.820689+0000", 2024-09-18T08:15:13.110 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.002390905 2024-09-18T08:15:13.110 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:15:13.110 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:15:13.110 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:15:13.110 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:13:36.917068+0000", 2024-09-18T08:15:13.110 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.096378909999999998 2024-09-18T08:15:13.110 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:15:13.110 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:15:13.110 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "commit_sent", 2024-09-18T08:15:13.110 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:13:36.917154+0000", 2024-09-18T08:15:13.110 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 8.5792999999999997e-05 2024-09-18T08:15:13.110 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:15:13.110 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:15:13.111 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-09-18T08:15:13.111 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:13:36.917243+0000", 2024-09-18T08:15:13.111 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 8.8708000000000005e-05 2024-09-18T08:15:13.111 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-18T08:15:13.111 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-09-18T08:15:13.111 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-18T08:15:13.111 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-18T08:15:13.111 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-09-18T08:15:13.111 INFO:teuthology.orchestra.run.smithi050.stdout:} 2024-09-18T08:15:13.112 INFO:teuthology.orchestra.run.smithi007.stdout:{"pg_ready":true,"pg_map":{"version":614,"stamp":"2024-09-18T08:15:11.353779+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":1941028198,"num_objects":29807,"num_object_clones":0,"num_object_copies":89421,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":29807,"num_whiteouts":0,"num_read":1354,"num_read_kb":1125,"num_write":30350,"num_write_kb":1895189,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3978,"num_bytes_recovered":235195177,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":31282,"ondisk_log_size":31282,"up":474,"acting":474,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":474,"num_osds":6,"num_per_pool_osds":6,"num_per_pool_omap_osds":6,"kb":562470912,"kb_used":7933920,"kb_used_data":5607668,"kb_used_omap":15,"kb_used_meta":2326128,"kb_avail":554536992,"statfs":{"total":575970213888,"available":567845879808,"internally_reserved":0,"allocated":5742252032,"data_stored":5736910047,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":16050,"internal_metadata":2381955406},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":828571648,"num_objects":12643,"num_object_clones":0,"num_object_copies":37929,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-565,"num_objects_unfound":0,"num_objects_dirty":12643,"num_whiteouts":0,"num_read":5,"num_read_kb":5,"num_write":12643,"num_write_kb":809152,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":565,"num_bytes_recovered":37027840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.004159"},"pg_stats":[{"pgid":"3.1f","version":"0'0","reported_seq":597,"reported_epoch":326,"state":"active+clean","last_fresh":"2024-09-18T08:15:07.732272+0000","last_change":"2024-09-18T08:14:52.682070+0000","last_active":"2024-09-18T08:15:07.732272+0000","last_peered":"2024-09-18T08:15:07.732272+0000","last_clean":"2024-09-18T08:15:07.732272+0000","last_became_active":"2024-09-18T08:14:52.681703+0000","last_became_peered":"2024-09-18T08:14:52.681703+0000","last_unstale":"2024-09-18T08:15:07.732272+0000","last_undegraded":"2024-09-18T08:15:07.732272+0000","last_fullsized":"2024-09-18T08:15:07.732272+0000","mapping_epoch":316,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":317,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:14:29.060206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:46.860092+0000","last_clean_scrub_stamp":"2024-09-18T08:14:29.060206+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:15:37.414506+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,6],"acting":[0,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"271'15","reported_seq":357,"reported_epoch":326,"state":"active+clean","last_fresh":"2024-09-18T08:15:07.715024+0000","last_change":"2024-09-18T08:15:04.677032+0000","last_active":"2024-09-18T08:15:07.715024+0000","last_peered":"2024-09-18T08:15:07.715024+0000","last_clean":"2024-09-18T08:15:07.715024+0000","last_became_active":"2024-09-18T08:14:39.372321+0000","last_became_peered":"2024-09-18T08:14:39.372321+0000","last_unstale":"2024-09-18T08:15:07.715024+0000","last_undegraded":"2024-09-18T08:15:07.715024+0000","last_fullsized":"2024-09-18T08:15:07.715024+0000","mapping_epoch":307,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":308,"parent":"0.0","parent_split_bits":0,"last_scrub":"271'15","last_scrub_stamp":"2024-09-18T08:15:04.676980+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_clean_scrub_stamp":"2024-09-18T08:15:04.676980+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:16:27.967770+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":25,"num_read_kb":19,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":405,"reported_epoch":325,"state":"active+clean","last_fresh":"2024-09-18T08:15:01.378729+0000","last_change":"2024-09-18T08:14:52.674822+0000","last_active":"2024-09-18T08:15:01.378729+0000","last_peered":"2024-09-18T08:15:01.378729+0000","last_clean":"2024-09-18T08:15:01.378729+0000","last_became_active":"2024-09-18T08:14:52.674469+0000","last_became_peered":"2024-09-18T08:14:52.674469+0000","last_unstale":"2024-09-18T08:15:01.378729+0000","last_undegraded":"2024-09-18T08:15:01.378729+0000","last_fullsized":"2024-09-18T08:15:01.378729+0000","mapping_epoch":316,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":317,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:10:15.603289+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:50.886830+0000","last_clean_scrub_stamp":"2024-09-18T08:10:15.603289+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":328,"reported_epoch":326,"state":"active+clean","last_fresh":"2024-09-18T08:15:07.716514+0000","last_change":"2024-09-18T08:14:52.655718+0000","last_active":"2024-09-18T08:15:07.716514+0000","last_peered":"2024-09-18T08:15:07.716514+0000","last_clean":"2024-09-18T08:15:07.716514+0000","last_became_active":"2024-09-18T08:14:52.655416+0000","last_became_peered":"2024-09-18T08:14:52.655416+0000","last_unstale":"2024-09-18T08:15:07.716514+0000","last_undegraded":"2024-09-18T08:15:07.716514+0000","last_fullsized":"2024-09-18T08:15:07.716514+0000","mapping_epoch":316,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":317,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:14:41.746083+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_clean_scrub_stamp":"2024-09-18T08:14:41.746083+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:15:47.250691+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,6],"acting":[4,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":590,"reported_epoch":326,"state":"active+clean","last_fresh":"2024-09-18T08:15:07.722034+0000","last_change":"2024-09-18T08:14:40.357598+0000","last_active":"2024-09-18T08:15:07.722034+0000","last_peered":"2024-09-18T08:15:07.722034+0000","last_clean":"2024-09-18T08:15:07.722034+0000","last_became_active":"2024-09-18T08:14:40.357298+0000","last_became_peered":"2024-09-18T08:14:40.357298+0000","last_unstale":"2024-09-18T08:15:07.722034+0000","last_undegraded":"2024-09-18T08:15:07.722034+0000","last_fullsized":"2024-09-18T08:15:07.722034+0000","mapping_epoch":308,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":309,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:12:38.915501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_clean_scrub_stamp":"2024-09-18T08:12:38.915501+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,1],"acting":[3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":600,"reported_epoch":326,"state":"active+clean","last_fresh":"2024-09-18T08:15:07.720562+0000","last_change":"2024-09-18T08:13:37.627262+0000","last_active":"2024-09-18T08:15:07.720562+0000","last_peered":"2024-09-18T08:15:07.720562+0000","last_clean":"2024-09-18T08:15:07.720562+0000","last_became_active":"2024-09-18T08:06:24.126194+0000","last_became_peered":"2024-09-18T08:06:24.126194+0000","last_unstale":"2024-09-18T08:15:07.720562+0000","last_undegraded":"2024-09-18T08:15:07.720562+0000","last_fullsized":"2024-09-18T08:15:07.720562+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:13:37.627195+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:46.860092+0000","last_clean_scrub_stamp":"2024-09-18T08:13:37.627195+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"271'27","reported_seq":600,"reported_epoch":326,"state":"active+clean","last_fresh":"2024-09-18T08:15:07.719437+0000","last_change":"2024-09-18T08:14:26.565324+0000","last_active":"2024-09-18T08:15:07.719437+0000","last_peered":"2024-09-18T08:15:07.719437+0000","last_clean":"2024-09-18T08:15:07.719437+0000","last_became_active":"2024-09-18T08:02:49.886106+0000","last_became_peered":"2024-09-18T08:02:49.886106+0000","last_unstale":"2024-09-18T08:15:07.719437+0000","last_undegraded":"2024-09-18T08:15:07.719437+0000","last_fullsized":"2024-09-18T08:15:07.719437+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"271'27","last_scrub_stamp":"2024-09-18T08:14:26.565071+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_clean_scrub_stamp":"2024-09-18T08:14:26.565071+0000","objects_scrubbed":7,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:15:42.902791+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":44,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":313,"reported_epoch":326,"state":"active+clean","last_fresh":"2024-09-18T08:15:07.715071+0000","last_change":"2024-09-18T08:14:39.371821+0000","last_active":"2024-09-18T08:15:07.715071+0000","last_peered":"2024-09-18T08:15:07.715071+0000","last_clean":"2024-09-18T08:15:07.715071+0000","last_became_active":"2024-09-18T08:14:39.371563+0000","last_became_peered":"2024-09-18T08:14:39.371563+0000","last_unstale":"2024-09-18T08:15:07.715071+0000","last_undegraded":"2024-09-18T08:15:07.715071+0000","last_fullsized":"2024-09-18T08:15:07.715071+0000","mapping_epoch":307,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":308,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:08:09.142188+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:50.886830+0000","last_clean_scrub_stamp":"2024-09-18T08:08:09.142188+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:15:42.766622+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":318,"reported_epoch":326,"state":"active+clean","last_fresh":"2024-09-18T08:15:07.719139+0000","last_change":"2024-09-18T08:14:52.651395+0000","last_active":"2024-09-18T08:15:07.719139+0000","last_peered":"2024-09-18T08:15:07.719139+0000","last_clean":"2024-09-18T08:15:07.719139+0000","last_became_active":"2024-09-18T08:14:52.651028+0000","last_became_peered":"2024-09-18T08:14:52.651028+0000","last_unstale":"2024-09-18T08:15:07.719139+0000","last_undegraded":"2024-09-18T08:15:07.719139+0000","last_fullsized":"2024-09-18T08:15:07.719139+0000","mapping_epoch":316,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":317,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:10:51.225974+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:46.860092+0000","last_clean_scrub_stamp":"2024-09-18T08:10:51.225974+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1a","version":"271'15","reported_seq":337,"reported_epoch":326,"state":"active+clean","last_fresh":"2024-09-18T08:15:07.716601+0000","last_change":"2024-09-18T08:12:00.466295+0000","last_active":"2024-09-18T08:15:07.716601+0000","last_peered":"2024-09-18T08:15:07.716601+0000","last_clean":"2024-09-18T08:15:07.716601+0000","last_became_active":"2024-09-18T08:08:18.550960+0000","last_became_peered":"2024-09-18T08:08:18.550960+0000","last_unstale":"2024-09-18T08:15:07.716601+0000","last_undegraded":"2024-09-18T08:15:07.716601+0000","last_fullsized":"2024-09-18T08:15:07.716601+0000","mapping_epoch":181,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":182,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'9","last_scrub_stamp":"2024-09-18T08:12:00.466202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_clean_scrub_stamp":"2024-09-18T08:12:00.466202+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":25,"num_read_kb":19,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":521,"reported_epoch":326,"state":"active+clean","last_fresh":"2024-09-18T08:15:07.733010+0000","last_change":"2024-09-18T08:14:52.659770+0000","last_active":"2024-09-18T08:15:07.733010+0000","last_peered":"2024-09-18T08:15:07.733010+0000","last_clean":"2024-09-18T08:15:07.733010+0000","last_became_active":"2024-09-18T08:14:52.657808+0000","last_became_peered":"2024-09-18T08:14:52.657808+0000","last_unstale":"2024-09-18T08:15:07.733010+0000","last_undegraded":"2024-09-18T08:15:07.733010+0000","last_fullsized":"2024-09-18T08:15:07.733010+0000","mapping_epoch":316,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":317,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:12:57.218286+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:50.886830+0000","last_clean_scrub_stamp":"2024-09-18T08:12:57.218286+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,6],"acting":[0,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":565,"reported_epoch":326,"state":"active+clean","last_fresh":"2024-09-18T08:15:07.731327+0000","last_change":"2024-09-18T08:14:40.356444+0000","last_active":"2024-09-18T08:15:07.731327+0000","last_peered":"2024-09-18T08:15:07.731327+0000","last_clean":"2024-09-18T08:15:07.731327+0000","last_became_active":"2024-09-18T08:14:40.354651+0000","last_became_peered":"2024-09-18T08:14:40.354651+0000","last_unstale":"2024-09-18T08:15:07.731327+0000","last_undegraded":"2024-09-18T08:15:07.731327+0000","last_fullsized":"2024-09-18T08:15:07.731327+0000","mapping_epoch":308,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":309,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:08:14.645993+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_clean_scrub_stamp":"2024-09-18T08:08:14.645993+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:15:43.181645+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":569,"reported_epoch":326,"state":"active+clean","last_fresh":"2024-09-18T08:15:07.715494+0000","last_change":"2024-09-18T08:15:01.636578+0000","last_active":"2024-09-18T08:15:07.715494+0000","last_peered":"2024-09-18T08:15:07.715494+0000","last_clean":"2024-09-18T08:15:07.715494+0000","last_became_active":"2024-09-18T08:14:52.647692+0000","last_became_peered":"2024-09-18T08:14:52.647692+0000","last_unstale":"2024-09-18T08:15:07.715494+0000","last_undegraded":"2024-09-18T08:15:07.715494+0000","last_fullsized":"2024-09-18T08:15:07.715494+0000","mapping_epoch":316,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":317,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:15:01.636504+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:46.860092+0000","last_clean_scrub_stamp":"2024-09-18T08:15:01.636504+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:16:11.682400+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,6],"acting":[4,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1b","version":"271'7","reported_seq":335,"reported_epoch":326,"state":"active+clean","last_fresh":"2024-09-18T08:15:07.715239+0000","last_change":"2024-09-18T08:14:39.425570+0000","last_active":"2024-09-18T08:15:07.715239+0000","last_peered":"2024-09-18T08:15:07.715239+0000","last_clean":"2024-09-18T08:15:07.715239+0000","last_became_active":"2024-09-18T08:14:39.368637+0000","last_became_peered":"2024-09-18T08:14:39.368637+0000","last_unstale":"2024-09-18T08:15:07.715239+0000","last_undegraded":"2024-09-18T08:15:07.715239+0000","last_fullsized":"2024-09-18T08:15:07.715239+0000","mapping_epoch":307,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":308,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'5","last_scrub_stamp":"2024-09-18T08:10:52.400411+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_clean_scrub_stamp":"2024-09-18T08:10:52.400411+0000","objects_scrubbed":2,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":13,"num_read_kb":9,"num_write":8,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":515,"reported_epoch":326,"state":"active+clean","last_fresh":"2024-09-18T08:15:07.716272+0000","last_change":"2024-09-18T08:14:40.353773+0000","last_active":"2024-09-18T08:15:07.716272+0000","last_peered":"2024-09-18T08:15:07.716272+0000","last_clean":"2024-09-18T08:15:07.716272+0000","last_became_active":"2024-09-18T08:14:40.353507+0000","last_became_peered":"2024-09-18T08:14:40.353507+0000","last_unstale":"2024-09-18T08:15:07.716272+0000","last_undegraded":"2024-09-18T08:15:07.716272+0000","last_fullsized":"2024-09-18T08:15:07.716272+0000","mapping_epoch":308,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":309,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:11:30.837341+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:50.886830+0000","last_clean_scrub_stamp":"2024-09-18T08:11:30.837341+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,0],"acting":[4,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":550,"reported_epoch":325,"state":"active+clean","last_fresh":"2024-09-18T08:15:01.378999+0000","last_change":"2024-09-18T08:14:52.674212+0000","last_active":"2024-09-18T08:15:01.378999+0000","last_peered":"2024-09-18T08:15:01.378999+0000","last_clean":"2024-09-18T08:15:01.378999+0000","last_became_active":"2024-09-18T08:14:52.673809+0000","last_became_peered":"2024-09-18T08:14:52.673809+0000","last_unstale":"2024-09-18T08:15:01.378999+0000","last_undegraded":"2024-09-18T08:15:01.378999+0000","last_fullsized":"2024-09-18T08:15:01.378999+0000","mapping_epoch":316,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":317,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:11:01.108648+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_clean_scrub_stamp":"2024-09-18T08:11:01.108648+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":318,"reported_epoch":326,"state":"active+clean","last_fresh":"2024-09-18T08:15:07.715916+0000","last_change":"2024-09-18T08:14:52.663738+0000","last_active":"2024-09-18T08:15:07.715916+0000","last_peered":"2024-09-18T08:15:07.715916+0000","last_clean":"2024-09-18T08:15:07.715916+0000","last_became_active":"2024-09-18T08:14:52.662945+0000","last_became_peered":"2024-09-18T08:14:52.662945+0000","last_unstale":"2024-09-18T08:15:07.715916+0000","last_undegraded":"2024-09-18T08:15:07.715916+0000","last_fullsized":"2024-09-18T08:15:07.715916+0000","mapping_epoch":316,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":317,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:11:34.625572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_clean_scrub_stamp":"2024-09-18T08:11:34.625572+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,3],"acting":[4,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":593,"reported_epoch":326,"state":"active+clean","last_fresh":"2024-09-18T08:15:07.733414+0000","last_change":"2024-09-18T08:14:40.354122+0000","last_active":"2024-09-18T08:15:07.733414+0000","last_peered":"2024-09-18T08:15:07.733414+0000","last_clean":"2024-09-18T08:15:07.733414+0000","last_became_active":"2024-09-18T08:14:40.353509+0000","last_became_peered":"2024-09-18T08:14:40.353509+0000","last_unstale":"2024-09-18T08:15:07.733414+0000","last_undegraded":"2024-09-18T08:15:07.733414+0000","last_fullsized":"2024-09-18T08:15:07.733414+0000","mapping_epoch":308,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":309,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:12:40.711507+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:46.860092+0000","last_clean_scrub_stamp":"2024-09-18T08:12:40.711507+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,1],"acting":[0,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1c","version":"271'27","reported_seq":336,"reported_epoch":326,"state":"active+clean","last_fresh":"2024-09-18T08:15:07.725901+0000","last_change":"2024-09-18T08:14:40.711299+0000","last_active":"2024-09-18T08:15:07.725901+0000","last_peered":"2024-09-18T08:15:07.725901+0000","last_clean":"2024-09-18T08:15:07.725901+0000","last_became_active":"2024-09-18T08:14:39.480913+0000","last_became_peered":"2024-09-18T08:14:39.480913+0000","last_unstale":"2024-09-18T08:15:07.725901+0000","last_undegraded":"2024-09-18T08:15:07.725901+0000","last_fullsized":"2024-09-18T08:15:07.725901+0000","mapping_epoch":307,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":308,"parent":"0.0","parent_split_bits":0,"last_scrub":"271'27","last_scrub_stamp":"2024-09-18T08:14:40.711228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_clean_scrub_stamp":"2024-09-18T08:14:40.711228+0000","objects_scrubbed":7,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:16:07.663050+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":44,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":441,"reported_epoch":325,"state":"active+clean","last_fresh":"2024-09-18T08:15:01.379635+0000","last_change":"2024-09-18T08:14:39.389703+0000","last_active":"2024-09-18T08:15:01.379635+0000","last_peered":"2024-09-18T08:15:01.379635+0000","last_clean":"2024-09-18T08:15:01.379635+0000","last_became_active":"2024-09-18T08:14:39.389465+0000","last_became_peered":"2024-09-18T08:14:39.389465+0000","last_unstale":"2024-09-18T08:15:01.379635+0000","last_undegraded":"2024-09-18T08:15:01.379635+0000","last_fullsized":"2024-09-18T08:15:01.379635+0000","mapping_epoch":307,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":308,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:10:59.645112+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:50.886830+0000","last_clean_scrub_stamp":"2024-09-18T08:10:59.645112+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":578,"reported_epoch":326,"state":"active+clean","last_fresh":"2024-09-18T08:15:07.720642+0000","last_change":"2024-09-18T08:15:05.719448+0000","last_active":"2024-09-18T08:15:07.720642+0000","last_peered":"2024-09-18T08:15:07.720642+0000","last_clean":"2024-09-18T08:15:07.720642+0000","last_became_active":"2024-09-18T08:14:52.673598+0000","last_became_peered":"2024-09-18T08:14:52.673598+0000","last_unstale":"2024-09-18T08:15:07.720642+0000","last_undegraded":"2024-09-18T08:15:07.720642+0000","last_fullsized":"2024-09-18T08:15:07.720642+0000","mapping_epoch":316,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":317,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:15:05.719269+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_clean_scrub_stamp":"2024-09-18T08:15:05.719269+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:16:26.405201+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,1],"acting":[3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":321,"reported_epoch":326,"state":"active+clean","last_fresh":"2024-09-18T08:15:07.716430+0000","last_change":"2024-09-18T08:14:40.774144+0000","last_active":"2024-09-18T08:15:07.716430+0000","last_peered":"2024-09-18T08:15:07.716430+0000","last_clean":"2024-09-18T08:15:07.716430+0000","last_became_active":"2024-09-18T08:14:39.376194+0000","last_became_peered":"2024-09-18T08:14:39.376194+0000","last_unstale":"2024-09-18T08:15:07.716430+0000","last_undegraded":"2024-09-18T08:15:07.716430+0000","last_fullsized":"2024-09-18T08:15:07.716430+0000","mapping_epoch":307,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":308,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:14:40.774086+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:46.860092+0000","last_clean_scrub_stamp":"2024-09-18T08:14:40.774086+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:15:44.963051+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"271'18","reported_seq":662,"reported_epoch":326,"state":"active+clean","last_fresh":"2024-09-18T08:15:07.722548+0000","last_change":"2024-09-18T08:14:39.453388+0000","last_active":"2024-09-18T08:15:07.722548+0000","last_peered":"2024-09-18T08:15:07.722548+0000","last_clean":"2024-09-18T08:15:07.722548+0000","last_became_active":"2024-09-18T08:14:39.381418+0000","last_became_peered":"2024-09-18T08:14:39.381418+0000","last_unstale":"2024-09-18T08:15:07.722548+0000","last_undegraded":"2024-09-18T08:15:07.722548+0000","last_fullsized":"2024-09-18T08:15:07.722548+0000","mapping_epoch":307,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":308,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'12","last_scrub_stamp":"2024-09-18T08:11:32.304472+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_clean_scrub_stamp":"2024-09-18T08:11:32.304472+0000","objects_scrubbed":5,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":34,"num_read_kb":25,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":310,"reported_epoch":326,"state":"active+clean","last_fresh":"2024-09-18T08:15:07.715861+0000","last_change":"2024-09-18T08:10:46.303912+0000","last_active":"2024-09-18T08:15:07.715861+0000","last_peered":"2024-09-18T08:15:07.715861+0000","last_clean":"2024-09-18T08:15:07.715861+0000","last_became_active":"2024-09-18T08:08:18.531438+0000","last_became_peered":"2024-09-18T08:08:18.531438+0000","last_unstale":"2024-09-18T08:15:07.715861+0000","last_undegraded":"2024-09-18T08:15:07.715861+0000","last_fullsized":"2024-09-18T08:15:07.715861+0000","mapping_epoch":181,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":182,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:10:46.303848+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:50.886830+0000","last_clean_scrub_stamp":"2024-09-18T08:10:46.303848+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1c","version":"69'1","reported_seq":55,"reported_epoch":326,"state":"active+clean","last_fresh":"2024-09-18T08:15:07.726311+0000","last_change":"2024-09-18T08:15:06.709065+0000","last_active":"2024-09-18T08:15:07.726311+0000","last_peered":"2024-09-18T08:15:07.726311+0000","last_clean":"2024-09-18T08:15:07.726311+0000","last_became_active":"2024-09-18T08:14:52.687145+0000","last_became_peered":"2024-09-18T08:14:52.687145+0000","last_unstale":"2024-09-18T08:15:07.726311+0000","last_undegraded":"2024-09-18T08:15:07.726311+0000","last_fullsized":"2024-09-18T08:15:07.726311+0000","mapping_epoch":316,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":317,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'1","last_scrub_stamp":"2024-09-18T08:15:06.708980+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_clean_scrub_stamp":"2024-09-18T08:15:06.708980+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:16:21.680740+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":403,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":403,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.19","version":"62'1","reported_seq":450,"reported_epoch":325,"state":"active+clean","last_fresh":"2024-09-18T08:15:01.378898+0000","last_change":"2024-09-18T08:14:39.414662+0000","last_active":"2024-09-18T08:15:01.378898+0000","last_peered":"2024-09-18T08:15:01.378898+0000","last_clean":"2024-09-18T08:15:01.378898+0000","last_became_active":"2024-09-18T08:14:39.414309+0000","last_became_peered":"2024-09-18T08:14:39.414309+0000","last_unstale":"2024-09-18T08:15:01.378898+0000","last_undegraded":"2024-09-18T08:15:01.378898+0000","last_fullsized":"2024-09-18T08:15:01.378898+0000","mapping_epoch":307,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":308,"parent":"0.0","parent_split_bits":0,"last_scrub":"62'1","last_scrub_stamp":"2024-09-18T08:11:00.634190+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:46.860092+0000","last_clean_scrub_stamp":"2024-09-18T08:11:00.634190+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1e","version":"271'20","reported_seq":611,"reported_epoch":326,"state":"active+clean","last_fresh":"2024-09-18T08:15:07.735741+0000","last_change":"2024-09-18T08:14:40.375239+0000","last_active":"2024-09-18T08:15:07.735741+0000","last_peered":"2024-09-18T08:15:07.735741+0000","last_clean":"2024-09-18T08:15:07.735741+0000","last_became_active":"2024-09-18T08:14:40.359879+0000","last_became_peered":"2024-09-18T08:14:40.359879+0000","last_unstale":"2024-09-18T08:15:07.735741+0000","last_undegraded":"2024-09-18T08:15:07.735741+0000","last_fullsized":"2024-09-18T08:15:07.735741+0000","mapping_epoch":308,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":309,"parent":"0.0","parent_split_bits":0,"last_scrub":"271'20","last_scrub_stamp":"2024-09-18T08:13:30.833726+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_clean_scrub_stamp":"2024-09-18T08:13:30.833726+0000","objects_scrubbed":5,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":25,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1],"acting":[0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1f","version":"293'14","reported_seq":410,"reported_epoch":326,"state":"active+clean","last_fresh":"2024-09-18T08:15:07.715761+0000","last_change":"2024-09-18T08:14:05.592694+0000","last_active":"2024-09-18T08:15:07.715761+0000","last_peered":"2024-09-18T08:15:07.715761+0000","last_clean":"2024-09-18T08:15:07.715761+0000","last_became_active":"2024-09-18T08:14:05.581783+0000","last_became_peered":"2024-09-18T08:14:05.581783+0000","last_unstale":"2024-09-18T08:15:07.715761+0000","last_undegraded":"2024-09-18T08:15:07.715761+0000","last_fullsized":"2024-09-18T08:15:07.715761+0000","mapping_epoch":292,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":293,"parent":"0.0","parent_split_bits":0,"last_scrub":"182'13","last_scrub_stamp":"2024-09-18T08:12:59.403054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:50.886830+0000","last_clean_scrub_stamp":"2024-09-18T08:12:59.403054+0000","objects_scrubbed":1,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,2],"acting":[4,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.1","version":"326'1797","reported_seq":1837,"reported_epoch":326,"state":"active+clean","last_fresh":"2024-09-18T08:15:08.385440+0000","last_change":"2024-09-18T08:14:44.432369+0000","last_active":"2024-09-18T08:15:08.385440+0000","last_peered":"2024-09-18T08:15:08.385440+0000","last_clean":"2024-09-18T08:15:08.385440+0000","last_became_active":"2024-09-18T08:14:44.432020+0000","last_became_peered":"2024-09-18T08:14:44.432020+0000","last_unstale":"2024-09-18T08:15:08.385440+0000","last_undegraded":"2024-09-18T08:15:08.385440+0000","last_fullsized":"2024-09-18T08:15:08.385440+0000","mapping_epoch":311,"log_start":"0'0","ondisk_log_start":"0'0","created":311,"last_epoch_clean":312,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:14:43.412748+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:14:43.412748+0000","last_clean_scrub_stamp":"2024-09-18T08:14:43.412748+0000","objects_scrubbed":0,"log_size":1797,"log_dups_size":0,"ondisk_log_size":1797,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:15:57.575857+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117768192,"num_objects":1797,"num_object_clones":0,"num_object_copies":5391,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1797,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1797,"num_write_kb":115008,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,1],"acting":[4,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.a","version":"74'79","reported_seq":35,"reported_epoch":326,"state":"active+clean","last_fresh":"2024-09-18T08:15:07.726607+0000","last_change":"2024-09-18T08:14:53.826448+0000","last_active":"2024-09-18T08:15:07.726607+0000","last_peered":"2024-09-18T08:15:07.726607+0000","last_clean":"2024-09-18T08:15:07.726607+0000","last_became_active":"2024-09-18T08:14:52.660773+0000","last_became_peered":"2024-09-18T08:14:52.660773+0000","last_unstale":"2024-09-18T08:15:07.726607+0000","last_undegraded":"2024-09-18T08:15:07.726607+0000","last_fullsized":"2024-09-18T08:15:07.726607+0000","mapping_epoch":316,"log_start":"0'0","ondisk_log_start":"0'0","created":152,"last_epoch_clean":317,"parent":"0.0","parent_split_bits":4,"last_scrub":"74'79","last_scrub_stamp":"2024-09-18T08:11:29.068355+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:00:11.893742+0000","last_clean_scrub_stamp":"2024-09-18T08:11:29.068355+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"user requested scrub scheduled @ 2024-09-18T08:15:23.691166+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":53669,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":10,"num_read_kb":20,"num_write":10,"num_write_kb":148,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":53669,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,4],"acting":[2,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.f","version":"271'27","reported_seq":494,"reported_epoch":325,"state":"active+clean","last_fresh":"2024-09-18T08:15:01.379885+0000","last_change":"2024-09-18T08:14:39.499528+0000","last_active":"2024-09-18T08:15:01.379885+0000","last_peered":"2024-09-18T08:15:01.379885+0000","last_clean":"2024-09-18T08:15:01.379885+0000","last_became_active":"2024-09-18T08:14:39.400606+0000","last_became_peered":"2024-09-18T08:14:39.400606+0000","last_unstale":"2024-09-18T08:15:01.379885+0000","last_undegraded":"2024-09-18T08:15:01.379885+0000","last_fullsized":"2024-09-18T08:15:01.379885+0000","mapping_epoch":307,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":308,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'15","last_scrub_stamp":"2024-09-18T08:10:41.548188+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_clean_scrub_stamp":"2024-09-18T08:10:41.548188+0000","objects_scrubbed":0,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":43,"num_read_kb":34,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":577,"reported_epoch":326,"state":"active+clean","last_fresh":"2024-09-18T08:15:07.719468+0000","last_change":"2024-09-18T08:14:40.354751+0000","last_active":"2024-09-18T08:15:07.719468+0000","last_peered":"2024-09-18T08:15:07.719468+0000","last_clean":"2024-09-18T08:15:07.719468+0000","last_became_active":"2024-09-18T08:14:40.354437+0000","last_became_peered":"2024-09-18T08:14:40.354437+0000","last_unstale":"2024-09-18T08:15:07.719468+0000","last_undegraded":"2024-09-18T08:15:07.719468+0000","last_fullsized":"2024-09-18T08:15:07.719468+0000","mapping_epoch":308,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":309,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:08:18.389273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:46.860092+0000","last_clean_scrub_stamp":"2024-09-18T08:08:18.389273+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:15:45.781017+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.e","version":"317'13","reported_seq":417,"reported_epoch":326,"state":"active+clean","last_fresh":"2024-09-18T08:15:07.715750+0000","last_change":"2024-09-18T08:14:52.698084+0000","last_active":"2024-09-18T08:15:07.715750+0000","last_peered":"2024-09-18T08:15:07.715750+0000","last_clean":"2024-09-18T08:15:07.715750+0000","last_became_active":"2024-09-18T08:14:52.664078+0000","last_became_peered":"2024-09-18T08:14:52.664078+0000","last_unstale":"2024-09-18T08:15:07.715750+0000","last_undegraded":"2024-09-18T08:15:07.715750+0000","last_fullsized":"2024-09-18T08:15:07.715750+0000","mapping_epoch":316,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":317,"parent":"0.0","parent_split_bits":0,"last_scrub":"182'12","last_scrub_stamp":"2024-09-18T08:13:31.620205+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:50.886830+0000","last_clean_scrub_stamp":"2024-09-18T08:13:31.620205+0000","objects_scrubbed":1,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,2],"acting":[4,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":559,"reported_epoch":325,"state":"active+clean","last_fresh":"2024-09-18T08:15:01.379538+0000","last_change":"2024-09-18T08:15:00.180364+0000","last_active":"2024-09-18T08:15:01.379538+0000","last_peered":"2024-09-18T08:15:01.379538+0000","last_clean":"2024-09-18T08:15:01.379538+0000","last_became_active":"2024-09-18T08:14:52.679357+0000","last_became_peered":"2024-09-18T08:14:52.679357+0000","last_unstale":"2024-09-18T08:15:01.379538+0000","last_undegraded":"2024-09-18T08:15:01.379538+0000","last_fullsized":"2024-09-18T08:15:01.379538+0000","mapping_epoch":316,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":317,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:15:00.180309+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_clean_scrub_stamp":"2024-09-18T08:15:00.180309+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:16:01.929095+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.e","version":"326'1775","reported_seq":1816,"reported_epoch":326,"state":"active+clean","last_fresh":"2024-09-18T08:15:08.371836+0000","last_change":"2024-09-18T08:14:44.435797+0000","last_active":"2024-09-18T08:15:08.371836+0000","last_peered":"2024-09-18T08:15:08.371836+0000","last_clean":"2024-09-18T08:15:08.371836+0000","last_became_active":"2024-09-18T08:14:44.435489+0000","last_became_peered":"2024-09-18T08:14:44.435489+0000","last_unstale":"2024-09-18T08:15:08.371836+0000","last_undegraded":"2024-09-18T08:15:08.371836+0000","last_fullsized":"2024-09-18T08:15:08.371836+0000","mapping_epoch":311,"log_start":"0'0","ondisk_log_start":"0'0","created":311,"last_epoch_clean":312,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:14:43.412748+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:14:43.412748+0000","last_clean_scrub_stamp":"2024-09-18T08:14:43.412748+0000","objects_scrubbed":0,"log_size":1775,"log_dups_size":0,"ondisk_log_size":1775,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:15:53.639006+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":116326400,"num_objects":1775,"num_object_clones":0,"num_object_copies":5325,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1775,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1775,"num_write_kb":113600,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,2],"acting":[4,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.5","version":"74'79","reported_seq":327,"reported_epoch":326,"state":"active+clean","last_fresh":"2024-09-18T08:15:07.719220+0000","last_change":"2024-09-18T08:11:54.503697+0000","last_active":"2024-09-18T08:15:07.719220+0000","last_peered":"2024-09-18T08:15:07.719220+0000","last_clean":"2024-09-18T08:15:07.719220+0000","last_became_active":"2024-09-18T08:08:52.690344+0000","last_became_peered":"2024-09-18T08:08:52.690344+0000","last_unstale":"2024-09-18T08:15:07.719220+0000","last_undegraded":"2024-09-18T08:15:07.719220+0000","last_fullsized":"2024-09-18T08:15:07.719220+0000","mapping_epoch":198,"log_start":"0'0","ondisk_log_start":"0'0","created":152,"last_epoch_clean":199,"parent":"0.0","parent_split_bits":3,"last_scrub":"74'79","last_scrub_stamp":"2024-09-18T08:11:54.503603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:00:11.893742+0000","last_clean_scrub_stamp":"2024-09-18T08:11:54.503603+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,6],"acting":[4,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.0","version":"271'30","reported_seq":654,"reported_epoch":326,"state":"active+clean","last_fresh":"2024-09-18T08:15:07.719214+0000","last_change":"2024-09-18T08:14:40.377265+0000","last_active":"2024-09-18T08:15:07.719214+0000","last_peered":"2024-09-18T08:15:07.719214+0000","last_clean":"2024-09-18T08:15:07.719214+0000","last_became_active":"2024-09-18T08:14:40.358956+0000","last_became_peered":"2024-09-18T08:14:40.358956+0000","last_unstale":"2024-09-18T08:15:07.719214+0000","last_undegraded":"2024-09-18T08:15:07.719214+0000","last_fullsized":"2024-09-18T08:15:07.719214+0000","mapping_epoch":308,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":309,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'18","last_scrub_stamp":"2024-09-18T08:12:36.956258+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_clean_scrub_stamp":"2024-09-18T08:12:36.956258+0000","objects_scrubbed":8,"log_size":30,"log_dups_size":0,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":52,"num_read_kb":40,"num_write":32,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":607,"reported_epoch":326,"state":"active+clean","last_fresh":"2024-09-18T08:15:08.780990+0000","last_change":"2024-09-18T08:15:08.780990+0000","last_active":"2024-09-18T08:15:08.780990+0000","last_peered":"2024-09-18T08:15:08.780990+0000","last_clean":"2024-09-18T08:15:08.780990+0000","last_became_active":"2024-09-18T08:14:52.674884+0000","last_became_peered":"2024-09-18T08:14:52.674884+0000","last_unstale":"2024-09-18T08:15:08.780990+0000","last_undegraded":"2024-09-18T08:15:08.780990+0000","last_fullsized":"2024-09-18T08:15:08.780990+0000","mapping_epoch":316,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":317,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:15:08.780913+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:46.860092+0000","last_clean_scrub_stamp":"2024-09-18T08:15:08.780913+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:16:20.828429+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":326,"reported_epoch":326,"state":"active+clean","last_fresh":"2024-09-18T08:15:07.716127+0000","last_change":"2024-09-18T08:14:05.588953+0000","last_active":"2024-09-18T08:15:07.716127+0000","last_peered":"2024-09-18T08:15:07.716127+0000","last_clean":"2024-09-18T08:15:07.716127+0000","last_became_active":"2024-09-18T08:14:05.588085+0000","last_became_peered":"2024-09-18T08:14:05.588085+0000","last_unstale":"2024-09-18T08:15:07.716127+0000","last_undegraded":"2024-09-18T08:15:07.716127+0000","last_fullsized":"2024-09-18T08:15:07.716127+0000","mapping_epoch":292,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":293,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:12:30.732539+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:50.886830+0000","last_clean_scrub_stamp":"2024-09-18T08:12:30.732539+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":319,"reported_epoch":326,"state":"active+clean","last_fresh":"2024-09-18T08:15:07.716521+0000","last_change":"2024-09-18T08:12:02.473229+0000","last_active":"2024-09-18T08:15:07.716521+0000","last_peered":"2024-09-18T08:15:07.716521+0000","last_clean":"2024-09-18T08:15:07.716521+0000","last_became_active":"2024-09-18T08:08:18.543694+0000","last_became_peered":"2024-09-18T08:08:18.543694+0000","last_unstale":"2024-09-18T08:15:07.716521+0000","last_undegraded":"2024-09-18T08:15:07.716521+0000","last_fullsized":"2024-09-18T08:15:07.716521+0000","mapping_epoch":181,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":182,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:12:02.473168+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_clean_scrub_stamp":"2024-09-18T08:12:02.473168+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.f","version":"326'1993","reported_seq":2032,"reported_epoch":326,"state":"active+clean","last_fresh":"2024-09-18T08:15:09.139759+0000","last_change":"2024-09-18T08:14:44.437378+0000","last_active":"2024-09-18T08:15:09.139759+0000","last_peered":"2024-09-18T08:15:09.139759+0000","last_clean":"2024-09-18T08:15:09.139759+0000","last_became_active":"2024-09-18T08:14:44.437103+0000","last_became_peered":"2024-09-18T08:14:44.437103+0000","last_unstale":"2024-09-18T08:15:09.139759+0000","last_undegraded":"2024-09-18T08:15:09.139759+0000","last_fullsized":"2024-09-18T08:15:09.139759+0000","mapping_epoch":311,"log_start":"0'0","ondisk_log_start":"0'0","created":311,"last_epoch_clean":312,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:14:43.412748+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:14:43.412748+0000","last_clean_scrub_stamp":"2024-09-18T08:14:43.412748+0000","objects_scrubbed":0,"log_size":1993,"log_dups_size":0,"ondisk_log_size":1993,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:16:00.051694+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":130613248,"num_objects":1993,"num_object_clones":0,"num_object_copies":5979,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1993,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1993,"num_write_kb":127552,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.4","version":"74'79","reported_seq":418,"reported_epoch":326,"state":"active+clean","last_fresh":"2024-09-18T08:15:07.723836+0000","last_change":"2024-09-18T08:14:52.695500+0000","last_active":"2024-09-18T08:15:07.723836+0000","last_peered":"2024-09-18T08:15:07.723836+0000","last_clean":"2024-09-18T08:15:07.723836+0000","last_became_active":"2024-09-18T08:14:52.680822+0000","last_became_peered":"2024-09-18T08:14:52.680822+0000","last_unstale":"2024-09-18T08:15:07.723836+0000","last_undegraded":"2024-09-18T08:15:07.723836+0000","last_fullsized":"2024-09-18T08:15:07.723836+0000","mapping_epoch":316,"log_start":"0'0","ondisk_log_start":"0'0","created":152,"last_epoch_clean":317,"parent":"0.0","parent_split_bits":3,"last_scrub":"74'79","last_scrub_stamp":"2024-09-18T08:12:37.920342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:00:11.893742+0000","last_clean_scrub_stamp":"2024-09-18T08:12:37.920342+0000","objects_scrubbed":1,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":327680,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":655360,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,4],"acting":[3,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"271'28","reported_seq":371,"reported_epoch":326,"state":"active+clean","last_fresh":"2024-09-18T08:15:07.716571+0000","last_change":"2024-09-18T08:14:52.690847+0000","last_active":"2024-09-18T08:15:07.716571+0000","last_peered":"2024-09-18T08:15:07.716571+0000","last_clean":"2024-09-18T08:15:07.716571+0000","last_became_active":"2024-09-18T08:14:52.648301+0000","last_became_peered":"2024-09-18T08:14:52.648301+0000","last_unstale":"2024-09-18T08:15:07.716571+0000","last_undegraded":"2024-09-18T08:15:07.716571+0000","last_fullsized":"2024-09-18T08:15:07.716571+0000","mapping_epoch":316,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":317,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'14","last_scrub_stamp":"2024-09-18T08:11:53.524045+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_clean_scrub_stamp":"2024-09-18T08:11:53.524045+0000","objects_scrubbed":7,"log_size":28,"log_dups_size":0,"ondisk_log_size":28,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":41,"num_read_kb":34,"num_write":28,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,0],"acting":[4,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"62'2","reported_seq":612,"reported_epoch":326,"state":"active+clean","last_fresh":"2024-09-18T08:15:07.736029+0000","last_change":"2024-09-18T08:14:42.145896+0000","last_active":"2024-09-18T08:15:07.736029+0000","last_peered":"2024-09-18T08:15:07.736029+0000","last_clean":"2024-09-18T08:15:07.736029+0000","last_became_active":"2024-09-18T08:14:39.452371+0000","last_became_peered":"2024-09-18T08:14:39.452371+0000","last_unstale":"2024-09-18T08:15:07.736029+0000","last_undegraded":"2024-09-18T08:15:07.736029+0000","last_fullsized":"2024-09-18T08:15:07.736029+0000","mapping_epoch":307,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":308,"parent":"0.0","parent_split_bits":0,"last_scrub":"62'2","last_scrub_stamp":"2024-09-18T08:14:42.145836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:46.860092+0000","last_clean_scrub_stamp":"2024-09-18T08:14:42.145836+0000","objects_scrubbed":2,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:15:44.657069+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1039,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":2078,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.0","version":"310'12","reported_seq":746,"reported_epoch":326,"state":"active+clean","last_fresh":"2024-09-18T08:15:07.719056+0000","last_change":"2024-09-18T08:14:39.404911+0000","last_active":"2024-09-18T08:15:07.719056+0000","last_peered":"2024-09-18T08:15:07.719056+0000","last_clean":"2024-09-18T08:15:07.719056+0000","last_became_active":"2024-09-18T08:14:39.387233+0000","last_became_peered":"2024-09-18T08:14:39.387233+0000","last_unstale":"2024-09-18T08:15:07.719056+0000","last_undegraded":"2024-09-18T08:15:07.719056+0000","last_fullsized":"2024-09-18T08:15:07.719056+0000","mapping_epoch":307,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":308,"parent":"0.0","parent_split_bits":0,"last_scrub":"182'10","last_scrub_stamp":"2024-09-18T08:09:51.467349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:50.886830+0000","last_clean_scrub_stamp":"2024-09-18T08:09:51.467349+0000","objects_scrubbed":1,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:15:47.809103+0000","scrub_duration":19,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":195,"reported_epoch":326,"state":"active+clean","last_fresh":"2024-09-18T08:15:07.727091+0000","last_change":"2024-09-18T08:14:52.675479+0000","last_active":"2024-09-18T08:15:07.727091+0000","last_peered":"2024-09-18T08:15:07.727091+0000","last_clean":"2024-09-18T08:15:07.727091+0000","last_became_active":"2024-09-18T08:14:52.675157+0000","last_became_peered":"2024-09-18T08:14:52.675157+0000","last_unstale":"2024-09-18T08:15:07.727091+0000","last_undegraded":"2024-09-18T08:15:07.727091+0000","last_fullsized":"2024-09-18T08:15:07.727091+0000","mapping_epoch":316,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":317,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:13:33.138734+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_clean_scrub_stamp":"2024-09-18T08:13:33.138734+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,1],"acting":[2,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.c","version":"326'1771","reported_seq":1812,"reported_epoch":326,"state":"active+clean","last_fresh":"2024-09-18T08:15:08.390170+0000","last_change":"2024-09-18T08:14:44.436817+0000","last_active":"2024-09-18T08:15:08.390170+0000","last_peered":"2024-09-18T08:15:08.390170+0000","last_clean":"2024-09-18T08:15:08.390170+0000","last_became_active":"2024-09-18T08:14:44.436502+0000","last_became_peered":"2024-09-18T08:14:44.436502+0000","last_unstale":"2024-09-18T08:15:08.390170+0000","last_undegraded":"2024-09-18T08:15:08.390170+0000","last_fullsized":"2024-09-18T08:15:08.390170+0000","mapping_epoch":311,"log_start":"0'0","ondisk_log_start":"0'0","created":311,"last_epoch_clean":312,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:14:43.412748+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:14:43.412748+0000","last_clean_scrub_stamp":"2024-09-18T08:14:43.412748+0000","objects_scrubbed":0,"log_size":1771,"log_dups_size":0,"ondisk_log_size":1771,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:15:48.229102+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":116064256,"num_objects":1771,"num_object_clones":0,"num_object_copies":5313,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1771,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1771,"num_write_kb":113344,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,3],"acting":[4,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.7","version":"74'79","reported_seq":405,"reported_epoch":326,"state":"active+clean","last_fresh":"2024-09-18T08:15:07.720985+0000","last_change":"2024-09-18T08:14:52.686947+0000","last_active":"2024-09-18T08:15:07.720985+0000","last_peered":"2024-09-18T08:15:07.720985+0000","last_clean":"2024-09-18T08:15:07.720985+0000","last_became_active":"2024-09-18T08:14:52.686060+0000","last_became_peered":"2024-09-18T08:14:52.686060+0000","last_unstale":"2024-09-18T08:15:07.720985+0000","last_undegraded":"2024-09-18T08:15:07.720985+0000","last_fullsized":"2024-09-18T08:15:07.720985+0000","mapping_epoch":316,"log_start":"0'0","ondisk_log_start":"0'0","created":152,"last_epoch_clean":317,"parent":"0.0","parent_split_bits":3,"last_scrub":"74'79","last_scrub_stamp":"2024-09-18T08:12:33.846913+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:00:11.893742+0000","last_clean_scrub_stamp":"2024-09-18T08:12:33.846913+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.2","version":"271'20","reported_seq":534,"reported_epoch":325,"state":"active+clean","last_fresh":"2024-09-18T08:15:01.379368+0000","last_change":"2024-09-18T08:14:52.711880+0000","last_active":"2024-09-18T08:15:01.379368+0000","last_peered":"2024-09-18T08:15:01.379368+0000","last_clean":"2024-09-18T08:15:01.379368+0000","last_became_active":"2024-09-18T08:14:52.676563+0000","last_became_peered":"2024-09-18T08:14:52.676563+0000","last_unstale":"2024-09-18T08:15:01.379368+0000","last_undegraded":"2024-09-18T08:15:01.379368+0000","last_fullsized":"2024-09-18T08:15:01.379368+0000","mapping_epoch":316,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":317,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'10","last_scrub_stamp":"2024-09-18T08:11:24.347134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_clean_scrub_stamp":"2024-09-18T08:11:24.347134+0000","objects_scrubbed":0,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":28,"num_read_kb":23,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":32,"reported_epoch":326,"state":"active+clean","last_fresh":"2024-09-18T08:15:07.722728+0000","last_change":"2024-09-18T08:14:52.678957+0000","last_active":"2024-09-18T08:15:07.722728+0000","last_peered":"2024-09-18T08:15:07.722728+0000","last_clean":"2024-09-18T08:15:07.722728+0000","last_became_active":"2024-09-18T08:14:52.678655+0000","last_became_peered":"2024-09-18T08:14:52.678655+0000","last_unstale":"2024-09-18T08:15:07.722728+0000","last_undegraded":"2024-09-18T08:15:07.722728+0000","last_fullsized":"2024-09-18T08:15:07.722728+0000","mapping_epoch":316,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":317,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:12:06.876541+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:46.860092+0000","last_clean_scrub_stamp":"2024-09-18T08:12:06.876541+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,6],"acting":[3,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.3","version":"317'11","reported_seq":751,"reported_epoch":326,"state":"active+clean","last_fresh":"2024-09-18T08:15:07.733201+0000","last_change":"2024-09-18T08:14:59.207730+0000","last_active":"2024-09-18T08:15:07.733201+0000","last_peered":"2024-09-18T08:15:07.733201+0000","last_clean":"2024-09-18T08:15:07.733201+0000","last_became_active":"2024-09-18T08:14:52.658272+0000","last_became_peered":"2024-09-18T08:14:52.658272+0000","last_unstale":"2024-09-18T08:15:07.733201+0000","last_undegraded":"2024-09-18T08:15:07.733201+0000","last_fullsized":"2024-09-18T08:15:07.733201+0000","mapping_epoch":316,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":317,"parent":"0.0","parent_split_bits":0,"last_scrub":"317'11","last_scrub_stamp":"2024-09-18T08:14:59.207667+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:50.886830+0000","last_clean_scrub_stamp":"2024-09-18T08:14:59.207667+0000","objects_scrubbed":1,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:16:23.090265+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,3],"acting":[0,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":588,"reported_epoch":326,"state":"active+clean","last_fresh":"2024-09-18T08:15:07.731434+0000","last_change":"2024-09-18T08:14:26.087160+0000","last_active":"2024-09-18T08:15:07.731434+0000","last_peered":"2024-09-18T08:15:07.731434+0000","last_clean":"2024-09-18T08:15:07.731434+0000","last_became_active":"2024-09-18T08:08:52.700080+0000","last_became_peered":"2024-09-18T08:08:52.700080+0000","last_unstale":"2024-09-18T08:15:07.731434+0000","last_undegraded":"2024-09-18T08:15:07.731434+0000","last_fullsized":"2024-09-18T08:15:07.731434+0000","mapping_epoch":198,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":199,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:14:26.087098+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_clean_scrub_stamp":"2024-09-18T08:14:26.087098+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:15:35.849543+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.d","version":"326'1721","reported_seq":1762,"reported_epoch":326,"state":"active+clean","last_fresh":"2024-09-18T08:15:08.381999+0000","last_change":"2024-09-18T08:14:44.430244+0000","last_active":"2024-09-18T08:15:08.381999+0000","last_peered":"2024-09-18T08:15:08.381999+0000","last_clean":"2024-09-18T08:15:08.381999+0000","last_became_active":"2024-09-18T08:14:44.429930+0000","last_became_peered":"2024-09-18T08:14:44.429930+0000","last_unstale":"2024-09-18T08:15:08.381999+0000","last_undegraded":"2024-09-18T08:15:08.381999+0000","last_fullsized":"2024-09-18T08:15:08.381999+0000","mapping_epoch":311,"log_start":"0'0","ondisk_log_start":"0'0","created":311,"last_epoch_clean":312,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:14:43.412748+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:14:43.412748+0000","last_clean_scrub_stamp":"2024-09-18T08:14:43.412748+0000","objects_scrubbed":0,"log_size":1721,"log_dups_size":0,"ondisk_log_size":1721,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:16:11.045324+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112787456,"num_objects":1721,"num_object_clones":0,"num_object_copies":5163,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1721,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1721,"num_write_kb":110144,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,0],"acting":[4,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.6","version":"74'79","reported_seq":479,"reported_epoch":326,"state":"active+clean","last_fresh":"2024-09-18T08:15:07.734049+0000","last_change":"2024-09-18T08:14:02.837236+0000","last_active":"2024-09-18T08:15:07.734049+0000","last_peered":"2024-09-18T08:15:07.734049+0000","last_clean":"2024-09-18T08:15:07.734049+0000","last_became_active":"2024-09-18T08:07:26.426532+0000","last_became_peered":"2024-09-18T08:07:26.426532+0000","last_unstale":"2024-09-18T08:15:07.734049+0000","last_undegraded":"2024-09-18T08:15:07.734049+0000","last_fullsized":"2024-09-18T08:15:07.734049+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":152,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":3,"last_scrub":"74'79","last_scrub_stamp":"2024-09-18T08:14:02.837167+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:00:11.893742+0000","last_clean_scrub_stamp":"2024-09-18T08:14:02.837167+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:15:32.552339+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,3],"acting":[0,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.3","version":"271'29","reported_seq":670,"reported_epoch":326,"state":"active+clean","last_fresh":"2024-09-18T08:15:07.733253+0000","last_change":"2024-09-18T08:14:52.699901+0000","last_active":"2024-09-18T08:15:07.733253+0000","last_peered":"2024-09-18T08:15:07.733253+0000","last_clean":"2024-09-18T08:15:07.733253+0000","last_became_active":"2024-09-18T08:14:52.684649+0000","last_became_peered":"2024-09-18T08:14:52.684649+0000","last_unstale":"2024-09-18T08:15:07.733253+0000","last_undegraded":"2024-09-18T08:15:07.733253+0000","last_fullsized":"2024-09-18T08:15:07.733253+0000","mapping_epoch":316,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":317,"parent":"0.0","parent_split_bits":0,"last_scrub":"271'29","last_scrub_stamp":"2024-09-18T08:13:33.814992+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_clean_scrub_stamp":"2024-09-18T08:13:33.814992+0000","objects_scrubbed":8,"log_size":29,"log_dups_size":0,"ondisk_log_size":29,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":54,"num_read_kb":40,"num_write":32,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":660,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.4","version":"62'1","reported_seq":434,"reported_epoch":325,"state":"active+clean","last_fresh":"2024-09-18T08:15:01.379654+0000","last_change":"2024-09-18T08:14:52.706224+0000","last_active":"2024-09-18T08:15:01.379654+0000","last_peered":"2024-09-18T08:15:01.379654+0000","last_clean":"2024-09-18T08:15:01.379654+0000","last_became_active":"2024-09-18T08:14:52.671462+0000","last_became_peered":"2024-09-18T08:14:52.671462+0000","last_unstale":"2024-09-18T08:15:01.379654+0000","last_undegraded":"2024-09-18T08:15:01.379654+0000","last_fullsized":"2024-09-18T08:15:01.379654+0000","mapping_epoch":316,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":317,"parent":"0.0","parent_split_bits":0,"last_scrub":"62'1","last_scrub_stamp":"2024-09-18T08:10:07.593004+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:46.860092+0000","last_clean_scrub_stamp":"2024-09-18T08:10:07.593004+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":436,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":872,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":494,"reported_epoch":326,"state":"active+clean","last_fresh":"2024-09-18T08:15:07.732475+0000","last_change":"2024-09-18T08:14:52.674482+0000","last_active":"2024-09-18T08:15:07.732475+0000","last_peered":"2024-09-18T08:15:07.732475+0000","last_clean":"2024-09-18T08:15:07.732475+0000","last_became_active":"2024-09-18T08:14:52.674161+0000","last_became_peered":"2024-09-18T08:14:52.674161+0000","last_unstale":"2024-09-18T08:15:07.732475+0000","last_undegraded":"2024-09-18T08:15:07.732475+0000","last_fullsized":"2024-09-18T08:15:07.732475+0000","mapping_epoch":316,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":317,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:12:37.732260+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:50.886830+0000","last_clean_scrub_stamp":"2024-09-18T08:12:37.732260+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,2],"acting":[0,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":464,"reported_epoch":325,"state":"active+clean","last_fresh":"2024-09-18T08:15:01.378660+0000","last_change":"2024-09-18T08:14:39.358424+0000","last_active":"2024-09-18T08:15:01.378660+0000","last_peered":"2024-09-18T08:15:01.378660+0000","last_clean":"2024-09-18T08:15:01.378660+0000","last_became_active":"2024-09-18T08:14:39.358132+0000","last_became_peered":"2024-09-18T08:14:39.358132+0000","last_unstale":"2024-09-18T08:15:01.378660+0000","last_undegraded":"2024-09-18T08:15:01.378660+0000","last_fullsized":"2024-09-18T08:15:01.378660+0000","mapping_epoch":307,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":308,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:11:28.346395+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_clean_scrub_stamp":"2024-09-18T08:11:28.346395+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.a","version":"326'1922","reported_seq":3139,"reported_epoch":326,"state":"active+clean","last_fresh":"2024-09-18T08:15:09.204908+0000","last_change":"2024-09-18T08:15:00.359731+0000","last_active":"2024-09-18T08:15:09.204908+0000","last_peered":"2024-09-18T08:15:09.204908+0000","last_clean":"2024-09-18T08:15:09.204908+0000","last_became_active":"2024-09-18T08:15:00.356111+0000","last_became_peered":"2024-09-18T08:15:00.356111+0000","last_unstale":"2024-09-18T08:15:09.204908+0000","last_undegraded":"2024-09-18T08:15:09.204908+0000","last_fullsized":"2024-09-18T08:15:09.204908+0000","mapping_epoch":323,"log_start":"0'0","ondisk_log_start":"0'0","created":311,"last_epoch_clean":324,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:14:43.412748+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:14:43.412748+0000","last_clean_scrub_stamp":"2024-09-18T08:14:43.412748+0000","objects_scrubbed":0,"log_size":1922,"log_dups_size":0,"ondisk_log_size":1922,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:15:46.641425+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":125960192,"num_objects":1922,"num_object_clones":0,"num_object_copies":5766,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1922,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1922,"num_write_kb":123008,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":582,"num_bytes_recovered":38141952,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,2],"acting":[0,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"74'79","reported_seq":114,"reported_epoch":326,"state":"active+clean","last_fresh":"2024-09-18T08:15:07.732647+0000","last_change":"2024-09-18T08:14:53.660283+0000","last_active":"2024-09-18T08:15:07.732647+0000","last_peered":"2024-09-18T08:15:07.732647+0000","last_clean":"2024-09-18T08:15:07.732647+0000","last_became_active":"2024-09-18T08:14:52.680350+0000","last_became_peered":"2024-09-18T08:14:52.680350+0000","last_unstale":"2024-09-18T08:15:07.732647+0000","last_undegraded":"2024-09-18T08:15:07.732647+0000","last_fullsized":"2024-09-18T08:15:07.732647+0000","mapping_epoch":316,"log_start":"0'0","ondisk_log_start":"0'0","created":152,"last_epoch_clean":317,"parent":"0.0","parent_split_bits":4,"last_scrub":"74'79","last_scrub_stamp":"2024-09-18T08:13:55.194411+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:00:11.893742+0000","last_clean_scrub_stamp":"2024-09-18T08:13:55.194411+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"user requested scrub scheduled @ 2024-09-18T08:15:23.169136+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":53670,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":11,"num_read_kb":20,"num_write":11,"num_write_kb":148,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":53670,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,4],"acting":[0,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.4","version":"310'42","reported_seq":572,"reported_epoch":325,"state":"active+clean","last_fresh":"2024-09-18T08:15:01.379891+0000","last_change":"2024-09-18T08:14:39.529072+0000","last_active":"2024-09-18T08:15:01.379891+0000","last_peered":"2024-09-18T08:15:01.379891+0000","last_clean":"2024-09-18T08:15:01.379891+0000","last_became_active":"2024-09-18T08:14:39.486174+0000","last_became_peered":"2024-09-18T08:14:39.486174+0000","last_unstale":"2024-09-18T08:15:01.379891+0000","last_undegraded":"2024-09-18T08:15:01.379891+0000","last_fullsized":"2024-09-18T08:15:01.379891+0000","mapping_epoch":307,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":308,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'28","last_scrub_stamp":"2024-09-18T08:10:46.497874+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_clean_scrub_stamp":"2024-09-18T08:10:46.497874+0000","objects_scrubbed":0,"log_size":42,"log_dups_size":0,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":65,"num_read_kb":50,"num_write":38,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":311,"reported_epoch":326,"state":"active+clean","last_fresh":"2024-09-18T08:15:07.715257+0000","last_change":"2024-09-18T08:13:52.760213+0000","last_active":"2024-09-18T08:15:07.715257+0000","last_peered":"2024-09-18T08:15:07.715257+0000","last_clean":"2024-09-18T08:15:07.715257+0000","last_became_active":"2024-09-18T08:08:18.538852+0000","last_became_peered":"2024-09-18T08:08:18.538852+0000","last_unstale":"2024-09-18T08:15:07.715257+0000","last_undegraded":"2024-09-18T08:15:07.715257+0000","last_fullsized":"2024-09-18T08:15:07.715257+0000","mapping_epoch":181,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":182,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:13:52.760146+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:46.860092+0000","last_clean_scrub_stamp":"2024-09-18T08:13:52.760146+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"64'2","reported_seq":54,"reported_epoch":325,"state":"active+clean","last_fresh":"2024-09-18T08:15:01.379081+0000","last_change":"2024-09-18T08:14:52.694635+0000","last_active":"2024-09-18T08:15:01.379081+0000","last_peered":"2024-09-18T08:15:01.379081+0000","last_clean":"2024-09-18T08:15:01.379081+0000","last_became_active":"2024-09-18T08:14:52.670795+0000","last_became_peered":"2024-09-18T08:14:52.670795+0000","last_unstale":"2024-09-18T08:15:01.379081+0000","last_undegraded":"2024-09-18T08:15:01.379081+0000","last_fullsized":"2024-09-18T08:15:01.379081+0000","mapping_epoch":316,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":317,"parent":"0.0","parent_split_bits":0,"last_scrub":"64'2","last_scrub_stamp":"2024-09-18T08:05:44.802942+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:45.855979+0000","last_clean_scrub_stamp":"2024-09-18T08:05:44.802942+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:15:53.129301+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,0],"acting":[1,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":585,"reported_epoch":326,"state":"active+clean","last_fresh":"2024-09-18T08:15:07.737426+0000","last_change":"2024-09-18T08:15:04.260609+0000","last_active":"2024-09-18T08:15:07.737426+0000","last_peered":"2024-09-18T08:15:07.737426+0000","last_clean":"2024-09-18T08:15:07.737426+0000","last_became_active":"2024-09-18T08:14:39.397429+0000","last_became_peered":"2024-09-18T08:14:39.397429+0000","last_unstale":"2024-09-18T08:15:07.737426+0000","last_undegraded":"2024-09-18T08:15:07.737426+0000","last_fullsized":"2024-09-18T08:15:07.737426+0000","mapping_epoch":307,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":308,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:15:04.260540+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:50.886830+0000","last_clean_scrub_stamp":"2024-09-18T08:15:04.260540+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:16:09.549342+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":587,"reported_epoch":326,"state":"active+clean","last_fresh":"2024-09-18T08:15:07.720290+0000","last_change":"2024-09-18T08:14:05.578066+0000","last_active":"2024-09-18T08:15:07.720290+0000","last_peered":"2024-09-18T08:15:07.720290+0000","last_clean":"2024-09-18T08:15:07.720290+0000","last_became_active":"2024-09-18T08:14:05.577377+0000","last_became_peered":"2024-09-18T08:14:05.577377+0000","last_unstale":"2024-09-18T08:15:07.720290+0000","last_undegraded":"2024-09-18T08:15:07.720290+0000","last_fullsized":"2024-09-18T08:15:07.720290+0000","mapping_epoch":292,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":293,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:13:36.653164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_clean_scrub_stamp":"2024-09-18T08:13:36.653164+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,6],"acting":[3,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.9","version":"326'1881","reported_seq":3098,"reported_epoch":326,"state":"active+clean","last_fresh":"2024-09-18T08:15:08.826638+0000","last_change":"2024-09-18T08:15:01.375810+0000","last_active":"2024-09-18T08:15:08.826638+0000","last_peered":"2024-09-18T08:15:08.826638+0000","last_clean":"2024-09-18T08:15:08.826638+0000","last_became_active":"2024-09-18T08:15:01.373545+0000","last_became_peered":"2024-09-18T08:15:01.373545+0000","last_unstale":"2024-09-18T08:15:08.826638+0000","last_undegraded":"2024-09-18T08:15:08.826638+0000","last_fullsized":"2024-09-18T08:15:08.826638+0000","mapping_epoch":324,"log_start":"0'0","ondisk_log_start":"0'0","created":311,"last_epoch_clean":325,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:14:43.412748+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:14:43.412748+0000","last_clean_scrub_stamp":"2024-09-18T08:14:43.412748+0000","objects_scrubbed":0,"log_size":1881,"log_dups_size":0,"ondisk_log_size":1881,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:16:05.947077+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":123273216,"num_objects":1881,"num_object_clones":0,"num_object_copies":5643,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1881,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1881,"num_write_kb":120384,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":582,"num_bytes_recovered":38141952,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,3],"acting":[2,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"74'79","reported_seq":499,"reported_epoch":326,"state":"active+clean","last_fresh":"2024-09-18T08:15:07.732269+0000","last_change":"2024-09-18T08:14:05.586022+0000","last_active":"2024-09-18T08:15:07.732269+0000","last_peered":"2024-09-18T08:15:07.732269+0000","last_clean":"2024-09-18T08:15:07.732269+0000","last_became_active":"2024-09-18T08:14:05.585608+0000","last_became_peered":"2024-09-18T08:14:05.585608+0000","last_unstale":"2024-09-18T08:15:07.732269+0000","last_undegraded":"2024-09-18T08:15:07.732269+0000","last_fullsized":"2024-09-18T08:15:07.732269+0000","mapping_epoch":292,"log_start":"0'0","ondisk_log_start":"0'0","created":152,"last_epoch_clean":293,"parent":"0.0","parent_split_bits":4,"last_scrub":"74'79","last_scrub_stamp":"2024-09-18T08:14:01.846338+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:00:11.893742+0000","last_clean_scrub_stamp":"2024-09-18T08:14:01.846338+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:15:31.647562+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,2],"acting":[0,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.7","version":"271'17","reported_seq":544,"reported_epoch":326,"state":"active+clean","last_fresh":"2024-09-18T08:15:07.728384+0000","last_change":"2024-09-18T08:14:52.706583+0000","last_active":"2024-09-18T08:15:07.728384+0000","last_peered":"2024-09-18T08:15:07.728384+0000","last_clean":"2024-09-18T08:15:07.728384+0000","last_became_active":"2024-09-18T08:14:52.675852+0000","last_became_peered":"2024-09-18T08:14:52.675852+0000","last_unstale":"2024-09-18T08:15:07.728384+0000","last_undegraded":"2024-09-18T08:15:07.728384+0000","last_fullsized":"2024-09-18T08:15:07.728384+0000","mapping_epoch":316,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":317,"parent":"0.0","parent_split_bits":0,"last_scrub":"271'17","last_scrub_stamp":"2024-09-18T08:13:30.296881+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_clean_scrub_stamp":"2024-09-18T08:13:30.296881+0000","objects_scrubbed":5,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":35,"num_read_kb":24,"num_write":20,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":660,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":454,"reported_epoch":325,"state":"active+clean","last_fresh":"2024-09-18T08:15:01.378789+0000","last_change":"2024-09-18T08:14:39.432590+0000","last_active":"2024-09-18T08:15:01.378789+0000","last_peered":"2024-09-18T08:15:01.378789+0000","last_clean":"2024-09-18T08:15:01.378789+0000","last_became_active":"2024-09-18T08:14:39.432254+0000","last_became_peered":"2024-09-18T08:14:39.432254+0000","last_unstale":"2024-09-18T08:15:01.378789+0000","last_undegraded":"2024-09-18T08:15:01.378789+0000","last_fullsized":"2024-09-18T08:15:01.378789+0000","mapping_epoch":307,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":308,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:11:31.367525+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:46.860092+0000","last_clean_scrub_stamp":"2024-09-18T08:11:31.367525+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"62'1","reported_seq":586,"reported_epoch":326,"state":"active+clean","last_fresh":"2024-09-18T08:15:07.730374+0000","last_change":"2024-09-18T08:14:31.765507+0000","last_active":"2024-09-18T08:15:07.730374+0000","last_peered":"2024-09-18T08:15:07.730374+0000","last_clean":"2024-09-18T08:15:07.730374+0000","last_became_active":"2024-09-18T08:02:46.868175+0000","last_became_peered":"2024-09-18T08:02:46.868175+0000","last_unstale":"2024-09-18T08:15:07.730374+0000","last_undegraded":"2024-09-18T08:15:07.730374+0000","last_fullsized":"2024-09-18T08:15:07.730374+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"62'1","last_scrub_stamp":"2024-09-18T08:14:31.765455+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:45.855979+0000","last_clean_scrub_stamp":"2024-09-18T08:14:31.765455+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:15:42.305137+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.00066785499999999997,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":290,"reported_epoch":326,"state":"active+clean","last_fresh":"2024-09-18T08:15:07.730023+0000","last_change":"2024-09-18T08:15:00.702895+0000","last_active":"2024-09-18T08:15:07.730023+0000","last_peered":"2024-09-18T08:15:07.730023+0000","last_clean":"2024-09-18T08:15:07.730023+0000","last_became_active":"2024-09-18T08:14:52.663536+0000","last_became_peered":"2024-09-18T08:14:52.663536+0000","last_unstale":"2024-09-18T08:15:07.730023+0000","last_undegraded":"2024-09-18T08:15:07.730023+0000","last_fullsized":"2024-09-18T08:15:07.730023+0000","mapping_epoch":316,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":317,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:15:00.702835+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:50.886830+0000","last_clean_scrub_stamp":"2024-09-18T08:15:00.702835+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:16:20.118343+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,1],"acting":[2,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":258,"reported_epoch":326,"state":"active+clean","last_fresh":"2024-09-18T08:15:07.722354+0000","last_change":"2024-09-18T08:14:05.578356+0000","last_active":"2024-09-18T08:15:07.722354+0000","last_peered":"2024-09-18T08:15:07.722354+0000","last_clean":"2024-09-18T08:15:07.722354+0000","last_became_active":"2024-09-18T08:14:05.577516+0000","last_became_peered":"2024-09-18T08:14:05.577516+0000","last_unstale":"2024-09-18T08:15:07.722354+0000","last_undegraded":"2024-09-18T08:15:07.722354+0000","last_fullsized":"2024-09-18T08:15:07.722354+0000","mapping_epoch":292,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":293,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:13:36.978582+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_clean_scrub_stamp":"2024-09-18T08:13:36.978582+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.8","version":"326'1939","reported_seq":3183,"reported_epoch":326,"state":"active+clean","last_fresh":"2024-09-18T08:15:09.210072+0000","last_change":"2024-09-18T08:15:00.360983+0000","last_active":"2024-09-18T08:15:09.210072+0000","last_peered":"2024-09-18T08:15:09.210072+0000","last_clean":"2024-09-18T08:15:09.210072+0000","last_became_active":"2024-09-18T08:15:00.358350+0000","last_became_peered":"2024-09-18T08:15:00.358350+0000","last_unstale":"2024-09-18T08:15:09.210072+0000","last_undegraded":"2024-09-18T08:15:09.210072+0000","last_fullsized":"2024-09-18T08:15:09.210072+0000","mapping_epoch":323,"log_start":"0'0","ondisk_log_start":"0'0","created":311,"last_epoch_clean":324,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:14:43.412748+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:14:43.412748+0000","last_clean_scrub_stamp":"2024-09-18T08:14:43.412748+0000","objects_scrubbed":0,"log_size":1939,"log_dups_size":0,"ondisk_log_size":1939,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:15:59.704225+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":127074304,"num_objects":1939,"num_object_clones":0,"num_object_copies":5817,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1939,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1939,"num_write_kb":124096,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":596,"num_bytes_recovered":39059456,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,4],"acting":[0,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"74'79","reported_seq":222,"reported_epoch":325,"state":"active+clean","last_fresh":"2024-09-18T08:15:01.380420+0000","last_change":"2024-09-18T08:14:40.223015+0000","last_active":"2024-09-18T08:15:01.380420+0000","last_peered":"2024-09-18T08:15:01.380420+0000","last_clean":"2024-09-18T08:15:01.380420+0000","last_became_active":"2024-09-18T08:14:39.384188+0000","last_became_peered":"2024-09-18T08:14:39.384188+0000","last_unstale":"2024-09-18T08:15:01.380420+0000","last_undegraded":"2024-09-18T08:15:01.380420+0000","last_fullsized":"2024-09-18T08:15:01.380420+0000","mapping_epoch":307,"log_start":"0'0","ondisk_log_start":"0'0","created":152,"last_epoch_clean":308,"parent":"0.0","parent_split_bits":3,"last_scrub":"74'79","last_scrub_stamp":"2024-09-18T08:14:40.222933+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:00:11.893742+0000","last_clean_scrub_stamp":"2024-09-18T08:14:40.222933+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:15:59.106182+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.6","version":"271'24","reported_seq":668,"reported_epoch":326,"state":"active+clean","last_fresh":"2024-09-18T08:15:07.735834+0000","last_change":"2024-09-18T08:14:39.510679+0000","last_active":"2024-09-18T08:15:07.735834+0000","last_peered":"2024-09-18T08:15:07.735834+0000","last_clean":"2024-09-18T08:15:07.735834+0000","last_became_active":"2024-09-18T08:14:39.438033+0000","last_became_peered":"2024-09-18T08:14:39.438033+0000","last_unstale":"2024-09-18T08:15:07.735834+0000","last_undegraded":"2024-09-18T08:15:07.735834+0000","last_fullsized":"2024-09-18T08:15:07.735834+0000","mapping_epoch":307,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":308,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'12","last_scrub_stamp":"2024-09-18T08:10:58.837531+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_clean_scrub_stamp":"2024-09-18T08:10:58.837531+0000","objects_scrubbed":6,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":36,"num_read_kb":30,"num_write":24,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":598,"reported_epoch":326,"state":"active+clean","last_fresh":"2024-09-18T08:15:07.732931+0000","last_change":"2024-09-18T08:14:37.133590+0000","last_active":"2024-09-18T08:15:07.732931+0000","last_peered":"2024-09-18T08:15:07.732931+0000","last_clean":"2024-09-18T08:15:07.732931+0000","last_became_active":"2024-09-18T08:08:18.523135+0000","last_became_peered":"2024-09-18T08:08:18.523135+0000","last_unstale":"2024-09-18T08:15:07.732931+0000","last_undegraded":"2024-09-18T08:15:07.732931+0000","last_fullsized":"2024-09-18T08:15:07.732931+0000","mapping_epoch":181,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":182,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:14:37.133544+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:46.860092+0000","last_clean_scrub_stamp":"2024-09-18T08:14:37.133544+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:15:38.579695+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"70'5","reported_seq":554,"reported_epoch":325,"state":"active+clean","last_fresh":"2024-09-18T08:15:06.397326+0000","last_change":"2024-09-18T08:14:40.354226+0000","last_active":"2024-09-18T08:15:06.397326+0000","last_peered":"2024-09-18T08:15:06.397326+0000","last_clean":"2024-09-18T08:15:06.397326+0000","last_became_active":"2024-09-18T08:14:40.353801+0000","last_became_peered":"2024-09-18T08:14:40.353801+0000","last_unstale":"2024-09-18T08:15:06.397326+0000","last_undegraded":"2024-09-18T08:15:06.397326+0000","last_fullsized":"2024-09-18T08:15:06.397326+0000","mapping_epoch":308,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":309,"parent":"0.0","parent_split_bits":0,"last_scrub":"70'5","last_scrub_stamp":"2024-09-18T08:14:35.082297+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:45.855979+0000","last_clean_scrub_stamp":"2024-09-18T08:14:35.082297+0000","objects_scrubbed":1,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:15:43.041640+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.000504055,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":201,"num_read_kb":196,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":389,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":562,"reported_epoch":325,"state":"active+clean","last_fresh":"2024-09-18T08:15:02.179771+0000","last_change":"2024-09-18T08:15:02.179771+0000","last_active":"2024-09-18T08:15:02.179771+0000","last_peered":"2024-09-18T08:15:02.179771+0000","last_clean":"2024-09-18T08:15:02.179771+0000","last_became_active":"2024-09-18T08:14:52.659649+0000","last_became_peered":"2024-09-18T08:14:52.659649+0000","last_unstale":"2024-09-18T08:15:02.179771+0000","last_undegraded":"2024-09-18T08:15:02.179771+0000","last_fullsized":"2024-09-18T08:15:02.179771+0000","mapping_epoch":316,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":317,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:15:02.179693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:50.886830+0000","last_clean_scrub_stamp":"2024-09-18T08:15:02.179693+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:16:12.533133+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,4],"acting":[1,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":377,"reported_epoch":325,"state":"active+clean","last_fresh":"2024-09-18T08:15:07.219352+0000","last_change":"2024-09-18T08:15:07.219352+0000","last_active":"2024-09-18T08:15:07.219352+0000","last_peered":"2024-09-18T08:15:07.219352+0000","last_clean":"2024-09-18T08:15:07.219352+0000","last_became_active":"2024-09-18T08:14:52.682501+0000","last_became_peered":"2024-09-18T08:14:52.682501+0000","last_unstale":"2024-09-18T08:15:07.219352+0000","last_undegraded":"2024-09-18T08:15:07.219352+0000","last_fullsized":"2024-09-18T08:15:07.219352+0000","mapping_epoch":316,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":317,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:15:07.219279+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_clean_scrub_stamp":"2024-09-18T08:15:07.219279+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:16:14.605231+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,6],"acting":[1,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.b","version":"325'1792","reported_seq":1831,"reported_epoch":325,"state":"active+clean","last_fresh":"2024-09-18T08:15:07.404571+0000","last_change":"2024-09-18T08:14:44.432885+0000","last_active":"2024-09-18T08:15:07.404571+0000","last_peered":"2024-09-18T08:15:07.404571+0000","last_clean":"2024-09-18T08:15:07.404571+0000","last_became_active":"2024-09-18T08:14:44.432599+0000","last_became_peered":"2024-09-18T08:14:44.432599+0000","last_unstale":"2024-09-18T08:15:07.404571+0000","last_undegraded":"2024-09-18T08:15:07.404571+0000","last_fullsized":"2024-09-18T08:15:07.404571+0000","mapping_epoch":311,"log_start":"0'0","ondisk_log_start":"0'0","created":311,"last_epoch_clean":312,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:14:43.412748+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:14:43.412748+0000","last_clean_scrub_stamp":"2024-09-18T08:14:43.412748+0000","objects_scrubbed":0,"log_size":1792,"log_dups_size":0,"ondisk_log_size":1792,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:16:11.284218+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117440512,"num_objects":1792,"num_object_clones":0,"num_object_copies":5376,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1792,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1792,"num_write_kb":114688,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,0],"acting":[1,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.5","version":"293'34","reported_seq":464,"reported_epoch":326,"state":"active+clean","last_fresh":"2024-09-18T08:15:07.719438+0000","last_change":"2024-09-18T08:14:05.604747+0000","last_active":"2024-09-18T08:15:07.719438+0000","last_peered":"2024-09-18T08:15:07.719438+0000","last_clean":"2024-09-18T08:15:07.719438+0000","last_became_active":"2024-09-18T08:14:05.585190+0000","last_became_peered":"2024-09-18T08:14:05.585190+0000","last_unstale":"2024-09-18T08:15:07.719438+0000","last_undegraded":"2024-09-18T08:15:07.719438+0000","last_fullsized":"2024-09-18T08:15:07.719438+0000","mapping_epoch":292,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":293,"parent":"0.0","parent_split_bits":0,"last_scrub":"271'33","last_scrub_stamp":"2024-09-18T08:13:28.626987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_clean_scrub_stamp":"2024-09-18T08:13:28.626987+0000","objects_scrubbed":8,"log_size":34,"log_dups_size":0,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":24,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":42,"num_read_kb":32,"num_write":25,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":308,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,0],"acting":[4,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":590,"reported_epoch":326,"state":"active+clean","last_fresh":"2024-09-18T08:15:07.719926+0000","last_change":"2024-09-18T08:14:52.660184+0000","last_active":"2024-09-18T08:15:07.719926+0000","last_peered":"2024-09-18T08:15:07.719926+0000","last_clean":"2024-09-18T08:15:07.719926+0000","last_became_active":"2024-09-18T08:14:52.659804+0000","last_became_peered":"2024-09-18T08:14:52.659804+0000","last_unstale":"2024-09-18T08:15:07.719926+0000","last_undegraded":"2024-09-18T08:15:07.719926+0000","last_fullsized":"2024-09-18T08:15:07.719926+0000","mapping_epoch":316,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":317,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:14:32.693460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:46.860092+0000","last_clean_scrub_stamp":"2024-09-18T08:14:32.693460+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:15:56.988130+0000","scrub_duration":38,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,4],"acting":[3,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"74'79","reported_seq":187,"reported_epoch":326,"state":"active+clean","last_fresh":"2024-09-18T08:15:07.731752+0000","last_change":"2024-09-18T08:14:40.357413+0000","last_active":"2024-09-18T08:15:07.731752+0000","last_peered":"2024-09-18T08:15:07.731752+0000","last_clean":"2024-09-18T08:15:07.731752+0000","last_became_active":"2024-09-18T08:14:40.357063+0000","last_became_peered":"2024-09-18T08:14:40.357063+0000","last_unstale":"2024-09-18T08:15:07.731752+0000","last_undegraded":"2024-09-18T08:15:07.731752+0000","last_fullsized":"2024-09-18T08:15:07.731752+0000","mapping_epoch":308,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":309,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'79","last_scrub_stamp":"2024-09-18T08:13:51.810296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:00:11.893742+0000","last_clean_scrub_stamp":"2024-09-18T08:13:51.810296+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:15:15.992804+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,1],"acting":[0,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":427,"reported_epoch":326,"state":"active+clean","last_fresh":"2024-09-18T08:15:07.726203+0000","last_change":"2024-09-18T08:14:52.670532+0000","last_active":"2024-09-18T08:15:07.726203+0000","last_peered":"2024-09-18T08:15:07.726203+0000","last_clean":"2024-09-18T08:15:07.726203+0000","last_became_active":"2024-09-18T08:14:52.670120+0000","last_became_peered":"2024-09-18T08:14:52.670120+0000","last_unstale":"2024-09-18T08:15:07.726203+0000","last_undegraded":"2024-09-18T08:15:07.726203+0000","last_fullsized":"2024-09-18T08:15:07.726203+0000","mapping_epoch":316,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":317,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:14:38.727133+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:50.886830+0000","last_clean_scrub_stamp":"2024-09-18T08:14:38.727133+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:15:52.339814+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":293,"reported_epoch":326,"state":"active+clean","last_fresh":"2024-09-18T08:15:07.720554+0000","last_change":"2024-09-18T08:14:52.686618+0000","last_active":"2024-09-18T08:15:07.720554+0000","last_peered":"2024-09-18T08:15:07.720554+0000","last_clean":"2024-09-18T08:15:07.720554+0000","last_became_active":"2024-09-18T08:14:52.685204+0000","last_became_peered":"2024-09-18T08:14:52.685204+0000","last_unstale":"2024-09-18T08:15:07.720554+0000","last_undegraded":"2024-09-18T08:15:07.720554+0000","last_fullsized":"2024-09-18T08:15:07.720554+0000","mapping_epoch":316,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":317,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:12:05.886281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_clean_scrub_stamp":"2024-09-18T08:12:05.886281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,1],"acting":[3,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.0","version":"325'1753","reported_seq":1792,"reported_epoch":325,"state":"active+clean","last_fresh":"2024-09-18T08:15:07.420191+0000","last_change":"2024-09-18T08:14:44.432450+0000","last_active":"2024-09-18T08:15:07.420191+0000","last_peered":"2024-09-18T08:15:07.420191+0000","last_clean":"2024-09-18T08:15:07.420191+0000","last_became_active":"2024-09-18T08:14:44.432101+0000","last_became_peered":"2024-09-18T08:14:44.432101+0000","last_unstale":"2024-09-18T08:15:07.420191+0000","last_undegraded":"2024-09-18T08:15:07.420191+0000","last_fullsized":"2024-09-18T08:15:07.420191+0000","mapping_epoch":311,"log_start":"0'0","ondisk_log_start":"0'0","created":311,"last_epoch_clean":312,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:14:43.412748+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:14:43.412748+0000","last_clean_scrub_stamp":"2024-09-18T08:14:43.412748+0000","objects_scrubbed":0,"log_size":1753,"log_dups_size":0,"ondisk_log_size":1753,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:16:00.690018+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":114884608,"num_objects":1753,"num_object_clones":0,"num_object_copies":5259,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1753,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1753,"num_write_kb":112192,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.e","version":"271'19","reported_seq":357,"reported_epoch":326,"state":"active+clean","last_fresh":"2024-09-18T08:15:07.715587+0000","last_change":"2024-09-18T08:14:39.515354+0000","last_active":"2024-09-18T08:15:07.715587+0000","last_peered":"2024-09-18T08:15:07.715587+0000","last_clean":"2024-09-18T08:15:07.715587+0000","last_became_active":"2024-09-18T08:14:39.376973+0000","last_became_peered":"2024-09-18T08:14:39.376973+0000","last_unstale":"2024-09-18T08:15:07.715587+0000","last_undegraded":"2024-09-18T08:15:07.715587+0000","last_fullsized":"2024-09-18T08:15:07.715587+0000","mapping_epoch":307,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":308,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'11","last_scrub_stamp":"2024-09-18T08:10:53.406574+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_clean_scrub_stamp":"2024-09-18T08:10:53.406574+0000","objects_scrubbed":5,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":31,"num_read_kb":24,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":331,"reported_epoch":326,"state":"active+clean","last_fresh":"2024-09-18T08:15:07.716274+0000","last_change":"2024-09-18T08:14:40.355155+0000","last_active":"2024-09-18T08:15:07.716274+0000","last_peered":"2024-09-18T08:15:07.716274+0000","last_clean":"2024-09-18T08:15:07.716274+0000","last_became_active":"2024-09-18T08:14:40.354788+0000","last_became_peered":"2024-09-18T08:14:40.354788+0000","last_unstale":"2024-09-18T08:15:07.716274+0000","last_undegraded":"2024-09-18T08:15:07.716274+0000","last_fullsized":"2024-09-18T08:15:07.716274+0000","mapping_epoch":308,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":309,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:11:59.458232+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:46.860092+0000","last_clean_scrub_stamp":"2024-09-18T08:11:59.458232+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,1],"acting":[4,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":310,"reported_epoch":326,"state":"active+clean","last_fresh":"2024-09-18T08:15:07.715458+0000","last_change":"2024-09-18T08:14:52.674134+0000","last_active":"2024-09-18T08:15:07.715458+0000","last_peered":"2024-09-18T08:15:07.715458+0000","last_clean":"2024-09-18T08:15:07.715458+0000","last_became_active":"2024-09-18T08:14:52.673806+0000","last_became_peered":"2024-09-18T08:14:52.673806+0000","last_unstale":"2024-09-18T08:15:07.715458+0000","last_undegraded":"2024-09-18T08:15:07.715458+0000","last_fullsized":"2024-09-18T08:15:07.715458+0000","mapping_epoch":316,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":317,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:13:34.187158+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:50.886830+0000","last_clean_scrub_stamp":"2024-09-18T08:13:34.187158+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,2],"acting":[4,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":578,"reported_epoch":326,"state":"active+clean","last_fresh":"2024-09-18T08:15:07.724013+0000","last_change":"2024-09-18T08:15:02.861629+0000","last_active":"2024-09-18T08:15:07.724013+0000","last_peered":"2024-09-18T08:15:07.724013+0000","last_clean":"2024-09-18T08:15:07.724013+0000","last_became_active":"2024-09-18T08:14:52.680568+0000","last_became_peered":"2024-09-18T08:14:52.680568+0000","last_unstale":"2024-09-18T08:15:07.724013+0000","last_undegraded":"2024-09-18T08:15:07.724013+0000","last_fullsized":"2024-09-18T08:15:07.724013+0000","mapping_epoch":316,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":317,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:15:02.861116+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_clean_scrub_stamp":"2024-09-18T08:15:02.861116+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:16:09.588151+0000","scrub_duration":81,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,1],"acting":[3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.3","version":"326'1849","reported_seq":3174,"reported_epoch":326,"state":"active+clean","last_fresh":"2024-09-18T08:15:08.757076+0000","last_change":"2024-09-18T08:15:00.361856+0000","last_active":"2024-09-18T08:15:08.757076+0000","last_peered":"2024-09-18T08:15:08.757076+0000","last_clean":"2024-09-18T08:15:08.757076+0000","last_became_active":"2024-09-18T08:15:00.358946+0000","last_became_peered":"2024-09-18T08:15:00.358946+0000","last_unstale":"2024-09-18T08:15:08.757076+0000","last_undegraded":"2024-09-18T08:15:08.757076+0000","last_fullsized":"2024-09-18T08:15:08.757076+0000","mapping_epoch":323,"log_start":"0'0","ondisk_log_start":"0'0","created":311,"last_epoch_clean":324,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:14:43.412748+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:14:43.412748+0000","last_clean_scrub_stamp":"2024-09-18T08:14:43.412748+0000","objects_scrubbed":0,"log_size":1849,"log_dups_size":0,"ondisk_log_size":1849,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:16:00.012992+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":121176064,"num_objects":1849,"num_object_clones":0,"num_object_copies":5547,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1849,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1849,"num_write_kb":118336,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":636,"num_bytes_recovered":41680896,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.8","version":"74'79","reported_seq":44,"reported_epoch":326,"state":"active+clean","last_fresh":"2024-09-18T08:15:07.725969+0000","last_change":"2024-09-18T08:14:54.675481+0000","last_active":"2024-09-18T08:15:07.725969+0000","last_peered":"2024-09-18T08:15:07.725969+0000","last_clean":"2024-09-18T08:15:07.725969+0000","last_became_active":"2024-09-18T08:14:52.670253+0000","last_became_peered":"2024-09-18T08:14:52.670253+0000","last_unstale":"2024-09-18T08:15:07.725969+0000","last_undegraded":"2024-09-18T08:15:07.725969+0000","last_fullsized":"2024-09-18T08:15:07.725969+0000","mapping_epoch":316,"log_start":"0'0","ondisk_log_start":"0'0","created":152,"last_epoch_clean":317,"parent":"0.0","parent_split_bits":4,"last_scrub":"74'79","last_scrub_stamp":"2024-09-18T08:14:54.675411+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:00:11.893742+0000","last_clean_scrub_stamp":"2024-09-18T08:14:54.675411+0000","objects_scrubbed":1,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:16:18.103969+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262688,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.d","version":"271'31","reported_seq":378,"reported_epoch":326,"state":"active+clean","last_fresh":"2024-09-18T08:15:07.719500+0000","last_change":"2024-09-18T08:15:03.678756+0000","last_active":"2024-09-18T08:15:07.719500+0000","last_peered":"2024-09-18T08:15:07.719500+0000","last_clean":"2024-09-18T08:15:07.719500+0000","last_became_active":"2024-09-18T08:14:39.377819+0000","last_became_peered":"2024-09-18T08:14:39.377819+0000","last_unstale":"2024-09-18T08:15:07.719500+0000","last_undegraded":"2024-09-18T08:15:07.719500+0000","last_fullsized":"2024-09-18T08:15:07.719500+0000","mapping_epoch":307,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":308,"parent":"0.0","parent_split_bits":0,"last_scrub":"271'31","last_scrub_stamp":"2024-09-18T08:15:03.678670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_clean_scrub_stamp":"2024-09-18T08:15:03.678670+0000","objects_scrubbed":8,"log_size":31,"log_dups_size":0,"ondisk_log_size":31,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:16:30.987390+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":49,"num_read_kb":39,"num_write":32,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,1],"acting":[4,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":322,"reported_epoch":326,"state":"active+clean","last_fresh":"2024-09-18T08:15:07.716380+0000","last_change":"2024-09-18T08:14:39.375355+0000","last_active":"2024-09-18T08:15:07.716380+0000","last_peered":"2024-09-18T08:15:07.716380+0000","last_clean":"2024-09-18T08:15:07.716380+0000","last_became_active":"2024-09-18T08:14:39.375087+0000","last_became_peered":"2024-09-18T08:14:39.375087+0000","last_unstale":"2024-09-18T08:15:07.716380+0000","last_undegraded":"2024-09-18T08:15:07.716380+0000","last_fullsized":"2024-09-18T08:15:07.716380+0000","mapping_epoch":307,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":308,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:10:22.243148+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:46.860092+0000","last_clean_scrub_stamp":"2024-09-18T08:10:22.243148+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":462,"reported_epoch":325,"state":"active+clean","last_fresh":"2024-09-18T08:15:01.379553+0000","last_change":"2024-09-18T08:14:42.230469+0000","last_active":"2024-09-18T08:15:01.379553+0000","last_peered":"2024-09-18T08:15:01.379553+0000","last_clean":"2024-09-18T08:15:01.379553+0000","last_became_active":"2024-09-18T08:14:39.442274+0000","last_became_peered":"2024-09-18T08:14:39.442274+0000","last_unstale":"2024-09-18T08:15:01.379553+0000","last_undegraded":"2024-09-18T08:15:01.379553+0000","last_fullsized":"2024-09-18T08:15:01.379553+0000","mapping_epoch":307,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":308,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:14:42.230420+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:50.886830+0000","last_clean_scrub_stamp":"2024-09-18T08:14:42.230420+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:16:00.658277+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":292,"reported_epoch":326,"state":"active+clean","last_fresh":"2024-09-18T08:15:07.728244+0000","last_change":"2024-09-18T08:14:35.757447+0000","last_active":"2024-09-18T08:15:07.728244+0000","last_peered":"2024-09-18T08:15:07.728244+0000","last_clean":"2024-09-18T08:15:07.728244+0000","last_became_active":"2024-09-18T08:08:52.696300+0000","last_became_peered":"2024-09-18T08:08:52.696300+0000","last_unstale":"2024-09-18T08:15:07.728244+0000","last_undegraded":"2024-09-18T08:15:07.728244+0000","last_fullsized":"2024-09-18T08:15:07.728244+0000","mapping_epoch":198,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":199,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:14:35.757380+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_clean_scrub_stamp":"2024-09-18T08:14:35.757380+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:16:03.865522+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.2","version":"325'1736","reported_seq":1775,"reported_epoch":325,"state":"active+clean","last_fresh":"2024-09-18T08:15:07.415785+0000","last_change":"2024-09-18T08:14:44.437864+0000","last_active":"2024-09-18T08:15:07.415785+0000","last_peered":"2024-09-18T08:15:07.415785+0000","last_clean":"2024-09-18T08:15:07.415785+0000","last_became_active":"2024-09-18T08:14:44.437430+0000","last_became_peered":"2024-09-18T08:14:44.437430+0000","last_unstale":"2024-09-18T08:15:07.415785+0000","last_undegraded":"2024-09-18T08:15:07.415785+0000","last_fullsized":"2024-09-18T08:15:07.415785+0000","mapping_epoch":311,"log_start":"0'0","ondisk_log_start":"0'0","created":311,"last_epoch_clean":312,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:14:43.412748+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:14:43.412748+0000","last_clean_scrub_stamp":"2024-09-18T08:14:43.412748+0000","objects_scrubbed":0,"log_size":1736,"log_dups_size":0,"ondisk_log_size":1736,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:16:12.065164+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":113770496,"num_objects":1736,"num_object_clones":0,"num_object_copies":5208,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1736,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1736,"num_write_kb":111104,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"74'79","reported_seq":396,"reported_epoch":326,"state":"active+clean","last_fresh":"2024-09-18T08:15:07.718736+0000","last_change":"2024-09-18T08:14:52.679032+0000","last_active":"2024-09-18T08:15:07.718736+0000","last_peered":"2024-09-18T08:15:07.718736+0000","last_clean":"2024-09-18T08:15:07.718736+0000","last_became_active":"2024-09-18T08:14:52.678308+0000","last_became_peered":"2024-09-18T08:14:52.678308+0000","last_unstale":"2024-09-18T08:15:07.718736+0000","last_undegraded":"2024-09-18T08:15:07.718736+0000","last_fullsized":"2024-09-18T08:15:07.718736+0000","mapping_epoch":316,"log_start":"0'0","ondisk_log_start":"0'0","created":152,"last_epoch_clean":317,"parent":"0.0","parent_split_bits":4,"last_scrub":"74'79","last_scrub_stamp":"2024-09-18T08:13:52.510099+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:00:11.893742+0000","last_clean_scrub_stamp":"2024-09-18T08:13:52.510099+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:15:20.086990+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,6],"acting":[3,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.c","version":"271'20","reported_seq":330,"reported_epoch":326,"state":"active+clean","last_fresh":"2024-09-18T08:15:07.715812+0000","last_change":"2024-09-18T08:08:18.642452+0000","last_active":"2024-09-18T08:15:07.715812+0000","last_peered":"2024-09-18T08:15:07.715812+0000","last_clean":"2024-09-18T08:15:07.715812+0000","last_became_active":"2024-09-18T08:08:18.622980+0000","last_became_peered":"2024-09-18T08:08:18.622980+0000","last_unstale":"2024-09-18T08:15:07.715812+0000","last_undegraded":"2024-09-18T08:15:07.715812+0000","last_fullsized":"2024-09-18T08:15:07.715812+0000","mapping_epoch":181,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":182,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'10","last_scrub_stamp":"2024-09-18T08:05:41.474813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_clean_scrub_stamp":"2024-09-18T08:05:41.474813+0000","objects_scrubbed":0,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:15:52.641137+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":29,"num_read_kb":24,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":591,"reported_epoch":326,"state":"active+clean","last_fresh":"2024-09-18T08:15:07.722517+0000","last_change":"2024-09-18T08:14:04.626151+0000","last_active":"2024-09-18T08:15:07.722517+0000","last_peered":"2024-09-18T08:15:07.722517+0000","last_clean":"2024-09-18T08:15:07.722517+0000","last_became_active":"2024-09-18T08:08:18.542424+0000","last_became_peered":"2024-09-18T08:08:18.542424+0000","last_unstale":"2024-09-18T08:15:07.722517+0000","last_undegraded":"2024-09-18T08:15:07.722517+0000","last_fullsized":"2024-09-18T08:15:07.722517+0000","mapping_epoch":181,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":182,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:14:04.626103+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:46.860092+0000","last_clean_scrub_stamp":"2024-09-18T08:14:04.626103+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:15:13.477322+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.d","version":"317'10","reported_seq":757,"reported_epoch":326,"state":"active+clean","last_fresh":"2024-09-18T08:15:07.727222+0000","last_change":"2024-09-18T08:14:52.703675+0000","last_active":"2024-09-18T08:15:07.727222+0000","last_peered":"2024-09-18T08:15:07.727222+0000","last_clean":"2024-09-18T08:15:07.727222+0000","last_became_active":"2024-09-18T08:14:52.685439+0000","last_became_peered":"2024-09-18T08:14:52.685439+0000","last_unstale":"2024-09-18T08:15:07.727222+0000","last_undegraded":"2024-09-18T08:15:07.727222+0000","last_fullsized":"2024-09-18T08:15:07.727222+0000","mapping_epoch":316,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":317,"parent":"0.0","parent_split_bits":0,"last_scrub":"293'9","last_scrub_stamp":"2024-09-18T08:14:37.759701+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:50.886830+0000","last_clean_scrub_stamp":"2024-09-18T08:14:37.759701+0000","objects_scrubbed":1,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:15:47.359435+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":329,"reported_epoch":326,"state":"active+clean","last_fresh":"2024-09-18T08:15:07.716032+0000","last_change":"2024-09-18T08:14:39.380919+0000","last_active":"2024-09-18T08:15:07.716032+0000","last_peered":"2024-09-18T08:15:07.716032+0000","last_clean":"2024-09-18T08:15:07.716032+0000","last_became_active":"2024-09-18T08:14:39.380178+0000","last_became_peered":"2024-09-18T08:14:39.380178+0000","last_unstale":"2024-09-18T08:15:07.716032+0000","last_undegraded":"2024-09-18T08:15:07.716032+0000","last_fullsized":"2024-09-18T08:15:07.716032+0000","mapping_epoch":307,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":308,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:11:18.475967+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_clean_scrub_stamp":"2024-09-18T08:11:18.475967+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.5","version":"326'1918","reported_seq":1959,"reported_epoch":326,"state":"active+clean","last_fresh":"2024-09-18T08:15:08.833779+0000","last_change":"2024-09-18T08:14:44.432113+0000","last_active":"2024-09-18T08:15:08.833779+0000","last_peered":"2024-09-18T08:15:08.833779+0000","last_clean":"2024-09-18T08:15:08.833779+0000","last_became_active":"2024-09-18T08:14:44.431835+0000","last_became_peered":"2024-09-18T08:14:44.431835+0000","last_unstale":"2024-09-18T08:15:08.833779+0000","last_undegraded":"2024-09-18T08:15:08.833779+0000","last_fullsized":"2024-09-18T08:15:08.833779+0000","mapping_epoch":311,"log_start":"0'0","ondisk_log_start":"0'0","created":311,"last_epoch_clean":312,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:14:43.412748+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:14:43.412748+0000","last_clean_scrub_stamp":"2024-09-18T08:14:43.412748+0000","objects_scrubbed":0,"log_size":1918,"log_dups_size":0,"ondisk_log_size":1918,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:16:04.244471+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":125698048,"num_objects":1918,"num_object_clones":0,"num_object_copies":5754,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1918,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1918,"num_write_kb":122752,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.b","version":"271'15","reported_seq":643,"reported_epoch":326,"state":"active+clean","last_fresh":"2024-09-18T08:15:07.733169+0000","last_change":"2024-09-18T08:14:39.482777+0000","last_active":"2024-09-18T08:15:07.733169+0000","last_peered":"2024-09-18T08:15:07.733169+0000","last_clean":"2024-09-18T08:15:07.733169+0000","last_became_active":"2024-09-18T08:14:39.371063+0000","last_became_peered":"2024-09-18T08:14:39.371063+0000","last_unstale":"2024-09-18T08:15:07.733169+0000","last_undegraded":"2024-09-18T08:15:07.733169+0000","last_fullsized":"2024-09-18T08:15:07.733169+0000","mapping_epoch":307,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":308,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'9","last_scrub_stamp":"2024-09-18T08:10:47.928596+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_clean_scrub_stamp":"2024-09-18T08:10:47.928596+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:15:49.090755+0000","scrub_duration":16,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":206,"reported_epoch":326,"state":"active+clean","last_fresh":"2024-09-18T08:15:07.724727+0000","last_change":"2024-09-18T08:14:52.682594+0000","last_active":"2024-09-18T08:15:07.724727+0000","last_peered":"2024-09-18T08:15:07.724727+0000","last_clean":"2024-09-18T08:15:07.724727+0000","last_became_active":"2024-09-18T08:14:52.680202+0000","last_became_peered":"2024-09-18T08:14:52.680202+0000","last_unstale":"2024-09-18T08:15:07.724727+0000","last_undegraded":"2024-09-18T08:15:07.724727+0000","last_fullsized":"2024-09-18T08:15:07.724727+0000","mapping_epoch":316,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":317,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:11:22.076452+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:46.860092+0000","last_clean_scrub_stamp":"2024-09-18T08:11:22.076452+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,1],"acting":[3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":575,"reported_epoch":326,"state":"active+clean","last_fresh":"2024-09-18T08:15:07.726927+0000","last_change":"2024-09-18T08:08:18.509651+0000","last_active":"2024-09-18T08:15:07.726927+0000","last_peered":"2024-09-18T08:15:07.726927+0000","last_clean":"2024-09-18T08:15:07.726927+0000","last_became_active":"2024-09-18T08:08:18.509247+0000","last_became_peered":"2024-09-18T08:08:18.509247+0000","last_unstale":"2024-09-18T08:15:07.726927+0000","last_undegraded":"2024-09-18T08:15:07.726927+0000","last_fullsized":"2024-09-18T08:15:07.726927+0000","mapping_epoch":181,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":182,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:07:10.728679+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:50.886830+0000","last_clean_scrub_stamp":"2024-09-18T08:07:10.728679+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:15:52.643929+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,3],"acting":[2,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":594,"reported_epoch":326,"state":"active+clean","last_fresh":"2024-09-18T08:15:07.732798+0000","last_change":"2024-09-18T08:14:05.585049+0000","last_active":"2024-09-18T08:15:07.732798+0000","last_peered":"2024-09-18T08:15:07.732798+0000","last_clean":"2024-09-18T08:15:07.732798+0000","last_became_active":"2024-09-18T08:14:05.583887+0000","last_became_peered":"2024-09-18T08:14:05.583887+0000","last_unstale":"2024-09-18T08:15:07.732798+0000","last_undegraded":"2024-09-18T08:15:07.732798+0000","last_fullsized":"2024-09-18T08:15:07.732798+0000","mapping_epoch":292,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":293,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:12:38.702798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_clean_scrub_stamp":"2024-09-18T08:12:38.702798+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,3],"acting":[0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.4","version":"326'1914","reported_seq":1955,"reported_epoch":326,"state":"active+clean","last_fresh":"2024-09-18T08:15:09.187358+0000","last_change":"2024-09-18T08:14:44.437938+0000","last_active":"2024-09-18T08:15:09.187358+0000","last_peered":"2024-09-18T08:15:09.187358+0000","last_clean":"2024-09-18T08:15:09.187358+0000","last_became_active":"2024-09-18T08:14:44.437396+0000","last_became_peered":"2024-09-18T08:14:44.437396+0000","last_unstale":"2024-09-18T08:15:09.187358+0000","last_undegraded":"2024-09-18T08:15:09.187358+0000","last_fullsized":"2024-09-18T08:15:09.187358+0000","mapping_epoch":311,"log_start":"0'0","ondisk_log_start":"0'0","created":311,"last_epoch_clean":312,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:14:43.412748+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:14:43.412748+0000","last_clean_scrub_stamp":"2024-09-18T08:14:43.412748+0000","objects_scrubbed":0,"log_size":1914,"log_dups_size":0,"ondisk_log_size":1914,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:15:49.213907+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":125435904,"num_objects":1914,"num_object_clones":0,"num_object_copies":5742,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1914,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1914,"num_write_kb":122496,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.a","version":"271'35","reported_seq":464,"reported_epoch":325,"state":"active+clean","last_fresh":"2024-09-18T08:15:01.379477+0000","last_change":"2024-09-18T08:14:52.713747+0000","last_active":"2024-09-18T08:15:01.379477+0000","last_peered":"2024-09-18T08:15:01.379477+0000","last_clean":"2024-09-18T08:15:01.379477+0000","last_became_active":"2024-09-18T08:14:52.675761+0000","last_became_peered":"2024-09-18T08:14:52.675761+0000","last_unstale":"2024-09-18T08:15:01.379477+0000","last_undegraded":"2024-09-18T08:15:01.379477+0000","last_fullsized":"2024-09-18T08:15:01.379477+0000","mapping_epoch":316,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":317,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'19","last_scrub_stamp":"2024-09-18T08:10:19.603091+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_clean_scrub_stamp":"2024-09-18T08:10:19.603091+0000","objects_scrubbed":0,"log_size":35,"log_dups_size":0,"ondisk_log_size":35,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":55,"num_read_kb":44,"num_write":36,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":35,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,0],"acting":[1,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":540,"reported_epoch":326,"state":"active+clean","last_fresh":"2024-09-18T08:15:07.722197+0000","last_change":"2024-09-18T08:14:52.672030+0000","last_active":"2024-09-18T08:15:07.722197+0000","last_peered":"2024-09-18T08:15:07.722197+0000","last_clean":"2024-09-18T08:15:07.722197+0000","last_became_active":"2024-09-18T08:14:52.671709+0000","last_became_peered":"2024-09-18T08:14:52.671709+0000","last_unstale":"2024-09-18T08:15:07.722197+0000","last_undegraded":"2024-09-18T08:15:07.722197+0000","last_fullsized":"2024-09-18T08:15:07.722197+0000","mapping_epoch":316,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":317,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:12:31.205468+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:46.860092+0000","last_clean_scrub_stamp":"2024-09-18T08:12:31.205468+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,4],"acting":[3,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":283,"reported_epoch":326,"state":"active+clean","last_fresh":"2024-09-18T08:15:07.726991+0000","last_change":"2024-09-18T08:14:52.678658+0000","last_active":"2024-09-18T08:15:07.726991+0000","last_peered":"2024-09-18T08:15:07.726991+0000","last_clean":"2024-09-18T08:15:07.726991+0000","last_became_active":"2024-09-18T08:14:52.678295+0000","last_became_peered":"2024-09-18T08:14:52.678295+0000","last_unstale":"2024-09-18T08:15:07.726991+0000","last_undegraded":"2024-09-18T08:15:07.726991+0000","last_fullsized":"2024-09-18T08:15:07.726991+0000","mapping_epoch":316,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":317,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:10:41.682297+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:50.886830+0000","last_clean_scrub_stamp":"2024-09-18T08:10:41.682297+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":411,"reported_epoch":326,"state":"active+clean","last_fresh":"2024-09-18T08:15:07.726071+0000","last_change":"2024-09-18T08:14:28.801364+0000","last_active":"2024-09-18T08:15:07.726071+0000","last_peered":"2024-09-18T08:15:07.726071+0000","last_clean":"2024-09-18T08:15:07.726071+0000","last_became_active":"2024-09-18T08:14:05.578053+0000","last_became_peered":"2024-09-18T08:14:05.578053+0000","last_unstale":"2024-09-18T08:15:07.726071+0000","last_undegraded":"2024-09-18T08:15:07.726071+0000","last_fullsized":"2024-09-18T08:15:07.726071+0000","mapping_epoch":292,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":293,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:14:28.801076+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_clean_scrub_stamp":"2024-09-18T08:14:28.801076+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:15:37.886886+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.7","version":"326'1919","reported_seq":3138,"reported_epoch":326,"state":"active+clean","last_fresh":"2024-09-18T08:15:09.118459+0000","last_change":"2024-09-18T08:14:59.356126+0000","last_active":"2024-09-18T08:15:09.118459+0000","last_peered":"2024-09-18T08:15:09.118459+0000","last_clean":"2024-09-18T08:15:09.118459+0000","last_became_active":"2024-09-18T08:14:59.354517+0000","last_became_peered":"2024-09-18T08:14:59.354517+0000","last_unstale":"2024-09-18T08:15:09.118459+0000","last_undegraded":"2024-09-18T08:15:09.118459+0000","last_fullsized":"2024-09-18T08:15:09.118459+0000","mapping_epoch":322,"log_start":"0'0","ondisk_log_start":"0'0","created":311,"last_epoch_clean":323,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:14:43.412748+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:14:43.412748+0000","last_clean_scrub_stamp":"2024-09-18T08:14:43.412748+0000","objects_scrubbed":0,"log_size":1919,"log_dups_size":0,"ondisk_log_size":1919,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:15:58.185132+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":125763584,"num_objects":1919,"num_object_clones":0,"num_object_copies":5757,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1919,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1919,"num_write_kb":122816,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":585,"num_bytes_recovered":38338560,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.9","version":"271'18","reported_seq":370,"reported_epoch":326,"state":"active+clean","last_fresh":"2024-09-18T08:15:07.715305+0000","last_change":"2024-09-18T08:14:52.681876+0000","last_active":"2024-09-18T08:15:07.715305+0000","last_peered":"2024-09-18T08:15:07.715305+0000","last_clean":"2024-09-18T08:15:07.715305+0000","last_became_active":"2024-09-18T08:14:52.649706+0000","last_became_peered":"2024-09-18T08:14:52.649706+0000","last_unstale":"2024-09-18T08:15:07.715305+0000","last_undegraded":"2024-09-18T08:15:07.715305+0000","last_fullsized":"2024-09-18T08:15:07.715305+0000","mapping_epoch":316,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":317,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'12","last_scrub_stamp":"2024-09-18T08:11:27.607049+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_clean_scrub_stamp":"2024-09-18T08:11:27.607049+0000","objects_scrubbed":5,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":33,"num_read_kb":24,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":880,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":515,"reported_epoch":326,"state":"active+clean","last_fresh":"2024-09-18T08:15:07.715496+0000","last_change":"2024-09-18T08:14:40.355356+0000","last_active":"2024-09-18T08:15:07.715496+0000","last_peered":"2024-09-18T08:15:07.715496+0000","last_clean":"2024-09-18T08:15:07.715496+0000","last_became_active":"2024-09-18T08:14:40.355030+0000","last_became_peered":"2024-09-18T08:14:40.355030+0000","last_unstale":"2024-09-18T08:15:07.715496+0000","last_undegraded":"2024-09-18T08:15:07.715496+0000","last_fullsized":"2024-09-18T08:15:07.715496+0000","mapping_epoch":308,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":309,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:10:57.579577+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:46.860092+0000","last_clean_scrub_stamp":"2024-09-18T08:10:57.579577+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,0],"acting":[4,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":580,"reported_epoch":326,"state":"active+clean","last_fresh":"2024-09-18T08:15:07.726159+0000","last_change":"2024-09-18T08:14:41.671839+0000","last_active":"2024-09-18T08:15:07.726159+0000","last_peered":"2024-09-18T08:15:07.726159+0000","last_clean":"2024-09-18T08:15:07.726159+0000","last_became_active":"2024-09-18T08:14:39.399301+0000","last_became_peered":"2024-09-18T08:14:39.399301+0000","last_unstale":"2024-09-18T08:15:07.726159+0000","last_undegraded":"2024-09-18T08:15:07.726159+0000","last_fullsized":"2024-09-18T08:15:07.726159+0000","mapping_epoch":307,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":308,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:14:41.671793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:50.886830+0000","last_clean_scrub_stamp":"2024-09-18T08:14:41.671793+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:16:03.918137+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":582,"reported_epoch":326,"state":"active+clean","last_fresh":"2024-09-18T08:15:07.722133+0000","last_change":"2024-09-18T08:14:40.356757+0000","last_active":"2024-09-18T08:15:07.722133+0000","last_peered":"2024-09-18T08:15:07.722133+0000","last_clean":"2024-09-18T08:15:07.722133+0000","last_became_active":"2024-09-18T08:14:40.356451+0000","last_became_peered":"2024-09-18T08:14:40.356451+0000","last_unstale":"2024-09-18T08:15:07.722133+0000","last_undegraded":"2024-09-18T08:15:07.722133+0000","last_fullsized":"2024-09-18T08:15:07.722133+0000","mapping_epoch":308,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":309,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:11:21.372405+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_clean_scrub_stamp":"2024-09-18T08:11:21.372405+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.6","version":"326'1927","reported_seq":3172,"reported_epoch":326,"state":"active+clean","last_fresh":"2024-09-18T08:15:09.209552+0000","last_change":"2024-09-18T08:15:00.357304+0000","last_active":"2024-09-18T08:15:09.209552+0000","last_peered":"2024-09-18T08:15:09.209552+0000","last_clean":"2024-09-18T08:15:09.209552+0000","last_became_active":"2024-09-18T08:15:00.355826+0000","last_became_peered":"2024-09-18T08:15:00.355826+0000","last_unstale":"2024-09-18T08:15:09.209552+0000","last_undegraded":"2024-09-18T08:15:09.209552+0000","last_fullsized":"2024-09-18T08:15:09.209552+0000","mapping_epoch":323,"log_start":"0'0","ondisk_log_start":"0'0","created":311,"last_epoch_clean":324,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:14:43.412748+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:14:43.412748+0000","last_clean_scrub_stamp":"2024-09-18T08:14:43.412748+0000","objects_scrubbed":0,"log_size":1927,"log_dups_size":0,"ondisk_log_size":1927,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:15:44.733960+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":126287872,"num_objects":1927,"num_object_clones":0,"num_object_copies":5781,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1927,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1927,"num_write_kb":123328,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":596,"num_bytes_recovered":39059456,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,1],"acting":[0,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.8","version":"271'27","reported_seq":623,"reported_epoch":326,"state":"active+clean","last_fresh":"2024-09-18T08:15:07.714786+0000","last_change":"2024-09-18T08:15:00.680861+0000","last_active":"2024-09-18T08:15:07.714786+0000","last_peered":"2024-09-18T08:15:07.714786+0000","last_clean":"2024-09-18T08:15:07.714786+0000","last_became_active":"2024-09-18T08:14:52.649443+0000","last_became_peered":"2024-09-18T08:14:52.649443+0000","last_unstale":"2024-09-18T08:15:07.714786+0000","last_undegraded":"2024-09-18T08:15:07.714786+0000","last_fullsized":"2024-09-18T08:15:07.714786+0000","mapping_epoch":316,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":317,"parent":"0.0","parent_split_bits":0,"last_scrub":"271'27","last_scrub_stamp":"2024-09-18T08:15:00.680764+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_clean_scrub_stamp":"2024-09-18T08:15:00.680764+0000","objects_scrubbed":7,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:16:12.368714+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":44,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,3],"acting":[4,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.f","version":"62'2","reported_seq":389,"reported_epoch":326,"state":"active+clean","last_fresh":"2024-09-18T08:15:07.715540+0000","last_change":"2024-09-18T08:14:40.364260+0000","last_active":"2024-09-18T08:15:07.715540+0000","last_peered":"2024-09-18T08:15:07.715540+0000","last_clean":"2024-09-18T08:15:07.715540+0000","last_became_active":"2024-09-18T08:14:40.354828+0000","last_became_peered":"2024-09-18T08:14:40.354828+0000","last_unstale":"2024-09-18T08:15:07.715540+0000","last_undegraded":"2024-09-18T08:15:07.715540+0000","last_fullsized":"2024-09-18T08:15:07.715540+0000","mapping_epoch":308,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":309,"parent":"0.0","parent_split_bits":0,"last_scrub":"62'2","last_scrub_stamp":"2024-09-18T08:10:53.580444+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:46.860092+0000","last_clean_scrub_stamp":"2024-09-18T08:10:53.580444+0000","objects_scrubbed":2,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":4,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":184,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,1],"acting":[4,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.9","version":"310'16","reported_seq":489,"reported_epoch":326,"state":"active+clean","last_fresh":"2024-09-18T08:15:07.715037+0000","last_change":"2024-09-18T08:14:40.368033+0000","last_active":"2024-09-18T08:15:07.715037+0000","last_peered":"2024-09-18T08:15:07.715037+0000","last_clean":"2024-09-18T08:15:07.715037+0000","last_became_active":"2024-09-18T08:14:40.360883+0000","last_became_peered":"2024-09-18T08:14:40.360883+0000","last_unstale":"2024-09-18T08:15:07.715037+0000","last_undegraded":"2024-09-18T08:15:07.715037+0000","last_fullsized":"2024-09-18T08:15:07.715037+0000","mapping_epoch":308,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":309,"parent":"0.0","parent_split_bits":0,"last_scrub":"294'14","last_scrub_stamp":"2024-09-18T08:14:28.686640+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:50.886830+0000","last_clean_scrub_stamp":"2024-09-18T08:14:28.686640+0000","objects_scrubbed":1,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:15:49.989136+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":194,"reported_epoch":326,"state":"active+clean","last_fresh":"2024-09-18T08:15:07.732356+0000","last_change":"2024-09-18T08:14:52.672045+0000","last_active":"2024-09-18T08:15:07.732356+0000","last_peered":"2024-09-18T08:15:07.732356+0000","last_clean":"2024-09-18T08:15:07.732356+0000","last_became_active":"2024-09-18T08:14:52.671698+0000","last_became_peered":"2024-09-18T08:14:52.671698+0000","last_unstale":"2024-09-18T08:15:07.732356+0000","last_undegraded":"2024-09-18T08:15:07.732356+0000","last_fullsized":"2024-09-18T08:15:07.732356+0000","mapping_epoch":316,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":317,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:11:23.050755+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_clean_scrub_stamp":"2024-09-18T08:11:23.050755+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,2],"acting":[0,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":501,"reported_epoch":326,"state":"active+clean","last_fresh":"2024-09-18T08:15:07.731438+0000","last_change":"2024-09-18T08:14:52.677675+0000","last_active":"2024-09-18T08:15:07.731438+0000","last_peered":"2024-09-18T08:15:07.731438+0000","last_clean":"2024-09-18T08:15:07.731438+0000","last_became_active":"2024-09-18T08:14:52.677353+0000","last_became_peered":"2024-09-18T08:14:52.677353+0000","last_unstale":"2024-09-18T08:15:07.731438+0000","last_undegraded":"2024-09-18T08:15:07.731438+0000","last_fullsized":"2024-09-18T08:15:07.731438+0000","mapping_epoch":316,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":317,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:13:06.860320+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:46.860092+0000","last_clean_scrub_stamp":"2024-09-18T08:13:06.860320+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,3],"acting":[0,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.17","version":"271'12","reported_seq":632,"reported_epoch":326,"state":"active+clean","last_fresh":"2024-09-18T08:15:07.721065+0000","last_change":"2024-09-18T08:14:40.365750+0000","last_active":"2024-09-18T08:15:07.721065+0000","last_peered":"2024-09-18T08:15:07.721065+0000","last_clean":"2024-09-18T08:15:07.721065+0000","last_became_active":"2024-09-18T08:14:40.355751+0000","last_became_peered":"2024-09-18T08:14:40.355751+0000","last_unstale":"2024-09-18T08:15:07.721065+0000","last_undegraded":"2024-09-18T08:15:07.721065+0000","last_fullsized":"2024-09-18T08:15:07.721065+0000","mapping_epoch":308,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":309,"parent":"0.0","parent_split_bits":0,"last_scrub":"271'12","last_scrub_stamp":"2024-09-18T08:14:33.700931+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_clean_scrub_stamp":"2024-09-18T08:14:33.700931+0000","objects_scrubbed":3,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:15:52.526359+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":18,"num_read_kb":15,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":563,"reported_epoch":326,"state":"active+clean","last_fresh":"2024-09-18T08:15:07.747225+0000","last_change":"2024-09-18T08:15:07.747225+0000","last_active":"2024-09-18T08:15:07.747225+0000","last_peered":"2024-09-18T08:15:07.747225+0000","last_clean":"2024-09-18T08:15:07.747225+0000","last_became_active":"2024-09-18T08:14:52.661901+0000","last_became_peered":"2024-09-18T08:14:52.661901+0000","last_unstale":"2024-09-18T08:15:07.747225+0000","last_undegraded":"2024-09-18T08:15:07.747225+0000","last_fullsized":"2024-09-18T08:15:07.747225+0000","mapping_epoch":316,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":317,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:15:07.747147+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:50.886830+0000","last_clean_scrub_stamp":"2024-09-18T08:15:07.747147+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:16:25.205578+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":370,"reported_epoch":326,"state":"active+clean","last_fresh":"2024-09-18T08:15:07.719053+0000","last_change":"2024-09-18T08:14:52.658874+0000","last_active":"2024-09-18T08:15:07.719053+0000","last_peered":"2024-09-18T08:15:07.719053+0000","last_clean":"2024-09-18T08:15:07.719053+0000","last_became_active":"2024-09-18T08:14:52.658604+0000","last_became_peered":"2024-09-18T08:14:52.658604+0000","last_unstale":"2024-09-18T08:15:07.719053+0000","last_undegraded":"2024-09-18T08:15:07.719053+0000","last_fullsized":"2024-09-18T08:15:07.719053+0000","mapping_epoch":316,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":317,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:12:54.661578+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_clean_scrub_stamp":"2024-09-18T08:12:54.661578+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,0],"acting":[4,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.16","version":"271'15","reported_seq":606,"reported_epoch":326,"state":"active+clean","last_fresh":"2024-09-18T08:15:07.735935+0000","last_change":"2024-09-18T08:14:05.588589+0000","last_active":"2024-09-18T08:15:07.735935+0000","last_peered":"2024-09-18T08:15:07.735935+0000","last_clean":"2024-09-18T08:15:07.735935+0000","last_became_active":"2024-09-18T08:14:05.578519+0000","last_became_peered":"2024-09-18T08:14:05.578519+0000","last_unstale":"2024-09-18T08:15:07.735935+0000","last_undegraded":"2024-09-18T08:15:07.735935+0000","last_fullsized":"2024-09-18T08:15:07.735935+0000","mapping_epoch":292,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":293,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'9","last_scrub_stamp":"2024-09-18T08:09:44.116308+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_clean_scrub_stamp":"2024-09-18T08:09:44.116308+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:15:24.072965+0000","scrub_duration":43,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":325,"reported_epoch":326,"state":"active+clean","last_fresh":"2024-09-18T08:15:07.716046+0000","last_change":"2024-09-18T08:14:05.583783+0000","last_active":"2024-09-18T08:15:07.716046+0000","last_peered":"2024-09-18T08:15:07.716046+0000","last_clean":"2024-09-18T08:15:07.716046+0000","last_became_active":"2024-09-18T08:14:05.583435+0000","last_became_peered":"2024-09-18T08:14:05.583435+0000","last_unstale":"2024-09-18T08:15:07.716046+0000","last_undegraded":"2024-09-18T08:15:07.716046+0000","last_fullsized":"2024-09-18T08:15:07.716046+0000","mapping_epoch":292,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":293,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:12:03.433407+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:46.860092+0000","last_clean_scrub_stamp":"2024-09-18T08:12:03.433407+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,0],"acting":[4,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":577,"reported_epoch":326,"state":"active+clean","last_fresh":"2024-09-18T08:15:07.720860+0000","last_change":"2024-09-18T08:14:40.356857+0000","last_active":"2024-09-18T08:15:07.720860+0000","last_peered":"2024-09-18T08:15:07.720860+0000","last_clean":"2024-09-18T08:15:07.720860+0000","last_became_active":"2024-09-18T08:14:40.356531+0000","last_became_peered":"2024-09-18T08:14:40.356531+0000","last_unstale":"2024-09-18T08:15:07.720860+0000","last_undegraded":"2024-09-18T08:15:07.720860+0000","last_fullsized":"2024-09-18T08:15:07.720860+0000","mapping_epoch":308,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":309,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:11:17.424701+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:50.886830+0000","last_clean_scrub_stamp":"2024-09-18T08:11:17.424701+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":579,"reported_epoch":326,"state":"active+clean","last_fresh":"2024-09-18T08:15:07.726063+0000","last_change":"2024-09-18T08:14:05.582569+0000","last_active":"2024-09-18T08:15:07.726063+0000","last_peered":"2024-09-18T08:15:07.726063+0000","last_clean":"2024-09-18T08:15:07.726063+0000","last_became_active":"2024-09-18T08:14:05.582226+0000","last_became_peered":"2024-09-18T08:14:05.582226+0000","last_unstale":"2024-09-18T08:15:07.726063+0000","last_undegraded":"2024-09-18T08:15:07.726063+0000","last_fullsized":"2024-09-18T08:15:07.726063+0000","mapping_epoch":292,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":293,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:10:42.731845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_clean_scrub_stamp":"2024-09-18T08:10:42.731845+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:15:18.761012+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,0],"acting":[2,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.15","version":"271'15","reported_seq":509,"reported_epoch":326,"state":"active+clean","last_fresh":"2024-09-18T08:15:07.722452+0000","last_change":"2024-09-18T08:14:52.701580+0000","last_active":"2024-09-18T08:15:07.722452+0000","last_peered":"2024-09-18T08:15:07.722452+0000","last_clean":"2024-09-18T08:15:07.722452+0000","last_became_active":"2024-09-18T08:14:52.665818+0000","last_became_peered":"2024-09-18T08:14:52.665818+0000","last_unstale":"2024-09-18T08:15:07.722452+0000","last_undegraded":"2024-09-18T08:15:07.722452+0000","last_fullsized":"2024-09-18T08:15:07.722452+0000","mapping_epoch":316,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":317,"parent":"0.0","parent_split_bits":0,"last_scrub":"271'15","last_scrub_stamp":"2024-09-18T08:12:58.225295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_clean_scrub_stamp":"2024-09-18T08:12:58.225295+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":25,"num_read_kb":19,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,0],"acting":[3,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":596,"reported_epoch":326,"state":"active+clean","last_fresh":"2024-09-18T08:15:07.732575+0000","last_change":"2024-09-18T08:14:40.357339+0000","last_active":"2024-09-18T08:15:07.732575+0000","last_peered":"2024-09-18T08:15:07.732575+0000","last_clean":"2024-09-18T08:15:07.732575+0000","last_became_active":"2024-09-18T08:14:40.355709+0000","last_became_peered":"2024-09-18T08:14:40.355709+0000","last_unstale":"2024-09-18T08:15:07.732575+0000","last_undegraded":"2024-09-18T08:15:07.732575+0000","last_fullsized":"2024-09-18T08:15:07.732575+0000","mapping_epoch":308,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":309,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:14:28.097629+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:46.860092+0000","last_clean_scrub_stamp":"2024-09-18T08:14:28.097629+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:15:37.900561+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1],"acting":[0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.14","version":"310'16","reported_seq":777,"reported_epoch":326,"state":"active+clean","last_fresh":"2024-09-18T08:15:07.724160+0000","last_change":"2024-09-18T08:14:40.366956+0000","last_active":"2024-09-18T08:15:07.724160+0000","last_peered":"2024-09-18T08:15:07.724160+0000","last_clean":"2024-09-18T08:15:07.724160+0000","last_became_active":"2024-09-18T08:14:40.358971+0000","last_became_peered":"2024-09-18T08:14:40.358971+0000","last_unstale":"2024-09-18T08:15:07.724160+0000","last_undegraded":"2024-09-18T08:15:07.724160+0000","last_fullsized":"2024-09-18T08:15:07.724160+0000","mapping_epoch":308,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":309,"parent":"0.0","parent_split_bits":0,"last_scrub":"199'12","last_scrub_stamp":"2024-09-18T08:13:31.695354+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:50.886830+0000","last_clean_scrub_stamp":"2024-09-18T08:13:31.695354+0000","objects_scrubbed":1,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,1],"acting":[3,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":317,"reported_epoch":326,"state":"active+clean","last_fresh":"2024-09-18T08:15:07.715643+0000","last_change":"2024-09-18T08:14:52.664205+0000","last_active":"2024-09-18T08:15:07.715643+0000","last_peered":"2024-09-18T08:15:07.715643+0000","last_clean":"2024-09-18T08:15:07.715643+0000","last_became_active":"2024-09-18T08:14:52.663949+0000","last_became_peered":"2024-09-18T08:14:52.663949+0000","last_unstale":"2024-09-18T08:15:07.715643+0000","last_undegraded":"2024-09-18T08:15:07.715643+0000","last_fullsized":"2024-09-18T08:15:07.715643+0000","mapping_epoch":316,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":317,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:12:06.521774+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_clean_scrub_stamp":"2024-09-18T08:12:06.521774+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,0],"acting":[4,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.14","version":"271'20","reported_seq":638,"reported_epoch":326,"state":"active+clean","last_fresh":"2024-09-18T08:15:07.724577+0000","last_change":"2024-09-18T08:14:40.378165+0000","last_active":"2024-09-18T08:15:07.724577+0000","last_peered":"2024-09-18T08:15:07.724577+0000","last_clean":"2024-09-18T08:15:07.724577+0000","last_became_active":"2024-09-18T08:14:40.358983+0000","last_became_peered":"2024-09-18T08:14:40.358983+0000","last_unstale":"2024-09-18T08:15:07.724577+0000","last_undegraded":"2024-09-18T08:15:07.724577+0000","last_fullsized":"2024-09-18T08:15:07.724577+0000","mapping_epoch":308,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":309,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'10","last_scrub_stamp":"2024-09-18T08:12:39.946462+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_clean_scrub_stamp":"2024-09-18T08:12:39.946462+0000","objects_scrubbed":5,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":25,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":352,"reported_epoch":326,"state":"active+clean","last_fresh":"2024-09-18T08:15:07.715140+0000","last_change":"2024-09-18T08:14:05.588122+0000","last_active":"2024-09-18T08:15:07.715140+0000","last_peered":"2024-09-18T08:15:07.715140+0000","last_clean":"2024-09-18T08:15:07.715140+0000","last_became_active":"2024-09-18T08:14:05.587338+0000","last_became_peered":"2024-09-18T08:14:05.587338+0000","last_unstale":"2024-09-18T08:15:07.715140+0000","last_undegraded":"2024-09-18T08:15:07.715140+0000","last_fullsized":"2024-09-18T08:15:07.715140+0000","mapping_epoch":292,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":293,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:13:32.933940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:46.860092+0000","last_clean_scrub_stamp":"2024-09-18T08:13:32.933940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":21,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,6],"acting":[4,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.15","version":"317'11","reported_seq":721,"reported_epoch":325,"state":"active+clean","last_fresh":"2024-09-18T08:15:01.379776+0000","last_change":"2024-09-18T08:14:52.685554+0000","last_active":"2024-09-18T08:15:01.379776+0000","last_peered":"2024-09-18T08:15:01.379776+0000","last_clean":"2024-09-18T08:15:01.379776+0000","last_became_active":"2024-09-18T08:14:52.663054+0000","last_became_peered":"2024-09-18T08:14:52.663054+0000","last_unstale":"2024-09-18T08:15:01.379776+0000","last_undegraded":"2024-09-18T08:15:01.379776+0000","last_fullsized":"2024-09-18T08:15:01.379776+0000","mapping_epoch":316,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":317,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'8","last_scrub_stamp":"2024-09-18T08:10:54.148986+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:50.886830+0000","last_clean_scrub_stamp":"2024-09-18T08:10:54.148986+0000","objects_scrubbed":1,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":564,"reported_epoch":326,"state":"active+clean","last_fresh":"2024-09-18T08:15:07.734139+0000","last_change":"2024-09-18T08:14:05.590487+0000","last_active":"2024-09-18T08:15:07.734139+0000","last_peered":"2024-09-18T08:15:07.734139+0000","last_clean":"2024-09-18T08:15:07.734139+0000","last_became_active":"2024-09-18T08:14:05.590147+0000","last_became_peered":"2024-09-18T08:14:05.590147+0000","last_unstale":"2024-09-18T08:15:07.734139+0000","last_undegraded":"2024-09-18T08:15:07.734139+0000","last_fullsized":"2024-09-18T08:15:07.734139+0000","mapping_epoch":292,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":293,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:09:42.047270+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_clean_scrub_stamp":"2024-09-18T08:09:42.047270+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:15:25.093748+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.13","version":"271'19","reported_seq":356,"reported_epoch":326,"state":"active+clean","last_fresh":"2024-09-18T08:15:07.715169+0000","last_change":"2024-09-18T08:14:39.467914+0000","last_active":"2024-09-18T08:15:07.715169+0000","last_peered":"2024-09-18T08:15:07.715169+0000","last_clean":"2024-09-18T08:15:07.715169+0000","last_became_active":"2024-09-18T08:14:39.369949+0000","last_became_peered":"2024-09-18T08:14:39.369949+0000","last_unstale":"2024-09-18T08:15:07.715169+0000","last_undegraded":"2024-09-18T08:15:07.715169+0000","last_fullsized":"2024-09-18T08:15:07.715169+0000","mapping_epoch":307,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":308,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'11","last_scrub_stamp":"2024-09-18T08:11:22.565415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_clean_scrub_stamp":"2024-09-18T08:11:22.565415+0000","objects_scrubbed":5,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":31,"num_read_kb":24,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":317,"reported_epoch":326,"state":"active+clean","last_fresh":"2024-09-18T08:15:07.715923+0000","last_change":"2024-09-18T08:14:05.582592+0000","last_active":"2024-09-18T08:15:07.715923+0000","last_peered":"2024-09-18T08:15:07.715923+0000","last_clean":"2024-09-18T08:15:07.715923+0000","last_became_active":"2024-09-18T08:14:05.581338+0000","last_became_peered":"2024-09-18T08:14:05.581338+0000","last_unstale":"2024-09-18T08:15:07.715923+0000","last_undegraded":"2024-09-18T08:15:07.715923+0000","last_fullsized":"2024-09-18T08:15:07.715923+0000","mapping_epoch":292,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":293,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:10:58.402308+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:46.860092+0000","last_clean_scrub_stamp":"2024-09-18T08:10:58.402308+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,2],"acting":[4,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":446,"reported_epoch":325,"state":"active+clean","last_fresh":"2024-09-18T08:15:01.379300+0000","last_change":"2024-09-18T08:14:52.673639+0000","last_active":"2024-09-18T08:15:01.379300+0000","last_peered":"2024-09-18T08:15:01.379300+0000","last_clean":"2024-09-18T08:15:01.379300+0000","last_became_active":"2024-09-18T08:14:52.673318+0000","last_became_peered":"2024-09-18T08:14:52.673318+0000","last_unstale":"2024-09-18T08:15:01.379300+0000","last_undegraded":"2024-09-18T08:15:01.379300+0000","last_fullsized":"2024-09-18T08:15:01.379300+0000","mapping_epoch":316,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":317,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:11:26.316948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:50.886830+0000","last_clean_scrub_stamp":"2024-09-18T08:11:26.316948+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,6],"acting":[1,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":581,"reported_epoch":326,"state":"active+clean","last_fresh":"2024-09-18T08:15:07.718567+0000","last_change":"2024-09-18T08:14:52.679314+0000","last_active":"2024-09-18T08:15:07.718567+0000","last_peered":"2024-09-18T08:15:07.718567+0000","last_clean":"2024-09-18T08:15:07.718567+0000","last_became_active":"2024-09-18T08:14:52.678725+0000","last_became_peered":"2024-09-18T08:14:52.678725+0000","last_unstale":"2024-09-18T08:15:07.718567+0000","last_undegraded":"2024-09-18T08:15:07.718567+0000","last_fullsized":"2024-09-18T08:15:07.718567+0000","mapping_epoch":316,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":317,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:13:28.680237+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_clean_scrub_stamp":"2024-09-18T08:13:28.680237+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.12","version":"271'15","reported_seq":441,"reported_epoch":325,"state":"active+clean","last_fresh":"2024-09-18T08:15:01.379468+0000","last_change":"2024-09-18T08:14:39.480181+0000","last_active":"2024-09-18T08:15:01.379468+0000","last_peered":"2024-09-18T08:15:01.379468+0000","last_clean":"2024-09-18T08:15:01.379468+0000","last_became_active":"2024-09-18T08:14:39.416284+0000","last_became_peered":"2024-09-18T08:14:39.416284+0000","last_unstale":"2024-09-18T08:15:01.379468+0000","last_undegraded":"2024-09-18T08:15:01.379468+0000","last_fullsized":"2024-09-18T08:15:01.379468+0000","mapping_epoch":307,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":308,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'9","last_scrub_stamp":"2024-09-18T08:10:06.582804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_clean_scrub_stamp":"2024-09-18T08:10:06.582804+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:15:52.149563+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":25,"num_read_kb":19,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":367,"reported_epoch":326,"state":"active+clean","last_fresh":"2024-09-18T08:15:07.722643+0000","last_change":"2024-09-18T08:14:05.581553+0000","last_active":"2024-09-18T08:15:07.722643+0000","last_peered":"2024-09-18T08:15:07.722643+0000","last_clean":"2024-09-18T08:15:07.722643+0000","last_became_active":"2024-09-18T08:14:05.581038+0000","last_became_peered":"2024-09-18T08:14:05.581038+0000","last_unstale":"2024-09-18T08:15:07.722643+0000","last_undegraded":"2024-09-18T08:15:07.722643+0000","last_fullsized":"2024-09-18T08:15:07.722643+0000","mapping_epoch":292,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":293,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:13:35.000383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:46.860092+0000","last_clean_scrub_stamp":"2024-09-18T08:13:35.000383+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,0],"acting":[3,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":572,"reported_epoch":326,"state":"active+clean","last_fresh":"2024-09-18T08:15:07.722282+0000","last_change":"2024-09-18T08:14:39.403741+0000","last_active":"2024-09-18T08:15:07.722282+0000","last_peered":"2024-09-18T08:15:07.722282+0000","last_clean":"2024-09-18T08:15:07.722282+0000","last_became_active":"2024-09-18T08:14:39.402406+0000","last_became_peered":"2024-09-18T08:14:39.402406+0000","last_unstale":"2024-09-18T08:15:07.722282+0000","last_undegraded":"2024-09-18T08:15:07.722282+0000","last_fullsized":"2024-09-18T08:15:07.722282+0000","mapping_epoch":307,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":308,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:11:30.302179+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:50.886830+0000","last_clean_scrub_stamp":"2024-09-18T08:11:30.302179+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.10","version":"69'1","reported_seq":575,"reported_epoch":326,"state":"active+clean","last_fresh":"2024-09-18T08:15:07.731257+0000","last_change":"2024-09-18T08:15:00.219728+0000","last_active":"2024-09-18T08:15:07.731257+0000","last_peered":"2024-09-18T08:15:07.731257+0000","last_clean":"2024-09-18T08:15:07.731257+0000","last_became_active":"2024-09-18T08:14:52.666199+0000","last_became_peered":"2024-09-18T08:14:52.666199+0000","last_unstale":"2024-09-18T08:15:07.731257+0000","last_undegraded":"2024-09-18T08:15:07.731257+0000","last_fullsized":"2024-09-18T08:15:07.731257+0000","mapping_epoch":316,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":317,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'1","last_scrub_stamp":"2024-09-18T08:15:00.219678+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_clean_scrub_stamp":"2024-09-18T08:15:00.219678+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:16:22.427445+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":13,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,6],"acting":[0,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.11","version":"271'19","reported_seq":696,"reported_epoch":326,"state":"active+clean","last_fresh":"2024-09-18T08:15:07.718842+0000","last_change":"2024-09-18T08:14:52.703314+0000","last_active":"2024-09-18T08:15:07.718842+0000","last_peered":"2024-09-18T08:15:07.718842+0000","last_clean":"2024-09-18T08:15:07.718842+0000","last_became_active":"2024-09-18T08:14:52.665732+0000","last_became_peered":"2024-09-18T08:14:52.665732+0000","last_unstale":"2024-09-18T08:15:07.718842+0000","last_undegraded":"2024-09-18T08:15:07.718842+0000","last_fullsized":"2024-09-18T08:15:07.718842+0000","mapping_epoch":316,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":317,"parent":"0.0","parent_split_bits":0,"last_scrub":"271'19","last_scrub_stamp":"2024-09-18T08:14:38.809490+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_clean_scrub_stamp":"2024-09-18T08:14:38.809490+0000","objects_scrubbed":5,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:16:03.873746+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":32,"num_read_kb":25,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,0],"acting":[3,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":568,"reported_epoch":325,"state":"active+clean","last_fresh":"2024-09-18T08:15:01.380432+0000","last_change":"2024-09-18T08:14:52.664843+0000","last_active":"2024-09-18T08:15:01.380432+0000","last_peered":"2024-09-18T08:15:01.380432+0000","last_clean":"2024-09-18T08:15:01.380432+0000","last_became_active":"2024-09-18T08:14:52.664526+0000","last_became_peered":"2024-09-18T08:14:52.664526+0000","last_unstale":"2024-09-18T08:15:01.380432+0000","last_undegraded":"2024-09-18T08:15:01.380432+0000","last_fullsized":"2024-09-18T08:15:01.380432+0000","mapping_epoch":316,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":317,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:10:53.145315+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:46.860092+0000","last_clean_scrub_stamp":"2024-09-18T08:10:53.145315+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":374,"reported_epoch":326,"state":"active+clean","last_fresh":"2024-09-18T08:15:07.714848+0000","last_change":"2024-09-18T08:14:37.743954+0000","last_active":"2024-09-18T08:15:07.714848+0000","last_peered":"2024-09-18T08:15:07.714848+0000","last_clean":"2024-09-18T08:15:07.714848+0000","last_became_active":"2024-09-18T08:14:05.585776+0000","last_became_peered":"2024-09-18T08:14:05.585776+0000","last_unstale":"2024-09-18T08:15:07.714848+0000","last_undegraded":"2024-09-18T08:15:07.714848+0000","last_fullsized":"2024-09-18T08:15:07.714848+0000","mapping_epoch":292,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":293,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:14:37.743856+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:50.886830+0000","last_clean_scrub_stamp":"2024-09-18T08:14:37.743856+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:15:43.360811+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,0],"acting":[4,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":571,"reported_epoch":326,"state":"active+clean","last_fresh":"2024-09-18T08:15:07.722420+0000","last_change":"2024-09-18T08:11:29.344444+0000","last_active":"2024-09-18T08:15:07.722420+0000","last_peered":"2024-09-18T08:15:07.722420+0000","last_clean":"2024-09-18T08:15:07.722420+0000","last_became_active":"2024-09-18T08:06:24.128582+0000","last_became_peered":"2024-09-18T08:06:24.128582+0000","last_unstale":"2024-09-18T08:15:07.722420+0000","last_undegraded":"2024-09-18T08:15:07.722420+0000","last_fullsized":"2024-09-18T08:15:07.722420+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:11:29.344382+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_clean_scrub_stamp":"2024-09-18T08:11:29.344382+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.10","version":"271'8","reported_seq":621,"reported_epoch":326,"state":"active+clean","last_fresh":"2024-09-18T08:15:07.719109+0000","last_change":"2024-09-18T08:14:39.475047+0000","last_active":"2024-09-18T08:15:07.719109+0000","last_peered":"2024-09-18T08:15:07.719109+0000","last_clean":"2024-09-18T08:15:07.719109+0000","last_became_active":"2024-09-18T08:14:39.404257+0000","last_became_peered":"2024-09-18T08:14:39.404257+0000","last_unstale":"2024-09-18T08:15:07.719109+0000","last_undegraded":"2024-09-18T08:15:07.719109+0000","last_fullsized":"2024-09-18T08:15:07.719109+0000","mapping_epoch":307,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":308,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'4","last_scrub_stamp":"2024-09-18T08:10:18.542984+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_clean_scrub_stamp":"2024-09-18T08:10:18.542984+0000","objects_scrubbed":2,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:15:48.821303+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":12,"num_read_kb":10,"num_write":8,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":591,"reported_epoch":326,"state":"active+clean","last_fresh":"2024-09-18T08:15:07.732382+0000","last_change":"2024-09-18T08:14:52.677347+0000","last_active":"2024-09-18T08:15:07.732382+0000","last_peered":"2024-09-18T08:15:07.732382+0000","last_clean":"2024-09-18T08:15:07.732382+0000","last_became_active":"2024-09-18T08:14:52.676933+0000","last_became_peered":"2024-09-18T08:14:52.676933+0000","last_unstale":"2024-09-18T08:15:07.732382+0000","last_undegraded":"2024-09-18T08:15:07.732382+0000","last_fullsized":"2024-09-18T08:15:07.732382+0000","mapping_epoch":316,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":317,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:14:33.064199+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:46.860092+0000","last_clean_scrub_stamp":"2024-09-18T08:14:33.064199+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:15:35.392614+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,6],"acting":[0,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":325,"reported_epoch":326,"state":"active+clean","last_fresh":"2024-09-18T08:15:07.716201+0000","last_change":"2024-09-18T08:14:05.589315+0000","last_active":"2024-09-18T08:15:07.716201+0000","last_peered":"2024-09-18T08:15:07.716201+0000","last_clean":"2024-09-18T08:15:07.716201+0000","last_became_active":"2024-09-18T08:14:05.588551+0000","last_became_peered":"2024-09-18T08:14:05.588551+0000","last_unstale":"2024-09-18T08:15:07.716201+0000","last_undegraded":"2024-09-18T08:15:07.716201+0000","last_fullsized":"2024-09-18T08:15:07.716201+0000","mapping_epoch":292,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":293,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:13:32.640669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:50.886830+0000","last_clean_scrub_stamp":"2024-09-18T08:13:32.640669+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":20,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,0],"acting":[4,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":372,"reported_epoch":326,"state":"active+clean","last_fresh":"2024-09-18T08:15:07.726768+0000","last_change":"2024-09-18T08:14:05.582585+0000","last_active":"2024-09-18T08:15:07.726768+0000","last_peered":"2024-09-18T08:15:07.726768+0000","last_clean":"2024-09-18T08:15:07.726768+0000","last_became_active":"2024-09-18T08:14:05.582250+0000","last_became_peered":"2024-09-18T08:14:05.582250+0000","last_unstale":"2024-09-18T08:15:07.726768+0000","last_undegraded":"2024-09-18T08:15:07.726768+0000","last_fullsized":"2024-09-18T08:15:07.726768+0000","mapping_epoch":292,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":293,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:13:56.525167+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_clean_scrub_stamp":"2024-09-18T08:13:56.525167+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:15:15.184232+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,3],"acting":[2,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":411,"reported_epoch":325,"state":"active+clean","last_fresh":"2024-09-18T08:15:01.379486+0000","last_change":"2024-09-18T08:14:52.680060+0000","last_active":"2024-09-18T08:15:01.379486+0000","last_peered":"2024-09-18T08:15:01.379486+0000","last_clean":"2024-09-18T08:15:01.379486+0000","last_became_active":"2024-09-18T08:14:52.679756+0000","last_became_peered":"2024-09-18T08:14:52.679756+0000","last_unstale":"2024-09-18T08:15:01.379486+0000","last_undegraded":"2024-09-18T08:15:01.379486+0000","last_fullsized":"2024-09-18T08:15:01.379486+0000","mapping_epoch":316,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":317,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:10:52.641537+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_clean_scrub_stamp":"2024-09-18T08:10:52.641537+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,2],"acting":[1,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":572,"reported_epoch":326,"state":"active+clean","last_fresh":"2024-09-18T08:15:07.719379+0000","last_change":"2024-09-18T08:14:39.405772+0000","last_active":"2024-09-18T08:15:07.719379+0000","last_peered":"2024-09-18T08:15:07.719379+0000","last_clean":"2024-09-18T08:15:07.719379+0000","last_became_active":"2024-09-18T08:14:39.405347+0000","last_became_peered":"2024-09-18T08:14:39.405347+0000","last_unstale":"2024-09-18T08:15:07.719379+0000","last_undegraded":"2024-09-18T08:15:07.719379+0000","last_fullsized":"2024-09-18T08:15:07.719379+0000","mapping_epoch":307,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":308,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:07:17.277894+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:46.860092+0000","last_clean_scrub_stamp":"2024-09-18T08:07:17.277894+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:15:42.744290+0000","scrub_duration":16,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"271'19","reported_seq":576,"reported_epoch":325,"state":"active+clean","last_fresh":"2024-09-18T08:15:01.380046+0000","last_change":"2024-09-18T08:14:52.689980+0000","last_active":"2024-09-18T08:15:01.380046+0000","last_peered":"2024-09-18T08:15:01.380046+0000","last_clean":"2024-09-18T08:15:01.380046+0000","last_became_active":"2024-09-18T08:14:52.668969+0000","last_became_peered":"2024-09-18T08:14:52.668969+0000","last_unstale":"2024-09-18T08:15:01.380046+0000","last_undegraded":"2024-09-18T08:15:01.380046+0000","last_fullsized":"2024-09-18T08:15:01.380046+0000","mapping_epoch":316,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":317,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'11","last_scrub_stamp":"2024-09-18T08:10:50.208543+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_clean_scrub_stamp":"2024-09-18T08:10:50.208543+0000","objects_scrubbed":5,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":31,"num_read_kb":24,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,0],"acting":[1,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":595,"reported_epoch":326,"state":"active+clean","last_fresh":"2024-09-18T08:15:07.735599+0000","last_change":"2024-09-18T08:14:40.358776+0000","last_active":"2024-09-18T08:15:07.735599+0000","last_peered":"2024-09-18T08:15:07.735599+0000","last_clean":"2024-09-18T08:15:07.735599+0000","last_became_active":"2024-09-18T08:14:40.358457+0000","last_became_peered":"2024-09-18T08:14:40.358457+0000","last_unstale":"2024-09-18T08:15:07.735599+0000","last_undegraded":"2024-09-18T08:15:07.735599+0000","last_fullsized":"2024-09-18T08:15:07.735599+0000","mapping_epoch":308,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":309,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:12:31.664456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:50.886830+0000","last_clean_scrub_stamp":"2024-09-18T08:12:31.664456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":10,"num_pg":16,"stat_sum":{"num_bytes":1940324352,"num_objects":29607,"num_object_clones":0,"num_object_copies":88821,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":29607,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":29607,"num_write_kb":1894848,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3577,"num_bytes_recovered":234422272,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":5972688896,"data_stored":5972688896,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":29607,"ondisk_log_size":29607,"up":48,"acting":48,"num_store_stats":7},{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":416,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":416,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":103,"ondisk_log_size":103,"up":96,"acting":96,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":178,"num_object_clones":0,"num_object_copies":534,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":178,"num_whiteouts":0,"num_read":1108,"num_read_kb":865,"num_write":703,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":363,"num_bytes_recovered":6248,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":442368,"data_stored":11766,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3639,"internal_metadata":0},"log_size":687,"ondisk_log_size":687,"up":96,"acting":96,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":1613,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":24,"num_read_kb":24,"num_write":10,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":3134,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":81920,"data_stored":4931,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":6,"ondisk_log_size":6,"up":96,"acting":96,"num_store_stats":8},{"poolid":2,"num_pg":3,"stat_sum":{"num_bytes":408,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":201,"num_read_kb":196,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":408,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1613,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":8,"ondisk_log_size":8,"up":9,"acting":9,"num_store_stats":7},{"poolid":1,"num_pg":11,"stat_sum":{"num_bytes":697707,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":21,"num_read_kb":40,"num_write":21,"num_write_kb":296,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":762699,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":869,"ondisk_log_size":869,"up":33,"acting":33,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":0,"seq":240518168728,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":47,"seq":201863463070,"num_pgs":74,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1168688,"kb_used_data":812636,"kb_used_omap":1,"kb_used_meta":356030,"kb_avail":92576464,"statfs":{"total":95995035648,"available":94798299136,"internally_reserved":0,"allocated":832139264,"data_stored":831284431,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":364575174},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":0,"seq":167503724708,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":32,"seq":137438953641,"num_pgs":85,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1634876,"kb_used_data":1216364,"kb_used_omap":1,"kb_used_meta":418494,"kb_avail":92110276,"statfs":{"total":95995035648,"available":94320922624,"internally_reserved":0,"allocated":1245556736,"data_stored":1244680053,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":428538309},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":26,"seq":111669149872,"num_pgs":81,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1449508,"kb_used_data":844944,"kb_used_omap":1,"kb_used_meta":604542,"kb_avail":92295644,"statfs":{"total":95995035648,"available":94510739456,"internally_reserved":0,"allocated":865222656,"data_stored":864316575,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":619051464},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411510,"num_pgs":70,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1409476,"kb_used_data":959024,"kb_used_omap":1,"kb_used_meta":450430,"kb_avail":92335676,"statfs":{"total":95995035648,"available":94551732224,"internally_reserved":0,"allocated":982040576,"data_stored":981166494,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":461240773},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":307,"seq":1318554959879,"num_pgs":83,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":609264,"kb_used_data":549152,"kb_used_omap":7,"kb_used_meta":60088,"kb_avail":93135888,"statfs":{"total":95995035648,"available":95371149312,"internally_reserved":0,"allocated":562331648,"data_stored":561403145,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":8079,"internal_metadata":61530225},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738561,"num_pgs":81,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1662108,"kb_used_data":1225548,"kb_used_omap":1,"kb_used_meta":436542,"kb_avail":92083044,"statfs":{"total":95995035648,"available":94293037056,"internally_reserved":0,"allocated":1254961152,"data_stored":1254059349,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":447019461},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":266240,"data_stored":262688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1567,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1613,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":436,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":46,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1177,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":69632,"data_stored":1914,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":98304,"data_stored":2558,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3639,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":1568,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":90112,"data_stored":2338,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":1584,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":1144,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":660,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":403,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":403,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":403,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":1259995136,"data_stored":1259995136,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":568590336,"data_stored":568590336,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":989331456,"data_stored":989331456,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":880607232,"data_stored":880607232,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":1306656768,"data_stored":1306656768,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":128385024,"data_stored":128385024,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":839122944,"data_stored":839122944,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T08:15:13.119 INFO:tasks.thrashosds.thrasher:Setting 10.c to [1, 6, 7, 2, 3, 5] 2024-09-18T08:15:13.119 INFO:tasks.thrashosds.thrasher:cmd ['osd', 'pg-upmap-items', '10.c', '1', '6', '7', '2', '3', '5'] 2024-09-18T08:15:13.120 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid 779349ca-7593-11ef-bceb-c7b262605968 -- ceph osd pg-upmap-items 10.c 1 6 7 2 3 5 2024-09-18T08:15:13.120 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/779349ca-7593-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T08:15:13.131 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2 2024-09-18T08:15:13.214 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:15:13.215 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:15:13.216 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2' 2024-09-18T08:15:13.317 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7 2024-09-18T08:15:13.339 INFO:tasks.radosbench.radosbench.0.smithi007.stdout: 25 16 37453 37437 93.5708 103.812 0.0122612 0.0106811 2024-09-18T08:15:13.344 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:15:13.345 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:15:13.345 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7' 2024-09-18T08:15:13.446 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3 2024-09-18T08:15:13.512 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:15:13.512 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:15:13.513 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3' 2024-09-18T08:15:13.613 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5 2024-09-18T08:15:13.644 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:15:13.645 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:15:13.646 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5' 2024-09-18T08:15:13.746 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0 2024-09-18T08:15:13.788 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:15:13.789 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:15:13.790 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0' 2024-09-18T08:15:13.890 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2 2024-09-18T08:15:13.928 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:15:13.929 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:15:13.929 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2' 2024-09-18T08:15:13.998 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid 779349ca-7593-11ef-bceb-c7b262605968 -- ceph daemon osd.1 dump_ops_in_flight 2024-09-18T08:15:14.029 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4 2024-09-18T08:15:14.054 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:15:14.055 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:15:14.055 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4' 2024-09-18T08:15:14.057 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:15:13 smithi007 ceph-mon[39249]: 3.4 scrub starts 2024-09-18T08:15:14.057 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:15:13 smithi007 ceph-mon[39249]: 6.a scrub starts 2024-09-18T08:15:14.057 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:15:13 smithi007 ceph-mon[39249]: 6.a scrub ok 2024-09-18T08:15:14.057 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:15:13 smithi007 ceph-mon[39249]: 3.4 scrub ok 2024-09-18T08:15:14.057 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:15:13 smithi007 ceph-mon[39249]: pgmap v615: 158 pgs: 158 active+clean; 1.9 GiB data, 7.8 GiB used, 529 GiB / 536 GiB avail; 1.2 KiB/s rd, 82 MiB/s wr, 1.32k op/s 2024-09-18T08:15:14.057 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:15:13 smithi007 ceph-mon[29459]: 3.4 scrub starts 2024-09-18T08:15:14.058 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:15:13 smithi007 ceph-mon[29459]: 6.a scrub starts 2024-09-18T08:15:14.058 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:15:13 smithi007 ceph-mon[29459]: 6.a scrub ok 2024-09-18T08:15:14.058 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:15:13 smithi007 ceph-mon[29459]: 3.4 scrub ok 2024-09-18T08:15:14.058 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:15:13 smithi007 ceph-mon[29459]: pgmap v615: 158 pgs: 158 active+clean; 1.9 GiB data, 7.8 GiB used, 529 GiB / 536 GiB avail; 1.2 KiB/s rd, 82 MiB/s wr, 1.32k op/s 2024-09-18T08:15:14.065 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/779349ca-7593-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T08:15:14.155 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:15:13 smithi050 ceph-mon[32371]: 3.4 scrub starts 2024-09-18T08:15:14.155 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:15:13 smithi050 ceph-mon[32371]: 6.a scrub starts 2024-09-18T08:15:14.155 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:15:13 smithi050 ceph-mon[32371]: 6.a scrub ok 2024-09-18T08:15:14.156 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:15:13 smithi050 ceph-mon[32371]: 3.4 scrub ok 2024-09-18T08:15:14.156 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:15:13 smithi050 ceph-mon[32371]: pgmap v615: 158 pgs: 158 active+clean; 1.9 GiB data, 7.8 GiB used, 529 GiB / 536 GiB avail; 1.2 KiB/s rd, 82 MiB/s wr, 1.32k op/s 2024-09-18T08:15:14.156 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2 2024-09-18T08:15:14.196 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:15:14.197 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:15:14.197 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2' 2024-09-18T08:15:14.298 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0 2024-09-18T08:15:14.333 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid 779349ca-7593-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-18T08:15:14.339 INFO:tasks.radosbench.radosbench.0.smithi007.stdout: 26 16 38916 38900 93.4882 91.4375 0.0071939 0.0106904 2024-09-18T08:15:14.358 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:15:14.359 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:15:14.359 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0' 2024-09-18T08:15:14.460 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7 2024-09-18T08:15:14.485 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:15:14.485 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:15:14.486 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7' 2024-09-18T08:15:14.587 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2 2024-09-18T08:15:14.625 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:15:14.626 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:15:14.626 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2' 2024-09-18T08:15:14.727 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7 2024-09-18T08:15:14.753 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:15:14.753 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:15:14.754 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7' 2024-09-18T08:15:14.797 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/779349ca-7593-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T08:15:14.855 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5 2024-09-18T08:15:14.881 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:15:14.881 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:15:14.882 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5' 2024-09-18T08:15:14.982 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5 2024-09-18T08:15:15.010 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:15:15.010 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:15:15.011 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5' 2024-09-18T08:15:15.112 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6 2024-09-18T08:15:15.115 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:15:14 smithi007 ceph-mon[39249]: 4.1b scrub starts 2024-09-18T08:15:15.115 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:15:14 smithi007 ceph-mon[39249]: 4.1b scrub ok 2024-09-18T08:15:15.115 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:15:14 smithi007 ceph-mon[39249]: 4.7 scrub starts 2024-09-18T08:15:15.115 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:15:14 smithi007 ceph-mon[39249]: 4.7 scrub ok 2024-09-18T08:15:15.115 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:15:14 smithi007 ceph-mon[39249]: from='client.? 172.21.15.7:0/3975342528' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-18T08:15:15.115 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:15:14 smithi007 ceph-mon[39249]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-18T08:15:15.115 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:15:14 smithi007 ceph-mon[29459]: 4.1b scrub starts 2024-09-18T08:15:15.116 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:15:14 smithi007 ceph-mon[29459]: 4.1b scrub ok 2024-09-18T08:15:15.116 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:15:14 smithi007 ceph-mon[29459]: 4.7 scrub starts 2024-09-18T08:15:15.116 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:15:14 smithi007 ceph-mon[29459]: 4.7 scrub ok 2024-09-18T08:15:15.116 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:15:14 smithi007 ceph-mon[29459]: from='client.? 172.21.15.7:0/3975342528' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-18T08:15:15.116 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:15:14 smithi007 ceph-mon[29459]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-18T08:15:15.137 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:15:15.137 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:15:15.138 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6' 2024-09-18T08:15:15.155 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:15:14 smithi050 ceph-mon[32371]: 4.1b scrub starts 2024-09-18T08:15:15.155 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:15:14 smithi050 ceph-mon[32371]: 4.1b scrub ok 2024-09-18T08:15:15.155 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:15:14 smithi050 ceph-mon[32371]: 4.7 scrub starts 2024-09-18T08:15:15.155 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:15:14 smithi050 ceph-mon[32371]: 4.7 scrub ok 2024-09-18T08:15:15.155 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:15:14 smithi050 ceph-mon[32371]: from='client.? 172.21.15.7:0/3975342528' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-18T08:15:15.155 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:15:14 smithi050 ceph-mon[32371]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-18T08:15:15.239 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2 2024-09-18T08:15:15.283 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:15:15.284 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:15:15.284 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2' 2024-09-18T08:15:15.339 INFO:tasks.radosbench.radosbench.0.smithi007.stdout: 27 15 40271 40256 93.1643 84.75 0.0061944 0.0107286 2024-09-18T08:15:15.358 INFO:teuthology.orchestra.run.smithi007.stderr:set 10.c pg_upmap_items mapping to [1->6,7->2,3->5] 2024-09-18T08:15:15.364 INFO:teuthology.orchestra.run.smithi007.stderr:noscrub is set 2024-09-18T08:15:15.385 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0 2024-09-18T08:15:15.429 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:15:15.430 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:15:15.430 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0' 2024-09-18T08:15:15.480 INFO:teuthology.orchestra.run.smithi007.stdout:{ 2024-09-18T08:15:15.480 INFO:teuthology.orchestra.run.smithi007.stdout: "ops": [ 2024-09-18T08:15:15.480 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:15:15.480 INFO:teuthology.orchestra.run.smithi007.stdout: "description": "osd_repop_reply(client.15966.0:40415 10.0 e328/311)", 2024-09-18T08:15:15.481 INFO:teuthology.orchestra.run.smithi007.stdout: "initiated_at": "2024-09-18T08:15:15.475146+0000", 2024-09-18T08:15:15.481 INFO:teuthology.orchestra.run.smithi007.stdout: "age": 0.0029788839999999998, 2024-09-18T08:15:15.481 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.0030006260000000002, 2024-09-18T08:15:15.481 INFO:teuthology.orchestra.run.smithi007.stdout: "continuous": false, 2024-09-18T08:15:15.481 INFO:teuthology.orchestra.run.smithi007.stdout: "type_data": { 2024-09-18T08:15:15.481 INFO:teuthology.orchestra.run.smithi007.stdout: "flag_point": "queued for pg", 2024-09-18T08:15:15.481 INFO:teuthology.orchestra.run.smithi007.stdout: "events": [ 2024-09-18T08:15:15.481 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:15:15.481 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "initiated", 2024-09-18T08:15:15.481 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:15:15.475146+0000", 2024-09-18T08:15:15.481 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-09-18T08:15:15.481 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:15:15.482 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:15:15.482 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "header_read", 2024-09-18T08:15:15.482 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:15:15.475146+0000", 2024-09-18T08:15:15.482 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-09-18T08:15:15.482 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:15:15.482 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:15:15.482 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "throttled", 2024-09-18T08:15:15.482 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:15:15.475147+0000", 2024-09-18T08:15:15.482 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 7.4600000000000004e-07 2024-09-18T08:15:15.482 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:15:15.482 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:15:15.482 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "all_read", 2024-09-18T08:15:15.482 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:15:15.475151+0000", 2024-09-18T08:15:15.483 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 4.2350000000000001e-06 2024-09-18T08:15:15.483 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:15:15.483 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:15:15.483 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "dispatched", 2024-09-18T08:15:15.483 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:15:15.475264+0000", 2024-09-18T08:15:15.483 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.00011299200000000001 2024-09-18T08:15:15.483 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:15:15.483 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:15:15.483 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "queued_for_pg", 2024-09-18T08:15:15.483 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:15:15.475270+0000", 2024-09-18T08:15:15.483 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 6.195e-06 2024-09-18T08:15:15.483 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-09-18T08:15:15.483 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-09-18T08:15:15.483 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-09-18T08:15:15.484 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:15:15.484 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:15:15.484 INFO:teuthology.orchestra.run.smithi007.stdout: "description": "osd_repop_reply(client.15966.0:40414 10.b e328/311)", 2024-09-18T08:15:15.484 INFO:teuthology.orchestra.run.smithi007.stdout: "initiated_at": "2024-09-18T08:15:15.477755+0000", 2024-09-18T08:15:15.484 INFO:teuthology.orchestra.run.smithi007.stdout: "age": 0.00036981299999999998, 2024-09-18T08:15:15.484 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.00043702299999999998, 2024-09-18T08:15:15.484 INFO:teuthology.orchestra.run.smithi007.stdout: "continuous": false, 2024-09-18T08:15:15.484 INFO:teuthology.orchestra.run.smithi007.stdout: "type_data": { 2024-09-18T08:15:15.484 INFO:teuthology.orchestra.run.smithi007.stdout: "flag_point": "queued for pg", 2024-09-18T08:15:15.484 INFO:teuthology.orchestra.run.smithi007.stdout: "events": [ 2024-09-18T08:15:15.484 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:15:15.484 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "initiated", 2024-09-18T08:15:15.484 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:15:15.477755+0000", 2024-09-18T08:15:15.485 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-09-18T08:15:15.485 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:15:15.485 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:15:15.485 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "header_read", 2024-09-18T08:15:15.485 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:15:15.477755+0000", 2024-09-18T08:15:15.485 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-09-18T08:15:15.485 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:15:15.485 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:15:15.485 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "throttled", 2024-09-18T08:15:15.485 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:15:15.477756+0000", 2024-09-18T08:15:15.485 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 6.4000000000000001e-07 2024-09-18T08:15:15.485 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:15:15.485 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:15:15.485 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "all_read", 2024-09-18T08:15:15.486 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:15:15.477761+0000", 2024-09-18T08:15:15.486 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 4.7469999999999996e-06 2024-09-18T08:15:15.486 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:15:15.486 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:15:15.486 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "dispatched", 2024-09-18T08:15:15.486 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:15:15.477770+0000", 2024-09-18T08:15:15.486 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 9.1630000000000002e-06 2024-09-18T08:15:15.486 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:15:15.486 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:15:15.486 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "queued_for_pg", 2024-09-18T08:15:15.486 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:15:15.477776+0000", 2024-09-18T08:15:15.486 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 6.6170000000000003e-06 2024-09-18T08:15:15.486 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-09-18T08:15:15.487 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-09-18T08:15:15.487 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-09-18T08:15:15.487 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:15:15.487 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:15:15.487 INFO:teuthology.orchestra.run.smithi007.stdout: "description": "osd_op(client.15966.0:40409 10.2 10:497b4870:::benchmark_data_smithi007_148834_object40408:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536 in=65536b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e328)", 2024-09-18T08:15:15.487 INFO:teuthology.orchestra.run.smithi007.stdout: "initiated_at": "2024-09-18T08:15:15.468117+0000", 2024-09-18T08:15:15.487 INFO:teuthology.orchestra.run.smithi007.stdout: "age": 0.010008339999999999, 2024-09-18T08:15:15.487 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.010136977, 2024-09-18T08:15:15.487 INFO:teuthology.orchestra.run.smithi007.stdout: "continuous": false, 2024-09-18T08:15:15.487 INFO:teuthology.orchestra.run.smithi007.stdout: "type_data": { 2024-09-18T08:15:15.487 INFO:teuthology.orchestra.run.smithi007.stdout: "flag_point": "waiting for sub ops", 2024-09-18T08:15:15.487 INFO:teuthology.orchestra.run.smithi007.stdout: "client_info": { 2024-09-18T08:15:15.487 INFO:teuthology.orchestra.run.smithi007.stdout: "client": "client.15966", 2024-09-18T08:15:15.487 INFO:teuthology.orchestra.run.smithi007.stdout: "client_addr": "172.21.15.7:0/529784850", 2024-09-18T08:15:15.488 INFO:teuthology.orchestra.run.smithi007.stdout: "tid": 40409 2024-09-18T08:15:15.488 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:15:15.488 INFO:teuthology.orchestra.run.smithi007.stdout: "events": [ 2024-09-18T08:15:15.488 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:15:15.488 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "initiated", 2024-09-18T08:15:15.488 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:15:15.468117+0000", 2024-09-18T08:15:15.488 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-09-18T08:15:15.488 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:15:15.488 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:15:15.488 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "header_read", 2024-09-18T08:15:15.488 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:15:15.468117+0000", 2024-09-18T08:15:15.488 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-09-18T08:15:15.488 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:15:15.488 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:15:15.489 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "throttled", 2024-09-18T08:15:15.489 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:15:15.468125+0000", 2024-09-18T08:15:15.489 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 8.3000000000000002e-06 2024-09-18T08:15:15.489 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:15:15.489 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:15:15.489 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "all_read", 2024-09-18T08:15:15.489 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:15:15.468154+0000", 2024-09-18T08:15:15.489 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 2.9430000000000001e-05 2024-09-18T08:15:15.489 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:15:15.489 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:15:15.489 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "dispatched", 2024-09-18T08:15:15.489 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:15:15.468335+0000", 2024-09-18T08:15:15.489 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.00018059400000000001 2024-09-18T08:15:15.489 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:15:15.490 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:15:15.490 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "queued_for_pg", 2024-09-18T08:15:15.490 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:15:15.468354+0000", 2024-09-18T08:15:15.490 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.8800999999999998e-05 2024-09-18T08:15:15.490 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:15:15.490 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:15:15.490 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-09-18T08:15:15.490 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:15:15.469007+0000", 2024-09-18T08:15:15.490 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.00065350399999999996 2024-09-18T08:15:15.490 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:15:15.490 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:15:15.490 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "started", 2024-09-18T08:15:15.490 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:15:15.469122+0000", 2024-09-18T08:15:15.490 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.00011422200000000001 2024-09-18T08:15:15.491 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:15:15.491 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:15:15.491 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "waiting for subops from 0,4", 2024-09-18T08:15:15.491 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:15:15.469257+0000", 2024-09-18T08:15:15.491 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.000135482 2024-09-18T08:15:15.491 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:15:15.491 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:15:15.491 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:15:15.491 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:15:15.473258+0000", 2024-09-18T08:15:15.491 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.0040009019999999998 2024-09-18T08:15:15.491 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:15:15.491 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:15:15.491 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "op_commit", 2024-09-18T08:15:15.491 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:15:15.473414+0000", 2024-09-18T08:15:15.492 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.00015647799999999999 2024-09-18T08:15:15.492 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-09-18T08:15:15.492 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-09-18T08:15:15.492 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-09-18T08:15:15.492 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:15:15.492 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:15:15.492 INFO:teuthology.orchestra.run.smithi007.stdout: "description": "osd_op(client.15966.0:40414 10.b 10:deeb982e:::benchmark_data_smithi007_148834_object40413:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536 in=65536b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e328)", 2024-09-18T08:15:15.492 INFO:teuthology.orchestra.run.smithi007.stdout: "initiated_at": "2024-09-18T08:15:15.471096+0000", 2024-09-18T08:15:15.492 INFO:teuthology.orchestra.run.smithi007.stdout: "age": 0.0070290600000000002, 2024-09-18T08:15:15.492 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.0072408680000000001, 2024-09-18T08:15:15.492 INFO:teuthology.orchestra.run.smithi007.stdout: "continuous": false, 2024-09-18T08:15:15.492 INFO:teuthology.orchestra.run.smithi007.stdout: "type_data": { 2024-09-18T08:15:15.492 INFO:teuthology.orchestra.run.smithi007.stdout: "flag_point": "waiting for sub ops", 2024-09-18T08:15:15.493 INFO:teuthology.orchestra.run.smithi007.stdout: "client_info": { 2024-09-18T08:15:15.493 INFO:teuthology.orchestra.run.smithi007.stdout: "client": "client.15966", 2024-09-18T08:15:15.493 INFO:teuthology.orchestra.run.smithi007.stdout: "client_addr": "172.21.15.7:0/529784850", 2024-09-18T08:15:15.493 INFO:teuthology.orchestra.run.smithi007.stdout: "tid": 40414 2024-09-18T08:15:15.493 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:15:15.493 INFO:teuthology.orchestra.run.smithi007.stdout: "events": [ 2024-09-18T08:15:15.493 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:15:15.493 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "initiated", 2024-09-18T08:15:15.493 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:15:15.471096+0000", 2024-09-18T08:15:15.493 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-09-18T08:15:15.493 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:15:15.493 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:15:15.493 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "header_read", 2024-09-18T08:15:15.493 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:15:15.471096+0000", 2024-09-18T08:15:15.494 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-09-18T08:15:15.494 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:15:15.494 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:15:15.494 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "throttled", 2024-09-18T08:15:15.494 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:15:15.471097+0000", 2024-09-18T08:15:15.494 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.3799999999999999e-06 2024-09-18T08:15:15.494 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:15:15.494 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:15:15.494 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "all_read", 2024-09-18T08:15:15.494 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:15:15.471121+0000", 2024-09-18T08:15:15.494 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 2.3451e-05 2024-09-18T08:15:15.494 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:15:15.494 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:15:15.494 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "dispatched", 2024-09-18T08:15:15.495 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:15:15.471132+0000", 2024-09-18T08:15:15.495 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.1268e-05 2024-09-18T08:15:15.495 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:15:15.495 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:15:15.495 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "queued_for_pg", 2024-09-18T08:15:15.495 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:15:15.471139+0000", 2024-09-18T08:15:15.495 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 7.3579999999999997e-06 2024-09-18T08:15:15.495 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:15:15.495 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:15:15.496 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-09-18T08:15:15.496 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:15:15.475662+0000", 2024-09-18T08:15:15.496 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.004522369 2024-09-18T08:15:15.496 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:15:15.496 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:15:15.496 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "started", 2024-09-18T08:15:15.496 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:15:15.475803+0000", 2024-09-18T08:15:15.496 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.00014111399999999999 2024-09-18T08:15:15.496 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:15:15.496 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:15:15.496 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "waiting for subops from 0,6", 2024-09-18T08:15:15.496 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:15:15.476031+0000", 2024-09-18T08:15:15.496 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.00022839 2024-09-18T08:15:15.497 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-09-18T08:15:15.497 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-09-18T08:15:15.497 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-09-18T08:15:15.497 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:15:15.497 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:15:15.497 INFO:teuthology.orchestra.run.smithi007.stdout: "description": "osd_op(client.15966.0:40415 10.0 10:008cf56e:::benchmark_data_smithi007_148834_object40414:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536 in=65536b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e328)", 2024-09-18T08:15:15.497 INFO:teuthology.orchestra.run.smithi007.stdout: "initiated_at": "2024-09-18T08:15:15.471609+0000", 2024-09-18T08:15:15.497 INFO:teuthology.orchestra.run.smithi007.stdout: "age": 0.0065159770000000001, 2024-09-18T08:15:15.497 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.0068022730000000002, 2024-09-18T08:15:15.497 INFO:teuthology.orchestra.run.smithi007.stdout: "continuous": false, 2024-09-18T08:15:15.497 INFO:teuthology.orchestra.run.smithi007.stdout: "type_data": { 2024-09-18T08:15:15.497 INFO:teuthology.orchestra.run.smithi007.stdout: "flag_point": "waiting for sub ops", 2024-09-18T08:15:15.497 INFO:teuthology.orchestra.run.smithi007.stdout: "client_info": { 2024-09-18T08:15:15.497 INFO:teuthology.orchestra.run.smithi007.stdout: "client": "client.15966", 2024-09-18T08:15:15.498 INFO:teuthology.orchestra.run.smithi007.stdout: "client_addr": "172.21.15.7:0/529784850", 2024-09-18T08:15:15.498 INFO:teuthology.orchestra.run.smithi007.stdout: "tid": 40415 2024-09-18T08:15:15.498 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:15:15.498 INFO:teuthology.orchestra.run.smithi007.stdout: "events": [ 2024-09-18T08:15:15.498 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:15:15.498 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "initiated", 2024-09-18T08:15:15.498 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:15:15.471609+0000", 2024-09-18T08:15:15.498 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-09-18T08:15:15.498 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:15:15.498 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:15:15.498 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "header_read", 2024-09-18T08:15:15.498 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:15:15.471609+0000", 2024-09-18T08:15:15.498 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-09-18T08:15:15.498 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:15:15.499 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:15:15.499 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "throttled", 2024-09-18T08:15:15.499 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:15:15.471610+0000", 2024-09-18T08:15:15.499 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 9.8899999999999998e-07 2024-09-18T08:15:15.499 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:15:15.499 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:15:15.499 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "all_read", 2024-09-18T08:15:15.499 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:15:15.471634+0000", 2024-09-18T08:15:15.499 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 2.3969999999999999e-05 2024-09-18T08:15:15.499 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:15:15.499 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:15:15.499 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "dispatched", 2024-09-18T08:15:15.499 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:15:15.471643+0000", 2024-09-18T08:15:15.499 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 8.5329999999999994e-06 2024-09-18T08:15:15.500 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:15:15.500 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:15:15.500 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "queued_for_pg", 2024-09-18T08:15:15.500 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:15:15.471649+0000", 2024-09-18T08:15:15.500 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 6.5989999999999998e-06 2024-09-18T08:15:15.500 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:15:15.500 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:15:15.500 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-09-18T08:15:15.500 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:15:15.472923+0000", 2024-09-18T08:15:15.500 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.00127406 2024-09-18T08:15:15.500 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:15:15.500 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:15:15.500 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "started", 2024-09-18T08:15:15.501 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:15:15.473441+0000", 2024-09-18T08:15:15.501 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.00051775699999999998 2024-09-18T08:15:15.501 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:15:15.501 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:15:15.501 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "waiting for subops from 4,6", 2024-09-18T08:15:15.501 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:15:15.473585+0000", 2024-09-18T08:15:15.501 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.000144124 2024-09-18T08:15:15.501 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-09-18T08:15:15.501 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-09-18T08:15:15.501 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-09-18T08:15:15.501 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:15:15.501 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:15:15.501 INFO:teuthology.orchestra.run.smithi007.stdout: "description": "osd_op(client.15966.0:40417 10.b 10:d658d302:::benchmark_data_smithi007_148834_object40416:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536 in=65536b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e328)", 2024-09-18T08:15:15.502 INFO:teuthology.orchestra.run.smithi007.stdout: "initiated_at": "2024-09-18T08:15:15.474123+0000", 2024-09-18T08:15:15.502 INFO:teuthology.orchestra.run.smithi007.stdout: "age": 0.0040020109999999998, 2024-09-18T08:15:15.502 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.0043331180000000004, 2024-09-18T08:15:15.502 INFO:teuthology.orchestra.run.smithi007.stdout: "continuous": false, 2024-09-18T08:15:15.502 INFO:teuthology.orchestra.run.smithi007.stdout: "type_data": { 2024-09-18T08:15:15.502 INFO:teuthology.orchestra.run.smithi007.stdout: "flag_point": "waiting for sub ops", 2024-09-18T08:15:15.502 INFO:teuthology.orchestra.run.smithi007.stdout: "client_info": { 2024-09-18T08:15:15.502 INFO:teuthology.orchestra.run.smithi007.stdout: "client": "client.15966", 2024-09-18T08:15:15.502 INFO:teuthology.orchestra.run.smithi007.stdout: "client_addr": "172.21.15.7:0/529784850", 2024-09-18T08:15:15.502 INFO:teuthology.orchestra.run.smithi007.stdout: "tid": 40417 2024-09-18T08:15:15.502 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:15:15.502 INFO:teuthology.orchestra.run.smithi007.stdout: "events": [ 2024-09-18T08:15:15.502 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:15:15.502 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "initiated", 2024-09-18T08:15:15.503 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:15:15.474123+0000", 2024-09-18T08:15:15.503 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-09-18T08:15:15.503 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:15:15.503 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:15:15.503 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "header_read", 2024-09-18T08:15:15.503 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:15:15.474123+0000", 2024-09-18T08:15:15.503 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-09-18T08:15:15.503 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:15:15.503 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:15:15.503 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "throttled", 2024-09-18T08:15:15.503 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:15:15.474125+0000", 2024-09-18T08:15:15.503 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.556e-06 2024-09-18T08:15:15.503 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:15:15.503 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:15:15.503 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "all_read", 2024-09-18T08:15:15.504 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:15:15.474148+0000", 2024-09-18T08:15:15.504 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 2.3586000000000001e-05 2024-09-18T08:15:15.504 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:15:15.504 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:15:15.504 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "dispatched", 2024-09-18T08:15:15.504 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:15:15.474164+0000", 2024-09-18T08:15:15.504 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.6021000000000001e-05 2024-09-18T08:15:15.504 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:15:15.504 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:15:15.504 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "queued_for_pg", 2024-09-18T08:15:15.504 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:15:15.474172+0000", 2024-09-18T08:15:15.504 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 7.9279999999999993e-06 2024-09-18T08:15:15.504 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:15:15.505 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:15:15.505 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-09-18T08:15:15.505 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:15:15.477601+0000", 2024-09-18T08:15:15.505 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.0034292189999999998 2024-09-18T08:15:15.505 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:15:15.505 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:15:15.505 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "started", 2024-09-18T08:15:15.505 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:15:15.477731+0000", 2024-09-18T08:15:15.505 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.000130068 2024-09-18T08:15:15.505 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:15:15.505 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:15:15.505 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "waiting for subops from 0,6", 2024-09-18T08:15:15.505 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:15:15.477888+0000", 2024-09-18T08:15:15.506 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.00015650500000000001 2024-09-18T08:15:15.506 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-09-18T08:15:15.506 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-09-18T08:15:15.506 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-09-18T08:15:15.506 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:15:15.506 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:15:15.506 INFO:teuthology.orchestra.run.smithi007.stdout: "description": "osd_op(client.15966.0:40419 10.2 10:4ac48a8a:::benchmark_data_smithi007_148834_object40418:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536 in=65536b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e328)", 2024-09-18T08:15:15.506 INFO:teuthology.orchestra.run.smithi007.stdout: "initiated_at": "2024-09-18T08:15:15.474933+0000", 2024-09-18T08:15:15.506 INFO:teuthology.orchestra.run.smithi007.stdout: "age": 0.0031915989999999998, 2024-09-18T08:15:15.506 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.003557036, 2024-09-18T08:15:15.506 INFO:teuthology.orchestra.run.smithi007.stdout: "continuous": false, 2024-09-18T08:15:15.506 INFO:teuthology.orchestra.run.smithi007.stdout: "type_data": { 2024-09-18T08:15:15.506 INFO:teuthology.orchestra.run.smithi007.stdout: "flag_point": "waiting for sub ops", 2024-09-18T08:15:15.506 INFO:teuthology.orchestra.run.smithi007.stdout: "client_info": { 2024-09-18T08:15:15.507 INFO:teuthology.orchestra.run.smithi007.stdout: "client": "client.15966", 2024-09-18T08:15:15.507 INFO:teuthology.orchestra.run.smithi007.stdout: "client_addr": "172.21.15.7:0/529784850", 2024-09-18T08:15:15.507 INFO:teuthology.orchestra.run.smithi007.stdout: "tid": 40419 2024-09-18T08:15:15.507 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:15:15.507 INFO:teuthology.orchestra.run.smithi007.stdout: "events": [ 2024-09-18T08:15:15.507 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:15:15.507 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "initiated", 2024-09-18T08:15:15.507 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:15:15.474933+0000", 2024-09-18T08:15:15.507 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-09-18T08:15:15.507 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:15:15.507 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:15:15.507 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "header_read", 2024-09-18T08:15:15.507 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:15:15.474933+0000", 2024-09-18T08:15:15.508 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-09-18T08:15:15.508 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:15:15.508 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:15:15.508 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "throttled", 2024-09-18T08:15:15.508 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:15:15.474934+0000", 2024-09-18T08:15:15.508 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 9.33e-07 2024-09-18T08:15:15.508 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:15:15.508 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:15:15.508 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "all_read", 2024-09-18T08:15:15.508 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:15:15.474956+0000", 2024-09-18T08:15:15.508 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 2.1203999999999999e-05 2024-09-18T08:15:15.508 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:15:15.508 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:15:15.508 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "dispatched", 2024-09-18T08:15:15.509 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:15:15.474966+0000", 2024-09-18T08:15:15.509 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.0564000000000001e-05 2024-09-18T08:15:15.509 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:15:15.509 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:15:15.509 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "queued_for_pg", 2024-09-18T08:15:15.509 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:15:15.474974+0000", 2024-09-18T08:15:15.509 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 7.4089999999999997e-06 2024-09-18T08:15:15.509 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:15:15.509 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:15:15.509 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-09-18T08:15:15.509 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:15:15.475026+0000", 2024-09-18T08:15:15.509 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 5.2290000000000002e-05 2024-09-18T08:15:15.509 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:15:15.509 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:15:15.510 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "started", 2024-09-18T08:15:15.510 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:15:15.475341+0000", 2024-09-18T08:15:15.510 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.000314634 2024-09-18T08:15:15.510 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:15:15.510 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:15:15.510 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "waiting for subops from 0,4", 2024-09-18T08:15:15.510 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:15:15.475477+0000", 2024-09-18T08:15:15.510 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.000136673 2024-09-18T08:15:15.510 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:15:15.510 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:15:15.510 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:15:15.510 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:15:15.477854+0000", 2024-09-18T08:15:15.510 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.0023763339999999999 2024-09-18T08:15:15.510 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:15:15.511 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:15:15.511 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "op_commit", 2024-09-18T08:15:15.511 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:15:15.477875+0000", 2024-09-18T08:15:15.511 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 2.1892e-05 2024-09-18T08:15:15.511 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-09-18T08:15:15.511 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-09-18T08:15:15.511 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-09-18T08:15:15.511 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:15:15.511 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:15:15.511 INFO:teuthology.orchestra.run.smithi007.stdout: "description": "osd_repop_reply(client.15966.0:40415 10.0 e328/311)", 2024-09-18T08:15:15.511 INFO:teuthology.orchestra.run.smithi007.stdout: "initiated_at": "2024-09-18T08:15:15.475194+0000", 2024-09-18T08:15:15.511 INFO:teuthology.orchestra.run.smithi007.stdout: "age": 0.002930869, 2024-09-18T08:15:15.511 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.0033581879999999998, 2024-09-18T08:15:15.511 INFO:teuthology.orchestra.run.smithi007.stdout: "continuous": false, 2024-09-18T08:15:15.512 INFO:teuthology.orchestra.run.smithi007.stdout: "type_data": { 2024-09-18T08:15:15.512 INFO:teuthology.orchestra.run.smithi007.stdout: "flag_point": "queued for pg", 2024-09-18T08:15:15.512 INFO:teuthology.orchestra.run.smithi007.stdout: "events": [ 2024-09-18T08:15:15.512 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:15:15.512 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "initiated", 2024-09-18T08:15:15.512 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:15:15.475194+0000", 2024-09-18T08:15:15.512 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-09-18T08:15:15.512 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:15:15.512 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:15:15.512 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "header_read", 2024-09-18T08:15:15.512 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:15:15.475194+0000", 2024-09-18T08:15:15.512 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-09-18T08:15:15.512 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:15:15.513 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:15:15.513 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "throttled", 2024-09-18T08:15:15.513 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:15:15.475195+0000", 2024-09-18T08:15:15.513 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 5.7000000000000005e-07 2024-09-18T08:15:15.513 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:15:15.513 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:15:15.513 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "all_read", 2024-09-18T08:15:15.513 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:15:15.475198+0000", 2024-09-18T08:15:15.513 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 3.63e-06 2024-09-18T08:15:15.513 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:15:15.513 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:15:15.513 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "dispatched", 2024-09-18T08:15:15.513 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:15:15.475213+0000", 2024-09-18T08:15:15.513 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.4970000000000001e-05 2024-09-18T08:15:15.514 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:15:15.514 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:15:15.514 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "queued_for_pg", 2024-09-18T08:15:15.514 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:15:15.475227+0000", 2024-09-18T08:15:15.514 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.3225e-05 2024-09-18T08:15:15.514 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-09-18T08:15:15.514 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-09-18T08:15:15.514 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-09-18T08:15:15.514 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-09-18T08:15:15.514 INFO:teuthology.orchestra.run.smithi007.stdout: ], 2024-09-18T08:15:15.514 INFO:teuthology.orchestra.run.smithi007.stdout: "num_ops": 8 2024-09-18T08:15:15.514 INFO:teuthology.orchestra.run.smithi007.stdout:} 2024-09-18T08:15:15.531 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6 2024-09-18T08:15:15.559 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:15:15.559 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:15:15.560 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6' 2024-09-18T08:15:15.660 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3 2024-09-18T08:15:15.701 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:15:15.702 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:15:15.702 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3' 2024-09-18T08:15:15.803 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6 2024-09-18T08:15:15.830 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:15:15.831 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:15:15.831 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6' 2024-09-18T08:15:15.932 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2 2024-09-18T08:15:15.970 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:15:15.970 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:15:15.971 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2' 2024-09-18T08:15:16.072 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6 2024-09-18T08:15:16.100 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:15:16.101 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:15:16.101 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6' 2024-09-18T08:15:16.155 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:15:15 smithi050 ceph-mon[32371]: 6.3 scrub starts 2024-09-18T08:15:16.155 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:15:15 smithi050 ceph-mon[32371]: 6.3 scrub ok 2024-09-18T08:15:16.155 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:15:15 smithi050 ceph-mon[32371]: from='client.? 172.21.15.7:0/1820026387' entity='client.admin' cmd={"prefix": "osd pg-upmap-items", "pgid": "10.c", "id": [1, 6, 7, 2, 3, 5]} : dispatch 2024-09-18T08:15:16.156 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:15:15 smithi050 ceph-mon[32371]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-18T08:15:16.156 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:15:15 smithi050 ceph-mon[32371]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-18T08:15:16.156 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:15:15 smithi050 ceph-mon[32371]: from='client.? 172.21.15.7:0/1820026387' entity='client.admin' cmd='[{"prefix": "osd pg-upmap-items", "pgid": "10.c", "id": [1, 6, 7, 2, 3, 5]}]': finished 2024-09-18T08:15:16.156 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:15:15 smithi050 ceph-mon[32371]: pgmap v616: 158 pgs: 158 active+clean; 2.3 GiB data, 9.2 GiB used, 527 GiB / 536 GiB avail; 1.2 KiB/s rd, 134 MiB/s wr, 2.14k op/s 2024-09-18T08:15:16.156 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:15:15 smithi050 ceph-mon[32371]: osdmap e328: 8 total, 8 up, 6 in 2024-09-18T08:15:16.203 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7 2024-09-18T08:15:16.230 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:15:15 smithi007 ceph-mon[39249]: 6.3 scrub starts 2024-09-18T08:15:16.230 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:15:15 smithi007 ceph-mon[39249]: 6.3 scrub ok 2024-09-18T08:15:16.230 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:15:15 smithi007 ceph-mon[39249]: from='client.? 172.21.15.7:0/1820026387' entity='client.admin' cmd={"prefix": "osd pg-upmap-items", "pgid": "10.c", "id": [1, 6, 7, 2, 3, 5]} : dispatch 2024-09-18T08:15:16.230 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:15:15 smithi007 ceph-mon[39249]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-18T08:15:16.231 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:15:15 smithi007 ceph-mon[39249]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-18T08:15:16.231 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:15:15 smithi007 ceph-mon[39249]: from='client.? 172.21.15.7:0/1820026387' entity='client.admin' cmd='[{"prefix": "osd pg-upmap-items", "pgid": "10.c", "id": [1, 6, 7, 2, 3, 5]}]': finished 2024-09-18T08:15:16.231 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:15:15 smithi007 ceph-mon[39249]: pgmap v616: 158 pgs: 158 active+clean; 2.3 GiB data, 9.2 GiB used, 527 GiB / 536 GiB avail; 1.2 KiB/s rd, 134 MiB/s wr, 2.14k op/s 2024-09-18T08:15:16.231 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:15:15 smithi007 ceph-mon[39249]: osdmap e328: 8 total, 8 up, 6 in 2024-09-18T08:15:16.231 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:15:15 smithi007 ceph-mon[29459]: 6.3 scrub starts 2024-09-18T08:15:16.231 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:15:15 smithi007 ceph-mon[29459]: 6.3 scrub ok 2024-09-18T08:15:16.231 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:15:15 smithi007 ceph-mon[29459]: from='client.? 172.21.15.7:0/1820026387' entity='client.admin' cmd={"prefix": "osd pg-upmap-items", "pgid": "10.c", "id": [1, 6, 7, 2, 3, 5]} : dispatch 2024-09-18T08:15:16.231 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:15:15 smithi007 ceph-mon[29459]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-18T08:15:16.231 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:15:15 smithi007 ceph-mon[29459]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-18T08:15:16.231 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:15:15 smithi007 ceph-mon[29459]: from='client.? 172.21.15.7:0/1820026387' entity='client.admin' cmd='[{"prefix": "osd pg-upmap-items", "pgid": "10.c", "id": [1, 6, 7, 2, 3, 5]}]': finished 2024-09-18T08:15:16.231 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:15:15 smithi007 ceph-mon[29459]: pgmap v616: 158 pgs: 158 active+clean; 2.3 GiB data, 9.2 GiB used, 527 GiB / 536 GiB avail; 1.2 KiB/s rd, 134 MiB/s wr, 2.14k op/s 2024-09-18T08:15:16.232 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:15:15 smithi007 ceph-mon[29459]: osdmap e328: 8 total, 8 up, 6 in 2024-09-18T08:15:16.232 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:15:16.233 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:15:16.233 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7' 2024-09-18T08:15:16.334 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0 2024-09-18T08:15:16.339 INFO:tasks.radosbench.radosbench.0.smithi007.stdout: 28 16 41805 41789 93.2585 95.8125 0.00246269 0.0107021 2024-09-18T08:15:16.364 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:15:16.365 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:15:16.365 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0' 2024-09-18T08:15:16.466 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1 2024-09-18T08:15:16.505 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:15:16.505 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:15:16.506 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1' 2024-09-18T08:15:16.606 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3 2024-09-18T08:15:16.648 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:15:16.649 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:15:16.649 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3' 2024-09-18T08:15:16.750 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3 2024-09-18T08:15:16.786 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:15:16.787 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:15:16.787 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3' 2024-09-18T08:15:16.855 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/779349ca-7593-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T08:15:16.889 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7 2024-09-18T08:15:16.917 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:15:16.917 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:15:16.918 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7' 2024-09-18T08:15:16.957 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Sep 18 08:15:16 smithi007 ceph-779349ca-7593-11ef-bceb-c7b262605968-mgr-y[29688]: ::ffff:172.21.15.50 - - [18/Sep/2024:08:15:16] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-09-18T08:15:17.019 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4 2024-09-18T08:15:17.045 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:15:17.046 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:15:17.046 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4' 2024-09-18T08:15:17.147 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0 2024-09-18T08:15:17.190 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:15:17.191 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:15:17.191 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0' 2024-09-18T08:15:17.292 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5 2024-09-18T08:15:17.320 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:15:17.321 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:15:17.321 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5' 2024-09-18T08:15:17.340 INFO:tasks.radosbench.radosbench.0.smithi007.stdout: 29 15 43428 43413 93.5424 101.5 0.0159561 0.0106845 2024-09-18T08:15:17.422 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2 2024-09-18T08:15:17.467 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:15:17.468 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:15:17.469 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2' 2024-09-18T08:15:17.569 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6 2024-09-18T08:15:17.597 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:15:17.597 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:15:17.598 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6' 2024-09-18T08:15:17.699 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2 2024-09-18T08:15:17.737 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:15:17.739 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:15:17.739 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2' 2024-09-18T08:15:17.840 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7 2024-09-18T08:15:17.868 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:15:17.869 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:15:17.869 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7' 2024-09-18T08:15:17.970 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5 2024-09-18T08:15:17.999 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:15:18.000 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:15:18.000 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5' 2024-09-18T08:15:18.101 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3 2024-09-18T08:15:18.385 INFO:tasks.radosbench.radosbench.0.smithi007.stdout: 30 16 45048 45032 93.7969 101.188 0.00932754 0.010656 2024-09-18T08:15:18.448 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:15:18.449 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:15:18.449 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3' 2024-09-18T08:15:18.453 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid 779349ca-7593-11ef-bceb-c7b262605968 -- ceph daemon osd.1 dump_blocked_ops 2024-09-18T08:15:18.550 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7 2024-09-18T08:15:18.578 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:15:18.579 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:15:18.579 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7' 2024-09-18T08:15:18.680 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0 2024-09-18T08:15:18.715 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:15:18 smithi007 ceph-mon[29459]: pgmap v618: 158 pgs: 158 active+clean; 2.3 GiB data, 9.2 GiB used, 527 GiB / 536 GiB avail; 639 B/s rd, 64 MiB/s wr, 1.02k op/s 2024-09-18T08:15:18.715 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:15:18 smithi007 ceph-mon[29459]: from='mgr.24470 172.21.15.7:0/2594390261' entity='mgr.y' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.b", "id": [4, 2]} : dispatch 2024-09-18T08:15:18.715 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:15:18 smithi007 ceph-mon[29459]: from='mgr.24470 172.21.15.7:0/2594390261' entity='mgr.y' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.18", "id": [1, 2]} : dispatch 2024-09-18T08:15:18.717 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:15:18 smithi007 ceph-mon[39249]: pgmap v618: 158 pgs: 158 active+clean; 2.3 GiB data, 9.2 GiB used, 527 GiB / 536 GiB avail; 639 B/s rd, 64 MiB/s wr, 1.02k op/s 2024-09-18T08:15:18.717 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:15:18 smithi007 ceph-mon[39249]: from='mgr.24470 172.21.15.7:0/2594390261' entity='mgr.y' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.b", "id": [4, 2]} : dispatch 2024-09-18T08:15:18.717 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:15:18 smithi007 ceph-mon[39249]: from='mgr.24470 172.21.15.7:0/2594390261' entity='mgr.y' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.18", "id": [1, 2]} : dispatch 2024-09-18T08:15:18.734 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:15:18.735 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:15:18.735 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0' 2024-09-18T08:15:18.836 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0 2024-09-18T08:15:18.892 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:15:18.893 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:15:18.893 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0' 2024-09-18T08:15:18.905 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:15:18 smithi050 ceph-mon[32371]: pgmap v618: 158 pgs: 158 active+clean; 2.3 GiB data, 9.2 GiB used, 527 GiB / 536 GiB avail; 639 B/s rd, 64 MiB/s wr, 1.02k op/s 2024-09-18T08:15:18.905 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:15:18 smithi050 ceph-mon[32371]: from='mgr.24470 172.21.15.7:0/2594390261' entity='mgr.y' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.b", "id": [4, 2]} : dispatch 2024-09-18T08:15:18.905 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:15:18 smithi050 ceph-mon[32371]: from='mgr.24470 172.21.15.7:0/2594390261' entity='mgr.y' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.18", "id": [1, 2]} : dispatch 2024-09-18T08:15:18.994 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4 2024-09-18T08:15:19.020 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:15:19.021 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:15:19.022 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4' 2024-09-18T08:15:19.056 INFO:teuthology.orchestra.run.smithi007.stdout:osd.0: {} 2024-09-18T08:15:19.056 INFO:teuthology.orchestra.run.smithi007.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-18T08:15:19.094 INFO:teuthology.orchestra.run.smithi007.stdout:osd.1: {} 2024-09-18T08:15:19.094 INFO:teuthology.orchestra.run.smithi007.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-18T08:15:19.123 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/779349ca-7593-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T08:15:19.123 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3 2024-09-18T08:15:19.132 INFO:teuthology.orchestra.run.smithi007.stdout:osd.2: {} 2024-09-18T08:15:19.133 INFO:teuthology.orchestra.run.smithi007.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-18T08:15:19.171 INFO:teuthology.orchestra.run.smithi007.stdout:osd.3: {} 2024-09-18T08:15:19.171 INFO:teuthology.orchestra.run.smithi007.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-18T08:15:19.181 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:15:19.181 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:15:19.182 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3' 2024-09-18T08:15:19.203 INFO:teuthology.orchestra.run.smithi007.stdout:osd.4: {} 2024-09-18T08:15:19.203 INFO:teuthology.orchestra.run.smithi007.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-18T08:15:19.235 INFO:teuthology.orchestra.run.smithi007.stdout:osd.5: {} 2024-09-18T08:15:19.235 INFO:teuthology.orchestra.run.smithi007.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-18T08:15:19.270 INFO:teuthology.orchestra.run.smithi007.stdout:osd.6: {} 2024-09-18T08:15:19.270 INFO:teuthology.orchestra.run.smithi007.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-18T08:15:19.283 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0 2024-09-18T08:15:19.304 INFO:teuthology.orchestra.run.smithi007.stdout:osd.7: {} 2024-09-18T08:15:19.304 INFO:teuthology.orchestra.run.smithi007.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-18T08:15:19.331 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:15:19.332 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:15:19.333 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0' 2024-09-18T08:15:19.340 INFO:tasks.radosbench.radosbench.0.smithi007.stdout: 31 16 46379 46363 93.4545 83.1875 0.0179363 0.010695 2024-09-18T08:15:19.433 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2 2024-09-18T08:15:19.478 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:15:19.478 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:15:19.479 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2' 2024-09-18T08:15:19.580 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3 2024-09-18T08:15:19.618 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:15:19.619 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:15:19.619 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3' 2024-09-18T08:15:19.720 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0 2024-09-18T08:15:19.731 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:15:19 smithi007 ceph-mon[29459]: from='mgr.24470 172.21.15.7:0/2594390261' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.b", "id": [4, 2]}]': finished 2024-09-18T08:15:19.731 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:15:19 smithi007 ceph-mon[29459]: from='mgr.24470 172.21.15.7:0/2594390261' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.18", "id": [1, 2]}]': finished 2024-09-18T08:15:19.731 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:15:19 smithi007 ceph-mon[29459]: osdmap e329: 8 total, 8 up, 6 in 2024-09-18T08:15:19.731 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:15:19 smithi007 ceph-mon[29459]: from='mgr.24470 172.21.15.7:0/2594390261' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-18T08:15:19.731 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:15:19 smithi007 ceph-mon[29459]: from='client.? 172.21.15.7:0/1684100107' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-18T08:15:19.731 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:15:19 smithi007 ceph-mon[39249]: from='mgr.24470 172.21.15.7:0/2594390261' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.b", "id": [4, 2]}]': finished 2024-09-18T08:15:19.731 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:15:19 smithi007 ceph-mon[39249]: from='mgr.24470 172.21.15.7:0/2594390261' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.18", "id": [1, 2]}]': finished 2024-09-18T08:15:19.732 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:15:19 smithi007 ceph-mon[39249]: osdmap e329: 8 total, 8 up, 6 in 2024-09-18T08:15:19.732 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:15:19 smithi007 ceph-mon[39249]: from='mgr.24470 172.21.15.7:0/2594390261' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-18T08:15:19.732 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:15:19 smithi007 ceph-mon[39249]: from='client.? 172.21.15.7:0/1684100107' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-18T08:15:19.760 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:15:19.761 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:15:19.762 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0' 2024-09-18T08:15:19.863 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4 2024-09-18T08:15:19.889 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:15:19.890 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:15:19.890 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4' 2024-09-18T08:15:19.905 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:15:19 smithi050 ceph-mon[32371]: from='mgr.24470 172.21.15.7:0/2594390261' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.b", "id": [4, 2]}]': finished 2024-09-18T08:15:19.905 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:15:19 smithi050 ceph-mon[32371]: from='mgr.24470 172.21.15.7:0/2594390261' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.18", "id": [1, 2]}]': finished 2024-09-18T08:15:19.905 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:15:19 smithi050 ceph-mon[32371]: osdmap e329: 8 total, 8 up, 6 in 2024-09-18T08:15:19.905 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:15:19 smithi050 ceph-mon[32371]: from='mgr.24470 172.21.15.7:0/2594390261' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-18T08:15:19.905 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:15:19 smithi050 ceph-mon[32371]: from='client.? 172.21.15.7:0/1684100107' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-18T08:15:19.906 INFO:journalctl@ceph.iscsi.iscsi.a.smithi050.stdout:Sep 18 08:15:19 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-iscsi-iscsi-a[57782]: debug there is no tcmu-runner data available 2024-09-18T08:15:19.927 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid 779349ca-7593-11ef-bceb-c7b262605968 -- ceph osd set nodeep-scrub 2024-09-18T08:15:19.991 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7 2024-09-18T08:15:20.019 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:15:20.019 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:15:20.020 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7' 2024-09-18T08:15:20.121 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0 2024-09-18T08:15:20.174 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:15:20.175 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:15:20.175 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0' 2024-09-18T08:15:20.276 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3 2024-09-18T08:15:20.300 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:15:20.300 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:15:20.301 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3' 2024-09-18T08:15:20.340 INFO:tasks.radosbench.radosbench.0.smithi007.stdout: 32 16 47736 47720 93.1841 84.8125 0.00275203 0.0107023 2024-09-18T08:15:20.401 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0 2024-09-18T08:15:20.440 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:15:20.441 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:15:20.441 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0' 2024-09-18T08:15:20.542 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7 2024-09-18T08:15:20.568 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:15:20.569 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:15:20.570 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7' 2024-09-18T08:15:20.670 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4 2024-09-18T08:15:20.698 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:15:20.699 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:15:20.699 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4' 2024-09-18T08:15:20.730 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:15:20 smithi007 ceph-mon[39249]: pgmap v620: 158 pgs: 1 peering, 157 active+clean; 2.8 GiB data, 11 GiB used, 526 GiB / 536 GiB avail; 1.2 KiB/s rd, 123 MiB/s wr, 1.97k op/s 2024-09-18T08:15:20.730 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:15:20 smithi007 ceph-mon[39249]: osdmap e330: 8 total, 8 up, 6 in 2024-09-18T08:15:20.730 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:15:20 smithi007 ceph-mon[39249]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-18T08:15:20.730 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:15:20 smithi007 ceph-mon[29459]: pgmap v620: 158 pgs: 1 peering, 157 active+clean; 2.8 GiB data, 11 GiB used, 526 GiB / 536 GiB avail; 1.2 KiB/s rd, 123 MiB/s wr, 1.97k op/s 2024-09-18T08:15:20.730 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:15:20 smithi007 ceph-mon[29459]: osdmap e330: 8 total, 8 up, 6 in 2024-09-18T08:15:20.730 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:15:20 smithi007 ceph-mon[29459]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-18T08:15:20.800 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4 2024-09-18T08:15:20.824 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:15:20.825 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:15:20.825 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4' 2024-09-18T08:15:20.905 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:15:20 smithi050 ceph-mon[32371]: pgmap v620: 158 pgs: 1 peering, 157 active+clean; 2.8 GiB data, 11 GiB used, 526 GiB / 536 GiB avail; 1.2 KiB/s rd, 123 MiB/s wr, 1.97k op/s 2024-09-18T08:15:20.905 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:15:20 smithi050 ceph-mon[32371]: osdmap e330: 8 total, 8 up, 6 in 2024-09-18T08:15:20.905 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:15:20 smithi050 ceph-mon[32371]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-18T08:15:20.926 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7 2024-09-18T08:15:20.957 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:15:20.957 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:15:20.958 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7' 2024-09-18T08:15:21.059 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4 2024-09-18T08:15:21.089 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:15:21.090 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:15:21.090 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4' 2024-09-18T08:15:21.191 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0 2024-09-18T08:15:21.258 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:15:21.258 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:15:21.259 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0' 2024-09-18T08:15:21.341 INFO:tasks.radosbench.radosbench.0.smithi007.stdout: 33 16 49041 49025 92.8299 81.5625 0.00464061 0.0107668 2024-09-18T08:15:21.360 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6 2024-09-18T08:15:21.389 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:15:21.389 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:15:21.390 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6' 2024-09-18T08:15:21.465 INFO:teuthology.orchestra.run.smithi007.stdout:ERROR: (22) Invalid argument 2024-09-18T08:15:21.466 INFO:teuthology.orchestra.run.smithi007.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-18T08:15:21.490 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6 2024-09-18T08:15:21.518 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:15:21.519 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:15:21.519 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6' 2024-09-18T08:15:21.620 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1 2024-09-18T08:15:21.660 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:15:21.661 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:15:21.662 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1' 2024-09-18T08:15:21.762 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2 2024-09-18T08:15:21.808 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:15:21.809 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:15:21.809 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2' 2024-09-18T08:15:21.910 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3 2024-09-18T08:15:21.948 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:15:21.948 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:15:21.949 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3' 2024-09-18T08:15:22.050 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0 2024-09-18T08:15:22.091 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:15:22.092 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:15:22.092 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0' 2024-09-18T08:15:22.193 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7 2024-09-18T08:15:22.221 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:15:22.221 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:15:22.222 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7' 2024-09-18T08:15:22.323 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0 2024-09-18T08:15:22.341 INFO:tasks.radosbench.radosbench.0.smithi007.stdout: 34 16 50489 50473 92.7603 90.5 0.0112952 0.010775 2024-09-18T08:15:22.366 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:15:22.367 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:15:22.368 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0' 2024-09-18T08:15:22.433 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 6, 4] out_osds: [7, 5] dead_osds: [] live_osds: [0, 2, 3, 4, 5, 6, 7, 1] 2024-09-18T08:15:22.434 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-18T08:15:22.434 INFO:tasks.thrashosds.thrasher:primary_affinity 2024-09-18T08:15:22.434 INFO:tasks.thrashosds.thrasher:Setting osd 1 primary_affinity to 1.000000 2024-09-18T08:15:22.434 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid 779349ca-7593-11ef-bceb-c7b262605968 -- ceph osd primary-affinity 1 1 2024-09-18T08:15:22.469 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5 2024-09-18T08:15:22.499 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:15:22.499 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:15:22.500 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5' 2024-09-18T08:15:22.601 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7 2024-09-18T08:15:22.627 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:15:22.628 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:15:22.628 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7' 2024-09-18T08:15:22.729 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7 2024-09-18T08:15:22.730 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:15:22 smithi007 ceph-mon[29459]: pgmap v622: 158 pgs: 1 peering, 157 active+clean; 2.8 GiB data, 11 GiB used, 526 GiB / 536 GiB avail; 853 B/s rd, 79 MiB/s wr, 1.26k op/s 2024-09-18T08:15:22.730 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:15:22 smithi007 ceph-mon[39249]: pgmap v622: 158 pgs: 1 peering, 157 active+clean; 2.8 GiB data, 11 GiB used, 526 GiB / 536 GiB avail; 853 B/s rd, 79 MiB/s wr, 1.26k op/s 2024-09-18T08:15:22.753 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:15:22.754 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:15:22.754 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7' 2024-09-18T08:15:22.755 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/779349ca-7593-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T08:15:22.803 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid 779349ca-7593-11ef-bceb-c7b262605968 -- ceph daemon osd.1 dump_historic_ops 2024-09-18T08:15:22.855 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7 2024-09-18T08:15:22.885 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:15:22.885 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:15:22.886 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7' 2024-09-18T08:15:22.905 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:15:22 smithi050 ceph-mon[32371]: pgmap v622: 158 pgs: 1 peering, 157 active+clean; 2.8 GiB data, 11 GiB used, 526 GiB / 536 GiB avail; 853 B/s rd, 79 MiB/s wr, 1.26k op/s 2024-09-18T08:15:22.987 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5 2024-09-18T08:15:23.014 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:15:23.015 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:15:23.015 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5' 2024-09-18T08:15:23.116 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2 2024-09-18T08:15:23.179 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:15:23.181 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:15:23.182 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2' 2024-09-18T08:15:23.283 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3 2024-09-18T08:15:23.331 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:15:23.332 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:15:23.332 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3' 2024-09-18T08:15:23.341 INFO:tasks.radosbench.radosbench.0.smithi007.stdout: 35 16 51809 51793 92.467 82.5 0.010203 0.0108094 2024-09-18T08:15:23.374 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/779349ca-7593-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T08:15:23.433 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6 2024-09-18T08:15:23.458 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:15:23.458 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:15:23.459 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6' 2024-09-18T08:15:23.560 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6 2024-09-18T08:15:23.585 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:15:23.586 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:15:23.586 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6' 2024-09-18T08:15:23.687 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5 2024-09-18T08:15:23.712 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:15:23.713 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:15:23.713 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5' 2024-09-18T08:15:23.814 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1 2024-09-18T08:15:23.861 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:15:23.861 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:15:23.862 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1' 2024-09-18T08:15:23.963 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2 2024-09-18T08:15:24.003 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid 779349ca-7593-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-18T08:15:24.007 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:15:24.009 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:15:24.010 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2' 2024-09-18T08:15:24.111 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7 2024-09-18T08:15:24.127 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/779349ca-7593-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T08:15:24.136 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:15:24.136 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:15:24.137 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7' 2024-09-18T08:15:24.238 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3 2024-09-18T08:15:24.298 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:15:24.299 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:15:24.299 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3' 2024-09-18T08:15:24.341 INFO:tasks.radosbench.radosbench.0.smithi007.stdout: 36 16 52902 52886 91.7959 68.3125 0.00908415 0.010888 2024-09-18T08:15:24.371 INFO:teuthology.orchestra.run.smithi007.stderr:nodeep-scrub is set 2024-09-18T08:15:24.400 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1 2024-09-18T08:15:24.455 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:15:24.456 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:15:24.456 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1' 2024-09-18T08:15:24.557 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0 2024-09-18T08:15:24.579 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:15:24.580 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:15:24.580 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0' 2024-09-18T08:15:24.681 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2 2024-09-18T08:15:24.703 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:15:24.704 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:15:24.704 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2' 2024-09-18T08:15:24.730 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:15:24 smithi007 ceph-mon[29459]: 1.1 scrub starts 2024-09-18T08:15:24.730 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:15:24 smithi007 ceph-mon[29459]: pgmap v623: 158 pgs: 1 peering, 157 active+clean; 2.8 GiB data, 11 GiB used, 526 GiB / 536 GiB avail; 1.2 KiB/s rd, 69 MiB/s wr, 1.11k op/s 2024-09-18T08:15:24.730 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:15:24 smithi007 ceph-mon[29459]: from='client.? 172.21.15.7:0/1838842132' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-18T08:15:24.730 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:15:24 smithi007 ceph-mon[29459]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-18T08:15:24.730 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:15:24 smithi007 ceph-mon[29459]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-18T08:15:24.730 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:15:24 smithi007 ceph-mon[29459]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-18T08:15:24.730 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:15:24 smithi007 ceph-mon[29459]: osdmap e331: 8 total, 8 up, 6 in 2024-09-18T08:15:24.730 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:15:24 smithi007 ceph-mon[29459]: from='client.? 172.21.15.7:0/39197799' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 1, "weight": 1.0} : dispatch 2024-09-18T08:15:24.730 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:15:24 smithi007 ceph-mon[29459]: from='client.? ' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 1, "weight": 1.0} : dispatch 2024-09-18T08:15:24.731 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:15:24 smithi007 ceph-mon[39249]: 1.1 scrub starts 2024-09-18T08:15:24.731 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:15:24 smithi007 ceph-mon[39249]: pgmap v623: 158 pgs: 1 peering, 157 active+clean; 2.8 GiB data, 11 GiB used, 526 GiB / 536 GiB avail; 1.2 KiB/s rd, 69 MiB/s wr, 1.11k op/s 2024-09-18T08:15:24.731 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:15:24 smithi007 ceph-mon[39249]: from='client.? 172.21.15.7:0/1838842132' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-18T08:15:24.731 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:15:24 smithi007 ceph-mon[39249]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-18T08:15:24.731 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:15:24 smithi007 ceph-mon[39249]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-18T08:15:24.731 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:15:24 smithi007 ceph-mon[39249]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-18T08:15:24.731 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:15:24 smithi007 ceph-mon[39249]: osdmap e331: 8 total, 8 up, 6 in 2024-09-18T08:15:24.731 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:15:24 smithi007 ceph-mon[39249]: from='client.? 172.21.15.7:0/39197799' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 1, "weight": 1.0} : dispatch 2024-09-18T08:15:24.731 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:15:24 smithi007 ceph-mon[39249]: from='client.? ' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 1, "weight": 1.0} : dispatch 2024-09-18T08:15:24.805 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0 2024-09-18T08:15:24.827 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:15:24.828 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:15:24.828 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0' 2024-09-18T08:15:24.905 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:15:24 smithi050 ceph-mon[32371]: 1.1 scrub starts 2024-09-18T08:15:24.906 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:15:24 smithi050 ceph-mon[32371]: pgmap v623: 158 pgs: 1 peering, 157 active+clean; 2.8 GiB data, 11 GiB used, 526 GiB / 536 GiB avail; 1.2 KiB/s rd, 69 MiB/s wr, 1.11k op/s 2024-09-18T08:15:24.906 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:15:24 smithi050 ceph-mon[32371]: from='client.? 172.21.15.7:0/1838842132' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-18T08:15:24.906 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:15:24 smithi050 ceph-mon[32371]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-18T08:15:24.906 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:15:24 smithi050 ceph-mon[32371]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-18T08:15:24.906 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:15:24 smithi050 ceph-mon[32371]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-18T08:15:24.906 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:15:24 smithi050 ceph-mon[32371]: osdmap e331: 8 total, 8 up, 6 in 2024-09-18T08:15:24.906 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:15:24 smithi050 ceph-mon[32371]: from='client.? 172.21.15.7:0/39197799' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 1, "weight": 1.0} : dispatch 2024-09-18T08:15:24.906 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:15:24 smithi050 ceph-mon[32371]: from='client.? ' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 1, "weight": 1.0} : dispatch 2024-09-18T08:15:24.929 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4 2024-09-18T08:15:24.956 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:15:24.957 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:15:24.957 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4' 2024-09-18T08:15:25.059 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6 2024-09-18T08:15:25.087 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:15:25.088 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:15:25.088 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6' 2024-09-18T08:15:25.189 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3 2024-09-18T08:15:25.232 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:15:25.232 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:15:25.233 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3' 2024-09-18T08:15:25.333 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1 2024-09-18T08:15:25.343 INFO:tasks.radosbench.radosbench.0.smithi007.stdout: 37 16 53867 53851 90.9413 60.3125 0.0107685 0.01099 2024-09-18T08:15:25.367 INFO:teuthology.orchestra.run.smithi007.stderr:set osd.1 primary-affinity to 1 (8655362) 2024-09-18T08:15:25.389 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:15:25.390 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:15:25.391 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1' 2024-09-18T08:15:25.492 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5 2024-09-18T08:15:25.520 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:15:25.521 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:15:25.521 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5' 2024-09-18T08:15:25.622 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6 2024-09-18T08:15:25.650 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:15:25.650 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:15:25.651 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6' 2024-09-18T08:15:25.730 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:15:25 smithi007 ceph-mon[29459]: 1.a scrub starts 2024-09-18T08:15:25.730 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:15:25 smithi007 ceph-mon[29459]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 1, "weight": 1.0}]': finished 2024-09-18T08:15:25.730 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:15:25 smithi007 ceph-mon[29459]: osdmap e332: 8 total, 8 up, 6 in 2024-09-18T08:15:25.730 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:15:25 smithi007 ceph-mon[39249]: 1.a scrub starts 2024-09-18T08:15:25.730 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:15:25 smithi007 ceph-mon[39249]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 1, "weight": 1.0}]': finished 2024-09-18T08:15:25.731 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:15:25 smithi007 ceph-mon[39249]: osdmap e332: 8 total, 8 up, 6 in 2024-09-18T08:15:25.751 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4 2024-09-18T08:15:25.778 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:15:25.778 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:15:25.779 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4' 2024-09-18T08:15:25.880 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5 2024-09-18T08:15:25.905 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:15:25 smithi050 ceph-mon[32371]: 1.a scrub starts 2024-09-18T08:15:25.905 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:15:25 smithi050 ceph-mon[32371]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 1, "weight": 1.0}]': finished 2024-09-18T08:15:25.905 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:15:25 smithi050 ceph-mon[32371]: osdmap e332: 8 total, 8 up, 6 in 2024-09-18T08:15:25.906 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:15:25.907 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:15:25.908 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5' 2024-09-18T08:15:25.938 INFO:teuthology.orchestra.run.smithi007.stdout:ERROR: (22) Invalid argument 2024-09-18T08:15:25.938 INFO:teuthology.orchestra.run.smithi007.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-18T08:15:26.008 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2 2024-09-18T08:15:26.048 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:15:26.048 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:15:26.049 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2' 2024-09-18T08:15:26.149 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6 2024-09-18T08:15:26.177 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:15:26.178 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:15:26.178 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6' 2024-09-18T08:15:26.279 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7 2024-09-18T08:15:26.306 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:15:26.307 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:15:26.307 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7' 2024-09-18T08:15:26.343 INFO:tasks.radosbench.radosbench.0.smithi007.stdout: 38 16 55225 55209 90.7814 84.875 0.00892814 0.0110103 2024-09-18T08:15:26.408 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3 2024-09-18T08:15:26.461 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:15:26.462 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:15:26.462 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3' 2024-09-18T08:15:26.564 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7 2024-09-18T08:15:26.570 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:15:26 smithi007 ceph-mon[29459]: pgmap v625: 158 pgs: 1 active+clean+scrubbing, 157 active+clean; 3.2 GiB data, 12 GiB used, 525 GiB / 536 GiB avail; 739 B/s rd, 98 MiB/s wr, 1.56k op/s; 15 B/s, 0 objects/s recovering 2024-09-18T08:15:26.571 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:15:26 smithi007 ceph-mon[39249]: pgmap v625: 158 pgs: 1 active+clean+scrubbing, 157 active+clean; 3.2 GiB data, 12 GiB used, 525 GiB / 536 GiB avail; 739 B/s rd, 98 MiB/s wr, 1.56k op/s; 15 B/s, 0 objects/s recovering 2024-09-18T08:15:26.591 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:15:26.592 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:15:26.592 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7' 2024-09-18T08:15:26.693 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2 2024-09-18T08:15:26.730 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:15:26.731 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:15:26.731 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2' 2024-09-18T08:15:26.832 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3 2024-09-18T08:15:26.875 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:15:26.876 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:15:26.877 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3' 2024-09-18T08:15:26.905 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:15:26 smithi050 ceph-mon[32371]: pgmap v625: 158 pgs: 1 active+clean+scrubbing, 157 active+clean; 3.2 GiB data, 12 GiB used, 525 GiB / 536 GiB avail; 739 B/s rd, 98 MiB/s wr, 1.56k op/s; 15 B/s, 0 objects/s recovering 2024-09-18T08:15:26.978 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6 2024-09-18T08:15:26.980 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Sep 18 08:15:26 smithi007 ceph-779349ca-7593-11ef-bceb-c7b262605968-mgr-y[29688]: ::ffff:172.21.15.50 - - [18/Sep/2024:08:15:26] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-09-18T08:15:27.004 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:15:27.005 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:15:27.005 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6' 2024-09-18T08:15:27.106 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4 2024-09-18T08:15:27.135 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:15:27.136 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:15:27.136 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4' 2024-09-18T08:15:27.237 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7 2024-09-18T08:15:27.265 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:15:27.265 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:15:27.266 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7' 2024-09-18T08:15:27.343 INFO:tasks.radosbench.radosbench.0.smithi007.stdout: 39 16 56655 56639 90.7452 89.375 0.00539328 0.0110145 2024-09-18T08:15:27.366 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5 2024-09-18T08:15:27.394 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:15:27.395 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:15:27.395 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5' 2024-09-18T08:15:27.496 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1 2024-09-18T08:15:27.541 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:15:27.542 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:15:27.542 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1' 2024-09-18T08:15:27.626 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid 779349ca-7593-11ef-bceb-c7b262605968 -- ceph daemon osd.0 dump_ops_in_flight 2024-09-18T08:15:27.643 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3 2024-09-18T08:15:27.713 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:15:27.713 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:15:27.713 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3' 2024-09-18T08:15:27.814 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7 2024-09-18T08:15:27.830 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/779349ca-7593-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T08:15:27.839 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:15:27.840 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:15:27.840 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7' 2024-09-18T08:15:27.941 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2 2024-09-18T08:15:27.991 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:15:27.992 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:15:27.993 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2' 2024-09-18T08:15:28.093 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7 2024-09-18T08:15:28.123 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:15:28.124 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:15:28.124 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7' 2024-09-18T08:15:28.225 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0 2024-09-18T08:15:28.260 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:15:28.260 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:15:28.261 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0' 2024-09-18T08:15:28.343 INFO:tasks.radosbench.radosbench.0.smithi007.stdout:2024-09-18T08:15:28.342719+0000 min lat: 0.00117701 max lat: 2.07039 avg lat: 0.0110597 2024-09-18T08:15:28.343 INFO:tasks.radosbench.radosbench.0.smithi007.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-09-18T08:15:28.343 INFO:tasks.radosbench.radosbench.0.smithi007.stdout: 40 16 57871 57855 90.3764 76 0.00979944 0.0110597 2024-09-18T08:15:28.362 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1 2024-09-18T08:15:28.398 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:15:28.399 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:15:28.399 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1' 2024-09-18T08:15:28.486 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid 779349ca-7593-11ef-bceb-c7b262605968 -- ceph osd unset noscrub 2024-09-18T08:15:28.494 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/779349ca-7593-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T08:15:28.500 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6 2024-09-18T08:15:28.523 INFO:teuthology.orchestra.run.smithi007.stdout:osd.0: {} 2024-09-18T08:15:28.523 INFO:teuthology.orchestra.run.smithi007.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-18T08:15:28.525 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:15:28.526 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:15:28.526 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6' 2024-09-18T08:15:28.560 INFO:teuthology.orchestra.run.smithi007.stdout:osd.1: {} 2024-09-18T08:15:28.560 INFO:teuthology.orchestra.run.smithi007.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-18T08:15:28.601 INFO:teuthology.orchestra.run.smithi007.stdout:osd.2: {} 2024-09-18T08:15:28.601 INFO:teuthology.orchestra.run.smithi007.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-18T08:15:28.601 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:15:28 smithi007 ceph-mon[39249]: purged_snaps scrub starts 2024-09-18T08:15:28.601 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:15:28 smithi007 ceph-mon[39249]: purged_snaps scrub ok 2024-09-18T08:15:28.602 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:15:28 smithi007 ceph-mon[39249]: pgmap v627: 158 pgs: 1 active+clean+scrubbing, 157 active+clean; 3.2 GiB data, 12 GiB used, 525 GiB / 536 GiB avail; 646 B/s rd, 53 MiB/s wr, 845 op/s; 13 B/s, 0 objects/s recovering 2024-09-18T08:15:28.602 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:15:28 smithi007 ceph-mon[39249]: osdmap e333: 8 total, 8 up, 6 in 2024-09-18T08:15:28.602 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:15:28 smithi007 ceph-mon[39249]: from='client.? 172.21.15.7:0/3171762843' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-18T08:15:28.602 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:15:28 smithi007 ceph-mon[29459]: purged_snaps scrub starts 2024-09-18T08:15:28.602 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:15:28 smithi007 ceph-mon[29459]: purged_snaps scrub ok 2024-09-18T08:15:28.602 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:15:28 smithi007 ceph-mon[29459]: pgmap v627: 158 pgs: 1 active+clean+scrubbing, 157 active+clean; 3.2 GiB data, 12 GiB used, 525 GiB / 536 GiB avail; 646 B/s rd, 53 MiB/s wr, 845 op/s; 13 B/s, 0 objects/s recovering 2024-09-18T08:15:28.602 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:15:28 smithi007 ceph-mon[29459]: osdmap e333: 8 total, 8 up, 6 in 2024-09-18T08:15:28.602 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:15:28 smithi007 ceph-mon[29459]: from='client.? 172.21.15.7:0/3171762843' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-18T08:15:28.627 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5 2024-09-18T08:15:28.634 INFO:teuthology.orchestra.run.smithi007.stdout:osd.3: {} 2024-09-18T08:15:28.634 INFO:teuthology.orchestra.run.smithi007.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-18T08:15:28.653 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:15:28.653 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:15:28.654 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5' 2024-09-18T08:15:28.676 INFO:teuthology.orchestra.run.smithi007.stdout:osd.4: {} 2024-09-18T08:15:28.676 INFO:teuthology.orchestra.run.smithi007.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-18T08:15:28.710 INFO:teuthology.orchestra.run.smithi007.stdout:osd.5: {} 2024-09-18T08:15:28.710 INFO:teuthology.orchestra.run.smithi007.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-18T08:15:28.747 INFO:teuthology.orchestra.run.smithi007.stdout:osd.6: {} 2024-09-18T08:15:28.748 INFO:teuthology.orchestra.run.smithi007.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-18T08:15:28.754 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3 2024-09-18T08:15:28.786 INFO:teuthology.orchestra.run.smithi007.stdout:osd.7: {} 2024-09-18T08:15:28.786 INFO:teuthology.orchestra.run.smithi007.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-18T08:15:28.849 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:15:28.850 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:15:28.850 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3' 2024-09-18T08:15:28.905 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:15:28 smithi050 ceph-mon[32371]: purged_snaps scrub starts 2024-09-18T08:15:28.906 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:15:28 smithi050 ceph-mon[32371]: purged_snaps scrub ok 2024-09-18T08:15:28.906 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:15:28 smithi050 ceph-mon[32371]: pgmap v627: 158 pgs: 1 active+clean+scrubbing, 157 active+clean; 3.2 GiB data, 12 GiB used, 525 GiB / 536 GiB avail; 646 B/s rd, 53 MiB/s wr, 845 op/s; 13 B/s, 0 objects/s recovering 2024-09-18T08:15:28.906 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:15:28 smithi050 ceph-mon[32371]: osdmap e333: 8 total, 8 up, 6 in 2024-09-18T08:15:28.906 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:15:28 smithi050 ceph-mon[32371]: from='client.? 172.21.15.7:0/3171762843' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-18T08:15:28.951 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7 2024-09-18T08:15:28.980 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:15:28.980 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:15:28.980 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7' 2024-09-18T08:15:29.081 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7 2024-09-18T08:15:29.107 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:15:29.107 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:15:29.108 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7' 2024-09-18T08:15:29.208 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4 2024-09-18T08:15:29.234 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:15:29.235 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:15:29.235 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4' 2024-09-18T08:15:29.285 INFO:teuthology.orchestra.run.smithi007.stdout:{ 2024-09-18T08:15:29.285 INFO:teuthology.orchestra.run.smithi007.stdout: "ops": [ 2024-09-18T08:15:29.285 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:15:29.285 INFO:teuthology.orchestra.run.smithi007.stdout: "description": "osd_repop(client.15966.0:58926 10.5 e333/311 10:a4e3dcb2:::benchmark_data_smithi007_148834_object58925:head v 333'3733, mlcod=333'3729)", 2024-09-18T08:15:29.285 INFO:teuthology.orchestra.run.smithi007.stdout: "initiated_at": "2024-09-18T08:15:29.276831+0000", 2024-09-18T08:15:29.286 INFO:teuthology.orchestra.run.smithi007.stdout: "age": 0.0043954140000000003, 2024-09-18T08:15:29.286 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.0044219790000000004, 2024-09-18T08:15:29.286 INFO:teuthology.orchestra.run.smithi007.stdout: "continuous": false, 2024-09-18T08:15:29.286 INFO:teuthology.orchestra.run.smithi007.stdout: "type_data": { 2024-09-18T08:15:29.286 INFO:teuthology.orchestra.run.smithi007.stdout: "flag_point": "started", 2024-09-18T08:15:29.286 INFO:teuthology.orchestra.run.smithi007.stdout: "events": [ 2024-09-18T08:15:29.286 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:15:29.286 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "initiated", 2024-09-18T08:15:29.286 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:15:29.276831+0000", 2024-09-18T08:15:29.286 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-09-18T08:15:29.286 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:15:29.286 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:15:29.286 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "header_read", 2024-09-18T08:15:29.287 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:15:29.276831+0000", 2024-09-18T08:15:29.287 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-09-18T08:15:29.287 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:15:29.287 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:15:29.287 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "throttled", 2024-09-18T08:15:29.287 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:15:29.276832+0000", 2024-09-18T08:15:29.287 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 8.9700000000000005e-07 2024-09-18T08:15:29.287 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:15:29.287 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:15:29.287 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "all_read", 2024-09-18T08:15:29.287 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:15:29.276860+0000", 2024-09-18T08:15:29.287 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 2.8626e-05 2024-09-18T08:15:29.287 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:15:29.288 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:15:29.288 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "dispatched", 2024-09-18T08:15:29.288 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:15:29.276870+0000", 2024-09-18T08:15:29.288 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 9.5559999999999996e-06 2024-09-18T08:15:29.288 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:15:29.288 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:15:29.288 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "queued_for_pg", 2024-09-18T08:15:29.288 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:15:29.276876+0000", 2024-09-18T08:15:29.288 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 6.2940000000000002e-06 2024-09-18T08:15:29.288 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:15:29.288 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:15:29.288 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-09-18T08:15:29.288 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:15:29.276946+0000", 2024-09-18T08:15:29.288 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 6.9562999999999998e-05 2024-09-18T08:15:29.289 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:15:29.289 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:15:29.289 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "started", 2024-09-18T08:15:29.289 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:15:29.276965+0000", 2024-09-18T08:15:29.289 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.9521000000000002e-05 2024-09-18T08:15:29.289 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-09-18T08:15:29.289 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-09-18T08:15:29.289 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-09-18T08:15:29.290 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:15:29.290 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:15:29.290 INFO:teuthology.orchestra.run.smithi007.stdout: "description": "osd_op(client.15966.0:58928 10.6 10:628c2a6f:::benchmark_data_smithi007_148834_object58927:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536 in=65536b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e333)", 2024-09-18T08:15:29.290 INFO:teuthology.orchestra.run.smithi007.stdout: "initiated_at": "2024-09-18T08:15:29.277227+0000", 2024-09-18T08:15:29.290 INFO:teuthology.orchestra.run.smithi007.stdout: "age": 0.0039991389999999996, 2024-09-18T08:15:29.290 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.0040874889999999997, 2024-09-18T08:15:29.290 INFO:teuthology.orchestra.run.smithi007.stdout: "continuous": false, 2024-09-18T08:15:29.290 INFO:teuthology.orchestra.run.smithi007.stdout: "type_data": { 2024-09-18T08:15:29.291 INFO:teuthology.orchestra.run.smithi007.stdout: "flag_point": "waiting for sub ops", 2024-09-18T08:15:29.291 INFO:teuthology.orchestra.run.smithi007.stdout: "client_info": { 2024-09-18T08:15:29.291 INFO:teuthology.orchestra.run.smithi007.stdout: "client": "client.15966", 2024-09-18T08:15:29.291 INFO:teuthology.orchestra.run.smithi007.stdout: "client_addr": "172.21.15.7:0/529784850", 2024-09-18T08:15:29.291 INFO:teuthology.orchestra.run.smithi007.stdout: "tid": 58928 2024-09-18T08:15:29.291 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:15:29.291 INFO:teuthology.orchestra.run.smithi007.stdout: "events": [ 2024-09-18T08:15:29.291 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:15:29.291 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "initiated", 2024-09-18T08:15:29.291 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:15:29.277227+0000", 2024-09-18T08:15:29.291 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-09-18T08:15:29.291 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:15:29.291 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:15:29.292 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "header_read", 2024-09-18T08:15:29.292 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:15:29.277227+0000", 2024-09-18T08:15:29.292 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-09-18T08:15:29.292 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:15:29.292 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:15:29.292 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "throttled", 2024-09-18T08:15:29.292 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:15:29.277229+0000", 2024-09-18T08:15:29.292 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.57e-06 2024-09-18T08:15:29.292 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:15:29.292 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:15:29.292 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "all_read", 2024-09-18T08:15:29.292 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:15:29.277280+0000", 2024-09-18T08:15:29.293 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 5.1122999999999998e-05 2024-09-18T08:15:29.293 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:15:29.293 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:15:29.293 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "dispatched", 2024-09-18T08:15:29.293 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:15:29.277298+0000", 2024-09-18T08:15:29.293 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.7983e-05 2024-09-18T08:15:29.293 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:15:29.293 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:15:29.293 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "queued_for_pg", 2024-09-18T08:15:29.293 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:15:29.277306+0000", 2024-09-18T08:15:29.293 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 7.9389999999999997e-06 2024-09-18T08:15:29.293 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:15:29.293 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:15:29.294 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-09-18T08:15:29.294 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:15:29.277452+0000", 2024-09-18T08:15:29.294 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.000146768 2024-09-18T08:15:29.294 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:15:29.294 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:15:29.294 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "started", 2024-09-18T08:15:29.294 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:15:29.277612+0000", 2024-09-18T08:15:29.294 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.00015961999999999999 2024-09-18T08:15:29.294 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:15:29.294 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:15:29.294 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "waiting for subops from 1,4", 2024-09-18T08:15:29.294 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:15:29.277754+0000", 2024-09-18T08:15:29.294 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.000141998 2024-09-18T08:15:29.294 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:15:29.295 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:15:29.295 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "op_commit", 2024-09-18T08:15:29.295 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:15:29.279840+0000", 2024-09-18T08:15:29.295 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.002085726 2024-09-18T08:15:29.295 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-09-18T08:15:29.295 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-09-18T08:15:29.295 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-09-18T08:15:29.295 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:15:29.295 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:15:29.295 INFO:teuthology.orchestra.run.smithi007.stdout: "description": "osd_op(client.15966.0:58929 10.6 10:605114dc:::benchmark_data_smithi007_148834_object58928:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536 in=65536b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e333)", 2024-09-18T08:15:29.295 INFO:teuthology.orchestra.run.smithi007.stdout: "initiated_at": "2024-09-18T08:15:29.277473+0000", 2024-09-18T08:15:29.295 INFO:teuthology.orchestra.run.smithi007.stdout: "age": 0.0037533969999999999, 2024-09-18T08:15:29.295 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.0039108010000000002, 2024-09-18T08:15:29.295 INFO:teuthology.orchestra.run.smithi007.stdout: "continuous": false, 2024-09-18T08:15:29.296 INFO:teuthology.orchestra.run.smithi007.stdout: "type_data": { 2024-09-18T08:15:29.296 INFO:teuthology.orchestra.run.smithi007.stdout: "flag_point": "waiting for sub ops", 2024-09-18T08:15:29.296 INFO:teuthology.orchestra.run.smithi007.stdout: "client_info": { 2024-09-18T08:15:29.296 INFO:teuthology.orchestra.run.smithi007.stdout: "client": "client.15966", 2024-09-18T08:15:29.296 INFO:teuthology.orchestra.run.smithi007.stdout: "client_addr": "172.21.15.7:0/529784850", 2024-09-18T08:15:29.296 INFO:teuthology.orchestra.run.smithi007.stdout: "tid": 58929 2024-09-18T08:15:29.296 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:15:29.296 INFO:teuthology.orchestra.run.smithi007.stdout: "events": [ 2024-09-18T08:15:29.296 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:15:29.296 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "initiated", 2024-09-18T08:15:29.296 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:15:29.277473+0000", 2024-09-18T08:15:29.296 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-09-18T08:15:29.296 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:15:29.297 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:15:29.297 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "header_read", 2024-09-18T08:15:29.297 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:15:29.277473+0000", 2024-09-18T08:15:29.297 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-09-18T08:15:29.297 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:15:29.297 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:15:29.297 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "throttled", 2024-09-18T08:15:29.297 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:15:29.277474+0000", 2024-09-18T08:15:29.297 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 7.3099999999999997e-07 2024-09-18T08:15:29.297 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:15:29.297 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:15:29.297 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "all_read", 2024-09-18T08:15:29.297 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:15:29.277524+0000", 2024-09-18T08:15:29.298 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 5.0659999999999999e-05 2024-09-18T08:15:29.298 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:15:29.298 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:15:29.298 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "dispatched", 2024-09-18T08:15:29.298 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:15:29.277533+0000", 2024-09-18T08:15:29.298 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 9.0729999999999994e-06 2024-09-18T08:15:29.298 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:15:29.298 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:15:29.298 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "queued_for_pg", 2024-09-18T08:15:29.298 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:15:29.277540+0000", 2024-09-18T08:15:29.298 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 6.4479999999999999e-06 2024-09-18T08:15:29.298 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:15:29.298 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:15:29.298 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-09-18T08:15:29.299 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:15:29.280321+0000", 2024-09-18T08:15:29.299 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.002781478 2024-09-18T08:15:29.299 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:15:29.299 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:15:29.299 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "started", 2024-09-18T08:15:29.299 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:15:29.280483+0000", 2024-09-18T08:15:29.299 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.00016129800000000001 2024-09-18T08:15:29.299 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:15:29.299 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:15:29.299 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "waiting for subops from 1,4", 2024-09-18T08:15:29.299 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:15:29.280626+0000", 2024-09-18T08:15:29.299 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.00014371300000000001 2024-09-18T08:15:29.299 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-09-18T08:15:29.300 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-09-18T08:15:29.300 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-09-18T08:15:29.300 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:15:29.300 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:15:29.300 INFO:teuthology.orchestra.run.smithi007.stdout: "description": "osd_op(client.15966.0:58930 10.a 10:5facbbe2:::benchmark_data_smithi007_148834_object58929:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536 in=65536b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e333)", 2024-09-18T08:15:29.300 INFO:teuthology.orchestra.run.smithi007.stdout: "initiated_at": "2024-09-18T08:15:29.277667+0000", 2024-09-18T08:15:29.300 INFO:teuthology.orchestra.run.smithi007.stdout: "age": 0.0035593249999999999, 2024-09-18T08:15:29.300 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.0037787680000000001, 2024-09-18T08:15:29.300 INFO:teuthology.orchestra.run.smithi007.stdout: "continuous": false, 2024-09-18T08:15:29.300 INFO:teuthology.orchestra.run.smithi007.stdout: "type_data": { 2024-09-18T08:15:29.300 INFO:teuthology.orchestra.run.smithi007.stdout: "flag_point": "waiting for sub ops", 2024-09-18T08:15:29.300 INFO:teuthology.orchestra.run.smithi007.stdout: "client_info": { 2024-09-18T08:15:29.301 INFO:teuthology.orchestra.run.smithi007.stdout: "client": "client.15966", 2024-09-18T08:15:29.301 INFO:teuthology.orchestra.run.smithi007.stdout: "client_addr": "172.21.15.7:0/529784850", 2024-09-18T08:15:29.301 INFO:teuthology.orchestra.run.smithi007.stdout: "tid": 58930 2024-09-18T08:15:29.301 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:15:29.301 INFO:teuthology.orchestra.run.smithi007.stdout: "events": [ 2024-09-18T08:15:29.301 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:15:29.301 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "initiated", 2024-09-18T08:15:29.301 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:15:29.277667+0000", 2024-09-18T08:15:29.301 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-09-18T08:15:29.301 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:15:29.301 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:15:29.301 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "header_read", 2024-09-18T08:15:29.301 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:15:29.277667+0000", 2024-09-18T08:15:29.301 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-09-18T08:15:29.302 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:15:29.302 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:15:29.302 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "throttled", 2024-09-18T08:15:29.302 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:15:29.277668+0000", 2024-09-18T08:15:29.302 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 6.7000000000000004e-07 2024-09-18T08:15:29.302 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:15:29.302 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:15:29.302 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "all_read", 2024-09-18T08:15:29.302 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:15:29.277714+0000", 2024-09-18T08:15:29.302 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 4.6254000000000001e-05 2024-09-18T08:15:29.302 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:15:29.302 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:15:29.302 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "dispatched", 2024-09-18T08:15:29.302 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:15:29.277723+0000", 2024-09-18T08:15:29.303 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 9.0739999999999999e-06 2024-09-18T08:15:29.303 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:15:29.303 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:15:29.303 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "queued_for_pg", 2024-09-18T08:15:29.303 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:15:29.277730+0000", 2024-09-18T08:15:29.303 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 6.781e-06 2024-09-18T08:15:29.303 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:15:29.303 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:15:29.303 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-09-18T08:15:29.303 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:15:29.277793+0000", 2024-09-18T08:15:29.303 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 6.3268999999999993e-05 2024-09-18T08:15:29.303 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:15:29.303 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:15:29.304 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "started", 2024-09-18T08:15:29.304 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:15:29.277930+0000", 2024-09-18T08:15:29.304 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.00013753800000000001 2024-09-18T08:15:29.304 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:15:29.304 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:15:29.304 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "waiting for subops from 2,6", 2024-09-18T08:15:29.304 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:15:29.278062+0000", 2024-09-18T08:15:29.304 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.00013138000000000001 2024-09-18T08:15:29.304 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:15:29.304 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:15:29.304 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:15:29.304 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:15:29.280839+0000", 2024-09-18T08:15:29.304 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.0027776300000000001 2024-09-18T08:15:29.305 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:15:29.305 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:15:29.305 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "op_commit", 2024-09-18T08:15:29.305 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:15:29.281450+0000", 2024-09-18T08:15:29.305 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.00061013599999999995 2024-09-18T08:15:29.305 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-09-18T08:15:29.305 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-09-18T08:15:29.305 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-09-18T08:15:29.305 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:15:29.305 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:15:29.305 INFO:teuthology.orchestra.run.smithi007.stdout: "description": "osd_repop(client.15966.0:58927 10.2 e333/311 10:477ac7e7:::benchmark_data_smithi007_148834_object58926:head v 333'3682, mlcod=333'3680)", 2024-09-18T08:15:29.305 INFO:teuthology.orchestra.run.smithi007.stdout: "initiated_at": "2024-09-18T08:15:29.279309+0000", 2024-09-18T08:15:29.305 INFO:teuthology.orchestra.run.smithi007.stdout: "age": 0.0019176180000000001, 2024-09-18T08:15:29.305 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.0022027800000000001, 2024-09-18T08:15:29.306 INFO:teuthology.orchestra.run.smithi007.stdout: "continuous": false, 2024-09-18T08:15:29.306 INFO:teuthology.orchestra.run.smithi007.stdout: "type_data": { 2024-09-18T08:15:29.306 INFO:teuthology.orchestra.run.smithi007.stdout: "flag_point": "started", 2024-09-18T08:15:29.306 INFO:teuthology.orchestra.run.smithi007.stdout: "events": [ 2024-09-18T08:15:29.306 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:15:29.306 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "initiated", 2024-09-18T08:15:29.306 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:15:29.279309+0000", 2024-09-18T08:15:29.306 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-09-18T08:15:29.306 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:15:29.306 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:15:29.306 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "header_read", 2024-09-18T08:15:29.306 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:15:29.279309+0000", 2024-09-18T08:15:29.306 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-09-18T08:15:29.307 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:15:29.307 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:15:29.307 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "throttled", 2024-09-18T08:15:29.307 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:15:29.279310+0000", 2024-09-18T08:15:29.307 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 9.1100000000000004e-07 2024-09-18T08:15:29.307 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:15:29.307 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:15:29.307 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "all_read", 2024-09-18T08:15:29.307 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:15:29.279336+0000", 2024-09-18T08:15:29.307 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 2.6367e-05 2024-09-18T08:15:29.307 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:15:29.307 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:15:29.307 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "dispatched", 2024-09-18T08:15:29.307 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:15:29.279346+0000", 2024-09-18T08:15:29.308 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 9.6900000000000004e-06 2024-09-18T08:15:29.308 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:15:29.308 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:15:29.308 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "queued_for_pg", 2024-09-18T08:15:29.308 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:15:29.279352+0000", 2024-09-18T08:15:29.308 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 6.7850000000000003e-06 2024-09-18T08:15:29.308 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:15:29.308 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:15:29.308 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-09-18T08:15:29.308 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:15:29.279477+0000", 2024-09-18T08:15:29.308 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.00012455699999999999 2024-09-18T08:15:29.308 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:15:29.308 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:15:29.308 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "started", 2024-09-18T08:15:29.309 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:15:29.279495+0000", 2024-09-18T08:15:29.309 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.8249000000000001e-05 2024-09-18T08:15:29.309 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-09-18T08:15:29.309 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-09-18T08:15:29.309 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-09-18T08:15:29.309 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:15:29.309 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:15:29.309 INFO:teuthology.orchestra.run.smithi007.stdout: "description": "osd_op(client.15966.0:58924 10.a 10:589dae44:::benchmark_data_smithi007_148834_object58923:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536 in=65536b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e333)", 2024-09-18T08:15:29.309 INFO:teuthology.orchestra.run.smithi007.stdout: "initiated_at": "2024-09-18T08:15:29.269356+0000", 2024-09-18T08:15:29.309 INFO:teuthology.orchestra.run.smithi007.stdout: "age": 0.011870502999999999, 2024-09-18T08:15:29.309 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.012207605, 2024-09-18T08:15:29.309 INFO:teuthology.orchestra.run.smithi007.stdout: "continuous": false, 2024-09-18T08:15:29.309 INFO:teuthology.orchestra.run.smithi007.stdout: "type_data": { 2024-09-18T08:15:29.310 INFO:teuthology.orchestra.run.smithi007.stdout: "flag_point": "waiting for sub ops", 2024-09-18T08:15:29.310 INFO:teuthology.orchestra.run.smithi007.stdout: "client_info": { 2024-09-18T08:15:29.310 INFO:teuthology.orchestra.run.smithi007.stdout: "client": "client.15966", 2024-09-18T08:15:29.310 INFO:teuthology.orchestra.run.smithi007.stdout: "client_addr": "172.21.15.7:0/529784850", 2024-09-18T08:15:29.310 INFO:teuthology.orchestra.run.smithi007.stdout: "tid": 58924 2024-09-18T08:15:29.310 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:15:29.310 INFO:teuthology.orchestra.run.smithi007.stdout: "events": [ 2024-09-18T08:15:29.310 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:15:29.310 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "initiated", 2024-09-18T08:15:29.310 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:15:29.269356+0000", 2024-09-18T08:15:29.310 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-09-18T08:15:29.310 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:15:29.310 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:15:29.310 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "header_read", 2024-09-18T08:15:29.311 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:15:29.269356+0000", 2024-09-18T08:15:29.311 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-09-18T08:15:29.311 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:15:29.311 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:15:29.311 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "throttled", 2024-09-18T08:15:29.311 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:15:29.269358+0000", 2024-09-18T08:15:29.311 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.807e-06 2024-09-18T08:15:29.311 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:15:29.311 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:15:29.311 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "all_read", 2024-09-18T08:15:29.311 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:15:29.269388+0000", 2024-09-18T08:15:29.311 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 3.0494999999999999e-05 2024-09-18T08:15:29.311 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:15:29.311 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:15:29.312 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "dispatched", 2024-09-18T08:15:29.312 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:15:29.269402+0000", 2024-09-18T08:15:29.312 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.3816e-05 2024-09-18T08:15:29.312 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:15:29.312 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:15:29.312 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "queued_for_pg", 2024-09-18T08:15:29.312 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:15:29.269410+0000", 2024-09-18T08:15:29.312 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 8.5019999999999992e-06 2024-09-18T08:15:29.312 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:15:29.312 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:15:29.312 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-09-18T08:15:29.312 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:15:29.269476+0000", 2024-09-18T08:15:29.312 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 6.5834999999999996e-05 2024-09-18T08:15:29.313 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:15:29.313 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:15:29.313 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "started", 2024-09-18T08:15:29.313 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:15:29.269624+0000", 2024-09-18T08:15:29.313 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.00014783799999999999 2024-09-18T08:15:29.313 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:15:29.313 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:15:29.313 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "waiting for subops from 2,6", 2024-09-18T08:15:29.313 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:15:29.269773+0000", 2024-09-18T08:15:29.313 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.00014912399999999999 2024-09-18T08:15:29.313 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:15:29.313 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:15:29.313 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:15:29.313 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:15:29.272611+0000", 2024-09-18T08:15:29.313 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.0028375539999999999 2024-09-18T08:15:29.314 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:15:29.314 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:15:29.314 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "op_commit", 2024-09-18T08:15:29.314 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:15:29.275008+0000", 2024-09-18T08:15:29.314 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.0023975110000000002 2024-09-18T08:15:29.314 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-09-18T08:15:29.314 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-09-18T08:15:29.314 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-09-18T08:15:29.314 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-09-18T08:15:29.314 INFO:teuthology.orchestra.run.smithi007.stdout: ], 2024-09-18T08:15:29.314 INFO:teuthology.orchestra.run.smithi007.stdout: "num_ops": 6 2024-09-18T08:15:29.314 INFO:teuthology.orchestra.run.smithi007.stdout:} 2024-09-18T08:15:29.336 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5 2024-09-18T08:15:29.343 INFO:tasks.radosbench.radosbench.0.smithi007.stdout: 41 16 58994 58978 89.8837 70.1875 0.00410941 0.0111192 2024-09-18T08:15:29.362 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:15:29.363 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:15:29.364 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5' 2024-09-18T08:15:29.465 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2 2024-09-18T08:15:29.509 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:15:29.510 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:15:29.510 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2' 2024-09-18T08:15:29.611 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2 2024-09-18T08:15:29.648 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:15:29.649 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:15:29.649 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2' 2024-09-18T08:15:29.750 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5 2024-09-18T08:15:29.779 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:15:29.779 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:15:29.780 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5' 2024-09-18T08:15:29.880 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0 2024-09-18T08:15:29.905 INFO:journalctl@ceph.iscsi.iscsi.a.smithi050.stdout:Sep 18 08:15:29 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-iscsi-iscsi-a[57782]: debug there is no tcmu-runner data available 2024-09-18T08:15:29.922 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:15:29.923 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:15:29.923 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0' 2024-09-18T08:15:30.024 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4 2024-09-18T08:15:30.052 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:15:30.053 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:15:30.054 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4' 2024-09-18T08:15:30.154 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4 2024-09-18T08:15:30.178 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:15:30.179 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:15:30.179 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4' 2024-09-18T08:15:30.280 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4 2024-09-18T08:15:30.311 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:15:30.311 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:15:30.311 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4' 2024-09-18T08:15:30.343 INFO:tasks.radosbench.radosbench.0.smithi007.stdout: 42 16 60143 60127 89.4533 71.8125 0.0147993 0.0111735 2024-09-18T08:15:30.412 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3 2024-09-18T08:15:30.434 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:15:30.434 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:15:30.435 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3' 2024-09-18T08:15:30.535 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5 2024-09-18T08:15:30.562 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:15:30.562 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:15:30.563 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5' 2024-09-18T08:15:30.655 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:15:30 smithi050 ceph-mon[32371]: pgmap v629: 158 pgs: 158 active+clean; 3.5 GiB data, 13 GiB used, 523 GiB / 536 GiB avail; 853 B/s rd, 120 MiB/s wr, 1.91k op/s; 18 B/s, 0 objects/s recovering 2024-09-18T08:15:30.655 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:15:30 smithi050 ceph-mon[32371]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-18T08:15:30.664 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4 2024-09-18T08:15:30.693 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:15:30.693 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:15:30.694 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4' 2024-09-18T08:15:30.730 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:15:30 smithi007 ceph-mon[29459]: pgmap v629: 158 pgs: 158 active+clean; 3.5 GiB data, 13 GiB used, 523 GiB / 536 GiB avail; 853 B/s rd, 120 MiB/s wr, 1.91k op/s; 18 B/s, 0 objects/s recovering 2024-09-18T08:15:30.730 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:15:30 smithi007 ceph-mon[29459]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-18T08:15:30.730 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:15:30 smithi007 ceph-mon[39249]: pgmap v629: 158 pgs: 158 active+clean; 3.5 GiB data, 13 GiB used, 523 GiB / 536 GiB avail; 853 B/s rd, 120 MiB/s wr, 1.91k op/s; 18 B/s, 0 objects/s recovering 2024-09-18T08:15:30.730 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:15:30 smithi007 ceph-mon[39249]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-18T08:15:30.795 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3 2024-09-18T08:15:30.837 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:15:30.837 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:15:30.837 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3' 2024-09-18T08:15:30.938 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4 2024-09-18T08:15:30.966 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:15:30.967 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:15:30.967 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4' 2024-09-18T08:15:31.068 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3 2024-09-18T08:15:31.090 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:15:31.091 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:15:31.091 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3' 2024-09-18T08:15:31.192 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4 2024-09-18T08:15:31.220 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:15:31.220 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:15:31.221 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4' 2024-09-18T08:15:31.322 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0 2024-09-18T08:15:31.343 INFO:tasks.radosbench.radosbench.0.smithi007.stdout: 43 16 61201 61185 88.9105 66.125 0.00839383 0.0112415 2024-09-18T08:15:31.359 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:15:31.360 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:15:31.360 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0' 2024-09-18T08:15:31.461 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7 2024-09-18T08:15:31.483 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:15:31.484 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:15:31.485 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7' 2024-09-18T08:15:31.585 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0 2024-09-18T08:15:31.627 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:15:31.627 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:15:31.628 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0' 2024-09-18T08:15:31.729 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0 2024-09-18T08:15:31.769 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:15:31.769 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:15:31.769 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0' 2024-09-18T08:15:31.870 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2 2024-09-18T08:15:31.893 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:15:31.893 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:15:31.894 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2' 2024-09-18T08:15:31.994 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6 2024-09-18T08:15:32.017 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:15:32.018 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:15:32.018 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6' 2024-09-18T08:15:32.119 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7 2024-09-18T08:15:32.141 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:15:32.142 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:15:32.142 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7' 2024-09-18T08:15:32.243 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3 2024-09-18T08:15:32.266 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:15:32.267 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:15:32.267 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3' 2024-09-18T08:15:32.344 INFO:tasks.radosbench.radosbench.0.smithi007.stdout: 44 16 61810 61794 87.755 38.0625 0.0145726 0.0113895 2024-09-18T08:15:32.368 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2 2024-09-18T08:15:32.413 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:15:32.414 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:15:32.414 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2' 2024-09-18T08:15:32.515 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5 2024-09-18T08:15:32.540 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:15:32.540 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:15:32.541 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5' 2024-09-18T08:15:32.641 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1 2024-09-18T08:15:32.655 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:15:32 smithi050 ceph-mon[32371]: pgmap v630: 158 pgs: 158 active+clean; 3.5 GiB data, 13 GiB used, 523 GiB / 536 GiB avail; 731 B/s rd, 54 MiB/s wr, 869 op/s 2024-09-18T08:15:32.683 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:15:32.684 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:15:32.684 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1' 2024-09-18T08:15:32.730 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:15:32 smithi007 ceph-mon[29459]: pgmap v630: 158 pgs: 158 active+clean; 3.5 GiB data, 13 GiB used, 523 GiB / 536 GiB avail; 731 B/s rd, 54 MiB/s wr, 869 op/s 2024-09-18T08:15:32.730 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:15:32 smithi007 ceph-mon[39249]: pgmap v630: 158 pgs: 158 active+clean; 3.5 GiB data, 13 GiB used, 523 GiB / 536 GiB avail; 731 B/s rd, 54 MiB/s wr, 869 op/s 2024-09-18T08:15:32.785 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7 2024-09-18T08:15:32.793 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 6, 4] out_osds: [7, 5] dead_osds: [] live_osds: [0, 2, 3, 4, 5, 6, 7, 1] 2024-09-18T08:15:32.793 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-18T08:15:32.793 INFO:tasks.thrashosds.thrasher:Adding osd 5 2024-09-18T08:15:32.794 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid 779349ca-7593-11ef-bceb-c7b262605968 -- ceph osd in 5 2024-09-18T08:15:32.812 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:15:32.813 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:15:32.813 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7' 2024-09-18T08:15:32.914 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1 2024-09-18T08:15:32.954 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:15:32.955 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:15:32.955 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1' 2024-09-18T08:15:33.056 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7 2024-09-18T08:15:33.084 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:15:33.084 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:15:33.085 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7' 2024-09-18T08:15:33.185 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0 2024-09-18T08:15:33.208 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:15:33.208 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:15:33.209 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0' 2024-09-18T08:15:33.310 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7 2024-09-18T08:15:33.339 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:15:33.339 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:15:33.340 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7' 2024-09-18T08:15:33.344 INFO:tasks.radosbench.radosbench.0.smithi007.stdout: 45 16 62903 62887 87.3228 68.3125 0.0178771 0.0114467 2024-09-18T08:15:33.441 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2 2024-09-18T08:15:33.479 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:15:33.480 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:15:33.480 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2' 2024-09-18T08:15:33.581 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4 2024-09-18T08:15:33.610 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:15:33.610 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:15:33.611 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4' 2024-09-18T08:15:33.711 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6 2024-09-18T08:15:33.739 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:15:33.739 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:15:33.740 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6' 2024-09-18T08:15:33.841 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1 2024-09-18T08:15:33.862 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:15:33.863 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:15:33.863 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1' 2024-09-18T08:15:33.964 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0 2024-09-18T08:15:34.001 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:15:34.002 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:15:34.002 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0' 2024-09-18T08:15:34.103 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6 2024-09-18T08:15:34.133 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:15:34.134 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:15:34.134 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6' 2024-09-18T08:15:34.235 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5 2024-09-18T08:15:34.257 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:15:34.258 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:15:34.258 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5' 2024-09-18T08:15:34.344 INFO:tasks.radosbench.radosbench.0.smithi007.stdout: 46 16 64152 64136 87.1214 78.0625 0.00429337 0.0114723 2024-09-18T08:15:34.358 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5 2024-09-18T08:15:34.386 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:15:34.387 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:15:34.387 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5' 2024-09-18T08:15:34.488 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3 2024-09-18T08:15:34.510 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:15:34.511 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:15:34.511 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3' 2024-09-18T08:15:34.612 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1 2024-09-18T08:15:34.652 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:15:34.653 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:15:34.653 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1' 2024-09-18T08:15:34.655 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:15:34 smithi050 ceph-mon[32371]: pgmap v631: 158 pgs: 158 active+clean; 3.6 GiB data, 13 GiB used, 523 GiB / 536 GiB avail; 1.1 KiB/s rd, 55 MiB/s wr, 887 op/s 2024-09-18T08:15:34.655 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:15:34 smithi050 ceph-mon[32371]: from='mgr.24470 172.21.15.7:0/2594390261' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-18T08:15:34.730 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:15:34 smithi007 ceph-mon[29459]: pgmap v631: 158 pgs: 158 active+clean; 3.6 GiB data, 13 GiB used, 523 GiB / 536 GiB avail; 1.1 KiB/s rd, 55 MiB/s wr, 887 op/s 2024-09-18T08:15:34.730 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:15:34 smithi007 ceph-mon[29459]: from='mgr.24470 172.21.15.7:0/2594390261' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-18T08:15:34.731 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:15:34 smithi007 ceph-mon[39249]: pgmap v631: 158 pgs: 158 active+clean; 3.6 GiB data, 13 GiB used, 523 GiB / 536 GiB avail; 1.1 KiB/s rd, 55 MiB/s wr, 887 op/s 2024-09-18T08:15:34.731 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:15:34 smithi007 ceph-mon[39249]: from='mgr.24470 172.21.15.7:0/2594390261' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-18T08:15:34.754 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0 2024-09-18T08:15:34.787 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:15:34.788 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:15:34.788 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0' 2024-09-18T08:15:34.889 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1 2024-09-18T08:15:34.928 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:15:34.928 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:15:34.929 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1' 2024-09-18T08:15:35.029 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0 2024-09-18T08:15:35.071 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:15:35.072 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:15:35.072 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0' 2024-09-18T08:15:35.173 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7 2024-09-18T08:15:35.195 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:15:35.196 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:15:35.196 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7' 2024-09-18T08:15:35.297 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0 2024-09-18T08:15:35.332 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:15:35.332 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:15:35.333 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0' 2024-09-18T08:15:35.344 INFO:tasks.radosbench.radosbench.0.smithi007.stdout: 47 16 65194 65178 86.6533 65.125 0.016079 0.0115339 2024-09-18T08:15:35.433 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7 2024-09-18T08:15:35.456 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:15:35.457 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:15:35.457 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7' 2024-09-18T08:15:35.558 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6 2024-09-18T08:15:35.583 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:15:35.584 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:15:35.585 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6' 2024-09-18T08:15:35.685 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4 2024-09-18T08:15:35.715 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:15:35.715 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:15:35.716 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4' 2024-09-18T08:15:35.817 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7 2024-09-18T08:15:35.844 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:15:35.845 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:15:35.845 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7' 2024-09-18T08:15:35.946 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3 2024-09-18T08:15:35.983 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:15:35.983 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:15:35.984 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3' 2024-09-18T08:15:36.085 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3 2024-09-18T08:15:36.127 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:15:36.127 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:15:36.128 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3' 2024-09-18T08:15:36.229 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7 2024-09-18T08:15:36.256 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:15:36.257 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:15:36.257 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7' 2024-09-18T08:15:36.344 INFO:tasks.radosbench.radosbench.0.smithi007.stdout: 48 16 66319 66303 86.3128 70.3125 0.0103222 0.0115806 2024-09-18T08:15:36.358 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7 2024-09-18T08:15:36.384 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:15:36.385 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:15:36.385 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7' 2024-09-18T08:15:36.486 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6 2024-09-18T08:15:36.513 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:15:36.514 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:15:36.514 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6' 2024-09-18T08:15:36.615 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0 2024-09-18T08:15:36.655 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:15:36 smithi050 ceph-mon[32371]: pgmap v632: 158 pgs: 158 active+clean; 3.9 GiB data, 14 GiB used, 522 GiB / 536 GiB avail; 921 B/s rd, 70 MiB/s wr, 1.12k op/s 2024-09-18T08:15:36.659 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:15:36.660 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:15:36.660 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0' 2024-09-18T08:15:36.699 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:15:36 smithi007 ceph-mon[29459]: pgmap v632: 158 pgs: 158 active+clean; 3.9 GiB data, 14 GiB used, 522 GiB / 536 GiB avail; 921 B/s rd, 70 MiB/s wr, 1.12k op/s 2024-09-18T08:15:36.699 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:15:36 smithi007 ceph-mon[39249]: pgmap v632: 158 pgs: 158 active+clean; 3.9 GiB data, 14 GiB used, 522 GiB / 536 GiB avail; 921 B/s rd, 70 MiB/s wr, 1.12k op/s 2024-09-18T08:15:36.761 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0 2024-09-18T08:15:36.803 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:15:36.804 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:15:36.804 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0' 2024-09-18T08:15:36.905 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2 2024-09-18T08:15:36.955 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:15:36.956 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:15:36.957 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2' 2024-09-18T08:15:36.980 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Sep 18 08:15:36 smithi007 ceph-779349ca-7593-11ef-bceb-c7b262605968-mgr-y[29688]: ::ffff:172.21.15.50 - - [18/Sep/2024:08:15:36] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-09-18T08:15:37.057 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5 2024-09-18T08:15:37.087 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:15:37.088 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:15:37.089 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5' 2024-09-18T08:15:37.190 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3 2024-09-18T08:15:37.236 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:15:37.236 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:15:37.237 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3' 2024-09-18T08:15:37.338 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7 2024-09-18T08:15:37.344 INFO:tasks.radosbench.radosbench.0.smithi007.stdout: 49 16 67577 67561 86.1558 78.625 0.0159654 0.0116013 2024-09-18T08:15:37.366 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:15:37.366 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:15:37.367 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7' 2024-09-18T08:15:37.468 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7 2024-09-18T08:15:37.497 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:15:37.497 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:15:37.498 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7' 2024-09-18T08:15:37.599 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0 2024-09-18T08:15:37.637 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:15:37.638 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:15:37.638 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0' 2024-09-18T08:15:37.739 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0 2024-09-18T08:15:37.783 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:15:37.784 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:15:37.784 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0' 2024-09-18T08:15:37.885 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3 2024-09-18T08:15:37.925 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:15:37.927 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:15:37.927 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3' 2024-09-18T08:15:37.980 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/779349ca-7593-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T08:15:38.028 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1 2024-09-18T08:15:38.042 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid 779349ca-7593-11ef-bceb-c7b262605968 -- ceph daemon osd.0 dump_blocked_ops 2024-09-18T08:15:38.060 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/779349ca-7593-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T08:15:38.091 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:15:38.092 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:15:38.092 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1' 2024-09-18T08:15:38.193 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7 2024-09-18T08:15:38.219 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:15:38.219 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:15:38.220 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7' 2024-09-18T08:15:38.321 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3 2024-09-18T08:15:38.346 INFO:tasks.radosbench.radosbench.0.smithi007.stdout: 50 16 68808 68792 85.9674 76.9375 0.0114653 0.0116258 2024-09-18T08:15:38.377 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:15:38.377 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:15:38.377 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3' 2024-09-18T08:15:38.478 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1 2024-09-18T08:15:38.520 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:15:38.520 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:15:38.521 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1' 2024-09-18T08:15:38.621 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0 2024-09-18T08:15:38.665 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:15:38 smithi007 ceph-mon[29459]: purged_snaps scrub starts 2024-09-18T08:15:38.665 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:15:38 smithi007 ceph-mon[29459]: purged_snaps scrub ok 2024-09-18T08:15:38.665 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:15:38 smithi007 ceph-mon[29459]: pgmap v633: 158 pgs: 158 active+clean; 3.9 GiB data, 14 GiB used, 522 GiB / 536 GiB avail; 921 B/s rd, 70 MiB/s wr, 1.12k op/s 2024-09-18T08:15:38.665 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:15:38 smithi007 ceph-mon[29459]: osdmap e334: 8 total, 8 up, 6 in 2024-09-18T08:15:38.665 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:15:38 smithi007 ceph-mon[39249]: purged_snaps scrub starts 2024-09-18T08:15:38.666 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:15:38 smithi007 ceph-mon[39249]: purged_snaps scrub ok 2024-09-18T08:15:38.666 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:15:38 smithi007 ceph-mon[39249]: pgmap v633: 158 pgs: 158 active+clean; 3.9 GiB data, 14 GiB used, 522 GiB / 536 GiB avail; 921 B/s rd, 70 MiB/s wr, 1.12k op/s 2024-09-18T08:15:38.666 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:15:38 smithi007 ceph-mon[39249]: osdmap e334: 8 total, 8 up, 6 in 2024-09-18T08:15:38.668 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:15:38.669 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:15:38.669 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0' 2024-09-18T08:15:38.770 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0 2024-09-18T08:15:38.810 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:15:38.810 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:15:38.811 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0' 2024-09-18T08:15:38.905 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:15:38 smithi050 ceph-mon[32371]: purged_snaps scrub starts 2024-09-18T08:15:38.905 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:15:38 smithi050 ceph-mon[32371]: purged_snaps scrub ok 2024-09-18T08:15:38.905 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:15:38 smithi050 ceph-mon[32371]: pgmap v633: 158 pgs: 158 active+clean; 3.9 GiB data, 14 GiB used, 522 GiB / 536 GiB avail; 921 B/s rd, 70 MiB/s wr, 1.12k op/s 2024-09-18T08:15:38.906 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:15:38 smithi050 ceph-mon[32371]: osdmap e334: 8 total, 8 up, 6 in 2024-09-18T08:15:38.912 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3 2024-09-18T08:15:38.964 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:15:38.964 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:15:38.965 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3' 2024-09-18T08:15:39.065 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7 2024-09-18T08:15:39.089 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:15:39.090 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:15:39.090 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7' 2024-09-18T08:15:39.191 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7 2024-09-18T08:15:39.209 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/779349ca-7593-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T08:15:39.217 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:15:39.218 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:15:39.219 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7' 2024-09-18T08:15:39.321 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0 2024-09-18T08:15:39.346 INFO:tasks.radosbench.radosbench.0.smithi007.stdout: 51 16 69875 69859 85.5893 66.6875 0.0126632 0.0116778 2024-09-18T08:15:39.366 INFO:teuthology.orchestra.run.smithi007.stderr:noscrub is unset 2024-09-18T08:15:39.367 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:15:39.368 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:15:39.369 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0' 2024-09-18T08:15:39.469 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0 2024-09-18T08:15:39.510 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:15:39.511 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:15:39.512 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0' 2024-09-18T08:15:39.607 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:15:39 smithi007 ceph-mon[39249]: from='client.? 172.21.15.7:0/765945659' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-18T08:15:39.607 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:15:39 smithi007 ceph-mon[39249]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-18T08:15:39.607 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:15:39 smithi007 ceph-mon[39249]: from='client.? 172.21.15.7:0/765945659' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-18T08:15:39.607 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:15:39 smithi007 ceph-mon[39249]: osdmap e335: 8 total, 8 up, 6 in 2024-09-18T08:15:39.608 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:15:39 smithi007 ceph-mon[29459]: from='client.? 172.21.15.7:0/765945659' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-18T08:15:39.608 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:15:39 smithi007 ceph-mon[29459]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-18T08:15:39.608 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:15:39 smithi007 ceph-mon[29459]: from='client.? 172.21.15.7:0/765945659' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-18T08:15:39.608 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:15:39 smithi007 ceph-mon[29459]: osdmap e335: 8 total, 8 up, 6 in 2024-09-18T08:15:39.613 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3 2024-09-18T08:15:39.653 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:15:39.653 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:15:39.654 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3' 2024-09-18T08:15:39.755 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5 2024-09-18T08:15:39.782 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:15:39.783 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:15:39.783 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5' 2024-09-18T08:15:39.839 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid 779349ca-7593-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-18T08:15:39.878 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:15:39 smithi007 ceph-779349ca-7593-11ef-bceb-c7b262605968-mon-a[29455]: 2024-09-18T08:15:39.613+0000 7fcd2b595640 -1 mon.a@0(leader).osd e335 definitely_dead 0 2024-09-18T08:15:39.883 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2 2024-09-18T08:15:39.905 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:15:39 smithi050 ceph-mon[32371]: from='client.? 172.21.15.7:0/765945659' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-18T08:15:39.905 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:15:39 smithi050 ceph-mon[32371]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-18T08:15:39.905 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:15:39 smithi050 ceph-mon[32371]: from='client.? 172.21.15.7:0/765945659' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-18T08:15:39.905 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:15:39 smithi050 ceph-mon[32371]: osdmap e335: 8 total, 8 up, 6 in 2024-09-18T08:15:39.906 INFO:journalctl@ceph.iscsi.iscsi.a.smithi050.stdout:Sep 18 08:15:39 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-iscsi-iscsi-a[57782]: debug there is no tcmu-runner data available 2024-09-18T08:15:39.929 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:15:39.929 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:15:39.930 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2' 2024-09-18T08:15:40.030 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4 2024-09-18T08:15:40.056 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:15:40.056 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:15:40.057 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4' 2024-09-18T08:15:40.158 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7 2024-09-18T08:15:40.184 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:15:40.185 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:15:40.185 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7' 2024-09-18T08:15:40.286 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6 2024-09-18T08:15:40.313 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:15:40.314 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:15:40.314 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6' 2024-09-18T08:15:40.347 INFO:tasks.radosbench.radosbench.0.smithi007.stdout: 52 16 71017 71001 85.315 71.375 0.012756 0.0117161 2024-09-18T08:15:40.415 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0 2024-09-18T08:15:40.437 INFO:teuthology.orchestra.run.smithi007.stderr:marked in osd.5. 2024-09-18T08:15:40.474 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:15:40.475 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:15:40.475 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0' 2024-09-18T08:15:40.480 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:15:40 smithi007 ceph-mon[29459]: pgmap v635: 158 pgs: 158 active+clean; 4.2 GiB data, 15 GiB used, 521 GiB / 536 GiB avail; 921 B/s rd, 68 MiB/s wr, 1.09k op/s 2024-09-18T08:15:40.480 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:15:40 smithi007 ceph-mon[29459]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-18T08:15:40.480 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:15:40 smithi007 ceph-mon[29459]: from='client.? 172.21.15.7:0/3145453097' entity='client.admin' cmd={"prefix": "osd in", "ids": ["5"]} : dispatch 2024-09-18T08:15:40.480 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:15:40 smithi007 ceph-mon[29459]: from='client.? ' entity='client.admin' cmd={"prefix": "osd in", "ids": ["5"]} : dispatch 2024-09-18T08:15:40.480 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:15:40 smithi007 ceph-mon[29459]: 3.5 scrub starts 2024-09-18T08:15:40.480 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:15:40 smithi007 ceph-mon[29459]: 3.5 scrub ok 2024-09-18T08:15:40.481 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:15:40 smithi007 ceph-mon[29459]: 4.2 scrub starts 2024-09-18T08:15:40.481 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:15:40 smithi007 ceph-mon[29459]: 4.1e scrub starts 2024-09-18T08:15:40.481 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:15:40 smithi007 ceph-mon[29459]: 4.2 scrub ok 2024-09-18T08:15:40.481 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:15:40 smithi007 ceph-mon[29459]: 4.1e scrub ok 2024-09-18T08:15:40.481 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:15:40 smithi007 ceph-mon[39249]: pgmap v635: 158 pgs: 158 active+clean; 4.2 GiB data, 15 GiB used, 521 GiB / 536 GiB avail; 921 B/s rd, 68 MiB/s wr, 1.09k op/s 2024-09-18T08:15:40.481 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:15:40 smithi007 ceph-mon[39249]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-18T08:15:40.481 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:15:40 smithi007 ceph-mon[39249]: from='client.? 172.21.15.7:0/3145453097' entity='client.admin' cmd={"prefix": "osd in", "ids": ["5"]} : dispatch 2024-09-18T08:15:40.481 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:15:40 smithi007 ceph-mon[39249]: from='client.? ' entity='client.admin' cmd={"prefix": "osd in", "ids": ["5"]} : dispatch 2024-09-18T08:15:40.482 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:15:40 smithi007 ceph-mon[39249]: 3.5 scrub starts 2024-09-18T08:15:40.482 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:15:40 smithi007 ceph-mon[39249]: 3.5 scrub ok 2024-09-18T08:15:40.482 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:15:40 smithi007 ceph-mon[39249]: 4.2 scrub starts 2024-09-18T08:15:40.482 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:15:40 smithi007 ceph-mon[39249]: 4.1e scrub starts 2024-09-18T08:15:40.482 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:15:40 smithi007 ceph-mon[39249]: 4.2 scrub ok 2024-09-18T08:15:40.482 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:15:40 smithi007 ceph-mon[39249]: 4.1e scrub ok 2024-09-18T08:15:40.527 INFO:teuthology.orchestra.run.smithi007.stdout:{ 2024-09-18T08:15:40.527 INFO:teuthology.orchestra.run.smithi007.stdout: "ops": [], 2024-09-18T08:15:40.527 INFO:teuthology.orchestra.run.smithi007.stdout: "complaint_time": 30, 2024-09-18T08:15:40.527 INFO:teuthology.orchestra.run.smithi007.stdout: "num_blocked_ops": 0 2024-09-18T08:15:40.527 INFO:teuthology.orchestra.run.smithi007.stdout:} 2024-09-18T08:15:40.576 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6 2024-09-18T08:15:40.598 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:15:40.599 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:15:40.599 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6' 2024-09-18T08:15:40.700 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4 2024-09-18T08:15:40.722 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:15:40.723 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:15:40.723 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4' 2024-09-18T08:15:40.824 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2 2024-09-18T08:15:40.847 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:15:40.847 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:15:40.848 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2' 2024-09-18T08:15:40.905 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:15:40 smithi050 ceph-mon[32371]: pgmap v635: 158 pgs: 158 active+clean; 4.2 GiB data, 15 GiB used, 521 GiB / 536 GiB avail; 921 B/s rd, 68 MiB/s wr, 1.09k op/s 2024-09-18T08:15:40.905 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:15:40 smithi050 ceph-mon[32371]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-18T08:15:40.905 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:15:40 smithi050 ceph-mon[32371]: from='client.? 172.21.15.7:0/3145453097' entity='client.admin' cmd={"prefix": "osd in", "ids": ["5"]} : dispatch 2024-09-18T08:15:40.906 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:15:40 smithi050 ceph-mon[32371]: from='client.? ' entity='client.admin' cmd={"prefix": "osd in", "ids": ["5"]} : dispatch 2024-09-18T08:15:40.906 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:15:40 smithi050 ceph-mon[32371]: 3.5 scrub starts 2024-09-18T08:15:40.906 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:15:40 smithi050 ceph-mon[32371]: 3.5 scrub ok 2024-09-18T08:15:40.906 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:15:40 smithi050 ceph-mon[32371]: 4.2 scrub starts 2024-09-18T08:15:40.906 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:15:40 smithi050 ceph-mon[32371]: 4.1e scrub starts 2024-09-18T08:15:40.906 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:15:40 smithi050 ceph-mon[32371]: 4.2 scrub ok 2024-09-18T08:15:40.906 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:15:40 smithi050 ceph-mon[32371]: 4.1e scrub ok 2024-09-18T08:15:40.948 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5 2024-09-18T08:15:40.972 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:15:40.972 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:15:40.973 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5' 2024-09-18T08:15:41.074 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2 2024-09-18T08:15:41.096 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:15:41.097 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:15:41.097 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2' 2024-09-18T08:15:41.198 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2 2024-09-18T08:15:41.220 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:15:41.220 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:15:41.221 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2' 2024-09-18T08:15:41.322 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4 2024-09-18T08:15:41.346 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:15:41.347 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:15:41.347 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4' 2024-09-18T08:15:41.347 INFO:tasks.radosbench.radosbench.0.smithi007.stdout: 53 16 71136 71120 83.8458 7.4375 0.00275652 0.0117233 2024-09-18T08:15:41.436 INFO:tasks.thrashosds.thrasher:Added osd 5 2024-09-18T08:15:41.448 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6 2024-09-18T08:15:41.484 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:15:41.485 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:15:41.485 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6' 2024-09-18T08:15:41.504 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/779349ca-7593-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T08:15:41.586 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2 2024-09-18T08:15:41.609 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:15:41.610 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:15:41.610 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2' 2024-09-18T08:15:41.711 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2 2024-09-18T08:15:41.730 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:15:41 smithi007 ceph-mon[29459]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd in", "ids": ["5"]}]': finished 2024-09-18T08:15:41.730 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:15:41 smithi007 ceph-mon[29459]: osdmap e336: 8 total, 8 up, 7 in 2024-09-18T08:15:41.730 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:15:41 smithi007 ceph-mon[29459]: osdmap e337: 8 total, 8 up, 7 in 2024-09-18T08:15:41.730 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:15:41 smithi007 ceph-mon[39249]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd in", "ids": ["5"]}]': finished 2024-09-18T08:15:41.730 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:15:41 smithi007 ceph-mon[39249]: osdmap e336: 8 total, 8 up, 7 in 2024-09-18T08:15:41.730 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:15:41 smithi007 ceph-mon[39249]: osdmap e337: 8 total, 8 up, 7 in 2024-09-18T08:15:41.734 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:15:41.734 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:15:41.735 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2' 2024-09-18T08:15:41.835 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5 2024-09-18T08:15:41.860 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:15:41.861 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:15:41.862 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5' 2024-09-18T08:15:41.905 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:15:41 smithi050 ceph-mon[32371]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd in", "ids": ["5"]}]': finished 2024-09-18T08:15:41.905 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:15:41 smithi050 ceph-mon[32371]: osdmap e336: 8 total, 8 up, 7 in 2024-09-18T08:15:41.905 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:15:41 smithi050 ceph-mon[32371]: osdmap e337: 8 total, 8 up, 7 in 2024-09-18T08:15:41.948 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid 779349ca-7593-11ef-bceb-c7b262605968 -- ceph daemon osd.0 dump_historic_ops 2024-09-18T08:15:41.963 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3 2024-09-18T08:15:42.014 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:15:42.015 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:15:42.016 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3' 2024-09-18T08:15:42.117 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2 2024-09-18T08:15:42.139 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:15:42.140 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:15:42.141 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2' 2024-09-18T08:15:42.241 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6 2024-09-18T08:15:42.269 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:15:42.269 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:15:42.270 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6' 2024-09-18T08:15:42.295 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid 779349ca-7593-11ef-bceb-c7b262605968 -- ceph osd unset nodeep-scrub 2024-09-18T08:15:42.347 INFO:tasks.radosbench.radosbench.0.smithi007.stdout: 54 16 71136 71120 82.2934 0 - 0.0117233 2024-09-18T08:15:42.371 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5 2024-09-18T08:15:42.397 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:15:42.398 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:15:42.398 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5' 2024-09-18T08:15:42.480 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:15:42 smithi007 ceph-mon[29459]: 5.1a scrub starts 2024-09-18T08:15:42.480 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:15:42 smithi007 ceph-mon[29459]: 5.1a scrub ok 2024-09-18T08:15:42.480 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:15:42 smithi007 ceph-mon[29459]: pgmap v638: 158 pgs: 158 active+clean; 4.2 GiB data, 15 GiB used, 521 GiB / 536 GiB avail; 853 B/s rd, 60 MiB/s wr, 959 op/s 2024-09-18T08:15:42.480 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:15:42 smithi007 ceph-mon[29459]: 6.8 scrub starts 2024-09-18T08:15:42.480 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:15:42 smithi007 ceph-mon[29459]: 6.8 scrub ok 2024-09-18T08:15:42.481 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:15:42 smithi007 ceph-mon[29459]: 1.4 scrub starts 2024-09-18T08:15:42.481 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:15:42 smithi007 ceph-mon[29459]: 1.4 scrub ok 2024-09-18T08:15:42.481 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:15:42 smithi007 ceph-mon[29459]: osdmap e338: 8 total, 8 up, 7 in 2024-09-18T08:15:42.481 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:15:42 smithi007 ceph-mon[39249]: 5.1a scrub starts 2024-09-18T08:15:42.481 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:15:42 smithi007 ceph-mon[39249]: 5.1a scrub ok 2024-09-18T08:15:42.481 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:15:42 smithi007 ceph-mon[39249]: pgmap v638: 158 pgs: 158 active+clean; 4.2 GiB data, 15 GiB used, 521 GiB / 536 GiB avail; 853 B/s rd, 60 MiB/s wr, 959 op/s 2024-09-18T08:15:42.481 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:15:42 smithi007 ceph-mon[39249]: 6.8 scrub starts 2024-09-18T08:15:42.481 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:15:42 smithi007 ceph-mon[39249]: 6.8 scrub ok 2024-09-18T08:15:42.481 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:15:42 smithi007 ceph-mon[39249]: 1.4 scrub starts 2024-09-18T08:15:42.481 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:15:42 smithi007 ceph-mon[39249]: 1.4 scrub ok 2024-09-18T08:15:42.481 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:15:42 smithi007 ceph-mon[39249]: osdmap e338: 8 total, 8 up, 7 in 2024-09-18T08:15:42.493 INFO:teuthology.orchestra.run.smithi007.stdout:osd.0: {} 2024-09-18T08:15:42.493 INFO:teuthology.orchestra.run.smithi007.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-18T08:15:42.499 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3 2024-09-18T08:15:42.513 INFO:teuthology.orchestra.run.smithi007.stdout:osd.1: {} 2024-09-18T08:15:42.513 INFO:teuthology.orchestra.run.smithi007.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-18T08:15:42.514 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/779349ca-7593-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T08:15:42.526 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:15:42.527 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:15:42.527 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3' 2024-09-18T08:15:42.533 INFO:teuthology.orchestra.run.smithi007.stdout:osd.2: {} 2024-09-18T08:15:42.534 INFO:teuthology.orchestra.run.smithi007.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-18T08:15:42.557 INFO:teuthology.orchestra.run.smithi007.stdout:osd.3: {} 2024-09-18T08:15:42.557 INFO:teuthology.orchestra.run.smithi007.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-18T08:15:42.593 INFO:teuthology.orchestra.run.smithi007.stdout:osd.4: {} 2024-09-18T08:15:42.593 INFO:teuthology.orchestra.run.smithi007.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-18T08:15:42.628 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2 2024-09-18T08:15:42.629 INFO:teuthology.orchestra.run.smithi007.stdout:osd.5: {} 2024-09-18T08:15:42.629 INFO:teuthology.orchestra.run.smithi007.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-18T08:15:42.665 INFO:teuthology.orchestra.run.smithi007.stdout:osd.6: {} 2024-09-18T08:15:42.665 INFO:teuthology.orchestra.run.smithi007.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-18T08:15:42.697 INFO:teuthology.orchestra.run.smithi007.stdout:osd.7: {} 2024-09-18T08:15:42.698 INFO:teuthology.orchestra.run.smithi007.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-18T08:15:42.699 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:15:42.699 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:15:42.699 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2' 2024-09-18T08:15:42.800 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1 2024-09-18T08:15:42.842 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:15:42.842 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:15:42.842 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1' 2024-09-18T08:15:42.905 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:15:42 smithi050 ceph-mon[32371]: 5.1a scrub starts 2024-09-18T08:15:42.905 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:15:42 smithi050 ceph-mon[32371]: 5.1a scrub ok 2024-09-18T08:15:42.905 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:15:42 smithi050 ceph-mon[32371]: pgmap v638: 158 pgs: 158 active+clean; 4.2 GiB data, 15 GiB used, 521 GiB / 536 GiB avail; 853 B/s rd, 60 MiB/s wr, 959 op/s 2024-09-18T08:15:42.906 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:15:42 smithi050 ceph-mon[32371]: 6.8 scrub starts 2024-09-18T08:15:42.906 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:15:42 smithi050 ceph-mon[32371]: 6.8 scrub ok 2024-09-18T08:15:42.906 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:15:42 smithi050 ceph-mon[32371]: 1.4 scrub starts 2024-09-18T08:15:42.906 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:15:42 smithi050 ceph-mon[32371]: 1.4 scrub ok 2024-09-18T08:15:42.906 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:15:42 smithi050 ceph-mon[32371]: osdmap e338: 8 total, 8 up, 7 in 2024-09-18T08:15:42.944 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3 2024-09-18T08:15:42.986 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:15:42.986 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:15:42.987 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3' 2024-09-18T08:15:43.087 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6 2024-09-18T08:15:43.117 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:15:43.117 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:15:43.117 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6' 2024-09-18T08:15:43.218 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2 2024-09-18T08:15:43.258 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:15:43.259 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:15:43.259 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2' 2024-09-18T08:15:43.347 INFO:tasks.radosbench.radosbench.0.smithi007.stdout: 55 16 71971 71955 81.746 26.0938 0.0151675 0.0122276 2024-09-18T08:15:43.360 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4 2024-09-18T08:15:43.390 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:15:43.391 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:15:43.391 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4' 2024-09-18T08:15:43.469 INFO:journalctl@ceph.rgw.foo.a.smithi007.stdout:Sep 18 08:15:43 smithi007 ceph-779349ca-7593-11ef-bceb-c7b262605968-rgw-foo-a[63271]: 2024-09-18T08:15:43.079+0000 7fdec1ad7640 -1 rgw watcher librados: RGWWatcher::handle_error cookie 94700888683008 err (110) Connection timed out 2024-09-18T08:15:43.469 INFO:journalctl@ceph.rgw.foo.a.smithi007.stdout:Sep 18 08:15:43 smithi007 ceph-779349ca-7593-11ef-bceb-c7b262605968-rgw-foo-a[63271]: 2024-09-18T08:15:43.080+0000 7fdec12d6640 -1 rgw watcher librados: RGWWatcher::handle_error cookie 94700888684160 err (110) Connection timed out 2024-09-18T08:15:43.469 INFO:journalctl@ceph.rgw.foo.a.smithi007.stdout:Sep 18 08:15:43 smithi007 ceph-779349ca-7593-11ef-bceb-c7b262605968-rgw-foo-a[63271]: 2024-09-18T08:15:43.080+0000 7fdec1ad7640 -1 rgw watcher librados: RGWWatcher::handle_error cookie 94700888685312 err (110) Connection timed out 2024-09-18T08:15:43.491 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4 2024-09-18T08:15:43.520 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:15:43.520 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:15:43.521 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4' 2024-09-18T08:15:43.622 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7 2024-09-18T08:15:43.652 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:15:43.653 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:15:43.653 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7' 2024-09-18T08:15:43.730 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:15:43 smithi007 ceph-mon[29459]: 1.a scrub starts 2024-09-18T08:15:43.730 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:15:43 smithi007 ceph-mon[29459]: 1.a scrub ok 2024-09-18T08:15:43.730 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:15:43 smithi007 ceph-mon[29459]: 10.7 starting backfill to osd.5 from (0'0,0'0] MAX to 335'4440 2024-09-18T08:15:43.730 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:15:43 smithi007 ceph-mon[29459]: 10.6 starting backfill to osd.5 from (0'0,0'0] MAX to 335'4366 2024-09-18T08:15:43.730 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:15:43 smithi007 ceph-mon[29459]: 10.a starting backfill to osd.5 from (0'0,0'0] MAX to 335'4416 2024-09-18T08:15:43.731 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:15:43 smithi007 ceph-mon[29459]: 10.9 starting backfill to osd.5 from (0'0,0'0] MAX to 335'4495 2024-09-18T08:15:43.731 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:15:43 smithi007 ceph-mon[29459]: 10.3 starting backfill to osd.5 from (0'0,0'0] MAX to 335'4396 2024-09-18T08:15:43.731 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:15:43 smithi007 ceph-mon[29459]: 10.8 starting backfill to osd.5 from (0'0,0'0] MAX to 335'4404 2024-09-18T08:15:43.731 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:15:43 smithi007 ceph-mon[29459]: from='client.? 172.21.15.7:0/2698720237' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-18T08:15:43.731 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:15:43 smithi007 ceph-mon[29459]: 4.a scrub starts 2024-09-18T08:15:43.731 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:15:43 smithi007 ceph-mon[29459]: 4.a scrub ok 2024-09-18T08:15:43.731 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:15:43 smithi007 ceph-mon[39249]: 1.a scrub starts 2024-09-18T08:15:43.731 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:15:43 smithi007 ceph-mon[39249]: 1.a scrub ok 2024-09-18T08:15:43.731 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:15:43 smithi007 ceph-mon[39249]: 10.7 starting backfill to osd.5 from (0'0,0'0] MAX to 335'4440 2024-09-18T08:15:43.732 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:15:43 smithi007 ceph-mon[39249]: 10.6 starting backfill to osd.5 from (0'0,0'0] MAX to 335'4366 2024-09-18T08:15:43.732 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:15:43 smithi007 ceph-mon[39249]: 10.a starting backfill to osd.5 from (0'0,0'0] MAX to 335'4416 2024-09-18T08:15:43.732 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:15:43 smithi007 ceph-mon[39249]: 10.9 starting backfill to osd.5 from (0'0,0'0] MAX to 335'4495 2024-09-18T08:15:43.732 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:15:43 smithi007 ceph-mon[39249]: 10.3 starting backfill to osd.5 from (0'0,0'0] MAX to 335'4396 2024-09-18T08:15:43.732 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:15:43 smithi007 ceph-mon[39249]: 10.8 starting backfill to osd.5 from (0'0,0'0] MAX to 335'4404 2024-09-18T08:15:43.732 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:15:43 smithi007 ceph-mon[39249]: from='client.? 172.21.15.7:0/2698720237' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-18T08:15:43.732 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:15:43 smithi007 ceph-mon[39249]: 4.a scrub starts 2024-09-18T08:15:43.732 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:15:43 smithi007 ceph-mon[39249]: 4.a scrub ok 2024-09-18T08:15:43.754 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0 2024-09-18T08:15:43.799 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:15:43.799 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:15:43.800 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0' 2024-09-18T08:15:43.869 INFO:teuthology.orchestra.run.smithi007.stdout:ERROR: (22) Invalid argument 2024-09-18T08:15:43.870 INFO:teuthology.orchestra.run.smithi007.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-18T08:15:43.901 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6 2024-09-18T08:15:43.906 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:15:43 smithi050 ceph-mon[32371]: 1.a scrub starts 2024-09-18T08:15:43.906 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:15:43 smithi050 ceph-mon[32371]: 1.a scrub ok 2024-09-18T08:15:43.906 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:15:43 smithi050 ceph-mon[32371]: 10.7 starting backfill to osd.5 from (0'0,0'0] MAX to 335'4440 2024-09-18T08:15:43.906 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:15:43 smithi050 ceph-mon[32371]: 10.6 starting backfill to osd.5 from (0'0,0'0] MAX to 335'4366 2024-09-18T08:15:43.906 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:15:43 smithi050 ceph-mon[32371]: 10.a starting backfill to osd.5 from (0'0,0'0] MAX to 335'4416 2024-09-18T08:15:43.906 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:15:43 smithi050 ceph-mon[32371]: 10.9 starting backfill to osd.5 from (0'0,0'0] MAX to 335'4495 2024-09-18T08:15:43.906 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:15:43 smithi050 ceph-mon[32371]: 10.3 starting backfill to osd.5 from (0'0,0'0] MAX to 335'4396 2024-09-18T08:15:43.906 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:15:43 smithi050 ceph-mon[32371]: 10.8 starting backfill to osd.5 from (0'0,0'0] MAX to 335'4404 2024-09-18T08:15:43.907 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:15:43 smithi050 ceph-mon[32371]: from='client.? 172.21.15.7:0/2698720237' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-18T08:15:43.907 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:15:43 smithi050 ceph-mon[32371]: 4.a scrub starts 2024-09-18T08:15:43.907 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:15:43 smithi050 ceph-mon[32371]: 4.a scrub ok 2024-09-18T08:15:43.931 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:15:43.932 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:15:43.932 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6' 2024-09-18T08:15:44.033 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1 2024-09-18T08:15:44.077 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:15:44.078 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:15:44.078 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1' 2024-09-18T08:15:44.179 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3 2024-09-18T08:15:44.228 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:15:44.229 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:15:44.229 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3' 2024-09-18T08:15:44.330 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7 2024-09-18T08:15:44.347 INFO:tasks.radosbench.radosbench.0.smithi007.stdout: 56 16 73081 73065 81.525 69.375 0.0160436 0.0122607 2024-09-18T08:15:44.359 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:15:44.360 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:15:44.361 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7' 2024-09-18T08:15:44.461 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3 2024-09-18T08:15:44.505 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:15:44.506 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:15:44.507 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3' 2024-09-18T08:15:44.607 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2 2024-09-18T08:15:44.651 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:15:44.652 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:15:44.653 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2' 2024-09-18T08:15:44.667 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/779349ca-7593-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T08:15:44.722 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid 779349ca-7593-11ef-bceb-c7b262605968 -- ceph daemon osd.2 dump_ops_in_flight 2024-09-18T08:15:44.731 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:15:44 smithi007 ceph-mon[39249]: 6.1e scrub starts 2024-09-18T08:15:44.732 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:15:44 smithi007 ceph-mon[39249]: 6.1e scrub ok 2024-09-18T08:15:44.732 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:15:44 smithi007 ceph-mon[39249]: 1.1 scrub starts 2024-09-18T08:15:44.732 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:15:44 smithi007 ceph-mon[39249]: 1.1 scrub ok 2024-09-18T08:15:44.732 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:15:44 smithi007 ceph-mon[39249]: pgmap v641: 158 pgs: 158 active+clean; 4.3 GiB data, 15 GiB used, 521 GiB / 536 GiB avail; 1.2 KiB/s rd, 11 MiB/s wr, 171 op/s; 136 B/s, 3 objects/s recovering 2024-09-18T08:15:44.732 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:15:44 smithi007 ceph-mon[39249]: 3.18 scrub starts 2024-09-18T08:15:44.732 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:15:44 smithi007 ceph-mon[39249]: 3.18 scrub ok 2024-09-18T08:15:44.732 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:15:44 smithi007 ceph-mon[39249]: osdmap e339: 8 total, 8 up, 7 in 2024-09-18T08:15:44.733 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:15:44 smithi007 ceph-mon[29459]: 6.1e scrub starts 2024-09-18T08:15:44.733 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:15:44 smithi007 ceph-mon[29459]: 6.1e scrub ok 2024-09-18T08:15:44.733 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:15:44 smithi007 ceph-mon[29459]: 1.1 scrub starts 2024-09-18T08:15:44.733 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:15:44 smithi007 ceph-mon[29459]: 1.1 scrub ok 2024-09-18T08:15:44.733 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:15:44 smithi007 ceph-mon[29459]: pgmap v641: 158 pgs: 158 active+clean; 4.3 GiB data, 15 GiB used, 521 GiB / 536 GiB avail; 1.2 KiB/s rd, 11 MiB/s wr, 171 op/s; 136 B/s, 3 objects/s recovering 2024-09-18T08:15:44.733 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:15:44 smithi007 ceph-mon[29459]: 3.18 scrub starts 2024-09-18T08:15:44.733 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:15:44 smithi007 ceph-mon[29459]: 3.18 scrub ok 2024-09-18T08:15:44.733 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:15:44 smithi007 ceph-mon[29459]: osdmap e339: 8 total, 8 up, 7 in 2024-09-18T08:15:44.754 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2 2024-09-18T08:15:44.844 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:15:44.845 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:15:44.845 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2' 2024-09-18T08:15:44.905 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:15:44 smithi050 ceph-mon[32371]: 6.1e scrub starts 2024-09-18T08:15:44.905 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:15:44 smithi050 ceph-mon[32371]: 6.1e scrub ok 2024-09-18T08:15:44.905 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:15:44 smithi050 ceph-mon[32371]: 1.1 scrub starts 2024-09-18T08:15:44.905 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:15:44 smithi050 ceph-mon[32371]: 1.1 scrub ok 2024-09-18T08:15:44.905 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:15:44 smithi050 ceph-mon[32371]: pgmap v641: 158 pgs: 158 active+clean; 4.3 GiB data, 15 GiB used, 521 GiB / 536 GiB avail; 1.2 KiB/s rd, 11 MiB/s wr, 171 op/s; 136 B/s, 3 objects/s recovering 2024-09-18T08:15:44.906 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:15:44 smithi050 ceph-mon[32371]: 3.18 scrub starts 2024-09-18T08:15:44.906 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:15:44 smithi050 ceph-mon[32371]: 3.18 scrub ok 2024-09-18T08:15:44.906 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:15:44 smithi050 ceph-mon[32371]: osdmap e339: 8 total, 8 up, 7 in 2024-09-18T08:15:44.946 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5 2024-09-18T08:15:44.975 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:15:44.976 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:15:44.977 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5' 2024-09-18T08:15:45.077 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0 2024-09-18T08:15:45.118 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:15:45.118 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:15:45.118 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0' 2024-09-18T08:15:45.219 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3 2024-09-18T08:15:45.254 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:15:45.255 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:15:45.255 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3' 2024-09-18T08:15:45.348 INFO:tasks.radosbench.radosbench.0.smithi007.stdout: 57 16 73943 73927 81.0399 53.875 0.00744539 0.0123228 2024-09-18T08:15:45.356 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1 2024-09-18T08:15:45.398 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:15:45.399 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:15:45.400 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1' 2024-09-18T08:15:45.500 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4 2024-09-18T08:15:45.529 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:15:45.530 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:15:45.530 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4' 2024-09-18T08:15:45.621 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid 779349ca-7593-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-18T08:15:45.631 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5 2024-09-18T08:15:45.659 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:15:45.660 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:15:45.660 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5' 2024-09-18T08:15:45.761 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4 2024-09-18T08:15:45.788 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:15:45.789 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:15:45.789 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4' 2024-09-18T08:15:45.811 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:15:45 smithi007 ceph-mon[39249]: 1.7 scrub starts 2024-09-18T08:15:45.811 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:15:45 smithi007 ceph-mon[39249]: 1.7 scrub ok 2024-09-18T08:15:45.812 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:15:45 smithi007 ceph-mon[39249]: 6.1d scrub starts 2024-09-18T08:15:45.812 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:15:45 smithi007 ceph-mon[39249]: 6.1d scrub ok 2024-09-18T08:15:45.812 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:15:45 smithi007 ceph-mon[29459]: 1.7 scrub starts 2024-09-18T08:15:45.812 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:15:45 smithi007 ceph-mon[29459]: 1.7 scrub ok 2024-09-18T08:15:45.812 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:15:45 smithi007 ceph-mon[29459]: 6.1d scrub starts 2024-09-18T08:15:45.812 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:15:45 smithi007 ceph-mon[29459]: 6.1d scrub ok 2024-09-18T08:15:45.890 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2 2024-09-18T08:15:45.905 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:15:45 smithi050 ceph-mon[32371]: 1.7 scrub starts 2024-09-18T08:15:45.905 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:15:45 smithi050 ceph-mon[32371]: 1.7 scrub ok 2024-09-18T08:15:45.905 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:15:45 smithi050 ceph-mon[32371]: 6.1d scrub starts 2024-09-18T08:15:45.905 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:15:45 smithi050 ceph-mon[32371]: 6.1d scrub ok 2024-09-18T08:15:45.937 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:15:45.938 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:15:45.938 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2' 2024-09-18T08:15:46.039 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1 2024-09-18T08:15:46.081 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:15:46.081 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:15:46.082 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1' 2024-09-18T08:15:46.182 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3 2024-09-18T08:15:46.232 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:15:46.233 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:15:46.233 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3' 2024-09-18T08:15:46.334 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7 2024-09-18T08:15:46.348 INFO:tasks.radosbench.radosbench.0.smithi007.stdout: 58 16 74812 74796 80.579 54.3125 0.0135999 0.0124044 2024-09-18T08:15:46.365 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:15:46.366 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:15:46.366 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7' 2024-09-18T08:15:46.437 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 6, 4, 5] out_osds: [7] dead_osds: [] live_osds: [0, 2, 3, 4, 5, 6, 7, 1] 2024-09-18T08:15:46.437 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-18T08:15:46.437 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid 779349ca-7593-11ef-bceb-c7b262605968 -- ceph pg dump --format=json 2024-09-18T08:15:46.467 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3 2024-09-18T08:15:46.487 INFO:teuthology.orchestra.run.smithi007.stderr:nodeep-scrub is unset 2024-09-18T08:15:46.546 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:15:46.547 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:15:46.548 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3' 2024-09-18T08:15:46.648 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6 2024-09-18T08:15:46.677 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:15:46.677 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:15:46.678 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6' 2024-09-18T08:15:46.696 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:15:46 smithi007 ceph-mon[29459]: 5.18 scrub starts 2024-09-18T08:15:46.697 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:15:46 smithi007 ceph-mon[29459]: 5.18 scrub ok 2024-09-18T08:15:46.697 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:15:46 smithi007 ceph-mon[29459]: pgmap v643: 158 pgs: 3 active+remapped+backfilling, 2 active+remapped+backfill_wait, 1 active+remapped+backfill_toofull, 152 active+clean; 4.4 GiB data, 16 GiB used, 610 GiB / 626 GiB avail; 42 MiB/s wr, 661 op/s; 26508/217626 objects misplaced (12.181%); 8.3 MiB/s, 140 objects/s recovering 2024-09-18T08:15:46.697 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:15:46 smithi007 ceph-mon[29459]: Health check failed: Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull (PG_BACKFILL_FULL) 2024-09-18T08:15:46.697 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:15:46 smithi007 ceph-mon[29459]: from='client.? 172.21.15.7:0/3991898767' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-18T08:15:46.697 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:15:46 smithi007 ceph-mon[29459]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-18T08:15:46.697 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:15:46 smithi007 ceph-mon[39249]: 5.18 scrub starts 2024-09-18T08:15:46.697 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:15:46 smithi007 ceph-mon[39249]: 5.18 scrub ok 2024-09-18T08:15:46.697 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:15:46 smithi007 ceph-mon[39249]: pgmap v643: 158 pgs: 3 active+remapped+backfilling, 2 active+remapped+backfill_wait, 1 active+remapped+backfill_toofull, 152 active+clean; 4.4 GiB data, 16 GiB used, 610 GiB / 626 GiB avail; 42 MiB/s wr, 661 op/s; 26508/217626 objects misplaced (12.181%); 8.3 MiB/s, 140 objects/s recovering 2024-09-18T08:15:46.697 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:15:46 smithi007 ceph-mon[39249]: Health check failed: Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull (PG_BACKFILL_FULL) 2024-09-18T08:15:46.698 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:15:46 smithi007 ceph-mon[39249]: from='client.? 172.21.15.7:0/3991898767' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-18T08:15:46.698 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:15:46 smithi007 ceph-mon[39249]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-18T08:15:46.779 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7 2024-09-18T08:15:46.806 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:15:46.806 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:15:46.807 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7' 2024-09-18T08:15:46.905 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:15:46 smithi050 ceph-mon[32371]: 5.18 scrub starts 2024-09-18T08:15:46.905 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:15:46 smithi050 ceph-mon[32371]: 5.18 scrub ok 2024-09-18T08:15:46.905 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:15:46 smithi050 ceph-mon[32371]: pgmap v643: 158 pgs: 3 active+remapped+backfilling, 2 active+remapped+backfill_wait, 1 active+remapped+backfill_toofull, 152 active+clean; 4.4 GiB data, 16 GiB used, 610 GiB / 626 GiB avail; 42 MiB/s wr, 661 op/s; 26508/217626 objects misplaced (12.181%); 8.3 MiB/s, 140 objects/s recovering 2024-09-18T08:15:46.905 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:15:46 smithi050 ceph-mon[32371]: Health check failed: Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull (PG_BACKFILL_FULL) 2024-09-18T08:15:46.905 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:15:46 smithi050 ceph-mon[32371]: from='client.? 172.21.15.7:0/3991898767' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-18T08:15:46.906 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:15:46 smithi050 ceph-mon[32371]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-18T08:15:46.908 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6 2024-09-18T08:15:46.936 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:15:46.937 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:15:46.937 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6' 2024-09-18T08:15:46.980 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Sep 18 08:15:46 smithi007 ceph-779349ca-7593-11ef-bceb-c7b262605968-mgr-y[29688]: ::ffff:172.21.15.50 - - [18/Sep/2024:08:15:46] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-09-18T08:15:47.038 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2 2024-09-18T08:15:47.083 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:15:47.084 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:15:47.084 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2' 2024-09-18T08:15:47.185 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5 2024-09-18T08:15:47.213 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:15:47.213 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:15:47.214 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5' 2024-09-18T08:15:47.232 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/779349ca-7593-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T08:15:47.314 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5 2024-09-18T08:15:47.319 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/779349ca-7593-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T08:15:47.344 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:15:47.345 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:15:47.345 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5' 2024-09-18T08:15:47.348 INFO:tasks.radosbench.radosbench.0.smithi007.stdout: 59 16 75794 75778 80.2526 61.375 0.0202203 0.0124548 2024-09-18T08:15:47.446 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1 2024-09-18T08:15:47.503 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:15:47.504 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:15:47.504 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1' 2024-09-18T08:15:47.605 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6 2024-09-18T08:15:47.635 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:15:47.636 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:15:47.637 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6' 2024-09-18T08:15:47.738 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1 2024-09-18T08:15:47.782 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:15:47.782 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:15:47.783 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1' 2024-09-18T08:15:47.828 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:15:47 smithi007 ceph-mon[29459]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-18T08:15:47.828 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:15:47 smithi007 ceph-mon[29459]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-18T08:15:47.828 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:15:47 smithi007 ceph-mon[29459]: osdmap e340: 8 total, 8 up, 7 in 2024-09-18T08:15:47.828 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:15:47 smithi007 ceph-mon[29459]: 3.8 scrub starts 2024-09-18T08:15:47.828 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:15:47 smithi007 ceph-mon[29459]: 3.8 scrub ok 2024-09-18T08:15:47.829 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:15:47 smithi007 ceph-mon[39249]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-18T08:15:47.829 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:15:47 smithi007 ceph-mon[39249]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-18T08:15:47.829 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:15:47 smithi007 ceph-mon[39249]: osdmap e340: 8 total, 8 up, 7 in 2024-09-18T08:15:47.829 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:15:47 smithi007 ceph-mon[39249]: 3.8 scrub starts 2024-09-18T08:15:47.829 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:15:47 smithi007 ceph-mon[39249]: 3.8 scrub ok 2024-09-18T08:15:47.884 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3 2024-09-18T08:15:47.905 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:15:47 smithi050 ceph-mon[32371]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-18T08:15:47.905 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:15:47 smithi050 ceph-mon[32371]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-18T08:15:47.905 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:15:47 smithi050 ceph-mon[32371]: osdmap e340: 8 total, 8 up, 7 in 2024-09-18T08:15:47.905 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:15:47 smithi050 ceph-mon[32371]: 3.8 scrub starts 2024-09-18T08:15:47.906 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:15:47 smithi050 ceph-mon[32371]: 3.8 scrub ok 2024-09-18T08:15:47.925 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:15:47.926 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:15:47.926 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3' 2024-09-18T08:15:48.027 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3 2024-09-18T08:15:48.056 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/779349ca-7593-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T08:15:48.074 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:15:48.074 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:15:48.075 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3' 2024-09-18T08:15:48.138 INFO:teuthology.orchestra.run.smithi007.stdout:ERROR: (22) Invalid argument 2024-09-18T08:15:48.139 INFO:teuthology.orchestra.run.smithi007.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-18T08:15:48.176 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1 2024-09-18T08:15:48.224 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:15:48.225 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:15:48.225 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1' 2024-09-18T08:15:48.326 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2 2024-09-18T08:15:48.349 INFO:tasks.radosbench.radosbench.0.smithi007.stdout:2024-09-18T08:15:48.348392+0000 min lat: 0.00117701 max lat: 2.16978 avg lat: 0.0125013 2024-09-18T08:15:48.349 INFO:tasks.radosbench.radosbench.0.smithi007.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-09-18T08:15:48.349 INFO:tasks.radosbench.radosbench.0.smithi007.stdout: 60 16 76789 76773 79.9513 62.1875 0.0106733 0.0125013 2024-09-18T08:15:48.373 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:15:48.374 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:15:48.374 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2' 2024-09-18T08:15:48.475 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7 2024-09-18T08:15:48.496 INFO:teuthology.orchestra.run.smithi007.stdout:osd.0: {} 2024-09-18T08:15:48.497 INFO:teuthology.orchestra.run.smithi007.stdout:osd.1: {} 2024-09-18T08:15:48.497 INFO:teuthology.orchestra.run.smithi007.stdout:osd.2: {} 2024-09-18T08:15:48.497 INFO:teuthology.orchestra.run.smithi007.stdout:osd.3: {} 2024-09-18T08:15:48.497 INFO:teuthology.orchestra.run.smithi007.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-18T08:15:48.497 INFO:teuthology.orchestra.run.smithi007.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-18T08:15:48.497 INFO:teuthology.orchestra.run.smithi007.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-18T08:15:48.497 INFO:teuthology.orchestra.run.smithi007.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-18T08:15:48.506 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:15:48.507 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:15:48.507 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7' 2024-09-18T08:15:48.528 INFO:teuthology.orchestra.run.smithi007.stdout:osd.4: {} 2024-09-18T08:15:48.529 INFO:teuthology.orchestra.run.smithi007.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-18T08:15:48.563 INFO:teuthology.orchestra.run.smithi007.stdout:osd.5: {} 2024-09-18T08:15:48.564 INFO:teuthology.orchestra.run.smithi007.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-18T08:15:48.599 INFO:teuthology.orchestra.run.smithi007.stdout:osd.6: {} 2024-09-18T08:15:48.599 INFO:teuthology.orchestra.run.smithi007.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-18T08:15:48.608 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4 2024-09-18T08:15:48.635 INFO:teuthology.orchestra.run.smithi007.stdout:osd.7: {} 2024-09-18T08:15:48.636 INFO:teuthology.orchestra.run.smithi007.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-18T08:15:48.640 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:15:48.641 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:15:48.641 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4' 2024-09-18T08:15:48.730 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:15:48 smithi007 ceph-mon[39249]: pgmap v645: 158 pgs: 3 active+remapped+backfilling, 2 active+remapped+backfill_wait, 1 active+remapped+backfill_toofull, 152 active+clean; 4.4 GiB data, 16 GiB used, 610 GiB / 626 GiB avail; 35 MiB/s wr, 550 op/s; 26508/217626 objects misplaced (12.181%); 6.9 MiB/s, 116 objects/s recovering 2024-09-18T08:15:48.730 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:15:48 smithi007 ceph-mon[39249]: 6.e scrub starts 2024-09-18T08:15:48.730 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:15:48 smithi007 ceph-mon[39249]: 6.e scrub ok 2024-09-18T08:15:48.730 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:15:48 smithi007 ceph-mon[39249]: from='client.? 172.21.15.7:0/1361951961' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-18T08:15:48.731 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:15:48 smithi007 ceph-mon[29459]: pgmap v645: 158 pgs: 3 active+remapped+backfilling, 2 active+remapped+backfill_wait, 1 active+remapped+backfill_toofull, 152 active+clean; 4.4 GiB data, 16 GiB used, 610 GiB / 626 GiB avail; 35 MiB/s wr, 550 op/s; 26508/217626 objects misplaced (12.181%); 6.9 MiB/s, 116 objects/s recovering 2024-09-18T08:15:48.731 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:15:48 smithi007 ceph-mon[29459]: 6.e scrub starts 2024-09-18T08:15:48.731 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:15:48 smithi007 ceph-mon[29459]: 6.e scrub ok 2024-09-18T08:15:48.731 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:15:48 smithi007 ceph-mon[29459]: from='client.? 172.21.15.7:0/1361951961' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-18T08:15:48.742 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6 2024-09-18T08:15:48.774 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:15:48.775 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:15:48.775 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6' 2024-09-18T08:15:48.876 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1 2024-09-18T08:15:48.905 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:15:48 smithi050 ceph-mon[32371]: pgmap v645: 158 pgs: 3 active+remapped+backfilling, 2 active+remapped+backfill_wait, 1 active+remapped+backfill_toofull, 152 active+clean; 4.4 GiB data, 16 GiB used, 610 GiB / 626 GiB avail; 35 MiB/s wr, 550 op/s; 26508/217626 objects misplaced (12.181%); 6.9 MiB/s, 116 objects/s recovering 2024-09-18T08:15:48.905 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:15:48 smithi050 ceph-mon[32371]: 6.e scrub starts 2024-09-18T08:15:48.905 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:15:48 smithi050 ceph-mon[32371]: 6.e scrub ok 2024-09-18T08:15:48.905 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:15:48 smithi050 ceph-mon[32371]: from='client.? 172.21.15.7:0/1361951961' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-18T08:15:48.921 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:15:48.922 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:15:48.923 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1' 2024-09-18T08:15:49.024 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7 2024-09-18T08:15:49.056 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:15:49.175 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:15:49.177 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid 779349ca-7593-11ef-bceb-c7b262605968 -- ceph daemon osd.2 dump_blocked_ops 2024-09-18T08:15:49.177 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7' 2024-09-18T08:15:49.250 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid 779349ca-7593-11ef-bceb-c7b262605968 -- ceph osd set noscrub 2024-09-18T08:15:49.278 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1 2024-09-18T08:15:49.349 INFO:tasks.radosbench.radosbench.0.smithi007.stdout: 61 16 77803 77787 79.6795 63.375 0.0207046 0.0125437 2024-09-18T08:15:49.371 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:15:49.371 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:15:49.372 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1' 2024-09-18T08:15:49.473 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3 2024-09-18T08:15:49.576 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:15:49.577 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:15:49.577 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3' 2024-09-18T08:15:49.678 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1 2024-09-18T08:15:49.729 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:15:49.730 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:15:49.731 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1' 2024-09-18T08:15:49.831 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0 2024-09-18T08:15:49.882 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:15:49.882 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:15:49.883 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0' 2024-09-18T08:15:49.905 INFO:journalctl@ceph.iscsi.iscsi.a.smithi050.stdout:Sep 18 08:15:49 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-iscsi-iscsi-a[57782]: debug there is no tcmu-runner data available 2024-09-18T08:15:49.905 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:15:49 smithi050 ceph-mon[32371]: from='mgr.24470 172.21.15.7:0/2594390261' entity='mgr.y' 2024-09-18T08:15:49.905 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:15:49 smithi050 ceph-mon[32371]: from='mgr.24470 172.21.15.7:0/2594390261' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-18T08:15:49.905 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:15:49 smithi050 ceph-mon[32371]: 3.0 scrub starts 2024-09-18T08:15:49.906 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:15:49 smithi050 ceph-mon[32371]: 3.0 scrub ok 2024-09-18T08:15:49.906 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:15:49 smithi050 ceph-mon[32371]: osdmap e341: 8 total, 8 up, 7 in 2024-09-18T08:15:49.960 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:15:49 smithi007 ceph-mon[29459]: from='mgr.24470 172.21.15.7:0/2594390261' entity='mgr.y' 2024-09-18T08:15:49.960 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:15:49 smithi007 ceph-mon[29459]: from='mgr.24470 172.21.15.7:0/2594390261' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-18T08:15:49.960 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:15:49 smithi007 ceph-mon[29459]: 3.0 scrub starts 2024-09-18T08:15:49.960 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:15:49 smithi007 ceph-mon[29459]: 3.0 scrub ok 2024-09-18T08:15:49.960 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:15:49 smithi007 ceph-mon[29459]: osdmap e341: 8 total, 8 up, 7 in 2024-09-18T08:15:49.961 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:15:49 smithi007 ceph-mon[39249]: from='mgr.24470 172.21.15.7:0/2594390261' entity='mgr.y' 2024-09-18T08:15:49.961 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:15:49 smithi007 ceph-mon[39249]: from='mgr.24470 172.21.15.7:0/2594390261' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-18T08:15:49.961 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:15:49 smithi007 ceph-mon[39249]: 3.0 scrub starts 2024-09-18T08:15:49.961 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:15:49 smithi007 ceph-mon[39249]: 3.0 scrub ok 2024-09-18T08:15:49.961 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:15:49 smithi007 ceph-mon[39249]: osdmap e341: 8 total, 8 up, 7 in 2024-09-18T08:15:49.983 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2 2024-09-18T08:15:50.033 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:15:50.034 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:15:50.034 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2' 2024-09-18T08:15:50.135 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7 2024-09-18T08:15:50.164 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:15:50.164 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:15:50.165 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7' 2024-09-18T08:15:50.265 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3 2024-09-18T08:15:50.314 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:15:50.315 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:15:50.316 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3' 2024-09-18T08:15:50.349 INFO:tasks.radosbench.radosbench.0.smithi007.stdout: 62 16 78740 78724 79.3389 58.5625 0.00652867 0.0125976 2024-09-18T08:15:50.417 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3 2024-09-18T08:15:50.457 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/779349ca-7593-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T08:15:50.475 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:15:50.475 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:15:50.476 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3' 2024-09-18T08:15:50.547 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/779349ca-7593-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T08:15:50.577 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7 2024-09-18T08:15:50.606 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:15:50.607 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:15:50.607 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7' 2024-09-18T08:15:50.708 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5 2024-09-18T08:15:50.739 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:15:50.740 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:15:50.741 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5' 2024-09-18T08:15:50.841 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7 2024-09-18T08:15:50.872 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:15:50.872 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:15:50.873 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7' 2024-09-18T08:15:50.907 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-09-18T08:15:50.907 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:15:50 smithi007 ceph-mon[29459]: purged_snaps scrub starts 2024-09-18T08:15:50.907 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:15:50 smithi007 ceph-mon[29459]: purged_snaps scrub ok 2024-09-18T08:15:50.907 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:15:50 smithi007 ceph-mon[29459]: pgmap v646: 158 pgs: 3 active+remapped+backfilling, 2 active+remapped+backfill_wait, 1 active+remapped+backfill_toofull, 152 active+clean; 4.7 GiB data, 17 GiB used, 609 GiB / 626 GiB avail; 73 MiB/s wr, 1.16k op/s; 26071/232005 objects misplaced (11.237%); 25 MiB/s, 410 objects/s recovering 2024-09-18T08:15:50.908 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:15:50 smithi007 ceph-mon[29459]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-18T08:15:50.908 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:15:50 smithi007 ceph-mon[29459]: 4.9 scrub starts 2024-09-18T08:15:50.908 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:15:50 smithi007 ceph-mon[29459]: 4.9 scrub ok 2024-09-18T08:15:50.908 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:15:50 smithi007 ceph-mon[29459]: 5.0 scrub starts 2024-09-18T08:15:50.908 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:15:50 smithi007 ceph-mon[29459]: 5.0 scrub ok 2024-09-18T08:15:50.908 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:15:50 smithi007 ceph-mon[29459]: 6.1 scrub starts 2024-09-18T08:15:50.908 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:15:50 smithi007 ceph-mon[29459]: 6.1 scrub ok 2024-09-18T08:15:50.909 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:15:50 smithi007 ceph-mon[39249]: purged_snaps scrub starts 2024-09-18T08:15:50.909 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:15:50 smithi007 ceph-mon[39249]: purged_snaps scrub ok 2024-09-18T08:15:50.909 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:15:50 smithi007 ceph-mon[39249]: pgmap v646: 158 pgs: 3 active+remapped+backfilling, 2 active+remapped+backfill_wait, 1 active+remapped+backfill_toofull, 152 active+clean; 4.7 GiB data, 17 GiB used, 609 GiB / 626 GiB avail; 73 MiB/s wr, 1.16k op/s; 26071/232005 objects misplaced (11.237%); 25 MiB/s, 410 objects/s recovering 2024-09-18T08:15:50.909 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:15:50 smithi007 ceph-mon[39249]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-18T08:15:50.909 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:15:50 smithi007 ceph-mon[39249]: 4.9 scrub starts 2024-09-18T08:15:50.909 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:15:50 smithi007 ceph-mon[39249]: 4.9 scrub ok 2024-09-18T08:15:50.909 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:15:50 smithi007 ceph-mon[39249]: 5.0 scrub starts 2024-09-18T08:15:50.909 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:15:50 smithi007 ceph-mon[39249]: 5.0 scrub ok 2024-09-18T08:15:50.909 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:15:50 smithi007 ceph-mon[39249]: 6.1 scrub starts 2024-09-18T08:15:50.909 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:15:50 smithi007 ceph-mon[39249]: 6.1 scrub ok 2024-09-18T08:15:50.914 INFO:teuthology.orchestra.run.smithi007.stderr:dumped all 2024-09-18T08:15:50.974 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6 2024-09-18T08:15:51.005 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:15:51.006 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:15:51.006 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6' 2024-09-18T08:15:51.107 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2 2024-09-18T08:15:51.153 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:15:51.154 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:15:51.154 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2' 2024-09-18T08:15:51.155 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:15:50 smithi050 ceph-mon[32371]: purged_snaps scrub starts 2024-09-18T08:15:51.155 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:15:50 smithi050 ceph-mon[32371]: purged_snaps scrub ok 2024-09-18T08:15:51.155 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:15:50 smithi050 ceph-mon[32371]: pgmap v646: 158 pgs: 3 active+remapped+backfilling, 2 active+remapped+backfill_wait, 1 active+remapped+backfill_toofull, 152 active+clean; 4.7 GiB data, 17 GiB used, 609 GiB / 626 GiB avail; 73 MiB/s wr, 1.16k op/s; 26071/232005 objects misplaced (11.237%); 25 MiB/s, 410 objects/s recovering 2024-09-18T08:15:51.155 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:15:50 smithi050 ceph-mon[32371]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-18T08:15:51.156 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:15:50 smithi050 ceph-mon[32371]: 4.9 scrub starts 2024-09-18T08:15:51.156 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:15:50 smithi050 ceph-mon[32371]: 4.9 scrub ok 2024-09-18T08:15:51.156 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:15:50 smithi050 ceph-mon[32371]: 5.0 scrub starts 2024-09-18T08:15:51.156 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:15:50 smithi050 ceph-mon[32371]: 5.0 scrub ok 2024-09-18T08:15:51.156 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:15:50 smithi050 ceph-mon[32371]: 6.1 scrub starts 2024-09-18T08:15:51.156 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:15:50 smithi050 ceph-mon[32371]: 6.1 scrub ok 2024-09-18T08:15:51.255 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5 2024-09-18T08:15:51.284 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:15:51.284 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:15:51.285 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5' 2024-09-18T08:15:51.349 INFO:tasks.radosbench.radosbench.0.smithi007.stdout: 63 16 79615 79599 78.9476 54.6875 0.0170051 0.0126613 2024-09-18T08:15:51.386 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0 2024-09-18T08:15:51.427 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:15:51.427 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:15:51.428 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0' 2024-09-18T08:15:51.528 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6 2024-09-18T08:15:51.559 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:15:51.559 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:15:51.560 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6' 2024-09-18T08:15:51.661 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1 2024-09-18T08:15:51.700 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:15:51.701 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:15:51.702 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1' 2024-09-18T08:15:51.802 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4 2024-09-18T08:15:51.824 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:15:51.825 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:15:51.825 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4' 2024-09-18T08:15:51.926 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0 2024-09-18T08:15:51.965 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid 779349ca-7593-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-18T08:15:51.971 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:15:51.974 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:15:51.974 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0' 2024-09-18T08:15:51.980 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:15:51 smithi007 ceph-mon[29459]: 4.10 scrub starts 2024-09-18T08:15:51.980 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:15:51 smithi007 ceph-mon[29459]: 4.10 scrub ok 2024-09-18T08:15:51.980 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:15:51 smithi007 ceph-mon[39249]: 4.10 scrub starts 2024-09-18T08:15:51.980 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:15:51 smithi007 ceph-mon[39249]: 4.10 scrub ok 2024-09-18T08:15:52.075 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2 2024-09-18T08:15:52.122 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:15:52.123 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:15:52.123 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2' 2024-09-18T08:15:52.155 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:15:51 smithi050 ceph-mon[32371]: 4.10 scrub starts 2024-09-18T08:15:52.155 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:15:51 smithi050 ceph-mon[32371]: 4.10 scrub ok 2024-09-18T08:15:52.224 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7 2024-09-18T08:15:52.245 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:15:52.246 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:15:52.246 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7' 2024-09-18T08:15:52.347 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5 2024-09-18T08:15:52.349 INFO:tasks.radosbench.radosbench.0.smithi007.stdout: 64 16 80497 80481 78.5752 55.125 0.0112747 0.0127207 2024-09-18T08:15:52.378 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:15:52.378 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:15:52.379 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5' 2024-09-18T08:15:52.479 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4 2024-09-18T08:15:52.509 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:15:52.509 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:15:52.510 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4' 2024-09-18T08:15:52.610 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4 2024-09-18T08:15:52.633 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:15:52.633 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:15:52.634 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4' 2024-09-18T08:15:52.735 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5 2024-09-18T08:15:52.767 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:15:52.767 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:15:52.768 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5' 2024-09-18T08:15:52.868 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7 2024-09-18T08:15:52.900 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:15:52.901 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:15:52.901 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7' 2024-09-18T08:15:52.980 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:15:52 smithi007 ceph-mon[29459]: from='client.16059 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-18T08:15:52.980 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:15:52 smithi007 ceph-mon[29459]: pgmap v648: 158 pgs: 3 active+remapped+backfilling, 2 active+remapped+backfill_wait, 1 active+remapped+backfill_toofull, 152 active+clean; 4.7 GiB data, 17 GiB used, 609 GiB / 626 GiB avail; 731 B/s rd, 43 MiB/s wr, 685 op/s; 26071/232005 objects misplaced (11.237%); 19 MiB/s, 308 objects/s recovering 2024-09-18T08:15:52.980 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:15:52 smithi007 ceph-mon[29459]: 4.1 scrub starts 2024-09-18T08:15:52.980 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:15:52 smithi007 ceph-mon[29459]: 4.1 scrub ok 2024-09-18T08:15:52.980 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:15:52 smithi007 ceph-mon[29459]: 6.1b scrub starts 2024-09-18T08:15:52.980 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:15:52 smithi007 ceph-mon[29459]: 6.1b scrub ok 2024-09-18T08:15:52.980 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:15:52 smithi007 ceph-mon[29459]: 4.12 scrub starts 2024-09-18T08:15:52.980 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:15:52 smithi007 ceph-mon[29459]: 4.12 scrub ok 2024-09-18T08:15:52.981 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:15:52 smithi007 ceph-mon[39249]: from='client.16059 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-18T08:15:52.981 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:15:52 smithi007 ceph-mon[39249]: pgmap v648: 158 pgs: 3 active+remapped+backfilling, 2 active+remapped+backfill_wait, 1 active+remapped+backfill_toofull, 152 active+clean; 4.7 GiB data, 17 GiB used, 609 GiB / 626 GiB avail; 731 B/s rd, 43 MiB/s wr, 685 op/s; 26071/232005 objects misplaced (11.237%); 19 MiB/s, 308 objects/s recovering 2024-09-18T08:15:52.981 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:15:52 smithi007 ceph-mon[39249]: 4.1 scrub starts 2024-09-18T08:15:52.981 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:15:52 smithi007 ceph-mon[39249]: 4.1 scrub ok 2024-09-18T08:15:52.981 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:15:52 smithi007 ceph-mon[39249]: 6.1b scrub starts 2024-09-18T08:15:52.981 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:15:52 smithi007 ceph-mon[39249]: 6.1b scrub ok 2024-09-18T08:15:52.981 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:15:52 smithi007 ceph-mon[39249]: 4.12 scrub starts 2024-09-18T08:15:52.981 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:15:52 smithi007 ceph-mon[39249]: 4.12 scrub ok 2024-09-18T08:15:53.001 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7 2024-09-18T08:15:53.028 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:15:53.029 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:15:53.029 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7' 2024-09-18T08:15:53.130 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2 2024-09-18T08:15:53.155 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:15:52 smithi050 ceph-mon[32371]: from='client.16059 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-18T08:15:53.155 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:15:52 smithi050 ceph-mon[32371]: pgmap v648: 158 pgs: 3 active+remapped+backfilling, 2 active+remapped+backfill_wait, 1 active+remapped+backfill_toofull, 152 active+clean; 4.7 GiB data, 17 GiB used, 609 GiB / 626 GiB avail; 731 B/s rd, 43 MiB/s wr, 685 op/s; 26071/232005 objects misplaced (11.237%); 19 MiB/s, 308 objects/s recovering 2024-09-18T08:15:53.155 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:15:52 smithi050 ceph-mon[32371]: 4.1 scrub starts 2024-09-18T08:15:53.155 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:15:52 smithi050 ceph-mon[32371]: 4.1 scrub ok 2024-09-18T08:15:53.155 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:15:52 smithi050 ceph-mon[32371]: 6.1b scrub starts 2024-09-18T08:15:53.156 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:15:52 smithi050 ceph-mon[32371]: 6.1b scrub ok 2024-09-18T08:15:53.156 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:15:52 smithi050 ceph-mon[32371]: 4.12 scrub starts 2024-09-18T08:15:53.156 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:15:52 smithi050 ceph-mon[32371]: 4.12 scrub ok 2024-09-18T08:15:53.167 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:15:53.167 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:15:53.168 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2' 2024-09-18T08:15:53.268 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7 2024-09-18T08:15:53.297 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:15:53.297 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:15:53.297 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7' 2024-09-18T08:15:53.349 INFO:tasks.radosbench.radosbench.0.smithi007.stdout: 65 16 81379 81363 78.2144 55.125 0.0187795 0.0127801 2024-09-18T08:15:53.398 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7 2024-09-18T08:15:53.420 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:15:53.421 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:15:53.421 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7' 2024-09-18T08:15:53.522 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7 2024-09-18T08:15:53.553 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:15:53.553 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:15:53.554 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7' 2024-09-18T08:15:53.654 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6 2024-09-18T08:15:53.680 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:15:53.681 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:15:53.681 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6' 2024-09-18T08:15:53.782 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4 2024-09-18T08:15:53.807 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:15:53.808 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:15:53.808 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4' 2024-09-18T08:15:53.909 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2 2024-09-18T08:15:53.953 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:15:53.954 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:15:53.954 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2' 2024-09-18T08:15:53.980 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:15:53 smithi007 ceph-mon[29459]: 4.14 scrub starts 2024-09-18T08:15:53.980 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:15:53 smithi007 ceph-mon[29459]: 4.14 scrub ok 2024-09-18T08:15:53.980 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:15:53 smithi007 ceph-mon[29459]: pgmap v649: 158 pgs: 3 active+remapped+backfilling, 2 active+remapped+backfill_wait, 1 active+remapped+backfill_toofull, 152 active+clean; 4.8 GiB data, 17 GiB used, 608 GiB / 626 GiB avail; 1.2 KiB/s rd, 44 MiB/s wr, 707 op/s; 26071/234582 objects misplaced (11.114%); 17 MiB/s, 269 objects/s recovering 2024-09-18T08:15:53.980 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:15:53 smithi007 ceph-mon[39249]: 4.14 scrub starts 2024-09-18T08:15:53.980 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:15:53 smithi007 ceph-mon[39249]: 4.14 scrub ok 2024-09-18T08:15:53.980 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:15:53 smithi007 ceph-mon[39249]: pgmap v649: 158 pgs: 3 active+remapped+backfilling, 2 active+remapped+backfill_wait, 1 active+remapped+backfill_toofull, 152 active+clean; 4.8 GiB data, 17 GiB used, 608 GiB / 626 GiB avail; 1.2 KiB/s rd, 44 MiB/s wr, 707 op/s; 26071/234582 objects misplaced (11.114%); 17 MiB/s, 269 objects/s recovering 2024-09-18T08:15:54.055 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5 2024-09-18T08:15:54.084 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:15:54.085 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:15:54.085 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5' 2024-09-18T08:15:54.155 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:15:53 smithi050 ceph-mon[32371]: 4.14 scrub starts 2024-09-18T08:15:54.155 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:15:53 smithi050 ceph-mon[32371]: 4.14 scrub ok 2024-09-18T08:15:54.155 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:15:53 smithi050 ceph-mon[32371]: pgmap v649: 158 pgs: 3 active+remapped+backfilling, 2 active+remapped+backfill_wait, 1 active+remapped+backfill_toofull, 152 active+clean; 4.8 GiB data, 17 GiB used, 608 GiB / 626 GiB avail; 1.2 KiB/s rd, 44 MiB/s wr, 707 op/s; 26071/234582 objects misplaced (11.114%); 17 MiB/s, 269 objects/s recovering 2024-09-18T08:15:54.186 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5 2024-09-18T08:15:54.217 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:15:54.218 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:15:54.218 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5' 2024-09-18T08:15:54.319 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7 2024-09-18T08:15:54.349 INFO:tasks.radosbench.radosbench.0.smithi007.stdout: 66 16 81970 81954 77.5891 36.9375 0.0136885 0.0128834 2024-09-18T08:15:54.352 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:15:54.352 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:15:54.353 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7' 2024-09-18T08:15:54.454 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7 2024-09-18T08:15:54.497 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:15:54.497 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:15:54.498 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7' 2024-09-18T08:15:54.599 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0 2024-09-18T08:15:54.672 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:15:54.673 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:15:54.673 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0' 2024-09-18T08:15:54.774 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5 2024-09-18T08:15:54.796 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:15:54.797 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:15:54.798 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5' 2024-09-18T08:15:54.898 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0 2024-09-18T08:15:54.945 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:15:54.946 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:15:54.946 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0' 2024-09-18T08:15:54.980 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:15:54 smithi007 ceph-mon[29459]: from='mgr.24470 172.21.15.7:0/2594390261' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-18T08:15:54.980 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:15:54 smithi007 ceph-mon[39249]: from='mgr.24470 172.21.15.7:0/2594390261' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-18T08:15:55.047 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0 2024-09-18T08:15:55.089 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:15:55.089 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:15:55.090 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0' 2024-09-18T08:15:55.155 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:15:54 smithi050 ceph-mon[32371]: from='mgr.24470 172.21.15.7:0/2594390261' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-18T08:15:55.191 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0 2024-09-18T08:15:55.229 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:15:55.229 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:15:55.230 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0' 2024-09-18T08:15:55.331 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2 2024-09-18T08:15:55.349 INFO:tasks.radosbench.radosbench.0.smithi007.stdout: 67 15 82738 82723 77.1484 48.0625 0.0179368 0.0129566 2024-09-18T08:15:55.375 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:15:55.376 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:15:55.376 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2' 2024-09-18T08:15:55.477 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2 2024-09-18T08:15:55.518 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:15:55.519 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:15:55.519 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2' 2024-09-18T08:15:55.620 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5 2024-09-18T08:15:55.652 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:15:55.652 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:15:55.653 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5' 2024-09-18T08:15:55.754 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1 2024-09-18T08:15:55.778 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:15:55.778 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:15:55.779 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1' 2024-09-18T08:15:55.880 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7 2024-09-18T08:15:55.911 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:15:55.911 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:15:55.911 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7' 2024-09-18T08:15:55.980 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:15:55 smithi007 ceph-mon[29459]: 5.12 scrub starts 2024-09-18T08:15:55.980 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:15:55 smithi007 ceph-mon[29459]: 5.12 scrub ok 2024-09-18T08:15:55.980 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:15:55 smithi007 ceph-mon[29459]: pgmap v650: 158 pgs: 3 active+remapped+backfilling, 2 active+remapped+backfill_wait, 1 active+remapped+backfill_toofull, 152 active+clean; 5.0 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 1.1 KiB/s rd, 64 MiB/s wr, 1.03k op/s; 25122/245079 objects misplaced (10.251%); 31 MiB/s, 502 objects/s recovering 2024-09-18T08:15:55.980 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:15:55 smithi007 ceph-mon[39249]: 5.12 scrub starts 2024-09-18T08:15:55.980 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:15:55 smithi007 ceph-mon[39249]: 5.12 scrub ok 2024-09-18T08:15:55.980 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:15:55 smithi007 ceph-mon[39249]: pgmap v650: 158 pgs: 3 active+remapped+backfilling, 2 active+remapped+backfill_wait, 1 active+remapped+backfill_toofull, 152 active+clean; 5.0 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 1.1 KiB/s rd, 64 MiB/s wr, 1.03k op/s; 25122/245079 objects misplaced (10.251%); 31 MiB/s, 502 objects/s recovering 2024-09-18T08:15:56.012 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5 2024-09-18T08:15:56.044 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:15:56.044 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:15:56.045 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5' 2024-09-18T08:15:56.145 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2 2024-09-18T08:15:56.155 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:15:55 smithi050 ceph-mon[32371]: 5.12 scrub starts 2024-09-18T08:15:56.155 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:15:55 smithi050 ceph-mon[32371]: 5.12 scrub ok 2024-09-18T08:15:56.155 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:15:55 smithi050 ceph-mon[32371]: pgmap v650: 158 pgs: 3 active+remapped+backfilling, 2 active+remapped+backfill_wait, 1 active+remapped+backfill_toofull, 152 active+clean; 5.0 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 1.1 KiB/s rd, 64 MiB/s wr, 1.03k op/s; 25122/245079 objects misplaced (10.251%); 31 MiB/s, 502 objects/s recovering 2024-09-18T08:15:56.186 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:15:56.187 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:15:56.187 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2' 2024-09-18T08:15:56.288 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6 2024-09-18T08:15:56.318 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:15:56.318 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:15:56.319 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6' 2024-09-18T08:15:56.349 INFO:tasks.radosbench.radosbench.0.smithi007.stdout: 68 16 83585 83569 76.7914 52.875 0.00620127 0.0130074 2024-09-18T08:15:56.419 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3 2024-09-18T08:15:56.462 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:15:56.463 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:15:56.463 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3' 2024-09-18T08:15:56.564 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7 2024-09-18T08:15:56.594 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:15:56.595 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:15:56.596 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7' 2024-09-18T08:15:56.696 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2 2024-09-18T08:15:56.722 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:15:56.722 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:15:56.723 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2' 2024-09-18T08:15:56.824 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0 2024-09-18T08:15:56.858 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:15:56.858 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:15:56.859 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0' 2024-09-18T08:15:56.960 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4 2024-09-18T08:15:56.980 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:15:56 smithi007 ceph-mon[29459]: purged_snaps scrub starts 2024-09-18T08:15:56.980 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:15:56 smithi007 ceph-mon[29459]: purged_snaps scrub ok 2024-09-18T08:15:56.980 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:15:56 smithi007 ceph-mon[29459]: osdmap e342: 8 total, 8 up, 7 in 2024-09-18T08:15:56.980 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Sep 18 08:15:56 smithi007 ceph-779349ca-7593-11ef-bceb-c7b262605968-mgr-y[29688]: ::ffff:172.21.15.50 - - [18/Sep/2024:08:15:56] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-09-18T08:15:56.981 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:15:56 smithi007 ceph-mon[39249]: purged_snaps scrub starts 2024-09-18T08:15:56.981 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:15:56 smithi007 ceph-mon[39249]: purged_snaps scrub ok 2024-09-18T08:15:56.981 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:15:56 smithi007 ceph-mon[39249]: osdmap e342: 8 total, 8 up, 7 in 2024-09-18T08:15:56.991 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:15:56.991 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:15:56.992 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4' 2024-09-18T08:15:57.093 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0 2024-09-18T08:15:57.117 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:15:57.118 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:15:57.118 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0' 2024-09-18T08:15:57.155 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:15:56 smithi050 ceph-mon[32371]: purged_snaps scrub starts 2024-09-18T08:15:57.155 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:15:56 smithi050 ceph-mon[32371]: purged_snaps scrub ok 2024-09-18T08:15:57.155 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:15:56 smithi050 ceph-mon[32371]: osdmap e342: 8 total, 8 up, 7 in 2024-09-18T08:15:57.219 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1 2024-09-18T08:15:57.261 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:15:57.261 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:15:57.262 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1' 2024-09-18T08:15:57.350 INFO:tasks.radosbench.radosbench.0.smithi007.stdout: 69 16 84487 84471 76.4955 56.375 0.00988299 0.0130673 2024-09-18T08:15:57.362 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2 2024-09-18T08:15:57.405 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:15:57.406 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:15:57.406 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2' 2024-09-18T08:15:57.507 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7 2024-09-18T08:15:57.538 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:15:57.539 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:15:57.539 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7' 2024-09-18T08:15:57.640 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3 2024-09-18T08:15:57.682 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:15:57.682 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:15:57.683 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3' 2024-09-18T08:15:57.784 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7 2024-09-18T08:15:57.808 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:15:57.809 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:15:57.809 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7' 2024-09-18T08:15:57.910 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0 2024-09-18T08:15:57.953 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:15:57.953 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:15:57.954 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0' 2024-09-18T08:15:58.057 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1 2024-09-18T08:15:58.101 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:15:58.101 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:15:58.102 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1' 2024-09-18T08:15:58.155 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:15:57 smithi050 ceph-mon[32371]: 4.c scrub starts 2024-09-18T08:15:58.155 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:15:57 smithi050 ceph-mon[32371]: 4.c scrub ok 2024-09-18T08:15:58.155 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:15:57 smithi050 ceph-mon[32371]: purged_snaps scrub starts 2024-09-18T08:15:58.155 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:15:57 smithi050 ceph-mon[32371]: purged_snaps scrub ok 2024-09-18T08:15:58.156 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:15:57 smithi050 ceph-mon[32371]: pgmap v652: 158 pgs: 3 active+remapped+backfilling, 2 active+remapped+backfill_wait, 1 active+remapped+backfill_toofull, 152 active+clean; 5.0 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 639 B/s rd, 34 MiB/s wr, 545 op/s; 25122/245079 objects misplaced (10.251%); 18 MiB/s, 287 objects/s recovering 2024-09-18T08:15:58.156 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:15:57 smithi050 ceph-mon[32371]: 2.0 scrub starts 2024-09-18T08:15:58.156 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:15:57 smithi050 ceph-mon[32371]: 2.0 scrub ok 2024-09-18T08:15:58.202 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5 2024-09-18T08:15:58.230 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:15:57 smithi007 ceph-mon[29459]: 4.c scrub starts 2024-09-18T08:15:58.230 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:15:57 smithi007 ceph-mon[29459]: 4.c scrub ok 2024-09-18T08:15:58.230 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:15:57 smithi007 ceph-mon[29459]: purged_snaps scrub starts 2024-09-18T08:15:58.230 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:15:57 smithi007 ceph-mon[29459]: purged_snaps scrub ok 2024-09-18T08:15:58.230 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:15:57 smithi007 ceph-mon[29459]: pgmap v652: 158 pgs: 3 active+remapped+backfilling, 2 active+remapped+backfill_wait, 1 active+remapped+backfill_toofull, 152 active+clean; 5.0 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 639 B/s rd, 34 MiB/s wr, 545 op/s; 25122/245079 objects misplaced (10.251%); 18 MiB/s, 287 objects/s recovering 2024-09-18T08:15:58.230 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:15:57 smithi007 ceph-mon[29459]: 2.0 scrub starts 2024-09-18T08:15:58.230 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:15:57 smithi007 ceph-mon[29459]: 2.0 scrub ok 2024-09-18T08:15:58.230 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:15:57 smithi007 ceph-mon[39249]: 4.c scrub starts 2024-09-18T08:15:58.231 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:15:57 smithi007 ceph-mon[39249]: 4.c scrub ok 2024-09-18T08:15:58.231 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:15:57 smithi007 ceph-mon[39249]: purged_snaps scrub starts 2024-09-18T08:15:58.231 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:15:57 smithi007 ceph-mon[39249]: purged_snaps scrub ok 2024-09-18T08:15:58.231 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:15:57 smithi007 ceph-mon[39249]: pgmap v652: 158 pgs: 3 active+remapped+backfilling, 2 active+remapped+backfill_wait, 1 active+remapped+backfill_toofull, 152 active+clean; 5.0 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 639 B/s rd, 34 MiB/s wr, 545 op/s; 25122/245079 objects misplaced (10.251%); 18 MiB/s, 287 objects/s recovering 2024-09-18T08:15:58.231 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:15:57 smithi007 ceph-mon[39249]: 2.0 scrub starts 2024-09-18T08:15:58.231 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:15:57 smithi007 ceph-mon[39249]: 2.0 scrub ok 2024-09-18T08:15:58.232 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:15:58.233 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:15:58.233 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5' 2024-09-18T08:15:58.334 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6 2024-09-18T08:15:58.349 INFO:tasks.radosbench.radosbench.0.smithi007.stdout: 70 16 85279 85263 76.1098 49.5 0.0156556 0.0131329 2024-09-18T08:15:58.363 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:15:58.364 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:15:58.364 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6' 2024-09-18T08:15:58.465 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0 2024-09-18T08:15:58.511 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:15:58.511 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:15:58.511 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0' 2024-09-18T08:15:58.612 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4 2024-09-18T08:15:58.642 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:15:58.643 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:15:58.643 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4' 2024-09-18T08:15:58.744 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1 2024-09-18T08:15:58.791 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:15:58.791 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:15:58.791 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1' 2024-09-18T08:15:58.892 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2 2024-09-18T08:15:58.938 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:15:58.938 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:15:58.939 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2' 2024-09-18T08:15:59.039 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5 2024-09-18T08:15:59.071 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:15:59.072 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:15:59.072 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5' 2024-09-18T08:15:59.155 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:15:58 smithi050 ceph-mon[32371]: osdmap e343: 8 total, 8 up, 7 in 2024-09-18T08:15:59.155 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:15:58 smithi050 ceph-mon[32371]: 4.0 scrub starts 2024-09-18T08:15:59.155 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:15:58 smithi050 ceph-mon[32371]: 4.0 scrub ok 2024-09-18T08:15:59.174 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5 2024-09-18T08:15:59.196 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:15:59.197 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:15:59.197 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5' 2024-09-18T08:15:59.230 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:15:58 smithi007 ceph-mon[29459]: osdmap e343: 8 total, 8 up, 7 in 2024-09-18T08:15:59.230 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:15:58 smithi007 ceph-mon[29459]: 4.0 scrub starts 2024-09-18T08:15:59.230 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:15:58 smithi007 ceph-mon[29459]: 4.0 scrub ok 2024-09-18T08:15:59.230 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:15:58 smithi007 ceph-mon[39249]: osdmap e343: 8 total, 8 up, 7 in 2024-09-18T08:15:59.230 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:15:58 smithi007 ceph-mon[39249]: 4.0 scrub starts 2024-09-18T08:15:59.231 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:15:58 smithi007 ceph-mon[39249]: 4.0 scrub ok 2024-09-18T08:15:59.298 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2 2024-09-18T08:15:59.340 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:15:59.341 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:15:59.341 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2' 2024-09-18T08:15:59.350 INFO:tasks.radosbench.radosbench.0.smithi007.stdout: 71 15 86328 86313 75.9621 65.625 0.0194455 0.0131594 2024-09-18T08:15:59.442 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1 2024-09-18T08:15:59.486 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:15:59.487 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:15:59.488 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1' 2024-09-18T08:15:59.588 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3 2024-09-18T08:15:59.632 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:15:59.633 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:15:59.633 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3' 2024-09-18T08:15:59.734 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5 2024-09-18T08:15:59.767 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:15:59.767 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:15:59.768 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5' 2024-09-18T08:15:59.869 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5 2024-09-18T08:15:59.904 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:15:59.905 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:15:59.905 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5' 2024-09-18T08:15:59.905 INFO:journalctl@ceph.iscsi.iscsi.a.smithi050.stdout:Sep 18 08:15:59 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-iscsi-iscsi-a[57782]: debug there is no tcmu-runner data available 2024-09-18T08:15:59.906 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:15:59 smithi050 ceph-mon[32371]: 1.5 scrub starts 2024-09-18T08:15:59.906 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:15:59 smithi050 ceph-mon[32371]: 1.5 scrub ok 2024-09-18T08:15:59.906 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:15:59 smithi050 ceph-mon[32371]: 3.15 scrub starts 2024-09-18T08:15:59.906 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:15:59 smithi050 ceph-mon[32371]: 3.15 scrub ok 2024-09-18T08:15:59.906 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:15:59 smithi050 ceph-mon[32371]: pgmap v654: 158 pgs: 3 active+remapped+backfilling, 2 active+remapped+backfill_wait, 1 active+remapped+backfill_toofull, 152 active+clean; 5.2 GiB data, 19 GiB used, 607 GiB / 626 GiB avail; 1.2 KiB/s rd, 58 MiB/s wr, 923 op/s; 25068/254163 objects misplaced (9.863%); 24 MiB/s, 378 objects/s recovering 2024-09-18T08:15:59.907 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:15:59 smithi050 ceph-mon[32371]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-18T08:16:00.007 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3 2024-09-18T08:16:00.046 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:16:00.047 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:16:00.047 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3' 2024-09-18T08:16:00.148 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0 2024-09-18T08:16:00.169 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:15:59 smithi007 ceph-mon[29459]: 1.5 scrub starts 2024-09-18T08:16:00.169 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:15:59 smithi007 ceph-mon[29459]: 1.5 scrub ok 2024-09-18T08:16:00.169 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:15:59 smithi007 ceph-mon[29459]: 3.15 scrub starts 2024-09-18T08:16:00.169 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:15:59 smithi007 ceph-mon[29459]: 3.15 scrub ok 2024-09-18T08:16:00.169 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:15:59 smithi007 ceph-mon[29459]: pgmap v654: 158 pgs: 3 active+remapped+backfilling, 2 active+remapped+backfill_wait, 1 active+remapped+backfill_toofull, 152 active+clean; 5.2 GiB data, 19 GiB used, 607 GiB / 626 GiB avail; 1.2 KiB/s rd, 58 MiB/s wr, 923 op/s; 25068/254163 objects misplaced (9.863%); 24 MiB/s, 378 objects/s recovering 2024-09-18T08:16:00.170 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:15:59 smithi007 ceph-mon[29459]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-18T08:16:00.170 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:15:59 smithi007 ceph-mon[39249]: 1.5 scrub starts 2024-09-18T08:16:00.170 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:15:59 smithi007 ceph-mon[39249]: 1.5 scrub ok 2024-09-18T08:16:00.170 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:15:59 smithi007 ceph-mon[39249]: 3.15 scrub starts 2024-09-18T08:16:00.170 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:15:59 smithi007 ceph-mon[39249]: 3.15 scrub ok 2024-09-18T08:16:00.170 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:15:59 smithi007 ceph-mon[39249]: pgmap v654: 158 pgs: 3 active+remapped+backfilling, 2 active+remapped+backfill_wait, 1 active+remapped+backfill_toofull, 152 active+clean; 5.2 GiB data, 19 GiB used, 607 GiB / 626 GiB avail; 1.2 KiB/s rd, 58 MiB/s wr, 923 op/s; 25068/254163 objects misplaced (9.863%); 24 MiB/s, 378 objects/s recovering 2024-09-18T08:16:00.170 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:15:59 smithi007 ceph-mon[39249]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-18T08:16:00.197 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:16:00.198 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:16:00.198 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0' 2024-09-18T08:16:00.226 INFO:teuthology.orchestra.run.smithi007.stdout:{"pg_ready":true,"pg_map":{"version":647,"stamp":"2024-09-18T08:15:49.368832+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":5055781403,"num_objects":77335,"num_object_clones":0,"num_object_copies":232005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":26071,"num_objects_unfound":0,"num_objects_dirty":77335,"num_whiteouts":0,"num_read":1372,"num_read_kb":1124,"num_write":77858,"num_write_kb":4936749,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6835,"num_bytes_recovered":419639600,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":68220,"ondisk_log_size":68220,"up":474,"acting":474,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":480,"num_osds":7,"num_per_pool_osds":7,"num_per_pool_omap_osds":7,"kb":656216064,"kb_used":18059720,"kb_used_data":14926220,"kb_used_omap":17,"kb_used_meta":3133358,"kb_avail":638156344,"statfs":{"total":671965249536,"available":653472096256,"internally_reserved":0,"allocated":15284449280,"data_stored":15277784803,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17644,"internal_metadata":3208559380},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":484990133,"num_objects":7401,"num_object_clones":0,"num_object_copies":22203,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":26071,"num_objects_unfound":0,"num_objects_dirty":7401,"num_whiteouts":0,"num_read":-16,"num_read_kb":-35,"num_write":7381,"num_write_kb":473432,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2838,"num_bytes_recovered":184443767,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.005544"},"pg_stats":[{"pgid":"3.1f","version":"0'0","reported_seq":633,"reported_epoch":340,"state":"active+clean","last_fresh":"2024-09-18T08:15:46.508191+0000","last_change":"2024-09-18T08:15:41.451370+0000","last_active":"2024-09-18T08:15:46.508191+0000","last_peered":"2024-09-18T08:15:46.508191+0000","last_clean":"2024-09-18T08:15:46.508191+0000","last_became_active":"2024-09-18T08:15:41.451012+0000","last_became_peered":"2024-09-18T08:15:41.451012+0000","last_unstale":"2024-09-18T08:15:46.508191+0000","last_undegraded":"2024-09-18T08:15:46.508191+0000","last_fullsized":"2024-09-18T08:15:46.508191+0000","mapping_epoch":336,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":337,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:14:29.060206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:46.860092+0000","last_clean_scrub_stamp":"2024-09-18T08:14:29.060206+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"271'15","reported_seq":403,"reported_epoch":340,"state":"active+clean","last_fresh":"2024-09-18T08:15:46.485049+0000","last_change":"2024-09-18T08:15:19.462945+0000","last_active":"2024-09-18T08:15:46.485049+0000","last_peered":"2024-09-18T08:15:46.485049+0000","last_clean":"2024-09-18T08:15:46.485049+0000","last_became_active":"2024-09-18T08:15:19.451028+0000","last_became_peered":"2024-09-18T08:15:19.451028+0000","last_unstale":"2024-09-18T08:15:46.485049+0000","last_undegraded":"2024-09-18T08:15:46.485049+0000","last_fullsized":"2024-09-18T08:15:46.485049+0000","mapping_epoch":329,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":330,"parent":"0.0","parent_split_bits":0,"last_scrub":"271'15","last_scrub_stamp":"2024-09-18T08:15:04.676980+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_clean_scrub_stamp":"2024-09-18T08:15:04.676980+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:16:09.396272+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":25,"num_read_kb":19,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,2],"acting":[4,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":450,"reported_epoch":340,"state":"active+clean","last_fresh":"2024-09-18T08:15:46.494910+0000","last_change":"2024-09-18T08:15:41.440121+0000","last_active":"2024-09-18T08:15:46.494910+0000","last_peered":"2024-09-18T08:15:46.494910+0000","last_clean":"2024-09-18T08:15:46.494910+0000","last_became_active":"2024-09-18T08:15:41.438512+0000","last_became_peered":"2024-09-18T08:15:41.438512+0000","last_unstale":"2024-09-18T08:15:46.494910+0000","last_undegraded":"2024-09-18T08:15:46.494910+0000","last_fullsized":"2024-09-18T08:15:46.494910+0000","mapping_epoch":336,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":337,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:15:12.171225+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:50.886830+0000","last_clean_scrub_stamp":"2024-09-18T08:15:12.171225+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:16:40.681407+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,2],"acting":[1,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":364,"reported_epoch":340,"state":"active+clean","last_fresh":"2024-09-18T08:15:46.485591+0000","last_change":"2024-09-18T08:15:41.444849+0000","last_active":"2024-09-18T08:15:46.485591+0000","last_peered":"2024-09-18T08:15:46.485591+0000","last_clean":"2024-09-18T08:15:46.485591+0000","last_became_active":"2024-09-18T08:15:41.444477+0000","last_became_peered":"2024-09-18T08:15:41.444477+0000","last_unstale":"2024-09-18T08:15:46.485591+0000","last_undegraded":"2024-09-18T08:15:46.485591+0000","last_fullsized":"2024-09-18T08:15:46.485591+0000","mapping_epoch":336,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":337,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:14:41.746083+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_clean_scrub_stamp":"2024-09-18T08:14:41.746083+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":619,"reported_epoch":340,"state":"active+clean","last_fresh":"2024-09-18T08:15:46.536025+0000","last_change":"2024-09-18T08:14:40.357598+0000","last_active":"2024-09-18T08:15:46.536025+0000","last_peered":"2024-09-18T08:15:46.536025+0000","last_clean":"2024-09-18T08:15:46.536025+0000","last_became_active":"2024-09-18T08:14:40.357298+0000","last_became_peered":"2024-09-18T08:14:40.357298+0000","last_unstale":"2024-09-18T08:15:46.536025+0000","last_undegraded":"2024-09-18T08:15:46.536025+0000","last_fullsized":"2024-09-18T08:15:46.536025+0000","mapping_epoch":308,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":309,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:12:38.915501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_clean_scrub_stamp":"2024-09-18T08:12:38.915501+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,1],"acting":[3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":628,"reported_epoch":340,"state":"active+clean","last_fresh":"2024-09-18T08:15:46.519551+0000","last_change":"2024-09-18T08:13:37.627262+0000","last_active":"2024-09-18T08:15:46.519551+0000","last_peered":"2024-09-18T08:15:46.519551+0000","last_clean":"2024-09-18T08:15:46.519551+0000","last_became_active":"2024-09-18T08:06:24.126194+0000","last_became_peered":"2024-09-18T08:06:24.126194+0000","last_unstale":"2024-09-18T08:15:46.519551+0000","last_undegraded":"2024-09-18T08:15:46.519551+0000","last_fullsized":"2024-09-18T08:15:46.519551+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:13:37.627195+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:46.860092+0000","last_clean_scrub_stamp":"2024-09-18T08:13:37.627195+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"271'27","reported_seq":628,"reported_epoch":340,"state":"active+clean","last_fresh":"2024-09-18T08:15:46.513251+0000","last_change":"2024-09-18T08:14:26.565324+0000","last_active":"2024-09-18T08:15:46.513251+0000","last_peered":"2024-09-18T08:15:46.513251+0000","last_clean":"2024-09-18T08:15:46.513251+0000","last_became_active":"2024-09-18T08:02:49.886106+0000","last_became_peered":"2024-09-18T08:02:49.886106+0000","last_unstale":"2024-09-18T08:15:46.513251+0000","last_undegraded":"2024-09-18T08:15:46.513251+0000","last_fullsized":"2024-09-18T08:15:46.513251+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"271'27","last_scrub_stamp":"2024-09-18T08:14:26.565071+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_clean_scrub_stamp":"2024-09-18T08:14:26.565071+0000","objects_scrubbed":7,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":44,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":350,"reported_epoch":340,"state":"active+clean","last_fresh":"2024-09-18T08:15:46.485157+0000","last_change":"2024-09-18T08:15:44.585165+0000","last_active":"2024-09-18T08:15:46.485157+0000","last_peered":"2024-09-18T08:15:46.485157+0000","last_clean":"2024-09-18T08:15:46.485157+0000","last_became_active":"2024-09-18T08:14:39.371563+0000","last_became_peered":"2024-09-18T08:14:39.371563+0000","last_unstale":"2024-09-18T08:15:46.485157+0000","last_undegraded":"2024-09-18T08:15:46.485157+0000","last_fullsized":"2024-09-18T08:15:46.485157+0000","mapping_epoch":307,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":308,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:15:44.585123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:50.886830+0000","last_clean_scrub_stamp":"2024-09-18T08:15:44.585123+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:16:58.747980+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":18,"reported_epoch":340,"state":"active+clean","last_fresh":"2024-09-18T08:15:46.512179+0000","last_change":"2024-09-18T08:15:41.450610+0000","last_active":"2024-09-18T08:15:46.512179+0000","last_peered":"2024-09-18T08:15:46.512179+0000","last_clean":"2024-09-18T08:15:46.512179+0000","last_became_active":"2024-09-18T08:15:41.446845+0000","last_became_peered":"2024-09-18T08:15:41.446845+0000","last_unstale":"2024-09-18T08:15:46.512179+0000","last_undegraded":"2024-09-18T08:15:46.512179+0000","last_fullsized":"2024-09-18T08:15:46.512179+0000","mapping_epoch":336,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":337,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:10:51.225974+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:46.860092+0000","last_clean_scrub_stamp":"2024-09-18T08:10:51.225974+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"271'15","reported_seq":366,"reported_epoch":340,"state":"active+clean","last_fresh":"2024-09-18T08:15:46.485579+0000","last_change":"2024-09-18T08:12:00.466295+0000","last_active":"2024-09-18T08:15:46.485579+0000","last_peered":"2024-09-18T08:15:46.485579+0000","last_clean":"2024-09-18T08:15:46.485579+0000","last_became_active":"2024-09-18T08:08:18.550960+0000","last_became_peered":"2024-09-18T08:08:18.550960+0000","last_unstale":"2024-09-18T08:15:46.485579+0000","last_undegraded":"2024-09-18T08:15:46.485579+0000","last_fullsized":"2024-09-18T08:15:46.485579+0000","mapping_epoch":181,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":182,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'9","last_scrub_stamp":"2024-09-18T08:12:00.466202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_clean_scrub_stamp":"2024-09-18T08:12:00.466202+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":25,"num_read_kb":19,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":18,"reported_epoch":340,"state":"active+clean","last_fresh":"2024-09-18T08:15:46.512171+0000","last_change":"2024-09-18T08:15:41.455870+0000","last_active":"2024-09-18T08:15:46.512171+0000","last_peered":"2024-09-18T08:15:46.512171+0000","last_clean":"2024-09-18T08:15:46.512171+0000","last_became_active":"2024-09-18T08:15:41.453708+0000","last_became_peered":"2024-09-18T08:15:41.453708+0000","last_unstale":"2024-09-18T08:15:46.512171+0000","last_undegraded":"2024-09-18T08:15:46.512171+0000","last_fullsized":"2024-09-18T08:15:46.512171+0000","mapping_epoch":336,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":337,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:12:57.218286+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:50.886830+0000","last_clean_scrub_stamp":"2024-09-18T08:12:57.218286+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,4],"acting":[5,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":593,"reported_epoch":340,"state":"active+clean","last_fresh":"2024-09-18T08:15:46.507211+0000","last_change":"2024-09-18T08:14:40.356444+0000","last_active":"2024-09-18T08:15:46.507211+0000","last_peered":"2024-09-18T08:15:46.507211+0000","last_clean":"2024-09-18T08:15:46.507211+0000","last_became_active":"2024-09-18T08:14:40.354651+0000","last_became_peered":"2024-09-18T08:14:40.354651+0000","last_unstale":"2024-09-18T08:15:46.507211+0000","last_undegraded":"2024-09-18T08:15:46.507211+0000","last_fullsized":"2024-09-18T08:15:46.507211+0000","mapping_epoch":308,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":309,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:08:14.645993+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_clean_scrub_stamp":"2024-09-18T08:08:14.645993+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":18,"reported_epoch":340,"state":"active+clean","last_fresh":"2024-09-18T08:15:46.512361+0000","last_change":"2024-09-18T08:15:41.445550+0000","last_active":"2024-09-18T08:15:46.512361+0000","last_peered":"2024-09-18T08:15:46.512361+0000","last_clean":"2024-09-18T08:15:46.512361+0000","last_became_active":"2024-09-18T08:15:41.444994+0000","last_became_peered":"2024-09-18T08:15:41.444994+0000","last_unstale":"2024-09-18T08:15:46.512361+0000","last_undegraded":"2024-09-18T08:15:46.512361+0000","last_fullsized":"2024-09-18T08:15:46.512361+0000","mapping_epoch":336,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":337,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:15:01.636504+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:46.860092+0000","last_clean_scrub_stamp":"2024-09-18T08:15:01.636504+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:16:29.683472+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1b","version":"271'7","reported_seq":372,"reported_epoch":340,"state":"active+clean","last_fresh":"2024-09-18T08:15:46.486543+0000","last_change":"2024-09-18T08:15:13.551423+0000","last_active":"2024-09-18T08:15:46.486543+0000","last_peered":"2024-09-18T08:15:46.486543+0000","last_clean":"2024-09-18T08:15:46.486543+0000","last_became_active":"2024-09-18T08:14:39.368637+0000","last_became_peered":"2024-09-18T08:14:39.368637+0000","last_unstale":"2024-09-18T08:15:46.486543+0000","last_undegraded":"2024-09-18T08:15:46.486543+0000","last_fullsized":"2024-09-18T08:15:46.486543+0000","mapping_epoch":307,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":308,"parent":"0.0","parent_split_bits":0,"last_scrub":"271'7","last_scrub_stamp":"2024-09-18T08:15:13.551351+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_clean_scrub_stamp":"2024-09-18T08:15:13.551351+0000","objects_scrubbed":2,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:16:37.311690+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":13,"num_read_kb":9,"num_write":8,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":551,"reported_epoch":340,"state":"active+clean","last_fresh":"2024-09-18T08:15:46.484851+0000","last_change":"2024-09-18T08:15:40.589093+0000","last_active":"2024-09-18T08:15:46.484851+0000","last_peered":"2024-09-18T08:15:46.484851+0000","last_clean":"2024-09-18T08:15:46.484851+0000","last_became_active":"2024-09-18T08:14:40.353507+0000","last_became_peered":"2024-09-18T08:14:40.353507+0000","last_unstale":"2024-09-18T08:15:46.484851+0000","last_undegraded":"2024-09-18T08:15:46.484851+0000","last_fullsized":"2024-09-18T08:15:46.484851+0000","mapping_epoch":308,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":309,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:15:40.589048+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:50.886830+0000","last_clean_scrub_stamp":"2024-09-18T08:15:40.589048+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:16:43.761662+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,0],"acting":[4,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":18,"reported_epoch":340,"state":"active+clean","last_fresh":"2024-09-18T08:15:46.512536+0000","last_change":"2024-09-18T08:15:41.444316+0000","last_active":"2024-09-18T08:15:46.512536+0000","last_peered":"2024-09-18T08:15:46.512536+0000","last_clean":"2024-09-18T08:15:46.512536+0000","last_became_active":"2024-09-18T08:15:41.441932+0000","last_became_peered":"2024-09-18T08:15:41.441932+0000","last_unstale":"2024-09-18T08:15:46.512536+0000","last_undegraded":"2024-09-18T08:15:46.512536+0000","last_fullsized":"2024-09-18T08:15:46.512536+0000","mapping_epoch":336,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":337,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:11:01.108648+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_clean_scrub_stamp":"2024-09-18T08:11:01.108648+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":361,"reported_epoch":340,"state":"active+clean","last_fresh":"2024-09-18T08:15:46.488992+0000","last_change":"2024-09-18T08:15:42.600909+0000","last_active":"2024-09-18T08:15:46.488992+0000","last_peered":"2024-09-18T08:15:46.488992+0000","last_clean":"2024-09-18T08:15:46.488992+0000","last_became_active":"2024-09-18T08:15:41.442994+0000","last_became_peered":"2024-09-18T08:15:41.442994+0000","last_unstale":"2024-09-18T08:15:46.488992+0000","last_undegraded":"2024-09-18T08:15:46.488992+0000","last_fullsized":"2024-09-18T08:15:46.488992+0000","mapping_epoch":336,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":337,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:15:42.600840+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_clean_scrub_stamp":"2024-09-18T08:15:42.600840+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:17:07.473295+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":622,"reported_epoch":340,"state":"active+clean","last_fresh":"2024-09-18T08:15:46.509075+0000","last_change":"2024-09-18T08:14:40.354122+0000","last_active":"2024-09-18T08:15:46.509075+0000","last_peered":"2024-09-18T08:15:46.509075+0000","last_clean":"2024-09-18T08:15:46.509075+0000","last_became_active":"2024-09-18T08:14:40.353509+0000","last_became_peered":"2024-09-18T08:14:40.353509+0000","last_unstale":"2024-09-18T08:15:46.509075+0000","last_undegraded":"2024-09-18T08:15:46.509075+0000","last_fullsized":"2024-09-18T08:15:46.509075+0000","mapping_epoch":308,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":309,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:12:40.711507+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:46.860092+0000","last_clean_scrub_stamp":"2024-09-18T08:12:40.711507+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:16:19.287139+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,1],"acting":[0,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1c","version":"271'27","reported_seq":364,"reported_epoch":340,"state":"active+clean","last_fresh":"2024-09-18T08:15:46.514733+0000","last_change":"2024-09-18T08:14:40.711299+0000","last_active":"2024-09-18T08:15:46.514733+0000","last_peered":"2024-09-18T08:15:46.514733+0000","last_clean":"2024-09-18T08:15:46.514733+0000","last_became_active":"2024-09-18T08:14:39.480913+0000","last_became_peered":"2024-09-18T08:14:39.480913+0000","last_unstale":"2024-09-18T08:15:46.514733+0000","last_undegraded":"2024-09-18T08:15:46.514733+0000","last_fullsized":"2024-09-18T08:15:46.514733+0000","mapping_epoch":307,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":308,"parent":"0.0","parent_split_bits":0,"last_scrub":"271'27","last_scrub_stamp":"2024-09-18T08:14:40.711228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_clean_scrub_stamp":"2024-09-18T08:14:40.711228+0000","objects_scrubbed":7,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:16:07.663050+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":44,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":471,"reported_epoch":340,"state":"active+clean","last_fresh":"2024-09-18T08:15:46.502146+0000","last_change":"2024-09-18T08:14:39.389703+0000","last_active":"2024-09-18T08:15:46.502146+0000","last_peered":"2024-09-18T08:15:46.502146+0000","last_clean":"2024-09-18T08:15:46.502146+0000","last_became_active":"2024-09-18T08:14:39.389465+0000","last_became_peered":"2024-09-18T08:14:39.389465+0000","last_unstale":"2024-09-18T08:15:46.502146+0000","last_undegraded":"2024-09-18T08:15:46.502146+0000","last_fullsized":"2024-09-18T08:15:46.502146+0000","mapping_epoch":307,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":308,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:10:59.645112+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:50.886830+0000","last_clean_scrub_stamp":"2024-09-18T08:10:59.645112+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:16:34.272900+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":614,"reported_epoch":340,"state":"active+clean","last_fresh":"2024-09-18T08:15:46.513294+0000","last_change":"2024-09-18T08:15:41.469278+0000","last_active":"2024-09-18T08:15:46.513294+0000","last_peered":"2024-09-18T08:15:46.513294+0000","last_clean":"2024-09-18T08:15:46.513294+0000","last_became_active":"2024-09-18T08:15:41.466371+0000","last_became_peered":"2024-09-18T08:15:41.466371+0000","last_unstale":"2024-09-18T08:15:46.513294+0000","last_undegraded":"2024-09-18T08:15:46.513294+0000","last_fullsized":"2024-09-18T08:15:46.513294+0000","mapping_epoch":336,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":337,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:15:05.719269+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_clean_scrub_stamp":"2024-09-18T08:15:05.719269+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:16:11.199075+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":350,"reported_epoch":340,"state":"active+clean","last_fresh":"2024-09-18T08:15:46.485089+0000","last_change":"2024-09-18T08:14:40.774144+0000","last_active":"2024-09-18T08:15:46.485089+0000","last_peered":"2024-09-18T08:15:46.485089+0000","last_clean":"2024-09-18T08:15:46.485089+0000","last_became_active":"2024-09-18T08:14:39.376194+0000","last_became_peered":"2024-09-18T08:14:39.376194+0000","last_unstale":"2024-09-18T08:15:46.485089+0000","last_undegraded":"2024-09-18T08:15:46.485089+0000","last_fullsized":"2024-09-18T08:15:46.485089+0000","mapping_epoch":307,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":308,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:14:40.774086+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:46.860092+0000","last_clean_scrub_stamp":"2024-09-18T08:14:40.774086+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"271'18","reported_seq":691,"reported_epoch":340,"state":"active+clean","last_fresh":"2024-09-18T08:15:46.523484+0000","last_change":"2024-09-18T08:14:39.453388+0000","last_active":"2024-09-18T08:15:46.523484+0000","last_peered":"2024-09-18T08:15:46.523484+0000","last_clean":"2024-09-18T08:15:46.523484+0000","last_became_active":"2024-09-18T08:14:39.381418+0000","last_became_peered":"2024-09-18T08:14:39.381418+0000","last_unstale":"2024-09-18T08:15:46.523484+0000","last_undegraded":"2024-09-18T08:15:46.523484+0000","last_fullsized":"2024-09-18T08:15:46.523484+0000","mapping_epoch":307,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":308,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'12","last_scrub_stamp":"2024-09-18T08:11:32.304472+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_clean_scrub_stamp":"2024-09-18T08:11:32.304472+0000","objects_scrubbed":5,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:16:28.876460+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":34,"num_read_kb":25,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":339,"reported_epoch":340,"state":"active+clean","last_fresh":"2024-09-18T08:15:46.485555+0000","last_change":"2024-09-18T08:10:46.303912+0000","last_active":"2024-09-18T08:15:46.485555+0000","last_peered":"2024-09-18T08:15:46.485555+0000","last_clean":"2024-09-18T08:15:46.485555+0000","last_became_active":"2024-09-18T08:08:18.531438+0000","last_became_peered":"2024-09-18T08:08:18.531438+0000","last_unstale":"2024-09-18T08:15:46.485555+0000","last_undegraded":"2024-09-18T08:15:46.485555+0000","last_fullsized":"2024-09-18T08:15:46.485555+0000","mapping_epoch":181,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":182,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:10:46.303848+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:50.886830+0000","last_clean_scrub_stamp":"2024-09-18T08:10:46.303848+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:16:25.558230+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1c","version":"69'1","reported_seq":22,"reported_epoch":340,"state":"active+clean","last_fresh":"2024-09-18T08:15:46.512460+0000","last_change":"2024-09-18T08:15:41.489871+0000","last_active":"2024-09-18T08:15:46.512460+0000","last_peered":"2024-09-18T08:15:46.512460+0000","last_clean":"2024-09-18T08:15:46.512460+0000","last_became_active":"2024-09-18T08:15:41.466665+0000","last_became_peered":"2024-09-18T08:15:41.466665+0000","last_unstale":"2024-09-18T08:15:46.512460+0000","last_undegraded":"2024-09-18T08:15:46.512460+0000","last_fullsized":"2024-09-18T08:15:46.512460+0000","mapping_epoch":336,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":337,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'1","last_scrub_stamp":"2024-09-18T08:15:06.708980+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_clean_scrub_stamp":"2024-09-18T08:15:06.708980+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:16:24.251484+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":403,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":806,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,3],"acting":[5,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"62'1","reported_seq":480,"reported_epoch":340,"state":"active+clean","last_fresh":"2024-09-18T08:15:46.495073+0000","last_change":"2024-09-18T08:14:39.414662+0000","last_active":"2024-09-18T08:15:46.495073+0000","last_peered":"2024-09-18T08:15:46.495073+0000","last_clean":"2024-09-18T08:15:46.495073+0000","last_became_active":"2024-09-18T08:14:39.414309+0000","last_became_peered":"2024-09-18T08:14:39.414309+0000","last_unstale":"2024-09-18T08:15:46.495073+0000","last_undegraded":"2024-09-18T08:15:46.495073+0000","last_fullsized":"2024-09-18T08:15:46.495073+0000","mapping_epoch":307,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":308,"parent":"0.0","parent_split_bits":0,"last_scrub":"62'1","last_scrub_stamp":"2024-09-18T08:11:00.634190+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:46.860092+0000","last_clean_scrub_stamp":"2024-09-18T08:11:00.634190+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:16:38.249487+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1e","version":"271'20","reported_seq":647,"reported_epoch":340,"state":"active+clean","last_fresh":"2024-09-18T08:15:46.509089+0000","last_change":"2024-09-18T08:15:40.251794+0000","last_active":"2024-09-18T08:15:46.509089+0000","last_peered":"2024-09-18T08:15:46.509089+0000","last_clean":"2024-09-18T08:15:46.509089+0000","last_became_active":"2024-09-18T08:14:40.359879+0000","last_became_peered":"2024-09-18T08:14:40.359879+0000","last_unstale":"2024-09-18T08:15:46.509089+0000","last_undegraded":"2024-09-18T08:15:46.509089+0000","last_fullsized":"2024-09-18T08:15:46.509089+0000","mapping_epoch":308,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":309,"parent":"0.0","parent_split_bits":0,"last_scrub":"271'20","last_scrub_stamp":"2024-09-18T08:15:40.251706+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_clean_scrub_stamp":"2024-09-18T08:15:40.251706+0000","objects_scrubbed":5,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:17:02.901805+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":25,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1],"acting":[0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1f","version":"293'14","reported_seq":446,"reported_epoch":340,"state":"active+clean","last_fresh":"2024-09-18T08:15:46.485821+0000","last_change":"2024-09-18T08:14:05.592694+0000","last_active":"2024-09-18T08:15:46.485821+0000","last_peered":"2024-09-18T08:15:46.485821+0000","last_clean":"2024-09-18T08:15:46.485821+0000","last_became_active":"2024-09-18T08:14:05.581783+0000","last_became_peered":"2024-09-18T08:14:05.581783+0000","last_unstale":"2024-09-18T08:15:46.485821+0000","last_undegraded":"2024-09-18T08:15:46.485821+0000","last_fullsized":"2024-09-18T08:15:46.485821+0000","mapping_epoch":292,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":293,"parent":"0.0","parent_split_bits":0,"last_scrub":"182'13","last_scrub_stamp":"2024-09-18T08:12:59.403054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:50.886830+0000","last_clean_scrub_stamp":"2024-09-18T08:12:59.403054+0000","objects_scrubbed":1,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,2],"acting":[4,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.1","version":"340'4871","reported_seq":4940,"reported_epoch":340,"state":"active+clean","last_fresh":"2024-09-18T08:15:48.334262+0000","last_change":"2024-09-18T08:14:44.432369+0000","last_active":"2024-09-18T08:15:48.334262+0000","last_peered":"2024-09-18T08:15:48.334262+0000","last_clean":"2024-09-18T08:15:48.334262+0000","last_became_active":"2024-09-18T08:14:44.432020+0000","last_became_peered":"2024-09-18T08:14:44.432020+0000","last_unstale":"2024-09-18T08:15:48.334262+0000","last_undegraded":"2024-09-18T08:15:48.334262+0000","last_fullsized":"2024-09-18T08:15:48.334262+0000","mapping_epoch":311,"log_start":"321'900","ondisk_log_start":"321'900","created":311,"last_epoch_clean":312,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:14:43.412748+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:14:43.412748+0000","last_clean_scrub_stamp":"2024-09-18T08:14:43.412748+0000","objects_scrubbed":0,"log_size":3971,"log_dups_size":0,"ondisk_log_size":3971,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:15:57.575857+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":319225856,"num_objects":4871,"num_object_clones":0,"num_object_copies":14613,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4871,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4871,"num_write_kb":311744,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,1],"acting":[4,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.a","version":"74'79","reported_seq":26,"reported_epoch":340,"state":"active+clean","last_fresh":"2024-09-18T08:15:46.511986+0000","last_change":"2024-09-18T08:15:42.288426+0000","last_active":"2024-09-18T08:15:46.511986+0000","last_peered":"2024-09-18T08:15:46.511986+0000","last_clean":"2024-09-18T08:15:46.511986+0000","last_became_active":"2024-09-18T08:15:41.443376+0000","last_became_peered":"2024-09-18T08:15:41.443376+0000","last_unstale":"2024-09-18T08:15:46.511986+0000","last_undegraded":"2024-09-18T08:15:46.511986+0000","last_fullsized":"2024-09-18T08:15:46.511986+0000","mapping_epoch":336,"log_start":"0'0","ondisk_log_start":"0'0","created":152,"last_epoch_clean":337,"parent":"0.0","parent_split_bits":4,"last_scrub":"74'79","last_scrub_stamp":"2024-09-18T08:15:42.288363+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:00:11.893742+0000","last_clean_scrub_stamp":"2024-09-18T08:15:42.288363+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:16:43.198731+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,6],"acting":[5,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.f","version":"271'27","reported_seq":524,"reported_epoch":340,"state":"active+clean","last_fresh":"2024-09-18T08:15:46.502662+0000","last_change":"2024-09-18T08:14:39.499528+0000","last_active":"2024-09-18T08:15:46.502662+0000","last_peered":"2024-09-18T08:15:46.502662+0000","last_clean":"2024-09-18T08:15:46.502662+0000","last_became_active":"2024-09-18T08:14:39.400606+0000","last_became_peered":"2024-09-18T08:14:39.400606+0000","last_unstale":"2024-09-18T08:15:46.502662+0000","last_undegraded":"2024-09-18T08:15:46.502662+0000","last_fullsized":"2024-09-18T08:15:46.502662+0000","mapping_epoch":307,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":308,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'15","last_scrub_stamp":"2024-09-18T08:10:41.548188+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_clean_scrub_stamp":"2024-09-18T08:10:41.548188+0000","objects_scrubbed":0,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:16:20.097961+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":43,"num_read_kb":34,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":613,"reported_epoch":340,"state":"active+clean","last_fresh":"2024-09-18T08:15:46.809475+0000","last_change":"2024-09-18T08:15:46.809475+0000","last_active":"2024-09-18T08:15:46.809475+0000","last_peered":"2024-09-18T08:15:46.809475+0000","last_clean":"2024-09-18T08:15:46.809475+0000","last_became_active":"2024-09-18T08:14:40.354437+0000","last_became_peered":"2024-09-18T08:14:40.354437+0000","last_unstale":"2024-09-18T08:15:46.809475+0000","last_undegraded":"2024-09-18T08:15:46.809475+0000","last_fullsized":"2024-09-18T08:15:46.809475+0000","mapping_epoch":308,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":309,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:15:46.809403+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:46.860092+0000","last_clean_scrub_stamp":"2024-09-18T08:15:46.809403+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:16:58.744046+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.e","version":"338'15","reported_seq":465,"reported_epoch":340,"state":"active+clean","last_fresh":"2024-09-18T08:15:46.488946+0000","last_change":"2024-09-18T08:15:41.483990+0000","last_active":"2024-09-18T08:15:46.488946+0000","last_peered":"2024-09-18T08:15:46.488946+0000","last_clean":"2024-09-18T08:15:46.488946+0000","last_became_active":"2024-09-18T08:15:41.450846+0000","last_became_peered":"2024-09-18T08:15:41.450846+0000","last_unstale":"2024-09-18T08:15:46.488946+0000","last_undegraded":"2024-09-18T08:15:46.488946+0000","last_fullsized":"2024-09-18T08:15:46.488946+0000","mapping_epoch":336,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":337,"parent":"0.0","parent_split_bits":0,"last_scrub":"182'12","last_scrub_stamp":"2024-09-18T08:13:31.620205+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:50.886830+0000","last_clean_scrub_stamp":"2024-09-18T08:13:31.620205+0000","objects_scrubbed":1,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":18,"reported_epoch":340,"state":"active+clean","last_fresh":"2024-09-18T08:15:46.511730+0000","last_change":"2024-09-18T08:15:41.438999+0000","last_active":"2024-09-18T08:15:46.511730+0000","last_peered":"2024-09-18T08:15:46.511730+0000","last_clean":"2024-09-18T08:15:46.511730+0000","last_became_active":"2024-09-18T08:15:41.438610+0000","last_became_peered":"2024-09-18T08:15:41.438610+0000","last_unstale":"2024-09-18T08:15:46.511730+0000","last_undegraded":"2024-09-18T08:15:46.511730+0000","last_fullsized":"2024-09-18T08:15:46.511730+0000","mapping_epoch":336,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":337,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:15:00.180309+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_clean_scrub_stamp":"2024-09-18T08:15:00.180309+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:16:07.486152+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.e","version":"340'4756","reported_seq":4824,"reported_epoch":340,"state":"active+clean","last_fresh":"2024-09-18T08:15:48.379299+0000","last_change":"2024-09-18T08:14:44.435797+0000","last_active":"2024-09-18T08:15:48.379299+0000","last_peered":"2024-09-18T08:15:48.379299+0000","last_clean":"2024-09-18T08:15:48.379299+0000","last_became_active":"2024-09-18T08:14:44.435489+0000","last_became_peered":"2024-09-18T08:14:44.435489+0000","last_unstale":"2024-09-18T08:15:48.379299+0000","last_undegraded":"2024-09-18T08:15:48.379299+0000","last_fullsized":"2024-09-18T08:15:48.379299+0000","mapping_epoch":311,"log_start":"320'800","ondisk_log_start":"320'800","created":311,"last_epoch_clean":312,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:14:43.412748+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:14:43.412748+0000","last_clean_scrub_stamp":"2024-09-18T08:14:43.412748+0000","objects_scrubbed":0,"log_size":3956,"log_dups_size":0,"ondisk_log_size":3956,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:15:53.639006+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":311689216,"num_objects":4756,"num_object_clones":0,"num_object_copies":14268,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4756,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4756,"num_write_kb":304384,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,2],"acting":[4,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.5","version":"74'79","reported_seq":355,"reported_epoch":340,"state":"active+clean","last_fresh":"2024-09-18T08:15:46.485612+0000","last_change":"2024-09-18T08:11:54.503697+0000","last_active":"2024-09-18T08:15:46.485612+0000","last_peered":"2024-09-18T08:15:46.485612+0000","last_clean":"2024-09-18T08:15:46.485612+0000","last_became_active":"2024-09-18T08:08:52.690344+0000","last_became_peered":"2024-09-18T08:08:52.690344+0000","last_unstale":"2024-09-18T08:15:46.485612+0000","last_undegraded":"2024-09-18T08:15:46.485612+0000","last_fullsized":"2024-09-18T08:15:46.485612+0000","mapping_epoch":198,"log_start":"0'0","ondisk_log_start":"0'0","created":152,"last_epoch_clean":199,"parent":"0.0","parent_split_bits":3,"last_scrub":"74'79","last_scrub_stamp":"2024-09-18T08:11:54.503603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:00:11.893742+0000","last_clean_scrub_stamp":"2024-09-18T08:11:54.503603+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,6],"acting":[4,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.0","version":"271'30","reported_seq":682,"reported_epoch":340,"state":"active+clean","last_fresh":"2024-09-18T08:15:46.518364+0000","last_change":"2024-09-18T08:14:40.377265+0000","last_active":"2024-09-18T08:15:46.518364+0000","last_peered":"2024-09-18T08:15:46.518364+0000","last_clean":"2024-09-18T08:15:46.518364+0000","last_became_active":"2024-09-18T08:14:40.358956+0000","last_became_peered":"2024-09-18T08:14:40.358956+0000","last_unstale":"2024-09-18T08:15:46.518364+0000","last_undegraded":"2024-09-18T08:15:46.518364+0000","last_fullsized":"2024-09-18T08:15:46.518364+0000","mapping_epoch":308,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":309,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'18","last_scrub_stamp":"2024-09-18T08:12:36.956258+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_clean_scrub_stamp":"2024-09-18T08:12:36.956258+0000","objects_scrubbed":8,"log_size":30,"log_dups_size":0,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":52,"num_read_kb":40,"num_write":32,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":642,"reported_epoch":340,"state":"active+clean","last_fresh":"2024-09-18T08:15:46.513800+0000","last_change":"2024-09-18T08:15:41.461616+0000","last_active":"2024-09-18T08:15:46.513800+0000","last_peered":"2024-09-18T08:15:46.513800+0000","last_clean":"2024-09-18T08:15:46.513800+0000","last_became_active":"2024-09-18T08:15:41.461057+0000","last_became_peered":"2024-09-18T08:15:41.461057+0000","last_unstale":"2024-09-18T08:15:46.513800+0000","last_undegraded":"2024-09-18T08:15:46.513800+0000","last_fullsized":"2024-09-18T08:15:46.513800+0000","mapping_epoch":336,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":337,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:15:08.780913+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:46.860092+0000","last_clean_scrub_stamp":"2024-09-18T08:15:08.780913+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:16:34.295364+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":354,"reported_epoch":340,"state":"active+clean","last_fresh":"2024-09-18T08:15:46.486061+0000","last_change":"2024-09-18T08:14:05.588953+0000","last_active":"2024-09-18T08:15:46.486061+0000","last_peered":"2024-09-18T08:15:46.486061+0000","last_clean":"2024-09-18T08:15:46.486061+0000","last_became_active":"2024-09-18T08:14:05.588085+0000","last_became_peered":"2024-09-18T08:14:05.588085+0000","last_unstale":"2024-09-18T08:15:46.486061+0000","last_undegraded":"2024-09-18T08:15:46.486061+0000","last_fullsized":"2024-09-18T08:15:46.486061+0000","mapping_epoch":292,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":293,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:12:30.732539+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:50.886830+0000","last_clean_scrub_stamp":"2024-09-18T08:12:30.732539+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":347,"reported_epoch":340,"state":"active+clean","last_fresh":"2024-09-18T08:15:46.485408+0000","last_change":"2024-09-18T08:12:02.473229+0000","last_active":"2024-09-18T08:15:46.485408+0000","last_peered":"2024-09-18T08:15:46.485408+0000","last_clean":"2024-09-18T08:15:46.485408+0000","last_became_active":"2024-09-18T08:08:18.543694+0000","last_became_peered":"2024-09-18T08:08:18.543694+0000","last_unstale":"2024-09-18T08:15:46.485408+0000","last_undegraded":"2024-09-18T08:15:46.485408+0000","last_fullsized":"2024-09-18T08:15:46.485408+0000","mapping_epoch":181,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":182,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:12:02.473168+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_clean_scrub_stamp":"2024-09-18T08:12:02.473168+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.f","version":"340'4918","reported_seq":4987,"reported_epoch":340,"state":"active+clean","last_fresh":"2024-09-18T08:15:49.176972+0000","last_change":"2024-09-18T08:14:44.437378+0000","last_active":"2024-09-18T08:15:49.176972+0000","last_peered":"2024-09-18T08:15:49.176972+0000","last_clean":"2024-09-18T08:15:49.176972+0000","last_became_active":"2024-09-18T08:14:44.437103+0000","last_became_peered":"2024-09-18T08:14:44.437103+0000","last_unstale":"2024-09-18T08:15:49.176972+0000","last_undegraded":"2024-09-18T08:15:49.176972+0000","last_fullsized":"2024-09-18T08:15:49.176972+0000","mapping_epoch":311,"log_start":"320'900","ondisk_log_start":"320'900","created":311,"last_epoch_clean":312,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:14:43.412748+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:14:43.412748+0000","last_clean_scrub_stamp":"2024-09-18T08:14:43.412748+0000","objects_scrubbed":0,"log_size":4018,"log_dups_size":0,"ondisk_log_size":4018,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:16:00.051694+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":322306048,"num_objects":4918,"num_object_clones":0,"num_object_copies":14754,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4918,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4918,"num_write_kb":314752,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.4","version":"74'79","reported_seq":466,"reported_epoch":340,"state":"active+clean","last_fresh":"2024-09-18T08:15:46.531293+0000","last_change":"2024-09-18T08:15:41.898345+0000","last_active":"2024-09-18T08:15:46.531293+0000","last_peered":"2024-09-18T08:15:46.531293+0000","last_clean":"2024-09-18T08:15:46.531293+0000","last_became_active":"2024-09-18T08:15:41.464987+0000","last_became_peered":"2024-09-18T08:15:41.464987+0000","last_unstale":"2024-09-18T08:15:46.531293+0000","last_undegraded":"2024-09-18T08:15:46.531293+0000","last_fullsized":"2024-09-18T08:15:46.531293+0000","mapping_epoch":336,"log_start":"0'0","ondisk_log_start":"0'0","created":152,"last_epoch_clean":337,"parent":"0.0","parent_split_bits":3,"last_scrub":"74'79","last_scrub_stamp":"2024-09-18T08:15:41.898294+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:00:11.893742+0000","last_clean_scrub_stamp":"2024-09-18T08:15:41.898294+0000","objects_scrubbed":1,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:17:06.273734+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":327680,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":983040,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"271'28","reported_seq":422,"reported_epoch":340,"state":"active+clean","last_fresh":"2024-09-18T08:15:46.485957+0000","last_change":"2024-09-18T08:15:41.482295+0000","last_active":"2024-09-18T08:15:46.485957+0000","last_peered":"2024-09-18T08:15:46.485957+0000","last_clean":"2024-09-18T08:15:46.485957+0000","last_became_active":"2024-09-18T08:15:41.442693+0000","last_became_peered":"2024-09-18T08:15:41.442693+0000","last_unstale":"2024-09-18T08:15:46.485957+0000","last_undegraded":"2024-09-18T08:15:46.485957+0000","last_fullsized":"2024-09-18T08:15:46.485957+0000","mapping_epoch":336,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":337,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'14","last_scrub_stamp":"2024-09-18T08:11:53.524045+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_clean_scrub_stamp":"2024-09-18T08:11:53.524045+0000","objects_scrubbed":7,"log_size":28,"log_dups_size":0,"ondisk_log_size":28,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":41,"num_read_kb":34,"num_write":28,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"62'2","reported_seq":641,"reported_epoch":340,"state":"active+clean","last_fresh":"2024-09-18T08:15:46.509348+0000","last_change":"2024-09-18T08:14:42.145896+0000","last_active":"2024-09-18T08:15:46.509348+0000","last_peered":"2024-09-18T08:15:46.509348+0000","last_clean":"2024-09-18T08:15:46.509348+0000","last_became_active":"2024-09-18T08:14:39.452371+0000","last_became_peered":"2024-09-18T08:14:39.452371+0000","last_unstale":"2024-09-18T08:15:46.509348+0000","last_undegraded":"2024-09-18T08:15:46.509348+0000","last_fullsized":"2024-09-18T08:15:46.509348+0000","mapping_epoch":307,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":308,"parent":"0.0","parent_split_bits":0,"last_scrub":"62'2","last_scrub_stamp":"2024-09-18T08:14:42.145836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:46.860092+0000","last_clean_scrub_stamp":"2024-09-18T08:14:42.145836+0000","objects_scrubbed":2,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1039,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":2078,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.0","version":"310'12","reported_seq":782,"reported_epoch":340,"state":"active+clean","last_fresh":"2024-09-18T08:15:46.518221+0000","last_change":"2024-09-18T08:14:39.404911+0000","last_active":"2024-09-18T08:15:46.518221+0000","last_peered":"2024-09-18T08:15:46.518221+0000","last_clean":"2024-09-18T08:15:46.518221+0000","last_became_active":"2024-09-18T08:14:39.387233+0000","last_became_peered":"2024-09-18T08:14:39.387233+0000","last_unstale":"2024-09-18T08:15:46.518221+0000","last_undegraded":"2024-09-18T08:15:46.518221+0000","last_fullsized":"2024-09-18T08:15:46.518221+0000","mapping_epoch":307,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":308,"parent":"0.0","parent_split_bits":0,"last_scrub":"182'10","last_scrub_stamp":"2024-09-18T08:09:51.467349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:50.886830+0000","last_clean_scrub_stamp":"2024-09-18T08:09:51.467349+0000","objects_scrubbed":1,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":19,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":18,"reported_epoch":340,"state":"active+clean","last_fresh":"2024-09-18T08:15:46.512073+0000","last_change":"2024-09-18T08:15:41.441653+0000","last_active":"2024-09-18T08:15:46.512073+0000","last_peered":"2024-09-18T08:15:46.512073+0000","last_clean":"2024-09-18T08:15:46.512073+0000","last_became_active":"2024-09-18T08:15:41.440876+0000","last_became_peered":"2024-09-18T08:15:41.440876+0000","last_unstale":"2024-09-18T08:15:46.512073+0000","last_undegraded":"2024-09-18T08:15:46.512073+0000","last_fullsized":"2024-09-18T08:15:46.512073+0000","mapping_epoch":336,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":337,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:15:14.781554+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_clean_scrub_stamp":"2024-09-18T08:15:14.781554+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:16:41.378995+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.c","version":"340'4773","reported_seq":4842,"reported_epoch":340,"state":"active+clean","last_fresh":"2024-09-18T08:15:48.390226+0000","last_change":"2024-09-18T08:14:44.436817+0000","last_active":"2024-09-18T08:15:48.390226+0000","last_peered":"2024-09-18T08:15:48.390226+0000","last_clean":"2024-09-18T08:15:48.390226+0000","last_became_active":"2024-09-18T08:14:44.436502+0000","last_became_peered":"2024-09-18T08:14:44.436502+0000","last_unstale":"2024-09-18T08:15:48.390226+0000","last_undegraded":"2024-09-18T08:15:48.390226+0000","last_fullsized":"2024-09-18T08:15:48.390226+0000","mapping_epoch":311,"log_start":"320'800","ondisk_log_start":"320'800","created":311,"last_epoch_clean":312,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:14:43.412748+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:14:43.412748+0000","last_clean_scrub_stamp":"2024-09-18T08:14:43.412748+0000","objects_scrubbed":0,"log_size":3973,"log_dups_size":0,"ondisk_log_size":3973,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":312803328,"num_objects":4773,"num_object_clones":0,"num_object_copies":14319,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4773,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4773,"num_write_kb":305472,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,3],"acting":[4,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.7","version":"74'79","reported_seq":448,"reported_epoch":340,"state":"active+clean","last_fresh":"2024-09-18T08:15:46.513633+0000","last_change":"2024-09-18T08:15:44.872544+0000","last_active":"2024-09-18T08:15:46.513633+0000","last_peered":"2024-09-18T08:15:46.513633+0000","last_clean":"2024-09-18T08:15:46.513633+0000","last_became_active":"2024-09-18T08:15:41.466505+0000","last_became_peered":"2024-09-18T08:15:41.466505+0000","last_unstale":"2024-09-18T08:15:46.513633+0000","last_undegraded":"2024-09-18T08:15:46.513633+0000","last_fullsized":"2024-09-18T08:15:46.513633+0000","mapping_epoch":336,"log_start":"0'0","ondisk_log_start":"0'0","created":152,"last_epoch_clean":337,"parent":"0.0","parent_split_bits":3,"last_scrub":"74'79","last_scrub_stamp":"2024-09-18T08:15:44.872332+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:00:11.893742+0000","last_clean_scrub_stamp":"2024-09-18T08:15:44.872332+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:16:57.125165+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.2","version":"271'20","reported_seq":591,"reported_epoch":340,"state":"active+clean","last_fresh":"2024-09-18T08:15:46.493237+0000","last_change":"2024-09-18T08:15:41.480265+0000","last_active":"2024-09-18T08:15:46.493237+0000","last_peered":"2024-09-18T08:15:46.493237+0000","last_clean":"2024-09-18T08:15:46.493237+0000","last_became_active":"2024-09-18T08:15:41.439413+0000","last_became_peered":"2024-09-18T08:15:41.439413+0000","last_unstale":"2024-09-18T08:15:46.493237+0000","last_undegraded":"2024-09-18T08:15:46.493237+0000","last_fullsized":"2024-09-18T08:15:46.493237+0000","mapping_epoch":336,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":337,"parent":"0.0","parent_split_bits":0,"last_scrub":"271'20","last_scrub_stamp":"2024-09-18T08:15:40.251008+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_clean_scrub_stamp":"2024-09-18T08:15:40.251008+0000","objects_scrubbed":5,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:17:05.726664+0000","scrub_duration":23,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":28,"num_read_kb":23,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":18,"reported_epoch":340,"state":"active+clean","last_fresh":"2024-09-18T08:15:46.512354+0000","last_change":"2024-09-18T08:15:41.452981+0000","last_active":"2024-09-18T08:15:46.512354+0000","last_peered":"2024-09-18T08:15:46.512354+0000","last_clean":"2024-09-18T08:15:46.512354+0000","last_became_active":"2024-09-18T08:15:41.449639+0000","last_became_peered":"2024-09-18T08:15:41.449639+0000","last_unstale":"2024-09-18T08:15:46.512354+0000","last_undegraded":"2024-09-18T08:15:46.512354+0000","last_fullsized":"2024-09-18T08:15:46.512354+0000","mapping_epoch":336,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":337,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:15:39.858184+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:46.860092+0000","last_clean_scrub_stamp":"2024-09-18T08:15:39.858184+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:16:40.848073+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.3","version":"338'14","reported_seq":801,"reported_epoch":340,"state":"active+clean","last_fresh":"2024-09-18T08:15:46.508677+0000","last_change":"2024-09-18T08:15:41.467411+0000","last_active":"2024-09-18T08:15:46.508677+0000","last_peered":"2024-09-18T08:15:46.508677+0000","last_clean":"2024-09-18T08:15:46.508677+0000","last_became_active":"2024-09-18T08:15:41.447667+0000","last_became_peered":"2024-09-18T08:15:41.447667+0000","last_unstale":"2024-09-18T08:15:46.508677+0000","last_undegraded":"2024-09-18T08:15:46.508677+0000","last_fullsized":"2024-09-18T08:15:46.508677+0000","mapping_epoch":336,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":337,"parent":"0.0","parent_split_bits":0,"last_scrub":"317'11","last_scrub_stamp":"2024-09-18T08:14:59.207667+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:50.886830+0000","last_clean_scrub_stamp":"2024-09-18T08:14:59.207667+0000","objects_scrubbed":1,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:16:18.555373+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":616,"reported_epoch":340,"state":"active+clean","last_fresh":"2024-09-18T08:15:46.507330+0000","last_change":"2024-09-18T08:14:26.087160+0000","last_active":"2024-09-18T08:15:46.507330+0000","last_peered":"2024-09-18T08:15:46.507330+0000","last_clean":"2024-09-18T08:15:46.507330+0000","last_became_active":"2024-09-18T08:08:52.700080+0000","last_became_peered":"2024-09-18T08:08:52.700080+0000","last_unstale":"2024-09-18T08:15:46.507330+0000","last_undegraded":"2024-09-18T08:15:46.507330+0000","last_fullsized":"2024-09-18T08:15:46.507330+0000","mapping_epoch":198,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":199,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:14:26.087098+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_clean_scrub_stamp":"2024-09-18T08:14:26.087098+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.d","version":"340'4696","reported_seq":4764,"reported_epoch":340,"state":"active+clean","last_fresh":"2024-09-18T08:15:48.389892+0000","last_change":"2024-09-18T08:14:44.430244+0000","last_active":"2024-09-18T08:15:48.389892+0000","last_peered":"2024-09-18T08:15:48.389892+0000","last_clean":"2024-09-18T08:15:48.389892+0000","last_became_active":"2024-09-18T08:14:44.429930+0000","last_became_peered":"2024-09-18T08:14:44.429930+0000","last_unstale":"2024-09-18T08:15:48.389892+0000","last_undegraded":"2024-09-18T08:15:48.389892+0000","last_fullsized":"2024-09-18T08:15:48.389892+0000","mapping_epoch":311,"log_start":"319'700","ondisk_log_start":"319'700","created":311,"last_epoch_clean":312,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:14:43.412748+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:14:43.412748+0000","last_clean_scrub_stamp":"2024-09-18T08:14:43.412748+0000","objects_scrubbed":0,"log_size":3996,"log_dups_size":0,"ondisk_log_size":3996,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:16:11.045324+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":307757056,"num_objects":4696,"num_object_clones":0,"num_object_copies":14088,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4696,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4696,"num_write_kb":300544,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,0],"acting":[4,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.6","version":"74'79","reported_seq":507,"reported_epoch":340,"state":"active+clean","last_fresh":"2024-09-18T08:15:46.508662+0000","last_change":"2024-09-18T08:14:02.837236+0000","last_active":"2024-09-18T08:15:46.508662+0000","last_peered":"2024-09-18T08:15:46.508662+0000","last_clean":"2024-09-18T08:15:46.508662+0000","last_became_active":"2024-09-18T08:07:26.426532+0000","last_became_peered":"2024-09-18T08:07:26.426532+0000","last_unstale":"2024-09-18T08:15:46.508662+0000","last_undegraded":"2024-09-18T08:15:46.508662+0000","last_fullsized":"2024-09-18T08:15:46.508662+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":152,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":3,"last_scrub":"74'79","last_scrub_stamp":"2024-09-18T08:14:02.837167+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:00:11.893742+0000","last_clean_scrub_stamp":"2024-09-18T08:14:02.837167+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,3],"acting":[0,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.3","version":"271'29","reported_seq":724,"reported_epoch":340,"state":"active+clean","last_fresh":"2024-09-18T08:15:46.508826+0000","last_change":"2024-09-18T08:15:41.493514+0000","last_active":"2024-09-18T08:15:46.508826+0000","last_peered":"2024-09-18T08:15:46.508826+0000","last_clean":"2024-09-18T08:15:46.508826+0000","last_became_active":"2024-09-18T08:15:41.451985+0000","last_became_peered":"2024-09-18T08:15:41.451985+0000","last_unstale":"2024-09-18T08:15:46.508826+0000","last_undegraded":"2024-09-18T08:15:46.508826+0000","last_fullsized":"2024-09-18T08:15:46.508826+0000","mapping_epoch":336,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":337,"parent":"0.0","parent_split_bits":0,"last_scrub":"271'29","last_scrub_stamp":"2024-09-18T08:13:33.814992+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_clean_scrub_stamp":"2024-09-18T08:13:33.814992+0000","objects_scrubbed":8,"log_size":29,"log_dups_size":0,"ondisk_log_size":29,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":54,"num_read_kb":40,"num_write":32,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":990,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.4","version":"62'1","reported_seq":483,"reported_epoch":340,"state":"active+clean","last_fresh":"2024-09-18T08:15:46.503158+0000","last_change":"2024-09-18T08:15:41.454692+0000","last_active":"2024-09-18T08:15:46.503158+0000","last_peered":"2024-09-18T08:15:46.503158+0000","last_clean":"2024-09-18T08:15:46.503158+0000","last_became_active":"2024-09-18T08:15:41.441822+0000","last_became_peered":"2024-09-18T08:15:41.441822+0000","last_unstale":"2024-09-18T08:15:46.503158+0000","last_undegraded":"2024-09-18T08:15:46.503158+0000","last_fullsized":"2024-09-18T08:15:46.503158+0000","mapping_epoch":336,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":337,"parent":"0.0","parent_split_bits":0,"last_scrub":"62'1","last_scrub_stamp":"2024-09-18T08:15:13.165826+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:46.860092+0000","last_clean_scrub_stamp":"2024-09-18T08:15:13.165826+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:16:27.292767+0000","scrub_duration":17,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":436,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":1308,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":529,"reported_epoch":340,"state":"active+clean","last_fresh":"2024-09-18T08:15:46.507924+0000","last_change":"2024-09-18T08:15:41.450294+0000","last_active":"2024-09-18T08:15:46.507924+0000","last_peered":"2024-09-18T08:15:46.507924+0000","last_clean":"2024-09-18T08:15:46.507924+0000","last_became_active":"2024-09-18T08:15:41.449832+0000","last_became_peered":"2024-09-18T08:15:41.449832+0000","last_unstale":"2024-09-18T08:15:46.507924+0000","last_undegraded":"2024-09-18T08:15:46.507924+0000","last_fullsized":"2024-09-18T08:15:46.507924+0000","mapping_epoch":336,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":337,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:12:37.732260+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:50.886830+0000","last_clean_scrub_stamp":"2024-09-18T08:12:37.732260+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":494,"reported_epoch":340,"state":"active+clean","last_fresh":"2024-09-18T08:15:46.494793+0000","last_change":"2024-09-18T08:14:39.358424+0000","last_active":"2024-09-18T08:15:46.494793+0000","last_peered":"2024-09-18T08:15:46.494793+0000","last_clean":"2024-09-18T08:15:46.494793+0000","last_became_active":"2024-09-18T08:14:39.358132+0000","last_became_peered":"2024-09-18T08:14:39.358132+0000","last_unstale":"2024-09-18T08:15:46.494793+0000","last_undegraded":"2024-09-18T08:15:46.494793+0000","last_fullsized":"2024-09-18T08:15:46.494793+0000","mapping_epoch":307,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":308,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:11:28.346395+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_clean_scrub_stamp":"2024-09-18T08:11:28.346395+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.a","version":"340'4836","reported_seq":7686,"reported_epoch":340,"state":"active+remapped+backfilling","last_fresh":"2024-09-18T08:15:49.345238+0000","last_change":"2024-09-18T08:15:42.582806+0000","last_active":"2024-09-18T08:15:49.345238+0000","last_peered":"2024-09-18T08:15:49.345238+0000","last_clean":"2024-09-18T08:15:40.417711+0000","last_became_active":"2024-09-18T08:15:42.567231+0000","last_became_peered":"2024-09-18T08:15:42.567231+0000","last_unstale":"2024-09-18T08:15:49.345238+0000","last_undegraded":"2024-09-18T08:15:49.345238+0000","last_fullsized":"2024-09-18T08:15:49.345238+0000","mapping_epoch":337,"log_start":"319'700","ondisk_log_start":"319'700","created":311,"last_epoch_clean":324,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:14:43.412748+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:14:43.412748+0000","last_clean_scrub_stamp":"2024-09-18T08:14:43.412748+0000","objects_scrubbed":0,"log_size":4136,"log_dups_size":0,"ondisk_log_size":4136,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":316932096,"num_objects":4836,"num_object_clones":0,"num_object_copies":14508,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":4001,"num_objects_unfound":0,"num_objects_dirty":4836,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4836,"num_write_kb":309504,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1383,"num_bytes_recovered":90636288,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,2],"avail_no_missing":["0","2","6"],"object_location_counts":[{"shards":"0,2,6","objects":4836}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"74'79","reported_seq":26,"reported_epoch":340,"state":"active+clean","last_fresh":"2024-09-18T08:15:46.511643+0000","last_change":"2024-09-18T08:15:43.275398+0000","last_active":"2024-09-18T08:15:46.511643+0000","last_peered":"2024-09-18T08:15:46.511643+0000","last_clean":"2024-09-18T08:15:46.511643+0000","last_became_active":"2024-09-18T08:15:41.452895+0000","last_became_peered":"2024-09-18T08:15:41.452895+0000","last_unstale":"2024-09-18T08:15:46.511643+0000","last_undegraded":"2024-09-18T08:15:46.511643+0000","last_fullsized":"2024-09-18T08:15:46.511643+0000","mapping_epoch":336,"log_start":"0'0","ondisk_log_start":"0'0","created":152,"last_epoch_clean":337,"parent":"0.0","parent_split_bits":4,"last_scrub":"74'79","last_scrub_stamp":"2024-09-18T08:15:43.275329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:00:11.893742+0000","last_clean_scrub_stamp":"2024-09-18T08:15:43.275329+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:17:09.642634+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,2],"acting":[5,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.4","version":"310'42","reported_seq":610,"reported_epoch":340,"state":"active+clean","last_fresh":"2024-09-18T08:15:46.503298+0000","last_change":"2024-09-18T08:14:39.529072+0000","last_active":"2024-09-18T08:15:46.503298+0000","last_peered":"2024-09-18T08:15:46.503298+0000","last_clean":"2024-09-18T08:15:46.503298+0000","last_became_active":"2024-09-18T08:14:39.486174+0000","last_became_peered":"2024-09-18T08:14:39.486174+0000","last_unstale":"2024-09-18T08:15:46.503298+0000","last_undegraded":"2024-09-18T08:15:46.503298+0000","last_fullsized":"2024-09-18T08:15:46.503298+0000","mapping_epoch":307,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":308,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'28","last_scrub_stamp":"2024-09-18T08:10:46.497874+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_clean_scrub_stamp":"2024-09-18T08:10:46.497874+0000","objects_scrubbed":0,"log_size":42,"log_dups_size":0,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:16:22.143132+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":65,"num_read_kb":50,"num_write":38,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":339,"reported_epoch":340,"state":"active+clean","last_fresh":"2024-09-18T08:15:46.486551+0000","last_change":"2024-09-18T08:13:52.760213+0000","last_active":"2024-09-18T08:15:46.486551+0000","last_peered":"2024-09-18T08:15:46.486551+0000","last_clean":"2024-09-18T08:15:46.486551+0000","last_became_active":"2024-09-18T08:08:18.538852+0000","last_became_peered":"2024-09-18T08:08:18.538852+0000","last_unstale":"2024-09-18T08:15:46.486551+0000","last_undegraded":"2024-09-18T08:15:46.486551+0000","last_fullsized":"2024-09-18T08:15:46.486551+0000","mapping_epoch":181,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":182,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:13:52.760146+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:46.860092+0000","last_clean_scrub_stamp":"2024-09-18T08:13:52.760146+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"64'2","reported_seq":22,"reported_epoch":340,"state":"active+clean","last_fresh":"2024-09-18T08:15:46.511856+0000","last_change":"2024-09-18T08:15:41.483311+0000","last_active":"2024-09-18T08:15:46.511856+0000","last_peered":"2024-09-18T08:15:46.511856+0000","last_clean":"2024-09-18T08:15:46.511856+0000","last_became_active":"2024-09-18T08:15:41.453351+0000","last_became_peered":"2024-09-18T08:15:41.453351+0000","last_unstale":"2024-09-18T08:15:46.511856+0000","last_undegraded":"2024-09-18T08:15:46.511856+0000","last_fullsized":"2024-09-18T08:15:46.511856+0000","mapping_epoch":336,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":337,"parent":"0.0","parent_split_bits":0,"last_scrub":"64'2","last_scrub_stamp":"2024-09-18T08:05:44.802942+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:45.855979+0000","last_clean_scrub_stamp":"2024-09-18T08:05:44.802942+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":614,"reported_epoch":340,"state":"active+clean","last_fresh":"2024-09-18T08:15:46.509364+0000","last_change":"2024-09-18T08:15:04.260609+0000","last_active":"2024-09-18T08:15:46.509364+0000","last_peered":"2024-09-18T08:15:46.509364+0000","last_clean":"2024-09-18T08:15:46.509364+0000","last_became_active":"2024-09-18T08:14:39.397429+0000","last_became_peered":"2024-09-18T08:14:39.397429+0000","last_unstale":"2024-09-18T08:15:46.509364+0000","last_undegraded":"2024-09-18T08:15:46.509364+0000","last_fullsized":"2024-09-18T08:15:46.509364+0000","mapping_epoch":307,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":308,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:15:04.260540+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:50.886830+0000","last_clean_scrub_stamp":"2024-09-18T08:15:04.260540+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:16:09.549342+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":615,"reported_epoch":340,"state":"active+clean","last_fresh":"2024-09-18T08:15:46.519293+0000","last_change":"2024-09-18T08:14:05.578066+0000","last_active":"2024-09-18T08:15:46.519293+0000","last_peered":"2024-09-18T08:15:46.519293+0000","last_clean":"2024-09-18T08:15:46.519293+0000","last_became_active":"2024-09-18T08:14:05.577377+0000","last_became_peered":"2024-09-18T08:14:05.577377+0000","last_unstale":"2024-09-18T08:15:46.519293+0000","last_undegraded":"2024-09-18T08:15:46.519293+0000","last_fullsized":"2024-09-18T08:15:46.519293+0000","mapping_epoch":292,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":293,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:13:36.653164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_clean_scrub_stamp":"2024-09-18T08:13:36.653164+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,6],"acting":[3,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.9","version":"340'4912","reported_seq":8330,"reported_epoch":340,"state":"active+remapped+backfilling","last_fresh":"2024-09-18T08:15:48.880382+0000","last_change":"2024-09-18T08:15:42.600499+0000","last_active":"2024-09-18T08:15:48.880382+0000","last_peered":"2024-09-18T08:15:48.880382+0000","last_clean":"2024-09-18T08:15:40.379727+0000","last_became_active":"2024-09-18T08:15:42.550774+0000","last_became_peered":"2024-09-18T08:15:42.550774+0000","last_unstale":"2024-09-18T08:15:48.880382+0000","last_undegraded":"2024-09-18T08:15:48.880382+0000","last_fullsized":"2024-09-18T08:15:48.880382+0000","mapping_epoch":337,"log_start":"315'300","ondisk_log_start":"315'300","created":311,"last_epoch_clean":325,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:14:43.412748+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:14:43.412748+0000","last_clean_scrub_stamp":"2024-09-18T08:14:43.412748+0000","objects_scrubbed":0,"log_size":4612,"log_dups_size":0,"ondisk_log_size":4612,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":321912832,"num_objects":4912,"num_object_clones":0,"num_object_copies":14736,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":3786,"num_objects_unfound":0,"num_objects_dirty":4912,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4912,"num_write_kb":314368,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1664,"num_bytes_recovered":109051904,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,4],"acting":[2,4,3],"avail_no_missing":["2","3","4"],"object_location_counts":[{"shards":"2,3,4","objects":4912}],"blocked_by":[],"up_primary":5,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"74'79","reported_seq":528,"reported_epoch":340,"state":"active+clean","last_fresh":"2024-09-18T08:15:46.507811+0000","last_change":"2024-09-18T08:14:05.586022+0000","last_active":"2024-09-18T08:15:46.507811+0000","last_peered":"2024-09-18T08:15:46.507811+0000","last_clean":"2024-09-18T08:15:46.507811+0000","last_became_active":"2024-09-18T08:14:05.585608+0000","last_became_peered":"2024-09-18T08:14:05.585608+0000","last_unstale":"2024-09-18T08:15:46.507811+0000","last_undegraded":"2024-09-18T08:15:46.507811+0000","last_fullsized":"2024-09-18T08:15:46.507811+0000","mapping_epoch":292,"log_start":"0'0","ondisk_log_start":"0'0","created":152,"last_epoch_clean":293,"parent":"0.0","parent_split_bits":4,"last_scrub":"74'79","last_scrub_stamp":"2024-09-18T08:14:01.846338+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:00:11.893742+0000","last_clean_scrub_stamp":"2024-09-18T08:14:01.846338+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,2],"acting":[0,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.7","version":"271'17","reported_seq":599,"reported_epoch":340,"state":"active+clean","last_fresh":"2024-09-18T08:15:46.523305+0000","last_change":"2024-09-18T08:15:41.494329+0000","last_active":"2024-09-18T08:15:46.523305+0000","last_peered":"2024-09-18T08:15:46.523305+0000","last_clean":"2024-09-18T08:15:46.523305+0000","last_became_active":"2024-09-18T08:15:41.456482+0000","last_became_peered":"2024-09-18T08:15:41.456482+0000","last_unstale":"2024-09-18T08:15:46.523305+0000","last_undegraded":"2024-09-18T08:15:46.523305+0000","last_fullsized":"2024-09-18T08:15:46.523305+0000","mapping_epoch":336,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":337,"parent":"0.0","parent_split_bits":0,"last_scrub":"271'17","last_scrub_stamp":"2024-09-18T08:15:13.792602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_clean_scrub_stamp":"2024-09-18T08:15:13.792602+0000","objects_scrubbed":5,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:16:20.588620+0000","scrub_duration":16,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":35,"num_read_kb":24,"num_write":20,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":990,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,0],"acting":[2,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":484,"reported_epoch":340,"state":"active+clean","last_fresh":"2024-09-18T08:15:46.492584+0000","last_change":"2024-09-18T08:14:39.432590+0000","last_active":"2024-09-18T08:15:46.492584+0000","last_peered":"2024-09-18T08:15:46.492584+0000","last_clean":"2024-09-18T08:15:46.492584+0000","last_became_active":"2024-09-18T08:14:39.432254+0000","last_became_peered":"2024-09-18T08:14:39.432254+0000","last_unstale":"2024-09-18T08:15:46.492584+0000","last_undegraded":"2024-09-18T08:15:46.492584+0000","last_fullsized":"2024-09-18T08:15:46.492584+0000","mapping_epoch":307,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":308,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:11:31.367525+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:46.860092+0000","last_clean_scrub_stamp":"2024-09-18T08:11:31.367525+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"62'1","reported_seq":615,"reported_epoch":340,"state":"active+clean","last_fresh":"2024-09-18T08:15:46.520328+0000","last_change":"2024-09-18T08:14:31.765507+0000","last_active":"2024-09-18T08:15:46.520328+0000","last_peered":"2024-09-18T08:15:46.520328+0000","last_clean":"2024-09-18T08:15:46.520328+0000","last_became_active":"2024-09-18T08:02:46.868175+0000","last_became_peered":"2024-09-18T08:02:46.868175+0000","last_unstale":"2024-09-18T08:15:46.520328+0000","last_undegraded":"2024-09-18T08:15:46.520328+0000","last_fullsized":"2024-09-18T08:15:46.520328+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"62'1","last_scrub_stamp":"2024-09-18T08:14:31.765455+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:45.855979+0000","last_clean_scrub_stamp":"2024-09-18T08:14:31.765455+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.00066785499999999997,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":326,"reported_epoch":340,"state":"active+clean","last_fresh":"2024-09-18T08:15:46.515544+0000","last_change":"2024-09-18T08:15:41.442049+0000","last_active":"2024-09-18T08:15:46.515544+0000","last_peered":"2024-09-18T08:15:46.515544+0000","last_clean":"2024-09-18T08:15:46.515544+0000","last_became_active":"2024-09-18T08:15:41.441612+0000","last_became_peered":"2024-09-18T08:15:41.441612+0000","last_unstale":"2024-09-18T08:15:46.515544+0000","last_undegraded":"2024-09-18T08:15:46.515544+0000","last_fullsized":"2024-09-18T08:15:46.515544+0000","mapping_epoch":336,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":337,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:15:00.702835+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:50.886830+0000","last_clean_scrub_stamp":"2024-09-18T08:15:00.702835+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:16:07.836116+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,4],"acting":[2,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":286,"reported_epoch":340,"state":"active+clean","last_fresh":"2024-09-18T08:15:46.523339+0000","last_change":"2024-09-18T08:14:05.578356+0000","last_active":"2024-09-18T08:15:46.523339+0000","last_peered":"2024-09-18T08:15:46.523339+0000","last_clean":"2024-09-18T08:15:46.523339+0000","last_became_active":"2024-09-18T08:14:05.577516+0000","last_became_peered":"2024-09-18T08:14:05.577516+0000","last_unstale":"2024-09-18T08:15:46.523339+0000","last_undegraded":"2024-09-18T08:15:46.523339+0000","last_fullsized":"2024-09-18T08:15:46.523339+0000","mapping_epoch":292,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":293,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:13:36.978582+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_clean_scrub_stamp":"2024-09-18T08:13:36.978582+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.8","version":"340'4809","reported_seq":7936,"reported_epoch":340,"state":"active+remapped+backfilling","last_fresh":"2024-09-18T08:15:49.359986+0000","last_change":"2024-09-18T08:15:42.561230+0000","last_active":"2024-09-18T08:15:49.359986+0000","last_peered":"2024-09-18T08:15:49.359986+0000","last_clean":"2024-09-18T08:15:40.418681+0000","last_became_active":"2024-09-18T08:15:42.544149+0000","last_became_peered":"2024-09-18T08:15:42.544149+0000","last_unstale":"2024-09-18T08:15:49.359986+0000","last_undegraded":"2024-09-18T08:15:49.359986+0000","last_fullsized":"2024-09-18T08:15:49.359986+0000","mapping_epoch":337,"log_start":"319'700","ondisk_log_start":"319'700","created":311,"last_epoch_clean":324,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:14:43.412748+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:14:43.412748+0000","last_clean_scrub_stamp":"2024-09-18T08:14:43.412748+0000","objects_scrubbed":0,"log_size":4109,"log_dups_size":0,"ondisk_log_size":4109,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":315162624,"num_objects":4809,"num_object_clones":0,"num_object_copies":14427,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":3840,"num_objects_unfound":0,"num_objects_dirty":4809,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4809,"num_write_kb":307776,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1519,"num_bytes_recovered":99549184,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,5],"acting":[0,2,4],"avail_no_missing":["0","2","4"],"object_location_counts":[{"shards":"0,2,4","objects":4809}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"74'79","reported_seq":252,"reported_epoch":340,"state":"active+clean","last_fresh":"2024-09-18T08:15:46.494016+0000","last_change":"2024-09-18T08:14:40.223015+0000","last_active":"2024-09-18T08:15:46.494016+0000","last_peered":"2024-09-18T08:15:46.494016+0000","last_clean":"2024-09-18T08:15:46.494016+0000","last_became_active":"2024-09-18T08:14:39.384188+0000","last_became_peered":"2024-09-18T08:14:39.384188+0000","last_unstale":"2024-09-18T08:15:46.494016+0000","last_undegraded":"2024-09-18T08:15:46.494016+0000","last_fullsized":"2024-09-18T08:15:46.494016+0000","mapping_epoch":307,"log_start":"0'0","ondisk_log_start":"0'0","created":152,"last_epoch_clean":308,"parent":"0.0","parent_split_bits":3,"last_scrub":"74'79","last_scrub_stamp":"2024-09-18T08:14:40.222933+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:00:11.893742+0000","last_clean_scrub_stamp":"2024-09-18T08:14:40.222933+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:15:59.106182+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.6","version":"271'24","reported_seq":705,"reported_epoch":340,"state":"active+clean","last_fresh":"2024-09-18T08:15:46.509235+0000","last_change":"2024-09-18T08:15:10.187667+0000","last_active":"2024-09-18T08:15:46.509235+0000","last_peered":"2024-09-18T08:15:46.509235+0000","last_clean":"2024-09-18T08:15:46.509235+0000","last_became_active":"2024-09-18T08:14:39.438033+0000","last_became_peered":"2024-09-18T08:14:39.438033+0000","last_unstale":"2024-09-18T08:15:46.509235+0000","last_undegraded":"2024-09-18T08:15:46.509235+0000","last_fullsized":"2024-09-18T08:15:46.509235+0000","mapping_epoch":307,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":308,"parent":"0.0","parent_split_bits":0,"last_scrub":"271'24","last_scrub_stamp":"2024-09-18T08:15:10.187593+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_clean_scrub_stamp":"2024-09-18T08:15:10.187593+0000","objects_scrubbed":6,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:16:19.659283+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":36,"num_read_kb":30,"num_write":24,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":627,"reported_epoch":340,"state":"active+clean","last_fresh":"2024-09-18T08:15:46.507032+0000","last_change":"2024-09-18T08:14:37.133590+0000","last_active":"2024-09-18T08:15:46.507032+0000","last_peered":"2024-09-18T08:15:46.507032+0000","last_clean":"2024-09-18T08:15:46.507032+0000","last_became_active":"2024-09-18T08:08:18.523135+0000","last_became_peered":"2024-09-18T08:08:18.523135+0000","last_unstale":"2024-09-18T08:15:46.507032+0000","last_undegraded":"2024-09-18T08:15:46.507032+0000","last_fullsized":"2024-09-18T08:15:46.507032+0000","mapping_epoch":181,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":182,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:14:37.133544+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:46.860092+0000","last_clean_scrub_stamp":"2024-09-18T08:14:37.133544+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"70'5","reported_seq":623,"reported_epoch":340,"state":"active+clean","last_fresh":"2024-09-18T08:15:46.492142+0000","last_change":"2024-09-18T08:14:40.354226+0000","last_active":"2024-09-18T08:15:46.492142+0000","last_peered":"2024-09-18T08:15:46.492142+0000","last_clean":"2024-09-18T08:15:46.492142+0000","last_became_active":"2024-09-18T08:14:40.353801+0000","last_became_peered":"2024-09-18T08:14:40.353801+0000","last_unstale":"2024-09-18T08:15:46.492142+0000","last_undegraded":"2024-09-18T08:15:46.492142+0000","last_fullsized":"2024-09-18T08:15:46.492142+0000","mapping_epoch":308,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":309,"parent":"0.0","parent_split_bits":0,"last_scrub":"70'5","last_scrub_stamp":"2024-09-18T08:14:35.082297+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:45.855979+0000","last_clean_scrub_stamp":"2024-09-18T08:14:35.082297+0000","objects_scrubbed":1,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.000504055,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":240,"num_read_kb":235,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":389,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":18,"reported_epoch":340,"state":"active+clean","last_fresh":"2024-09-18T08:15:46.512686+0000","last_change":"2024-09-18T08:15:41.443577+0000","last_active":"2024-09-18T08:15:46.512686+0000","last_peered":"2024-09-18T08:15:46.512686+0000","last_clean":"2024-09-18T08:15:46.512686+0000","last_became_active":"2024-09-18T08:15:41.443080+0000","last_became_peered":"2024-09-18T08:15:41.443080+0000","last_unstale":"2024-09-18T08:15:46.512686+0000","last_undegraded":"2024-09-18T08:15:46.512686+0000","last_fullsized":"2024-09-18T08:15:46.512686+0000","mapping_epoch":336,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":337,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:15:02.179693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:50.886830+0000","last_clean_scrub_stamp":"2024-09-18T08:15:02.179693+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:16:29.603020+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":414,"reported_epoch":340,"state":"active+clean","last_fresh":"2024-09-18T08:15:46.503428+0000","last_change":"2024-09-18T08:15:41.437061+0000","last_active":"2024-09-18T08:15:46.503428+0000","last_peered":"2024-09-18T08:15:46.503428+0000","last_clean":"2024-09-18T08:15:46.503428+0000","last_became_active":"2024-09-18T08:15:41.436692+0000","last_became_peered":"2024-09-18T08:15:41.436692+0000","last_unstale":"2024-09-18T08:15:46.503428+0000","last_undegraded":"2024-09-18T08:15:46.503428+0000","last_fullsized":"2024-09-18T08:15:46.503428+0000","mapping_epoch":336,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":337,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:15:07.219279+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_clean_scrub_stamp":"2024-09-18T08:15:07.219279+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:16:27.847114+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.b","version":"340'4826","reported_seq":4895,"reported_epoch":340,"state":"active+clean","last_fresh":"2024-09-18T08:15:47.436001+0000","last_change":"2024-09-18T08:14:44.432885+0000","last_active":"2024-09-18T08:15:47.436001+0000","last_peered":"2024-09-18T08:15:47.436001+0000","last_clean":"2024-09-18T08:15:47.436001+0000","last_became_active":"2024-09-18T08:14:44.432599+0000","last_became_peered":"2024-09-18T08:14:44.432599+0000","last_unstale":"2024-09-18T08:15:47.436001+0000","last_undegraded":"2024-09-18T08:15:47.436001+0000","last_fullsized":"2024-09-18T08:15:47.436001+0000","mapping_epoch":311,"log_start":"320'800","ondisk_log_start":"320'800","created":311,"last_epoch_clean":312,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:14:43.412748+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:14:43.412748+0000","last_clean_scrub_stamp":"2024-09-18T08:14:43.412748+0000","objects_scrubbed":0,"log_size":4026,"log_dups_size":0,"ondisk_log_size":4026,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:16:11.284218+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":316276736,"num_objects":4826,"num_object_clones":0,"num_object_copies":14478,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4826,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4826,"num_write_kb":308864,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,0],"acting":[1,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.5","version":"293'34","reported_seq":500,"reported_epoch":340,"state":"active+clean","last_fresh":"2024-09-18T08:15:46.485765+0000","last_change":"2024-09-18T08:14:05.604747+0000","last_active":"2024-09-18T08:15:46.485765+0000","last_peered":"2024-09-18T08:15:46.485765+0000","last_clean":"2024-09-18T08:15:46.485765+0000","last_became_active":"2024-09-18T08:14:05.585190+0000","last_became_peered":"2024-09-18T08:14:05.585190+0000","last_unstale":"2024-09-18T08:15:46.485765+0000","last_undegraded":"2024-09-18T08:15:46.485765+0000","last_fullsized":"2024-09-18T08:15:46.485765+0000","mapping_epoch":292,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":293,"parent":"0.0","parent_split_bits":0,"last_scrub":"271'33","last_scrub_stamp":"2024-09-18T08:13:28.626987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_clean_scrub_stamp":"2024-09-18T08:13:28.626987+0000","objects_scrubbed":8,"log_size":34,"log_dups_size":0,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":24,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":42,"num_read_kb":32,"num_write":25,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":308,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,0],"acting":[4,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":626,"reported_epoch":340,"state":"active+clean","last_fresh":"2024-09-18T08:15:46.519025+0000","last_change":"2024-09-18T08:15:41.464630+0000","last_active":"2024-09-18T08:15:46.519025+0000","last_peered":"2024-09-18T08:15:46.519025+0000","last_clean":"2024-09-18T08:15:46.519025+0000","last_became_active":"2024-09-18T08:15:41.464313+0000","last_became_peered":"2024-09-18T08:15:41.464313+0000","last_unstale":"2024-09-18T08:15:46.519025+0000","last_undegraded":"2024-09-18T08:15:46.519025+0000","last_fullsized":"2024-09-18T08:15:46.519025+0000","mapping_epoch":336,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":337,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:14:32.693460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:46.860092+0000","last_clean_scrub_stamp":"2024-09-18T08:14:32.693460+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":38,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"74'79","reported_seq":215,"reported_epoch":340,"state":"active+clean","last_fresh":"2024-09-18T08:15:46.507551+0000","last_change":"2024-09-18T08:14:40.357413+0000","last_active":"2024-09-18T08:15:46.507551+0000","last_peered":"2024-09-18T08:15:46.507551+0000","last_clean":"2024-09-18T08:15:46.507551+0000","last_became_active":"2024-09-18T08:14:40.357063+0000","last_became_peered":"2024-09-18T08:14:40.357063+0000","last_unstale":"2024-09-18T08:15:46.507551+0000","last_undegraded":"2024-09-18T08:15:46.507551+0000","last_fullsized":"2024-09-18T08:15:46.507551+0000","mapping_epoch":308,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":309,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'79","last_scrub_stamp":"2024-09-18T08:13:51.810296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:00:11.893742+0000","last_clean_scrub_stamp":"2024-09-18T08:13:51.810296+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,1],"acting":[0,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":462,"reported_epoch":340,"state":"active+clean","last_fresh":"2024-09-18T08:15:46.514955+0000","last_change":"2024-09-18T08:15:41.441734+0000","last_active":"2024-09-18T08:15:46.514955+0000","last_peered":"2024-09-18T08:15:46.514955+0000","last_clean":"2024-09-18T08:15:46.514955+0000","last_became_active":"2024-09-18T08:15:41.441392+0000","last_became_peered":"2024-09-18T08:15:41.441392+0000","last_unstale":"2024-09-18T08:15:46.514955+0000","last_undegraded":"2024-09-18T08:15:46.514955+0000","last_fullsized":"2024-09-18T08:15:46.514955+0000","mapping_epoch":336,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":337,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:14:38.727133+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:50.886830+0000","last_clean_scrub_stamp":"2024-09-18T08:14:38.727133+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:15:49.423164+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,0],"acting":[2,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":18,"reported_epoch":340,"state":"active+clean","last_fresh":"2024-09-18T08:15:46.512581+0000","last_change":"2024-09-18T08:15:41.466410+0000","last_active":"2024-09-18T08:15:46.512581+0000","last_peered":"2024-09-18T08:15:46.512581+0000","last_clean":"2024-09-18T08:15:46.512581+0000","last_became_active":"2024-09-18T08:15:41.464638+0000","last_became_peered":"2024-09-18T08:15:41.464638+0000","last_unstale":"2024-09-18T08:15:46.512581+0000","last_undegraded":"2024-09-18T08:15:46.512581+0000","last_fullsized":"2024-09-18T08:15:46.512581+0000","mapping_epoch":336,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":337,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:12:05.886281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_clean_scrub_stamp":"2024-09-18T08:12:05.886281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.0","version":"340'4844","reported_seq":4913,"reported_epoch":340,"state":"active+clean","last_fresh":"2024-09-18T08:15:47.405852+0000","last_change":"2024-09-18T08:14:44.432450+0000","last_active":"2024-09-18T08:15:47.405852+0000","last_peered":"2024-09-18T08:15:47.405852+0000","last_clean":"2024-09-18T08:15:47.405852+0000","last_became_active":"2024-09-18T08:14:44.432101+0000","last_became_peered":"2024-09-18T08:14:44.432101+0000","last_unstale":"2024-09-18T08:15:47.405852+0000","last_undegraded":"2024-09-18T08:15:47.405852+0000","last_fullsized":"2024-09-18T08:15:47.405852+0000","mapping_epoch":311,"log_start":"319'800","ondisk_log_start":"319'800","created":311,"last_epoch_clean":312,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:14:43.412748+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:14:43.412748+0000","last_clean_scrub_stamp":"2024-09-18T08:14:43.412748+0000","objects_scrubbed":0,"log_size":4044,"log_dups_size":0,"ondisk_log_size":4044,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:16:00.690018+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":317456384,"num_objects":4844,"num_object_clones":0,"num_object_copies":14532,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4844,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4844,"num_write_kb":310016,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.e","version":"271'19","reported_seq":385,"reported_epoch":340,"state":"active+clean","last_fresh":"2024-09-18T08:15:46.488884+0000","last_change":"2024-09-18T08:14:39.515354+0000","last_active":"2024-09-18T08:15:46.488884+0000","last_peered":"2024-09-18T08:15:46.488884+0000","last_clean":"2024-09-18T08:15:46.488884+0000","last_became_active":"2024-09-18T08:14:39.376973+0000","last_became_peered":"2024-09-18T08:14:39.376973+0000","last_unstale":"2024-09-18T08:15:46.488884+0000","last_undegraded":"2024-09-18T08:15:46.488884+0000","last_fullsized":"2024-09-18T08:15:46.488884+0000","mapping_epoch":307,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":308,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'11","last_scrub_stamp":"2024-09-18T08:10:53.406574+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_clean_scrub_stamp":"2024-09-18T08:10:53.406574+0000","objects_scrubbed":5,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:16:27.580193+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":31,"num_read_kb":24,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":359,"reported_epoch":340,"state":"active+clean","last_fresh":"2024-09-18T08:15:46.486169+0000","last_change":"2024-09-18T08:14:40.355155+0000","last_active":"2024-09-18T08:15:46.486169+0000","last_peered":"2024-09-18T08:15:46.486169+0000","last_clean":"2024-09-18T08:15:46.486169+0000","last_became_active":"2024-09-18T08:14:40.354788+0000","last_became_peered":"2024-09-18T08:14:40.354788+0000","last_unstale":"2024-09-18T08:15:46.486169+0000","last_undegraded":"2024-09-18T08:15:46.486169+0000","last_fullsized":"2024-09-18T08:15:46.486169+0000","mapping_epoch":308,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":309,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:11:59.458232+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:46.860092+0000","last_clean_scrub_stamp":"2024-09-18T08:11:59.458232+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,1],"acting":[4,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":18,"reported_epoch":340,"state":"active+clean","last_fresh":"2024-09-18T08:15:46.512573+0000","last_change":"2024-09-18T08:15:41.453585+0000","last_active":"2024-09-18T08:15:46.512573+0000","last_peered":"2024-09-18T08:15:46.512573+0000","last_clean":"2024-09-18T08:15:46.512573+0000","last_became_active":"2024-09-18T08:15:41.447057+0000","last_became_peered":"2024-09-18T08:15:41.447057+0000","last_unstale":"2024-09-18T08:15:46.512573+0000","last_undegraded":"2024-09-18T08:15:46.512573+0000","last_fullsized":"2024-09-18T08:15:46.512573+0000","mapping_epoch":336,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":337,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:13:34.187158+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:50.886830+0000","last_clean_scrub_stamp":"2024-09-18T08:13:34.187158+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":613,"reported_epoch":340,"state":"active+clean","last_fresh":"2024-09-18T08:15:46.527602+0000","last_change":"2024-09-18T08:15:41.465035+0000","last_active":"2024-09-18T08:15:46.527602+0000","last_peered":"2024-09-18T08:15:46.527602+0000","last_clean":"2024-09-18T08:15:46.527602+0000","last_became_active":"2024-09-18T08:15:41.464500+0000","last_became_peered":"2024-09-18T08:15:41.464500+0000","last_unstale":"2024-09-18T08:15:46.527602+0000","last_undegraded":"2024-09-18T08:15:46.527602+0000","last_fullsized":"2024-09-18T08:15:46.527602+0000","mapping_epoch":336,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":337,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:15:02.861116+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_clean_scrub_stamp":"2024-09-18T08:15:02.861116+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:16:07.987232+0000","scrub_duration":81,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.3","version":"340'4813","reported_seq":6177,"reported_epoch":340,"state":"active+remapped+backfill_wait","last_fresh":"2024-09-18T08:15:48.857159+0000","last_change":"2024-09-18T08:15:42.598246+0000","last_active":"2024-09-18T08:15:48.857159+0000","last_peered":"2024-09-18T08:15:48.857159+0000","last_clean":"2024-09-18T08:15:40.403525+0000","last_became_active":"2024-09-18T08:15:42.566169+0000","last_became_peered":"2024-09-18T08:15:42.566169+0000","last_unstale":"2024-09-18T08:15:48.857159+0000","last_undegraded":"2024-09-18T08:15:48.857159+0000","last_fullsized":"2024-09-18T08:15:48.857159+0000","mapping_epoch":337,"log_start":"314'200","ondisk_log_start":"314'200","created":311,"last_epoch_clean":324,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:14:43.412748+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:14:43.412748+0000","last_clean_scrub_stamp":"2024-09-18T08:14:43.412748+0000","objects_scrubbed":0,"log_size":4613,"log_dups_size":0,"ondisk_log_size":4613,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":315424768,"num_objects":4813,"num_object_clones":0,"num_object_copies":14439,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":4813,"num_objects_unfound":0,"num_objects_dirty":4813,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4813,"num_write_kb":308032,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":637,"num_bytes_recovered":41746432,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,3],"acting":[2,3,4],"avail_no_missing":["2","3","4"],"object_location_counts":[{"shards":"2,3,4","objects":4813}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.8","version":"74'79","reported_seq":22,"reported_epoch":340,"state":"active+clean","last_fresh":"2024-09-18T08:15:46.511329+0000","last_change":"2024-09-18T08:15:41.485726+0000","last_active":"2024-09-18T08:15:46.511329+0000","last_peered":"2024-09-18T08:15:46.511329+0000","last_clean":"2024-09-18T08:15:46.511329+0000","last_became_active":"2024-09-18T08:15:41.454781+0000","last_became_peered":"2024-09-18T08:15:41.454781+0000","last_unstale":"2024-09-18T08:15:46.511329+0000","last_undegraded":"2024-09-18T08:15:46.511329+0000","last_fullsized":"2024-09-18T08:15:46.511329+0000","mapping_epoch":336,"log_start":"0'0","ondisk_log_start":"0'0","created":152,"last_epoch_clean":337,"parent":"0.0","parent_split_bits":4,"last_scrub":"74'79","last_scrub_stamp":"2024-09-18T08:14:54.675411+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:00:11.893742+0000","last_clean_scrub_stamp":"2024-09-18T08:14:54.675411+0000","objects_scrubbed":1,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:15:58.195611+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262688,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":262688,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,3],"acting":[5,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.d","version":"271'31","reported_seq":406,"reported_epoch":340,"state":"active+clean","last_fresh":"2024-09-18T08:15:46.485830+0000","last_change":"2024-09-18T08:15:03.678756+0000","last_active":"2024-09-18T08:15:46.485830+0000","last_peered":"2024-09-18T08:15:46.485830+0000","last_clean":"2024-09-18T08:15:46.485830+0000","last_became_active":"2024-09-18T08:14:39.377819+0000","last_became_peered":"2024-09-18T08:14:39.377819+0000","last_unstale":"2024-09-18T08:15:46.485830+0000","last_undegraded":"2024-09-18T08:15:46.485830+0000","last_fullsized":"2024-09-18T08:15:46.485830+0000","mapping_epoch":307,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":308,"parent":"0.0","parent_split_bits":0,"last_scrub":"271'31","last_scrub_stamp":"2024-09-18T08:15:03.678670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_clean_scrub_stamp":"2024-09-18T08:15:03.678670+0000","objects_scrubbed":8,"log_size":31,"log_dups_size":0,"ondisk_log_size":31,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:16:30.987390+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":49,"num_read_kb":39,"num_write":32,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,1],"acting":[4,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":358,"reported_epoch":340,"state":"active+clean","last_fresh":"2024-09-18T08:15:46.485020+0000","last_change":"2024-09-18T08:15:09.614801+0000","last_active":"2024-09-18T08:15:46.485020+0000","last_peered":"2024-09-18T08:15:46.485020+0000","last_clean":"2024-09-18T08:15:46.485020+0000","last_became_active":"2024-09-18T08:14:39.375087+0000","last_became_peered":"2024-09-18T08:14:39.375087+0000","last_unstale":"2024-09-18T08:15:46.485020+0000","last_undegraded":"2024-09-18T08:15:46.485020+0000","last_fullsized":"2024-09-18T08:15:46.485020+0000","mapping_epoch":307,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":308,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:15:09.614752+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:46.860092+0000","last_clean_scrub_stamp":"2024-09-18T08:15:09.614752+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:16:38.410150+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":492,"reported_epoch":340,"state":"active+clean","last_fresh":"2024-09-18T08:15:46.503068+0000","last_change":"2024-09-18T08:14:42.230469+0000","last_active":"2024-09-18T08:15:46.503068+0000","last_peered":"2024-09-18T08:15:46.503068+0000","last_clean":"2024-09-18T08:15:46.503068+0000","last_became_active":"2024-09-18T08:14:39.442274+0000","last_became_peered":"2024-09-18T08:14:39.442274+0000","last_unstale":"2024-09-18T08:15:46.503068+0000","last_undegraded":"2024-09-18T08:15:46.503068+0000","last_fullsized":"2024-09-18T08:15:46.503068+0000","mapping_epoch":307,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":308,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:14:42.230420+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:50.886830+0000","last_clean_scrub_stamp":"2024-09-18T08:14:42.230420+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:16:00.658277+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":321,"reported_epoch":340,"state":"active+clean","last_fresh":"2024-09-18T08:15:46.523206+0000","last_change":"2024-09-18T08:14:35.757447+0000","last_active":"2024-09-18T08:15:46.523206+0000","last_peered":"2024-09-18T08:15:46.523206+0000","last_clean":"2024-09-18T08:15:46.523206+0000","last_became_active":"2024-09-18T08:08:52.696300+0000","last_became_peered":"2024-09-18T08:08:52.696300+0000","last_unstale":"2024-09-18T08:15:46.523206+0000","last_undegraded":"2024-09-18T08:15:46.523206+0000","last_fullsized":"2024-09-18T08:15:46.523206+0000","mapping_epoch":198,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":199,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:14:35.757380+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_clean_scrub_stamp":"2024-09-18T08:14:35.757380+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:16:03.865522+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.2","version":"340'4742","reported_seq":4811,"reported_epoch":340,"state":"active+clean","last_fresh":"2024-09-18T08:15:47.448317+0000","last_change":"2024-09-18T08:14:44.437864+0000","last_active":"2024-09-18T08:15:47.448317+0000","last_peered":"2024-09-18T08:15:47.448317+0000","last_clean":"2024-09-18T08:15:47.448317+0000","last_became_active":"2024-09-18T08:14:44.437430+0000","last_became_peered":"2024-09-18T08:14:44.437430+0000","last_unstale":"2024-09-18T08:15:47.448317+0000","last_undegraded":"2024-09-18T08:15:47.448317+0000","last_fullsized":"2024-09-18T08:15:47.448317+0000","mapping_epoch":311,"log_start":"319'700","ondisk_log_start":"319'700","created":311,"last_epoch_clean":312,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:14:43.412748+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:14:43.412748+0000","last_clean_scrub_stamp":"2024-09-18T08:14:43.412748+0000","objects_scrubbed":0,"log_size":4042,"log_dups_size":0,"ondisk_log_size":4042,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:16:12.065164+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":310771712,"num_objects":4742,"num_object_clones":0,"num_object_copies":14226,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4742,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4742,"num_write_kb":303488,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"74'79","reported_seq":431,"reported_epoch":340,"state":"active+clean","last_fresh":"2024-09-18T08:15:46.512747+0000","last_change":"2024-09-18T08:15:41.467055+0000","last_active":"2024-09-18T08:15:46.512747+0000","last_peered":"2024-09-18T08:15:46.512747+0000","last_clean":"2024-09-18T08:15:46.512747+0000","last_became_active":"2024-09-18T08:15:41.465341+0000","last_became_peered":"2024-09-18T08:15:41.465341+0000","last_unstale":"2024-09-18T08:15:46.512747+0000","last_undegraded":"2024-09-18T08:15:46.512747+0000","last_fullsized":"2024-09-18T08:15:46.512747+0000","mapping_epoch":336,"log_start":"0'0","ondisk_log_start":"0'0","created":152,"last_epoch_clean":337,"parent":"0.0","parent_split_bits":4,"last_scrub":"74'79","last_scrub_stamp":"2024-09-18T08:13:52.510099+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:00:11.893742+0000","last_clean_scrub_stamp":"2024-09-18T08:13:52.510099+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.c","version":"271'20","reported_seq":358,"reported_epoch":340,"state":"active+clean","last_fresh":"2024-09-18T08:15:46.485408+0000","last_change":"2024-09-18T08:08:18.642452+0000","last_active":"2024-09-18T08:15:46.485408+0000","last_peered":"2024-09-18T08:15:46.485408+0000","last_clean":"2024-09-18T08:15:46.485408+0000","last_became_active":"2024-09-18T08:08:18.622980+0000","last_became_peered":"2024-09-18T08:08:18.622980+0000","last_unstale":"2024-09-18T08:15:46.485408+0000","last_undegraded":"2024-09-18T08:15:46.485408+0000","last_fullsized":"2024-09-18T08:15:46.485408+0000","mapping_epoch":181,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":182,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'10","last_scrub_stamp":"2024-09-18T08:05:41.474813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_clean_scrub_stamp":"2024-09-18T08:05:41.474813+0000","objects_scrubbed":0,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:15:52.641137+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":29,"num_read_kb":24,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":626,"reported_epoch":340,"state":"active+clean","last_fresh":"2024-09-18T08:15:46.536624+0000","last_change":"2024-09-18T08:15:19.461006+0000","last_active":"2024-09-18T08:15:46.536624+0000","last_peered":"2024-09-18T08:15:46.536624+0000","last_clean":"2024-09-18T08:15:46.536624+0000","last_became_active":"2024-09-18T08:15:19.460612+0000","last_became_peered":"2024-09-18T08:15:19.460612+0000","last_unstale":"2024-09-18T08:15:46.536624+0000","last_undegraded":"2024-09-18T08:15:46.536624+0000","last_fullsized":"2024-09-18T08:15:46.536624+0000","mapping_epoch":329,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":330,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:14:04.626103+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:46.860092+0000","last_clean_scrub_stamp":"2024-09-18T08:14:04.626103+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.d","version":"338'13","reported_seq":806,"reported_epoch":340,"state":"active+clean","last_fresh":"2024-09-18T08:15:46.514961+0000","last_change":"2024-09-18T08:15:41.471980+0000","last_active":"2024-09-18T08:15:46.514961+0000","last_peered":"2024-09-18T08:15:46.514961+0000","last_clean":"2024-09-18T08:15:46.514961+0000","last_became_active":"2024-09-18T08:15:41.441562+0000","last_became_peered":"2024-09-18T08:15:41.441562+0000","last_unstale":"2024-09-18T08:15:46.514961+0000","last_undegraded":"2024-09-18T08:15:46.514961+0000","last_fullsized":"2024-09-18T08:15:46.514961+0000","mapping_epoch":336,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":337,"parent":"0.0","parent_split_bits":0,"last_scrub":"293'9","last_scrub_stamp":"2024-09-18T08:14:37.759701+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:50.886830+0000","last_clean_scrub_stamp":"2024-09-18T08:14:37.759701+0000","objects_scrubbed":1,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:15:50.937902+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,0],"acting":[2,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":365,"reported_epoch":340,"state":"active+clean","last_fresh":"2024-09-18T08:15:47.605040+0000","last_change":"2024-09-18T08:15:47.605040+0000","last_active":"2024-09-18T08:15:47.605040+0000","last_peered":"2024-09-18T08:15:47.605040+0000","last_clean":"2024-09-18T08:15:47.605040+0000","last_became_active":"2024-09-18T08:14:39.380178+0000","last_became_peered":"2024-09-18T08:14:39.380178+0000","last_unstale":"2024-09-18T08:15:47.605040+0000","last_undegraded":"2024-09-18T08:15:47.605040+0000","last_fullsized":"2024-09-18T08:15:47.605040+0000","mapping_epoch":307,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":308,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:15:47.604976+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_clean_scrub_stamp":"2024-09-18T08:15:47.604976+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:16:48.951103+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.5","version":"340'4853","reported_seq":4922,"reported_epoch":340,"state":"active+clean","last_fresh":"2024-09-18T08:15:48.836950+0000","last_change":"2024-09-18T08:14:44.432113+0000","last_active":"2024-09-18T08:15:48.836950+0000","last_peered":"2024-09-18T08:15:48.836950+0000","last_clean":"2024-09-18T08:15:48.836950+0000","last_became_active":"2024-09-18T08:14:44.431835+0000","last_became_peered":"2024-09-18T08:14:44.431835+0000","last_unstale":"2024-09-18T08:15:48.836950+0000","last_undegraded":"2024-09-18T08:15:48.836950+0000","last_fullsized":"2024-09-18T08:15:48.836950+0000","mapping_epoch":311,"log_start":"314'200","ondisk_log_start":"314'200","created":311,"last_epoch_clean":312,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:14:43.412748+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:14:43.412748+0000","last_clean_scrub_stamp":"2024-09-18T08:14:43.412748+0000","objects_scrubbed":0,"log_size":4653,"log_dups_size":0,"ondisk_log_size":4653,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:16:04.244471+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":318046208,"num_objects":4853,"num_object_clones":0,"num_object_copies":14559,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4853,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4853,"num_write_kb":310592,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.b","version":"271'15","reported_seq":671,"reported_epoch":340,"state":"active+clean","last_fresh":"2024-09-18T08:15:46.508732+0000","last_change":"2024-09-18T08:14:39.482777+0000","last_active":"2024-09-18T08:15:46.508732+0000","last_peered":"2024-09-18T08:15:46.508732+0000","last_clean":"2024-09-18T08:15:46.508732+0000","last_became_active":"2024-09-18T08:14:39.371063+0000","last_became_peered":"2024-09-18T08:14:39.371063+0000","last_unstale":"2024-09-18T08:15:46.508732+0000","last_undegraded":"2024-09-18T08:15:46.508732+0000","last_fullsized":"2024-09-18T08:15:46.508732+0000","mapping_epoch":307,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":308,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'9","last_scrub_stamp":"2024-09-18T08:10:47.928596+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_clean_scrub_stamp":"2024-09-18T08:10:47.928596+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:15:49.090755+0000","scrub_duration":16,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":18,"reported_epoch":340,"state":"active+clean","last_fresh":"2024-09-18T08:15:46.511742+0000","last_change":"2024-09-18T08:15:41.443068+0000","last_active":"2024-09-18T08:15:46.511742+0000","last_peered":"2024-09-18T08:15:46.511742+0000","last_clean":"2024-09-18T08:15:46.511742+0000","last_became_active":"2024-09-18T08:15:41.442697+0000","last_became_peered":"2024-09-18T08:15:41.442697+0000","last_unstale":"2024-09-18T08:15:46.511742+0000","last_undegraded":"2024-09-18T08:15:46.511742+0000","last_fullsized":"2024-09-18T08:15:46.511742+0000","mapping_epoch":336,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":337,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:11:22.076452+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:46.860092+0000","last_clean_scrub_stamp":"2024-09-18T08:11:22.076452+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":603,"reported_epoch":340,"state":"active+clean","last_fresh":"2024-09-18T08:15:46.520062+0000","last_change":"2024-09-18T08:08:18.509651+0000","last_active":"2024-09-18T08:15:46.520062+0000","last_peered":"2024-09-18T08:15:46.520062+0000","last_clean":"2024-09-18T08:15:46.520062+0000","last_became_active":"2024-09-18T08:08:18.509247+0000","last_became_peered":"2024-09-18T08:08:18.509247+0000","last_unstale":"2024-09-18T08:15:46.520062+0000","last_undegraded":"2024-09-18T08:15:46.520062+0000","last_fullsized":"2024-09-18T08:15:46.520062+0000","mapping_epoch":181,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":182,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:07:10.728679+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:50.886830+0000","last_clean_scrub_stamp":"2024-09-18T08:07:10.728679+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:15:52.643929+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,3],"acting":[2,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":622,"reported_epoch":340,"state":"active+clean","last_fresh":"2024-09-18T08:15:46.506941+0000","last_change":"2024-09-18T08:14:05.585049+0000","last_active":"2024-09-18T08:15:46.506941+0000","last_peered":"2024-09-18T08:15:46.506941+0000","last_clean":"2024-09-18T08:15:46.506941+0000","last_became_active":"2024-09-18T08:14:05.583887+0000","last_became_peered":"2024-09-18T08:14:05.583887+0000","last_unstale":"2024-09-18T08:15:46.506941+0000","last_undegraded":"2024-09-18T08:15:46.506941+0000","last_fullsized":"2024-09-18T08:15:46.506941+0000","mapping_epoch":292,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":293,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:12:38.702798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_clean_scrub_stamp":"2024-09-18T08:12:38.702798+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:16:18.256244+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,3],"acting":[0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.4","version":"340'4856","reported_seq":4925,"reported_epoch":340,"state":"active+clean","last_fresh":"2024-09-18T08:15:49.309505+0000","last_change":"2024-09-18T08:14:44.437938+0000","last_active":"2024-09-18T08:15:49.309505+0000","last_peered":"2024-09-18T08:15:49.309505+0000","last_clean":"2024-09-18T08:15:49.309505+0000","last_became_active":"2024-09-18T08:14:44.437396+0000","last_became_peered":"2024-09-18T08:14:44.437396+0000","last_unstale":"2024-09-18T08:15:49.309505+0000","last_undegraded":"2024-09-18T08:15:49.309505+0000","last_fullsized":"2024-09-18T08:15:49.309505+0000","mapping_epoch":311,"log_start":"319'700","ondisk_log_start":"319'700","created":311,"last_epoch_clean":312,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:14:43.412748+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:14:43.412748+0000","last_clean_scrub_stamp":"2024-09-18T08:14:43.412748+0000","objects_scrubbed":0,"log_size":4156,"log_dups_size":0,"ondisk_log_size":4156,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":318242816,"num_objects":4856,"num_object_clones":0,"num_object_copies":14568,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4856,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4856,"num_write_kb":310784,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.a","version":"271'35","reported_seq":529,"reported_epoch":340,"state":"active+clean","last_fresh":"2024-09-18T08:15:46.493466+0000","last_change":"2024-09-18T08:15:43.282823+0000","last_active":"2024-09-18T08:15:46.493466+0000","last_peered":"2024-09-18T08:15:46.493466+0000","last_clean":"2024-09-18T08:15:46.493466+0000","last_became_active":"2024-09-18T08:15:41.441186+0000","last_became_peered":"2024-09-18T08:15:41.441186+0000","last_unstale":"2024-09-18T08:15:46.493466+0000","last_undegraded":"2024-09-18T08:15:46.493466+0000","last_fullsized":"2024-09-18T08:15:46.493466+0000","mapping_epoch":336,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":337,"parent":"0.0","parent_split_bits":0,"last_scrub":"271'35","last_scrub_stamp":"2024-09-18T08:15:43.282736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_clean_scrub_stamp":"2024-09-18T08:15:43.282736+0000","objects_scrubbed":9,"log_size":35,"log_dups_size":0,"ondisk_log_size":35,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:16:50.012638+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":55,"num_read_kb":44,"num_write":36,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":44,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,5],"acting":[1,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":18,"reported_epoch":340,"state":"active+clean","last_fresh":"2024-09-18T08:15:46.511936+0000","last_change":"2024-09-18T08:15:41.452272+0000","last_active":"2024-09-18T08:15:46.511936+0000","last_peered":"2024-09-18T08:15:46.511936+0000","last_clean":"2024-09-18T08:15:46.511936+0000","last_became_active":"2024-09-18T08:15:41.446405+0000","last_became_peered":"2024-09-18T08:15:41.446405+0000","last_unstale":"2024-09-18T08:15:46.511936+0000","last_undegraded":"2024-09-18T08:15:46.511936+0000","last_fullsized":"2024-09-18T08:15:46.511936+0000","mapping_epoch":336,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":337,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:12:31.205468+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:46.860092+0000","last_clean_scrub_stamp":"2024-09-18T08:12:31.205468+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,3],"acting":[5,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":327,"reported_epoch":340,"state":"active+clean","last_fresh":"2024-09-18T08:15:46.515577+0000","last_change":"2024-09-18T08:15:41.447379+0000","last_active":"2024-09-18T08:15:46.515577+0000","last_peered":"2024-09-18T08:15:46.515577+0000","last_clean":"2024-09-18T08:15:46.515577+0000","last_became_active":"2024-09-18T08:15:41.447075+0000","last_became_peered":"2024-09-18T08:15:41.447075+0000","last_unstale":"2024-09-18T08:15:46.515577+0000","last_undegraded":"2024-09-18T08:15:46.515577+0000","last_fullsized":"2024-09-18T08:15:46.515577+0000","mapping_epoch":336,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":337,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:15:10.767720+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:50.886830+0000","last_clean_scrub_stamp":"2024-09-18T08:15:10.767720+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:16:21.548420+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":448,"reported_epoch":340,"state":"active+clean","last_fresh":"2024-09-18T08:15:46.519480+0000","last_change":"2024-09-18T08:15:41.722212+0000","last_active":"2024-09-18T08:15:46.519480+0000","last_peered":"2024-09-18T08:15:46.519480+0000","last_clean":"2024-09-18T08:15:46.519480+0000","last_became_active":"2024-09-18T08:14:05.578053+0000","last_became_peered":"2024-09-18T08:14:05.578053+0000","last_unstale":"2024-09-18T08:15:46.519480+0000","last_undegraded":"2024-09-18T08:15:46.519480+0000","last_fullsized":"2024-09-18T08:15:46.519480+0000","mapping_epoch":292,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":293,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:15:41.722163+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_clean_scrub_stamp":"2024-09-18T08:15:41.722163+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:17:00.305704+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.7","version":"340'4853","reported_seq":6107,"reported_epoch":340,"state":"active+remapped+backfill_toofull","last_fresh":"2024-09-18T08:15:49.174006+0000","last_change":"2024-09-18T08:15:42.580001+0000","last_active":"2024-09-18T08:15:49.174006+0000","last_peered":"2024-09-18T08:15:49.174006+0000","last_clean":"2024-09-18T08:15:40.409092+0000","last_became_active":"2024-09-18T08:15:42.564409+0000","last_became_peered":"2024-09-18T08:15:42.564409+0000","last_unstale":"2024-09-18T08:15:49.174006+0000","last_undegraded":"2024-09-18T08:15:49.174006+0000","last_fullsized":"2024-09-18T08:15:49.174006+0000","mapping_epoch":337,"log_start":"320'800","ondisk_log_start":"320'800","created":311,"last_epoch_clean":323,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:14:43.412748+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:14:43.412748+0000","last_clean_scrub_stamp":"2024-09-18T08:14:43.412748+0000","objects_scrubbed":0,"log_size":4053,"log_dups_size":0,"ondisk_log_size":4053,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":318046208,"num_objects":4853,"num_object_clones":0,"num_object_copies":14559,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":4853,"num_objects_unfound":0,"num_objects_dirty":4853,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4853,"num_write_kb":310592,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":585,"num_bytes_recovered":38338560,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,2],"avail_no_missing":["3","2","6"],"object_location_counts":[{"shards":"2,3,6","objects":4853}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.9","version":"271'18","reported_seq":398,"reported_epoch":340,"state":"active+clean","last_fresh":"2024-09-18T08:15:46.485886+0000","last_change":"2024-09-18T08:14:52.681876+0000","last_active":"2024-09-18T08:15:46.485886+0000","last_peered":"2024-09-18T08:15:46.485886+0000","last_clean":"2024-09-18T08:15:46.485886+0000","last_became_active":"2024-09-18T08:14:52.649706+0000","last_became_peered":"2024-09-18T08:14:52.649706+0000","last_unstale":"2024-09-18T08:15:46.485886+0000","last_undegraded":"2024-09-18T08:15:46.485886+0000","last_fullsized":"2024-09-18T08:15:46.485886+0000","mapping_epoch":316,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":317,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'12","last_scrub_stamp":"2024-09-18T08:11:27.607049+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_clean_scrub_stamp":"2024-09-18T08:11:27.607049+0000","objects_scrubbed":5,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":33,"num_read_kb":24,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":880,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":543,"reported_epoch":340,"state":"active+clean","last_fresh":"2024-09-18T08:15:46.488761+0000","last_change":"2024-09-18T08:14:40.355356+0000","last_active":"2024-09-18T08:15:46.488761+0000","last_peered":"2024-09-18T08:15:46.488761+0000","last_clean":"2024-09-18T08:15:46.488761+0000","last_became_active":"2024-09-18T08:14:40.355030+0000","last_became_peered":"2024-09-18T08:14:40.355030+0000","last_unstale":"2024-09-18T08:15:46.488761+0000","last_undegraded":"2024-09-18T08:15:46.488761+0000","last_fullsized":"2024-09-18T08:15:46.488761+0000","mapping_epoch":308,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":309,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:10:57.579577+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:46.860092+0000","last_clean_scrub_stamp":"2024-09-18T08:10:57.579577+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:16:22.554174+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,0],"acting":[4,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":608,"reported_epoch":340,"state":"active+clean","last_fresh":"2024-09-18T08:15:46.519575+0000","last_change":"2024-09-18T08:14:41.671839+0000","last_active":"2024-09-18T08:15:46.519575+0000","last_peered":"2024-09-18T08:15:46.519575+0000","last_clean":"2024-09-18T08:15:46.519575+0000","last_became_active":"2024-09-18T08:14:39.399301+0000","last_became_peered":"2024-09-18T08:14:39.399301+0000","last_unstale":"2024-09-18T08:15:46.519575+0000","last_undegraded":"2024-09-18T08:15:46.519575+0000","last_fullsized":"2024-09-18T08:15:46.519575+0000","mapping_epoch":307,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":308,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:14:41.671793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:50.886830+0000","last_clean_scrub_stamp":"2024-09-18T08:14:41.671793+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:16:03.918137+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":610,"reported_epoch":340,"state":"active+clean","last_fresh":"2024-09-18T08:15:46.536155+0000","last_change":"2024-09-18T08:14:40.356757+0000","last_active":"2024-09-18T08:15:46.536155+0000","last_peered":"2024-09-18T08:15:46.536155+0000","last_clean":"2024-09-18T08:15:46.536155+0000","last_became_active":"2024-09-18T08:14:40.356451+0000","last_became_peered":"2024-09-18T08:14:40.356451+0000","last_unstale":"2024-09-18T08:15:46.536155+0000","last_undegraded":"2024-09-18T08:15:46.536155+0000","last_fullsized":"2024-09-18T08:15:46.536155+0000","mapping_epoch":308,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":309,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:11:21.372405+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_clean_scrub_stamp":"2024-09-18T08:11:21.372405+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:16:19.970924+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.6","version":"340'4778","reported_seq":6057,"reported_epoch":340,"state":"active+remapped+backfill_wait","last_fresh":"2024-09-18T08:15:49.360649+0000","last_change":"2024-09-18T08:15:42.600218+0000","last_active":"2024-09-18T08:15:49.360649+0000","last_peered":"2024-09-18T08:15:49.360649+0000","last_clean":"2024-09-18T08:15:40.390287+0000","last_became_active":"2024-09-18T08:15:42.559473+0000","last_became_peered":"2024-09-18T08:15:42.559473+0000","last_unstale":"2024-09-18T08:15:49.360649+0000","last_undegraded":"2024-09-18T08:15:49.360649+0000","last_fullsized":"2024-09-18T08:15:49.360649+0000","mapping_epoch":337,"log_start":"318'600","ondisk_log_start":"318'600","created":311,"last_epoch_clean":324,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:14:43.412748+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:14:43.412748+0000","last_clean_scrub_stamp":"2024-09-18T08:14:43.412748+0000","objects_scrubbed":0,"log_size":4178,"log_dups_size":0,"ondisk_log_size":4178,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":313131008,"num_objects":4778,"num_object_clones":0,"num_object_copies":14334,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":4778,"num_objects_unfound":0,"num_objects_dirty":4778,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4778,"num_write_kb":305792,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":596,"num_bytes_recovered":39059456,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,4],"acting":[0,4,1],"avail_no_missing":["0","1","4"],"object_location_counts":[{"shards":"0,1,4","objects":4778}],"blocked_by":[],"up_primary":5,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.8","version":"271'27","reported_seq":34,"reported_epoch":340,"state":"active+clean","last_fresh":"2024-09-18T08:15:46.511401+0000","last_change":"2024-09-18T08:15:41.480348+0000","last_active":"2024-09-18T08:15:46.511401+0000","last_peered":"2024-09-18T08:15:46.511401+0000","last_clean":"2024-09-18T08:15:46.511401+0000","last_became_active":"2024-09-18T08:15:41.446549+0000","last_became_peered":"2024-09-18T08:15:41.446549+0000","last_unstale":"2024-09-18T08:15:46.511401+0000","last_undegraded":"2024-09-18T08:15:46.511401+0000","last_fullsized":"2024-09-18T08:15:46.511401+0000","mapping_epoch":336,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":337,"parent":"0.0","parent_split_bits":0,"last_scrub":"271'27","last_scrub_stamp":"2024-09-18T08:15:00.680764+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_clean_scrub_stamp":"2024-09-18T08:15:00.680764+0000","objects_scrubbed":7,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:16:14.939742+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":44,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,4],"acting":[5,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.f","version":"62'2","reported_seq":417,"reported_epoch":340,"state":"active+clean","last_fresh":"2024-09-18T08:15:46.485674+0000","last_change":"2024-09-18T08:14:40.364260+0000","last_active":"2024-09-18T08:15:46.485674+0000","last_peered":"2024-09-18T08:15:46.485674+0000","last_clean":"2024-09-18T08:15:46.485674+0000","last_became_active":"2024-09-18T08:14:40.354828+0000","last_became_peered":"2024-09-18T08:14:40.354828+0000","last_unstale":"2024-09-18T08:15:46.485674+0000","last_undegraded":"2024-09-18T08:15:46.485674+0000","last_fullsized":"2024-09-18T08:15:46.485674+0000","mapping_epoch":308,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":309,"parent":"0.0","parent_split_bits":0,"last_scrub":"62'2","last_scrub_stamp":"2024-09-18T08:10:53.580444+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:46.860092+0000","last_clean_scrub_stamp":"2024-09-18T08:10:53.580444+0000","objects_scrubbed":2,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:16:17.545269+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":4,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":184,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,1],"acting":[4,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.9","version":"310'16","reported_seq":525,"reported_epoch":340,"state":"active+clean","last_fresh":"2024-09-18T08:15:46.485708+0000","last_change":"2024-09-18T08:14:40.368033+0000","last_active":"2024-09-18T08:15:46.485708+0000","last_peered":"2024-09-18T08:15:46.485708+0000","last_clean":"2024-09-18T08:15:46.485708+0000","last_became_active":"2024-09-18T08:14:40.360883+0000","last_became_peered":"2024-09-18T08:14:40.360883+0000","last_unstale":"2024-09-18T08:15:46.485708+0000","last_undegraded":"2024-09-18T08:15:46.485708+0000","last_fullsized":"2024-09-18T08:15:46.485708+0000","mapping_epoch":308,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":309,"parent":"0.0","parent_split_bits":0,"last_scrub":"294'14","last_scrub_stamp":"2024-09-18T08:14:28.686640+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:50.886830+0000","last_clean_scrub_stamp":"2024-09-18T08:14:28.686640+0000","objects_scrubbed":1,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:15:49.989136+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":18,"reported_epoch":340,"state":"active+clean","last_fresh":"2024-09-18T08:15:46.511996+0000","last_change":"2024-09-18T08:15:41.454432+0000","last_active":"2024-09-18T08:15:46.511996+0000","last_peered":"2024-09-18T08:15:46.511996+0000","last_clean":"2024-09-18T08:15:46.511996+0000","last_became_active":"2024-09-18T08:15:41.449889+0000","last_became_peered":"2024-09-18T08:15:41.449889+0000","last_unstale":"2024-09-18T08:15:46.511996+0000","last_undegraded":"2024-09-18T08:15:46.511996+0000","last_fullsized":"2024-09-18T08:15:46.511996+0000","mapping_epoch":336,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":337,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:15:13.161614+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_clean_scrub_stamp":"2024-09-18T08:15:13.161614+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:16:17.907918+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":536,"reported_epoch":340,"state":"active+clean","last_fresh":"2024-09-18T08:15:46.507361+0000","last_change":"2024-09-18T08:15:41.448597+0000","last_active":"2024-09-18T08:15:46.507361+0000","last_peered":"2024-09-18T08:15:46.507361+0000","last_clean":"2024-09-18T08:15:46.507361+0000","last_became_active":"2024-09-18T08:15:41.446544+0000","last_became_peered":"2024-09-18T08:15:41.446544+0000","last_unstale":"2024-09-18T08:15:46.507361+0000","last_undegraded":"2024-09-18T08:15:46.507361+0000","last_fullsized":"2024-09-18T08:15:46.507361+0000","mapping_epoch":336,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":337,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:13:06.860320+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:46.860092+0000","last_clean_scrub_stamp":"2024-09-18T08:13:06.860320+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.17","version":"271'12","reported_seq":660,"reported_epoch":340,"state":"active+clean","last_fresh":"2024-09-18T08:15:46.513688+0000","last_change":"2024-09-18T08:14:40.365750+0000","last_active":"2024-09-18T08:15:46.513688+0000","last_peered":"2024-09-18T08:15:46.513688+0000","last_clean":"2024-09-18T08:15:46.513688+0000","last_became_active":"2024-09-18T08:14:40.355751+0000","last_became_peered":"2024-09-18T08:14:40.355751+0000","last_unstale":"2024-09-18T08:15:46.513688+0000","last_undegraded":"2024-09-18T08:15:46.513688+0000","last_fullsized":"2024-09-18T08:15:46.513688+0000","mapping_epoch":308,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":309,"parent":"0.0","parent_split_bits":0,"last_scrub":"271'12","last_scrub_stamp":"2024-09-18T08:14:33.700931+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_clean_scrub_stamp":"2024-09-18T08:14:33.700931+0000","objects_scrubbed":3,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:15:52.526359+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":18,"num_read_kb":15,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":18,"reported_epoch":340,"state":"active+clean","last_fresh":"2024-09-18T08:15:46.511894+0000","last_change":"2024-09-18T08:15:41.448984+0000","last_active":"2024-09-18T08:15:46.511894+0000","last_peered":"2024-09-18T08:15:46.511894+0000","last_clean":"2024-09-18T08:15:46.511894+0000","last_became_active":"2024-09-18T08:15:41.445332+0000","last_became_peered":"2024-09-18T08:15:41.445332+0000","last_unstale":"2024-09-18T08:15:46.511894+0000","last_undegraded":"2024-09-18T08:15:46.511894+0000","last_fullsized":"2024-09-18T08:15:46.511894+0000","mapping_epoch":336,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":337,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:15:07.747147+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:50.886830+0000","last_clean_scrub_stamp":"2024-09-18T08:15:07.747147+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:16:08.674990+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,1],"acting":[5,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":405,"reported_epoch":340,"state":"active+clean","last_fresh":"2024-09-18T08:15:46.485324+0000","last_change":"2024-09-18T08:15:41.444603+0000","last_active":"2024-09-18T08:15:46.485324+0000","last_peered":"2024-09-18T08:15:46.485324+0000","last_clean":"2024-09-18T08:15:46.485324+0000","last_became_active":"2024-09-18T08:15:41.444268+0000","last_became_peered":"2024-09-18T08:15:41.444268+0000","last_unstale":"2024-09-18T08:15:46.485324+0000","last_undegraded":"2024-09-18T08:15:46.485324+0000","last_fullsized":"2024-09-18T08:15:46.485324+0000","mapping_epoch":336,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":337,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:12:54.661578+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_clean_scrub_stamp":"2024-09-18T08:12:54.661578+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.16","version":"271'15","reported_seq":634,"reported_epoch":340,"state":"active+clean","last_fresh":"2024-09-18T08:15:46.509253+0000","last_change":"2024-09-18T08:14:05.588589+0000","last_active":"2024-09-18T08:15:46.509253+0000","last_peered":"2024-09-18T08:15:46.509253+0000","last_clean":"2024-09-18T08:15:46.509253+0000","last_became_active":"2024-09-18T08:14:05.578519+0000","last_became_peered":"2024-09-18T08:14:05.578519+0000","last_unstale":"2024-09-18T08:15:46.509253+0000","last_undegraded":"2024-09-18T08:15:46.509253+0000","last_fullsized":"2024-09-18T08:15:46.509253+0000","mapping_epoch":292,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":293,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'9","last_scrub_stamp":"2024-09-18T08:09:44.116308+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_clean_scrub_stamp":"2024-09-18T08:09:44.116308+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:16:24.309658+0000","scrub_duration":43,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":354,"reported_epoch":340,"state":"active+clean","last_fresh":"2024-09-18T08:15:46.485991+0000","last_change":"2024-09-18T08:14:05.583783+0000","last_active":"2024-09-18T08:15:46.485991+0000","last_peered":"2024-09-18T08:15:46.485991+0000","last_clean":"2024-09-18T08:15:46.485991+0000","last_became_active":"2024-09-18T08:14:05.583435+0000","last_became_peered":"2024-09-18T08:14:05.583435+0000","last_unstale":"2024-09-18T08:15:46.485991+0000","last_undegraded":"2024-09-18T08:15:46.485991+0000","last_fullsized":"2024-09-18T08:15:46.485991+0000","mapping_epoch":292,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":293,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:12:03.433407+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:46.860092+0000","last_clean_scrub_stamp":"2024-09-18T08:12:03.433407+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,0],"acting":[4,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":605,"reported_epoch":340,"state":"active+clean","last_fresh":"2024-09-18T08:15:46.513534+0000","last_change":"2024-09-18T08:14:40.356857+0000","last_active":"2024-09-18T08:15:46.513534+0000","last_peered":"2024-09-18T08:15:46.513534+0000","last_clean":"2024-09-18T08:15:46.513534+0000","last_became_active":"2024-09-18T08:14:40.356531+0000","last_became_peered":"2024-09-18T08:14:40.356531+0000","last_unstale":"2024-09-18T08:15:46.513534+0000","last_undegraded":"2024-09-18T08:15:46.513534+0000","last_fullsized":"2024-09-18T08:15:46.513534+0000","mapping_epoch":308,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":309,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:11:17.424701+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:50.886830+0000","last_clean_scrub_stamp":"2024-09-18T08:11:17.424701+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:16:17.892236+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":607,"reported_epoch":340,"state":"active+clean","last_fresh":"2024-09-18T08:15:46.514910+0000","last_change":"2024-09-18T08:14:05.582569+0000","last_active":"2024-09-18T08:15:46.514910+0000","last_peered":"2024-09-18T08:15:46.514910+0000","last_clean":"2024-09-18T08:15:46.514910+0000","last_became_active":"2024-09-18T08:14:05.582226+0000","last_became_peered":"2024-09-18T08:14:05.582226+0000","last_unstale":"2024-09-18T08:15:46.514910+0000","last_undegraded":"2024-09-18T08:15:46.514910+0000","last_fullsized":"2024-09-18T08:15:46.514910+0000","mapping_epoch":292,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":293,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:10:42.731845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_clean_scrub_stamp":"2024-09-18T08:10:42.731845+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:16:18.787695+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,0],"acting":[2,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.15","version":"271'15","reported_seq":28,"reported_epoch":340,"state":"active+clean","last_fresh":"2024-09-18T08:15:46.512087+0000","last_change":"2024-09-18T08:15:41.485598+0000","last_active":"2024-09-18T08:15:46.512087+0000","last_peered":"2024-09-18T08:15:46.512087+0000","last_clean":"2024-09-18T08:15:46.512087+0000","last_became_active":"2024-09-18T08:15:41.447339+0000","last_became_peered":"2024-09-18T08:15:41.447339+0000","last_unstale":"2024-09-18T08:15:46.512087+0000","last_undegraded":"2024-09-18T08:15:46.512087+0000","last_fullsized":"2024-09-18T08:15:46.512087+0000","mapping_epoch":336,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":337,"parent":"0.0","parent_split_bits":0,"last_scrub":"271'15","last_scrub_stamp":"2024-09-18T08:12:58.225295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_clean_scrub_stamp":"2024-09-18T08:12:58.225295+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":25,"num_read_kb":19,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":624,"reported_epoch":340,"state":"active+clean","last_fresh":"2024-09-18T08:15:46.508063+0000","last_change":"2024-09-18T08:14:40.357339+0000","last_active":"2024-09-18T08:15:46.508063+0000","last_peered":"2024-09-18T08:15:46.508063+0000","last_clean":"2024-09-18T08:15:46.508063+0000","last_became_active":"2024-09-18T08:14:40.355709+0000","last_became_peered":"2024-09-18T08:14:40.355709+0000","last_unstale":"2024-09-18T08:15:46.508063+0000","last_undegraded":"2024-09-18T08:15:46.508063+0000","last_fullsized":"2024-09-18T08:15:46.508063+0000","mapping_epoch":308,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":309,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:14:28.097629+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:46.860092+0000","last_clean_scrub_stamp":"2024-09-18T08:14:28.097629+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1],"acting":[0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.14","version":"310'16","reported_seq":813,"reported_epoch":340,"state":"active+clean","last_fresh":"2024-09-18T08:15:46.527775+0000","last_change":"2024-09-18T08:14:40.366956+0000","last_active":"2024-09-18T08:15:46.527775+0000","last_peered":"2024-09-18T08:15:46.527775+0000","last_clean":"2024-09-18T08:15:46.527775+0000","last_became_active":"2024-09-18T08:14:40.358971+0000","last_became_peered":"2024-09-18T08:14:40.358971+0000","last_unstale":"2024-09-18T08:15:46.527775+0000","last_undegraded":"2024-09-18T08:15:46.527775+0000","last_fullsized":"2024-09-18T08:15:46.527775+0000","mapping_epoch":308,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":309,"parent":"0.0","parent_split_bits":0,"last_scrub":"199'12","last_scrub_stamp":"2024-09-18T08:13:31.695354+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:50.886830+0000","last_clean_scrub_stamp":"2024-09-18T08:13:31.695354+0000","objects_scrubbed":1,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,1],"acting":[3,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":353,"reported_epoch":340,"state":"active+clean","last_fresh":"2024-09-18T08:15:46.485725+0000","last_change":"2024-09-18T08:15:41.441640+0000","last_active":"2024-09-18T08:15:46.485725+0000","last_peered":"2024-09-18T08:15:46.485725+0000","last_clean":"2024-09-18T08:15:46.485725+0000","last_became_active":"2024-09-18T08:15:41.441327+0000","last_became_peered":"2024-09-18T08:15:41.441327+0000","last_unstale":"2024-09-18T08:15:46.485725+0000","last_undegraded":"2024-09-18T08:15:46.485725+0000","last_fullsized":"2024-09-18T08:15:46.485725+0000","mapping_epoch":336,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":337,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:12:06.521774+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_clean_scrub_stamp":"2024-09-18T08:12:06.521774+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.14","version":"271'20","reported_seq":666,"reported_epoch":340,"state":"active+clean","last_fresh":"2024-09-18T08:15:46.528136+0000","last_change":"2024-09-18T08:14:40.378165+0000","last_active":"2024-09-18T08:15:46.528136+0000","last_peered":"2024-09-18T08:15:46.528136+0000","last_clean":"2024-09-18T08:15:46.528136+0000","last_became_active":"2024-09-18T08:14:40.358983+0000","last_became_peered":"2024-09-18T08:14:40.358983+0000","last_unstale":"2024-09-18T08:15:46.528136+0000","last_undegraded":"2024-09-18T08:15:46.528136+0000","last_fullsized":"2024-09-18T08:15:46.528136+0000","mapping_epoch":308,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":309,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'10","last_scrub_stamp":"2024-09-18T08:12:39.946462+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_clean_scrub_stamp":"2024-09-18T08:12:39.946462+0000","objects_scrubbed":5,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":25,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":380,"reported_epoch":340,"state":"active+clean","last_fresh":"2024-09-18T08:15:46.486100+0000","last_change":"2024-09-18T08:14:05.588122+0000","last_active":"2024-09-18T08:15:46.486100+0000","last_peered":"2024-09-18T08:15:46.486100+0000","last_clean":"2024-09-18T08:15:46.486100+0000","last_became_active":"2024-09-18T08:14:05.587338+0000","last_became_peered":"2024-09-18T08:14:05.587338+0000","last_unstale":"2024-09-18T08:15:46.486100+0000","last_undegraded":"2024-09-18T08:15:46.486100+0000","last_fullsized":"2024-09-18T08:15:46.486100+0000","mapping_epoch":292,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":293,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:13:32.933940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:46.860092+0000","last_clean_scrub_stamp":"2024-09-18T08:13:32.933940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":21,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,6],"acting":[4,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.15","version":"338'12","reported_seq":24,"reported_epoch":340,"state":"active+clean","last_fresh":"2024-09-18T08:15:46.512032+0000","last_change":"2024-09-18T08:15:41.489414+0000","last_active":"2024-09-18T08:15:46.512032+0000","last_peered":"2024-09-18T08:15:46.512032+0000","last_clean":"2024-09-18T08:15:46.512032+0000","last_became_active":"2024-09-18T08:15:41.455243+0000","last_became_peered":"2024-09-18T08:15:41.455243+0000","last_unstale":"2024-09-18T08:15:46.512032+0000","last_undegraded":"2024-09-18T08:15:46.512032+0000","last_fullsized":"2024-09-18T08:15:46.512032+0000","mapping_epoch":336,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":337,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'8","last_scrub_stamp":"2024-09-18T08:10:54.148986+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:50.886830+0000","last_clean_scrub_stamp":"2024-09-18T08:10:54.148986+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":592,"reported_epoch":340,"state":"active+clean","last_fresh":"2024-09-18T08:15:46.508771+0000","last_change":"2024-09-18T08:14:05.590487+0000","last_active":"2024-09-18T08:15:46.508771+0000","last_peered":"2024-09-18T08:15:46.508771+0000","last_clean":"2024-09-18T08:15:46.508771+0000","last_became_active":"2024-09-18T08:14:05.590147+0000","last_became_peered":"2024-09-18T08:14:05.590147+0000","last_unstale":"2024-09-18T08:15:46.508771+0000","last_undegraded":"2024-09-18T08:15:46.508771+0000","last_fullsized":"2024-09-18T08:15:46.508771+0000","mapping_epoch":292,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":293,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:09:42.047270+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_clean_scrub_stamp":"2024-09-18T08:09:42.047270+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:16:25.282745+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.13","version":"271'19","reported_seq":385,"reported_epoch":340,"state":"active+clean","last_fresh":"2024-09-18T08:15:46.486194+0000","last_change":"2024-09-18T08:14:39.467914+0000","last_active":"2024-09-18T08:15:46.486194+0000","last_peered":"2024-09-18T08:15:46.486194+0000","last_clean":"2024-09-18T08:15:46.486194+0000","last_became_active":"2024-09-18T08:14:39.369949+0000","last_became_peered":"2024-09-18T08:14:39.369949+0000","last_unstale":"2024-09-18T08:15:46.486194+0000","last_undegraded":"2024-09-18T08:15:46.486194+0000","last_fullsized":"2024-09-18T08:15:46.486194+0000","mapping_epoch":307,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":308,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'11","last_scrub_stamp":"2024-09-18T08:11:22.565415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_clean_scrub_stamp":"2024-09-18T08:11:22.565415+0000","objects_scrubbed":5,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:16:35.591742+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":31,"num_read_kb":24,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":346,"reported_epoch":340,"state":"active+clean","last_fresh":"2024-09-18T08:15:46.485611+0000","last_change":"2024-09-18T08:14:05.582592+0000","last_active":"2024-09-18T08:15:46.485611+0000","last_peered":"2024-09-18T08:15:46.485611+0000","last_clean":"2024-09-18T08:15:46.485611+0000","last_became_active":"2024-09-18T08:14:05.581338+0000","last_became_peered":"2024-09-18T08:14:05.581338+0000","last_unstale":"2024-09-18T08:15:46.485611+0000","last_undegraded":"2024-09-18T08:15:46.485611+0000","last_fullsized":"2024-09-18T08:15:46.485611+0000","mapping_epoch":292,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":293,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:10:58.402308+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:46.860092+0000","last_clean_scrub_stamp":"2024-09-18T08:10:58.402308+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:16:31.538697+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,2],"acting":[4,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":483,"reported_epoch":340,"state":"active+clean","last_fresh":"2024-09-18T08:15:46.493102+0000","last_change":"2024-09-18T08:15:41.442065+0000","last_active":"2024-09-18T08:15:46.493102+0000","last_peered":"2024-09-18T08:15:46.493102+0000","last_clean":"2024-09-18T08:15:46.493102+0000","last_became_active":"2024-09-18T08:15:41.438048+0000","last_became_peered":"2024-09-18T08:15:41.438048+0000","last_unstale":"2024-09-18T08:15:46.493102+0000","last_undegraded":"2024-09-18T08:15:46.493102+0000","last_fullsized":"2024-09-18T08:15:46.493102+0000","mapping_epoch":336,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":337,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:11:26.316948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:50.886830+0000","last_clean_scrub_stamp":"2024-09-18T08:11:26.316948+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":616,"reported_epoch":340,"state":"active+clean","last_fresh":"2024-09-18T08:15:46.512587+0000","last_change":"2024-09-18T08:15:41.466178+0000","last_active":"2024-09-18T08:15:46.512587+0000","last_peered":"2024-09-18T08:15:46.512587+0000","last_clean":"2024-09-18T08:15:46.512587+0000","last_became_active":"2024-09-18T08:15:41.464961+0000","last_became_peered":"2024-09-18T08:15:41.464961+0000","last_unstale":"2024-09-18T08:15:46.512587+0000","last_undegraded":"2024-09-18T08:15:46.512587+0000","last_fullsized":"2024-09-18T08:15:46.512587+0000","mapping_epoch":336,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":337,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:13:28.680237+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_clean_scrub_stamp":"2024-09-18T08:13:28.680237+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.12","version":"271'15","reported_seq":471,"reported_epoch":340,"state":"active+clean","last_fresh":"2024-09-18T08:15:46.493371+0000","last_change":"2024-09-18T08:14:39.480181+0000","last_active":"2024-09-18T08:15:46.493371+0000","last_peered":"2024-09-18T08:15:46.493371+0000","last_clean":"2024-09-18T08:15:46.493371+0000","last_became_active":"2024-09-18T08:14:39.416284+0000","last_became_peered":"2024-09-18T08:14:39.416284+0000","last_unstale":"2024-09-18T08:15:46.493371+0000","last_undegraded":"2024-09-18T08:15:46.493371+0000","last_fullsized":"2024-09-18T08:15:46.493371+0000","mapping_epoch":307,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":308,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'9","last_scrub_stamp":"2024-09-18T08:10:06.582804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_clean_scrub_stamp":"2024-09-18T08:10:06.582804+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:15:52.149563+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":25,"num_read_kb":19,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":395,"reported_epoch":340,"state":"active+clean","last_fresh":"2024-09-18T08:15:46.523565+0000","last_change":"2024-09-18T08:14:05.581553+0000","last_active":"2024-09-18T08:15:46.523565+0000","last_peered":"2024-09-18T08:15:46.523565+0000","last_clean":"2024-09-18T08:15:46.523565+0000","last_became_active":"2024-09-18T08:14:05.581038+0000","last_became_peered":"2024-09-18T08:14:05.581038+0000","last_unstale":"2024-09-18T08:15:46.523565+0000","last_undegraded":"2024-09-18T08:15:46.523565+0000","last_fullsized":"2024-09-18T08:15:46.523565+0000","mapping_epoch":292,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":293,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:13:35.000383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:46.860092+0000","last_clean_scrub_stamp":"2024-09-18T08:13:35.000383+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,0],"acting":[3,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":600,"reported_epoch":340,"state":"active+clean","last_fresh":"2024-09-18T08:15:46.536321+0000","last_change":"2024-09-18T08:14:39.403741+0000","last_active":"2024-09-18T08:15:46.536321+0000","last_peered":"2024-09-18T08:15:46.536321+0000","last_clean":"2024-09-18T08:15:46.536321+0000","last_became_active":"2024-09-18T08:14:39.402406+0000","last_became_peered":"2024-09-18T08:14:39.402406+0000","last_unstale":"2024-09-18T08:15:46.536321+0000","last_undegraded":"2024-09-18T08:15:46.536321+0000","last_fullsized":"2024-09-18T08:15:46.536321+0000","mapping_epoch":307,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":308,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:11:30.302179+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:50.886830+0000","last_clean_scrub_stamp":"2024-09-18T08:11:30.302179+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:16:34.883373+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.10","version":"69'1","reported_seq":614,"reported_epoch":340,"state":"active+clean","last_fresh":"2024-09-18T08:15:46.507063+0000","last_change":"2024-09-18T08:15:41.480106+0000","last_active":"2024-09-18T08:15:46.507063+0000","last_peered":"2024-09-18T08:15:46.507063+0000","last_clean":"2024-09-18T08:15:46.507063+0000","last_became_active":"2024-09-18T08:15:41.444085+0000","last_became_peered":"2024-09-18T08:15:41.444085+0000","last_unstale":"2024-09-18T08:15:46.507063+0000","last_undegraded":"2024-09-18T08:15:46.507063+0000","last_fullsized":"2024-09-18T08:15:46.507063+0000","mapping_epoch":336,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":337,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'1","last_scrub_stamp":"2024-09-18T08:15:00.219678+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_clean_scrub_stamp":"2024-09-18T08:15:00.219678+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:16:13.737108+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":26,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.11","version":"271'19","reported_seq":743,"reported_epoch":340,"state":"active+clean","last_fresh":"2024-09-18T08:15:46.512858+0000","last_change":"2024-09-18T08:15:41.491797+0000","last_active":"2024-09-18T08:15:46.512858+0000","last_peered":"2024-09-18T08:15:46.512858+0000","last_clean":"2024-09-18T08:15:46.512858+0000","last_became_active":"2024-09-18T08:15:41.465631+0000","last_became_peered":"2024-09-18T08:15:41.465631+0000","last_unstale":"2024-09-18T08:15:46.512858+0000","last_undegraded":"2024-09-18T08:15:46.512858+0000","last_fullsized":"2024-09-18T08:15:46.512858+0000","mapping_epoch":336,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":337,"parent":"0.0","parent_split_bits":0,"last_scrub":"271'19","last_scrub_stamp":"2024-09-18T08:14:38.809490+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_clean_scrub_stamp":"2024-09-18T08:14:38.809490+0000","objects_scrubbed":5,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:16:00.032157+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":32,"num_read_kb":25,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":550,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":18,"reported_epoch":340,"state":"active+clean","last_fresh":"2024-09-18T08:15:46.511887+0000","last_change":"2024-09-18T08:15:41.449283+0000","last_active":"2024-09-18T08:15:46.511887+0000","last_peered":"2024-09-18T08:15:46.511887+0000","last_clean":"2024-09-18T08:15:46.511887+0000","last_became_active":"2024-09-18T08:15:41.445583+0000","last_became_peered":"2024-09-18T08:15:41.445583+0000","last_unstale":"2024-09-18T08:15:46.511887+0000","last_undegraded":"2024-09-18T08:15:46.511887+0000","last_fullsized":"2024-09-18T08:15:46.511887+0000","mapping_epoch":336,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":337,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:10:53.145315+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:46.860092+0000","last_clean_scrub_stamp":"2024-09-18T08:10:53.145315+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":402,"reported_epoch":340,"state":"active+clean","last_fresh":"2024-09-18T08:15:46.484888+0000","last_change":"2024-09-18T08:14:37.743954+0000","last_active":"2024-09-18T08:15:46.484888+0000","last_peered":"2024-09-18T08:15:46.484888+0000","last_clean":"2024-09-18T08:15:46.484888+0000","last_became_active":"2024-09-18T08:14:05.585776+0000","last_became_peered":"2024-09-18T08:14:05.585776+0000","last_unstale":"2024-09-18T08:15:46.484888+0000","last_undegraded":"2024-09-18T08:15:46.484888+0000","last_fullsized":"2024-09-18T08:15:46.484888+0000","mapping_epoch":292,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":293,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:14:37.743856+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:50.886830+0000","last_clean_scrub_stamp":"2024-09-18T08:14:37.743856+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,0],"acting":[4,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":599,"reported_epoch":340,"state":"active+clean","last_fresh":"2024-09-18T08:15:46.536528+0000","last_change":"2024-09-18T08:11:29.344444+0000","last_active":"2024-09-18T08:15:46.536528+0000","last_peered":"2024-09-18T08:15:46.536528+0000","last_clean":"2024-09-18T08:15:46.536528+0000","last_became_active":"2024-09-18T08:06:24.128582+0000","last_became_peered":"2024-09-18T08:06:24.128582+0000","last_unstale":"2024-09-18T08:15:46.536528+0000","last_undegraded":"2024-09-18T08:15:46.536528+0000","last_fullsized":"2024-09-18T08:15:46.536528+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:11:29.344382+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_clean_scrub_stamp":"2024-09-18T08:11:29.344382+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:16:26.894147+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.10","version":"271'8","reported_seq":649,"reported_epoch":340,"state":"active+clean","last_fresh":"2024-09-18T08:15:46.518272+0000","last_change":"2024-09-18T08:14:39.475047+0000","last_active":"2024-09-18T08:15:46.518272+0000","last_peered":"2024-09-18T08:15:46.518272+0000","last_clean":"2024-09-18T08:15:46.518272+0000","last_became_active":"2024-09-18T08:14:39.404257+0000","last_became_peered":"2024-09-18T08:14:39.404257+0000","last_unstale":"2024-09-18T08:15:46.518272+0000","last_undegraded":"2024-09-18T08:15:46.518272+0000","last_fullsized":"2024-09-18T08:15:46.518272+0000","mapping_epoch":307,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":308,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'4","last_scrub_stamp":"2024-09-18T08:10:18.542984+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_clean_scrub_stamp":"2024-09-18T08:10:18.542984+0000","objects_scrubbed":2,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:15:48.821303+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":12,"num_read_kb":10,"num_write":8,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":626,"reported_epoch":340,"state":"active+clean","last_fresh":"2024-09-18T08:15:46.508270+0000","last_change":"2024-09-18T08:15:41.467281+0000","last_active":"2024-09-18T08:15:46.508270+0000","last_peered":"2024-09-18T08:15:46.508270+0000","last_clean":"2024-09-18T08:15:46.508270+0000","last_became_active":"2024-09-18T08:15:41.466964+0000","last_became_peered":"2024-09-18T08:15:41.466964+0000","last_unstale":"2024-09-18T08:15:46.508270+0000","last_undegraded":"2024-09-18T08:15:46.508270+0000","last_fullsized":"2024-09-18T08:15:46.508270+0000","mapping_epoch":336,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":337,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:14:33.064199+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:46.860092+0000","last_clean_scrub_stamp":"2024-09-18T08:14:33.064199+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":354,"reported_epoch":340,"state":"active+clean","last_fresh":"2024-09-18T08:15:46.486111+0000","last_change":"2024-09-18T08:14:05.589315+0000","last_active":"2024-09-18T08:15:46.486111+0000","last_peered":"2024-09-18T08:15:46.486111+0000","last_clean":"2024-09-18T08:15:46.486111+0000","last_became_active":"2024-09-18T08:14:05.588551+0000","last_became_peered":"2024-09-18T08:14:05.588551+0000","last_unstale":"2024-09-18T08:15:46.486111+0000","last_undegraded":"2024-09-18T08:15:46.486111+0000","last_fullsized":"2024-09-18T08:15:46.486111+0000","mapping_epoch":292,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":293,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:13:32.640669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:50.886830+0000","last_clean_scrub_stamp":"2024-09-18T08:13:32.640669+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":20,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,0],"acting":[4,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":400,"reported_epoch":340,"state":"active+clean","last_fresh":"2024-09-18T08:15:46.519916+0000","last_change":"2024-09-18T08:14:05.582585+0000","last_active":"2024-09-18T08:15:46.519916+0000","last_peered":"2024-09-18T08:15:46.519916+0000","last_clean":"2024-09-18T08:15:46.519916+0000","last_became_active":"2024-09-18T08:14:05.582250+0000","last_became_peered":"2024-09-18T08:14:05.582250+0000","last_unstale":"2024-09-18T08:15:46.519916+0000","last_undegraded":"2024-09-18T08:15:46.519916+0000","last_fullsized":"2024-09-18T08:15:46.519916+0000","mapping_epoch":292,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":293,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:13:56.525167+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_clean_scrub_stamp":"2024-09-18T08:13:56.525167+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:16:15.801368+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,3],"acting":[2,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":456,"reported_epoch":340,"state":"active+clean","last_fresh":"2024-09-18T08:15:46.501960+0000","last_change":"2024-09-18T08:15:45.297283+0000","last_active":"2024-09-18T08:15:46.501960+0000","last_peered":"2024-09-18T08:15:46.501960+0000","last_clean":"2024-09-18T08:15:46.501960+0000","last_became_active":"2024-09-18T08:15:41.437638+0000","last_became_peered":"2024-09-18T08:15:41.437638+0000","last_unstale":"2024-09-18T08:15:46.501960+0000","last_undegraded":"2024-09-18T08:15:46.501960+0000","last_fullsized":"2024-09-18T08:15:46.501960+0000","mapping_epoch":336,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":337,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:15:45.297240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_clean_scrub_stamp":"2024-09-18T08:15:45.297240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:17:03.256928+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":608,"reported_epoch":340,"state":"active+clean","last_fresh":"2024-09-18T08:15:46.518496+0000","last_change":"2024-09-18T08:15:43.836627+0000","last_active":"2024-09-18T08:15:46.518496+0000","last_peered":"2024-09-18T08:15:46.518496+0000","last_clean":"2024-09-18T08:15:46.518496+0000","last_became_active":"2024-09-18T08:14:39.405347+0000","last_became_peered":"2024-09-18T08:14:39.405347+0000","last_unstale":"2024-09-18T08:15:46.518496+0000","last_undegraded":"2024-09-18T08:15:46.518496+0000","last_fullsized":"2024-09-18T08:15:46.518496+0000","mapping_epoch":307,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":308,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:15:43.836549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:46.860092+0000","last_clean_scrub_stamp":"2024-09-18T08:15:43.836549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:17:02.503647+0000","scrub_duration":20,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"271'19","reported_seq":30,"reported_epoch":340,"state":"active+clean","last_fresh":"2024-09-18T08:15:46.512483+0000","last_change":"2024-09-18T08:15:41.472787+0000","last_active":"2024-09-18T08:15:46.512483+0000","last_peered":"2024-09-18T08:15:46.512483+0000","last_clean":"2024-09-18T08:15:46.512483+0000","last_became_active":"2024-09-18T08:15:41.445198+0000","last_became_peered":"2024-09-18T08:15:41.445198+0000","last_unstale":"2024-09-18T08:15:46.512483+0000","last_undegraded":"2024-09-18T08:15:46.512483+0000","last_fullsized":"2024-09-18T08:15:46.512483+0000","mapping_epoch":336,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":337,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'11","last_scrub_stamp":"2024-09-18T08:10:50.208543+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_clean_scrub_stamp":"2024-09-18T08:10:50.208543+0000","objects_scrubbed":5,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":31,"num_read_kb":24,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,1],"acting":[5,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":623,"reported_epoch":340,"state":"active+clean","last_fresh":"2024-09-18T08:15:46.508943+0000","last_change":"2024-09-18T08:14:40.358776+0000","last_active":"2024-09-18T08:15:46.508943+0000","last_peered":"2024-09-18T08:15:46.508943+0000","last_clean":"2024-09-18T08:15:46.508943+0000","last_became_active":"2024-09-18T08:14:40.358457+0000","last_became_peered":"2024-09-18T08:14:40.358457+0000","last_unstale":"2024-09-18T08:15:46.508943+0000","last_undegraded":"2024-09-18T08:15:46.508943+0000","last_fullsized":"2024-09-18T08:15:46.508943+0000","mapping_epoch":308,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":309,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:12:31.664456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:50.886830+0000","last_clean_scrub_stamp":"2024-09-18T08:12:31.664456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:16:17.252353+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":10,"num_pg":16,"stat_sum":{"num_bytes":5055184896,"num_objects":77136,"num_object_clones":0,"num_object_copies":231408,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":26071,"num_objects_unfound":0,"num_objects_dirty":77136,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":77136,"num_write_kb":4936704,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6384,"num_bytes_recovered":418381824,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":15346499584,"data_stored":15346499584,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":66536,"ondisk_log_size":66536,"up":48,"acting":48,"num_store_stats":7},{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":416,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":832,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":112,"ondisk_log_size":112,"up":96,"acting":96,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":178,"num_object_clones":0,"num_object_copies":534,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":178,"num_whiteouts":0,"num_read":1108,"num_read_kb":865,"num_write":703,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":406,"num_bytes_recovered":7238,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":442368,"data_stored":11766,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3639,"internal_metadata":0},"log_size":687,"ondisk_log_size":687,"up":96,"acting":96,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":1613,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":24,"num_read_kb":24,"num_write":10,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11,"num_bytes_recovered":3570,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":81920,"data_stored":4931,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":6,"ondisk_log_size":6,"up":96,"acting":96,"num_store_stats":8},{"poolid":2,"num_pg":3,"stat_sum":{"num_bytes":408,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":240,"num_read_kb":235,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":408,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1613,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":8,"ondisk_log_size":8,"up":9,"acting":9,"num_store_stats":7},{"poolid":1,"num_pg":11,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1245728,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":869,"ondisk_log_size":869,"up":33,"acting":33,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":0,"seq":240518168736,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":47,"seq":201863463078,"num_pgs":60,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2545228,"kb_used_data":2153084,"kb_used_omap":1,"kb_used_meta":392126,"kb_avail":91199924,"statfs":{"total":95995035648,"available":93388722176,"internally_reserved":0,"allocated":2204758016,"data_stored":2203838185,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":401537478},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":39,"seq":167503724716,"num_pgs":64,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":719844,"kb_used_data":165264,"kb_used_omap":1,"kb_used_meta":554558,"kb_avail":93025308,"statfs":{"total":95995035648,"available":95257915392,"internally_reserved":0,"allocated":169230336,"data_stored":168291487,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":567867846},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":32,"seq":137438953649,"num_pgs":76,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":3791920,"kb_used_data":3337308,"kb_used_omap":1,"kb_used_meta":454590,"kb_avail":89953232,"statfs":{"total":95995035648,"available":92112109568,"internally_reserved":0,"allocated":3417403392,"data_stored":3416452391,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":465500613},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":26,"seq":111669149880,"num_pgs":75,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2798712,"kb_used_data":2171944,"kb_used_omap":1,"kb_used_meta":626750,"kb_avail":90946440,"statfs":{"total":95995035648,"available":93129154560,"internally_reserved":0,"allocated":2224070656,"data_stored":2223107886,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":641792456},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411518,"num_pgs":61,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2957344,"kb_used_data":2474960,"kb_used_omap":1,"kb_used_meta":482366,"kb_avail":90787808,"statfs":{"total":95995035648,"available":92966715392,"internally_reserved":0,"allocated":2534359040,"data_stored":2533420297,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":493943237},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":307,"seq":1318554959887,"num_pgs":73,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1630628,"kb_used_data":1526096,"kb_used_omap":7,"kb_used_meta":104504,"kb_avail":92114524,"statfs":{"total":95995035648,"available":94325272576,"internally_reserved":0,"allocated":1562722304,"data_stored":1561719776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":8079,"internal_metadata":107012209},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738569,"num_pgs":71,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":3616044,"kb_used_data":3097564,"kb_used_omap":1,"kb_used_meta":518462,"kb_avail":90129108,"statfs":{"total":95995035648,"available":92292206592,"internally_reserved":0,"allocated":3171905536,"data_stored":3170954781,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":530905541},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":266240,"data_stored":262688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1131,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1613,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":436,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":46,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1177,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":436,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1474,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":94208,"data_stored":2448,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3639,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1348,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":1898,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":1584,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1210,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":1144,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":660,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":403,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":403,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":416,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":3176857600,"data_stored":3176857600,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":1564999680,"data_stored":1564999680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":2535653376,"data_stored":2535653376,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":2233335808,"data_stored":2233335808,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":3451846656,"data_stored":3451846656,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":177602560,"data_stored":177602560,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":2206203904,"data_stored":2206203904,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T08:16:00.299 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2 2024-09-18T08:16:00.342 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:16:00.342 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:16:00.343 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2' 2024-09-18T08:16:00.350 INFO:tasks.radosbench.radosbench.0.smithi007.stdout: 72 16 87396 87380 75.8332 66.6875 0.0237021 0.0131813 2024-09-18T08:16:00.444 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2 2024-09-18T08:16:00.483 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:16:00.483 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:16:00.484 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2' 2024-09-18T08:16:00.584 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7 2024-09-18T08:16:00.615 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:16:00.616 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:16:00.616 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7' 2024-09-18T08:16:00.717 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2 2024-09-18T08:16:00.764 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:16:00.764 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:16:00.765 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2' 2024-09-18T08:16:00.866 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2 2024-09-18T08:16:00.915 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:16:00.916 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:16:00.916 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2' 2024-09-18T08:16:00.929 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:16:00 smithi007 ceph-mon[29459]: 6.6 scrub starts 2024-09-18T08:16:00.929 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:16:00 smithi007 ceph-mon[29459]: 6.6 scrub ok 2024-09-18T08:16:00.929 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:16:00 smithi007 ceph-mon[29459]: 1.3 scrub starts 2024-09-18T08:16:00.929 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:16:00 smithi007 ceph-mon[29459]: 1.3 scrub ok 2024-09-18T08:16:00.929 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:16:00 smithi007 ceph-mon[39249]: 6.6 scrub starts 2024-09-18T08:16:00.930 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:16:00 smithi007 ceph-mon[39249]: 6.6 scrub ok 2024-09-18T08:16:00.930 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:16:00 smithi007 ceph-mon[39249]: 1.3 scrub starts 2024-09-18T08:16:00.930 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:16:00 smithi007 ceph-mon[39249]: 1.3 scrub ok 2024-09-18T08:16:01.017 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3 2024-09-18T08:16:01.065 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:16:01.066 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:16:01.066 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3' 2024-09-18T08:16:01.155 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:16:00 smithi050 ceph-mon[32371]: 6.6 scrub starts 2024-09-18T08:16:01.155 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:16:00 smithi050 ceph-mon[32371]: 6.6 scrub ok 2024-09-18T08:16:01.156 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:16:00 smithi050 ceph-mon[32371]: 1.3 scrub starts 2024-09-18T08:16:01.156 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:16:00 smithi050 ceph-mon[32371]: 1.3 scrub ok 2024-09-18T08:16:01.167 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2 2024-09-18T08:16:01.214 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:16:01.214 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:16:01.215 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2' 2024-09-18T08:16:01.316 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7 2024-09-18T08:16:01.346 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:16:01.346 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:16:01.347 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7' 2024-09-18T08:16:01.350 INFO:tasks.radosbench.radosbench.0.smithi007.stdout: 73 16 88258 88242 75.5324 53.875 0.0292979 0.0132324 2024-09-18T08:16:01.399 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/779349ca-7593-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T08:16:01.447 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2 2024-09-18T08:16:01.462 INFO:teuthology.orchestra.run.smithi007.stdout:{ 2024-09-18T08:16:01.463 INFO:teuthology.orchestra.run.smithi007.stdout: "ops": [], 2024-09-18T08:16:01.463 INFO:teuthology.orchestra.run.smithi007.stdout: "complaint_time": 30, 2024-09-18T08:16:01.463 INFO:teuthology.orchestra.run.smithi007.stdout: "num_blocked_ops": 0 2024-09-18T08:16:01.463 INFO:teuthology.orchestra.run.smithi007.stdout:} 2024-09-18T08:16:01.508 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:16:01.509 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:16:01.510 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2' 2024-09-18T08:16:01.610 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1 2024-09-18T08:16:01.650 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:16:01.651 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:16:01.651 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1' 2024-09-18T08:16:01.752 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4 2024-09-18T08:16:01.772 INFO:teuthology.orchestra.run.smithi007.stderr:noscrub is set 2024-09-18T08:16:01.779 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:16:01.780 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:16:01.780 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4' 2024-09-18T08:16:01.818 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:16:01 smithi007 ceph-mon[29459]: from='client.? 172.21.15.7:0/666752146' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-18T08:16:01.819 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:16:01 smithi007 ceph-mon[29459]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-18T08:16:01.819 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:16:01 smithi007 ceph-mon[29459]: pgmap v655: 158 pgs: 3 active+remapped+backfilling, 2 active+remapped+backfill_wait, 1 active+remapped+backfill_toofull, 152 active+clean; 5.2 GiB data, 19 GiB used, 607 GiB / 626 GiB avail; 639 B/s rd, 60 MiB/s wr, 957 op/s; 24716/257544 objects misplaced (9.597%); 31 MiB/s, 497 objects/s recovering 2024-09-18T08:16:01.819 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:16:01 smithi007 ceph-mon[29459]: from='mgr.24470 172.21.15.7:0/2594390261' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-18T08:16:01.819 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:16:01 smithi007 ceph-mon[29459]: from='mgr.24470 172.21.15.7:0/2594390261' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-18T08:16:01.819 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:16:01 smithi007 ceph-mon[29459]: from='mgr.24470 172.21.15.7:0/2594390261' entity='mgr.y' 2024-09-18T08:16:01.819 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:16:01 smithi007 ceph-mon[39249]: from='client.? 172.21.15.7:0/666752146' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-18T08:16:01.819 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:16:01 smithi007 ceph-mon[39249]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-18T08:16:01.819 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:16:01 smithi007 ceph-mon[39249]: pgmap v655: 158 pgs: 3 active+remapped+backfilling, 2 active+remapped+backfill_wait, 1 active+remapped+backfill_toofull, 152 active+clean; 5.2 GiB data, 19 GiB used, 607 GiB / 626 GiB avail; 639 B/s rd, 60 MiB/s wr, 957 op/s; 24716/257544 objects misplaced (9.597%); 31 MiB/s, 497 objects/s recovering 2024-09-18T08:16:01.819 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:16:01 smithi007 ceph-mon[39249]: from='mgr.24470 172.21.15.7:0/2594390261' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-18T08:16:01.819 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:16:01 smithi007 ceph-mon[39249]: from='mgr.24470 172.21.15.7:0/2594390261' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-18T08:16:01.819 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:16:01 smithi007 ceph-mon[39249]: from='mgr.24470 172.21.15.7:0/2594390261' entity='mgr.y' 2024-09-18T08:16:01.881 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5 2024-09-18T08:16:01.909 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:16:01.909 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:16:01.910 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5' 2024-09-18T08:16:02.010 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1 2024-09-18T08:16:02.066 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:16:02.067 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:16:02.068 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1' 2024-09-18T08:16:02.133 INFO:teuthology.orchestra.run.smithi007.stdout:osd.0: {} 2024-09-18T08:16:02.133 INFO:teuthology.orchestra.run.smithi007.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-18T08:16:02.155 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:16:01 smithi050 ceph-mon[32371]: from='client.? 172.21.15.7:0/666752146' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-18T08:16:02.155 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:16:01 smithi050 ceph-mon[32371]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-18T08:16:02.155 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:16:01 smithi050 ceph-mon[32371]: pgmap v655: 158 pgs: 3 active+remapped+backfilling, 2 active+remapped+backfill_wait, 1 active+remapped+backfill_toofull, 152 active+clean; 5.2 GiB data, 19 GiB used, 607 GiB / 626 GiB avail; 639 B/s rd, 60 MiB/s wr, 957 op/s; 24716/257544 objects misplaced (9.597%); 31 MiB/s, 497 objects/s recovering 2024-09-18T08:16:02.156 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:16:01 smithi050 ceph-mon[32371]: from='mgr.24470 172.21.15.7:0/2594390261' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-18T08:16:02.156 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:16:01 smithi050 ceph-mon[32371]: from='mgr.24470 172.21.15.7:0/2594390261' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-18T08:16:02.156 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:16:01 smithi050 ceph-mon[32371]: from='mgr.24470 172.21.15.7:0/2594390261' entity='mgr.y' 2024-09-18T08:16:02.168 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6 2024-09-18T08:16:02.173 INFO:teuthology.orchestra.run.smithi007.stdout:osd.1: {} 2024-09-18T08:16:02.173 INFO:teuthology.orchestra.run.smithi007.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-18T08:16:02.195 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:16:02.195 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:16:02.196 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6' 2024-09-18T08:16:02.208 INFO:teuthology.orchestra.run.smithi007.stdout:osd.2: {} 2024-09-18T08:16:02.209 INFO:teuthology.orchestra.run.smithi007.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-18T08:16:02.247 INFO:teuthology.orchestra.run.smithi007.stdout:osd.3: {} 2024-09-18T08:16:02.247 INFO:teuthology.orchestra.run.smithi007.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-18T08:16:02.283 INFO:teuthology.orchestra.run.smithi007.stdout:osd.4: {} 2024-09-18T08:16:02.283 INFO:teuthology.orchestra.run.smithi007.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-18T08:16:02.297 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4 2024-09-18T08:16:02.318 INFO:teuthology.orchestra.run.smithi007.stdout:osd.5: {} 2024-09-18T08:16:02.318 INFO:teuthology.orchestra.run.smithi007.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-18T08:16:02.329 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:16:02.330 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:16:02.330 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4' 2024-09-18T08:16:02.350 INFO:tasks.radosbench.radosbench.0.smithi007.stdout: 74 16 89105 89089 75.227 52.9375 0.00598309 0.0132879 2024-09-18T08:16:02.356 INFO:teuthology.orchestra.run.smithi007.stdout:osd.6: {} 2024-09-18T08:16:02.356 INFO:teuthology.orchestra.run.smithi007.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-18T08:16:02.396 INFO:teuthology.orchestra.run.smithi007.stdout:osd.7: {} 2024-09-18T08:16:02.396 INFO:teuthology.orchestra.run.smithi007.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-18T08:16:02.430 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7 2024-09-18T08:16:02.463 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:16:02.463 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:16:02.463 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7' 2024-09-18T08:16:02.564 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7 2024-09-18T08:16:02.594 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:16:02.594 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:16:02.595 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7' 2024-09-18T08:16:02.696 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7 2024-09-18T08:16:02.729 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:16:02.730 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:16:02.730 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7' 2024-09-18T08:16:02.831 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6 2024-09-18T08:16:02.860 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:16:02.860 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:16:02.861 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6' 2024-09-18T08:16:02.961 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2 2024-09-18T08:16:03.010 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:16:03.011 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:16:03.011 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2' 2024-09-18T08:16:03.112 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4 2024-09-18T08:16:03.114 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:16:02 smithi007 ceph-mon[29459]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-18T08:16:03.115 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:16:02 smithi007 ceph-mon[29459]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-18T08:16:03.115 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:16:02 smithi007 ceph-mon[29459]: osdmap e344: 8 total, 8 up, 7 in 2024-09-18T08:16:03.115 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:16:02 smithi007 ceph-mon[29459]: from='client.? 172.21.15.7:0/1547102837' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-18T08:16:03.115 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:16:02 smithi007 ceph-mon[39249]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-18T08:16:03.115 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:16:02 smithi007 ceph-mon[39249]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-18T08:16:03.115 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:16:02 smithi007 ceph-mon[39249]: osdmap e344: 8 total, 8 up, 7 in 2024-09-18T08:16:03.115 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:16:02 smithi007 ceph-mon[39249]: from='client.? 172.21.15.7:0/1547102837' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-18T08:16:03.143 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:16:03.144 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:16:03.144 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4' 2024-09-18T08:16:03.155 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:16:02 smithi050 ceph-mon[32371]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-18T08:16:03.155 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:16:02 smithi050 ceph-mon[32371]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-18T08:16:03.155 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:16:02 smithi050 ceph-mon[32371]: osdmap e344: 8 total, 8 up, 7 in 2024-09-18T08:16:03.155 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:16:02 smithi050 ceph-mon[32371]: from='client.? 172.21.15.7:0/1547102837' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-18T08:16:03.245 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3 2024-09-18T08:16:03.282 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:16:03.283 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:16:03.283 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3' 2024-09-18T08:16:03.350 INFO:tasks.radosbench.radosbench.0.smithi007.stdout: 75 16 90181 90165 75.1206 67.25 0.0209258 0.0133067 2024-09-18T08:16:03.384 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7 2024-09-18T08:16:03.414 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:16:03.414 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:16:03.415 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7' 2024-09-18T08:16:03.515 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0 2024-09-18T08:16:03.569 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:16:03.569 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:16:03.570 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0' 2024-09-18T08:16:03.671 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7 2024-09-18T08:16:03.703 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:16:03.703 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:16:03.704 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7' 2024-09-18T08:16:03.804 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6 2024-09-18T08:16:03.834 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:16:03.835 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:16:03.835 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6' 2024-09-18T08:16:03.925 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:16:03 smithi007 ceph-mon[39249]: pgmap v657: 158 pgs: 3 active+remapped+backfilling, 2 active+remapped+backfill_wait, 1 active+remapped+backfill_toofull, 152 active+clean; 5.3 GiB data, 19 GiB used, 606 GiB / 626 GiB avail; 1.3 KiB/s rd, 41 MiB/s wr, 660 op/s; 24716/260217 objects misplaced (9.498%); 14 MiB/s, 219 objects/s recovering 2024-09-18T08:16:03.925 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:16:03 smithi007 ceph-mon[39249]: from='mgr.24470 172.21.15.7:0/2594390261' entity='mgr.y' 2024-09-18T08:16:03.925 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:16:03 smithi007 ceph-mon[39249]: from='mgr.24470 172.21.15.7:0/2594390261' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-18T08:16:03.925 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:16:03 smithi007 ceph-mon[29459]: pgmap v657: 158 pgs: 3 active+remapped+backfilling, 2 active+remapped+backfill_wait, 1 active+remapped+backfill_toofull, 152 active+clean; 5.3 GiB data, 19 GiB used, 606 GiB / 626 GiB avail; 1.3 KiB/s rd, 41 MiB/s wr, 660 op/s; 24716/260217 objects misplaced (9.498%); 14 MiB/s, 219 objects/s recovering 2024-09-18T08:16:03.926 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:16:03 smithi007 ceph-mon[29459]: from='mgr.24470 172.21.15.7:0/2594390261' entity='mgr.y' 2024-09-18T08:16:03.926 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:16:03 smithi007 ceph-mon[29459]: from='mgr.24470 172.21.15.7:0/2594390261' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-18T08:16:03.936 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1 2024-09-18T08:16:03.974 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:16:03.975 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:16:03.975 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1' 2024-09-18T08:16:04.076 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4 2024-09-18T08:16:04.108 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:16:04.109 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:16:04.109 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4' 2024-09-18T08:16:04.155 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:16:03 smithi050 ceph-mon[32371]: pgmap v657: 158 pgs: 3 active+remapped+backfilling, 2 active+remapped+backfill_wait, 1 active+remapped+backfill_toofull, 152 active+clean; 5.3 GiB data, 19 GiB used, 606 GiB / 626 GiB avail; 1.3 KiB/s rd, 41 MiB/s wr, 660 op/s; 24716/260217 objects misplaced (9.498%); 14 MiB/s, 219 objects/s recovering 2024-09-18T08:16:04.155 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:16:03 smithi050 ceph-mon[32371]: from='mgr.24470 172.21.15.7:0/2594390261' entity='mgr.y' 2024-09-18T08:16:04.155 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:16:03 smithi050 ceph-mon[32371]: from='mgr.24470 172.21.15.7:0/2594390261' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-18T08:16:04.210 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1 2024-09-18T08:16:04.254 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:16:04.255 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:16:04.255 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1' 2024-09-18T08:16:04.350 INFO:tasks.radosbench.radosbench.0.smithi007.stdout: 76 16 91246 91230 75.0079 66.5625 0.00954093 0.0133269 2024-09-18T08:16:04.356 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1 2024-09-18T08:16:04.400 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:16:04.400 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:16:04.400 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1' 2024-09-18T08:16:04.501 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5 2024-09-18T08:16:04.530 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:16:04.531 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:16:04.531 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5' 2024-09-18T08:16:04.570 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid 779349ca-7593-11ef-bceb-c7b262605968 -- ceph daemon osd.2 dump_historic_ops 2024-09-18T08:16:04.633 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2 2024-09-18T08:16:04.681 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:16:04.681 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:16:04.681 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2' 2024-09-18T08:16:04.782 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7 2024-09-18T08:16:04.811 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:16:04.812 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:16:04.812 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7' 2024-09-18T08:16:04.906 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid 779349ca-7593-11ef-bceb-c7b262605968 -- ceph osd set nodeep-scrub 2024-09-18T08:16:04.913 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0 2024-09-18T08:16:05.001 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:16:05.002 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:16:05.002 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0' 2024-09-18T08:16:05.069 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/779349ca-7593-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T08:16:05.102 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1 2024-09-18T08:16:05.152 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:16:05.152 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:16:05.153 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1' 2024-09-18T08:16:05.238 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 6, 4, 5] out_osds: [7] dead_osds: [] live_osds: [0, 2, 3, 4, 5, 6, 7, 1] 2024-09-18T08:16:05.239 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-18T08:16:05.239 INFO:tasks.thrashosds.thrasher:Removing osd 4, in_osds are: [0, 1, 2, 3, 6, 4, 5] 2024-09-18T08:16:05.239 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid 779349ca-7593-11ef-bceb-c7b262605968 -- ceph osd out 4 2024-09-18T08:16:05.254 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5 2024-09-18T08:16:05.283 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:16:05.284 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:16:05.284 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5' 2024-09-18T08:16:05.353 INFO:tasks.radosbench.radosbench.0.smithi007.stdout: 77 16 92151 92135 74.7679 56.5625 0.0111211 0.0133691 2024-09-18T08:16:05.385 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2 2024-09-18T08:16:05.415 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:16:05.415 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:16:05.416 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2' 2024-09-18T08:16:05.480 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:16:05 smithi007 ceph-mon[29459]: osdmap e345: 8 total, 8 up, 7 in 2024-09-18T08:16:05.480 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:16:05 smithi007 ceph-mon[39249]: osdmap e345: 8 total, 8 up, 7 in 2024-09-18T08:16:05.517 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4 2024-09-18T08:16:05.546 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:16:05.547 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:16:05.547 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4' 2024-09-18T08:16:05.606 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid 779349ca-7593-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-18T08:16:05.648 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2 2024-09-18T08:16:05.655 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:16:05 smithi050 ceph-mon[32371]: osdmap e345: 8 total, 8 up, 7 in 2024-09-18T08:16:05.713 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:16:05.714 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:16:05.715 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2' 2024-09-18T08:16:05.816 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1 2024-09-18T08:16:05.865 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:16:05.865 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:16:05.866 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1' 2024-09-18T08:16:05.966 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7 2024-09-18T08:16:05.994 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:16:05.995 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:16:05.995 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7' 2024-09-18T08:16:06.096 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6 2024-09-18T08:16:06.126 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:16:06.127 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:16:06.127 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6' 2024-09-18T08:16:06.228 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7 2024-09-18T08:16:06.261 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:16:06.261 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:16:06.262 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7' 2024-09-18T08:16:06.353 INFO:tasks.radosbench.radosbench.0.smithi007.stdout: 78 16 93046 93030 74.5264 55.9375 0.00774358 0.0134121 2024-09-18T08:16:06.362 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3 2024-09-18T08:16:06.385 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:16:06.385 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:16:06.386 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3' 2024-09-18T08:16:06.486 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6 2024-09-18T08:16:06.515 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:16:06.515 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:16:06.516 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6' 2024-09-18T08:16:06.617 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7 2024-09-18T08:16:06.650 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:16:06.650 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:16:06.651 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7' 2024-09-18T08:16:06.655 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:16:06 smithi050 ceph-mon[32371]: purged_snaps scrub starts 2024-09-18T08:16:06.655 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:16:06 smithi050 ceph-mon[32371]: purged_snaps scrub ok 2024-09-18T08:16:06.655 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:16:06 smithi050 ceph-mon[32371]: pgmap v659: 158 pgs: 3 active+remapped+backfilling, 2 active+remapped+backfill_wait, 1 active+remapped+backfill_toofull, 152 active+clean; 5.5 GiB data, 20 GiB used, 606 GiB / 626 GiB avail; 669 B/s rd, 67 MiB/s wr, 1.08k op/s; 24017/272112 objects misplaced (8.826%); 30 MiB/s, 486 objects/s recovering 2024-09-18T08:16:06.655 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:16:06 smithi050 ceph-mon[32371]: osdmap e346: 8 total, 8 up, 7 in 2024-09-18T08:16:06.697 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:16:06 smithi007 ceph-mon[29459]: purged_snaps scrub starts 2024-09-18T08:16:06.698 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:16:06 smithi007 ceph-mon[29459]: purged_snaps scrub ok 2024-09-18T08:16:06.698 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:16:06 smithi007 ceph-mon[29459]: pgmap v659: 158 pgs: 3 active+remapped+backfilling, 2 active+remapped+backfill_wait, 1 active+remapped+backfill_toofull, 152 active+clean; 5.5 GiB data, 20 GiB used, 606 GiB / 626 GiB avail; 669 B/s rd, 67 MiB/s wr, 1.08k op/s; 24017/272112 objects misplaced (8.826%); 30 MiB/s, 486 objects/s recovering 2024-09-18T08:16:06.698 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:16:06 smithi007 ceph-mon[29459]: osdmap e346: 8 total, 8 up, 7 in 2024-09-18T08:16:06.698 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:16:06 smithi007 ceph-mon[39249]: purged_snaps scrub starts 2024-09-18T08:16:06.698 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:16:06 smithi007 ceph-mon[39249]: purged_snaps scrub ok 2024-09-18T08:16:06.698 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:16:06 smithi007 ceph-mon[39249]: pgmap v659: 158 pgs: 3 active+remapped+backfilling, 2 active+remapped+backfill_wait, 1 active+remapped+backfill_toofull, 152 active+clean; 5.5 GiB data, 20 GiB used, 606 GiB / 626 GiB avail; 669 B/s rd, 67 MiB/s wr, 1.08k op/s; 24017/272112 objects misplaced (8.826%); 30 MiB/s, 486 objects/s recovering 2024-09-18T08:16:06.698 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:16:06 smithi007 ceph-mon[39249]: osdmap e346: 8 total, 8 up, 7 in 2024-09-18T08:16:06.752 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4 2024-09-18T08:16:06.784 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:16:06.784 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:16:06.785 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4' 2024-09-18T08:16:06.886 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6 2024-09-18T08:16:06.915 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:16:06.915 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:16:06.916 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6' 2024-09-18T08:16:06.980 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Sep 18 08:16:06 smithi007 ceph-779349ca-7593-11ef-bceb-c7b262605968-mgr-y[29688]: ::ffff:172.21.15.50 - - [18/Sep/2024:08:16:06] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-09-18T08:16:07.017 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1 2024-09-18T08:16:07.061 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:16:07.061 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:16:07.062 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1' 2024-09-18T08:16:07.162 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0 2024-09-18T08:16:07.202 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:16:07.202 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:16:07.203 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0' 2024-09-18T08:16:07.303 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0 2024-09-18T08:16:07.335 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:16:07.336 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:16:07.336 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0' 2024-09-18T08:16:07.351 INFO:tasks.radosbench.radosbench.0.smithi007.stdout: 79 16 93829 93813 74.2025 48.9375 0.0277763 0.0134678 2024-09-18T08:16:07.437 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6 2024-09-18T08:16:07.466 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:16:07.467 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:16:07.467 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6' 2024-09-18T08:16:07.568 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7 2024-09-18T08:16:07.595 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:16:07.596 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:16:07.596 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7' 2024-09-18T08:16:07.697 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0 2024-09-18T08:16:07.729 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:16:07.729 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:16:07.729 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0' 2024-09-18T08:16:07.830 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3 2024-09-18T08:16:07.869 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:16:07.870 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:16:07.870 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3' 2024-09-18T08:16:07.971 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6 2024-09-18T08:16:08.000 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:16:08.000 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:16:08.001 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6' 2024-09-18T08:16:08.102 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2 2024-09-18T08:16:08.144 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:16:08.145 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:16:08.145 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2' 2024-09-18T08:16:08.246 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2 2024-09-18T08:16:08.287 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:16:08.287 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:16:08.288 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2' 2024-09-18T08:16:08.351 INFO:tasks.radosbench.radosbench.0.smithi007.stdout:2024-09-18T08:16:08.350793+0000 min lat: 0.00117701 max lat: 2.16978 avg lat: 0.013502 2024-09-18T08:16:08.351 INFO:tasks.radosbench.radosbench.0.smithi007.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-09-18T08:16:08.351 INFO:tasks.radosbench.radosbench.0.smithi007.stdout: 80 16 94801 94785 74.0343 60.75 0.0233226 0.013502 2024-09-18T08:16:08.389 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3 2024-09-18T08:16:08.429 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:16:08.430 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:16:08.430 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3' 2024-09-18T08:16:08.531 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7 2024-09-18T08:16:08.559 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:16:08.560 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:16:08.560 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7' 2024-09-18T08:16:08.655 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:16:08 smithi050 ceph-mon[32371]: pgmap v661: 158 pgs: 3 active+remapped+backfilling, 2 active+remapped+backfill_wait, 1 active+remapped+backfill_toofull, 152 active+clean; 5.5 GiB data, 20 GiB used, 606 GiB / 626 GiB avail; 852 B/s rd, 51 MiB/s wr, 809 op/s; 24017/272112 objects misplaced (8.826%); 21 MiB/s, 339 objects/s recovering 2024-09-18T08:16:08.662 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3 2024-09-18T08:16:08.705 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:16:08.706 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:16:08.706 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3' 2024-09-18T08:16:08.713 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:16:08 smithi007 ceph-mon[39249]: pgmap v661: 158 pgs: 3 active+remapped+backfilling, 2 active+remapped+backfill_wait, 1 active+remapped+backfill_toofull, 152 active+clean; 5.5 GiB data, 20 GiB used, 606 GiB / 626 GiB avail; 852 B/s rd, 51 MiB/s wr, 809 op/s; 24017/272112 objects misplaced (8.826%); 21 MiB/s, 339 objects/s recovering 2024-09-18T08:16:08.713 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:16:08 smithi007 ceph-mon[29459]: pgmap v661: 158 pgs: 3 active+remapped+backfilling, 2 active+remapped+backfill_wait, 1 active+remapped+backfill_toofull, 152 active+clean; 5.5 GiB data, 20 GiB used, 606 GiB / 626 GiB avail; 852 B/s rd, 51 MiB/s wr, 809 op/s; 24017/272112 objects misplaced (8.826%); 21 MiB/s, 339 objects/s recovering 2024-09-18T08:16:08.768 INFO:teuthology.orchestra.run.smithi007.stdout:ERROR: (22) Invalid argument 2024-09-18T08:16:08.768 INFO:teuthology.orchestra.run.smithi007.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-18T08:16:08.807 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1 2024-09-18T08:16:08.847 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:16:08.848 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:16:08.848 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1' 2024-09-18T08:16:08.949 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0 2024-09-18T08:16:08.992 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:16:08.993 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:16:08.993 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0' 2024-09-18T08:16:09.094 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3 2024-09-18T08:16:09.140 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:16:09.141 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:16:09.141 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3' 2024-09-18T08:16:09.242 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2 2024-09-18T08:16:09.286 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:16:09.287 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:16:09.288 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2' 2024-09-18T08:16:09.351 INFO:tasks.radosbench.radosbench.0.smithi007.stdout: 81 16 95843 95827 73.9243 65.125 0.004875 0.0135228 2024-09-18T08:16:09.389 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5 2024-09-18T08:16:09.421 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:16:09.422 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:16:09.422 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5' 2024-09-18T08:16:09.522 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0 2024-09-18T08:16:09.548 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:16:09.549 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:16:09.549 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0' 2024-09-18T08:16:09.649 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2 2024-09-18T08:16:09.688 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:16:09.689 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:16:09.689 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2' 2024-09-18T08:16:09.790 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3 2024-09-18T08:16:09.837 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:16:09.838 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:16:09.838 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3' 2024-09-18T08:16:09.905 INFO:journalctl@ceph.iscsi.iscsi.a.smithi050.stdout:Sep 18 08:16:09 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-iscsi-iscsi-a[57782]: debug there is no tcmu-runner data available 2024-09-18T08:16:09.939 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2 2024-09-18T08:16:09.980 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:16:09.981 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:16:09.981 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2' 2024-09-18T08:16:10.082 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2 2024-09-18T08:16:10.119 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:16:10.120 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:16:10.120 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2' 2024-09-18T08:16:10.221 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0 2024-09-18T08:16:10.247 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:16:10.247 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:16:10.248 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0' 2024-09-18T08:16:10.348 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6 2024-09-18T08:16:10.351 INFO:tasks.radosbench.radosbench.0.smithi007.stdout: 82 16 96776 96760 73.7338 58.3125 0.0124795 0.0135569 2024-09-18T08:16:10.378 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:16:10.379 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:16:10.379 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6' 2024-09-18T08:16:10.480 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1 2024-09-18T08:16:10.518 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:16:10.519 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:16:10.519 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1' 2024-09-18T08:16:10.620 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2 2024-09-18T08:16:10.649 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:16:10 smithi007 ceph-mon[29459]: pgmap v662: 158 pgs: 3 active+remapped+backfilling, 2 active+remapped+backfill_wait, 1 active+remapped+backfill_toofull, 152 active+clean; 5.7 GiB data, 21 GiB used, 605 GiB / 626 GiB avail; 1.3 KiB/s rd, 69 MiB/s wr, 1.10k op/s; 23830/282636 objects misplaced (8.431%); 24 MiB/s, 377 objects/s recovering 2024-09-18T08:16:10.649 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:16:10 smithi007 ceph-mon[29459]: osdmap e347: 8 total, 8 up, 7 in 2024-09-18T08:16:10.649 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:16:10 smithi007 ceph-mon[29459]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-18T08:16:10.649 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:16:10 smithi007 ceph-mon[39249]: pgmap v662: 158 pgs: 3 active+remapped+backfilling, 2 active+remapped+backfill_wait, 1 active+remapped+backfill_toofull, 152 active+clean; 5.7 GiB data, 21 GiB used, 605 GiB / 626 GiB avail; 1.3 KiB/s rd, 69 MiB/s wr, 1.10k op/s; 23830/282636 objects misplaced (8.431%); 24 MiB/s, 377 objects/s recovering 2024-09-18T08:16:10.649 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:16:10 smithi007 ceph-mon[39249]: osdmap e347: 8 total, 8 up, 7 in 2024-09-18T08:16:10.650 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:16:10 smithi007 ceph-mon[39249]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-18T08:16:10.655 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:16:10 smithi050 ceph-mon[32371]: pgmap v662: 158 pgs: 3 active+remapped+backfilling, 2 active+remapped+backfill_wait, 1 active+remapped+backfill_toofull, 152 active+clean; 5.7 GiB data, 21 GiB used, 605 GiB / 626 GiB avail; 1.3 KiB/s rd, 69 MiB/s wr, 1.10k op/s; 23830/282636 objects misplaced (8.431%); 24 MiB/s, 377 objects/s recovering 2024-09-18T08:16:10.655 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:16:10 smithi050 ceph-mon[32371]: osdmap e347: 8 total, 8 up, 7 in 2024-09-18T08:16:10.655 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:16:10 smithi050 ceph-mon[32371]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-18T08:16:10.661 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:16:10.661 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:16:10.661 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2' 2024-09-18T08:16:10.706 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid 779349ca-7593-11ef-bceb-c7b262605968 -- ceph daemon osd.3 dump_ops_in_flight 2024-09-18T08:16:10.748 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/779349ca-7593-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T08:16:10.762 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1 2024-09-18T08:16:10.800 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/779349ca-7593-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T08:16:10.832 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/779349ca-7593-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T08:16:10.844 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:16:10.845 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:16:10.845 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1' 2024-09-18T08:16:10.946 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7 2024-09-18T08:16:10.976 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:16:10.977 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:16:10.977 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7' 2024-09-18T08:16:11.078 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4 2024-09-18T08:16:11.106 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:16:11.107 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:16:11.108 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4' 2024-09-18T08:16:11.209 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7 2024-09-18T08:16:11.240 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:16:11.241 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:16:11.242 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7' 2024-09-18T08:16:11.342 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5 2024-09-18T08:16:11.353 INFO:tasks.radosbench.radosbench.0.smithi007.stdout: 83 16 97708 97692 73.5465 58.25 0.023589 0.0135911 2024-09-18T08:16:11.369 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:16:11.370 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:16:11.370 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5' 2024-09-18T08:16:11.471 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5 2024-09-18T08:16:11.501 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:16:11.501 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:16:11.502 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5' 2024-09-18T08:16:11.603 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2 2024-09-18T08:16:11.656 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:16:11.657 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:16:11.658 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2' 2024-09-18T08:16:11.758 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4 2024-09-18T08:16:11.786 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:16:11.787 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:16:11.787 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4' 2024-09-18T08:16:11.888 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0 2024-09-18T08:16:11.935 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:16:11.935 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:16:11.936 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0' 2024-09-18T08:16:12.036 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4 2024-09-18T08:16:12.066 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:16:12.066 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:16:12.066 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4' 2024-09-18T08:16:12.162 INFO:teuthology.orchestra.run.smithi007.stdout:osd.0: {} 2024-09-18T08:16:12.162 INFO:teuthology.orchestra.run.smithi007.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-18T08:16:12.167 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4 2024-09-18T08:16:12.193 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:16:12.194 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:16:12.194 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4' 2024-09-18T08:16:12.208 INFO:teuthology.orchestra.run.smithi007.stdout:osd.1: {} 2024-09-18T08:16:12.208 INFO:teuthology.orchestra.run.smithi007.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-18T08:16:12.260 INFO:teuthology.orchestra.run.smithi007.stdout:osd.2: {} 2024-09-18T08:16:12.260 INFO:teuthology.orchestra.run.smithi007.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-18T08:16:12.295 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5 2024-09-18T08:16:12.300 INFO:teuthology.orchestra.run.smithi007.stdout:osd.3: {} 2024-09-18T08:16:12.301 INFO:teuthology.orchestra.run.smithi007.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-18T08:16:12.322 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:16:12.322 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:16:12.323 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5' 2024-09-18T08:16:12.340 INFO:teuthology.orchestra.run.smithi007.stdout:osd.4: {} 2024-09-18T08:16:12.340 INFO:teuthology.orchestra.run.smithi007.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-18T08:16:12.352 INFO:tasks.radosbench.radosbench.0.smithi007.stdout: 84 16 98598 98582 73.3331 55.625 0.0158123 0.0136298 2024-09-18T08:16:12.384 INFO:teuthology.orchestra.run.smithi007.stdout:osd.5: {} 2024-09-18T08:16:12.384 INFO:teuthology.orchestra.run.smithi007.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-18T08:16:12.414 INFO:teuthology.orchestra.run.smithi007.stderr:marked out osd.4. 2024-09-18T08:16:12.417 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:16:12 smithi007 ceph-779349ca-7593-11ef-bceb-c7b262605968-mon-a[29455]: 2024-09-18T08:16:12.294+0000 7fcd2b595640 -1 mon.a@0(leader).osd e347 definitely_dead 0 2024-09-18T08:16:12.423 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6 2024-09-18T08:16:12.433 INFO:teuthology.orchestra.run.smithi007.stdout:osd.6: {} 2024-09-18T08:16:12.433 INFO:teuthology.orchestra.run.smithi007.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-18T08:16:12.446 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:16:12.447 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:16:12.447 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6' 2024-09-18T08:16:12.482 INFO:teuthology.orchestra.run.smithi007.stdout:osd.7: {} 2024-09-18T08:16:12.482 INFO:teuthology.orchestra.run.smithi007.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-18T08:16:12.548 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7 2024-09-18T08:16:12.571 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:16:12.571 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:16:12.572 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7' 2024-09-18T08:16:12.672 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4 2024-09-18T08:16:12.700 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:16:12.700 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:16:12.701 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4' 2024-09-18T08:16:12.730 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:16:12 smithi007 ceph-mon[29459]: pgmap v664: 158 pgs: 3 active+remapped+backfilling, 2 active+remapped+backfill_wait, 1 active+remapped+backfill_toofull, 152 active+clean; 5.8 GiB data, 21 GiB used, 605 GiB / 626 GiB avail; 731 B/s rd, 52 MiB/s wr, 839 op/s; 22885/286155 objects misplaced (7.997%); 32 MiB/s, 507 objects/s recovering 2024-09-18T08:16:12.730 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:16:12 smithi007 ceph-mon[29459]: from='client.? 172.21.15.7:0/274200980' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-18T08:16:12.730 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:16:12 smithi007 ceph-mon[29459]: from='client.? 172.21.15.7:0/924444237' entity='client.admin' cmd={"prefix": "osd out", "ids": ["4"]} : dispatch 2024-09-18T08:16:12.730 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:16:12 smithi007 ceph-mon[29459]: from='client.? ' entity='client.admin' cmd={"prefix": "osd out", "ids": ["4"]} : dispatch 2024-09-18T08:16:12.730 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:16:12 smithi007 ceph-mon[29459]: Client client.admin marked osd.4 out, while it was still marked up 2024-09-18T08:16:12.730 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:16:12 smithi007 ceph-mon[39249]: pgmap v664: 158 pgs: 3 active+remapped+backfilling, 2 active+remapped+backfill_wait, 1 active+remapped+backfill_toofull, 152 active+clean; 5.8 GiB data, 21 GiB used, 605 GiB / 626 GiB avail; 731 B/s rd, 52 MiB/s wr, 839 op/s; 22885/286155 objects misplaced (7.997%); 32 MiB/s, 507 objects/s recovering 2024-09-18T08:16:12.731 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:16:12 smithi007 ceph-mon[39249]: from='client.? 172.21.15.7:0/274200980' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-18T08:16:12.731 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:16:12 smithi007 ceph-mon[39249]: from='client.? 172.21.15.7:0/924444237' entity='client.admin' cmd={"prefix": "osd out", "ids": ["4"]} : dispatch 2024-09-18T08:16:12.731 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:16:12 smithi007 ceph-mon[39249]: from='client.? ' entity='client.admin' cmd={"prefix": "osd out", "ids": ["4"]} : dispatch 2024-09-18T08:16:12.731 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:16:12 smithi007 ceph-mon[39249]: Client client.admin marked osd.4 out, while it was still marked up 2024-09-18T08:16:12.801 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7 2024-09-18T08:16:12.827 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:16:12.827 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:16:12.828 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7' 2024-09-18T08:16:12.905 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:16:12 smithi050 ceph-mon[32371]: pgmap v664: 158 pgs: 3 active+remapped+backfilling, 2 active+remapped+backfill_wait, 1 active+remapped+backfill_toofull, 152 active+clean; 5.8 GiB data, 21 GiB used, 605 GiB / 626 GiB avail; 731 B/s rd, 52 MiB/s wr, 839 op/s; 22885/286155 objects misplaced (7.997%); 32 MiB/s, 507 objects/s recovering 2024-09-18T08:16:12.905 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:16:12 smithi050 ceph-mon[32371]: from='client.? 172.21.15.7:0/274200980' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-18T08:16:12.905 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:16:12 smithi050 ceph-mon[32371]: from='client.? 172.21.15.7:0/924444237' entity='client.admin' cmd={"prefix": "osd out", "ids": ["4"]} : dispatch 2024-09-18T08:16:12.905 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:16:12 smithi050 ceph-mon[32371]: from='client.? ' entity='client.admin' cmd={"prefix": "osd out", "ids": ["4"]} : dispatch 2024-09-18T08:16:12.905 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:16:12 smithi050 ceph-mon[32371]: Client client.admin marked osd.4 out, while it was still marked up 2024-09-18T08:16:12.929 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7 2024-09-18T08:16:12.951 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:16:12.952 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:16:12.953 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7' 2024-09-18T08:16:13.053 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2 2024-09-18T08:16:13.082 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:16:13.083 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:16:13.083 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2' 2024-09-18T08:16:13.184 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2 2024-09-18T08:16:13.207 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:16:13.208 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:16:13.209 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2' 2024-09-18T08:16:13.309 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2 2024-09-18T08:16:13.338 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:16:13.339 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:16:13.339 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2' 2024-09-18T08:16:13.352 INFO:tasks.radosbench.radosbench.0.smithi007.stdout: 85 16 98650 98634 72.5087 3.25 0.0679157 0.0136369 2024-09-18T08:16:13.432 INFO:teuthology.orchestra.run.smithi007.stderr:nodeep-scrub is set 2024-09-18T08:16:13.440 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2 2024-09-18T08:16:13.481 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:16:13.481 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:16:13.482 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2' 2024-09-18T08:16:13.583 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6 2024-09-18T08:16:13.607 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:16:13.607 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:16:13.608 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6' 2024-09-18T08:16:13.709 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3 2024-09-18T08:16:13.730 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:16:13 smithi007 ceph-mon[29459]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd out", "ids": ["4"]}]': finished 2024-09-18T08:16:13.730 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:16:13 smithi007 ceph-mon[29459]: osdmap e348: 8 total, 8 up, 6 in 2024-09-18T08:16:13.730 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:16:13 smithi007 ceph-mon[29459]: from='client.? 172.21.15.7:0/947667032' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-18T08:16:13.730 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:16:13 smithi007 ceph-mon[29459]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-18T08:16:13.730 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:16:13 smithi007 ceph-mon[29459]: Health check failed: Reduced data availability: 10 pgs inactive, 10 pgs peering (PG_AVAILABILITY) 2024-09-18T08:16:13.730 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:16:13 smithi007 ceph-mon[39249]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd out", "ids": ["4"]}]': finished 2024-09-18T08:16:13.730 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:16:13 smithi007 ceph-mon[39249]: osdmap e348: 8 total, 8 up, 6 in 2024-09-18T08:16:13.730 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:16:13 smithi007 ceph-mon[39249]: from='client.? 172.21.15.7:0/947667032' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-18T08:16:13.730 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:16:13 smithi007 ceph-mon[39249]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-18T08:16:13.730 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:16:13 smithi007 ceph-mon[39249]: Health check failed: Reduced data availability: 10 pgs inactive, 10 pgs peering (PG_AVAILABILITY) 2024-09-18T08:16:13.735 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:16:13.736 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:16:13.736 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3' 2024-09-18T08:16:13.837 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5 2024-09-18T08:16:13.862 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:16:13.863 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:16:13.863 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5' 2024-09-18T08:16:13.905 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:16:13 smithi050 ceph-mon[32371]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd out", "ids": ["4"]}]': finished 2024-09-18T08:16:13.905 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:16:13 smithi050 ceph-mon[32371]: osdmap e348: 8 total, 8 up, 6 in 2024-09-18T08:16:13.905 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:16:13 smithi050 ceph-mon[32371]: from='client.? 172.21.15.7:0/947667032' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-18T08:16:13.905 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:16:13 smithi050 ceph-mon[32371]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-18T08:16:13.905 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:16:13 smithi050 ceph-mon[32371]: Health check failed: Reduced data availability: 10 pgs inactive, 10 pgs peering (PG_AVAILABILITY) 2024-09-18T08:16:13.964 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6 2024-09-18T08:16:13.989 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:16:13.990 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:16:13.990 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6' 2024-09-18T08:16:14.091 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7 2024-09-18T08:16:14.115 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:16:14.115 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:16:14.116 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7' 2024-09-18T08:16:14.217 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3 2024-09-18T08:16:14.242 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:16:14.243 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:16:14.244 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3' 2024-09-18T08:16:14.344 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0 2024-09-18T08:16:14.352 INFO:tasks.radosbench.radosbench.0.smithi007.stdout: 86 16 98650 98634 71.6657 0 - 0.0136369 2024-09-18T08:16:14.367 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:16:14.367 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:16:14.368 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0' 2024-09-18T08:16:14.469 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0 2024-09-18T08:16:14.520 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:16:14.520 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:16:14.521 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0' 2024-09-18T08:16:14.621 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1 2024-09-18T08:16:14.644 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:16:14.644 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:16:14.645 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1' 2024-09-18T08:16:14.730 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:16:14 smithi007 ceph-mon[29459]: pgmap v666: 158 pgs: 19 peering, 3 active+remapped+backfilling, 2 active+remapped+backfill_wait, 1 active+remapped+backfill_toofull, 133 active+clean; 5.9 GiB data, 17 GiB used, 519 GiB / 536 GiB avail; 1.3 KiB/s rd, 43 MiB/s wr, 690 op/s; 22885/288645 objects misplaced (7.928%); 18 MiB/s, 288 objects/s recovering 2024-09-18T08:16:14.730 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:16:14 smithi007 ceph-mon[29459]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-18T08:16:14.730 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:16:14 smithi007 ceph-mon[29459]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-18T08:16:14.730 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:16:14 smithi007 ceph-mon[29459]: osdmap e349: 8 total, 8 up, 6 in 2024-09-18T08:16:14.731 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:16:14 smithi007 ceph-mon[39249]: pgmap v666: 158 pgs: 19 peering, 3 active+remapped+backfilling, 2 active+remapped+backfill_wait, 1 active+remapped+backfill_toofull, 133 active+clean; 5.9 GiB data, 17 GiB used, 519 GiB / 536 GiB avail; 1.3 KiB/s rd, 43 MiB/s wr, 690 op/s; 22885/288645 objects misplaced (7.928%); 18 MiB/s, 288 objects/s recovering 2024-09-18T08:16:14.731 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:16:14 smithi007 ceph-mon[39249]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-18T08:16:14.731 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:16:14 smithi007 ceph-mon[39249]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-18T08:16:14.731 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:16:14 smithi007 ceph-mon[39249]: osdmap e349: 8 total, 8 up, 6 in 2024-09-18T08:16:14.745 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1 2024-09-18T08:16:14.789 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:16:14.791 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:16:14.792 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1' 2024-09-18T08:16:14.892 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1 2024-09-18T08:16:14.905 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:16:14 smithi050 ceph-mon[32371]: pgmap v666: 158 pgs: 19 peering, 3 active+remapped+backfilling, 2 active+remapped+backfill_wait, 1 active+remapped+backfill_toofull, 133 active+clean; 5.9 GiB data, 17 GiB used, 519 GiB / 536 GiB avail; 1.3 KiB/s rd, 43 MiB/s wr, 690 op/s; 22885/288645 objects misplaced (7.928%); 18 MiB/s, 288 objects/s recovering 2024-09-18T08:16:14.905 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:16:14 smithi050 ceph-mon[32371]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-18T08:16:14.905 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:16:14 smithi050 ceph-mon[32371]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-18T08:16:14.905 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:16:14 smithi050 ceph-mon[32371]: osdmap e349: 8 total, 8 up, 6 in 2024-09-18T08:16:14.930 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:16:14.931 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:16:14.931 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1' 2024-09-18T08:16:15.032 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6 2024-09-18T08:16:15.062 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:16:15.063 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:16:15.063 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6' 2024-09-18T08:16:15.164 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4 2024-09-18T08:16:15.194 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:16:15.195 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:16:15.195 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4' 2024-09-18T08:16:15.296 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4 2024-09-18T08:16:15.325 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:16:15.326 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:16:15.326 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4' 2024-09-18T08:16:15.353 INFO:tasks.radosbench.radosbench.0.smithi007.stdout: 87 16 99338 99322 71.3362 21.5 0.00939974 0.014013 2024-09-18T08:16:15.427 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4 2024-09-18T08:16:15.450 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:16:15.450 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:16:15.450 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4' 2024-09-18T08:16:15.551 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2 2024-09-18T08:16:15.601 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:16:15.602 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:16:15.602 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2' 2024-09-18T08:16:15.703 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4 2024-09-18T08:16:15.730 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:16:15 smithi007 ceph-mon[29459]: 10.0 starting backfill to osd.0 from (0'0,0'0] MAX to 347'6278 2024-09-18T08:16:15.731 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:16:15 smithi007 ceph-mon[29459]: osdmap e350: 8 total, 8 up, 6 in 2024-09-18T08:16:15.731 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:16:15 smithi007 ceph-mon[29459]: 10.9 continuing backfill to osd.5 from (325'1300,347'6296] 10:9eb81a5c:::benchmark_data_smithi007_148834_object65296:head to 347'6296 2024-09-18T08:16:15.731 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:16:15 smithi007 ceph-mon[29459]: 10.e starting backfill to osd.5 from (0'0,0'0] MAX to 347'6074 2024-09-18T08:16:15.731 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:16:15 smithi007 ceph-mon[29459]: 10.4 starting backfill to osd.1 from (0'0,0'0] MAX to 347'6146 2024-09-18T08:16:15.731 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:16:15 smithi007 ceph-mon[29459]: 10.1 starting backfill to osd.2 from (0'0,0'0] MAX to 347'6314 2024-09-18T08:16:15.731 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:16:15 smithi007 ceph-mon[29459]: 10.c starting backfill to osd.6 from (0'0,0'0] MAX to 347'6051 2024-09-18T08:16:15.731 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:16:15 smithi007 ceph-mon[29459]: 10.d starting backfill to osd.3 from (0'0,0'0] MAX to 347'6050 2024-09-18T08:16:15.731 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:16:15 smithi007 ceph-mon[29459]: 10.2 starting backfill to osd.3 from (0'0,0'0] MAX to 347'6125 2024-09-18T08:16:15.731 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:16:15 smithi007 ceph-mon[29459]: 10.6 continuing backfill to osd.5 from (326'1800,347'6112] MIN to 347'6112 2024-09-18T08:16:15.731 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:16:15 smithi007 ceph-mon[29459]: purged_snaps scrub starts 2024-09-18T08:16:15.731 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:16:15 smithi007 ceph-mon[29459]: purged_snaps scrub ok 2024-09-18T08:16:15.732 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:16:15 smithi007 ceph-mon[39249]: 10.0 starting backfill to osd.0 from (0'0,0'0] MAX to 347'6278 2024-09-18T08:16:15.732 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:16:15 smithi007 ceph-mon[39249]: osdmap e350: 8 total, 8 up, 6 in 2024-09-18T08:16:15.732 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:16:15 smithi007 ceph-mon[39249]: 10.9 continuing backfill to osd.5 from (325'1300,347'6296] 10:9eb81a5c:::benchmark_data_smithi007_148834_object65296:head to 347'6296 2024-09-18T08:16:15.732 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:16:15 smithi007 ceph-mon[39249]: 10.e starting backfill to osd.5 from (0'0,0'0] MAX to 347'6074 2024-09-18T08:16:15.732 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:16:15 smithi007 ceph-mon[39249]: 10.4 starting backfill to osd.1 from (0'0,0'0] MAX to 347'6146 2024-09-18T08:16:15.732 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:16:15 smithi007 ceph-mon[39249]: 10.1 starting backfill to osd.2 from (0'0,0'0] MAX to 347'6314 2024-09-18T08:16:15.732 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:16:15 smithi007 ceph-mon[39249]: 10.c starting backfill to osd.6 from (0'0,0'0] MAX to 347'6051 2024-09-18T08:16:15.732 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:16:15 smithi007 ceph-mon[39249]: 10.d starting backfill to osd.3 from (0'0,0'0] MAX to 347'6050 2024-09-18T08:16:15.732 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:16:15 smithi007 ceph-mon[39249]: 10.2 starting backfill to osd.3 from (0'0,0'0] MAX to 347'6125 2024-09-18T08:16:15.732 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:16:15 smithi007 ceph-mon[39249]: 10.6 continuing backfill to osd.5 from (326'1800,347'6112] MIN to 347'6112 2024-09-18T08:16:15.732 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:16:15 smithi007 ceph-mon[39249]: purged_snaps scrub starts 2024-09-18T08:16:15.732 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:16:15 smithi007 ceph-mon[39249]: purged_snaps scrub ok 2024-09-18T08:16:15.733 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:16:15.733 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:16:15.733 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4' 2024-09-18T08:16:15.834 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3 2024-09-18T08:16:15.881 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:16:15.882 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:16:15.883 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3' 2024-09-18T08:16:15.905 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:16:15 smithi050 ceph-mon[32371]: 10.0 starting backfill to osd.0 from (0'0,0'0] MAX to 347'6278 2024-09-18T08:16:15.905 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:16:15 smithi050 ceph-mon[32371]: osdmap e350: 8 total, 8 up, 6 in 2024-09-18T08:16:15.906 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:16:15 smithi050 ceph-mon[32371]: 10.9 continuing backfill to osd.5 from (325'1300,347'6296] 10:9eb81a5c:::benchmark_data_smithi007_148834_object65296:head to 347'6296 2024-09-18T08:16:15.906 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:16:15 smithi050 ceph-mon[32371]: 10.e starting backfill to osd.5 from (0'0,0'0] MAX to 347'6074 2024-09-18T08:16:15.906 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:16:15 smithi050 ceph-mon[32371]: 10.4 starting backfill to osd.1 from (0'0,0'0] MAX to 347'6146 2024-09-18T08:16:15.906 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:16:15 smithi050 ceph-mon[32371]: 10.1 starting backfill to osd.2 from (0'0,0'0] MAX to 347'6314 2024-09-18T08:16:15.906 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:16:15 smithi050 ceph-mon[32371]: 10.c starting backfill to osd.6 from (0'0,0'0] MAX to 347'6051 2024-09-18T08:16:15.906 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:16:15 smithi050 ceph-mon[32371]: 10.d starting backfill to osd.3 from (0'0,0'0] MAX to 347'6050 2024-09-18T08:16:15.906 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:16:15 smithi050 ceph-mon[32371]: 10.2 starting backfill to osd.3 from (0'0,0'0] MAX to 347'6125 2024-09-18T08:16:15.906 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:16:15 smithi050 ceph-mon[32371]: 10.6 continuing backfill to osd.5 from (326'1800,347'6112] MIN to 347'6112 2024-09-18T08:16:15.906 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:16:15 smithi050 ceph-mon[32371]: purged_snaps scrub starts 2024-09-18T08:16:15.906 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:16:15 smithi050 ceph-mon[32371]: purged_snaps scrub ok 2024-09-18T08:16:15.984 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7 2024-09-18T08:16:16.012 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:16:16.013 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:16:16.013 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7' 2024-09-18T08:16:16.114 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6 2024-09-18T08:16:16.141 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:16:16.142 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:16:16.142 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6' 2024-09-18T08:16:16.243 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7 2024-09-18T08:16:16.274 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:16:16.275 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:16:16.275 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7' 2024-09-18T08:16:16.353 INFO:tasks.radosbench.radosbench.0.smithi007.stdout: 88 16 100109 100093 71.0732 48.1875 0.0438671 0.0140646 2024-09-18T08:16:16.376 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2 2024-09-18T08:16:16.418 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:16:16.418 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:16:16.419 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2' 2024-09-18T08:16:16.520 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6 2024-09-18T08:16:16.542 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:16:16.542 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:16:16.543 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6' 2024-09-18T08:16:16.644 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3 2024-09-18T08:16:16.689 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:16:16.689 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:16:16.690 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3' 2024-09-18T08:16:16.700 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:16:16 smithi007 ceph-mon[29459]: pgmap v669: 158 pgs: 1 activating+remapped, 2 remapped+peering, 19 peering, 3 active+remapped+backfilling, 3 active+remapped+backfill_wait, 2 active+remapped+backfill_toofull, 128 active+clean; 6.0 GiB data, 18 GiB used, 519 GiB / 536 GiB avail; 48 MiB/s wr, 773 op/s; 43806/296559 objects misplaced (14.771%); 55 MiB/s, 889 objects/s recovering 2024-09-18T08:16:16.700 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:16:16 smithi007 ceph-mon[29459]: Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 2 pgs backfill_toofull (PG_BACKFILL_FULL) 2024-09-18T08:16:16.701 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:16:16 smithi007 ceph-mon[29459]: osdmap e351: 8 total, 8 up, 6 in 2024-09-18T08:16:16.701 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:16:16 smithi007 ceph-mon[39249]: pgmap v669: 158 pgs: 1 activating+remapped, 2 remapped+peering, 19 peering, 3 active+remapped+backfilling, 3 active+remapped+backfill_wait, 2 active+remapped+backfill_toofull, 128 active+clean; 6.0 GiB data, 18 GiB used, 519 GiB / 536 GiB avail; 48 MiB/s wr, 773 op/s; 43806/296559 objects misplaced (14.771%); 55 MiB/s, 889 objects/s recovering 2024-09-18T08:16:16.701 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:16:16 smithi007 ceph-mon[39249]: Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 2 pgs backfill_toofull (PG_BACKFILL_FULL) 2024-09-18T08:16:16.701 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:16:16 smithi007 ceph-mon[39249]: osdmap e351: 8 total, 8 up, 6 in 2024-09-18T08:16:16.791 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1 2024-09-18T08:16:16.836 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:16:16.837 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:16:16.837 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1' 2024-09-18T08:16:16.905 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:16:16 smithi050 ceph-mon[32371]: pgmap v669: 158 pgs: 1 activating+remapped, 2 remapped+peering, 19 peering, 3 active+remapped+backfilling, 3 active+remapped+backfill_wait, 2 active+remapped+backfill_toofull, 128 active+clean; 6.0 GiB data, 18 GiB used, 519 GiB / 536 GiB avail; 48 MiB/s wr, 773 op/s; 43806/296559 objects misplaced (14.771%); 55 MiB/s, 889 objects/s recovering 2024-09-18T08:16:16.905 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:16:16 smithi050 ceph-mon[32371]: Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 2 pgs backfill_toofull (PG_BACKFILL_FULL) 2024-09-18T08:16:16.905 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:16:16 smithi050 ceph-mon[32371]: osdmap e351: 8 total, 8 up, 6 in 2024-09-18T08:16:16.938 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0 2024-09-18T08:16:16.979 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:16:16.979 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:16:16.980 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0' 2024-09-18T08:16:16.980 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Sep 18 08:16:16 smithi007 ceph-779349ca-7593-11ef-bceb-c7b262605968-mgr-y[29688]: ::ffff:172.21.15.50 - - [18/Sep/2024:08:16:16] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-09-18T08:16:17.081 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4 2024-09-18T08:16:17.111 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:16:17.112 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:16:17.113 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4' 2024-09-18T08:16:17.213 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6 2024-09-18T08:16:17.244 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:16:17.245 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:16:17.245 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6' 2024-09-18T08:16:17.346 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3 2024-09-18T08:16:17.353 INFO:tasks.radosbench.radosbench.0.smithi007.stdout: 89 16 100965 100949 70.8755 53.5 0.0135952 0.0141037 2024-09-18T08:16:17.380 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:16:17.380 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:16:17.380 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3' 2024-09-18T08:16:17.481 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5 2024-09-18T08:16:17.509 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:16:17.509 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:16:17.510 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5' 2024-09-18T08:16:17.610 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6 2024-09-18T08:16:17.633 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:16:17.633 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:16:17.633 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6' 2024-09-18T08:16:17.734 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4 2024-09-18T08:16:17.764 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:16:17.764 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:16:17.765 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4' 2024-09-18T08:16:17.866 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3 2024-09-18T08:16:17.888 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:16:17.889 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:16:17.889 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3' 2024-09-18T08:16:17.990 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7 2024-09-18T08:16:18.020 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:16:18.021 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:16:18.021 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7' 2024-09-18T08:16:18.122 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2 2024-09-18T08:16:18.165 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:16:18.166 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:16:18.166 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2' 2024-09-18T08:16:18.267 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0 2024-09-18T08:16:18.312 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:16:18.313 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:16:18.313 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0' 2024-09-18T08:16:18.353 INFO:tasks.radosbench.radosbench.0.smithi007.stdout:Total time run: 90.008 2024-09-18T08:16:18.353 INFO:tasks.radosbench.radosbench.0.smithi007.stdout:Total writes made: 101717 2024-09-18T08:16:18.353 INFO:tasks.radosbench.radosbench.0.smithi007.stdout:Write size: 65536 2024-09-18T08:16:18.354 INFO:tasks.radosbench.radosbench.0.smithi007.stdout:Object size: 65536 2024-09-18T08:16:18.354 INFO:tasks.radosbench.radosbench.0.smithi007.stdout:Bandwidth (MB/sec): 70.6305 2024-09-18T08:16:18.354 INFO:tasks.radosbench.radosbench.0.smithi007.stdout:Stddev Bandwidth: 34.6868 2024-09-18T08:16:18.354 INFO:tasks.radosbench.radosbench.0.smithi007.stdout:Max bandwidth (MB/sec): 198.625 2024-09-18T08:16:18.354 INFO:tasks.radosbench.radosbench.0.smithi007.stdout:Min bandwidth (MB/sec): 0 2024-09-18T08:16:18.354 INFO:tasks.radosbench.radosbench.0.smithi007.stdout:Average IOPS: 1130 2024-09-18T08:16:18.354 INFO:tasks.radosbench.radosbench.0.smithi007.stdout:Stddev IOPS: 555.002 2024-09-18T08:16:18.354 INFO:tasks.radosbench.radosbench.0.smithi007.stdout:Max IOPS: 3178 2024-09-18T08:16:18.354 INFO:tasks.radosbench.radosbench.0.smithi007.stdout:Min IOPS: 0 2024-09-18T08:16:18.354 INFO:tasks.radosbench.radosbench.0.smithi007.stdout:Average Latency(s): 0.0141539 2024-09-18T08:16:18.354 INFO:tasks.radosbench.radosbench.0.smithi007.stdout:Stddev Latency(s): 0.0508865 2024-09-18T08:16:18.354 INFO:tasks.radosbench.radosbench.0.smithi007.stdout:Max latency(s): 2.26083 2024-09-18T08:16:18.354 INFO:tasks.radosbench.radosbench.0.smithi007.stdout:Min latency(s): 0.00117701 2024-09-18T08:16:18.358 INFO:tasks.radosbench.radosbench.0.smithi007.stdout:Cleaning up (deleting benchmark objects) 2024-09-18T08:16:18.414 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6 2024-09-18T08:16:18.443 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:16:18.444 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:16:18.444 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6' 2024-09-18T08:16:18.468 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/779349ca-7593-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T08:16:18.545 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7 2024-09-18T08:16:18.573 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:16:18.574 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:16:18.574 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7' 2024-09-18T08:16:18.675 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2 2024-09-18T08:16:18.720 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:16:18.721 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:16:18.721 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2' 2024-09-18T08:16:18.730 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:16:18 smithi007 ceph-mon[29459]: pgmap v671: 158 pgs: 1 activating+remapped, 2 remapped+peering, 19 peering, 3 active+remapped+backfilling, 3 active+remapped+backfill_wait, 2 active+remapped+backfill_toofull, 128 active+clean; 6.0 GiB data, 18 GiB used, 519 GiB / 536 GiB avail; 44 MiB/s wr, 696 op/s; 43806/296559 objects misplaced (14.771%); 48 MiB/s, 774 objects/s recovering 2024-09-18T08:16:18.730 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:16:18 smithi007 ceph-mon[39249]: pgmap v671: 158 pgs: 1 activating+remapped, 2 remapped+peering, 19 peering, 3 active+remapped+backfilling, 3 active+remapped+backfill_wait, 2 active+remapped+backfill_toofull, 128 active+clean; 6.0 GiB data, 18 GiB used, 519 GiB / 536 GiB avail; 44 MiB/s wr, 696 op/s; 43806/296559 objects misplaced (14.771%); 48 MiB/s, 774 objects/s recovering 2024-09-18T08:16:18.822 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1 2024-09-18T08:16:18.868 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:16:18.868 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:16:18.869 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1' 2024-09-18T08:16:18.905 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:16:18 smithi050 ceph-mon[32371]: pgmap v671: 158 pgs: 1 activating+remapped, 2 remapped+peering, 19 peering, 3 active+remapped+backfilling, 3 active+remapped+backfill_wait, 2 active+remapped+backfill_toofull, 128 active+clean; 6.0 GiB data, 18 GiB used, 519 GiB / 536 GiB avail; 44 MiB/s wr, 696 op/s; 43806/296559 objects misplaced (14.771%); 48 MiB/s, 774 objects/s recovering 2024-09-18T08:16:18.970 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2 2024-09-18T08:16:19.017 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:16:19.018 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:16:19.018 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2' 2024-09-18T08:16:19.119 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2 2024-09-18T08:16:19.161 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:16:19.162 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:16:19.162 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2' 2024-09-18T08:16:19.263 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3 2024-09-18T08:16:19.304 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:16:19.304 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:16:19.305 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3' 2024-09-18T08:16:19.406 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1 2024-09-18T08:16:19.454 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:16:19.454 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:16:19.455 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1' 2024-09-18T08:16:19.480 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:16:19 smithi007 ceph-mon[29459]: purged_snaps scrub starts 2024-09-18T08:16:19.481 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:16:19 smithi007 ceph-mon[29459]: purged_snaps scrub ok 2024-09-18T08:16:19.481 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:16:19 smithi007 ceph-mon[29459]: osdmap e352: 8 total, 8 up, 6 in 2024-09-18T08:16:19.481 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:16:19 smithi007 ceph-mon[29459]: from='mgr.24470 172.21.15.7:0/2594390261' entity='mgr.y' 2024-09-18T08:16:19.481 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:16:19 smithi007 ceph-mon[29459]: from='mgr.24470 172.21.15.7:0/2594390261' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-18T08:16:19.481 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:16:19 smithi007 ceph-mon[29459]: osdmap e353: 8 total, 8 up, 6 in 2024-09-18T08:16:19.481 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:16:19 smithi007 ceph-mon[39249]: purged_snaps scrub starts 2024-09-18T08:16:19.482 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:16:19 smithi007 ceph-mon[39249]: purged_snaps scrub ok 2024-09-18T08:16:19.482 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:16:19 smithi007 ceph-mon[39249]: osdmap e352: 8 total, 8 up, 6 in 2024-09-18T08:16:19.482 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:16:19 smithi007 ceph-mon[39249]: from='mgr.24470 172.21.15.7:0/2594390261' entity='mgr.y' 2024-09-18T08:16:19.482 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:16:19 smithi007 ceph-mon[39249]: from='mgr.24470 172.21.15.7:0/2594390261' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-18T08:16:19.482 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:16:19 smithi007 ceph-mon[39249]: osdmap e353: 8 total, 8 up, 6 in 2024-09-18T08:16:19.556 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2 2024-09-18T08:16:19.591 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:16:19.592 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:16:19.593 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2' 2024-09-18T08:16:19.694 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5 2024-09-18T08:16:19.721 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:16:19.721 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:16:19.722 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5' 2024-09-18T08:16:19.823 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0 2024-09-18T08:16:19.853 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:16:19.853 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:16:19.854 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0' 2024-09-18T08:16:19.905 INFO:journalctl@ceph.iscsi.iscsi.a.smithi050.stdout:Sep 18 08:16:19 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-iscsi-iscsi-a[57782]: debug there is no tcmu-runner data available 2024-09-18T08:16:19.906 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:16:19 smithi050 ceph-mon[32371]: purged_snaps scrub starts 2024-09-18T08:16:19.906 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:16:19 smithi050 ceph-mon[32371]: purged_snaps scrub ok 2024-09-18T08:16:19.906 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:16:19 smithi050 ceph-mon[32371]: osdmap e352: 8 total, 8 up, 6 in 2024-09-18T08:16:19.906 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:16:19 smithi050 ceph-mon[32371]: from='mgr.24470 172.21.15.7:0/2594390261' entity='mgr.y' 2024-09-18T08:16:19.906 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:16:19 smithi050 ceph-mon[32371]: from='mgr.24470 172.21.15.7:0/2594390261' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-18T08:16:19.906 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:16:19 smithi050 ceph-mon[32371]: osdmap e353: 8 total, 8 up, 6 in 2024-09-18T08:16:19.954 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6 2024-09-18T08:16:19.981 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:16:19.982 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:16:19.982 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6' 2024-09-18T08:16:20.083 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4 2024-09-18T08:16:20.123 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:16:20.124 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:16:20.124 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4' 2024-09-18T08:16:20.225 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5 2024-09-18T08:16:20.250 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:16:20.251 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:16:20.252 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5' 2024-09-18T08:16:20.352 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2 2024-09-18T08:16:20.385 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:16:20.386 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:16:20.387 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2' 2024-09-18T08:16:20.448 INFO:teuthology.orchestra.run.smithi007.stdout:{ 2024-09-18T08:16:20.448 INFO:teuthology.orchestra.run.smithi007.stdout: "ops": [], 2024-09-18T08:16:20.448 INFO:teuthology.orchestra.run.smithi007.stdout: "num_ops": 0 2024-09-18T08:16:20.448 INFO:teuthology.orchestra.run.smithi007.stdout:} 2024-09-18T08:16:20.488 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0 2024-09-18T08:16:20.530 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:16:20.530 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:16:20.531 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0' 2024-09-18T08:16:20.632 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6 2024-09-18T08:16:20.659 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:16:20.660 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:16:20.660 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6' 2024-09-18T08:16:20.730 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:16:20 smithi007 ceph-mon[29459]: pgmap v674: 158 pgs: 1 active+clean+remapped, 1 activating+remapped, 4 active+remapped+backfilling, 3 active+remapped+backfill_wait, 4 active+remapped+backfill_toofull, 145 active+clean; 6.1 GiB data, 18 GiB used, 518 GiB / 536 GiB avail; 38 MiB/s wr, 652 op/s; 61817/300675 objects misplaced (20.559%); 53 MiB/s, 850 objects/s recovering 2024-09-18T08:16:20.730 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:16:20 smithi007 ceph-mon[29459]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-18T08:16:20.730 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:16:20 smithi007 ceph-mon[29459]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 10 pgs inactive, 10 pgs peering) 2024-09-18T08:16:20.730 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:16:20 smithi007 ceph-mon[29459]: osdmap e354: 8 total, 8 up, 6 in 2024-09-18T08:16:20.731 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:16:20 smithi007 ceph-mon[39249]: pgmap v674: 158 pgs: 1 active+clean+remapped, 1 activating+remapped, 4 active+remapped+backfilling, 3 active+remapped+backfill_wait, 4 active+remapped+backfill_toofull, 145 active+clean; 6.1 GiB data, 18 GiB used, 518 GiB / 536 GiB avail; 38 MiB/s wr, 652 op/s; 61817/300675 objects misplaced (20.559%); 53 MiB/s, 850 objects/s recovering 2024-09-18T08:16:20.731 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:16:20 smithi007 ceph-mon[39249]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-18T08:16:20.731 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:16:20 smithi007 ceph-mon[39249]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 10 pgs inactive, 10 pgs peering) 2024-09-18T08:16:20.731 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:16:20 smithi007 ceph-mon[39249]: osdmap e354: 8 total, 8 up, 6 in 2024-09-18T08:16:20.761 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3 2024-09-18T08:16:20.794 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:16:20.795 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:16:20.796 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3' 2024-09-18T08:16:20.897 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3 2024-09-18T08:16:20.905 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:16:20 smithi050 ceph-mon[32371]: pgmap v674: 158 pgs: 1 active+clean+remapped, 1 activating+remapped, 4 active+remapped+backfilling, 3 active+remapped+backfill_wait, 4 active+remapped+backfill_toofull, 145 active+clean; 6.1 GiB data, 18 GiB used, 518 GiB / 536 GiB avail; 38 MiB/s wr, 652 op/s; 61817/300675 objects misplaced (20.559%); 53 MiB/s, 850 objects/s recovering 2024-09-18T08:16:20.905 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:16:20 smithi050 ceph-mon[32371]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-18T08:16:20.905 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:16:20 smithi050 ceph-mon[32371]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 10 pgs inactive, 10 pgs peering) 2024-09-18T08:16:20.905 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:16:20 smithi050 ceph-mon[32371]: osdmap e354: 8 total, 8 up, 6 in 2024-09-18T08:16:20.928 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:16:20.928 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:16:20.929 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3' 2024-09-18T08:16:20.952 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid 779349ca-7593-11ef-bceb-c7b262605968 -- ceph osd unset noscrub 2024-09-18T08:16:21.030 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5 2024-09-18T08:16:21.055 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:16:21.056 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:16:21.056 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5' 2024-09-18T08:16:21.157 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2 2024-09-18T08:16:21.193 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:16:21.193 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:16:21.194 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2' 2024-09-18T08:16:21.295 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3 2024-09-18T08:16:21.329 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:16:21.329 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:16:21.330 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3' 2024-09-18T08:16:21.430 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4 2024-09-18T08:16:21.461 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:16:21.462 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:16:21.463 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4' 2024-09-18T08:16:21.564 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5 2024-09-18T08:16:21.592 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:16:21.592 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:16:21.593 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5' 2024-09-18T08:16:21.694 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4 2024-09-18T08:16:21.723 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:16:21.724 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:16:21.724 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4' 2024-09-18T08:16:21.825 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3 2024-09-18T08:16:21.871 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:16:21.872 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:16:21.872 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3' 2024-09-18T08:16:21.973 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5 2024-09-18T08:16:22.001 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:16:22.001 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:16:22.002 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5' 2024-09-18T08:16:22.103 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2 2024-09-18T08:16:22.139 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid 779349ca-7593-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-18T08:16:22.147 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:16:22.147 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:16:22.148 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2' 2024-09-18T08:16:22.249 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4 2024-09-18T08:16:22.276 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:16:22.277 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:16:22.277 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4' 2024-09-18T08:16:22.378 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6 2024-09-18T08:16:22.405 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:16:22.406 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:16:22.406 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6' 2024-09-18T08:16:22.480 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:16:22 smithi007 ceph-mon[29459]: pgmap v676: 158 pgs: 1 active+clean+remapped, 5 active+remapped+backfilling, 2 active+remapped+backfill_wait, 5 active+remapped+backfill_toofull, 145 active+clean; 6.2 GiB data, 18 GiB used, 518 GiB / 536 GiB avail; 1.0 KiB/s rd, 31 MiB/s wr, 627 op/s; 61215/303183 objects misplaced (20.191%); 31 MiB/s, 502 objects/s recovering 2024-09-18T08:16:22.480 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:16:22 smithi007 ceph-mon[29459]: osdmap e355: 8 total, 8 up, 6 in 2024-09-18T08:16:22.480 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:16:22 smithi007 ceph-mon[29459]: Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 5 pgs backfill_toofull (PG_BACKFILL_FULL) 2024-09-18T08:16:22.480 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:16:22 smithi007 ceph-mon[39249]: pgmap v676: 158 pgs: 1 active+clean+remapped, 5 active+remapped+backfilling, 2 active+remapped+backfill_wait, 5 active+remapped+backfill_toofull, 145 active+clean; 6.2 GiB data, 18 GiB used, 518 GiB / 536 GiB avail; 1.0 KiB/s rd, 31 MiB/s wr, 627 op/s; 61215/303183 objects misplaced (20.191%); 31 MiB/s, 502 objects/s recovering 2024-09-18T08:16:22.481 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:16:22 smithi007 ceph-mon[39249]: osdmap e355: 8 total, 8 up, 6 in 2024-09-18T08:16:22.481 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:16:22 smithi007 ceph-mon[39249]: Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 5 pgs backfill_toofull (PG_BACKFILL_FULL) 2024-09-18T08:16:22.507 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4 2024-09-18T08:16:22.535 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:16:22.535 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:16:22.536 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4' 2024-09-18T08:16:22.637 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5 2024-09-18T08:16:22.655 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:16:22 smithi050 ceph-mon[32371]: pgmap v676: 158 pgs: 1 active+clean+remapped, 5 active+remapped+backfilling, 2 active+remapped+backfill_wait, 5 active+remapped+backfill_toofull, 145 active+clean; 6.2 GiB data, 18 GiB used, 518 GiB / 536 GiB avail; 1.0 KiB/s rd, 31 MiB/s wr, 627 op/s; 61215/303183 objects misplaced (20.191%); 31 MiB/s, 502 objects/s recovering 2024-09-18T08:16:22.655 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:16:22 smithi050 ceph-mon[32371]: osdmap e355: 8 total, 8 up, 6 in 2024-09-18T08:16:22.656 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:16:22 smithi050 ceph-mon[32371]: Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 5 pgs backfill_toofull (PG_BACKFILL_FULL) 2024-09-18T08:16:22.665 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:16:22.666 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:16:22.667 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5' 2024-09-18T08:16:22.768 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3 2024-09-18T08:16:22.807 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:16:22.808 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:16:22.808 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3' 2024-09-18T08:16:22.909 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3 2024-09-18T08:16:22.934 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:16:22.934 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:16:22.935 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3' 2024-09-18T08:16:23.035 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6 2024-09-18T08:16:23.064 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:16:23.064 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:16:23.065 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6' 2024-09-18T08:16:23.166 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1 2024-09-18T08:16:23.206 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:16:23.207 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:16:23.207 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1' 2024-09-18T08:16:23.308 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7 2024-09-18T08:16:23.333 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:16:23.334 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:16:23.334 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7' 2024-09-18T08:16:23.435 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2 2024-09-18T08:16:23.477 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:16:23.477 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:16:23.477 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2' 2024-09-18T08:16:23.578 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6 2024-09-18T08:16:23.605 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:16:23.606 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:16:23.606 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6' 2024-09-18T08:16:23.707 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2 2024-09-18T08:16:23.750 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:16:23.751 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:16:23.751 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2' 2024-09-18T08:16:23.852 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0 2024-09-18T08:16:23.875 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:16:23.875 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:16:23.876 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0' 2024-09-18T08:16:23.977 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0 2024-09-18T08:16:24.015 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:16:24.016 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:16:24.017 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0' 2024-09-18T08:16:24.117 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3 2024-09-18T08:16:24.157 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:16:24.158 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:16:24.158 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3' 2024-09-18T08:16:24.259 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3 2024-09-18T08:16:24.288 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:16:24.289 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:16:24.290 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3' 2024-09-18T08:16:24.390 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3 2024-09-18T08:16:24.433 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:16:24.434 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:16:24.435 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3' 2024-09-18T08:16:24.535 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5 2024-09-18T08:16:24.565 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:16:24.565 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:16:24.566 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5' 2024-09-18T08:16:24.655 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:16:24 smithi050 ceph-mon[32371]: pgmap v678: 158 pgs: 1 active+clean+remapped, 5 active+remapped+backfilling, 2 active+remapped+backfill_wait, 5 active+remapped+backfill_toofull, 145 active+clean; 6.1 GiB data, 18 GiB used, 518 GiB / 536 GiB avail; 1.2 KiB/s rd, 33 MiB/s wr, 784 op/s; 59933/302058 objects misplaced (19.842%); 48 MiB/s, 763 objects/s recovering 2024-09-18T08:16:24.667 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0 2024-09-18T08:16:24.672 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 6, 5] out_osds: [7, 4] dead_osds: [] live_osds: [0, 2, 3, 4, 5, 6, 7, 1] 2024-09-18T08:16:24.673 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid 779349ca-7593-11ef-bceb-c7b262605968 -- ceph osd reweight 0 1 2024-09-18T08:16:24.693 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:16:24.693 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:16:24.694 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0' 2024-09-18T08:16:24.714 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:16:24 smithi007 ceph-mon[29459]: pgmap v678: 158 pgs: 1 active+clean+remapped, 5 active+remapped+backfilling, 2 active+remapped+backfill_wait, 5 active+remapped+backfill_toofull, 145 active+clean; 6.1 GiB data, 18 GiB used, 518 GiB / 536 GiB avail; 1.2 KiB/s rd, 33 MiB/s wr, 784 op/s; 59933/302058 objects misplaced (19.842%); 48 MiB/s, 763 objects/s recovering 2024-09-18T08:16:24.714 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:16:24 smithi007 ceph-mon[39249]: pgmap v678: 158 pgs: 1 active+clean+remapped, 5 active+remapped+backfilling, 2 active+remapped+backfill_wait, 5 active+remapped+backfill_toofull, 145 active+clean; 6.1 GiB data, 18 GiB used, 518 GiB / 536 GiB avail; 1.2 KiB/s rd, 33 MiB/s wr, 784 op/s; 59933/302058 objects misplaced (19.842%); 48 MiB/s, 763 objects/s recovering 2024-09-18T08:16:24.795 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3 2024-09-18T08:16:24.853 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:16:24.853 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:16:24.854 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3' 2024-09-18T08:16:24.955 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3 2024-09-18T08:16:24.987 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:16:24.988 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:16:24.988 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3' 2024-09-18T08:16:25.089 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6 2024-09-18T08:16:25.115 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:16:25.116 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:16:25.116 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6' 2024-09-18T08:16:25.217 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4 2024-09-18T08:16:25.242 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:16:25.242 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:16:25.243 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4' 2024-09-18T08:16:25.343 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0 2024-09-18T08:16:25.385 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:16:25.386 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:16:25.386 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0' 2024-09-18T08:16:25.487 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0 2024-09-18T08:16:25.523 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:16:25.523 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:16:25.524 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0' 2024-09-18T08:16:25.625 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0 2024-09-18T08:16:25.656 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:16:25.657 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:16:25.658 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0' 2024-09-18T08:16:25.758 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6 2024-09-18T08:16:25.783 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:16:25.784 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:16:25.785 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6' 2024-09-18T08:16:25.885 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6 2024-09-18T08:16:25.912 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:16:25.912 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:16:25.912 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6' 2024-09-18T08:16:26.014 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3 2024-09-18T08:16:26.050 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:16:26.051 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:16:26.051 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3' 2024-09-18T08:16:26.152 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5 2024-09-18T08:16:26.181 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:16:26.181 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:16:26.182 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5' 2024-09-18T08:16:26.282 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1 2024-09-18T08:16:26.305 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:16:26.305 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:16:26.306 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1' 2024-09-18T08:16:26.407 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2 2024-09-18T08:16:26.445 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:16:26.445 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:16:26.446 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2' 2024-09-18T08:16:26.546 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7 2024-09-18T08:16:26.576 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:16:26.576 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:16:26.577 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7' 2024-09-18T08:16:26.655 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:16:26 smithi050 ceph-mon[32371]: pgmap v679: 158 pgs: 5 active+remapped+backfilling, 6 active+remapped+backfill_toofull, 147 active+clean; 6.0 GiB data, 18 GiB used, 518 GiB / 536 GiB avail; 1023 B/s rd, 15 MiB/s wr, 830 op/s; 53996/294462 objects misplaced (18.337%); 71 MiB/s, 1.13k objects/s recovering 2024-09-18T08:16:26.677 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7 2024-09-18T08:16:26.697 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:16:26 smithi007 ceph-mon[29459]: pgmap v679: 158 pgs: 5 active+remapped+backfilling, 6 active+remapped+backfill_toofull, 147 active+clean; 6.0 GiB data, 18 GiB used, 518 GiB / 536 GiB avail; 1023 B/s rd, 15 MiB/s wr, 830 op/s; 53996/294462 objects misplaced (18.337%); 71 MiB/s, 1.13k objects/s recovering 2024-09-18T08:16:26.697 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:16:26 smithi007 ceph-mon[39249]: pgmap v679: 158 pgs: 5 active+remapped+backfilling, 6 active+remapped+backfill_toofull, 147 active+clean; 6.0 GiB data, 18 GiB used, 518 GiB / 536 GiB avail; 1023 B/s rd, 15 MiB/s wr, 830 op/s; 53996/294462 objects misplaced (18.337%); 71 MiB/s, 1.13k objects/s recovering 2024-09-18T08:16:26.706 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:16:26.707 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:16:26.707 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7' 2024-09-18T08:16:26.808 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0 2024-09-18T08:16:26.844 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:16:26.845 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:16:26.845 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0' 2024-09-18T08:16:26.946 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0 2024-09-18T08:16:26.980 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Sep 18 08:16:26 smithi007 ceph-779349ca-7593-11ef-bceb-c7b262605968-mgr-y[29688]: ::ffff:172.21.15.50 - - [18/Sep/2024:08:16:26] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-09-18T08:16:26.993 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:16:26.993 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:16:26.994 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0' 2024-09-18T08:16:27.095 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5 2024-09-18T08:16:27.126 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:16:27.126 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:16:27.127 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5' 2024-09-18T08:16:27.228 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0 2024-09-18T08:16:27.253 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:16:27.254 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:16:27.254 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0' 2024-09-18T08:16:27.355 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7 2024-09-18T08:16:27.380 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:16:27.380 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:16:27.381 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7' 2024-09-18T08:16:27.482 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0 2024-09-18T08:16:27.513 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:16:27.514 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:16:27.514 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0' 2024-09-18T08:16:27.615 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2 2024-09-18T08:16:27.641 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:16:27.641 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:16:27.642 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2' 2024-09-18T08:16:27.742 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6 2024-09-18T08:16:27.773 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:16:27.773 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:16:27.774 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6' 2024-09-18T08:16:27.875 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7 2024-09-18T08:16:27.901 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:16:27.902 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:16:27.902 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7' 2024-09-18T08:16:28.003 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6 2024-09-18T08:16:28.028 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:16:28.028 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:16:28.028 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6' 2024-09-18T08:16:28.129 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4 2024-09-18T08:16:28.157 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:16:28.157 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:16:28.158 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4' 2024-09-18T08:16:28.259 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0 2024-09-18T08:16:28.300 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:16:28.301 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:16:28.301 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0' 2024-09-18T08:16:28.402 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2 2024-09-18T08:16:28.431 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid 779349ca-7593-11ef-bceb-c7b262605968 -- ceph daemon osd.3 dump_blocked_ops 2024-09-18T08:16:28.458 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/779349ca-7593-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T08:16:28.471 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:16:28.472 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:16:28.472 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2' 2024-09-18T08:16:28.523 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/779349ca-7593-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T08:16:28.535 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/779349ca-7593-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T08:16:28.573 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7 2024-09-18T08:16:28.600 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:16:28.601 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:16:28.601 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7' 2024-09-18T08:16:28.702 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1 2024-09-18T08:16:28.730 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:16:28 smithi007 ceph-mon[39249]: pgmap v680: 158 pgs: 5 active+remapped+backfilling, 6 active+remapped+backfill_toofull, 147 active+clean; 6.0 GiB data, 18 GiB used, 518 GiB / 536 GiB avail; 767 B/s rd, 11 MiB/s wr, 623 op/s; 53996/294462 objects misplaced (18.337%); 53 MiB/s, 848 objects/s recovering 2024-09-18T08:16:28.730 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:16:28 smithi007 ceph-mon[39249]: purged_snaps scrub starts 2024-09-18T08:16:28.730 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:16:28 smithi007 ceph-mon[39249]: purged_snaps scrub ok 2024-09-18T08:16:28.730 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:16:28 smithi007 ceph-mon[29459]: pgmap v680: 158 pgs: 5 active+remapped+backfilling, 6 active+remapped+backfill_toofull, 147 active+clean; 6.0 GiB data, 18 GiB used, 518 GiB / 536 GiB avail; 767 B/s rd, 11 MiB/s wr, 623 op/s; 53996/294462 objects misplaced (18.337%); 53 MiB/s, 848 objects/s recovering 2024-09-18T08:16:28.730 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:16:28 smithi007 ceph-mon[29459]: purged_snaps scrub starts 2024-09-18T08:16:28.730 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:16:28 smithi007 ceph-mon[29459]: purged_snaps scrub ok 2024-09-18T08:16:28.752 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:16:28.753 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:16:28.753 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1' 2024-09-18T08:16:28.854 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6 2024-09-18T08:16:28.876 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:16:28.877 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:16:28.877 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6' 2024-09-18T08:16:28.905 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:16:28 smithi050 ceph-mon[32371]: pgmap v680: 158 pgs: 5 active+remapped+backfilling, 6 active+remapped+backfill_toofull, 147 active+clean; 6.0 GiB data, 18 GiB used, 518 GiB / 536 GiB avail; 767 B/s rd, 11 MiB/s wr, 623 op/s; 53996/294462 objects misplaced (18.337%); 53 MiB/s, 848 objects/s recovering 2024-09-18T08:16:28.905 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:16:28 smithi050 ceph-mon[32371]: purged_snaps scrub starts 2024-09-18T08:16:28.905 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:16:28 smithi050 ceph-mon[32371]: purged_snaps scrub ok 2024-09-18T08:16:28.979 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0 2024-09-18T08:16:29.026 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:16:29.027 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:16:29.027 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0' 2024-09-18T08:16:29.128 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6 2024-09-18T08:16:29.156 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:16:29.156 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:16:29.157 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6' 2024-09-18T08:16:29.258 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5 2024-09-18T08:16:29.289 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:16:29.289 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:16:29.290 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5' 2024-09-18T08:16:29.390 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3 2024-09-18T08:16:29.414 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:16:29.415 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:16:29.415 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3' 2024-09-18T08:16:29.516 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5 2024-09-18T08:16:29.544 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:16:29.545 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:16:29.545 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5' 2024-09-18T08:16:29.646 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4 2024-09-18T08:16:29.675 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:16:29.676 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:16:29.677 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4' 2024-09-18T08:16:29.730 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:16:29 smithi007 ceph-mon[29459]: osdmap e356: 8 total, 8 up, 6 in 2024-09-18T08:16:29.730 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:16:29 smithi007 ceph-mon[29459]: Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 6 pgs backfill_toofull (PG_BACKFILL_FULL) 2024-09-18T08:16:29.730 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:16:29 smithi007 ceph-mon[39249]: osdmap e356: 8 total, 8 up, 6 in 2024-09-18T08:16:29.730 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:16:29 smithi007 ceph-mon[39249]: Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 6 pgs backfill_toofull (PG_BACKFILL_FULL) 2024-09-18T08:16:29.777 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1 2024-09-18T08:16:29.819 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:16:29.820 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:16:29.820 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1' 2024-09-18T08:16:29.905 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:16:29 smithi050 ceph-mon[32371]: osdmap e356: 8 total, 8 up, 6 in 2024-09-18T08:16:29.905 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:16:29 smithi050 ceph-mon[32371]: Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 6 pgs backfill_toofull (PG_BACKFILL_FULL) 2024-09-18T08:16:29.906 INFO:journalctl@ceph.iscsi.iscsi.a.smithi050.stdout:Sep 18 08:16:29 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-iscsi-iscsi-a[57782]: debug there is no tcmu-runner data available 2024-09-18T08:16:29.921 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3 2024-09-18T08:16:29.963 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:16:29.964 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:16:29.964 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3' 2024-09-18T08:16:30.065 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6 2024-09-18T08:16:30.093 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:16:30.093 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:16:30.094 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6' 2024-09-18T08:16:30.195 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3 2024-09-18T08:16:30.232 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:16:30.233 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:16:30.233 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3' 2024-09-18T08:16:30.334 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6 2024-09-18T08:16:30.356 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:16:30.357 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:16:30.357 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6' 2024-09-18T08:16:30.458 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6 2024-09-18T08:16:30.486 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:16:30.487 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:16:30.487 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6' 2024-09-18T08:16:30.588 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3 2024-09-18T08:16:30.631 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:16:30.632 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:16:30.633 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3' 2024-09-18T08:16:30.730 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:16:30 smithi007 ceph-mon[29459]: pgmap v682: 158 pgs: 5 active+remapped+backfilling, 6 active+remapped+backfill_toofull, 147 active+clean; 5.9 GiB data, 18 GiB used, 519 GiB / 536 GiB avail; 1.2 KiB/s rd, 776 KiB/s wr, 648 op/s; 51623/288225 objects misplaced (17.911%); 55 MiB/s, 873 objects/s recovering 2024-09-18T08:16:30.730 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:16:30 smithi007 ceph-mon[29459]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-18T08:16:30.730 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:16:30 smithi007 ceph-mon[39249]: pgmap v682: 158 pgs: 5 active+remapped+backfilling, 6 active+remapped+backfill_toofull, 147 active+clean; 5.9 GiB data, 18 GiB used, 519 GiB / 536 GiB avail; 1.2 KiB/s rd, 776 KiB/s wr, 648 op/s; 51623/288225 objects misplaced (17.911%); 55 MiB/s, 873 objects/s recovering 2024-09-18T08:16:30.731 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:16:30 smithi007 ceph-mon[39249]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-18T08:16:30.734 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0 2024-09-18T08:16:30.772 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:16:30.773 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:16:30.773 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0' 2024-09-18T08:16:30.874 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3 2024-09-18T08:16:30.904 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:16:30.904 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:16:30.905 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3' 2024-09-18T08:16:30.905 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:16:30 smithi050 ceph-mon[32371]: pgmap v682: 158 pgs: 5 active+remapped+backfilling, 6 active+remapped+backfill_toofull, 147 active+clean; 5.9 GiB data, 18 GiB used, 519 GiB / 536 GiB avail; 1.2 KiB/s rd, 776 KiB/s wr, 648 op/s; 51623/288225 objects misplaced (17.911%); 55 MiB/s, 873 objects/s recovering 2024-09-18T08:16:30.905 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:16:30 smithi050 ceph-mon[32371]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-18T08:16:31.006 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0 2024-09-18T08:16:31.050 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:16:31.051 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:16:31.051 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0' 2024-09-18T08:16:31.152 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3 2024-09-18T08:16:31.191 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:16:31.192 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:16:31.192 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3' 2024-09-18T08:16:31.293 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3 2024-09-18T08:16:31.336 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:16:31.336 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:16:31.337 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3' 2024-09-18T08:16:31.437 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7 2024-09-18T08:16:31.465 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:16:31.466 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:16:31.466 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7' 2024-09-18T08:16:31.567 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3 2024-09-18T08:16:31.611 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:16:31.612 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:16:31.612 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3' 2024-09-18T08:16:31.713 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5 2024-09-18T08:16:31.740 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:16:31.741 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:16:31.741 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5' 2024-09-18T08:16:31.842 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2 2024-09-18T08:16:31.884 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:16:31.885 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:16:31.886 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2' 2024-09-18T08:16:31.987 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0 2024-09-18T08:16:32.024 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:16:32.025 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:16:32.025 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0' 2024-09-18T08:16:32.126 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5 2024-09-18T08:16:32.155 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:16:32.156 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:16:32.156 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5' 2024-09-18T08:16:32.257 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3 2024-09-18T08:16:32.296 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:16:32.296 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:16:32.297 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3' 2024-09-18T08:16:32.397 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4 2024-09-18T08:16:32.426 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:16:32.427 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:16:32.427 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4' 2024-09-18T08:16:32.438 INFO:teuthology.orchestra.run.smithi007.stderr:reweighted osd.0 to 1 (10000) 2024-09-18T08:16:32.528 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2 2024-09-18T08:16:32.566 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:16:32.567 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:16:32.567 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2' 2024-09-18T08:16:32.668 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3 2024-09-18T08:16:32.712 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:16:32 smithi007 ceph-mon[39249]: pgmap v683: 158 pgs: 5 active+remapped+backfilling, 6 active+remapped+backfill_toofull, 147 active+clean; 5.8 GiB data, 18 GiB used, 519 GiB / 536 GiB avail; 1 KiB/s rd, 621 KiB/s wr, 670 op/s; 48772/283668 objects misplaced (17.193%); 55 MiB/s, 887 objects/s recovering 2024-09-18T08:16:32.712 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:16:32 smithi007 ceph-mon[39249]: from='client.? 172.21.15.7:0/1098713728' entity='client.admin' cmd={"prefix": "osd reweight", "id": 0, "weight": 1.0} : dispatch 2024-09-18T08:16:32.712 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:16:32 smithi007 ceph-mon[39249]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 0, "weight": 1.0} : dispatch 2024-09-18T08:16:32.712 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:16:32 smithi007 ceph-mon[29459]: pgmap v683: 158 pgs: 5 active+remapped+backfilling, 6 active+remapped+backfill_toofull, 147 active+clean; 5.8 GiB data, 18 GiB used, 519 GiB / 536 GiB avail; 1 KiB/s rd, 621 KiB/s wr, 670 op/s; 48772/283668 objects misplaced (17.193%); 55 MiB/s, 887 objects/s recovering 2024-09-18T08:16:32.712 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:16:32 smithi007 ceph-mon[29459]: from='client.? 172.21.15.7:0/1098713728' entity='client.admin' cmd={"prefix": "osd reweight", "id": 0, "weight": 1.0} : dispatch 2024-09-18T08:16:32.712 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:16:32 smithi007 ceph-mon[29459]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 0, "weight": 1.0} : dispatch 2024-09-18T08:16:32.719 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:16:32.720 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:16:32.720 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3' 2024-09-18T08:16:32.821 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2 2024-09-18T08:16:32.861 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:16:32.862 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:16:32.863 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2' 2024-09-18T08:16:32.905 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:16:32 smithi050 ceph-mon[32371]: pgmap v683: 158 pgs: 5 active+remapped+backfilling, 6 active+remapped+backfill_toofull, 147 active+clean; 5.8 GiB data, 18 GiB used, 519 GiB / 536 GiB avail; 1 KiB/s rd, 621 KiB/s wr, 670 op/s; 48772/283668 objects misplaced (17.193%); 55 MiB/s, 887 objects/s recovering 2024-09-18T08:16:32.905 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:16:32 smithi050 ceph-mon[32371]: from='client.? 172.21.15.7:0/1098713728' entity='client.admin' cmd={"prefix": "osd reweight", "id": 0, "weight": 1.0} : dispatch 2024-09-18T08:16:32.905 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:16:32 smithi050 ceph-mon[32371]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 0, "weight": 1.0} : dispatch 2024-09-18T08:16:32.964 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2 2024-09-18T08:16:33.005 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:16:33.005 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:16:33.006 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2' 2024-09-18T08:16:33.106 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2 2024-09-18T08:16:33.145 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:16:33.146 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:16:33.146 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2' 2024-09-18T08:16:33.247 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0 2024-09-18T08:16:33.278 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:16:33.279 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:16:33.280 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0' 2024-09-18T08:16:33.381 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5 2024-09-18T08:16:33.410 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:16:33.411 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:16:33.411 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5' 2024-09-18T08:16:33.512 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2 2024-09-18T08:16:33.556 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:16:33.557 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:16:33.557 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2' 2024-09-18T08:16:33.658 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4 2024-09-18T08:16:33.685 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:16:33.685 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:16:33.686 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4' 2024-09-18T08:16:33.730 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:16:33 smithi007 ceph-mon[29459]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]': finished 2024-09-18T08:16:33.730 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:16:33 smithi007 ceph-mon[29459]: osdmap e357: 8 total, 8 up, 6 in 2024-09-18T08:16:33.730 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:16:33 smithi007 ceph-mon[39249]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]': finished 2024-09-18T08:16:33.730 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:16:33 smithi007 ceph-mon[39249]: osdmap e357: 8 total, 8 up, 6 in 2024-09-18T08:16:33.787 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0 2024-09-18T08:16:33.836 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:16:33.836 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:16:33.837 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0' 2024-09-18T08:16:33.905 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:16:33 smithi050 ceph-mon[32371]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]': finished 2024-09-18T08:16:33.905 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:16:33 smithi050 ceph-mon[32371]: osdmap e357: 8 total, 8 up, 6 in 2024-09-18T08:16:33.938 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6 2024-09-18T08:16:33.964 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:16:33.965 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:16:33.965 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6' 2024-09-18T08:16:34.047 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid 779349ca-7593-11ef-bceb-c7b262605968 -- ceph osd reweight 1 1 2024-09-18T08:16:34.067 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0 2024-09-18T08:16:34.141 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:16:34.142 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:16:34.142 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0' 2024-09-18T08:16:34.243 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4 2024-09-18T08:16:34.270 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:16:34.270 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:16:34.271 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4' 2024-09-18T08:16:34.372 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2 2024-09-18T08:16:34.419 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:16:34.419 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:16:34.419 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2' 2024-09-18T08:16:34.520 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2 2024-09-18T08:16:34.584 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:16:34.585 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:16:34.585 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2' 2024-09-18T08:16:34.608 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:16:34 smithi007 ceph-mon[39249]: pgmap v685: 158 pgs: 5 active+remapped+backfilling, 6 active+remapped+backfill_toofull, 147 active+clean; 5.7 GiB data, 18 GiB used, 519 GiB / 536 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 548 op/s; 47972/281310 objects misplaced (17.053%); 29 MiB/s, 456 objects/s recovering 2024-09-18T08:16:34.608 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:16:34 smithi007 ceph-mon[39249]: from='mgr.24470 172.21.15.7:0/2594390261' entity='mgr.y' 2024-09-18T08:16:34.608 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:16:34 smithi007 ceph-mon[39249]: from='mgr.24470 172.21.15.7:0/2594390261' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-18T08:16:34.609 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:16:34 smithi007 ceph-mon[29459]: pgmap v685: 158 pgs: 5 active+remapped+backfilling, 6 active+remapped+backfill_toofull, 147 active+clean; 5.7 GiB data, 18 GiB used, 519 GiB / 536 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 548 op/s; 47972/281310 objects misplaced (17.053%); 29 MiB/s, 456 objects/s recovering 2024-09-18T08:16:34.609 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:16:34 smithi007 ceph-mon[29459]: from='mgr.24470 172.21.15.7:0/2594390261' entity='mgr.y' 2024-09-18T08:16:34.609 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:16:34 smithi007 ceph-mon[29459]: from='mgr.24470 172.21.15.7:0/2594390261' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-18T08:16:34.687 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4 2024-09-18T08:16:34.713 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:16:34.714 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:16:34.714 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4' 2024-09-18T08:16:34.815 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5 2024-09-18T08:16:34.844 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:16:34.844 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:16:34.845 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5' 2024-09-18T08:16:34.905 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:16:34 smithi050 ceph-mon[32371]: pgmap v685: 158 pgs: 5 active+remapped+backfilling, 6 active+remapped+backfill_toofull, 147 active+clean; 5.7 GiB data, 18 GiB used, 519 GiB / 536 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 548 op/s; 47972/281310 objects misplaced (17.053%); 29 MiB/s, 456 objects/s recovering 2024-09-18T08:16:34.905 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:16:34 smithi050 ceph-mon[32371]: from='mgr.24470 172.21.15.7:0/2594390261' entity='mgr.y' 2024-09-18T08:16:34.905 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:16:34 smithi050 ceph-mon[32371]: from='mgr.24470 172.21.15.7:0/2594390261' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-18T08:16:34.946 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4 2024-09-18T08:16:34.976 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:16:34.976 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:16:34.977 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4' 2024-09-18T08:16:35.078 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5 2024-09-18T08:16:35.105 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:16:35.106 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:16:35.106 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5' 2024-09-18T08:16:35.207 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0 2024-09-18T08:16:35.247 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:16:35.248 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:16:35.248 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0' 2024-09-18T08:16:35.349 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3 2024-09-18T08:16:35.396 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:16:35.397 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:16:35.397 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3' 2024-09-18T08:16:35.474 INFO:teuthology.orchestra.run.smithi007.stderr:noscrub is unset 2024-09-18T08:16:35.499 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3 2024-09-18T08:16:35.541 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:16:35.542 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:16:35.542 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3' 2024-09-18T08:16:35.553 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:16:35 smithi007 ceph-mon[29459]: from='client.? 172.21.15.7:0/3673041745' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-18T08:16:35.553 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:16:35 smithi007 ceph-mon[29459]: from='client.? 172.21.15.7:0/3623281740' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-18T08:16:35.553 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:16:35 smithi007 ceph-mon[39249]: from='client.? 172.21.15.7:0/3673041745' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-18T08:16:35.553 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:16:35 smithi007 ceph-mon[39249]: from='client.? 172.21.15.7:0/3623281740' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-18T08:16:35.643 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6 2024-09-18T08:16:35.670 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:16:35.671 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:16:35.672 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6' 2024-09-18T08:16:35.773 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6 2024-09-18T08:16:35.799 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:16:35.800 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:16:35.800 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6' 2024-09-18T08:16:35.901 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6 2024-09-18T08:16:35.902 INFO:teuthology.orchestra.run.smithi007.stdout:osd.0: {} 2024-09-18T08:16:35.902 INFO:teuthology.orchestra.run.smithi007.stdout:osd.1: {} 2024-09-18T08:16:35.903 INFO:teuthology.orchestra.run.smithi007.stdout:osd.2: {} 2024-09-18T08:16:35.903 INFO:teuthology.orchestra.run.smithi007.stdout:osd.3: {} 2024-09-18T08:16:35.903 INFO:teuthology.orchestra.run.smithi007.stdout:osd.4: {} 2024-09-18T08:16:35.903 INFO:teuthology.orchestra.run.smithi007.stdout:osd.5: {} 2024-09-18T08:16:35.903 INFO:teuthology.orchestra.run.smithi007.stdout:osd.6: {} 2024-09-18T08:16:35.903 INFO:teuthology.orchestra.run.smithi007.stdout:osd.7: {} 2024-09-18T08:16:35.903 INFO:teuthology.orchestra.run.smithi007.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-18T08:16:35.903 INFO:teuthology.orchestra.run.smithi007.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-18T08:16:35.903 INFO:teuthology.orchestra.run.smithi007.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-18T08:16:35.903 INFO:teuthology.orchestra.run.smithi007.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-18T08:16:35.903 INFO:teuthology.orchestra.run.smithi007.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-18T08:16:35.904 INFO:teuthology.orchestra.run.smithi007.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-18T08:16:35.904 INFO:teuthology.orchestra.run.smithi007.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-18T08:16:35.904 INFO:teuthology.orchestra.run.smithi007.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-18T08:16:35.905 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:16:35 smithi050 ceph-mon[32371]: from='client.? 172.21.15.7:0/3673041745' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-18T08:16:35.905 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:16:35 smithi050 ceph-mon[32371]: from='client.? 172.21.15.7:0/3623281740' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-18T08:16:35.932 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:16:35.932 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:16:35.933 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6' 2024-09-18T08:16:36.033 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2 2024-09-18T08:16:36.073 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:16:36.074 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:16:36.074 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2' 2024-09-18T08:16:36.174 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0 2024-09-18T08:16:36.209 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:16:36.210 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:16:36.210 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0' 2024-09-18T08:16:36.311 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5 2024-09-18T08:16:36.333 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:16:36.334 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:16:36.334 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5' 2024-09-18T08:16:36.435 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3 2024-09-18T08:16:36.477 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:16:36.477 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:16:36.477 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3' 2024-09-18T08:16:36.578 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6 2024-09-18T08:16:36.608 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:16:36.609 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:16:36.609 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6' 2024-09-18T08:16:36.710 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7 2024-09-18T08:16:36.730 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Sep 18 08:16:36 smithi007 ceph-779349ca-7593-11ef-bceb-c7b262605968-mgr-y[29688]: ::ffff:172.21.15.50 - - [18/Sep/2024:08:16:36] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-09-18T08:16:36.732 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:16:36 smithi007 ceph-mon[29459]: pgmap v686: 158 pgs: 5 active+remapped+backfilling, 6 active+remapped+backfill_toofull, 147 active+clean; 5.5 GiB data, 17 GiB used, 519 GiB / 536 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 977 op/s; 43747/271002 objects misplaced (16.143%); 49 MiB/s, 777 objects/s recovering 2024-09-18T08:16:36.733 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:16:36 smithi007 ceph-mon[29459]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-18T08:16:36.733 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:16:36 smithi007 ceph-mon[29459]: from='client.? 172.21.15.7:0/3673041745' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-18T08:16:36.733 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:16:36 smithi007 ceph-mon[29459]: osdmap e358: 8 total, 8 up, 6 in 2024-09-18T08:16:36.733 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:16:36 smithi007 ceph-mon[39249]: pgmap v686: 158 pgs: 5 active+remapped+backfilling, 6 active+remapped+backfill_toofull, 147 active+clean; 5.5 GiB data, 17 GiB used, 519 GiB / 536 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 977 op/s; 43747/271002 objects misplaced (16.143%); 49 MiB/s, 777 objects/s recovering 2024-09-18T08:16:36.733 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:16:36 smithi007 ceph-mon[39249]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-18T08:16:36.734 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:16:36 smithi007 ceph-mon[39249]: from='client.? 172.21.15.7:0/3673041745' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-18T08:16:36.734 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:16:36 smithi007 ceph-mon[39249]: osdmap e358: 8 total, 8 up, 6 in 2024-09-18T08:16:36.737 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:16:36.738 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:16:36.738 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7' 2024-09-18T08:16:36.839 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7 2024-09-18T08:16:36.864 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:16:36.865 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:16:36.865 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7' 2024-09-18T08:16:36.905 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:16:36 smithi050 ceph-mon[32371]: pgmap v686: 158 pgs: 5 active+remapped+backfilling, 6 active+remapped+backfill_toofull, 147 active+clean; 5.5 GiB data, 17 GiB used, 519 GiB / 536 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 977 op/s; 43747/271002 objects misplaced (16.143%); 49 MiB/s, 777 objects/s recovering 2024-09-18T08:16:36.905 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:16:36 smithi050 ceph-mon[32371]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-18T08:16:36.905 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:16:36 smithi050 ceph-mon[32371]: from='client.? 172.21.15.7:0/3673041745' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-18T08:16:36.905 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:16:36 smithi050 ceph-mon[32371]: osdmap e358: 8 total, 8 up, 6 in 2024-09-18T08:16:36.966 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3 2024-09-18T08:16:37.006 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:16:37.007 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:16:37.007 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3' 2024-09-18T08:16:37.108 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7 2024-09-18T08:16:37.137 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:16:37.138 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:16:37.138 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7' 2024-09-18T08:16:37.239 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6 2024-09-18T08:16:37.265 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:16:37.266 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:16:37.266 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6' 2024-09-18T08:16:37.367 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3 2024-09-18T08:16:37.405 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:16:37.406 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:16:37.406 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3' 2024-09-18T08:16:37.507 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0 2024-09-18T08:16:37.555 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:16:37.555 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:16:37.556 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0' 2024-09-18T08:16:37.657 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2 2024-09-18T08:16:37.696 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:16:37.697 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:16:37.697 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2' 2024-09-18T08:16:37.798 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2 2024-09-18T08:16:37.832 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:16:37.833 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:16:37.833 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2' 2024-09-18T08:16:37.934 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5 2024-09-18T08:16:37.962 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:16:37.963 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:16:37.963 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5' 2024-09-18T08:16:38.064 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6 2024-09-18T08:16:38.092 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:16:38.093 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:16:38.093 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6' 2024-09-18T08:16:38.194 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0 2024-09-18T08:16:38.228 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:16:38.229 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:16:38.230 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0' 2024-09-18T08:16:38.330 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1 2024-09-18T08:16:38.376 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:16:38.376 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:16:38.377 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1' 2024-09-18T08:16:38.477 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0 2024-09-18T08:16:38.518 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:16:38.519 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:16:38.519 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0' 2024-09-18T08:16:38.620 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5 2024-09-18T08:16:38.642 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:16:38.643 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:16:38.643 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5' 2024-09-18T08:16:38.730 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:16:38 smithi007 ceph-mon[39249]: pgmap v688: 158 pgs: 5 active+remapped+backfilling, 6 active+remapped+backfill_toofull, 147 active+clean; 5.5 GiB data, 17 GiB used, 519 GiB / 536 GiB avail; 639 B/s rd, 0 B/s wr, 717 op/s; 43747/271002 objects misplaced (16.143%); 38 MiB/s, 600 objects/s recovering 2024-09-18T08:16:38.730 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:16:38 smithi007 ceph-mon[29459]: pgmap v688: 158 pgs: 5 active+remapped+backfilling, 6 active+remapped+backfill_toofull, 147 active+clean; 5.5 GiB data, 17 GiB used, 519 GiB / 536 GiB avail; 639 B/s rd, 0 B/s wr, 717 op/s; 43747/271002 objects misplaced (16.143%); 38 MiB/s, 600 objects/s recovering 2024-09-18T08:16:38.743 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5 2024-09-18T08:16:38.773 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:16:38.773 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:16:38.774 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5' 2024-09-18T08:16:38.875 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7 2024-09-18T08:16:38.904 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:16:38.905 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:16:38 smithi050 ceph-mon[32371]: pgmap v688: 158 pgs: 5 active+remapped+backfilling, 6 active+remapped+backfill_toofull, 147 active+clean; 5.5 GiB data, 17 GiB used, 519 GiB / 536 GiB avail; 639 B/s rd, 0 B/s wr, 717 op/s; 43747/271002 objects misplaced (16.143%); 38 MiB/s, 600 objects/s recovering 2024-09-18T08:16:38.905 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:16:38.906 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7' 2024-09-18T08:16:39.006 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7 2024-09-18T08:16:39.035 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:16:39.036 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:16:39.036 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7' 2024-09-18T08:16:39.137 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7 2024-09-18T08:16:39.163 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:16:39.164 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:16:39.165 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7' 2024-09-18T08:16:39.266 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5 2024-09-18T08:16:39.295 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:16:39.296 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:16:39.296 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5' 2024-09-18T08:16:39.397 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5 2024-09-18T08:16:39.425 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:16:39.425 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:16:39.426 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5' 2024-09-18T08:16:39.527 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1 2024-09-18T08:16:39.566 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:16:39.566 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:16:39.567 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1' 2024-09-18T08:16:39.667 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6 2024-09-18T08:16:39.694 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:16:39.694 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:16:39.695 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6' 2024-09-18T08:16:39.763 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/779349ca-7593-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T08:16:39.763 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/779349ca-7593-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T08:16:39.796 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4 2024-09-18T08:16:39.820 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:16:39.821 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:16:39.821 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4' 2024-09-18T08:16:39.905 INFO:journalctl@ceph.iscsi.iscsi.a.smithi050.stdout:Sep 18 08:16:39 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-iscsi-iscsi-a[57782]: debug there is no tcmu-runner data available 2024-09-18T08:16:39.922 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0 2024-09-18T08:16:39.954 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:16:39.954 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:16:39.955 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0' 2024-09-18T08:16:40.056 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0 2024-09-18T08:16:40.087 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:16:40.088 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:16:40.088 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0' 2024-09-18T08:16:40.189 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6 2024-09-18T08:16:40.213 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:16:40.215 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:16:40.215 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6' 2024-09-18T08:16:40.317 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7 2024-09-18T08:16:40.340 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:16:40.341 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:16:40.341 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7' 2024-09-18T08:16:40.442 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2 2024-09-18T08:16:40.494 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:16:40.495 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:16:40.495 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2' 2024-09-18T08:16:40.594 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:16:40 smithi007 ceph-mon[39249]: pgmap v690: 158 pgs: 1 active+clean+remapped, 4 active+remapped+backfilling, 6 active+remapped+backfill_toofull, 147 active+clean; 5.4 GiB data, 17 GiB used, 520 GiB / 536 GiB avail; 1.4 KiB/s rd, 0 B/s wr, 964 op/s; 41408/263556 objects misplaced (15.711%); 38 MiB/s, 614 objects/s recovering 2024-09-18T08:16:40.594 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:16:40 smithi007 ceph-mon[39249]: osdmap e359: 8 total, 8 up, 6 in 2024-09-18T08:16:40.594 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:16:40 smithi007 ceph-mon[39249]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-18T08:16:40.594 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:16:40 smithi007 ceph-mon[29459]: pgmap v690: 158 pgs: 1 active+clean+remapped, 4 active+remapped+backfilling, 6 active+remapped+backfill_toofull, 147 active+clean; 5.4 GiB data, 17 GiB used, 520 GiB / 536 GiB avail; 1.4 KiB/s rd, 0 B/s wr, 964 op/s; 41408/263556 objects misplaced (15.711%); 38 MiB/s, 614 objects/s recovering 2024-09-18T08:16:40.594 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:16:40 smithi007 ceph-mon[29459]: osdmap e359: 8 total, 8 up, 6 in 2024-09-18T08:16:40.594 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:16:40 smithi007 ceph-mon[29459]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-18T08:16:40.596 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5 2024-09-18T08:16:40.621 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:16:40.621 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:16:40.622 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5' 2024-09-18T08:16:40.655 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:16:40 smithi050 ceph-mon[32371]: pgmap v690: 158 pgs: 1 active+clean+remapped, 4 active+remapped+backfilling, 6 active+remapped+backfill_toofull, 147 active+clean; 5.4 GiB data, 17 GiB used, 520 GiB / 536 GiB avail; 1.4 KiB/s rd, 0 B/s wr, 964 op/s; 41408/263556 objects misplaced (15.711%); 38 MiB/s, 614 objects/s recovering 2024-09-18T08:16:40.655 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:16:40 smithi050 ceph-mon[32371]: osdmap e359: 8 total, 8 up, 6 in 2024-09-18T08:16:40.655 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:16:40 smithi050 ceph-mon[32371]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-18T08:16:40.723 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2 2024-09-18T08:16:40.766 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:16:40.767 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:16:40.767 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2' 2024-09-18T08:16:40.868 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6 2024-09-18T08:16:40.895 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:16:40.896 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:16:40.896 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6' 2024-09-18T08:16:40.997 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0 2024-09-18T08:16:41.020 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:16:41.020 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:16:41.021 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0' 2024-09-18T08:16:41.121 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6 2024-09-18T08:16:41.151 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:16:41.152 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:16:41.152 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6' 2024-09-18T08:16:41.253 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6 2024-09-18T08:16:41.281 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:16:41.281 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:16:41.282 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6' 2024-09-18T08:16:41.382 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0 2024-09-18T08:16:41.429 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:16:41.430 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:16:41.430 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0' 2024-09-18T08:16:41.531 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4 2024-09-18T08:16:41.561 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:16:41.561 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:16:41.562 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4' 2024-09-18T08:16:41.655 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:16:41 smithi050 ceph-mon[32371]: osdmap e360: 8 total, 8 up, 6 in 2024-09-18T08:16:41.662 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1 2024-09-18T08:16:41.708 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:16:41.708 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:16:41.709 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1' 2024-09-18T08:16:41.729 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:16:41 smithi007 ceph-mon[39249]: osdmap e360: 8 total, 8 up, 6 in 2024-09-18T08:16:41.730 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:16:41 smithi007 ceph-mon[29459]: osdmap e360: 8 total, 8 up, 6 in 2024-09-18T08:16:41.746 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid 779349ca-7593-11ef-bceb-c7b262605968 -- ceph osd unset nodeep-scrub 2024-09-18T08:16:41.810 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1 2024-09-18T08:16:41.838 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:16:41.839 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:16:41.839 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1' 2024-09-18T08:16:41.940 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5 2024-09-18T08:16:41.965 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:16:41.966 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:16:41.966 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5' 2024-09-18T08:16:42.067 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1 2024-09-18T08:16:42.129 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:16:42.129 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:16:42.135 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1' 2024-09-18T08:16:42.236 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2 2024-09-18T08:16:42.280 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:16:42.281 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:16:42.282 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2' 2024-09-18T08:16:42.306 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid 779349ca-7593-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-18T08:16:42.383 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3 2024-09-18T08:16:42.408 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:16:42.409 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:16:42.410 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3' 2024-09-18T08:16:42.511 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1 2024-09-18T08:16:42.557 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:16:42.558 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:16:42.558 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1' 2024-09-18T08:16:42.659 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6 2024-09-18T08:16:42.686 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:16:42.687 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:16:42.687 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6' 2024-09-18T08:16:42.730 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:16:42 smithi007 ceph-mon[29459]: pgmap v692: 158 pgs: 1 active+clean+remapped, 4 active+remapped+backfilling, 6 active+remapped+backfill_toofull, 147 active+clean; 5.2 GiB data, 17 GiB used, 520 GiB / 536 GiB avail; 852 B/s rd, 0 B/s wr, 727 op/s; 38669/257913 objects misplaced (14.993%); 32 MiB/s, 508 objects/s recovering 2024-09-18T08:16:42.730 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:16:42 smithi007 ceph-mon[39249]: pgmap v692: 158 pgs: 1 active+clean+remapped, 4 active+remapped+backfilling, 6 active+remapped+backfill_toofull, 147 active+clean; 5.2 GiB data, 17 GiB used, 520 GiB / 536 GiB avail; 852 B/s rd, 0 B/s wr, 727 op/s; 38669/257913 objects misplaced (14.993%); 32 MiB/s, 508 objects/s recovering 2024-09-18T08:16:42.788 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6 2024-09-18T08:16:42.810 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:16:42.810 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:16:42.811 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6' 2024-09-18T08:16:42.905 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:16:42 smithi050 ceph-mon[32371]: pgmap v692: 158 pgs: 1 active+clean+remapped, 4 active+remapped+backfilling, 6 active+remapped+backfill_toofull, 147 active+clean; 5.2 GiB data, 17 GiB used, 520 GiB / 536 GiB avail; 852 B/s rd, 0 B/s wr, 727 op/s; 38669/257913 objects misplaced (14.993%); 32 MiB/s, 508 objects/s recovering 2024-09-18T08:16:42.911 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6 2024-09-18T08:16:42.939 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:16:42.940 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:16:42.940 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6' 2024-09-18T08:16:43.041 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7 2024-09-18T08:16:43.071 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:16:43.071 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:16:43.072 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7' 2024-09-18T08:16:43.172 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5 2024-09-18T08:16:43.194 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:16:43.195 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:16:43.195 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5' 2024-09-18T08:16:43.296 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6 2024-09-18T08:16:43.323 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:16:43.323 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:16:43.324 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6' 2024-09-18T08:16:43.425 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7 2024-09-18T08:16:43.449 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:16:43.450 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:16:43.450 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7' 2024-09-18T08:16:43.551 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4 2024-09-18T08:16:43.577 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:16:43.578 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:16:43.579 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4' 2024-09-18T08:16:43.679 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3 2024-09-18T08:16:43.704 INFO:teuthology.orchestra.run.smithi007.stdout:ERROR: (22) Invalid argument 2024-09-18T08:16:43.704 INFO:teuthology.orchestra.run.smithi007.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-18T08:16:43.724 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:16:43.725 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:16:43.725 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3' 2024-09-18T08:16:43.826 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5 2024-09-18T08:16:43.848 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:16:43.849 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:16:43.849 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5' 2024-09-18T08:16:43.950 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4 2024-09-18T08:16:43.978 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:16:43.978 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:16:43.979 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4' 2024-09-18T08:16:44.080 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5 2024-09-18T08:16:44.103 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:16:44.103 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:16:44.104 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5' 2024-09-18T08:16:44.205 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5 2024-09-18T08:16:44.231 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:16:44.231 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:16:44.232 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5' 2024-09-18T08:16:44.333 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7 2024-09-18T08:16:44.363 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:16:44.363 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:16:44.363 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7' 2024-09-18T08:16:44.464 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2 2024-09-18T08:16:44.495 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:16:44.496 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:16:44.496 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2' 2024-09-18T08:16:44.597 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2 2024-09-18T08:16:44.642 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:16:44.642 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:16:44.642 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2' 2024-09-18T08:16:44.730 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:16:44 smithi007 ceph-mon[29459]: pgmap v693: 158 pgs: 1 active+clean+remapped, 4 active+remapped+backfilling, 6 active+remapped+backfill_toofull, 147 active+clean; 5.2 GiB data, 17 GiB used, 520 GiB / 536 GiB avail; 1.3 KiB/s rd, 0 B/s wr, 649 op/s; 37511/255603 objects misplaced (14.675%); 30 MiB/s, 481 objects/s recovering 2024-09-18T08:16:44.730 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:16:44 smithi007 ceph-mon[39249]: pgmap v693: 158 pgs: 1 active+clean+remapped, 4 active+remapped+backfilling, 6 active+remapped+backfill_toofull, 147 active+clean; 5.2 GiB data, 17 GiB used, 520 GiB / 536 GiB avail; 1.3 KiB/s rd, 0 B/s wr, 649 op/s; 37511/255603 objects misplaced (14.675%); 30 MiB/s, 481 objects/s recovering 2024-09-18T08:16:44.743 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2 2024-09-18T08:16:44.790 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:16:44.791 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:16:44.791 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2' 2024-09-18T08:16:44.892 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4 2024-09-18T08:16:44.905 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:16:44 smithi050 ceph-mon[32371]: pgmap v693: 158 pgs: 1 active+clean+remapped, 4 active+remapped+backfilling, 6 active+remapped+backfill_toofull, 147 active+clean; 5.2 GiB data, 17 GiB used, 520 GiB / 536 GiB avail; 1.3 KiB/s rd, 0 B/s wr, 649 op/s; 37511/255603 objects misplaced (14.675%); 30 MiB/s, 481 objects/s recovering 2024-09-18T08:16:44.917 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:16:44.918 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:16:44.919 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4' 2024-09-18T08:16:45.019 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5 2024-09-18T08:16:45.047 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:16:45.048 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:16:45.048 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5' 2024-09-18T08:16:45.149 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1 2024-09-18T08:16:45.181 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:16:45.181 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:16:45.182 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1' 2024-09-18T08:16:45.283 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5 2024-09-18T08:16:45.311 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:16:45.312 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:16:45.312 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5' 2024-09-18T08:16:45.413 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6 2024-09-18T08:16:45.435 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:16:45.435 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:16:45.436 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6' 2024-09-18T08:16:45.536 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5 2024-09-18T08:16:45.562 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:16:45.563 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:16:45.563 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5' 2024-09-18T08:16:45.664 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6 2024-09-18T08:16:45.688 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:16:45.689 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:16:45.689 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6' 2024-09-18T08:16:45.730 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:16:45 smithi007 ceph-mon[29459]: Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 5 pgs backfill_toofull (PG_BACKFILL_FULL) 2024-09-18T08:16:45.730 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:16:45 smithi007 ceph-mon[39249]: Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 5 pgs backfill_toofull (PG_BACKFILL_FULL) 2024-09-18T08:16:45.790 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7 2024-09-18T08:16:45.820 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:16:45.820 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:16:45.821 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7' 2024-09-18T08:16:45.905 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:16:45 smithi050 ceph-mon[32371]: Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 5 pgs backfill_toofull (PG_BACKFILL_FULL) 2024-09-18T08:16:45.922 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1 2024-09-18T08:16:45.961 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:16:45.962 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:16:45.962 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1' 2024-09-18T08:16:46.063 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0 2024-09-18T08:16:46.088 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:16:46.089 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:16:46.089 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0' 2024-09-18T08:16:46.190 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2 2024-09-18T08:16:46.236 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:16:46.237 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:16:46.238 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2' 2024-09-18T08:16:46.338 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1 2024-09-18T08:16:46.383 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:16:46.384 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:16:46.384 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1' 2024-09-18T08:16:46.485 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7 2024-09-18T08:16:46.513 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:16:46.513 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:16:46.514 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7' 2024-09-18T08:16:46.615 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3 2024-09-18T08:16:46.653 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:16:46.654 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:16:46.655 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3' 2024-09-18T08:16:46.730 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Sep 18 08:16:46 smithi007 ceph-779349ca-7593-11ef-bceb-c7b262605968-mgr-y[29688]: ::ffff:172.21.15.50 - - [18/Sep/2024:08:16:46] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-09-18T08:16:46.730 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:16:46 smithi007 ceph-mon[29459]: pgmap v694: 158 pgs: 2 active+remapped+backfill_wait+backfill_toofull, 5 active+remapped+backfilling, 3 active+remapped+backfill_toofull, 148 active+clean; 5.0 GiB data, 16 GiB used, 520 GiB / 536 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 990 op/s; 33468/247242 objects misplaced (13.537%); 52 MiB/s, 838 objects/s recovering 2024-09-18T08:16:46.731 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:16:46 smithi007 ceph-mon[39249]: pgmap v694: 158 pgs: 2 active+remapped+backfill_wait+backfill_toofull, 5 active+remapped+backfilling, 3 active+remapped+backfill_toofull, 148 active+clean; 5.0 GiB data, 16 GiB used, 520 GiB / 536 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 990 op/s; 33468/247242 objects misplaced (13.537%); 52 MiB/s, 838 objects/s recovering 2024-09-18T08:16:46.756 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6 2024-09-18T08:16:46.785 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:16:46.786 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:16:46.786 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6' 2024-09-18T08:16:46.888 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3 2024-09-18T08:16:46.905 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:16:46 smithi050 ceph-mon[32371]: pgmap v694: 158 pgs: 2 active+remapped+backfill_wait+backfill_toofull, 5 active+remapped+backfilling, 3 active+remapped+backfill_toofull, 148 active+clean; 5.0 GiB data, 16 GiB used, 520 GiB / 536 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 990 op/s; 33468/247242 objects misplaced (13.537%); 52 MiB/s, 838 objects/s recovering 2024-09-18T08:16:46.941 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:16:46.941 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:16:46.942 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3' 2024-09-18T08:16:47.043 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6 2024-09-18T08:16:47.070 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:16:47.070 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:16:47.071 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6' 2024-09-18T08:16:47.172 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2 2024-09-18T08:16:47.213 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:16:47.214 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:16:47.214 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2' 2024-09-18T08:16:47.315 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4 2024-09-18T08:16:47.343 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:16:47.344 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:16:47.344 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4' 2024-09-18T08:16:47.445 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5 2024-09-18T08:16:47.474 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:16:47.475 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:16:47.475 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5' 2024-09-18T08:16:47.576 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0 2024-09-18T08:16:47.625 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid 779349ca-7593-11ef-bceb-c7b262605968 -- ceph daemon osd.3 dump_historic_ops 2024-09-18T08:16:47.641 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:16:47.641 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:16:47.642 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0' 2024-09-18T08:16:47.731 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:16:47 smithi007 ceph-mon[29459]: 5.a scrub starts 2024-09-18T08:16:47.731 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:16:47 smithi007 ceph-mon[29459]: 5.a scrub ok 2024-09-18T08:16:47.731 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:16:47 smithi007 ceph-mon[39249]: 5.a scrub starts 2024-09-18T08:16:47.731 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:16:47 smithi007 ceph-mon[39249]: 5.a scrub ok 2024-09-18T08:16:47.743 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2 2024-09-18T08:16:47.818 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:16:47.818 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:16:47.819 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2' 2024-09-18T08:16:47.905 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:16:47 smithi050 ceph-mon[32371]: 5.a scrub starts 2024-09-18T08:16:47.905 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:16:47 smithi050 ceph-mon[32371]: 5.a scrub ok 2024-09-18T08:16:47.920 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7 2024-09-18T08:16:47.944 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:16:47.945 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:16:47.945 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7' 2024-09-18T08:16:47.973 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/779349ca-7593-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T08:16:48.016 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/779349ca-7593-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T08:16:48.046 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5 2024-09-18T08:16:48.071 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:16:48.072 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:16:48.072 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5' 2024-09-18T08:16:48.173 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6 2024-09-18T08:16:48.198 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:16:48.199 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:16:48.199 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6' 2024-09-18T08:16:48.300 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7 2024-09-18T08:16:48.330 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:16:48.331 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:16:48.331 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7' 2024-09-18T08:16:48.432 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4 2024-09-18T08:16:48.458 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:16:48.458 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:16:48.459 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4' 2024-09-18T08:16:48.484 INFO:teuthology.orchestra.run.smithi007.stderr:reweighted osd.1 to 1 (10000) 2024-09-18T08:16:48.560 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5 2024-09-18T08:16:48.588 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:16:48.589 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:16:48.589 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5' 2024-09-18T08:16:48.690 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7 2024-09-18T08:16:48.712 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:16:48.713 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:16:48.713 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7' 2024-09-18T08:16:48.814 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6 2024-09-18T08:16:48.842 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:16:48.842 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:16:48.843 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6' 2024-09-18T08:16:48.905 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:16:48 smithi050 ceph-mon[32371]: pgmap v695: 158 pgs: 2 active+remapped+backfill_wait+backfill_toofull, 5 active+remapped+backfilling, 3 active+remapped+backfill_toofull, 148 active+clean; 5.0 GiB data, 16 GiB used, 520 GiB / 536 GiB avail; 639 B/s rd, 0 B/s wr, 679 op/s; 33468/247242 objects misplaced (13.537%); 42 MiB/s, 668 objects/s recovering 2024-09-18T08:16:48.905 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:16:48 smithi050 ceph-mon[32371]: from='client.? 172.21.15.7:0/1222672588' entity='client.admin' cmd={"prefix": "osd reweight", "id": 1, "weight": 1.0} : dispatch 2024-09-18T08:16:48.906 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:16:48 smithi050 ceph-mon[32371]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 1, "weight": 1.0} : dispatch 2024-09-18T08:16:48.943 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2 2024-09-18T08:16:48.980 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:16:48 smithi007 ceph-mon[29459]: pgmap v695: 158 pgs: 2 active+remapped+backfill_wait+backfill_toofull, 5 active+remapped+backfilling, 3 active+remapped+backfill_toofull, 148 active+clean; 5.0 GiB data, 16 GiB used, 520 GiB / 536 GiB avail; 639 B/s rd, 0 B/s wr, 679 op/s; 33468/247242 objects misplaced (13.537%); 42 MiB/s, 668 objects/s recovering 2024-09-18T08:16:48.980 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:16:48 smithi007 ceph-mon[29459]: from='client.? 172.21.15.7:0/1222672588' entity='client.admin' cmd={"prefix": "osd reweight", "id": 1, "weight": 1.0} : dispatch 2024-09-18T08:16:48.980 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:16:48 smithi007 ceph-mon[29459]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 1, "weight": 1.0} : dispatch 2024-09-18T08:16:48.980 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:16:48 smithi007 ceph-mon[39249]: pgmap v695: 158 pgs: 2 active+remapped+backfill_wait+backfill_toofull, 5 active+remapped+backfilling, 3 active+remapped+backfill_toofull, 148 active+clean; 5.0 GiB data, 16 GiB used, 520 GiB / 536 GiB avail; 639 B/s rd, 0 B/s wr, 679 op/s; 33468/247242 objects misplaced (13.537%); 42 MiB/s, 668 objects/s recovering 2024-09-18T08:16:48.980 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:16:48 smithi007 ceph-mon[39249]: from='client.? 172.21.15.7:0/1222672588' entity='client.admin' cmd={"prefix": "osd reweight", "id": 1, "weight": 1.0} : dispatch 2024-09-18T08:16:48.980 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:16:48 smithi007 ceph-mon[39249]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 1, "weight": 1.0} : dispatch 2024-09-18T08:16:48.983 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:16:48.984 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:16:48.984 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2' 2024-09-18T08:16:49.085 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7 2024-09-18T08:16:49.107 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:16:49.108 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:16:49.108 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7' 2024-09-18T08:16:49.209 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0 2024-09-18T08:16:49.241 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:16:49.242 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:16:49.242 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0' 2024-09-18T08:16:49.343 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6 2024-09-18T08:16:49.369 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:16:49.369 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:16:49.370 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6' 2024-09-18T08:16:49.471 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6 2024-09-18T08:16:49.499 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:16:49.500 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:16:49.500 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6' 2024-09-18T08:16:49.601 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2 2024-09-18T08:16:49.638 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:16:49.639 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:16:49.639 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2' 2024-09-18T08:16:49.740 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3 2024-09-18T08:16:49.779 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:16:49.780 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:16:49.780 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3' 2024-09-18T08:16:49.826 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:16:49 smithi007 ceph-mon[29459]: 3.14 scrub starts 2024-09-18T08:16:49.826 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:16:49 smithi007 ceph-mon[29459]: 3.14 scrub ok 2024-09-18T08:16:49.826 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:16:49 smithi007 ceph-mon[29459]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]': finished 2024-09-18T08:16:49.826 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:16:49 smithi007 ceph-mon[29459]: osdmap e361: 8 total, 8 up, 6 in 2024-09-18T08:16:49.826 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:16:49 smithi007 ceph-mon[29459]: from='mgr.24470 172.21.15.7:0/2594390261' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-18T08:16:49.826 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:16:49 smithi007 ceph-mon[39249]: 3.14 scrub starts 2024-09-18T08:16:49.826 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:16:49 smithi007 ceph-mon[39249]: 3.14 scrub ok 2024-09-18T08:16:49.827 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:16:49 smithi007 ceph-mon[39249]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]': finished 2024-09-18T08:16:49.827 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:16:49 smithi007 ceph-mon[39249]: osdmap e361: 8 total, 8 up, 6 in 2024-09-18T08:16:49.827 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:16:49 smithi007 ceph-mon[39249]: from='mgr.24470 172.21.15.7:0/2594390261' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-18T08:16:49.881 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1 2024-09-18T08:16:49.905 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:16:49 smithi050 ceph-mon[32371]: 3.14 scrub starts 2024-09-18T08:16:49.905 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:16:49 smithi050 ceph-mon[32371]: 3.14 scrub ok 2024-09-18T08:16:49.905 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:16:49 smithi050 ceph-mon[32371]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]': finished 2024-09-18T08:16:49.905 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:16:49 smithi050 ceph-mon[32371]: osdmap e361: 8 total, 8 up, 6 in 2024-09-18T08:16:49.905 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:16:49 smithi050 ceph-mon[32371]: from='mgr.24470 172.21.15.7:0/2594390261' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-18T08:16:49.906 INFO:journalctl@ceph.iscsi.iscsi.a.smithi050.stdout:Sep 18 08:16:49 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-iscsi-iscsi-a[57782]: debug there is no tcmu-runner data available 2024-09-18T08:16:49.947 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:16:49.948 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:16:49.949 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1' 2024-09-18T08:16:50.050 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0 2024-09-18T08:16:50.073 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:16:50.073 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:16:50.073 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0' 2024-09-18T08:16:50.174 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5 2024-09-18T08:16:50.203 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:16:50.203 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:16:50.204 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5' 2024-09-18T08:16:50.305 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2 2024-09-18T08:16:50.349 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:16:50.349 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:16:50.350 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2' 2024-09-18T08:16:50.451 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4 2024-09-18T08:16:50.481 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:16:50.482 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:16:50.482 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4' 2024-09-18T08:16:50.583 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3 2024-09-18T08:16:50.638 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:16:50.639 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:16:50.639 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3' 2024-09-18T08:16:50.740 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0 2024-09-18T08:16:50.793 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:16:50.794 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:16:50.794 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0' 2024-09-18T08:16:50.819 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:16:50 smithi007 ceph-mon[29459]: 6.14 scrub starts 2024-09-18T08:16:50.819 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:16:50 smithi007 ceph-mon[29459]: 6.14 scrub ok 2024-09-18T08:16:50.819 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:16:50 smithi007 ceph-mon[29459]: pgmap v697: 158 pgs: 2 active+remapped+backfill_wait+backfill_toofull, 5 active+remapped+backfilling, 3 active+remapped+backfill_toofull, 148 active+clean; 4.9 GiB data, 16 GiB used, 520 GiB / 536 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 662 op/s; 31319/240030 objects misplaced (13.048%); 34 MiB/s, 545 objects/s recovering 2024-09-18T08:16:50.819 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:16:50 smithi007 ceph-mon[29459]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-18T08:16:50.820 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:16:50 smithi007 ceph-mon[39249]: 6.14 scrub starts 2024-09-18T08:16:50.820 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:16:50 smithi007 ceph-mon[39249]: 6.14 scrub ok 2024-09-18T08:16:50.820 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:16:50 smithi007 ceph-mon[39249]: pgmap v697: 158 pgs: 2 active+remapped+backfill_wait+backfill_toofull, 5 active+remapped+backfilling, 3 active+remapped+backfill_toofull, 148 active+clean; 4.9 GiB data, 16 GiB used, 520 GiB / 536 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 662 op/s; 31319/240030 objects misplaced (13.048%); 34 MiB/s, 545 objects/s recovering 2024-09-18T08:16:50.820 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:16:50 smithi007 ceph-mon[39249]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-18T08:16:50.895 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6 2024-09-18T08:16:50.905 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:16:50 smithi050 ceph-mon[32371]: 6.14 scrub starts 2024-09-18T08:16:50.905 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:16:50 smithi050 ceph-mon[32371]: 6.14 scrub ok 2024-09-18T08:16:50.905 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:16:50 smithi050 ceph-mon[32371]: pgmap v697: 158 pgs: 2 active+remapped+backfill_wait+backfill_toofull, 5 active+remapped+backfilling, 3 active+remapped+backfill_toofull, 148 active+clean; 4.9 GiB data, 16 GiB used, 520 GiB / 536 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 662 op/s; 31319/240030 objects misplaced (13.048%); 34 MiB/s, 545 objects/s recovering 2024-09-18T08:16:50.905 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:16:50 smithi050 ceph-mon[32371]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-18T08:16:50.920 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:16:50.920 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:16:50.921 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6' 2024-09-18T08:16:50.958 INFO:teuthology.orchestra.run.smithi007.stdout:osd.0: {} 2024-09-18T08:16:50.958 INFO:teuthology.orchestra.run.smithi007.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-18T08:16:51.004 INFO:teuthology.orchestra.run.smithi007.stdout:osd.1: {} 2024-09-18T08:16:51.004 INFO:teuthology.orchestra.run.smithi007.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-18T08:16:51.022 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0 2024-09-18T08:16:51.045 INFO:teuthology.orchestra.run.smithi007.stdout:osd.2: {} 2024-09-18T08:16:51.045 INFO:teuthology.orchestra.run.smithi007.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-18T08:16:51.058 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:16:51.059 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:16:51.059 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0' 2024-09-18T08:16:51.088 INFO:teuthology.orchestra.run.smithi007.stdout:osd.3: {} 2024-09-18T08:16:51.089 INFO:teuthology.orchestra.run.smithi007.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-18T08:16:51.124 INFO:teuthology.orchestra.run.smithi007.stdout:osd.4: {} 2024-09-18T08:16:51.125 INFO:teuthology.orchestra.run.smithi007.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-18T08:16:51.160 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3 2024-09-18T08:16:51.181 INFO:teuthology.orchestra.run.smithi007.stdout:osd.5: {} 2024-09-18T08:16:51.182 INFO:teuthology.orchestra.run.smithi007.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-18T08:16:51.209 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/779349ca-7593-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T08:16:51.216 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:16:51.217 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:16:51.217 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3' 2024-09-18T08:16:51.227 INFO:teuthology.orchestra.run.smithi007.stdout:osd.6: {} 2024-09-18T08:16:51.227 INFO:teuthology.orchestra.run.smithi007.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-18T08:16:51.264 INFO:teuthology.orchestra.run.smithi007.stdout:osd.7: {} 2024-09-18T08:16:51.264 INFO:teuthology.orchestra.run.smithi007.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-18T08:16:51.318 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5 2024-09-18T08:16:51.348 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:16:51.348 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:16:51.349 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5' 2024-09-18T08:16:51.449 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5 2024-09-18T08:16:51.476 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:16:51.477 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:16:51.477 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5' 2024-09-18T08:16:51.515 INFO:teuthology.orchestra.run.smithi007.stderr:nodeep-scrub is unset 2024-09-18T08:16:51.578 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6 2024-09-18T08:16:51.605 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:16:51.605 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:16:51.606 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6' 2024-09-18T08:16:51.706 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0 2024-09-18T08:16:51.752 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:16:51.753 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:16:51.753 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0' 2024-09-18T08:16:51.771 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:16:51 smithi007 ceph-mon[39249]: from='client.? 172.21.15.7:0/739558993' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-18T08:16:51.772 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:16:51 smithi007 ceph-mon[39249]: from='client.? 172.21.15.7:0/4218900062' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-18T08:16:51.772 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:16:51 smithi007 ceph-mon[29459]: from='client.? 172.21.15.7:0/739558993' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-18T08:16:51.772 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:16:51 smithi007 ceph-mon[29459]: from='client.? 172.21.15.7:0/4218900062' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-18T08:16:51.791 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid 779349ca-7593-11ef-bceb-c7b262605968 -- ceph osd reweight 2 1 2024-09-18T08:16:51.854 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2 2024-09-18T08:16:51.905 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:16:51 smithi050 ceph-mon[32371]: from='client.? 172.21.15.7:0/739558993' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-18T08:16:51.906 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:16:51 smithi050 ceph-mon[32371]: from='client.? 172.21.15.7:0/4218900062' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-18T08:16:51.909 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:16:51.910 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:16:51.910 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2' 2024-09-18T08:16:52.012 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6 2024-09-18T08:16:52.037 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:16:52.038 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:16:52.038 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6' 2024-09-18T08:16:52.139 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6 2024-09-18T08:16:52.163 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:16:52.164 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:16:52.164 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6' 2024-09-18T08:16:52.265 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6 2024-09-18T08:16:52.288 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:16:52.289 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:16:52.289 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6' 2024-09-18T08:16:52.390 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3 2024-09-18T08:16:52.422 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:16:52.423 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:16:52.423 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3' 2024-09-18T08:16:52.524 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2 2024-09-18T08:16:52.556 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:16:52.557 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:16:52.557 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2' 2024-09-18T08:16:52.658 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3 2024-09-18T08:16:52.697 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:16:52.698 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:16:52.699 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3' 2024-09-18T08:16:52.716 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:16:52 smithi007 ceph-mon[29459]: pgmap v698: 158 pgs: 2 active+remapped+backfill_wait+backfill_toofull, 5 active+remapped+backfilling, 3 active+remapped+backfill_toofull, 148 active+clean; 4.8 GiB data, 16 GiB used, 520 GiB / 536 GiB avail; 1023 B/s rd, 0 B/s wr, 777 op/s; 28775/234609 objects misplaced (12.265%); 42 MiB/s, 669 objects/s recovering 2024-09-18T08:16:52.716 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:16:52 smithi007 ceph-mon[29459]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-18T08:16:52.717 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:16:52 smithi007 ceph-mon[29459]: from='client.? 172.21.15.7:0/4218900062' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-18T08:16:52.717 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:16:52 smithi007 ceph-mon[29459]: osdmap e362: 8 total, 8 up, 6 in 2024-09-18T08:16:52.717 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:16:52 smithi007 ceph-mon[29459]: 6.17 scrub starts 2024-09-18T08:16:52.717 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:16:52 smithi007 ceph-mon[29459]: 6.17 scrub ok 2024-09-18T08:16:52.717 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:16:52 smithi007 ceph-mon[39249]: pgmap v698: 158 pgs: 2 active+remapped+backfill_wait+backfill_toofull, 5 active+remapped+backfilling, 3 active+remapped+backfill_toofull, 148 active+clean; 4.8 GiB data, 16 GiB used, 520 GiB / 536 GiB avail; 1023 B/s rd, 0 B/s wr, 777 op/s; 28775/234609 objects misplaced (12.265%); 42 MiB/s, 669 objects/s recovering 2024-09-18T08:16:52.717 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:16:52 smithi007 ceph-mon[39249]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-18T08:16:52.717 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:16:52 smithi007 ceph-mon[39249]: from='client.? 172.21.15.7:0/4218900062' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-18T08:16:52.717 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:16:52 smithi007 ceph-mon[39249]: osdmap e362: 8 total, 8 up, 6 in 2024-09-18T08:16:52.717 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:16:52 smithi007 ceph-mon[39249]: 6.17 scrub starts 2024-09-18T08:16:52.717 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:16:52 smithi007 ceph-mon[39249]: 6.17 scrub ok 2024-09-18T08:16:52.799 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0 2024-09-18T08:16:52.833 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:16:52.833 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:16:52.834 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0' 2024-09-18T08:16:52.905 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:16:52 smithi050 ceph-mon[32371]: pgmap v698: 158 pgs: 2 active+remapped+backfill_wait+backfill_toofull, 5 active+remapped+backfilling, 3 active+remapped+backfill_toofull, 148 active+clean; 4.8 GiB data, 16 GiB used, 520 GiB / 536 GiB avail; 1023 B/s rd, 0 B/s wr, 777 op/s; 28775/234609 objects misplaced (12.265%); 42 MiB/s, 669 objects/s recovering 2024-09-18T08:16:52.906 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:16:52 smithi050 ceph-mon[32371]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-18T08:16:52.906 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:16:52 smithi050 ceph-mon[32371]: from='client.? 172.21.15.7:0/4218900062' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-18T08:16:52.906 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:16:52 smithi050 ceph-mon[32371]: osdmap e362: 8 total, 8 up, 6 in 2024-09-18T08:16:52.906 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:16:52 smithi050 ceph-mon[32371]: 6.17 scrub starts 2024-09-18T08:16:52.906 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:16:52 smithi050 ceph-mon[32371]: 6.17 scrub ok 2024-09-18T08:16:52.934 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1 2024-09-18T08:16:52.966 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:16:52.967 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:16:52.967 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1' 2024-09-18T08:16:53.068 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0 2024-09-18T08:16:53.105 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:16:53.106 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:16:53.107 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0' 2024-09-18T08:16:53.207 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5 2024-09-18T08:16:53.230 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:16:53.230 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:16:53.231 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5' 2024-09-18T08:16:53.332 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3 2024-09-18T08:16:53.364 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:16:53.365 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:16:53.365 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3' 2024-09-18T08:16:53.466 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2 2024-09-18T08:16:53.499 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:16:53.500 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:16:53.500 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2' 2024-09-18T08:16:53.601 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1 2024-09-18T08:16:53.656 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:16:53 smithi007 ceph-mon[29459]: osdmap e363: 8 total, 8 up, 6 in 2024-09-18T08:16:53.656 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:16:53 smithi007 ceph-mon[39249]: osdmap e363: 8 total, 8 up, 6 in 2024-09-18T08:16:53.665 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:16:53.665 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:16:53.666 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1' 2024-09-18T08:16:53.693 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/779349ca-7593-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T08:16:53.725 INFO:teuthology.orchestra.run.smithi007.stdout:{ 2024-09-18T08:16:53.726 INFO:teuthology.orchestra.run.smithi007.stdout: "size": 20, 2024-09-18T08:16:53.726 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 600, 2024-09-18T08:16:53.726 INFO:teuthology.orchestra.run.smithi007.stdout: "ops": [ 2024-09-18T08:16:53.726 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:16:53.726 INFO:teuthology.orchestra.run.smithi007.stdout: "description": "osd_op(client.15216.0:69187 8.f 8:f51333c2:::benchmark_data_smithi007_101583_object69186:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e198)", 2024-09-18T08:16:53.726 INFO:teuthology.orchestra.run.smithi007.stdout: "initiated_at": "2024-09-18T08:08:51.699105+0000", 2024-09-18T08:16:53.726 INFO:teuthology.orchestra.run.smithi007.stdout: "age": 482.02133713000001, 2024-09-18T08:16:53.726 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 2.1312144279999998, 2024-09-18T08:16:53.726 INFO:teuthology.orchestra.run.smithi007.stdout: "continuous": false, 2024-09-18T08:16:53.726 INFO:teuthology.orchestra.run.smithi007.stdout: "type_data": { 2024-09-18T08:16:53.726 INFO:teuthology.orchestra.run.smithi007.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T08:16:53.726 INFO:teuthology.orchestra.run.smithi007.stdout: "client_info": { 2024-09-18T08:16:53.727 INFO:teuthology.orchestra.run.smithi007.stdout: "client": "client.15216", 2024-09-18T08:16:53.727 INFO:teuthology.orchestra.run.smithi007.stdout: "client_addr": "172.21.15.7:0/2596041177", 2024-09-18T08:16:53.727 INFO:teuthology.orchestra.run.smithi007.stdout: "tid": 69187 2024-09-18T08:16:53.727 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:16:53.727 INFO:teuthology.orchestra.run.smithi007.stdout: "events": [ 2024-09-18T08:16:53.727 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:16:53.727 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "initiated", 2024-09-18T08:16:53.727 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:51.699105+0000", 2024-09-18T08:16:53.727 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-09-18T08:16:53.727 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:16:53.727 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:16:53.727 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "header_read", 2024-09-18T08:16:53.727 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:51.699105+0000", 2024-09-18T08:16:53.727 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-09-18T08:16:53.728 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:16:53.728 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:16:53.728 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "throttled", 2024-09-18T08:16:53.728 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:51.699107+0000", 2024-09-18T08:16:53.728 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.5149999999999999e-06 2024-09-18T08:16:53.728 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:16:53.728 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:16:53.728 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "all_read", 2024-09-18T08:16:53.728 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:51.699133+0000", 2024-09-18T08:16:53.728 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 2.5752999999999999e-05 2024-09-18T08:16:53.728 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:16:53.728 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:16:53.729 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "dispatched", 2024-09-18T08:16:53.729 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:51.699141+0000", 2024-09-18T08:16:53.729 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 8.2709999999999992e-06 2024-09-18T08:16:53.729 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:16:53.729 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:16:53.729 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "queued_for_pg", 2024-09-18T08:16:53.729 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:51.699163+0000", 2024-09-18T08:16:53.729 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 2.2432000000000001e-05 2024-09-18T08:16:53.729 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:16:53.729 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:16:53.730 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-09-18T08:16:53.730 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:51.699402+0000", 2024-09-18T08:16:53.730 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.000239046 2024-09-18T08:16:53.730 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:16:53.730 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:16:53.730 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "waiting for peered", 2024-09-18T08:16:53.730 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:51.699413+0000", 2024-09-18T08:16:53.730 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.0387000000000001e-05 2024-09-18T08:16:53.730 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:16:53.730 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:16:53.730 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-09-18T08:16:53.730 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:52.696975+0000", 2024-09-18T08:16:53.730 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.99756218100000005 2024-09-18T08:16:53.731 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:16:53.731 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:16:53.731 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "waiting for peered", 2024-09-18T08:16:53.731 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:52.697002+0000", 2024-09-18T08:16:53.731 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 2.7464e-05 2024-09-18T08:16:53.731 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:16:53.731 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:16:53.731 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-09-18T08:16:53.731 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:53.824414+0000", 2024-09-18T08:16:53.732 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.1274112789999999 2024-09-18T08:16:53.732 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:16:53.732 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:16:53.732 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "started", 2024-09-18T08:16:53.732 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:53.824573+0000", 2024-09-18T08:16:53.732 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.00015908400000000001 2024-09-18T08:16:53.732 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:16:53.732 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:16:53.732 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "waiting for subops from 1,2,4", 2024-09-18T08:16:53.732 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:53.824756+0000", 2024-09-18T08:16:53.732 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.00018345199999999999 2024-09-18T08:16:53.733 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:16:53.733 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:16:53.733 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:16:53.733 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:53.827276+0000", 2024-09-18T08:16:53.733 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.002519444 2024-09-18T08:16:53.733 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:16:53.733 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:16:53.733 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:16:53.733 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:53.827356+0000", 2024-09-18T08:16:53.733 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 8.0029000000000002e-05 2024-09-18T08:16:53.733 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:16:53.734 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:16:53.734 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "op_commit", 2024-09-18T08:16:53.734 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:53.827376+0000", 2024-09-18T08:16:53.734 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 2.0250000000000001e-05 2024-09-18T08:16:53.734 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:16:53.734 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:16:53.734 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:16:53.734 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:53.830181+0000", 2024-09-18T08:16:53.734 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.0028047469999999998 2024-09-18T08:16:53.734 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:16:53.735 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:16:53.735 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "commit_sent", 2024-09-18T08:16:53.735 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:53.830246+0000", 2024-09-18T08:16:53.735 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 6.5140000000000003e-05 2024-09-18T08:16:53.735 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:16:53.735 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:16:53.735 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "done", 2024-09-18T08:16:53.735 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:53.830320+0000", 2024-09-18T08:16:53.735 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 7.3954000000000004e-05 2024-09-18T08:16:53.735 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-09-18T08:16:53.735 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-09-18T08:16:53.736 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-09-18T08:16:53.736 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:16:53.736 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:16:53.736 INFO:teuthology.orchestra.run.smithi007.stdout: "description": "osd_op(client.15216.0:69195 8.f 8:fafe41e6:::benchmark_data_smithi007_101583_object69194:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e198)", 2024-09-18T08:16:53.736 INFO:teuthology.orchestra.run.smithi007.stdout: "initiated_at": "2024-09-18T08:08:51.711966+0000", 2024-09-18T08:16:53.736 INFO:teuthology.orchestra.run.smithi007.stdout: "age": 482.008476368, 2024-09-18T08:16:53.736 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 2.1187014670000002, 2024-09-18T08:16:53.736 INFO:teuthology.orchestra.run.smithi007.stdout: "continuous": false, 2024-09-18T08:16:53.736 INFO:teuthology.orchestra.run.smithi007.stdout: "type_data": { 2024-09-18T08:16:53.736 INFO:teuthology.orchestra.run.smithi007.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T08:16:53.737 INFO:teuthology.orchestra.run.smithi007.stdout: "client_info": { 2024-09-18T08:16:53.737 INFO:teuthology.orchestra.run.smithi007.stdout: "client": "client.15216", 2024-09-18T08:16:53.737 INFO:teuthology.orchestra.run.smithi007.stdout: "client_addr": "172.21.15.7:0/2596041177", 2024-09-18T08:16:53.737 INFO:teuthology.orchestra.run.smithi007.stdout: "tid": 69195 2024-09-18T08:16:53.737 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:16:53.737 INFO:teuthology.orchestra.run.smithi007.stdout: "events": [ 2024-09-18T08:16:53.737 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:16:53.737 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "initiated", 2024-09-18T08:16:53.737 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:51.711966+0000", 2024-09-18T08:16:53.737 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-09-18T08:16:53.737 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:16:53.737 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:16:53.737 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "header_read", 2024-09-18T08:16:53.738 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:51.711966+0000", 2024-09-18T08:16:53.738 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-09-18T08:16:53.738 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:16:53.738 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:16:53.738 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "throttled", 2024-09-18T08:16:53.738 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:51.711968+0000", 2024-09-18T08:16:53.738 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 2.013e-06 2024-09-18T08:16:53.738 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:16:53.738 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:16:53.738 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "all_read", 2024-09-18T08:16:53.738 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:51.711997+0000", 2024-09-18T08:16:53.738 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 2.8561999999999999e-05 2024-09-18T08:16:53.738 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:16:53.739 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:16:53.739 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "dispatched", 2024-09-18T08:16:53.739 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:51.712012+0000", 2024-09-18T08:16:53.739 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.5184000000000001e-05 2024-09-18T08:16:53.739 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:16:53.739 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:16:53.739 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "queued_for_pg", 2024-09-18T08:16:53.739 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:51.712022+0000", 2024-09-18T08:16:53.739 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 9.6530000000000006e-06 2024-09-18T08:16:53.739 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:16:53.739 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:16:53.739 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-09-18T08:16:53.739 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:51.712088+0000", 2024-09-18T08:16:53.740 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 6.6921000000000005e-05 2024-09-18T08:16:53.740 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:16:53.740 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:16:53.740 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "waiting for peered", 2024-09-18T08:16:53.740 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:51.712101+0000", 2024-09-18T08:16:53.740 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.2768e-05 2024-09-18T08:16:53.740 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:16:53.740 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:16:53.740 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-09-18T08:16:53.740 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:52.697039+0000", 2024-09-18T08:16:53.740 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.98493815299999998 2024-09-18T08:16:53.740 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:16:53.740 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:16:53.741 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "waiting for peered", 2024-09-18T08:16:53.741 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:52.697052+0000", 2024-09-18T08:16:53.741 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.2475e-05 2024-09-18T08:16:53.741 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:16:53.741 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:16:53.741 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-09-18T08:16:53.741 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:53.827400+0000", 2024-09-18T08:16:53.741 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.1303482419999999 2024-09-18T08:16:53.741 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:16:53.741 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:16:53.741 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "started", 2024-09-18T08:16:53.741 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:53.827528+0000", 2024-09-18T08:16:53.742 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.000128009 2024-09-18T08:16:53.742 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:16:53.742 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:16:53.742 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "waiting for subops from 1,2,4", 2024-09-18T08:16:53.742 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:53.827701+0000", 2024-09-18T08:16:53.742 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.00017307100000000001 2024-09-18T08:16:53.742 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:16:53.742 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:16:53.742 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:16:53.742 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:53.830376+0000", 2024-09-18T08:16:53.742 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.002675144 2024-09-18T08:16:53.743 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:16:53.743 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:16:53.743 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:16:53.743 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:53.830419+0000", 2024-09-18T08:16:53.743 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 4.3087000000000003e-05 2024-09-18T08:16:53.743 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:16:53.743 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:16:53.743 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "op_commit", 2024-09-18T08:16:53.743 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:53.830444+0000", 2024-09-18T08:16:53.743 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 2.4579e-05 2024-09-18T08:16:53.743 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:16:53.743 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:16:53.743 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:16:53.744 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:53.830485+0000", 2024-09-18T08:16:53.744 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 4.1236999999999998e-05 2024-09-18T08:16:53.744 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:16:53.744 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:16:53.744 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "commit_sent", 2024-09-18T08:16:53.744 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:53.830541+0000", 2024-09-18T08:16:53.744 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 5.6075000000000001e-05 2024-09-18T08:16:53.744 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:16:53.744 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:16:53.744 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "done", 2024-09-18T08:16:53.744 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:53.830668+0000", 2024-09-18T08:16:53.744 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.00012629399999999999 2024-09-18T08:16:53.744 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-09-18T08:16:53.745 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-09-18T08:16:53.745 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-09-18T08:16:53.745 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:16:53.745 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:16:53.745 INFO:teuthology.orchestra.run.smithi007.stdout: "description": "osd_op(client.15216.0:69207 8.f 8:f63411d3:::benchmark_data_smithi007_101583_object69206:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e198)", 2024-09-18T08:16:53.745 INFO:teuthology.orchestra.run.smithi007.stdout: "initiated_at": "2024-09-18T08:08:51.719518+0000", 2024-09-18T08:16:53.745 INFO:teuthology.orchestra.run.smithi007.stdout: "age": 482.00092400800003, 2024-09-18T08:16:53.745 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 2.1171781319999998, 2024-09-18T08:16:53.745 INFO:teuthology.orchestra.run.smithi007.stdout: "continuous": false, 2024-09-18T08:16:53.745 INFO:teuthology.orchestra.run.smithi007.stdout: "type_data": { 2024-09-18T08:16:53.746 INFO:teuthology.orchestra.run.smithi007.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T08:16:53.746 INFO:teuthology.orchestra.run.smithi007.stdout: "client_info": { 2024-09-18T08:16:53.746 INFO:teuthology.orchestra.run.smithi007.stdout: "client": "client.15216", 2024-09-18T08:16:53.746 INFO:teuthology.orchestra.run.smithi007.stdout: "client_addr": "172.21.15.7:0/2596041177", 2024-09-18T08:16:53.746 INFO:teuthology.orchestra.run.smithi007.stdout: "tid": 69207 2024-09-18T08:16:53.746 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:16:53.746 INFO:teuthology.orchestra.run.smithi007.stdout: "events": [ 2024-09-18T08:16:53.746 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:16:53.746 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "initiated", 2024-09-18T08:16:53.746 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:51.719518+0000", 2024-09-18T08:16:53.747 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-09-18T08:16:53.747 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:16:53.747 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:16:53.747 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "header_read", 2024-09-18T08:16:53.747 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:51.719518+0000", 2024-09-18T08:16:53.747 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-09-18T08:16:53.747 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:16:53.747 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:16:53.747 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "throttled", 2024-09-18T08:16:53.747 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:51.719519+0000", 2024-09-18T08:16:53.747 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 5.7999999999999995e-07 2024-09-18T08:16:53.747 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:16:53.747 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:16:53.747 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "all_read", 2024-09-18T08:16:53.748 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:51.719540+0000", 2024-09-18T08:16:53.748 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 2.1097000000000001e-05 2024-09-18T08:16:53.748 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:16:53.748 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:16:53.748 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "dispatched", 2024-09-18T08:16:53.748 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:51.719547+0000", 2024-09-18T08:16:53.748 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 6.4810000000000003e-06 2024-09-18T08:16:53.748 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:16:53.748 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:16:53.748 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "queued_for_pg", 2024-09-18T08:16:53.748 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:51.719552+0000", 2024-09-18T08:16:53.748 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 5.2100000000000001e-06 2024-09-18T08:16:53.748 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:16:53.749 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:16:53.749 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-09-18T08:16:53.749 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:51.719676+0000", 2024-09-18T08:16:53.749 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.00012415899999999999 2024-09-18T08:16:53.749 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:16:53.749 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:16:53.749 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "waiting for peered", 2024-09-18T08:16:53.750 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:51.719686+0000", 2024-09-18T08:16:53.750 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 9.5000000000000005e-06 2024-09-18T08:16:53.750 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:16:53.750 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:16:53.750 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-09-18T08:16:53.750 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:52.697757+0000", 2024-09-18T08:16:53.751 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.97807110100000005 2024-09-18T08:16:53.751 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:16:53.751 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:16:53.751 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "waiting for peered", 2024-09-18T08:16:53.751 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:52.697767+0000", 2024-09-18T08:16:53.751 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 9.8810000000000008e-06 2024-09-18T08:16:53.752 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:16:53.752 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:16:53.752 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-09-18T08:16:53.752 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:53.830718+0000", 2024-09-18T08:16:53.752 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.1329519370000001 2024-09-18T08:16:53.752 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:16:53.753 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:16:53.753 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "started", 2024-09-18T08:16:53.753 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:53.830870+0000", 2024-09-18T08:16:53.753 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.00015120300000000001 2024-09-18T08:16:53.753 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:16:53.753 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:16:53.754 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "waiting for subops from 1,2,4", 2024-09-18T08:16:53.754 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:53.831043+0000", 2024-09-18T08:16:53.754 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.00017385499999999999 2024-09-18T08:16:53.754 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:16:53.754 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:16:53.754 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "op_commit", 2024-09-18T08:16:53.754 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:53.836372+0000", 2024-09-18T08:16:53.755 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.0053287689999999997 2024-09-18T08:16:53.755 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:16:53.755 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:16:53.755 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:16:53.755 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:53.836420+0000", 2024-09-18T08:16:53.755 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 4.7490000000000001e-05 2024-09-18T08:16:53.755 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:16:53.756 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:16:53.756 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:16:53.756 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:53.836464+0000", 2024-09-18T08:16:53.756 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 4.4254e-05 2024-09-18T08:16:53.756 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:16:53.756 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:16:53.757 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:16:53.757 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:53.836505+0000", 2024-09-18T08:16:53.757 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 4.1414e-05 2024-09-18T08:16:53.757 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:16:53.757 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:16:53.757 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "commit_sent", 2024-09-18T08:16:53.758 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:53.836566+0000", 2024-09-18T08:16:53.758 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 6.0538999999999998e-05 2024-09-18T08:16:53.758 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:16:53.758 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:16:53.758 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "done", 2024-09-18T08:16:53.758 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:53.836697+0000", 2024-09-18T08:16:53.759 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.000130662 2024-09-18T08:16:53.759 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-09-18T08:16:53.759 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-09-18T08:16:53.759 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-09-18T08:16:53.759 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:16:53.759 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:16:53.759 INFO:teuthology.orchestra.run.smithi007.stdout: "description": "osd_op(client.15216.0:69218 8.f 8:fd5c3396:::benchmark_data_smithi007_101583_object69217:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e198)", 2024-09-18T08:16:53.759 INFO:teuthology.orchestra.run.smithi007.stdout: "initiated_at": "2024-09-18T08:08:51.731702+0000", 2024-09-18T08:16:53.759 INFO:teuthology.orchestra.run.smithi007.stdout: "age": 481.98874075200001, 2024-09-18T08:16:53.759 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 2.1112667140000001, 2024-09-18T08:16:53.760 INFO:teuthology.orchestra.run.smithi007.stdout: "continuous": false, 2024-09-18T08:16:53.760 INFO:teuthology.orchestra.run.smithi007.stdout: "type_data": { 2024-09-18T08:16:53.760 INFO:teuthology.orchestra.run.smithi007.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T08:16:53.760 INFO:teuthology.orchestra.run.smithi007.stdout: "client_info": { 2024-09-18T08:16:53.760 INFO:teuthology.orchestra.run.smithi007.stdout: "client": "client.15216", 2024-09-18T08:16:53.760 INFO:teuthology.orchestra.run.smithi007.stdout: "client_addr": "172.21.15.7:0/2596041177", 2024-09-18T08:16:53.760 INFO:teuthology.orchestra.run.smithi007.stdout: "tid": 69218 2024-09-18T08:16:53.760 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:16:53.760 INFO:teuthology.orchestra.run.smithi007.stdout: "events": [ 2024-09-18T08:16:53.760 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:16:53.760 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "initiated", 2024-09-18T08:16:53.761 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:51.731702+0000", 2024-09-18T08:16:53.761 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-09-18T08:16:53.761 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:16:53.761 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:16:53.761 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "header_read", 2024-09-18T08:16:53.761 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:51.731702+0000", 2024-09-18T08:16:53.761 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-09-18T08:16:53.761 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:16:53.761 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:16:53.761 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "throttled", 2024-09-18T08:16:53.761 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:51.731703+0000", 2024-09-18T08:16:53.761 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.265e-06 2024-09-18T08:16:53.761 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:16:53.762 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:16:53.762 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "all_read", 2024-09-18T08:16:53.762 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:51.731771+0000", 2024-09-18T08:16:53.762 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 6.7945000000000001e-05 2024-09-18T08:16:53.762 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:16:53.762 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:16:53.762 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "dispatched", 2024-09-18T08:16:53.763 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:51.731781+0000", 2024-09-18T08:16:53.763 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 9.8030000000000008e-06 2024-09-18T08:16:53.763 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:16:53.763 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:16:53.763 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "queued_for_pg", 2024-09-18T08:16:53.763 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:51.731788+0000", 2024-09-18T08:16:53.764 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 7.1620000000000002e-06 2024-09-18T08:16:53.764 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:16:53.764 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:16:53.764 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-09-18T08:16:53.764 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:51.731861+0000", 2024-09-18T08:16:53.764 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 7.2669999999999994e-05 2024-09-18T08:16:53.764 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:16:53.764 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:16:53.764 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "waiting for peered", 2024-09-18T08:16:53.764 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:51.731875+0000", 2024-09-18T08:16:53.764 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.4209999999999999e-05 2024-09-18T08:16:53.765 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:16:53.765 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:16:53.765 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-09-18T08:16:53.765 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:52.697807+0000", 2024-09-18T08:16:53.765 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.96593184200000004 2024-09-18T08:16:53.765 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:16:53.765 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:16:53.765 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "waiting for peered", 2024-09-18T08:16:53.765 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:52.697815+0000", 2024-09-18T08:16:53.765 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 8.4239999999999993e-06 2024-09-18T08:16:53.765 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:16:53.766 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:16:53.766 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-09-18T08:16:53.766 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:53.833616+0000", 2024-09-18T08:16:53.766 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.1358008079999999 2024-09-18T08:16:53.766 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:16:53.766 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:16:53.766 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "started", 2024-09-18T08:16:53.766 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:53.833763+0000", 2024-09-18T08:16:53.766 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.000146663 2024-09-18T08:16:53.766 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:16:53.766 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:16:53.766 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "waiting for subops from 1,2,4", 2024-09-18T08:16:53.766 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:53.833911+0000", 2024-09-18T08:16:53.766 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.00014877900000000001 2024-09-18T08:16:53.767 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:16:53.767 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:16:53.767 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:16:53.767 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:53.836740+0000", 2024-09-18T08:16:53.767 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.002828882 2024-09-18T08:16:53.767 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:16:53.767 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:16:53.767 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "op_commit", 2024-09-18T08:16:53.767 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:53.836756+0000", 2024-09-18T08:16:53.767 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.6195999999999999e-05 2024-09-18T08:16:53.767 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:16:53.767 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:16:53.767 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:16:53.767 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:53.842695+0000", 2024-09-18T08:16:53.768 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.0059383209999999999 2024-09-18T08:16:53.768 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:16:53.768 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:16:53.768 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:16:53.768 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:53.842746+0000", 2024-09-18T08:16:53.768 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 5.0966e-05 2024-09-18T08:16:53.768 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:16:53.768 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:16:53.768 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "commit_sent", 2024-09-18T08:16:53.768 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:53.842817+0000", 2024-09-18T08:16:53.768 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 7.1437000000000001e-05 2024-09-18T08:16:53.769 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:16:53.769 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:16:53.769 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "done", 2024-09-18T08:16:53.769 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:53.842968+0000", 2024-09-18T08:16:53.769 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.00015134100000000001 2024-09-18T08:16:53.769 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-09-18T08:16:53.769 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-09-18T08:16:53.769 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-09-18T08:16:53.769 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:16:53.769 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:16:53.769 INFO:teuthology.orchestra.run.smithi007.stdout: "description": "osd_op(client.15216.0:69239 8.f 8:f2d8399e:::benchmark_data_smithi007_101583_object69238:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e198)", 2024-09-18T08:16:53.769 INFO:teuthology.orchestra.run.smithi007.stdout: "initiated_at": "2024-09-18T08:08:51.752739+0000", 2024-09-18T08:16:53.770 INFO:teuthology.orchestra.run.smithi007.stdout: "age": 481.96770322100002, 2024-09-18T08:16:53.770 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 2.0962772900000002, 2024-09-18T08:16:53.770 INFO:teuthology.orchestra.run.smithi007.stdout: "continuous": false, 2024-09-18T08:16:53.770 INFO:teuthology.orchestra.run.smithi007.stdout: "type_data": { 2024-09-18T08:16:53.770 INFO:teuthology.orchestra.run.smithi007.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T08:16:53.770 INFO:teuthology.orchestra.run.smithi007.stdout: "client_info": { 2024-09-18T08:16:53.770 INFO:teuthology.orchestra.run.smithi007.stdout: "client": "client.15216", 2024-09-18T08:16:53.770 INFO:teuthology.orchestra.run.smithi007.stdout: "client_addr": "172.21.15.7:0/2596041177", 2024-09-18T08:16:53.770 INFO:teuthology.orchestra.run.smithi007.stdout: "tid": 69239 2024-09-18T08:16:53.770 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:16:53.770 INFO:teuthology.orchestra.run.smithi007.stdout: "events": [ 2024-09-18T08:16:53.771 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:16:53.771 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "initiated", 2024-09-18T08:16:53.771 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:51.752739+0000", 2024-09-18T08:16:53.771 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-09-18T08:16:53.771 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:16:53.771 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:16:53.771 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "header_read", 2024-09-18T08:16:53.771 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:51.752739+0000", 2024-09-18T08:16:53.771 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-09-18T08:16:53.771 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:16:53.771 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:16:53.771 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "throttled", 2024-09-18T08:16:53.771 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:51.752741+0000", 2024-09-18T08:16:53.771 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.829e-06 2024-09-18T08:16:53.772 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:16:53.772 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:16:53.772 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "all_read", 2024-09-18T08:16:53.772 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:51.752766+0000", 2024-09-18T08:16:53.772 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 2.4842e-05 2024-09-18T08:16:53.772 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:16:53.772 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:16:53.772 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "dispatched", 2024-09-18T08:16:53.772 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:51.752777+0000", 2024-09-18T08:16:53.772 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.1395e-05 2024-09-18T08:16:53.772 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:16:53.772 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:16:53.772 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "queued_for_pg", 2024-09-18T08:16:53.773 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:51.752785+0000", 2024-09-18T08:16:53.773 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 7.401e-06 2024-09-18T08:16:53.773 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:16:53.773 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:16:53.773 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-09-18T08:16:53.773 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:51.753101+0000", 2024-09-18T08:16:53.773 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.00031663600000000002 2024-09-18T08:16:53.773 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:16:53.773 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:16:53.773 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "waiting for peered", 2024-09-18T08:16:53.773 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:51.753112+0000", 2024-09-18T08:16:53.773 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.0825000000000001e-05 2024-09-18T08:16:53.773 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:16:53.774 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:16:53.774 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-09-18T08:16:53.774 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:52.697883+0000", 2024-09-18T08:16:53.774 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.94477077700000001 2024-09-18T08:16:53.774 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:16:53.774 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:16:53.774 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "waiting for peered", 2024-09-18T08:16:53.774 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:52.697893+0000", 2024-09-18T08:16:53.774 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 9.5820000000000008e-06 2024-09-18T08:16:53.774 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:16:53.774 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:16:53.774 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-09-18T08:16:53.775 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:53.836781+0000", 2024-09-18T08:16:53.775 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.1388879329999999 2024-09-18T08:16:53.775 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:16:53.775 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:16:53.775 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "started", 2024-09-18T08:16:53.775 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:53.836912+0000", 2024-09-18T08:16:53.775 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.00013171799999999999 2024-09-18T08:16:53.775 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:16:53.775 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:16:53.775 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "waiting for subops from 1,2,4", 2024-09-18T08:16:53.775 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:53.837063+0000", 2024-09-18T08:16:53.775 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.00015040799999999999 2024-09-18T08:16:53.775 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:16:53.776 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:16:53.776 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:16:53.776 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:53.843015+0000", 2024-09-18T08:16:53.776 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.0059519250000000003 2024-09-18T08:16:53.776 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:16:53.776 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:16:53.776 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:16:53.776 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:53.843063+0000", 2024-09-18T08:16:53.776 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 4.8199999999999999e-05 2024-09-18T08:16:53.776 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:16:53.776 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:16:53.776 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "op_commit", 2024-09-18T08:16:53.776 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:53.848550+0000", 2024-09-18T08:16:53.777 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.0054868929999999996 2024-09-18T08:16:53.777 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:16:53.777 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:16:53.777 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:16:53.777 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:53.848663+0000", 2024-09-18T08:16:53.777 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.000113195 2024-09-18T08:16:53.777 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:16:53.777 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:16:53.777 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "commit_sent", 2024-09-18T08:16:53.777 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:53.848860+0000", 2024-09-18T08:16:53.777 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.00019676000000000001 2024-09-18T08:16:53.777 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:16:53.777 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:16:53.778 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "done", 2024-09-18T08:16:53.778 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:53.849017+0000", 2024-09-18T08:16:53.778 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.000156971 2024-09-18T08:16:53.778 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-09-18T08:16:53.778 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-09-18T08:16:53.778 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-09-18T08:16:53.778 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:16:53.778 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:16:53.778 INFO:teuthology.orchestra.run.smithi007.stdout: "description": "osd_op(client.15216.0:69262 8.f 8:f2d117f8:::benchmark_data_smithi007_101583_object69261:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e198)", 2024-09-18T08:16:53.778 INFO:teuthology.orchestra.run.smithi007.stdout: "initiated_at": "2024-09-18T08:08:51.776735+0000", 2024-09-18T08:16:53.778 INFO:teuthology.orchestra.run.smithi007.stdout: "age": 481.94370758100001, 2024-09-18T08:16:53.778 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 2.0727967970000001, 2024-09-18T08:16:53.778 INFO:teuthology.orchestra.run.smithi007.stdout: "continuous": false, 2024-09-18T08:16:53.779 INFO:teuthology.orchestra.run.smithi007.stdout: "type_data": { 2024-09-18T08:16:53.779 INFO:teuthology.orchestra.run.smithi007.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T08:16:53.779 INFO:teuthology.orchestra.run.smithi007.stdout: "client_info": { 2024-09-18T08:16:53.779 INFO:teuthology.orchestra.run.smithi007.stdout: "client": "client.15216", 2024-09-18T08:16:53.779 INFO:teuthology.orchestra.run.smithi007.stdout: "client_addr": "172.21.15.7:0/2596041177", 2024-09-18T08:16:53.779 INFO:teuthology.orchestra.run.smithi007.stdout: "tid": 69262 2024-09-18T08:16:53.779 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:16:53.779 INFO:teuthology.orchestra.run.smithi007.stdout: "events": [ 2024-09-18T08:16:53.779 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:16:53.779 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "initiated", 2024-09-18T08:16:53.779 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:51.776735+0000", 2024-09-18T08:16:53.779 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-09-18T08:16:53.779 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:16:53.780 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:16:53.780 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "header_read", 2024-09-18T08:16:53.780 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:51.776735+0000", 2024-09-18T08:16:53.780 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-09-18T08:16:53.780 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:16:53.780 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:16:53.780 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "throttled", 2024-09-18T08:16:53.780 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:51.776737+0000", 2024-09-18T08:16:53.780 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 2.0990000000000001e-06 2024-09-18T08:16:53.780 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:16:53.780 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:16:53.780 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "all_read", 2024-09-18T08:16:53.780 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:51.776765+0000", 2024-09-18T08:16:53.781 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 2.7562000000000001e-05 2024-09-18T08:16:53.781 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:16:53.781 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:16:53.781 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "dispatched", 2024-09-18T08:16:53.781 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:51.776785+0000", 2024-09-18T08:16:53.781 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 2.0136000000000002e-05 2024-09-18T08:16:53.781 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:16:53.781 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:16:53.781 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "queued_for_pg", 2024-09-18T08:16:53.781 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:51.776794+0000", 2024-09-18T08:16:53.781 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 9.3109999999999995e-06 2024-09-18T08:16:53.781 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:16:53.781 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:16:53.782 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-09-18T08:16:53.782 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:51.776887+0000", 2024-09-18T08:16:53.782 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 9.3382999999999997e-05 2024-09-18T08:16:53.782 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:16:53.782 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:16:53.782 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "waiting for peered", 2024-09-18T08:16:53.782 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:51.776898+0000", 2024-09-18T08:16:53.782 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.0278999999999999e-05 2024-09-18T08:16:53.782 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:16:53.782 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:16:53.782 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-09-18T08:16:53.782 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:52.697921+0000", 2024-09-18T08:16:53.782 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.92102297700000002 2024-09-18T08:16:53.783 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:16:53.783 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:16:53.783 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "waiting for peered", 2024-09-18T08:16:53.783 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:52.697929+0000", 2024-09-18T08:16:53.783 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 8.0709999999999995e-06 2024-09-18T08:16:53.783 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:16:53.783 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:16:53.783 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-09-18T08:16:53.783 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:53.839602+0000", 2024-09-18T08:16:53.783 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.1416730690000001 2024-09-18T08:16:53.783 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:16:53.783 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:16:53.783 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "started", 2024-09-18T08:16:53.784 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:53.839748+0000", 2024-09-18T08:16:53.784 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.00014631299999999999 2024-09-18T08:16:53.784 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:16:53.784 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:16:53.784 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "waiting for subops from 1,2,4", 2024-09-18T08:16:53.784 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:53.839927+0000", 2024-09-18T08:16:53.784 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.00017868800000000001 2024-09-18T08:16:53.784 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:16:53.784 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:16:53.784 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:16:53.784 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:53.843113+0000", 2024-09-18T08:16:53.785 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.003186595 2024-09-18T08:16:53.785 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:16:53.785 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:16:53.785 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:16:53.785 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:53.848599+0000", 2024-09-18T08:16:53.785 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.0054851049999999997 2024-09-18T08:16:53.785 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:16:53.785 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:16:53.786 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "op_commit", 2024-09-18T08:16:53.786 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:53.848616+0000", 2024-09-18T08:16:53.786 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.785e-05 2024-09-18T08:16:53.786 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:16:53.786 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:16:53.786 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:16:53.786 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:53.849125+0000", 2024-09-18T08:16:53.786 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.000508579 2024-09-18T08:16:53.786 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:16:53.786 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:16:53.786 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "commit_sent", 2024-09-18T08:16:53.786 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:53.849311+0000", 2024-09-18T08:16:53.787 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.00018621299999999999 2024-09-18T08:16:53.787 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:16:53.787 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:16:53.787 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "done", 2024-09-18T08:16:53.787 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:08:53.849532+0000", 2024-09-18T08:16:53.787 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.00022056699999999999 2024-09-18T08:16:53.787 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-09-18T08:16:53.787 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-09-18T08:16:53.787 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-09-18T08:16:53.787 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:16:53.788 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:16:53.788 INFO:teuthology.orchestra.run.smithi007.stdout: "description": "osd_op(client.15582.0:252481 9.1 9:8590ecfa:::benchmark_data_smithi007_124019_object110596:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e291)", 2024-09-18T08:16:53.788 INFO:teuthology.orchestra.run.smithi007.stdout: "initiated_at": "2024-09-18T08:14:04.554046+0000", 2024-09-18T08:16:53.788 INFO:teuthology.orchestra.run.smithi007.stdout: "age": 169.16639612500001, 2024-09-18T08:16:53.788 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 2.257659055, 2024-09-18T08:16:53.788 INFO:teuthology.orchestra.run.smithi007.stdout: "continuous": false, 2024-09-18T08:16:53.788 INFO:teuthology.orchestra.run.smithi007.stdout: "type_data": { 2024-09-18T08:16:53.788 INFO:teuthology.orchestra.run.smithi007.stdout: "flag_point": "reached pg", 2024-09-18T08:16:53.788 INFO:teuthology.orchestra.run.smithi007.stdout: "client_info": { 2024-09-18T08:16:53.788 INFO:teuthology.orchestra.run.smithi007.stdout: "client": "client.15582", 2024-09-18T08:16:53.788 INFO:teuthology.orchestra.run.smithi007.stdout: "client_addr": "172.21.15.7:0/3598605839", 2024-09-18T08:16:53.789 INFO:teuthology.orchestra.run.smithi007.stdout: "tid": 252481 2024-09-18T08:16:53.789 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:16:53.789 INFO:teuthology.orchestra.run.smithi007.stdout: "events": [ 2024-09-18T08:16:53.789 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:16:53.789 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "initiated", 2024-09-18T08:16:53.789 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:14:04.554046+0000", 2024-09-18T08:16:53.789 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-09-18T08:16:53.789 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:16:53.789 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:16:53.789 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "header_read", 2024-09-18T08:16:53.790 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:14:04.554046+0000", 2024-09-18T08:16:53.790 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-09-18T08:16:53.790 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:16:53.790 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:16:53.790 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "throttled", 2024-09-18T08:16:53.790 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:14:04.554048+0000", 2024-09-18T08:16:53.790 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.4270000000000001e-06 2024-09-18T08:16:53.790 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:16:53.790 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:16:53.790 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "all_read", 2024-09-18T08:16:53.790 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:14:04.554054+0000", 2024-09-18T08:16:53.791 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 6.1800000000000001e-06 2024-09-18T08:16:53.791 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:16:53.791 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:16:53.791 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "dispatched", 2024-09-18T08:16:53.791 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:14:04.554064+0000", 2024-09-18T08:16:53.791 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.0468e-05 2024-09-18T08:16:53.791 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:16:53.791 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:16:53.791 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "queued_for_pg", 2024-09-18T08:16:53.791 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:14:04.554072+0000", 2024-09-18T08:16:53.791 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 7.2089999999999999e-06 2024-09-18T08:16:53.791 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:16:53.792 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:16:53.792 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-09-18T08:16:53.792 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:14:04.557405+0000", 2024-09-18T08:16:53.792 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.0033328590000000001 2024-09-18T08:16:53.792 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:16:53.792 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:16:53.792 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "started", 2024-09-18T08:16:53.792 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:14:04.557507+0000", 2024-09-18T08:16:53.792 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.000102092 2024-09-18T08:16:53.792 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:16:53.792 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:16:53.792 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "waiting for subops from 6,7", 2024-09-18T08:16:53.792 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:14:04.557610+0000", 2024-09-18T08:16:53.793 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.000103459 2024-09-18T08:16:53.793 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:16:53.793 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:16:53.793 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:16:53.793 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:14:04.561424+0000", 2024-09-18T08:16:53.793 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.0038139010000000002 2024-09-18T08:16:53.793 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:16:53.793 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:16:53.793 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "op_commit", 2024-09-18T08:16:53.793 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:14:04.565070+0000", 2024-09-18T08:16:53.793 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.0036458910000000001 2024-09-18T08:16:53.793 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:16:53.793 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:16:53.794 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-09-18T08:16:53.794 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:14:04.593896+0000", 2024-09-18T08:16:53.794 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.028826368000000002 2024-09-18T08:16:53.794 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:16:53.794 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:16:53.794 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "waiting for peered", 2024-09-18T08:16:53.794 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:14:04.593907+0000", 2024-09-18T08:16:53.794 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.0774999999999999e-05 2024-09-18T08:16:53.794 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:16:53.794 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:16:53.794 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-09-18T08:16:53.794 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:14:05.577311+0000", 2024-09-18T08:16:53.794 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.98340420900000003 2024-09-18T08:16:53.794 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:16:53.795 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:16:53.795 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "waiting for peered", 2024-09-18T08:16:53.795 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:14:05.577322+0000", 2024-09-18T08:16:53.795 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.0917999999999999e-05 2024-09-18T08:16:53.795 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:16:53.795 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:16:53.795 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-09-18T08:16:53.795 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:14:06.811620+0000", 2024-09-18T08:16:53.795 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.234297778 2024-09-18T08:16:53.795 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:16:53.795 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:16:53.795 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "done", 2024-09-18T08:16:53.795 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:14:06.811705+0000", 2024-09-18T08:16:53.796 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 8.5520999999999997e-05 2024-09-18T08:16:53.796 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-09-18T08:16:53.796 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-09-18T08:16:53.796 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-09-18T08:16:53.796 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:16:53.796 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:16:53.796 INFO:teuthology.orchestra.run.smithi007.stdout: "description": "osd_op(client.24356.0:2783 5.14 5:2b04a3e9:::notify.6:head [watch reconnect cookie 94700928708992 gen 2] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e292)", 2024-09-18T08:16:53.796 INFO:teuthology.orchestra.run.smithi007.stdout: "initiated_at": "2024-09-18T08:14:04.565629+0000", 2024-09-18T08:16:53.796 INFO:teuthology.orchestra.run.smithi007.stdout: "age": 169.15481323099999, 2024-09-18T08:16:53.796 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 2.057368002, 2024-09-18T08:16:53.797 INFO:teuthology.orchestra.run.smithi007.stdout: "continuous": false, 2024-09-18T08:16:53.797 INFO:teuthology.orchestra.run.smithi007.stdout: "type_data": { 2024-09-18T08:16:53.797 INFO:teuthology.orchestra.run.smithi007.stdout: "flag_point": "started", 2024-09-18T08:16:53.797 INFO:teuthology.orchestra.run.smithi007.stdout: "client_info": { 2024-09-18T08:16:53.797 INFO:teuthology.orchestra.run.smithi007.stdout: "client": "client.24356", 2024-09-18T08:16:53.797 INFO:teuthology.orchestra.run.smithi007.stdout: "client_addr": "172.21.15.7:0/219168083", 2024-09-18T08:16:53.797 INFO:teuthology.orchestra.run.smithi007.stdout: "tid": 2783 2024-09-18T08:16:53.797 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:16:53.798 INFO:teuthology.orchestra.run.smithi007.stdout: "events": [ 2024-09-18T08:16:53.798 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:16:53.798 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "initiated", 2024-09-18T08:16:53.798 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:14:04.565629+0000", 2024-09-18T08:16:53.798 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-09-18T08:16:53.798 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:16:53.798 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:16:53.798 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "header_read", 2024-09-18T08:16:53.798 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:14:04.565629+0000", 2024-09-18T08:16:53.798 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-09-18T08:16:53.799 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:16:53.799 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:16:53.799 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "throttled", 2024-09-18T08:16:53.799 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:14:04.565631+0000", 2024-09-18T08:16:53.799 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.2899999999999999e-06 2024-09-18T08:16:53.799 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:16:53.799 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:16:53.799 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "all_read", 2024-09-18T08:16:53.799 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:14:04.565636+0000", 2024-09-18T08:16:53.799 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 5.0379999999999999e-06 2024-09-18T08:16:53.799 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:16:53.799 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:16:53.800 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "dispatched", 2024-09-18T08:16:53.800 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:14:04.565647+0000", 2024-09-18T08:16:53.800 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.154e-05 2024-09-18T08:16:53.800 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:16:53.800 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:16:53.800 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "queued_for_pg", 2024-09-18T08:16:53.800 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:14:04.565654+0000", 2024-09-18T08:16:53.800 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 7.3479999999999998e-06 2024-09-18T08:16:53.800 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:16:53.800 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:16:53.800 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-09-18T08:16:53.800 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:14:04.565753+0000", 2024-09-18T08:16:53.800 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 9.8768e-05 2024-09-18T08:16:53.801 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:16:53.801 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:16:53.801 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "op must wait for map", 2024-09-18T08:16:53.801 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:14:04.565912+0000", 2024-09-18T08:16:53.801 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.00015838199999999999 2024-09-18T08:16:53.801 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:16:53.801 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:16:53.801 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-09-18T08:16:53.801 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:14:04.587017+0000", 2024-09-18T08:16:53.801 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.021105261 2024-09-18T08:16:53.801 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:16:53.802 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:16:53.802 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "waiting for peered", 2024-09-18T08:16:53.802 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:14:04.587024+0000", 2024-09-18T08:16:53.802 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 6.6449999999999999e-06 2024-09-18T08:16:53.802 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:16:53.802 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:16:53.802 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-09-18T08:16:53.802 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:14:05.582156+0000", 2024-09-18T08:16:53.802 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.99513244899999997 2024-09-18T08:16:53.802 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:16:53.802 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:16:53.802 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "waiting for peered", 2024-09-18T08:16:53.802 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:14:05.582170+0000", 2024-09-18T08:16:53.803 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.4248e-05 2024-09-18T08:16:53.803 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:16:53.803 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:16:53.803 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-09-18T08:16:53.803 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:14:06.605977+0000", 2024-09-18T08:16:53.803 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.0238063529999999 2024-09-18T08:16:53.803 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:16:53.803 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:16:53.803 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "started", 2024-09-18T08:16:53.803 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:14:06.606129+0000", 2024-09-18T08:16:53.803 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.00015248799999999999 2024-09-18T08:16:53.803 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:16:53.803 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:16:53.804 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "done", 2024-09-18T08:16:53.804 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:14:06.622997+0000", 2024-09-18T08:16:53.804 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.016868192000000001 2024-09-18T08:16:53.804 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-09-18T08:16:53.804 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-09-18T08:16:53.804 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-09-18T08:16:53.804 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:16:53.804 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:16:53.804 INFO:teuthology.orchestra.run.smithi007.stdout: "description": "osd_op(client.15582.0:252490 9.1 9:891fe784:::benchmark_data_smithi007_124019_object110605:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e291)", 2024-09-18T08:16:53.804 INFO:teuthology.orchestra.run.smithi007.stdout: "initiated_at": "2024-09-18T08:14:04.565679+0000", 2024-09-18T08:16:53.804 INFO:teuthology.orchestra.run.smithi007.stdout: "age": 169.15476351300001, 2024-09-18T08:16:53.804 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 2.2588554969999999, 2024-09-18T08:16:53.805 INFO:teuthology.orchestra.run.smithi007.stdout: "continuous": false, 2024-09-18T08:16:53.805 INFO:teuthology.orchestra.run.smithi007.stdout: "type_data": { 2024-09-18T08:16:53.805 INFO:teuthology.orchestra.run.smithi007.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T08:16:53.805 INFO:teuthology.orchestra.run.smithi007.stdout: "client_info": { 2024-09-18T08:16:53.805 INFO:teuthology.orchestra.run.smithi007.stdout: "client": "client.15582", 2024-09-18T08:16:53.805 INFO:teuthology.orchestra.run.smithi007.stdout: "client_addr": "172.21.15.7:0/3598605839", 2024-09-18T08:16:53.805 INFO:teuthology.orchestra.run.smithi007.stdout: "tid": 252490 2024-09-18T08:16:53.805 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:16:53.805 INFO:teuthology.orchestra.run.smithi007.stdout: "events": [ 2024-09-18T08:16:53.805 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:16:53.806 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "initiated", 2024-09-18T08:16:53.806 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:14:04.565679+0000", 2024-09-18T08:16:53.806 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-09-18T08:16:53.806 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:16:53.806 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:16:53.806 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "header_read", 2024-09-18T08:16:53.806 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:14:04.565679+0000", 2024-09-18T08:16:53.806 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-09-18T08:16:53.806 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:16:53.806 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:16:53.806 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "throttled", 2024-09-18T08:16:53.806 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:14:04.565680+0000", 2024-09-18T08:16:53.807 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 8.9700000000000005e-07 2024-09-18T08:16:53.807 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:16:53.807 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:16:53.807 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "all_read", 2024-09-18T08:16:53.807 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:14:04.565683+0000", 2024-09-18T08:16:53.807 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 3.2940000000000001e-06 2024-09-18T08:16:53.807 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:16:53.807 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:16:53.807 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "dispatched", 2024-09-18T08:16:53.807 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:14:04.565691+0000", 2024-09-18T08:16:53.807 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 7.7060000000000004e-06 2024-09-18T08:16:53.807 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:16:53.807 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:16:53.807 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "queued_for_pg", 2024-09-18T08:16:53.808 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:14:04.565697+0000", 2024-09-18T08:16:53.808 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 5.9139999999999996e-06 2024-09-18T08:16:53.808 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:16:53.808 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:16:53.808 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-09-18T08:16:53.808 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:14:04.593936+0000", 2024-09-18T08:16:53.808 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.028238722000000001 2024-09-18T08:16:53.808 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:16:53.808 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:16:53.808 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "waiting for peered", 2024-09-18T08:16:53.808 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:14:04.593944+0000", 2024-09-18T08:16:53.808 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 8.2039999999999997e-06 2024-09-18T08:16:53.808 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:16:53.809 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:16:53.809 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-09-18T08:16:53.809 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:14:05.577351+0000", 2024-09-18T08:16:53.809 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.98340685800000005 2024-09-18T08:16:53.809 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:16:53.809 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:16:53.809 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "waiting for peered", 2024-09-18T08:16:53.809 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:14:05.577357+0000", 2024-09-18T08:16:53.809 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 6.8299999999999998e-06 2024-09-18T08:16:53.809 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:16:53.809 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:16:53.809 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-09-18T08:16:53.809 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:14:06.811976+0000", 2024-09-18T08:16:53.810 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.234618384 2024-09-18T08:16:53.810 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:16:53.810 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:16:53.810 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "started", 2024-09-18T08:16:53.810 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:14:06.812084+0000", 2024-09-18T08:16:53.810 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.00010839 2024-09-18T08:16:53.810 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:16:53.810 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:16:53.810 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3 2024-09-18T08:16:53.812 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "waiting for subops from 4,6,7", 2024-09-18T08:16:53.812 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:14:06.812465+0000", 2024-09-18T08:16:53.812 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.00038073600000000001 2024-09-18T08:16:53.812 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:16:53.812 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:16:53.812 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "op_commit", 2024-09-18T08:16:53.812 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:14:06.816196+0000", 2024-09-18T08:16:53.812 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.0037314879999999998 2024-09-18T08:16:53.812 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:16:53.812 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:16:53.813 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:16:53.813 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:14:06.824173+0000", 2024-09-18T08:16:53.813 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.0079768100000000008 2024-09-18T08:16:53.813 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:16:53.813 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:16:53.813 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:16:53.813 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:14:06.824235+0000", 2024-09-18T08:16:53.813 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 6.1746000000000001e-05 2024-09-18T08:16:53.813 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:16:53.813 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:16:53.813 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:16:53.814 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:14:06.824291+0000", 2024-09-18T08:16:53.814 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 5.5689999999999997e-05 2024-09-18T08:16:53.814 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:16:53.814 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:16:53.814 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "commit_sent", 2024-09-18T08:16:53.814 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:14:06.824367+0000", 2024-09-18T08:16:53.814 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 7.6524000000000001e-05 2024-09-18T08:16:53.814 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:16:53.814 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:16:53.814 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "done", 2024-09-18T08:16:53.814 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:14:06.824534+0000", 2024-09-18T08:16:53.815 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.00016730400000000001 2024-09-18T08:16:53.815 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-09-18T08:16:53.815 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-09-18T08:16:53.815 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-09-18T08:16:53.815 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:16:53.815 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:16:53.815 INFO:teuthology.orchestra.run.smithi007.stdout: "description": "osd_op(client.15582.0:252491 9.c 9:326143b5:::benchmark_data_smithi007_124019_object110606:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e291)", 2024-09-18T08:16:53.815 INFO:teuthology.orchestra.run.smithi007.stdout: "initiated_at": "2024-09-18T08:14:04.576096+0000", 2024-09-18T08:16:53.815 INFO:teuthology.orchestra.run.smithi007.stdout: "age": 169.144346474, 2024-09-18T08:16:53.815 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 2.4808478369999998, 2024-09-18T08:16:53.815 INFO:teuthology.orchestra.run.smithi007.stdout: "continuous": false, 2024-09-18T08:16:53.815 INFO:teuthology.orchestra.run.smithi007.stdout: "type_data": { 2024-09-18T08:16:53.815 INFO:teuthology.orchestra.run.smithi007.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T08:16:53.816 INFO:teuthology.orchestra.run.smithi007.stdout: "client_info": { 2024-09-18T08:16:53.816 INFO:teuthology.orchestra.run.smithi007.stdout: "client": "client.15582", 2024-09-18T08:16:53.816 INFO:teuthology.orchestra.run.smithi007.stdout: "client_addr": "172.21.15.7:0/3598605839", 2024-09-18T08:16:53.816 INFO:teuthology.orchestra.run.smithi007.stdout: "tid": 252491 2024-09-18T08:16:53.816 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:16:53.816 INFO:teuthology.orchestra.run.smithi007.stdout: "events": [ 2024-09-18T08:16:53.816 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:16:53.816 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "initiated", 2024-09-18T08:16:53.816 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:14:04.576096+0000", 2024-09-18T08:16:53.816 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-09-18T08:16:53.816 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:16:53.816 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:16:53.816 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "header_read", 2024-09-18T08:16:53.817 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:14:04.576096+0000", 2024-09-18T08:16:53.817 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-09-18T08:16:53.817 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:16:53.817 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:16:53.817 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "throttled", 2024-09-18T08:16:53.817 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:14:04.576098+0000", 2024-09-18T08:16:53.817 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.7659999999999999e-06 2024-09-18T08:16:53.817 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:16:53.817 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:16:53.817 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "all_read", 2024-09-18T08:16:53.817 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:14:04.576105+0000", 2024-09-18T08:16:53.817 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 6.8839999999999997e-06 2024-09-18T08:16:53.818 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:16:53.818 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:16:53.818 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "dispatched", 2024-09-18T08:16:53.818 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:14:04.576118+0000", 2024-09-18T08:16:53.818 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.332e-05 2024-09-18T08:16:53.818 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:16:53.818 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:16:53.818 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "queued_for_pg", 2024-09-18T08:16:53.818 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:14:04.576127+0000", 2024-09-18T08:16:53.818 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 8.9700000000000005e-06 2024-09-18T08:16:53.819 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:16:53.819 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:16:53.819 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-09-18T08:16:53.819 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:14:04.586985+0000", 2024-09-18T08:16:53.819 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.010858247 2024-09-18T08:16:53.819 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:16:53.819 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:16:53.819 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "waiting for peered", 2024-09-18T08:16:53.819 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:14:04.586995+0000", 2024-09-18T08:16:53.819 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 9.3619999999999995e-06 2024-09-18T08:16:53.820 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:16:53.820 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:16:53.820 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-09-18T08:16:53.820 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:14:05.581931+0000", 2024-09-18T08:16:53.820 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.99493684800000004 2024-09-18T08:16:53.820 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:16:53.820 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:16:53.820 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "waiting for peered", 2024-09-18T08:16:53.820 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:14:05.581944+0000", 2024-09-18T08:16:53.820 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.2191999999999999e-05 2024-09-18T08:16:53.820 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:16:53.820 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:16:53.820 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-09-18T08:16:53.821 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:14:06.798281+0000", 2024-09-18T08:16:53.821 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.2163375969999999 2024-09-18T08:16:53.821 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:16:53.821 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:16:53.821 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "started", 2024-09-18T08:16:53.821 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:14:06.798419+0000", 2024-09-18T08:16:53.821 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.000137546 2024-09-18T08:16:53.821 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:16:53.821 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:16:53.821 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "waiting for subops from 0,4,7", 2024-09-18T08:16:53.821 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:14:06.798827+0000", 2024-09-18T08:16:53.821 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.00040866200000000001 2024-09-18T08:16:53.821 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:16:53.822 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:16:53.822 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:16:53.822 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:14:06.804658+0000", 2024-09-18T08:16:53.822 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.0058310159999999996 2024-09-18T08:16:53.822 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:16:53.822 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:16:53.822 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:16:53.822 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:14:06.808523+0000", 2024-09-18T08:16:53.822 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.0038643739999999998 2024-09-18T08:16:53.822 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:16:53.822 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:16:53.822 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "op_commit", 2024-09-18T08:16:53.822 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:14:06.808532+0000", 2024-09-18T08:16:53.822 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 9.3479999999999993e-06 2024-09-18T08:16:53.823 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:16:53.823 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:16:53.823 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:16:53.823 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:14:07.056320+0000", 2024-09-18T08:16:53.823 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.247787753 2024-09-18T08:16:53.823 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:16:53.823 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:16:53.823 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "commit_sent", 2024-09-18T08:16:53.823 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:14:07.056553+0000", 2024-09-18T08:16:53.823 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.00023296999999999999 2024-09-18T08:16:53.823 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:16:53.823 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:16:53.823 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "done", 2024-09-18T08:16:53.823 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:14:07.056944+0000", 2024-09-18T08:16:53.824 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.000390982 2024-09-18T08:16:53.824 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-09-18T08:16:53.824 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-09-18T08:16:53.824 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-09-18T08:16:53.824 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:16:53.824 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:16:53.824 INFO:teuthology.orchestra.run.smithi007.stdout: "description": "osd_op(client.15582.0:252477 9.3 9:c97eebf1:::benchmark_data_smithi007_124019_object110592:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e292)", 2024-09-18T08:16:53.824 INFO:teuthology.orchestra.run.smithi007.stdout: "initiated_at": "2024-09-18T08:14:04.578529+0000", 2024-09-18T08:16:53.824 INFO:teuthology.orchestra.run.smithi007.stdout: "age": 169.14191320800001, 2024-09-18T08:16:53.824 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 2.2331727560000001, 2024-09-18T08:16:53.824 INFO:teuthology.orchestra.run.smithi007.stdout: "continuous": false, 2024-09-18T08:16:53.824 INFO:teuthology.orchestra.run.smithi007.stdout: "type_data": { 2024-09-18T08:16:53.824 INFO:teuthology.orchestra.run.smithi007.stdout: "flag_point": "reached pg", 2024-09-18T08:16:53.825 INFO:teuthology.orchestra.run.smithi007.stdout: "client_info": { 2024-09-18T08:16:53.825 INFO:teuthology.orchestra.run.smithi007.stdout: "client": "client.15582", 2024-09-18T08:16:53.825 INFO:teuthology.orchestra.run.smithi007.stdout: "client_addr": "172.21.15.7:0/3598605839", 2024-09-18T08:16:53.825 INFO:teuthology.orchestra.run.smithi007.stdout: "tid": 252477 2024-09-18T08:16:53.825 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:16:53.825 INFO:teuthology.orchestra.run.smithi007.stdout: "events": [ 2024-09-18T08:16:53.825 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:16:53.825 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "initiated", 2024-09-18T08:16:53.825 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:14:04.578529+0000", 2024-09-18T08:16:53.825 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-09-18T08:16:53.825 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:16:53.826 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:16:53.826 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "header_read", 2024-09-18T08:16:53.826 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:14:04.578529+0000", 2024-09-18T08:16:53.826 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-09-18T08:16:53.826 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:16:53.826 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:16:53.826 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "throttled", 2024-09-18T08:16:53.826 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:14:04.578530+0000", 2024-09-18T08:16:53.826 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 4.3099999999999998e-07 2024-09-18T08:16:53.826 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:16:53.826 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:16:53.826 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "all_read", 2024-09-18T08:16:53.827 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:14:04.578532+0000", 2024-09-18T08:16:53.827 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 2.5579999999999999e-06 2024-09-18T08:16:53.827 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:16:53.827 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:16:53.827 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "dispatched", 2024-09-18T08:16:53.827 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:14:04.578538+0000", 2024-09-18T08:16:53.827 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 6.2140000000000001e-06 2024-09-18T08:16:53.827 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:16:53.827 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:16:53.827 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "queued_for_pg", 2024-09-18T08:16:53.827 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:14:04.578544+0000", 2024-09-18T08:16:53.827 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 5.9769999999999997e-06 2024-09-18T08:16:53.827 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:16:53.828 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:16:53.828 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-09-18T08:16:53.828 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:14:04.578776+0000", 2024-09-18T08:16:53.828 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.000231306 2024-09-18T08:16:53.828 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:16:53.828 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:16:53.828 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "waiting for peered", 2024-09-18T08:16:53.828 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:14:04.578784+0000", 2024-09-18T08:16:53.828 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 8.0989999999999999e-06 2024-09-18T08:16:53.828 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:16:53.828 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:16:53.828 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-09-18T08:16:53.828 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:14:05.571602+0000", 2024-09-18T08:16:53.829 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.992817799 2024-09-18T08:16:53.829 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:16:53.829 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:16:53.829 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "waiting for peered", 2024-09-18T08:16:53.829 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:14:05.571612+0000", 2024-09-18T08:16:53.829 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.0336000000000001e-05 2024-09-18T08:16:53.829 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:16:53.829 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:16:53.829 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-09-18T08:16:53.829 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:14:06.811618+0000", 2024-09-18T08:16:53.829 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.240005698 2024-09-18T08:16:53.829 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:16:53.829 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:16:53.830 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "done", 2024-09-18T08:16:53.830 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:14:06.811702+0000", 2024-09-18T08:16:53.830 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 8.4338000000000005e-05 2024-09-18T08:16:53.830 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-09-18T08:16:53.830 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-09-18T08:16:53.830 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-09-18T08:16:53.830 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:16:53.830 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:16:53.830 INFO:teuthology.orchestra.run.smithi007.stdout: "description": "osd_op(client.15582.0:252478 9.3 9:ce475113:::benchmark_data_smithi007_124019_object110593:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e292)", 2024-09-18T08:16:53.830 INFO:teuthology.orchestra.run.smithi007.stdout: "initiated_at": "2024-09-18T08:14:04.578552+0000", 2024-09-18T08:16:53.830 INFO:teuthology.orchestra.run.smithi007.stdout: "age": 169.14189016099999, 2024-09-18T08:16:53.831 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 2.2335192359999998, 2024-09-18T08:16:53.831 INFO:teuthology.orchestra.run.smithi007.stdout: "continuous": false, 2024-09-18T08:16:53.831 INFO:teuthology.orchestra.run.smithi007.stdout: "type_data": { 2024-09-18T08:16:53.831 INFO:teuthology.orchestra.run.smithi007.stdout: "flag_point": "reached pg", 2024-09-18T08:16:53.831 INFO:teuthology.orchestra.run.smithi007.stdout: "client_info": { 2024-09-18T08:16:53.831 INFO:teuthology.orchestra.run.smithi007.stdout: "client": "client.15582", 2024-09-18T08:16:53.831 INFO:teuthology.orchestra.run.smithi007.stdout: "client_addr": "172.21.15.7:0/3598605839", 2024-09-18T08:16:53.831 INFO:teuthology.orchestra.run.smithi007.stdout: "tid": 252478 2024-09-18T08:16:53.831 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:16:53.831 INFO:teuthology.orchestra.run.smithi007.stdout: "events": [ 2024-09-18T08:16:53.831 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:16:53.831 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "initiated", 2024-09-18T08:16:53.831 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:14:04.578552+0000", 2024-09-18T08:16:53.831 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-09-18T08:16:53.832 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:16:53.832 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:16:53.832 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "header_read", 2024-09-18T08:16:53.832 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:14:04.578552+0000", 2024-09-18T08:16:53.832 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-09-18T08:16:53.832 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:16:53.832 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:16:53.832 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "throttled", 2024-09-18T08:16:53.832 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:14:04.578553+0000", 2024-09-18T08:16:53.832 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 4.2399999999999999e-07 2024-09-18T08:16:53.832 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:16:53.832 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:16:53.832 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "all_read", 2024-09-18T08:16:53.833 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:14:04.578555+0000", 2024-09-18T08:16:53.833 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 2.1959999999999998e-06 2024-09-18T08:16:53.833 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:16:53.833 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:16:53.833 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "dispatched", 2024-09-18T08:16:53.833 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:14:04.578560+0000", 2024-09-18T08:16:53.833 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 5.2440000000000001e-06 2024-09-18T08:16:53.833 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:16:53.833 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:16:53.833 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "queued_for_pg", 2024-09-18T08:16:53.833 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:14:04.578565+0000", 2024-09-18T08:16:53.833 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 4.6009999999999997e-06 2024-09-18T08:16:53.833 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:16:53.834 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:16:53.834 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-09-18T08:16:53.834 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:14:04.578814+0000", 2024-09-18T08:16:53.834 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.00024935199999999999 2024-09-18T08:16:53.834 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:16:53.834 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:16:53.836 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "waiting for peered", 2024-09-18T08:16:53.836 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:14:04.578820+0000", 2024-09-18T08:16:53.836 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 6.0220000000000001e-06 2024-09-18T08:16:53.836 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:16:53.836 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:16:53.836 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-09-18T08:16:53.836 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:14:05.571644+0000", 2024-09-18T08:16:53.836 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.99282350200000002 2024-09-18T08:16:53.836 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:16:53.836 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:16:53.836 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "waiting for peered", 2024-09-18T08:16:53.837 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:14:05.571651+0000", 2024-09-18T08:16:53.837 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 7.2250000000000003e-06 2024-09-18T08:16:53.837 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:16:53.837 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:16:53.837 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-09-18T08:16:53.837 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:14:06.811970+0000", 2024-09-18T08:16:53.837 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.240318786 2024-09-18T08:16:53.837 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:16:53.837 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:16:53.837 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "done", 2024-09-18T08:16:53.837 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:14:06.812072+0000", 2024-09-18T08:16:53.837 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.000101884 2024-09-18T08:16:53.837 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-09-18T08:16:53.838 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-09-18T08:16:53.838 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-09-18T08:16:53.838 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:16:53.838 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:16:53.838 INFO:teuthology.orchestra.run.smithi007.stdout: "description": "osd_op(client.15582.0:252483 9.3 9:cd866060:::benchmark_data_smithi007_124019_object110598:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e292)", 2024-09-18T08:16:53.838 INFO:teuthology.orchestra.run.smithi007.stdout: "initiated_at": "2024-09-18T08:14:04.578571+0000", 2024-09-18T08:16:53.838 INFO:teuthology.orchestra.run.smithi007.stdout: "age": 169.14187165600001, 2024-09-18T08:16:53.838 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 2.2336163340000001, 2024-09-18T08:16:53.838 INFO:teuthology.orchestra.run.smithi007.stdout: "continuous": false, 2024-09-18T08:16:53.838 INFO:teuthology.orchestra.run.smithi007.stdout: "type_data": { 2024-09-18T08:16:53.838 INFO:teuthology.orchestra.run.smithi007.stdout: "flag_point": "reached pg", 2024-09-18T08:16:53.838 INFO:teuthology.orchestra.run.smithi007.stdout: "client_info": { 2024-09-18T08:16:53.839 INFO:teuthology.orchestra.run.smithi007.stdout: "client": "client.15582", 2024-09-18T08:16:53.839 INFO:teuthology.orchestra.run.smithi007.stdout: "client_addr": "172.21.15.7:0/3598605839", 2024-09-18T08:16:53.839 INFO:teuthology.orchestra.run.smithi007.stdout: "tid": 252483 2024-09-18T08:16:53.839 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:16:53.839 INFO:teuthology.orchestra.run.smithi007.stdout: "events": [ 2024-09-18T08:16:53.839 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:16:53.839 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "initiated", 2024-09-18T08:16:53.839 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:14:04.578571+0000", 2024-09-18T08:16:53.839 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-09-18T08:16:53.839 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:16:53.839 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:16:53.839 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "header_read", 2024-09-18T08:16:53.840 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:14:04.578571+0000", 2024-09-18T08:16:53.840 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-09-18T08:16:53.840 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:16:53.840 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:16:53.840 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "throttled", 2024-09-18T08:16:53.840 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:14:04.578571+0000", 2024-09-18T08:16:53.840 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 3.0600000000000001e-07 2024-09-18T08:16:53.840 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:16:53.840 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:16:53.841 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "all_read", 2024-09-18T08:16:53.841 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:14:04.578573+0000", 2024-09-18T08:16:53.841 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 2.0279999999999999e-06 2024-09-18T08:16:53.841 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:16:53.841 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:16:53.841 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "dispatched", 2024-09-18T08:16:53.841 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:14:04.578578+0000", 2024-09-18T08:16:53.841 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 4.6070000000000001e-06 2024-09-18T08:16:53.841 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:16:53.841 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:16:53.841 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "queued_for_pg", 2024-09-18T08:16:53.841 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:14:04.578582+0000", 2024-09-18T08:16:53.841 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 4.0770000000000001e-06 2024-09-18T08:16:53.842 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:16:53.842 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:16:53.842 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-09-18T08:16:53.842 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:14:04.578846+0000", 2024-09-18T08:16:53.842 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.00026379700000000002 2024-09-18T08:16:53.842 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:16:53.842 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:16:53.842 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "waiting for peered", 2024-09-18T08:16:53.842 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:14:04.578864+0000", 2024-09-18T08:16:53.842 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.8329e-05 2024-09-18T08:16:53.842 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:16:53.843 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:16:53.843 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-09-18T08:16:53.843 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:14:05.571675+0000", 2024-09-18T08:16:53.843 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.99281106600000002 2024-09-18T08:16:53.843 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:16:53.843 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:16:53.843 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "waiting for peered", 2024-09-18T08:16:53.843 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:14:05.571681+0000", 2024-09-18T08:16:53.843 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 6.4169999999999997e-06 2024-09-18T08:16:53.843 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:16:53.843 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:16:53.843 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-09-18T08:16:53.843 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:14:06.812113+0000", 2024-09-18T08:16:53.843 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.240431778 2024-09-18T08:16:53.844 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:16:53.844 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:16:53.844 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "done", 2024-09-18T08:16:53.844 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:14:06.812187+0000", 2024-09-18T08:16:53.844 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 7.3929000000000003e-05 2024-09-18T08:16:53.844 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-09-18T08:16:53.844 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-09-18T08:16:53.844 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-09-18T08:16:53.844 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:16:53.844 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:16:53.844 INFO:teuthology.orchestra.run.smithi007.stdout: "description": "osd_op(client.15582.0:252485 9.3 9:cf447bb3:::benchmark_data_smithi007_124019_object110600:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e292)", 2024-09-18T08:16:53.844 INFO:teuthology.orchestra.run.smithi007.stdout: "initiated_at": "2024-09-18T08:14:04.578588+0000", 2024-09-18T08:16:53.844 INFO:teuthology.orchestra.run.smithi007.stdout: "age": 169.14185457799999, 2024-09-18T08:16:53.845 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 2.2469260480000002, 2024-09-18T08:16:53.845 INFO:teuthology.orchestra.run.smithi007.stdout: "continuous": false, 2024-09-18T08:16:53.845 INFO:teuthology.orchestra.run.smithi007.stdout: "type_data": { 2024-09-18T08:16:53.845 INFO:teuthology.orchestra.run.smithi007.stdout: "flag_point": "reached pg", 2024-09-18T08:16:53.845 INFO:teuthology.orchestra.run.smithi007.stdout: "client_info": { 2024-09-18T08:16:53.845 INFO:teuthology.orchestra.run.smithi007.stdout: "client": "client.15582", 2024-09-18T08:16:53.845 INFO:teuthology.orchestra.run.smithi007.stdout: "client_addr": "172.21.15.7:0/3598605839", 2024-09-18T08:16:53.845 INFO:teuthology.orchestra.run.smithi007.stdout: "tid": 252485 2024-09-18T08:16:53.845 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:16:53.845 INFO:teuthology.orchestra.run.smithi007.stdout: "events": [ 2024-09-18T08:16:53.845 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:16:53.845 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "initiated", 2024-09-18T08:16:53.845 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:14:04.578588+0000", 2024-09-18T08:16:53.846 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-09-18T08:16:53.846 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:16:53.846 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:16:53.846 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "header_read", 2024-09-18T08:16:53.846 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:14:04.578588+0000", 2024-09-18T08:16:53.846 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-09-18T08:16:53.846 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:16:53.846 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:16:53.846 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "throttled", 2024-09-18T08:16:53.846 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:14:04.578588+0000", 2024-09-18T08:16:53.847 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 3.5999999999999999e-07 2024-09-18T08:16:53.847 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:16:53.847 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:16:53.847 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "all_read", 2024-09-18T08:16:53.847 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:14:04.578590+0000", 2024-09-18T08:16:53.847 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 2.1600000000000001e-06 2024-09-18T08:16:53.847 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:16:53.847 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:16:53.847 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "dispatched", 2024-09-18T08:16:53.847 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:14:04.578596+0000", 2024-09-18T08:16:53.847 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 5.48e-06 2024-09-18T08:16:53.847 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:16:53.847 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:16:53.848 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "queued_for_pg", 2024-09-18T08:16:53.848 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:14:04.578601+0000", 2024-09-18T08:16:53.848 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 5.1599999999999997e-06 2024-09-18T08:16:53.848 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:16:53.848 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:16:53.848 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-09-18T08:16:53.848 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:14:04.578909+0000", 2024-09-18T08:16:53.848 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.00030810599999999999 2024-09-18T08:16:53.848 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:16:53.848 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:16:53.848 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "waiting for peered", 2024-09-18T08:16:53.849 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:14:04.578915+0000", 2024-09-18T08:16:53.849 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 5.9190000000000004e-06 2024-09-18T08:16:53.849 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:16:53.849 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:16:53.849 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-09-18T08:16:53.849 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:14:05.571706+0000", 2024-09-18T08:16:53.849 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.99279081800000002 2024-09-18T08:16:53.849 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:16:53.849 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:16:53.849 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "waiting for peered", 2024-09-18T08:16:53.849 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:14:05.571712+0000", 2024-09-18T08:16:53.849 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 6.1770000000000003e-06 2024-09-18T08:16:53.849 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:16:53.850 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:16:53.850 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-09-18T08:16:53.850 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:14:06.812226+0000", 2024-09-18T08:16:53.850 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.2405141289999999 2024-09-18T08:16:53.850 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:16:53.850 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:16:53.850 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "waiting for missing object", 2024-09-18T08:16:53.850 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:14:06.812335+0000", 2024-09-18T08:16:53.850 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.00010895600000000001 2024-09-18T08:16:53.850 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:16:53.850 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:16:53.850 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-09-18T08:16:53.850 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:14:06.825411+0000", 2024-09-18T08:16:53.851 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.013075429 2024-09-18T08:16:53.851 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:16:53.851 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:16:53.851 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "done", 2024-09-18T08:16:53.851 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:14:06.825514+0000", 2024-09-18T08:16:53.851 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.000103354 2024-09-18T08:16:53.851 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-09-18T08:16:53.851 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-09-18T08:16:53.851 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-09-18T08:16:53.852 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:16:53.852 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:16:53.852 INFO:teuthology.orchestra.run.smithi007.stdout: "description": "osd_op(client.15582.0:252497 9.3 9:cca43518:::benchmark_data_smithi007_124019_object110612:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e292)", 2024-09-18T08:16:53.852 INFO:teuthology.orchestra.run.smithi007.stdout: "initiated_at": "2024-09-18T08:14:04.579093+0000", 2024-09-18T08:16:53.852 INFO:teuthology.orchestra.run.smithi007.stdout: "age": 169.14134981699999, 2024-09-18T08:16:53.852 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 2.2514827070000001, 2024-09-18T08:16:53.852 INFO:teuthology.orchestra.run.smithi007.stdout: "continuous": false, 2024-09-18T08:16:53.852 INFO:teuthology.orchestra.run.smithi007.stdout: "type_data": { 2024-09-18T08:16:53.852 INFO:teuthology.orchestra.run.smithi007.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T08:16:53.852 INFO:teuthology.orchestra.run.smithi007.stdout: "client_info": { 2024-09-18T08:16:53.852 INFO:teuthology.orchestra.run.smithi007.stdout: "client": "client.15582", 2024-09-18T08:16:53.852 INFO:teuthology.orchestra.run.smithi007.stdout: "client_addr": "172.21.15.7:0/3598605839", 2024-09-18T08:16:53.853 INFO:teuthology.orchestra.run.smithi007.stdout: "tid": 252497 2024-09-18T08:16:53.853 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:16:53.853 INFO:teuthology.orchestra.run.smithi007.stdout: "events": [ 2024-09-18T08:16:53.853 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:16:53.853 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "initiated", 2024-09-18T08:16:53.853 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:14:04.579093+0000", 2024-09-18T08:16:53.853 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-09-18T08:16:53.853 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:16:53.853 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:16:53.853 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "header_read", 2024-09-18T08:16:53.854 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:14:04.579093+0000", 2024-09-18T08:16:53.854 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-09-18T08:16:53.854 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:16:53.854 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:16:53.854 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "throttled", 2024-09-18T08:16:53.854 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:14:04.579094+0000", 2024-09-18T08:16:53.854 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 8.5600000000000004e-07 2024-09-18T08:16:53.854 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:16:53.854 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:16:53.854 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "all_read", 2024-09-18T08:16:53.854 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:14:04.579098+0000", 2024-09-18T08:16:53.854 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 4.143e-06 2024-09-18T08:16:53.854 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:16:53.854 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:16:53.855 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "dispatched", 2024-09-18T08:16:53.855 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:14:04.579105+0000", 2024-09-18T08:16:53.855 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 7.4309999999999997e-06 2024-09-18T08:16:53.855 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:16:53.855 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:16:53.855 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "queued_for_pg", 2024-09-18T08:16:53.855 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:14:04.579114+0000", 2024-09-18T08:16:53.855 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 8.6119999999999999e-06 2024-09-18T08:16:53.855 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:16:53.855 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:16:53.855 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-09-18T08:16:53.855 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:14:04.579160+0000", 2024-09-18T08:16:53.855 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 4.5977999999999999e-05 2024-09-18T08:16:53.856 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:16:53.856 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:16:53.856 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "waiting for peered", 2024-09-18T08:16:53.856 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:14:04.579167+0000", 2024-09-18T08:16:53.856 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 7.5000000000000002e-06 2024-09-18T08:16:53.856 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:16:53.856 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:16:53.856 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-09-18T08:16:53.856 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:14:05.571734+0000", 2024-09-18T08:16:53.856 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.99256691500000005 2024-09-18T08:16:53.856 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:16:53.856 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:16:53.856 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "waiting for peered", 2024-09-18T08:16:53.856 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:14:05.571740+0000", 2024-09-18T08:16:53.857 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 5.9410000000000004e-06 2024-09-18T08:16:53.857 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:16:53.857 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:16:53.857 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-09-18T08:16:53.857 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:14:06.814366+0000", 2024-09-18T08:16:53.857 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.242625654 2024-09-18T08:16:53.857 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:16:53.857 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:16:53.857 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "started", 2024-09-18T08:16:53.857 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:14:06.814471+0000", 2024-09-18T08:16:53.857 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.00010566 2024-09-18T08:16:53.857 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:16:53.857 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:16:53.858 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "waiting for subops from 5,7", 2024-09-18T08:16:53.858 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:14:06.814842+0000", 2024-09-18T08:16:53.858 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.00037091799999999997 2024-09-18T08:16:53.858 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:16:53.858 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:16:53.858 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "op_commit", 2024-09-18T08:16:53.858 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:14:06.825034+0000", 2024-09-18T08:16:53.858 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.010192084000000001 2024-09-18T08:16:53.858 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:16:53.858 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:16:53.858 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:16:53.858 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:14:06.828000+0000", 2024-09-18T08:16:53.858 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.0029653050000000001 2024-09-18T08:16:53.858 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:16:53.859 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:16:53.859 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:16:53.859 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:14:06.830366+0000", 2024-09-18T08:16:53.859 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.0023659689999999999 2024-09-18T08:16:53.859 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:16:53.859 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:16:53.859 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "commit_sent", 2024-09-18T08:16:53.859 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:14:06.830439+0000", 2024-09-18T08:16:53.859 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 7.3083000000000005e-05 2024-09-18T08:16:53.859 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:16:53.859 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:16:53.859 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "done", 2024-09-18T08:16:53.859 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:14:06.830575+0000", 2024-09-18T08:16:53.859 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.00013665800000000001 2024-09-18T08:16:53.860 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-09-18T08:16:53.860 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-09-18T08:16:53.860 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-09-18T08:16:53.860 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:16:53.860 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:16:53.860 INFO:teuthology.orchestra.run.smithi007.stdout: "description": "osd_op(client.15582.0:252500 9.c 9:3c726e3d:::benchmark_data_smithi007_124019_object110615:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e292)", 2024-09-18T08:16:53.860 INFO:teuthology.orchestra.run.smithi007.stdout: "initiated_at": "2024-09-18T08:14:04.592965+0000", 2024-09-18T08:16:53.860 INFO:teuthology.orchestra.run.smithi007.stdout: "age": 169.12747776399999, 2024-09-18T08:16:53.860 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 2.4642305090000001, 2024-09-18T08:16:53.860 INFO:teuthology.orchestra.run.smithi007.stdout: "continuous": false, 2024-09-18T08:16:53.860 INFO:teuthology.orchestra.run.smithi007.stdout: "type_data": { 2024-09-18T08:16:53.860 INFO:teuthology.orchestra.run.smithi007.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T08:16:53.861 INFO:teuthology.orchestra.run.smithi007.stdout: "client_info": { 2024-09-18T08:16:53.861 INFO:teuthology.orchestra.run.smithi007.stdout: "client": "client.15582", 2024-09-18T08:16:53.861 INFO:teuthology.orchestra.run.smithi007.stdout: "client_addr": "172.21.15.7:0/3598605839", 2024-09-18T08:16:53.861 INFO:teuthology.orchestra.run.smithi007.stdout: "tid": 252500 2024-09-18T08:16:53.861 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:16:53.861 INFO:teuthology.orchestra.run.smithi007.stdout: "events": [ 2024-09-18T08:16:53.861 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:16:53.861 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "initiated", 2024-09-18T08:16:53.862 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:14:04.592965+0000", 2024-09-18T08:16:53.862 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-09-18T08:16:53.862 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:16:53.862 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:16:53.862 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "header_read", 2024-09-18T08:16:53.862 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:14:04.592965+0000", 2024-09-18T08:16:53.862 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-09-18T08:16:53.862 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:16:53.862 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:16:53.862 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "throttled", 2024-09-18T08:16:53.862 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:14:04.592966+0000", 2024-09-18T08:16:53.862 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.6899999999999999e-06 2024-09-18T08:16:53.862 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:16:53.863 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:16:53.863 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "all_read", 2024-09-18T08:16:53.863 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:14:04.592973+0000", 2024-09-18T08:16:53.863 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 6.3600000000000001e-06 2024-09-18T08:16:53.863 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:16:53.863 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:16:53.863 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "dispatched", 2024-09-18T08:16:53.863 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:14:04.592985+0000", 2024-09-18T08:16:53.863 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.2500999999999999e-05 2024-09-18T08:16:53.863 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:16:53.864 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:16:53.864 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "queued_for_pg", 2024-09-18T08:16:53.864 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:14:04.592995+0000", 2024-09-18T08:16:53.864 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 9.5580000000000006e-06 2024-09-18T08:16:53.864 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:16:53.864 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:16:53.864 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-09-18T08:16:53.864 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:14:04.593104+0000", 2024-09-18T08:16:53.864 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.000108935 2024-09-18T08:16:53.864 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:16:53.864 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:16:53.864 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "waiting for peered", 2024-09-18T08:16:53.864 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:14:04.593116+0000", 2024-09-18T08:16:53.865 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.1891e-05 2024-09-18T08:16:53.865 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:16:53.865 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:16:53.865 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-09-18T08:16:53.865 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:14:05.582200+0000", 2024-09-18T08:16:53.865 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.98908412899999998 2024-09-18T08:16:53.865 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:16:53.865 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:16:53.865 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "waiting for peered", 2024-09-18T08:16:53.866 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:14:05.582209+0000", 2024-09-18T08:16:53.866 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 8.8710000000000003e-06 2024-09-18T08:16:53.866 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:16:53.866 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:16:53.866 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-09-18T08:16:53.866 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:14:06.804708+0000", 2024-09-18T08:16:53.866 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.2224989939999999 2024-09-18T08:16:53.866 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:16:53.866 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:16:53.866 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "started", 2024-09-18T08:16:53.866 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:14:06.804843+0000", 2024-09-18T08:16:53.866 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.00013566300000000001 2024-09-18T08:16:53.866 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:16:53.867 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:16:53.867 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "waiting for subops from 0,4,7", 2024-09-18T08:16:53.867 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:14:06.804991+0000", 2024-09-18T08:16:53.867 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.00014795800000000001 2024-09-18T08:16:53.867 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:16:53.867 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:16:53.867 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:16:53.867 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:14:06.808569+0000", 2024-09-18T08:16:53.867 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.0035780719999999998 2024-09-18T08:16:53.867 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:16:53.867 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:16:53.867 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:16:53.867 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:14:06.808595+0000", 2024-09-18T08:16:53.867 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 2.5677000000000002e-05 2024-09-18T08:16:53.868 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:16:53.868 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:16:53.868 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "op_commit", 2024-09-18T08:16:53.868 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:14:06.808602+0000", 2024-09-18T08:16:53.868 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 6.9430000000000003e-06 2024-09-18T08:16:53.868 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:16:53.868 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:16:53.868 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:16:53.868 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:14:07.057024+0000", 2024-09-18T08:16:53.868 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.24842249599999999 2024-09-18T08:16:53.869 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:16:53.869 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:16:53.869 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "commit_sent", 2024-09-18T08:16:53.869 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:14:07.057085+0000", 2024-09-18T08:16:53.869 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 6.0189999999999998e-05 2024-09-18T08:16:53.869 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:16:53.869 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:16:53.869 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "done", 2024-09-18T08:16:53.869 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:14:07.057195+0000", 2024-09-18T08:16:53.869 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.000110581 2024-09-18T08:16:53.869 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-09-18T08:16:53.869 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-09-18T08:16:53.869 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-09-18T08:16:53.870 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:16:53.870 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:16:53.870 INFO:teuthology.orchestra.run.smithi007.stdout: "description": "osd_op(client.15582.0:252501 9.3 9:c1421981:::benchmark_data_smithi007_124019_object110616:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e292)", 2024-09-18T08:16:53.870 INFO:teuthology.orchestra.run.smithi007.stdout: "initiated_at": "2024-09-18T08:14:04.593015+0000", 2024-09-18T08:16:53.870 INFO:teuthology.orchestra.run.smithi007.stdout: "age": 169.12742752299999, 2024-09-18T08:16:53.870 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 2.2377744819999998, 2024-09-18T08:16:53.870 INFO:teuthology.orchestra.run.smithi007.stdout: "continuous": false, 2024-09-18T08:16:53.870 INFO:teuthology.orchestra.run.smithi007.stdout: "type_data": { 2024-09-18T08:16:53.870 INFO:teuthology.orchestra.run.smithi007.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T08:16:53.870 INFO:teuthology.orchestra.run.smithi007.stdout: "client_info": { 2024-09-18T08:16:53.870 INFO:teuthology.orchestra.run.smithi007.stdout: "client": "client.15582", 2024-09-18T08:16:53.870 INFO:teuthology.orchestra.run.smithi007.stdout: "client_addr": "172.21.15.7:0/3598605839", 2024-09-18T08:16:53.870 INFO:teuthology.orchestra.run.smithi007.stdout: "tid": 252501 2024-09-18T08:16:53.870 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:16:53.871 INFO:teuthology.orchestra.run.smithi007.stdout: "events": [ 2024-09-18T08:16:53.871 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:16:53.871 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "initiated", 2024-09-18T08:16:53.871 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:14:04.593015+0000", 2024-09-18T08:16:53.871 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-09-18T08:16:53.871 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:16:53.871 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:16:53.871 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "header_read", 2024-09-18T08:16:53.871 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:14:04.593015+0000", 2024-09-18T08:16:53.871 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-09-18T08:16:53.871 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:16:53.871 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:16:53.871 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "throttled", 2024-09-18T08:16:53.871 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:14:04.593015+0000", 2024-09-18T08:16:53.872 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 5.2099999999999997e-07 2024-09-18T08:16:53.872 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:16:53.872 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:16:53.872 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "all_read", 2024-09-18T08:16:53.872 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:14:04.593018+0000", 2024-09-18T08:16:53.872 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 2.8820000000000001e-06 2024-09-18T08:16:53.872 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:16:53.872 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:16:53.872 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "dispatched", 2024-09-18T08:16:53.873 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:14:04.593025+0000", 2024-09-18T08:16:53.873 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 6.3589999999999996e-06 2024-09-18T08:16:53.873 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:16:53.873 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:16:53.873 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "queued_for_pg", 2024-09-18T08:16:53.873 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:14:04.593031+0000", 2024-09-18T08:16:53.873 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 5.8030000000000002e-06 2024-09-18T08:16:53.873 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:16:53.873 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:16:53.873 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-09-18T08:16:53.873 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:14:04.593087+0000", 2024-09-18T08:16:53.874 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 5.6628000000000001e-05 2024-09-18T08:16:53.874 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:16:53.874 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:16:53.874 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "waiting for peered", 2024-09-18T08:16:53.874 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:14:04.593097+0000", 2024-09-18T08:16:53.874 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 9.8209999999999997e-06 2024-09-18T08:16:53.874 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:16:53.874 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:16:53.874 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-09-18T08:16:53.874 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:14:05.571762+0000", 2024-09-18T08:16:53.874 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.97866520700000004 2024-09-18T08:16:53.874 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:16:53.874 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:16:53.874 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "waiting for peered", 2024-09-18T08:16:53.875 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:14:05.571768+0000", 2024-09-18T08:16:53.875 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 5.9710000000000001e-06 2024-09-18T08:16:53.875 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:16:53.875 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:16:53.875 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-09-18T08:16:53.875 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:14:06.820762+0000", 2024-09-18T08:16:53.875 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.248993587 2024-09-18T08:16:53.875 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:16:53.875 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:16:53.875 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "started", 2024-09-18T08:16:53.875 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:14:06.820911+0000", 2024-09-18T08:16:53.876 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.00014937800000000001 2024-09-18T08:16:53.876 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:16:53.877 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:16:53.877 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "waiting for subops from 5,7", 2024-09-18T08:16:53.877 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:14:06.821051+0000", 2024-09-18T08:16:53.877 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.00013987200000000001 2024-09-18T08:16:53.877 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:16:53.877 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:16:53.877 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "op_commit", 2024-09-18T08:16:53.877 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:14:06.825043+0000", 2024-09-18T08:16:53.877 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.0039923010000000002 2024-09-18T08:16:53.877 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:16:53.877 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:16:53.878 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:16:53.878 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:14:06.828045+0000", 2024-09-18T08:16:53.878 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.0030014629999999998 2024-09-18T08:16:53.878 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:16:53.878 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:16:53.878 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:16:53.878 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:14:06.830636+0000", 2024-09-18T08:16:53.878 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.0025915980000000001 2024-09-18T08:16:53.878 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:16:53.878 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:16:53.878 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "commit_sent", 2024-09-18T08:16:53.878 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:14:06.830687+0000", 2024-09-18T08:16:53.878 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 5.0912000000000003e-05 2024-09-18T08:16:53.879 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:16:53.879 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:16:53.879 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "done", 2024-09-18T08:16:53.879 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:14:06.830789+0000", 2024-09-18T08:16:53.879 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.000102179 2024-09-18T08:16:53.879 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-09-18T08:16:53.879 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-09-18T08:16:53.879 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-09-18T08:16:53.879 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:16:53.879 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:16:53.879 INFO:teuthology.orchestra.run.smithi007.stdout: "description": "osd_op(client.15966.0:9472 10.7 10:ebde9e8a:::benchmark_data_smithi007_148834_object9471:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e315)", 2024-09-18T08:16:53.879 INFO:teuthology.orchestra.run.smithi007.stdout: "initiated_at": "2024-09-18T08:14:51.634759+0000", 2024-09-18T08:16:53.879 INFO:teuthology.orchestra.run.smithi007.stdout: "age": 122.085683855, 2024-09-18T08:16:53.880 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 2.0688344779999999, 2024-09-18T08:16:53.880 INFO:teuthology.orchestra.run.smithi007.stdout: "continuous": false, 2024-09-18T08:16:53.880 INFO:teuthology.orchestra.run.smithi007.stdout: "type_data": { 2024-09-18T08:16:53.880 INFO:teuthology.orchestra.run.smithi007.stdout: "flag_point": "reached pg", 2024-09-18T08:16:53.880 INFO:teuthology.orchestra.run.smithi007.stdout: "client_info": { 2024-09-18T08:16:53.880 INFO:teuthology.orchestra.run.smithi007.stdout: "client": "client.15966", 2024-09-18T08:16:53.880 INFO:teuthology.orchestra.run.smithi007.stdout: "client_addr": "172.21.15.7:0/529784850", 2024-09-18T08:16:53.880 INFO:teuthology.orchestra.run.smithi007.stdout: "tid": 9472 2024-09-18T08:16:53.880 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:16:53.881 INFO:teuthology.orchestra.run.smithi007.stdout: "events": [ 2024-09-18T08:16:53.881 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:16:53.881 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "initiated", 2024-09-18T08:16:53.881 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:14:51.634759+0000", 2024-09-18T08:16:53.881 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-09-18T08:16:53.881 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:16:53.881 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:16:53.881 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "header_read", 2024-09-18T08:16:53.881 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:14:51.634759+0000", 2024-09-18T08:16:53.881 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-09-18T08:16:53.881 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:16:53.881 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:16:53.881 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "throttled", 2024-09-18T08:16:53.881 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:14:51.634760+0000", 2024-09-18T08:16:53.882 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.4959999999999999e-06 2024-09-18T08:16:53.882 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:16:53.882 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:16:53.882 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "all_read", 2024-09-18T08:16:53.882 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:14:51.634843+0000", 2024-09-18T08:16:53.882 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 8.3253999999999999e-05 2024-09-18T08:16:53.882 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:16:53.882 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:16:53.882 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "dispatched", 2024-09-18T08:16:53.883 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:14:51.634861+0000", 2024-09-18T08:16:53.883 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.7368999999999999e-05 2024-09-18T08:16:53.883 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:16:53.883 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:16:53.883 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "queued_for_pg", 2024-09-18T08:16:53.883 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:14:51.634869+0000", 2024-09-18T08:16:53.883 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 8.2470000000000008e-06 2024-09-18T08:16:53.883 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:16:53.883 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:16:53.883 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-09-18T08:16:53.883 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:14:51.635928+0000", 2024-09-18T08:16:53.883 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.001059321 2024-09-18T08:16:53.883 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:16:53.884 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:16:53.884 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "started", 2024-09-18T08:16:53.884 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:14:51.636070+0000", 2024-09-18T08:16:53.884 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.000141526 2024-09-18T08:16:53.884 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:16:53.884 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:16:53.884 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "waiting for subops from 5,6", 2024-09-18T08:16:53.884 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:14:51.636200+0000", 2024-09-18T08:16:53.884 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.00013038599999999999 2024-09-18T08:16:53.884 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:16:53.884 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:16:53.884 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "op_commit", 2024-09-18T08:16:53.884 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:14:51.638724+0000", 2024-09-18T08:16:53.885 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.0025236899999999999 2024-09-18T08:16:53.885 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:16:53.885 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:16:53.885 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-09-18T08:16:53.885 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:14:51.652898+0000", 2024-09-18T08:16:53.885 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.014173715999999999 2024-09-18T08:16:53.885 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:16:53.885 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:16:53.885 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "waiting for peered", 2024-09-18T08:16:53.885 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:14:51.652911+0000", 2024-09-18T08:16:53.885 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.3540000000000001e-05 2024-09-18T08:16:53.885 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:16:53.885 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:16:53.886 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-09-18T08:16:53.886 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:14:52.657538+0000", 2024-09-18T08:16:53.886 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.004626684 2024-09-18T08:16:53.886 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:16:53.886 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:16:53.886 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "waiting for peered", 2024-09-18T08:16:53.886 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:14:52.657552+0000", 2024-09-18T08:16:53.886 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.3854e-05 2024-09-18T08:16:53.886 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:16:53.886 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:16:53.886 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-09-18T08:16:53.886 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:14:53.703400+0000", 2024-09-18T08:16:53.886 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.0458485280000001 2024-09-18T08:16:53.887 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:16:53.887 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:16:53.887 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "done", 2024-09-18T08:16:53.887 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:14:53.703593+0000", 2024-09-18T08:16:53.887 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.000192867 2024-09-18T08:16:53.887 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-09-18T08:16:53.887 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-09-18T08:16:53.887 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-09-18T08:16:53.887 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:16:53.887 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:16:53.887 INFO:teuthology.orchestra.run.smithi007.stdout: "description": "osd_op(client.15966.0:9474 10.7 10:efdd08be:::benchmark_data_smithi007_148834_object9473:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e315)", 2024-09-18T08:16:53.887 INFO:teuthology.orchestra.run.smithi007.stdout: "initiated_at": "2024-09-18T08:14:51.634886+0000", 2024-09-18T08:16:53.887 INFO:teuthology.orchestra.run.smithi007.stdout: "age": 122.085556146, 2024-09-18T08:16:53.887 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 2.0695741889999999, 2024-09-18T08:16:53.888 INFO:teuthology.orchestra.run.smithi007.stdout: "continuous": false, 2024-09-18T08:16:53.888 INFO:teuthology.orchestra.run.smithi007.stdout: "type_data": { 2024-09-18T08:16:53.888 INFO:teuthology.orchestra.run.smithi007.stdout: "flag_point": "reached pg", 2024-09-18T08:16:53.888 INFO:teuthology.orchestra.run.smithi007.stdout: "client_info": { 2024-09-18T08:16:53.888 INFO:teuthology.orchestra.run.smithi007.stdout: "client": "client.15966", 2024-09-18T08:16:53.888 INFO:teuthology.orchestra.run.smithi007.stdout: "client_addr": "172.21.15.7:0/529784850", 2024-09-18T08:16:53.888 INFO:teuthology.orchestra.run.smithi007.stdout: "tid": 9474 2024-09-18T08:16:53.888 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:16:53.888 INFO:teuthology.orchestra.run.smithi007.stdout: "events": [ 2024-09-18T08:16:53.888 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:16:53.888 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "initiated", 2024-09-18T08:16:53.889 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:14:51.634886+0000", 2024-09-18T08:16:53.889 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-09-18T08:16:53.889 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:16:53.889 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:16:53.889 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "header_read", 2024-09-18T08:16:53.889 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:14:51.634886+0000", 2024-09-18T08:16:53.889 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-09-18T08:16:53.889 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:16:53.889 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:16:53.889 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "throttled", 2024-09-18T08:16:53.889 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:14:51.634887+0000", 2024-09-18T08:16:53.889 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 6.4600000000000004e-07 2024-09-18T08:16:53.889 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:16:53.890 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:16:53.890 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "all_read", 2024-09-18T08:16:53.890 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:14:51.634960+0000", 2024-09-18T08:16:53.890 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 7.3404000000000004e-05 2024-09-18T08:16:53.890 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:16:53.890 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:16:53.890 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "dispatched", 2024-09-18T08:16:53.890 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:14:51.634983+0000", 2024-09-18T08:16:53.890 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 2.2880000000000001e-05 2024-09-18T08:16:53.890 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:16:53.890 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:16:53.890 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "queued_for_pg", 2024-09-18T08:16:53.890 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:14:51.634990+0000", 2024-09-18T08:16:53.891 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 7.0859999999999996e-06 2024-09-18T08:16:53.891 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:16:53.891 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:16:53.891 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-09-18T08:16:53.891 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:14:51.637114+0000", 2024-09-18T08:16:53.891 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.0021231639999999999 2024-09-18T08:16:53.891 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:16:53.891 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:16:53.891 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "started", 2024-09-18T08:16:53.891 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:14:51.637239+0000", 2024-09-18T08:16:53.891 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.000125959 2024-09-18T08:16:53.891 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:16:53.891 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:16:53.891 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "waiting for subops from 5,6", 2024-09-18T08:16:53.892 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:14:51.637369+0000", 2024-09-18T08:16:53.892 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.00012985899999999999 2024-09-18T08:16:53.892 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:16:53.892 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:16:53.892 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "op_commit", 2024-09-18T08:16:53.892 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:14:51.639867+0000", 2024-09-18T08:16:53.892 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.0024973999999999999 2024-09-18T08:16:53.892 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:16:53.892 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:16:53.892 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-09-18T08:16:53.893 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:14:51.652947+0000", 2024-09-18T08:16:53.893 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.013080253999999999 2024-09-18T08:16:53.893 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:16:53.893 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:16:53.893 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "waiting for peered", 2024-09-18T08:16:53.893 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:14:51.652956+0000", 2024-09-18T08:16:53.893 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 9.1269999999999992e-06 2024-09-18T08:16:53.893 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:16:53.893 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:16:53.894 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-09-18T08:16:53.894 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:14:52.657686+0000", 2024-09-18T08:16:53.894 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.0047295039999999 2024-09-18T08:16:53.894 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:16:53.894 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:16:53.894 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "waiting for peered", 2024-09-18T08:16:53.894 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:14:52.657696+0000", 2024-09-18T08:16:53.894 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.0710000000000001e-05 2024-09-18T08:16:53.894 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:16:53.895 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:16:53.895 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-09-18T08:16:53.895 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:14:53.704057+0000", 2024-09-18T08:16:53.895 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.046360814 2024-09-18T08:16:53.895 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:16:53.895 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:16:53.895 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "done", 2024-09-18T08:16:53.895 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:14:53.704461+0000", 2024-09-18T08:16:53.895 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.00040338199999999998 2024-09-18T08:16:53.895 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-09-18T08:16:53.895 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-09-18T08:16:53.895 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-09-18T08:16:53.895 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:16:53.895 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:16:53.896 INFO:teuthology.orchestra.run.smithi007.stdout: "description": "osd_op(client.15966.0:9477 10.7 10:e773b142:::benchmark_data_smithi007_148834_object9476:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e315)", 2024-09-18T08:16:53.896 INFO:teuthology.orchestra.run.smithi007.stdout: "initiated_at": "2024-09-18T08:14:51.635151+0000", 2024-09-18T08:16:53.896 INFO:teuthology.orchestra.run.smithi007.stdout: "age": 122.085291864, 2024-09-18T08:16:53.896 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 2.0696481750000002, 2024-09-18T08:16:53.896 INFO:teuthology.orchestra.run.smithi007.stdout: "continuous": false, 2024-09-18T08:16:53.896 INFO:teuthology.orchestra.run.smithi007.stdout: "type_data": { 2024-09-18T08:16:53.896 INFO:teuthology.orchestra.run.smithi007.stdout: "flag_point": "reached pg", 2024-09-18T08:16:53.896 INFO:teuthology.orchestra.run.smithi007.stdout: "client_info": { 2024-09-18T08:16:53.896 INFO:teuthology.orchestra.run.smithi007.stdout: "client": "client.15966", 2024-09-18T08:16:53.896 INFO:teuthology.orchestra.run.smithi007.stdout: "client_addr": "172.21.15.7:0/529784850", 2024-09-18T08:16:53.896 INFO:teuthology.orchestra.run.smithi007.stdout: "tid": 9477 2024-09-18T08:16:53.897 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:16:53.897 INFO:teuthology.orchestra.run.smithi007.stdout: "events": [ 2024-09-18T08:16:53.897 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:16:53.897 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "initiated", 2024-09-18T08:16:53.897 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:14:51.635151+0000", 2024-09-18T08:16:53.897 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-09-18T08:16:53.897 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:16:53.897 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:16:53.897 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "header_read", 2024-09-18T08:16:53.897 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:14:51.635151+0000", 2024-09-18T08:16:53.898 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-09-18T08:16:53.898 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:16:53.898 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:16:53.898 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "throttled", 2024-09-18T08:16:53.898 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:14:51.635152+0000", 2024-09-18T08:16:53.898 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 9.4600000000000003e-07 2024-09-18T08:16:53.898 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:16:53.898 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:16:53.898 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "all_read", 2024-09-18T08:16:53.898 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:14:51.635174+0000", 2024-09-18T08:16:53.898 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 2.2035000000000002e-05 2024-09-18T08:16:53.898 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:16:53.898 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:16:53.898 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "dispatched", 2024-09-18T08:16:53.899 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:14:51.635187+0000", 2024-09-18T08:16:53.899 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.378e-05 2024-09-18T08:16:53.899 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:16:53.899 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:16:53.899 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "queued_for_pg", 2024-09-18T08:16:53.899 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:14:51.635195+0000", 2024-09-18T08:16:53.899 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 7.661e-06 2024-09-18T08:16:53.899 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:16:53.899 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:16:53.900 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-09-18T08:16:53.900 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:14:51.637873+0000", 2024-09-18T08:16:53.900 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.002677539 2024-09-18T08:16:53.900 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:16:53.900 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:16:53.900 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "started", 2024-09-18T08:16:53.900 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:14:51.637961+0000", 2024-09-18T08:16:53.900 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 8.8047000000000006e-05 2024-09-18T08:16:53.900 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:16:53.900 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:16:53.900 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "waiting for subops from 5,6", 2024-09-18T08:16:53.900 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:14:51.638041+0000", 2024-09-18T08:16:53.901 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 7.9957999999999996e-05 2024-09-18T08:16:53.901 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:16:53.901 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:16:53.901 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "op_commit", 2024-09-18T08:16:53.901 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:14:51.640125+0000", 2024-09-18T08:16:53.901 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.00208421 2024-09-18T08:16:53.901 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:16:53.901 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:16:53.901 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-09-18T08:16:53.901 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:14:51.652990+0000", 2024-09-18T08:16:53.902 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.01286488 2024-09-18T08:16:53.902 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:16:53.902 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:16:53.902 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "waiting for peered", 2024-09-18T08:16:53.902 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:14:51.652999+0000", 2024-09-18T08:16:53.902 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 9.1810000000000007e-06 2024-09-18T08:16:53.902 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:16:53.902 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:16:53.902 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-09-18T08:16:53.903 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:14:52.657733+0000", 2024-09-18T08:16:53.903 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.0047345940000001 2024-09-18T08:16:53.903 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:16:53.903 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:16:53.903 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "waiting for peered", 2024-09-18T08:16:53.903 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:14:52.657743+0000", 2024-09-18T08:16:53.903 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 9.0599999999999997e-06 2024-09-18T08:16:53.903 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:16:53.903 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:16:53.903 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-09-18T08:16:53.903 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:14:53.704501+0000", 2024-09-18T08:16:53.903 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.0467583199999999 2024-09-18T08:16:53.903 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:16:53.904 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:16:53.904 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "done", 2024-09-18T08:16:53.904 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:14:53.704799+0000", 2024-09-18T08:16:53.904 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.00029796400000000001 2024-09-18T08:16:53.904 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-09-18T08:16:53.904 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-09-18T08:16:53.904 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-09-18T08:16:53.904 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-09-18T08:16:53.904 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-09-18T08:16:53.904 INFO:teuthology.orchestra.run.smithi007.stdout:} 2024-09-18T08:16:53.905 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:16:53 smithi050 ceph-mon[32371]: osdmap e363: 8 total, 8 up, 6 in 2024-09-18T08:16:53.909 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:16:53.910 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:16:53.910 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3' 2024-09-18T08:16:54.011 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0 2024-09-18T08:16:54.045 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:16:54.046 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:16:54.047 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0' 2024-09-18T08:16:54.147 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5 2024-09-18T08:16:54.170 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:16:54.171 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:16:54.171 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5' 2024-09-18T08:16:54.272 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7 2024-09-18T08:16:54.295 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:16:54.295 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:16:54.296 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7' 2024-09-18T08:16:54.397 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2 2024-09-18T08:16:54.430 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:16:54.431 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:16:54.431 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2' 2024-09-18T08:16:54.462 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid 779349ca-7593-11ef-bceb-c7b262605968 -- ceph osd set noscrub 2024-09-18T08:16:54.532 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1 2024-09-18T08:16:54.581 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:16:54.582 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:16:54.582 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1' 2024-09-18T08:16:54.683 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4 2024-09-18T08:16:54.708 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:16:54.709 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:16:54.709 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4' 2024-09-18T08:16:54.810 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4 2024-09-18T08:16:54.837 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:16:54.837 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:16:54.838 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4' 2024-09-18T08:16:54.905 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:16:54 smithi050 ceph-mon[32371]: pgmap v701: 158 pgs: 2 active+remapped+backfill_wait+backfill_toofull, 5 active+remapped+backfilling, 3 active+remapped+backfill_toofull, 148 active+clean; 4.7 GiB data, 16 GiB used, 521 GiB / 536 GiB avail; 1.7 KiB/s rd, 0 B/s wr, 804 op/s; 27879/232791 objects misplaced (11.976%); 38 MiB/s, 613 objects/s recovering 2024-09-18T08:16:54.905 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:16:54 smithi050 ceph-mon[32371]: osdmap e364: 8 total, 8 up, 6 in 2024-09-18T08:16:54.905 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:16:54 smithi050 ceph-mon[32371]: purged_snaps scrub starts 2024-09-18T08:16:54.905 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:16:54 smithi050 ceph-mon[32371]: purged_snaps scrub ok 2024-09-18T08:16:54.939 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2 2024-09-18T08:16:54.980 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:16:54 smithi007 ceph-mon[29459]: pgmap v701: 158 pgs: 2 active+remapped+backfill_wait+backfill_toofull, 5 active+remapped+backfilling, 3 active+remapped+backfill_toofull, 148 active+clean; 4.7 GiB data, 16 GiB used, 521 GiB / 536 GiB avail; 1.7 KiB/s rd, 0 B/s wr, 804 op/s; 27879/232791 objects misplaced (11.976%); 38 MiB/s, 613 objects/s recovering 2024-09-18T08:16:54.980 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:16:54 smithi007 ceph-mon[29459]: osdmap e364: 8 total, 8 up, 6 in 2024-09-18T08:16:54.980 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:16:54 smithi007 ceph-mon[29459]: purged_snaps scrub starts 2024-09-18T08:16:54.980 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:16:54 smithi007 ceph-mon[29459]: purged_snaps scrub ok 2024-09-18T08:16:54.981 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:16:54 smithi007 ceph-mon[39249]: pgmap v701: 158 pgs: 2 active+remapped+backfill_wait+backfill_toofull, 5 active+remapped+backfilling, 3 active+remapped+backfill_toofull, 148 active+clean; 4.7 GiB data, 16 GiB used, 521 GiB / 536 GiB avail; 1.7 KiB/s rd, 0 B/s wr, 804 op/s; 27879/232791 objects misplaced (11.976%); 38 MiB/s, 613 objects/s recovering 2024-09-18T08:16:54.981 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:16:54 smithi007 ceph-mon[39249]: osdmap e364: 8 total, 8 up, 6 in 2024-09-18T08:16:54.981 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:16:54 smithi007 ceph-mon[39249]: purged_snaps scrub starts 2024-09-18T08:16:54.981 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:16:54 smithi007 ceph-mon[39249]: purged_snaps scrub ok 2024-09-18T08:16:54.982 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:16:54.982 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:16:54.982 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2' 2024-09-18T08:16:55.083 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4 2024-09-18T08:16:55.087 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid 779349ca-7593-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-18T08:16:55.108 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:16:55.109 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:16:55.110 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4' 2024-09-18T08:16:55.210 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7 2024-09-18T08:16:55.235 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:16:55.236 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:16:55.236 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7' 2024-09-18T08:16:55.296 DEBUG:teuthology.orchestra.run.smithi050:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid 779349ca-7593-11ef-bceb-c7b262605968 -- ceph daemon osd.4 dump_ops_in_flight 2024-09-18T08:16:55.337 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0 2024-09-18T08:16:55.421 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:16:55.421 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:16:55.422 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0' 2024-09-18T08:16:55.522 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1 2024-09-18T08:16:55.569 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:16:55.571 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:16:55.571 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1' 2024-09-18T08:16:55.610 INFO:teuthology.orchestra.run.smithi050.stderr:Inferring config /var/lib/ceph/779349ca-7593-11ef-bceb-c7b262605968/mon.b/config 2024-09-18T08:16:55.655 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:16:55 smithi050 ceph-mon[32371]: osdmap e365: 8 total, 8 up, 6 in 2024-09-18T08:16:55.672 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5 2024-09-18T08:16:55.700 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:16:55.701 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:16:55.701 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5' 2024-09-18T08:16:55.802 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2 2024-09-18T08:16:55.839 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:16:55.840 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:16:55.840 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2' 2024-09-18T08:16:55.941 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3 2024-09-18T08:16:55.980 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:16:55 smithi007 ceph-mon[29459]: osdmap e365: 8 total, 8 up, 6 in 2024-09-18T08:16:55.980 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:16:55 smithi007 ceph-mon[39249]: osdmap e365: 8 total, 8 up, 6 in 2024-09-18T08:16:55.981 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:16:55.982 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:16:55.982 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3' 2024-09-18T08:16:56.083 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0 2024-09-18T08:16:56.124 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:16:56.125 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:16:56.126 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0' 2024-09-18T08:16:56.159 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2024-09-18T08:16:56.159 INFO:teuthology.orchestra.run.smithi050.stdout: "ops": [], 2024-09-18T08:16:56.159 INFO:teuthology.orchestra.run.smithi050.stdout: "num_ops": 0 2024-09-18T08:16:56.159 INFO:teuthology.orchestra.run.smithi050.stdout:} 2024-09-18T08:16:56.226 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5 2024-09-18T08:16:56.255 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:16:56.256 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:16:56.256 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5' 2024-09-18T08:16:56.357 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4 2024-09-18T08:16:56.383 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:16:56.384 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:16:56.384 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4' 2024-09-18T08:16:56.485 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1 2024-09-18T08:16:56.527 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:16:56.528 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:16:56.528 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1' 2024-09-18T08:16:56.629 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6 2024-09-18T08:16:56.655 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:16:56.655 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:16:56.656 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6' 2024-09-18T08:16:56.756 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6 2024-09-18T08:16:56.782 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:16:56.782 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:16:56.783 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6' 2024-09-18T08:16:56.884 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6 2024-09-18T08:16:56.905 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:16:56 smithi050 ceph-mon[32371]: pgmap v704: 158 pgs: 5 active+remapped+backfilling, 2 active+remapped+backfill_toofull, 151 active+clean; 4.6 GiB data, 16 GiB used, 521 GiB / 536 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 641 op/s; 21691/226917 objects misplaced (9.559%); 97 MiB/s, 1.55k objects/s recovering 2024-09-18T08:16:56.905 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:16:56 smithi050 ceph-mon[32371]: Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 2 pgs backfill_toofull (PG_BACKFILL_FULL) 2024-09-18T08:16:56.906 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:16:56.907 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:16:56.907 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6' 2024-09-18T08:16:56.977 DEBUG:teuthology.orchestra.run.smithi050:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid 779349ca-7593-11ef-bceb-c7b262605968 -- ceph daemon osd.4 dump_blocked_ops 2024-09-18T08:16:56.980 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:16:56 smithi007 ceph-mon[29459]: pgmap v704: 158 pgs: 5 active+remapped+backfilling, 2 active+remapped+backfill_toofull, 151 active+clean; 4.6 GiB data, 16 GiB used, 521 GiB / 536 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 641 op/s; 21691/226917 objects misplaced (9.559%); 97 MiB/s, 1.55k objects/s recovering 2024-09-18T08:16:56.980 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:16:56 smithi007 ceph-mon[29459]: Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 2 pgs backfill_toofull (PG_BACKFILL_FULL) 2024-09-18T08:16:56.980 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:16:56 smithi007 ceph-mon[39249]: pgmap v704: 158 pgs: 5 active+remapped+backfilling, 2 active+remapped+backfill_toofull, 151 active+clean; 4.6 GiB data, 16 GiB used, 521 GiB / 536 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 641 op/s; 21691/226917 objects misplaced (9.559%); 97 MiB/s, 1.55k objects/s recovering 2024-09-18T08:16:56.980 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:16:56 smithi007 ceph-mon[39249]: Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 2 pgs backfill_toofull (PG_BACKFILL_FULL) 2024-09-18T08:16:56.980 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Sep 18 08:16:56 smithi007 ceph-779349ca-7593-11ef-bceb-c7b262605968-mgr-y[29688]: ::ffff:172.21.15.50 - - [18/Sep/2024:08:16:56] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-09-18T08:16:57.008 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2 2024-09-18T08:16:57.043 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:16:57.044 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:16:57.044 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2' 2024-09-18T08:16:57.145 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5 2024-09-18T08:16:57.167 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:16:57.167 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:16:57.168 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5' 2024-09-18T08:16:57.249 INFO:teuthology.orchestra.run.smithi050.stderr:Inferring config /var/lib/ceph/779349ca-7593-11ef-bceb-c7b262605968/mon.b/config 2024-09-18T08:16:57.269 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5 2024-09-18T08:16:57.292 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:16:57.293 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:16:57.294 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5' 2024-09-18T08:16:57.395 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1 2024-09-18T08:16:57.420 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:16:57.420 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:16:57.421 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1' 2024-09-18T08:16:57.522 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5 2024-09-18T08:16:57.547 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:16:57.547 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:16:57.547 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5' 2024-09-18T08:16:57.648 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6 2024-09-18T08:16:57.655 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:16:57 smithi050 ceph-mon[32371]: osdmap e366: 8 total, 8 up, 6 in 2024-09-18T08:16:57.655 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:16:57 smithi050 ceph-mon[32371]: from='client.? 172.21.15.7:0/3528506086' entity='client.admin' cmd={"prefix": "osd reweight", "id": 2, "weight": 1.0} : dispatch 2024-09-18T08:16:57.655 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:16:57 smithi050 ceph-mon[32371]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 2, "weight": 1.0} : dispatch 2024-09-18T08:16:57.677 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:16:57.677 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:16:57.678 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6' 2024-09-18T08:16:57.779 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4 2024-09-18T08:16:57.804 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:16:57.805 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:16:57.805 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4' 2024-09-18T08:16:57.812 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2024-09-18T08:16:57.813 INFO:teuthology.orchestra.run.smithi050.stdout: "ops": [], 2024-09-18T08:16:57.813 INFO:teuthology.orchestra.run.smithi050.stdout: "complaint_time": 30, 2024-09-18T08:16:57.813 INFO:teuthology.orchestra.run.smithi050.stdout: "num_blocked_ops": 0 2024-09-18T08:16:57.813 INFO:teuthology.orchestra.run.smithi050.stdout:} 2024-09-18T08:16:57.891 INFO:teuthology.orchestra.run.smithi007.stderr:reweighted osd.2 to 1 (10000) 2024-09-18T08:16:57.892 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:16:57 smithi007 ceph-mon[29459]: osdmap e366: 8 total, 8 up, 6 in 2024-09-18T08:16:57.892 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:16:57 smithi007 ceph-mon[29459]: from='client.? 172.21.15.7:0/3528506086' entity='client.admin' cmd={"prefix": "osd reweight", "id": 2, "weight": 1.0} : dispatch 2024-09-18T08:16:57.892 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:16:57 smithi007 ceph-mon[29459]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 2, "weight": 1.0} : dispatch 2024-09-18T08:16:57.892 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:16:57 smithi007 ceph-mon[39249]: osdmap e366: 8 total, 8 up, 6 in 2024-09-18T08:16:57.892 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:16:57 smithi007 ceph-mon[39249]: from='client.? 172.21.15.7:0/3528506086' entity='client.admin' cmd={"prefix": "osd reweight", "id": 2, "weight": 1.0} : dispatch 2024-09-18T08:16:57.893 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:16:57 smithi007 ceph-mon[39249]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 2, "weight": 1.0} : dispatch 2024-09-18T08:16:57.906 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5 2024-09-18T08:16:57.933 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:16:57.934 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:16:57.934 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5' 2024-09-18T08:16:58.035 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1 2024-09-18T08:16:58.073 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:16:58.074 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:16:58.074 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1' 2024-09-18T08:16:58.175 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2 2024-09-18T08:16:58.198 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:16:58.198 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:16:58.198 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2' 2024-09-18T08:16:58.299 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1 2024-09-18T08:16:58.342 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:16:58.342 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:16:58.343 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1' 2024-09-18T08:16:58.443 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0 2024-09-18T08:16:58.480 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:16:58.481 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:16:58.481 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0' 2024-09-18T08:16:58.511 DEBUG:teuthology.orchestra.run.smithi050:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid 779349ca-7593-11ef-bceb-c7b262605968 -- ceph daemon osd.4 dump_historic_ops 2024-09-18T08:16:58.582 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2 2024-09-18T08:16:58.627 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:16:58.628 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:16:58.628 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2' 2024-09-18T08:16:58.729 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0 2024-09-18T08:16:58.770 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:16:58.771 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:16:58.771 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0' 2024-09-18T08:16:58.824 INFO:teuthology.orchestra.run.smithi050.stderr:Inferring config /var/lib/ceph/779349ca-7593-11ef-bceb-c7b262605968/mon.b/config 2024-09-18T08:16:58.872 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7 2024-09-18T08:16:58.902 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:16:58.903 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:16:58.903 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7' 2024-09-18T08:16:58.905 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:16:58 smithi050 ceph-mon[32371]: purged_snaps scrub starts 2024-09-18T08:16:58.905 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:16:58 smithi050 ceph-mon[32371]: purged_snaps scrub ok 2024-09-18T08:16:58.905 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:16:58 smithi050 ceph-mon[32371]: pgmap v706: 158 pgs: 5 active+remapped+backfilling, 2 active+remapped+backfill_toofull, 151 active+clean; 4.6 GiB data, 16 GiB used, 521 GiB / 536 GiB avail; 0 B/s wr, 400 op/s; 21691/226917 objects misplaced (9.559%); 71 MiB/s, 1.14k objects/s recovering 2024-09-18T08:16:58.905 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:16:58 smithi050 ceph-mon[32371]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]': finished 2024-09-18T08:16:58.905 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:16:58 smithi050 ceph-mon[32371]: osdmap e367: 8 total, 8 up, 6 in 2024-09-18T08:16:58.980 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:16:58 smithi007 ceph-mon[29459]: purged_snaps scrub starts 2024-09-18T08:16:58.980 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:16:58 smithi007 ceph-mon[29459]: purged_snaps scrub ok 2024-09-18T08:16:58.980 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:16:58 smithi007 ceph-mon[29459]: pgmap v706: 158 pgs: 5 active+remapped+backfilling, 2 active+remapped+backfill_toofull, 151 active+clean; 4.6 GiB data, 16 GiB used, 521 GiB / 536 GiB avail; 0 B/s wr, 400 op/s; 21691/226917 objects misplaced (9.559%); 71 MiB/s, 1.14k objects/s recovering 2024-09-18T08:16:58.980 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:16:58 smithi007 ceph-mon[29459]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]': finished 2024-09-18T08:16:58.980 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:16:58 smithi007 ceph-mon[29459]: osdmap e367: 8 total, 8 up, 6 in 2024-09-18T08:16:58.981 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:16:58 smithi007 ceph-mon[39249]: purged_snaps scrub starts 2024-09-18T08:16:58.981 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:16:58 smithi007 ceph-mon[39249]: purged_snaps scrub ok 2024-09-18T08:16:58.981 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:16:58 smithi007 ceph-mon[39249]: pgmap v706: 158 pgs: 5 active+remapped+backfilling, 2 active+remapped+backfill_toofull, 151 active+clean; 4.6 GiB data, 16 GiB used, 521 GiB / 536 GiB avail; 0 B/s wr, 400 op/s; 21691/226917 objects misplaced (9.559%); 71 MiB/s, 1.14k objects/s recovering 2024-09-18T08:16:58.981 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:16:58 smithi007 ceph-mon[39249]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]': finished 2024-09-18T08:16:58.981 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:16:58 smithi007 ceph-mon[39249]: osdmap e367: 8 total, 8 up, 6 in 2024-09-18T08:16:59.004 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5 2024-09-18T08:16:59.028 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:16:59.029 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:16:59.029 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5' 2024-09-18T08:16:59.130 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5 2024-09-18T08:16:59.157 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:16:59.157 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:16:59.157 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5' 2024-09-18T08:16:59.258 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3 2024-09-18T08:16:59.284 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:16:59.285 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:16:59.285 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3' 2024-09-18T08:16:59.386 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1 2024-09-18T08:16:59.391 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2024-09-18T08:16:59.391 INFO:teuthology.orchestra.run.smithi050.stdout: "size": 20, 2024-09-18T08:16:59.391 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 600, 2024-09-18T08:16:59.391 INFO:teuthology.orchestra.run.smithi050.stdout: "ops": [ 2024-09-18T08:16:59.391 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:16:59.391 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.15216.0:86225 8.5 8:ae8df458:::benchmark_data_smithi007_101583_object86224:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e205)", 2024-09-18T08:16:59.391 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-09-18T08:09:12.178973+0000", 2024-09-18T08:16:59.392 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 467.20955180999999, 2024-09-18T08:16:59.392 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.99102768600000002, 2024-09-18T08:16:59.392 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-09-18T08:16:59.392 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-09-18T08:16:59.392 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "reached pg", 2024-09-18T08:16:59.392 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2024-09-18T08:16:59.392 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.15216", 2024-09-18T08:16:59.392 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "172.21.15.7:0/2596041177", 2024-09-18T08:16:59.392 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 86225 2024-09-18T08:16:59.392 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:16:59.392 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-09-18T08:16:59.392 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:16:59.392 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-09-18T08:16:59.393 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:12.178973+0000", 2024-09-18T08:16:59.393 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-18T08:16:59.393 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:16:59.393 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:16:59.393 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-09-18T08:16:59.393 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:12.178973+0000", 2024-09-18T08:16:59.393 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-18T08:16:59.393 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:16:59.393 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:16:59.393 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-09-18T08:16:59.393 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:12.178975+0000", 2024-09-18T08:16:59.393 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.3969999999999999e-06 2024-09-18T08:16:59.393 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:16:59.394 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:16:59.394 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-09-18T08:16:59.394 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:12.179022+0000", 2024-09-18T08:16:59.394 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4.7580000000000002e-05 2024-09-18T08:16:59.394 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:16:59.394 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:16:59.394 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-09-18T08:16:59.394 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:12.179034+0000", 2024-09-18T08:16:59.394 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.1477e-05 2024-09-18T08:16:59.394 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:16:59.394 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:16:59.394 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-09-18T08:16:59.394 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:12.179040+0000", 2024-09-18T08:16:59.394 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 5.7529999999999998e-06 2024-09-18T08:16:59.395 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:16:59.395 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:16:59.395 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-18T08:16:59.395 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:12.179092+0000", 2024-09-18T08:16:59.395 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 5.2840000000000002e-05 2024-09-18T08:16:59.395 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:16:59.395 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:16:59.395 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "waiting for peered", 2024-09-18T08:16:59.395 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:12.179102+0000", 2024-09-18T08:16:59.395 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 9.5210000000000008e-06 2024-09-18T08:16:59.395 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:16:59.395 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:16:59.395 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-18T08:16:59.395 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:13.169945+0000", 2024-09-18T08:16:59.396 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.99084323600000002 2024-09-18T08:16:59.396 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:16:59.396 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:16:59.396 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-09-18T08:16:59.396 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:13.170001+0000", 2024-09-18T08:16:59.396 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 5.5881999999999998e-05 2024-09-18T08:16:59.396 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-18T08:16:59.396 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-09-18T08:16:59.396 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-18T08:16:59.396 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:16:59.396 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:16:59.396 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.15216.0:86263 8.5 8:af593c7a:::benchmark_data_smithi007_101583_object86262:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e205)", 2024-09-18T08:16:59.396 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-09-18T08:09:12.204249+0000", 2024-09-18T08:16:59.397 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 467.18427580500003, 2024-09-18T08:16:59.397 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.98012944999999996, 2024-09-18T08:16:59.397 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-09-18T08:16:59.397 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-09-18T08:16:59.397 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T08:16:59.397 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2024-09-18T08:16:59.397 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.15216", 2024-09-18T08:16:59.397 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "172.21.15.7:0/2596041177", 2024-09-18T08:16:59.397 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 86263 2024-09-18T08:16:59.397 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:16:59.397 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-09-18T08:16:59.397 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:16:59.397 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-09-18T08:16:59.398 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:12.204249+0000", 2024-09-18T08:16:59.398 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-18T08:16:59.398 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:16:59.398 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:16:59.398 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-09-18T08:16:59.398 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:12.204249+0000", 2024-09-18T08:16:59.398 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-18T08:16:59.398 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:16:59.398 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:16:59.398 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-09-18T08:16:59.398 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:12.204251+0000", 2024-09-18T08:16:59.398 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.635e-06 2024-09-18T08:16:59.398 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:16:59.398 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:16:59.398 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-09-18T08:16:59.399 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:12.204299+0000", 2024-09-18T08:16:59.399 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4.8026999999999999e-05 2024-09-18T08:16:59.399 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:16:59.399 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:16:59.399 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-09-18T08:16:59.399 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:12.204313+0000", 2024-09-18T08:16:59.399 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.4066000000000001e-05 2024-09-18T08:16:59.399 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:16:59.399 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:16:59.399 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-09-18T08:16:59.399 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:12.204321+0000", 2024-09-18T08:16:59.399 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 8.1839999999999999e-06 2024-09-18T08:16:59.399 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:16:59.399 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:16:59.400 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-18T08:16:59.400 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:12.204392+0000", 2024-09-18T08:16:59.400 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 7.0340999999999996e-05 2024-09-18T08:16:59.400 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:16:59.400 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:16:59.400 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "waiting for peered", 2024-09-18T08:16:59.400 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:12.204403+0000", 2024-09-18T08:16:59.400 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.1654000000000001e-05 2024-09-18T08:16:59.400 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:16:59.400 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:16:59.400 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-18T08:16:59.400 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:13.170414+0000", 2024-09-18T08:16:59.400 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.96601117000000003 2024-09-18T08:16:59.400 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:16:59.401 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:16:59.401 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-09-18T08:16:59.401 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:13.170546+0000", 2024-09-18T08:16:59.401 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00013120100000000001 2024-09-18T08:16:59.401 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:16:59.401 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:16:59.401 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "waiting for subops from 6,7", 2024-09-18T08:16:59.401 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:13.170965+0000", 2024-09-18T08:16:59.401 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00041976200000000001 2024-09-18T08:16:59.401 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:16:59.401 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:16:59.401 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:16:59.401 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:13.181803+0000", 2024-09-18T08:16:59.402 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.010837167999999999 2024-09-18T08:16:59.402 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:16:59.402 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:16:59.402 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:16:59.402 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:13.181864+0000", 2024-09-18T08:16:59.402 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 6.1857000000000001e-05 2024-09-18T08:16:59.402 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:16:59.402 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:16:59.402 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "op_commit", 2024-09-18T08:16:59.402 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:13.184283+0000", 2024-09-18T08:16:59.402 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0024184340000000001 2024-09-18T08:16:59.402 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:16:59.402 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:16:59.402 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "commit_sent", 2024-09-18T08:16:59.403 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:13.184339+0000", 2024-09-18T08:16:59.403 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 5.6378999999999997e-05 2024-09-18T08:16:59.403 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:16:59.403 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:16:59.403 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-09-18T08:16:59.403 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:13.184379+0000", 2024-09-18T08:16:59.403 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 3.9572e-05 2024-09-18T08:16:59.403 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-18T08:16:59.403 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-09-18T08:16:59.403 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-18T08:16:59.403 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:16:59.403 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:16:59.403 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.15216.0:86267 8.5 8:a135397b:::benchmark_data_smithi007_101583_object86266:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e205)", 2024-09-18T08:16:59.404 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-09-18T08:09:12.207725+0000", 2024-09-18T08:16:59.404 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 467.18079975500001, 2024-09-18T08:16:59.404 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.97676783599999994, 2024-09-18T08:16:59.404 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-09-18T08:16:59.404 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-09-18T08:16:59.404 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T08:16:59.404 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2024-09-18T08:16:59.404 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.15216", 2024-09-18T08:16:59.404 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "172.21.15.7:0/2596041177", 2024-09-18T08:16:59.404 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 86267 2024-09-18T08:16:59.404 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:16:59.404 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-09-18T08:16:59.404 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:16:59.405 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-09-18T08:16:59.405 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:12.207725+0000", 2024-09-18T08:16:59.405 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-18T08:16:59.405 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:16:59.405 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:16:59.406 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-09-18T08:16:59.406 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:12.207725+0000", 2024-09-18T08:16:59.406 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-18T08:16:59.406 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:16:59.406 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:16:59.406 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-09-18T08:16:59.406 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:12.207727+0000", 2024-09-18T08:16:59.406 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.79e-06 2024-09-18T08:16:59.406 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:16:59.406 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:16:59.406 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-09-18T08:16:59.406 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:12.207765+0000", 2024-09-18T08:16:59.406 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 3.7864000000000002e-05 2024-09-18T08:16:59.407 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:16:59.407 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:16:59.407 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-09-18T08:16:59.407 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:12.207783+0000", 2024-09-18T08:16:59.407 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.7779999999999999e-05 2024-09-18T08:16:59.407 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:16:59.407 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:16:59.407 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-09-18T08:16:59.407 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:12.207791+0000", 2024-09-18T08:16:59.407 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 7.9729999999999997e-06 2024-09-18T08:16:59.407 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:16:59.407 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:16:59.407 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-18T08:16:59.408 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:12.207872+0000", 2024-09-18T08:16:59.408 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 8.1318999999999998e-05 2024-09-18T08:16:59.408 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:16:59.408 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:16:59.408 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "waiting for peered", 2024-09-18T08:16:59.408 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:12.207905+0000", 2024-09-18T08:16:59.408 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 3.2694000000000002e-05 2024-09-18T08:16:59.408 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:16:59.408 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:16:59.408 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-18T08:16:59.408 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:13.179332+0000", 2024-09-18T08:16:59.408 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.97142748800000001 2024-09-18T08:16:59.408 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:16:59.409 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:16:59.409 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-09-18T08:16:59.409 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:13.179470+0000", 2024-09-18T08:16:59.409 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00013747400000000001 2024-09-18T08:16:59.409 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:16:59.409 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:16:59.409 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "waiting for subops from 6,7", 2024-09-18T08:16:59.409 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:13.179634+0000", 2024-09-18T08:16:59.409 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00016417600000000001 2024-09-18T08:16:59.409 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:16:59.409 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:16:59.409 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:16:59.410 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:13.184267+0000", 2024-09-18T08:16:59.410 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0046332079999999998 2024-09-18T08:16:59.410 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:16:59.410 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:16:59.410 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "op_commit", 2024-09-18T08:16:59.410 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:13.184386+0000", 2024-09-18T08:16:59.410 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000118993 2024-09-18T08:16:59.410 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:16:59.410 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:16:59.410 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:16:59.410 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:13.184437+0000", 2024-09-18T08:16:59.410 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 5.0988e-05 2024-09-18T08:16:59.410 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:16:59.410 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:16:59.411 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "commit_sent", 2024-09-18T08:16:59.411 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:13.184469+0000", 2024-09-18T08:16:59.411 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 3.1603000000000002e-05 2024-09-18T08:16:59.411 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:16:59.411 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:16:59.411 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-09-18T08:16:59.411 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:13.184493+0000", 2024-09-18T08:16:59.411 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 2.4485999999999999e-05 2024-09-18T08:16:59.411 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-18T08:16:59.411 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-09-18T08:16:59.411 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-18T08:16:59.411 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:16:59.412 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:16:59.412 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.15216.0:86268 8.5 8:aa7096fb:::benchmark_data_smithi007_101583_object86267:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e205)", 2024-09-18T08:16:59.412 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-09-18T08:09:12.208946+0000", 2024-09-18T08:16:59.412 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 467.17957914800002, 2024-09-18T08:16:59.412 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.980827699, 2024-09-18T08:16:59.412 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-09-18T08:16:59.412 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-09-18T08:16:59.412 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T08:16:59.412 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2024-09-18T08:16:59.412 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.15216", 2024-09-18T08:16:59.412 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "172.21.15.7:0/2596041177", 2024-09-18T08:16:59.412 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 86268 2024-09-18T08:16:59.413 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:16:59.413 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-09-18T08:16:59.413 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:16:59.413 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-09-18T08:16:59.413 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:12.208946+0000", 2024-09-18T08:16:59.413 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-18T08:16:59.413 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:16:59.413 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:16:59.413 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-09-18T08:16:59.413 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:12.208946+0000", 2024-09-18T08:16:59.413 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-18T08:16:59.413 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:16:59.413 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:16:59.414 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-09-18T08:16:59.414 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:12.208947+0000", 2024-09-18T08:16:59.414 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.201e-06 2024-09-18T08:16:59.414 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:16:59.414 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:16:59.414 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-09-18T08:16:59.414 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:12.209036+0000", 2024-09-18T08:16:59.414 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 8.8865999999999999e-05 2024-09-18T08:16:59.414 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:16:59.414 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:16:59.414 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-09-18T08:16:59.414 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:12.209049+0000", 2024-09-18T08:16:59.415 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.2795e-05 2024-09-18T08:16:59.415 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:16:59.415 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:16:59.415 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-09-18T08:16:59.415 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:12.209056+0000", 2024-09-18T08:16:59.415 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 7.1160000000000001e-06 2024-09-18T08:16:59.415 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:16:59.415 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:16:59.415 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-18T08:16:59.415 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:12.209116+0000", 2024-09-18T08:16:59.415 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 5.9517e-05 2024-09-18T08:16:59.415 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:16:59.415 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:16:59.415 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "waiting for peered", 2024-09-18T08:16:59.416 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:12.209125+0000", 2024-09-18T08:16:59.416 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 9.6630000000000005e-06 2024-09-18T08:16:59.416 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:16:59.416 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:16:59.416 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-18T08:16:59.416 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:13.182004+0000", 2024-09-18T08:16:59.416 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.97287885799999996 2024-09-18T08:16:59.416 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:16:59.416 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:16:59.416 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-09-18T08:16:59.416 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:13.182091+0000", 2024-09-18T08:16:59.416 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 8.6907e-05 2024-09-18T08:16:59.417 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:16:59.417 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:16:59.417 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "waiting for subops from 6,7", 2024-09-18T08:16:59.417 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:13.182183+0000", 2024-09-18T08:16:59.417 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 9.2131000000000003e-05 2024-09-18T08:16:59.417 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:16:59.417 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:16:59.417 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "op_commit", 2024-09-18T08:16:59.417 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:13.187041+0000", 2024-09-18T08:16:59.417 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0048579210000000003 2024-09-18T08:16:59.417 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:16:59.417 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:16:59.417 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:16:59.418 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:13.189581+0000", 2024-09-18T08:16:59.418 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.002539707 2024-09-18T08:16:59.418 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:16:59.418 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:16:59.418 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:16:59.418 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:13.189646+0000", 2024-09-18T08:16:59.418 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 6.5699999999999998e-05 2024-09-18T08:16:59.418 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:16:59.418 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:16:59.419 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "commit_sent", 2024-09-18T08:16:59.419 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:13.189714+0000", 2024-09-18T08:16:59.419 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 6.7108999999999997e-05 2024-09-18T08:16:59.419 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:16:59.419 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:16:59.419 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-09-18T08:16:59.419 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:13.189774+0000", 2024-09-18T08:16:59.419 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 6.0207999999999997e-05 2024-09-18T08:16:59.419 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-18T08:16:59.419 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-09-18T08:16:59.419 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-18T08:16:59.419 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:16:59.419 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:16:59.420 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.15216.0:86303 8.5 8:aebc4459:::benchmark_data_smithi007_101583_object86302:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e205)", 2024-09-18T08:16:59.420 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-09-18T08:09:12.250052+0000", 2024-09-18T08:16:59.420 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 467.13847359200003, 2024-09-18T08:16:59.420 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.94992247699999999, 2024-09-18T08:16:59.420 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-09-18T08:16:59.420 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-09-18T08:16:59.420 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T08:16:59.420 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2024-09-18T08:16:59.420 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.15216", 2024-09-18T08:16:59.420 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "172.21.15.7:0/2596041177", 2024-09-18T08:16:59.420 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 86303 2024-09-18T08:16:59.420 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:16:59.421 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-09-18T08:16:59.421 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:16:59.421 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-09-18T08:16:59.421 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:12.250052+0000", 2024-09-18T08:16:59.421 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-18T08:16:59.421 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:16:59.421 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:16:59.421 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-09-18T08:16:59.421 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:12.250052+0000", 2024-09-18T08:16:59.421 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-18T08:16:59.421 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:16:59.422 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:16:59.422 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-09-18T08:16:59.422 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:12.250053+0000", 2024-09-18T08:16:59.422 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.6130000000000001e-06 2024-09-18T08:16:59.422 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:16:59.422 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:16:59.422 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-09-18T08:16:59.422 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:12.250090+0000", 2024-09-18T08:16:59.422 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 3.6307999999999999e-05 2024-09-18T08:16:59.422 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:16:59.422 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:16:59.422 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-09-18T08:16:59.422 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:12.250102+0000", 2024-09-18T08:16:59.422 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.2624e-05 2024-09-18T08:16:59.423 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:16:59.423 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:16:59.423 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-09-18T08:16:59.423 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:12.250109+0000", 2024-09-18T08:16:59.423 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 6.426e-06 2024-09-18T08:16:59.423 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:16:59.423 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:16:59.423 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-18T08:16:59.423 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:12.250173+0000", 2024-09-18T08:16:59.423 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 6.3966000000000004e-05 2024-09-18T08:16:59.423 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:16:59.423 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:16:59.424 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "waiting for peered", 2024-09-18T08:16:59.424 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:12.250184+0000", 2024-09-18T08:16:59.424 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.1401e-05 2024-09-18T08:16:59.424 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:16:59.424 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:16:59.424 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-18T08:16:59.424 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:13.184533+0000", 2024-09-18T08:16:59.424 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.93434866299999997 2024-09-18T08:16:59.424 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:16:59.424 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:16:59.424 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-09-18T08:16:59.424 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:13.184634+0000", 2024-09-18T08:16:59.424 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000101139 2024-09-18T08:16:59.424 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:16:59.425 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:16:59.425 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "waiting for subops from 6,7", 2024-09-18T08:16:59.425 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:13.184793+0000", 2024-09-18T08:16:59.425 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00015932000000000001 2024-09-18T08:16:59.425 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:16:59.425 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:16:59.425 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:16:59.425 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:13.194839+0000", 2024-09-18T08:16:59.425 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.010046144 2024-09-18T08:16:59.425 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:16:59.425 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:16:59.425 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:16:59.426 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:13.194928+0000", 2024-09-18T08:16:59.426 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 8.9037999999999997e-05 2024-09-18T08:16:59.426 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:16:59.426 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:16:59.426 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "op_commit", 2024-09-18T08:16:59.426 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:13.199865+0000", 2024-09-18T08:16:59.426 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0049365879999999996 2024-09-18T08:16:59.426 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:16:59.426 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:16:59.426 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "commit_sent", 2024-09-18T08:16:59.426 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:13.199924+0000", 2024-09-18T08:16:59.426 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 5.9383999999999999e-05 2024-09-18T08:16:59.426 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:16:59.427 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:16:59.427 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-09-18T08:16:59.427 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:13.199974+0000", 2024-09-18T08:16:59.427 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4.9863e-05 2024-09-18T08:16:59.427 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-18T08:16:59.427 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-09-18T08:16:59.427 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-18T08:16:59.427 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:16:59.427 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:16:59.427 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.15216.0:86344 8.5 8:ac157b17:::benchmark_data_smithi007_101583_object86343:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e205)", 2024-09-18T08:16:59.427 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-09-18T08:09:12.295743+0000", 2024-09-18T08:16:59.427 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 467.09278191200002, 2024-09-18T08:16:59.428 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.92374953400000004, 2024-09-18T08:16:59.428 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-09-18T08:16:59.428 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-09-18T08:16:59.428 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T08:16:59.428 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2024-09-18T08:16:59.428 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.15216", 2024-09-18T08:16:59.428 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "172.21.15.7:0/2596041177", 2024-09-18T08:16:59.428 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 86344 2024-09-18T08:16:59.428 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:16:59.428 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-09-18T08:16:59.428 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:16:59.428 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-09-18T08:16:59.428 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:12.295743+0000", 2024-09-18T08:16:59.429 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-18T08:16:59.429 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:16:59.429 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:16:59.429 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-09-18T08:16:59.429 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:12.295743+0000", 2024-09-18T08:16:59.429 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-18T08:16:59.429 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:16:59.429 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:16:59.429 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-09-18T08:16:59.429 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:12.295745+0000", 2024-09-18T08:16:59.429 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.57e-06 2024-09-18T08:16:59.429 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:16:59.429 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:16:59.430 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-09-18T08:16:59.430 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:12.295795+0000", 2024-09-18T08:16:59.430 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 5.0062999999999998e-05 2024-09-18T08:16:59.430 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:16:59.430 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:16:59.430 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-09-18T08:16:59.430 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:12.295807+0000", 2024-09-18T08:16:59.430 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.1578e-05 2024-09-18T08:16:59.430 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:16:59.430 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:16:59.430 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-09-18T08:16:59.430 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:12.295812+0000", 2024-09-18T08:16:59.430 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 5.8930000000000002e-06 2024-09-18T08:16:59.431 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:16:59.431 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:16:59.431 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-18T08:16:59.431 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:12.295875+0000", 2024-09-18T08:16:59.431 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 6.2168000000000005e-05 2024-09-18T08:16:59.431 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:16:59.431 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:16:59.431 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "waiting for peered", 2024-09-18T08:16:59.431 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:12.295883+0000", 2024-09-18T08:16:59.431 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 7.9969999999999998e-06 2024-09-18T08:16:59.431 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:16:59.431 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:16:59.431 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-18T08:16:59.431 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:13.187072+0000", 2024-09-18T08:16:59.432 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.89118984899999998 2024-09-18T08:16:59.432 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:16:59.432 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:16:59.432 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-09-18T08:16:59.432 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:13.187193+0000", 2024-09-18T08:16:59.432 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00012062599999999999 2024-09-18T08:16:59.432 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:16:59.432 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:16:59.432 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "waiting for subops from 6,7", 2024-09-18T08:16:59.432 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:13.187351+0000", 2024-09-18T08:16:59.432 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00015781500000000001 2024-09-18T08:16:59.432 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:16:59.432 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:16:59.433 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:16:59.433 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:13.194877+0000", 2024-09-18T08:16:59.433 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0075262139999999998 2024-09-18T08:16:59.433 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:16:59.433 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:16:59.433 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:16:59.433 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:13.195031+0000", 2024-09-18T08:16:59.433 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00015384799999999999 2024-09-18T08:16:59.433 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:16:59.433 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:16:59.433 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "op_commit", 2024-09-18T08:16:59.433 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:13.219386+0000", 2024-09-18T08:16:59.433 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.024355226000000001 2024-09-18T08:16:59.434 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:16:59.434 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:16:59.434 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "commit_sent", 2024-09-18T08:16:59.434 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:13.219440+0000", 2024-09-18T08:16:59.434 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 5.3786999999999998e-05 2024-09-18T08:16:59.434 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:16:59.434 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:16:59.434 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-09-18T08:16:59.434 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:13.219493+0000", 2024-09-18T08:16:59.434 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 5.2899999999999998e-05 2024-09-18T08:16:59.434 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-18T08:16:59.434 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-09-18T08:16:59.434 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-18T08:16:59.435 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:16:59.435 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:16:59.435 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.15216.0:86350 8.5 8:a14e8599:::benchmark_data_smithi007_101583_object86349:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e205)", 2024-09-18T08:16:59.435 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-09-18T08:09:12.297534+0000", 2024-09-18T08:16:59.435 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 467.090991428, 2024-09-18T08:16:59.435 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.922055505, 2024-09-18T08:16:59.435 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-09-18T08:16:59.435 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-09-18T08:16:59.435 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T08:16:59.435 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2024-09-18T08:16:59.435 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.15216", 2024-09-18T08:16:59.435 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "172.21.15.7:0/2596041177", 2024-09-18T08:16:59.435 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 86350 2024-09-18T08:16:59.436 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:16:59.436 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-09-18T08:16:59.436 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:16:59.436 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-09-18T08:16:59.436 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:12.297534+0000", 2024-09-18T08:16:59.436 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-18T08:16:59.436 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:16:59.436 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:16:59.436 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-09-18T08:16:59.436 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:12.297534+0000", 2024-09-18T08:16:59.436 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-18T08:16:59.436 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:16:59.436 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:16:59.437 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-09-18T08:16:59.437 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:12.297534+0000", 2024-09-18T08:16:59.437 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 7.0800000000000004e-07 2024-09-18T08:16:59.437 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:16:59.437 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:16:59.437 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-09-18T08:16:59.437 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:12.297606+0000", 2024-09-18T08:16:59.437 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 7.1075000000000006e-05 2024-09-18T08:16:59.437 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:16:59.437 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:16:59.437 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-09-18T08:16:59.437 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:12.297613+0000", 2024-09-18T08:16:59.438 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 7.6260000000000003e-06 2024-09-18T08:16:59.438 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:16:59.438 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:16:59.438 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-09-18T08:16:59.438 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:12.297618+0000", 2024-09-18T08:16:59.438 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4.8550000000000001e-06 2024-09-18T08:16:59.438 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:16:59.438 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:16:59.438 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-18T08:16:59.438 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:12.297660+0000", 2024-09-18T08:16:59.438 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4.1968000000000002e-05 2024-09-18T08:16:59.438 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:16:59.438 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:16:59.439 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "waiting for peered", 2024-09-18T08:16:59.439 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:12.297666+0000", 2024-09-18T08:16:59.439 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 5.806e-06 2024-09-18T08:16:59.439 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:16:59.439 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:16:59.439 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-18T08:16:59.439 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:13.189816+0000", 2024-09-18T08:16:59.439 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.89215055300000001 2024-09-18T08:16:59.439 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:16:59.439 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:16:59.439 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-09-18T08:16:59.439 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:13.190014+0000", 2024-09-18T08:16:59.439 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000197894 2024-09-18T08:16:59.440 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:16:59.440 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:16:59.440 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "waiting for subops from 6,7", 2024-09-18T08:16:59.440 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:13.190197+0000", 2024-09-18T08:16:59.440 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00018230399999999999 2024-09-18T08:16:59.440 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:16:59.440 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:16:59.440 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:16:59.440 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:13.194999+0000", 2024-09-18T08:16:59.440 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0048023019999999996 2024-09-18T08:16:59.440 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:16:59.440 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:16:59.440 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:16:59.441 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:13.197606+0000", 2024-09-18T08:16:59.441 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0026074890000000002 2024-09-18T08:16:59.441 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:16:59.441 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:16:59.441 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "op_commit", 2024-09-18T08:16:59.441 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:13.219502+0000", 2024-09-18T08:16:59.441 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.02189514 2024-09-18T08:16:59.441 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:16:59.441 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:16:59.441 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "commit_sent", 2024-09-18T08:16:59.441 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:13.219546+0000", 2024-09-18T08:16:59.441 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4.4574999999999999e-05 2024-09-18T08:16:59.442 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:16:59.442 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:16:59.442 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-09-18T08:16:59.442 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:13.219589+0000", 2024-09-18T08:16:59.442 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4.3210000000000001e-05 2024-09-18T08:16:59.442 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-18T08:16:59.442 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-09-18T08:16:59.442 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-18T08:16:59.442 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:16:59.442 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:16:59.442 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.15216.0:86366 8.5 8:a8c4536d:::benchmark_data_smithi007_101583_object86365:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e205)", 2024-09-18T08:16:59.442 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-09-18T08:09:12.318202+0000", 2024-09-18T08:16:59.442 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 467.07032275199998, 2024-09-18T08:16:59.442 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.90148185700000005, 2024-09-18T08:16:59.443 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-09-18T08:16:59.443 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-09-18T08:16:59.443 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T08:16:59.443 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2024-09-18T08:16:59.443 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.15216", 2024-09-18T08:16:59.443 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "172.21.15.7:0/2596041177", 2024-09-18T08:16:59.443 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 86366 2024-09-18T08:16:59.443 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:16:59.443 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-09-18T08:16:59.443 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:16:59.443 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-09-18T08:16:59.443 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:12.318202+0000", 2024-09-18T08:16:59.443 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-18T08:16:59.444 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:16:59.444 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:16:59.444 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-09-18T08:16:59.444 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:12.318202+0000", 2024-09-18T08:16:59.444 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-18T08:16:59.444 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:16:59.444 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:16:59.444 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-09-18T08:16:59.444 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:12.318205+0000", 2024-09-18T08:16:59.444 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 2.074e-06 2024-09-18T08:16:59.444 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:16:59.444 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:16:59.444 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-09-18T08:16:59.445 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:12.318242+0000", 2024-09-18T08:16:59.445 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 3.6996999999999998e-05 2024-09-18T08:16:59.445 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:16:59.445 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:16:59.445 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-09-18T08:16:59.445 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:12.318258+0000", 2024-09-18T08:16:59.445 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.6382e-05 2024-09-18T08:16:59.445 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:16:59.445 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:16:59.445 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-09-18T08:16:59.445 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:12.318266+0000", 2024-09-18T08:16:59.445 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 7.9820000000000008e-06 2024-09-18T08:16:59.445 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:16:59.446 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:16:59.446 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-18T08:16:59.446 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:12.318402+0000", 2024-09-18T08:16:59.446 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000136477 2024-09-18T08:16:59.446 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:16:59.446 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:16:59.446 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "waiting for peered", 2024-09-18T08:16:59.446 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:12.318414+0000", 2024-09-18T08:16:59.446 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.2096000000000001e-05 2024-09-18T08:16:59.446 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:16:59.446 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:16:59.446 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-18T08:16:59.446 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:13.192024+0000", 2024-09-18T08:16:59.447 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.87360952300000005 2024-09-18T08:16:59.447 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:16:59.447 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:16:59.447 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-09-18T08:16:59.447 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:13.192155+0000", 2024-09-18T08:16:59.447 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00013099499999999999 2024-09-18T08:16:59.447 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:16:59.447 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:16:59.447 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "waiting for subops from 6,7", 2024-09-18T08:16:59.447 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:13.192313+0000", 2024-09-18T08:16:59.447 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00015802500000000001 2024-09-18T08:16:59.447 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:16:59.447 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:16:59.447 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:16:59.448 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:13.195063+0000", 2024-09-18T08:16:59.448 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0027500279999999998 2024-09-18T08:16:59.448 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:16:59.448 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:16:59.448 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:16:59.448 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:13.200043+0000", 2024-09-18T08:16:59.448 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0049800929999999997 2024-09-18T08:16:59.448 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:16:59.448 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:16:59.448 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "op_commit", 2024-09-18T08:16:59.448 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:13.219598+0000", 2024-09-18T08:16:59.448 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.019554768 2024-09-18T08:16:59.448 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:16:59.449 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:16:59.449 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "commit_sent", 2024-09-18T08:16:59.449 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:13.219647+0000", 2024-09-18T08:16:59.449 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4.9327999999999999e-05 2024-09-18T08:16:59.449 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:16:59.449 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:16:59.449 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-09-18T08:16:59.449 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:13.219684+0000", 2024-09-18T08:16:59.449 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 3.7088999999999997e-05 2024-09-18T08:16:59.449 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-18T08:16:59.449 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-09-18T08:16:59.449 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-18T08:16:59.450 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:16:59.450 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:16:59.452 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:16:59.452 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.15216.0:86390 8.5 8:a8c7b3fd:::benchmark_data_smithi007_101583_object86389:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e205)", 2024-09-18T08:16:59.452 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-09-18T08:09:12.356640+0000", 2024-09-18T08:16:59.452 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 467.03188545099999, 2024-09-18T08:16:59.452 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.86366743700000004, 2024-09-18T08:16:59.452 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-09-18T08:16:59.452 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-09-18T08:16:59.452 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T08:16:59.452 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2024-09-18T08:16:59.453 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.15216", 2024-09-18T08:16:59.453 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "172.21.15.7:0/2596041177", 2024-09-18T08:16:59.453 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 86390 2024-09-18T08:16:59.453 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:16:59.453 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-09-18T08:16:59.453 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:16:59.453 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-09-18T08:16:59.453 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:12.356640+0000", 2024-09-18T08:16:59.453 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-18T08:16:59.453 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:16:59.453 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:16:59.453 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-09-18T08:16:59.453 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:12.356640+0000", 2024-09-18T08:16:59.453 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-18T08:16:59.454 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:16:59.454 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:16:59.454 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-09-18T08:16:59.454 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:12.356641+0000", 2024-09-18T08:16:59.454 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.5430000000000001e-06 2024-09-18T08:16:59.454 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:16:59.454 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:16:59.454 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-09-18T08:16:59.454 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:12.356684+0000", 2024-09-18T08:16:59.454 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4.2222999999999999e-05 2024-09-18T08:16:59.454 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:16:59.454 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:16:59.454 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-09-18T08:16:59.454 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:12.356696+0000", 2024-09-18T08:16:59.455 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.2663000000000001e-05 2024-09-18T08:16:59.455 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:16:59.455 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:16:59.455 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-09-18T08:16:59.455 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:12.356703+0000", 2024-09-18T08:16:59.455 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 6.6710000000000002e-06 2024-09-18T08:16:59.455 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:16:59.455 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:16:59.455 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-18T08:16:59.455 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:12.356759+0000", 2024-09-18T08:16:59.455 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 5.6382000000000003e-05 2024-09-18T08:16:59.455 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:16:59.455 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:16:59.455 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "waiting for peered", 2024-09-18T08:16:59.456 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:12.356767+0000", 2024-09-18T08:16:59.456 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 7.5190000000000003e-06 2024-09-18T08:16:59.456 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:16:59.456 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:16:59.456 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-18T08:16:59.456 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:13.195090+0000", 2024-09-18T08:16:59.456 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.83832288600000004 2024-09-18T08:16:59.456 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:16:59.456 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:16:59.456 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-09-18T08:16:59.456 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:13.195221+0000", 2024-09-18T08:16:59.456 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00013146699999999999 2024-09-18T08:16:59.456 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:16:59.457 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:16:59.457 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "waiting for subops from 6,7", 2024-09-18T08:16:59.457 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:13.195370+0000", 2024-09-18T08:16:59.457 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000148598 2024-09-18T08:16:59.457 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:16:59.457 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:16:59.457 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:16:59.457 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:13.199845+0000", 2024-09-18T08:16:59.457 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0044749999999999998 2024-09-18T08:16:59.457 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:16:59.457 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:16:59.457 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:16:59.458 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:13.207294+0000", 2024-09-18T08:16:59.458 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0074489309999999998 2024-09-18T08:16:59.458 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:16:59.458 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:16:59.458 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "op_commit", 2024-09-18T08:16:59.458 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:13.220218+0000", 2024-09-18T08:16:59.458 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.012924715 2024-09-18T08:16:59.458 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:16:59.458 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:16:59.458 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "commit_sent", 2024-09-18T08:16:59.458 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:13.220272+0000", 2024-09-18T08:16:59.458 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 5.3748000000000001e-05 2024-09-18T08:16:59.458 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:16:59.458 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:16:59.459 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-09-18T08:16:59.459 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:13.220307+0000", 2024-09-18T08:16:59.459 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 3.5091e-05 2024-09-18T08:16:59.459 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-18T08:16:59.459 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-09-18T08:16:59.459 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-18T08:16:59.459 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:16:59.459 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:16:59.459 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.15216.0:86394 8.5 8:a810cd9d:::benchmark_data_smithi007_101583_object86393:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e205)", 2024-09-18T08:16:59.459 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-09-18T08:09:12.360967+0000", 2024-09-18T08:16:59.459 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 467.02755787400002, 2024-09-18T08:16:59.459 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.85950780100000002, 2024-09-18T08:16:59.459 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-09-18T08:16:59.460 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-09-18T08:16:59.460 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T08:16:59.460 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2024-09-18T08:16:59.460 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.15216", 2024-09-18T08:16:59.460 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "172.21.15.7:0/2596041177", 2024-09-18T08:16:59.460 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 86394 2024-09-18T08:16:59.460 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:16:59.460 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-09-18T08:16:59.460 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:16:59.460 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-09-18T08:16:59.460 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:12.360967+0000", 2024-09-18T08:16:59.460 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-18T08:16:59.460 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:16:59.461 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:16:59.461 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-09-18T08:16:59.461 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:12.360967+0000", 2024-09-18T08:16:59.461 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-18T08:16:59.461 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:16:59.461 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:16:59.461 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-09-18T08:16:59.461 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:12.360969+0000", 2024-09-18T08:16:59.461 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.8500000000000001e-06 2024-09-18T08:16:59.461 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:16:59.461 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:16:59.461 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-09-18T08:16:59.461 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:12.361013+0000", 2024-09-18T08:16:59.462 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4.3693000000000003e-05 2024-09-18T08:16:59.462 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:16:59.462 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:16:59.462 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-09-18T08:16:59.462 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:12.361031+0000", 2024-09-18T08:16:59.462 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.8340000000000001e-05 2024-09-18T08:16:59.462 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:16:59.462 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:16:59.462 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-09-18T08:16:59.462 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:12.361040+0000", 2024-09-18T08:16:59.462 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 8.5320000000000006e-06 2024-09-18T08:16:59.462 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:16:59.462 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:16:59.463 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-18T08:16:59.463 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:12.361108+0000", 2024-09-18T08:16:59.463 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 6.7879000000000005e-05 2024-09-18T08:16:59.463 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:16:59.463 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:16:59.463 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "waiting for peered", 2024-09-18T08:16:59.463 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:12.361119+0000", 2024-09-18T08:16:59.463 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.1163e-05 2024-09-18T08:16:59.463 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:16:59.463 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:16:59.463 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-18T08:16:59.463 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:13.197641+0000", 2024-09-18T08:16:59.463 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.83652230299999997 2024-09-18T08:16:59.463 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:16:59.464 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:16:59.464 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-09-18T08:16:59.464 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:13.197776+0000", 2024-09-18T08:16:59.464 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00013499000000000001 2024-09-18T08:16:59.464 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:16:59.464 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:16:59.464 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "waiting for subops from 6,7", 2024-09-18T08:16:59.464 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:13.197954+0000", 2024-09-18T08:16:59.464 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00017745200000000001 2024-09-18T08:16:59.464 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:16:59.464 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:16:59.464 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:16:59.464 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:13.207329+0000", 2024-09-18T08:16:59.465 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0093754460000000008 2024-09-18T08:16:59.465 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:16:59.465 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:16:59.465 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:16:59.465 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:13.207363+0000", 2024-09-18T08:16:59.465 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 3.3535999999999999e-05 2024-09-18T08:16:59.465 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:16:59.465 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:16:59.465 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "op_commit", 2024-09-18T08:16:59.465 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:13.220323+0000", 2024-09-18T08:16:59.465 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.012960121999999999 2024-09-18T08:16:59.465 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:16:59.465 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:16:59.466 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "commit_sent", 2024-09-18T08:16:59.466 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:13.220445+0000", 2024-09-18T08:16:59.466 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00012217099999999999 2024-09-18T08:16:59.466 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:16:59.466 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:16:59.466 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-09-18T08:16:59.466 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:13.220475+0000", 2024-09-18T08:16:59.466 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 3.0324e-05 2024-09-18T08:16:59.466 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-18T08:16:59.466 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-09-18T08:16:59.466 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-18T08:16:59.466 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:16:59.466 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:16:59.467 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.15216.0:86399 8.5 8:a535189e:::benchmark_data_smithi007_101583_object86398:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e205)", 2024-09-18T08:16:59.467 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-09-18T08:09:12.364040+0000", 2024-09-18T08:16:59.467 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 467.02448480999999, 2024-09-18T08:16:59.467 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.85654366599999998, 2024-09-18T08:16:59.467 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-09-18T08:16:59.467 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-09-18T08:16:59.467 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T08:16:59.467 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2024-09-18T08:16:59.467 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.15216", 2024-09-18T08:16:59.467 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "172.21.15.7:0/2596041177", 2024-09-18T08:16:59.467 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 86399 2024-09-18T08:16:59.467 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:16:59.467 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-09-18T08:16:59.468 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:16:59.468 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-09-18T08:16:59.468 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:12.364040+0000", 2024-09-18T08:16:59.468 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-18T08:16:59.468 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:16:59.468 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:16:59.468 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-09-18T08:16:59.468 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:12.364040+0000", 2024-09-18T08:16:59.468 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-18T08:16:59.468 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:16:59.468 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:16:59.468 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-09-18T08:16:59.468 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:12.364042+0000", 2024-09-18T08:16:59.468 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.407e-06 2024-09-18T08:16:59.469 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:16:59.469 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:16:59.469 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-09-18T08:16:59.469 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:12.364174+0000", 2024-09-18T08:16:59.469 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000132545 2024-09-18T08:16:59.469 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:16:59.469 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:16:59.469 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-09-18T08:16:59.469 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:12.364186+0000", 2024-09-18T08:16:59.469 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.1912e-05 2024-09-18T08:16:59.469 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:16:59.469 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:16:59.469 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-09-18T08:16:59.470 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:12.364192+0000", 2024-09-18T08:16:59.470 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 5.4600000000000002e-06 2024-09-18T08:16:59.470 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:16:59.470 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:16:59.470 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-18T08:16:59.470 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:12.364257+0000", 2024-09-18T08:16:59.470 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 6.5606999999999997e-05 2024-09-18T08:16:59.470 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:16:59.470 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:16:59.470 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "waiting for peered", 2024-09-18T08:16:59.470 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:12.364271+0000", 2024-09-18T08:16:59.470 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.3433e-05 2024-09-18T08:16:59.470 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:16:59.471 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:16:59.471 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-18T08:16:59.471 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:13.200082+0000", 2024-09-18T08:16:59.471 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.83581135500000003 2024-09-18T08:16:59.471 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:16:59.471 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:16:59.471 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-09-18T08:16:59.471 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:13.200201+0000", 2024-09-18T08:16:59.471 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00011913099999999999 2024-09-18T08:16:59.471 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:16:59.471 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:16:59.471 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "waiting for subops from 6,7", 2024-09-18T08:16:59.471 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:13.200360+0000", 2024-09-18T08:16:59.471 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000159173 2024-09-18T08:16:59.472 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:16:59.472 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:16:59.472 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:16:59.472 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:13.207395+0000", 2024-09-18T08:16:59.472 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0070341090000000002 2024-09-18T08:16:59.472 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:16:59.472 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:16:59.472 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:16:59.472 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:13.207427+0000", 2024-09-18T08:16:59.472 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 3.2138000000000003e-05 2024-09-18T08:16:59.472 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:16:59.472 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:16:59.473 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "op_commit", 2024-09-18T08:16:59.473 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:13.220524+0000", 2024-09-18T08:16:59.473 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.013097698 2024-09-18T08:16:59.473 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:16:59.473 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:16:59.473 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "commit_sent", 2024-09-18T08:16:59.473 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:13.220557+0000", 2024-09-18T08:16:59.473 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 3.2682999999999999e-05 2024-09-18T08:16:59.473 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:16:59.473 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:16:59.473 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-09-18T08:16:59.473 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:13.220584+0000", 2024-09-18T08:16:59.473 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 2.7015000000000001e-05 2024-09-18T08:16:59.474 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-18T08:16:59.474 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-09-18T08:16:59.474 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-18T08:16:59.474 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:16:59.474 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:16:59.474 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.15216.0:86429 8.5 8:a7f1f687:::benchmark_data_smithi007_101583_object86428:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e205)", 2024-09-18T08:16:59.474 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-09-18T08:09:12.416915+0000", 2024-09-18T08:16:59.474 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 466.97161004100002, 2024-09-18T08:16:59.474 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.80373175900000005, 2024-09-18T08:16:59.474 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-09-18T08:16:59.474 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-09-18T08:16:59.474 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T08:16:59.475 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2024-09-18T08:16:59.475 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.15216", 2024-09-18T08:16:59.475 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "172.21.15.7:0/2596041177", 2024-09-18T08:16:59.475 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 86429 2024-09-18T08:16:59.475 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:16:59.475 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-09-18T08:16:59.475 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:16:59.475 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-09-18T08:16:59.475 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:12.416915+0000", 2024-09-18T08:16:59.475 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-18T08:16:59.475 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:16:59.475 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:16:59.475 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-09-18T08:16:59.475 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:12.416915+0000", 2024-09-18T08:16:59.476 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-18T08:16:59.476 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:16:59.476 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:16:59.476 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-09-18T08:16:59.476 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:12.416917+0000", 2024-09-18T08:16:59.476 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.7150000000000001e-06 2024-09-18T08:16:59.476 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:16:59.476 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:16:59.476 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-09-18T08:16:59.476 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:12.416952+0000", 2024-09-18T08:16:59.476 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 3.4860000000000002e-05 2024-09-18T08:16:59.476 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:16:59.476 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:16:59.476 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-09-18T08:16:59.477 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:12.416966+0000", 2024-09-18T08:16:59.477 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.3884e-05 2024-09-18T08:16:59.477 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:16:59.477 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:16:59.477 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-09-18T08:16:59.477 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:12.416973+0000", 2024-09-18T08:16:59.477 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 7.5809999999999999e-06 2024-09-18T08:16:59.477 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:16:59.477 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:16:59.477 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-18T08:16:59.477 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:12.417031+0000", 2024-09-18T08:16:59.477 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 5.7544999999999999e-05 2024-09-18T08:16:59.477 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:16:59.478 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:16:59.478 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "waiting for peered", 2024-09-18T08:16:59.478 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:12.417039+0000", 2024-09-18T08:16:59.478 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 8.5369999999999997e-06 2024-09-18T08:16:59.478 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:16:59.478 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:16:59.478 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-18T08:16:59.478 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:13.202704+0000", 2024-09-18T08:16:59.478 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.78566510099999998 2024-09-18T08:16:59.478 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:16:59.478 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:16:59.478 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-09-18T08:16:59.478 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:13.202825+0000", 2024-09-18T08:16:59.479 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00012023600000000001 2024-09-18T08:16:59.479 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:16:59.479 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:16:59.479 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "waiting for subops from 6,7", 2024-09-18T08:16:59.479 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:13.202989+0000", 2024-09-18T08:16:59.479 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00016451000000000001 2024-09-18T08:16:59.479 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:16:59.479 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:16:59.479 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:16:59.479 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:13.207464+0000", 2024-09-18T08:16:59.479 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0044748909999999999 2024-09-18T08:16:59.479 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:16:59.479 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:16:59.480 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:16:59.480 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:13.214034+0000", 2024-09-18T08:16:59.480 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0065698229999999998 2024-09-18T08:16:59.480 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:16:59.480 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:16:59.480 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "op_commit", 2024-09-18T08:16:59.480 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:13.220590+0000", 2024-09-18T08:16:59.480 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0065556909999999998 2024-09-18T08:16:59.480 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:16:59.480 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:16:59.480 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "commit_sent", 2024-09-18T08:16:59.480 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:13.220620+0000", 2024-09-18T08:16:59.480 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 3.0429999999999998e-05 2024-09-18T08:16:59.480 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:16:59.481 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:16:59.481 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-09-18T08:16:59.481 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:13.220647+0000", 2024-09-18T08:16:59.481 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 2.6954999999999999e-05 2024-09-18T08:16:59.481 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-18T08:16:59.481 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-09-18T08:16:59.481 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-18T08:16:59.481 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:16:59.481 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:16:59.481 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.15582.0:53562 9.e 9:7f5cd26d:::benchmark_data_smithi007_124019_object53561:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e243)", 2024-09-18T08:16:59.481 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-09-18T08:11:41.305003+0000", 2024-09-18T08:16:59.481 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 318.08352251700001, 2024-09-18T08:16:59.481 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.032254158, 2024-09-18T08:16:59.481 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-09-18T08:16:59.482 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-09-18T08:16:59.482 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "reached pg", 2024-09-18T08:16:59.482 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2024-09-18T08:16:59.482 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.15582", 2024-09-18T08:16:59.482 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "172.21.15.7:0/3598605839", 2024-09-18T08:16:59.482 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 53562 2024-09-18T08:16:59.482 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:16:59.482 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-09-18T08:16:59.482 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:16:59.482 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-09-18T08:16:59.482 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:11:41.305003+0000", 2024-09-18T08:16:59.482 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-18T08:16:59.482 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:16:59.482 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:16:59.483 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-09-18T08:16:59.483 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:11:41.305003+0000", 2024-09-18T08:16:59.483 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-18T08:16:59.483 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:16:59.483 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:16:59.483 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-09-18T08:16:59.483 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:11:41.305004+0000", 2024-09-18T08:16:59.483 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.6750000000000001e-06 2024-09-18T08:16:59.483 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:16:59.483 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:16:59.483 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-09-18T08:16:59.483 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:11:41.305059+0000", 2024-09-18T08:16:59.483 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 5.4181999999999997e-05 2024-09-18T08:16:59.484 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:16:59.484 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:16:59.484 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-09-18T08:16:59.484 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:11:41.305074+0000", 2024-09-18T08:16:59.484 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.5379999999999998e-05 2024-09-18T08:16:59.484 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:16:59.484 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:16:59.484 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-09-18T08:16:59.484 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:11:41.305083+0000", 2024-09-18T08:16:59.484 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 8.9549999999999998e-06 2024-09-18T08:16:59.484 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:16:59.484 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:16:59.484 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-18T08:16:59.485 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:11:41.305153+0000", 2024-09-18T08:16:59.485 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 6.9703999999999995e-05 2024-09-18T08:16:59.485 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:16:59.485 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:16:59.485 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-09-18T08:16:59.485 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:11:41.305277+0000", 2024-09-18T08:16:59.485 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0001247 2024-09-18T08:16:59.485 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:16:59.485 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:16:59.485 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "waiting for subops from 1,7", 2024-09-18T08:16:59.485 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:11:41.305414+0000", 2024-09-18T08:16:59.485 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00013647900000000001 2024-09-18T08:16:59.485 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:16:59.486 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:16:59.486 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "op_commit", 2024-09-18T08:16:59.486 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:11:41.307315+0000", 2024-09-18T08:16:59.486 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.001901511 2024-09-18T08:16:59.486 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:16:59.486 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:16:59.486 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:16:59.486 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:11:41.307516+0000", 2024-09-18T08:16:59.486 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00020116900000000001 2024-09-18T08:16:59.486 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:16:59.486 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:16:59.486 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-18T08:16:59.486 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:11:41.323519+0000", 2024-09-18T08:16:59.487 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.016002283999999999 2024-09-18T08:16:59.487 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:16:59.487 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:16:59.487 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "waiting for peered", 2024-09-18T08:16:59.487 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:11:41.323530+0000", 2024-09-18T08:16:59.487 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.1729000000000001e-05 2024-09-18T08:16:59.487 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:16:59.487 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:16:59.487 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-18T08:16:59.487 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:11:42.337165+0000", 2024-09-18T08:16:59.487 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.0136343050000001 2024-09-18T08:16:59.487 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:16:59.487 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:16:59.487 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-09-18T08:16:59.488 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:11:42.337257+0000", 2024-09-18T08:16:59.488 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 9.2084999999999997e-05 2024-09-18T08:16:59.488 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-18T08:16:59.488 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-09-18T08:16:59.488 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-18T08:16:59.488 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:16:59.488 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:16:59.488 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.15582.0:53574 9.e 9:7c49e58c:::benchmark_data_smithi007_124019_object53573:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e243)", 2024-09-18T08:16:59.488 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-09-18T08:11:41.323835+0000", 2024-09-18T08:16:59.488 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 318.06469013399999, 2024-09-18T08:16:59.488 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.0202460840000001, 2024-09-18T08:16:59.488 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-09-18T08:16:59.488 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-09-18T08:16:59.489 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T08:16:59.489 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2024-09-18T08:16:59.489 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.15582", 2024-09-18T08:16:59.489 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "172.21.15.7:0/3598605839", 2024-09-18T08:16:59.489 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 53574 2024-09-18T08:16:59.489 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:16:59.489 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-09-18T08:16:59.489 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:16:59.489 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-09-18T08:16:59.489 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:11:41.323835+0000", 2024-09-18T08:16:59.489 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-18T08:16:59.489 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:16:59.489 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:16:59.489 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-09-18T08:16:59.490 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:11:41.323835+0000", 2024-09-18T08:16:59.490 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-18T08:16:59.490 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:16:59.490 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:16:59.490 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-09-18T08:16:59.490 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:11:41.323837+0000", 2024-09-18T08:16:59.490 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.8470000000000001e-06 2024-09-18T08:16:59.490 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:16:59.490 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:16:59.490 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-09-18T08:16:59.490 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:11:41.323872+0000", 2024-09-18T08:16:59.490 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 3.4660999999999999e-05 2024-09-18T08:16:59.490 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:16:59.490 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:16:59.491 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-09-18T08:16:59.491 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:11:41.323886+0000", 2024-09-18T08:16:59.491 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.4779e-05 2024-09-18T08:16:59.491 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:16:59.491 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:16:59.491 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-09-18T08:16:59.491 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:11:41.323895+0000", 2024-09-18T08:16:59.491 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 8.4630000000000001e-06 2024-09-18T08:16:59.491 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:16:59.491 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:16:59.491 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-18T08:16:59.491 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:11:41.323994+0000", 2024-09-18T08:16:59.491 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 9.9152000000000002e-05 2024-09-18T08:16:59.492 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:16:59.492 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:16:59.492 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "waiting for peered", 2024-09-18T08:16:59.492 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:11:41.324002+0000", 2024-09-18T08:16:59.492 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 8.4039999999999995e-06 2024-09-18T08:16:59.492 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:16:59.492 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:16:59.492 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-18T08:16:59.492 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:11:42.337953+0000", 2024-09-18T08:16:59.492 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.013950753 2024-09-18T08:16:59.492 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:16:59.492 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:16:59.492 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-09-18T08:16:59.492 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:11:42.338094+0000", 2024-09-18T08:16:59.493 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000140371 2024-09-18T08:16:59.493 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:16:59.493 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:16:59.493 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "waiting for subops from 7", 2024-09-18T08:16:59.493 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:11:42.338248+0000", 2024-09-18T08:16:59.493 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000154723 2024-09-18T08:16:59.493 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:16:59.493 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:16:59.493 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:16:59.493 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:11:42.343852+0000", 2024-09-18T08:16:59.493 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0056041909999999997 2024-09-18T08:16:59.493 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:16:59.493 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:16:59.493 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "op_commit", 2024-09-18T08:16:59.494 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:11:42.343917+0000", 2024-09-18T08:16:59.494 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 6.4090000000000005e-05 2024-09-18T08:16:59.494 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:16:59.494 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:16:59.494 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "commit_sent", 2024-09-18T08:16:59.494 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:11:42.343980+0000", 2024-09-18T08:16:59.494 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 6.3769999999999994e-05 2024-09-18T08:16:59.494 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:16:59.494 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:16:59.494 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-09-18T08:16:59.494 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:11:42.344081+0000", 2024-09-18T08:16:59.494 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00010088 2024-09-18T08:16:59.494 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-18T08:16:59.495 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-09-18T08:16:59.495 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-18T08:16:59.495 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:16:59.495 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:16:59.495 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.15582.0:53552 9.4 9:2d548aaa:::benchmark_data_smithi007_124019_object53551:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e244)", 2024-09-18T08:16:59.495 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-09-18T08:11:41.324333+0000", 2024-09-18T08:16:59.495 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 318.06419270999999, 2024-09-18T08:16:59.495 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.0111114809999999, 2024-09-18T08:16:59.495 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-09-18T08:16:59.495 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-09-18T08:16:59.495 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "reached pg", 2024-09-18T08:16:59.495 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2024-09-18T08:16:59.495 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.15582", 2024-09-18T08:16:59.495 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "172.21.15.7:0/3598605839", 2024-09-18T08:16:59.496 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 53552 2024-09-18T08:16:59.496 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:16:59.496 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-09-18T08:16:59.496 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:16:59.496 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-09-18T08:16:59.496 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:11:41.324333+0000", 2024-09-18T08:16:59.496 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-18T08:16:59.496 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:16:59.496 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:16:59.496 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-09-18T08:16:59.496 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:11:41.324333+0000", 2024-09-18T08:16:59.496 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-18T08:16:59.496 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:16:59.497 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:16:59.497 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-09-18T08:16:59.497 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:11:41.324334+0000", 2024-09-18T08:16:59.497 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.217e-06 2024-09-18T08:16:59.497 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:16:59.497 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:16:59.497 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-09-18T08:16:59.497 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:11:41.324369+0000", 2024-09-18T08:16:59.497 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 3.4805999999999998e-05 2024-09-18T08:16:59.497 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:16:59.497 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:16:59.497 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-09-18T08:16:59.498 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:11:41.324382+0000", 2024-09-18T08:16:59.498 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.3739e-05 2024-09-18T08:16:59.498 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:16:59.498 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:16:59.498 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-09-18T08:16:59.498 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:11:41.324390+0000", 2024-09-18T08:16:59.498 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 7.3989999999999998e-06 2024-09-18T08:16:59.498 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:16:59.498 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:16:59.498 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-18T08:16:59.498 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:11:41.324445+0000", 2024-09-18T08:16:59.498 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 5.5535999999999998e-05 2024-09-18T08:16:59.498 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:16:59.498 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:16:59.499 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "waiting for peered", 2024-09-18T08:16:59.499 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:11:41.324453+0000", 2024-09-18T08:16:59.499 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 8.2139999999999996e-06 2024-09-18T08:16:59.499 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:16:59.499 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:16:59.499 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-18T08:16:59.499 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:11:42.335357+0000", 2024-09-18T08:16:59.499 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.010903675 2024-09-18T08:16:59.499 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:16:59.499 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:16:59.499 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-09-18T08:16:59.499 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:11:42.335444+0000", 2024-09-18T08:16:59.499 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 8.6895000000000001e-05 2024-09-18T08:16:59.500 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-18T08:16:59.500 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-09-18T08:16:59.500 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-18T08:16:59.500 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:16:59.500 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:16:59.500 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.15582.0:53563 9.4 9:2254eea2:::benchmark_data_smithi007_124019_object53562:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e244)", 2024-09-18T08:16:59.500 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-09-18T08:11:41.324409+0000", 2024-09-18T08:16:59.500 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 318.06411603200002, 2024-09-18T08:16:59.500 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.022272439, 2024-09-18T08:16:59.500 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-09-18T08:16:59.500 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-09-18T08:16:59.500 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "reached pg", 2024-09-18T08:16:59.500 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2024-09-18T08:16:59.501 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.15582", 2024-09-18T08:16:59.501 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "172.21.15.7:0/3598605839", 2024-09-18T08:16:59.501 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 53563 2024-09-18T08:16:59.501 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:16:59.501 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-09-18T08:16:59.501 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:16:59.501 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-09-18T08:16:59.501 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:11:41.324409+0000", 2024-09-18T08:16:59.501 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-18T08:16:59.501 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:16:59.501 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:16:59.501 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-09-18T08:16:59.501 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:11:41.324409+0000", 2024-09-18T08:16:59.502 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-18T08:16:59.502 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:16:59.502 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:16:59.502 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-09-18T08:16:59.502 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:11:41.324410+0000", 2024-09-18T08:16:59.502 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 5.8400000000000004e-07 2024-09-18T08:16:59.502 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:16:59.502 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:16:59.502 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-09-18T08:16:59.502 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:11:41.324438+0000", 2024-09-18T08:16:59.502 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 2.8438000000000001e-05 2024-09-18T08:16:59.502 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:16:59.502 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:16:59.502 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-09-18T08:16:59.503 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:11:41.324446+0000", 2024-09-18T08:16:59.503 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 8.0020000000000006e-06 2024-09-18T08:16:59.503 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:16:59.503 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:16:59.503 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-09-18T08:16:59.503 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:11:41.324453+0000", 2024-09-18T08:16:59.503 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 6.4910000000000002e-06 2024-09-18T08:16:59.503 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:16:59.503 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:16:59.503 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-18T08:16:59.503 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:11:41.324565+0000", 2024-09-18T08:16:59.503 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000111983 2024-09-18T08:16:59.503 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:16:59.503 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:16:59.504 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "waiting for peered", 2024-09-18T08:16:59.504 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:11:41.324572+0000", 2024-09-18T08:16:59.504 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 7.8080000000000005e-06 2024-09-18T08:16:59.504 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:16:59.504 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:16:59.504 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-18T08:16:59.504 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:11:42.335472+0000", 2024-09-18T08:16:59.504 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.010899996 2024-09-18T08:16:59.504 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:16:59.504 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:16:59.504 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "waiting for missing object", 2024-09-18T08:16:59.504 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:11:42.335604+0000", 2024-09-18T08:16:59.504 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00013134999999999999 2024-09-18T08:16:59.505 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:16:59.505 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:16:59.505 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-18T08:16:59.505 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:11:42.346593+0000", 2024-09-18T08:16:59.505 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.01098872 2024-09-18T08:16:59.505 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:16:59.505 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:16:59.505 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-09-18T08:16:59.505 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:11:42.346682+0000", 2024-09-18T08:16:59.505 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 8.9067000000000006e-05 2024-09-18T08:16:59.505 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-18T08:16:59.505 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-09-18T08:16:59.506 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-18T08:16:59.506 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:16:59.506 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:16:59.506 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.24356.0:2784 5.1f 5:f95f44c2:::notify.0:head [watch reconnect cookie 94700888679552 gen 6] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e292)", 2024-09-18T08:16:59.506 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-09-18T08:14:04.566002+0000", 2024-09-18T08:16:59.506 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 174.822523437, 2024-09-18T08:16:59.506 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.026328065, 2024-09-18T08:16:59.506 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-09-18T08:16:59.506 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-09-18T08:16:59.506 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "started", 2024-09-18T08:16:59.506 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2024-09-18T08:16:59.506 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.24356", 2024-09-18T08:16:59.506 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "172.21.15.7:0/219168083", 2024-09-18T08:16:59.506 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 2784 2024-09-18T08:16:59.507 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:16:59.507 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-09-18T08:16:59.507 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:16:59.507 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-09-18T08:16:59.507 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:14:04.566002+0000", 2024-09-18T08:16:59.507 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-18T08:16:59.507 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:16:59.507 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:16:59.507 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-09-18T08:16:59.507 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:14:04.566002+0000", 2024-09-18T08:16:59.507 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-18T08:16:59.507 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:16:59.508 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:16:59.508 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-09-18T08:16:59.508 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:14:04.566004+0000", 2024-09-18T08:16:59.508 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.9350000000000001e-06 2024-09-18T08:16:59.508 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:16:59.508 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:16:59.508 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-09-18T08:16:59.508 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:14:04.566009+0000", 2024-09-18T08:16:59.508 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 5.5589999999999996e-06 2024-09-18T08:16:59.508 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:16:59.508 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:16:59.508 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-09-18T08:16:59.508 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:14:04.566021+0000", 2024-09-18T08:16:59.509 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.1398e-05 2024-09-18T08:16:59.509 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:16:59.509 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:16:59.509 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-09-18T08:16:59.509 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:14:04.566028+0000", 2024-09-18T08:16:59.509 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 7.8029999999999997e-06 2024-09-18T08:16:59.509 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:16:59.509 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:16:59.509 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-18T08:16:59.509 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:14:04.574060+0000", 2024-09-18T08:16:59.509 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0080314459999999994 2024-09-18T08:16:59.509 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:16:59.510 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:16:59.510 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "waiting for peered", 2024-09-18T08:16:59.510 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:14:04.574071+0000", 2024-09-18T08:16:59.510 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.0693000000000001e-05 2024-09-18T08:16:59.510 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:16:59.510 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:16:59.510 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-18T08:16:59.510 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:14:05.582227+0000", 2024-09-18T08:16:59.510 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.0081559760000001 2024-09-18T08:16:59.510 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:16:59.510 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:16:59.510 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "waiting for degraded object", 2024-09-18T08:16:59.510 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:14:05.582584+0000", 2024-09-18T08:16:59.510 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000357096 2024-09-18T08:16:59.511 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:16:59.511 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:16:59.511 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-18T08:16:59.511 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:14:05.585756+0000", 2024-09-18T08:16:59.511 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0031727320000000002 2024-09-18T08:16:59.511 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:16:59.511 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:16:59.511 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-09-18T08:16:59.511 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:14:05.585848+0000", 2024-09-18T08:16:59.511 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 9.1587000000000003e-05 2024-09-18T08:16:59.511 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:16:59.511 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:16:59.511 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-09-18T08:16:59.512 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:14:05.592330+0000", 2024-09-18T08:16:59.512 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00648184 2024-09-18T08:16:59.512 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-18T08:16:59.512 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-09-18T08:16:59.512 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-18T08:16:59.512 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:16:59.512 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:16:59.512 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.24356.0:2785 4.5 4:a60f9269:lc::lua_package_allowlist:head [watch reconnect cookie 94700889442560 gen 6] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e292)", 2024-09-18T08:16:59.512 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-09-18T08:14:04.566047+0000", 2024-09-18T08:16:59.512 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 174.82247820200001, 2024-09-18T08:16:59.512 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.027464521, 2024-09-18T08:16:59.512 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-09-18T08:16:59.512 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-09-18T08:16:59.513 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "started", 2024-09-18T08:16:59.513 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2024-09-18T08:16:59.513 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.24356", 2024-09-18T08:16:59.513 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "172.21.15.7:0/219168083", 2024-09-18T08:16:59.513 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 2785 2024-09-18T08:16:59.513 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:16:59.513 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-09-18T08:16:59.513 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:16:59.513 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-09-18T08:16:59.513 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:14:04.566047+0000", 2024-09-18T08:16:59.513 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-18T08:16:59.513 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:16:59.513 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:16:59.514 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-09-18T08:16:59.514 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:14:04.566047+0000", 2024-09-18T08:16:59.514 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-18T08:16:59.514 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:16:59.514 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:16:59.514 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-09-18T08:16:59.514 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:14:04.566048+0000", 2024-09-18T08:16:59.514 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 5.8500000000000001e-07 2024-09-18T08:16:59.514 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:16:59.514 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:16:59.514 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-09-18T08:16:59.514 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:14:04.566050+0000", 2024-09-18T08:16:59.514 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 2.8509999999999999e-06 2024-09-18T08:16:59.514 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:16:59.515 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:16:59.515 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-09-18T08:16:59.515 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:14:04.566057+0000", 2024-09-18T08:16:59.515 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 6.9369999999999998e-06 2024-09-18T08:16:59.515 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:16:59.515 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:16:59.515 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-09-18T08:16:59.515 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:14:04.566063+0000", 2024-09-18T08:16:59.515 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 5.6049999999999997e-06 2024-09-18T08:16:59.515 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:16:59.515 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:16:59.515 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-18T08:16:59.515 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:14:04.568597+0000", 2024-09-18T08:16:59.516 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0025344730000000002 2024-09-18T08:16:59.516 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:16:59.516 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:16:59.516 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "waiting for peered", 2024-09-18T08:16:59.516 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:14:04.568606+0000", 2024-09-18T08:16:59.516 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 8.0649999999999999e-06 2024-09-18T08:16:59.516 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:16:59.516 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:16:59.516 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-18T08:16:59.516 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:14:05.586207+0000", 2024-09-18T08:16:59.516 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.017601309 2024-09-18T08:16:59.516 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:16:59.516 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:16:59.516 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "waiting for degraded object", 2024-09-18T08:16:59.517 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:14:05.586551+0000", 2024-09-18T08:16:59.517 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00034398799999999999 2024-09-18T08:16:59.517 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:16:59.517 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:16:59.517 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-18T08:16:59.517 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:14:05.590223+0000", 2024-09-18T08:16:59.517 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0036724499999999998 2024-09-18T08:16:59.517 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:16:59.517 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:16:59.517 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-09-18T08:16:59.517 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:14:05.590327+0000", 2024-09-18T08:16:59.517 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000103261 2024-09-18T08:16:59.518 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:16:59.518 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:16:59.518 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-09-18T08:16:59.518 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:14:05.593512+0000", 2024-09-18T08:16:59.518 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0031849970000000002 2024-09-18T08:16:59.518 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-18T08:16:59.518 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-09-18T08:16:59.518 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-18T08:16:59.518 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:16:59.518 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:16:59.518 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.24356.0:2786 5.9 5:93e5b521:::notify.7:head [watch reconnect cookie 94700888687616 gen 5] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e292)", 2024-09-18T08:16:59.518 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-09-18T08:14:04.566075+0000", 2024-09-18T08:16:59.518 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 174.82245055000001, 2024-09-18T08:16:59.518 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 2.0238415459999999, 2024-09-18T08:16:59.519 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-09-18T08:16:59.519 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-09-18T08:16:59.519 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "started", 2024-09-18T08:16:59.519 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2024-09-18T08:16:59.519 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.24356", 2024-09-18T08:16:59.519 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "172.21.15.7:0/219168083", 2024-09-18T08:16:59.519 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 2786 2024-09-18T08:16:59.519 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:16:59.519 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-09-18T08:16:59.520 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:16:59.520 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-09-18T08:16:59.520 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:14:04.566075+0000", 2024-09-18T08:16:59.520 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-18T08:16:59.520 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:16:59.520 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:16:59.520 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-09-18T08:16:59.520 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:14:04.566075+0000", 2024-09-18T08:16:59.520 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-18T08:16:59.520 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:16:59.520 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:16:59.520 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-09-18T08:16:59.520 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:14:04.566075+0000", 2024-09-18T08:16:59.521 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4.6699999999999999e-07 2024-09-18T08:16:59.521 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:16:59.521 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:16:59.521 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-09-18T08:16:59.521 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:14:04.566077+0000", 2024-09-18T08:16:59.521 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 2.283e-06 2024-09-18T08:16:59.521 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:16:59.521 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:16:59.521 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-09-18T08:16:59.521 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:14:04.566083+0000", 2024-09-18T08:16:59.521 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 5.4739999999999996e-06 2024-09-18T08:16:59.521 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:16:59.521 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:16:59.521 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-09-18T08:16:59.522 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:14:04.566088+0000", 2024-09-18T08:16:59.522 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4.887e-06 2024-09-18T08:16:59.522 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:16:59.522 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:16:59.522 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-18T08:16:59.522 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:14:04.567450+0000", 2024-09-18T08:16:59.522 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.001362261 2024-09-18T08:16:59.522 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:16:59.522 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:16:59.522 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "waiting for peered", 2024-09-18T08:16:59.522 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:14:04.567459+0000", 2024-09-18T08:16:59.522 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 8.5110000000000003e-06 2024-09-18T08:16:59.522 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:16:59.522 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:16:59.523 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-18T08:16:59.523 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:14:05.568619+0000", 2024-09-18T08:16:59.523 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.001160869 2024-09-18T08:16:59.523 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:16:59.523 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:16:59.523 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "waiting for peered", 2024-09-18T08:16:59.523 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:14:05.568637+0000", 2024-09-18T08:16:59.523 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.7801000000000001e-05 2024-09-18T08:16:59.523 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:16:59.523 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:16:59.523 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-18T08:16:59.523 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:14:06.586669+0000", 2024-09-18T08:16:59.523 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.01803212 2024-09-18T08:16:59.523 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:16:59.524 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:16:59.524 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-09-18T08:16:59.524 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:14:06.586966+0000", 2024-09-18T08:16:59.524 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00029684900000000002 2024-09-18T08:16:59.524 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:16:59.524 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:16:59.524 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-09-18T08:16:59.524 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:14:06.589916+0000", 2024-09-18T08:16:59.524 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0029500239999999999 2024-09-18T08:16:59.524 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-18T08:16:59.524 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-09-18T08:16:59.524 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-18T08:16:59.524 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:16:59.525 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:16:59.525 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.24356.0:2788 5.9 5:93e5b521:::notify.7:head [watch reconnect cookie 94700888687616 gen 6] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e293)", 2024-09-18T08:16:59.525 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-09-18T08:14:05.569076+0000", 2024-09-18T08:16:59.525 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 173.81944901200001, 2024-09-18T08:16:59.525 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.021128322, 2024-09-18T08:16:59.525 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-09-18T08:16:59.525 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-09-18T08:16:59.525 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "started", 2024-09-18T08:16:59.525 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2024-09-18T08:16:59.525 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.24356", 2024-09-18T08:16:59.525 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "172.21.15.7:0/219168083", 2024-09-18T08:16:59.525 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 2788 2024-09-18T08:16:59.525 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:16:59.526 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-09-18T08:16:59.526 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:16:59.526 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-09-18T08:16:59.526 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:14:05.569076+0000", 2024-09-18T08:16:59.526 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-18T08:16:59.526 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:16:59.526 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:16:59.526 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-09-18T08:16:59.526 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:14:05.569076+0000", 2024-09-18T08:16:59.526 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-18T08:16:59.526 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:16:59.526 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:16:59.526 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-09-18T08:16:59.526 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:14:05.569078+0000", 2024-09-18T08:16:59.527 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 2.0250000000000001e-06 2024-09-18T08:16:59.527 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:16:59.527 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:16:59.527 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-09-18T08:16:59.527 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:14:05.569086+0000", 2024-09-18T08:16:59.527 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 7.9500000000000001e-06 2024-09-18T08:16:59.527 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:16:59.527 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:16:59.527 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-09-18T08:16:59.527 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:14:05.569099+0000", 2024-09-18T08:16:59.527 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.2755000000000001e-05 2024-09-18T08:16:59.527 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:16:59.527 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:16:59.527 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-09-18T08:16:59.528 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:14:05.569110+0000", 2024-09-18T08:16:59.528 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.1085e-05 2024-09-18T08:16:59.528 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:16:59.528 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:16:59.528 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-18T08:16:59.528 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:14:05.569176+0000", 2024-09-18T08:16:59.528 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 6.5499000000000004e-05 2024-09-18T08:16:59.528 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:16:59.528 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:16:59.528 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "waiting for peered", 2024-09-18T08:16:59.528 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:14:05.569187+0000", 2024-09-18T08:16:59.528 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.1537e-05 2024-09-18T08:16:59.528 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:16:59.529 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:16:59.529 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-18T08:16:59.529 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:14:06.588218+0000", 2024-09-18T08:16:59.529 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.0190308189999999 2024-09-18T08:16:59.529 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:16:59.529 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:16:59.529 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-09-18T08:16:59.529 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:14:06.588315+0000", 2024-09-18T08:16:59.529 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 9.7161e-05 2024-09-18T08:16:59.529 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:16:59.529 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:16:59.529 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-09-18T08:16:59.529 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:14:06.590205+0000", 2024-09-18T08:16:59.529 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0018894910000000001 2024-09-18T08:16:59.530 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-18T08:16:59.530 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-09-18T08:16:59.530 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-18T08:16:59.530 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-18T08:16:59.530 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-09-18T08:16:59.530 INFO:teuthology.orchestra.run.smithi050.stdout:} 2024-09-18T08:16:59.530 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:16:59.531 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1' 2024-09-18T08:16:59.631 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3 2024-09-18T08:16:59.677 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:16:59.678 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:16:59.678 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3' 2024-09-18T08:16:59.779 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2 2024-09-18T08:16:59.823 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:16:59.824 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:16:59.825 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2' 2024-09-18T08:16:59.905 INFO:journalctl@ceph.iscsi.iscsi.a.smithi050.stdout:Sep 18 08:16:59 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-iscsi-iscsi-a[57782]: debug there is no tcmu-runner data available 2024-09-18T08:16:59.925 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7 2024-09-18T08:16:59.954 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:16:59.955 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:16:59.955 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7' 2024-09-18T08:17:00.056 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4 2024-09-18T08:17:00.082 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:17:00.082 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:17:00.083 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4' 2024-09-18T08:17:00.184 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3 2024-09-18T08:17:00.238 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:17:00.238 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:17:00.239 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3' 2024-09-18T08:17:00.340 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7 2024-09-18T08:17:00.366 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:17:00.367 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:17:00.367 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7' 2024-09-18T08:17:00.468 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7 2024-09-18T08:17:00.493 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:17:00.494 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:17:00.494 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7' 2024-09-18T08:17:00.595 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5 2024-09-18T08:17:00.621 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:17:00.621 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:17:00.622 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5' 2024-09-18T08:17:00.655 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:17:00 smithi050 ceph-mon[32371]: purged_snaps scrub starts 2024-09-18T08:17:00.655 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:17:00 smithi050 ceph-mon[32371]: purged_snaps scrub ok 2024-09-18T08:17:00.655 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:17:00 smithi050 ceph-mon[32371]: pgmap v708: 158 pgs: 1 peering, 4 active+remapped+backfilling, 2 active+remapped+backfill_toofull, 151 active+clean; 4.5 GiB data, 16 GiB used, 521 GiB / 536 GiB avail; 870 B/s rd, 0 B/s wr, 662 op/s; 19237/221118 objects misplaced (8.700%); 80 MiB/s, 1.29k objects/s recovering 2024-09-18T08:17:00.655 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:17:00 smithi050 ceph-mon[32371]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-18T08:17:00.655 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:17:00 smithi050 ceph-mon[32371]: osdmap e368: 8 total, 8 up, 6 in 2024-09-18T08:17:00.701 DEBUG:teuthology.orchestra.run.smithi050:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid 779349ca-7593-11ef-bceb-c7b262605968 -- ceph daemon osd.5 dump_ops_in_flight 2024-09-18T08:17:00.723 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2 2024-09-18T08:17:00.730 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:17:00 smithi007 ceph-mon[29459]: purged_snaps scrub starts 2024-09-18T08:17:00.730 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:17:00 smithi007 ceph-mon[29459]: purged_snaps scrub ok 2024-09-18T08:17:00.730 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:17:00 smithi007 ceph-mon[29459]: pgmap v708: 158 pgs: 1 peering, 4 active+remapped+backfilling, 2 active+remapped+backfill_toofull, 151 active+clean; 4.5 GiB data, 16 GiB used, 521 GiB / 536 GiB avail; 870 B/s rd, 0 B/s wr, 662 op/s; 19237/221118 objects misplaced (8.700%); 80 MiB/s, 1.29k objects/s recovering 2024-09-18T08:17:00.730 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:17:00 smithi007 ceph-mon[29459]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-18T08:17:00.730 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:17:00 smithi007 ceph-mon[29459]: osdmap e368: 8 total, 8 up, 6 in 2024-09-18T08:17:00.731 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:17:00 smithi007 ceph-mon[39249]: purged_snaps scrub starts 2024-09-18T08:17:00.731 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:17:00 smithi007 ceph-mon[39249]: purged_snaps scrub ok 2024-09-18T08:17:00.731 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:17:00 smithi007 ceph-mon[39249]: pgmap v708: 158 pgs: 1 peering, 4 active+remapped+backfilling, 2 active+remapped+backfill_toofull, 151 active+clean; 4.5 GiB data, 16 GiB used, 521 GiB / 536 GiB avail; 870 B/s rd, 0 B/s wr, 662 op/s; 19237/221118 objects misplaced (8.700%); 80 MiB/s, 1.29k objects/s recovering 2024-09-18T08:17:00.731 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:17:00 smithi007 ceph-mon[39249]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-18T08:17:00.731 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:17:00 smithi007 ceph-mon[39249]: osdmap e368: 8 total, 8 up, 6 in 2024-09-18T08:17:00.768 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:17:00.769 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:17:00.769 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2' 2024-09-18T08:17:00.870 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6 2024-09-18T08:17:00.900 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:17:00.900 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:17:00.901 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6' 2024-09-18T08:17:01.001 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3 2024-09-18T08:17:01.052 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:17:01.053 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:17:01.053 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3' 2024-09-18T08:17:01.056 INFO:teuthology.orchestra.run.smithi050.stderr:Inferring config /var/lib/ceph/779349ca-7593-11ef-bceb-c7b262605968/mon.b/config 2024-09-18T08:17:01.154 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1 2024-09-18T08:17:01.198 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:17:01.199 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:17:01.200 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1' 2024-09-18T08:17:01.300 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2 2024-09-18T08:17:01.343 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:17:01.344 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:17:01.345 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2' 2024-09-18T08:17:01.445 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6 2024-09-18T08:17:01.474 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:17:01.475 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:17:01.475 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6' 2024-09-18T08:17:01.486 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:17:01 smithi050 ceph-mon[32371]: purged_snaps scrub starts 2024-09-18T08:17:01.486 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:17:01 smithi050 ceph-mon[32371]: purged_snaps scrub ok 2024-09-18T08:17:01.577 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0 2024-09-18T08:17:01.602 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2024-09-18T08:17:01.602 INFO:teuthology.orchestra.run.smithi050.stdout: "ops": [], 2024-09-18T08:17:01.602 INFO:teuthology.orchestra.run.smithi050.stdout: "num_ops": 0 2024-09-18T08:17:01.603 INFO:teuthology.orchestra.run.smithi050.stdout:} 2024-09-18T08:17:01.614 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:17:01.615 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:17:01.616 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0' 2024-09-18T08:17:01.717 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0 2024-09-18T08:17:01.730 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:17:01 smithi007 ceph-mon[29459]: purged_snaps scrub starts 2024-09-18T08:17:01.730 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:17:01 smithi007 ceph-mon[29459]: purged_snaps scrub ok 2024-09-18T08:17:01.730 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:17:01 smithi007 ceph-mon[39249]: purged_snaps scrub starts 2024-09-18T08:17:01.730 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:17:01 smithi007 ceph-mon[39249]: purged_snaps scrub ok 2024-09-18T08:17:01.747 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:17:01.748 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:17:01.749 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0' 2024-09-18T08:17:01.849 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7 2024-09-18T08:17:01.875 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:17:01.875 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:17:01.876 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7' 2024-09-18T08:17:01.977 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1 2024-09-18T08:17:02.017 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:17:02.019 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:17:02.019 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1' 2024-09-18T08:17:02.120 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2 2024-09-18T08:17:02.157 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:17:02.158 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:17:02.158 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2' 2024-09-18T08:17:02.259 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1 2024-09-18T08:17:02.282 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:17:02.282 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:17:02.283 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1' 2024-09-18T08:17:02.341 DEBUG:teuthology.orchestra.run.smithi050:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid 779349ca-7593-11ef-bceb-c7b262605968 -- ceph daemon osd.5 dump_blocked_ops 2024-09-18T08:17:02.384 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7 2024-09-18T08:17:02.414 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:17:02.414 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:17:02.415 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7' 2024-09-18T08:17:02.516 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5 2024-09-18T08:17:02.540 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:17:02.541 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:17:02.541 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5' 2024-09-18T08:17:02.646 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6 2024-09-18T08:17:02.655 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:17:02 smithi050 ceph-mon[32371]: pgmap v710: 158 pgs: 1 peering, 4 active+remapped+backfilling, 2 active+remapped+backfill_toofull, 151 active+clean; 4.4 GiB data, 16 GiB used, 521 GiB / 536 GiB avail; 852 B/s rd, 0 B/s wr, 570 op/s; 17066/216648 objects misplaced (7.877%); 39 MiB/s, 621 objects/s recovering 2024-09-18T08:17:02.655 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:17:02 smithi050 ceph-mon[32371]: osdmap e369: 8 total, 8 up, 6 in 2024-09-18T08:17:02.655 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:17:02 smithi050 ceph-mon[32371]: from='mgr.24470 172.21.15.7:0/2594390261' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-18T08:17:02.671 INFO:teuthology.orchestra.run.smithi050.stderr:Inferring config /var/lib/ceph/779349ca-7593-11ef-bceb-c7b262605968/mon.b/config 2024-09-18T08:17:02.684 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:17:02.685 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:17:02.685 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6' 2024-09-18T08:17:02.730 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:17:02 smithi007 ceph-mon[29459]: pgmap v710: 158 pgs: 1 peering, 4 active+remapped+backfilling, 2 active+remapped+backfill_toofull, 151 active+clean; 4.4 GiB data, 16 GiB used, 521 GiB / 536 GiB avail; 852 B/s rd, 0 B/s wr, 570 op/s; 17066/216648 objects misplaced (7.877%); 39 MiB/s, 621 objects/s recovering 2024-09-18T08:17:02.730 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:17:02 smithi007 ceph-mon[29459]: osdmap e369: 8 total, 8 up, 6 in 2024-09-18T08:17:02.730 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:17:02 smithi007 ceph-mon[29459]: from='mgr.24470 172.21.15.7:0/2594390261' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-18T08:17:02.730 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:17:02 smithi007 ceph-mon[39249]: pgmap v710: 158 pgs: 1 peering, 4 active+remapped+backfilling, 2 active+remapped+backfill_toofull, 151 active+clean; 4.4 GiB data, 16 GiB used, 521 GiB / 536 GiB avail; 852 B/s rd, 0 B/s wr, 570 op/s; 17066/216648 objects misplaced (7.877%); 39 MiB/s, 621 objects/s recovering 2024-09-18T08:17:02.731 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:17:02 smithi007 ceph-mon[39249]: osdmap e369: 8 total, 8 up, 6 in 2024-09-18T08:17:02.731 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:17:02 smithi007 ceph-mon[39249]: from='mgr.24470 172.21.15.7:0/2594390261' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-18T08:17:02.786 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2 2024-09-18T08:17:02.827 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:17:02.828 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:17:02.828 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2' 2024-09-18T08:17:02.929 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5 2024-09-18T08:17:02.959 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:17:02.960 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:17:02.960 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5' 2024-09-18T08:17:03.061 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3 2024-09-18T08:17:03.109 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:17:03.109 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:17:03.110 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3' 2024-09-18T08:17:03.211 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1 2024-09-18T08:17:03.257 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:17:03.258 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:17:03.258 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1' 2024-09-18T08:17:03.359 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4 2024-09-18T08:17:03.390 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:17:03.390 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:17:03.391 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4' 2024-09-18T08:17:03.442 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2024-09-18T08:17:03.442 INFO:teuthology.orchestra.run.smithi050.stdout: "ops": [], 2024-09-18T08:17:03.442 INFO:teuthology.orchestra.run.smithi050.stdout: "complaint_time": 30, 2024-09-18T08:17:03.442 INFO:teuthology.orchestra.run.smithi050.stdout: "num_blocked_ops": 0 2024-09-18T08:17:03.442 INFO:teuthology.orchestra.run.smithi050.stdout:} 2024-09-18T08:17:03.492 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5 2024-09-18T08:17:03.520 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:17:03.520 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:17:03.521 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5' 2024-09-18T08:17:03.622 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3 2024-09-18T08:17:03.666 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:17:03.666 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:17:03.667 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3' 2024-09-18T08:17:03.768 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2 2024-09-18T08:17:03.795 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:17:03.795 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:17:03.795 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2' 2024-09-18T08:17:03.896 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7 2024-09-18T08:17:03.921 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:17:03.921 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:17:03.922 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7' 2024-09-18T08:17:04.022 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5 2024-09-18T08:17:04.050 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:17:04.051 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:17:04.051 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5' 2024-09-18T08:17:04.152 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1 2024-09-18T08:17:04.196 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:17:04.196 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:17:04.196 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1' 2024-09-18T08:17:04.297 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5 2024-09-18T08:17:04.324 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:17:04.324 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:17:04.325 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5' 2024-09-18T08:17:04.425 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2 2024-09-18T08:17:04.455 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:17:04.456 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:17:04.456 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2' 2024-09-18T08:17:04.498 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:17:04 smithi050 ceph-mon[32371]: pgmap v712: 158 pgs: 4 active+remapped+backfilling, 2 active+remapped+backfill_toofull, 152 active+clean; 4.3 GiB data, 15 GiB used, 521 GiB / 536 GiB avail; 1.7 KiB/s rd, 0 B/s wr, 703 op/s; 16823/214284 objects misplaced (7.851%); 39 MiB/s, 621 objects/s recovering 2024-09-18T08:17:04.498 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:17:04 smithi050 ceph-mon[32371]: from='mgr.24470 172.21.15.7:0/2594390261' entity='mgr.y' 2024-09-18T08:17:04.498 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:17:04 smithi050 ceph-mon[32371]: from='mgr.24470 172.21.15.7:0/2594390261' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-18T08:17:04.528 DEBUG:teuthology.orchestra.run.smithi050:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid 779349ca-7593-11ef-bceb-c7b262605968 -- ceph daemon osd.5 dump_historic_ops 2024-09-18T08:17:04.558 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2 2024-09-18T08:17:04.585 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:17:04.586 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:17:04.586 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2' 2024-09-18T08:17:04.687 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7 2024-09-18T08:17:04.714 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:17:04.715 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:17:04.715 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7' 2024-09-18T08:17:04.730 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:17:04 smithi007 ceph-mon[29459]: pgmap v712: 158 pgs: 4 active+remapped+backfilling, 2 active+remapped+backfill_toofull, 152 active+clean; 4.3 GiB data, 15 GiB used, 521 GiB / 536 GiB avail; 1.7 KiB/s rd, 0 B/s wr, 703 op/s; 16823/214284 objects misplaced (7.851%); 39 MiB/s, 621 objects/s recovering 2024-09-18T08:17:04.730 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:17:04 smithi007 ceph-mon[29459]: from='mgr.24470 172.21.15.7:0/2594390261' entity='mgr.y' 2024-09-18T08:17:04.730 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:17:04 smithi007 ceph-mon[29459]: from='mgr.24470 172.21.15.7:0/2594390261' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-18T08:17:04.730 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:17:04 smithi007 ceph-mon[39249]: pgmap v712: 158 pgs: 4 active+remapped+backfilling, 2 active+remapped+backfill_toofull, 152 active+clean; 4.3 GiB data, 15 GiB used, 521 GiB / 536 GiB avail; 1.7 KiB/s rd, 0 B/s wr, 703 op/s; 16823/214284 objects misplaced (7.851%); 39 MiB/s, 621 objects/s recovering 2024-09-18T08:17:04.730 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:17:04 smithi007 ceph-mon[39249]: from='mgr.24470 172.21.15.7:0/2594390261' entity='mgr.y' 2024-09-18T08:17:04.730 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:17:04 smithi007 ceph-mon[39249]: from='mgr.24470 172.21.15.7:0/2594390261' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-18T08:17:04.816 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0 2024-09-18T08:17:04.857 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:17:04.857 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:17:04.858 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0' 2024-09-18T08:17:04.874 INFO:teuthology.orchestra.run.smithi050.stderr:Inferring config /var/lib/ceph/779349ca-7593-11ef-bceb-c7b262605968/mon.b/config 2024-09-18T08:17:04.959 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1 2024-09-18T08:17:05.005 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:17:05.005 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:17:05.006 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1' 2024-09-18T08:17:05.106 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4 2024-09-18T08:17:05.136 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:17:05.137 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:17:05.137 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4' 2024-09-18T08:17:05.238 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7 2024-09-18T08:17:05.269 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:17:05.270 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:17:05.270 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7' 2024-09-18T08:17:05.371 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7 2024-09-18T08:17:05.397 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:17:05.398 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:17:05.398 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7' 2024-09-18T08:17:05.499 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7 2024-09-18T08:17:05.525 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:17:05.526 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:17:05.526 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7' 2024-09-18T08:17:05.627 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0 2024-09-18T08:17:05.656 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:17:05.657 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:17:05.658 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0' 2024-09-18T08:17:05.717 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:17:05 smithi050 ceph-mon[32371]: 4.1d scrub starts 2024-09-18T08:17:05.718 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:17:05 smithi050 ceph-mon[32371]: 4.1d scrub ok 2024-09-18T08:17:05.730 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:17:05 smithi007 ceph-mon[29459]: 4.1d scrub starts 2024-09-18T08:17:05.730 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:17:05 smithi007 ceph-mon[29459]: 4.1d scrub ok 2024-09-18T08:17:05.730 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:17:05 smithi007 ceph-mon[39249]: 4.1d scrub starts 2024-09-18T08:17:05.730 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:17:05 smithi007 ceph-mon[39249]: 4.1d scrub ok 2024-09-18T08:17:05.759 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1 2024-09-18T08:17:05.785 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:17:05.786 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:17:05.786 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1' 2024-09-18T08:17:05.887 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2 2024-09-18T08:17:05.915 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:17:05.915 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:17:05.916 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2' 2024-09-18T08:17:06.017 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4 2024-09-18T08:17:06.041 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:17:06.042 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:17:06.042 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4' 2024-09-18T08:17:06.089 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2024-09-18T08:17:06.089 INFO:teuthology.orchestra.run.smithi050.stdout: "size": 20, 2024-09-18T08:17:06.090 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 600, 2024-09-18T08:17:06.090 INFO:teuthology.orchestra.run.smithi050.stdout: "ops": [ 2024-09-18T08:17:06.090 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:17:06.090 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.15216.0:22610 8.4 8:2876ee9f:::benchmark_data_smithi007_101583_object22609:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e174)", 2024-09-18T08:17:06.090 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-09-18T08:08:08.437530+0000", 2024-09-18T08:17:06.090 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 537.589463593, 2024-09-18T08:17:06.090 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.7699730649999998, 2024-09-18T08:17:06.090 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-09-18T08:17:06.090 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-09-18T08:17:06.090 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "reached pg", 2024-09-18T08:17:06.090 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2024-09-18T08:17:06.090 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.15216", 2024-09-18T08:17:06.090 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "172.21.15.7:0/2596041177", 2024-09-18T08:17:06.091 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 22610 2024-09-18T08:17:06.091 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:17:06.091 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-09-18T08:17:06.091 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:17:06.091 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-09-18T08:17:06.091 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:08:08.437530+0000", 2024-09-18T08:17:06.091 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-18T08:17:06.091 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:17:06.091 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:17:06.091 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-09-18T08:17:06.091 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:08:08.437530+0000", 2024-09-18T08:17:06.091 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-18T08:17:06.091 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:17:06.092 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:17:06.092 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-09-18T08:17:06.092 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:08:08.437531+0000", 2024-09-18T08:17:06.092 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.0890000000000001e-06 2024-09-18T08:17:06.092 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:17:06.092 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:17:06.092 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-09-18T08:17:06.092 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:08:08.437611+0000", 2024-09-18T08:17:06.092 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 7.9339999999999996e-05 2024-09-18T08:17:06.092 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:17:06.092 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:17:06.092 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-09-18T08:17:06.092 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:08:08.437623+0000", 2024-09-18T08:17:06.093 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.2442999999999999e-05 2024-09-18T08:17:06.093 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:17:06.093 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:17:06.093 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-09-18T08:17:06.093 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:08:08.437631+0000", 2024-09-18T08:17:06.093 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 8.0420000000000003e-06 2024-09-18T08:17:06.093 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:17:06.093 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:17:06.093 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-18T08:17:06.093 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:08:08.437684+0000", 2024-09-18T08:17:06.093 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 5.3430000000000002e-05 2024-09-18T08:17:06.093 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:17:06.093 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:17:06.094 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-09-18T08:17:06.094 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:08:08.437818+0000", 2024-09-18T08:17:06.094 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00013359400000000001 2024-09-18T08:17:06.094 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:17:06.094 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:17:06.094 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "waiting for subops from 2,6", 2024-09-18T08:17:06.094 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:08:08.438886+0000", 2024-09-18T08:17:06.094 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0010681 2024-09-18T08:17:06.094 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:17:06.094 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:17:06.094 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "op_commit", 2024-09-18T08:17:06.094 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:08:08.442178+0000", 2024-09-18T08:17:06.094 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0032919889999999999 2024-09-18T08:17:06.094 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:17:06.095 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:17:06.095 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-18T08:17:06.095 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:08:08.447653+0000", 2024-09-18T08:17:06.095 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0054751269999999998 2024-09-18T08:17:06.095 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:17:06.095 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:17:06.095 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "waiting for peered", 2024-09-18T08:17:06.095 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:08:08.447695+0000", 2024-09-18T08:17:06.095 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4.1869999999999997e-05 2024-09-18T08:17:06.095 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:17:06.095 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:17:06.095 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-18T08:17:06.095 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:08:09.153169+0000", 2024-09-18T08:17:06.096 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.70547423099999995 2024-09-18T08:17:06.096 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:17:06.096 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:17:06.096 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "waiting for peered", 2024-09-18T08:17:06.096 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:08:09.153246+0000", 2024-09-18T08:17:06.096 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 7.6285000000000006e-05 2024-09-18T08:17:06.096 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:17:06.096 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:17:06.096 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-18T08:17:06.096 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:08:10.207423+0000", 2024-09-18T08:17:06.096 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.0541769270000001 2024-09-18T08:17:06.096 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:17:06.097 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:17:06.097 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-09-18T08:17:06.097 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:08:10.207503+0000", 2024-09-18T08:17:06.097 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 8.0598000000000003e-05 2024-09-18T08:17:06.097 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-18T08:17:06.097 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-09-18T08:17:06.097 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-18T08:17:06.097 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:17:06.097 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:17:06.097 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.15216.0:22651 8.4 8:2e29b373:::benchmark_data_smithi007_101583_object22650:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e175)", 2024-09-18T08:17:06.097 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-09-18T08:08:08.478683+0000", 2024-09-18T08:17:06.097 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 537.54831113600005, 2024-09-18T08:17:06.097 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.7382695699999999, 2024-09-18T08:17:06.098 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-09-18T08:17:06.098 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-09-18T08:17:06.098 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T08:17:06.098 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2024-09-18T08:17:06.098 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.15216", 2024-09-18T08:17:06.098 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "172.21.15.7:0/2596041177", 2024-09-18T08:17:06.098 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 22651 2024-09-18T08:17:06.098 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:17:06.098 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-09-18T08:17:06.098 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:17:06.098 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-09-18T08:17:06.098 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:08:08.478683+0000", 2024-09-18T08:17:06.098 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-18T08:17:06.099 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:17:06.099 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:17:06.099 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-09-18T08:17:06.099 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:08:08.478683+0000", 2024-09-18T08:17:06.099 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-18T08:17:06.099 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:17:06.099 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:17:06.099 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-09-18T08:17:06.099 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:08:08.478683+0000", 2024-09-18T08:17:06.099 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 6.3099999999999997e-07 2024-09-18T08:17:06.099 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:17:06.099 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:17:06.099 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-09-18T08:17:06.100 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:08:08.478752+0000", 2024-09-18T08:17:06.100 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 6.9239999999999994e-05 2024-09-18T08:17:06.100 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:17:06.100 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:17:06.100 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-09-18T08:17:06.100 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:08:08.478761+0000", 2024-09-18T08:17:06.100 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 8.7979999999999995e-06 2024-09-18T08:17:06.100 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:17:06.100 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:17:06.100 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-09-18T08:17:06.100 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:08:08.478768+0000", 2024-09-18T08:17:06.100 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 6.7680000000000003e-06 2024-09-18T08:17:06.100 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:17:06.100 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:17:06.101 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-18T08:17:06.101 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:08:08.478835+0000", 2024-09-18T08:17:06.101 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 6.6914000000000003e-05 2024-09-18T08:17:06.101 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:17:06.101 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:17:06.101 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "waiting for peered", 2024-09-18T08:17:06.101 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:08:08.478845+0000", 2024-09-18T08:17:06.101 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 9.9969999999999993e-06 2024-09-18T08:17:06.101 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:17:06.101 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:17:06.101 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-18T08:17:06.101 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:08:09.155047+0000", 2024-09-18T08:17:06.101 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.67620250999999998 2024-09-18T08:17:06.101 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:17:06.102 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:17:06.102 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "waiting for peered", 2024-09-18T08:17:06.102 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:08:09.155057+0000", 2024-09-18T08:17:06.102 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 9.2739999999999996e-06 2024-09-18T08:17:06.102 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:17:06.102 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:17:06.102 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-18T08:17:06.102 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:08:10.208122+0000", 2024-09-18T08:17:06.102 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.0530651179999999 2024-09-18T08:17:06.102 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:17:06.102 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:17:06.102 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-09-18T08:17:06.102 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:08:10.208293+0000", 2024-09-18T08:17:06.102 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00017156699999999999 2024-09-18T08:17:06.103 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:17:06.103 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:17:06.103 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "waiting for subops from 3,6", 2024-09-18T08:17:06.103 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:08:10.208594+0000", 2024-09-18T08:17:06.103 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00030062200000000001 2024-09-18T08:17:06.103 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:17:06.103 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:17:06.103 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:17:06.103 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:08:10.213477+0000", 2024-09-18T08:17:06.103 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0048826920000000001 2024-09-18T08:17:06.103 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:17:06.104 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:17:06.104 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "op_commit", 2024-09-18T08:17:06.104 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:08:10.213660+0000", 2024-09-18T08:17:06.104 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00018310599999999999 2024-09-18T08:17:06.104 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:17:06.104 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:17:06.104 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:17:06.104 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:08:10.215742+0000", 2024-09-18T08:17:06.104 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0020817399999999999 2024-09-18T08:17:06.104 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:17:06.104 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:17:06.104 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "commit_sent", 2024-09-18T08:17:06.105 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:08:10.215809+0000", 2024-09-18T08:17:06.105 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 6.7220999999999999e-05 2024-09-18T08:17:06.105 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:17:06.105 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:17:06.105 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-09-18T08:17:06.105 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:08:10.216952+0000", 2024-09-18T08:17:06.105 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0011433719999999999 2024-09-18T08:17:06.105 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-18T08:17:06.105 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-09-18T08:17:06.105 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-18T08:17:06.105 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:17:06.105 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:17:06.105 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.15216.0:22653 8.4 8:28542a20:::benchmark_data_smithi007_101583_object22652:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e175)", 2024-09-18T08:17:06.106 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-09-18T08:08:08.478788+0000", 2024-09-18T08:17:06.106 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 537.54820607500005, 2024-09-18T08:17:06.106 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.7390986750000001, 2024-09-18T08:17:06.106 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-09-18T08:17:06.106 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-09-18T08:17:06.106 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T08:17:06.106 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2024-09-18T08:17:06.106 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.15216", 2024-09-18T08:17:06.106 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "172.21.15.7:0/2596041177", 2024-09-18T08:17:06.106 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 22653 2024-09-18T08:17:06.106 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:17:06.106 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-09-18T08:17:06.106 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:17:06.107 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-09-18T08:17:06.107 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:08:08.478788+0000", 2024-09-18T08:17:06.107 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-18T08:17:06.107 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:17:06.107 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:17:06.107 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-09-18T08:17:06.107 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:08:08.478788+0000", 2024-09-18T08:17:06.107 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-18T08:17:06.107 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:17:06.107 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:17:06.107 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-09-18T08:17:06.107 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:08:08.478788+0000", 2024-09-18T08:17:06.108 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 8.2600000000000001e-07 2024-09-18T08:17:06.108 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:17:06.108 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:17:06.108 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-09-18T08:17:06.108 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:08:08.478822+0000", 2024-09-18T08:17:06.108 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 3.3343999999999998e-05 2024-09-18T08:17:06.108 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:17:06.108 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:17:06.108 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-09-18T08:17:06.108 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:08:08.478830+0000", 2024-09-18T08:17:06.108 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 8.4549999999999995e-06 2024-09-18T08:17:06.108 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:17:06.109 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:17:06.109 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-09-18T08:17:06.109 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:08:08.478837+0000", 2024-09-18T08:17:06.109 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 6.3199999999999996e-06 2024-09-18T08:17:06.109 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:17:06.109 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:17:06.109 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-18T08:17:06.109 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:08:08.478877+0000", 2024-09-18T08:17:06.109 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4.0157999999999997e-05 2024-09-18T08:17:06.109 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:17:06.109 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:17:06.109 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "waiting for peered", 2024-09-18T08:17:06.109 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:08:08.478882+0000", 2024-09-18T08:17:06.110 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 5.0520000000000001e-06 2024-09-18T08:17:06.110 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:17:06.110 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:17:06.110 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-18T08:17:06.110 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:08:09.155087+0000", 2024-09-18T08:17:06.110 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.67620514899999995 2024-09-18T08:17:06.110 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:17:06.110 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:17:06.110 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "waiting for peered", 2024-09-18T08:17:06.110 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:08:09.155095+0000", 2024-09-18T08:17:06.110 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 8.4039999999999995e-06 2024-09-18T08:17:06.110 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:17:06.110 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:17:06.110 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-18T08:17:06.111 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:08:10.209614+0000", 2024-09-18T08:17:06.111 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.0545187579999999 2024-09-18T08:17:06.111 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:17:06.111 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:17:06.111 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-09-18T08:17:06.111 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:08:10.209770+0000", 2024-09-18T08:17:06.111 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00015591699999999999 2024-09-18T08:17:06.111 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:17:06.111 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:17:06.111 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "waiting for subops from 3,6", 2024-09-18T08:17:06.111 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:08:10.209987+0000", 2024-09-18T08:17:06.111 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000216565 2024-09-18T08:17:06.111 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:17:06.111 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:17:06.112 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:17:06.112 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:08:10.213551+0000", 2024-09-18T08:17:06.112 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0035644280000000001 2024-09-18T08:17:06.112 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:17:06.112 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:17:06.112 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "op_commit", 2024-09-18T08:17:06.112 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:08:10.213670+0000", 2024-09-18T08:17:06.112 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00011882700000000001 2024-09-18T08:17:06.112 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:17:06.112 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:17:06.112 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:17:06.112 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:08:10.217201+0000", 2024-09-18T08:17:06.113 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.003531307 2024-09-18T08:17:06.113 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:17:06.113 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:17:06.113 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "commit_sent", 2024-09-18T08:17:06.113 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:08:10.217269+0000", 2024-09-18T08:17:06.113 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 6.7539999999999994e-05 2024-09-18T08:17:06.113 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:17:06.113 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:17:06.113 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-09-18T08:17:06.113 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:08:10.217886+0000", 2024-09-18T08:17:06.113 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00061762500000000001 2024-09-18T08:17:06.113 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-18T08:17:06.113 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-09-18T08:17:06.114 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-18T08:17:06.114 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:17:06.114 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:17:06.114 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.15216.0:22670 8.4 8:2c79527e:::benchmark_data_smithi007_101583_object22669:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e175)", 2024-09-18T08:17:06.114 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-09-18T08:08:08.488620+0000", 2024-09-18T08:17:06.114 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 537.53837421499998, 2024-09-18T08:17:06.114 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.7300631360000001, 2024-09-18T08:17:06.114 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-09-18T08:17:06.114 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-09-18T08:17:06.114 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T08:17:06.114 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2024-09-18T08:17:06.114 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.15216", 2024-09-18T08:17:06.114 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "172.21.15.7:0/2596041177", 2024-09-18T08:17:06.114 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 22670 2024-09-18T08:17:06.114 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:17:06.115 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-09-18T08:17:06.115 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:17:06.115 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-09-18T08:17:06.115 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:08:08.488620+0000", 2024-09-18T08:17:06.115 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-18T08:17:06.115 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:17:06.115 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:17:06.115 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-09-18T08:17:06.115 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:08:08.488620+0000", 2024-09-18T08:17:06.115 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-18T08:17:06.115 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:17:06.115 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:17:06.115 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-09-18T08:17:06.116 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:08:08.488621+0000", 2024-09-18T08:17:06.116 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.3820000000000001e-06 2024-09-18T08:17:06.116 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:17:06.116 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:17:06.116 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-09-18T08:17:06.116 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:08:08.488653+0000", 2024-09-18T08:17:06.116 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 3.1826999999999999e-05 2024-09-18T08:17:06.116 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:17:06.116 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:17:06.116 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-09-18T08:17:06.116 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:08:08.488667+0000", 2024-09-18T08:17:06.116 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.4042e-05 2024-09-18T08:17:06.116 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:17:06.116 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:17:06.117 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-09-18T08:17:06.117 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:08:08.488674+0000", 2024-09-18T08:17:06.117 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 7.1570000000000003e-06 2024-09-18T08:17:06.117 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:17:06.117 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:17:06.117 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-18T08:17:06.117 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:08:08.489170+0000", 2024-09-18T08:17:06.117 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00049649999999999998 2024-09-18T08:17:06.117 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:17:06.117 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:17:06.117 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "waiting for peered", 2024-09-18T08:17:06.117 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:08:08.489181+0000", 2024-09-18T08:17:06.117 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.0991e-05 2024-09-18T08:17:06.118 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:17:06.118 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:17:06.118 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-18T08:17:06.118 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:08:09.155124+0000", 2024-09-18T08:17:06.118 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.66594271500000002 2024-09-18T08:17:06.118 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:17:06.118 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:17:06.118 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "waiting for peered", 2024-09-18T08:17:06.118 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:08:09.155132+0000", 2024-09-18T08:17:06.118 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 7.8509999999999999e-06 2024-09-18T08:17:06.118 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:17:06.118 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:17:06.118 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-18T08:17:06.119 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:08:10.212427+0000", 2024-09-18T08:17:06.119 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.0572947880000001 2024-09-18T08:17:06.119 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:17:06.119 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:17:06.119 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-09-18T08:17:06.119 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:08:10.212575+0000", 2024-09-18T08:17:06.119 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00014837199999999999 2024-09-18T08:17:06.119 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:17:06.119 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:17:06.119 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "waiting for subops from 3,6", 2024-09-18T08:17:06.119 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:08:10.212772+0000", 2024-09-18T08:17:06.119 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00019651400000000001 2024-09-18T08:17:06.120 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:17:06.120 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:17:06.120 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "op_commit", 2024-09-18T08:17:06.120 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:08:10.214660+0000", 2024-09-18T08:17:06.120 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.001887854 2024-09-18T08:17:06.120 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:17:06.120 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:17:06.120 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:17:06.120 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:08:10.217118+0000", 2024-09-18T08:17:06.120 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0024580309999999998 2024-09-18T08:17:06.120 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:17:06.120 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:17:06.120 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:17:06.121 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:08:10.218023+0000", 2024-09-18T08:17:06.121 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00090551 2024-09-18T08:17:06.121 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:17:06.121 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:17:06.121 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "commit_sent", 2024-09-18T08:17:06.121 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:08:10.218067+0000", 2024-09-18T08:17:06.121 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4.3547999999999997e-05 2024-09-18T08:17:06.121 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:17:06.121 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:17:06.121 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-09-18T08:17:06.121 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:08:10.218683+0000", 2024-09-18T08:17:06.121 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00061605399999999999 2024-09-18T08:17:06.121 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-18T08:17:06.122 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-09-18T08:17:06.122 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-18T08:17:06.122 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:17:06.122 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:17:06.122 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.15216.0:22690 8.4 8:29940999:::benchmark_data_smithi007_101583_object22689:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e175)", 2024-09-18T08:17:06.122 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-09-18T08:08:08.501955+0000", 2024-09-18T08:17:06.122 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 537.52503907300002, 2024-09-18T08:17:06.122 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.717490398, 2024-09-18T08:17:06.122 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-09-18T08:17:06.122 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-09-18T08:17:06.122 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T08:17:06.122 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2024-09-18T08:17:06.122 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.15216", 2024-09-18T08:17:06.122 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "172.21.15.7:0/2596041177", 2024-09-18T08:17:06.122 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 22690 2024-09-18T08:17:06.123 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:17:06.123 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-09-18T08:17:06.123 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:17:06.123 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-09-18T08:17:06.123 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:08:08.501955+0000", 2024-09-18T08:17:06.123 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-18T08:17:06.123 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:17:06.123 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:17:06.123 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-09-18T08:17:06.123 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:08:08.501955+0000", 2024-09-18T08:17:06.123 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-18T08:17:06.123 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:17:06.123 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:17:06.123 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-09-18T08:17:06.124 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:08:08.501956+0000", 2024-09-18T08:17:06.124 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.4190000000000001e-06 2024-09-18T08:17:06.124 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:17:06.124 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:17:06.124 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-09-18T08:17:06.124 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:08:08.502052+0000", 2024-09-18T08:17:06.124 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 9.5538999999999994e-05 2024-09-18T08:17:06.124 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:17:06.124 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:17:06.124 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-09-18T08:17:06.124 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:08:08.502067+0000", 2024-09-18T08:17:06.124 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.5014999999999999e-05 2024-09-18T08:17:06.125 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:17:06.125 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:17:06.125 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-09-18T08:17:06.125 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:08:08.502075+0000", 2024-09-18T08:17:06.125 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 8.0029999999999994e-06 2024-09-18T08:17:06.125 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:17:06.125 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:17:06.125 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-18T08:17:06.125 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:08:08.502136+0000", 2024-09-18T08:17:06.125 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 6.1029999999999997e-05 2024-09-18T08:17:06.125 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:17:06.125 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:17:06.125 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "waiting for peered", 2024-09-18T08:17:06.125 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:08:08.502142+0000", 2024-09-18T08:17:06.126 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 6.5749999999999997e-06 2024-09-18T08:17:06.126 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:17:06.126 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:17:06.126 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-18T08:17:06.126 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:08:09.155157+0000", 2024-09-18T08:17:06.126 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.65301457600000001 2024-09-18T08:17:06.126 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:17:06.126 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:17:06.126 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "waiting for peered", 2024-09-18T08:17:06.126 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:08:09.155165+0000", 2024-09-18T08:17:06.126 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 7.7889999999999995e-06 2024-09-18T08:17:06.126 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:17:06.126 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:17:06.127 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-18T08:17:06.127 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:08:10.213708+0000", 2024-09-18T08:17:06.127 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.0585432850000001 2024-09-18T08:17:06.127 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:17:06.127 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:17:06.127 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-09-18T08:17:06.127 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:08:10.213852+0000", 2024-09-18T08:17:06.127 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00014445899999999999 2024-09-18T08:17:06.127 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:17:06.127 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:17:06.127 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "waiting for subops from 3,6", 2024-09-18T08:17:06.127 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:08:10.214051+0000", 2024-09-18T08:17:06.127 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000198758 2024-09-18T08:17:06.127 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:17:06.128 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:17:06.128 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "op_commit", 2024-09-18T08:17:06.128 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:08:10.217140+0000", 2024-09-18T08:17:06.128 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0030884929999999999 2024-09-18T08:17:06.128 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:17:06.128 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:17:06.128 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:17:06.128 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:08:10.217957+0000", 2024-09-18T08:17:06.128 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00081764700000000001 2024-09-18T08:17:06.128 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:17:06.128 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:17:06.128 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:17:06.128 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:08:10.218885+0000", 2024-09-18T08:17:06.129 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00092746099999999998 2024-09-18T08:17:06.129 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:17:06.129 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:17:06.129 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "commit_sent", 2024-09-18T08:17:06.129 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:08:10.218945+0000", 2024-09-18T08:17:06.129 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 6.0149000000000003e-05 2024-09-18T08:17:06.129 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:17:06.129 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:17:06.129 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-09-18T08:17:06.129 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:08:10.219445+0000", 2024-09-18T08:17:06.129 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00050020000000000002 2024-09-18T08:17:06.129 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-18T08:17:06.129 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-09-18T08:17:06.130 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-18T08:17:06.130 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:17:06.130 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:17:06.130 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.15216.0:22694 8.4 8:285f3aa9:::benchmark_data_smithi007_101583_object22693:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e175)", 2024-09-18T08:17:06.130 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-09-18T08:08:08.509228+0000", 2024-09-18T08:17:06.130 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 537.51776616999996, 2024-09-18T08:17:06.130 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.710799596, 2024-09-18T08:17:06.130 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-09-18T08:17:06.130 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-09-18T08:17:06.130 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T08:17:06.130 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2024-09-18T08:17:06.130 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.15216", 2024-09-18T08:17:06.130 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "172.21.15.7:0/2596041177", 2024-09-18T08:17:06.131 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 22694 2024-09-18T08:17:06.131 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:17:06.131 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-09-18T08:17:06.131 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:17:06.131 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-09-18T08:17:06.131 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:08:08.509228+0000", 2024-09-18T08:17:06.131 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-18T08:17:06.131 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:17:06.131 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:17:06.131 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-09-18T08:17:06.131 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:08:08.509228+0000", 2024-09-18T08:17:06.131 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-18T08:17:06.131 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:17:06.132 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:17:06.132 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-09-18T08:17:06.132 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:08:08.509229+0000", 2024-09-18T08:17:06.132 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.6640000000000001e-06 2024-09-18T08:17:06.132 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:17:06.132 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:17:06.132 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-09-18T08:17:06.132 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:08:08.509266+0000", 2024-09-18T08:17:06.132 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 3.6563000000000002e-05 2024-09-18T08:17:06.132 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:17:06.132 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:17:06.132 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-09-18T08:17:06.132 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:08:08.509280+0000", 2024-09-18T08:17:06.132 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.4272e-05 2024-09-18T08:17:06.133 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:17:06.133 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:17:06.133 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-09-18T08:17:06.133 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:08:08.509288+0000", 2024-09-18T08:17:06.133 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 8.439e-06 2024-09-18T08:17:06.133 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:17:06.133 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:17:06.133 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-18T08:17:06.133 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:08:08.509344+0000", 2024-09-18T08:17:06.133 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 5.5538000000000003e-05 2024-09-18T08:17:06.133 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:17:06.133 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:17:06.133 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "waiting for peered", 2024-09-18T08:17:06.134 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:08:08.509352+0000", 2024-09-18T08:17:06.134 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 7.5329999999999997e-06 2024-09-18T08:17:06.134 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:17:06.134 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:17:06.134 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-18T08:17:06.134 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:08:09.155189+0000", 2024-09-18T08:17:06.134 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.64583694300000005 2024-09-18T08:17:06.134 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:17:06.134 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:17:06.134 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "waiting for peered", 2024-09-18T08:17:06.134 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:08:09.155196+0000", 2024-09-18T08:17:06.134 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 7.6760000000000007e-06 2024-09-18T08:17:06.134 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:17:06.135 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:17:06.135 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-18T08:17:06.135 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:08:10.214718+0000", 2024-09-18T08:17:06.135 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.059522045 2024-09-18T08:17:06.135 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:17:06.135 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:17:06.135 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-09-18T08:17:06.135 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:08:10.214859+0000", 2024-09-18T08:17:06.135 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000140922 2024-09-18T08:17:06.135 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:17:06.135 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:17:06.135 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "waiting for subops from 3,6", 2024-09-18T08:17:06.135 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:08:10.215042+0000", 2024-09-18T08:17:06.136 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000183141 2024-09-18T08:17:06.136 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:17:06.136 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:17:06.136 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "op_commit", 2024-09-18T08:17:06.136 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:08:10.218698+0000", 2024-09-18T08:17:06.136 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.003655845 2024-09-18T08:17:06.136 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:17:06.136 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:17:06.136 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:17:06.136 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:08:10.218758+0000", 2024-09-18T08:17:06.136 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 5.9463999999999999e-05 2024-09-18T08:17:06.136 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:17:06.136 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:17:06.137 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:17:06.137 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:08:10.219503+0000", 2024-09-18T08:17:06.137 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00074575399999999995 2024-09-18T08:17:06.137 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:17:06.137 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:17:06.137 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "commit_sent", 2024-09-18T08:17:06.137 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:08:10.219563+0000", 2024-09-18T08:17:06.137 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 5.9209999999999997e-05 2024-09-18T08:17:06.137 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:17:06.137 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:17:06.137 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-09-18T08:17:06.137 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:08:10.220027+0000", 2024-09-18T08:17:06.137 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00046458699999999998 2024-09-18T08:17:06.138 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-18T08:17:06.138 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-09-18T08:17:06.138 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-18T08:17:06.138 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:17:06.138 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:17:06.138 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.15216.0:69189 8.4 8:298d5120:::benchmark_data_smithi007_101583_object69188:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e198)", 2024-09-18T08:17:06.138 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-09-18T08:08:51.700999+0000", 2024-09-18T08:17:06.138 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 494.325994503, 2024-09-18T08:17:06.138 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 2.1566947330000001, 2024-09-18T08:17:06.138 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-09-18T08:17:06.138 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-09-18T08:17:06.138 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T08:17:06.138 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2024-09-18T08:17:06.139 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.15216", 2024-09-18T08:17:06.139 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "172.21.15.7:0/2596041177", 2024-09-18T08:17:06.139 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 69189 2024-09-18T08:17:06.139 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:17:06.139 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-09-18T08:17:06.139 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:17:06.139 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-09-18T08:17:06.139 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:08:51.700999+0000", 2024-09-18T08:17:06.139 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-18T08:17:06.139 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:17:06.139 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:17:06.139 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-09-18T08:17:06.139 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:08:51.700999+0000", 2024-09-18T08:17:06.140 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-18T08:17:06.140 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:17:06.140 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:17:06.140 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-09-18T08:17:06.140 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:08:51.701001+0000", 2024-09-18T08:17:06.140 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.9889999999999999e-06 2024-09-18T08:17:06.140 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:17:06.140 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:17:06.140 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-09-18T08:17:06.140 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:08:51.701054+0000", 2024-09-18T08:17:06.140 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 5.3019999999999997e-05 2024-09-18T08:17:06.140 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:17:06.140 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:17:06.141 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-09-18T08:17:06.141 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:08:51.701066+0000", 2024-09-18T08:17:06.141 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.204e-05 2024-09-18T08:17:06.141 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:17:06.141 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:17:06.141 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-09-18T08:17:06.141 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:08:51.701073+0000", 2024-09-18T08:17:06.141 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 6.5060000000000001e-06 2024-09-18T08:17:06.141 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:17:06.141 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:17:06.141 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-18T08:17:06.141 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:08:51.701115+0000", 2024-09-18T08:17:06.141 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4.2219999999999999e-05 2024-09-18T08:17:06.142 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:17:06.142 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:17:06.142 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "waiting for peered", 2024-09-18T08:17:06.142 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:08:51.701123+0000", 2024-09-18T08:17:06.142 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 8.3790000000000006e-06 2024-09-18T08:17:06.142 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:17:06.142 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:17:06.142 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-18T08:17:06.142 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:08:52.682893+0000", 2024-09-18T08:17:06.142 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.981770001 2024-09-18T08:17:06.142 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:17:06.142 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:17:06.142 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "waiting for peered", 2024-09-18T08:17:06.143 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:08:52.682917+0000", 2024-09-18T08:17:06.143 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 2.3329999999999999e-05 2024-09-18T08:17:06.143 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:17:06.143 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:17:06.143 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-18T08:17:06.143 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:08:53.834777+0000", 2024-09-18T08:17:06.143 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.151860171 2024-09-18T08:17:06.143 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:17:06.143 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:17:06.143 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-09-18T08:17:06.143 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:08:53.834944+0000", 2024-09-18T08:17:06.143 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00016724700000000001 2024-09-18T08:17:06.143 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:17:06.143 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:17:06.144 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "waiting for subops from 2,4", 2024-09-18T08:17:06.144 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:08:53.835122+0000", 2024-09-18T08:17:06.144 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00017796399999999999 2024-09-18T08:17:06.144 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:17:06.144 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:17:06.144 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:17:06.144 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:08:53.840827+0000", 2024-09-18T08:17:06.144 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0057047570000000004 2024-09-18T08:17:06.144 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:17:06.144 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:17:06.145 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "op_commit", 2024-09-18T08:17:06.145 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:08:53.840849+0000", 2024-09-18T08:17:06.145 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 2.1985e-05 2024-09-18T08:17:06.145 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:17:06.145 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:17:06.145 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:17:06.145 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:08:53.855347+0000", 2024-09-18T08:17:06.145 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.014498603000000001 2024-09-18T08:17:06.145 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:17:06.145 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:17:06.145 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "commit_sent", 2024-09-18T08:17:06.145 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:08:53.855403+0000", 2024-09-18T08:17:06.146 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 5.5977999999999998e-05 2024-09-18T08:17:06.146 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:17:06.146 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:17:06.146 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-09-18T08:17:06.146 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:08:53.857694+0000", 2024-09-18T08:17:06.146 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0022905429999999999 2024-09-18T08:17:06.146 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-18T08:17:06.146 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-09-18T08:17:06.146 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-18T08:17:06.146 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:17:06.146 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:17:06.146 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.15216.0:69191 8.4 8:2175794a:::benchmark_data_smithi007_101583_object69190:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e198)", 2024-09-18T08:17:06.146 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-09-18T08:08:51.705131+0000", 2024-09-18T08:17:06.146 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 494.32186301500002, 2024-09-18T08:17:06.147 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 2.1552133659999999, 2024-09-18T08:17:06.147 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-09-18T08:17:06.147 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-09-18T08:17:06.147 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T08:17:06.147 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2024-09-18T08:17:06.147 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.15216", 2024-09-18T08:17:06.147 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "172.21.15.7:0/2596041177", 2024-09-18T08:17:06.147 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 69191 2024-09-18T08:17:06.147 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:17:06.147 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-09-18T08:17:06.147 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:17:06.147 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-09-18T08:17:06.147 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:08:51.705131+0000", 2024-09-18T08:17:06.148 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-18T08:17:06.148 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:17:06.148 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:17:06.148 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-09-18T08:17:06.148 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:08:51.705131+0000", 2024-09-18T08:17:06.148 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-18T08:17:06.148 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:17:06.148 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:17:06.148 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-09-18T08:17:06.148 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:08:51.705138+0000", 2024-09-18T08:17:06.148 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 7.4560000000000003e-06 2024-09-18T08:17:06.148 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:17:06.148 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:17:06.149 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-09-18T08:17:06.149 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:08:51.705169+0000", 2024-09-18T08:17:06.149 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 3.1266999999999997e-05 2024-09-18T08:17:06.149 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:17:06.149 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:17:06.149 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-09-18T08:17:06.149 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:08:51.705181+0000", 2024-09-18T08:17:06.149 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.1610000000000001e-05 2024-09-18T08:17:06.149 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:17:06.149 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:17:06.149 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-09-18T08:17:06.149 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:08:51.705187+0000", 2024-09-18T08:17:06.149 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 5.5639999999999996e-06 2024-09-18T08:17:06.150 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:17:06.150 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:17:06.150 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-18T08:17:06.150 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:08:51.705248+0000", 2024-09-18T08:17:06.150 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 6.1258000000000002e-05 2024-09-18T08:17:06.150 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:17:06.150 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:17:06.150 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "waiting for peered", 2024-09-18T08:17:06.150 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:08:51.705260+0000", 2024-09-18T08:17:06.150 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.2305e-05 2024-09-18T08:17:06.150 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:17:06.150 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:17:06.150 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-18T08:17:06.151 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:08:52.683701+0000", 2024-09-18T08:17:06.151 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.97844060799999999 2024-09-18T08:17:06.151 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:17:06.151 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:17:06.151 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "waiting for peered", 2024-09-18T08:17:06.151 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:08:52.683710+0000", 2024-09-18T08:17:06.151 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 9.6760000000000002e-06 2024-09-18T08:17:06.151 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:17:06.151 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:17:06.151 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-18T08:17:06.151 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:08:53.838687+0000", 2024-09-18T08:17:06.151 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.1549762699999999 2024-09-18T08:17:06.151 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:17:06.151 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:17:06.151 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-09-18T08:17:06.152 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:08:53.838837+0000", 2024-09-18T08:17:06.152 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00015000300000000001 2024-09-18T08:17:06.152 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:17:06.152 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:17:06.152 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "waiting for subops from 2,4", 2024-09-18T08:17:06.152 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:08:53.838962+0000", 2024-09-18T08:17:06.152 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00012499200000000001 2024-09-18T08:17:06.152 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:17:06.152 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:17:06.152 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "op_commit", 2024-09-18T08:17:06.152 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:08:53.843446+0000", 2024-09-18T08:17:06.152 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0044846399999999998 2024-09-18T08:17:06.152 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:17:06.152 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:17:06.153 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:17:06.153 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:08:53.843632+0000", 2024-09-18T08:17:06.153 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00018546 2024-09-18T08:17:06.153 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:17:06.153 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:17:06.153 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:17:06.153 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:08:53.858138+0000", 2024-09-18T08:17:06.153 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.014506204 2024-09-18T08:17:06.153 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:17:06.153 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:17:06.153 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "commit_sent", 2024-09-18T08:17:06.153 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:08:53.858201+0000", 2024-09-18T08:17:06.153 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 6.2593000000000002e-05 2024-09-18T08:17:06.154 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:17:06.154 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:17:06.154 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-09-18T08:17:06.154 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:08:53.860344+0000", 2024-09-18T08:17:06.154 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0021434599999999998 2024-09-18T08:17:06.154 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-18T08:17:06.154 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-09-18T08:17:06.154 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-18T08:17:06.154 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:17:06.154 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:17:06.154 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.15216.0:69225 8.4 8:23f8fd92:::benchmark_data_smithi007_101583_object69224:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e198)", 2024-09-18T08:17:06.154 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-09-18T08:08:51.739781+0000", 2024-09-18T08:17:06.154 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 494.287212334, 2024-09-18T08:17:06.154 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 2.127987402, 2024-09-18T08:17:06.154 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-09-18T08:17:06.155 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-09-18T08:17:06.155 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T08:17:06.155 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2024-09-18T08:17:06.155 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.15216", 2024-09-18T08:17:06.155 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "172.21.15.7:0/2596041177", 2024-09-18T08:17:06.155 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 69225 2024-09-18T08:17:06.155 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:17:06.155 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-09-18T08:17:06.155 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:17:06.155 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-09-18T08:17:06.155 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:08:51.739781+0000", 2024-09-18T08:17:06.155 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-18T08:17:06.155 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:17:06.155 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:17:06.156 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-09-18T08:17:06.156 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:08:51.739781+0000", 2024-09-18T08:17:06.156 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-18T08:17:06.156 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:17:06.156 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:17:06.156 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-09-18T08:17:06.156 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:08:51.739782+0000", 2024-09-18T08:17:06.156 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.0550000000000001e-06 2024-09-18T08:17:06.156 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:17:06.156 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:17:06.156 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-09-18T08:17:06.156 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:08:51.739840+0000", 2024-09-18T08:17:06.156 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 5.7058999999999997e-05 2024-09-18T08:17:06.157 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:17:06.157 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:17:06.157 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-09-18T08:17:06.157 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:08:51.739850+0000", 2024-09-18T08:17:06.157 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.0926e-05 2024-09-18T08:17:06.157 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:17:06.157 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:17:06.157 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-09-18T08:17:06.157 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:08:51.739856+0000", 2024-09-18T08:17:06.157 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 5.5079999999999996e-06 2024-09-18T08:17:06.157 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:17:06.157 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:17:06.157 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-18T08:17:06.158 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:08:51.739923+0000", 2024-09-18T08:17:06.158 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 6.7543e-05 2024-09-18T08:17:06.158 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:17:06.158 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:17:06.158 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "waiting for peered", 2024-09-18T08:17:06.158 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:08:51.739936+0000", 2024-09-18T08:17:06.158 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.2027e-05 2024-09-18T08:17:06.158 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:17:06.158 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:17:06.158 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-18T08:17:06.158 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:08:52.683740+0000", 2024-09-18T08:17:06.158 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.94380415500000003 2024-09-18T08:17:06.158 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:17:06.158 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:17:06.159 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "waiting for peered", 2024-09-18T08:17:06.159 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:08:52.683748+0000", 2024-09-18T08:17:06.159 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 7.8390000000000007e-06 2024-09-18T08:17:06.159 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:17:06.159 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:17:06.159 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-18T08:17:06.159 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:08:53.840880+0000", 2024-09-18T08:17:06.159 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.157132941 2024-09-18T08:17:06.159 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:17:06.159 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:17:06.159 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-09-18T08:17:06.159 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:08:53.841039+0000", 2024-09-18T08:17:06.159 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00015902099999999999 2024-09-18T08:17:06.159 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:17:06.160 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:17:06.160 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "waiting for subops from 2,4", 2024-09-18T08:17:06.160 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:08:53.841222+0000", 2024-09-18T08:17:06.160 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000182215 2024-09-18T08:17:06.160 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:17:06.160 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:17:06.160 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:17:06.160 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:08:53.848840+0000", 2024-09-18T08:17:06.160 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0076179430000000003 2024-09-18T08:17:06.160 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:17:06.160 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:17:06.160 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "op_commit", 2024-09-18T08:17:06.160 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:08:53.848901+0000", 2024-09-18T08:17:06.160 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 6.1829999999999996e-05 2024-09-18T08:17:06.161 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:17:06.161 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:17:06.161 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:17:06.161 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:08:53.866059+0000", 2024-09-18T08:17:06.161 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.017157466999999999 2024-09-18T08:17:06.161 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:17:06.161 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:17:06.161 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "commit_sent", 2024-09-18T08:17:06.161 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:08:53.866105+0000", 2024-09-18T08:17:06.161 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4.6094000000000003e-05 2024-09-18T08:17:06.161 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:17:06.161 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:17:06.162 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-09-18T08:17:06.162 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:08:53.867769+0000", 2024-09-18T08:17:06.162 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.001663779 2024-09-18T08:17:06.162 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-18T08:17:06.162 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-09-18T08:17:06.162 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-18T08:17:06.162 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:17:06.162 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:17:06.162 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.15216.0:69249 8.4 8:21d6a20b:::benchmark_data_smithi007_101583_object69248:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e198)", 2024-09-18T08:17:06.162 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-09-18T08:08:51.761740+0000", 2024-09-18T08:17:06.162 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 494.26525370100001, 2024-09-18T08:17:06.162 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 2.1074109980000002, 2024-09-18T08:17:06.162 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-09-18T08:17:06.162 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-09-18T08:17:06.163 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T08:17:06.163 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2024-09-18T08:17:06.163 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.15216", 2024-09-18T08:17:06.163 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "172.21.15.7:0/2596041177", 2024-09-18T08:17:06.163 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 69249 2024-09-18T08:17:06.163 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:17:06.163 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-09-18T08:17:06.163 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:17:06.163 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-09-18T08:17:06.163 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:08:51.761740+0000", 2024-09-18T08:17:06.163 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-18T08:17:06.163 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:17:06.163 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:17:06.163 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-09-18T08:17:06.164 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:08:51.761740+0000", 2024-09-18T08:17:06.164 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-18T08:17:06.164 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:17:06.164 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:17:06.164 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-09-18T08:17:06.164 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:08:51.761741+0000", 2024-09-18T08:17:06.164 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.051e-06 2024-09-18T08:17:06.164 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:17:06.164 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:17:06.164 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-09-18T08:17:06.164 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:08:51.761813+0000", 2024-09-18T08:17:06.164 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 7.1748000000000005e-05 2024-09-18T08:17:06.165 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:17:06.165 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:17:06.165 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-09-18T08:17:06.165 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:08:51.761822+0000", 2024-09-18T08:17:06.165 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 9.6520000000000001e-06 2024-09-18T08:17:06.165 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:17:06.165 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:17:06.165 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-09-18T08:17:06.165 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:08:51.761828+0000", 2024-09-18T08:17:06.165 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 5.305e-06 2024-09-18T08:17:06.165 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:17:06.165 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:17:06.165 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-18T08:17:06.166 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:08:51.761893+0000", 2024-09-18T08:17:06.166 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 6.5613999999999999e-05 2024-09-18T08:17:06.166 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:17:06.166 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:17:06.166 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "waiting for peered", 2024-09-18T08:17:06.166 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:08:51.761919+0000", 2024-09-18T08:17:06.166 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 2.567e-05 2024-09-18T08:17:06.166 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:17:06.166 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:17:06.166 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-18T08:17:06.166 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:08:52.683789+0000", 2024-09-18T08:17:06.166 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.92186995000000005 2024-09-18T08:17:06.166 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:17:06.167 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:17:06.167 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "waiting for peered", 2024-09-18T08:17:06.167 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:08:52.683797+0000", 2024-09-18T08:17:06.167 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 7.9079999999999995e-06 2024-09-18T08:17:06.167 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:17:06.167 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:17:06.167 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-18T08:17:06.167 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:08:53.843803+0000", 2024-09-18T08:17:06.167 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.1600056759999999 2024-09-18T08:17:06.167 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:17:06.167 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:17:06.167 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-09-18T08:17:06.167 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:08:53.843954+0000", 2024-09-18T08:17:06.167 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000151037 2024-09-18T08:17:06.168 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:17:06.168 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:17:06.168 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "waiting for subops from 2,4", 2024-09-18T08:17:06.168 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:08:53.844132+0000", 2024-09-18T08:17:06.168 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00017874000000000001 2024-09-18T08:17:06.168 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:17:06.168 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:17:06.168 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:17:06.168 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:08:53.848886+0000", 2024-09-18T08:17:06.168 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0047536779999999999 2024-09-18T08:17:06.168 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:17:06.168 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:17:06.168 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "op_commit", 2024-09-18T08:17:06.168 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:08:53.848906+0000", 2024-09-18T08:17:06.169 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.9758999999999998e-05 2024-09-18T08:17:06.169 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:17:06.169 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:17:06.169 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:17:06.169 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:08:53.867840+0000", 2024-09-18T08:17:06.169 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.018934359000000001 2024-09-18T08:17:06.169 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:17:06.169 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:17:06.169 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "commit_sent", 2024-09-18T08:17:06.169 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:08:53.867882+0000", 2024-09-18T08:17:06.169 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4.1526000000000002e-05 2024-09-18T08:17:06.169 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:17:06.169 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:17:06.170 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-09-18T08:17:06.170 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:08:53.869151+0000", 2024-09-18T08:17:06.170 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.001269325 2024-09-18T08:17:06.170 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-18T08:17:06.170 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-09-18T08:17:06.170 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-18T08:17:06.170 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:17:06.170 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:17:06.170 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.15216.0:69269 8.4 8:2819c4dc:::benchmark_data_smithi007_101583_object69268:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e198)", 2024-09-18T08:17:06.170 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-09-18T08:08:51.782860+0000", 2024-09-18T08:17:06.170 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 494.244133912, 2024-09-18T08:17:06.170 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 2.0970977999999998, 2024-09-18T08:17:06.170 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-09-18T08:17:06.170 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-09-18T08:17:06.171 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T08:17:06.171 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2024-09-18T08:17:06.171 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.15216", 2024-09-18T08:17:06.171 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "172.21.15.7:0/2596041177", 2024-09-18T08:17:06.171 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 69269 2024-09-18T08:17:06.171 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:17:06.171 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-09-18T08:17:06.171 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:17:06.171 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-09-18T08:17:06.171 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:08:51.782860+0000", 2024-09-18T08:17:06.171 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-18T08:17:06.171 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:17:06.171 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:17:06.171 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-09-18T08:17:06.172 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:08:51.782860+0000", 2024-09-18T08:17:06.172 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-18T08:17:06.172 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:17:06.172 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:17:06.172 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-09-18T08:17:06.172 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:08:51.782861+0000", 2024-09-18T08:17:06.172 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.3039999999999999e-06 2024-09-18T08:17:06.172 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:17:06.172 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:17:06.172 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-09-18T08:17:06.172 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:08:51.782932+0000", 2024-09-18T08:17:06.172 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 7.1074000000000004e-05 2024-09-18T08:17:06.172 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:17:06.172 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:17:06.173 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-09-18T08:17:06.173 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:08:51.782944+0000", 2024-09-18T08:17:06.173 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.2164000000000001e-05 2024-09-18T08:17:06.173 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:17:06.173 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:17:06.173 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-09-18T08:17:06.173 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:08:51.782952+0000", 2024-09-18T08:17:06.173 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 7.1559999999999998e-06 2024-09-18T08:17:06.173 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:17:06.173 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:17:06.173 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-18T08:17:06.173 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:08:51.783016+0000", 2024-09-18T08:17:06.174 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 6.4747999999999997e-05 2024-09-18T08:17:06.174 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:17:06.174 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:17:06.174 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "waiting for peered", 2024-09-18T08:17:06.174 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:08:51.783028+0000", 2024-09-18T08:17:06.174 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.1463e-05 2024-09-18T08:17:06.174 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:17:06.174 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:17:06.174 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-18T08:17:06.174 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:08:52.683825+0000", 2024-09-18T08:17:06.174 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.90079731799999996 2024-09-18T08:17:06.174 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:17:06.174 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:17:06.175 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "waiting for peered", 2024-09-18T08:17:06.175 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:08:52.683833+0000", 2024-09-18T08:17:06.175 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 7.763e-06 2024-09-18T08:17:06.175 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:17:06.175 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:17:06.175 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-18T08:17:06.175 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:08:53.846992+0000", 2024-09-18T08:17:06.175 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.163159319 2024-09-18T08:17:06.175 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:17:06.175 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:17:06.175 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-09-18T08:17:06.175 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:08:53.847095+0000", 2024-09-18T08:17:06.176 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000102392 2024-09-18T08:17:06.176 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:17:06.176 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:17:06.176 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "waiting for subops from 2,4", 2024-09-18T08:17:06.176 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:08:53.847204+0000", 2024-09-18T08:17:06.176 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00010955000000000001 2024-09-18T08:17:06.176 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:17:06.176 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:17:06.176 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:17:06.176 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:08:53.853079+0000", 2024-09-18T08:17:06.176 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0058749609999999997 2024-09-18T08:17:06.176 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:17:06.176 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:17:06.177 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "op_commit", 2024-09-18T08:17:06.177 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:08:53.853167+0000", 2024-09-18T08:17:06.177 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 8.7645000000000005e-05 2024-09-18T08:17:06.177 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:17:06.177 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:17:06.177 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:17:06.177 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:08:53.878127+0000", 2024-09-18T08:17:06.177 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.024960336 2024-09-18T08:17:06.177 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:17:06.177 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:17:06.177 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "commit_sent", 2024-09-18T08:17:06.177 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:08:53.878167+0000", 2024-09-18T08:17:06.177 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 3.9728999999999999e-05 2024-09-18T08:17:06.178 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:17:06.178 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:17:06.178 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-09-18T08:17:06.178 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:08:53.879958+0000", 2024-09-18T08:17:06.178 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0017908780000000001 2024-09-18T08:17:06.178 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-18T08:17:06.178 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-09-18T08:17:06.178 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-18T08:17:06.178 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:17:06.178 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:17:06.178 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.15216.0:69283 8.4 8:2031b74c:::benchmark_data_smithi007_101583_object69282:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e198)", 2024-09-18T08:17:06.178 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-09-18T08:08:51.806433+0000", 2024-09-18T08:17:06.178 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 494.220560944, 2024-09-18T08:17:06.179 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 2.086781915, 2024-09-18T08:17:06.179 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-09-18T08:17:06.179 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-09-18T08:17:06.179 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T08:17:06.179 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2024-09-18T08:17:06.179 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.15216", 2024-09-18T08:17:06.179 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "172.21.15.7:0/2596041177", 2024-09-18T08:17:06.179 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 69283 2024-09-18T08:17:06.179 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:17:06.179 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-09-18T08:17:06.179 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:17:06.179 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-09-18T08:17:06.179 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:08:51.806433+0000", 2024-09-18T08:17:06.180 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-18T08:17:06.180 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:17:06.180 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:17:06.180 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-09-18T08:17:06.180 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:08:51.806433+0000", 2024-09-18T08:17:06.180 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-18T08:17:06.180 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:17:06.180 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:17:06.180 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-09-18T08:17:06.181 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:08:51.806434+0000", 2024-09-18T08:17:06.181 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.341e-06 2024-09-18T08:17:06.181 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:17:06.181 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:17:06.181 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-09-18T08:17:06.181 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:08:51.806474+0000", 2024-09-18T08:17:06.181 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 3.9456999999999998e-05 2024-09-18T08:17:06.181 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:17:06.181 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:17:06.181 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-09-18T08:17:06.181 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:08:51.806486+0000", 2024-09-18T08:17:06.181 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.2619999999999999e-05 2024-09-18T08:17:06.181 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:17:06.182 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:17:06.182 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-09-18T08:17:06.182 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:08:51.806494+0000", 2024-09-18T08:17:06.182 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 7.3019999999999997e-06 2024-09-18T08:17:06.182 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:17:06.182 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:17:06.182 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-18T08:17:06.182 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:08:51.806557+0000", 2024-09-18T08:17:06.182 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 6.3452000000000002e-05 2024-09-18T08:17:06.182 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:17:06.182 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:17:06.182 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "waiting for peered", 2024-09-18T08:17:06.182 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:08:51.806568+0000", 2024-09-18T08:17:06.182 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.0671e-05 2024-09-18T08:17:06.183 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:17:06.183 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:17:06.183 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-18T08:17:06.183 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:08:52.683860+0000", 2024-09-18T08:17:06.183 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.87729208700000005 2024-09-18T08:17:06.183 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:17:06.183 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:17:06.183 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "waiting for peered", 2024-09-18T08:17:06.183 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:08:52.683868+0000", 2024-09-18T08:17:06.183 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 7.8050000000000007e-06 2024-09-18T08:17:06.183 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:17:06.183 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:17:06.183 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-18T08:17:06.184 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:08:53.848952+0000", 2024-09-18T08:17:06.184 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.1650841270000001 2024-09-18T08:17:06.184 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:17:06.184 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:17:06.184 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-09-18T08:17:06.184 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:08:53.849101+0000", 2024-09-18T08:17:06.184 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00014899999999999999 2024-09-18T08:17:06.184 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:17:06.184 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:17:06.184 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "waiting for subops from 2,4", 2024-09-18T08:17:06.184 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:08:53.849292+0000", 2024-09-18T08:17:06.184 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000191349 2024-09-18T08:17:06.184 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:17:06.185 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:17:06.185 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:17:06.185 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:08:53.853148+0000", 2024-09-18T08:17:06.185 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0038562010000000001 2024-09-18T08:17:06.185 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:17:06.185 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:17:06.185 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "op_commit", 2024-09-18T08:17:06.185 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:08:53.853173+0000", 2024-09-18T08:17:06.185 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 2.4482e-05 2024-09-18T08:17:06.185 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:17:06.185 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:17:06.185 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:17:06.185 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:08:53.891653+0000", 2024-09-18T08:17:06.186 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.038480225 2024-09-18T08:17:06.186 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:17:06.186 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:17:06.186 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "commit_sent", 2024-09-18T08:17:06.186 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:08:53.891697+0000", 2024-09-18T08:17:06.186 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4.4171000000000001e-05 2024-09-18T08:17:06.186 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:17:06.186 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:17:06.186 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-09-18T08:17:06.186 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:08:53.893215+0000", 2024-09-18T08:17:06.186 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0015176250000000001 2024-09-18T08:17:06.186 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-18T08:17:06.187 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-09-18T08:17:06.187 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-18T08:17:06.187 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:17:06.187 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:17:06.187 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.15216.0:69323 8.4 8:2053bc8e:::benchmark_data_smithi007_101583_object69322:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e198)", 2024-09-18T08:17:06.187 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-09-18T08:08:51.863425+0000", 2024-09-18T08:17:06.187 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 494.16356902000001, 2024-09-18T08:17:06.187 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 2.0310337920000001, 2024-09-18T08:17:06.187 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-09-18T08:17:06.187 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-09-18T08:17:06.187 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T08:17:06.187 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2024-09-18T08:17:06.187 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.15216", 2024-09-18T08:17:06.188 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "172.21.15.7:0/2596041177", 2024-09-18T08:17:06.188 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 69323 2024-09-18T08:17:06.188 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:17:06.188 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-09-18T08:17:06.188 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:17:06.188 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-09-18T08:17:06.188 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:08:51.863425+0000", 2024-09-18T08:17:06.188 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-18T08:17:06.188 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:17:06.188 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:17:06.188 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-09-18T08:17:06.188 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:08:51.863425+0000", 2024-09-18T08:17:06.188 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-18T08:17:06.189 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:17:06.189 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:17:06.189 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-09-18T08:17:06.189 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:08:51.863427+0000", 2024-09-18T08:17:06.189 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.846e-06 2024-09-18T08:17:06.189 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:17:06.189 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:17:06.189 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-09-18T08:17:06.189 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:08:51.863472+0000", 2024-09-18T08:17:06.189 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4.5639000000000001e-05 2024-09-18T08:17:06.189 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:17:06.189 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:17:06.189 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-09-18T08:17:06.189 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:08:51.863487+0000", 2024-09-18T08:17:06.190 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.4688999999999999e-05 2024-09-18T08:17:06.190 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:17:06.190 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:17:06.190 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-09-18T08:17:06.190 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:08:51.863495+0000", 2024-09-18T08:17:06.190 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 7.6850000000000001e-06 2024-09-18T08:17:06.190 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:17:06.190 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:17:06.190 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-18T08:17:06.190 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:08:51.863561+0000", 2024-09-18T08:17:06.190 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 6.6688999999999998e-05 2024-09-18T08:17:06.190 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:17:06.190 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:17:06.191 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "waiting for peered", 2024-09-18T08:17:06.191 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:08:51.863573+0000", 2024-09-18T08:17:06.191 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.1663e-05 2024-09-18T08:17:06.191 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:17:06.191 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:17:06.191 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-18T08:17:06.191 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:08:52.683896+0000", 2024-09-18T08:17:06.191 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.82032282700000003 2024-09-18T08:17:06.191 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:17:06.191 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:17:06.191 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "waiting for peered", 2024-09-18T08:17:06.191 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:08:52.683910+0000", 2024-09-18T08:17:06.191 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.4576000000000001e-05 2024-09-18T08:17:06.191 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:17:06.191 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:17:06.192 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-18T08:17:06.192 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:08:53.851037+0000", 2024-09-18T08:17:06.192 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.1671269989999999 2024-09-18T08:17:06.192 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:17:06.192 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:17:06.192 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-09-18T08:17:06.192 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:08:53.851121+0000", 2024-09-18T08:17:06.192 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 8.3844999999999994e-05 2024-09-18T08:17:06.192 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:17:06.192 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:17:06.192 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "waiting for subops from 2,4", 2024-09-18T08:17:06.192 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:08:53.851225+0000", 2024-09-18T08:17:06.192 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000103806 2024-09-18T08:17:06.193 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:17:06.193 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:17:06.193 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:17:06.193 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:08:53.860401+0000", 2024-09-18T08:17:06.193 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0091763390000000004 2024-09-18T08:17:06.193 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:17:06.193 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:17:06.193 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "op_commit", 2024-09-18T08:17:06.193 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:08:53.860538+0000", 2024-09-18T08:17:06.193 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00013657 2024-09-18T08:17:06.193 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:17:06.193 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:17:06.193 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:17:06.193 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:08:53.893287+0000", 2024-09-18T08:17:06.194 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.032748691000000003 2024-09-18T08:17:06.194 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:17:06.194 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:17:06.194 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "commit_sent", 2024-09-18T08:17:06.194 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:08:53.893357+0000", 2024-09-18T08:17:06.194 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 7.0751000000000001e-05 2024-09-18T08:17:06.194 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:17:06.194 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:17:06.194 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-09-18T08:17:06.194 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:08:53.894459+0000", 2024-09-18T08:17:06.194 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0011011770000000001 2024-09-18T08:17:06.194 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-18T08:17:06.194 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-09-18T08:17:06.195 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-18T08:17:06.195 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:17:06.195 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:17:06.195 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.15216.0:69346 8.4 8:28700287:::benchmark_data_smithi007_101583_object69345:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e198)", 2024-09-18T08:17:06.195 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-09-18T08:08:51.900473+0000", 2024-09-18T08:17:06.195 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 494.12652063899998, 2024-09-18T08:17:06.195 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 2.013468198, 2024-09-18T08:17:06.195 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-09-18T08:17:06.195 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-09-18T08:17:06.195 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T08:17:06.195 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2024-09-18T08:17:06.195 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.15216", 2024-09-18T08:17:06.195 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "172.21.15.7:0/2596041177", 2024-09-18T08:17:06.195 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 69346 2024-09-18T08:17:06.195 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:17:06.196 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-09-18T08:17:06.196 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:17:06.196 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-09-18T08:17:06.196 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:08:51.900473+0000", 2024-09-18T08:17:06.196 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-18T08:17:06.196 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:17:06.196 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:17:06.196 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-09-18T08:17:06.196 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:08:51.900473+0000", 2024-09-18T08:17:06.196 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-18T08:17:06.196 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:17:06.196 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:17:06.197 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-09-18T08:17:06.197 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:08:51.900474+0000", 2024-09-18T08:17:06.197 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.3060000000000001e-06 2024-09-18T08:17:06.197 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:17:06.197 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:17:06.197 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-09-18T08:17:06.197 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:08:51.900534+0000", 2024-09-18T08:17:06.197 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 5.9429000000000003e-05 2024-09-18T08:17:06.197 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:17:06.197 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:17:06.197 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-09-18T08:17:06.197 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:08:51.900547+0000", 2024-09-18T08:17:06.197 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.2716e-05 2024-09-18T08:17:06.197 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:17:06.197 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:17:06.198 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-09-18T08:17:06.198 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:08:51.900555+0000", 2024-09-18T08:17:06.198 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 8.0390000000000005e-06 2024-09-18T08:17:06.198 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:17:06.198 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:17:06.198 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-18T08:17:06.198 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:08:51.900615+0000", 2024-09-18T08:17:06.198 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 6.0646000000000002e-05 2024-09-18T08:17:06.198 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:17:06.198 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:17:06.198 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "waiting for peered", 2024-09-18T08:17:06.198 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:08:51.900621+0000", 2024-09-18T08:17:06.198 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 6.1820000000000003e-06 2024-09-18T08:17:06.199 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:17:06.199 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:17:06.199 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-18T08:17:06.199 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:08:52.683937+0000", 2024-09-18T08:17:06.199 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.78331511700000001 2024-09-18T08:17:06.199 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:17:06.199 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:17:06.199 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "waiting for peered", 2024-09-18T08:17:06.199 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:08:52.683945+0000", 2024-09-18T08:17:06.199 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 8.0630000000000006e-06 2024-09-18T08:17:06.199 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:17:06.199 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:17:06.199 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-18T08:17:06.199 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:08:53.853192+0000", 2024-09-18T08:17:06.200 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.169247726 2024-09-18T08:17:06.200 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:17:06.200 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:17:06.200 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-09-18T08:17:06.200 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:08:53.853272+0000", 2024-09-18T08:17:06.200 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 8.0107999999999999e-05 2024-09-18T08:17:06.200 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:17:06.200 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:17:06.200 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "waiting for subops from 2,4", 2024-09-18T08:17:06.200 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:08:53.853369+0000", 2024-09-18T08:17:06.200 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 9.6391000000000006e-05 2024-09-18T08:17:06.200 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:17:06.200 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:17:06.200 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:17:06.201 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:08:53.860443+0000", 2024-09-18T08:17:06.201 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0070746749999999999 2024-09-18T08:17:06.201 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:17:06.201 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:17:06.201 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "op_commit", 2024-09-18T08:17:06.201 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:08:53.860549+0000", 2024-09-18T08:17:06.201 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000105679 2024-09-18T08:17:06.201 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:17:06.201 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:17:06.201 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:17:06.201 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:08:53.912224+0000", 2024-09-18T08:17:06.201 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.051674987999999998 2024-09-18T08:17:06.201 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:17:06.202 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:17:06.202 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "commit_sent", 2024-09-18T08:17:06.202 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:08:53.912271+0000", 2024-09-18T08:17:06.202 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4.6724000000000002e-05 2024-09-18T08:17:06.202 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:17:06.202 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:17:06.202 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-09-18T08:17:06.202 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:08:53.913941+0000", 2024-09-18T08:17:06.202 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0016704090000000001 2024-09-18T08:17:06.202 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-18T08:17:06.202 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-09-18T08:17:06.202 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-18T08:17:06.202 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:17:06.202 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:17:06.202 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.15216.0:151197 8.4 8:28e40b7f:::benchmark_data_smithi007_101583_object50445:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e218)", 2024-09-18T08:17:06.202 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-09-18T08:10:14.236495+0000", 2024-09-18T08:17:06.203 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 411.79049909499997, 2024-09-18T08:17:06.203 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.027586009, 2024-09-18T08:17:06.203 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-09-18T08:17:06.203 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-09-18T08:17:06.203 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "reached pg", 2024-09-18T08:17:06.203 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2024-09-18T08:17:06.203 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.15216", 2024-09-18T08:17:06.203 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "172.21.15.7:0/2596041177", 2024-09-18T08:17:06.203 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 151197 2024-09-18T08:17:06.203 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:17:06.203 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-09-18T08:17:06.203 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:17:06.203 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-09-18T08:17:06.203 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:10:14.236495+0000", 2024-09-18T08:17:06.203 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-18T08:17:06.204 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:17:06.204 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:17:06.204 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-09-18T08:17:06.204 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:10:14.236495+0000", 2024-09-18T08:17:06.204 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-18T08:17:06.204 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:17:06.204 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:17:06.204 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-09-18T08:17:06.204 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:10:14.236495+0000", 2024-09-18T08:17:06.204 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4.3099999999999998e-07 2024-09-18T08:17:06.204 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:17:06.204 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:17:06.204 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-09-18T08:17:06.204 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:10:14.236497+0000", 2024-09-18T08:17:06.205 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 2.057e-06 2024-09-18T08:17:06.205 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:17:06.205 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:17:06.205 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-09-18T08:17:06.205 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:10:14.236503+0000", 2024-09-18T08:17:06.205 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 5.3870000000000003e-06 2024-09-18T08:17:06.205 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:17:06.205 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:17:06.205 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-09-18T08:17:06.205 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:10:14.236508+0000", 2024-09-18T08:17:06.205 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 5.2190000000000003e-06 2024-09-18T08:17:06.205 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:17:06.205 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:17:06.205 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-18T08:17:06.205 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:10:14.236572+0000", 2024-09-18T08:17:06.206 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 6.3966000000000004e-05 2024-09-18T08:17:06.206 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:17:06.206 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:17:06.206 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-09-18T08:17:06.206 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:10:14.236702+0000", 2024-09-18T08:17:06.206 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00013004299999999999 2024-09-18T08:17:06.206 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:17:06.206 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:17:06.206 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "waiting for subops from 2,4", 2024-09-18T08:17:06.206 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:10:14.236830+0000", 2024-09-18T08:17:06.206 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00012811100000000001 2024-09-18T08:17:06.206 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:17:06.206 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:17:06.207 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:17:06.207 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:10:14.241959+0000", 2024-09-18T08:17:06.207 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.005129471 2024-09-18T08:17:06.207 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:17:06.207 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:17:06.207 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "op_commit", 2024-09-18T08:17:06.207 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:10:14.242315+0000", 2024-09-18T08:17:06.207 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00035617899999999998 2024-09-18T08:17:06.207 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:17:06.207 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:17:06.207 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-18T08:17:06.207 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:10:14.249236+0000", 2024-09-18T08:17:06.208 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0069204399999999999 2024-09-18T08:17:06.208 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:17:06.208 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:17:06.208 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "waiting for peered", 2024-09-18T08:17:06.208 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:10:14.249256+0000", 2024-09-18T08:17:06.208 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.9888999999999999e-05 2024-09-18T08:17:06.208 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:17:06.208 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:17:06.208 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-18T08:17:06.208 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:10:15.264014+0000", 2024-09-18T08:17:06.208 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.014757718 2024-09-18T08:17:06.208 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:17:06.208 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:17:06.209 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-09-18T08:17:06.209 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:10:15.264081+0000", 2024-09-18T08:17:06.209 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 6.7098e-05 2024-09-18T08:17:06.209 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-18T08:17:06.209 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-09-18T08:17:06.209 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-18T08:17:06.209 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:17:06.209 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:17:06.209 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.15582.0:252487 9.7 9:eae9c550:::benchmark_data_smithi007_124019_object110602:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e291)", 2024-09-18T08:17:06.209 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-09-18T08:14:04.561104+0000", 2024-09-18T08:17:06.209 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 181.46589013100001, 2024-09-18T08:17:06.210 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 2.165593307, 2024-09-18T08:17:06.210 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-09-18T08:17:06.210 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-09-18T08:17:06.210 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "reached pg", 2024-09-18T08:17:06.210 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2024-09-18T08:17:06.210 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.15582", 2024-09-18T08:17:06.210 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "172.21.15.7:0/3598605839", 2024-09-18T08:17:06.210 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 252487 2024-09-18T08:17:06.210 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:17:06.210 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-09-18T08:17:06.210 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:17:06.210 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-09-18T08:17:06.210 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:14:04.561104+0000", 2024-09-18T08:17:06.211 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-18T08:17:06.211 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:17:06.211 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:17:06.211 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-09-18T08:17:06.211 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:14:04.561104+0000", 2024-09-18T08:17:06.211 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-18T08:17:06.211 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:17:06.211 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:17:06.211 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-09-18T08:17:06.211 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:14:04.561105+0000", 2024-09-18T08:17:06.211 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.263e-06 2024-09-18T08:17:06.212 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:17:06.212 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:17:06.212 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-09-18T08:17:06.212 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:14:04.561110+0000", 2024-09-18T08:17:06.212 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 5.4369999999999998e-06 2024-09-18T08:17:06.212 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:17:06.212 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:17:06.212 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-09-18T08:17:06.212 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:14:04.561134+0000", 2024-09-18T08:17:06.212 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 2.3430000000000001e-05 2024-09-18T08:17:06.212 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:17:06.212 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:17:06.212 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-09-18T08:17:06.213 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:14:04.561142+0000", 2024-09-18T08:17:06.213 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 8.1689999999999992e-06 2024-09-18T08:17:06.213 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:17:06.213 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:17:06.213 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-18T08:17:06.213 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:14:04.561226+0000", 2024-09-18T08:17:06.213 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 8.3919999999999996e-05 2024-09-18T08:17:06.213 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:17:06.213 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:17:06.213 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-09-18T08:17:06.213 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:14:04.561335+0000", 2024-09-18T08:17:06.213 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000109313 2024-09-18T08:17:06.214 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:17:06.214 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:17:06.214 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "waiting for subops from 3,7", 2024-09-18T08:17:06.214 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:14:04.561452+0000", 2024-09-18T08:17:06.214 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000116501 2024-09-18T08:17:06.214 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:17:06.214 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:17:06.214 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-18T08:17:06.214 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:14:04.585657+0000", 2024-09-18T08:17:06.214 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.024205082999999999 2024-09-18T08:17:06.214 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:17:06.214 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:17:06.215 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "waiting for peered", 2024-09-18T08:17:06.215 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:14:04.585671+0000", 2024-09-18T08:17:06.215 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.3998e-05 2024-09-18T08:17:06.215 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:17:06.215 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:17:06.215 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-18T08:17:06.215 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:14:05.593401+0000", 2024-09-18T08:17:06.215 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.0077304380000001 2024-09-18T08:17:06.215 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:17:06.215 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:17:06.215 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "waiting for peered", 2024-09-18T08:17:06.215 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:14:05.593414+0000", 2024-09-18T08:17:06.215 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.2656999999999999e-05 2024-09-18T08:17:06.216 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:17:06.216 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:17:06.216 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-18T08:17:06.216 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:14:06.726634+0000", 2024-09-18T08:17:06.216 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.133220077 2024-09-18T08:17:06.216 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:17:06.216 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:17:06.216 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-09-18T08:17:06.216 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:14:06.726697+0000", 2024-09-18T08:17:06.216 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 6.3021000000000005e-05 2024-09-18T08:17:06.216 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-18T08:17:06.216 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-09-18T08:17:06.217 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-18T08:17:06.217 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:17:06.217 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:17:06.217 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.15582.0:252492 9.7 9:e97f18e4:::benchmark_data_smithi007_124019_object110607:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e291)", 2024-09-18T08:17:06.217 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-09-18T08:14:04.568168+0000", 2024-09-18T08:17:06.217 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 181.45882621999999, 2024-09-18T08:17:06.217 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 2.1587796849999998, 2024-09-18T08:17:06.217 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-09-18T08:17:06.217 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-09-18T08:17:06.217 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "reached pg", 2024-09-18T08:17:06.217 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2024-09-18T08:17:06.218 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.15582", 2024-09-18T08:17:06.218 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "172.21.15.7:0/3598605839", 2024-09-18T08:17:06.218 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 252492 2024-09-18T08:17:06.218 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:17:06.218 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-09-18T08:17:06.218 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:17:06.218 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-09-18T08:17:06.218 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:14:04.568168+0000", 2024-09-18T08:17:06.218 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-18T08:17:06.218 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:17:06.218 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:17:06.218 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-09-18T08:17:06.219 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:14:04.568168+0000", 2024-09-18T08:17:06.219 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-18T08:17:06.219 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:17:06.219 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:17:06.219 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-09-18T08:17:06.219 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:14:04.568169+0000", 2024-09-18T08:17:06.219 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.342e-06 2024-09-18T08:17:06.219 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:17:06.219 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:17:06.219 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-09-18T08:17:06.219 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:14:04.568174+0000", 2024-09-18T08:17:06.220 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4.6539999999999998e-06 2024-09-18T08:17:06.220 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:17:06.220 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:17:06.220 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-09-18T08:17:06.220 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:14:04.568184+0000", 2024-09-18T08:17:06.220 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.0883999999999999e-05 2024-09-18T08:17:06.220 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:17:06.220 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:17:06.220 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-09-18T08:17:06.220 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:14:04.568192+0000", 2024-09-18T08:17:06.220 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 7.5739999999999998e-06 2024-09-18T08:17:06.221 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:17:06.221 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:17:06.221 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-18T08:17:06.221 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:14:04.568688+0000", 2024-09-18T08:17:06.221 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00049632500000000004 2024-09-18T08:17:06.221 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:17:06.221 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:17:06.221 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-09-18T08:17:06.221 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:14:04.568817+0000", 2024-09-18T08:17:06.221 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00012876000000000001 2024-09-18T08:17:06.221 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:17:06.221 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:17:06.221 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "waiting for subops from 3,7", 2024-09-18T08:17:06.222 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:14:04.568949+0000", 2024-09-18T08:17:06.222 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00013194300000000001 2024-09-18T08:17:06.222 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:17:06.222 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:17:06.222 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-18T08:17:06.222 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:14:04.585705+0000", 2024-09-18T08:17:06.222 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.016755823 2024-09-18T08:17:06.222 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:17:06.222 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:17:06.222 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "waiting for peered", 2024-09-18T08:17:06.222 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:14:04.585715+0000", 2024-09-18T08:17:06.222 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.0547000000000001e-05 2024-09-18T08:17:06.222 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:17:06.222 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:17:06.223 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-18T08:17:06.223 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:14:05.593509+0000", 2024-09-18T08:17:06.223 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.007793607 2024-09-18T08:17:06.223 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:17:06.223 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:17:06.223 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "waiting for peered", 2024-09-18T08:17:06.223 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:14:05.593521+0000", 2024-09-18T08:17:06.223 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.1834000000000001e-05 2024-09-18T08:17:06.223 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:17:06.223 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:17:06.223 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-18T08:17:06.223 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:14:06.726906+0000", 2024-09-18T08:17:06.223 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.1333849229999999 2024-09-18T08:17:06.223 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:17:06.223 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:17:06.224 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-09-18T08:17:06.224 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:14:06.726947+0000", 2024-09-18T08:17:06.224 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4.1468999999999999e-05 2024-09-18T08:17:06.224 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-18T08:17:06.224 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-09-18T08:17:06.224 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-18T08:17:06.224 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:17:06.224 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:17:06.224 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.15582.0:252495 9.7 9:e76824cc:::benchmark_data_smithi007_124019_object110610:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e292)", 2024-09-18T08:17:06.224 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-09-18T08:14:04.576440+0000", 2024-09-18T08:17:06.224 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 181.450553942, 2024-09-18T08:17:06.224 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 2.2199907799999998, 2024-09-18T08:17:06.224 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-09-18T08:17:06.224 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-09-18T08:17:06.225 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T08:17:06.225 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2024-09-18T08:17:06.225 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.15582", 2024-09-18T08:17:06.225 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "172.21.15.7:0/3598605839", 2024-09-18T08:17:06.225 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 252495 2024-09-18T08:17:06.225 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:17:06.225 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-09-18T08:17:06.225 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:17:06.225 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-09-18T08:17:06.225 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:14:04.576440+0000", 2024-09-18T08:17:06.225 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-18T08:17:06.225 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:17:06.225 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:17:06.225 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-09-18T08:17:06.226 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:14:04.576440+0000", 2024-09-18T08:17:06.226 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-18T08:17:06.226 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:17:06.226 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:17:06.226 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-09-18T08:17:06.226 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:14:04.576441+0000", 2024-09-18T08:17:06.226 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.663e-06 2024-09-18T08:17:06.226 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:17:06.226 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:17:06.226 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-09-18T08:17:06.226 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:14:04.576447+0000", 2024-09-18T08:17:06.226 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 5.806e-06 2024-09-18T08:17:06.226 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:17:06.226 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:17:06.227 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-09-18T08:17:06.227 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:14:04.576458+0000", 2024-09-18T08:17:06.227 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.0458999999999999e-05 2024-09-18T08:17:06.227 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:17:06.227 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:17:06.227 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-09-18T08:17:06.227 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:14:04.576467+0000", 2024-09-18T08:17:06.227 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 9.2229999999999996e-06 2024-09-18T08:17:06.227 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:17:06.227 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:17:06.227 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-18T08:17:06.227 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:14:04.585743+0000", 2024-09-18T08:17:06.227 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0092757240000000008 2024-09-18T08:17:06.227 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:17:06.228 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:17:06.228 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "waiting for peered", 2024-09-18T08:17:06.228 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:14:04.585752+0000", 2024-09-18T08:17:06.228 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 9.0790000000000007e-06 2024-09-18T08:17:06.228 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:17:06.228 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:17:06.228 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-18T08:17:06.228 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:14:05.593554+0000", 2024-09-18T08:17:06.228 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.007802423 2024-09-18T08:17:06.228 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:17:06.228 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:17:06.228 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "waiting for peered", 2024-09-18T08:17:06.228 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:14:05.593563+0000", 2024-09-18T08:17:06.228 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 9.0359999999999995e-06 2024-09-18T08:17:06.229 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:17:06.229 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:17:06.229 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-18T08:17:06.229 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:14:06.727007+0000", 2024-09-18T08:17:06.229 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.1334437239999999 2024-09-18T08:17:06.229 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:17:06.229 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:17:06.229 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-09-18T08:17:06.229 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:14:06.727083+0000", 2024-09-18T08:17:06.229 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 7.5791000000000007e-05 2024-09-18T08:17:06.229 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:17:06.229 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:17:06.229 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "waiting for subops from 2,3,7", 2024-09-18T08:17:06.229 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:14:06.727428+0000", 2024-09-18T08:17:06.230 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00034489900000000002 2024-09-18T08:17:06.230 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:17:06.230 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:17:06.230 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:17:06.230 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:14:06.732851+0000", 2024-09-18T08:17:06.230 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0054237729999999998 2024-09-18T08:17:06.230 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:17:06.230 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:17:06.230 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "op_commit", 2024-09-18T08:17:06.230 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:14:06.733574+0000", 2024-09-18T08:17:06.230 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00072280300000000005 2024-09-18T08:17:06.230 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:17:06.230 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:17:06.231 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:17:06.231 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:14:06.733630+0000", 2024-09-18T08:17:06.231 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 5.5566000000000003e-05 2024-09-18T08:17:06.231 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:17:06.231 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:17:06.231 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:17:06.231 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:14:06.796241+0000", 2024-09-18T08:17:06.231 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.062611555999999999 2024-09-18T08:17:06.231 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:17:06.231 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:17:06.231 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "commit_sent", 2024-09-18T08:17:06.231 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:14:06.796314+0000", 2024-09-18T08:17:06.231 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 7.2207999999999997e-05 2024-09-18T08:17:06.232 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:17:06.232 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:17:06.232 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-09-18T08:17:06.232 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:14:06.796431+0000", 2024-09-18T08:17:06.232 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00011704700000000001 2024-09-18T08:17:06.232 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-18T08:17:06.232 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-09-18T08:17:06.232 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-18T08:17:06.232 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:17:06.232 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:17:06.232 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.15582.0:252496 9.a 9:5293ceb8:::benchmark_data_smithi007_124019_object110611:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e292)", 2024-09-18T08:17:06.232 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-09-18T08:14:04.579029+0000", 2024-09-18T08:17:06.232 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 181.44796448899999, 2024-09-18T08:17:06.233 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 2.224896416, 2024-09-18T08:17:06.233 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-09-18T08:17:06.233 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-09-18T08:17:06.233 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T08:17:06.233 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2024-09-18T08:17:06.233 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.15582", 2024-09-18T08:17:06.233 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "172.21.15.7:0/3598605839", 2024-09-18T08:17:06.233 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 252496 2024-09-18T08:17:06.233 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:17:06.233 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-09-18T08:17:06.233 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:17:06.233 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-09-18T08:17:06.233 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:14:04.579029+0000", 2024-09-18T08:17:06.234 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-18T08:17:06.234 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:17:06.234 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:17:06.234 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-09-18T08:17:06.234 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:14:04.579029+0000", 2024-09-18T08:17:06.234 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-18T08:17:06.234 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:17:06.234 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:17:06.234 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-09-18T08:17:06.234 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:14:04.579030+0000", 2024-09-18T08:17:06.234 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.091e-06 2024-09-18T08:17:06.234 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:17:06.234 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:17:06.235 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-09-18T08:17:06.235 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:14:04.579036+0000", 2024-09-18T08:17:06.235 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 5.7069999999999997e-06 2024-09-18T08:17:06.235 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:17:06.235 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:17:06.235 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-09-18T08:17:06.235 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:14:04.579045+0000", 2024-09-18T08:17:06.235 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 9.2560000000000008e-06 2024-09-18T08:17:06.235 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:17:06.235 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:17:06.235 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-09-18T08:17:06.235 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:14:04.579054+0000", 2024-09-18T08:17:06.236 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 8.6510000000000007e-06 2024-09-18T08:17:06.236 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:17:06.236 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:17:06.236 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-18T08:17:06.236 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:14:04.586401+0000", 2024-09-18T08:17:06.236 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0073470150000000001 2024-09-18T08:17:06.236 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:17:06.236 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:17:06.236 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "waiting for peered", 2024-09-18T08:17:06.236 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:14:04.586409+0000", 2024-09-18T08:17:06.236 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 7.7919999999999993e-06 2024-09-18T08:17:06.236 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:17:06.236 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:17:06.237 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-18T08:17:06.237 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:14:05.578537+0000", 2024-09-18T08:17:06.237 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.99212811400000001 2024-09-18T08:17:06.237 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:17:06.237 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:17:06.237 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "waiting for peered", 2024-09-18T08:17:06.237 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:14:05.578550+0000", 2024-09-18T08:17:06.237 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.3069e-05 2024-09-18T08:17:06.237 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:17:06.237 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:17:06.237 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-18T08:17:06.237 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:14:06.795766+0000", 2024-09-18T08:17:06.237 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.2172160480000001 2024-09-18T08:17:06.238 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:17:06.238 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:17:06.238 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-09-18T08:17:06.238 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:14:06.795884+0000", 2024-09-18T08:17:06.238 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00011841 2024-09-18T08:17:06.238 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:17:06.238 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:17:06.238 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "waiting for subops from 3,7", 2024-09-18T08:17:06.238 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:14:06.796256+0000", 2024-09-18T08:17:06.238 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00037132700000000002 2024-09-18T08:17:06.238 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:17:06.238 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:17:06.238 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:17:06.238 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:14:06.803555+0000", 2024-09-18T08:17:06.238 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0072990449999999997 2024-09-18T08:17:06.238 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:17:06.239 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:17:06.239 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:17:06.239 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:14:06.803736+0000", 2024-09-18T08:17:06.239 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00018114300000000001 2024-09-18T08:17:06.239 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:17:06.239 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:17:06.239 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "op_commit", 2024-09-18T08:17:06.239 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:14:06.803750+0000", 2024-09-18T08:17:06.239 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.414e-05 2024-09-18T08:17:06.239 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:17:06.239 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:17:06.239 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "commit_sent", 2024-09-18T08:17:06.239 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:14:06.803809+0000", 2024-09-18T08:17:06.240 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 5.8817999999999998e-05 2024-09-18T08:17:06.240 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:17:06.240 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:17:06.240 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-09-18T08:17:06.240 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:14:06.803926+0000", 2024-09-18T08:17:06.240 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00011679 2024-09-18T08:17:06.240 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-18T08:17:06.240 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-09-18T08:17:06.240 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-18T08:17:06.240 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:17:06.240 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:17:06.240 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.15582.0:252503 9.d 9:b2496e7b:::benchmark_data_smithi007_124019_object110618:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e292)", 2024-09-18T08:17:06.240 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-09-18T08:14:04.603611+0000", 2024-09-18T08:17:06.240 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 181.42338272399999, 2024-09-18T08:17:06.240 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 2.134165211, 2024-09-18T08:17:06.241 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-09-18T08:17:06.241 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-09-18T08:17:06.241 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T08:17:06.241 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2024-09-18T08:17:06.241 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.15582", 2024-09-18T08:17:06.241 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "172.21.15.7:0/3598605839", 2024-09-18T08:17:06.241 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 252503 2024-09-18T08:17:06.241 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:17:06.241 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-09-18T08:17:06.241 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:17:06.241 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-09-18T08:17:06.241 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:14:04.603611+0000", 2024-09-18T08:17:06.241 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-18T08:17:06.242 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:17:06.242 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:17:06.242 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-09-18T08:17:06.242 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:14:04.603611+0000", 2024-09-18T08:17:06.242 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-18T08:17:06.242 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:17:06.242 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:17:06.242 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-09-18T08:17:06.242 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:14:04.603612+0000", 2024-09-18T08:17:06.242 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.2219999999999999e-06 2024-09-18T08:17:06.242 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:17:06.242 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:17:06.242 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-09-18T08:17:06.242 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:14:04.603617+0000", 2024-09-18T08:17:06.242 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4.8899999999999998e-06 2024-09-18T08:17:06.243 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:17:06.243 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:17:06.243 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-09-18T08:17:06.243 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:14:04.603628+0000", 2024-09-18T08:17:06.243 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.0899e-05 2024-09-18T08:17:06.243 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:17:06.243 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:17:06.243 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-09-18T08:17:06.243 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:14:04.603634+0000", 2024-09-18T08:17:06.243 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 5.5330000000000002e-06 2024-09-18T08:17:06.243 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:17:06.243 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:17:06.243 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-18T08:17:06.243 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:14:04.603676+0000", 2024-09-18T08:17:06.243 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4.2120999999999999e-05 2024-09-18T08:17:06.244 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:17:06.244 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:17:06.244 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "waiting for peered", 2024-09-18T08:17:06.244 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:14:04.603682+0000", 2024-09-18T08:17:06.244 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 5.835e-06 2024-09-18T08:17:06.244 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:17:06.244 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:17:06.244 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-18T08:17:06.244 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:14:05.602418+0000", 2024-09-18T08:17:06.244 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.998736654 2024-09-18T08:17:06.244 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:17:06.244 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:17:06.245 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "waiting for peered", 2024-09-18T08:17:06.245 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:14:05.602429+0000", 2024-09-18T08:17:06.245 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.044e-05 2024-09-18T08:17:06.245 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:17:06.245 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:17:06.245 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-18T08:17:06.245 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:14:06.731289+0000", 2024-09-18T08:17:06.245 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.1288608339999999 2024-09-18T08:17:06.245 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:17:06.245 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:17:06.245 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-09-18T08:17:06.246 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:14:06.731419+0000", 2024-09-18T08:17:06.246 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00012965200000000001 2024-09-18T08:17:06.246 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:17:06.246 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:17:06.246 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "waiting for subops from 4,7", 2024-09-18T08:17:06.246 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:14:06.731883+0000", 2024-09-18T08:17:06.246 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00046438000000000002 2024-09-18T08:17:06.246 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:17:06.246 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:17:06.246 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:17:06.246 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:14:06.737536+0000", 2024-09-18T08:17:06.246 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0056523299999999997 2024-09-18T08:17:06.246 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:17:06.246 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:17:06.247 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "op_commit", 2024-09-18T08:17:06.247 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:14:06.737568+0000", 2024-09-18T08:17:06.247 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 3.2668e-05 2024-09-18T08:17:06.247 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:17:06.247 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:17:06.247 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:17:06.247 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:14:06.737612+0000", 2024-09-18T08:17:06.247 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4.3686999999999997e-05 2024-09-18T08:17:06.247 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:17:06.247 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:17:06.247 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "commit_sent", 2024-09-18T08:17:06.247 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:14:06.737674+0000", 2024-09-18T08:17:06.247 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 6.2100000000000005e-05 2024-09-18T08:17:06.247 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:17:06.248 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:17:06.248 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-09-18T08:17:06.248 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:14:06.737776+0000", 2024-09-18T08:17:06.248 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00010196600000000001 2024-09-18T08:17:06.248 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-18T08:17:06.248 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-09-18T08:17:06.248 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-18T08:17:06.248 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-18T08:17:06.248 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-09-18T08:17:06.248 INFO:teuthology.orchestra.run.smithi050.stdout:} 2024-09-18T08:17:06.249 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0 2024-09-18T08:17:06.250 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/779349ca-7593-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T08:17:06.261 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid 779349ca-7593-11ef-bceb-c7b262605968 -- ceph osd reweight 3 1 2024-09-18T08:17:06.281 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:17:06.282 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:17:06.282 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0' 2024-09-18T08:17:06.383 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7 2024-09-18T08:17:06.406 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:17:06.407 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:17:06.407 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7' 2024-09-18T08:17:06.480 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:17:06 smithi007 ceph-mon[29459]: pgmap v713: 158 pgs: 1 active+clean+remapped, 3 active+remapped+backfilling, 2 active+remapped+backfill_toofull, 152 active+clean; 4.1 GiB data, 15 GiB used, 521 GiB / 536 GiB avail; 652 B/s rd, 0 B/s wr, 895 op/s; 13963/204357 objects misplaced (6.833%); 41 MiB/s, 653 objects/s recovering 2024-09-18T08:17:06.480 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:17:06 smithi007 ceph-mon[29459]: osdmap e370: 8 total, 8 up, 6 in 2024-09-18T08:17:06.480 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:17:06 smithi007 ceph-mon[39249]: pgmap v713: 158 pgs: 1 active+clean+remapped, 3 active+remapped+backfilling, 2 active+remapped+backfill_toofull, 152 active+clean; 4.1 GiB data, 15 GiB used, 521 GiB / 536 GiB avail; 652 B/s rd, 0 B/s wr, 895 op/s; 13963/204357 objects misplaced (6.833%); 41 MiB/s, 653 objects/s recovering 2024-09-18T08:17:06.480 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:17:06 smithi007 ceph-mon[39249]: osdmap e370: 8 total, 8 up, 6 in 2024-09-18T08:17:06.508 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7 2024-09-18T08:17:06.532 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:17:06.533 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:17:06.533 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7' 2024-09-18T08:17:06.634 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7 2024-09-18T08:17:06.655 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:17:06 smithi050 ceph-mon[32371]: pgmap v713: 158 pgs: 1 active+clean+remapped, 3 active+remapped+backfilling, 2 active+remapped+backfill_toofull, 152 active+clean; 4.1 GiB data, 15 GiB used, 521 GiB / 536 GiB avail; 652 B/s rd, 0 B/s wr, 895 op/s; 13963/204357 objects misplaced (6.833%); 41 MiB/s, 653 objects/s recovering 2024-09-18T08:17:06.655 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:17:06 smithi050 ceph-mon[32371]: osdmap e370: 8 total, 8 up, 6 in 2024-09-18T08:17:06.660 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:17:06.661 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:17:06.661 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7' 2024-09-18T08:17:06.762 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4 2024-09-18T08:17:06.788 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:17:06.789 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:17:06.789 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4' 2024-09-18T08:17:06.878 DEBUG:teuthology.orchestra.run.smithi050:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid 779349ca-7593-11ef-bceb-c7b262605968 -- ceph daemon osd.6 dump_ops_in_flight 2024-09-18T08:17:06.891 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7 2024-09-18T08:17:06.949 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:17:06.950 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:17:06.950 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7' 2024-09-18T08:17:06.980 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Sep 18 08:17:06 smithi007 ceph-779349ca-7593-11ef-bceb-c7b262605968-mgr-y[29688]: ::ffff:172.21.15.50 - - [18/Sep/2024:08:17:06] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-09-18T08:17:07.051 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6 2024-09-18T08:17:07.073 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:17:07.074 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:17:07.074 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6' 2024-09-18T08:17:07.175 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7 2024-09-18T08:17:07.203 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:17:07.204 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:17:07.204 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7' 2024-09-18T08:17:07.305 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6 2024-09-18T08:17:07.330 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:17:07.331 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:17:07.331 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6' 2024-09-18T08:17:07.432 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0 2024-09-18T08:17:07.473 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:17:07.474 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:17:07.474 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0' 2024-09-18T08:17:07.575 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7 2024-09-18T08:17:07.600 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:17:07.601 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:17:07.601 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7' 2024-09-18T08:17:07.702 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6 2024-09-18T08:17:07.723 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:17:07 smithi050 ceph-mon[32371]: osdmap e371: 8 total, 8 up, 6 in 2024-09-18T08:17:07.724 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:17:07 smithi050 ceph-mon[32371]: 5.17 scrub starts 2024-09-18T08:17:07.724 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:17:07 smithi050 ceph-mon[32371]: 5.17 scrub ok 2024-09-18T08:17:07.730 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:17:07 smithi007 ceph-mon[29459]: osdmap e371: 8 total, 8 up, 6 in 2024-09-18T08:17:07.730 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:17:07 smithi007 ceph-mon[29459]: 5.17 scrub starts 2024-09-18T08:17:07.730 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:17:07 smithi007 ceph-mon[29459]: 5.17 scrub ok 2024-09-18T08:17:07.731 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:17:07 smithi007 ceph-mon[39249]: osdmap e371: 8 total, 8 up, 6 in 2024-09-18T08:17:07.731 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:17:07 smithi007 ceph-mon[39249]: 5.17 scrub starts 2024-09-18T08:17:07.731 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:17:07 smithi007 ceph-mon[39249]: 5.17 scrub ok 2024-09-18T08:17:07.732 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:17:07.733 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:17:07.733 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6' 2024-09-18T08:17:07.763 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/779349ca-7593-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T08:17:07.834 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6 2024-09-18T08:17:07.861 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:17:07.862 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:17:07.862 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6' 2024-09-18T08:17:07.963 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1 2024-09-18T08:17:08.006 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:17:08.006 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:17:08.007 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1' 2024-09-18T08:17:08.107 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3 2024-09-18T08:17:08.146 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:17:08.147 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:17:08.147 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3' 2024-09-18T08:17:08.188 INFO:teuthology.orchestra.run.smithi050.stderr:Inferring config /var/lib/ceph/779349ca-7593-11ef-bceb-c7b262605968/mon.b/config 2024-09-18T08:17:08.248 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3 2024-09-18T08:17:08.270 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:17:08.271 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:17:08.271 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3' 2024-09-18T08:17:08.372 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2 2024-09-18T08:17:08.399 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:17:08.400 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:17:08.400 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2' 2024-09-18T08:17:08.501 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5 2024-09-18T08:17:08.548 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:17:08.549 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:17:08.549 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5' 2024-09-18T08:17:08.591 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:17:08 smithi050 ceph-mon[32371]: pgmap v716: 158 pgs: 1 active+clean+remapped, 3 active+remapped+backfilling, 2 active+remapped+backfill_toofull, 152 active+clean; 4.1 GiB data, 15 GiB used, 521 GiB / 536 GiB avail; 852 B/s rd, 0 B/s wr, 683 op/s; 13963/204357 objects misplaced (6.833%); 23 MiB/s, 371 objects/s recovering 2024-09-18T08:17:08.650 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5 2024-09-18T08:17:08.681 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:17:08.681 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:17:08.682 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5' 2024-09-18T08:17:08.720 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:17:08 smithi007 ceph-mon[29459]: pgmap v716: 158 pgs: 1 active+clean+remapped, 3 active+remapped+backfilling, 2 active+remapped+backfill_toofull, 152 active+clean; 4.1 GiB data, 15 GiB used, 521 GiB / 536 GiB avail; 852 B/s rd, 0 B/s wr, 683 op/s; 13963/204357 objects misplaced (6.833%); 23 MiB/s, 371 objects/s recovering 2024-09-18T08:17:08.720 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:17:08 smithi007 ceph-mon[39249]: pgmap v716: 158 pgs: 1 active+clean+remapped, 3 active+remapped+backfilling, 2 active+remapped+backfill_toofull, 152 active+clean; 4.1 GiB data, 15 GiB used, 521 GiB / 536 GiB avail; 852 B/s rd, 0 B/s wr, 683 op/s; 13963/204357 objects misplaced (6.833%); 23 MiB/s, 371 objects/s recovering 2024-09-18T08:17:08.728 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2024-09-18T08:17:08.728 INFO:teuthology.orchestra.run.smithi050.stdout: "ops": [ 2024-09-18T08:17:08.728 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:17:08.728 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_repop(client.15966.0:139049 10.7 e371/370 10:e75a2e46:::benchmark_data_smithi007_148834_object37329:head v 371'8679, mlcod=371'8677)", 2024-09-18T08:17:08.728 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-09-18T08:17:08.724106+0000", 2024-09-18T08:17:08.728 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 0.001928693, 2024-09-18T08:17:08.728 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0019482729999999999, 2024-09-18T08:17:08.728 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-09-18T08:17:08.728 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-09-18T08:17:08.728 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "started", 2024-09-18T08:17:08.728 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-09-18T08:17:08.729 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:17:08.729 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-09-18T08:17:08.729 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:17:08.724106+0000", 2024-09-18T08:17:08.729 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-18T08:17:08.729 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:17:08.729 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:17:08.729 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-09-18T08:17:08.729 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:17:08.724106+0000", 2024-09-18T08:17:08.729 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-18T08:17:08.729 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:17:08.729 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:17:08.729 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-09-18T08:17:08.729 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:17:08.724107+0000", 2024-09-18T08:17:08.730 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.0809999999999999e-06 2024-09-18T08:17:08.730 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:17:08.730 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:17:08.730 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-09-18T08:17:08.730 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:17:08.724114+0000", 2024-09-18T08:17:08.730 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 7.5299999999999999e-06 2024-09-18T08:17:08.730 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:17:08.730 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:17:08.730 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-09-18T08:17:08.730 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:17:08.724130+0000", 2024-09-18T08:17:08.730 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.6124e-05 2024-09-18T08:17:08.730 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:17:08.730 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:17:08.730 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-09-18T08:17:08.731 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:17:08.724140+0000", 2024-09-18T08:17:08.731 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 9.268e-06 2024-09-18T08:17:08.731 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:17:08.731 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:17:08.731 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-18T08:17:08.731 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:17:08.724198+0000", 2024-09-18T08:17:08.731 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 5.8281e-05 2024-09-18T08:17:08.731 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:17:08.731 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:17:08.731 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-09-18T08:17:08.731 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:17:08.724213+0000", 2024-09-18T08:17:08.731 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.5108e-05 2024-09-18T08:17:08.731 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-18T08:17:08.731 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-09-18T08:17:08.732 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-18T08:17:08.732 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-18T08:17:08.732 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2024-09-18T08:17:08.732 INFO:teuthology.orchestra.run.smithi050.stdout: "num_ops": 1 2024-09-18T08:17:08.732 INFO:teuthology.orchestra.run.smithi050.stdout:} 2024-09-18T08:17:08.783 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0 2024-09-18T08:17:08.812 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:17:08.812 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:17:08.813 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0' 2024-09-18T08:17:08.914 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4 2024-09-18T08:17:08.943 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:17:08.944 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:17:08.944 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4' 2024-09-18T08:17:09.045 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3 2024-09-18T08:17:09.084 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:17:09.085 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:17:09.086 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3' 2024-09-18T08:17:09.186 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3 2024-09-18T08:17:09.225 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:17:09.225 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:17:09.226 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3' 2024-09-18T08:17:09.327 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1 2024-09-18T08:17:09.366 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:17:09.367 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:17:09.367 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1' 2024-09-18T08:17:09.468 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0 2024-09-18T08:17:09.503 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:17:09.504 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:17:09.504 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0' 2024-09-18T08:17:09.572 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:17:09 smithi050 ceph-mon[32371]: 2.2 scrub starts 2024-09-18T08:17:09.572 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:17:09 smithi050 ceph-mon[32371]: 2.2 scrub ok 2024-09-18T08:17:09.605 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6 2024-09-18T08:17:09.632 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:17:09.633 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:17:09.634 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6' 2024-09-18T08:17:09.728 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:17:09 smithi007 ceph-mon[39249]: 2.2 scrub starts 2024-09-18T08:17:09.728 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:17:09 smithi007 ceph-mon[39249]: 2.2 scrub ok 2024-09-18T08:17:09.728 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:17:09 smithi007 ceph-mon[29459]: 2.2 scrub starts 2024-09-18T08:17:09.728 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:17:09 smithi007 ceph-mon[29459]: 2.2 scrub ok 2024-09-18T08:17:09.735 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3 2024-09-18T08:17:09.793 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:17:09.794 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:17:09.794 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3' 2024-09-18T08:17:09.895 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0 2024-09-18T08:17:09.905 INFO:journalctl@ceph.iscsi.iscsi.a.smithi050.stdout:Sep 18 08:17:09 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-iscsi-iscsi-a[57782]: debug there is no tcmu-runner data available 2024-09-18T08:17:09.937 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:17:09.938 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:17:09.938 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0' 2024-09-18T08:17:10.039 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2 2024-09-18T08:17:10.069 DEBUG:teuthology.orchestra.run.smithi050:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid 779349ca-7593-11ef-bceb-c7b262605968 -- ceph daemon osd.6 dump_blocked_ops 2024-09-18T08:17:10.086 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:17:10.087 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:17:10.087 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2' 2024-09-18T08:17:10.188 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0 2024-09-18T08:17:10.235 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:17:10.236 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:17:10.237 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0' 2024-09-18T08:17:10.337 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5 2024-09-18T08:17:10.363 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:17:10.364 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:17:10.364 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5' 2024-09-18T08:17:10.465 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2 2024-09-18T08:17:10.466 INFO:teuthology.orchestra.run.smithi007.stderr:noscrub is set 2024-09-18T08:17:10.527 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:17:10.528 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:17:10.529 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2' 2024-09-18T08:17:10.553 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:17:10 smithi007 ceph-mon[29459]: pgmap v717: 158 pgs: 1 active+clean+remapped, 2 active+remapped+backfilling, 2 active+remapped+backfill_toofull, 153 active+clean; 4.0 GiB data, 14 GiB used, 522 GiB / 536 GiB avail; 1.3 KiB/s rd, 0 B/s wr, 795 op/s; 13280/197625 objects misplaced (6.720%); 19 MiB/s, 305 objects/s recovering 2024-09-18T08:17:10.553 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:17:10 smithi007 ceph-mon[29459]: 6.b scrub starts 2024-09-18T08:17:10.553 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:17:10 smithi007 ceph-mon[29459]: 6.b scrub ok 2024-09-18T08:17:10.553 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:17:10 smithi007 ceph-mon[29459]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-18T08:17:10.553 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:17:10 smithi007 ceph-mon[29459]: from='client.? 172.21.15.7:0/4019798034' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-18T08:17:10.553 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:17:10 smithi007 ceph-mon[29459]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-18T08:17:10.553 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:17:10 smithi007 ceph-mon[39249]: pgmap v717: 158 pgs: 1 active+clean+remapped, 2 active+remapped+backfilling, 2 active+remapped+backfill_toofull, 153 active+clean; 4.0 GiB data, 14 GiB used, 522 GiB / 536 GiB avail; 1.3 KiB/s rd, 0 B/s wr, 795 op/s; 13280/197625 objects misplaced (6.720%); 19 MiB/s, 305 objects/s recovering 2024-09-18T08:17:10.553 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:17:10 smithi007 ceph-mon[39249]: 6.b scrub starts 2024-09-18T08:17:10.553 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:17:10 smithi007 ceph-mon[39249]: 6.b scrub ok 2024-09-18T08:17:10.554 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:17:10 smithi007 ceph-mon[39249]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-18T08:17:10.554 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:17:10 smithi007 ceph-mon[39249]: from='client.? 172.21.15.7:0/4019798034' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-18T08:17:10.554 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:17:10 smithi007 ceph-mon[39249]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-18T08:17:10.629 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1 2024-09-18T08:17:10.647 INFO:teuthology.orchestra.run.smithi007.stdout:osd.0: {} 2024-09-18T08:17:10.647 INFO:teuthology.orchestra.run.smithi007.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-18T08:17:10.655 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:17:10 smithi050 ceph-mon[32371]: pgmap v717: 158 pgs: 1 active+clean+remapped, 2 active+remapped+backfilling, 2 active+remapped+backfill_toofull, 153 active+clean; 4.0 GiB data, 14 GiB used, 522 GiB / 536 GiB avail; 1.3 KiB/s rd, 0 B/s wr, 795 op/s; 13280/197625 objects misplaced (6.720%); 19 MiB/s, 305 objects/s recovering 2024-09-18T08:17:10.655 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:17:10 smithi050 ceph-mon[32371]: 6.b scrub starts 2024-09-18T08:17:10.655 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:17:10 smithi050 ceph-mon[32371]: 6.b scrub ok 2024-09-18T08:17:10.655 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:17:10 smithi050 ceph-mon[32371]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-18T08:17:10.655 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:17:10 smithi050 ceph-mon[32371]: from='client.? 172.21.15.7:0/4019798034' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-18T08:17:10.655 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:17:10 smithi050 ceph-mon[32371]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-18T08:17:10.666 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:17:10.667 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:17:10.667 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1' 2024-09-18T08:17:10.688 INFO:teuthology.orchestra.run.smithi007.stdout:osd.1: {} 2024-09-18T08:17:10.688 INFO:teuthology.orchestra.run.smithi007.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-18T08:17:10.724 INFO:teuthology.orchestra.run.smithi007.stdout:osd.2: {} 2024-09-18T08:17:10.724 INFO:teuthology.orchestra.run.smithi007.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-18T08:17:10.761 INFO:teuthology.orchestra.run.smithi007.stdout:osd.3: {} 2024-09-18T08:17:10.762 INFO:teuthology.orchestra.run.smithi007.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-18T08:17:10.768 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4 2024-09-18T08:17:10.795 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:17:10.796 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:17:10.796 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4' 2024-09-18T08:17:10.800 INFO:teuthology.orchestra.run.smithi007.stdout:osd.4: {} 2024-09-18T08:17:10.800 INFO:teuthology.orchestra.run.smithi007.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-18T08:17:10.841 INFO:teuthology.orchestra.run.smithi007.stdout:osd.5: {} 2024-09-18T08:17:10.842 INFO:teuthology.orchestra.run.smithi007.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-18T08:17:10.873 INFO:teuthology.orchestra.run.smithi007.stdout:osd.6: {} 2024-09-18T08:17:10.873 INFO:teuthology.orchestra.run.smithi007.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-18T08:17:10.897 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4 2024-09-18T08:17:10.913 INFO:teuthology.orchestra.run.smithi007.stdout:osd.7: {} 2024-09-18T08:17:10.913 INFO:teuthology.orchestra.run.smithi007.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-18T08:17:10.927 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:17:10.927 INFO:teuthology.orchestra.run.smithi050.stderr:Inferring config /var/lib/ceph/779349ca-7593-11ef-bceb-c7b262605968/mon.b/config 2024-09-18T08:17:10.927 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:17:10.928 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4' 2024-09-18T08:17:11.029 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1 2024-09-18T08:17:11.068 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:17:11.069 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:17:11.069 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1' 2024-09-18T08:17:11.170 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3 2024-09-18T08:17:11.214 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:17:11.215 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:17:11.215 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3' 2024-09-18T08:17:11.316 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7 2024-09-18T08:17:11.350 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:17:11.351 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:17:11.351 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7' 2024-09-18T08:17:11.451 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0 2024-09-18T08:17:11.490 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:17:11.543 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:17:11.544 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0' 2024-09-18T08:17:11.644 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5 2024-09-18T08:17:11.655 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:17:11 smithi050 ceph-mon[32371]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-18T08:17:11.655 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:17:11 smithi050 ceph-mon[32371]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-18T08:17:11.655 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:17:11 smithi050 ceph-mon[32371]: osdmap e372: 8 total, 8 up, 6 in 2024-09-18T08:17:11.656 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:17:11 smithi050 ceph-mon[32371]: from='client.? 172.21.15.7:0/2517297500' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-18T08:17:11.656 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:17:11 smithi050 ceph-mon[32371]: from='mgr.24470 172.21.15.7:0/2594390261' entity='mgr.y' 2024-09-18T08:17:11.656 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:17:11 smithi050 ceph-mon[32371]: from='mgr.24470 172.21.15.7:0/2594390261' entity='mgr.y' 2024-09-18T08:17:11.671 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:17:11.671 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:17:11.672 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5' 2024-09-18T08:17:11.730 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:17:11 smithi007 ceph-mon[29459]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-18T08:17:11.730 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:17:11 smithi007 ceph-mon[29459]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-18T08:17:11.730 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:17:11 smithi007 ceph-mon[29459]: osdmap e372: 8 total, 8 up, 6 in 2024-09-18T08:17:11.730 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:17:11 smithi007 ceph-mon[29459]: from='client.? 172.21.15.7:0/2517297500' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-18T08:17:11.730 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:17:11 smithi007 ceph-mon[29459]: from='mgr.24470 172.21.15.7:0/2594390261' entity='mgr.y' 2024-09-18T08:17:11.730 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:17:11 smithi007 ceph-mon[29459]: from='mgr.24470 172.21.15.7:0/2594390261' entity='mgr.y' 2024-09-18T08:17:11.731 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:17:11 smithi007 ceph-mon[39249]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-18T08:17:11.731 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:17:11 smithi007 ceph-mon[39249]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-18T08:17:11.731 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:17:11 smithi007 ceph-mon[39249]: osdmap e372: 8 total, 8 up, 6 in 2024-09-18T08:17:11.731 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:17:11 smithi007 ceph-mon[39249]: from='client.? 172.21.15.7:0/2517297500' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-18T08:17:11.731 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:17:11 smithi007 ceph-mon[39249]: from='mgr.24470 172.21.15.7:0/2594390261' entity='mgr.y' 2024-09-18T08:17:11.731 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:17:11 smithi007 ceph-mon[39249]: from='mgr.24470 172.21.15.7:0/2594390261' entity='mgr.y' 2024-09-18T08:17:11.773 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1 2024-09-18T08:17:11.822 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:17:11.823 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:17:11.823 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1' 2024-09-18T08:17:11.924 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5 2024-09-18T08:17:11.929 INFO:teuthology.orchestra.run.smithi050.stdout:ERROR: (22) Invalid argument 2024-09-18T08:17:11.929 INFO:teuthology.orchestra.run.smithi050.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-18T08:17:11.950 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:17:11.950 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:17:11.951 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5' 2024-09-18T08:17:12.052 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3 2024-09-18T08:17:12.099 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:17:12.103 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:17:12.104 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3' 2024-09-18T08:17:12.205 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2 2024-09-18T08:17:12.241 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:17:12.242 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:17:12.242 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2' 2024-09-18T08:17:12.343 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6 2024-09-18T08:17:12.370 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:17:12.371 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:17:12.371 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6' 2024-09-18T08:17:12.472 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2 2024-09-18T08:17:12.479 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:17:12 smithi007 ceph-mon[39249]: pgmap v719: 158 pgs: 2 active+remapped+backfilling, 2 active+remapped+backfill_toofull, 154 active+clean; 3.9 GiB data, 14 GiB used, 522 GiB / 536 GiB avail; 852 B/s rd, 0 B/s wr, 639 op/s; 11387/192855 objects misplaced (5.904%); 19 MiB/s, 305 objects/s recovering 2024-09-18T08:17:12.511 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:17:12.511 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:17:12.512 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2' 2024-09-18T08:17:12.539 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/779349ca-7593-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T08:17:12.610 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:17:12 smithi050 ceph-mon[32371]: pgmap v719: 158 pgs: 2 active+remapped+backfilling, 2 active+remapped+backfill_toofull, 154 active+clean; 3.9 GiB data, 14 GiB used, 522 GiB / 536 GiB avail; 852 B/s rd, 0 B/s wr, 639 op/s; 11387/192855 objects misplaced (5.904%); 19 MiB/s, 305 objects/s recovering 2024-09-18T08:17:12.612 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3 2024-09-18T08:17:12.647 DEBUG:teuthology.orchestra.run.smithi050:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid 779349ca-7593-11ef-bceb-c7b262605968 -- ceph daemon osd.6 dump_historic_ops 2024-09-18T08:17:12.680 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:17:12.680 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:17:12.681 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3' 2024-09-18T08:17:12.730 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:17:12 smithi007 ceph-mon[29459]: pgmap v719: 158 pgs: 2 active+remapped+backfilling, 2 active+remapped+backfill_toofull, 154 active+clean; 3.9 GiB data, 14 GiB used, 522 GiB / 536 GiB avail; 852 B/s rd, 0 B/s wr, 639 op/s; 11387/192855 objects misplaced (5.904%); 19 MiB/s, 305 objects/s recovering 2024-09-18T08:17:12.782 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4 2024-09-18T08:17:12.814 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:17:12.815 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:17:12.815 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4' 2024-09-18T08:17:12.916 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5 2024-09-18T08:17:12.945 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:17:12.946 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:17:12.946 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5' 2024-09-18T08:17:12.979 INFO:teuthology.orchestra.run.smithi050.stderr:Inferring config /var/lib/ceph/779349ca-7593-11ef-bceb-c7b262605968/mon.b/config 2024-09-18T08:17:13.047 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3 2024-09-18T08:17:13.092 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:17:13.093 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:17:13.093 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3' 2024-09-18T08:17:13.194 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7 2024-09-18T08:17:13.219 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:17:13.220 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:17:13.220 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7' 2024-09-18T08:17:13.321 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5 2024-09-18T08:17:13.349 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:17:13.349 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:17:13.350 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5' 2024-09-18T08:17:13.428 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid 779349ca-7593-11ef-bceb-c7b262605968 -- ceph osd set nodeep-scrub 2024-09-18T08:17:13.451 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3 2024-09-18T08:17:13.513 INFO:teuthology.orchestra.run.smithi050.stdout:ERROR: (22) Invalid argument 2024-09-18T08:17:13.513 INFO:teuthology.orchestra.run.smithi050.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-18T08:17:13.548 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:17:13.549 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:17:13.549 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3' 2024-09-18T08:17:13.650 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0 2024-09-18T08:17:13.680 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:17:13.681 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:17:13.681 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0' 2024-09-18T08:17:13.782 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3 2024-09-18T08:17:13.849 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:17:13.850 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:17:13.850 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3' 2024-09-18T08:17:13.951 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7 2024-09-18T08:17:13.977 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:17:13.978 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:17:13.978 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7' 2024-09-18T08:17:14.079 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2 2024-09-18T08:17:14.116 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:17:14.117 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:17:14.117 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2' 2024-09-18T08:17:14.218 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6 2024-09-18T08:17:14.229 DEBUG:teuthology.orchestra.run.smithi050:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid 779349ca-7593-11ef-bceb-c7b262605968 -- ceph daemon osd.7 dump_ops_in_flight 2024-09-18T08:17:14.244 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:17:14.245 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:17:14.245 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6' 2024-09-18T08:17:14.347 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3 2024-09-18T08:17:14.385 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:17:14.385 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:17:14.386 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3' 2024-09-18T08:17:14.487 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4 2024-09-18T08:17:14.497 INFO:teuthology.orchestra.run.smithi007.stderr:reweighted osd.3 to 1 (10000) 2024-09-18T08:17:14.500 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:17:14 smithi050 ceph-mon[32371]: pgmap v720: 158 pgs: 2 active+remapped+backfilling, 2 active+remapped+backfill_toofull, 154 active+clean; 3.9 GiB data, 14 GiB used, 522 GiB / 536 GiB avail; 1.3 KiB/s rd, 0 B/s wr, 606 op/s; 11058/189870 objects misplaced (5.824%); 14 MiB/s, 229 objects/s recovering 2024-09-18T08:17:14.501 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:17:14 smithi050 ceph-mon[32371]: from='client.? 172.21.15.7:0/3406563058' entity='client.admin' cmd={"prefix": "osd reweight", "id": 3, "weight": 1.0} : dispatch 2024-09-18T08:17:14.501 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:17:14 smithi050 ceph-mon[32371]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 3, "weight": 1.0} : dispatch 2024-09-18T08:17:14.515 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:17:14.516 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:17:14.516 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4' 2024-09-18T08:17:14.561 INFO:teuthology.orchestra.run.smithi050.stderr:Inferring config /var/lib/ceph/779349ca-7593-11ef-bceb-c7b262605968/mon.b/config 2024-09-18T08:17:14.591 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid 779349ca-7593-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-18T08:17:14.618 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4 2024-09-18T08:17:14.643 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:17:14.644 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:17:14.644 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4' 2024-09-18T08:17:14.745 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1 2024-09-18T08:17:14.778 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:17:14.779 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:17:14.779 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1' 2024-09-18T08:17:14.880 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2 2024-09-18T08:17:14.927 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:17:14.927 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:17:14.928 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2' 2024-09-18T08:17:14.980 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:17:14 smithi007 ceph-mon[29459]: pgmap v720: 158 pgs: 2 active+remapped+backfilling, 2 active+remapped+backfill_toofull, 154 active+clean; 3.9 GiB data, 14 GiB used, 522 GiB / 536 GiB avail; 1.3 KiB/s rd, 0 B/s wr, 606 op/s; 11058/189870 objects misplaced (5.824%); 14 MiB/s, 229 objects/s recovering 2024-09-18T08:17:14.980 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:17:14 smithi007 ceph-mon[29459]: from='client.? 172.21.15.7:0/3406563058' entity='client.admin' cmd={"prefix": "osd reweight", "id": 3, "weight": 1.0} : dispatch 2024-09-18T08:17:14.980 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:17:14 smithi007 ceph-mon[29459]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 3, "weight": 1.0} : dispatch 2024-09-18T08:17:14.981 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:17:14 smithi007 ceph-mon[39249]: pgmap v720: 158 pgs: 2 active+remapped+backfilling, 2 active+remapped+backfill_toofull, 154 active+clean; 3.9 GiB data, 14 GiB used, 522 GiB / 536 GiB avail; 1.3 KiB/s rd, 0 B/s wr, 606 op/s; 11058/189870 objects misplaced (5.824%); 14 MiB/s, 229 objects/s recovering 2024-09-18T08:17:14.981 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:17:14 smithi007 ceph-mon[39249]: from='client.? 172.21.15.7:0/3406563058' entity='client.admin' cmd={"prefix": "osd reweight", "id": 3, "weight": 1.0} : dispatch 2024-09-18T08:17:14.981 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:17:14 smithi007 ceph-mon[39249]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 3, "weight": 1.0} : dispatch 2024-09-18T08:17:15.029 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3 2024-09-18T08:17:15.052 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:17:15.052 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:17:15.053 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3' 2024-09-18T08:17:15.116 INFO:teuthology.orchestra.run.smithi050.stdout:ERROR: (22) Invalid argument 2024-09-18T08:17:15.116 INFO:teuthology.orchestra.run.smithi050.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-18T08:17:15.153 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1 2024-09-18T08:17:15.191 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:17:15.191 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:17:15.192 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1' 2024-09-18T08:17:15.292 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7 2024-09-18T08:17:15.319 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:17:15.320 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:17:15.320 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7' 2024-09-18T08:17:15.421 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0 2024-09-18T08:17:15.465 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:17:15.466 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:17:15.466 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0' 2024-09-18T08:17:15.567 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5 2024-09-18T08:17:15.593 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:17:15.594 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:17:15.595 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5' 2024-09-18T08:17:15.695 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0 2024-09-18T08:17:15.743 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:17:15.743 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:17:15.744 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0' 2024-09-18T08:17:15.845 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5 2024-09-18T08:17:15.871 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:17:15.871 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:17:15.872 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5' 2024-09-18T08:17:15.905 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:17:15 smithi050 ceph-mon[32371]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]': finished 2024-09-18T08:17:15.905 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:17:15 smithi050 ceph-mon[32371]: osdmap e373: 8 total, 8 up, 6 in 2024-09-18T08:17:15.973 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5 2024-09-18T08:17:15.980 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:17:15 smithi007 ceph-mon[29459]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]': finished 2024-09-18T08:17:15.980 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:17:15 smithi007 ceph-mon[29459]: osdmap e373: 8 total, 8 up, 6 in 2024-09-18T08:17:15.980 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:17:15 smithi007 ceph-mon[39249]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]': finished 2024-09-18T08:17:15.980 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:17:15 smithi007 ceph-mon[39249]: osdmap e373: 8 total, 8 up, 6 in 2024-09-18T08:17:16.000 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:17:16.000 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:17:16.001 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5' 2024-09-18T08:17:16.102 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6 2024-09-18T08:17:16.125 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:17:16.126 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:17:16.127 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6' 2024-09-18T08:17:16.196 DEBUG:teuthology.orchestra.run.smithi050:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid 779349ca-7593-11ef-bceb-c7b262605968 -- ceph daemon osd.7 dump_blocked_ops 2024-09-18T08:17:16.228 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1 2024-09-18T08:17:16.275 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:17:16.277 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:17:16.277 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1' 2024-09-18T08:17:16.378 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0 2024-09-18T08:17:16.424 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:17:16.425 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:17:16.426 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0' 2024-09-18T08:17:16.500 INFO:teuthology.orchestra.run.smithi050.stderr:Inferring config /var/lib/ceph/779349ca-7593-11ef-bceb-c7b262605968/mon.b/config 2024-09-18T08:17:16.526 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4 2024-09-18T08:17:16.554 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:17:16.555 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:17:16.555 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4' 2024-09-18T08:17:16.656 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3 2024-09-18T08:17:16.681 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:17:16.682 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:17:16.682 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3' 2024-09-18T08:17:16.783 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7 2024-09-18T08:17:16.798 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:17:16 smithi050 ceph-mon[32371]: pgmap v722: 158 pgs: 2 active+remapped+backfilling, 2 active+remapped+backfill_toofull, 154 active+clean; 3.6 GiB data, 13 GiB used, 523 GiB / 536 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.12k op/s; 9656/177357 objects misplaced (5.444%); 21 MiB/s, 331 objects/s recovering 2024-09-18T08:17:16.806 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:17:16.806 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:17:16.807 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7' 2024-09-18T08:17:16.908 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1 2024-09-18T08:17:16.930 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:17:16.931 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:17:16.931 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1' 2024-09-18T08:17:16.980 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:17:16 smithi007 ceph-mon[29459]: pgmap v722: 158 pgs: 2 active+remapped+backfilling, 2 active+remapped+backfill_toofull, 154 active+clean; 3.6 GiB data, 13 GiB used, 523 GiB / 536 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.12k op/s; 9656/177357 objects misplaced (5.444%); 21 MiB/s, 331 objects/s recovering 2024-09-18T08:17:16.980 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Sep 18 08:17:16 smithi007 ceph-779349ca-7593-11ef-bceb-c7b262605968-mgr-y[29688]: ::ffff:172.21.15.50 - - [18/Sep/2024:08:17:16] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-09-18T08:17:16.980 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:17:16 smithi007 ceph-mon[39249]: pgmap v722: 158 pgs: 2 active+remapped+backfilling, 2 active+remapped+backfill_toofull, 154 active+clean; 3.6 GiB data, 13 GiB used, 523 GiB / 536 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.12k op/s; 9656/177357 objects misplaced (5.444%); 21 MiB/s, 331 objects/s recovering 2024-09-18T08:17:17.032 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1 2024-09-18T08:17:17.071 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:17:17.071 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:17:17.072 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1' 2024-09-18T08:17:17.173 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5 2024-09-18T08:17:17.203 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:17:17.203 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:17:17.204 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5' 2024-09-18T08:17:17.305 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5 2024-09-18T08:17:17.330 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:17:17.330 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:17:17.331 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5' 2024-09-18T08:17:17.368 INFO:teuthology.orchestra.run.smithi050.stdout:ERROR: (22) Invalid argument 2024-09-18T08:17:17.368 INFO:teuthology.orchestra.run.smithi050.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-18T08:17:17.432 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1 2024-09-18T08:17:17.468 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:17:17.469 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:17:17.470 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1' 2024-09-18T08:17:17.571 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0 2024-09-18T08:17:17.612 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:17:17.613 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:17:17.613 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0' 2024-09-18T08:17:17.714 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3 2024-09-18T08:17:17.754 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:17:17.754 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:17:17.755 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3' 2024-09-18T08:17:17.856 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6 2024-09-18T08:17:17.881 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:17:17.881 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:17:17.882 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6' 2024-09-18T08:17:17.925 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid 779349ca-7593-11ef-bceb-c7b262605968 -- ceph osd reweight 6 1 2024-09-18T08:17:17.982 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3 2024-09-18T08:17:18.043 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:17:18.044 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:17:18.044 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3' 2024-09-18T08:17:18.145 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3 2024-09-18T08:17:18.202 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:17:18.203 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:17:18.204 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3' 2024-09-18T08:17:18.290 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/779349ca-7593-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T08:17:18.305 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2 2024-09-18T08:17:18.343 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/779349ca-7593-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T08:17:18.382 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:17:18.383 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:17:18.383 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2' 2024-09-18T08:17:18.484 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6 2024-09-18T08:17:18.512 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:17:18.512 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:17:18.513 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6' 2024-09-18T08:17:18.613 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0 2024-09-18T08:17:18.671 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:17:18.672 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:17:18.673 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0' 2024-09-18T08:17:18.730 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:17:18 smithi007 ceph-mon[39249]: pgmap v723: 158 pgs: 2 active+remapped+backfilling, 2 active+remapped+backfill_toofull, 154 active+clean; 3.6 GiB data, 13 GiB used, 523 GiB / 536 GiB avail; 639 B/s rd, 0 B/s wr, 844 op/s; 9656/177357 objects misplaced (5.444%); 19 MiB/s, 305 objects/s recovering 2024-09-18T08:17:18.735 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:17:18 smithi007 ceph-mon[29459]: pgmap v723: 158 pgs: 2 active+remapped+backfilling, 2 active+remapped+backfill_toofull, 154 active+clean; 3.6 GiB data, 13 GiB used, 523 GiB / 536 GiB avail; 639 B/s rd, 0 B/s wr, 844 op/s; 9656/177357 objects misplaced (5.444%); 19 MiB/s, 305 objects/s recovering 2024-09-18T08:17:18.774 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5 2024-09-18T08:17:18.801 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:17:18.801 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:17:18.802 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5' 2024-09-18T08:17:18.882 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:17:18 smithi050 ceph-mon[32371]: pgmap v723: 158 pgs: 2 active+remapped+backfilling, 2 active+remapped+backfill_toofull, 154 active+clean; 3.6 GiB data, 13 GiB used, 523 GiB / 536 GiB avail; 639 B/s rd, 0 B/s wr, 844 op/s; 9656/177357 objects misplaced (5.444%); 19 MiB/s, 305 objects/s recovering 2024-09-18T08:17:18.902 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6 2024-09-18T08:17:18.930 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:17:18.930 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:17:18.931 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6' 2024-09-18T08:17:19.031 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2 2024-09-18T08:17:19.081 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:17:19.082 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:17:19.082 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2' 2024-09-18T08:17:19.183 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0 2024-09-18T08:17:19.221 DEBUG:teuthology.orchestra.run.smithi050:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid 779349ca-7593-11ef-bceb-c7b262605968 -- ceph daemon osd.7 dump_historic_ops 2024-09-18T08:17:19.227 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:17:19.228 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:17:19.228 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0' 2024-09-18T08:17:19.329 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5 2024-09-18T08:17:19.356 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:17:19.356 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:17:19.357 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5' 2024-09-18T08:17:19.459 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0 2024-09-18T08:17:19.530 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:17:19 smithi050 ceph-mon[32371]: purged_snaps scrub starts 2024-09-18T08:17:19.530 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:17:19 smithi050 ceph-mon[32371]: purged_snaps scrub ok 2024-09-18T08:17:19.530 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:17:19 smithi050 ceph-mon[32371]: osdmap e374: 8 total, 8 up, 6 in 2024-09-18T08:17:19.531 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:17:19 smithi050 ceph-mon[32371]: from='mgr.24470 172.21.15.7:0/2594390261' entity='mgr.y' 2024-09-18T08:17:19.531 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:17:19 smithi050 ceph-mon[32371]: from='mgr.24470 172.21.15.7:0/2594390261' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-18T08:17:19.531 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:17:19 smithi050 ceph-mon[32371]: from='mgr.24470 172.21.15.7:0/2594390261' entity='mgr.y' 2024-09-18T08:17:19.531 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:17:19 smithi050 ceph-mon[32371]: from='mgr.24470 172.21.15.7:0/2594390261' entity='mgr.y' 2024-09-18T08:17:19.533 INFO:teuthology.orchestra.run.smithi050.stderr:Inferring config /var/lib/ceph/779349ca-7593-11ef-bceb-c7b262605968/mon.b/config 2024-09-18T08:17:19.550 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:17:19.551 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:17:19.551 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0' 2024-09-18T08:17:19.597 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/779349ca-7593-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T08:17:19.652 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6 2024-09-18T08:17:19.677 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:17:19.678 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:17:19.678 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6' 2024-09-18T08:17:19.730 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:17:19 smithi007 ceph-mon[29459]: purged_snaps scrub starts 2024-09-18T08:17:19.730 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:17:19 smithi007 ceph-mon[29459]: purged_snaps scrub ok 2024-09-18T08:17:19.730 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:17:19 smithi007 ceph-mon[29459]: osdmap e374: 8 total, 8 up, 6 in 2024-09-18T08:17:19.730 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:17:19 smithi007 ceph-mon[29459]: from='mgr.24470 172.21.15.7:0/2594390261' entity='mgr.y' 2024-09-18T08:17:19.730 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:17:19 smithi007 ceph-mon[29459]: from='mgr.24470 172.21.15.7:0/2594390261' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-18T08:17:19.730 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:17:19 smithi007 ceph-mon[29459]: from='mgr.24470 172.21.15.7:0/2594390261' entity='mgr.y' 2024-09-18T08:17:19.731 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:17:19 smithi007 ceph-mon[29459]: from='mgr.24470 172.21.15.7:0/2594390261' entity='mgr.y' 2024-09-18T08:17:19.732 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:17:19 smithi007 ceph-mon[39249]: purged_snaps scrub starts 2024-09-18T08:17:19.732 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:17:19 smithi007 ceph-mon[39249]: purged_snaps scrub ok 2024-09-18T08:17:19.732 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:17:19 smithi007 ceph-mon[39249]: osdmap e374: 8 total, 8 up, 6 in 2024-09-18T08:17:19.732 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:17:19 smithi007 ceph-mon[39249]: from='mgr.24470 172.21.15.7:0/2594390261' entity='mgr.y' 2024-09-18T08:17:19.732 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:17:19 smithi007 ceph-mon[39249]: from='mgr.24470 172.21.15.7:0/2594390261' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-18T08:17:19.732 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:17:19 smithi007 ceph-mon[39249]: from='mgr.24470 172.21.15.7:0/2594390261' entity='mgr.y' 2024-09-18T08:17:19.732 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:17:19 smithi007 ceph-mon[39249]: from='mgr.24470 172.21.15.7:0/2594390261' entity='mgr.y' 2024-09-18T08:17:19.779 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2 2024-09-18T08:17:19.799 INFO:journalctl@ceph.iscsi.iscsi.a.smithi050.stdout:Sep 18 08:17:19 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-iscsi-iscsi-a[57782]: debug there is no tcmu-runner data available 2024-09-18T08:17:19.830 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:17:19.831 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:17:19.831 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2' 2024-09-18T08:17:19.932 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2 2024-09-18T08:17:19.970 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:17:19.971 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:17:19.972 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2' 2024-09-18T08:17:20.068 INFO:teuthology.orchestra.run.smithi050.stdout:ERROR: (22) Invalid argument 2024-09-18T08:17:20.068 INFO:teuthology.orchestra.run.smithi050.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-18T08:17:20.072 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3 2024-09-18T08:17:20.111 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:17:20.111 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:17:20.111 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3' 2024-09-18T08:17:20.212 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5 2024-09-18T08:17:20.241 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:17:20.241 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:17:20.242 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5' 2024-09-18T08:17:20.343 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7 2024-09-18T08:17:20.368 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:17:20.369 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:17:20.369 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7' 2024-09-18T08:17:20.470 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7 2024-09-18T08:17:20.498 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:17:20.499 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:17:20.499 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7' 2024-09-18T08:17:20.547 INFO:teuthology.orchestra.run.smithi007.stderr:nodeep-scrub is set 2024-09-18T08:17:20.600 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6 2024-09-18T08:17:20.628 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:17:20.628 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:17:20.629 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6' 2024-09-18T08:17:20.730 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5 2024-09-18T08:17:20.759 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:17:20.760 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:17:20.761 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5' 2024-09-18T08:17:20.803 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:17:20 smithi050 ceph-mon[32371]: pgmap v725: 158 pgs: 2 active+remapped+backfilling, 2 active+remapped+backfill_toofull, 154 active+clean; 3.4 GiB data, 13 GiB used, 524 GiB / 536 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.01k op/s; 9077/168555 objects misplaced (5.385%); 6.4 MiB/s, 102 objects/s recovering 2024-09-18T08:17:20.803 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:17:20 smithi050 ceph-mon[32371]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-18T08:17:20.803 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:17:20 smithi050 ceph-mon[32371]: from='client.? 172.21.15.7:0/1507270997' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-18T08:17:20.804 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:17:20 smithi050 ceph-mon[32371]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-18T08:17:20.847 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid 779349ca-7593-11ef-bceb-c7b262605968 -- ceph daemon osd.1 dump_ops_in_flight 2024-09-18T08:17:20.862 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5 2024-09-18T08:17:20.870 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:17:20 smithi007 ceph-mon[29459]: pgmap v725: 158 pgs: 2 active+remapped+backfilling, 2 active+remapped+backfill_toofull, 154 active+clean; 3.4 GiB data, 13 GiB used, 524 GiB / 536 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.01k op/s; 9077/168555 objects misplaced (5.385%); 6.4 MiB/s, 102 objects/s recovering 2024-09-18T08:17:20.871 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:17:20 smithi007 ceph-mon[29459]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-18T08:17:20.871 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:17:20 smithi007 ceph-mon[29459]: from='client.? 172.21.15.7:0/1507270997' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-18T08:17:20.871 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:17:20 smithi007 ceph-mon[29459]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-18T08:17:20.871 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:17:20 smithi007 ceph-mon[39249]: pgmap v725: 158 pgs: 2 active+remapped+backfilling, 2 active+remapped+backfill_toofull, 154 active+clean; 3.4 GiB data, 13 GiB used, 524 GiB / 536 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.01k op/s; 9077/168555 objects misplaced (5.385%); 6.4 MiB/s, 102 objects/s recovering 2024-09-18T08:17:20.871 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:17:20 smithi007 ceph-mon[39249]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-18T08:17:20.871 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:17:20 smithi007 ceph-mon[39249]: from='client.? 172.21.15.7:0/1507270997' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-18T08:17:20.871 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:17:20 smithi007 ceph-mon[39249]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-18T08:17:20.884 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:17:20.885 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:17:20.885 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5' 2024-09-18T08:17:20.986 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2 2024-09-18T08:17:21.011 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:17:21.012 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:17:21.012 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2' 2024-09-18T08:17:21.113 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5 2024-09-18T08:17:21.137 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:17:21.137 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:17:21.138 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5' 2024-09-18T08:17:21.239 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3 2024-09-18T08:17:21.265 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:17:21.266 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:17:21.266 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3' 2024-09-18T08:17:21.367 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1 2024-09-18T08:17:21.402 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:17:21.403 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:17:21.403 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1' 2024-09-18T08:17:21.504 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0 2024-09-18T08:17:21.539 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:17:21 smithi007 ceph-mon[29459]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-18T08:17:21.539 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:17:21.540 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:17:21.540 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0' 2024-09-18T08:17:21.641 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7 2024-09-18T08:17:21.667 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:17:21.668 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:17:21.668 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7' 2024-09-18T08:17:21.769 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4 2024-09-18T08:17:21.794 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:17:21.794 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:17:21.795 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4' 2024-09-18T08:17:21.896 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3 2024-09-18T08:17:21.905 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:17:21 smithi050 ceph-mon[32371]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-18T08:17:21.905 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:17:21 smithi050 ceph-mon[32371]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-18T08:17:21.905 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:17:21 smithi050 ceph-mon[32371]: osdmap e375: 8 total, 8 up, 6 in 2024-09-18T08:17:21.934 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:17:21.935 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:17:21.936 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3' 2024-09-18T08:17:21.980 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:17:21 smithi007 ceph-mon[29459]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-18T08:17:21.980 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:17:21 smithi007 ceph-mon[29459]: osdmap e375: 8 total, 8 up, 6 in 2024-09-18T08:17:21.980 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:17:21 smithi007 ceph-mon[39249]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-18T08:17:21.980 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:17:21 smithi007 ceph-mon[39249]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-18T08:17:21.980 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:17:21 smithi007 ceph-mon[39249]: osdmap e375: 8 total, 8 up, 6 in 2024-09-18T08:17:22.036 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6 2024-09-18T08:17:22.064 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:17:22.065 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:17:22.065 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6' 2024-09-18T08:17:22.166 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1 2024-09-18T08:17:22.208 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:17:22.209 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:17:22.209 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1' 2024-09-18T08:17:22.310 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3 2024-09-18T08:17:22.338 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:17:22.338 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:17:22.339 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3' 2024-09-18T08:17:22.439 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1 2024-09-18T08:17:22.480 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:17:22.481 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:17:22.481 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1' 2024-09-18T08:17:22.582 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5 2024-09-18T08:17:22.608 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:17:22.609 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:17:22.609 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5' 2024-09-18T08:17:22.674 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:17:22 smithi007 ceph-mon[29459]: pgmap v727: 158 pgs: 2 active+remapped+backfilling, 2 active+remapped+backfill_toofull, 154 active+clean; 3.3 GiB data, 13 GiB used, 524 GiB / 536 GiB avail; 738 B/s rd, 0 B/s wr, 954 op/s; 8099/163290 objects misplaced (4.960%); 15 MiB/s, 236 objects/s recovering 2024-09-18T08:17:22.677 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:17:22 smithi007 ceph-mon[39249]: pgmap v727: 158 pgs: 2 active+remapped+backfilling, 2 active+remapped+backfill_toofull, 154 active+clean; 3.3 GiB data, 13 GiB used, 524 GiB / 536 GiB avail; 738 B/s rd, 0 B/s wr, 954 op/s; 8099/163290 objects misplaced (4.960%); 15 MiB/s, 236 objects/s recovering 2024-09-18T08:17:22.710 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0 2024-09-18T08:17:22.771 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:17:22.772 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:17:22.772 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0' 2024-09-18T08:17:22.873 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6 2024-09-18T08:17:22.899 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:17:22.899 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:17:22.900 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6' 2024-09-18T08:17:22.905 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:17:22 smithi050 ceph-mon[32371]: pgmap v727: 158 pgs: 2 active+remapped+backfilling, 2 active+remapped+backfill_toofull, 154 active+clean; 3.3 GiB data, 13 GiB used, 524 GiB / 536 GiB avail; 738 B/s rd, 0 B/s wr, 954 op/s; 8099/163290 objects misplaced (4.960%); 15 MiB/s, 236 objects/s recovering 2024-09-18T08:17:23.001 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7 2024-09-18T08:17:23.026 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:17:23.027 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:17:23.027 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7' 2024-09-18T08:17:23.128 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3 2024-09-18T08:17:23.167 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:17:23.168 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:17:23.168 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3' 2024-09-18T08:17:23.269 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7 2024-09-18T08:17:23.295 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:17:23.296 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:17:23.296 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7' 2024-09-18T08:17:23.397 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6 2024-09-18T08:17:23.423 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:17:23.423 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:17:23.424 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6' 2024-09-18T08:17:23.525 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7 2024-09-18T08:17:23.548 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:17:23.549 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:17:23.550 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7' 2024-09-18T08:17:23.616 INFO:teuthology.orchestra.run.smithi007.stdout:osd.0: {} 2024-09-18T08:17:23.617 INFO:teuthology.orchestra.run.smithi007.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-18T08:17:23.650 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0 2024-09-18T08:17:23.657 INFO:teuthology.orchestra.run.smithi007.stdout:osd.1: {} 2024-09-18T08:17:23.657 INFO:teuthology.orchestra.run.smithi007.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-18T08:17:23.696 INFO:teuthology.orchestra.run.smithi007.stdout:osd.2: {} 2024-09-18T08:17:23.696 INFO:teuthology.orchestra.run.smithi007.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-18T08:17:23.701 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:17:23.701 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:17:23.702 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0' 2024-09-18T08:17:23.730 INFO:teuthology.orchestra.run.smithi007.stdout:osd.3: {} 2024-09-18T08:17:23.731 INFO:teuthology.orchestra.run.smithi007.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-18T08:17:23.766 INFO:teuthology.orchestra.run.smithi007.stdout:osd.4: {} 2024-09-18T08:17:23.766 INFO:teuthology.orchestra.run.smithi007.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-18T08:17:23.803 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5 2024-09-18T08:17:23.803 INFO:teuthology.orchestra.run.smithi007.stdout:osd.5: {} 2024-09-18T08:17:23.804 INFO:teuthology.orchestra.run.smithi007.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-18T08:17:23.827 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:17:23.828 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:17:23.828 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5' 2024-09-18T08:17:23.840 INFO:teuthology.orchestra.run.smithi007.stdout:osd.6: {} 2024-09-18T08:17:23.841 INFO:teuthology.orchestra.run.smithi007.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-18T08:17:23.892 INFO:teuthology.orchestra.run.smithi007.stdout:osd.7: {} 2024-09-18T08:17:23.892 INFO:teuthology.orchestra.run.smithi007.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-18T08:17:23.929 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4 2024-09-18T08:17:23.952 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:17:23.953 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:17:23.953 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4' 2024-09-18T08:17:24.054 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6 2024-09-18T08:17:24.080 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:17:24.081 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:17:24.086 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6' 2024-09-18T08:17:24.187 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2 2024-09-18T08:17:24.233 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:17:24.233 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:17:24.234 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2' 2024-09-18T08:17:24.334 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2 2024-09-18T08:17:24.376 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:17:24.377 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:17:24.377 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2' 2024-09-18T08:17:24.478 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1 2024-09-18T08:17:24.519 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:17:24.520 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:17:24.520 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1' 2024-09-18T08:17:24.621 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5 2024-09-18T08:17:24.647 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:17:24.647 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:17:24.647 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5' 2024-09-18T08:17:24.730 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:17:24 smithi007 ceph-mon[39249]: pgmap v728: 158 pgs: 2 active+remapped+backfilling, 2 active+remapped+backfill_toofull, 154 active+clean; 3.3 GiB data, 12 GiB used, 524 GiB / 536 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 694 op/s; 7836/160683 objects misplaced (4.877%); 6.4 MiB/s, 102 objects/s recovering 2024-09-18T08:17:24.730 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:17:24 smithi007 ceph-mon[39249]: from='client.? 172.21.15.7:0/106516978' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-18T08:17:24.730 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:17:24 smithi007 ceph-mon[39249]: from='mgr.24470 172.21.15.7:0/2594390261' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-18T08:17:24.730 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:17:24 smithi007 ceph-mon[39249]: from='mgr.24470 172.21.15.7:0/2594390261' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-18T08:17:24.730 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:17:24 smithi007 ceph-mon[39249]: from='mgr.24470 172.21.15.7:0/2594390261' entity='mgr.y' 2024-09-18T08:17:24.731 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:17:24 smithi007 ceph-mon[39249]: osdmap e376: 8 total, 8 up, 6 in 2024-09-18T08:17:24.731 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:17:24 smithi007 ceph-mon[39249]: from='client.? 172.21.15.7:0/3168136186' entity='client.admin' cmd={"prefix": "osd reweight", "id": 6, "weight": 1.0} : dispatch 2024-09-18T08:17:24.731 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:17:24 smithi007 ceph-mon[39249]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 6, "weight": 1.0} : dispatch 2024-09-18T08:17:24.731 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:17:24 smithi007 ceph-mon[29459]: pgmap v728: 158 pgs: 2 active+remapped+backfilling, 2 active+remapped+backfill_toofull, 154 active+clean; 3.3 GiB data, 12 GiB used, 524 GiB / 536 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 694 op/s; 7836/160683 objects misplaced (4.877%); 6.4 MiB/s, 102 objects/s recovering 2024-09-18T08:17:24.731 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:17:24 smithi007 ceph-mon[29459]: from='client.? 172.21.15.7:0/106516978' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-18T08:17:24.731 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:17:24 smithi007 ceph-mon[29459]: from='mgr.24470 172.21.15.7:0/2594390261' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-18T08:17:24.731 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:17:24 smithi007 ceph-mon[29459]: from='mgr.24470 172.21.15.7:0/2594390261' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-18T08:17:24.731 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:17:24 smithi007 ceph-mon[29459]: from='mgr.24470 172.21.15.7:0/2594390261' entity='mgr.y' 2024-09-18T08:17:24.731 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:17:24 smithi007 ceph-mon[29459]: osdmap e376: 8 total, 8 up, 6 in 2024-09-18T08:17:24.731 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:17:24 smithi007 ceph-mon[29459]: from='client.? 172.21.15.7:0/3168136186' entity='client.admin' cmd={"prefix": "osd reweight", "id": 6, "weight": 1.0} : dispatch 2024-09-18T08:17:24.732 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:17:24 smithi007 ceph-mon[29459]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 6, "weight": 1.0} : dispatch 2024-09-18T08:17:24.748 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4 2024-09-18T08:17:24.770 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:17:24.771 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:17:24.771 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4' 2024-09-18T08:17:24.830 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/779349ca-7593-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T08:17:24.872 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6 2024-09-18T08:17:24.895 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:17:24.895 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:17:24.895 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6' 2024-09-18T08:17:24.905 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:17:24 smithi050 ceph-mon[32371]: pgmap v728: 158 pgs: 2 active+remapped+backfilling, 2 active+remapped+backfill_toofull, 154 active+clean; 3.3 GiB data, 12 GiB used, 524 GiB / 536 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 694 op/s; 7836/160683 objects misplaced (4.877%); 6.4 MiB/s, 102 objects/s recovering 2024-09-18T08:17:24.906 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:17:24 smithi050 ceph-mon[32371]: from='client.? 172.21.15.7:0/106516978' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-18T08:17:24.906 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:17:24 smithi050 ceph-mon[32371]: from='mgr.24470 172.21.15.7:0/2594390261' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-18T08:17:24.906 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:17:24 smithi050 ceph-mon[32371]: from='mgr.24470 172.21.15.7:0/2594390261' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-18T08:17:24.906 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:17:24 smithi050 ceph-mon[32371]: from='mgr.24470 172.21.15.7:0/2594390261' entity='mgr.y' 2024-09-18T08:17:24.906 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:17:24 smithi050 ceph-mon[32371]: osdmap e376: 8 total, 8 up, 6 in 2024-09-18T08:17:24.906 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:17:24 smithi050 ceph-mon[32371]: from='client.? 172.21.15.7:0/3168136186' entity='client.admin' cmd={"prefix": "osd reweight", "id": 6, "weight": 1.0} : dispatch 2024-09-18T08:17:24.906 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:17:24 smithi050 ceph-mon[32371]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 6, "weight": 1.0} : dispatch 2024-09-18T08:17:24.997 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0 2024-09-18T08:17:25.021 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:17:25.022 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:17:25.022 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0' 2024-09-18T08:17:25.123 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5 2024-09-18T08:17:25.146 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:17:25.147 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:17:25.147 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5' 2024-09-18T08:17:25.248 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2 2024-09-18T08:17:25.271 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:17:25.271 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:17:25.272 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2' 2024-09-18T08:17:25.279 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid 779349ca-7593-11ef-bceb-c7b262605968 -- ceph osd unset noscrub 2024-09-18T08:17:25.373 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6 2024-09-18T08:17:25.381 INFO:teuthology.orchestra.run.smithi007.stdout:{ 2024-09-18T08:17:25.381 INFO:teuthology.orchestra.run.smithi007.stdout: "ops": [], 2024-09-18T08:17:25.381 INFO:teuthology.orchestra.run.smithi007.stdout: "num_ops": 0 2024-09-18T08:17:25.381 INFO:teuthology.orchestra.run.smithi007.stdout:} 2024-09-18T08:17:25.396 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:17:25.396 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:17:25.397 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6' 2024-09-18T08:17:25.411 INFO:teuthology.orchestra.run.smithi007.stderr:reweighted osd.6 to 1 (10000) 2024-09-18T08:17:25.498 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3 2024-09-18T08:17:25.543 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:17:25.543 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:17:25.544 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3' 2024-09-18T08:17:25.645 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0 2024-09-18T08:17:25.683 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:17:25.684 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:17:25.684 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0' 2024-09-18T08:17:25.785 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7 2024-09-18T08:17:25.808 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:17:25.809 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:17:25.809 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7' 2024-09-18T08:17:25.910 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6 2024-09-18T08:17:25.932 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:17:25.933 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:17:25.933 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6' 2024-09-18T08:17:26.034 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1 2024-09-18T08:17:26.058 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:17:26.059 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:17:26.059 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1' 2024-09-18T08:17:26.160 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3 2024-09-18T08:17:26.183 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:17:26.184 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:17:26.184 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3' 2024-09-18T08:17:26.285 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0 2024-09-18T08:17:26.308 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:17:26.309 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:17:26.309 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0' 2024-09-18T08:17:26.378 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid 779349ca-7593-11ef-bceb-c7b262605968 -- ceph osd reweight 5 1 2024-09-18T08:17:26.410 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5 2024-09-18T08:17:26.435 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:17:26.436 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:17:26.436 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5' 2024-09-18T08:17:26.537 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3 2024-09-18T08:17:26.587 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:17:26.588 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:17:26.588 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3' 2024-09-18T08:17:26.654 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid 779349ca-7593-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-18T08:17:26.689 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5 2024-09-18T08:17:26.697 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:17:26 smithi007 ceph-mon[29459]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]': finished 2024-09-18T08:17:26.698 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:17:26 smithi007 ceph-mon[29459]: osdmap e377: 8 total, 8 up, 6 in 2024-09-18T08:17:26.698 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:17:26 smithi007 ceph-mon[29459]: pgmap v731: 158 pgs: 1 active+remapped+backfilling, 1 peering, 2 active+remapped+backfill_toofull, 154 active+clean; 3.0 GiB data, 12 GiB used, 524 GiB / 536 GiB avail; 851 B/s rd, 0 B/s wr, 1.07k op/s; 6372/149238 objects misplaced (4.270%); 20 MiB/s, 319 objects/s recovering 2024-09-18T08:17:26.700 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:17:26 smithi007 ceph-mon[39249]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]': finished 2024-09-18T08:17:26.700 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:17:26 smithi007 ceph-mon[39249]: osdmap e377: 8 total, 8 up, 6 in 2024-09-18T08:17:26.700 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:17:26 smithi007 ceph-mon[39249]: pgmap v731: 158 pgs: 1 active+remapped+backfilling, 1 peering, 2 active+remapped+backfill_toofull, 154 active+clean; 3.0 GiB data, 12 GiB used, 524 GiB / 536 GiB avail; 851 B/s rd, 0 B/s wr, 1.07k op/s; 6372/149238 objects misplaced (4.270%); 20 MiB/s, 319 objects/s recovering 2024-09-18T08:17:26.714 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:17:26.715 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:17:26.715 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5' 2024-09-18T08:17:26.816 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5 2024-09-18T08:17:26.843 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:17:26.844 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:17:26.844 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5' 2024-09-18T08:17:26.896 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid 779349ca-7593-11ef-bceb-c7b262605968 -- ceph daemon osd.1 dump_blocked_ops 2024-09-18T08:17:26.905 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:17:26 smithi050 ceph-mon[32371]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]': finished 2024-09-18T08:17:26.905 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:17:26 smithi050 ceph-mon[32371]: osdmap e377: 8 total, 8 up, 6 in 2024-09-18T08:17:26.905 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:17:26 smithi050 ceph-mon[32371]: pgmap v731: 158 pgs: 1 active+remapped+backfilling, 1 peering, 2 active+remapped+backfill_toofull, 154 active+clean; 3.0 GiB data, 12 GiB used, 524 GiB / 536 GiB avail; 851 B/s rd, 0 B/s wr, 1.07k op/s; 6372/149238 objects misplaced (4.270%); 20 MiB/s, 319 objects/s recovering 2024-09-18T08:17:26.945 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5 2024-09-18T08:17:26.966 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/779349ca-7593-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T08:17:26.970 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:17:26.970 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:17:26.971 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5' 2024-09-18T08:17:26.984 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Sep 18 08:17:26 smithi007 ceph-779349ca-7593-11ef-bceb-c7b262605968-mgr-y[29688]: ::ffff:172.21.15.50 - - [18/Sep/2024:08:17:26] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-09-18T08:17:27.064 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/779349ca-7593-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T08:17:27.071 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0 2024-09-18T08:17:27.136 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:17:27.137 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:17:27.137 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0' 2024-09-18T08:17:27.238 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6 2024-09-18T08:17:27.265 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:17:27.265 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:17:27.265 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6' 2024-09-18T08:17:27.366 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4 2024-09-18T08:17:27.390 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:17:27.391 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:17:27.391 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4' 2024-09-18T08:17:27.492 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7 2024-09-18T08:17:27.518 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:17:27.519 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:17:27.519 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7' 2024-09-18T08:17:27.561 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:17:27 smithi007 ceph-mon[29459]: osdmap e378: 8 total, 8 up, 6 in 2024-09-18T08:17:27.561 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:17:27 smithi007 ceph-mon[39249]: osdmap e378: 8 total, 8 up, 6 in 2024-09-18T08:17:27.620 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3 2024-09-18T08:17:27.677 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:17:27.678 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:17:27.679 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3' 2024-09-18T08:17:27.780 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2 2024-09-18T08:17:27.825 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:17:27.826 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:17:27.826 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2' 2024-09-18T08:17:27.905 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:17:27 smithi050 ceph-mon[32371]: osdmap e378: 8 total, 8 up, 6 in 2024-09-18T08:17:27.926 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4 2024-09-18T08:17:27.949 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:17:27.950 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:17:27.950 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4' 2024-09-18T08:17:28.051 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6 2024-09-18T08:17:28.075 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:17:28.075 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:17:28.076 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6' 2024-09-18T08:17:28.146 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/779349ca-7593-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T08:17:28.177 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6 2024-09-18T08:17:28.201 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:17:28.202 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:17:28.203 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6' 2024-09-18T08:17:28.232 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/779349ca-7593-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T08:17:28.304 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4 2024-09-18T08:17:28.330 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:17:28.330 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:17:28.331 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4' 2024-09-18T08:17:28.430 INFO:teuthology.orchestra.run.smithi007.stderr:reweighted osd.5 to 1 (10000) 2024-09-18T08:17:28.431 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6 2024-09-18T08:17:28.434 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:17:28 smithi007 ceph-mon[29459]: pgmap v733: 158 pgs: 1 active+remapped+backfilling, 1 peering, 2 active+remapped+backfill_toofull, 154 active+clean; 3.0 GiB data, 12 GiB used, 524 GiB / 536 GiB avail; 851 B/s rd, 0 B/s wr, 780 op/s; 6372/149238 objects misplaced (4.270%); 11 MiB/s, 182 objects/s recovering 2024-09-18T08:17:28.435 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:17:28 smithi007 ceph-mon[29459]: from='client.? 172.21.15.7:0/740523084' entity='client.admin' cmd={"prefix": "osd reweight", "id": 5, "weight": 1.0} : dispatch 2024-09-18T08:17:28.435 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:17:28 smithi007 ceph-mon[29459]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 5, "weight": 1.0} : dispatch 2024-09-18T08:17:28.435 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:17:28 smithi007 ceph-mon[39249]: pgmap v733: 158 pgs: 1 active+remapped+backfilling, 1 peering, 2 active+remapped+backfill_toofull, 154 active+clean; 3.0 GiB data, 12 GiB used, 524 GiB / 536 GiB avail; 851 B/s rd, 0 B/s wr, 780 op/s; 6372/149238 objects misplaced (4.270%); 11 MiB/s, 182 objects/s recovering 2024-09-18T08:17:28.458 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:17:28.458 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:17:28.459 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6' 2024-09-18T08:17:28.560 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7 2024-09-18T08:17:28.586 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:17:28.587 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:17:28.587 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7' 2024-09-18T08:17:28.688 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6 2024-09-18T08:17:28.715 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:17:28.716 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:17:28.716 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6' 2024-09-18T08:17:28.730 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:17:28 smithi007 ceph-mon[39249]: from='client.? 172.21.15.7:0/740523084' entity='client.admin' cmd={"prefix": "osd reweight", "id": 5, "weight": 1.0} : dispatch 2024-09-18T08:17:28.730 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:17:28 smithi007 ceph-mon[39249]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 5, "weight": 1.0} : dispatch 2024-09-18T08:17:28.817 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6 2024-09-18T08:17:28.842 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:17:28.843 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:17:28.844 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6' 2024-09-18T08:17:28.905 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:17:28 smithi050 ceph-mon[32371]: pgmap v733: 158 pgs: 1 active+remapped+backfilling, 1 peering, 2 active+remapped+backfill_toofull, 154 active+clean; 3.0 GiB data, 12 GiB used, 524 GiB / 536 GiB avail; 851 B/s rd, 0 B/s wr, 780 op/s; 6372/149238 objects misplaced (4.270%); 11 MiB/s, 182 objects/s recovering 2024-09-18T08:17:28.906 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:17:28 smithi050 ceph-mon[32371]: from='client.? 172.21.15.7:0/740523084' entity='client.admin' cmd={"prefix": "osd reweight", "id": 5, "weight": 1.0} : dispatch 2024-09-18T08:17:28.906 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:17:28 smithi050 ceph-mon[32371]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 5, "weight": 1.0} : dispatch 2024-09-18T08:17:28.945 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0 2024-09-18T08:17:28.992 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:17:28.992 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:17:28.993 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0' 2024-09-18T08:17:29.093 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4 2024-09-18T08:17:29.121 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:17:29.122 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:17:29.122 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4' 2024-09-18T08:17:29.223 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1 2024-09-18T08:17:29.261 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:17:29.261 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:17:29.262 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1' 2024-09-18T08:17:29.363 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5 2024-09-18T08:17:29.389 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:17:29.390 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:17:29.390 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5' 2024-09-18T08:17:29.436 INFO:teuthology.orchestra.run.smithi007.stderr:noscrub is unset 2024-09-18T08:17:29.491 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7 2024-09-18T08:17:29.516 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:17:29.517 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:17:29.517 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7' 2024-09-18T08:17:29.618 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2 2024-09-18T08:17:29.658 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:17:29.659 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:17:29.659 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2' 2024-09-18T08:17:29.710 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:17:29 smithi007 ceph-mon[39249]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]': finished 2024-09-18T08:17:29.710 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:17:29 smithi007 ceph-mon[39249]: osdmap e379: 8 total, 8 up, 6 in 2024-09-18T08:17:29.710 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:17:29 smithi007 ceph-mon[39249]: from='client.? 172.21.15.7:0/2603512712' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-18T08:17:29.711 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:17:29 smithi007 ceph-mon[39249]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-18T08:17:29.711 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:17:29 smithi007 ceph-mon[29459]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]': finished 2024-09-18T08:17:29.711 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:17:29 smithi007 ceph-mon[29459]: osdmap e379: 8 total, 8 up, 6 in 2024-09-18T08:17:29.711 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:17:29 smithi007 ceph-mon[29459]: from='client.? 172.21.15.7:0/2603512712' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-18T08:17:29.711 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:17:29 smithi007 ceph-mon[29459]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-18T08:17:29.760 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7 2024-09-18T08:17:29.786 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:17:29.786 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:17:29.787 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7' 2024-09-18T08:17:29.803 INFO:teuthology.orchestra.run.smithi007.stdout:osd.0: {} 2024-09-18T08:17:29.804 INFO:teuthology.orchestra.run.smithi007.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-18T08:17:29.838 INFO:teuthology.orchestra.run.smithi007.stdout:osd.1: {} 2024-09-18T08:17:29.839 INFO:teuthology.orchestra.run.smithi007.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-18T08:17:29.882 INFO:teuthology.orchestra.run.smithi007.stdout:osd.2: {} 2024-09-18T08:17:29.882 INFO:teuthology.orchestra.run.smithi007.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-18T08:17:29.888 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3 2024-09-18T08:17:29.905 INFO:journalctl@ceph.iscsi.iscsi.a.smithi050.stdout:Sep 18 08:17:29 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-iscsi-iscsi-a[57782]: debug there is no tcmu-runner data available 2024-09-18T08:17:29.905 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:17:29 smithi050 ceph-mon[32371]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]': finished 2024-09-18T08:17:29.905 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:17:29 smithi050 ceph-mon[32371]: osdmap e379: 8 total, 8 up, 6 in 2024-09-18T08:17:29.905 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:17:29 smithi050 ceph-mon[32371]: from='client.? 172.21.15.7:0/2603512712' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-18T08:17:29.906 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:17:29 smithi050 ceph-mon[32371]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-18T08:17:29.927 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:17:29.928 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:17:29.928 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3' 2024-09-18T08:17:29.931 INFO:teuthology.orchestra.run.smithi007.stdout:osd.3: {} 2024-09-18T08:17:29.932 INFO:teuthology.orchestra.run.smithi007.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-18T08:17:29.975 INFO:teuthology.orchestra.run.smithi007.stdout:osd.4: {} 2024-09-18T08:17:29.976 INFO:teuthology.orchestra.run.smithi007.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-18T08:17:30.011 INFO:teuthology.orchestra.run.smithi007.stdout:osd.5: {} 2024-09-18T08:17:30.011 INFO:teuthology.orchestra.run.smithi007.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-18T08:17:30.018 INFO:teuthology.orchestra.run.smithi007.stdout:ERROR: (22) Invalid argument 2024-09-18T08:17:30.018 INFO:teuthology.orchestra.run.smithi007.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-18T08:17:30.029 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7 2024-09-18T08:17:30.050 INFO:teuthology.orchestra.run.smithi007.stdout:osd.6: {} 2024-09-18T08:17:30.050 INFO:teuthology.orchestra.run.smithi007.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-18T08:17:30.053 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:17:30.054 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:17:30.054 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7' 2024-09-18T08:17:30.089 INFO:teuthology.orchestra.run.smithi007.stdout:osd.7: {} 2024-09-18T08:17:30.089 INFO:teuthology.orchestra.run.smithi007.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-18T08:17:30.155 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3 2024-09-18T08:17:30.191 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:17:30.191 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:17:30.192 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3' 2024-09-18T08:17:30.293 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3 2024-09-18T08:17:30.332 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:17:30.333 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:17:30.334 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3' 2024-09-18T08:17:30.434 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1 2024-09-18T08:17:30.475 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:17:30.475 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:17:30.476 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1' 2024-09-18T08:17:30.577 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5 2024-09-18T08:17:30.604 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:17:30.604 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:17:30.605 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5' 2024-09-18T08:17:30.705 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7 2024-09-18T08:17:30.722 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:17:30 smithi007 ceph-mon[29459]: pgmap v735: 158 pgs: 1 active+remapped+backfilling, 1 peering, 2 active+remapped+backfill_toofull, 154 active+clean; 2.9 GiB data, 11 GiB used, 525 GiB / 536 GiB avail; 1020 B/s rd, 0 B/s wr, 822 op/s; 5909/142095 objects misplaced (4.158%); 14 MiB/s, 218 objects/s recovering 2024-09-18T08:17:30.722 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:17:30 smithi007 ceph-mon[29459]: from='client.? 172.21.15.7:0/2603512712' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-18T08:17:30.723 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:17:30 smithi007 ceph-mon[29459]: osdmap e380: 8 total, 8 up, 6 in 2024-09-18T08:17:30.723 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:17:30 smithi007 ceph-mon[29459]: 6.18 scrub starts 2024-09-18T08:17:30.723 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:17:30 smithi007 ceph-mon[29459]: 6.18 scrub ok 2024-09-18T08:17:30.723 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:17:30 smithi007 ceph-mon[29459]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-18T08:17:30.723 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:17:30 smithi007 ceph-mon[29459]: from='client.? 172.21.15.7:0/4172383751' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-18T08:17:30.723 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:17:30 smithi007 ceph-mon[29459]: 5.1f scrub starts 2024-09-18T08:17:30.723 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:17:30 smithi007 ceph-mon[29459]: 5.1f scrub ok 2024-09-18T08:17:30.723 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:17:30 smithi007 ceph-mon[39249]: pgmap v735: 158 pgs: 1 active+remapped+backfilling, 1 peering, 2 active+remapped+backfill_toofull, 154 active+clean; 2.9 GiB data, 11 GiB used, 525 GiB / 536 GiB avail; 1020 B/s rd, 0 B/s wr, 822 op/s; 5909/142095 objects misplaced (4.158%); 14 MiB/s, 218 objects/s recovering 2024-09-18T08:17:30.723 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:17:30 smithi007 ceph-mon[39249]: from='client.? 172.21.15.7:0/2603512712' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-18T08:17:30.723 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:17:30 smithi007 ceph-mon[39249]: osdmap e380: 8 total, 8 up, 6 in 2024-09-18T08:17:30.723 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:17:30 smithi007 ceph-mon[39249]: 6.18 scrub starts 2024-09-18T08:17:30.724 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:17:30 smithi007 ceph-mon[39249]: 6.18 scrub ok 2024-09-18T08:17:30.724 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:17:30 smithi007 ceph-mon[39249]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-18T08:17:30.724 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:17:30 smithi007 ceph-mon[39249]: from='client.? 172.21.15.7:0/4172383751' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-18T08:17:30.724 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:17:30 smithi007 ceph-mon[39249]: 5.1f scrub starts 2024-09-18T08:17:30.724 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:17:30 smithi007 ceph-mon[39249]: 5.1f scrub ok 2024-09-18T08:17:30.732 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:17:30.733 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:17:30.733 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7' 2024-09-18T08:17:30.834 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3 2024-09-18T08:17:30.873 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:17:30.873 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:17:30.874 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3' 2024-09-18T08:17:30.905 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:17:30 smithi050 ceph-mon[32371]: pgmap v735: 158 pgs: 1 active+remapped+backfilling, 1 peering, 2 active+remapped+backfill_toofull, 154 active+clean; 2.9 GiB data, 11 GiB used, 525 GiB / 536 GiB avail; 1020 B/s rd, 0 B/s wr, 822 op/s; 5909/142095 objects misplaced (4.158%); 14 MiB/s, 218 objects/s recovering 2024-09-18T08:17:30.905 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:17:30 smithi050 ceph-mon[32371]: from='client.? 172.21.15.7:0/2603512712' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-18T08:17:30.905 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:17:30 smithi050 ceph-mon[32371]: osdmap e380: 8 total, 8 up, 6 in 2024-09-18T08:17:30.905 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:17:30 smithi050 ceph-mon[32371]: 6.18 scrub starts 2024-09-18T08:17:30.905 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:17:30 smithi050 ceph-mon[32371]: 6.18 scrub ok 2024-09-18T08:17:30.905 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:17:30 smithi050 ceph-mon[32371]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-18T08:17:30.906 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:17:30 smithi050 ceph-mon[32371]: from='client.? 172.21.15.7:0/4172383751' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-18T08:17:30.906 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:17:30 smithi050 ceph-mon[32371]: 5.1f scrub starts 2024-09-18T08:17:30.906 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:17:30 smithi050 ceph-mon[32371]: 5.1f scrub ok 2024-09-18T08:17:30.974 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0 2024-09-18T08:17:31.017 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:17:31.022 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:17:31.022 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0' 2024-09-18T08:17:31.123 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2 2024-09-18T08:17:31.167 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:17:31.167 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:17:31.168 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2' 2024-09-18T08:17:31.269 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7 2024-09-18T08:17:31.294 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:17:31.294 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:17:31.294 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7' 2024-09-18T08:17:31.395 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6 2024-09-18T08:17:31.421 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:17:31.422 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:17:31.422 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6' 2024-09-18T08:17:31.525 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4 2024-09-18T08:17:31.550 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:17:31.551 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:17:31.551 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4' 2024-09-18T08:17:31.613 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:17:31 smithi007 ceph-mon[29459]: 4.f scrub starts 2024-09-18T08:17:31.613 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:17:31 smithi007 ceph-mon[29459]: 4.f scrub ok 2024-09-18T08:17:31.613 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:17:31 smithi007 ceph-mon[29459]: 5.1b scrub starts 2024-09-18T08:17:31.613 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:17:31 smithi007 ceph-mon[29459]: 5.1b scrub ok 2024-09-18T08:17:31.613 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:17:31 smithi007 ceph-mon[29459]: 3.13 scrub starts 2024-09-18T08:17:31.613 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:17:31 smithi007 ceph-mon[29459]: 3.13 scrub ok 2024-09-18T08:17:31.614 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:17:31 smithi007 ceph-mon[39249]: 4.f scrub starts 2024-09-18T08:17:31.614 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:17:31 smithi007 ceph-mon[39249]: 4.f scrub ok 2024-09-18T08:17:31.614 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:17:31 smithi007 ceph-mon[39249]: 5.1b scrub starts 2024-09-18T08:17:31.614 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:17:31 smithi007 ceph-mon[39249]: 5.1b scrub ok 2024-09-18T08:17:31.614 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:17:31 smithi007 ceph-mon[39249]: 3.13 scrub starts 2024-09-18T08:17:31.614 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:17:31 smithi007 ceph-mon[39249]: 3.13 scrub ok 2024-09-18T08:17:31.652 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2 2024-09-18T08:17:31.669 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid 779349ca-7593-11ef-bceb-c7b262605968 -- ceph daemon osd.1 dump_historic_ops 2024-09-18T08:17:31.692 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:17:31.693 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:17:31.694 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2' 2024-09-18T08:17:31.795 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7 2024-09-18T08:17:31.818 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:17:31.819 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:17:31.819 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7' 2024-09-18T08:17:31.905 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:17:31 smithi050 ceph-mon[32371]: 4.f scrub starts 2024-09-18T08:17:31.905 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:17:31 smithi050 ceph-mon[32371]: 4.f scrub ok 2024-09-18T08:17:31.905 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:17:31 smithi050 ceph-mon[32371]: 5.1b scrub starts 2024-09-18T08:17:31.905 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:17:31 smithi050 ceph-mon[32371]: 5.1b scrub ok 2024-09-18T08:17:31.905 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:17:31 smithi050 ceph-mon[32371]: 3.13 scrub starts 2024-09-18T08:17:31.906 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:17:31 smithi050 ceph-mon[32371]: 3.13 scrub ok 2024-09-18T08:17:31.920 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6 2024-09-18T08:17:31.946 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:17:31.947 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:17:31.947 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6' 2024-09-18T08:17:32.048 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2 2024-09-18T08:17:32.106 INFO:tasks.cephadm.ceph_manager.ceph:waiting for recovery to complete 2024-09-18T08:17:32.106 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid 779349ca-7593-11ef-bceb-c7b262605968 -- ceph pg dump --format=json 2024-09-18T08:17:32.110 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:17:32.110 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:17:32.110 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2' 2024-09-18T08:17:32.211 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1 2024-09-18T08:17:32.259 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:17:32.260 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:17:32.260 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1' 2024-09-18T08:17:32.361 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5 2024-09-18T08:17:32.386 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:17:32.387 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:17:32.387 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5' 2024-09-18T08:17:32.488 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6 2024-09-18T08:17:32.515 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:17:32.516 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:17:32.516 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6' 2024-09-18T08:17:32.617 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0 2024-09-18T08:17:32.676 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:17:32.677 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:17:32.677 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0' 2024-09-18T08:17:32.734 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:17:32 smithi007 ceph-mon[29459]: 4.e scrub starts 2024-09-18T08:17:32.734 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:17:32 smithi007 ceph-mon[29459]: 4.e scrub ok 2024-09-18T08:17:32.734 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:17:32 smithi007 ceph-mon[29459]: 5.1 scrub starts 2024-09-18T08:17:32.734 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:17:32 smithi007 ceph-mon[29459]: 5.1 scrub ok 2024-09-18T08:17:32.734 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:17:32 smithi007 ceph-mon[29459]: pgmap v737: 158 pgs: 2 active+remapped+backfill_toofull, 156 active+clean; 2.8 GiB data, 11 GiB used, 525 GiB / 536 GiB avail; 853 B/s rd, 0 B/s wr, 640 op/s; 5744/137718 objects misplaced (4.171%); 1.7 MiB/s, 27 objects/s recovering 2024-09-18T08:17:32.734 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:17:32 smithi007 ceph-mon[29459]: 6.12 scrub starts 2024-09-18T08:17:32.734 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:17:32 smithi007 ceph-mon[29459]: 6.12 scrub ok 2024-09-18T08:17:32.735 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:17:32 smithi007 ceph-mon[39249]: 4.e scrub starts 2024-09-18T08:17:32.735 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:17:32 smithi007 ceph-mon[39249]: 4.e scrub ok 2024-09-18T08:17:32.735 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:17:32 smithi007 ceph-mon[39249]: 5.1 scrub starts 2024-09-18T08:17:32.735 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:17:32 smithi007 ceph-mon[39249]: 5.1 scrub ok 2024-09-18T08:17:32.735 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:17:32 smithi007 ceph-mon[39249]: pgmap v737: 158 pgs: 2 active+remapped+backfill_toofull, 156 active+clean; 2.8 GiB data, 11 GiB used, 525 GiB / 536 GiB avail; 853 B/s rd, 0 B/s wr, 640 op/s; 5744/137718 objects misplaced (4.171%); 1.7 MiB/s, 27 objects/s recovering 2024-09-18T08:17:32.735 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:17:32 smithi007 ceph-mon[39249]: 6.12 scrub starts 2024-09-18T08:17:32.735 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:17:32 smithi007 ceph-mon[39249]: 6.12 scrub ok 2024-09-18T08:17:32.747 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/779349ca-7593-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T08:17:32.756 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/779349ca-7593-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T08:17:32.775 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid 779349ca-7593-11ef-bceb-c7b262605968 -- ceph osd unset nodeep-scrub 2024-09-18T08:17:32.778 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4 2024-09-18T08:17:32.803 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:17:32.804 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:17:32.804 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4' 2024-09-18T08:17:32.905 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:17:32 smithi050 ceph-mon[32371]: 4.e scrub starts 2024-09-18T08:17:32.905 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:17:32 smithi050 ceph-mon[32371]: 4.e scrub ok 2024-09-18T08:17:32.905 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:17:32 smithi050 ceph-mon[32371]: 5.1 scrub starts 2024-09-18T08:17:32.905 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:17:32 smithi050 ceph-mon[32371]: 5.1 scrub ok 2024-09-18T08:17:32.906 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:17:32 smithi050 ceph-mon[32371]: pgmap v737: 158 pgs: 2 active+remapped+backfill_toofull, 156 active+clean; 2.8 GiB data, 11 GiB used, 525 GiB / 536 GiB avail; 853 B/s rd, 0 B/s wr, 640 op/s; 5744/137718 objects misplaced (4.171%); 1.7 MiB/s, 27 objects/s recovering 2024-09-18T08:17:32.906 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:17:32 smithi050 ceph-mon[32371]: 6.12 scrub starts 2024-09-18T08:17:32.906 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:17:32 smithi050 ceph-mon[32371]: 6.12 scrub ok 2024-09-18T08:17:32.906 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1 2024-09-18T08:17:32.951 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:17:32.951 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:17:32.951 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1' 2024-09-18T08:17:33.052 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2 2024-09-18T08:17:33.107 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:17:33.107 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:17:33.107 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2' 2024-09-18T08:17:33.208 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3 2024-09-18T08:17:33.291 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:17:33.292 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:17:33.292 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3' 2024-09-18T08:17:33.393 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3 2024-09-18T08:17:33.437 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:17:33.438 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:17:33.438 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3' 2024-09-18T08:17:33.539 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5 2024-09-18T08:17:33.564 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:17:33.564 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:17:33.565 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5' 2024-09-18T08:17:33.646 INFO:teuthology.orchestra.run.smithi007.stdout:ERROR: (22) Invalid argument 2024-09-18T08:17:33.646 INFO:teuthology.orchestra.run.smithi007.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-18T08:17:33.666 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4 2024-09-18T08:17:33.691 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:17:33.692 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:17:33.693 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4' 2024-09-18T08:17:33.730 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:17:33 smithi007 ceph-mon[29459]: 4.16 scrub starts 2024-09-18T08:17:33.730 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:17:33 smithi007 ceph-mon[29459]: 4.16 scrub ok 2024-09-18T08:17:33.730 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:17:33 smithi007 ceph-mon[29459]: 4.15 scrub starts 2024-09-18T08:17:33.730 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:17:33 smithi007 ceph-mon[29459]: 4.15 scrub ok 2024-09-18T08:17:33.730 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:17:33 smithi007 ceph-mon[29459]: purged_snaps scrub starts 2024-09-18T08:17:33.730 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:17:33 smithi007 ceph-mon[29459]: purged_snaps scrub ok 2024-09-18T08:17:33.731 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:17:33 smithi007 ceph-mon[39249]: 4.16 scrub starts 2024-09-18T08:17:33.731 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:17:33 smithi007 ceph-mon[39249]: 4.16 scrub ok 2024-09-18T08:17:33.731 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:17:33 smithi007 ceph-mon[39249]: 4.15 scrub starts 2024-09-18T08:17:33.731 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:17:33 smithi007 ceph-mon[39249]: 4.15 scrub ok 2024-09-18T08:17:33.731 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:17:33 smithi007 ceph-mon[39249]: purged_snaps scrub starts 2024-09-18T08:17:33.731 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:17:33 smithi007 ceph-mon[39249]: purged_snaps scrub ok 2024-09-18T08:17:33.793 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1 2024-09-18T08:17:33.831 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:17:33.832 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:17:33.833 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1' 2024-09-18T08:17:33.905 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:17:33 smithi050 ceph-mon[32371]: 4.16 scrub starts 2024-09-18T08:17:33.905 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:17:33 smithi050 ceph-mon[32371]: 4.16 scrub ok 2024-09-18T08:17:33.905 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:17:33 smithi050 ceph-mon[32371]: 4.15 scrub starts 2024-09-18T08:17:33.905 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:17:33 smithi050 ceph-mon[32371]: 4.15 scrub ok 2024-09-18T08:17:33.905 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:17:33 smithi050 ceph-mon[32371]: purged_snaps scrub starts 2024-09-18T08:17:33.905 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:17:33 smithi050 ceph-mon[32371]: purged_snaps scrub ok 2024-09-18T08:17:33.930 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-09-18T08:17:33.933 INFO:teuthology.orchestra.run.smithi007.stderr:dumped all 2024-09-18T08:17:33.933 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6 2024-09-18T08:17:33.959 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:17:33.959 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:17:33.960 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6' 2024-09-18T08:17:34.061 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6 2024-09-18T08:17:34.087 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:17:34.088 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:17:34.088 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6' 2024-09-18T08:17:34.189 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6 2024-09-18T08:17:34.214 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:17:34.215 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:17:34.215 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6' 2024-09-18T08:17:34.316 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0 2024-09-18T08:17:34.367 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:17:34.368 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:17:34.368 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0' 2024-09-18T08:17:34.469 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2 2024-09-18T08:17:34.509 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:17:34.510 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:17:34.510 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2' 2024-09-18T08:17:34.611 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6 2024-09-18T08:17:34.636 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:17:34.637 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:17:34.637 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6' 2024-09-18T08:17:34.686 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid 779349ca-7593-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-18T08:17:34.730 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:17:34 smithi007 ceph-mon[29459]: purged_snaps scrub starts 2024-09-18T08:17:34.730 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:17:34 smithi007 ceph-mon[29459]: purged_snaps scrub ok 2024-09-18T08:17:34.730 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:17:34 smithi007 ceph-mon[29459]: pgmap v738: 158 pgs: 2 active+remapped+backfill_toofull, 156 active+clean; 2.7 GiB data, 11 GiB used, 525 GiB / 536 GiB avail; 1.4 KiB/s rd, 0 B/s wr, 701 op/s; 5347/134490 objects misplaced (3.976%); 1.5 MiB/s, 23 objects/s recovering 2024-09-18T08:17:34.730 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:17:34 smithi007 ceph-mon[29459]: osdmap e381: 8 total, 8 up, 6 in 2024-09-18T08:17:34.730 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:17:34 smithi007 ceph-mon[29459]: from='mgr.24470 172.21.15.7:0/2594390261' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-18T08:17:34.730 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:17:34 smithi007 ceph-mon[29459]: 6.f scrub starts 2024-09-18T08:17:34.730 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:17:34 smithi007 ceph-mon[29459]: 6.f scrub ok 2024-09-18T08:17:34.730 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:17:34 smithi007 ceph-mon[39249]: purged_snaps scrub starts 2024-09-18T08:17:34.730 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:17:34 smithi007 ceph-mon[39249]: purged_snaps scrub ok 2024-09-18T08:17:34.730 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:17:34 smithi007 ceph-mon[39249]: pgmap v738: 158 pgs: 2 active+remapped+backfill_toofull, 156 active+clean; 2.7 GiB data, 11 GiB used, 525 GiB / 536 GiB avail; 1.4 KiB/s rd, 0 B/s wr, 701 op/s; 5347/134490 objects misplaced (3.976%); 1.5 MiB/s, 23 objects/s recovering 2024-09-18T08:17:34.730 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:17:34 smithi007 ceph-mon[39249]: osdmap e381: 8 total, 8 up, 6 in 2024-09-18T08:17:34.731 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:17:34 smithi007 ceph-mon[39249]: from='mgr.24470 172.21.15.7:0/2594390261' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-18T08:17:34.731 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:17:34 smithi007 ceph-mon[39249]: 6.f scrub starts 2024-09-18T08:17:34.731 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:17:34 smithi007 ceph-mon[39249]: 6.f scrub ok 2024-09-18T08:17:34.738 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7 2024-09-18T08:17:34.760 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:17:34.761 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:17:34.761 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7' 2024-09-18T08:17:34.862 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2 2024-09-18T08:17:34.906 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:17:34 smithi050 ceph-mon[32371]: purged_snaps scrub starts 2024-09-18T08:17:34.906 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:17:34 smithi050 ceph-mon[32371]: purged_snaps scrub ok 2024-09-18T08:17:34.906 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:17:34 smithi050 ceph-mon[32371]: pgmap v738: 158 pgs: 2 active+remapped+backfill_toofull, 156 active+clean; 2.7 GiB data, 11 GiB used, 525 GiB / 536 GiB avail; 1.4 KiB/s rd, 0 B/s wr, 701 op/s; 5347/134490 objects misplaced (3.976%); 1.5 MiB/s, 23 objects/s recovering 2024-09-18T08:17:34.907 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:17:34 smithi050 ceph-mon[32371]: osdmap e381: 8 total, 8 up, 6 in 2024-09-18T08:17:34.907 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:17:34 smithi050 ceph-mon[32371]: from='mgr.24470 172.21.15.7:0/2594390261' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-18T08:17:34.907 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:17:34 smithi050 ceph-mon[32371]: 6.f scrub starts 2024-09-18T08:17:34.907 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:17:34 smithi050 ceph-mon[32371]: 6.f scrub ok 2024-09-18T08:17:34.907 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:17:34.908 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:17:34.908 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2' 2024-09-18T08:17:35.009 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7 2024-09-18T08:17:35.036 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:17:35.037 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:17:35.037 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7' 2024-09-18T08:17:35.138 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5 2024-09-18T08:17:35.160 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:17:35.161 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:17:35.161 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5' 2024-09-18T08:17:35.262 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2 2024-09-18T08:17:35.300 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:17:35.301 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:17:35.301 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2' 2024-09-18T08:17:35.402 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7 2024-09-18T08:17:35.428 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:17:35.429 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:17:35.429 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7' 2024-09-18T08:17:35.530 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2 2024-09-18T08:17:35.571 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:17:35.572 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:17:35.573 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2' 2024-09-18T08:17:35.674 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6 2024-09-18T08:17:35.698 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:17:35.699 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:17:35.699 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6' 2024-09-18T08:17:35.730 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:17:35 smithi007 ceph-mon[29459]: from='client.25973 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-18T08:17:35.730 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:17:35 smithi007 ceph-mon[29459]: 2.1 scrub starts 2024-09-18T08:17:35.730 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:17:35 smithi007 ceph-mon[29459]: 2.1 scrub ok 2024-09-18T08:17:35.730 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:17:35 smithi007 ceph-mon[39249]: from='client.25973 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-18T08:17:35.730 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:17:35 smithi007 ceph-mon[39249]: 2.1 scrub starts 2024-09-18T08:17:35.731 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:17:35 smithi007 ceph-mon[39249]: 2.1 scrub ok 2024-09-18T08:17:35.800 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3 2024-09-18T08:17:35.839 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:17:35.839 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:17:35.840 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3' 2024-09-18T08:17:35.905 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:17:35 smithi050 ceph-mon[32371]: from='client.25973 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-18T08:17:35.905 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:17:35 smithi050 ceph-mon[32371]: 2.1 scrub starts 2024-09-18T08:17:35.905 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:17:35 smithi050 ceph-mon[32371]: 2.1 scrub ok 2024-09-18T08:17:35.941 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6 2024-09-18T08:17:35.964 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:17:35.964 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:17:35.964 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6' 2024-09-18T08:17:36.065 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1 2024-09-18T08:17:36.111 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:17:36.111 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:17:36.112 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1' 2024-09-18T08:17:36.213 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0 2024-09-18T08:17:36.249 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:17:36.250 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:17:36.250 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0' 2024-09-18T08:17:36.351 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1 2024-09-18T08:17:36.392 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:17:36.392 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:17:36.392 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1' 2024-09-18T08:17:36.493 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5 2024-09-18T08:17:36.518 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:17:36.519 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:17:36.519 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5' 2024-09-18T08:17:36.620 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7 2024-09-18T08:17:36.646 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:17:36.647 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:17:36.647 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7' 2024-09-18T08:17:36.730 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:17:36 smithi007 ceph-mon[29459]: 6.5 scrub starts 2024-09-18T08:17:36.730 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:17:36 smithi007 ceph-mon[29459]: 6.5 scrub ok 2024-09-18T08:17:36.730 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:17:36 smithi007 ceph-mon[29459]: pgmap v740: 158 pgs: 2 active+remapped+backfill_toofull, 156 active+clean; 2.4 GiB data, 10 GiB used, 526 GiB / 536 GiB avail; 730 B/s rd, 0 B/s wr, 1.25k op/s; 4985/120804 objects misplaced (4.127%); 1.5 MiB/s, 23 objects/s recovering 2024-09-18T08:17:36.730 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Sep 18 08:17:36 smithi007 ceph-779349ca-7593-11ef-bceb-c7b262605968-mgr-y[29688]: ::ffff:172.21.15.50 - - [18/Sep/2024:08:17:36] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-09-18T08:17:36.731 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:17:36 smithi007 ceph-mon[39249]: 6.5 scrub starts 2024-09-18T08:17:36.731 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:17:36 smithi007 ceph-mon[39249]: 6.5 scrub ok 2024-09-18T08:17:36.731 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:17:36 smithi007 ceph-mon[39249]: pgmap v740: 158 pgs: 2 active+remapped+backfill_toofull, 156 active+clean; 2.4 GiB data, 10 GiB used, 526 GiB / 536 GiB avail; 730 B/s rd, 0 B/s wr, 1.25k op/s; 4985/120804 objects misplaced (4.127%); 1.5 MiB/s, 23 objects/s recovering 2024-09-18T08:17:36.748 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0 2024-09-18T08:17:36.787 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:17:36.787 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:17:36.788 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0' 2024-09-18T08:17:36.889 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1 2024-09-18T08:17:36.905 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:17:36 smithi050 ceph-mon[32371]: 6.5 scrub starts 2024-09-18T08:17:36.905 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:17:36 smithi050 ceph-mon[32371]: 6.5 scrub ok 2024-09-18T08:17:36.905 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:17:36 smithi050 ceph-mon[32371]: pgmap v740: 158 pgs: 2 active+remapped+backfill_toofull, 156 active+clean; 2.4 GiB data, 10 GiB used, 526 GiB / 536 GiB avail; 730 B/s rd, 0 B/s wr, 1.25k op/s; 4985/120804 objects misplaced (4.127%); 1.5 MiB/s, 23 objects/s recovering 2024-09-18T08:17:36.914 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:17:36.915 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:17:36.915 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1' 2024-09-18T08:17:37.016 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1 2024-09-18T08:17:37.053 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:17:37.054 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:17:37.054 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1' 2024-09-18T08:17:37.155 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7 2024-09-18T08:17:37.193 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:17:37.194 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:17:37.194 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7' 2024-09-18T08:17:37.295 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6 2024-09-18T08:17:37.320 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:17:37.321 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:17:37.321 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6' 2024-09-18T08:17:37.422 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6 2024-09-18T08:17:37.452 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:17:37.452 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:17:37.452 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6' 2024-09-18T08:17:37.553 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7 2024-09-18T08:17:37.580 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:17:37.580 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:17:37.580 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7' 2024-09-18T08:17:37.681 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6 2024-09-18T08:17:37.707 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:17:37.708 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:17:37.708 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6' 2024-09-18T08:17:37.809 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1 2024-09-18T08:17:37.844 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:17:37.844 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:17:37.845 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1' 2024-09-18T08:17:37.905 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:17:37 smithi050 ceph-mon[32371]: 3.e scrub starts 2024-09-18T08:17:37.905 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:17:37 smithi050 ceph-mon[32371]: 3.e scrub ok 2024-09-18T08:17:37.905 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:17:37 smithi050 ceph-mon[32371]: 5.4 scrub starts 2024-09-18T08:17:37.905 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:17:37 smithi050 ceph-mon[32371]: 5.4 scrub ok 2024-09-18T08:17:37.946 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7 2024-09-18T08:17:37.972 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:17:37.973 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:17:37.973 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7' 2024-09-18T08:17:37.980 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:17:37 smithi007 ceph-mon[39249]: 3.e scrub starts 2024-09-18T08:17:37.980 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:17:37 smithi007 ceph-mon[39249]: 3.e scrub ok 2024-09-18T08:17:37.980 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:17:37 smithi007 ceph-mon[39249]: 5.4 scrub starts 2024-09-18T08:17:37.980 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:17:37 smithi007 ceph-mon[39249]: 5.4 scrub ok 2024-09-18T08:17:37.980 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:17:37 smithi007 ceph-mon[29459]: 3.e scrub starts 2024-09-18T08:17:37.980 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:17:37 smithi007 ceph-mon[29459]: 3.e scrub ok 2024-09-18T08:17:37.980 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:17:37 smithi007 ceph-mon[29459]: 5.4 scrub starts 2024-09-18T08:17:37.980 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:17:37 smithi007 ceph-mon[29459]: 5.4 scrub ok 2024-09-18T08:17:38.074 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2 2024-09-18T08:17:38.107 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:17:38.108 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:17:38.108 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2' 2024-09-18T08:17:38.209 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1 2024-09-18T08:17:38.251 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:17:38.252 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:17:38.252 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1' 2024-09-18T08:17:38.353 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6 2024-09-18T08:17:38.377 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:17:38.378 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:17:38.378 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6' 2024-09-18T08:17:38.479 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1 2024-09-18T08:17:38.524 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:17:38.525 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:17:38.525 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1' 2024-09-18T08:17:38.626 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1 2024-09-18T08:17:38.663 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:17:38.665 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:17:38.665 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1' 2024-09-18T08:17:38.766 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2 2024-09-18T08:17:38.808 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:17:38.809 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:17:38.809 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2' 2024-09-18T08:17:38.905 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:17:38 smithi050 ceph-mon[32371]: 5.1d scrub starts 2024-09-18T08:17:38.905 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:17:38 smithi050 ceph-mon[32371]: 5.1d scrub ok 2024-09-18T08:17:38.905 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:17:38 smithi050 ceph-mon[32371]: 6.11 scrub starts 2024-09-18T08:17:38.905 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:17:38 smithi050 ceph-mon[32371]: 6.11 scrub ok 2024-09-18T08:17:38.905 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:17:38 smithi050 ceph-mon[32371]: pgmap v741: 158 pgs: 2 active+remapped+backfill_toofull, 156 active+clean; 2.4 GiB data, 10 GiB used, 526 GiB / 536 GiB avail; 639 B/s rd, 0 B/s wr, 887 op/s; 4985/120804 objects misplaced (4.127%); 1.3 MiB/s, 20 objects/s recovering 2024-09-18T08:17:38.905 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:17:38 smithi050 ceph-mon[32371]: 5.e scrub starts 2024-09-18T08:17:38.906 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:17:38 smithi050 ceph-mon[32371]: 5.e scrub ok 2024-09-18T08:17:38.910 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6 2024-09-18T08:17:38.936 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:17:38.937 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:17:38.937 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6' 2024-09-18T08:17:38.980 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:17:38 smithi007 ceph-mon[29459]: 5.1d scrub starts 2024-09-18T08:17:38.980 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:17:38 smithi007 ceph-mon[29459]: 5.1d scrub ok 2024-09-18T08:17:38.980 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:17:38 smithi007 ceph-mon[29459]: 6.11 scrub starts 2024-09-18T08:17:38.980 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:17:38 smithi007 ceph-mon[29459]: 6.11 scrub ok 2024-09-18T08:17:38.980 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:17:38 smithi007 ceph-mon[29459]: pgmap v741: 158 pgs: 2 active+remapped+backfill_toofull, 156 active+clean; 2.4 GiB data, 10 GiB used, 526 GiB / 536 GiB avail; 639 B/s rd, 0 B/s wr, 887 op/s; 4985/120804 objects misplaced (4.127%); 1.3 MiB/s, 20 objects/s recovering 2024-09-18T08:17:38.980 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:17:38 smithi007 ceph-mon[29459]: 5.e scrub starts 2024-09-18T08:17:38.980 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:17:38 smithi007 ceph-mon[29459]: 5.e scrub ok 2024-09-18T08:17:38.980 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:17:38 smithi007 ceph-mon[39249]: 5.1d scrub starts 2024-09-18T08:17:38.981 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:17:38 smithi007 ceph-mon[39249]: 5.1d scrub ok 2024-09-18T08:17:38.981 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:17:38 smithi007 ceph-mon[39249]: 6.11 scrub starts 2024-09-18T08:17:38.981 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:17:38 smithi007 ceph-mon[39249]: 6.11 scrub ok 2024-09-18T08:17:38.981 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:17:38 smithi007 ceph-mon[39249]: pgmap v741: 158 pgs: 2 active+remapped+backfill_toofull, 156 active+clean; 2.4 GiB data, 10 GiB used, 526 GiB / 536 GiB avail; 639 B/s rd, 0 B/s wr, 887 op/s; 4985/120804 objects misplaced (4.127%); 1.3 MiB/s, 20 objects/s recovering 2024-09-18T08:17:38.981 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:17:38 smithi007 ceph-mon[39249]: 5.e scrub starts 2024-09-18T08:17:38.981 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:17:38 smithi007 ceph-mon[39249]: 5.e scrub ok 2024-09-18T08:17:39.038 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6 2024-09-18T08:17:39.064 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:17:39.064 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:17:39.064 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6' 2024-09-18T08:17:39.165 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1 2024-09-18T08:17:39.205 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:17:39.205 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:17:39.206 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1' 2024-09-18T08:17:39.307 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4 2024-09-18T08:17:39.334 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:17:39.334 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:17:39.335 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4' 2024-09-18T08:17:39.436 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7 2024-09-18T08:17:39.457 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:17:39.458 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:17:39.459 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7' 2024-09-18T08:17:39.559 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0 2024-09-18T08:17:39.602 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:17:39.603 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:17:39.603 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0' 2024-09-18T08:17:39.704 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0 2024-09-18T08:17:39.748 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:17:39.749 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:17:39.749 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0' 2024-09-18T08:17:39.850 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5 2024-09-18T08:17:39.877 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:17:39.878 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:17:39.879 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5' 2024-09-18T08:17:39.905 INFO:journalctl@ceph.iscsi.iscsi.a.smithi050.stdout:Sep 18 08:17:39 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-iscsi-iscsi-a[57782]: debug there is no tcmu-runner data available 2024-09-18T08:17:39.906 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:17:39 smithi050 ceph-mon[32371]: 3.19 scrub starts 2024-09-18T08:17:39.906 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:17:39 smithi050 ceph-mon[32371]: 3.19 scrub ok 2024-09-18T08:17:39.906 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:17:39 smithi050 ceph-mon[32371]: 3.1e scrub starts 2024-09-18T08:17:39.906 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:17:39 smithi050 ceph-mon[32371]: 3.1e scrub ok 2024-09-18T08:17:39.906 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:17:39 smithi050 ceph-mon[32371]: 5.f scrub starts 2024-09-18T08:17:39.906 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:17:39 smithi050 ceph-mon[32371]: 5.f scrub ok 2024-09-18T08:17:39.906 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:17:39 smithi050 ceph-mon[32371]: 5.d scrub starts 2024-09-18T08:17:39.906 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:17:39 smithi050 ceph-mon[32371]: 5.d scrub ok 2024-09-18T08:17:39.980 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6 2024-09-18T08:17:39.981 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:17:39 smithi007 ceph-mon[29459]: 3.19 scrub starts 2024-09-18T08:17:39.981 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:17:39 smithi007 ceph-mon[29459]: 3.19 scrub ok 2024-09-18T08:17:39.981 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:17:39 smithi007 ceph-mon[29459]: 3.1e scrub starts 2024-09-18T08:17:39.981 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:17:39 smithi007 ceph-mon[29459]: 3.1e scrub ok 2024-09-18T08:17:39.981 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:17:39 smithi007 ceph-mon[29459]: 5.f scrub starts 2024-09-18T08:17:39.981 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:17:39 smithi007 ceph-mon[29459]: 5.f scrub ok 2024-09-18T08:17:39.981 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:17:39 smithi007 ceph-mon[29459]: 5.d scrub starts 2024-09-18T08:17:39.981 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:17:39 smithi007 ceph-mon[29459]: 5.d scrub ok 2024-09-18T08:17:39.982 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:17:39 smithi007 ceph-mon[39249]: 3.19 scrub starts 2024-09-18T08:17:39.982 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:17:39 smithi007 ceph-mon[39249]: 3.19 scrub ok 2024-09-18T08:17:39.982 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:17:39 smithi007 ceph-mon[39249]: 3.1e scrub starts 2024-09-18T08:17:39.982 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:17:39 smithi007 ceph-mon[39249]: 3.1e scrub ok 2024-09-18T08:17:39.982 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:17:39 smithi007 ceph-mon[39249]: 5.f scrub starts 2024-09-18T08:17:39.982 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:17:39 smithi007 ceph-mon[39249]: 5.f scrub ok 2024-09-18T08:17:39.982 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:17:39 smithi007 ceph-mon[39249]: 5.d scrub starts 2024-09-18T08:17:39.982 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:17:39 smithi007 ceph-mon[39249]: 5.d scrub ok 2024-09-18T08:17:40.010 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:17:40.010 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:17:40.010 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6' 2024-09-18T08:17:40.111 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2 2024-09-18T08:17:40.145 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:17:40.146 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:17:40.146 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2' 2024-09-18T08:17:40.247 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2 2024-09-18T08:17:40.290 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:17:40.290 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:17:40.291 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2' 2024-09-18T08:17:40.392 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4 2024-09-18T08:17:40.419 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:17:40.420 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:17:40.420 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4' 2024-09-18T08:17:40.521 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6 2024-09-18T08:17:40.546 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:17:40.546 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:17:40.547 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6' 2024-09-18T08:17:40.610 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid 779349ca-7593-11ef-bceb-c7b262605968 -- ceph daemon osd.0 dump_ops_in_flight 2024-09-18T08:17:40.647 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5 2024-09-18T08:17:40.671 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:17:40.672 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:17:40.672 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5' 2024-09-18T08:17:40.773 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0 2024-09-18T08:17:40.794 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:17:40.795 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:17:40.796 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0' 2024-09-18T08:17:40.897 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6 2024-09-18T08:17:40.905 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:17:40 smithi050 ceph-mon[32371]: 5.14 scrub starts 2024-09-18T08:17:40.905 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:17:40 smithi050 ceph-mon[32371]: 5.14 scrub ok 2024-09-18T08:17:40.905 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:17:40 smithi050 ceph-mon[32371]: pgmap v742: 158 pgs: 2 active+remapped+backfill_toofull, 156 active+clean; 2.2 GiB data, 9.6 GiB used, 527 GiB / 536 GiB avail; 1 KiB/s rd, 0 B/s wr, 1.05k op/s; 4304/110520 objects misplaced (3.894%); 1.0 MiB/s, 16 objects/s recovering 2024-09-18T08:17:40.905 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:17:40 smithi050 ceph-mon[32371]: 6.16 scrub starts 2024-09-18T08:17:40.905 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:17:40 smithi050 ceph-mon[32371]: 6.16 scrub ok 2024-09-18T08:17:40.905 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:17:40 smithi050 ceph-mon[32371]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-18T08:17:40.921 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:17:40.922 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:17:40.922 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6' 2024-09-18T08:17:40.979 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:17:40 smithi007 ceph-mon[29459]: 5.14 scrub starts 2024-09-18T08:17:40.980 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:17:40 smithi007 ceph-mon[29459]: 5.14 scrub ok 2024-09-18T08:17:40.980 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:17:40 smithi007 ceph-mon[29459]: pgmap v742: 158 pgs: 2 active+remapped+backfill_toofull, 156 active+clean; 2.2 GiB data, 9.6 GiB used, 527 GiB / 536 GiB avail; 1 KiB/s rd, 0 B/s wr, 1.05k op/s; 4304/110520 objects misplaced (3.894%); 1.0 MiB/s, 16 objects/s recovering 2024-09-18T08:17:40.980 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:17:40 smithi007 ceph-mon[29459]: 6.16 scrub starts 2024-09-18T08:17:40.980 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:17:40 smithi007 ceph-mon[29459]: 6.16 scrub ok 2024-09-18T08:17:40.980 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:17:40 smithi007 ceph-mon[29459]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-18T08:17:40.981 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:17:40 smithi007 ceph-mon[39249]: 5.14 scrub starts 2024-09-18T08:17:40.981 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:17:40 smithi007 ceph-mon[39249]: 5.14 scrub ok 2024-09-18T08:17:40.981 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:17:40 smithi007 ceph-mon[39249]: pgmap v742: 158 pgs: 2 active+remapped+backfill_toofull, 156 active+clean; 2.2 GiB data, 9.6 GiB used, 527 GiB / 536 GiB avail; 1 KiB/s rd, 0 B/s wr, 1.05k op/s; 4304/110520 objects misplaced (3.894%); 1.0 MiB/s, 16 objects/s recovering 2024-09-18T08:17:40.981 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:17:40 smithi007 ceph-mon[39249]: 6.16 scrub starts 2024-09-18T08:17:40.981 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:17:40 smithi007 ceph-mon[39249]: 6.16 scrub ok 2024-09-18T08:17:40.981 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:17:40 smithi007 ceph-mon[39249]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-18T08:17:41.023 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6 2024-09-18T08:17:41.048 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:17:41.048 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:17:41.049 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6' 2024-09-18T08:17:41.149 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3 2024-09-18T08:17:41.187 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:17:41.187 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:17:41.188 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3' 2024-09-18T08:17:41.289 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4 2024-09-18T08:17:41.314 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:17:41.315 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:17:41.315 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4' 2024-09-18T08:17:41.416 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3 2024-09-18T08:17:41.439 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/779349ca-7593-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T08:17:41.456 INFO:teuthology.orchestra.run.smithi007.stdout:{"pg_ready":true,"pg_map":{"version":739,"stamp":"2024-09-18T08:17:33.467360+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":2925468211,"num_objects":44830,"num_object_clones":0,"num_object_copies":134490,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":5347,"num_objects_unfound":0,"num_objects_dirty":44830,"num_whiteouts":0,"num_read":1464,"num_read_kb":1216,"num_write":159527,"num_write_kb":6509934,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":60474,"num_bytes_recovered":3931978101,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":63462,"ondisk_log_size":63462,"up":474,"acting":474,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":474,"num_osds":6,"num_per_pool_osds":6,"num_per_pool_omap_osds":6,"kb":562470912,"kb_used":11551400,"kb_used_data":8200316,"kb_used_omap":15,"kb_used_meta":3350704,"kb_avail":550919512,"statfs":{"total":575970213888,"available":564141580288,"internally_reserved":0,"allocated":8397123584,"data_stored":8394511052,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":16049,"internal_metadata":3431121231},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,7,6,9,4,3],"upper_bound":64},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-322174976,"num_objects":-4916,"num_object_clones":0,"num_object_copies":-14748,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-1025,"num_objects_unfound":0,"num_objects_dirty":-4916,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":4916,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":164,"num_bytes_recovered":10747904,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.037466"},"pg_stats":[{"pgid":"3.1f","version":"0'0","reported_seq":714,"reported_epoch":380,"state":"active+clean","last_fresh":"2024-09-18T08:17:29.477830+0000","last_change":"2024-09-18T08:15:41.451370+0000","last_active":"2024-09-18T08:17:29.477830+0000","last_peered":"2024-09-18T08:17:29.477830+0000","last_clean":"2024-09-18T08:17:29.477830+0000","last_became_active":"2024-09-18T08:15:41.451012+0000","last_became_peered":"2024-09-18T08:15:41.451012+0000","last_unstale":"2024-09-18T08:17:29.477830+0000","last_undegraded":"2024-09-18T08:17:29.477830+0000","last_fullsized":"2024-09-18T08:17:29.477830+0000","mapping_epoch":336,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":337,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:14:29.060206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:46.860092+0000","last_clean_scrub_stamp":"2024-09-18T08:14:29.060206+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"271'15","reported_seq":445,"reported_epoch":379,"state":"active+clean","last_fresh":"2024-09-18T08:17:28.435950+0000","last_change":"2024-09-18T08:16:13.499531+0000","last_active":"2024-09-18T08:17:28.435950+0000","last_peered":"2024-09-18T08:17:28.435950+0000","last_clean":"2024-09-18T08:17:28.435950+0000","last_became_active":"2024-09-18T08:16:13.456332+0000","last_became_peered":"2024-09-18T08:16:13.456332+0000","last_unstale":"2024-09-18T08:17:28.435950+0000","last_undegraded":"2024-09-18T08:17:28.435950+0000","last_fullsized":"2024-09-18T08:17:28.435950+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"271'15","last_scrub_stamp":"2024-09-18T08:15:04.676980+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_clean_scrub_stamp":"2024-09-18T08:15:04.676980+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":25,"num_read_kb":19,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,3],"acting":[2,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":531,"reported_epoch":380,"state":"active+clean","last_fresh":"2024-09-18T08:17:29.461377+0000","last_change":"2024-09-18T08:15:41.440121+0000","last_active":"2024-09-18T08:17:29.461377+0000","last_peered":"2024-09-18T08:17:29.461377+0000","last_clean":"2024-09-18T08:17:29.461377+0000","last_became_active":"2024-09-18T08:15:41.438512+0000","last_became_peered":"2024-09-18T08:15:41.438512+0000","last_unstale":"2024-09-18T08:17:29.461377+0000","last_undegraded":"2024-09-18T08:17:29.461377+0000","last_fullsized":"2024-09-18T08:17:29.461377+0000","mapping_epoch":336,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":337,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:15:12.171225+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:50.886830+0000","last_clean_scrub_stamp":"2024-09-18T08:15:12.171225+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,2],"acting":[1,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":420,"reported_epoch":379,"state":"active+clean","last_fresh":"2024-09-18T08:17:28.426574+0000","last_change":"2024-09-18T08:16:13.489265+0000","last_active":"2024-09-18T08:17:28.426574+0000","last_peered":"2024-09-18T08:17:28.426574+0000","last_clean":"2024-09-18T08:17:28.426574+0000","last_became_active":"2024-09-18T08:16:13.488969+0000","last_became_peered":"2024-09-18T08:16:13.488969+0000","last_unstale":"2024-09-18T08:17:28.426574+0000","last_undegraded":"2024-09-18T08:17:28.426574+0000","last_fullsized":"2024-09-18T08:17:28.426574+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:14:41.746083+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_clean_scrub_stamp":"2024-09-18T08:14:41.746083+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":705,"reported_epoch":379,"state":"active+clean","last_fresh":"2024-09-18T08:17:28.459147+0000","last_change":"2024-09-18T08:15:51.826092+0000","last_active":"2024-09-18T08:17:28.459147+0000","last_peered":"2024-09-18T08:17:28.459147+0000","last_clean":"2024-09-18T08:17:28.459147+0000","last_became_active":"2024-09-18T08:14:40.357298+0000","last_became_peered":"2024-09-18T08:14:40.357298+0000","last_unstale":"2024-09-18T08:17:28.459147+0000","last_undegraded":"2024-09-18T08:17:28.459147+0000","last_fullsized":"2024-09-18T08:17:28.459147+0000","mapping_epoch":308,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":309,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:15:51.826050+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_clean_scrub_stamp":"2024-09-18T08:15:51.826050+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,1],"acting":[3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":707,"reported_epoch":379,"state":"active+clean","last_fresh":"2024-09-18T08:17:28.461752+0000","last_change":"2024-09-18T08:13:37.627262+0000","last_active":"2024-09-18T08:17:28.461752+0000","last_peered":"2024-09-18T08:17:28.461752+0000","last_clean":"2024-09-18T08:17:28.461752+0000","last_became_active":"2024-09-18T08:06:24.126194+0000","last_became_peered":"2024-09-18T08:06:24.126194+0000","last_unstale":"2024-09-18T08:17:28.461752+0000","last_undegraded":"2024-09-18T08:17:28.461752+0000","last_fullsized":"2024-09-18T08:17:28.461752+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:13:37.627195+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:46.860092+0000","last_clean_scrub_stamp":"2024-09-18T08:13:37.627195+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"271'27","reported_seq":707,"reported_epoch":379,"state":"active+clean","last_fresh":"2024-09-18T08:17:28.457067+0000","last_change":"2024-09-18T08:14:26.565324+0000","last_active":"2024-09-18T08:17:28.457067+0000","last_peered":"2024-09-18T08:17:28.457067+0000","last_clean":"2024-09-18T08:17:28.457067+0000","last_became_active":"2024-09-18T08:02:49.886106+0000","last_became_peered":"2024-09-18T08:02:49.886106+0000","last_unstale":"2024-09-18T08:17:28.457067+0000","last_undegraded":"2024-09-18T08:17:28.457067+0000","last_fullsized":"2024-09-18T08:17:28.457067+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"271'27","last_scrub_stamp":"2024-09-18T08:14:26.565071+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_clean_scrub_stamp":"2024-09-18T08:14:26.565071+0000","objects_scrubbed":7,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":44,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":342,"reported_epoch":380,"state":"active+clean","last_fresh":"2024-09-18T08:17:29.456564+0000","last_change":"2024-09-18T08:16:13.483763+0000","last_active":"2024-09-18T08:17:29.456564+0000","last_peered":"2024-09-18T08:17:29.456564+0000","last_clean":"2024-09-18T08:17:29.456564+0000","last_became_active":"2024-09-18T08:16:13.474963+0000","last_became_peered":"2024-09-18T08:16:13.474963+0000","last_unstale":"2024-09-18T08:17:29.456564+0000","last_undegraded":"2024-09-18T08:17:29.456564+0000","last_fullsized":"2024-09-18T08:17:29.456564+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:15:44.585123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:50.886830+0000","last_clean_scrub_stamp":"2024-09-18T08:15:44.585123+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":103,"reported_epoch":379,"state":"active+clean","last_fresh":"2024-09-18T08:17:28.427685+0000","last_change":"2024-09-18T08:16:13.459018+0000","last_active":"2024-09-18T08:17:28.427685+0000","last_peered":"2024-09-18T08:17:28.427685+0000","last_clean":"2024-09-18T08:17:28.427685+0000","last_became_active":"2024-09-18T08:16:13.458654+0000","last_became_peered":"2024-09-18T08:16:13.458654+0000","last_unstale":"2024-09-18T08:17:28.427685+0000","last_undegraded":"2024-09-18T08:17:28.427685+0000","last_fullsized":"2024-09-18T08:17:28.427685+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:10:51.225974+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:46.860092+0000","last_clean_scrub_stamp":"2024-09-18T08:10:51.225974+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:18:13.341070+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,1],"acting":[5,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"271'15","reported_seq":307,"reported_epoch":379,"state":"active+clean","last_fresh":"2024-09-18T08:17:28.450479+0000","last_change":"2024-09-18T08:16:13.499713+0000","last_active":"2024-09-18T08:17:28.450479+0000","last_peered":"2024-09-18T08:17:28.450479+0000","last_clean":"2024-09-18T08:17:28.450479+0000","last_became_active":"2024-09-18T08:16:13.463397+0000","last_became_peered":"2024-09-18T08:16:13.463397+0000","last_unstale":"2024-09-18T08:17:28.450479+0000","last_undegraded":"2024-09-18T08:17:28.450479+0000","last_fullsized":"2024-09-18T08:17:28.450479+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'9","last_scrub_stamp":"2024-09-18T08:12:00.466202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_clean_scrub_stamp":"2024-09-18T08:12:00.466202+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:18:24.407823+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":24,"num_read_kb":18,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":103,"reported_epoch":379,"state":"active+clean","last_fresh":"2024-09-18T08:17:28.427211+0000","last_change":"2024-09-18T08:16:13.461165+0000","last_active":"2024-09-18T08:17:28.427211+0000","last_peered":"2024-09-18T08:17:28.427211+0000","last_clean":"2024-09-18T08:17:28.427211+0000","last_became_active":"2024-09-18T08:16:13.460761+0000","last_became_peered":"2024-09-18T08:16:13.460761+0000","last_unstale":"2024-09-18T08:17:28.427211+0000","last_undegraded":"2024-09-18T08:17:28.427211+0000","last_fullsized":"2024-09-18T08:17:28.427211+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:12:57.218286+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:50.886830+0000","last_clean_scrub_stamp":"2024-09-18T08:12:57.218286+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,6],"acting":[5,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":682,"reported_epoch":380,"state":"active+clean","last_fresh":"2024-09-18T08:17:29.542441+0000","last_change":"2024-09-18T08:17:29.542358+0000","last_active":"2024-09-18T08:17:29.542441+0000","last_peered":"2024-09-18T08:17:29.542441+0000","last_clean":"2024-09-18T08:17:29.542441+0000","last_became_active":"2024-09-18T08:14:40.354651+0000","last_became_peered":"2024-09-18T08:14:40.354651+0000","last_unstale":"2024-09-18T08:17:29.542441+0000","last_undegraded":"2024-09-18T08:17:29.542441+0000","last_fullsized":"2024-09-18T08:17:29.542441+0000","mapping_epoch":308,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":309,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:17:29.542288+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_clean_scrub_stamp":"2024-09-18T08:17:29.542288+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:18:42.683374+0000","scrub_duration":18,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":103,"reported_epoch":379,"state":"active+clean","last_fresh":"2024-09-18T08:17:28.427437+0000","last_change":"2024-09-18T08:16:13.471332+0000","last_active":"2024-09-18T08:17:28.427437+0000","last_peered":"2024-09-18T08:17:28.427437+0000","last_clean":"2024-09-18T08:17:28.427437+0000","last_became_active":"2024-09-18T08:16:13.471003+0000","last_became_peered":"2024-09-18T08:16:13.471003+0000","last_unstale":"2024-09-18T08:17:28.427437+0000","last_undegraded":"2024-09-18T08:17:28.427437+0000","last_fullsized":"2024-09-18T08:17:28.427437+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:15:01.636504+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:46.860092+0000","last_clean_scrub_stamp":"2024-09-18T08:15:01.636504+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1b","version":"271'7","reported_seq":289,"reported_epoch":379,"state":"active+clean","last_fresh":"2024-09-18T08:17:28.460313+0000","last_change":"2024-09-18T08:16:13.514798+0000","last_active":"2024-09-18T08:17:28.460313+0000","last_peered":"2024-09-18T08:17:28.460313+0000","last_clean":"2024-09-18T08:17:28.460313+0000","last_became_active":"2024-09-18T08:16:13.473146+0000","last_became_peered":"2024-09-18T08:16:13.473146+0000","last_unstale":"2024-09-18T08:17:28.460313+0000","last_undegraded":"2024-09-18T08:17:28.460313+0000","last_fullsized":"2024-09-18T08:17:28.460313+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"271'7","last_scrub_stamp":"2024-09-18T08:15:13.551351+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_clean_scrub_stamp":"2024-09-18T08:15:13.551351+0000","objects_scrubbed":2,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":12,"num_read_kb":8,"num_write":8,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":546,"reported_epoch":380,"state":"active+clean","last_fresh":"2024-09-18T08:17:29.457586+0000","last_change":"2024-09-18T08:16:13.487519+0000","last_active":"2024-09-18T08:17:29.457586+0000","last_peered":"2024-09-18T08:17:29.457586+0000","last_clean":"2024-09-18T08:17:29.457586+0000","last_became_active":"2024-09-18T08:16:13.486136+0000","last_became_peered":"2024-09-18T08:16:13.486136+0000","last_unstale":"2024-09-18T08:17:29.457586+0000","last_undegraded":"2024-09-18T08:17:29.457586+0000","last_fullsized":"2024-09-18T08:17:29.457586+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:15:40.589048+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:50.886830+0000","last_clean_scrub_stamp":"2024-09-18T08:15:40.589048+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,6],"acting":[1,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":96,"reported_epoch":379,"state":"active+clean","last_fresh":"2024-09-18T08:17:28.426724+0000","last_change":"2024-09-18T08:15:41.444316+0000","last_active":"2024-09-18T08:17:28.426724+0000","last_peered":"2024-09-18T08:17:28.426724+0000","last_clean":"2024-09-18T08:17:28.426724+0000","last_became_active":"2024-09-18T08:15:41.441932+0000","last_became_peered":"2024-09-18T08:15:41.441932+0000","last_unstale":"2024-09-18T08:17:28.426724+0000","last_undegraded":"2024-09-18T08:17:28.426724+0000","last_fullsized":"2024-09-18T08:17:28.426724+0000","mapping_epoch":336,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":337,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:11:01.108648+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_clean_scrub_stamp":"2024-09-18T08:11:01.108648+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:18:18.451529+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":410,"reported_epoch":379,"state":"active+clean","last_fresh":"2024-09-18T08:17:28.429535+0000","last_change":"2024-09-18T08:16:13.453737+0000","last_active":"2024-09-18T08:17:28.429535+0000","last_peered":"2024-09-18T08:17:28.429535+0000","last_clean":"2024-09-18T08:17:28.429535+0000","last_became_active":"2024-09-18T08:16:13.453129+0000","last_became_peered":"2024-09-18T08:16:13.453129+0000","last_unstale":"2024-09-18T08:17:28.429535+0000","last_undegraded":"2024-09-18T08:17:28.429535+0000","last_fullsized":"2024-09-18T08:17:28.429535+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:15:42.600840+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_clean_scrub_stamp":"2024-09-18T08:15:42.600840+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,3],"acting":[5,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":709,"reported_epoch":380,"state":"active+clean","last_fresh":"2024-09-18T08:17:29.472287+0000","last_change":"2024-09-18T08:16:13.477657+0000","last_active":"2024-09-18T08:17:29.472287+0000","last_peered":"2024-09-18T08:17:29.472287+0000","last_clean":"2024-09-18T08:17:29.472287+0000","last_became_active":"2024-09-18T08:16:13.477401+0000","last_became_peered":"2024-09-18T08:16:13.477401+0000","last_unstale":"2024-09-18T08:17:29.472287+0000","last_undegraded":"2024-09-18T08:17:29.472287+0000","last_fullsized":"2024-09-18T08:17:29.472287+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:12:40.711507+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:46.860092+0000","last_clean_scrub_stamp":"2024-09-18T08:12:40.711507+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,3],"acting":[0,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1c","version":"271'27","reported_seq":443,"reported_epoch":379,"state":"active+clean","last_fresh":"2024-09-18T08:17:28.435323+0000","last_change":"2024-09-18T08:14:40.711299+0000","last_active":"2024-09-18T08:17:28.435323+0000","last_peered":"2024-09-18T08:17:28.435323+0000","last_clean":"2024-09-18T08:17:28.435323+0000","last_became_active":"2024-09-18T08:14:39.480913+0000","last_became_peered":"2024-09-18T08:14:39.480913+0000","last_unstale":"2024-09-18T08:17:28.435323+0000","last_undegraded":"2024-09-18T08:17:28.435323+0000","last_fullsized":"2024-09-18T08:17:28.435323+0000","mapping_epoch":307,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":308,"parent":"0.0","parent_split_bits":0,"last_scrub":"271'27","last_scrub_stamp":"2024-09-18T08:14:40.711228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_clean_scrub_stamp":"2024-09-18T08:14:40.711228+0000","objects_scrubbed":7,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":44,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":559,"reported_epoch":380,"state":"active+clean","last_fresh":"2024-09-18T08:17:29.464091+0000","last_change":"2024-09-18T08:16:13.462617+0000","last_active":"2024-09-18T08:17:29.464091+0000","last_peered":"2024-09-18T08:17:29.464091+0000","last_clean":"2024-09-18T08:17:29.464091+0000","last_became_active":"2024-09-18T08:16:13.462303+0000","last_became_peered":"2024-09-18T08:16:13.462303+0000","last_unstale":"2024-09-18T08:17:29.464091+0000","last_undegraded":"2024-09-18T08:17:29.464091+0000","last_fullsized":"2024-09-18T08:17:29.464091+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:10:59.645112+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:50.886830+0000","last_clean_scrub_stamp":"2024-09-18T08:10:59.645112+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,5],"acting":[1,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":693,"reported_epoch":379,"state":"active+clean","last_fresh":"2024-09-18T08:17:28.462239+0000","last_change":"2024-09-18T08:15:41.469278+0000","last_active":"2024-09-18T08:17:28.462239+0000","last_peered":"2024-09-18T08:17:28.462239+0000","last_clean":"2024-09-18T08:17:28.462239+0000","last_became_active":"2024-09-18T08:15:41.466371+0000","last_became_peered":"2024-09-18T08:15:41.466371+0000","last_unstale":"2024-09-18T08:17:28.462239+0000","last_undegraded":"2024-09-18T08:17:28.462239+0000","last_fullsized":"2024-09-18T08:17:28.462239+0000","mapping_epoch":336,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":337,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:15:05.719269+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_clean_scrub_stamp":"2024-09-18T08:15:05.719269+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":342,"reported_epoch":380,"state":"active+clean","last_fresh":"2024-09-18T08:17:29.459924+0000","last_change":"2024-09-18T08:16:13.487949+0000","last_active":"2024-09-18T08:17:29.459924+0000","last_peered":"2024-09-18T08:17:29.459924+0000","last_clean":"2024-09-18T08:17:29.459924+0000","last_became_active":"2024-09-18T08:16:13.487001+0000","last_became_peered":"2024-09-18T08:16:13.487001+0000","last_unstale":"2024-09-18T08:17:29.459924+0000","last_undegraded":"2024-09-18T08:17:29.459924+0000","last_fullsized":"2024-09-18T08:17:29.459924+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:14:40.774086+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:46.860092+0000","last_clean_scrub_stamp":"2024-09-18T08:14:40.774086+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1d","version":"271'18","reported_seq":796,"reported_epoch":379,"state":"active+clean","last_fresh":"2024-09-18T08:17:28.466020+0000","last_change":"2024-09-18T08:17:04.627991+0000","last_active":"2024-09-18T08:17:28.466020+0000","last_peered":"2024-09-18T08:17:28.466020+0000","last_clean":"2024-09-18T08:17:28.466020+0000","last_became_active":"2024-09-18T08:16:13.473248+0000","last_became_peered":"2024-09-18T08:16:13.473248+0000","last_unstale":"2024-09-18T08:17:28.466020+0000","last_undegraded":"2024-09-18T08:17:28.466020+0000","last_fullsized":"2024-09-18T08:17:28.466020+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"271'18","last_scrub_stamp":"2024-09-18T08:17:04.627913+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_clean_scrub_stamp":"2024-09-18T08:17:04.627913+0000","objects_scrubbed":5,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:18:07.331175+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":34,"num_read_kb":25,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":660,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":101,"reported_epoch":379,"state":"active+clean","last_fresh":"2024-09-18T08:17:28.469649+0000","last_change":"2024-09-18T08:16:13.485702+0000","last_active":"2024-09-18T08:17:28.469649+0000","last_peered":"2024-09-18T08:17:28.469649+0000","last_clean":"2024-09-18T08:17:28.469649+0000","last_became_active":"2024-09-18T08:16:13.485316+0000","last_became_peered":"2024-09-18T08:16:13.485316+0000","last_unstale":"2024-09-18T08:17:28.469649+0000","last_undegraded":"2024-09-18T08:17:28.469649+0000","last_fullsized":"2024-09-18T08:17:28.469649+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:10:46.303848+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:50.886830+0000","last_clean_scrub_stamp":"2024-09-18T08:10:46.303848+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:18:16.412467+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,1],"acting":[3,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.1c","version":"69'1","reported_seq":100,"reported_epoch":379,"state":"active+clean","last_fresh":"2024-09-18T08:17:28.427484+0000","last_change":"2024-09-18T08:15:41.489871+0000","last_active":"2024-09-18T08:17:28.427484+0000","last_peered":"2024-09-18T08:17:28.427484+0000","last_clean":"2024-09-18T08:17:28.427484+0000","last_became_active":"2024-09-18T08:15:41.466665+0000","last_became_peered":"2024-09-18T08:15:41.466665+0000","last_unstale":"2024-09-18T08:17:28.427484+0000","last_undegraded":"2024-09-18T08:17:28.427484+0000","last_fullsized":"2024-09-18T08:17:28.427484+0000","mapping_epoch":336,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":337,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'1","last_scrub_stamp":"2024-09-18T08:15:06.708980+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_clean_scrub_stamp":"2024-09-18T08:15:06.708980+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":403,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":806,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,3],"acting":[5,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"62'1","reported_seq":572,"reported_epoch":380,"state":"active+clean","last_fresh":"2024-09-18T08:17:29.461241+0000","last_change":"2024-09-18T08:16:13.516657+0000","last_active":"2024-09-18T08:17:29.461241+0000","last_peered":"2024-09-18T08:17:29.461241+0000","last_clean":"2024-09-18T08:17:29.461241+0000","last_became_active":"2024-09-18T08:16:13.483397+0000","last_became_peered":"2024-09-18T08:16:13.483397+0000","last_unstale":"2024-09-18T08:17:29.461241+0000","last_undegraded":"2024-09-18T08:17:29.461241+0000","last_fullsized":"2024-09-18T08:17:29.461241+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"62'1","last_scrub_stamp":"2024-09-18T08:11:00.634190+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:46.860092+0000","last_clean_scrub_stamp":"2024-09-18T08:11:00.634190+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":46,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,6],"acting":[1,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1e","version":"271'20","reported_seq":728,"reported_epoch":380,"state":"active+clean","last_fresh":"2024-09-18T08:17:29.477257+0000","last_change":"2024-09-18T08:15:40.251794+0000","last_active":"2024-09-18T08:17:29.477257+0000","last_peered":"2024-09-18T08:17:29.477257+0000","last_clean":"2024-09-18T08:17:29.477257+0000","last_became_active":"2024-09-18T08:14:40.359879+0000","last_became_peered":"2024-09-18T08:14:40.359879+0000","last_unstale":"2024-09-18T08:17:29.477257+0000","last_undegraded":"2024-09-18T08:17:29.477257+0000","last_fullsized":"2024-09-18T08:17:29.477257+0000","mapping_epoch":308,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":309,"parent":"0.0","parent_split_bits":0,"last_scrub":"271'20","last_scrub_stamp":"2024-09-18T08:15:40.251706+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_clean_scrub_stamp":"2024-09-18T08:15:40.251706+0000","objects_scrubbed":5,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":25,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1],"acting":[0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1f","version":"349'15","reported_seq":408,"reported_epoch":379,"state":"active+clean","last_fresh":"2024-09-18T08:17:28.436811+0000","last_change":"2024-09-18T08:16:13.502776+0000","last_active":"2024-09-18T08:17:28.436811+0000","last_peered":"2024-09-18T08:17:28.436811+0000","last_clean":"2024-09-18T08:17:28.436811+0000","last_became_active":"2024-09-18T08:16:13.467898+0000","last_became_peered":"2024-09-18T08:16:13.467898+0000","last_unstale":"2024-09-18T08:17:28.436811+0000","last_undegraded":"2024-09-18T08:17:28.436811+0000","last_fullsized":"2024-09-18T08:17:28.436811+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"182'13","last_scrub_stamp":"2024-09-18T08:12:59.403054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:50.886830+0000","last_clean_scrub_stamp":"2024-09-18T08:12:59.403054+0000","objects_scrubbed":1,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,1],"acting":[2,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.1","version":"380'10108","reported_seq":18615,"reported_epoch":380,"state":"active+clean","last_fresh":"2024-09-18T08:17:29.779460+0000","last_change":"2024-09-18T08:17:25.434104+0000","last_active":"2024-09-18T08:17:29.779460+0000","last_peered":"2024-09-18T08:17:29.779460+0000","last_clean":"2024-09-18T08:17:29.779460+0000","last_became_active":"2024-09-18T08:17:25.417680+0000","last_became_peered":"2024-09-18T08:17:25.417680+0000","last_unstale":"2024-09-18T08:17:29.779460+0000","last_undegraded":"2024-09-18T08:17:29.779460+0000","last_fullsized":"2024-09-18T08:17:29.779460+0000","mapping_epoch":376,"log_start":"347'6311","ondisk_log_start":"347'6311","created":311,"last_epoch_clean":377,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:14:43.412748+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:14:43.412748+0000","last_clean_scrub_stamp":"2024-09-18T08:14:43.412748+0000","objects_scrubbed":0,"log_size":3797,"log_dups_size":0,"ondisk_log_size":3797,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":188350464,"num_objects":2874,"num_object_clones":0,"num_object_copies":8622,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2874,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10108,"num_write_kb":415424,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4169,"num_bytes_recovered":273219584,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.a","version":"74'79","reported_seq":104,"reported_epoch":379,"state":"active+clean","last_fresh":"2024-09-18T08:17:28.426831+0000","last_change":"2024-09-18T08:15:42.288426+0000","last_active":"2024-09-18T08:17:28.426831+0000","last_peered":"2024-09-18T08:17:28.426831+0000","last_clean":"2024-09-18T08:17:28.426831+0000","last_became_active":"2024-09-18T08:15:41.443376+0000","last_became_peered":"2024-09-18T08:15:41.443376+0000","last_unstale":"2024-09-18T08:17:28.426831+0000","last_undegraded":"2024-09-18T08:17:28.426831+0000","last_fullsized":"2024-09-18T08:17:28.426831+0000","mapping_epoch":336,"log_start":"0'0","ondisk_log_start":"0'0","created":152,"last_epoch_clean":337,"parent":"0.0","parent_split_bits":4,"last_scrub":"74'79","last_scrub_stamp":"2024-09-18T08:15:42.288363+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:00:11.893742+0000","last_clean_scrub_stamp":"2024-09-18T08:15:42.288363+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,6],"acting":[5,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.f","version":"271'27","reported_seq":636,"reported_epoch":380,"state":"active+clean","last_fresh":"2024-09-18T08:17:29.732577+0000","last_change":"2024-09-18T08:17:29.732577+0000","last_active":"2024-09-18T08:17:29.732577+0000","last_peered":"2024-09-18T08:17:29.732577+0000","last_clean":"2024-09-18T08:17:29.732577+0000","last_became_active":"2024-09-18T08:16:13.456369+0000","last_became_peered":"2024-09-18T08:16:13.456369+0000","last_unstale":"2024-09-18T08:17:29.732577+0000","last_undegraded":"2024-09-18T08:17:29.732577+0000","last_fullsized":"2024-09-18T08:17:29.732577+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"271'27","last_scrub_stamp":"2024-09-18T08:17:29.732496+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_clean_scrub_stamp":"2024-09-18T08:17:29.732496+0000","objects_scrubbed":7,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:18:58.561148+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":43,"num_read_kb":34,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,5],"acting":[1,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":699,"reported_epoch":379,"state":"active+clean","last_fresh":"2024-09-18T08:17:28.455492+0000","last_change":"2024-09-18T08:16:13.472791+0000","last_active":"2024-09-18T08:17:28.455492+0000","last_peered":"2024-09-18T08:17:28.455492+0000","last_clean":"2024-09-18T08:17:28.455492+0000","last_became_active":"2024-09-18T08:16:13.472353+0000","last_became_peered":"2024-09-18T08:16:13.472353+0000","last_unstale":"2024-09-18T08:17:28.455492+0000","last_undegraded":"2024-09-18T08:17:28.455492+0000","last_fullsized":"2024-09-18T08:17:28.455492+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:15:46.809403+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:46.860092+0000","last_clean_scrub_stamp":"2024-09-18T08:15:46.809403+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,5],"acting":[3,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.e","version":"349'16","reported_seq":551,"reported_epoch":379,"state":"active+clean","last_fresh":"2024-09-18T08:17:28.429559+0000","last_change":"2024-09-18T08:16:13.477202+0000","last_active":"2024-09-18T08:17:28.429559+0000","last_peered":"2024-09-18T08:17:28.429559+0000","last_clean":"2024-09-18T08:17:28.429559+0000","last_became_active":"2024-09-18T08:16:13.451199+0000","last_became_peered":"2024-09-18T08:16:13.451199+0000","last_unstale":"2024-09-18T08:17:28.429559+0000","last_undegraded":"2024-09-18T08:17:28.429559+0000","last_fullsized":"2024-09-18T08:17:28.429559+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"182'12","last_scrub_stamp":"2024-09-18T08:13:31.620205+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:50.886830+0000","last_clean_scrub_stamp":"2024-09-18T08:13:31.620205+0000","objects_scrubbed":1,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,2],"acting":[5,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":96,"reported_epoch":379,"state":"active+clean","last_fresh":"2024-09-18T08:17:28.427358+0000","last_change":"2024-09-18T08:15:41.438999+0000","last_active":"2024-09-18T08:17:28.427358+0000","last_peered":"2024-09-18T08:17:28.427358+0000","last_clean":"2024-09-18T08:17:28.427358+0000","last_became_active":"2024-09-18T08:15:41.438610+0000","last_became_peered":"2024-09-18T08:15:41.438610+0000","last_unstale":"2024-09-18T08:17:28.427358+0000","last_undegraded":"2024-09-18T08:17:28.427358+0000","last_fullsized":"2024-09-18T08:17:28.427358+0000","mapping_epoch":336,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":337,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:15:00.180309+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_clean_scrub_stamp":"2024-09-18T08:15:00.180309+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.e","version":"379'9713","reported_seq":9875,"reported_epoch":379,"state":"active+remapped+backfill_toofull","last_fresh":"2024-09-18T08:17:28.996637+0000","last_change":"2024-09-18T08:17:14.523313+0000","last_active":"2024-09-18T08:17:28.996637+0000","last_peered":"2024-09-18T08:17:28.996637+0000","last_clean":"2024-09-18T08:16:12.353466+0000","last_became_active":"2024-09-18T08:16:14.500187+0000","last_became_peered":"2024-09-18T08:16:14.500187+0000","last_unstale":"2024-09-18T08:17:28.996637+0000","last_undegraded":"2024-09-18T08:17:28.996637+0000","last_fullsized":"2024-09-18T08:17:28.996637+0000","mapping_epoch":349,"log_start":"341'5000","ondisk_log_start":"341'5000","created":311,"last_epoch_clean":312,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:14:43.412748+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:14:43.412748+0000","last_clean_scrub_stamp":"2024-09-18T08:14:43.412748+0000","objects_scrubbed":0,"log_size":4713,"log_dups_size":0,"ondisk_log_size":4713,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":182910976,"num_objects":2791,"num_object_clones":0,"num_object_copies":8373,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":2791,"num_objects_unfound":0,"num_objects_dirty":2791,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9713,"num_write_kb":400128,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,5],"acting":[2,6,4],"avail_no_missing":["2","4","6"],"object_location_counts":[{"shards":"2,4,6","objects":2791}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.5","version":"74'79","reported_seq":124,"reported_epoch":379,"state":"active+clean","last_fresh":"2024-09-18T08:17:28.435990+0000","last_change":"2024-09-18T08:16:13.445824+0000","last_active":"2024-09-18T08:17:28.435990+0000","last_peered":"2024-09-18T08:17:28.435990+0000","last_clean":"2024-09-18T08:17:28.435990+0000","last_became_active":"2024-09-18T08:16:13.445467+0000","last_became_peered":"2024-09-18T08:16:13.445467+0000","last_unstale":"2024-09-18T08:17:28.435990+0000","last_undegraded":"2024-09-18T08:17:28.435990+0000","last_fullsized":"2024-09-18T08:17:28.435990+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":152,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":3,"last_scrub":"74'79","last_scrub_stamp":"2024-09-18T08:15:58.701823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:00:11.893742+0000","last_clean_scrub_stamp":"2024-09-18T08:15:58.701823+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,5],"acting":[2,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.0","version":"271'30","reported_seq":769,"reported_epoch":379,"state":"active+clean","last_fresh":"2024-09-18T08:17:28.453085+0000","last_change":"2024-09-18T08:15:57.927197+0000","last_active":"2024-09-18T08:17:28.453085+0000","last_peered":"2024-09-18T08:17:28.453085+0000","last_clean":"2024-09-18T08:17:28.453085+0000","last_became_active":"2024-09-18T08:14:40.358956+0000","last_became_peered":"2024-09-18T08:14:40.358956+0000","last_unstale":"2024-09-18T08:17:28.453085+0000","last_undegraded":"2024-09-18T08:17:28.453085+0000","last_fullsized":"2024-09-18T08:17:28.453085+0000","mapping_epoch":308,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":309,"parent":"0.0","parent_split_bits":0,"last_scrub":"271'30","last_scrub_stamp":"2024-09-18T08:15:57.927111+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_clean_scrub_stamp":"2024-09-18T08:15:57.927111+0000","objects_scrubbed":8,"log_size":30,"log_dups_size":0,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":18,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":52,"num_read_kb":40,"num_write":32,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":721,"reported_epoch":379,"state":"active+clean","last_fresh":"2024-09-18T08:17:28.462781+0000","last_change":"2024-09-18T08:15:41.461616+0000","last_active":"2024-09-18T08:17:28.462781+0000","last_peered":"2024-09-18T08:17:28.462781+0000","last_clean":"2024-09-18T08:17:28.462781+0000","last_became_active":"2024-09-18T08:15:41.461057+0000","last_became_peered":"2024-09-18T08:15:41.461057+0000","last_unstale":"2024-09-18T08:17:28.462781+0000","last_undegraded":"2024-09-18T08:17:28.462781+0000","last_fullsized":"2024-09-18T08:17:28.462781+0000","mapping_epoch":336,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":337,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:15:08.780913+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:46.860092+0000","last_clean_scrub_stamp":"2024-09-18T08:15:08.780913+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":322,"reported_epoch":379,"state":"active+clean","last_fresh":"2024-09-18T08:17:28.457567+0000","last_change":"2024-09-18T08:16:13.479056+0000","last_active":"2024-09-18T08:17:28.457567+0000","last_peered":"2024-09-18T08:17:28.457567+0000","last_clean":"2024-09-18T08:17:28.457567+0000","last_became_active":"2024-09-18T08:16:13.478687+0000","last_became_peered":"2024-09-18T08:16:13.478687+0000","last_unstale":"2024-09-18T08:17:28.457567+0000","last_undegraded":"2024-09-18T08:17:28.457567+0000","last_fullsized":"2024-09-18T08:17:28.457567+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:12:30.732539+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:50.886830+0000","last_clean_scrub_stamp":"2024-09-18T08:12:30.732539+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":102,"reported_epoch":379,"state":"active+clean","last_fresh":"2024-09-18T08:17:28.450370+0000","last_change":"2024-09-18T08:16:13.461117+0000","last_active":"2024-09-18T08:17:28.450370+0000","last_peered":"2024-09-18T08:17:28.450370+0000","last_clean":"2024-09-18T08:17:28.450370+0000","last_became_active":"2024-09-18T08:16:13.460785+0000","last_became_peered":"2024-09-18T08:16:13.460785+0000","last_unstale":"2024-09-18T08:17:28.450370+0000","last_undegraded":"2024-09-18T08:17:28.450370+0000","last_fullsized":"2024-09-18T08:17:28.450370+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:12:02.473168+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_clean_scrub_stamp":"2024-09-18T08:12:02.473168+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:18:26.348422+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,6],"acting":[3,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.f","version":"379'10047","reported_seq":10195,"reported_epoch":379,"state":"active+clean","last_fresh":"2024-09-18T08:17:29.327889+0000","last_change":"2024-09-18T08:14:44.437378+0000","last_active":"2024-09-18T08:17:29.327889+0000","last_peered":"2024-09-18T08:17:29.327889+0000","last_clean":"2024-09-18T08:17:29.327889+0000","last_became_active":"2024-09-18T08:14:44.437103+0000","last_became_peered":"2024-09-18T08:14:44.437103+0000","last_unstale":"2024-09-18T08:17:29.327889+0000","last_undegraded":"2024-09-18T08:17:29.327889+0000","last_fullsized":"2024-09-18T08:17:29.327889+0000","mapping_epoch":311,"log_start":"352'6501","ondisk_log_start":"352'6501","created":311,"last_epoch_clean":312,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:14:43.412748+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:14:43.412748+0000","last_clean_scrub_stamp":"2024-09-18T08:14:43.412748+0000","objects_scrubbed":0,"log_size":3546,"log_dups_size":0,"ondisk_log_size":3546,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":187105280,"num_objects":2855,"num_object_clones":0,"num_object_copies":8565,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2855,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10047,"num_write_kb":412864,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.4","version":"74'79","reported_seq":544,"reported_epoch":379,"state":"active+clean","last_fresh":"2024-09-18T08:17:28.469330+0000","last_change":"2024-09-18T08:15:41.898345+0000","last_active":"2024-09-18T08:17:28.469330+0000","last_peered":"2024-09-18T08:17:28.469330+0000","last_clean":"2024-09-18T08:17:28.469330+0000","last_became_active":"2024-09-18T08:15:41.464987+0000","last_became_peered":"2024-09-18T08:15:41.464987+0000","last_unstale":"2024-09-18T08:17:28.469330+0000","last_undegraded":"2024-09-18T08:17:28.469330+0000","last_fullsized":"2024-09-18T08:17:28.469330+0000","mapping_epoch":336,"log_start":"0'0","ondisk_log_start":"0'0","created":152,"last_epoch_clean":337,"parent":"0.0","parent_split_bits":3,"last_scrub":"74'79","last_scrub_stamp":"2024-09-18T08:15:41.898294+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:00:11.893742+0000","last_clean_scrub_stamp":"2024-09-18T08:15:41.898294+0000","objects_scrubbed":1,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":327680,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":983040,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"271'28","reported_seq":479,"reported_epoch":379,"state":"active+clean","last_fresh":"2024-09-18T08:17:28.426628+0000","last_change":"2024-09-18T08:16:13.506782+0000","last_active":"2024-09-18T08:17:28.426628+0000","last_peered":"2024-09-18T08:17:28.426628+0000","last_clean":"2024-09-18T08:17:28.426628+0000","last_became_active":"2024-09-18T08:16:13.451205+0000","last_became_peered":"2024-09-18T08:16:13.451205+0000","last_unstale":"2024-09-18T08:17:28.426628+0000","last_undegraded":"2024-09-18T08:17:28.426628+0000","last_fullsized":"2024-09-18T08:17:28.426628+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"271'28","last_scrub_stamp":"2024-09-18T08:15:51.716119+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_clean_scrub_stamp":"2024-09-18T08:15:51.716119+0000","objects_scrubbed":7,"log_size":28,"log_dups_size":0,"ondisk_log_size":28,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":41,"num_read_kb":34,"num_write":28,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.6","version":"62'2","reported_seq":734,"reported_epoch":380,"state":"active+clean","last_fresh":"2024-09-18T08:17:29.478267+0000","last_change":"2024-09-18T08:16:13.508628+0000","last_active":"2024-09-18T08:17:29.478267+0000","last_peered":"2024-09-18T08:17:29.478267+0000","last_clean":"2024-09-18T08:17:29.478267+0000","last_became_active":"2024-09-18T08:16:13.451171+0000","last_became_peered":"2024-09-18T08:16:13.451171+0000","last_unstale":"2024-09-18T08:17:29.478267+0000","last_undegraded":"2024-09-18T08:17:29.478267+0000","last_fullsized":"2024-09-18T08:17:29.478267+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"62'2","last_scrub_stamp":"2024-09-18T08:14:42.145836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:46.860092+0000","last_clean_scrub_stamp":"2024-09-18T08:14:42.145836+0000","objects_scrubbed":2,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1039,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":3117,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,5],"acting":[0,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.0","version":"349'13","reported_seq":900,"reported_epoch":379,"state":"active+clean","last_fresh":"2024-09-18T08:17:28.441898+0000","last_change":"2024-09-18T08:16:13.516195+0000","last_active":"2024-09-18T08:17:28.441898+0000","last_peered":"2024-09-18T08:17:28.441898+0000","last_clean":"2024-09-18T08:17:28.441898+0000","last_became_active":"2024-09-18T08:16:13.477431+0000","last_became_peered":"2024-09-18T08:16:13.477431+0000","last_unstale":"2024-09-18T08:17:28.441898+0000","last_undegraded":"2024-09-18T08:17:28.441898+0000","last_fullsized":"2024-09-18T08:17:28.441898+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"310'12","last_scrub_stamp":"2024-09-18T08:15:49.795095+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:50.886830+0000","last_clean_scrub_stamp":"2024-09-18T08:15:49.795095+0000","objects_scrubbed":1,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":96,"reported_epoch":379,"state":"active+clean","last_fresh":"2024-09-18T08:17:28.427148+0000","last_change":"2024-09-18T08:15:41.441653+0000","last_active":"2024-09-18T08:17:28.427148+0000","last_peered":"2024-09-18T08:17:28.427148+0000","last_clean":"2024-09-18T08:17:28.427148+0000","last_became_active":"2024-09-18T08:15:41.440876+0000","last_became_peered":"2024-09-18T08:15:41.440876+0000","last_unstale":"2024-09-18T08:17:28.427148+0000","last_undegraded":"2024-09-18T08:17:28.427148+0000","last_fullsized":"2024-09-18T08:17:28.427148+0000","mapping_epoch":336,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":337,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:15:14.781554+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_clean_scrub_stamp":"2024-09-18T08:15:14.781554+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.c","version":"380'9791","reported_seq":21069,"reported_epoch":380,"state":"active+clean","last_fresh":"2024-09-18T08:17:29.785007+0000","last_change":"2024-09-18T08:16:52.555960+0000","last_active":"2024-09-18T08:17:29.785007+0000","last_peered":"2024-09-18T08:17:29.785007+0000","last_clean":"2024-09-18T08:17:29.785007+0000","last_became_active":"2024-09-18T08:16:52.532776+0000","last_became_peered":"2024-09-18T08:16:52.532776+0000","last_unstale":"2024-09-18T08:17:29.785007+0000","last_undegraded":"2024-09-18T08:17:29.785007+0000","last_fullsized":"2024-09-18T08:17:29.785007+0000","mapping_epoch":362,"log_start":"347'6048","ondisk_log_start":"347'6048","created":311,"last_epoch_clean":363,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:14:43.412748+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:14:43.412748+0000","last_clean_scrub_stamp":"2024-09-18T08:14:43.412748+0000","objects_scrubbed":0,"log_size":3743,"log_dups_size":0,"ondisk_log_size":3743,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":178716672,"num_objects":2727,"num_object_clones":0,"num_object_copies":8181,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2727,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9791,"num_write_kb":400576,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5557,"num_bytes_recovered":364183552,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,6],"acting":[0,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.7","version":"74'79","reported_seq":527,"reported_epoch":379,"state":"active+clean","last_fresh":"2024-09-18T08:17:28.462569+0000","last_change":"2024-09-18T08:15:44.872544+0000","last_active":"2024-09-18T08:17:28.462569+0000","last_peered":"2024-09-18T08:17:28.462569+0000","last_clean":"2024-09-18T08:17:28.462569+0000","last_became_active":"2024-09-18T08:15:41.466505+0000","last_became_peered":"2024-09-18T08:15:41.466505+0000","last_unstale":"2024-09-18T08:17:28.462569+0000","last_undegraded":"2024-09-18T08:17:28.462569+0000","last_fullsized":"2024-09-18T08:17:28.462569+0000","mapping_epoch":336,"log_start":"0'0","ondisk_log_start":"0'0","created":152,"last_epoch_clean":337,"parent":"0.0","parent_split_bits":3,"last_scrub":"74'79","last_scrub_stamp":"2024-09-18T08:15:44.872332+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:00:11.893742+0000","last_clean_scrub_stamp":"2024-09-18T08:15:44.872332+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.2","version":"271'20","reported_seq":691,"reported_epoch":380,"state":"active+clean","last_fresh":"2024-09-18T08:17:29.460006+0000","last_change":"2024-09-18T08:16:13.525620+0000","last_active":"2024-09-18T08:17:29.460006+0000","last_peered":"2024-09-18T08:17:29.460006+0000","last_clean":"2024-09-18T08:17:29.460006+0000","last_became_active":"2024-09-18T08:16:13.486720+0000","last_became_peered":"2024-09-18T08:16:13.486720+0000","last_unstale":"2024-09-18T08:17:29.460006+0000","last_undegraded":"2024-09-18T08:17:29.460006+0000","last_fullsized":"2024-09-18T08:17:29.460006+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"271'20","last_scrub_stamp":"2024-09-18T08:15:40.251008+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_clean_scrub_stamp":"2024-09-18T08:15:40.251008+0000","objects_scrubbed":5,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":23,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":28,"num_read_kb":23,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":96,"reported_epoch":379,"state":"active+clean","last_fresh":"2024-09-18T08:17:28.427800+0000","last_change":"2024-09-18T08:15:41.452981+0000","last_active":"2024-09-18T08:17:28.427800+0000","last_peered":"2024-09-18T08:17:28.427800+0000","last_clean":"2024-09-18T08:17:28.427800+0000","last_became_active":"2024-09-18T08:15:41.449639+0000","last_became_peered":"2024-09-18T08:15:41.449639+0000","last_unstale":"2024-09-18T08:17:28.427800+0000","last_undegraded":"2024-09-18T08:17:28.427800+0000","last_fullsized":"2024-09-18T08:17:28.427800+0000","mapping_epoch":336,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":337,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:15:39.858184+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:46.860092+0000","last_clean_scrub_stamp":"2024-09-18T08:15:39.858184+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.3","version":"338'14","reported_seq":902,"reported_epoch":380,"state":"active+clean","last_fresh":"2024-09-18T08:17:29.471610+0000","last_change":"2024-09-18T08:15:41.467411+0000","last_active":"2024-09-18T08:17:29.471610+0000","last_peered":"2024-09-18T08:17:29.471610+0000","last_clean":"2024-09-18T08:17:29.471610+0000","last_became_active":"2024-09-18T08:15:41.447667+0000","last_became_peered":"2024-09-18T08:15:41.447667+0000","last_unstale":"2024-09-18T08:17:29.471610+0000","last_undegraded":"2024-09-18T08:17:29.471610+0000","last_fullsized":"2024-09-18T08:17:29.471610+0000","mapping_epoch":336,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":337,"parent":"0.0","parent_split_bits":0,"last_scrub":"317'11","last_scrub_stamp":"2024-09-18T08:14:59.207667+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:50.886830+0000","last_clean_scrub_stamp":"2024-09-18T08:14:59.207667+0000","objects_scrubbed":1,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":697,"reported_epoch":380,"state":"active+clean","last_fresh":"2024-09-18T08:17:29.473124+0000","last_change":"2024-09-18T08:14:26.087160+0000","last_active":"2024-09-18T08:17:29.473124+0000","last_peered":"2024-09-18T08:17:29.473124+0000","last_clean":"2024-09-18T08:17:29.473124+0000","last_became_active":"2024-09-18T08:08:52.700080+0000","last_became_peered":"2024-09-18T08:08:52.700080+0000","last_unstale":"2024-09-18T08:17:29.473124+0000","last_undegraded":"2024-09-18T08:17:29.473124+0000","last_fullsized":"2024-09-18T08:17:29.473124+0000","mapping_epoch":198,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":199,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:14:26.087098+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_clean_scrub_stamp":"2024-09-18T08:14:26.087098+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.d","version":"380'9694","reported_seq":19994,"reported_epoch":380,"state":"active+clean","last_fresh":"2024-09-18T08:17:29.823693+0000","last_change":"2024-09-18T08:17:06.468945+0000","last_active":"2024-09-18T08:17:29.823693+0000","last_peered":"2024-09-18T08:17:29.823693+0000","last_clean":"2024-09-18T08:17:29.823693+0000","last_became_active":"2024-09-18T08:17:06.449951+0000","last_became_peered":"2024-09-18T08:17:06.449951+0000","last_unstale":"2024-09-18T08:17:29.823693+0000","last_undegraded":"2024-09-18T08:17:29.823693+0000","last_fullsized":"2024-09-18T08:17:29.823693+0000","mapping_epoch":370,"log_start":"347'5947","ondisk_log_start":"347'5947","created":311,"last_epoch_clean":371,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:14:43.412748+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:14:43.412748+0000","last_clean_scrub_stamp":"2024-09-18T08:14:43.412748+0000","objects_scrubbed":0,"log_size":3747,"log_dups_size":0,"ondisk_log_size":3747,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":183959576,"num_objects":2808,"num_object_clones":0,"num_object_copies":8424,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2808,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":9694,"num_write_kb":400001,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5068,"num_bytes_recovered":332070936,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,3],"acting":[0,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.6","version":"74'79","reported_seq":588,"reported_epoch":380,"state":"active+clean","last_fresh":"2024-09-18T08:17:29.476325+0000","last_change":"2024-09-18T08:14:02.837236+0000","last_active":"2024-09-18T08:17:29.476325+0000","last_peered":"2024-09-18T08:17:29.476325+0000","last_clean":"2024-09-18T08:17:29.476325+0000","last_became_active":"2024-09-18T08:07:26.426532+0000","last_became_peered":"2024-09-18T08:07:26.426532+0000","last_unstale":"2024-09-18T08:17:29.476325+0000","last_undegraded":"2024-09-18T08:17:29.476325+0000","last_fullsized":"2024-09-18T08:17:29.476325+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":152,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":3,"last_scrub":"74'79","last_scrub_stamp":"2024-09-18T08:14:02.837167+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:00:11.893742+0000","last_clean_scrub_stamp":"2024-09-18T08:14:02.837167+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,3],"acting":[0,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.3","version":"271'29","reported_seq":805,"reported_epoch":380,"state":"active+clean","last_fresh":"2024-09-18T08:17:29.472016+0000","last_change":"2024-09-18T08:15:41.493514+0000","last_active":"2024-09-18T08:17:29.472016+0000","last_peered":"2024-09-18T08:17:29.472016+0000","last_clean":"2024-09-18T08:17:29.472016+0000","last_became_active":"2024-09-18T08:15:41.451985+0000","last_became_peered":"2024-09-18T08:15:41.451985+0000","last_unstale":"2024-09-18T08:17:29.472016+0000","last_undegraded":"2024-09-18T08:17:29.472016+0000","last_fullsized":"2024-09-18T08:17:29.472016+0000","mapping_epoch":336,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":337,"parent":"0.0","parent_split_bits":0,"last_scrub":"271'29","last_scrub_stamp":"2024-09-18T08:13:33.814992+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_clean_scrub_stamp":"2024-09-18T08:13:33.814992+0000","objects_scrubbed":8,"log_size":29,"log_dups_size":0,"ondisk_log_size":29,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":54,"num_read_kb":40,"num_write":32,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":990,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.4","version":"62'1","reported_seq":564,"reported_epoch":380,"state":"active+clean","last_fresh":"2024-09-18T08:17:29.462784+0000","last_change":"2024-09-18T08:15:41.454692+0000","last_active":"2024-09-18T08:17:29.462784+0000","last_peered":"2024-09-18T08:17:29.462784+0000","last_clean":"2024-09-18T08:17:29.462784+0000","last_became_active":"2024-09-18T08:15:41.441822+0000","last_became_peered":"2024-09-18T08:15:41.441822+0000","last_unstale":"2024-09-18T08:17:29.462784+0000","last_undegraded":"2024-09-18T08:17:29.462784+0000","last_fullsized":"2024-09-18T08:17:29.462784+0000","mapping_epoch":336,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":337,"parent":"0.0","parent_split_bits":0,"last_scrub":"62'1","last_scrub_stamp":"2024-09-18T08:15:13.165826+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:46.860092+0000","last_clean_scrub_stamp":"2024-09-18T08:15:13.165826+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":17,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":436,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":1308,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":610,"reported_epoch":380,"state":"active+clean","last_fresh":"2024-09-18T08:17:29.471833+0000","last_change":"2024-09-18T08:15:41.450294+0000","last_active":"2024-09-18T08:17:29.471833+0000","last_peered":"2024-09-18T08:17:29.471833+0000","last_clean":"2024-09-18T08:17:29.471833+0000","last_became_active":"2024-09-18T08:15:41.449832+0000","last_became_peered":"2024-09-18T08:15:41.449832+0000","last_unstale":"2024-09-18T08:17:29.471833+0000","last_undegraded":"2024-09-18T08:17:29.471833+0000","last_fullsized":"2024-09-18T08:17:29.471833+0000","mapping_epoch":336,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":337,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:12:37.732260+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:50.886830+0000","last_clean_scrub_stamp":"2024-09-18T08:12:37.732260+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":583,"reported_epoch":380,"state":"active+clean","last_fresh":"2024-09-18T08:17:29.460973+0000","last_change":"2024-09-18T08:15:50.294289+0000","last_active":"2024-09-18T08:17:29.460973+0000","last_peered":"2024-09-18T08:17:29.460973+0000","last_clean":"2024-09-18T08:17:29.460973+0000","last_became_active":"2024-09-18T08:14:39.358132+0000","last_became_peered":"2024-09-18T08:14:39.358132+0000","last_unstale":"2024-09-18T08:17:29.460973+0000","last_undegraded":"2024-09-18T08:17:29.460973+0000","last_fullsized":"2024-09-18T08:17:29.460973+0000","mapping_epoch":307,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":308,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:15:50.294213+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_clean_scrub_stamp":"2024-09-18T08:15:50.294213+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":28,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.a","version":"380'9851","reported_seq":22337,"reported_epoch":380,"state":"active+clean","last_fresh":"2024-09-18T08:17:29.819158+0000","last_change":"2024-09-18T08:16:21.405814+0000","last_active":"2024-09-18T08:17:29.819158+0000","last_peered":"2024-09-18T08:17:29.819158+0000","last_clean":"2024-09-18T08:17:29.819158+0000","last_became_active":"2024-09-18T08:16:21.396170+0000","last_became_peered":"2024-09-18T08:16:21.396170+0000","last_unstale":"2024-09-18T08:17:29.819158+0000","last_undegraded":"2024-09-18T08:17:29.819158+0000","last_fullsized":"2024-09-18T08:17:29.819158+0000","mapping_epoch":354,"log_start":"347'6092","ondisk_log_start":"347'6092","created":311,"last_epoch_clean":355,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:14:43.412748+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:14:43.412748+0000","last_clean_scrub_stamp":"2024-09-18T08:14:43.412748+0000","objects_scrubbed":0,"log_size":3759,"log_dups_size":0,"ondisk_log_size":3759,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":186187776,"num_objects":2841,"num_object_clones":0,"num_object_copies":8523,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2841,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9851,"num_write_kb":406144,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6152,"num_bytes_recovered":403177472,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"74'79","reported_seq":104,"reported_epoch":379,"state":"active+clean","last_fresh":"2024-09-18T08:17:28.426830+0000","last_change":"2024-09-18T08:15:43.275398+0000","last_active":"2024-09-18T08:17:28.426830+0000","last_peered":"2024-09-18T08:17:28.426830+0000","last_clean":"2024-09-18T08:17:28.426830+0000","last_became_active":"2024-09-18T08:15:41.452895+0000","last_became_peered":"2024-09-18T08:15:41.452895+0000","last_unstale":"2024-09-18T08:17:28.426830+0000","last_undegraded":"2024-09-18T08:17:28.426830+0000","last_fullsized":"2024-09-18T08:17:28.426830+0000","mapping_epoch":336,"log_start":"0'0","ondisk_log_start":"0'0","created":152,"last_epoch_clean":337,"parent":"0.0","parent_split_bits":4,"last_scrub":"74'79","last_scrub_stamp":"2024-09-18T08:15:43.275329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:00:11.893742+0000","last_clean_scrub_stamp":"2024-09-18T08:15:43.275329+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,2],"acting":[5,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.4","version":"310'42","reported_seq":712,"reported_epoch":380,"state":"active+clean","last_fresh":"2024-09-18T08:17:29.462946+0000","last_change":"2024-09-18T08:14:39.529072+0000","last_active":"2024-09-18T08:17:29.462946+0000","last_peered":"2024-09-18T08:17:29.462946+0000","last_clean":"2024-09-18T08:17:29.462946+0000","last_became_active":"2024-09-18T08:14:39.486174+0000","last_became_peered":"2024-09-18T08:14:39.486174+0000","last_unstale":"2024-09-18T08:17:29.462946+0000","last_undegraded":"2024-09-18T08:17:29.462946+0000","last_fullsized":"2024-09-18T08:17:29.462946+0000","mapping_epoch":307,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":308,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'28","last_scrub_stamp":"2024-09-18T08:10:46.497874+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_clean_scrub_stamp":"2024-09-18T08:10:46.497874+0000","objects_scrubbed":0,"log_size":42,"log_dups_size":0,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:18:26.645395+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":65,"num_read_kb":50,"num_write":38,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":278,"reported_epoch":380,"state":"active+clean","last_fresh":"2024-09-18T08:17:29.472209+0000","last_change":"2024-09-18T08:16:13.476275+0000","last_active":"2024-09-18T08:17:29.472209+0000","last_peered":"2024-09-18T08:17:29.472209+0000","last_clean":"2024-09-18T08:17:29.472209+0000","last_became_active":"2024-09-18T08:16:13.475879+0000","last_became_peered":"2024-09-18T08:16:13.475879+0000","last_unstale":"2024-09-18T08:17:29.472209+0000","last_undegraded":"2024-09-18T08:17:29.472209+0000","last_fullsized":"2024-09-18T08:17:29.472209+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:13:52.760146+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:46.860092+0000","last_clean_scrub_stamp":"2024-09-18T08:13:52.760146+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,1],"acting":[0,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"64'2","reported_seq":108,"reported_epoch":379,"state":"active+clean","last_fresh":"2024-09-18T08:17:28.426734+0000","last_change":"2024-09-18T08:17:08.332007+0000","last_active":"2024-09-18T08:17:28.426734+0000","last_peered":"2024-09-18T08:17:28.426734+0000","last_clean":"2024-09-18T08:17:28.426734+0000","last_became_active":"2024-09-18T08:15:41.453351+0000","last_became_peered":"2024-09-18T08:15:41.453351+0000","last_unstale":"2024-09-18T08:17:28.426734+0000","last_undegraded":"2024-09-18T08:17:28.426734+0000","last_fullsized":"2024-09-18T08:17:28.426734+0000","mapping_epoch":336,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":337,"parent":"0.0","parent_split_bits":0,"last_scrub":"64'2","last_scrub_stamp":"2024-09-18T08:17:08.331955+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:45.855979+0000","last_clean_scrub_stamp":"2024-09-18T08:17:08.331955+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:18:28.285273+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":701,"reported_epoch":380,"state":"active+clean","last_fresh":"2024-09-18T08:17:29.477564+0000","last_change":"2024-09-18T08:16:13.481778+0000","last_active":"2024-09-18T08:17:29.477564+0000","last_peered":"2024-09-18T08:17:29.477564+0000","last_clean":"2024-09-18T08:17:29.477564+0000","last_became_active":"2024-09-18T08:16:13.481456+0000","last_became_peered":"2024-09-18T08:16:13.481456+0000","last_unstale":"2024-09-18T08:17:29.477564+0000","last_undegraded":"2024-09-18T08:17:29.477564+0000","last_fullsized":"2024-09-18T08:17:29.477564+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:15:04.260540+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:50.886830+0000","last_clean_scrub_stamp":"2024-09-18T08:15:04.260540+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,6],"acting":[0,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":709,"reported_epoch":379,"state":"active+clean","last_fresh":"2024-09-18T08:17:28.461496+0000","last_change":"2024-09-18T08:16:13.462219+0000","last_active":"2024-09-18T08:17:28.461496+0000","last_peered":"2024-09-18T08:17:28.461496+0000","last_clean":"2024-09-18T08:17:28.461496+0000","last_became_active":"2024-09-18T08:16:13.459207+0000","last_became_peered":"2024-09-18T08:16:13.459207+0000","last_unstale":"2024-09-18T08:17:28.461496+0000","last_undegraded":"2024-09-18T08:17:28.461496+0000","last_fullsized":"2024-09-18T08:17:28.461496+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:15:59.948801+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_clean_scrub_stamp":"2024-09-18T08:15:59.948801+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.9","version":"379'10002","reported_seq":22240,"reported_epoch":379,"state":"active+clean","last_fresh":"2024-09-18T08:17:28.665469+0000","last_change":"2024-09-18T08:16:20.408102+0000","last_active":"2024-09-18T08:17:28.665469+0000","last_peered":"2024-09-18T08:17:28.665469+0000","last_clean":"2024-09-18T08:17:28.665469+0000","last_became_active":"2024-09-18T08:16:20.398277+0000","last_became_peered":"2024-09-18T08:16:20.398277+0000","last_unstale":"2024-09-18T08:17:28.665469+0000","last_undegraded":"2024-09-18T08:17:28.665469+0000","last_fullsized":"2024-09-18T08:17:28.665469+0000","mapping_epoch":353,"log_start":"350'6332","ondisk_log_start":"350'6332","created":311,"last_epoch_clean":354,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:14:43.412748+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:14:43.412748+0000","last_clean_scrub_stamp":"2024-09-18T08:14:43.412748+0000","objects_scrubbed":0,"log_size":3670,"log_dups_size":0,"ondisk_log_size":3670,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":195297280,"num_objects":2980,"num_object_clones":0,"num_object_copies":8940,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2980,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10002,"num_write_kb":415424,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6027,"num_bytes_recovered":394985472,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,3],"acting":[5,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.2","version":"74'79","reported_seq":615,"reported_epoch":380,"state":"active+clean","last_fresh":"2024-09-18T08:17:29.471597+0000","last_change":"2024-09-18T08:16:13.453252+0000","last_active":"2024-09-18T08:17:29.471597+0000","last_peered":"2024-09-18T08:17:29.471597+0000","last_clean":"2024-09-18T08:17:29.471597+0000","last_became_active":"2024-09-18T08:16:13.451687+0000","last_became_peered":"2024-09-18T08:16:13.451687+0000","last_unstale":"2024-09-18T08:17:29.471597+0000","last_undegraded":"2024-09-18T08:17:29.471597+0000","last_fullsized":"2024-09-18T08:17:29.471597+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":152,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":4,"last_scrub":"74'79","last_scrub_stamp":"2024-09-18T08:14:01.846338+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:00:11.893742+0000","last_clean_scrub_stamp":"2024-09-18T08:14:01.846338+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,5],"acting":[0,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.7","version":"271'17","reported_seq":678,"reported_epoch":379,"state":"active+clean","last_fresh":"2024-09-18T08:17:28.437044+0000","last_change":"2024-09-18T08:15:41.494329+0000","last_active":"2024-09-18T08:17:28.437044+0000","last_peered":"2024-09-18T08:17:28.437044+0000","last_clean":"2024-09-18T08:17:28.437044+0000","last_became_active":"2024-09-18T08:15:41.456482+0000","last_became_peered":"2024-09-18T08:15:41.456482+0000","last_unstale":"2024-09-18T08:17:28.437044+0000","last_undegraded":"2024-09-18T08:17:28.437044+0000","last_fullsized":"2024-09-18T08:17:28.437044+0000","mapping_epoch":336,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":337,"parent":"0.0","parent_split_bits":0,"last_scrub":"271'17","last_scrub_stamp":"2024-09-18T08:15:13.792602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_clean_scrub_stamp":"2024-09-18T08:15:13.792602+0000","objects_scrubbed":5,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":16,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":35,"num_read_kb":24,"num_write":20,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":990,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,0],"acting":[2,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":573,"reported_epoch":380,"state":"active+clean","last_fresh":"2024-09-18T08:17:29.456658+0000","last_change":"2024-09-18T08:15:49.244718+0000","last_active":"2024-09-18T08:17:29.456658+0000","last_peered":"2024-09-18T08:17:29.456658+0000","last_clean":"2024-09-18T08:17:29.456658+0000","last_became_active":"2024-09-18T08:14:39.432254+0000","last_became_peered":"2024-09-18T08:14:39.432254+0000","last_unstale":"2024-09-18T08:17:29.456658+0000","last_undegraded":"2024-09-18T08:17:29.456658+0000","last_fullsized":"2024-09-18T08:17:29.456658+0000","mapping_epoch":307,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":308,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:15:49.244156+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:46.860092+0000","last_clean_scrub_stamp":"2024-09-18T08:15:49.244156+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"62'1","reported_seq":694,"reported_epoch":379,"state":"active+clean","last_fresh":"2024-09-18T08:17:28.446126+0000","last_change":"2024-09-18T08:14:31.765507+0000","last_active":"2024-09-18T08:17:28.446126+0000","last_peered":"2024-09-18T08:17:28.446126+0000","last_clean":"2024-09-18T08:17:28.446126+0000","last_became_active":"2024-09-18T08:02:46.868175+0000","last_became_peered":"2024-09-18T08:02:46.868175+0000","last_unstale":"2024-09-18T08:17:28.446126+0000","last_undegraded":"2024-09-18T08:17:28.446126+0000","last_fullsized":"2024-09-18T08:17:28.446126+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"62'1","last_scrub_stamp":"2024-09-18T08:14:31.765455+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:45.855979+0000","last_clean_scrub_stamp":"2024-09-18T08:14:31.765455+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.00066785499999999997,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":411,"reported_epoch":379,"state":"active+clean","last_fresh":"2024-09-18T08:17:28.438446+0000","last_change":"2024-09-18T08:16:13.467275+0000","last_active":"2024-09-18T08:17:28.438446+0000","last_peered":"2024-09-18T08:17:28.438446+0000","last_clean":"2024-09-18T08:17:28.438446+0000","last_became_active":"2024-09-18T08:16:13.466910+0000","last_became_peered":"2024-09-18T08:16:13.466910+0000","last_unstale":"2024-09-18T08:17:28.438446+0000","last_undegraded":"2024-09-18T08:17:28.438446+0000","last_fullsized":"2024-09-18T08:17:28.438446+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:15:00.702835+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:50.886830+0000","last_clean_scrub_stamp":"2024-09-18T08:15:00.702835+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,1],"acting":[2,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":365,"reported_epoch":379,"state":"active+clean","last_fresh":"2024-09-18T08:17:28.465864+0000","last_change":"2024-09-18T08:14:05.578356+0000","last_active":"2024-09-18T08:17:28.465864+0000","last_peered":"2024-09-18T08:17:28.465864+0000","last_clean":"2024-09-18T08:17:28.465864+0000","last_became_active":"2024-09-18T08:14:05.577516+0000","last_became_peered":"2024-09-18T08:14:05.577516+0000","last_unstale":"2024-09-18T08:17:28.465864+0000","last_undegraded":"2024-09-18T08:17:28.465864+0000","last_fullsized":"2024-09-18T08:17:28.465864+0000","mapping_epoch":292,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":293,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:13:36.978582+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_clean_scrub_stamp":"2024-09-18T08:13:36.978582+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.8","version":"380'9860","reported_seq":21689,"reported_epoch":380,"state":"active+clean","last_fresh":"2024-09-18T08:17:29.828110+0000","last_change":"2024-09-18T08:16:54.572396+0000","last_active":"2024-09-18T08:17:29.828110+0000","last_peered":"2024-09-18T08:17:29.828110+0000","last_clean":"2024-09-18T08:17:29.828110+0000","last_became_active":"2024-09-18T08:16:54.554366+0000","last_became_peered":"2024-09-18T08:16:54.554366+0000","last_unstale":"2024-09-18T08:17:29.828110+0000","last_undegraded":"2024-09-18T08:17:29.828110+0000","last_fullsized":"2024-09-18T08:17:29.828110+0000","mapping_epoch":364,"log_start":"347'6148","ondisk_log_start":"347'6148","created":311,"last_epoch_clean":365,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:14:43.412748+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:14:43.412748+0000","last_clean_scrub_stamp":"2024-09-18T08:14:43.412748+0000","objects_scrubbed":0,"log_size":3712,"log_dups_size":0,"ondisk_log_size":3712,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":184811520,"num_objects":2820,"num_object_clones":0,"num_object_copies":8460,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2820,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9860,"num_write_kb":405760,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5820,"num_bytes_recovered":381419520,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,5],"acting":[0,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"74'79","reported_seq":348,"reported_epoch":380,"state":"active+clean","last_fresh":"2024-09-18T08:17:29.462198+0000","last_change":"2024-09-18T08:16:13.455530+0000","last_active":"2024-09-18T08:17:29.462198+0000","last_peered":"2024-09-18T08:17:29.462198+0000","last_clean":"2024-09-18T08:17:29.462198+0000","last_became_active":"2024-09-18T08:16:13.454138+0000","last_became_peered":"2024-09-18T08:16:13.454138+0000","last_unstale":"2024-09-18T08:17:29.462198+0000","last_undegraded":"2024-09-18T08:17:29.462198+0000","last_fullsized":"2024-09-18T08:17:29.462198+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":152,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":3,"last_scrub":"74'79","last_scrub_stamp":"2024-09-18T08:16:00.303863+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:00:11.893742+0000","last_clean_scrub_stamp":"2024-09-18T08:16:00.303863+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,5],"acting":[1,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.6","version":"271'24","reported_seq":785,"reported_epoch":380,"state":"active+clean","last_fresh":"2024-09-18T08:17:29.478095+0000","last_change":"2024-09-18T08:15:10.187667+0000","last_active":"2024-09-18T08:17:29.478095+0000","last_peered":"2024-09-18T08:17:29.478095+0000","last_clean":"2024-09-18T08:17:29.478095+0000","last_became_active":"2024-09-18T08:14:39.438033+0000","last_became_peered":"2024-09-18T08:14:39.438033+0000","last_unstale":"2024-09-18T08:17:29.478095+0000","last_undegraded":"2024-09-18T08:17:29.478095+0000","last_fullsized":"2024-09-18T08:17:29.478095+0000","mapping_epoch":307,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":308,"parent":"0.0","parent_split_bits":0,"last_scrub":"271'24","last_scrub_stamp":"2024-09-18T08:15:10.187593+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_clean_scrub_stamp":"2024-09-18T08:15:10.187593+0000","objects_scrubbed":6,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":36,"num_read_kb":30,"num_write":24,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":715,"reported_epoch":380,"state":"active+clean","last_fresh":"2024-09-18T08:17:29.471549+0000","last_change":"2024-09-18T08:16:13.461291+0000","last_active":"2024-09-18T08:17:29.471549+0000","last_peered":"2024-09-18T08:17:29.471549+0000","last_clean":"2024-09-18T08:17:29.471549+0000","last_became_active":"2024-09-18T08:16:13.460391+0000","last_became_peered":"2024-09-18T08:16:13.460391+0000","last_unstale":"2024-09-18T08:17:29.471549+0000","last_undegraded":"2024-09-18T08:17:29.471549+0000","last_fullsized":"2024-09-18T08:17:29.471549+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:14:37.133544+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:46.860092+0000","last_clean_scrub_stamp":"2024-09-18T08:14:37.133544+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,5],"acting":[0,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"70'5","reported_seq":817,"reported_epoch":380,"state":"active+clean","last_fresh":"2024-09-18T08:17:30.927139+0000","last_change":"2024-09-18T08:15:57.384301+0000","last_active":"2024-09-18T08:17:30.927139+0000","last_peered":"2024-09-18T08:17:30.927139+0000","last_clean":"2024-09-18T08:17:30.927139+0000","last_became_active":"2024-09-18T08:14:40.353801+0000","last_became_peered":"2024-09-18T08:14:40.353801+0000","last_unstale":"2024-09-18T08:17:30.927139+0000","last_undegraded":"2024-09-18T08:17:30.927139+0000","last_fullsized":"2024-09-18T08:17:30.927139+0000","mapping_epoch":308,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":309,"parent":"0.0","parent_split_bits":0,"last_scrub":"70'5","last_scrub_stamp":"2024-09-18T08:15:57.384224+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:45.855979+0000","last_clean_scrub_stamp":"2024-09-18T08:15:57.384224+0000","objects_scrubbed":1,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0.000504055,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":345,"num_read_kb":340,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":389,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":96,"reported_epoch":379,"state":"active+clean","last_fresh":"2024-09-18T08:17:28.428098+0000","last_change":"2024-09-18T08:15:41.443577+0000","last_active":"2024-09-18T08:17:28.428098+0000","last_peered":"2024-09-18T08:17:28.428098+0000","last_clean":"2024-09-18T08:17:28.428098+0000","last_became_active":"2024-09-18T08:15:41.443080+0000","last_became_peered":"2024-09-18T08:15:41.443080+0000","last_unstale":"2024-09-18T08:17:28.428098+0000","last_undegraded":"2024-09-18T08:17:28.428098+0000","last_fullsized":"2024-09-18T08:17:28.428098+0000","mapping_epoch":336,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":337,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:15:02.179693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:50.886830+0000","last_clean_scrub_stamp":"2024-09-18T08:15:02.179693+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":495,"reported_epoch":380,"state":"active+clean","last_fresh":"2024-09-18T08:17:29.462617+0000","last_change":"2024-09-18T08:15:41.437061+0000","last_active":"2024-09-18T08:17:29.462617+0000","last_peered":"2024-09-18T08:17:29.462617+0000","last_clean":"2024-09-18T08:17:29.462617+0000","last_became_active":"2024-09-18T08:15:41.436692+0000","last_became_peered":"2024-09-18T08:15:41.436692+0000","last_unstale":"2024-09-18T08:17:29.462617+0000","last_undegraded":"2024-09-18T08:17:29.462617+0000","last_fullsized":"2024-09-18T08:17:29.462617+0000","mapping_epoch":336,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":337,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:15:07.219279+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_clean_scrub_stamp":"2024-09-18T08:15:07.219279+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.b","version":"380'10273","reported_seq":10423,"reported_epoch":380,"state":"active+clean","last_fresh":"2024-09-18T08:17:32.528058+0000","last_change":"2024-09-18T08:14:44.432885+0000","last_active":"2024-09-18T08:17:32.528058+0000","last_peered":"2024-09-18T08:17:32.528058+0000","last_clean":"2024-09-18T08:17:32.528058+0000","last_became_active":"2024-09-18T08:14:44.432599+0000","last_became_peered":"2024-09-18T08:14:44.432599+0000","last_unstale":"2024-09-18T08:17:32.528058+0000","last_undegraded":"2024-09-18T08:17:32.528058+0000","last_fullsized":"2024-09-18T08:17:32.528058+0000","mapping_epoch":311,"log_start":"355'6701","ondisk_log_start":"355'6701","created":311,"last_epoch_clean":312,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:14:43.412748+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:14:43.412748+0000","last_clean_scrub_stamp":"2024-09-18T08:14:43.412748+0000","objects_scrubbed":0,"log_size":3572,"log_dups_size":0,"ondisk_log_size":3572,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":172818432,"num_objects":2637,"num_object_clones":0,"num_object_copies":7911,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2637,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10273,"num_write_kb":413120,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,0],"acting":[1,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.5","version":"349'35","reported_seq":464,"reported_epoch":380,"state":"active+clean","last_fresh":"2024-09-18T08:17:29.477593+0000","last_change":"2024-09-18T08:16:13.521519+0000","last_active":"2024-09-18T08:17:29.477593+0000","last_peered":"2024-09-18T08:17:29.477593+0000","last_clean":"2024-09-18T08:17:29.477593+0000","last_became_active":"2024-09-18T08:16:13.460467+0000","last_became_peered":"2024-09-18T08:16:13.460467+0000","last_unstale":"2024-09-18T08:17:29.477593+0000","last_undegraded":"2024-09-18T08:17:29.477593+0000","last_fullsized":"2024-09-18T08:17:29.477593+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"271'33","last_scrub_stamp":"2024-09-18T08:13:28.626987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_clean_scrub_stamp":"2024-09-18T08:13:28.626987+0000","objects_scrubbed":8,"log_size":35,"log_dups_size":0,"ondisk_log_size":35,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":24,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":42,"num_read_kb":32,"num_write":25,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":308,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,3],"acting":[0,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":705,"reported_epoch":379,"state":"active+clean","last_fresh":"2024-09-18T08:17:28.450622+0000","last_change":"2024-09-18T08:15:41.464630+0000","last_active":"2024-09-18T08:17:28.450622+0000","last_peered":"2024-09-18T08:17:28.450622+0000","last_clean":"2024-09-18T08:17:28.450622+0000","last_became_active":"2024-09-18T08:15:41.464313+0000","last_became_peered":"2024-09-18T08:15:41.464313+0000","last_unstale":"2024-09-18T08:17:28.450622+0000","last_undegraded":"2024-09-18T08:17:28.450622+0000","last_fullsized":"2024-09-18T08:17:28.450622+0000","mapping_epoch":336,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":337,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:14:32.693460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:46.860092+0000","last_clean_scrub_stamp":"2024-09-18T08:14:32.693460+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":38,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"74'79","reported_seq":296,"reported_epoch":380,"state":"active+clean","last_fresh":"2024-09-18T08:17:29.475434+0000","last_change":"2024-09-18T08:14:40.357413+0000","last_active":"2024-09-18T08:17:29.475434+0000","last_peered":"2024-09-18T08:17:29.475434+0000","last_clean":"2024-09-18T08:17:29.475434+0000","last_became_active":"2024-09-18T08:14:40.357063+0000","last_became_peered":"2024-09-18T08:14:40.357063+0000","last_unstale":"2024-09-18T08:17:29.475434+0000","last_undegraded":"2024-09-18T08:17:29.475434+0000","last_fullsized":"2024-09-18T08:17:29.475434+0000","mapping_epoch":308,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":309,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'79","last_scrub_stamp":"2024-09-18T08:13:51.810296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:00:11.893742+0000","last_clean_scrub_stamp":"2024-09-18T08:13:51.810296+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,1],"acting":[0,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":541,"reported_epoch":379,"state":"active+clean","last_fresh":"2024-09-18T08:17:28.435571+0000","last_change":"2024-09-18T08:15:41.441734+0000","last_active":"2024-09-18T08:17:28.435571+0000","last_peered":"2024-09-18T08:17:28.435571+0000","last_clean":"2024-09-18T08:17:28.435571+0000","last_became_active":"2024-09-18T08:15:41.441392+0000","last_became_peered":"2024-09-18T08:15:41.441392+0000","last_unstale":"2024-09-18T08:17:28.435571+0000","last_undegraded":"2024-09-18T08:17:28.435571+0000","last_fullsized":"2024-09-18T08:17:28.435571+0000","mapping_epoch":336,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":337,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:14:38.727133+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:50.886830+0000","last_clean_scrub_stamp":"2024-09-18T08:14:38.727133+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,0],"acting":[2,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":103,"reported_epoch":379,"state":"active+clean","last_fresh":"2024-09-18T08:17:28.428043+0000","last_change":"2024-09-18T08:16:13.472241+0000","last_active":"2024-09-18T08:17:28.428043+0000","last_peered":"2024-09-18T08:17:28.428043+0000","last_clean":"2024-09-18T08:17:28.428043+0000","last_became_active":"2024-09-18T08:16:13.471911+0000","last_became_peered":"2024-09-18T08:16:13.471911+0000","last_unstale":"2024-09-18T08:17:28.428043+0000","last_undegraded":"2024-09-18T08:17:28.428043+0000","last_fullsized":"2024-09-18T08:17:28.428043+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:12:05.886281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_clean_scrub_stamp":"2024-09-18T08:12:05.886281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:18:25.503591+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,1],"acting":[5,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.0","version":"380'10245","reported_seq":21361,"reported_epoch":380,"state":"active+clean","last_fresh":"2024-09-18T08:17:32.514645+0000","last_change":"2024-09-18T08:16:57.592438+0000","last_active":"2024-09-18T08:17:32.514645+0000","last_peered":"2024-09-18T08:17:32.514645+0000","last_clean":"2024-09-18T08:17:32.514645+0000","last_became_active":"2024-09-18T08:16:57.574026+0000","last_became_peered":"2024-09-18T08:16:57.574026+0000","last_unstale":"2024-09-18T08:17:32.514645+0000","last_undegraded":"2024-09-18T08:17:32.514645+0000","last_fullsized":"2024-09-18T08:17:32.514645+0000","mapping_epoch":366,"log_start":"355'6607","ondisk_log_start":"355'6607","created":311,"last_epoch_clean":367,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:14:43.412748+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:14:43.412748+0000","last_clean_scrub_stamp":"2024-09-18T08:14:43.412748+0000","objects_scrubbed":0,"log_size":3638,"log_dups_size":0,"ondisk_log_size":3638,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":173342720,"num_objects":2645,"num_object_clones":0,"num_object_copies":7935,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2645,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10245,"num_write_kb":412480,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5473,"num_bytes_recovered":358678528,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,0],"acting":[1,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.e","version":"271'19","reported_seq":395,"reported_epoch":380,"state":"active+clean","last_fresh":"2024-09-18T08:17:30.690655+0000","last_change":"2024-09-18T08:17:30.690655+0000","last_active":"2024-09-18T08:17:30.690655+0000","last_peered":"2024-09-18T08:17:30.690655+0000","last_clean":"2024-09-18T08:17:30.690655+0000","last_became_active":"2024-09-18T08:16:13.458001+0000","last_became_peered":"2024-09-18T08:16:13.458001+0000","last_unstale":"2024-09-18T08:17:30.690655+0000","last_undegraded":"2024-09-18T08:17:30.690655+0000","last_fullsized":"2024-09-18T08:17:30.690655+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"271'19","last_scrub_stamp":"2024-09-18T08:17:30.690581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_clean_scrub_stamp":"2024-09-18T08:17:30.690581+0000","objects_scrubbed":5,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:18:59.500251+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":31,"num_read_kb":24,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,3],"acting":[1,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":359,"reported_epoch":379,"state":"active+clean","last_fresh":"2024-09-18T08:17:28.438393+0000","last_change":"2024-09-18T08:16:13.476914+0000","last_active":"2024-09-18T08:17:28.438393+0000","last_peered":"2024-09-18T08:17:28.438393+0000","last_clean":"2024-09-18T08:17:28.438393+0000","last_became_active":"2024-09-18T08:16:13.476579+0000","last_became_peered":"2024-09-18T08:16:13.476579+0000","last_unstale":"2024-09-18T08:17:28.438393+0000","last_undegraded":"2024-09-18T08:17:28.438393+0000","last_fullsized":"2024-09-18T08:17:28.438393+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:11:59.458232+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:46.860092+0000","last_clean_scrub_stamp":"2024-09-18T08:11:59.458232+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:18:25.909992+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,5],"acting":[2,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":103,"reported_epoch":379,"state":"active+clean","last_fresh":"2024-09-18T08:17:28.428046+0000","last_change":"2024-09-18T08:16:13.444166+0000","last_active":"2024-09-18T08:17:28.428046+0000","last_peered":"2024-09-18T08:17:28.428046+0000","last_clean":"2024-09-18T08:17:28.428046+0000","last_became_active":"2024-09-18T08:16:13.443884+0000","last_became_peered":"2024-09-18T08:16:13.443884+0000","last_unstale":"2024-09-18T08:17:28.428046+0000","last_undegraded":"2024-09-18T08:17:28.428046+0000","last_fullsized":"2024-09-18T08:17:28.428046+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:13:34.187158+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:50.886830+0000","last_clean_scrub_stamp":"2024-09-18T08:13:34.187158+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,2],"acting":[5,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":692,"reported_epoch":379,"state":"active+clean","last_fresh":"2024-09-18T08:17:28.469446+0000","last_change":"2024-09-18T08:15:41.465035+0000","last_active":"2024-09-18T08:17:28.469446+0000","last_peered":"2024-09-18T08:17:28.469446+0000","last_clean":"2024-09-18T08:17:28.469446+0000","last_became_active":"2024-09-18T08:15:41.464500+0000","last_became_peered":"2024-09-18T08:15:41.464500+0000","last_unstale":"2024-09-18T08:17:28.469446+0000","last_undegraded":"2024-09-18T08:17:28.469446+0000","last_fullsized":"2024-09-18T08:17:28.469446+0000","mapping_epoch":336,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":337,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:15:02.861116+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_clean_scrub_stamp":"2024-09-18T08:15:02.861116+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":81,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.3","version":"379'9706","reported_seq":22905,"reported_epoch":379,"state":"active+clean","last_fresh":"2024-09-18T08:17:29.004733+0000","last_change":"2024-09-18T08:16:40.417668+0000","last_active":"2024-09-18T08:17:29.004733+0000","last_peered":"2024-09-18T08:17:29.004733+0000","last_clean":"2024-09-18T08:17:29.004733+0000","last_became_active":"2024-09-18T08:16:40.406526+0000","last_became_peered":"2024-09-18T08:16:40.406526+0000","last_unstale":"2024-09-18T08:17:29.004733+0000","last_undegraded":"2024-09-18T08:17:29.004733+0000","last_fullsized":"2024-09-18T08:17:29.004733+0000","mapping_epoch":359,"log_start":"341'5013","ondisk_log_start":"341'5013","created":311,"last_epoch_clean":360,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:14:43.412748+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:14:43.412748+0000","last_clean_scrub_stamp":"2024-09-18T08:14:43.412748+0000","objects_scrubbed":0,"log_size":4693,"log_dups_size":0,"ondisk_log_size":4693,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":187301888,"num_objects":2858,"num_object_clones":0,"num_object_copies":8574,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2858,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9706,"num_write_kb":402048,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6505,"num_bytes_recovered":426311680,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,3],"acting":[2,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.8","version":"74'79","reported_seq":100,"reported_epoch":379,"state":"active+clean","last_fresh":"2024-09-18T08:17:28.426405+0000","last_change":"2024-09-18T08:15:41.485726+0000","last_active":"2024-09-18T08:17:28.426405+0000","last_peered":"2024-09-18T08:17:28.426405+0000","last_clean":"2024-09-18T08:17:28.426405+0000","last_became_active":"2024-09-18T08:15:41.454781+0000","last_became_peered":"2024-09-18T08:15:41.454781+0000","last_unstale":"2024-09-18T08:17:28.426405+0000","last_undegraded":"2024-09-18T08:17:28.426405+0000","last_fullsized":"2024-09-18T08:17:28.426405+0000","mapping_epoch":336,"log_start":"0'0","ondisk_log_start":"0'0","created":152,"last_epoch_clean":337,"parent":"0.0","parent_split_bits":4,"last_scrub":"74'79","last_scrub_stamp":"2024-09-18T08:14:54.675411+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:00:11.893742+0000","last_clean_scrub_stamp":"2024-09-18T08:14:54.675411+0000","objects_scrubbed":1,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262688,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":262688,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,3],"acting":[5,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.d","version":"271'31","reported_seq":324,"reported_epoch":379,"state":"active+clean","last_fresh":"2024-09-18T08:17:28.447426+0000","last_change":"2024-09-18T08:16:13.511045+0000","last_active":"2024-09-18T08:17:28.447426+0000","last_peered":"2024-09-18T08:17:28.447426+0000","last_clean":"2024-09-18T08:17:28.447426+0000","last_became_active":"2024-09-18T08:16:13.466471+0000","last_became_peered":"2024-09-18T08:16:13.466471+0000","last_unstale":"2024-09-18T08:17:28.447426+0000","last_undegraded":"2024-09-18T08:17:28.447426+0000","last_fullsized":"2024-09-18T08:17:28.447426+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"271'31","last_scrub_stamp":"2024-09-18T08:15:03.678670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_clean_scrub_stamp":"2024-09-18T08:15:03.678670+0000","objects_scrubbed":0,"log_size":31,"log_dups_size":0,"ondisk_log_size":31,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":48,"num_read_kb":38,"num_write":32,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,6],"acting":[2,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":350,"reported_epoch":380,"state":"active+clean","last_fresh":"2024-09-18T08:17:29.460088+0000","last_change":"2024-09-18T08:16:13.487196+0000","last_active":"2024-09-18T08:17:29.460088+0000","last_peered":"2024-09-18T08:17:29.460088+0000","last_clean":"2024-09-18T08:17:29.460088+0000","last_became_active":"2024-09-18T08:16:13.483143+0000","last_became_peered":"2024-09-18T08:16:13.483143+0000","last_unstale":"2024-09-18T08:17:29.460088+0000","last_undegraded":"2024-09-18T08:17:29.460088+0000","last_fullsized":"2024-09-18T08:17:29.460088+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:15:09.614752+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:46.860092+0000","last_clean_scrub_stamp":"2024-09-18T08:15:09.614752+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,3],"acting":[1,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":580,"reported_epoch":380,"state":"active+clean","last_fresh":"2024-09-18T08:17:29.462718+0000","last_change":"2024-09-18T08:16:13.460968+0000","last_active":"2024-09-18T08:17:29.462718+0000","last_peered":"2024-09-18T08:17:29.462718+0000","last_clean":"2024-09-18T08:17:29.462718+0000","last_became_active":"2024-09-18T08:16:13.460650+0000","last_became_peered":"2024-09-18T08:16:13.460650+0000","last_unstale":"2024-09-18T08:17:29.462718+0000","last_undegraded":"2024-09-18T08:17:29.462718+0000","last_fullsized":"2024-09-18T08:17:29.462718+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:14:42.230420+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:50.886830+0000","last_clean_scrub_stamp":"2024-09-18T08:14:42.230420+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,6],"acting":[1,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":406,"reported_epoch":379,"state":"active+clean","last_fresh":"2024-09-18T08:17:28.436959+0000","last_change":"2024-09-18T08:16:13.458430+0000","last_active":"2024-09-18T08:17:28.436959+0000","last_peered":"2024-09-18T08:17:28.436959+0000","last_clean":"2024-09-18T08:17:28.436959+0000","last_became_active":"2024-09-18T08:16:13.457516+0000","last_became_peered":"2024-09-18T08:16:13.457516+0000","last_unstale":"2024-09-18T08:17:28.436959+0000","last_undegraded":"2024-09-18T08:17:28.436959+0000","last_fullsized":"2024-09-18T08:17:28.436959+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:14:35.757380+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_clean_scrub_stamp":"2024-09-18T08:14:35.757380+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.2","version":"380'10060","reported_seq":10231,"reported_epoch":380,"state":"active+remapped+backfill_toofull","last_fresh":"2024-09-18T08:17:32.557444+0000","last_change":"2024-09-18T08:17:14.611706+0000","last_active":"2024-09-18T08:17:32.557444+0000","last_peered":"2024-09-18T08:17:32.557444+0000","last_clean":"2024-09-18T08:16:12.397450+0000","last_became_active":"2024-09-18T08:16:14.545962+0000","last_became_peered":"2024-09-18T08:16:14.545962+0000","last_unstale":"2024-09-18T08:17:32.557444+0000","last_undegraded":"2024-09-18T08:17:32.557444+0000","last_fullsized":"2024-09-18T08:17:32.557444+0000","mapping_epoch":349,"log_start":"355'6454","ondisk_log_start":"355'6454","created":311,"last_epoch_clean":312,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:14:43.412748+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:14:43.412748+0000","last_clean_scrub_stamp":"2024-09-18T08:14:43.412748+0000","objects_scrubbed":0,"log_size":3606,"log_dups_size":0,"ondisk_log_size":3606,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":167510016,"num_objects":2556,"num_object_clones":0,"num_object_copies":7668,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":2556,"num_objects_unfound":0,"num_objects_dirty":2556,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10060,"num_write_kb":403712,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":65536,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,4],"avail_no_missing":["1","0","4"],"object_location_counts":[{"shards":"0,1,4","objects":2556}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"74'79","reported_seq":510,"reported_epoch":379,"state":"active+clean","last_fresh":"2024-09-18T08:17:28.457482+0000","last_change":"2024-09-18T08:15:41.467055+0000","last_active":"2024-09-18T08:17:28.457482+0000","last_peered":"2024-09-18T08:17:28.457482+0000","last_clean":"2024-09-18T08:17:28.457482+0000","last_became_active":"2024-09-18T08:15:41.465341+0000","last_became_peered":"2024-09-18T08:15:41.465341+0000","last_unstale":"2024-09-18T08:17:28.457482+0000","last_undegraded":"2024-09-18T08:17:28.457482+0000","last_fullsized":"2024-09-18T08:17:28.457482+0000","mapping_epoch":336,"log_start":"0'0","ondisk_log_start":"0'0","created":152,"last_epoch_clean":337,"parent":"0.0","parent_split_bits":4,"last_scrub":"74'79","last_scrub_stamp":"2024-09-18T08:13:52.510099+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:00:11.893742+0000","last_clean_scrub_stamp":"2024-09-18T08:13:52.510099+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.c","version":"271'20","reported_seq":302,"reported_epoch":379,"state":"active+clean","last_fresh":"2024-09-18T08:17:28.469890+0000","last_change":"2024-09-18T08:16:13.526582+0000","last_active":"2024-09-18T08:17:28.469890+0000","last_peered":"2024-09-18T08:17:28.469890+0000","last_clean":"2024-09-18T08:17:28.469890+0000","last_became_active":"2024-09-18T08:16:13.476918+0000","last_became_peered":"2024-09-18T08:16:13.476918+0000","last_unstale":"2024-09-18T08:17:28.469890+0000","last_undegraded":"2024-09-18T08:17:28.469890+0000","last_fullsized":"2024-09-18T08:17:28.469890+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"271'20","last_scrub_stamp":"2024-09-18T08:15:56.673129+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_clean_scrub_stamp":"2024-09-18T08:15:56.673129+0000","objects_scrubbed":0,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":28,"num_read_kb":23,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,1],"acting":[3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":712,"reported_epoch":379,"state":"active+clean","last_fresh":"2024-09-18T08:17:28.460511+0000","last_change":"2024-09-18T08:16:13.479508+0000","last_active":"2024-09-18T08:17:28.460511+0000","last_peered":"2024-09-18T08:17:28.460511+0000","last_clean":"2024-09-18T08:17:28.460511+0000","last_became_active":"2024-09-18T08:16:13.479148+0000","last_became_peered":"2024-09-18T08:16:13.479148+0000","last_unstale":"2024-09-18T08:17:28.460511+0000","last_undegraded":"2024-09-18T08:17:28.460511+0000","last_fullsized":"2024-09-18T08:17:28.460511+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:14:04.626103+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:46.860092+0000","last_clean_scrub_stamp":"2024-09-18T08:14:04.626103+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.d","version":"338'13","reported_seq":905,"reported_epoch":379,"state":"active+clean","last_fresh":"2024-09-18T08:17:28.438126+0000","last_change":"2024-09-18T08:15:41.471980+0000","last_active":"2024-09-18T08:17:28.438126+0000","last_peered":"2024-09-18T08:17:28.438126+0000","last_clean":"2024-09-18T08:17:28.438126+0000","last_became_active":"2024-09-18T08:15:41.441562+0000","last_became_peered":"2024-09-18T08:15:41.441562+0000","last_unstale":"2024-09-18T08:17:28.438126+0000","last_undegraded":"2024-09-18T08:17:28.438126+0000","last_fullsized":"2024-09-18T08:17:28.438126+0000","mapping_epoch":336,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":337,"parent":"0.0","parent_split_bits":0,"last_scrub":"293'9","last_scrub_stamp":"2024-09-18T08:14:37.759701+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:50.886830+0000","last_clean_scrub_stamp":"2024-09-18T08:14:37.759701+0000","objects_scrubbed":1,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,0],"acting":[2,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":357,"reported_epoch":380,"state":"active+clean","last_fresh":"2024-09-18T08:17:29.463317+0000","last_change":"2024-09-18T08:16:13.465932+0000","last_active":"2024-09-18T08:17:29.463317+0000","last_peered":"2024-09-18T08:17:29.463317+0000","last_clean":"2024-09-18T08:17:29.463317+0000","last_became_active":"2024-09-18T08:16:13.464924+0000","last_became_peered":"2024-09-18T08:16:13.464924+0000","last_unstale":"2024-09-18T08:17:29.463317+0000","last_undegraded":"2024-09-18T08:17:29.463317+0000","last_fullsized":"2024-09-18T08:17:29.463317+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:15:47.604976+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_clean_scrub_stamp":"2024-09-18T08:15:47.604976+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.5","version":"379'9912","reported_seq":10060,"reported_epoch":379,"state":"active+clean","last_fresh":"2024-09-18T08:17:28.975673+0000","last_change":"2024-09-18T08:14:44.432113+0000","last_active":"2024-09-18T08:17:28.975673+0000","last_peered":"2024-09-18T08:17:28.975673+0000","last_clean":"2024-09-18T08:17:28.975673+0000","last_became_active":"2024-09-18T08:14:44.431835+0000","last_became_peered":"2024-09-18T08:14:44.431835+0000","last_unstale":"2024-09-18T08:17:28.975673+0000","last_undegraded":"2024-09-18T08:17:28.975673+0000","last_fullsized":"2024-09-18T08:17:28.975673+0000","mapping_epoch":311,"log_start":"341'5213","ondisk_log_start":"341'5213","created":311,"last_epoch_clean":312,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:14:43.412748+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:14:43.412748+0000","last_clean_scrub_stamp":"2024-09-18T08:14:43.412748+0000","objects_scrubbed":0,"log_size":4699,"log_dups_size":0,"ondisk_log_size":4699,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":184287232,"num_objects":2812,"num_object_clones":0,"num_object_copies":8436,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2812,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9912,"num_write_kb":407168,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.b","version":"271'15","reported_seq":769,"reported_epoch":380,"state":"active+clean","last_fresh":"2024-09-18T08:17:29.471663+0000","last_change":"2024-09-18T08:16:13.523741+0000","last_active":"2024-09-18T08:17:29.471663+0000","last_peered":"2024-09-18T08:17:29.471663+0000","last_clean":"2024-09-18T08:17:29.471663+0000","last_became_active":"2024-09-18T08:16:13.480279+0000","last_became_peered":"2024-09-18T08:16:13.480279+0000","last_unstale":"2024-09-18T08:17:29.471663+0000","last_undegraded":"2024-09-18T08:17:29.471663+0000","last_fullsized":"2024-09-18T08:17:29.471663+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'9","last_scrub_stamp":"2024-09-18T08:10:47.928596+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_clean_scrub_stamp":"2024-09-18T08:10:47.928596+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":16,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,3],"acting":[0,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":96,"reported_epoch":379,"state":"active+clean","last_fresh":"2024-09-18T08:17:28.427343+0000","last_change":"2024-09-18T08:15:41.443068+0000","last_active":"2024-09-18T08:17:28.427343+0000","last_peered":"2024-09-18T08:17:28.427343+0000","last_clean":"2024-09-18T08:17:28.427343+0000","last_became_active":"2024-09-18T08:15:41.442697+0000","last_became_peered":"2024-09-18T08:15:41.442697+0000","last_unstale":"2024-09-18T08:17:28.427343+0000","last_undegraded":"2024-09-18T08:17:28.427343+0000","last_fullsized":"2024-09-18T08:17:28.427343+0000","mapping_epoch":336,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":337,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:11:22.076452+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:46.860092+0000","last_clean_scrub_stamp":"2024-09-18T08:11:22.076452+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:18:24.518634+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":697,"reported_epoch":379,"state":"active+clean","last_fresh":"2024-09-18T08:17:28.434890+0000","last_change":"2024-09-18T08:16:46.007260+0000","last_active":"2024-09-18T08:17:28.434890+0000","last_peered":"2024-09-18T08:17:28.434890+0000","last_clean":"2024-09-18T08:17:28.434890+0000","last_became_active":"2024-09-18T08:16:13.465129+0000","last_became_peered":"2024-09-18T08:16:13.465129+0000","last_unstale":"2024-09-18T08:17:28.434890+0000","last_undegraded":"2024-09-18T08:17:28.434890+0000","last_fullsized":"2024-09-18T08:17:28.434890+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:16:46.007194+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:50.886830+0000","last_clean_scrub_stamp":"2024-09-18T08:16:46.007194+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:17:53.322101+0000","scrub_duration":27,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":703,"reported_epoch":380,"state":"active+clean","last_fresh":"2024-09-18T08:17:29.471453+0000","last_change":"2024-09-18T08:14:05.585049+0000","last_active":"2024-09-18T08:17:29.471453+0000","last_peered":"2024-09-18T08:17:29.471453+0000","last_clean":"2024-09-18T08:17:29.471453+0000","last_became_active":"2024-09-18T08:14:05.583887+0000","last_became_peered":"2024-09-18T08:14:05.583887+0000","last_unstale":"2024-09-18T08:17:29.471453+0000","last_undegraded":"2024-09-18T08:17:29.471453+0000","last_fullsized":"2024-09-18T08:17:29.471453+0000","mapping_epoch":292,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":293,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:12:38.702798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_clean_scrub_stamp":"2024-09-18T08:12:38.702798+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,3],"acting":[0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.4","version":"380'9897","reported_seq":17935,"reported_epoch":380,"state":"active+clean","last_fresh":"2024-09-18T08:17:29.789586+0000","last_change":"2024-09-18T08:17:26.449992+0000","last_active":"2024-09-18T08:17:29.789586+0000","last_peered":"2024-09-18T08:17:29.789586+0000","last_clean":"2024-09-18T08:17:29.789586+0000","last_became_active":"2024-09-18T08:17:26.419323+0000","last_became_peered":"2024-09-18T08:17:26.419323+0000","last_unstale":"2024-09-18T08:17:29.789586+0000","last_undegraded":"2024-09-18T08:17:29.789586+0000","last_fullsized":"2024-09-18T08:17:29.789586+0000","mapping_epoch":377,"log_start":"347'6143","ondisk_log_start":"347'6143","created":311,"last_epoch_clean":378,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:14:43.412748+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:14:43.412748+0000","last_clean_scrub_stamp":"2024-09-18T08:14:43.412748+0000","objects_scrubbed":0,"log_size":3754,"log_dups_size":0,"ondisk_log_size":3754,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":182255616,"num_objects":2781,"num_object_clones":0,"num_object_copies":8343,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2781,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9897,"num_write_kb":405696,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3934,"num_bytes_recovered":257818624,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1],"acting":[0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.a","version":"271'35","reported_seq":610,"reported_epoch":380,"state":"active+clean","last_fresh":"2024-09-18T08:17:29.460245+0000","last_change":"2024-09-18T08:15:43.282823+0000","last_active":"2024-09-18T08:17:29.460245+0000","last_peered":"2024-09-18T08:17:29.460245+0000","last_clean":"2024-09-18T08:17:29.460245+0000","last_became_active":"2024-09-18T08:15:41.441186+0000","last_became_peered":"2024-09-18T08:15:41.441186+0000","last_unstale":"2024-09-18T08:17:29.460245+0000","last_undegraded":"2024-09-18T08:17:29.460245+0000","last_fullsized":"2024-09-18T08:17:29.460245+0000","mapping_epoch":336,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":337,"parent":"0.0","parent_split_bits":0,"last_scrub":"271'35","last_scrub_stamp":"2024-09-18T08:15:43.282736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_clean_scrub_stamp":"2024-09-18T08:15:43.282736+0000","objects_scrubbed":9,"log_size":35,"log_dups_size":0,"ondisk_log_size":35,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":55,"num_read_kb":44,"num_write":36,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":44,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,5],"acting":[1,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":96,"reported_epoch":379,"state":"active+clean","last_fresh":"2024-09-18T08:17:28.427510+0000","last_change":"2024-09-18T08:15:41.452272+0000","last_active":"2024-09-18T08:17:28.427510+0000","last_peered":"2024-09-18T08:17:28.427510+0000","last_clean":"2024-09-18T08:17:28.427510+0000","last_became_active":"2024-09-18T08:15:41.446405+0000","last_became_peered":"2024-09-18T08:15:41.446405+0000","last_unstale":"2024-09-18T08:17:28.427510+0000","last_undegraded":"2024-09-18T08:17:28.427510+0000","last_fullsized":"2024-09-18T08:17:28.427510+0000","mapping_epoch":336,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":337,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:12:31.205468+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:46.860092+0000","last_clean_scrub_stamp":"2024-09-18T08:12:31.205468+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:18:27.536995+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,3],"acting":[5,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":405,"reported_epoch":379,"state":"active+clean","last_fresh":"2024-09-18T08:17:28.437343+0000","last_change":"2024-09-18T08:15:41.447379+0000","last_active":"2024-09-18T08:17:28.437343+0000","last_peered":"2024-09-18T08:17:28.437343+0000","last_clean":"2024-09-18T08:17:28.437343+0000","last_became_active":"2024-09-18T08:15:41.447075+0000","last_became_peered":"2024-09-18T08:15:41.447075+0000","last_unstale":"2024-09-18T08:17:28.437343+0000","last_undegraded":"2024-09-18T08:17:28.437343+0000","last_fullsized":"2024-09-18T08:17:28.437343+0000","mapping_epoch":336,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":337,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:15:10.767720+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:50.886830+0000","last_clean_scrub_stamp":"2024-09-18T08:15:10.767720+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":526,"reported_epoch":379,"state":"active+clean","last_fresh":"2024-09-18T08:17:28.436358+0000","last_change":"2024-09-18T08:15:41.722212+0000","last_active":"2024-09-18T08:17:28.436358+0000","last_peered":"2024-09-18T08:17:28.436358+0000","last_clean":"2024-09-18T08:17:28.436358+0000","last_became_active":"2024-09-18T08:14:05.578053+0000","last_became_peered":"2024-09-18T08:14:05.578053+0000","last_unstale":"2024-09-18T08:17:28.436358+0000","last_undegraded":"2024-09-18T08:17:28.436358+0000","last_fullsized":"2024-09-18T08:17:28.436358+0000","mapping_epoch":292,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":293,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:15:41.722163+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_clean_scrub_stamp":"2024-09-18T08:15:41.722163+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.7","version":"379'9889","reported_seq":20482,"reported_epoch":379,"state":"active+clean","last_fresh":"2024-09-18T08:17:29.309254+0000","last_change":"2024-09-18T08:17:06.465603+0000","last_active":"2024-09-18T08:17:29.309254+0000","last_peered":"2024-09-18T08:17:29.309254+0000","last_clean":"2024-09-18T08:17:29.309254+0000","last_became_active":"2024-09-18T08:17:06.445544+0000","last_became_peered":"2024-09-18T08:17:06.445544+0000","last_unstale":"2024-09-18T08:17:29.309254+0000","last_undegraded":"2024-09-18T08:17:29.309254+0000","last_fullsized":"2024-09-18T08:17:29.309254+0000","mapping_epoch":370,"log_start":"354'6405","ondisk_log_start":"354'6405","created":311,"last_epoch_clean":371,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:14:43.412748+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:14:43.412748+0000","last_clean_scrub_stamp":"2024-09-18T08:14:43.412748+0000","objects_scrubbed":0,"log_size":3484,"log_dups_size":0,"ondisk_log_size":3484,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":182386688,"num_objects":2783,"num_object_clones":0,"num_object_copies":8349,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2783,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9889,"num_write_kb":405504,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5204,"num_bytes_recovered":341049344,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.9","version":"271'18","reported_seq":417,"reported_epoch":380,"state":"active+clean","last_fresh":"2024-09-18T08:17:29.461151+0000","last_change":"2024-09-18T08:16:13.525130+0000","last_active":"2024-09-18T08:17:29.461151+0000","last_peered":"2024-09-18T08:17:29.461151+0000","last_clean":"2024-09-18T08:17:29.461151+0000","last_became_active":"2024-09-18T08:16:13.482580+0000","last_became_peered":"2024-09-18T08:16:13.482580+0000","last_unstale":"2024-09-18T08:17:29.461151+0000","last_undegraded":"2024-09-18T08:17:29.461151+0000","last_fullsized":"2024-09-18T08:17:29.461151+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"271'18","last_scrub_stamp":"2024-09-18T08:15:49.666776+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_clean_scrub_stamp":"2024-09-18T08:15:49.666776+0000","objects_scrubbed":5,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":33,"num_read_kb":24,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":880,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,6],"acting":[1,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":546,"reported_epoch":380,"state":"active+clean","last_fresh":"2024-09-18T08:17:29.463621+0000","last_change":"2024-09-18T08:16:13.466230+0000","last_active":"2024-09-18T08:17:29.463621+0000","last_peered":"2024-09-18T08:17:29.463621+0000","last_clean":"2024-09-18T08:17:29.463621+0000","last_became_active":"2024-09-18T08:16:13.462268+0000","last_became_peered":"2024-09-18T08:16:13.462268+0000","last_unstale":"2024-09-18T08:17:29.463621+0000","last_undegraded":"2024-09-18T08:17:29.463621+0000","last_fullsized":"2024-09-18T08:17:29.463621+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:10:57.579577+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:46.860092+0000","last_clean_scrub_stamp":"2024-09-18T08:10:57.579577+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,5],"acting":[1,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":687,"reported_epoch":379,"state":"active+clean","last_fresh":"2024-09-18T08:17:28.436448+0000","last_change":"2024-09-18T08:14:41.671839+0000","last_active":"2024-09-18T08:17:28.436448+0000","last_peered":"2024-09-18T08:17:28.436448+0000","last_clean":"2024-09-18T08:17:28.436448+0000","last_became_active":"2024-09-18T08:14:39.399301+0000","last_became_peered":"2024-09-18T08:14:39.399301+0000","last_unstale":"2024-09-18T08:17:28.436448+0000","last_undegraded":"2024-09-18T08:17:28.436448+0000","last_fullsized":"2024-09-18T08:17:28.436448+0000","mapping_epoch":307,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":308,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:14:41.671793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:50.886830+0000","last_clean_scrub_stamp":"2024-09-18T08:14:41.671793+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":697,"reported_epoch":379,"state":"active+clean","last_fresh":"2024-09-18T08:17:28.459295+0000","last_change":"2024-09-18T08:17:09.487909+0000","last_active":"2024-09-18T08:17:28.459295+0000","last_peered":"2024-09-18T08:17:28.459295+0000","last_clean":"2024-09-18T08:17:28.459295+0000","last_became_active":"2024-09-18T08:14:40.356451+0000","last_became_peered":"2024-09-18T08:14:40.356451+0000","last_unstale":"2024-09-18T08:17:28.459295+0000","last_undegraded":"2024-09-18T08:17:28.459295+0000","last_fullsized":"2024-09-18T08:17:28.459295+0000","mapping_epoch":308,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":309,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:17:09.487823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_clean_scrub_stamp":"2024-09-18T08:17:09.487823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:18:35.062728+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.6","version":"379'9757","reported_seq":22078,"reported_epoch":379,"state":"active+clean","last_fresh":"2024-09-18T08:17:28.676630+0000","last_change":"2024-09-18T08:16:53.555560+0000","last_active":"2024-09-18T08:17:28.676630+0000","last_peered":"2024-09-18T08:17:28.676630+0000","last_clean":"2024-09-18T08:17:28.676630+0000","last_became_active":"2024-09-18T08:16:53.541405+0000","last_became_peered":"2024-09-18T08:16:53.541405+0000","last_unstale":"2024-09-18T08:17:28.676630+0000","last_undegraded":"2024-09-18T08:17:28.676630+0000","last_fullsized":"2024-09-18T08:17:28.676630+0000","mapping_epoch":363,"log_start":"350'6117","ondisk_log_start":"350'6117","created":311,"last_epoch_clean":364,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:14:43.412748+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:14:43.412748+0000","last_clean_scrub_stamp":"2024-09-18T08:14:43.412748+0000","objects_scrubbed":0,"log_size":3640,"log_dups_size":0,"ondisk_log_size":3640,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":187629568,"num_objects":2863,"num_object_clones":0,"num_object_copies":8589,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2863,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9757,"num_write_kb":403840,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6069,"num_bytes_recovered":397737984,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,1],"acting":[5,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.8","version":"271'27","reported_seq":135,"reported_epoch":379,"state":"active+clean","last_fresh":"2024-09-18T08:17:28.426467+0000","last_change":"2024-09-18T08:16:13.502022+0000","last_active":"2024-09-18T08:17:28.426467+0000","last_peered":"2024-09-18T08:17:28.426467+0000","last_clean":"2024-09-18T08:17:28.426467+0000","last_became_active":"2024-09-18T08:16:13.451720+0000","last_became_peered":"2024-09-18T08:16:13.451720+0000","last_unstale":"2024-09-18T08:17:28.426467+0000","last_undegraded":"2024-09-18T08:17:28.426467+0000","last_fullsized":"2024-09-18T08:17:28.426467+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"271'27","last_scrub_stamp":"2024-09-18T08:15:00.680764+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_clean_scrub_stamp":"2024-09-18T08:15:00.680764+0000","objects_scrubbed":7,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":44,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":28,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,3],"acting":[5,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.f","version":"62'2","reported_seq":92,"reported_epoch":380,"state":"active+clean","last_fresh":"2024-09-18T08:17:29.477596+0000","last_change":"2024-09-18T08:16:13.496793+0000","last_active":"2024-09-18T08:17:29.477596+0000","last_peered":"2024-09-18T08:17:29.477596+0000","last_clean":"2024-09-18T08:17:29.477596+0000","last_became_active":"2024-09-18T08:16:13.450206+0000","last_became_peered":"2024-09-18T08:16:13.450206+0000","last_unstale":"2024-09-18T08:17:29.477596+0000","last_undegraded":"2024-09-18T08:17:29.477596+0000","last_fullsized":"2024-09-18T08:17:29.477596+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"62'2","last_scrub_stamp":"2024-09-18T08:10:53.580444+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:46.860092+0000","last_clean_scrub_stamp":"2024-09-18T08:10:53.580444+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":92,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,5],"acting":[0,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.9","version":"349'17","reported_seq":529,"reported_epoch":380,"state":"active+clean","last_fresh":"2024-09-18T08:17:29.460625+0000","last_change":"2024-09-18T08:16:13.513889+0000","last_active":"2024-09-18T08:17:29.460625+0000","last_peered":"2024-09-18T08:17:29.460625+0000","last_clean":"2024-09-18T08:17:29.460625+0000","last_became_active":"2024-09-18T08:16:13.474474+0000","last_became_peered":"2024-09-18T08:16:13.474474+0000","last_unstale":"2024-09-18T08:17:29.460625+0000","last_undegraded":"2024-09-18T08:17:29.460625+0000","last_fullsized":"2024-09-18T08:17:29.460625+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"294'14","last_scrub_stamp":"2024-09-18T08:14:28.686640+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:50.886830+0000","last_clean_scrub_stamp":"2024-09-18T08:14:28.686640+0000","objects_scrubbed":1,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,3],"acting":[1,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":96,"reported_epoch":379,"state":"active+clean","last_fresh":"2024-09-18T08:17:28.426771+0000","last_change":"2024-09-18T08:15:41.454432+0000","last_active":"2024-09-18T08:17:28.426771+0000","last_peered":"2024-09-18T08:17:28.426771+0000","last_clean":"2024-09-18T08:17:28.426771+0000","last_became_active":"2024-09-18T08:15:41.449889+0000","last_became_peered":"2024-09-18T08:15:41.449889+0000","last_unstale":"2024-09-18T08:17:28.426771+0000","last_undegraded":"2024-09-18T08:17:28.426771+0000","last_fullsized":"2024-09-18T08:17:28.426771+0000","mapping_epoch":336,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":337,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:15:13.161614+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_clean_scrub_stamp":"2024-09-18T08:15:13.161614+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":617,"reported_epoch":380,"state":"active+clean","last_fresh":"2024-09-18T08:17:29.473212+0000","last_change":"2024-09-18T08:15:41.448597+0000","last_active":"2024-09-18T08:17:29.473212+0000","last_peered":"2024-09-18T08:17:29.473212+0000","last_clean":"2024-09-18T08:17:29.473212+0000","last_became_active":"2024-09-18T08:15:41.446544+0000","last_became_peered":"2024-09-18T08:15:41.446544+0000","last_unstale":"2024-09-18T08:17:29.473212+0000","last_undegraded":"2024-09-18T08:17:29.473212+0000","last_fullsized":"2024-09-18T08:17:29.473212+0000","mapping_epoch":336,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":337,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:13:06.860320+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:46.860092+0000","last_clean_scrub_stamp":"2024-09-18T08:13:06.860320+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.17","version":"271'12","reported_seq":739,"reported_epoch":379,"state":"active+clean","last_fresh":"2024-09-18T08:17:28.462654+0000","last_change":"2024-09-18T08:14:40.365750+0000","last_active":"2024-09-18T08:17:28.462654+0000","last_peered":"2024-09-18T08:17:28.462654+0000","last_clean":"2024-09-18T08:17:28.462654+0000","last_became_active":"2024-09-18T08:14:40.355751+0000","last_became_peered":"2024-09-18T08:14:40.355751+0000","last_unstale":"2024-09-18T08:17:28.462654+0000","last_undegraded":"2024-09-18T08:17:28.462654+0000","last_fullsized":"2024-09-18T08:17:28.462654+0000","mapping_epoch":308,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":309,"parent":"0.0","parent_split_bits":0,"last_scrub":"271'12","last_scrub_stamp":"2024-09-18T08:14:33.700931+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_clean_scrub_stamp":"2024-09-18T08:14:33.700931+0000","objects_scrubbed":3,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":18,"num_read_kb":15,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":96,"reported_epoch":379,"state":"active+clean","last_fresh":"2024-09-18T08:17:28.429667+0000","last_change":"2024-09-18T08:15:41.448984+0000","last_active":"2024-09-18T08:17:28.429667+0000","last_peered":"2024-09-18T08:17:28.429667+0000","last_clean":"2024-09-18T08:17:28.429667+0000","last_became_active":"2024-09-18T08:15:41.445332+0000","last_became_peered":"2024-09-18T08:15:41.445332+0000","last_unstale":"2024-09-18T08:17:28.429667+0000","last_undegraded":"2024-09-18T08:17:28.429667+0000","last_fullsized":"2024-09-18T08:17:28.429667+0000","mapping_epoch":336,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":337,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:15:07.747147+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:50.886830+0000","last_clean_scrub_stamp":"2024-09-18T08:15:07.747147+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,1],"acting":[5,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":462,"reported_epoch":379,"state":"active+clean","last_fresh":"2024-09-18T08:17:28.427787+0000","last_change":"2024-09-18T08:16:13.453780+0000","last_active":"2024-09-18T08:17:28.427787+0000","last_peered":"2024-09-18T08:17:28.427787+0000","last_clean":"2024-09-18T08:17:28.427787+0000","last_became_active":"2024-09-18T08:16:13.452993+0000","last_became_peered":"2024-09-18T08:16:13.452993+0000","last_unstale":"2024-09-18T08:17:28.427787+0000","last_undegraded":"2024-09-18T08:17:28.427787+0000","last_fullsized":"2024-09-18T08:17:28.427787+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:12:54.661578+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_clean_scrub_stamp":"2024-09-18T08:12:54.661578+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.16","version":"271'15","reported_seq":715,"reported_epoch":380,"state":"active+clean","last_fresh":"2024-09-18T08:17:29.478184+0000","last_change":"2024-09-18T08:14:05.588589+0000","last_active":"2024-09-18T08:17:29.478184+0000","last_peered":"2024-09-18T08:17:29.478184+0000","last_clean":"2024-09-18T08:17:29.478184+0000","last_became_active":"2024-09-18T08:14:05.578519+0000","last_became_peered":"2024-09-18T08:14:05.578519+0000","last_unstale":"2024-09-18T08:17:29.478184+0000","last_undegraded":"2024-09-18T08:17:29.478184+0000","last_fullsized":"2024-09-18T08:17:29.478184+0000","mapping_epoch":292,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":293,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'9","last_scrub_stamp":"2024-09-18T08:09:44.116308+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_clean_scrub_stamp":"2024-09-18T08:09:44.116308+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":43,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":324,"reported_epoch":380,"state":"active+clean","last_fresh":"2024-09-18T08:17:29.471235+0000","last_change":"2024-09-18T08:16:13.460704+0000","last_active":"2024-09-18T08:17:29.471235+0000","last_peered":"2024-09-18T08:17:29.471235+0000","last_clean":"2024-09-18T08:17:29.471235+0000","last_became_active":"2024-09-18T08:16:13.460000+0000","last_became_peered":"2024-09-18T08:16:13.460000+0000","last_unstale":"2024-09-18T08:17:29.471235+0000","last_undegraded":"2024-09-18T08:17:29.471235+0000","last_fullsized":"2024-09-18T08:17:29.471235+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:12:03.433407+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:46.860092+0000","last_clean_scrub_stamp":"2024-09-18T08:12:03.433407+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,3],"acting":[0,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":692,"reported_epoch":379,"state":"active+clean","last_fresh":"2024-09-18T08:17:28.462440+0000","last_change":"2024-09-18T08:17:06.569916+0000","last_active":"2024-09-18T08:17:28.462440+0000","last_peered":"2024-09-18T08:17:28.462440+0000","last_clean":"2024-09-18T08:17:28.462440+0000","last_became_active":"2024-09-18T08:14:40.356531+0000","last_became_peered":"2024-09-18T08:14:40.356531+0000","last_unstale":"2024-09-18T08:17:28.462440+0000","last_undegraded":"2024-09-18T08:17:28.462440+0000","last_fullsized":"2024-09-18T08:17:28.462440+0000","mapping_epoch":308,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":309,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:17:06.569828+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:50.886830+0000","last_clean_scrub_stamp":"2024-09-18T08:17:06.569828+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:18:35.645500+0000","scrub_duration":31,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":701,"reported_epoch":379,"state":"active+clean","last_fresh":"2024-09-18T08:17:28.435476+0000","last_change":"2024-09-18T08:16:49.005144+0000","last_active":"2024-09-18T08:17:28.435476+0000","last_peered":"2024-09-18T08:17:28.435476+0000","last_clean":"2024-09-18T08:17:28.435476+0000","last_became_active":"2024-09-18T08:16:13.466171+0000","last_became_peered":"2024-09-18T08:16:13.466171+0000","last_unstale":"2024-09-18T08:17:28.435476+0000","last_undegraded":"2024-09-18T08:17:28.435476+0000","last_fullsized":"2024-09-18T08:17:28.435476+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:16:49.005064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_clean_scrub_stamp":"2024-09-18T08:16:49.005064+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:18:17.185217+0000","scrub_duration":22,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.15","version":"271'15","reported_seq":106,"reported_epoch":379,"state":"active+clean","last_fresh":"2024-09-18T08:17:28.427657+0000","last_change":"2024-09-18T08:15:41.485598+0000","last_active":"2024-09-18T08:17:28.427657+0000","last_peered":"2024-09-18T08:17:28.427657+0000","last_clean":"2024-09-18T08:17:28.427657+0000","last_became_active":"2024-09-18T08:15:41.447339+0000","last_became_peered":"2024-09-18T08:15:41.447339+0000","last_unstale":"2024-09-18T08:17:28.427657+0000","last_undegraded":"2024-09-18T08:17:28.427657+0000","last_fullsized":"2024-09-18T08:17:28.427657+0000","mapping_epoch":336,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":337,"parent":"0.0","parent_split_bits":0,"last_scrub":"271'15","last_scrub_stamp":"2024-09-18T08:12:58.225295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_clean_scrub_stamp":"2024-09-18T08:12:58.225295+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":25,"num_read_kb":19,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":705,"reported_epoch":380,"state":"active+clean","last_fresh":"2024-09-18T08:17:29.472522+0000","last_change":"2024-09-18T08:14:40.357339+0000","last_active":"2024-09-18T08:17:29.472522+0000","last_peered":"2024-09-18T08:17:29.472522+0000","last_clean":"2024-09-18T08:17:29.472522+0000","last_became_active":"2024-09-18T08:14:40.355709+0000","last_became_peered":"2024-09-18T08:14:40.355709+0000","last_unstale":"2024-09-18T08:17:29.472522+0000","last_undegraded":"2024-09-18T08:17:29.472522+0000","last_fullsized":"2024-09-18T08:17:29.472522+0000","mapping_epoch":308,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":309,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:14:28.097629+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:46.860092+0000","last_clean_scrub_stamp":"2024-09-18T08:14:28.097629+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1],"acting":[0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.14","version":"310'16","reported_seq":912,"reported_epoch":379,"state":"active+clean","last_fresh":"2024-09-18T08:17:28.469549+0000","last_change":"2024-09-18T08:14:40.366956+0000","last_active":"2024-09-18T08:17:28.469549+0000","last_peered":"2024-09-18T08:17:28.469549+0000","last_clean":"2024-09-18T08:17:28.469549+0000","last_became_active":"2024-09-18T08:14:40.358971+0000","last_became_peered":"2024-09-18T08:14:40.358971+0000","last_unstale":"2024-09-18T08:17:28.469549+0000","last_undegraded":"2024-09-18T08:17:28.469549+0000","last_fullsized":"2024-09-18T08:17:28.469549+0000","mapping_epoch":308,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":309,"parent":"0.0","parent_split_bits":0,"last_scrub":"199'12","last_scrub_stamp":"2024-09-18T08:13:31.695354+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:50.886830+0000","last_clean_scrub_stamp":"2024-09-18T08:13:31.695354+0000","objects_scrubbed":1,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,1],"acting":[3,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":418,"reported_epoch":379,"state":"active+clean","last_fresh":"2024-09-18T08:17:28.437126+0000","last_change":"2024-09-18T08:16:51.923862+0000","last_active":"2024-09-18T08:17:28.437126+0000","last_peered":"2024-09-18T08:17:28.437126+0000","last_clean":"2024-09-18T08:17:28.437126+0000","last_became_active":"2024-09-18T08:16:13.467534+0000","last_became_peered":"2024-09-18T08:16:13.467534+0000","last_unstale":"2024-09-18T08:17:28.437126+0000","last_undegraded":"2024-09-18T08:17:28.437126+0000","last_fullsized":"2024-09-18T08:17:28.437126+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:16:51.923777+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_clean_scrub_stamp":"2024-09-18T08:16:51.923777+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:17:59.666455+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,0],"acting":[2,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.14","version":"271'20","reported_seq":753,"reported_epoch":379,"state":"active+clean","last_fresh":"2024-09-18T08:17:28.459386+0000","last_change":"2024-09-18T08:15:52.819894+0000","last_active":"2024-09-18T08:17:28.459386+0000","last_peered":"2024-09-18T08:17:28.459386+0000","last_clean":"2024-09-18T08:17:28.459386+0000","last_became_active":"2024-09-18T08:14:40.358983+0000","last_became_peered":"2024-09-18T08:14:40.358983+0000","last_unstale":"2024-09-18T08:17:28.459386+0000","last_undegraded":"2024-09-18T08:17:28.459386+0000","last_fullsized":"2024-09-18T08:17:28.459386+0000","mapping_epoch":308,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":309,"parent":"0.0","parent_split_bits":0,"last_scrub":"271'20","last_scrub_stamp":"2024-09-18T08:15:52.819801+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_clean_scrub_stamp":"2024-09-18T08:15:52.819801+0000","objects_scrubbed":5,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":16,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":25,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":348,"reported_epoch":379,"state":"active+clean","last_fresh":"2024-09-18T08:17:28.437557+0000","last_change":"2024-09-18T08:16:13.461715+0000","last_active":"2024-09-18T08:17:28.437557+0000","last_peered":"2024-09-18T08:17:28.437557+0000","last_clean":"2024-09-18T08:17:28.437557+0000","last_became_active":"2024-09-18T08:16:13.461199+0000","last_became_peered":"2024-09-18T08:16:13.461199+0000","last_unstale":"2024-09-18T08:17:28.437557+0000","last_undegraded":"2024-09-18T08:17:28.437557+0000","last_fullsized":"2024-09-18T08:17:28.437557+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:13:32.933940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:46.860092+0000","last_clean_scrub_stamp":"2024-09-18T08:13:32.933940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":21,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,0],"acting":[2,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.15","version":"338'12","reported_seq":122,"reported_epoch":379,"state":"active+clean","last_fresh":"2024-09-18T08:17:28.427403+0000","last_change":"2024-09-18T08:15:41.489414+0000","last_active":"2024-09-18T08:17:28.427403+0000","last_peered":"2024-09-18T08:17:28.427403+0000","last_clean":"2024-09-18T08:17:28.427403+0000","last_became_active":"2024-09-18T08:15:41.455243+0000","last_became_peered":"2024-09-18T08:15:41.455243+0000","last_unstale":"2024-09-18T08:17:28.427403+0000","last_undegraded":"2024-09-18T08:17:28.427403+0000","last_fullsized":"2024-09-18T08:17:28.427403+0000","mapping_epoch":336,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":337,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'8","last_scrub_stamp":"2024-09-18T08:10:54.148986+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:50.886830+0000","last_clean_scrub_stamp":"2024-09-18T08:10:54.148986+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:18:23.530255+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":680,"reported_epoch":380,"state":"active+clean","last_fresh":"2024-09-18T08:17:29.476926+0000","last_change":"2024-09-18T08:16:13.472592+0000","last_active":"2024-09-18T08:17:29.476926+0000","last_peered":"2024-09-18T08:17:29.476926+0000","last_clean":"2024-09-18T08:17:29.476926+0000","last_became_active":"2024-09-18T08:16:13.472297+0000","last_became_peered":"2024-09-18T08:16:13.472297+0000","last_unstale":"2024-09-18T08:17:29.476926+0000","last_undegraded":"2024-09-18T08:17:29.476926+0000","last_fullsized":"2024-09-18T08:17:29.476926+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:09:42.047270+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_clean_scrub_stamp":"2024-09-18T08:09:42.047270+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,5],"acting":[0,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.13","version":"271'19","reported_seq":387,"reported_epoch":380,"state":"active+clean","last_fresh":"2024-09-18T08:17:29.462103+0000","last_change":"2024-09-18T08:16:13.494619+0000","last_active":"2024-09-18T08:17:29.462103+0000","last_peered":"2024-09-18T08:17:29.462103+0000","last_clean":"2024-09-18T08:17:29.462103+0000","last_became_active":"2024-09-18T08:16:13.454640+0000","last_became_peered":"2024-09-18T08:16:13.454640+0000","last_unstale":"2024-09-18T08:17:29.462103+0000","last_undegraded":"2024-09-18T08:17:29.462103+0000","last_fullsized":"2024-09-18T08:17:29.462103+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'11","last_scrub_stamp":"2024-09-18T08:11:22.565415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_clean_scrub_stamp":"2024-09-18T08:11:22.565415+0000","objects_scrubbed":5,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":31,"num_read_kb":24,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,5],"acting":[1,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":322,"reported_epoch":379,"state":"active+clean","last_fresh":"2024-09-18T08:17:28.435120+0000","last_change":"2024-09-18T08:16:48.007441+0000","last_active":"2024-09-18T08:17:28.435120+0000","last_peered":"2024-09-18T08:17:28.435120+0000","last_clean":"2024-09-18T08:17:28.435120+0000","last_became_active":"2024-09-18T08:16:13.457932+0000","last_became_peered":"2024-09-18T08:16:13.457932+0000","last_unstale":"2024-09-18T08:17:28.435120+0000","last_undegraded":"2024-09-18T08:17:28.435120+0000","last_fullsized":"2024-09-18T08:17:28.435120+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:16:48.007357+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:46.860092+0000","last_clean_scrub_stamp":"2024-09-18T08:16:48.007357+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:18:00.038865+0000","scrub_duration":27,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,0],"acting":[2,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":572,"reported_epoch":380,"state":"active+clean","last_fresh":"2024-09-18T08:17:29.460350+0000","last_change":"2024-09-18T08:15:55.335233+0000","last_active":"2024-09-18T08:17:29.460350+0000","last_peered":"2024-09-18T08:17:29.460350+0000","last_clean":"2024-09-18T08:17:29.460350+0000","last_became_active":"2024-09-18T08:15:41.438048+0000","last_became_peered":"2024-09-18T08:15:41.438048+0000","last_unstale":"2024-09-18T08:17:29.460350+0000","last_undegraded":"2024-09-18T08:17:29.460350+0000","last_fullsized":"2024-09-18T08:17:29.460350+0000","mapping_epoch":336,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":337,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:15:55.334983+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:50.886830+0000","last_clean_scrub_stamp":"2024-09-18T08:15:55.334983+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":695,"reported_epoch":379,"state":"active+clean","last_fresh":"2024-09-18T08:17:28.457642+0000","last_change":"2024-09-18T08:15:41.466178+0000","last_active":"2024-09-18T08:17:28.457642+0000","last_peered":"2024-09-18T08:17:28.457642+0000","last_clean":"2024-09-18T08:17:28.457642+0000","last_became_active":"2024-09-18T08:15:41.464961+0000","last_became_peered":"2024-09-18T08:15:41.464961+0000","last_unstale":"2024-09-18T08:17:28.457642+0000","last_undegraded":"2024-09-18T08:17:28.457642+0000","last_fullsized":"2024-09-18T08:17:28.457642+0000","mapping_epoch":336,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":337,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:13:28.680237+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_clean_scrub_stamp":"2024-09-18T08:13:28.680237+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.12","version":"271'15","reported_seq":560,"reported_epoch":380,"state":"active+clean","last_fresh":"2024-09-18T08:17:29.460162+0000","last_change":"2024-09-18T08:15:52.304936+0000","last_active":"2024-09-18T08:17:29.460162+0000","last_peered":"2024-09-18T08:17:29.460162+0000","last_clean":"2024-09-18T08:17:29.460162+0000","last_became_active":"2024-09-18T08:14:39.416284+0000","last_became_peered":"2024-09-18T08:14:39.416284+0000","last_unstale":"2024-09-18T08:17:29.460162+0000","last_undegraded":"2024-09-18T08:17:29.460162+0000","last_fullsized":"2024-09-18T08:17:29.460162+0000","mapping_epoch":307,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":308,"parent":"0.0","parent_split_bits":0,"last_scrub":"271'15","last_scrub_stamp":"2024-09-18T08:15:52.304832+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_clean_scrub_stamp":"2024-09-18T08:15:52.304832+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":25,"num_read_kb":19,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":489,"reported_epoch":379,"state":"active+clean","last_fresh":"2024-09-18T08:17:28.466118+0000","last_change":"2024-09-18T08:16:13.476132+0000","last_active":"2024-09-18T08:17:28.466118+0000","last_peered":"2024-09-18T08:17:28.466118+0000","last_clean":"2024-09-18T08:17:28.466118+0000","last_became_active":"2024-09-18T08:16:13.475647+0000","last_became_peered":"2024-09-18T08:16:13.475647+0000","last_unstale":"2024-09-18T08:17:28.466118+0000","last_undegraded":"2024-09-18T08:17:28.466118+0000","last_fullsized":"2024-09-18T08:17:28.466118+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:15:58.978411+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:46.860092+0000","last_clean_scrub_stamp":"2024-09-18T08:15:58.978411+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":18,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":679,"reported_epoch":379,"state":"active+clean","last_fresh":"2024-09-18T08:17:28.459463+0000","last_change":"2024-09-18T08:14:39.403741+0000","last_active":"2024-09-18T08:17:28.459463+0000","last_peered":"2024-09-18T08:17:28.459463+0000","last_clean":"2024-09-18T08:17:28.459463+0000","last_became_active":"2024-09-18T08:14:39.402406+0000","last_became_peered":"2024-09-18T08:14:39.402406+0000","last_unstale":"2024-09-18T08:17:28.459463+0000","last_undegraded":"2024-09-18T08:17:28.459463+0000","last_fullsized":"2024-09-18T08:17:28.459463+0000","mapping_epoch":307,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":308,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:11:30.302179+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:50.886830+0000","last_clean_scrub_stamp":"2024-09-18T08:11:30.302179+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:18:20.349750+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.10","version":"69'1","reported_seq":695,"reported_epoch":380,"state":"active+clean","last_fresh":"2024-09-18T08:17:29.472834+0000","last_change":"2024-09-18T08:15:41.480106+0000","last_active":"2024-09-18T08:17:29.472834+0000","last_peered":"2024-09-18T08:17:29.472834+0000","last_clean":"2024-09-18T08:17:29.472834+0000","last_became_active":"2024-09-18T08:15:41.444085+0000","last_became_peered":"2024-09-18T08:15:41.444085+0000","last_unstale":"2024-09-18T08:17:29.472834+0000","last_undegraded":"2024-09-18T08:17:29.472834+0000","last_fullsized":"2024-09-18T08:17:29.472834+0000","mapping_epoch":336,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":337,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'1","last_scrub_stamp":"2024-09-18T08:15:00.219678+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_clean_scrub_stamp":"2024-09-18T08:15:00.219678+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":26,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.11","version":"271'19","reported_seq":822,"reported_epoch":379,"state":"active+clean","last_fresh":"2024-09-18T08:17:28.457397+0000","last_change":"2024-09-18T08:15:41.491797+0000","last_active":"2024-09-18T08:17:28.457397+0000","last_peered":"2024-09-18T08:17:28.457397+0000","last_clean":"2024-09-18T08:17:28.457397+0000","last_became_active":"2024-09-18T08:15:41.465631+0000","last_became_peered":"2024-09-18T08:15:41.465631+0000","last_unstale":"2024-09-18T08:17:28.457397+0000","last_undegraded":"2024-09-18T08:17:28.457397+0000","last_fullsized":"2024-09-18T08:17:28.457397+0000","mapping_epoch":336,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":337,"parent":"0.0","parent_split_bits":0,"last_scrub":"271'19","last_scrub_stamp":"2024-09-18T08:14:38.809490+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_clean_scrub_stamp":"2024-09-18T08:14:38.809490+0000","objects_scrubbed":5,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":32,"num_read_kb":25,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":550,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":96,"reported_epoch":379,"state":"active+clean","last_fresh":"2024-09-18T08:17:28.429645+0000","last_change":"2024-09-18T08:15:41.449283+0000","last_active":"2024-09-18T08:17:28.429645+0000","last_peered":"2024-09-18T08:17:28.429645+0000","last_clean":"2024-09-18T08:17:28.429645+0000","last_became_active":"2024-09-18T08:15:41.445583+0000","last_became_peered":"2024-09-18T08:15:41.445583+0000","last_unstale":"2024-09-18T08:17:28.429645+0000","last_undegraded":"2024-09-18T08:17:28.429645+0000","last_fullsized":"2024-09-18T08:17:28.429645+0000","mapping_epoch":336,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":337,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:10:53.145315+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:46.860092+0000","last_clean_scrub_stamp":"2024-09-18T08:10:53.145315+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:18:14.386231+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":364,"reported_epoch":380,"state":"active+clean","last_fresh":"2024-09-18T08:17:29.472754+0000","last_change":"2024-09-18T08:16:13.444745+0000","last_active":"2024-09-18T08:17:29.472754+0000","last_peered":"2024-09-18T08:17:29.472754+0000","last_clean":"2024-09-18T08:17:29.472754+0000","last_became_active":"2024-09-18T08:16:13.443940+0000","last_became_peered":"2024-09-18T08:16:13.443940+0000","last_unstale":"2024-09-18T08:17:29.472754+0000","last_undegraded":"2024-09-18T08:17:29.472754+0000","last_fullsized":"2024-09-18T08:17:29.472754+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:14:37.743856+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:50.886830+0000","last_clean_scrub_stamp":"2024-09-18T08:14:37.743856+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":678,"reported_epoch":379,"state":"active+clean","last_fresh":"2024-09-18T08:17:28.460422+0000","last_change":"2024-09-18T08:11:29.344444+0000","last_active":"2024-09-18T08:17:28.460422+0000","last_peered":"2024-09-18T08:17:28.460422+0000","last_clean":"2024-09-18T08:17:28.460422+0000","last_became_active":"2024-09-18T08:06:24.128582+0000","last_became_peered":"2024-09-18T08:06:24.128582+0000","last_unstale":"2024-09-18T08:17:28.460422+0000","last_undegraded":"2024-09-18T08:17:28.460422+0000","last_fullsized":"2024-09-18T08:17:28.460422+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:11:29.344382+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_clean_scrub_stamp":"2024-09-18T08:11:29.344382+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:18:19.388633+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.10","version":"271'8","reported_seq":736,"reported_epoch":379,"state":"active+clean","last_fresh":"2024-09-18T08:17:28.456422+0000","last_change":"2024-09-18T08:15:50.790606+0000","last_active":"2024-09-18T08:17:28.456422+0000","last_peered":"2024-09-18T08:17:28.456422+0000","last_clean":"2024-09-18T08:17:28.456422+0000","last_became_active":"2024-09-18T08:14:39.404257+0000","last_became_peered":"2024-09-18T08:14:39.404257+0000","last_unstale":"2024-09-18T08:17:28.456422+0000","last_undegraded":"2024-09-18T08:17:28.456422+0000","last_fullsized":"2024-09-18T08:17:28.456422+0000","mapping_epoch":307,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":308,"parent":"0.0","parent_split_bits":0,"last_scrub":"271'8","last_scrub_stamp":"2024-09-18T08:15:50.790525+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_clean_scrub_stamp":"2024-09-18T08:15:50.790525+0000","objects_scrubbed":2,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":12,"num_read_kb":10,"num_write":8,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":707,"reported_epoch":380,"state":"active+clean","last_fresh":"2024-09-18T08:17:29.477918+0000","last_change":"2024-09-18T08:15:41.467281+0000","last_active":"2024-09-18T08:17:29.477918+0000","last_peered":"2024-09-18T08:17:29.477918+0000","last_clean":"2024-09-18T08:17:29.477918+0000","last_became_active":"2024-09-18T08:15:41.466964+0000","last_became_peered":"2024-09-18T08:15:41.466964+0000","last_unstale":"2024-09-18T08:17:29.477918+0000","last_undegraded":"2024-09-18T08:17:29.477918+0000","last_fullsized":"2024-09-18T08:17:29.477918+0000","mapping_epoch":336,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":337,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:14:33.064199+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:46.860092+0000","last_clean_scrub_stamp":"2024-09-18T08:14:33.064199+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":324,"reported_epoch":380,"state":"active+clean","last_fresh":"2024-09-18T08:17:29.471029+0000","last_change":"2024-09-18T08:16:13.453084+0000","last_active":"2024-09-18T08:17:29.471029+0000","last_peered":"2024-09-18T08:17:29.471029+0000","last_clean":"2024-09-18T08:17:29.471029+0000","last_became_active":"2024-09-18T08:16:13.452169+0000","last_became_peered":"2024-09-18T08:16:13.452169+0000","last_unstale":"2024-09-18T08:17:29.471029+0000","last_undegraded":"2024-09-18T08:17:29.471029+0000","last_fullsized":"2024-09-18T08:17:29.471029+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:13:32.640669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:50.886830+0000","last_clean_scrub_stamp":"2024-09-18T08:13:32.640669+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":20,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,1],"acting":[0,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":486,"reported_epoch":379,"state":"active+clean","last_fresh":"2024-09-18T08:17:28.434664+0000","last_change":"2024-09-18T08:16:13.492104+0000","last_active":"2024-09-18T08:17:28.434664+0000","last_peered":"2024-09-18T08:17:28.434664+0000","last_clean":"2024-09-18T08:17:28.434664+0000","last_became_active":"2024-09-18T08:16:13.491755+0000","last_became_peered":"2024-09-18T08:16:13.491755+0000","last_unstale":"2024-09-18T08:17:28.434664+0000","last_undegraded":"2024-09-18T08:17:28.434664+0000","last_fullsized":"2024-09-18T08:17:28.434664+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:13:56.525167+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_clean_scrub_stamp":"2024-09-18T08:13:56.525167+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,1],"acting":[2,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":544,"reported_epoch":380,"state":"active+clean","last_fresh":"2024-09-18T08:17:29.463053+0000","last_change":"2024-09-18T08:16:13.468068+0000","last_active":"2024-09-18T08:17:29.463053+0000","last_peered":"2024-09-18T08:17:29.463053+0000","last_clean":"2024-09-18T08:17:29.463053+0000","last_became_active":"2024-09-18T08:16:13.466812+0000","last_became_peered":"2024-09-18T08:16:13.466812+0000","last_unstale":"2024-09-18T08:17:29.463053+0000","last_undegraded":"2024-09-18T08:17:29.463053+0000","last_fullsized":"2024-09-18T08:17:29.463053+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:15:45.297240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_clean_scrub_stamp":"2024-09-18T08:15:45.297240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,2],"acting":[1,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":687,"reported_epoch":379,"state":"active+clean","last_fresh":"2024-09-18T08:17:28.455355+0000","last_change":"2024-09-18T08:15:43.836627+0000","last_active":"2024-09-18T08:17:28.455355+0000","last_peered":"2024-09-18T08:17:28.455355+0000","last_clean":"2024-09-18T08:17:28.455355+0000","last_became_active":"2024-09-18T08:14:39.405347+0000","last_became_peered":"2024-09-18T08:14:39.405347+0000","last_unstale":"2024-09-18T08:17:28.455355+0000","last_undegraded":"2024-09-18T08:17:28.455355+0000","last_fullsized":"2024-09-18T08:17:28.455355+0000","mapping_epoch":307,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":308,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:15:43.836549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:46.860092+0000","last_clean_scrub_stamp":"2024-09-18T08:15:43.836549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":20,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"271'19","reported_seq":108,"reported_epoch":379,"state":"active+clean","last_fresh":"2024-09-18T08:17:28.427989+0000","last_change":"2024-09-18T08:15:41.472787+0000","last_active":"2024-09-18T08:17:28.427989+0000","last_peered":"2024-09-18T08:17:28.427989+0000","last_clean":"2024-09-18T08:17:28.427989+0000","last_became_active":"2024-09-18T08:15:41.445198+0000","last_became_peered":"2024-09-18T08:15:41.445198+0000","last_unstale":"2024-09-18T08:17:28.427989+0000","last_undegraded":"2024-09-18T08:17:28.427989+0000","last_fullsized":"2024-09-18T08:17:28.427989+0000","mapping_epoch":336,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":337,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'11","last_scrub_stamp":"2024-09-18T08:10:50.208543+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_clean_scrub_stamp":"2024-09-18T08:10:50.208543+0000","objects_scrubbed":5,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:18:15.381752+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":31,"num_read_kb":24,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,1],"acting":[5,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":704,"reported_epoch":380,"state":"active+clean","last_fresh":"2024-09-18T08:17:29.477091+0000","last_change":"2024-09-18T08:14:40.358776+0000","last_active":"2024-09-18T08:17:29.477091+0000","last_peered":"2024-09-18T08:17:29.477091+0000","last_clean":"2024-09-18T08:17:29.477091+0000","last_became_active":"2024-09-18T08:14:40.358457+0000","last_became_peered":"2024-09-18T08:14:40.358457+0000","last_unstale":"2024-09-18T08:17:29.477091+0000","last_undegraded":"2024-09-18T08:17:29.477091+0000","last_fullsized":"2024-09-18T08:17:29.477091+0000","mapping_epoch":308,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":309,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:12:31.664456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:50.886830+0000","last_clean_scrub_stamp":"2024-09-18T08:12:31.664456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":10,"num_pg":16,"stat_sum":{"num_bytes":2924871704,"num_objects":44631,"num_object_clones":0,"num_object_copies":133893,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":5347,"num_objects_unfound":0,"num_objects_dirty":44631,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":158805,"num_write_kb":6509889,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":59979,"num_bytes_recovered":3930718232,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":10679631872,"data_stored":10679615584,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":61773,"ondisk_log_size":61773,"up":48,"acting":48,"num_store_stats":7},{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":416,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":832,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":26,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":116,"ondisk_log_size":116,"up":96,"acting":96,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":178,"num_object_clones":0,"num_object_copies":534,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":178,"num_whiteouts":0,"num_read":1104,"num_read_kb":861,"num_write":703,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":447,"num_bytes_recovered":8338,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":499712,"data_stored":13350,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3639,"internal_metadata":0},"log_size":688,"ondisk_log_size":688,"up":96,"acting":96,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":1613,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":14,"num_read_kb":14,"num_write":10,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":4563,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":102400,"data_stored":6108,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":6,"ondisk_log_size":6,"up":96,"acting":96,"num_store_stats":8},{"poolid":2,"num_pg":3,"stat_sum":{"num_bytes":408,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":345,"num_read_kb":340,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":408,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1613,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":8,"ondisk_log_size":8,"up":9,"acting":9,"num_store_stats":7},{"poolid":1,"num_pg":11,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1245728,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":869,"ondisk_log_size":869,"up":33,"acting":33,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":0,"seq":240518168756,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":47,"seq":201863463099,"num_pgs":74,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1801516,"kb_used_data":1323292,"kb_used_omap":1,"kb_used_meta":478206,"kb_avail":91943636,"statfs":{"total":95995035648,"available":94150283264,"internally_reserved":0,"allocated":1355051008,"data_stored":1354631829,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":489683398},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":39,"seq":167503724736,"num_pgs":83,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1879476,"kb_used_data":1118048,"kb_used_omap":1,"kb_used_meta":761406,"kb_avail":91865676,"statfs":{"total":95995035648,"available":94070452224,"internally_reserved":0,"allocated":1144881152,"data_stored":1144453115,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":779680198},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,1,3,1],"upper_bound":32},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":0,"seq":137438953670,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":26,"seq":111669149900,"num_pgs":87,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2165000,"kb_used_data":1447992,"kb_used_omap":1,"kb_used_meta":716990,"kb_avail":91580152,"statfs":{"total":95995035648,"available":93778075648,"internally_reserved":0,"allocated":1482743808,"data_stored":1482291117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":734198216},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,3,1],"upper_bound":32},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411538,"num_pgs":65,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1919844,"kb_used_data":1293008,"kb_used_omap":1,"kb_used_meta":626750,"kb_avail":91825308,"statfs":{"total":95995035648,"available":94029115392,"internally_reserved":0,"allocated":1324040192,"data_stored":1323627514,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":641792453},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,1,1],"upper_bound":64},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":307,"seq":1318554959908,"num_pgs":84,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1178212,"kb_used_data":1027860,"kb_used_omap":7,"kb_used_meta":150328,"kb_avail":92566940,"statfs":{"total":95995035648,"available":94788546560,"internally_reserved":0,"allocated":1052528640,"data_stored":1052054816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":8079,"internal_metadata":153935985},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,3,0,1],"upper_bound":32},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738589,"num_pgs":81,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2607352,"kb_used_data":1990116,"kb_used_omap":1,"kb_used_meta":617022,"kb_avail":91137800,"statfs":{"total":95995035648,"available":93325107200,"internally_reserved":0,"allocated":2037878784,"data_stored":2037452661,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":631830981},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,5,2,3,0,2],"upper_bound":64},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":266240,"data_stored":262688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1131,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1613,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":436,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":46,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1177,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1567,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":46,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1474,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":94208,"data_stored":2448,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3639,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":1568,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":94208,"data_stored":2492,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":1584,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1430,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":1694,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":660,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":403,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":403,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":416,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":2020806656,"data_stored":2020802584,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":1028915200,"data_stored":1028915200,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":1318322176,"data_stored":1318322176,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":1481117696,"data_stored":1481113624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":2360020992,"data_stored":2360016920,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":1139605504,"data_stored":1139605504,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":1330843648,"data_stored":1330839576,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T08:17:41.464 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid 779349ca-7593-11ef-bceb-c7b262605968 -- ceph pg dump --format=json 2024-09-18T08:17:41.469 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:17:41.469 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:17:41.470 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3' 2024-09-18T08:17:41.491 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/779349ca-7593-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T08:17:41.570 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0 2024-09-18T08:17:41.622 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:17:41.623 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:17:41.623 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0' 2024-09-18T08:17:41.724 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1 2024-09-18T08:17:41.731 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:17:41 smithi007 ceph-mon[29459]: 4.13 scrub starts 2024-09-18T08:17:41.731 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:17:41 smithi007 ceph-mon[29459]: 4.13 scrub ok 2024-09-18T08:17:41.731 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:17:41 smithi007 ceph-mon[29459]: 6.1a scrub starts 2024-09-18T08:17:41.731 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:17:41 smithi007 ceph-mon[29459]: 6.1a scrub ok 2024-09-18T08:17:41.731 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:17:41 smithi007 ceph-mon[29459]: 5.8 scrub starts 2024-09-18T08:17:41.731 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:17:41 smithi007 ceph-mon[29459]: 5.8 scrub ok 2024-09-18T08:17:41.731 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:17:41 smithi007 ceph-mon[39249]: 4.13 scrub starts 2024-09-18T08:17:41.731 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:17:41 smithi007 ceph-mon[39249]: 4.13 scrub ok 2024-09-18T08:17:41.731 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:17:41 smithi007 ceph-mon[39249]: 6.1a scrub starts 2024-09-18T08:17:41.732 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:17:41 smithi007 ceph-mon[39249]: 6.1a scrub ok 2024-09-18T08:17:41.732 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:17:41 smithi007 ceph-mon[39249]: 5.8 scrub starts 2024-09-18T08:17:41.732 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:17:41 smithi007 ceph-mon[39249]: 5.8 scrub ok 2024-09-18T08:17:41.789 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:17:41.790 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:17:41.790 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1' 2024-09-18T08:17:41.891 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1 2024-09-18T08:17:41.905 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:17:41 smithi050 ceph-mon[32371]: 4.13 scrub starts 2024-09-18T08:17:41.905 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:17:41 smithi050 ceph-mon[32371]: 4.13 scrub ok 2024-09-18T08:17:41.905 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:17:41 smithi050 ceph-mon[32371]: 6.1a scrub starts 2024-09-18T08:17:41.905 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:17:41 smithi050 ceph-mon[32371]: 6.1a scrub ok 2024-09-18T08:17:41.905 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:17:41 smithi050 ceph-mon[32371]: 5.8 scrub starts 2024-09-18T08:17:41.905 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:17:41 smithi050 ceph-mon[32371]: 5.8 scrub ok 2024-09-18T08:17:41.932 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:17:41.933 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:17:41.933 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1' 2024-09-18T08:17:42.034 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7 2024-09-18T08:17:42.059 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:17:42.060 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:17:42.060 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7' 2024-09-18T08:17:42.161 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6 2024-09-18T08:17:42.187 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:17:42.188 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:17:42.188 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6' 2024-09-18T08:17:42.289 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1 2024-09-18T08:17:42.333 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:17:42.334 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:17:42.334 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1' 2024-09-18T08:17:42.416 INFO:teuthology.orchestra.run.smithi007.stdout:osd.0: {} 2024-09-18T08:17:42.416 INFO:teuthology.orchestra.run.smithi007.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-18T08:17:42.435 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2 2024-09-18T08:17:42.456 INFO:teuthology.orchestra.run.smithi007.stdout:osd.1: {} 2024-09-18T08:17:42.456 INFO:teuthology.orchestra.run.smithi007.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-18T08:17:42.478 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:17:42.478 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:17:42.479 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2' 2024-09-18T08:17:42.501 INFO:teuthology.orchestra.run.smithi007.stdout:osd.2: {} 2024-09-18T08:17:42.502 INFO:teuthology.orchestra.run.smithi007.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-18T08:17:42.569 INFO:teuthology.orchestra.run.smithi007.stdout:osd.3: {} 2024-09-18T08:17:42.569 INFO:teuthology.orchestra.run.smithi007.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-18T08:17:42.580 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7 2024-09-18T08:17:42.605 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:17:42.605 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:17:42.606 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7' 2024-09-18T08:17:42.616 INFO:teuthology.orchestra.run.smithi007.stdout:osd.4: {} 2024-09-18T08:17:42.616 INFO:teuthology.orchestra.run.smithi007.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-18T08:17:42.623 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/779349ca-7593-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T08:17:42.660 INFO:teuthology.orchestra.run.smithi007.stdout:osd.5: {} 2024-09-18T08:17:42.661 INFO:teuthology.orchestra.run.smithi007.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-18T08:17:42.662 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/779349ca-7593-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T08:17:42.699 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:17:42 smithi007 ceph-mon[29459]: pgmap v743: 158 pgs: 2 active+remapped+backfill_toofull, 156 active+clean; 2.1 GiB data, 9.4 GiB used, 527 GiB / 536 GiB avail; 1023 B/s rd, 0 B/s wr, 1.12k op/s; 4304/104151 objects misplaced (4.132%) 2024-09-18T08:17:42.699 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:17:42 smithi007 ceph-mon[29459]: 10.6 scrub starts 2024-09-18T08:17:42.699 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:17:42 smithi007 ceph-mon[29459]: 10.5 scrub starts 2024-09-18T08:17:42.699 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:17:42 smithi007 ceph-mon[29459]: from='client.? 172.21.15.7:0/1782636519' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-18T08:17:42.699 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:17:42 smithi007 ceph-mon[39249]: pgmap v743: 158 pgs: 2 active+remapped+backfill_toofull, 156 active+clean; 2.1 GiB data, 9.4 GiB used, 527 GiB / 536 GiB avail; 1023 B/s rd, 0 B/s wr, 1.12k op/s; 4304/104151 objects misplaced (4.132%) 2024-09-18T08:17:42.699 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:17:42 smithi007 ceph-mon[39249]: 10.6 scrub starts 2024-09-18T08:17:42.699 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:17:42 smithi007 ceph-mon[39249]: 10.5 scrub starts 2024-09-18T08:17:42.700 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:17:42 smithi007 ceph-mon[39249]: from='client.? 172.21.15.7:0/1782636519' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-18T08:17:42.700 INFO:teuthology.orchestra.run.smithi007.stdout:osd.6: {} 2024-09-18T08:17:42.700 INFO:teuthology.orchestra.run.smithi007.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-18T08:17:42.707 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5 2024-09-18T08:17:42.730 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:17:42.730 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:17:42.731 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5' 2024-09-18T08:17:42.739 INFO:teuthology.orchestra.run.smithi007.stdout:osd.7: {} 2024-09-18T08:17:42.740 INFO:teuthology.orchestra.run.smithi007.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-18T08:17:42.831 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7 2024-09-18T08:17:42.857 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:17:42.858 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:17:42.858 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7' 2024-09-18T08:17:42.905 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:17:42 smithi050 ceph-mon[32371]: pgmap v743: 158 pgs: 2 active+remapped+backfill_toofull, 156 active+clean; 2.1 GiB data, 9.4 GiB used, 527 GiB / 536 GiB avail; 1023 B/s rd, 0 B/s wr, 1.12k op/s; 4304/104151 objects misplaced (4.132%) 2024-09-18T08:17:42.906 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:17:42 smithi050 ceph-mon[32371]: 10.6 scrub starts 2024-09-18T08:17:42.906 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:17:42 smithi050 ceph-mon[32371]: 10.5 scrub starts 2024-09-18T08:17:42.906 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:17:42 smithi050 ceph-mon[32371]: from='client.? 172.21.15.7:0/1782636519' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-18T08:17:42.959 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4 2024-09-18T08:17:42.985 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:17:42.986 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:17:42.986 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4' 2024-09-18T08:17:43.087 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2 2024-09-18T08:17:43.128 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:17:43.128 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:17:43.128 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2' 2024-09-18T08:17:43.229 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0 2024-09-18T08:17:43.272 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:17:43.272 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:17:43.273 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0' 2024-09-18T08:17:43.374 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5 2024-09-18T08:17:43.399 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:17:43.400 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:17:43.400 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5' 2024-09-18T08:17:43.501 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2 2024-09-18T08:17:43.541 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:17:43.541 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:17:43.542 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2' 2024-09-18T08:17:43.547 INFO:teuthology.orchestra.run.smithi007.stderr:nodeep-scrub is unset 2024-09-18T08:17:43.643 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4 2024-09-18T08:17:43.670 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:17:43.670 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:17:43.671 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4' 2024-09-18T08:17:43.771 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0 2024-09-18T08:17:43.811 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:17:43.811 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:17:43.811 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0' 2024-09-18T08:17:43.905 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:17:43 smithi050 ceph-mon[32371]: 10.6 scrub ok 2024-09-18T08:17:43.905 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:17:43 smithi050 ceph-mon[32371]: purged_snaps scrub starts 2024-09-18T08:17:43.905 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:17:43 smithi050 ceph-mon[32371]: purged_snaps scrub ok 2024-09-18T08:17:43.905 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:17:43 smithi050 ceph-mon[32371]: 10.9 scrub starts 2024-09-18T08:17:43.905 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:17:43 smithi050 ceph-mon[32371]: 10.5 scrub ok 2024-09-18T08:17:43.905 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:17:43 smithi050 ceph-mon[32371]: from='client.? 172.21.15.7:0/2019403317' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-18T08:17:43.906 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:17:43 smithi050 ceph-mon[32371]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-18T08:17:43.912 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7 2024-09-18T08:17:43.939 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:17:43.939 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:17:43.940 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7' 2024-09-18T08:17:43.980 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:17:43 smithi007 ceph-mon[29459]: 10.6 scrub ok 2024-09-18T08:17:43.980 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:17:43 smithi007 ceph-mon[29459]: purged_snaps scrub starts 2024-09-18T08:17:43.980 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:17:43 smithi007 ceph-mon[29459]: purged_snaps scrub ok 2024-09-18T08:17:43.980 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:17:43 smithi007 ceph-mon[29459]: 10.9 scrub starts 2024-09-18T08:17:43.980 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:17:43 smithi007 ceph-mon[29459]: 10.5 scrub ok 2024-09-18T08:17:43.980 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:17:43 smithi007 ceph-mon[29459]: from='client.? 172.21.15.7:0/2019403317' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-18T08:17:43.980 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:17:43 smithi007 ceph-mon[29459]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-18T08:17:43.981 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:17:43 smithi007 ceph-mon[39249]: 10.6 scrub ok 2024-09-18T08:17:43.981 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:17:43 smithi007 ceph-mon[39249]: purged_snaps scrub starts 2024-09-18T08:17:43.981 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:17:43 smithi007 ceph-mon[39249]: purged_snaps scrub ok 2024-09-18T08:17:43.981 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:17:43 smithi007 ceph-mon[39249]: 10.9 scrub starts 2024-09-18T08:17:43.981 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:17:43 smithi007 ceph-mon[39249]: 10.5 scrub ok 2024-09-18T08:17:43.981 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:17:43 smithi007 ceph-mon[39249]: from='client.? 172.21.15.7:0/2019403317' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-18T08:17:43.981 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:17:43 smithi007 ceph-mon[39249]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-18T08:17:44.040 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6 2024-09-18T08:17:44.066 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:17:44.066 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:17:44.067 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6' 2024-09-18T08:17:44.167 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4 2024-09-18T08:17:44.192 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:17:44.193 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:17:44.193 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4' 2024-09-18T08:17:44.294 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7 2024-09-18T08:17:44.320 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:17:44.320 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:17:44.321 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7' 2024-09-18T08:17:44.421 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3 2024-09-18T08:17:44.467 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:17:44.467 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:17:44.468 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3' 2024-09-18T08:17:44.569 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2 2024-09-18T08:17:44.612 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:17:44.613 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:17:44.613 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2' 2024-09-18T08:17:44.714 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7 2024-09-18T08:17:44.730 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:17:44 smithi007 ceph-mon[29459]: 10.9 scrub ok 2024-09-18T08:17:44.730 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:17:44 smithi007 ceph-mon[29459]: pgmap v744: 158 pgs: 2 active+remapped+backfill_toofull, 156 active+clean; 2.0 GiB data, 9.1 GiB used, 527 GiB / 536 GiB avail; 1023 B/s rd, 0 B/s wr, 1.12k op/s; 3959/100878 objects misplaced (3.925%) 2024-09-18T08:17:44.730 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:17:44 smithi007 ceph-mon[29459]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-18T08:17:44.730 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:17:44 smithi007 ceph-mon[29459]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-18T08:17:44.731 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:17:44 smithi007 ceph-mon[29459]: osdmap e382: 8 total, 8 up, 6 in 2024-09-18T08:17:44.731 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:17:44 smithi007 ceph-mon[29459]: 1.8 scrub starts 2024-09-18T08:17:44.731 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:17:44 smithi007 ceph-mon[29459]: 1.8 scrub ok 2024-09-18T08:17:44.731 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:17:44 smithi007 ceph-mon[29459]: 1.9 scrub starts 2024-09-18T08:17:44.731 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:17:44 smithi007 ceph-mon[29459]: 1.9 scrub ok 2024-09-18T08:17:44.731 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:17:44 smithi007 ceph-mon[29459]: osdmap e383: 8 total, 8 up, 6 in 2024-09-18T08:17:44.731 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:17:44 smithi007 ceph-mon[39249]: 10.9 scrub ok 2024-09-18T08:17:44.731 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:17:44 smithi007 ceph-mon[39249]: pgmap v744: 158 pgs: 2 active+remapped+backfill_toofull, 156 active+clean; 2.0 GiB data, 9.1 GiB used, 527 GiB / 536 GiB avail; 1023 B/s rd, 0 B/s wr, 1.12k op/s; 3959/100878 objects misplaced (3.925%) 2024-09-18T08:17:44.731 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:17:44 smithi007 ceph-mon[39249]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-18T08:17:44.731 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:17:44 smithi007 ceph-mon[39249]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-18T08:17:44.731 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:17:44 smithi007 ceph-mon[39249]: osdmap e382: 8 total, 8 up, 6 in 2024-09-18T08:17:44.731 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:17:44 smithi007 ceph-mon[39249]: 1.8 scrub starts 2024-09-18T08:17:44.732 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:17:44 smithi007 ceph-mon[39249]: 1.8 scrub ok 2024-09-18T08:17:44.732 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:17:44 smithi007 ceph-mon[39249]: 1.9 scrub starts 2024-09-18T08:17:44.732 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:17:44 smithi007 ceph-mon[39249]: 1.9 scrub ok 2024-09-18T08:17:44.732 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:17:44 smithi007 ceph-mon[39249]: osdmap e383: 8 total, 8 up, 6 in 2024-09-18T08:17:44.739 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:17:44.740 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:17:44.740 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7' 2024-09-18T08:17:44.841 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5 2024-09-18T08:17:44.867 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:17:44.868 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:17:44.868 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5' 2024-09-18T08:17:44.905 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:17:44 smithi050 ceph-mon[32371]: 10.9 scrub ok 2024-09-18T08:17:44.905 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:17:44 smithi050 ceph-mon[32371]: pgmap v744: 158 pgs: 2 active+remapped+backfill_toofull, 156 active+clean; 2.0 GiB data, 9.1 GiB used, 527 GiB / 536 GiB avail; 1023 B/s rd, 0 B/s wr, 1.12k op/s; 3959/100878 objects misplaced (3.925%) 2024-09-18T08:17:44.905 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:17:44 smithi050 ceph-mon[32371]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-18T08:17:44.905 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:17:44 smithi050 ceph-mon[32371]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-18T08:17:44.905 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:17:44 smithi050 ceph-mon[32371]: osdmap e382: 8 total, 8 up, 6 in 2024-09-18T08:17:44.905 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:17:44 smithi050 ceph-mon[32371]: 1.8 scrub starts 2024-09-18T08:17:44.905 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:17:44 smithi050 ceph-mon[32371]: 1.8 scrub ok 2024-09-18T08:17:44.905 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:17:44 smithi050 ceph-mon[32371]: 1.9 scrub starts 2024-09-18T08:17:44.905 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:17:44 smithi050 ceph-mon[32371]: 1.9 scrub ok 2024-09-18T08:17:44.906 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:17:44 smithi050 ceph-mon[32371]: osdmap e383: 8 total, 8 up, 6 in 2024-09-18T08:17:44.969 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5 2024-09-18T08:17:44.996 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:17:44.997 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:17:44.997 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5' 2024-09-18T08:17:45.098 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0 2024-09-18T08:17:45.133 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:17:45.134 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:17:45.135 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0' 2024-09-18T08:17:45.236 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6 2024-09-18T08:17:45.259 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:17:45.260 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:17:45.260 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6' 2024-09-18T08:17:45.307 INFO:teuthology.orchestra.run.smithi007.stdout:{ 2024-09-18T08:17:45.307 INFO:teuthology.orchestra.run.smithi007.stdout: "ops": [ 2024-09-18T08:17:45.307 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:17:45.307 INFO:teuthology.orchestra.run.smithi007.stdout: "description": "osd_op(client.15966.0:175836 10.8 10:1963719f:::benchmark_data_smithi007_148834_object74116:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e383)", 2024-09-18T08:17:45.307 INFO:teuthology.orchestra.run.smithi007.stdout: "initiated_at": "2024-09-18T08:17:45.298925+0000", 2024-09-18T08:17:45.307 INFO:teuthology.orchestra.run.smithi007.stdout: "age": 0.0054585340000000001, 2024-09-18T08:17:45.307 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.0054862019999999999, 2024-09-18T08:17:45.307 INFO:teuthology.orchestra.run.smithi007.stdout: "continuous": false, 2024-09-18T08:17:45.307 INFO:teuthology.orchestra.run.smithi007.stdout: "type_data": { 2024-09-18T08:17:45.307 INFO:teuthology.orchestra.run.smithi007.stdout: "flag_point": "started", 2024-09-18T08:17:45.308 INFO:teuthology.orchestra.run.smithi007.stdout: "client_info": { 2024-09-18T08:17:45.308 INFO:teuthology.orchestra.run.smithi007.stdout: "client": "client.15966", 2024-09-18T08:17:45.308 INFO:teuthology.orchestra.run.smithi007.stdout: "client_addr": "172.21.15.7:0/529784850", 2024-09-18T08:17:45.308 INFO:teuthology.orchestra.run.smithi007.stdout: "tid": 175836 2024-09-18T08:17:45.308 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:17:45.308 INFO:teuthology.orchestra.run.smithi007.stdout: "events": [ 2024-09-18T08:17:45.308 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:17:45.308 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "initiated", 2024-09-18T08:17:45.308 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:17:45.298925+0000", 2024-09-18T08:17:45.308 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-09-18T08:17:45.308 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:17:45.308 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:17:45.308 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "header_read", 2024-09-18T08:17:45.309 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:17:45.298925+0000", 2024-09-18T08:17:45.309 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-09-18T08:17:45.309 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:17:45.309 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:17:45.309 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "throttled", 2024-09-18T08:17:45.309 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:17:45.298926+0000", 2024-09-18T08:17:45.309 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 9.3399999999999997e-07 2024-09-18T08:17:45.309 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:17:45.309 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:17:45.309 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "all_read", 2024-09-18T08:17:45.309 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:17:45.298930+0000", 2024-09-18T08:17:45.309 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 3.5549999999999998e-06 2024-09-18T08:17:45.310 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:17:45.310 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:17:45.310 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "dispatched", 2024-09-18T08:17:45.310 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:17:45.298939+0000", 2024-09-18T08:17:45.310 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 9.6220000000000004e-06 2024-09-18T08:17:45.310 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:17:45.310 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:17:45.310 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "queued_for_pg", 2024-09-18T08:17:45.310 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:17:45.298946+0000", 2024-09-18T08:17:45.310 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 7.0180000000000004e-06 2024-09-18T08:17:45.310 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:17:45.310 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:17:45.310 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-09-18T08:17:45.310 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:17:45.302808+0000", 2024-09-18T08:17:45.311 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.0038614230000000001 2024-09-18T08:17:45.311 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:17:45.311 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:17:45.311 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "started", 2024-09-18T08:17:45.311 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:17:45.304315+0000", 2024-09-18T08:17:45.311 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.0015075640000000001 2024-09-18T08:17:45.311 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:17:45.311 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:17:45.311 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "waiting for subops from 2,5", 2024-09-18T08:17:45.311 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:17:45.304420+0000", 2024-09-18T08:17:45.311 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.000104546 2024-09-18T08:17:45.311 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-09-18T08:17:45.311 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-09-18T08:17:45.312 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-09-18T08:17:45.312 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:17:45.312 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:17:45.312 INFO:teuthology.orchestra.run.smithi007.stdout: "description": "osd_op(client.15966.0:175837 10.c 10.12573cac (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e383)", 2024-09-18T08:17:45.312 INFO:teuthology.orchestra.run.smithi007.stdout: "initiated_at": "2024-09-18T08:17:45.299011+0000", 2024-09-18T08:17:45.312 INFO:teuthology.orchestra.run.smithi007.stdout: "age": 0.0053725769999999999, 2024-09-18T08:17:45.312 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.0054713039999999998, 2024-09-18T08:17:45.312 INFO:teuthology.orchestra.run.smithi007.stdout: "continuous": false, 2024-09-18T08:17:45.312 INFO:teuthology.orchestra.run.smithi007.stdout: "type_data": { 2024-09-18T08:17:45.312 INFO:teuthology.orchestra.run.smithi007.stdout: "flag_point": "queued for pg", 2024-09-18T08:17:45.312 INFO:teuthology.orchestra.run.smithi007.stdout: "client_info": { 2024-09-18T08:17:45.312 INFO:teuthology.orchestra.run.smithi007.stdout: "client": "client.15966", 2024-09-18T08:17:45.312 INFO:teuthology.orchestra.run.smithi007.stdout: "client_addr": "172.21.15.7:0/529784850", 2024-09-18T08:17:45.313 INFO:teuthology.orchestra.run.smithi007.stdout: "tid": 175837 2024-09-18T08:17:45.313 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:17:45.313 INFO:teuthology.orchestra.run.smithi007.stdout: "events": [ 2024-09-18T08:17:45.313 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:17:45.313 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "initiated", 2024-09-18T08:17:45.313 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:17:45.299011+0000", 2024-09-18T08:17:45.313 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-09-18T08:17:45.314 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:17:45.314 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:17:45.314 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "header_read", 2024-09-18T08:17:45.314 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:17:45.299011+0000", 2024-09-18T08:17:45.314 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-09-18T08:17:45.314 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:17:45.314 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:17:45.314 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "throttled", 2024-09-18T08:17:45.314 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:17:45.299012+0000", 2024-09-18T08:17:45.314 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 7.3200000000000004e-07 2024-09-18T08:17:45.314 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:17:45.314 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:17:45.315 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "all_read", 2024-09-18T08:17:45.315 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:17:45.299015+0000", 2024-09-18T08:17:45.315 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 3.5300000000000001e-06 2024-09-18T08:17:45.315 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:17:45.315 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:17:45.315 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "dispatched", 2024-09-18T08:17:45.315 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:17:45.299048+0000", 2024-09-18T08:17:45.315 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 3.2233999999999997e-05 2024-09-18T08:17:45.315 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:17:45.315 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:17:45.315 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "queued_for_pg", 2024-09-18T08:17:45.315 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:17:45.299054+0000", 2024-09-18T08:17:45.315 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 6.319e-06 2024-09-18T08:17:45.316 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-09-18T08:17:45.316 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-09-18T08:17:45.316 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-09-18T08:17:45.316 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:17:45.316 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:17:45.316 INFO:teuthology.orchestra.run.smithi007.stdout: "description": "osd_op(client.15966.0:175839 10.a 10:5ac23374:::benchmark_data_smithi007_148834_object74119:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e383)", 2024-09-18T08:17:45.316 INFO:teuthology.orchestra.run.smithi007.stdout: "initiated_at": "2024-09-18T08:17:45.299207+0000", 2024-09-18T08:17:45.316 INFO:teuthology.orchestra.run.smithi007.stdout: "age": 0.0051763729999999997, 2024-09-18T08:17:45.316 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.005330324, 2024-09-18T08:17:45.316 INFO:teuthology.orchestra.run.smithi007.stdout: "continuous": false, 2024-09-18T08:17:45.316 INFO:teuthology.orchestra.run.smithi007.stdout: "type_data": { 2024-09-18T08:17:45.316 INFO:teuthology.orchestra.run.smithi007.stdout: "flag_point": "waiting for sub ops", 2024-09-18T08:17:45.316 INFO:teuthology.orchestra.run.smithi007.stdout: "client_info": { 2024-09-18T08:17:45.317 INFO:teuthology.orchestra.run.smithi007.stdout: "client": "client.15966", 2024-09-18T08:17:45.317 INFO:teuthology.orchestra.run.smithi007.stdout: "client_addr": "172.21.15.7:0/529784850", 2024-09-18T08:17:45.317 INFO:teuthology.orchestra.run.smithi007.stdout: "tid": 175839 2024-09-18T08:17:45.317 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:17:45.317 INFO:teuthology.orchestra.run.smithi007.stdout: "events": [ 2024-09-18T08:17:45.317 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:17:45.317 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "initiated", 2024-09-18T08:17:45.317 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:17:45.299207+0000", 2024-09-18T08:17:45.318 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-09-18T08:17:45.318 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:17:45.318 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:17:45.318 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "header_read", 2024-09-18T08:17:45.318 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:17:45.299207+0000", 2024-09-18T08:17:45.318 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-09-18T08:17:45.318 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:17:45.318 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:17:45.318 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "throttled", 2024-09-18T08:17:45.318 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:17:45.299208+0000", 2024-09-18T08:17:45.318 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 7.1999999999999999e-07 2024-09-18T08:17:45.318 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:17:45.318 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:17:45.319 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "all_read", 2024-09-18T08:17:45.319 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:17:45.299212+0000", 2024-09-18T08:17:45.319 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 4.0439999999999997e-06 2024-09-18T08:17:45.319 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:17:45.319 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:17:45.319 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "dispatched", 2024-09-18T08:17:45.319 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:17:45.299220+0000", 2024-09-18T08:17:45.319 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 7.9419999999999995e-06 2024-09-18T08:17:45.319 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:17:45.319 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:17:45.319 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "queued_for_pg", 2024-09-18T08:17:45.319 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:17:45.299226+0000", 2024-09-18T08:17:45.319 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 5.9089999999999997e-06 2024-09-18T08:17:45.319 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:17:45.320 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:17:45.320 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-09-18T08:17:45.320 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:17:45.299610+0000", 2024-09-18T08:17:45.320 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.00038435800000000001 2024-09-18T08:17:45.320 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:17:45.320 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:17:45.320 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "started", 2024-09-18T08:17:45.320 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:17:45.299727+0000", 2024-09-18T08:17:45.320 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.000116384 2024-09-18T08:17:45.320 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:17:45.320 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:17:45.320 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "waiting for subops from 5,6", 2024-09-18T08:17:45.320 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:17:45.299840+0000", 2024-09-18T08:17:45.321 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.000113001 2024-09-18T08:17:45.321 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:17:45.321 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:17:45.321 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:17:45.321 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:17:45.303534+0000", 2024-09-18T08:17:45.321 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.0036942979999999999 2024-09-18T08:17:45.321 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-09-18T08:17:45.321 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-09-18T08:17:45.321 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-09-18T08:17:45.321 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:17:45.321 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:17:45.321 INFO:teuthology.orchestra.run.smithi007.stdout: "description": "osd_repop_reply(client.15966.0:175839 10.a e383/354)", 2024-09-18T08:17:45.321 INFO:teuthology.orchestra.run.smithi007.stdout: "initiated_at": "2024-09-18T08:17:45.303170+0000", 2024-09-18T08:17:45.322 INFO:teuthology.orchestra.run.smithi007.stdout: "age": 0.001213754, 2024-09-18T08:17:45.322 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.0014311739999999999, 2024-09-18T08:17:45.322 INFO:teuthology.orchestra.run.smithi007.stdout: "continuous": false, 2024-09-18T08:17:45.322 INFO:teuthology.orchestra.run.smithi007.stdout: "type_data": { 2024-09-18T08:17:45.322 INFO:teuthology.orchestra.run.smithi007.stdout: "flag_point": "queued for pg", 2024-09-18T08:17:45.322 INFO:teuthology.orchestra.run.smithi007.stdout: "events": [ 2024-09-18T08:17:45.322 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:17:45.322 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "initiated", 2024-09-18T08:17:45.322 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:17:45.303170+0000", 2024-09-18T08:17:45.322 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-09-18T08:17:45.322 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:17:45.322 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:17:45.322 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "header_read", 2024-09-18T08:17:45.322 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:17:45.303170+0000", 2024-09-18T08:17:45.323 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-09-18T08:17:45.323 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:17:45.323 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:17:45.324 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "throttled", 2024-09-18T08:17:45.324 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:17:45.303171+0000", 2024-09-18T08:17:45.324 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 9.1299999999999998e-07 2024-09-18T08:17:45.324 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:17:45.324 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:17:45.324 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "all_read", 2024-09-18T08:17:45.324 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:17:45.303176+0000", 2024-09-18T08:17:45.324 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 5.5140000000000001e-06 2024-09-18T08:17:45.324 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:17:45.324 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:17:45.324 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "dispatched", 2024-09-18T08:17:45.324 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:17:45.303186+0000", 2024-09-18T08:17:45.325 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 9.7820000000000005e-06 2024-09-18T08:17:45.325 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:17:45.325 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:17:45.325 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "queued_for_pg", 2024-09-18T08:17:45.325 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:17:45.303193+0000", 2024-09-18T08:17:45.325 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 6.6039999999999998e-06 2024-09-18T08:17:45.325 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-09-18T08:17:45.325 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-09-18T08:17:45.325 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-09-18T08:17:45.325 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:17:45.325 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:17:45.325 INFO:teuthology.orchestra.run.smithi007.stdout: "description": "osd_repop(client.15966.0:175840 10.b e383/311)", 2024-09-18T08:17:45.325 INFO:teuthology.orchestra.run.smithi007.stdout: "initiated_at": "2024-09-18T08:17:45.303835+0000", 2024-09-18T08:17:45.326 INFO:teuthology.orchestra.run.smithi007.stdout: "age": 0.00054855099999999999, 2024-09-18T08:17:45.326 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.00080311900000000003, 2024-09-18T08:17:45.326 INFO:teuthology.orchestra.run.smithi007.stdout: "continuous": false, 2024-09-18T08:17:45.326 INFO:teuthology.orchestra.run.smithi007.stdout: "type_data": { 2024-09-18T08:17:45.326 INFO:teuthology.orchestra.run.smithi007.stdout: "flag_point": "queued for pg", 2024-09-18T08:17:45.326 INFO:teuthology.orchestra.run.smithi007.stdout: "events": [ 2024-09-18T08:17:45.326 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:17:45.326 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "initiated", 2024-09-18T08:17:45.326 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:17:45.303835+0000", 2024-09-18T08:17:45.326 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-09-18T08:17:45.326 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:17:45.326 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:17:45.326 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "header_read", 2024-09-18T08:17:45.327 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:17:45.303835+0000", 2024-09-18T08:17:45.327 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-09-18T08:17:45.327 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:17:45.327 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:17:45.327 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "throttled", 2024-09-18T08:17:45.327 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:17:45.303836+0000", 2024-09-18T08:17:45.327 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 6.4899999999999995e-07 2024-09-18T08:17:45.327 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:17:45.327 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:17:45.327 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "all_read", 2024-09-18T08:17:45.327 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:17:45.303841+0000", 2024-09-18T08:17:45.327 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 5.3870000000000003e-06 2024-09-18T08:17:45.327 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:17:45.328 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:17:45.328 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "dispatched", 2024-09-18T08:17:45.328 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:17:45.303858+0000", 2024-09-18T08:17:45.328 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.7302000000000001e-05 2024-09-18T08:17:45.328 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:17:45.328 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:17:45.328 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "queued_for_pg", 2024-09-18T08:17:45.328 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:17:45.303865+0000", 2024-09-18T08:17:45.328 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 6.1879999999999999e-06 2024-09-18T08:17:45.328 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-09-18T08:17:45.328 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-09-18T08:17:45.328 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-09-18T08:17:45.328 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:17:45.329 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:17:45.329 INFO:teuthology.orchestra.run.smithi007.stdout: "description": "osd_op(client.15966.0:175843 10.a 10.b03f6cca (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e383)", 2024-09-18T08:17:45.329 INFO:teuthology.orchestra.run.smithi007.stdout: "initiated_at": "2024-09-18T08:17:45.304174+0000", 2024-09-18T08:17:45.329 INFO:teuthology.orchestra.run.smithi007.stdout: "age": 0.000210046, 2024-09-18T08:17:45.329 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.00050194499999999995, 2024-09-18T08:17:45.329 INFO:teuthology.orchestra.run.smithi007.stdout: "continuous": false, 2024-09-18T08:17:45.329 INFO:teuthology.orchestra.run.smithi007.stdout: "type_data": { 2024-09-18T08:17:45.329 INFO:teuthology.orchestra.run.smithi007.stdout: "flag_point": "queued for pg", 2024-09-18T08:17:45.329 INFO:teuthology.orchestra.run.smithi007.stdout: "client_info": { 2024-09-18T08:17:45.329 INFO:teuthology.orchestra.run.smithi007.stdout: "client": "client.15966", 2024-09-18T08:17:45.329 INFO:teuthology.orchestra.run.smithi007.stdout: "client_addr": "172.21.15.7:0/529784850", 2024-09-18T08:17:45.329 INFO:teuthology.orchestra.run.smithi007.stdout: "tid": 175843 2024-09-18T08:17:45.329 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:17:45.330 INFO:teuthology.orchestra.run.smithi007.stdout: "events": [ 2024-09-18T08:17:45.330 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:17:45.330 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "initiated", 2024-09-18T08:17:45.330 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:17:45.304174+0000", 2024-09-18T08:17:45.330 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-09-18T08:17:45.330 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:17:45.330 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:17:45.330 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "header_read", 2024-09-18T08:17:45.330 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:17:45.304174+0000", 2024-09-18T08:17:45.330 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-09-18T08:17:45.331 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:17:45.331 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:17:45.331 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "throttled", 2024-09-18T08:17:45.331 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:17:45.304175+0000", 2024-09-18T08:17:45.331 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.3629999999999999e-06 2024-09-18T08:17:45.331 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:17:45.331 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:17:45.331 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "all_read", 2024-09-18T08:17:45.331 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:17:45.304179+0000", 2024-09-18T08:17:45.331 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 4.0849999999999999e-06 2024-09-18T08:17:45.331 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:17:45.331 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:17:45.331 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "dispatched", 2024-09-18T08:17:45.332 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:17:45.304194+0000", 2024-09-18T08:17:45.332 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.5122e-05 2024-09-18T08:17:45.332 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:17:45.332 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:17:45.332 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "queued_for_pg", 2024-09-18T08:17:45.332 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:17:45.304202+0000", 2024-09-18T08:17:45.332 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 7.3320000000000003e-06 2024-09-18T08:17:45.332 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-09-18T08:17:45.332 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-09-18T08:17:45.332 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-09-18T08:17:45.332 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:17:45.332 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:17:45.332 INFO:teuthology.orchestra.run.smithi007.stdout: "description": "osd_op(client.15966.0:175825 10.4 10:2a4b59ed:::benchmark_data_smithi007_148834_object74105:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e383)", 2024-09-18T08:17:45.333 INFO:teuthology.orchestra.run.smithi007.stdout: "initiated_at": "2024-09-18T08:17:45.291086+0000", 2024-09-18T08:17:45.333 INFO:teuthology.orchestra.run.smithi007.stdout: "age": 0.013297349999999999, 2024-09-18T08:17:45.333 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.013637167, 2024-09-18T08:17:45.333 INFO:teuthology.orchestra.run.smithi007.stdout: "continuous": false, 2024-09-18T08:17:45.333 INFO:teuthology.orchestra.run.smithi007.stdout: "type_data": { 2024-09-18T08:17:45.333 INFO:teuthology.orchestra.run.smithi007.stdout: "flag_point": "waiting for sub ops", 2024-09-18T08:17:45.333 INFO:teuthology.orchestra.run.smithi007.stdout: "client_info": { 2024-09-18T08:17:45.333 INFO:teuthology.orchestra.run.smithi007.stdout: "client": "client.15966", 2024-09-18T08:17:45.333 INFO:teuthology.orchestra.run.smithi007.stdout: "client_addr": "172.21.15.7:0/529784850", 2024-09-18T08:17:45.333 INFO:teuthology.orchestra.run.smithi007.stdout: "tid": 175825 2024-09-18T08:17:45.333 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:17:45.333 INFO:teuthology.orchestra.run.smithi007.stdout: "events": [ 2024-09-18T08:17:45.333 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:17:45.333 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "initiated", 2024-09-18T08:17:45.334 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:17:45.291086+0000", 2024-09-18T08:17:45.334 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-09-18T08:17:45.334 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:17:45.334 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:17:45.334 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "header_read", 2024-09-18T08:17:45.334 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:17:45.291086+0000", 2024-09-18T08:17:45.334 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-09-18T08:17:45.334 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:17:45.334 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:17:45.334 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "throttled", 2024-09-18T08:17:45.334 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:17:45.291088+0000", 2024-09-18T08:17:45.334 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.463e-06 2024-09-18T08:17:45.335 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:17:45.335 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:17:45.335 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "all_read", 2024-09-18T08:17:45.335 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:17:45.291093+0000", 2024-09-18T08:17:45.335 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 5.293e-06 2024-09-18T08:17:45.335 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:17:45.335 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:17:45.335 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "dispatched", 2024-09-18T08:17:45.335 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:17:45.291104+0000", 2024-09-18T08:17:45.335 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.1218e-05 2024-09-18T08:17:45.335 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:17:45.335 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:17:45.335 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "queued_for_pg", 2024-09-18T08:17:45.335 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:17:45.291111+0000", 2024-09-18T08:17:45.336 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 6.8669999999999996e-06 2024-09-18T08:17:45.336 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:17:45.336 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:17:45.336 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-09-18T08:17:45.336 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:17:45.298768+0000", 2024-09-18T08:17:45.336 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.0076571510000000001 2024-09-18T08:17:45.336 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:17:45.336 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:17:45.336 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "started", 2024-09-18T08:17:45.336 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:17:45.299918+0000", 2024-09-18T08:17:45.336 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.0011500130000000001 2024-09-18T08:17:45.336 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:17:45.336 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:17:45.337 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "waiting for subops from 1,3", 2024-09-18T08:17:45.337 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:17:45.300035+0000", 2024-09-18T08:17:45.337 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.00011617599999999999 2024-09-18T08:17:45.337 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-09-18T08:17:45.337 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-09-18T08:17:45.337 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-09-18T08:17:45.337 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:17:45.337 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:17:45.337 INFO:teuthology.orchestra.run.smithi007.stdout: "description": "osd_op(client.15966.0:175827 10.8 10:1e530228:::benchmark_data_smithi007_148834_object74107:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e383)", 2024-09-18T08:17:45.337 INFO:teuthology.orchestra.run.smithi007.stdout: "initiated_at": "2024-09-18T08:17:45.291122+0000", 2024-09-18T08:17:45.337 INFO:teuthology.orchestra.run.smithi007.stdout: "age": 0.013261254, 2024-09-18T08:17:45.337 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.013660011999999999, 2024-09-18T08:17:45.337 INFO:teuthology.orchestra.run.smithi007.stdout: "continuous": false, 2024-09-18T08:17:45.338 INFO:teuthology.orchestra.run.smithi007.stdout: "type_data": { 2024-09-18T08:17:45.338 INFO:teuthology.orchestra.run.smithi007.stdout: "flag_point": "waiting for sub ops", 2024-09-18T08:17:45.338 INFO:teuthology.orchestra.run.smithi007.stdout: "client_info": { 2024-09-18T08:17:45.338 INFO:teuthology.orchestra.run.smithi007.stdout: "client": "client.15966", 2024-09-18T08:17:45.338 INFO:teuthology.orchestra.run.smithi007.stdout: "client_addr": "172.21.15.7:0/529784850", 2024-09-18T08:17:45.338 INFO:teuthology.orchestra.run.smithi007.stdout: "tid": 175827 2024-09-18T08:17:45.338 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:17:45.338 INFO:teuthology.orchestra.run.smithi007.stdout: "events": [ 2024-09-18T08:17:45.338 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:17:45.338 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "initiated", 2024-09-18T08:17:45.338 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:17:45.291122+0000", 2024-09-18T08:17:45.338 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-09-18T08:17:45.338 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:17:45.339 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:17:45.339 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "header_read", 2024-09-18T08:17:45.339 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:17:45.291122+0000", 2024-09-18T08:17:45.339 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-09-18T08:17:45.339 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:17:45.339 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:17:45.339 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "throttled", 2024-09-18T08:17:45.339 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:17:45.291123+0000", 2024-09-18T08:17:45.339 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 4.0999999999999999e-07 2024-09-18T08:17:45.339 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:17:45.339 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:17:45.339 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "all_read", 2024-09-18T08:17:45.339 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:17:45.291125+0000", 2024-09-18T08:17:45.340 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 2.294e-06 2024-09-18T08:17:45.340 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:17:45.340 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:17:45.340 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "dispatched", 2024-09-18T08:17:45.340 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:17:45.291130+0000", 2024-09-18T08:17:45.340 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 5.0239999999999997e-06 2024-09-18T08:17:45.340 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:17:45.340 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:17:45.340 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "queued_for_pg", 2024-09-18T08:17:45.340 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:17:45.291135+0000", 2024-09-18T08:17:45.340 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 4.549e-06 2024-09-18T08:17:45.340 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:17:45.341 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:17:45.341 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-09-18T08:17:45.341 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:17:45.292889+0000", 2024-09-18T08:17:45.341 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.0017547089999999999 2024-09-18T08:17:45.341 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:17:45.341 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:17:45.341 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "started", 2024-09-18T08:17:45.341 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:17:45.293007+0000", 2024-09-18T08:17:45.341 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.000117551 2024-09-18T08:17:45.341 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:17:45.341 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:17:45.341 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "waiting for subops from 2,5", 2024-09-18T08:17:45.341 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:17:45.293120+0000", 2024-09-18T08:17:45.342 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.000113393 2024-09-18T08:17:45.342 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:17:45.343 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:17:45.343 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:17:45.344 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:17:45.302667+0000", 2024-09-18T08:17:45.344 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.0095463530000000005 2024-09-18T08:17:45.344 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:17:45.344 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:17:45.344 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:17:45.344 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:17:45.302706+0000", 2024-09-18T08:17:45.344 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 3.8801999999999999e-05 2024-09-18T08:17:45.344 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-09-18T08:17:45.344 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-09-18T08:17:45.344 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-09-18T08:17:45.344 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:17:45.344 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:17:45.344 INFO:teuthology.orchestra.run.smithi007.stdout: "description": "osd_op(client.15966.0:175830 10.c 10:3603fc77:::benchmark_data_smithi007_148834_object74110:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e383)", 2024-09-18T08:17:45.345 INFO:teuthology.orchestra.run.smithi007.stdout: "initiated_at": "2024-09-18T08:17:45.295878+0000", 2024-09-18T08:17:45.345 INFO:teuthology.orchestra.run.smithi007.stdout: "age": 0.0085055530000000008, 2024-09-18T08:17:45.345 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.0090160150000000005, 2024-09-18T08:17:45.345 INFO:teuthology.orchestra.run.smithi007.stdout: "continuous": false, 2024-09-18T08:17:45.345 INFO:teuthology.orchestra.run.smithi007.stdout: "type_data": { 2024-09-18T08:17:45.345 INFO:teuthology.orchestra.run.smithi007.stdout: "flag_point": "waiting for sub ops", 2024-09-18T08:17:45.345 INFO:teuthology.orchestra.run.smithi007.stdout: "client_info": { 2024-09-18T08:17:45.345 INFO:teuthology.orchestra.run.smithi007.stdout: "client": "client.15966", 2024-09-18T08:17:45.345 INFO:teuthology.orchestra.run.smithi007.stdout: "client_addr": "172.21.15.7:0/529784850", 2024-09-18T08:17:45.345 INFO:teuthology.orchestra.run.smithi007.stdout: "tid": 175830 2024-09-18T08:17:45.345 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:17:45.345 INFO:teuthology.orchestra.run.smithi007.stdout: "events": [ 2024-09-18T08:17:45.345 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:17:45.346 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "initiated", 2024-09-18T08:17:45.346 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:17:45.295878+0000", 2024-09-18T08:17:45.346 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-09-18T08:17:45.346 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:17:45.346 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:17:45.346 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "header_read", 2024-09-18T08:17:45.346 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:17:45.295878+0000", 2024-09-18T08:17:45.346 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-09-18T08:17:45.346 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:17:45.346 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:17:45.347 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "throttled", 2024-09-18T08:17:45.347 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:17:45.295880+0000", 2024-09-18T08:17:45.347 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.86e-06 2024-09-18T08:17:45.347 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:17:45.347 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:17:45.347 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "all_read", 2024-09-18T08:17:45.347 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:17:45.295885+0000", 2024-09-18T08:17:45.347 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 4.5800000000000002e-06 2024-09-18T08:17:45.347 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:17:45.347 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:17:45.347 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "dispatched", 2024-09-18T08:17:45.347 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:17:45.295895+0000", 2024-09-18T08:17:45.347 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.0739e-05 2024-09-18T08:17:45.348 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:17:45.348 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:17:45.348 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "queued_for_pg", 2024-09-18T08:17:45.348 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:17:45.295902+0000", 2024-09-18T08:17:45.348 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 7.1690000000000003e-06 2024-09-18T08:17:45.348 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:17:45.348 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:17:45.348 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-09-18T08:17:45.348 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:17:45.299132+0000", 2024-09-18T08:17:45.348 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.0032292089999999998 2024-09-18T08:17:45.348 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:17:45.348 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:17:45.348 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "started", 2024-09-18T08:17:45.348 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:17:45.301649+0000", 2024-09-18T08:17:45.349 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.0025174640000000001 2024-09-18T08:17:45.349 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:17:45.349 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:17:45.349 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "waiting for subops from 3,6", 2024-09-18T08:17:45.349 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:17:45.301937+0000", 2024-09-18T08:17:45.349 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.00028776000000000002 2024-09-18T08:17:45.349 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-09-18T08:17:45.349 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-09-18T08:17:45.349 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-09-18T08:17:45.349 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:17:45.349 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:17:45.349 INFO:teuthology.orchestra.run.smithi007.stdout: "description": "osd_repop(client.15966.0:175829 10.0 e383/366 10:013aefd4:::benchmark_data_smithi007_148834_object74109:head v 383'11173, mlcod=383'11170)", 2024-09-18T08:17:45.349 INFO:teuthology.orchestra.run.smithi007.stdout: "initiated_at": "2024-09-18T08:17:45.296161+0000", 2024-09-18T08:17:45.350 INFO:teuthology.orchestra.run.smithi007.stdout: "age": 0.0082223350000000008, 2024-09-18T08:17:45.350 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.0087928360000000001, 2024-09-18T08:17:45.350 INFO:teuthology.orchestra.run.smithi007.stdout: "continuous": false, 2024-09-18T08:17:45.350 INFO:teuthology.orchestra.run.smithi007.stdout: "type_data": { 2024-09-18T08:17:45.350 INFO:teuthology.orchestra.run.smithi007.stdout: "flag_point": "started", 2024-09-18T08:17:45.350 INFO:teuthology.orchestra.run.smithi007.stdout: "events": [ 2024-09-18T08:17:45.350 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:17:45.350 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "initiated", 2024-09-18T08:17:45.350 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:17:45.296161+0000", 2024-09-18T08:17:45.350 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-09-18T08:17:45.350 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:17:45.350 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:17:45.350 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "header_read", 2024-09-18T08:17:45.351 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:17:45.296161+0000", 2024-09-18T08:17:45.351 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-09-18T08:17:45.351 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:17:45.351 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:17:45.351 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "throttled", 2024-09-18T08:17:45.351 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:17:45.296162+0000", 2024-09-18T08:17:45.351 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 6.37e-07 2024-09-18T08:17:45.351 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:17:45.351 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:17:45.351 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "all_read", 2024-09-18T08:17:45.351 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:17:45.296169+0000", 2024-09-18T08:17:45.351 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 6.8929999999999999e-06 2024-09-18T08:17:45.351 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:17:45.352 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:17:45.352 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "dispatched", 2024-09-18T08:17:45.352 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:17:45.296178+0000", 2024-09-18T08:17:45.352 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 9.3230000000000004e-06 2024-09-18T08:17:45.352 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:17:45.352 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:17:45.352 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "queued_for_pg", 2024-09-18T08:17:45.352 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:17:45.296185+0000", 2024-09-18T08:17:45.352 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 6.5810000000000002e-06 2024-09-18T08:17:45.352 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:17:45.352 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:17:45.352 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-09-18T08:17:45.352 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:17:45.296221+0000", 2024-09-18T08:17:45.353 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 3.6284e-05 2024-09-18T08:17:45.353 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:17:45.353 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:17:45.353 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "started", 2024-09-18T08:17:45.353 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:17:45.296241+0000", 2024-09-18T08:17:45.353 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.9514e-05 2024-09-18T08:17:45.353 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-09-18T08:17:45.353 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-09-18T08:17:45.353 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-09-18T08:17:45.353 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:17:45.353 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:17:45.353 INFO:teuthology.orchestra.run.smithi007.stdout: "description": "osd_op(client.15966.0:175831 10.8 10:11721065:::benchmark_data_smithi007_148834_object74111:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e383)", 2024-09-18T08:17:45.354 INFO:teuthology.orchestra.run.smithi007.stdout: "initiated_at": "2024-09-18T08:17:45.296609+0000", 2024-09-18T08:17:45.354 INFO:teuthology.orchestra.run.smithi007.stdout: "age": 0.0077750320000000003, 2024-09-18T08:17:45.354 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.0084056559999999992, 2024-09-18T08:17:45.354 INFO:teuthology.orchestra.run.smithi007.stdout: "continuous": false, 2024-09-18T08:17:45.354 INFO:teuthology.orchestra.run.smithi007.stdout: "type_data": { 2024-09-18T08:17:45.354 INFO:teuthology.orchestra.run.smithi007.stdout: "flag_point": "waiting for sub ops", 2024-09-18T08:17:45.354 INFO:teuthology.orchestra.run.smithi007.stdout: "client_info": { 2024-09-18T08:17:45.354 INFO:teuthology.orchestra.run.smithi007.stdout: "client": "client.15966", 2024-09-18T08:17:45.354 INFO:teuthology.orchestra.run.smithi007.stdout: "client_addr": "172.21.15.7:0/529784850", 2024-09-18T08:17:45.354 INFO:teuthology.orchestra.run.smithi007.stdout: "tid": 175831 2024-09-18T08:17:45.354 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:17:45.354 INFO:teuthology.orchestra.run.smithi007.stdout: "events": [ 2024-09-18T08:17:45.354 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:17:45.355 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "initiated", 2024-09-18T08:17:45.355 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:17:45.296609+0000", 2024-09-18T08:17:45.355 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-09-18T08:17:45.355 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:17:45.355 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:17:45.355 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "header_read", 2024-09-18T08:17:45.355 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:17:45.296609+0000", 2024-09-18T08:17:45.355 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-09-18T08:17:45.355 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:17:45.355 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:17:45.355 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "throttled", 2024-09-18T08:17:45.355 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:17:45.296610+0000", 2024-09-18T08:17:45.355 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.1480000000000001e-06 2024-09-18T08:17:45.356 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:17:45.356 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:17:45.356 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "all_read", 2024-09-18T08:17:45.356 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:17:45.296615+0000", 2024-09-18T08:17:45.356 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 5.1320000000000002e-06 2024-09-18T08:17:45.356 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:17:45.356 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:17:45.356 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "dispatched", 2024-09-18T08:17:45.356 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:17:45.296631+0000", 2024-09-18T08:17:45.356 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.5698000000000001e-05 2024-09-18T08:17:45.356 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:17:45.356 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:17:45.356 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "queued_for_pg", 2024-09-18T08:17:45.357 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:17:45.296639+0000", 2024-09-18T08:17:45.357 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 8.6209999999999993e-06 2024-09-18T08:17:45.357 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:17:45.357 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:17:45.357 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-09-18T08:17:45.357 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:17:45.296702+0000", 2024-09-18T08:17:45.357 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 6.2959000000000005e-05 2024-09-18T08:17:45.357 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:17:45.357 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:17:45.357 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "started", 2024-09-18T08:17:45.357 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:17:45.297042+0000", 2024-09-18T08:17:45.357 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.00033969399999999998 2024-09-18T08:17:45.357 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:17:45.358 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:17:45.358 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "waiting for subops from 2,5", 2024-09-18T08:17:45.358 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:17:45.297152+0000", 2024-09-18T08:17:45.358 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.00011021100000000001 2024-09-18T08:17:45.358 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:17:45.358 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:17:45.358 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:17:45.358 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:17:45.302745+0000", 2024-09-18T08:17:45.358 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.0055932860000000003 2024-09-18T08:17:45.358 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:17:45.358 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:17:45.358 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:17:45.358 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:17:45.302779+0000", 2024-09-18T08:17:45.358 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 3.3940999999999999e-05 2024-09-18T08:17:45.358 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-09-18T08:17:45.359 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-09-18T08:17:45.359 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-09-18T08:17:45.359 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:17:45.359 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:17:45.359 INFO:teuthology.orchestra.run.smithi007.stdout: "description": "osd_op(client.15966.0:175833 10.4 10:2fe6c280:::benchmark_data_smithi007_148834_object74113:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e383)", 2024-09-18T08:17:45.359 INFO:teuthology.orchestra.run.smithi007.stdout: "initiated_at": "2024-09-18T08:17:45.296796+0000", 2024-09-18T08:17:45.359 INFO:teuthology.orchestra.run.smithi007.stdout: "age": 0.007587664, 2024-09-18T08:17:45.359 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.0082884740000000005, 2024-09-18T08:17:45.359 INFO:teuthology.orchestra.run.smithi007.stdout: "continuous": false, 2024-09-18T08:17:45.359 INFO:teuthology.orchestra.run.smithi007.stdout: "type_data": { 2024-09-18T08:17:45.359 INFO:teuthology.orchestra.run.smithi007.stdout: "flag_point": "waiting for sub ops", 2024-09-18T08:17:45.359 INFO:teuthology.orchestra.run.smithi007.stdout: "client_info": { 2024-09-18T08:17:45.359 INFO:teuthology.orchestra.run.smithi007.stdout: "client": "client.15966", 2024-09-18T08:17:45.359 INFO:teuthology.orchestra.run.smithi007.stdout: "client_addr": "172.21.15.7:0/529784850", 2024-09-18T08:17:45.359 INFO:teuthology.orchestra.run.smithi007.stdout: "tid": 175833 2024-09-18T08:17:45.359 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:17:45.360 INFO:teuthology.orchestra.run.smithi007.stdout: "events": [ 2024-09-18T08:17:45.360 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:17:45.360 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "initiated", 2024-09-18T08:17:45.360 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:17:45.296796+0000", 2024-09-18T08:17:45.360 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-09-18T08:17:45.360 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:17:45.360 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:17:45.360 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "header_read", 2024-09-18T08:17:45.360 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:17:45.296796+0000", 2024-09-18T08:17:45.360 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-09-18T08:17:45.360 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:17:45.360 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:17:45.360 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "throttled", 2024-09-18T08:17:45.361 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:17:45.296797+0000", 2024-09-18T08:17:45.361 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.0809999999999999e-06 2024-09-18T08:17:45.361 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:17:45.361 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:17:45.361 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "all_read", 2024-09-18T08:17:45.361 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:17:45.296802+0000", 2024-09-18T08:17:45.361 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 4.4129999999999999e-06 2024-09-18T08:17:45.361 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:17:45.361 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:17:45.361 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "dispatched", 2024-09-18T08:17:45.361 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:17:45.296810+0000", 2024-09-18T08:17:45.361 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 8.6300000000000004e-06 2024-09-18T08:17:45.362 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:17:45.362 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:17:45.362 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "queued_for_pg", 2024-09-18T08:17:45.362 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:17:45.296817+0000", 2024-09-18T08:17:45.362 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 6.8519999999999998e-06 2024-09-18T08:17:45.362 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:17:45.362 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:17:45.362 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-09-18T08:17:45.362 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:17:45.302549+0000", 2024-09-18T08:17:45.362 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.0057320219999999998 2024-09-18T08:17:45.362 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:17:45.363 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:17:45.363 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "started", 2024-09-18T08:17:45.363 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:17:45.302677+0000", 2024-09-18T08:17:45.363 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.00012797500000000001 2024-09-18T08:17:45.363 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:17:45.363 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:17:45.363 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "waiting for subops from 1,3", 2024-09-18T08:17:45.363 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:17:45.303124+0000", 2024-09-18T08:17:45.363 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.00044719999999999997 2024-09-18T08:17:45.363 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-09-18T08:17:45.363 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-09-18T08:17:45.363 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-09-18T08:17:45.363 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-09-18T08:17:45.364 INFO:teuthology.orchestra.run.smithi007.stdout: ], 2024-09-18T08:17:45.364 INFO:teuthology.orchestra.run.smithi007.stdout: "num_ops": 12 2024-09-18T08:17:45.364 INFO:teuthology.orchestra.run.smithi007.stdout:} 2024-09-18T08:17:45.364 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2 2024-09-18T08:17:45.406 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:17:45.407 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:17:45.408 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2' 2024-09-18T08:17:45.508 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1 2024-09-18T08:17:45.549 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:17:45.549 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:17:45.549 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1' 2024-09-18T08:17:45.650 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4 2024-09-18T08:17:45.674 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:17:45.675 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:17:45.675 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4' 2024-09-18T08:17:45.776 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4 2024-09-18T08:17:45.801 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:17:45.801 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:17:45.802 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4' 2024-09-18T08:17:45.902 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1 2024-09-18T08:17:45.944 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:17:45.945 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:17:45.946 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1' 2024-09-18T08:17:46.046 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0 2024-09-18T08:17:46.089 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:17:46.089 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:17:46.090 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0' 2024-09-18T08:17:46.190 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3 2024-09-18T08:17:46.237 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:17:46.237 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:17:46.238 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3' 2024-09-18T08:17:46.339 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2 2024-09-18T08:17:46.378 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:17:46.379 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:17:46.379 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2' 2024-09-18T08:17:46.480 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7 2024-09-18T08:17:46.504 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:17:46.505 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:17:46.505 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7' 2024-09-18T08:17:46.606 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6 2024-09-18T08:17:46.633 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:17:46.634 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:17:46.634 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6' 2024-09-18T08:17:46.680 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:17:46 smithi007 ceph-mon[29459]: pgmap v747: 158 pgs: 2 active+remapped+backfill_toofull, 156 active+clean; 1.8 GiB data, 8.4 GiB used, 528 GiB / 536 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.39k op/s; 3613/87507 objects misplaced (4.129%) 2024-09-18T08:17:46.681 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:17:46 smithi007 ceph-mon[39249]: pgmap v747: 158 pgs: 2 active+remapped+backfill_toofull, 156 active+clean; 1.8 GiB data, 8.4 GiB used, 528 GiB / 536 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.39k op/s; 3613/87507 objects misplaced (4.129%) 2024-09-18T08:17:46.735 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6 2024-09-18T08:17:46.760 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:17:46.761 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:17:46.761 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6' 2024-09-18T08:17:46.766 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-09-18T08:17:46.769 INFO:teuthology.orchestra.run.smithi007.stderr:dumped all 2024-09-18T08:17:46.862 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7 2024-09-18T08:17:46.887 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:17:46.888 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:17:46.888 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7' 2024-09-18T08:17:46.905 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:17:46 smithi050 ceph-mon[32371]: pgmap v747: 158 pgs: 2 active+remapped+backfill_toofull, 156 active+clean; 1.8 GiB data, 8.4 GiB used, 528 GiB / 536 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.39k op/s; 3613/87507 objects misplaced (4.129%) 2024-09-18T08:17:46.980 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Sep 18 08:17:46 smithi007 ceph-779349ca-7593-11ef-bceb-c7b262605968-mgr-y[29688]: ::ffff:172.21.15.50 - - [18/Sep/2024:08:17:46] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-09-18T08:17:46.989 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3 2024-09-18T08:17:47.035 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:17:47.036 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:17:47.036 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3' 2024-09-18T08:17:47.059 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid 779349ca-7593-11ef-bceb-c7b262605968 -- ceph osd set noscrub 2024-09-18T08:17:47.137 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3 2024-09-18T08:17:47.196 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid 779349ca-7593-11ef-bceb-c7b262605968 -- ceph daemon osd.0 dump_blocked_ops 2024-09-18T08:17:47.201 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:17:47.201 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:17:47.202 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3' 2024-09-18T08:17:47.302 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4 2024-09-18T08:17:47.328 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:17:47.328 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:17:47.329 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4' 2024-09-18T08:17:47.429 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1 2024-09-18T08:17:47.497 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:17:47.498 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:17:47.498 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1' 2024-09-18T08:17:47.599 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6 2024-09-18T08:17:47.627 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:17:47.627 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:17:47.628 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6' 2024-09-18T08:17:47.728 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4 2024-09-18T08:17:47.754 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:17:47.755 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:17:47.755 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4' 2024-09-18T08:17:47.856 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0 2024-09-18T08:17:47.898 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:17:47.899 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:17:47.899 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0' 2024-09-18T08:17:47.904 INFO:teuthology.orchestra.run.smithi007.stdout:{"pg_ready":true,"pg_map":{"version":747,"stamp":"2024-09-18T08:17:45.436458+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":1899108915,"num_objects":29169,"num_object_clones":0,"num_object_copies":87507,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":3613,"num_objects_unfound":0,"num_objects_dirty":29169,"num_whiteouts":0,"num_read":1474,"num_read_kb":1226,"num_write":175188,"num_write_kb":6509934,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":60474,"num_bytes_recovered":3931978101,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":63223,"ondisk_log_size":63223,"up":474,"acting":474,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":474,"num_osds":6,"num_per_pool_osds":6,"num_per_pool_omap_osds":6,"kb":562470912,"kb_used":8813524,"kb_used_data":5440228,"kb_used_omap":15,"kb_used_meta":3372912,"kb_avail":553657388,"statfs":{"total":575970213888,"available":566945165312,"internally_reserved":0,"allocated":5570793472,"data_stored":5568107078,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":16049,"internal_metadata":3453862223},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,2,5,7,4,2],"upper_bound":64},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-727384064,"num_objects":-11099,"num_object_clones":0,"num_object_copies":-33297,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-1372,"num_objects_unfound":0,"num_objects_dirty":-11099,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":11099,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.003081"},"pg_stats":[{"pgid":"3.1f","version":"0'0","reported_seq":720,"reported_epoch":383,"state":"active+clean","last_fresh":"2024-09-18T08:17:44.418117+0000","last_change":"2024-09-18T08:15:41.451370+0000","last_active":"2024-09-18T08:17:44.418117+0000","last_peered":"2024-09-18T08:17:44.418117+0000","last_clean":"2024-09-18T08:17:44.418117+0000","last_became_active":"2024-09-18T08:15:41.451012+0000","last_became_peered":"2024-09-18T08:15:41.451012+0000","last_unstale":"2024-09-18T08:17:44.418117+0000","last_undegraded":"2024-09-18T08:17:44.418117+0000","last_fullsized":"2024-09-18T08:17:44.418117+0000","mapping_epoch":336,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":337,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:14:29.060206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:46.860092+0000","last_clean_scrub_stamp":"2024-09-18T08:14:29.060206+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"271'15","reported_seq":451,"reported_epoch":382,"state":"active+clean","last_fresh":"2024-09-18T08:17:43.530314+0000","last_change":"2024-09-18T08:16:13.499531+0000","last_active":"2024-09-18T08:17:43.530314+0000","last_peered":"2024-09-18T08:17:43.530314+0000","last_clean":"2024-09-18T08:17:43.530314+0000","last_became_active":"2024-09-18T08:16:13.456332+0000","last_became_peered":"2024-09-18T08:16:13.456332+0000","last_unstale":"2024-09-18T08:17:43.530314+0000","last_undegraded":"2024-09-18T08:17:43.530314+0000","last_fullsized":"2024-09-18T08:17:43.530314+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"271'15","last_scrub_stamp":"2024-09-18T08:15:04.676980+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_clean_scrub_stamp":"2024-09-18T08:15:04.676980+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":25,"num_read_kb":19,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,3],"acting":[2,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":533,"reported_epoch":381,"state":"active+clean","last_fresh":"2024-09-18T08:17:33.479596+0000","last_change":"2024-09-18T08:15:41.440121+0000","last_active":"2024-09-18T08:17:33.479596+0000","last_peered":"2024-09-18T08:17:33.479596+0000","last_clean":"2024-09-18T08:17:33.479596+0000","last_became_active":"2024-09-18T08:15:41.438512+0000","last_became_peered":"2024-09-18T08:15:41.438512+0000","last_unstale":"2024-09-18T08:17:33.479596+0000","last_undegraded":"2024-09-18T08:17:33.479596+0000","last_fullsized":"2024-09-18T08:17:33.479596+0000","mapping_epoch":336,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":337,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:15:12.171225+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:50.886830+0000","last_clean_scrub_stamp":"2024-09-18T08:15:12.171225+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,2],"acting":[1,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":434,"reported_epoch":382,"state":"active+clean","last_fresh":"2024-09-18T08:17:43.528081+0000","last_change":"2024-09-18T08:17:40.565146+0000","last_active":"2024-09-18T08:17:43.528081+0000","last_peered":"2024-09-18T08:17:43.528081+0000","last_clean":"2024-09-18T08:17:43.528081+0000","last_became_active":"2024-09-18T08:16:13.488969+0000","last_became_peered":"2024-09-18T08:16:13.488969+0000","last_unstale":"2024-09-18T08:17:43.528081+0000","last_undegraded":"2024-09-18T08:17:43.528081+0000","last_fullsized":"2024-09-18T08:17:43.528081+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:17:40.565100+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_clean_scrub_stamp":"2024-09-18T08:17:40.565100+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:19:01.428914+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":711,"reported_epoch":382,"state":"active+clean","last_fresh":"2024-09-18T08:17:43.547906+0000","last_change":"2024-09-18T08:15:51.826092+0000","last_active":"2024-09-18T08:17:43.547906+0000","last_peered":"2024-09-18T08:17:43.547906+0000","last_clean":"2024-09-18T08:17:43.547906+0000","last_became_active":"2024-09-18T08:14:40.357298+0000","last_became_peered":"2024-09-18T08:14:40.357298+0000","last_unstale":"2024-09-18T08:17:43.547906+0000","last_undegraded":"2024-09-18T08:17:43.547906+0000","last_fullsized":"2024-09-18T08:17:43.547906+0000","mapping_epoch":308,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":309,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:15:51.826050+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_clean_scrub_stamp":"2024-09-18T08:15:51.826050+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,1],"acting":[3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":721,"reported_epoch":382,"state":"active+clean","last_fresh":"2024-09-18T08:17:43.548933+0000","last_change":"2024-09-18T08:17:38.359429+0000","last_active":"2024-09-18T08:17:43.548933+0000","last_peered":"2024-09-18T08:17:43.548933+0000","last_clean":"2024-09-18T08:17:43.548933+0000","last_became_active":"2024-09-18T08:06:24.126194+0000","last_became_peered":"2024-09-18T08:06:24.126194+0000","last_unstale":"2024-09-18T08:17:43.548933+0000","last_undegraded":"2024-09-18T08:17:43.548933+0000","last_fullsized":"2024-09-18T08:17:43.548933+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:17:38.359366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:46.860092+0000","last_clean_scrub_stamp":"2024-09-18T08:17:38.359366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:19:07.133806+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"271'27","reported_seq":713,"reported_epoch":382,"state":"active+clean","last_fresh":"2024-09-18T08:17:43.545335+0000","last_change":"2024-09-18T08:14:26.565324+0000","last_active":"2024-09-18T08:17:43.545335+0000","last_peered":"2024-09-18T08:17:43.545335+0000","last_clean":"2024-09-18T08:17:43.545335+0000","last_became_active":"2024-09-18T08:02:49.886106+0000","last_became_peered":"2024-09-18T08:02:49.886106+0000","last_unstale":"2024-09-18T08:17:43.545335+0000","last_undegraded":"2024-09-18T08:17:43.545335+0000","last_fullsized":"2024-09-18T08:17:43.545335+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"271'27","last_scrub_stamp":"2024-09-18T08:14:26.565071+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_clean_scrub_stamp":"2024-09-18T08:14:26.565071+0000","objects_scrubbed":7,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":44,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":344,"reported_epoch":381,"state":"active+clean","last_fresh":"2024-09-18T08:17:33.478070+0000","last_change":"2024-09-18T08:16:13.483763+0000","last_active":"2024-09-18T08:17:33.478070+0000","last_peered":"2024-09-18T08:17:33.478070+0000","last_clean":"2024-09-18T08:17:33.478070+0000","last_became_active":"2024-09-18T08:16:13.474963+0000","last_became_peered":"2024-09-18T08:16:13.474963+0000","last_unstale":"2024-09-18T08:17:33.478070+0000","last_undegraded":"2024-09-18T08:17:33.478070+0000","last_fullsized":"2024-09-18T08:17:33.478070+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:15:44.585123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:50.886830+0000","last_clean_scrub_stamp":"2024-09-18T08:15:44.585123+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":109,"reported_epoch":382,"state":"active+clean","last_fresh":"2024-09-18T08:17:43.529188+0000","last_change":"2024-09-18T08:16:13.459018+0000","last_active":"2024-09-18T08:17:43.529188+0000","last_peered":"2024-09-18T08:17:43.529188+0000","last_clean":"2024-09-18T08:17:43.529188+0000","last_became_active":"2024-09-18T08:16:13.458654+0000","last_became_peered":"2024-09-18T08:16:13.458654+0000","last_unstale":"2024-09-18T08:17:43.529188+0000","last_undegraded":"2024-09-18T08:17:43.529188+0000","last_fullsized":"2024-09-18T08:17:43.529188+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:10:51.225974+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:46.860092+0000","last_clean_scrub_stamp":"2024-09-18T08:10:51.225974+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:18:13.341070+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,1],"acting":[5,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"271'15","reported_seq":313,"reported_epoch":382,"state":"active+clean","last_fresh":"2024-09-18T08:17:43.546333+0000","last_change":"2024-09-18T08:16:13.499713+0000","last_active":"2024-09-18T08:17:43.546333+0000","last_peered":"2024-09-18T08:17:43.546333+0000","last_clean":"2024-09-18T08:17:43.546333+0000","last_became_active":"2024-09-18T08:16:13.463397+0000","last_became_peered":"2024-09-18T08:16:13.463397+0000","last_unstale":"2024-09-18T08:17:43.546333+0000","last_undegraded":"2024-09-18T08:17:43.546333+0000","last_fullsized":"2024-09-18T08:17:43.546333+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'9","last_scrub_stamp":"2024-09-18T08:12:00.466202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_clean_scrub_stamp":"2024-09-18T08:12:00.466202+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:18:24.407823+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":24,"num_read_kb":18,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":117,"reported_epoch":382,"state":"active+clean","last_fresh":"2024-09-18T08:17:43.529352+0000","last_change":"2024-09-18T08:17:30.548045+0000","last_active":"2024-09-18T08:17:43.529352+0000","last_peered":"2024-09-18T08:17:43.529352+0000","last_clean":"2024-09-18T08:17:43.529352+0000","last_became_active":"2024-09-18T08:16:13.460761+0000","last_became_peered":"2024-09-18T08:16:13.460761+0000","last_unstale":"2024-09-18T08:17:43.529352+0000","last_undegraded":"2024-09-18T08:17:43.529352+0000","last_fullsized":"2024-09-18T08:17:43.529352+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:17:30.547979+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:50.886830+0000","last_clean_scrub_stamp":"2024-09-18T08:17:30.547979+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:18:36.571323+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,6],"acting":[5,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":688,"reported_epoch":383,"state":"active+clean","last_fresh":"2024-09-18T08:17:44.417784+0000","last_change":"2024-09-18T08:17:29.542358+0000","last_active":"2024-09-18T08:17:44.417784+0000","last_peered":"2024-09-18T08:17:44.417784+0000","last_clean":"2024-09-18T08:17:44.417784+0000","last_became_active":"2024-09-18T08:14:40.354651+0000","last_became_peered":"2024-09-18T08:14:40.354651+0000","last_unstale":"2024-09-18T08:17:44.417784+0000","last_undegraded":"2024-09-18T08:17:44.417784+0000","last_fullsized":"2024-09-18T08:17:44.417784+0000","mapping_epoch":308,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":309,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:17:29.542288+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_clean_scrub_stamp":"2024-09-18T08:17:29.542288+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:18:42.683374+0000","scrub_duration":18,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":109,"reported_epoch":382,"state":"active+clean","last_fresh":"2024-09-18T08:17:43.528583+0000","last_change":"2024-09-18T08:16:13.471332+0000","last_active":"2024-09-18T08:17:43.528583+0000","last_peered":"2024-09-18T08:17:43.528583+0000","last_clean":"2024-09-18T08:17:43.528583+0000","last_became_active":"2024-09-18T08:16:13.471003+0000","last_became_peered":"2024-09-18T08:16:13.471003+0000","last_unstale":"2024-09-18T08:17:43.528583+0000","last_undegraded":"2024-09-18T08:17:43.528583+0000","last_fullsized":"2024-09-18T08:17:43.528583+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:15:01.636504+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:46.860092+0000","last_clean_scrub_stamp":"2024-09-18T08:15:01.636504+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1b","version":"271'7","reported_seq":295,"reported_epoch":382,"state":"active+clean","last_fresh":"2024-09-18T08:17:43.548288+0000","last_change":"2024-09-18T08:16:13.514798+0000","last_active":"2024-09-18T08:17:43.548288+0000","last_peered":"2024-09-18T08:17:43.548288+0000","last_clean":"2024-09-18T08:17:43.548288+0000","last_became_active":"2024-09-18T08:16:13.473146+0000","last_became_peered":"2024-09-18T08:16:13.473146+0000","last_unstale":"2024-09-18T08:17:43.548288+0000","last_undegraded":"2024-09-18T08:17:43.548288+0000","last_fullsized":"2024-09-18T08:17:43.548288+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"271'7","last_scrub_stamp":"2024-09-18T08:15:13.551351+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_clean_scrub_stamp":"2024-09-18T08:15:13.551351+0000","objects_scrubbed":2,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":12,"num_read_kb":8,"num_write":8,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":548,"reported_epoch":381,"state":"active+clean","last_fresh":"2024-09-18T08:17:33.479038+0000","last_change":"2024-09-18T08:16:13.487519+0000","last_active":"2024-09-18T08:17:33.479038+0000","last_peered":"2024-09-18T08:17:33.479038+0000","last_clean":"2024-09-18T08:17:33.479038+0000","last_became_active":"2024-09-18T08:16:13.486136+0000","last_became_peered":"2024-09-18T08:16:13.486136+0000","last_unstale":"2024-09-18T08:17:33.479038+0000","last_undegraded":"2024-09-18T08:17:33.479038+0000","last_fullsized":"2024-09-18T08:17:33.479038+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:15:40.589048+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:50.886830+0000","last_clean_scrub_stamp":"2024-09-18T08:15:40.589048+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,6],"acting":[1,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":102,"reported_epoch":382,"state":"active+clean","last_fresh":"2024-09-18T08:17:43.528182+0000","last_change":"2024-09-18T08:15:41.444316+0000","last_active":"2024-09-18T08:17:43.528182+0000","last_peered":"2024-09-18T08:17:43.528182+0000","last_clean":"2024-09-18T08:17:43.528182+0000","last_became_active":"2024-09-18T08:15:41.441932+0000","last_became_peered":"2024-09-18T08:15:41.441932+0000","last_unstale":"2024-09-18T08:17:43.528182+0000","last_undegraded":"2024-09-18T08:17:43.528182+0000","last_fullsized":"2024-09-18T08:17:43.528182+0000","mapping_epoch":336,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":337,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:11:01.108648+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_clean_scrub_stamp":"2024-09-18T08:11:01.108648+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:18:18.451529+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":416,"reported_epoch":382,"state":"active+clean","last_fresh":"2024-09-18T08:17:43.528570+0000","last_change":"2024-09-18T08:16:13.453737+0000","last_active":"2024-09-18T08:17:43.528570+0000","last_peered":"2024-09-18T08:17:43.528570+0000","last_clean":"2024-09-18T08:17:43.528570+0000","last_became_active":"2024-09-18T08:16:13.453129+0000","last_became_peered":"2024-09-18T08:16:13.453129+0000","last_unstale":"2024-09-18T08:17:43.528570+0000","last_undegraded":"2024-09-18T08:17:43.528570+0000","last_fullsized":"2024-09-18T08:17:43.528570+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:15:42.600840+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_clean_scrub_stamp":"2024-09-18T08:15:42.600840+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,3],"acting":[5,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":715,"reported_epoch":383,"state":"active+clean","last_fresh":"2024-09-18T08:17:44.425700+0000","last_change":"2024-09-18T08:16:13.477657+0000","last_active":"2024-09-18T08:17:44.425700+0000","last_peered":"2024-09-18T08:17:44.425700+0000","last_clean":"2024-09-18T08:17:44.425700+0000","last_became_active":"2024-09-18T08:16:13.477401+0000","last_became_peered":"2024-09-18T08:16:13.477401+0000","last_unstale":"2024-09-18T08:17:44.425700+0000","last_undegraded":"2024-09-18T08:17:44.425700+0000","last_fullsized":"2024-09-18T08:17:44.425700+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:12:40.711507+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:46.860092+0000","last_clean_scrub_stamp":"2024-09-18T08:12:40.711507+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,3],"acting":[0,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1c","version":"271'27","reported_seq":449,"reported_epoch":382,"state":"active+clean","last_fresh":"2024-09-18T08:17:43.535397+0000","last_change":"2024-09-18T08:14:40.711299+0000","last_active":"2024-09-18T08:17:43.535397+0000","last_peered":"2024-09-18T08:17:43.535397+0000","last_clean":"2024-09-18T08:17:43.535397+0000","last_became_active":"2024-09-18T08:14:39.480913+0000","last_became_peered":"2024-09-18T08:14:39.480913+0000","last_unstale":"2024-09-18T08:17:43.535397+0000","last_undegraded":"2024-09-18T08:17:43.535397+0000","last_fullsized":"2024-09-18T08:17:43.535397+0000","mapping_epoch":307,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":308,"parent":"0.0","parent_split_bits":0,"last_scrub":"271'27","last_scrub_stamp":"2024-09-18T08:14:40.711228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_clean_scrub_stamp":"2024-09-18T08:14:40.711228+0000","objects_scrubbed":7,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":44,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":569,"reported_epoch":381,"state":"active+clean","last_fresh":"2024-09-18T08:17:36.633467+0000","last_change":"2024-09-18T08:17:36.633467+0000","last_active":"2024-09-18T08:17:36.633467+0000","last_peered":"2024-09-18T08:17:36.633467+0000","last_clean":"2024-09-18T08:17:36.633467+0000","last_became_active":"2024-09-18T08:16:13.462303+0000","last_became_peered":"2024-09-18T08:16:13.462303+0000","last_unstale":"2024-09-18T08:17:36.633467+0000","last_undegraded":"2024-09-18T08:17:36.633467+0000","last_fullsized":"2024-09-18T08:17:36.633467+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:17:36.633392+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:50.886830+0000","last_clean_scrub_stamp":"2024-09-18T08:17:36.633392+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:18:36.654140+0000","scrub_duration":20,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,5],"acting":[1,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":699,"reported_epoch":382,"state":"active+clean","last_fresh":"2024-09-18T08:17:43.549209+0000","last_change":"2024-09-18T08:15:41.469278+0000","last_active":"2024-09-18T08:17:43.549209+0000","last_peered":"2024-09-18T08:17:43.549209+0000","last_clean":"2024-09-18T08:17:43.549209+0000","last_became_active":"2024-09-18T08:15:41.466371+0000","last_became_peered":"2024-09-18T08:15:41.466371+0000","last_unstale":"2024-09-18T08:17:43.549209+0000","last_undegraded":"2024-09-18T08:17:43.549209+0000","last_fullsized":"2024-09-18T08:17:43.549209+0000","mapping_epoch":336,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":337,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:15:05.719269+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_clean_scrub_stamp":"2024-09-18T08:15:05.719269+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":344,"reported_epoch":381,"state":"active+clean","last_fresh":"2024-09-18T08:17:33.479213+0000","last_change":"2024-09-18T08:16:13.487949+0000","last_active":"2024-09-18T08:17:33.479213+0000","last_peered":"2024-09-18T08:17:33.479213+0000","last_clean":"2024-09-18T08:17:33.479213+0000","last_became_active":"2024-09-18T08:16:13.487001+0000","last_became_peered":"2024-09-18T08:16:13.487001+0000","last_unstale":"2024-09-18T08:17:33.479213+0000","last_undegraded":"2024-09-18T08:17:33.479213+0000","last_fullsized":"2024-09-18T08:17:33.479213+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:14:40.774086+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:46.860092+0000","last_clean_scrub_stamp":"2024-09-18T08:14:40.774086+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1d","version":"271'18","reported_seq":802,"reported_epoch":382,"state":"active+clean","last_fresh":"2024-09-18T08:17:43.550802+0000","last_change":"2024-09-18T08:17:04.627991+0000","last_active":"2024-09-18T08:17:43.550802+0000","last_peered":"2024-09-18T08:17:43.550802+0000","last_clean":"2024-09-18T08:17:43.550802+0000","last_became_active":"2024-09-18T08:16:13.473248+0000","last_became_peered":"2024-09-18T08:16:13.473248+0000","last_unstale":"2024-09-18T08:17:43.550802+0000","last_undegraded":"2024-09-18T08:17:43.550802+0000","last_fullsized":"2024-09-18T08:17:43.550802+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"271'18","last_scrub_stamp":"2024-09-18T08:17:04.627913+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_clean_scrub_stamp":"2024-09-18T08:17:04.627913+0000","objects_scrubbed":5,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:18:07.331175+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":34,"num_read_kb":25,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":660,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":107,"reported_epoch":382,"state":"active+clean","last_fresh":"2024-09-18T08:17:43.551838+0000","last_change":"2024-09-18T08:16:13.485702+0000","last_active":"2024-09-18T08:17:43.551838+0000","last_peered":"2024-09-18T08:17:43.551838+0000","last_clean":"2024-09-18T08:17:43.551838+0000","last_became_active":"2024-09-18T08:16:13.485316+0000","last_became_peered":"2024-09-18T08:16:13.485316+0000","last_unstale":"2024-09-18T08:17:43.551838+0000","last_undegraded":"2024-09-18T08:17:43.551838+0000","last_fullsized":"2024-09-18T08:17:43.551838+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:10:46.303848+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:50.886830+0000","last_clean_scrub_stamp":"2024-09-18T08:10:46.303848+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:18:16.412467+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,1],"acting":[3,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.1c","version":"69'1","reported_seq":106,"reported_epoch":382,"state":"active+clean","last_fresh":"2024-09-18T08:17:43.528681+0000","last_change":"2024-09-18T08:15:41.489871+0000","last_active":"2024-09-18T08:17:43.528681+0000","last_peered":"2024-09-18T08:17:43.528681+0000","last_clean":"2024-09-18T08:17:43.528681+0000","last_became_active":"2024-09-18T08:15:41.466665+0000","last_became_peered":"2024-09-18T08:15:41.466665+0000","last_unstale":"2024-09-18T08:17:43.528681+0000","last_undegraded":"2024-09-18T08:17:43.528681+0000","last_fullsized":"2024-09-18T08:17:43.528681+0000","mapping_epoch":336,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":337,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'1","last_scrub_stamp":"2024-09-18T08:15:06.708980+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_clean_scrub_stamp":"2024-09-18T08:15:06.708980+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":403,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":806,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,3],"acting":[5,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"62'1","reported_seq":582,"reported_epoch":381,"state":"active+clean","last_fresh":"2024-09-18T08:17:37.618000+0000","last_change":"2024-09-18T08:17:37.618000+0000","last_active":"2024-09-18T08:17:37.618000+0000","last_peered":"2024-09-18T08:17:37.618000+0000","last_clean":"2024-09-18T08:17:37.618000+0000","last_became_active":"2024-09-18T08:16:13.483397+0000","last_became_peered":"2024-09-18T08:16:13.483397+0000","last_unstale":"2024-09-18T08:17:37.618000+0000","last_undegraded":"2024-09-18T08:17:37.618000+0000","last_fullsized":"2024-09-18T08:17:37.618000+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"62'1","last_scrub_stamp":"2024-09-18T08:17:37.617954+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:46.860092+0000","last_clean_scrub_stamp":"2024-09-18T08:17:37.617954+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:18:50.504684+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":46,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,6],"acting":[1,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1e","version":"271'20","reported_seq":734,"reported_epoch":383,"state":"active+clean","last_fresh":"2024-09-18T08:17:44.423125+0000","last_change":"2024-09-18T08:15:40.251794+0000","last_active":"2024-09-18T08:17:44.423125+0000","last_peered":"2024-09-18T08:17:44.423125+0000","last_clean":"2024-09-18T08:17:44.423125+0000","last_became_active":"2024-09-18T08:14:40.359879+0000","last_became_peered":"2024-09-18T08:14:40.359879+0000","last_unstale":"2024-09-18T08:17:44.423125+0000","last_undegraded":"2024-09-18T08:17:44.423125+0000","last_fullsized":"2024-09-18T08:17:44.423125+0000","mapping_epoch":308,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":309,"parent":"0.0","parent_split_bits":0,"last_scrub":"271'20","last_scrub_stamp":"2024-09-18T08:15:40.251706+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_clean_scrub_stamp":"2024-09-18T08:15:40.251706+0000","objects_scrubbed":5,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":25,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1],"acting":[0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1f","version":"349'15","reported_seq":425,"reported_epoch":382,"state":"active+clean","last_fresh":"2024-09-18T08:17:43.535643+0000","last_change":"2024-09-18T08:17:29.878587+0000","last_active":"2024-09-18T08:17:43.535643+0000","last_peered":"2024-09-18T08:17:43.535643+0000","last_clean":"2024-09-18T08:17:43.535643+0000","last_became_active":"2024-09-18T08:16:13.467898+0000","last_became_peered":"2024-09-18T08:16:13.467898+0000","last_unstale":"2024-09-18T08:17:43.535643+0000","last_undegraded":"2024-09-18T08:17:43.535643+0000","last_fullsized":"2024-09-18T08:17:43.535643+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"349'15","last_scrub_stamp":"2024-09-18T08:17:29.878376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:50.886830+0000","last_clean_scrub_stamp":"2024-09-18T08:17:29.878376+0000","objects_scrubbed":1,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:18:46.725267+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,1],"acting":[2,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.1","version":"383'11158","reported_seq":19671,"reported_epoch":383,"state":"active+clean","last_fresh":"2024-09-18T08:17:44.827846+0000","last_change":"2024-09-18T08:17:25.434104+0000","last_active":"2024-09-18T08:17:44.827846+0000","last_peered":"2024-09-18T08:17:44.827846+0000","last_clean":"2024-09-18T08:17:44.827846+0000","last_became_active":"2024-09-18T08:17:25.417680+0000","last_became_peered":"2024-09-18T08:17:25.417680+0000","last_unstale":"2024-09-18T08:17:44.827846+0000","last_undegraded":"2024-09-18T08:17:44.827846+0000","last_fullsized":"2024-09-18T08:17:44.827846+0000","mapping_epoch":376,"log_start":"358'7411","ondisk_log_start":"358'7411","created":311,"last_epoch_clean":377,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:14:43.412748+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:14:43.412748+0000","last_clean_scrub_stamp":"2024-09-18T08:14:43.412748+0000","objects_scrubbed":0,"log_size":3747,"log_dups_size":0,"ondisk_log_size":3747,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":119537664,"num_objects":1824,"num_object_clones":0,"num_object_copies":5472,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1824,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11158,"num_write_kb":415424,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4169,"num_bytes_recovered":273219584,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.a","version":"74'79","reported_seq":110,"reported_epoch":382,"state":"active+clean","last_fresh":"2024-09-18T08:17:43.528709+0000","last_change":"2024-09-18T08:15:42.288426+0000","last_active":"2024-09-18T08:17:43.528709+0000","last_peered":"2024-09-18T08:17:43.528709+0000","last_clean":"2024-09-18T08:17:43.528709+0000","last_became_active":"2024-09-18T08:15:41.443376+0000","last_became_peered":"2024-09-18T08:15:41.443376+0000","last_unstale":"2024-09-18T08:17:43.528709+0000","last_undegraded":"2024-09-18T08:17:43.528709+0000","last_fullsized":"2024-09-18T08:17:43.528709+0000","mapping_epoch":336,"log_start":"0'0","ondisk_log_start":"0'0","created":152,"last_epoch_clean":337,"parent":"0.0","parent_split_bits":4,"last_scrub":"74'79","last_scrub_stamp":"2024-09-18T08:15:42.288363+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:00:11.893742+0000","last_clean_scrub_stamp":"2024-09-18T08:15:42.288363+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,6],"acting":[5,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.f","version":"271'27","reported_seq":638,"reported_epoch":381,"state":"active+clean","last_fresh":"2024-09-18T08:17:33.478034+0000","last_change":"2024-09-18T08:17:29.732577+0000","last_active":"2024-09-18T08:17:33.478034+0000","last_peered":"2024-09-18T08:17:33.478034+0000","last_clean":"2024-09-18T08:17:33.478034+0000","last_became_active":"2024-09-18T08:16:13.456369+0000","last_became_peered":"2024-09-18T08:16:13.456369+0000","last_unstale":"2024-09-18T08:17:33.478034+0000","last_undegraded":"2024-09-18T08:17:33.478034+0000","last_fullsized":"2024-09-18T08:17:33.478034+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"271'27","last_scrub_stamp":"2024-09-18T08:17:29.732496+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_clean_scrub_stamp":"2024-09-18T08:17:29.732496+0000","objects_scrubbed":7,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:18:58.561148+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":43,"num_read_kb":34,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,5],"acting":[1,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":705,"reported_epoch":382,"state":"active+clean","last_fresh":"2024-09-18T08:17:43.545067+0000","last_change":"2024-09-18T08:16:13.472791+0000","last_active":"2024-09-18T08:17:43.545067+0000","last_peered":"2024-09-18T08:17:43.545067+0000","last_clean":"2024-09-18T08:17:43.545067+0000","last_became_active":"2024-09-18T08:16:13.472353+0000","last_became_peered":"2024-09-18T08:16:13.472353+0000","last_unstale":"2024-09-18T08:17:43.545067+0000","last_undegraded":"2024-09-18T08:17:43.545067+0000","last_fullsized":"2024-09-18T08:17:43.545067+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:15:46.809403+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:46.860092+0000","last_clean_scrub_stamp":"2024-09-18T08:15:46.809403+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,5],"acting":[3,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.e","version":"349'16","reported_seq":568,"reported_epoch":382,"state":"active+clean","last_fresh":"2024-09-18T08:17:43.528613+0000","last_change":"2024-09-18T08:17:37.704743+0000","last_active":"2024-09-18T08:17:43.528613+0000","last_peered":"2024-09-18T08:17:43.528613+0000","last_clean":"2024-09-18T08:17:43.528613+0000","last_became_active":"2024-09-18T08:16:13.451199+0000","last_became_peered":"2024-09-18T08:16:13.451199+0000","last_unstale":"2024-09-18T08:17:43.528613+0000","last_undegraded":"2024-09-18T08:17:43.528613+0000","last_fullsized":"2024-09-18T08:17:43.528613+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"349'16","last_scrub_stamp":"2024-09-18T08:17:37.704683+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:50.886830+0000","last_clean_scrub_stamp":"2024-09-18T08:17:37.704683+0000","objects_scrubbed":1,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:18:50.732146+0000","scrub_duration":71,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,2],"acting":[5,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":102,"reported_epoch":382,"state":"active+clean","last_fresh":"2024-09-18T08:17:43.528923+0000","last_change":"2024-09-18T08:15:41.438999+0000","last_active":"2024-09-18T08:17:43.528923+0000","last_peered":"2024-09-18T08:17:43.528923+0000","last_clean":"2024-09-18T08:17:43.528923+0000","last_became_active":"2024-09-18T08:15:41.438610+0000","last_became_peered":"2024-09-18T08:15:41.438610+0000","last_unstale":"2024-09-18T08:17:43.528923+0000","last_undegraded":"2024-09-18T08:17:43.528923+0000","last_fullsized":"2024-09-18T08:17:43.528923+0000","mapping_epoch":336,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":337,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:15:00.180309+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_clean_scrub_stamp":"2024-09-18T08:15:00.180309+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.e","version":"382'10763","reported_seq":10931,"reported_epoch":382,"state":"active+remapped+backfill_toofull","last_fresh":"2024-09-18T08:17:44.051574+0000","last_change":"2024-09-18T08:17:14.523313+0000","last_active":"2024-09-18T08:17:44.051574+0000","last_peered":"2024-09-18T08:17:44.051574+0000","last_clean":"2024-09-18T08:16:12.353466+0000","last_became_active":"2024-09-18T08:16:14.500187+0000","last_became_peered":"2024-09-18T08:16:14.500187+0000","last_unstale":"2024-09-18T08:17:44.051574+0000","last_undegraded":"2024-09-18T08:17:44.051574+0000","last_fullsized":"2024-09-18T08:17:44.051574+0000","mapping_epoch":349,"log_start":"350'6100","ondisk_log_start":"350'6100","created":311,"last_epoch_clean":312,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:14:43.412748+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:14:43.412748+0000","last_clean_scrub_stamp":"2024-09-18T08:14:43.412748+0000","objects_scrubbed":0,"log_size":4663,"log_dups_size":0,"ondisk_log_size":4663,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":114098176,"num_objects":1741,"num_object_clones":0,"num_object_copies":5223,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1741,"num_objects_unfound":0,"num_objects_dirty":1741,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10763,"num_write_kb":400128,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,5],"acting":[2,6,4],"avail_no_missing":["2","4","6"],"object_location_counts":[{"shards":"2,4,6","objects":1741}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.5","version":"74'79","reported_seq":130,"reported_epoch":382,"state":"active+clean","last_fresh":"2024-09-18T08:17:43.537678+0000","last_change":"2024-09-18T08:16:13.445824+0000","last_active":"2024-09-18T08:17:43.537678+0000","last_peered":"2024-09-18T08:17:43.537678+0000","last_clean":"2024-09-18T08:17:43.537678+0000","last_became_active":"2024-09-18T08:16:13.445467+0000","last_became_peered":"2024-09-18T08:16:13.445467+0000","last_unstale":"2024-09-18T08:17:43.537678+0000","last_undegraded":"2024-09-18T08:17:43.537678+0000","last_fullsized":"2024-09-18T08:17:43.537678+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":152,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":3,"last_scrub":"74'79","last_scrub_stamp":"2024-09-18T08:15:58.701823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:00:11.893742+0000","last_clean_scrub_stamp":"2024-09-18T08:15:58.701823+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,5],"acting":[2,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.0","version":"271'30","reported_seq":775,"reported_epoch":382,"state":"active+clean","last_fresh":"2024-09-18T08:17:43.544841+0000","last_change":"2024-09-18T08:15:57.927197+0000","last_active":"2024-09-18T08:17:43.544841+0000","last_peered":"2024-09-18T08:17:43.544841+0000","last_clean":"2024-09-18T08:17:43.544841+0000","last_became_active":"2024-09-18T08:14:40.358956+0000","last_became_peered":"2024-09-18T08:14:40.358956+0000","last_unstale":"2024-09-18T08:17:43.544841+0000","last_undegraded":"2024-09-18T08:17:43.544841+0000","last_fullsized":"2024-09-18T08:17:43.544841+0000","mapping_epoch":308,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":309,"parent":"0.0","parent_split_bits":0,"last_scrub":"271'30","last_scrub_stamp":"2024-09-18T08:15:57.927111+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_clean_scrub_stamp":"2024-09-18T08:15:57.927111+0000","objects_scrubbed":8,"log_size":30,"log_dups_size":0,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":18,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":52,"num_read_kb":40,"num_write":32,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":727,"reported_epoch":382,"state":"active+clean","last_fresh":"2024-09-18T08:17:43.552682+0000","last_change":"2024-09-18T08:15:41.461616+0000","last_active":"2024-09-18T08:17:43.552682+0000","last_peered":"2024-09-18T08:17:43.552682+0000","last_clean":"2024-09-18T08:17:43.552682+0000","last_became_active":"2024-09-18T08:15:41.461057+0000","last_became_peered":"2024-09-18T08:15:41.461057+0000","last_unstale":"2024-09-18T08:17:43.552682+0000","last_undegraded":"2024-09-18T08:17:43.552682+0000","last_fullsized":"2024-09-18T08:17:43.552682+0000","mapping_epoch":336,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":337,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:15:08.780913+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:46.860092+0000","last_clean_scrub_stamp":"2024-09-18T08:15:08.780913+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":336,"reported_epoch":382,"state":"active+clean","last_fresh":"2024-09-18T08:17:43.549568+0000","last_change":"2024-09-18T08:17:31.417637+0000","last_active":"2024-09-18T08:17:43.549568+0000","last_peered":"2024-09-18T08:17:43.549568+0000","last_clean":"2024-09-18T08:17:43.549568+0000","last_became_active":"2024-09-18T08:16:13.478687+0000","last_became_peered":"2024-09-18T08:16:13.478687+0000","last_unstale":"2024-09-18T08:17:43.549568+0000","last_undegraded":"2024-09-18T08:17:43.549568+0000","last_fullsized":"2024-09-18T08:17:43.549568+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:17:31.417453+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:50.886830+0000","last_clean_scrub_stamp":"2024-09-18T08:17:31.417453+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:18:55.886528+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":108,"reported_epoch":382,"state":"active+clean","last_fresh":"2024-09-18T08:17:43.546245+0000","last_change":"2024-09-18T08:16:13.461117+0000","last_active":"2024-09-18T08:17:43.546245+0000","last_peered":"2024-09-18T08:17:43.546245+0000","last_clean":"2024-09-18T08:17:43.546245+0000","last_became_active":"2024-09-18T08:16:13.460785+0000","last_became_peered":"2024-09-18T08:16:13.460785+0000","last_unstale":"2024-09-18T08:17:43.546245+0000","last_undegraded":"2024-09-18T08:17:43.546245+0000","last_fullsized":"2024-09-18T08:17:43.546245+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:12:02.473168+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_clean_scrub_stamp":"2024-09-18T08:12:02.473168+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:18:26.348422+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,6],"acting":[3,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.f","version":"382'11086","reported_seq":11240,"reported_epoch":382,"state":"active+clean","last_fresh":"2024-09-18T08:17:44.308193+0000","last_change":"2024-09-18T08:14:44.437378+0000","last_active":"2024-09-18T08:17:44.308193+0000","last_peered":"2024-09-18T08:17:44.308193+0000","last_clean":"2024-09-18T08:17:44.308193+0000","last_became_active":"2024-09-18T08:14:44.437103+0000","last_became_peered":"2024-09-18T08:14:44.437103+0000","last_unstale":"2024-09-18T08:17:44.308193+0000","last_undegraded":"2024-09-18T08:17:44.308193+0000","last_fullsized":"2024-09-18T08:17:44.308193+0000","mapping_epoch":311,"log_start":"360'7601","ondisk_log_start":"360'7601","created":311,"last_epoch_clean":312,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:14:43.412748+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:14:43.412748+0000","last_clean_scrub_stamp":"2024-09-18T08:14:43.412748+0000","objects_scrubbed":0,"log_size":3485,"log_dups_size":0,"ondisk_log_size":3485,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":119013376,"num_objects":1816,"num_object_clones":0,"num_object_copies":5448,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1816,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11086,"num_write_kb":412864,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.4","version":"74'79","reported_seq":550,"reported_epoch":382,"state":"active+clean","last_fresh":"2024-09-18T08:17:43.548102+0000","last_change":"2024-09-18T08:15:41.898345+0000","last_active":"2024-09-18T08:17:43.548102+0000","last_peered":"2024-09-18T08:17:43.548102+0000","last_clean":"2024-09-18T08:17:43.548102+0000","last_became_active":"2024-09-18T08:15:41.464987+0000","last_became_peered":"2024-09-18T08:15:41.464987+0000","last_unstale":"2024-09-18T08:17:43.548102+0000","last_undegraded":"2024-09-18T08:17:43.548102+0000","last_fullsized":"2024-09-18T08:17:43.548102+0000","mapping_epoch":336,"log_start":"0'0","ondisk_log_start":"0'0","created":152,"last_epoch_clean":337,"parent":"0.0","parent_split_bits":3,"last_scrub":"74'79","last_scrub_stamp":"2024-09-18T08:15:41.898294+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:00:11.893742+0000","last_clean_scrub_stamp":"2024-09-18T08:15:41.898294+0000","objects_scrubbed":1,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":327680,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":983040,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"271'28","reported_seq":485,"reported_epoch":382,"state":"active+clean","last_fresh":"2024-09-18T08:17:43.528124+0000","last_change":"2024-09-18T08:16:13.506782+0000","last_active":"2024-09-18T08:17:43.528124+0000","last_peered":"2024-09-18T08:17:43.528124+0000","last_clean":"2024-09-18T08:17:43.528124+0000","last_became_active":"2024-09-18T08:16:13.451205+0000","last_became_peered":"2024-09-18T08:16:13.451205+0000","last_unstale":"2024-09-18T08:17:43.528124+0000","last_undegraded":"2024-09-18T08:17:43.528124+0000","last_fullsized":"2024-09-18T08:17:43.528124+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"271'28","last_scrub_stamp":"2024-09-18T08:15:51.716119+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_clean_scrub_stamp":"2024-09-18T08:15:51.716119+0000","objects_scrubbed":7,"log_size":28,"log_dups_size":0,"ondisk_log_size":28,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":41,"num_read_kb":34,"num_write":28,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.6","version":"62'2","reported_seq":740,"reported_epoch":383,"state":"active+clean","last_fresh":"2024-09-18T08:17:44.429614+0000","last_change":"2024-09-18T08:16:13.508628+0000","last_active":"2024-09-18T08:17:44.429614+0000","last_peered":"2024-09-18T08:17:44.429614+0000","last_clean":"2024-09-18T08:17:44.429614+0000","last_became_active":"2024-09-18T08:16:13.451171+0000","last_became_peered":"2024-09-18T08:16:13.451171+0000","last_unstale":"2024-09-18T08:17:44.429614+0000","last_undegraded":"2024-09-18T08:17:44.429614+0000","last_fullsized":"2024-09-18T08:17:44.429614+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"62'2","last_scrub_stamp":"2024-09-18T08:14:42.145836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:46.860092+0000","last_clean_scrub_stamp":"2024-09-18T08:14:42.145836+0000","objects_scrubbed":2,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1039,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":3117,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,5],"acting":[0,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.0","version":"349'13","reported_seq":909,"reported_epoch":382,"state":"active+clean","last_fresh":"2024-09-18T08:17:43.544628+0000","last_change":"2024-09-18T08:16:13.516195+0000","last_active":"2024-09-18T08:17:43.544628+0000","last_peered":"2024-09-18T08:17:43.544628+0000","last_clean":"2024-09-18T08:17:43.544628+0000","last_became_active":"2024-09-18T08:16:13.477431+0000","last_became_peered":"2024-09-18T08:16:13.477431+0000","last_unstale":"2024-09-18T08:17:43.544628+0000","last_undegraded":"2024-09-18T08:17:43.544628+0000","last_fullsized":"2024-09-18T08:17:43.544628+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"310'12","last_scrub_stamp":"2024-09-18T08:15:49.795095+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:50.886830+0000","last_clean_scrub_stamp":"2024-09-18T08:15:49.795095+0000","objects_scrubbed":1,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":102,"reported_epoch":382,"state":"active+clean","last_fresh":"2024-09-18T08:17:43.529227+0000","last_change":"2024-09-18T08:15:41.441653+0000","last_active":"2024-09-18T08:17:43.529227+0000","last_peered":"2024-09-18T08:17:43.529227+0000","last_clean":"2024-09-18T08:17:43.529227+0000","last_became_active":"2024-09-18T08:15:41.440876+0000","last_became_peered":"2024-09-18T08:15:41.440876+0000","last_unstale":"2024-09-18T08:17:43.529227+0000","last_undegraded":"2024-09-18T08:17:43.529227+0000","last_fullsized":"2024-09-18T08:17:43.529227+0000","mapping_epoch":336,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":337,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:15:14.781554+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_clean_scrub_stamp":"2024-09-18T08:15:14.781554+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.c","version":"383'10855","reported_seq":22139,"reported_epoch":383,"state":"active+clean","last_fresh":"2024-09-18T08:17:44.859352+0000","last_change":"2024-09-18T08:16:52.555960+0000","last_active":"2024-09-18T08:17:44.859352+0000","last_peered":"2024-09-18T08:17:44.859352+0000","last_clean":"2024-09-18T08:17:44.859352+0000","last_became_active":"2024-09-18T08:16:52.532776+0000","last_became_peered":"2024-09-18T08:16:52.532776+0000","last_unstale":"2024-09-18T08:17:44.859352+0000","last_undegraded":"2024-09-18T08:17:44.859352+0000","last_fullsized":"2024-09-18T08:17:44.859352+0000","mapping_epoch":362,"log_start":"358'7148","ondisk_log_start":"358'7148","created":311,"last_epoch_clean":363,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:14:43.412748+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:14:43.412748+0000","last_clean_scrub_stamp":"2024-09-18T08:14:43.412748+0000","objects_scrubbed":0,"log_size":3707,"log_dups_size":0,"ondisk_log_size":3707,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":108986368,"num_objects":1663,"num_object_clones":0,"num_object_copies":4989,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1663,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10855,"num_write_kb":400576,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5557,"num_bytes_recovered":364183552,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,6],"acting":[0,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.7","version":"74'79","reported_seq":533,"reported_epoch":382,"state":"active+clean","last_fresh":"2024-09-18T08:17:43.549563+0000","last_change":"2024-09-18T08:15:44.872544+0000","last_active":"2024-09-18T08:17:43.549563+0000","last_peered":"2024-09-18T08:17:43.549563+0000","last_clean":"2024-09-18T08:17:43.549563+0000","last_became_active":"2024-09-18T08:15:41.466505+0000","last_became_peered":"2024-09-18T08:15:41.466505+0000","last_unstale":"2024-09-18T08:17:43.549563+0000","last_undegraded":"2024-09-18T08:17:43.549563+0000","last_fullsized":"2024-09-18T08:17:43.549563+0000","mapping_epoch":336,"log_start":"0'0","ondisk_log_start":"0'0","created":152,"last_epoch_clean":337,"parent":"0.0","parent_split_bits":3,"last_scrub":"74'79","last_scrub_stamp":"2024-09-18T08:15:44.872332+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:00:11.893742+0000","last_clean_scrub_stamp":"2024-09-18T08:15:44.872332+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.2","version":"271'20","reported_seq":693,"reported_epoch":381,"state":"active+clean","last_fresh":"2024-09-18T08:17:33.479302+0000","last_change":"2024-09-18T08:16:13.525620+0000","last_active":"2024-09-18T08:17:33.479302+0000","last_peered":"2024-09-18T08:17:33.479302+0000","last_clean":"2024-09-18T08:17:33.479302+0000","last_became_active":"2024-09-18T08:16:13.486720+0000","last_became_peered":"2024-09-18T08:16:13.486720+0000","last_unstale":"2024-09-18T08:17:33.479302+0000","last_undegraded":"2024-09-18T08:17:33.479302+0000","last_fullsized":"2024-09-18T08:17:33.479302+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"271'20","last_scrub_stamp":"2024-09-18T08:15:40.251008+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_clean_scrub_stamp":"2024-09-18T08:15:40.251008+0000","objects_scrubbed":5,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":23,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":28,"num_read_kb":23,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":102,"reported_epoch":382,"state":"active+clean","last_fresh":"2024-09-18T08:17:43.529291+0000","last_change":"2024-09-18T08:15:41.452981+0000","last_active":"2024-09-18T08:17:43.529291+0000","last_peered":"2024-09-18T08:17:43.529291+0000","last_clean":"2024-09-18T08:17:43.529291+0000","last_became_active":"2024-09-18T08:15:41.449639+0000","last_became_peered":"2024-09-18T08:15:41.449639+0000","last_unstale":"2024-09-18T08:17:43.529291+0000","last_undegraded":"2024-09-18T08:17:43.529291+0000","last_fullsized":"2024-09-18T08:17:43.529291+0000","mapping_epoch":336,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":337,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:15:39.858184+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:46.860092+0000","last_clean_scrub_stamp":"2024-09-18T08:15:39.858184+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.3","version":"338'14","reported_seq":911,"reported_epoch":383,"state":"active+clean","last_fresh":"2024-09-18T08:17:44.424999+0000","last_change":"2024-09-18T08:15:41.467411+0000","last_active":"2024-09-18T08:17:44.424999+0000","last_peered":"2024-09-18T08:17:44.424999+0000","last_clean":"2024-09-18T08:17:44.424999+0000","last_became_active":"2024-09-18T08:15:41.447667+0000","last_became_peered":"2024-09-18T08:15:41.447667+0000","last_unstale":"2024-09-18T08:17:44.424999+0000","last_undegraded":"2024-09-18T08:17:44.424999+0000","last_fullsized":"2024-09-18T08:17:44.424999+0000","mapping_epoch":336,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":337,"parent":"0.0","parent_split_bits":0,"last_scrub":"317'11","last_scrub_stamp":"2024-09-18T08:14:59.207667+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:50.886830+0000","last_clean_scrub_stamp":"2024-09-18T08:14:59.207667+0000","objects_scrubbed":1,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":703,"reported_epoch":383,"state":"active+clean","last_fresh":"2024-09-18T08:17:44.417945+0000","last_change":"2024-09-18T08:14:26.087160+0000","last_active":"2024-09-18T08:17:44.417945+0000","last_peered":"2024-09-18T08:17:44.417945+0000","last_clean":"2024-09-18T08:17:44.417945+0000","last_became_active":"2024-09-18T08:08:52.700080+0000","last_became_peered":"2024-09-18T08:08:52.700080+0000","last_unstale":"2024-09-18T08:17:44.417945+0000","last_undegraded":"2024-09-18T08:17:44.417945+0000","last_fullsized":"2024-09-18T08:17:44.417945+0000","mapping_epoch":198,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":199,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:14:26.087098+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_clean_scrub_stamp":"2024-09-18T08:14:26.087098+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.d","version":"383'10728","reported_seq":21034,"reported_epoch":383,"state":"active+clean","last_fresh":"2024-09-18T08:17:44.859396+0000","last_change":"2024-09-18T08:17:06.468945+0000","last_active":"2024-09-18T08:17:44.859396+0000","last_peered":"2024-09-18T08:17:44.859396+0000","last_clean":"2024-09-18T08:17:44.859396+0000","last_became_active":"2024-09-18T08:17:06.449951+0000","last_became_peered":"2024-09-18T08:17:06.449951+0000","last_unstale":"2024-09-18T08:17:44.859396+0000","last_undegraded":"2024-09-18T08:17:44.859396+0000","last_fullsized":"2024-09-18T08:17:44.859396+0000","mapping_epoch":370,"log_start":"357'6947","ondisk_log_start":"357'6947","created":311,"last_epoch_clean":371,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:14:43.412748+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:14:43.412748+0000","last_clean_scrub_stamp":"2024-09-18T08:14:43.412748+0000","objects_scrubbed":0,"log_size":3781,"log_dups_size":0,"ondisk_log_size":3781,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":116195352,"num_objects":1774,"num_object_clones":0,"num_object_copies":5322,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1774,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10728,"num_write_kb":400001,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5068,"num_bytes_recovered":332070936,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,3],"acting":[0,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.6","version":"74'79","reported_seq":594,"reported_epoch":383,"state":"active+clean","last_fresh":"2024-09-18T08:17:44.422692+0000","last_change":"2024-09-18T08:14:02.837236+0000","last_active":"2024-09-18T08:17:44.422692+0000","last_peered":"2024-09-18T08:17:44.422692+0000","last_clean":"2024-09-18T08:17:44.422692+0000","last_became_active":"2024-09-18T08:07:26.426532+0000","last_became_peered":"2024-09-18T08:07:26.426532+0000","last_unstale":"2024-09-18T08:17:44.422692+0000","last_undegraded":"2024-09-18T08:17:44.422692+0000","last_fullsized":"2024-09-18T08:17:44.422692+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":152,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":3,"last_scrub":"74'79","last_scrub_stamp":"2024-09-18T08:14:02.837167+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:00:11.893742+0000","last_clean_scrub_stamp":"2024-09-18T08:14:02.837167+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,3],"acting":[0,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.3","version":"271'29","reported_seq":811,"reported_epoch":383,"state":"active+clean","last_fresh":"2024-09-18T08:17:44.425443+0000","last_change":"2024-09-18T08:15:41.493514+0000","last_active":"2024-09-18T08:17:44.425443+0000","last_peered":"2024-09-18T08:17:44.425443+0000","last_clean":"2024-09-18T08:17:44.425443+0000","last_became_active":"2024-09-18T08:15:41.451985+0000","last_became_peered":"2024-09-18T08:15:41.451985+0000","last_unstale":"2024-09-18T08:17:44.425443+0000","last_undegraded":"2024-09-18T08:17:44.425443+0000","last_fullsized":"2024-09-18T08:17:44.425443+0000","mapping_epoch":336,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":337,"parent":"0.0","parent_split_bits":0,"last_scrub":"271'29","last_scrub_stamp":"2024-09-18T08:13:33.814992+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_clean_scrub_stamp":"2024-09-18T08:13:33.814992+0000","objects_scrubbed":8,"log_size":29,"log_dups_size":0,"ondisk_log_size":29,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":54,"num_read_kb":40,"num_write":32,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":990,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.4","version":"62'1","reported_seq":566,"reported_epoch":381,"state":"active+clean","last_fresh":"2024-09-18T08:17:33.481684+0000","last_change":"2024-09-18T08:15:41.454692+0000","last_active":"2024-09-18T08:17:33.481684+0000","last_peered":"2024-09-18T08:17:33.481684+0000","last_clean":"2024-09-18T08:17:33.481684+0000","last_became_active":"2024-09-18T08:15:41.441822+0000","last_became_peered":"2024-09-18T08:15:41.441822+0000","last_unstale":"2024-09-18T08:17:33.481684+0000","last_undegraded":"2024-09-18T08:17:33.481684+0000","last_fullsized":"2024-09-18T08:17:33.481684+0000","mapping_epoch":336,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":337,"parent":"0.0","parent_split_bits":0,"last_scrub":"62'1","last_scrub_stamp":"2024-09-18T08:15:13.165826+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:46.860092+0000","last_clean_scrub_stamp":"2024-09-18T08:15:13.165826+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":17,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":436,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":1308,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":616,"reported_epoch":383,"state":"active+clean","last_fresh":"2024-09-18T08:17:44.419786+0000","last_change":"2024-09-18T08:15:41.450294+0000","last_active":"2024-09-18T08:17:44.419786+0000","last_peered":"2024-09-18T08:17:44.419786+0000","last_clean":"2024-09-18T08:17:44.419786+0000","last_became_active":"2024-09-18T08:15:41.449832+0000","last_became_peered":"2024-09-18T08:15:41.449832+0000","last_unstale":"2024-09-18T08:17:44.419786+0000","last_undegraded":"2024-09-18T08:17:44.419786+0000","last_fullsized":"2024-09-18T08:17:44.419786+0000","mapping_epoch":336,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":337,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:12:37.732260+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:50.886830+0000","last_clean_scrub_stamp":"2024-09-18T08:12:37.732260+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":585,"reported_epoch":381,"state":"active+clean","last_fresh":"2024-09-18T08:17:33.479511+0000","last_change":"2024-09-18T08:15:50.294289+0000","last_active":"2024-09-18T08:17:33.479511+0000","last_peered":"2024-09-18T08:17:33.479511+0000","last_clean":"2024-09-18T08:17:33.479511+0000","last_became_active":"2024-09-18T08:14:39.358132+0000","last_became_peered":"2024-09-18T08:14:39.358132+0000","last_unstale":"2024-09-18T08:17:33.479511+0000","last_undegraded":"2024-09-18T08:17:33.479511+0000","last_fullsized":"2024-09-18T08:17:33.479511+0000","mapping_epoch":307,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":308,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:15:50.294213+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_clean_scrub_stamp":"2024-09-18T08:15:50.294213+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":28,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.a","version":"383'10912","reported_seq":23404,"reported_epoch":383,"state":"active+clean","last_fresh":"2024-09-18T08:17:44.848008+0000","last_change":"2024-09-18T08:16:21.405814+0000","last_active":"2024-09-18T08:17:44.848008+0000","last_peered":"2024-09-18T08:17:44.848008+0000","last_clean":"2024-09-18T08:17:44.848008+0000","last_became_active":"2024-09-18T08:16:21.396170+0000","last_became_peered":"2024-09-18T08:16:21.396170+0000","last_unstale":"2024-09-18T08:17:44.848008+0000","last_undegraded":"2024-09-18T08:17:44.848008+0000","last_fullsized":"2024-09-18T08:17:44.848008+0000","mapping_epoch":354,"log_start":"358'7192","ondisk_log_start":"358'7192","created":311,"last_epoch_clean":355,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:14:43.412748+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:14:43.412748+0000","last_clean_scrub_stamp":"2024-09-18T08:14:43.412748+0000","objects_scrubbed":0,"log_size":3720,"log_dups_size":0,"ondisk_log_size":3720,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":116654080,"num_objects":1780,"num_object_clones":0,"num_object_copies":5340,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1780,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10912,"num_write_kb":406144,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6152,"num_bytes_recovered":403177472,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"74'79","reported_seq":110,"reported_epoch":382,"state":"active+clean","last_fresh":"2024-09-18T08:17:43.528225+0000","last_change":"2024-09-18T08:15:43.275398+0000","last_active":"2024-09-18T08:17:43.528225+0000","last_peered":"2024-09-18T08:17:43.528225+0000","last_clean":"2024-09-18T08:17:43.528225+0000","last_became_active":"2024-09-18T08:15:41.452895+0000","last_became_peered":"2024-09-18T08:15:41.452895+0000","last_unstale":"2024-09-18T08:17:43.528225+0000","last_undegraded":"2024-09-18T08:17:43.528225+0000","last_fullsized":"2024-09-18T08:17:43.528225+0000","mapping_epoch":336,"log_start":"0'0","ondisk_log_start":"0'0","created":152,"last_epoch_clean":337,"parent":"0.0","parent_split_bits":4,"last_scrub":"74'79","last_scrub_stamp":"2024-09-18T08:15:43.275329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:00:11.893742+0000","last_clean_scrub_stamp":"2024-09-18T08:15:43.275329+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,2],"acting":[5,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.4","version":"310'42","reported_seq":716,"reported_epoch":381,"state":"active+clean","last_fresh":"2024-09-18T08:17:33.481906+0000","last_change":"2024-09-18T08:14:39.529072+0000","last_active":"2024-09-18T08:17:33.481906+0000","last_peered":"2024-09-18T08:17:33.481906+0000","last_clean":"2024-09-18T08:17:33.481906+0000","last_became_active":"2024-09-18T08:14:39.486174+0000","last_became_peered":"2024-09-18T08:14:39.486174+0000","last_unstale":"2024-09-18T08:17:33.481906+0000","last_undegraded":"2024-09-18T08:17:33.481906+0000","last_fullsized":"2024-09-18T08:17:33.481906+0000","mapping_epoch":307,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":308,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'28","last_scrub_stamp":"2024-09-18T08:10:46.497874+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_clean_scrub_stamp":"2024-09-18T08:10:46.497874+0000","objects_scrubbed":0,"log_size":42,"log_dups_size":0,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:18:26.645395+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":65,"num_read_kb":50,"num_write":38,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":284,"reported_epoch":383,"state":"active+clean","last_fresh":"2024-09-18T08:17:44.425625+0000","last_change":"2024-09-18T08:16:13.476275+0000","last_active":"2024-09-18T08:17:44.425625+0000","last_peered":"2024-09-18T08:17:44.425625+0000","last_clean":"2024-09-18T08:17:44.425625+0000","last_became_active":"2024-09-18T08:16:13.475879+0000","last_became_peered":"2024-09-18T08:16:13.475879+0000","last_unstale":"2024-09-18T08:17:44.425625+0000","last_undegraded":"2024-09-18T08:17:44.425625+0000","last_fullsized":"2024-09-18T08:17:44.425625+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:13:52.760146+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:46.860092+0000","last_clean_scrub_stamp":"2024-09-18T08:13:52.760146+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,1],"acting":[0,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"64'2","reported_seq":114,"reported_epoch":382,"state":"active+clean","last_fresh":"2024-09-18T08:17:43.528438+0000","last_change":"2024-09-18T08:17:08.332007+0000","last_active":"2024-09-18T08:17:43.528438+0000","last_peered":"2024-09-18T08:17:43.528438+0000","last_clean":"2024-09-18T08:17:43.528438+0000","last_became_active":"2024-09-18T08:15:41.453351+0000","last_became_peered":"2024-09-18T08:15:41.453351+0000","last_unstale":"2024-09-18T08:17:43.528438+0000","last_undegraded":"2024-09-18T08:17:43.528438+0000","last_fullsized":"2024-09-18T08:17:43.528438+0000","mapping_epoch":336,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":337,"parent":"0.0","parent_split_bits":0,"last_scrub":"64'2","last_scrub_stamp":"2024-09-18T08:17:08.331955+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:45.855979+0000","last_clean_scrub_stamp":"2024-09-18T08:17:08.331955+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:18:28.285273+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":708,"reported_epoch":383,"state":"active+clean","last_fresh":"2024-09-18T08:17:44.420463+0000","last_change":"2024-09-18T08:16:13.481778+0000","last_active":"2024-09-18T08:17:44.420463+0000","last_peered":"2024-09-18T08:17:44.420463+0000","last_clean":"2024-09-18T08:17:44.420463+0000","last_became_active":"2024-09-18T08:16:13.481456+0000","last_became_peered":"2024-09-18T08:16:13.481456+0000","last_unstale":"2024-09-18T08:17:44.420463+0000","last_undegraded":"2024-09-18T08:17:44.420463+0000","last_fullsized":"2024-09-18T08:17:44.420463+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:15:04.260540+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:50.886830+0000","last_clean_scrub_stamp":"2024-09-18T08:15:04.260540+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,6],"acting":[0,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":715,"reported_epoch":382,"state":"active+clean","last_fresh":"2024-09-18T08:17:43.548671+0000","last_change":"2024-09-18T08:16:13.462219+0000","last_active":"2024-09-18T08:17:43.548671+0000","last_peered":"2024-09-18T08:17:43.548671+0000","last_clean":"2024-09-18T08:17:43.548671+0000","last_became_active":"2024-09-18T08:16:13.459207+0000","last_became_peered":"2024-09-18T08:16:13.459207+0000","last_unstale":"2024-09-18T08:17:43.548671+0000","last_undegraded":"2024-09-18T08:17:43.548671+0000","last_fullsized":"2024-09-18T08:17:43.548671+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:15:59.948801+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_clean_scrub_stamp":"2024-09-18T08:15:59.948801+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.9","version":"382'11065","reported_seq":23337,"reported_epoch":382,"state":"active+clean","last_fresh":"2024-09-18T08:17:43.676268+0000","last_change":"2024-09-18T08:17:43.058512+0000","last_active":"2024-09-18T08:17:43.676268+0000","last_peered":"2024-09-18T08:17:43.676268+0000","last_clean":"2024-09-18T08:17:43.676268+0000","last_became_active":"2024-09-18T08:16:20.398277+0000","last_became_peered":"2024-09-18T08:16:20.398277+0000","last_unstale":"2024-09-18T08:17:43.676268+0000","last_undegraded":"2024-09-18T08:17:43.676268+0000","last_fullsized":"2024-09-18T08:17:43.676268+0000","mapping_epoch":353,"log_start":"358'7432","ondisk_log_start":"358'7432","created":311,"last_epoch_clean":354,"parent":"0.0","parent_split_bits":0,"last_scrub":"381'11012","last_scrub_stamp":"2024-09-18T08:17:43.058365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:14:43.412748+0000","last_clean_scrub_stamp":"2024-09-18T08:17:43.058365+0000","objects_scrubbed":1991,"log_size":3633,"log_dups_size":0,"ondisk_log_size":3633,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:18:43.655486+0000","scrub_duration":512,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":125632512,"num_objects":1917,"num_object_clones":0,"num_object_copies":5751,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1917,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11065,"num_write_kb":415424,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6027,"num_bytes_recovered":394985472,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,3],"acting":[5,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.2","version":"74'79","reported_seq":621,"reported_epoch":383,"state":"active+clean","last_fresh":"2024-09-18T08:17:44.419565+0000","last_change":"2024-09-18T08:16:13.453252+0000","last_active":"2024-09-18T08:17:44.419565+0000","last_peered":"2024-09-18T08:17:44.419565+0000","last_clean":"2024-09-18T08:17:44.419565+0000","last_became_active":"2024-09-18T08:16:13.451687+0000","last_became_peered":"2024-09-18T08:16:13.451687+0000","last_unstale":"2024-09-18T08:17:44.419565+0000","last_undegraded":"2024-09-18T08:17:44.419565+0000","last_fullsized":"2024-09-18T08:17:44.419565+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":152,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":4,"last_scrub":"74'79","last_scrub_stamp":"2024-09-18T08:14:01.846338+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:00:11.893742+0000","last_clean_scrub_stamp":"2024-09-18T08:14:01.846338+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,5],"acting":[0,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.7","version":"271'17","reported_seq":684,"reported_epoch":382,"state":"active+clean","last_fresh":"2024-09-18T08:17:43.535912+0000","last_change":"2024-09-18T08:15:41.494329+0000","last_active":"2024-09-18T08:17:43.535912+0000","last_peered":"2024-09-18T08:17:43.535912+0000","last_clean":"2024-09-18T08:17:43.535912+0000","last_became_active":"2024-09-18T08:15:41.456482+0000","last_became_peered":"2024-09-18T08:15:41.456482+0000","last_unstale":"2024-09-18T08:17:43.535912+0000","last_undegraded":"2024-09-18T08:17:43.535912+0000","last_fullsized":"2024-09-18T08:17:43.535912+0000","mapping_epoch":336,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":337,"parent":"0.0","parent_split_bits":0,"last_scrub":"271'17","last_scrub_stamp":"2024-09-18T08:15:13.792602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_clean_scrub_stamp":"2024-09-18T08:15:13.792602+0000","objects_scrubbed":5,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":16,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":35,"num_read_kb":24,"num_write":20,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":990,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,0],"acting":[2,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":575,"reported_epoch":381,"state":"active+clean","last_fresh":"2024-09-18T08:17:33.478152+0000","last_change":"2024-09-18T08:15:49.244718+0000","last_active":"2024-09-18T08:17:33.478152+0000","last_peered":"2024-09-18T08:17:33.478152+0000","last_clean":"2024-09-18T08:17:33.478152+0000","last_became_active":"2024-09-18T08:14:39.432254+0000","last_became_peered":"2024-09-18T08:14:39.432254+0000","last_unstale":"2024-09-18T08:17:33.478152+0000","last_undegraded":"2024-09-18T08:17:33.478152+0000","last_fullsized":"2024-09-18T08:17:33.478152+0000","mapping_epoch":307,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":308,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:15:49.244156+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:46.860092+0000","last_clean_scrub_stamp":"2024-09-18T08:15:49.244156+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"62'1","reported_seq":708,"reported_epoch":382,"state":"active+clean","last_fresh":"2024-09-18T08:17:43.536784+0000","last_change":"2024-09-18T08:17:34.935837+0000","last_active":"2024-09-18T08:17:43.536784+0000","last_peered":"2024-09-18T08:17:43.536784+0000","last_clean":"2024-09-18T08:17:43.536784+0000","last_became_active":"2024-09-18T08:02:46.868175+0000","last_became_peered":"2024-09-18T08:02:46.868175+0000","last_unstale":"2024-09-18T08:17:43.536784+0000","last_undegraded":"2024-09-18T08:17:43.536784+0000","last_fullsized":"2024-09-18T08:17:43.536784+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"62'1","last_scrub_stamp":"2024-09-18T08:17:34.935618+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:45.855979+0000","last_clean_scrub_stamp":"2024-09-18T08:17:34.935618+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:18:36.691557+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0.00066785499999999997,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":418,"reported_epoch":382,"state":"active+clean","last_fresh":"2024-09-18T08:17:43.536643+0000","last_change":"2024-09-18T08:16:13.467275+0000","last_active":"2024-09-18T08:17:43.536643+0000","last_peered":"2024-09-18T08:17:43.536643+0000","last_clean":"2024-09-18T08:17:43.536643+0000","last_became_active":"2024-09-18T08:16:13.466910+0000","last_became_peered":"2024-09-18T08:16:13.466910+0000","last_unstale":"2024-09-18T08:17:43.536643+0000","last_undegraded":"2024-09-18T08:17:43.536643+0000","last_fullsized":"2024-09-18T08:17:43.536643+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:15:00.702835+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:50.886830+0000","last_clean_scrub_stamp":"2024-09-18T08:15:00.702835+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,1],"acting":[2,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":379,"reported_epoch":382,"state":"active+clean","last_fresh":"2024-09-18T08:17:43.550643+0000","last_change":"2024-09-18T08:17:35.381592+0000","last_active":"2024-09-18T08:17:43.550643+0000","last_peered":"2024-09-18T08:17:43.550643+0000","last_clean":"2024-09-18T08:17:43.550643+0000","last_became_active":"2024-09-18T08:14:05.577516+0000","last_became_peered":"2024-09-18T08:14:05.577516+0000","last_unstale":"2024-09-18T08:17:43.550643+0000","last_undegraded":"2024-09-18T08:17:43.550643+0000","last_fullsized":"2024-09-18T08:17:43.550643+0000","mapping_epoch":292,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":293,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:17:35.381401+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_clean_scrub_stamp":"2024-09-18T08:17:35.381401+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:18:54.224811+0000","scrub_duration":23,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.8","version":"383'10895","reported_seq":22730,"reported_epoch":383,"state":"active+clean","last_fresh":"2024-09-18T08:17:44.842726+0000","last_change":"2024-09-18T08:16:54.572396+0000","last_active":"2024-09-18T08:17:44.842726+0000","last_peered":"2024-09-18T08:17:44.842726+0000","last_clean":"2024-09-18T08:17:44.842726+0000","last_became_active":"2024-09-18T08:16:54.554366+0000","last_became_peered":"2024-09-18T08:16:54.554366+0000","last_unstale":"2024-09-18T08:17:44.842726+0000","last_undegraded":"2024-09-18T08:17:44.842726+0000","last_fullsized":"2024-09-18T08:17:44.842726+0000","mapping_epoch":364,"log_start":"358'7148","ondisk_log_start":"358'7148","created":311,"last_epoch_clean":365,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:14:43.412748+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:14:43.412748+0000","last_clean_scrub_stamp":"2024-09-18T08:14:43.412748+0000","objects_scrubbed":0,"log_size":3747,"log_dups_size":0,"ondisk_log_size":3747,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":116981760,"num_objects":1785,"num_object_clones":0,"num_object_copies":5355,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1785,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10895,"num_write_kb":405760,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5820,"num_bytes_recovered":381419520,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,5],"acting":[0,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"74'79","reported_seq":350,"reported_epoch":381,"state":"active+clean","last_fresh":"2024-09-18T08:17:33.483892+0000","last_change":"2024-09-18T08:16:13.455530+0000","last_active":"2024-09-18T08:17:33.483892+0000","last_peered":"2024-09-18T08:17:33.483892+0000","last_clean":"2024-09-18T08:17:33.483892+0000","last_became_active":"2024-09-18T08:16:13.454138+0000","last_became_peered":"2024-09-18T08:16:13.454138+0000","last_unstale":"2024-09-18T08:17:33.483892+0000","last_undegraded":"2024-09-18T08:17:33.483892+0000","last_fullsized":"2024-09-18T08:17:33.483892+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":152,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":3,"last_scrub":"74'79","last_scrub_stamp":"2024-09-18T08:16:00.303863+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:00:11.893742+0000","last_clean_scrub_stamp":"2024-09-18T08:16:00.303863+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,5],"acting":[1,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.6","version":"271'24","reported_seq":791,"reported_epoch":383,"state":"active+clean","last_fresh":"2024-09-18T08:17:44.425991+0000","last_change":"2024-09-18T08:15:10.187667+0000","last_active":"2024-09-18T08:17:44.425991+0000","last_peered":"2024-09-18T08:17:44.425991+0000","last_clean":"2024-09-18T08:17:44.425991+0000","last_became_active":"2024-09-18T08:14:39.438033+0000","last_became_peered":"2024-09-18T08:14:39.438033+0000","last_unstale":"2024-09-18T08:17:44.425991+0000","last_undegraded":"2024-09-18T08:17:44.425991+0000","last_fullsized":"2024-09-18T08:17:44.425991+0000","mapping_epoch":307,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":308,"parent":"0.0","parent_split_bits":0,"last_scrub":"271'24","last_scrub_stamp":"2024-09-18T08:15:10.187593+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_clean_scrub_stamp":"2024-09-18T08:15:10.187593+0000","objects_scrubbed":6,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":36,"num_read_kb":30,"num_write":24,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":721,"reported_epoch":383,"state":"active+clean","last_fresh":"2024-09-18T08:17:44.431569+0000","last_change":"2024-09-18T08:16:13.461291+0000","last_active":"2024-09-18T08:17:44.431569+0000","last_peered":"2024-09-18T08:17:44.431569+0000","last_clean":"2024-09-18T08:17:44.431569+0000","last_became_active":"2024-09-18T08:16:13.460391+0000","last_became_peered":"2024-09-18T08:16:13.460391+0000","last_unstale":"2024-09-18T08:17:44.431569+0000","last_undegraded":"2024-09-18T08:17:44.431569+0000","last_fullsized":"2024-09-18T08:17:44.431569+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:14:37.133544+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:46.860092+0000","last_clean_scrub_stamp":"2024-09-18T08:14:37.133544+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,5],"acting":[0,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"70'5","reported_seq":829,"reported_epoch":381,"state":"active+clean","last_fresh":"2024-09-18T08:17:40.989246+0000","last_change":"2024-09-18T08:15:57.384301+0000","last_active":"2024-09-18T08:17:40.989246+0000","last_peered":"2024-09-18T08:17:40.989246+0000","last_clean":"2024-09-18T08:17:40.989246+0000","last_became_active":"2024-09-18T08:14:40.353801+0000","last_became_peered":"2024-09-18T08:14:40.353801+0000","last_unstale":"2024-09-18T08:17:40.989246+0000","last_undegraded":"2024-09-18T08:17:40.989246+0000","last_fullsized":"2024-09-18T08:17:40.989246+0000","mapping_epoch":308,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":309,"parent":"0.0","parent_split_bits":0,"last_scrub":"70'5","last_scrub_stamp":"2024-09-18T08:15:57.384224+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:45.855979+0000","last_clean_scrub_stamp":"2024-09-18T08:15:57.384224+0000","objects_scrubbed":1,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0.000504055,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":355,"num_read_kb":350,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":389,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":102,"reported_epoch":382,"state":"active+clean","last_fresh":"2024-09-18T08:17:43.529228+0000","last_change":"2024-09-18T08:15:41.443577+0000","last_active":"2024-09-18T08:17:43.529228+0000","last_peered":"2024-09-18T08:17:43.529228+0000","last_clean":"2024-09-18T08:17:43.529228+0000","last_became_active":"2024-09-18T08:15:41.443080+0000","last_became_peered":"2024-09-18T08:15:41.443080+0000","last_unstale":"2024-09-18T08:17:43.529228+0000","last_undegraded":"2024-09-18T08:17:43.529228+0000","last_fullsized":"2024-09-18T08:17:43.529228+0000","mapping_epoch":336,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":337,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:15:02.179693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:50.886830+0000","last_clean_scrub_stamp":"2024-09-18T08:15:02.179693+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":497,"reported_epoch":381,"state":"active+clean","last_fresh":"2024-09-18T08:17:33.481492+0000","last_change":"2024-09-18T08:15:41.437061+0000","last_active":"2024-09-18T08:17:33.481492+0000","last_peered":"2024-09-18T08:17:33.481492+0000","last_clean":"2024-09-18T08:17:33.481492+0000","last_became_active":"2024-09-18T08:15:41.436692+0000","last_became_peered":"2024-09-18T08:15:41.436692+0000","last_unstale":"2024-09-18T08:17:33.481492+0000","last_undegraded":"2024-09-18T08:17:33.481492+0000","last_fullsized":"2024-09-18T08:17:33.481492+0000","mapping_epoch":336,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":337,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:15:07.219279+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_clean_scrub_stamp":"2024-09-18T08:15:07.219279+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.b","version":"381'10992","reported_seq":11144,"reported_epoch":381,"state":"active+clean","last_fresh":"2024-09-18T08:17:42.517081+0000","last_change":"2024-09-18T08:14:44.432885+0000","last_active":"2024-09-18T08:17:42.517081+0000","last_peered":"2024-09-18T08:17:42.517081+0000","last_clean":"2024-09-18T08:17:42.517081+0000","last_became_active":"2024-09-18T08:14:44.432599+0000","last_became_peered":"2024-09-18T08:14:44.432599+0000","last_unstale":"2024-09-18T08:17:42.517081+0000","last_undegraded":"2024-09-18T08:17:42.517081+0000","last_fullsized":"2024-09-18T08:17:42.517081+0000","mapping_epoch":311,"log_start":"358'7401","ondisk_log_start":"358'7401","created":311,"last_epoch_clean":312,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:14:43.412748+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:14:43.412748+0000","last_clean_scrub_stamp":"2024-09-18T08:14:43.412748+0000","objects_scrubbed":0,"log_size":3591,"log_dups_size":0,"ondisk_log_size":3591,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":125698048,"num_objects":1918,"num_object_clones":0,"num_object_copies":5754,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1918,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10992,"num_write_kb":413120,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,0],"acting":[1,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.5","version":"349'35","reported_seq":473,"reported_epoch":383,"state":"active+clean","last_fresh":"2024-09-18T08:17:44.420169+0000","last_change":"2024-09-18T08:16:13.521519+0000","last_active":"2024-09-18T08:17:44.420169+0000","last_peered":"2024-09-18T08:17:44.420169+0000","last_clean":"2024-09-18T08:17:44.420169+0000","last_became_active":"2024-09-18T08:16:13.460467+0000","last_became_peered":"2024-09-18T08:16:13.460467+0000","last_unstale":"2024-09-18T08:17:44.420169+0000","last_undegraded":"2024-09-18T08:17:44.420169+0000","last_fullsized":"2024-09-18T08:17:44.420169+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"271'33","last_scrub_stamp":"2024-09-18T08:13:28.626987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_clean_scrub_stamp":"2024-09-18T08:13:28.626987+0000","objects_scrubbed":8,"log_size":35,"log_dups_size":0,"ondisk_log_size":35,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":24,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":42,"num_read_kb":32,"num_write":25,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":308,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,3],"acting":[0,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":711,"reported_epoch":382,"state":"active+clean","last_fresh":"2024-09-18T08:17:43.546481+0000","last_change":"2024-09-18T08:15:41.464630+0000","last_active":"2024-09-18T08:17:43.546481+0000","last_peered":"2024-09-18T08:17:43.546481+0000","last_clean":"2024-09-18T08:17:43.546481+0000","last_became_active":"2024-09-18T08:15:41.464313+0000","last_became_peered":"2024-09-18T08:15:41.464313+0000","last_unstale":"2024-09-18T08:17:43.546481+0000","last_undegraded":"2024-09-18T08:17:43.546481+0000","last_fullsized":"2024-09-18T08:17:43.546481+0000","mapping_epoch":336,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":337,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:14:32.693460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:46.860092+0000","last_clean_scrub_stamp":"2024-09-18T08:14:32.693460+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":38,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"74'79","reported_seq":302,"reported_epoch":383,"state":"active+clean","last_fresh":"2024-09-18T08:17:44.418372+0000","last_change":"2024-09-18T08:14:40.357413+0000","last_active":"2024-09-18T08:17:44.418372+0000","last_peered":"2024-09-18T08:17:44.418372+0000","last_clean":"2024-09-18T08:17:44.418372+0000","last_became_active":"2024-09-18T08:14:40.357063+0000","last_became_peered":"2024-09-18T08:14:40.357063+0000","last_unstale":"2024-09-18T08:17:44.418372+0000","last_undegraded":"2024-09-18T08:17:44.418372+0000","last_fullsized":"2024-09-18T08:17:44.418372+0000","mapping_epoch":308,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":309,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'79","last_scrub_stamp":"2024-09-18T08:13:51.810296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:00:11.893742+0000","last_clean_scrub_stamp":"2024-09-18T08:13:51.810296+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,1],"acting":[0,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":555,"reported_epoch":382,"state":"active+clean","last_fresh":"2024-09-18T08:17:43.535718+0000","last_change":"2024-09-18T08:17:36.886577+0000","last_active":"2024-09-18T08:17:43.535718+0000","last_peered":"2024-09-18T08:17:43.535718+0000","last_clean":"2024-09-18T08:17:43.535718+0000","last_became_active":"2024-09-18T08:15:41.441392+0000","last_became_peered":"2024-09-18T08:15:41.441392+0000","last_unstale":"2024-09-18T08:17:43.535718+0000","last_undegraded":"2024-09-18T08:17:43.535718+0000","last_fullsized":"2024-09-18T08:17:43.535718+0000","mapping_epoch":336,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":337,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:17:36.886507+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:50.886830+0000","last_clean_scrub_stamp":"2024-09-18T08:17:36.886507+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:18:41.767214+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,0],"acting":[2,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":109,"reported_epoch":382,"state":"active+clean","last_fresh":"2024-09-18T08:17:43.529175+0000","last_change":"2024-09-18T08:16:13.472241+0000","last_active":"2024-09-18T08:17:43.529175+0000","last_peered":"2024-09-18T08:17:43.529175+0000","last_clean":"2024-09-18T08:17:43.529175+0000","last_became_active":"2024-09-18T08:16:13.471911+0000","last_became_peered":"2024-09-18T08:16:13.471911+0000","last_unstale":"2024-09-18T08:17:43.529175+0000","last_undegraded":"2024-09-18T08:17:43.529175+0000","last_fullsized":"2024-09-18T08:17:43.529175+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:12:05.886281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_clean_scrub_stamp":"2024-09-18T08:12:05.886281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:18:25.503591+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,1],"acting":[5,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.0","version":"381'10997","reported_seq":22115,"reported_epoch":381,"state":"active+clean","last_fresh":"2024-09-18T08:17:42.554920+0000","last_change":"2024-09-18T08:16:57.592438+0000","last_active":"2024-09-18T08:17:42.554920+0000","last_peered":"2024-09-18T08:17:42.554920+0000","last_clean":"2024-09-18T08:17:42.554920+0000","last_became_active":"2024-09-18T08:16:57.574026+0000","last_became_peered":"2024-09-18T08:16:57.574026+0000","last_unstale":"2024-09-18T08:17:42.554920+0000","last_undegraded":"2024-09-18T08:17:42.554920+0000","last_fullsized":"2024-09-18T08:17:42.554920+0000","mapping_epoch":366,"log_start":"358'7407","ondisk_log_start":"358'7407","created":311,"last_epoch_clean":367,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:14:43.412748+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:14:43.412748+0000","last_clean_scrub_stamp":"2024-09-18T08:14:43.412748+0000","objects_scrubbed":0,"log_size":3590,"log_dups_size":0,"ondisk_log_size":3590,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":124059648,"num_objects":1893,"num_object_clones":0,"num_object_copies":5679,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1893,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10997,"num_write_kb":412480,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5473,"num_bytes_recovered":358678528,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,0],"acting":[1,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.e","version":"271'19","reported_seq":397,"reported_epoch":381,"state":"active+clean","last_fresh":"2024-09-18T08:17:33.478701+0000","last_change":"2024-09-18T08:17:30.690655+0000","last_active":"2024-09-18T08:17:33.478701+0000","last_peered":"2024-09-18T08:17:33.478701+0000","last_clean":"2024-09-18T08:17:33.478701+0000","last_became_active":"2024-09-18T08:16:13.458001+0000","last_became_peered":"2024-09-18T08:16:13.458001+0000","last_unstale":"2024-09-18T08:17:33.478701+0000","last_undegraded":"2024-09-18T08:17:33.478701+0000","last_fullsized":"2024-09-18T08:17:33.478701+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"271'19","last_scrub_stamp":"2024-09-18T08:17:30.690581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_clean_scrub_stamp":"2024-09-18T08:17:30.690581+0000","objects_scrubbed":5,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:18:59.500251+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":31,"num_read_kb":24,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,3],"acting":[1,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":365,"reported_epoch":382,"state":"active+clean","last_fresh":"2024-09-18T08:17:43.535476+0000","last_change":"2024-09-18T08:16:13.476914+0000","last_active":"2024-09-18T08:17:43.535476+0000","last_peered":"2024-09-18T08:17:43.535476+0000","last_clean":"2024-09-18T08:17:43.535476+0000","last_became_active":"2024-09-18T08:16:13.476579+0000","last_became_peered":"2024-09-18T08:16:13.476579+0000","last_unstale":"2024-09-18T08:17:43.535476+0000","last_undegraded":"2024-09-18T08:17:43.535476+0000","last_fullsized":"2024-09-18T08:17:43.535476+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:11:59.458232+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:46.860092+0000","last_clean_scrub_stamp":"2024-09-18T08:11:59.458232+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:18:25.909992+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,5],"acting":[2,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":117,"reported_epoch":382,"state":"active+clean","last_fresh":"2024-09-18T08:17:43.529089+0000","last_change":"2024-09-18T08:17:38.616479+0000","last_active":"2024-09-18T08:17:43.529089+0000","last_peered":"2024-09-18T08:17:43.529089+0000","last_clean":"2024-09-18T08:17:43.529089+0000","last_became_active":"2024-09-18T08:16:13.443884+0000","last_became_peered":"2024-09-18T08:16:13.443884+0000","last_unstale":"2024-09-18T08:17:43.529089+0000","last_undegraded":"2024-09-18T08:17:43.529089+0000","last_fullsized":"2024-09-18T08:17:43.529089+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:17:38.616405+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:50.886830+0000","last_clean_scrub_stamp":"2024-09-18T08:17:38.616405+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:19:06.381094+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,2],"acting":[5,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":698,"reported_epoch":382,"state":"active+clean","last_fresh":"2024-09-18T08:17:43.548209+0000","last_change":"2024-09-18T08:15:41.465035+0000","last_active":"2024-09-18T08:17:43.548209+0000","last_peered":"2024-09-18T08:17:43.548209+0000","last_clean":"2024-09-18T08:17:43.548209+0000","last_became_active":"2024-09-18T08:15:41.464500+0000","last_became_peered":"2024-09-18T08:15:41.464500+0000","last_unstale":"2024-09-18T08:17:43.548209+0000","last_undegraded":"2024-09-18T08:17:43.548209+0000","last_fullsized":"2024-09-18T08:17:43.548209+0000","mapping_epoch":336,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":337,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:15:02.861116+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_clean_scrub_stamp":"2024-09-18T08:15:02.861116+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":81,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.3","version":"382'10769","reported_seq":23973,"reported_epoch":382,"state":"active+clean","last_fresh":"2024-09-18T08:17:44.038582+0000","last_change":"2024-09-18T08:16:40.417668+0000","last_active":"2024-09-18T08:17:44.038582+0000","last_peered":"2024-09-18T08:17:44.038582+0000","last_clean":"2024-09-18T08:17:44.038582+0000","last_became_active":"2024-09-18T08:16:40.406526+0000","last_became_peered":"2024-09-18T08:16:40.406526+0000","last_unstale":"2024-09-18T08:17:44.038582+0000","last_undegraded":"2024-09-18T08:17:44.038582+0000","last_fullsized":"2024-09-18T08:17:44.038582+0000","mapping_epoch":359,"log_start":"350'6113","ondisk_log_start":"350'6113","created":311,"last_epoch_clean":360,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:14:43.412748+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:14:43.412748+0000","last_clean_scrub_stamp":"2024-09-18T08:14:43.412748+0000","objects_scrubbed":0,"log_size":4656,"log_dups_size":0,"ondisk_log_size":4656,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117637120,"num_objects":1795,"num_object_clones":0,"num_object_copies":5385,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1795,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10769,"num_write_kb":402048,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6505,"num_bytes_recovered":426311680,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,3],"acting":[2,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.8","version":"74'79","reported_seq":114,"reported_epoch":382,"state":"active+clean","last_fresh":"2024-09-18T08:17:43.558527+0000","last_change":"2024-09-18T08:17:43.558527+0000","last_active":"2024-09-18T08:17:43.558527+0000","last_peered":"2024-09-18T08:17:43.558527+0000","last_clean":"2024-09-18T08:17:43.558527+0000","last_became_active":"2024-09-18T08:15:41.454781+0000","last_became_peered":"2024-09-18T08:15:41.454781+0000","last_unstale":"2024-09-18T08:17:43.558527+0000","last_undegraded":"2024-09-18T08:17:43.558527+0000","last_fullsized":"2024-09-18T08:17:43.558527+0000","mapping_epoch":336,"log_start":"0'0","ondisk_log_start":"0'0","created":152,"last_epoch_clean":337,"parent":"0.0","parent_split_bits":4,"last_scrub":"74'79","last_scrub_stamp":"2024-09-18T08:17:43.558474+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:00:11.893742+0000","last_clean_scrub_stamp":"2024-09-18T08:17:43.558474+0000","objects_scrubbed":1,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:18:45.862783+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262688,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":262688,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,3],"acting":[5,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.d","version":"271'31","reported_seq":330,"reported_epoch":382,"state":"active+clean","last_fresh":"2024-09-18T08:17:43.537891+0000","last_change":"2024-09-18T08:16:13.511045+0000","last_active":"2024-09-18T08:17:43.537891+0000","last_peered":"2024-09-18T08:17:43.537891+0000","last_clean":"2024-09-18T08:17:43.537891+0000","last_became_active":"2024-09-18T08:16:13.466471+0000","last_became_peered":"2024-09-18T08:16:13.466471+0000","last_unstale":"2024-09-18T08:17:43.537891+0000","last_undegraded":"2024-09-18T08:17:43.537891+0000","last_fullsized":"2024-09-18T08:17:43.537891+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"271'31","last_scrub_stamp":"2024-09-18T08:15:03.678670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_clean_scrub_stamp":"2024-09-18T08:15:03.678670+0000","objects_scrubbed":0,"log_size":31,"log_dups_size":0,"ondisk_log_size":31,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":48,"num_read_kb":38,"num_write":32,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,6],"acting":[2,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":352,"reported_epoch":381,"state":"active+clean","last_fresh":"2024-09-18T08:17:33.480414+0000","last_change":"2024-09-18T08:16:13.487196+0000","last_active":"2024-09-18T08:17:33.480414+0000","last_peered":"2024-09-18T08:17:33.480414+0000","last_clean":"2024-09-18T08:17:33.480414+0000","last_became_active":"2024-09-18T08:16:13.483143+0000","last_became_peered":"2024-09-18T08:16:13.483143+0000","last_unstale":"2024-09-18T08:17:33.480414+0000","last_undegraded":"2024-09-18T08:17:33.480414+0000","last_fullsized":"2024-09-18T08:17:33.480414+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:15:09.614752+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:46.860092+0000","last_clean_scrub_stamp":"2024-09-18T08:15:09.614752+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,3],"acting":[1,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":582,"reported_epoch":381,"state":"active+clean","last_fresh":"2024-09-18T08:17:33.481593+0000","last_change":"2024-09-18T08:16:13.460968+0000","last_active":"2024-09-18T08:17:33.481593+0000","last_peered":"2024-09-18T08:17:33.481593+0000","last_clean":"2024-09-18T08:17:33.481593+0000","last_became_active":"2024-09-18T08:16:13.460650+0000","last_became_peered":"2024-09-18T08:16:13.460650+0000","last_unstale":"2024-09-18T08:17:33.481593+0000","last_undegraded":"2024-09-18T08:17:33.481593+0000","last_fullsized":"2024-09-18T08:17:33.481593+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:14:42.230420+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:50.886830+0000","last_clean_scrub_stamp":"2024-09-18T08:14:42.230420+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,6],"acting":[1,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":420,"reported_epoch":382,"state":"active+clean","last_fresh":"2024-09-18T08:17:43.535809+0000","last_change":"2024-09-18T08:17:33.946061+0000","last_active":"2024-09-18T08:17:43.535809+0000","last_peered":"2024-09-18T08:17:43.535809+0000","last_clean":"2024-09-18T08:17:43.535809+0000","last_became_active":"2024-09-18T08:16:13.457516+0000","last_became_peered":"2024-09-18T08:16:13.457516+0000","last_unstale":"2024-09-18T08:17:43.535809+0000","last_undegraded":"2024-09-18T08:17:43.535809+0000","last_fullsized":"2024-09-18T08:17:43.535809+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:17:33.945785+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_clean_scrub_stamp":"2024-09-18T08:17:33.945785+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:18:58.386817+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.2","version":"381'10744","reported_seq":10917,"reported_epoch":381,"state":"active+remapped+backfill_toofull","last_fresh":"2024-09-18T08:17:42.550690+0000","last_change":"2024-09-18T08:17:14.611706+0000","last_active":"2024-09-18T08:17:42.550690+0000","last_peered":"2024-09-18T08:17:42.550690+0000","last_clean":"2024-09-18T08:16:12.397450+0000","last_became_active":"2024-09-18T08:16:14.545962+0000","last_became_peered":"2024-09-18T08:16:14.545962+0000","last_unstale":"2024-09-18T08:17:42.550690+0000","last_undegraded":"2024-09-18T08:17:42.550690+0000","last_fullsized":"2024-09-18T08:17:42.550690+0000","mapping_epoch":349,"log_start":"358'7154","ondisk_log_start":"358'7154","created":311,"last_epoch_clean":312,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:14:43.412748+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:14:43.412748+0000","last_clean_scrub_stamp":"2024-09-18T08:14:43.412748+0000","objects_scrubbed":0,"log_size":3590,"log_dups_size":0,"ondisk_log_size":3590,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":122683392,"num_objects":1872,"num_object_clones":0,"num_object_copies":5616,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1872,"num_objects_unfound":0,"num_objects_dirty":1872,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10744,"num_write_kb":403712,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":65536,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,4],"avail_no_missing":["1","0","4"],"object_location_counts":[{"shards":"0,1,4","objects":1872}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"74'79","reported_seq":524,"reported_epoch":382,"state":"active+clean","last_fresh":"2024-09-18T08:17:44.331632+0000","last_change":"2024-09-18T08:17:44.331632+0000","last_active":"2024-09-18T08:17:44.331632+0000","last_peered":"2024-09-18T08:17:44.331632+0000","last_clean":"2024-09-18T08:17:44.331632+0000","last_became_active":"2024-09-18T08:15:41.465341+0000","last_became_peered":"2024-09-18T08:15:41.465341+0000","last_unstale":"2024-09-18T08:17:44.331632+0000","last_undegraded":"2024-09-18T08:17:44.331632+0000","last_fullsized":"2024-09-18T08:17:44.331632+0000","mapping_epoch":336,"log_start":"0'0","ondisk_log_start":"0'0","created":152,"last_epoch_clean":337,"parent":"0.0","parent_split_bits":4,"last_scrub":"74'79","last_scrub_stamp":"2024-09-18T08:17:44.331322+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:00:11.893742+0000","last_clean_scrub_stamp":"2024-09-18T08:17:44.331322+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:18:49.579430+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.c","version":"271'20","reported_seq":308,"reported_epoch":382,"state":"active+clean","last_fresh":"2024-09-18T08:17:43.552063+0000","last_change":"2024-09-18T08:16:13.526582+0000","last_active":"2024-09-18T08:17:43.552063+0000","last_peered":"2024-09-18T08:17:43.552063+0000","last_clean":"2024-09-18T08:17:43.552063+0000","last_became_active":"2024-09-18T08:16:13.476918+0000","last_became_peered":"2024-09-18T08:16:13.476918+0000","last_unstale":"2024-09-18T08:17:43.552063+0000","last_undegraded":"2024-09-18T08:17:43.552063+0000","last_fullsized":"2024-09-18T08:17:43.552063+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"271'20","last_scrub_stamp":"2024-09-18T08:15:56.673129+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_clean_scrub_stamp":"2024-09-18T08:15:56.673129+0000","objects_scrubbed":0,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":28,"num_read_kb":23,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,1],"acting":[3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":718,"reported_epoch":382,"state":"active+clean","last_fresh":"2024-09-18T08:17:43.549108+0000","last_change":"2024-09-18T08:16:13.479508+0000","last_active":"2024-09-18T08:17:43.549108+0000","last_peered":"2024-09-18T08:17:43.549108+0000","last_clean":"2024-09-18T08:17:43.549108+0000","last_became_active":"2024-09-18T08:16:13.479148+0000","last_became_peered":"2024-09-18T08:16:13.479148+0000","last_unstale":"2024-09-18T08:17:43.549108+0000","last_undegraded":"2024-09-18T08:17:43.549108+0000","last_fullsized":"2024-09-18T08:17:43.549108+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:14:04.626103+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:46.860092+0000","last_clean_scrub_stamp":"2024-09-18T08:14:04.626103+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.d","version":"338'13","reported_seq":922,"reported_epoch":382,"state":"active+clean","last_fresh":"2024-09-18T08:17:43.537778+0000","last_change":"2024-09-18T08:17:38.886726+0000","last_active":"2024-09-18T08:17:43.537778+0000","last_peered":"2024-09-18T08:17:43.537778+0000","last_clean":"2024-09-18T08:17:43.537778+0000","last_became_active":"2024-09-18T08:15:41.441562+0000","last_became_peered":"2024-09-18T08:15:41.441562+0000","last_unstale":"2024-09-18T08:17:43.537778+0000","last_undegraded":"2024-09-18T08:17:43.537778+0000","last_fullsized":"2024-09-18T08:17:43.537778+0000","mapping_epoch":336,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":337,"parent":"0.0","parent_split_bits":0,"last_scrub":"338'13","last_scrub_stamp":"2024-09-18T08:17:38.886654+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:50.886830+0000","last_clean_scrub_stamp":"2024-09-18T08:17:38.886654+0000","objects_scrubbed":1,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:18:41.947513+0000","scrub_duration":26,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,0],"acting":[2,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":359,"reported_epoch":381,"state":"active+clean","last_fresh":"2024-09-18T08:17:33.478523+0000","last_change":"2024-09-18T08:16:13.465932+0000","last_active":"2024-09-18T08:17:33.478523+0000","last_peered":"2024-09-18T08:17:33.478523+0000","last_clean":"2024-09-18T08:17:33.478523+0000","last_became_active":"2024-09-18T08:16:13.464924+0000","last_became_peered":"2024-09-18T08:16:13.464924+0000","last_unstale":"2024-09-18T08:17:33.478523+0000","last_undegraded":"2024-09-18T08:17:33.478523+0000","last_fullsized":"2024-09-18T08:17:33.478523+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:15:47.604976+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_clean_scrub_stamp":"2024-09-18T08:15:47.604976+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.5","version":"382'10927","reported_seq":11109,"reported_epoch":382,"state":"active+clean","last_fresh":"2024-09-18T08:17:44.043789+0000","last_change":"2024-09-18T08:17:42.813269+0000","last_active":"2024-09-18T08:17:44.043789+0000","last_peered":"2024-09-18T08:17:44.043789+0000","last_clean":"2024-09-18T08:17:44.043789+0000","last_became_active":"2024-09-18T08:14:44.431835+0000","last_became_peered":"2024-09-18T08:14:44.431835+0000","last_unstale":"2024-09-18T08:17:44.043789+0000","last_undegraded":"2024-09-18T08:17:44.043789+0000","last_fullsized":"2024-09-18T08:17:44.043789+0000","mapping_epoch":311,"log_start":"351'6213","ondisk_log_start":"351'6213","created":311,"last_epoch_clean":312,"parent":"0.0","parent_split_bits":0,"last_scrub":"381'10819","last_scrub_stamp":"2024-09-18T08:17:42.813063+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:14:43.412748+0000","last_clean_scrub_stamp":"2024-09-18T08:17:42.813063+0000","objects_scrubbed":1926,"log_size":4714,"log_dups_size":0,"ondisk_log_size":4714,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:18:59.853937+0000","scrub_duration":861,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117768192,"num_objects":1797,"num_object_clones":0,"num_object_copies":5391,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1797,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10927,"num_write_kb":407168,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.b","version":"271'15","reported_seq":775,"reported_epoch":383,"state":"active+clean","last_fresh":"2024-09-18T08:17:44.425352+0000","last_change":"2024-09-18T08:16:13.523741+0000","last_active":"2024-09-18T08:17:44.425352+0000","last_peered":"2024-09-18T08:17:44.425352+0000","last_clean":"2024-09-18T08:17:44.425352+0000","last_became_active":"2024-09-18T08:16:13.480279+0000","last_became_peered":"2024-09-18T08:16:13.480279+0000","last_unstale":"2024-09-18T08:17:44.425352+0000","last_undegraded":"2024-09-18T08:17:44.425352+0000","last_fullsized":"2024-09-18T08:17:44.425352+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'9","last_scrub_stamp":"2024-09-18T08:10:47.928596+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_clean_scrub_stamp":"2024-09-18T08:10:47.928596+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":16,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,3],"acting":[0,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":102,"reported_epoch":382,"state":"active+clean","last_fresh":"2024-09-18T08:17:43.528617+0000","last_change":"2024-09-18T08:15:41.443068+0000","last_active":"2024-09-18T08:17:43.528617+0000","last_peered":"2024-09-18T08:17:43.528617+0000","last_clean":"2024-09-18T08:17:43.528617+0000","last_became_active":"2024-09-18T08:15:41.442697+0000","last_became_peered":"2024-09-18T08:15:41.442697+0000","last_unstale":"2024-09-18T08:17:43.528617+0000","last_undegraded":"2024-09-18T08:17:43.528617+0000","last_fullsized":"2024-09-18T08:17:43.528617+0000","mapping_epoch":336,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":337,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:11:22.076452+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:46.860092+0000","last_clean_scrub_stamp":"2024-09-18T08:11:22.076452+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:18:24.518634+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":703,"reported_epoch":382,"state":"active+clean","last_fresh":"2024-09-18T08:17:43.537225+0000","last_change":"2024-09-18T08:16:46.007260+0000","last_active":"2024-09-18T08:17:43.537225+0000","last_peered":"2024-09-18T08:17:43.537225+0000","last_clean":"2024-09-18T08:17:43.537225+0000","last_became_active":"2024-09-18T08:16:13.465129+0000","last_became_peered":"2024-09-18T08:16:13.465129+0000","last_unstale":"2024-09-18T08:17:43.537225+0000","last_undegraded":"2024-09-18T08:17:43.537225+0000","last_fullsized":"2024-09-18T08:17:43.537225+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:16:46.007194+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:50.886830+0000","last_clean_scrub_stamp":"2024-09-18T08:16:46.007194+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:17:53.322101+0000","scrub_duration":27,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":709,"reported_epoch":383,"state":"active+clean","last_fresh":"2024-09-18T08:17:44.431443+0000","last_change":"2024-09-18T08:14:05.585049+0000","last_active":"2024-09-18T08:17:44.431443+0000","last_peered":"2024-09-18T08:17:44.431443+0000","last_clean":"2024-09-18T08:17:44.431443+0000","last_became_active":"2024-09-18T08:14:05.583887+0000","last_became_peered":"2024-09-18T08:14:05.583887+0000","last_unstale":"2024-09-18T08:17:44.431443+0000","last_undegraded":"2024-09-18T08:17:44.431443+0000","last_fullsized":"2024-09-18T08:17:44.431443+0000","mapping_epoch":292,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":293,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:12:38.702798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_clean_scrub_stamp":"2024-09-18T08:12:38.702798+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,3],"acting":[0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.4","version":"383'10927","reported_seq":18971,"reported_epoch":383,"state":"active+clean","last_fresh":"2024-09-18T08:17:44.742888+0000","last_change":"2024-09-18T08:17:26.449992+0000","last_active":"2024-09-18T08:17:44.742888+0000","last_peered":"2024-09-18T08:17:44.742888+0000","last_clean":"2024-09-18T08:17:44.742888+0000","last_became_active":"2024-09-18T08:17:26.419323+0000","last_became_peered":"2024-09-18T08:17:26.419323+0000","last_unstale":"2024-09-18T08:17:44.742888+0000","last_undegraded":"2024-09-18T08:17:44.742888+0000","last_fullsized":"2024-09-18T08:17:44.742888+0000","mapping_epoch":377,"log_start":"357'7143","ondisk_log_start":"357'7143","created":311,"last_epoch_clean":378,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:14:43.412748+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:14:43.412748+0000","last_clean_scrub_stamp":"2024-09-18T08:14:43.412748+0000","objects_scrubbed":0,"log_size":3784,"log_dups_size":0,"ondisk_log_size":3784,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":114753536,"num_objects":1751,"num_object_clones":0,"num_object_copies":5253,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1751,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10927,"num_write_kb":405696,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3934,"num_bytes_recovered":257818624,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1],"acting":[0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.a","version":"271'35","reported_seq":612,"reported_epoch":381,"state":"active+clean","last_fresh":"2024-09-18T08:17:33.482104+0000","last_change":"2024-09-18T08:15:43.282823+0000","last_active":"2024-09-18T08:17:33.482104+0000","last_peered":"2024-09-18T08:17:33.482104+0000","last_clean":"2024-09-18T08:17:33.482104+0000","last_became_active":"2024-09-18T08:15:41.441186+0000","last_became_peered":"2024-09-18T08:15:41.441186+0000","last_unstale":"2024-09-18T08:17:33.482104+0000","last_undegraded":"2024-09-18T08:17:33.482104+0000","last_fullsized":"2024-09-18T08:17:33.482104+0000","mapping_epoch":336,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":337,"parent":"0.0","parent_split_bits":0,"last_scrub":"271'35","last_scrub_stamp":"2024-09-18T08:15:43.282736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_clean_scrub_stamp":"2024-09-18T08:15:43.282736+0000","objects_scrubbed":9,"log_size":35,"log_dups_size":0,"ondisk_log_size":35,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":55,"num_read_kb":44,"num_write":36,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":44,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,5],"acting":[1,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":102,"reported_epoch":382,"state":"active+clean","last_fresh":"2024-09-18T08:17:43.529065+0000","last_change":"2024-09-18T08:15:41.452272+0000","last_active":"2024-09-18T08:17:43.529065+0000","last_peered":"2024-09-18T08:17:43.529065+0000","last_clean":"2024-09-18T08:17:43.529065+0000","last_became_active":"2024-09-18T08:15:41.446405+0000","last_became_peered":"2024-09-18T08:15:41.446405+0000","last_unstale":"2024-09-18T08:17:43.529065+0000","last_undegraded":"2024-09-18T08:17:43.529065+0000","last_fullsized":"2024-09-18T08:17:43.529065+0000","mapping_epoch":336,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":337,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:12:31.205468+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:46.860092+0000","last_clean_scrub_stamp":"2024-09-18T08:12:31.205468+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:18:27.536995+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,3],"acting":[5,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":411,"reported_epoch":382,"state":"active+clean","last_fresh":"2024-09-18T08:17:43.543274+0000","last_change":"2024-09-18T08:15:41.447379+0000","last_active":"2024-09-18T08:17:43.543274+0000","last_peered":"2024-09-18T08:17:43.543274+0000","last_clean":"2024-09-18T08:17:43.543274+0000","last_became_active":"2024-09-18T08:15:41.447075+0000","last_became_peered":"2024-09-18T08:15:41.447075+0000","last_unstale":"2024-09-18T08:17:43.543274+0000","last_undegraded":"2024-09-18T08:17:43.543274+0000","last_fullsized":"2024-09-18T08:17:43.543274+0000","mapping_epoch":336,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":337,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:15:10.767720+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:50.886830+0000","last_clean_scrub_stamp":"2024-09-18T08:15:10.767720+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":533,"reported_epoch":382,"state":"active+clean","last_fresh":"2024-09-18T08:17:43.537314+0000","last_change":"2024-09-18T08:15:41.722212+0000","last_active":"2024-09-18T08:17:43.537314+0000","last_peered":"2024-09-18T08:17:43.537314+0000","last_clean":"2024-09-18T08:17:43.537314+0000","last_became_active":"2024-09-18T08:14:05.578053+0000","last_became_peered":"2024-09-18T08:14:05.578053+0000","last_unstale":"2024-09-18T08:17:43.537314+0000","last_undegraded":"2024-09-18T08:17:43.537314+0000","last_fullsized":"2024-09-18T08:17:43.537314+0000","mapping_epoch":292,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":293,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:15:41.722163+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_clean_scrub_stamp":"2024-09-18T08:15:41.722163+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.7","version":"382'10905","reported_seq":21504,"reported_epoch":382,"state":"active+clean","last_fresh":"2024-09-18T08:17:44.295854+0000","last_change":"2024-09-18T08:17:06.465603+0000","last_active":"2024-09-18T08:17:44.295854+0000","last_peered":"2024-09-18T08:17:44.295854+0000","last_clean":"2024-09-18T08:17:44.295854+0000","last_became_active":"2024-09-18T08:17:06.445544+0000","last_became_peered":"2024-09-18T08:17:06.445544+0000","last_unstale":"2024-09-18T08:17:44.295854+0000","last_undegraded":"2024-09-18T08:17:44.295854+0000","last_fullsized":"2024-09-18T08:17:44.295854+0000","mapping_epoch":370,"log_start":"360'7405","ondisk_log_start":"360'7405","created":311,"last_epoch_clean":371,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:14:43.412748+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:14:43.412748+0000","last_clean_scrub_stamp":"2024-09-18T08:14:43.412748+0000","objects_scrubbed":0,"log_size":3500,"log_dups_size":0,"ondisk_log_size":3500,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":115802112,"num_objects":1767,"num_object_clones":0,"num_object_copies":5301,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1767,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10905,"num_write_kb":405504,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5204,"num_bytes_recovered":341049344,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.9","version":"271'18","reported_seq":419,"reported_epoch":381,"state":"active+clean","last_fresh":"2024-09-18T08:17:33.479674+0000","last_change":"2024-09-18T08:16:13.525130+0000","last_active":"2024-09-18T08:17:33.479674+0000","last_peered":"2024-09-18T08:17:33.479674+0000","last_clean":"2024-09-18T08:17:33.479674+0000","last_became_active":"2024-09-18T08:16:13.482580+0000","last_became_peered":"2024-09-18T08:16:13.482580+0000","last_unstale":"2024-09-18T08:17:33.479674+0000","last_undegraded":"2024-09-18T08:17:33.479674+0000","last_fullsized":"2024-09-18T08:17:33.479674+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"271'18","last_scrub_stamp":"2024-09-18T08:15:49.666776+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_clean_scrub_stamp":"2024-09-18T08:15:49.666776+0000","objects_scrubbed":5,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":33,"num_read_kb":24,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":880,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,6],"acting":[1,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":556,"reported_epoch":381,"state":"active+clean","last_fresh":"2024-09-18T08:17:35.673112+0000","last_change":"2024-09-18T08:17:35.673112+0000","last_active":"2024-09-18T08:17:35.673112+0000","last_peered":"2024-09-18T08:17:35.673112+0000","last_clean":"2024-09-18T08:17:35.673112+0000","last_became_active":"2024-09-18T08:16:13.462268+0000","last_became_peered":"2024-09-18T08:16:13.462268+0000","last_unstale":"2024-09-18T08:17:35.673112+0000","last_undegraded":"2024-09-18T08:17:35.673112+0000","last_fullsized":"2024-09-18T08:17:35.673112+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:17:35.673047+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:46.860092+0000","last_clean_scrub_stamp":"2024-09-18T08:17:35.673047+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:18:39.728756+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,5],"acting":[1,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":701,"reported_epoch":382,"state":"active+clean","last_fresh":"2024-09-18T08:17:43.535019+0000","last_change":"2024-09-18T08:17:40.958509+0000","last_active":"2024-09-18T08:17:43.535019+0000","last_peered":"2024-09-18T08:17:43.535019+0000","last_clean":"2024-09-18T08:17:43.535019+0000","last_became_active":"2024-09-18T08:14:39.399301+0000","last_became_peered":"2024-09-18T08:14:39.399301+0000","last_unstale":"2024-09-18T08:17:43.535019+0000","last_undegraded":"2024-09-18T08:17:43.535019+0000","last_fullsized":"2024-09-18T08:17:43.535019+0000","mapping_epoch":307,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":308,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:17:40.958275+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:50.886830+0000","last_clean_scrub_stamp":"2024-09-18T08:17:40.958275+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:18:57.914660+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":703,"reported_epoch":382,"state":"active+clean","last_fresh":"2024-09-18T08:17:43.547948+0000","last_change":"2024-09-18T08:17:09.487909+0000","last_active":"2024-09-18T08:17:43.547948+0000","last_peered":"2024-09-18T08:17:43.547948+0000","last_clean":"2024-09-18T08:17:43.547948+0000","last_became_active":"2024-09-18T08:14:40.356451+0000","last_became_peered":"2024-09-18T08:14:40.356451+0000","last_unstale":"2024-09-18T08:17:43.547948+0000","last_undegraded":"2024-09-18T08:17:43.547948+0000","last_fullsized":"2024-09-18T08:17:43.547948+0000","mapping_epoch":308,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":309,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:17:09.487823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_clean_scrub_stamp":"2024-09-18T08:17:09.487823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:18:35.062728+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.6","version":"382'10743","reported_seq":23099,"reported_epoch":382,"state":"active+clean","last_fresh":"2024-09-18T08:17:43.677300+0000","last_change":"2024-09-18T08:17:42.087296+0000","last_active":"2024-09-18T08:17:43.677300+0000","last_peered":"2024-09-18T08:17:43.677300+0000","last_clean":"2024-09-18T08:17:43.677300+0000","last_became_active":"2024-09-18T08:16:53.541405+0000","last_became_peered":"2024-09-18T08:16:53.541405+0000","last_unstale":"2024-09-18T08:17:43.677300+0000","last_undegraded":"2024-09-18T08:17:43.677300+0000","last_fullsized":"2024-09-18T08:17:43.677300+0000","mapping_epoch":363,"log_start":"358'7117","ondisk_log_start":"358'7117","created":311,"last_epoch_clean":364,"parent":"0.0","parent_split_bits":0,"last_scrub":"381'10659","last_scrub_stamp":"2024-09-18T08:17:42.087203+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:14:43.412748+0000","last_clean_scrub_stamp":"2024-09-18T08:17:42.087203+0000","objects_scrubbed":1978,"log_size":3626,"log_dups_size":0,"ondisk_log_size":3626,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:19:00.210365+0000","scrub_duration":518,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":123011072,"num_objects":1877,"num_object_clones":0,"num_object_copies":5631,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1877,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10743,"num_write_kb":403840,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6069,"num_bytes_recovered":397737984,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,1],"acting":[5,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.8","version":"271'27","reported_seq":141,"reported_epoch":382,"state":"active+clean","last_fresh":"2024-09-18T08:17:43.527952+0000","last_change":"2024-09-18T08:16:13.502022+0000","last_active":"2024-09-18T08:17:43.527952+0000","last_peered":"2024-09-18T08:17:43.527952+0000","last_clean":"2024-09-18T08:17:43.527952+0000","last_became_active":"2024-09-18T08:16:13.451720+0000","last_became_peered":"2024-09-18T08:16:13.451720+0000","last_unstale":"2024-09-18T08:17:43.527952+0000","last_undegraded":"2024-09-18T08:17:43.527952+0000","last_fullsized":"2024-09-18T08:17:43.527952+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"271'27","last_scrub_stamp":"2024-09-18T08:15:00.680764+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_clean_scrub_stamp":"2024-09-18T08:15:00.680764+0000","objects_scrubbed":7,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":44,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":28,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,3],"acting":[5,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.f","version":"62'2","reported_seq":98,"reported_epoch":383,"state":"active+clean","last_fresh":"2024-09-18T08:17:44.417845+0000","last_change":"2024-09-18T08:16:13.496793+0000","last_active":"2024-09-18T08:17:44.417845+0000","last_peered":"2024-09-18T08:17:44.417845+0000","last_clean":"2024-09-18T08:17:44.417845+0000","last_became_active":"2024-09-18T08:16:13.450206+0000","last_became_peered":"2024-09-18T08:16:13.450206+0000","last_unstale":"2024-09-18T08:17:44.417845+0000","last_undegraded":"2024-09-18T08:17:44.417845+0000","last_fullsized":"2024-09-18T08:17:44.417845+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"62'2","last_scrub_stamp":"2024-09-18T08:10:53.580444+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:46.860092+0000","last_clean_scrub_stamp":"2024-09-18T08:10:53.580444+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":92,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,5],"acting":[0,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.9","version":"349'17","reported_seq":533,"reported_epoch":381,"state":"active+clean","last_fresh":"2024-09-18T08:17:33.479403+0000","last_change":"2024-09-18T08:16:13.513889+0000","last_active":"2024-09-18T08:17:33.479403+0000","last_peered":"2024-09-18T08:17:33.479403+0000","last_clean":"2024-09-18T08:17:33.479403+0000","last_became_active":"2024-09-18T08:16:13.474474+0000","last_became_peered":"2024-09-18T08:16:13.474474+0000","last_unstale":"2024-09-18T08:17:33.479403+0000","last_undegraded":"2024-09-18T08:17:33.479403+0000","last_fullsized":"2024-09-18T08:17:33.479403+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"294'14","last_scrub_stamp":"2024-09-18T08:14:28.686640+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:50.886830+0000","last_clean_scrub_stamp":"2024-09-18T08:14:28.686640+0000","objects_scrubbed":1,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,3],"acting":[1,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":102,"reported_epoch":382,"state":"active+clean","last_fresh":"2024-09-18T08:17:43.528549+0000","last_change":"2024-09-18T08:15:41.454432+0000","last_active":"2024-09-18T08:17:43.528549+0000","last_peered":"2024-09-18T08:17:43.528549+0000","last_clean":"2024-09-18T08:17:43.528549+0000","last_became_active":"2024-09-18T08:15:41.449889+0000","last_became_peered":"2024-09-18T08:15:41.449889+0000","last_unstale":"2024-09-18T08:17:43.528549+0000","last_undegraded":"2024-09-18T08:17:43.528549+0000","last_fullsized":"2024-09-18T08:17:43.528549+0000","mapping_epoch":336,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":337,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:15:13.161614+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_clean_scrub_stamp":"2024-09-18T08:15:13.161614+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":623,"reported_epoch":383,"state":"active+clean","last_fresh":"2024-09-18T08:17:44.418034+0000","last_change":"2024-09-18T08:15:41.448597+0000","last_active":"2024-09-18T08:17:44.418034+0000","last_peered":"2024-09-18T08:17:44.418034+0000","last_clean":"2024-09-18T08:17:44.418034+0000","last_became_active":"2024-09-18T08:15:41.446544+0000","last_became_peered":"2024-09-18T08:15:41.446544+0000","last_unstale":"2024-09-18T08:17:44.418034+0000","last_undegraded":"2024-09-18T08:17:44.418034+0000","last_fullsized":"2024-09-18T08:17:44.418034+0000","mapping_epoch":336,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":337,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:13:06.860320+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:46.860092+0000","last_clean_scrub_stamp":"2024-09-18T08:13:06.860320+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.17","version":"271'12","reported_seq":745,"reported_epoch":382,"state":"active+clean","last_fresh":"2024-09-18T08:17:43.549657+0000","last_change":"2024-09-18T08:14:40.365750+0000","last_active":"2024-09-18T08:17:43.549657+0000","last_peered":"2024-09-18T08:17:43.549657+0000","last_clean":"2024-09-18T08:17:43.549657+0000","last_became_active":"2024-09-18T08:14:40.355751+0000","last_became_peered":"2024-09-18T08:14:40.355751+0000","last_unstale":"2024-09-18T08:17:43.549657+0000","last_undegraded":"2024-09-18T08:17:43.549657+0000","last_fullsized":"2024-09-18T08:17:43.549657+0000","mapping_epoch":308,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":309,"parent":"0.0","parent_split_bits":0,"last_scrub":"271'12","last_scrub_stamp":"2024-09-18T08:14:33.700931+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_clean_scrub_stamp":"2024-09-18T08:14:33.700931+0000","objects_scrubbed":3,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":18,"num_read_kb":15,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":102,"reported_epoch":382,"state":"active+clean","last_fresh":"2024-09-18T08:17:43.528741+0000","last_change":"2024-09-18T08:15:41.448984+0000","last_active":"2024-09-18T08:17:43.528741+0000","last_peered":"2024-09-18T08:17:43.528741+0000","last_clean":"2024-09-18T08:17:43.528741+0000","last_became_active":"2024-09-18T08:15:41.445332+0000","last_became_peered":"2024-09-18T08:15:41.445332+0000","last_unstale":"2024-09-18T08:17:43.528741+0000","last_undegraded":"2024-09-18T08:17:43.528741+0000","last_fullsized":"2024-09-18T08:17:43.528741+0000","mapping_epoch":336,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":337,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:15:07.747147+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:50.886830+0000","last_clean_scrub_stamp":"2024-09-18T08:15:07.747147+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,1],"acting":[5,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":468,"reported_epoch":382,"state":"active+clean","last_fresh":"2024-09-18T08:17:43.529289+0000","last_change":"2024-09-18T08:16:13.453780+0000","last_active":"2024-09-18T08:17:43.529289+0000","last_peered":"2024-09-18T08:17:43.529289+0000","last_clean":"2024-09-18T08:17:43.529289+0000","last_became_active":"2024-09-18T08:16:13.452993+0000","last_became_peered":"2024-09-18T08:16:13.452993+0000","last_unstale":"2024-09-18T08:17:43.529289+0000","last_undegraded":"2024-09-18T08:17:43.529289+0000","last_fullsized":"2024-09-18T08:17:43.529289+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:12:54.661578+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_clean_scrub_stamp":"2024-09-18T08:12:54.661578+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:18:28.558405+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.16","version":"271'15","reported_seq":729,"reported_epoch":383,"state":"active+clean","last_fresh":"2024-09-18T08:17:44.426108+0000","last_change":"2024-09-18T08:17:32.508478+0000","last_active":"2024-09-18T08:17:44.426108+0000","last_peered":"2024-09-18T08:17:44.426108+0000","last_clean":"2024-09-18T08:17:44.426108+0000","last_became_active":"2024-09-18T08:14:05.578519+0000","last_became_peered":"2024-09-18T08:14:05.578519+0000","last_unstale":"2024-09-18T08:17:44.426108+0000","last_undegraded":"2024-09-18T08:17:44.426108+0000","last_fullsized":"2024-09-18T08:17:44.426108+0000","mapping_epoch":292,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":293,"parent":"0.0","parent_split_bits":0,"last_scrub":"271'15","last_scrub_stamp":"2024-09-18T08:17:32.508401+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_clean_scrub_stamp":"2024-09-18T08:17:32.508401+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:18:58.925490+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":330,"reported_epoch":383,"state":"active+clean","last_fresh":"2024-09-18T08:17:44.430628+0000","last_change":"2024-09-18T08:16:13.460704+0000","last_active":"2024-09-18T08:17:44.430628+0000","last_peered":"2024-09-18T08:17:44.430628+0000","last_clean":"2024-09-18T08:17:44.430628+0000","last_became_active":"2024-09-18T08:16:13.460000+0000","last_became_peered":"2024-09-18T08:16:13.460000+0000","last_unstale":"2024-09-18T08:17:44.430628+0000","last_undegraded":"2024-09-18T08:17:44.430628+0000","last_fullsized":"2024-09-18T08:17:44.430628+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:12:03.433407+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:46.860092+0000","last_clean_scrub_stamp":"2024-09-18T08:12:03.433407+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,3],"acting":[0,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":698,"reported_epoch":382,"state":"active+clean","last_fresh":"2024-09-18T08:17:43.549425+0000","last_change":"2024-09-18T08:17:06.569916+0000","last_active":"2024-09-18T08:17:43.549425+0000","last_peered":"2024-09-18T08:17:43.549425+0000","last_clean":"2024-09-18T08:17:43.549425+0000","last_became_active":"2024-09-18T08:14:40.356531+0000","last_became_peered":"2024-09-18T08:14:40.356531+0000","last_unstale":"2024-09-18T08:17:43.549425+0000","last_undegraded":"2024-09-18T08:17:43.549425+0000","last_fullsized":"2024-09-18T08:17:43.549425+0000","mapping_epoch":308,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":309,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:17:06.569828+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:50.886830+0000","last_clean_scrub_stamp":"2024-09-18T08:17:06.569828+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:18:35.645500+0000","scrub_duration":31,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":707,"reported_epoch":382,"state":"active+clean","last_fresh":"2024-09-18T08:17:43.535613+0000","last_change":"2024-09-18T08:16:49.005144+0000","last_active":"2024-09-18T08:17:43.535613+0000","last_peered":"2024-09-18T08:17:43.535613+0000","last_clean":"2024-09-18T08:17:43.535613+0000","last_became_active":"2024-09-18T08:16:13.466171+0000","last_became_peered":"2024-09-18T08:16:13.466171+0000","last_unstale":"2024-09-18T08:17:43.535613+0000","last_undegraded":"2024-09-18T08:17:43.535613+0000","last_fullsized":"2024-09-18T08:17:43.535613+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:16:49.005064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_clean_scrub_stamp":"2024-09-18T08:16:49.005064+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:18:17.185217+0000","scrub_duration":22,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.15","version":"271'15","reported_seq":120,"reported_epoch":382,"state":"active+clean","last_fresh":"2024-09-18T08:17:43.529181+0000","last_change":"2024-09-18T08:17:32.638867+0000","last_active":"2024-09-18T08:17:43.529181+0000","last_peered":"2024-09-18T08:17:43.529181+0000","last_clean":"2024-09-18T08:17:43.529181+0000","last_became_active":"2024-09-18T08:15:41.447339+0000","last_became_peered":"2024-09-18T08:15:41.447339+0000","last_unstale":"2024-09-18T08:17:43.529181+0000","last_undegraded":"2024-09-18T08:17:43.529181+0000","last_fullsized":"2024-09-18T08:17:43.529181+0000","mapping_epoch":336,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":337,"parent":"0.0","parent_split_bits":0,"last_scrub":"271'15","last_scrub_stamp":"2024-09-18T08:17:32.638811+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_clean_scrub_stamp":"2024-09-18T08:17:32.638811+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:18:33.277962+0000","scrub_duration":20,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":25,"num_read_kb":19,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":711,"reported_epoch":383,"state":"active+clean","last_fresh":"2024-09-18T08:17:44.419957+0000","last_change":"2024-09-18T08:14:40.357339+0000","last_active":"2024-09-18T08:17:44.419957+0000","last_peered":"2024-09-18T08:17:44.419957+0000","last_clean":"2024-09-18T08:17:44.419957+0000","last_became_active":"2024-09-18T08:14:40.355709+0000","last_became_peered":"2024-09-18T08:14:40.355709+0000","last_unstale":"2024-09-18T08:17:44.419957+0000","last_undegraded":"2024-09-18T08:17:44.419957+0000","last_fullsized":"2024-09-18T08:17:44.419957+0000","mapping_epoch":308,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":309,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:14:28.097629+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:46.860092+0000","last_clean_scrub_stamp":"2024-09-18T08:14:28.097629+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1],"acting":[0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.14","version":"310'16","reported_seq":929,"reported_epoch":382,"state":"active+clean","last_fresh":"2024-09-18T08:17:43.548312+0000","last_change":"2024-09-18T08:17:39.424179+0000","last_active":"2024-09-18T08:17:43.548312+0000","last_peered":"2024-09-18T08:17:43.548312+0000","last_clean":"2024-09-18T08:17:43.548312+0000","last_became_active":"2024-09-18T08:14:40.358971+0000","last_became_peered":"2024-09-18T08:14:40.358971+0000","last_unstale":"2024-09-18T08:17:43.548312+0000","last_undegraded":"2024-09-18T08:17:43.548312+0000","last_fullsized":"2024-09-18T08:17:43.548312+0000","mapping_epoch":308,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":309,"parent":"0.0","parent_split_bits":0,"last_scrub":"310'16","last_scrub_stamp":"2024-09-18T08:17:39.424098+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:50.886830+0000","last_clean_scrub_stamp":"2024-09-18T08:17:39.424098+0000","objects_scrubbed":1,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:19:08.498783+0000","scrub_duration":16,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,1],"acting":[3,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":424,"reported_epoch":382,"state":"active+clean","last_fresh":"2024-09-18T08:17:43.536012+0000","last_change":"2024-09-18T08:16:51.923862+0000","last_active":"2024-09-18T08:17:43.536012+0000","last_peered":"2024-09-18T08:17:43.536012+0000","last_clean":"2024-09-18T08:17:43.536012+0000","last_became_active":"2024-09-18T08:16:13.467534+0000","last_became_peered":"2024-09-18T08:16:13.467534+0000","last_unstale":"2024-09-18T08:17:43.536012+0000","last_undegraded":"2024-09-18T08:17:43.536012+0000","last_fullsized":"2024-09-18T08:17:43.536012+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:16:51.923777+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_clean_scrub_stamp":"2024-09-18T08:16:51.923777+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:17:59.666455+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,0],"acting":[2,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.14","version":"271'20","reported_seq":759,"reported_epoch":382,"state":"active+clean","last_fresh":"2024-09-18T08:17:43.552207+0000","last_change":"2024-09-18T08:15:52.819894+0000","last_active":"2024-09-18T08:17:43.552207+0000","last_peered":"2024-09-18T08:17:43.552207+0000","last_clean":"2024-09-18T08:17:43.552207+0000","last_became_active":"2024-09-18T08:14:40.358983+0000","last_became_peered":"2024-09-18T08:14:40.358983+0000","last_unstale":"2024-09-18T08:17:43.552207+0000","last_undegraded":"2024-09-18T08:17:43.552207+0000","last_fullsized":"2024-09-18T08:17:43.552207+0000","mapping_epoch":308,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":309,"parent":"0.0","parent_split_bits":0,"last_scrub":"271'20","last_scrub_stamp":"2024-09-18T08:15:52.819801+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_clean_scrub_stamp":"2024-09-18T08:15:52.819801+0000","objects_scrubbed":5,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":16,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":25,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":362,"reported_epoch":382,"state":"active+clean","last_fresh":"2024-09-18T08:17:43.543494+0000","last_change":"2024-09-18T08:17:30.909075+0000","last_active":"2024-09-18T08:17:43.543494+0000","last_peered":"2024-09-18T08:17:43.543494+0000","last_clean":"2024-09-18T08:17:43.543494+0000","last_became_active":"2024-09-18T08:16:13.461199+0000","last_became_peered":"2024-09-18T08:16:13.461199+0000","last_unstale":"2024-09-18T08:17:43.543494+0000","last_undegraded":"2024-09-18T08:17:43.543494+0000","last_fullsized":"2024-09-18T08:17:43.543494+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:17:30.908835+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:46.860092+0000","last_clean_scrub_stamp":"2024-09-18T08:17:30.908835+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:18:34.331728+0000","scrub_duration":22,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,0],"acting":[2,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.15","version":"338'12","reported_seq":131,"reported_epoch":382,"state":"active+clean","last_fresh":"2024-09-18T08:17:43.528950+0000","last_change":"2024-09-18T08:15:41.489414+0000","last_active":"2024-09-18T08:17:43.528950+0000","last_peered":"2024-09-18T08:17:43.528950+0000","last_clean":"2024-09-18T08:17:43.528950+0000","last_became_active":"2024-09-18T08:15:41.455243+0000","last_became_peered":"2024-09-18T08:15:41.455243+0000","last_unstale":"2024-09-18T08:17:43.528950+0000","last_undegraded":"2024-09-18T08:17:43.528950+0000","last_fullsized":"2024-09-18T08:17:43.528950+0000","mapping_epoch":336,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":337,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'8","last_scrub_stamp":"2024-09-18T08:10:54.148986+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:50.886830+0000","last_clean_scrub_stamp":"2024-09-18T08:10:54.148986+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:18:23.530255+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":694,"reported_epoch":383,"state":"active+clean","last_fresh":"2024-09-18T08:17:44.422787+0000","last_change":"2024-09-18T08:17:39.513217+0000","last_active":"2024-09-18T08:17:44.422787+0000","last_peered":"2024-09-18T08:17:44.422787+0000","last_clean":"2024-09-18T08:17:44.422787+0000","last_became_active":"2024-09-18T08:16:13.472297+0000","last_became_peered":"2024-09-18T08:16:13.472297+0000","last_unstale":"2024-09-18T08:17:44.422787+0000","last_undegraded":"2024-09-18T08:17:44.422787+0000","last_fullsized":"2024-09-18T08:17:44.422787+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:17:39.513160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_clean_scrub_stamp":"2024-09-18T08:17:39.513160+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:18:41.265902+0000","scrub_duration":31,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,5],"acting":[0,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.13","version":"271'19","reported_seq":397,"reported_epoch":381,"state":"active+clean","last_fresh":"2024-09-18T08:17:39.665796+0000","last_change":"2024-09-18T08:17:39.665796+0000","last_active":"2024-09-18T08:17:39.665796+0000","last_peered":"2024-09-18T08:17:39.665796+0000","last_clean":"2024-09-18T08:17:39.665796+0000","last_became_active":"2024-09-18T08:16:13.454640+0000","last_became_peered":"2024-09-18T08:16:13.454640+0000","last_unstale":"2024-09-18T08:17:39.665796+0000","last_undegraded":"2024-09-18T08:17:39.665796+0000","last_fullsized":"2024-09-18T08:17:39.665796+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"271'19","last_scrub_stamp":"2024-09-18T08:17:39.665588+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_clean_scrub_stamp":"2024-09-18T08:17:39.665588+0000","objects_scrubbed":5,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:18:55.676655+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":31,"num_read_kb":24,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,5],"acting":[1,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":328,"reported_epoch":382,"state":"active+clean","last_fresh":"2024-09-18T08:17:43.534687+0000","last_change":"2024-09-18T08:16:48.007441+0000","last_active":"2024-09-18T08:17:43.534687+0000","last_peered":"2024-09-18T08:17:43.534687+0000","last_clean":"2024-09-18T08:17:43.534687+0000","last_became_active":"2024-09-18T08:16:13.457932+0000","last_became_peered":"2024-09-18T08:16:13.457932+0000","last_unstale":"2024-09-18T08:17:43.534687+0000","last_undegraded":"2024-09-18T08:17:43.534687+0000","last_fullsized":"2024-09-18T08:17:43.534687+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:16:48.007357+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:46.860092+0000","last_clean_scrub_stamp":"2024-09-18T08:16:48.007357+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:18:00.038865+0000","scrub_duration":27,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,0],"acting":[2,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":574,"reported_epoch":381,"state":"active+clean","last_fresh":"2024-09-18T08:17:33.479126+0000","last_change":"2024-09-18T08:15:55.335233+0000","last_active":"2024-09-18T08:17:33.479126+0000","last_peered":"2024-09-18T08:17:33.479126+0000","last_clean":"2024-09-18T08:17:33.479126+0000","last_became_active":"2024-09-18T08:15:41.438048+0000","last_became_peered":"2024-09-18T08:15:41.438048+0000","last_unstale":"2024-09-18T08:17:33.479126+0000","last_undegraded":"2024-09-18T08:17:33.479126+0000","last_fullsized":"2024-09-18T08:17:33.479126+0000","mapping_epoch":336,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":337,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:15:55.334983+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:50.886830+0000","last_clean_scrub_stamp":"2024-09-18T08:15:55.334983+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":709,"reported_epoch":382,"state":"active+clean","last_fresh":"2024-09-18T08:17:43.549672+0000","last_change":"2024-09-18T08:17:37.394735+0000","last_active":"2024-09-18T08:17:43.549672+0000","last_peered":"2024-09-18T08:17:43.549672+0000","last_clean":"2024-09-18T08:17:43.549672+0000","last_became_active":"2024-09-18T08:15:41.464961+0000","last_became_peered":"2024-09-18T08:15:41.464961+0000","last_unstale":"2024-09-18T08:17:43.549672+0000","last_undegraded":"2024-09-18T08:17:43.549672+0000","last_fullsized":"2024-09-18T08:17:43.549672+0000","mapping_epoch":336,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":337,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:17:37.394532+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_clean_scrub_stamp":"2024-09-18T08:17:37.394532+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:18:46.957682+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.12","version":"271'15","reported_seq":562,"reported_epoch":381,"state":"active+clean","last_fresh":"2024-09-18T08:17:33.481998+0000","last_change":"2024-09-18T08:15:52.304936+0000","last_active":"2024-09-18T08:17:33.481998+0000","last_peered":"2024-09-18T08:17:33.481998+0000","last_clean":"2024-09-18T08:17:33.481998+0000","last_became_active":"2024-09-18T08:14:39.416284+0000","last_became_peered":"2024-09-18T08:14:39.416284+0000","last_unstale":"2024-09-18T08:17:33.481998+0000","last_undegraded":"2024-09-18T08:17:33.481998+0000","last_fullsized":"2024-09-18T08:17:33.481998+0000","mapping_epoch":307,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":308,"parent":"0.0","parent_split_bits":0,"last_scrub":"271'15","last_scrub_stamp":"2024-09-18T08:15:52.304832+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_clean_scrub_stamp":"2024-09-18T08:15:52.304832+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":25,"num_read_kb":19,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":495,"reported_epoch":382,"state":"active+clean","last_fresh":"2024-09-18T08:17:43.550913+0000","last_change":"2024-09-18T08:16:13.476132+0000","last_active":"2024-09-18T08:17:43.550913+0000","last_peered":"2024-09-18T08:17:43.550913+0000","last_clean":"2024-09-18T08:17:43.550913+0000","last_became_active":"2024-09-18T08:16:13.475647+0000","last_became_peered":"2024-09-18T08:16:13.475647+0000","last_unstale":"2024-09-18T08:17:43.550913+0000","last_undegraded":"2024-09-18T08:17:43.550913+0000","last_fullsized":"2024-09-18T08:17:43.550913+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:15:58.978411+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:46.860092+0000","last_clean_scrub_stamp":"2024-09-18T08:15:58.978411+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":18,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":685,"reported_epoch":382,"state":"active+clean","last_fresh":"2024-09-18T08:17:43.548119+0000","last_change":"2024-09-18T08:14:39.403741+0000","last_active":"2024-09-18T08:17:43.548119+0000","last_peered":"2024-09-18T08:17:43.548119+0000","last_clean":"2024-09-18T08:17:43.548119+0000","last_became_active":"2024-09-18T08:14:39.402406+0000","last_became_peered":"2024-09-18T08:14:39.402406+0000","last_unstale":"2024-09-18T08:17:43.548119+0000","last_undegraded":"2024-09-18T08:17:43.548119+0000","last_fullsized":"2024-09-18T08:17:43.548119+0000","mapping_epoch":307,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":308,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:11:30.302179+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:50.886830+0000","last_clean_scrub_stamp":"2024-09-18T08:11:30.302179+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:18:20.349750+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.10","version":"69'1","reported_seq":701,"reported_epoch":383,"state":"active+clean","last_fresh":"2024-09-18T08:17:44.417628+0000","last_change":"2024-09-18T08:15:41.480106+0000","last_active":"2024-09-18T08:17:44.417628+0000","last_peered":"2024-09-18T08:17:44.417628+0000","last_clean":"2024-09-18T08:17:44.417628+0000","last_became_active":"2024-09-18T08:15:41.444085+0000","last_became_peered":"2024-09-18T08:15:41.444085+0000","last_unstale":"2024-09-18T08:17:44.417628+0000","last_undegraded":"2024-09-18T08:17:44.417628+0000","last_fullsized":"2024-09-18T08:17:44.417628+0000","mapping_epoch":336,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":337,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'1","last_scrub_stamp":"2024-09-18T08:15:00.219678+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_clean_scrub_stamp":"2024-09-18T08:15:00.219678+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":26,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.11","version":"271'19","reported_seq":828,"reported_epoch":382,"state":"active+clean","last_fresh":"2024-09-18T08:17:43.549469+0000","last_change":"2024-09-18T08:15:41.491797+0000","last_active":"2024-09-18T08:17:43.549469+0000","last_peered":"2024-09-18T08:17:43.549469+0000","last_clean":"2024-09-18T08:17:43.549469+0000","last_became_active":"2024-09-18T08:15:41.465631+0000","last_became_peered":"2024-09-18T08:15:41.465631+0000","last_unstale":"2024-09-18T08:17:43.549469+0000","last_undegraded":"2024-09-18T08:17:43.549469+0000","last_fullsized":"2024-09-18T08:17:43.549469+0000","mapping_epoch":336,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":337,"parent":"0.0","parent_split_bits":0,"last_scrub":"271'19","last_scrub_stamp":"2024-09-18T08:14:38.809490+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_clean_scrub_stamp":"2024-09-18T08:14:38.809490+0000","objects_scrubbed":5,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":32,"num_read_kb":25,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":550,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":102,"reported_epoch":382,"state":"active+clean","last_fresh":"2024-09-18T08:17:43.528709+0000","last_change":"2024-09-18T08:15:41.449283+0000","last_active":"2024-09-18T08:17:43.528709+0000","last_peered":"2024-09-18T08:17:43.528709+0000","last_clean":"2024-09-18T08:17:43.528709+0000","last_became_active":"2024-09-18T08:15:41.445583+0000","last_became_peered":"2024-09-18T08:15:41.445583+0000","last_unstale":"2024-09-18T08:17:43.528709+0000","last_undegraded":"2024-09-18T08:17:43.528709+0000","last_fullsized":"2024-09-18T08:17:43.528709+0000","mapping_epoch":336,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":337,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:10:53.145315+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:46.860092+0000","last_clean_scrub_stamp":"2024-09-18T08:10:53.145315+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:18:14.386231+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":370,"reported_epoch":383,"state":"active+clean","last_fresh":"2024-09-18T08:17:44.417533+0000","last_change":"2024-09-18T08:16:13.444745+0000","last_active":"2024-09-18T08:17:44.417533+0000","last_peered":"2024-09-18T08:17:44.417533+0000","last_clean":"2024-09-18T08:17:44.417533+0000","last_became_active":"2024-09-18T08:16:13.443940+0000","last_became_peered":"2024-09-18T08:16:13.443940+0000","last_unstale":"2024-09-18T08:17:44.417533+0000","last_undegraded":"2024-09-18T08:17:44.417533+0000","last_fullsized":"2024-09-18T08:17:44.417533+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:14:37.743856+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:50.886830+0000","last_clean_scrub_stamp":"2024-09-18T08:14:37.743856+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":684,"reported_epoch":382,"state":"active+clean","last_fresh":"2024-09-18T08:17:43.549008+0000","last_change":"2024-09-18T08:11:29.344444+0000","last_active":"2024-09-18T08:17:43.549008+0000","last_peered":"2024-09-18T08:17:43.549008+0000","last_clean":"2024-09-18T08:17:43.549008+0000","last_became_active":"2024-09-18T08:06:24.128582+0000","last_became_peered":"2024-09-18T08:06:24.128582+0000","last_unstale":"2024-09-18T08:17:43.549008+0000","last_undegraded":"2024-09-18T08:17:43.549008+0000","last_fullsized":"2024-09-18T08:17:43.549008+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:11:29.344382+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_clean_scrub_stamp":"2024-09-18T08:11:29.344382+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:18:19.388633+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.10","version":"271'8","reported_seq":742,"reported_epoch":382,"state":"active+clean","last_fresh":"2024-09-18T08:17:43.544725+0000","last_change":"2024-09-18T08:15:50.790606+0000","last_active":"2024-09-18T08:17:43.544725+0000","last_peered":"2024-09-18T08:17:43.544725+0000","last_clean":"2024-09-18T08:17:43.544725+0000","last_became_active":"2024-09-18T08:14:39.404257+0000","last_became_peered":"2024-09-18T08:14:39.404257+0000","last_unstale":"2024-09-18T08:17:43.544725+0000","last_undegraded":"2024-09-18T08:17:43.544725+0000","last_fullsized":"2024-09-18T08:17:43.544725+0000","mapping_epoch":307,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":308,"parent":"0.0","parent_split_bits":0,"last_scrub":"271'8","last_scrub_stamp":"2024-09-18T08:15:50.790525+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_clean_scrub_stamp":"2024-09-18T08:15:50.790525+0000","objects_scrubbed":2,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":12,"num_read_kb":10,"num_write":8,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":713,"reported_epoch":383,"state":"active+clean","last_fresh":"2024-09-18T08:17:44.418204+0000","last_change":"2024-09-18T08:15:41.467281+0000","last_active":"2024-09-18T08:17:44.418204+0000","last_peered":"2024-09-18T08:17:44.418204+0000","last_clean":"2024-09-18T08:17:44.418204+0000","last_became_active":"2024-09-18T08:15:41.466964+0000","last_became_peered":"2024-09-18T08:15:41.466964+0000","last_unstale":"2024-09-18T08:17:44.418204+0000","last_undegraded":"2024-09-18T08:17:44.418204+0000","last_fullsized":"2024-09-18T08:17:44.418204+0000","mapping_epoch":336,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":337,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:14:33.064199+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:46.860092+0000","last_clean_scrub_stamp":"2024-09-18T08:14:33.064199+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":330,"reported_epoch":383,"state":"active+clean","last_fresh":"2024-09-18T08:17:44.430442+0000","last_change":"2024-09-18T08:16:13.453084+0000","last_active":"2024-09-18T08:17:44.430442+0000","last_peered":"2024-09-18T08:17:44.430442+0000","last_clean":"2024-09-18T08:17:44.430442+0000","last_became_active":"2024-09-18T08:16:13.452169+0000","last_became_peered":"2024-09-18T08:16:13.452169+0000","last_unstale":"2024-09-18T08:17:44.430442+0000","last_undegraded":"2024-09-18T08:17:44.430442+0000","last_fullsized":"2024-09-18T08:17:44.430442+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:13:32.640669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:50.886830+0000","last_clean_scrub_stamp":"2024-09-18T08:13:32.640669+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":20,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,1],"acting":[0,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":500,"reported_epoch":382,"state":"active+clean","last_fresh":"2024-09-18T08:17:43.530390+0000","last_change":"2024-09-18T08:17:31.928942+0000","last_active":"2024-09-18T08:17:43.530390+0000","last_peered":"2024-09-18T08:17:43.530390+0000","last_clean":"2024-09-18T08:17:43.530390+0000","last_became_active":"2024-09-18T08:16:13.491755+0000","last_became_peered":"2024-09-18T08:16:13.491755+0000","last_unstale":"2024-09-18T08:17:43.530390+0000","last_undegraded":"2024-09-18T08:17:43.530390+0000","last_fullsized":"2024-09-18T08:17:43.530390+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:17:31.928876+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_clean_scrub_stamp":"2024-09-18T08:17:31.928876+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:18:33.514219+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,1],"acting":[2,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":546,"reported_epoch":381,"state":"active+clean","last_fresh":"2024-09-18T08:17:33.477053+0000","last_change":"2024-09-18T08:16:13.468068+0000","last_active":"2024-09-18T08:17:33.477053+0000","last_peered":"2024-09-18T08:17:33.477053+0000","last_clean":"2024-09-18T08:17:33.477053+0000","last_became_active":"2024-09-18T08:16:13.466812+0000","last_became_peered":"2024-09-18T08:16:13.466812+0000","last_unstale":"2024-09-18T08:17:33.477053+0000","last_undegraded":"2024-09-18T08:17:33.477053+0000","last_fullsized":"2024-09-18T08:17:33.477053+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:15:45.297240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_clean_scrub_stamp":"2024-09-18T08:15:45.297240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,2],"acting":[1,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":693,"reported_epoch":382,"state":"active+clean","last_fresh":"2024-09-18T08:17:43.544990+0000","last_change":"2024-09-18T08:15:43.836627+0000","last_active":"2024-09-18T08:17:43.544990+0000","last_peered":"2024-09-18T08:17:43.544990+0000","last_clean":"2024-09-18T08:17:43.544990+0000","last_became_active":"2024-09-18T08:14:39.405347+0000","last_became_peered":"2024-09-18T08:14:39.405347+0000","last_unstale":"2024-09-18T08:17:43.544990+0000","last_undegraded":"2024-09-18T08:17:43.544990+0000","last_fullsized":"2024-09-18T08:17:43.544990+0000","mapping_epoch":307,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":308,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:15:43.836549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:46.860092+0000","last_clean_scrub_stamp":"2024-09-18T08:15:43.836549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":20,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"271'19","reported_seq":114,"reported_epoch":382,"state":"active+clean","last_fresh":"2024-09-18T08:17:43.529128+0000","last_change":"2024-09-18T08:15:41.472787+0000","last_active":"2024-09-18T08:17:43.529128+0000","last_peered":"2024-09-18T08:17:43.529128+0000","last_clean":"2024-09-18T08:17:43.529128+0000","last_became_active":"2024-09-18T08:15:41.445198+0000","last_became_peered":"2024-09-18T08:15:41.445198+0000","last_unstale":"2024-09-18T08:17:43.529128+0000","last_undegraded":"2024-09-18T08:17:43.529128+0000","last_fullsized":"2024-09-18T08:17:43.529128+0000","mapping_epoch":336,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":337,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'11","last_scrub_stamp":"2024-09-18T08:10:50.208543+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_clean_scrub_stamp":"2024-09-18T08:10:50.208543+0000","objects_scrubbed":5,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:18:15.381752+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":31,"num_read_kb":24,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,1],"acting":[5,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":710,"reported_epoch":383,"state":"active+clean","last_fresh":"2024-09-18T08:17:44.422969+0000","last_change":"2024-09-18T08:14:40.358776+0000","last_active":"2024-09-18T08:17:44.422969+0000","last_peered":"2024-09-18T08:17:44.422969+0000","last_clean":"2024-09-18T08:17:44.422969+0000","last_became_active":"2024-09-18T08:14:40.358457+0000","last_became_peered":"2024-09-18T08:14:40.358457+0000","last_unstale":"2024-09-18T08:17:44.422969+0000","last_undegraded":"2024-09-18T08:17:44.422969+0000","last_fullsized":"2024-09-18T08:17:44.422969+0000","mapping_epoch":308,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":309,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:12:31.664456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:50.886830+0000","last_clean_scrub_stamp":"2024-09-18T08:12:31.664456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":10,"num_pg":16,"stat_sum":{"num_bytes":1898512408,"num_objects":28970,"num_object_clones":0,"num_object_copies":86910,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":3613,"num_objects_unfound":0,"num_objects_dirty":28970,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":174466,"num_write_kb":6509889,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":59979,"num_bytes_recovered":3930718232,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":7592165376,"data_stored":7592149088,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":61534,"ondisk_log_size":61534,"up":48,"acting":48,"num_store_stats":7},{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":416,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":832,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":26,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":116,"ondisk_log_size":116,"up":96,"acting":96,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":178,"num_object_clones":0,"num_object_copies":534,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":178,"num_whiteouts":0,"num_read":1104,"num_read_kb":861,"num_write":703,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":447,"num_bytes_recovered":8338,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":499712,"data_stored":13350,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3639,"internal_metadata":0},"log_size":688,"ondisk_log_size":688,"up":96,"acting":96,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":1613,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":14,"num_read_kb":14,"num_write":10,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":4563,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":102400,"data_stored":6108,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":6,"ondisk_log_size":6,"up":96,"acting":96,"num_store_stats":8},{"poolid":2,"num_pg":3,"stat_sum":{"num_bytes":408,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":355,"num_read_kb":350,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":408,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1613,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":8,"ondisk_log_size":8,"up":9,"acting":9,"num_store_stats":7},{"poolid":1,"num_pg":11,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1245728,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":869,"ondisk_log_size":869,"up":33,"acting":33,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":0,"seq":240518168759,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":47,"seq":201863463101,"num_pgs":74,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1450296,"kb_used_data":967912,"kb_used_omap":1,"kb_used_meta":482366,"kb_avail":92294856,"statfs":{"total":95995035648,"available":94509932544,"internally_reserved":0,"allocated":991141888,"data_stored":990717476,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":493943238},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":39,"seq":167503724739,"num_pgs":83,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1478040,"kb_used_data":720772,"kb_used_omap":1,"kb_used_meta":757246,"kb_avail":92267112,"statfs":{"total":95995035648,"available":94481522688,"internally_reserved":0,"allocated":738070528,"data_stored":737626642,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":775420358},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":0,"seq":137438953672,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":26,"seq":111669149903,"num_pgs":87,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1633520,"kb_used_data":916508,"kb_used_omap":1,"kb_used_meta":716990,"kb_avail":92111632,"statfs":{"total":95995035648,"available":94322311168,"internally_reserved":0,"allocated":938504192,"data_stored":938035652,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":734198216},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,7,1],"upper_bound":32},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411541,"num_pgs":65,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1458120,"kb_used_data":822836,"kb_used_omap":1,"kb_used_meta":635070,"kb_avail":92287032,"statfs":{"total":95995035648,"available":94501920768,"internally_reserved":0,"allocated":842584064,"data_stored":842155537,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":650312133},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":307,"seq":1318554959910,"num_pgs":84,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":908400,"kb_used_data":762208,"kb_used_omap":7,"kb_used_meta":146168,"kb_avail":92836752,"statfs":{"total":95995035648,"available":95064834048,"internally_reserved":0,"allocated":780500992,"data_stored":780021935,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":8079,"internal_metadata":149676145},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738592,"num_pgs":81,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1885148,"kb_used_data":1249992,"kb_used_omap":1,"kb_used_meta":635070,"kb_avail":91860004,"statfs":{"total":95995035648,"available":94064644096,"internally_reserved":0,"allocated":1279991808,"data_stored":1279549836,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":650312133},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,5,0,3,2],"upper_bound":64},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":266240,"data_stored":262688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1131,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1613,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":436,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":46,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1177,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1567,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":46,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1474,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":94208,"data_stored":2448,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3639,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":1568,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":94208,"data_stored":2492,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":1584,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1430,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":1694,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":660,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":403,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":403,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":416,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":1264914432,"data_stored":1264910360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":756809728,"data_stored":756809728,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":838205440,"data_stored":838205440,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":935661568,"data_stored":935657496,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":2093944832,"data_stored":2093940760,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":732495872,"data_stored":732495872,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":970133504,"data_stored":970129432,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T08:17:47.909 INFO:tasks.cephadm.ceph_manager.ceph:recovered! 2024-09-18T08:17:47.910 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-18T08:17:47.910 INFO:tasks.thrashosds.thrasher:thrash_pg_upmap 2024-09-18T08:17:47.910 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid 779349ca-7593-11ef-bceb-c7b262605968 -- ceph osd dump -f json-pretty 2024-09-18T08:17:48.000 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4 2024-09-18T08:17:48.021 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/779349ca-7593-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T08:17:48.024 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:17:48.025 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:17:48.025 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4' 2024-09-18T08:17:48.041 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/779349ca-7593-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T08:17:48.126 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4 2024-09-18T08:17:48.126 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid 779349ca-7593-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-18T08:17:48.150 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:17:48.151 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:17:48.152 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4' 2024-09-18T08:17:48.252 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0 2024-09-18T08:17:48.328 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:17:48.329 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:17:48.329 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0' 2024-09-18T08:17:48.430 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4 2024-09-18T08:17:48.456 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:17:48.457 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:17:48.457 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4' 2024-09-18T08:17:48.558 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0 2024-09-18T08:17:48.602 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:17:48.603 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:17:48.604 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0' 2024-09-18T08:17:48.705 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5 2024-09-18T08:17:48.732 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:17:48.732 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:17:48.733 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5' 2024-09-18T08:17:48.833 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7 2024-09-18T08:17:48.861 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:17:48.861 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:17:48.862 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7' 2024-09-18T08:17:48.905 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:17:48 smithi050 ceph-mon[32371]: from='client.25813 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-18T08:17:48.905 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:17:48 smithi050 ceph-mon[32371]: pgmap v748: 158 pgs: 2 active+remapped+backfill_toofull, 156 active+clean; 1.8 GiB data, 8.4 GiB used, 528 GiB / 536 GiB avail; 639 B/s rd, 0 B/s wr, 959 op/s; 3613/87507 objects misplaced (4.129%) 2024-09-18T08:17:48.963 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1 2024-09-18T08:17:48.981 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:17:48 smithi007 ceph-mon[29459]: from='client.25813 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-18T08:17:48.981 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:17:48 smithi007 ceph-mon[29459]: pgmap v748: 158 pgs: 2 active+remapped+backfill_toofull, 156 active+clean; 1.8 GiB data, 8.4 GiB used, 528 GiB / 536 GiB avail; 639 B/s rd, 0 B/s wr, 959 op/s; 3613/87507 objects misplaced (4.129%) 2024-09-18T08:17:48.981 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:17:48 smithi007 ceph-mon[39249]: from='client.25813 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-18T08:17:48.981 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:17:48 smithi007 ceph-mon[39249]: pgmap v748: 158 pgs: 2 active+remapped+backfill_toofull, 156 active+clean; 1.8 GiB data, 8.4 GiB used, 528 GiB / 536 GiB avail; 639 B/s rd, 0 B/s wr, 959 op/s; 3613/87507 objects misplaced (4.129%) 2024-09-18T08:17:49.009 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:17:49.010 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:17:49.010 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1' 2024-09-18T08:17:49.111 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2 2024-09-18T08:17:49.155 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:17:49.156 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:17:49.156 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2' 2024-09-18T08:17:49.257 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7 2024-09-18T08:17:49.280 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:17:49.281 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:17:49.281 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7' 2024-09-18T08:17:49.382 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5 2024-09-18T08:17:49.410 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:17:49.411 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:17:49.411 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5' 2024-09-18T08:17:49.512 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1 2024-09-18T08:17:49.547 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:17:49.547 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:17:49.548 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1' 2024-09-18T08:17:49.649 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2 2024-09-18T08:17:49.690 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:17:49.691 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:17:49.692 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2' 2024-09-18T08:17:49.792 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0 2024-09-18T08:17:49.840 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:17:49.840 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:17:49.841 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0' 2024-09-18T08:17:49.873 INFO:journalctl@ceph.iscsi.iscsi.a.smithi050.stdout:Sep 18 08:17:49 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-iscsi-iscsi-a[57782]: debug there is no tcmu-runner data available 2024-09-18T08:17:49.941 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1 2024-09-18T08:17:49.969 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:17:49.970 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:17:49.970 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1' 2024-09-18T08:17:50.071 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4 2024-09-18T08:17:50.096 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:17:50.097 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:17:50.097 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4' 2024-09-18T08:17:50.155 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:17:49 smithi050 ceph-mon[32371]: 4.b scrub starts 2024-09-18T08:17:50.155 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:17:49 smithi050 ceph-mon[32371]: 4.b scrub ok 2024-09-18T08:17:50.155 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:17:49 smithi050 ceph-mon[32371]: from='mgr.24470 172.21.15.7:0/2594390261' entity='mgr.y' 2024-09-18T08:17:50.155 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:17:49 smithi050 ceph-mon[32371]: from='mgr.24470 172.21.15.7:0/2594390261' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-18T08:17:50.155 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:17:49 smithi050 ceph-mon[32371]: osdmap e384: 8 total, 8 up, 6 in 2024-09-18T08:17:50.155 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:17:49 smithi050 ceph-mon[32371]: pgmap v750: 158 pgs: 2 active+remapped+backfilling, 156 active+clean; 1.6 GiB data, 7.8 GiB used, 529 GiB / 536 GiB avail; 853 B/s rd, 0 B/s wr, 1.28k op/s; 1975/77775 objects misplaced (2.539%); 11 MiB/s, 180 objects/s recovering 2024-09-18T08:17:50.155 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:17:49 smithi050 ceph-mon[32371]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-18T08:17:50.198 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7 2024-09-18T08:17:50.225 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:17:50.226 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:17:50.226 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7' 2024-09-18T08:17:50.230 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:17:49 smithi007 ceph-mon[29459]: 4.b scrub starts 2024-09-18T08:17:50.230 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:17:49 smithi007 ceph-mon[29459]: 4.b scrub ok 2024-09-18T08:17:50.230 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:17:49 smithi007 ceph-mon[29459]: from='mgr.24470 172.21.15.7:0/2594390261' entity='mgr.y' 2024-09-18T08:17:50.230 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:17:49 smithi007 ceph-mon[29459]: from='mgr.24470 172.21.15.7:0/2594390261' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-18T08:17:50.230 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:17:49 smithi007 ceph-mon[29459]: osdmap e384: 8 total, 8 up, 6 in 2024-09-18T08:17:50.230 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:17:49 smithi007 ceph-mon[29459]: pgmap v750: 158 pgs: 2 active+remapped+backfilling, 156 active+clean; 1.6 GiB data, 7.8 GiB used, 529 GiB / 536 GiB avail; 853 B/s rd, 0 B/s wr, 1.28k op/s; 1975/77775 objects misplaced (2.539%); 11 MiB/s, 180 objects/s recovering 2024-09-18T08:17:50.230 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:17:49 smithi007 ceph-mon[29459]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-18T08:17:50.231 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:17:49 smithi007 ceph-mon[39249]: 4.b scrub starts 2024-09-18T08:17:50.231 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:17:49 smithi007 ceph-mon[39249]: 4.b scrub ok 2024-09-18T08:17:50.231 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:17:49 smithi007 ceph-mon[39249]: from='mgr.24470 172.21.15.7:0/2594390261' entity='mgr.y' 2024-09-18T08:17:50.231 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:17:49 smithi007 ceph-mon[39249]: from='mgr.24470 172.21.15.7:0/2594390261' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-18T08:17:50.232 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:17:49 smithi007 ceph-mon[39249]: osdmap e384: 8 total, 8 up, 6 in 2024-09-18T08:17:50.232 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:17:49 smithi007 ceph-mon[39249]: pgmap v750: 158 pgs: 2 active+remapped+backfilling, 156 active+clean; 1.6 GiB data, 7.8 GiB used, 529 GiB / 536 GiB avail; 853 B/s rd, 0 B/s wr, 1.28k op/s; 1975/77775 objects misplaced (2.539%); 11 MiB/s, 180 objects/s recovering 2024-09-18T08:17:50.232 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:17:49 smithi007 ceph-mon[39249]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-18T08:17:50.327 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1 2024-09-18T08:17:50.372 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:17:50.373 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:17:50.373 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1' 2024-09-18T08:17:50.474 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3 2024-09-18T08:17:50.515 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:17:50.516 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:17:50.516 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3' 2024-09-18T08:17:50.617 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7 2024-09-18T08:17:50.643 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:17:50.644 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:17:50.644 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7' 2024-09-18T08:17:50.745 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4 2024-09-18T08:17:50.771 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:17:50.771 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:17:50.772 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4' 2024-09-18T08:17:50.872 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5 2024-09-18T08:17:50.897 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:17:50.898 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:17:50.898 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5' 2024-09-18T08:17:50.999 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6 2024-09-18T08:17:51.024 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:17:51.024 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:17:51.025 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6' 2024-09-18T08:17:51.096 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:17:50 smithi007 ceph-mon[29459]: 3.f scrub starts 2024-09-18T08:17:51.097 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:17:50 smithi007 ceph-mon[29459]: 3.f scrub ok 2024-09-18T08:17:51.097 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:17:50 smithi007 ceph-mon[29459]: Health check cleared: PG_BACKFILL_FULL (was: Low space hindering backfill (add storage if this doesn't resolve itself): 2 pgs backfill_toofull) 2024-09-18T08:17:51.097 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:17:50 smithi007 ceph-mon[29459]: Cluster is now healthy 2024-09-18T08:17:51.097 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:17:50 smithi007 ceph-mon[39249]: 3.f scrub starts 2024-09-18T08:17:51.097 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:17:50 smithi007 ceph-mon[39249]: 3.f scrub ok 2024-09-18T08:17:51.097 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:17:50 smithi007 ceph-mon[39249]: Health check cleared: PG_BACKFILL_FULL (was: Low space hindering backfill (add storage if this doesn't resolve itself): 2 pgs backfill_toofull) 2024-09-18T08:17:51.097 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:17:50 smithi007 ceph-mon[39249]: Cluster is now healthy 2024-09-18T08:17:51.126 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5 2024-09-18T08:17:51.151 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:17:51.151 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:17:51.152 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5' 2024-09-18T08:17:51.155 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:17:50 smithi050 ceph-mon[32371]: 3.f scrub starts 2024-09-18T08:17:51.155 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:17:50 smithi050 ceph-mon[32371]: 3.f scrub ok 2024-09-18T08:17:51.155 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:17:50 smithi050 ceph-mon[32371]: Health check cleared: PG_BACKFILL_FULL (was: Low space hindering backfill (add storage if this doesn't resolve itself): 2 pgs backfill_toofull) 2024-09-18T08:17:51.155 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:17:50 smithi050 ceph-mon[32371]: Cluster is now healthy 2024-09-18T08:17:51.166 INFO:teuthology.orchestra.run.smithi007.stdout:{ 2024-09-18T08:17:51.167 INFO:teuthology.orchestra.run.smithi007.stdout: "ops": [], 2024-09-18T08:17:51.167 INFO:teuthology.orchestra.run.smithi007.stdout: "complaint_time": 30, 2024-09-18T08:17:51.167 INFO:teuthology.orchestra.run.smithi007.stdout: "num_blocked_ops": 0 2024-09-18T08:17:51.167 INFO:teuthology.orchestra.run.smithi007.stdout:} 2024-09-18T08:17:51.252 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5 2024-09-18T08:17:51.278 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:17:51.278 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:17:51.279 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5' 2024-09-18T08:17:51.380 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3 2024-09-18T08:17:51.422 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:17:51.422 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:17:51.423 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3' 2024-09-18T08:17:51.524 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7 2024-09-18T08:17:51.549 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:17:51.550 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:17:51.550 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7' 2024-09-18T08:17:51.651 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2 2024-09-18T08:17:51.730 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:17:51.731 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:17:51.731 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2' 2024-09-18T08:17:51.832 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0 2024-09-18T08:17:51.872 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:17:51.873 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:17:51.874 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0' 2024-09-18T08:17:51.907 INFO:teuthology.orchestra.run.smithi007.stderr:noscrub is set 2024-09-18T08:17:51.975 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6 2024-09-18T08:17:52.001 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:17:52.001 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:17:52.002 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6' 2024-09-18T08:17:52.102 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0 2024-09-18T08:17:52.145 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:17:52.146 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:17:52.146 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0' 2024-09-18T08:17:52.155 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:17:51 smithi050 ceph-mon[32371]: 3.11 scrub starts 2024-09-18T08:17:52.155 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:17:51 smithi050 ceph-mon[32371]: 3.11 scrub ok 2024-09-18T08:17:52.155 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:17:51 smithi050 ceph-mon[32371]: pgmap v751: 158 pgs: 2 active+remapped+backfilling, 156 active+clean; 1.5 GiB data, 7.6 GiB used, 529 GiB / 536 GiB avail; 647 B/s rd, 0 B/s wr, 1.21k op/s; 1975/72180 objects misplaced (2.736%); 8.6 MiB/s, 137 objects/s recovering 2024-09-18T08:17:52.155 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:17:51 smithi050 ceph-mon[32371]: from='client.? 172.21.15.7:0/16665188' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-18T08:17:52.155 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:17:51 smithi050 ceph-mon[32371]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-18T08:17:52.222 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:17:51 smithi007 ceph-mon[29459]: 3.11 scrub starts 2024-09-18T08:17:52.222 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:17:51 smithi007 ceph-mon[29459]: 3.11 scrub ok 2024-09-18T08:17:52.222 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:17:51 smithi007 ceph-mon[29459]: pgmap v751: 158 pgs: 2 active+remapped+backfilling, 156 active+clean; 1.5 GiB data, 7.6 GiB used, 529 GiB / 536 GiB avail; 647 B/s rd, 0 B/s wr, 1.21k op/s; 1975/72180 objects misplaced (2.736%); 8.6 MiB/s, 137 objects/s recovering 2024-09-18T08:17:52.222 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:17:51 smithi007 ceph-mon[29459]: from='client.? 172.21.15.7:0/16665188' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-18T08:17:52.222 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:17:51 smithi007 ceph-mon[29459]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-18T08:17:52.222 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:17:51 smithi007 ceph-mon[39249]: 3.11 scrub starts 2024-09-18T08:17:52.222 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:17:51 smithi007 ceph-mon[39249]: 3.11 scrub ok 2024-09-18T08:17:52.223 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:17:51 smithi007 ceph-mon[39249]: pgmap v751: 158 pgs: 2 active+remapped+backfilling, 156 active+clean; 1.5 GiB data, 7.6 GiB used, 529 GiB / 536 GiB avail; 647 B/s rd, 0 B/s wr, 1.21k op/s; 1975/72180 objects misplaced (2.736%); 8.6 MiB/s, 137 objects/s recovering 2024-09-18T08:17:52.223 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:17:51 smithi007 ceph-mon[39249]: from='client.? 172.21.15.7:0/16665188' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-18T08:17:52.223 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:17:51 smithi007 ceph-mon[39249]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-18T08:17:52.247 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7 2024-09-18T08:17:52.272 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:17:52.273 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:17:52.273 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7' 2024-09-18T08:17:52.274 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid 779349ca-7593-11ef-bceb-c7b262605968 -- ceph daemon osd.0 dump_historic_ops 2024-09-18T08:17:52.374 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2 2024-09-18T08:17:52.428 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:17:52.428 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:17:52.429 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2' 2024-09-18T08:17:52.485 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/779349ca-7593-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T08:17:52.486 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/779349ca-7593-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T08:17:52.530 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4 2024-09-18T08:17:52.555 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:17:52.557 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:17:52.557 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4' 2024-09-18T08:17:52.658 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7 2024-09-18T08:17:52.683 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:17:52.683 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:17:52.684 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7' 2024-09-18T08:17:52.785 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3 2024-09-18T08:17:52.833 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:17:52.834 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:17:52.834 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3' 2024-09-18T08:17:52.935 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3 2024-09-18T08:17:53.006 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:17:53.007 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:17:53.007 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3' 2024-09-18T08:17:53.108 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5 2024-09-18T08:17:53.132 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:17:53.132 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:17:53.133 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5' 2024-09-18T08:17:53.153 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:17:52 smithi007 ceph-mon[29459]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-18T08:17:53.153 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:17:52 smithi007 ceph-mon[29459]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-18T08:17:53.153 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:17:52 smithi007 ceph-mon[29459]: osdmap e385: 8 total, 8 up, 6 in 2024-09-18T08:17:53.154 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:17:52 smithi007 ceph-mon[39249]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-18T08:17:53.154 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:17:52 smithi007 ceph-mon[39249]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-18T08:17:53.154 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:17:52 smithi007 ceph-mon[39249]: osdmap e385: 8 total, 8 up, 6 in 2024-09-18T08:17:53.155 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:17:52 smithi050 ceph-mon[32371]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-18T08:17:53.155 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:17:52 smithi050 ceph-mon[32371]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-18T08:17:53.155 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:17:52 smithi050 ceph-mon[32371]: osdmap e385: 8 total, 8 up, 6 in 2024-09-18T08:17:53.233 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4 2024-09-18T08:17:53.256 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:17:53.257 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:17:53.257 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4' 2024-09-18T08:17:53.358 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6 2024-09-18T08:17:53.380 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:17:53.380 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:17:53.381 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6' 2024-09-18T08:17:53.481 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7 2024-09-18T08:17:53.503 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:17:53.504 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:17:53.504 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7' 2024-09-18T08:17:53.549 INFO:teuthology.orchestra.run.smithi007.stdout:osd.0: {} 2024-09-18T08:17:53.549 INFO:teuthology.orchestra.run.smithi007.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-18T08:17:53.557 INFO:teuthology.orchestra.run.smithi007.stdout:osd.1: {} 2024-09-18T08:17:53.557 INFO:teuthology.orchestra.run.smithi007.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-18T08:17:53.573 INFO:teuthology.orchestra.run.smithi007.stdout:osd.2: {} 2024-09-18T08:17:53.573 INFO:teuthology.orchestra.run.smithi007.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-18T08:17:53.590 INFO:teuthology.orchestra.run.smithi007.stdout:osd.3: {} 2024-09-18T08:17:53.590 INFO:teuthology.orchestra.run.smithi007.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-18T08:17:53.605 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6 2024-09-18T08:17:53.607 INFO:teuthology.orchestra.run.smithi007.stdout:osd.4: {} 2024-09-18T08:17:53.607 INFO:teuthology.orchestra.run.smithi007.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-18T08:17:53.625 INFO:teuthology.orchestra.run.smithi007.stdout:osd.5: {} 2024-09-18T08:17:53.625 INFO:teuthology.orchestra.run.smithi007.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-18T08:17:53.628 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:17:53.629 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:17:53.629 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6' 2024-09-18T08:17:53.643 INFO:teuthology.orchestra.run.smithi007.stdout:osd.6: {} 2024-09-18T08:17:53.643 INFO:teuthology.orchestra.run.smithi007.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-18T08:17:53.659 INFO:teuthology.orchestra.run.smithi007.stdout:osd.7: {} 2024-09-18T08:17:53.660 INFO:teuthology.orchestra.run.smithi007.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-18T08:17:53.729 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7 2024-09-18T08:17:53.752 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:17:53.753 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:17:53.753 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7' 2024-09-18T08:17:53.854 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2 2024-09-18T08:17:53.876 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:17:53.877 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:17:53.877 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2' 2024-09-18T08:17:53.978 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3 2024-09-18T08:17:53.981 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:17:53 smithi007 ceph-mon[29459]: osdmap e386: 8 total, 8 up, 6 in 2024-09-18T08:17:53.981 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:17:53 smithi007 ceph-mon[29459]: pgmap v754: 158 pgs: 2 active+remapped+backfilling, 156 active+clean; 1.4 GiB data, 7.3 GiB used, 529 GiB / 536 GiB avail; 1.7 KiB/s rd, 0 B/s wr, 996 op/s; 1164/69597 objects misplaced (1.672%); 18 MiB/s, 286 objects/s recovering 2024-09-18T08:17:53.981 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:17:53 smithi007 ceph-mon[29459]: from='client.? 172.21.15.7:0/868141989' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-18T08:17:53.981 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:17:53 smithi007 ceph-mon[39249]: osdmap e386: 8 total, 8 up, 6 in 2024-09-18T08:17:53.981 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:17:53 smithi007 ceph-mon[39249]: pgmap v754: 158 pgs: 2 active+remapped+backfilling, 156 active+clean; 1.4 GiB data, 7.3 GiB used, 529 GiB / 536 GiB avail; 1.7 KiB/s rd, 0 B/s wr, 996 op/s; 1164/69597 objects misplaced (1.672%); 18 MiB/s, 286 objects/s recovering 2024-09-18T08:17:53.981 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:17:53 smithi007 ceph-mon[39249]: from='client.? 172.21.15.7:0/868141989' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-18T08:17:54.015 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:17:54.016 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:17:54.016 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3' 2024-09-18T08:17:54.117 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7 2024-09-18T08:17:54.140 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:17:54.140 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:17:54.141 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7' 2024-09-18T08:17:54.242 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1 2024-09-18T08:17:54.264 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:17:54.265 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:17:54.265 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1' 2024-09-18T08:17:54.366 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6 2024-09-18T08:17:54.389 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:17:54.390 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:17:54.390 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6' 2024-09-18T08:17:54.405 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:17:53 smithi050 ceph-mon[32371]: osdmap e386: 8 total, 8 up, 6 in 2024-09-18T08:17:54.405 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:17:53 smithi050 ceph-mon[32371]: pgmap v754: 158 pgs: 2 active+remapped+backfilling, 156 active+clean; 1.4 GiB data, 7.3 GiB used, 529 GiB / 536 GiB avail; 1.7 KiB/s rd, 0 B/s wr, 996 op/s; 1164/69597 objects misplaced (1.672%); 18 MiB/s, 286 objects/s recovering 2024-09-18T08:17:54.405 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:17:53 smithi050 ceph-mon[32371]: from='client.? 172.21.15.7:0/868141989' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-18T08:17:54.491 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1 2024-09-18T08:17:54.513 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:17:54.513 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:17:54.513 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1' 2024-09-18T08:17:54.614 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5 2024-09-18T08:17:54.642 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:17:54.643 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:17:54.643 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5' 2024-09-18T08:17:54.745 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4 2024-09-18T08:17:54.771 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:17:54.772 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:17:54.772 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4' 2024-09-18T08:17:54.873 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4 2024-09-18T08:17:54.899 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:17:54.900 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:17:54.900 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4' 2024-09-18T08:17:54.986 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid 779349ca-7593-11ef-bceb-c7b262605968 -- ceph osd set nodeep-scrub 2024-09-18T08:17:55.001 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7 2024-09-18T08:17:55.025 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:17:55.026 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:17:55.026 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7' 2024-09-18T08:17:55.127 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1 2024-09-18T08:17:55.165 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:17:55.166 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:17:55.166 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1' 2024-09-18T08:17:55.229 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:17:54 smithi007 ceph-mon[29459]: osdmap e387: 8 total, 8 up, 6 in 2024-09-18T08:17:55.230 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:17:54 smithi007 ceph-mon[29459]: osdmap e388: 8 total, 8 up, 6 in 2024-09-18T08:17:55.230 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:17:54 smithi007 ceph-mon[39249]: osdmap e387: 8 total, 8 up, 6 in 2024-09-18T08:17:55.230 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:17:54 smithi007 ceph-mon[39249]: osdmap e388: 8 total, 8 up, 6 in 2024-09-18T08:17:55.267 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0 2024-09-18T08:17:55.317 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:17:55.317 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:17:55.318 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0' 2024-09-18T08:17:55.405 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:17:54 smithi050 ceph-mon[32371]: osdmap e387: 8 total, 8 up, 6 in 2024-09-18T08:17:55.405 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:17:54 smithi050 ceph-mon[32371]: osdmap e388: 8 total, 8 up, 6 in 2024-09-18T08:17:55.418 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4 2024-09-18T08:17:55.440 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:17:55.441 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:17:55.441 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4' 2024-09-18T08:17:55.542 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2 2024-09-18T08:17:55.584 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:17:55.585 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:17:55.585 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2' 2024-09-18T08:17:55.686 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7 2024-09-18T08:17:55.708 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:17:55.709 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:17:55.710 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7' 2024-09-18T08:17:55.811 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1 2024-09-18T08:17:55.852 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:17:55.853 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:17:55.853 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1' 2024-09-18T08:17:55.954 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7 2024-09-18T08:17:55.977 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:17:55.978 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:17:55.978 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7' 2024-09-18T08:17:55.981 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:17:55 smithi007 ceph-mon[29459]: pgmap v757: 158 pgs: 1 active+remapped+backfilling, 157 active+clean; 1.2 GiB data, 6.9 GiB used, 530 GiB / 536 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.05k op/s; 408/59571 objects misplaced (0.685%); 21 MiB/s, 334 objects/s recovering 2024-09-18T08:17:55.981 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:17:55 smithi007 ceph-mon[39249]: pgmap v757: 158 pgs: 1 active+remapped+backfilling, 157 active+clean; 1.2 GiB data, 6.9 GiB used, 530 GiB / 536 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.05k op/s; 408/59571 objects misplaced (0.685%); 21 MiB/s, 334 objects/s recovering 2024-09-18T08:17:56.079 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2 2024-09-18T08:17:56.127 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:17:56.127 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:17:56.128 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2' 2024-09-18T08:17:56.200 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-09-18T08:17:56.200 INFO:teuthology.orchestra.run.smithi007.stdout:{ 2024-09-18T08:17:56.200 INFO:teuthology.orchestra.run.smithi007.stdout: "epoch": 388, 2024-09-18T08:17:56.200 INFO:teuthology.orchestra.run.smithi007.stdout: "fsid": "779349ca-7593-11ef-bceb-c7b262605968", 2024-09-18T08:17:56.200 INFO:teuthology.orchestra.run.smithi007.stdout: "created": "2024-09-18T07:57:09.295710+0000", 2024-09-18T08:17:56.200 INFO:teuthology.orchestra.run.smithi007.stdout: "modified": "2024-09-18T08:17:54.412045+0000", 2024-09-18T08:17:56.201 INFO:teuthology.orchestra.run.smithi007.stdout: "last_up_change": "2024-09-18T08:14:38.391465+0000", 2024-09-18T08:17:56.201 INFO:teuthology.orchestra.run.smithi007.stdout: "last_in_change": "2024-09-18T08:16:12.396744+0000", 2024-09-18T08:17:56.201 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": "noscrub,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit", 2024-09-18T08:17:56.201 INFO:teuthology.orchestra.run.smithi007.stdout: "flags_num": 5801984, 2024-09-18T08:17:56.201 INFO:teuthology.orchestra.run.smithi007.stdout: "flags_set": [ 2024-09-18T08:17:56.201 INFO:teuthology.orchestra.run.smithi007.stdout: "noscrub", 2024-09-18T08:17:56.201 INFO:teuthology.orchestra.run.smithi007.stdout: "pglog_hardlimit", 2024-09-18T08:17:56.201 INFO:teuthology.orchestra.run.smithi007.stdout: "purged_snapdirs", 2024-09-18T08:17:56.201 INFO:teuthology.orchestra.run.smithi007.stdout: "recovery_deletes", 2024-09-18T08:17:56.201 INFO:teuthology.orchestra.run.smithi007.stdout: "sortbitwise" 2024-09-18T08:17:56.201 INFO:teuthology.orchestra.run.smithi007.stdout: ], 2024-09-18T08:17:56.201 INFO:teuthology.orchestra.run.smithi007.stdout: "crush_version": 19, 2024-09-18T08:17:56.202 INFO:teuthology.orchestra.run.smithi007.stdout: "full_ratio": 0.94999998807907104, 2024-09-18T08:17:56.202 INFO:teuthology.orchestra.run.smithi007.stdout: "backfillfull_ratio": 0.89999997615814209, 2024-09-18T08:17:56.202 INFO:teuthology.orchestra.run.smithi007.stdout: "nearfull_ratio": 0.85000002384185791, 2024-09-18T08:17:56.202 INFO:teuthology.orchestra.run.smithi007.stdout: "cluster_snapshot": "", 2024-09-18T08:17:56.202 INFO:teuthology.orchestra.run.smithi007.stdout: "pool_max": 10, 2024-09-18T08:17:56.202 INFO:teuthology.orchestra.run.smithi007.stdout: "max_osd": 8, 2024-09-18T08:17:56.202 INFO:teuthology.orchestra.run.smithi007.stdout: "require_min_compat_client": "luminous", 2024-09-18T08:17:56.202 INFO:teuthology.orchestra.run.smithi007.stdout: "min_compat_client": "luminous", 2024-09-18T08:17:56.202 INFO:teuthology.orchestra.run.smithi007.stdout: "require_osd_release": "squid", 2024-09-18T08:17:56.202 INFO:teuthology.orchestra.run.smithi007.stdout: "allow_crimson": false, 2024-09-18T08:17:56.202 INFO:teuthology.orchestra.run.smithi007.stdout: "pools": [ 2024-09-18T08:17:56.202 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:17:56.202 INFO:teuthology.orchestra.run.smithi007.stdout: "pool": 1, 2024-09-18T08:17:56.203 INFO:teuthology.orchestra.run.smithi007.stdout: "pool_name": ".mgr", 2024-09-18T08:17:56.203 INFO:teuthology.orchestra.run.smithi007.stdout: "create_time": "2024-09-18T08:00:11.802627+0000", 2024-09-18T08:17:56.203 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 1, 2024-09-18T08:17:56.203 INFO:teuthology.orchestra.run.smithi007.stdout: "flags_names": "hashpspool", 2024-09-18T08:17:56.203 INFO:teuthology.orchestra.run.smithi007.stdout: "type": 1, 2024-09-18T08:17:56.203 INFO:teuthology.orchestra.run.smithi007.stdout: "size": 3, 2024-09-18T08:17:56.203 INFO:teuthology.orchestra.run.smithi007.stdout: "min_size": 2, 2024-09-18T08:17:56.203 INFO:teuthology.orchestra.run.smithi007.stdout: "crush_rule": 0, 2024-09-18T08:17:56.203 INFO:teuthology.orchestra.run.smithi007.stdout: "peering_crush_bucket_count": 0, 2024-09-18T08:17:56.203 INFO:teuthology.orchestra.run.smithi007.stdout: "peering_crush_bucket_target": 0, 2024-09-18T08:17:56.203 INFO:teuthology.orchestra.run.smithi007.stdout: "peering_crush_bucket_barrier": 0, 2024-09-18T08:17:56.203 INFO:teuthology.orchestra.run.smithi007.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-09-18T08:17:56.203 INFO:teuthology.orchestra.run.smithi007.stdout: "is_stretch_pool": false, 2024-09-18T08:17:56.204 INFO:teuthology.orchestra.run.smithi007.stdout: "object_hash": 2, 2024-09-18T08:17:56.204 INFO:teuthology.orchestra.run.smithi007.stdout: "pg_autoscale_mode": "off", 2024-09-18T08:17:56.204 INFO:teuthology.orchestra.run.smithi007.stdout: "pg_num": 11, 2024-09-18T08:17:56.204 INFO:teuthology.orchestra.run.smithi007.stdout: "pg_placement_num": 11, 2024-09-18T08:17:56.204 INFO:teuthology.orchestra.run.smithi007.stdout: "pg_placement_num_target": 11, 2024-09-18T08:17:56.204 INFO:teuthology.orchestra.run.smithi007.stdout: "pg_num_target": 11, 2024-09-18T08:17:56.204 INFO:teuthology.orchestra.run.smithi007.stdout: "pg_num_pending": 11, 2024-09-18T08:17:56.204 INFO:teuthology.orchestra.run.smithi007.stdout: "last_pg_merge_meta": { 2024-09-18T08:17:56.204 INFO:teuthology.orchestra.run.smithi007.stdout: "source_pgid": "0.0", 2024-09-18T08:17:56.204 INFO:teuthology.orchestra.run.smithi007.stdout: "ready_epoch": 0, 2024-09-18T08:17:56.204 INFO:teuthology.orchestra.run.smithi007.stdout: "last_epoch_started": 0, 2024-09-18T08:17:56.204 INFO:teuthology.orchestra.run.smithi007.stdout: "last_epoch_clean": 0, 2024-09-18T08:17:56.204 INFO:teuthology.orchestra.run.smithi007.stdout: "source_version": "0'0", 2024-09-18T08:17:56.205 INFO:teuthology.orchestra.run.smithi007.stdout: "target_version": "0'0" 2024-09-18T08:17:56.205 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:17:56.205 INFO:teuthology.orchestra.run.smithi007.stdout: "last_change": "285", 2024-09-18T08:17:56.205 INFO:teuthology.orchestra.run.smithi007.stdout: "last_force_op_resend": "0", 2024-09-18T08:17:56.205 INFO:teuthology.orchestra.run.smithi007.stdout: "last_force_op_resend_prenautilus": "0", 2024-09-18T08:17:56.205 INFO:teuthology.orchestra.run.smithi007.stdout: "last_force_op_resend_preluminous": "152", 2024-09-18T08:17:56.205 INFO:teuthology.orchestra.run.smithi007.stdout: "auid": 0, 2024-09-18T08:17:56.205 INFO:teuthology.orchestra.run.smithi007.stdout: "snap_mode": "selfmanaged", 2024-09-18T08:17:56.205 INFO:teuthology.orchestra.run.smithi007.stdout: "snap_seq": 0, 2024-09-18T08:17:56.205 INFO:teuthology.orchestra.run.smithi007.stdout: "snap_epoch": 0, 2024-09-18T08:17:56.205 INFO:teuthology.orchestra.run.smithi007.stdout: "pool_snaps": [], 2024-09-18T08:17:56.205 INFO:teuthology.orchestra.run.smithi007.stdout: "removed_snaps": "[]", 2024-09-18T08:17:56.205 INFO:teuthology.orchestra.run.smithi007.stdout: "quota_max_bytes": 0, 2024-09-18T08:17:56.205 INFO:teuthology.orchestra.run.smithi007.stdout: "quota_max_objects": 0, 2024-09-18T08:17:56.206 INFO:teuthology.orchestra.run.smithi007.stdout: "tiers": [], 2024-09-18T08:17:56.206 INFO:teuthology.orchestra.run.smithi007.stdout: "tier_of": -1, 2024-09-18T08:17:56.206 INFO:teuthology.orchestra.run.smithi007.stdout: "read_tier": -1, 2024-09-18T08:17:56.206 INFO:teuthology.orchestra.run.smithi007.stdout: "write_tier": -1, 2024-09-18T08:17:56.206 INFO:teuthology.orchestra.run.smithi007.stdout: "cache_mode": "none", 2024-09-18T08:17:56.206 INFO:teuthology.orchestra.run.smithi007.stdout: "target_max_bytes": 0, 2024-09-18T08:17:56.206 INFO:teuthology.orchestra.run.smithi007.stdout: "target_max_objects": 0, 2024-09-18T08:17:56.206 INFO:teuthology.orchestra.run.smithi007.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-09-18T08:17:56.206 INFO:teuthology.orchestra.run.smithi007.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-09-18T08:17:56.206 INFO:teuthology.orchestra.run.smithi007.stdout: "cache_target_full_ratio_micro": 800000, 2024-09-18T08:17:56.206 INFO:teuthology.orchestra.run.smithi007.stdout: "cache_min_flush_age": 0, 2024-09-18T08:17:56.206 INFO:teuthology.orchestra.run.smithi007.stdout: "cache_min_evict_age": 0, 2024-09-18T08:17:56.206 INFO:teuthology.orchestra.run.smithi007.stdout: "erasure_code_profile": "", 2024-09-18T08:17:56.206 INFO:teuthology.orchestra.run.smithi007.stdout: "hit_set_params": { 2024-09-18T08:17:56.207 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "none" 2024-09-18T08:17:56.207 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:17:56.207 INFO:teuthology.orchestra.run.smithi007.stdout: "hit_set_period": 0, 2024-09-18T08:17:56.207 INFO:teuthology.orchestra.run.smithi007.stdout: "hit_set_count": 0, 2024-09-18T08:17:56.207 INFO:teuthology.orchestra.run.smithi007.stdout: "use_gmt_hitset": true, 2024-09-18T08:17:56.207 INFO:teuthology.orchestra.run.smithi007.stdout: "min_read_recency_for_promote": 0, 2024-09-18T08:17:56.207 INFO:teuthology.orchestra.run.smithi007.stdout: "min_write_recency_for_promote": 0, 2024-09-18T08:17:56.207 INFO:teuthology.orchestra.run.smithi007.stdout: "hit_set_grade_decay_rate": 0, 2024-09-18T08:17:56.207 INFO:teuthology.orchestra.run.smithi007.stdout: "hit_set_search_last_n": 0, 2024-09-18T08:17:56.207 INFO:teuthology.orchestra.run.smithi007.stdout: "grade_table": [], 2024-09-18T08:17:56.207 INFO:teuthology.orchestra.run.smithi007.stdout: "stripe_width": 0, 2024-09-18T08:17:56.207 INFO:teuthology.orchestra.run.smithi007.stdout: "expected_num_objects": 0, 2024-09-18T08:17:56.207 INFO:teuthology.orchestra.run.smithi007.stdout: "fast_read": false, 2024-09-18T08:17:56.207 INFO:teuthology.orchestra.run.smithi007.stdout: "options": { 2024-09-18T08:17:56.208 INFO:teuthology.orchestra.run.smithi007.stdout: "pg_num_max": 32, 2024-09-18T08:17:56.208 INFO:teuthology.orchestra.run.smithi007.stdout: "pg_num_min": 1 2024-09-18T08:17:56.208 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:17:56.208 INFO:teuthology.orchestra.run.smithi007.stdout: "application_metadata": { 2024-09-18T08:17:56.208 INFO:teuthology.orchestra.run.smithi007.stdout: "mgr": {} 2024-09-18T08:17:56.208 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:17:56.208 INFO:teuthology.orchestra.run.smithi007.stdout: "read_balance": { 2024-09-18T08:17:56.208 INFO:teuthology.orchestra.run.smithi007.stdout: "score_type": "Fair distribution", 2024-09-18T08:17:56.208 INFO:teuthology.orchestra.run.smithi007.stdout: "score_acting": 1.9099999666213989, 2024-09-18T08:17:56.208 INFO:teuthology.orchestra.run.smithi007.stdout: "score_stable": 1.9099999666213989, 2024-09-18T08:17:56.208 INFO:teuthology.orchestra.run.smithi007.stdout: "optimal_score": 0.86000001430511475, 2024-09-18T08:17:56.208 INFO:teuthology.orchestra.run.smithi007.stdout: "raw_score_acting": 1.6399999856948853, 2024-09-18T08:17:56.208 INFO:teuthology.orchestra.run.smithi007.stdout: "raw_score_stable": 1.6399999856948853, 2024-09-18T08:17:56.208 INFO:teuthology.orchestra.run.smithi007.stdout: "primary_affinity_weighted": 0.80927085876464844, 2024-09-18T08:17:56.209 INFO:teuthology.orchestra.run.smithi007.stdout: "average_primary_affinity": 0.92000001668930054, 2024-09-18T08:17:56.209 INFO:teuthology.orchestra.run.smithi007.stdout: "average_primary_affinity_weighted": 0.81000000238418579 2024-09-18T08:17:56.209 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-09-18T08:17:56.209 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:17:56.209 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:17:56.209 INFO:teuthology.orchestra.run.smithi007.stdout: "pool": 2, 2024-09-18T08:17:56.209 INFO:teuthology.orchestra.run.smithi007.stdout: "pool_name": "datapool", 2024-09-18T08:17:56.209 INFO:teuthology.orchestra.run.smithi007.stdout: "create_time": "2024-09-18T08:02:45.374099+0000", 2024-09-18T08:17:56.209 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 8193, 2024-09-18T08:17:56.209 INFO:teuthology.orchestra.run.smithi007.stdout: "flags_names": "hashpspool,selfmanaged_snaps", 2024-09-18T08:17:56.209 INFO:teuthology.orchestra.run.smithi007.stdout: "type": 1, 2024-09-18T08:17:56.209 INFO:teuthology.orchestra.run.smithi007.stdout: "size": 3, 2024-09-18T08:17:56.209 INFO:teuthology.orchestra.run.smithi007.stdout: "min_size": 2, 2024-09-18T08:17:56.210 INFO:teuthology.orchestra.run.smithi007.stdout: "crush_rule": 0, 2024-09-18T08:17:56.210 INFO:teuthology.orchestra.run.smithi007.stdout: "peering_crush_bucket_count": 0, 2024-09-18T08:17:56.210 INFO:teuthology.orchestra.run.smithi007.stdout: "peering_crush_bucket_target": 0, 2024-09-18T08:17:56.210 INFO:teuthology.orchestra.run.smithi007.stdout: "peering_crush_bucket_barrier": 0, 2024-09-18T08:17:56.210 INFO:teuthology.orchestra.run.smithi007.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-09-18T08:17:56.210 INFO:teuthology.orchestra.run.smithi007.stdout: "is_stretch_pool": false, 2024-09-18T08:17:56.210 INFO:teuthology.orchestra.run.smithi007.stdout: "object_hash": 2, 2024-09-18T08:17:56.210 INFO:teuthology.orchestra.run.smithi007.stdout: "pg_autoscale_mode": "off", 2024-09-18T08:17:56.210 INFO:teuthology.orchestra.run.smithi007.stdout: "pg_num": 3, 2024-09-18T08:17:56.210 INFO:teuthology.orchestra.run.smithi007.stdout: "pg_placement_num": 3, 2024-09-18T08:17:56.210 INFO:teuthology.orchestra.run.smithi007.stdout: "pg_placement_num_target": 3, 2024-09-18T08:17:56.210 INFO:teuthology.orchestra.run.smithi007.stdout: "pg_num_target": 3, 2024-09-18T08:17:56.210 INFO:teuthology.orchestra.run.smithi007.stdout: "pg_num_pending": 3, 2024-09-18T08:17:56.210 INFO:teuthology.orchestra.run.smithi007.stdout: "last_pg_merge_meta": { 2024-09-18T08:17:56.211 INFO:teuthology.orchestra.run.smithi007.stdout: "source_pgid": "0.0", 2024-09-18T08:17:56.211 INFO:teuthology.orchestra.run.smithi007.stdout: "ready_epoch": 0, 2024-09-18T08:17:56.211 INFO:teuthology.orchestra.run.smithi007.stdout: "last_epoch_started": 0, 2024-09-18T08:17:56.211 INFO:teuthology.orchestra.run.smithi007.stdout: "last_epoch_clean": 0, 2024-09-18T08:17:56.211 INFO:teuthology.orchestra.run.smithi007.stdout: "source_version": "0'0", 2024-09-18T08:17:56.211 INFO:teuthology.orchestra.run.smithi007.stdout: "target_version": "0'0" 2024-09-18T08:17:56.211 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:17:56.211 INFO:teuthology.orchestra.run.smithi007.stdout: "last_change": "64", 2024-09-18T08:17:56.211 INFO:teuthology.orchestra.run.smithi007.stdout: "last_force_op_resend": "0", 2024-09-18T08:17:56.211 INFO:teuthology.orchestra.run.smithi007.stdout: "last_force_op_resend_prenautilus": "0", 2024-09-18T08:17:56.211 INFO:teuthology.orchestra.run.smithi007.stdout: "last_force_op_resend_preluminous": "0", 2024-09-18T08:17:56.211 INFO:teuthology.orchestra.run.smithi007.stdout: "auid": 0, 2024-09-18T08:17:56.211 INFO:teuthology.orchestra.run.smithi007.stdout: "snap_mode": "selfmanaged", 2024-09-18T08:17:56.211 INFO:teuthology.orchestra.run.smithi007.stdout: "snap_seq": 2, 2024-09-18T08:17:56.212 INFO:teuthology.orchestra.run.smithi007.stdout: "snap_epoch": 64, 2024-09-18T08:17:56.212 INFO:teuthology.orchestra.run.smithi007.stdout: "pool_snaps": [], 2024-09-18T08:17:56.212 INFO:teuthology.orchestra.run.smithi007.stdout: "removed_snaps": "[]", 2024-09-18T08:17:56.212 INFO:teuthology.orchestra.run.smithi007.stdout: "quota_max_bytes": 0, 2024-09-18T08:17:56.212 INFO:teuthology.orchestra.run.smithi007.stdout: "quota_max_objects": 0, 2024-09-18T08:17:56.212 INFO:teuthology.orchestra.run.smithi007.stdout: "tiers": [], 2024-09-18T08:17:56.212 INFO:teuthology.orchestra.run.smithi007.stdout: "tier_of": -1, 2024-09-18T08:17:56.212 INFO:teuthology.orchestra.run.smithi007.stdout: "read_tier": -1, 2024-09-18T08:17:56.212 INFO:teuthology.orchestra.run.smithi007.stdout: "write_tier": -1, 2024-09-18T08:17:56.212 INFO:teuthology.orchestra.run.smithi007.stdout: "cache_mode": "none", 2024-09-18T08:17:56.212 INFO:teuthology.orchestra.run.smithi007.stdout: "target_max_bytes": 0, 2024-09-18T08:17:56.212 INFO:teuthology.orchestra.run.smithi007.stdout: "target_max_objects": 0, 2024-09-18T08:17:56.212 INFO:teuthology.orchestra.run.smithi007.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-09-18T08:17:56.213 INFO:teuthology.orchestra.run.smithi007.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-09-18T08:17:56.213 INFO:teuthology.orchestra.run.smithi007.stdout: "cache_target_full_ratio_micro": 800000, 2024-09-18T08:17:56.213 INFO:teuthology.orchestra.run.smithi007.stdout: "cache_min_flush_age": 0, 2024-09-18T08:17:56.213 INFO:teuthology.orchestra.run.smithi007.stdout: "cache_min_evict_age": 0, 2024-09-18T08:17:56.213 INFO:teuthology.orchestra.run.smithi007.stdout: "erasure_code_profile": "", 2024-09-18T08:17:56.213 INFO:teuthology.orchestra.run.smithi007.stdout: "hit_set_params": { 2024-09-18T08:17:56.213 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "none" 2024-09-18T08:17:56.213 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:17:56.213 INFO:teuthology.orchestra.run.smithi007.stdout: "hit_set_period": 0, 2024-09-18T08:17:56.213 INFO:teuthology.orchestra.run.smithi007.stdout: "hit_set_count": 0, 2024-09-18T08:17:56.213 INFO:teuthology.orchestra.run.smithi007.stdout: "use_gmt_hitset": true, 2024-09-18T08:17:56.213 INFO:teuthology.orchestra.run.smithi007.stdout: "min_read_recency_for_promote": 0, 2024-09-18T08:17:56.213 INFO:teuthology.orchestra.run.smithi007.stdout: "min_write_recency_for_promote": 0, 2024-09-18T08:17:56.213 INFO:teuthology.orchestra.run.smithi007.stdout: "hit_set_grade_decay_rate": 0, 2024-09-18T08:17:56.214 INFO:teuthology.orchestra.run.smithi007.stdout: "hit_set_search_last_n": 0, 2024-09-18T08:17:56.214 INFO:teuthology.orchestra.run.smithi007.stdout: "grade_table": [], 2024-09-18T08:17:56.214 INFO:teuthology.orchestra.run.smithi007.stdout: "stripe_width": 0, 2024-09-18T08:17:56.214 INFO:teuthology.orchestra.run.smithi007.stdout: "expected_num_objects": 0, 2024-09-18T08:17:56.214 INFO:teuthology.orchestra.run.smithi007.stdout: "fast_read": false, 2024-09-18T08:17:56.214 INFO:teuthology.orchestra.run.smithi007.stdout: "options": {}, 2024-09-18T08:17:56.214 INFO:teuthology.orchestra.run.smithi007.stdout: "application_metadata": { 2024-09-18T08:17:56.214 INFO:teuthology.orchestra.run.smithi007.stdout: "rbd": {} 2024-09-18T08:17:56.214 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:17:56.214 INFO:teuthology.orchestra.run.smithi007.stdout: "read_balance": { 2024-09-18T08:17:56.214 INFO:teuthology.orchestra.run.smithi007.stdout: "score_type": "Fair distribution", 2024-09-18T08:17:56.214 INFO:teuthology.orchestra.run.smithi007.stdout: "score_acting": 2.3299999237060547, 2024-09-18T08:17:56.214 INFO:teuthology.orchestra.run.smithi007.stdout: "score_stable": 2.3299999237060547, 2024-09-18T08:17:56.214 INFO:teuthology.orchestra.run.smithi007.stdout: "optimal_score": 0.86000001430511475, 2024-09-18T08:17:56.215 INFO:teuthology.orchestra.run.smithi007.stdout: "raw_score_acting": 2, 2024-09-18T08:17:56.215 INFO:teuthology.orchestra.run.smithi007.stdout: "raw_score_stable": 2, 2024-09-18T08:17:56.215 INFO:teuthology.orchestra.run.smithi007.stdout: "primary_affinity_weighted": 0.80927085876464844, 2024-09-18T08:17:56.215 INFO:teuthology.orchestra.run.smithi007.stdout: "average_primary_affinity": 0.92000001668930054, 2024-09-18T08:17:56.215 INFO:teuthology.orchestra.run.smithi007.stdout: "average_primary_affinity_weighted": 0.81000000238418579 2024-09-18T08:17:56.215 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-09-18T08:17:56.215 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:17:56.215 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:17:56.215 INFO:teuthology.orchestra.run.smithi007.stdout: "pool": 3, 2024-09-18T08:17:56.215 INFO:teuthology.orchestra.run.smithi007.stdout: "pool_name": ".rgw.root", 2024-09-18T08:17:56.215 INFO:teuthology.orchestra.run.smithi007.stdout: "create_time": "2024-09-18T08:02:46.319947+0000", 2024-09-18T08:17:56.215 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 1, 2024-09-18T08:17:56.215 INFO:teuthology.orchestra.run.smithi007.stdout: "flags_names": "hashpspool", 2024-09-18T08:17:56.215 INFO:teuthology.orchestra.run.smithi007.stdout: "type": 1, 2024-09-18T08:17:56.216 INFO:teuthology.orchestra.run.smithi007.stdout: "size": 3, 2024-09-18T08:17:56.216 INFO:teuthology.orchestra.run.smithi007.stdout: "min_size": 2, 2024-09-18T08:17:56.216 INFO:teuthology.orchestra.run.smithi007.stdout: "crush_rule": 0, 2024-09-18T08:17:56.216 INFO:teuthology.orchestra.run.smithi007.stdout: "peering_crush_bucket_count": 0, 2024-09-18T08:17:56.216 INFO:teuthology.orchestra.run.smithi007.stdout: "peering_crush_bucket_target": 0, 2024-09-18T08:17:56.216 INFO:teuthology.orchestra.run.smithi007.stdout: "peering_crush_bucket_barrier": 0, 2024-09-18T08:17:56.216 INFO:teuthology.orchestra.run.smithi007.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-09-18T08:17:56.216 INFO:teuthology.orchestra.run.smithi007.stdout: "is_stretch_pool": false, 2024-09-18T08:17:56.216 INFO:teuthology.orchestra.run.smithi007.stdout: "object_hash": 2, 2024-09-18T08:17:56.216 INFO:teuthology.orchestra.run.smithi007.stdout: "pg_autoscale_mode": "off", 2024-09-18T08:17:56.216 INFO:teuthology.orchestra.run.smithi007.stdout: "pg_num": 32, 2024-09-18T08:17:56.216 INFO:teuthology.orchestra.run.smithi007.stdout: "pg_placement_num": 32, 2024-09-18T08:17:56.216 INFO:teuthology.orchestra.run.smithi007.stdout: "pg_placement_num_target": 32, 2024-09-18T08:17:56.216 INFO:teuthology.orchestra.run.smithi007.stdout: "pg_num_target": 32, 2024-09-18T08:17:56.217 INFO:teuthology.orchestra.run.smithi007.stdout: "pg_num_pending": 32, 2024-09-18T08:17:56.217 INFO:teuthology.orchestra.run.smithi007.stdout: "last_pg_merge_meta": { 2024-09-18T08:17:56.217 INFO:teuthology.orchestra.run.smithi007.stdout: "source_pgid": "0.0", 2024-09-18T08:17:56.217 INFO:teuthology.orchestra.run.smithi007.stdout: "ready_epoch": 0, 2024-09-18T08:17:56.217 INFO:teuthology.orchestra.run.smithi007.stdout: "last_epoch_started": 0, 2024-09-18T08:17:56.217 INFO:teuthology.orchestra.run.smithi007.stdout: "last_epoch_clean": 0, 2024-09-18T08:17:56.217 INFO:teuthology.orchestra.run.smithi007.stdout: "source_version": "0'0", 2024-09-18T08:17:56.217 INFO:teuthology.orchestra.run.smithi007.stdout: "target_version": "0'0" 2024-09-18T08:17:56.217 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:17:56.217 INFO:teuthology.orchestra.run.smithi007.stdout: "last_change": "63", 2024-09-18T08:17:56.217 INFO:teuthology.orchestra.run.smithi007.stdout: "last_force_op_resend": "0", 2024-09-18T08:17:56.217 INFO:teuthology.orchestra.run.smithi007.stdout: "last_force_op_resend_prenautilus": "0", 2024-09-18T08:17:56.217 INFO:teuthology.orchestra.run.smithi007.stdout: "last_force_op_resend_preluminous": "0", 2024-09-18T08:17:56.218 INFO:teuthology.orchestra.run.smithi007.stdout: "auid": 0, 2024-09-18T08:17:56.218 INFO:teuthology.orchestra.run.smithi007.stdout: "snap_mode": "selfmanaged", 2024-09-18T08:17:56.218 INFO:teuthology.orchestra.run.smithi007.stdout: "snap_seq": 0, 2024-09-18T08:17:56.218 INFO:teuthology.orchestra.run.smithi007.stdout: "snap_epoch": 0, 2024-09-18T08:17:56.218 INFO:teuthology.orchestra.run.smithi007.stdout: "pool_snaps": [], 2024-09-18T08:17:56.218 INFO:teuthology.orchestra.run.smithi007.stdout: "removed_snaps": "[]", 2024-09-18T08:17:56.218 INFO:teuthology.orchestra.run.smithi007.stdout: "quota_max_bytes": 0, 2024-09-18T08:17:56.218 INFO:teuthology.orchestra.run.smithi007.stdout: "quota_max_objects": 0, 2024-09-18T08:17:56.218 INFO:teuthology.orchestra.run.smithi007.stdout: "tiers": [], 2024-09-18T08:17:56.218 INFO:teuthology.orchestra.run.smithi007.stdout: "tier_of": -1, 2024-09-18T08:17:56.218 INFO:teuthology.orchestra.run.smithi007.stdout: "read_tier": -1, 2024-09-18T08:17:56.218 INFO:teuthology.orchestra.run.smithi007.stdout: "write_tier": -1, 2024-09-18T08:17:56.218 INFO:teuthology.orchestra.run.smithi007.stdout: "cache_mode": "none", 2024-09-18T08:17:56.218 INFO:teuthology.orchestra.run.smithi007.stdout: "target_max_bytes": 0, 2024-09-18T08:17:56.218 INFO:teuthology.orchestra.run.smithi007.stdout: "target_max_objects": 0, 2024-09-18T08:17:56.219 INFO:teuthology.orchestra.run.smithi007.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-09-18T08:17:56.219 INFO:teuthology.orchestra.run.smithi007.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-09-18T08:17:56.219 INFO:teuthology.orchestra.run.smithi007.stdout: "cache_target_full_ratio_micro": 800000, 2024-09-18T08:17:56.219 INFO:teuthology.orchestra.run.smithi007.stdout: "cache_min_flush_age": 0, 2024-09-18T08:17:56.219 INFO:teuthology.orchestra.run.smithi007.stdout: "cache_min_evict_age": 0, 2024-09-18T08:17:56.219 INFO:teuthology.orchestra.run.smithi007.stdout: "erasure_code_profile": "", 2024-09-18T08:17:56.219 INFO:teuthology.orchestra.run.smithi007.stdout: "hit_set_params": { 2024-09-18T08:17:56.219 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "none" 2024-09-18T08:17:56.219 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:17:56.219 INFO:teuthology.orchestra.run.smithi007.stdout: "hit_set_period": 0, 2024-09-18T08:17:56.219 INFO:teuthology.orchestra.run.smithi007.stdout: "hit_set_count": 0, 2024-09-18T08:17:56.219 INFO:teuthology.orchestra.run.smithi007.stdout: "use_gmt_hitset": true, 2024-09-18T08:17:56.219 INFO:teuthology.orchestra.run.smithi007.stdout: "min_read_recency_for_promote": 0, 2024-09-18T08:17:56.220 INFO:teuthology.orchestra.run.smithi007.stdout: "min_write_recency_for_promote": 0, 2024-09-18T08:17:56.220 INFO:teuthology.orchestra.run.smithi007.stdout: "hit_set_grade_decay_rate": 0, 2024-09-18T08:17:56.220 INFO:teuthology.orchestra.run.smithi007.stdout: "hit_set_search_last_n": 0, 2024-09-18T08:17:56.220 INFO:teuthology.orchestra.run.smithi007.stdout: "grade_table": [], 2024-09-18T08:17:56.220 INFO:teuthology.orchestra.run.smithi007.stdout: "stripe_width": 0, 2024-09-18T08:17:56.220 INFO:teuthology.orchestra.run.smithi007.stdout: "expected_num_objects": 0, 2024-09-18T08:17:56.220 INFO:teuthology.orchestra.run.smithi007.stdout: "fast_read": false, 2024-09-18T08:17:56.220 INFO:teuthology.orchestra.run.smithi007.stdout: "options": {}, 2024-09-18T08:17:56.220 INFO:teuthology.orchestra.run.smithi007.stdout: "application_metadata": { 2024-09-18T08:17:56.220 INFO:teuthology.orchestra.run.smithi007.stdout: "rgw": {} 2024-09-18T08:17:56.220 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:17:56.220 INFO:teuthology.orchestra.run.smithi007.stdout: "read_balance": { 2024-09-18T08:17:56.220 INFO:teuthology.orchestra.run.smithi007.stdout: "score_type": "Fair distribution", 2024-09-18T08:17:56.221 INFO:teuthology.orchestra.run.smithi007.stdout: "score_acting": 2.190000057220459, 2024-09-18T08:17:56.221 INFO:teuthology.orchestra.run.smithi007.stdout: "score_stable": 2.190000057220459, 2024-09-18T08:17:56.221 INFO:teuthology.orchestra.run.smithi007.stdout: "optimal_score": 0.86000001430511475, 2024-09-18T08:17:56.221 INFO:teuthology.orchestra.run.smithi007.stdout: "raw_score_acting": 1.8799999952316284, 2024-09-18T08:17:56.221 INFO:teuthology.orchestra.run.smithi007.stdout: "raw_score_stable": 1.8799999952316284, 2024-09-18T08:17:56.221 INFO:teuthology.orchestra.run.smithi007.stdout: "primary_affinity_weighted": 0.80927085876464844, 2024-09-18T08:17:56.221 INFO:teuthology.orchestra.run.smithi007.stdout: "average_primary_affinity": 0.92000001668930054, 2024-09-18T08:17:56.221 INFO:teuthology.orchestra.run.smithi007.stdout: "average_primary_affinity_weighted": 0.81000000238418579 2024-09-18T08:17:56.221 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-09-18T08:17:56.221 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:17:56.221 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:17:56.221 INFO:teuthology.orchestra.run.smithi007.stdout: "pool": 4, 2024-09-18T08:17:56.221 INFO:teuthology.orchestra.run.smithi007.stdout: "pool_name": "default.rgw.log", 2024-09-18T08:17:56.222 INFO:teuthology.orchestra.run.smithi007.stdout: "create_time": "2024-09-18T08:02:47.948552+0000", 2024-09-18T08:17:56.222 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 1, 2024-09-18T08:17:56.222 INFO:teuthology.orchestra.run.smithi007.stdout: "flags_names": "hashpspool", 2024-09-18T08:17:56.222 INFO:teuthology.orchestra.run.smithi007.stdout: "type": 1, 2024-09-18T08:17:56.222 INFO:teuthology.orchestra.run.smithi007.stdout: "size": 3, 2024-09-18T08:17:56.222 INFO:teuthology.orchestra.run.smithi007.stdout: "min_size": 2, 2024-09-18T08:17:56.222 INFO:teuthology.orchestra.run.smithi007.stdout: "crush_rule": 0, 2024-09-18T08:17:56.222 INFO:teuthology.orchestra.run.smithi007.stdout: "peering_crush_bucket_count": 0, 2024-09-18T08:17:56.222 INFO:teuthology.orchestra.run.smithi007.stdout: "peering_crush_bucket_target": 0, 2024-09-18T08:17:56.222 INFO:teuthology.orchestra.run.smithi007.stdout: "peering_crush_bucket_barrier": 0, 2024-09-18T08:17:56.222 INFO:teuthology.orchestra.run.smithi007.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-09-18T08:17:56.222 INFO:teuthology.orchestra.run.smithi007.stdout: "is_stretch_pool": false, 2024-09-18T08:17:56.222 INFO:teuthology.orchestra.run.smithi007.stdout: "object_hash": 2, 2024-09-18T08:17:56.222 INFO:teuthology.orchestra.run.smithi007.stdout: "pg_autoscale_mode": "off", 2024-09-18T08:17:56.223 INFO:teuthology.orchestra.run.smithi007.stdout: "pg_num": 32, 2024-09-18T08:17:56.223 INFO:teuthology.orchestra.run.smithi007.stdout: "pg_placement_num": 32, 2024-09-18T08:17:56.223 INFO:teuthology.orchestra.run.smithi007.stdout: "pg_placement_num_target": 32, 2024-09-18T08:17:56.223 INFO:teuthology.orchestra.run.smithi007.stdout: "pg_num_target": 32, 2024-09-18T08:17:56.223 INFO:teuthology.orchestra.run.smithi007.stdout: "pg_num_pending": 32, 2024-09-18T08:17:56.223 INFO:teuthology.orchestra.run.smithi007.stdout: "last_pg_merge_meta": { 2024-09-18T08:17:56.223 INFO:teuthology.orchestra.run.smithi007.stdout: "source_pgid": "0.0", 2024-09-18T08:17:56.223 INFO:teuthology.orchestra.run.smithi007.stdout: "ready_epoch": 0, 2024-09-18T08:17:56.223 INFO:teuthology.orchestra.run.smithi007.stdout: "last_epoch_started": 0, 2024-09-18T08:17:56.223 INFO:teuthology.orchestra.run.smithi007.stdout: "last_epoch_clean": 0, 2024-09-18T08:17:56.223 INFO:teuthology.orchestra.run.smithi007.stdout: "source_version": "0'0", 2024-09-18T08:17:56.223 INFO:teuthology.orchestra.run.smithi007.stdout: "target_version": "0'0" 2024-09-18T08:17:56.224 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:17:56.224 INFO:teuthology.orchestra.run.smithi007.stdout: "last_change": "65", 2024-09-18T08:17:56.224 INFO:teuthology.orchestra.run.smithi007.stdout: "last_force_op_resend": "0", 2024-09-18T08:17:56.224 INFO:teuthology.orchestra.run.smithi007.stdout: "last_force_op_resend_prenautilus": "0", 2024-09-18T08:17:56.224 INFO:teuthology.orchestra.run.smithi007.stdout: "last_force_op_resend_preluminous": "0", 2024-09-18T08:17:56.224 INFO:teuthology.orchestra.run.smithi007.stdout: "auid": 0, 2024-09-18T08:17:56.224 INFO:teuthology.orchestra.run.smithi007.stdout: "snap_mode": "selfmanaged", 2024-09-18T08:17:56.224 INFO:teuthology.orchestra.run.smithi007.stdout: "snap_seq": 0, 2024-09-18T08:17:56.224 INFO:teuthology.orchestra.run.smithi007.stdout: "snap_epoch": 0, 2024-09-18T08:17:56.224 INFO:teuthology.orchestra.run.smithi007.stdout: "pool_snaps": [], 2024-09-18T08:17:56.224 INFO:teuthology.orchestra.run.smithi007.stdout: "removed_snaps": "[]", 2024-09-18T08:17:56.224 INFO:teuthology.orchestra.run.smithi007.stdout: "quota_max_bytes": 0, 2024-09-18T08:17:56.224 INFO:teuthology.orchestra.run.smithi007.stdout: "quota_max_objects": 0, 2024-09-18T08:17:56.225 INFO:teuthology.orchestra.run.smithi007.stdout: "tiers": [], 2024-09-18T08:17:56.225 INFO:teuthology.orchestra.run.smithi007.stdout: "tier_of": -1, 2024-09-18T08:17:56.225 INFO:teuthology.orchestra.run.smithi007.stdout: "read_tier": -1, 2024-09-18T08:17:56.225 INFO:teuthology.orchestra.run.smithi007.stdout: "write_tier": -1, 2024-09-18T08:17:56.225 INFO:teuthology.orchestra.run.smithi007.stdout: "cache_mode": "none", 2024-09-18T08:17:56.225 INFO:teuthology.orchestra.run.smithi007.stdout: "target_max_bytes": 0, 2024-09-18T08:17:56.225 INFO:teuthology.orchestra.run.smithi007.stdout: "target_max_objects": 0, 2024-09-18T08:17:56.225 INFO:teuthology.orchestra.run.smithi007.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-09-18T08:17:56.225 INFO:teuthology.orchestra.run.smithi007.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-09-18T08:17:56.225 INFO:teuthology.orchestra.run.smithi007.stdout: "cache_target_full_ratio_micro": 800000, 2024-09-18T08:17:56.225 INFO:teuthology.orchestra.run.smithi007.stdout: "cache_min_flush_age": 0, 2024-09-18T08:17:56.225 INFO:teuthology.orchestra.run.smithi007.stdout: "cache_min_evict_age": 0, 2024-09-18T08:17:56.225 INFO:teuthology.orchestra.run.smithi007.stdout: "erasure_code_profile": "", 2024-09-18T08:17:56.225 INFO:teuthology.orchestra.run.smithi007.stdout: "hit_set_params": { 2024-09-18T08:17:56.226 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "none" 2024-09-18T08:17:56.226 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:17:56.226 INFO:teuthology.orchestra.run.smithi007.stdout: "hit_set_period": 0, 2024-09-18T08:17:56.226 INFO:teuthology.orchestra.run.smithi007.stdout: "hit_set_count": 0, 2024-09-18T08:17:56.226 INFO:teuthology.orchestra.run.smithi007.stdout: "use_gmt_hitset": true, 2024-09-18T08:17:56.226 INFO:teuthology.orchestra.run.smithi007.stdout: "min_read_recency_for_promote": 0, 2024-09-18T08:17:56.226 INFO:teuthology.orchestra.run.smithi007.stdout: "min_write_recency_for_promote": 0, 2024-09-18T08:17:56.226 INFO:teuthology.orchestra.run.smithi007.stdout: "hit_set_grade_decay_rate": 0, 2024-09-18T08:17:56.226 INFO:teuthology.orchestra.run.smithi007.stdout: "hit_set_search_last_n": 0, 2024-09-18T08:17:56.226 INFO:teuthology.orchestra.run.smithi007.stdout: "grade_table": [], 2024-09-18T08:17:56.226 INFO:teuthology.orchestra.run.smithi007.stdout: "stripe_width": 0, 2024-09-18T08:17:56.226 INFO:teuthology.orchestra.run.smithi007.stdout: "expected_num_objects": 0, 2024-09-18T08:17:56.226 INFO:teuthology.orchestra.run.smithi007.stdout: "fast_read": false, 2024-09-18T08:17:56.226 INFO:teuthology.orchestra.run.smithi007.stdout: "options": {}, 2024-09-18T08:17:56.227 INFO:teuthology.orchestra.run.smithi007.stdout: "application_metadata": { 2024-09-18T08:17:56.227 INFO:teuthology.orchestra.run.smithi007.stdout: "rgw": {} 2024-09-18T08:17:56.227 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:17:56.227 INFO:teuthology.orchestra.run.smithi007.stdout: "read_balance": { 2024-09-18T08:17:56.227 INFO:teuthology.orchestra.run.smithi007.stdout: "score_type": "Fair distribution", 2024-09-18T08:17:56.227 INFO:teuthology.orchestra.run.smithi007.stdout: "score_acting": 2.190000057220459, 2024-09-18T08:17:56.227 INFO:teuthology.orchestra.run.smithi007.stdout: "score_stable": 2.190000057220459, 2024-09-18T08:17:56.227 INFO:teuthology.orchestra.run.smithi007.stdout: "optimal_score": 0.86000001430511475, 2024-09-18T08:17:56.227 INFO:teuthology.orchestra.run.smithi007.stdout: "raw_score_acting": 1.8799999952316284, 2024-09-18T08:17:56.227 INFO:teuthology.orchestra.run.smithi007.stdout: "raw_score_stable": 1.8799999952316284, 2024-09-18T08:17:56.227 INFO:teuthology.orchestra.run.smithi007.stdout: "primary_affinity_weighted": 0.80927085876464844, 2024-09-18T08:17:56.227 INFO:teuthology.orchestra.run.smithi007.stdout: "average_primary_affinity": 0.92000001668930054, 2024-09-18T08:17:56.227 INFO:teuthology.orchestra.run.smithi007.stdout: "average_primary_affinity_weighted": 0.81000000238418579 2024-09-18T08:17:56.227 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-09-18T08:17:56.228 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:17:56.228 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:17:56.228 INFO:teuthology.orchestra.run.smithi007.stdout: "pool": 5, 2024-09-18T08:17:56.228 INFO:teuthology.orchestra.run.smithi007.stdout: "pool_name": "default.rgw.control", 2024-09-18T08:17:56.228 INFO:teuthology.orchestra.run.smithi007.stdout: "create_time": "2024-09-18T08:02:49.904302+0000", 2024-09-18T08:17:56.228 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 1, 2024-09-18T08:17:56.228 INFO:teuthology.orchestra.run.smithi007.stdout: "flags_names": "hashpspool", 2024-09-18T08:17:56.228 INFO:teuthology.orchestra.run.smithi007.stdout: "type": 1, 2024-09-18T08:17:56.228 INFO:teuthology.orchestra.run.smithi007.stdout: "size": 3, 2024-09-18T08:17:56.228 INFO:teuthology.orchestra.run.smithi007.stdout: "min_size": 2, 2024-09-18T08:17:56.228 INFO:teuthology.orchestra.run.smithi007.stdout: "crush_rule": 0, 2024-09-18T08:17:56.228 INFO:teuthology.orchestra.run.smithi007.stdout: "peering_crush_bucket_count": 0, 2024-09-18T08:17:56.228 INFO:teuthology.orchestra.run.smithi007.stdout: "peering_crush_bucket_target": 0, 2024-09-18T08:17:56.228 INFO:teuthology.orchestra.run.smithi007.stdout: "peering_crush_bucket_barrier": 0, 2024-09-18T08:17:56.228 INFO:teuthology.orchestra.run.smithi007.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-09-18T08:17:56.229 INFO:teuthology.orchestra.run.smithi007.stdout: "is_stretch_pool": false, 2024-09-18T08:17:56.229 INFO:teuthology.orchestra.run.smithi007.stdout: "object_hash": 2, 2024-09-18T08:17:56.229 INFO:teuthology.orchestra.run.smithi007.stdout: "pg_autoscale_mode": "off", 2024-09-18T08:17:56.229 INFO:teuthology.orchestra.run.smithi007.stdout: "pg_num": 32, 2024-09-18T08:17:56.229 INFO:teuthology.orchestra.run.smithi007.stdout: "pg_placement_num": 32, 2024-09-18T08:17:56.229 INFO:teuthology.orchestra.run.smithi007.stdout: "pg_placement_num_target": 32, 2024-09-18T08:17:56.229 INFO:teuthology.orchestra.run.smithi007.stdout: "pg_num_target": 32, 2024-09-18T08:17:56.229 INFO:teuthology.orchestra.run.smithi007.stdout: "pg_num_pending": 32, 2024-09-18T08:17:56.229 INFO:teuthology.orchestra.run.smithi007.stdout: "last_pg_merge_meta": { 2024-09-18T08:17:56.229 INFO:teuthology.orchestra.run.smithi007.stdout: "source_pgid": "0.0", 2024-09-18T08:17:56.229 INFO:teuthology.orchestra.run.smithi007.stdout: "ready_epoch": 0, 2024-09-18T08:17:56.229 INFO:teuthology.orchestra.run.smithi007.stdout: "last_epoch_started": 0, 2024-09-18T08:17:56.229 INFO:teuthology.orchestra.run.smithi007.stdout: "last_epoch_clean": 0, 2024-09-18T08:17:56.229 INFO:teuthology.orchestra.run.smithi007.stdout: "source_version": "0'0", 2024-09-18T08:17:56.230 INFO:teuthology.orchestra.run.smithi007.stdout: "target_version": "0'0" 2024-09-18T08:17:56.230 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:17:56.230 INFO:teuthology.orchestra.run.smithi007.stdout: "last_change": "67", 2024-09-18T08:17:56.230 INFO:teuthology.orchestra.run.smithi007.stdout: "last_force_op_resend": "0", 2024-09-18T08:17:56.230 INFO:teuthology.orchestra.run.smithi007.stdout: "last_force_op_resend_prenautilus": "0", 2024-09-18T08:17:56.230 INFO:teuthology.orchestra.run.smithi007.stdout: "last_force_op_resend_preluminous": "0", 2024-09-18T08:17:56.230 INFO:teuthology.orchestra.run.smithi007.stdout: "auid": 0, 2024-09-18T08:17:56.230 INFO:teuthology.orchestra.run.smithi007.stdout: "snap_mode": "selfmanaged", 2024-09-18T08:17:56.230 INFO:teuthology.orchestra.run.smithi007.stdout: "snap_seq": 0, 2024-09-18T08:17:56.230 INFO:teuthology.orchestra.run.smithi007.stdout: "snap_epoch": 0, 2024-09-18T08:17:56.230 INFO:teuthology.orchestra.run.smithi007.stdout: "pool_snaps": [], 2024-09-18T08:17:56.230 INFO:teuthology.orchestra.run.smithi007.stdout: "removed_snaps": "[]", 2024-09-18T08:17:56.230 INFO:teuthology.orchestra.run.smithi007.stdout: "quota_max_bytes": 0, 2024-09-18T08:17:56.231 INFO:teuthology.orchestra.run.smithi007.stdout: "quota_max_objects": 0, 2024-09-18T08:17:56.231 INFO:teuthology.orchestra.run.smithi007.stdout: "tiers": [], 2024-09-18T08:17:56.231 INFO:teuthology.orchestra.run.smithi007.stdout: "tier_of": -1, 2024-09-18T08:17:56.231 INFO:teuthology.orchestra.run.smithi007.stdout: "read_tier": -1, 2024-09-18T08:17:56.231 INFO:teuthology.orchestra.run.smithi007.stdout: "write_tier": -1, 2024-09-18T08:17:56.231 INFO:teuthology.orchestra.run.smithi007.stdout: "cache_mode": "none", 2024-09-18T08:17:56.231 INFO:teuthology.orchestra.run.smithi007.stdout: "target_max_bytes": 0, 2024-09-18T08:17:56.231 INFO:teuthology.orchestra.run.smithi007.stdout: "target_max_objects": 0, 2024-09-18T08:17:56.231 INFO:teuthology.orchestra.run.smithi007.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-09-18T08:17:56.231 INFO:teuthology.orchestra.run.smithi007.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-09-18T08:17:56.231 INFO:teuthology.orchestra.run.smithi007.stdout: "cache_target_full_ratio_micro": 800000, 2024-09-18T08:17:56.231 INFO:teuthology.orchestra.run.smithi007.stdout: "cache_min_flush_age": 0, 2024-09-18T08:17:56.231 INFO:teuthology.orchestra.run.smithi007.stdout: "cache_min_evict_age": 0, 2024-09-18T08:17:56.231 INFO:teuthology.orchestra.run.smithi007.stdout: "erasure_code_profile": "", 2024-09-18T08:17:56.232 INFO:teuthology.orchestra.run.smithi007.stdout: "hit_set_params": { 2024-09-18T08:17:56.232 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "none" 2024-09-18T08:17:56.232 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:17:56.232 INFO:teuthology.orchestra.run.smithi007.stdout: "hit_set_period": 0, 2024-09-18T08:17:56.232 INFO:teuthology.orchestra.run.smithi007.stdout: "hit_set_count": 0, 2024-09-18T08:17:56.232 INFO:teuthology.orchestra.run.smithi007.stdout: "use_gmt_hitset": true, 2024-09-18T08:17:56.232 INFO:teuthology.orchestra.run.smithi007.stdout: "min_read_recency_for_promote": 0, 2024-09-18T08:17:56.232 INFO:teuthology.orchestra.run.smithi007.stdout: "min_write_recency_for_promote": 0, 2024-09-18T08:17:56.232 INFO:teuthology.orchestra.run.smithi007.stdout: "hit_set_grade_decay_rate": 0, 2024-09-18T08:17:56.232 INFO:teuthology.orchestra.run.smithi007.stdout: "hit_set_search_last_n": 0, 2024-09-18T08:17:56.232 INFO:teuthology.orchestra.run.smithi007.stdout: "grade_table": [], 2024-09-18T08:17:56.232 INFO:teuthology.orchestra.run.smithi007.stdout: "stripe_width": 0, 2024-09-18T08:17:56.232 INFO:teuthology.orchestra.run.smithi007.stdout: "expected_num_objects": 0, 2024-09-18T08:17:56.232 INFO:teuthology.orchestra.run.smithi007.stdout: "fast_read": false, 2024-09-18T08:17:56.233 INFO:teuthology.orchestra.run.smithi007.stdout: "options": {}, 2024-09-18T08:17:56.233 INFO:teuthology.orchestra.run.smithi007.stdout: "application_metadata": { 2024-09-18T08:17:56.233 INFO:teuthology.orchestra.run.smithi007.stdout: "rgw": {} 2024-09-18T08:17:56.233 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:17:56.233 INFO:teuthology.orchestra.run.smithi007.stdout: "read_balance": { 2024-09-18T08:17:56.233 INFO:teuthology.orchestra.run.smithi007.stdout: "score_type": "Fair distribution", 2024-09-18T08:17:56.233 INFO:teuthology.orchestra.run.smithi007.stdout: "score_acting": 1.5199999809265137, 2024-09-18T08:17:56.233 INFO:teuthology.orchestra.run.smithi007.stdout: "score_stable": 1.5199999809265137, 2024-09-18T08:17:56.233 INFO:teuthology.orchestra.run.smithi007.stdout: "optimal_score": 0.86000001430511475, 2024-09-18T08:17:56.233 INFO:teuthology.orchestra.run.smithi007.stdout: "raw_score_acting": 1.309999942779541, 2024-09-18T08:17:56.233 INFO:teuthology.orchestra.run.smithi007.stdout: "raw_score_stable": 1.309999942779541, 2024-09-18T08:17:56.233 INFO:teuthology.orchestra.run.smithi007.stdout: "primary_affinity_weighted": 0.80927085876464844, 2024-09-18T08:17:56.233 INFO:teuthology.orchestra.run.smithi007.stdout: "average_primary_affinity": 0.92000001668930054, 2024-09-18T08:17:56.234 INFO:teuthology.orchestra.run.smithi007.stdout: "average_primary_affinity_weighted": 0.81000000238418579 2024-09-18T08:17:56.234 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-09-18T08:17:56.234 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:17:56.234 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:17:56.234 INFO:teuthology.orchestra.run.smithi007.stdout: "pool": 6, 2024-09-18T08:17:56.234 INFO:teuthology.orchestra.run.smithi007.stdout: "pool_name": "default.rgw.meta", 2024-09-18T08:17:56.234 INFO:teuthology.orchestra.run.smithi007.stdout: "create_time": "2024-09-18T08:02:51.971465+0000", 2024-09-18T08:17:56.234 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 1, 2024-09-18T08:17:56.234 INFO:teuthology.orchestra.run.smithi007.stdout: "flags_names": "hashpspool", 2024-09-18T08:17:56.234 INFO:teuthology.orchestra.run.smithi007.stdout: "type": 1, 2024-09-18T08:17:56.234 INFO:teuthology.orchestra.run.smithi007.stdout: "size": 3, 2024-09-18T08:17:56.234 INFO:teuthology.orchestra.run.smithi007.stdout: "min_size": 2, 2024-09-18T08:17:56.234 INFO:teuthology.orchestra.run.smithi007.stdout: "crush_rule": 0, 2024-09-18T08:17:56.235 INFO:teuthology.orchestra.run.smithi007.stdout: "peering_crush_bucket_count": 0, 2024-09-18T08:17:56.235 INFO:teuthology.orchestra.run.smithi007.stdout: "peering_crush_bucket_target": 0, 2024-09-18T08:17:56.235 INFO:teuthology.orchestra.run.smithi007.stdout: "peering_crush_bucket_barrier": 0, 2024-09-18T08:17:56.235 INFO:teuthology.orchestra.run.smithi007.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-09-18T08:17:56.235 INFO:teuthology.orchestra.run.smithi007.stdout: "is_stretch_pool": false, 2024-09-18T08:17:56.235 INFO:teuthology.orchestra.run.smithi007.stdout: "object_hash": 2, 2024-09-18T08:17:56.235 INFO:teuthology.orchestra.run.smithi007.stdout: "pg_autoscale_mode": "off", 2024-09-18T08:17:56.235 INFO:teuthology.orchestra.run.smithi007.stdout: "pg_num": 32, 2024-09-18T08:17:56.235 INFO:teuthology.orchestra.run.smithi007.stdout: "pg_placement_num": 32, 2024-09-18T08:17:56.235 INFO:teuthology.orchestra.run.smithi007.stdout: "pg_placement_num_target": 32, 2024-09-18T08:17:56.235 INFO:teuthology.orchestra.run.smithi007.stdout: "pg_num_target": 32, 2024-09-18T08:17:56.235 INFO:teuthology.orchestra.run.smithi007.stdout: "pg_num_pending": 32, 2024-09-18T08:17:56.235 INFO:teuthology.orchestra.run.smithi007.stdout: "last_pg_merge_meta": { 2024-09-18T08:17:56.235 INFO:teuthology.orchestra.run.smithi007.stdout: "source_pgid": "0.0", 2024-09-18T08:17:56.236 INFO:teuthology.orchestra.run.smithi007.stdout: "ready_epoch": 0, 2024-09-18T08:17:56.236 INFO:teuthology.orchestra.run.smithi007.stdout: "last_epoch_started": 0, 2024-09-18T08:17:56.236 INFO:teuthology.orchestra.run.smithi007.stdout: "last_epoch_clean": 0, 2024-09-18T08:17:56.236 INFO:teuthology.orchestra.run.smithi007.stdout: "source_version": "0'0", 2024-09-18T08:17:56.236 INFO:teuthology.orchestra.run.smithi007.stdout: "target_version": "0'0" 2024-09-18T08:17:56.236 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:17:56.236 INFO:teuthology.orchestra.run.smithi007.stdout: "last_change": "70", 2024-09-18T08:17:56.236 INFO:teuthology.orchestra.run.smithi007.stdout: "last_force_op_resend": "0", 2024-09-18T08:17:56.236 INFO:teuthology.orchestra.run.smithi007.stdout: "last_force_op_resend_prenautilus": "0", 2024-09-18T08:17:56.236 INFO:teuthology.orchestra.run.smithi007.stdout: "last_force_op_resend_preluminous": "0", 2024-09-18T08:17:56.236 INFO:teuthology.orchestra.run.smithi007.stdout: "auid": 0, 2024-09-18T08:17:56.236 INFO:teuthology.orchestra.run.smithi007.stdout: "snap_mode": "selfmanaged", 2024-09-18T08:17:56.236 INFO:teuthology.orchestra.run.smithi007.stdout: "snap_seq": 0, 2024-09-18T08:17:56.237 INFO:teuthology.orchestra.run.smithi007.stdout: "snap_epoch": 0, 2024-09-18T08:17:56.237 INFO:teuthology.orchestra.run.smithi007.stdout: "pool_snaps": [], 2024-09-18T08:17:56.237 INFO:teuthology.orchestra.run.smithi007.stdout: "removed_snaps": "[]", 2024-09-18T08:17:56.237 INFO:teuthology.orchestra.run.smithi007.stdout: "quota_max_bytes": 0, 2024-09-18T08:17:56.237 INFO:teuthology.orchestra.run.smithi007.stdout: "quota_max_objects": 0, 2024-09-18T08:17:56.237 INFO:teuthology.orchestra.run.smithi007.stdout: "tiers": [], 2024-09-18T08:17:56.237 INFO:teuthology.orchestra.run.smithi007.stdout: "tier_of": -1, 2024-09-18T08:17:56.237 INFO:teuthology.orchestra.run.smithi007.stdout: "read_tier": -1, 2024-09-18T08:17:56.237 INFO:teuthology.orchestra.run.smithi007.stdout: "write_tier": -1, 2024-09-18T08:17:56.237 INFO:teuthology.orchestra.run.smithi007.stdout: "cache_mode": "none", 2024-09-18T08:17:56.237 INFO:teuthology.orchestra.run.smithi007.stdout: "target_max_bytes": 0, 2024-09-18T08:17:56.237 INFO:teuthology.orchestra.run.smithi007.stdout: "target_max_objects": 0, 2024-09-18T08:17:56.237 INFO:teuthology.orchestra.run.smithi007.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-09-18T08:17:56.238 INFO:teuthology.orchestra.run.smithi007.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-09-18T08:17:56.238 INFO:teuthology.orchestra.run.smithi007.stdout: "cache_target_full_ratio_micro": 800000, 2024-09-18T08:17:56.238 INFO:teuthology.orchestra.run.smithi007.stdout: "cache_min_flush_age": 0, 2024-09-18T08:17:56.238 INFO:teuthology.orchestra.run.smithi007.stdout: "cache_min_evict_age": 0, 2024-09-18T08:17:56.238 INFO:teuthology.orchestra.run.smithi007.stdout: "erasure_code_profile": "", 2024-09-18T08:17:56.238 INFO:teuthology.orchestra.run.smithi007.stdout: "hit_set_params": { 2024-09-18T08:17:56.238 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "none" 2024-09-18T08:17:56.238 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:17:56.238 INFO:teuthology.orchestra.run.smithi007.stdout: "hit_set_period": 0, 2024-09-18T08:17:56.238 INFO:teuthology.orchestra.run.smithi007.stdout: "hit_set_count": 0, 2024-09-18T08:17:56.238 INFO:teuthology.orchestra.run.smithi007.stdout: "use_gmt_hitset": true, 2024-09-18T08:17:56.238 INFO:teuthology.orchestra.run.smithi007.stdout: "min_read_recency_for_promote": 0, 2024-09-18T08:17:56.238 INFO:teuthology.orchestra.run.smithi007.stdout: "min_write_recency_for_promote": 0, 2024-09-18T08:17:56.239 INFO:teuthology.orchestra.run.smithi007.stdout: "hit_set_grade_decay_rate": 0, 2024-09-18T08:17:56.239 INFO:teuthology.orchestra.run.smithi007.stdout: "hit_set_search_last_n": 0, 2024-09-18T08:17:56.239 INFO:teuthology.orchestra.run.smithi007.stdout: "grade_table": [], 2024-09-18T08:17:56.239 INFO:teuthology.orchestra.run.smithi007.stdout: "stripe_width": 0, 2024-09-18T08:17:56.239 INFO:teuthology.orchestra.run.smithi007.stdout: "expected_num_objects": 0, 2024-09-18T08:17:56.239 INFO:teuthology.orchestra.run.smithi007.stdout: "fast_read": false, 2024-09-18T08:17:56.239 INFO:teuthology.orchestra.run.smithi007.stdout: "options": { 2024-09-18T08:17:56.239 INFO:teuthology.orchestra.run.smithi007.stdout: "pg_autoscale_bias": 4 2024-09-18T08:17:56.239 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:17:56.239 INFO:teuthology.orchestra.run.smithi007.stdout: "application_metadata": { 2024-09-18T08:17:56.239 INFO:teuthology.orchestra.run.smithi007.stdout: "rgw": {} 2024-09-18T08:17:56.239 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:17:56.239 INFO:teuthology.orchestra.run.smithi007.stdout: "read_balance": { 2024-09-18T08:17:56.239 INFO:teuthology.orchestra.run.smithi007.stdout: "score_type": "Fair distribution", 2024-09-18T08:17:56.240 INFO:teuthology.orchestra.run.smithi007.stdout: "score_acting": 1.9700000286102295, 2024-09-18T08:17:56.240 INFO:teuthology.orchestra.run.smithi007.stdout: "score_stable": 1.9700000286102295, 2024-09-18T08:17:56.240 INFO:teuthology.orchestra.run.smithi007.stdout: "optimal_score": 0.86000001430511475, 2024-09-18T08:17:56.240 INFO:teuthology.orchestra.run.smithi007.stdout: "raw_score_acting": 1.690000057220459, 2024-09-18T08:17:56.240 INFO:teuthology.orchestra.run.smithi007.stdout: "raw_score_stable": 1.690000057220459, 2024-09-18T08:17:56.240 INFO:teuthology.orchestra.run.smithi007.stdout: "primary_affinity_weighted": 0.80927085876464844, 2024-09-18T08:17:56.240 INFO:teuthology.orchestra.run.smithi007.stdout: "average_primary_affinity": 0.92000001668930054, 2024-09-18T08:17:56.240 INFO:teuthology.orchestra.run.smithi007.stdout: "average_primary_affinity_weighted": 0.81000000238418579 2024-09-18T08:17:56.240 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-09-18T08:17:56.240 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:17:56.240 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:17:56.240 INFO:teuthology.orchestra.run.smithi007.stdout: "pool": 10, 2024-09-18T08:17:56.240 INFO:teuthology.orchestra.run.smithi007.stdout: "pool_name": "unique_pool_3", 2024-09-18T08:17:56.241 INFO:teuthology.orchestra.run.smithi007.stdout: "create_time": "2024-09-18T08:14:42.583623+0000", 2024-09-18T08:17:56.241 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 1, 2024-09-18T08:17:56.241 INFO:teuthology.orchestra.run.smithi007.stdout: "flags_names": "hashpspool", 2024-09-18T08:17:56.241 INFO:teuthology.orchestra.run.smithi007.stdout: "type": 1, 2024-09-18T08:17:56.241 INFO:teuthology.orchestra.run.smithi007.stdout: "size": 3, 2024-09-18T08:17:56.241 INFO:teuthology.orchestra.run.smithi007.stdout: "min_size": 2, 2024-09-18T08:17:56.241 INFO:teuthology.orchestra.run.smithi007.stdout: "crush_rule": 0, 2024-09-18T08:17:56.241 INFO:teuthology.orchestra.run.smithi007.stdout: "peering_crush_bucket_count": 0, 2024-09-18T08:17:56.241 INFO:teuthology.orchestra.run.smithi007.stdout: "peering_crush_bucket_target": 0, 2024-09-18T08:17:56.241 INFO:teuthology.orchestra.run.smithi007.stdout: "peering_crush_bucket_barrier": 0, 2024-09-18T08:17:56.241 INFO:teuthology.orchestra.run.smithi007.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-09-18T08:17:56.241 INFO:teuthology.orchestra.run.smithi007.stdout: "is_stretch_pool": false, 2024-09-18T08:17:56.241 INFO:teuthology.orchestra.run.smithi007.stdout: "object_hash": 2, 2024-09-18T08:17:56.242 INFO:teuthology.orchestra.run.smithi007.stdout: "pg_autoscale_mode": "off", 2024-09-18T08:17:56.242 INFO:teuthology.orchestra.run.smithi007.stdout: "pg_num": 16, 2024-09-18T08:17:56.242 INFO:teuthology.orchestra.run.smithi007.stdout: "pg_placement_num": 16, 2024-09-18T08:17:56.242 INFO:teuthology.orchestra.run.smithi007.stdout: "pg_placement_num_target": 16, 2024-09-18T08:17:56.242 INFO:teuthology.orchestra.run.smithi007.stdout: "pg_num_target": 16, 2024-09-18T08:17:56.242 INFO:teuthology.orchestra.run.smithi007.stdout: "pg_num_pending": 16, 2024-09-18T08:17:56.242 INFO:teuthology.orchestra.run.smithi007.stdout: "last_pg_merge_meta": { 2024-09-18T08:17:56.242 INFO:teuthology.orchestra.run.smithi007.stdout: "source_pgid": "0.0", 2024-09-18T08:17:56.242 INFO:teuthology.orchestra.run.smithi007.stdout: "ready_epoch": 0, 2024-09-18T08:17:56.242 INFO:teuthology.orchestra.run.smithi007.stdout: "last_epoch_started": 0, 2024-09-18T08:17:56.242 INFO:teuthology.orchestra.run.smithi007.stdout: "last_epoch_clean": 0, 2024-09-18T08:17:56.242 INFO:teuthology.orchestra.run.smithi007.stdout: "source_version": "0'0", 2024-09-18T08:17:56.242 INFO:teuthology.orchestra.run.smithi007.stdout: "target_version": "0'0" 2024-09-18T08:17:56.242 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:17:56.243 INFO:teuthology.orchestra.run.smithi007.stdout: "last_change": "314", 2024-09-18T08:17:56.243 INFO:teuthology.orchestra.run.smithi007.stdout: "last_force_op_resend": "0", 2024-09-18T08:17:56.243 INFO:teuthology.orchestra.run.smithi007.stdout: "last_force_op_resend_prenautilus": "0", 2024-09-18T08:17:56.243 INFO:teuthology.orchestra.run.smithi007.stdout: "last_force_op_resend_preluminous": "0", 2024-09-18T08:17:56.243 INFO:teuthology.orchestra.run.smithi007.stdout: "auid": 0, 2024-09-18T08:17:56.243 INFO:teuthology.orchestra.run.smithi007.stdout: "snap_mode": "selfmanaged", 2024-09-18T08:17:56.243 INFO:teuthology.orchestra.run.smithi007.stdout: "snap_seq": 0, 2024-09-18T08:17:56.243 INFO:teuthology.orchestra.run.smithi007.stdout: "snap_epoch": 0, 2024-09-18T08:17:56.243 INFO:teuthology.orchestra.run.smithi007.stdout: "pool_snaps": [], 2024-09-18T08:17:56.243 INFO:teuthology.orchestra.run.smithi007.stdout: "removed_snaps": "[]", 2024-09-18T08:17:56.243 INFO:teuthology.orchestra.run.smithi007.stdout: "quota_max_bytes": 0, 2024-09-18T08:17:56.243 INFO:teuthology.orchestra.run.smithi007.stdout: "quota_max_objects": 0, 2024-09-18T08:17:56.243 INFO:teuthology.orchestra.run.smithi007.stdout: "tiers": [], 2024-09-18T08:17:56.244 INFO:teuthology.orchestra.run.smithi007.stdout: "tier_of": -1, 2024-09-18T08:17:56.244 INFO:teuthology.orchestra.run.smithi007.stdout: "read_tier": -1, 2024-09-18T08:17:56.244 INFO:teuthology.orchestra.run.smithi007.stdout: "write_tier": -1, 2024-09-18T08:17:56.244 INFO:teuthology.orchestra.run.smithi007.stdout: "cache_mode": "none", 2024-09-18T08:17:56.244 INFO:teuthology.orchestra.run.smithi007.stdout: "target_max_bytes": 0, 2024-09-18T08:17:56.244 INFO:teuthology.orchestra.run.smithi007.stdout: "target_max_objects": 0, 2024-09-18T08:17:56.244 INFO:teuthology.orchestra.run.smithi007.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-09-18T08:17:56.244 INFO:teuthology.orchestra.run.smithi007.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-09-18T08:17:56.244 INFO:teuthology.orchestra.run.smithi007.stdout: "cache_target_full_ratio_micro": 800000, 2024-09-18T08:17:56.244 INFO:teuthology.orchestra.run.smithi007.stdout: "cache_min_flush_age": 0, 2024-09-18T08:17:56.244 INFO:teuthology.orchestra.run.smithi007.stdout: "cache_min_evict_age": 0, 2024-09-18T08:17:56.244 INFO:teuthology.orchestra.run.smithi007.stdout: "erasure_code_profile": "", 2024-09-18T08:17:56.244 INFO:teuthology.orchestra.run.smithi007.stdout: "hit_set_params": { 2024-09-18T08:17:56.244 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "none" 2024-09-18T08:17:56.245 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:17:56.245 INFO:teuthology.orchestra.run.smithi007.stdout: "hit_set_period": 0, 2024-09-18T08:17:56.245 INFO:teuthology.orchestra.run.smithi007.stdout: "hit_set_count": 0, 2024-09-18T08:17:56.245 INFO:teuthology.orchestra.run.smithi007.stdout: "use_gmt_hitset": true, 2024-09-18T08:17:56.245 INFO:teuthology.orchestra.run.smithi007.stdout: "min_read_recency_for_promote": 0, 2024-09-18T08:17:56.245 INFO:teuthology.orchestra.run.smithi007.stdout: "min_write_recency_for_promote": 0, 2024-09-18T08:17:56.245 INFO:teuthology.orchestra.run.smithi007.stdout: "hit_set_grade_decay_rate": 0, 2024-09-18T08:17:56.245 INFO:teuthology.orchestra.run.smithi007.stdout: "hit_set_search_last_n": 0, 2024-09-18T08:17:56.245 INFO:teuthology.orchestra.run.smithi007.stdout: "grade_table": [], 2024-09-18T08:17:56.245 INFO:teuthology.orchestra.run.smithi007.stdout: "stripe_width": 0, 2024-09-18T08:17:56.245 INFO:teuthology.orchestra.run.smithi007.stdout: "expected_num_objects": 0, 2024-09-18T08:17:56.245 INFO:teuthology.orchestra.run.smithi007.stdout: "fast_read": false, 2024-09-18T08:17:56.245 INFO:teuthology.orchestra.run.smithi007.stdout: "options": {}, 2024-09-18T08:17:56.245 INFO:teuthology.orchestra.run.smithi007.stdout: "application_metadata": { 2024-09-18T08:17:56.246 INFO:teuthology.orchestra.run.smithi007.stdout: "rados": {} 2024-09-18T08:17:56.246 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:17:56.246 INFO:teuthology.orchestra.run.smithi007.stdout: "read_balance": { 2024-09-18T08:17:56.246 INFO:teuthology.orchestra.run.smithi007.stdout: "score_type": "Fair distribution", 2024-09-18T08:17:56.246 INFO:teuthology.orchestra.run.smithi007.stdout: "score_acting": 2.619999885559082, 2024-09-18T08:17:56.246 INFO:teuthology.orchestra.run.smithi007.stdout: "score_stable": 2.619999885559082, 2024-09-18T08:17:56.246 INFO:teuthology.orchestra.run.smithi007.stdout: "optimal_score": 0.86000001430511475, 2024-09-18T08:17:56.246 INFO:teuthology.orchestra.run.smithi007.stdout: "raw_score_acting": 2.25, 2024-09-18T08:17:56.246 INFO:teuthology.orchestra.run.smithi007.stdout: "raw_score_stable": 2.25, 2024-09-18T08:17:56.246 INFO:teuthology.orchestra.run.smithi007.stdout: "primary_affinity_weighted": 0.80927085876464844, 2024-09-18T08:17:56.246 INFO:teuthology.orchestra.run.smithi007.stdout: "average_primary_affinity": 0.92000001668930054, 2024-09-18T08:17:56.246 INFO:teuthology.orchestra.run.smithi007.stdout: "average_primary_affinity_weighted": 0.81000000238418579 2024-09-18T08:17:56.246 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-09-18T08:17:56.246 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-09-18T08:17:56.247 INFO:teuthology.orchestra.run.smithi007.stdout: ], 2024-09-18T08:17:56.247 INFO:teuthology.orchestra.run.smithi007.stdout: "osds": [ 2024-09-18T08:17:56.247 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:17:56.247 INFO:teuthology.orchestra.run.smithi007.stdout: "osd": 0, 2024-09-18T08:17:56.247 INFO:teuthology.orchestra.run.smithi007.stdout: "uuid": "ae8f60df-72ac-46fb-91a1-f132b851c876", 2024-09-18T08:17:56.247 INFO:teuthology.orchestra.run.smithi007.stdout: "up": 1, 2024-09-18T08:17:56.247 INFO:teuthology.orchestra.run.smithi007.stdout: "in": 1, 2024-09-18T08:17:56.247 INFO:teuthology.orchestra.run.smithi007.stdout: "weight": 1, 2024-09-18T08:17:56.247 INFO:teuthology.orchestra.run.smithi007.stdout: "primary_affinity": 1, 2024-09-18T08:17:56.247 INFO:teuthology.orchestra.run.smithi007.stdout: "last_clean_begin": 0, 2024-09-18T08:17:56.247 INFO:teuthology.orchestra.run.smithi007.stdout: "last_clean_end": 0, 2024-09-18T08:17:56.247 INFO:teuthology.orchestra.run.smithi007.stdout: "up_from": 8, 2024-09-18T08:17:56.247 INFO:teuthology.orchestra.run.smithi007.stdout: "up_thru": 377, 2024-09-18T08:17:56.248 INFO:teuthology.orchestra.run.smithi007.stdout: "down_at": 0, 2024-09-18T08:17:56.248 INFO:teuthology.orchestra.run.smithi007.stdout: "lost_at": 0, 2024-09-18T08:17:56.248 INFO:teuthology.orchestra.run.smithi007.stdout: "public_addrs": { 2024-09-18T08:17:56.248 INFO:teuthology.orchestra.run.smithi007.stdout: "addrvec": [ 2024-09-18T08:17:56.248 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:17:56.248 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "v2", 2024-09-18T08:17:56.248 INFO:teuthology.orchestra.run.smithi007.stdout: "addr": "172.21.15.7:6801", 2024-09-18T08:17:56.248 INFO:teuthology.orchestra.run.smithi007.stdout: "nonce": 2896425794 2024-09-18T08:17:56.248 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-09-18T08:17:56.248 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-09-18T08:17:56.248 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:17:56.248 INFO:teuthology.orchestra.run.smithi007.stdout: "cluster_addrs": { 2024-09-18T08:17:56.248 INFO:teuthology.orchestra.run.smithi007.stdout: "addrvec": [ 2024-09-18T08:17:56.248 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:17:56.249 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "v2", 2024-09-18T08:17:56.249 INFO:teuthology.orchestra.run.smithi007.stdout: "addr": "172.21.15.7:6802", 2024-09-18T08:17:56.249 INFO:teuthology.orchestra.run.smithi007.stdout: "nonce": 2896425794 2024-09-18T08:17:56.249 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-09-18T08:17:56.249 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-09-18T08:17:56.249 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:17:56.249 INFO:teuthology.orchestra.run.smithi007.stdout: "heartbeat_back_addrs": { 2024-09-18T08:17:56.249 INFO:teuthology.orchestra.run.smithi007.stdout: "addrvec": [ 2024-09-18T08:17:56.249 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:17:56.249 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "v2", 2024-09-18T08:17:56.249 INFO:teuthology.orchestra.run.smithi007.stdout: "addr": "172.21.15.7:6804", 2024-09-18T08:17:56.249 INFO:teuthology.orchestra.run.smithi007.stdout: "nonce": 2896425794 2024-09-18T08:17:56.249 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-09-18T08:17:56.249 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-09-18T08:17:56.250 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:17:56.250 INFO:teuthology.orchestra.run.smithi007.stdout: "heartbeat_front_addrs": { 2024-09-18T08:17:56.250 INFO:teuthology.orchestra.run.smithi007.stdout: "addrvec": [ 2024-09-18T08:17:56.250 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:17:56.250 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "v2", 2024-09-18T08:17:56.250 INFO:teuthology.orchestra.run.smithi007.stdout: "addr": "172.21.15.7:6803", 2024-09-18T08:17:56.250 INFO:teuthology.orchestra.run.smithi007.stdout: "nonce": 2896425794 2024-09-18T08:17:56.250 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-09-18T08:17:56.250 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-09-18T08:17:56.250 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:17:56.250 INFO:teuthology.orchestra.run.smithi007.stdout: "public_addr": "172.21.15.7:6801/2896425794", 2024-09-18T08:17:56.250 INFO:teuthology.orchestra.run.smithi007.stdout: "cluster_addr": "172.21.15.7:6802/2896425794", 2024-09-18T08:17:56.250 INFO:teuthology.orchestra.run.smithi007.stdout: "heartbeat_back_addr": "172.21.15.7:6804/2896425794", 2024-09-18T08:17:56.251 INFO:teuthology.orchestra.run.smithi007.stdout: "heartbeat_front_addr": "172.21.15.7:6803/2896425794", 2024-09-18T08:17:56.251 INFO:teuthology.orchestra.run.smithi007.stdout: "state": [ 2024-09-18T08:17:56.251 INFO:teuthology.orchestra.run.smithi007.stdout: "exists", 2024-09-18T08:17:56.251 INFO:teuthology.orchestra.run.smithi007.stdout: "up" 2024-09-18T08:17:56.251 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-09-18T08:17:56.251 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:17:56.251 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:17:56.251 INFO:teuthology.orchestra.run.smithi007.stdout: "osd": 1, 2024-09-18T08:17:56.251 INFO:teuthology.orchestra.run.smithi007.stdout: "uuid": "be28b70c-1f73-494f-ad4b-cb7ec5311999", 2024-09-18T08:17:56.251 INFO:teuthology.orchestra.run.smithi007.stdout: "up": 1, 2024-09-18T08:17:56.251 INFO:teuthology.orchestra.run.smithi007.stdout: "in": 1, 2024-09-18T08:17:56.251 INFO:teuthology.orchestra.run.smithi007.stdout: "weight": 1, 2024-09-18T08:17:56.251 INFO:teuthology.orchestra.run.smithi007.stdout: "primary_affinity": 1, 2024-09-18T08:17:56.252 INFO:teuthology.orchestra.run.smithi007.stdout: "last_clean_begin": 13, 2024-09-18T08:17:56.252 INFO:teuthology.orchestra.run.smithi007.stdout: "last_clean_end": 243, 2024-09-18T08:17:56.252 INFO:teuthology.orchestra.run.smithi007.stdout: "up_from": 307, 2024-09-18T08:17:56.252 INFO:teuthology.orchestra.run.smithi007.stdout: "up_thru": 387, 2024-09-18T08:17:56.252 INFO:teuthology.orchestra.run.smithi007.stdout: "down_at": 244, 2024-09-18T08:17:56.252 INFO:teuthology.orchestra.run.smithi007.stdout: "lost_at": 0, 2024-09-18T08:17:56.252 INFO:teuthology.orchestra.run.smithi007.stdout: "public_addrs": { 2024-09-18T08:17:56.253 INFO:teuthology.orchestra.run.smithi007.stdout: "addrvec": [ 2024-09-18T08:17:56.253 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:17:56.253 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "v2", 2024-09-18T08:17:56.253 INFO:teuthology.orchestra.run.smithi007.stdout: "addr": "172.21.15.7:6805", 2024-09-18T08:17:56.253 INFO:teuthology.orchestra.run.smithi007.stdout: "nonce": 3300107791 2024-09-18T08:17:56.253 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-09-18T08:17:56.253 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-09-18T08:17:56.253 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:17:56.253 INFO:teuthology.orchestra.run.smithi007.stdout: "cluster_addrs": { 2024-09-18T08:17:56.253 INFO:teuthology.orchestra.run.smithi007.stdout: "addrvec": [ 2024-09-18T08:17:56.253 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:17:56.254 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "v2", 2024-09-18T08:17:56.254 INFO:teuthology.orchestra.run.smithi007.stdout: "addr": "172.21.15.7:6806", 2024-09-18T08:17:56.254 INFO:teuthology.orchestra.run.smithi007.stdout: "nonce": 3300107791 2024-09-18T08:17:56.254 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-09-18T08:17:56.254 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-09-18T08:17:56.254 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:17:56.254 INFO:teuthology.orchestra.run.smithi007.stdout: "heartbeat_back_addrs": { 2024-09-18T08:17:56.254 INFO:teuthology.orchestra.run.smithi007.stdout: "addrvec": [ 2024-09-18T08:17:56.254 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:17:56.254 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "v2", 2024-09-18T08:17:56.255 INFO:teuthology.orchestra.run.smithi007.stdout: "addr": "172.21.15.7:6808", 2024-09-18T08:17:56.255 INFO:teuthology.orchestra.run.smithi007.stdout: "nonce": 3300107791 2024-09-18T08:17:56.255 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-09-18T08:17:56.255 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-09-18T08:17:56.255 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:17:56.255 INFO:teuthology.orchestra.run.smithi007.stdout: "heartbeat_front_addrs": { 2024-09-18T08:17:56.255 INFO:teuthology.orchestra.run.smithi007.stdout: "addrvec": [ 2024-09-18T08:17:56.255 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:17:56.255 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "v2", 2024-09-18T08:17:56.255 INFO:teuthology.orchestra.run.smithi007.stdout: "addr": "172.21.15.7:6807", 2024-09-18T08:17:56.255 INFO:teuthology.orchestra.run.smithi007.stdout: "nonce": 3300107791 2024-09-18T08:17:56.255 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-09-18T08:17:56.255 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-09-18T08:17:56.255 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:17:56.256 INFO:teuthology.orchestra.run.smithi007.stdout: "public_addr": "172.21.15.7:6805/3300107791", 2024-09-18T08:17:56.256 INFO:teuthology.orchestra.run.smithi007.stdout: "cluster_addr": "172.21.15.7:6806/3300107791", 2024-09-18T08:17:56.256 INFO:teuthology.orchestra.run.smithi007.stdout: "heartbeat_back_addr": "172.21.15.7:6808/3300107791", 2024-09-18T08:17:56.256 INFO:teuthology.orchestra.run.smithi007.stdout: "heartbeat_front_addr": "172.21.15.7:6807/3300107791", 2024-09-18T08:17:56.256 INFO:teuthology.orchestra.run.smithi007.stdout: "state": [ 2024-09-18T08:17:56.256 INFO:teuthology.orchestra.run.smithi007.stdout: "exists", 2024-09-18T08:17:56.256 INFO:teuthology.orchestra.run.smithi007.stdout: "up" 2024-09-18T08:17:56.256 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-09-18T08:17:56.256 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:17:56.256 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:17:56.256 INFO:teuthology.orchestra.run.smithi007.stdout: "osd": 2, 2024-09-18T08:17:56.256 INFO:teuthology.orchestra.run.smithi007.stdout: "uuid": "cfa4f083-c7fd-4743-be65-84a3040fe6dc", 2024-09-18T08:17:56.256 INFO:teuthology.orchestra.run.smithi007.stdout: "up": 1, 2024-09-18T08:17:56.257 INFO:teuthology.orchestra.run.smithi007.stdout: "in": 1, 2024-09-18T08:17:56.257 INFO:teuthology.orchestra.run.smithi007.stdout: "weight": 1, 2024-09-18T08:17:56.257 INFO:teuthology.orchestra.run.smithi007.stdout: "primary_affinity": 1, 2024-09-18T08:17:56.257 INFO:teuthology.orchestra.run.smithi007.stdout: "last_clean_begin": 0, 2024-09-18T08:17:56.257 INFO:teuthology.orchestra.run.smithi007.stdout: "last_clean_end": 0, 2024-09-18T08:17:56.257 INFO:teuthology.orchestra.run.smithi007.stdout: "up_from": 18, 2024-09-18T08:17:56.257 INFO:teuthology.orchestra.run.smithi007.stdout: "up_thru": 386, 2024-09-18T08:17:56.257 INFO:teuthology.orchestra.run.smithi007.stdout: "down_at": 0, 2024-09-18T08:17:56.257 INFO:teuthology.orchestra.run.smithi007.stdout: "lost_at": 0, 2024-09-18T08:17:56.257 INFO:teuthology.orchestra.run.smithi007.stdout: "public_addrs": { 2024-09-18T08:17:56.257 INFO:teuthology.orchestra.run.smithi007.stdout: "addrvec": [ 2024-09-18T08:17:56.257 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:17:56.257 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "v2", 2024-09-18T08:17:56.257 INFO:teuthology.orchestra.run.smithi007.stdout: "addr": "172.21.15.7:6809", 2024-09-18T08:17:56.258 INFO:teuthology.orchestra.run.smithi007.stdout: "nonce": 556142559 2024-09-18T08:17:56.258 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-09-18T08:17:56.258 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-09-18T08:17:56.258 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:17:56.258 INFO:teuthology.orchestra.run.smithi007.stdout: "cluster_addrs": { 2024-09-18T08:17:56.258 INFO:teuthology.orchestra.run.smithi007.stdout: "addrvec": [ 2024-09-18T08:17:56.258 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:17:56.258 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "v2", 2024-09-18T08:17:56.258 INFO:teuthology.orchestra.run.smithi007.stdout: "addr": "172.21.15.7:6810", 2024-09-18T08:17:56.258 INFO:teuthology.orchestra.run.smithi007.stdout: "nonce": 556142559 2024-09-18T08:17:56.258 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-09-18T08:17:56.258 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-09-18T08:17:56.258 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:17:56.258 INFO:teuthology.orchestra.run.smithi007.stdout: "heartbeat_back_addrs": { 2024-09-18T08:17:56.259 INFO:teuthology.orchestra.run.smithi007.stdout: "addrvec": [ 2024-09-18T08:17:56.259 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:17:56.259 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "v2", 2024-09-18T08:17:56.259 INFO:teuthology.orchestra.run.smithi007.stdout: "addr": "172.21.15.7:6812", 2024-09-18T08:17:56.259 INFO:teuthology.orchestra.run.smithi007.stdout: "nonce": 556142559 2024-09-18T08:17:56.259 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-09-18T08:17:56.259 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-09-18T08:17:56.259 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:17:56.259 INFO:teuthology.orchestra.run.smithi007.stdout: "heartbeat_front_addrs": { 2024-09-18T08:17:56.259 INFO:teuthology.orchestra.run.smithi007.stdout: "addrvec": [ 2024-09-18T08:17:56.259 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:17:56.259 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "v2", 2024-09-18T08:17:56.259 INFO:teuthology.orchestra.run.smithi007.stdout: "addr": "172.21.15.7:6811", 2024-09-18T08:17:56.260 INFO:teuthology.orchestra.run.smithi007.stdout: "nonce": 556142559 2024-09-18T08:17:56.260 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-09-18T08:17:56.260 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-09-18T08:17:56.260 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:17:56.260 INFO:teuthology.orchestra.run.smithi007.stdout: "public_addr": "172.21.15.7:6809/556142559", 2024-09-18T08:17:56.260 INFO:teuthology.orchestra.run.smithi007.stdout: "cluster_addr": "172.21.15.7:6810/556142559", 2024-09-18T08:17:56.260 INFO:teuthology.orchestra.run.smithi007.stdout: "heartbeat_back_addr": "172.21.15.7:6812/556142559", 2024-09-18T08:17:56.260 INFO:teuthology.orchestra.run.smithi007.stdout: "heartbeat_front_addr": "172.21.15.7:6811/556142559", 2024-09-18T08:17:56.260 INFO:teuthology.orchestra.run.smithi007.stdout: "state": [ 2024-09-18T08:17:56.260 INFO:teuthology.orchestra.run.smithi007.stdout: "exists", 2024-09-18T08:17:56.260 INFO:teuthology.orchestra.run.smithi007.stdout: "up" 2024-09-18T08:17:56.260 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-09-18T08:17:56.260 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:17:56.261 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:17:56.261 INFO:teuthology.orchestra.run.smithi007.stdout: "osd": 3, 2024-09-18T08:17:56.261 INFO:teuthology.orchestra.run.smithi007.stdout: "uuid": "11ecd7e1-a751-4b75-95f5-00b32d2a5bc4", 2024-09-18T08:17:56.261 INFO:teuthology.orchestra.run.smithi007.stdout: "up": 1, 2024-09-18T08:17:56.261 INFO:teuthology.orchestra.run.smithi007.stdout: "in": 1, 2024-09-18T08:17:56.261 INFO:teuthology.orchestra.run.smithi007.stdout: "weight": 1, 2024-09-18T08:17:56.261 INFO:teuthology.orchestra.run.smithi007.stdout: "primary_affinity": 1, 2024-09-18T08:17:56.261 INFO:teuthology.orchestra.run.smithi007.stdout: "last_clean_begin": 0, 2024-09-18T08:17:56.261 INFO:teuthology.orchestra.run.smithi007.stdout: "last_clean_end": 0, 2024-09-18T08:17:56.261 INFO:teuthology.orchestra.run.smithi007.stdout: "up_from": 26, 2024-09-18T08:17:56.261 INFO:teuthology.orchestra.run.smithi007.stdout: "up_thru": 370, 2024-09-18T08:17:56.261 INFO:teuthology.orchestra.run.smithi007.stdout: "down_at": 0, 2024-09-18T08:17:56.261 INFO:teuthology.orchestra.run.smithi007.stdout: "lost_at": 0, 2024-09-18T08:17:56.261 INFO:teuthology.orchestra.run.smithi007.stdout: "public_addrs": { 2024-09-18T08:17:56.262 INFO:teuthology.orchestra.run.smithi007.stdout: "addrvec": [ 2024-09-18T08:17:56.262 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:17:56.262 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "v2", 2024-09-18T08:17:56.262 INFO:teuthology.orchestra.run.smithi007.stdout: "addr": "172.21.15.7:6813", 2024-09-18T08:17:56.262 INFO:teuthology.orchestra.run.smithi007.stdout: "nonce": 163587772 2024-09-18T08:17:56.262 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-09-18T08:17:56.262 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-09-18T08:17:56.262 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:17:56.262 INFO:teuthology.orchestra.run.smithi007.stdout: "cluster_addrs": { 2024-09-18T08:17:56.262 INFO:teuthology.orchestra.run.smithi007.stdout: "addrvec": [ 2024-09-18T08:17:56.262 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:17:56.262 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "v2", 2024-09-18T08:17:56.262 INFO:teuthology.orchestra.run.smithi007.stdout: "addr": "172.21.15.7:6814", 2024-09-18T08:17:56.263 INFO:teuthology.orchestra.run.smithi007.stdout: "nonce": 163587772 2024-09-18T08:17:56.263 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-09-18T08:17:56.263 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-09-18T08:17:56.263 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:17:56.263 INFO:teuthology.orchestra.run.smithi007.stdout: "heartbeat_back_addrs": { 2024-09-18T08:17:56.263 INFO:teuthology.orchestra.run.smithi007.stdout: "addrvec": [ 2024-09-18T08:17:56.263 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:17:56.263 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "v2", 2024-09-18T08:17:56.263 INFO:teuthology.orchestra.run.smithi007.stdout: "addr": "172.21.15.7:6816", 2024-09-18T08:17:56.263 INFO:teuthology.orchestra.run.smithi007.stdout: "nonce": 163587772 2024-09-18T08:17:56.263 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-09-18T08:17:56.263 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-09-18T08:17:56.263 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:17:56.264 INFO:teuthology.orchestra.run.smithi007.stdout: "heartbeat_front_addrs": { 2024-09-18T08:17:56.264 INFO:teuthology.orchestra.run.smithi007.stdout: "addrvec": [ 2024-09-18T08:17:56.264 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:17:56.264 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "v2", 2024-09-18T08:17:56.264 INFO:teuthology.orchestra.run.smithi007.stdout: "addr": "172.21.15.7:6815", 2024-09-18T08:17:56.264 INFO:teuthology.orchestra.run.smithi007.stdout: "nonce": 163587772 2024-09-18T08:17:56.264 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-09-18T08:17:56.264 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-09-18T08:17:56.264 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:17:56.264 INFO:teuthology.orchestra.run.smithi007.stdout: "public_addr": "172.21.15.7:6813/163587772", 2024-09-18T08:17:56.264 INFO:teuthology.orchestra.run.smithi007.stdout: "cluster_addr": "172.21.15.7:6814/163587772", 2024-09-18T08:17:56.265 INFO:teuthology.orchestra.run.smithi007.stdout: "heartbeat_back_addr": "172.21.15.7:6816/163587772", 2024-09-18T08:17:56.265 INFO:teuthology.orchestra.run.smithi007.stdout: "heartbeat_front_addr": "172.21.15.7:6815/163587772", 2024-09-18T08:17:56.265 INFO:teuthology.orchestra.run.smithi007.stdout: "state": [ 2024-09-18T08:17:56.265 INFO:teuthology.orchestra.run.smithi007.stdout: "exists", 2024-09-18T08:17:56.265 INFO:teuthology.orchestra.run.smithi007.stdout: "up" 2024-09-18T08:17:56.265 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-09-18T08:17:56.265 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:17:56.265 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:17:56.265 INFO:teuthology.orchestra.run.smithi007.stdout: "osd": 4, 2024-09-18T08:17:56.265 INFO:teuthology.orchestra.run.smithi007.stdout: "uuid": "98992b34-cfa1-45bf-97a1-7d557739252f", 2024-09-18T08:17:56.265 INFO:teuthology.orchestra.run.smithi007.stdout: "up": 1, 2024-09-18T08:17:56.265 INFO:teuthology.orchestra.run.smithi007.stdout: "in": 0, 2024-09-18T08:17:56.265 INFO:teuthology.orchestra.run.smithi007.stdout: "weight": 0, 2024-09-18T08:17:56.265 INFO:teuthology.orchestra.run.smithi007.stdout: "primary_affinity": 1, 2024-09-18T08:17:56.266 INFO:teuthology.orchestra.run.smithi007.stdout: "last_clean_begin": 0, 2024-09-18T08:17:56.266 INFO:teuthology.orchestra.run.smithi007.stdout: "last_clean_end": 0, 2024-09-18T08:17:56.266 INFO:teuthology.orchestra.run.smithi007.stdout: "up_from": 32, 2024-09-18T08:17:56.266 INFO:teuthology.orchestra.run.smithi007.stdout: "up_thru": 336, 2024-09-18T08:17:56.266 INFO:teuthology.orchestra.run.smithi007.stdout: "down_at": 0, 2024-09-18T08:17:56.266 INFO:teuthology.orchestra.run.smithi007.stdout: "lost_at": 0, 2024-09-18T08:17:56.266 INFO:teuthology.orchestra.run.smithi007.stdout: "public_addrs": { 2024-09-18T08:17:56.266 INFO:teuthology.orchestra.run.smithi007.stdout: "addrvec": [ 2024-09-18T08:17:56.266 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:17:56.266 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "v2", 2024-09-18T08:17:56.266 INFO:teuthology.orchestra.run.smithi007.stdout: "addr": "172.21.15.50:6800", 2024-09-18T08:17:56.266 INFO:teuthology.orchestra.run.smithi007.stdout: "nonce": 3662119805 2024-09-18T08:17:56.266 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-09-18T08:17:56.266 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-09-18T08:17:56.266 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:17:56.267 INFO:teuthology.orchestra.run.smithi007.stdout: "cluster_addrs": { 2024-09-18T08:17:56.267 INFO:teuthology.orchestra.run.smithi007.stdout: "addrvec": [ 2024-09-18T08:17:56.267 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:17:56.267 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "v2", 2024-09-18T08:17:56.267 INFO:teuthology.orchestra.run.smithi007.stdout: "addr": "172.21.15.50:6801", 2024-09-18T08:17:56.267 INFO:teuthology.orchestra.run.smithi007.stdout: "nonce": 3662119805 2024-09-18T08:17:56.267 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-09-18T08:17:56.267 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-09-18T08:17:56.267 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:17:56.267 INFO:teuthology.orchestra.run.smithi007.stdout: "heartbeat_back_addrs": { 2024-09-18T08:17:56.267 INFO:teuthology.orchestra.run.smithi007.stdout: "addrvec": [ 2024-09-18T08:17:56.267 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:17:56.267 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "v2", 2024-09-18T08:17:56.267 INFO:teuthology.orchestra.run.smithi007.stdout: "addr": "172.21.15.50:6803", 2024-09-18T08:17:56.267 INFO:teuthology.orchestra.run.smithi007.stdout: "nonce": 3662119805 2024-09-18T08:17:56.268 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-09-18T08:17:56.268 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-09-18T08:17:56.268 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:17:56.268 INFO:teuthology.orchestra.run.smithi007.stdout: "heartbeat_front_addrs": { 2024-09-18T08:17:56.268 INFO:teuthology.orchestra.run.smithi007.stdout: "addrvec": [ 2024-09-18T08:17:56.268 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:17:56.268 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "v2", 2024-09-18T08:17:56.268 INFO:teuthology.orchestra.run.smithi007.stdout: "addr": "172.21.15.50:6802", 2024-09-18T08:17:56.268 INFO:teuthology.orchestra.run.smithi007.stdout: "nonce": 3662119805 2024-09-18T08:17:56.268 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-09-18T08:17:56.268 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-09-18T08:17:56.268 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:17:56.268 INFO:teuthology.orchestra.run.smithi007.stdout: "public_addr": "172.21.15.50:6800/3662119805", 2024-09-18T08:17:56.268 INFO:teuthology.orchestra.run.smithi007.stdout: "cluster_addr": "172.21.15.50:6801/3662119805", 2024-09-18T08:17:56.269 INFO:teuthology.orchestra.run.smithi007.stdout: "heartbeat_back_addr": "172.21.15.50:6803/3662119805", 2024-09-18T08:17:56.269 INFO:teuthology.orchestra.run.smithi007.stdout: "heartbeat_front_addr": "172.21.15.50:6802/3662119805", 2024-09-18T08:17:56.269 INFO:teuthology.orchestra.run.smithi007.stdout: "state": [ 2024-09-18T08:17:56.269 INFO:teuthology.orchestra.run.smithi007.stdout: "exists", 2024-09-18T08:17:56.269 INFO:teuthology.orchestra.run.smithi007.stdout: "up" 2024-09-18T08:17:56.269 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-09-18T08:17:56.269 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:17:56.269 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:17:56.269 INFO:teuthology.orchestra.run.smithi007.stdout: "osd": 5, 2024-09-18T08:17:56.269 INFO:teuthology.orchestra.run.smithi007.stdout: "uuid": "7f290005-ebca-467b-87e4-02ead1063610", 2024-09-18T08:17:56.269 INFO:teuthology.orchestra.run.smithi007.stdout: "up": 1, 2024-09-18T08:17:56.269 INFO:teuthology.orchestra.run.smithi007.stdout: "in": 1, 2024-09-18T08:17:56.269 INFO:teuthology.orchestra.run.smithi007.stdout: "weight": 1, 2024-09-18T08:17:56.269 INFO:teuthology.orchestra.run.smithi007.stdout: "primary_affinity": 1, 2024-09-18T08:17:56.270 INFO:teuthology.orchestra.run.smithi007.stdout: "last_clean_begin": 0, 2024-09-18T08:17:56.270 INFO:teuthology.orchestra.run.smithi007.stdout: "last_clean_end": 0, 2024-09-18T08:17:56.270 INFO:teuthology.orchestra.run.smithi007.stdout: "up_from": 39, 2024-09-18T08:17:56.270 INFO:teuthology.orchestra.run.smithi007.stdout: "up_thru": 363, 2024-09-18T08:17:56.270 INFO:teuthology.orchestra.run.smithi007.stdout: "down_at": 0, 2024-09-18T08:17:56.270 INFO:teuthology.orchestra.run.smithi007.stdout: "lost_at": 0, 2024-09-18T08:17:56.270 INFO:teuthology.orchestra.run.smithi007.stdout: "public_addrs": { 2024-09-18T08:17:56.270 INFO:teuthology.orchestra.run.smithi007.stdout: "addrvec": [ 2024-09-18T08:17:56.270 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:17:56.270 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "v2", 2024-09-18T08:17:56.270 INFO:teuthology.orchestra.run.smithi007.stdout: "addr": "172.21.15.50:6804", 2024-09-18T08:17:56.270 INFO:teuthology.orchestra.run.smithi007.stdout: "nonce": 1199797954 2024-09-18T08:17:56.270 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-09-18T08:17:56.270 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-09-18T08:17:56.270 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:17:56.271 INFO:teuthology.orchestra.run.smithi007.stdout: "cluster_addrs": { 2024-09-18T08:17:56.271 INFO:teuthology.orchestra.run.smithi007.stdout: "addrvec": [ 2024-09-18T08:17:56.271 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:17:56.271 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "v2", 2024-09-18T08:17:56.271 INFO:teuthology.orchestra.run.smithi007.stdout: "addr": "172.21.15.50:6805", 2024-09-18T08:17:56.271 INFO:teuthology.orchestra.run.smithi007.stdout: "nonce": 1199797954 2024-09-18T08:17:56.271 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-09-18T08:17:56.271 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-09-18T08:17:56.271 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:17:56.271 INFO:teuthology.orchestra.run.smithi007.stdout: "heartbeat_back_addrs": { 2024-09-18T08:17:56.271 INFO:teuthology.orchestra.run.smithi007.stdout: "addrvec": [ 2024-09-18T08:17:56.271 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:17:56.272 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "v2", 2024-09-18T08:17:56.272 INFO:teuthology.orchestra.run.smithi007.stdout: "addr": "172.21.15.50:6807", 2024-09-18T08:17:56.272 INFO:teuthology.orchestra.run.smithi007.stdout: "nonce": 1199797954 2024-09-18T08:17:56.272 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-09-18T08:17:56.272 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-09-18T08:17:56.272 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:17:56.272 INFO:teuthology.orchestra.run.smithi007.stdout: "heartbeat_front_addrs": { 2024-09-18T08:17:56.272 INFO:teuthology.orchestra.run.smithi007.stdout: "addrvec": [ 2024-09-18T08:17:56.272 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:17:56.272 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "v2", 2024-09-18T08:17:56.272 INFO:teuthology.orchestra.run.smithi007.stdout: "addr": "172.21.15.50:6806", 2024-09-18T08:17:56.272 INFO:teuthology.orchestra.run.smithi007.stdout: "nonce": 1199797954 2024-09-18T08:17:56.272 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-09-18T08:17:56.272 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-09-18T08:17:56.273 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:17:56.273 INFO:teuthology.orchestra.run.smithi007.stdout: "public_addr": "172.21.15.50:6804/1199797954", 2024-09-18T08:17:56.273 INFO:teuthology.orchestra.run.smithi007.stdout: "cluster_addr": "172.21.15.50:6805/1199797954", 2024-09-18T08:17:56.273 INFO:teuthology.orchestra.run.smithi007.stdout: "heartbeat_back_addr": "172.21.15.50:6807/1199797954", 2024-09-18T08:17:56.273 INFO:teuthology.orchestra.run.smithi007.stdout: "heartbeat_front_addr": "172.21.15.50:6806/1199797954", 2024-09-18T08:17:56.273 INFO:teuthology.orchestra.run.smithi007.stdout: "state": [ 2024-09-18T08:17:56.273 INFO:teuthology.orchestra.run.smithi007.stdout: "exists", 2024-09-18T08:17:56.273 INFO:teuthology.orchestra.run.smithi007.stdout: "up" 2024-09-18T08:17:56.273 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-09-18T08:17:56.273 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:17:56.273 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:17:56.273 INFO:teuthology.orchestra.run.smithi007.stdout: "osd": 6, 2024-09-18T08:17:56.273 INFO:teuthology.orchestra.run.smithi007.stdout: "uuid": "e07d846c-ae42-4fea-aa5c-3a25835d18a6", 2024-09-18T08:17:56.274 INFO:teuthology.orchestra.run.smithi007.stdout: "up": 1, 2024-09-18T08:17:56.274 INFO:teuthology.orchestra.run.smithi007.stdout: "in": 1, 2024-09-18T08:17:56.274 INFO:teuthology.orchestra.run.smithi007.stdout: "weight": 1, 2024-09-18T08:17:56.274 INFO:teuthology.orchestra.run.smithi007.stdout: "primary_affinity": 0, 2024-09-18T08:17:56.274 INFO:teuthology.orchestra.run.smithi007.stdout: "last_clean_begin": 0, 2024-09-18T08:17:56.274 INFO:teuthology.orchestra.run.smithi007.stdout: "last_clean_end": 0, 2024-09-18T08:17:56.274 INFO:teuthology.orchestra.run.smithi007.stdout: "up_from": 47, 2024-09-18T08:17:56.274 INFO:teuthology.orchestra.run.smithi007.stdout: "up_thru": 244, 2024-09-18T08:17:56.274 INFO:teuthology.orchestra.run.smithi007.stdout: "down_at": 0, 2024-09-18T08:17:56.274 INFO:teuthology.orchestra.run.smithi007.stdout: "lost_at": 0, 2024-09-18T08:17:56.274 INFO:teuthology.orchestra.run.smithi007.stdout: "public_addrs": { 2024-09-18T08:17:56.274 INFO:teuthology.orchestra.run.smithi007.stdout: "addrvec": [ 2024-09-18T08:17:56.274 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:17:56.274 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "v2", 2024-09-18T08:17:56.275 INFO:teuthology.orchestra.run.smithi007.stdout: "addr": "172.21.15.50:6808", 2024-09-18T08:17:56.275 INFO:teuthology.orchestra.run.smithi007.stdout: "nonce": 255960422 2024-09-18T08:17:56.275 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-09-18T08:17:56.275 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-09-18T08:17:56.275 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:17:56.275 INFO:teuthology.orchestra.run.smithi007.stdout: "cluster_addrs": { 2024-09-18T08:17:56.275 INFO:teuthology.orchestra.run.smithi007.stdout: "addrvec": [ 2024-09-18T08:17:56.275 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:17:56.275 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "v2", 2024-09-18T08:17:56.275 INFO:teuthology.orchestra.run.smithi007.stdout: "addr": "172.21.15.50:6809", 2024-09-18T08:17:56.275 INFO:teuthology.orchestra.run.smithi007.stdout: "nonce": 255960422 2024-09-18T08:17:56.275 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-09-18T08:17:56.275 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-09-18T08:17:56.276 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:17:56.276 INFO:teuthology.orchestra.run.smithi007.stdout: "heartbeat_back_addrs": { 2024-09-18T08:17:56.276 INFO:teuthology.orchestra.run.smithi007.stdout: "addrvec": [ 2024-09-18T08:17:56.276 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:17:56.276 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "v2", 2024-09-18T08:17:56.276 INFO:teuthology.orchestra.run.smithi007.stdout: "addr": "172.21.15.50:6811", 2024-09-18T08:17:56.276 INFO:teuthology.orchestra.run.smithi007.stdout: "nonce": 255960422 2024-09-18T08:17:56.276 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-09-18T08:17:56.276 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-09-18T08:17:56.276 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:17:56.276 INFO:teuthology.orchestra.run.smithi007.stdout: "heartbeat_front_addrs": { 2024-09-18T08:17:56.276 INFO:teuthology.orchestra.run.smithi007.stdout: "addrvec": [ 2024-09-18T08:17:56.276 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:17:56.276 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "v2", 2024-09-18T08:17:56.277 INFO:teuthology.orchestra.run.smithi007.stdout: "addr": "172.21.15.50:6810", 2024-09-18T08:17:56.277 INFO:teuthology.orchestra.run.smithi007.stdout: "nonce": 255960422 2024-09-18T08:17:56.277 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-09-18T08:17:56.277 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-09-18T08:17:56.277 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:17:56.277 INFO:teuthology.orchestra.run.smithi007.stdout: "public_addr": "172.21.15.50:6808/255960422", 2024-09-18T08:17:56.277 INFO:teuthology.orchestra.run.smithi007.stdout: "cluster_addr": "172.21.15.50:6809/255960422", 2024-09-18T08:17:56.277 INFO:teuthology.orchestra.run.smithi007.stdout: "heartbeat_back_addr": "172.21.15.50:6811/255960422", 2024-09-18T08:17:56.277 INFO:teuthology.orchestra.run.smithi007.stdout: "heartbeat_front_addr": "172.21.15.50:6810/255960422", 2024-09-18T08:17:56.277 INFO:teuthology.orchestra.run.smithi007.stdout: "state": [ 2024-09-18T08:17:56.277 INFO:teuthology.orchestra.run.smithi007.stdout: "exists", 2024-09-18T08:17:56.277 INFO:teuthology.orchestra.run.smithi007.stdout: "up" 2024-09-18T08:17:56.277 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-09-18T08:17:56.278 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:17:56.278 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:17:56.278 INFO:teuthology.orchestra.run.smithi007.stdout: "osd": 7, 2024-09-18T08:17:56.278 INFO:teuthology.orchestra.run.smithi007.stdout: "uuid": "c9dac3b7-d651-489d-b96e-5b10fa32d9af", 2024-09-18T08:17:56.278 INFO:teuthology.orchestra.run.smithi007.stdout: "up": 1, 2024-09-18T08:17:56.278 INFO:teuthology.orchestra.run.smithi007.stdout: "in": 0, 2024-09-18T08:17:56.278 INFO:teuthology.orchestra.run.smithi007.stdout: "weight": 0, 2024-09-18T08:17:56.278 INFO:teuthology.orchestra.run.smithi007.stdout: "primary_affinity": 0.4741668701171875, 2024-09-18T08:17:56.278 INFO:teuthology.orchestra.run.smithi007.stdout: "last_clean_begin": 0, 2024-09-18T08:17:56.278 INFO:teuthology.orchestra.run.smithi007.stdout: "last_clean_end": 0, 2024-09-18T08:17:56.278 INFO:teuthology.orchestra.run.smithi007.stdout: "up_from": 56, 2024-09-18T08:17:56.278 INFO:teuthology.orchestra.run.smithi007.stdout: "up_thru": 244, 2024-09-18T08:17:56.278 INFO:teuthology.orchestra.run.smithi007.stdout: "down_at": 0, 2024-09-18T08:17:56.279 INFO:teuthology.orchestra.run.smithi007.stdout: "lost_at": 0, 2024-09-18T08:17:56.279 INFO:teuthology.orchestra.run.smithi007.stdout: "public_addrs": { 2024-09-18T08:17:56.279 INFO:teuthology.orchestra.run.smithi007.stdout: "addrvec": [ 2024-09-18T08:17:56.279 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:17:56.279 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "v2", 2024-09-18T08:17:56.279 INFO:teuthology.orchestra.run.smithi007.stdout: "addr": "172.21.15.50:6812", 2024-09-18T08:17:56.279 INFO:teuthology.orchestra.run.smithi007.stdout: "nonce": 874923904 2024-09-18T08:17:56.279 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-09-18T08:17:56.279 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-09-18T08:17:56.279 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:17:56.279 INFO:teuthology.orchestra.run.smithi007.stdout: "cluster_addrs": { 2024-09-18T08:17:56.279 INFO:teuthology.orchestra.run.smithi007.stdout: "addrvec": [ 2024-09-18T08:17:56.279 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:17:56.280 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "v2", 2024-09-18T08:17:56.280 INFO:teuthology.orchestra.run.smithi007.stdout: "addr": "172.21.15.50:6813", 2024-09-18T08:17:56.280 INFO:teuthology.orchestra.run.smithi007.stdout: "nonce": 874923904 2024-09-18T08:17:56.280 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-09-18T08:17:56.280 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-09-18T08:17:56.280 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:17:56.280 INFO:teuthology.orchestra.run.smithi007.stdout: "heartbeat_back_addrs": { 2024-09-18T08:17:56.280 INFO:teuthology.orchestra.run.smithi007.stdout: "addrvec": [ 2024-09-18T08:17:56.280 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:17:56.280 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "v2", 2024-09-18T08:17:56.280 INFO:teuthology.orchestra.run.smithi007.stdout: "addr": "172.21.15.50:6815", 2024-09-18T08:17:56.280 INFO:teuthology.orchestra.run.smithi007.stdout: "nonce": 874923904 2024-09-18T08:17:56.280 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-09-18T08:17:56.280 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-09-18T08:17:56.281 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:17:56.281 INFO:teuthology.orchestra.run.smithi007.stdout: "heartbeat_front_addrs": { 2024-09-18T08:17:56.281 INFO:teuthology.orchestra.run.smithi007.stdout: "addrvec": [ 2024-09-18T08:17:56.281 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:17:56.281 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "v2", 2024-09-18T08:17:56.281 INFO:teuthology.orchestra.run.smithi007.stdout: "addr": "172.21.15.50:6814", 2024-09-18T08:17:56.281 INFO:teuthology.orchestra.run.smithi007.stdout: "nonce": 874923904 2024-09-18T08:17:56.281 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-09-18T08:17:56.281 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-09-18T08:17:56.281 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:17:56.281 INFO:teuthology.orchestra.run.smithi007.stdout: "public_addr": "172.21.15.50:6812/874923904", 2024-09-18T08:17:56.281 INFO:teuthology.orchestra.run.smithi007.stdout: "cluster_addr": "172.21.15.50:6813/874923904", 2024-09-18T08:17:56.281 INFO:teuthology.orchestra.run.smithi007.stdout: "heartbeat_back_addr": "172.21.15.50:6815/874923904", 2024-09-18T08:17:56.282 INFO:teuthology.orchestra.run.smithi007.stdout: "heartbeat_front_addr": "172.21.15.50:6814/874923904", 2024-09-18T08:17:56.282 INFO:teuthology.orchestra.run.smithi007.stdout: "state": [ 2024-09-18T08:17:56.282 INFO:teuthology.orchestra.run.smithi007.stdout: "exists", 2024-09-18T08:17:56.282 INFO:teuthology.orchestra.run.smithi007.stdout: "up" 2024-09-18T08:17:56.282 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-09-18T08:17:56.282 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-09-18T08:17:56.282 INFO:teuthology.orchestra.run.smithi007.stdout: ], 2024-09-18T08:17:56.282 INFO:teuthology.orchestra.run.smithi007.stdout: "osd_xinfo": [ 2024-09-18T08:17:56.282 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:17:56.282 INFO:teuthology.orchestra.run.smithi007.stdout: "osd": 0, 2024-09-18T08:17:56.282 INFO:teuthology.orchestra.run.smithi007.stdout: "down_stamp": "0.000000", 2024-09-18T08:17:56.282 INFO:teuthology.orchestra.run.smithi007.stdout: "laggy_probability": 0, 2024-09-18T08:17:56.282 INFO:teuthology.orchestra.run.smithi007.stdout: "laggy_interval": 0, 2024-09-18T08:17:56.283 INFO:teuthology.orchestra.run.smithi007.stdout: "features": 4540719139924082687, 2024-09-18T08:17:56.283 INFO:teuthology.orchestra.run.smithi007.stdout: "old_weight": 0, 2024-09-18T08:17:56.283 INFO:teuthology.orchestra.run.smithi007.stdout: "last_purged_snaps_scrub": "2024-09-18T08:17:33.038707+0000", 2024-09-18T08:17:56.283 INFO:teuthology.orchestra.run.smithi007.stdout: "dead_epoch": 0 2024-09-18T08:17:56.283 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:17:56.283 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:17:56.283 INFO:teuthology.orchestra.run.smithi007.stdout: "osd": 1, 2024-09-18T08:17:56.283 INFO:teuthology.orchestra.run.smithi007.stdout: "down_stamp": "0.000000", 2024-09-18T08:17:56.283 INFO:teuthology.orchestra.run.smithi007.stdout: "laggy_probability": 0, 2024-09-18T08:17:56.283 INFO:teuthology.orchestra.run.smithi007.stdout: "laggy_interval": 0, 2024-09-18T08:17:56.283 INFO:teuthology.orchestra.run.smithi007.stdout: "features": 4540719139924082687, 2024-09-18T08:17:56.283 INFO:teuthology.orchestra.run.smithi007.stdout: "old_weight": 0, 2024-09-18T08:17:56.283 INFO:teuthology.orchestra.run.smithi007.stdout: "last_purged_snaps_scrub": "2024-09-18T08:16:58.977612+0000", 2024-09-18T08:17:56.283 INFO:teuthology.orchestra.run.smithi007.stdout: "dead_epoch": 243 2024-09-18T08:17:56.284 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:17:56.284 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:17:56.284 INFO:teuthology.orchestra.run.smithi007.stdout: "osd": 2, 2024-09-18T08:17:56.284 INFO:teuthology.orchestra.run.smithi007.stdout: "down_stamp": "0.000000", 2024-09-18T08:17:56.284 INFO:teuthology.orchestra.run.smithi007.stdout: "laggy_probability": 0, 2024-09-18T08:17:56.284 INFO:teuthology.orchestra.run.smithi007.stdout: "laggy_interval": 0, 2024-09-18T08:17:56.284 INFO:teuthology.orchestra.run.smithi007.stdout: "features": 4540719139924082687, 2024-09-18T08:17:56.284 INFO:teuthology.orchestra.run.smithi007.stdout: "old_weight": 0, 2024-09-18T08:17:56.284 INFO:teuthology.orchestra.run.smithi007.stdout: "last_purged_snaps_scrub": "2024-09-18T08:17:32.795625+0000", 2024-09-18T08:17:56.284 INFO:teuthology.orchestra.run.smithi007.stdout: "dead_epoch": 0 2024-09-18T08:17:56.284 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:17:56.284 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:17:56.284 INFO:teuthology.orchestra.run.smithi007.stdout: "osd": 3, 2024-09-18T08:17:56.285 INFO:teuthology.orchestra.run.smithi007.stdout: "down_stamp": "0.000000", 2024-09-18T08:17:56.285 INFO:teuthology.orchestra.run.smithi007.stdout: "laggy_probability": 0, 2024-09-18T08:17:56.285 INFO:teuthology.orchestra.run.smithi007.stdout: "laggy_interval": 0, 2024-09-18T08:17:56.285 INFO:teuthology.orchestra.run.smithi007.stdout: "features": 4540719139924082687, 2024-09-18T08:17:56.285 INFO:teuthology.orchestra.run.smithi007.stdout: "old_weight": 0, 2024-09-18T08:17:56.285 INFO:teuthology.orchestra.run.smithi007.stdout: "last_purged_snaps_scrub": "2024-09-18T08:16:57.272929+0000", 2024-09-18T08:17:56.285 INFO:teuthology.orchestra.run.smithi007.stdout: "dead_epoch": 0 2024-09-18T08:17:56.285 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:17:56.285 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:17:56.285 INFO:teuthology.orchestra.run.smithi007.stdout: "osd": 4, 2024-09-18T08:17:56.285 INFO:teuthology.orchestra.run.smithi007.stdout: "down_stamp": "0.000000", 2024-09-18T08:17:56.285 INFO:teuthology.orchestra.run.smithi007.stdout: "laggy_probability": 0, 2024-09-18T08:17:56.285 INFO:teuthology.orchestra.run.smithi007.stdout: "laggy_interval": 0, 2024-09-18T08:17:56.285 INFO:teuthology.orchestra.run.smithi007.stdout: "features": 4540719139924082687, 2024-09-18T08:17:56.286 INFO:teuthology.orchestra.run.smithi007.stdout: "old_weight": 65536, 2024-09-18T08:17:56.286 INFO:teuthology.orchestra.run.smithi007.stdout: "last_purged_snaps_scrub": "2024-09-18T08:16:53.844644+0000", 2024-09-18T08:17:56.286 INFO:teuthology.orchestra.run.smithi007.stdout: "dead_epoch": 0 2024-09-18T08:17:56.286 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:17:56.286 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:17:56.286 INFO:teuthology.orchestra.run.smithi007.stdout: "osd": 5, 2024-09-18T08:17:56.286 INFO:teuthology.orchestra.run.smithi007.stdout: "down_stamp": "0.000000", 2024-09-18T08:17:56.286 INFO:teuthology.orchestra.run.smithi007.stdout: "laggy_probability": 0, 2024-09-18T08:17:56.286 INFO:teuthology.orchestra.run.smithi007.stdout: "laggy_interval": 0, 2024-09-18T08:17:56.286 INFO:teuthology.orchestra.run.smithi007.stdout: "features": 4540719139924082687, 2024-09-18T08:17:56.286 INFO:teuthology.orchestra.run.smithi007.stdout: "old_weight": 0, 2024-09-18T08:17:56.286 INFO:teuthology.orchestra.run.smithi007.stdout: "last_purged_snaps_scrub": "2024-09-18T08:17:17.949416+0000", 2024-09-18T08:17:56.286 INFO:teuthology.orchestra.run.smithi007.stdout: "dead_epoch": 0 2024-09-18T08:17:56.287 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:17:56.287 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:17:56.287 INFO:teuthology.orchestra.run.smithi007.stdout: "osd": 6, 2024-09-18T08:17:56.287 INFO:teuthology.orchestra.run.smithi007.stdout: "down_stamp": "0.000000", 2024-09-18T08:17:56.287 INFO:teuthology.orchestra.run.smithi007.stdout: "laggy_probability": 0, 2024-09-18T08:17:56.287 INFO:teuthology.orchestra.run.smithi007.stdout: "laggy_interval": 0, 2024-09-18T08:17:56.287 INFO:teuthology.orchestra.run.smithi007.stdout: "features": 4540719139924082687, 2024-09-18T08:17:56.287 INFO:teuthology.orchestra.run.smithi007.stdout: "old_weight": 0, 2024-09-18T08:17:56.287 INFO:teuthology.orchestra.run.smithi007.stdout: "last_purged_snaps_scrub": "2024-09-18T08:17:00.738958+0000", 2024-09-18T08:17:56.287 INFO:teuthology.orchestra.run.smithi007.stdout: "dead_epoch": 0 2024-09-18T08:17:56.287 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:17:56.287 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:17:56.287 INFO:teuthology.orchestra.run.smithi007.stdout: "osd": 7, 2024-09-18T08:17:56.287 INFO:teuthology.orchestra.run.smithi007.stdout: "down_stamp": "0.000000", 2024-09-18T08:17:56.288 INFO:teuthology.orchestra.run.smithi007.stdout: "laggy_probability": 0, 2024-09-18T08:17:56.288 INFO:teuthology.orchestra.run.smithi007.stdout: "laggy_interval": 0, 2024-09-18T08:17:56.288 INFO:teuthology.orchestra.run.smithi007.stdout: "features": 4540719139924082687, 2024-09-18T08:17:56.288 INFO:teuthology.orchestra.run.smithi007.stdout: "old_weight": 65536, 2024-09-18T08:17:56.288 INFO:teuthology.orchestra.run.smithi007.stdout: "last_purged_snaps_scrub": "2024-09-18T08:17:42.528532+0000", 2024-09-18T08:17:56.288 INFO:teuthology.orchestra.run.smithi007.stdout: "dead_epoch": 0 2024-09-18T08:17:56.288 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-09-18T08:17:56.288 INFO:teuthology.orchestra.run.smithi007.stdout: ], 2024-09-18T08:17:56.288 INFO:teuthology.orchestra.run.smithi007.stdout: "pg_upmap": [ 2024-09-18T08:17:56.288 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:17:56.288 INFO:teuthology.orchestra.run.smithi007.stdout: "pgid": "6.3", 2024-09-18T08:17:56.288 INFO:teuthology.orchestra.run.smithi007.stdout: "osds": [ 2024-09-18T08:17:56.288 INFO:teuthology.orchestra.run.smithi007.stdout: 5, 2024-09-18T08:17:56.288 INFO:teuthology.orchestra.run.smithi007.stdout: 3, 2024-09-18T08:17:56.289 INFO:teuthology.orchestra.run.smithi007.stdout: 2 2024-09-18T08:17:56.289 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-09-18T08:17:56.289 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-09-18T08:17:56.289 INFO:teuthology.orchestra.run.smithi007.stdout: ], 2024-09-18T08:17:56.289 INFO:teuthology.orchestra.run.smithi007.stdout: "pg_upmap_items": [ 2024-09-18T08:17:56.289 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:17:56.289 INFO:teuthology.orchestra.run.smithi007.stdout: "pgid": "4.7", 2024-09-18T08:17:56.289 INFO:teuthology.orchestra.run.smithi007.stdout: "mappings": [ 2024-09-18T08:17:56.289 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:17:56.289 INFO:teuthology.orchestra.run.smithi007.stdout: "from": 1, 2024-09-18T08:17:56.289 INFO:teuthology.orchestra.run.smithi007.stdout: "to": 2 2024-09-18T08:17:56.289 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-09-18T08:17:56.289 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-09-18T08:17:56.290 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:17:56.290 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:17:56.290 INFO:teuthology.orchestra.run.smithi007.stdout: "pgid": "4.14", 2024-09-18T08:17:56.290 INFO:teuthology.orchestra.run.smithi007.stdout: "mappings": [ 2024-09-18T08:17:56.290 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:17:56.290 INFO:teuthology.orchestra.run.smithi007.stdout: "from": 1, 2024-09-18T08:17:56.290 INFO:teuthology.orchestra.run.smithi007.stdout: "to": 2 2024-09-18T08:17:56.290 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-09-18T08:17:56.290 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-09-18T08:17:56.290 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:17:56.290 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:17:56.290 INFO:teuthology.orchestra.run.smithi007.stdout: "pgid": "4.18", 2024-09-18T08:17:56.290 INFO:teuthology.orchestra.run.smithi007.stdout: "mappings": [ 2024-09-18T08:17:56.291 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:17:56.291 INFO:teuthology.orchestra.run.smithi007.stdout: "from": 1, 2024-09-18T08:17:56.291 INFO:teuthology.orchestra.run.smithi007.stdout: "to": 2 2024-09-18T08:17:56.291 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-09-18T08:17:56.291 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-09-18T08:17:56.291 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-09-18T08:17:56.291 INFO:teuthology.orchestra.run.smithi007.stdout: ], 2024-09-18T08:17:56.291 INFO:teuthology.orchestra.run.smithi007.stdout: "pg_upmap_primaries": [], 2024-09-18T08:17:56.291 INFO:teuthology.orchestra.run.smithi007.stdout: "pg_temp": [], 2024-09-18T08:17:56.291 INFO:teuthology.orchestra.run.smithi007.stdout: "primary_temp": [], 2024-09-18T08:17:56.291 INFO:teuthology.orchestra.run.smithi007.stdout: "blocklist": { 2024-09-18T08:17:56.291 INFO:teuthology.orchestra.run.smithi007.stdout: "172.21.15.7:0/398367658": "2024-09-19T08:03:17.052903+0000", 2024-09-18T08:17:56.292 INFO:teuthology.orchestra.run.smithi007.stdout: "172.21.15.7:6800/2918496291": "2024-09-19T07:57:31.015269+0000", 2024-09-18T08:17:56.292 INFO:teuthology.orchestra.run.smithi007.stdout: "172.21.15.7:0/2514354773": "2024-09-19T07:57:31.015269+0000", 2024-09-18T08:17:56.292 INFO:teuthology.orchestra.run.smithi007.stdout: "172.21.15.7:0/3924911335": "2024-09-19T07:57:31.015269+0000", 2024-09-18T08:17:56.292 INFO:teuthology.orchestra.run.smithi007.stdout: "172.21.15.7:0/1668598084": "2024-09-19T07:57:57.346149+0000", 2024-09-18T08:17:56.292 INFO:teuthology.orchestra.run.smithi007.stdout: "172.21.15.7:0/4272027292": "2024-09-19T07:57:31.015269+0000", 2024-09-18T08:17:56.292 INFO:teuthology.orchestra.run.smithi007.stdout: "172.21.15.7:6800/1013061071": "2024-09-19T07:57:57.346149+0000", 2024-09-18T08:17:56.292 INFO:teuthology.orchestra.run.smithi007.stdout: "172.21.15.7:6800/1491061076": "2024-09-19T08:03:17.052903+0000", 2024-09-18T08:17:56.292 INFO:teuthology.orchestra.run.smithi007.stdout: "172.21.15.7:0/533727535": "2024-09-19T07:57:57.346149+0000", 2024-09-18T08:17:56.292 INFO:teuthology.orchestra.run.smithi007.stdout: "172.21.15.7:0/1605217610": "2024-09-19T08:03:17.052903+0000", 2024-09-18T08:17:56.292 INFO:teuthology.orchestra.run.smithi007.stdout: "172.21.15.7:0/1722968065": "2024-09-19T08:03:17.052903+0000", 2024-09-18T08:17:56.292 INFO:teuthology.orchestra.run.smithi007.stdout: "172.21.15.7:0/1277549846": "2024-09-19T07:57:57.346149+0000", 2024-09-18T08:17:56.292 INFO:teuthology.orchestra.run.smithi007.stdout: "172.21.15.7:0/232452890": "2024-09-19T08:03:17.052903+0000", 2024-09-18T08:17:56.292 INFO:teuthology.orchestra.run.smithi007.stdout: "172.21.15.7:0/3881670470": "2024-09-19T08:03:17.052903+0000" 2024-09-18T08:17:56.293 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:17:56.293 INFO:teuthology.orchestra.run.smithi007.stdout: "range_blocklist": {}, 2024-09-18T08:17:56.293 INFO:teuthology.orchestra.run.smithi007.stdout: "erasure_code_profiles": { 2024-09-18T08:17:56.293 INFO:teuthology.orchestra.run.smithi007.stdout: "default": { 2024-09-18T08:17:56.293 INFO:teuthology.orchestra.run.smithi007.stdout: "crush-failure-domain": "osd", 2024-09-18T08:17:56.293 INFO:teuthology.orchestra.run.smithi007.stdout: "k": "2", 2024-09-18T08:17:56.293 INFO:teuthology.orchestra.run.smithi007.stdout: "m": "1", 2024-09-18T08:17:56.293 INFO:teuthology.orchestra.run.smithi007.stdout: "plugin": "jerasure", 2024-09-18T08:17:56.293 INFO:teuthology.orchestra.run.smithi007.stdout: "technique": "reed_sol_van" 2024-09-18T08:17:56.293 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-09-18T08:17:56.293 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:17:56.293 INFO:teuthology.orchestra.run.smithi007.stdout: "removed_snaps_queue": [], 2024-09-18T08:17:56.293 INFO:teuthology.orchestra.run.smithi007.stdout: "new_removed_snaps": [], 2024-09-18T08:17:56.294 INFO:teuthology.orchestra.run.smithi007.stdout: "new_purged_snaps": [], 2024-09-18T08:17:56.294 INFO:teuthology.orchestra.run.smithi007.stdout: "crush_node_flags": {}, 2024-09-18T08:17:56.294 INFO:teuthology.orchestra.run.smithi007.stdout: "device_class_flags": {}, 2024-09-18T08:17:56.294 INFO:teuthology.orchestra.run.smithi007.stdout: "stretch_mode": { 2024-09-18T08:17:56.294 INFO:teuthology.orchestra.run.smithi007.stdout: "stretch_mode_enabled": false, 2024-09-18T08:17:56.294 INFO:teuthology.orchestra.run.smithi007.stdout: "stretch_bucket_count": 0, 2024-09-18T08:17:56.294 INFO:teuthology.orchestra.run.smithi007.stdout: "degraded_stretch_mode": 0, 2024-09-18T08:17:56.294 INFO:teuthology.orchestra.run.smithi007.stdout: "recovering_stretch_mode": 0, 2024-09-18T08:17:56.294 INFO:teuthology.orchestra.run.smithi007.stdout: "stretch_mode_bucket": 0 2024-09-18T08:17:56.294 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-09-18T08:17:56.294 INFO:teuthology.orchestra.run.smithi007.stdout:} 2024-09-18T08:17:56.295 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3 2024-09-18T08:17:56.334 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:17:56.334 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:17:56.334 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3' 2024-09-18T08:17:56.405 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:17:55 smithi050 ceph-mon[32371]: pgmap v757: 158 pgs: 1 active+remapped+backfilling, 157 active+clean; 1.2 GiB data, 6.9 GiB used, 530 GiB / 536 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.05k op/s; 408/59571 objects misplaced (0.685%); 21 MiB/s, 334 objects/s recovering 2024-09-18T08:17:56.435 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0 2024-09-18T08:17:56.478 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:17:56.479 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:17:56.479 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0' 2024-09-18T08:17:56.580 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6 2024-09-18T08:17:56.604 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:17:56.605 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:17:56.605 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6' 2024-09-18T08:17:56.706 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3 2024-09-18T08:17:56.728 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:17:56.729 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:17:56.729 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3' 2024-09-18T08:17:56.830 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7 2024-09-18T08:17:56.853 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:17:56.854 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:17:56.854 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7' 2024-09-18T08:17:56.955 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1 2024-09-18T08:17:56.980 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:17:56 smithi007 ceph-mon[29459]: from='client.? 172.21.15.7:0/2960423741' entity='client.admin' cmd={"prefix": "osd dump", "format": "json-pretty"} : dispatch 2024-09-18T08:17:56.981 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Sep 18 08:17:56 smithi007 ceph-779349ca-7593-11ef-bceb-c7b262605968-mgr-y[29688]: ::ffff:172.21.15.50 - - [18/Sep/2024:08:17:56] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-09-18T08:17:56.981 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:17:56 smithi007 ceph-mon[39249]: from='client.? 172.21.15.7:0/2960423741' entity='client.admin' cmd={"prefix": "osd dump", "format": "json-pretty"} : dispatch 2024-09-18T08:17:57.000 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:17:57.001 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:17:57.001 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1' 2024-09-18T08:17:57.102 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1 2024-09-18T08:17:57.137 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:17:57.138 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:17:57.139 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1' 2024-09-18T08:17:57.240 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4 2024-09-18T08:17:57.265 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:17:57.265 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:17:57.266 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4' 2024-09-18T08:17:57.367 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6 2024-09-18T08:17:57.391 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:17:57.392 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:17:57.392 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6' 2024-09-18T08:17:57.405 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:17:56 smithi050 ceph-mon[32371]: from='client.? 172.21.15.7:0/2960423741' entity='client.admin' cmd={"prefix": "osd dump", "format": "json-pretty"} : dispatch 2024-09-18T08:17:57.493 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5 2024-09-18T08:17:57.519 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:17:57.519 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:17:57.519 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5' 2024-09-18T08:17:57.620 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0 2024-09-18T08:17:57.657 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:17:57.658 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:17:57.658 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0' 2024-09-18T08:17:57.759 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7 2024-09-18T08:17:57.784 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:17:57.784 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:17:57.785 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7' 2024-09-18T08:17:57.886 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2 2024-09-18T08:17:57.935 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:17:57.936 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:17:57.936 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2' 2024-09-18T08:17:57.980 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:17:57 smithi007 ceph-mon[29459]: pgmap v758: 158 pgs: 1 active+remapped+backfilling, 157 active+clean; 1.2 GiB data, 6.9 GiB used, 530 GiB / 536 GiB avail; 922 B/s rd, 0 B/s wr, 758 op/s; 408/59571 objects misplaced (0.685%); 15 MiB/s, 241 objects/s recovering 2024-09-18T08:17:57.980 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:17:57 smithi007 ceph-mon[39249]: pgmap v758: 158 pgs: 1 active+remapped+backfilling, 157 active+clean; 1.2 GiB data, 6.9 GiB used, 530 GiB / 536 GiB avail; 922 B/s rd, 0 B/s wr, 758 op/s; 408/59571 objects misplaced (0.685%); 15 MiB/s, 241 objects/s recovering 2024-09-18T08:17:58.037 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6 2024-09-18T08:17:58.060 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:17:58.061 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:17:58.061 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6' 2024-09-18T08:17:58.162 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2 2024-09-18T08:17:58.206 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:17:58.207 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:17:58.207 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2' 2024-09-18T08:17:58.308 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4 2024-09-18T08:17:58.333 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:17:58.334 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:17:58.334 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4' 2024-09-18T08:17:58.405 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:17:57 smithi050 ceph-mon[32371]: pgmap v758: 158 pgs: 1 active+remapped+backfilling, 157 active+clean; 1.2 GiB data, 6.9 GiB used, 530 GiB / 536 GiB avail; 922 B/s rd, 0 B/s wr, 758 op/s; 408/59571 objects misplaced (0.685%); 15 MiB/s, 241 objects/s recovering 2024-09-18T08:17:58.434 INFO:tasks.thrashosds.thrasher:j is {'epoch': 388, 'fsid': '779349ca-7593-11ef-bceb-c7b262605968', 'created': '2024-09-18T07:57:09.295710+0000', 'modified': '2024-09-18T08:17:54.412045+0000', 'last_up_change': '2024-09-18T08:14:38.391465+0000', 'last_in_change': '2024-09-18T08:16:12.396744+0000', 'flags': 'noscrub,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit', 'flags_num': 5801984, 'flags_set': ['noscrub', 'pglog_hardlimit', 'purged_snapdirs', 'recovery_deletes', 'sortbitwise'], 'crush_version': 19, 'full_ratio': 0.949999988079071, 'backfillfull_ratio': 0.8999999761581421, 'nearfull_ratio': 0.8500000238418579, 'cluster_snapshot': '', 'pool_max': 10, 'max_osd': 8, 'require_min_compat_client': 'luminous', 'min_compat_client': 'luminous', 'require_osd_release': 'squid', 'allow_crimson': False, 'pools': [{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-09-18T08:00:11.802627+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'is_stretch_pool': False, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 11, 'pg_placement_num': 11, 'pg_placement_num_target': 11, 'pg_num_target': 11, 'pg_num_pending': 11, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '285', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '152', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.909999966621399, 'score_stable': 1.909999966621399, 'optimal_score': 0.8600000143051147, 'raw_score_acting': 1.6399999856948853, 'raw_score_stable': 1.6399999856948853, 'primary_affinity_weighted': 0.8092708587646484, 'average_primary_affinity': 0.9200000166893005, 'average_primary_affinity_weighted': 0.8100000023841858}}, {'pool': 2, 'pool_name': 'datapool', 'create_time': '2024-09-18T08:02:45.374099+0000', 'flags': 8193, 'flags_names': 'hashpspool,selfmanaged_snaps', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'is_stretch_pool': False, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 3, 'pg_placement_num': 3, 'pg_placement_num_target': 3, 'pg_num_target': 3, 'pg_num_pending': 3, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '64', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 2, 'snap_epoch': 64, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.3299999237060547, 'score_stable': 2.3299999237060547, 'optimal_score': 0.8600000143051147, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 0.8092708587646484, 'average_primary_affinity': 0.9200000166893005, 'average_primary_affinity_weighted': 0.8100000023841858}}, {'pool': 3, 'pool_name': '.rgw.root', 'create_time': '2024-09-18T08:02:46.319947+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'is_stretch_pool': False, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '63', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rgw': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.190000057220459, 'score_stable': 2.190000057220459, 'optimal_score': 0.8600000143051147, 'raw_score_acting': 1.8799999952316284, 'raw_score_stable': 1.8799999952316284, 'primary_affinity_weighted': 0.8092708587646484, 'average_primary_affinity': 0.9200000166893005, 'average_primary_affinity_weighted': 0.8100000023841858}}, {'pool': 4, 'pool_name': 'default.rgw.log', 'create_time': '2024-09-18T08:02:47.948552+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'is_stretch_pool': False, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '65', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rgw': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.190000057220459, 'score_stable': 2.190000057220459, 'optimal_score': 0.8600000143051147, 'raw_score_acting': 1.8799999952316284, 'raw_score_stable': 1.8799999952316284, 'primary_affinity_weighted': 0.8092708587646484, 'average_primary_affinity': 0.9200000166893005, 'average_primary_affinity_weighted': 0.8100000023841858}}, {'pool': 5, 'pool_name': 'default.rgw.control', 'create_time': '2024-09-18T08:02:49.904302+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'is_stretch_pool': False, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '67', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rgw': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5199999809265137, 'score_stable': 1.5199999809265137, 'optimal_score': 0.8600000143051147, 'raw_score_acting': 1.309999942779541, 'raw_score_stable': 1.309999942779541, 'primary_affinity_weighted': 0.8092708587646484, 'average_primary_affinity': 0.9200000166893005, 'average_primary_affinity_weighted': 0.8100000023841858}}, {'pool': 6, 'pool_name': 'default.rgw.meta', 'create_time': '2024-09-18T08:02:51.971465+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'is_stretch_pool': False, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '70', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4}, 'application_metadata': {'rgw': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.9700000286102295, 'score_stable': 1.9700000286102295, 'optimal_score': 0.8600000143051147, 'raw_score_acting': 1.690000057220459, 'raw_score_stable': 1.690000057220459, 'primary_affinity_weighted': 0.8092708587646484, 'average_primary_affinity': 0.9200000166893005, 'average_primary_affinity_weighted': 0.8100000023841858}}, {'pool': 10, 'pool_name': 'unique_pool_3', 'create_time': '2024-09-18T08:14:42.583623+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'is_stretch_pool': False, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 16, 'pg_placement_num': 16, 'pg_placement_num_target': 16, 'pg_num_target': 16, 'pg_num_pending': 16, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '314', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rados': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.619999885559082, 'score_stable': 2.619999885559082, 'optimal_score': 0.8600000143051147, 'raw_score_acting': 2.25, 'raw_score_stable': 2.25, 'primary_affinity_weighted': 0.8092708587646484, 'average_primary_affinity': 0.9200000166893005, 'average_primary_affinity_weighted': 0.8100000023841858}}], 'osds': [{'osd': 0, 'uuid': 'ae8f60df-72ac-46fb-91a1-f132b851c876', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 8, 'up_thru': 377, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.7:6801', 'nonce': 2896425794}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.7:6802', 'nonce': 2896425794}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.7:6804', 'nonce': 2896425794}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.7:6803', 'nonce': 2896425794}]}, 'public_addr': '172.21.15.7:6801/2896425794', 'cluster_addr': '172.21.15.7:6802/2896425794', 'heartbeat_back_addr': '172.21.15.7:6804/2896425794', 'heartbeat_front_addr': '172.21.15.7:6803/2896425794', 'state': ['exists', 'up']}, {'osd': 1, 'uuid': 'be28b70c-1f73-494f-ad4b-cb7ec5311999', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 13, 'last_clean_end': 243, 'up_from': 307, 'up_thru': 387, 'down_at': 244, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.7:6805', 'nonce': 3300107791}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.7:6806', 'nonce': 3300107791}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.7:6808', 'nonce': 3300107791}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.7:6807', 'nonce': 3300107791}]}, 'public_addr': '172.21.15.7:6805/3300107791', 'cluster_addr': '172.21.15.7:6806/3300107791', 'heartbeat_back_addr': '172.21.15.7:6808/3300107791', 'heartbeat_front_addr': '172.21.15.7:6807/3300107791', 'state': ['exists', 'up']}, {'osd': 2, 'uuid': 'cfa4f083-c7fd-4743-be65-84a3040fe6dc', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 18, 'up_thru': 386, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.7:6809', 'nonce': 556142559}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.7:6810', 'nonce': 556142559}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.7:6812', 'nonce': 556142559}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.7:6811', 'nonce': 556142559}]}, 'public_addr': '172.21.15.7:6809/556142559', 'cluster_addr': '172.21.15.7:6810/556142559', 'heartbeat_back_addr': '172.21.15.7:6812/556142559', 'heartbeat_front_addr': '172.21.15.7:6811/556142559', 'state': ['exists', 'up']}, {'osd': 3, 'uuid': '11ecd7e1-a751-4b75-95f5-00b32d2a5bc4', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 26, 'up_thru': 370, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.7:6813', 'nonce': 163587772}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.7:6814', 'nonce': 163587772}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.7:6816', 'nonce': 163587772}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.7:6815', 'nonce': 163587772}]}, 'public_addr': '172.21.15.7:6813/163587772', 'cluster_addr': '172.21.15.7:6814/163587772', 'heartbeat_back_addr': '172.21.15.7:6816/163587772', 'heartbeat_front_addr': '172.21.15.7:6815/163587772', 'state': ['exists', 'up']}, {'osd': 4, 'uuid': '98992b34-cfa1-45bf-97a1-7d557739252f', 'up': 1, 'in': 0, 'weight': 0, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 32, 'up_thru': 336, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.50:6800', 'nonce': 3662119805}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.50:6801', 'nonce': 3662119805}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.50:6803', 'nonce': 3662119805}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.50:6802', 'nonce': 3662119805}]}, 'public_addr': '172.21.15.50:6800/3662119805', 'cluster_addr': '172.21.15.50:6801/3662119805', 'heartbeat_back_addr': '172.21.15.50:6803/3662119805', 'heartbeat_front_addr': '172.21.15.50:6802/3662119805', 'state': ['exists', 'up']}, {'osd': 5, 'uuid': '7f290005-ebca-467b-87e4-02ead1063610', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 39, 'up_thru': 363, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.50:6804', 'nonce': 1199797954}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.50:6805', 'nonce': 1199797954}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.50:6807', 'nonce': 1199797954}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.50:6806', 'nonce': 1199797954}]}, 'public_addr': '172.21.15.50:6804/1199797954', 'cluster_addr': '172.21.15.50:6805/1199797954', 'heartbeat_back_addr': '172.21.15.50:6807/1199797954', 'heartbeat_front_addr': '172.21.15.50:6806/1199797954', 'state': ['exists', 'up']}, {'osd': 6, 'uuid': 'e07d846c-ae42-4fea-aa5c-3a25835d18a6', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 0, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 47, 'up_thru': 244, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.50:6808', 'nonce': 255960422}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.50:6809', 'nonce': 255960422}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.50:6811', 'nonce': 255960422}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.50:6810', 'nonce': 255960422}]}, 'public_addr': '172.21.15.50:6808/255960422', 'cluster_addr': '172.21.15.50:6809/255960422', 'heartbeat_back_addr': '172.21.15.50:6811/255960422', 'heartbeat_front_addr': '172.21.15.50:6810/255960422', 'state': ['exists', 'up']}, {'osd': 7, 'uuid': 'c9dac3b7-d651-489d-b96e-5b10fa32d9af', 'up': 1, 'in': 0, 'weight': 0, 'primary_affinity': 0.4741668701171875, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 56, 'up_thru': 244, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.50:6812', 'nonce': 874923904}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.50:6813', 'nonce': 874923904}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.50:6815', 'nonce': 874923904}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.50:6814', 'nonce': 874923904}]}, 'public_addr': '172.21.15.50:6812/874923904', 'cluster_addr': '172.21.15.50:6813/874923904', 'heartbeat_back_addr': '172.21.15.50:6815/874923904', 'heartbeat_front_addr': '172.21.15.50:6814/874923904', 'state': ['exists', 'up']}], 'osd_xinfo': [{'osd': 0, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540719139924082687, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-18T08:17:33.038707+0000', 'dead_epoch': 0}, {'osd': 1, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540719139924082687, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-18T08:16:58.977612+0000', 'dead_epoch': 243}, {'osd': 2, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540719139924082687, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-18T08:17:32.795625+0000', 'dead_epoch': 0}, {'osd': 3, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540719139924082687, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-18T08:16:57.272929+0000', 'dead_epoch': 0}, {'osd': 4, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540719139924082687, 'old_weight': 65536, 'last_purged_snaps_scrub': '2024-09-18T08:16:53.844644+0000', 'dead_epoch': 0}, {'osd': 5, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540719139924082687, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-18T08:17:17.949416+0000', 'dead_epoch': 0}, {'osd': 6, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540719139924082687, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-18T08:17:00.738958+0000', 'dead_epoch': 0}, {'osd': 7, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540719139924082687, 'old_weight': 65536, 'last_purged_snaps_scrub': '2024-09-18T08:17:42.528532+0000', 'dead_epoch': 0}], 'pg_upmap': [{'pgid': '6.3', 'osds': [5, 3, 2]}], 'pg_upmap_items': [{'pgid': '4.7', 'mappings': [{'from': 1, 'to': 2}]}, {'pgid': '4.14', 'mappings': [{'from': 1, 'to': 2}]}, {'pgid': '4.18', 'mappings': [{'from': 1, 'to': 2}]}], 'pg_upmap_primaries': [], 'pg_temp': [], 'primary_temp': [], 'blocklist': {'172.21.15.7:0/398367658': '2024-09-19T08:03:17.052903+0000', '172.21.15.7:6800/2918496291': '2024-09-19T07:57:31.015269+0000', '172.21.15.7:0/2514354773': '2024-09-19T07:57:31.015269+0000', '172.21.15.7:0/3924911335': '2024-09-19T07:57:31.015269+0000', '172.21.15.7:0/1668598084': '2024-09-19T07:57:57.346149+0000', '172.21.15.7:0/4272027292': '2024-09-19T07:57:31.015269+0000', '172.21.15.7:6800/1013061071': '2024-09-19T07:57:57.346149+0000', '172.21.15.7:6800/1491061076': '2024-09-19T08:03:17.052903+0000', '172.21.15.7:0/533727535': '2024-09-19T07:57:57.346149+0000', '172.21.15.7:0/1605217610': '2024-09-19T08:03:17.052903+0000', '172.21.15.7:0/1722968065': '2024-09-19T08:03:17.052903+0000', '172.21.15.7:0/1277549846': '2024-09-19T07:57:57.346149+0000', '172.21.15.7:0/232452890': '2024-09-19T08:03:17.052903+0000', '172.21.15.7:0/3881670470': '2024-09-19T08:03:17.052903+0000'}, 'range_blocklist': {}, 'erasure_code_profiles': {'default': {'crush-failure-domain': 'osd', 'k': '2', 'm': '1', 'plugin': 'jerasure', 'technique': 'reed_sol_van'}}, 'removed_snaps_queue': [], 'new_removed_snaps': [], 'new_purged_snaps': [], 'crush_node_flags': {}, 'device_class_flags': {}, 'stretch_mode': {'stretch_mode_enabled': False, 'stretch_bucket_count': 0, 'degraded_stretch_mode': 0, 'recovering_stretch_mode': 0, 'stretch_mode_bucket': 0}} 2024-09-18T08:17:58.434 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid 779349ca-7593-11ef-bceb-c7b262605968 -- ceph pg dump --format=json 2024-09-18T08:17:58.435 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1 2024-09-18T08:17:58.471 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/779349ca-7593-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T08:17:58.499 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:17:58.500 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:17:58.500 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1' 2024-09-18T08:17:58.560 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/779349ca-7593-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T08:17:58.601 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6 2024-09-18T08:17:58.625 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:17:58.626 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:17:58.626 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6' 2024-09-18T08:17:58.727 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6 2024-09-18T08:17:58.752 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:17:58.752 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:17:58.753 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6' 2024-09-18T08:17:58.853 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1 2024-09-18T08:17:58.913 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:17:58.918 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:17:58.918 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1' 2024-09-18T08:17:59.019 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7 2024-09-18T08:17:59.044 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:17:59.045 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:17:59.045 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7' 2024-09-18T08:17:59.146 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6 2024-09-18T08:17:59.170 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:17:59.170 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:17:59.171 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6' 2024-09-18T08:17:59.272 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2 2024-09-18T08:17:59.316 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:17:59.316 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:17:59.317 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2' 2024-09-18T08:17:59.418 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7 2024-09-18T08:17:59.444 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:17:59.444 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:17:59.445 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7' 2024-09-18T08:17:59.520 INFO:teuthology.orchestra.run.smithi007.stdout:ERROR: (22) Invalid argument 2024-09-18T08:17:59.520 INFO:teuthology.orchestra.run.smithi007.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-18T08:17:59.546 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1 2024-09-18T08:17:59.589 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:17:59.590 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:17:59.591 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1' 2024-09-18T08:17:59.691 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7 2024-09-18T08:17:59.716 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:17:59.717 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:17:59.717 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7' 2024-09-18T08:17:59.772 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid 779349ca-7593-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-18T08:17:59.818 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5 2024-09-18T08:17:59.843 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:17:59.844 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:17:59.845 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5' 2024-09-18T08:17:59.905 INFO:journalctl@ceph.iscsi.iscsi.a.smithi050.stdout:Sep 18 08:17:59 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-iscsi-iscsi-a[57782]: debug there is no tcmu-runner data available 2024-09-18T08:17:59.945 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1 2024-09-18T08:17:59.978 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:17:59.979 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:17:59.979 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1' 2024-09-18T08:18:00.080 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5 2024-09-18T08:18:00.104 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:18:00.105 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:18:00.106 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5' 2024-09-18T08:18:00.206 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3 2024-09-18T08:18:00.245 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:18:00.246 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:18:00.246 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3' 2024-09-18T08:18:00.347 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3 2024-09-18T08:18:00.386 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:18:00.387 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:18:00.387 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3' 2024-09-18T08:18:00.488 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1 2024-09-18T08:18:00.521 INFO:teuthology.orchestra.run.smithi007.stderr:nodeep-scrub is set 2024-09-18T08:18:00.526 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:18:00.527 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:18:00.527 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1' 2024-09-18T08:18:00.627 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2 2024-09-18T08:18:00.664 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:18:00.665 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:18:00.665 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2' 2024-09-18T08:18:00.766 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2 2024-09-18T08:18:00.804 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:18:00.805 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:18:00.805 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2' 2024-09-18T08:18:00.905 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:18:00 smithi050 ceph-mon[32371]: pgmap v759: 158 pgs: 158 active+clean; 1.0 GiB data, 6.3 GiB used, 530 GiB / 536 GiB avail; 783 B/s rd, 0 B/s wr, 939 op/s; 11 MiB/s, 168 objects/s recovering 2024-09-18T08:18:00.905 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:18:00 smithi050 ceph-mon[32371]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-18T08:18:00.905 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:18:00 smithi050 ceph-mon[32371]: from='client.? 172.21.15.7:0/861730615' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-18T08:18:00.905 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:18:00 smithi050 ceph-mon[32371]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-18T08:18:00.909 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2 2024-09-18T08:18:00.931 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:18:00.931 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:18:00.932 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2' 2024-09-18T08:18:00.980 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:18:00 smithi007 ceph-mon[29459]: pgmap v759: 158 pgs: 158 active+clean; 1.0 GiB data, 6.3 GiB used, 530 GiB / 536 GiB avail; 783 B/s rd, 0 B/s wr, 939 op/s; 11 MiB/s, 168 objects/s recovering 2024-09-18T08:18:00.980 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:18:00 smithi007 ceph-mon[29459]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-18T08:18:00.980 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:18:00 smithi007 ceph-mon[29459]: from='client.? 172.21.15.7:0/861730615' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-18T08:18:00.980 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:18:00 smithi007 ceph-mon[29459]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-18T08:18:00.980 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:18:00 smithi007 ceph-mon[39249]: pgmap v759: 158 pgs: 158 active+clean; 1.0 GiB data, 6.3 GiB used, 530 GiB / 536 GiB avail; 783 B/s rd, 0 B/s wr, 939 op/s; 11 MiB/s, 168 objects/s recovering 2024-09-18T08:18:00.980 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:18:00 smithi007 ceph-mon[39249]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-18T08:18:00.980 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:18:00 smithi007 ceph-mon[39249]: from='client.? 172.21.15.7:0/861730615' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-18T08:18:00.980 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:18:00 smithi007 ceph-mon[39249]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-18T08:18:01.033 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6 2024-09-18T08:18:01.059 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:18:01.059 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:18:01.060 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6' 2024-09-18T08:18:01.160 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5 2024-09-18T08:18:01.186 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:18:01.187 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:18:01.187 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5' 2024-09-18T08:18:01.288 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1 2024-09-18T08:18:01.310 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:18:01.310 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:18:01.311 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1' 2024-09-18T08:18:01.411 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4 2024-09-18T08:18:01.436 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:18:01.437 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:18:01.437 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4' 2024-09-18T08:18:01.538 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1 2024-09-18T08:18:01.576 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:18:01.577 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:18:01.577 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1' 2024-09-18T08:18:01.678 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7 2024-09-18T08:18:01.703 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:18:01.704 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:18:01.704 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7' 2024-09-18T08:18:01.805 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7 2024-09-18T08:18:01.830 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:18:01.831 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:18:01.831 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7' 2024-09-18T08:18:01.905 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:18:01 smithi050 ceph-mon[32371]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-18T08:18:01.905 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:18:01 smithi050 ceph-mon[32371]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-18T08:18:01.905 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:18:01 smithi050 ceph-mon[32371]: osdmap e389: 8 total, 8 up, 6 in 2024-09-18T08:18:01.932 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3 2024-09-18T08:18:01.967 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:18:01.968 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:18:01.969 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3' 2024-09-18T08:18:01.979 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:18:01 smithi007 ceph-mon[29459]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-18T08:18:01.979 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:18:01 smithi007 ceph-mon[29459]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-18T08:18:01.980 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:18:01 smithi007 ceph-mon[29459]: osdmap e389: 8 total, 8 up, 6 in 2024-09-18T08:18:01.980 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:18:01 smithi007 ceph-mon[39249]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-18T08:18:01.980 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:18:01 smithi007 ceph-mon[39249]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-18T08:18:01.980 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:18:01 smithi007 ceph-mon[39249]: osdmap e389: 8 total, 8 up, 6 in 2024-09-18T08:18:02.070 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0 2024-09-18T08:18:02.092 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:18:02.093 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:18:02.093 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0' 2024-09-18T08:18:02.194 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1 2024-09-18T08:18:02.233 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:18:02.233 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:18:02.234 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1' 2024-09-18T08:18:02.335 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5 2024-09-18T08:18:02.360 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:18:02.360 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:18:02.361 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5' 2024-09-18T08:18:02.461 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2 2024-09-18T08:18:02.489 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:18:02.490 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:18:02.490 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2' 2024-09-18T08:18:02.591 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1 2024-09-18T08:18:02.637 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:18:02.638 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:18:02.638 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1' 2024-09-18T08:18:02.730 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:18:02 smithi007 ceph-mon[29459]: pgmap v761: 158 pgs: 158 active+clean; 932 MiB data, 6.1 GiB used, 530 GiB / 536 GiB avail; 679 B/s rd, 0 B/s wr, 999 op/s; 9.1 MiB/s, 145 objects/s recovering 2024-09-18T08:18:02.730 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:18:02 smithi007 ceph-mon[39249]: pgmap v761: 158 pgs: 158 active+clean; 932 MiB data, 6.1 GiB used, 530 GiB / 536 GiB avail; 679 B/s rd, 0 B/s wr, 999 op/s; 9.1 MiB/s, 145 objects/s recovering 2024-09-18T08:18:02.739 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4 2024-09-18T08:18:02.764 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:18:02.765 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:18:02.765 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4' 2024-09-18T08:18:02.866 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1 2024-09-18T08:18:02.905 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:18:02 smithi050 ceph-mon[32371]: pgmap v761: 158 pgs: 158 active+clean; 932 MiB data, 6.1 GiB used, 530 GiB / 536 GiB avail; 679 B/s rd, 0 B/s wr, 999 op/s; 9.1 MiB/s, 145 objects/s recovering 2024-09-18T08:18:02.907 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:18:02.908 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:18:02.908 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1' 2024-09-18T08:18:03.009 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6 2024-09-18T08:18:03.031 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:18:03.032 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:18:03.032 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6' 2024-09-18T08:18:03.133 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2 2024-09-18T08:18:03.173 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:18:03.173 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:18:03.174 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2' 2024-09-18T08:18:03.275 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3 2024-09-18T08:18:03.316 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:18:03.317 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:18:03.318 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3' 2024-09-18T08:18:03.418 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2 2024-09-18T08:18:03.461 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:18:03.461 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:18:03.462 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2' 2024-09-18T08:18:03.563 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3 2024-09-18T08:18:03.610 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:18:03.611 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:18:03.611 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3' 2024-09-18T08:18:03.712 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5 2024-09-18T08:18:03.736 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:18:03.736 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:18:03.737 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5' 2024-09-18T08:18:03.838 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4 2024-09-18T08:18:03.862 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:18:03.863 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:18:03.864 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4' 2024-09-18T08:18:03.964 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0 2024-09-18T08:18:03.986 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:18:03.987 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:18:03.988 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0' 2024-09-18T08:18:04.088 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4 2024-09-18T08:18:04.114 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:18:04.115 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:18:04.116 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4' 2024-09-18T08:18:04.216 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6 2024-09-18T08:18:04.241 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:18:04.241 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:18:04.242 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6' 2024-09-18T08:18:04.342 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0 2024-09-18T08:18:04.380 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:18:04.380 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:18:04.381 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0' 2024-09-18T08:18:04.481 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5 2024-09-18T08:18:04.508 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:18:04.509 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:18:04.510 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5' 2024-09-18T08:18:04.611 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4 2024-09-18T08:18:04.637 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:18:04.638 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:18:04.638 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4' 2024-09-18T08:18:04.739 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3 2024-09-18T08:18:04.770 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:18:04 smithi007 ceph-mon[39249]: pgmap v762: 158 pgs: 158 active+clean; 868 MiB data, 5.8 GiB used, 531 GiB / 536 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 812 op/s; 2.7 MiB/s, 43 objects/s recovering 2024-09-18T08:18:04.770 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:18:04 smithi007 ceph-mon[39249]: from='mgr.24470 172.21.15.7:0/2594390261' entity='mgr.y' 2024-09-18T08:18:04.770 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:18:04 smithi007 ceph-mon[39249]: from='mgr.24470 172.21.15.7:0/2594390261' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-18T08:18:04.770 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:18:04 smithi007 ceph-mon[39249]: osdmap e390: 8 total, 8 up, 6 in 2024-09-18T08:18:04.770 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:18:04 smithi007 ceph-mon[29459]: pgmap v762: 158 pgs: 158 active+clean; 868 MiB data, 5.8 GiB used, 531 GiB / 536 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 812 op/s; 2.7 MiB/s, 43 objects/s recovering 2024-09-18T08:18:04.770 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:18:04 smithi007 ceph-mon[29459]: from='mgr.24470 172.21.15.7:0/2594390261' entity='mgr.y' 2024-09-18T08:18:04.771 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:18:04 smithi007 ceph-mon[29459]: from='mgr.24470 172.21.15.7:0/2594390261' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-18T08:18:04.771 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:18:04 smithi007 ceph-mon[29459]: osdmap e390: 8 total, 8 up, 6 in 2024-09-18T08:18:04.772 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid 779349ca-7593-11ef-bceb-c7b262605968 -- ceph daemon osd.2 dump_ops_in_flight 2024-09-18T08:18:04.800 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:18:04.800 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:18:04.800 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3' 2024-09-18T08:18:04.901 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5 2024-09-18T08:18:04.905 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:18:04 smithi050 ceph-mon[32371]: pgmap v762: 158 pgs: 158 active+clean; 868 MiB data, 5.8 GiB used, 531 GiB / 536 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 812 op/s; 2.7 MiB/s, 43 objects/s recovering 2024-09-18T08:18:04.905 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:18:04 smithi050 ceph-mon[32371]: from='mgr.24470 172.21.15.7:0/2594390261' entity='mgr.y' 2024-09-18T08:18:04.905 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:18:04 smithi050 ceph-mon[32371]: from='mgr.24470 172.21.15.7:0/2594390261' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-18T08:18:04.905 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:18:04 smithi050 ceph-mon[32371]: osdmap e390: 8 total, 8 up, 6 in 2024-09-18T08:18:04.925 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:18:04.926 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:18:04.926 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5' 2024-09-18T08:18:05.027 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7 2024-09-18T08:18:05.051 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:18:05.051 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:18:05.052 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7' 2024-09-18T08:18:05.153 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4 2024-09-18T08:18:05.179 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:18:05.180 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:18:05.180 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4' 2024-09-18T08:18:05.281 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2 2024-09-18T08:18:05.333 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:18:05.334 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:18:05.334 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2' 2024-09-18T08:18:05.435 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0 2024-09-18T08:18:05.474 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:18:05.474 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:18:05.475 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0' 2024-09-18T08:18:05.541 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/779349ca-7593-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T08:18:05.576 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1 2024-09-18T08:18:05.604 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/779349ca-7593-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T08:18:05.660 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:18:05.661 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:18:05.661 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1' 2024-09-18T08:18:05.762 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4 2024-09-18T08:18:05.787 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:18:05.788 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:18:05.788 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4' 2024-09-18T08:18:05.889 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3 2024-09-18T08:18:05.935 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:18:05.935 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:18:05.936 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3' 2024-09-18T08:18:06.037 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5 2024-09-18T08:18:06.061 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:18:06.062 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:18:06.062 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5' 2024-09-18T08:18:06.163 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2 2024-09-18T08:18:06.201 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:18:06.202 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:18:06.202 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2' 2024-09-18T08:18:06.241 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/779349ca-7593-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T08:18:06.304 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7 2024-09-18T08:18:06.327 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:18:06.328 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:18:06.329 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7' 2024-09-18T08:18:06.429 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7 2024-09-18T08:18:06.457 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:18:06.458 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:18:06.458 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7' 2024-09-18T08:18:06.559 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5 2024-09-18T08:18:06.583 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:18:06.585 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:18:06.585 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5' 2024-09-18T08:18:06.629 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-09-18T08:18:06.632 INFO:teuthology.orchestra.run.smithi007.stderr:dumped all 2024-09-18T08:18:06.646 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:18:06 smithi007 ceph-mon[29459]: pgmap v764: 158 pgs: 158 active+clean; 604 MiB data, 5.1 GiB used, 531 GiB / 536 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.25k op/s; 3.1 MiB/s, 49 objects/s recovering 2024-09-18T08:18:06.646 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:18:06 smithi007 ceph-mon[39249]: pgmap v764: 158 pgs: 158 active+clean; 604 MiB data, 5.1 GiB used, 531 GiB / 536 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.25k op/s; 3.1 MiB/s, 49 objects/s recovering 2024-09-18T08:18:06.686 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0 2024-09-18T08:18:06.712 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:18:06.712 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:18:06.713 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0' 2024-09-18T08:18:06.814 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1 2024-09-18T08:18:06.852 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:18:06.852 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:18:06.853 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1' 2024-09-18T08:18:06.905 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:18:06 smithi050 ceph-mon[32371]: pgmap v764: 158 pgs: 158 active+clean; 604 MiB data, 5.1 GiB used, 531 GiB / 536 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.25k op/s; 3.1 MiB/s, 49 objects/s recovering 2024-09-18T08:18:06.954 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3 2024-09-18T08:18:06.980 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Sep 18 08:18:06 smithi007 ceph-779349ca-7593-11ef-bceb-c7b262605968-mgr-y[29688]: ::ffff:172.21.15.50 - - [18/Sep/2024:08:18:06] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-09-18T08:18:06.995 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:18:06.996 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:18:06.997 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3' 2024-09-18T08:18:07.098 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3 2024-09-18T08:18:07.122 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:18:07.123 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:18:07.123 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3' 2024-09-18T08:18:07.224 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0 2024-09-18T08:18:07.263 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:18:07.264 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:18:07.264 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0' 2024-09-18T08:18:07.365 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4 2024-09-18T08:18:07.387 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:18:07.387 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:18:07.388 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4' 2024-09-18T08:18:07.489 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2 2024-09-18T08:18:07.529 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:18:07.529 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:18:07.530 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2' 2024-09-18T08:18:07.551 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid 779349ca-7593-11ef-bceb-c7b262605968 -- ceph osd unset noscrub 2024-09-18T08:18:07.630 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4 2024-09-18T08:18:07.654 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:18:07.655 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:18:07.655 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4' 2024-09-18T08:18:07.756 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4 2024-09-18T08:18:07.779 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:18:07.780 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:18:07.780 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4' 2024-09-18T08:18:07.881 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3 2024-09-18T08:18:07.905 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:18:07 smithi050 ceph-mon[32371]: from='client.16182 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-18T08:18:07.905 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:18:07 smithi050 ceph-mon[32371]: purged_snaps scrub starts 2024-09-18T08:18:07.905 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:18:07 smithi050 ceph-mon[32371]: purged_snaps scrub ok 2024-09-18T08:18:07.927 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:18:07.928 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:18:07.929 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3' 2024-09-18T08:18:07.980 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:18:07 smithi007 ceph-mon[29459]: from='client.16182 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-18T08:18:07.980 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:18:07 smithi007 ceph-mon[29459]: purged_snaps scrub starts 2024-09-18T08:18:07.980 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:18:07 smithi007 ceph-mon[29459]: purged_snaps scrub ok 2024-09-18T08:18:07.980 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:18:07 smithi007 ceph-mon[39249]: from='client.16182 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-18T08:18:07.980 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:18:07 smithi007 ceph-mon[39249]: purged_snaps scrub starts 2024-09-18T08:18:07.980 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:18:07 smithi007 ceph-mon[39249]: purged_snaps scrub ok 2024-09-18T08:18:08.029 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1 2024-09-18T08:18:08.066 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:18:08.067 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:18:08.067 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1' 2024-09-18T08:18:08.168 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5 2024-09-18T08:18:08.192 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:18:08.193 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:18:08.193 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5' 2024-09-18T08:18:08.294 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3 2024-09-18T08:18:08.338 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:18:08.339 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:18:08.339 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3' 2024-09-18T08:18:08.440 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0 2024-09-18T08:18:08.477 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:18:08.477 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:18:08.478 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0' 2024-09-18T08:18:08.578 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0 2024-09-18T08:18:08.618 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:18:08.618 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:18:08.619 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0' 2024-09-18T08:18:08.720 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2 2024-09-18T08:18:08.763 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:18:08.764 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:18:08.764 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2' 2024-09-18T08:18:08.865 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2 2024-09-18T08:18:08.905 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:18:08.906 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:18:08 smithi050 ceph-mon[32371]: pgmap v765: 158 pgs: 158 active+clean; 604 MiB data, 5.1 GiB used, 531 GiB / 536 GiB avail; 639 B/s rd, 0 B/s wr, 901 op/s 2024-09-18T08:18:08.906 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:18:08 smithi050 ceph-mon[32371]: osdmap e391: 8 total, 8 up, 6 in 2024-09-18T08:18:08.906 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:18:08.907 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2' 2024-09-18T08:18:08.920 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:18:08 smithi007 ceph-mon[29459]: pgmap v765: 158 pgs: 158 active+clean; 604 MiB data, 5.1 GiB used, 531 GiB / 536 GiB avail; 639 B/s rd, 0 B/s wr, 901 op/s 2024-09-18T08:18:08.920 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:18:08 smithi007 ceph-mon[29459]: osdmap e391: 8 total, 8 up, 6 in 2024-09-18T08:18:08.920 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:18:08 smithi007 ceph-mon[39249]: pgmap v765: 158 pgs: 158 active+clean; 604 MiB data, 5.1 GiB used, 531 GiB / 536 GiB avail; 639 B/s rd, 0 B/s wr, 901 op/s 2024-09-18T08:18:08.920 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:18:08 smithi007 ceph-mon[39249]: osdmap e391: 8 total, 8 up, 6 in 2024-09-18T08:18:09.008 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2 2024-09-18T08:18:09.047 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:18:09.048 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:18:09.048 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2' 2024-09-18T08:18:09.149 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0 2024-09-18T08:18:09.188 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:18:09.188 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:18:09.189 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0' 2024-09-18T08:18:09.290 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3 2024-09-18T08:18:09.327 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:18:09.328 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:18:09.328 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3' 2024-09-18T08:18:09.339 INFO:teuthology.orchestra.run.smithi007.stdout:osd.0: {} 2024-09-18T08:18:09.340 INFO:teuthology.orchestra.run.smithi007.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-18T08:18:09.377 INFO:teuthology.orchestra.run.smithi007.stdout:osd.1: {} 2024-09-18T08:18:09.377 INFO:teuthology.orchestra.run.smithi007.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-18T08:18:09.413 INFO:teuthology.orchestra.run.smithi007.stdout:osd.2: {} 2024-09-18T08:18:09.413 INFO:teuthology.orchestra.run.smithi007.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-18T08:18:09.429 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3 2024-09-18T08:18:09.456 INFO:teuthology.orchestra.run.smithi007.stdout:osd.3: {} 2024-09-18T08:18:09.457 INFO:teuthology.orchestra.run.smithi007.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-18T08:18:09.469 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:18:09.470 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:18:09.470 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3' 2024-09-18T08:18:09.487 INFO:teuthology.orchestra.run.smithi007.stdout:osd.4: {} 2024-09-18T08:18:09.488 INFO:teuthology.orchestra.run.smithi007.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-18T08:18:09.523 INFO:teuthology.orchestra.run.smithi007.stdout:osd.5: {} 2024-09-18T08:18:09.524 INFO:teuthology.orchestra.run.smithi007.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-18T08:18:09.560 INFO:teuthology.orchestra.run.smithi007.stdout:osd.6: {} 2024-09-18T08:18:09.560 INFO:teuthology.orchestra.run.smithi007.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-18T08:18:09.572 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3 2024-09-18T08:18:09.594 INFO:teuthology.orchestra.run.smithi007.stdout:osd.7: {} 2024-09-18T08:18:09.594 INFO:teuthology.orchestra.run.smithi007.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-18T08:18:09.611 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:18:09.612 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:18:09.613 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3' 2024-09-18T08:18:09.714 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0 2024-09-18T08:18:09.753 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:18:09.754 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:18:09.754 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0' 2024-09-18T08:18:09.831 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:18:09 smithi007 ceph-mon[29459]: from='client.? 172.21.15.7:0/103380881' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-18T08:18:09.831 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:18:09 smithi007 ceph-mon[29459]: osdmap e392: 8 total, 8 up, 6 in 2024-09-18T08:18:09.831 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:18:09 smithi007 ceph-mon[39249]: from='client.? 172.21.15.7:0/103380881' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-18T08:18:09.831 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:18:09 smithi007 ceph-mon[39249]: osdmap e392: 8 total, 8 up, 6 in 2024-09-18T08:18:09.855 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2 2024-09-18T08:18:09.888 INFO:teuthology.orchestra.run.smithi007.stdout:{"pg_ready":true,"pg_map":{"version":764,"stamp":"2024-09-18T08:18:05.443579+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":633412147,"num_objects":9856,"num_object_clones":0,"num_object_copies":29568,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9856,"num_whiteouts":0,"num_read":1494,"num_read_kb":1246,"num_write":194501,"num_write_kb":6509934,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":63293,"num_bytes_recovered":4116724085,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":63336,"ondisk_log_size":63336,"up":474,"acting":474,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":474,"num_osds":6,"num_per_pool_osds":6,"num_per_pool_omap_osds":6,"kb":562470912,"kb_used":5397980,"kb_used_data":1994356,"kb_used_omap":15,"kb_used_meta":3403440,"kb_avail":557072932,"statfs":{"total":575970213888,"available":570442682368,"internally_reserved":0,"allocated":2042220544,"data_stored":2039298590,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":16049,"internal_metadata":3485122895},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-655425536,"num_objects":-10001,"num_object_clones":0,"num_object_copies":-30003,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-408,"num_objects_unfound":0,"num_objects_dirty":-10001,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":10001,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":395,"num_bytes_recovered":25886720,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.004273"},"pg_stats":[{"pgid":"3.1f","version":"0'0","reported_seq":734,"reported_epoch":390,"state":"active+clean","last_fresh":"2024-09-18T08:18:04.438035+0000","last_change":"2024-09-18T08:15:41.451370+0000","last_active":"2024-09-18T08:18:04.438035+0000","last_peered":"2024-09-18T08:18:04.438035+0000","last_clean":"2024-09-18T08:18:04.438035+0000","last_became_active":"2024-09-18T08:15:41.451012+0000","last_became_peered":"2024-09-18T08:15:41.451012+0000","last_unstale":"2024-09-18T08:18:04.438035+0000","last_undegraded":"2024-09-18T08:18:04.438035+0000","last_fullsized":"2024-09-18T08:18:04.438035+0000","mapping_epoch":336,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":337,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:14:29.060206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:46.860092+0000","last_clean_scrub_stamp":"2024-09-18T08:14:29.060206+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"271'15","reported_seq":465,"reported_epoch":389,"state":"active+clean","last_fresh":"2024-09-18T08:18:00.536526+0000","last_change":"2024-09-18T08:16:13.499531+0000","last_active":"2024-09-18T08:18:00.536526+0000","last_peered":"2024-09-18T08:18:00.536526+0000","last_clean":"2024-09-18T08:18:00.536526+0000","last_became_active":"2024-09-18T08:16:13.456332+0000","last_became_peered":"2024-09-18T08:16:13.456332+0000","last_unstale":"2024-09-18T08:18:00.536526+0000","last_undegraded":"2024-09-18T08:18:00.536526+0000","last_fullsized":"2024-09-18T08:18:00.536526+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"271'15","last_scrub_stamp":"2024-09-18T08:15:04.676980+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_clean_scrub_stamp":"2024-09-18T08:15:04.676980+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":25,"num_read_kb":19,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,3],"acting":[2,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":549,"reported_epoch":389,"state":"active+clean","last_fresh":"2024-09-18T08:18:00.525417+0000","last_change":"2024-09-18T08:15:41.440121+0000","last_active":"2024-09-18T08:18:00.525417+0000","last_peered":"2024-09-18T08:18:00.525417+0000","last_clean":"2024-09-18T08:18:00.525417+0000","last_became_active":"2024-09-18T08:15:41.438512+0000","last_became_peered":"2024-09-18T08:15:41.438512+0000","last_unstale":"2024-09-18T08:18:00.525417+0000","last_undegraded":"2024-09-18T08:18:00.525417+0000","last_fullsized":"2024-09-18T08:18:00.525417+0000","mapping_epoch":336,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":337,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:15:12.171225+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:50.886830+0000","last_clean_scrub_stamp":"2024-09-18T08:15:12.171225+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,2],"acting":[1,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":448,"reported_epoch":389,"state":"active+clean","last_fresh":"2024-09-18T08:18:00.533655+0000","last_change":"2024-09-18T08:17:40.565146+0000","last_active":"2024-09-18T08:18:00.533655+0000","last_peered":"2024-09-18T08:18:00.533655+0000","last_clean":"2024-09-18T08:18:00.533655+0000","last_became_active":"2024-09-18T08:16:13.488969+0000","last_became_peered":"2024-09-18T08:16:13.488969+0000","last_unstale":"2024-09-18T08:18:00.533655+0000","last_undegraded":"2024-09-18T08:18:00.533655+0000","last_fullsized":"2024-09-18T08:18:00.533655+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:17:40.565100+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_clean_scrub_stamp":"2024-09-18T08:17:40.565100+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:19:01.428914+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":726,"reported_epoch":389,"state":"active+clean","last_fresh":"2024-09-18T08:18:00.542513+0000","last_change":"2024-09-18T08:15:51.826092+0000","last_active":"2024-09-18T08:18:00.542513+0000","last_peered":"2024-09-18T08:18:00.542513+0000","last_clean":"2024-09-18T08:18:00.542513+0000","last_became_active":"2024-09-18T08:14:40.357298+0000","last_became_peered":"2024-09-18T08:14:40.357298+0000","last_unstale":"2024-09-18T08:18:00.542513+0000","last_undegraded":"2024-09-18T08:18:00.542513+0000","last_fullsized":"2024-09-18T08:18:00.542513+0000","mapping_epoch":308,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":309,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:15:51.826050+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_clean_scrub_stamp":"2024-09-18T08:15:51.826050+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,1],"acting":[3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":735,"reported_epoch":389,"state":"active+clean","last_fresh":"2024-09-18T08:18:00.543112+0000","last_change":"2024-09-18T08:17:38.359429+0000","last_active":"2024-09-18T08:18:00.543112+0000","last_peered":"2024-09-18T08:18:00.543112+0000","last_clean":"2024-09-18T08:18:00.543112+0000","last_became_active":"2024-09-18T08:06:24.126194+0000","last_became_peered":"2024-09-18T08:06:24.126194+0000","last_unstale":"2024-09-18T08:18:00.543112+0000","last_undegraded":"2024-09-18T08:18:00.543112+0000","last_fullsized":"2024-09-18T08:18:00.543112+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:17:38.359366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:46.860092+0000","last_clean_scrub_stamp":"2024-09-18T08:17:38.359366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:19:07.133806+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"271'27","reported_seq":727,"reported_epoch":389,"state":"active+clean","last_fresh":"2024-09-18T08:18:00.536222+0000","last_change":"2024-09-18T08:14:26.565324+0000","last_active":"2024-09-18T08:18:00.536222+0000","last_peered":"2024-09-18T08:18:00.536222+0000","last_clean":"2024-09-18T08:18:00.536222+0000","last_became_active":"2024-09-18T08:02:49.886106+0000","last_became_peered":"2024-09-18T08:02:49.886106+0000","last_unstale":"2024-09-18T08:18:00.536222+0000","last_undegraded":"2024-09-18T08:18:00.536222+0000","last_fullsized":"2024-09-18T08:18:00.536222+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"271'27","last_scrub_stamp":"2024-09-18T08:14:26.565071+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_clean_scrub_stamp":"2024-09-18T08:14:26.565071+0000","objects_scrubbed":7,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:19:00.445844+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":44,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":360,"reported_epoch":389,"state":"active+clean","last_fresh":"2024-09-18T08:18:00.524954+0000","last_change":"2024-09-18T08:16:13.483763+0000","last_active":"2024-09-18T08:18:00.524954+0000","last_peered":"2024-09-18T08:18:00.524954+0000","last_clean":"2024-09-18T08:18:00.524954+0000","last_became_active":"2024-09-18T08:16:13.474963+0000","last_became_peered":"2024-09-18T08:16:13.474963+0000","last_unstale":"2024-09-18T08:18:00.524954+0000","last_undegraded":"2024-09-18T08:18:00.524954+0000","last_fullsized":"2024-09-18T08:18:00.524954+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:15:44.585123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:50.886830+0000","last_clean_scrub_stamp":"2024-09-18T08:15:44.585123+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":123,"reported_epoch":389,"state":"active+clean","last_fresh":"2024-09-18T08:18:00.535121+0000","last_change":"2024-09-18T08:16:13.459018+0000","last_active":"2024-09-18T08:18:00.535121+0000","last_peered":"2024-09-18T08:18:00.535121+0000","last_clean":"2024-09-18T08:18:00.535121+0000","last_became_active":"2024-09-18T08:16:13.458654+0000","last_became_peered":"2024-09-18T08:16:13.458654+0000","last_unstale":"2024-09-18T08:18:00.535121+0000","last_undegraded":"2024-09-18T08:18:00.535121+0000","last_fullsized":"2024-09-18T08:18:00.535121+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:10:51.225974+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:46.860092+0000","last_clean_scrub_stamp":"2024-09-18T08:10:51.225974+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:18:13.341070+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,1],"acting":[5,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"271'15","reported_seq":327,"reported_epoch":389,"state":"active+clean","last_fresh":"2024-09-18T08:18:00.542676+0000","last_change":"2024-09-18T08:16:13.499713+0000","last_active":"2024-09-18T08:18:00.542676+0000","last_peered":"2024-09-18T08:18:00.542676+0000","last_clean":"2024-09-18T08:18:00.542676+0000","last_became_active":"2024-09-18T08:16:13.463397+0000","last_became_peered":"2024-09-18T08:16:13.463397+0000","last_unstale":"2024-09-18T08:18:00.542676+0000","last_undegraded":"2024-09-18T08:18:00.542676+0000","last_fullsized":"2024-09-18T08:18:00.542676+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'9","last_scrub_stamp":"2024-09-18T08:12:00.466202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_clean_scrub_stamp":"2024-09-18T08:12:00.466202+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:18:24.407823+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":24,"num_read_kb":18,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":131,"reported_epoch":389,"state":"active+clean","last_fresh":"2024-09-18T08:18:00.534438+0000","last_change":"2024-09-18T08:17:30.548045+0000","last_active":"2024-09-18T08:18:00.534438+0000","last_peered":"2024-09-18T08:18:00.534438+0000","last_clean":"2024-09-18T08:18:00.534438+0000","last_became_active":"2024-09-18T08:16:13.460761+0000","last_became_peered":"2024-09-18T08:16:13.460761+0000","last_unstale":"2024-09-18T08:18:00.534438+0000","last_undegraded":"2024-09-18T08:18:00.534438+0000","last_fullsized":"2024-09-18T08:18:00.534438+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:17:30.547979+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:50.886830+0000","last_clean_scrub_stamp":"2024-09-18T08:17:30.547979+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:18:36.571323+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,6],"acting":[5,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":702,"reported_epoch":390,"state":"active+clean","last_fresh":"2024-09-18T08:18:04.439783+0000","last_change":"2024-09-18T08:17:29.542358+0000","last_active":"2024-09-18T08:18:04.439783+0000","last_peered":"2024-09-18T08:18:04.439783+0000","last_clean":"2024-09-18T08:18:04.439783+0000","last_became_active":"2024-09-18T08:14:40.354651+0000","last_became_peered":"2024-09-18T08:14:40.354651+0000","last_unstale":"2024-09-18T08:18:04.439783+0000","last_undegraded":"2024-09-18T08:18:04.439783+0000","last_fullsized":"2024-09-18T08:18:04.439783+0000","mapping_epoch":308,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":309,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:17:29.542288+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_clean_scrub_stamp":"2024-09-18T08:17:29.542288+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:18:42.683374+0000","scrub_duration":18,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":123,"reported_epoch":389,"state":"active+clean","last_fresh":"2024-09-18T08:18:00.534363+0000","last_change":"2024-09-18T08:16:13.471332+0000","last_active":"2024-09-18T08:18:00.534363+0000","last_peered":"2024-09-18T08:18:00.534363+0000","last_clean":"2024-09-18T08:18:00.534363+0000","last_became_active":"2024-09-18T08:16:13.471003+0000","last_became_peered":"2024-09-18T08:16:13.471003+0000","last_unstale":"2024-09-18T08:18:00.534363+0000","last_undegraded":"2024-09-18T08:18:00.534363+0000","last_fullsized":"2024-09-18T08:18:00.534363+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:15:01.636504+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:46.860092+0000","last_clean_scrub_stamp":"2024-09-18T08:15:01.636504+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1b","version":"271'7","reported_seq":309,"reported_epoch":389,"state":"active+clean","last_fresh":"2024-09-18T08:18:00.544000+0000","last_change":"2024-09-18T08:16:13.514798+0000","last_active":"2024-09-18T08:18:00.544000+0000","last_peered":"2024-09-18T08:18:00.544000+0000","last_clean":"2024-09-18T08:18:00.544000+0000","last_became_active":"2024-09-18T08:16:13.473146+0000","last_became_peered":"2024-09-18T08:16:13.473146+0000","last_unstale":"2024-09-18T08:18:00.544000+0000","last_undegraded":"2024-09-18T08:18:00.544000+0000","last_fullsized":"2024-09-18T08:18:00.544000+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"271'7","last_scrub_stamp":"2024-09-18T08:15:13.551351+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_clean_scrub_stamp":"2024-09-18T08:15:13.551351+0000","objects_scrubbed":2,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":12,"num_read_kb":8,"num_write":8,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":564,"reported_epoch":389,"state":"active+clean","last_fresh":"2024-09-18T08:18:00.525859+0000","last_change":"2024-09-18T08:16:13.487519+0000","last_active":"2024-09-18T08:18:00.525859+0000","last_peered":"2024-09-18T08:18:00.525859+0000","last_clean":"2024-09-18T08:18:00.525859+0000","last_became_active":"2024-09-18T08:16:13.486136+0000","last_became_peered":"2024-09-18T08:16:13.486136+0000","last_unstale":"2024-09-18T08:18:00.525859+0000","last_undegraded":"2024-09-18T08:18:00.525859+0000","last_fullsized":"2024-09-18T08:18:00.525859+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:15:40.589048+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:50.886830+0000","last_clean_scrub_stamp":"2024-09-18T08:15:40.589048+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,6],"acting":[1,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":116,"reported_epoch":389,"state":"active+clean","last_fresh":"2024-09-18T08:18:00.533571+0000","last_change":"2024-09-18T08:15:41.444316+0000","last_active":"2024-09-18T08:18:00.533571+0000","last_peered":"2024-09-18T08:18:00.533571+0000","last_clean":"2024-09-18T08:18:00.533571+0000","last_became_active":"2024-09-18T08:15:41.441932+0000","last_became_peered":"2024-09-18T08:15:41.441932+0000","last_unstale":"2024-09-18T08:18:00.533571+0000","last_undegraded":"2024-09-18T08:18:00.533571+0000","last_fullsized":"2024-09-18T08:18:00.533571+0000","mapping_epoch":336,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":337,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:11:01.108648+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_clean_scrub_stamp":"2024-09-18T08:11:01.108648+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:18:18.451529+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":430,"reported_epoch":389,"state":"active+clean","last_fresh":"2024-09-18T08:18:00.536048+0000","last_change":"2024-09-18T08:16:13.453737+0000","last_active":"2024-09-18T08:18:00.536048+0000","last_peered":"2024-09-18T08:18:00.536048+0000","last_clean":"2024-09-18T08:18:00.536048+0000","last_became_active":"2024-09-18T08:16:13.453129+0000","last_became_peered":"2024-09-18T08:16:13.453129+0000","last_unstale":"2024-09-18T08:18:00.536048+0000","last_undegraded":"2024-09-18T08:18:00.536048+0000","last_fullsized":"2024-09-18T08:18:00.536048+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:15:42.600840+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_clean_scrub_stamp":"2024-09-18T08:15:42.600840+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,3],"acting":[5,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":730,"reported_epoch":390,"state":"active+clean","last_fresh":"2024-09-18T08:18:04.436444+0000","last_change":"2024-09-18T08:16:13.477657+0000","last_active":"2024-09-18T08:18:04.436444+0000","last_peered":"2024-09-18T08:18:04.436444+0000","last_clean":"2024-09-18T08:18:04.436444+0000","last_became_active":"2024-09-18T08:16:13.477401+0000","last_became_peered":"2024-09-18T08:16:13.477401+0000","last_unstale":"2024-09-18T08:18:04.436444+0000","last_undegraded":"2024-09-18T08:18:04.436444+0000","last_fullsized":"2024-09-18T08:18:04.436444+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:12:40.711507+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:46.860092+0000","last_clean_scrub_stamp":"2024-09-18T08:12:40.711507+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:18:59.784008+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,3],"acting":[0,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1c","version":"271'27","reported_seq":463,"reported_epoch":389,"state":"active+clean","last_fresh":"2024-09-18T08:18:00.537976+0000","last_change":"2024-09-18T08:14:40.711299+0000","last_active":"2024-09-18T08:18:00.537976+0000","last_peered":"2024-09-18T08:18:00.537976+0000","last_clean":"2024-09-18T08:18:00.537976+0000","last_became_active":"2024-09-18T08:14:39.480913+0000","last_became_peered":"2024-09-18T08:14:39.480913+0000","last_unstale":"2024-09-18T08:18:00.537976+0000","last_undegraded":"2024-09-18T08:18:00.537976+0000","last_fullsized":"2024-09-18T08:18:00.537976+0000","mapping_epoch":307,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":308,"parent":"0.0","parent_split_bits":0,"last_scrub":"271'27","last_scrub_stamp":"2024-09-18T08:14:40.711228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_clean_scrub_stamp":"2024-09-18T08:14:40.711228+0000","objects_scrubbed":7,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:18:52.983685+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":44,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":585,"reported_epoch":389,"state":"active+clean","last_fresh":"2024-09-18T08:18:00.522842+0000","last_change":"2024-09-18T08:17:36.633467+0000","last_active":"2024-09-18T08:18:00.522842+0000","last_peered":"2024-09-18T08:18:00.522842+0000","last_clean":"2024-09-18T08:18:00.522842+0000","last_became_active":"2024-09-18T08:16:13.462303+0000","last_became_peered":"2024-09-18T08:16:13.462303+0000","last_unstale":"2024-09-18T08:18:00.522842+0000","last_undegraded":"2024-09-18T08:18:00.522842+0000","last_fullsized":"2024-09-18T08:18:00.522842+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:17:36.633392+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:50.886830+0000","last_clean_scrub_stamp":"2024-09-18T08:17:36.633392+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:18:36.654140+0000","scrub_duration":20,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,5],"acting":[1,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":713,"reported_epoch":389,"state":"active+clean","last_fresh":"2024-09-18T08:18:00.543160+0000","last_change":"2024-09-18T08:15:41.469278+0000","last_active":"2024-09-18T08:18:00.543160+0000","last_peered":"2024-09-18T08:18:00.543160+0000","last_clean":"2024-09-18T08:18:00.543160+0000","last_became_active":"2024-09-18T08:15:41.466371+0000","last_became_peered":"2024-09-18T08:15:41.466371+0000","last_unstale":"2024-09-18T08:18:00.543160+0000","last_undegraded":"2024-09-18T08:18:00.543160+0000","last_fullsized":"2024-09-18T08:18:00.543160+0000","mapping_epoch":336,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":337,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:15:05.719269+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_clean_scrub_stamp":"2024-09-18T08:15:05.719269+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":360,"reported_epoch":389,"state":"active+clean","last_fresh":"2024-09-18T08:18:00.526023+0000","last_change":"2024-09-18T08:16:13.487949+0000","last_active":"2024-09-18T08:18:00.526023+0000","last_peered":"2024-09-18T08:18:00.526023+0000","last_clean":"2024-09-18T08:18:00.526023+0000","last_became_active":"2024-09-18T08:16:13.487001+0000","last_became_peered":"2024-09-18T08:16:13.487001+0000","last_unstale":"2024-09-18T08:18:00.526023+0000","last_undegraded":"2024-09-18T08:18:00.526023+0000","last_fullsized":"2024-09-18T08:18:00.526023+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:14:40.774086+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:46.860092+0000","last_clean_scrub_stamp":"2024-09-18T08:14:40.774086+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1d","version":"271'18","reported_seq":817,"reported_epoch":389,"state":"active+clean","last_fresh":"2024-09-18T08:18:00.547034+0000","last_change":"2024-09-18T08:17:04.627991+0000","last_active":"2024-09-18T08:18:00.547034+0000","last_peered":"2024-09-18T08:18:00.547034+0000","last_clean":"2024-09-18T08:18:00.547034+0000","last_became_active":"2024-09-18T08:16:13.473248+0000","last_became_peered":"2024-09-18T08:16:13.473248+0000","last_unstale":"2024-09-18T08:18:00.547034+0000","last_undegraded":"2024-09-18T08:18:00.547034+0000","last_fullsized":"2024-09-18T08:18:00.547034+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"271'18","last_scrub_stamp":"2024-09-18T08:17:04.627913+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_clean_scrub_stamp":"2024-09-18T08:17:04.627913+0000","objects_scrubbed":5,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:18:07.331175+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":34,"num_read_kb":25,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":660,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":121,"reported_epoch":389,"state":"active+clean","last_fresh":"2024-09-18T08:18:00.544519+0000","last_change":"2024-09-18T08:16:13.485702+0000","last_active":"2024-09-18T08:18:00.544519+0000","last_peered":"2024-09-18T08:18:00.544519+0000","last_clean":"2024-09-18T08:18:00.544519+0000","last_became_active":"2024-09-18T08:16:13.485316+0000","last_became_peered":"2024-09-18T08:16:13.485316+0000","last_unstale":"2024-09-18T08:18:00.544519+0000","last_undegraded":"2024-09-18T08:18:00.544519+0000","last_fullsized":"2024-09-18T08:18:00.544519+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:10:46.303848+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:50.886830+0000","last_clean_scrub_stamp":"2024-09-18T08:10:46.303848+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:18:16.412467+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,1],"acting":[3,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.1c","version":"69'1","reported_seq":120,"reported_epoch":389,"state":"active+clean","last_fresh":"2024-09-18T08:18:00.534457+0000","last_change":"2024-09-18T08:15:41.489871+0000","last_active":"2024-09-18T08:18:00.534457+0000","last_peered":"2024-09-18T08:18:00.534457+0000","last_clean":"2024-09-18T08:18:00.534457+0000","last_became_active":"2024-09-18T08:15:41.466665+0000","last_became_peered":"2024-09-18T08:15:41.466665+0000","last_unstale":"2024-09-18T08:18:00.534457+0000","last_undegraded":"2024-09-18T08:18:00.534457+0000","last_fullsized":"2024-09-18T08:18:00.534457+0000","mapping_epoch":336,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":337,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'1","last_scrub_stamp":"2024-09-18T08:15:06.708980+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_clean_scrub_stamp":"2024-09-18T08:15:06.708980+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":403,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":806,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,3],"acting":[5,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"62'1","reported_seq":598,"reported_epoch":389,"state":"active+clean","last_fresh":"2024-09-18T08:18:00.525835+0000","last_change":"2024-09-18T08:17:37.618000+0000","last_active":"2024-09-18T08:18:00.525835+0000","last_peered":"2024-09-18T08:18:00.525835+0000","last_clean":"2024-09-18T08:18:00.525835+0000","last_became_active":"2024-09-18T08:16:13.483397+0000","last_became_peered":"2024-09-18T08:16:13.483397+0000","last_unstale":"2024-09-18T08:18:00.525835+0000","last_undegraded":"2024-09-18T08:18:00.525835+0000","last_fullsized":"2024-09-18T08:18:00.525835+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"62'1","last_scrub_stamp":"2024-09-18T08:17:37.617954+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:46.860092+0000","last_clean_scrub_stamp":"2024-09-18T08:17:37.617954+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:18:50.504684+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":46,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,6],"acting":[1,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1e","version":"271'20","reported_seq":748,"reported_epoch":390,"state":"active+clean","last_fresh":"2024-09-18T08:18:04.438492+0000","last_change":"2024-09-18T08:15:40.251794+0000","last_active":"2024-09-18T08:18:04.438492+0000","last_peered":"2024-09-18T08:18:04.438492+0000","last_clean":"2024-09-18T08:18:04.438492+0000","last_became_active":"2024-09-18T08:14:40.359879+0000","last_became_peered":"2024-09-18T08:14:40.359879+0000","last_unstale":"2024-09-18T08:18:04.438492+0000","last_undegraded":"2024-09-18T08:18:04.438492+0000","last_fullsized":"2024-09-18T08:18:04.438492+0000","mapping_epoch":308,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":309,"parent":"0.0","parent_split_bits":0,"last_scrub":"271'20","last_scrub_stamp":"2024-09-18T08:15:40.251706+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_clean_scrub_stamp":"2024-09-18T08:15:40.251706+0000","objects_scrubbed":5,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":25,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1],"acting":[0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1f","version":"349'15","reported_seq":443,"reported_epoch":389,"state":"active+clean","last_fresh":"2024-09-18T08:18:00.539292+0000","last_change":"2024-09-18T08:17:29.878587+0000","last_active":"2024-09-18T08:18:00.539292+0000","last_peered":"2024-09-18T08:18:00.539292+0000","last_clean":"2024-09-18T08:18:00.539292+0000","last_became_active":"2024-09-18T08:16:13.467898+0000","last_became_peered":"2024-09-18T08:16:13.467898+0000","last_unstale":"2024-09-18T08:18:00.539292+0000","last_undegraded":"2024-09-18T08:18:00.539292+0000","last_fullsized":"2024-09-18T08:18:00.539292+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"349'15","last_scrub_stamp":"2024-09-18T08:17:29.878376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:50.886830+0000","last_clean_scrub_stamp":"2024-09-18T08:17:29.878376+0000","objects_scrubbed":1,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:18:46.725267+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,1],"acting":[2,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.1","version":"390'12464","reported_seq":20991,"reported_epoch":390,"state":"active+clean","last_fresh":"2024-09-18T08:18:05.054053+0000","last_change":"2024-09-18T08:17:25.434104+0000","last_active":"2024-09-18T08:18:05.054053+0000","last_peered":"2024-09-18T08:18:05.054053+0000","last_clean":"2024-09-18T08:18:05.054053+0000","last_became_active":"2024-09-18T08:17:25.417680+0000","last_became_peered":"2024-09-18T08:17:25.417680+0000","last_unstale":"2024-09-18T08:18:05.054053+0000","last_undegraded":"2024-09-18T08:18:05.054053+0000","last_fullsized":"2024-09-18T08:18:05.054053+0000","mapping_epoch":376,"log_start":"371'8711","ondisk_log_start":"371'8711","created":311,"last_epoch_clean":377,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:14:43.412748+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:14:43.412748+0000","last_clean_scrub_stamp":"2024-09-18T08:14:43.412748+0000","objects_scrubbed":0,"log_size":3753,"log_dups_size":0,"ondisk_log_size":3753,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33947648,"num_objects":518,"num_object_clones":0,"num_object_copies":1554,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":518,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12464,"num_write_kb":415424,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4169,"num_bytes_recovered":273219584,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.a","version":"74'79","reported_seq":125,"reported_epoch":389,"state":"active+clean","last_fresh":"2024-09-18T08:18:00.534030+0000","last_change":"2024-09-18T08:15:42.288426+0000","last_active":"2024-09-18T08:18:00.534030+0000","last_peered":"2024-09-18T08:18:00.534030+0000","last_clean":"2024-09-18T08:18:00.534030+0000","last_became_active":"2024-09-18T08:15:41.443376+0000","last_became_peered":"2024-09-18T08:15:41.443376+0000","last_unstale":"2024-09-18T08:18:00.534030+0000","last_undegraded":"2024-09-18T08:18:00.534030+0000","last_fullsized":"2024-09-18T08:18:00.534030+0000","mapping_epoch":336,"log_start":"0'0","ondisk_log_start":"0'0","created":152,"last_epoch_clean":337,"parent":"0.0","parent_split_bits":4,"last_scrub":"74'79","last_scrub_stamp":"2024-09-18T08:15:42.288363+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:00:11.893742+0000","last_clean_scrub_stamp":"2024-09-18T08:15:42.288363+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,6],"acting":[5,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.f","version":"271'27","reported_seq":654,"reported_epoch":389,"state":"active+clean","last_fresh":"2024-09-18T08:18:00.522888+0000","last_change":"2024-09-18T08:17:29.732577+0000","last_active":"2024-09-18T08:18:00.522888+0000","last_peered":"2024-09-18T08:18:00.522888+0000","last_clean":"2024-09-18T08:18:00.522888+0000","last_became_active":"2024-09-18T08:16:13.456369+0000","last_became_peered":"2024-09-18T08:16:13.456369+0000","last_unstale":"2024-09-18T08:18:00.522888+0000","last_undegraded":"2024-09-18T08:18:00.522888+0000","last_fullsized":"2024-09-18T08:18:00.522888+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"271'27","last_scrub_stamp":"2024-09-18T08:17:29.732496+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_clean_scrub_stamp":"2024-09-18T08:17:29.732496+0000","objects_scrubbed":7,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:18:58.561148+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":43,"num_read_kb":34,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,5],"acting":[1,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":720,"reported_epoch":389,"state":"active+clean","last_fresh":"2024-09-18T08:18:00.533368+0000","last_change":"2024-09-18T08:16:13.472791+0000","last_active":"2024-09-18T08:18:00.533368+0000","last_peered":"2024-09-18T08:18:00.533368+0000","last_clean":"2024-09-18T08:18:00.533368+0000","last_became_active":"2024-09-18T08:16:13.472353+0000","last_became_peered":"2024-09-18T08:16:13.472353+0000","last_unstale":"2024-09-18T08:18:00.533368+0000","last_undegraded":"2024-09-18T08:18:00.533368+0000","last_fullsized":"2024-09-18T08:18:00.533368+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:15:46.809403+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:46.860092+0000","last_clean_scrub_stamp":"2024-09-18T08:15:46.809403+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,5],"acting":[3,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.e","version":"349'16","reported_seq":586,"reported_epoch":389,"state":"active+clean","last_fresh":"2024-09-18T08:18:00.536189+0000","last_change":"2024-09-18T08:17:37.704743+0000","last_active":"2024-09-18T08:18:00.536189+0000","last_peered":"2024-09-18T08:18:00.536189+0000","last_clean":"2024-09-18T08:18:00.536189+0000","last_became_active":"2024-09-18T08:16:13.451199+0000","last_became_peered":"2024-09-18T08:16:13.451199+0000","last_unstale":"2024-09-18T08:18:00.536189+0000","last_undegraded":"2024-09-18T08:18:00.536189+0000","last_fullsized":"2024-09-18T08:18:00.536189+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"349'16","last_scrub_stamp":"2024-09-18T08:17:37.704683+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:50.886830+0000","last_clean_scrub_stamp":"2024-09-18T08:17:37.704683+0000","objects_scrubbed":1,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:18:50.732146+0000","scrub_duration":71,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,2],"acting":[5,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":117,"reported_epoch":389,"state":"active+clean","last_fresh":"2024-09-18T08:18:00.534705+0000","last_change":"2024-09-18T08:15:41.438999+0000","last_active":"2024-09-18T08:18:00.534705+0000","last_peered":"2024-09-18T08:18:00.534705+0000","last_clean":"2024-09-18T08:18:00.534705+0000","last_became_active":"2024-09-18T08:15:41.438610+0000","last_became_peered":"2024-09-18T08:15:41.438610+0000","last_unstale":"2024-09-18T08:18:00.534705+0000","last_undegraded":"2024-09-18T08:18:00.534705+0000","last_fullsized":"2024-09-18T08:18:00.534705+0000","mapping_epoch":336,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":337,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:15:00.180309+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_clean_scrub_stamp":"2024-09-18T08:15:00.180309+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:18:54.566550+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.e","version":"389'11906","reported_seq":14992,"reported_epoch":389,"state":"active+clean","last_fresh":"2024-09-18T08:18:04.104017+0000","last_change":"2024-09-18T08:17:53.954702+0000","last_active":"2024-09-18T08:18:04.104017+0000","last_peered":"2024-09-18T08:18:04.104017+0000","last_clean":"2024-09-18T08:18:04.104017+0000","last_became_active":"2024-09-18T08:17:53.917048+0000","last_became_peered":"2024-09-18T08:17:53.917048+0000","last_unstale":"2024-09-18T08:18:04.104017+0000","last_undegraded":"2024-09-18T08:18:04.104017+0000","last_fullsized":"2024-09-18T08:18:04.104017+0000","mapping_epoch":386,"log_start":"358'7200","ondisk_log_start":"358'7200","created":311,"last_epoch_clean":387,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:14:43.412748+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:14:43.412748+0000","last_clean_scrub_stamp":"2024-09-18T08:14:43.412748+0000","objects_scrubbed":0,"log_size":4706,"log_dups_size":0,"ondisk_log_size":4706,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:18:54.998788+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39190528,"num_objects":598,"num_object_clones":0,"num_object_copies":1794,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":598,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11906,"num_write_kb":400128,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1446,"num_bytes_recovered":94765056,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,5],"acting":[2,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.5","version":"74'79","reported_seq":144,"reported_epoch":389,"state":"active+clean","last_fresh":"2024-09-18T08:18:00.545522+0000","last_change":"2024-09-18T08:16:13.445824+0000","last_active":"2024-09-18T08:18:00.545522+0000","last_peered":"2024-09-18T08:18:00.545522+0000","last_clean":"2024-09-18T08:18:00.545522+0000","last_became_active":"2024-09-18T08:16:13.445467+0000","last_became_peered":"2024-09-18T08:16:13.445467+0000","last_unstale":"2024-09-18T08:18:00.545522+0000","last_undegraded":"2024-09-18T08:18:00.545522+0000","last_fullsized":"2024-09-18T08:18:00.545522+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":152,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":3,"last_scrub":"74'79","last_scrub_stamp":"2024-09-18T08:15:58.701823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:00:11.893742+0000","last_clean_scrub_stamp":"2024-09-18T08:15:58.701823+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,5],"acting":[2,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.0","version":"271'30","reported_seq":789,"reported_epoch":389,"state":"active+clean","last_fresh":"2024-09-18T08:18:00.533085+0000","last_change":"2024-09-18T08:15:57.927197+0000","last_active":"2024-09-18T08:18:00.533085+0000","last_peered":"2024-09-18T08:18:00.533085+0000","last_clean":"2024-09-18T08:18:00.533085+0000","last_became_active":"2024-09-18T08:14:40.358956+0000","last_became_peered":"2024-09-18T08:14:40.358956+0000","last_unstale":"2024-09-18T08:18:00.533085+0000","last_undegraded":"2024-09-18T08:18:00.533085+0000","last_fullsized":"2024-09-18T08:18:00.533085+0000","mapping_epoch":308,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":309,"parent":"0.0","parent_split_bits":0,"last_scrub":"271'30","last_scrub_stamp":"2024-09-18T08:15:57.927111+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_clean_scrub_stamp":"2024-09-18T08:15:57.927111+0000","objects_scrubbed":8,"log_size":30,"log_dups_size":0,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":18,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":52,"num_read_kb":40,"num_write":32,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":742,"reported_epoch":389,"state":"active+clean","last_fresh":"2024-09-18T08:18:00.543698+0000","last_change":"2024-09-18T08:15:41.461616+0000","last_active":"2024-09-18T08:18:00.543698+0000","last_peered":"2024-09-18T08:18:00.543698+0000","last_clean":"2024-09-18T08:18:00.543698+0000","last_became_active":"2024-09-18T08:15:41.461057+0000","last_became_peered":"2024-09-18T08:15:41.461057+0000","last_unstale":"2024-09-18T08:18:00.543698+0000","last_undegraded":"2024-09-18T08:18:00.543698+0000","last_fullsized":"2024-09-18T08:18:00.543698+0000","mapping_epoch":336,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":337,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:15:08.780913+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:46.860092+0000","last_clean_scrub_stamp":"2024-09-18T08:15:08.780913+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":350,"reported_epoch":389,"state":"active+clean","last_fresh":"2024-09-18T08:18:00.536766+0000","last_change":"2024-09-18T08:17:31.417637+0000","last_active":"2024-09-18T08:18:00.536766+0000","last_peered":"2024-09-18T08:18:00.536766+0000","last_clean":"2024-09-18T08:18:00.536766+0000","last_became_active":"2024-09-18T08:16:13.478687+0000","last_became_peered":"2024-09-18T08:16:13.478687+0000","last_unstale":"2024-09-18T08:18:00.536766+0000","last_undegraded":"2024-09-18T08:18:00.536766+0000","last_fullsized":"2024-09-18T08:18:00.536766+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:17:31.417453+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:50.886830+0000","last_clean_scrub_stamp":"2024-09-18T08:17:31.417453+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:18:55.886528+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":122,"reported_epoch":389,"state":"active+clean","last_fresh":"2024-09-18T08:18:00.542574+0000","last_change":"2024-09-18T08:16:13.461117+0000","last_active":"2024-09-18T08:18:00.542574+0000","last_peered":"2024-09-18T08:18:00.542574+0000","last_clean":"2024-09-18T08:18:00.542574+0000","last_became_active":"2024-09-18T08:16:13.460785+0000","last_became_peered":"2024-09-18T08:16:13.460785+0000","last_unstale":"2024-09-18T08:18:00.542574+0000","last_undegraded":"2024-09-18T08:18:00.542574+0000","last_fullsized":"2024-09-18T08:18:00.542574+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:12:02.473168+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_clean_scrub_stamp":"2024-09-18T08:12:02.473168+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:18:26.348422+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,6],"acting":[3,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.f","version":"389'12313","reported_seq":12481,"reported_epoch":389,"state":"active+clean","last_fresh":"2024-09-18T08:18:04.349456+0000","last_change":"2024-09-18T08:14:44.437378+0000","last_active":"2024-09-18T08:18:04.349456+0000","last_peered":"2024-09-18T08:18:04.349456+0000","last_clean":"2024-09-18T08:18:04.349456+0000","last_became_active":"2024-09-18T08:14:44.437103+0000","last_became_peered":"2024-09-18T08:14:44.437103+0000","last_unstale":"2024-09-18T08:18:04.349456+0000","last_undegraded":"2024-09-18T08:18:04.349456+0000","last_fullsized":"2024-09-18T08:18:04.349456+0000","mapping_epoch":311,"log_start":"371'8801","ondisk_log_start":"371'8801","created":311,"last_epoch_clean":312,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:14:43.412748+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:14:43.412748+0000","last_clean_scrub_stamp":"2024-09-18T08:14:43.412748+0000","objects_scrubbed":0,"log_size":3512,"log_dups_size":0,"ondisk_log_size":3512,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38600704,"num_objects":589,"num_object_clones":0,"num_object_copies":1767,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":589,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12313,"num_write_kb":412864,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.4","version":"74'79","reported_seq":565,"reported_epoch":389,"state":"active+clean","last_fresh":"2024-09-18T08:18:00.543716+0000","last_change":"2024-09-18T08:15:41.898345+0000","last_active":"2024-09-18T08:18:00.543716+0000","last_peered":"2024-09-18T08:18:00.543716+0000","last_clean":"2024-09-18T08:18:00.543716+0000","last_became_active":"2024-09-18T08:15:41.464987+0000","last_became_peered":"2024-09-18T08:15:41.464987+0000","last_unstale":"2024-09-18T08:18:00.543716+0000","last_undegraded":"2024-09-18T08:18:00.543716+0000","last_fullsized":"2024-09-18T08:18:00.543716+0000","mapping_epoch":336,"log_start":"0'0","ondisk_log_start":"0'0","created":152,"last_epoch_clean":337,"parent":"0.0","parent_split_bits":3,"last_scrub":"74'79","last_scrub_stamp":"2024-09-18T08:15:41.898294+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:00:11.893742+0000","last_clean_scrub_stamp":"2024-09-18T08:15:41.898294+0000","objects_scrubbed":1,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":327680,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":983040,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"271'28","reported_seq":499,"reported_epoch":389,"state":"active+clean","last_fresh":"2024-09-18T08:18:00.533527+0000","last_change":"2024-09-18T08:16:13.506782+0000","last_active":"2024-09-18T08:18:00.533527+0000","last_peered":"2024-09-18T08:18:00.533527+0000","last_clean":"2024-09-18T08:18:00.533527+0000","last_became_active":"2024-09-18T08:16:13.451205+0000","last_became_peered":"2024-09-18T08:16:13.451205+0000","last_unstale":"2024-09-18T08:18:00.533527+0000","last_undegraded":"2024-09-18T08:18:00.533527+0000","last_fullsized":"2024-09-18T08:18:00.533527+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"271'28","last_scrub_stamp":"2024-09-18T08:15:51.716119+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_clean_scrub_stamp":"2024-09-18T08:15:51.716119+0000","objects_scrubbed":7,"log_size":28,"log_dups_size":0,"ondisk_log_size":28,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":41,"num_read_kb":34,"num_write":28,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.6","version":"62'2","reported_seq":755,"reported_epoch":390,"state":"active+clean","last_fresh":"2024-09-18T08:18:04.438760+0000","last_change":"2024-09-18T08:16:13.508628+0000","last_active":"2024-09-18T08:18:04.438760+0000","last_peered":"2024-09-18T08:18:04.438760+0000","last_clean":"2024-09-18T08:18:04.438760+0000","last_became_active":"2024-09-18T08:16:13.451171+0000","last_became_peered":"2024-09-18T08:16:13.451171+0000","last_unstale":"2024-09-18T08:18:04.438760+0000","last_undegraded":"2024-09-18T08:18:04.438760+0000","last_fullsized":"2024-09-18T08:18:04.438760+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"62'2","last_scrub_stamp":"2024-09-18T08:14:42.145836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:46.860092+0000","last_clean_scrub_stamp":"2024-09-18T08:14:42.145836+0000","objects_scrubbed":2,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1039,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":3117,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,5],"acting":[0,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.0","version":"349'13","reported_seq":927,"reported_epoch":389,"state":"active+clean","last_fresh":"2024-09-18T08:18:00.532215+0000","last_change":"2024-09-18T08:16:13.516195+0000","last_active":"2024-09-18T08:18:00.532215+0000","last_peered":"2024-09-18T08:18:00.532215+0000","last_clean":"2024-09-18T08:18:00.532215+0000","last_became_active":"2024-09-18T08:16:13.477431+0000","last_became_peered":"2024-09-18T08:16:13.477431+0000","last_unstale":"2024-09-18T08:18:00.532215+0000","last_undegraded":"2024-09-18T08:18:00.532215+0000","last_fullsized":"2024-09-18T08:18:00.532215+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"310'12","last_scrub_stamp":"2024-09-18T08:15:49.795095+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:50.886830+0000","last_clean_scrub_stamp":"2024-09-18T08:15:49.795095+0000","objects_scrubbed":1,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":117,"reported_epoch":389,"state":"active+clean","last_fresh":"2024-09-18T08:18:00.534366+0000","last_change":"2024-09-18T08:15:41.441653+0000","last_active":"2024-09-18T08:18:00.534366+0000","last_peered":"2024-09-18T08:18:00.534366+0000","last_clean":"2024-09-18T08:18:00.534366+0000","last_became_active":"2024-09-18T08:15:41.440876+0000","last_became_peered":"2024-09-18T08:15:41.440876+0000","last_unstale":"2024-09-18T08:18:00.534366+0000","last_undegraded":"2024-09-18T08:18:00.534366+0000","last_fullsized":"2024-09-18T08:18:00.534366+0000","mapping_epoch":336,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":337,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:15:14.781554+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_clean_scrub_stamp":"2024-09-18T08:15:14.781554+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.c","version":"390'11997","reported_seq":23295,"reported_epoch":390,"state":"active+clean","last_fresh":"2024-09-18T08:18:05.058134+0000","last_change":"2024-09-18T08:16:52.555960+0000","last_active":"2024-09-18T08:18:05.058134+0000","last_peered":"2024-09-18T08:18:05.058134+0000","last_clean":"2024-09-18T08:18:05.058134+0000","last_became_active":"2024-09-18T08:16:52.532776+0000","last_became_peered":"2024-09-18T08:16:52.532776+0000","last_unstale":"2024-09-18T08:18:05.058134+0000","last_undegraded":"2024-09-18T08:18:05.058134+0000","last_fullsized":"2024-09-18T08:18:05.058134+0000","mapping_epoch":362,"log_start":"369'8248","ondisk_log_start":"369'8248","created":311,"last_epoch_clean":363,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:14:43.412748+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:14:43.412748+0000","last_clean_scrub_stamp":"2024-09-18T08:14:43.412748+0000","objects_scrubbed":0,"log_size":3749,"log_dups_size":0,"ondisk_log_size":3749,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34144256,"num_objects":521,"num_object_clones":0,"num_object_copies":1563,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":521,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11997,"num_write_kb":400576,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5557,"num_bytes_recovered":364183552,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,6],"acting":[0,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.7","version":"74'79","reported_seq":547,"reported_epoch":389,"state":"active+clean","last_fresh":"2024-09-18T08:18:00.543538+0000","last_change":"2024-09-18T08:15:44.872544+0000","last_active":"2024-09-18T08:18:00.543538+0000","last_peered":"2024-09-18T08:18:00.543538+0000","last_clean":"2024-09-18T08:18:00.543538+0000","last_became_active":"2024-09-18T08:15:41.466505+0000","last_became_peered":"2024-09-18T08:15:41.466505+0000","last_unstale":"2024-09-18T08:18:00.543538+0000","last_undegraded":"2024-09-18T08:18:00.543538+0000","last_fullsized":"2024-09-18T08:18:00.543538+0000","mapping_epoch":336,"log_start":"0'0","ondisk_log_start":"0'0","created":152,"last_epoch_clean":337,"parent":"0.0","parent_split_bits":3,"last_scrub":"74'79","last_scrub_stamp":"2024-09-18T08:15:44.872332+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:00:11.893742+0000","last_clean_scrub_stamp":"2024-09-18T08:15:44.872332+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.2","version":"271'20","reported_seq":709,"reported_epoch":389,"state":"active+clean","last_fresh":"2024-09-18T08:18:00.526106+0000","last_change":"2024-09-18T08:16:13.525620+0000","last_active":"2024-09-18T08:18:00.526106+0000","last_peered":"2024-09-18T08:18:00.526106+0000","last_clean":"2024-09-18T08:18:00.526106+0000","last_became_active":"2024-09-18T08:16:13.486720+0000","last_became_peered":"2024-09-18T08:16:13.486720+0000","last_unstale":"2024-09-18T08:18:00.526106+0000","last_undegraded":"2024-09-18T08:18:00.526106+0000","last_fullsized":"2024-09-18T08:18:00.526106+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"271'20","last_scrub_stamp":"2024-09-18T08:15:40.251008+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_clean_scrub_stamp":"2024-09-18T08:15:40.251008+0000","objects_scrubbed":5,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":23,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":28,"num_read_kb":23,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":117,"reported_epoch":389,"state":"active+clean","last_fresh":"2024-09-18T08:18:00.535208+0000","last_change":"2024-09-18T08:15:41.452981+0000","last_active":"2024-09-18T08:18:00.535208+0000","last_peered":"2024-09-18T08:18:00.535208+0000","last_clean":"2024-09-18T08:18:00.535208+0000","last_became_active":"2024-09-18T08:15:41.449639+0000","last_became_peered":"2024-09-18T08:15:41.449639+0000","last_unstale":"2024-09-18T08:18:00.535208+0000","last_undegraded":"2024-09-18T08:18:00.535208+0000","last_fullsized":"2024-09-18T08:18:00.535208+0000","mapping_epoch":336,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":337,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:15:39.858184+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:46.860092+0000","last_clean_scrub_stamp":"2024-09-18T08:15:39.858184+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.3","version":"338'14","reported_seq":929,"reported_epoch":390,"state":"active+clean","last_fresh":"2024-09-18T08:18:04.436161+0000","last_change":"2024-09-18T08:15:41.467411+0000","last_active":"2024-09-18T08:18:04.436161+0000","last_peered":"2024-09-18T08:18:04.436161+0000","last_clean":"2024-09-18T08:18:04.436161+0000","last_became_active":"2024-09-18T08:15:41.447667+0000","last_became_peered":"2024-09-18T08:15:41.447667+0000","last_unstale":"2024-09-18T08:18:04.436161+0000","last_undegraded":"2024-09-18T08:18:04.436161+0000","last_fullsized":"2024-09-18T08:18:04.436161+0000","mapping_epoch":336,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":337,"parent":"0.0","parent_split_bits":0,"last_scrub":"317'11","last_scrub_stamp":"2024-09-18T08:14:59.207667+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:50.886830+0000","last_clean_scrub_stamp":"2024-09-18T08:14:59.207667+0000","objects_scrubbed":1,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":717,"reported_epoch":390,"state":"active+clean","last_fresh":"2024-09-18T08:18:04.437504+0000","last_change":"2024-09-18T08:14:26.087160+0000","last_active":"2024-09-18T08:18:04.437504+0000","last_peered":"2024-09-18T08:18:04.437504+0000","last_clean":"2024-09-18T08:18:04.437504+0000","last_became_active":"2024-09-18T08:08:52.700080+0000","last_became_peered":"2024-09-18T08:08:52.700080+0000","last_unstale":"2024-09-18T08:18:04.437504+0000","last_undegraded":"2024-09-18T08:18:04.437504+0000","last_fullsized":"2024-09-18T08:18:04.437504+0000","mapping_epoch":198,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":199,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:14:26.087098+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_clean_scrub_stamp":"2024-09-18T08:14:26.087098+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.d","version":"390'11970","reported_seq":22290,"reported_epoch":390,"state":"active+clean","last_fresh":"2024-09-18T08:18:05.078089+0000","last_change":"2024-09-18T08:17:06.468945+0000","last_active":"2024-09-18T08:18:05.078089+0000","last_peered":"2024-09-18T08:18:05.078089+0000","last_clean":"2024-09-18T08:18:05.078089+0000","last_became_active":"2024-09-18T08:17:06.449951+0000","last_became_peered":"2024-09-18T08:17:06.449951+0000","last_unstale":"2024-09-18T08:18:05.078089+0000","last_undegraded":"2024-09-18T08:18:05.078089+0000","last_fullsized":"2024-09-18T08:18:05.078089+0000","mapping_epoch":370,"log_start":"369'8247","ondisk_log_start":"369'8247","created":311,"last_epoch_clean":371,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:14:43.412748+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:14:43.412748+0000","last_clean_scrub_stamp":"2024-09-18T08:14:43.412748+0000","objects_scrubbed":0,"log_size":3723,"log_dups_size":0,"ondisk_log_size":3723,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34799640,"num_objects":532,"num_object_clones":0,"num_object_copies":1596,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":532,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":11970,"num_write_kb":400001,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5068,"num_bytes_recovered":332070936,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,3],"acting":[0,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.6","version":"74'79","reported_seq":608,"reported_epoch":390,"state":"active+clean","last_fresh":"2024-09-18T08:18:04.435688+0000","last_change":"2024-09-18T08:14:02.837236+0000","last_active":"2024-09-18T08:18:04.435688+0000","last_peered":"2024-09-18T08:18:04.435688+0000","last_clean":"2024-09-18T08:18:04.435688+0000","last_became_active":"2024-09-18T08:07:26.426532+0000","last_became_peered":"2024-09-18T08:07:26.426532+0000","last_unstale":"2024-09-18T08:18:04.435688+0000","last_undegraded":"2024-09-18T08:18:04.435688+0000","last_fullsized":"2024-09-18T08:18:04.435688+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":152,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":3,"last_scrub":"74'79","last_scrub_stamp":"2024-09-18T08:14:02.837167+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:00:11.893742+0000","last_clean_scrub_stamp":"2024-09-18T08:14:02.837167+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,3],"acting":[0,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.3","version":"271'29","reported_seq":825,"reported_epoch":390,"state":"active+clean","last_fresh":"2024-09-18T08:18:04.436265+0000","last_change":"2024-09-18T08:15:41.493514+0000","last_active":"2024-09-18T08:18:04.436265+0000","last_peered":"2024-09-18T08:18:04.436265+0000","last_clean":"2024-09-18T08:18:04.436265+0000","last_became_active":"2024-09-18T08:15:41.451985+0000","last_became_peered":"2024-09-18T08:15:41.451985+0000","last_unstale":"2024-09-18T08:18:04.436265+0000","last_undegraded":"2024-09-18T08:18:04.436265+0000","last_fullsized":"2024-09-18T08:18:04.436265+0000","mapping_epoch":336,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":337,"parent":"0.0","parent_split_bits":0,"last_scrub":"271'29","last_scrub_stamp":"2024-09-18T08:13:33.814992+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_clean_scrub_stamp":"2024-09-18T08:13:33.814992+0000","objects_scrubbed":8,"log_size":29,"log_dups_size":0,"ondisk_log_size":29,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":54,"num_read_kb":40,"num_write":32,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":990,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.4","version":"62'1","reported_seq":582,"reported_epoch":389,"state":"active+clean","last_fresh":"2024-09-18T08:18:00.526923+0000","last_change":"2024-09-18T08:15:41.454692+0000","last_active":"2024-09-18T08:18:00.526923+0000","last_peered":"2024-09-18T08:18:00.526923+0000","last_clean":"2024-09-18T08:18:00.526923+0000","last_became_active":"2024-09-18T08:15:41.441822+0000","last_became_peered":"2024-09-18T08:15:41.441822+0000","last_unstale":"2024-09-18T08:18:00.526923+0000","last_undegraded":"2024-09-18T08:18:00.526923+0000","last_fullsized":"2024-09-18T08:18:00.526923+0000","mapping_epoch":336,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":337,"parent":"0.0","parent_split_bits":0,"last_scrub":"62'1","last_scrub_stamp":"2024-09-18T08:15:13.165826+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:46.860092+0000","last_clean_scrub_stamp":"2024-09-18T08:15:13.165826+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":17,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":436,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":1308,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":630,"reported_epoch":390,"state":"active+clean","last_fresh":"2024-09-18T08:18:04.436718+0000","last_change":"2024-09-18T08:15:41.450294+0000","last_active":"2024-09-18T08:18:04.436718+0000","last_peered":"2024-09-18T08:18:04.436718+0000","last_clean":"2024-09-18T08:18:04.436718+0000","last_became_active":"2024-09-18T08:15:41.449832+0000","last_became_peered":"2024-09-18T08:15:41.449832+0000","last_unstale":"2024-09-18T08:18:04.436718+0000","last_undegraded":"2024-09-18T08:18:04.436718+0000","last_fullsized":"2024-09-18T08:18:04.436718+0000","mapping_epoch":336,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":337,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:12:37.732260+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:50.886830+0000","last_clean_scrub_stamp":"2024-09-18T08:12:37.732260+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:18:56.820396+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":601,"reported_epoch":389,"state":"active+clean","last_fresh":"2024-09-18T08:18:00.523434+0000","last_change":"2024-09-18T08:15:50.294289+0000","last_active":"2024-09-18T08:18:00.523434+0000","last_peered":"2024-09-18T08:18:00.523434+0000","last_clean":"2024-09-18T08:18:00.523434+0000","last_became_active":"2024-09-18T08:14:39.358132+0000","last_became_peered":"2024-09-18T08:14:39.358132+0000","last_unstale":"2024-09-18T08:18:00.523434+0000","last_undegraded":"2024-09-18T08:18:00.523434+0000","last_fullsized":"2024-09-18T08:18:00.523434+0000","mapping_epoch":307,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":308,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:15:50.294213+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_clean_scrub_stamp":"2024-09-18T08:15:50.294213+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":28,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.a","version":"390'12152","reported_seq":24658,"reported_epoch":390,"state":"active+clean","last_fresh":"2024-09-18T08:18:05.083911+0000","last_change":"2024-09-18T08:16:21.405814+0000","last_active":"2024-09-18T08:18:05.083911+0000","last_peered":"2024-09-18T08:18:05.083911+0000","last_clean":"2024-09-18T08:18:05.083911+0000","last_became_active":"2024-09-18T08:16:21.396170+0000","last_became_peered":"2024-09-18T08:16:21.396170+0000","last_unstale":"2024-09-18T08:18:05.083911+0000","last_undegraded":"2024-09-18T08:18:05.083911+0000","last_fullsized":"2024-09-18T08:18:05.083911+0000","mapping_epoch":354,"log_start":"369'8392","ondisk_log_start":"369'8392","created":311,"last_epoch_clean":355,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:14:43.412748+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:14:43.412748+0000","last_clean_scrub_stamp":"2024-09-18T08:14:43.412748+0000","objects_scrubbed":0,"log_size":3760,"log_dups_size":0,"ondisk_log_size":3760,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35389440,"num_objects":540,"num_object_clones":0,"num_object_copies":1620,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":540,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12152,"num_write_kb":406144,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6152,"num_bytes_recovered":403177472,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"74'79","reported_seq":125,"reported_epoch":389,"state":"active+clean","last_fresh":"2024-09-18T08:18:00.533636+0000","last_change":"2024-09-18T08:15:43.275398+0000","last_active":"2024-09-18T08:18:00.533636+0000","last_peered":"2024-09-18T08:18:00.533636+0000","last_clean":"2024-09-18T08:18:00.533636+0000","last_became_active":"2024-09-18T08:15:41.452895+0000","last_became_peered":"2024-09-18T08:15:41.452895+0000","last_unstale":"2024-09-18T08:18:00.533636+0000","last_undegraded":"2024-09-18T08:18:00.533636+0000","last_fullsized":"2024-09-18T08:18:00.533636+0000","mapping_epoch":336,"log_start":"0'0","ondisk_log_start":"0'0","created":152,"last_epoch_clean":337,"parent":"0.0","parent_split_bits":4,"last_scrub":"74'79","last_scrub_stamp":"2024-09-18T08:15:43.275329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:00:11.893742+0000","last_clean_scrub_stamp":"2024-09-18T08:15:43.275329+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,2],"acting":[5,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.4","version":"310'42","reported_seq":736,"reported_epoch":389,"state":"active+clean","last_fresh":"2024-09-18T08:18:00.527059+0000","last_change":"2024-09-18T08:14:39.529072+0000","last_active":"2024-09-18T08:18:00.527059+0000","last_peered":"2024-09-18T08:18:00.527059+0000","last_clean":"2024-09-18T08:18:00.527059+0000","last_became_active":"2024-09-18T08:14:39.486174+0000","last_became_peered":"2024-09-18T08:14:39.486174+0000","last_unstale":"2024-09-18T08:18:00.527059+0000","last_undegraded":"2024-09-18T08:18:00.527059+0000","last_fullsized":"2024-09-18T08:18:00.527059+0000","mapping_epoch":307,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":308,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'28","last_scrub_stamp":"2024-09-18T08:10:46.497874+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_clean_scrub_stamp":"2024-09-18T08:10:46.497874+0000","objects_scrubbed":0,"log_size":42,"log_dups_size":0,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:18:26.645395+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":65,"num_read_kb":50,"num_write":38,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":298,"reported_epoch":390,"state":"active+clean","last_fresh":"2024-09-18T08:18:04.436403+0000","last_change":"2024-09-18T08:16:13.476275+0000","last_active":"2024-09-18T08:18:04.436403+0000","last_peered":"2024-09-18T08:18:04.436403+0000","last_clean":"2024-09-18T08:18:04.436403+0000","last_became_active":"2024-09-18T08:16:13.475879+0000","last_became_peered":"2024-09-18T08:16:13.475879+0000","last_unstale":"2024-09-18T08:18:04.436403+0000","last_undegraded":"2024-09-18T08:18:04.436403+0000","last_fullsized":"2024-09-18T08:18:04.436403+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:13:52.760146+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:46.860092+0000","last_clean_scrub_stamp":"2024-09-18T08:13:52.760146+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,1],"acting":[0,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"64'2","reported_seq":129,"reported_epoch":389,"state":"active+clean","last_fresh":"2024-09-18T08:18:00.533833+0000","last_change":"2024-09-18T08:17:08.332007+0000","last_active":"2024-09-18T08:18:00.533833+0000","last_peered":"2024-09-18T08:18:00.533833+0000","last_clean":"2024-09-18T08:18:00.533833+0000","last_became_active":"2024-09-18T08:15:41.453351+0000","last_became_peered":"2024-09-18T08:15:41.453351+0000","last_unstale":"2024-09-18T08:18:00.533833+0000","last_undegraded":"2024-09-18T08:18:00.533833+0000","last_fullsized":"2024-09-18T08:18:00.533833+0000","mapping_epoch":336,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":337,"parent":"0.0","parent_split_bits":0,"last_scrub":"64'2","last_scrub_stamp":"2024-09-18T08:17:08.331955+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:45.855979+0000","last_clean_scrub_stamp":"2024-09-18T08:17:08.331955+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:18:28.285273+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":722,"reported_epoch":390,"state":"active+clean","last_fresh":"2024-09-18T08:18:04.443051+0000","last_change":"2024-09-18T08:16:13.481778+0000","last_active":"2024-09-18T08:18:04.443051+0000","last_peered":"2024-09-18T08:18:04.443051+0000","last_clean":"2024-09-18T08:18:04.443051+0000","last_became_active":"2024-09-18T08:16:13.481456+0000","last_became_peered":"2024-09-18T08:16:13.481456+0000","last_unstale":"2024-09-18T08:18:04.443051+0000","last_undegraded":"2024-09-18T08:18:04.443051+0000","last_fullsized":"2024-09-18T08:18:04.443051+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:15:04.260540+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:50.886830+0000","last_clean_scrub_stamp":"2024-09-18T08:15:04.260540+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,6],"acting":[0,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":730,"reported_epoch":389,"state":"active+clean","last_fresh":"2024-09-18T08:18:00.542893+0000","last_change":"2024-09-18T08:16:13.462219+0000","last_active":"2024-09-18T08:18:00.542893+0000","last_peered":"2024-09-18T08:18:00.542893+0000","last_clean":"2024-09-18T08:18:00.542893+0000","last_became_active":"2024-09-18T08:16:13.459207+0000","last_became_peered":"2024-09-18T08:16:13.459207+0000","last_unstale":"2024-09-18T08:18:00.542893+0000","last_undegraded":"2024-09-18T08:18:00.542893+0000","last_fullsized":"2024-09-18T08:18:00.542893+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:15:59.948801+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_clean_scrub_stamp":"2024-09-18T08:15:59.948801+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.9","version":"389'12293","reported_seq":24579,"reported_epoch":389,"state":"active+clean","last_fresh":"2024-09-18T08:18:03.626302+0000","last_change":"2024-09-18T08:17:43.058512+0000","last_active":"2024-09-18T08:18:03.626302+0000","last_peered":"2024-09-18T08:18:03.626302+0000","last_clean":"2024-09-18T08:18:03.626302+0000","last_became_active":"2024-09-18T08:16:20.398277+0000","last_became_peered":"2024-09-18T08:16:20.398277+0000","last_unstale":"2024-09-18T08:18:03.626302+0000","last_undegraded":"2024-09-18T08:18:03.626302+0000","last_fullsized":"2024-09-18T08:18:03.626302+0000","mapping_epoch":353,"log_start":"369'8632","ondisk_log_start":"369'8632","created":311,"last_epoch_clean":354,"parent":"0.0","parent_split_bits":0,"last_scrub":"381'11012","last_scrub_stamp":"2024-09-18T08:17:43.058365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:14:43.412748+0000","last_clean_scrub_stamp":"2024-09-18T08:17:43.058365+0000","objects_scrubbed":1991,"log_size":3661,"log_dups_size":0,"ondisk_log_size":3661,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:18:43.655486+0000","scrub_duration":512,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":45154304,"num_objects":689,"num_object_clones":0,"num_object_copies":2067,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":689,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12293,"num_write_kb":415424,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6027,"num_bytes_recovered":394985472,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,3],"acting":[5,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.2","version":"74'79","reported_seq":636,"reported_epoch":390,"state":"active+clean","last_fresh":"2024-09-18T08:18:04.435918+0000","last_change":"2024-09-18T08:16:13.453252+0000","last_active":"2024-09-18T08:18:04.435918+0000","last_peered":"2024-09-18T08:18:04.435918+0000","last_clean":"2024-09-18T08:18:04.435918+0000","last_became_active":"2024-09-18T08:16:13.451687+0000","last_became_peered":"2024-09-18T08:16:13.451687+0000","last_unstale":"2024-09-18T08:18:04.435918+0000","last_undegraded":"2024-09-18T08:18:04.435918+0000","last_fullsized":"2024-09-18T08:18:04.435918+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":152,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":4,"last_scrub":"74'79","last_scrub_stamp":"2024-09-18T08:14:01.846338+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:00:11.893742+0000","last_clean_scrub_stamp":"2024-09-18T08:14:01.846338+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,5],"acting":[0,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.7","version":"271'17","reported_seq":698,"reported_epoch":389,"state":"active+clean","last_fresh":"2024-09-18T08:18:00.539475+0000","last_change":"2024-09-18T08:15:41.494329+0000","last_active":"2024-09-18T08:18:00.539475+0000","last_peered":"2024-09-18T08:18:00.539475+0000","last_clean":"2024-09-18T08:18:00.539475+0000","last_became_active":"2024-09-18T08:15:41.456482+0000","last_became_peered":"2024-09-18T08:15:41.456482+0000","last_unstale":"2024-09-18T08:18:00.539475+0000","last_undegraded":"2024-09-18T08:18:00.539475+0000","last_fullsized":"2024-09-18T08:18:00.539475+0000","mapping_epoch":336,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":337,"parent":"0.0","parent_split_bits":0,"last_scrub":"271'17","last_scrub_stamp":"2024-09-18T08:15:13.792602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_clean_scrub_stamp":"2024-09-18T08:15:13.792602+0000","objects_scrubbed":5,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":16,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":35,"num_read_kb":24,"num_write":20,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":990,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,0],"acting":[2,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":591,"reported_epoch":389,"state":"active+clean","last_fresh":"2024-09-18T08:18:00.525620+0000","last_change":"2024-09-18T08:15:49.244718+0000","last_active":"2024-09-18T08:18:00.525620+0000","last_peered":"2024-09-18T08:18:00.525620+0000","last_clean":"2024-09-18T08:18:00.525620+0000","last_became_active":"2024-09-18T08:14:39.432254+0000","last_became_peered":"2024-09-18T08:14:39.432254+0000","last_unstale":"2024-09-18T08:18:00.525620+0000","last_undegraded":"2024-09-18T08:18:00.525620+0000","last_fullsized":"2024-09-18T08:18:00.525620+0000","mapping_epoch":307,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":308,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:15:49.244156+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:46.860092+0000","last_clean_scrub_stamp":"2024-09-18T08:15:49.244156+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"62'1","reported_seq":722,"reported_epoch":389,"state":"active+clean","last_fresh":"2024-09-18T08:18:00.537957+0000","last_change":"2024-09-18T08:17:34.935837+0000","last_active":"2024-09-18T08:18:00.537957+0000","last_peered":"2024-09-18T08:18:00.537957+0000","last_clean":"2024-09-18T08:18:00.537957+0000","last_became_active":"2024-09-18T08:02:46.868175+0000","last_became_peered":"2024-09-18T08:02:46.868175+0000","last_unstale":"2024-09-18T08:18:00.537957+0000","last_undegraded":"2024-09-18T08:18:00.537957+0000","last_fullsized":"2024-09-18T08:18:00.537957+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"62'1","last_scrub_stamp":"2024-09-18T08:17:34.935618+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:45.855979+0000","last_clean_scrub_stamp":"2024-09-18T08:17:34.935618+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:18:36.691557+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0.00066785499999999997,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":432,"reported_epoch":389,"state":"active+clean","last_fresh":"2024-09-18T08:18:00.546433+0000","last_change":"2024-09-18T08:16:13.467275+0000","last_active":"2024-09-18T08:18:00.546433+0000","last_peered":"2024-09-18T08:18:00.546433+0000","last_clean":"2024-09-18T08:18:00.546433+0000","last_became_active":"2024-09-18T08:16:13.466910+0000","last_became_peered":"2024-09-18T08:16:13.466910+0000","last_unstale":"2024-09-18T08:18:00.546433+0000","last_undegraded":"2024-09-18T08:18:00.546433+0000","last_fullsized":"2024-09-18T08:18:00.546433+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:15:00.702835+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:50.886830+0000","last_clean_scrub_stamp":"2024-09-18T08:15:00.702835+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,1],"acting":[2,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":394,"reported_epoch":389,"state":"active+clean","last_fresh":"2024-09-18T08:18:00.546934+0000","last_change":"2024-09-18T08:17:35.381592+0000","last_active":"2024-09-18T08:18:00.546934+0000","last_peered":"2024-09-18T08:18:00.546934+0000","last_clean":"2024-09-18T08:18:00.546934+0000","last_became_active":"2024-09-18T08:14:05.577516+0000","last_became_peered":"2024-09-18T08:14:05.577516+0000","last_unstale":"2024-09-18T08:18:00.546934+0000","last_undegraded":"2024-09-18T08:18:00.546934+0000","last_fullsized":"2024-09-18T08:18:00.546934+0000","mapping_epoch":292,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":293,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:17:35.381401+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_clean_scrub_stamp":"2024-09-18T08:17:35.381401+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:18:54.224811+0000","scrub_duration":23,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.8","version":"390'12141","reported_seq":23991,"reported_epoch":390,"state":"active+clean","last_fresh":"2024-09-18T08:18:05.063632+0000","last_change":"2024-09-18T08:16:54.572396+0000","last_active":"2024-09-18T08:18:05.063632+0000","last_peered":"2024-09-18T08:18:05.063632+0000","last_clean":"2024-09-18T08:18:05.063632+0000","last_became_active":"2024-09-18T08:16:54.554366+0000","last_became_peered":"2024-09-18T08:16:54.554366+0000","last_unstale":"2024-09-18T08:18:05.063632+0000","last_undegraded":"2024-09-18T08:18:05.063632+0000","last_fullsized":"2024-09-18T08:18:05.063632+0000","mapping_epoch":364,"log_start":"368'8348","ondisk_log_start":"368'8348","created":311,"last_epoch_clean":365,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:14:43.412748+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:14:43.412748+0000","last_clean_scrub_stamp":"2024-09-18T08:14:43.412748+0000","objects_scrubbed":0,"log_size":3793,"log_dups_size":0,"ondisk_log_size":3793,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35323904,"num_objects":539,"num_object_clones":0,"num_object_copies":1617,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":539,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12141,"num_write_kb":405760,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5820,"num_bytes_recovered":381419520,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,5],"acting":[0,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"74'79","reported_seq":366,"reported_epoch":389,"state":"active+clean","last_fresh":"2024-09-18T08:18:00.526832+0000","last_change":"2024-09-18T08:16:13.455530+0000","last_active":"2024-09-18T08:18:00.526832+0000","last_peered":"2024-09-18T08:18:00.526832+0000","last_clean":"2024-09-18T08:18:00.526832+0000","last_became_active":"2024-09-18T08:16:13.454138+0000","last_became_peered":"2024-09-18T08:16:13.454138+0000","last_unstale":"2024-09-18T08:18:00.526832+0000","last_undegraded":"2024-09-18T08:18:00.526832+0000","last_fullsized":"2024-09-18T08:18:00.526832+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":152,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":3,"last_scrub":"74'79","last_scrub_stamp":"2024-09-18T08:16:00.303863+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:00:11.893742+0000","last_clean_scrub_stamp":"2024-09-18T08:16:00.303863+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,5],"acting":[1,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.6","version":"271'24","reported_seq":806,"reported_epoch":390,"state":"active+clean","last_fresh":"2024-09-18T08:18:04.438579+0000","last_change":"2024-09-18T08:15:10.187667+0000","last_active":"2024-09-18T08:18:04.438579+0000","last_peered":"2024-09-18T08:18:04.438579+0000","last_clean":"2024-09-18T08:18:04.438579+0000","last_became_active":"2024-09-18T08:14:39.438033+0000","last_became_peered":"2024-09-18T08:14:39.438033+0000","last_unstale":"2024-09-18T08:18:04.438579+0000","last_undegraded":"2024-09-18T08:18:04.438579+0000","last_fullsized":"2024-09-18T08:18:04.438579+0000","mapping_epoch":307,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":308,"parent":"0.0","parent_split_bits":0,"last_scrub":"271'24","last_scrub_stamp":"2024-09-18T08:15:10.187593+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_clean_scrub_stamp":"2024-09-18T08:15:10.187593+0000","objects_scrubbed":6,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":36,"num_read_kb":30,"num_write":24,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":735,"reported_epoch":390,"state":"active+clean","last_fresh":"2024-09-18T08:18:04.437216+0000","last_change":"2024-09-18T08:16:13.461291+0000","last_active":"2024-09-18T08:18:04.437216+0000","last_peered":"2024-09-18T08:18:04.437216+0000","last_clean":"2024-09-18T08:18:04.437216+0000","last_became_active":"2024-09-18T08:16:13.460391+0000","last_became_peered":"2024-09-18T08:16:13.460391+0000","last_unstale":"2024-09-18T08:18:04.437216+0000","last_undegraded":"2024-09-18T08:18:04.437216+0000","last_fullsized":"2024-09-18T08:18:04.437216+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:14:37.133544+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:46.860092+0000","last_clean_scrub_stamp":"2024-09-18T08:14:37.133544+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,5],"acting":[0,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"70'5","reported_seq":865,"reported_epoch":389,"state":"active+clean","last_fresh":"2024-09-18T08:18:01.039991+0000","last_change":"2024-09-18T08:15:57.384301+0000","last_active":"2024-09-18T08:18:01.039991+0000","last_peered":"2024-09-18T08:18:01.039991+0000","last_clean":"2024-09-18T08:18:01.039991+0000","last_became_active":"2024-09-18T08:14:40.353801+0000","last_became_peered":"2024-09-18T08:14:40.353801+0000","last_unstale":"2024-09-18T08:18:01.039991+0000","last_undegraded":"2024-09-18T08:18:01.039991+0000","last_fullsized":"2024-09-18T08:18:01.039991+0000","mapping_epoch":308,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":309,"parent":"0.0","parent_split_bits":0,"last_scrub":"70'5","last_scrub_stamp":"2024-09-18T08:15:57.384224+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:45.855979+0000","last_clean_scrub_stamp":"2024-09-18T08:15:57.384224+0000","objects_scrubbed":1,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0.000504055,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":375,"num_read_kb":370,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":389,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":117,"reported_epoch":389,"state":"active+clean","last_fresh":"2024-09-18T08:18:00.536145+0000","last_change":"2024-09-18T08:15:41.443577+0000","last_active":"2024-09-18T08:18:00.536145+0000","last_peered":"2024-09-18T08:18:00.536145+0000","last_clean":"2024-09-18T08:18:00.536145+0000","last_became_active":"2024-09-18T08:15:41.443080+0000","last_became_peered":"2024-09-18T08:15:41.443080+0000","last_unstale":"2024-09-18T08:18:00.536145+0000","last_undegraded":"2024-09-18T08:18:00.536145+0000","last_fullsized":"2024-09-18T08:18:00.536145+0000","mapping_epoch":336,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":337,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:15:02.179693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:50.886830+0000","last_clean_scrub_stamp":"2024-09-18T08:15:02.179693+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":513,"reported_epoch":389,"state":"active+clean","last_fresh":"2024-09-18T08:18:00.522798+0000","last_change":"2024-09-18T08:15:41.437061+0000","last_active":"2024-09-18T08:18:00.522798+0000","last_peered":"2024-09-18T08:18:00.522798+0000","last_clean":"2024-09-18T08:18:00.522798+0000","last_became_active":"2024-09-18T08:15:41.436692+0000","last_became_peered":"2024-09-18T08:15:41.436692+0000","last_unstale":"2024-09-18T08:18:00.522798+0000","last_undegraded":"2024-09-18T08:18:00.522798+0000","last_fullsized":"2024-09-18T08:18:00.522798+0000","mapping_epoch":336,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":337,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:15:07.219279+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_clean_scrub_stamp":"2024-09-18T08:15:07.219279+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.b","version":"389'12216","reported_seq":12384,"reported_epoch":389,"state":"active+clean","last_fresh":"2024-09-18T08:18:02.554620+0000","last_change":"2024-09-18T08:14:44.432885+0000","last_active":"2024-09-18T08:18:02.554620+0000","last_peered":"2024-09-18T08:18:02.554620+0000","last_clean":"2024-09-18T08:18:02.554620+0000","last_became_active":"2024-09-18T08:14:44.432599+0000","last_became_peered":"2024-09-18T08:14:44.432599+0000","last_unstale":"2024-09-18T08:18:02.554620+0000","last_undegraded":"2024-09-18T08:18:02.554620+0000","last_fullsized":"2024-09-18T08:18:02.554620+0000","mapping_epoch":311,"log_start":"369'8601","ondisk_log_start":"369'8601","created":311,"last_epoch_clean":312,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:14:43.412748+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:14:43.412748+0000","last_clean_scrub_stamp":"2024-09-18T08:14:43.412748+0000","objects_scrubbed":0,"log_size":3615,"log_dups_size":0,"ondisk_log_size":3615,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":45481984,"num_objects":694,"num_object_clones":0,"num_object_copies":2082,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":694,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12216,"num_write_kb":413120,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,0],"acting":[1,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.5","version":"349'35","reported_seq":491,"reported_epoch":390,"state":"active+clean","last_fresh":"2024-09-18T08:18:04.442382+0000","last_change":"2024-09-18T08:16:13.521519+0000","last_active":"2024-09-18T08:18:04.442382+0000","last_peered":"2024-09-18T08:18:04.442382+0000","last_clean":"2024-09-18T08:18:04.442382+0000","last_became_active":"2024-09-18T08:16:13.460467+0000","last_became_peered":"2024-09-18T08:16:13.460467+0000","last_unstale":"2024-09-18T08:18:04.442382+0000","last_undegraded":"2024-09-18T08:18:04.442382+0000","last_fullsized":"2024-09-18T08:18:04.442382+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"271'33","last_scrub_stamp":"2024-09-18T08:13:28.626987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_clean_scrub_stamp":"2024-09-18T08:13:28.626987+0000","objects_scrubbed":8,"log_size":35,"log_dups_size":0,"ondisk_log_size":35,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":24,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":42,"num_read_kb":32,"num_write":25,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":308,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,3],"acting":[0,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":725,"reported_epoch":389,"state":"active+clean","last_fresh":"2024-09-18T08:18:00.542833+0000","last_change":"2024-09-18T08:15:41.464630+0000","last_active":"2024-09-18T08:18:00.542833+0000","last_peered":"2024-09-18T08:18:00.542833+0000","last_clean":"2024-09-18T08:18:00.542833+0000","last_became_active":"2024-09-18T08:15:41.464313+0000","last_became_peered":"2024-09-18T08:15:41.464313+0000","last_unstale":"2024-09-18T08:18:00.542833+0000","last_undegraded":"2024-09-18T08:18:00.542833+0000","last_fullsized":"2024-09-18T08:18:00.542833+0000","mapping_epoch":336,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":337,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:14:32.693460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:46.860092+0000","last_clean_scrub_stamp":"2024-09-18T08:14:32.693460+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":38,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"74'79","reported_seq":317,"reported_epoch":390,"state":"active+clean","last_fresh":"2024-09-18T08:18:04.437868+0000","last_change":"2024-09-18T08:14:40.357413+0000","last_active":"2024-09-18T08:18:04.437868+0000","last_peered":"2024-09-18T08:18:04.437868+0000","last_clean":"2024-09-18T08:18:04.437868+0000","last_became_active":"2024-09-18T08:14:40.357063+0000","last_became_peered":"2024-09-18T08:14:40.357063+0000","last_unstale":"2024-09-18T08:18:04.437868+0000","last_undegraded":"2024-09-18T08:18:04.437868+0000","last_fullsized":"2024-09-18T08:18:04.437868+0000","mapping_epoch":308,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":309,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'79","last_scrub_stamp":"2024-09-18T08:13:51.810296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:00:11.893742+0000","last_clean_scrub_stamp":"2024-09-18T08:13:51.810296+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,1],"acting":[0,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":569,"reported_epoch":389,"state":"active+clean","last_fresh":"2024-09-18T08:18:00.538198+0000","last_change":"2024-09-18T08:17:36.886577+0000","last_active":"2024-09-18T08:18:00.538198+0000","last_peered":"2024-09-18T08:18:00.538198+0000","last_clean":"2024-09-18T08:18:00.538198+0000","last_became_active":"2024-09-18T08:15:41.441392+0000","last_became_peered":"2024-09-18T08:15:41.441392+0000","last_unstale":"2024-09-18T08:18:00.538198+0000","last_undegraded":"2024-09-18T08:18:00.538198+0000","last_fullsized":"2024-09-18T08:18:00.538198+0000","mapping_epoch":336,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":337,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:17:36.886507+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:50.886830+0000","last_clean_scrub_stamp":"2024-09-18T08:17:36.886507+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:18:41.767214+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,0],"acting":[2,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":124,"reported_epoch":389,"state":"active+clean","last_fresh":"2024-09-18T08:18:00.536042+0000","last_change":"2024-09-18T08:16:13.472241+0000","last_active":"2024-09-18T08:18:00.536042+0000","last_peered":"2024-09-18T08:18:00.536042+0000","last_clean":"2024-09-18T08:18:00.536042+0000","last_became_active":"2024-09-18T08:16:13.471911+0000","last_became_peered":"2024-09-18T08:16:13.471911+0000","last_unstale":"2024-09-18T08:18:00.536042+0000","last_undegraded":"2024-09-18T08:18:00.536042+0000","last_fullsized":"2024-09-18T08:18:00.536042+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:12:05.886281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_clean_scrub_stamp":"2024-09-18T08:12:05.886281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:18:25.503591+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,1],"acting":[5,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.0","version":"389'12210","reported_seq":23344,"reported_epoch":389,"state":"active+clean","last_fresh":"2024-09-18T08:18:02.573198+0000","last_change":"2024-09-18T08:16:57.592438+0000","last_active":"2024-09-18T08:18:02.573198+0000","last_peered":"2024-09-18T08:18:02.573198+0000","last_clean":"2024-09-18T08:18:02.573198+0000","last_became_active":"2024-09-18T08:16:57.574026+0000","last_became_peered":"2024-09-18T08:16:57.574026+0000","last_unstale":"2024-09-18T08:18:02.573198+0000","last_undegraded":"2024-09-18T08:18:02.573198+0000","last_fullsized":"2024-09-18T08:18:02.573198+0000","mapping_epoch":366,"log_start":"369'8607","ondisk_log_start":"369'8607","created":311,"last_epoch_clean":367,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:14:43.412748+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:14:43.412748+0000","last_clean_scrub_stamp":"2024-09-18T08:14:43.412748+0000","objects_scrubbed":0,"log_size":3603,"log_dups_size":0,"ondisk_log_size":3603,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":44564480,"num_objects":680,"num_object_clones":0,"num_object_copies":2040,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":680,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12210,"num_write_kb":412480,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5473,"num_bytes_recovered":358678528,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,0],"acting":[1,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.e","version":"271'19","reported_seq":413,"reported_epoch":389,"state":"active+clean","last_fresh":"2024-09-18T08:18:00.523517+0000","last_change":"2024-09-18T08:17:30.690655+0000","last_active":"2024-09-18T08:18:00.523517+0000","last_peered":"2024-09-18T08:18:00.523517+0000","last_clean":"2024-09-18T08:18:00.523517+0000","last_became_active":"2024-09-18T08:16:13.458001+0000","last_became_peered":"2024-09-18T08:16:13.458001+0000","last_unstale":"2024-09-18T08:18:00.523517+0000","last_undegraded":"2024-09-18T08:18:00.523517+0000","last_fullsized":"2024-09-18T08:18:00.523517+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"271'19","last_scrub_stamp":"2024-09-18T08:17:30.690581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_clean_scrub_stamp":"2024-09-18T08:17:30.690581+0000","objects_scrubbed":5,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:18:59.500251+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":31,"num_read_kb":24,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,3],"acting":[1,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":379,"reported_epoch":389,"state":"active+clean","last_fresh":"2024-09-18T08:18:00.537743+0000","last_change":"2024-09-18T08:16:13.476914+0000","last_active":"2024-09-18T08:18:00.537743+0000","last_peered":"2024-09-18T08:18:00.537743+0000","last_clean":"2024-09-18T08:18:00.537743+0000","last_became_active":"2024-09-18T08:16:13.476579+0000","last_became_peered":"2024-09-18T08:16:13.476579+0000","last_unstale":"2024-09-18T08:18:00.537743+0000","last_undegraded":"2024-09-18T08:18:00.537743+0000","last_fullsized":"2024-09-18T08:18:00.537743+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:11:59.458232+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:46.860092+0000","last_clean_scrub_stamp":"2024-09-18T08:11:59.458232+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:18:25.909992+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,5],"acting":[2,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":132,"reported_epoch":389,"state":"active+clean","last_fresh":"2024-09-18T08:18:00.535975+0000","last_change":"2024-09-18T08:17:38.616479+0000","last_active":"2024-09-18T08:18:00.535975+0000","last_peered":"2024-09-18T08:18:00.535975+0000","last_clean":"2024-09-18T08:18:00.535975+0000","last_became_active":"2024-09-18T08:16:13.443884+0000","last_became_peered":"2024-09-18T08:16:13.443884+0000","last_unstale":"2024-09-18T08:18:00.535975+0000","last_undegraded":"2024-09-18T08:18:00.535975+0000","last_fullsized":"2024-09-18T08:18:00.535975+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:17:38.616405+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:50.886830+0000","last_clean_scrub_stamp":"2024-09-18T08:17:38.616405+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:19:06.381094+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,2],"acting":[5,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":712,"reported_epoch":389,"state":"active+clean","last_fresh":"2024-09-18T08:18:00.543839+0000","last_change":"2024-09-18T08:15:41.465035+0000","last_active":"2024-09-18T08:18:00.543839+0000","last_peered":"2024-09-18T08:18:00.543839+0000","last_clean":"2024-09-18T08:18:00.543839+0000","last_became_active":"2024-09-18T08:15:41.464500+0000","last_became_peered":"2024-09-18T08:15:41.464500+0000","last_unstale":"2024-09-18T08:18:00.543839+0000","last_undegraded":"2024-09-18T08:18:00.543839+0000","last_fullsized":"2024-09-18T08:18:00.543839+0000","mapping_epoch":336,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":337,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:15:02.861116+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_clean_scrub_stamp":"2024-09-18T08:15:02.861116+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":81,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.3","version":"389'11932","reported_seq":25150,"reported_epoch":389,"state":"active+clean","last_fresh":"2024-09-18T08:18:04.093238+0000","last_change":"2024-09-18T08:16:40.417668+0000","last_active":"2024-09-18T08:18:04.093238+0000","last_peered":"2024-09-18T08:18:04.093238+0000","last_clean":"2024-09-18T08:18:04.093238+0000","last_became_active":"2024-09-18T08:16:40.406526+0000","last_became_peered":"2024-09-18T08:16:40.406526+0000","last_unstale":"2024-09-18T08:18:04.093238+0000","last_undegraded":"2024-09-18T08:18:04.093238+0000","last_fullsized":"2024-09-18T08:18:04.093238+0000","mapping_epoch":359,"log_start":"358'7313","ondisk_log_start":"358'7313","created":311,"last_epoch_clean":360,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:14:43.412748+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:14:43.412748+0000","last_clean_scrub_stamp":"2024-09-18T08:14:43.412748+0000","objects_scrubbed":0,"log_size":4619,"log_dups_size":0,"ondisk_log_size":4619,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:18:58.960501+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41418752,"num_objects":632,"num_object_clones":0,"num_object_copies":1896,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":632,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11932,"num_write_kb":402048,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6505,"num_bytes_recovered":426311680,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,3],"acting":[2,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.8","version":"74'79","reported_seq":129,"reported_epoch":389,"state":"active+clean","last_fresh":"2024-09-18T08:18:00.533409+0000","last_change":"2024-09-18T08:17:43.558527+0000","last_active":"2024-09-18T08:18:00.533409+0000","last_peered":"2024-09-18T08:18:00.533409+0000","last_clean":"2024-09-18T08:18:00.533409+0000","last_became_active":"2024-09-18T08:15:41.454781+0000","last_became_peered":"2024-09-18T08:15:41.454781+0000","last_unstale":"2024-09-18T08:18:00.533409+0000","last_undegraded":"2024-09-18T08:18:00.533409+0000","last_fullsized":"2024-09-18T08:18:00.533409+0000","mapping_epoch":336,"log_start":"0'0","ondisk_log_start":"0'0","created":152,"last_epoch_clean":337,"parent":"0.0","parent_split_bits":4,"last_scrub":"74'79","last_scrub_stamp":"2024-09-18T08:17:43.558474+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:00:11.893742+0000","last_clean_scrub_stamp":"2024-09-18T08:17:43.558474+0000","objects_scrubbed":1,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:18:45.862783+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262688,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":262688,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,3],"acting":[5,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.d","version":"271'31","reported_seq":344,"reported_epoch":389,"state":"active+clean","last_fresh":"2024-09-18T08:18:00.545744+0000","last_change":"2024-09-18T08:16:13.511045+0000","last_active":"2024-09-18T08:18:00.545744+0000","last_peered":"2024-09-18T08:18:00.545744+0000","last_clean":"2024-09-18T08:18:00.545744+0000","last_became_active":"2024-09-18T08:16:13.466471+0000","last_became_peered":"2024-09-18T08:16:13.466471+0000","last_unstale":"2024-09-18T08:18:00.545744+0000","last_undegraded":"2024-09-18T08:18:00.545744+0000","last_fullsized":"2024-09-18T08:18:00.545744+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"271'31","last_scrub_stamp":"2024-09-18T08:15:03.678670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_clean_scrub_stamp":"2024-09-18T08:15:03.678670+0000","objects_scrubbed":0,"log_size":31,"log_dups_size":0,"ondisk_log_size":31,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":48,"num_read_kb":38,"num_write":32,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,6],"acting":[2,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":368,"reported_epoch":389,"state":"active+clean","last_fresh":"2024-09-18T08:18:00.526232+0000","last_change":"2024-09-18T08:16:13.487196+0000","last_active":"2024-09-18T08:18:00.526232+0000","last_peered":"2024-09-18T08:18:00.526232+0000","last_clean":"2024-09-18T08:18:00.526232+0000","last_became_active":"2024-09-18T08:16:13.483143+0000","last_became_peered":"2024-09-18T08:16:13.483143+0000","last_unstale":"2024-09-18T08:18:00.526232+0000","last_undegraded":"2024-09-18T08:18:00.526232+0000","last_fullsized":"2024-09-18T08:18:00.526232+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:15:09.614752+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:46.860092+0000","last_clean_scrub_stamp":"2024-09-18T08:15:09.614752+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,3],"acting":[1,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":598,"reported_epoch":389,"state":"active+clean","last_fresh":"2024-09-18T08:18:00.522844+0000","last_change":"2024-09-18T08:16:13.460968+0000","last_active":"2024-09-18T08:18:00.522844+0000","last_peered":"2024-09-18T08:18:00.522844+0000","last_clean":"2024-09-18T08:18:00.522844+0000","last_became_active":"2024-09-18T08:16:13.460650+0000","last_became_peered":"2024-09-18T08:16:13.460650+0000","last_unstale":"2024-09-18T08:18:00.522844+0000","last_undegraded":"2024-09-18T08:18:00.522844+0000","last_fullsized":"2024-09-18T08:18:00.522844+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:14:42.230420+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:50.886830+0000","last_clean_scrub_stamp":"2024-09-18T08:14:42.230420+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:18:57.718405+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,6],"acting":[1,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":435,"reported_epoch":389,"state":"active+clean","last_fresh":"2024-09-18T08:18:00.539390+0000","last_change":"2024-09-18T08:17:33.946061+0000","last_active":"2024-09-18T08:18:00.539390+0000","last_peered":"2024-09-18T08:18:00.539390+0000","last_clean":"2024-09-18T08:18:00.539390+0000","last_became_active":"2024-09-18T08:16:13.457516+0000","last_became_peered":"2024-09-18T08:16:13.457516+0000","last_unstale":"2024-09-18T08:18:00.539390+0000","last_undegraded":"2024-09-18T08:18:00.539390+0000","last_fullsized":"2024-09-18T08:18:00.539390+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:17:33.945785+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_clean_scrub_stamp":"2024-09-18T08:17:33.945785+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:18:58.386817+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.2","version":"389'11929","reported_seq":14878,"reported_epoch":389,"state":"active+clean","last_fresh":"2024-09-18T08:18:02.460825+0000","last_change":"2024-09-18T08:17:54.452837+0000","last_active":"2024-09-18T08:18:02.460825+0000","last_peered":"2024-09-18T08:18:02.460825+0000","last_clean":"2024-09-18T08:18:02.460825+0000","last_became_active":"2024-09-18T08:17:54.424707+0000","last_became_peered":"2024-09-18T08:17:54.424707+0000","last_unstale":"2024-09-18T08:18:02.460825+0000","last_undegraded":"2024-09-18T08:18:02.460825+0000","last_fullsized":"2024-09-18T08:18:02.460825+0000","mapping_epoch":387,"log_start":"369'8354","ondisk_log_start":"369'8354","created":311,"last_epoch_clean":388,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:14:43.412748+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:14:43.412748+0000","last_clean_scrub_stamp":"2024-09-18T08:14:43.412748+0000","objects_scrubbed":0,"log_size":3575,"log_dups_size":0,"ondisk_log_size":3575,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:18:58.685912+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":45023232,"num_objects":687,"num_object_clones":0,"num_object_copies":2061,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":687,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11929,"num_write_kb":403712,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1374,"num_bytes_recovered":90046464,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"74'79","reported_seq":538,"reported_epoch":389,"state":"active+clean","last_fresh":"2024-09-18T08:18:00.536681+0000","last_change":"2024-09-18T08:17:44.331632+0000","last_active":"2024-09-18T08:18:00.536681+0000","last_peered":"2024-09-18T08:18:00.536681+0000","last_clean":"2024-09-18T08:18:00.536681+0000","last_became_active":"2024-09-18T08:15:41.465341+0000","last_became_peered":"2024-09-18T08:15:41.465341+0000","last_unstale":"2024-09-18T08:18:00.536681+0000","last_undegraded":"2024-09-18T08:18:00.536681+0000","last_fullsized":"2024-09-18T08:18:00.536681+0000","mapping_epoch":336,"log_start":"0'0","ondisk_log_start":"0'0","created":152,"last_epoch_clean":337,"parent":"0.0","parent_split_bits":4,"last_scrub":"74'79","last_scrub_stamp":"2024-09-18T08:17:44.331322+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:00:11.893742+0000","last_clean_scrub_stamp":"2024-09-18T08:17:44.331322+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:18:49.579430+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.c","version":"271'20","reported_seq":322,"reported_epoch":389,"state":"active+clean","last_fresh":"2024-09-18T08:18:00.544759+0000","last_change":"2024-09-18T08:16:13.526582+0000","last_active":"2024-09-18T08:18:00.544759+0000","last_peered":"2024-09-18T08:18:00.544759+0000","last_clean":"2024-09-18T08:18:00.544759+0000","last_became_active":"2024-09-18T08:16:13.476918+0000","last_became_peered":"2024-09-18T08:16:13.476918+0000","last_unstale":"2024-09-18T08:18:00.544759+0000","last_undegraded":"2024-09-18T08:18:00.544759+0000","last_fullsized":"2024-09-18T08:18:00.544759+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"271'20","last_scrub_stamp":"2024-09-18T08:15:56.673129+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_clean_scrub_stamp":"2024-09-18T08:15:56.673129+0000","objects_scrubbed":0,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":28,"num_read_kb":23,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,1],"acting":[3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":732,"reported_epoch":389,"state":"active+clean","last_fresh":"2024-09-18T08:18:00.544181+0000","last_change":"2024-09-18T08:16:13.479508+0000","last_active":"2024-09-18T08:18:00.544181+0000","last_peered":"2024-09-18T08:18:00.544181+0000","last_clean":"2024-09-18T08:18:00.544181+0000","last_became_active":"2024-09-18T08:16:13.479148+0000","last_became_peered":"2024-09-18T08:16:13.479148+0000","last_unstale":"2024-09-18T08:18:00.544181+0000","last_undegraded":"2024-09-18T08:18:00.544181+0000","last_fullsized":"2024-09-18T08:18:00.544181+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:14:04.626103+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:46.860092+0000","last_clean_scrub_stamp":"2024-09-18T08:14:04.626103+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:18:59.479308+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.d","version":"338'13","reported_seq":940,"reported_epoch":389,"state":"active+clean","last_fresh":"2024-09-18T08:18:00.545687+0000","last_change":"2024-09-18T08:17:38.886726+0000","last_active":"2024-09-18T08:18:00.545687+0000","last_peered":"2024-09-18T08:18:00.545687+0000","last_clean":"2024-09-18T08:18:00.545687+0000","last_became_active":"2024-09-18T08:15:41.441562+0000","last_became_peered":"2024-09-18T08:15:41.441562+0000","last_unstale":"2024-09-18T08:18:00.545687+0000","last_undegraded":"2024-09-18T08:18:00.545687+0000","last_fullsized":"2024-09-18T08:18:00.545687+0000","mapping_epoch":336,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":337,"parent":"0.0","parent_split_bits":0,"last_scrub":"338'13","last_scrub_stamp":"2024-09-18T08:17:38.886654+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:50.886830+0000","last_clean_scrub_stamp":"2024-09-18T08:17:38.886654+0000","objects_scrubbed":1,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:18:41.947513+0000","scrub_duration":26,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,0],"acting":[2,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":375,"reported_epoch":389,"state":"active+clean","last_fresh":"2024-09-18T08:18:00.523111+0000","last_change":"2024-09-18T08:16:13.465932+0000","last_active":"2024-09-18T08:18:00.523111+0000","last_peered":"2024-09-18T08:18:00.523111+0000","last_clean":"2024-09-18T08:18:00.523111+0000","last_became_active":"2024-09-18T08:16:13.464924+0000","last_became_peered":"2024-09-18T08:16:13.464924+0000","last_unstale":"2024-09-18T08:18:00.523111+0000","last_undegraded":"2024-09-18T08:18:00.523111+0000","last_fullsized":"2024-09-18T08:18:00.523111+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:15:47.604976+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_clean_scrub_stamp":"2024-09-18T08:15:47.604976+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.5","version":"389'12115","reported_seq":12311,"reported_epoch":389,"state":"active+clean","last_fresh":"2024-09-18T08:18:04.097308+0000","last_change":"2024-09-18T08:17:42.813269+0000","last_active":"2024-09-18T08:18:04.097308+0000","last_peered":"2024-09-18T08:18:04.097308+0000","last_clean":"2024-09-18T08:18:04.097308+0000","last_became_active":"2024-09-18T08:14:44.431835+0000","last_became_peered":"2024-09-18T08:14:44.431835+0000","last_unstale":"2024-09-18T08:18:04.097308+0000","last_undegraded":"2024-09-18T08:18:04.097308+0000","last_fullsized":"2024-09-18T08:18:04.097308+0000","mapping_epoch":311,"log_start":"358'7413","ondisk_log_start":"358'7413","created":311,"last_epoch_clean":312,"parent":"0.0","parent_split_bits":0,"last_scrub":"381'10819","last_scrub_stamp":"2024-09-18T08:17:42.813063+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:14:43.412748+0000","last_clean_scrub_stamp":"2024-09-18T08:17:42.813063+0000","objects_scrubbed":1926,"log_size":4702,"log_dups_size":0,"ondisk_log_size":4702,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:18:59.853937+0000","scrub_duration":861,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39911424,"num_objects":609,"num_object_clones":0,"num_object_copies":1827,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":609,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12115,"num_write_kb":407168,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.b","version":"271'15","reported_seq":797,"reported_epoch":390,"state":"active+clean","last_fresh":"2024-09-18T08:18:04.436229+0000","last_change":"2024-09-18T08:17:48.720750+0000","last_active":"2024-09-18T08:18:04.436229+0000","last_peered":"2024-09-18T08:18:04.436229+0000","last_clean":"2024-09-18T08:18:04.436229+0000","last_became_active":"2024-09-18T08:16:13.480279+0000","last_became_peered":"2024-09-18T08:16:13.480279+0000","last_unstale":"2024-09-18T08:18:04.436229+0000","last_undegraded":"2024-09-18T08:18:04.436229+0000","last_fullsized":"2024-09-18T08:18:04.436229+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"271'15","last_scrub_stamp":"2024-09-18T08:17:48.720672+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_clean_scrub_stamp":"2024-09-18T08:17:48.720672+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:19:11.267567+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,3],"acting":[0,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":117,"reported_epoch":389,"state":"active+clean","last_fresh":"2024-09-18T08:18:00.534148+0000","last_change":"2024-09-18T08:15:41.443068+0000","last_active":"2024-09-18T08:18:00.534148+0000","last_peered":"2024-09-18T08:18:00.534148+0000","last_clean":"2024-09-18T08:18:00.534148+0000","last_became_active":"2024-09-18T08:15:41.442697+0000","last_became_peered":"2024-09-18T08:15:41.442697+0000","last_unstale":"2024-09-18T08:18:00.534148+0000","last_undegraded":"2024-09-18T08:18:00.534148+0000","last_fullsized":"2024-09-18T08:18:00.534148+0000","mapping_epoch":336,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":337,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:11:22.076452+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:46.860092+0000","last_clean_scrub_stamp":"2024-09-18T08:11:22.076452+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:18:24.518634+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":718,"reported_epoch":389,"state":"active+clean","last_fresh":"2024-09-18T08:18:00.538751+0000","last_change":"2024-09-18T08:16:46.007260+0000","last_active":"2024-09-18T08:18:00.538751+0000","last_peered":"2024-09-18T08:18:00.538751+0000","last_clean":"2024-09-18T08:18:00.538751+0000","last_became_active":"2024-09-18T08:16:13.465129+0000","last_became_peered":"2024-09-18T08:16:13.465129+0000","last_unstale":"2024-09-18T08:18:00.538751+0000","last_undegraded":"2024-09-18T08:18:00.538751+0000","last_fullsized":"2024-09-18T08:18:00.538751+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:16:46.007194+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:50.886830+0000","last_clean_scrub_stamp":"2024-09-18T08:16:46.007194+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":27,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":723,"reported_epoch":390,"state":"active+clean","last_fresh":"2024-09-18T08:18:04.437123+0000","last_change":"2024-09-18T08:14:05.585049+0000","last_active":"2024-09-18T08:18:04.437123+0000","last_peered":"2024-09-18T08:18:04.437123+0000","last_clean":"2024-09-18T08:18:04.437123+0000","last_became_active":"2024-09-18T08:14:05.583887+0000","last_became_peered":"2024-09-18T08:14:05.583887+0000","last_unstale":"2024-09-18T08:18:04.437123+0000","last_undegraded":"2024-09-18T08:18:04.437123+0000","last_fullsized":"2024-09-18T08:18:04.437123+0000","mapping_epoch":292,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":293,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:12:38.702798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_clean_scrub_stamp":"2024-09-18T08:12:38.702798+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:18:54.733346+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,3],"acting":[0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.4","version":"390'12120","reported_seq":20178,"reported_epoch":390,"state":"active+clean","last_fresh":"2024-09-18T08:18:05.058006+0000","last_change":"2024-09-18T08:17:26.449992+0000","last_active":"2024-09-18T08:18:05.058006+0000","last_peered":"2024-09-18T08:18:05.058006+0000","last_clean":"2024-09-18T08:18:05.058006+0000","last_became_active":"2024-09-18T08:17:26.419323+0000","last_became_peered":"2024-09-18T08:17:26.419323+0000","last_unstale":"2024-09-18T08:18:05.058006+0000","last_undegraded":"2024-09-18T08:18:05.058006+0000","last_fullsized":"2024-09-18T08:18:05.058006+0000","mapping_epoch":377,"log_start":"368'8343","ondisk_log_start":"368'8343","created":311,"last_epoch_clean":378,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:14:43.412748+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:14:43.412748+0000","last_clean_scrub_stamp":"2024-09-18T08:14:43.412748+0000","objects_scrubbed":0,"log_size":3777,"log_dups_size":0,"ondisk_log_size":3777,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36569088,"num_objects":558,"num_object_clones":0,"num_object_copies":1674,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":558,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12120,"num_write_kb":405696,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3934,"num_bytes_recovered":257818624,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1],"acting":[0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.a","version":"271'35","reported_seq":628,"reported_epoch":389,"state":"active+clean","last_fresh":"2024-09-18T08:18:00.526467+0000","last_change":"2024-09-18T08:15:43.282823+0000","last_active":"2024-09-18T08:18:00.526467+0000","last_peered":"2024-09-18T08:18:00.526467+0000","last_clean":"2024-09-18T08:18:00.526467+0000","last_became_active":"2024-09-18T08:15:41.441186+0000","last_became_peered":"2024-09-18T08:15:41.441186+0000","last_unstale":"2024-09-18T08:18:00.526467+0000","last_undegraded":"2024-09-18T08:18:00.526467+0000","last_fullsized":"2024-09-18T08:18:00.526467+0000","mapping_epoch":336,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":337,"parent":"0.0","parent_split_bits":0,"last_scrub":"271'35","last_scrub_stamp":"2024-09-18T08:15:43.282736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_clean_scrub_stamp":"2024-09-18T08:15:43.282736+0000","objects_scrubbed":9,"log_size":35,"log_dups_size":0,"ondisk_log_size":35,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":55,"num_read_kb":44,"num_write":36,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":44,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,5],"acting":[1,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":117,"reported_epoch":389,"state":"active+clean","last_fresh":"2024-09-18T08:18:00.535021+0000","last_change":"2024-09-18T08:15:41.452272+0000","last_active":"2024-09-18T08:18:00.535021+0000","last_peered":"2024-09-18T08:18:00.535021+0000","last_clean":"2024-09-18T08:18:00.535021+0000","last_became_active":"2024-09-18T08:15:41.446405+0000","last_became_peered":"2024-09-18T08:15:41.446405+0000","last_unstale":"2024-09-18T08:18:00.535021+0000","last_undegraded":"2024-09-18T08:18:00.535021+0000","last_fullsized":"2024-09-18T08:18:00.535021+0000","mapping_epoch":336,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":337,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:12:31.205468+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:46.860092+0000","last_clean_scrub_stamp":"2024-09-18T08:12:31.205468+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:18:27.536995+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,3],"acting":[5,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":426,"reported_epoch":389,"state":"active+clean","last_fresh":"2024-09-18T08:18:00.539558+0000","last_change":"2024-09-18T08:15:41.447379+0000","last_active":"2024-09-18T08:18:00.539558+0000","last_peered":"2024-09-18T08:18:00.539558+0000","last_clean":"2024-09-18T08:18:00.539558+0000","last_became_active":"2024-09-18T08:15:41.447075+0000","last_became_peered":"2024-09-18T08:15:41.447075+0000","last_unstale":"2024-09-18T08:18:00.539558+0000","last_undegraded":"2024-09-18T08:18:00.539558+0000","last_fullsized":"2024-09-18T08:18:00.539558+0000","mapping_epoch":336,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":337,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:15:10.767720+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:50.886830+0000","last_clean_scrub_stamp":"2024-09-18T08:15:10.767720+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":547,"reported_epoch":389,"state":"active+clean","last_fresh":"2024-09-18T08:18:00.536898+0000","last_change":"2024-09-18T08:15:41.722212+0000","last_active":"2024-09-18T08:18:00.536898+0000","last_peered":"2024-09-18T08:18:00.536898+0000","last_clean":"2024-09-18T08:18:00.536898+0000","last_became_active":"2024-09-18T08:14:05.578053+0000","last_became_peered":"2024-09-18T08:14:05.578053+0000","last_unstale":"2024-09-18T08:18:00.536898+0000","last_undegraded":"2024-09-18T08:18:00.536898+0000","last_fullsized":"2024-09-18T08:18:00.536898+0000","mapping_epoch":292,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":293,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:15:41.722163+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_clean_scrub_stamp":"2024-09-18T08:15:41.722163+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.7","version":"389'12084","reported_seq":22697,"reported_epoch":389,"state":"active+clean","last_fresh":"2024-09-18T08:18:04.349309+0000","last_change":"2024-09-18T08:17:06.465603+0000","last_active":"2024-09-18T08:18:04.349309+0000","last_peered":"2024-09-18T08:18:04.349309+0000","last_clean":"2024-09-18T08:18:04.349309+0000","last_became_active":"2024-09-18T08:17:06.445544+0000","last_became_peered":"2024-09-18T08:17:06.445544+0000","last_unstale":"2024-09-18T08:18:04.349309+0000","last_undegraded":"2024-09-18T08:18:04.349309+0000","last_fullsized":"2024-09-18T08:18:04.349309+0000","mapping_epoch":370,"log_start":"371'8605","ondisk_log_start":"371'8605","created":311,"last_epoch_clean":371,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:14:43.412748+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:14:43.412748+0000","last_clean_scrub_stamp":"2024-09-18T08:14:43.412748+0000","objects_scrubbed":0,"log_size":3479,"log_dups_size":0,"ondisk_log_size":3479,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38535168,"num_objects":588,"num_object_clones":0,"num_object_copies":1764,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":588,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12084,"num_write_kb":405504,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5204,"num_bytes_recovered":341049344,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.9","version":"271'18","reported_seq":435,"reported_epoch":389,"state":"active+clean","last_fresh":"2024-09-18T08:18:00.525510+0000","last_change":"2024-09-18T08:16:13.525130+0000","last_active":"2024-09-18T08:18:00.525510+0000","last_peered":"2024-09-18T08:18:00.525510+0000","last_clean":"2024-09-18T08:18:00.525510+0000","last_became_active":"2024-09-18T08:16:13.482580+0000","last_became_peered":"2024-09-18T08:16:13.482580+0000","last_unstale":"2024-09-18T08:18:00.525510+0000","last_undegraded":"2024-09-18T08:18:00.525510+0000","last_fullsized":"2024-09-18T08:18:00.525510+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"271'18","last_scrub_stamp":"2024-09-18T08:15:49.666776+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_clean_scrub_stamp":"2024-09-18T08:15:49.666776+0000","objects_scrubbed":5,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":33,"num_read_kb":24,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":880,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,6],"acting":[1,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":572,"reported_epoch":389,"state":"active+clean","last_fresh":"2024-09-18T08:18:00.526131+0000","last_change":"2024-09-18T08:17:35.673112+0000","last_active":"2024-09-18T08:18:00.526131+0000","last_peered":"2024-09-18T08:18:00.526131+0000","last_clean":"2024-09-18T08:18:00.526131+0000","last_became_active":"2024-09-18T08:16:13.462268+0000","last_became_peered":"2024-09-18T08:16:13.462268+0000","last_unstale":"2024-09-18T08:18:00.526131+0000","last_undegraded":"2024-09-18T08:18:00.526131+0000","last_fullsized":"2024-09-18T08:18:00.526131+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:17:35.673047+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:46.860092+0000","last_clean_scrub_stamp":"2024-09-18T08:17:35.673047+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:18:39.728756+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,5],"acting":[1,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":716,"reported_epoch":389,"state":"active+clean","last_fresh":"2024-09-18T08:18:00.537027+0000","last_change":"2024-09-18T08:17:40.958509+0000","last_active":"2024-09-18T08:18:00.537027+0000","last_peered":"2024-09-18T08:18:00.537027+0000","last_clean":"2024-09-18T08:18:00.537027+0000","last_became_active":"2024-09-18T08:14:39.399301+0000","last_became_peered":"2024-09-18T08:14:39.399301+0000","last_unstale":"2024-09-18T08:18:00.537027+0000","last_undegraded":"2024-09-18T08:18:00.537027+0000","last_fullsized":"2024-09-18T08:18:00.537027+0000","mapping_epoch":307,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":308,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:17:40.958275+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:50.886830+0000","last_clean_scrub_stamp":"2024-09-18T08:17:40.958275+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:18:57.914660+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":717,"reported_epoch":389,"state":"active+clean","last_fresh":"2024-09-18T08:18:00.542630+0000","last_change":"2024-09-18T08:17:09.487909+0000","last_active":"2024-09-18T08:18:00.542630+0000","last_peered":"2024-09-18T08:18:00.542630+0000","last_clean":"2024-09-18T08:18:00.542630+0000","last_became_active":"2024-09-18T08:14:40.356451+0000","last_became_peered":"2024-09-18T08:14:40.356451+0000","last_unstale":"2024-09-18T08:18:00.542630+0000","last_undegraded":"2024-09-18T08:18:00.542630+0000","last_fullsized":"2024-09-18T08:18:00.542630+0000","mapping_epoch":308,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":309,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:17:09.487823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_clean_scrub_stamp":"2024-09-18T08:17:09.487823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:18:35.062728+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.6","version":"389'11937","reported_seq":24307,"reported_epoch":389,"state":"active+clean","last_fresh":"2024-09-18T08:18:03.650889+0000","last_change":"2024-09-18T08:17:42.087296+0000","last_active":"2024-09-18T08:18:03.650889+0000","last_peered":"2024-09-18T08:18:03.650889+0000","last_clean":"2024-09-18T08:18:03.650889+0000","last_became_active":"2024-09-18T08:16:53.541405+0000","last_became_peered":"2024-09-18T08:16:53.541405+0000","last_unstale":"2024-09-18T08:18:03.650889+0000","last_undegraded":"2024-09-18T08:18:03.650889+0000","last_fullsized":"2024-09-18T08:18:03.650889+0000","mapping_epoch":363,"log_start":"369'8317","ondisk_log_start":"369'8317","created":311,"last_epoch_clean":364,"parent":"0.0","parent_split_bits":0,"last_scrub":"381'10659","last_scrub_stamp":"2024-09-18T08:17:42.087203+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:14:43.412748+0000","last_clean_scrub_stamp":"2024-09-18T08:17:42.087203+0000","objects_scrubbed":1978,"log_size":3620,"log_dups_size":0,"ondisk_log_size":3620,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:19:00.210365+0000","scrub_duration":518,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":44761088,"num_objects":683,"num_object_clones":0,"num_object_copies":2049,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":683,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11937,"num_write_kb":403840,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6069,"num_bytes_recovered":397737984,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,1],"acting":[5,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.8","version":"271'27","reported_seq":156,"reported_epoch":389,"state":"active+clean","last_fresh":"2024-09-18T08:18:00.533539+0000","last_change":"2024-09-18T08:16:13.502022+0000","last_active":"2024-09-18T08:18:00.533539+0000","last_peered":"2024-09-18T08:18:00.533539+0000","last_clean":"2024-09-18T08:18:00.533539+0000","last_became_active":"2024-09-18T08:16:13.451720+0000","last_became_peered":"2024-09-18T08:16:13.451720+0000","last_unstale":"2024-09-18T08:18:00.533539+0000","last_undegraded":"2024-09-18T08:18:00.533539+0000","last_fullsized":"2024-09-18T08:18:00.533539+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"271'27","last_scrub_stamp":"2024-09-18T08:15:00.680764+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_clean_scrub_stamp":"2024-09-18T08:15:00.680764+0000","objects_scrubbed":7,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:18:59.697017+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":44,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":28,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,3],"acting":[5,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.f","version":"62'2","reported_seq":120,"reported_epoch":390,"state":"active+clean","last_fresh":"2024-09-18T08:18:04.437782+0000","last_change":"2024-09-18T08:17:49.718337+0000","last_active":"2024-09-18T08:18:04.437782+0000","last_peered":"2024-09-18T08:18:04.437782+0000","last_clean":"2024-09-18T08:18:04.437782+0000","last_became_active":"2024-09-18T08:16:13.450206+0000","last_became_peered":"2024-09-18T08:16:13.450206+0000","last_unstale":"2024-09-18T08:18:04.437782+0000","last_undegraded":"2024-09-18T08:18:04.437782+0000","last_fullsized":"2024-09-18T08:18:04.437782+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"62'2","last_scrub_stamp":"2024-09-18T08:17:49.718261+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:46.860092+0000","last_clean_scrub_stamp":"2024-09-18T08:17:49.718261+0000","objects_scrubbed":2,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:19:10.780602+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":92,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,5],"acting":[0,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.9","version":"349'17","reported_seq":553,"reported_epoch":389,"state":"active+clean","last_fresh":"2024-09-18T08:18:00.522419+0000","last_change":"2024-09-18T08:16:13.513889+0000","last_active":"2024-09-18T08:18:00.522419+0000","last_peered":"2024-09-18T08:18:00.522419+0000","last_clean":"2024-09-18T08:18:00.522419+0000","last_became_active":"2024-09-18T08:16:13.474474+0000","last_became_peered":"2024-09-18T08:16:13.474474+0000","last_unstale":"2024-09-18T08:18:00.522419+0000","last_undegraded":"2024-09-18T08:18:00.522419+0000","last_fullsized":"2024-09-18T08:18:00.522419+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"294'14","last_scrub_stamp":"2024-09-18T08:14:28.686640+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:50.886830+0000","last_clean_scrub_stamp":"2024-09-18T08:14:28.686640+0000","objects_scrubbed":1,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:18:56.713614+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,3],"acting":[1,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":117,"reported_epoch":389,"state":"active+clean","last_fresh":"2024-09-18T08:18:00.533937+0000","last_change":"2024-09-18T08:15:41.454432+0000","last_active":"2024-09-18T08:18:00.533937+0000","last_peered":"2024-09-18T08:18:00.533937+0000","last_clean":"2024-09-18T08:18:00.533937+0000","last_became_active":"2024-09-18T08:15:41.449889+0000","last_became_peered":"2024-09-18T08:15:41.449889+0000","last_unstale":"2024-09-18T08:18:00.533937+0000","last_undegraded":"2024-09-18T08:18:00.533937+0000","last_fullsized":"2024-09-18T08:18:00.533937+0000","mapping_epoch":336,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":337,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:15:13.161614+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_clean_scrub_stamp":"2024-09-18T08:15:13.161614+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":637,"reported_epoch":390,"state":"active+clean","last_fresh":"2024-09-18T08:18:04.437592+0000","last_change":"2024-09-18T08:15:41.448597+0000","last_active":"2024-09-18T08:18:04.437592+0000","last_peered":"2024-09-18T08:18:04.437592+0000","last_clean":"2024-09-18T08:18:04.437592+0000","last_became_active":"2024-09-18T08:15:41.446544+0000","last_became_peered":"2024-09-18T08:15:41.446544+0000","last_unstale":"2024-09-18T08:18:04.437592+0000","last_undegraded":"2024-09-18T08:18:04.437592+0000","last_fullsized":"2024-09-18T08:18:04.437592+0000","mapping_epoch":336,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":337,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:13:06.860320+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:46.860092+0000","last_clean_scrub_stamp":"2024-09-18T08:13:06.860320+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:19:00.797409+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.17","version":"271'12","reported_seq":759,"reported_epoch":389,"state":"active+clean","last_fresh":"2024-09-18T08:18:00.543548+0000","last_change":"2024-09-18T08:14:40.365750+0000","last_active":"2024-09-18T08:18:00.543548+0000","last_peered":"2024-09-18T08:18:00.543548+0000","last_clean":"2024-09-18T08:18:00.543548+0000","last_became_active":"2024-09-18T08:14:40.355751+0000","last_became_peered":"2024-09-18T08:14:40.355751+0000","last_unstale":"2024-09-18T08:18:00.543548+0000","last_undegraded":"2024-09-18T08:18:00.543548+0000","last_fullsized":"2024-09-18T08:18:00.543548+0000","mapping_epoch":308,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":309,"parent":"0.0","parent_split_bits":0,"last_scrub":"271'12","last_scrub_stamp":"2024-09-18T08:14:33.700931+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_clean_scrub_stamp":"2024-09-18T08:14:33.700931+0000","objects_scrubbed":3,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":18,"num_read_kb":15,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":116,"reported_epoch":389,"state":"active+clean","last_fresh":"2024-09-18T08:18:00.536319+0000","last_change":"2024-09-18T08:15:41.448984+0000","last_active":"2024-09-18T08:18:00.536319+0000","last_peered":"2024-09-18T08:18:00.536319+0000","last_clean":"2024-09-18T08:18:00.536319+0000","last_became_active":"2024-09-18T08:15:41.445332+0000","last_became_peered":"2024-09-18T08:15:41.445332+0000","last_unstale":"2024-09-18T08:18:00.536319+0000","last_undegraded":"2024-09-18T08:18:00.536319+0000","last_fullsized":"2024-09-18T08:18:00.536319+0000","mapping_epoch":336,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":337,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:15:07.747147+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:50.886830+0000","last_clean_scrub_stamp":"2024-09-18T08:15:07.747147+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:18:56.666371+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,1],"acting":[5,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":482,"reported_epoch":389,"state":"active+clean","last_fresh":"2024-09-18T08:18:00.535199+0000","last_change":"2024-09-18T08:16:13.453780+0000","last_active":"2024-09-18T08:18:00.535199+0000","last_peered":"2024-09-18T08:18:00.535199+0000","last_clean":"2024-09-18T08:18:00.535199+0000","last_became_active":"2024-09-18T08:16:13.452993+0000","last_became_peered":"2024-09-18T08:16:13.452993+0000","last_unstale":"2024-09-18T08:18:00.535199+0000","last_undegraded":"2024-09-18T08:18:00.535199+0000","last_fullsized":"2024-09-18T08:18:00.535199+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:12:54.661578+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_clean_scrub_stamp":"2024-09-18T08:12:54.661578+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:18:28.558405+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.16","version":"271'15","reported_seq":743,"reported_epoch":390,"state":"active+clean","last_fresh":"2024-09-18T08:18:04.438664+0000","last_change":"2024-09-18T08:17:32.508478+0000","last_active":"2024-09-18T08:18:04.438664+0000","last_peered":"2024-09-18T08:18:04.438664+0000","last_clean":"2024-09-18T08:18:04.438664+0000","last_became_active":"2024-09-18T08:14:05.578519+0000","last_became_peered":"2024-09-18T08:14:05.578519+0000","last_unstale":"2024-09-18T08:18:04.438664+0000","last_undegraded":"2024-09-18T08:18:04.438664+0000","last_fullsized":"2024-09-18T08:18:04.438664+0000","mapping_epoch":292,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":293,"parent":"0.0","parent_split_bits":0,"last_scrub":"271'15","last_scrub_stamp":"2024-09-18T08:17:32.508401+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_clean_scrub_stamp":"2024-09-18T08:17:32.508401+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:18:58.925490+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":352,"reported_epoch":390,"state":"active+clean","last_fresh":"2024-09-18T08:18:04.435727+0000","last_change":"2024-09-18T08:17:50.772102+0000","last_active":"2024-09-18T08:18:04.435727+0000","last_peered":"2024-09-18T08:18:04.435727+0000","last_clean":"2024-09-18T08:18:04.435727+0000","last_became_active":"2024-09-18T08:16:13.460000+0000","last_became_peered":"2024-09-18T08:16:13.460000+0000","last_unstale":"2024-09-18T08:18:04.435727+0000","last_undegraded":"2024-09-18T08:18:04.435727+0000","last_fullsized":"2024-09-18T08:18:04.435727+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:17:50.771903+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:46.860092+0000","last_clean_scrub_stamp":"2024-09-18T08:17:50.771903+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:18:57.832543+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,3],"acting":[0,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":712,"reported_epoch":389,"state":"active+clean","last_fresh":"2024-09-18T08:18:00.543445+0000","last_change":"2024-09-18T08:17:06.569916+0000","last_active":"2024-09-18T08:18:00.543445+0000","last_peered":"2024-09-18T08:18:00.543445+0000","last_clean":"2024-09-18T08:18:00.543445+0000","last_became_active":"2024-09-18T08:14:40.356531+0000","last_became_peered":"2024-09-18T08:14:40.356531+0000","last_unstale":"2024-09-18T08:18:00.543445+0000","last_undegraded":"2024-09-18T08:18:00.543445+0000","last_fullsized":"2024-09-18T08:18:00.543445+0000","mapping_epoch":308,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":309,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:17:06.569828+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:50.886830+0000","last_clean_scrub_stamp":"2024-09-18T08:17:06.569828+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:18:35.645500+0000","scrub_duration":31,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":721,"reported_epoch":389,"state":"active+clean","last_fresh":"2024-09-18T08:18:00.538116+0000","last_change":"2024-09-18T08:16:49.005144+0000","last_active":"2024-09-18T08:18:00.538116+0000","last_peered":"2024-09-18T08:18:00.538116+0000","last_clean":"2024-09-18T08:18:00.538116+0000","last_became_active":"2024-09-18T08:16:13.466171+0000","last_became_peered":"2024-09-18T08:16:13.466171+0000","last_unstale":"2024-09-18T08:18:00.538116+0000","last_undegraded":"2024-09-18T08:18:00.538116+0000","last_fullsized":"2024-09-18T08:18:00.538116+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:16:49.005064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_clean_scrub_stamp":"2024-09-18T08:16:49.005064+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:18:17.185217+0000","scrub_duration":22,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.15","version":"271'15","reported_seq":134,"reported_epoch":389,"state":"active+clean","last_fresh":"2024-09-18T08:18:00.535090+0000","last_change":"2024-09-18T08:17:32.638867+0000","last_active":"2024-09-18T08:18:00.535090+0000","last_peered":"2024-09-18T08:18:00.535090+0000","last_clean":"2024-09-18T08:18:00.535090+0000","last_became_active":"2024-09-18T08:15:41.447339+0000","last_became_peered":"2024-09-18T08:15:41.447339+0000","last_unstale":"2024-09-18T08:18:00.535090+0000","last_undegraded":"2024-09-18T08:18:00.535090+0000","last_fullsized":"2024-09-18T08:18:00.535090+0000","mapping_epoch":336,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":337,"parent":"0.0","parent_split_bits":0,"last_scrub":"271'15","last_scrub_stamp":"2024-09-18T08:17:32.638811+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_clean_scrub_stamp":"2024-09-18T08:17:32.638811+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:18:33.277962+0000","scrub_duration":20,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":25,"num_read_kb":19,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":725,"reported_epoch":390,"state":"active+clean","last_fresh":"2024-09-18T08:18:04.436913+0000","last_change":"2024-09-18T08:14:40.357339+0000","last_active":"2024-09-18T08:18:04.436913+0000","last_peered":"2024-09-18T08:18:04.436913+0000","last_clean":"2024-09-18T08:18:04.436913+0000","last_became_active":"2024-09-18T08:14:40.355709+0000","last_became_peered":"2024-09-18T08:14:40.355709+0000","last_unstale":"2024-09-18T08:18:04.436913+0000","last_undegraded":"2024-09-18T08:18:04.436913+0000","last_fullsized":"2024-09-18T08:18:04.436913+0000","mapping_epoch":308,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":309,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:14:28.097629+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:46.860092+0000","last_clean_scrub_stamp":"2024-09-18T08:14:28.097629+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1],"acting":[0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.14","version":"310'16","reported_seq":947,"reported_epoch":389,"state":"active+clean","last_fresh":"2024-09-18T08:18:00.544467+0000","last_change":"2024-09-18T08:17:39.424179+0000","last_active":"2024-09-18T08:18:00.544467+0000","last_peered":"2024-09-18T08:18:00.544467+0000","last_clean":"2024-09-18T08:18:00.544467+0000","last_became_active":"2024-09-18T08:14:40.358971+0000","last_became_peered":"2024-09-18T08:14:40.358971+0000","last_unstale":"2024-09-18T08:18:00.544467+0000","last_undegraded":"2024-09-18T08:18:00.544467+0000","last_fullsized":"2024-09-18T08:18:00.544467+0000","mapping_epoch":308,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":309,"parent":"0.0","parent_split_bits":0,"last_scrub":"310'16","last_scrub_stamp":"2024-09-18T08:17:39.424098+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:50.886830+0000","last_clean_scrub_stamp":"2024-09-18T08:17:39.424098+0000","objects_scrubbed":1,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:19:08.498783+0000","scrub_duration":16,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,1],"acting":[3,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":438,"reported_epoch":389,"state":"active+clean","last_fresh":"2024-09-18T08:18:00.539560+0000","last_change":"2024-09-18T08:16:51.923862+0000","last_active":"2024-09-18T08:18:00.539560+0000","last_peered":"2024-09-18T08:18:00.539560+0000","last_clean":"2024-09-18T08:18:00.539560+0000","last_became_active":"2024-09-18T08:16:13.467534+0000","last_became_peered":"2024-09-18T08:16:13.467534+0000","last_unstale":"2024-09-18T08:18:00.539560+0000","last_undegraded":"2024-09-18T08:18:00.539560+0000","last_fullsized":"2024-09-18T08:18:00.539560+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:16:51.923777+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_clean_scrub_stamp":"2024-09-18T08:16:51.923777+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,0],"acting":[2,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.14","version":"271'20","reported_seq":773,"reported_epoch":389,"state":"active+clean","last_fresh":"2024-09-18T08:18:00.544940+0000","last_change":"2024-09-18T08:15:52.819894+0000","last_active":"2024-09-18T08:18:00.544940+0000","last_peered":"2024-09-18T08:18:00.544940+0000","last_clean":"2024-09-18T08:18:00.544940+0000","last_became_active":"2024-09-18T08:14:40.358983+0000","last_became_peered":"2024-09-18T08:14:40.358983+0000","last_unstale":"2024-09-18T08:18:00.544940+0000","last_undegraded":"2024-09-18T08:18:00.544940+0000","last_fullsized":"2024-09-18T08:18:00.544940+0000","mapping_epoch":308,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":309,"parent":"0.0","parent_split_bits":0,"last_scrub":"271'20","last_scrub_stamp":"2024-09-18T08:15:52.819801+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_clean_scrub_stamp":"2024-09-18T08:15:52.819801+0000","objects_scrubbed":5,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":16,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":25,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":376,"reported_epoch":389,"state":"active+clean","last_fresh":"2024-09-18T08:18:00.539807+0000","last_change":"2024-09-18T08:17:30.909075+0000","last_active":"2024-09-18T08:18:00.539807+0000","last_peered":"2024-09-18T08:18:00.539807+0000","last_clean":"2024-09-18T08:18:00.539807+0000","last_became_active":"2024-09-18T08:16:13.461199+0000","last_became_peered":"2024-09-18T08:16:13.461199+0000","last_unstale":"2024-09-18T08:18:00.539807+0000","last_undegraded":"2024-09-18T08:18:00.539807+0000","last_fullsized":"2024-09-18T08:18:00.539807+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:17:30.908835+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:46.860092+0000","last_clean_scrub_stamp":"2024-09-18T08:17:30.908835+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:18:34.331728+0000","scrub_duration":22,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,0],"acting":[2,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.15","version":"338'12","reported_seq":149,"reported_epoch":389,"state":"active+clean","last_fresh":"2024-09-18T08:18:00.534862+0000","last_change":"2024-09-18T08:15:41.489414+0000","last_active":"2024-09-18T08:18:00.534862+0000","last_peered":"2024-09-18T08:18:00.534862+0000","last_clean":"2024-09-18T08:18:00.534862+0000","last_became_active":"2024-09-18T08:15:41.455243+0000","last_became_peered":"2024-09-18T08:15:41.455243+0000","last_unstale":"2024-09-18T08:18:00.534862+0000","last_undegraded":"2024-09-18T08:18:00.534862+0000","last_fullsized":"2024-09-18T08:18:00.534862+0000","mapping_epoch":336,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":337,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'8","last_scrub_stamp":"2024-09-18T08:10:54.148986+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:50.886830+0000","last_clean_scrub_stamp":"2024-09-18T08:10:54.148986+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:18:23.530255+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":708,"reported_epoch":390,"state":"active+clean","last_fresh":"2024-09-18T08:18:04.438218+0000","last_change":"2024-09-18T08:17:39.513217+0000","last_active":"2024-09-18T08:18:04.438218+0000","last_peered":"2024-09-18T08:18:04.438218+0000","last_clean":"2024-09-18T08:18:04.438218+0000","last_became_active":"2024-09-18T08:16:13.472297+0000","last_became_peered":"2024-09-18T08:16:13.472297+0000","last_unstale":"2024-09-18T08:18:04.438218+0000","last_undegraded":"2024-09-18T08:18:04.438218+0000","last_fullsized":"2024-09-18T08:18:04.438218+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:17:39.513160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_clean_scrub_stamp":"2024-09-18T08:17:39.513160+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:18:41.265902+0000","scrub_duration":31,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,5],"acting":[0,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.13","version":"271'19","reported_seq":413,"reported_epoch":389,"state":"active+clean","last_fresh":"2024-09-18T08:18:00.526735+0000","last_change":"2024-09-18T08:17:39.665796+0000","last_active":"2024-09-18T08:18:00.526735+0000","last_peered":"2024-09-18T08:18:00.526735+0000","last_clean":"2024-09-18T08:18:00.526735+0000","last_became_active":"2024-09-18T08:16:13.454640+0000","last_became_peered":"2024-09-18T08:16:13.454640+0000","last_unstale":"2024-09-18T08:18:00.526735+0000","last_undegraded":"2024-09-18T08:18:00.526735+0000","last_fullsized":"2024-09-18T08:18:00.526735+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"271'19","last_scrub_stamp":"2024-09-18T08:17:39.665588+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_clean_scrub_stamp":"2024-09-18T08:17:39.665588+0000","objects_scrubbed":5,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:18:55.676655+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":31,"num_read_kb":24,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,5],"acting":[1,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":342,"reported_epoch":389,"state":"active+clean","last_fresh":"2024-09-18T08:18:00.537776+0000","last_change":"2024-09-18T08:16:48.007441+0000","last_active":"2024-09-18T08:18:00.537776+0000","last_peered":"2024-09-18T08:18:00.537776+0000","last_clean":"2024-09-18T08:18:00.537776+0000","last_became_active":"2024-09-18T08:16:13.457932+0000","last_became_peered":"2024-09-18T08:16:13.457932+0000","last_unstale":"2024-09-18T08:18:00.537776+0000","last_undegraded":"2024-09-18T08:18:00.537776+0000","last_fullsized":"2024-09-18T08:18:00.537776+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:16:48.007357+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:46.860092+0000","last_clean_scrub_stamp":"2024-09-18T08:16:48.007357+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":27,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,0],"acting":[2,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":590,"reported_epoch":389,"state":"active+clean","last_fresh":"2024-09-18T08:18:00.525938+0000","last_change":"2024-09-18T08:15:55.335233+0000","last_active":"2024-09-18T08:18:00.525938+0000","last_peered":"2024-09-18T08:18:00.525938+0000","last_clean":"2024-09-18T08:18:00.525938+0000","last_became_active":"2024-09-18T08:15:41.438048+0000","last_became_peered":"2024-09-18T08:15:41.438048+0000","last_unstale":"2024-09-18T08:18:00.525938+0000","last_undegraded":"2024-09-18T08:18:00.525938+0000","last_fullsized":"2024-09-18T08:18:00.525938+0000","mapping_epoch":336,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":337,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:15:55.334983+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:50.886830+0000","last_clean_scrub_stamp":"2024-09-18T08:15:55.334983+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":723,"reported_epoch":389,"state":"active+clean","last_fresh":"2024-09-18T08:18:00.536843+0000","last_change":"2024-09-18T08:17:37.394735+0000","last_active":"2024-09-18T08:18:00.536843+0000","last_peered":"2024-09-18T08:18:00.536843+0000","last_clean":"2024-09-18T08:18:00.536843+0000","last_became_active":"2024-09-18T08:15:41.464961+0000","last_became_peered":"2024-09-18T08:15:41.464961+0000","last_unstale":"2024-09-18T08:18:00.536843+0000","last_undegraded":"2024-09-18T08:18:00.536843+0000","last_fullsized":"2024-09-18T08:18:00.536843+0000","mapping_epoch":336,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":337,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:17:37.394532+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_clean_scrub_stamp":"2024-09-18T08:17:37.394532+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:18:46.957682+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.12","version":"271'15","reported_seq":578,"reported_epoch":389,"state":"active+clean","last_fresh":"2024-09-18T08:18:00.526320+0000","last_change":"2024-09-18T08:15:52.304936+0000","last_active":"2024-09-18T08:18:00.526320+0000","last_peered":"2024-09-18T08:18:00.526320+0000","last_clean":"2024-09-18T08:18:00.526320+0000","last_became_active":"2024-09-18T08:14:39.416284+0000","last_became_peered":"2024-09-18T08:14:39.416284+0000","last_unstale":"2024-09-18T08:18:00.526320+0000","last_undegraded":"2024-09-18T08:18:00.526320+0000","last_fullsized":"2024-09-18T08:18:00.526320+0000","mapping_epoch":307,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":308,"parent":"0.0","parent_split_bits":0,"last_scrub":"271'15","last_scrub_stamp":"2024-09-18T08:15:52.304832+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_clean_scrub_stamp":"2024-09-18T08:15:52.304832+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":25,"num_read_kb":19,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":509,"reported_epoch":389,"state":"active+clean","last_fresh":"2024-09-18T08:18:00.547124+0000","last_change":"2024-09-18T08:16:13.476132+0000","last_active":"2024-09-18T08:18:00.547124+0000","last_peered":"2024-09-18T08:18:00.547124+0000","last_clean":"2024-09-18T08:18:00.547124+0000","last_became_active":"2024-09-18T08:16:13.475647+0000","last_became_peered":"2024-09-18T08:16:13.475647+0000","last_unstale":"2024-09-18T08:18:00.547124+0000","last_undegraded":"2024-09-18T08:18:00.547124+0000","last_fullsized":"2024-09-18T08:18:00.547124+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:15:58.978411+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:46.860092+0000","last_clean_scrub_stamp":"2024-09-18T08:15:58.978411+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":18,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":699,"reported_epoch":389,"state":"active+clean","last_fresh":"2024-09-18T08:18:00.543825+0000","last_change":"2024-09-18T08:14:39.403741+0000","last_active":"2024-09-18T08:18:00.543825+0000","last_peered":"2024-09-18T08:18:00.543825+0000","last_clean":"2024-09-18T08:18:00.543825+0000","last_became_active":"2024-09-18T08:14:39.402406+0000","last_became_peered":"2024-09-18T08:14:39.402406+0000","last_unstale":"2024-09-18T08:18:00.543825+0000","last_undegraded":"2024-09-18T08:18:00.543825+0000","last_fullsized":"2024-09-18T08:18:00.543825+0000","mapping_epoch":307,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":308,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:11:30.302179+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:50.886830+0000","last_clean_scrub_stamp":"2024-09-18T08:11:30.302179+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:18:20.349750+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.10","version":"69'1","reported_seq":715,"reported_epoch":390,"state":"active+clean","last_fresh":"2024-09-18T08:18:04.435764+0000","last_change":"2024-09-18T08:15:41.480106+0000","last_active":"2024-09-18T08:18:04.435764+0000","last_peered":"2024-09-18T08:18:04.435764+0000","last_clean":"2024-09-18T08:18:04.435764+0000","last_became_active":"2024-09-18T08:15:41.444085+0000","last_became_peered":"2024-09-18T08:15:41.444085+0000","last_unstale":"2024-09-18T08:18:04.435764+0000","last_undegraded":"2024-09-18T08:18:04.435764+0000","last_fullsized":"2024-09-18T08:18:04.435764+0000","mapping_epoch":336,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":337,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'1","last_scrub_stamp":"2024-09-18T08:15:00.219678+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_clean_scrub_stamp":"2024-09-18T08:15:00.219678+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":26,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.11","version":"271'19","reported_seq":842,"reported_epoch":389,"state":"active+clean","last_fresh":"2024-09-18T08:18:00.536589+0000","last_change":"2024-09-18T08:15:41.491797+0000","last_active":"2024-09-18T08:18:00.536589+0000","last_peered":"2024-09-18T08:18:00.536589+0000","last_clean":"2024-09-18T08:18:00.536589+0000","last_became_active":"2024-09-18T08:15:41.465631+0000","last_became_peered":"2024-09-18T08:15:41.465631+0000","last_unstale":"2024-09-18T08:18:00.536589+0000","last_undegraded":"2024-09-18T08:18:00.536589+0000","last_fullsized":"2024-09-18T08:18:00.536589+0000","mapping_epoch":336,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":337,"parent":"0.0","parent_split_bits":0,"last_scrub":"271'19","last_scrub_stamp":"2024-09-18T08:14:38.809490+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_clean_scrub_stamp":"2024-09-18T08:14:38.809490+0000","objects_scrubbed":5,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":32,"num_read_kb":25,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":550,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":116,"reported_epoch":389,"state":"active+clean","last_fresh":"2024-09-18T08:18:00.536318+0000","last_change":"2024-09-18T08:15:41.449283+0000","last_active":"2024-09-18T08:18:00.536318+0000","last_peered":"2024-09-18T08:18:00.536318+0000","last_clean":"2024-09-18T08:18:00.536318+0000","last_became_active":"2024-09-18T08:15:41.445583+0000","last_became_peered":"2024-09-18T08:15:41.445583+0000","last_unstale":"2024-09-18T08:18:00.536318+0000","last_undegraded":"2024-09-18T08:18:00.536318+0000","last_fullsized":"2024-09-18T08:18:00.536318+0000","mapping_epoch":336,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":337,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:10:53.145315+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:46.860092+0000","last_clean_scrub_stamp":"2024-09-18T08:10:53.145315+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:18:14.386231+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":384,"reported_epoch":390,"state":"active+clean","last_fresh":"2024-09-18T08:18:04.435629+0000","last_change":"2024-09-18T08:16:13.444745+0000","last_active":"2024-09-18T08:18:04.435629+0000","last_peered":"2024-09-18T08:18:04.435629+0000","last_clean":"2024-09-18T08:18:04.435629+0000","last_became_active":"2024-09-18T08:16:13.443940+0000","last_became_peered":"2024-09-18T08:16:13.443940+0000","last_unstale":"2024-09-18T08:18:04.435629+0000","last_undegraded":"2024-09-18T08:18:04.435629+0000","last_fullsized":"2024-09-18T08:18:04.435629+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:14:37.743856+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:50.886830+0000","last_clean_scrub_stamp":"2024-09-18T08:14:37.743856+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":698,"reported_epoch":389,"state":"active+clean","last_fresh":"2024-09-18T08:18:00.544090+0000","last_change":"2024-09-18T08:11:29.344444+0000","last_active":"2024-09-18T08:18:00.544090+0000","last_peered":"2024-09-18T08:18:00.544090+0000","last_clean":"2024-09-18T08:18:00.544090+0000","last_became_active":"2024-09-18T08:06:24.128582+0000","last_became_peered":"2024-09-18T08:06:24.128582+0000","last_unstale":"2024-09-18T08:18:00.544090+0000","last_undegraded":"2024-09-18T08:18:00.544090+0000","last_fullsized":"2024-09-18T08:18:00.544090+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:11:29.344382+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_clean_scrub_stamp":"2024-09-18T08:11:29.344382+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:18:19.388633+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.10","version":"271'8","reported_seq":756,"reported_epoch":389,"state":"active+clean","last_fresh":"2024-09-18T08:18:00.532270+0000","last_change":"2024-09-18T08:15:50.790606+0000","last_active":"2024-09-18T08:18:00.532270+0000","last_peered":"2024-09-18T08:18:00.532270+0000","last_clean":"2024-09-18T08:18:00.532270+0000","last_became_active":"2024-09-18T08:14:39.404257+0000","last_became_peered":"2024-09-18T08:14:39.404257+0000","last_unstale":"2024-09-18T08:18:00.532270+0000","last_undegraded":"2024-09-18T08:18:00.532270+0000","last_fullsized":"2024-09-18T08:18:00.532270+0000","mapping_epoch":307,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":308,"parent":"0.0","parent_split_bits":0,"last_scrub":"271'8","last_scrub_stamp":"2024-09-18T08:15:50.790525+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_clean_scrub_stamp":"2024-09-18T08:15:50.790525+0000","objects_scrubbed":2,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":12,"num_read_kb":10,"num_write":8,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":727,"reported_epoch":390,"state":"active+clean","last_fresh":"2024-09-18T08:18:04.438113+0000","last_change":"2024-09-18T08:15:41.467281+0000","last_active":"2024-09-18T08:18:04.438113+0000","last_peered":"2024-09-18T08:18:04.438113+0000","last_clean":"2024-09-18T08:18:04.438113+0000","last_became_active":"2024-09-18T08:15:41.466964+0000","last_became_peered":"2024-09-18T08:15:41.466964+0000","last_unstale":"2024-09-18T08:18:04.438113+0000","last_undegraded":"2024-09-18T08:18:04.438113+0000","last_fullsized":"2024-09-18T08:18:04.438113+0000","mapping_epoch":336,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":337,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:14:33.064199+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:46.860092+0000","last_clean_scrub_stamp":"2024-09-18T08:14:33.064199+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":344,"reported_epoch":390,"state":"active+clean","last_fresh":"2024-09-18T08:18:04.435438+0000","last_change":"2024-09-18T08:16:13.453084+0000","last_active":"2024-09-18T08:18:04.435438+0000","last_peered":"2024-09-18T08:18:04.435438+0000","last_clean":"2024-09-18T08:18:04.435438+0000","last_became_active":"2024-09-18T08:16:13.452169+0000","last_became_peered":"2024-09-18T08:16:13.452169+0000","last_unstale":"2024-09-18T08:18:04.435438+0000","last_undegraded":"2024-09-18T08:18:04.435438+0000","last_fullsized":"2024-09-18T08:18:04.435438+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:13:32.640669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:50.886830+0000","last_clean_scrub_stamp":"2024-09-18T08:13:32.640669+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":20,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,1],"acting":[0,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":514,"reported_epoch":389,"state":"active+clean","last_fresh":"2024-09-18T08:18:00.537507+0000","last_change":"2024-09-18T08:17:31.928942+0000","last_active":"2024-09-18T08:18:00.537507+0000","last_peered":"2024-09-18T08:18:00.537507+0000","last_clean":"2024-09-18T08:18:00.537507+0000","last_became_active":"2024-09-18T08:16:13.491755+0000","last_became_peered":"2024-09-18T08:16:13.491755+0000","last_unstale":"2024-09-18T08:18:00.537507+0000","last_undegraded":"2024-09-18T08:18:00.537507+0000","last_fullsized":"2024-09-18T08:18:00.537507+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:17:31.928876+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_clean_scrub_stamp":"2024-09-18T08:17:31.928876+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:18:33.514219+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,1],"acting":[2,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":562,"reported_epoch":389,"state":"active+clean","last_fresh":"2024-09-18T08:18:00.522364+0000","last_change":"2024-09-18T08:16:13.468068+0000","last_active":"2024-09-18T08:18:00.522364+0000","last_peered":"2024-09-18T08:18:00.522364+0000","last_clean":"2024-09-18T08:18:00.522364+0000","last_became_active":"2024-09-18T08:16:13.466812+0000","last_became_peered":"2024-09-18T08:16:13.466812+0000","last_unstale":"2024-09-18T08:18:00.522364+0000","last_undegraded":"2024-09-18T08:18:00.522364+0000","last_fullsized":"2024-09-18T08:18:00.522364+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:15:45.297240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_clean_scrub_stamp":"2024-09-18T08:15:45.297240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,2],"acting":[1,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":707,"reported_epoch":389,"state":"active+clean","last_fresh":"2024-09-18T08:18:00.533245+0000","last_change":"2024-09-18T08:15:43.836627+0000","last_active":"2024-09-18T08:18:00.533245+0000","last_peered":"2024-09-18T08:18:00.533245+0000","last_clean":"2024-09-18T08:18:00.533245+0000","last_became_active":"2024-09-18T08:14:39.405347+0000","last_became_peered":"2024-09-18T08:14:39.405347+0000","last_unstale":"2024-09-18T08:18:00.533245+0000","last_undegraded":"2024-09-18T08:18:00.533245+0000","last_fullsized":"2024-09-18T08:18:00.533245+0000","mapping_epoch":307,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":308,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:15:43.836549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:46.860092+0000","last_clean_scrub_stamp":"2024-09-18T08:15:43.836549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":20,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"271'19","reported_seq":129,"reported_epoch":389,"state":"active+clean","last_fresh":"2024-09-18T08:18:00.535890+0000","last_change":"2024-09-18T08:15:41.472787+0000","last_active":"2024-09-18T08:18:00.535890+0000","last_peered":"2024-09-18T08:18:00.535890+0000","last_clean":"2024-09-18T08:18:00.535890+0000","last_became_active":"2024-09-18T08:15:41.445198+0000","last_became_peered":"2024-09-18T08:15:41.445198+0000","last_unstale":"2024-09-18T08:18:00.535890+0000","last_undegraded":"2024-09-18T08:18:00.535890+0000","last_fullsized":"2024-09-18T08:18:00.535890+0000","mapping_epoch":336,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":337,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'11","last_scrub_stamp":"2024-09-18T08:10:50.208543+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_clean_scrub_stamp":"2024-09-18T08:10:50.208543+0000","objects_scrubbed":5,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:18:15.381752+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":31,"num_read_kb":24,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,1],"acting":[5,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":724,"reported_epoch":390,"state":"active+clean","last_fresh":"2024-09-18T08:18:04.438360+0000","last_change":"2024-09-18T08:14:40.358776+0000","last_active":"2024-09-18T08:18:04.438360+0000","last_peered":"2024-09-18T08:18:04.438360+0000","last_clean":"2024-09-18T08:18:04.438360+0000","last_became_active":"2024-09-18T08:14:40.358457+0000","last_became_peered":"2024-09-18T08:14:40.358457+0000","last_unstale":"2024-09-18T08:18:04.438360+0000","last_undegraded":"2024-09-18T08:18:04.438360+0000","last_fullsized":"2024-09-18T08:18:04.438360+0000","mapping_epoch":308,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":309,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:12:31.664456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:50.886830+0000","last_clean_scrub_stamp":"2024-09-18T08:12:31.664456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:18:52.758648+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":10,"num_pg":16,"stat_sum":{"num_bytes":632815640,"num_objects":9657,"num_object_clones":0,"num_object_copies":28971,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9657,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":193779,"num_write_kb":6509889,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":62798,"num_bytes_recovered":4115464216,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":3554557952,"data_stored":3554541664,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":61647,"ondisk_log_size":61647,"up":48,"acting":48,"num_store_stats":7},{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":416,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":832,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":26,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":116,"ondisk_log_size":116,"up":96,"acting":96,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":178,"num_object_clones":0,"num_object_copies":534,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":178,"num_whiteouts":0,"num_read":1104,"num_read_kb":861,"num_write":703,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":447,"num_bytes_recovered":8338,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":499712,"data_stored":13350,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3639,"internal_metadata":0},"log_size":688,"ondisk_log_size":688,"up":96,"acting":96,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":1613,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":14,"num_read_kb":14,"num_write":10,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":4563,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":102400,"data_stored":6108,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":6,"ondisk_log_size":6,"up":96,"acting":96,"num_store_stats":8},{"poolid":2,"num_pg":3,"stat_sum":{"num_bytes":408,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":375,"num_read_kb":370,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":408,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1613,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":8,"ondisk_log_size":8,"up":9,"acting":9,"num_store_stats":7},{"poolid":1,"num_pg":11,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1245728,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":869,"ondisk_log_size":869,"up":33,"acting":33,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":0,"seq":240518168763,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":47,"seq":201863463105,"num_pgs":74,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":841560,"kb_used_data":350856,"kb_used_omap":1,"kb_used_meta":490686,"kb_avail":92903592,"statfs":{"total":95995035648,"available":95133278208,"internally_reserved":0,"allocated":359276544,"data_stored":358809361,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":502462918},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":39,"seq":167503724743,"num_pgs":83,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1066920,"kb_used_data":295768,"kb_used_omap":1,"kb_used_meta":771134,"kb_avail":92678232,"statfs":{"total":95995035648,"available":94902509568,"internally_reserved":0,"allocated":302866432,"data_stored":302385058,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":789641670},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":0,"seq":137438953676,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":26,"seq":111669149907,"num_pgs":87,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1107520,"kb_used_data":382192,"kb_used_omap":1,"kb_used_meta":725310,"kb_avail":92637632,"statfs":{"total":95995035648,"available":94860935168,"internally_reserved":0,"allocated":391364608,"data_stored":390858580,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":742717896},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411545,"num_pgs":65,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":911704,"kb_used_data":280712,"kb_used_omap":1,"kb_used_meta":630910,"kb_avail":92833448,"statfs":{"total":95995035648,"available":95061450752,"internally_reserved":0,"allocated":287449088,"data_stored":286983073,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":646052293},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":307,"seq":1318554959914,"num_pgs":84,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":454804,"kb_used_data":296128,"kb_used_omap":7,"kb_used_meta":158648,"kb_avail":93290348,"statfs":{"total":95995035648,"available":95529316352,"internally_reserved":0,"allocated":303235072,"data_stored":302713244,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":8079,"internal_metadata":162455665},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738596,"num_pgs":81,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1015472,"kb_used_data":388700,"kb_used_omap":1,"kb_used_meta":626750,"kb_avail":92729680,"statfs":{"total":95995035648,"available":94955192320,"internally_reserved":0,"allocated":398028800,"data_stored":397549274,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":641792453},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":266240,"data_stored":262688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1131,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1613,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":436,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":46,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1177,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1567,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":46,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1474,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":94208,"data_stored":2448,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3639,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":1568,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":94208,"data_stored":2492,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":1584,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1430,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":1694,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":660,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":403,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":403,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":416,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":383913984,"data_stored":383909912,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":277676032,"data_stored":277676032,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":280428544,"data_stored":280428544,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":348065792,"data_stored":348061720,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":1626017792,"data_stored":1626013720,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":299171840,"data_stored":299171840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":339283968,"data_stored":339279896,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T08:18:09.894 INFO:tasks.thrashosds.thrasher:Setting 3.c to [3, 2, 6] 2024-09-18T08:18:09.894 INFO:tasks.thrashosds.thrasher:cmd ['osd', 'pg-upmap', '3.c', '3', '2', '6'] 2024-09-18T08:18:09.895 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid 779349ca-7593-11ef-bceb-c7b262605968 -- ceph osd pg-upmap 3.c 3 2 6 2024-09-18T08:18:09.895 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:18:09.896 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:18:09.896 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2' 2024-09-18T08:18:09.905 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:18:09 smithi050 ceph-mon[32371]: from='client.? 172.21.15.7:0/103380881' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-18T08:18:09.905 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:18:09 smithi050 ceph-mon[32371]: osdmap e392: 8 total, 8 up, 6 in 2024-09-18T08:18:09.905 INFO:journalctl@ceph.iscsi.iscsi.a.smithi050.stdout:Sep 18 08:18:09 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-iscsi-iscsi-a[57782]: debug there is no tcmu-runner data available 2024-09-18T08:18:09.997 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4 2024-09-18T08:18:10.023 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:18:10.024 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:18:10.024 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4' 2024-09-18T08:18:10.125 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1 2024-09-18T08:18:10.194 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:18:10.195 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:18:10.195 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1' 2024-09-18T08:18:10.296 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6 2024-09-18T08:18:10.320 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:18:10.321 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:18:10.321 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6' 2024-09-18T08:18:10.370 INFO:teuthology.orchestra.run.smithi007.stdout:{ 2024-09-18T08:18:10.370 INFO:teuthology.orchestra.run.smithi007.stdout: "ops": [ 2024-09-18T08:18:10.370 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:18:10.370 INFO:teuthology.orchestra.run.smithi007.stdout: "description": "osd_repop_reply(client.15966.0:200476 10.e e392/386)", 2024-09-18T08:18:10.370 INFO:teuthology.orchestra.run.smithi007.stdout: "initiated_at": "2024-09-18T08:18:10.361922+0000", 2024-09-18T08:18:10.370 INFO:teuthology.orchestra.run.smithi007.stdout: "age": 0.0058261119999999996, 2024-09-18T08:18:10.371 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.0058476800000000001, 2024-09-18T08:18:10.371 INFO:teuthology.orchestra.run.smithi007.stdout: "continuous": false, 2024-09-18T08:18:10.371 INFO:teuthology.orchestra.run.smithi007.stdout: "type_data": { 2024-09-18T08:18:10.371 INFO:teuthology.orchestra.run.smithi007.stdout: "flag_point": "queued for pg", 2024-09-18T08:18:10.371 INFO:teuthology.orchestra.run.smithi007.stdout: "events": [ 2024-09-18T08:18:10.371 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:18:10.371 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "initiated", 2024-09-18T08:18:10.371 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:18:10.361922+0000", 2024-09-18T08:18:10.371 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-09-18T08:18:10.371 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:18:10.371 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:18:10.371 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "header_read", 2024-09-18T08:18:10.371 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:18:10.361922+0000", 2024-09-18T08:18:10.372 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-09-18T08:18:10.372 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:18:10.372 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:18:10.372 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "throttled", 2024-09-18T08:18:10.372 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:18:10.361923+0000", 2024-09-18T08:18:10.372 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 5.8999999999999996e-07 2024-09-18T08:18:10.372 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:18:10.372 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:18:10.372 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "all_read", 2024-09-18T08:18:10.372 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:18:10.361926+0000", 2024-09-18T08:18:10.373 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 3.6950000000000002e-06 2024-09-18T08:18:10.373 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:18:10.373 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:18:10.373 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "dispatched", 2024-09-18T08:18:10.373 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:18:10.361934+0000", 2024-09-18T08:18:10.373 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 8.1359999999999997e-06 2024-09-18T08:18:10.373 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:18:10.373 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:18:10.373 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "queued_for_pg", 2024-09-18T08:18:10.373 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:18:10.361940+0000", 2024-09-18T08:18:10.373 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 6.0519999999999998e-06 2024-09-18T08:18:10.373 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-09-18T08:18:10.373 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-09-18T08:18:10.373 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-09-18T08:18:10.374 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:18:10.374 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:18:10.374 INFO:teuthology.orchestra.run.smithi007.stdout: "description": "osd_op(client.15966.0:200486 10.e 10.c052715e (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e392)", 2024-09-18T08:18:10.374 INFO:teuthology.orchestra.run.smithi007.stdout: "initiated_at": "2024-09-18T08:18:10.364358+0000", 2024-09-18T08:18:10.374 INFO:teuthology.orchestra.run.smithi007.stdout: "age": 0.0033905609999999998, 2024-09-18T08:18:10.374 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.003458062, 2024-09-18T08:18:10.374 INFO:teuthology.orchestra.run.smithi007.stdout: "continuous": false, 2024-09-18T08:18:10.374 INFO:teuthology.orchestra.run.smithi007.stdout: "type_data": { 2024-09-18T08:18:10.374 INFO:teuthology.orchestra.run.smithi007.stdout: "flag_point": "queued for pg", 2024-09-18T08:18:10.374 INFO:teuthology.orchestra.run.smithi007.stdout: "client_info": { 2024-09-18T08:18:10.374 INFO:teuthology.orchestra.run.smithi007.stdout: "client": "client.15966", 2024-09-18T08:18:10.374 INFO:teuthology.orchestra.run.smithi007.stdout: "client_addr": "172.21.15.7:0/529784850", 2024-09-18T08:18:10.374 INFO:teuthology.orchestra.run.smithi007.stdout: "tid": 200486 2024-09-18T08:18:10.375 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:18:10.375 INFO:teuthology.orchestra.run.smithi007.stdout: "events": [ 2024-09-18T08:18:10.375 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:18:10.375 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "initiated", 2024-09-18T08:18:10.375 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:18:10.364358+0000", 2024-09-18T08:18:10.375 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-09-18T08:18:10.375 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:18:10.375 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:18:10.375 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "header_read", 2024-09-18T08:18:10.375 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:18:10.364358+0000", 2024-09-18T08:18:10.375 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-09-18T08:18:10.375 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:18:10.375 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:18:10.375 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "throttled", 2024-09-18T08:18:10.376 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:18:10.364359+0000", 2024-09-18T08:18:10.376 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.606e-06 2024-09-18T08:18:10.376 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:18:10.376 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:18:10.376 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "all_read", 2024-09-18T08:18:10.376 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:18:10.364366+0000", 2024-09-18T08:18:10.376 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 6.6590000000000001e-06 2024-09-18T08:18:10.376 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:18:10.376 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:18:10.376 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "dispatched", 2024-09-18T08:18:10.376 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:18:10.364378+0000", 2024-09-18T08:18:10.376 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.2363e-05 2024-09-18T08:18:10.376 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:18:10.377 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:18:10.377 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "queued_for_pg", 2024-09-18T08:18:10.377 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:18:10.364386+0000", 2024-09-18T08:18:10.377 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 7.9200000000000004e-06 2024-09-18T08:18:10.377 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-09-18T08:18:10.377 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-09-18T08:18:10.377 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-09-18T08:18:10.377 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:18:10.377 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:18:10.377 INFO:teuthology.orchestra.run.smithi007.stdout: "description": "osd_repop(client.15966.0:200480 10.f e392/311 10:ff42a053:::benchmark_data_smithi007_148834_object98760:head v 392'12708, mlcod=392'12706)", 2024-09-18T08:18:10.377 INFO:teuthology.orchestra.run.smithi007.stdout: "initiated_at": "2024-09-18T08:18:10.366043+0000", 2024-09-18T08:18:10.377 INFO:teuthology.orchestra.run.smithi007.stdout: "age": 0.0017046190000000001, 2024-09-18T08:18:10.377 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.0018301560000000001, 2024-09-18T08:18:10.378 INFO:teuthology.orchestra.run.smithi007.stdout: "continuous": false, 2024-09-18T08:18:10.378 INFO:teuthology.orchestra.run.smithi007.stdout: "type_data": { 2024-09-18T08:18:10.378 INFO:teuthology.orchestra.run.smithi007.stdout: "flag_point": "started", 2024-09-18T08:18:10.378 INFO:teuthology.orchestra.run.smithi007.stdout: "events": [ 2024-09-18T08:18:10.378 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:18:10.378 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "initiated", 2024-09-18T08:18:10.378 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:18:10.366043+0000", 2024-09-18T08:18:10.378 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-09-18T08:18:10.378 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:18:10.378 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:18:10.378 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "header_read", 2024-09-18T08:18:10.378 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:18:10.366043+0000", 2024-09-18T08:18:10.378 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-09-18T08:18:10.378 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:18:10.379 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:18:10.379 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "throttled", 2024-09-18T08:18:10.379 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:18:10.366044+0000", 2024-09-18T08:18:10.379 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 7.7100000000000001e-07 2024-09-18T08:18:10.379 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:18:10.379 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:18:10.379 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "all_read", 2024-09-18T08:18:10.379 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:18:10.366050+0000", 2024-09-18T08:18:10.379 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 5.2920000000000003e-06 2024-09-18T08:18:10.379 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:18:10.379 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:18:10.379 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "dispatched", 2024-09-18T08:18:10.379 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:18:10.366058+0000", 2024-09-18T08:18:10.379 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 8.7900000000000005e-06 2024-09-18T08:18:10.380 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:18:10.380 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:18:10.380 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "queued_for_pg", 2024-09-18T08:18:10.380 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:18:10.366065+0000", 2024-09-18T08:18:10.380 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 6.793e-06 2024-09-18T08:18:10.380 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:18:10.380 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:18:10.380 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-09-18T08:18:10.380 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:18:10.366180+0000", 2024-09-18T08:18:10.380 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.000115029 2024-09-18T08:18:10.380 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:18:10.380 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:18:10.380 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "started", 2024-09-18T08:18:10.381 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:18:10.366198+0000", 2024-09-18T08:18:10.381 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.8233999999999999e-05 2024-09-18T08:18:10.381 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-09-18T08:18:10.381 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-09-18T08:18:10.381 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-09-18T08:18:10.381 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:18:10.381 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:18:10.381 INFO:teuthology.orchestra.run.smithi007.stdout: "description": "osd_op(client.15966.0:200489 10.e 10.858a639e (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e392)", 2024-09-18T08:18:10.381 INFO:teuthology.orchestra.run.smithi007.stdout: "initiated_at": "2024-09-18T08:18:10.366832+0000", 2024-09-18T08:18:10.381 INFO:teuthology.orchestra.run.smithi007.stdout: "age": 0.00091584299999999995, 2024-09-18T08:18:10.381 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.0010895620000000001, 2024-09-18T08:18:10.381 INFO:teuthology.orchestra.run.smithi007.stdout: "continuous": false, 2024-09-18T08:18:10.381 INFO:teuthology.orchestra.run.smithi007.stdout: "type_data": { 2024-09-18T08:18:10.382 INFO:teuthology.orchestra.run.smithi007.stdout: "flag_point": "queued for pg", 2024-09-18T08:18:10.382 INFO:teuthology.orchestra.run.smithi007.stdout: "client_info": { 2024-09-18T08:18:10.382 INFO:teuthology.orchestra.run.smithi007.stdout: "client": "client.15966", 2024-09-18T08:18:10.382 INFO:teuthology.orchestra.run.smithi007.stdout: "client_addr": "172.21.15.7:0/529784850", 2024-09-18T08:18:10.382 INFO:teuthology.orchestra.run.smithi007.stdout: "tid": 200489 2024-09-18T08:18:10.382 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:18:10.382 INFO:teuthology.orchestra.run.smithi007.stdout: "events": [ 2024-09-18T08:18:10.382 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:18:10.382 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "initiated", 2024-09-18T08:18:10.382 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:18:10.366832+0000", 2024-09-18T08:18:10.382 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-09-18T08:18:10.382 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:18:10.382 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:18:10.382 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "header_read", 2024-09-18T08:18:10.383 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:18:10.366832+0000", 2024-09-18T08:18:10.383 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-09-18T08:18:10.383 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:18:10.383 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:18:10.383 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "throttled", 2024-09-18T08:18:10.383 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:18:10.366833+0000", 2024-09-18T08:18:10.383 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.235e-06 2024-09-18T08:18:10.383 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:18:10.383 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:18:10.383 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "all_read", 2024-09-18T08:18:10.383 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:18:10.366838+0000", 2024-09-18T08:18:10.383 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 4.8110000000000002e-06 2024-09-18T08:18:10.383 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:18:10.383 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:18:10.384 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "dispatched", 2024-09-18T08:18:10.384 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:18:10.366895+0000", 2024-09-18T08:18:10.384 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 5.7198999999999999e-05 2024-09-18T08:18:10.384 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:18:10.384 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:18:10.384 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "queued_for_pg", 2024-09-18T08:18:10.384 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:18:10.366905+0000", 2024-09-18T08:18:10.384 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 9.6809999999999993e-06 2024-09-18T08:18:10.384 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-09-18T08:18:10.384 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-09-18T08:18:10.384 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-09-18T08:18:10.384 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:18:10.384 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:18:10.385 INFO:teuthology.orchestra.run.smithi007.stdout: "description": "osd_op(client.15966.0:200490 10.5 10:ae562584:::benchmark_data_smithi007_148834_object98770:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e392)", 2024-09-18T08:18:10.385 INFO:teuthology.orchestra.run.smithi007.stdout: "initiated_at": "2024-09-18T08:18:10.367336+0000", 2024-09-18T08:18:10.385 INFO:teuthology.orchestra.run.smithi007.stdout: "age": 0.00041211699999999998, 2024-09-18T08:18:10.385 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.00063308400000000001, 2024-09-18T08:18:10.385 INFO:teuthology.orchestra.run.smithi007.stdout: "continuous": false, 2024-09-18T08:18:10.385 INFO:teuthology.orchestra.run.smithi007.stdout: "type_data": { 2024-09-18T08:18:10.386 INFO:teuthology.orchestra.run.smithi007.stdout: "flag_point": "waiting for sub ops", 2024-09-18T08:18:10.386 INFO:teuthology.orchestra.run.smithi007.stdout: "client_info": { 2024-09-18T08:18:10.386 INFO:teuthology.orchestra.run.smithi007.stdout: "client": "client.15966", 2024-09-18T08:18:10.386 INFO:teuthology.orchestra.run.smithi007.stdout: "client_addr": "172.21.15.7:0/529784850", 2024-09-18T08:18:10.386 INFO:teuthology.orchestra.run.smithi007.stdout: "tid": 200490 2024-09-18T08:18:10.386 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:18:10.386 INFO:teuthology.orchestra.run.smithi007.stdout: "events": [ 2024-09-18T08:18:10.386 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:18:10.386 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "initiated", 2024-09-18T08:18:10.386 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:18:10.367336+0000", 2024-09-18T08:18:10.386 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-09-18T08:18:10.386 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:18:10.386 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:18:10.387 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "header_read", 2024-09-18T08:18:10.387 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:18:10.367336+0000", 2024-09-18T08:18:10.387 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-09-18T08:18:10.387 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:18:10.387 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:18:10.387 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "throttled", 2024-09-18T08:18:10.387 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:18:10.367337+0000", 2024-09-18T08:18:10.387 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 9.2399999999999996e-07 2024-09-18T08:18:10.387 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:18:10.387 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:18:10.387 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "all_read", 2024-09-18T08:18:10.387 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:18:10.367341+0000", 2024-09-18T08:18:10.387 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 4.5469999999999998e-06 2024-09-18T08:18:10.388 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:18:10.388 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:18:10.388 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "dispatched", 2024-09-18T08:18:10.388 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:18:10.367352+0000", 2024-09-18T08:18:10.388 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.0522e-05 2024-09-18T08:18:10.388 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:18:10.388 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:18:10.388 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "queued_for_pg", 2024-09-18T08:18:10.388 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:18:10.367360+0000", 2024-09-18T08:18:10.388 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 7.7409999999999992e-06 2024-09-18T08:18:10.388 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:18:10.388 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:18:10.388 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-09-18T08:18:10.389 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:18:10.367415+0000", 2024-09-18T08:18:10.389 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 5.4948000000000003e-05 2024-09-18T08:18:10.389 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:18:10.389 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:18:10.389 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "started", 2024-09-18T08:18:10.389 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:18:10.367513+0000", 2024-09-18T08:18:10.389 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 9.8407999999999997e-05 2024-09-18T08:18:10.389 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:18:10.389 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:18:10.389 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "waiting for subops from 0,3", 2024-09-18T08:18:10.389 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:18:10.367616+0000", 2024-09-18T08:18:10.389 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.00010304 2024-09-18T08:18:10.389 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-09-18T08:18:10.390 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-09-18T08:18:10.390 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-09-18T08:18:10.390 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:18:10.390 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:18:10.390 INFO:teuthology.orchestra.run.smithi007.stdout: "description": "osd_op(client.15966.0:200471 10.e 10:72336719:::benchmark_data_smithi007_148834_object98751:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e392)", 2024-09-18T08:18:10.390 INFO:teuthology.orchestra.run.smithi007.stdout: "initiated_at": "2024-09-18T08:18:10.353410+0000", 2024-09-18T08:18:10.390 INFO:teuthology.orchestra.run.smithi007.stdout: "age": 0.014337956000000001, 2024-09-18T08:18:10.390 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.014621252, 2024-09-18T08:18:10.390 INFO:teuthology.orchestra.run.smithi007.stdout: "continuous": false, 2024-09-18T08:18:10.390 INFO:teuthology.orchestra.run.smithi007.stdout: "type_data": { 2024-09-18T08:18:10.390 INFO:teuthology.orchestra.run.smithi007.stdout: "flag_point": "waiting for sub ops", 2024-09-18T08:18:10.390 INFO:teuthology.orchestra.run.smithi007.stdout: "client_info": { 2024-09-18T08:18:10.390 INFO:teuthology.orchestra.run.smithi007.stdout: "client": "client.15966", 2024-09-18T08:18:10.390 INFO:teuthology.orchestra.run.smithi007.stdout: "client_addr": "172.21.15.7:0/529784850", 2024-09-18T08:18:10.391 INFO:teuthology.orchestra.run.smithi007.stdout: "tid": 200471 2024-09-18T08:18:10.391 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:18:10.391 INFO:teuthology.orchestra.run.smithi007.stdout: "events": [ 2024-09-18T08:18:10.391 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:18:10.391 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "initiated", 2024-09-18T08:18:10.391 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:18:10.353410+0000", 2024-09-18T08:18:10.391 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-09-18T08:18:10.391 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:18:10.391 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:18:10.391 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "header_read", 2024-09-18T08:18:10.391 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:18:10.353410+0000", 2024-09-18T08:18:10.391 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-09-18T08:18:10.391 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:18:10.392 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:18:10.392 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "throttled", 2024-09-18T08:18:10.392 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:18:10.353412+0000", 2024-09-18T08:18:10.392 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.4139999999999999e-06 2024-09-18T08:18:10.392 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:18:10.392 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:18:10.392 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "all_read", 2024-09-18T08:18:10.392 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:18:10.353417+0000", 2024-09-18T08:18:10.392 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 5.5589999999999996e-06 2024-09-18T08:18:10.392 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:18:10.392 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:18:10.392 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "dispatched", 2024-09-18T08:18:10.392 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:18:10.353432+0000", 2024-09-18T08:18:10.393 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.4491000000000001e-05 2024-09-18T08:18:10.393 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:18:10.393 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:18:10.393 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "queued_for_pg", 2024-09-18T08:18:10.393 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:18:10.353439+0000", 2024-09-18T08:18:10.393 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 7.1670000000000002e-06 2024-09-18T08:18:10.393 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:18:10.393 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:18:10.393 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-09-18T08:18:10.393 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:18:10.353545+0000", 2024-09-18T08:18:10.393 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.000106001 2024-09-18T08:18:10.393 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:18:10.393 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:18:10.393 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "started", 2024-09-18T08:18:10.394 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:18:10.353664+0000", 2024-09-18T08:18:10.394 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.00011897500000000001 2024-09-18T08:18:10.394 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:18:10.394 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:18:10.394 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "waiting for subops from 5,6", 2024-09-18T08:18:10.394 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:18:10.353775+0000", 2024-09-18T08:18:10.394 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.00011128699999999999 2024-09-18T08:18:10.394 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:18:10.394 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:18:10.394 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:18:10.394 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:18:10.359248+0000", 2024-09-18T08:18:10.394 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.0054732360000000002 2024-09-18T08:18:10.394 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:18:10.394 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:18:10.395 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:18:10.395 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:18:10.359296+0000", 2024-09-18T08:18:10.395 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 4.7580999999999997e-05 2024-09-18T08:18:10.395 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-09-18T08:18:10.395 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-09-18T08:18:10.395 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-09-18T08:18:10.395 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:18:10.395 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:18:10.395 INFO:teuthology.orchestra.run.smithi007.stdout: "description": "osd_op(client.15966.0:200476 10.e 10:77728443:::benchmark_data_smithi007_148834_object98756:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e392)", 2024-09-18T08:18:10.395 INFO:teuthology.orchestra.run.smithi007.stdout: "initiated_at": "2024-09-18T08:18:10.353466+0000", 2024-09-18T08:18:10.395 INFO:teuthology.orchestra.run.smithi007.stdout: "age": 0.014282125999999999, 2024-09-18T08:18:10.395 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.014633734000000001, 2024-09-18T08:18:10.395 INFO:teuthology.orchestra.run.smithi007.stdout: "continuous": false, 2024-09-18T08:18:10.396 INFO:teuthology.orchestra.run.smithi007.stdout: "type_data": { 2024-09-18T08:18:10.396 INFO:teuthology.orchestra.run.smithi007.stdout: "flag_point": "waiting for sub ops", 2024-09-18T08:18:10.396 INFO:teuthology.orchestra.run.smithi007.stdout: "client_info": { 2024-09-18T08:18:10.396 INFO:teuthology.orchestra.run.smithi007.stdout: "client": "client.15966", 2024-09-18T08:18:10.396 INFO:teuthology.orchestra.run.smithi007.stdout: "client_addr": "172.21.15.7:0/529784850", 2024-09-18T08:18:10.396 INFO:teuthology.orchestra.run.smithi007.stdout: "tid": 200476 2024-09-18T08:18:10.396 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:18:10.396 INFO:teuthology.orchestra.run.smithi007.stdout: "events": [ 2024-09-18T08:18:10.396 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:18:10.396 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "initiated", 2024-09-18T08:18:10.396 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:18:10.353466+0000", 2024-09-18T08:18:10.396 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-09-18T08:18:10.396 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:18:10.396 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:18:10.397 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "header_read", 2024-09-18T08:18:10.397 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:18:10.353466+0000", 2024-09-18T08:18:10.397 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-09-18T08:18:10.397 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:18:10.397 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:18:10.397 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "throttled", 2024-09-18T08:18:10.397 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:18:10.353466+0000", 2024-09-18T08:18:10.397 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 4.82e-07 2024-09-18T08:18:10.397 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:18:10.397 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:18:10.397 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "all_read", 2024-09-18T08:18:10.397 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:18:10.353469+0000", 2024-09-18T08:18:10.397 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 2.7860000000000001e-06 2024-09-18T08:18:10.398 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:18:10.398 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:18:10.398 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "dispatched", 2024-09-18T08:18:10.398 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:18:10.353475+0000", 2024-09-18T08:18:10.398 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 6.1029999999999998e-06 2024-09-18T08:18:10.398 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:18:10.398 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:18:10.398 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "queued_for_pg", 2024-09-18T08:18:10.398 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:18:10.353480+0000", 2024-09-18T08:18:10.398 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 5.0690000000000001e-06 2024-09-18T08:18:10.398 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:18:10.398 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:18:10.398 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-09-18T08:18:10.398 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:18:10.359330+0000", 2024-09-18T08:18:10.398 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.0058500949999999996 2024-09-18T08:18:10.399 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:18:10.399 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:18:10.399 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "started", 2024-09-18T08:18:10.399 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:18:10.359449+0000", 2024-09-18T08:18:10.399 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.000118292 2024-09-18T08:18:10.399 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:18:10.399 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:18:10.399 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "waiting for subops from 5,6", 2024-09-18T08:18:10.399 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:18:10.359561+0000", 2024-09-18T08:18:10.399 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.00011217000000000001 2024-09-18T08:18:10.399 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-09-18T08:18:10.399 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-09-18T08:18:10.399 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-09-18T08:18:10.400 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:18:10.400 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:18:10.400 INFO:teuthology.orchestra.run.smithi007.stdout: "description": "osd_op(client.15966.0:200477 10.e 10:7d0a463c:::benchmark_data_smithi007_148834_object98757:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e392)", 2024-09-18T08:18:10.400 INFO:teuthology.orchestra.run.smithi007.stdout: "initiated_at": "2024-09-18T08:18:10.353488+0000", 2024-09-18T08:18:10.400 INFO:teuthology.orchestra.run.smithi007.stdout: "age": 0.014260557, 2024-09-18T08:18:10.400 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.014672892, 2024-09-18T08:18:10.400 INFO:teuthology.orchestra.run.smithi007.stdout: "continuous": false, 2024-09-18T08:18:10.400 INFO:teuthology.orchestra.run.smithi007.stdout: "type_data": { 2024-09-18T08:18:10.400 INFO:teuthology.orchestra.run.smithi007.stdout: "flag_point": "waiting for sub ops", 2024-09-18T08:18:10.400 INFO:teuthology.orchestra.run.smithi007.stdout: "client_info": { 2024-09-18T08:18:10.400 INFO:teuthology.orchestra.run.smithi007.stdout: "client": "client.15966", 2024-09-18T08:18:10.400 INFO:teuthology.orchestra.run.smithi007.stdout: "client_addr": "172.21.15.7:0/529784850", 2024-09-18T08:18:10.400 INFO:teuthology.orchestra.run.smithi007.stdout: "tid": 200477 2024-09-18T08:18:10.401 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:18:10.401 INFO:teuthology.orchestra.run.smithi007.stdout: "events": [ 2024-09-18T08:18:10.401 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:18:10.401 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "initiated", 2024-09-18T08:18:10.401 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:18:10.353488+0000", 2024-09-18T08:18:10.401 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-09-18T08:18:10.401 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:18:10.401 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:18:10.401 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "header_read", 2024-09-18T08:18:10.401 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:18:10.353488+0000", 2024-09-18T08:18:10.401 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-09-18T08:18:10.401 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:18:10.401 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:18:10.402 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "throttled", 2024-09-18T08:18:10.402 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:18:10.353488+0000", 2024-09-18T08:18:10.402 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 3.0899999999999997e-07 2024-09-18T08:18:10.402 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:18:10.402 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:18:10.402 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "all_read", 2024-09-18T08:18:10.402 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:18:10.353490+0000", 2024-09-18T08:18:10.402 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.9989999999999998e-06 2024-09-18T08:18:10.402 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:18:10.402 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:18:10.402 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "dispatched", 2024-09-18T08:18:10.402 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:18:10.353494+0000", 2024-09-18T08:18:10.402 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 4.6759999999999998e-06 2024-09-18T08:18:10.403 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:18:10.403 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:18:10.403 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "queued_for_pg", 2024-09-18T08:18:10.403 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:18:10.353499+0000", 2024-09-18T08:18:10.403 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 4.3780000000000003e-06 2024-09-18T08:18:10.403 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:18:10.403 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:18:10.403 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-09-18T08:18:10.403 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:18:10.366594+0000", 2024-09-18T08:18:10.403 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.013095258 2024-09-18T08:18:10.403 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:18:10.403 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:18:10.403 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "started", 2024-09-18T08:18:10.404 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:18:10.366695+0000", 2024-09-18T08:18:10.404 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.00010111999999999999 2024-09-18T08:18:10.404 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:18:10.404 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:18:10.404 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "waiting for subops from 5,6", 2024-09-18T08:18:10.404 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:18:10.366804+0000", 2024-09-18T08:18:10.404 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.00010886 2024-09-18T08:18:10.404 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-09-18T08:18:10.404 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-09-18T08:18:10.404 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-09-18T08:18:10.404 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:18:10.404 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:18:10.405 INFO:teuthology.orchestra.run.smithi007.stdout: "description": "osd_op(client.15966.0:200482 10.5 10:ad07d638:::benchmark_data_smithi007_148834_object98762:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e392)", 2024-09-18T08:18:10.405 INFO:teuthology.orchestra.run.smithi007.stdout: "initiated_at": "2024-09-18T08:18:10.360653+0000", 2024-09-18T08:18:10.405 INFO:teuthology.orchestra.run.smithi007.stdout: "age": 0.0070952910000000001, 2024-09-18T08:18:10.405 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.0075668819999999996, 2024-09-18T08:18:10.405 INFO:teuthology.orchestra.run.smithi007.stdout: "continuous": false, 2024-09-18T08:18:10.405 INFO:teuthology.orchestra.run.smithi007.stdout: "type_data": { 2024-09-18T08:18:10.405 INFO:teuthology.orchestra.run.smithi007.stdout: "flag_point": "waiting for sub ops", 2024-09-18T08:18:10.405 INFO:teuthology.orchestra.run.smithi007.stdout: "client_info": { 2024-09-18T08:18:10.405 INFO:teuthology.orchestra.run.smithi007.stdout: "client": "client.15966", 2024-09-18T08:18:10.405 INFO:teuthology.orchestra.run.smithi007.stdout: "client_addr": "172.21.15.7:0/529784850", 2024-09-18T08:18:10.405 INFO:teuthology.orchestra.run.smithi007.stdout: "tid": 200482 2024-09-18T08:18:10.405 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:18:10.405 INFO:teuthology.orchestra.run.smithi007.stdout: "events": [ 2024-09-18T08:18:10.405 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:18:10.406 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "initiated", 2024-09-18T08:18:10.406 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:18:10.360653+0000", 2024-09-18T08:18:10.406 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-09-18T08:18:10.406 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:18:10.406 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:18:10.406 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "header_read", 2024-09-18T08:18:10.406 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:18:10.360653+0000", 2024-09-18T08:18:10.406 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-09-18T08:18:10.406 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:18:10.406 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:18:10.406 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "throttled", 2024-09-18T08:18:10.406 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:18:10.360654+0000", 2024-09-18T08:18:10.406 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 7.7100000000000001e-07 2024-09-18T08:18:10.407 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:18:10.407 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:18:10.407 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "all_read", 2024-09-18T08:18:10.407 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:18:10.360657+0000", 2024-09-18T08:18:10.407 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 3.1470000000000001e-06 2024-09-18T08:18:10.407 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:18:10.407 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:18:10.407 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "dispatched", 2024-09-18T08:18:10.407 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:18:10.360665+0000", 2024-09-18T08:18:10.407 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 7.9359999999999999e-06 2024-09-18T08:18:10.407 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:18:10.407 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:18:10.407 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "queued_for_pg", 2024-09-18T08:18:10.407 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:18:10.360671+0000", 2024-09-18T08:18:10.408 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 6.6850000000000004e-06 2024-09-18T08:18:10.408 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:18:10.408 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:18:10.408 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-09-18T08:18:10.408 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:18:10.360731+0000", 2024-09-18T08:18:10.408 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 5.9753999999999997e-05 2024-09-18T08:18:10.408 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:18:10.408 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:18:10.408 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "started", 2024-09-18T08:18:10.408 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:18:10.360859+0000", 2024-09-18T08:18:10.408 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.00012825400000000001 2024-09-18T08:18:10.408 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:18:10.408 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:18:10.409 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "waiting for subops from 0,3", 2024-09-18T08:18:10.409 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:18:10.360978+0000", 2024-09-18T08:18:10.409 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.000118515 2024-09-18T08:18:10.409 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:18:10.409 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:18:10.409 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:18:10.409 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:18:10.364809+0000", 2024-09-18T08:18:10.409 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.0038311679999999998 2024-09-18T08:18:10.409 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:18:10.409 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:18:10.409 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "op_commit", 2024-09-18T08:18:10.409 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:18:10.364878+0000", 2024-09-18T08:18:10.409 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 6.8966000000000003e-05 2024-09-18T08:18:10.409 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-09-18T08:18:10.410 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-09-18T08:18:10.410 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-09-18T08:18:10.410 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:18:10.410 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:18:10.410 INFO:teuthology.orchestra.run.smithi007.stdout: "description": "osd_repop_reply(client.15966.0:200476 10.e e392/386)", 2024-09-18T08:18:10.410 INFO:teuthology.orchestra.run.smithi007.stdout: "initiated_at": "2024-09-18T08:18:10.361841+0000", 2024-09-18T08:18:10.410 INFO:teuthology.orchestra.run.smithi007.stdout: "age": 0.0059068259999999996, 2024-09-18T08:18:10.410 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.0064422469999999999, 2024-09-18T08:18:10.410 INFO:teuthology.orchestra.run.smithi007.stdout: "continuous": false, 2024-09-18T08:18:10.410 INFO:teuthology.orchestra.run.smithi007.stdout: "type_data": { 2024-09-18T08:18:10.410 INFO:teuthology.orchestra.run.smithi007.stdout: "flag_point": "queued for pg", 2024-09-18T08:18:10.410 INFO:teuthology.orchestra.run.smithi007.stdout: "events": [ 2024-09-18T08:18:10.411 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:18:10.411 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "initiated", 2024-09-18T08:18:10.411 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:18:10.361841+0000", 2024-09-18T08:18:10.411 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-09-18T08:18:10.411 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:18:10.411 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:18:10.411 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "header_read", 2024-09-18T08:18:10.411 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:18:10.361841+0000", 2024-09-18T08:18:10.411 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-09-18T08:18:10.411 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:18:10.412 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:18:10.412 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "throttled", 2024-09-18T08:18:10.412 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:18:10.361842+0000", 2024-09-18T08:18:10.412 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 8.7599999999999996e-07 2024-09-18T08:18:10.412 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:18:10.412 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:18:10.412 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "all_read", 2024-09-18T08:18:10.412 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:18:10.361857+0000", 2024-09-18T08:18:10.412 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.5115e-05 2024-09-18T08:18:10.412 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:18:10.412 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:18:10.412 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "dispatched", 2024-09-18T08:18:10.412 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:18:10.361871+0000", 2024-09-18T08:18:10.413 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.3788000000000001e-05 2024-09-18T08:18:10.413 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:18:10.413 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:18:10.413 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "queued_for_pg", 2024-09-18T08:18:10.413 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:18:10.361878+0000", 2024-09-18T08:18:10.413 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 6.7909999999999999e-06 2024-09-18T08:18:10.413 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-09-18T08:18:10.413 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-09-18T08:18:10.413 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-09-18T08:18:10.413 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-09-18T08:18:10.413 INFO:teuthology.orchestra.run.smithi007.stdout: ], 2024-09-18T08:18:10.413 INFO:teuthology.orchestra.run.smithi007.stdout: "num_ops": 10 2024-09-18T08:18:10.413 INFO:teuthology.orchestra.run.smithi007.stdout:} 2024-09-18T08:18:10.422 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0 2024-09-18T08:18:10.472 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:18:10.473 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:18:10.473 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0' 2024-09-18T08:18:10.574 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0 2024-09-18T08:18:10.615 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:18:10.616 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:18:10.616 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0' 2024-09-18T08:18:10.717 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7 2024-09-18T08:18:10.730 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:18:10 smithi007 ceph-mon[29459]: pgmap v768: 158 pgs: 158 active+clean; 389 MiB data, 4.4 GiB used, 532 GiB / 536 GiB avail; 852 B/s rd, 0 B/s wr, 1.28k op/s 2024-09-18T08:18:10.730 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:18:10 smithi007 ceph-mon[29459]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-18T08:18:10.730 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:18:10 smithi007 ceph-mon[39249]: pgmap v768: 158 pgs: 158 active+clean; 389 MiB data, 4.4 GiB used, 532 GiB / 536 GiB avail; 852 B/s rd, 0 B/s wr, 1.28k op/s 2024-09-18T08:18:10.730 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:18:10 smithi007 ceph-mon[39249]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-18T08:18:10.743 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:18:10.744 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:18:10.744 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7' 2024-09-18T08:18:10.845 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3 2024-09-18T08:18:10.883 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:18:10.884 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:18:10.884 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3' 2024-09-18T08:18:10.905 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:18:10 smithi050 ceph-mon[32371]: pgmap v768: 158 pgs: 158 active+clean; 389 MiB data, 4.4 GiB used, 532 GiB / 536 GiB avail; 852 B/s rd, 0 B/s wr, 1.28k op/s 2024-09-18T08:18:10.905 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:18:10 smithi050 ceph-mon[32371]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-18T08:18:10.985 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2 2024-09-18T08:18:11.024 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:18:11.025 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:18:11.026 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2' 2024-09-18T08:18:11.126 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7 2024-09-18T08:18:11.153 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:18:11.154 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:18:11.154 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7' 2024-09-18T08:18:11.255 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1 2024-09-18T08:18:11.297 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:18:11.298 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:18:11.298 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1' 2024-09-18T08:18:11.337 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/779349ca-7593-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T08:18:11.399 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6 2024-09-18T08:18:11.424 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:18:11.425 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:18:11.425 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6' 2024-09-18T08:18:11.526 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6 2024-09-18T08:18:11.550 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:18:11.550 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:18:11.551 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6' 2024-09-18T08:18:11.651 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4 2024-09-18T08:18:11.678 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:18:11.678 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:18:11.679 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4' 2024-09-18T08:18:11.730 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:18:11 smithi007 ceph-mon[29459]: purged_snaps scrub starts 2024-09-18T08:18:11.730 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:18:11 smithi007 ceph-mon[29459]: purged_snaps scrub ok 2024-09-18T08:18:11.730 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:18:11 smithi007 ceph-mon[29459]: osdmap e393: 8 total, 8 up, 6 in 2024-09-18T08:18:11.730 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:18:11 smithi007 ceph-mon[39249]: purged_snaps scrub starts 2024-09-18T08:18:11.731 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:18:11 smithi007 ceph-mon[39249]: purged_snaps scrub ok 2024-09-18T08:18:11.731 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:18:11 smithi007 ceph-mon[39249]: osdmap e393: 8 total, 8 up, 6 in 2024-09-18T08:18:11.779 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2 2024-09-18T08:18:11.816 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:18:11.816 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:18:11.817 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2' 2024-09-18T08:18:11.905 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:18:11 smithi050 ceph-mon[32371]: purged_snaps scrub starts 2024-09-18T08:18:11.905 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:18:11 smithi050 ceph-mon[32371]: purged_snaps scrub ok 2024-09-18T08:18:11.905 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:18:11 smithi050 ceph-mon[32371]: osdmap e393: 8 total, 8 up, 6 in 2024-09-18T08:18:11.917 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6 2024-09-18T08:18:11.943 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:18:11.944 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:18:11.944 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6' 2024-09-18T08:18:12.045 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6 2024-09-18T08:18:12.070 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:18:12.071 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:18:12.071 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6' 2024-09-18T08:18:12.172 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0 2024-09-18T08:18:12.214 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:18:12.214 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:18:12.215 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0' 2024-09-18T08:18:12.315 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4 2024-09-18T08:18:12.338 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:18:12.339 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:18:12.339 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4' 2024-09-18T08:18:12.440 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1 2024-09-18T08:18:12.473 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:18:12.473 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:18:12.474 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1' 2024-09-18T08:18:12.575 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0 2024-09-18T08:18:12.610 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:18:12.611 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:18:12.611 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0' 2024-09-18T08:18:12.712 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3 2024-09-18T08:18:12.751 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:18:12.752 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:18:12.752 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3' 2024-09-18T08:18:12.802 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid 779349ca-7593-11ef-bceb-c7b262605968 -- ceph daemon osd.2 dump_blocked_ops 2024-09-18T08:18:12.842 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:18:12 smithi007 ceph-mon[29459]: pgmap v770: 158 pgs: 158 active+clean; 265 MiB data, 4.2 GiB used, 532 GiB / 536 GiB avail; 852 B/s rd, 0 B/s wr, 904 op/s 2024-09-18T08:18:12.842 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:18:12 smithi007 ceph-mon[39249]: pgmap v770: 158 pgs: 158 active+clean; 265 MiB data, 4.2 GiB used, 532 GiB / 536 GiB avail; 852 B/s rd, 0 B/s wr, 904 op/s 2024-09-18T08:18:12.853 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3 2024-09-18T08:18:12.904 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:18:12.905 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:18:12.905 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3' 2024-09-18T08:18:12.905 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:18:12 smithi050 ceph-mon[32371]: pgmap v770: 158 pgs: 158 active+clean; 265 MiB data, 4.2 GiB used, 532 GiB / 536 GiB avail; 852 B/s rd, 0 B/s wr, 904 op/s 2024-09-18T08:18:12.946 INFO:tasks.radosbench.radosbench.0.smithi007.stdout:Removed 101717 objects 2024-09-18T08:18:12.946 INFO:tasks.radosbench.radosbench.0.smithi007.stdout:Clean up completed and total clean up time :114.587 2024-09-18T08:18:13.006 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2 2024-09-18T08:18:13.028 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:18:13.029 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:18:13.029 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2' 2024-09-18T08:18:13.129 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid 779349ca-7593-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-18T08:18:13.130 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0 2024-09-18T08:18:13.152 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:18:13.153 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:18:13.153 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0' 2024-09-18T08:18:13.254 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4 2024-09-18T08:18:13.278 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:18:13.278 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:18:13.279 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4' 2024-09-18T08:18:13.380 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4 2024-09-18T08:18:13.403 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:18:13.404 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:18:13.404 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4' 2024-09-18T08:18:13.505 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5 2024-09-18T08:18:13.528 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:18:13.529 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:18:13.530 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5' 2024-09-18T08:18:13.630 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6 2024-09-18T08:18:13.653 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:18:13.654 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:18:13.654 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6' 2024-09-18T08:18:13.755 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3 2024-09-18T08:18:13.778 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:18:13.779 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:18:13.779 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3' 2024-09-18T08:18:13.880 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6 2024-09-18T08:18:13.903 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:18:13.904 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:18:13.905 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6' 2024-09-18T08:18:13.905 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:18:13 smithi050 ceph-mon[32371]: from='client.? 172.21.15.7:0/1442393512' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-18T08:18:13.905 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:18:13 smithi050 ceph-mon[32371]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-18T08:18:13.980 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:18:13 smithi007 ceph-mon[29459]: from='client.? 172.21.15.7:0/1442393512' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-18T08:18:13.980 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:18:13 smithi007 ceph-mon[29459]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-18T08:18:13.980 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:18:13 smithi007 ceph-mon[39249]: from='client.? 172.21.15.7:0/1442393512' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-18T08:18:13.980 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:18:13 smithi007 ceph-mon[39249]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-18T08:18:14.005 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7 2024-09-18T08:18:14.027 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:18:14.028 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:18:14.029 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7' 2024-09-18T08:18:14.084 INFO:teuthology.orchestra.run.smithi007.stderr:noscrub is unset 2024-09-18T08:18:14.129 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2 2024-09-18T08:18:14.157 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:18:14.158 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:18:14.158 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2' 2024-09-18T08:18:14.259 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1 2024-09-18T08:18:14.286 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:18:14.287 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:18:14.287 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1' 2024-09-18T08:18:14.388 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7 2024-09-18T08:18:14.413 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:18:14.413 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:18:14.414 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7' 2024-09-18T08:18:14.514 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5 2024-09-18T08:18:14.540 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:18:14.540 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:18:14.541 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5' 2024-09-18T08:18:14.628 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/779349ca-7593-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T08:18:14.641 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3 2024-09-18T08:18:14.670 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:18:14.670 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:18:14.671 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3' 2024-09-18T08:18:14.772 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2 2024-09-18T08:18:14.796 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:18:14.797 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:18:14.797 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2' 2024-09-18T08:18:14.898 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2 2024-09-18T08:18:14.905 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:18:14 smithi050 ceph-mon[32371]: pgmap v771: 158 pgs: 158 active+clean; 204 MiB data, 3.9 GiB used, 533 GiB / 536 GiB avail; 1.7 KiB/s rd, 0 B/s wr, 1.07k op/s 2024-09-18T08:18:14.905 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:18:14 smithi050 ceph-mon[32371]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-18T08:18:14.905 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:18:14 smithi050 ceph-mon[32371]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-18T08:18:14.905 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:18:14 smithi050 ceph-mon[32371]: osdmap e394: 8 total, 8 up, 6 in 2024-09-18T08:18:14.923 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:18:14.924 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:18:14.924 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2' 2024-09-18T08:18:14.980 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:18:14 smithi007 ceph-mon[39249]: pgmap v771: 158 pgs: 158 active+clean; 204 MiB data, 3.9 GiB used, 533 GiB / 536 GiB avail; 1.7 KiB/s rd, 0 B/s wr, 1.07k op/s 2024-09-18T08:18:14.980 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:18:14 smithi007 ceph-mon[39249]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-18T08:18:14.980 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:18:14 smithi007 ceph-mon[39249]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-18T08:18:14.980 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:18:14 smithi007 ceph-mon[39249]: osdmap e394: 8 total, 8 up, 6 in 2024-09-18T08:18:14.980 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:18:14 smithi007 ceph-mon[29459]: pgmap v771: 158 pgs: 158 active+clean; 204 MiB data, 3.9 GiB used, 533 GiB / 536 GiB avail; 1.7 KiB/s rd, 0 B/s wr, 1.07k op/s 2024-09-18T08:18:14.980 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:18:14 smithi007 ceph-mon[29459]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-18T08:18:14.980 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:18:14 smithi007 ceph-mon[29459]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-18T08:18:14.981 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:18:14 smithi007 ceph-mon[29459]: osdmap e394: 8 total, 8 up, 6 in 2024-09-18T08:18:15.025 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2 2024-09-18T08:18:15.050 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:18:15.051 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:18:15.051 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2' 2024-09-18T08:18:15.152 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5 2024-09-18T08:18:15.179 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:18:15.180 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:18:15.180 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5' 2024-09-18T08:18:15.281 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2 2024-09-18T08:18:15.306 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:18:15.306 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:18:15.307 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2' 2024-09-18T08:18:15.407 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4 2024-09-18T08:18:15.434 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:18:15.435 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:18:15.435 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4' 2024-09-18T08:18:15.536 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6 2024-09-18T08:18:15.562 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:18:15.563 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:18:15.563 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6' 2024-09-18T08:18:15.664 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2 2024-09-18T08:18:15.689 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:18:15.690 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:18:15.690 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2' 2024-09-18T08:18:15.716 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/779349ca-7593-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T08:18:15.721 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/779349ca-7593-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T08:18:15.729 INFO:tasks.cephadm.ceph_manager.ceph:removing pool_name unique_pool_3 2024-09-18T08:18:15.729 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid 779349ca-7593-11ef-bceb-c7b262605968 -- ceph osd pool rm unique_pool_3 unique_pool_3 --yes-i-really-really-mean-it 2024-09-18T08:18:15.730 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:18:15 smithi007 ceph-mon[29459]: 1.5 scrub starts 2024-09-18T08:18:15.730 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:18:15 smithi007 ceph-mon[29459]: 1.5 scrub ok 2024-09-18T08:18:15.730 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:18:15 smithi007 ceph-mon[29459]: 10.7 scrub starts 2024-09-18T08:18:15.730 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:18:15 smithi007 ceph-mon[29459]: 10.7 scrub ok 2024-09-18T08:18:15.730 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:18:15 smithi007 ceph-mon[29459]: 5.11 scrub starts 2024-09-18T08:18:15.730 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:18:15 smithi007 ceph-mon[29459]: 5.11 scrub ok 2024-09-18T08:18:15.731 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:18:15 smithi007 ceph-mon[39249]: 1.5 scrub starts 2024-09-18T08:18:15.731 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:18:15 smithi007 ceph-mon[39249]: 1.5 scrub ok 2024-09-18T08:18:15.732 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:18:15 smithi007 ceph-mon[39249]: 10.7 scrub starts 2024-09-18T08:18:15.732 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:18:15 smithi007 ceph-mon[39249]: 10.7 scrub ok 2024-09-18T08:18:15.732 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:18:15 smithi007 ceph-mon[39249]: 5.11 scrub starts 2024-09-18T08:18:15.732 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:18:15 smithi007 ceph-mon[39249]: 5.11 scrub ok 2024-09-18T08:18:15.791 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0 2024-09-18T08:18:15.813 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:18:15.814 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:18:15.814 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0' 2024-09-18T08:18:15.905 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:18:15 smithi050 ceph-mon[32371]: 1.5 scrub starts 2024-09-18T08:18:15.905 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:18:15 smithi050 ceph-mon[32371]: 1.5 scrub ok 2024-09-18T08:18:15.905 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:18:15 smithi050 ceph-mon[32371]: 10.7 scrub starts 2024-09-18T08:18:15.905 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:18:15 smithi050 ceph-mon[32371]: 10.7 scrub ok 2024-09-18T08:18:15.905 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:18:15 smithi050 ceph-mon[32371]: 5.11 scrub starts 2024-09-18T08:18:15.906 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:18:15 smithi050 ceph-mon[32371]: 5.11 scrub ok 2024-09-18T08:18:15.915 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3 2024-09-18T08:18:15.938 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:18:15.939 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:18:15.939 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3' 2024-09-18T08:18:16.040 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5 2024-09-18T08:18:16.067 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:18:16.068 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:18:16.068 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5' 2024-09-18T08:18:16.169 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2 2024-09-18T08:18:16.191 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:18:16.192 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:18:16.192 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2' 2024-09-18T08:18:16.293 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1 2024-09-18T08:18:16.315 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:18:16.315 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:18:16.316 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1' 2024-09-18T08:18:16.417 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4 2024-09-18T08:18:16.444 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:18:16.444 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:18:16.445 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4' 2024-09-18T08:18:16.460 INFO:teuthology.orchestra.run.smithi007.stdout:osd.0: {} 2024-09-18T08:18:16.461 INFO:teuthology.orchestra.run.smithi007.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-18T08:18:16.477 INFO:teuthology.orchestra.run.smithi007.stdout:osd.1: {} 2024-09-18T08:18:16.477 INFO:teuthology.orchestra.run.smithi007.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-18T08:18:16.493 INFO:teuthology.orchestra.run.smithi007.stdout:osd.2: {} 2024-09-18T08:18:16.493 INFO:teuthology.orchestra.run.smithi007.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-18T08:18:16.510 INFO:teuthology.orchestra.run.smithi007.stdout:osd.3: {} 2024-09-18T08:18:16.510 INFO:teuthology.orchestra.run.smithi007.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-18T08:18:16.527 INFO:teuthology.orchestra.run.smithi007.stdout:osd.4: {} 2024-09-18T08:18:16.527 INFO:teuthology.orchestra.run.smithi007.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-18T08:18:16.544 INFO:teuthology.orchestra.run.smithi007.stdout:osd.5: {} 2024-09-18T08:18:16.544 INFO:teuthology.orchestra.run.smithi007.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-18T08:18:16.546 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4 2024-09-18T08:18:16.561 INFO:teuthology.orchestra.run.smithi007.stdout:osd.6: {} 2024-09-18T08:18:16.561 INFO:teuthology.orchestra.run.smithi007.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-18T08:18:16.572 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:18:16.573 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:18:16.573 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4' 2024-09-18T08:18:16.579 INFO:teuthology.orchestra.run.smithi007.stderr:set 3.c pg_upmap mapping to [3,2,6] 2024-09-18T08:18:16.579 INFO:teuthology.orchestra.run.smithi007.stdout:osd.7: {} 2024-09-18T08:18:16.580 INFO:teuthology.orchestra.run.smithi007.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-18T08:18:16.674 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6 2024-09-18T08:18:16.676 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:18:16 smithi007 ceph-mon[39249]: 6.8 scrub starts 2024-09-18T08:18:16.677 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:18:16 smithi007 ceph-mon[39249]: 6.8 scrub ok 2024-09-18T08:18:16.677 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:18:16 smithi007 ceph-mon[39249]: pgmap v773: 158 pgs: 158 active+clean; 3.9 MiB data, 3.3 GiB used, 533 GiB / 536 GiB avail; 849 B/s rd, 0 B/s wr, 1.02k op/s 2024-09-18T08:18:16.677 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:18:16 smithi007 ceph-mon[39249]: from='client.? 172.21.15.7:0/2415988260' entity='client.admin' cmd={"prefix": "osd pg-upmap", "pgid": "3.c", "id": [3, 2, 6]} : dispatch 2024-09-18T08:18:16.677 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:18:16 smithi007 ceph-mon[39249]: 4.3 scrub starts 2024-09-18T08:18:16.677 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:18:16 smithi007 ceph-mon[39249]: 4.3 scrub ok 2024-09-18T08:18:16.677 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:18:16 smithi007 ceph-mon[39249]: from='client.? 172.21.15.7:0/2620840712' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-18T08:18:16.677 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:18:16 smithi007 ceph-mon[29459]: 6.8 scrub starts 2024-09-18T08:18:16.677 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:18:16 smithi007 ceph-mon[29459]: 6.8 scrub ok 2024-09-18T08:18:16.677 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:18:16 smithi007 ceph-mon[29459]: pgmap v773: 158 pgs: 158 active+clean; 3.9 MiB data, 3.3 GiB used, 533 GiB / 536 GiB avail; 849 B/s rd, 0 B/s wr, 1.02k op/s 2024-09-18T08:18:16.677 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:18:16 smithi007 ceph-mon[29459]: from='client.? 172.21.15.7:0/2415988260' entity='client.admin' cmd={"prefix": "osd pg-upmap", "pgid": "3.c", "id": [3, 2, 6]} : dispatch 2024-09-18T08:18:16.678 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:18:16 smithi007 ceph-mon[29459]: 4.3 scrub starts 2024-09-18T08:18:16.678 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:18:16 smithi007 ceph-mon[29459]: 4.3 scrub ok 2024-09-18T08:18:16.678 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:18:16 smithi007 ceph-mon[29459]: from='client.? 172.21.15.7:0/2620840712' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-18T08:18:16.687 INFO:teuthology.orchestra.run.smithi007.stdout:ERROR: (22) Invalid argument 2024-09-18T08:18:16.687 INFO:teuthology.orchestra.run.smithi007.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-18T08:18:16.700 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:18:16.701 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:18:16.701 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6' 2024-09-18T08:18:16.802 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3 2024-09-18T08:18:16.824 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:18:16.825 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:18:16.825 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3' 2024-09-18T08:18:16.905 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:18:16 smithi050 ceph-mon[32371]: 6.8 scrub starts 2024-09-18T08:18:16.905 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:18:16 smithi050 ceph-mon[32371]: 6.8 scrub ok 2024-09-18T08:18:16.905 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:18:16 smithi050 ceph-mon[32371]: pgmap v773: 158 pgs: 158 active+clean; 3.9 MiB data, 3.3 GiB used, 533 GiB / 536 GiB avail; 849 B/s rd, 0 B/s wr, 1.02k op/s 2024-09-18T08:18:16.905 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:18:16 smithi050 ceph-mon[32371]: from='client.? 172.21.15.7:0/2415988260' entity='client.admin' cmd={"prefix": "osd pg-upmap", "pgid": "3.c", "id": [3, 2, 6]} : dispatch 2024-09-18T08:18:16.905 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:18:16 smithi050 ceph-mon[32371]: 4.3 scrub starts 2024-09-18T08:18:16.905 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:18:16 smithi050 ceph-mon[32371]: 4.3 scrub ok 2024-09-18T08:18:16.905 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:18:16 smithi050 ceph-mon[32371]: from='client.? 172.21.15.7:0/2620840712' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-18T08:18:16.926 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5 2024-09-18T08:18:16.954 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:18:16.955 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:18:16.955 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5' 2024-09-18T08:18:16.979 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Sep 18 08:18:16 smithi007 ceph-779349ca-7593-11ef-bceb-c7b262605968-mgr-y[29688]: ::ffff:172.21.15.50 - - [18/Sep/2024:08:18:16] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-09-18T08:18:17.056 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4 2024-09-18T08:18:17.082 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:18:17.083 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:18:17.083 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4' 2024-09-18T08:18:17.184 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4 2024-09-18T08:18:17.189 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid 779349ca-7593-11ef-bceb-c7b262605968 -- ceph osd unset nodeep-scrub 2024-09-18T08:18:17.210 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:18:17.211 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:18:17.211 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4' 2024-09-18T08:18:17.312 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4 2024-09-18T08:18:17.338 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:18:17.339 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:18:17.339 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4' 2024-09-18T08:18:17.440 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5 2024-09-18T08:18:17.466 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:18:17.466 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:18:17.467 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5' 2024-09-18T08:18:17.568 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1 2024-09-18T08:18:17.594 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:18:17.594 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:18:17.595 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1' 2024-09-18T08:18:17.696 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3 2024-09-18T08:18:17.718 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:18:17.719 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:18:17.719 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3' 2024-09-18T08:18:17.820 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5 2024-09-18T08:18:17.847 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:18:17.848 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:18:17.848 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5' 2024-09-18T08:18:17.905 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:18:17 smithi050 ceph-mon[32371]: 6.4 scrub starts 2024-09-18T08:18:17.905 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:18:17 smithi050 ceph-mon[32371]: 6.4 scrub ok 2024-09-18T08:18:17.905 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:18:17 smithi050 ceph-mon[32371]: from='client.? 172.21.15.7:0/2415988260' entity='client.admin' cmd='[{"prefix": "osd pg-upmap", "pgid": "3.c", "id": [3, 2, 6]}]': finished 2024-09-18T08:18:17.905 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:18:17 smithi050 ceph-mon[32371]: osdmap e395: 8 total, 8 up, 6 in 2024-09-18T08:18:17.905 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:18:17 smithi050 ceph-mon[32371]: 3.3 scrub starts 2024-09-18T08:18:17.905 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:18:17 smithi050 ceph-mon[32371]: 3.3 scrub ok 2024-09-18T08:18:17.949 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4 2024-09-18T08:18:17.976 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:18:17.977 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:18:17.977 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4' 2024-09-18T08:18:17.980 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:18:17 smithi007 ceph-mon[29459]: 6.4 scrub starts 2024-09-18T08:18:17.980 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:18:17 smithi007 ceph-mon[29459]: 6.4 scrub ok 2024-09-18T08:18:17.980 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:18:17 smithi007 ceph-mon[29459]: from='client.? 172.21.15.7:0/2415988260' entity='client.admin' cmd='[{"prefix": "osd pg-upmap", "pgid": "3.c", "id": [3, 2, 6]}]': finished 2024-09-18T08:18:17.980 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:18:17 smithi007 ceph-mon[29459]: osdmap e395: 8 total, 8 up, 6 in 2024-09-18T08:18:17.980 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:18:17 smithi007 ceph-mon[29459]: 3.3 scrub starts 2024-09-18T08:18:17.980 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:18:17 smithi007 ceph-mon[29459]: 3.3 scrub ok 2024-09-18T08:18:17.980 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:18:17 smithi007 ceph-mon[39249]: 6.4 scrub starts 2024-09-18T08:18:17.980 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:18:17 smithi007 ceph-mon[39249]: 6.4 scrub ok 2024-09-18T08:18:17.981 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:18:17 smithi007 ceph-mon[39249]: from='client.? 172.21.15.7:0/2415988260' entity='client.admin' cmd='[{"prefix": "osd pg-upmap", "pgid": "3.c", "id": [3, 2, 6]}]': finished 2024-09-18T08:18:17.981 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:18:17 smithi007 ceph-mon[39249]: osdmap e395: 8 total, 8 up, 6 in 2024-09-18T08:18:17.981 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:18:17 smithi007 ceph-mon[39249]: 3.3 scrub starts 2024-09-18T08:18:17.981 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:18:17 smithi007 ceph-mon[39249]: 3.3 scrub ok 2024-09-18T08:18:18.078 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2 2024-09-18T08:18:18.441 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:18:18.442 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:18:18.442 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2' 2024-09-18T08:18:18.543 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7 2024-09-18T08:18:18.571 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:18:18.571 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:18:18.572 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7' 2024-09-18T08:18:18.673 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5 2024-09-18T08:18:18.699 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:18:18.700 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:18:18.700 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5' 2024-09-18T08:18:18.799 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:18:18 smithi007 ceph-mon[29459]: pgmap v775: 158 pgs: 158 active+clean; 3.9 MiB data, 3.3 GiB used, 533 GiB / 536 GiB avail; 742 B/s rd, 0 B/s wr, 606 op/s 2024-09-18T08:18:18.800 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:18:18 smithi007 ceph-mon[29459]: 3.1d scrub starts 2024-09-18T08:18:18.800 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:18:18 smithi007 ceph-mon[29459]: 3.1d scrub ok 2024-09-18T08:18:18.800 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:18:18 smithi007 ceph-mon[29459]: osdmap e396: 8 total, 8 up, 6 in 2024-09-18T08:18:18.800 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:18:18 smithi007 ceph-mon[29459]: 1.2 scrub starts 2024-09-18T08:18:18.800 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:18:18 smithi007 ceph-mon[29459]: 1.2 scrub ok 2024-09-18T08:18:18.800 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:18:18 smithi007 ceph-mon[29459]: from='mgr.24470 172.21.15.7:0/2594390261' entity='mgr.y' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.8", "id": [3, 2]} : dispatch 2024-09-18T08:18:18.800 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:18:18 smithi007 ceph-mon[29459]: from='mgr.24470 172.21.15.7:0/2594390261' entity='mgr.y' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1b", "id": [3, 5]} : dispatch 2024-09-18T08:18:18.800 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:18:18 smithi007 ceph-mon[29459]: from='mgr.24470 172.21.15.7:0/2594390261' entity='mgr.y' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1d", "id": [1, 0]} : dispatch 2024-09-18T08:18:18.800 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:18:18 smithi007 ceph-mon[29459]: from='mgr.24470 172.21.15.7:0/2594390261' entity='mgr.y' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "5.b", "id": [0, 3]} : dispatch 2024-09-18T08:18:18.800 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:18:18 smithi007 ceph-mon[29459]: 3.a scrub starts 2024-09-18T08:18:18.800 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:18:18 smithi007 ceph-mon[29459]: 3.a scrub ok 2024-09-18T08:18:18.801 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:18:18 smithi007 ceph-mon[39249]: pgmap v775: 158 pgs: 158 active+clean; 3.9 MiB data, 3.3 GiB used, 533 GiB / 536 GiB avail; 742 B/s rd, 0 B/s wr, 606 op/s 2024-09-18T08:18:18.801 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:18:18 smithi007 ceph-mon[39249]: 3.1d scrub starts 2024-09-18T08:18:18.801 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:18:18 smithi007 ceph-mon[39249]: 3.1d scrub ok 2024-09-18T08:18:18.801 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:18:18 smithi007 ceph-mon[39249]: osdmap e396: 8 total, 8 up, 6 in 2024-09-18T08:18:18.801 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:18:18 smithi007 ceph-mon[39249]: 1.2 scrub starts 2024-09-18T08:18:18.801 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:18:18 smithi007 ceph-mon[39249]: 1.2 scrub ok 2024-09-18T08:18:18.801 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:18:18 smithi007 ceph-mon[39249]: from='mgr.24470 172.21.15.7:0/2594390261' entity='mgr.y' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.8", "id": [3, 2]} : dispatch 2024-09-18T08:18:18.801 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:18:18 smithi007 ceph-mon[39249]: from='mgr.24470 172.21.15.7:0/2594390261' entity='mgr.y' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1b", "id": [3, 5]} : dispatch 2024-09-18T08:18:18.801 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:18:18 smithi007 ceph-mon[39249]: from='mgr.24470 172.21.15.7:0/2594390261' entity='mgr.y' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1d", "id": [1, 0]} : dispatch 2024-09-18T08:18:18.802 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:18:18 smithi007 ceph-mon[39249]: from='mgr.24470 172.21.15.7:0/2594390261' entity='mgr.y' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "5.b", "id": [0, 3]} : dispatch 2024-09-18T08:18:18.802 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:18:18 smithi007 ceph-mon[39249]: 3.a scrub starts 2024-09-18T08:18:18.802 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:18:18 smithi007 ceph-mon[39249]: 3.a scrub ok 2024-09-18T08:18:18.802 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6 2024-09-18T08:18:18.828 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid 779349ca-7593-11ef-bceb-c7b262605968 -- ceph daemon osd.2 dump_historic_ops 2024-09-18T08:18:18.828 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:18:18.829 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:18:18.829 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6' 2024-09-18T08:18:18.853 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/779349ca-7593-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T08:18:18.905 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:18:18 smithi050 ceph-mon[32371]: pgmap v775: 158 pgs: 158 active+clean; 3.9 MiB data, 3.3 GiB used, 533 GiB / 536 GiB avail; 742 B/s rd, 0 B/s wr, 606 op/s 2024-09-18T08:18:18.905 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:18:18 smithi050 ceph-mon[32371]: 3.1d scrub starts 2024-09-18T08:18:18.905 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:18:18 smithi050 ceph-mon[32371]: 3.1d scrub ok 2024-09-18T08:18:18.905 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:18:18 smithi050 ceph-mon[32371]: osdmap e396: 8 total, 8 up, 6 in 2024-09-18T08:18:18.905 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:18:18 smithi050 ceph-mon[32371]: 1.2 scrub starts 2024-09-18T08:18:18.905 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:18:18 smithi050 ceph-mon[32371]: 1.2 scrub ok 2024-09-18T08:18:18.906 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:18:18 smithi050 ceph-mon[32371]: from='mgr.24470 172.21.15.7:0/2594390261' entity='mgr.y' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.8", "id": [3, 2]} : dispatch 2024-09-18T08:18:18.906 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:18:18 smithi050 ceph-mon[32371]: from='mgr.24470 172.21.15.7:0/2594390261' entity='mgr.y' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1b", "id": [3, 5]} : dispatch 2024-09-18T08:18:18.906 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:18:18 smithi050 ceph-mon[32371]: from='mgr.24470 172.21.15.7:0/2594390261' entity='mgr.y' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1d", "id": [1, 0]} : dispatch 2024-09-18T08:18:18.906 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:18:18 smithi050 ceph-mon[32371]: from='mgr.24470 172.21.15.7:0/2594390261' entity='mgr.y' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "5.b", "id": [0, 3]} : dispatch 2024-09-18T08:18:18.906 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:18:18 smithi050 ceph-mon[32371]: 3.a scrub starts 2024-09-18T08:18:18.906 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:18:18 smithi050 ceph-mon[32371]: 3.a scrub ok 2024-09-18T08:18:18.930 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2 2024-09-18T08:18:18.953 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:18:18.954 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:18:18.954 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2' 2024-09-18T08:18:19.055 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3 2024-09-18T08:18:19.077 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:18:19.077 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:18:19.078 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3' 2024-09-18T08:18:19.178 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6 2024-09-18T08:18:19.205 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:18:19.206 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:18:19.206 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6' 2024-09-18T08:18:19.307 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1 2024-09-18T08:18:19.332 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:18:19.333 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:18:19.333 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1' 2024-09-18T08:18:19.359 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/779349ca-7593-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T08:18:19.434 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4 2024-09-18T08:18:19.461 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:18:19.461 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:18:19.462 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4' 2024-09-18T08:18:19.563 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4 2024-09-18T08:18:19.588 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:18:19.588 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:18:19.589 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4' 2024-09-18T08:18:19.660 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid 779349ca-7593-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-18T08:18:19.689 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0 2024-09-18T08:18:19.725 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:18:19.725 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:18:19.726 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0' 2024-09-18T08:18:19.774 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:18:19 smithi007 ceph-mon[29459]: from='mgr.24470 172.21.15.7:0/2594390261' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.8", "id": [3, 2]}]': finished 2024-09-18T08:18:19.775 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:18:19 smithi007 ceph-mon[29459]: from='mgr.24470 172.21.15.7:0/2594390261' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1b", "id": [3, 5]}]': finished 2024-09-18T08:18:19.775 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:18:19 smithi007 ceph-mon[29459]: from='mgr.24470 172.21.15.7:0/2594390261' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1d", "id": [1, 0]}]': finished 2024-09-18T08:18:19.775 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:18:19 smithi007 ceph-mon[29459]: from='mgr.24470 172.21.15.7:0/2594390261' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "5.b", "id": [0, 3]}]': finished 2024-09-18T08:18:19.775 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:18:19 smithi007 ceph-mon[29459]: osdmap e397: 8 total, 8 up, 6 in 2024-09-18T08:18:19.775 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:18:19 smithi007 ceph-mon[29459]: 3.16 scrub starts 2024-09-18T08:18:19.775 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:18:19 smithi007 ceph-mon[29459]: 3.16 scrub ok 2024-09-18T08:18:19.775 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:18:19 smithi007 ceph-mon[29459]: 5.1c scrub starts 2024-09-18T08:18:19.775 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:18:19 smithi007 ceph-mon[29459]: 5.1c scrub ok 2024-09-18T08:18:19.775 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:18:19 smithi007 ceph-mon[29459]: 1.0 scrub starts 2024-09-18T08:18:19.775 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:18:19 smithi007 ceph-mon[29459]: 1.0 scrub ok 2024-09-18T08:18:19.775 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:18:19 smithi007 ceph-mon[29459]: purged_snaps scrub starts 2024-09-18T08:18:19.775 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:18:19 smithi007 ceph-mon[29459]: purged_snaps scrub ok 2024-09-18T08:18:19.776 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:18:19 smithi007 ceph-mon[29459]: from='mgr.24470 172.21.15.7:0/2594390261' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-18T08:18:19.776 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:18:19 smithi007 ceph-mon[29459]: 5.a scrub starts 2024-09-18T08:18:19.776 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:18:19 smithi007 ceph-mon[29459]: osdmap e398: 8 total, 8 up, 6 in 2024-09-18T08:18:19.776 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:18:19 smithi007 ceph-mon[29459]: from='client.? 172.21.15.7:0/1345658327' entity='client.admin' cmd={"prefix": "osd pool rm", "pool": "unique_pool_3", "pool2": "unique_pool_3", "yes_i_really_really_mean_it": true} : dispatch 2024-09-18T08:18:19.776 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:18:19 smithi007 ceph-mon[39249]: from='mgr.24470 172.21.15.7:0/2594390261' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.8", "id": [3, 2]}]': finished 2024-09-18T08:18:19.776 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:18:19 smithi007 ceph-mon[39249]: from='mgr.24470 172.21.15.7:0/2594390261' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1b", "id": [3, 5]}]': finished 2024-09-18T08:18:19.776 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:18:19 smithi007 ceph-mon[39249]: from='mgr.24470 172.21.15.7:0/2594390261' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1d", "id": [1, 0]}]': finished 2024-09-18T08:18:19.776 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:18:19 smithi007 ceph-mon[39249]: from='mgr.24470 172.21.15.7:0/2594390261' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "5.b", "id": [0, 3]}]': finished 2024-09-18T08:18:19.776 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:18:19 smithi007 ceph-mon[39249]: osdmap e397: 8 total, 8 up, 6 in 2024-09-18T08:18:19.776 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:18:19 smithi007 ceph-mon[39249]: 3.16 scrub starts 2024-09-18T08:18:19.776 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:18:19 smithi007 ceph-mon[39249]: 3.16 scrub ok 2024-09-18T08:18:19.777 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:18:19 smithi007 ceph-mon[39249]: 5.1c scrub starts 2024-09-18T08:18:19.777 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:18:19 smithi007 ceph-mon[39249]: 5.1c scrub ok 2024-09-18T08:18:19.777 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:18:19 smithi007 ceph-mon[39249]: 1.0 scrub starts 2024-09-18T08:18:19.777 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:18:19 smithi007 ceph-mon[39249]: 1.0 scrub ok 2024-09-18T08:18:19.777 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:18:19 smithi007 ceph-mon[39249]: purged_snaps scrub starts 2024-09-18T08:18:19.777 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:18:19 smithi007 ceph-mon[39249]: purged_snaps scrub ok 2024-09-18T08:18:19.777 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:18:19 smithi007 ceph-mon[39249]: from='mgr.24470 172.21.15.7:0/2594390261' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-18T08:18:19.777 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:18:19 smithi007 ceph-mon[39249]: 5.a scrub starts 2024-09-18T08:18:19.777 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:18:19 smithi007 ceph-mon[39249]: osdmap e398: 8 total, 8 up, 6 in 2024-09-18T08:18:19.777 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:18:19 smithi007 ceph-mon[39249]: from='client.? 172.21.15.7:0/1345658327' entity='client.admin' cmd={"prefix": "osd pool rm", "pool": "unique_pool_3", "pool2": "unique_pool_3", "yes_i_really_really_mean_it": true} : dispatch 2024-09-18T08:18:19.827 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5 2024-09-18T08:18:19.854 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:18:19.855 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:18:19.855 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5' 2024-09-18T08:18:19.905 INFO:journalctl@ceph.iscsi.iscsi.a.smithi050.stdout:Sep 18 08:18:19 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-iscsi-iscsi-a[57782]: debug there is no tcmu-runner data available 2024-09-18T08:18:19.906 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:18:19 smithi050 ceph-mon[32371]: from='mgr.24470 172.21.15.7:0/2594390261' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.8", "id": [3, 2]}]': finished 2024-09-18T08:18:19.906 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:18:19 smithi050 ceph-mon[32371]: from='mgr.24470 172.21.15.7:0/2594390261' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1b", "id": [3, 5]}]': finished 2024-09-18T08:18:19.906 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:18:19 smithi050 ceph-mon[32371]: from='mgr.24470 172.21.15.7:0/2594390261' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1d", "id": [1, 0]}]': finished 2024-09-18T08:18:19.906 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:18:19 smithi050 ceph-mon[32371]: from='mgr.24470 172.21.15.7:0/2594390261' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "5.b", "id": [0, 3]}]': finished 2024-09-18T08:18:19.906 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:18:19 smithi050 ceph-mon[32371]: osdmap e397: 8 total, 8 up, 6 in 2024-09-18T08:18:19.906 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:18:19 smithi050 ceph-mon[32371]: 3.16 scrub starts 2024-09-18T08:18:19.906 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:18:19 smithi050 ceph-mon[32371]: 3.16 scrub ok 2024-09-18T08:18:19.906 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:18:19 smithi050 ceph-mon[32371]: 5.1c scrub starts 2024-09-18T08:18:19.906 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:18:19 smithi050 ceph-mon[32371]: 5.1c scrub ok 2024-09-18T08:18:19.906 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:18:19 smithi050 ceph-mon[32371]: 1.0 scrub starts 2024-09-18T08:18:19.906 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:18:19 smithi050 ceph-mon[32371]: 1.0 scrub ok 2024-09-18T08:18:19.907 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:18:19 smithi050 ceph-mon[32371]: purged_snaps scrub starts 2024-09-18T08:18:19.907 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:18:19 smithi050 ceph-mon[32371]: purged_snaps scrub ok 2024-09-18T08:18:19.907 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:18:19 smithi050 ceph-mon[32371]: from='mgr.24470 172.21.15.7:0/2594390261' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-18T08:18:19.907 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:18:19 smithi050 ceph-mon[32371]: 5.a scrub starts 2024-09-18T08:18:19.907 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:18:19 smithi050 ceph-mon[32371]: osdmap e398: 8 total, 8 up, 6 in 2024-09-18T08:18:19.907 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:18:19 smithi050 ceph-mon[32371]: from='client.? 172.21.15.7:0/1345658327' entity='client.admin' cmd={"prefix": "osd pool rm", "pool": "unique_pool_3", "pool2": "unique_pool_3", "yes_i_really_really_mean_it": true} : dispatch 2024-09-18T08:18:19.941 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/779349ca-7593-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T08:18:19.944 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/779349ca-7593-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T08:18:19.956 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7 2024-09-18T08:18:19.983 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:18:19.983 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:18:19.984 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7' 2024-09-18T08:18:20.085 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6 2024-09-18T08:18:20.111 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:18:20.112 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:18:20.112 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6' 2024-09-18T08:18:20.213 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3 2024-09-18T08:18:20.236 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:18:20.236 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:18:20.237 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3' 2024-09-18T08:18:20.338 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2 2024-09-18T08:18:20.360 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:18:20.361 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:18:20.361 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2' 2024-09-18T08:18:20.435 INFO:teuthology.orchestra.run.smithi007.stderr:pool 'unique_pool_3' removed 2024-09-18T08:18:20.445 INFO:teuthology.orchestra.run.smithi007.stderr:nodeep-scrub is unset 2024-09-18T08:18:20.461 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0 2024-09-18T08:18:20.527 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:18:20.528 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:18:20.528 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0' 2024-09-18T08:18:20.629 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3 2024-09-18T08:18:20.654 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:18:20.654 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:18:20.654 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3' 2024-09-18T08:18:20.755 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2 2024-09-18T08:18:20.781 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:18:20.782 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:18:20.782 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2' 2024-09-18T08:18:20.883 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2 2024-09-18T08:18:20.905 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:18:20 smithi050 ceph-mon[32371]: 5.a scrub ok 2024-09-18T08:18:20.905 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:18:20 smithi050 ceph-mon[32371]: pgmap v779: 158 pgs: 4 peering, 154 active+clean; 583 KiB data, 3.3 GiB used, 533 GiB / 536 GiB avail; 1.5 KiB/s rd, 0 B/s wr, 14 op/s 2024-09-18T08:18:20.905 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:18:20 smithi050 ceph-mon[32371]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-18T08:18:20.905 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:18:20 smithi050 ceph-mon[32371]: from='client.? 172.21.15.7:0/3283653646' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-18T08:18:20.905 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:18:20 smithi050 ceph-mon[32371]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-18T08:18:20.905 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:18:20 smithi050 ceph-mon[32371]: Health check failed: Reduced data availability: 2 pgs inactive, 2 pgs peering (PG_AVAILABILITY) 2024-09-18T08:18:20.906 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:18:20 smithi050 ceph-mon[32371]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-18T08:18:20.906 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:18:20 smithi050 ceph-mon[32371]: Cluster is now healthy 2024-09-18T08:18:20.906 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:18:20 smithi050 ceph-mon[32371]: from='client.? 172.21.15.7:0/1345658327' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "unique_pool_3", "pool2": "unique_pool_3", "yes_i_really_really_mean_it": true}]': finished 2024-09-18T08:18:20.906 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:18:20 smithi050 ceph-mon[32371]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-18T08:18:20.906 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:18:20 smithi050 ceph-mon[32371]: osdmap e399: 8 total, 8 up, 6 in 2024-09-18T08:18:20.906 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:18:20.907 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:18:20.907 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2' 2024-09-18T08:18:20.980 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:18:20 smithi007 ceph-mon[29459]: 5.a scrub ok 2024-09-18T08:18:20.980 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:18:20 smithi007 ceph-mon[29459]: pgmap v779: 158 pgs: 4 peering, 154 active+clean; 583 KiB data, 3.3 GiB used, 533 GiB / 536 GiB avail; 1.5 KiB/s rd, 0 B/s wr, 14 op/s 2024-09-18T08:18:20.980 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:18:20 smithi007 ceph-mon[29459]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-18T08:18:20.980 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:18:20 smithi007 ceph-mon[29459]: from='client.? 172.21.15.7:0/3283653646' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-18T08:18:20.980 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:18:20 smithi007 ceph-mon[29459]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-18T08:18:20.980 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:18:20 smithi007 ceph-mon[29459]: Health check failed: Reduced data availability: 2 pgs inactive, 2 pgs peering (PG_AVAILABILITY) 2024-09-18T08:18:20.980 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:18:20 smithi007 ceph-mon[29459]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-18T08:18:20.980 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:18:20 smithi007 ceph-mon[29459]: Cluster is now healthy 2024-09-18T08:18:20.980 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:18:20 smithi007 ceph-mon[29459]: from='client.? 172.21.15.7:0/1345658327' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "unique_pool_3", "pool2": "unique_pool_3", "yes_i_really_really_mean_it": true}]': finished 2024-09-18T08:18:20.980 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:18:20 smithi007 ceph-mon[29459]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-18T08:18:20.981 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:18:20 smithi007 ceph-mon[29459]: osdmap e399: 8 total, 8 up, 6 in 2024-09-18T08:18:20.981 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:18:20 smithi007 ceph-mon[39249]: 5.a scrub ok 2024-09-18T08:18:20.981 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:18:20 smithi007 ceph-mon[39249]: pgmap v779: 158 pgs: 4 peering, 154 active+clean; 583 KiB data, 3.3 GiB used, 533 GiB / 536 GiB avail; 1.5 KiB/s rd, 0 B/s wr, 14 op/s 2024-09-18T08:18:20.981 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:18:20 smithi007 ceph-mon[39249]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-18T08:18:20.981 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:18:20 smithi007 ceph-mon[39249]: from='client.? 172.21.15.7:0/3283653646' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-18T08:18:20.981 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:18:20 smithi007 ceph-mon[39249]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-18T08:18:20.981 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:18:20 smithi007 ceph-mon[39249]: Health check failed: Reduced data availability: 2 pgs inactive, 2 pgs peering (PG_AVAILABILITY) 2024-09-18T08:18:20.981 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:18:20 smithi007 ceph-mon[39249]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-18T08:18:20.981 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:18:20 smithi007 ceph-mon[39249]: Cluster is now healthy 2024-09-18T08:18:20.981 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:18:20 smithi007 ceph-mon[39249]: from='client.? 172.21.15.7:0/1345658327' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "unique_pool_3", "pool2": "unique_pool_3", "yes_i_really_really_mean_it": true}]': finished 2024-09-18T08:18:20.982 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:18:20 smithi007 ceph-mon[39249]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-18T08:18:20.982 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:18:20 smithi007 ceph-mon[39249]: osdmap e399: 8 total, 8 up, 6 in 2024-09-18T08:18:21.008 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1 2024-09-18T08:18:21.031 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:18:21.031 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:18:21.032 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1' 2024-09-18T08:18:21.132 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3 2024-09-18T08:18:21.154 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:18:21.155 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:18:21.155 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3' 2024-09-18T08:18:21.256 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1 2024-09-18T08:18:21.278 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:18:21.279 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:18:21.279 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1' 2024-09-18T08:18:21.380 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2 2024-09-18T08:18:21.402 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:18:21.402 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:18:21.403 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2' 2024-09-18T08:18:21.494 INFO:teuthology.orchestra.run.smithi007.stdout:{ 2024-09-18T08:18:21.495 INFO:teuthology.orchestra.run.smithi007.stdout: "size": 20, 2024-09-18T08:18:21.495 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 600, 2024-09-18T08:18:21.495 INFO:teuthology.orchestra.run.smithi007.stdout: "ops": [ 2024-09-18T08:18:21.495 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:18:21.495 INFO:teuthology.orchestra.run.smithi007.stdout: "description": "osd_op(client.15582.0:53557 9.0 9:0cf6806d:::benchmark_data_smithi007_124019_object53556:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e244)", 2024-09-18T08:18:21.495 INFO:teuthology.orchestra.run.smithi007.stdout: "initiated_at": "2024-09-18T08:11:41.343691+0000", 2024-09-18T08:18:21.495 INFO:teuthology.orchestra.run.smithi007.stdout: "age": 399.42163947099999, 2024-09-18T08:18:21.495 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.98520017699999995, 2024-09-18T08:18:21.495 INFO:teuthology.orchestra.run.smithi007.stdout: "continuous": false, 2024-09-18T08:18:21.495 INFO:teuthology.orchestra.run.smithi007.stdout: "type_data": { 2024-09-18T08:18:21.496 INFO:teuthology.orchestra.run.smithi007.stdout: "flag_point": "reached pg", 2024-09-18T08:18:21.496 INFO:teuthology.orchestra.run.smithi007.stdout: "client_info": { 2024-09-18T08:18:21.496 INFO:teuthology.orchestra.run.smithi007.stdout: "client": "client.15582", 2024-09-18T08:18:21.496 INFO:teuthology.orchestra.run.smithi007.stdout: "client_addr": "172.21.15.7:0/3598605839", 2024-09-18T08:18:21.496 INFO:teuthology.orchestra.run.smithi007.stdout: "tid": 53557 2024-09-18T08:18:21.496 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:18:21.496 INFO:teuthology.orchestra.run.smithi007.stdout: "events": [ 2024-09-18T08:18:21.496 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:18:21.496 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "initiated", 2024-09-18T08:18:21.496 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:11:41.343691+0000", 2024-09-18T08:18:21.496 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-09-18T08:18:21.497 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:18:21.497 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:18:21.497 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "header_read", 2024-09-18T08:18:21.497 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:11:41.343691+0000", 2024-09-18T08:18:21.497 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-09-18T08:18:21.497 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:18:21.497 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:18:21.497 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "throttled", 2024-09-18T08:18:21.497 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:11:41.343693+0000", 2024-09-18T08:18:21.497 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.804e-06 2024-09-18T08:18:21.497 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:18:21.498 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:18:21.498 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "all_read", 2024-09-18T08:18:21.498 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:11:41.343759+0000", 2024-09-18T08:18:21.498 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 6.6369000000000001e-05 2024-09-18T08:18:21.498 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:18:21.498 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:18:21.498 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "dispatched", 2024-09-18T08:18:21.498 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:11:41.343774+0000", 2024-09-18T08:18:21.498 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.5489999999999999e-05 2024-09-18T08:18:21.498 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:18:21.498 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:18:21.498 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "queued_for_pg", 2024-09-18T08:18:21.499 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:11:41.343785+0000", 2024-09-18T08:18:21.499 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.078e-05 2024-09-18T08:18:21.499 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:18:21.499 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:18:21.499 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-09-18T08:18:21.499 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:11:41.344215+0000", 2024-09-18T08:18:21.499 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.00042940799999999999 2024-09-18T08:18:21.499 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:18:21.499 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:18:21.499 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "waiting for peered", 2024-09-18T08:18:21.499 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:11:41.344258+0000", 2024-09-18T08:18:21.499 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 4.3460000000000001e-05 2024-09-18T08:18:21.500 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:18:21.500 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:18:21.500 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-09-18T08:18:21.500 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:11:42.326886+0000", 2024-09-18T08:18:21.500 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.98262749800000004 2024-09-18T08:18:21.500 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:18:21.500 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:18:21.500 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "done", 2024-09-18T08:18:21.500 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:11:42.328891+0000", 2024-09-18T08:18:21.500 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.002005368 2024-09-18T08:18:21.500 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-09-18T08:18:21.501 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-09-18T08:18:21.501 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-09-18T08:18:21.501 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:18:21.501 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:18:21.501 INFO:teuthology.orchestra.run.smithi007.stdout: "description": "osd_op(client.15582.0:53566 9.0 9:09a05dd0:::benchmark_data_smithi007_124019_object53565:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e244)", 2024-09-18T08:18:21.501 INFO:teuthology.orchestra.run.smithi007.stdout: "initiated_at": "2024-09-18T08:11:41.343807+0000", 2024-09-18T08:18:21.501 INFO:teuthology.orchestra.run.smithi007.stdout: "age": 399.421523643, 2024-09-18T08:18:21.501 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.007200514, 2024-09-18T08:18:21.501 INFO:teuthology.orchestra.run.smithi007.stdout: "continuous": false, 2024-09-18T08:18:21.501 INFO:teuthology.orchestra.run.smithi007.stdout: "type_data": { 2024-09-18T08:18:21.502 INFO:teuthology.orchestra.run.smithi007.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T08:18:21.502 INFO:teuthology.orchestra.run.smithi007.stdout: "client_info": { 2024-09-18T08:18:21.502 INFO:teuthology.orchestra.run.smithi007.stdout: "client": "client.15582", 2024-09-18T08:18:21.502 INFO:teuthology.orchestra.run.smithi007.stdout: "client_addr": "172.21.15.7:0/3598605839", 2024-09-18T08:18:21.502 INFO:teuthology.orchestra.run.smithi007.stdout: "tid": 53566 2024-09-18T08:18:21.502 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:18:21.502 INFO:teuthology.orchestra.run.smithi007.stdout: "events": [ 2024-09-18T08:18:21.502 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:18:21.502 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "initiated", 2024-09-18T08:18:21.502 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:11:41.343807+0000", 2024-09-18T08:18:21.502 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-09-18T08:18:21.502 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:18:21.503 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:18:21.503 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "header_read", 2024-09-18T08:18:21.503 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:11:41.343807+0000", 2024-09-18T08:18:21.503 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-09-18T08:18:21.503 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:18:21.503 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:18:21.503 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "throttled", 2024-09-18T08:18:21.503 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:11:41.343807+0000", 2024-09-18T08:18:21.503 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 5.3000000000000001e-07 2024-09-18T08:18:21.503 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:18:21.503 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:18:21.503 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "all_read", 2024-09-18T08:18:21.504 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:11:41.343826+0000", 2024-09-18T08:18:21.504 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.9338999999999999e-05 2024-09-18T08:18:21.504 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:18:21.504 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:18:21.504 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "dispatched", 2024-09-18T08:18:21.504 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:11:41.343833+0000", 2024-09-18T08:18:21.504 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 6.7710000000000001e-06 2024-09-18T08:18:21.504 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:18:21.504 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:18:21.504 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "queued_for_pg", 2024-09-18T08:18:21.504 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:11:41.343839+0000", 2024-09-18T08:18:21.504 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 5.4269999999999999e-06 2024-09-18T08:18:21.505 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:18:21.505 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:18:21.505 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-09-18T08:18:21.505 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:11:41.344332+0000", 2024-09-18T08:18:21.505 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.00049369799999999999 2024-09-18T08:18:21.505 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:18:21.505 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:18:21.505 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "waiting for peered", 2024-09-18T08:18:21.505 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:11:41.344339+0000", 2024-09-18T08:18:21.505 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 6.8589999999999999e-06 2024-09-18T08:18:21.505 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:18:21.506 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:18:21.506 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-09-18T08:18:21.506 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:11:42.328960+0000", 2024-09-18T08:18:21.506 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.98462126599999999 2024-09-18T08:18:21.506 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:18:21.506 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:18:21.506 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "started", 2024-09-18T08:18:21.506 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:11:42.329310+0000", 2024-09-18T08:18:21.506 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.00034983799999999999 2024-09-18T08:18:21.506 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:18:21.506 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:18:21.506 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "waiting for subops from 6", 2024-09-18T08:18:21.507 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:11:42.330445+0000", 2024-09-18T08:18:21.507 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.0011349859999999999 2024-09-18T08:18:21.507 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:18:21.507 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:18:21.507 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "op_commit", 2024-09-18T08:18:21.507 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:11:42.339902+0000", 2024-09-18T08:18:21.507 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.0094567869999999995 2024-09-18T08:18:21.507 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:18:21.507 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:18:21.507 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:18:21.507 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:11:42.350663+0000", 2024-09-18T08:18:21.508 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.010761299 2024-09-18T08:18:21.508 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:18:21.508 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:18:21.508 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "commit_sent", 2024-09-18T08:18:21.508 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:11:42.350832+0000", 2024-09-18T08:18:21.508 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.00016828399999999999 2024-09-18T08:18:21.508 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:18:21.508 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:18:21.508 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "done", 2024-09-18T08:18:21.508 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:11:42.351007+0000", 2024-09-18T08:18:21.508 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.00017542999999999999 2024-09-18T08:18:21.509 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-09-18T08:18:21.509 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-09-18T08:18:21.509 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-09-18T08:18:21.509 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:18:21.509 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:18:21.509 INFO:teuthology.orchestra.run.smithi007.stdout: "description": "osd_op(client.15582.0:53575 9.0 9:0f4fbab7:::benchmark_data_smithi007_124019_object53574:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e244)", 2024-09-18T08:18:21.509 INFO:teuthology.orchestra.run.smithi007.stdout: "initiated_at": "2024-09-18T08:11:41.343854+0000", 2024-09-18T08:18:21.509 INFO:teuthology.orchestra.run.smithi007.stdout: "age": 399.42147620100002, 2024-09-18T08:18:21.509 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.0074121789999999, 2024-09-18T08:18:21.509 INFO:teuthology.orchestra.run.smithi007.stdout: "continuous": false, 2024-09-18T08:18:21.509 INFO:teuthology.orchestra.run.smithi007.stdout: "type_data": { 2024-09-18T08:18:21.510 INFO:teuthology.orchestra.run.smithi007.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T08:18:21.510 INFO:teuthology.orchestra.run.smithi007.stdout: "client_info": { 2024-09-18T08:18:21.510 INFO:teuthology.orchestra.run.smithi007.stdout: "client": "client.15582", 2024-09-18T08:18:21.510 INFO:teuthology.orchestra.run.smithi007.stdout: "client_addr": "172.21.15.7:0/3598605839", 2024-09-18T08:18:21.510 INFO:teuthology.orchestra.run.smithi007.stdout: "tid": 53575 2024-09-18T08:18:21.510 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:18:21.510 INFO:teuthology.orchestra.run.smithi007.stdout: "events": [ 2024-09-18T08:18:21.510 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:18:21.510 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "initiated", 2024-09-18T08:18:21.510 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:11:41.343854+0000", 2024-09-18T08:18:21.510 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-09-18T08:18:21.510 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:18:21.511 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:18:21.511 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "header_read", 2024-09-18T08:18:21.511 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:11:41.343854+0000", 2024-09-18T08:18:21.511 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-09-18T08:18:21.511 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:18:21.511 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:18:21.511 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "throttled", 2024-09-18T08:18:21.511 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:11:41.343855+0000", 2024-09-18T08:18:21.511 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 5.5000000000000003e-07 2024-09-18T08:18:21.511 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:18:21.511 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:18:21.512 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "all_read", 2024-09-18T08:18:21.512 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:11:41.343878+0000", 2024-09-18T08:18:21.512 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 2.3740999999999999e-05 2024-09-18T08:18:21.512 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:18:21.512 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:18:21.512 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "dispatched", 2024-09-18T08:18:21.512 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:11:41.343886+0000", 2024-09-18T08:18:21.512 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 7.8809999999999996e-06 2024-09-18T08:18:21.512 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:18:21.512 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:18:21.512 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "queued_for_pg", 2024-09-18T08:18:21.513 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:11:41.343892+0000", 2024-09-18T08:18:21.513 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 5.8200000000000002e-06 2024-09-18T08:18:21.513 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:18:21.513 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:18:21.513 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-09-18T08:18:21.513 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:11:41.344821+0000", 2024-09-18T08:18:21.513 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.000928874 2024-09-18T08:18:21.513 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:18:21.513 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:18:21.513 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "waiting for peered", 2024-09-18T08:18:21.513 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:11:41.344861+0000", 2024-09-18T08:18:21.514 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 3.9919999999999997e-05 2024-09-18T08:18:21.514 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:18:21.514 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:18:21.514 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-09-18T08:18:21.514 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:11:42.334161+0000", 2024-09-18T08:18:21.514 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.98930047799999998 2024-09-18T08:18:21.514 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:18:21.514 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:18:21.514 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "started", 2024-09-18T08:18:21.514 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:11:42.334487+0000", 2024-09-18T08:18:21.514 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.00032586300000000002 2024-09-18T08:18:21.514 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:18:21.515 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:18:21.515 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "waiting for subops from 6", 2024-09-18T08:18:21.515 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:11:42.334655+0000", 2024-09-18T08:18:21.515 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.000167419 2024-09-18T08:18:21.515 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:18:21.515 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:18:21.515 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "op_commit", 2024-09-18T08:18:21.515 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:11:42.346631+0000", 2024-09-18T08:18:21.515 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.011976496 2024-09-18T08:18:21.515 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:18:21.515 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:18:21.516 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:18:21.516 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:11:42.351115+0000", 2024-09-18T08:18:21.516 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.0044843590000000003 2024-09-18T08:18:21.516 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:18:21.516 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:18:21.516 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "commit_sent", 2024-09-18T08:18:21.516 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:11:42.351173+0000", 2024-09-18T08:18:21.516 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 5.7061000000000001e-05 2024-09-18T08:18:21.516 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:18:21.516 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:18:21.516 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "done", 2024-09-18T08:18:21.517 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:11:42.351266+0000", 2024-09-18T08:18:21.517 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 9.3716999999999997e-05 2024-09-18T08:18:21.517 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-09-18T08:18:21.517 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-09-18T08:18:21.517 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-09-18T08:18:21.517 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:18:21.517 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:18:21.517 INFO:teuthology.orchestra.run.smithi007.stdout: "description": "osd_op(client.15582.0:53576 9.0 9:0f9d6982:::benchmark_data_smithi007_124019_object53575:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e244)", 2024-09-18T08:18:21.517 INFO:teuthology.orchestra.run.smithi007.stdout: "initiated_at": "2024-09-18T08:11:41.343901+0000", 2024-09-18T08:18:21.517 INFO:teuthology.orchestra.run.smithi007.stdout: "age": 399.42142898100002, 2024-09-18T08:18:21.517 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.01002054, 2024-09-18T08:18:21.518 INFO:teuthology.orchestra.run.smithi007.stdout: "continuous": false, 2024-09-18T08:18:21.518 INFO:teuthology.orchestra.run.smithi007.stdout: "type_data": { 2024-09-18T08:18:21.518 INFO:teuthology.orchestra.run.smithi007.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T08:18:21.518 INFO:teuthology.orchestra.run.smithi007.stdout: "client_info": { 2024-09-18T08:18:21.518 INFO:teuthology.orchestra.run.smithi007.stdout: "client": "client.15582", 2024-09-18T08:18:21.518 INFO:teuthology.orchestra.run.smithi007.stdout: "client_addr": "172.21.15.7:0/3598605839", 2024-09-18T08:18:21.518 INFO:teuthology.orchestra.run.smithi007.stdout: "tid": 53576 2024-09-18T08:18:21.518 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:18:21.518 INFO:teuthology.orchestra.run.smithi007.stdout: "events": [ 2024-09-18T08:18:21.518 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:18:21.518 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "initiated", 2024-09-18T08:18:21.519 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:11:41.343901+0000", 2024-09-18T08:18:21.519 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-09-18T08:18:21.519 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:18:21.519 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:18:21.519 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "header_read", 2024-09-18T08:18:21.519 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:11:41.343901+0000", 2024-09-18T08:18:21.519 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-09-18T08:18:21.519 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:18:21.519 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:18:21.519 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "throttled", 2024-09-18T08:18:21.519 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:11:41.343902+0000", 2024-09-18T08:18:21.520 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 4.7100000000000002e-07 2024-09-18T08:18:21.520 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:18:21.520 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:18:21.520 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "all_read", 2024-09-18T08:18:21.520 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:11:41.343923+0000", 2024-09-18T08:18:21.520 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 2.1009000000000002e-05 2024-09-18T08:18:21.520 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:18:21.520 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:18:21.520 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "dispatched", 2024-09-18T08:18:21.520 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:11:41.343930+0000", 2024-09-18T08:18:21.520 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 7.0040000000000002e-06 2024-09-18T08:18:21.521 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:18:21.521 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:18:21.521 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "queued_for_pg", 2024-09-18T08:18:21.521 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:11:41.343935+0000", 2024-09-18T08:18:21.521 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 5.0549999999999999e-06 2024-09-18T08:18:21.521 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:18:21.521 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:18:21.521 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-09-18T08:18:21.521 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:11:41.345163+0000", 2024-09-18T08:18:21.522 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.0012281200000000001 2024-09-18T08:18:21.522 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:18:21.522 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:18:21.522 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "waiting for peered", 2024-09-18T08:18:21.522 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:11:41.345169+0000", 2024-09-18T08:18:21.522 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 5.6999999999999996e-06 2024-09-18T08:18:21.522 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:18:21.522 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:18:21.522 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-09-18T08:18:21.522 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:11:42.336827+0000", 2024-09-18T08:18:21.522 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.99165844599999997 2024-09-18T08:18:21.523 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:18:21.523 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:18:21.523 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "started", 2024-09-18T08:18:21.523 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:11:42.336957+0000", 2024-09-18T08:18:21.523 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.00013009 2024-09-18T08:18:21.523 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:18:21.523 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:18:21.523 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "waiting for subops from 6", 2024-09-18T08:18:21.523 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:11:42.337097+0000", 2024-09-18T08:18:21.523 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.00014021899999999999 2024-09-18T08:18:21.523 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:18:21.523 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:18:21.523 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "op_commit", 2024-09-18T08:18:21.524 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:11:42.346639+0000", 2024-09-18T08:18:21.524 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.0095413370000000004 2024-09-18T08:18:21.524 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:18:21.524 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:18:21.524 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:18:21.524 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:11:42.353750+0000", 2024-09-18T08:18:21.524 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.0071108530000000003 2024-09-18T08:18:21.524 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:18:21.524 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:18:21.524 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "commit_sent", 2024-09-18T08:18:21.524 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:11:42.353813+0000", 2024-09-18T08:18:21.525 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 6.3721999999999997e-05 2024-09-18T08:18:21.525 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:18:21.525 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:18:21.525 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "done", 2024-09-18T08:18:21.525 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:11:42.353922+0000", 2024-09-18T08:18:21.525 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.000108514 2024-09-18T08:18:21.525 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-09-18T08:18:21.525 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-09-18T08:18:21.525 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-09-18T08:18:21.525 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:18:21.525 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:18:21.525 INFO:teuthology.orchestra.run.smithi007.stdout: "description": "osd_op(client.24356.0:2782 5.d 5:b4812045:::notify.4:head [watch reconnect cookie 94700888684160 gen 1] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e292)", 2024-09-18T08:18:21.526 INFO:teuthology.orchestra.run.smithi007.stdout: "initiated_at": "2024-09-18T08:14:04.565391+0000", 2024-09-18T08:18:21.526 INFO:teuthology.orchestra.run.smithi007.stdout: "age": 256.19993899999997, 2024-09-18T08:18:21.526 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.0413668460000001, 2024-09-18T08:18:21.526 INFO:teuthology.orchestra.run.smithi007.stdout: "continuous": false, 2024-09-18T08:18:21.526 INFO:teuthology.orchestra.run.smithi007.stdout: "type_data": { 2024-09-18T08:18:21.526 INFO:teuthology.orchestra.run.smithi007.stdout: "flag_point": "started", 2024-09-18T08:18:21.526 INFO:teuthology.orchestra.run.smithi007.stdout: "client_info": { 2024-09-18T08:18:21.526 INFO:teuthology.orchestra.run.smithi007.stdout: "client": "client.24356", 2024-09-18T08:18:21.526 INFO:teuthology.orchestra.run.smithi007.stdout: "client_addr": "172.21.15.7:0/219168083", 2024-09-18T08:18:21.526 INFO:teuthology.orchestra.run.smithi007.stdout: "tid": 2782 2024-09-18T08:18:21.526 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:18:21.526 INFO:teuthology.orchestra.run.smithi007.stdout: "events": [ 2024-09-18T08:18:21.527 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:18:21.527 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "initiated", 2024-09-18T08:18:21.527 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:14:04.565391+0000", 2024-09-18T08:18:21.527 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-09-18T08:18:21.527 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:18:21.527 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:18:21.527 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "header_read", 2024-09-18T08:18:21.527 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:14:04.565391+0000", 2024-09-18T08:18:21.527 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-09-18T08:18:21.527 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:18:21.528 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:18:21.528 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "throttled", 2024-09-18T08:18:21.528 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:14:04.565393+0000", 2024-09-18T08:18:21.528 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.8199999999999999e-06 2024-09-18T08:18:21.528 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:18:21.528 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:18:21.528 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "all_read", 2024-09-18T08:18:21.528 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:14:04.565400+0000", 2024-09-18T08:18:21.528 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 6.968e-06 2024-09-18T08:18:21.528 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:18:21.528 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:18:21.529 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "dispatched", 2024-09-18T08:18:21.529 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:14:04.565413+0000", 2024-09-18T08:18:21.529 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.3003000000000001e-05 2024-09-18T08:18:21.529 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:18:21.529 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:18:21.529 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "queued_for_pg", 2024-09-18T08:18:21.529 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:14:04.565422+0000", 2024-09-18T08:18:21.529 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 9.3870000000000001e-06 2024-09-18T08:18:21.529 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:18:21.529 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:18:21.529 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-09-18T08:18:21.530 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:14:04.565500+0000", 2024-09-18T08:18:21.530 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 7.7395e-05 2024-09-18T08:18:21.530 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:18:21.530 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:18:21.530 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "op must wait for map", 2024-09-18T08:18:21.530 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:14:04.565506+0000", 2024-09-18T08:18:21.530 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 6.5180000000000001e-06 2024-09-18T08:18:21.530 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:18:21.530 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:18:21.530 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-09-18T08:18:21.531 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:14:04.576465+0000", 2024-09-18T08:18:21.531 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.010958183 2024-09-18T08:18:21.531 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:18:21.531 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:18:21.531 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "waiting for peered", 2024-09-18T08:18:21.531 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:14:04.576476+0000", 2024-09-18T08:18:21.531 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.1877e-05 2024-09-18T08:18:21.531 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:18:21.531 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:18:21.531 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-09-18T08:18:21.531 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:14:05.594014+0000", 2024-09-18T08:18:21.531 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.0175379360000001 2024-09-18T08:18:21.532 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:18:21.532 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:18:21.532 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "waiting for degraded object", 2024-09-18T08:18:21.532 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:14:05.594307+0000", 2024-09-18T08:18:21.532 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.00029306699999999999 2024-09-18T08:18:21.532 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:18:21.532 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:18:21.532 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-09-18T08:18:21.532 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:14:05.595721+0000", 2024-09-18T08:18:21.532 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.001413826 2024-09-18T08:18:21.532 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:18:21.533 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:18:21.533 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "started", 2024-09-18T08:18:21.533 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:14:05.595822+0000", 2024-09-18T08:18:21.533 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.000100814 2024-09-18T08:18:21.533 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:18:21.533 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:18:21.533 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "done", 2024-09-18T08:18:21.533 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:14:05.606758+0000", 2024-09-18T08:18:21.533 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.010936052 2024-09-18T08:18:21.533 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-09-18T08:18:21.533 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-09-18T08:18:21.534 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-09-18T08:18:21.534 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:18:21.534 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:18:21.534 INFO:teuthology.orchestra.run.smithi007.stdout: "description": "osd_op(client.15966.0:9476 10.9 10:9f7935f1:::benchmark_data_smithi007_148834_object9475:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e316)", 2024-09-18T08:18:21.534 INFO:teuthology.orchestra.run.smithi007.stdout: "initiated_at": "2024-09-18T08:14:51.654386+0000", 2024-09-18T08:18:21.534 INFO:teuthology.orchestra.run.smithi007.stdout: "age": 209.11094458700001, 2024-09-18T08:18:21.534 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 2.0447284200000002, 2024-09-18T08:18:21.534 INFO:teuthology.orchestra.run.smithi007.stdout: "continuous": false, 2024-09-18T08:18:21.534 INFO:teuthology.orchestra.run.smithi007.stdout: "type_data": { 2024-09-18T08:18:21.534 INFO:teuthology.orchestra.run.smithi007.stdout: "flag_point": "reached pg", 2024-09-18T08:18:21.534 INFO:teuthology.orchestra.run.smithi007.stdout: "client_info": { 2024-09-18T08:18:21.534 INFO:teuthology.orchestra.run.smithi007.stdout: "client": "client.15966", 2024-09-18T08:18:21.535 INFO:teuthology.orchestra.run.smithi007.stdout: "client_addr": "172.21.15.7:0/529784850", 2024-09-18T08:18:21.535 INFO:teuthology.orchestra.run.smithi007.stdout: "tid": 9476 2024-09-18T08:18:21.535 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:18:21.535 INFO:teuthology.orchestra.run.smithi007.stdout: "events": [ 2024-09-18T08:18:21.535 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:18:21.535 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "initiated", 2024-09-18T08:18:21.535 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:14:51.654386+0000", 2024-09-18T08:18:21.535 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-09-18T08:18:21.535 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:18:21.535 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:18:21.535 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "header_read", 2024-09-18T08:18:21.536 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:14:51.654386+0000", 2024-09-18T08:18:21.536 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-09-18T08:18:21.536 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:18:21.536 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:18:21.536 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "throttled", 2024-09-18T08:18:21.536 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:14:51.654387+0000", 2024-09-18T08:18:21.536 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.714e-06 2024-09-18T08:18:21.536 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:18:21.536 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:18:21.536 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "all_read", 2024-09-18T08:18:21.536 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:14:51.654415+0000", 2024-09-18T08:18:21.537 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 2.7216000000000002e-05 2024-09-18T08:18:21.537 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:18:21.537 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:18:21.537 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "dispatched", 2024-09-18T08:18:21.537 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:14:51.654427+0000", 2024-09-18T08:18:21.537 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.2340999999999999e-05 2024-09-18T08:18:21.537 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:18:21.537 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:18:21.537 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "queued_for_pg", 2024-09-18T08:18:21.537 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:14:51.654438+0000", 2024-09-18T08:18:21.538 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.0591000000000001e-05 2024-09-18T08:18:21.538 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:18:21.538 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:18:21.538 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-09-18T08:18:21.538 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:14:51.667444+0000", 2024-09-18T08:18:21.538 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.013006734000000001 2024-09-18T08:18:21.538 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:18:21.538 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:18:21.538 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "waiting for peered", 2024-09-18T08:18:21.538 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:14:51.667454+0000", 2024-09-18T08:18:21.538 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.0095000000000001e-05 2024-09-18T08:18:21.538 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:18:21.539 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:18:21.539 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-09-18T08:18:21.539 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:14:52.671410+0000", 2024-09-18T08:18:21.539 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.0039560489999999 2024-09-18T08:18:21.539 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:18:21.539 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:18:21.539 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "waiting for peered", 2024-09-18T08:18:21.539 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:14:52.671421+0000", 2024-09-18T08:18:21.539 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.0103e-05 2024-09-18T08:18:21.539 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:18:21.539 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:18:21.540 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-09-18T08:18:21.540 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:14:53.698810+0000", 2024-09-18T08:18:21.540 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.0273892680000001 2024-09-18T08:18:21.540 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:18:21.540 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:18:21.540 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "done", 2024-09-18T08:18:21.540 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:14:53.699114+0000", 2024-09-18T08:18:21.540 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.000304309 2024-09-18T08:18:21.540 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-09-18T08:18:21.540 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-09-18T08:18:21.540 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-09-18T08:18:21.540 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:18:21.541 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:18:21.541 INFO:teuthology.orchestra.run.smithi007.stdout: "description": "osd_op(client.15966.0:9493 10.9 10:99175b59:::benchmark_data_smithi007_148834_object9492:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e316)", 2024-09-18T08:18:21.541 INFO:teuthology.orchestra.run.smithi007.stdout: "initiated_at": "2024-09-18T08:14:51.668505+0000", 2024-09-18T08:18:21.541 INFO:teuthology.orchestra.run.smithi007.stdout: "age": 209.096825091, 2024-09-18T08:18:21.541 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 2.0445903859999999, 2024-09-18T08:18:21.541 INFO:teuthology.orchestra.run.smithi007.stdout: "continuous": false, 2024-09-18T08:18:21.541 INFO:teuthology.orchestra.run.smithi007.stdout: "type_data": { 2024-09-18T08:18:21.541 INFO:teuthology.orchestra.run.smithi007.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T08:18:21.541 INFO:teuthology.orchestra.run.smithi007.stdout: "client_info": { 2024-09-18T08:18:21.541 INFO:teuthology.orchestra.run.smithi007.stdout: "client": "client.15966", 2024-09-18T08:18:21.541 INFO:teuthology.orchestra.run.smithi007.stdout: "client_addr": "172.21.15.7:0/529784850", 2024-09-18T08:18:21.542 INFO:teuthology.orchestra.run.smithi007.stdout: "tid": 9493 2024-09-18T08:18:21.542 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:18:21.542 INFO:teuthology.orchestra.run.smithi007.stdout: "events": [ 2024-09-18T08:18:21.542 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:18:21.542 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "initiated", 2024-09-18T08:18:21.542 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:14:51.668505+0000", 2024-09-18T08:18:21.542 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-09-18T08:18:21.542 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:18:21.542 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:18:21.542 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "header_read", 2024-09-18T08:18:21.543 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:14:51.668505+0000", 2024-09-18T08:18:21.543 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-09-18T08:18:21.543 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:18:21.543 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:18:21.543 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "throttled", 2024-09-18T08:18:21.543 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:14:51.668507+0000", 2024-09-18T08:18:21.543 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.7999999999999999e-06 2024-09-18T08:18:21.543 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:18:21.543 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:18:21.543 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "all_read", 2024-09-18T08:18:21.543 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:14:51.668533+0000", 2024-09-18T08:18:21.543 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 2.6472e-05 2024-09-18T08:18:21.543 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:18:21.544 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:18:21.544 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "dispatched", 2024-09-18T08:18:21.544 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:14:51.668546+0000", 2024-09-18T08:18:21.544 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.3064999999999999e-05 2024-09-18T08:18:21.544 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:18:21.544 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:18:21.544 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "queued_for_pg", 2024-09-18T08:18:21.544 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:14:51.668630+0000", 2024-09-18T08:18:21.544 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 8.3813000000000005e-05 2024-09-18T08:18:21.544 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:18:21.545 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:18:21.545 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-09-18T08:18:21.545 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:14:51.668810+0000", 2024-09-18T08:18:21.545 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.00018000500000000001 2024-09-18T08:18:21.545 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:18:21.545 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:18:21.545 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "waiting for peered", 2024-09-18T08:18:21.545 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:14:51.668820+0000", 2024-09-18T08:18:21.545 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 9.8770000000000005e-06 2024-09-18T08:18:21.545 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:18:21.545 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:18:21.546 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-09-18T08:18:21.546 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:14:52.671448+0000", 2024-09-18T08:18:21.546 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.0026282019999999 2024-09-18T08:18:21.546 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:18:21.546 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:18:21.546 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "waiting for peered", 2024-09-18T08:18:21.546 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:14:52.671457+0000", 2024-09-18T08:18:21.546 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 8.4859999999999997e-06 2024-09-18T08:18:21.546 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:18:21.546 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:18:21.547 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-09-18T08:18:21.547 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:14:53.699656+0000", 2024-09-18T08:18:21.547 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.0281992200000001 2024-09-18T08:18:21.547 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:18:21.547 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:18:21.547 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "started", 2024-09-18T08:18:21.547 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:14:53.699827+0000", 2024-09-18T08:18:21.547 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.00017075200000000001 2024-09-18T08:18:21.547 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:18:21.547 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:18:21.547 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "waiting for subops from 3,4", 2024-09-18T08:18:21.547 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:14:53.700023+0000", 2024-09-18T08:18:21.548 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.00019650399999999999 2024-09-18T08:18:21.548 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:18:21.548 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:18:21.548 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:18:21.548 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:14:53.701118+0000", 2024-09-18T08:18:21.548 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.0010951069999999999 2024-09-18T08:18:21.548 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:18:21.548 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:18:21.548 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:18:21.548 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:14:53.702736+0000", 2024-09-18T08:18:21.549 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.001617063 2024-09-18T08:18:21.549 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:18:21.549 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:18:21.549 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "op_commit", 2024-09-18T08:18:21.549 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:14:53.712454+0000", 2024-09-18T08:18:21.549 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.0097184779999999991 2024-09-18T08:18:21.549 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:18:21.549 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:18:21.549 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "commit_sent", 2024-09-18T08:18:21.549 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:14:53.712512+0000", 2024-09-18T08:18:21.549 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 5.7602999999999997e-05 2024-09-18T08:18:21.549 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:18:21.550 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:18:21.550 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "done", 2024-09-18T08:18:21.550 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:14:53.713096+0000", 2024-09-18T08:18:21.550 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.00058393900000000001 2024-09-18T08:18:21.550 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-09-18T08:18:21.550 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-09-18T08:18:21.550 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-09-18T08:18:21.550 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:18:21.550 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:18:21.550 INFO:teuthology.orchestra.run.smithi007.stdout: "description": "osd_op(client.15966.0:9495 10.3 10:ceed0e84:::benchmark_data_smithi007_148834_object9494:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e316)", 2024-09-18T08:18:21.551 INFO:teuthology.orchestra.run.smithi007.stdout: "initiated_at": "2024-09-18T08:14:51.668764+0000", 2024-09-18T08:18:21.551 INFO:teuthology.orchestra.run.smithi007.stdout: "age": 209.096566392, 2024-09-18T08:18:21.551 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 2.0458681990000001, 2024-09-18T08:18:21.551 INFO:teuthology.orchestra.run.smithi007.stdout: "continuous": false, 2024-09-18T08:18:21.551 INFO:teuthology.orchestra.run.smithi007.stdout: "type_data": { 2024-09-18T08:18:21.551 INFO:teuthology.orchestra.run.smithi007.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T08:18:21.551 INFO:teuthology.orchestra.run.smithi007.stdout: "client_info": { 2024-09-18T08:18:21.551 INFO:teuthology.orchestra.run.smithi007.stdout: "client": "client.15966", 2024-09-18T08:18:21.551 INFO:teuthology.orchestra.run.smithi007.stdout: "client_addr": "172.21.15.7:0/529784850", 2024-09-18T08:18:21.551 INFO:teuthology.orchestra.run.smithi007.stdout: "tid": 9495 2024-09-18T08:18:21.551 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:18:21.551 INFO:teuthology.orchestra.run.smithi007.stdout: "events": [ 2024-09-18T08:18:21.552 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:18:21.552 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "initiated", 2024-09-18T08:18:21.552 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:14:51.668764+0000", 2024-09-18T08:18:21.552 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-09-18T08:18:21.552 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:18:21.552 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:18:21.552 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "header_read", 2024-09-18T08:18:21.552 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:14:51.668764+0000", 2024-09-18T08:18:21.552 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-09-18T08:18:21.552 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:18:21.552 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:18:21.552 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "throttled", 2024-09-18T08:18:21.553 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:14:51.668765+0000", 2024-09-18T08:18:21.553 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 9.0999999999999997e-07 2024-09-18T08:18:21.553 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:18:21.553 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:18:21.553 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "all_read", 2024-09-18T08:18:21.553 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:14:51.668788+0000", 2024-09-18T08:18:21.553 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 2.3133e-05 2024-09-18T08:18:21.554 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:18:21.554 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:18:21.554 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "dispatched", 2024-09-18T08:18:21.554 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:14:51.668802+0000", 2024-09-18T08:18:21.554 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.3984e-05 2024-09-18T08:18:21.554 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:18:21.554 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:18:21.554 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "queued_for_pg", 2024-09-18T08:18:21.554 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:14:51.668809+0000", 2024-09-18T08:18:21.554 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 6.7909999999999999e-06 2024-09-18T08:18:21.555 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:18:21.555 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:18:21.555 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-09-18T08:18:21.555 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:14:51.668866+0000", 2024-09-18T08:18:21.555 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 5.7133999999999999e-05 2024-09-18T08:18:21.555 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:18:21.555 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:18:21.555 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "waiting for peered", 2024-09-18T08:18:21.555 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:14:51.668874+0000", 2024-09-18T08:18:21.555 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 8.5310000000000001e-06 2024-09-18T08:18:21.555 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:18:21.555 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:18:21.556 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-09-18T08:18:21.556 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:14:52.654764+0000", 2024-09-18T08:18:21.556 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.98588983399999996 2024-09-18T08:18:21.556 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:18:21.556 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:18:21.556 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "waiting for peered", 2024-09-18T08:18:21.556 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:14:52.654779+0000", 2024-09-18T08:18:21.556 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.4501999999999999e-05 2024-09-18T08:18:21.556 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:18:21.556 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:18:21.556 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-09-18T08:18:21.556 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:14:53.695130+0000", 2024-09-18T08:18:21.557 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.0403511240000001 2024-09-18T08:18:21.557 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:18:21.557 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:18:21.557 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "started", 2024-09-18T08:18:21.557 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:14:53.695281+0000", 2024-09-18T08:18:21.557 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.00015081200000000001 2024-09-18T08:18:21.557 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:18:21.557 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:18:21.557 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "waiting for subops from 3,4", 2024-09-18T08:18:21.557 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:14:53.695455+0000", 2024-09-18T08:18:21.557 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.00017440500000000001 2024-09-18T08:18:21.558 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:18:21.558 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:18:21.558 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:18:21.558 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:14:53.699916+0000", 2024-09-18T08:18:21.558 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.0044613769999999999 2024-09-18T08:18:21.558 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:18:21.558 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:18:21.558 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:18:21.558 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:14:53.699961+0000", 2024-09-18T08:18:21.558 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 4.4354999999999998e-05 2024-09-18T08:18:21.558 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:18:21.558 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:18:21.559 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "op_commit", 2024-09-18T08:18:21.559 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:14:53.714186+0000", 2024-09-18T08:18:21.559 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.014225632 2024-09-18T08:18:21.559 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:18:21.559 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:18:21.559 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "commit_sent", 2024-09-18T08:18:21.559 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:14:53.714331+0000", 2024-09-18T08:18:21.559 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.00014509500000000001 2024-09-18T08:18:21.559 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:18:21.559 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:18:21.559 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "done", 2024-09-18T08:18:21.560 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:14:53.714632+0000", 2024-09-18T08:18:21.560 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.00030058 2024-09-18T08:18:21.560 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-09-18T08:18:21.560 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-09-18T08:18:21.560 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-09-18T08:18:21.560 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:18:21.560 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:18:21.560 INFO:teuthology.orchestra.run.smithi007.stdout: "description": "osd_op(client.15966.0:9501 10.3 10:ca7811de:::benchmark_data_smithi007_148834_object9500:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e316)", 2024-09-18T08:18:21.560 INFO:teuthology.orchestra.run.smithi007.stdout: "initiated_at": "2024-09-18T08:14:51.676714+0000", 2024-09-18T08:18:21.560 INFO:teuthology.orchestra.run.smithi007.stdout: "age": 209.08861617599999, 2024-09-18T08:18:21.560 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 2.0382764990000002, 2024-09-18T08:18:21.560 INFO:teuthology.orchestra.run.smithi007.stdout: "continuous": false, 2024-09-18T08:18:21.561 INFO:teuthology.orchestra.run.smithi007.stdout: "type_data": { 2024-09-18T08:18:21.561 INFO:teuthology.orchestra.run.smithi007.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T08:18:21.561 INFO:teuthology.orchestra.run.smithi007.stdout: "client_info": { 2024-09-18T08:18:21.561 INFO:teuthology.orchestra.run.smithi007.stdout: "client": "client.15966", 2024-09-18T08:18:21.561 INFO:teuthology.orchestra.run.smithi007.stdout: "client_addr": "172.21.15.7:0/529784850", 2024-09-18T08:18:21.561 INFO:teuthology.orchestra.run.smithi007.stdout: "tid": 9501 2024-09-18T08:18:21.561 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:18:21.561 INFO:teuthology.orchestra.run.smithi007.stdout: "events": [ 2024-09-18T08:18:21.561 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:18:21.561 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "initiated", 2024-09-18T08:18:21.561 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:14:51.676714+0000", 2024-09-18T08:18:21.561 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-09-18T08:18:21.562 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:18:21.562 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:18:21.562 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "header_read", 2024-09-18T08:18:21.562 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:14:51.676714+0000", 2024-09-18T08:18:21.562 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-09-18T08:18:21.562 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:18:21.562 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:18:21.562 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "throttled", 2024-09-18T08:18:21.562 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:14:51.676716+0000", 2024-09-18T08:18:21.562 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.6130000000000001e-06 2024-09-18T08:18:21.562 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:18:21.563 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:18:21.563 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "all_read", 2024-09-18T08:18:21.563 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:14:51.676738+0000", 2024-09-18T08:18:21.563 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 2.2714e-05 2024-09-18T08:18:21.563 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:18:21.563 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:18:21.563 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "dispatched", 2024-09-18T08:18:21.563 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:14:51.676753+0000", 2024-09-18T08:18:21.563 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.4439e-05 2024-09-18T08:18:21.563 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:18:21.563 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:18:21.564 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "queued_for_pg", 2024-09-18T08:18:21.564 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:14:51.676761+0000", 2024-09-18T08:18:21.564 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 7.8059999999999995e-06 2024-09-18T08:18:21.564 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:18:21.564 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:18:21.564 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-09-18T08:18:21.564 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:14:51.676814+0000", 2024-09-18T08:18:21.564 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 5.3504999999999997e-05 2024-09-18T08:18:21.564 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:18:21.564 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:18:21.564 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "waiting for peered", 2024-09-18T08:18:21.565 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:14:51.676821+0000", 2024-09-18T08:18:21.565 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 6.6379999999999998e-06 2024-09-18T08:18:21.565 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:18:21.565 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:18:21.565 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-09-18T08:18:21.565 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:14:52.654807+0000", 2024-09-18T08:18:21.565 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.97798668300000002 2024-09-18T08:18:21.565 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:18:21.565 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:18:21.565 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "waiting for peered", 2024-09-18T08:18:21.565 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:14:52.654816+0000", 2024-09-18T08:18:21.566 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 8.2949999999999994e-06 2024-09-18T08:18:21.566 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:18:21.566 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:18:21.566 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-09-18T08:18:21.566 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:14:53.697902+0000", 2024-09-18T08:18:21.566 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.0430859640000001 2024-09-18T08:18:21.566 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:18:21.566 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:18:21.566 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "started", 2024-09-18T08:18:21.566 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:14:53.698062+0000", 2024-09-18T08:18:21.566 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.00016033900000000001 2024-09-18T08:18:21.566 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:18:21.567 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:18:21.567 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "waiting for subops from 3,4", 2024-09-18T08:18:21.567 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:14:53.698234+0000", 2024-09-18T08:18:21.567 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.00017239800000000001 2024-09-18T08:18:21.567 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:18:21.567 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:18:21.567 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:18:21.567 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:14:53.700006+0000", 2024-09-18T08:18:21.567 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.001771667 2024-09-18T08:18:21.567 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:18:21.567 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:18:21.567 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:18:21.568 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:14:53.701950+0000", 2024-09-18T08:18:21.568 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.0019442330000000001 2024-09-18T08:18:21.568 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:18:21.568 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:18:21.568 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "op_commit", 2024-09-18T08:18:21.568 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:14:53.714642+0000", 2024-09-18T08:18:21.568 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.012691578 2024-09-18T08:18:21.568 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:18:21.568 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:18:21.568 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "commit_sent", 2024-09-18T08:18:21.568 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:14:53.714697+0000", 2024-09-18T08:18:21.568 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 5.5316999999999999e-05 2024-09-18T08:18:21.568 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:18:21.569 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:18:21.569 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "done", 2024-09-18T08:18:21.569 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:14:53.714991+0000", 2024-09-18T08:18:21.569 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.00029331000000000002 2024-09-18T08:18:21.569 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-09-18T08:18:21.569 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-09-18T08:18:21.569 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-09-18T08:18:21.569 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:18:21.569 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:18:21.569 INFO:teuthology.orchestra.run.smithi007.stdout: "description": "osd_op(client.15966.0:16666 10.3 10:c7bced0d:::benchmark_data_smithi007_148834_object16665:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e323)", 2024-09-18T08:18:21.570 INFO:teuthology.orchestra.run.smithi007.stdout: "initiated_at": "2024-09-18T08:14:59.368483+0000", 2024-09-18T08:18:21.570 INFO:teuthology.orchestra.run.smithi007.stdout: "age": 201.396846966, 2024-09-18T08:18:21.570 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.0017785939999999, 2024-09-18T08:18:21.570 INFO:teuthology.orchestra.run.smithi007.stdout: "continuous": false, 2024-09-18T08:18:21.570 INFO:teuthology.orchestra.run.smithi007.stdout: "type_data": { 2024-09-18T08:18:21.570 INFO:teuthology.orchestra.run.smithi007.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T08:18:21.570 INFO:teuthology.orchestra.run.smithi007.stdout: "client_info": { 2024-09-18T08:18:21.570 INFO:teuthology.orchestra.run.smithi007.stdout: "client": "client.15966", 2024-09-18T08:18:21.570 INFO:teuthology.orchestra.run.smithi007.stdout: "client_addr": "172.21.15.7:0/529784850", 2024-09-18T08:18:21.570 INFO:teuthology.orchestra.run.smithi007.stdout: "tid": 16666 2024-09-18T08:18:21.570 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:18:21.571 INFO:teuthology.orchestra.run.smithi007.stdout: "events": [ 2024-09-18T08:18:21.571 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:18:21.571 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "initiated", 2024-09-18T08:18:21.571 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:14:59.368483+0000", 2024-09-18T08:18:21.571 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-09-18T08:18:21.571 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:18:21.571 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:18:21.571 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "header_read", 2024-09-18T08:18:21.571 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:14:59.368483+0000", 2024-09-18T08:18:21.571 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-09-18T08:18:21.571 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:18:21.571 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:18:21.572 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "throttled", 2024-09-18T08:18:21.572 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:14:59.368485+0000", 2024-09-18T08:18:21.572 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.3769999999999999e-06 2024-09-18T08:18:21.572 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:18:21.572 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:18:21.572 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "all_read", 2024-09-18T08:18:21.572 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:14:59.368519+0000", 2024-09-18T08:18:21.572 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 3.4001000000000002e-05 2024-09-18T08:18:21.572 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:18:21.572 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:18:21.572 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "dispatched", 2024-09-18T08:18:21.573 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:14:59.368532+0000", 2024-09-18T08:18:21.573 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.2931999999999999e-05 2024-09-18T08:18:21.573 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:18:21.573 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:18:21.573 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "queued_for_pg", 2024-09-18T08:18:21.573 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:14:59.368541+0000", 2024-09-18T08:18:21.573 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 9.098e-06 2024-09-18T08:18:21.573 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:18:21.573 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:18:21.573 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-09-18T08:18:21.573 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:14:59.368675+0000", 2024-09-18T08:18:21.574 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.000134738 2024-09-18T08:18:21.574 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:18:21.574 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:18:21.574 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "waiting for peered", 2024-09-18T08:18:21.574 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:14:59.368681+0000", 2024-09-18T08:18:21.574 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 5.8590000000000001e-06 2024-09-18T08:18:21.574 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:18:21.574 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:18:21.574 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-09-18T08:18:21.574 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:15:00.360513+0000", 2024-09-18T08:18:21.574 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.99183212899999995 2024-09-18T08:18:21.574 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:18:21.575 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:18:21.575 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "started", 2024-09-18T08:18:21.575 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:15:00.360688+0000", 2024-09-18T08:18:21.575 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.00017474600000000001 2024-09-18T08:18:21.575 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:18:21.575 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:18:21.575 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "waiting for subops from 3,4", 2024-09-18T08:18:21.575 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:15:00.360836+0000", 2024-09-18T08:18:21.575 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.000148064 2024-09-18T08:18:21.575 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:18:21.575 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:18:21.575 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "op_commit", 2024-09-18T08:18:21.576 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:15:00.363513+0000", 2024-09-18T08:18:21.576 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.0026769110000000001 2024-09-18T08:18:21.576 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:18:21.576 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:18:21.576 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:18:21.576 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:15:00.363640+0000", 2024-09-18T08:18:21.576 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.000126857 2024-09-18T08:18:21.576 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:18:21.576 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:18:21.576 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:18:21.576 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:15:00.370161+0000", 2024-09-18T08:18:21.577 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.0065207370000000004 2024-09-18T08:18:21.577 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:18:21.577 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:18:21.577 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "commit_sent", 2024-09-18T08:18:21.577 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:15:00.370216+0000", 2024-09-18T08:18:21.577 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 5.5447e-05 2024-09-18T08:18:21.577 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:18:21.577 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:18:21.577 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "done", 2024-09-18T08:18:21.577 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:15:00.370262+0000", 2024-09-18T08:18:21.578 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 4.5698000000000002e-05 2024-09-18T08:18:21.578 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-09-18T08:18:21.578 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-09-18T08:18:21.578 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-09-18T08:18:21.578 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:18:21.578 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:18:21.578 INFO:teuthology.orchestra.run.smithi007.stdout: "description": "osd_op(client.15966.0:16693 10.3 10:c1956f28:::benchmark_data_smithi007_148834_object16692:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e323)", 2024-09-18T08:18:21.578 INFO:teuthology.orchestra.run.smithi007.stdout: "initiated_at": "2024-09-18T08:14:59.380410+0000", 2024-09-18T08:18:21.578 INFO:teuthology.orchestra.run.smithi007.stdout: "age": 201.38492002800001, 2024-09-18T08:18:21.578 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.99034093599999995, 2024-09-18T08:18:21.578 INFO:teuthology.orchestra.run.smithi007.stdout: "continuous": false, 2024-09-18T08:18:21.578 INFO:teuthology.orchestra.run.smithi007.stdout: "type_data": { 2024-09-18T08:18:21.579 INFO:teuthology.orchestra.run.smithi007.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T08:18:21.579 INFO:teuthology.orchestra.run.smithi007.stdout: "client_info": { 2024-09-18T08:18:21.579 INFO:teuthology.orchestra.run.smithi007.stdout: "client": "client.15966", 2024-09-18T08:18:21.579 INFO:teuthology.orchestra.run.smithi007.stdout: "client_addr": "172.21.15.7:0/529784850", 2024-09-18T08:18:21.579 INFO:teuthology.orchestra.run.smithi007.stdout: "tid": 16693 2024-09-18T08:18:21.579 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:18:21.579 INFO:teuthology.orchestra.run.smithi007.stdout: "events": [ 2024-09-18T08:18:21.579 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:18:21.579 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "initiated", 2024-09-18T08:18:21.579 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:14:59.380410+0000", 2024-09-18T08:18:21.580 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-09-18T08:18:21.580 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:18:21.580 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:18:21.580 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "header_read", 2024-09-18T08:18:21.580 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:14:59.380410+0000", 2024-09-18T08:18:21.580 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-09-18T08:18:21.580 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:18:21.580 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:18:21.580 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "throttled", 2024-09-18T08:18:21.580 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:14:59.380411+0000", 2024-09-18T08:18:21.580 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 5.1600000000000001e-07 2024-09-18T08:18:21.581 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:18:21.581 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:18:21.581 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "all_read", 2024-09-18T08:18:21.581 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:14:59.380460+0000", 2024-09-18T08:18:21.581 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 4.9724e-05 2024-09-18T08:18:21.581 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:18:21.581 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:18:21.581 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "dispatched", 2024-09-18T08:18:21.581 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:14:59.380468+0000", 2024-09-18T08:18:21.581 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 7.1779999999999997e-06 2024-09-18T08:18:21.581 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:18:21.582 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:18:21.582 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "queued_for_pg", 2024-09-18T08:18:21.582 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:14:59.380473+0000", 2024-09-18T08:18:21.582 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 5.4399999999999996e-06 2024-09-18T08:18:21.582 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:18:21.582 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:18:21.582 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-09-18T08:18:21.582 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:14:59.380568+0000", 2024-09-18T08:18:21.582 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 9.4908999999999995e-05 2024-09-18T08:18:21.582 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:18:21.582 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:18:21.583 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "waiting for peered", 2024-09-18T08:18:21.583 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:14:59.380577+0000", 2024-09-18T08:18:21.584 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 8.8079999999999994e-06 2024-09-18T08:18:21.584 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:18:21.584 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:18:21.584 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-09-18T08:18:21.584 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:15:00.362125+0000", 2024-09-18T08:18:21.585 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.98154793200000001 2024-09-18T08:18:21.585 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:18:21.585 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:18:21.585 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "started", 2024-09-18T08:18:21.585 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:15:00.362248+0000", 2024-09-18T08:18:21.585 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.00012318400000000001 2024-09-18T08:18:21.585 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:18:21.585 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:18:21.585 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "waiting for subops from 3,4", 2024-09-18T08:18:21.585 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:15:00.362390+0000", 2024-09-18T08:18:21.586 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.000142366 2024-09-18T08:18:21.586 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:18:21.586 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:18:21.586 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "op_commit", 2024-09-18T08:18:21.586 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:15:00.363599+0000", 2024-09-18T08:18:21.586 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.001208836 2024-09-18T08:18:21.586 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:18:21.586 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:18:21.586 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:18:21.586 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:15:00.364371+0000", 2024-09-18T08:18:21.586 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.00077206300000000002 2024-09-18T08:18:21.586 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:18:21.587 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:18:21.587 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:18:21.587 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:15:00.370667+0000", 2024-09-18T08:18:21.587 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.0062957500000000001 2024-09-18T08:18:21.587 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:18:21.587 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:18:21.587 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "commit_sent", 2024-09-18T08:18:21.587 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:15:00.370716+0000", 2024-09-18T08:18:21.587 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 4.9055999999999999e-05 2024-09-18T08:18:21.587 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:18:21.587 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:18:21.588 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "done", 2024-09-18T08:18:21.588 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:15:00.370751+0000", 2024-09-18T08:18:21.588 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 3.5173999999999999e-05 2024-09-18T08:18:21.588 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-09-18T08:18:21.588 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-09-18T08:18:21.588 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-09-18T08:18:21.588 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:18:21.588 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:18:21.588 INFO:teuthology.orchestra.run.smithi007.stdout: "description": "osd_op(client.15966.0:16740 10.9 10:9658d6e1:::benchmark_data_smithi007_148834_object16739:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e324)", 2024-09-18T08:18:21.589 INFO:teuthology.orchestra.run.smithi007.stdout: "initiated_at": "2024-09-18T08:15:00.363218+0000", 2024-09-18T08:18:21.589 INFO:teuthology.orchestra.run.smithi007.stdout: "age": 200.40211196000001, 2024-09-18T08:18:21.589 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.013596299, 2024-09-18T08:18:21.589 INFO:teuthology.orchestra.run.smithi007.stdout: "continuous": false, 2024-09-18T08:18:21.589 INFO:teuthology.orchestra.run.smithi007.stdout: "type_data": { 2024-09-18T08:18:21.589 INFO:teuthology.orchestra.run.smithi007.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T08:18:21.589 INFO:teuthology.orchestra.run.smithi007.stdout: "client_info": { 2024-09-18T08:18:21.589 INFO:teuthology.orchestra.run.smithi007.stdout: "client": "client.15966", 2024-09-18T08:18:21.589 INFO:teuthology.orchestra.run.smithi007.stdout: "client_addr": "172.21.15.7:0/529784850", 2024-09-18T08:18:21.589 INFO:teuthology.orchestra.run.smithi007.stdout: "tid": 16740 2024-09-18T08:18:21.589 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:18:21.590 INFO:teuthology.orchestra.run.smithi007.stdout: "events": [ 2024-09-18T08:18:21.590 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:18:21.590 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "initiated", 2024-09-18T08:18:21.590 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:15:00.363218+0000", 2024-09-18T08:18:21.590 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-09-18T08:18:21.590 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:18:21.590 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:18:21.590 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "header_read", 2024-09-18T08:18:21.590 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:15:00.363218+0000", 2024-09-18T08:18:21.590 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-09-18T08:18:21.590 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:18:21.591 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:18:21.591 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "throttled", 2024-09-18T08:18:21.591 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:15:00.363220+0000", 2024-09-18T08:18:21.591 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.268e-06 2024-09-18T08:18:21.591 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:18:21.591 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:18:21.591 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "all_read", 2024-09-18T08:18:21.591 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:15:00.363241+0000", 2024-09-18T08:18:21.591 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 2.1866e-05 2024-09-18T08:18:21.591 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:18:21.591 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:18:21.591 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "dispatched", 2024-09-18T08:18:21.592 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:15:00.363253+0000", 2024-09-18T08:18:21.592 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.1394e-05 2024-09-18T08:18:21.592 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:18:21.592 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:18:21.592 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "queued_for_pg", 2024-09-18T08:18:21.592 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:15:00.363260+0000", 2024-09-18T08:18:21.592 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 7.6620000000000005e-06 2024-09-18T08:18:21.592 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:18:21.592 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:18:21.592 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-09-18T08:18:21.592 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:15:00.363329+0000", 2024-09-18T08:18:21.593 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 6.8176999999999995e-05 2024-09-18T08:18:21.593 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:18:21.593 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:18:21.593 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "waiting for peered", 2024-09-18T08:18:21.593 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:15:00.363369+0000", 2024-09-18T08:18:21.593 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 3.9879999999999998e-05 2024-09-18T08:18:21.593 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:18:21.593 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:18:21.593 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-09-18T08:18:21.593 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:15:01.374467+0000", 2024-09-18T08:18:21.593 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.0110987739999999 2024-09-18T08:18:21.594 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:18:21.594 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:18:21.594 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "started", 2024-09-18T08:18:21.594 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:15:01.374631+0000", 2024-09-18T08:18:21.594 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.00016364799999999999 2024-09-18T08:18:21.594 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:18:21.594 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:18:21.594 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "waiting for subops from 3,4", 2024-09-18T08:18:21.594 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:15:01.374806+0000", 2024-09-18T08:18:21.594 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.00017538800000000001 2024-09-18T08:18:21.594 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:18:21.594 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:18:21.594 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:18:21.595 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:15:01.376700+0000", 2024-09-18T08:18:21.595 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.001893884 2024-09-18T08:18:21.595 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:18:21.595 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:18:21.595 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:18:21.595 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:15:01.376734+0000", 2024-09-18T08:18:21.595 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 3.4124000000000001e-05 2024-09-18T08:18:21.595 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:18:21.595 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:18:21.595 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "op_commit", 2024-09-18T08:18:21.595 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:15:01.376744+0000", 2024-09-18T08:18:21.595 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 9.893e-06 2024-09-18T08:18:21.595 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:18:21.596 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:18:21.596 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "commit_sent", 2024-09-18T08:18:21.596 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:15:01.376772+0000", 2024-09-18T08:18:21.596 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 2.8170999999999998e-05 2024-09-18T08:18:21.596 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:18:21.596 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:18:21.596 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "done", 2024-09-18T08:18:21.596 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:15:01.376815+0000", 2024-09-18T08:18:21.596 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 4.2169999999999998e-05 2024-09-18T08:18:21.596 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-09-18T08:18:21.597 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-09-18T08:18:21.597 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-09-18T08:18:21.597 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:18:21.597 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:18:21.597 INFO:teuthology.orchestra.run.smithi007.stdout: "description": "osd_op(client.15966.0:16746 10.9 10:988f7ffd:::benchmark_data_smithi007_148834_object16745:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e324)", 2024-09-18T08:18:21.597 INFO:teuthology.orchestra.run.smithi007.stdout: "initiated_at": "2024-09-18T08:15:00.367585+0000", 2024-09-18T08:18:21.597 INFO:teuthology.orchestra.run.smithi007.stdout: "age": 200.39774545200001, 2024-09-18T08:18:21.597 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.011225958, 2024-09-18T08:18:21.597 INFO:teuthology.orchestra.run.smithi007.stdout: "continuous": false, 2024-09-18T08:18:21.597 INFO:teuthology.orchestra.run.smithi007.stdout: "type_data": { 2024-09-18T08:18:21.598 INFO:teuthology.orchestra.run.smithi007.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T08:18:21.598 INFO:teuthology.orchestra.run.smithi007.stdout: "client_info": { 2024-09-18T08:18:21.598 INFO:teuthology.orchestra.run.smithi007.stdout: "client": "client.15966", 2024-09-18T08:18:21.598 INFO:teuthology.orchestra.run.smithi007.stdout: "client_addr": "172.21.15.7:0/529784850", 2024-09-18T08:18:21.598 INFO:teuthology.orchestra.run.smithi007.stdout: "tid": 16746 2024-09-18T08:18:21.598 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:18:21.598 INFO:teuthology.orchestra.run.smithi007.stdout: "events": [ 2024-09-18T08:18:21.598 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:18:21.598 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "initiated", 2024-09-18T08:18:21.598 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:15:00.367585+0000", 2024-09-18T08:18:21.598 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-09-18T08:18:21.599 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:18:21.599 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:18:21.599 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "header_read", 2024-09-18T08:18:21.599 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:15:00.367585+0000", 2024-09-18T08:18:21.599 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-09-18T08:18:21.599 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:18:21.599 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:18:21.599 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "throttled", 2024-09-18T08:18:21.599 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:15:00.367586+0000", 2024-09-18T08:18:21.599 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.359e-06 2024-09-18T08:18:21.599 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:18:21.599 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:18:21.600 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "all_read", 2024-09-18T08:18:21.600 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:15:00.367607+0000", 2024-09-18T08:18:21.600 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 2.1055000000000001e-05 2024-09-18T08:18:21.600 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:18:21.600 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:18:21.600 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "dispatched", 2024-09-18T08:18:21.600 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:15:00.367618+0000", 2024-09-18T08:18:21.600 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.0336999999999999e-05 2024-09-18T08:18:21.600 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:18:21.600 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:18:21.600 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "queued_for_pg", 2024-09-18T08:18:21.601 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:15:00.367625+0000", 2024-09-18T08:18:21.601 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 7.3930000000000002e-06 2024-09-18T08:18:21.601 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:18:21.601 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:18:21.601 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-09-18T08:18:21.601 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:15:00.368768+0000", 2024-09-18T08:18:21.601 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.001143055 2024-09-18T08:18:21.601 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:18:21.601 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:18:21.601 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "waiting for peered", 2024-09-18T08:18:21.601 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:15:00.368779+0000", 2024-09-18T08:18:21.601 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.0553999999999999e-05 2024-09-18T08:18:21.602 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:18:21.602 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:18:21.602 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-09-18T08:18:21.602 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:15:01.376069+0000", 2024-09-18T08:18:21.602 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.007290206 2024-09-18T08:18:21.602 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:18:21.602 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:18:21.602 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "started", 2024-09-18T08:18:21.602 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:15:01.376154+0000", 2024-09-18T08:18:21.602 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 8.4840999999999996e-05 2024-09-18T08:18:21.602 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:18:21.603 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:18:21.603 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "waiting for subops from 3,4", 2024-09-18T08:18:21.603 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:15:01.376239+0000", 2024-09-18T08:18:21.603 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 8.5001999999999997e-05 2024-09-18T08:18:21.603 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:18:21.603 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:18:21.603 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "op_commit", 2024-09-18T08:18:21.603 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:15:01.378633+0000", 2024-09-18T08:18:21.603 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.002394471 2024-09-18T08:18:21.603 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:18:21.603 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:18:21.603 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:18:21.604 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:15:01.378671+0000", 2024-09-18T08:18:21.604 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 3.7676000000000003e-05 2024-09-18T08:18:21.604 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:18:21.604 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:18:21.604 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:18:21.604 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:15:01.378716+0000", 2024-09-18T08:18:21.604 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 4.5269999999999999e-05 2024-09-18T08:18:21.604 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:18:21.604 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:18:21.604 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "commit_sent", 2024-09-18T08:18:21.605 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:15:01.378771+0000", 2024-09-18T08:18:21.605 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 5.5050000000000003e-05 2024-09-18T08:18:21.605 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:18:21.605 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:18:21.605 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "done", 2024-09-18T08:18:21.605 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:15:01.378811+0000", 2024-09-18T08:18:21.605 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 3.9688999999999999e-05 2024-09-18T08:18:21.605 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-09-18T08:18:21.605 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-09-18T08:18:21.605 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-09-18T08:18:21.605 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:18:21.605 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:18:21.606 INFO:teuthology.orchestra.run.smithi007.stdout: "description": "osd_op(client.15966.0:16793 10.9 10:94d3fd58:::benchmark_data_smithi007_148834_object16792:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e324)", 2024-09-18T08:18:21.606 INFO:teuthology.orchestra.run.smithi007.stdout: "initiated_at": "2024-09-18T08:15:00.387215+0000", 2024-09-18T08:18:21.606 INFO:teuthology.orchestra.run.smithi007.stdout: "age": 200.37811536199999, 2024-09-18T08:18:21.606 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.99187469100000003, 2024-09-18T08:18:21.606 INFO:teuthology.orchestra.run.smithi007.stdout: "continuous": false, 2024-09-18T08:18:21.606 INFO:teuthology.orchestra.run.smithi007.stdout: "type_data": { 2024-09-18T08:18:21.606 INFO:teuthology.orchestra.run.smithi007.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T08:18:21.606 INFO:teuthology.orchestra.run.smithi007.stdout: "client_info": { 2024-09-18T08:18:21.606 INFO:teuthology.orchestra.run.smithi007.stdout: "client": "client.15966", 2024-09-18T08:18:21.606 INFO:teuthology.orchestra.run.smithi007.stdout: "client_addr": "172.21.15.7:0/529784850", 2024-09-18T08:18:21.606 INFO:teuthology.orchestra.run.smithi007.stdout: "tid": 16793 2024-09-18T08:18:21.607 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:18:21.607 INFO:teuthology.orchestra.run.smithi007.stdout: "events": [ 2024-09-18T08:18:21.607 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:18:21.607 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "initiated", 2024-09-18T08:18:21.607 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:15:00.387215+0000", 2024-09-18T08:18:21.607 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-09-18T08:18:21.607 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:18:21.607 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:18:21.607 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "header_read", 2024-09-18T08:18:21.607 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:15:00.387215+0000", 2024-09-18T08:18:21.608 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-09-18T08:18:21.608 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:18:21.608 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:18:21.608 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "throttled", 2024-09-18T08:18:21.608 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:15:00.387216+0000", 2024-09-18T08:18:21.608 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.5960000000000001e-06 2024-09-18T08:18:21.608 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:18:21.608 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:18:21.608 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "all_read", 2024-09-18T08:18:21.608 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:15:00.387243+0000", 2024-09-18T08:18:21.608 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 2.635e-05 2024-09-18T08:18:21.609 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:18:21.609 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:18:21.609 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "dispatched", 2024-09-18T08:18:21.609 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:15:00.387257+0000", 2024-09-18T08:18:21.609 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.4081e-05 2024-09-18T08:18:21.609 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:18:21.609 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:18:21.609 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "queued_for_pg", 2024-09-18T08:18:21.609 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:15:00.387264+0000", 2024-09-18T08:18:21.609 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 7.5630000000000003e-06 2024-09-18T08:18:21.609 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:18:21.610 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:18:21.610 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-09-18T08:18:21.610 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:15:00.387346+0000", 2024-09-18T08:18:21.610 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 8.1583999999999997e-05 2024-09-18T08:18:21.610 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:18:21.610 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:18:21.610 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "waiting for peered", 2024-09-18T08:18:21.610 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:15:00.387357+0000", 2024-09-18T08:18:21.610 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.0572e-05 2024-09-18T08:18:21.610 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:18:21.610 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:18:21.611 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-09-18T08:18:21.611 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:15:01.376952+0000", 2024-09-18T08:18:21.611 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.98959566499999996 2024-09-18T08:18:21.611 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:18:21.611 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:18:21.611 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "started", 2024-09-18T08:18:21.611 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:15:01.377094+0000", 2024-09-18T08:18:21.611 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.00014199500000000001 2024-09-18T08:18:21.611 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:18:21.611 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:18:21.611 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "waiting for subops from 3,4", 2024-09-18T08:18:21.611 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:15:01.377242+0000", 2024-09-18T08:18:21.612 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.00014731799999999999 2024-09-18T08:18:21.612 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:18:21.612 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:18:21.612 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "op_commit", 2024-09-18T08:18:21.612 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:15:01.378682+0000", 2024-09-18T08:18:21.612 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.0014402919999999999 2024-09-18T08:18:21.612 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:18:21.612 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:18:21.612 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:18:21.612 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:15:01.378856+0000", 2024-09-18T08:18:21.612 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.00017438800000000001 2024-09-18T08:18:21.613 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:18:21.613 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:18:21.613 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:18:21.613 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:15:01.378943+0000", 2024-09-18T08:18:21.613 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 8.7101000000000005e-05 2024-09-18T08:18:21.613 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:18:21.613 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:18:21.613 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "commit_sent", 2024-09-18T08:18:21.613 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:15:01.379052+0000", 2024-09-18T08:18:21.613 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.00010877799999999999 2024-09-18T08:18:21.613 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:18:21.614 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:18:21.614 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "done", 2024-09-18T08:18:21.614 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:15:01.379090+0000", 2024-09-18T08:18:21.614 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 3.7407999999999998e-05 2024-09-18T08:18:21.614 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-09-18T08:18:21.614 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-09-18T08:18:21.614 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-09-18T08:18:21.614 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:18:21.614 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:18:21.614 INFO:teuthology.orchestra.run.smithi007.stdout: "description": "osd_op(client.15966.0:16818 10.9 10:9b4b2220:::benchmark_data_smithi007_148834_object16817:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e324)", 2024-09-18T08:18:21.614 INFO:teuthology.orchestra.run.smithi007.stdout: "initiated_at": "2024-09-18T08:15:00.398499+0000", 2024-09-18T08:18:21.615 INFO:teuthology.orchestra.run.smithi007.stdout: "age": 200.36683074999999, 2024-09-18T08:18:21.615 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.98326930499999998, 2024-09-18T08:18:21.615 INFO:teuthology.orchestra.run.smithi007.stdout: "continuous": false, 2024-09-18T08:18:21.615 INFO:teuthology.orchestra.run.smithi007.stdout: "type_data": { 2024-09-18T08:18:21.615 INFO:teuthology.orchestra.run.smithi007.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T08:18:21.615 INFO:teuthology.orchestra.run.smithi007.stdout: "client_info": { 2024-09-18T08:18:21.615 INFO:teuthology.orchestra.run.smithi007.stdout: "client": "client.15966", 2024-09-18T08:18:21.615 INFO:teuthology.orchestra.run.smithi007.stdout: "client_addr": "172.21.15.7:0/529784850", 2024-09-18T08:18:21.615 INFO:teuthology.orchestra.run.smithi007.stdout: "tid": 16818 2024-09-18T08:18:21.615 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:18:21.615 INFO:teuthology.orchestra.run.smithi007.stdout: "events": [ 2024-09-18T08:18:21.615 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:18:21.616 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "initiated", 2024-09-18T08:18:21.616 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:15:00.398499+0000", 2024-09-18T08:18:21.616 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-09-18T08:18:21.616 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:18:21.616 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:18:21.616 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "header_read", 2024-09-18T08:18:21.616 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:15:00.398499+0000", 2024-09-18T08:18:21.616 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-09-18T08:18:21.616 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:18:21.616 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:18:21.617 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "throttled", 2024-09-18T08:18:21.617 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:15:00.398501+0000", 2024-09-18T08:18:21.617 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.246e-06 2024-09-18T08:18:21.617 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:18:21.617 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:18:21.617 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "all_read", 2024-09-18T08:18:21.617 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:15:00.398529+0000", 2024-09-18T08:18:21.617 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 2.7789000000000001e-05 2024-09-18T08:18:21.617 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:18:21.617 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:18:21.617 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "dispatched", 2024-09-18T08:18:21.618 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:15:00.398539+0000", 2024-09-18T08:18:21.618 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.0176e-05 2024-09-18T08:18:21.618 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:18:21.618 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:18:21.618 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "queued_for_pg", 2024-09-18T08:18:21.618 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:15:00.398553+0000", 2024-09-18T08:18:21.618 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.4038e-05 2024-09-18T08:18:21.618 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:18:21.618 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:18:21.618 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-09-18T08:18:21.618 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:15:00.398785+0000", 2024-09-18T08:18:21.619 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.000232582 2024-09-18T08:18:21.619 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:18:21.619 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:18:21.619 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "waiting for peered", 2024-09-18T08:18:21.619 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:15:00.398795+0000", 2024-09-18T08:18:21.619 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 9.8449999999999998e-06 2024-09-18T08:18:21.619 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:18:21.619 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:18:21.619 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-09-18T08:18:21.619 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:15:01.377825+0000", 2024-09-18T08:18:21.619 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.97902953400000003 2024-09-18T08:18:21.620 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:18:21.620 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:18:21.620 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "started", 2024-09-18T08:18:21.620 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:15:01.377954+0000", 2024-09-18T08:18:21.620 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.00012964300000000001 2024-09-18T08:18:21.620 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:18:21.620 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:18:21.620 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "waiting for subops from 3,4", 2024-09-18T08:18:21.620 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:15:01.378088+0000", 2024-09-18T08:18:21.620 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.00013414300000000001 2024-09-18T08:18:21.620 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:18:21.620 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:18:21.621 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:18:21.621 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:15:01.380051+0000", 2024-09-18T08:18:21.621 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.0019621199999999999 2024-09-18T08:18:21.621 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:18:21.621 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:18:21.621 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:18:21.621 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:15:01.380096+0000", 2024-09-18T08:18:21.621 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 4.5011e-05 2024-09-18T08:18:21.621 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:18:21.621 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:18:21.621 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "op_commit", 2024-09-18T08:18:21.622 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:15:01.381685+0000", 2024-09-18T08:18:21.622 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.0015894189999999999 2024-09-18T08:18:21.622 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:18:21.622 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:18:21.622 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "commit_sent", 2024-09-18T08:18:21.622 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:15:01.381728+0000", 2024-09-18T08:18:21.622 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 4.3421999999999999e-05 2024-09-18T08:18:21.622 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:18:21.622 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:18:21.622 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "done", 2024-09-18T08:18:21.622 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:15:01.381769+0000", 2024-09-18T08:18:21.623 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 4.0336999999999997e-05 2024-09-18T08:18:21.623 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-09-18T08:18:21.623 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-09-18T08:18:21.623 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-09-18T08:18:21.623 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:18:21.623 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:18:21.623 INFO:teuthology.orchestra.run.smithi007.stdout: "description": "osd_op(client.15966.0:16840 10.9 10:9b2ea096:::benchmark_data_smithi007_148834_object16839:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e324)", 2024-09-18T08:18:21.623 INFO:teuthology.orchestra.run.smithi007.stdout: "initiated_at": "2024-09-18T08:15:00.407638+0000", 2024-09-18T08:18:21.623 INFO:teuthology.orchestra.run.smithi007.stdout: "age": 200.35769261499999, 2024-09-18T08:18:21.623 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.97589707299999995, 2024-09-18T08:18:21.623 INFO:teuthology.orchestra.run.smithi007.stdout: "continuous": false, 2024-09-18T08:18:21.623 INFO:teuthology.orchestra.run.smithi007.stdout: "type_data": { 2024-09-18T08:18:21.624 INFO:teuthology.orchestra.run.smithi007.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T08:18:21.624 INFO:teuthology.orchestra.run.smithi007.stdout: "client_info": { 2024-09-18T08:18:21.624 INFO:teuthology.orchestra.run.smithi007.stdout: "client": "client.15966", 2024-09-18T08:18:21.624 INFO:teuthology.orchestra.run.smithi007.stdout: "client_addr": "172.21.15.7:0/529784850", 2024-09-18T08:18:21.624 INFO:teuthology.orchestra.run.smithi007.stdout: "tid": 16840 2024-09-18T08:18:21.624 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:18:21.624 INFO:teuthology.orchestra.run.smithi007.stdout: "events": [ 2024-09-18T08:18:21.624 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:18:21.624 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "initiated", 2024-09-18T08:18:21.624 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:15:00.407638+0000", 2024-09-18T08:18:21.624 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-09-18T08:18:21.625 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:18:21.625 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:18:21.625 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "header_read", 2024-09-18T08:18:21.625 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:15:00.407638+0000", 2024-09-18T08:18:21.625 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-09-18T08:18:21.625 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:18:21.625 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:18:21.625 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "throttled", 2024-09-18T08:18:21.625 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:15:00.407639+0000", 2024-09-18T08:18:21.625 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.8640000000000001e-06 2024-09-18T08:18:21.625 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:18:21.625 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:18:21.626 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "all_read", 2024-09-18T08:18:21.626 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:15:00.407679+0000", 2024-09-18T08:18:21.626 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 3.9249999999999999e-05 2024-09-18T08:18:21.626 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:18:21.626 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:18:21.626 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "dispatched", 2024-09-18T08:18:21.626 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:15:00.407701+0000", 2024-09-18T08:18:21.626 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 2.2648999999999999e-05 2024-09-18T08:18:21.626 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:18:21.626 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:18:21.626 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "queued_for_pg", 2024-09-18T08:18:21.627 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:15:00.407710+0000", 2024-09-18T08:18:21.627 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 8.4970000000000001e-06 2024-09-18T08:18:21.627 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:18:21.627 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:18:21.627 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-09-18T08:18:21.627 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:15:00.407825+0000", 2024-09-18T08:18:21.627 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.000115239 2024-09-18T08:18:21.627 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:18:21.627 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:18:21.627 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "waiting for peered", 2024-09-18T08:18:21.627 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:15:00.407836+0000", 2024-09-18T08:18:21.627 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.0549e-05 2024-09-18T08:18:21.628 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:18:21.628 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:18:21.628 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-09-18T08:18:21.628 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:15:01.379173+0000", 2024-09-18T08:18:21.628 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.97133749700000005 2024-09-18T08:18:21.628 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:18:21.628 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:18:21.628 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "started", 2024-09-18T08:18:21.628 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:15:01.379323+0000", 2024-09-18T08:18:21.628 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.00014971900000000001 2024-09-18T08:18:21.628 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:18:21.629 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:18:21.629 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "waiting for subops from 3,4", 2024-09-18T08:18:21.629 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:15:01.379452+0000", 2024-09-18T08:18:21.629 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.00012921499999999999 2024-09-18T08:18:21.629 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:18:21.629 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:18:21.629 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "op_commit", 2024-09-18T08:18:21.629 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:15:01.383346+0000", 2024-09-18T08:18:21.629 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.0038943049999999998 2024-09-18T08:18:21.629 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:18:21.629 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:18:21.629 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:18:21.630 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:15:01.383394+0000", 2024-09-18T08:18:21.630 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 4.7095000000000002e-05 2024-09-18T08:18:21.630 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:18:21.630 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:18:21.630 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:18:21.630 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:15:01.383437+0000", 2024-09-18T08:18:21.630 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 4.3872000000000003e-05 2024-09-18T08:18:21.630 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:18:21.630 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:18:21.630 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "commit_sent", 2024-09-18T08:18:21.630 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:15:01.383492+0000", 2024-09-18T08:18:21.630 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 5.4193000000000001e-05 2024-09-18T08:18:21.631 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:18:21.631 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:18:21.631 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "done", 2024-09-18T08:18:21.631 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:15:01.383535+0000", 2024-09-18T08:18:21.631 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 4.3129e-05 2024-09-18T08:18:21.631 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-09-18T08:18:21.631 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-09-18T08:18:21.631 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-09-18T08:18:21.631 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:18:21.631 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:18:21.631 INFO:teuthology.orchestra.run.smithi007.stdout: "description": "osd_op(client.15966.0:16843 10.9 10:9f4a9fc4:::benchmark_data_smithi007_148834_object16842:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e324)", 2024-09-18T08:18:21.632 INFO:teuthology.orchestra.run.smithi007.stdout: "initiated_at": "2024-09-18T08:15:00.408739+0000", 2024-09-18T08:18:21.632 INFO:teuthology.orchestra.run.smithi007.stdout: "age": 200.35659101799999, 2024-09-18T08:18:21.632 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.97496165499999998, 2024-09-18T08:18:21.632 INFO:teuthology.orchestra.run.smithi007.stdout: "continuous": false, 2024-09-18T08:18:21.632 INFO:teuthology.orchestra.run.smithi007.stdout: "type_data": { 2024-09-18T08:18:21.632 INFO:teuthology.orchestra.run.smithi007.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T08:18:21.632 INFO:teuthology.orchestra.run.smithi007.stdout: "client_info": { 2024-09-18T08:18:21.632 INFO:teuthology.orchestra.run.smithi007.stdout: "client": "client.15966", 2024-09-18T08:18:21.632 INFO:teuthology.orchestra.run.smithi007.stdout: "client_addr": "172.21.15.7:0/529784850", 2024-09-18T08:18:21.632 INFO:teuthology.orchestra.run.smithi007.stdout: "tid": 16843 2024-09-18T08:18:21.632 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:18:21.633 INFO:teuthology.orchestra.run.smithi007.stdout: "events": [ 2024-09-18T08:18:21.633 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:18:21.633 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "initiated", 2024-09-18T08:18:21.633 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:15:00.408739+0000", 2024-09-18T08:18:21.633 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-09-18T08:18:21.633 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:18:21.633 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:18:21.633 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "header_read", 2024-09-18T08:18:21.633 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:15:00.408739+0000", 2024-09-18T08:18:21.633 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-09-18T08:18:21.633 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:18:21.633 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:18:21.634 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "throttled", 2024-09-18T08:18:21.634 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:15:00.408740+0000", 2024-09-18T08:18:21.634 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.145e-06 2024-09-18T08:18:21.634 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:18:21.634 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:18:21.634 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "all_read", 2024-09-18T08:18:21.634 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:15:00.408803+0000", 2024-09-18T08:18:21.634 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 6.2834000000000002e-05 2024-09-18T08:18:21.634 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:18:21.634 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:18:21.634 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "dispatched", 2024-09-18T08:18:21.635 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:15:00.408817+0000", 2024-09-18T08:18:21.635 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.3434999999999999e-05 2024-09-18T08:18:21.635 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:18:21.635 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:18:21.635 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "queued_for_pg", 2024-09-18T08:18:21.635 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:15:00.408825+0000", 2024-09-18T08:18:21.635 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 8.2789999999999998e-06 2024-09-18T08:18:21.635 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:18:21.635 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:18:21.635 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-09-18T08:18:21.635 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:15:00.408900+0000", 2024-09-18T08:18:21.635 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 7.5226000000000001e-05 2024-09-18T08:18:21.636 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:18:21.636 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:18:21.636 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "waiting for peered", 2024-09-18T08:18:21.636 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:15:00.408909+0000", 2024-09-18T08:18:21.636 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 9.2730000000000008e-06 2024-09-18T08:18:21.636 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:18:21.636 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:18:21.636 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-09-18T08:18:21.636 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:15:01.380129+0000", 2024-09-18T08:18:21.636 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.97121982100000004 2024-09-18T08:18:21.636 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:18:21.637 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:18:21.637 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "started", 2024-09-18T08:18:21.637 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:15:01.380236+0000", 2024-09-18T08:18:21.637 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.000106405 2024-09-18T08:18:21.637 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:18:21.637 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:18:21.637 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "waiting for subops from 3,4", 2024-09-18T08:18:21.637 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:15:01.380356+0000", 2024-09-18T08:18:21.637 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.000120686 2024-09-18T08:18:21.637 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:18:21.637 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:18:21.638 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "op_commit", 2024-09-18T08:18:21.638 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:15:01.383352+0000", 2024-09-18T08:18:21.638 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.0029956919999999999 2024-09-18T08:18:21.638 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:18:21.638 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:18:21.638 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:18:21.638 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:15:01.383578+0000", 2024-09-18T08:18:21.638 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.00022547800000000001 2024-09-18T08:18:21.638 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:18:21.638 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:18:21.638 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:18:21.638 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:15:01.383622+0000", 2024-09-18T08:18:21.639 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 4.4311999999999999e-05 2024-09-18T08:18:21.639 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:18:21.639 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:18:21.639 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "commit_sent", 2024-09-18T08:18:21.639 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:15:01.383666+0000", 2024-09-18T08:18:21.639 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 4.4579000000000001e-05 2024-09-18T08:18:21.639 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:18:21.639 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:18:21.639 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "done", 2024-09-18T08:18:21.639 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:15:01.383701+0000", 2024-09-18T08:18:21.639 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 3.4489999999999997e-05 2024-09-18T08:18:21.639 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-09-18T08:18:21.640 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-09-18T08:18:21.640 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-09-18T08:18:21.640 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:18:21.640 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:18:21.640 INFO:teuthology.orchestra.run.smithi007.stdout: "description": "osd_op(client.15966.0:16844 10.9 10:9c7b6630:::benchmark_data_smithi007_148834_object16843:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e324)", 2024-09-18T08:18:21.640 INFO:teuthology.orchestra.run.smithi007.stdout: "initiated_at": "2024-09-18T08:15:00.408844+0000", 2024-09-18T08:18:21.640 INFO:teuthology.orchestra.run.smithi007.stdout: "age": 200.35648584099999, 2024-09-18T08:18:21.640 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.97895429899999997, 2024-09-18T08:18:21.640 INFO:teuthology.orchestra.run.smithi007.stdout: "continuous": false, 2024-09-18T08:18:21.640 INFO:teuthology.orchestra.run.smithi007.stdout: "type_data": { 2024-09-18T08:18:21.640 INFO:teuthology.orchestra.run.smithi007.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T08:18:21.641 INFO:teuthology.orchestra.run.smithi007.stdout: "client_info": { 2024-09-18T08:18:21.641 INFO:teuthology.orchestra.run.smithi007.stdout: "client": "client.15966", 2024-09-18T08:18:21.641 INFO:teuthology.orchestra.run.smithi007.stdout: "client_addr": "172.21.15.7:0/529784850", 2024-09-18T08:18:21.641 INFO:teuthology.orchestra.run.smithi007.stdout: "tid": 16844 2024-09-18T08:18:21.641 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:18:21.641 INFO:teuthology.orchestra.run.smithi007.stdout: "events": [ 2024-09-18T08:18:21.641 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:18:21.641 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "initiated", 2024-09-18T08:18:21.641 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:15:00.408844+0000", 2024-09-18T08:18:21.641 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-09-18T08:18:21.641 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:18:21.642 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:18:21.642 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "header_read", 2024-09-18T08:18:21.642 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:15:00.408844+0000", 2024-09-18T08:18:21.642 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-09-18T08:18:21.642 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:18:21.642 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:18:21.642 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "throttled", 2024-09-18T08:18:21.642 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:15:00.408845+0000", 2024-09-18T08:18:21.642 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 7.8400000000000003e-07 2024-09-18T08:18:21.642 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:18:21.642 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:18:21.642 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "all_read", 2024-09-18T08:18:21.643 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:15:00.408949+0000", 2024-09-18T08:18:21.643 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.000103719 2024-09-18T08:18:21.643 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:18:21.643 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:18:21.643 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "dispatched", 2024-09-18T08:18:21.643 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:15:00.408960+0000", 2024-09-18T08:18:21.643 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.1082e-05 2024-09-18T08:18:21.643 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:18:21.643 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:18:21.643 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "queued_for_pg", 2024-09-18T08:18:21.643 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:15:00.408967+0000", 2024-09-18T08:18:21.643 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 6.9589999999999998e-06 2024-09-18T08:18:21.644 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:18:21.644 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:18:21.644 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-09-18T08:18:21.644 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:15:00.409026+0000", 2024-09-18T08:18:21.644 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 5.8570000000000003e-05 2024-09-18T08:18:21.644 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:18:21.644 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:18:21.644 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "waiting for peered", 2024-09-18T08:18:21.644 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:15:00.409033+0000", 2024-09-18T08:18:21.644 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 7.4850000000000003e-06 2024-09-18T08:18:21.644 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:18:21.645 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:18:21.645 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-09-18T08:18:21.645 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:15:01.380947+0000", 2024-09-18T08:18:21.645 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.9719139 2024-09-18T08:18:21.645 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:18:21.645 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:18:21.645 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "started", 2024-09-18T08:18:21.645 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:15:01.381063+0000", 2024-09-18T08:18:21.645 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.000115841 2024-09-18T08:18:21.645 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:18:21.645 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:18:21.646 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "waiting for subops from 3,4", 2024-09-18T08:18:21.646 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:15:01.381182+0000", 2024-09-18T08:18:21.646 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.000119503 2024-09-18T08:18:21.646 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:18:21.646 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:18:21.646 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "op_commit", 2024-09-18T08:18:21.646 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:15:01.383357+0000", 2024-09-18T08:18:21.646 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.002174619 2024-09-18T08:18:21.646 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:18:21.646 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:18:21.646 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:18:21.646 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:15:01.386021+0000", 2024-09-18T08:18:21.647 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.002663941 2024-09-18T08:18:21.647 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:18:21.647 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:18:21.647 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:18:21.647 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:15:01.387705+0000", 2024-09-18T08:18:21.647 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.001684261 2024-09-18T08:18:21.647 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:18:21.647 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:18:21.647 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "commit_sent", 2024-09-18T08:18:21.647 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:15:01.387756+0000", 2024-09-18T08:18:21.647 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 5.0906999999999999e-05 2024-09-18T08:18:21.648 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:18:21.648 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:18:21.648 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "done", 2024-09-18T08:18:21.648 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:15:01.387799+0000", 2024-09-18T08:18:21.648 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 4.2728000000000002e-05 2024-09-18T08:18:21.648 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-09-18T08:18:21.648 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-09-18T08:18:21.648 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-09-18T08:18:21.648 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:18:21.648 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:18:21.648 INFO:teuthology.orchestra.run.smithi007.stdout: "description": "osd_op(client.15966.0:98625 10.9 10:9690b9c0:::benchmark_data_smithi007_148834_object98624:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e347)", 2024-09-18T08:18:21.649 INFO:teuthology.orchestra.run.smithi007.stdout: "initiated_at": "2024-09-18T08:16:12.377991+0000", 2024-09-18T08:18:21.649 INFO:teuthology.orchestra.run.smithi007.stdout: "age": 128.38733883099999, 2024-09-18T08:18:21.649 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 2.0645955950000001, 2024-09-18T08:18:21.649 INFO:teuthology.orchestra.run.smithi007.stdout: "continuous": false, 2024-09-18T08:18:21.649 INFO:teuthology.orchestra.run.smithi007.stdout: "type_data": { 2024-09-18T08:18:21.649 INFO:teuthology.orchestra.run.smithi007.stdout: "flag_point": "reached pg", 2024-09-18T08:18:21.649 INFO:teuthology.orchestra.run.smithi007.stdout: "client_info": { 2024-09-18T08:18:21.649 INFO:teuthology.orchestra.run.smithi007.stdout: "client": "client.15966", 2024-09-18T08:18:21.649 INFO:teuthology.orchestra.run.smithi007.stdout: "client_addr": "172.21.15.7:0/529784850", 2024-09-18T08:18:21.649 INFO:teuthology.orchestra.run.smithi007.stdout: "tid": 98625 2024-09-18T08:18:21.649 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:18:21.649 INFO:teuthology.orchestra.run.smithi007.stdout: "events": [ 2024-09-18T08:18:21.650 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:18:21.650 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "initiated", 2024-09-18T08:18:21.650 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:16:12.377991+0000", 2024-09-18T08:18:21.650 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-09-18T08:18:21.650 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:18:21.650 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:18:21.650 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "header_read", 2024-09-18T08:18:21.650 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:16:12.377991+0000", 2024-09-18T08:18:21.650 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-09-18T08:18:21.650 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:18:21.651 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:18:21.651 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "throttled", 2024-09-18T08:18:21.651 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:16:12.377992+0000", 2024-09-18T08:18:21.651 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 4.9999999999999998e-07 2024-09-18T08:18:21.651 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:18:21.651 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:18:21.651 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "all_read", 2024-09-18T08:18:21.651 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:16:12.378041+0000", 2024-09-18T08:18:21.651 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 4.8764000000000003e-05 2024-09-18T08:18:21.651 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:18:21.651 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:18:21.652 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "dispatched", 2024-09-18T08:18:21.652 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:16:12.378047+0000", 2024-09-18T08:18:21.652 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 6.6409999999999996e-06 2024-09-18T08:18:21.652 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:18:21.652 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:18:21.652 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "queued_for_pg", 2024-09-18T08:18:21.652 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:16:12.378053+0000", 2024-09-18T08:18:21.652 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 5.4140000000000002e-06 2024-09-18T08:18:21.652 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:18:21.652 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:18:21.652 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-09-18T08:18:21.653 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:16:12.387065+0000", 2024-09-18T08:18:21.653 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.0090127599999999999 2024-09-18T08:18:21.653 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:18:21.653 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:18:21.653 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "started", 2024-09-18T08:18:21.653 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:16:12.387212+0000", 2024-09-18T08:18:21.653 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.000146169 2024-09-18T08:18:21.653 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:18:21.653 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:18:21.653 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "waiting for subops from 3,4,5", 2024-09-18T08:18:21.653 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:16:12.387395+0000", 2024-09-18T08:18:21.654 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.00018346500000000001 2024-09-18T08:18:21.654 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:18:21.654 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:18:21.654 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:18:21.654 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:16:12.396723+0000", 2024-09-18T08:18:21.654 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.0093279699999999997 2024-09-18T08:18:21.654 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:18:21.654 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:18:21.654 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:18:21.654 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:16:12.396785+0000", 2024-09-18T08:18:21.654 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 6.2169999999999996e-05 2024-09-18T08:18:21.655 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:18:21.655 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:18:21.655 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:18:21.655 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:16:12.422101+0000", 2024-09-18T08:18:21.655 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.025315666000000001 2024-09-18T08:18:21.655 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:18:21.655 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:18:21.655 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-09-18T08:18:21.655 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:16:12.425399+0000", 2024-09-18T08:18:21.655 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.0032976490000000002 2024-09-18T08:18:21.655 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:18:21.655 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:18:21.656 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "waiting for peered", 2024-09-18T08:18:21.656 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:16:12.425410+0000", 2024-09-18T08:18:21.656 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.1423000000000001e-05 2024-09-18T08:18:21.656 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:18:21.656 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:18:21.656 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-09-18T08:18:21.656 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:16:13.429965+0000", 2024-09-18T08:18:21.656 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.004555144 2024-09-18T08:18:21.656 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:18:21.656 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:18:21.656 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "waiting for peered", 2024-09-18T08:18:21.656 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:16:13.429977+0000", 2024-09-18T08:18:21.657 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.1511e-05 2024-09-18T08:18:21.657 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:18:21.657 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:18:21.657 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-09-18T08:18:21.657 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:16:14.442511+0000", 2024-09-18T08:18:21.657 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.012533948 2024-09-18T08:18:21.657 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:18:21.657 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:18:21.657 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "done", 2024-09-18T08:18:21.657 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:16:14.442587+0000", 2024-09-18T08:18:21.657 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 7.6401000000000003e-05 2024-09-18T08:18:21.657 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-09-18T08:18:21.658 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-09-18T08:18:21.658 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-09-18T08:18:21.658 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:18:21.658 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:18:21.658 INFO:teuthology.orchestra.run.smithi007.stdout: "description": "osd_op(client.15966.0:98630 10.9 10:93666e30:::benchmark_data_smithi007_148834_object98629:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e347)", 2024-09-18T08:18:21.658 INFO:teuthology.orchestra.run.smithi007.stdout: "initiated_at": "2024-09-18T08:16:12.389940+0000", 2024-09-18T08:18:21.658 INFO:teuthology.orchestra.run.smithi007.stdout: "age": 128.37538989399999, 2024-09-18T08:18:21.658 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 2.0536796430000002, 2024-09-18T08:18:21.658 INFO:teuthology.orchestra.run.smithi007.stdout: "continuous": false, 2024-09-18T08:18:21.658 INFO:teuthology.orchestra.run.smithi007.stdout: "type_data": { 2024-09-18T08:18:21.658 INFO:teuthology.orchestra.run.smithi007.stdout: "flag_point": "reached pg", 2024-09-18T08:18:21.659 INFO:teuthology.orchestra.run.smithi007.stdout: "client_info": { 2024-09-18T08:18:21.659 INFO:teuthology.orchestra.run.smithi007.stdout: "client": "client.15966", 2024-09-18T08:18:21.659 INFO:teuthology.orchestra.run.smithi007.stdout: "client_addr": "172.21.15.7:0/529784850", 2024-09-18T08:18:21.659 INFO:teuthology.orchestra.run.smithi007.stdout: "tid": 98630 2024-09-18T08:18:21.659 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:18:21.659 INFO:teuthology.orchestra.run.smithi007.stdout: "events": [ 2024-09-18T08:18:21.659 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:18:21.659 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "initiated", 2024-09-18T08:18:21.659 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:16:12.389940+0000", 2024-09-18T08:18:21.659 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-09-18T08:18:21.659 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:18:21.659 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:18:21.660 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "header_read", 2024-09-18T08:18:21.660 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:16:12.389940+0000", 2024-09-18T08:18:21.660 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-09-18T08:18:21.660 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:18:21.660 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:18:21.660 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "throttled", 2024-09-18T08:18:21.660 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:16:12.389941+0000", 2024-09-18T08:18:21.660 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.049e-06 2024-09-18T08:18:21.660 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:18:21.660 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:18:21.660 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "all_read", 2024-09-18T08:18:21.660 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:16:12.389964+0000", 2024-09-18T08:18:21.661 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 2.3085e-05 2024-09-18T08:18:21.661 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:18:21.661 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:18:21.661 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "dispatched", 2024-09-18T08:18:21.661 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:16:12.389975+0000", 2024-09-18T08:18:21.661 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.0988999999999999e-05 2024-09-18T08:18:21.661 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:18:21.661 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:18:21.661 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "queued_for_pg", 2024-09-18T08:18:21.661 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:16:12.389983+0000", 2024-09-18T08:18:21.661 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 7.9349999999999994e-06 2024-09-18T08:18:21.661 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:18:21.662 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:18:21.662 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-09-18T08:18:21.662 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:16:12.396823+0000", 2024-09-18T08:18:21.662 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.0068393330000000004 2024-09-18T08:18:21.662 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:18:21.662 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:18:21.662 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "started", 2024-09-18T08:18:21.662 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:16:12.396999+0000", 2024-09-18T08:18:21.662 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.00017637999999999999 2024-09-18T08:18:21.662 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:18:21.662 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:18:21.663 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "waiting for subops from 3,4,5", 2024-09-18T08:18:21.663 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:16:12.397192+0000", 2024-09-18T08:18:21.663 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.00019290900000000001 2024-09-18T08:18:21.663 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:18:21.663 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:18:21.663 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:18:21.663 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:16:12.422147+0000", 2024-09-18T08:18:21.663 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.024954787999999999 2024-09-18T08:18:21.663 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:18:21.663 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:18:21.663 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-09-18T08:18:21.664 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:16:12.425442+0000", 2024-09-18T08:18:21.664 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.0032947530000000001 2024-09-18T08:18:21.664 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:18:21.664 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:18:21.664 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "waiting for peered", 2024-09-18T08:18:21.664 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:16:12.425449+0000", 2024-09-18T08:18:21.664 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 7.5390000000000002e-06 2024-09-18T08:18:21.664 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:18:21.664 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:18:21.664 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-09-18T08:18:21.664 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:16:13.430015+0000", 2024-09-18T08:18:21.665 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.0045655259999999 2024-09-18T08:18:21.665 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:18:21.665 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:18:21.665 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "waiting for peered", 2024-09-18T08:18:21.665 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:16:13.430024+0000", 2024-09-18T08:18:21.665 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 9.1670000000000005e-06 2024-09-18T08:18:21.665 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:18:21.665 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:18:21.665 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-09-18T08:18:21.665 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:16:14.443546+0000", 2024-09-18T08:18:21.665 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.0135222079999999 2024-09-18T08:18:21.666 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:18:21.666 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:18:21.666 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "done", 2024-09-18T08:18:21.666 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:16:14.443620+0000", 2024-09-18T08:18:21.666 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 7.3981999999999997e-05 2024-09-18T08:18:21.666 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-09-18T08:18:21.666 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-09-18T08:18:21.666 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-09-18T08:18:21.666 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-09-18T08:18:21.666 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-09-18T08:18:21.666 INFO:teuthology.orchestra.run.smithi007.stdout:} 2024-09-18T08:18:21.667 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5 2024-09-18T08:18:21.692 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:18:21.693 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:18:21.693 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5' 2024-09-18T08:18:21.782 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:18:21 smithi007 ceph-mon[39249]: 4.1f scrub starts 2024-09-18T08:18:21.782 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:18:21 smithi007 ceph-mon[39249]: 4.1f scrub ok 2024-09-18T08:18:21.782 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:18:21 smithi007 ceph-mon[39249]: from='client.? 172.21.15.7:0/1458934672' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-18T08:18:21.782 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:18:21 smithi007 ceph-mon[39249]: 1.6 scrub starts 2024-09-18T08:18:21.782 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:18:21 smithi007 ceph-mon[39249]: 1.6 scrub ok 2024-09-18T08:18:21.782 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:18:21 smithi007 ceph-mon[29459]: 4.1f scrub starts 2024-09-18T08:18:21.782 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:18:21 smithi007 ceph-mon[29459]: 4.1f scrub ok 2024-09-18T08:18:21.782 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:18:21 smithi007 ceph-mon[29459]: from='client.? 172.21.15.7:0/1458934672' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-18T08:18:21.783 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:18:21 smithi007 ceph-mon[29459]: 1.6 scrub starts 2024-09-18T08:18:21.783 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:18:21 smithi007 ceph-mon[29459]: 1.6 scrub ok 2024-09-18T08:18:21.794 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3 2024-09-18T08:18:21.817 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:18:21.817 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:18:21.818 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3' 2024-09-18T08:18:21.905 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:18:21 smithi050 ceph-mon[32371]: 4.1f scrub starts 2024-09-18T08:18:21.905 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:18:21 smithi050 ceph-mon[32371]: 4.1f scrub ok 2024-09-18T08:18:21.905 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:18:21 smithi050 ceph-mon[32371]: from='client.? 172.21.15.7:0/1458934672' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-18T08:18:21.905 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:18:21 smithi050 ceph-mon[32371]: 1.6 scrub starts 2024-09-18T08:18:21.905 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:18:21 smithi050 ceph-mon[32371]: 1.6 scrub ok 2024-09-18T08:18:21.920 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2 2024-09-18T08:18:21.943 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:18:21.943 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:18:21.944 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2' 2024-09-18T08:18:22.045 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7 2024-09-18T08:18:22.072 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:18:22.073 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:18:22.073 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7' 2024-09-18T08:18:22.174 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5 2024-09-18T08:18:22.201 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:18:22.201 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:18:22.202 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5' 2024-09-18T08:18:22.212 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid 779349ca-7593-11ef-bceb-c7b262605968 -- ceph daemon osd.3 dump_ops_in_flight 2024-09-18T08:18:22.302 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7 2024-09-18T08:18:22.329 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:18:22.330 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:18:22.330 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7' 2024-09-18T08:18:22.431 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1 2024-09-18T08:18:22.453 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:18:22.454 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:18:22.454 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1' 2024-09-18T08:18:22.555 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4 2024-09-18T08:18:22.582 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:18:22.583 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:18:22.584 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4' 2024-09-18T08:18:22.685 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5 2024-09-18T08:18:22.711 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:18:22.711 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:18:22.712 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5' 2024-09-18T08:18:22.721 INFO:teuthology.task.full_sequential:In full_sequential, running task radosbench... 2024-09-18T08:18:22.722 INFO:tasks.radosbench:Beginning radosbench... 2024-09-18T08:18:22.722 INFO:tasks.cephadm.ceph_manager.ceph:creating pool_name unique_pool_4 2024-09-18T08:18:22.722 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid 779349ca-7593-11ef-bceb-c7b262605968 -- ceph osd pool create unique_pool_4 16 2024-09-18T08:18:22.813 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5 2024-09-18T08:18:22.839 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:18:22.840 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:18:22.840 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5' 2024-09-18T08:18:22.905 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:18:22 smithi050 ceph-mon[32371]: pgmap v781: 142 pgs: 4 peering, 138 active+clean; 583 KiB data, 3.3 GiB used, 533 GiB / 536 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-18T08:18:22.905 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:18:22 smithi050 ceph-mon[32371]: 5.19 scrub starts 2024-09-18T08:18:22.905 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:18:22 smithi050 ceph-mon[32371]: 5.19 scrub ok 2024-09-18T08:18:22.905 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:18:22 smithi050 ceph-mon[32371]: 6.13 scrub starts 2024-09-18T08:18:22.905 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:18:22 smithi050 ceph-mon[32371]: 6.13 scrub ok 2024-09-18T08:18:22.942 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7 2024-09-18T08:18:22.945 INFO:teuthology.orchestra.run.smithi007.stdout:osd.0: {} 2024-09-18T08:18:22.945 INFO:teuthology.orchestra.run.smithi007.stdout:osd.1: {} 2024-09-18T08:18:22.945 INFO:teuthology.orchestra.run.smithi007.stdout:osd.2: {} 2024-09-18T08:18:22.945 INFO:teuthology.orchestra.run.smithi007.stdout:osd.3: {} 2024-09-18T08:18:22.945 INFO:teuthology.orchestra.run.smithi007.stdout:osd.4: {} 2024-09-18T08:18:22.946 INFO:teuthology.orchestra.run.smithi007.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-18T08:18:22.946 INFO:teuthology.orchestra.run.smithi007.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-18T08:18:22.946 INFO:teuthology.orchestra.run.smithi007.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-18T08:18:22.946 INFO:teuthology.orchestra.run.smithi007.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-18T08:18:22.946 INFO:teuthology.orchestra.run.smithi007.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-18T08:18:22.946 INFO:teuthology.orchestra.run.smithi007.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-18T08:18:22.947 INFO:teuthology.orchestra.run.smithi007.stdout:osd.5: {} 2024-09-18T08:18:22.947 INFO:teuthology.orchestra.run.smithi007.stdout:osd.6: {} 2024-09-18T08:18:22.947 INFO:teuthology.orchestra.run.smithi007.stdout:osd.7: {} 2024-09-18T08:18:22.947 INFO:teuthology.orchestra.run.smithi007.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-18T08:18:22.947 INFO:teuthology.orchestra.run.smithi007.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-18T08:18:22.947 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:18:22 smithi007 ceph-mon[39249]: pgmap v781: 142 pgs: 4 peering, 138 active+clean; 583 KiB data, 3.3 GiB used, 533 GiB / 536 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-18T08:18:22.948 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:18:22 smithi007 ceph-mon[39249]: 5.19 scrub starts 2024-09-18T08:18:22.948 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:18:22 smithi007 ceph-mon[39249]: 5.19 scrub ok 2024-09-18T08:18:22.948 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:18:22 smithi007 ceph-mon[39249]: 6.13 scrub starts 2024-09-18T08:18:22.948 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:18:22 smithi007 ceph-mon[39249]: 6.13 scrub ok 2024-09-18T08:18:22.949 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:18:22 smithi007 ceph-mon[29459]: pgmap v781: 142 pgs: 4 peering, 138 active+clean; 583 KiB data, 3.3 GiB used, 533 GiB / 536 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-18T08:18:22.949 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:18:22 smithi007 ceph-mon[29459]: 5.19 scrub starts 2024-09-18T08:18:22.949 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:18:22 smithi007 ceph-mon[29459]: 5.19 scrub ok 2024-09-18T08:18:22.949 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:18:22 smithi007 ceph-mon[29459]: 6.13 scrub starts 2024-09-18T08:18:22.949 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:18:22 smithi007 ceph-mon[29459]: 6.13 scrub ok 2024-09-18T08:18:22.969 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:18:22.969 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:18:22.970 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7' 2024-09-18T08:18:23.071 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3 2024-09-18T08:18:23.093 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:18:23.094 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:18:23.095 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3' 2024-09-18T08:18:23.196 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6 2024-09-18T08:18:23.223 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:18:23.224 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:18:23.225 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6' 2024-09-18T08:18:23.325 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6 2024-09-18T08:18:23.353 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:18:23.354 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:18:23.354 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6' 2024-09-18T08:18:23.380 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 6, 5] out_osds: [7, 4] dead_osds: [] live_osds: [0, 2, 3, 4, 5, 6, 7, 1] 2024-09-18T08:18:23.381 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-18T08:18:23.381 INFO:tasks.thrashosds.thrasher:Adding osd 7 2024-09-18T08:18:23.381 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid 779349ca-7593-11ef-bceb-c7b262605968 -- ceph osd in 7 2024-09-18T08:18:23.455 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7 2024-09-18T08:18:23.483 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:18:23.484 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:18:23.484 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7' 2024-09-18T08:18:23.585 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1 2024-09-18T08:18:23.612 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:18:23.613 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:18:23.614 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1' 2024-09-18T08:18:23.715 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0 2024-09-18T08:18:23.737 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:18:23.738 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:18:23.739 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0' 2024-09-18T08:18:23.840 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1 2024-09-18T08:18:23.862 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:18:23.863 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:18:23.864 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1' 2024-09-18T08:18:23.905 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:18:23 smithi050 ceph-mon[32371]: 5.b scrub starts 2024-09-18T08:18:23.905 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:18:23 smithi050 ceph-mon[32371]: 5.b scrub ok 2024-09-18T08:18:23.905 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:18:23 smithi050 ceph-mon[32371]: 6.19 scrub starts 2024-09-18T08:18:23.905 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:18:23 smithi050 ceph-mon[32371]: 6.19 scrub ok 2024-09-18T08:18:23.906 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:18:23 smithi050 ceph-mon[32371]: 3.c scrub starts 2024-09-18T08:18:23.906 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:18:23 smithi050 ceph-mon[32371]: 3.c scrub ok 2024-09-18T08:18:23.906 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:18:23 smithi050 ceph-mon[32371]: 4.2 scrub starts 2024-09-18T08:18:23.906 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:18:23 smithi050 ceph-mon[32371]: 4.2 scrub ok 2024-09-18T08:18:23.906 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:18:23 smithi050 ceph-mon[32371]: 3.1f scrub starts 2024-09-18T08:18:23.906 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:18:23 smithi050 ceph-mon[32371]: 3.1f scrub ok 2024-09-18T08:18:23.964 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4 2024-09-18T08:18:23.980 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:18:23 smithi007 ceph-mon[29459]: 5.b scrub starts 2024-09-18T08:18:23.980 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:18:23 smithi007 ceph-mon[29459]: 5.b scrub ok 2024-09-18T08:18:23.980 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:18:23 smithi007 ceph-mon[29459]: 6.19 scrub starts 2024-09-18T08:18:23.980 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:18:23 smithi007 ceph-mon[29459]: 6.19 scrub ok 2024-09-18T08:18:23.980 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:18:23 smithi007 ceph-mon[29459]: 3.c scrub starts 2024-09-18T08:18:23.980 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:18:23 smithi007 ceph-mon[29459]: 3.c scrub ok 2024-09-18T08:18:23.980 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:18:23 smithi007 ceph-mon[29459]: 4.2 scrub starts 2024-09-18T08:18:23.980 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:18:23 smithi007 ceph-mon[29459]: 4.2 scrub ok 2024-09-18T08:18:23.981 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:18:23 smithi007 ceph-mon[29459]: 3.1f scrub starts 2024-09-18T08:18:23.981 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:18:23 smithi007 ceph-mon[29459]: 3.1f scrub ok 2024-09-18T08:18:23.981 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:18:23 smithi007 ceph-mon[39249]: 5.b scrub starts 2024-09-18T08:18:23.981 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:18:23 smithi007 ceph-mon[39249]: 5.b scrub ok 2024-09-18T08:18:23.981 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:18:23 smithi007 ceph-mon[39249]: 6.19 scrub starts 2024-09-18T08:18:23.981 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:18:23 smithi007 ceph-mon[39249]: 6.19 scrub ok 2024-09-18T08:18:23.981 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:18:23 smithi007 ceph-mon[39249]: 3.c scrub starts 2024-09-18T08:18:23.981 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:18:23 smithi007 ceph-mon[39249]: 3.c scrub ok 2024-09-18T08:18:23.981 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:18:23 smithi007 ceph-mon[39249]: 4.2 scrub starts 2024-09-18T08:18:23.981 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:18:23 smithi007 ceph-mon[39249]: 4.2 scrub ok 2024-09-18T08:18:23.981 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:18:23 smithi007 ceph-mon[39249]: 3.1f scrub starts 2024-09-18T08:18:23.981 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:18:23 smithi007 ceph-mon[39249]: 3.1f scrub ok 2024-09-18T08:18:23.992 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:18:23.992 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:18:23.993 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4' 2024-09-18T08:18:24.035 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/779349ca-7593-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T08:18:24.094 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7 2024-09-18T08:18:24.121 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:18:24.122 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:18:24.122 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7' 2024-09-18T08:18:24.223 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6 2024-09-18T08:18:24.249 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:18:24.250 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:18:24.250 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6' 2024-09-18T08:18:24.302 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/779349ca-7593-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T08:18:24.351 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7 2024-09-18T08:18:24.378 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:18:24.379 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:18:24.379 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7' 2024-09-18T08:18:24.480 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0 2024-09-18T08:18:24.505 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:18:24.506 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:18:24.506 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0' 2024-09-18T08:18:24.588 INFO:teuthology.orchestra.run.smithi007.stdout:{ 2024-09-18T08:18:24.588 INFO:teuthology.orchestra.run.smithi007.stdout: "ops": [], 2024-09-18T08:18:24.589 INFO:teuthology.orchestra.run.smithi007.stdout: "num_ops": 0 2024-09-18T08:18:24.589 INFO:teuthology.orchestra.run.smithi007.stdout:} 2024-09-18T08:18:24.589 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/779349ca-7593-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T08:18:24.607 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6 2024-09-18T08:18:24.635 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:18:24.636 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:18:24.636 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6' 2024-09-18T08:18:24.737 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5 2024-09-18T08:18:24.764 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:18:24.764 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:18:24.765 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5' 2024-09-18T08:18:24.866 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5 2024-09-18T08:18:24.891 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:18:24.892 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:18:24.892 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5' 2024-09-18T08:18:24.905 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:18:24 smithi050 ceph-mon[32371]: 3.8 scrub starts 2024-09-18T08:18:24.905 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:18:24 smithi050 ceph-mon[32371]: 3.8 scrub ok 2024-09-18T08:18:24.905 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:18:24 smithi050 ceph-mon[32371]: pgmap v782: 142 pgs: 4 peering, 138 active+clean; 583 KiB data, 3.3 GiB used, 533 GiB / 536 GiB avail; 873 B/s rd, 0 op/s 2024-09-18T08:18:24.905 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:18:24 smithi050 ceph-mon[32371]: 4.a scrub starts 2024-09-18T08:18:24.905 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:18:24 smithi050 ceph-mon[32371]: 4.a scrub ok 2024-09-18T08:18:24.905 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:18:24 smithi050 ceph-mon[32371]: 6.0 scrub starts 2024-09-18T08:18:24.905 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:18:24 smithi050 ceph-mon[32371]: 6.0 scrub ok 2024-09-18T08:18:24.906 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:18:24 smithi050 ceph-mon[32371]: purged_snaps scrub starts 2024-09-18T08:18:24.906 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:18:24 smithi050 ceph-mon[32371]: purged_snaps scrub ok 2024-09-18T08:18:24.906 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:18:24 smithi050 ceph-mon[32371]: osdmap e400: 8 total, 8 up, 6 in 2024-09-18T08:18:24.929 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:18:24 smithi007 ceph-mon[29459]: 3.8 scrub starts 2024-09-18T08:18:24.929 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:18:24 smithi007 ceph-mon[29459]: 3.8 scrub ok 2024-09-18T08:18:24.929 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:18:24 smithi007 ceph-mon[29459]: pgmap v782: 142 pgs: 4 peering, 138 active+clean; 583 KiB data, 3.3 GiB used, 533 GiB / 536 GiB avail; 873 B/s rd, 0 op/s 2024-09-18T08:18:24.929 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:18:24 smithi007 ceph-mon[29459]: 4.a scrub starts 2024-09-18T08:18:24.929 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:18:24 smithi007 ceph-mon[29459]: 4.a scrub ok 2024-09-18T08:18:24.929 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:18:24 smithi007 ceph-mon[29459]: 6.0 scrub starts 2024-09-18T08:18:24.929 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:18:24 smithi007 ceph-mon[29459]: 6.0 scrub ok 2024-09-18T08:18:24.929 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:18:24 smithi007 ceph-mon[29459]: purged_snaps scrub starts 2024-09-18T08:18:24.929 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:18:24 smithi007 ceph-mon[29459]: purged_snaps scrub ok 2024-09-18T08:18:24.929 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:18:24 smithi007 ceph-mon[29459]: osdmap e400: 8 total, 8 up, 6 in 2024-09-18T08:18:24.930 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:18:24 smithi007 ceph-mon[39249]: 3.8 scrub starts 2024-09-18T08:18:24.930 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:18:24 smithi007 ceph-mon[39249]: 3.8 scrub ok 2024-09-18T08:18:24.930 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:18:24 smithi007 ceph-mon[39249]: pgmap v782: 142 pgs: 4 peering, 138 active+clean; 583 KiB data, 3.3 GiB used, 533 GiB / 536 GiB avail; 873 B/s rd, 0 op/s 2024-09-18T08:18:24.930 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:18:24 smithi007 ceph-mon[39249]: 4.a scrub starts 2024-09-18T08:18:24.930 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:18:24 smithi007 ceph-mon[39249]: 4.a scrub ok 2024-09-18T08:18:24.930 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:18:24 smithi007 ceph-mon[39249]: 6.0 scrub starts 2024-09-18T08:18:24.930 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:18:24 smithi007 ceph-mon[39249]: 6.0 scrub ok 2024-09-18T08:18:24.930 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:18:24 smithi007 ceph-mon[39249]: purged_snaps scrub starts 2024-09-18T08:18:24.930 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:18:24 smithi007 ceph-mon[39249]: purged_snaps scrub ok 2024-09-18T08:18:24.930 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:18:24 smithi007 ceph-mon[39249]: osdmap e400: 8 total, 8 up, 6 in 2024-09-18T08:18:24.993 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1 2024-09-18T08:18:25.015 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:18:25.016 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:18:25.016 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1' 2024-09-18T08:18:25.117 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0 2024-09-18T08:18:25.139 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:18:25.139 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:18:25.140 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0' 2024-09-18T08:18:25.241 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4 2024-09-18T08:18:25.253 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid 779349ca-7593-11ef-bceb-c7b262605968 -- ceph daemon osd.3 dump_blocked_ops 2024-09-18T08:18:25.263 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:18:25.263 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:18:25.264 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4' 2024-09-18T08:18:25.364 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6 2024-09-18T08:18:25.386 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:18:25.387 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:18:25.387 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6' 2024-09-18T08:18:25.488 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1 2024-09-18T08:18:25.512 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:18:25.512 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:18:25.513 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1' 2024-09-18T08:18:25.582 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid 779349ca-7593-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-18T08:18:25.614 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0 2024-09-18T08:18:25.647 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:18:25.648 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:18:25.648 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0' 2024-09-18T08:18:25.749 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2 2024-09-18T08:18:25.766 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:18:25 smithi007 ceph-mon[29459]: purged_snaps scrub starts 2024-09-18T08:18:25.767 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:18:25 smithi007 ceph-mon[29459]: purged_snaps scrub ok 2024-09-18T08:18:25.767 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:18:25 smithi007 ceph-mon[29459]: 6.17 scrub starts 2024-09-18T08:18:25.767 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:18:25 smithi007 ceph-mon[29459]: 6.17 scrub ok 2024-09-18T08:18:25.767 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:18:25 smithi007 ceph-mon[29459]: 5.15 scrub starts 2024-09-18T08:18:25.767 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:18:25 smithi007 ceph-mon[29459]: 5.15 scrub ok 2024-09-18T08:18:25.767 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:18:25 smithi007 ceph-mon[29459]: 3.12 scrub starts 2024-09-18T08:18:25.767 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:18:25 smithi007 ceph-mon[29459]: 3.12 scrub ok 2024-09-18T08:18:25.767 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:18:25 smithi007 ceph-mon[29459]: from='mgr.24470 172.21.15.7:0/2594390261' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-18T08:18:25.767 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:18:25 smithi007 ceph-mon[29459]: from='mgr.24470 172.21.15.7:0/2594390261' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-18T08:18:25.767 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:18:25 smithi007 ceph-mon[29459]: from='mgr.24470 172.21.15.7:0/2594390261' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-18T08:18:25.767 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:18:25 smithi007 ceph-mon[29459]: from='mgr.24470 172.21.15.7:0/2594390261' entity='mgr.y' 2024-09-18T08:18:25.768 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:18:25 smithi007 ceph-mon[39249]: purged_snaps scrub starts 2024-09-18T08:18:25.768 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:18:25 smithi007 ceph-mon[39249]: purged_snaps scrub ok 2024-09-18T08:18:25.768 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:18:25 smithi007 ceph-mon[39249]: 6.17 scrub starts 2024-09-18T08:18:25.768 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:18:25 smithi007 ceph-mon[39249]: 6.17 scrub ok 2024-09-18T08:18:25.768 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:18:25 smithi007 ceph-mon[39249]: 5.15 scrub starts 2024-09-18T08:18:25.768 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:18:25 smithi007 ceph-mon[39249]: 5.15 scrub ok 2024-09-18T08:18:25.768 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:18:25 smithi007 ceph-mon[39249]: 3.12 scrub starts 2024-09-18T08:18:25.768 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:18:25 smithi007 ceph-mon[39249]: 3.12 scrub ok 2024-09-18T08:18:25.768 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:18:25 smithi007 ceph-mon[39249]: from='mgr.24470 172.21.15.7:0/2594390261' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-18T08:18:25.768 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:18:25 smithi007 ceph-mon[39249]: from='mgr.24470 172.21.15.7:0/2594390261' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-18T08:18:25.768 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:18:25 smithi007 ceph-mon[39249]: from='mgr.24470 172.21.15.7:0/2594390261' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-18T08:18:25.768 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:18:25 smithi007 ceph-mon[39249]: from='mgr.24470 172.21.15.7:0/2594390261' entity='mgr.y' 2024-09-18T08:18:25.777 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:18:25.779 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:18:25.779 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2' 2024-09-18T08:18:25.880 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0 2024-09-18T08:18:25.903 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:18:25.904 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:18:25.904 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0' 2024-09-18T08:18:25.905 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:18:25 smithi050 ceph-mon[32371]: purged_snaps scrub starts 2024-09-18T08:18:25.905 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:18:25 smithi050 ceph-mon[32371]: purged_snaps scrub ok 2024-09-18T08:18:25.905 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:18:25 smithi050 ceph-mon[32371]: 6.17 scrub starts 2024-09-18T08:18:25.905 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:18:25 smithi050 ceph-mon[32371]: 6.17 scrub ok 2024-09-18T08:18:25.905 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:18:25 smithi050 ceph-mon[32371]: 5.15 scrub starts 2024-09-18T08:18:25.905 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:18:25 smithi050 ceph-mon[32371]: 5.15 scrub ok 2024-09-18T08:18:25.905 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:18:25 smithi050 ceph-mon[32371]: 3.12 scrub starts 2024-09-18T08:18:25.905 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:18:25 smithi050 ceph-mon[32371]: 3.12 scrub ok 2024-09-18T08:18:25.905 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:18:25 smithi050 ceph-mon[32371]: from='mgr.24470 172.21.15.7:0/2594390261' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-18T08:18:25.906 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:18:25 smithi050 ceph-mon[32371]: from='mgr.24470 172.21.15.7:0/2594390261' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-18T08:18:25.906 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:18:25 smithi050 ceph-mon[32371]: from='mgr.24470 172.21.15.7:0/2594390261' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-18T08:18:25.906 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:18:25 smithi050 ceph-mon[32371]: from='mgr.24470 172.21.15.7:0/2594390261' entity='mgr.y' 2024-09-18T08:18:26.005 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5 2024-09-18T08:18:26.033 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:18:26.034 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:18:26.034 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5' 2024-09-18T08:18:26.046 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid 779349ca-7593-11ef-bceb-c7b262605968 -- ceph osd set noscrub 2024-09-18T08:18:26.135 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5 2024-09-18T08:18:26.161 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:18:26.162 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:18:26.162 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5' 2024-09-18T08:18:26.219 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/779349ca-7593-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T08:18:26.263 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5 2024-09-18T08:18:26.270 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/779349ca-7593-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T08:18:26.289 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:18:26.293 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:18:26.293 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5' 2024-09-18T08:18:26.394 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1 2024-09-18T08:18:26.417 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:18:26.417 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:18:26.418 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1' 2024-09-18T08:18:26.518 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6 2024-09-18T08:18:26.545 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:18:26.546 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:18:26.546 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6' 2024-09-18T08:18:26.614 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:18:26 smithi007 ceph-779349ca-7593-11ef-bceb-c7b262605968-mon-a[29455]: 2024-09-18T08:18:26.442+0000 7fcd2b595640 -1 mon.a@0(leader).osd e400 definitely_dead 0 2024-09-18T08:18:26.622 INFO:teuthology.orchestra.run.smithi007.stderr:marked in osd.7. 2024-09-18T08:18:26.629 INFO:teuthology.orchestra.run.smithi007.stderr:pool 'unique_pool_4' created 2024-09-18T08:18:26.647 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2 2024-09-18T08:18:26.676 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:18:26.676 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:18:26.677 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2' 2024-09-18T08:18:26.778 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3 2024-09-18T08:18:26.800 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:18:26.801 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:18:26.802 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3' 2024-09-18T08:18:26.906 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4 2024-09-18T08:18:26.908 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:18:26 smithi050 ceph-mon[32371]: 3.14 scrub starts 2024-09-18T08:18:26.908 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:18:26 smithi050 ceph-mon[32371]: 3.14 scrub ok 2024-09-18T08:18:26.908 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:18:26 smithi050 ceph-mon[32371]: pgmap v784: 142 pgs: 142 active+clean; 583 KiB data, 3.3 GiB used, 533 GiB / 536 GiB avail; 679 B/s rd, 0 op/s 2024-09-18T08:18:26.908 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:18:26 smithi050 ceph-mon[32371]: 6.7 scrub starts 2024-09-18T08:18:26.908 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:18:26 smithi050 ceph-mon[32371]: 6.7 scrub ok 2024-09-18T08:18:26.909 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:18:26 smithi050 ceph-mon[32371]: 3.17 scrub starts 2024-09-18T08:18:26.909 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:18:26 smithi050 ceph-mon[32371]: 3.17 scrub ok 2024-09-18T08:18:26.909 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:18:26 smithi050 ceph-mon[32371]: from='client.? 172.21.15.7:0/3613013601' entity='client.admin' cmd={"prefix": "osd pool create", "pool": "unique_pool_4", "pg_num": 16} : dispatch 2024-09-18T08:18:26.909 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:18:26 smithi050 ceph-mon[32371]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool create", "pool": "unique_pool_4", "pg_num": 16} : dispatch 2024-09-18T08:18:26.909 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:18:26 smithi050 ceph-mon[32371]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 2 pgs inactive, 2 pgs peering) 2024-09-18T08:18:26.909 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:18:26 smithi050 ceph-mon[32371]: Cluster is now healthy 2024-09-18T08:18:26.909 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:18:26 smithi050 ceph-mon[32371]: from='client.? 172.21.15.7:0/1710466144' entity='client.admin' cmd={"prefix": "osd in", "ids": ["7"]} : dispatch 2024-09-18T08:18:26.909 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:18:26 smithi050 ceph-mon[32371]: from='client.? ' entity='client.admin' cmd={"prefix": "osd in", "ids": ["7"]} : dispatch 2024-09-18T08:18:26.935 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:18:26.936 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:18:26.937 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4' 2024-09-18T08:18:26.980 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Sep 18 08:18:26 smithi007 ceph-779349ca-7593-11ef-bceb-c7b262605968-mgr-y[29688]: ::ffff:172.21.15.50 - - [18/Sep/2024:08:18:26] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-09-18T08:18:26.980 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:18:26 smithi007 ceph-mon[29459]: 3.14 scrub starts 2024-09-18T08:18:26.980 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:18:26 smithi007 ceph-mon[29459]: 3.14 scrub ok 2024-09-18T08:18:26.980 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:18:26 smithi007 ceph-mon[29459]: pgmap v784: 142 pgs: 142 active+clean; 583 KiB data, 3.3 GiB used, 533 GiB / 536 GiB avail; 679 B/s rd, 0 op/s 2024-09-18T08:18:26.980 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:18:26 smithi007 ceph-mon[29459]: 6.7 scrub starts 2024-09-18T08:18:26.980 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:18:26 smithi007 ceph-mon[29459]: 6.7 scrub ok 2024-09-18T08:18:26.980 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:18:26 smithi007 ceph-mon[29459]: 3.17 scrub starts 2024-09-18T08:18:26.981 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:18:26 smithi007 ceph-mon[29459]: 3.17 scrub ok 2024-09-18T08:18:26.981 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:18:26 smithi007 ceph-mon[29459]: from='client.? 172.21.15.7:0/3613013601' entity='client.admin' cmd={"prefix": "osd pool create", "pool": "unique_pool_4", "pg_num": 16} : dispatch 2024-09-18T08:18:26.981 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:18:26 smithi007 ceph-mon[29459]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool create", "pool": "unique_pool_4", "pg_num": 16} : dispatch 2024-09-18T08:18:26.981 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:18:26 smithi007 ceph-mon[29459]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 2 pgs inactive, 2 pgs peering) 2024-09-18T08:18:26.981 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:18:26 smithi007 ceph-mon[29459]: Cluster is now healthy 2024-09-18T08:18:26.981 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:18:26 smithi007 ceph-mon[29459]: from='client.? 172.21.15.7:0/1710466144' entity='client.admin' cmd={"prefix": "osd in", "ids": ["7"]} : dispatch 2024-09-18T08:18:26.981 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:18:26 smithi007 ceph-mon[29459]: from='client.? ' entity='client.admin' cmd={"prefix": "osd in", "ids": ["7"]} : dispatch 2024-09-18T08:18:26.981 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:18:26 smithi007 ceph-mon[39249]: 3.14 scrub starts 2024-09-18T08:18:26.981 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:18:26 smithi007 ceph-mon[39249]: 3.14 scrub ok 2024-09-18T08:18:26.981 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:18:26 smithi007 ceph-mon[39249]: pgmap v784: 142 pgs: 142 active+clean; 583 KiB data, 3.3 GiB used, 533 GiB / 536 GiB avail; 679 B/s rd, 0 op/s 2024-09-18T08:18:26.981 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:18:26 smithi007 ceph-mon[39249]: 6.7 scrub starts 2024-09-18T08:18:26.982 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:18:26 smithi007 ceph-mon[39249]: 6.7 scrub ok 2024-09-18T08:18:26.982 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:18:26 smithi007 ceph-mon[39249]: 3.17 scrub starts 2024-09-18T08:18:26.982 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:18:26 smithi007 ceph-mon[39249]: 3.17 scrub ok 2024-09-18T08:18:26.982 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:18:26 smithi007 ceph-mon[39249]: from='client.? 172.21.15.7:0/3613013601' entity='client.admin' cmd={"prefix": "osd pool create", "pool": "unique_pool_4", "pg_num": 16} : dispatch 2024-09-18T08:18:26.982 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:18:26 smithi007 ceph-mon[39249]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool create", "pool": "unique_pool_4", "pg_num": 16} : dispatch 2024-09-18T08:18:26.982 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:18:26 smithi007 ceph-mon[39249]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 2 pgs inactive, 2 pgs peering) 2024-09-18T08:18:26.982 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:18:26 smithi007 ceph-mon[39249]: Cluster is now healthy 2024-09-18T08:18:26.982 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:18:26 smithi007 ceph-mon[39249]: from='client.? 172.21.15.7:0/1710466144' entity='client.admin' cmd={"prefix": "osd in", "ids": ["7"]} : dispatch 2024-09-18T08:18:26.982 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:18:26 smithi007 ceph-mon[39249]: from='client.? ' entity='client.admin' cmd={"prefix": "osd in", "ids": ["7"]} : dispatch 2024-09-18T08:18:27.037 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4 2024-09-18T08:18:27.064 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:18:27.064 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:18:27.065 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4' 2024-09-18T08:18:27.165 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2 2024-09-18T08:18:27.188 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:18:27.189 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:18:27.189 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2' 2024-09-18T08:18:27.190 INFO:teuthology.orchestra.run.smithi007.stdout:{ 2024-09-18T08:18:27.190 INFO:teuthology.orchestra.run.smithi007.stdout: "ops": [], 2024-09-18T08:18:27.190 INFO:teuthology.orchestra.run.smithi007.stdout: "complaint_time": 30, 2024-09-18T08:18:27.190 INFO:teuthology.orchestra.run.smithi007.stdout: "num_blocked_ops": 0 2024-09-18T08:18:27.190 INFO:teuthology.orchestra.run.smithi007.stdout:} 2024-09-18T08:18:27.290 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5 2024-09-18T08:18:27.317 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:18:27.318 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:18:27.318 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5' 2024-09-18T08:18:27.419 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2 2024-09-18T08:18:27.442 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:18:27.442 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:18:27.442 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2' 2024-09-18T08:18:27.543 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7 2024-09-18T08:18:27.570 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:18:27.571 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:18:27.571 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7' 2024-09-18T08:18:27.672 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5 2024-09-18T08:18:27.701 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:18:27.702 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:18:27.702 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5' 2024-09-18T08:18:27.725 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:18:27 smithi007 ceph-mon[29459]: 3.9 scrub starts 2024-09-18T08:18:27.725 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:18:27 smithi007 ceph-mon[29459]: 3.9 scrub ok 2024-09-18T08:18:27.725 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:18:27 smithi007 ceph-mon[29459]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "unique_pool_4", "pg_num": 16}]': finished 2024-09-18T08:18:27.725 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:18:27 smithi007 ceph-mon[29459]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd in", "ids": ["7"]}]': finished 2024-09-18T08:18:27.725 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:18:27 smithi007 ceph-mon[29459]: osdmap e401: 8 total, 8 up, 7 in 2024-09-18T08:18:27.726 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:18:27 smithi007 ceph-mon[39249]: 3.9 scrub starts 2024-09-18T08:18:27.726 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:18:27 smithi007 ceph-mon[39249]: 3.9 scrub ok 2024-09-18T08:18:27.726 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:18:27 smithi007 ceph-mon[39249]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "unique_pool_4", "pg_num": 16}]': finished 2024-09-18T08:18:27.726 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:18:27 smithi007 ceph-mon[39249]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd in", "ids": ["7"]}]': finished 2024-09-18T08:18:27.726 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:18:27 smithi007 ceph-mon[39249]: osdmap e401: 8 total, 8 up, 7 in 2024-09-18T08:18:27.803 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0 2024-09-18T08:18:27.827 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:18:27.827 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:18:27.828 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0' 2024-09-18T08:18:27.905 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:18:27 smithi050 ceph-mon[32371]: 3.9 scrub starts 2024-09-18T08:18:27.905 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:18:27 smithi050 ceph-mon[32371]: 3.9 scrub ok 2024-09-18T08:18:27.905 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:18:27 smithi050 ceph-mon[32371]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "unique_pool_4", "pg_num": 16}]': finished 2024-09-18T08:18:27.905 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:18:27 smithi050 ceph-mon[32371]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd in", "ids": ["7"]}]': finished 2024-09-18T08:18:27.905 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:18:27 smithi050 ceph-mon[32371]: osdmap e401: 8 total, 8 up, 7 in 2024-09-18T08:18:27.929 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3 2024-09-18T08:18:27.951 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:18:27.952 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:18:27.952 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3' 2024-09-18T08:18:28.053 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1 2024-09-18T08:18:28.075 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:18:28.076 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:18:28.076 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1' 2024-09-18T08:18:28.177 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3 2024-09-18T08:18:28.199 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:18:28.200 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:18:28.200 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3' 2024-09-18T08:18:28.301 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3 2024-09-18T08:18:28.305 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid 779349ca-7593-11ef-bceb-c7b262605968 -- ceph osd pool application enable unique_pool_4 rados --yes-i-really-mean-it || true 2024-09-18T08:18:28.324 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:18:28.324 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:18:28.325 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3' 2024-09-18T08:18:28.425 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1 2024-09-18T08:18:28.447 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:18:28.448 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:18:28.448 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1' 2024-09-18T08:18:28.549 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1 2024-09-18T08:18:28.571 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:18:28.572 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:18:28.572 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1' 2024-09-18T08:18:28.673 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0 2024-09-18T08:18:28.697 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:18:28.697 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:18:28.698 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0' 2024-09-18T08:18:28.798 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6 2024-09-18T08:18:28.825 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:18:28.826 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:18:28.827 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6' 2024-09-18T08:18:28.875 INFO:teuthology.orchestra.run.smithi007.stdout:osd.0: {} 2024-09-18T08:18:28.876 INFO:teuthology.orchestra.run.smithi007.stdout:osd.1: {} 2024-09-18T08:18:28.876 INFO:teuthology.orchestra.run.smithi007.stdout:osd.2: {} 2024-09-18T08:18:28.876 INFO:teuthology.orchestra.run.smithi007.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-18T08:18:28.876 INFO:teuthology.orchestra.run.smithi007.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-18T08:18:28.876 INFO:teuthology.orchestra.run.smithi007.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-18T08:18:28.876 INFO:teuthology.orchestra.run.smithi007.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-18T08:18:28.877 INFO:teuthology.orchestra.run.smithi007.stdout:osd.3: {} 2024-09-18T08:18:28.877 INFO:teuthology.orchestra.run.smithi007.stdout:osd.4: {} 2024-09-18T08:18:28.877 INFO:teuthology.orchestra.run.smithi007.stdout:osd.5: {} 2024-09-18T08:18:28.877 INFO:teuthology.orchestra.run.smithi007.stdout:osd.6: {} 2024-09-18T08:18:28.877 INFO:teuthology.orchestra.run.smithi007.stdout:osd.7: {} 2024-09-18T08:18:28.877 INFO:teuthology.orchestra.run.smithi007.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-18T08:18:28.877 INFO:teuthology.orchestra.run.smithi007.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-18T08:18:28.877 INFO:teuthology.orchestra.run.smithi007.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-18T08:18:28.877 INFO:teuthology.orchestra.run.smithi007.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-18T08:18:28.878 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:18:28 smithi007 ceph-mon[39249]: pgmap v786: 158 pgs: 16 unknown, 142 active+clean; 583 KiB data, 3.3 GiB used, 533 GiB / 536 GiB avail; 729 B/s rd, 0 op/s; 47 B/s, 0 objects/s recovering 2024-09-18T08:18:28.878 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:18:28 smithi007 ceph-mon[39249]: 4.1b scrub starts 2024-09-18T08:18:28.878 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:18:28 smithi007 ceph-mon[39249]: 4.1b scrub ok 2024-09-18T08:18:28.878 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:18:28 smithi007 ceph-mon[39249]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-18T08:18:28.878 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:18:28 smithi007 ceph-mon[39249]: osdmap e402: 8 total, 8 up, 7 in 2024-09-18T08:18:28.878 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:18:28 smithi007 ceph-mon[39249]: 4.4 scrub starts 2024-09-18T08:18:28.878 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:18:28 smithi007 ceph-mon[39249]: 4.4 scrub ok 2024-09-18T08:18:28.878 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:18:28 smithi007 ceph-mon[39249]: from='client.? 172.21.15.7:0/3987220350' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-18T08:18:28.878 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:18:28 smithi007 ceph-mon[29459]: pgmap v786: 158 pgs: 16 unknown, 142 active+clean; 583 KiB data, 3.3 GiB used, 533 GiB / 536 GiB avail; 729 B/s rd, 0 op/s; 47 B/s, 0 objects/s recovering 2024-09-18T08:18:28.879 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:18:28 smithi007 ceph-mon[29459]: 4.1b scrub starts 2024-09-18T08:18:28.879 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:18:28 smithi007 ceph-mon[29459]: 4.1b scrub ok 2024-09-18T08:18:28.879 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:18:28 smithi007 ceph-mon[29459]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-18T08:18:28.879 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:18:28 smithi007 ceph-mon[29459]: osdmap e402: 8 total, 8 up, 7 in 2024-09-18T08:18:28.879 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:18:28 smithi007 ceph-mon[29459]: 4.4 scrub starts 2024-09-18T08:18:28.879 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:18:28 smithi007 ceph-mon[29459]: 4.4 scrub ok 2024-09-18T08:18:28.879 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:18:28 smithi007 ceph-mon[29459]: from='client.? 172.21.15.7:0/3987220350' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-18T08:18:28.905 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:18:28 smithi050 ceph-mon[32371]: pgmap v786: 158 pgs: 16 unknown, 142 active+clean; 583 KiB data, 3.3 GiB used, 533 GiB / 536 GiB avail; 729 B/s rd, 0 op/s; 47 B/s, 0 objects/s recovering 2024-09-18T08:18:28.905 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:18:28 smithi050 ceph-mon[32371]: 4.1b scrub starts 2024-09-18T08:18:28.905 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:18:28 smithi050 ceph-mon[32371]: 4.1b scrub ok 2024-09-18T08:18:28.905 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:18:28 smithi050 ceph-mon[32371]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-18T08:18:28.906 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:18:28 smithi050 ceph-mon[32371]: osdmap e402: 8 total, 8 up, 7 in 2024-09-18T08:18:28.906 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:18:28 smithi050 ceph-mon[32371]: 4.4 scrub starts 2024-09-18T08:18:28.906 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:18:28 smithi050 ceph-mon[32371]: 4.4 scrub ok 2024-09-18T08:18:28.906 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:18:28 smithi050 ceph-mon[32371]: from='client.? 172.21.15.7:0/3987220350' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-18T08:18:28.927 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6 2024-09-18T08:18:28.953 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:18:28.954 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:18:28.955 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6' 2024-09-18T08:18:29.055 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6 2024-09-18T08:18:29.081 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:18:29.081 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:18:29.082 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6' 2024-09-18T08:18:29.183 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3 2024-09-18T08:18:29.210 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:18:29.210 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:18:29.211 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3' 2024-09-18T08:18:29.312 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0 2024-09-18T08:18:29.338 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:18:29.339 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:18:29.339 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0' 2024-09-18T08:18:29.440 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0 2024-09-18T08:18:29.467 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:18:29.467 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:18:29.468 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0' 2024-09-18T08:18:29.569 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6 2024-09-18T08:18:29.594 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:18:29.595 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:18:29.596 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6' 2024-09-18T08:18:29.696 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4 2024-09-18T08:18:29.721 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:18:29.721 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:18:29.722 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4' 2024-09-18T08:18:29.738 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid 779349ca-7593-11ef-bceb-c7b262605968 -- ceph daemon osd.3 dump_historic_ops 2024-09-18T08:18:29.819 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:18:29 smithi007 ceph-mon[29459]: 5.6 scrub starts 2024-09-18T08:18:29.819 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:18:29 smithi007 ceph-mon[29459]: 5.6 scrub ok 2024-09-18T08:18:29.819 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:18:29 smithi007 ceph-mon[29459]: osdmap e403: 8 total, 8 up, 7 in 2024-09-18T08:18:29.819 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:18:29 smithi007 ceph-mon[29459]: 5.13 scrub starts 2024-09-18T08:18:29.819 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:18:29 smithi007 ceph-mon[29459]: 5.13 scrub ok 2024-09-18T08:18:29.819 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:18:29 smithi007 ceph-mon[29459]: 6.9 scrub starts 2024-09-18T08:18:29.819 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:18:29 smithi007 ceph-mon[29459]: 6.9 scrub ok 2024-09-18T08:18:29.819 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:18:29 smithi007 ceph-mon[29459]: osdmap e404: 8 total, 8 up, 7 in 2024-09-18T08:18:29.820 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:18:29 smithi007 ceph-mon[39249]: 5.6 scrub starts 2024-09-18T08:18:29.820 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:18:29 smithi007 ceph-mon[39249]: 5.6 scrub ok 2024-09-18T08:18:29.820 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:18:29 smithi007 ceph-mon[39249]: osdmap e403: 8 total, 8 up, 7 in 2024-09-18T08:18:29.820 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:18:29 smithi007 ceph-mon[39249]: 5.13 scrub starts 2024-09-18T08:18:29.820 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:18:29 smithi007 ceph-mon[39249]: 5.13 scrub ok 2024-09-18T08:18:29.820 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:18:29 smithi007 ceph-mon[39249]: 6.9 scrub starts 2024-09-18T08:18:29.820 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:18:29 smithi007 ceph-mon[39249]: 6.9 scrub ok 2024-09-18T08:18:29.820 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:18:29 smithi007 ceph-mon[39249]: osdmap e404: 8 total, 8 up, 7 in 2024-09-18T08:18:29.822 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5 2024-09-18T08:18:29.846 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:18:29.847 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:18:29.847 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5' 2024-09-18T08:18:29.905 INFO:journalctl@ceph.iscsi.iscsi.a.smithi050.stdout:Sep 18 08:18:29 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-iscsi-iscsi-a[57782]: debug there is no tcmu-runner data available 2024-09-18T08:18:29.905 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:18:29 smithi050 ceph-mon[32371]: 5.6 scrub starts 2024-09-18T08:18:29.905 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:18:29 smithi050 ceph-mon[32371]: 5.6 scrub ok 2024-09-18T08:18:29.905 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:18:29 smithi050 ceph-mon[32371]: osdmap e403: 8 total, 8 up, 7 in 2024-09-18T08:18:29.905 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:18:29 smithi050 ceph-mon[32371]: 5.13 scrub starts 2024-09-18T08:18:29.906 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:18:29 smithi050 ceph-mon[32371]: 5.13 scrub ok 2024-09-18T08:18:29.906 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:18:29 smithi050 ceph-mon[32371]: 6.9 scrub starts 2024-09-18T08:18:29.906 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:18:29 smithi050 ceph-mon[32371]: 6.9 scrub ok 2024-09-18T08:18:29.906 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:18:29 smithi050 ceph-mon[32371]: osdmap e404: 8 total, 8 up, 7 in 2024-09-18T08:18:29.949 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3 2024-09-18T08:18:29.971 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:18:29.972 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:18:29.973 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3' 2024-09-18T08:18:30.074 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2 2024-09-18T08:18:30.096 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:18:30.097 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:18:30.098 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2' 2024-09-18T08:18:30.122 INFO:tasks.thrashosds.thrasher:Added osd 7 2024-09-18T08:18:30.190 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/779349ca-7593-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T08:18:30.192 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/779349ca-7593-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T08:18:30.192 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/779349ca-7593-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T08:18:30.199 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5 2024-09-18T08:18:30.225 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:18:30.226 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:18:30.227 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5' 2024-09-18T08:18:30.327 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1 2024-09-18T08:18:30.351 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:18:30.351 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:18:30.352 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1' 2024-09-18T08:18:30.453 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6 2024-09-18T08:18:30.480 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:18:30.481 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:18:30.482 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6' 2024-09-18T08:18:30.582 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6 2024-09-18T08:18:30.610 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:18:30.611 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:18:30.611 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6' 2024-09-18T08:18:30.712 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2 2024-09-18T08:18:30.734 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:18:30.735 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:18:30.735 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2' 2024-09-18T08:18:30.836 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1 2024-09-18T08:18:30.858 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:18:30.859 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:18:30.859 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1' 2024-09-18T08:18:30.905 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:18:30 smithi050 ceph-mon[32371]: pgmap v790: 158 pgs: 4 creating+peering, 4 peering, 150 active+clean; 583 KiB data, 3.8 GiB used, 622 GiB / 626 GiB avail; 511 B/s rd, 0 op/s; 81 KiB/s, 16 objects/s recovering 2024-09-18T08:18:30.905 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:18:30 smithi050 ceph-mon[32371]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-18T08:18:30.961 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7 2024-09-18T08:18:30.966 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:18:30 smithi007 ceph-mon[39249]: pgmap v790: 158 pgs: 4 creating+peering, 4 peering, 150 active+clean; 583 KiB data, 3.8 GiB used, 622 GiB / 626 GiB avail; 511 B/s rd, 0 op/s; 81 KiB/s, 16 objects/s recovering 2024-09-18T08:18:30.966 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:18:30 smithi007 ceph-mon[39249]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-18T08:18:30.967 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:18:30 smithi007 ceph-mon[29459]: pgmap v790: 158 pgs: 4 creating+peering, 4 peering, 150 active+clean; 583 KiB data, 3.8 GiB used, 622 GiB / 626 GiB avail; 511 B/s rd, 0 op/s; 81 KiB/s, 16 objects/s recovering 2024-09-18T08:18:30.967 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:18:30 smithi007 ceph-mon[29459]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-18T08:18:30.968 INFO:teuthology.orchestra.run.smithi007.stdout:ERROR: (22) Invalid argument 2024-09-18T08:18:30.968 INFO:teuthology.orchestra.run.smithi007.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-18T08:18:30.988 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:18:30.988 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:18:30.989 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7' 2024-09-18T08:18:31.090 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4 2024-09-18T08:18:31.115 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:18:31.116 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:18:31.117 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4' 2024-09-18T08:18:31.217 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7 2024-09-18T08:18:31.242 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:18:31.243 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:18:31.243 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7' 2024-09-18T08:18:31.344 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3 2024-09-18T08:18:31.368 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:18:31.368 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:18:31.369 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3' 2024-09-18T08:18:31.470 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6 2024-09-18T08:18:31.495 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:18:31.496 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:18:31.496 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6' 2024-09-18T08:18:31.546 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid 779349ca-7593-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-18T08:18:31.597 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1 2024-09-18T08:18:31.620 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:18:31.620 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:18:31.621 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1' 2024-09-18T08:18:31.709 DEBUG:teuthology.orchestra.run.smithi050:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid 779349ca-7593-11ef-bceb-c7b262605968 -- ceph daemon osd.4 dump_ops_in_flight 2024-09-18T08:18:31.722 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5 2024-09-18T08:18:31.777 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:18:31.777 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:18:31.778 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5' 2024-09-18T08:18:31.848 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:18:31 smithi007 ceph-mon[39249]: 6.15 scrub starts 2024-09-18T08:18:31.848 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:18:31 smithi007 ceph-mon[39249]: 6.15 scrub ok 2024-09-18T08:18:31.848 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:18:31 smithi007 ceph-mon[39249]: 3.1b scrub starts 2024-09-18T08:18:31.848 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:18:31 smithi007 ceph-mon[39249]: 3.1b scrub ok 2024-09-18T08:18:31.848 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:18:31 smithi007 ceph-mon[29459]: 6.15 scrub starts 2024-09-18T08:18:31.848 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:18:31 smithi007 ceph-mon[29459]: 6.15 scrub ok 2024-09-18T08:18:31.848 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:18:31 smithi007 ceph-mon[29459]: 3.1b scrub starts 2024-09-18T08:18:31.848 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:18:31 smithi007 ceph-mon[29459]: 3.1b scrub ok 2024-09-18T08:18:31.878 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4 2024-09-18T08:18:31.900 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:18:31.901 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:18:31.901 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4' 2024-09-18T08:18:31.905 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:18:31 smithi050 ceph-mon[32371]: 6.15 scrub starts 2024-09-18T08:18:31.905 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:18:31 smithi050 ceph-mon[32371]: 6.15 scrub ok 2024-09-18T08:18:31.905 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:18:31 smithi050 ceph-mon[32371]: 3.1b scrub starts 2024-09-18T08:18:31.905 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:18:31 smithi050 ceph-mon[32371]: 3.1b scrub ok 2024-09-18T08:18:31.977 INFO:teuthology.orchestra.run.smithi050.stderr:Inferring config /var/lib/ceph/779349ca-7593-11ef-bceb-c7b262605968/mon.b/config 2024-09-18T08:18:32.006 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5 2024-09-18T08:18:32.029 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:18:32.029 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:18:32.030 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5' 2024-09-18T08:18:32.130 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3 2024-09-18T08:18:32.153 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:18:32.153 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:18:32.154 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3' 2024-09-18T08:18:32.254 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4 2024-09-18T08:18:32.304 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:18:32.304 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:18:32.304 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4' 2024-09-18T08:18:32.384 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/779349ca-7593-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T08:18:32.405 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5 2024-09-18T08:18:32.430 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:18:32.431 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:18:32.431 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5' 2024-09-18T08:18:32.486 INFO:teuthology.orchestra.run.smithi050.stdout:ERROR: (22) Invalid argument 2024-09-18T08:18:32.486 INFO:teuthology.orchestra.run.smithi050.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-18T08:18:32.533 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6 2024-09-18T08:18:32.556 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:18:32.556 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:18:32.557 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6' 2024-09-18T08:18:32.638 INFO:teuthology.orchestra.run.smithi007.stderr:enabled application 'rados' on pool 'unique_pool_4' 2024-09-18T08:18:32.642 INFO:teuthology.orchestra.run.smithi007.stderr:noscrub is set 2024-09-18T08:18:32.658 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5 2024-09-18T08:18:32.681 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:18:32.681 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:18:32.682 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5' 2024-09-18T08:18:32.782 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3 2024-09-18T08:18:32.805 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:18:32.806 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:18:32.806 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3' 2024-09-18T08:18:32.890 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:18:32 smithi007 ceph-mon[39249]: 4.5 scrub starts 2024-09-18T08:18:32.890 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:18:32 smithi007 ceph-mon[39249]: 4.5 scrub ok 2024-09-18T08:18:32.891 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:18:32 smithi007 ceph-mon[39249]: 6.14 scrub starts 2024-09-18T08:18:32.891 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:18:32 smithi007 ceph-mon[39249]: 6.14 scrub ok 2024-09-18T08:18:32.891 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:18:32 smithi007 ceph-mon[39249]: pgmap v791: 158 pgs: 4 creating+peering, 4 peering, 150 active+clean; 583 KiB data, 3.8 GiB used, 622 GiB / 626 GiB avail; 423 B/s rd, 0 op/s; 67 KiB/s, 13 objects/s recovering 2024-09-18T08:18:32.891 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:18:32 smithi007 ceph-mon[39249]: 4.12 scrub starts 2024-09-18T08:18:32.891 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:18:32 smithi007 ceph-mon[39249]: 4.12 scrub ok 2024-09-18T08:18:32.891 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:18:32 smithi007 ceph-mon[39249]: 4.1a scrub starts 2024-09-18T08:18:32.891 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:18:32 smithi007 ceph-mon[39249]: 4.1a scrub ok 2024-09-18T08:18:32.891 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:18:32 smithi007 ceph-mon[39249]: from='client.? 172.21.15.7:0/2841720426' entity='client.admin' cmd={"prefix": "osd pool application enable", "pool": "unique_pool_4", "app": "rados", "yes_i_really_mean_it": true} : dispatch 2024-09-18T08:18:32.891 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:18:32 smithi007 ceph-mon[39249]: from='client.? 172.21.15.7:0/1186291158' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-18T08:18:32.891 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:18:32 smithi007 ceph-mon[39249]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-18T08:18:32.892 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:18:32 smithi007 ceph-mon[29459]: 4.5 scrub starts 2024-09-18T08:18:32.892 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:18:32 smithi007 ceph-mon[29459]: 4.5 scrub ok 2024-09-18T08:18:32.892 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:18:32 smithi007 ceph-mon[29459]: 6.14 scrub starts 2024-09-18T08:18:32.892 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:18:32 smithi007 ceph-mon[29459]: 6.14 scrub ok 2024-09-18T08:18:32.892 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:18:32 smithi007 ceph-mon[29459]: pgmap v791: 158 pgs: 4 creating+peering, 4 peering, 150 active+clean; 583 KiB data, 3.8 GiB used, 622 GiB / 626 GiB avail; 423 B/s rd, 0 op/s; 67 KiB/s, 13 objects/s recovering 2024-09-18T08:18:32.892 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:18:32 smithi007 ceph-mon[29459]: 4.12 scrub starts 2024-09-18T08:18:32.892 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:18:32 smithi007 ceph-mon[29459]: 4.12 scrub ok 2024-09-18T08:18:32.892 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:18:32 smithi007 ceph-mon[29459]: 4.1a scrub starts 2024-09-18T08:18:32.892 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:18:32 smithi007 ceph-mon[29459]: 4.1a scrub ok 2024-09-18T08:18:32.892 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:18:32 smithi007 ceph-mon[29459]: from='client.? 172.21.15.7:0/2841720426' entity='client.admin' cmd={"prefix": "osd pool application enable", "pool": "unique_pool_4", "app": "rados", "yes_i_really_mean_it": true} : dispatch 2024-09-18T08:18:32.892 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:18:32 smithi007 ceph-mon[29459]: from='client.? 172.21.15.7:0/1186291158' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-18T08:18:32.892 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:18:32 smithi007 ceph-mon[29459]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-18T08:18:32.905 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:18:32 smithi050 ceph-mon[32371]: 4.5 scrub starts 2024-09-18T08:18:32.905 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:18:32 smithi050 ceph-mon[32371]: 4.5 scrub ok 2024-09-18T08:18:32.905 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:18:32 smithi050 ceph-mon[32371]: 6.14 scrub starts 2024-09-18T08:18:32.905 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:18:32 smithi050 ceph-mon[32371]: 6.14 scrub ok 2024-09-18T08:18:32.905 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:18:32 smithi050 ceph-mon[32371]: pgmap v791: 158 pgs: 4 creating+peering, 4 peering, 150 active+clean; 583 KiB data, 3.8 GiB used, 622 GiB / 626 GiB avail; 423 B/s rd, 0 op/s; 67 KiB/s, 13 objects/s recovering 2024-09-18T08:18:32.905 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:18:32 smithi050 ceph-mon[32371]: 4.12 scrub starts 2024-09-18T08:18:32.905 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:18:32 smithi050 ceph-mon[32371]: 4.12 scrub ok 2024-09-18T08:18:32.905 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:18:32 smithi050 ceph-mon[32371]: 4.1a scrub starts 2024-09-18T08:18:32.905 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:18:32 smithi050 ceph-mon[32371]: 4.1a scrub ok 2024-09-18T08:18:32.906 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:18:32 smithi050 ceph-mon[32371]: from='client.? 172.21.15.7:0/2841720426' entity='client.admin' cmd={"prefix": "osd pool application enable", "pool": "unique_pool_4", "app": "rados", "yes_i_really_mean_it": true} : dispatch 2024-09-18T08:18:32.906 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:18:32 smithi050 ceph-mon[32371]: from='client.? 172.21.15.7:0/1186291158' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-18T08:18:32.906 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:18:32 smithi050 ceph-mon[32371]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-18T08:18:32.907 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6 2024-09-18T08:18:32.929 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:18:32.930 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:18:32.930 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6' 2024-09-18T08:18:33.031 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7 2024-09-18T08:18:33.053 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:18:33.054 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:18:33.054 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7' 2024-09-18T08:18:33.155 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1 2024-09-18T08:18:33.178 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:18:33.178 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:18:33.179 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1' 2024-09-18T08:18:33.252 DEBUG:teuthology.orchestra.run.smithi050:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid 779349ca-7593-11ef-bceb-c7b262605968 -- ceph daemon osd.4 dump_blocked_ops 2024-09-18T08:18:33.280 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3 2024-09-18T08:18:33.302 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:18:33.303 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:18:33.303 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3' 2024-09-18T08:18:33.336 INFO:teuthology.orchestra.run.smithi007.stdout:osd.0: {} 2024-09-18T08:18:33.336 INFO:teuthology.orchestra.run.smithi007.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-18T08:18:33.353 INFO:teuthology.orchestra.run.smithi007.stdout:osd.1: {} 2024-09-18T08:18:33.353 INFO:teuthology.orchestra.run.smithi007.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-18T08:18:33.369 INFO:teuthology.orchestra.run.smithi007.stdout:osd.2: {} 2024-09-18T08:18:33.370 INFO:teuthology.orchestra.run.smithi007.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-18T08:18:33.386 INFO:teuthology.orchestra.run.smithi007.stdout:osd.3: {} 2024-09-18T08:18:33.386 INFO:teuthology.orchestra.run.smithi007.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-18T08:18:33.402 INFO:teuthology.orchestra.run.smithi007.stdout:osd.4: {} 2024-09-18T08:18:33.403 INFO:teuthology.orchestra.run.smithi007.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-18T08:18:33.404 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7 2024-09-18T08:18:33.421 INFO:teuthology.orchestra.run.smithi007.stdout:osd.5: {} 2024-09-18T08:18:33.421 INFO:teuthology.orchestra.run.smithi007.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-18T08:18:33.426 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:18:33.426 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:18:33.427 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7' 2024-09-18T08:18:33.437 INFO:teuthology.orchestra.run.smithi007.stdout:osd.6: {} 2024-09-18T08:18:33.437 INFO:teuthology.orchestra.run.smithi007.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-18T08:18:33.456 INFO:teuthology.orchestra.run.smithi007.stdout:osd.7: {} 2024-09-18T08:18:33.456 INFO:teuthology.orchestra.run.smithi007.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-18T08:18:33.510 INFO:teuthology.orchestra.run.smithi050.stderr:Inferring config /var/lib/ceph/779349ca-7593-11ef-bceb-c7b262605968/mon.b/config 2024-09-18T08:18:33.527 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7 2024-09-18T08:18:33.550 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:18:33.550 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:18:33.551 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7' 2024-09-18T08:18:33.651 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0 2024-09-18T08:18:33.673 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:18:33.674 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:18:33.674 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0' 2024-09-18T08:18:33.775 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2 2024-09-18T08:18:33.797 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:18:33.798 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:18:33.798 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2' 2024-09-18T08:18:33.899 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4 2024-09-18T08:18:33.900 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:18:33 smithi007 ceph-mon[29459]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-18T08:18:33.900 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:18:33 smithi007 ceph-mon[29459]: from='client.? 172.21.15.7:0/2841720426' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "unique_pool_4", "app": "rados", "yes_i_really_mean_it": true}]': finished 2024-09-18T08:18:33.900 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:18:33 smithi007 ceph-mon[29459]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-18T08:18:33.900 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:18:33 smithi007 ceph-mon[29459]: osdmap e405: 8 total, 8 up, 7 in 2024-09-18T08:18:33.901 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:18:33 smithi007 ceph-mon[29459]: from='client.? 172.21.15.7:0/4104372398' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-18T08:18:33.901 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:18:33 smithi007 ceph-mon[39249]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-18T08:18:33.901 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:18:33 smithi007 ceph-mon[39249]: from='client.? 172.21.15.7:0/2841720426' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "unique_pool_4", "app": "rados", "yes_i_really_mean_it": true}]': finished 2024-09-18T08:18:33.901 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:18:33 smithi007 ceph-mon[39249]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-18T08:18:33.901 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:18:33 smithi007 ceph-mon[39249]: osdmap e405: 8 total, 8 up, 7 in 2024-09-18T08:18:33.901 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:18:33 smithi007 ceph-mon[39249]: from='client.? 172.21.15.7:0/4104372398' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-18T08:18:33.905 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:18:33 smithi050 ceph-mon[32371]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-18T08:18:33.905 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:18:33 smithi050 ceph-mon[32371]: from='client.? 172.21.15.7:0/2841720426' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "unique_pool_4", "app": "rados", "yes_i_really_mean_it": true}]': finished 2024-09-18T08:18:33.905 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:18:33 smithi050 ceph-mon[32371]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-18T08:18:33.905 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:18:33 smithi050 ceph-mon[32371]: osdmap e405: 8 total, 8 up, 7 in 2024-09-18T08:18:33.905 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:18:33 smithi050 ceph-mon[32371]: from='client.? 172.21.15.7:0/4104372398' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-18T08:18:33.923 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:18:33.924 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:18:33.925 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4' 2024-09-18T08:18:34.025 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7 2024-09-18T08:18:34.034 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2024-09-18T08:18:34.035 INFO:teuthology.orchestra.run.smithi050.stdout: "ops": [], 2024-09-18T08:18:34.035 INFO:teuthology.orchestra.run.smithi050.stdout: "complaint_time": 30, 2024-09-18T08:18:34.035 INFO:teuthology.orchestra.run.smithi050.stdout: "num_blocked_ops": 0 2024-09-18T08:18:34.035 INFO:teuthology.orchestra.run.smithi050.stdout:} 2024-09-18T08:18:34.048 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:18:34.049 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:18:34.049 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7' 2024-09-18T08:18:34.150 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1 2024-09-18T08:18:34.172 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:18:34.173 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:18:34.173 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1' 2024-09-18T08:18:34.274 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1 2024-09-18T08:18:34.296 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:18:34.297 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:18:34.297 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1' 2024-09-18T08:18:34.398 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7 2024-09-18T08:18:34.422 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:18:34.422 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:18:34.422 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7' 2024-09-18T08:18:34.523 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6 2024-09-18T08:18:34.546 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:18:34.547 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:18:34.547 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6' 2024-09-18T08:18:34.648 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0 2024-09-18T08:18:34.672 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:18:34.673 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:18:34.673 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0' 2024-09-18T08:18:34.726 DEBUG:teuthology.orchestra.run.smithi050:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid 779349ca-7593-11ef-bceb-c7b262605968 -- ceph daemon osd.4 dump_historic_ops 2024-09-18T08:18:34.774 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4 2024-09-18T08:18:34.796 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:18:34.796 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:18:34.797 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4' 2024-09-18T08:18:34.898 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4 2024-09-18T08:18:34.902 DEBUG:teuthology.orchestra.run.smithi007:> /bin/sh -c 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados --no-log-to-stderr --name client.0 -b 65536 --object-size 65536 -p unique_pool_4 bench 90 write' 2024-09-18T08:18:34.905 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:18:34 smithi050 ceph-mon[32371]: pgmap v793: 158 pgs: 158 active+clean; 583 KiB data, 3.8 GiB used, 622 GiB / 626 GiB avail; 878 B/s rd, 0 op/s; 55 KiB/s, 14 objects/s recovering 2024-09-18T08:18:34.905 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:18:34 smithi050 ceph-mon[32371]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-09-18T08:18:34.905 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:18:34 smithi050 ceph-mon[32371]: from='mgr.24470 172.21.15.7:0/2594390261' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-18T08:18:34.905 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:18:34 smithi050 ceph-mon[32371]: osdmap e406: 8 total, 8 up, 7 in 2024-09-18T08:18:34.906 INFO:tasks.radosbench:joining radosbench (timing out after 3000s) 2024-09-18T08:18:34.906 INFO:teuthology.orchestra.run:waiting for 3000 2024-09-18T08:18:34.921 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:18:34.922 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:18:34.922 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4' 2024-09-18T08:18:34.928 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:18:34 smithi007 ceph-mon[29459]: pgmap v793: 158 pgs: 158 active+clean; 583 KiB data, 3.8 GiB used, 622 GiB / 626 GiB avail; 878 B/s rd, 0 op/s; 55 KiB/s, 14 objects/s recovering 2024-09-18T08:18:34.928 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:18:34 smithi007 ceph-mon[29459]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-09-18T08:18:34.928 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:18:34 smithi007 ceph-mon[29459]: from='mgr.24470 172.21.15.7:0/2594390261' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-18T08:18:34.928 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:18:34 smithi007 ceph-mon[29459]: osdmap e406: 8 total, 8 up, 7 in 2024-09-18T08:18:34.929 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:18:34 smithi007 ceph-mon[39249]: pgmap v793: 158 pgs: 158 active+clean; 583 KiB data, 3.8 GiB used, 622 GiB / 626 GiB avail; 878 B/s rd, 0 op/s; 55 KiB/s, 14 objects/s recovering 2024-09-18T08:18:34.929 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:18:34 smithi007 ceph-mon[39249]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-09-18T08:18:34.929 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:18:34 smithi007 ceph-mon[39249]: from='mgr.24470 172.21.15.7:0/2594390261' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-18T08:18:34.929 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:18:34 smithi007 ceph-mon[39249]: osdmap e406: 8 total, 8 up, 7 in 2024-09-18T08:18:34.982 INFO:teuthology.orchestra.run.smithi050.stderr:Inferring config /var/lib/ceph/779349ca-7593-11ef-bceb-c7b262605968/mon.b/config 2024-09-18T08:18:35.023 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0 2024-09-18T08:18:35.029 INFO:tasks.radosbench.radosbench.0.smithi007.stdout:hints = 1 2024-09-18T08:18:35.029 INFO:tasks.radosbench.radosbench.0.smithi007.stdout:Maintaining 16 concurrent writes of 65536 bytes to objects of size 65536 for up to 90 seconds or 0 objects 2024-09-18T08:18:35.029 INFO:tasks.radosbench.radosbench.0.smithi007.stdout:Object prefix: benchmark_data_smithi007_172546 2024-09-18T08:18:35.030 INFO:tasks.radosbench.radosbench.0.smithi007.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-09-18T08:18:35.030 INFO:tasks.radosbench.radosbench.0.smithi007.stdout: 0 0 0 0 0 0 - 0 2024-09-18T08:18:35.065 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:18:35.066 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:18:35.066 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0' 2024-09-18T08:18:35.123 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 6, 5, 7] out_osds: [4] dead_osds: [] live_osds: [0, 2, 3, 4, 5, 6, 7, 1] 2024-09-18T08:18:35.123 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-18T08:18:35.123 INFO:tasks.thrashosds.thrasher:thrash_pg_upmap_items 2024-09-18T08:18:35.124 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid 779349ca-7593-11ef-bceb-c7b262605968 -- ceph osd dump -f json-pretty 2024-09-18T08:18:35.168 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7 2024-09-18T08:18:35.198 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:18:35.198 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:18:35.199 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7' 2024-09-18T08:18:35.300 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2 2024-09-18T08:18:35.340 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:18:35.341 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:18:35.341 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2' 2024-09-18T08:18:35.442 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5 2024-09-18T08:18:35.477 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:18:35.477 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:18:35.477 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5' 2024-09-18T08:18:35.534 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2024-09-18T08:18:35.535 INFO:teuthology.orchestra.run.smithi050.stdout: "size": 20, 2024-09-18T08:18:35.535 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 600, 2024-09-18T08:18:35.535 INFO:teuthology.orchestra.run.smithi050.stdout: "ops": [ 2024-09-18T08:18:35.535 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:18:35.535 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.15216.0:86225 8.5 8:ae8df458:::benchmark_data_smithi007_101583_object86224:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e205)", 2024-09-18T08:18:35.535 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-09-18T08:09:12.178973+0000", 2024-09-18T08:18:35.535 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 563.35311794699999, 2024-09-18T08:18:35.535 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.99102768600000002, 2024-09-18T08:18:35.535 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-09-18T08:18:35.535 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-09-18T08:18:35.535 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "reached pg", 2024-09-18T08:18:35.535 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2024-09-18T08:18:35.536 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.15216", 2024-09-18T08:18:35.536 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "172.21.15.7:0/2596041177", 2024-09-18T08:18:35.536 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 86225 2024-09-18T08:18:35.536 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:18:35.536 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-09-18T08:18:35.536 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:18:35.536 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-09-18T08:18:35.536 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:12.178973+0000", 2024-09-18T08:18:35.536 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-18T08:18:35.536 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:18:35.536 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:18:35.536 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-09-18T08:18:35.536 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:12.178973+0000", 2024-09-18T08:18:35.537 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-18T08:18:35.537 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:18:35.537 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:18:35.537 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-09-18T08:18:35.537 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:12.178975+0000", 2024-09-18T08:18:35.537 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.3969999999999999e-06 2024-09-18T08:18:35.537 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:18:35.537 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:18:35.537 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-09-18T08:18:35.537 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:12.179022+0000", 2024-09-18T08:18:35.537 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4.7580000000000002e-05 2024-09-18T08:18:35.537 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:18:35.537 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:18:35.538 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-09-18T08:18:35.538 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:12.179034+0000", 2024-09-18T08:18:35.538 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.1477e-05 2024-09-18T08:18:35.538 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:18:35.538 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:18:35.538 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-09-18T08:18:35.538 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:12.179040+0000", 2024-09-18T08:18:35.538 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 5.7529999999999998e-06 2024-09-18T08:18:35.538 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:18:35.538 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:18:35.538 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-18T08:18:35.538 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:12.179092+0000", 2024-09-18T08:18:35.538 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 5.2840000000000002e-05 2024-09-18T08:18:35.538 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:18:35.539 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:18:35.539 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "waiting for peered", 2024-09-18T08:18:35.539 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:12.179102+0000", 2024-09-18T08:18:35.539 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 9.5210000000000008e-06 2024-09-18T08:18:35.539 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:18:35.539 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:18:35.539 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-18T08:18:35.539 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:13.169945+0000", 2024-09-18T08:18:35.539 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.99084323600000002 2024-09-18T08:18:35.539 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:18:35.539 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:18:35.539 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-09-18T08:18:35.539 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:13.170001+0000", 2024-09-18T08:18:35.540 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 5.5881999999999998e-05 2024-09-18T08:18:35.540 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-18T08:18:35.540 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-09-18T08:18:35.540 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-18T08:18:35.540 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:18:35.540 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:18:35.540 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.15216.0:86263 8.5 8:af593c7a:::benchmark_data_smithi007_101583_object86262:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e205)", 2024-09-18T08:18:35.540 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-09-18T08:09:12.204249+0000", 2024-09-18T08:18:35.540 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 563.32784194199996, 2024-09-18T08:18:35.540 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.98012944999999996, 2024-09-18T08:18:35.541 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-09-18T08:18:35.541 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-09-18T08:18:35.541 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T08:18:35.541 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2024-09-18T08:18:35.541 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.15216", 2024-09-18T08:18:35.541 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "172.21.15.7:0/2596041177", 2024-09-18T08:18:35.541 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 86263 2024-09-18T08:18:35.541 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:18:35.541 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-09-18T08:18:35.541 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:18:35.541 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-09-18T08:18:35.541 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:12.204249+0000", 2024-09-18T08:18:35.542 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-18T08:18:35.542 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:18:35.542 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:18:35.542 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-09-18T08:18:35.542 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:12.204249+0000", 2024-09-18T08:18:35.542 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-18T08:18:35.542 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:18:35.542 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:18:35.542 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-09-18T08:18:35.542 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:12.204251+0000", 2024-09-18T08:18:35.542 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.635e-06 2024-09-18T08:18:35.542 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:18:35.542 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:18:35.543 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-09-18T08:18:35.543 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:12.204299+0000", 2024-09-18T08:18:35.543 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4.8026999999999999e-05 2024-09-18T08:18:35.543 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:18:35.543 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:18:35.543 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-09-18T08:18:35.543 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:12.204313+0000", 2024-09-18T08:18:35.543 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.4066000000000001e-05 2024-09-18T08:18:35.543 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:18:35.544 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:18:35.544 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-09-18T08:18:35.544 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:12.204321+0000", 2024-09-18T08:18:35.544 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 8.1839999999999999e-06 2024-09-18T08:18:35.544 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:18:35.544 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:18:35.544 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-18T08:18:35.544 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:12.204392+0000", 2024-09-18T08:18:35.544 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 7.0340999999999996e-05 2024-09-18T08:18:35.544 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:18:35.544 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:18:35.544 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "waiting for peered", 2024-09-18T08:18:35.544 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:12.204403+0000", 2024-09-18T08:18:35.544 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.1654000000000001e-05 2024-09-18T08:18:35.545 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:18:35.545 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:18:35.545 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-18T08:18:35.545 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:13.170414+0000", 2024-09-18T08:18:35.545 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.96601117000000003 2024-09-18T08:18:35.545 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:18:35.545 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:18:35.545 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-09-18T08:18:35.545 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:13.170546+0000", 2024-09-18T08:18:35.545 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00013120100000000001 2024-09-18T08:18:35.545 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:18:35.545 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:18:35.545 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "waiting for subops from 6,7", 2024-09-18T08:18:35.546 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:13.170965+0000", 2024-09-18T08:18:35.546 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00041976200000000001 2024-09-18T08:18:35.546 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:18:35.546 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:18:35.546 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:18:35.546 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:13.181803+0000", 2024-09-18T08:18:35.546 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.010837167999999999 2024-09-18T08:18:35.546 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:18:35.546 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:18:35.546 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:18:35.546 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:13.181864+0000", 2024-09-18T08:18:35.546 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 6.1857000000000001e-05 2024-09-18T08:18:35.546 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:18:35.547 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:18:35.547 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "op_commit", 2024-09-18T08:18:35.547 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:13.184283+0000", 2024-09-18T08:18:35.547 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0024184340000000001 2024-09-18T08:18:35.547 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:18:35.547 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:18:35.547 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "commit_sent", 2024-09-18T08:18:35.547 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:13.184339+0000", 2024-09-18T08:18:35.547 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 5.6378999999999997e-05 2024-09-18T08:18:35.547 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:18:35.547 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:18:35.547 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-09-18T08:18:35.548 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:13.184379+0000", 2024-09-18T08:18:35.548 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 3.9572e-05 2024-09-18T08:18:35.548 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-18T08:18:35.548 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-09-18T08:18:35.548 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-18T08:18:35.548 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:18:35.548 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:18:35.548 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.15216.0:86267 8.5 8:a135397b:::benchmark_data_smithi007_101583_object86266:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e205)", 2024-09-18T08:18:35.548 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-09-18T08:09:12.207725+0000", 2024-09-18T08:18:35.548 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 563.32436589199995, 2024-09-18T08:18:35.548 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.97676783599999994, 2024-09-18T08:18:35.548 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-09-18T08:18:35.548 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-09-18T08:18:35.549 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T08:18:35.549 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2024-09-18T08:18:35.549 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.15216", 2024-09-18T08:18:35.549 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "172.21.15.7:0/2596041177", 2024-09-18T08:18:35.549 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 86267 2024-09-18T08:18:35.549 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:18:35.549 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-09-18T08:18:35.549 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:18:35.549 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-09-18T08:18:35.549 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:12.207725+0000", 2024-09-18T08:18:35.549 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-18T08:18:35.550 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:18:35.550 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:18:35.551 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-09-18T08:18:35.551 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:12.207725+0000", 2024-09-18T08:18:35.551 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-18T08:18:35.551 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:18:35.551 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:18:35.551 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-09-18T08:18:35.551 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:12.207727+0000", 2024-09-18T08:18:35.551 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.79e-06 2024-09-18T08:18:35.551 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:18:35.551 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:18:35.551 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-09-18T08:18:35.551 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:12.207765+0000", 2024-09-18T08:18:35.552 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 3.7864000000000002e-05 2024-09-18T08:18:35.552 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:18:35.552 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:18:35.552 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-09-18T08:18:35.552 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:12.207783+0000", 2024-09-18T08:18:35.552 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.7779999999999999e-05 2024-09-18T08:18:35.552 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:18:35.552 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:18:35.552 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-09-18T08:18:35.552 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:12.207791+0000", 2024-09-18T08:18:35.552 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 7.9729999999999997e-06 2024-09-18T08:18:35.553 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:18:35.553 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:18:35.553 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-18T08:18:35.553 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:12.207872+0000", 2024-09-18T08:18:35.553 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 8.1318999999999998e-05 2024-09-18T08:18:35.553 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:18:35.553 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:18:35.553 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "waiting for peered", 2024-09-18T08:18:35.553 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:12.207905+0000", 2024-09-18T08:18:35.554 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 3.2694000000000002e-05 2024-09-18T08:18:35.554 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:18:35.554 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:18:35.554 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-18T08:18:35.554 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:13.179332+0000", 2024-09-18T08:18:35.554 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.97142748800000001 2024-09-18T08:18:35.554 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:18:35.554 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:18:35.554 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-09-18T08:18:35.554 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:13.179470+0000", 2024-09-18T08:18:35.554 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00013747400000000001 2024-09-18T08:18:35.554 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:18:35.554 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:18:35.555 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "waiting for subops from 6,7", 2024-09-18T08:18:35.555 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:13.179634+0000", 2024-09-18T08:18:35.555 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00016417600000000001 2024-09-18T08:18:35.555 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:18:35.555 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:18:35.555 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:18:35.555 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:13.184267+0000", 2024-09-18T08:18:35.555 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0046332079999999998 2024-09-18T08:18:35.555 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:18:35.555 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:18:35.555 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "op_commit", 2024-09-18T08:18:35.555 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:13.184386+0000", 2024-09-18T08:18:35.556 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000118993 2024-09-18T08:18:35.556 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:18:35.556 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:18:35.556 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:18:35.556 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:13.184437+0000", 2024-09-18T08:18:35.556 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 5.0988e-05 2024-09-18T08:18:35.556 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:18:35.556 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:18:35.556 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "commit_sent", 2024-09-18T08:18:35.556 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:13.184469+0000", 2024-09-18T08:18:35.556 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 3.1603000000000002e-05 2024-09-18T08:18:35.556 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:18:35.556 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:18:35.557 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-09-18T08:18:35.557 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:13.184493+0000", 2024-09-18T08:18:35.557 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 2.4485999999999999e-05 2024-09-18T08:18:35.557 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-18T08:18:35.557 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-09-18T08:18:35.557 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-18T08:18:35.557 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:18:35.557 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:18:35.557 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.15216.0:86268 8.5 8:aa7096fb:::benchmark_data_smithi007_101583_object86267:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e205)", 2024-09-18T08:18:35.557 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-09-18T08:09:12.208946+0000", 2024-09-18T08:18:35.557 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 563.32314528500001, 2024-09-18T08:18:35.557 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.980827699, 2024-09-18T08:18:35.557 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-09-18T08:18:35.558 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-09-18T08:18:35.558 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T08:18:35.558 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2024-09-18T08:18:35.558 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.15216", 2024-09-18T08:18:35.558 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "172.21.15.7:0/2596041177", 2024-09-18T08:18:35.558 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 86268 2024-09-18T08:18:35.558 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:18:35.558 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-09-18T08:18:35.558 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:18:35.558 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-09-18T08:18:35.558 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:12.208946+0000", 2024-09-18T08:18:35.558 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-18T08:18:35.559 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:18:35.559 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:18:35.559 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-09-18T08:18:35.559 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:12.208946+0000", 2024-09-18T08:18:35.559 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-18T08:18:35.559 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:18:35.559 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:18:35.559 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-09-18T08:18:35.559 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:12.208947+0000", 2024-09-18T08:18:35.559 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.201e-06 2024-09-18T08:18:35.559 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:18:35.560 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:18:35.560 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-09-18T08:18:35.560 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:12.209036+0000", 2024-09-18T08:18:35.560 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 8.8865999999999999e-05 2024-09-18T08:18:35.560 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:18:35.560 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:18:35.560 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-09-18T08:18:35.560 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:12.209049+0000", 2024-09-18T08:18:35.560 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.2795e-05 2024-09-18T08:18:35.560 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:18:35.560 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:18:35.560 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-09-18T08:18:35.561 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:12.209056+0000", 2024-09-18T08:18:35.561 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 7.1160000000000001e-06 2024-09-18T08:18:35.561 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:18:35.561 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:18:35.561 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-18T08:18:35.561 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:12.209116+0000", 2024-09-18T08:18:35.561 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 5.9517e-05 2024-09-18T08:18:35.561 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:18:35.561 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:18:35.561 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "waiting for peered", 2024-09-18T08:18:35.561 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:12.209125+0000", 2024-09-18T08:18:35.561 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 9.6630000000000005e-06 2024-09-18T08:18:35.561 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:18:35.561 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:18:35.562 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-18T08:18:35.562 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:13.182004+0000", 2024-09-18T08:18:35.562 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.97287885799999996 2024-09-18T08:18:35.562 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:18:35.562 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:18:35.562 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-09-18T08:18:35.562 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:13.182091+0000", 2024-09-18T08:18:35.562 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 8.6907e-05 2024-09-18T08:18:35.562 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:18:35.562 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:18:35.563 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "waiting for subops from 6,7", 2024-09-18T08:18:35.563 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:13.182183+0000", 2024-09-18T08:18:35.563 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 9.2131000000000003e-05 2024-09-18T08:18:35.563 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:18:35.563 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:18:35.563 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "op_commit", 2024-09-18T08:18:35.563 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:13.187041+0000", 2024-09-18T08:18:35.563 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0048579210000000003 2024-09-18T08:18:35.563 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:18:35.563 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:18:35.563 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:18:35.564 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:13.189581+0000", 2024-09-18T08:18:35.564 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.002539707 2024-09-18T08:18:35.564 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:18:35.564 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:18:35.564 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:18:35.564 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:13.189646+0000", 2024-09-18T08:18:35.564 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 6.5699999999999998e-05 2024-09-18T08:18:35.564 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:18:35.564 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:18:35.564 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "commit_sent", 2024-09-18T08:18:35.564 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:13.189714+0000", 2024-09-18T08:18:35.564 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 6.7108999999999997e-05 2024-09-18T08:18:35.565 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:18:35.565 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:18:35.565 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-09-18T08:18:35.565 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:13.189774+0000", 2024-09-18T08:18:35.565 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 6.0207999999999997e-05 2024-09-18T08:18:35.565 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-18T08:18:35.565 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-09-18T08:18:35.565 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-18T08:18:35.565 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:18:35.565 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:18:35.565 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.15216.0:86303 8.5 8:aebc4459:::benchmark_data_smithi007_101583_object86302:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e205)", 2024-09-18T08:18:35.565 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-09-18T08:09:12.250052+0000", 2024-09-18T08:18:35.565 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 563.28203972899996, 2024-09-18T08:18:35.566 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.94992247699999999, 2024-09-18T08:18:35.566 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-09-18T08:18:35.566 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-09-18T08:18:35.566 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T08:18:35.566 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2024-09-18T08:18:35.566 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.15216", 2024-09-18T08:18:35.566 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "172.21.15.7:0/2596041177", 2024-09-18T08:18:35.566 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 86303 2024-09-18T08:18:35.566 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:18:35.566 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-09-18T08:18:35.566 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:18:35.566 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-09-18T08:18:35.566 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:12.250052+0000", 2024-09-18T08:18:35.566 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-18T08:18:35.567 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:18:35.567 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:18:35.567 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-09-18T08:18:35.567 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:12.250052+0000", 2024-09-18T08:18:35.567 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-18T08:18:35.567 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:18:35.567 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:18:35.568 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-09-18T08:18:35.568 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:12.250053+0000", 2024-09-18T08:18:35.568 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.6130000000000001e-06 2024-09-18T08:18:35.568 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:18:35.568 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:18:35.568 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-09-18T08:18:35.569 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:12.250090+0000", 2024-09-18T08:18:35.569 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 3.6307999999999999e-05 2024-09-18T08:18:35.569 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:18:35.569 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:18:35.569 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-09-18T08:18:35.569 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:12.250102+0000", 2024-09-18T08:18:35.570 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.2624e-05 2024-09-18T08:18:35.570 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:18:35.570 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:18:35.570 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-09-18T08:18:35.570 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:12.250109+0000", 2024-09-18T08:18:35.571 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 6.426e-06 2024-09-18T08:18:35.571 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:18:35.571 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:18:35.571 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-18T08:18:35.571 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:12.250173+0000", 2024-09-18T08:18:35.571 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 6.3966000000000004e-05 2024-09-18T08:18:35.571 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:18:35.572 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:18:35.572 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "waiting for peered", 2024-09-18T08:18:35.572 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:12.250184+0000", 2024-09-18T08:18:35.572 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.1401e-05 2024-09-18T08:18:35.572 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:18:35.572 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:18:35.573 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-18T08:18:35.573 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:13.184533+0000", 2024-09-18T08:18:35.573 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.93434866299999997 2024-09-18T08:18:35.573 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:18:35.573 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:18:35.573 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-09-18T08:18:35.574 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:13.184634+0000", 2024-09-18T08:18:35.574 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000101139 2024-09-18T08:18:35.574 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:18:35.574 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:18:35.575 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "waiting for subops from 6,7", 2024-09-18T08:18:35.575 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:13.184793+0000", 2024-09-18T08:18:35.575 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00015932000000000001 2024-09-18T08:18:35.575 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:18:35.575 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:18:35.575 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:18:35.576 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:13.194839+0000", 2024-09-18T08:18:35.576 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.010046144 2024-09-18T08:18:35.576 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:18:35.576 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:18:35.576 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:18:35.576 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:13.194928+0000", 2024-09-18T08:18:35.577 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 8.9037999999999997e-05 2024-09-18T08:18:35.577 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:18:35.577 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:18:35.577 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "op_commit", 2024-09-18T08:18:35.577 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:13.199865+0000", 2024-09-18T08:18:35.578 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0049365879999999996 2024-09-18T08:18:35.578 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:18:35.578 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:18:35.578 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "commit_sent", 2024-09-18T08:18:35.578 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:13.199924+0000", 2024-09-18T08:18:35.578 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 5.9383999999999999e-05 2024-09-18T08:18:35.579 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:18:35.579 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:18:35.579 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-09-18T08:18:35.579 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:13.199974+0000", 2024-09-18T08:18:35.579 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4.9863e-05 2024-09-18T08:18:35.579 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-18T08:18:35.580 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-09-18T08:18:35.580 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-18T08:18:35.580 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:18:35.580 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:18:35.580 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.15216.0:86344 8.5 8:ac157b17:::benchmark_data_smithi007_101583_object86343:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e205)", 2024-09-18T08:18:35.580 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-09-18T08:09:12.295743+0000", 2024-09-18T08:18:35.581 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 563.23634804899996, 2024-09-18T08:18:35.581 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.92374953400000004, 2024-09-18T08:18:35.581 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-09-18T08:18:35.581 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-09-18T08:18:35.581 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T08:18:35.582 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2024-09-18T08:18:35.582 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.15216", 2024-09-18T08:18:35.582 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "172.21.15.7:0/2596041177", 2024-09-18T08:18:35.582 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 86344 2024-09-18T08:18:35.582 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:18:35.582 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-09-18T08:18:35.583 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:18:35.583 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-09-18T08:18:35.583 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:12.295743+0000", 2024-09-18T08:18:35.583 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-18T08:18:35.583 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:18:35.583 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:18:35.584 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-09-18T08:18:35.584 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:12.295743+0000", 2024-09-18T08:18:35.584 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-18T08:18:35.584 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:18:35.584 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:18:35.585 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-09-18T08:18:35.585 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:12.295745+0000", 2024-09-18T08:18:35.585 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.57e-06 2024-09-18T08:18:35.585 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:18:35.585 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:18:35.585 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-09-18T08:18:35.586 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:12.295795+0000", 2024-09-18T08:18:35.586 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 5.0062999999999998e-05 2024-09-18T08:18:35.586 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:18:35.586 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:18:35.586 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-09-18T08:18:35.586 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:12.295807+0000", 2024-09-18T08:18:35.587 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.1578e-05 2024-09-18T08:18:35.587 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:18:35.587 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:18:35.587 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-09-18T08:18:35.587 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:12.295812+0000", 2024-09-18T08:18:35.587 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 5.8930000000000002e-06 2024-09-18T08:18:35.588 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:18:35.588 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:18:35.588 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-18T08:18:35.588 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:12.295875+0000", 2024-09-18T08:18:35.588 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 6.2168000000000005e-05 2024-09-18T08:18:35.589 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:18:35.589 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:18:35.589 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "waiting for peered", 2024-09-18T08:18:35.589 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:12.295883+0000", 2024-09-18T08:18:35.589 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 7.9969999999999998e-06 2024-09-18T08:18:35.589 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:18:35.590 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:18:35.590 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-18T08:18:35.590 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:13.187072+0000", 2024-09-18T08:18:35.590 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.89118984899999998 2024-09-18T08:18:35.590 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:18:35.591 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:18:35.591 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-09-18T08:18:35.591 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:13.187193+0000", 2024-09-18T08:18:35.591 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00012062599999999999 2024-09-18T08:18:35.591 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:18:35.591 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:18:35.592 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "waiting for subops from 6,7", 2024-09-18T08:18:35.592 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:13.187351+0000", 2024-09-18T08:18:35.592 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00015781500000000001 2024-09-18T08:18:35.592 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:18:35.592 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:18:35.592 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:18:35.593 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:13.194877+0000", 2024-09-18T08:18:35.593 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0075262139999999998 2024-09-18T08:18:35.593 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:18:35.593 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:18:35.593 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:18:35.594 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:13.195031+0000", 2024-09-18T08:18:35.594 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00015384799999999999 2024-09-18T08:18:35.594 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:18:35.594 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:18:35.594 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "op_commit", 2024-09-18T08:18:35.594 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:13.219386+0000", 2024-09-18T08:18:35.595 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.024355226000000001 2024-09-18T08:18:35.595 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:18:35.595 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:18:35.595 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "commit_sent", 2024-09-18T08:18:35.595 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:13.219440+0000", 2024-09-18T08:18:35.595 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 5.3786999999999998e-05 2024-09-18T08:18:35.596 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:18:35.596 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:18:35.596 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-09-18T08:18:35.596 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:13.219493+0000", 2024-09-18T08:18:35.596 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 5.2899999999999998e-05 2024-09-18T08:18:35.596 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-18T08:18:35.597 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-09-18T08:18:35.597 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-18T08:18:35.597 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:18:35.597 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:18:35.597 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.15216.0:86350 8.5 8:a14e8599:::benchmark_data_smithi007_101583_object86349:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e205)", 2024-09-18T08:18:35.597 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-09-18T08:09:12.297534+0000", 2024-09-18T08:18:35.598 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 563.23455756500005, 2024-09-18T08:18:35.598 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.922055505, 2024-09-18T08:18:35.598 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-09-18T08:18:35.598 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-09-18T08:18:35.599 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T08:18:35.599 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2024-09-18T08:18:35.599 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.15216", 2024-09-18T08:18:35.599 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "172.21.15.7:0/2596041177", 2024-09-18T08:18:35.599 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 86350 2024-09-18T08:18:35.599 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:18:35.600 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-09-18T08:18:35.600 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:18:35.600 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-09-18T08:18:35.600 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:12.297534+0000", 2024-09-18T08:18:35.600 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-18T08:18:35.601 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:18:35.601 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:18:35.601 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-09-18T08:18:35.601 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:12.297534+0000", 2024-09-18T08:18:35.601 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-18T08:18:35.602 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:18:35.602 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:18:35.602 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-09-18T08:18:35.602 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:12.297534+0000", 2024-09-18T08:18:35.602 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 7.0800000000000004e-07 2024-09-18T08:18:35.602 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:18:35.603 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:18:35.603 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-09-18T08:18:35.603 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:12.297606+0000", 2024-09-18T08:18:35.603 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 7.1075000000000006e-05 2024-09-18T08:18:35.603 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:18:35.603 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:18:35.604 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-09-18T08:18:35.604 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:12.297613+0000", 2024-09-18T08:18:35.604 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 7.6260000000000003e-06 2024-09-18T08:18:35.604 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:18:35.604 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:18:35.605 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-09-18T08:18:35.605 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:12.297618+0000", 2024-09-18T08:18:35.605 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4.8550000000000001e-06 2024-09-18T08:18:35.605 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:18:35.605 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:18:35.605 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-18T08:18:35.606 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:12.297660+0000", 2024-09-18T08:18:35.606 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4.1968000000000002e-05 2024-09-18T08:18:35.606 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:18:35.606 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:18:35.606 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "waiting for peered", 2024-09-18T08:18:35.607 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:12.297666+0000", 2024-09-18T08:18:35.607 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 5.806e-06 2024-09-18T08:18:35.607 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:18:35.607 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:18:35.607 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-18T08:18:35.607 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:13.189816+0000", 2024-09-18T08:18:35.608 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.89215055300000001 2024-09-18T08:18:35.608 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:18:35.608 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:18:35.608 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-09-18T08:18:35.608 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:13.190014+0000", 2024-09-18T08:18:35.608 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000197894 2024-09-18T08:18:35.609 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:18:35.609 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:18:35.609 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "waiting for subops from 6,7", 2024-09-18T08:18:35.609 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:13.190197+0000", 2024-09-18T08:18:35.609 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00018230399999999999 2024-09-18T08:18:35.610 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:18:35.610 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:18:35.610 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:18:35.610 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:13.194999+0000", 2024-09-18T08:18:35.610 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0048023019999999996 2024-09-18T08:18:35.610 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:18:35.611 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:18:35.611 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:18:35.611 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:13.197606+0000", 2024-09-18T08:18:35.611 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0026074890000000002 2024-09-18T08:18:35.611 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:18:35.612 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:18:35.612 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "op_commit", 2024-09-18T08:18:35.612 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:13.219502+0000", 2024-09-18T08:18:35.612 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.02189514 2024-09-18T08:18:35.612 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:18:35.613 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:18:35.613 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "commit_sent", 2024-09-18T08:18:35.613 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:13.219546+0000", 2024-09-18T08:18:35.613 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4.4574999999999999e-05 2024-09-18T08:18:35.613 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:18:35.613 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:18:35.614 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-09-18T08:18:35.614 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:13.219589+0000", 2024-09-18T08:18:35.614 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4.3210000000000001e-05 2024-09-18T08:18:35.614 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-18T08:18:35.614 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-09-18T08:18:35.615 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-18T08:18:35.615 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:18:35.615 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:18:35.615 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.15216.0:86366 8.5 8:a8c4536d:::benchmark_data_smithi007_101583_object86365:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e205)", 2024-09-18T08:18:35.615 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-09-18T08:09:12.318202+0000", 2024-09-18T08:18:35.615 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 563.21388888900003, 2024-09-18T08:18:35.615 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.90148185700000005, 2024-09-18T08:18:35.615 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-09-18T08:18:35.615 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-09-18T08:18:35.616 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T08:18:35.616 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2024-09-18T08:18:35.616 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.15216", 2024-09-18T08:18:35.616 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "172.21.15.7:0/2596041177", 2024-09-18T08:18:35.616 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 86366 2024-09-18T08:18:35.616 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:18:35.616 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-09-18T08:18:35.616 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:18:35.616 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-09-18T08:18:35.616 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:12.318202+0000", 2024-09-18T08:18:35.616 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-18T08:18:35.616 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:18:35.616 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:18:35.617 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-09-18T08:18:35.617 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:12.318202+0000", 2024-09-18T08:18:35.617 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-18T08:18:35.617 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:18:35.617 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:18:35.617 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-09-18T08:18:35.617 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:12.318205+0000", 2024-09-18T08:18:35.617 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 2.074e-06 2024-09-18T08:18:35.617 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:18:35.617 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:18:35.617 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-09-18T08:18:35.617 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:12.318242+0000", 2024-09-18T08:18:35.617 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 3.6996999999999998e-05 2024-09-18T08:18:35.617 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:18:35.618 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:18:35.618 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-09-18T08:18:35.618 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:12.318258+0000", 2024-09-18T08:18:35.618 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.6382e-05 2024-09-18T08:18:35.618 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:18:35.618 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:18:35.618 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-09-18T08:18:35.618 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:12.318266+0000", 2024-09-18T08:18:35.618 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 7.9820000000000008e-06 2024-09-18T08:18:35.618 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:18:35.618 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:18:35.618 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-18T08:18:35.618 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:12.318402+0000", 2024-09-18T08:18:35.618 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000136477 2024-09-18T08:18:35.619 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:18:35.619 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:18:35.619 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "waiting for peered", 2024-09-18T08:18:35.619 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:12.318414+0000", 2024-09-18T08:18:35.619 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.2096000000000001e-05 2024-09-18T08:18:35.619 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:18:35.619 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:18:35.619 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-18T08:18:35.619 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:13.192024+0000", 2024-09-18T08:18:35.619 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.87360952300000005 2024-09-18T08:18:35.619 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:18:35.619 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:18:35.619 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-09-18T08:18:35.620 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:13.192155+0000", 2024-09-18T08:18:35.620 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00013099499999999999 2024-09-18T08:18:35.620 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:18:35.620 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:18:35.620 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "waiting for subops from 6,7", 2024-09-18T08:18:35.620 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:13.192313+0000", 2024-09-18T08:18:35.620 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00015802500000000001 2024-09-18T08:18:35.620 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:18:35.620 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:18:35.620 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:18:35.620 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:13.195063+0000", 2024-09-18T08:18:35.620 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0027500279999999998 2024-09-18T08:18:35.621 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:18:35.621 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:18:35.621 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:18:35.621 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:13.200043+0000", 2024-09-18T08:18:35.621 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0049800929999999997 2024-09-18T08:18:35.621 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:18:35.621 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:18:35.621 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "op_commit", 2024-09-18T08:18:35.621 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:13.219598+0000", 2024-09-18T08:18:35.621 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.019554768 2024-09-18T08:18:35.621 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:18:35.622 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:18:35.622 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "commit_sent", 2024-09-18T08:18:35.622 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:13.219647+0000", 2024-09-18T08:18:35.622 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4.9327999999999999e-05 2024-09-18T08:18:35.622 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:18:35.622 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:18:35.622 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-09-18T08:18:35.622 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:13.219684+0000", 2024-09-18T08:18:35.622 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 3.7088999999999997e-05 2024-09-18T08:18:35.622 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-18T08:18:35.622 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-09-18T08:18:35.622 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-18T08:18:35.622 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:18:35.622 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:18:35.623 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6 2024-09-18T08:18:35.624 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/779349ca-7593-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T08:18:35.625 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.15216.0:86390 8.5 8:a8c7b3fd:::benchmark_data_smithi007_101583_object86389:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e205)", 2024-09-18T08:18:35.625 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-09-18T08:09:12.356640+0000", 2024-09-18T08:18:35.625 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 563.17545158799999, 2024-09-18T08:18:35.625 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.86366743700000004, 2024-09-18T08:18:35.625 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-09-18T08:18:35.625 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-09-18T08:18:35.625 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T08:18:35.626 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2024-09-18T08:18:35.626 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.15216", 2024-09-18T08:18:35.626 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "172.21.15.7:0/2596041177", 2024-09-18T08:18:35.626 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 86390 2024-09-18T08:18:35.626 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:18:35.626 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-09-18T08:18:35.626 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:18:35.626 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-09-18T08:18:35.626 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:12.356640+0000", 2024-09-18T08:18:35.626 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-18T08:18:35.626 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:18:35.627 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:18:35.627 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-09-18T08:18:35.627 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:12.356640+0000", 2024-09-18T08:18:35.627 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-18T08:18:35.627 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:18:35.627 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:18:35.627 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-09-18T08:18:35.627 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:12.356641+0000", 2024-09-18T08:18:35.627 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.5430000000000001e-06 2024-09-18T08:18:35.627 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:18:35.627 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:18:35.627 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-09-18T08:18:35.627 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:12.356684+0000", 2024-09-18T08:18:35.627 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4.2222999999999999e-05 2024-09-18T08:18:35.628 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:18:35.628 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:18:35.628 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-09-18T08:18:35.628 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:12.356696+0000", 2024-09-18T08:18:35.628 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.2663000000000001e-05 2024-09-18T08:18:35.628 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:18:35.628 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:18:35.628 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-09-18T08:18:35.628 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:12.356703+0000", 2024-09-18T08:18:35.628 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 6.6710000000000002e-06 2024-09-18T08:18:35.628 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:18:35.628 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:18:35.628 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-18T08:18:35.628 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:12.356759+0000", 2024-09-18T08:18:35.629 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 5.6382000000000003e-05 2024-09-18T08:18:35.629 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:18:35.629 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:18:35.629 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "waiting for peered", 2024-09-18T08:18:35.629 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:12.356767+0000", 2024-09-18T08:18:35.629 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 7.5190000000000003e-06 2024-09-18T08:18:35.629 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:18:35.629 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:18:35.629 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-18T08:18:35.629 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:13.195090+0000", 2024-09-18T08:18:35.629 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.83832288600000004 2024-09-18T08:18:35.629 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:18:35.629 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:18:35.630 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-09-18T08:18:35.630 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:13.195221+0000", 2024-09-18T08:18:35.630 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00013146699999999999 2024-09-18T08:18:35.630 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:18:35.630 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:18:35.630 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "waiting for subops from 6,7", 2024-09-18T08:18:35.630 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:13.195370+0000", 2024-09-18T08:18:35.630 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000148598 2024-09-18T08:18:35.630 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:18:35.630 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:18:35.630 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:18:35.630 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:13.199845+0000", 2024-09-18T08:18:35.630 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0044749999999999998 2024-09-18T08:18:35.630 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:18:35.631 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:18:35.631 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:18:35.631 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:13.207294+0000", 2024-09-18T08:18:35.631 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0074489309999999998 2024-09-18T08:18:35.631 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:18:35.631 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:18:35.631 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "op_commit", 2024-09-18T08:18:35.631 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:13.220218+0000", 2024-09-18T08:18:35.631 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.012924715 2024-09-18T08:18:35.631 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:18:35.631 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:18:35.631 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "commit_sent", 2024-09-18T08:18:35.631 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:13.220272+0000", 2024-09-18T08:18:35.632 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 5.3748000000000001e-05 2024-09-18T08:18:35.632 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:18:35.632 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:18:35.632 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-09-18T08:18:35.632 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:13.220307+0000", 2024-09-18T08:18:35.632 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 3.5091e-05 2024-09-18T08:18:35.632 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-18T08:18:35.632 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-09-18T08:18:35.632 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-18T08:18:35.632 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:18:35.632 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:18:35.632 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.15216.0:86394 8.5 8:a810cd9d:::benchmark_data_smithi007_101583_object86393:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e205)", 2024-09-18T08:18:35.632 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-09-18T08:09:12.360967+0000", 2024-09-18T08:18:35.633 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 563.17112401099996, 2024-09-18T08:18:35.633 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.85950780100000002, 2024-09-18T08:18:35.633 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-09-18T08:18:35.633 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-09-18T08:18:35.633 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T08:18:35.633 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2024-09-18T08:18:35.633 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.15216", 2024-09-18T08:18:35.633 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "172.21.15.7:0/2596041177", 2024-09-18T08:18:35.633 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 86394 2024-09-18T08:18:35.633 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:18:35.633 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-09-18T08:18:35.633 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:18:35.633 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-09-18T08:18:35.634 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:12.360967+0000", 2024-09-18T08:18:35.634 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-18T08:18:35.634 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:18:35.634 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:18:35.634 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-09-18T08:18:35.634 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:12.360967+0000", 2024-09-18T08:18:35.634 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-18T08:18:35.634 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:18:35.634 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:18:35.634 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-09-18T08:18:35.634 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:12.360969+0000", 2024-09-18T08:18:35.634 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.8500000000000001e-06 2024-09-18T08:18:35.635 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:18:35.635 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:18:35.635 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-09-18T08:18:35.635 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:12.361013+0000", 2024-09-18T08:18:35.635 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4.3693000000000003e-05 2024-09-18T08:18:35.635 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:18:35.635 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:18:35.635 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-09-18T08:18:35.635 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:12.361031+0000", 2024-09-18T08:18:35.635 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.8340000000000001e-05 2024-09-18T08:18:35.635 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:18:35.635 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:18:35.635 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-09-18T08:18:35.635 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:12.361040+0000", 2024-09-18T08:18:35.636 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 8.5320000000000006e-06 2024-09-18T08:18:35.636 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:18:35.636 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:18:35.636 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-18T08:18:35.636 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:12.361108+0000", 2024-09-18T08:18:35.636 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 6.7879000000000005e-05 2024-09-18T08:18:35.636 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:18:35.636 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:18:35.636 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "waiting for peered", 2024-09-18T08:18:35.636 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:12.361119+0000", 2024-09-18T08:18:35.636 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.1163e-05 2024-09-18T08:18:35.636 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:18:35.636 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:18:35.637 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-18T08:18:35.637 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:13.197641+0000", 2024-09-18T08:18:35.637 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.83652230299999997 2024-09-18T08:18:35.637 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:18:35.637 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:18:35.637 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-09-18T08:18:35.637 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:13.197776+0000", 2024-09-18T08:18:35.637 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00013499000000000001 2024-09-18T08:18:35.637 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:18:35.637 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:18:35.637 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "waiting for subops from 6,7", 2024-09-18T08:18:35.637 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:13.197954+0000", 2024-09-18T08:18:35.637 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00017745200000000001 2024-09-18T08:18:35.637 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:18:35.638 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:18:35.638 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:18:35.638 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:13.207329+0000", 2024-09-18T08:18:35.638 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0093754460000000008 2024-09-18T08:18:35.638 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:18:35.638 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:18:35.638 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:18:35.638 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:13.207363+0000", 2024-09-18T08:18:35.638 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 3.3535999999999999e-05 2024-09-18T08:18:35.638 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:18:35.638 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:18:35.638 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "op_commit", 2024-09-18T08:18:35.638 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:13.220323+0000", 2024-09-18T08:18:35.638 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.012960121999999999 2024-09-18T08:18:35.639 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:18:35.639 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:18:35.639 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "commit_sent", 2024-09-18T08:18:35.639 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:13.220445+0000", 2024-09-18T08:18:35.639 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00012217099999999999 2024-09-18T08:18:35.639 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:18:35.639 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:18:35.639 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-09-18T08:18:35.639 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:13.220475+0000", 2024-09-18T08:18:35.639 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 3.0324e-05 2024-09-18T08:18:35.639 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-18T08:18:35.639 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-09-18T08:18:35.639 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-18T08:18:35.639 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:18:35.640 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:18:35.640 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.15216.0:86399 8.5 8:a535189e:::benchmark_data_smithi007_101583_object86398:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e205)", 2024-09-18T08:18:35.640 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-09-18T08:09:12.364040+0000", 2024-09-18T08:18:35.640 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 563.16805094699998, 2024-09-18T08:18:35.640 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.85654366599999998, 2024-09-18T08:18:35.640 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-09-18T08:18:35.640 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-09-18T08:18:35.640 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T08:18:35.640 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2024-09-18T08:18:35.640 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.15216", 2024-09-18T08:18:35.640 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "172.21.15.7:0/2596041177", 2024-09-18T08:18:35.640 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 86399 2024-09-18T08:18:35.640 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:18:35.640 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-09-18T08:18:35.641 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:18:35.641 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-09-18T08:18:35.641 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:12.364040+0000", 2024-09-18T08:18:35.641 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-18T08:18:35.641 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:18:35.641 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:18:35.641 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-09-18T08:18:35.641 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:12.364040+0000", 2024-09-18T08:18:35.641 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-18T08:18:35.641 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:18:35.641 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:18:35.641 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-09-18T08:18:35.641 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:12.364042+0000", 2024-09-18T08:18:35.642 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.407e-06 2024-09-18T08:18:35.642 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:18:35.642 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:18:35.642 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-09-18T08:18:35.642 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:12.364174+0000", 2024-09-18T08:18:35.642 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000132545 2024-09-18T08:18:35.642 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:18:35.642 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:18:35.642 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-09-18T08:18:35.642 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:12.364186+0000", 2024-09-18T08:18:35.642 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.1912e-05 2024-09-18T08:18:35.642 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:18:35.642 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:18:35.642 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-09-18T08:18:35.643 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:12.364192+0000", 2024-09-18T08:18:35.643 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 5.4600000000000002e-06 2024-09-18T08:18:35.643 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:18:35.643 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:18:35.643 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-18T08:18:35.643 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:12.364257+0000", 2024-09-18T08:18:35.643 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 6.5606999999999997e-05 2024-09-18T08:18:35.643 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:18:35.643 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:18:35.643 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "waiting for peered", 2024-09-18T08:18:35.643 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:12.364271+0000", 2024-09-18T08:18:35.643 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.3433e-05 2024-09-18T08:18:35.643 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:18:35.643 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:18:35.644 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-18T08:18:35.644 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:13.200082+0000", 2024-09-18T08:18:35.644 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.83581135500000003 2024-09-18T08:18:35.644 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:18:35.644 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:18:35.644 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-09-18T08:18:35.644 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:13.200201+0000", 2024-09-18T08:18:35.644 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00011913099999999999 2024-09-18T08:18:35.644 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:18:35.644 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:18:35.644 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "waiting for subops from 6,7", 2024-09-18T08:18:35.644 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:13.200360+0000", 2024-09-18T08:18:35.644 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000159173 2024-09-18T08:18:35.644 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:18:35.645 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:18:35.645 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:18:35.645 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:13.207395+0000", 2024-09-18T08:18:35.645 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0070341090000000002 2024-09-18T08:18:35.645 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:18:35.645 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:18:35.645 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:18:35.645 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:13.207427+0000", 2024-09-18T08:18:35.645 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 3.2138000000000003e-05 2024-09-18T08:18:35.645 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:18:35.645 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:18:35.645 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "op_commit", 2024-09-18T08:18:35.645 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:13.220524+0000", 2024-09-18T08:18:35.646 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.013097698 2024-09-18T08:18:35.646 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:18:35.646 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:18:35.646 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "commit_sent", 2024-09-18T08:18:35.646 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:13.220557+0000", 2024-09-18T08:18:35.646 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 3.2682999999999999e-05 2024-09-18T08:18:35.646 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:18:35.646 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:18:35.646 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-09-18T08:18:35.646 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:13.220584+0000", 2024-09-18T08:18:35.646 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 2.7015000000000001e-05 2024-09-18T08:18:35.646 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-18T08:18:35.646 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-09-18T08:18:35.647 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-18T08:18:35.647 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:18:35.647 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:18:35.647 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.15216.0:86429 8.5 8:a7f1f687:::benchmark_data_smithi007_101583_object86428:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e205)", 2024-09-18T08:18:35.647 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-09-18T08:09:12.416915+0000", 2024-09-18T08:18:35.647 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 563.11517617799996, 2024-09-18T08:18:35.647 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.80373175900000005, 2024-09-18T08:18:35.647 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-09-18T08:18:35.647 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-09-18T08:18:35.647 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T08:18:35.647 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2024-09-18T08:18:35.647 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.15216", 2024-09-18T08:18:35.648 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "172.21.15.7:0/2596041177", 2024-09-18T08:18:35.648 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 86429 2024-09-18T08:18:35.648 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:18:35.648 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-09-18T08:18:35.648 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:18:35.648 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-09-18T08:18:35.648 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:12.416915+0000", 2024-09-18T08:18:35.648 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-18T08:18:35.648 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:18:35.648 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:18:35.648 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-09-18T08:18:35.648 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:12.416915+0000", 2024-09-18T08:18:35.648 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-18T08:18:35.648 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:18:35.649 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:18:35.649 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-09-18T08:18:35.649 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:12.416917+0000", 2024-09-18T08:18:35.649 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.7150000000000001e-06 2024-09-18T08:18:35.649 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:18:35.649 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:18:35.649 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-09-18T08:18:35.649 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:12.416952+0000", 2024-09-18T08:18:35.649 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 3.4860000000000002e-05 2024-09-18T08:18:35.649 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:18:35.649 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:18:35.649 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-09-18T08:18:35.649 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:12.416966+0000", 2024-09-18T08:18:35.649 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.3884e-05 2024-09-18T08:18:35.650 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:18:35.650 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:18:35.650 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-09-18T08:18:35.650 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:12.416973+0000", 2024-09-18T08:18:35.650 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 7.5809999999999999e-06 2024-09-18T08:18:35.650 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:18:35.650 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:18:35.650 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-18T08:18:35.650 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:12.417031+0000", 2024-09-18T08:18:35.650 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 5.7544999999999999e-05 2024-09-18T08:18:35.650 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:18:35.650 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:18:35.650 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "waiting for peered", 2024-09-18T08:18:35.650 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:12.417039+0000", 2024-09-18T08:18:35.651 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 8.5369999999999997e-06 2024-09-18T08:18:35.651 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:18:35.651 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:18:35.651 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-18T08:18:35.651 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:13.202704+0000", 2024-09-18T08:18:35.651 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.78566510099999998 2024-09-18T08:18:35.651 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:18:35.651 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:18:35.651 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-09-18T08:18:35.651 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:13.202825+0000", 2024-09-18T08:18:35.651 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00012023600000000001 2024-09-18T08:18:35.651 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:18:35.651 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:18:35.651 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "waiting for subops from 6,7", 2024-09-18T08:18:35.652 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:13.202989+0000", 2024-09-18T08:18:35.652 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00016451000000000001 2024-09-18T08:18:35.652 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:18:35.652 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:18:35.652 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:18:35.652 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:13.207464+0000", 2024-09-18T08:18:35.652 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0044748909999999999 2024-09-18T08:18:35.652 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:18:35.652 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:18:35.652 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:18:35.652 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:13.214034+0000", 2024-09-18T08:18:35.652 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0065698229999999998 2024-09-18T08:18:35.652 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:18:35.653 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:18:35.653 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "op_commit", 2024-09-18T08:18:35.653 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:13.220590+0000", 2024-09-18T08:18:35.653 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0065556909999999998 2024-09-18T08:18:35.653 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:18:35.653 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:18:35.653 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "commit_sent", 2024-09-18T08:18:35.653 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:13.220620+0000", 2024-09-18T08:18:35.653 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 3.0429999999999998e-05 2024-09-18T08:18:35.653 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:18:35.653 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:18:35.653 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-09-18T08:18:35.654 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:09:13.220647+0000", 2024-09-18T08:18:35.654 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 2.6954999999999999e-05 2024-09-18T08:18:35.654 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-18T08:18:35.654 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-09-18T08:18:35.654 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-18T08:18:35.654 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:18:35.654 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:18:35.654 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.15582.0:53562 9.e 9:7f5cd26d:::benchmark_data_smithi007_124019_object53561:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e243)", 2024-09-18T08:18:35.654 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-09-18T08:11:41.305003+0000", 2024-09-18T08:18:35.654 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 414.227088654, 2024-09-18T08:18:35.654 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.032254158, 2024-09-18T08:18:35.654 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-09-18T08:18:35.654 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-09-18T08:18:35.654 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "reached pg", 2024-09-18T08:18:35.655 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2024-09-18T08:18:35.655 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.15582", 2024-09-18T08:18:35.655 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "172.21.15.7:0/3598605839", 2024-09-18T08:18:35.655 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 53562 2024-09-18T08:18:35.655 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:18:35.655 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-09-18T08:18:35.655 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:18:35.655 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-09-18T08:18:35.655 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:11:41.305003+0000", 2024-09-18T08:18:35.655 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-18T08:18:35.655 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:18:35.655 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:18:35.655 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-09-18T08:18:35.656 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:11:41.305003+0000", 2024-09-18T08:18:35.656 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-18T08:18:35.656 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:18:35.656 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:18:35.656 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-09-18T08:18:35.656 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:11:41.305004+0000", 2024-09-18T08:18:35.656 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.6750000000000001e-06 2024-09-18T08:18:35.656 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:18:35.656 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:18:35.656 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-09-18T08:18:35.656 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:11:41.305059+0000", 2024-09-18T08:18:35.656 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 5.4181999999999997e-05 2024-09-18T08:18:35.657 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:18:35.657 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:18:35.657 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-09-18T08:18:35.657 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:11:41.305074+0000", 2024-09-18T08:18:35.657 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.5379999999999998e-05 2024-09-18T08:18:35.657 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:18:35.657 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:18:35.657 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-09-18T08:18:35.657 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:11:41.305083+0000", 2024-09-18T08:18:35.657 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 8.9549999999999998e-06 2024-09-18T08:18:35.657 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:18:35.657 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:18:35.657 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-18T08:18:35.658 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:11:41.305153+0000", 2024-09-18T08:18:35.658 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 6.9703999999999995e-05 2024-09-18T08:18:35.658 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:18:35.658 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:18:35.658 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-09-18T08:18:35.658 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:11:41.305277+0000", 2024-09-18T08:18:35.658 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0001247 2024-09-18T08:18:35.658 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:18:35.658 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:18:35.658 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "waiting for subops from 1,7", 2024-09-18T08:18:35.658 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:11:41.305414+0000", 2024-09-18T08:18:35.658 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00013647900000000001 2024-09-18T08:18:35.658 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:18:35.659 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:18:35.659 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "op_commit", 2024-09-18T08:18:35.659 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:11:41.307315+0000", 2024-09-18T08:18:35.659 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.001901511 2024-09-18T08:18:35.659 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:18:35.659 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:18:35.659 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:18:35.659 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:11:41.307516+0000", 2024-09-18T08:18:35.659 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00020116900000000001 2024-09-18T08:18:35.659 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:18:35.659 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:18:35.659 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-18T08:18:35.659 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:11:41.323519+0000", 2024-09-18T08:18:35.659 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.016002283999999999 2024-09-18T08:18:35.659 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:18:35.660 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:18:35.660 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "waiting for peered", 2024-09-18T08:18:35.660 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:11:41.323530+0000", 2024-09-18T08:18:35.660 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.1729000000000001e-05 2024-09-18T08:18:35.660 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:18:35.660 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:18:35.660 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-18T08:18:35.660 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:11:42.337165+0000", 2024-09-18T08:18:35.660 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.0136343050000001 2024-09-18T08:18:35.660 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:18:35.660 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:18:35.660 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-09-18T08:18:35.660 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:11:42.337257+0000", 2024-09-18T08:18:35.660 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 9.2084999999999997e-05 2024-09-18T08:18:35.661 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-18T08:18:35.661 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-09-18T08:18:35.661 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-18T08:18:35.661 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:18:35.661 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:18:35.661 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.15582.0:53574 9.e 9:7c49e58c:::benchmark_data_smithi007_124019_object53573:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e243)", 2024-09-18T08:18:35.661 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-09-18T08:11:41.323835+0000", 2024-09-18T08:18:35.661 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 414.20825627099998, 2024-09-18T08:18:35.661 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.0202460840000001, 2024-09-18T08:18:35.661 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-09-18T08:18:35.661 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-09-18T08:18:35.661 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T08:18:35.661 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2024-09-18T08:18:35.661 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.15582", 2024-09-18T08:18:35.662 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "172.21.15.7:0/3598605839", 2024-09-18T08:18:35.662 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 53574 2024-09-18T08:18:35.662 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:18:35.662 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-09-18T08:18:35.662 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:18:35.662 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-09-18T08:18:35.662 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:11:41.323835+0000", 2024-09-18T08:18:35.662 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-18T08:18:35.662 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:18:35.662 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:18:35.662 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-09-18T08:18:35.662 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:11:41.323835+0000", 2024-09-18T08:18:35.662 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-18T08:18:35.663 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:18:35.663 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:18:35.663 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-09-18T08:18:35.663 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:11:41.323837+0000", 2024-09-18T08:18:35.663 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.8470000000000001e-06 2024-09-18T08:18:35.663 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:18:35.663 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:18:35.663 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-09-18T08:18:35.663 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:11:41.323872+0000", 2024-09-18T08:18:35.663 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 3.4660999999999999e-05 2024-09-18T08:18:35.663 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:18:35.663 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:18:35.663 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-09-18T08:18:35.663 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:11:41.323886+0000", 2024-09-18T08:18:35.664 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.4779e-05 2024-09-18T08:18:35.664 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:18:35.664 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:18:35.664 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-09-18T08:18:35.664 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:11:41.323895+0000", 2024-09-18T08:18:35.664 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 8.4630000000000001e-06 2024-09-18T08:18:35.664 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:18:35.664 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:18:35.664 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-18T08:18:35.664 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:11:41.323994+0000", 2024-09-18T08:18:35.664 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 9.9152000000000002e-05 2024-09-18T08:18:35.664 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:18:35.664 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:18:35.664 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "waiting for peered", 2024-09-18T08:18:35.665 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:11:41.324002+0000", 2024-09-18T08:18:35.665 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 8.4039999999999995e-06 2024-09-18T08:18:35.665 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:18:35.665 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:18:35.665 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-18T08:18:35.665 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:11:42.337953+0000", 2024-09-18T08:18:35.665 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.013950753 2024-09-18T08:18:35.665 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:18:35.665 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:18:35.665 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-09-18T08:18:35.665 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:11:42.338094+0000", 2024-09-18T08:18:35.665 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000140371 2024-09-18T08:18:35.665 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:18:35.666 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:18:35.666 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "waiting for subops from 7", 2024-09-18T08:18:35.666 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:11:42.338248+0000", 2024-09-18T08:18:35.666 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000154723 2024-09-18T08:18:35.666 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:18:35.666 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:18:35.666 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:18:35.666 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:11:42.343852+0000", 2024-09-18T08:18:35.666 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0056041909999999997 2024-09-18T08:18:35.666 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:18:35.666 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:18:35.666 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "op_commit", 2024-09-18T08:18:35.666 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:11:42.343917+0000", 2024-09-18T08:18:35.666 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 6.4090000000000005e-05 2024-09-18T08:18:35.667 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:18:35.667 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:18:35.667 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "commit_sent", 2024-09-18T08:18:35.667 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:11:42.343980+0000", 2024-09-18T08:18:35.667 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 6.3769999999999994e-05 2024-09-18T08:18:35.667 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:18:35.667 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:18:35.667 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-09-18T08:18:35.667 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:11:42.344081+0000", 2024-09-18T08:18:35.667 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00010088 2024-09-18T08:18:35.667 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-18T08:18:35.667 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-09-18T08:18:35.667 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-18T08:18:35.667 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:18:35.668 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:18:35.668 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.15582.0:53552 9.4 9:2d548aaa:::benchmark_data_smithi007_124019_object53551:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e244)", 2024-09-18T08:18:35.668 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-09-18T08:11:41.324333+0000", 2024-09-18T08:18:35.668 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 414.20775884699998, 2024-09-18T08:18:35.668 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.0111114809999999, 2024-09-18T08:18:35.668 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-09-18T08:18:35.668 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-09-18T08:18:35.668 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "reached pg", 2024-09-18T08:18:35.668 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2024-09-18T08:18:35.668 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.15582", 2024-09-18T08:18:35.668 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "172.21.15.7:0/3598605839", 2024-09-18T08:18:35.668 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 53552 2024-09-18T08:18:35.668 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:18:35.668 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-09-18T08:18:35.669 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:18:35.669 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-09-18T08:18:35.669 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:11:41.324333+0000", 2024-09-18T08:18:35.669 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-18T08:18:35.669 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:18:35.669 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:18:35.669 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-09-18T08:18:35.669 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:11:41.324333+0000", 2024-09-18T08:18:35.669 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-18T08:18:35.669 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:18:35.669 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:18:35.669 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-09-18T08:18:35.669 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:11:41.324334+0000", 2024-09-18T08:18:35.669 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.217e-06 2024-09-18T08:18:35.670 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:18:35.670 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:18:35.670 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-09-18T08:18:35.670 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:11:41.324369+0000", 2024-09-18T08:18:35.670 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 3.4805999999999998e-05 2024-09-18T08:18:35.670 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:18:35.670 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:18:35.670 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-09-18T08:18:35.670 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:11:41.324382+0000", 2024-09-18T08:18:35.670 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.3739e-05 2024-09-18T08:18:35.670 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:18:35.670 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:18:35.670 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-09-18T08:18:35.671 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:11:41.324390+0000", 2024-09-18T08:18:35.671 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 7.3989999999999998e-06 2024-09-18T08:18:35.671 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:18:35.671 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:18:35.671 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-18T08:18:35.671 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:11:41.324445+0000", 2024-09-18T08:18:35.671 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 5.5535999999999998e-05 2024-09-18T08:18:35.671 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:18:35.671 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:18:35.671 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "waiting for peered", 2024-09-18T08:18:35.671 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:11:41.324453+0000", 2024-09-18T08:18:35.671 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 8.2139999999999996e-06 2024-09-18T08:18:35.671 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:18:35.672 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:18:35.672 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-18T08:18:35.672 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:11:42.335357+0000", 2024-09-18T08:18:35.672 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.010903675 2024-09-18T08:18:35.672 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:18:35.672 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:18:35.672 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-09-18T08:18:35.672 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:11:42.335444+0000", 2024-09-18T08:18:35.672 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 8.6895000000000001e-05 2024-09-18T08:18:35.672 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-18T08:18:35.672 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-09-18T08:18:35.672 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-18T08:18:35.672 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:18:35.672 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:18:35.673 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.15582.0:53563 9.4 9:2254eea2:::benchmark_data_smithi007_124019_object53562:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e244)", 2024-09-18T08:18:35.673 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-09-18T08:11:41.324409+0000", 2024-09-18T08:18:35.673 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 414.20768216900001, 2024-09-18T08:18:35.673 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.022272439, 2024-09-18T08:18:35.673 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-09-18T08:18:35.673 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-09-18T08:18:35.673 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "reached pg", 2024-09-18T08:18:35.673 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2024-09-18T08:18:35.673 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.15582", 2024-09-18T08:18:35.673 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "172.21.15.7:0/3598605839", 2024-09-18T08:18:35.673 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 53563 2024-09-18T08:18:35.673 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:18:35.674 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-09-18T08:18:35.674 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:18:35.674 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-09-18T08:18:35.674 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:11:41.324409+0000", 2024-09-18T08:18:35.674 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-18T08:18:35.674 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:18:35.674 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:18:35.674 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-09-18T08:18:35.674 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:11:41.324409+0000", 2024-09-18T08:18:35.674 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-18T08:18:35.674 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:18:35.674 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:18:35.674 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-09-18T08:18:35.674 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:11:41.324410+0000", 2024-09-18T08:18:35.675 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 5.8400000000000004e-07 2024-09-18T08:18:35.675 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:18:35.675 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:18:35.675 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-09-18T08:18:35.675 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:11:41.324438+0000", 2024-09-18T08:18:35.675 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 2.8438000000000001e-05 2024-09-18T08:18:35.675 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:18:35.675 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:18:35.675 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-09-18T08:18:35.675 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:11:41.324446+0000", 2024-09-18T08:18:35.675 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 8.0020000000000006e-06 2024-09-18T08:18:35.675 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:18:35.675 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:18:35.675 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-09-18T08:18:35.676 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:11:41.324453+0000", 2024-09-18T08:18:35.676 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 6.4910000000000002e-06 2024-09-18T08:18:35.676 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:18:35.676 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:18:35.676 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-18T08:18:35.676 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:11:41.324565+0000", 2024-09-18T08:18:35.676 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000111983 2024-09-18T08:18:35.676 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:18:35.676 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:18:35.676 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "waiting for peered", 2024-09-18T08:18:35.676 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:11:41.324572+0000", 2024-09-18T08:18:35.676 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 7.8080000000000005e-06 2024-09-18T08:18:35.676 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:18:35.676 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:18:35.677 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-18T08:18:35.677 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:11:42.335472+0000", 2024-09-18T08:18:35.677 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.010899996 2024-09-18T08:18:35.677 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:18:35.677 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:18:35.677 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "waiting for missing object", 2024-09-18T08:18:35.677 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:11:42.335604+0000", 2024-09-18T08:18:35.677 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00013134999999999999 2024-09-18T08:18:35.677 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:18:35.677 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:18:35.677 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-18T08:18:35.677 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:11:42.346593+0000", 2024-09-18T08:18:35.677 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.01098872 2024-09-18T08:18:35.678 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:18:35.678 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:18:35.678 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-09-18T08:18:35.678 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:11:42.346682+0000", 2024-09-18T08:18:35.678 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 8.9067000000000006e-05 2024-09-18T08:18:35.678 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-18T08:18:35.678 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-09-18T08:18:35.678 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-18T08:18:35.678 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:18:35.678 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:18:35.678 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.24356.0:2784 5.1f 5:f95f44c2:::notify.0:head [watch reconnect cookie 94700888679552 gen 6] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e292)", 2024-09-18T08:18:35.678 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-09-18T08:14:04.566002+0000", 2024-09-18T08:18:35.678 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 270.96608957400002, 2024-09-18T08:18:35.679 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.026328065, 2024-09-18T08:18:35.679 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-09-18T08:18:35.679 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-09-18T08:18:35.679 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "started", 2024-09-18T08:18:35.679 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2024-09-18T08:18:35.679 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.24356", 2024-09-18T08:18:35.679 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "172.21.15.7:0/219168083", 2024-09-18T08:18:35.679 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 2784 2024-09-18T08:18:35.679 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:18:35.679 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-09-18T08:18:35.679 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:18:35.679 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-09-18T08:18:35.679 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:14:04.566002+0000", 2024-09-18T08:18:35.679 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-18T08:18:35.680 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:18:35.680 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:18:35.680 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-09-18T08:18:35.680 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:14:04.566002+0000", 2024-09-18T08:18:35.680 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-18T08:18:35.680 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:18:35.680 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:18:35.680 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-09-18T08:18:35.680 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:14:04.566004+0000", 2024-09-18T08:18:35.680 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.9350000000000001e-06 2024-09-18T08:18:35.680 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:18:35.680 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:18:35.680 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-09-18T08:18:35.680 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:14:04.566009+0000", 2024-09-18T08:18:35.681 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 5.5589999999999996e-06 2024-09-18T08:18:35.681 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:18:35.681 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:18:35.681 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-09-18T08:18:35.681 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:14:04.566021+0000", 2024-09-18T08:18:35.681 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.1398e-05 2024-09-18T08:18:35.681 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:18:35.681 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:18:35.681 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-09-18T08:18:35.681 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:14:04.566028+0000", 2024-09-18T08:18:35.681 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 7.8029999999999997e-06 2024-09-18T08:18:35.681 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:18:35.681 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:18:35.682 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-18T08:18:35.682 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:14:04.574060+0000", 2024-09-18T08:18:35.682 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0080314459999999994 2024-09-18T08:18:35.682 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:18:35.682 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:18:35.682 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "waiting for peered", 2024-09-18T08:18:35.682 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:14:04.574071+0000", 2024-09-18T08:18:35.682 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.0693000000000001e-05 2024-09-18T08:18:35.682 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:18:35.682 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:18:35.682 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-18T08:18:35.682 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:14:05.582227+0000", 2024-09-18T08:18:35.682 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.0081559760000001 2024-09-18T08:18:35.682 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:18:35.683 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:18:35.683 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "waiting for degraded object", 2024-09-18T08:18:35.683 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:14:05.582584+0000", 2024-09-18T08:18:35.683 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000357096 2024-09-18T08:18:35.683 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:18:35.683 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:18:35.683 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-18T08:18:35.683 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:14:05.585756+0000", 2024-09-18T08:18:35.683 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0031727320000000002 2024-09-18T08:18:35.683 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:18:35.683 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:18:35.683 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-09-18T08:18:35.683 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:14:05.585848+0000", 2024-09-18T08:18:35.684 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 9.1587000000000003e-05 2024-09-18T08:18:35.684 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:18:35.684 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:18:35.684 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-09-18T08:18:35.684 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:14:05.592330+0000", 2024-09-18T08:18:35.684 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00648184 2024-09-18T08:18:35.684 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-18T08:18:35.684 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-09-18T08:18:35.684 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-18T08:18:35.684 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:18:35.684 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:18:35.684 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.24356.0:2785 4.5 4:a60f9269:lc::lua_package_allowlist:head [watch reconnect cookie 94700889442560 gen 6] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e292)", 2024-09-18T08:18:35.684 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-09-18T08:14:04.566047+0000", 2024-09-18T08:18:35.685 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 270.96604433900001, 2024-09-18T08:18:35.685 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.027464521, 2024-09-18T08:18:35.685 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-09-18T08:18:35.685 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-09-18T08:18:35.685 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "started", 2024-09-18T08:18:35.685 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2024-09-18T08:18:35.685 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.24356", 2024-09-18T08:18:35.685 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "172.21.15.7:0/219168083", 2024-09-18T08:18:35.685 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 2785 2024-09-18T08:18:35.685 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:18:35.685 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-09-18T08:18:35.685 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:18:35.685 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-09-18T08:18:35.685 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:14:04.566047+0000", 2024-09-18T08:18:35.686 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-18T08:18:35.686 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:18:35.686 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:18:35.686 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-09-18T08:18:35.686 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:14:04.566047+0000", 2024-09-18T08:18:35.686 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-18T08:18:35.686 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:18:35.686 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:18:35.686 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-09-18T08:18:35.686 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:14:04.566048+0000", 2024-09-18T08:18:35.686 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 5.8500000000000001e-07 2024-09-18T08:18:35.686 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:18:35.686 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:18:35.687 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-09-18T08:18:35.687 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:14:04.566050+0000", 2024-09-18T08:18:35.687 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 2.8509999999999999e-06 2024-09-18T08:18:35.687 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:18:35.687 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:18:35.687 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-09-18T08:18:35.687 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:14:04.566057+0000", 2024-09-18T08:18:35.687 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 6.9369999999999998e-06 2024-09-18T08:18:35.687 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:18:35.687 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:18:35.687 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-09-18T08:18:35.687 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:14:04.566063+0000", 2024-09-18T08:18:35.688 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 5.6049999999999997e-06 2024-09-18T08:18:35.688 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:18:35.688 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:18:35.688 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-18T08:18:35.688 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:14:04.568597+0000", 2024-09-18T08:18:35.688 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0025344730000000002 2024-09-18T08:18:35.688 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:18:35.688 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:18:35.688 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "waiting for peered", 2024-09-18T08:18:35.688 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:14:04.568606+0000", 2024-09-18T08:18:35.688 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 8.0649999999999999e-06 2024-09-18T08:18:35.688 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:18:35.688 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:18:35.689 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-18T08:18:35.689 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:14:05.586207+0000", 2024-09-18T08:18:35.689 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.017601309 2024-09-18T08:18:35.689 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:18:35.689 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:18:35.689 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "waiting for degraded object", 2024-09-18T08:18:35.689 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:14:05.586551+0000", 2024-09-18T08:18:35.689 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00034398799999999999 2024-09-18T08:18:35.689 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:18:35.689 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:18:35.689 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-18T08:18:35.689 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:14:05.590223+0000", 2024-09-18T08:18:35.689 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0036724499999999998 2024-09-18T08:18:35.689 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:18:35.690 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:18:35.690 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-09-18T08:18:35.690 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:14:05.590327+0000", 2024-09-18T08:18:35.690 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000103261 2024-09-18T08:18:35.690 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:18:35.690 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:18:35.690 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-09-18T08:18:35.690 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:14:05.593512+0000", 2024-09-18T08:18:35.690 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0031849970000000002 2024-09-18T08:18:35.690 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-18T08:18:35.690 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-09-18T08:18:35.690 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-18T08:18:35.690 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:18:35.691 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:18:35.691 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.24356.0:2786 5.9 5:93e5b521:::notify.7:head [watch reconnect cookie 94700888687616 gen 5] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e292)", 2024-09-18T08:18:35.691 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-09-18T08:14:04.566075+0000", 2024-09-18T08:18:35.691 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 270.96601668699998, 2024-09-18T08:18:35.691 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 2.0238415459999999, 2024-09-18T08:18:35.691 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-09-18T08:18:35.691 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-09-18T08:18:35.691 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "started", 2024-09-18T08:18:35.691 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2024-09-18T08:18:35.691 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.24356", 2024-09-18T08:18:35.691 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "172.21.15.7:0/219168083", 2024-09-18T08:18:35.691 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 2786 2024-09-18T08:18:35.691 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:18:35.692 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-09-18T08:18:35.692 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:18:35.692 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-09-18T08:18:35.692 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:14:04.566075+0000", 2024-09-18T08:18:35.692 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-18T08:18:35.692 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:18:35.692 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:18:35.692 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-09-18T08:18:35.692 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:14:04.566075+0000", 2024-09-18T08:18:35.692 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-18T08:18:35.692 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:18:35.692 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:18:35.692 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-09-18T08:18:35.692 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:14:04.566075+0000", 2024-09-18T08:18:35.693 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4.6699999999999999e-07 2024-09-18T08:18:35.693 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:18:35.693 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:18:35.693 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-09-18T08:18:35.693 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:14:04.566077+0000", 2024-09-18T08:18:35.693 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 2.283e-06 2024-09-18T08:18:35.693 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:18:35.693 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:18:35.693 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-09-18T08:18:35.693 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:14:04.566083+0000", 2024-09-18T08:18:35.693 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 5.4739999999999996e-06 2024-09-18T08:18:35.693 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:18:35.693 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:18:35.693 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-09-18T08:18:35.694 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:14:04.566088+0000", 2024-09-18T08:18:35.694 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4.887e-06 2024-09-18T08:18:35.694 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:18:35.694 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:18:35.694 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-18T08:18:35.694 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:14:04.567450+0000", 2024-09-18T08:18:35.694 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.001362261 2024-09-18T08:18:35.694 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:18:35.694 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:18:35.694 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "waiting for peered", 2024-09-18T08:18:35.694 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:14:04.567459+0000", 2024-09-18T08:18:35.694 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 8.5110000000000003e-06 2024-09-18T08:18:35.694 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:18:35.695 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:18:35.695 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-18T08:18:35.695 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:14:05.568619+0000", 2024-09-18T08:18:35.695 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.001160869 2024-09-18T08:18:35.695 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:18:35.695 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:18:35.695 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "waiting for peered", 2024-09-18T08:18:35.695 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:14:05.568637+0000", 2024-09-18T08:18:35.695 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.7801000000000001e-05 2024-09-18T08:18:35.695 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:18:35.695 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:18:35.695 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-18T08:18:35.695 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:14:06.586669+0000", 2024-09-18T08:18:35.695 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.01803212 2024-09-18T08:18:35.696 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:18:35.696 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:18:35.696 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-09-18T08:18:35.696 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:14:06.586966+0000", 2024-09-18T08:18:35.696 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00029684900000000002 2024-09-18T08:18:35.696 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:18:35.696 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:18:35.696 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-09-18T08:18:35.696 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:14:06.589916+0000", 2024-09-18T08:18:35.696 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0029500239999999999 2024-09-18T08:18:35.696 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-18T08:18:35.696 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-09-18T08:18:35.697 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-18T08:18:35.697 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:18:35.697 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:18:35.697 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.24356.0:2788 5.9 5:93e5b521:::notify.7:head [watch reconnect cookie 94700888687616 gen 6] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e293)", 2024-09-18T08:18:35.697 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-09-18T08:14:05.569076+0000", 2024-09-18T08:18:35.697 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 269.963015149, 2024-09-18T08:18:35.697 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.021128322, 2024-09-18T08:18:35.697 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-09-18T08:18:35.697 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-09-18T08:18:35.697 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "started", 2024-09-18T08:18:35.697 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2024-09-18T08:18:35.697 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.24356", 2024-09-18T08:18:35.697 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "172.21.15.7:0/219168083", 2024-09-18T08:18:35.698 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 2788 2024-09-18T08:18:35.698 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:18:35.698 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-09-18T08:18:35.698 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:18:35.698 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-09-18T08:18:35.698 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:14:05.569076+0000", 2024-09-18T08:18:35.698 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-18T08:18:35.698 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:18:35.698 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:18:35.698 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-09-18T08:18:35.698 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:14:05.569076+0000", 2024-09-18T08:18:35.698 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-18T08:18:35.698 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:18:35.699 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:18:35.699 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-09-18T08:18:35.699 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:14:05.569078+0000", 2024-09-18T08:18:35.699 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 2.0250000000000001e-06 2024-09-18T08:18:35.699 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:18:35.699 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:18:35.699 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-09-18T08:18:35.699 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:14:05.569086+0000", 2024-09-18T08:18:35.699 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 7.9500000000000001e-06 2024-09-18T08:18:35.699 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:18:35.699 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:18:35.699 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-09-18T08:18:35.699 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:14:05.569099+0000", 2024-09-18T08:18:35.699 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.2755000000000001e-05 2024-09-18T08:18:35.700 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:18:35.700 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:18:35.700 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-09-18T08:18:35.700 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:14:05.569110+0000", 2024-09-18T08:18:35.700 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.1085e-05 2024-09-18T08:18:35.700 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:18:35.700 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:18:35.700 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-18T08:18:35.700 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:14:05.569176+0000", 2024-09-18T08:18:35.700 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 6.5499000000000004e-05 2024-09-18T08:18:35.700 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:18:35.700 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:18:35.700 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "waiting for peered", 2024-09-18T08:18:35.700 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:14:05.569187+0000", 2024-09-18T08:18:35.701 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.1537e-05 2024-09-18T08:18:35.701 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:18:35.701 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:18:35.701 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-18T08:18:35.701 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:14:06.588218+0000", 2024-09-18T08:18:35.701 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.0190308189999999 2024-09-18T08:18:35.701 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:18:35.701 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:18:35.701 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-09-18T08:18:35.701 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:14:06.588315+0000", 2024-09-18T08:18:35.701 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 9.7161e-05 2024-09-18T08:18:35.701 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:18:35.702 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:18:35.702 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-09-18T08:18:35.702 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:14:06.590205+0000", 2024-09-18T08:18:35.702 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0018894910000000001 2024-09-18T08:18:35.702 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-18T08:18:35.702 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-09-18T08:18:35.702 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-18T08:18:35.702 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-18T08:18:35.702 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-09-18T08:18:35.702 INFO:teuthology.orchestra.run.smithi050.stdout:} 2024-09-18T08:18:35.737 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:18:35.738 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:18:35.738 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6' 2024-09-18T08:18:35.840 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1 2024-09-18T08:18:35.863 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:18:35.864 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:18:35.865 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1' 2024-09-18T08:18:35.966 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3 2024-09-18T08:18:35.988 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:18:35.989 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:18:35.989 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3' 2024-09-18T08:18:36.030 INFO:tasks.radosbench.radosbench.0.smithi007.stdout: 1 16 2255 2239 139.924 139.938 0.001193 0.00577963 2024-09-18T08:18:36.091 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6 2024-09-18T08:18:36.120 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:18:36.122 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:18:36.122 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6' 2024-09-18T08:18:36.223 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6 2024-09-18T08:18:36.247 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:18:36.247 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:18:36.248 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6' 2024-09-18T08:18:36.278 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid 779349ca-7593-11ef-bceb-c7b262605968 -- ceph osd set nodeep-scrub 2024-09-18T08:18:36.349 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3 2024-09-18T08:18:36.399 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:18:36.400 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:18:36.400 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3' 2024-09-18T08:18:36.446 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-09-18T08:18:36.446 INFO:teuthology.orchestra.run.smithi007.stdout:{ 2024-09-18T08:18:36.446 INFO:teuthology.orchestra.run.smithi007.stdout: "epoch": 406, 2024-09-18T08:18:36.446 INFO:teuthology.orchestra.run.smithi007.stdout: "fsid": "779349ca-7593-11ef-bceb-c7b262605968", 2024-09-18T08:18:36.446 INFO:teuthology.orchestra.run.smithi007.stdout: "created": "2024-09-18T07:57:09.295710+0000", 2024-09-18T08:18:36.446 INFO:teuthology.orchestra.run.smithi007.stdout: "modified": "2024-09-18T08:18:34.424531+0000", 2024-09-18T08:18:36.447 INFO:teuthology.orchestra.run.smithi007.stdout: "last_up_change": "2024-09-18T08:14:38.391465+0000", 2024-09-18T08:18:36.447 INFO:teuthology.orchestra.run.smithi007.stdout: "last_in_change": "2024-09-18T08:18:26.612016+0000", 2024-09-18T08:18:36.447 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": "noscrub,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit", 2024-09-18T08:18:36.447 INFO:teuthology.orchestra.run.smithi007.stdout: "flags_num": 5801984, 2024-09-18T08:18:36.447 INFO:teuthology.orchestra.run.smithi007.stdout: "flags_set": [ 2024-09-18T08:18:36.447 INFO:teuthology.orchestra.run.smithi007.stdout: "noscrub", 2024-09-18T08:18:36.447 INFO:teuthology.orchestra.run.smithi007.stdout: "pglog_hardlimit", 2024-09-18T08:18:36.447 INFO:teuthology.orchestra.run.smithi007.stdout: "purged_snapdirs", 2024-09-18T08:18:36.447 INFO:teuthology.orchestra.run.smithi007.stdout: "recovery_deletes", 2024-09-18T08:18:36.447 INFO:teuthology.orchestra.run.smithi007.stdout: "sortbitwise" 2024-09-18T08:18:36.447 INFO:teuthology.orchestra.run.smithi007.stdout: ], 2024-09-18T08:18:36.447 INFO:teuthology.orchestra.run.smithi007.stdout: "crush_version": 19, 2024-09-18T08:18:36.448 INFO:teuthology.orchestra.run.smithi007.stdout: "full_ratio": 0.94999998807907104, 2024-09-18T08:18:36.448 INFO:teuthology.orchestra.run.smithi007.stdout: "backfillfull_ratio": 0.89999997615814209, 2024-09-18T08:18:36.448 INFO:teuthology.orchestra.run.smithi007.stdout: "nearfull_ratio": 0.85000002384185791, 2024-09-18T08:18:36.448 INFO:teuthology.orchestra.run.smithi007.stdout: "cluster_snapshot": "", 2024-09-18T08:18:36.448 INFO:teuthology.orchestra.run.smithi007.stdout: "pool_max": 11, 2024-09-18T08:18:36.448 INFO:teuthology.orchestra.run.smithi007.stdout: "max_osd": 8, 2024-09-18T08:18:36.448 INFO:teuthology.orchestra.run.smithi007.stdout: "require_min_compat_client": "luminous", 2024-09-18T08:18:36.448 INFO:teuthology.orchestra.run.smithi007.stdout: "min_compat_client": "luminous", 2024-09-18T08:18:36.448 INFO:teuthology.orchestra.run.smithi007.stdout: "require_osd_release": "squid", 2024-09-18T08:18:36.449 INFO:teuthology.orchestra.run.smithi007.stdout: "allow_crimson": false, 2024-09-18T08:18:36.449 INFO:teuthology.orchestra.run.smithi007.stdout: "pools": [ 2024-09-18T08:18:36.449 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:18:36.449 INFO:teuthology.orchestra.run.smithi007.stdout: "pool": 1, 2024-09-18T08:18:36.449 INFO:teuthology.orchestra.run.smithi007.stdout: "pool_name": ".mgr", 2024-09-18T08:18:36.449 INFO:teuthology.orchestra.run.smithi007.stdout: "create_time": "2024-09-18T08:00:11.802627+0000", 2024-09-18T08:18:36.449 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 1, 2024-09-18T08:18:36.449 INFO:teuthology.orchestra.run.smithi007.stdout: "flags_names": "hashpspool", 2024-09-18T08:18:36.449 INFO:teuthology.orchestra.run.smithi007.stdout: "type": 1, 2024-09-18T08:18:36.449 INFO:teuthology.orchestra.run.smithi007.stdout: "size": 3, 2024-09-18T08:18:36.449 INFO:teuthology.orchestra.run.smithi007.stdout: "min_size": 2, 2024-09-18T08:18:36.449 INFO:teuthology.orchestra.run.smithi007.stdout: "crush_rule": 0, 2024-09-18T08:18:36.449 INFO:teuthology.orchestra.run.smithi007.stdout: "peering_crush_bucket_count": 0, 2024-09-18T08:18:36.450 INFO:teuthology.orchestra.run.smithi007.stdout: "peering_crush_bucket_target": 0, 2024-09-18T08:18:36.450 INFO:teuthology.orchestra.run.smithi007.stdout: "peering_crush_bucket_barrier": 0, 2024-09-18T08:18:36.450 INFO:teuthology.orchestra.run.smithi007.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-09-18T08:18:36.450 INFO:teuthology.orchestra.run.smithi007.stdout: "is_stretch_pool": false, 2024-09-18T08:18:36.450 INFO:teuthology.orchestra.run.smithi007.stdout: "object_hash": 2, 2024-09-18T08:18:36.450 INFO:teuthology.orchestra.run.smithi007.stdout: "pg_autoscale_mode": "off", 2024-09-18T08:18:36.450 INFO:teuthology.orchestra.run.smithi007.stdout: "pg_num": 11, 2024-09-18T08:18:36.450 INFO:teuthology.orchestra.run.smithi007.stdout: "pg_placement_num": 11, 2024-09-18T08:18:36.450 INFO:teuthology.orchestra.run.smithi007.stdout: "pg_placement_num_target": 11, 2024-09-18T08:18:36.450 INFO:teuthology.orchestra.run.smithi007.stdout: "pg_num_target": 11, 2024-09-18T08:18:36.450 INFO:teuthology.orchestra.run.smithi007.stdout: "pg_num_pending": 11, 2024-09-18T08:18:36.450 INFO:teuthology.orchestra.run.smithi007.stdout: "last_pg_merge_meta": { 2024-09-18T08:18:36.450 INFO:teuthology.orchestra.run.smithi007.stdout: "source_pgid": "0.0", 2024-09-18T08:18:36.451 INFO:teuthology.orchestra.run.smithi007.stdout: "ready_epoch": 0, 2024-09-18T08:18:36.451 INFO:teuthology.orchestra.run.smithi007.stdout: "last_epoch_started": 0, 2024-09-18T08:18:36.451 INFO:teuthology.orchestra.run.smithi007.stdout: "last_epoch_clean": 0, 2024-09-18T08:18:36.451 INFO:teuthology.orchestra.run.smithi007.stdout: "source_version": "0'0", 2024-09-18T08:18:36.451 INFO:teuthology.orchestra.run.smithi007.stdout: "target_version": "0'0" 2024-09-18T08:18:36.451 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:18:36.451 INFO:teuthology.orchestra.run.smithi007.stdout: "last_change": "285", 2024-09-18T08:18:36.451 INFO:teuthology.orchestra.run.smithi007.stdout: "last_force_op_resend": "0", 2024-09-18T08:18:36.451 INFO:teuthology.orchestra.run.smithi007.stdout: "last_force_op_resend_prenautilus": "0", 2024-09-18T08:18:36.451 INFO:teuthology.orchestra.run.smithi007.stdout: "last_force_op_resend_preluminous": "152", 2024-09-18T08:18:36.451 INFO:teuthology.orchestra.run.smithi007.stdout: "auid": 0, 2024-09-18T08:18:36.451 INFO:teuthology.orchestra.run.smithi007.stdout: "snap_mode": "selfmanaged", 2024-09-18T08:18:36.451 INFO:teuthology.orchestra.run.smithi007.stdout: "snap_seq": 0, 2024-09-18T08:18:36.452 INFO:teuthology.orchestra.run.smithi007.stdout: "snap_epoch": 0, 2024-09-18T08:18:36.452 INFO:teuthology.orchestra.run.smithi007.stdout: "pool_snaps": [], 2024-09-18T08:18:36.452 INFO:teuthology.orchestra.run.smithi007.stdout: "removed_snaps": "[]", 2024-09-18T08:18:36.452 INFO:teuthology.orchestra.run.smithi007.stdout: "quota_max_bytes": 0, 2024-09-18T08:18:36.452 INFO:teuthology.orchestra.run.smithi007.stdout: "quota_max_objects": 0, 2024-09-18T08:18:36.452 INFO:teuthology.orchestra.run.smithi007.stdout: "tiers": [], 2024-09-18T08:18:36.452 INFO:teuthology.orchestra.run.smithi007.stdout: "tier_of": -1, 2024-09-18T08:18:36.452 INFO:teuthology.orchestra.run.smithi007.stdout: "read_tier": -1, 2024-09-18T08:18:36.452 INFO:teuthology.orchestra.run.smithi007.stdout: "write_tier": -1, 2024-09-18T08:18:36.452 INFO:teuthology.orchestra.run.smithi007.stdout: "cache_mode": "none", 2024-09-18T08:18:36.452 INFO:teuthology.orchestra.run.smithi007.stdout: "target_max_bytes": 0, 2024-09-18T08:18:36.452 INFO:teuthology.orchestra.run.smithi007.stdout: "target_max_objects": 0, 2024-09-18T08:18:36.453 INFO:teuthology.orchestra.run.smithi007.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-09-18T08:18:36.453 INFO:teuthology.orchestra.run.smithi007.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-09-18T08:18:36.453 INFO:teuthology.orchestra.run.smithi007.stdout: "cache_target_full_ratio_micro": 800000, 2024-09-18T08:18:36.453 INFO:teuthology.orchestra.run.smithi007.stdout: "cache_min_flush_age": 0, 2024-09-18T08:18:36.453 INFO:teuthology.orchestra.run.smithi007.stdout: "cache_min_evict_age": 0, 2024-09-18T08:18:36.453 INFO:teuthology.orchestra.run.smithi007.stdout: "erasure_code_profile": "", 2024-09-18T08:18:36.453 INFO:teuthology.orchestra.run.smithi007.stdout: "hit_set_params": { 2024-09-18T08:18:36.453 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "none" 2024-09-18T08:18:36.453 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:18:36.453 INFO:teuthology.orchestra.run.smithi007.stdout: "hit_set_period": 0, 2024-09-18T08:18:36.453 INFO:teuthology.orchestra.run.smithi007.stdout: "hit_set_count": 0, 2024-09-18T08:18:36.453 INFO:teuthology.orchestra.run.smithi007.stdout: "use_gmt_hitset": true, 2024-09-18T08:18:36.453 INFO:teuthology.orchestra.run.smithi007.stdout: "min_read_recency_for_promote": 0, 2024-09-18T08:18:36.454 INFO:teuthology.orchestra.run.smithi007.stdout: "min_write_recency_for_promote": 0, 2024-09-18T08:18:36.454 INFO:teuthology.orchestra.run.smithi007.stdout: "hit_set_grade_decay_rate": 0, 2024-09-18T08:18:36.454 INFO:teuthology.orchestra.run.smithi007.stdout: "hit_set_search_last_n": 0, 2024-09-18T08:18:36.454 INFO:teuthology.orchestra.run.smithi007.stdout: "grade_table": [], 2024-09-18T08:18:36.454 INFO:teuthology.orchestra.run.smithi007.stdout: "stripe_width": 0, 2024-09-18T08:18:36.454 INFO:teuthology.orchestra.run.smithi007.stdout: "expected_num_objects": 0, 2024-09-18T08:18:36.454 INFO:teuthology.orchestra.run.smithi007.stdout: "fast_read": false, 2024-09-18T08:18:36.454 INFO:teuthology.orchestra.run.smithi007.stdout: "options": { 2024-09-18T08:18:36.454 INFO:teuthology.orchestra.run.smithi007.stdout: "pg_num_max": 32, 2024-09-18T08:18:36.454 INFO:teuthology.orchestra.run.smithi007.stdout: "pg_num_min": 1 2024-09-18T08:18:36.454 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:18:36.454 INFO:teuthology.orchestra.run.smithi007.stdout: "application_metadata": { 2024-09-18T08:18:36.455 INFO:teuthology.orchestra.run.smithi007.stdout: "mgr": {} 2024-09-18T08:18:36.455 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:18:36.455 INFO:teuthology.orchestra.run.smithi007.stdout: "read_balance": { 2024-09-18T08:18:36.455 INFO:teuthology.orchestra.run.smithi007.stdout: "score_type": "Fair distribution", 2024-09-18T08:18:36.455 INFO:teuthology.orchestra.run.smithi007.stdout: "score_acting": 1.9099999666213989, 2024-09-18T08:18:36.455 INFO:teuthology.orchestra.run.smithi007.stdout: "score_stable": 1.9099999666213989, 2024-09-18T08:18:36.455 INFO:teuthology.orchestra.run.smithi007.stdout: "optimal_score": 1, 2024-09-18T08:18:36.455 INFO:teuthology.orchestra.run.smithi007.stdout: "raw_score_acting": 1.9099999666213989, 2024-09-18T08:18:36.455 INFO:teuthology.orchestra.run.smithi007.stdout: "raw_score_stable": 1.9099999666213989, 2024-09-18T08:18:36.455 INFO:teuthology.orchestra.run.smithi007.stdout: "primary_affinity_weighted": 0.80927085876464844, 2024-09-18T08:18:36.455 INFO:teuthology.orchestra.run.smithi007.stdout: "average_primary_affinity": 0.92000001668930054, 2024-09-18T08:18:36.455 INFO:teuthology.orchestra.run.smithi007.stdout: "average_primary_affinity_weighted": 0.81000000238418579 2024-09-18T08:18:36.455 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-09-18T08:18:36.456 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:18:36.456 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:18:36.456 INFO:teuthology.orchestra.run.smithi007.stdout: "pool": 2, 2024-09-18T08:18:36.456 INFO:teuthology.orchestra.run.smithi007.stdout: "pool_name": "datapool", 2024-09-18T08:18:36.456 INFO:teuthology.orchestra.run.smithi007.stdout: "create_time": "2024-09-18T08:02:45.374099+0000", 2024-09-18T08:18:36.456 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 8193, 2024-09-18T08:18:36.456 INFO:teuthology.orchestra.run.smithi007.stdout: "flags_names": "hashpspool,selfmanaged_snaps", 2024-09-18T08:18:36.456 INFO:teuthology.orchestra.run.smithi007.stdout: "type": 1, 2024-09-18T08:18:36.456 INFO:teuthology.orchestra.run.smithi007.stdout: "size": 3, 2024-09-18T08:18:36.456 INFO:teuthology.orchestra.run.smithi007.stdout: "min_size": 2, 2024-09-18T08:18:36.456 INFO:teuthology.orchestra.run.smithi007.stdout: "crush_rule": 0, 2024-09-18T08:18:36.456 INFO:teuthology.orchestra.run.smithi007.stdout: "peering_crush_bucket_count": 0, 2024-09-18T08:18:36.456 INFO:teuthology.orchestra.run.smithi007.stdout: "peering_crush_bucket_target": 0, 2024-09-18T08:18:36.457 INFO:teuthology.orchestra.run.smithi007.stdout: "peering_crush_bucket_barrier": 0, 2024-09-18T08:18:36.457 INFO:teuthology.orchestra.run.smithi007.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-09-18T08:18:36.457 INFO:teuthology.orchestra.run.smithi007.stdout: "is_stretch_pool": false, 2024-09-18T08:18:36.457 INFO:teuthology.orchestra.run.smithi007.stdout: "object_hash": 2, 2024-09-18T08:18:36.457 INFO:teuthology.orchestra.run.smithi007.stdout: "pg_autoscale_mode": "off", 2024-09-18T08:18:36.457 INFO:teuthology.orchestra.run.smithi007.stdout: "pg_num": 3, 2024-09-18T08:18:36.457 INFO:teuthology.orchestra.run.smithi007.stdout: "pg_placement_num": 3, 2024-09-18T08:18:36.457 INFO:teuthology.orchestra.run.smithi007.stdout: "pg_placement_num_target": 3, 2024-09-18T08:18:36.457 INFO:teuthology.orchestra.run.smithi007.stdout: "pg_num_target": 3, 2024-09-18T08:18:36.457 INFO:teuthology.orchestra.run.smithi007.stdout: "pg_num_pending": 3, 2024-09-18T08:18:36.457 INFO:teuthology.orchestra.run.smithi007.stdout: "last_pg_merge_meta": { 2024-09-18T08:18:36.457 INFO:teuthology.orchestra.run.smithi007.stdout: "source_pgid": "0.0", 2024-09-18T08:18:36.457 INFO:teuthology.orchestra.run.smithi007.stdout: "ready_epoch": 0, 2024-09-18T08:18:36.457 INFO:teuthology.orchestra.run.smithi007.stdout: "last_epoch_started": 0, 2024-09-18T08:18:36.458 INFO:teuthology.orchestra.run.smithi007.stdout: "last_epoch_clean": 0, 2024-09-18T08:18:36.458 INFO:teuthology.orchestra.run.smithi007.stdout: "source_version": "0'0", 2024-09-18T08:18:36.458 INFO:teuthology.orchestra.run.smithi007.stdout: "target_version": "0'0" 2024-09-18T08:18:36.458 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:18:36.458 INFO:teuthology.orchestra.run.smithi007.stdout: "last_change": "64", 2024-09-18T08:18:36.458 INFO:teuthology.orchestra.run.smithi007.stdout: "last_force_op_resend": "0", 2024-09-18T08:18:36.458 INFO:teuthology.orchestra.run.smithi007.stdout: "last_force_op_resend_prenautilus": "0", 2024-09-18T08:18:36.458 INFO:teuthology.orchestra.run.smithi007.stdout: "last_force_op_resend_preluminous": "0", 2024-09-18T08:18:36.458 INFO:teuthology.orchestra.run.smithi007.stdout: "auid": 0, 2024-09-18T08:18:36.458 INFO:teuthology.orchestra.run.smithi007.stdout: "snap_mode": "selfmanaged", 2024-09-18T08:18:36.458 INFO:teuthology.orchestra.run.smithi007.stdout: "snap_seq": 2, 2024-09-18T08:18:36.458 INFO:teuthology.orchestra.run.smithi007.stdout: "snap_epoch": 64, 2024-09-18T08:18:36.458 INFO:teuthology.orchestra.run.smithi007.stdout: "pool_snaps": [], 2024-09-18T08:18:36.459 INFO:teuthology.orchestra.run.smithi007.stdout: "removed_snaps": "[]", 2024-09-18T08:18:36.459 INFO:teuthology.orchestra.run.smithi007.stdout: "quota_max_bytes": 0, 2024-09-18T08:18:36.459 INFO:teuthology.orchestra.run.smithi007.stdout: "quota_max_objects": 0, 2024-09-18T08:18:36.459 INFO:teuthology.orchestra.run.smithi007.stdout: "tiers": [], 2024-09-18T08:18:36.459 INFO:teuthology.orchestra.run.smithi007.stdout: "tier_of": -1, 2024-09-18T08:18:36.459 INFO:teuthology.orchestra.run.smithi007.stdout: "read_tier": -1, 2024-09-18T08:18:36.459 INFO:teuthology.orchestra.run.smithi007.stdout: "write_tier": -1, 2024-09-18T08:18:36.459 INFO:teuthology.orchestra.run.smithi007.stdout: "cache_mode": "none", 2024-09-18T08:18:36.459 INFO:teuthology.orchestra.run.smithi007.stdout: "target_max_bytes": 0, 2024-09-18T08:18:36.459 INFO:teuthology.orchestra.run.smithi007.stdout: "target_max_objects": 0, 2024-09-18T08:18:36.459 INFO:teuthology.orchestra.run.smithi007.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-09-18T08:18:36.459 INFO:teuthology.orchestra.run.smithi007.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-09-18T08:18:36.459 INFO:teuthology.orchestra.run.smithi007.stdout: "cache_target_full_ratio_micro": 800000, 2024-09-18T08:18:36.460 INFO:teuthology.orchestra.run.smithi007.stdout: "cache_min_flush_age": 0, 2024-09-18T08:18:36.460 INFO:teuthology.orchestra.run.smithi007.stdout: "cache_min_evict_age": 0, 2024-09-18T08:18:36.460 INFO:teuthology.orchestra.run.smithi007.stdout: "erasure_code_profile": "", 2024-09-18T08:18:36.460 INFO:teuthology.orchestra.run.smithi007.stdout: "hit_set_params": { 2024-09-18T08:18:36.460 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "none" 2024-09-18T08:18:36.460 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:18:36.460 INFO:teuthology.orchestra.run.smithi007.stdout: "hit_set_period": 0, 2024-09-18T08:18:36.460 INFO:teuthology.orchestra.run.smithi007.stdout: "hit_set_count": 0, 2024-09-18T08:18:36.460 INFO:teuthology.orchestra.run.smithi007.stdout: "use_gmt_hitset": true, 2024-09-18T08:18:36.460 INFO:teuthology.orchestra.run.smithi007.stdout: "min_read_recency_for_promote": 0, 2024-09-18T08:18:36.460 INFO:teuthology.orchestra.run.smithi007.stdout: "min_write_recency_for_promote": 0, 2024-09-18T08:18:36.460 INFO:teuthology.orchestra.run.smithi007.stdout: "hit_set_grade_decay_rate": 0, 2024-09-18T08:18:36.460 INFO:teuthology.orchestra.run.smithi007.stdout: "hit_set_search_last_n": 0, 2024-09-18T08:18:36.461 INFO:teuthology.orchestra.run.smithi007.stdout: "grade_table": [], 2024-09-18T08:18:36.461 INFO:teuthology.orchestra.run.smithi007.stdout: "stripe_width": 0, 2024-09-18T08:18:36.461 INFO:teuthology.orchestra.run.smithi007.stdout: "expected_num_objects": 0, 2024-09-18T08:18:36.461 INFO:teuthology.orchestra.run.smithi007.stdout: "fast_read": false, 2024-09-18T08:18:36.461 INFO:teuthology.orchestra.run.smithi007.stdout: "options": {}, 2024-09-18T08:18:36.461 INFO:teuthology.orchestra.run.smithi007.stdout: "application_metadata": { 2024-09-18T08:18:36.461 INFO:teuthology.orchestra.run.smithi007.stdout: "rbd": {} 2024-09-18T08:18:36.461 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:18:36.461 INFO:teuthology.orchestra.run.smithi007.stdout: "read_balance": { 2024-09-18T08:18:36.461 INFO:teuthology.orchestra.run.smithi007.stdout: "score_type": "Fair distribution", 2024-09-18T08:18:36.461 INFO:teuthology.orchestra.run.smithi007.stdout: "score_acting": 2.3299999237060547, 2024-09-18T08:18:36.461 INFO:teuthology.orchestra.run.smithi007.stdout: "score_stable": 2.3299999237060547, 2024-09-18T08:18:36.461 INFO:teuthology.orchestra.run.smithi007.stdout: "optimal_score": 1, 2024-09-18T08:18:36.461 INFO:teuthology.orchestra.run.smithi007.stdout: "raw_score_acting": 2.3299999237060547, 2024-09-18T08:18:36.462 INFO:teuthology.orchestra.run.smithi007.stdout: "raw_score_stable": 2.3299999237060547, 2024-09-18T08:18:36.462 INFO:teuthology.orchestra.run.smithi007.stdout: "primary_affinity_weighted": 0.80927085876464844, 2024-09-18T08:18:36.462 INFO:teuthology.orchestra.run.smithi007.stdout: "average_primary_affinity": 0.92000001668930054, 2024-09-18T08:18:36.462 INFO:teuthology.orchestra.run.smithi007.stdout: "average_primary_affinity_weighted": 0.81000000238418579 2024-09-18T08:18:36.462 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-09-18T08:18:36.462 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:18:36.462 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:18:36.462 INFO:teuthology.orchestra.run.smithi007.stdout: "pool": 3, 2024-09-18T08:18:36.462 INFO:teuthology.orchestra.run.smithi007.stdout: "pool_name": ".rgw.root", 2024-09-18T08:18:36.462 INFO:teuthology.orchestra.run.smithi007.stdout: "create_time": "2024-09-18T08:02:46.319947+0000", 2024-09-18T08:18:36.462 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 1, 2024-09-18T08:18:36.462 INFO:teuthology.orchestra.run.smithi007.stdout: "flags_names": "hashpspool", 2024-09-18T08:18:36.462 INFO:teuthology.orchestra.run.smithi007.stdout: "type": 1, 2024-09-18T08:18:36.462 INFO:teuthology.orchestra.run.smithi007.stdout: "size": 3, 2024-09-18T08:18:36.463 INFO:teuthology.orchestra.run.smithi007.stdout: "min_size": 2, 2024-09-18T08:18:36.463 INFO:teuthology.orchestra.run.smithi007.stdout: "crush_rule": 0, 2024-09-18T08:18:36.463 INFO:teuthology.orchestra.run.smithi007.stdout: "peering_crush_bucket_count": 0, 2024-09-18T08:18:36.463 INFO:teuthology.orchestra.run.smithi007.stdout: "peering_crush_bucket_target": 0, 2024-09-18T08:18:36.463 INFO:teuthology.orchestra.run.smithi007.stdout: "peering_crush_bucket_barrier": 0, 2024-09-18T08:18:36.463 INFO:teuthology.orchestra.run.smithi007.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-09-18T08:18:36.463 INFO:teuthology.orchestra.run.smithi007.stdout: "is_stretch_pool": false, 2024-09-18T08:18:36.463 INFO:teuthology.orchestra.run.smithi007.stdout: "object_hash": 2, 2024-09-18T08:18:36.463 INFO:teuthology.orchestra.run.smithi007.stdout: "pg_autoscale_mode": "off", 2024-09-18T08:18:36.463 INFO:teuthology.orchestra.run.smithi007.stdout: "pg_num": 32, 2024-09-18T08:18:36.464 INFO:teuthology.orchestra.run.smithi007.stdout: "pg_placement_num": 32, 2024-09-18T08:18:36.464 INFO:teuthology.orchestra.run.smithi007.stdout: "pg_placement_num_target": 32, 2024-09-18T08:18:36.464 INFO:teuthology.orchestra.run.smithi007.stdout: "pg_num_target": 32, 2024-09-18T08:18:36.464 INFO:teuthology.orchestra.run.smithi007.stdout: "pg_num_pending": 32, 2024-09-18T08:18:36.464 INFO:teuthology.orchestra.run.smithi007.stdout: "last_pg_merge_meta": { 2024-09-18T08:18:36.465 INFO:teuthology.orchestra.run.smithi007.stdout: "source_pgid": "0.0", 2024-09-18T08:18:36.465 INFO:teuthology.orchestra.run.smithi007.stdout: "ready_epoch": 0, 2024-09-18T08:18:36.465 INFO:teuthology.orchestra.run.smithi007.stdout: "last_epoch_started": 0, 2024-09-18T08:18:36.465 INFO:teuthology.orchestra.run.smithi007.stdout: "last_epoch_clean": 0, 2024-09-18T08:18:36.465 INFO:teuthology.orchestra.run.smithi007.stdout: "source_version": "0'0", 2024-09-18T08:18:36.465 INFO:teuthology.orchestra.run.smithi007.stdout: "target_version": "0'0" 2024-09-18T08:18:36.466 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:18:36.466 INFO:teuthology.orchestra.run.smithi007.stdout: "last_change": "63", 2024-09-18T08:18:36.466 INFO:teuthology.orchestra.run.smithi007.stdout: "last_force_op_resend": "0", 2024-09-18T08:18:36.466 INFO:teuthology.orchestra.run.smithi007.stdout: "last_force_op_resend_prenautilus": "0", 2024-09-18T08:18:36.466 INFO:teuthology.orchestra.run.smithi007.stdout: "last_force_op_resend_preluminous": "0", 2024-09-18T08:18:36.466 INFO:teuthology.orchestra.run.smithi007.stdout: "auid": 0, 2024-09-18T08:18:36.466 INFO:teuthology.orchestra.run.smithi007.stdout: "snap_mode": "selfmanaged", 2024-09-18T08:18:36.466 INFO:teuthology.orchestra.run.smithi007.stdout: "snap_seq": 0, 2024-09-18T08:18:36.466 INFO:teuthology.orchestra.run.smithi007.stdout: "snap_epoch": 0, 2024-09-18T08:18:36.466 INFO:teuthology.orchestra.run.smithi007.stdout: "pool_snaps": [], 2024-09-18T08:18:36.466 INFO:teuthology.orchestra.run.smithi007.stdout: "removed_snaps": "[]", 2024-09-18T08:18:36.466 INFO:teuthology.orchestra.run.smithi007.stdout: "quota_max_bytes": 0, 2024-09-18T08:18:36.466 INFO:teuthology.orchestra.run.smithi007.stdout: "quota_max_objects": 0, 2024-09-18T08:18:36.467 INFO:teuthology.orchestra.run.smithi007.stdout: "tiers": [], 2024-09-18T08:18:36.467 INFO:teuthology.orchestra.run.smithi007.stdout: "tier_of": -1, 2024-09-18T08:18:36.467 INFO:teuthology.orchestra.run.smithi007.stdout: "read_tier": -1, 2024-09-18T08:18:36.467 INFO:teuthology.orchestra.run.smithi007.stdout: "write_tier": -1, 2024-09-18T08:18:36.467 INFO:teuthology.orchestra.run.smithi007.stdout: "cache_mode": "none", 2024-09-18T08:18:36.467 INFO:teuthology.orchestra.run.smithi007.stdout: "target_max_bytes": 0, 2024-09-18T08:18:36.467 INFO:teuthology.orchestra.run.smithi007.stdout: "target_max_objects": 0, 2024-09-18T08:18:36.467 INFO:teuthology.orchestra.run.smithi007.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-09-18T08:18:36.467 INFO:teuthology.orchestra.run.smithi007.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-09-18T08:18:36.467 INFO:teuthology.orchestra.run.smithi007.stdout: "cache_target_full_ratio_micro": 800000, 2024-09-18T08:18:36.467 INFO:teuthology.orchestra.run.smithi007.stdout: "cache_min_flush_age": 0, 2024-09-18T08:18:36.467 INFO:teuthology.orchestra.run.smithi007.stdout: "cache_min_evict_age": 0, 2024-09-18T08:18:36.467 INFO:teuthology.orchestra.run.smithi007.stdout: "erasure_code_profile": "", 2024-09-18T08:18:36.468 INFO:teuthology.orchestra.run.smithi007.stdout: "hit_set_params": { 2024-09-18T08:18:36.468 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "none" 2024-09-18T08:18:36.468 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:18:36.468 INFO:teuthology.orchestra.run.smithi007.stdout: "hit_set_period": 0, 2024-09-18T08:18:36.468 INFO:teuthology.orchestra.run.smithi007.stdout: "hit_set_count": 0, 2024-09-18T08:18:36.468 INFO:teuthology.orchestra.run.smithi007.stdout: "use_gmt_hitset": true, 2024-09-18T08:18:36.468 INFO:teuthology.orchestra.run.smithi007.stdout: "min_read_recency_for_promote": 0, 2024-09-18T08:18:36.468 INFO:teuthology.orchestra.run.smithi007.stdout: "min_write_recency_for_promote": 0, 2024-09-18T08:18:36.468 INFO:teuthology.orchestra.run.smithi007.stdout: "hit_set_grade_decay_rate": 0, 2024-09-18T08:18:36.468 INFO:teuthology.orchestra.run.smithi007.stdout: "hit_set_search_last_n": 0, 2024-09-18T08:18:36.468 INFO:teuthology.orchestra.run.smithi007.stdout: "grade_table": [], 2024-09-18T08:18:36.468 INFO:teuthology.orchestra.run.smithi007.stdout: "stripe_width": 0, 2024-09-18T08:18:36.468 INFO:teuthology.orchestra.run.smithi007.stdout: "expected_num_objects": 0, 2024-09-18T08:18:36.469 INFO:teuthology.orchestra.run.smithi007.stdout: "fast_read": false, 2024-09-18T08:18:36.469 INFO:teuthology.orchestra.run.smithi007.stdout: "options": {}, 2024-09-18T08:18:36.469 INFO:teuthology.orchestra.run.smithi007.stdout: "application_metadata": { 2024-09-18T08:18:36.469 INFO:teuthology.orchestra.run.smithi007.stdout: "rgw": {} 2024-09-18T08:18:36.469 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:18:36.469 INFO:teuthology.orchestra.run.smithi007.stdout: "read_balance": { 2024-09-18T08:18:36.469 INFO:teuthology.orchestra.run.smithi007.stdout: "score_type": "Fair distribution", 2024-09-18T08:18:36.469 INFO:teuthology.orchestra.run.smithi007.stdout: "score_acting": 1.9700000286102295, 2024-09-18T08:18:36.469 INFO:teuthology.orchestra.run.smithi007.stdout: "score_stable": 1.9700000286102295, 2024-09-18T08:18:36.469 INFO:teuthology.orchestra.run.smithi007.stdout: "optimal_score": 1, 2024-09-18T08:18:36.469 INFO:teuthology.orchestra.run.smithi007.stdout: "raw_score_acting": 1.9700000286102295, 2024-09-18T08:18:36.469 INFO:teuthology.orchestra.run.smithi007.stdout: "raw_score_stable": 1.9700000286102295, 2024-09-18T08:18:36.469 INFO:teuthology.orchestra.run.smithi007.stdout: "primary_affinity_weighted": 0.80927085876464844, 2024-09-18T08:18:36.470 INFO:teuthology.orchestra.run.smithi007.stdout: "average_primary_affinity": 0.92000001668930054, 2024-09-18T08:18:36.470 INFO:teuthology.orchestra.run.smithi007.stdout: "average_primary_affinity_weighted": 0.81000000238418579 2024-09-18T08:18:36.470 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-09-18T08:18:36.470 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:18:36.470 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:18:36.470 INFO:teuthology.orchestra.run.smithi007.stdout: "pool": 4, 2024-09-18T08:18:36.470 INFO:teuthology.orchestra.run.smithi007.stdout: "pool_name": "default.rgw.log", 2024-09-18T08:18:36.470 INFO:teuthology.orchestra.run.smithi007.stdout: "create_time": "2024-09-18T08:02:47.948552+0000", 2024-09-18T08:18:36.470 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 1, 2024-09-18T08:18:36.470 INFO:teuthology.orchestra.run.smithi007.stdout: "flags_names": "hashpspool", 2024-09-18T08:18:36.470 INFO:teuthology.orchestra.run.smithi007.stdout: "type": 1, 2024-09-18T08:18:36.470 INFO:teuthology.orchestra.run.smithi007.stdout: "size": 3, 2024-09-18T08:18:36.470 INFO:teuthology.orchestra.run.smithi007.stdout: "min_size": 2, 2024-09-18T08:18:36.471 INFO:teuthology.orchestra.run.smithi007.stdout: "crush_rule": 0, 2024-09-18T08:18:36.471 INFO:teuthology.orchestra.run.smithi007.stdout: "peering_crush_bucket_count": 0, 2024-09-18T08:18:36.471 INFO:teuthology.orchestra.run.smithi007.stdout: "peering_crush_bucket_target": 0, 2024-09-18T08:18:36.471 INFO:teuthology.orchestra.run.smithi007.stdout: "peering_crush_bucket_barrier": 0, 2024-09-18T08:18:36.471 INFO:teuthology.orchestra.run.smithi007.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-09-18T08:18:36.471 INFO:teuthology.orchestra.run.smithi007.stdout: "is_stretch_pool": false, 2024-09-18T08:18:36.471 INFO:teuthology.orchestra.run.smithi007.stdout: "object_hash": 2, 2024-09-18T08:18:36.471 INFO:teuthology.orchestra.run.smithi007.stdout: "pg_autoscale_mode": "off", 2024-09-18T08:18:36.471 INFO:teuthology.orchestra.run.smithi007.stdout: "pg_num": 32, 2024-09-18T08:18:36.471 INFO:teuthology.orchestra.run.smithi007.stdout: "pg_placement_num": 32, 2024-09-18T08:18:36.471 INFO:teuthology.orchestra.run.smithi007.stdout: "pg_placement_num_target": 32, 2024-09-18T08:18:36.471 INFO:teuthology.orchestra.run.smithi007.stdout: "pg_num_target": 32, 2024-09-18T08:18:36.471 INFO:teuthology.orchestra.run.smithi007.stdout: "pg_num_pending": 32, 2024-09-18T08:18:36.472 INFO:teuthology.orchestra.run.smithi007.stdout: "last_pg_merge_meta": { 2024-09-18T08:18:36.472 INFO:teuthology.orchestra.run.smithi007.stdout: "source_pgid": "0.0", 2024-09-18T08:18:36.472 INFO:teuthology.orchestra.run.smithi007.stdout: "ready_epoch": 0, 2024-09-18T08:18:36.472 INFO:teuthology.orchestra.run.smithi007.stdout: "last_epoch_started": 0, 2024-09-18T08:18:36.472 INFO:teuthology.orchestra.run.smithi007.stdout: "last_epoch_clean": 0, 2024-09-18T08:18:36.472 INFO:teuthology.orchestra.run.smithi007.stdout: "source_version": "0'0", 2024-09-18T08:18:36.472 INFO:teuthology.orchestra.run.smithi007.stdout: "target_version": "0'0" 2024-09-18T08:18:36.472 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:18:36.472 INFO:teuthology.orchestra.run.smithi007.stdout: "last_change": "65", 2024-09-18T08:18:36.472 INFO:teuthology.orchestra.run.smithi007.stdout: "last_force_op_resend": "0", 2024-09-18T08:18:36.472 INFO:teuthology.orchestra.run.smithi007.stdout: "last_force_op_resend_prenautilus": "0", 2024-09-18T08:18:36.472 INFO:teuthology.orchestra.run.smithi007.stdout: "last_force_op_resend_preluminous": "0", 2024-09-18T08:18:36.472 INFO:teuthology.orchestra.run.smithi007.stdout: "auid": 0, 2024-09-18T08:18:36.473 INFO:teuthology.orchestra.run.smithi007.stdout: "snap_mode": "selfmanaged", 2024-09-18T08:18:36.473 INFO:teuthology.orchestra.run.smithi007.stdout: "snap_seq": 0, 2024-09-18T08:18:36.473 INFO:teuthology.orchestra.run.smithi007.stdout: "snap_epoch": 0, 2024-09-18T08:18:36.473 INFO:teuthology.orchestra.run.smithi007.stdout: "pool_snaps": [], 2024-09-18T08:18:36.473 INFO:teuthology.orchestra.run.smithi007.stdout: "removed_snaps": "[]", 2024-09-18T08:18:36.473 INFO:teuthology.orchestra.run.smithi007.stdout: "quota_max_bytes": 0, 2024-09-18T08:18:36.473 INFO:teuthology.orchestra.run.smithi007.stdout: "quota_max_objects": 0, 2024-09-18T08:18:36.473 INFO:teuthology.orchestra.run.smithi007.stdout: "tiers": [], 2024-09-18T08:18:36.473 INFO:teuthology.orchestra.run.smithi007.stdout: "tier_of": -1, 2024-09-18T08:18:36.473 INFO:teuthology.orchestra.run.smithi007.stdout: "read_tier": -1, 2024-09-18T08:18:36.473 INFO:teuthology.orchestra.run.smithi007.stdout: "write_tier": -1, 2024-09-18T08:18:36.473 INFO:teuthology.orchestra.run.smithi007.stdout: "cache_mode": "none", 2024-09-18T08:18:36.473 INFO:teuthology.orchestra.run.smithi007.stdout: "target_max_bytes": 0, 2024-09-18T08:18:36.474 INFO:teuthology.orchestra.run.smithi007.stdout: "target_max_objects": 0, 2024-09-18T08:18:36.474 INFO:teuthology.orchestra.run.smithi007.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-09-18T08:18:36.474 INFO:teuthology.orchestra.run.smithi007.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-09-18T08:18:36.474 INFO:teuthology.orchestra.run.smithi007.stdout: "cache_target_full_ratio_micro": 800000, 2024-09-18T08:18:36.474 INFO:teuthology.orchestra.run.smithi007.stdout: "cache_min_flush_age": 0, 2024-09-18T08:18:36.474 INFO:teuthology.orchestra.run.smithi007.stdout: "cache_min_evict_age": 0, 2024-09-18T08:18:36.474 INFO:teuthology.orchestra.run.smithi007.stdout: "erasure_code_profile": "", 2024-09-18T08:18:36.474 INFO:teuthology.orchestra.run.smithi007.stdout: "hit_set_params": { 2024-09-18T08:18:36.474 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "none" 2024-09-18T08:18:36.474 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:18:36.474 INFO:teuthology.orchestra.run.smithi007.stdout: "hit_set_period": 0, 2024-09-18T08:18:36.475 INFO:teuthology.orchestra.run.smithi007.stdout: "hit_set_count": 0, 2024-09-18T08:18:36.475 INFO:teuthology.orchestra.run.smithi007.stdout: "use_gmt_hitset": true, 2024-09-18T08:18:36.475 INFO:teuthology.orchestra.run.smithi007.stdout: "min_read_recency_for_promote": 0, 2024-09-18T08:18:36.475 INFO:teuthology.orchestra.run.smithi007.stdout: "min_write_recency_for_promote": 0, 2024-09-18T08:18:36.475 INFO:teuthology.orchestra.run.smithi007.stdout: "hit_set_grade_decay_rate": 0, 2024-09-18T08:18:36.475 INFO:teuthology.orchestra.run.smithi007.stdout: "hit_set_search_last_n": 0, 2024-09-18T08:18:36.475 INFO:teuthology.orchestra.run.smithi007.stdout: "grade_table": [], 2024-09-18T08:18:36.475 INFO:teuthology.orchestra.run.smithi007.stdout: "stripe_width": 0, 2024-09-18T08:18:36.475 INFO:teuthology.orchestra.run.smithi007.stdout: "expected_num_objects": 0, 2024-09-18T08:18:36.475 INFO:teuthology.orchestra.run.smithi007.stdout: "fast_read": false, 2024-09-18T08:18:36.475 INFO:teuthology.orchestra.run.smithi007.stdout: "options": {}, 2024-09-18T08:18:36.475 INFO:teuthology.orchestra.run.smithi007.stdout: "application_metadata": { 2024-09-18T08:18:36.475 INFO:teuthology.orchestra.run.smithi007.stdout: "rgw": {} 2024-09-18T08:18:36.476 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:18:36.476 INFO:teuthology.orchestra.run.smithi007.stdout: "read_balance": { 2024-09-18T08:18:36.476 INFO:teuthology.orchestra.run.smithi007.stdout: "score_type": "Fair distribution", 2024-09-18T08:18:36.476 INFO:teuthology.orchestra.run.smithi007.stdout: "score_acting": 1.9700000286102295, 2024-09-18T08:18:36.476 INFO:teuthology.orchestra.run.smithi007.stdout: "score_stable": 1.9700000286102295, 2024-09-18T08:18:36.476 INFO:teuthology.orchestra.run.smithi007.stdout: "optimal_score": 1, 2024-09-18T08:18:36.476 INFO:teuthology.orchestra.run.smithi007.stdout: "raw_score_acting": 1.9700000286102295, 2024-09-18T08:18:36.476 INFO:teuthology.orchestra.run.smithi007.stdout: "raw_score_stable": 1.9700000286102295, 2024-09-18T08:18:36.476 INFO:teuthology.orchestra.run.smithi007.stdout: "primary_affinity_weighted": 0.80927085876464844, 2024-09-18T08:18:36.476 INFO:teuthology.orchestra.run.smithi007.stdout: "average_primary_affinity": 0.92000001668930054, 2024-09-18T08:18:36.476 INFO:teuthology.orchestra.run.smithi007.stdout: "average_primary_affinity_weighted": 0.81000000238418579 2024-09-18T08:18:36.476 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-09-18T08:18:36.477 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:18:36.477 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:18:36.477 INFO:teuthology.orchestra.run.smithi007.stdout: "pool": 5, 2024-09-18T08:18:36.477 INFO:teuthology.orchestra.run.smithi007.stdout: "pool_name": "default.rgw.control", 2024-09-18T08:18:36.477 INFO:teuthology.orchestra.run.smithi007.stdout: "create_time": "2024-09-18T08:02:49.904302+0000", 2024-09-18T08:18:36.477 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 1, 2024-09-18T08:18:36.477 INFO:teuthology.orchestra.run.smithi007.stdout: "flags_names": "hashpspool", 2024-09-18T08:18:36.477 INFO:teuthology.orchestra.run.smithi007.stdout: "type": 1, 2024-09-18T08:18:36.477 INFO:teuthology.orchestra.run.smithi007.stdout: "size": 3, 2024-09-18T08:18:36.477 INFO:teuthology.orchestra.run.smithi007.stdout: "min_size": 2, 2024-09-18T08:18:36.477 INFO:teuthology.orchestra.run.smithi007.stdout: "crush_rule": 0, 2024-09-18T08:18:36.477 INFO:teuthology.orchestra.run.smithi007.stdout: "peering_crush_bucket_count": 0, 2024-09-18T08:18:36.477 INFO:teuthology.orchestra.run.smithi007.stdout: "peering_crush_bucket_target": 0, 2024-09-18T08:18:36.477 INFO:teuthology.orchestra.run.smithi007.stdout: "peering_crush_bucket_barrier": 0, 2024-09-18T08:18:36.478 INFO:teuthology.orchestra.run.smithi007.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-09-18T08:18:36.478 INFO:teuthology.orchestra.run.smithi007.stdout: "is_stretch_pool": false, 2024-09-18T08:18:36.478 INFO:teuthology.orchestra.run.smithi007.stdout: "object_hash": 2, 2024-09-18T08:18:36.478 INFO:teuthology.orchestra.run.smithi007.stdout: "pg_autoscale_mode": "off", 2024-09-18T08:18:36.478 INFO:teuthology.orchestra.run.smithi007.stdout: "pg_num": 32, 2024-09-18T08:18:36.478 INFO:teuthology.orchestra.run.smithi007.stdout: "pg_placement_num": 32, 2024-09-18T08:18:36.478 INFO:teuthology.orchestra.run.smithi007.stdout: "pg_placement_num_target": 32, 2024-09-18T08:18:36.478 INFO:teuthology.orchestra.run.smithi007.stdout: "pg_num_target": 32, 2024-09-18T08:18:36.478 INFO:teuthology.orchestra.run.smithi007.stdout: "pg_num_pending": 32, 2024-09-18T08:18:36.478 INFO:teuthology.orchestra.run.smithi007.stdout: "last_pg_merge_meta": { 2024-09-18T08:18:36.478 INFO:teuthology.orchestra.run.smithi007.stdout: "source_pgid": "0.0", 2024-09-18T08:18:36.478 INFO:teuthology.orchestra.run.smithi007.stdout: "ready_epoch": 0, 2024-09-18T08:18:36.478 INFO:teuthology.orchestra.run.smithi007.stdout: "last_epoch_started": 0, 2024-09-18T08:18:36.479 INFO:teuthology.orchestra.run.smithi007.stdout: "last_epoch_clean": 0, 2024-09-18T08:18:36.479 INFO:teuthology.orchestra.run.smithi007.stdout: "source_version": "0'0", 2024-09-18T08:18:36.479 INFO:teuthology.orchestra.run.smithi007.stdout: "target_version": "0'0" 2024-09-18T08:18:36.479 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:18:36.479 INFO:teuthology.orchestra.run.smithi007.stdout: "last_change": "67", 2024-09-18T08:18:36.479 INFO:teuthology.orchestra.run.smithi007.stdout: "last_force_op_resend": "0", 2024-09-18T08:18:36.479 INFO:teuthology.orchestra.run.smithi007.stdout: "last_force_op_resend_prenautilus": "0", 2024-09-18T08:18:36.479 INFO:teuthology.orchestra.run.smithi007.stdout: "last_force_op_resend_preluminous": "0", 2024-09-18T08:18:36.479 INFO:teuthology.orchestra.run.smithi007.stdout: "auid": 0, 2024-09-18T08:18:36.480 INFO:teuthology.orchestra.run.smithi007.stdout: "snap_mode": "selfmanaged", 2024-09-18T08:18:36.480 INFO:teuthology.orchestra.run.smithi007.stdout: "snap_seq": 0, 2024-09-18T08:18:36.480 INFO:teuthology.orchestra.run.smithi007.stdout: "snap_epoch": 0, 2024-09-18T08:18:36.480 INFO:teuthology.orchestra.run.smithi007.stdout: "pool_snaps": [], 2024-09-18T08:18:36.480 INFO:teuthology.orchestra.run.smithi007.stdout: "removed_snaps": "[]", 2024-09-18T08:18:36.480 INFO:teuthology.orchestra.run.smithi007.stdout: "quota_max_bytes": 0, 2024-09-18T08:18:36.480 INFO:teuthology.orchestra.run.smithi007.stdout: "quota_max_objects": 0, 2024-09-18T08:18:36.480 INFO:teuthology.orchestra.run.smithi007.stdout: "tiers": [], 2024-09-18T08:18:36.480 INFO:teuthology.orchestra.run.smithi007.stdout: "tier_of": -1, 2024-09-18T08:18:36.480 INFO:teuthology.orchestra.run.smithi007.stdout: "read_tier": -1, 2024-09-18T08:18:36.480 INFO:teuthology.orchestra.run.smithi007.stdout: "write_tier": -1, 2024-09-18T08:18:36.480 INFO:teuthology.orchestra.run.smithi007.stdout: "cache_mode": "none", 2024-09-18T08:18:36.480 INFO:teuthology.orchestra.run.smithi007.stdout: "target_max_bytes": 0, 2024-09-18T08:18:36.481 INFO:teuthology.orchestra.run.smithi007.stdout: "target_max_objects": 0, 2024-09-18T08:18:36.481 INFO:teuthology.orchestra.run.smithi007.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-09-18T08:18:36.481 INFO:teuthology.orchestra.run.smithi007.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-09-18T08:18:36.481 INFO:teuthology.orchestra.run.smithi007.stdout: "cache_target_full_ratio_micro": 800000, 2024-09-18T08:18:36.481 INFO:teuthology.orchestra.run.smithi007.stdout: "cache_min_flush_age": 0, 2024-09-18T08:18:36.481 INFO:teuthology.orchestra.run.smithi007.stdout: "cache_min_evict_age": 0, 2024-09-18T08:18:36.481 INFO:teuthology.orchestra.run.smithi007.stdout: "erasure_code_profile": "", 2024-09-18T08:18:36.481 INFO:teuthology.orchestra.run.smithi007.stdout: "hit_set_params": { 2024-09-18T08:18:36.481 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "none" 2024-09-18T08:18:36.481 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:18:36.482 INFO:teuthology.orchestra.run.smithi007.stdout: "hit_set_period": 0, 2024-09-18T08:18:36.482 INFO:teuthology.orchestra.run.smithi007.stdout: "hit_set_count": 0, 2024-09-18T08:18:36.482 INFO:teuthology.orchestra.run.smithi007.stdout: "use_gmt_hitset": true, 2024-09-18T08:18:36.482 INFO:teuthology.orchestra.run.smithi007.stdout: "min_read_recency_for_promote": 0, 2024-09-18T08:18:36.482 INFO:teuthology.orchestra.run.smithi007.stdout: "min_write_recency_for_promote": 0, 2024-09-18T08:18:36.482 INFO:teuthology.orchestra.run.smithi007.stdout: "hit_set_grade_decay_rate": 0, 2024-09-18T08:18:36.482 INFO:teuthology.orchestra.run.smithi007.stdout: "hit_set_search_last_n": 0, 2024-09-18T08:18:36.482 INFO:teuthology.orchestra.run.smithi007.stdout: "grade_table": [], 2024-09-18T08:18:36.482 INFO:teuthology.orchestra.run.smithi007.stdout: "stripe_width": 0, 2024-09-18T08:18:36.482 INFO:teuthology.orchestra.run.smithi007.stdout: "expected_num_objects": 0, 2024-09-18T08:18:36.482 INFO:teuthology.orchestra.run.smithi007.stdout: "fast_read": false, 2024-09-18T08:18:36.482 INFO:teuthology.orchestra.run.smithi007.stdout: "options": {}, 2024-09-18T08:18:36.483 INFO:teuthology.orchestra.run.smithi007.stdout: "application_metadata": { 2024-09-18T08:18:36.483 INFO:teuthology.orchestra.run.smithi007.stdout: "rgw": {} 2024-09-18T08:18:36.483 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:18:36.483 INFO:teuthology.orchestra.run.smithi007.stdout: "read_balance": { 2024-09-18T08:18:36.483 INFO:teuthology.orchestra.run.smithi007.stdout: "score_type": "Fair distribution", 2024-09-18T08:18:36.483 INFO:teuthology.orchestra.run.smithi007.stdout: "score_acting": 1.309999942779541, 2024-09-18T08:18:36.483 INFO:teuthology.orchestra.run.smithi007.stdout: "score_stable": 1.309999942779541, 2024-09-18T08:18:36.483 INFO:teuthology.orchestra.run.smithi007.stdout: "optimal_score": 1, 2024-09-18T08:18:36.483 INFO:teuthology.orchestra.run.smithi007.stdout: "raw_score_acting": 1.309999942779541, 2024-09-18T08:18:36.483 INFO:teuthology.orchestra.run.smithi007.stdout: "raw_score_stable": 1.309999942779541, 2024-09-18T08:18:36.483 INFO:teuthology.orchestra.run.smithi007.stdout: "primary_affinity_weighted": 0.80927085876464844, 2024-09-18T08:18:36.483 INFO:teuthology.orchestra.run.smithi007.stdout: "average_primary_affinity": 0.92000001668930054, 2024-09-18T08:18:36.483 INFO:teuthology.orchestra.run.smithi007.stdout: "average_primary_affinity_weighted": 0.81000000238418579 2024-09-18T08:18:36.484 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-09-18T08:18:36.484 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:18:36.484 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:18:36.484 INFO:teuthology.orchestra.run.smithi007.stdout: "pool": 6, 2024-09-18T08:18:36.484 INFO:teuthology.orchestra.run.smithi007.stdout: "pool_name": "default.rgw.meta", 2024-09-18T08:18:36.484 INFO:teuthology.orchestra.run.smithi007.stdout: "create_time": "2024-09-18T08:02:51.971465+0000", 2024-09-18T08:18:36.484 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 1, 2024-09-18T08:18:36.484 INFO:teuthology.orchestra.run.smithi007.stdout: "flags_names": "hashpspool", 2024-09-18T08:18:36.484 INFO:teuthology.orchestra.run.smithi007.stdout: "type": 1, 2024-09-18T08:18:36.484 INFO:teuthology.orchestra.run.smithi007.stdout: "size": 3, 2024-09-18T08:18:36.484 INFO:teuthology.orchestra.run.smithi007.stdout: "min_size": 2, 2024-09-18T08:18:36.484 INFO:teuthology.orchestra.run.smithi007.stdout: "crush_rule": 0, 2024-09-18T08:18:36.485 INFO:teuthology.orchestra.run.smithi007.stdout: "peering_crush_bucket_count": 0, 2024-09-18T08:18:36.485 INFO:teuthology.orchestra.run.smithi007.stdout: "peering_crush_bucket_target": 0, 2024-09-18T08:18:36.485 INFO:teuthology.orchestra.run.smithi007.stdout: "peering_crush_bucket_barrier": 0, 2024-09-18T08:18:36.485 INFO:teuthology.orchestra.run.smithi007.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-09-18T08:18:36.485 INFO:teuthology.orchestra.run.smithi007.stdout: "is_stretch_pool": false, 2024-09-18T08:18:36.485 INFO:teuthology.orchestra.run.smithi007.stdout: "object_hash": 2, 2024-09-18T08:18:36.485 INFO:teuthology.orchestra.run.smithi007.stdout: "pg_autoscale_mode": "off", 2024-09-18T08:18:36.485 INFO:teuthology.orchestra.run.smithi007.stdout: "pg_num": 32, 2024-09-18T08:18:36.485 INFO:teuthology.orchestra.run.smithi007.stdout: "pg_placement_num": 32, 2024-09-18T08:18:36.485 INFO:teuthology.orchestra.run.smithi007.stdout: "pg_placement_num_target": 32, 2024-09-18T08:18:36.485 INFO:teuthology.orchestra.run.smithi007.stdout: "pg_num_target": 32, 2024-09-18T08:18:36.485 INFO:teuthology.orchestra.run.smithi007.stdout: "pg_num_pending": 32, 2024-09-18T08:18:36.485 INFO:teuthology.orchestra.run.smithi007.stdout: "last_pg_merge_meta": { 2024-09-18T08:18:36.486 INFO:teuthology.orchestra.run.smithi007.stdout: "source_pgid": "0.0", 2024-09-18T08:18:36.486 INFO:teuthology.orchestra.run.smithi007.stdout: "ready_epoch": 0, 2024-09-18T08:18:36.486 INFO:teuthology.orchestra.run.smithi007.stdout: "last_epoch_started": 0, 2024-09-18T08:18:36.486 INFO:teuthology.orchestra.run.smithi007.stdout: "last_epoch_clean": 0, 2024-09-18T08:18:36.486 INFO:teuthology.orchestra.run.smithi007.stdout: "source_version": "0'0", 2024-09-18T08:18:36.486 INFO:teuthology.orchestra.run.smithi007.stdout: "target_version": "0'0" 2024-09-18T08:18:36.486 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:18:36.486 INFO:teuthology.orchestra.run.smithi007.stdout: "last_change": "70", 2024-09-18T08:18:36.486 INFO:teuthology.orchestra.run.smithi007.stdout: "last_force_op_resend": "0", 2024-09-18T08:18:36.486 INFO:teuthology.orchestra.run.smithi007.stdout: "last_force_op_resend_prenautilus": "0", 2024-09-18T08:18:36.486 INFO:teuthology.orchestra.run.smithi007.stdout: "last_force_op_resend_preluminous": "0", 2024-09-18T08:18:36.486 INFO:teuthology.orchestra.run.smithi007.stdout: "auid": 0, 2024-09-18T08:18:36.486 INFO:teuthology.orchestra.run.smithi007.stdout: "snap_mode": "selfmanaged", 2024-09-18T08:18:36.487 INFO:teuthology.orchestra.run.smithi007.stdout: "snap_seq": 0, 2024-09-18T08:18:36.487 INFO:teuthology.orchestra.run.smithi007.stdout: "snap_epoch": 0, 2024-09-18T08:18:36.487 INFO:teuthology.orchestra.run.smithi007.stdout: "pool_snaps": [], 2024-09-18T08:18:36.487 INFO:teuthology.orchestra.run.smithi007.stdout: "removed_snaps": "[]", 2024-09-18T08:18:36.487 INFO:teuthology.orchestra.run.smithi007.stdout: "quota_max_bytes": 0, 2024-09-18T08:18:36.487 INFO:teuthology.orchestra.run.smithi007.stdout: "quota_max_objects": 0, 2024-09-18T08:18:36.487 INFO:teuthology.orchestra.run.smithi007.stdout: "tiers": [], 2024-09-18T08:18:36.487 INFO:teuthology.orchestra.run.smithi007.stdout: "tier_of": -1, 2024-09-18T08:18:36.487 INFO:teuthology.orchestra.run.smithi007.stdout: "read_tier": -1, 2024-09-18T08:18:36.487 INFO:teuthology.orchestra.run.smithi007.stdout: "write_tier": -1, 2024-09-18T08:18:36.487 INFO:teuthology.orchestra.run.smithi007.stdout: "cache_mode": "none", 2024-09-18T08:18:36.487 INFO:teuthology.orchestra.run.smithi007.stdout: "target_max_bytes": 0, 2024-09-18T08:18:36.487 INFO:teuthology.orchestra.run.smithi007.stdout: "target_max_objects": 0, 2024-09-18T08:18:36.488 INFO:teuthology.orchestra.run.smithi007.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-09-18T08:18:36.488 INFO:teuthology.orchestra.run.smithi007.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-09-18T08:18:36.488 INFO:teuthology.orchestra.run.smithi007.stdout: "cache_target_full_ratio_micro": 800000, 2024-09-18T08:18:36.488 INFO:teuthology.orchestra.run.smithi007.stdout: "cache_min_flush_age": 0, 2024-09-18T08:18:36.488 INFO:teuthology.orchestra.run.smithi007.stdout: "cache_min_evict_age": 0, 2024-09-18T08:18:36.488 INFO:teuthology.orchestra.run.smithi007.stdout: "erasure_code_profile": "", 2024-09-18T08:18:36.488 INFO:teuthology.orchestra.run.smithi007.stdout: "hit_set_params": { 2024-09-18T08:18:36.488 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "none" 2024-09-18T08:18:36.488 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:18:36.488 INFO:teuthology.orchestra.run.smithi007.stdout: "hit_set_period": 0, 2024-09-18T08:18:36.488 INFO:teuthology.orchestra.run.smithi007.stdout: "hit_set_count": 0, 2024-09-18T08:18:36.488 INFO:teuthology.orchestra.run.smithi007.stdout: "use_gmt_hitset": true, 2024-09-18T08:18:36.488 INFO:teuthology.orchestra.run.smithi007.stdout: "min_read_recency_for_promote": 0, 2024-09-18T08:18:36.489 INFO:teuthology.orchestra.run.smithi007.stdout: "min_write_recency_for_promote": 0, 2024-09-18T08:18:36.489 INFO:teuthology.orchestra.run.smithi007.stdout: "hit_set_grade_decay_rate": 0, 2024-09-18T08:18:36.489 INFO:teuthology.orchestra.run.smithi007.stdout: "hit_set_search_last_n": 0, 2024-09-18T08:18:36.489 INFO:teuthology.orchestra.run.smithi007.stdout: "grade_table": [], 2024-09-18T08:18:36.489 INFO:teuthology.orchestra.run.smithi007.stdout: "stripe_width": 0, 2024-09-18T08:18:36.489 INFO:teuthology.orchestra.run.smithi007.stdout: "expected_num_objects": 0, 2024-09-18T08:18:36.489 INFO:teuthology.orchestra.run.smithi007.stdout: "fast_read": false, 2024-09-18T08:18:36.489 INFO:teuthology.orchestra.run.smithi007.stdout: "options": { 2024-09-18T08:18:36.489 INFO:teuthology.orchestra.run.smithi007.stdout: "pg_autoscale_bias": 4 2024-09-18T08:18:36.489 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:18:36.489 INFO:teuthology.orchestra.run.smithi007.stdout: "application_metadata": { 2024-09-18T08:18:36.489 INFO:teuthology.orchestra.run.smithi007.stdout: "rgw": {} 2024-09-18T08:18:36.490 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:18:36.490 INFO:teuthology.orchestra.run.smithi007.stdout: "read_balance": { 2024-09-18T08:18:36.490 INFO:teuthology.orchestra.run.smithi007.stdout: "score_type": "Fair distribution", 2024-09-18T08:18:36.490 INFO:teuthology.orchestra.run.smithi007.stdout: "score_acting": 1.9700000286102295, 2024-09-18T08:18:36.490 INFO:teuthology.orchestra.run.smithi007.stdout: "score_stable": 1.9700000286102295, 2024-09-18T08:18:36.490 INFO:teuthology.orchestra.run.smithi007.stdout: "optimal_score": 1, 2024-09-18T08:18:36.490 INFO:teuthology.orchestra.run.smithi007.stdout: "raw_score_acting": 1.9700000286102295, 2024-09-18T08:18:36.490 INFO:teuthology.orchestra.run.smithi007.stdout: "raw_score_stable": 1.9700000286102295, 2024-09-18T08:18:36.490 INFO:teuthology.orchestra.run.smithi007.stdout: "primary_affinity_weighted": 0.80927085876464844, 2024-09-18T08:18:36.490 INFO:teuthology.orchestra.run.smithi007.stdout: "average_primary_affinity": 0.92000001668930054, 2024-09-18T08:18:36.490 INFO:teuthology.orchestra.run.smithi007.stdout: "average_primary_affinity_weighted": 0.81000000238418579 2024-09-18T08:18:36.490 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-09-18T08:18:36.491 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:18:36.491 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:18:36.491 INFO:teuthology.orchestra.run.smithi007.stdout: "pool": 11, 2024-09-18T08:18:36.491 INFO:teuthology.orchestra.run.smithi007.stdout: "pool_name": "unique_pool_4", 2024-09-18T08:18:36.491 INFO:teuthology.orchestra.run.smithi007.stdout: "create_time": "2024-09-18T08:18:25.969782+0000", 2024-09-18T08:18:36.491 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 1, 2024-09-18T08:18:36.491 INFO:teuthology.orchestra.run.smithi007.stdout: "flags_names": "hashpspool", 2024-09-18T08:18:36.491 INFO:teuthology.orchestra.run.smithi007.stdout: "type": 1, 2024-09-18T08:18:36.491 INFO:teuthology.orchestra.run.smithi007.stdout: "size": 3, 2024-09-18T08:18:36.491 INFO:teuthology.orchestra.run.smithi007.stdout: "min_size": 2, 2024-09-18T08:18:36.491 INFO:teuthology.orchestra.run.smithi007.stdout: "crush_rule": 0, 2024-09-18T08:18:36.491 INFO:teuthology.orchestra.run.smithi007.stdout: "peering_crush_bucket_count": 0, 2024-09-18T08:18:36.492 INFO:teuthology.orchestra.run.smithi007.stdout: "peering_crush_bucket_target": 0, 2024-09-18T08:18:36.492 INFO:teuthology.orchestra.run.smithi007.stdout: "peering_crush_bucket_barrier": 0, 2024-09-18T08:18:36.492 INFO:teuthology.orchestra.run.smithi007.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-09-18T08:18:36.492 INFO:teuthology.orchestra.run.smithi007.stdout: "is_stretch_pool": false, 2024-09-18T08:18:36.492 INFO:teuthology.orchestra.run.smithi007.stdout: "object_hash": 2, 2024-09-18T08:18:36.492 INFO:teuthology.orchestra.run.smithi007.stdout: "pg_autoscale_mode": "off", 2024-09-18T08:18:36.492 INFO:teuthology.orchestra.run.smithi007.stdout: "pg_num": 16, 2024-09-18T08:18:36.492 INFO:teuthology.orchestra.run.smithi007.stdout: "pg_placement_num": 16, 2024-09-18T08:18:36.492 INFO:teuthology.orchestra.run.smithi007.stdout: "pg_placement_num_target": 16, 2024-09-18T08:18:36.492 INFO:teuthology.orchestra.run.smithi007.stdout: "pg_num_target": 16, 2024-09-18T08:18:36.492 INFO:teuthology.orchestra.run.smithi007.stdout: "pg_num_pending": 16, 2024-09-18T08:18:36.492 INFO:teuthology.orchestra.run.smithi007.stdout: "last_pg_merge_meta": { 2024-09-18T08:18:36.493 INFO:teuthology.orchestra.run.smithi007.stdout: "source_pgid": "0.0", 2024-09-18T08:18:36.493 INFO:teuthology.orchestra.run.smithi007.stdout: "ready_epoch": 0, 2024-09-18T08:18:36.493 INFO:teuthology.orchestra.run.smithi007.stdout: "last_epoch_started": 0, 2024-09-18T08:18:36.493 INFO:teuthology.orchestra.run.smithi007.stdout: "last_epoch_clean": 0, 2024-09-18T08:18:36.493 INFO:teuthology.orchestra.run.smithi007.stdout: "source_version": "0'0", 2024-09-18T08:18:36.493 INFO:teuthology.orchestra.run.smithi007.stdout: "target_version": "0'0" 2024-09-18T08:18:36.493 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:18:36.493 INFO:teuthology.orchestra.run.smithi007.stdout: "last_change": "405", 2024-09-18T08:18:36.493 INFO:teuthology.orchestra.run.smithi007.stdout: "last_force_op_resend": "0", 2024-09-18T08:18:36.493 INFO:teuthology.orchestra.run.smithi007.stdout: "last_force_op_resend_prenautilus": "0", 2024-09-18T08:18:36.493 INFO:teuthology.orchestra.run.smithi007.stdout: "last_force_op_resend_preluminous": "0", 2024-09-18T08:18:36.493 INFO:teuthology.orchestra.run.smithi007.stdout: "auid": 0, 2024-09-18T08:18:36.493 INFO:teuthology.orchestra.run.smithi007.stdout: "snap_mode": "selfmanaged", 2024-09-18T08:18:36.493 INFO:teuthology.orchestra.run.smithi007.stdout: "snap_seq": 0, 2024-09-18T08:18:36.494 INFO:teuthology.orchestra.run.smithi007.stdout: "snap_epoch": 0, 2024-09-18T08:18:36.494 INFO:teuthology.orchestra.run.smithi007.stdout: "pool_snaps": [], 2024-09-18T08:18:36.494 INFO:teuthology.orchestra.run.smithi007.stdout: "removed_snaps": "[]", 2024-09-18T08:18:36.494 INFO:teuthology.orchestra.run.smithi007.stdout: "quota_max_bytes": 0, 2024-09-18T08:18:36.494 INFO:teuthology.orchestra.run.smithi007.stdout: "quota_max_objects": 0, 2024-09-18T08:18:36.494 INFO:teuthology.orchestra.run.smithi007.stdout: "tiers": [], 2024-09-18T08:18:36.494 INFO:teuthology.orchestra.run.smithi007.stdout: "tier_of": -1, 2024-09-18T08:18:36.494 INFO:teuthology.orchestra.run.smithi007.stdout: "read_tier": -1, 2024-09-18T08:18:36.494 INFO:teuthology.orchestra.run.smithi007.stdout: "write_tier": -1, 2024-09-18T08:18:36.494 INFO:teuthology.orchestra.run.smithi007.stdout: "cache_mode": "none", 2024-09-18T08:18:36.494 INFO:teuthology.orchestra.run.smithi007.stdout: "target_max_bytes": 0, 2024-09-18T08:18:36.494 INFO:teuthology.orchestra.run.smithi007.stdout: "target_max_objects": 0, 2024-09-18T08:18:36.494 INFO:teuthology.orchestra.run.smithi007.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-09-18T08:18:36.495 INFO:teuthology.orchestra.run.smithi007.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-09-18T08:18:36.495 INFO:teuthology.orchestra.run.smithi007.stdout: "cache_target_full_ratio_micro": 800000, 2024-09-18T08:18:36.495 INFO:teuthology.orchestra.run.smithi007.stdout: "cache_min_flush_age": 0, 2024-09-18T08:18:36.495 INFO:teuthology.orchestra.run.smithi007.stdout: "cache_min_evict_age": 0, 2024-09-18T08:18:36.495 INFO:teuthology.orchestra.run.smithi007.stdout: "erasure_code_profile": "", 2024-09-18T08:18:36.495 INFO:teuthology.orchestra.run.smithi007.stdout: "hit_set_params": { 2024-09-18T08:18:36.495 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "none" 2024-09-18T08:18:36.495 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:18:36.495 INFO:teuthology.orchestra.run.smithi007.stdout: "hit_set_period": 0, 2024-09-18T08:18:36.495 INFO:teuthology.orchestra.run.smithi007.stdout: "hit_set_count": 0, 2024-09-18T08:18:36.495 INFO:teuthology.orchestra.run.smithi007.stdout: "use_gmt_hitset": true, 2024-09-18T08:18:36.495 INFO:teuthology.orchestra.run.smithi007.stdout: "min_read_recency_for_promote": 0, 2024-09-18T08:18:36.495 INFO:teuthology.orchestra.run.smithi007.stdout: "min_write_recency_for_promote": 0, 2024-09-18T08:18:36.495 INFO:teuthology.orchestra.run.smithi007.stdout: "hit_set_grade_decay_rate": 0, 2024-09-18T08:18:36.496 INFO:teuthology.orchestra.run.smithi007.stdout: "hit_set_search_last_n": 0, 2024-09-18T08:18:36.496 INFO:teuthology.orchestra.run.smithi007.stdout: "grade_table": [], 2024-09-18T08:18:36.496 INFO:teuthology.orchestra.run.smithi007.stdout: "stripe_width": 0, 2024-09-18T08:18:36.496 INFO:teuthology.orchestra.run.smithi007.stdout: "expected_num_objects": 0, 2024-09-18T08:18:36.496 INFO:teuthology.orchestra.run.smithi007.stdout: "fast_read": false, 2024-09-18T08:18:36.496 INFO:teuthology.orchestra.run.smithi007.stdout: "options": {}, 2024-09-18T08:18:36.496 INFO:teuthology.orchestra.run.smithi007.stdout: "application_metadata": { 2024-09-18T08:18:36.496 INFO:teuthology.orchestra.run.smithi007.stdout: "rados": {} 2024-09-18T08:18:36.496 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:18:36.496 INFO:teuthology.orchestra.run.smithi007.stdout: "read_balance": { 2024-09-18T08:18:36.496 INFO:teuthology.orchestra.run.smithi007.stdout: "score_type": "Fair distribution", 2024-09-18T08:18:36.496 INFO:teuthology.orchestra.run.smithi007.stdout: "score_acting": 2.190000057220459, 2024-09-18T08:18:36.496 INFO:teuthology.orchestra.run.smithi007.stdout: "score_stable": 2.190000057220459, 2024-09-18T08:18:36.496 INFO:teuthology.orchestra.run.smithi007.stdout: "optimal_score": 1, 2024-09-18T08:18:36.497 INFO:teuthology.orchestra.run.smithi007.stdout: "raw_score_acting": 2.190000057220459, 2024-09-18T08:18:36.497 INFO:teuthology.orchestra.run.smithi007.stdout: "raw_score_stable": 2.190000057220459, 2024-09-18T08:18:36.497 INFO:teuthology.orchestra.run.smithi007.stdout: "primary_affinity_weighted": 0.80927085876464844, 2024-09-18T08:18:36.497 INFO:teuthology.orchestra.run.smithi007.stdout: "average_primary_affinity": 0.92000001668930054, 2024-09-18T08:18:36.497 INFO:teuthology.orchestra.run.smithi007.stdout: "average_primary_affinity_weighted": 0.81000000238418579 2024-09-18T08:18:36.497 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-09-18T08:18:36.497 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-09-18T08:18:36.497 INFO:teuthology.orchestra.run.smithi007.stdout: ], 2024-09-18T08:18:36.497 INFO:teuthology.orchestra.run.smithi007.stdout: "osds": [ 2024-09-18T08:18:36.497 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:18:36.497 INFO:teuthology.orchestra.run.smithi007.stdout: "osd": 0, 2024-09-18T08:18:36.497 INFO:teuthology.orchestra.run.smithi007.stdout: "uuid": "ae8f60df-72ac-46fb-91a1-f132b851c876", 2024-09-18T08:18:36.498 INFO:teuthology.orchestra.run.smithi007.stdout: "up": 1, 2024-09-18T08:18:36.498 INFO:teuthology.orchestra.run.smithi007.stdout: "in": 1, 2024-09-18T08:18:36.498 INFO:teuthology.orchestra.run.smithi007.stdout: "weight": 1, 2024-09-18T08:18:36.498 INFO:teuthology.orchestra.run.smithi007.stdout: "primary_affinity": 1, 2024-09-18T08:18:36.498 INFO:teuthology.orchestra.run.smithi007.stdout: "last_clean_begin": 0, 2024-09-18T08:18:36.498 INFO:teuthology.orchestra.run.smithi007.stdout: "last_clean_end": 0, 2024-09-18T08:18:36.498 INFO:teuthology.orchestra.run.smithi007.stdout: "up_from": 8, 2024-09-18T08:18:36.498 INFO:teuthology.orchestra.run.smithi007.stdout: "up_thru": 401, 2024-09-18T08:18:36.498 INFO:teuthology.orchestra.run.smithi007.stdout: "down_at": 0, 2024-09-18T08:18:36.498 INFO:teuthology.orchestra.run.smithi007.stdout: "lost_at": 0, 2024-09-18T08:18:36.498 INFO:teuthology.orchestra.run.smithi007.stdout: "public_addrs": { 2024-09-18T08:18:36.498 INFO:teuthology.orchestra.run.smithi007.stdout: "addrvec": [ 2024-09-18T08:18:36.498 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:18:36.499 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "v2", 2024-09-18T08:18:36.499 INFO:teuthology.orchestra.run.smithi007.stdout: "addr": "172.21.15.7:6801", 2024-09-18T08:18:36.499 INFO:teuthology.orchestra.run.smithi007.stdout: "nonce": 2896425794 2024-09-18T08:18:36.499 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-09-18T08:18:36.499 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-09-18T08:18:36.499 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:18:36.499 INFO:teuthology.orchestra.run.smithi007.stdout: "cluster_addrs": { 2024-09-18T08:18:36.499 INFO:teuthology.orchestra.run.smithi007.stdout: "addrvec": [ 2024-09-18T08:18:36.499 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:18:36.499 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "v2", 2024-09-18T08:18:36.499 INFO:teuthology.orchestra.run.smithi007.stdout: "addr": "172.21.15.7:6802", 2024-09-18T08:18:36.499 INFO:teuthology.orchestra.run.smithi007.stdout: "nonce": 2896425794 2024-09-18T08:18:36.500 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-09-18T08:18:36.500 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-09-18T08:18:36.500 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:18:36.500 INFO:teuthology.orchestra.run.smithi007.stdout: "heartbeat_back_addrs": { 2024-09-18T08:18:36.500 INFO:teuthology.orchestra.run.smithi007.stdout: "addrvec": [ 2024-09-18T08:18:36.500 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:18:36.500 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "v2", 2024-09-18T08:18:36.500 INFO:teuthology.orchestra.run.smithi007.stdout: "addr": "172.21.15.7:6804", 2024-09-18T08:18:36.500 INFO:teuthology.orchestra.run.smithi007.stdout: "nonce": 2896425794 2024-09-18T08:18:36.500 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-09-18T08:18:36.500 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-09-18T08:18:36.500 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:18:36.500 INFO:teuthology.orchestra.run.smithi007.stdout: "heartbeat_front_addrs": { 2024-09-18T08:18:36.501 INFO:teuthology.orchestra.run.smithi007.stdout: "addrvec": [ 2024-09-18T08:18:36.501 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:18:36.501 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "v2", 2024-09-18T08:18:36.501 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4 2024-09-18T08:18:36.502 INFO:teuthology.orchestra.run.smithi007.stdout: "addr": "172.21.15.7:6803", 2024-09-18T08:18:36.502 INFO:teuthology.orchestra.run.smithi007.stdout: "nonce": 2896425794 2024-09-18T08:18:36.502 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-09-18T08:18:36.502 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-09-18T08:18:36.502 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:18:36.502 INFO:teuthology.orchestra.run.smithi007.stdout: "public_addr": "172.21.15.7:6801/2896425794", 2024-09-18T08:18:36.502 INFO:teuthology.orchestra.run.smithi007.stdout: "cluster_addr": "172.21.15.7:6802/2896425794", 2024-09-18T08:18:36.503 INFO:teuthology.orchestra.run.smithi007.stdout: "heartbeat_back_addr": "172.21.15.7:6804/2896425794", 2024-09-18T08:18:36.503 INFO:teuthology.orchestra.run.smithi007.stdout: "heartbeat_front_addr": "172.21.15.7:6803/2896425794", 2024-09-18T08:18:36.503 INFO:teuthology.orchestra.run.smithi007.stdout: "state": [ 2024-09-18T08:18:36.503 INFO:teuthology.orchestra.run.smithi007.stdout: "exists", 2024-09-18T08:18:36.503 INFO:teuthology.orchestra.run.smithi007.stdout: "up" 2024-09-18T08:18:36.503 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-09-18T08:18:36.503 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:18:36.503 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:18:36.503 INFO:teuthology.orchestra.run.smithi007.stdout: "osd": 1, 2024-09-18T08:18:36.503 INFO:teuthology.orchestra.run.smithi007.stdout: "uuid": "be28b70c-1f73-494f-ad4b-cb7ec5311999", 2024-09-18T08:18:36.503 INFO:teuthology.orchestra.run.smithi007.stdout: "up": 1, 2024-09-18T08:18:36.503 INFO:teuthology.orchestra.run.smithi007.stdout: "in": 1, 2024-09-18T08:18:36.503 INFO:teuthology.orchestra.run.smithi007.stdout: "weight": 1, 2024-09-18T08:18:36.504 INFO:teuthology.orchestra.run.smithi007.stdout: "primary_affinity": 1, 2024-09-18T08:18:36.504 INFO:teuthology.orchestra.run.smithi007.stdout: "last_clean_begin": 13, 2024-09-18T08:18:36.504 INFO:teuthology.orchestra.run.smithi007.stdout: "last_clean_end": 243, 2024-09-18T08:18:36.504 INFO:teuthology.orchestra.run.smithi007.stdout: "up_from": 307, 2024-09-18T08:18:36.504 INFO:teuthology.orchestra.run.smithi007.stdout: "up_thru": 401, 2024-09-18T08:18:36.504 INFO:teuthology.orchestra.run.smithi007.stdout: "down_at": 244, 2024-09-18T08:18:36.504 INFO:teuthology.orchestra.run.smithi007.stdout: "lost_at": 0, 2024-09-18T08:18:36.504 INFO:teuthology.orchestra.run.smithi007.stdout: "public_addrs": { 2024-09-18T08:18:36.504 INFO:teuthology.orchestra.run.smithi007.stdout: "addrvec": [ 2024-09-18T08:18:36.504 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:18:36.504 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "v2", 2024-09-18T08:18:36.504 INFO:teuthology.orchestra.run.smithi007.stdout: "addr": "172.21.15.7:6805", 2024-09-18T08:18:36.505 INFO:teuthology.orchestra.run.smithi007.stdout: "nonce": 3300107791 2024-09-18T08:18:36.505 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-09-18T08:18:36.505 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-09-18T08:18:36.505 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:18:36.505 INFO:teuthology.orchestra.run.smithi007.stdout: "cluster_addrs": { 2024-09-18T08:18:36.505 INFO:teuthology.orchestra.run.smithi007.stdout: "addrvec": [ 2024-09-18T08:18:36.505 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:18:36.505 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "v2", 2024-09-18T08:18:36.505 INFO:teuthology.orchestra.run.smithi007.stdout: "addr": "172.21.15.7:6806", 2024-09-18T08:18:36.505 INFO:teuthology.orchestra.run.smithi007.stdout: "nonce": 3300107791 2024-09-18T08:18:36.505 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-09-18T08:18:36.505 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-09-18T08:18:36.506 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:18:36.506 INFO:teuthology.orchestra.run.smithi007.stdout: "heartbeat_back_addrs": { 2024-09-18T08:18:36.506 INFO:teuthology.orchestra.run.smithi007.stdout: "addrvec": [ 2024-09-18T08:18:36.506 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:18:36.506 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "v2", 2024-09-18T08:18:36.506 INFO:teuthology.orchestra.run.smithi007.stdout: "addr": "172.21.15.7:6808", 2024-09-18T08:18:36.506 INFO:teuthology.orchestra.run.smithi007.stdout: "nonce": 3300107791 2024-09-18T08:18:36.506 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-09-18T08:18:36.506 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-09-18T08:18:36.506 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:18:36.506 INFO:teuthology.orchestra.run.smithi007.stdout: "heartbeat_front_addrs": { 2024-09-18T08:18:36.506 INFO:teuthology.orchestra.run.smithi007.stdout: "addrvec": [ 2024-09-18T08:18:36.506 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:18:36.507 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "v2", 2024-09-18T08:18:36.507 INFO:teuthology.orchestra.run.smithi007.stdout: "addr": "172.21.15.7:6807", 2024-09-18T08:18:36.507 INFO:teuthology.orchestra.run.smithi007.stdout: "nonce": 3300107791 2024-09-18T08:18:36.507 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-09-18T08:18:36.507 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-09-18T08:18:36.507 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:18:36.507 INFO:teuthology.orchestra.run.smithi007.stdout: "public_addr": "172.21.15.7:6805/3300107791", 2024-09-18T08:18:36.507 INFO:teuthology.orchestra.run.smithi007.stdout: "cluster_addr": "172.21.15.7:6806/3300107791", 2024-09-18T08:18:36.507 INFO:teuthology.orchestra.run.smithi007.stdout: "heartbeat_back_addr": "172.21.15.7:6808/3300107791", 2024-09-18T08:18:36.507 INFO:teuthology.orchestra.run.smithi007.stdout: "heartbeat_front_addr": "172.21.15.7:6807/3300107791", 2024-09-18T08:18:36.507 INFO:teuthology.orchestra.run.smithi007.stdout: "state": [ 2024-09-18T08:18:36.507 INFO:teuthology.orchestra.run.smithi007.stdout: "exists", 2024-09-18T08:18:36.507 INFO:teuthology.orchestra.run.smithi007.stdout: "up" 2024-09-18T08:18:36.508 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-09-18T08:18:36.508 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:18:36.508 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:18:36.508 INFO:teuthology.orchestra.run.smithi007.stdout: "osd": 2, 2024-09-18T08:18:36.508 INFO:teuthology.orchestra.run.smithi007.stdout: "uuid": "cfa4f083-c7fd-4743-be65-84a3040fe6dc", 2024-09-18T08:18:36.508 INFO:teuthology.orchestra.run.smithi007.stdout: "up": 1, 2024-09-18T08:18:36.508 INFO:teuthology.orchestra.run.smithi007.stdout: "in": 1, 2024-09-18T08:18:36.508 INFO:teuthology.orchestra.run.smithi007.stdout: "weight": 1, 2024-09-18T08:18:36.508 INFO:teuthology.orchestra.run.smithi007.stdout: "primary_affinity": 1, 2024-09-18T08:18:36.508 INFO:teuthology.orchestra.run.smithi007.stdout: "last_clean_begin": 0, 2024-09-18T08:18:36.508 INFO:teuthology.orchestra.run.smithi007.stdout: "last_clean_end": 0, 2024-09-18T08:18:36.508 INFO:teuthology.orchestra.run.smithi007.stdout: "up_from": 18, 2024-09-18T08:18:36.508 INFO:teuthology.orchestra.run.smithi007.stdout: "up_thru": 401, 2024-09-18T08:18:36.508 INFO:teuthology.orchestra.run.smithi007.stdout: "down_at": 0, 2024-09-18T08:18:36.509 INFO:teuthology.orchestra.run.smithi007.stdout: "lost_at": 0, 2024-09-18T08:18:36.509 INFO:teuthology.orchestra.run.smithi007.stdout: "public_addrs": { 2024-09-18T08:18:36.509 INFO:teuthology.orchestra.run.smithi007.stdout: "addrvec": [ 2024-09-18T08:18:36.509 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:18:36.509 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "v2", 2024-09-18T08:18:36.509 INFO:teuthology.orchestra.run.smithi007.stdout: "addr": "172.21.15.7:6809", 2024-09-18T08:18:36.509 INFO:teuthology.orchestra.run.smithi007.stdout: "nonce": 556142559 2024-09-18T08:18:36.509 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-09-18T08:18:36.509 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-09-18T08:18:36.509 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:18:36.509 INFO:teuthology.orchestra.run.smithi007.stdout: "cluster_addrs": { 2024-09-18T08:18:36.509 INFO:teuthology.orchestra.run.smithi007.stdout: "addrvec": [ 2024-09-18T08:18:36.509 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:18:36.510 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "v2", 2024-09-18T08:18:36.510 INFO:teuthology.orchestra.run.smithi007.stdout: "addr": "172.21.15.7:6810", 2024-09-18T08:18:36.510 INFO:teuthology.orchestra.run.smithi007.stdout: "nonce": 556142559 2024-09-18T08:18:36.510 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-09-18T08:18:36.510 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-09-18T08:18:36.510 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:18:36.510 INFO:teuthology.orchestra.run.smithi007.stdout: "heartbeat_back_addrs": { 2024-09-18T08:18:36.510 INFO:teuthology.orchestra.run.smithi007.stdout: "addrvec": [ 2024-09-18T08:18:36.510 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:18:36.510 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "v2", 2024-09-18T08:18:36.510 INFO:teuthology.orchestra.run.smithi007.stdout: "addr": "172.21.15.7:6812", 2024-09-18T08:18:36.510 INFO:teuthology.orchestra.run.smithi007.stdout: "nonce": 556142559 2024-09-18T08:18:36.510 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-09-18T08:18:36.511 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-09-18T08:18:36.511 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:18:36.511 INFO:teuthology.orchestra.run.smithi007.stdout: "heartbeat_front_addrs": { 2024-09-18T08:18:36.511 INFO:teuthology.orchestra.run.smithi007.stdout: "addrvec": [ 2024-09-18T08:18:36.511 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:18:36.511 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "v2", 2024-09-18T08:18:36.511 INFO:teuthology.orchestra.run.smithi007.stdout: "addr": "172.21.15.7:6811", 2024-09-18T08:18:36.511 INFO:teuthology.orchestra.run.smithi007.stdout: "nonce": 556142559 2024-09-18T08:18:36.511 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-09-18T08:18:36.511 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-09-18T08:18:36.511 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:18:36.511 INFO:teuthology.orchestra.run.smithi007.stdout: "public_addr": "172.21.15.7:6809/556142559", 2024-09-18T08:18:36.511 INFO:teuthology.orchestra.run.smithi007.stdout: "cluster_addr": "172.21.15.7:6810/556142559", 2024-09-18T08:18:36.512 INFO:teuthology.orchestra.run.smithi007.stdout: "heartbeat_back_addr": "172.21.15.7:6812/556142559", 2024-09-18T08:18:36.512 INFO:teuthology.orchestra.run.smithi007.stdout: "heartbeat_front_addr": "172.21.15.7:6811/556142559", 2024-09-18T08:18:36.512 INFO:teuthology.orchestra.run.smithi007.stdout: "state": [ 2024-09-18T08:18:36.512 INFO:teuthology.orchestra.run.smithi007.stdout: "exists", 2024-09-18T08:18:36.512 INFO:teuthology.orchestra.run.smithi007.stdout: "up" 2024-09-18T08:18:36.512 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-09-18T08:18:36.512 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:18:36.512 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:18:36.512 INFO:teuthology.orchestra.run.smithi007.stdout: "osd": 3, 2024-09-18T08:18:36.512 INFO:teuthology.orchestra.run.smithi007.stdout: "uuid": "11ecd7e1-a751-4b75-95f5-00b32d2a5bc4", 2024-09-18T08:18:36.512 INFO:teuthology.orchestra.run.smithi007.stdout: "up": 1, 2024-09-18T08:18:36.512 INFO:teuthology.orchestra.run.smithi007.stdout: "in": 1, 2024-09-18T08:18:36.512 INFO:teuthology.orchestra.run.smithi007.stdout: "weight": 1, 2024-09-18T08:18:36.513 INFO:teuthology.orchestra.run.smithi007.stdout: "primary_affinity": 1, 2024-09-18T08:18:36.513 INFO:teuthology.orchestra.run.smithi007.stdout: "last_clean_begin": 0, 2024-09-18T08:18:36.513 INFO:teuthology.orchestra.run.smithi007.stdout: "last_clean_end": 0, 2024-09-18T08:18:36.513 INFO:teuthology.orchestra.run.smithi007.stdout: "up_from": 26, 2024-09-18T08:18:36.513 INFO:teuthology.orchestra.run.smithi007.stdout: "up_thru": 401, 2024-09-18T08:18:36.513 INFO:teuthology.orchestra.run.smithi007.stdout: "down_at": 0, 2024-09-18T08:18:36.513 INFO:teuthology.orchestra.run.smithi007.stdout: "lost_at": 0, 2024-09-18T08:18:36.513 INFO:teuthology.orchestra.run.smithi007.stdout: "public_addrs": { 2024-09-18T08:18:36.513 INFO:teuthology.orchestra.run.smithi007.stdout: "addrvec": [ 2024-09-18T08:18:36.513 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:18:36.513 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "v2", 2024-09-18T08:18:36.513 INFO:teuthology.orchestra.run.smithi007.stdout: "addr": "172.21.15.7:6813", 2024-09-18T08:18:36.513 INFO:teuthology.orchestra.run.smithi007.stdout: "nonce": 163587772 2024-09-18T08:18:36.514 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-09-18T08:18:36.514 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-09-18T08:18:36.514 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:18:36.514 INFO:teuthology.orchestra.run.smithi007.stdout: "cluster_addrs": { 2024-09-18T08:18:36.514 INFO:teuthology.orchestra.run.smithi007.stdout: "addrvec": [ 2024-09-18T08:18:36.514 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:18:36.514 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "v2", 2024-09-18T08:18:36.514 INFO:teuthology.orchestra.run.smithi007.stdout: "addr": "172.21.15.7:6814", 2024-09-18T08:18:36.514 INFO:teuthology.orchestra.run.smithi007.stdout: "nonce": 163587772 2024-09-18T08:18:36.514 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-09-18T08:18:36.514 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-09-18T08:18:36.514 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:18:36.514 INFO:teuthology.orchestra.run.smithi007.stdout: "heartbeat_back_addrs": { 2024-09-18T08:18:36.514 INFO:teuthology.orchestra.run.smithi007.stdout: "addrvec": [ 2024-09-18T08:18:36.514 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:18:36.515 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "v2", 2024-09-18T08:18:36.515 INFO:teuthology.orchestra.run.smithi007.stdout: "addr": "172.21.15.7:6816", 2024-09-18T08:18:36.515 INFO:teuthology.orchestra.run.smithi007.stdout: "nonce": 163587772 2024-09-18T08:18:36.515 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-09-18T08:18:36.515 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-09-18T08:18:36.515 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:18:36.515 INFO:teuthology.orchestra.run.smithi007.stdout: "heartbeat_front_addrs": { 2024-09-18T08:18:36.515 INFO:teuthology.orchestra.run.smithi007.stdout: "addrvec": [ 2024-09-18T08:18:36.515 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:18:36.515 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "v2", 2024-09-18T08:18:36.515 INFO:teuthology.orchestra.run.smithi007.stdout: "addr": "172.21.15.7:6815", 2024-09-18T08:18:36.515 INFO:teuthology.orchestra.run.smithi007.stdout: "nonce": 163587772 2024-09-18T08:18:36.515 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-09-18T08:18:36.516 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-09-18T08:18:36.516 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:18:36.516 INFO:teuthology.orchestra.run.smithi007.stdout: "public_addr": "172.21.15.7:6813/163587772", 2024-09-18T08:18:36.516 INFO:teuthology.orchestra.run.smithi007.stdout: "cluster_addr": "172.21.15.7:6814/163587772", 2024-09-18T08:18:36.516 INFO:teuthology.orchestra.run.smithi007.stdout: "heartbeat_back_addr": "172.21.15.7:6816/163587772", 2024-09-18T08:18:36.516 INFO:teuthology.orchestra.run.smithi007.stdout: "heartbeat_front_addr": "172.21.15.7:6815/163587772", 2024-09-18T08:18:36.516 INFO:teuthology.orchestra.run.smithi007.stdout: "state": [ 2024-09-18T08:18:36.516 INFO:teuthology.orchestra.run.smithi007.stdout: "exists", 2024-09-18T08:18:36.516 INFO:teuthology.orchestra.run.smithi007.stdout: "up" 2024-09-18T08:18:36.516 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-09-18T08:18:36.516 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:18:36.516 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:18:36.516 INFO:teuthology.orchestra.run.smithi007.stdout: "osd": 4, 2024-09-18T08:18:36.517 INFO:teuthology.orchestra.run.smithi007.stdout: "uuid": "98992b34-cfa1-45bf-97a1-7d557739252f", 2024-09-18T08:18:36.517 INFO:teuthology.orchestra.run.smithi007.stdout: "up": 1, 2024-09-18T08:18:36.517 INFO:teuthology.orchestra.run.smithi007.stdout: "in": 0, 2024-09-18T08:18:36.517 INFO:teuthology.orchestra.run.smithi007.stdout: "weight": 0, 2024-09-18T08:18:36.517 INFO:teuthology.orchestra.run.smithi007.stdout: "primary_affinity": 1, 2024-09-18T08:18:36.517 INFO:teuthology.orchestra.run.smithi007.stdout: "last_clean_begin": 0, 2024-09-18T08:18:36.517 INFO:teuthology.orchestra.run.smithi007.stdout: "last_clean_end": 0, 2024-09-18T08:18:36.517 INFO:teuthology.orchestra.run.smithi007.stdout: "up_from": 32, 2024-09-18T08:18:36.517 INFO:teuthology.orchestra.run.smithi007.stdout: "up_thru": 336, 2024-09-18T08:18:36.517 INFO:teuthology.orchestra.run.smithi007.stdout: "down_at": 0, 2024-09-18T08:18:36.517 INFO:teuthology.orchestra.run.smithi007.stdout: "lost_at": 0, 2024-09-18T08:18:36.517 INFO:teuthology.orchestra.run.smithi007.stdout: "public_addrs": { 2024-09-18T08:18:36.517 INFO:teuthology.orchestra.run.smithi007.stdout: "addrvec": [ 2024-09-18T08:18:36.517 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:18:36.518 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "v2", 2024-09-18T08:18:36.518 INFO:teuthology.orchestra.run.smithi007.stdout: "addr": "172.21.15.50:6800", 2024-09-18T08:18:36.518 INFO:teuthology.orchestra.run.smithi007.stdout: "nonce": 3662119805 2024-09-18T08:18:36.518 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-09-18T08:18:36.518 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-09-18T08:18:36.518 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:18:36.518 INFO:teuthology.orchestra.run.smithi007.stdout: "cluster_addrs": { 2024-09-18T08:18:36.518 INFO:teuthology.orchestra.run.smithi007.stdout: "addrvec": [ 2024-09-18T08:18:36.518 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:18:36.518 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "v2", 2024-09-18T08:18:36.518 INFO:teuthology.orchestra.run.smithi007.stdout: "addr": "172.21.15.50:6801", 2024-09-18T08:18:36.518 INFO:teuthology.orchestra.run.smithi007.stdout: "nonce": 3662119805 2024-09-18T08:18:36.518 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-09-18T08:18:36.518 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-09-18T08:18:36.519 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:18:36.519 INFO:teuthology.orchestra.run.smithi007.stdout: "heartbeat_back_addrs": { 2024-09-18T08:18:36.519 INFO:teuthology.orchestra.run.smithi007.stdout: "addrvec": [ 2024-09-18T08:18:36.519 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:18:36.520 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "v2", 2024-09-18T08:18:36.520 INFO:teuthology.orchestra.run.smithi007.stdout: "addr": "172.21.15.50:6803", 2024-09-18T08:18:36.520 INFO:teuthology.orchestra.run.smithi007.stdout: "nonce": 3662119805 2024-09-18T08:18:36.520 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-09-18T08:18:36.520 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-09-18T08:18:36.520 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:18:36.520 INFO:teuthology.orchestra.run.smithi007.stdout: "heartbeat_front_addrs": { 2024-09-18T08:18:36.520 INFO:teuthology.orchestra.run.smithi007.stdout: "addrvec": [ 2024-09-18T08:18:36.520 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:18:36.520 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "v2", 2024-09-18T08:18:36.521 INFO:teuthology.orchestra.run.smithi007.stdout: "addr": "172.21.15.50:6802", 2024-09-18T08:18:36.521 INFO:teuthology.orchestra.run.smithi007.stdout: "nonce": 3662119805 2024-09-18T08:18:36.521 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-09-18T08:18:36.521 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-09-18T08:18:36.521 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:18:36.521 INFO:teuthology.orchestra.run.smithi007.stdout: "public_addr": "172.21.15.50:6800/3662119805", 2024-09-18T08:18:36.521 INFO:teuthology.orchestra.run.smithi007.stdout: "cluster_addr": "172.21.15.50:6801/3662119805", 2024-09-18T08:18:36.521 INFO:teuthology.orchestra.run.smithi007.stdout: "heartbeat_back_addr": "172.21.15.50:6803/3662119805", 2024-09-18T08:18:36.521 INFO:teuthology.orchestra.run.smithi007.stdout: "heartbeat_front_addr": "172.21.15.50:6802/3662119805", 2024-09-18T08:18:36.521 INFO:teuthology.orchestra.run.smithi007.stdout: "state": [ 2024-09-18T08:18:36.521 INFO:teuthology.orchestra.run.smithi007.stdout: "exists", 2024-09-18T08:18:36.521 INFO:teuthology.orchestra.run.smithi007.stdout: "up" 2024-09-18T08:18:36.521 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-09-18T08:18:36.522 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:18:36.522 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:18:36.522 INFO:teuthology.orchestra.run.smithi007.stdout: "osd": 5, 2024-09-18T08:18:36.522 INFO:teuthology.orchestra.run.smithi007.stdout: "uuid": "7f290005-ebca-467b-87e4-02ead1063610", 2024-09-18T08:18:36.522 INFO:teuthology.orchestra.run.smithi007.stdout: "up": 1, 2024-09-18T08:18:36.522 INFO:teuthology.orchestra.run.smithi007.stdout: "in": 1, 2024-09-18T08:18:36.522 INFO:teuthology.orchestra.run.smithi007.stdout: "weight": 1, 2024-09-18T08:18:36.522 INFO:teuthology.orchestra.run.smithi007.stdout: "primary_affinity": 1, 2024-09-18T08:18:36.522 INFO:teuthology.orchestra.run.smithi007.stdout: "last_clean_begin": 0, 2024-09-18T08:18:36.523 INFO:teuthology.orchestra.run.smithi007.stdout: "last_clean_end": 0, 2024-09-18T08:18:36.523 INFO:teuthology.orchestra.run.smithi007.stdout: "up_from": 39, 2024-09-18T08:18:36.523 INFO:teuthology.orchestra.run.smithi007.stdout: "up_thru": 401, 2024-09-18T08:18:36.523 INFO:teuthology.orchestra.run.smithi007.stdout: "down_at": 0, 2024-09-18T08:18:36.523 INFO:teuthology.orchestra.run.smithi007.stdout: "lost_at": 0, 2024-09-18T08:18:36.523 INFO:teuthology.orchestra.run.smithi007.stdout: "public_addrs": { 2024-09-18T08:18:36.524 INFO:teuthology.orchestra.run.smithi007.stdout: "addrvec": [ 2024-09-18T08:18:36.524 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:18:36.524 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "v2", 2024-09-18T08:18:36.524 INFO:teuthology.orchestra.run.smithi007.stdout: "addr": "172.21.15.50:6804", 2024-09-18T08:18:36.524 INFO:teuthology.orchestra.run.smithi007.stdout: "nonce": 1199797954 2024-09-18T08:18:36.524 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-09-18T08:18:36.524 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-09-18T08:18:36.524 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:18:36.524 INFO:teuthology.orchestra.run.smithi007.stdout: "cluster_addrs": { 2024-09-18T08:18:36.524 INFO:teuthology.orchestra.run.smithi007.stdout: "addrvec": [ 2024-09-18T08:18:36.524 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:18:36.524 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "v2", 2024-09-18T08:18:36.525 INFO:teuthology.orchestra.run.smithi007.stdout: "addr": "172.21.15.50:6805", 2024-09-18T08:18:36.525 INFO:teuthology.orchestra.run.smithi007.stdout: "nonce": 1199797954 2024-09-18T08:18:36.525 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-09-18T08:18:36.525 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-09-18T08:18:36.525 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:18:36.525 INFO:teuthology.orchestra.run.smithi007.stdout: "heartbeat_back_addrs": { 2024-09-18T08:18:36.525 INFO:teuthology.orchestra.run.smithi007.stdout: "addrvec": [ 2024-09-18T08:18:36.525 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:18:36.525 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "v2", 2024-09-18T08:18:36.525 INFO:teuthology.orchestra.run.smithi007.stdout: "addr": "172.21.15.50:6807", 2024-09-18T08:18:36.526 INFO:teuthology.orchestra.run.smithi007.stdout: "nonce": 1199797954 2024-09-18T08:18:36.526 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-09-18T08:18:36.526 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-09-18T08:18:36.526 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:18:36.526 INFO:teuthology.orchestra.run.smithi007.stdout: "heartbeat_front_addrs": { 2024-09-18T08:18:36.526 INFO:teuthology.orchestra.run.smithi007.stdout: "addrvec": [ 2024-09-18T08:18:36.526 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:18:36.526 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "v2", 2024-09-18T08:18:36.526 INFO:teuthology.orchestra.run.smithi007.stdout: "addr": "172.21.15.50:6806", 2024-09-18T08:18:36.526 INFO:teuthology.orchestra.run.smithi007.stdout: "nonce": 1199797954 2024-09-18T08:18:36.526 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-09-18T08:18:36.526 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-09-18T08:18:36.526 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:18:36.527 INFO:teuthology.orchestra.run.smithi007.stdout: "public_addr": "172.21.15.50:6804/1199797954", 2024-09-18T08:18:36.527 INFO:teuthology.orchestra.run.smithi007.stdout: "cluster_addr": "172.21.15.50:6805/1199797954", 2024-09-18T08:18:36.527 INFO:teuthology.orchestra.run.smithi007.stdout: "heartbeat_back_addr": "172.21.15.50:6807/1199797954", 2024-09-18T08:18:36.527 INFO:teuthology.orchestra.run.smithi007.stdout: "heartbeat_front_addr": "172.21.15.50:6806/1199797954", 2024-09-18T08:18:36.527 INFO:teuthology.orchestra.run.smithi007.stdout: "state": [ 2024-09-18T08:18:36.527 INFO:teuthology.orchestra.run.smithi007.stdout: "exists", 2024-09-18T08:18:36.527 INFO:teuthology.orchestra.run.smithi007.stdout: "up" 2024-09-18T08:18:36.527 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-09-18T08:18:36.527 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:18:36.527 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:18:36.527 INFO:teuthology.orchestra.run.smithi007.stdout: "osd": 6, 2024-09-18T08:18:36.527 INFO:teuthology.orchestra.run.smithi007.stdout: "uuid": "e07d846c-ae42-4fea-aa5c-3a25835d18a6", 2024-09-18T08:18:36.527 INFO:teuthology.orchestra.run.smithi007.stdout: "up": 1, 2024-09-18T08:18:36.528 INFO:teuthology.orchestra.run.smithi007.stdout: "in": 1, 2024-09-18T08:18:36.528 INFO:teuthology.orchestra.run.smithi007.stdout: "weight": 1, 2024-09-18T08:18:36.528 INFO:teuthology.orchestra.run.smithi007.stdout: "primary_affinity": 0, 2024-09-18T08:18:36.528 INFO:teuthology.orchestra.run.smithi007.stdout: "last_clean_begin": 0, 2024-09-18T08:18:36.528 INFO:teuthology.orchestra.run.smithi007.stdout: "last_clean_end": 0, 2024-09-18T08:18:36.528 INFO:teuthology.orchestra.run.smithi007.stdout: "up_from": 47, 2024-09-18T08:18:36.528 INFO:teuthology.orchestra.run.smithi007.stdout: "up_thru": 244, 2024-09-18T08:18:36.528 INFO:teuthology.orchestra.run.smithi007.stdout: "down_at": 0, 2024-09-18T08:18:36.528 INFO:teuthology.orchestra.run.smithi007.stdout: "lost_at": 0, 2024-09-18T08:18:36.528 INFO:teuthology.orchestra.run.smithi007.stdout: "public_addrs": { 2024-09-18T08:18:36.528 INFO:teuthology.orchestra.run.smithi007.stdout: "addrvec": [ 2024-09-18T08:18:36.528 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:18:36.528 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "v2", 2024-09-18T08:18:36.529 INFO:teuthology.orchestra.run.smithi007.stdout: "addr": "172.21.15.50:6808", 2024-09-18T08:18:36.529 INFO:teuthology.orchestra.run.smithi007.stdout: "nonce": 255960422 2024-09-18T08:18:36.529 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-09-18T08:18:36.529 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-09-18T08:18:36.529 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:18:36.529 INFO:teuthology.orchestra.run.smithi007.stdout: "cluster_addrs": { 2024-09-18T08:18:36.529 INFO:teuthology.orchestra.run.smithi007.stdout: "addrvec": [ 2024-09-18T08:18:36.529 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:18:36.529 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "v2", 2024-09-18T08:18:36.529 INFO:teuthology.orchestra.run.smithi007.stdout: "addr": "172.21.15.50:6809", 2024-09-18T08:18:36.529 INFO:teuthology.orchestra.run.smithi007.stdout: "nonce": 255960422 2024-09-18T08:18:36.529 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-09-18T08:18:36.529 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-09-18T08:18:36.530 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:18:36.530 INFO:teuthology.orchestra.run.smithi007.stdout: "heartbeat_back_addrs": { 2024-09-18T08:18:36.530 INFO:teuthology.orchestra.run.smithi007.stdout: "addrvec": [ 2024-09-18T08:18:36.530 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:18:36.530 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "v2", 2024-09-18T08:18:36.530 INFO:teuthology.orchestra.run.smithi007.stdout: "addr": "172.21.15.50:6811", 2024-09-18T08:18:36.530 INFO:teuthology.orchestra.run.smithi007.stdout: "nonce": 255960422 2024-09-18T08:18:36.530 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-09-18T08:18:36.530 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-09-18T08:18:36.530 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:18:36.530 INFO:teuthology.orchestra.run.smithi007.stdout: "heartbeat_front_addrs": { 2024-09-18T08:18:36.530 INFO:teuthology.orchestra.run.smithi007.stdout: "addrvec": [ 2024-09-18T08:18:36.530 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:18:36.530 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "v2", 2024-09-18T08:18:36.530 INFO:teuthology.orchestra.run.smithi007.stdout: "addr": "172.21.15.50:6810", 2024-09-18T08:18:36.531 INFO:teuthology.orchestra.run.smithi007.stdout: "nonce": 255960422 2024-09-18T08:18:36.531 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-09-18T08:18:36.531 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-09-18T08:18:36.531 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:18:36.531 INFO:teuthology.orchestra.run.smithi007.stdout: "public_addr": "172.21.15.50:6808/255960422", 2024-09-18T08:18:36.531 INFO:teuthology.orchestra.run.smithi007.stdout: "cluster_addr": "172.21.15.50:6809/255960422", 2024-09-18T08:18:36.531 INFO:teuthology.orchestra.run.smithi007.stdout: "heartbeat_back_addr": "172.21.15.50:6811/255960422", 2024-09-18T08:18:36.531 INFO:teuthology.orchestra.run.smithi007.stdout: "heartbeat_front_addr": "172.21.15.50:6810/255960422", 2024-09-18T08:18:36.531 INFO:teuthology.orchestra.run.smithi007.stdout: "state": [ 2024-09-18T08:18:36.531 INFO:teuthology.orchestra.run.smithi007.stdout: "exists", 2024-09-18T08:18:36.531 INFO:teuthology.orchestra.run.smithi007.stdout: "up" 2024-09-18T08:18:36.531 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-09-18T08:18:36.531 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:18:36.531 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:18:36.531 INFO:teuthology.orchestra.run.smithi007.stdout: "osd": 7, 2024-09-18T08:18:36.532 INFO:teuthology.orchestra.run.smithi007.stdout: "uuid": "c9dac3b7-d651-489d-b96e-5b10fa32d9af", 2024-09-18T08:18:36.532 INFO:teuthology.orchestra.run.smithi007.stdout: "up": 1, 2024-09-18T08:18:36.532 INFO:teuthology.orchestra.run.smithi007.stdout: "in": 1, 2024-09-18T08:18:36.532 INFO:teuthology.orchestra.run.smithi007.stdout: "weight": 1, 2024-09-18T08:18:36.532 INFO:teuthology.orchestra.run.smithi007.stdout: "primary_affinity": 0.4741668701171875, 2024-09-18T08:18:36.532 INFO:teuthology.orchestra.run.smithi007.stdout: "last_clean_begin": 0, 2024-09-18T08:18:36.532 INFO:teuthology.orchestra.run.smithi007.stdout: "last_clean_end": 0, 2024-09-18T08:18:36.532 INFO:teuthology.orchestra.run.smithi007.stdout: "up_from": 56, 2024-09-18T08:18:36.532 INFO:teuthology.orchestra.run.smithi007.stdout: "up_thru": 401, 2024-09-18T08:18:36.532 INFO:teuthology.orchestra.run.smithi007.stdout: "down_at": 0, 2024-09-18T08:18:36.532 INFO:teuthology.orchestra.run.smithi007.stdout: "lost_at": 0, 2024-09-18T08:18:36.532 INFO:teuthology.orchestra.run.smithi007.stdout: "public_addrs": { 2024-09-18T08:18:36.532 INFO:teuthology.orchestra.run.smithi007.stdout: "addrvec": [ 2024-09-18T08:18:36.532 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:18:36.532 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "v2", 2024-09-18T08:18:36.533 INFO:teuthology.orchestra.run.smithi007.stdout: "addr": "172.21.15.50:6812", 2024-09-18T08:18:36.533 INFO:teuthology.orchestra.run.smithi007.stdout: "nonce": 874923904 2024-09-18T08:18:36.533 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-09-18T08:18:36.533 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-09-18T08:18:36.533 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:18:36.533 INFO:teuthology.orchestra.run.smithi007.stdout: "cluster_addrs": { 2024-09-18T08:18:36.533 INFO:teuthology.orchestra.run.smithi007.stdout: "addrvec": [ 2024-09-18T08:18:36.533 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:18:36.533 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "v2", 2024-09-18T08:18:36.533 INFO:teuthology.orchestra.run.smithi007.stdout: "addr": "172.21.15.50:6813", 2024-09-18T08:18:36.533 INFO:teuthology.orchestra.run.smithi007.stdout: "nonce": 874923904 2024-09-18T08:18:36.533 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-09-18T08:18:36.534 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-09-18T08:18:36.534 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:18:36.534 INFO:teuthology.orchestra.run.smithi007.stdout: "heartbeat_back_addrs": { 2024-09-18T08:18:36.534 INFO:teuthology.orchestra.run.smithi007.stdout: "addrvec": [ 2024-09-18T08:18:36.534 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:18:36.534 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "v2", 2024-09-18T08:18:36.534 INFO:teuthology.orchestra.run.smithi007.stdout: "addr": "172.21.15.50:6815", 2024-09-18T08:18:36.534 INFO:teuthology.orchestra.run.smithi007.stdout: "nonce": 874923904 2024-09-18T08:18:36.534 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-09-18T08:18:36.534 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-09-18T08:18:36.534 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:18:36.534 INFO:teuthology.orchestra.run.smithi007.stdout: "heartbeat_front_addrs": { 2024-09-18T08:18:36.534 INFO:teuthology.orchestra.run.smithi007.stdout: "addrvec": [ 2024-09-18T08:18:36.534 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:18:36.535 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "v2", 2024-09-18T08:18:36.535 INFO:teuthology.orchestra.run.smithi007.stdout: "addr": "172.21.15.50:6814", 2024-09-18T08:18:36.535 INFO:teuthology.orchestra.run.smithi007.stdout: "nonce": 874923904 2024-09-18T08:18:36.535 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-09-18T08:18:36.535 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-09-18T08:18:36.535 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:18:36.535 INFO:teuthology.orchestra.run.smithi007.stdout: "public_addr": "172.21.15.50:6812/874923904", 2024-09-18T08:18:36.535 INFO:teuthology.orchestra.run.smithi007.stdout: "cluster_addr": "172.21.15.50:6813/874923904", 2024-09-18T08:18:36.535 INFO:teuthology.orchestra.run.smithi007.stdout: "heartbeat_back_addr": "172.21.15.50:6815/874923904", 2024-09-18T08:18:36.535 INFO:teuthology.orchestra.run.smithi007.stdout: "heartbeat_front_addr": "172.21.15.50:6814/874923904", 2024-09-18T08:18:36.535 INFO:teuthology.orchestra.run.smithi007.stdout: "state": [ 2024-09-18T08:18:36.535 INFO:teuthology.orchestra.run.smithi007.stdout: "exists", 2024-09-18T08:18:36.536 INFO:teuthology.orchestra.run.smithi007.stdout: "up" 2024-09-18T08:18:36.536 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-09-18T08:18:36.536 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-09-18T08:18:36.536 INFO:teuthology.orchestra.run.smithi007.stdout: ], 2024-09-18T08:18:36.536 INFO:teuthology.orchestra.run.smithi007.stdout: "osd_xinfo": [ 2024-09-18T08:18:36.536 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:18:36.536 INFO:teuthology.orchestra.run.smithi007.stdout: "osd": 0, 2024-09-18T08:18:36.536 INFO:teuthology.orchestra.run.smithi007.stdout: "down_stamp": "0.000000", 2024-09-18T08:18:36.536 INFO:teuthology.orchestra.run.smithi007.stdout: "laggy_probability": 0, 2024-09-18T08:18:36.536 INFO:teuthology.orchestra.run.smithi007.stdout: "laggy_interval": 0, 2024-09-18T08:18:36.536 INFO:teuthology.orchestra.run.smithi007.stdout: "features": 4540719139924082687, 2024-09-18T08:18:36.536 INFO:teuthology.orchestra.run.smithi007.stdout: "old_weight": 0, 2024-09-18T08:18:36.536 INFO:teuthology.orchestra.run.smithi007.stdout: "last_purged_snaps_scrub": "2024-09-18T08:17:33.038707+0000", 2024-09-18T08:18:36.537 INFO:teuthology.orchestra.run.smithi007.stdout: "dead_epoch": 0 2024-09-18T08:18:36.537 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:18:36.537 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:18:36.537 INFO:teuthology.orchestra.run.smithi007.stdout: "osd": 1, 2024-09-18T08:18:36.537 INFO:teuthology.orchestra.run.smithi007.stdout: "down_stamp": "0.000000", 2024-09-18T08:18:36.537 INFO:teuthology.orchestra.run.smithi007.stdout: "laggy_probability": 0, 2024-09-18T08:18:36.537 INFO:teuthology.orchestra.run.smithi007.stdout: "laggy_interval": 0, 2024-09-18T08:18:36.537 INFO:teuthology.orchestra.run.smithi007.stdout: "features": 4540719139924082687, 2024-09-18T08:18:36.537 INFO:teuthology.orchestra.run.smithi007.stdout: "old_weight": 0, 2024-09-18T08:18:36.537 INFO:teuthology.orchestra.run.smithi007.stdout: "last_purged_snaps_scrub": "2024-09-18T08:18:24.029220+0000", 2024-09-18T08:18:36.537 INFO:teuthology.orchestra.run.smithi007.stdout: "dead_epoch": 243 2024-09-18T08:18:36.537 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:18:36.538 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:18:36.538 INFO:teuthology.orchestra.run.smithi007.stdout: "osd": 2, 2024-09-18T08:18:36.538 INFO:teuthology.orchestra.run.smithi007.stdout: "down_stamp": "0.000000", 2024-09-18T08:18:36.538 INFO:teuthology.orchestra.run.smithi007.stdout: "laggy_probability": 0, 2024-09-18T08:18:36.538 INFO:teuthology.orchestra.run.smithi007.stdout: "laggy_interval": 0, 2024-09-18T08:18:36.538 INFO:teuthology.orchestra.run.smithi007.stdout: "features": 4540719139924082687, 2024-09-18T08:18:36.538 INFO:teuthology.orchestra.run.smithi007.stdout: "old_weight": 0, 2024-09-18T08:18:36.538 INFO:teuthology.orchestra.run.smithi007.stdout: "last_purged_snaps_scrub": "2024-09-18T08:17:32.795625+0000", 2024-09-18T08:18:36.538 INFO:teuthology.orchestra.run.smithi007.stdout: "dead_epoch": 0 2024-09-18T08:18:36.538 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:18:36.538 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:18:36.538 INFO:teuthology.orchestra.run.smithi007.stdout: "osd": 3, 2024-09-18T08:18:36.538 INFO:teuthology.orchestra.run.smithi007.stdout: "down_stamp": "0.000000", 2024-09-18T08:18:36.538 INFO:teuthology.orchestra.run.smithi007.stdout: "laggy_probability": 0, 2024-09-18T08:18:36.539 INFO:teuthology.orchestra.run.smithi007.stdout: "laggy_interval": 0, 2024-09-18T08:18:36.539 INFO:teuthology.orchestra.run.smithi007.stdout: "features": 4540719139924082687, 2024-09-18T08:18:36.539 INFO:teuthology.orchestra.run.smithi007.stdout: "old_weight": 0, 2024-09-18T08:18:36.539 INFO:teuthology.orchestra.run.smithi007.stdout: "last_purged_snaps_scrub": "2024-09-18T08:18:06.641839+0000", 2024-09-18T08:18:36.539 INFO:teuthology.orchestra.run.smithi007.stdout: "dead_epoch": 0 2024-09-18T08:18:36.539 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:18:36.539 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:18:36.539 INFO:teuthology.orchestra.run.smithi007.stdout: "osd": 4, 2024-09-18T08:18:36.539 INFO:teuthology.orchestra.run.smithi007.stdout: "down_stamp": "0.000000", 2024-09-18T08:18:36.539 INFO:teuthology.orchestra.run.smithi007.stdout: "laggy_probability": 0, 2024-09-18T08:18:36.539 INFO:teuthology.orchestra.run.smithi007.stdout: "laggy_interval": 0, 2024-09-18T08:18:36.539 INFO:teuthology.orchestra.run.smithi007.stdout: "features": 4540719139924082687, 2024-09-18T08:18:36.539 INFO:teuthology.orchestra.run.smithi007.stdout: "old_weight": 65536, 2024-09-18T08:18:36.540 INFO:teuthology.orchestra.run.smithi007.stdout: "last_purged_snaps_scrub": "2024-09-18T08:18:18.900056+0000", 2024-09-18T08:18:36.540 INFO:teuthology.orchestra.run.smithi007.stdout: "dead_epoch": 0 2024-09-18T08:18:36.540 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:18:36.540 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:18:36.540 INFO:teuthology.orchestra.run.smithi007.stdout: "osd": 5, 2024-09-18T08:18:36.540 INFO:teuthology.orchestra.run.smithi007.stdout: "down_stamp": "0.000000", 2024-09-18T08:18:36.540 INFO:teuthology.orchestra.run.smithi007.stdout: "laggy_probability": 0, 2024-09-18T08:18:36.540 INFO:teuthology.orchestra.run.smithi007.stdout: "laggy_interval": 0, 2024-09-18T08:18:36.540 INFO:teuthology.orchestra.run.smithi007.stdout: "features": 4540719139924082687, 2024-09-18T08:18:36.540 INFO:teuthology.orchestra.run.smithi007.stdout: "old_weight": 0, 2024-09-18T08:18:36.540 INFO:teuthology.orchestra.run.smithi007.stdout: "last_purged_snaps_scrub": "2024-09-18T08:18:24.016232+0000", 2024-09-18T08:18:36.540 INFO:teuthology.orchestra.run.smithi007.stdout: "dead_epoch": 0 2024-09-18T08:18:36.540 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:18:36.540 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:18:36.541 INFO:teuthology.orchestra.run.smithi007.stdout: "osd": 6, 2024-09-18T08:18:36.541 INFO:teuthology.orchestra.run.smithi007.stdout: "down_stamp": "0.000000", 2024-09-18T08:18:36.541 INFO:teuthology.orchestra.run.smithi007.stdout: "laggy_probability": 0, 2024-09-18T08:18:36.541 INFO:teuthology.orchestra.run.smithi007.stdout: "laggy_interval": 0, 2024-09-18T08:18:36.541 INFO:teuthology.orchestra.run.smithi007.stdout: "features": 4540719139924082687, 2024-09-18T08:18:36.541 INFO:teuthology.orchestra.run.smithi007.stdout: "old_weight": 0, 2024-09-18T08:18:36.541 INFO:teuthology.orchestra.run.smithi007.stdout: "last_purged_snaps_scrub": "2024-09-18T08:18:09.854168+0000", 2024-09-18T08:18:36.541 INFO:teuthology.orchestra.run.smithi007.stdout: "dead_epoch": 0 2024-09-18T08:18:36.541 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:18:36.541 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:18:36.541 INFO:teuthology.orchestra.run.smithi007.stdout: "osd": 7, 2024-09-18T08:18:36.541 INFO:teuthology.orchestra.run.smithi007.stdout: "down_stamp": "0.000000", 2024-09-18T08:18:36.542 INFO:teuthology.orchestra.run.smithi007.stdout: "laggy_probability": 0, 2024-09-18T08:18:36.542 INFO:teuthology.orchestra.run.smithi007.stdout: "laggy_interval": 0, 2024-09-18T08:18:36.542 INFO:teuthology.orchestra.run.smithi007.stdout: "features": 4540719139924082687, 2024-09-18T08:18:36.542 INFO:teuthology.orchestra.run.smithi007.stdout: "old_weight": 0, 2024-09-18T08:18:36.542 INFO:teuthology.orchestra.run.smithi007.stdout: "last_purged_snaps_scrub": "2024-09-18T08:17:42.528532+0000", 2024-09-18T08:18:36.542 INFO:teuthology.orchestra.run.smithi007.stdout: "dead_epoch": 0 2024-09-18T08:18:36.542 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-09-18T08:18:36.542 INFO:teuthology.orchestra.run.smithi007.stdout: ], 2024-09-18T08:18:36.542 INFO:teuthology.orchestra.run.smithi007.stdout: "pg_upmap": [ 2024-09-18T08:18:36.542 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:18:36.542 INFO:teuthology.orchestra.run.smithi007.stdout: "pgid": "3.c", 2024-09-18T08:18:36.542 INFO:teuthology.orchestra.run.smithi007.stdout: "osds": [ 2024-09-18T08:18:36.542 INFO:teuthology.orchestra.run.smithi007.stdout: 3, 2024-09-18T08:18:36.543 INFO:teuthology.orchestra.run.smithi007.stdout: 2, 2024-09-18T08:18:36.543 INFO:teuthology.orchestra.run.smithi007.stdout: 6 2024-09-18T08:18:36.543 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-09-18T08:18:36.543 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:18:36.543 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:18:36.543 INFO:teuthology.orchestra.run.smithi007.stdout: "pgid": "6.3", 2024-09-18T08:18:36.543 INFO:teuthology.orchestra.run.smithi007.stdout: "osds": [ 2024-09-18T08:18:36.543 INFO:teuthology.orchestra.run.smithi007.stdout: 5, 2024-09-18T08:18:36.543 INFO:teuthology.orchestra.run.smithi007.stdout: 3, 2024-09-18T08:18:36.543 INFO:teuthology.orchestra.run.smithi007.stdout: 2 2024-09-18T08:18:36.543 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-09-18T08:18:36.543 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-09-18T08:18:36.543 INFO:teuthology.orchestra.run.smithi007.stdout: ], 2024-09-18T08:18:36.543 INFO:teuthology.orchestra.run.smithi007.stdout: "pg_upmap_items": [ 2024-09-18T08:18:36.544 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:18:36.544 INFO:teuthology.orchestra.run.smithi007.stdout: "pgid": "3.8", 2024-09-18T08:18:36.544 INFO:teuthology.orchestra.run.smithi007.stdout: "mappings": [ 2024-09-18T08:18:36.544 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:18:36.544 INFO:teuthology.orchestra.run.smithi007.stdout: "from": 3, 2024-09-18T08:18:36.544 INFO:teuthology.orchestra.run.smithi007.stdout: "to": 2 2024-09-18T08:18:36.544 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-09-18T08:18:36.544 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-09-18T08:18:36.544 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:18:36.544 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:18:36.544 INFO:teuthology.orchestra.run.smithi007.stdout: "pgid": "4.7", 2024-09-18T08:18:36.544 INFO:teuthology.orchestra.run.smithi007.stdout: "mappings": [ 2024-09-18T08:18:36.545 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:18:36.545 INFO:teuthology.orchestra.run.smithi007.stdout: "from": 1, 2024-09-18T08:18:36.545 INFO:teuthology.orchestra.run.smithi007.stdout: "to": 2 2024-09-18T08:18:36.546 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-09-18T08:18:36.546 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-09-18T08:18:36.546 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:18:36.546 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:18:36.546 INFO:teuthology.orchestra.run.smithi007.stdout: "pgid": "4.14", 2024-09-18T08:18:36.546 INFO:teuthology.orchestra.run.smithi007.stdout: "mappings": [ 2024-09-18T08:18:36.546 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:18:36.546 INFO:teuthology.orchestra.run.smithi007.stdout: "from": 1, 2024-09-18T08:18:36.546 INFO:teuthology.orchestra.run.smithi007.stdout: "to": 2 2024-09-18T08:18:36.546 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-09-18T08:18:36.546 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-09-18T08:18:36.546 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:18:36.546 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:18:36.547 INFO:teuthology.orchestra.run.smithi007.stdout: "pgid": "4.18", 2024-09-18T08:18:36.547 INFO:teuthology.orchestra.run.smithi007.stdout: "mappings": [ 2024-09-18T08:18:36.547 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:18:36.547 INFO:teuthology.orchestra.run.smithi007.stdout: "from": 1, 2024-09-18T08:18:36.547 INFO:teuthology.orchestra.run.smithi007.stdout: "to": 2 2024-09-18T08:18:36.547 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-09-18T08:18:36.547 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-09-18T08:18:36.547 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:18:36.547 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:18:36.547 INFO:teuthology.orchestra.run.smithi007.stdout: "pgid": "4.1b", 2024-09-18T08:18:36.547 INFO:teuthology.orchestra.run.smithi007.stdout: "mappings": [ 2024-09-18T08:18:36.547 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:18:36.547 INFO:teuthology.orchestra.run.smithi007.stdout: "from": 3, 2024-09-18T08:18:36.548 INFO:teuthology.orchestra.run.smithi007.stdout: "to": 5 2024-09-18T08:18:36.548 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-09-18T08:18:36.548 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-09-18T08:18:36.548 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:18:36.548 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:18:36.548 INFO:teuthology.orchestra.run.smithi007.stdout: "pgid": "4.1d", 2024-09-18T08:18:36.548 INFO:teuthology.orchestra.run.smithi007.stdout: "mappings": [ 2024-09-18T08:18:36.548 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:18:36.548 INFO:teuthology.orchestra.run.smithi007.stdout: "from": 1, 2024-09-18T08:18:36.548 INFO:teuthology.orchestra.run.smithi007.stdout: "to": 0 2024-09-18T08:18:36.548 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-09-18T08:18:36.548 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-09-18T08:18:36.548 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:18:36.548 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:18:36.549 INFO:teuthology.orchestra.run.smithi007.stdout: "pgid": "5.b", 2024-09-18T08:18:36.549 INFO:teuthology.orchestra.run.smithi007.stdout: "mappings": [ 2024-09-18T08:18:36.549 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:18:36.549 INFO:teuthology.orchestra.run.smithi007.stdout: "from": 0, 2024-09-18T08:18:36.549 INFO:teuthology.orchestra.run.smithi007.stdout: "to": 3 2024-09-18T08:18:36.549 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-09-18T08:18:36.549 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-09-18T08:18:36.549 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-09-18T08:18:36.549 INFO:teuthology.orchestra.run.smithi007.stdout: ], 2024-09-18T08:18:36.549 INFO:teuthology.orchestra.run.smithi007.stdout: "pg_upmap_primaries": [], 2024-09-18T08:18:36.549 INFO:teuthology.orchestra.run.smithi007.stdout: "pg_temp": [], 2024-09-18T08:18:36.549 INFO:teuthology.orchestra.run.smithi007.stdout: "primary_temp": [], 2024-09-18T08:18:36.549 INFO:teuthology.orchestra.run.smithi007.stdout: "blocklist": { 2024-09-18T08:18:36.550 INFO:teuthology.orchestra.run.smithi007.stdout: "172.21.15.7:0/398367658": "2024-09-19T08:03:17.052903+0000", 2024-09-18T08:18:36.550 INFO:teuthology.orchestra.run.smithi007.stdout: "172.21.15.7:6800/2918496291": "2024-09-19T07:57:31.015269+0000", 2024-09-18T08:18:36.550 INFO:teuthology.orchestra.run.smithi007.stdout: "172.21.15.7:0/2514354773": "2024-09-19T07:57:31.015269+0000", 2024-09-18T08:18:36.550 INFO:teuthology.orchestra.run.smithi007.stdout: "172.21.15.7:0/3924911335": "2024-09-19T07:57:31.015269+0000", 2024-09-18T08:18:36.550 INFO:teuthology.orchestra.run.smithi007.stdout: "172.21.15.7:0/1668598084": "2024-09-19T07:57:57.346149+0000", 2024-09-18T08:18:36.550 INFO:teuthology.orchestra.run.smithi007.stdout: "172.21.15.7:0/4272027292": "2024-09-19T07:57:31.015269+0000", 2024-09-18T08:18:36.550 INFO:teuthology.orchestra.run.smithi007.stdout: "172.21.15.7:6800/1013061071": "2024-09-19T07:57:57.346149+0000", 2024-09-18T08:18:36.550 INFO:teuthology.orchestra.run.smithi007.stdout: "172.21.15.7:6800/1491061076": "2024-09-19T08:03:17.052903+0000", 2024-09-18T08:18:36.550 INFO:teuthology.orchestra.run.smithi007.stdout: "172.21.15.7:0/533727535": "2024-09-19T07:57:57.346149+0000", 2024-09-18T08:18:36.550 INFO:teuthology.orchestra.run.smithi007.stdout: "172.21.15.7:0/1605217610": "2024-09-19T08:03:17.052903+0000", 2024-09-18T08:18:36.550 INFO:teuthology.orchestra.run.smithi007.stdout: "172.21.15.7:0/1722968065": "2024-09-19T08:03:17.052903+0000", 2024-09-18T08:18:36.550 INFO:teuthology.orchestra.run.smithi007.stdout: "172.21.15.7:0/1277549846": "2024-09-19T07:57:57.346149+0000", 2024-09-18T08:18:36.550 INFO:teuthology.orchestra.run.smithi007.stdout: "172.21.15.7:0/232452890": "2024-09-19T08:03:17.052903+0000", 2024-09-18T08:18:36.550 INFO:teuthology.orchestra.run.smithi007.stdout: "172.21.15.7:0/3881670470": "2024-09-19T08:03:17.052903+0000" 2024-09-18T08:18:36.551 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:18:36.551 INFO:teuthology.orchestra.run.smithi007.stdout: "range_blocklist": {}, 2024-09-18T08:18:36.551 INFO:teuthology.orchestra.run.smithi007.stdout: "erasure_code_profiles": { 2024-09-18T08:18:36.551 INFO:teuthology.orchestra.run.smithi007.stdout: "default": { 2024-09-18T08:18:36.551 INFO:teuthology.orchestra.run.smithi007.stdout: "crush-failure-domain": "osd", 2024-09-18T08:18:36.551 INFO:teuthology.orchestra.run.smithi007.stdout: "k": "2", 2024-09-18T08:18:36.551 INFO:teuthology.orchestra.run.smithi007.stdout: "m": "1", 2024-09-18T08:18:36.551 INFO:teuthology.orchestra.run.smithi007.stdout: "plugin": "jerasure", 2024-09-18T08:18:36.551 INFO:teuthology.orchestra.run.smithi007.stdout: "technique": "reed_sol_van" 2024-09-18T08:18:36.551 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-09-18T08:18:36.551 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:18:36.551 INFO:teuthology.orchestra.run.smithi007.stdout: "removed_snaps_queue": [], 2024-09-18T08:18:36.551 INFO:teuthology.orchestra.run.smithi007.stdout: "new_removed_snaps": [], 2024-09-18T08:18:36.552 INFO:teuthology.orchestra.run.smithi007.stdout: "new_purged_snaps": [], 2024-09-18T08:18:36.552 INFO:teuthology.orchestra.run.smithi007.stdout: "crush_node_flags": {}, 2024-09-18T08:18:36.552 INFO:teuthology.orchestra.run.smithi007.stdout: "device_class_flags": {}, 2024-09-18T08:18:36.552 INFO:teuthology.orchestra.run.smithi007.stdout: "stretch_mode": { 2024-09-18T08:18:36.552 INFO:teuthology.orchestra.run.smithi007.stdout: "stretch_mode_enabled": false, 2024-09-18T08:18:36.552 INFO:teuthology.orchestra.run.smithi007.stdout: "stretch_bucket_count": 0, 2024-09-18T08:18:36.552 INFO:teuthology.orchestra.run.smithi007.stdout: "degraded_stretch_mode": 0, 2024-09-18T08:18:36.552 INFO:teuthology.orchestra.run.smithi007.stdout: "recovering_stretch_mode": 0, 2024-09-18T08:18:36.552 INFO:teuthology.orchestra.run.smithi007.stdout: "stretch_mode_bucket": 0 2024-09-18T08:18:36.552 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-09-18T08:18:36.552 INFO:teuthology.orchestra.run.smithi007.stdout:} 2024-09-18T08:18:36.575 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:18:36.575 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:18:36.576 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4' 2024-09-18T08:18:36.677 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4 2024-09-18T08:18:36.697 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:18:36 smithi007 ceph-mon[29459]: pgmap v795: 158 pgs: 158 active+clean; 583 KiB data, 3.8 GiB used, 622 GiB / 626 GiB avail; 849 B/s rd, 0 op/s; 101 B/s, 2 objects/s recovering 2024-09-18T08:18:36.697 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:18:36 smithi007 ceph-mon[29459]: from='client.? 172.21.15.7:0/2170569184' entity='client.admin' cmd={"prefix": "osd dump", "format": "json-pretty"} : dispatch 2024-09-18T08:18:36.697 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:18:36 smithi007 ceph-mon[39249]: pgmap v795: 158 pgs: 158 active+clean; 583 KiB data, 3.8 GiB used, 622 GiB / 626 GiB avail; 849 B/s rd, 0 op/s; 101 B/s, 2 objects/s recovering 2024-09-18T08:18:36.697 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:18:36 smithi007 ceph-mon[39249]: from='client.? 172.21.15.7:0/2170569184' entity='client.admin' cmd={"prefix": "osd dump", "format": "json-pretty"} : dispatch 2024-09-18T08:18:36.709 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:18:36.709 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:18:36.710 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4' 2024-09-18T08:18:36.810 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6 2024-09-18T08:18:36.838 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:18:36.839 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:18:36.839 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6' 2024-09-18T08:18:36.940 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0 2024-09-18T08:18:36.958 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid 779349ca-7593-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-18T08:18:36.980 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Sep 18 08:18:36 smithi007 ceph-779349ca-7593-11ef-bceb-c7b262605968-mgr-y[29688]: ::ffff:172.21.15.50 - - [18/Sep/2024:08:18:36] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-09-18T08:18:36.983 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:18:36.984 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:18:36.985 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0' 2024-09-18T08:18:37.030 INFO:tasks.radosbench.radosbench.0.smithi007.stdout: 2 16 4358 4342 135.677 131.438 0.00476313 0.00735665 2024-09-18T08:18:37.078 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:18:36 smithi050 ceph-mon[32371]: pgmap v795: 158 pgs: 158 active+clean; 583 KiB data, 3.8 GiB used, 622 GiB / 626 GiB avail; 849 B/s rd, 0 op/s; 101 B/s, 2 objects/s recovering 2024-09-18T08:18:37.078 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:18:36 smithi050 ceph-mon[32371]: from='client.? 172.21.15.7:0/2170569184' entity='client.admin' cmd={"prefix": "osd dump", "format": "json-pretty"} : dispatch 2024-09-18T08:18:37.085 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4 2024-09-18T08:18:37.118 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:18:37.119 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:18:37.119 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4' 2024-09-18T08:18:37.126 DEBUG:teuthology.orchestra.run.smithi050:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid 779349ca-7593-11ef-bceb-c7b262605968 -- ceph daemon osd.5 dump_ops_in_flight 2024-09-18T08:18:37.220 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1 2024-09-18T08:18:37.263 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:18:37.264 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:18:37.264 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1' 2024-09-18T08:18:37.365 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2 2024-09-18T08:18:37.410 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:18:37.412 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:18:37.413 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2' 2024-09-18T08:18:37.443 INFO:tasks.thrashosds.thrasher:j is {'epoch': 406, 'fsid': '779349ca-7593-11ef-bceb-c7b262605968', 'created': '2024-09-18T07:57:09.295710+0000', 'modified': '2024-09-18T08:18:34.424531+0000', 'last_up_change': '2024-09-18T08:14:38.391465+0000', 'last_in_change': '2024-09-18T08:18:26.612016+0000', 'flags': 'noscrub,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit', 'flags_num': 5801984, 'flags_set': ['noscrub', 'pglog_hardlimit', 'purged_snapdirs', 'recovery_deletes', 'sortbitwise'], 'crush_version': 19, 'full_ratio': 0.949999988079071, 'backfillfull_ratio': 0.8999999761581421, 'nearfull_ratio': 0.8500000238418579, 'cluster_snapshot': '', 'pool_max': 11, 'max_osd': 8, 'require_min_compat_client': 'luminous', 'min_compat_client': 'luminous', 'require_osd_release': 'squid', 'allow_crimson': False, 'pools': [{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-09-18T08:00:11.802627+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'is_stretch_pool': False, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 11, 'pg_placement_num': 11, 'pg_placement_num_target': 11, 'pg_num_target': 11, 'pg_num_pending': 11, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '285', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '152', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.909999966621399, 'score_stable': 1.909999966621399, 'optimal_score': 1, 'raw_score_acting': 1.909999966621399, 'raw_score_stable': 1.909999966621399, 'primary_affinity_weighted': 0.8092708587646484, 'average_primary_affinity': 0.9200000166893005, 'average_primary_affinity_weighted': 0.8100000023841858}}, {'pool': 2, 'pool_name': 'datapool', 'create_time': '2024-09-18T08:02:45.374099+0000', 'flags': 8193, 'flags_names': 'hashpspool,selfmanaged_snaps', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'is_stretch_pool': False, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 3, 'pg_placement_num': 3, 'pg_placement_num_target': 3, 'pg_num_target': 3, 'pg_num_pending': 3, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '64', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 2, 'snap_epoch': 64, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.3299999237060547, 'score_stable': 2.3299999237060547, 'optimal_score': 1, 'raw_score_acting': 2.3299999237060547, 'raw_score_stable': 2.3299999237060547, 'primary_affinity_weighted': 0.8092708587646484, 'average_primary_affinity': 0.9200000166893005, 'average_primary_affinity_weighted': 0.8100000023841858}}, {'pool': 3, 'pool_name': '.rgw.root', 'create_time': '2024-09-18T08:02:46.319947+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'is_stretch_pool': False, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '63', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rgw': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.9700000286102295, 'score_stable': 1.9700000286102295, 'optimal_score': 1, 'raw_score_acting': 1.9700000286102295, 'raw_score_stable': 1.9700000286102295, 'primary_affinity_weighted': 0.8092708587646484, 'average_primary_affinity': 0.9200000166893005, 'average_primary_affinity_weighted': 0.8100000023841858}}, {'pool': 4, 'pool_name': 'default.rgw.log', 'create_time': '2024-09-18T08:02:47.948552+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'is_stretch_pool': False, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '65', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rgw': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.9700000286102295, 'score_stable': 1.9700000286102295, 'optimal_score': 1, 'raw_score_acting': 1.9700000286102295, 'raw_score_stable': 1.9700000286102295, 'primary_affinity_weighted': 0.8092708587646484, 'average_primary_affinity': 0.9200000166893005, 'average_primary_affinity_weighted': 0.8100000023841858}}, {'pool': 5, 'pool_name': 'default.rgw.control', 'create_time': '2024-09-18T08:02:49.904302+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'is_stretch_pool': False, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '67', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rgw': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.309999942779541, 'score_stable': 1.309999942779541, 'optimal_score': 1, 'raw_score_acting': 1.309999942779541, 'raw_score_stable': 1.309999942779541, 'primary_affinity_weighted': 0.8092708587646484, 'average_primary_affinity': 0.9200000166893005, 'average_primary_affinity_weighted': 0.8100000023841858}}, {'pool': 6, 'pool_name': 'default.rgw.meta', 'create_time': '2024-09-18T08:02:51.971465+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'is_stretch_pool': False, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '70', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4}, 'application_metadata': {'rgw': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.9700000286102295, 'score_stable': 1.9700000286102295, 'optimal_score': 1, 'raw_score_acting': 1.9700000286102295, 'raw_score_stable': 1.9700000286102295, 'primary_affinity_weighted': 0.8092708587646484, 'average_primary_affinity': 0.9200000166893005, 'average_primary_affinity_weighted': 0.8100000023841858}}, {'pool': 11, 'pool_name': 'unique_pool_4', 'create_time': '2024-09-18T08:18:25.969782+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'is_stretch_pool': False, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 16, 'pg_placement_num': 16, 'pg_placement_num_target': 16, 'pg_num_target': 16, 'pg_num_pending': 16, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '405', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rados': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.190000057220459, 'score_stable': 2.190000057220459, 'optimal_score': 1, 'raw_score_acting': 2.190000057220459, 'raw_score_stable': 2.190000057220459, 'primary_affinity_weighted': 0.8092708587646484, 'average_primary_affinity': 0.9200000166893005, 'average_primary_affinity_weighted': 0.8100000023841858}}], 'osds': [{'osd': 0, 'uuid': 'ae8f60df-72ac-46fb-91a1-f132b851c876', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 8, 'up_thru': 401, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.7:6801', 'nonce': 2896425794}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.7:6802', 'nonce': 2896425794}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.7:6804', 'nonce': 2896425794}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.7:6803', 'nonce': 2896425794}]}, 'public_addr': '172.21.15.7:6801/2896425794', 'cluster_addr': '172.21.15.7:6802/2896425794', 'heartbeat_back_addr': '172.21.15.7:6804/2896425794', 'heartbeat_front_addr': '172.21.15.7:6803/2896425794', 'state': ['exists', 'up']}, {'osd': 1, 'uuid': 'be28b70c-1f73-494f-ad4b-cb7ec5311999', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 13, 'last_clean_end': 243, 'up_from': 307, 'up_thru': 401, 'down_at': 244, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.7:6805', 'nonce': 3300107791}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.7:6806', 'nonce': 3300107791}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.7:6808', 'nonce': 3300107791}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.7:6807', 'nonce': 3300107791}]}, 'public_addr': '172.21.15.7:6805/3300107791', 'cluster_addr': '172.21.15.7:6806/3300107791', 'heartbeat_back_addr': '172.21.15.7:6808/3300107791', 'heartbeat_front_addr': '172.21.15.7:6807/3300107791', 'state': ['exists', 'up']}, {'osd': 2, 'uuid': 'cfa4f083-c7fd-4743-be65-84a3040fe6dc', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 18, 'up_thru': 401, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.7:6809', 'nonce': 556142559}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.7:6810', 'nonce': 556142559}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.7:6812', 'nonce': 556142559}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.7:6811', 'nonce': 556142559}]}, 'public_addr': '172.21.15.7:6809/556142559', 'cluster_addr': '172.21.15.7:6810/556142559', 'heartbeat_back_addr': '172.21.15.7:6812/556142559', 'heartbeat_front_addr': '172.21.15.7:6811/556142559', 'state': ['exists', 'up']}, {'osd': 3, 'uuid': '11ecd7e1-a751-4b75-95f5-00b32d2a5bc4', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 26, 'up_thru': 401, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.7:6813', 'nonce': 163587772}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.7:6814', 'nonce': 163587772}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.7:6816', 'nonce': 163587772}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.7:6815', 'nonce': 163587772}]}, 'public_addr': '172.21.15.7:6813/163587772', 'cluster_addr': '172.21.15.7:6814/163587772', 'heartbeat_back_addr': '172.21.15.7:6816/163587772', 'heartbeat_front_addr': '172.21.15.7:6815/163587772', 'state': ['exists', 'up']}, {'osd': 4, 'uuid': '98992b34-cfa1-45bf-97a1-7d557739252f', 'up': 1, 'in': 0, 'weight': 0, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 32, 'up_thru': 336, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.50:6800', 'nonce': 3662119805}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.50:6801', 'nonce': 3662119805}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.50:6803', 'nonce': 3662119805}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.50:6802', 'nonce': 3662119805}]}, 'public_addr': '172.21.15.50:6800/3662119805', 'cluster_addr': '172.21.15.50:6801/3662119805', 'heartbeat_back_addr': '172.21.15.50:6803/3662119805', 'heartbeat_front_addr': '172.21.15.50:6802/3662119805', 'state': ['exists', 'up']}, {'osd': 5, 'uuid': '7f290005-ebca-467b-87e4-02ead1063610', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 39, 'up_thru': 401, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.50:6804', 'nonce': 1199797954}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.50:6805', 'nonce': 1199797954}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.50:6807', 'nonce': 1199797954}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.50:6806', 'nonce': 1199797954}]}, 'public_addr': '172.21.15.50:6804/1199797954', 'cluster_addr': '172.21.15.50:6805/1199797954', 'heartbeat_back_addr': '172.21.15.50:6807/1199797954', 'heartbeat_front_addr': '172.21.15.50:6806/1199797954', 'state': ['exists', 'up']}, {'osd': 6, 'uuid': 'e07d846c-ae42-4fea-aa5c-3a25835d18a6', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 0, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 47, 'up_thru': 244, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.50:6808', 'nonce': 255960422}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.50:6809', 'nonce': 255960422}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.50:6811', 'nonce': 255960422}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.50:6810', 'nonce': 255960422}]}, 'public_addr': '172.21.15.50:6808/255960422', 'cluster_addr': '172.21.15.50:6809/255960422', 'heartbeat_back_addr': '172.21.15.50:6811/255960422', 'heartbeat_front_addr': '172.21.15.50:6810/255960422', 'state': ['exists', 'up']}, {'osd': 7, 'uuid': 'c9dac3b7-d651-489d-b96e-5b10fa32d9af', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 0.4741668701171875, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 56, 'up_thru': 401, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.50:6812', 'nonce': 874923904}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.50:6813', 'nonce': 874923904}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.50:6815', 'nonce': 874923904}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.50:6814', 'nonce': 874923904}]}, 'public_addr': '172.21.15.50:6812/874923904', 'cluster_addr': '172.21.15.50:6813/874923904', 'heartbeat_back_addr': '172.21.15.50:6815/874923904', 'heartbeat_front_addr': '172.21.15.50:6814/874923904', 'state': ['exists', 'up']}], 'osd_xinfo': [{'osd': 0, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540719139924082687, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-18T08:17:33.038707+0000', 'dead_epoch': 0}, {'osd': 1, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540719139924082687, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-18T08:18:24.029220+0000', 'dead_epoch': 243}, {'osd': 2, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540719139924082687, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-18T08:17:32.795625+0000', 'dead_epoch': 0}, {'osd': 3, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540719139924082687, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-18T08:18:06.641839+0000', 'dead_epoch': 0}, {'osd': 4, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540719139924082687, 'old_weight': 65536, 'last_purged_snaps_scrub': '2024-09-18T08:18:18.900056+0000', 'dead_epoch': 0}, {'osd': 5, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540719139924082687, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-18T08:18:24.016232+0000', 'dead_epoch': 0}, {'osd': 6, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540719139924082687, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-18T08:18:09.854168+0000', 'dead_epoch': 0}, {'osd': 7, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540719139924082687, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-18T08:17:42.528532+0000', 'dead_epoch': 0}], 'pg_upmap': [{'pgid': '3.c', 'osds': [3, 2, 6]}, {'pgid': '6.3', 'osds': [5, 3, 2]}], 'pg_upmap_items': [{'pgid': '3.8', 'mappings': [{'from': 3, 'to': 2}]}, {'pgid': '4.7', 'mappings': [{'from': 1, 'to': 2}]}, {'pgid': '4.14', 'mappings': [{'from': 1, 'to': 2}]}, {'pgid': '4.18', 'mappings': [{'from': 1, 'to': 2}]}, {'pgid': '4.1b', 'mappings': [{'from': 3, 'to': 5}]}, {'pgid': '4.1d', 'mappings': [{'from': 1, 'to': 0}]}, {'pgid': '5.b', 'mappings': [{'from': 0, 'to': 3}]}], 'pg_upmap_primaries': [], 'pg_temp': [], 'primary_temp': [], 'blocklist': {'172.21.15.7:0/398367658': '2024-09-19T08:03:17.052903+0000', '172.21.15.7:6800/2918496291': '2024-09-19T07:57:31.015269+0000', '172.21.15.7:0/2514354773': '2024-09-19T07:57:31.015269+0000', '172.21.15.7:0/3924911335': '2024-09-19T07:57:31.015269+0000', '172.21.15.7:0/1668598084': '2024-09-19T07:57:57.346149+0000', '172.21.15.7:0/4272027292': '2024-09-19T07:57:31.015269+0000', '172.21.15.7:6800/1013061071': '2024-09-19T07:57:57.346149+0000', '172.21.15.7:6800/1491061076': '2024-09-19T08:03:17.052903+0000', '172.21.15.7:0/533727535': '2024-09-19T07:57:57.346149+0000', '172.21.15.7:0/1605217610': '2024-09-19T08:03:17.052903+0000', '172.21.15.7:0/1722968065': '2024-09-19T08:03:17.052903+0000', '172.21.15.7:0/1277549846': '2024-09-19T07:57:57.346149+0000', '172.21.15.7:0/232452890': '2024-09-19T08:03:17.052903+0000', '172.21.15.7:0/3881670470': '2024-09-19T08:03:17.052903+0000'}, 'range_blocklist': {}, 'erasure_code_profiles': {'default': {'crush-failure-domain': 'osd', 'k': '2', 'm': '1', 'plugin': 'jerasure', 'technique': 'reed_sol_van'}}, 'removed_snaps_queue': [], 'new_removed_snaps': [], 'new_purged_snaps': [], 'crush_node_flags': {}, 'device_class_flags': {}, 'stretch_mode': {'stretch_mode_enabled': False, 'stretch_bucket_count': 0, 'degraded_stretch_mode': 0, 'recovering_stretch_mode': 0, 'stretch_mode_bucket': 0}} 2024-09-18T08:18:37.444 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid 779349ca-7593-11ef-bceb-c7b262605968 -- ceph pg dump --format=json 2024-09-18T08:18:37.504 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/779349ca-7593-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T08:18:37.514 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3 2024-09-18T08:18:37.517 INFO:teuthology.orchestra.run.smithi050.stderr:Inferring config /var/lib/ceph/779349ca-7593-11ef-bceb-c7b262605968/mon.b/config 2024-09-18T08:18:37.534 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/779349ca-7593-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T08:18:37.590 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:18:37.591 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:18:37.591 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3' 2024-09-18T08:18:37.692 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1 2024-09-18T08:18:37.743 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:18:37.744 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:18:37.745 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1' 2024-09-18T08:18:37.846 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2 2024-09-18T08:18:37.888 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:18:37.889 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:18:37.889 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2' 2024-09-18T08:18:37.990 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3 2024-09-18T08:18:38.030 INFO:tasks.radosbench.radosbench.0.smithi007.stdout: 3 15 6909 6894 143.613 159.5 0.0061142 0.00694993 2024-09-18T08:18:38.036 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:18:38.036 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:18:38.037 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3' 2024-09-18T08:18:38.109 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2024-09-18T08:18:38.109 INFO:teuthology.orchestra.run.smithi050.stdout: "ops": [ 2024-09-18T08:18:38.109 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:18:38.109 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.26087.0:7134 11.7 11:e2dac240:::benchmark_data_smithi007_172546_object7133:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536 in=65536b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e406)", 2024-09-18T08:18:38.109 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-09-18T08:18:38.106772+0000", 2024-09-18T08:18:38.109 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 0.00053077199999999999, 2024-09-18T08:18:38.109 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00055951600000000003, 2024-09-18T08:18:38.109 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-09-18T08:18:38.109 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-09-18T08:18:38.109 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "waiting for sub ops", 2024-09-18T08:18:38.109 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2024-09-18T08:18:38.109 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.26087", 2024-09-18T08:18:38.110 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "172.21.15.7:0/2752273185", 2024-09-18T08:18:38.110 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 7134 2024-09-18T08:18:38.110 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:18:38.110 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-09-18T08:18:38.110 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:18:38.110 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-09-18T08:18:38.110 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:18:38.106772+0000", 2024-09-18T08:18:38.110 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-18T08:18:38.110 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:18:38.110 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:18:38.110 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-09-18T08:18:38.110 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:18:38.106772+0000", 2024-09-18T08:18:38.111 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-18T08:18:38.111 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:18:38.111 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:18:38.111 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-09-18T08:18:38.111 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:18:38.106774+0000", 2024-09-18T08:18:38.111 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 2.0420000000000001e-06 2024-09-18T08:18:38.111 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:18:38.111 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:18:38.111 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-09-18T08:18:38.111 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:18:38.106851+0000", 2024-09-18T08:18:38.111 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 7.7145999999999996e-05 2024-09-18T08:18:38.111 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:18:38.111 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:18:38.112 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-09-18T08:18:38.112 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:18:38.106866+0000", 2024-09-18T08:18:38.112 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.4823e-05 2024-09-18T08:18:38.112 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:18:38.112 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:18:38.112 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-09-18T08:18:38.112 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:18:38.106875+0000", 2024-09-18T08:18:38.112 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 9.3090000000000001e-06 2024-09-18T08:18:38.112 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:18:38.113 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:18:38.113 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-18T08:18:38.113 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:18:38.106942+0000", 2024-09-18T08:18:38.113 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 6.6817999999999996e-05 2024-09-18T08:18:38.113 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:18:38.113 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:18:38.113 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-09-18T08:18:38.113 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:18:38.107147+0000", 2024-09-18T08:18:38.113 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00020427100000000001 2024-09-18T08:18:38.113 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:18:38.113 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:18:38.113 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "waiting for subops from 0,1", 2024-09-18T08:18:38.114 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:18:38.107288+0000", 2024-09-18T08:18:38.114 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000141438 2024-09-18T08:18:38.114 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-18T08:18:38.114 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-09-18T08:18:38.114 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-18T08:18:38.114 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:18:38.114 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:18:38.114 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_repop(client.26087.0:7129 11.8 e406/401 11:1c744ec2:::benchmark_data_smithi007_172546_object7128:head v 406'402, mlcod=406'400)", 2024-09-18T08:18:38.114 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-09-18T08:18:38.107034+0000", 2024-09-18T08:18:38.114 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 0.00026927100000000002, 2024-09-18T08:18:38.114 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000375991, 2024-09-18T08:18:38.115 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-09-18T08:18:38.115 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-09-18T08:18:38.115 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "started", 2024-09-18T08:18:38.115 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-09-18T08:18:38.115 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:18:38.115 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-09-18T08:18:38.115 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:18:38.107034+0000", 2024-09-18T08:18:38.115 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-18T08:18:38.115 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:18:38.115 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:18:38.115 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-09-18T08:18:38.115 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:18:38.107034+0000", 2024-09-18T08:18:38.115 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-18T08:18:38.116 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:18:38.116 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:18:38.116 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-09-18T08:18:38.116 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:18:38.107034+0000", 2024-09-18T08:18:38.116 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 7.2500000000000005e-07 2024-09-18T08:18:38.116 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:18:38.116 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:18:38.116 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-09-18T08:18:38.116 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:18:38.107086+0000", 2024-09-18T08:18:38.116 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 5.1663000000000003e-05 2024-09-18T08:18:38.116 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:18:38.116 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:18:38.116 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-09-18T08:18:38.116 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:18:38.107097+0000", 2024-09-18T08:18:38.117 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.1085e-05 2024-09-18T08:18:38.117 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:18:38.117 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:18:38.117 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-09-18T08:18:38.117 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:18:38.107106+0000", 2024-09-18T08:18:38.117 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 8.9579999999999996e-06 2024-09-18T08:18:38.117 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:18:38.117 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:18:38.117 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-18T08:18:38.117 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:18:38.107164+0000", 2024-09-18T08:18:38.117 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 5.7737e-05 2024-09-18T08:18:38.117 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:18:38.118 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:18:38.118 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-09-18T08:18:38.118 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:18:38.107186+0000", 2024-09-18T08:18:38.118 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 2.1963999999999998e-05 2024-09-18T08:18:38.118 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-18T08:18:38.118 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-09-18T08:18:38.118 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-18T08:18:38.118 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-18T08:18:38.118 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2024-09-18T08:18:38.118 INFO:teuthology.orchestra.run.smithi050.stdout: "num_ops": 2 2024-09-18T08:18:38.118 INFO:teuthology.orchestra.run.smithi050.stdout:} 2024-09-18T08:18:38.138 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6 2024-09-18T08:18:38.172 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:18:38.173 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:18:38.173 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6' 2024-09-18T08:18:38.274 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4 2024-09-18T08:18:38.302 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:18:38.303 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:18:38.304 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4' 2024-09-18T08:18:38.404 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2 2024-09-18T08:18:38.429 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:18:38.430 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:18:38.430 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2' 2024-09-18T08:18:38.531 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5 2024-09-18T08:18:38.560 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:18:38.560 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:18:38.561 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5' 2024-09-18T08:18:38.661 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0 2024-09-18T08:18:38.702 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:18:38.703 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:18:38.703 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0' 2024-09-18T08:18:38.804 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5 2024-09-18T08:18:38.833 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:18:38.833 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:18:38.834 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5' 2024-09-18T08:18:38.926 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:18:38 smithi007 ceph-mon[29459]: pgmap v796: 158 pgs: 158 active+clean; 583 KiB data, 3.8 GiB used, 622 GiB / 626 GiB avail; 639 B/s rd, 0 op/s; 76 B/s, 2 objects/s recovering 2024-09-18T08:18:38.926 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:18:38 smithi007 ceph-mon[29459]: purged_snaps scrub starts 2024-09-18T08:18:38.926 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:18:38 smithi007 ceph-mon[29459]: purged_snaps scrub ok 2024-09-18T08:18:38.927 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:18:38 smithi007 ceph-mon[39249]: pgmap v796: 158 pgs: 158 active+clean; 583 KiB data, 3.8 GiB used, 622 GiB / 626 GiB avail; 639 B/s rd, 0 op/s; 76 B/s, 2 objects/s recovering 2024-09-18T08:18:38.927 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:18:38 smithi007 ceph-mon[39249]: purged_snaps scrub starts 2024-09-18T08:18:38.927 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:18:38 smithi007 ceph-mon[39249]: purged_snaps scrub ok 2024-09-18T08:18:38.934 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3 2024-09-18T08:18:38.976 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:18:38.977 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:18:38.977 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3' 2024-09-18T08:18:39.032 INFO:tasks.radosbench.radosbench.0.smithi007.stdout: 4 16 9511 9495 148.279 162.562 0.00548098 0.00673102 2024-09-18T08:18:39.079 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2 2024-09-18T08:18:39.129 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:18:39.130 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:18:39.130 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2' 2024-09-18T08:18:39.155 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:18:38 smithi050 ceph-mon[32371]: pgmap v796: 158 pgs: 158 active+clean; 583 KiB data, 3.8 GiB used, 622 GiB / 626 GiB avail; 639 B/s rd, 0 op/s; 76 B/s, 2 objects/s recovering 2024-09-18T08:18:39.155 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:18:38 smithi050 ceph-mon[32371]: purged_snaps scrub starts 2024-09-18T08:18:39.155 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:18:38 smithi050 ceph-mon[32371]: purged_snaps scrub ok 2024-09-18T08:18:39.231 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2 2024-09-18T08:18:39.275 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:18:39.276 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:18:39.277 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2' 2024-09-18T08:18:39.377 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6 2024-09-18T08:18:39.407 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:18:39.408 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:18:39.409 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6' 2024-09-18T08:18:39.435 INFO:teuthology.orchestra.run.smithi007.stderr:nodeep-scrub is set 2024-09-18T08:18:39.510 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2 2024-09-18T08:18:39.549 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:18:39.549 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:18:39.550 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2' 2024-09-18T08:18:39.632 DEBUG:teuthology.orchestra.run.smithi050:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid 779349ca-7593-11ef-bceb-c7b262605968 -- ceph daemon osd.5 dump_blocked_ops 2024-09-18T08:18:39.650 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4 2024-09-18T08:18:39.709 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:18:39.710 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:18:39.711 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4' 2024-09-18T08:18:39.754 INFO:teuthology.orchestra.run.smithi007.stdout:osd.0: {} 2024-09-18T08:18:39.754 INFO:teuthology.orchestra.run.smithi007.stdout:osd.1: {} 2024-09-18T08:18:39.754 INFO:teuthology.orchestra.run.smithi007.stdout:osd.2: {} 2024-09-18T08:18:39.754 INFO:teuthology.orchestra.run.smithi007.stdout:osd.3: {} 2024-09-18T08:18:39.754 INFO:teuthology.orchestra.run.smithi007.stdout:osd.4: {} 2024-09-18T08:18:39.754 INFO:teuthology.orchestra.run.smithi007.stdout:osd.5: {} 2024-09-18T08:18:39.754 INFO:teuthology.orchestra.run.smithi007.stdout:osd.6: {} 2024-09-18T08:18:39.754 INFO:teuthology.orchestra.run.smithi007.stdout:osd.7: {} 2024-09-18T08:18:39.754 INFO:teuthology.orchestra.run.smithi007.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-18T08:18:39.754 INFO:teuthology.orchestra.run.smithi007.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-18T08:18:39.755 INFO:teuthology.orchestra.run.smithi007.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-18T08:18:39.755 INFO:teuthology.orchestra.run.smithi007.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-18T08:18:39.755 INFO:teuthology.orchestra.run.smithi007.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-18T08:18:39.755 INFO:teuthology.orchestra.run.smithi007.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-18T08:18:39.755 INFO:teuthology.orchestra.run.smithi007.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-18T08:18:39.755 INFO:teuthology.orchestra.run.smithi007.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-18T08:18:39.811 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2 2024-09-18T08:18:39.851 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:18:39.851 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:18:39.852 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2' 2024-09-18T08:18:39.906 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:18:39 smithi050 ceph-mon[32371]: osdmap e407: 8 total, 8 up, 7 in 2024-09-18T08:18:39.907 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:18:39 smithi050 ceph-mon[32371]: from='client.? 172.21.15.7:0/1445629359' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-18T08:18:39.907 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:18:39 smithi050 ceph-mon[32371]: from='client.? 172.21.15.7:0/2588805105' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-18T08:18:39.907 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:18:39 smithi050 ceph-mon[32371]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-18T08:18:39.907 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:18:39 smithi050 ceph-mon[32371]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-18T08:18:39.907 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:18:39 smithi050 ceph-mon[32371]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-18T08:18:39.907 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:18:39 smithi050 ceph-mon[32371]: osdmap e408: 8 total, 8 up, 7 in 2024-09-18T08:18:39.907 INFO:journalctl@ceph.iscsi.iscsi.a.smithi050.stdout:Sep 18 08:18:39 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-iscsi-iscsi-a[57782]: debug there is no tcmu-runner data available 2024-09-18T08:18:39.953 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4 2024-09-18T08:18:39.980 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:18:39 smithi007 ceph-mon[29459]: osdmap e407: 8 total, 8 up, 7 in 2024-09-18T08:18:39.980 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:18:39 smithi007 ceph-mon[29459]: from='client.? 172.21.15.7:0/1445629359' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-18T08:18:39.980 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:18:39 smithi007 ceph-mon[29459]: from='client.? 172.21.15.7:0/2588805105' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-18T08:18:39.980 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:18:39 smithi007 ceph-mon[29459]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-18T08:18:39.980 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:18:39 smithi007 ceph-mon[29459]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-18T08:18:39.980 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:18:39 smithi007 ceph-mon[29459]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-18T08:18:39.980 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:18:39 smithi007 ceph-mon[29459]: osdmap e408: 8 total, 8 up, 7 in 2024-09-18T08:18:39.980 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:18:39 smithi007 ceph-mon[39249]: osdmap e407: 8 total, 8 up, 7 in 2024-09-18T08:18:39.980 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:18:39 smithi007 ceph-mon[39249]: from='client.? 172.21.15.7:0/1445629359' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-18T08:18:39.981 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:18:39 smithi007 ceph-mon[39249]: from='client.? 172.21.15.7:0/2588805105' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-18T08:18:39.981 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:18:39 smithi007 ceph-mon[39249]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-18T08:18:39.981 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:18:39 smithi007 ceph-mon[39249]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-18T08:18:39.981 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:18:39 smithi007 ceph-mon[39249]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-18T08:18:39.981 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:18:39 smithi007 ceph-mon[39249]: osdmap e408: 8 total, 8 up, 7 in 2024-09-18T08:18:39.985 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:18:39.986 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:18:39.986 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4' 2024-09-18T08:18:39.994 INFO:teuthology.orchestra.run.smithi050.stderr:Inferring config /var/lib/ceph/779349ca-7593-11ef-bceb-c7b262605968/mon.b/config 2024-09-18T08:18:40.032 INFO:tasks.radosbench.radosbench.0.smithi007.stdout: 5 16 11773 11757 146.897 141.375 0.00335903 0.00679741 2024-09-18T08:18:40.088 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2 2024-09-18T08:18:40.127 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:18:40.127 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:18:40.128 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2' 2024-09-18T08:18:40.229 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3 2024-09-18T08:18:40.269 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:18:40.270 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:18:40.270 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3' 2024-09-18T08:18:40.371 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7 2024-09-18T08:18:40.401 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:18:40.401 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:18:40.402 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7' 2024-09-18T08:18:40.502 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1 2024-09-18T08:18:40.544 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:18:40.544 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:18:40.545 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1' 2024-09-18T08:18:40.578 INFO:teuthology.orchestra.run.smithi050.stdout:ERROR: (22) Invalid argument 2024-09-18T08:18:40.579 INFO:teuthology.orchestra.run.smithi050.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-18T08:18:40.645 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1 2024-09-18T08:18:40.686 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:18:40.687 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:18:40.687 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1' 2024-09-18T08:18:40.788 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3 2024-09-18T08:18:40.827 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:18:40.828 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:18:40.829 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3' 2024-09-18T08:18:40.905 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:18:40 smithi050 ceph-mon[32371]: pgmap v799: 158 pgs: 158 active+clean; 523 MiB data, 5.1 GiB used, 621 GiB / 626 GiB avail; 853 B/s rd, 87 MiB/s wr, 1.39k op/s 2024-09-18T08:18:40.905 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:18:40 smithi050 ceph-mon[32371]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-18T08:18:40.925 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:18:40 smithi007 ceph-mon[29459]: pgmap v799: 158 pgs: 158 active+clean; 523 MiB data, 5.1 GiB used, 621 GiB / 626 GiB avail; 853 B/s rd, 87 MiB/s wr, 1.39k op/s 2024-09-18T08:18:40.925 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:18:40 smithi007 ceph-mon[29459]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-18T08:18:40.925 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:18:40 smithi007 ceph-mon[39249]: pgmap v799: 158 pgs: 158 active+clean; 523 MiB data, 5.1 GiB used, 621 GiB / 626 GiB avail; 853 B/s rd, 87 MiB/s wr, 1.39k op/s 2024-09-18T08:18:40.925 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:18:40 smithi007 ceph-mon[39249]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-18T08:18:40.930 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3 2024-09-18T08:18:40.974 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:18:40.974 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:18:40.975 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3' 2024-09-18T08:18:41.032 INFO:tasks.radosbench.radosbench.0.smithi007.stdout: 6 16 14311 14295 148.849 158.625 0.00544107 0.00671002 2024-09-18T08:18:41.076 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3 2024-09-18T08:18:41.115 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:18:41.116 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:18:41.116 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3' 2024-09-18T08:18:41.217 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7 2024-09-18T08:18:41.247 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:18:41.248 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:18:41.248 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7' 2024-09-18T08:18:41.349 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6 2024-09-18T08:18:41.376 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:18:41.377 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:18:41.378 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6' 2024-09-18T08:18:41.421 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/779349ca-7593-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T08:18:41.462 DEBUG:teuthology.orchestra.run.smithi050:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid 779349ca-7593-11ef-bceb-c7b262605968 -- ceph daemon osd.5 dump_historic_ops 2024-09-18T08:18:41.478 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4 2024-09-18T08:18:41.544 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:18:41.545 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:18:41.545 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4' 2024-09-18T08:18:41.646 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6 2024-09-18T08:18:41.680 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:18:41.680 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:18:41.681 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6' 2024-09-18T08:18:41.781 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1 2024-09-18T08:18:41.820 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:18:41.821 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:18:41.822 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1' 2024-09-18T08:18:41.831 INFO:teuthology.orchestra.run.smithi050.stderr:Inferring config /var/lib/ceph/779349ca-7593-11ef-bceb-c7b262605968/mon.b/config 2024-09-18T08:18:41.922 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0 2024-09-18T08:18:41.967 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:18:41.968 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:18:41.968 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0' 2024-09-18T08:18:42.032 INFO:tasks.radosbench.radosbench.0.smithi007.stdout: 7 15 16711 16696 149.02 150.062 0.00809677 0.00670254 2024-09-18T08:18:42.069 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7 2024-09-18T08:18:42.100 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:18:42.101 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:18:42.102 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7' 2024-09-18T08:18:42.202 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3 2024-09-18T08:18:42.240 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:18:42.240 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:18:42.240 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3' 2024-09-18T08:18:42.341 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4 2024-09-18T08:18:42.373 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:18:42.374 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:18:42.374 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4' 2024-09-18T08:18:42.456 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-09-18T08:18:42.458 INFO:teuthology.orchestra.run.smithi007.stderr:dumped all 2024-09-18T08:18:42.475 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2 2024-09-18T08:18:42.497 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:18:42.497 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:18:42.498 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2' 2024-09-18T08:18:42.599 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1 2024-09-18T08:18:42.605 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid 779349ca-7593-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-18T08:18:42.637 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:18:42.638 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:18:42.638 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1' 2024-09-18T08:18:42.730 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:18:42 smithi007 ceph-mon[29459]: pgmap v800: 158 pgs: 158 active+clean; 523 MiB data, 5.1 GiB used, 621 GiB / 626 GiB avail; 728 B/s rd, 74 MiB/s wr, 1.19k op/s 2024-09-18T08:18:42.730 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:18:42 smithi007 ceph-mon[39249]: pgmap v800: 158 pgs: 158 active+clean; 523 MiB data, 5.1 GiB used, 621 GiB / 626 GiB avail; 728 B/s rd, 74 MiB/s wr, 1.19k op/s 2024-09-18T08:18:42.740 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7 2024-09-18T08:18:42.747 INFO:teuthology.orchestra.run.smithi050.stdout:ERROR: (22) Invalid argument 2024-09-18T08:18:42.747 INFO:teuthology.orchestra.run.smithi050.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-18T08:18:42.770 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:18:42.771 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:18:42.772 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7' 2024-09-18T08:18:42.872 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2 2024-09-18T08:18:42.928 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:18:42.929 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:18:42.929 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2' 2024-09-18T08:18:43.030 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5 2024-09-18T08:18:43.033 INFO:tasks.radosbench.radosbench.0.smithi007.stdout: 8 16 18715 18699 146.021 125.188 0.00301704 0.0068419 2024-09-18T08:18:43.059 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:18:43.060 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:18:43.061 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5' 2024-09-18T08:18:43.155 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:18:42 smithi050 ceph-mon[32371]: pgmap v800: 158 pgs: 158 active+clean; 523 MiB data, 5.1 GiB used, 621 GiB / 626 GiB avail; 728 B/s rd, 74 MiB/s wr, 1.19k op/s 2024-09-18T08:18:43.161 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7 2024-09-18T08:18:43.193 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:18:43.193 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:18:43.194 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7' 2024-09-18T08:18:43.295 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5 2024-09-18T08:18:43.322 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:18:43.323 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:18:43.323 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5' 2024-09-18T08:18:43.349 INFO:teuthology.orchestra.run.smithi007.stdout:{"pg_ready":true,"pg_map":{"version":800,"stamp":"2024-09-18T08:18:41.453914+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":548411931,"num_objects":8558,"num_object_clones":0,"num_object_copies":25674,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8558,"num_whiteouts":0,"num_read":1526,"num_read_kb":1278,"num_write":9081,"num_write_kb":535021,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":583,"num_bytes_recovered":1591056,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":10053,"ondisk_log_size":10053,"up":474,"acting":474,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":474,"num_osds":7,"num_per_pool_osds":7,"num_per_pool_omap_osds":7,"kb":656216064,"kb_used":5301452,"kb_used_data":1351640,"kb_used_omap":17,"kb_used_meta":3949678,"kb_avail":650914612,"statfs":{"total":671965249536,"available":666536562688,"internally_reserved":0,"allocated":1384079360,"data_stored":1380131311,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17639,"internal_metadata":4044471065},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[5,0,1,4],"upper_bound":16},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":547815424,"num_objects":8359,"num_object_clones":0,"num_object_copies":25077,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8359,"num_whiteouts":0,"num_read":5,"num_read_kb":5,"num_write":8359,"num_write_kb":534976,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"7.027894"},"pg_stats":[{"pgid":"3.1f","version":"0'0","reported_seq":774,"reported_epoch":406,"state":"active+clean","last_fresh":"2024-09-18T08:18:34.922750+0000","last_change":"2024-09-18T08:18:22.822605+0000","last_active":"2024-09-18T08:18:34.922750+0000","last_peered":"2024-09-18T08:18:34.922750+0000","last_clean":"2024-09-18T08:18:34.922750+0000","last_became_active":"2024-09-18T08:15:41.451012+0000","last_became_peered":"2024-09-18T08:15:41.451012+0000","last_unstale":"2024-09-18T08:18:34.922750+0000","last_undegraded":"2024-09-18T08:18:34.922750+0000","last_fullsized":"2024-09-18T08:18:34.922750+0000","mapping_epoch":336,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":337,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:18:22.822559+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:46.860092+0000","last_clean_scrub_stamp":"2024-09-18T08:18:22.822559+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:19:49.749025+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"271'15","reported_seq":502,"reported_epoch":407,"state":"active+clean","last_fresh":"2024-09-18T08:18:38.671991+0000","last_change":"2024-09-18T08:16:13.499531+0000","last_active":"2024-09-18T08:18:38.671991+0000","last_peered":"2024-09-18T08:18:38.671991+0000","last_clean":"2024-09-18T08:18:38.671991+0000","last_became_active":"2024-09-18T08:16:13.456332+0000","last_became_peered":"2024-09-18T08:16:13.456332+0000","last_unstale":"2024-09-18T08:18:38.671991+0000","last_undegraded":"2024-09-18T08:18:38.671991+0000","last_fullsized":"2024-09-18T08:18:38.671991+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"271'15","last_scrub_stamp":"2024-09-18T08:15:04.676980+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_clean_scrub_stamp":"2024-09-18T08:15:04.676980+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:19:11.019339+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":25,"num_read_kb":19,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,3],"acting":[2,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":598,"reported_epoch":406,"state":"active+clean","last_fresh":"2024-09-18T08:18:34.427581+0000","last_change":"2024-09-18T08:18:27.641641+0000","last_active":"2024-09-18T08:18:34.427581+0000","last_peered":"2024-09-18T08:18:34.427581+0000","last_clean":"2024-09-18T08:18:34.427581+0000","last_became_active":"2024-09-18T08:18:27.641323+0000","last_became_peered":"2024-09-18T08:18:27.641323+0000","last_unstale":"2024-09-18T08:18:34.427581+0000","last_undegraded":"2024-09-18T08:18:34.427581+0000","last_fullsized":"2024-09-18T08:18:34.427581+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:18:21.610930+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:50.886830+0000","last_clean_scrub_stamp":"2024-09-18T08:18:21.610930+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:19:39.039833+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":491,"reported_epoch":407,"state":"active+clean","last_fresh":"2024-09-18T08:18:38.668004+0000","last_change":"2024-09-18T08:18:27.639074+0000","last_active":"2024-09-18T08:18:38.668004+0000","last_peered":"2024-09-18T08:18:38.668004+0000","last_clean":"2024-09-18T08:18:38.668004+0000","last_became_active":"2024-09-18T08:18:27.638698+0000","last_became_peered":"2024-09-18T08:18:27.638698+0000","last_unstale":"2024-09-18T08:18:38.668004+0000","last_undegraded":"2024-09-18T08:18:38.668004+0000","last_fullsized":"2024-09-18T08:18:38.668004+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:17:40.565100+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_clean_scrub_stamp":"2024-09-18T08:17:40.565100+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:19:00.161214+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":769,"reported_epoch":407,"state":"active+clean","last_fresh":"2024-09-18T08:18:38.673996+0000","last_change":"2024-09-18T08:18:27.662859+0000","last_active":"2024-09-18T08:18:38.673996+0000","last_peered":"2024-09-18T08:18:38.673996+0000","last_clean":"2024-09-18T08:18:38.673996+0000","last_became_active":"2024-09-18T08:18:27.662545+0000","last_became_peered":"2024-09-18T08:18:27.662545+0000","last_unstale":"2024-09-18T08:18:38.673996+0000","last_undegraded":"2024-09-18T08:18:38.673996+0000","last_fullsized":"2024-09-18T08:18:38.673996+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:15:51.826050+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_clean_scrub_stamp":"2024-09-18T08:15:51.826050+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":771,"reported_epoch":407,"state":"active+clean","last_fresh":"2024-09-18T08:18:38.675094+0000","last_change":"2024-09-18T08:17:38.359429+0000","last_active":"2024-09-18T08:18:38.675094+0000","last_peered":"2024-09-18T08:18:38.675094+0000","last_clean":"2024-09-18T08:18:38.675094+0000","last_became_active":"2024-09-18T08:06:24.126194+0000","last_became_peered":"2024-09-18T08:06:24.126194+0000","last_unstale":"2024-09-18T08:18:38.675094+0000","last_undegraded":"2024-09-18T08:18:38.675094+0000","last_fullsized":"2024-09-18T08:18:38.675094+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:17:38.359366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:46.860092+0000","last_clean_scrub_stamp":"2024-09-18T08:17:38.359366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:19:07.133806+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"271'27","reported_seq":763,"reported_epoch":407,"state":"active+clean","last_fresh":"2024-09-18T08:18:38.673187+0000","last_change":"2024-09-18T08:14:26.565324+0000","last_active":"2024-09-18T08:18:38.673187+0000","last_peered":"2024-09-18T08:18:38.673187+0000","last_clean":"2024-09-18T08:18:38.673187+0000","last_became_active":"2024-09-18T08:02:49.886106+0000","last_became_peered":"2024-09-18T08:02:49.886106+0000","last_unstale":"2024-09-18T08:18:38.673187+0000","last_undegraded":"2024-09-18T08:18:38.673187+0000","last_fullsized":"2024-09-18T08:18:38.673187+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"271'27","last_scrub_stamp":"2024-09-18T08:14:26.565071+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_clean_scrub_stamp":"2024-09-18T08:14:26.565071+0000","objects_scrubbed":7,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:19:00.445844+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":44,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":395,"reported_epoch":406,"state":"active+clean","last_fresh":"2024-09-18T08:18:34.428710+0000","last_change":"2024-09-18T08:16:13.483763+0000","last_active":"2024-09-18T08:18:34.428710+0000","last_peered":"2024-09-18T08:18:34.428710+0000","last_clean":"2024-09-18T08:18:34.428710+0000","last_became_active":"2024-09-18T08:16:13.474963+0000","last_became_peered":"2024-09-18T08:16:13.474963+0000","last_unstale":"2024-09-18T08:18:34.428710+0000","last_undegraded":"2024-09-18T08:18:34.428710+0000","last_fullsized":"2024-09-18T08:18:34.428710+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:15:44.585123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:50.886830+0000","last_clean_scrub_stamp":"2024-09-18T08:15:44.585123+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":168,"reported_epoch":407,"state":"active+clean","last_fresh":"2024-09-18T08:18:38.668960+0000","last_change":"2024-09-18T08:18:17.566526+0000","last_active":"2024-09-18T08:18:38.668960+0000","last_peered":"2024-09-18T08:18:38.668960+0000","last_clean":"2024-09-18T08:18:38.668960+0000","last_became_active":"2024-09-18T08:16:13.458654+0000","last_became_peered":"2024-09-18T08:16:13.458654+0000","last_unstale":"2024-09-18T08:18:38.668960+0000","last_undegraded":"2024-09-18T08:18:38.668960+0000","last_fullsized":"2024-09-18T08:18:38.668960+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:18:17.566467+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:46.860092+0000","last_clean_scrub_stamp":"2024-09-18T08:18:17.566467+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:19:32.066482+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,1],"acting":[5,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"271'15","reported_seq":371,"reported_epoch":407,"state":"active+clean","last_fresh":"2024-09-18T08:18:38.674804+0000","last_change":"2024-09-18T08:18:31.669144+0000","last_active":"2024-09-18T08:18:38.674804+0000","last_peered":"2024-09-18T08:18:38.674804+0000","last_clean":"2024-09-18T08:18:38.674804+0000","last_became_active":"2024-09-18T08:16:13.463397+0000","last_became_peered":"2024-09-18T08:16:13.463397+0000","last_unstale":"2024-09-18T08:18:38.674804+0000","last_undegraded":"2024-09-18T08:18:38.674804+0000","last_fullsized":"2024-09-18T08:18:38.674804+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"271'15","last_scrub_stamp":"2024-09-18T08:18:31.669089+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_clean_scrub_stamp":"2024-09-18T08:18:31.669089+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:19:39.096555+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":24,"num_read_kb":18,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":175,"reported_epoch":407,"state":"active+clean","last_fresh":"2024-09-18T08:18:38.668840+0000","last_change":"2024-09-18T08:18:27.651580+0000","last_active":"2024-09-18T08:18:38.668840+0000","last_peered":"2024-09-18T08:18:38.668840+0000","last_clean":"2024-09-18T08:18:38.668840+0000","last_became_active":"2024-09-18T08:18:27.650998+0000","last_became_peered":"2024-09-18T08:18:27.650998+0000","last_unstale":"2024-09-18T08:18:38.668840+0000","last_undegraded":"2024-09-18T08:18:38.668840+0000","last_fullsized":"2024-09-18T08:18:38.668840+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:17:30.547979+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:50.886830+0000","last_clean_scrub_stamp":"2024-09-18T08:17:30.547979+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:18:50.273358+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":741,"reported_epoch":406,"state":"active+clean","last_fresh":"2024-09-18T08:18:34.921409+0000","last_change":"2024-09-18T08:18:27.645124+0000","last_active":"2024-09-18T08:18:34.921409+0000","last_peered":"2024-09-18T08:18:34.921409+0000","last_clean":"2024-09-18T08:18:34.921409+0000","last_became_active":"2024-09-18T08:18:27.644825+0000","last_became_peered":"2024-09-18T08:18:27.644825+0000","last_unstale":"2024-09-18T08:18:34.921409+0000","last_undegraded":"2024-09-18T08:18:34.921409+0000","last_fullsized":"2024-09-18T08:18:34.921409+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:17:29.542288+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_clean_scrub_stamp":"2024-09-18T08:17:29.542288+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:18:41.491186+0000","scrub_duration":18,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":160,"reported_epoch":407,"state":"active+clean","last_fresh":"2024-09-18T08:18:38.668817+0000","last_change":"2024-09-18T08:16:13.471332+0000","last_active":"2024-09-18T08:18:38.668817+0000","last_peered":"2024-09-18T08:18:38.668817+0000","last_clean":"2024-09-18T08:18:38.668817+0000","last_became_active":"2024-09-18T08:16:13.471003+0000","last_became_peered":"2024-09-18T08:16:13.471003+0000","last_unstale":"2024-09-18T08:18:38.668817+0000","last_undegraded":"2024-09-18T08:18:38.668817+0000","last_fullsized":"2024-09-18T08:18:38.668817+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:15:01.636504+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:46.860092+0000","last_clean_scrub_stamp":"2024-09-18T08:15:01.636504+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1b","version":"271'7","reported_seq":44,"reported_epoch":407,"state":"active+clean","last_fresh":"2024-09-18T08:18:38.668642+0000","last_change":"2024-09-18T08:18:27.484997+0000","last_active":"2024-09-18T08:18:38.668642+0000","last_peered":"2024-09-18T08:18:38.668642+0000","last_clean":"2024-09-18T08:18:38.668642+0000","last_became_active":"2024-09-18T08:18:19.431475+0000","last_became_peered":"2024-09-18T08:18:19.431475+0000","last_unstale":"2024-09-18T08:18:38.668642+0000","last_undegraded":"2024-09-18T08:18:38.668642+0000","last_fullsized":"2024-09-18T08:18:38.668642+0000","mapping_epoch":397,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":398,"parent":"0.0","parent_split_bits":0,"last_scrub":"271'7","last_scrub_stamp":"2024-09-18T08:18:27.484925+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_clean_scrub_stamp":"2024-09-18T08:18:27.484925+0000","objects_scrubbed":2,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:19:31.516926+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":13,"num_read_kb":9,"num_write":8,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,2],"acting":[5,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":20,"reported_epoch":406,"state":"active+clean","last_fresh":"2024-09-18T08:18:34.427787+0000","last_change":"2024-09-18T08:18:27.656772+0000","last_active":"2024-09-18T08:18:34.427787+0000","last_peered":"2024-09-18T08:18:34.427787+0000","last_clean":"2024-09-18T08:18:34.427787+0000","last_became_active":"2024-09-18T08:18:27.656359+0000","last_became_peered":"2024-09-18T08:18:27.656359+0000","last_unstale":"2024-09-18T08:18:34.427787+0000","last_undegraded":"2024-09-18T08:18:34.427787+0000","last_fullsized":"2024-09-18T08:18:34.427787+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:15:40.589048+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:50.886830+0000","last_clean_scrub_stamp":"2024-09-18T08:15:40.589048+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":161,"reported_epoch":407,"state":"active+clean","last_fresh":"2024-09-18T08:18:38.668050+0000","last_change":"2024-09-18T08:18:22.573319+0000","last_active":"2024-09-18T08:18:38.668050+0000","last_peered":"2024-09-18T08:18:38.668050+0000","last_clean":"2024-09-18T08:18:38.668050+0000","last_became_active":"2024-09-18T08:15:41.441932+0000","last_became_peered":"2024-09-18T08:15:41.441932+0000","last_unstale":"2024-09-18T08:18:38.668050+0000","last_undegraded":"2024-09-18T08:18:38.668050+0000","last_fullsized":"2024-09-18T08:18:38.668050+0000","mapping_epoch":336,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":337,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:18:22.573259+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_clean_scrub_stamp":"2024-09-18T08:18:22.573259+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:19:32.335476+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":467,"reported_epoch":407,"state":"active+clean","last_fresh":"2024-09-18T08:18:38.669019+0000","last_change":"2024-09-18T08:16:13.453737+0000","last_active":"2024-09-18T08:18:38.669019+0000","last_peered":"2024-09-18T08:18:38.669019+0000","last_clean":"2024-09-18T08:18:38.669019+0000","last_became_active":"2024-09-18T08:16:13.453129+0000","last_became_peered":"2024-09-18T08:16:13.453129+0000","last_unstale":"2024-09-18T08:18:38.669019+0000","last_undegraded":"2024-09-18T08:18:38.669019+0000","last_fullsized":"2024-09-18T08:18:38.669019+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:15:42.600840+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_clean_scrub_stamp":"2024-09-18T08:15:42.600840+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,3],"acting":[5,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":777,"reported_epoch":406,"state":"active+clean","last_fresh":"2024-09-18T08:18:34.923420+0000","last_change":"2024-09-18T08:18:30.893923+0000","last_active":"2024-09-18T08:18:34.923420+0000","last_peered":"2024-09-18T08:18:34.923420+0000","last_clean":"2024-09-18T08:18:34.923420+0000","last_became_active":"2024-09-18T08:18:27.643517+0000","last_became_peered":"2024-09-18T08:18:27.643517+0000","last_unstale":"2024-09-18T08:18:34.923420+0000","last_undegraded":"2024-09-18T08:18:34.923420+0000","last_fullsized":"2024-09-18T08:18:34.923420+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:18:30.893875+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:46.860092+0000","last_clean_scrub_stamp":"2024-09-18T08:18:30.893875+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:19:46.217695+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,1],"acting":[0,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1c","version":"271'27","reported_seq":499,"reported_epoch":407,"state":"active+clean","last_fresh":"2024-09-18T08:18:38.672521+0000","last_change":"2024-09-18T08:14:40.711299+0000","last_active":"2024-09-18T08:18:38.672521+0000","last_peered":"2024-09-18T08:18:38.672521+0000","last_clean":"2024-09-18T08:18:38.672521+0000","last_became_active":"2024-09-18T08:14:39.480913+0000","last_became_peered":"2024-09-18T08:14:39.480913+0000","last_unstale":"2024-09-18T08:18:38.672521+0000","last_undegraded":"2024-09-18T08:18:38.672521+0000","last_fullsized":"2024-09-18T08:18:38.672521+0000","mapping_epoch":307,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":308,"parent":"0.0","parent_split_bits":0,"last_scrub":"271'27","last_scrub_stamp":"2024-09-18T08:14:40.711228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_clean_scrub_stamp":"2024-09-18T08:14:40.711228+0000","objects_scrubbed":7,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:18:52.983685+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":44,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":619,"reported_epoch":406,"state":"active+clean","last_fresh":"2024-09-18T08:18:34.427908+0000","last_change":"2024-09-18T08:17:36.633467+0000","last_active":"2024-09-18T08:18:34.427908+0000","last_peered":"2024-09-18T08:18:34.427908+0000","last_clean":"2024-09-18T08:18:34.427908+0000","last_became_active":"2024-09-18T08:16:13.462303+0000","last_became_peered":"2024-09-18T08:16:13.462303+0000","last_unstale":"2024-09-18T08:18:34.427908+0000","last_undegraded":"2024-09-18T08:18:34.427908+0000","last_fullsized":"2024-09-18T08:18:34.427908+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:17:36.633392+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:50.886830+0000","last_clean_scrub_stamp":"2024-09-18T08:17:36.633392+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:18:36.654140+0000","scrub_duration":20,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,5],"acting":[1,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":749,"reported_epoch":407,"state":"active+clean","last_fresh":"2024-09-18T08:18:38.675220+0000","last_change":"2024-09-18T08:15:41.469278+0000","last_active":"2024-09-18T08:18:38.675220+0000","last_peered":"2024-09-18T08:18:38.675220+0000","last_clean":"2024-09-18T08:18:38.675220+0000","last_became_active":"2024-09-18T08:15:41.466371+0000","last_became_peered":"2024-09-18T08:15:41.466371+0000","last_unstale":"2024-09-18T08:18:38.675220+0000","last_undegraded":"2024-09-18T08:18:38.675220+0000","last_fullsized":"2024-09-18T08:18:38.675220+0000","mapping_epoch":336,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":337,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:15:05.719269+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_clean_scrub_stamp":"2024-09-18T08:15:05.719269+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":394,"reported_epoch":406,"state":"active+clean","last_fresh":"2024-09-18T08:18:34.429101+0000","last_change":"2024-09-18T08:16:13.487949+0000","last_active":"2024-09-18T08:18:34.429101+0000","last_peered":"2024-09-18T08:18:34.429101+0000","last_clean":"2024-09-18T08:18:34.429101+0000","last_became_active":"2024-09-18T08:16:13.487001+0000","last_became_peered":"2024-09-18T08:16:13.487001+0000","last_unstale":"2024-09-18T08:18:34.429101+0000","last_undegraded":"2024-09-18T08:18:34.429101+0000","last_fullsized":"2024-09-18T08:18:34.429101+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:14:40.774086+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:46.860092+0000","last_clean_scrub_stamp":"2024-09-18T08:14:40.774086+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:19:03.581637+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1d","version":"271'18","reported_seq":872,"reported_epoch":407,"state":"active+clean","last_fresh":"2024-09-18T08:18:38.673763+0000","last_change":"2024-09-18T08:18:19.829522+0000","last_active":"2024-09-18T08:18:38.673763+0000","last_peered":"2024-09-18T08:18:38.673763+0000","last_clean":"2024-09-18T08:18:38.673763+0000","last_became_active":"2024-09-18T08:18:19.823337+0000","last_became_peered":"2024-09-18T08:18:19.823337+0000","last_unstale":"2024-09-18T08:18:38.673763+0000","last_undegraded":"2024-09-18T08:18:38.673763+0000","last_fullsized":"2024-09-18T08:18:38.673763+0000","mapping_epoch":397,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":398,"parent":"0.0","parent_split_bits":0,"last_scrub":"271'18","last_scrub_stamp":"2024-09-18T08:17:04.627913+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_clean_scrub_stamp":"2024-09-18T08:17:04.627913+0000","objects_scrubbed":5,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":34,"num_read_kb":25,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":23,"num_bytes_recovered":880,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":172,"reported_epoch":407,"state":"active+clean","last_fresh":"2024-09-18T08:18:38.673197+0000","last_change":"2024-09-18T08:18:27.662672+0000","last_active":"2024-09-18T08:18:38.673197+0000","last_peered":"2024-09-18T08:18:38.673197+0000","last_clean":"2024-09-18T08:18:38.673197+0000","last_became_active":"2024-09-18T08:18:27.662242+0000","last_became_peered":"2024-09-18T08:18:27.662242+0000","last_unstale":"2024-09-18T08:18:38.673197+0000","last_undegraded":"2024-09-18T08:18:38.673197+0000","last_fullsized":"2024-09-18T08:18:38.673197+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:18:18.614698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:50.886830+0000","last_clean_scrub_stamp":"2024-09-18T08:18:18.614698+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:19:23.411626+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,7],"acting":[3,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.1c","version":"69'1","reported_seq":168,"reported_epoch":407,"state":"active+clean","last_fresh":"2024-09-18T08:18:38.668827+0000","last_change":"2024-09-18T08:18:27.662597+0000","last_active":"2024-09-18T08:18:38.668827+0000","last_peered":"2024-09-18T08:18:38.668827+0000","last_clean":"2024-09-18T08:18:38.668827+0000","last_became_active":"2024-09-18T08:18:27.642951+0000","last_became_peered":"2024-09-18T08:18:27.642951+0000","last_unstale":"2024-09-18T08:18:38.668827+0000","last_undegraded":"2024-09-18T08:18:38.668827+0000","last_fullsized":"2024-09-18T08:18:38.668827+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'1","last_scrub_stamp":"2024-09-18T08:15:06.708980+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_clean_scrub_stamp":"2024-09-18T08:15:06.708980+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":403,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":1209,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,2],"acting":[5,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"62'1","reported_seq":632,"reported_epoch":406,"state":"active+clean","last_fresh":"2024-09-18T08:18:34.427767+0000","last_change":"2024-09-18T08:17:37.618000+0000","last_active":"2024-09-18T08:18:34.427767+0000","last_peered":"2024-09-18T08:18:34.427767+0000","last_clean":"2024-09-18T08:18:34.427767+0000","last_became_active":"2024-09-18T08:16:13.483397+0000","last_became_peered":"2024-09-18T08:16:13.483397+0000","last_unstale":"2024-09-18T08:18:34.427767+0000","last_undegraded":"2024-09-18T08:18:34.427767+0000","last_fullsized":"2024-09-18T08:18:34.427767+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"62'1","last_scrub_stamp":"2024-09-18T08:17:37.617954+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:46.860092+0000","last_clean_scrub_stamp":"2024-09-18T08:17:37.617954+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:18:50.504684+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":46,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,6],"acting":[1,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1e","version":"271'20","reported_seq":799,"reported_epoch":406,"state":"active+clean","last_fresh":"2024-09-18T08:18:34.923675+0000","last_change":"2024-09-18T08:18:27.690042+0000","last_active":"2024-09-18T08:18:34.923675+0000","last_peered":"2024-09-18T08:18:34.923675+0000","last_clean":"2024-09-18T08:18:34.923675+0000","last_became_active":"2024-09-18T08:18:27.657472+0000","last_became_peered":"2024-09-18T08:18:27.657472+0000","last_unstale":"2024-09-18T08:18:34.923675+0000","last_undegraded":"2024-09-18T08:18:34.923675+0000","last_fullsized":"2024-09-18T08:18:34.923675+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":0,"last_scrub":"271'20","last_scrub_stamp":"2024-09-18T08:15:40.251706+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_clean_scrub_stamp":"2024-09-18T08:15:40.251706+0000","objects_scrubbed":5,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":25,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1f","version":"402'16","reported_seq":27,"reported_epoch":406,"state":"active+clean","last_fresh":"2024-09-18T08:18:34.428517+0000","last_change":"2024-09-18T08:18:27.675519+0000","last_active":"2024-09-18T08:18:34.428517+0000","last_peered":"2024-09-18T08:18:34.428517+0000","last_clean":"2024-09-18T08:18:34.428517+0000","last_became_active":"2024-09-18T08:18:27.658656+0000","last_became_peered":"2024-09-18T08:18:27.658656+0000","last_unstale":"2024-09-18T08:18:34.428517+0000","last_undegraded":"2024-09-18T08:18:34.428517+0000","last_fullsized":"2024-09-18T08:18:34.428517+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":0,"last_scrub":"349'15","last_scrub_stamp":"2024-09-18T08:17:29.878376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:50.886830+0000","last_clean_scrub_stamp":"2024-09-18T08:17:29.878376+0000","objects_scrubbed":1,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:18:39.055954+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.0","version":"406'384","reported_seq":405,"reported_epoch":406,"state":"active+clean","last_fresh":"2024-09-18T08:18:37.564748+0000","last_change":"2024-09-18T08:18:27.643125+0000","last_active":"2024-09-18T08:18:37.564748+0000","last_peered":"2024-09-18T08:18:37.564748+0000","last_clean":"2024-09-18T08:18:37.564748+0000","last_became_active":"2024-09-18T08:18:27.642627+0000","last_became_peered":"2024-09-18T08:18:27.642627+0000","last_unstale":"2024-09-18T08:18:37.564748+0000","last_undegraded":"2024-09-18T08:18:37.564748+0000","last_fullsized":"2024-09-18T08:18:37.564748+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":401,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:18:26.612016+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:18:26.612016+0000","last_clean_scrub_stamp":"2024-09-18T08:18:26.612016+0000","objects_scrubbed":0,"log_size":384,"log_dups_size":0,"ondisk_log_size":384,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:19:28.785928+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25165824,"num_objects":384,"num_object_clones":0,"num_object_copies":1152,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":384,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":384,"num_write_kb":24576,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,7],"acting":[1,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.a","version":"74'79","reported_seq":161,"reported_epoch":407,"state":"active+clean","last_fresh":"2024-09-18T08:18:38.668367+0000","last_change":"2024-09-18T08:15:42.288426+0000","last_active":"2024-09-18T08:18:38.668367+0000","last_peered":"2024-09-18T08:18:38.668367+0000","last_clean":"2024-09-18T08:18:38.668367+0000","last_became_active":"2024-09-18T08:15:41.443376+0000","last_became_peered":"2024-09-18T08:15:41.443376+0000","last_unstale":"2024-09-18T08:18:38.668367+0000","last_undegraded":"2024-09-18T08:18:38.668367+0000","last_fullsized":"2024-09-18T08:18:38.668367+0000","mapping_epoch":336,"log_start":"0'0","ondisk_log_start":"0'0","created":152,"last_epoch_clean":337,"parent":"0.0","parent_split_bits":4,"last_scrub":"74'79","last_scrub_stamp":"2024-09-18T08:15:42.288363+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:00:11.893742+0000","last_clean_scrub_stamp":"2024-09-18T08:15:42.288363+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,6],"acting":[5,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.f","version":"271'27","reported_seq":711,"reported_epoch":406,"state":"active+clean","last_fresh":"2024-09-18T08:18:34.428253+0000","last_change":"2024-09-18T08:18:27.682257+0000","last_active":"2024-09-18T08:18:34.428253+0000","last_peered":"2024-09-18T08:18:34.428253+0000","last_clean":"2024-09-18T08:18:34.428253+0000","last_became_active":"2024-09-18T08:18:27.652553+0000","last_became_peered":"2024-09-18T08:18:27.652553+0000","last_unstale":"2024-09-18T08:18:34.428253+0000","last_undegraded":"2024-09-18T08:18:34.428253+0000","last_fullsized":"2024-09-18T08:18:34.428253+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":0,"last_scrub":"271'27","last_scrub_stamp":"2024-09-18T08:17:29.732496+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_clean_scrub_stamp":"2024-09-18T08:17:29.732496+0000","objects_scrubbed":7,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":43,"num_read_kb":34,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,7],"acting":[1,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":45,"reported_epoch":407,"state":"active+clean","last_fresh":"2024-09-18T08:18:38.671845+0000","last_change":"2024-09-18T08:18:27.655637+0000","last_active":"2024-09-18T08:18:38.671845+0000","last_peered":"2024-09-18T08:18:38.671845+0000","last_clean":"2024-09-18T08:18:38.671845+0000","last_became_active":"2024-09-18T08:18:27.655321+0000","last_became_peered":"2024-09-18T08:18:27.655321+0000","last_unstale":"2024-09-18T08:18:38.671845+0000","last_undegraded":"2024-09-18T08:18:38.671845+0000","last_fullsized":"2024-09-18T08:18:38.671845+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:18:23.067375+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:46.860092+0000","last_clean_scrub_stamp":"2024-09-18T08:18:23.067375+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:19:29.744845+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,7],"acting":[2,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.e","version":"349'16","reported_seq":630,"reported_epoch":407,"state":"active+clean","last_fresh":"2024-09-18T08:18:38.669141+0000","last_change":"2024-09-18T08:17:37.704743+0000","last_active":"2024-09-18T08:18:38.669141+0000","last_peered":"2024-09-18T08:18:38.669141+0000","last_clean":"2024-09-18T08:18:38.669141+0000","last_became_active":"2024-09-18T08:16:13.451199+0000","last_became_peered":"2024-09-18T08:16:13.451199+0000","last_unstale":"2024-09-18T08:18:38.669141+0000","last_undegraded":"2024-09-18T08:18:38.669141+0000","last_fullsized":"2024-09-18T08:18:38.669141+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"349'16","last_scrub_stamp":"2024-09-18T08:17:37.704683+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:50.886830+0000","last_clean_scrub_stamp":"2024-09-18T08:17:37.704683+0000","objects_scrubbed":1,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:18:50.732146+0000","scrub_duration":71,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,2],"acting":[5,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":153,"reported_epoch":407,"state":"active+clean","last_fresh":"2024-09-18T08:18:38.668508+0000","last_change":"2024-09-18T08:15:41.438999+0000","last_active":"2024-09-18T08:18:38.668508+0000","last_peered":"2024-09-18T08:18:38.668508+0000","last_clean":"2024-09-18T08:18:38.668508+0000","last_became_active":"2024-09-18T08:15:41.438610+0000","last_became_peered":"2024-09-18T08:15:41.438610+0000","last_unstale":"2024-09-18T08:18:38.668508+0000","last_undegraded":"2024-09-18T08:18:38.668508+0000","last_fullsized":"2024-09-18T08:18:38.668508+0000","mapping_epoch":336,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":337,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:15:00.180309+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_clean_scrub_stamp":"2024-09-18T08:15:00.180309+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:18:54.566550+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.f","version":"406'369","reported_seq":390,"reported_epoch":406,"state":"active+clean","last_fresh":"2024-09-18T08:18:37.573752+0000","last_change":"2024-09-18T08:18:27.652891+0000","last_active":"2024-09-18T08:18:37.573752+0000","last_peered":"2024-09-18T08:18:37.573752+0000","last_clean":"2024-09-18T08:18:37.573752+0000","last_became_active":"2024-09-18T08:18:27.652080+0000","last_became_peered":"2024-09-18T08:18:27.652080+0000","last_unstale":"2024-09-18T08:18:37.573752+0000","last_undegraded":"2024-09-18T08:18:37.573752+0000","last_fullsized":"2024-09-18T08:18:37.573752+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":401,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:18:26.612016+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:18:26.612016+0000","last_clean_scrub_stamp":"2024-09-18T08:18:26.612016+0000","objects_scrubbed":0,"log_size":369,"log_dups_size":0,"ondisk_log_size":369,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:19:30.958767+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24182784,"num_objects":369,"num_object_clones":0,"num_object_copies":1107,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":369,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":369,"num_write_kb":23616,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,2],"acting":[1,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.5","version":"74'79","reported_seq":189,"reported_epoch":407,"state":"active+clean","last_fresh":"2024-09-18T08:18:38.672573+0000","last_change":"2024-09-18T08:18:14.060040+0000","last_active":"2024-09-18T08:18:38.672573+0000","last_peered":"2024-09-18T08:18:38.672573+0000","last_clean":"2024-09-18T08:18:38.672573+0000","last_became_active":"2024-09-18T08:16:13.445467+0000","last_became_peered":"2024-09-18T08:16:13.445467+0000","last_unstale":"2024-09-18T08:18:38.672573+0000","last_undegraded":"2024-09-18T08:18:38.672573+0000","last_fullsized":"2024-09-18T08:18:38.672573+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":152,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":3,"last_scrub":"74'79","last_scrub_stamp":"2024-09-18T08:18:14.059977+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:00:11.893742+0000","last_clean_scrub_stamp":"2024-09-18T08:18:14.059977+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:19:39.257618+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,5],"acting":[2,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.0","version":"271'30","reported_seq":850,"reported_epoch":407,"state":"active+clean","last_fresh":"2024-09-18T08:18:38.673197+0000","last_change":"2024-09-18T08:18:27.693740+0000","last_active":"2024-09-18T08:18:38.673197+0000","last_peered":"2024-09-18T08:18:38.673197+0000","last_clean":"2024-09-18T08:18:38.673197+0000","last_became_active":"2024-09-18T08:18:27.651348+0000","last_became_peered":"2024-09-18T08:18:27.651348+0000","last_unstale":"2024-09-18T08:18:38.673197+0000","last_undegraded":"2024-09-18T08:18:38.673197+0000","last_fullsized":"2024-09-18T08:18:38.673197+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":0,"last_scrub":"271'30","last_scrub_stamp":"2024-09-18T08:15:57.927111+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_clean_scrub_stamp":"2024-09-18T08:15:57.927111+0000","objects_scrubbed":8,"log_size":30,"log_dups_size":0,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":18,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":52,"num_read_kb":40,"num_write":32,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":785,"reported_epoch":407,"state":"active+clean","last_fresh":"2024-09-18T08:18:38.675611+0000","last_change":"2024-09-18T08:18:27.664084+0000","last_active":"2024-09-18T08:18:38.675611+0000","last_peered":"2024-09-18T08:18:38.675611+0000","last_clean":"2024-09-18T08:18:38.675611+0000","last_became_active":"2024-09-18T08:18:27.663173+0000","last_became_peered":"2024-09-18T08:18:27.663173+0000","last_unstale":"2024-09-18T08:18:38.675611+0000","last_undegraded":"2024-09-18T08:18:38.675611+0000","last_fullsized":"2024-09-18T08:18:38.675611+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:15:08.780913+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:46.860092+0000","last_clean_scrub_stamp":"2024-09-18T08:15:08.780913+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":394,"reported_epoch":407,"state":"active+clean","last_fresh":"2024-09-18T08:18:38.673692+0000","last_change":"2024-09-18T08:18:27.661806+0000","last_active":"2024-09-18T08:18:38.673692+0000","last_peered":"2024-09-18T08:18:38.673692+0000","last_clean":"2024-09-18T08:18:38.673692+0000","last_became_active":"2024-09-18T08:18:27.661393+0000","last_became_peered":"2024-09-18T08:18:27.661393+0000","last_unstale":"2024-09-18T08:18:38.673692+0000","last_undegraded":"2024-09-18T08:18:38.673692+0000","last_fullsized":"2024-09-18T08:18:38.673692+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:17:31.417453+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:50.886830+0000","last_clean_scrub_stamp":"2024-09-18T08:17:31.417453+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:18:47.249365+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":159,"reported_epoch":407,"state":"active+clean","last_fresh":"2024-09-18T08:18:38.674707+0000","last_change":"2024-09-18T08:16:13.461117+0000","last_active":"2024-09-18T08:18:38.674707+0000","last_peered":"2024-09-18T08:18:38.674707+0000","last_clean":"2024-09-18T08:18:38.674707+0000","last_became_active":"2024-09-18T08:16:13.460785+0000","last_became_peered":"2024-09-18T08:16:13.460785+0000","last_unstale":"2024-09-18T08:18:38.674707+0000","last_undegraded":"2024-09-18T08:18:38.674707+0000","last_fullsized":"2024-09-18T08:18:38.674707+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:12:02.473168+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_clean_scrub_stamp":"2024-09-18T08:12:02.473168+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:19:38.554195+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,6],"acting":[3,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.e","version":"406'372","reported_seq":393,"reported_epoch":406,"state":"active+clean","last_fresh":"2024-09-18T08:18:37.561875+0000","last_change":"2024-09-18T08:18:27.652534+0000","last_active":"2024-09-18T08:18:37.561875+0000","last_peered":"2024-09-18T08:18:37.561875+0000","last_clean":"2024-09-18T08:18:37.561875+0000","last_became_active":"2024-09-18T08:18:27.652261+0000","last_became_peered":"2024-09-18T08:18:27.652261+0000","last_unstale":"2024-09-18T08:18:37.561875+0000","last_undegraded":"2024-09-18T08:18:37.561875+0000","last_fullsized":"2024-09-18T08:18:37.561875+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":401,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:18:26.612016+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:18:26.612016+0000","last_clean_scrub_stamp":"2024-09-18T08:18:26.612016+0000","objects_scrubbed":0,"log_size":372,"log_dups_size":0,"ondisk_log_size":372,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:19:27.279203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24379392,"num_objects":372,"num_object_clones":0,"num_object_copies":1116,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":372,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":372,"num_write_kb":23808,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"74'79","reported_seq":612,"reported_epoch":407,"state":"active+clean","last_fresh":"2024-09-18T08:18:38.672906+0000","last_change":"2024-09-18T08:18:27.688842+0000","last_active":"2024-09-18T08:18:38.672906+0000","last_peered":"2024-09-18T08:18:38.672906+0000","last_clean":"2024-09-18T08:18:38.672906+0000","last_became_active":"2024-09-18T08:18:27.662304+0000","last_became_peered":"2024-09-18T08:18:27.662304+0000","last_unstale":"2024-09-18T08:18:38.672906+0000","last_undegraded":"2024-09-18T08:18:38.672906+0000","last_fullsized":"2024-09-18T08:18:38.672906+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":152,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":3,"last_scrub":"74'79","last_scrub_stamp":"2024-09-18T08:15:41.898294+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:00:11.893742+0000","last_clean_scrub_stamp":"2024-09-18T08:15:41.898294+0000","objects_scrubbed":1,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":327680,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1310720,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,7],"acting":[3,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"271'28","reported_seq":536,"reported_epoch":407,"state":"active+clean","last_fresh":"2024-09-18T08:18:38.667989+0000","last_change":"2024-09-18T08:16:13.506782+0000","last_active":"2024-09-18T08:18:38.667989+0000","last_peered":"2024-09-18T08:18:38.667989+0000","last_clean":"2024-09-18T08:18:38.667989+0000","last_became_active":"2024-09-18T08:16:13.451205+0000","last_became_peered":"2024-09-18T08:16:13.451205+0000","last_unstale":"2024-09-18T08:18:38.667989+0000","last_undegraded":"2024-09-18T08:18:38.667989+0000","last_fullsized":"2024-09-18T08:18:38.667989+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"271'28","last_scrub_stamp":"2024-09-18T08:15:51.716119+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_clean_scrub_stamp":"2024-09-18T08:15:51.716119+0000","objects_scrubbed":7,"log_size":28,"log_dups_size":0,"ondisk_log_size":28,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":41,"num_read_kb":34,"num_write":28,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.6","version":"62'2","reported_seq":800,"reported_epoch":406,"state":"active+clean","last_fresh":"2024-09-18T08:18:34.923836+0000","last_change":"2024-09-18T08:18:27.670033+0000","last_active":"2024-09-18T08:18:34.923836+0000","last_peered":"2024-09-18T08:18:34.923836+0000","last_clean":"2024-09-18T08:18:34.923836+0000","last_became_active":"2024-09-18T08:18:27.645204+0000","last_became_peered":"2024-09-18T08:18:27.645204+0000","last_unstale":"2024-09-18T08:18:34.923836+0000","last_undegraded":"2024-09-18T08:18:34.923836+0000","last_fullsized":"2024-09-18T08:18:34.923836+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":0,"last_scrub":"62'2","last_scrub_stamp":"2024-09-18T08:14:42.145836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:46.860092+0000","last_clean_scrub_stamp":"2024-09-18T08:14:42.145836+0000","objects_scrubbed":2,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:19:33.893174+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1039,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":4156,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.0","version":"349'13","reported_seq":970,"reported_epoch":407,"state":"active+clean","last_fresh":"2024-09-18T08:18:38.672987+0000","last_change":"2024-09-18T08:16:13.516195+0000","last_active":"2024-09-18T08:18:38.672987+0000","last_peered":"2024-09-18T08:18:38.672987+0000","last_clean":"2024-09-18T08:18:38.672987+0000","last_became_active":"2024-09-18T08:16:13.477431+0000","last_became_peered":"2024-09-18T08:16:13.477431+0000","last_unstale":"2024-09-18T08:18:38.672987+0000","last_undegraded":"2024-09-18T08:18:38.672987+0000","last_fullsized":"2024-09-18T08:18:38.672987+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"310'12","last_scrub_stamp":"2024-09-18T08:15:49.795095+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:50.886830+0000","last_clean_scrub_stamp":"2024-09-18T08:15:49.795095+0000","objects_scrubbed":1,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":153,"reported_epoch":407,"state":"active+clean","last_fresh":"2024-09-18T08:18:38.668745+0000","last_change":"2024-09-18T08:15:41.441653+0000","last_active":"2024-09-18T08:18:38.668745+0000","last_peered":"2024-09-18T08:18:38.668745+0000","last_clean":"2024-09-18T08:18:38.668745+0000","last_became_active":"2024-09-18T08:15:41.440876+0000","last_became_peered":"2024-09-18T08:15:41.440876+0000","last_unstale":"2024-09-18T08:18:38.668745+0000","last_undegraded":"2024-09-18T08:18:38.668745+0000","last_fullsized":"2024-09-18T08:18:38.668745+0000","mapping_epoch":336,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":337,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:15:14.781554+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_clean_scrub_stamp":"2024-09-18T08:15:14.781554+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.d","version":"407'616","reported_seq":639,"reported_epoch":407,"state":"active+clean","last_fresh":"2024-09-18T08:18:39.132058+0000","last_change":"2024-09-18T08:18:27.638289+0000","last_active":"2024-09-18T08:18:39.132058+0000","last_peered":"2024-09-18T08:18:39.132058+0000","last_clean":"2024-09-18T08:18:39.132058+0000","last_became_active":"2024-09-18T08:18:27.637978+0000","last_became_peered":"2024-09-18T08:18:27.637978+0000","last_unstale":"2024-09-18T08:18:39.132058+0000","last_undegraded":"2024-09-18T08:18:39.132058+0000","last_fullsized":"2024-09-18T08:18:39.132058+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":401,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:18:26.612016+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:18:26.612016+0000","last_clean_scrub_stamp":"2024-09-18T08:18:26.612016+0000","objects_scrubbed":0,"log_size":616,"log_dups_size":0,"ondisk_log_size":616,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:19:50.943560+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40370176,"num_objects":616,"num_object_clones":0,"num_object_copies":1848,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":616,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":616,"num_write_kb":39424,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,1],"acting":[2,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.7","version":"74'79","reported_seq":591,"reported_epoch":407,"state":"active+clean","last_fresh":"2024-09-18T08:18:38.675658+0000","last_change":"2024-09-18T08:18:27.663511+0000","last_active":"2024-09-18T08:18:38.675658+0000","last_peered":"2024-09-18T08:18:38.675658+0000","last_clean":"2024-09-18T08:18:38.675658+0000","last_became_active":"2024-09-18T08:18:27.662723+0000","last_became_peered":"2024-09-18T08:18:27.662723+0000","last_unstale":"2024-09-18T08:18:38.675658+0000","last_undegraded":"2024-09-18T08:18:38.675658+0000","last_fullsized":"2024-09-18T08:18:38.675658+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":152,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":3,"last_scrub":"74'79","last_scrub_stamp":"2024-09-18T08:15:44.872332+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:00:11.893742+0000","last_clean_scrub_stamp":"2024-09-18T08:15:44.872332+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,7],"acting":[3,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.2","version":"271'20","reported_seq":751,"reported_epoch":406,"state":"active+clean","last_fresh":"2024-09-18T08:18:34.429192+0000","last_change":"2024-09-18T08:18:22.653813+0000","last_active":"2024-09-18T08:18:34.429192+0000","last_peered":"2024-09-18T08:18:34.429192+0000","last_clean":"2024-09-18T08:18:34.429192+0000","last_became_active":"2024-09-18T08:16:13.486720+0000","last_became_peered":"2024-09-18T08:16:13.486720+0000","last_unstale":"2024-09-18T08:18:34.429192+0000","last_undegraded":"2024-09-18T08:18:34.429192+0000","last_fullsized":"2024-09-18T08:18:34.429192+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"271'20","last_scrub_stamp":"2024-09-18T08:18:22.653761+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_clean_scrub_stamp":"2024-09-18T08:18:22.653761+0000","objects_scrubbed":5,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:19:43.290763+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":28,"num_read_kb":23,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":153,"reported_epoch":407,"state":"active+clean","last_fresh":"2024-09-18T08:18:38.669057+0000","last_change":"2024-09-18T08:15:41.452981+0000","last_active":"2024-09-18T08:18:38.669057+0000","last_peered":"2024-09-18T08:18:38.669057+0000","last_clean":"2024-09-18T08:18:38.669057+0000","last_became_active":"2024-09-18T08:15:41.449639+0000","last_became_peered":"2024-09-18T08:15:41.449639+0000","last_unstale":"2024-09-18T08:18:38.669057+0000","last_undegraded":"2024-09-18T08:18:38.669057+0000","last_fullsized":"2024-09-18T08:18:38.669057+0000","mapping_epoch":336,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":337,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:15:39.858184+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:46.860092+0000","last_clean_scrub_stamp":"2024-09-18T08:15:39.858184+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.3","version":"338'14","reported_seq":967,"reported_epoch":406,"state":"active+clean","last_fresh":"2024-09-18T08:18:34.923081+0000","last_change":"2024-09-18T08:15:41.467411+0000","last_active":"2024-09-18T08:18:34.923081+0000","last_peered":"2024-09-18T08:18:34.923081+0000","last_clean":"2024-09-18T08:18:34.923081+0000","last_became_active":"2024-09-18T08:15:41.447667+0000","last_became_peered":"2024-09-18T08:15:41.447667+0000","last_unstale":"2024-09-18T08:18:34.923081+0000","last_undegraded":"2024-09-18T08:18:34.923081+0000","last_fullsized":"2024-09-18T08:18:34.923081+0000","mapping_epoch":336,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":337,"parent":"0.0","parent_split_bits":0,"last_scrub":"317'11","last_scrub_stamp":"2024-09-18T08:14:59.207667+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:50.886830+0000","last_clean_scrub_stamp":"2024-09-18T08:14:59.207667+0000","objects_scrubbed":1,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":757,"reported_epoch":406,"state":"active+clean","last_fresh":"2024-09-18T08:18:34.921538+0000","last_change":"2024-09-18T08:18:23.800730+0000","last_active":"2024-09-18T08:18:34.921538+0000","last_peered":"2024-09-18T08:18:34.921538+0000","last_clean":"2024-09-18T08:18:34.921538+0000","last_became_active":"2024-09-18T08:08:52.700080+0000","last_became_peered":"2024-09-18T08:08:52.700080+0000","last_unstale":"2024-09-18T08:18:34.921538+0000","last_undegraded":"2024-09-18T08:18:34.921538+0000","last_fullsized":"2024-09-18T08:18:34.921538+0000","mapping_epoch":198,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":199,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:18:23.800677+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_clean_scrub_stamp":"2024-09-18T08:18:23.800677+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:19:26.101441+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.c","version":"406'369","reported_seq":390,"reported_epoch":406,"state":"active+clean","last_fresh":"2024-09-18T08:18:37.581132+0000","last_change":"2024-09-18T08:18:27.641565+0000","last_active":"2024-09-18T08:18:37.581132+0000","last_peered":"2024-09-18T08:18:37.581132+0000","last_clean":"2024-09-18T08:18:37.581132+0000","last_became_active":"2024-09-18T08:18:27.641287+0000","last_became_peered":"2024-09-18T08:18:27.641287+0000","last_unstale":"2024-09-18T08:18:37.581132+0000","last_undegraded":"2024-09-18T08:18:37.581132+0000","last_fullsized":"2024-09-18T08:18:37.581132+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":401,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:18:26.612016+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:18:26.612016+0000","last_clean_scrub_stamp":"2024-09-18T08:18:26.612016+0000","objects_scrubbed":0,"log_size":369,"log_dups_size":0,"ondisk_log_size":369,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:19:33.258270+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24182784,"num_objects":369,"num_object_clones":0,"num_object_copies":1107,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":369,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":369,"num_write_kb":23616,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,7],"acting":[1,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"74'79","reported_seq":649,"reported_epoch":406,"state":"active+clean","last_fresh":"2024-09-18T08:18:34.923319+0000","last_change":"2024-09-18T08:18:20.837291+0000","last_active":"2024-09-18T08:18:34.923319+0000","last_peered":"2024-09-18T08:18:34.923319+0000","last_clean":"2024-09-18T08:18:34.923319+0000","last_became_active":"2024-09-18T08:07:26.426532+0000","last_became_peered":"2024-09-18T08:07:26.426532+0000","last_unstale":"2024-09-18T08:18:34.923319+0000","last_undegraded":"2024-09-18T08:18:34.923319+0000","last_fullsized":"2024-09-18T08:18:34.923319+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":152,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":3,"last_scrub":"74'79","last_scrub_stamp":"2024-09-18T08:18:20.837241+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:00:11.893742+0000","last_clean_scrub_stamp":"2024-09-18T08:18:20.837241+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:19:31.957632+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,3],"acting":[0,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.3","version":"271'29","reported_seq":890,"reported_epoch":406,"state":"active+clean","last_fresh":"2024-09-18T08:18:34.923212+0000","last_change":"2024-09-18T08:18:27.685772+0000","last_active":"2024-09-18T08:18:34.923212+0000","last_peered":"2024-09-18T08:18:34.923212+0000","last_clean":"2024-09-18T08:18:34.923212+0000","last_became_active":"2024-09-18T08:18:27.654192+0000","last_became_peered":"2024-09-18T08:18:27.654192+0000","last_unstale":"2024-09-18T08:18:34.923212+0000","last_undegraded":"2024-09-18T08:18:34.923212+0000","last_fullsized":"2024-09-18T08:18:34.923212+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":0,"last_scrub":"271'29","last_scrub_stamp":"2024-09-18T08:18:15.879919+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_clean_scrub_stamp":"2024-09-18T08:18:15.879919+0000","objects_scrubbed":8,"log_size":29,"log_dups_size":0,"ondisk_log_size":29,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:19:43.408146+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":54,"num_read_kb":40,"num_write":32,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":32,"num_bytes_recovered":1320,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.4","version":"62'1","reported_seq":616,"reported_epoch":406,"state":"active+clean","last_fresh":"2024-09-18T08:18:34.429825+0000","last_change":"2024-09-18T08:15:41.454692+0000","last_active":"2024-09-18T08:18:34.429825+0000","last_peered":"2024-09-18T08:18:34.429825+0000","last_clean":"2024-09-18T08:18:34.429825+0000","last_became_active":"2024-09-18T08:15:41.441822+0000","last_became_peered":"2024-09-18T08:15:41.441822+0000","last_unstale":"2024-09-18T08:18:34.429825+0000","last_undegraded":"2024-09-18T08:18:34.429825+0000","last_fullsized":"2024-09-18T08:18:34.429825+0000","mapping_epoch":336,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":337,"parent":"0.0","parent_split_bits":0,"last_scrub":"62'1","last_scrub_stamp":"2024-09-18T08:15:13.165826+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:46.860092+0000","last_clean_scrub_stamp":"2024-09-18T08:15:13.165826+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:19:09.628900+0000","scrub_duration":17,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":436,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":1308,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":662,"reported_epoch":406,"state":"active+clean","last_fresh":"2024-09-18T08:18:34.922493+0000","last_change":"2024-09-18T08:15:41.450294+0000","last_active":"2024-09-18T08:18:34.922493+0000","last_peered":"2024-09-18T08:18:34.922493+0000","last_clean":"2024-09-18T08:18:34.922493+0000","last_became_active":"2024-09-18T08:15:41.449832+0000","last_became_peered":"2024-09-18T08:15:41.449832+0000","last_unstale":"2024-09-18T08:18:34.922493+0000","last_undegraded":"2024-09-18T08:18:34.922493+0000","last_fullsized":"2024-09-18T08:18:34.922493+0000","mapping_epoch":336,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":337,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:12:37.732260+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:50.886830+0000","last_clean_scrub_stamp":"2024-09-18T08:12:37.732260+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:18:56.820396+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":635,"reported_epoch":406,"state":"active+clean","last_fresh":"2024-09-18T08:18:34.427485+0000","last_change":"2024-09-18T08:15:50.294289+0000","last_active":"2024-09-18T08:18:34.427485+0000","last_peered":"2024-09-18T08:18:34.427485+0000","last_clean":"2024-09-18T08:18:34.427485+0000","last_became_active":"2024-09-18T08:14:39.358132+0000","last_became_peered":"2024-09-18T08:14:39.358132+0000","last_unstale":"2024-09-18T08:18:34.427485+0000","last_undegraded":"2024-09-18T08:18:34.427485+0000","last_fullsized":"2024-09-18T08:18:34.427485+0000","mapping_epoch":307,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":308,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:15:50.294213+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_clean_scrub_stamp":"2024-09-18T08:15:50.294213+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":28,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.b","version":"407'634","reported_seq":657,"reported_epoch":407,"state":"active+clean","last_fresh":"2024-09-18T08:18:39.373459+0000","last_change":"2024-09-18T08:18:27.656478+0000","last_active":"2024-09-18T08:18:39.373459+0000","last_peered":"2024-09-18T08:18:39.373459+0000","last_clean":"2024-09-18T08:18:39.373459+0000","last_became_active":"2024-09-18T08:18:27.656216+0000","last_became_peered":"2024-09-18T08:18:27.656216+0000","last_unstale":"2024-09-18T08:18:39.373459+0000","last_undegraded":"2024-09-18T08:18:39.373459+0000","last_fullsized":"2024-09-18T08:18:39.373459+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":401,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:18:26.612016+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:18:26.612016+0000","last_clean_scrub_stamp":"2024-09-18T08:18:26.612016+0000","objects_scrubbed":0,"log_size":634,"log_dups_size":0,"ondisk_log_size":634,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:19:51.525124+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41549824,"num_objects":634,"num_object_clones":0,"num_object_copies":1902,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":634,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":634,"num_write_kb":40576,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"74'79","reported_seq":161,"reported_epoch":407,"state":"active+clean","last_fresh":"2024-09-18T08:18:38.668102+0000","last_change":"2024-09-18T08:15:43.275398+0000","last_active":"2024-09-18T08:18:38.668102+0000","last_peered":"2024-09-18T08:18:38.668102+0000","last_clean":"2024-09-18T08:18:38.668102+0000","last_became_active":"2024-09-18T08:15:41.452895+0000","last_became_peered":"2024-09-18T08:15:41.452895+0000","last_unstale":"2024-09-18T08:18:38.668102+0000","last_undegraded":"2024-09-18T08:18:38.668102+0000","last_fullsized":"2024-09-18T08:18:38.668102+0000","mapping_epoch":336,"log_start":"0'0","ondisk_log_start":"0'0","created":152,"last_epoch_clean":337,"parent":"0.0","parent_split_bits":4,"last_scrub":"74'79","last_scrub_stamp":"2024-09-18T08:15:43.275329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:00:11.893742+0000","last_clean_scrub_stamp":"2024-09-18T08:15:43.275329+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,2],"acting":[5,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.4","version":"310'42","reported_seq":785,"reported_epoch":406,"state":"active+clean","last_fresh":"2024-09-18T08:18:34.429985+0000","last_change":"2024-09-18T08:18:27.690564+0000","last_active":"2024-09-18T08:18:34.429985+0000","last_peered":"2024-09-18T08:18:34.429985+0000","last_clean":"2024-09-18T08:18:34.429985+0000","last_became_active":"2024-09-18T08:14:39.486174+0000","last_became_peered":"2024-09-18T08:14:39.486174+0000","last_unstale":"2024-09-18T08:18:34.429985+0000","last_undegraded":"2024-09-18T08:18:34.429985+0000","last_fullsized":"2024-09-18T08:18:34.429985+0000","mapping_epoch":307,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":308,"parent":"0.0","parent_split_bits":0,"last_scrub":"310'42","last_scrub_stamp":"2024-09-18T08:18:27.690465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_clean_scrub_stamp":"2024-09-18T08:18:27.690465+0000","objects_scrubbed":10,"log_size":42,"log_dups_size":0,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:19:42.723207+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":65,"num_read_kb":50,"num_write":38,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":339,"reported_epoch":406,"state":"active+clean","last_fresh":"2024-09-18T08:18:34.923396+0000","last_change":"2024-09-18T08:18:16.839327+0000","last_active":"2024-09-18T08:18:34.923396+0000","last_peered":"2024-09-18T08:18:34.923396+0000","last_clean":"2024-09-18T08:18:34.923396+0000","last_became_active":"2024-09-18T08:16:13.475879+0000","last_became_peered":"2024-09-18T08:16:13.475879+0000","last_unstale":"2024-09-18T08:18:34.923396+0000","last_undegraded":"2024-09-18T08:18:34.923396+0000","last_fullsized":"2024-09-18T08:18:34.923396+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:18:16.839282+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:46.860092+0000","last_clean_scrub_stamp":"2024-09-18T08:18:16.839282+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:19:30.041390+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,1],"acting":[0,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"64'2","reported_seq":165,"reported_epoch":407,"state":"active+clean","last_fresh":"2024-09-18T08:18:38.668254+0000","last_change":"2024-09-18T08:17:08.332007+0000","last_active":"2024-09-18T08:18:38.668254+0000","last_peered":"2024-09-18T08:18:38.668254+0000","last_clean":"2024-09-18T08:18:38.668254+0000","last_became_active":"2024-09-18T08:15:41.453351+0000","last_became_peered":"2024-09-18T08:15:41.453351+0000","last_unstale":"2024-09-18T08:18:38.668254+0000","last_undegraded":"2024-09-18T08:18:38.668254+0000","last_fullsized":"2024-09-18T08:18:38.668254+0000","mapping_epoch":336,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":337,"parent":"0.0","parent_split_bits":0,"last_scrub":"64'2","last_scrub_stamp":"2024-09-18T08:17:08.331955+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:45.855979+0000","last_clean_scrub_stamp":"2024-09-18T08:17:08.331955+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":754,"reported_epoch":406,"state":"active+clean","last_fresh":"2024-09-18T08:18:34.923089+0000","last_change":"2024-09-18T08:16:13.481778+0000","last_active":"2024-09-18T08:18:34.923089+0000","last_peered":"2024-09-18T08:18:34.923089+0000","last_clean":"2024-09-18T08:18:34.923089+0000","last_became_active":"2024-09-18T08:16:13.481456+0000","last_became_peered":"2024-09-18T08:16:13.481456+0000","last_unstale":"2024-09-18T08:18:34.923089+0000","last_undegraded":"2024-09-18T08:18:34.923089+0000","last_fullsized":"2024-09-18T08:18:34.923089+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:15:04.260540+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:50.886830+0000","last_clean_scrub_stamp":"2024-09-18T08:15:04.260540+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,6],"acting":[0,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":773,"reported_epoch":407,"state":"active+clean","last_fresh":"2024-09-18T08:18:38.674797+0000","last_change":"2024-09-18T08:18:27.657904+0000","last_active":"2024-09-18T08:18:38.674797+0000","last_peered":"2024-09-18T08:18:38.674797+0000","last_clean":"2024-09-18T08:18:38.674797+0000","last_became_active":"2024-09-18T08:18:27.657591+0000","last_became_peered":"2024-09-18T08:18:27.657591+0000","last_unstale":"2024-09-18T08:18:38.674797+0000","last_undegraded":"2024-09-18T08:18:38.674797+0000","last_fullsized":"2024-09-18T08:18:38.674797+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:15:59.948801+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_clean_scrub_stamp":"2024-09-18T08:15:59.948801+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.8","version":"407'583","reported_seq":606,"reported_epoch":407,"state":"active+clean","last_fresh":"2024-09-18T08:18:39.353682+0000","last_change":"2024-09-18T08:18:27.641711+0000","last_active":"2024-09-18T08:18:39.353682+0000","last_peered":"2024-09-18T08:18:39.353682+0000","last_clean":"2024-09-18T08:18:39.353682+0000","last_became_active":"2024-09-18T08:18:27.641439+0000","last_became_peered":"2024-09-18T08:18:27.641439+0000","last_unstale":"2024-09-18T08:18:39.353682+0000","last_undegraded":"2024-09-18T08:18:39.353682+0000","last_fullsized":"2024-09-18T08:18:39.353682+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":401,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:18:26.612016+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:18:26.612016+0000","last_clean_scrub_stamp":"2024-09-18T08:18:26.612016+0000","objects_scrubbed":0,"log_size":583,"log_dups_size":0,"ondisk_log_size":583,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:19:55.081035+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38207488,"num_objects":583,"num_object_clones":0,"num_object_copies":1749,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":583,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":583,"num_write_kb":37312,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"74'79","reported_seq":683,"reported_epoch":406,"state":"active+clean","last_fresh":"2024-09-18T08:18:34.922280+0000","last_change":"2024-09-18T08:18:27.659593+0000","last_active":"2024-09-18T08:18:34.922280+0000","last_peered":"2024-09-18T08:18:34.922280+0000","last_clean":"2024-09-18T08:18:34.922280+0000","last_became_active":"2024-09-18T08:18:27.659302+0000","last_became_peered":"2024-09-18T08:18:27.659302+0000","last_unstale":"2024-09-18T08:18:34.922280+0000","last_undegraded":"2024-09-18T08:18:34.922280+0000","last_fullsized":"2024-09-18T08:18:34.922280+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":152,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":4,"last_scrub":"74'79","last_scrub_stamp":"2024-09-18T08:18:17.808269+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:00:11.893742+0000","last_clean_scrub_stamp":"2024-09-18T08:18:17.808269+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:19:26.115106+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.7","version":"271'17","reported_seq":734,"reported_epoch":407,"state":"active+clean","last_fresh":"2024-09-18T08:18:38.675177+0000","last_change":"2024-09-18T08:15:41.494329+0000","last_active":"2024-09-18T08:18:38.675177+0000","last_peered":"2024-09-18T08:18:38.675177+0000","last_clean":"2024-09-18T08:18:38.675177+0000","last_became_active":"2024-09-18T08:15:41.456482+0000","last_became_peered":"2024-09-18T08:15:41.456482+0000","last_unstale":"2024-09-18T08:18:38.675177+0000","last_undegraded":"2024-09-18T08:18:38.675177+0000","last_fullsized":"2024-09-18T08:18:38.675177+0000","mapping_epoch":336,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":337,"parent":"0.0","parent_split_bits":0,"last_scrub":"271'17","last_scrub_stamp":"2024-09-18T08:15:13.792602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_clean_scrub_stamp":"2024-09-18T08:15:13.792602+0000","objects_scrubbed":5,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:19:06.957984+0000","scrub_duration":16,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":35,"num_read_kb":24,"num_write":20,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":990,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,0],"acting":[2,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":626,"reported_epoch":406,"state":"active+clean","last_fresh":"2024-09-18T08:18:34.428741+0000","last_change":"2024-09-18T08:15:49.244718+0000","last_active":"2024-09-18T08:18:34.428741+0000","last_peered":"2024-09-18T08:18:34.428741+0000","last_clean":"2024-09-18T08:18:34.428741+0000","last_became_active":"2024-09-18T08:14:39.432254+0000","last_became_peered":"2024-09-18T08:14:39.432254+0000","last_unstale":"2024-09-18T08:18:34.428741+0000","last_undegraded":"2024-09-18T08:18:34.428741+0000","last_fullsized":"2024-09-18T08:18:34.428741+0000","mapping_epoch":307,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":308,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:15:49.244156+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:46.860092+0000","last_clean_scrub_stamp":"2024-09-18T08:15:49.244156+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"62'1","reported_seq":758,"reported_epoch":407,"state":"active+clean","last_fresh":"2024-09-18T08:18:38.672879+0000","last_change":"2024-09-18T08:17:34.935837+0000","last_active":"2024-09-18T08:18:38.672879+0000","last_peered":"2024-09-18T08:18:38.672879+0000","last_clean":"2024-09-18T08:18:38.672879+0000","last_became_active":"2024-09-18T08:02:46.868175+0000","last_became_peered":"2024-09-18T08:02:46.868175+0000","last_unstale":"2024-09-18T08:18:38.672879+0000","last_undegraded":"2024-09-18T08:18:38.672879+0000","last_fullsized":"2024-09-18T08:18:38.672879+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"62'1","last_scrub_stamp":"2024-09-18T08:17:34.935618+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:45.855979+0000","last_clean_scrub_stamp":"2024-09-18T08:17:34.935618+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:19:37.073987+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0.00066785499999999997,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":483,"reported_epoch":407,"state":"active+clean","last_fresh":"2024-09-18T08:18:38.672084+0000","last_change":"2024-09-18T08:18:28.112487+0000","last_active":"2024-09-18T08:18:38.672084+0000","last_peered":"2024-09-18T08:18:38.672084+0000","last_clean":"2024-09-18T08:18:38.672084+0000","last_became_active":"2024-09-18T08:18:27.652680+0000","last_became_peered":"2024-09-18T08:18:27.652680+0000","last_unstale":"2024-09-18T08:18:38.672084+0000","last_undegraded":"2024-09-18T08:18:38.672084+0000","last_fullsized":"2024-09-18T08:18:38.672084+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:18:28.112437+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:50.886830+0000","last_clean_scrub_stamp":"2024-09-18T08:18:28.112437+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:19:42.559131+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":437,"reported_epoch":407,"state":"active+clean","last_fresh":"2024-09-18T08:18:38.673611+0000","last_change":"2024-09-18T08:18:27.662433+0000","last_active":"2024-09-18T08:18:38.673611+0000","last_peered":"2024-09-18T08:18:38.673611+0000","last_clean":"2024-09-18T08:18:38.673611+0000","last_became_active":"2024-09-18T08:18:27.662116+0000","last_became_peered":"2024-09-18T08:18:27.662116+0000","last_unstale":"2024-09-18T08:18:38.673611+0000","last_undegraded":"2024-09-18T08:18:38.673611+0000","last_fullsized":"2024-09-18T08:18:38.673611+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:17:35.381401+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_clean_scrub_stamp":"2024-09-18T08:17:35.381401+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:19:02.247011+0000","scrub_duration":23,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.9","version":"407'614","reported_seq":637,"reported_epoch":407,"state":"active+clean","last_fresh":"2024-09-18T08:18:39.364813+0000","last_change":"2024-09-18T08:18:27.655199+0000","last_active":"2024-09-18T08:18:39.364813+0000","last_peered":"2024-09-18T08:18:39.364813+0000","last_clean":"2024-09-18T08:18:39.364813+0000","last_became_active":"2024-09-18T08:18:27.654866+0000","last_became_peered":"2024-09-18T08:18:27.654866+0000","last_unstale":"2024-09-18T08:18:39.364813+0000","last_undegraded":"2024-09-18T08:18:39.364813+0000","last_fullsized":"2024-09-18T08:18:39.364813+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":401,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:18:26.612016+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:18:26.612016+0000","last_clean_scrub_stamp":"2024-09-18T08:18:26.612016+0000","objects_scrubbed":0,"log_size":614,"log_dups_size":0,"ondisk_log_size":614,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:19:40.211623+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40239104,"num_objects":614,"num_object_clones":0,"num_object_copies":1842,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":614,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":614,"num_write_kb":39296,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,0],"acting":[3,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"74'79","reported_seq":400,"reported_epoch":406,"state":"active+clean","last_fresh":"2024-09-18T08:18:34.428990+0000","last_change":"2024-09-18T08:16:13.455530+0000","last_active":"2024-09-18T08:18:34.428990+0000","last_peered":"2024-09-18T08:18:34.428990+0000","last_clean":"2024-09-18T08:18:34.428990+0000","last_became_active":"2024-09-18T08:16:13.454138+0000","last_became_peered":"2024-09-18T08:16:13.454138+0000","last_unstale":"2024-09-18T08:18:34.428990+0000","last_undegraded":"2024-09-18T08:18:34.428990+0000","last_fullsized":"2024-09-18T08:18:34.428990+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":152,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":3,"last_scrub":"74'79","last_scrub_stamp":"2024-09-18T08:16:00.303863+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:00:11.893742+0000","last_clean_scrub_stamp":"2024-09-18T08:16:00.303863+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,5],"acting":[1,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.6","version":"271'24","reported_seq":838,"reported_epoch":406,"state":"active+clean","last_fresh":"2024-09-18T08:18:34.923681+0000","last_change":"2024-09-18T08:15:10.187667+0000","last_active":"2024-09-18T08:18:34.923681+0000","last_peered":"2024-09-18T08:18:34.923681+0000","last_clean":"2024-09-18T08:18:34.923681+0000","last_became_active":"2024-09-18T08:14:39.438033+0000","last_became_peered":"2024-09-18T08:14:39.438033+0000","last_unstale":"2024-09-18T08:18:34.923681+0000","last_undegraded":"2024-09-18T08:18:34.923681+0000","last_fullsized":"2024-09-18T08:18:34.923681+0000","mapping_epoch":307,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":308,"parent":"0.0","parent_split_bits":0,"last_scrub":"271'24","last_scrub_stamp":"2024-09-18T08:15:10.187593+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_clean_scrub_stamp":"2024-09-18T08:15:10.187593+0000","objects_scrubbed":6,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":36,"num_read_kb":30,"num_write":24,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":767,"reported_epoch":406,"state":"active+clean","last_fresh":"2024-09-18T08:18:34.922306+0000","last_change":"2024-09-18T08:16:13.461291+0000","last_active":"2024-09-18T08:18:34.922306+0000","last_peered":"2024-09-18T08:18:34.922306+0000","last_clean":"2024-09-18T08:18:34.922306+0000","last_became_active":"2024-09-18T08:16:13.460391+0000","last_became_peered":"2024-09-18T08:16:13.460391+0000","last_unstale":"2024-09-18T08:18:34.922306+0000","last_undegraded":"2024-09-18T08:18:34.922306+0000","last_fullsized":"2024-09-18T08:18:34.922306+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:14:37.133544+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:46.860092+0000","last_clean_scrub_stamp":"2024-09-18T08:14:37.133544+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:19:34.910202+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,5],"acting":[0,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"70'5","reported_seq":945,"reported_epoch":406,"state":"active+clean","last_fresh":"2024-09-18T08:18:36.663502+0000","last_change":"2024-09-18T08:18:27.672092+0000","last_active":"2024-09-18T08:18:36.663502+0000","last_peered":"2024-09-18T08:18:36.663502+0000","last_clean":"2024-09-18T08:18:36.663502+0000","last_became_active":"2024-09-18T08:18:27.646445+0000","last_became_peered":"2024-09-18T08:18:27.646445+0000","last_unstale":"2024-09-18T08:18:36.663502+0000","last_undegraded":"2024-09-18T08:18:36.663502+0000","last_fullsized":"2024-09-18T08:18:36.663502+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":0,"last_scrub":"70'5","last_scrub_stamp":"2024-09-18T08:15:57.384224+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:45.855979+0000","last_clean_scrub_stamp":"2024-09-18T08:15:57.384224+0000","objects_scrubbed":1,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0.000504055,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":407,"num_read_kb":402,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":778,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":153,"reported_epoch":407,"state":"active+clean","last_fresh":"2024-09-18T08:18:38.669346+0000","last_change":"2024-09-18T08:15:41.443577+0000","last_active":"2024-09-18T08:18:38.669346+0000","last_peered":"2024-09-18T08:18:38.669346+0000","last_clean":"2024-09-18T08:18:38.669346+0000","last_became_active":"2024-09-18T08:15:41.443080+0000","last_became_peered":"2024-09-18T08:15:41.443080+0000","last_unstale":"2024-09-18T08:18:38.669346+0000","last_undegraded":"2024-09-18T08:18:38.669346+0000","last_fullsized":"2024-09-18T08:18:38.669346+0000","mapping_epoch":336,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":337,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:15:02.179693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:50.886830+0000","last_clean_scrub_stamp":"2024-09-18T08:15:02.179693+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:19:12.621694+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":555,"reported_epoch":406,"state":"active+clean","last_fresh":"2024-09-18T08:18:34.429565+0000","last_change":"2024-09-18T08:18:16.556819+0000","last_active":"2024-09-18T08:18:34.429565+0000","last_peered":"2024-09-18T08:18:34.429565+0000","last_clean":"2024-09-18T08:18:34.429565+0000","last_became_active":"2024-09-18T08:15:41.436692+0000","last_became_peered":"2024-09-18T08:15:41.436692+0000","last_unstale":"2024-09-18T08:18:34.429565+0000","last_undegraded":"2024-09-18T08:18:34.429565+0000","last_fullsized":"2024-09-18T08:18:34.429565+0000","mapping_epoch":336,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":337,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:18:16.556771+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_clean_scrub_stamp":"2024-09-18T08:18:16.556771+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:19:20.511661+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.a","version":"407'670","reported_seq":693,"reported_epoch":407,"state":"active+clean","last_fresh":"2024-09-18T08:18:39.370783+0000","last_change":"2024-09-18T08:18:27.646142+0000","last_active":"2024-09-18T08:18:39.370783+0000","last_peered":"2024-09-18T08:18:39.370783+0000","last_clean":"2024-09-18T08:18:39.370783+0000","last_became_active":"2024-09-18T08:18:27.645863+0000","last_became_peered":"2024-09-18T08:18:27.645863+0000","last_unstale":"2024-09-18T08:18:39.370783+0000","last_undegraded":"2024-09-18T08:18:39.370783+0000","last_fullsized":"2024-09-18T08:18:39.370783+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":401,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:18:26.612016+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:18:26.612016+0000","last_clean_scrub_stamp":"2024-09-18T08:18:26.612016+0000","objects_scrubbed":0,"log_size":670,"log_dups_size":0,"ondisk_log_size":670,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:19:28.560219+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":43909120,"num_objects":670,"num_object_clones":0,"num_object_copies":2010,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":670,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":670,"num_write_kb":42880,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.5","version":"402'36","reported_seq":49,"reported_epoch":406,"state":"active+clean","last_fresh":"2024-09-18T08:18:34.428417+0000","last_change":"2024-09-18T08:18:30.788913+0000","last_active":"2024-09-18T08:18:34.428417+0000","last_peered":"2024-09-18T08:18:34.428417+0000","last_clean":"2024-09-18T08:18:34.428417+0000","last_became_active":"2024-09-18T08:18:27.657630+0000","last_became_peered":"2024-09-18T08:18:27.657630+0000","last_unstale":"2024-09-18T08:18:34.428417+0000","last_undegraded":"2024-09-18T08:18:34.428417+0000","last_fullsized":"2024-09-18T08:18:34.428417+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":0,"last_scrub":"402'36","last_scrub_stamp":"2024-09-18T08:18:30.788799+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_clean_scrub_stamp":"2024-09-18T08:18:30.788799+0000","objects_scrubbed":8,"log_size":36,"log_dups_size":0,"ondisk_log_size":36,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:19:47.408292+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":42,"num_read_kb":32,"num_write":25,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":462,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,0],"acting":[7,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":761,"reported_epoch":407,"state":"active+clean","last_fresh":"2024-09-18T08:18:38.675054+0000","last_change":"2024-09-18T08:15:41.464630+0000","last_active":"2024-09-18T08:18:38.675054+0000","last_peered":"2024-09-18T08:18:38.675054+0000","last_clean":"2024-09-18T08:18:38.675054+0000","last_became_active":"2024-09-18T08:15:41.464313+0000","last_became_peered":"2024-09-18T08:15:41.464313+0000","last_unstale":"2024-09-18T08:18:38.675054+0000","last_undegraded":"2024-09-18T08:18:38.675054+0000","last_fullsized":"2024-09-18T08:18:38.675054+0000","mapping_epoch":336,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":337,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:14:32.693460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:46.860092+0000","last_clean_scrub_stamp":"2024-09-18T08:14:32.693460+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:19:02.427610+0000","scrub_duration":38,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"74'79","reported_seq":364,"reported_epoch":406,"state":"active+clean","last_fresh":"2024-09-18T08:18:34.921734+0000","last_change":"2024-09-18T08:18:27.650334+0000","last_active":"2024-09-18T08:18:34.921734+0000","last_peered":"2024-09-18T08:18:34.921734+0000","last_clean":"2024-09-18T08:18:34.921734+0000","last_became_active":"2024-09-18T08:18:27.649964+0000","last_became_peered":"2024-09-18T08:18:27.649964+0000","last_unstale":"2024-09-18T08:18:34.921734+0000","last_undegraded":"2024-09-18T08:18:34.921734+0000","last_fullsized":"2024-09-18T08:18:34.921734+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'79","last_scrub_stamp":"2024-09-18T08:18:18.786155+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:00:11.893742+0000","last_clean_scrub_stamp":"2024-09-18T08:18:18.786155+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:19:47.551374+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,6],"acting":[0,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":612,"reported_epoch":407,"state":"active+clean","last_fresh":"2024-09-18T08:18:38.672776+0000","last_change":"2024-09-18T08:18:27.654069+0000","last_active":"2024-09-18T08:18:38.672776+0000","last_peered":"2024-09-18T08:18:38.672776+0000","last_clean":"2024-09-18T08:18:38.672776+0000","last_became_active":"2024-09-18T08:18:27.653602+0000","last_became_peered":"2024-09-18T08:18:27.653602+0000","last_unstale":"2024-09-18T08:18:38.672776+0000","last_undegraded":"2024-09-18T08:18:38.672776+0000","last_fullsized":"2024-09-18T08:18:38.672776+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:17:36.886507+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:50.886830+0000","last_clean_scrub_stamp":"2024-09-18T08:17:36.886507+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:19:02.530691+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":168,"reported_epoch":407,"state":"active+clean","last_fresh":"2024-09-18T08:18:38.669302+0000","last_change":"2024-09-18T08:18:25.515806+0000","last_active":"2024-09-18T08:18:38.669302+0000","last_peered":"2024-09-18T08:18:38.669302+0000","last_clean":"2024-09-18T08:18:38.669302+0000","last_became_active":"2024-09-18T08:16:13.471911+0000","last_became_peered":"2024-09-18T08:16:13.471911+0000","last_unstale":"2024-09-18T08:18:38.669302+0000","last_undegraded":"2024-09-18T08:18:38.669302+0000","last_fullsized":"2024-09-18T08:18:38.669302+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:18:25.515757+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_clean_scrub_stamp":"2024-09-18T08:18:25.515757+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:19:37.445115+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,1],"acting":[5,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.1","version":"407'629","reported_seq":650,"reported_epoch":407,"state":"active+clean","last_fresh":"2024-09-18T08:18:39.369662+0000","last_change":"2024-09-18T08:18:27.651222+0000","last_active":"2024-09-18T08:18:39.369662+0000","last_peered":"2024-09-18T08:18:39.369662+0000","last_clean":"2024-09-18T08:18:39.369662+0000","last_became_active":"2024-09-18T08:18:27.650844+0000","last_became_peered":"2024-09-18T08:18:27.650844+0000","last_unstale":"2024-09-18T08:18:39.369662+0000","last_undegraded":"2024-09-18T08:18:39.369662+0000","last_fullsized":"2024-09-18T08:18:39.369662+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":401,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:18:26.612016+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:18:26.612016+0000","last_clean_scrub_stamp":"2024-09-18T08:18:26.612016+0000","objects_scrubbed":0,"log_size":629,"log_dups_size":0,"ondisk_log_size":629,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:19:32.775774+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41222144,"num_objects":629,"num_object_clones":0,"num_object_copies":1887,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":629,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":629,"num_write_kb":40256,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,0],"acting":[3,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.e","version":"271'19","reported_seq":448,"reported_epoch":406,"state":"active+clean","last_fresh":"2024-09-18T08:18:34.428336+0000","last_change":"2024-09-18T08:17:30.690655+0000","last_active":"2024-09-18T08:18:34.428336+0000","last_peered":"2024-09-18T08:18:34.428336+0000","last_clean":"2024-09-18T08:18:34.428336+0000","last_became_active":"2024-09-18T08:16:13.458001+0000","last_became_peered":"2024-09-18T08:16:13.458001+0000","last_unstale":"2024-09-18T08:18:34.428336+0000","last_undegraded":"2024-09-18T08:18:34.428336+0000","last_fullsized":"2024-09-18T08:18:34.428336+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"271'19","last_scrub_stamp":"2024-09-18T08:17:30.690581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_clean_scrub_stamp":"2024-09-18T08:17:30.690581+0000","objects_scrubbed":5,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:18:59.500251+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":31,"num_read_kb":24,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,3],"acting":[1,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":431,"reported_epoch":407,"state":"active+clean","last_fresh":"2024-09-18T08:18:38.672714+0000","last_change":"2024-09-18T08:18:27.654952+0000","last_active":"2024-09-18T08:18:38.672714+0000","last_peered":"2024-09-18T08:18:38.672714+0000","last_clean":"2024-09-18T08:18:38.672714+0000","last_became_active":"2024-09-18T08:18:27.654623+0000","last_became_peered":"2024-09-18T08:18:27.654623+0000","last_unstale":"2024-09-18T08:18:38.672714+0000","last_undegraded":"2024-09-18T08:18:38.672714+0000","last_fullsized":"2024-09-18T08:18:38.672714+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:18:26.121022+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:46.860092+0000","last_clean_scrub_stamp":"2024-09-18T08:18:26.121022+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:19:36.375673+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,1],"acting":[2,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":168,"reported_epoch":407,"state":"active+clean","last_fresh":"2024-09-18T08:18:38.669258+0000","last_change":"2024-09-18T08:17:38.616479+0000","last_active":"2024-09-18T08:18:38.669258+0000","last_peered":"2024-09-18T08:18:38.669258+0000","last_clean":"2024-09-18T08:18:38.669258+0000","last_became_active":"2024-09-18T08:16:13.443884+0000","last_became_peered":"2024-09-18T08:16:13.443884+0000","last_unstale":"2024-09-18T08:18:38.669258+0000","last_undegraded":"2024-09-18T08:18:38.669258+0000","last_fullsized":"2024-09-18T08:18:38.669258+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:17:38.616405+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:50.886830+0000","last_clean_scrub_stamp":"2024-09-18T08:17:38.616405+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:19:06.381094+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,2],"acting":[5,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":748,"reported_epoch":407,"state":"active+clean","last_fresh":"2024-09-18T08:18:38.673005+0000","last_change":"2024-09-18T08:15:41.465035+0000","last_active":"2024-09-18T08:18:38.673005+0000","last_peered":"2024-09-18T08:18:38.673005+0000","last_clean":"2024-09-18T08:18:38.673005+0000","last_became_active":"2024-09-18T08:15:41.464500+0000","last_became_peered":"2024-09-18T08:15:41.464500+0000","last_unstale":"2024-09-18T08:18:38.673005+0000","last_undegraded":"2024-09-18T08:18:38.673005+0000","last_fullsized":"2024-09-18T08:18:38.673005+0000","mapping_epoch":336,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":337,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:15:02.861116+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_clean_scrub_stamp":"2024-09-18T08:15:02.861116+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":81,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.2","version":"407'630","reported_seq":653,"reported_epoch":407,"state":"active+clean","last_fresh":"2024-09-18T08:18:39.106647+0000","last_change":"2024-09-18T08:18:27.652381+0000","last_active":"2024-09-18T08:18:39.106647+0000","last_peered":"2024-09-18T08:18:39.106647+0000","last_clean":"2024-09-18T08:18:39.106647+0000","last_became_active":"2024-09-18T08:18:27.652112+0000","last_became_peered":"2024-09-18T08:18:27.652112+0000","last_unstale":"2024-09-18T08:18:39.106647+0000","last_undegraded":"2024-09-18T08:18:39.106647+0000","last_fullsized":"2024-09-18T08:18:39.106647+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":401,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:18:26.612016+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:18:26.612016+0000","last_clean_scrub_stamp":"2024-09-18T08:18:26.612016+0000","objects_scrubbed":0,"log_size":630,"log_dups_size":0,"ondisk_log_size":630,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:19:52.998008+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41287680,"num_objects":630,"num_object_clones":0,"num_object_copies":1890,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":630,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":630,"num_write_kb":40320,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.8","version":"74'79","reported_seq":165,"reported_epoch":407,"state":"active+clean","last_fresh":"2024-09-18T08:18:38.668034+0000","last_change":"2024-09-18T08:17:43.558527+0000","last_active":"2024-09-18T08:18:38.668034+0000","last_peered":"2024-09-18T08:18:38.668034+0000","last_clean":"2024-09-18T08:18:38.668034+0000","last_became_active":"2024-09-18T08:15:41.454781+0000","last_became_peered":"2024-09-18T08:15:41.454781+0000","last_unstale":"2024-09-18T08:18:38.668034+0000","last_undegraded":"2024-09-18T08:18:38.668034+0000","last_fullsized":"2024-09-18T08:18:38.668034+0000","mapping_epoch":336,"log_start":"0'0","ondisk_log_start":"0'0","created":152,"last_epoch_clean":337,"parent":"0.0","parent_split_bits":4,"last_scrub":"74'79","last_scrub_stamp":"2024-09-18T08:17:43.558474+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:00:11.893742+0000","last_clean_scrub_stamp":"2024-09-18T08:17:43.558474+0000","objects_scrubbed":1,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:18:45.862783+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262688,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":262688,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,3],"acting":[5,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.d","version":"271'31","reported_seq":381,"reported_epoch":407,"state":"active+clean","last_fresh":"2024-09-18T08:18:38.672765+0000","last_change":"2024-09-18T08:16:13.511045+0000","last_active":"2024-09-18T08:18:38.672765+0000","last_peered":"2024-09-18T08:18:38.672765+0000","last_clean":"2024-09-18T08:18:38.672765+0000","last_became_active":"2024-09-18T08:16:13.466471+0000","last_became_peered":"2024-09-18T08:16:13.466471+0000","last_unstale":"2024-09-18T08:18:38.672765+0000","last_undegraded":"2024-09-18T08:18:38.672765+0000","last_fullsized":"2024-09-18T08:18:38.672765+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"271'31","last_scrub_stamp":"2024-09-18T08:15:03.678670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_clean_scrub_stamp":"2024-09-18T08:15:03.678670+0000","objects_scrubbed":0,"log_size":31,"log_dups_size":0,"ondisk_log_size":31,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:19:03.954219+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":48,"num_read_kb":38,"num_write":32,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,6],"acting":[2,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":411,"reported_epoch":406,"state":"active+clean","last_fresh":"2024-09-18T08:18:34.429287+0000","last_change":"2024-09-18T08:18:18.567839+0000","last_active":"2024-09-18T08:18:34.429287+0000","last_peered":"2024-09-18T08:18:34.429287+0000","last_clean":"2024-09-18T08:18:34.429287+0000","last_became_active":"2024-09-18T08:16:13.483143+0000","last_became_peered":"2024-09-18T08:16:13.483143+0000","last_unstale":"2024-09-18T08:18:34.429287+0000","last_undegraded":"2024-09-18T08:18:34.429287+0000","last_fullsized":"2024-09-18T08:18:34.429287+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:18:18.567791+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:46.860092+0000","last_clean_scrub_stamp":"2024-09-18T08:18:18.567791+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:19:28.873689+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,3],"acting":[1,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":632,"reported_epoch":406,"state":"active+clean","last_fresh":"2024-09-18T08:18:34.429656+0000","last_change":"2024-09-18T08:16:13.460968+0000","last_active":"2024-09-18T08:18:34.429656+0000","last_peered":"2024-09-18T08:18:34.429656+0000","last_clean":"2024-09-18T08:18:34.429656+0000","last_became_active":"2024-09-18T08:16:13.460650+0000","last_became_peered":"2024-09-18T08:16:13.460650+0000","last_unstale":"2024-09-18T08:18:34.429656+0000","last_undegraded":"2024-09-18T08:18:34.429656+0000","last_fullsized":"2024-09-18T08:18:34.429656+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:14:42.230420+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:50.886830+0000","last_clean_scrub_stamp":"2024-09-18T08:14:42.230420+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:18:57.718405+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,6],"acting":[1,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":471,"reported_epoch":407,"state":"active+clean","last_fresh":"2024-09-18T08:18:38.675081+0000","last_change":"2024-09-18T08:17:33.946061+0000","last_active":"2024-09-18T08:18:38.675081+0000","last_peered":"2024-09-18T08:18:38.675081+0000","last_clean":"2024-09-18T08:18:38.675081+0000","last_became_active":"2024-09-18T08:16:13.457516+0000","last_became_peered":"2024-09-18T08:16:13.457516+0000","last_unstale":"2024-09-18T08:18:38.675081+0000","last_undegraded":"2024-09-18T08:18:38.675081+0000","last_fullsized":"2024-09-18T08:18:38.675081+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:17:33.945785+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_clean_scrub_stamp":"2024-09-18T08:17:33.945785+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:18:58.386817+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.3","version":"406'501","reported_seq":522,"reported_epoch":406,"state":"active+clean","last_fresh":"2024-09-18T08:18:38.516842+0000","last_change":"2024-09-18T08:18:27.658212+0000","last_active":"2024-09-18T08:18:38.516842+0000","last_peered":"2024-09-18T08:18:38.516842+0000","last_clean":"2024-09-18T08:18:38.516842+0000","last_became_active":"2024-09-18T08:18:27.657815+0000","last_became_peered":"2024-09-18T08:18:27.657815+0000","last_unstale":"2024-09-18T08:18:38.516842+0000","last_undegraded":"2024-09-18T08:18:38.516842+0000","last_fullsized":"2024-09-18T08:18:38.516842+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":401,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:18:26.612016+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:18:26.612016+0000","last_clean_scrub_stamp":"2024-09-18T08:18:26.612016+0000","objects_scrubbed":0,"log_size":501,"log_dups_size":0,"ondisk_log_size":501,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:19:45.500162+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32833536,"num_objects":501,"num_object_clones":0,"num_object_copies":1503,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":501,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":501,"num_write_kb":32064,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.9","version":"74'79","reported_seq":582,"reported_epoch":407,"state":"active+clean","last_fresh":"2024-09-18T08:18:38.673580+0000","last_change":"2024-09-18T08:18:27.660987+0000","last_active":"2024-09-18T08:18:38.673580+0000","last_peered":"2024-09-18T08:18:38.673580+0000","last_clean":"2024-09-18T08:18:38.673580+0000","last_became_active":"2024-09-18T08:18:27.660610+0000","last_became_peered":"2024-09-18T08:18:27.660610+0000","last_unstale":"2024-09-18T08:18:38.673580+0000","last_undegraded":"2024-09-18T08:18:38.673580+0000","last_fullsized":"2024-09-18T08:18:38.673580+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":152,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":4,"last_scrub":"74'79","last_scrub_stamp":"2024-09-18T08:17:44.331322+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:00:11.893742+0000","last_clean_scrub_stamp":"2024-09-18T08:17:44.331322+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:19:10.784557+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.c","version":"271'20","reported_seq":359,"reported_epoch":407,"state":"active+clean","last_fresh":"2024-09-18T08:18:38.673302+0000","last_change":"2024-09-18T08:16:13.526582+0000","last_active":"2024-09-18T08:18:38.673302+0000","last_peered":"2024-09-18T08:18:38.673302+0000","last_clean":"2024-09-18T08:18:38.673302+0000","last_became_active":"2024-09-18T08:16:13.476918+0000","last_became_peered":"2024-09-18T08:16:13.476918+0000","last_unstale":"2024-09-18T08:18:38.673302+0000","last_undegraded":"2024-09-18T08:18:38.673302+0000","last_fullsized":"2024-09-18T08:18:38.673302+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"271'20","last_scrub_stamp":"2024-09-18T08:15:56.673129+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_clean_scrub_stamp":"2024-09-18T08:15:56.673129+0000","objects_scrubbed":0,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":28,"num_read_kb":23,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,1],"acting":[3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":769,"reported_epoch":407,"state":"active+clean","last_fresh":"2024-09-18T08:18:38.674499+0000","last_change":"2024-09-18T08:16:13.479508+0000","last_active":"2024-09-18T08:18:38.674499+0000","last_peered":"2024-09-18T08:18:38.674499+0000","last_clean":"2024-09-18T08:18:38.674499+0000","last_became_active":"2024-09-18T08:16:13.479148+0000","last_became_peered":"2024-09-18T08:16:13.479148+0000","last_unstale":"2024-09-18T08:18:38.674499+0000","last_undegraded":"2024-09-18T08:18:38.674499+0000","last_fullsized":"2024-09-18T08:18:38.674499+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:14:04.626103+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:46.860092+0000","last_clean_scrub_stamp":"2024-09-18T08:14:04.626103+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:18:59.479308+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.d","version":"402'14","reported_seq":995,"reported_epoch":407,"state":"active+clean","last_fresh":"2024-09-18T08:18:38.672713+0000","last_change":"2024-09-18T08:18:27.675655+0000","last_active":"2024-09-18T08:18:38.672713+0000","last_peered":"2024-09-18T08:18:38.672713+0000","last_clean":"2024-09-18T08:18:38.672713+0000","last_became_active":"2024-09-18T08:18:27.653796+0000","last_became_peered":"2024-09-18T08:18:27.653796+0000","last_unstale":"2024-09-18T08:18:38.672713+0000","last_undegraded":"2024-09-18T08:18:38.672713+0000","last_fullsized":"2024-09-18T08:18:38.672713+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":0,"last_scrub":"338'13","last_scrub_stamp":"2024-09-18T08:17:38.886654+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:50.886830+0000","last_clean_scrub_stamp":"2024-09-18T08:17:38.886654+0000","objects_scrubbed":1,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:18:44.202678+0000","scrub_duration":26,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":410,"reported_epoch":406,"state":"active+clean","last_fresh":"2024-09-18T08:18:34.428192+0000","last_change":"2024-09-18T08:16:13.465932+0000","last_active":"2024-09-18T08:18:34.428192+0000","last_peered":"2024-09-18T08:18:34.428192+0000","last_clean":"2024-09-18T08:18:34.428192+0000","last_became_active":"2024-09-18T08:16:13.464924+0000","last_became_peered":"2024-09-18T08:16:13.464924+0000","last_unstale":"2024-09-18T08:18:34.428192+0000","last_undegraded":"2024-09-18T08:18:34.428192+0000","last_fullsized":"2024-09-18T08:18:34.428192+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:15:47.604976+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_clean_scrub_stamp":"2024-09-18T08:15:47.604976+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.4","version":"406'509","reported_seq":530,"reported_epoch":406,"state":"active+clean","last_fresh":"2024-09-18T08:18:38.518416+0000","last_change":"2024-09-18T08:18:27.659656+0000","last_active":"2024-09-18T08:18:38.518416+0000","last_peered":"2024-09-18T08:18:38.518416+0000","last_clean":"2024-09-18T08:18:38.518416+0000","last_became_active":"2024-09-18T08:18:27.659213+0000","last_became_peered":"2024-09-18T08:18:27.659213+0000","last_unstale":"2024-09-18T08:18:38.518416+0000","last_undegraded":"2024-09-18T08:18:38.518416+0000","last_fullsized":"2024-09-18T08:18:38.518416+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":401,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:18:26.612016+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:18:26.612016+0000","last_clean_scrub_stamp":"2024-09-18T08:18:26.612016+0000","objects_scrubbed":0,"log_size":509,"log_dups_size":0,"ondisk_log_size":509,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:19:43.814155+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33357824,"num_objects":509,"num_object_clones":0,"num_object_copies":1527,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":509,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":509,"num_write_kb":32576,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.b","version":"271'15","reported_seq":830,"reported_epoch":406,"state":"active+clean","last_fresh":"2024-09-18T08:18:34.923191+0000","last_change":"2024-09-18T08:17:48.720750+0000","last_active":"2024-09-18T08:18:34.923191+0000","last_peered":"2024-09-18T08:18:34.923191+0000","last_clean":"2024-09-18T08:18:34.923191+0000","last_became_active":"2024-09-18T08:16:13.480279+0000","last_became_peered":"2024-09-18T08:16:13.480279+0000","last_unstale":"2024-09-18T08:18:34.923191+0000","last_undegraded":"2024-09-18T08:18:34.923191+0000","last_fullsized":"2024-09-18T08:18:34.923191+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"271'15","last_scrub_stamp":"2024-09-18T08:17:48.720672+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_clean_scrub_stamp":"2024-09-18T08:17:48.720672+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:19:11.267567+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,3],"acting":[0,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":266,"reported_epoch":407,"state":"active+clean","last_fresh":"2024-09-18T08:18:38.675548+0000","last_change":"2024-09-18T08:18:22.633544+0000","last_active":"2024-09-18T08:18:38.675548+0000","last_peered":"2024-09-18T08:18:38.675548+0000","last_clean":"2024-09-18T08:18:38.675548+0000","last_became_active":"2024-09-18T08:18:17.892088+0000","last_became_peered":"2024-09-18T08:18:17.892088+0000","last_unstale":"2024-09-18T08:18:38.675548+0000","last_undegraded":"2024-09-18T08:18:38.675548+0000","last_fullsized":"2024-09-18T08:18:38.675548+0000","mapping_epoch":395,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":396,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:18:22.633492+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:46.860092+0000","last_clean_scrub_stamp":"2024-09-18T08:18:22.633492+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:19:42.386865+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,6],"acting":[3,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":762,"reported_epoch":407,"state":"active+clean","last_fresh":"2024-09-18T08:18:38.675814+0000","last_change":"2024-09-18T08:18:19.057374+0000","last_active":"2024-09-18T08:18:38.675814+0000","last_peered":"2024-09-18T08:18:38.675814+0000","last_clean":"2024-09-18T08:18:38.675814+0000","last_became_active":"2024-09-18T08:16:13.465129+0000","last_became_peered":"2024-09-18T08:16:13.465129+0000","last_unstale":"2024-09-18T08:18:38.675814+0000","last_undegraded":"2024-09-18T08:18:38.675814+0000","last_fullsized":"2024-09-18T08:18:38.675814+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:18:19.057302+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:50.886830+0000","last_clean_scrub_stamp":"2024-09-18T08:18:19.057302+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:19:40.451915+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":771,"reported_epoch":406,"state":"active+clean","last_fresh":"2024-09-18T08:18:34.922188+0000","last_change":"2024-09-18T08:18:28.901676+0000","last_active":"2024-09-18T08:18:34.922188+0000","last_peered":"2024-09-18T08:18:34.922188+0000","last_clean":"2024-09-18T08:18:34.922188+0000","last_became_active":"2024-09-18T08:18:27.660054+0000","last_became_peered":"2024-09-18T08:18:27.660054+0000","last_unstale":"2024-09-18T08:18:34.922188+0000","last_undegraded":"2024-09-18T08:18:34.922188+0000","last_fullsized":"2024-09-18T08:18:34.922188+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:18:28.901610+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_clean_scrub_stamp":"2024-09-18T08:18:28.901610+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:19:36.710308+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.5","version":"407'556","reported_seq":579,"reported_epoch":407,"state":"active+clean","last_fresh":"2024-09-18T08:18:38.682039+0000","last_change":"2024-09-18T08:18:27.643971+0000","last_active":"2024-09-18T08:18:38.682039+0000","last_peered":"2024-09-18T08:18:38.682039+0000","last_clean":"2024-09-18T08:18:38.682039+0000","last_became_active":"2024-09-18T08:18:27.642478+0000","last_became_peered":"2024-09-18T08:18:27.642478+0000","last_unstale":"2024-09-18T08:18:38.682039+0000","last_undegraded":"2024-09-18T08:18:38.682039+0000","last_fullsized":"2024-09-18T08:18:38.682039+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":401,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:18:26.612016+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:18:26.612016+0000","last_clean_scrub_stamp":"2024-09-18T08:18:26.612016+0000","objects_scrubbed":0,"log_size":556,"log_dups_size":0,"ondisk_log_size":556,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:19:28.507626+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36438016,"num_objects":556,"num_object_clones":0,"num_object_copies":1668,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":556,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":556,"num_write_kb":35584,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.a","version":"271'35","reported_seq":697,"reported_epoch":406,"state":"active+clean","last_fresh":"2024-09-18T08:18:34.429469+0000","last_change":"2024-09-18T08:18:27.683070+0000","last_active":"2024-09-18T08:18:34.429469+0000","last_peered":"2024-09-18T08:18:34.429469+0000","last_clean":"2024-09-18T08:18:34.429469+0000","last_became_active":"2024-09-18T08:18:27.652693+0000","last_became_peered":"2024-09-18T08:18:27.652693+0000","last_unstale":"2024-09-18T08:18:34.429469+0000","last_undegraded":"2024-09-18T08:18:34.429469+0000","last_fullsized":"2024-09-18T08:18:34.429469+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":0,"last_scrub":"271'35","last_scrub_stamp":"2024-09-18T08:18:23.687385+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_clean_scrub_stamp":"2024-09-18T08:18:23.687385+0000","objects_scrubbed":9,"log_size":35,"log_dups_size":0,"ondisk_log_size":35,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:19:52.740313+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":55,"num_read_kb":44,"num_write":36,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":53,"num_bytes_recovered":550,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,7],"acting":[1,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":160,"reported_epoch":407,"state":"active+clean","last_fresh":"2024-09-18T08:18:38.668799+0000","last_change":"2024-09-18T08:18:27.644368+0000","last_active":"2024-09-18T08:18:38.668799+0000","last_peered":"2024-09-18T08:18:38.668799+0000","last_clean":"2024-09-18T08:18:38.668799+0000","last_became_active":"2024-09-18T08:18:27.643663+0000","last_became_peered":"2024-09-18T08:18:27.643663+0000","last_unstale":"2024-09-18T08:18:38.668799+0000","last_undegraded":"2024-09-18T08:18:38.668799+0000","last_fullsized":"2024-09-18T08:18:38.668799+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:12:31.205468+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:46.860092+0000","last_clean_scrub_stamp":"2024-09-18T08:12:31.205468+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":477,"reported_epoch":407,"state":"active+clean","last_fresh":"2024-09-18T08:18:38.672995+0000","last_change":"2024-09-18T08:18:22.116736+0000","last_active":"2024-09-18T08:18:38.672995+0000","last_peered":"2024-09-18T08:18:38.672995+0000","last_clean":"2024-09-18T08:18:38.672995+0000","last_became_active":"2024-09-18T08:18:19.431247+0000","last_became_peered":"2024-09-18T08:18:19.431247+0000","last_unstale":"2024-09-18T08:18:38.672995+0000","last_undegraded":"2024-09-18T08:18:38.672995+0000","last_fullsized":"2024-09-18T08:18:38.672995+0000","mapping_epoch":397,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":398,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:18:22.116686+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:50.886830+0000","last_clean_scrub_stamp":"2024-09-18T08:18:22.116686+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:19:45.633126+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,5],"acting":[2,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":598,"reported_epoch":407,"state":"active+clean","last_fresh":"2024-09-18T08:18:38.672149+0000","last_change":"2024-09-18T08:18:27.655959+0000","last_active":"2024-09-18T08:18:38.672149+0000","last_peered":"2024-09-18T08:18:38.672149+0000","last_clean":"2024-09-18T08:18:38.672149+0000","last_became_active":"2024-09-18T08:18:27.654618+0000","last_became_peered":"2024-09-18T08:18:27.654618+0000","last_unstale":"2024-09-18T08:18:38.672149+0000","last_undegraded":"2024-09-18T08:18:38.672149+0000","last_fullsized":"2024-09-18T08:18:38.672149+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:18:15.051582+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_clean_scrub_stamp":"2024-09-18T08:18:15.051582+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:19:16.499274+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,3],"acting":[2,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.6","version":"406'397","reported_seq":418,"reported_epoch":406,"state":"active+clean","last_fresh":"2024-09-18T08:18:38.064935+0000","last_change":"2024-09-18T08:18:27.660043+0000","last_active":"2024-09-18T08:18:38.064935+0000","last_peered":"2024-09-18T08:18:38.064935+0000","last_clean":"2024-09-18T08:18:38.064935+0000","last_became_active":"2024-09-18T08:18:27.659247+0000","last_became_peered":"2024-09-18T08:18:27.659247+0000","last_unstale":"2024-09-18T08:18:38.064935+0000","last_undegraded":"2024-09-18T08:18:38.064935+0000","last_fullsized":"2024-09-18T08:18:38.064935+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":401,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:18:26.612016+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:18:26.612016+0000","last_clean_scrub_stamp":"2024-09-18T08:18:26.612016+0000","objects_scrubbed":0,"log_size":397,"log_dups_size":0,"ondisk_log_size":397,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:19:28.489488+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26017792,"num_objects":397,"num_object_clones":0,"num_object_copies":1191,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":397,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":397,"num_write_kb":25408,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.9","version":"271'18","reported_seq":470,"reported_epoch":406,"state":"active+clean","last_fresh":"2024-09-18T08:18:34.427669+0000","last_change":"2024-09-18T08:16:13.525130+0000","last_active":"2024-09-18T08:18:34.427669+0000","last_peered":"2024-09-18T08:18:34.427669+0000","last_clean":"2024-09-18T08:18:34.427669+0000","last_became_active":"2024-09-18T08:16:13.482580+0000","last_became_peered":"2024-09-18T08:16:13.482580+0000","last_unstale":"2024-09-18T08:18:34.427669+0000","last_undegraded":"2024-09-18T08:18:34.427669+0000","last_fullsized":"2024-09-18T08:18:34.427669+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"271'18","last_scrub_stamp":"2024-09-18T08:15:49.666776+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_clean_scrub_stamp":"2024-09-18T08:15:49.666776+0000","objects_scrubbed":5,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":33,"num_read_kb":24,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":880,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,6],"acting":[1,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":20,"reported_epoch":406,"state":"active+clean","last_fresh":"2024-09-18T08:18:34.428452+0000","last_change":"2024-09-18T08:18:27.654827+0000","last_active":"2024-09-18T08:18:34.428452+0000","last_peered":"2024-09-18T08:18:34.428452+0000","last_clean":"2024-09-18T08:18:34.428452+0000","last_became_active":"2024-09-18T08:18:27.654488+0000","last_became_peered":"2024-09-18T08:18:27.654488+0000","last_unstale":"2024-09-18T08:18:34.428452+0000","last_undegraded":"2024-09-18T08:18:34.428452+0000","last_fullsized":"2024-09-18T08:18:34.428452+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:17:35.673047+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:46.860092+0000","last_clean_scrub_stamp":"2024-09-18T08:17:35.673047+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:18:55.975483+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":752,"reported_epoch":407,"state":"active+clean","last_fresh":"2024-09-18T08:18:38.672238+0000","last_change":"2024-09-18T08:17:40.958509+0000","last_active":"2024-09-18T08:18:38.672238+0000","last_peered":"2024-09-18T08:18:38.672238+0000","last_clean":"2024-09-18T08:18:38.672238+0000","last_became_active":"2024-09-18T08:14:39.399301+0000","last_became_peered":"2024-09-18T08:14:39.399301+0000","last_unstale":"2024-09-18T08:18:38.672238+0000","last_undegraded":"2024-09-18T08:18:38.672238+0000","last_fullsized":"2024-09-18T08:18:38.672238+0000","mapping_epoch":307,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":308,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:17:40.958275+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:50.886830+0000","last_clean_scrub_stamp":"2024-09-18T08:17:40.958275+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:18:57.914660+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":761,"reported_epoch":407,"state":"active+clean","last_fresh":"2024-09-18T08:18:38.674305+0000","last_change":"2024-09-18T08:18:27.665000+0000","last_active":"2024-09-18T08:18:38.674305+0000","last_peered":"2024-09-18T08:18:38.674305+0000","last_clean":"2024-09-18T08:18:38.674305+0000","last_became_active":"2024-09-18T08:18:27.664412+0000","last_became_peered":"2024-09-18T08:18:27.664412+0000","last_unstale":"2024-09-18T08:18:38.674305+0000","last_undegraded":"2024-09-18T08:18:38.674305+0000","last_fullsized":"2024-09-18T08:18:38.674305+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:17:09.487823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_clean_scrub_stamp":"2024-09-18T08:17:09.487823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.7","version":"407'526","reported_seq":549,"reported_epoch":407,"state":"active+clean","last_fresh":"2024-09-18T08:18:38.677469+0000","last_change":"2024-09-18T08:18:27.651740+0000","last_active":"2024-09-18T08:18:38.677469+0000","last_peered":"2024-09-18T08:18:38.677469+0000","last_clean":"2024-09-18T08:18:38.677469+0000","last_became_active":"2024-09-18T08:18:27.651411+0000","last_became_peered":"2024-09-18T08:18:27.651411+0000","last_unstale":"2024-09-18T08:18:38.677469+0000","last_undegraded":"2024-09-18T08:18:38.677469+0000","last_fullsized":"2024-09-18T08:18:38.677469+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":401,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:18:26.612016+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:18:26.612016+0000","last_clean_scrub_stamp":"2024-09-18T08:18:26.612016+0000","objects_scrubbed":0,"log_size":526,"log_dups_size":0,"ondisk_log_size":526,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:19:56.058465+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34471936,"num_objects":526,"num_object_clones":0,"num_object_copies":1578,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":526,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":526,"num_write_kb":33664,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.8","version":"271'27","reported_seq":215,"reported_epoch":407,"state":"active+clean","last_fresh":"2024-09-18T08:18:38.667930+0000","last_change":"2024-09-18T08:18:27.673319+0000","last_active":"2024-09-18T08:18:38.667930+0000","last_peered":"2024-09-18T08:18:38.667930+0000","last_clean":"2024-09-18T08:18:38.667930+0000","last_became_active":"2024-09-18T08:18:27.643053+0000","last_became_peered":"2024-09-18T08:18:27.643053+0000","last_unstale":"2024-09-18T08:18:38.667930+0000","last_undegraded":"2024-09-18T08:18:38.667930+0000","last_fullsized":"2024-09-18T08:18:38.667930+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":0,"last_scrub":"271'27","last_scrub_stamp":"2024-09-18T08:15:00.680764+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_clean_scrub_stamp":"2024-09-18T08:15:00.680764+0000","objects_scrubbed":7,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":44,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":35,"num_bytes_recovered":550,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.f","version":"62'2","reported_seq":26,"reported_epoch":406,"state":"active+clean","last_fresh":"2024-09-18T08:18:34.428418+0000","last_change":"2024-09-18T08:18:27.672015+0000","last_active":"2024-09-18T08:18:34.428418+0000","last_peered":"2024-09-18T08:18:34.428418+0000","last_clean":"2024-09-18T08:18:34.428418+0000","last_became_active":"2024-09-18T08:18:27.654687+0000","last_became_peered":"2024-09-18T08:18:27.654687+0000","last_unstale":"2024-09-18T08:18:34.428418+0000","last_undegraded":"2024-09-18T08:18:34.428418+0000","last_fullsized":"2024-09-18T08:18:34.428418+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":0,"last_scrub":"62'2","last_scrub_stamp":"2024-09-18T08:17:49.718261+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:46.860092+0000","last_clean_scrub_stamp":"2024-09-18T08:17:49.718261+0000","objects_scrubbed":2,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:19:08.318845+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":184,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,1],"acting":[7,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.9","version":"402'18","reported_seq":27,"reported_epoch":406,"state":"active+clean","last_fresh":"2024-09-18T08:18:34.427904+0000","last_change":"2024-09-18T08:18:27.666046+0000","last_active":"2024-09-18T08:18:34.427904+0000","last_peered":"2024-09-18T08:18:34.427904+0000","last_clean":"2024-09-18T08:18:34.427904+0000","last_became_active":"2024-09-18T08:18:27.652480+0000","last_became_peered":"2024-09-18T08:18:27.652480+0000","last_unstale":"2024-09-18T08:18:34.427904+0000","last_undegraded":"2024-09-18T08:18:34.427904+0000","last_fullsized":"2024-09-18T08:18:34.427904+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":0,"last_scrub":"294'14","last_scrub_stamp":"2024-09-18T08:14:28.686640+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:50.886830+0000","last_clean_scrub_stamp":"2024-09-18T08:14:28.686640+0000","objects_scrubbed":1,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:19:36.702166+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":153,"reported_epoch":407,"state":"active+clean","last_fresh":"2024-09-18T08:18:38.668261+0000","last_change":"2024-09-18T08:15:41.454432+0000","last_active":"2024-09-18T08:18:38.668261+0000","last_peered":"2024-09-18T08:18:38.668261+0000","last_clean":"2024-09-18T08:18:38.668261+0000","last_became_active":"2024-09-18T08:15:41.449889+0000","last_became_peered":"2024-09-18T08:15:41.449889+0000","last_unstale":"2024-09-18T08:18:38.668261+0000","last_undegraded":"2024-09-18T08:18:38.668261+0000","last_fullsized":"2024-09-18T08:18:38.668261+0000","mapping_epoch":336,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":337,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:15:13.161614+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_clean_scrub_stamp":"2024-09-18T08:15:13.161614+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:19:04.647823+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":669,"reported_epoch":406,"state":"active+clean","last_fresh":"2024-09-18T08:18:34.921551+0000","last_change":"2024-09-18T08:15:41.448597+0000","last_active":"2024-09-18T08:18:34.921551+0000","last_peered":"2024-09-18T08:18:34.921551+0000","last_clean":"2024-09-18T08:18:34.921551+0000","last_became_active":"2024-09-18T08:15:41.446544+0000","last_became_peered":"2024-09-18T08:15:41.446544+0000","last_unstale":"2024-09-18T08:18:34.921551+0000","last_undegraded":"2024-09-18T08:18:34.921551+0000","last_fullsized":"2024-09-18T08:18:34.921551+0000","mapping_epoch":336,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":337,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:13:06.860320+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:46.860092+0000","last_clean_scrub_stamp":"2024-09-18T08:13:06.860320+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:19:00.797409+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.17","version":"271'12","reported_seq":810,"reported_epoch":407,"state":"active+clean","last_fresh":"2024-09-18T08:18:38.675430+0000","last_change":"2024-09-18T08:18:27.699070+0000","last_active":"2024-09-18T08:18:38.675430+0000","last_peered":"2024-09-18T08:18:38.675430+0000","last_clean":"2024-09-18T08:18:38.675430+0000","last_became_active":"2024-09-18T08:18:27.663338+0000","last_became_peered":"2024-09-18T08:18:27.663338+0000","last_unstale":"2024-09-18T08:18:38.675430+0000","last_undegraded":"2024-09-18T08:18:38.675430+0000","last_fullsized":"2024-09-18T08:18:38.675430+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":0,"last_scrub":"271'12","last_scrub_stamp":"2024-09-18T08:14:33.700931+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_clean_scrub_stamp":"2024-09-18T08:14:33.700931+0000","objects_scrubbed":3,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":18,"num_read_kb":15,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":153,"reported_epoch":407,"state":"active+clean","last_fresh":"2024-09-18T08:18:38.669301+0000","last_change":"2024-09-18T08:15:41.448984+0000","last_active":"2024-09-18T08:18:38.669301+0000","last_peered":"2024-09-18T08:18:38.669301+0000","last_clean":"2024-09-18T08:18:38.669301+0000","last_became_active":"2024-09-18T08:15:41.445332+0000","last_became_peered":"2024-09-18T08:15:41.445332+0000","last_unstale":"2024-09-18T08:18:38.669301+0000","last_undegraded":"2024-09-18T08:18:38.669301+0000","last_fullsized":"2024-09-18T08:18:38.669301+0000","mapping_epoch":336,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":337,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:15:07.747147+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:50.886830+0000","last_clean_scrub_stamp":"2024-09-18T08:15:07.747147+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:18:56.666371+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,1],"acting":[5,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":28,"reported_epoch":406,"state":"active+clean","last_fresh":"2024-09-18T08:18:34.428234+0000","last_change":"2024-09-18T08:18:29.808538+0000","last_active":"2024-09-18T08:18:34.428234+0000","last_peered":"2024-09-18T08:18:34.428234+0000","last_clean":"2024-09-18T08:18:34.428234+0000","last_became_active":"2024-09-18T08:18:27.653368+0000","last_became_peered":"2024-09-18T08:18:27.653368+0000","last_unstale":"2024-09-18T08:18:34.428234+0000","last_undegraded":"2024-09-18T08:18:34.428234+0000","last_fullsized":"2024-09-18T08:18:34.428234+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:18:29.808476+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_clean_scrub_stamp":"2024-09-18T08:18:29.808476+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:19:54.372875+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,5],"acting":[7,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"271'15","reported_seq":792,"reported_epoch":406,"state":"active+clean","last_fresh":"2024-09-18T08:18:34.923788+0000","last_change":"2024-09-18T08:18:27.694758+0000","last_active":"2024-09-18T08:18:34.923788+0000","last_peered":"2024-09-18T08:18:34.923788+0000","last_clean":"2024-09-18T08:18:34.923788+0000","last_became_active":"2024-09-18T08:18:27.658637+0000","last_became_peered":"2024-09-18T08:18:27.658637+0000","last_unstale":"2024-09-18T08:18:34.923788+0000","last_undegraded":"2024-09-18T08:18:34.923788+0000","last_fullsized":"2024-09-18T08:18:34.923788+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":0,"last_scrub":"271'15","last_scrub_stamp":"2024-09-18T08:17:32.508401+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_clean_scrub_stamp":"2024-09-18T08:17:32.508401+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:18:55.235151+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":392,"reported_epoch":406,"state":"active+clean","last_fresh":"2024-09-18T08:18:34.921841+0000","last_change":"2024-09-18T08:18:27.650329+0000","last_active":"2024-09-18T08:18:34.921841+0000","last_peered":"2024-09-18T08:18:34.921841+0000","last_clean":"2024-09-18T08:18:34.921841+0000","last_became_active":"2024-09-18T08:18:27.649876+0000","last_became_peered":"2024-09-18T08:18:27.649876+0000","last_unstale":"2024-09-18T08:18:34.921841+0000","last_undegraded":"2024-09-18T08:18:34.921841+0000","last_fullsized":"2024-09-18T08:18:34.921841+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:17:50.771903+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:46.860092+0000","last_clean_scrub_stamp":"2024-09-18T08:17:50.771903+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:19:13.310947+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,6],"acting":[0,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":755,"reported_epoch":407,"state":"active+clean","last_fresh":"2024-09-18T08:18:38.675316+0000","last_change":"2024-09-18T08:18:27.660413+0000","last_active":"2024-09-18T08:18:38.675316+0000","last_peered":"2024-09-18T08:18:38.675316+0000","last_clean":"2024-09-18T08:18:38.675316+0000","last_became_active":"2024-09-18T08:18:27.660093+0000","last_became_peered":"2024-09-18T08:18:27.660093+0000","last_unstale":"2024-09-18T08:18:38.675316+0000","last_undegraded":"2024-09-18T08:18:38.675316+0000","last_fullsized":"2024-09-18T08:18:38.675316+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:17:06.569828+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:50.886830+0000","last_clean_scrub_stamp":"2024-09-18T08:17:06.569828+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":31,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":772,"reported_epoch":407,"state":"active+clean","last_fresh":"2024-09-18T08:18:38.672683+0000","last_change":"2024-09-18T08:18:31.070762+0000","last_active":"2024-09-18T08:18:38.672683+0000","last_peered":"2024-09-18T08:18:38.672683+0000","last_clean":"2024-09-18T08:18:38.672683+0000","last_became_active":"2024-09-18T08:18:27.656776+0000","last_became_peered":"2024-09-18T08:18:27.656776+0000","last_unstale":"2024-09-18T08:18:38.672683+0000","last_undegraded":"2024-09-18T08:18:38.672683+0000","last_fullsized":"2024-09-18T08:18:38.672683+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:18:31.070711+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_clean_scrub_stamp":"2024-09-18T08:18:31.070711+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:19:41.398335+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,0],"acting":[2,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.15","version":"271'15","reported_seq":188,"reported_epoch":407,"state":"active+clean","last_fresh":"2024-09-18T08:18:38.668930+0000","last_change":"2024-09-18T08:18:27.674004+0000","last_active":"2024-09-18T08:18:38.668930+0000","last_peered":"2024-09-18T08:18:38.668930+0000","last_clean":"2024-09-18T08:18:38.668930+0000","last_became_active":"2024-09-18T08:18:27.646211+0000","last_became_peered":"2024-09-18T08:18:27.646211+0000","last_unstale":"2024-09-18T08:18:38.668930+0000","last_undegraded":"2024-09-18T08:18:38.668930+0000","last_fullsized":"2024-09-18T08:18:38.668930+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":0,"last_scrub":"271'15","last_scrub_stamp":"2024-09-18T08:17:32.638811+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_clean_scrub_stamp":"2024-09-18T08:17:32.638811+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:18:55.118013+0000","scrub_duration":20,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":25,"num_read_kb":19,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":772,"reported_epoch":406,"state":"active+clean","last_fresh":"2024-09-18T08:18:34.922647+0000","last_change":"2024-09-18T08:18:27.656075+0000","last_active":"2024-09-18T08:18:34.922647+0000","last_peered":"2024-09-18T08:18:34.922647+0000","last_clean":"2024-09-18T08:18:34.922647+0000","last_became_active":"2024-09-18T08:18:27.655758+0000","last_became_peered":"2024-09-18T08:18:27.655758+0000","last_unstale":"2024-09-18T08:18:34.922647+0000","last_undegraded":"2024-09-18T08:18:34.922647+0000","last_fullsized":"2024-09-18T08:18:34.922647+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:18:24.804845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:46.860092+0000","last_clean_scrub_stamp":"2024-09-18T08:18:24.804845+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:19:48.037934+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.14","version":"402'17","reported_seq":1001,"reported_epoch":407,"state":"active+clean","last_fresh":"2024-09-18T08:18:38.673100+0000","last_change":"2024-09-18T08:18:27.695586+0000","last_active":"2024-09-18T08:18:38.673100+0000","last_peered":"2024-09-18T08:18:38.673100+0000","last_clean":"2024-09-18T08:18:38.673100+0000","last_became_active":"2024-09-18T08:18:27.662747+0000","last_became_peered":"2024-09-18T08:18:27.662747+0000","last_unstale":"2024-09-18T08:18:38.673100+0000","last_undegraded":"2024-09-18T08:18:38.673100+0000","last_fullsized":"2024-09-18T08:18:38.673100+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":0,"last_scrub":"310'16","last_scrub_stamp":"2024-09-18T08:17:39.424098+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:50.886830+0000","last_clean_scrub_stamp":"2024-09-18T08:17:39.424098+0000","objects_scrubbed":1,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:19:08.589109+0000","scrub_duration":16,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":483,"reported_epoch":407,"state":"active+clean","last_fresh":"2024-09-18T08:18:38.675377+0000","last_change":"2024-09-18T08:18:24.109825+0000","last_active":"2024-09-18T08:18:38.675377+0000","last_peered":"2024-09-18T08:18:38.675377+0000","last_clean":"2024-09-18T08:18:38.675377+0000","last_became_active":"2024-09-18T08:16:13.467534+0000","last_became_peered":"2024-09-18T08:16:13.467534+0000","last_unstale":"2024-09-18T08:18:38.675377+0000","last_undegraded":"2024-09-18T08:18:38.675377+0000","last_fullsized":"2024-09-18T08:18:38.675377+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:18:24.109778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_clean_scrub_stamp":"2024-09-18T08:18:24.109778+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:19:32.079476+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,0],"acting":[2,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.14","version":"271'20","reported_seq":828,"reported_epoch":407,"state":"active+clean","last_fresh":"2024-09-18T08:18:38.675383+0000","last_change":"2024-09-18T08:18:27.701608+0000","last_active":"2024-09-18T08:18:38.675383+0000","last_peered":"2024-09-18T08:18:38.675383+0000","last_clean":"2024-09-18T08:18:38.675383+0000","last_became_active":"2024-09-18T08:18:27.663877+0000","last_became_peered":"2024-09-18T08:18:27.663877+0000","last_unstale":"2024-09-18T08:18:38.675383+0000","last_undegraded":"2024-09-18T08:18:38.675383+0000","last_fullsized":"2024-09-18T08:18:38.675383+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":0,"last_scrub":"271'20","last_scrub_stamp":"2024-09-18T08:15:52.819801+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_clean_scrub_stamp":"2024-09-18T08:15:52.819801+0000","objects_scrubbed":5,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":16,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":25,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,7],"acting":[3,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":20,"reported_epoch":406,"state":"active+clean","last_fresh":"2024-09-18T08:18:34.427864+0000","last_change":"2024-09-18T08:18:27.659991+0000","last_active":"2024-09-18T08:18:34.427864+0000","last_peered":"2024-09-18T08:18:34.427864+0000","last_clean":"2024-09-18T08:18:34.427864+0000","last_became_active":"2024-09-18T08:18:27.659410+0000","last_became_peered":"2024-09-18T08:18:27.659410+0000","last_unstale":"2024-09-18T08:18:34.427864+0000","last_undegraded":"2024-09-18T08:18:34.427864+0000","last_fullsized":"2024-09-18T08:18:34.427864+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:17:30.908835+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:46.860092+0000","last_clean_scrub_stamp":"2024-09-18T08:17:30.908835+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:18:42.403459+0000","scrub_duration":21,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,6],"acting":[7,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.15","version":"338'12","reported_seq":201,"reported_epoch":407,"state":"active+clean","last_fresh":"2024-09-18T08:18:38.668547+0000","last_change":"2024-09-18T08:18:24.511983+0000","last_active":"2024-09-18T08:18:38.668547+0000","last_peered":"2024-09-18T08:18:38.668547+0000","last_clean":"2024-09-18T08:18:38.668547+0000","last_became_active":"2024-09-18T08:15:41.455243+0000","last_became_peered":"2024-09-18T08:15:41.455243+0000","last_unstale":"2024-09-18T08:18:38.668547+0000","last_undegraded":"2024-09-18T08:18:38.668547+0000","last_fullsized":"2024-09-18T08:18:38.668547+0000","mapping_epoch":336,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":337,"parent":"0.0","parent_split_bits":0,"last_scrub":"338'12","last_scrub_stamp":"2024-09-18T08:18:24.511895+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:50.886830+0000","last_clean_scrub_stamp":"2024-09-18T08:18:24.511895+0000","objects_scrubbed":1,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:19:41.815901+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":747,"reported_epoch":406,"state":"active+clean","last_fresh":"2024-09-18T08:18:34.923420+0000","last_change":"2024-09-18T08:18:27.660826+0000","last_active":"2024-09-18T08:18:34.923420+0000","last_peered":"2024-09-18T08:18:34.923420+0000","last_clean":"2024-09-18T08:18:34.923420+0000","last_became_active":"2024-09-18T08:18:27.658009+0000","last_became_peered":"2024-09-18T08:18:27.658009+0000","last_unstale":"2024-09-18T08:18:34.923420+0000","last_undegraded":"2024-09-18T08:18:34.923420+0000","last_fullsized":"2024-09-18T08:18:34.923420+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:17:39.513160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_clean_scrub_stamp":"2024-09-18T08:17:39.513160+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:18:56.891390+0000","scrub_duration":31,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.13","version":"271'19","reported_seq":448,"reported_epoch":406,"state":"active+clean","last_fresh":"2024-09-18T08:18:34.428817+0000","last_change":"2024-09-18T08:17:39.665796+0000","last_active":"2024-09-18T08:18:34.428817+0000","last_peered":"2024-09-18T08:18:34.428817+0000","last_clean":"2024-09-18T08:18:34.428817+0000","last_became_active":"2024-09-18T08:16:13.454640+0000","last_became_peered":"2024-09-18T08:16:13.454640+0000","last_unstale":"2024-09-18T08:18:34.428817+0000","last_undegraded":"2024-09-18T08:18:34.428817+0000","last_fullsized":"2024-09-18T08:18:34.428817+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"271'19","last_scrub_stamp":"2024-09-18T08:17:39.665588+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_clean_scrub_stamp":"2024-09-18T08:17:39.665588+0000","objects_scrubbed":5,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:18:55.676655+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":31,"num_read_kb":24,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,5],"acting":[1,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":20,"reported_epoch":406,"state":"active+clean","last_fresh":"2024-09-18T08:18:34.428160+0000","last_change":"2024-09-18T08:18:27.658805+0000","last_active":"2024-09-18T08:18:34.428160+0000","last_peered":"2024-09-18T08:18:34.428160+0000","last_clean":"2024-09-18T08:18:34.428160+0000","last_became_active":"2024-09-18T08:18:27.658476+0000","last_became_peered":"2024-09-18T08:18:27.658476+0000","last_unstale":"2024-09-18T08:18:34.428160+0000","last_undegraded":"2024-09-18T08:18:34.428160+0000","last_fullsized":"2024-09-18T08:18:34.428160+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:18:25.097909+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:46.860092+0000","last_clean_scrub_stamp":"2024-09-18T08:18:25.097909+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:19:29.336077+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":624,"reported_epoch":406,"state":"active+clean","last_fresh":"2024-09-18T08:18:34.429011+0000","last_change":"2024-09-18T08:15:55.335233+0000","last_active":"2024-09-18T08:18:34.429011+0000","last_peered":"2024-09-18T08:18:34.429011+0000","last_clean":"2024-09-18T08:18:34.429011+0000","last_became_active":"2024-09-18T08:15:41.438048+0000","last_became_peered":"2024-09-18T08:15:41.438048+0000","last_unstale":"2024-09-18T08:18:34.429011+0000","last_undegraded":"2024-09-18T08:18:34.429011+0000","last_fullsized":"2024-09-18T08:18:34.429011+0000","mapping_epoch":336,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":337,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:15:55.334983+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:50.886830+0000","last_clean_scrub_stamp":"2024-09-18T08:15:55.334983+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":759,"reported_epoch":407,"state":"active+clean","last_fresh":"2024-09-18T08:18:38.673784+0000","last_change":"2024-09-18T08:17:37.394735+0000","last_active":"2024-09-18T08:18:38.673784+0000","last_peered":"2024-09-18T08:18:38.673784+0000","last_clean":"2024-09-18T08:18:38.673784+0000","last_became_active":"2024-09-18T08:15:41.464961+0000","last_became_peered":"2024-09-18T08:15:41.464961+0000","last_unstale":"2024-09-18T08:18:38.673784+0000","last_undegraded":"2024-09-18T08:18:38.673784+0000","last_fullsized":"2024-09-18T08:18:38.673784+0000","mapping_epoch":336,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":337,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:17:37.394532+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_clean_scrub_stamp":"2024-09-18T08:17:37.394532+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:18:46.957682+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.12","version":"271'15","reported_seq":620,"reported_epoch":406,"state":"active+clean","last_fresh":"2024-09-18T08:18:34.429374+0000","last_change":"2024-09-18T08:18:31.593288+0000","last_active":"2024-09-18T08:18:34.429374+0000","last_peered":"2024-09-18T08:18:34.429374+0000","last_clean":"2024-09-18T08:18:34.429374+0000","last_became_active":"2024-09-18T08:14:39.416284+0000","last_became_peered":"2024-09-18T08:14:39.416284+0000","last_unstale":"2024-09-18T08:18:34.429374+0000","last_undegraded":"2024-09-18T08:18:34.429374+0000","last_fullsized":"2024-09-18T08:18:34.429374+0000","mapping_epoch":307,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":308,"parent":"0.0","parent_split_bits":0,"last_scrub":"271'15","last_scrub_stamp":"2024-09-18T08:18:31.593238+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_clean_scrub_stamp":"2024-09-18T08:18:31.593238+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:19:56.196058+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":25,"num_read_kb":19,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":552,"reported_epoch":407,"state":"active+clean","last_fresh":"2024-09-18T08:18:38.673904+0000","last_change":"2024-09-18T08:18:27.663193+0000","last_active":"2024-09-18T08:18:38.673904+0000","last_peered":"2024-09-18T08:18:38.673904+0000","last_clean":"2024-09-18T08:18:38.673904+0000","last_became_active":"2024-09-18T08:18:27.662885+0000","last_became_peered":"2024-09-18T08:18:27.662885+0000","last_unstale":"2024-09-18T08:18:38.673904+0000","last_undegraded":"2024-09-18T08:18:38.673904+0000","last_fullsized":"2024-09-18T08:18:38.673904+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:15:58.978411+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:46.860092+0000","last_clean_scrub_stamp":"2024-09-18T08:15:58.978411+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":18,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":743,"reported_epoch":407,"state":"active+clean","last_fresh":"2024-09-18T08:18:38.674359+0000","last_change":"2024-09-18T08:18:28.695902+0000","last_active":"2024-09-18T08:18:38.674359+0000","last_peered":"2024-09-18T08:18:38.674359+0000","last_clean":"2024-09-18T08:18:38.674359+0000","last_became_active":"2024-09-18T08:14:39.402406+0000","last_became_peered":"2024-09-18T08:14:39.402406+0000","last_unstale":"2024-09-18T08:18:38.674359+0000","last_undegraded":"2024-09-18T08:18:38.674359+0000","last_fullsized":"2024-09-18T08:18:38.674359+0000","mapping_epoch":307,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":308,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:18:28.695842+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:50.886830+0000","last_clean_scrub_stamp":"2024-09-18T08:18:28.695842+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:19:50.760247+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.10","version":"69'1","reported_seq":747,"reported_epoch":406,"state":"active+clean","last_fresh":"2024-09-18T08:18:34.921372+0000","last_change":"2024-09-18T08:15:41.480106+0000","last_active":"2024-09-18T08:18:34.921372+0000","last_peered":"2024-09-18T08:18:34.921372+0000","last_clean":"2024-09-18T08:18:34.921372+0000","last_became_active":"2024-09-18T08:15:41.444085+0000","last_became_peered":"2024-09-18T08:15:41.444085+0000","last_unstale":"2024-09-18T08:18:34.921372+0000","last_undegraded":"2024-09-18T08:18:34.921372+0000","last_fullsized":"2024-09-18T08:18:34.921372+0000","mapping_epoch":336,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":337,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'1","last_scrub_stamp":"2024-09-18T08:15:00.219678+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_clean_scrub_stamp":"2024-09-18T08:15:00.219678+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":26,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.11","version":"271'19","reported_seq":897,"reported_epoch":407,"state":"active+clean","last_fresh":"2024-09-18T08:18:38.673474+0000","last_change":"2024-09-18T08:18:27.695649+0000","last_active":"2024-09-18T08:18:38.673474+0000","last_peered":"2024-09-18T08:18:38.673474+0000","last_clean":"2024-09-18T08:18:38.673474+0000","last_became_active":"2024-09-18T08:18:27.659293+0000","last_became_peered":"2024-09-18T08:18:27.659293+0000","last_unstale":"2024-09-18T08:18:38.673474+0000","last_undegraded":"2024-09-18T08:18:38.673474+0000","last_fullsized":"2024-09-18T08:18:38.673474+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":0,"last_scrub":"271'19","last_scrub_stamp":"2024-09-18T08:14:38.809490+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_clean_scrub_stamp":"2024-09-18T08:14:38.809490+0000","objects_scrubbed":5,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":32,"num_read_kb":25,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":30,"num_bytes_recovered":660,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":168,"reported_epoch":407,"state":"active+clean","last_fresh":"2024-09-18T08:18:38.669245+0000","last_change":"2024-09-18T08:18:27.644325+0000","last_active":"2024-09-18T08:18:38.669245+0000","last_peered":"2024-09-18T08:18:38.669245+0000","last_clean":"2024-09-18T08:18:38.669245+0000","last_became_active":"2024-09-18T08:18:27.643957+0000","last_became_peered":"2024-09-18T08:18:27.643957+0000","last_unstale":"2024-09-18T08:18:38.669245+0000","last_undegraded":"2024-09-18T08:18:38.669245+0000","last_fullsized":"2024-09-18T08:18:38.669245+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:18:18.611302+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:46.860092+0000","last_clean_scrub_stamp":"2024-09-18T08:18:18.611302+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:19:24.512155+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":20,"reported_epoch":406,"state":"active+clean","last_fresh":"2024-09-18T08:18:34.427603+0000","last_change":"2024-09-18T08:18:27.652157+0000","last_active":"2024-09-18T08:18:34.427603+0000","last_peered":"2024-09-18T08:18:34.427603+0000","last_clean":"2024-09-18T08:18:34.427603+0000","last_became_active":"2024-09-18T08:18:27.651810+0000","last_became_peered":"2024-09-18T08:18:27.651810+0000","last_unstale":"2024-09-18T08:18:34.427603+0000","last_undegraded":"2024-09-18T08:18:34.427603+0000","last_fullsized":"2024-09-18T08:18:34.427603+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:14:37.743856+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:50.886830+0000","last_clean_scrub_stamp":"2024-09-18T08:14:37.743856+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,0],"acting":[7,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":742,"reported_epoch":407,"state":"active+clean","last_fresh":"2024-09-18T08:18:38.674443+0000","last_change":"2024-09-18T08:18:21.660255+0000","last_active":"2024-09-18T08:18:38.674443+0000","last_peered":"2024-09-18T08:18:38.674443+0000","last_clean":"2024-09-18T08:18:38.674443+0000","last_became_active":"2024-09-18T08:06:24.128582+0000","last_became_peered":"2024-09-18T08:06:24.128582+0000","last_unstale":"2024-09-18T08:18:38.674443+0000","last_undegraded":"2024-09-18T08:18:38.674443+0000","last_fullsized":"2024-09-18T08:18:38.674443+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:18:21.660213+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_clean_scrub_stamp":"2024-09-18T08:18:21.660213+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:19:28.412505+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.10","version":"271'8","reported_seq":792,"reported_epoch":407,"state":"active+clean","last_fresh":"2024-09-18T08:18:38.673093+0000","last_change":"2024-09-18T08:15:50.790606+0000","last_active":"2024-09-18T08:18:38.673093+0000","last_peered":"2024-09-18T08:18:38.673093+0000","last_clean":"2024-09-18T08:18:38.673093+0000","last_became_active":"2024-09-18T08:14:39.404257+0000","last_became_peered":"2024-09-18T08:14:39.404257+0000","last_unstale":"2024-09-18T08:18:38.673093+0000","last_undegraded":"2024-09-18T08:18:38.673093+0000","last_fullsized":"2024-09-18T08:18:38.673093+0000","mapping_epoch":307,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":308,"parent":"0.0","parent_split_bits":0,"last_scrub":"271'8","last_scrub_stamp":"2024-09-18T08:15:50.790525+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_clean_scrub_stamp":"2024-09-18T08:15:50.790525+0000","objects_scrubbed":2,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":12,"num_read_kb":10,"num_write":8,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":767,"reported_epoch":406,"state":"active+clean","last_fresh":"2024-09-18T08:18:34.922844+0000","last_change":"2024-09-18T08:18:25.799322+0000","last_active":"2024-09-18T08:18:34.922844+0000","last_peered":"2024-09-18T08:18:34.922844+0000","last_clean":"2024-09-18T08:18:34.922844+0000","last_became_active":"2024-09-18T08:15:41.466964+0000","last_became_peered":"2024-09-18T08:15:41.466964+0000","last_unstale":"2024-09-18T08:18:34.922844+0000","last_undegraded":"2024-09-18T08:18:34.922844+0000","last_fullsized":"2024-09-18T08:18:34.922844+0000","mapping_epoch":336,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":337,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:18:25.799264+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:46.860092+0000","last_clean_scrub_stamp":"2024-09-18T08:18:25.799264+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:19:25.829120+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":392,"reported_epoch":406,"state":"active+clean","last_fresh":"2024-09-18T08:18:34.921388+0000","last_change":"2024-09-18T08:18:27.645673+0000","last_active":"2024-09-18T08:18:34.921388+0000","last_peered":"2024-09-18T08:18:34.921388+0000","last_clean":"2024-09-18T08:18:34.921388+0000","last_became_active":"2024-09-18T08:18:27.645122+0000","last_became_peered":"2024-09-18T08:18:27.645122+0000","last_unstale":"2024-09-18T08:18:34.921388+0000","last_undegraded":"2024-09-18T08:18:34.921388+0000","last_fullsized":"2024-09-18T08:18:34.921388+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:18:14.912464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:50.886830+0000","last_clean_scrub_stamp":"2024-09-18T08:18:14.912464+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:19:16.165963+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,7],"acting":[0,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":558,"reported_epoch":407,"state":"active+clean","last_fresh":"2024-09-18T08:18:38.673192+0000","last_change":"2024-09-18T08:18:27.655596+0000","last_active":"2024-09-18T08:18:38.673192+0000","last_peered":"2024-09-18T08:18:38.673192+0000","last_clean":"2024-09-18T08:18:38.673192+0000","last_became_active":"2024-09-18T08:18:27.654824+0000","last_became_peered":"2024-09-18T08:18:27.654824+0000","last_unstale":"2024-09-18T08:18:38.673192+0000","last_undegraded":"2024-09-18T08:18:38.673192+0000","last_fullsized":"2024-09-18T08:18:38.673192+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:17:31.928876+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_clean_scrub_stamp":"2024-09-18T08:17:31.928876+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:19:00.036673+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,3],"acting":[2,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":596,"reported_epoch":406,"state":"active+clean","last_fresh":"2024-09-18T08:18:34.427762+0000","last_change":"2024-09-18T08:16:13.468068+0000","last_active":"2024-09-18T08:18:34.427762+0000","last_peered":"2024-09-18T08:18:34.427762+0000","last_clean":"2024-09-18T08:18:34.427762+0000","last_became_active":"2024-09-18T08:16:13.466812+0000","last_became_peered":"2024-09-18T08:16:13.466812+0000","last_unstale":"2024-09-18T08:18:34.427762+0000","last_undegraded":"2024-09-18T08:18:34.427762+0000","last_fullsized":"2024-09-18T08:18:34.427762+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:15:45.297240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_clean_scrub_stamp":"2024-09-18T08:15:45.297240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,2],"acting":[1,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":743,"reported_epoch":407,"state":"active+clean","last_fresh":"2024-09-18T08:18:38.673368+0000","last_change":"2024-09-18T08:15:43.836627+0000","last_active":"2024-09-18T08:18:38.673368+0000","last_peered":"2024-09-18T08:18:38.673368+0000","last_clean":"2024-09-18T08:18:38.673368+0000","last_became_active":"2024-09-18T08:14:39.405347+0000","last_became_peered":"2024-09-18T08:14:39.405347+0000","last_unstale":"2024-09-18T08:18:38.673368+0000","last_undegraded":"2024-09-18T08:18:38.673368+0000","last_fullsized":"2024-09-18T08:18:38.673368+0000","mapping_epoch":307,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":308,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:15:43.836549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:46.860092+0000","last_clean_scrub_stamp":"2024-09-18T08:15:43.836549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":20,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"271'19","reported_seq":173,"reported_epoch":407,"state":"active+clean","last_fresh":"2024-09-18T08:18:38.669190+0000","last_change":"2024-09-18T08:18:20.608272+0000","last_active":"2024-09-18T08:18:38.669190+0000","last_peered":"2024-09-18T08:18:38.669190+0000","last_clean":"2024-09-18T08:18:38.669190+0000","last_became_active":"2024-09-18T08:15:41.445198+0000","last_became_peered":"2024-09-18T08:15:41.445198+0000","last_unstale":"2024-09-18T08:18:38.669190+0000","last_undegraded":"2024-09-18T08:18:38.669190+0000","last_fullsized":"2024-09-18T08:18:38.669190+0000","mapping_epoch":336,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":337,"parent":"0.0","parent_split_bits":0,"last_scrub":"271'19","last_scrub_stamp":"2024-09-18T08:18:20.608110+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_clean_scrub_stamp":"2024-09-18T08:18:20.608110+0000","objects_scrubbed":5,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:19:20.791591+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":31,"num_read_kb":24,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,1],"acting":[5,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":763,"reported_epoch":406,"state":"active+clean","last_fresh":"2024-09-18T08:18:34.923530+0000","last_change":"2024-09-18T08:18:27.660335+0000","last_active":"2024-09-18T08:18:34.923530+0000","last_peered":"2024-09-18T08:18:34.923530+0000","last_clean":"2024-09-18T08:18:34.923530+0000","last_became_active":"2024-09-18T08:18:27.659615+0000","last_became_peered":"2024-09-18T08:18:27.659615+0000","last_unstale":"2024-09-18T08:18:34.923530+0000","last_undegraded":"2024-09-18T08:18:34.923530+0000","last_fullsized":"2024-09-18T08:18:34.923530+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:12:31.664456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:50.886830+0000","last_clean_scrub_stamp":"2024-09-18T08:12:31.664456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:19:32.910058+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":11,"num_pg":16,"stat_sum":{"num_bytes":547815424,"num_objects":8359,"num_object_clones":0,"num_object_copies":25077,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8359,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8359,"num_write_kb":534976,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1583415296,"data_stored":1583415296,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":8359,"ondisk_log_size":8359,"up":48,"acting":48,"num_store_stats":7},{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":416,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":1235,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":29,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":120,"ondisk_log_size":120,"up":96,"acting":96,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":178,"num_object_clones":0,"num_object_copies":534,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":178,"num_whiteouts":0,"num_read":1105,"num_read_kb":862,"num_write":703,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":525,"num_bytes_recovered":9922,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":475136,"data_stored":12690,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3639,"internal_metadata":0},"log_size":689,"ondisk_log_size":689,"up":96,"acting":96,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":1613,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":14,"num_read_kb":14,"num_write":10,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":5694,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":94208,"data_stored":6016,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":6,"ondisk_log_size":6,"up":96,"acting":96,"num_store_stats":8},{"poolid":2,"num_pg":3,"stat_sum":{"num_bytes":408,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":407,"num_read_kb":402,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":797,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":8,"ondisk_log_size":8,"up":9,"acting":9,"num_store_stats":7},{"poolid":1,"num_pg":11,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":1573408,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":869,"ondisk_log_size":869,"up":33,"acting":33,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":56,"seq":240518168770,"num_pgs":67,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":642280,"kb_used_data":177944,"kb_used_omap":1,"kb_used_meta":464318,"kb_avail":93102872,"statfs":{"total":95995035648,"available":95337340928,"internally_reserved":0,"allocated":182214656,"data_stored":181657081,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":475462090},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":47,"seq":201863463112,"num_pgs":66,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":692768,"kb_used_data":184016,"kb_used_omap":1,"kb_used_meta":508734,"kb_avail":93052384,"statfs":{"total":95995035648,"available":95285641216,"internally_reserved":0,"allocated":188432384,"data_stored":187881363,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":520944070},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":39,"seq":167503724750,"num_pgs":63,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":914272,"kb_used_data":134800,"kb_used_omap":1,"kb_used_meta":779454,"kb_avail":92830880,"statfs":{"total":95995035648,"available":95058821120,"internally_reserved":0,"allocated":138035200,"data_stored":137485155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":798161350},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1],"upper_bound":2},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":0,"seq":137438953683,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":26,"seq":111669149914,"num_pgs":74,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":976376,"kb_used_data":233000,"kb_used_omap":1,"kb_used_meta":743358,"kb_avail":92768776,"statfs":{"total":95995035648,"available":94995226624,"internally_reserved":0,"allocated":238592000,"data_stored":238017462,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":761199048},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411552,"num_pgs":61,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":878836,"kb_used_data":234016,"kb_used_omap":1,"kb_used_meta":644798,"kb_avail":92866316,"statfs":{"total":95995035648,"available":95095107584,"internally_reserved":0,"allocated":239632384,"data_stored":239073259,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":660273605},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,1,4],"upper_bound":16},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":307,"seq":1318554959921,"num_pgs":72,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":376004,"kb_used_data":207604,"kb_used_omap":7,"kb_used_meta":168376,"kb_avail":93369148,"statfs":{"total":95995035648,"available":95610007552,"internally_reserved":0,"allocated":212586496,"data_stored":212000674,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":8079,"internal_metadata":172417137},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738603,"num_pgs":71,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":820916,"kb_used_data":180260,"kb_used_omap":1,"kb_used_meta":640638,"kb_avail":92924236,"statfs":{"total":95995035648,"available":95154417664,"internally_reserved":0,"allocated":184586240,"data_stored":184016317,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":656013765},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2],"upper_bound":2},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":266240,"data_stored":262688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1131,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1613,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":436,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":46,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1177,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":436,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":46,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1131,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":1694,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":1678,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3639,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":1458,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":81920,"data_stored":2118,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":1584,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1210,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":1584,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1364,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":403,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":416,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":403,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":198443008,"data_stored":198443008,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":213319680,"data_stored":213319680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":241500160,"data_stored":241500160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":291504128,"data_stored":291504128,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":140705792,"data_stored":140705792,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":257687552,"data_stored":257687552,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":240254976,"data_stored":240254976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T08:18:43.356 INFO:tasks.thrashosds.thrasher:Setting 6.8 to [5, 6, 7, 3, 2, 4] 2024-09-18T08:18:43.356 INFO:tasks.thrashosds.thrasher:cmd ['osd', 'pg-upmap-items', '6.8', '5', '6', '7', '3', '2', '4'] 2024-09-18T08:18:43.357 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid 779349ca-7593-11ef-bceb-c7b262605968 -- ceph osd pg-upmap-items 6.8 5 6 7 3 2 4 2024-09-18T08:18:43.405 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid 779349ca-7593-11ef-bceb-c7b262605968 -- ceph osd unset noscrub 2024-09-18T08:18:43.424 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0 2024-09-18T08:18:43.445 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/779349ca-7593-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T08:18:43.560 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:18:43.561 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:18:43.561 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0' 2024-09-18T08:18:43.662 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6 2024-09-18T08:18:43.688 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:18:43.689 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:18:43.689 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6' 2024-09-18T08:18:43.790 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4 2024-09-18T08:18:43.815 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:18:43.816 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:18:43.816 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4' 2024-09-18T08:18:43.917 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0 2024-09-18T08:18:43.945 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:18:43 smithi007 ceph-mon[29459]: from='client.26105 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-18T08:18:43.945 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:18:43 smithi007 ceph-mon[39249]: from='client.26105 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-18T08:18:43.961 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:18:43.961 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:18:43.961 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0' 2024-09-18T08:18:44.033 INFO:tasks.radosbench.radosbench.0.smithi007.stdout: 9 16 20379 20363 141.353 104 0.00781949 0.00706512 2024-09-18T08:18:44.063 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3 2024-09-18T08:18:44.117 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:18:44.118 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:18:44.118 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3' 2024-09-18T08:18:44.144 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/779349ca-7593-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T08:18:44.145 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/779349ca-7593-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T08:18:44.155 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:18:43 smithi050 ceph-mon[32371]: from='client.26105 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-18T08:18:44.157 INFO:teuthology.orchestra.run.smithi007.stdout:osd.0: {} 2024-09-18T08:18:44.157 INFO:teuthology.orchestra.run.smithi007.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-18T08:18:44.197 INFO:teuthology.orchestra.run.smithi007.stdout:osd.1: {} 2024-09-18T08:18:44.197 INFO:teuthology.orchestra.run.smithi007.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-18T08:18:44.219 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0 2024-09-18T08:18:44.234 INFO:teuthology.orchestra.run.smithi007.stdout:osd.2: {} 2024-09-18T08:18:44.234 INFO:teuthology.orchestra.run.smithi007.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-18T08:18:44.262 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:18:44.262 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:18:44.263 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0' 2024-09-18T08:18:44.271 INFO:teuthology.orchestra.run.smithi007.stdout:osd.3: {} 2024-09-18T08:18:44.271 INFO:teuthology.orchestra.run.smithi007.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-18T08:18:44.305 INFO:teuthology.orchestra.run.smithi007.stdout:osd.4: {} 2024-09-18T08:18:44.306 INFO:teuthology.orchestra.run.smithi007.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-18T08:18:44.324 INFO:teuthology.orchestra.run.smithi007.stdout:osd.5: {} 2024-09-18T08:18:44.324 INFO:teuthology.orchestra.run.smithi007.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-18T08:18:44.340 INFO:teuthology.orchestra.run.smithi007.stdout:osd.6: {} 2024-09-18T08:18:44.340 INFO:teuthology.orchestra.run.smithi007.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-18T08:18:44.357 INFO:teuthology.orchestra.run.smithi007.stdout:osd.7: {} 2024-09-18T08:18:44.357 INFO:teuthology.orchestra.run.smithi007.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-18T08:18:44.364 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5 2024-09-18T08:18:44.385 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:18:44.386 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:18:44.386 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5' 2024-09-18T08:18:44.487 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4 2024-09-18T08:18:44.518 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:18:44.519 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:18:44.519 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4' 2024-09-18T08:18:44.620 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5 2024-09-18T08:18:44.649 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:18:44.650 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:18:44.650 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5' 2024-09-18T08:18:44.751 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5 2024-09-18T08:18:44.783 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:18:44.784 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:18:44.784 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5' 2024-09-18T08:18:44.885 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0 2024-09-18T08:18:44.905 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:18:44 smithi050 ceph-mon[32371]: pgmap v801: 158 pgs: 158 active+clean; 759 MiB data, 6.1 GiB used, 620 GiB / 626 GiB avail; 1.2 KiB/s rd, 95 MiB/s wr, 1.52k op/s 2024-09-18T08:18:44.905 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:18:44 smithi050 ceph-mon[32371]: from='client.? 172.21.15.7:0/3886620571' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-18T08:18:44.928 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:18:44.929 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:18:44.929 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0' 2024-09-18T08:18:44.979 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:18:44 smithi007 ceph-mon[29459]: pgmap v801: 158 pgs: 158 active+clean; 759 MiB data, 6.1 GiB used, 620 GiB / 626 GiB avail; 1.2 KiB/s rd, 95 MiB/s wr, 1.52k op/s 2024-09-18T08:18:44.980 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:18:44 smithi007 ceph-mon[29459]: from='client.? 172.21.15.7:0/3886620571' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-18T08:18:44.980 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:18:44 smithi007 ceph-mon[39249]: pgmap v801: 158 pgs: 158 active+clean; 759 MiB data, 6.1 GiB used, 620 GiB / 626 GiB avail; 1.2 KiB/s rd, 95 MiB/s wr, 1.52k op/s 2024-09-18T08:18:44.980 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:18:44 smithi007 ceph-mon[39249]: from='client.? 172.21.15.7:0/3886620571' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-18T08:18:45.031 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6 2024-09-18T08:18:45.034 INFO:tasks.radosbench.radosbench.0.smithi007.stdout: 10 16 22115 22099 138.066 108.5 0.00468412 0.00718666 2024-09-18T08:18:45.054 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:18:45.055 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:18:45.055 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6' 2024-09-18T08:18:45.068 DEBUG:teuthology.orchestra.run.smithi050:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid 779349ca-7593-11ef-bceb-c7b262605968 -- ceph daemon osd.6 dump_ops_in_flight 2024-09-18T08:18:45.156 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5 2024-09-18T08:18:45.188 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:18:45.189 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:18:45.189 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5' 2024-09-18T08:18:45.290 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4 2024-09-18T08:18:45.325 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:18:45.325 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:18:45.326 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4' 2024-09-18T08:18:45.427 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2 2024-09-18T08:18:45.466 INFO:teuthology.orchestra.run.smithi050.stderr:Inferring config /var/lib/ceph/779349ca-7593-11ef-bceb-c7b262605968/mon.b/config 2024-09-18T08:18:45.466 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:18:45.467 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:18:45.467 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2' 2024-09-18T08:18:45.568 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7 2024-09-18T08:18:45.598 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:18:45.600 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:18:45.600 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7' 2024-09-18T08:18:45.701 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1 2024-09-18T08:18:45.729 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:18:45.729 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:18:45.730 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1' 2024-09-18T08:18:45.831 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2 2024-09-18T08:18:45.865 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:18:45.865 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:18:45.865 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2' 2024-09-18T08:18:45.966 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3 2024-09-18T08:18:46.008 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:18:46.009 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:18:46.010 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3' 2024-09-18T08:18:46.033 INFO:tasks.radosbench.radosbench.0.smithi007.stdout: 11 16 23970 23954 136.054 115.938 0.00175938 0.00733849 2024-09-18T08:18:46.078 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2024-09-18T08:18:46.078 INFO:teuthology.orchestra.run.smithi050.stdout: "ops": [], 2024-09-18T08:18:46.078 INFO:teuthology.orchestra.run.smithi050.stdout: "num_ops": 0 2024-09-18T08:18:46.078 INFO:teuthology.orchestra.run.smithi050.stdout:} 2024-09-18T08:18:46.112 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2 2024-09-18T08:18:46.146 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:18:46.146 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:18:46.147 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2' 2024-09-18T08:18:46.247 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0 2024-09-18T08:18:46.288 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:18:46.289 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:18:46.289 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0' 2024-09-18T08:18:46.390 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2 2024-09-18T08:18:46.431 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:18:46.431 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:18:46.432 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2' 2024-09-18T08:18:46.533 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1 2024-09-18T08:18:46.570 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:18:46.571 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:18:46.571 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1' 2024-09-18T08:18:46.672 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5 2024-09-18T08:18:46.699 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:18:46.700 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:18:46.700 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5' 2024-09-18T08:18:46.801 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2 2024-09-18T08:18:46.825 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:18:46.826 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:18:46.826 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2' 2024-09-18T08:18:46.927 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0 2024-09-18T08:18:46.968 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:18:46.969 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:18:46.969 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0' 2024-09-18T08:18:46.979 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:18:46 smithi007 ceph-mon[29459]: pgmap v802: 158 pgs: 158 active+clean; 1.2 GiB data, 7.2 GiB used, 619 GiB / 626 GiB avail; 1.2 KiB/s rd, 153 MiB/s wr, 2.46k op/s 2024-09-18T08:18:46.980 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:18:46 smithi007 ceph-mon[39249]: pgmap v802: 158 pgs: 158 active+clean; 1.2 GiB data, 7.2 GiB used, 619 GiB / 626 GiB avail; 1.2 KiB/s rd, 153 MiB/s wr, 2.46k op/s 2024-09-18T08:18:46.980 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Sep 18 08:18:46 smithi007 ceph-779349ca-7593-11ef-bceb-c7b262605968-mgr-y[29688]: ::ffff:172.21.15.50 - - [18/Sep/2024:08:18:46] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-09-18T08:18:47.034 INFO:tasks.radosbench.radosbench.0.smithi007.stdout: 12 16 25536 25520 132.873 97.875 0.0105202 0.00751894 2024-09-18T08:18:47.070 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1 2024-09-18T08:18:47.092 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:18:47.093 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:18:47.093 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1' 2024-09-18T08:18:47.155 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:18:46 smithi050 ceph-mon[32371]: pgmap v802: 158 pgs: 158 active+clean; 1.2 GiB data, 7.2 GiB used, 619 GiB / 626 GiB avail; 1.2 KiB/s rd, 153 MiB/s wr, 2.46k op/s 2024-09-18T08:18:47.194 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4 2024-09-18T08:18:47.217 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:18:47.217 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:18:47.218 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4' 2024-09-18T08:18:47.319 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7 2024-09-18T08:18:47.348 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:18:47.349 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:18:47.349 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7' 2024-09-18T08:18:47.423 DEBUG:teuthology.orchestra.run.smithi050:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid 779349ca-7593-11ef-bceb-c7b262605968 -- ceph daemon osd.6 dump_blocked_ops 2024-09-18T08:18:47.449 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1 2024-09-18T08:18:47.487 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:18:47.488 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:18:47.488 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1' 2024-09-18T08:18:47.589 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1 2024-09-18T08:18:47.617 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:18:47.618 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:18:47.619 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1' 2024-09-18T08:18:47.719 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3 2024-09-18T08:18:47.731 INFO:teuthology.orchestra.run.smithi050.stderr:Inferring config /var/lib/ceph/779349ca-7593-11ef-bceb-c7b262605968/mon.b/config 2024-09-18T08:18:47.764 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:18:47.765 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:18:47.765 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3' 2024-09-18T08:18:47.866 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5 2024-09-18T08:18:47.888 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:18:47.889 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:18:47.889 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5' 2024-09-18T08:18:47.992 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2 2024-09-18T08:18:48.029 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:18:48.029 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:18:48.030 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2' 2024-09-18T08:18:48.034 INFO:tasks.radosbench.radosbench.0.smithi007.stdout: 13 16 26701 26685 128.253 72.8125 0.00755313 0.00779076 2024-09-18T08:18:48.130 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3 2024-09-18T08:18:48.155 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:18:48.155 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:18:48.156 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3' 2024-09-18T08:18:48.257 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3 2024-09-18T08:18:48.294 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2024-09-18T08:18:48.294 INFO:teuthology.orchestra.run.smithi050.stdout: "ops": [], 2024-09-18T08:18:48.294 INFO:teuthology.orchestra.run.smithi050.stdout: "complaint_time": 30, 2024-09-18T08:18:48.294 INFO:teuthology.orchestra.run.smithi050.stdout: "num_blocked_ops": 0 2024-09-18T08:18:48.294 INFO:teuthology.orchestra.run.smithi050.stdout:} 2024-09-18T08:18:48.295 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:18:48.295 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:18:48.296 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3' 2024-09-18T08:18:48.397 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1 2024-09-18T08:18:48.438 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:18:48.439 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:18:48.439 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1' 2024-09-18T08:18:48.540 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0 2024-09-18T08:18:48.580 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:18:48.580 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:18:48.581 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0' 2024-09-18T08:18:48.681 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2 2024-09-18T08:18:48.721 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:18:48.722 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:18:48.722 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2' 2024-09-18T08:18:48.823 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1 2024-09-18T08:18:48.867 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:18:48.868 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:18:48.869 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1' 2024-09-18T08:18:48.970 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1 2024-09-18T08:18:48.980 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:18:48 smithi007 ceph-mon[29459]: pgmap v803: 158 pgs: 158 active+clean; 1.2 GiB data, 7.2 GiB used, 619 GiB / 626 GiB avail; 582 B/s rd, 119 MiB/s wr, 1.91k op/s 2024-09-18T08:18:48.980 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:18:48 smithi007 ceph-mon[39249]: pgmap v803: 158 pgs: 158 active+clean; 1.2 GiB data, 7.2 GiB used, 619 GiB / 626 GiB avail; 582 B/s rd, 119 MiB/s wr, 1.91k op/s 2024-09-18T08:18:49.009 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:18:49.009 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:18:49.010 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1' 2024-09-18T08:18:49.034 INFO:tasks.radosbench.radosbench.0.smithi007.stdout: 14 16 28402 28386 126.685 106.312 0.00888178 0.00787356 2024-09-18T08:18:49.080 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:18:48 smithi050 ceph-mon[32371]: pgmap v803: 158 pgs: 158 active+clean; 1.2 GiB data, 7.2 GiB used, 619 GiB / 626 GiB avail; 582 B/s rd, 119 MiB/s wr, 1.91k op/s 2024-09-18T08:18:49.081 DEBUG:teuthology.orchestra.run.smithi050:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid 779349ca-7593-11ef-bceb-c7b262605968 -- ceph daemon osd.6 dump_historic_ops 2024-09-18T08:18:49.111 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7 2024-09-18T08:18:49.153 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:18:49.154 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:18:49.154 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7' 2024-09-18T08:18:49.255 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1 2024-09-18T08:18:49.292 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:18:49.293 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:18:49.293 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1' 2024-09-18T08:18:49.394 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2 2024-09-18T08:18:49.408 INFO:teuthology.orchestra.run.smithi050.stderr:Inferring config /var/lib/ceph/779349ca-7593-11ef-bceb-c7b262605968/mon.b/config 2024-09-18T08:18:49.435 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:18:49.436 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:18:49.436 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2' 2024-09-18T08:18:49.537 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2 2024-09-18T08:18:49.575 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:18:49.576 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:18:49.577 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2' 2024-09-18T08:18:49.678 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7 2024-09-18T08:18:49.713 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:18:49.713 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:18:49.714 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7' 2024-09-18T08:18:49.815 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1 2024-09-18T08:18:49.839 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:18:49 smithi050 ceph-mon[32371]: from='mgr.24470 172.21.15.7:0/2594390261' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-18T08:18:49.839 INFO:journalctl@ceph.iscsi.iscsi.a.smithi050.stdout:Sep 18 08:18:49 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-iscsi-iscsi-a[57782]: debug there is no tcmu-runner data available 2024-09-18T08:18:49.862 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:18:49.863 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:18:49.863 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1' 2024-09-18T08:18:49.964 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4 2024-09-18T08:18:49.979 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:18:49 smithi007 ceph-mon[29459]: from='mgr.24470 172.21.15.7:0/2594390261' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-18T08:18:49.980 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:18:49 smithi007 ceph-mon[39249]: from='mgr.24470 172.21.15.7:0/2594390261' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-18T08:18:49.989 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:18:49.989 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:18:49.990 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4' 2024-09-18T08:18:50.034 INFO:tasks.radosbench.radosbench.0.smithi007.stdout: 15 16 29835 29819 124.21 89.5625 0.00633297 0.0080445 2024-09-18T08:18:50.090 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0 2024-09-18T08:18:50.125 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:18:50.126 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:18:50.126 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0' 2024-09-18T08:18:50.227 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2 2024-09-18T08:18:50.266 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:18:50.267 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:18:50.267 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2' 2024-09-18T08:18:50.307 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2024-09-18T08:18:50.307 INFO:teuthology.orchestra.run.smithi050.stdout: "size": 20, 2024-09-18T08:18:50.307 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 600, 2024-09-18T08:18:50.307 INFO:teuthology.orchestra.run.smithi050.stdout: "ops": [ 2024-09-18T08:18:50.307 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:18:50.307 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_repop(client.15582.0:123972 9.0 e261/244 9:0e6c2bd0:::benchmark_data_smithi007_124019_object123971:head v 261'7682, mlcod=261'7678)", 2024-09-18T08:18:50.307 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-09-18T08:12:31.108179+0000", 2024-09-18T08:18:50.307 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 378.85651754499997, 2024-09-18T08:18:50.307 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.029088182000000001, 2024-09-18T08:18:50.308 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-09-18T08:18:50.308 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-09-18T08:18:50.308 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T08:18:50.308 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-09-18T08:18:50.308 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:18:50.308 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-09-18T08:18:50.308 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:12:31.108179+0000", 2024-09-18T08:18:50.308 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-18T08:18:50.308 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:18:50.308 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:18:50.308 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-09-18T08:18:50.308 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:12:31.108179+0000", 2024-09-18T08:18:50.308 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-18T08:18:50.309 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:18:50.309 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:18:50.309 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-09-18T08:18:50.309 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:12:31.108183+0000", 2024-09-18T08:18:50.309 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4.0029999999999996e-06 2024-09-18T08:18:50.309 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:18:50.309 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:18:50.309 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-09-18T08:18:50.309 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:12:31.108232+0000", 2024-09-18T08:18:50.309 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4.9462000000000002e-05 2024-09-18T08:18:50.309 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:18:50.309 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:18:50.309 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-09-18T08:18:50.309 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:12:31.108247+0000", 2024-09-18T08:18:50.310 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.5350999999999999e-05 2024-09-18T08:18:50.310 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:18:50.310 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:18:50.310 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-09-18T08:18:50.310 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:12:31.108255+0000", 2024-09-18T08:18:50.310 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 7.9449999999999993e-06 2024-09-18T08:18:50.310 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:18:50.310 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:18:50.310 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-18T08:18:50.310 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:12:31.109933+0000", 2024-09-18T08:18:50.310 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.001677869 2024-09-18T08:18:50.310 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:18:50.310 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:18:50.311 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-09-18T08:18:50.311 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:12:31.109957+0000", 2024-09-18T08:18:50.311 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 2.3374e-05 2024-09-18T08:18:50.311 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:18:50.311 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:18:50.311 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "commit_sent", 2024-09-18T08:18:50.311 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:12:31.137170+0000", 2024-09-18T08:18:50.311 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.027213580000000001 2024-09-18T08:18:50.311 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:18:50.311 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:18:50.311 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-09-18T08:18:50.311 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:12:31.137267+0000", 2024-09-18T08:18:50.311 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 9.6597999999999999e-05 2024-09-18T08:18:50.311 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-18T08:18:50.312 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-09-18T08:18:50.312 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-18T08:18:50.312 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:18:50.312 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:18:50.312 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_repop(client.15582.0:123975 9.0 e261/244 9:030e6d7f:::benchmark_data_smithi007_124019_object123974:head v 261'7683, mlcod=261'7680)", 2024-09-18T08:18:50.312 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-09-18T08:12:31.113187+0000", 2024-09-18T08:18:50.312 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 378.85150888800001, 2024-09-18T08:18:50.312 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.027941135999999998, 2024-09-18T08:18:50.312 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-09-18T08:18:50.312 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-09-18T08:18:50.312 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T08:18:50.312 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-09-18T08:18:50.312 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:18:50.312 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-09-18T08:18:50.313 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:12:31.113187+0000", 2024-09-18T08:18:50.313 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-18T08:18:50.313 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:18:50.313 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:18:50.313 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-09-18T08:18:50.313 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:12:31.113187+0000", 2024-09-18T08:18:50.313 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-18T08:18:50.313 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:18:50.313 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:18:50.313 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-09-18T08:18:50.313 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:12:31.113189+0000", 2024-09-18T08:18:50.313 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.4500000000000001e-06 2024-09-18T08:18:50.313 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:18:50.313 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:18:50.314 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-09-18T08:18:50.314 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:12:31.113223+0000", 2024-09-18T08:18:50.314 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 3.4588000000000002e-05 2024-09-18T08:18:50.314 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:18:50.314 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:18:50.314 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-09-18T08:18:50.314 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:12:31.113240+0000", 2024-09-18T08:18:50.314 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.6580999999999999e-05 2024-09-18T08:18:50.314 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:18:50.314 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:18:50.314 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-09-18T08:18:50.314 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:12:31.113248+0000", 2024-09-18T08:18:50.314 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 8.0560000000000005e-06 2024-09-18T08:18:50.314 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:18:50.315 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:18:50.315 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-18T08:18:50.315 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:12:31.113470+0000", 2024-09-18T08:18:50.315 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00022196100000000001 2024-09-18T08:18:50.315 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:18:50.315 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:18:50.315 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-09-18T08:18:50.315 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:12:31.113490+0000", 2024-09-18T08:18:50.315 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 2.0377000000000002e-05 2024-09-18T08:18:50.315 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:18:50.315 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:18:50.315 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "commit_sent", 2024-09-18T08:18:50.315 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:12:31.140806+0000", 2024-09-18T08:18:50.315 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.027315715000000001 2024-09-18T08:18:50.316 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:18:50.316 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:18:50.316 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-09-18T08:18:50.316 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:12:31.141128+0000", 2024-09-18T08:18:50.316 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00032240799999999999 2024-09-18T08:18:50.316 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-18T08:18:50.316 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-09-18T08:18:50.316 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-18T08:18:50.316 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:18:50.316 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:18:50.316 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_repop(client.15966.0:12235 10.b e320/311 10:df316ae8:::benchmark_data_smithi007_148834_object12234:head v 320'796, mlcod=319'794)", 2024-09-18T08:18:50.316 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-09-18T08:14:55.670832+0000", 2024-09-18T08:18:50.316 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 234.29386409099999, 2024-09-18T08:18:50.317 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.070438830999999993, 2024-09-18T08:18:50.317 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-09-18T08:18:50.317 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-09-18T08:18:50.317 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T08:18:50.317 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-09-18T08:18:50.317 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:18:50.317 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-09-18T08:18:50.317 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:14:55.670832+0000", 2024-09-18T08:18:50.317 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-18T08:18:50.317 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:18:50.317 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:18:50.317 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-09-18T08:18:50.317 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:14:55.670832+0000", 2024-09-18T08:18:50.317 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-18T08:18:50.317 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:18:50.318 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:18:50.318 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-09-18T08:18:50.318 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:14:55.670833+0000", 2024-09-18T08:18:50.318 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 5.2600000000000002e-07 2024-09-18T08:18:50.318 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:18:50.318 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:18:50.318 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-09-18T08:18:50.318 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:14:55.670860+0000", 2024-09-18T08:18:50.318 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 2.7843000000000001e-05 2024-09-18T08:18:50.318 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:18:50.318 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:18:50.318 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-09-18T08:18:50.319 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:14:55.740135+0000", 2024-09-18T08:18:50.319 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.069274522000000005 2024-09-18T08:18:50.319 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:18:50.319 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:18:50.319 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-09-18T08:18:50.319 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:14:55.740144+0000", 2024-09-18T08:18:50.319 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 9.4709999999999996e-06 2024-09-18T08:18:50.319 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:18:50.320 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:18:50.320 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-18T08:18:50.320 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:14:55.740253+0000", 2024-09-18T08:18:50.320 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000109021 2024-09-18T08:18:50.320 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:18:50.320 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:18:50.320 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-09-18T08:18:50.320 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:14:55.740276+0000", 2024-09-18T08:18:50.320 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 2.2724000000000001e-05 2024-09-18T08:18:50.320 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:18:50.320 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:18:50.321 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "commit_sent", 2024-09-18T08:18:50.321 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:14:55.741226+0000", 2024-09-18T08:18:50.321 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00095007499999999999 2024-09-18T08:18:50.321 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:18:50.321 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:18:50.321 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-09-18T08:18:50.321 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:14:55.741271+0000", 2024-09-18T08:18:50.321 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4.4648999999999999e-05 2024-09-18T08:18:50.321 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-18T08:18:50.321 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-09-18T08:18:50.321 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-18T08:18:50.321 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:18:50.322 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:18:50.322 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_repop(client.15966.0:12238 10.f e320/311 10:f60a0ac6:::benchmark_data_smithi007_148834_object12237:head v 320'785, mlcod=319'783)", 2024-09-18T08:18:50.322 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-09-18T08:14:55.678110+0000", 2024-09-18T08:18:50.322 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 234.28658652799999, 2024-09-18T08:18:50.322 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.064983823999999996, 2024-09-18T08:18:50.322 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-09-18T08:18:50.322 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-09-18T08:18:50.322 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T08:18:50.322 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-09-18T08:18:50.322 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:18:50.322 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-09-18T08:18:50.322 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:14:55.678110+0000", 2024-09-18T08:18:50.322 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-18T08:18:50.323 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:18:50.323 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:18:50.323 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-09-18T08:18:50.323 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:14:55.678110+0000", 2024-09-18T08:18:50.323 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-18T08:18:50.323 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:18:50.323 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:18:50.323 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-09-18T08:18:50.323 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:14:55.678110+0000", 2024-09-18T08:18:50.323 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.6999999999999999e-07 2024-09-18T08:18:50.323 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:18:50.323 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:18:50.323 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-09-18T08:18:50.323 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:14:55.678173+0000", 2024-09-18T08:18:50.324 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 6.3546000000000004e-05 2024-09-18T08:18:50.324 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:18:50.324 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:18:50.324 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-09-18T08:18:50.324 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:14:55.739707+0000", 2024-09-18T08:18:50.324 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.061533746 2024-09-18T08:18:50.324 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:18:50.324 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:18:50.324 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-09-18T08:18:50.324 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:14:55.740687+0000", 2024-09-18T08:18:50.324 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00097965500000000007 2024-09-18T08:18:50.324 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:18:50.324 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:18:50.325 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-18T08:18:50.325 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:14:55.741323+0000", 2024-09-18T08:18:50.325 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00063652000000000001 2024-09-18T08:18:50.325 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:18:50.325 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:18:50.325 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-09-18T08:18:50.325 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:14:55.741342+0000", 2024-09-18T08:18:50.325 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.8774e-05 2024-09-18T08:18:50.325 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:18:50.325 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:18:50.325 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "commit_sent", 2024-09-18T08:18:50.325 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:14:55.743054+0000", 2024-09-18T08:18:50.325 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0017122890000000001 2024-09-18T08:18:50.325 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:18:50.326 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:18:50.326 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-09-18T08:18:50.326 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:14:55.743093+0000", 2024-09-18T08:18:50.326 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 3.9124e-05 2024-09-18T08:18:50.326 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-18T08:18:50.326 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-09-18T08:18:50.326 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-18T08:18:50.326 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:18:50.326 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:18:50.326 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_repop(client.15966.0:39969 10.b e327/311 10:d5cbced9:::benchmark_data_smithi007_148834_object39968:head v 327'2587, mlcod=327'2582)", 2024-09-18T08:18:50.326 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-09-18T08:15:15.163966+0000", 2024-09-18T08:18:50.326 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 214.80072980700001, 2024-09-18T08:18:50.326 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.043290203999999999, 2024-09-18T08:18:50.326 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-09-18T08:18:50.327 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-09-18T08:18:50.327 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T08:18:50.327 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-09-18T08:18:50.327 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:18:50.327 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-09-18T08:18:50.327 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:15:15.163966+0000", 2024-09-18T08:18:50.327 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-18T08:18:50.327 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:18:50.327 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:18:50.327 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-09-18T08:18:50.327 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:15:15.163966+0000", 2024-09-18T08:18:50.327 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-18T08:18:50.327 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:18:50.327 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:18:50.328 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-09-18T08:18:50.328 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:15:15.163967+0000", 2024-09-18T08:18:50.328 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4.5999999999999999e-07 2024-09-18T08:18:50.328 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:18:50.328 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:18:50.328 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-09-18T08:18:50.328 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:15:15.202633+0000", 2024-09-18T08:18:50.328 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.038666457000000001 2024-09-18T08:18:50.328 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:18:50.328 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:18:50.328 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-09-18T08:18:50.328 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:15:15.202641+0000", 2024-09-18T08:18:50.328 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 8.1780000000000003e-06 2024-09-18T08:18:50.328 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:18:50.329 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:18:50.329 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-09-18T08:18:50.329 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:15:15.202646+0000", 2024-09-18T08:18:50.329 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4.6800000000000001e-06 2024-09-18T08:18:50.329 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:18:50.329 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:18:50.329 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-18T08:18:50.329 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:15:15.202698+0000", 2024-09-18T08:18:50.329 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 5.198e-05 2024-09-18T08:18:50.329 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:18:50.329 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:18:50.329 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-09-18T08:18:50.329 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:15:15.202719+0000", 2024-09-18T08:18:50.329 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 2.0859000000000001e-05 2024-09-18T08:18:50.330 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:18:50.330 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:18:50.330 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "commit_sent", 2024-09-18T08:18:50.330 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:15:15.207225+0000", 2024-09-18T08:18:50.330 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0045059000000000002 2024-09-18T08:18:50.330 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:18:50.330 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:18:50.330 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-09-18T08:18:50.330 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:15:15.207257+0000", 2024-09-18T08:18:50.330 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 3.1690000000000003e-05 2024-09-18T08:18:50.330 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-18T08:18:50.330 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-09-18T08:18:50.331 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-18T08:18:50.331 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:18:50.331 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:18:50.331 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_repop(client.15966.0:107878 10.f e355/311 10:fc238020:::benchmark_data_smithi007_148834_object6158:head v 355'6839, mlcod=355'6837)", 2024-09-18T08:18:50.331 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-09-18T08:16:27.220525+0000", 2024-09-18T08:18:50.331 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 142.74417109699999, 2024-09-18T08:18:50.331 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.044386047999999997, 2024-09-18T08:18:50.332 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-09-18T08:18:50.332 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-09-18T08:18:50.332 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T08:18:50.332 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-09-18T08:18:50.332 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:18:50.332 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-09-18T08:18:50.332 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:16:27.220525+0000", 2024-09-18T08:18:50.332 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-18T08:18:50.332 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:18:50.332 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:18:50.332 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-09-18T08:18:50.332 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:16:27.220525+0000", 2024-09-18T08:18:50.332 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-18T08:18:50.332 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:18:50.333 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:18:50.333 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-09-18T08:18:50.333 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:16:27.220526+0000", 2024-09-18T08:18:50.333 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 7.6199999999999997e-07 2024-09-18T08:18:50.333 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:18:50.333 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:18:50.333 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-09-18T08:18:50.333 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:16:27.220533+0000", 2024-09-18T08:18:50.333 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 6.9280000000000004e-06 2024-09-18T08:18:50.333 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:18:50.333 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:18:50.333 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-09-18T08:18:50.333 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:16:27.220542+0000", 2024-09-18T08:18:50.333 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 9.1819999999999995e-06 2024-09-18T08:18:50.334 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:18:50.334 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:18:50.334 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-09-18T08:18:50.334 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:16:27.220549+0000", 2024-09-18T08:18:50.334 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 6.6669999999999999e-06 2024-09-18T08:18:50.334 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:18:50.334 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:18:50.334 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-18T08:18:50.334 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:16:27.220600+0000", 2024-09-18T08:18:50.334 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 5.1353000000000001e-05 2024-09-18T08:18:50.334 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:18:50.334 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:18:50.334 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-09-18T08:18:50.334 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:16:27.220618+0000", 2024-09-18T08:18:50.335 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.7642000000000002e-05 2024-09-18T08:18:50.335 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:18:50.335 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:18:50.335 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "commit_sent", 2024-09-18T08:18:50.335 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:16:27.222744+0000", 2024-09-18T08:18:50.335 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0021260519999999998 2024-09-18T08:18:50.335 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:18:50.335 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:18:50.335 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-09-18T08:18:50.335 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:16:27.264911+0000", 2024-09-18T08:18:50.335 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.042167462000000003 2024-09-18T08:18:50.335 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-18T08:18:50.336 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-09-18T08:18:50.336 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-18T08:18:50.336 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:18:50.336 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:18:50.336 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_repop(client.15966.0:107871 10.b e355/311 10:d19d8c53:::benchmark_data_smithi007_148834_object6151:head v 355'6844, mlcod=355'6842)", 2024-09-18T08:18:50.336 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-09-18T08:16:27.221250+0000", 2024-09-18T08:18:50.336 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 142.743446468, 2024-09-18T08:18:50.336 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.04515603, 2024-09-18T08:18:50.336 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-09-18T08:18:50.336 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-09-18T08:18:50.336 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T08:18:50.337 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-09-18T08:18:50.337 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:18:50.337 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-09-18T08:18:50.337 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:16:27.221250+0000", 2024-09-18T08:18:50.337 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-18T08:18:50.337 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:18:50.337 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:18:50.337 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-09-18T08:18:50.337 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:16:27.221250+0000", 2024-09-18T08:18:50.337 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-18T08:18:50.337 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:18:50.338 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:18:50.338 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-09-18T08:18:50.338 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:16:27.221250+0000", 2024-09-18T08:18:50.338 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 6.2600000000000002e-07 2024-09-18T08:18:50.338 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:18:50.338 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:18:50.338 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-09-18T08:18:50.338 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:16:27.221256+0000", 2024-09-18T08:18:50.338 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 5.1839999999999998e-06 2024-09-18T08:18:50.338 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:18:50.338 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:18:50.338 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-09-18T08:18:50.338 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:16:27.221263+0000", 2024-09-18T08:18:50.338 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 7.148e-06 2024-09-18T08:18:50.339 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:18:50.339 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:18:50.339 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-09-18T08:18:50.339 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:16:27.221267+0000", 2024-09-18T08:18:50.339 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4.2309999999999998e-06 2024-09-18T08:18:50.339 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:18:50.339 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:18:50.339 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-18T08:18:50.339 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:16:27.221307+0000", 2024-09-18T08:18:50.339 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 3.9848999999999997e-05 2024-09-18T08:18:50.339 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:18:50.339 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:18:50.339 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-09-18T08:18:50.339 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:16:27.221319+0000", 2024-09-18T08:18:50.340 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.2532e-05 2024-09-18T08:18:50.340 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:18:50.340 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:18:50.340 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "commit_sent", 2024-09-18T08:18:50.340 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:16:27.266370+0000", 2024-09-18T08:18:50.340 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.045050417000000002 2024-09-18T08:18:50.340 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:18:50.340 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:18:50.340 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-09-18T08:18:50.341 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:16:27.266406+0000", 2024-09-18T08:18:50.341 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 3.6043e-05 2024-09-18T08:18:50.341 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-18T08:18:50.341 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-09-18T08:18:50.341 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-18T08:18:50.341 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:18:50.341 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:18:50.341 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_repop(client.15966.0:107875 10.0 e355/349 10:0c1b5bdb:::benchmark_data_smithi007_148834_object6155:head v 355'6837, mlcod=355'6835)", 2024-09-18T08:18:50.341 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-09-18T08:16:27.221864+0000", 2024-09-18T08:18:50.341 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 142.74283202399999, 2024-09-18T08:18:50.341 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.043115863999999997, 2024-09-18T08:18:50.341 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-09-18T08:18:50.341 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-09-18T08:18:50.341 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T08:18:50.342 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-09-18T08:18:50.342 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:18:50.342 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-09-18T08:18:50.342 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:16:27.221864+0000", 2024-09-18T08:18:50.342 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-18T08:18:50.342 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:18:50.342 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:18:50.342 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-09-18T08:18:50.342 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:16:27.221864+0000", 2024-09-18T08:18:50.342 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-18T08:18:50.342 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:18:50.343 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:18:50.343 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-09-18T08:18:50.343 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:16:27.221865+0000", 2024-09-18T08:18:50.343 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 5.4199999999999996e-07 2024-09-18T08:18:50.343 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:18:50.343 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:18:50.343 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-09-18T08:18:50.343 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:16:27.221869+0000", 2024-09-18T08:18:50.343 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4.2220000000000004e-06 2024-09-18T08:18:50.343 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:18:50.343 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:18:50.343 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-09-18T08:18:50.344 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:16:27.221875+0000", 2024-09-18T08:18:50.344 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 6.4230000000000002e-06 2024-09-18T08:18:50.344 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:18:50.344 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:18:50.344 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-09-18T08:18:50.344 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:16:27.221879+0000", 2024-09-18T08:18:50.344 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 3.9020000000000001e-06 2024-09-18T08:18:50.344 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:18:50.344 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:18:50.344 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-18T08:18:50.344 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:16:27.221931+0000", 2024-09-18T08:18:50.344 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 5.1517000000000002e-05 2024-09-18T08:18:50.345 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:18:50.345 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:18:50.345 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-09-18T08:18:50.345 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:16:27.221946+0000", 2024-09-18T08:18:50.345 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.4795e-05 2024-09-18T08:18:50.345 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:18:50.345 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:18:50.345 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "commit_sent", 2024-09-18T08:18:50.345 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:16:27.264948+0000", 2024-09-18T08:18:50.345 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.043002315999999999 2024-09-18T08:18:50.345 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:18:50.345 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:18:50.345 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-09-18T08:18:50.345 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:16:27.264980+0000", 2024-09-18T08:18:50.346 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 3.2147000000000002e-05 2024-09-18T08:18:50.346 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-18T08:18:50.346 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-09-18T08:18:50.346 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-18T08:18:50.346 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:18:50.346 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:18:50.346 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_repop(client.15966.0:107873 10.b e355/311 10:d045f7ef:::benchmark_data_smithi007_148834_object6153:head v 355'6845, mlcod=355'6842)", 2024-09-18T08:18:50.346 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-09-18T08:16:27.222161+0000", 2024-09-18T08:18:50.346 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 142.74253536699999, 2024-09-18T08:18:50.346 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.045665511999999998, 2024-09-18T08:18:50.346 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-09-18T08:18:50.346 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-09-18T08:18:50.346 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T08:18:50.346 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-09-18T08:18:50.347 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:18:50.347 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-09-18T08:18:50.347 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:16:27.222161+0000", 2024-09-18T08:18:50.347 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-18T08:18:50.347 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:18:50.347 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:18:50.347 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-09-18T08:18:50.347 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:16:27.222161+0000", 2024-09-18T08:18:50.348 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-18T08:18:50.348 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:18:50.348 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:18:50.348 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-09-18T08:18:50.348 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:16:27.222161+0000", 2024-09-18T08:18:50.348 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 6.0699999999999997e-07 2024-09-18T08:18:50.348 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:18:50.348 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:18:50.348 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-09-18T08:18:50.348 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:16:27.222166+0000", 2024-09-18T08:18:50.348 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4.6179999999999997e-06 2024-09-18T08:18:50.348 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:18:50.349 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:18:50.349 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-09-18T08:18:50.349 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:16:27.222174+0000", 2024-09-18T08:18:50.349 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 8.0849999999999997e-06 2024-09-18T08:18:50.349 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:18:50.349 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:18:50.349 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-09-18T08:18:50.349 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:16:27.222180+0000", 2024-09-18T08:18:50.349 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 5.6470000000000003e-06 2024-09-18T08:18:50.349 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:18:50.349 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:18:50.349 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-18T08:18:50.350 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:16:27.222705+0000", 2024-09-18T08:18:50.350 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00052515500000000002 2024-09-18T08:18:50.350 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:18:50.350 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:18:50.350 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-09-18T08:18:50.350 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:16:27.222725+0000", 2024-09-18T08:18:50.350 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 2.0344999999999999e-05 2024-09-18T08:18:50.350 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:18:50.350 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:18:50.350 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "commit_sent", 2024-09-18T08:18:50.350 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:16:27.267787+0000", 2024-09-18T08:18:50.350 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.045061457999999999 2024-09-18T08:18:50.351 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:18:50.351 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:18:50.351 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-09-18T08:18:50.351 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:16:27.267826+0000", 2024-09-18T08:18:50.351 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 3.9597000000000001e-05 2024-09-18T08:18:50.351 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-18T08:18:50.351 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-09-18T08:18:50.351 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-18T08:18:50.351 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:18:50.351 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:18:50.351 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_repop(client.15966.0:107877 10.0 e355/349 10:0c59ffdb:::benchmark_data_smithi007_148834_object6157:head v 355'6838, mlcod=355'6835)", 2024-09-18T08:18:50.352 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-09-18T08:16:27.224340+0000", 2024-09-18T08:18:50.352 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 142.74035653999999, 2024-09-18T08:18:50.352 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.043959059000000002, 2024-09-18T08:18:50.352 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-09-18T08:18:50.352 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-09-18T08:18:50.352 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T08:18:50.353 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-09-18T08:18:50.353 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:18:50.353 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-09-18T08:18:50.353 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:16:27.224340+0000", 2024-09-18T08:18:50.353 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-18T08:18:50.353 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:18:50.353 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:18:50.353 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-09-18T08:18:50.354 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:16:27.224340+0000", 2024-09-18T08:18:50.354 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-18T08:18:50.354 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:18:50.354 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:18:50.354 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-09-18T08:18:50.354 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:16:27.224340+0000", 2024-09-18T08:18:50.354 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 6.2900000000000003e-07 2024-09-18T08:18:50.355 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:18:50.355 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:18:50.355 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-09-18T08:18:50.355 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:16:27.224345+0000", 2024-09-18T08:18:50.355 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 5.1209999999999998e-06 2024-09-18T08:18:50.355 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:18:50.355 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:18:50.355 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-09-18T08:18:50.356 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:16:27.264931+0000", 2024-09-18T08:18:50.356 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.040585860000000001 2024-09-18T08:18:50.356 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:18:50.356 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:18:50.357 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-09-18T08:18:50.357 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:16:27.264940+0000", 2024-09-18T08:18:50.357 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 9.0820000000000005e-06 2024-09-18T08:18:50.357 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:18:50.357 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:18:50.357 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-18T08:18:50.357 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:16:27.264998+0000", 2024-09-18T08:18:50.358 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 5.7964000000000003e-05 2024-09-18T08:18:50.358 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:18:50.358 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:18:50.358 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-09-18T08:18:50.358 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:16:27.265021+0000", 2024-09-18T08:18:50.358 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 2.2949999999999999e-05 2024-09-18T08:18:50.359 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:18:50.359 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:18:50.359 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "commit_sent", 2024-09-18T08:18:50.359 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:16:27.268261+0000", 2024-09-18T08:18:50.359 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.003240003 2024-09-18T08:18:50.359 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:18:50.359 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:18:50.360 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-09-18T08:18:50.360 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:16:27.268299+0000", 2024-09-18T08:18:50.360 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 3.7450000000000002e-05 2024-09-18T08:18:50.360 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-18T08:18:50.360 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-09-18T08:18:50.360 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-18T08:18:50.360 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:18:50.361 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:18:50.361 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_repop(client.15966.0:107876 10.c e355/349 10:3be0ad31:::benchmark_data_smithi007_148834_object6156:head v 355'6648, mlcod=355'6644)", 2024-09-18T08:18:50.361 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-09-18T08:16:27.228756+0000", 2024-09-18T08:18:50.361 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 142.735940271, 2024-09-18T08:18:50.361 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.039054169, 2024-09-18T08:18:50.361 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-09-18T08:18:50.362 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-09-18T08:18:50.362 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T08:18:50.362 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-09-18T08:18:50.362 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:18:50.362 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-09-18T08:18:50.362 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:16:27.228756+0000", 2024-09-18T08:18:50.362 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-18T08:18:50.363 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:18:50.363 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:18:50.363 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-09-18T08:18:50.363 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:16:27.228756+0000", 2024-09-18T08:18:50.363 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-18T08:18:50.363 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:18:50.363 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:18:50.364 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-09-18T08:18:50.364 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:16:27.228757+0000", 2024-09-18T08:18:50.364 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 7.7899999999999997e-07 2024-09-18T08:18:50.364 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:18:50.364 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:18:50.364 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-09-18T08:18:50.364 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:16:27.228766+0000", 2024-09-18T08:18:50.365 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 9.4870000000000008e-06 2024-09-18T08:18:50.365 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:18:50.365 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:18:50.365 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-09-18T08:18:50.365 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:16:27.264924+0000", 2024-09-18T08:18:50.365 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.036158269 2024-09-18T08:18:50.365 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:18:50.366 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:18:50.366 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-09-18T08:18:50.366 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:16:27.264936+0000", 2024-09-18T08:18:50.366 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.1449e-05 2024-09-18T08:18:50.366 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:18:50.366 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:18:50.367 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-18T08:18:50.367 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:16:27.264992+0000", 2024-09-18T08:18:50.367 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 5.5770000000000003e-05 2024-09-18T08:18:50.367 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:18:50.367 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:18:50.367 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-09-18T08:18:50.367 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:16:27.265013+0000", 2024-09-18T08:18:50.368 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 2.1396e-05 2024-09-18T08:18:50.368 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:18:50.368 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:18:50.368 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "commit_sent", 2024-09-18T08:18:50.368 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:16:27.267778+0000", 2024-09-18T08:18:50.368 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0027653999999999999 2024-09-18T08:18:50.369 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:18:50.369 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:18:50.369 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-09-18T08:18:50.369 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:16:27.267810+0000", 2024-09-18T08:18:50.369 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 3.1619000000000003e-05 2024-09-18T08:18:50.369 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-18T08:18:50.370 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-09-18T08:18:50.370 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-18T08:18:50.370 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:18:50.370 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:18:50.370 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_repop(client.26087.0:2173 11.d e406/401 11:b9d45e09:::benchmark_data_smithi007_172546_object2172:head v 406'140, mlcod=406'137)", 2024-09-18T08:18:50.370 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-09-18T08:18:35.762404+0000", 2024-09-18T08:18:50.371 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 14.202292061, 2024-09-18T08:18:50.371 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.065543183000000005, 2024-09-18T08:18:50.371 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-09-18T08:18:50.371 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-09-18T08:18:50.371 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T08:18:50.371 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-09-18T08:18:50.372 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:18:50.372 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-09-18T08:18:50.372 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:18:35.762404+0000", 2024-09-18T08:18:50.372 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-18T08:18:50.372 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:18:50.372 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:18:50.372 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-09-18T08:18:50.373 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:18:35.762404+0000", 2024-09-18T08:18:50.373 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-18T08:18:50.373 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:18:50.374 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:18:50.374 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-09-18T08:18:50.374 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:18:35.762405+0000", 2024-09-18T08:18:50.374 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 6.75e-07 2024-09-18T08:18:50.374 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:18:50.374 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:18:50.374 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-09-18T08:18:50.375 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:18:35.762463+0000", 2024-09-18T08:18:50.375 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 5.7889000000000001e-05 2024-09-18T08:18:50.375 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:18:50.375 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:18:50.375 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-09-18T08:18:50.375 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:18:35.762471+0000", 2024-09-18T08:18:50.375 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 8.0409999999999998e-06 2024-09-18T08:18:50.376 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:18:50.376 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:18:50.376 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-09-18T08:18:50.376 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:18:35.762475+0000", 2024-09-18T08:18:50.376 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4.6319999999999999e-06 2024-09-18T08:18:50.376 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:18:50.376 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:18:50.377 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-18T08:18:50.377 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:18:35.762509+0000", 2024-09-18T08:18:50.377 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 3.3590999999999997e-05 2024-09-18T08:18:50.377 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:18:50.377 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:18:50.377 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-09-18T08:18:50.377 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:18:35.827105+0000", 2024-09-18T08:18:50.377 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.064596555999999999 2024-09-18T08:18:50.377 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:18:50.377 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:18:50.378 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "commit_sent", 2024-09-18T08:18:50.378 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:18:35.827866+0000", 2024-09-18T08:18:50.378 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00076022299999999998 2024-09-18T08:18:50.378 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:18:50.378 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:18:50.378 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-09-18T08:18:50.378 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:18:35.827947+0000", 2024-09-18T08:18:50.378 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 8.1576000000000006e-05 2024-09-18T08:18:50.378 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-18T08:18:50.378 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-09-18T08:18:50.378 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-18T08:18:50.378 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:18:50.378 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:18:50.378 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_repop(client.26087.0:2174 11.d e406/401 11:b0e4e2b9:::benchmark_data_smithi007_172546_object2173:head v 406'141, mlcod=406'137)", 2024-09-18T08:18:50.379 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-09-18T08:18:35.762851+0000", 2024-09-18T08:18:50.379 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 14.201845584000001, 2024-09-18T08:18:50.379 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.065128555000000005, 2024-09-18T08:18:50.379 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-09-18T08:18:50.379 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-09-18T08:18:50.379 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T08:18:50.379 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-09-18T08:18:50.379 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:18:50.379 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-09-18T08:18:50.379 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:18:35.762851+0000", 2024-09-18T08:18:50.380 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-18T08:18:50.380 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:18:50.380 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:18:50.380 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-09-18T08:18:50.380 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:18:35.762851+0000", 2024-09-18T08:18:50.380 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-18T08:18:50.380 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:18:50.380 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:18:50.380 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-09-18T08:18:50.380 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:18:35.762851+0000", 2024-09-18T08:18:50.380 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4.0200000000000003e-07 2024-09-18T08:18:50.380 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:18:50.380 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:18:50.381 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-09-18T08:18:50.381 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:18:35.762923+0000", 2024-09-18T08:18:50.381 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 7.2439000000000002e-05 2024-09-18T08:18:50.381 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:18:50.381 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:18:50.381 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-09-18T08:18:50.381 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:18:35.827101+0000", 2024-09-18T08:18:50.381 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.064177925999999996 2024-09-18T08:18:50.381 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:18:50.381 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:18:50.381 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-09-18T08:18:50.381 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:18:35.827110+0000", 2024-09-18T08:18:50.382 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 8.4819999999999994e-06 2024-09-18T08:18:50.382 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:18:50.382 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:18:50.382 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-18T08:18:50.382 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:18:35.827421+0000", 2024-09-18T08:18:50.382 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00031155899999999999 2024-09-18T08:18:50.382 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:18:50.382 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:18:50.382 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-09-18T08:18:50.382 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:18:35.827440+0000", 2024-09-18T08:18:50.382 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.8708000000000001e-05 2024-09-18T08:18:50.382 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:18:50.383 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:18:50.383 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "commit_sent", 2024-09-18T08:18:50.383 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:18:35.827955+0000", 2024-09-18T08:18:50.383 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00051529099999999997 2024-09-18T08:18:50.383 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:18:50.383 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:18:50.383 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-09-18T08:18:50.383 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:18:35.827979+0000", 2024-09-18T08:18:50.383 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 2.3748e-05 2024-09-18T08:18:50.383 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-18T08:18:50.383 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-09-18T08:18:50.383 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-18T08:18:50.383 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:18:50.383 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:18:50.384 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_repop(client.26087.0:2177 11.1 e406/401 11:80039ec8:::benchmark_data_smithi007_172546_object2176:head v 406'119, mlcod=406'117)", 2024-09-18T08:18:50.384 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-09-18T08:18:35.763185+0000", 2024-09-18T08:18:50.384 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 14.201511392, 2024-09-18T08:18:50.384 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.064984449, 2024-09-18T08:18:50.384 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-09-18T08:18:50.384 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-09-18T08:18:50.384 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T08:18:50.384 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-09-18T08:18:50.384 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:18:50.384 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-09-18T08:18:50.384 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:18:35.763185+0000", 2024-09-18T08:18:50.384 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-18T08:18:50.384 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:18:50.385 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:18:50.385 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-09-18T08:18:50.385 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:18:35.763185+0000", 2024-09-18T08:18:50.385 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-18T08:18:50.385 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:18:50.385 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:18:50.385 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-09-18T08:18:50.385 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:18:35.763185+0000", 2024-09-18T08:18:50.385 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 5.3799999999999997e-07 2024-09-18T08:18:50.385 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:18:50.386 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:18:50.386 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-09-18T08:18:50.386 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:18:35.763228+0000", 2024-09-18T08:18:50.386 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4.2249999999999997e-05 2024-09-18T08:18:50.386 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:18:50.386 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:18:50.386 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-09-18T08:18:50.386 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:18:35.827100+0000", 2024-09-18T08:18:50.386 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.063872218999999994 2024-09-18T08:18:50.386 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:18:50.386 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:18:50.386 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-09-18T08:18:50.386 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:18:35.827110+0000", 2024-09-18T08:18:50.386 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.0594000000000001e-05 2024-09-18T08:18:50.387 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:18:50.387 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:18:50.387 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-18T08:18:50.387 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:18:35.827538+0000", 2024-09-18T08:18:50.387 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00042732399999999998 2024-09-18T08:18:50.387 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:18:50.387 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:18:50.387 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-09-18T08:18:50.387 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:18:35.827558+0000", 2024-09-18T08:18:50.388 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 2.0299e-05 2024-09-18T08:18:50.388 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:18:50.388 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:18:50.388 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "commit_sent", 2024-09-18T08:18:50.388 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:18:35.828135+0000", 2024-09-18T08:18:50.388 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00057689799999999995 2024-09-18T08:18:50.388 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:18:50.388 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:18:50.388 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-09-18T08:18:50.388 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:18:35.828169+0000", 2024-09-18T08:18:50.388 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 3.4326999999999998e-05 2024-09-18T08:18:50.388 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-18T08:18:50.388 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-09-18T08:18:50.389 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-18T08:18:50.389 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:18:50.389 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:18:50.389 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_repop(client.26087.0:2370 11.3 e406/401 11:cc7a7da7:::benchmark_data_smithi007_172546_object2369:head v 406'156, mlcod=406'154)", 2024-09-18T08:18:50.389 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-09-18T08:18:36.146109+0000", 2024-09-18T08:18:50.389 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 13.81858714, 2024-09-18T08:18:50.389 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.17769138600000001, 2024-09-18T08:18:50.389 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-09-18T08:18:50.389 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-09-18T08:18:50.389 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T08:18:50.390 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-09-18T08:18:50.390 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:18:50.390 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-09-18T08:18:50.390 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:18:36.146109+0000", 2024-09-18T08:18:50.390 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-18T08:18:50.390 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:18:50.390 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:18:50.390 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-09-18T08:18:50.390 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:18:36.146109+0000", 2024-09-18T08:18:50.390 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-18T08:18:50.390 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:18:50.390 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:18:50.390 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-09-18T08:18:50.391 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:18:36.146109+0000", 2024-09-18T08:18:50.391 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 3.9099999999999999e-07 2024-09-18T08:18:50.391 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:18:50.391 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:18:50.391 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-09-18T08:18:50.391 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:18:36.146133+0000", 2024-09-18T08:18:50.391 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 2.3978e-05 2024-09-18T08:18:50.391 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:18:50.391 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:18:50.391 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-09-18T08:18:50.391 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:18:36.146141+0000", 2024-09-18T08:18:50.391 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 7.4490000000000002e-06 2024-09-18T08:18:50.391 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:18:50.391 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:18:50.392 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-09-18T08:18:50.392 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:18:36.146160+0000", 2024-09-18T08:18:50.392 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.9395e-05 2024-09-18T08:18:50.392 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:18:50.392 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:18:50.392 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-18T08:18:50.392 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:18:36.146218+0000", 2024-09-18T08:18:50.392 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 5.8010999999999997e-05 2024-09-18T08:18:50.392 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:18:50.392 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:18:50.392 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-09-18T08:18:50.393 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:18:36.146238+0000", 2024-09-18T08:18:50.393 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.9636999999999999e-05 2024-09-18T08:18:50.393 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:18:50.393 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:18:50.393 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "commit_sent", 2024-09-18T08:18:50.393 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:18:36.323745+0000", 2024-09-18T08:18:50.393 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.177506784 2024-09-18T08:18:50.393 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:18:50.393 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:18:50.393 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-09-18T08:18:50.393 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:18:36.323800+0000", 2024-09-18T08:18:50.393 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 5.5741e-05 2024-09-18T08:18:50.393 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-18T08:18:50.393 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-09-18T08:18:50.394 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-18T08:18:50.394 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:18:50.394 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:18:50.394 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_repop(client.26087.0:2371 11.4 e406/401 11:2d817d2b:::benchmark_data_smithi007_172546_object2370:head v 406'158, mlcod=406'155)", 2024-09-18T08:18:50.394 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-09-18T08:18:36.146584+0000", 2024-09-18T08:18:50.394 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 13.818112097, 2024-09-18T08:18:50.394 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.178099175, 2024-09-18T08:18:50.394 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-09-18T08:18:50.394 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-09-18T08:18:50.395 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T08:18:50.395 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-09-18T08:18:50.395 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:18:50.395 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-09-18T08:18:50.395 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:18:36.146584+0000", 2024-09-18T08:18:50.395 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-18T08:18:50.395 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:18:50.396 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:18:50.396 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-09-18T08:18:50.396 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:18:36.146584+0000", 2024-09-18T08:18:50.396 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-18T08:18:50.396 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:18:50.396 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:18:50.397 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-09-18T08:18:50.397 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:18:36.146585+0000", 2024-09-18T08:18:50.397 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 6.2099999999999996e-07 2024-09-18T08:18:50.397 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:18:50.397 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:18:50.397 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-09-18T08:18:50.397 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:18:36.146606+0000", 2024-09-18T08:18:50.398 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 2.1342999999999999e-05 2024-09-18T08:18:50.398 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:18:50.398 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:18:50.398 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-09-18T08:18:50.398 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:18:36.146613+0000", 2024-09-18T08:18:50.398 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 7.1999999999999997e-06 2024-09-18T08:18:50.398 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:18:50.398 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:18:50.398 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-09-18T08:18:50.398 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:18:36.146618+0000", 2024-09-18T08:18:50.398 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4.8409999999999999e-06 2024-09-18T08:18:50.398 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:18:50.398 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:18:50.399 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-18T08:18:50.399 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:18:36.323730+0000", 2024-09-18T08:18:50.399 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.177112032 2024-09-18T08:18:50.399 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:18:50.399 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:18:50.399 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-09-18T08:18:50.399 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:18:36.323764+0000", 2024-09-18T08:18:50.399 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 3.4230999999999998e-05 2024-09-18T08:18:50.399 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:18:50.399 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:18:50.399 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "commit_sent", 2024-09-18T08:18:50.399 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:18:36.324645+0000", 2024-09-18T08:18:50.400 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00088089799999999997 2024-09-18T08:18:50.400 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:18:50.400 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:18:50.400 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-09-18T08:18:50.400 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:18:36.324683+0000", 2024-09-18T08:18:50.400 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 3.8009000000000001e-05 2024-09-18T08:18:50.400 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-18T08:18:50.400 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-09-18T08:18:50.400 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-18T08:18:50.400 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:18:50.400 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:18:50.400 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_repop(client.26087.0:2372 11.4 e406/401 11:2d6b5b97:::benchmark_data_smithi007_172546_object2371:head v 406'159, mlcod=406'155)", 2024-09-18T08:18:50.400 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-09-18T08:18:36.147079+0000", 2024-09-18T08:18:50.400 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 13.817617387, 2024-09-18T08:18:50.401 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.17764386700000001, 2024-09-18T08:18:50.401 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-09-18T08:18:50.401 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-09-18T08:18:50.401 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T08:18:50.401 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-09-18T08:18:50.401 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:18:50.401 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-09-18T08:18:50.401 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:18:36.147079+0000", 2024-09-18T08:18:50.401 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-18T08:18:50.401 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:18:50.401 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:18:50.401 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-09-18T08:18:50.401 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:18:36.147079+0000", 2024-09-18T08:18:50.402 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-18T08:18:50.402 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:18:50.402 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:18:50.402 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-09-18T08:18:50.402 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:18:36.147080+0000", 2024-09-18T08:18:50.402 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 7.6899999999999996e-07 2024-09-18T08:18:50.402 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:18:50.402 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:18:50.402 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-09-18T08:18:50.402 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:18:36.147105+0000", 2024-09-18T08:18:50.402 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 2.5347e-05 2024-09-18T08:18:50.402 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:18:50.402 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:18:50.402 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-09-18T08:18:50.403 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:18:36.323717+0000", 2024-09-18T08:18:50.403 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.17661196900000001 2024-09-18T08:18:50.403 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:18:50.403 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:18:50.403 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-09-18T08:18:50.403 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:18:36.323732+0000", 2024-09-18T08:18:50.403 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.4714e-05 2024-09-18T08:18:50.403 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:18:50.403 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:18:50.403 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-18T08:18:50.403 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:18:36.324186+0000", 2024-09-18T08:18:50.403 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00045396 2024-09-18T08:18:50.403 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:18:50.403 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:18:50.404 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-09-18T08:18:50.404 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:18:36.324205+0000", 2024-09-18T08:18:50.404 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.9233999999999999e-05 2024-09-18T08:18:50.404 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:18:50.404 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:18:50.404 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "commit_sent", 2024-09-18T08:18:50.404 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:18:36.324686+0000", 2024-09-18T08:18:50.404 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00048093799999999998 2024-09-18T08:18:50.404 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:18:50.404 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:18:50.404 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-09-18T08:18:50.404 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:18:36.324723+0000", 2024-09-18T08:18:50.404 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 3.6936e-05 2024-09-18T08:18:50.404 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-18T08:18:50.405 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-09-18T08:18:50.405 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-18T08:18:50.405 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:18:50.405 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:18:50.405 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_repop(client.26087.0:2376 11.5 e406/401 11:a232a325:::benchmark_data_smithi007_172546_object2375:head v 406'147, mlcod=406'145)", 2024-09-18T08:18:50.405 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-09-18T08:18:36.147233+0000", 2024-09-18T08:18:50.405 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 13.817463483999999, 2024-09-18T08:18:50.405 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.17764028800000001, 2024-09-18T08:18:50.405 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-09-18T08:18:50.405 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-09-18T08:18:50.405 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T08:18:50.405 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-09-18T08:18:50.405 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:18:50.405 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-09-18T08:18:50.406 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:18:36.147233+0000", 2024-09-18T08:18:50.406 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-18T08:18:50.406 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:18:50.406 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:18:50.406 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-09-18T08:18:50.406 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:18:36.147233+0000", 2024-09-18T08:18:50.406 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-18T08:18:50.406 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:18:50.406 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:18:50.406 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-09-18T08:18:50.406 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:18:36.147234+0000", 2024-09-18T08:18:50.406 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 8.9500000000000001e-07 2024-09-18T08:18:50.406 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:18:50.407 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:18:50.407 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-09-18T08:18:50.407 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:18:36.147262+0000", 2024-09-18T08:18:50.407 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 2.8813e-05 2024-09-18T08:18:50.407 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:18:50.407 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:18:50.407 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-09-18T08:18:50.407 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:18:36.323707+0000", 2024-09-18T08:18:50.407 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.17644468899999999 2024-09-18T08:18:50.407 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:18:50.407 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:18:50.407 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-09-18T08:18:50.407 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:18:36.323724+0000", 2024-09-18T08:18:50.407 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.6971999999999999e-05 2024-09-18T08:18:50.408 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:18:50.408 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:18:50.408 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-18T08:18:50.408 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:18:36.323791+0000", 2024-09-18T08:18:50.408 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 6.6975000000000002e-05 2024-09-18T08:18:50.408 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:18:50.408 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:18:50.408 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-09-18T08:18:50.408 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:18:36.323811+0000", 2024-09-18T08:18:50.408 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.9998000000000001e-05 2024-09-18T08:18:50.408 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:18:50.408 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:18:50.408 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "commit_sent", 2024-09-18T08:18:50.409 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:18:36.324827+0000", 2024-09-18T08:18:50.409 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.001015845 2024-09-18T08:18:50.409 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:18:50.409 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:18:50.409 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-09-18T08:18:50.409 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:18:36.324873+0000", 2024-09-18T08:18:50.409 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4.6100999999999998e-05 2024-09-18T08:18:50.409 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-18T08:18:50.409 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-09-18T08:18:50.409 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-18T08:18:50.409 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:18:50.409 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:18:50.409 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_repop(client.26087.0:2374 11.a e406/401 11:567bc3ae:::benchmark_data_smithi007_172546_object2373:head v 406'152, mlcod=406'149)", 2024-09-18T08:18:50.409 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-09-18T08:18:36.148323+0000", 2024-09-18T08:18:50.410 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 13.816372953, 2024-09-18T08:18:50.410 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.176389139, 2024-09-18T08:18:50.410 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-09-18T08:18:50.410 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-09-18T08:18:50.410 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T08:18:50.410 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-09-18T08:18:50.410 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:18:50.410 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-09-18T08:18:50.410 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:18:36.148323+0000", 2024-09-18T08:18:50.410 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-18T08:18:50.410 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:18:50.410 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:18:50.410 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-09-18T08:18:50.410 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:18:36.148323+0000", 2024-09-18T08:18:50.411 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-18T08:18:50.411 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:18:50.411 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:18:50.411 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-09-18T08:18:50.411 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:18:36.148324+0000", 2024-09-18T08:18:50.411 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 6.99e-07 2024-09-18T08:18:50.411 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:18:50.411 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:18:50.411 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-09-18T08:18:50.411 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:18:36.148410+0000", 2024-09-18T08:18:50.411 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 8.6580000000000001e-05 2024-09-18T08:18:50.411 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:18:50.411 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:18:50.411 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-09-18T08:18:50.412 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:18:36.323736+0000", 2024-09-18T08:18:50.412 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.17532525199999999 2024-09-18T08:18:50.412 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:18:50.412 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:18:50.412 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-09-18T08:18:50.412 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:18:36.323746+0000", 2024-09-18T08:18:50.412 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 9.9820000000000003e-06 2024-09-18T08:18:50.412 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:18:50.412 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:18:50.412 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-18T08:18:50.412 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:18:36.323808+0000", 2024-09-18T08:18:50.412 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 6.1853999999999994e-05 2024-09-18T08:18:50.412 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:18:50.413 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:18:50.413 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-09-18T08:18:50.413 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:18:36.323830+0000", 2024-09-18T08:18:50.413 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 2.2240999999999999e-05 2024-09-18T08:18:50.413 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:18:50.413 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:18:50.413 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "commit_sent", 2024-09-18T08:18:50.413 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:18:36.324673+0000", 2024-09-18T08:18:50.413 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00084346400000000004 2024-09-18T08:18:50.413 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:18:50.413 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:18:50.413 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-09-18T08:18:50.413 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:18:36.324712+0000", 2024-09-18T08:18:50.414 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 3.9066999999999997e-05 2024-09-18T08:18:50.414 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-18T08:18:50.414 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-09-18T08:18:50.414 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-18T08:18:50.414 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:18:50.414 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:18:50.414 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_repop(client.26087.0:24018 11.8 e408/401 11:173b91d1:::benchmark_data_smithi007_172546_object24017:head v 408'1484, mlcod=408'1481)", 2024-09-18T08:18:50.414 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-09-18T08:18:46.133021+0000", 2024-09-18T08:18:50.414 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 3.8316752350000001, 2024-09-18T08:18:50.414 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.026108600999999999, 2024-09-18T08:18:50.414 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-09-18T08:18:50.414 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-09-18T08:18:50.414 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T08:18:50.414 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-09-18T08:18:50.415 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:18:50.415 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-09-18T08:18:50.415 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:18:46.133021+0000", 2024-09-18T08:18:50.415 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-18T08:18:50.415 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:18:50.415 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:18:50.415 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-09-18T08:18:50.415 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:18:46.133021+0000", 2024-09-18T08:18:50.415 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-18T08:18:50.415 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:18:50.415 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:18:50.415 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-09-18T08:18:50.415 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:18:46.133022+0000", 2024-09-18T08:18:50.415 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 6.4300000000000003e-07 2024-09-18T08:18:50.416 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:18:50.416 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:18:50.416 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-09-18T08:18:50.416 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:18:46.133090+0000", 2024-09-18T08:18:50.416 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 6.8764999999999996e-05 2024-09-18T08:18:50.416 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:18:50.416 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:18:50.416 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-09-18T08:18:50.416 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:18:46.133096+0000", 2024-09-18T08:18:50.416 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 5.9309999999999996e-06 2024-09-18T08:18:50.416 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:18:50.416 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:18:50.416 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-09-18T08:18:50.417 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:18:46.133100+0000", 2024-09-18T08:18:50.417 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 3.8199999999999998e-06 2024-09-18T08:18:50.417 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:18:50.417 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:18:50.417 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-18T08:18:50.417 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:18:46.157714+0000", 2024-09-18T08:18:50.417 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.024613935 2024-09-18T08:18:50.417 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:18:50.417 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:18:50.417 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-09-18T08:18:50.417 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:18:46.157732+0000", 2024-09-18T08:18:50.417 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.7992999999999999e-05 2024-09-18T08:18:50.417 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:18:50.417 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:18:50.418 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "commit_sent", 2024-09-18T08:18:50.418 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:18:46.159092+0000", 2024-09-18T08:18:50.418 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.001360308 2024-09-18T08:18:50.418 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:18:50.418 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:18:50.418 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-09-18T08:18:50.418 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:18:46.159130+0000", 2024-09-18T08:18:50.418 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 3.7206000000000003e-05 2024-09-18T08:18:50.418 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-18T08:18:50.418 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-09-18T08:18:50.418 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-18T08:18:50.418 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-18T08:18:50.418 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-09-18T08:18:50.418 INFO:teuthology.orchestra.run.smithi050.stdout:} 2024-09-18T08:18:50.419 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3 2024-09-18T08:18:50.459 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:18:50.459 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:18:50.459 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3' 2024-09-18T08:18:50.560 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4 2024-09-18T08:18:50.590 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:18:50.590 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:18:50.591 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4' 2024-09-18T08:18:50.692 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2 2024-09-18T08:18:50.714 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:18:50.714 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:18:50.715 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2' 2024-09-18T08:18:50.816 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2 2024-09-18T08:18:50.855 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:18:50.855 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:18:50.856 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2' 2024-09-18T08:18:50.956 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1 2024-09-18T08:18:50.979 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:18:50 smithi007 ceph-mon[29459]: pgmap v804: 158 pgs: 158 active+clean; 1.7 GiB data, 8.8 GiB used, 617 GiB / 626 GiB avail; 1021 B/s rd, 121 MiB/s wr, 1.93k op/s 2024-09-18T08:18:50.980 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:18:50 smithi007 ceph-mon[29459]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-18T08:18:50.980 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:18:50 smithi007 ceph-mon[39249]: pgmap v804: 158 pgs: 158 active+clean; 1.7 GiB data, 8.8 GiB used, 617 GiB / 626 GiB avail; 1021 B/s rd, 121 MiB/s wr, 1.93k op/s 2024-09-18T08:18:50.980 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:18:50 smithi007 ceph-mon[39249]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-18T08:18:50.998 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:18:50.999 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:18:50.999 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1' 2024-09-18T08:18:51.034 INFO:tasks.radosbench.radosbench.0.smithi007.stdout: 16 16 31264 31248 122.029 89.3125 0.0093291 0.00818884 2024-09-18T08:18:51.100 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4 2024-09-18T08:18:51.130 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:18:51.131 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:18:51.131 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4' 2024-09-18T08:18:51.155 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:18:50 smithi050 ceph-mon[32371]: pgmap v804: 158 pgs: 158 active+clean; 1.7 GiB data, 8.8 GiB used, 617 GiB / 626 GiB avail; 1021 B/s rd, 121 MiB/s wr, 1.93k op/s 2024-09-18T08:18:51.155 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:18:50 smithi050 ceph-mon[32371]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-18T08:18:51.232 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7 2024-09-18T08:18:51.256 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:18:51.256 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:18:51.257 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7' 2024-09-18T08:18:51.357 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3 2024-09-18T08:18:51.403 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:18:51.403 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:18:51.404 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3' 2024-09-18T08:18:51.505 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7 2024-09-18T08:18:51.537 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:18:51.538 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:18:51.538 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7' 2024-09-18T08:18:51.639 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3 2024-09-18T08:18:51.678 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:18:51.679 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:18:51.679 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3' 2024-09-18T08:18:51.780 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6 2024-09-18T08:18:51.809 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:18:51.810 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:18:51.810 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6' 2024-09-18T08:18:51.911 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5 2024-09-18T08:18:51.940 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:18:51.941 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:18:51.941 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5' 2024-09-18T08:18:52.034 INFO:tasks.radosbench.radosbench.0.smithi007.stdout: 17 15 32873 32858 120.77 100.625 0.0105894 0.00827435 2024-09-18T08:18:52.042 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6 2024-09-18T08:18:52.073 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:18:52.074 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:18:52.074 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6' 2024-09-18T08:18:52.151 DEBUG:teuthology.orchestra.run.smithi050:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid 779349ca-7593-11ef-bceb-c7b262605968 -- ceph daemon osd.7 dump_ops_in_flight 2024-09-18T08:18:52.175 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5 2024-09-18T08:18:52.232 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:18:52.233 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:18:52.233 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5' 2024-09-18T08:18:52.334 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7 2024-09-18T08:18:52.356 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:18:52.357 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:18:52.357 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7' 2024-09-18T08:18:52.458 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2 2024-09-18T08:18:52.473 INFO:teuthology.orchestra.run.smithi050.stderr:Inferring config /var/lib/ceph/779349ca-7593-11ef-bceb-c7b262605968/mon.b/config 2024-09-18T08:18:52.501 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:18:52.501 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:18:52.502 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2' 2024-09-18T08:18:52.603 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6 2024-09-18T08:18:52.633 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:18:52.634 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:18:52.634 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6' 2024-09-18T08:18:52.730 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:18:52 smithi007 ceph-mon[29459]: pgmap v805: 158 pgs: 158 active+clean; 1.7 GiB data, 8.8 GiB used, 617 GiB / 626 GiB avail; 852 B/s rd, 101 MiB/s wr, 1.61k op/s 2024-09-18T08:18:52.730 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:18:52 smithi007 ceph-mon[39249]: pgmap v805: 158 pgs: 158 active+clean; 1.7 GiB data, 8.8 GiB used, 617 GiB / 626 GiB avail; 852 B/s rd, 101 MiB/s wr, 1.61k op/s 2024-09-18T08:18:52.734 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2 2024-09-18T08:18:52.773 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:18:52.774 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:18:52.774 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2' 2024-09-18T08:18:52.875 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4 2024-09-18T08:18:52.898 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:18:52.898 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:18:52.899 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4' 2024-09-18T08:18:52.905 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:18:52 smithi050 ceph-mon[32371]: pgmap v805: 158 pgs: 158 active+clean; 1.7 GiB data, 8.8 GiB used, 617 GiB / 626 GiB avail; 852 B/s rd, 101 MiB/s wr, 1.61k op/s 2024-09-18T08:18:52.999 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3 2024-09-18T08:18:53.020 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2024-09-18T08:18:53.021 INFO:teuthology.orchestra.run.smithi050.stdout: "ops": [ 2024-09-18T08:18:53.021 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:18:53.021 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.26087.0:34020 11.3 11:cfbccb88:::benchmark_data_smithi007_172546_object34019:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536 in=65536b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e408)", 2024-09-18T08:18:53.021 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-09-18T08:18:52.951931+0000", 2024-09-18T08:18:53.021 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 0.067643354000000003, 2024-09-18T08:18:53.021 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.067670329000000001, 2024-09-18T08:18:53.021 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-09-18T08:18:53.021 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-09-18T08:18:53.021 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "waiting for sub ops", 2024-09-18T08:18:53.021 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2024-09-18T08:18:53.021 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.26087", 2024-09-18T08:18:53.021 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "172.21.15.7:0/2752273185", 2024-09-18T08:18:53.021 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 34020 2024-09-18T08:18:53.022 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:18:53.022 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-09-18T08:18:53.022 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:18:53.022 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-09-18T08:18:53.022 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:18:52.951931+0000", 2024-09-18T08:18:53.022 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-18T08:18:53.022 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:18:53.022 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:18:53.022 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-09-18T08:18:53.022 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:18:52.951931+0000", 2024-09-18T08:18:53.022 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-18T08:18:53.022 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:18:53.022 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:18:53.023 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-09-18T08:18:53.023 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:18:52.951932+0000", 2024-09-18T08:18:53.023 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.147e-06 2024-09-18T08:18:53.023 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:18:53.023 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:18:53.023 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-09-18T08:18:53.023 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:18:52.951971+0000", 2024-09-18T08:18:53.023 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 3.8837000000000001e-05 2024-09-18T08:18:53.023 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:18:53.023 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:18:53.023 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-09-18T08:18:53.023 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:18:52.951984+0000", 2024-09-18T08:18:53.023 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.2765000000000001e-05 2024-09-18T08:18:53.023 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:18:53.024 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:18:53.024 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-09-18T08:18:53.024 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:18:52.951992+0000", 2024-09-18T08:18:53.024 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 7.9580000000000007e-06 2024-09-18T08:18:53.024 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:18:53.024 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:18:53.024 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-18T08:18:53.024 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:18:52.952038+0000", 2024-09-18T08:18:53.024 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4.5763999999999998e-05 2024-09-18T08:18:53.024 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:18:53.024 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:18:53.024 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-09-18T08:18:53.024 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:18:52.952186+0000", 2024-09-18T08:18:53.024 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00014854899999999999 2024-09-18T08:18:53.025 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:18:53.025 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:18:53.025 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "waiting for subops from 1,6", 2024-09-18T08:18:53.025 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:18:52.952320+0000", 2024-09-18T08:18:53.025 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00013358200000000001 2024-09-18T08:18:53.025 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:18:53.025 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:18:53.025 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "op_commit", 2024-09-18T08:18:53.025 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:18:52.953581+0000", 2024-09-18T08:18:53.025 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0012614880000000001 2024-09-18T08:18:53.025 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:18:53.025 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:18:53.025 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:18:53.025 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:18:52.953894+0000", 2024-09-18T08:18:53.026 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00031271400000000001 2024-09-18T08:18:53.026 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-18T08:18:53.026 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-09-18T08:18:53.026 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-18T08:18:53.026 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-18T08:18:53.026 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2024-09-18T08:18:53.026 INFO:teuthology.orchestra.run.smithi050.stdout: "num_ops": 1 2024-09-18T08:18:53.026 INFO:teuthology.orchestra.run.smithi050.stdout:} 2024-09-18T08:18:53.027 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:18:53.027 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:18:53.027 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3' 2024-09-18T08:18:53.034 INFO:tasks.radosbench.radosbench.0.smithi007.stdout: 18 16 34027 34011 118.064 72.0625 0.00266061 0.00842736 2024-09-18T08:18:53.129 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4 2024-09-18T08:18:53.151 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:18:53.151 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:18:53.152 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4' 2024-09-18T08:18:53.253 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7 2024-09-18T08:18:53.282 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:18:53.283 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:18:53.283 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7' 2024-09-18T08:18:53.384 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6 2024-09-18T08:18:53.416 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:18:53.417 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:18:53.417 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6' 2024-09-18T08:18:53.518 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7 2024-09-18T08:18:53.544 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:18:53.544 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:18:53.545 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7' 2024-09-18T08:18:53.646 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4 2024-09-18T08:18:53.675 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:18:53.676 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:18:53.676 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4' 2024-09-18T08:18:53.777 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2 2024-09-18T08:18:53.818 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:18:53.818 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:18:53.819 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2' 2024-09-18T08:18:53.882 DEBUG:teuthology.orchestra.run.smithi050:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid 779349ca-7593-11ef-bceb-c7b262605968 -- ceph daemon osd.7 dump_blocked_ops 2024-09-18T08:18:53.919 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5 2024-09-18T08:18:53.955 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:18:53.956 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:18:53.956 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5' 2024-09-18T08:18:54.034 INFO:tasks.radosbench.radosbench.0.smithi007.stdout: 19 16 35378 35362 116.294 84.4375 0.00730923 0.00859216 2024-09-18T08:18:54.057 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5 2024-09-18T08:18:54.089 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:18:54.090 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:18:54.090 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5' 2024-09-18T08:18:54.191 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6 2024-09-18T08:18:54.226 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:18:54.226 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:18:54.227 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6' 2024-09-18T08:18:54.233 INFO:teuthology.orchestra.run.smithi050.stderr:Inferring config /var/lib/ceph/779349ca-7593-11ef-bceb-c7b262605968/mon.b/config 2024-09-18T08:18:54.328 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2 2024-09-18T08:18:54.363 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:18:54.364 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:18:54.364 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2' 2024-09-18T08:18:54.465 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5 2024-09-18T08:18:54.494 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:18:54.495 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:18:54.495 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5' 2024-09-18T08:18:54.596 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1 2024-09-18T08:18:54.638 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:18:54.639 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:18:54.639 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1' 2024-09-18T08:18:54.716 INFO:teuthology.orchestra.run.smithi007.stderr:noscrub is unset 2024-09-18T08:18:54.740 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4 2024-09-18T08:18:54.778 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:18:54.778 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:18:54.779 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4' 2024-09-18T08:18:54.829 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2024-09-18T08:18:54.829 INFO:teuthology.orchestra.run.smithi050.stdout: "ops": [], 2024-09-18T08:18:54.830 INFO:teuthology.orchestra.run.smithi050.stdout: "complaint_time": 30, 2024-09-18T08:18:54.830 INFO:teuthology.orchestra.run.smithi050.stdout: "num_blocked_ops": 0 2024-09-18T08:18:54.830 INFO:teuthology.orchestra.run.smithi050.stdout:} 2024-09-18T08:18:54.880 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4 2024-09-18T08:18:54.911 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:18:54.911 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:18:54.912 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4' 2024-09-18T08:18:54.980 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:18:54 smithi007 ceph-mon[29459]: pgmap v806: 158 pgs: 158 active+clean; 1.8 GiB data, 9.4 GiB used, 616 GiB / 626 GiB avail; 1.2 KiB/s rd, 113 MiB/s wr, 1.80k op/s 2024-09-18T08:18:54.980 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:18:54 smithi007 ceph-mon[29459]: from='client.? 172.21.15.7:0/2090504115' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-18T08:18:54.980 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:18:54 smithi007 ceph-mon[29459]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-18T08:18:54.980 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:18:54 smithi007 ceph-mon[39249]: pgmap v806: 158 pgs: 158 active+clean; 1.8 GiB data, 9.4 GiB used, 616 GiB / 626 GiB avail; 1.2 KiB/s rd, 113 MiB/s wr, 1.80k op/s 2024-09-18T08:18:54.980 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:18:54 smithi007 ceph-mon[39249]: from='client.? 172.21.15.7:0/2090504115' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-18T08:18:54.980 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:18:54 smithi007 ceph-mon[39249]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-18T08:18:55.013 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6 2024-09-18T08:18:55.034 INFO:tasks.radosbench.radosbench.0.smithi007.stdout:2024-09-18T08:18:55.034443+0000 min lat: 0.00108817 max lat: 0.27165 avg lat: 0.0086701 2024-09-18T08:18:55.035 INFO:tasks.radosbench.radosbench.0.smithi007.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-09-18T08:18:55.035 INFO:tasks.radosbench.radosbench.0.smithi007.stdout: 20 16 36909 36893 115.263 95.6875 0.00742746 0.0086701 2024-09-18T08:18:55.038 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:18:55.039 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:18:55.039 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6' 2024-09-18T08:18:55.140 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2 2024-09-18T08:18:55.155 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:18:54 smithi050 ceph-mon[32371]: pgmap v806: 158 pgs: 158 active+clean; 1.8 GiB data, 9.4 GiB used, 616 GiB / 626 GiB avail; 1.2 KiB/s rd, 113 MiB/s wr, 1.80k op/s 2024-09-18T08:18:55.155 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:18:54 smithi050 ceph-mon[32371]: from='client.? 172.21.15.7:0/2090504115' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-18T08:18:55.155 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:18:54 smithi050 ceph-mon[32371]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-18T08:18:55.178 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:18:55.179 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:18:55.180 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2' 2024-09-18T08:18:55.280 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7 2024-09-18T08:18:55.311 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:18:55.311 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:18:55.312 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7' 2024-09-18T08:18:55.412 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1 2024-09-18T08:18:55.456 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:18:55.456 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:18:55.457 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1' 2024-09-18T08:18:55.557 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4 2024-09-18T08:18:55.586 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:18:55.586 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:18:55.587 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4' 2024-09-18T08:18:55.688 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4 2024-09-18T08:18:55.707 INFO:teuthology.orchestra.run.smithi007.stderr:set 6.8 pg_upmap_items mapping to [5->6,7->3,2->4] 2024-09-18T08:18:55.716 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:18:55.717 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:18:55.718 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4' 2024-09-18T08:18:55.819 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6 2024-09-18T08:18:55.848 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:18:55.849 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:18:55.849 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6' 2024-09-18T08:18:55.905 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:18:55 smithi050 ceph-mon[32371]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-18T08:18:55.905 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:18:55 smithi050 ceph-mon[32371]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-18T08:18:55.905 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:18:55 smithi050 ceph-mon[32371]: osdmap e409: 8 total, 8 up, 7 in 2024-09-18T08:18:55.905 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:18:55 smithi050 ceph-mon[32371]: from='client.? 172.21.15.7:0/1368771895' entity='client.admin' cmd={"prefix": "osd pg-upmap-items", "pgid": "6.8", "id": [5, 6, 7, 3, 2, 4]} : dispatch 2024-09-18T08:18:55.905 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:18:55 smithi050 ceph-mon[32371]: 1.3 scrub starts 2024-09-18T08:18:55.905 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:18:55 smithi050 ceph-mon[32371]: 1.3 scrub ok 2024-09-18T08:18:55.906 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:18:55 smithi050 ceph-mon[32371]: from='client.? 172.21.15.7:0/1368771895' entity='client.admin' cmd='[{"prefix": "osd pg-upmap-items", "pgid": "6.8", "id": [5, 6, 7, 3, 2, 4]}]': finished 2024-09-18T08:18:55.906 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:18:55 smithi050 ceph-mon[32371]: osdmap e410: 8 total, 8 up, 7 in 2024-09-18T08:18:55.950 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3 2024-09-18T08:18:55.978 DEBUG:teuthology.orchestra.run.smithi050:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid 779349ca-7593-11ef-bceb-c7b262605968 -- ceph daemon osd.7 dump_historic_ops 2024-09-18T08:18:55.979 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:18:55 smithi007 ceph-mon[29459]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-18T08:18:55.980 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:18:55 smithi007 ceph-mon[29459]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-18T08:18:55.980 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:18:55 smithi007 ceph-mon[29459]: osdmap e409: 8 total, 8 up, 7 in 2024-09-18T08:18:55.980 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:18:55 smithi007 ceph-mon[29459]: from='client.? 172.21.15.7:0/1368771895' entity='client.admin' cmd={"prefix": "osd pg-upmap-items", "pgid": "6.8", "id": [5, 6, 7, 3, 2, 4]} : dispatch 2024-09-18T08:18:55.980 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:18:55 smithi007 ceph-mon[29459]: 1.3 scrub starts 2024-09-18T08:18:55.980 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:18:55 smithi007 ceph-mon[29459]: 1.3 scrub ok 2024-09-18T08:18:55.980 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:18:55 smithi007 ceph-mon[29459]: from='client.? 172.21.15.7:0/1368771895' entity='client.admin' cmd='[{"prefix": "osd pg-upmap-items", "pgid": "6.8", "id": [5, 6, 7, 3, 2, 4]}]': finished 2024-09-18T08:18:55.980 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:18:55 smithi007 ceph-mon[29459]: osdmap e410: 8 total, 8 up, 7 in 2024-09-18T08:18:55.980 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:18:55 smithi007 ceph-mon[39249]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-18T08:18:55.980 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:18:55 smithi007 ceph-mon[39249]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-18T08:18:55.980 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:18:55 smithi007 ceph-mon[39249]: osdmap e409: 8 total, 8 up, 7 in 2024-09-18T08:18:55.981 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:18:55 smithi007 ceph-mon[39249]: from='client.? 172.21.15.7:0/1368771895' entity='client.admin' cmd={"prefix": "osd pg-upmap-items", "pgid": "6.8", "id": [5, 6, 7, 3, 2, 4]} : dispatch 2024-09-18T08:18:55.981 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:18:55 smithi007 ceph-mon[39249]: 1.3 scrub starts 2024-09-18T08:18:55.981 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:18:55 smithi007 ceph-mon[39249]: 1.3 scrub ok 2024-09-18T08:18:55.981 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:18:55 smithi007 ceph-mon[39249]: from='client.? 172.21.15.7:0/1368771895' entity='client.admin' cmd='[{"prefix": "osd pg-upmap-items", "pgid": "6.8", "id": [5, 6, 7, 3, 2, 4]}]': finished 2024-09-18T08:18:55.981 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:18:55 smithi007 ceph-mon[39249]: osdmap e410: 8 total, 8 up, 7 in 2024-09-18T08:18:55.990 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:18:55.991 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:18:55.991 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3' 2024-09-18T08:18:56.035 INFO:tasks.radosbench.radosbench.0.smithi007.stdout: 21 16 38647 38631 114.947 108.625 0.015701 0.00869358 2024-09-18T08:18:56.092 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7 2024-09-18T08:18:56.125 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:18:56.126 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:18:56.126 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7' 2024-09-18T08:18:56.227 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6 2024-09-18T08:18:56.266 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:18:56.266 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:18:56.267 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6' 2024-09-18T08:18:56.352 INFO:teuthology.orchestra.run.smithi050.stderr:Inferring config /var/lib/ceph/779349ca-7593-11ef-bceb-c7b262605968/mon.b/config 2024-09-18T08:18:56.368 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5 2024-09-18T08:18:56.405 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:18:56.408 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:18:56.408 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5' 2024-09-18T08:18:56.510 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1 2024-09-18T08:18:56.560 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:18:56.560 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:18:56.561 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1' 2024-09-18T08:18:56.662 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2 2024-09-18T08:18:56.691 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:18:56.692 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:18:56.692 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2' 2024-09-18T08:18:56.744 INFO:journalctl@ceph.grafana.a.smithi050.stdout:Sep 18 08:18:56 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-grafana-a[60403]: logger=sqlstore.transactions t=2024-09-18T08:18:56.474533647Z level=info msg="Database locked, sleeping then retrying" error="database is locked" retry=0 code="database is locked" 2024-09-18T08:18:56.793 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1 2024-09-18T08:18:56.824 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid 779349ca-7593-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-18T08:18:56.834 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:18:56.836 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:18:56.837 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1' 2024-09-18T08:18:56.872 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:18:56 smithi007 ceph-mon[29459]: pgmap v808: 158 pgs: 158 active+clean; 2.1 GiB data, 10 GiB used, 616 GiB / 626 GiB avail; 1023 B/s rd, 95 MiB/s wr, 1.52k op/s 2024-09-18T08:18:56.873 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:18:56 smithi007 ceph-mon[29459]: 3.e scrub starts 2024-09-18T08:18:56.873 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:18:56 smithi007 ceph-mon[29459]: 3.e scrub ok 2024-09-18T08:18:56.873 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Sep 18 08:18:56 smithi007 ceph-779349ca-7593-11ef-bceb-c7b262605968-mgr-y[29688]: ::ffff:172.21.15.50 - - [18/Sep/2024:08:18:56] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-09-18T08:18:56.873 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:18:56 smithi007 ceph-mon[39249]: pgmap v808: 158 pgs: 158 active+clean; 2.1 GiB data, 10 GiB used, 616 GiB / 626 GiB avail; 1023 B/s rd, 95 MiB/s wr, 1.52k op/s 2024-09-18T08:18:56.873 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:18:56 smithi007 ceph-mon[39249]: 3.e scrub starts 2024-09-18T08:18:56.873 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:18:56 smithi007 ceph-mon[39249]: 3.e scrub ok 2024-09-18T08:18:56.916 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2024-09-18T08:18:56.916 INFO:teuthology.orchestra.run.smithi050.stdout: "size": 20, 2024-09-18T08:18:56.917 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 600, 2024-09-18T08:18:56.917 INFO:teuthology.orchestra.run.smithi050.stdout: "ops": [ 2024-09-18T08:18:56.917 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:18:56.917 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.15582.0:53567 9.3 9:ce224b9e:::benchmark_data_smithi007_124019_object53566:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e243)", 2024-09-18T08:18:56.917 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-09-18T08:11:41.311629+0000", 2024-09-18T08:18:56.917 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 435.60184491699999, 2024-09-18T08:18:56.917 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.0246146169999999, 2024-09-18T08:18:56.917 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-09-18T08:18:56.917 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-09-18T08:18:56.917 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "reached pg", 2024-09-18T08:18:56.917 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2024-09-18T08:18:56.917 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.15582", 2024-09-18T08:18:56.918 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "172.21.15.7:0/3598605839", 2024-09-18T08:18:56.918 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 53567 2024-09-18T08:18:56.918 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:18:56.918 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-09-18T08:18:56.918 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:18:56.918 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-09-18T08:18:56.918 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:11:41.311629+0000", 2024-09-18T08:18:56.918 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-18T08:18:56.918 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:18:56.918 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:18:56.918 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-09-18T08:18:56.918 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:11:41.311629+0000", 2024-09-18T08:18:56.918 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-18T08:18:56.918 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:18:56.919 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:18:56.919 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-09-18T08:18:56.919 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:11:41.311631+0000", 2024-09-18T08:18:56.919 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.9929999999999998e-06 2024-09-18T08:18:56.919 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:18:56.919 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:18:56.919 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-09-18T08:18:56.919 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:11:41.311694+0000", 2024-09-18T08:18:56.919 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 6.3456999999999999e-05 2024-09-18T08:18:56.919 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:18:56.919 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:18:56.919 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-09-18T08:18:56.919 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:11:41.311711+0000", 2024-09-18T08:18:56.919 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.6816e-05 2024-09-18T08:18:56.920 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:18:56.920 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:18:56.920 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-09-18T08:18:56.920 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:11:41.311721+0000", 2024-09-18T08:18:56.920 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 9.7259999999999997e-06 2024-09-18T08:18:56.920 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:18:56.920 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:18:56.920 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-18T08:18:56.920 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:11:41.311783+0000", 2024-09-18T08:18:56.920 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 6.1737999999999997e-05 2024-09-18T08:18:56.920 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:18:56.920 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:18:56.920 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-09-18T08:18:56.921 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:11:41.311938+0000", 2024-09-18T08:18:56.921 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000155291 2024-09-18T08:18:56.921 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:18:56.921 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:18:56.921 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "waiting for subops from 1,3", 2024-09-18T08:18:56.921 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:11:41.312026+0000", 2024-09-18T08:18:56.921 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 8.8227999999999996e-05 2024-09-18T08:18:56.921 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:18:56.921 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:18:56.921 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "op_commit", 2024-09-18T08:18:56.921 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:11:41.313766+0000", 2024-09-18T08:18:56.921 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0017395819999999999 2024-09-18T08:18:56.921 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:18:56.921 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:18:56.922 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:18:56.922 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:11:41.320080+0000", 2024-09-18T08:18:56.922 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0063146590000000002 2024-09-18T08:18:56.922 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:18:56.922 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:18:56.922 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-18T08:18:56.922 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:11:41.329868+0000", 2024-09-18T08:18:56.922 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0097875719999999996 2024-09-18T08:18:56.922 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:18:56.922 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:18:56.922 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "waiting for peered", 2024-09-18T08:18:56.922 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:11:41.329881+0000", 2024-09-18T08:18:56.922 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.3154e-05 2024-09-18T08:18:56.923 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:18:56.923 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:18:56.923 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-18T08:18:56.923 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:11:42.336164+0000", 2024-09-18T08:18:56.923 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.0062827560000001 2024-09-18T08:18:56.923 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:18:56.923 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:18:56.923 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-09-18T08:18:56.923 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:11:42.336244+0000", 2024-09-18T08:18:56.923 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 7.9645000000000001e-05 2024-09-18T08:18:56.923 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-18T08:18:56.923 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-09-18T08:18:56.923 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-18T08:18:56.923 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:18:56.924 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:18:56.924 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.15582.0:53569 9.3 9:cedd1e96:::benchmark_data_smithi007_124019_object53568:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e243)", 2024-09-18T08:18:56.924 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-09-18T08:11:41.313768+0000", 2024-09-18T08:18:56.924 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 435.59970567800002, 2024-09-18T08:18:56.924 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.0232493, 2024-09-18T08:18:56.924 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-09-18T08:18:56.924 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-09-18T08:18:56.924 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "reached pg", 2024-09-18T08:18:56.924 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2024-09-18T08:18:56.924 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.15582", 2024-09-18T08:18:56.924 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "172.21.15.7:0/3598605839", 2024-09-18T08:18:56.924 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 53569 2024-09-18T08:18:56.924 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:18:56.924 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-09-18T08:18:56.925 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:18:56.925 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-09-18T08:18:56.925 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:11:41.313768+0000", 2024-09-18T08:18:56.925 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-18T08:18:56.925 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:18:56.925 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:18:56.925 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-09-18T08:18:56.925 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:11:41.313768+0000", 2024-09-18T08:18:56.925 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-18T08:18:56.925 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:18:56.925 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:18:56.925 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-09-18T08:18:56.925 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:11:41.313769+0000", 2024-09-18T08:18:56.925 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.2109999999999999e-06 2024-09-18T08:18:56.926 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:18:56.926 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:18:56.926 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-09-18T08:18:56.926 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:11:41.313829+0000", 2024-09-18T08:18:56.926 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 5.9518999999999997e-05 2024-09-18T08:18:56.926 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:18:56.926 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:18:56.926 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-09-18T08:18:56.926 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:11:41.313839+0000", 2024-09-18T08:18:56.926 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 9.7820000000000005e-06 2024-09-18T08:18:56.926 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:18:56.926 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:18:56.926 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-09-18T08:18:56.926 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:11:41.313846+0000", 2024-09-18T08:18:56.927 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 7.6410000000000002e-06 2024-09-18T08:18:56.927 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:18:56.927 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:18:56.927 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-18T08:18:56.927 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:11:41.313909+0000", 2024-09-18T08:18:56.927 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 6.3079999999999999e-05 2024-09-18T08:18:56.927 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:18:56.927 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:18:56.927 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-09-18T08:18:56.927 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:11:41.313999+0000", 2024-09-18T08:18:56.927 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 8.9446999999999999e-05 2024-09-18T08:18:56.927 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:18:56.927 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:18:56.927 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "waiting for subops from 1,3", 2024-09-18T08:18:56.928 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:11:41.314130+0000", 2024-09-18T08:18:56.928 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000130748 2024-09-18T08:18:56.928 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:18:56.928 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:18:56.928 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "op_commit", 2024-09-18T08:18:56.928 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:11:41.315241+0000", 2024-09-18T08:18:56.928 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.001111108 2024-09-18T08:18:56.928 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:18:56.928 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:18:56.928 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:18:56.928 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:11:41.320126+0000", 2024-09-18T08:18:56.928 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0048852110000000004 2024-09-18T08:18:56.928 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:18:56.928 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:18:56.929 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-18T08:18:56.929 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:11:41.329924+0000", 2024-09-18T08:18:56.929 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0097982309999999993 2024-09-18T08:18:56.929 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:18:56.929 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:18:56.929 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "waiting for peered", 2024-09-18T08:18:56.929 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:11:41.329932+0000", 2024-09-18T08:18:56.929 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 7.7670000000000003e-06 2024-09-18T08:18:56.929 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:18:56.929 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:18:56.929 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-18T08:18:56.929 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:11:42.336953+0000", 2024-09-18T08:18:56.929 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.007021433 2024-09-18T08:18:56.929 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:18:56.930 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:18:56.930 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-09-18T08:18:56.930 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:11:42.337018+0000", 2024-09-18T08:18:56.930 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 6.4121999999999994e-05 2024-09-18T08:18:56.930 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-18T08:18:56.930 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-09-18T08:18:56.930 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-18T08:18:56.930 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:18:56.930 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:18:56.931 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.15582.0:53578 9.d 9:bae76b75:::benchmark_data_smithi007_124019_object53577:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e244)", 2024-09-18T08:18:56.931 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-09-18T08:11:41.338704+0000", 2024-09-18T08:18:56.931 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 435.57477023299998, 2024-09-18T08:18:56.931 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.0150159940000001, 2024-09-18T08:18:56.931 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-09-18T08:18:56.931 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-09-18T08:18:56.931 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T08:18:56.931 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2024-09-18T08:18:56.931 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.15582", 2024-09-18T08:18:56.931 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "172.21.15.7:0/3598605839", 2024-09-18T08:18:56.931 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 53578 2024-09-18T08:18:56.931 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:18:56.932 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-09-18T08:18:56.932 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:18:56.932 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-09-18T08:18:56.932 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:11:41.338704+0000", 2024-09-18T08:18:56.932 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-18T08:18:56.932 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:18:56.932 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:18:56.932 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-09-18T08:18:56.932 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:11:41.338704+0000", 2024-09-18T08:18:56.932 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-18T08:18:56.932 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:18:56.932 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:18:56.932 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-09-18T08:18:56.932 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:11:41.338705+0000", 2024-09-18T08:18:56.933 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.339e-06 2024-09-18T08:18:56.933 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:18:56.933 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:18:56.933 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-09-18T08:18:56.933 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:11:41.338780+0000", 2024-09-18T08:18:56.933 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 7.5328999999999997e-05 2024-09-18T08:18:56.933 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:18:56.933 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:18:56.933 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-09-18T08:18:56.933 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:11:41.338795+0000", 2024-09-18T08:18:56.933 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.4476e-05 2024-09-18T08:18:56.933 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:18:56.933 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:18:56.934 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-09-18T08:18:56.934 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:11:41.338816+0000", 2024-09-18T08:18:56.934 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 2.1205000000000001e-05 2024-09-18T08:18:56.934 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:18:56.934 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:18:56.934 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-18T08:18:56.934 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:11:41.338884+0000", 2024-09-18T08:18:56.934 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 6.8219999999999994e-05 2024-09-18T08:18:56.934 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:18:56.934 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:18:56.934 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "waiting for peered", 2024-09-18T08:18:56.934 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:11:41.338895+0000", 2024-09-18T08:18:56.934 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.0559e-05 2024-09-18T08:18:56.935 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:18:56.935 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:18:56.935 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-18T08:18:56.935 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:11:42.344107+0000", 2024-09-18T08:18:56.935 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.0052120229999999 2024-09-18T08:18:56.935 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:18:56.935 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:18:56.935 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-09-18T08:18:56.935 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:11:42.344239+0000", 2024-09-18T08:18:56.935 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000132655 2024-09-18T08:18:56.935 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:18:56.935 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:18:56.935 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "waiting for subops from 5", 2024-09-18T08:18:56.935 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:11:42.344389+0000", 2024-09-18T08:18:56.936 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00014924099999999999 2024-09-18T08:18:56.936 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:18:56.936 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:18:56.936 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:18:56.936 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:11:42.353327+0000", 2024-09-18T08:18:56.936 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0089384949999999994 2024-09-18T08:18:56.936 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:18:56.936 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:18:56.936 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "op_commit", 2024-09-18T08:18:56.936 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:11:42.353567+0000", 2024-09-18T08:18:56.936 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000240232 2024-09-18T08:18:56.936 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:18:56.936 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:18:56.937 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "commit_sent", 2024-09-18T08:18:56.937 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:11:42.353618+0000", 2024-09-18T08:18:56.937 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 5.0488999999999997e-05 2024-09-18T08:18:56.937 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:18:56.937 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:18:56.937 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-09-18T08:18:56.937 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:11:42.353720+0000", 2024-09-18T08:18:56.937 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00010173100000000001 2024-09-18T08:18:56.937 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-18T08:18:56.937 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-09-18T08:18:56.937 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-18T08:18:56.937 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:18:56.937 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:18:56.937 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.15582.0:53579 9.d 9:bbcadf3f:::benchmark_data_smithi007_124019_object53578:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e244)", 2024-09-18T08:18:56.938 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-09-18T08:11:41.355809+0000", 2024-09-18T08:18:56.938 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 435.55766443599998, 2024-09-18T08:18:56.938 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.99805365400000001, 2024-09-18T08:18:56.938 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-09-18T08:18:56.938 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-09-18T08:18:56.938 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T08:18:56.938 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2024-09-18T08:18:56.938 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.15582", 2024-09-18T08:18:56.938 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "172.21.15.7:0/3598605839", 2024-09-18T08:18:56.938 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 53579 2024-09-18T08:18:56.938 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:18:56.938 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-09-18T08:18:56.938 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:18:56.938 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-09-18T08:18:56.939 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:11:41.355809+0000", 2024-09-18T08:18:56.939 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-18T08:18:56.939 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:18:56.939 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:18:56.939 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-09-18T08:18:56.939 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:11:41.355809+0000", 2024-09-18T08:18:56.939 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-18T08:18:56.939 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:18:56.939 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:18:56.939 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-09-18T08:18:56.939 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:11:41.355811+0000", 2024-09-18T08:18:56.939 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.204e-06 2024-09-18T08:18:56.939 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:18:56.939 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:18:56.940 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-09-18T08:18:56.940 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:11:41.355868+0000", 2024-09-18T08:18:56.940 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 5.6900000000000001e-05 2024-09-18T08:18:56.940 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:18:56.940 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:18:56.940 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-09-18T08:18:56.940 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:11:41.355879+0000", 2024-09-18T08:18:56.940 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.1314e-05 2024-09-18T08:18:56.940 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:18:56.940 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:18:56.940 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-09-18T08:18:56.940 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:11:41.355885+0000", 2024-09-18T08:18:56.940 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 5.8420000000000001e-06 2024-09-18T08:18:56.941 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:18:56.941 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:18:56.941 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-18T08:18:56.941 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:11:41.355958+0000", 2024-09-18T08:18:56.941 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 7.3339999999999999e-05 2024-09-18T08:18:56.941 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:18:56.941 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:18:56.941 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "waiting for peered", 2024-09-18T08:18:56.941 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:11:41.355964+0000", 2024-09-18T08:18:56.941 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 5.6790000000000002e-06 2024-09-18T08:18:56.941 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:18:56.941 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:18:56.941 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-18T08:18:56.941 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:11:42.346633+0000", 2024-09-18T08:18:56.942 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.99066880899999998 2024-09-18T08:18:56.942 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:18:56.942 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:18:56.942 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-09-18T08:18:56.942 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:11:42.346957+0000", 2024-09-18T08:18:56.942 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00032403199999999998 2024-09-18T08:18:56.942 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:18:56.942 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:18:56.942 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "waiting for subops from 5", 2024-09-18T08:18:56.942 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:11:42.347119+0000", 2024-09-18T08:18:56.942 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000162812 2024-09-18T08:18:56.942 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:18:56.942 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:18:56.942 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:18:56.943 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:11:42.353508+0000", 2024-09-18T08:18:56.943 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0063887969999999999 2024-09-18T08:18:56.943 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:18:56.943 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:18:56.943 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "op_commit", 2024-09-18T08:18:56.943 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:11:42.353736+0000", 2024-09-18T08:18:56.943 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000228163 2024-09-18T08:18:56.943 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:18:56.943 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:18:56.943 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "commit_sent", 2024-09-18T08:18:56.943 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:11:42.353782+0000", 2024-09-18T08:18:56.943 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4.5800000000000002e-05 2024-09-18T08:18:56.943 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:18:56.943 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:18:56.944 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-09-18T08:18:56.944 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:11:42.353863+0000", 2024-09-18T08:18:56.944 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 8.0962000000000002e-05 2024-09-18T08:18:56.944 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-18T08:18:56.944 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-09-18T08:18:56.944 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-18T08:18:56.944 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:18:56.944 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:18:56.944 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.26087.0:2236 11.4 11:24ac0a38:::benchmark_data_smithi007_172546_object2235:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e406)", 2024-09-18T08:18:56.944 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-09-18T08:18:35.837859+0000", 2024-09-18T08:18:56.944 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 21.075615358, 2024-09-18T08:18:56.945 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:18:56 smithi050 ceph-mon[32371]: pgmap v808: 158 pgs: 158 active+clean; 2.1 GiB data, 10 GiB used, 616 GiB / 626 GiB avail; 1023 B/s rd, 95 MiB/s wr, 1.52k op/s 2024-09-18T08:18:56.945 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:18:56 smithi050 ceph-mon[32371]: 3.e scrub starts 2024-09-18T08:18:56.945 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:18:56 smithi050 ceph-mon[32371]: 3.e scrub ok 2024-09-18T08:18:56.945 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.26786262, 2024-09-18T08:18:56.945 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-09-18T08:18:56.946 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-09-18T08:18:56.946 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T08:18:56.946 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2024-09-18T08:18:56.946 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.26087", 2024-09-18T08:18:56.946 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "172.21.15.7:0/2752273185", 2024-09-18T08:18:56.946 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 2236 2024-09-18T08:18:56.946 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:18:56.946 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-09-18T08:18:56.946 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:18:56.946 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-09-18T08:18:56.946 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:18:35.837859+0000", 2024-09-18T08:18:56.946 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-18T08:18:56.946 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:18:56.946 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:18:56.947 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-09-18T08:18:56.947 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:18:35.837859+0000", 2024-09-18T08:18:56.947 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-18T08:18:56.947 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:18:56.947 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:18:56.947 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-09-18T08:18:56.947 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:18:35.837859+0000", 2024-09-18T08:18:56.947 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 7.6000000000000003e-07 2024-09-18T08:18:56.947 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:18:56.947 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:18:56.947 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-09-18T08:18:56.947 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:18:35.838000+0000", 2024-09-18T08:18:56.947 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000141137 2024-09-18T08:18:56.947 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:18:56.948 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:18:56.948 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-09-18T08:18:56.948 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:18:35.838023+0000", 2024-09-18T08:18:56.948 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 2.2073999999999999e-05 2024-09-18T08:18:56.948 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:18:56.948 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:18:56.948 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-09-18T08:18:56.948 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:18:35.838032+0000", 2024-09-18T08:18:56.948 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 9.4900000000000006e-06 2024-09-18T08:18:56.948 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:18:56.948 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:18:56.948 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-18T08:18:56.948 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:18:35.838084+0000", 2024-09-18T08:18:56.948 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 5.2394e-05 2024-09-18T08:18:56.949 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:18:56.949 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:18:56.949 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-09-18T08:18:56.949 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:18:35.838191+0000", 2024-09-18T08:18:56.949 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00010650900000000001 2024-09-18T08:18:56.949 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:18:56.949 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:18:56.949 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "waiting for subops from 2,6", 2024-09-18T08:18:56.949 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:18:35.838278+0000", 2024-09-18T08:18:56.949 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 8.7084999999999997e-05 2024-09-18T08:18:56.949 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:18:56.949 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:18:56.949 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "op_commit", 2024-09-18T08:18:56.949 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:18:35.838715+0000", 2024-09-18T08:18:56.950 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000437231 2024-09-18T08:18:56.950 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:18:56.950 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:18:56.950 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:18:56.950 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:18:35.839045+0000", 2024-09-18T08:18:56.950 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000329632 2024-09-18T08:18:56.950 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:18:56.950 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:18:56.950 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:18:56.950 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:18:36.105610+0000", 2024-09-18T08:18:56.950 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.26656476600000001 2024-09-18T08:18:56.950 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:18:56.950 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:18:56.950 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "commit_sent", 2024-09-18T08:18:56.951 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:18:36.105666+0000", 2024-09-18T08:18:56.951 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 5.6251000000000001e-05 2024-09-18T08:18:56.951 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:18:56.951 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:18:56.951 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-09-18T08:18:56.951 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:18:36.105721+0000", 2024-09-18T08:18:56.951 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 5.5291000000000003e-05 2024-09-18T08:18:56.951 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-18T08:18:56.951 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-09-18T08:18:56.951 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-18T08:18:56.951 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:18:56.951 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:18:56.951 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_repop(client.26087.0:2225 11.f e406/401 11:fa9370c2:::benchmark_data_smithi007_172546_object2224:head v 406'146, mlcod=406'144)", 2024-09-18T08:18:56.951 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-09-18T08:18:35.838648+0000", 2024-09-18T08:18:56.952 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 21.074826287, 2024-09-18T08:18:56.952 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.26600996399999999, 2024-09-18T08:18:56.952 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-09-18T08:18:56.952 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-09-18T08:18:56.952 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T08:18:56.952 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-09-18T08:18:56.952 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:18:56.952 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-09-18T08:18:56.952 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:18:35.838648+0000", 2024-09-18T08:18:56.952 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-18T08:18:56.952 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:18:56.952 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:18:56.952 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-09-18T08:18:56.952 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:18:35.838648+0000", 2024-09-18T08:18:56.953 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-18T08:18:56.953 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:18:56.953 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:18:56.953 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-09-18T08:18:56.953 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:18:35.838648+0000", 2024-09-18T08:18:56.953 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4.4099999999999999e-07 2024-09-18T08:18:56.953 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:18:56.953 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:18:56.953 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-09-18T08:18:56.953 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:18:35.838686+0000", 2024-09-18T08:18:56.953 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 3.7471000000000001e-05 2024-09-18T08:18:56.953 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:18:56.953 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:18:56.953 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-09-18T08:18:56.954 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:18:35.838694+0000", 2024-09-18T08:18:56.954 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 7.9850000000000006e-06 2024-09-18T08:18:56.954 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:18:56.954 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:18:56.954 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-09-18T08:18:56.954 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:18:35.838700+0000", 2024-09-18T08:18:56.954 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 6.1009999999999997e-06 2024-09-18T08:18:56.954 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:18:56.954 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:18:56.954 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-18T08:18:56.954 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:18:35.838761+0000", 2024-09-18T08:18:56.954 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 6.1266000000000006e-05 2024-09-18T08:18:56.954 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:18:56.954 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:18:56.955 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-09-18T08:18:56.955 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:18:35.838779+0000", 2024-09-18T08:18:56.955 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.8179e-05 2024-09-18T08:18:56.955 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:18:56.955 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:18:56.955 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "commit_sent", 2024-09-18T08:18:56.955 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:18:35.839237+0000", 2024-09-18T08:18:56.955 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000457455 2024-09-18T08:18:56.955 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:18:56.955 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:18:56.955 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-09-18T08:18:56.955 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:18:36.104658+0000", 2024-09-18T08:18:56.955 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.26542106599999998 2024-09-18T08:18:56.955 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-18T08:18:56.956 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-09-18T08:18:56.956 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-18T08:18:56.956 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:18:56.956 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:18:56.956 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_repop(client.26087.0:2227 11.e e406/401 11:7e663daf:::benchmark_data_smithi007_172546_object2226:head v 406'136, mlcod=406'133)", 2024-09-18T08:18:56.956 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-09-18T08:18:35.838728+0000", 2024-09-18T08:18:56.956 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 21.074745492000002, 2024-09-18T08:18:56.956 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.26592518300000001, 2024-09-18T08:18:56.956 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-09-18T08:18:56.956 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-09-18T08:18:56.956 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T08:18:56.956 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-09-18T08:18:56.956 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:18:56.956 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-09-18T08:18:56.957 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:18:35.838728+0000", 2024-09-18T08:18:56.957 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-18T08:18:56.957 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:18:56.957 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:18:56.957 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-09-18T08:18:56.957 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:18:35.838728+0000", 2024-09-18T08:18:56.957 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-18T08:18:56.957 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:18:56.957 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:18:56.957 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-09-18T08:18:56.957 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:18:35.838729+0000", 2024-09-18T08:18:56.957 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 3.34e-07 2024-09-18T08:18:56.957 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:18:56.957 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:18:56.958 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-09-18T08:18:56.958 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:18:35.838756+0000", 2024-09-18T08:18:56.958 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 2.7030999999999999e-05 2024-09-18T08:18:56.958 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:18:56.958 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:18:56.958 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-09-18T08:18:56.958 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:18:35.838761+0000", 2024-09-18T08:18:56.958 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 5.5099999999999998e-06 2024-09-18T08:18:56.958 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:18:56.958 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:18:56.958 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-09-18T08:18:56.958 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:18:35.838765+0000", 2024-09-18T08:18:56.958 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 3.5090000000000002e-06 2024-09-18T08:18:56.958 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:18:56.959 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:18:56.959 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-18T08:18:56.959 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:18:35.838799+0000", 2024-09-18T08:18:56.959 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 3.3804999999999999e-05 2024-09-18T08:18:56.959 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:18:56.959 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:18:56.959 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-09-18T08:18:56.959 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:18:35.838810+0000", 2024-09-18T08:18:56.959 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.1861999999999999e-05 2024-09-18T08:18:56.959 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:18:56.959 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:18:56.960 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "commit_sent", 2024-09-18T08:18:56.960 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:18:35.839145+0000", 2024-09-18T08:18:56.960 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00033440100000000001 2024-09-18T08:18:56.960 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:18:56.960 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:18:56.960 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-09-18T08:18:56.960 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:18:36.104654+0000", 2024-09-18T08:18:56.960 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.265508731 2024-09-18T08:18:56.960 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-18T08:18:56.960 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-09-18T08:18:56.960 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-18T08:18:56.960 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:18:56.960 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:18:56.961 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_repop_reply(client.26087.0:2236 11.4 e406/401 ondisk, result = 0)", 2024-09-18T08:18:56.961 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-09-18T08:18:35.838973+0000", 2024-09-18T08:18:56.961 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 21.074500925999999, 2024-09-18T08:18:56.961 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.26563722400000001, 2024-09-18T08:18:56.961 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-09-18T08:18:56.961 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-09-18T08:18:56.961 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "started", 2024-09-18T08:18:56.961 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-09-18T08:18:56.961 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:18:56.961 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-09-18T08:18:56.961 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:18:35.838973+0000", 2024-09-18T08:18:56.961 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-18T08:18:56.961 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:18:56.961 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:18:56.962 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-09-18T08:18:56.962 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:18:35.838973+0000", 2024-09-18T08:18:56.962 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-18T08:18:56.962 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:18:56.962 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:18:56.962 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-09-18T08:18:56.962 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:18:35.838974+0000", 2024-09-18T08:18:56.962 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 6.5000000000000002e-07 2024-09-18T08:18:56.962 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:18:56.962 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:18:56.962 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-09-18T08:18:56.962 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:18:35.838978+0000", 2024-09-18T08:18:56.962 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4.0690000000000003e-06 2024-09-18T08:18:56.962 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:18:56.963 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:18:56.963 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-09-18T08:18:56.963 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:18:35.838987+0000", 2024-09-18T08:18:56.963 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 8.9109999999999999e-06 2024-09-18T08:18:56.963 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:18:56.963 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:18:56.963 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-09-18T08:18:56.963 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:18:35.838993+0000", 2024-09-18T08:18:56.963 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 6.1820000000000003e-06 2024-09-18T08:18:56.963 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:18:56.963 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:18:56.963 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-18T08:18:56.963 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:18:35.839036+0000", 2024-09-18T08:18:56.963 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4.3359999999999998e-05 2024-09-18T08:18:56.964 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:18:56.964 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:18:56.964 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-09-18T08:18:56.964 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:18:35.839041+0000", 2024-09-18T08:18:56.964 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 5.3340000000000001e-06 2024-09-18T08:18:56.964 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:18:56.964 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:18:56.964 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-09-18T08:18:56.964 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:18:36.104610+0000", 2024-09-18T08:18:56.964 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.26556871799999998 2024-09-18T08:18:56.964 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-18T08:18:56.964 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-09-18T08:18:56.964 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-18T08:18:56.964 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:18:56.965 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:18:56.965 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.26087.0:2242 11.4 11:251f9250:::benchmark_data_smithi007_172546_object2241:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e406)", 2024-09-18T08:18:56.965 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-09-18T08:18:35.839244+0000", 2024-09-18T08:18:56.965 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 21.074229412000001, 2024-09-18T08:18:56.965 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.26896450700000002, 2024-09-18T08:18:56.965 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-09-18T08:18:56.965 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-09-18T08:18:56.965 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T08:18:56.965 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2024-09-18T08:18:56.965 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.26087", 2024-09-18T08:18:56.965 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "172.21.15.7:0/2752273185", 2024-09-18T08:18:56.965 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 2242 2024-09-18T08:18:56.965 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:18:56.966 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-09-18T08:18:56.966 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:18:56.966 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-09-18T08:18:56.966 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:18:35.839244+0000", 2024-09-18T08:18:56.966 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-18T08:18:56.966 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:18:56.966 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:18:56.966 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-09-18T08:18:56.966 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:18:35.839244+0000", 2024-09-18T08:18:56.966 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-18T08:18:56.966 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:18:56.966 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:18:56.966 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-09-18T08:18:56.966 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:18:35.839245+0000", 2024-09-18T08:18:56.967 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 9.0400000000000005e-07 2024-09-18T08:18:56.967 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:18:56.967 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:18:56.967 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-09-18T08:18:56.967 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:18:35.839306+0000", 2024-09-18T08:18:56.967 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 6.0896000000000001e-05 2024-09-18T08:18:56.967 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:18:56.967 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:18:56.967 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-09-18T08:18:56.967 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:18:36.104634+0000", 2024-09-18T08:18:56.967 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.265327335 2024-09-18T08:18:56.967 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:18:56.967 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:18:56.967 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-09-18T08:18:56.967 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:18:36.104646+0000", 2024-09-18T08:18:56.968 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.2371000000000001e-05 2024-09-18T08:18:56.968 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:18:56.968 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:18:56.968 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-18T08:18:56.968 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:18:36.104789+0000", 2024-09-18T08:18:56.968 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000142683 2024-09-18T08:18:56.968 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:18:56.968 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:18:56.968 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-09-18T08:18:56.968 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:18:36.104994+0000", 2024-09-18T08:18:56.968 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00020551200000000001 2024-09-18T08:18:56.968 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:18:56.968 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:18:56.969 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "waiting for subops from 2,6", 2024-09-18T08:18:56.969 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:18:36.105169+0000", 2024-09-18T08:18:56.969 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00017453100000000001 2024-09-18T08:18:56.969 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:18:56.969 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:18:56.969 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:18:56.969 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:18:36.107918+0000", 2024-09-18T08:18:56.969 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.002749404 2024-09-18T08:18:56.969 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:18:56.969 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:18:56.969 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:18:56.969 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:18:36.107959+0000", 2024-09-18T08:18:56.969 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4.1193999999999999e-05 2024-09-18T08:18:56.969 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:18:56.969 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:18:56.970 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "op_commit", 2024-09-18T08:18:56.970 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:18:36.108113+0000", 2024-09-18T08:18:56.970 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00015322999999999999 2024-09-18T08:18:56.970 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:18:56.970 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:18:56.970 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "commit_sent", 2024-09-18T08:18:56.970 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:18:36.108164+0000", 2024-09-18T08:18:56.970 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 5.1425999999999999e-05 2024-09-18T08:18:56.970 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:18:56.970 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:18:56.970 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-09-18T08:18:56.970 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:18:36.108209+0000", 2024-09-18T08:18:56.970 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4.5021000000000001e-05 2024-09-18T08:18:56.970 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-18T08:18:56.971 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-09-18T08:18:56.971 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-18T08:18:56.971 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:18:56.971 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:18:56.971 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_repop(client.26087.0:2240 11.5 e406/401 11:a981f685:::benchmark_data_smithi007_172546_object2239:head v 406'138, mlcod=406'136)", 2024-09-18T08:18:56.971 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-09-18T08:18:35.839554+0000", 2024-09-18T08:18:56.971 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 21.073920090000001, 2024-09-18T08:18:56.971 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.26625396600000001, 2024-09-18T08:18:56.971 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-09-18T08:18:56.971 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-09-18T08:18:56.971 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T08:18:56.971 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-09-18T08:18:56.972 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:18:56.972 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-09-18T08:18:56.972 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:18:35.839554+0000", 2024-09-18T08:18:56.972 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-18T08:18:56.972 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:18:56.972 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:18:56.972 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-09-18T08:18:56.972 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:18:35.839554+0000", 2024-09-18T08:18:56.972 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-18T08:18:56.972 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:18:56.972 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:18:56.973 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-09-18T08:18:56.973 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:18:35.839555+0000", 2024-09-18T08:18:56.973 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 7.7400000000000002e-07 2024-09-18T08:18:56.973 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:18:56.973 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:18:56.973 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-09-18T08:18:56.973 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:18:35.839579+0000", 2024-09-18T08:18:56.973 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 2.4057e-05 2024-09-18T08:18:56.973 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:18:56.973 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:18:56.973 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-09-18T08:18:56.973 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:18:36.104615+0000", 2024-09-18T08:18:56.973 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.26503616200000002 2024-09-18T08:18:56.973 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:18:56.974 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:18:56.974 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-09-18T08:18:56.974 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:18:36.104632+0000", 2024-09-18T08:18:56.974 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.6804e-05 2024-09-18T08:18:56.974 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:18:56.974 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:18:56.974 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-18T08:18:56.974 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:18:36.104718+0000", 2024-09-18T08:18:56.974 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 8.6802e-05 2024-09-18T08:18:56.974 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:18:56.974 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:18:56.974 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-09-18T08:18:56.974 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:18:36.104745+0000", 2024-09-18T08:18:56.975 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 2.6571999999999999e-05 2024-09-18T08:18:56.975 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:18:56.975 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:18:56.975 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "commit_sent", 2024-09-18T08:18:56.975 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:18:36.105729+0000", 2024-09-18T08:18:56.975 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00098410699999999991 2024-09-18T08:18:56.975 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:18:56.975 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:18:56.975 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-09-18T08:18:56.975 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:18:36.105808+0000", 2024-09-18T08:18:56.975 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 7.8688000000000003e-05 2024-09-18T08:18:56.975 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-18T08:18:56.975 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-09-18T08:18:56.975 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-18T08:18:56.976 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:18:56.976 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:18:56.976 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.26087.0:2372 11.4 11:2d6b5b97:::benchmark_data_smithi007_172546_object2371:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e406)", 2024-09-18T08:18:56.976 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-09-18T08:18:36.145870+0000", 2024-09-18T08:18:56.976 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 20.767603823999998, 2024-09-18T08:18:56.976 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.179251092, 2024-09-18T08:18:56.976 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-09-18T08:18:56.976 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-09-18T08:18:56.976 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T08:18:56.976 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2024-09-18T08:18:56.976 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.26087", 2024-09-18T08:18:56.976 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "172.21.15.7:0/2752273185", 2024-09-18T08:18:56.976 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 2372 2024-09-18T08:18:56.976 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:18:56.977 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-09-18T08:18:56.977 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:18:56.977 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-09-18T08:18:56.977 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:18:36.145870+0000", 2024-09-18T08:18:56.977 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-18T08:18:56.977 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:18:56.977 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:18:56.977 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-09-18T08:18:56.977 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:18:36.145870+0000", 2024-09-18T08:18:56.977 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-18T08:18:56.977 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:18:56.977 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:18:56.977 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-09-18T08:18:56.977 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:18:36.145871+0000", 2024-09-18T08:18:56.978 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 6.0999999999999998e-07 2024-09-18T08:18:56.978 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:18:56.978 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:18:56.978 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-09-18T08:18:56.978 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:18:36.145937+0000", 2024-09-18T08:18:56.978 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 6.6215000000000002e-05 2024-09-18T08:18:56.978 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:18:56.978 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:18:56.978 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-09-18T08:18:56.978 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:18:36.145945+0000", 2024-09-18T08:18:56.978 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 8.5590000000000006e-06 2024-09-18T08:18:56.978 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:18:56.978 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:18:56.978 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-09-18T08:18:56.979 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:18:36.145952+0000", 2024-09-18T08:18:56.979 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 6.139e-06 2024-09-18T08:18:56.979 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:18:56.979 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:18:56.979 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-18T08:18:56.979 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:18:36.146752+0000", 2024-09-18T08:18:56.979 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00080020700000000004 2024-09-18T08:18:56.979 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:18:56.979 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:18:56.979 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-09-18T08:18:56.979 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:18:36.146875+0000", 2024-09-18T08:18:56.979 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00012306400000000001 2024-09-18T08:18:56.979 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:18:56.980 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:18:56.980 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "waiting for subops from 2,6", 2024-09-18T08:18:56.980 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:18:36.146982+0000", 2024-09-18T08:18:56.980 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000107086 2024-09-18T08:18:56.980 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:18:56.980 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:18:56.980 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "op_commit", 2024-09-18T08:18:56.980 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:18:36.147403+0000", 2024-09-18T08:18:56.980 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000421328 2024-09-18T08:18:56.980 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:18:56.980 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:18:56.980 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:18:56.980 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:18:36.148173+0000", 2024-09-18T08:18:56.980 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00076948700000000001 2024-09-18T08:18:56.981 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:18:56.981 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:18:56.981 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:18:56.981 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:18:36.325019+0000", 2024-09-18T08:18:56.981 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.176846475 2024-09-18T08:18:56.981 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:18:56.981 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:18:56.981 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "commit_sent", 2024-09-18T08:18:56.981 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:18:36.325075+0000", 2024-09-18T08:18:56.981 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 5.5557999999999999e-05 2024-09-18T08:18:56.981 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:18:56.981 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:18:56.981 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-09-18T08:18:56.981 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:18:36.325121+0000", 2024-09-18T08:18:56.982 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4.6363999999999999e-05 2024-09-18T08:18:56.982 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-18T08:18:56.982 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-09-18T08:18:56.982 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-18T08:18:56.982 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:18:56.982 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:18:56.982 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.26087.0:25616 11.3 11:c4d2ce35:::benchmark_data_smithi007_172546_object25615:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e408)", 2024-09-18T08:18:56.982 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-09-18T08:18:47.066119+0000", 2024-09-18T08:18:56.982 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 9.8473547850000003, 2024-09-18T08:18:56.982 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.18956592799999999, 2024-09-18T08:18:56.982 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-09-18T08:18:56.982 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-09-18T08:18:56.982 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T08:18:56.982 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2024-09-18T08:18:56.983 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.26087", 2024-09-18T08:18:56.983 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "172.21.15.7:0/2752273185", 2024-09-18T08:18:56.983 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 25616 2024-09-18T08:18:56.983 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:18:56.983 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-09-18T08:18:56.983 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:18:56.983 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-09-18T08:18:56.983 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:18:47.066119+0000", 2024-09-18T08:18:56.983 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-18T08:18:56.983 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:18:56.983 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:18:56.983 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-09-18T08:18:56.984 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:18:47.066119+0000", 2024-09-18T08:18:56.984 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-18T08:18:56.984 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:18:56.984 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:18:56.984 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-09-18T08:18:56.984 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:18:47.066120+0000", 2024-09-18T08:18:56.984 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 5.9800000000000003e-07 2024-09-18T08:18:56.984 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:18:56.984 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:18:56.984 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-09-18T08:18:56.984 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:18:47.066174+0000", 2024-09-18T08:18:56.984 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 5.3950999999999999e-05 2024-09-18T08:18:56.984 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:18:56.985 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:18:56.985 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-09-18T08:18:56.985 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:18:47.066184+0000", 2024-09-18T08:18:56.985 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 9.8409999999999995e-06 2024-09-18T08:18:56.985 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:18:56.985 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:18:56.985 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-09-18T08:18:56.985 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:18:47.066190+0000", 2024-09-18T08:18:56.985 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 6.4150000000000004e-06 2024-09-18T08:18:56.985 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:18:56.985 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:18:56.985 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-18T08:18:56.985 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:18:47.066242+0000", 2024-09-18T08:18:56.985 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 5.2528000000000003e-05 2024-09-18T08:18:56.985 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:18:56.986 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:18:56.986 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-09-18T08:18:56.986 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:18:47.066331+0000", 2024-09-18T08:18:56.986 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 8.8500999999999999e-05 2024-09-18T08:18:56.986 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:18:56.986 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:18:56.986 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "waiting for subops from 1,6", 2024-09-18T08:18:56.986 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:18:47.066466+0000", 2024-09-18T08:18:56.986 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00013544800000000001 2024-09-18T08:18:56.986 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:18:56.986 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:18:56.986 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "op_commit", 2024-09-18T08:18:56.986 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:18:47.067607+0000", 2024-09-18T08:18:56.986 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.001140552 2024-09-18T08:18:56.987 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:18:56.987 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:18:56.987 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:18:56.987 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:18:47.067928+0000", 2024-09-18T08:18:56.987 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000321145 2024-09-18T08:18:56.987 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:18:56.987 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:18:56.987 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:18:56.987 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:18:47.255567+0000", 2024-09-18T08:18:56.987 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.18763895 2024-09-18T08:18:56.987 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:18:56.987 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:18:56.987 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "commit_sent", 2024-09-18T08:18:56.988 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6 2024-09-18T08:18:56.989 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:18:47.255623+0000", 2024-09-18T08:18:56.989 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 5.5584000000000002e-05 2024-09-18T08:18:56.989 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:18:56.990 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:18:56.990 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-09-18T08:18:56.990 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:18:47.255685+0000", 2024-09-18T08:18:56.990 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 6.2415000000000005e-05 2024-09-18T08:18:56.990 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-18T08:18:56.990 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-09-18T08:18:56.990 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-18T08:18:56.990 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:18:56.990 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:18:56.990 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.26087.0:30778 11.4 11:21f1eaf9:::benchmark_data_smithi007_172546_object30777:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e408)", 2024-09-18T08:18:56.990 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-09-18T08:18:50.587271+0000", 2024-09-18T08:18:56.990 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 6.3262030200000003, 2024-09-18T08:18:56.990 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.202263475, 2024-09-18T08:18:56.990 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-09-18T08:18:56.991 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-09-18T08:18:56.991 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T08:18:56.991 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2024-09-18T08:18:56.991 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.26087", 2024-09-18T08:18:56.991 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "172.21.15.7:0/2752273185", 2024-09-18T08:18:56.991 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 30778 2024-09-18T08:18:56.991 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:18:56.991 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-09-18T08:18:56.991 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:18:56.991 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-09-18T08:18:56.991 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:18:50.587271+0000", 2024-09-18T08:18:56.991 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-18T08:18:56.991 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:18:56.991 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:18:56.992 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-09-18T08:18:56.992 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:18:50.587271+0000", 2024-09-18T08:18:56.992 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-18T08:18:56.992 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:18:56.992 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:18:56.992 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-09-18T08:18:56.992 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:18:50.587272+0000", 2024-09-18T08:18:56.992 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.15e-06 2024-09-18T08:18:56.992 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:18:56.992 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:18:56.992 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-09-18T08:18:56.992 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:18:50.587301+0000", 2024-09-18T08:18:56.992 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 2.8660999999999999e-05 2024-09-18T08:18:56.993 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:18:56.993 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:18:56.993 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-09-18T08:18:56.993 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:18:50.587312+0000", 2024-09-18T08:18:56.993 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.1039999999999999e-05 2024-09-18T08:18:56.993 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:18:56.993 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:18:56.993 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-09-18T08:18:56.993 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:18:50.587319+0000", 2024-09-18T08:18:56.993 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 6.8009999999999998e-06 2024-09-18T08:18:56.993 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:18:56.993 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:18:56.993 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-18T08:18:56.993 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:18:50.587369+0000", 2024-09-18T08:18:56.994 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 5.0084999999999999e-05 2024-09-18T08:18:56.994 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:18:56.994 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:18:56.994 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-09-18T08:18:56.994 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:18:50.587492+0000", 2024-09-18T08:18:56.994 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00012305000000000001 2024-09-18T08:18:56.994 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:18:56.994 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:18:56.994 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "waiting for subops from 2,6", 2024-09-18T08:18:56.994 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:18:50.587624+0000", 2024-09-18T08:18:56.994 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00013223999999999999 2024-09-18T08:18:56.994 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:18:56.994 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:18:56.994 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:18:56.995 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:18:50.603715+0000", 2024-09-18T08:18:56.995 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.016090637000000001 2024-09-18T08:18:56.995 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:18:56.995 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:18:56.995 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "op_commit", 2024-09-18T08:18:56.995 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:18:50.603787+0000", 2024-09-18T08:18:56.995 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 7.2210000000000002e-05 2024-09-18T08:18:56.995 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:18:56.995 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:18:56.995 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:18:56.995 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:18:50.789441+0000", 2024-09-18T08:18:56.995 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.185653754 2024-09-18T08:18:56.995 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:18:56.995 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:18:56.996 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "commit_sent", 2024-09-18T08:18:56.996 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:18:50.789491+0000", 2024-09-18T08:18:56.996 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 5.0954e-05 2024-09-18T08:18:56.996 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:18:56.996 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:18:56.996 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-09-18T08:18:56.996 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:18:50.789534+0000", 2024-09-18T08:18:56.996 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4.2892999999999997e-05 2024-09-18T08:18:56.996 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-18T08:18:56.996 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-09-18T08:18:56.996 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-18T08:18:56.996 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:18:56.996 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:18:56.997 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.26087.0:30781 11.4 11:24b88e2c:::benchmark_data_smithi007_172546_object30780:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e408)", 2024-09-18T08:18:56.997 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-09-18T08:18:50.587991+0000", 2024-09-18T08:18:56.997 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 6.3254833650000002, 2024-09-18T08:18:56.997 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.202302807, 2024-09-18T08:18:56.997 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-09-18T08:18:56.997 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-09-18T08:18:56.997 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T08:18:56.997 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2024-09-18T08:18:56.998 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.26087", 2024-09-18T08:18:56.998 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "172.21.15.7:0/2752273185", 2024-09-18T08:18:56.998 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 30781 2024-09-18T08:18:56.998 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:18:56.998 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-09-18T08:18:56.998 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:18:56.998 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-09-18T08:18:56.998 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:18:50.587991+0000", 2024-09-18T08:18:56.998 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-18T08:18:56.998 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:18:56.998 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:18:56.998 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-09-18T08:18:56.999 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:18:50.587991+0000", 2024-09-18T08:18:56.999 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-18T08:18:56.999 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:18:56.999 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:18:56.999 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-09-18T08:18:56.999 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:18:50.587991+0000", 2024-09-18T08:18:56.999 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 8.9199999999999999e-07 2024-09-18T08:18:56.999 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:18:56.999 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:18:56.999 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-09-18T08:18:56.999 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:18:50.588046+0000", 2024-09-18T08:18:56.999 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 5.4246000000000002e-05 2024-09-18T08:18:56.999 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:18:56.999 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:18:57.000 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-09-18T08:18:57.000 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:18:50.599257+0000", 2024-09-18T08:18:57.000 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.011211011 2024-09-18T08:18:57.000 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:18:57.000 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:18:57.000 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-09-18T08:18:57.000 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:18:50.599277+0000", 2024-09-18T08:18:57.000 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 2.0038999999999999e-05 2024-09-18T08:18:57.000 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:18:57.000 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:18:57.000 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-18T08:18:57.000 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:18:50.600845+0000", 2024-09-18T08:18:57.000 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0015685219999999999 2024-09-18T08:18:57.001 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:18:57.001 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:18:57.001 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-09-18T08:18:57.001 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:18:50.601008+0000", 2024-09-18T08:18:57.001 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000162533 2024-09-18T08:18:57.001 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:18:57.001 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:18:57.001 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "waiting for subops from 2,6", 2024-09-18T08:18:57.002 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:18:50.601103+0000", 2024-09-18T08:18:57.002 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 9.4895999999999994e-05 2024-09-18T08:18:57.002 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:18:57.002 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:18:57.002 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:18:57.002 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:18:50.603772+0000", 2024-09-18T08:18:57.002 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.002669214 2024-09-18T08:18:57.002 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:18:57.002 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:18:57.002 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "op_commit", 2024-09-18T08:18:57.002 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:18:50.603832+0000", 2024-09-18T08:18:57.002 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 5.9865999999999999e-05 2024-09-18T08:18:57.002 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:18:57.002 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:18:57.003 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:18:57.003 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:18:50.790183+0000", 2024-09-18T08:18:57.003 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.18635159800000001 2024-09-18T08:18:57.003 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:18:57.003 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:18:57.003 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "commit_sent", 2024-09-18T08:18:57.003 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:18:50.790241+0000", 2024-09-18T08:18:57.003 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 5.7398999999999997e-05 2024-09-18T08:18:57.003 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:18:57.003 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:18:57.003 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-09-18T08:18:57.003 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:18:50.790293+0000", 2024-09-18T08:18:57.003 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 5.2590999999999998e-05 2024-09-18T08:18:57.003 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-18T08:18:57.004 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-09-18T08:18:57.004 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-18T08:18:57.004 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:18:57.004 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:18:57.004 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.26087.0:30796 11.4 11:23e82cc6:::benchmark_data_smithi007_172546_object30795:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e408)", 2024-09-18T08:18:57.004 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-09-18T08:18:50.600948+0000", 2024-09-18T08:18:57.004 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 6.3125260990000003, 2024-09-18T08:18:57.004 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.18944663, 2024-09-18T08:18:57.004 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-09-18T08:18:57.004 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-09-18T08:18:57.004 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T08:18:57.004 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2024-09-18T08:18:57.004 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.26087", 2024-09-18T08:18:57.005 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "172.21.15.7:0/2752273185", 2024-09-18T08:18:57.005 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 30796 2024-09-18T08:18:57.005 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:18:57.005 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-09-18T08:18:57.005 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:18:57.005 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-09-18T08:18:57.005 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:18:50.600948+0000", 2024-09-18T08:18:57.005 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-18T08:18:57.005 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:18:57.005 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:18:57.005 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-09-18T08:18:57.005 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:18:50.600948+0000", 2024-09-18T08:18:57.005 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-18T08:18:57.005 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:18:57.006 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:18:57.006 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-09-18T08:18:57.006 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:18:50.600949+0000", 2024-09-18T08:18:57.006 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.2100000000000001e-06 2024-09-18T08:18:57.006 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:18:57.006 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:18:57.006 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-09-18T08:18:57.006 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:18:50.600977+0000", 2024-09-18T08:18:57.006 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 2.7820000000000001e-05 2024-09-18T08:18:57.006 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:18:57.006 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:18:57.006 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-09-18T08:18:57.006 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:18:50.600986+0000", 2024-09-18T08:18:57.006 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 9.251e-06 2024-09-18T08:18:57.007 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:18:57.007 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:18:57.007 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-09-18T08:18:57.007 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:18:50.600993+0000", 2024-09-18T08:18:57.007 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 7.1539999999999996e-06 2024-09-18T08:18:57.007 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:18:57.007 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:18:57.007 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-18T08:18:57.007 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:18:50.602395+0000", 2024-09-18T08:18:57.007 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0014016810000000001 2024-09-18T08:18:57.007 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:18:57.007 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:18:57.007 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-09-18T08:18:57.007 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:18:50.602520+0000", 2024-09-18T08:18:57.008 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000125105 2024-09-18T08:18:57.008 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:18:57.008 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:18:57.008 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "waiting for subops from 2,6", 2024-09-18T08:18:57.008 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:18:50.602678+0000", 2024-09-18T08:18:57.008 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00015803200000000001 2024-09-18T08:18:57.008 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:18:57.008 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:18:57.008 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "op_commit", 2024-09-18T08:18:57.008 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:18:50.603871+0000", 2024-09-18T08:18:57.008 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0011933390000000001 2024-09-18T08:18:57.008 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:18:57.008 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:18:57.008 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:18:57.009 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:18:50.605141+0000", 2024-09-18T08:18:57.009 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0012700120000000001 2024-09-18T08:18:57.009 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:18:57.009 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:18:57.009 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:18:57.009 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:18:50.790337+0000", 2024-09-18T08:18:57.009 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.18519549699999999 2024-09-18T08:18:57.009 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:18:57.009 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:18:57.009 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "commit_sent", 2024-09-18T08:18:57.009 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:18:50.790370+0000", 2024-09-18T08:18:57.009 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 3.2706999999999997e-05 2024-09-18T08:18:57.009 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:18:57.009 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:18:57.010 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-09-18T08:18:57.010 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:18:50.790394+0000", 2024-09-18T08:18:57.010 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 2.4822000000000001e-05 2024-09-18T08:18:57.010 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-18T08:18:57.010 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-09-18T08:18:57.010 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-18T08:18:57.010 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:18:57.010 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:18:57.010 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.26087.0:33250 11.3 11:cfa0d4b2:::benchmark_data_smithi007_172546_object33249:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e408)", 2024-09-18T08:18:57.010 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-09-18T08:18:52.223247+0000", 2024-09-18T08:18:57.010 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 4.6902272839999997, 2024-09-18T08:18:57.010 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.20316906300000001, 2024-09-18T08:18:57.010 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-09-18T08:18:57.010 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-09-18T08:18:57.011 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T08:18:57.011 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2024-09-18T08:18:57.011 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.26087", 2024-09-18T08:18:57.011 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "172.21.15.7:0/2752273185", 2024-09-18T08:18:57.011 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 33250 2024-09-18T08:18:57.011 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:18:57.011 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-09-18T08:18:57.011 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:18:57.011 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-09-18T08:18:57.011 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:18:52.223247+0000", 2024-09-18T08:18:57.011 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-18T08:18:57.011 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:18:57.011 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:18:57.011 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-09-18T08:18:57.012 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:18:52.223247+0000", 2024-09-18T08:18:57.012 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-18T08:18:57.012 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:18:57.012 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:18:57.012 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-09-18T08:18:57.012 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:18:52.223248+0000", 2024-09-18T08:18:57.012 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.4899999999999999e-06 2024-09-18T08:18:57.012 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:18:57.012 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:18:57.012 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-09-18T08:18:57.012 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:18:52.223279+0000", 2024-09-18T08:18:57.012 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 3.0514e-05 2024-09-18T08:18:57.012 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:18:57.012 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:18:57.013 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-09-18T08:18:57.013 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:18:52.223290+0000", 2024-09-18T08:18:57.013 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.0895e-05 2024-09-18T08:18:57.013 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:18:57.013 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:18:57.013 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-09-18T08:18:57.013 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:18:52.223297+0000", 2024-09-18T08:18:57.013 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 7.041e-06 2024-09-18T08:18:57.013 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:18:57.013 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:18:57.013 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-18T08:18:57.013 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:18:52.224985+0000", 2024-09-18T08:18:57.013 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0016882609999999999 2024-09-18T08:18:57.013 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:18:57.014 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:18:57.014 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-09-18T08:18:57.014 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:18:52.225244+0000", 2024-09-18T08:18:57.014 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00025908399999999997 2024-09-18T08:18:57.014 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:18:57.014 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:18:57.014 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "waiting for subops from 1,6", 2024-09-18T08:18:57.014 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:18:52.225413+0000", 2024-09-18T08:18:57.014 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00016919299999999999 2024-09-18T08:18:57.014 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:18:57.014 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:18:57.014 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:18:57.014 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:18:52.231539+0000", 2024-09-18T08:18:57.015 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0061261290000000001 2024-09-18T08:18:57.015 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:18:57.015 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:18:57.015 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "op_commit", 2024-09-18T08:18:57.015 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:18:52.231601+0000", 2024-09-18T08:18:57.015 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 6.1376000000000003e-05 2024-09-18T08:18:57.015 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:18:57.015 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:18:57.015 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:18:57.015 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:18:52.425925+0000", 2024-09-18T08:18:57.015 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.19432407900000001 2024-09-18T08:18:57.015 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:18:57.015 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:18:57.015 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "commit_sent", 2024-09-18T08:18:57.015 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:18:52.425978+0000", 2024-09-18T08:18:57.016 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 5.3300000000000001e-05 2024-09-18T08:18:57.016 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:18:57.016 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:18:57.016 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-09-18T08:18:57.016 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:18:52.426416+0000", 2024-09-18T08:18:57.016 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00043770099999999998 2024-09-18T08:18:57.016 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-18T08:18:57.016 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-09-18T08:18:57.016 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-18T08:18:57.016 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:18:57.016 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:18:57.016 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.26087.0:33251 11.3 11:cbf2a758:::benchmark_data_smithi007_172546_object33250:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e408)", 2024-09-18T08:18:57.016 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-09-18T08:18:52.223314+0000", 2024-09-18T08:18:57.016 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 4.690159543, 2024-09-18T08:18:57.017 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.20323029300000001, 2024-09-18T08:18:57.017 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-09-18T08:18:57.017 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-09-18T08:18:57.017 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T08:18:57.017 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2024-09-18T08:18:57.017 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.26087", 2024-09-18T08:18:57.017 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "172.21.15.7:0/2752273185", 2024-09-18T08:18:57.017 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 33251 2024-09-18T08:18:57.017 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:18:57.017 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-09-18T08:18:57.017 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:18:57.017 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-09-18T08:18:57.017 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:18:52.223314+0000", 2024-09-18T08:18:57.017 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-18T08:18:57.018 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:18:57.018 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:18:57.018 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-09-18T08:18:57.018 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:18:52.223314+0000", 2024-09-18T08:18:57.018 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-18T08:18:57.018 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:18:57.018 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:18:57.018 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-09-18T08:18:57.018 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:18:52.223315+0000", 2024-09-18T08:18:57.018 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4.2399999999999999e-07 2024-09-18T08:18:57.018 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:18:57.018 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:18:57.018 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-09-18T08:18:57.018 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:18:52.223343+0000", 2024-09-18T08:18:57.019 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 2.7869999999999999e-05 2024-09-18T08:18:57.019 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:18:57.019 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:18:57.019 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-09-18T08:18:57.019 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:18:52.223349+0000", 2024-09-18T08:18:57.019 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 6.348e-06 2024-09-18T08:18:57.019 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:18:57.019 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:18:57.019 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-09-18T08:18:57.019 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:18:52.223355+0000", 2024-09-18T08:18:57.019 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 6.2210000000000002e-06 2024-09-18T08:18:57.019 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:18:57.019 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:18:57.019 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-18T08:18:57.020 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:18:52.226684+0000", 2024-09-18T08:18:57.020 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.003329102 2024-09-18T08:18:57.020 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:18:57.020 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:18:57.020 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-09-18T08:18:57.020 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:18:52.226818+0000", 2024-09-18T08:18:57.020 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00013412999999999999 2024-09-18T08:18:57.020 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:18:57.020 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:18:57.020 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "waiting for subops from 1,6", 2024-09-18T08:18:57.020 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:18:52.226971+0000", 2024-09-18T08:18:57.020 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00015243900000000001 2024-09-18T08:18:57.020 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:18:57.021 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:18:57.021 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:18:57.021 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:18:52.231585+0000", 2024-09-18T08:18:57.021 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0046146010000000003 2024-09-18T08:18:57.021 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:18:57.021 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:18:57.021 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "op_commit", 2024-09-18T08:18:57.021 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:18:52.231605+0000", 2024-09-18T08:18:57.021 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.9777999999999999e-05 2024-09-18T08:18:57.021 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:18:57.021 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:18:57.021 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:18:57.021 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:18:52.426464+0000", 2024-09-18T08:18:57.021 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.194858749 2024-09-18T08:18:57.022 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:18:57.022 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:18:57.022 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "commit_sent", 2024-09-18T08:18:57.022 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:18:52.426512+0000", 2024-09-18T08:18:57.022 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4.8078000000000003e-05 2024-09-18T08:18:57.022 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:18:57.022 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:18:57.022 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-09-18T08:18:57.022 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:18:52.426545+0000", 2024-09-18T08:18:57.022 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 3.2552999999999998e-05 2024-09-18T08:18:57.022 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-18T08:18:57.022 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-09-18T08:18:57.022 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-18T08:18:57.022 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:18:57.023 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:18:57.023 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.26087.0:33260 11.3 11:c3ace94c:::benchmark_data_smithi007_172546_object33259:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e408)", 2024-09-18T08:18:57.023 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-09-18T08:18:52.226923+0000", 2024-09-18T08:18:57.023 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 4.6865510490000002, 2024-09-18T08:18:57.023 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.19972696000000001, 2024-09-18T08:18:57.023 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-09-18T08:18:57.023 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-09-18T08:18:57.023 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T08:18:57.023 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2024-09-18T08:18:57.023 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.26087", 2024-09-18T08:18:57.023 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "172.21.15.7:0/2752273185", 2024-09-18T08:18:57.023 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 33260 2024-09-18T08:18:57.023 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:18:57.023 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-09-18T08:18:57.024 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:18:57.024 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-09-18T08:18:57.024 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:18:52.226923+0000", 2024-09-18T08:18:57.024 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-18T08:18:57.024 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:18:57.024 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:18:57.024 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-09-18T08:18:57.024 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:18:52.226923+0000", 2024-09-18T08:18:57.024 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-18T08:18:57.024 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:18:57.024 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:18:57.024 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-09-18T08:18:57.024 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:18:52.226925+0000", 2024-09-18T08:18:57.025 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.81e-06 2024-09-18T08:18:57.025 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:18:57.025 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:18:57.025 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-09-18T08:18:57.025 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:18:52.226980+0000", 2024-09-18T08:18:57.025 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 5.5003000000000002e-05 2024-09-18T08:18:57.025 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:18:57.025 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:18:57.025 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-09-18T08:18:57.025 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:18:52.226989+0000", 2024-09-18T08:18:57.025 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 9.3179999999999996e-06 2024-09-18T08:18:57.025 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:18:57.025 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:18:57.025 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-09-18T08:18:57.026 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:18:52.226996+0000", 2024-09-18T08:18:57.028 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 6.9380000000000003e-06 2024-09-18T08:18:57.028 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:18:57.028 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:18:57.029 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-18T08:18:57.029 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:18:52.229666+0000", 2024-09-18T08:18:57.029 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.002670314 2024-09-18T08:18:57.029 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:18:57.029 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:18:57.029 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-09-18T08:18:57.029 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:18:52.229840+0000", 2024-09-18T08:18:57.029 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00017422600000000001 2024-09-18T08:18:57.029 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:18:57.029 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:18:57.029 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "waiting for subops from 1,6", 2024-09-18T08:18:57.029 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:18:52.229987+0000", 2024-09-18T08:18:57.029 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000146735 2024-09-18T08:18:57.029 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:18:57.030 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:18:57.030 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "op_commit", 2024-09-18T08:18:57.030 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:18:52.231674+0000", 2024-09-18T08:18:57.030 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.001686985 2024-09-18T08:18:57.030 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:18:57.030 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:18:57.030 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:18:57.030 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:18:52.232607+0000", 2024-09-18T08:18:57.030 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00093290699999999996 2024-09-18T08:18:57.030 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:18:57.030 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:18:57.030 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:18:57.030 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:18:52.426579+0000", 2024-09-18T08:18:57.030 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.19397212699999999 2024-09-18T08:18:57.030 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:18:57.031 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:18:57.031 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "commit_sent", 2024-09-18T08:18:57.031 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:18:52.426619+0000", 2024-09-18T08:18:57.031 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4.0132999999999997e-05 2024-09-18T08:18:57.031 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:18:57.031 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:18:57.031 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-09-18T08:18:57.031 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:18:52.426650+0000", 2024-09-18T08:18:57.031 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 3.0463999999999998e-05 2024-09-18T08:18:57.031 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-18T08:18:57.031 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-09-18T08:18:57.031 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-18T08:18:57.031 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:18:57.031 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:18:57.032 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.26087.0:33271 11.3 11:c9d2a71a:::benchmark_data_smithi007_172546_object33270:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e408)", 2024-09-18T08:18:57.032 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-09-18T08:18:52.232837+0000", 2024-09-18T08:18:57.032 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 4.6806369950000004, 2024-09-18T08:18:57.032 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.19880059, 2024-09-18T08:18:57.032 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-09-18T08:18:57.032 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-09-18T08:18:57.032 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T08:18:57.032 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2024-09-18T08:18:57.032 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.26087", 2024-09-18T08:18:57.032 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "172.21.15.7:0/2752273185", 2024-09-18T08:18:57.032 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 33271 2024-09-18T08:18:57.032 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:18:57.032 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-09-18T08:18:57.032 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:18:57.033 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-09-18T08:18:57.033 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:18:52.232837+0000", 2024-09-18T08:18:57.033 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-18T08:18:57.033 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:18:57.033 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:18:57.033 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-09-18T08:18:57.033 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:18:52.232837+0000", 2024-09-18T08:18:57.033 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-18T08:18:57.033 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:18:57.033 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:18:57.033 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-09-18T08:18:57.033 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:18:52.232839+0000", 2024-09-18T08:18:57.033 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.759e-06 2024-09-18T08:18:57.033 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:18:57.034 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:18:57.034 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-09-18T08:18:57.034 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:18:52.232881+0000", 2024-09-18T08:18:57.034 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4.2627999999999999e-05 2024-09-18T08:18:57.034 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:18:57.034 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:18:57.034 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-09-18T08:18:57.034 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:18:52.232895+0000", 2024-09-18T08:18:57.034 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.3818999999999999e-05 2024-09-18T08:18:57.034 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:18:57.034 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:18:57.034 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-09-18T08:18:57.034 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:18:52.232911+0000", 2024-09-18T08:18:57.034 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.6271999999999999e-05 2024-09-18T08:18:57.035 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:18:57.035 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:18:57.035 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-18T08:18:57.035 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:18:52.232961+0000", 2024-09-18T08:18:57.035 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 5.0068999999999998e-05 2024-09-18T08:18:57.035 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:18:57.035 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:18:57.035 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-09-18T08:18:57.035 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:18:52.233093+0000", 2024-09-18T08:18:57.035 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000131261 2024-09-18T08:18:57.035 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:18:57.035 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:18:57.035 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "waiting for subops from 1,6", 2024-09-18T08:18:57.035 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:18:52.233186+0000", 2024-09-18T08:18:57.036 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 9.3065000000000004e-05 2024-09-18T08:18:57.036 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:18:57.036 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:18:57.036 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "op_commit", 2024-09-18T08:18:57.036 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:18:52.234329+0000", 2024-09-18T08:18:57.036 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.001143633 2024-09-18T08:18:57.036 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:18:57.036 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:18:57.036 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:18:57.036 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:18:52.234488+0000", 2024-09-18T08:18:57.036 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000159071 2024-09-18T08:18:57.036 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:18:57.036 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:18:57.037 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:18:57.037 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:18:52.431537+0000", 2024-09-18T08:18:57.037 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.19704860099999999 2024-09-18T08:18:57.037 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:18:57.037 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:18:57.037 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "commit_sent", 2024-09-18T08:18:57.037 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:18:52.431591+0000", 2024-09-18T08:18:57.037 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 5.3542000000000003e-05 2024-09-18T08:18:57.037 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:18:57.037 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:18:57.037 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-09-18T08:18:57.037 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:18:52.431637+0000", 2024-09-18T08:18:57.037 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4.6869999999999997e-05 2024-09-18T08:18:57.037 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-18T08:18:57.038 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-09-18T08:18:57.038 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-18T08:18:57.038 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:18:57.038 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:18:57.038 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.26087.0:34020 11.3 11:cfbccb88:::benchmark_data_smithi007_172546_object34019:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536 in=65536b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e408)", 2024-09-18T08:18:57.038 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-09-18T08:18:52.951931+0000", 2024-09-18T08:18:57.038 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 3.9615428110000002, 2024-09-18T08:18:57.038 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.22312789599999999, 2024-09-18T08:18:57.038 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-09-18T08:18:57.038 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-09-18T08:18:57.038 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T08:18:57.038 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2024-09-18T08:18:57.038 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.26087", 2024-09-18T08:18:57.038 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "172.21.15.7:0/2752273185", 2024-09-18T08:18:57.039 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 34020 2024-09-18T08:18:57.039 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:18:57.039 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-09-18T08:18:57.039 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:18:57.039 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-09-18T08:18:57.039 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:18:52.951931+0000", 2024-09-18T08:18:57.039 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-18T08:18:57.039 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:18:57.039 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:18:57.039 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-09-18T08:18:57.039 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:18:52.951931+0000", 2024-09-18T08:18:57.039 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-18T08:18:57.039 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:18:57.039 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:18:57.040 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-09-18T08:18:57.040 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:18:52.951932+0000", 2024-09-18T08:18:57.040 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.147e-06 2024-09-18T08:18:57.040 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:18:57.040 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:18:57.040 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-09-18T08:18:57.040 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:18:52.951971+0000", 2024-09-18T08:18:57.040 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 3.8837000000000001e-05 2024-09-18T08:18:57.040 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:18:57.040 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:18:57.040 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-09-18T08:18:57.040 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:18:52.951984+0000", 2024-09-18T08:18:57.040 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.2765000000000001e-05 2024-09-18T08:18:57.040 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:18:57.041 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:18:57.041 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-09-18T08:18:57.041 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:18:52.951992+0000", 2024-09-18T08:18:57.041 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 7.9580000000000007e-06 2024-09-18T08:18:57.041 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:18:57.041 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:18:57.041 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-18T08:18:57.041 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:18:52.952038+0000", 2024-09-18T08:18:57.041 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4.5763999999999998e-05 2024-09-18T08:18:57.041 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:18:57.041 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:18:57.041 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-09-18T08:18:57.041 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:18:52.952186+0000", 2024-09-18T08:18:57.041 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00014854899999999999 2024-09-18T08:18:57.042 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:18:57.042 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:18:57.042 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "waiting for subops from 1,6", 2024-09-18T08:18:57.042 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:18:52.952320+0000", 2024-09-18T08:18:57.042 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00013358200000000001 2024-09-18T08:18:57.042 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:18:57.042 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:18:57.042 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "op_commit", 2024-09-18T08:18:57.042 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:18:52.953581+0000", 2024-09-18T08:18:57.042 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0012614880000000001 2024-09-18T08:18:57.042 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:18:57.042 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:18:57.042 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:18:57.042 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:18:52.953894+0000", 2024-09-18T08:18:57.042 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00031271400000000001 2024-09-18T08:18:57.043 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:18:57.043 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:18:57.043 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:18:57.043 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:18:53.174975+0000", 2024-09-18T08:18:57.043 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.221080945 2024-09-18T08:18:57.043 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:18:57.043 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:18:57.043 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "commit_sent", 2024-09-18T08:18:57.043 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:18:53.175015+0000", 2024-09-18T08:18:57.043 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 3.9978999999999998e-05 2024-09-18T08:18:57.043 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:18:57.043 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:18:57.043 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-09-18T08:18:57.043 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:18:53.175059+0000", 2024-09-18T08:18:57.044 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4.4168000000000001e-05 2024-09-18T08:18:57.044 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-18T08:18:57.044 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-09-18T08:18:57.044 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-18T08:18:57.044 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-18T08:18:57.044 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-09-18T08:18:57.044 INFO:teuthology.orchestra.run.smithi050.stdout:} 2024-09-18T08:18:57.045 INFO:tasks.radosbench.radosbench.0.smithi007.stdout: 22 16 40335 40319 114.515 105.5 0.00421501 0.00872665 2024-09-18T08:18:57.075 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:18:57.075 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:18:57.075 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6' 2024-09-18T08:18:57.176 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6 2024-09-18T08:18:57.205 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:18:57.206 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:18:57.206 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6' 2024-09-18T08:18:57.300 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/779349ca-7593-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T08:18:57.306 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7 2024-09-18T08:18:57.334 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:18:57.335 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:18:57.335 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7' 2024-09-18T08:18:57.437 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2 2024-09-18T08:18:57.487 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:18:57.487 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:18:57.488 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2' 2024-09-18T08:18:57.510 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid 779349ca-7593-11ef-bceb-c7b262605968 -- ceph osd unset nodeep-scrub 2024-09-18T08:18:57.588 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4 2024-09-18T08:18:57.613 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:18:57.614 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:18:57.615 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4' 2024-09-18T08:18:57.715 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0 2024-09-18T08:18:57.765 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:18:57.766 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:18:57.766 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0' 2024-09-18T08:18:57.867 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1 2024-09-18T08:18:57.905 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:18:57 smithi050 ceph-mon[32371]: 6.d scrub starts 2024-09-18T08:18:57.905 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:18:57 smithi050 ceph-mon[32371]: 6.d scrub ok 2024-09-18T08:18:57.905 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:18:57 smithi050 ceph-mon[32371]: pgmap v810: 158 pgs: 158 active+clean; 2.1 GiB data, 10 GiB used, 616 GiB / 626 GiB avail; 639 B/s rd, 56 MiB/s wr, 900 op/s 2024-09-18T08:18:57.906 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:18:57 smithi050 ceph-mon[32371]: 4.f scrub starts 2024-09-18T08:18:57.910 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:18:57.911 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:18:57.912 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1' 2024-09-18T08:18:57.982 INFO:teuthology.orchestra.run.smithi007.stdout:osd.0: {} 2024-09-18T08:18:57.982 INFO:teuthology.orchestra.run.smithi007.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-18T08:18:58.013 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1 2024-09-18T08:18:58.021 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:18:57 smithi007 ceph-mon[39249]: 6.d scrub starts 2024-09-18T08:18:58.021 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:18:57 smithi007 ceph-mon[39249]: 6.d scrub ok 2024-09-18T08:18:58.021 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:18:57 smithi007 ceph-mon[39249]: pgmap v810: 158 pgs: 158 active+clean; 2.1 GiB data, 10 GiB used, 616 GiB / 626 GiB avail; 639 B/s rd, 56 MiB/s wr, 900 op/s 2024-09-18T08:18:58.021 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:18:57 smithi007 ceph-mon[39249]: 4.f scrub starts 2024-09-18T08:18:58.021 INFO:teuthology.orchestra.run.smithi007.stdout:osd.1: {} 2024-09-18T08:18:58.021 INFO:teuthology.orchestra.run.smithi007.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-18T08:18:58.022 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:18:57 smithi007 ceph-mon[29459]: 6.d scrub starts 2024-09-18T08:18:58.022 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:18:57 smithi007 ceph-mon[29459]: 6.d scrub ok 2024-09-18T08:18:58.022 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:18:57 smithi007 ceph-mon[29459]: pgmap v810: 158 pgs: 158 active+clean; 2.1 GiB data, 10 GiB used, 616 GiB / 626 GiB avail; 639 B/s rd, 56 MiB/s wr, 900 op/s 2024-09-18T08:18:58.022 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:18:57 smithi007 ceph-mon[29459]: 4.f scrub starts 2024-09-18T08:18:58.035 INFO:tasks.radosbench.radosbench.0.smithi007.stdout: 23 16 41793 41777 113.497 91.125 0.0189382 0.00880398 2024-09-18T08:18:58.061 INFO:teuthology.orchestra.run.smithi007.stdout:osd.2: {} 2024-09-18T08:18:58.061 INFO:teuthology.orchestra.run.smithi007.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-18T08:18:58.068 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/779349ca-7593-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T08:18:58.071 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:18:58.072 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:18:58.072 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1' 2024-09-18T08:18:58.085 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid 779349ca-7593-11ef-bceb-c7b262605968 -- ceph daemon osd.1 dump_ops_in_flight 2024-09-18T08:18:58.097 INFO:teuthology.orchestra.run.smithi007.stdout:osd.3: {} 2024-09-18T08:18:58.098 INFO:teuthology.orchestra.run.smithi007.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-18T08:18:58.134 INFO:teuthology.orchestra.run.smithi007.stdout:osd.4: {} 2024-09-18T08:18:58.135 INFO:teuthology.orchestra.run.smithi007.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-18T08:18:58.168 INFO:teuthology.orchestra.run.smithi007.stdout:osd.5: {} 2024-09-18T08:18:58.168 INFO:teuthology.orchestra.run.smithi007.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-18T08:18:58.173 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0 2024-09-18T08:18:58.209 INFO:teuthology.orchestra.run.smithi007.stdout:osd.6: {} 2024-09-18T08:18:58.210 INFO:teuthology.orchestra.run.smithi007.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-18T08:18:58.224 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:18:58.225 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:18:58.225 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0' 2024-09-18T08:18:58.247 INFO:teuthology.orchestra.run.smithi007.stdout:osd.7: {} 2024-09-18T08:18:58.247 INFO:teuthology.orchestra.run.smithi007.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-18T08:18:58.326 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6 2024-09-18T08:18:58.355 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:18:58.356 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:18:58.357 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6' 2024-09-18T08:18:58.457 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1 2024-09-18T08:18:58.494 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:18:58.495 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:18:58.495 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1' 2024-09-18T08:18:58.596 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6 2024-09-18T08:18:58.626 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:18:58.627 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:18:58.627 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6' 2024-09-18T08:18:58.728 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3 2024-09-18T08:18:58.770 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:18:58.771 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:18:58.771 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3' 2024-09-18T08:18:58.872 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7 2024-09-18T08:18:58.901 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:18:58.901 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:18:58.902 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7' 2024-09-18T08:18:59.002 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1 2024-09-18T08:18:59.035 INFO:tasks.radosbench.radosbench.0.smithi007.stdout: 24 16 43205 43189 112.445 88.25 0.0127362 0.00888825 2024-09-18T08:18:59.039 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:18:59.039 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:18:59.040 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1' 2024-09-18T08:18:59.140 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6 2024-09-18T08:18:59.155 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:18:58 smithi050 ceph-mon[32371]: 4.f scrub ok 2024-09-18T08:18:59.155 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:18:58 smithi050 ceph-mon[32371]: from='client.? 172.21.15.7:0/2663456255' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-18T08:18:59.155 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:18:58 smithi050 ceph-mon[32371]: 5.2 scrub starts 2024-09-18T08:18:59.155 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:18:58 smithi050 ceph-mon[32371]: 5.2 scrub ok 2024-09-18T08:18:59.172 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:18:59.173 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:18:59.173 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6' 2024-09-18T08:18:59.229 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:18:58 smithi007 ceph-mon[29459]: 4.f scrub ok 2024-09-18T08:18:59.230 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:18:58 smithi007 ceph-mon[29459]: from='client.? 172.21.15.7:0/2663456255' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-18T08:18:59.230 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:18:58 smithi007 ceph-mon[29459]: 5.2 scrub starts 2024-09-18T08:18:59.230 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:18:58 smithi007 ceph-mon[29459]: 5.2 scrub ok 2024-09-18T08:18:59.230 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:18:58 smithi007 ceph-mon[39249]: 4.f scrub ok 2024-09-18T08:18:59.230 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:18:58 smithi007 ceph-mon[39249]: from='client.? 172.21.15.7:0/2663456255' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-18T08:18:59.230 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:18:58 smithi007 ceph-mon[39249]: 5.2 scrub starts 2024-09-18T08:18:59.230 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:18:58 smithi007 ceph-mon[39249]: 5.2 scrub ok 2024-09-18T08:18:59.274 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3 2024-09-18T08:18:59.314 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:18:59.315 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:18:59.315 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3' 2024-09-18T08:18:59.416 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2 2024-09-18T08:18:59.455 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:18:59.455 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:18:59.456 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2' 2024-09-18T08:18:59.556 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1 2024-09-18T08:18:59.583 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:18:59.583 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:18:59.584 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1' 2024-09-18T08:18:59.684 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4 2024-09-18T08:18:59.712 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:18:59.713 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:18:59.713 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4' 2024-09-18T08:18:59.814 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3 2024-09-18T08:18:59.860 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:18:59.861 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:18:59.861 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3' 2024-09-18T08:18:59.963 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0 2024-09-18T08:19:00.003 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:19:00.004 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:19:00.004 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0' 2024-09-18T08:19:00.035 INFO:tasks.radosbench.radosbench.0.smithi007.stdout: 25 16 44649 44633 111.557 90.25 0.0138113 0.0089587 2024-09-18T08:19:00.054 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:18:59 smithi007 ceph-mon[29459]: 5.16 scrub starts 2024-09-18T08:19:00.054 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:18:59 smithi007 ceph-mon[29459]: 5.16 scrub ok 2024-09-18T08:19:00.054 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:18:59 smithi007 ceph-mon[29459]: 4.16 scrub starts 2024-09-18T08:19:00.054 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:18:59 smithi007 ceph-mon[29459]: 4.16 scrub ok 2024-09-18T08:19:00.054 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:18:59 smithi007 ceph-mon[29459]: pgmap v811: 158 pgs: 158 active+clean; 2.6 GiB data, 12 GiB used, 614 GiB / 626 GiB avail; 1.2 KiB/s rd, 117 MiB/s wr, 1.88k op/s 2024-09-18T08:19:00.054 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:18:59 smithi007 ceph-mon[29459]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-18T08:19:00.055 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:18:59 smithi007 ceph-mon[39249]: 5.16 scrub starts 2024-09-18T08:19:00.055 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:18:59 smithi007 ceph-mon[39249]: 5.16 scrub ok 2024-09-18T08:19:00.055 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:18:59 smithi007 ceph-mon[39249]: 4.16 scrub starts 2024-09-18T08:19:00.055 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:18:59 smithi007 ceph-mon[39249]: 4.16 scrub ok 2024-09-18T08:19:00.055 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:18:59 smithi007 ceph-mon[39249]: pgmap v811: 158 pgs: 158 active+clean; 2.6 GiB data, 12 GiB used, 614 GiB / 626 GiB avail; 1.2 KiB/s rd, 117 MiB/s wr, 1.88k op/s 2024-09-18T08:19:00.055 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:18:59 smithi007 ceph-mon[39249]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-18T08:19:00.106 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0 2024-09-18T08:19:00.153 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:19:00.154 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:19:00.154 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0' 2024-09-18T08:19:00.155 INFO:journalctl@ceph.iscsi.iscsi.a.smithi050.stdout:Sep 18 08:18:59 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-iscsi-iscsi-a[57782]: debug there is no tcmu-runner data available 2024-09-18T08:19:00.155 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:18:59 smithi050 ceph-mon[32371]: 5.16 scrub starts 2024-09-18T08:19:00.155 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:18:59 smithi050 ceph-mon[32371]: 5.16 scrub ok 2024-09-18T08:19:00.155 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:18:59 smithi050 ceph-mon[32371]: 4.16 scrub starts 2024-09-18T08:19:00.155 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:18:59 smithi050 ceph-mon[32371]: 4.16 scrub ok 2024-09-18T08:19:00.155 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:18:59 smithi050 ceph-mon[32371]: pgmap v811: 158 pgs: 158 active+clean; 2.6 GiB data, 12 GiB used, 614 GiB / 626 GiB avail; 1.2 KiB/s rd, 117 MiB/s wr, 1.88k op/s 2024-09-18T08:19:00.155 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:18:59 smithi050 ceph-mon[32371]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-18T08:19:00.255 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4 2024-09-18T08:19:00.284 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:19:00.284 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:19:00.285 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4' 2024-09-18T08:19:00.386 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0 2024-09-18T08:19:00.424 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:19:00.424 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:19:00.424 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0' 2024-09-18T08:19:00.525 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5 2024-09-18T08:19:00.560 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:19:00.560 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:19:00.561 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5' 2024-09-18T08:19:00.659 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/779349ca-7593-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T08:19:00.662 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1 2024-09-18T08:19:00.719 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:19:00.719 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:19:00.720 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1' 2024-09-18T08:19:00.782 INFO:teuthology.orchestra.run.smithi007.stderr:nodeep-scrub is unset 2024-09-18T08:19:00.819 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:19:00 smithi007 ceph-mon[29459]: 5.c scrub starts 2024-09-18T08:19:00.819 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:19:00 smithi007 ceph-mon[29459]: 5.c scrub ok 2024-09-18T08:19:00.820 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:19:00 smithi007 ceph-mon[29459]: from='client.? 172.21.15.7:0/457137690' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-18T08:19:00.820 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:19:00 smithi007 ceph-mon[29459]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-18T08:19:00.820 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:19:00 smithi007 ceph-mon[29459]: 6.16 scrub starts 2024-09-18T08:19:00.820 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:19:00 smithi007 ceph-mon[29459]: 6.16 scrub ok 2024-09-18T08:19:00.820 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:19:00 smithi007 ceph-mon[39249]: 5.c scrub starts 2024-09-18T08:19:00.820 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:19:00 smithi007 ceph-mon[39249]: 5.c scrub ok 2024-09-18T08:19:00.820 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:19:00 smithi007 ceph-mon[39249]: from='client.? 172.21.15.7:0/457137690' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-18T08:19:00.820 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:19:00 smithi007 ceph-mon[39249]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-18T08:19:00.820 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:19:00 smithi007 ceph-mon[39249]: 6.16 scrub starts 2024-09-18T08:19:00.820 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:19:00 smithi007 ceph-mon[39249]: 6.16 scrub ok 2024-09-18T08:19:00.821 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4 2024-09-18T08:19:00.850 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:19:00.850 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:19:00.851 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4' 2024-09-18T08:19:00.952 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6 2024-09-18T08:19:00.974 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:19:00.975 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:19:00.975 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6' 2024-09-18T08:19:01.036 INFO:tasks.radosbench.radosbench.0.smithi007.stdout: 26 16 46021 46005 110.564 85.75 0.00270766 0.00903066 2024-09-18T08:19:01.075 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3 2024-09-18T08:19:01.109 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:19:01.109 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:19:01.110 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3' 2024-09-18T08:19:01.155 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:19:00 smithi050 ceph-mon[32371]: 5.c scrub starts 2024-09-18T08:19:01.155 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:19:00 smithi050 ceph-mon[32371]: 5.c scrub ok 2024-09-18T08:19:01.155 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:19:00 smithi050 ceph-mon[32371]: from='client.? 172.21.15.7:0/457137690' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-18T08:19:01.155 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:19:00 smithi050 ceph-mon[32371]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-18T08:19:01.155 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:19:00 smithi050 ceph-mon[32371]: 6.16 scrub starts 2024-09-18T08:19:01.155 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:19:00 smithi050 ceph-mon[32371]: 6.16 scrub ok 2024-09-18T08:19:01.211 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3 2024-09-18T08:19:01.249 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:19:01.249 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:19:01.250 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3' 2024-09-18T08:19:01.351 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5 2024-09-18T08:19:01.381 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:19:01.382 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:19:01.382 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5' 2024-09-18T08:19:01.483 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1 2024-09-18T08:19:01.527 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:19:01.528 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:19:01.529 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1' 2024-09-18T08:19:01.571 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 6, 5, 7] out_osds: [4] dead_osds: [] live_osds: [0, 2, 3, 4, 5, 6, 7, 1] 2024-09-18T08:19:01.571 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-18T08:19:01.571 INFO:tasks.thrashosds.thrasher:Reweighting osd 2 to 0.12225889780248589 2024-09-18T08:19:01.572 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid 779349ca-7593-11ef-bceb-c7b262605968 -- ceph osd reweight 2 0.12225889780248589 2024-09-18T08:19:01.629 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5 2024-09-18T08:19:01.655 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:19:01.655 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:19:01.656 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5' 2024-09-18T08:19:01.757 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0 2024-09-18T08:19:01.811 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:19:01.811 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:19:01.812 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0' 2024-09-18T08:19:01.905 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:19:01 smithi007 ceph-mon[29459]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-18T08:19:01.905 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:19:01 smithi007 ceph-mon[29459]: Cluster is now healthy 2024-09-18T08:19:01.905 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:19:01 smithi007 ceph-mon[29459]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-18T08:19:01.905 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:19:01 smithi007 ceph-mon[29459]: 2.2 scrub starts 2024-09-18T08:19:01.905 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:19:01 smithi007 ceph-mon[29459]: osdmap e411: 8 total, 8 up, 7 in 2024-09-18T08:19:01.905 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:19:01 smithi007 ceph-mon[29459]: 2.2 scrub ok 2024-09-18T08:19:01.905 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:19:01 smithi007 ceph-mon[29459]: pgmap v813: 158 pgs: 158 active+clean; 2.6 GiB data, 12 GiB used, 614 GiB / 626 GiB avail; 756 B/s rd, 72 MiB/s wr, 1.16k op/s 2024-09-18T08:19:01.906 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:19:01 smithi007 ceph-mon[39249]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-18T08:19:01.906 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:19:01 smithi007 ceph-mon[39249]: Cluster is now healthy 2024-09-18T08:19:01.906 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:19:01 smithi007 ceph-mon[39249]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-18T08:19:01.906 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:19:01 smithi007 ceph-mon[39249]: 2.2 scrub starts 2024-09-18T08:19:01.906 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:19:01 smithi007 ceph-mon[39249]: osdmap e411: 8 total, 8 up, 7 in 2024-09-18T08:19:01.906 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:19:01 smithi007 ceph-mon[39249]: 2.2 scrub ok 2024-09-18T08:19:01.906 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:19:01 smithi007 ceph-mon[39249]: pgmap v813: 158 pgs: 158 active+clean; 2.6 GiB data, 12 GiB used, 614 GiB / 626 GiB avail; 756 B/s rd, 72 MiB/s wr, 1.16k op/s 2024-09-18T08:19:01.906 INFO:teuthology.orchestra.run.smithi007.stdout:ERROR: (22) Invalid argument 2024-09-18T08:19:01.907 INFO:teuthology.orchestra.run.smithi007.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-18T08:19:01.913 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3 2024-09-18T08:19:01.959 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:19:01.959 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:19:01.960 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3' 2024-09-18T08:19:02.036 INFO:tasks.radosbench.radosbench.0.smithi007.stdout: 27 16 47458 47442 109.795 89.8125 0.00764277 0.00910094 2024-09-18T08:19:02.060 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2 2024-09-18T08:19:02.104 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:19:02.105 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:19:02.105 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2' 2024-09-18T08:19:02.157 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:19:01 smithi050 ceph-mon[32371]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-18T08:19:02.158 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:19:01 smithi050 ceph-mon[32371]: Cluster is now healthy 2024-09-18T08:19:02.158 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:19:01 smithi050 ceph-mon[32371]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-18T08:19:02.158 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:19:01 smithi050 ceph-mon[32371]: 2.2 scrub starts 2024-09-18T08:19:02.158 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:19:01 smithi050 ceph-mon[32371]: osdmap e411: 8 total, 8 up, 7 in 2024-09-18T08:19:02.158 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:19:01 smithi050 ceph-mon[32371]: 2.2 scrub ok 2024-09-18T08:19:02.158 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:19:01 smithi050 ceph-mon[32371]: pgmap v813: 158 pgs: 158 active+clean; 2.6 GiB data, 12 GiB used, 614 GiB / 626 GiB avail; 756 B/s rd, 72 MiB/s wr, 1.16k op/s 2024-09-18T08:19:02.206 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5 2024-09-18T08:19:02.243 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:19:02.243 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:19:02.243 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5' 2024-09-18T08:19:02.344 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4 2024-09-18T08:19:02.372 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:19:02.373 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:19:02.374 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4' 2024-09-18T08:19:02.474 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1 2024-09-18T08:19:02.513 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:19:02.514 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:19:02.514 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1' 2024-09-18T08:19:02.615 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1 2024-09-18T08:19:02.625 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid 779349ca-7593-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-18T08:19:02.662 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid 779349ca-7593-11ef-bceb-c7b262605968 -- ceph daemon osd.1 dump_blocked_ops 2024-09-18T08:19:02.681 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:19:02.683 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:19:02.684 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1' 2024-09-18T08:19:02.743 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/779349ca-7593-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T08:19:02.786 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6 2024-09-18T08:19:02.812 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:19:02.812 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:19:02.813 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6' 2024-09-18T08:19:02.898 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:19:02 smithi007 ceph-mon[29459]: 5.8 scrub starts 2024-09-18T08:19:02.898 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:19:02 smithi007 ceph-mon[29459]: 5.8 scrub ok 2024-09-18T08:19:02.898 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:19:02 smithi007 ceph-mon[29459]: 5.1d scrub starts 2024-09-18T08:19:02.898 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:19:02 smithi007 ceph-mon[29459]: 5.1d scrub ok 2024-09-18T08:19:02.898 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:19:02 smithi007 ceph-mon[29459]: 1.8 scrub starts 2024-09-18T08:19:02.898 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:19:02 smithi007 ceph-mon[29459]: 1.8 scrub ok 2024-09-18T08:19:02.899 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:19:02 smithi007 ceph-mon[39249]: 5.8 scrub starts 2024-09-18T08:19:02.899 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:19:02 smithi007 ceph-mon[39249]: 5.8 scrub ok 2024-09-18T08:19:02.899 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:19:02 smithi007 ceph-mon[39249]: 5.1d scrub starts 2024-09-18T08:19:02.899 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:19:02 smithi007 ceph-mon[39249]: 5.1d scrub ok 2024-09-18T08:19:02.899 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:19:02 smithi007 ceph-mon[39249]: 1.8 scrub starts 2024-09-18T08:19:02.899 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:19:02 smithi007 ceph-mon[39249]: 1.8 scrub ok 2024-09-18T08:19:02.913 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2 2024-09-18T08:19:02.957 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:19:02.958 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:19:02.958 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2' 2024-09-18T08:19:03.036 INFO:tasks.radosbench.radosbench.0.smithi007.stdout: 28 16 48841 48825 108.961 86.4375 0.00599419 0.00917158 2024-09-18T08:19:03.059 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4 2024-09-18T08:19:03.086 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:19:03.087 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:19:03.087 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4' 2024-09-18T08:19:03.155 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:19:02 smithi050 ceph-mon[32371]: 5.8 scrub starts 2024-09-18T08:19:03.155 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:19:02 smithi050 ceph-mon[32371]: 5.8 scrub ok 2024-09-18T08:19:03.155 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:19:02 smithi050 ceph-mon[32371]: 5.1d scrub starts 2024-09-18T08:19:03.155 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:19:02 smithi050 ceph-mon[32371]: 5.1d scrub ok 2024-09-18T08:19:03.155 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:19:02 smithi050 ceph-mon[32371]: 1.8 scrub starts 2024-09-18T08:19:03.155 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:19:02 smithi050 ceph-mon[32371]: 1.8 scrub ok 2024-09-18T08:19:03.188 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6 2024-09-18T08:19:03.217 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:19:03.218 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:19:03.218 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6' 2024-09-18T08:19:03.319 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2 2024-09-18T08:19:03.360 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:19:03.361 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:19:03.361 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2' 2024-09-18T08:19:03.462 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6 2024-09-18T08:19:03.478 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/779349ca-7593-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T08:19:03.479 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/779349ca-7593-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T08:19:03.487 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:19:03.487 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:19:03.488 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6' 2024-09-18T08:19:03.588 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5 2024-09-18T08:19:03.615 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:19:03.616 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:19:03.616 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5' 2024-09-18T08:19:03.645 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid 779349ca-7593-11ef-bceb-c7b262605968 -- ceph osd set noscrub 2024-09-18T08:19:03.717 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7 2024-09-18T08:19:03.746 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:19:03.747 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:19:03.747 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7' 2024-09-18T08:19:03.848 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6 2024-09-18T08:19:03.871 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:19:03 smithi007 ceph-mon[29459]: 6.f scrub starts 2024-09-18T08:19:03.871 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:19:03 smithi007 ceph-mon[29459]: 6.f scrub ok 2024-09-18T08:19:03.871 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:19:03 smithi007 ceph-mon[29459]: 3.b scrub starts 2024-09-18T08:19:03.871 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:19:03 smithi007 ceph-mon[29459]: 3.b scrub ok 2024-09-18T08:19:03.871 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:19:03 smithi007 ceph-mon[29459]: 3.10 scrub starts 2024-09-18T08:19:03.871 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:19:03 smithi007 ceph-mon[29459]: 3.10 scrub ok 2024-09-18T08:19:03.871 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:19:03 smithi007 ceph-mon[29459]: pgmap v814: 158 pgs: 158 active+clean; 2.7 GiB data, 12 GiB used, 614 GiB / 626 GiB avail; 1.2 KiB/s rd, 78 MiB/s wr, 1.25k op/s 2024-09-18T08:19:03.872 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:19:03 smithi007 ceph-mon[39249]: 6.f scrub starts 2024-09-18T08:19:03.872 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:19:03 smithi007 ceph-mon[39249]: 6.f scrub ok 2024-09-18T08:19:03.872 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:19:03 smithi007 ceph-mon[39249]: 3.b scrub starts 2024-09-18T08:19:03.872 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:19:03 smithi007 ceph-mon[39249]: 3.b scrub ok 2024-09-18T08:19:03.872 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:19:03 smithi007 ceph-mon[39249]: 3.10 scrub starts 2024-09-18T08:19:03.872 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:19:03 smithi007 ceph-mon[39249]: 3.10 scrub ok 2024-09-18T08:19:03.872 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:19:03 smithi007 ceph-mon[39249]: pgmap v814: 158 pgs: 158 active+clean; 2.7 GiB data, 12 GiB used, 614 GiB / 626 GiB avail; 1.2 KiB/s rd, 78 MiB/s wr, 1.25k op/s 2024-09-18T08:19:03.875 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:19:03.875 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:19:03.876 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6' 2024-09-18T08:19:03.977 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4 2024-09-18T08:19:04.004 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:19:04.005 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:19:04.005 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4' 2024-09-18T08:19:04.037 INFO:tasks.radosbench.radosbench.0.smithi007.stdout: 29 16 50062 50046 107.835 76.3125 0.0160722 0.00926771 2024-09-18T08:19:04.106 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3 2024-09-18T08:19:04.155 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:19:03 smithi050 ceph-mon[32371]: 6.f scrub starts 2024-09-18T08:19:04.155 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:19:03 smithi050 ceph-mon[32371]: 6.f scrub ok 2024-09-18T08:19:04.155 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:19:03 smithi050 ceph-mon[32371]: 3.b scrub starts 2024-09-18T08:19:04.156 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:19:03 smithi050 ceph-mon[32371]: 3.b scrub ok 2024-09-18T08:19:04.156 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:19:03 smithi050 ceph-mon[32371]: 3.10 scrub starts 2024-09-18T08:19:04.156 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:19:03 smithi050 ceph-mon[32371]: 3.10 scrub ok 2024-09-18T08:19:04.156 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:19:03 smithi050 ceph-mon[32371]: pgmap v814: 158 pgs: 158 active+clean; 2.7 GiB data, 12 GiB used, 614 GiB / 626 GiB avail; 1.2 KiB/s rd, 78 MiB/s wr, 1.25k op/s 2024-09-18T08:19:04.169 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:19:04.169 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:19:04.170 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3' 2024-09-18T08:19:04.270 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4 2024-09-18T08:19:04.297 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:19:04.298 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:19:04.298 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4' 2024-09-18T08:19:04.399 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3 2024-09-18T08:19:04.423 INFO:teuthology.orchestra.run.smithi007.stdout:osd.0: {} 2024-09-18T08:19:04.423 INFO:teuthology.orchestra.run.smithi007.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-18T08:19:04.436 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:19:04.437 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:19:04.437 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3' 2024-09-18T08:19:04.463 INFO:teuthology.orchestra.run.smithi007.stdout:osd.1: {} 2024-09-18T08:19:04.463 INFO:teuthology.orchestra.run.smithi007.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-18T08:19:04.498 INFO:teuthology.orchestra.run.smithi007.stdout:osd.2: {} 2024-09-18T08:19:04.498 INFO:teuthology.orchestra.run.smithi007.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-18T08:19:04.539 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4 2024-09-18T08:19:04.548 INFO:teuthology.orchestra.run.smithi007.stdout:osd.3: {} 2024-09-18T08:19:04.549 INFO:teuthology.orchestra.run.smithi007.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-18T08:19:04.564 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:19:04.565 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:19:04.565 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4' 2024-09-18T08:19:04.590 INFO:teuthology.orchestra.run.smithi007.stdout:osd.4: {} 2024-09-18T08:19:04.590 INFO:teuthology.orchestra.run.smithi007.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-18T08:19:04.632 INFO:teuthology.orchestra.run.smithi007.stdout:osd.5: {} 2024-09-18T08:19:04.633 INFO:teuthology.orchestra.run.smithi007.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-18T08:19:04.666 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5 2024-09-18T08:19:04.667 INFO:teuthology.orchestra.run.smithi007.stdout:{ 2024-09-18T08:19:04.667 INFO:teuthology.orchestra.run.smithi007.stdout: "ops": [], 2024-09-18T08:19:04.667 INFO:teuthology.orchestra.run.smithi007.stdout: "complaint_time": 30, 2024-09-18T08:19:04.667 INFO:teuthology.orchestra.run.smithi007.stdout: "num_blocked_ops": 0 2024-09-18T08:19:04.667 INFO:teuthology.orchestra.run.smithi007.stdout:} 2024-09-18T08:19:04.674 INFO:teuthology.orchestra.run.smithi007.stdout:osd.6: {} 2024-09-18T08:19:04.674 INFO:teuthology.orchestra.run.smithi007.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-18T08:19:04.693 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:19:04.694 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:19:04.694 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5' 2024-09-18T08:19:04.696 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/779349ca-7593-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T08:19:04.717 INFO:teuthology.orchestra.run.smithi007.stdout:osd.7: {} 2024-09-18T08:19:04.717 INFO:teuthology.orchestra.run.smithi007.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-18T08:19:04.795 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7 2024-09-18T08:19:04.816 INFO:teuthology.orchestra.run.smithi007.stderr:reweighted osd.2 to 0.122259 (1f4c) 2024-09-18T08:19:04.826 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:19:04.826 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:19:04.827 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7' 2024-09-18T08:19:04.928 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5 2024-09-18T08:19:04.951 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:19:04.951 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:19:04.952 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5' 2024-09-18T08:19:04.980 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:19:04 smithi007 ceph-mon[29459]: 6.12 scrub starts 2024-09-18T08:19:04.980 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:19:04 smithi007 ceph-mon[29459]: 6.12 scrub ok 2024-09-18T08:19:04.980 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:19:04 smithi007 ceph-mon[29459]: purged_snaps scrub starts 2024-09-18T08:19:04.980 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:19:04 smithi007 ceph-mon[29459]: purged_snaps scrub ok 2024-09-18T08:19:04.980 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:19:04 smithi007 ceph-mon[29459]: 5.1b scrub starts 2024-09-18T08:19:04.980 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:19:04 smithi007 ceph-mon[29459]: 5.1b scrub ok 2024-09-18T08:19:04.980 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:19:04 smithi007 ceph-mon[29459]: osdmap e412: 8 total, 8 up, 7 in 2024-09-18T08:19:04.980 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:19:04 smithi007 ceph-mon[29459]: from='client.? 172.21.15.7:0/1631680924' entity='client.admin' cmd={"prefix": "osd reweight", "id": 2, "weight": 0.12225889780248589} : dispatch 2024-09-18T08:19:04.980 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:19:04 smithi007 ceph-mon[29459]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 2, "weight": 0.12225889780248589} : dispatch 2024-09-18T08:19:04.980 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:19:04 smithi007 ceph-mon[29459]: from='mgr.24470 172.21.15.7:0/2594390261' entity='mgr.y' 2024-09-18T08:19:04.981 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:19:04 smithi007 ceph-mon[29459]: from='mgr.24470 172.21.15.7:0/2594390261' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-18T08:19:04.981 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:19:04 smithi007 ceph-mon[29459]: from='client.? 172.21.15.7:0/3830870973' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-18T08:19:04.981 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:19:04 smithi007 ceph-mon[39249]: 6.12 scrub starts 2024-09-18T08:19:04.981 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:19:04 smithi007 ceph-mon[39249]: 6.12 scrub ok 2024-09-18T08:19:04.981 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:19:04 smithi007 ceph-mon[39249]: purged_snaps scrub starts 2024-09-18T08:19:04.981 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:19:04 smithi007 ceph-mon[39249]: purged_snaps scrub ok 2024-09-18T08:19:04.981 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:19:04 smithi007 ceph-mon[39249]: 5.1b scrub starts 2024-09-18T08:19:04.981 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:19:04 smithi007 ceph-mon[39249]: 5.1b scrub ok 2024-09-18T08:19:04.981 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:19:04 smithi007 ceph-mon[39249]: osdmap e412: 8 total, 8 up, 7 in 2024-09-18T08:19:04.981 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:19:04 smithi007 ceph-mon[39249]: from='client.? 172.21.15.7:0/1631680924' entity='client.admin' cmd={"prefix": "osd reweight", "id": 2, "weight": 0.12225889780248589} : dispatch 2024-09-18T08:19:04.981 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:19:04 smithi007 ceph-mon[39249]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 2, "weight": 0.12225889780248589} : dispatch 2024-09-18T08:19:04.981 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:19:04 smithi007 ceph-mon[39249]: from='mgr.24470 172.21.15.7:0/2594390261' entity='mgr.y' 2024-09-18T08:19:04.982 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:19:04 smithi007 ceph-mon[39249]: from='mgr.24470 172.21.15.7:0/2594390261' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-18T08:19:04.982 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:19:04 smithi007 ceph-mon[39249]: from='client.? 172.21.15.7:0/3830870973' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-18T08:19:05.036 INFO:tasks.radosbench.radosbench.0.smithi007.stdout: 30 16 51092 51076 106.386 64.375 0.00234585 0.0093358 2024-09-18T08:19:05.053 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2 2024-09-18T08:19:05.076 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:19:05.077 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:19:05.077 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2' 2024-09-18T08:19:05.155 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:19:04 smithi050 ceph-mon[32371]: 6.12 scrub starts 2024-09-18T08:19:05.155 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:19:04 smithi050 ceph-mon[32371]: 6.12 scrub ok 2024-09-18T08:19:05.155 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:19:04 smithi050 ceph-mon[32371]: purged_snaps scrub starts 2024-09-18T08:19:05.155 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:19:04 smithi050 ceph-mon[32371]: purged_snaps scrub ok 2024-09-18T08:19:05.155 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:19:04 smithi050 ceph-mon[32371]: 5.1b scrub starts 2024-09-18T08:19:05.155 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:19:04 smithi050 ceph-mon[32371]: 5.1b scrub ok 2024-09-18T08:19:05.156 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:19:04 smithi050 ceph-mon[32371]: osdmap e412: 8 total, 8 up, 7 in 2024-09-18T08:19:05.156 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:19:04 smithi050 ceph-mon[32371]: from='client.? 172.21.15.7:0/1631680924' entity='client.admin' cmd={"prefix": "osd reweight", "id": 2, "weight": 0.12225889780248589} : dispatch 2024-09-18T08:19:05.156 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:19:04 smithi050 ceph-mon[32371]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 2, "weight": 0.12225889780248589} : dispatch 2024-09-18T08:19:05.156 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:19:04 smithi050 ceph-mon[32371]: from='mgr.24470 172.21.15.7:0/2594390261' entity='mgr.y' 2024-09-18T08:19:05.156 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:19:04 smithi050 ceph-mon[32371]: from='mgr.24470 172.21.15.7:0/2594390261' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-18T08:19:05.156 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:19:04 smithi050 ceph-mon[32371]: from='client.? 172.21.15.7:0/3830870973' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-18T08:19:05.177 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6 2024-09-18T08:19:05.201 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:19:05.201 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:19:05.202 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6' 2024-09-18T08:19:05.303 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1 2024-09-18T08:19:05.327 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:19:05.327 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:19:05.327 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1' 2024-09-18T08:19:05.428 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4 2024-09-18T08:19:05.453 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:19:05.453 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:19:05.454 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4' 2024-09-18T08:19:05.555 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4 2024-09-18T08:19:05.580 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:19:05.581 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:19:05.581 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4' 2024-09-18T08:19:05.682 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4 2024-09-18T08:19:05.707 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:19:05.707 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:19:05.707 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4' 2024-09-18T08:19:05.808 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6 2024-09-18T08:19:05.836 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:19:05.837 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:19:05.838 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6' 2024-09-18T08:19:05.938 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6 2024-09-18T08:19:05.961 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:19:05.962 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:19:05.962 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6' 2024-09-18T08:19:06.037 INFO:tasks.radosbench.radosbench.0.smithi007.stdout: 31 16 51092 51076 102.955 0 - 0.0093358 2024-09-18T08:19:06.062 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6 2024-09-18T08:19:06.085 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:19:06.085 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:19:06.086 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6' 2024-09-18T08:19:06.155 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:19:05 smithi050 ceph-mon[32371]: 6.a scrub starts 2024-09-18T08:19:06.155 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:19:05 smithi050 ceph-mon[32371]: 6.a scrub ok 2024-09-18T08:19:06.155 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:19:05 smithi050 ceph-mon[32371]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 2, "weight": 0.12225889780248589}]': finished 2024-09-18T08:19:06.155 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:19:05 smithi050 ceph-mon[32371]: osdmap e413: 8 total, 8 up, 7 in 2024-09-18T08:19:06.155 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:19:05 smithi050 ceph-mon[32371]: 5.4 scrub starts 2024-09-18T08:19:06.155 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:19:05 smithi050 ceph-mon[32371]: 5.4 scrub ok 2024-09-18T08:19:06.156 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:19:05 smithi050 ceph-mon[32371]: pgmap v817: 158 pgs: 158 active+clean; 3.0 GiB data, 13 GiB used, 613 GiB / 626 GiB avail; 852 B/s rd, 72 MiB/s wr, 1.15k op/s 2024-09-18T08:19:06.156 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:19:05 smithi050 ceph-mon[32371]: osdmap e414: 8 total, 8 up, 7 in 2024-09-18T08:19:06.187 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6 2024-09-18T08:19:06.210 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:19:06.211 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:19:06.211 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6' 2024-09-18T08:19:06.230 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:19:05 smithi007 ceph-mon[29459]: 6.a scrub starts 2024-09-18T08:19:06.230 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:19:05 smithi007 ceph-mon[29459]: 6.a scrub ok 2024-09-18T08:19:06.230 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:19:05 smithi007 ceph-mon[29459]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 2, "weight": 0.12225889780248589}]': finished 2024-09-18T08:19:06.230 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:19:05 smithi007 ceph-mon[29459]: osdmap e413: 8 total, 8 up, 7 in 2024-09-18T08:19:06.230 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:19:05 smithi007 ceph-mon[29459]: 5.4 scrub starts 2024-09-18T08:19:06.230 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:19:05 smithi007 ceph-mon[29459]: 5.4 scrub ok 2024-09-18T08:19:06.230 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:19:05 smithi007 ceph-mon[29459]: pgmap v817: 158 pgs: 158 active+clean; 3.0 GiB data, 13 GiB used, 613 GiB / 626 GiB avail; 852 B/s rd, 72 MiB/s wr, 1.15k op/s 2024-09-18T08:19:06.230 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:19:05 smithi007 ceph-mon[29459]: osdmap e414: 8 total, 8 up, 7 in 2024-09-18T08:19:06.230 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:19:05 smithi007 ceph-mon[39249]: 6.a scrub starts 2024-09-18T08:19:06.231 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:19:05 smithi007 ceph-mon[39249]: 6.a scrub ok 2024-09-18T08:19:06.231 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:19:05 smithi007 ceph-mon[39249]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 2, "weight": 0.12225889780248589}]': finished 2024-09-18T08:19:06.231 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:19:05 smithi007 ceph-mon[39249]: osdmap e413: 8 total, 8 up, 7 in 2024-09-18T08:19:06.231 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:19:05 smithi007 ceph-mon[39249]: 5.4 scrub starts 2024-09-18T08:19:06.231 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:19:05 smithi007 ceph-mon[39249]: 5.4 scrub ok 2024-09-18T08:19:06.231 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:19:05 smithi007 ceph-mon[39249]: pgmap v817: 158 pgs: 158 active+clean; 3.0 GiB data, 13 GiB used, 613 GiB / 626 GiB avail; 852 B/s rd, 72 MiB/s wr, 1.15k op/s 2024-09-18T08:19:06.231 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:19:05 smithi007 ceph-mon[39249]: osdmap e414: 8 total, 8 up, 7 in 2024-09-18T08:19:06.312 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4 2024-09-18T08:19:06.334 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:19:06.335 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:19:06.335 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4' 2024-09-18T08:19:06.436 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0 2024-09-18T08:19:06.464 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:19:06.464 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:19:06.464 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0' 2024-09-18T08:19:06.565 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4 2024-09-18T08:19:06.590 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:19:06.590 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:19:06.591 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4' 2024-09-18T08:19:06.691 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0 2024-09-18T08:19:06.719 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:19:06.720 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:19:06.721 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0' 2024-09-18T08:19:06.822 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6 2024-09-18T08:19:06.850 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:19:06.851 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:19:06.851 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6' 2024-09-18T08:19:06.952 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7 2024-09-18T08:19:06.979 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Sep 18 08:19:06 smithi007 ceph-779349ca-7593-11ef-bceb-c7b262605968-mgr-y[29688]: ::ffff:172.21.15.50 - - [18/Sep/2024:08:19:06] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-09-18T08:19:06.986 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:19:06.987 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:19:06.988 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7' 2024-09-18T08:19:07.036 INFO:tasks.radosbench.radosbench.0.smithi007.stdout: 32 16 51128 51112 99.8078 1.125 0.0118112 0.00966479 2024-09-18T08:19:07.089 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3 2024-09-18T08:19:07.126 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:19:07.126 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:19:07.127 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3' 2024-09-18T08:19:07.227 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3 2024-09-18T08:19:07.272 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:19:07.272 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:19:07.273 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3' 2024-09-18T08:19:07.374 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0 2024-09-18T08:19:07.415 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:19:07.416 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:19:07.416 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0' 2024-09-18T08:19:07.517 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2 2024-09-18T08:19:07.559 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:19:07.560 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:19:07.561 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2' 2024-09-18T08:19:07.661 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3 2024-09-18T08:19:07.702 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:19:07.703 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:19:07.703 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3' 2024-09-18T08:19:07.804 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6 2024-09-18T08:19:07.836 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:19:07.836 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:19:07.836 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6' 2024-09-18T08:19:07.938 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0 2024-09-18T08:19:07.977 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:19:07.978 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:19:07.978 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0' 2024-09-18T08:19:08.036 INFO:tasks.radosbench.radosbench.0.smithi007.stdout: 33 16 51997 51981 98.4292 54.3125 0.00507632 0.0101516 2024-09-18T08:19:08.079 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0 2024-09-18T08:19:08.126 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:19:07 smithi007 ceph-mon[29459]: 6.2 scrub starts 2024-09-18T08:19:08.126 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:19:07 smithi007 ceph-mon[29459]: 6.2 scrub ok 2024-09-18T08:19:08.126 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:19:07 smithi007 ceph-mon[29459]: 5.e scrub starts 2024-09-18T08:19:08.126 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:19:07 smithi007 ceph-mon[29459]: 5.e scrub ok 2024-09-18T08:19:08.126 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:19:07 smithi007 ceph-mon[29459]: osdmap e415: 8 total, 8 up, 7 in 2024-09-18T08:19:08.126 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:19:07 smithi007 ceph-mon[29459]: pgmap v820: 158 pgs: 158 active+clean; 3.0 GiB data, 13 GiB used, 613 GiB / 626 GiB avail; 74 MiB/s wr, 1.18k op/s 2024-09-18T08:19:08.127 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:19:07 smithi007 ceph-mon[39249]: 6.2 scrub starts 2024-09-18T08:19:08.127 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:19:07 smithi007 ceph-mon[39249]: 6.2 scrub ok 2024-09-18T08:19:08.127 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:19:07 smithi007 ceph-mon[39249]: 5.e scrub starts 2024-09-18T08:19:08.127 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:19:07 smithi007 ceph-mon[39249]: 5.e scrub ok 2024-09-18T08:19:08.127 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:19:07 smithi007 ceph-mon[39249]: osdmap e415: 8 total, 8 up, 7 in 2024-09-18T08:19:08.127 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:19:07 smithi007 ceph-mon[39249]: pgmap v820: 158 pgs: 158 active+clean; 3.0 GiB data, 13 GiB used, 613 GiB / 626 GiB avail; 74 MiB/s wr, 1.18k op/s 2024-09-18T08:19:08.132 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:19:08.133 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:19:08.134 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0' 2024-09-18T08:19:08.155 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:19:07 smithi050 ceph-mon[32371]: 6.2 scrub starts 2024-09-18T08:19:08.155 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:19:07 smithi050 ceph-mon[32371]: 6.2 scrub ok 2024-09-18T08:19:08.155 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:19:07 smithi050 ceph-mon[32371]: 5.e scrub starts 2024-09-18T08:19:08.155 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:19:07 smithi050 ceph-mon[32371]: 5.e scrub ok 2024-09-18T08:19:08.155 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:19:07 smithi050 ceph-mon[32371]: osdmap e415: 8 total, 8 up, 7 in 2024-09-18T08:19:08.155 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:19:07 smithi050 ceph-mon[32371]: pgmap v820: 158 pgs: 158 active+clean; 3.0 GiB data, 13 GiB used, 613 GiB / 626 GiB avail; 74 MiB/s wr, 1.18k op/s 2024-09-18T08:19:08.235 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0 2024-09-18T08:19:08.273 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:19:08.274 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:19:08.274 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0' 2024-09-18T08:19:08.375 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0 2024-09-18T08:19:08.397 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:19:08.397 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:19:08.398 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0' 2024-09-18T08:19:08.480 INFO:journalctl@ceph.rgw.foo.a.smithi007.stdout:Sep 18 08:19:08 smithi007 ceph-779349ca-7593-11ef-bceb-c7b262605968-rgw-foo-a[63271]: 2024-09-18T08:19:08.123+0000 7fdec1ad7640 -1 rgw watcher librados: RGWWatcher::handle_error cookie 94700928703232 err (110) Connection timed out 2024-09-18T08:19:08.498 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0 2024-09-18T08:19:08.548 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:19:08.549 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:19:08.549 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0' 2024-09-18T08:19:08.650 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6 2024-09-18T08:19:08.678 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:19:08.679 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:19:08.680 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6' 2024-09-18T08:19:08.780 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3 2024-09-18T08:19:08.816 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:19:08.816 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:19:08.817 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3' 2024-09-18T08:19:08.918 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7 2024-09-18T08:19:08.950 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:19:08.951 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:19:08.952 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7' 2024-09-18T08:19:09.037 INFO:tasks.radosbench.radosbench.0.smithi007.stdout: 34 16 52883 52867 97.1629 55.375 0.0254391 0.0102856 2024-09-18T08:19:09.052 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1 2024-09-18T08:19:09.095 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:19:09.096 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:19:09.096 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1' 2024-09-18T08:19:09.155 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:19:08 smithi050 ceph-mon[32371]: 4.7 scrub starts 2024-09-18T08:19:09.155 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:19:08 smithi050 ceph-mon[32371]: 4.7 scrub ok 2024-09-18T08:19:09.197 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4 2024-09-18T08:19:09.225 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:19:09.226 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:19:09.226 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4' 2024-09-18T08:19:09.230 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:19:08 smithi007 ceph-mon[29459]: 4.7 scrub starts 2024-09-18T08:19:09.230 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:19:08 smithi007 ceph-mon[29459]: 4.7 scrub ok 2024-09-18T08:19:09.230 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:19:08 smithi007 ceph-mon[39249]: 4.7 scrub starts 2024-09-18T08:19:09.230 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:19:08 smithi007 ceph-mon[39249]: 4.7 scrub ok 2024-09-18T08:19:09.327 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7 2024-09-18T08:19:09.356 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:19:09.357 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:19:09.357 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7' 2024-09-18T08:19:09.373 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid 779349ca-7593-11ef-bceb-c7b262605968 -- ceph daemon osd.1 dump_historic_ops 2024-09-18T08:19:09.457 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2 2024-09-18T08:19:09.525 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:19:09.526 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:19:09.526 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2' 2024-09-18T08:19:09.627 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1 2024-09-18T08:19:09.665 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:19:09.666 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:19:09.666 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1' 2024-09-18T08:19:09.767 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4 2024-09-18T08:19:09.789 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:19:09.790 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:19:09.790 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4' 2024-09-18T08:19:09.891 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5 2024-09-18T08:19:09.920 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:19:09.921 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:19:09.921 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5' 2024-09-18T08:19:10.022 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6 2024-09-18T08:19:10.036 INFO:tasks.radosbench.radosbench.0.smithi007.stdout: 35 16 53678 53662 95.8065 49.6875 0.00866967 0.010432 2024-09-18T08:19:10.052 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:19:10.053 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:19:10.054 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6' 2024-09-18T08:19:10.154 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4 2024-09-18T08:19:10.155 INFO:journalctl@ceph.iscsi.iscsi.a.smithi050.stdout:Sep 18 08:19:09 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-iscsi-iscsi-a[57782]: debug there is no tcmu-runner data available 2024-09-18T08:19:10.184 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:19:10.185 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:19:10.185 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4' 2024-09-18T08:19:10.286 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6 2024-09-18T08:19:10.313 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:19:10.313 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:19:10.313 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6' 2024-09-18T08:19:10.414 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6 2024-09-18T08:19:10.446 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:19:10.446 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:19:10.447 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6' 2024-09-18T08:19:10.548 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0 2024-09-18T08:19:10.596 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:19:10.596 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:19:10.596 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0' 2024-09-18T08:19:10.697 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3 2024-09-18T08:19:10.719 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:19:10.720 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:19:10.720 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3' 2024-09-18T08:19:10.729 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:19:10 smithi007 ceph-mon[29459]: osdmap e416: 8 total, 8 up, 7 in 2024-09-18T08:19:10.729 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:19:10 smithi007 ceph-mon[29459]: pgmap v822: 158 pgs: 6 active+recovering+undersized+remapped, 152 active+clean; 3.2 GiB data, 14 GiB used, 612 GiB / 626 GiB avail; 1.3 KiB/s rd, 39 MiB/s wr, 618 op/s; 18869/158070 objects misplaced (11.937%); 3.5 MiB/s, 64 objects/s recovering 2024-09-18T08:19:10.730 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:19:10 smithi007 ceph-mon[29459]: purged_snaps scrub starts 2024-09-18T08:19:10.730 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:19:10 smithi007 ceph-mon[29459]: purged_snaps scrub ok 2024-09-18T08:19:10.730 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:19:10 smithi007 ceph-mon[29459]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-18T08:19:10.730 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:19:10 smithi007 ceph-mon[39249]: osdmap e416: 8 total, 8 up, 7 in 2024-09-18T08:19:10.730 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:19:10 smithi007 ceph-mon[39249]: pgmap v822: 158 pgs: 6 active+recovering+undersized+remapped, 152 active+clean; 3.2 GiB data, 14 GiB used, 612 GiB / 626 GiB avail; 1.3 KiB/s rd, 39 MiB/s wr, 618 op/s; 18869/158070 objects misplaced (11.937%); 3.5 MiB/s, 64 objects/s recovering 2024-09-18T08:19:10.730 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:19:10 smithi007 ceph-mon[39249]: purged_snaps scrub starts 2024-09-18T08:19:10.730 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:19:10 smithi007 ceph-mon[39249]: purged_snaps scrub ok 2024-09-18T08:19:10.730 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:19:10 smithi007 ceph-mon[39249]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-18T08:19:10.821 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2 2024-09-18T08:19:10.860 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:19:10.861 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:19:10.861 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2' 2024-09-18T08:19:10.905 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:19:10 smithi050 ceph-mon[32371]: osdmap e416: 8 total, 8 up, 7 in 2024-09-18T08:19:10.905 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:19:10 smithi050 ceph-mon[32371]: pgmap v822: 158 pgs: 6 active+recovering+undersized+remapped, 152 active+clean; 3.2 GiB data, 14 GiB used, 612 GiB / 626 GiB avail; 1.3 KiB/s rd, 39 MiB/s wr, 618 op/s; 18869/158070 objects misplaced (11.937%); 3.5 MiB/s, 64 objects/s recovering 2024-09-18T08:19:10.905 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:19:10 smithi050 ceph-mon[32371]: purged_snaps scrub starts 2024-09-18T08:19:10.905 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:19:10 smithi050 ceph-mon[32371]: purged_snaps scrub ok 2024-09-18T08:19:10.905 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:19:10 smithi050 ceph-mon[32371]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-18T08:19:10.962 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6 2024-09-18T08:19:10.991 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:19:10.992 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:19:10.992 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6' 2024-09-18T08:19:11.037 INFO:tasks.radosbench.radosbench.0.smithi007.stdout: 36 16 54410 54394 94.4159 45.75 0.0121234 0.0105687 2024-09-18T08:19:11.093 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5 2024-09-18T08:19:11.122 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:19:11.123 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:19:11.124 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5' 2024-09-18T08:19:11.224 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4 2024-09-18T08:19:11.253 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:19:11.254 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:19:11.254 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4' 2024-09-18T08:19:11.355 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1 2024-09-18T08:19:11.396 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:19:11.397 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:19:11.398 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1' 2024-09-18T08:19:11.498 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3 2024-09-18T08:19:11.541 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:19:11.542 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:19:11.542 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3' 2024-09-18T08:19:11.643 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1 2024-09-18T08:19:11.689 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:19:11.689 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:19:11.690 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1' 2024-09-18T08:19:11.729 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:19:11 smithi007 ceph-mon[29459]: osdmap e417: 8 total, 8 up, 7 in 2024-09-18T08:19:11.730 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:19:11 smithi007 ceph-mon[39249]: osdmap e417: 8 total, 8 up, 7 in 2024-09-18T08:19:11.791 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2 2024-09-18T08:19:11.841 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:19:11.842 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:19:11.842 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2' 2024-09-18T08:19:11.905 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:19:11 smithi050 ceph-mon[32371]: osdmap e417: 8 total, 8 up, 7 in 2024-09-18T08:19:11.943 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1 2024-09-18T08:19:11.971 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:19:11.971 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:19:11.971 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1' 2024-09-18T08:19:12.037 INFO:tasks.radosbench.radosbench.0.smithi007.stdout: 37 16 55180 55164 93.1648 48.125 0.00788519 0.0106839 2024-09-18T08:19:12.072 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3 2024-09-18T08:19:12.114 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:19:12.114 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:19:12.115 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3' 2024-09-18T08:19:12.215 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4 2024-09-18T08:19:12.248 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:19:12.248 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:19:12.249 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4' 2024-09-18T08:19:12.350 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6 2024-09-18T08:19:12.379 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:19:12.380 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:19:12.380 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6' 2024-09-18T08:19:12.481 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7 2024-09-18T08:19:12.506 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:19:12.507 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:19:12.507 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7' 2024-09-18T08:19:12.608 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3 2024-09-18T08:19:12.647 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:19:12.648 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:19:12.648 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3' 2024-09-18T08:19:12.730 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:19:12 smithi007 ceph-mon[29459]: pgmap v824: 158 pgs: 6 active+recovering+undersized+remapped, 152 active+clean; 3.2 GiB data, 14 GiB used, 612 GiB / 626 GiB avail; 1.1 KiB/s rd, 32 MiB/s wr, 509 op/s; 18869/158070 objects misplaced (11.937%); 2.9 MiB/s, 52 objects/s recovering 2024-09-18T08:19:12.730 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:19:12 smithi007 ceph-mon[39249]: pgmap v824: 158 pgs: 6 active+recovering+undersized+remapped, 152 active+clean; 3.2 GiB data, 14 GiB used, 612 GiB / 626 GiB avail; 1.1 KiB/s rd, 32 MiB/s wr, 509 op/s; 18869/158070 objects misplaced (11.937%); 2.9 MiB/s, 52 objects/s recovering 2024-09-18T08:19:12.749 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2 2024-09-18T08:19:12.791 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:19:12.792 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:19:12.792 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2' 2024-09-18T08:19:12.893 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6 2024-09-18T08:19:12.905 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:19:12 smithi050 ceph-mon[32371]: pgmap v824: 158 pgs: 6 active+recovering+undersized+remapped, 152 active+clean; 3.2 GiB data, 14 GiB used, 612 GiB / 626 GiB avail; 1.1 KiB/s rd, 32 MiB/s wr, 509 op/s; 18869/158070 objects misplaced (11.937%); 2.9 MiB/s, 52 objects/s recovering 2024-09-18T08:19:12.921 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:19:12.921 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:19:12.922 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6' 2024-09-18T08:19:13.022 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0 2024-09-18T08:19:13.037 INFO:tasks.radosbench.radosbench.0.smithi007.stdout: 38 16 55972 55956 92.0152 49.5 0.0235509 0.010861 2024-09-18T08:19:13.069 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:19:13.070 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:19:13.070 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0' 2024-09-18T08:19:13.171 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0 2024-09-18T08:19:13.216 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:19:13.217 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:19:13.220 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0' 2024-09-18T08:19:13.321 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3 2024-09-18T08:19:13.358 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:19:13.360 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:19:13.361 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3' 2024-09-18T08:19:13.461 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3 2024-09-18T08:19:13.492 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:19:13.493 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:19:13.493 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3' 2024-09-18T08:19:13.594 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6 2024-09-18T08:19:13.625 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:19:13.625 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:19:13.626 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6' 2024-09-18T08:19:13.726 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0 2024-09-18T08:19:13.767 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:19:13.768 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:19:13.768 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0' 2024-09-18T08:19:13.869 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2 2024-09-18T08:19:13.895 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:19:13.895 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:19:13.896 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2' 2024-09-18T08:19:13.905 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:19:13 smithi050 ceph-mon[32371]: 4.18 scrub starts 2024-09-18T08:19:13.905 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:19:13 smithi050 ceph-mon[32371]: 4.18 scrub ok 2024-09-18T08:19:13.980 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:19:13 smithi007 ceph-mon[29459]: 4.18 scrub starts 2024-09-18T08:19:13.980 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:19:13 smithi007 ceph-mon[29459]: 4.18 scrub ok 2024-09-18T08:19:13.980 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:19:13 smithi007 ceph-mon[39249]: 4.18 scrub starts 2024-09-18T08:19:13.980 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:19:13 smithi007 ceph-mon[39249]: 4.18 scrub ok 2024-09-18T08:19:13.997 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2 2024-09-18T08:19:14.037 INFO:tasks.radosbench.radosbench.0.smithi007.stdout: 39 16 56711 56695 90.8401 46.1875 0.014914 0.0110021 2024-09-18T08:19:14.045 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:19:14.046 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:19:14.046 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2' 2024-09-18T08:19:14.147 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6 2024-09-18T08:19:14.176 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:19:14.176 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:19:14.177 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6' 2024-09-18T08:19:14.278 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2 2024-09-18T08:19:14.319 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:19:14.320 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:19:14.320 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2' 2024-09-18T08:19:14.421 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7 2024-09-18T08:19:14.453 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:19:14.453 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:19:14.454 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7' 2024-09-18T08:19:14.554 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4 2024-09-18T08:19:14.581 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:19:14.582 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:19:14.583 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4' 2024-09-18T08:19:14.683 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7 2024-09-18T08:19:14.720 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:19:14.721 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:19:14.722 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7' 2024-09-18T08:19:14.822 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5 2024-09-18T08:19:14.854 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:19:14.855 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:19:14.855 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5' 2024-09-18T08:19:14.905 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:19:14 smithi050 ceph-mon[32371]: pgmap v825: 158 pgs: 6 active+recovering+undersized+remapped, 152 active+clean; 3.3 GiB data, 14 GiB used, 612 GiB / 626 GiB avail; 1.6 KiB/s rd, 41 MiB/s wr, 651 op/s; 18141/162435 objects misplaced (11.168%); 9.3 MiB/s, 154 objects/s recovering 2024-09-18T08:19:14.905 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:19:14 smithi050 ceph-mon[32371]: osdmap e418: 8 total, 8 up, 7 in 2024-09-18T08:19:14.956 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6 2024-09-18T08:19:14.979 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:19:14 smithi007 ceph-mon[29459]: pgmap v825: 158 pgs: 6 active+recovering+undersized+remapped, 152 active+clean; 3.3 GiB data, 14 GiB used, 612 GiB / 626 GiB avail; 1.6 KiB/s rd, 41 MiB/s wr, 651 op/s; 18141/162435 objects misplaced (11.168%); 9.3 MiB/s, 154 objects/s recovering 2024-09-18T08:19:14.980 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:19:14 smithi007 ceph-mon[29459]: osdmap e418: 8 total, 8 up, 7 in 2024-09-18T08:19:14.980 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:19:14 smithi007 ceph-mon[39249]: pgmap v825: 158 pgs: 6 active+recovering+undersized+remapped, 152 active+clean; 3.3 GiB data, 14 GiB used, 612 GiB / 626 GiB avail; 1.6 KiB/s rd, 41 MiB/s wr, 651 op/s; 18141/162435 objects misplaced (11.168%); 9.3 MiB/s, 154 objects/s recovering 2024-09-18T08:19:14.980 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:19:14 smithi007 ceph-mon[39249]: osdmap e418: 8 total, 8 up, 7 in 2024-09-18T08:19:14.985 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:19:14.986 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:19:14.987 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6' 2024-09-18T08:19:15.037 INFO:tasks.radosbench.radosbench.0.smithi007.stdout:2024-09-18T08:19:15.037200+0000 min lat: 0.00108817 max lat: 2.21973 avg lat: 0.0111146 2024-09-18T08:19:15.037 INFO:tasks.radosbench.radosbench.0.smithi007.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-09-18T08:19:15.037 INFO:tasks.radosbench.radosbench.0.smithi007.stdout: 40 16 57570 57554 89.9113 53.6875 0.0236408 0.0111146 2024-09-18T08:19:15.088 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2 2024-09-18T08:19:15.132 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:19:15.132 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:19:15.133 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2' 2024-09-18T08:19:15.234 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5 2024-09-18T08:19:15.263 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:19:15.264 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:19:15.264 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5' 2024-09-18T08:19:15.365 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6 2024-09-18T08:19:15.395 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:19:15.396 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:19:15.397 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6' 2024-09-18T08:19:15.498 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2 2024-09-18T08:19:15.551 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:19:15.551 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:19:15.552 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2' 2024-09-18T08:19:15.652 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7 2024-09-18T08:19:15.685 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:19:15.686 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:19:15.686 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7' 2024-09-18T08:19:15.787 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2 2024-09-18T08:19:15.827 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:19:15.828 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:19:15.828 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2' 2024-09-18T08:19:15.929 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2 2024-09-18T08:19:15.968 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:19:15.969 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:19:15.969 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2' 2024-09-18T08:19:16.037 INFO:tasks.radosbench.radosbench.0.smithi007.stdout: 41 16 58426 58410 89.0231 53.5 0.0185313 0.0112246 2024-09-18T08:19:16.070 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7 2024-09-18T08:19:16.102 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:19:16.103 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:19:16.103 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7' 2024-09-18T08:19:16.204 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0 2024-09-18T08:19:16.245 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:19:16.246 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:19:16.246 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0' 2024-09-18T08:19:16.347 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2 2024-09-18T08:19:16.370 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:19:16.370 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:19:16.370 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2' 2024-09-18T08:19:16.471 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0 2024-09-18T08:19:16.514 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:19:16.515 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:19:16.515 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0' 2024-09-18T08:19:16.616 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4 2024-09-18T08:19:16.645 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:19:16.645 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:19:16.646 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4' 2024-09-18T08:19:16.746 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7 2024-09-18T08:19:16.777 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:19:16.778 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:19:16.778 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7' 2024-09-18T08:19:16.879 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6 2024-09-18T08:19:16.905 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:19:16 smithi050 ceph-mon[32371]: pgmap v827: 158 pgs: 6 active+recovering+undersized+remapped, 152 active+clean; 3.4 GiB data, 14 GiB used, 611 GiB / 626 GiB avail; 849 B/s rd, 51 MiB/s wr, 811 op/s; 17804/169752 objects misplaced (10.488%); 12 MiB/s, 196 objects/s recovering 2024-09-18T08:19:16.909 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:19:16.909 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:19:16.909 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6' 2024-09-18T08:19:16.979 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:19:16 smithi007 ceph-mon[29459]: pgmap v827: 158 pgs: 6 active+recovering+undersized+remapped, 152 active+clean; 3.4 GiB data, 14 GiB used, 611 GiB / 626 GiB avail; 849 B/s rd, 51 MiB/s wr, 811 op/s; 17804/169752 objects misplaced (10.488%); 12 MiB/s, 196 objects/s recovering 2024-09-18T08:19:16.980 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Sep 18 08:19:16 smithi007 ceph-779349ca-7593-11ef-bceb-c7b262605968-mgr-y[29688]: ::ffff:172.21.15.50 - - [18/Sep/2024:08:19:16] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-09-18T08:19:16.980 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:19:16 smithi007 ceph-mon[39249]: pgmap v827: 158 pgs: 6 active+recovering+undersized+remapped, 152 active+clean; 3.4 GiB data, 14 GiB used, 611 GiB / 626 GiB avail; 849 B/s rd, 51 MiB/s wr, 811 op/s; 17804/169752 objects misplaced (10.488%); 12 MiB/s, 196 objects/s recovering 2024-09-18T08:19:17.010 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3 2024-09-18T08:19:17.033 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:19:17.033 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:19:17.033 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3' 2024-09-18T08:19:17.037 INFO:tasks.radosbench.radosbench.0.smithi007.stdout: 42 16 59171 59155 88.0122 46.5625 0.0269276 0.0113308 2024-09-18T08:19:17.134 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5 2024-09-18T08:19:17.165 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:19:17.166 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:19:17.166 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5' 2024-09-18T08:19:17.267 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2 2024-09-18T08:19:17.302 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:19:17.303 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:19:17.304 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2' 2024-09-18T08:19:17.404 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7 2024-09-18T08:19:17.433 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:19:17.434 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:19:17.434 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7' 2024-09-18T08:19:17.535 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0 2024-09-18T08:19:17.581 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:19:17.581 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:19:17.582 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0' 2024-09-18T08:19:17.683 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0 2024-09-18T08:19:17.725 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:19:17.725 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:19:17.726 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0' 2024-09-18T08:19:17.826 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5 2024-09-18T08:19:17.859 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:19:17.860 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:19:17.860 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5' 2024-09-18T08:19:17.961 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4 2024-09-18T08:19:17.984 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:19:17.985 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:19:17.985 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4' 2024-09-18T08:19:18.037 INFO:tasks.radosbench.radosbench.0.smithi007.stdout: 43 16 59989 59973 87.1543 51.125 0.00867659 0.0114667 2024-09-18T08:19:18.086 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4 2024-09-18T08:19:18.124 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:19:18.125 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:19:18.125 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4' 2024-09-18T08:19:18.226 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1 2024-09-18T08:19:18.248 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:19:18.249 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:19:18.249 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1' 2024-09-18T08:19:18.350 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6 2024-09-18T08:19:18.382 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:19:18.382 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:19:18.383 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6' 2024-09-18T08:19:18.484 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1 2024-09-18T08:19:18.526 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:19:18.527 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:19:18.527 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1' 2024-09-18T08:19:18.628 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6 2024-09-18T08:19:18.658 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:19:18.659 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:19:18.659 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6' 2024-09-18T08:19:18.760 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5 2024-09-18T08:19:18.792 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:19:18.793 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:19:18.793 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5' 2024-09-18T08:19:18.894 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6 2024-09-18T08:19:18.905 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:19:18 smithi050 ceph-mon[32371]: pgmap v828: 158 pgs: 6 active+recovering+undersized+remapped, 152 active+clean; 3.4 GiB data, 14 GiB used, 611 GiB / 626 GiB avail; 639 B/s rd, 30 MiB/s wr, 487 op/s; 17804/169752 objects misplaced (10.488%); 8.3 MiB/s, 133 objects/s recovering 2024-09-18T08:19:18.926 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:19:18.927 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:19:18.927 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6' 2024-09-18T08:19:18.980 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:19:18 smithi007 ceph-mon[29459]: pgmap v828: 158 pgs: 6 active+recovering+undersized+remapped, 152 active+clean; 3.4 GiB data, 14 GiB used, 611 GiB / 626 GiB avail; 639 B/s rd, 30 MiB/s wr, 487 op/s; 17804/169752 objects misplaced (10.488%); 8.3 MiB/s, 133 objects/s recovering 2024-09-18T08:19:18.980 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:19:18 smithi007 ceph-mon[39249]: pgmap v828: 158 pgs: 6 active+recovering+undersized+remapped, 152 active+clean; 3.4 GiB data, 14 GiB used, 611 GiB / 626 GiB avail; 639 B/s rd, 30 MiB/s wr, 487 op/s; 17804/169752 objects misplaced (10.488%); 8.3 MiB/s, 133 objects/s recovering 2024-09-18T08:19:19.028 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6 2024-09-18T08:19:19.038 INFO:tasks.radosbench.radosbench.0.smithi007.stdout: 44 16 60768 60752 86.28 48.6875 0.0263772 0.0115738 2024-09-18T08:19:19.050 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:19:19.051 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:19:19.051 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6' 2024-09-18T08:19:19.152 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2 2024-09-18T08:19:19.191 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:19:19.191 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:19:19.191 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2' 2024-09-18T08:19:19.292 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4 2024-09-18T08:19:19.320 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:19:19.321 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:19:19.321 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4' 2024-09-18T08:19:19.422 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6 2024-09-18T08:19:19.445 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:19:19.446 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:19:19.446 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6' 2024-09-18T08:19:19.547 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5 2024-09-18T08:19:19.570 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:19:19.571 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:19:19.572 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5' 2024-09-18T08:19:19.672 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3 2024-09-18T08:19:19.707 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:19:19.708 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:19:19.709 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3' 2024-09-18T08:19:19.809 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2 2024-09-18T08:19:19.855 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:19:19.855 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:19:19.856 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2' 2024-09-18T08:19:19.905 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:19:19 smithi050 ceph-mon[32371]: from='mgr.24470 172.21.15.7:0/2594390261' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-18T08:19:19.905 INFO:journalctl@ceph.iscsi.iscsi.a.smithi050.stdout:Sep 18 08:19:19 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-iscsi-iscsi-a[57782]: debug there is no tcmu-runner data available 2024-09-18T08:19:19.957 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0 2024-09-18T08:19:19.979 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:19:19 smithi007 ceph-mon[39249]: from='mgr.24470 172.21.15.7:0/2594390261' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-18T08:19:19.980 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:19:19 smithi007 ceph-mon[29459]: from='mgr.24470 172.21.15.7:0/2594390261' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-18T08:19:19.996 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:19:19.997 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:19:19.998 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0' 2024-09-18T08:19:20.038 INFO:tasks.radosbench.radosbench.0.smithi007.stdout: 45 16 61301 61285 85.103 33.3125 0.0139374 0.0117438 2024-09-18T08:19:20.099 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2 2024-09-18T08:19:20.142 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:19:20.142 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:19:20.143 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2' 2024-09-18T08:19:20.244 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7 2024-09-18T08:19:20.265 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:19:20.266 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:19:20.266 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7' 2024-09-18T08:19:20.367 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7 2024-09-18T08:19:20.398 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:19:20.399 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:19:20.399 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7' 2024-09-18T08:19:20.500 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6 2024-09-18T08:19:20.529 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:19:20.530 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:19:20.530 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6' 2024-09-18T08:19:20.631 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2 2024-09-18T08:19:20.672 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:19:20.672 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:19:20.672 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2' 2024-09-18T08:19:20.773 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4 2024-09-18T08:19:20.800 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:19:20.801 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:19:20.801 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4' 2024-09-18T08:19:20.902 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0 2024-09-18T08:19:20.905 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:19:20 smithi050 ceph-mon[32371]: pgmap v829: 158 pgs: 6 active+recovering+undersized+remapped, 152 active+clean; 3.7 GiB data, 15 GiB used, 611 GiB / 626 GiB avail; 1.1 KiB/s rd, 55 MiB/s wr, 882 op/s; 16773/181881 objects misplaced (9.222%); 15 MiB/s, 232 objects/s recovering 2024-09-18T08:19:20.905 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:19:20 smithi050 ceph-mon[32371]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-18T08:19:20.945 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:19:20.946 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:19:20.947 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0' 2024-09-18T08:19:20.979 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:19:20 smithi007 ceph-mon[29459]: pgmap v829: 158 pgs: 6 active+recovering+undersized+remapped, 152 active+clean; 3.7 GiB data, 15 GiB used, 611 GiB / 626 GiB avail; 1.1 KiB/s rd, 55 MiB/s wr, 882 op/s; 16773/181881 objects misplaced (9.222%); 15 MiB/s, 232 objects/s recovering 2024-09-18T08:19:20.979 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:19:20 smithi007 ceph-mon[29459]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-18T08:19:20.980 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:19:20 smithi007 ceph-mon[39249]: pgmap v829: 158 pgs: 6 active+recovering+undersized+remapped, 152 active+clean; 3.7 GiB data, 15 GiB used, 611 GiB / 626 GiB avail; 1.1 KiB/s rd, 55 MiB/s wr, 882 op/s; 16773/181881 objects misplaced (9.222%); 15 MiB/s, 232 objects/s recovering 2024-09-18T08:19:20.980 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:19:20 smithi007 ceph-mon[39249]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-18T08:19:21.038 INFO:tasks.radosbench.radosbench.0.smithi007.stdout: 46 16 62071 62055 84.2991 48.125 0.00589108 0.0118554 2024-09-18T08:19:21.047 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4 2024-09-18T08:19:21.076 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:19:21.076 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:19:21.077 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4' 2024-09-18T08:19:21.178 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0 2024-09-18T08:19:21.217 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:19:21.218 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:19:21.218 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0' 2024-09-18T08:19:21.319 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6 2024-09-18T08:19:21.350 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:19:21.351 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:19:21.352 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6' 2024-09-18T08:19:21.452 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1 2024-09-18T08:19:21.495 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:19:21.496 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:19:21.496 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1' 2024-09-18T08:19:21.597 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5 2024-09-18T08:19:21.625 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:19:21.626 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:19:21.626 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5' 2024-09-18T08:19:21.727 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4 2024-09-18T08:19:21.761 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:19:21.762 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:19:21.762 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4' 2024-09-18T08:19:21.863 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1 2024-09-18T08:19:21.900 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:19:21.901 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:19:21.901 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1' 2024-09-18T08:19:22.002 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1 2024-09-18T08:19:22.038 INFO:tasks.radosbench.radosbench.0.smithi007.stdout: 47 16 62973 62957 83.7045 56.375 0.0166993 0.011938 2024-09-18T08:19:22.039 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:19:22.039 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:19:22.040 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1' 2024-09-18T08:19:22.140 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4 2024-09-18T08:19:22.175 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:19:22.175 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:19:22.176 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4' 2024-09-18T08:19:22.276 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7 2024-09-18T08:19:22.304 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:19:22.305 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:19:22.305 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7' 2024-09-18T08:19:22.406 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0 2024-09-18T08:19:22.457 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:19:22.458 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:19:22.458 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0' 2024-09-18T08:19:22.559 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2 2024-09-18T08:19:22.601 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:19:22.601 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:19:22.602 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2' 2024-09-18T08:19:22.703 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5 2024-09-18T08:19:22.729 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:19:22 smithi007 ceph-mon[29459]: purged_snaps scrub starts 2024-09-18T08:19:22.730 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:19:22 smithi007 ceph-mon[29459]: purged_snaps scrub ok 2024-09-18T08:19:22.730 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:19:22 smithi007 ceph-mon[29459]: pgmap v830: 158 pgs: 6 active+recovering+undersized+remapped, 152 active+clean; 3.7 GiB data, 15 GiB used, 611 GiB / 626 GiB avail; 1023 B/s rd, 50 MiB/s wr, 794 op/s; 16773/181881 objects misplaced (9.222%); 13 MiB/s, 209 objects/s recovering 2024-09-18T08:19:22.730 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:19:22 smithi007 ceph-mon[29459]: osdmap e419: 8 total, 8 up, 7 in 2024-09-18T08:19:22.730 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:19:22 smithi007 ceph-mon[39249]: purged_snaps scrub starts 2024-09-18T08:19:22.730 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:19:22 smithi007 ceph-mon[39249]: purged_snaps scrub ok 2024-09-18T08:19:22.730 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:19:22 smithi007 ceph-mon[39249]: pgmap v830: 158 pgs: 6 active+recovering+undersized+remapped, 152 active+clean; 3.7 GiB data, 15 GiB used, 611 GiB / 626 GiB avail; 1023 B/s rd, 50 MiB/s wr, 794 op/s; 16773/181881 objects misplaced (9.222%); 13 MiB/s, 209 objects/s recovering 2024-09-18T08:19:22.730 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:19:22 smithi007 ceph-mon[39249]: osdmap e419: 8 total, 8 up, 7 in 2024-09-18T08:19:22.735 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:19:22.736 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:19:22.736 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5' 2024-09-18T08:19:22.837 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0 2024-09-18T08:19:22.862 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:19:22.862 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:19:22.863 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0' 2024-09-18T08:19:22.905 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:19:22 smithi050 ceph-mon[32371]: purged_snaps scrub starts 2024-09-18T08:19:22.905 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:19:22 smithi050 ceph-mon[32371]: purged_snaps scrub ok 2024-09-18T08:19:22.905 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:19:22 smithi050 ceph-mon[32371]: pgmap v830: 158 pgs: 6 active+recovering+undersized+remapped, 152 active+clean; 3.7 GiB data, 15 GiB used, 611 GiB / 626 GiB avail; 1023 B/s rd, 50 MiB/s wr, 794 op/s; 16773/181881 objects misplaced (9.222%); 13 MiB/s, 209 objects/s recovering 2024-09-18T08:19:22.905 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:19:22 smithi050 ceph-mon[32371]: osdmap e419: 8 total, 8 up, 7 in 2024-09-18T08:19:22.963 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2 2024-09-18T08:19:23.012 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:19:23.013 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:19:23.014 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2' 2024-09-18T08:19:23.038 INFO:tasks.radosbench.radosbench.0.smithi007.stdout: 48 16 63847 63831 83.0986 54.625 0.0358117 0.0120248 2024-09-18T08:19:23.115 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1 2024-09-18T08:19:23.161 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:19:23.161 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:19:23.162 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1' 2024-09-18T08:19:23.262 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1 2024-09-18T08:19:23.300 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:19:23.300 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:19:23.301 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1' 2024-09-18T08:19:23.402 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3 2024-09-18T08:19:23.444 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:19:23.445 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:19:23.445 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3' 2024-09-18T08:19:23.546 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2 2024-09-18T08:19:23.585 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:19:23.586 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:19:23.586 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2' 2024-09-18T08:19:23.687 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1 2024-09-18T08:19:23.737 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:19:23.738 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:19:23.738 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1' 2024-09-18T08:19:23.839 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6 2024-09-18T08:19:23.862 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:19:23.863 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:19:23.863 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6' 2024-09-18T08:19:23.905 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:19:23 smithi050 ceph-mon[32371]: purged_snaps scrub starts 2024-09-18T08:19:23.905 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:19:23 smithi050 ceph-mon[32371]: purged_snaps scrub ok 2024-09-18T08:19:23.905 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:19:23 smithi050 ceph-mon[32371]: osdmap e420: 8 total, 8 up, 7 in 2024-09-18T08:19:23.965 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1 2024-09-18T08:19:23.979 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:19:23 smithi007 ceph-mon[39249]: purged_snaps scrub starts 2024-09-18T08:19:23.980 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:19:23 smithi007 ceph-mon[39249]: purged_snaps scrub ok 2024-09-18T08:19:23.980 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:19:23 smithi007 ceph-mon[39249]: osdmap e420: 8 total, 8 up, 7 in 2024-09-18T08:19:23.980 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:19:23 smithi007 ceph-mon[29459]: purged_snaps scrub starts 2024-09-18T08:19:23.981 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:19:23 smithi007 ceph-mon[29459]: purged_snaps scrub ok 2024-09-18T08:19:23.981 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:19:23 smithi007 ceph-mon[29459]: osdmap e420: 8 total, 8 up, 7 in 2024-09-18T08:19:24.001 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:19:24.002 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:19:24.002 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1' 2024-09-18T08:19:24.038 INFO:tasks.radosbench.radosbench.0.smithi007.stdout: 49 16 64631 64615 82.4027 49 0.02362 0.0121294 2024-09-18T08:19:24.104 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0 2024-09-18T08:19:24.139 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:19:24.140 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:19:24.140 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0' 2024-09-18T08:19:24.241 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4 2024-09-18T08:19:24.271 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:19:24.271 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:19:24.272 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4' 2024-09-18T08:19:24.357 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid 779349ca-7593-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-18T08:19:24.372 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4 2024-09-18T08:19:24.399 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:19:24.400 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:19:24.400 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4' 2024-09-18T08:19:24.501 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0 2024-09-18T08:19:24.552 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:19:24.552 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:19:24.553 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0' 2024-09-18T08:19:24.653 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0 2024-09-18T08:19:24.700 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:19:24.701 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:19:24.701 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0' 2024-09-18T08:19:24.730 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:19:24 smithi007 ceph-mon[39249]: purged_snaps scrub starts 2024-09-18T08:19:24.730 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:19:24 smithi007 ceph-mon[39249]: purged_snaps scrub ok 2024-09-18T08:19:24.730 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:19:24 smithi007 ceph-mon[39249]: pgmap v833: 158 pgs: 6 active+recovering+undersized+remapped, 152 active+clean; 3.8 GiB data, 16 GiB used, 610 GiB / 626 GiB avail; 1.2 KiB/s rd, 43 MiB/s wr, 689 op/s; 16059/186273 objects misplaced (8.621%); 14 MiB/s, 217 objects/s recovering 2024-09-18T08:19:24.730 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:19:24 smithi007 ceph-mon[39249]: osdmap e421: 8 total, 8 up, 7 in 2024-09-18T08:19:24.730 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:19:24 smithi007 ceph-mon[39249]: osdmap e422: 8 total, 8 up, 7 in 2024-09-18T08:19:24.731 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:19:24 smithi007 ceph-mon[29459]: purged_snaps scrub starts 2024-09-18T08:19:24.731 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:19:24 smithi007 ceph-mon[29459]: purged_snaps scrub ok 2024-09-18T08:19:24.731 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:19:24 smithi007 ceph-mon[29459]: pgmap v833: 158 pgs: 6 active+recovering+undersized+remapped, 152 active+clean; 3.8 GiB data, 16 GiB used, 610 GiB / 626 GiB avail; 1.2 KiB/s rd, 43 MiB/s wr, 689 op/s; 16059/186273 objects misplaced (8.621%); 14 MiB/s, 217 objects/s recovering 2024-09-18T08:19:24.731 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:19:24 smithi007 ceph-mon[29459]: osdmap e421: 8 total, 8 up, 7 in 2024-09-18T08:19:24.731 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:19:24 smithi007 ceph-mon[29459]: osdmap e422: 8 total, 8 up, 7 in 2024-09-18T08:19:24.802 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5 2024-09-18T08:19:24.830 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:19:24.830 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:19:24.831 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5' 2024-09-18T08:19:24.905 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:19:24 smithi050 ceph-mon[32371]: purged_snaps scrub starts 2024-09-18T08:19:24.909 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:19:24 smithi050 ceph-mon[32371]: purged_snaps scrub ok 2024-09-18T08:19:24.910 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:19:24 smithi050 ceph-mon[32371]: pgmap v833: 158 pgs: 6 active+recovering+undersized+remapped, 152 active+clean; 3.8 GiB data, 16 GiB used, 610 GiB / 626 GiB avail; 1.2 KiB/s rd, 43 MiB/s wr, 689 op/s; 16059/186273 objects misplaced (8.621%); 14 MiB/s, 217 objects/s recovering 2024-09-18T08:19:24.910 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:19:24 smithi050 ceph-mon[32371]: osdmap e421: 8 total, 8 up, 7 in 2024-09-18T08:19:24.910 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:19:24 smithi050 ceph-mon[32371]: osdmap e422: 8 total, 8 up, 7 in 2024-09-18T08:19:24.932 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3 2024-09-18T08:19:24.985 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:19:24.986 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:19:24.986 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3' 2024-09-18T08:19:25.038 INFO:tasks.radosbench.radosbench.0.smithi007.stdout: 50 16 65395 65379 81.7096 47.75 0.0118936 0.0122327 2024-09-18T08:19:25.087 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5 2024-09-18T08:19:25.119 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:19:25.120 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:19:25.120 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5' 2024-09-18T08:19:25.221 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0 2024-09-18T08:19:25.270 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:19:25.270 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:19:25.271 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0' 2024-09-18T08:19:25.372 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5 2024-09-18T08:19:25.403 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:19:25.404 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:19:25.404 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5' 2024-09-18T08:19:25.505 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0 2024-09-18T08:19:25.545 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:19:25.546 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:19:25.547 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0' 2024-09-18T08:19:25.647 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5 2024-09-18T08:19:25.675 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:19:25.675 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:19:25.676 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5' 2024-09-18T08:19:25.777 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6 2024-09-18T08:19:25.806 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:19:25.806 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:19:25.807 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6' 2024-09-18T08:19:25.907 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5 2024-09-18T08:19:25.936 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:19:25.937 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:19:25.937 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5' 2024-09-18T08:19:26.038 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5 2024-09-18T08:19:26.039 INFO:tasks.radosbench.radosbench.0.smithi007.stdout: 51 15 66219 66204 81.1184 51.5625 0.0236904 0.012322 2024-09-18T08:19:26.076 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:19:26.076 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:19:26.077 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5' 2024-09-18T08:19:26.178 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1 2024-09-18T08:19:26.230 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:19:26.230 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:19:26.231 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1' 2024-09-18T08:19:26.331 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6 2024-09-18T08:19:26.366 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:19:26.366 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:19:26.367 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6' 2024-09-18T08:19:26.467 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5 2024-09-18T08:19:26.491 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:19:26.491 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:19:26.492 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5' 2024-09-18T08:19:26.592 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7 2024-09-18T08:19:26.631 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:19:26.632 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:19:26.632 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7' 2024-09-18T08:19:26.698 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:19:26 smithi007 ceph-mon[29459]: pgmap v836: 158 pgs: 6 active+recovering+undersized+remapped, 152 active+clean; 3.9 GiB data, 16 GiB used, 610 GiB / 626 GiB avail; 1.2 KiB/s rd, 61 MiB/s wr, 976 op/s; 15648/193593 objects misplaced (8.083%); 18 MiB/s, 281 objects/s recovering 2024-09-18T08:19:26.698 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:19:26 smithi007 ceph-mon[29459]: from='mgr.24470 172.21.15.7:0/2594390261' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-18T08:19:26.698 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:19:26 smithi007 ceph-mon[29459]: from='mgr.24470 172.21.15.7:0/2594390261' entity='mgr.y' 2024-09-18T08:19:26.698 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:19:26 smithi007 ceph-mon[29459]: from='mgr.24470 172.21.15.7:0/2594390261' entity='mgr.y' 2024-09-18T08:19:26.698 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:19:26 smithi007 ceph-mon[39249]: pgmap v836: 158 pgs: 6 active+recovering+undersized+remapped, 152 active+clean; 3.9 GiB data, 16 GiB used, 610 GiB / 626 GiB avail; 1.2 KiB/s rd, 61 MiB/s wr, 976 op/s; 15648/193593 objects misplaced (8.083%); 18 MiB/s, 281 objects/s recovering 2024-09-18T08:19:26.699 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:19:26 smithi007 ceph-mon[39249]: from='mgr.24470 172.21.15.7:0/2594390261' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-18T08:19:26.699 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:19:26 smithi007 ceph-mon[39249]: from='mgr.24470 172.21.15.7:0/2594390261' entity='mgr.y' 2024-09-18T08:19:26.699 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:19:26 smithi007 ceph-mon[39249]: from='mgr.24470 172.21.15.7:0/2594390261' entity='mgr.y' 2024-09-18T08:19:26.733 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4 2024-09-18T08:19:26.756 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:19:26.757 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:19:26.757 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4' 2024-09-18T08:19:26.858 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4 2024-09-18T08:19:26.883 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:19:26.883 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:19:26.884 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4' 2024-09-18T08:19:26.905 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:19:26 smithi050 ceph-mon[32371]: pgmap v836: 158 pgs: 6 active+recovering+undersized+remapped, 152 active+clean; 3.9 GiB data, 16 GiB used, 610 GiB / 626 GiB avail; 1.2 KiB/s rd, 61 MiB/s wr, 976 op/s; 15648/193593 objects misplaced (8.083%); 18 MiB/s, 281 objects/s recovering 2024-09-18T08:19:26.905 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:19:26 smithi050 ceph-mon[32371]: from='mgr.24470 172.21.15.7:0/2594390261' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-18T08:19:26.905 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:19:26 smithi050 ceph-mon[32371]: from='mgr.24470 172.21.15.7:0/2594390261' entity='mgr.y' 2024-09-18T08:19:26.905 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:19:26 smithi050 ceph-mon[32371]: from='mgr.24470 172.21.15.7:0/2594390261' entity='mgr.y' 2024-09-18T08:19:26.980 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Sep 18 08:19:26 smithi007 ceph-779349ca-7593-11ef-bceb-c7b262605968-mgr-y[29688]: ::ffff:172.21.15.50 - - [18/Sep/2024:08:19:26] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-09-18T08:19:26.985 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5 2024-09-18T08:19:27.011 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:19:27.012 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:19:27.012 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5' 2024-09-18T08:19:27.039 INFO:tasks.radosbench.radosbench.0.smithi007.stdout: 52 16 66878 66862 80.3488 41.125 0.0297374 0.0124383 2024-09-18T08:19:27.113 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0 2024-09-18T08:19:27.136 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:19:27.137 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:19:27.137 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0' 2024-09-18T08:19:27.238 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3 2024-09-18T08:19:27.265 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:19:27.266 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:19:27.266 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3' 2024-09-18T08:19:27.367 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3 2024-09-18T08:19:27.392 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:19:27.393 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:19:27.393 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3' 2024-09-18T08:19:27.494 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4 2024-09-18T08:19:27.522 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:19:27.523 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:19:27.523 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4' 2024-09-18T08:19:27.624 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3 2024-09-18T08:19:27.663 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:19:27.663 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:19:27.664 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3' 2024-09-18T08:19:27.765 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3 2024-09-18T08:19:27.790 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:19:27.794 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:19:27.795 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3' 2024-09-18T08:19:27.896 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3 2024-09-18T08:19:27.905 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:19:27 smithi050 ceph-mon[32371]: from='mgr.24470 172.21.15.7:0/2594390261' entity='mgr.y' 2024-09-18T08:19:27.905 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:19:27 smithi050 ceph-mon[32371]: from='mgr.24470 172.21.15.7:0/2594390261' entity='mgr.y' 2024-09-18T08:19:27.942 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:19:27.943 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:19:27.943 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3' 2024-09-18T08:19:27.980 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:19:27 smithi007 ceph-mon[29459]: from='mgr.24470 172.21.15.7:0/2594390261' entity='mgr.y' 2024-09-18T08:19:27.980 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:19:27 smithi007 ceph-mon[29459]: from='mgr.24470 172.21.15.7:0/2594390261' entity='mgr.y' 2024-09-18T08:19:27.980 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:19:27 smithi007 ceph-mon[39249]: from='mgr.24470 172.21.15.7:0/2594390261' entity='mgr.y' 2024-09-18T08:19:27.980 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:19:27 smithi007 ceph-mon[39249]: from='mgr.24470 172.21.15.7:0/2594390261' entity='mgr.y' 2024-09-18T08:19:28.039 INFO:tasks.radosbench.radosbench.0.smithi007.stdout: 53 16 67407 67391 79.4567 33.0625 0.0136901 0.012577 2024-09-18T08:19:28.043 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6 2024-09-18T08:19:28.073 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:19:28.073 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:19:28.074 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6' 2024-09-18T08:19:28.175 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5 2024-09-18T08:19:28.197 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:19:28.198 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:19:28.198 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5' 2024-09-18T08:19:28.299 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6 2024-09-18T08:19:28.328 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:19:28.329 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:19:28.329 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6' 2024-09-18T08:19:28.430 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7 2024-09-18T08:19:28.461 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:19:28.462 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:19:28.463 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7' 2024-09-18T08:19:28.564 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2 2024-09-18T08:19:28.592 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:19:28.592 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:19:28.593 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2' 2024-09-18T08:19:28.652 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 6, 5, 7] out_osds: [4] dead_osds: [] live_osds: [0, 2, 3, 4, 5, 6, 7, 1] 2024-09-18T08:19:28.652 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-18T08:19:28.653 INFO:tasks.thrashosds.thrasher:fixing pg num pool unique_pool_4 2024-09-18T08:19:28.653 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid 779349ca-7593-11ef-bceb-c7b262605968 -- ceph pg dump --format=json 2024-09-18T08:19:28.694 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6 2024-09-18T08:19:28.719 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:19:28.720 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:19:28.720 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6' 2024-09-18T08:19:28.821 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5 2024-09-18T08:19:28.848 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:19:28.849 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:19:28.850 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5' 2024-09-18T08:19:28.905 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:19:28 smithi050 ceph-mon[32371]: pgmap v837: 158 pgs: 6 active+recovering+undersized+remapped, 152 active+clean; 3.9 GiB data, 16 GiB used, 610 GiB / 626 GiB avail; 857 B/s rd, 41 MiB/s wr, 654 op/s; 15648/193593 objects misplaced (8.083%); 12 MiB/s, 188 objects/s recovering 2024-09-18T08:19:28.905 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:19:28 smithi050 ceph-mon[32371]: from='client.? 172.21.15.7:0/3647927239' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-18T08:19:28.905 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:19:28 smithi050 ceph-mon[32371]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-18T08:19:28.951 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7 2024-09-18T08:19:28.977 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:19:28.978 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:19:28.978 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7' 2024-09-18T08:19:28.980 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:19:28 smithi007 ceph-mon[29459]: pgmap v837: 158 pgs: 6 active+recovering+undersized+remapped, 152 active+clean; 3.9 GiB data, 16 GiB used, 610 GiB / 626 GiB avail; 857 B/s rd, 41 MiB/s wr, 654 op/s; 15648/193593 objects misplaced (8.083%); 12 MiB/s, 188 objects/s recovering 2024-09-18T08:19:28.980 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:19:28 smithi007 ceph-mon[29459]: from='client.? 172.21.15.7:0/3647927239' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-18T08:19:28.980 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:19:28 smithi007 ceph-mon[29459]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-18T08:19:28.983 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:19:28 smithi007 ceph-mon[39249]: pgmap v837: 158 pgs: 6 active+recovering+undersized+remapped, 152 active+clean; 3.9 GiB data, 16 GiB used, 610 GiB / 626 GiB avail; 857 B/s rd, 41 MiB/s wr, 654 op/s; 15648/193593 objects misplaced (8.083%); 12 MiB/s, 188 objects/s recovering 2024-09-18T08:19:28.983 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:19:28 smithi007 ceph-mon[39249]: from='client.? 172.21.15.7:0/3647927239' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-18T08:19:28.983 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:19:28 smithi007 ceph-mon[39249]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-18T08:19:29.040 INFO:tasks.radosbench.radosbench.0.smithi007.stdout: 54 16 68046 68030 78.7239 39.9375 0.00991458 0.0126961 2024-09-18T08:19:29.079 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7 2024-09-18T08:19:29.113 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:19:29.114 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:19:29.114 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7' 2024-09-18T08:19:29.142 INFO:teuthology.orchestra.run.smithi007.stderr:noscrub is set 2024-09-18T08:19:29.215 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4 2024-09-18T08:19:29.244 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:19:29.244 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:19:29.245 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4' 2024-09-18T08:19:29.346 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0 2024-09-18T08:19:29.386 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:19:29.387 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:19:29.387 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0' 2024-09-18T08:19:29.488 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0 2024-09-18T08:19:29.530 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:19:29.531 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:19:29.531 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0' 2024-09-18T08:19:29.632 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0 2024-09-18T08:19:29.675 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:19:29.676 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:19:29.676 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0' 2024-09-18T08:19:29.777 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2 2024-09-18T08:19:29.824 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:19:29.825 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:19:29.825 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2' 2024-09-18T08:19:29.905 INFO:journalctl@ceph.iscsi.iscsi.a.smithi050.stdout:Sep 18 08:19:29 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-iscsi-iscsi-a[57782]: debug there is no tcmu-runner data available 2024-09-18T08:19:29.905 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:19:29 smithi050 ceph-mon[32371]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-18T08:19:29.905 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:19:29 smithi050 ceph-mon[32371]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-18T08:19:29.906 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:19:29 smithi050 ceph-mon[32371]: osdmap e423: 8 total, 8 up, 7 in 2024-09-18T08:19:29.906 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:19:29 smithi050 ceph-mon[32371]: osdmap e424: 8 total, 8 up, 7 in 2024-09-18T08:19:29.926 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1 2024-09-18T08:19:29.970 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:19:29.971 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:19:29.971 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1' 2024-09-18T08:19:29.979 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:19:29 smithi007 ceph-mon[29459]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-18T08:19:29.980 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:19:29 smithi007 ceph-mon[29459]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-18T08:19:29.980 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:19:29 smithi007 ceph-mon[29459]: osdmap e423: 8 total, 8 up, 7 in 2024-09-18T08:19:29.980 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:19:29 smithi007 ceph-mon[29459]: osdmap e424: 8 total, 8 up, 7 in 2024-09-18T08:19:29.980 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:19:29 smithi007 ceph-mon[39249]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-18T08:19:29.980 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:19:29 smithi007 ceph-mon[39249]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-18T08:19:29.980 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:19:29 smithi007 ceph-mon[39249]: osdmap e423: 8 total, 8 up, 7 in 2024-09-18T08:19:29.980 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:19:29 smithi007 ceph-mon[39249]: osdmap e424: 8 total, 8 up, 7 in 2024-09-18T08:19:30.041 INFO:tasks.radosbench.radosbench.0.smithi007.stdout: 55 16 68885 68869 78.246 52.4375 0.0132376 0.0127738 2024-09-18T08:19:30.072 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7 2024-09-18T08:19:30.103 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:19:30.104 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:19:30.104 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7' 2024-09-18T08:19:30.205 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5 2024-09-18T08:19:30.236 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:19:30.237 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:19:30.237 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5' 2024-09-18T08:19:30.339 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5 2024-09-18T08:19:30.354 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/779349ca-7593-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T08:19:30.366 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:19:30.367 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:19:30.367 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5' 2024-09-18T08:19:30.468 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3 2024-09-18T08:19:30.528 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:19:30.529 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:19:30.529 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3' 2024-09-18T08:19:30.630 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5 2024-09-18T08:19:30.658 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:19:30.658 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:19:30.659 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5' 2024-09-18T08:19:30.760 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1 2024-09-18T08:19:30.816 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:19:30.816 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:19:30.817 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1' 2024-09-18T08:19:30.905 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:19:30 smithi050 ceph-mon[32371]: pgmap v840: 158 pgs: 6 active+recovering+undersized+remapped, 152 active+clean; 4.1 GiB data, 17 GiB used, 609 GiB / 626 GiB avail; 861 B/s rd, 49 MiB/s wr, 785 op/s; 14753/200268 objects misplaced (7.367%); 14 MiB/s, 219 objects/s recovering 2024-09-18T08:19:30.905 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:19:30 smithi050 ceph-mon[32371]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-18T08:19:30.918 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4 2024-09-18T08:19:30.944 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:19:30.945 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:19:30.945 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4' 2024-09-18T08:19:30.980 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:19:30 smithi007 ceph-mon[29459]: pgmap v840: 158 pgs: 6 active+recovering+undersized+remapped, 152 active+clean; 4.1 GiB data, 17 GiB used, 609 GiB / 626 GiB avail; 861 B/s rd, 49 MiB/s wr, 785 op/s; 14753/200268 objects misplaced (7.367%); 14 MiB/s, 219 objects/s recovering 2024-09-18T08:19:30.980 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:19:30 smithi007 ceph-mon[29459]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-18T08:19:30.980 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:19:30 smithi007 ceph-mon[39249]: pgmap v840: 158 pgs: 6 active+recovering+undersized+remapped, 152 active+clean; 4.1 GiB data, 17 GiB used, 609 GiB / 626 GiB avail; 861 B/s rd, 49 MiB/s wr, 785 op/s; 14753/200268 objects misplaced (7.367%); 14 MiB/s, 219 objects/s recovering 2024-09-18T08:19:30.980 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:19:30 smithi007 ceph-mon[39249]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-18T08:19:31.040 INFO:tasks.radosbench.radosbench.0.smithi007.stdout: 56 16 69681 69665 77.7371 49.75 0.0162246 0.0128575 2024-09-18T08:19:31.046 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7 2024-09-18T08:19:31.075 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:19:31.076 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:19:31.076 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7' 2024-09-18T08:19:31.177 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6 2024-09-18T08:19:31.206 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:19:31.206 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:19:31.207 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6' 2024-09-18T08:19:31.270 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/779349ca-7593-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T08:19:31.308 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5 2024-09-18T08:19:31.334 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:19:31.335 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:19:31.335 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5' 2024-09-18T08:19:31.436 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1 2024-09-18T08:19:31.491 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:19:31.492 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:19:31.492 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1' 2024-09-18T08:19:31.593 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7 2024-09-18T08:19:31.598 INFO:teuthology.orchestra.run.smithi007.stdout:{ 2024-09-18T08:19:31.598 INFO:teuthology.orchestra.run.smithi007.stdout: "size": 20, 2024-09-18T08:19:31.598 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 600, 2024-09-18T08:19:31.599 INFO:teuthology.orchestra.run.smithi007.stdout: "ops": [ 2024-09-18T08:19:31.599 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:19:31.599 INFO:teuthology.orchestra.run.smithi007.stdout: "description": "osd_op(client.24419.0:696 2.0 2:353ec8f9:::gateway.conf:head [getxattr epoch (1) in=5b] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e308)", 2024-09-18T08:19:31.599 INFO:teuthology.orchestra.run.smithi007.stdout: "initiated_at": "2024-09-18T08:14:39.405732+0000", 2024-09-18T08:19:31.599 INFO:teuthology.orchestra.run.smithi007.stdout: "age": 292.18371731600001, 2024-09-18T08:19:31.599 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.94917402399999995, 2024-09-18T08:19:31.599 INFO:teuthology.orchestra.run.smithi007.stdout: "continuous": false, 2024-09-18T08:19:31.599 INFO:teuthology.orchestra.run.smithi007.stdout: "type_data": { 2024-09-18T08:19:31.599 INFO:teuthology.orchestra.run.smithi007.stdout: "flag_point": "started", 2024-09-18T08:19:31.599 INFO:teuthology.orchestra.run.smithi007.stdout: "client_info": { 2024-09-18T08:19:31.599 INFO:teuthology.orchestra.run.smithi007.stdout: "client": "client.24419", 2024-09-18T08:19:31.599 INFO:teuthology.orchestra.run.smithi007.stdout: "client_addr": "172.21.15.50:0/4007330110", 2024-09-18T08:19:31.599 INFO:teuthology.orchestra.run.smithi007.stdout: "tid": 696 2024-09-18T08:19:31.600 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:19:31.600 INFO:teuthology.orchestra.run.smithi007.stdout: "events": [ 2024-09-18T08:19:31.600 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:19:31.600 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "initiated", 2024-09-18T08:19:31.600 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:14:39.405732+0000", 2024-09-18T08:19:31.600 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-09-18T08:19:31.600 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:19:31.600 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:19:31.600 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "header_read", 2024-09-18T08:19:31.600 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:14:39.405732+0000", 2024-09-18T08:19:31.600 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-09-18T08:19:31.600 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:19:31.600 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:19:31.601 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "throttled", 2024-09-18T08:19:31.601 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:14:39.405733+0000", 2024-09-18T08:19:31.601 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.491e-06 2024-09-18T08:19:31.601 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:19:31.601 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:19:31.601 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "all_read", 2024-09-18T08:19:31.601 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:14:39.405739+0000", 2024-09-18T08:19:31.601 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 6.0050000000000001e-06 2024-09-18T08:19:31.601 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:19:31.601 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:19:31.601 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "dispatched", 2024-09-18T08:19:31.601 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:14:39.406166+0000", 2024-09-18T08:19:31.601 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.00042718500000000001 2024-09-18T08:19:31.602 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:19:31.602 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:19:31.602 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "queued_for_pg", 2024-09-18T08:19:31.602 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:14:39.406178+0000", 2024-09-18T08:19:31.602 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.1693e-05 2024-09-18T08:19:31.602 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:19:31.602 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:19:31.602 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-09-18T08:19:31.602 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:14:39.407800+0000", 2024-09-18T08:19:31.602 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.0016220570000000001 2024-09-18T08:19:31.602 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:19:31.602 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:19:31.602 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "waiting for peered", 2024-09-18T08:19:31.603 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:14:39.407810+0000", 2024-09-18T08:19:31.603 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.0111e-05 2024-09-18T08:19:31.603 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:19:31.603 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:19:31.603 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-09-18T08:19:31.603 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:14:40.354668+0000", 2024-09-18T08:19:31.603 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.94685731699999998 2024-09-18T08:19:31.603 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:19:31.603 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:19:31.603 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "started", 2024-09-18T08:19:31.603 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:14:40.354804+0000", 2024-09-18T08:19:31.603 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.00013648999999999999 2024-09-18T08:19:31.603 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:19:31.604 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:19:31.604 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "done", 2024-09-18T08:19:31.604 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:14:40.354906+0000", 2024-09-18T08:19:31.604 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.000101675 2024-09-18T08:19:31.604 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-09-18T08:19:31.604 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-09-18T08:19:31.604 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-09-18T08:19:31.604 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:19:31.604 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:19:31.604 INFO:teuthology.orchestra.run.smithi007.stdout: "description": "osd_op(client.15966.0:98634 10.0 10:0787cdb8:::benchmark_data_smithi007_148834_object98633:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e347)", 2024-09-18T08:19:31.604 INFO:teuthology.orchestra.run.smithi007.stdout: "initiated_at": "2024-09-18T08:16:12.399976+0000", 2024-09-18T08:19:31.604 INFO:teuthology.orchestra.run.smithi007.stdout: "age": 199.189472948, 2024-09-18T08:19:31.604 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 2.1342004050000001, 2024-09-18T08:19:31.605 INFO:teuthology.orchestra.run.smithi007.stdout: "continuous": false, 2024-09-18T08:19:31.605 INFO:teuthology.orchestra.run.smithi007.stdout: "type_data": { 2024-09-18T08:19:31.605 INFO:teuthology.orchestra.run.smithi007.stdout: "flag_point": "reached pg", 2024-09-18T08:19:31.605 INFO:teuthology.orchestra.run.smithi007.stdout: "client_info": { 2024-09-18T08:19:31.605 INFO:teuthology.orchestra.run.smithi007.stdout: "client": "client.15966", 2024-09-18T08:19:31.605 INFO:teuthology.orchestra.run.smithi007.stdout: "client_addr": "172.21.15.7:0/529784850", 2024-09-18T08:19:31.605 INFO:teuthology.orchestra.run.smithi007.stdout: "tid": 98634 2024-09-18T08:19:31.605 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:19:31.605 INFO:teuthology.orchestra.run.smithi007.stdout: "events": [ 2024-09-18T08:19:31.605 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:19:31.605 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "initiated", 2024-09-18T08:19:31.605 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:16:12.399976+0000", 2024-09-18T08:19:31.605 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-09-18T08:19:31.606 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:19:31.606 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:19:31.606 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "header_read", 2024-09-18T08:19:31.606 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:16:12.399976+0000", 2024-09-18T08:19:31.606 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-09-18T08:19:31.606 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:19:31.606 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:19:31.606 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "throttled", 2024-09-18T08:19:31.606 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:16:12.399978+0000", 2024-09-18T08:19:31.606 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.9750000000000001e-06 2024-09-18T08:19:31.606 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:19:31.606 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:19:31.606 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "all_read", 2024-09-18T08:19:31.607 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:16:12.400001+0000", 2024-09-18T08:19:31.607 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 2.3230999999999998e-05 2024-09-18T08:19:31.607 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:19:31.607 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:19:31.607 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "dispatched", 2024-09-18T08:19:31.607 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:16:12.400014+0000", 2024-09-18T08:19:31.607 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.3071000000000001e-05 2024-09-18T08:19:31.607 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:19:31.607 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:19:31.607 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "queued_for_pg", 2024-09-18T08:19:31.607 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:16:12.400022+0000", 2024-09-18T08:19:31.607 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 7.7400000000000004e-06 2024-09-18T08:19:31.608 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:19:31.608 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:19:31.608 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-09-18T08:19:31.608 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:16:12.402679+0000", 2024-09-18T08:19:31.608 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.0026564039999999998 2024-09-18T08:19:31.608 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:19:31.608 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:19:31.608 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "started", 2024-09-18T08:19:31.608 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:16:12.402821+0000", 2024-09-18T08:19:31.608 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.00014268700000000001 2024-09-18T08:19:31.608 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:19:31.608 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:19:31.608 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "waiting for subops from 4,6", 2024-09-18T08:19:31.609 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:16:12.402970+0000", 2024-09-18T08:19:31.609 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.00014844700000000001 2024-09-18T08:19:31.609 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:19:31.609 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:19:31.609 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-09-18T08:19:31.609 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:16:12.457276+0000", 2024-09-18T08:19:31.609 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.054306818999999999 2024-09-18T08:19:31.609 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:19:31.609 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:19:31.609 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "waiting for peered", 2024-09-18T08:19:31.609 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:16:12.457291+0000", 2024-09-18T08:19:31.609 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.4063000000000001e-05 2024-09-18T08:19:31.609 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:19:31.609 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:19:31.610 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-09-18T08:19:31.610 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:16:13.443181+0000", 2024-09-18T08:19:31.610 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.98589081899999997 2024-09-18T08:19:31.610 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:19:31.610 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:19:31.610 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "waiting for peered", 2024-09-18T08:19:31.610 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:16:13.443195+0000", 2024-09-18T08:19:31.610 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.3278999999999999e-05 2024-09-18T08:19:31.610 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:19:31.611 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:19:31.611 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-09-18T08:19:31.611 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:16:14.529134+0000", 2024-09-18T08:19:31.611 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.085939787 2024-09-18T08:19:31.611 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:19:31.611 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:19:31.611 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "waiting for degraded object", 2024-09-18T08:19:31.611 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:16:14.529905+0000", 2024-09-18T08:19:31.611 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.00077055999999999999 2024-09-18T08:19:31.611 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:19:31.611 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:19:31.611 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-09-18T08:19:31.611 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:16:14.534096+0000", 2024-09-18T08:19:31.612 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.0041908589999999999 2024-09-18T08:19:31.612 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:19:31.612 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:19:31.612 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "done", 2024-09-18T08:19:31.612 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:16:14.534177+0000", 2024-09-18T08:19:31.612 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 8.0663999999999999e-05 2024-09-18T08:19:31.612 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-09-18T08:19:31.612 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-09-18T08:19:31.612 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-09-18T08:19:31.612 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:19:31.612 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:19:31.612 INFO:teuthology.orchestra.run.smithi007.stdout: "description": "osd_op(client.15966.0:98637 10.2 10:4026e598:::benchmark_data_smithi007_148834_object98636:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e347)", 2024-09-18T08:19:31.612 INFO:teuthology.orchestra.run.smithi007.stdout: "initiated_at": "2024-09-18T08:16:12.400092+0000", 2024-09-18T08:19:31.613 INFO:teuthology.orchestra.run.smithi007.stdout: "age": 199.189357398, 2024-09-18T08:19:31.613 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 2.1934846079999999, 2024-09-18T08:19:31.613 INFO:teuthology.orchestra.run.smithi007.stdout: "continuous": false, 2024-09-18T08:19:31.613 INFO:teuthology.orchestra.run.smithi007.stdout: "type_data": { 2024-09-18T08:19:31.613 INFO:teuthology.orchestra.run.smithi007.stdout: "flag_point": "reached pg", 2024-09-18T08:19:31.613 INFO:teuthology.orchestra.run.smithi007.stdout: "client_info": { 2024-09-18T08:19:31.613 INFO:teuthology.orchestra.run.smithi007.stdout: "client": "client.15966", 2024-09-18T08:19:31.613 INFO:teuthology.orchestra.run.smithi007.stdout: "client_addr": "172.21.15.7:0/529784850", 2024-09-18T08:19:31.613 INFO:teuthology.orchestra.run.smithi007.stdout: "tid": 98637 2024-09-18T08:19:31.613 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:19:31.613 INFO:teuthology.orchestra.run.smithi007.stdout: "events": [ 2024-09-18T08:19:31.613 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:19:31.613 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "initiated", 2024-09-18T08:19:31.614 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:16:12.400092+0000", 2024-09-18T08:19:31.614 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-09-18T08:19:31.614 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:19:31.614 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:19:31.614 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "header_read", 2024-09-18T08:19:31.614 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:16:12.400092+0000", 2024-09-18T08:19:31.614 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-09-18T08:19:31.614 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:19:31.614 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:19:31.614 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "throttled", 2024-09-18T08:19:31.614 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:16:12.400092+0000", 2024-09-18T08:19:31.614 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 4.0900000000000002e-07 2024-09-18T08:19:31.614 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:19:31.615 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:19:31.615 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "all_read", 2024-09-18T08:19:31.615 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:16:12.400111+0000", 2024-09-18T08:19:31.615 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.8771e-05 2024-09-18T08:19:31.615 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:19:31.615 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:19:31.615 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "dispatched", 2024-09-18T08:19:31.615 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:16:12.400117+0000", 2024-09-18T08:19:31.615 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 5.964e-06 2024-09-18T08:19:31.615 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:19:31.615 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:19:31.615 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "queued_for_pg", 2024-09-18T08:19:31.615 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:16:12.400122+0000", 2024-09-18T08:19:31.615 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 5.006e-06 2024-09-18T08:19:31.616 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:19:31.616 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:19:31.616 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-09-18T08:19:31.616 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:16:12.405026+0000", 2024-09-18T08:19:31.616 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.0049037550000000001 2024-09-18T08:19:31.616 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:19:31.616 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:19:31.616 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "started", 2024-09-18T08:19:31.616 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:16:12.405149+0000", 2024-09-18T08:19:31.616 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.000123165 2024-09-18T08:19:31.616 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:19:31.616 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:19:31.616 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "waiting for subops from 0,4", 2024-09-18T08:19:31.617 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:16:12.405296+0000", 2024-09-18T08:19:31.617 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.00014697000000000001 2024-09-18T08:19:31.617 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:19:31.617 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:19:31.617 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-09-18T08:19:31.617 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:16:12.465899+0000", 2024-09-18T08:19:31.617 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.060603539999999997 2024-09-18T08:19:31.617 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:19:31.617 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:19:31.617 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "waiting for peered", 2024-09-18T08:19:31.617 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:16:12.465913+0000", 2024-09-18T08:19:31.617 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.3733999999999999e-05 2024-09-18T08:19:31.617 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:19:31.617 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:19:31.618 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-09-18T08:19:31.618 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:16:13.449381+0000", 2024-09-18T08:19:31.618 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.98346793600000004 2024-09-18T08:19:31.618 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:19:31.618 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:19:31.618 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "waiting for peered", 2024-09-18T08:19:31.618 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:16:13.449394+0000", 2024-09-18T08:19:31.618 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.3207000000000001e-05 2024-09-18T08:19:31.618 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:19:31.618 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:19:31.618 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-09-18T08:19:31.618 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:16:14.551054+0000", 2024-09-18T08:19:31.618 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.101660063 2024-09-18T08:19:31.619 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:19:31.619 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:19:31.619 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "waiting for degraded object", 2024-09-18T08:19:31.619 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:16:14.551966+0000", 2024-09-18T08:19:31.619 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.00091193800000000005 2024-09-18T08:19:31.619 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:19:31.619 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:19:31.619 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-09-18T08:19:31.619 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:16:14.593429+0000", 2024-09-18T08:19:31.619 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.041462859999999997 2024-09-18T08:19:31.619 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:19:31.619 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:19:31.619 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "done", 2024-09-18T08:19:31.619 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:16:14.593576+0000", 2024-09-18T08:19:31.620 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.00014729000000000001 2024-09-18T08:19:31.620 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-09-18T08:19:31.620 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-09-18T08:19:31.620 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-09-18T08:19:31.620 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:19:31.620 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:19:31.620 INFO:teuthology.orchestra.run.smithi007.stdout: "description": "osd_op(client.15966.0:98646 10.2 10:482c60da:::benchmark_data_smithi007_148834_object98645:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e348)", 2024-09-18T08:19:31.620 INFO:teuthology.orchestra.run.smithi007.stdout: "initiated_at": "2024-09-18T08:16:12.464818+0000", 2024-09-18T08:19:31.620 INFO:teuthology.orchestra.run.smithi007.stdout: "age": 199.12463089900001, 2024-09-18T08:19:31.620 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 2.1291308149999999, 2024-09-18T08:19:31.620 INFO:teuthology.orchestra.run.smithi007.stdout: "continuous": false, 2024-09-18T08:19:31.620 INFO:teuthology.orchestra.run.smithi007.stdout: "type_data": { 2024-09-18T08:19:31.620 INFO:teuthology.orchestra.run.smithi007.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T08:19:31.621 INFO:teuthology.orchestra.run.smithi007.stdout: "client_info": { 2024-09-18T08:19:31.621 INFO:teuthology.orchestra.run.smithi007.stdout: "client": "client.15966", 2024-09-18T08:19:31.621 INFO:teuthology.orchestra.run.smithi007.stdout: "client_addr": "172.21.15.7:0/529784850", 2024-09-18T08:19:31.621 INFO:teuthology.orchestra.run.smithi007.stdout: "tid": 98646 2024-09-18T08:19:31.621 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:19:31.621 INFO:teuthology.orchestra.run.smithi007.stdout: "events": [ 2024-09-18T08:19:31.621 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:19:31.621 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "initiated", 2024-09-18T08:19:31.621 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:16:12.464818+0000", 2024-09-18T08:19:31.621 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-09-18T08:19:31.621 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:19:31.621 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:19:31.621 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "header_read", 2024-09-18T08:19:31.621 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:16:12.464818+0000", 2024-09-18T08:19:31.622 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-09-18T08:19:31.622 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:19:31.622 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:19:31.622 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "throttled", 2024-09-18T08:19:31.622 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:16:12.464820+0000", 2024-09-18T08:19:31.622 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.5579999999999999e-06 2024-09-18T08:19:31.622 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:19:31.622 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:19:31.622 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "all_read", 2024-09-18T08:19:31.622 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:16:12.464844+0000", 2024-09-18T08:19:31.622 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 2.4484000000000001e-05 2024-09-18T08:19:31.622 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:19:31.622 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:19:31.622 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "dispatched", 2024-09-18T08:19:31.623 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:16:12.464860+0000", 2024-09-18T08:19:31.623 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.6246e-05 2024-09-18T08:19:31.623 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:19:31.623 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:19:31.623 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "queued_for_pg", 2024-09-18T08:19:31.623 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:16:12.464869+0000", 2024-09-18T08:19:31.623 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 8.9910000000000008e-06 2024-09-18T08:19:31.623 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:19:31.623 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:19:31.623 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-09-18T08:19:31.623 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:16:12.466278+0000", 2024-09-18T08:19:31.623 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.001408627 2024-09-18T08:19:31.623 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:19:31.624 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:19:31.624 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "waiting for peered", 2024-09-18T08:19:31.624 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:16:12.466288+0000", 2024-09-18T08:19:31.624 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 9.5980000000000003e-06 2024-09-18T08:19:31.624 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:19:31.624 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:19:31.624 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-09-18T08:19:31.624 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:16:13.449420+0000", 2024-09-18T08:19:31.624 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.98313272600000001 2024-09-18T08:19:31.624 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:19:31.624 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:19:31.624 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "waiting for peered", 2024-09-18T08:19:31.624 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:16:13.449428+0000", 2024-09-18T08:19:31.625 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 7.6299999999999998e-06 2024-09-18T08:19:31.625 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:19:31.625 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:19:31.625 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-09-18T08:19:31.625 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:16:14.552301+0000", 2024-09-18T08:19:31.625 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.1028732960000001 2024-09-18T08:19:31.625 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:19:31.625 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:19:31.625 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "started", 2024-09-18T08:19:31.625 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:16:14.552434+0000", 2024-09-18T08:19:31.625 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.00013269099999999999 2024-09-18T08:19:31.625 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:19:31.625 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:19:31.625 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "waiting for subops from 0,3,4", 2024-09-18T08:19:31.626 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:16:14.553011+0000", 2024-09-18T08:19:31.626 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.00057736099999999998 2024-09-18T08:19:31.626 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:19:31.626 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:19:31.626 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:19:31.626 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:16:14.569316+0000", 2024-09-18T08:19:31.626 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.016304869 2024-09-18T08:19:31.626 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:19:31.626 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:19:31.626 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "op_commit", 2024-09-18T08:19:31.626 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:16:14.573668+0000", 2024-09-18T08:19:31.626 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.0043519930000000002 2024-09-18T08:19:31.626 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:19:31.627 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:19:31.627 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:19:31.627 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:16:14.592879+0000", 2024-09-18T08:19:31.627 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.019210628 2024-09-18T08:19:31.627 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:19:31.627 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:19:31.627 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:19:31.627 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:16:14.593619+0000", 2024-09-18T08:19:31.627 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.00073994499999999999 2024-09-18T08:19:31.627 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:19:31.627 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:19:31.627 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "commit_sent", 2024-09-18T08:19:31.627 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:16:14.593833+0000", 2024-09-18T08:19:31.628 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.000214064 2024-09-18T08:19:31.628 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:19:31.628 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:19:31.628 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "done", 2024-09-18T08:19:31.628 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:16:14.593949+0000", 2024-09-18T08:19:31.628 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.00011610799999999999 2024-09-18T08:19:31.628 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-09-18T08:19:31.628 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-09-18T08:19:31.628 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-09-18T08:19:31.628 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:19:31.628 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:19:31.628 INFO:teuthology.orchestra.run.smithi007.stdout: "description": "osd_op(client.15966.0:129869 10.0 10:04ddbf68:::benchmark_data_smithi007_148834_object28149:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e366)", 2024-09-18T08:19:31.628 INFO:teuthology.orchestra.run.smithi007.stdout: "initiated_at": "2024-09-18T08:16:56.596150+0000", 2024-09-18T08:19:31.628 INFO:teuthology.orchestra.run.smithi007.stdout: "age": 154.993299116, 2024-09-18T08:19:31.629 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.0014386099999999, 2024-09-18T08:19:31.629 INFO:teuthology.orchestra.run.smithi007.stdout: "continuous": false, 2024-09-18T08:19:31.629 INFO:teuthology.orchestra.run.smithi007.stdout: "type_data": { 2024-09-18T08:19:31.629 INFO:teuthology.orchestra.run.smithi007.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T08:19:31.629 INFO:teuthology.orchestra.run.smithi007.stdout: "client_info": { 2024-09-18T08:19:31.629 INFO:teuthology.orchestra.run.smithi007.stdout: "client": "client.15966", 2024-09-18T08:19:31.629 INFO:teuthology.orchestra.run.smithi007.stdout: "client_addr": "172.21.15.7:0/529784850", 2024-09-18T08:19:31.629 INFO:teuthology.orchestra.run.smithi007.stdout: "tid": 129869 2024-09-18T08:19:31.629 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:19:31.629 INFO:teuthology.orchestra.run.smithi007.stdout: "events": [ 2024-09-18T08:19:31.629 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:19:31.629 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "initiated", 2024-09-18T08:19:31.629 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:16:56.596150+0000", 2024-09-18T08:19:31.630 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-09-18T08:19:31.630 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:19:31.630 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:19:31.630 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "header_read", 2024-09-18T08:19:31.630 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:16:56.596150+0000", 2024-09-18T08:19:31.630 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-09-18T08:19:31.630 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:19:31.630 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:19:31.630 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "throttled", 2024-09-18T08:19:31.630 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:16:56.596152+0000", 2024-09-18T08:19:31.630 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.68e-06 2024-09-18T08:19:31.630 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:19:31.630 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:19:31.631 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "all_read", 2024-09-18T08:19:31.631 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:16:56.596157+0000", 2024-09-18T08:19:31.631 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 5.7520000000000002e-06 2024-09-18T08:19:31.631 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:19:31.631 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:19:31.631 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "dispatched", 2024-09-18T08:19:31.631 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:16:56.596170+0000", 2024-09-18T08:19:31.631 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.22e-05 2024-09-18T08:19:31.631 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:19:31.631 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:19:31.631 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "queued_for_pg", 2024-09-18T08:19:31.631 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:16:56.596177+0000", 2024-09-18T08:19:31.631 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 7.7829999999999999e-06 2024-09-18T08:19:31.632 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:19:31.632 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:19:31.632 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-09-18T08:19:31.632 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:16:56.596236+0000", 2024-09-18T08:19:31.632 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 5.8554999999999998e-05 2024-09-18T08:19:31.632 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:19:31.632 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:19:31.632 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "waiting for peered", 2024-09-18T08:19:31.632 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:16:56.596244+0000", 2024-09-18T08:19:31.632 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 7.9519999999999994e-06 2024-09-18T08:19:31.632 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:19:31.632 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:19:31.633 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-09-18T08:19:31.633 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:16:57.589955+0000", 2024-09-18T08:19:31.633 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.99371131499999998 2024-09-18T08:19:31.633 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:19:31.633 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:19:31.633 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "started", 2024-09-18T08:19:31.633 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:16:57.590089+0000", 2024-09-18T08:19:31.633 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.00013384700000000001 2024-09-18T08:19:31.633 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:19:31.633 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:19:31.633 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "waiting for subops from 0,6", 2024-09-18T08:19:31.633 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:16:57.590223+0000", 2024-09-18T08:19:31.633 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.000134046 2024-09-18T08:19:31.634 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:19:31.634 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:19:31.634 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:19:31.634 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:16:57.592665+0000", 2024-09-18T08:19:31.634 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.0024419720000000002 2024-09-18T08:19:31.634 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:19:31.634 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:19:31.634 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "op_commit", 2024-09-18T08:19:31.634 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:16:57.592679+0000", 2024-09-18T08:19:31.634 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.3652000000000001e-05 2024-09-18T08:19:31.634 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:19:31.634 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:19:31.635 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:19:31.635 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:16:57.597278+0000", 2024-09-18T08:19:31.635 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.0045995439999999997 2024-09-18T08:19:31.635 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:19:31.635 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:19:31.635 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "commit_sent", 2024-09-18T08:19:31.635 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:16:57.597541+0000", 2024-09-18T08:19:31.635 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.00026299400000000001 2024-09-18T08:19:31.635 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:19:31.635 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:19:31.635 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "done", 2024-09-18T08:19:31.635 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:16:57.597589+0000", 2024-09-18T08:19:31.635 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 4.7318000000000003e-05 2024-09-18T08:19:31.636 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-09-18T08:19:31.636 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-09-18T08:19:31.636 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-09-18T08:19:31.636 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:19:31.636 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:19:31.636 INFO:teuthology.orchestra.run.smithi007.stdout: "description": "osd_op(client.15966.0:129891 10.0 10:0ca5a76f:::benchmark_data_smithi007_148834_object28171:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e366)", 2024-09-18T08:19:31.636 INFO:teuthology.orchestra.run.smithi007.stdout: "initiated_at": "2024-09-18T08:16:56.615109+0000", 2024-09-18T08:19:31.636 INFO:teuthology.orchestra.run.smithi007.stdout: "age": 154.97434001100001, 2024-09-18T08:19:31.636 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.98279396500000005, 2024-09-18T08:19:31.636 INFO:teuthology.orchestra.run.smithi007.stdout: "continuous": false, 2024-09-18T08:19:31.636 INFO:teuthology.orchestra.run.smithi007.stdout: "type_data": { 2024-09-18T08:19:31.636 INFO:teuthology.orchestra.run.smithi007.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T08:19:31.636 INFO:teuthology.orchestra.run.smithi007.stdout: "client_info": { 2024-09-18T08:19:31.637 INFO:teuthology.orchestra.run.smithi007.stdout: "client": "client.15966", 2024-09-18T08:19:31.637 INFO:teuthology.orchestra.run.smithi007.stdout: "client_addr": "172.21.15.7:0/529784850", 2024-09-18T08:19:31.637 INFO:teuthology.orchestra.run.smithi007.stdout: "tid": 129891 2024-09-18T08:19:31.637 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:19:31.637 INFO:teuthology.orchestra.run.smithi007.stdout: "events": [ 2024-09-18T08:19:31.637 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:19:31.637 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "initiated", 2024-09-18T08:19:31.637 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:16:56.615109+0000", 2024-09-18T08:19:31.637 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-09-18T08:19:31.637 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:19:31.637 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:19:31.637 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "header_read", 2024-09-18T08:19:31.637 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:16:56.615109+0000", 2024-09-18T08:19:31.638 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-09-18T08:19:31.638 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:19:31.638 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:19:31.638 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "throttled", 2024-09-18T08:19:31.638 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:16:56.615111+0000", 2024-09-18T08:19:31.638 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.813e-06 2024-09-18T08:19:31.638 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:19:31.638 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:19:31.638 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "all_read", 2024-09-18T08:19:31.638 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:16:56.615116+0000", 2024-09-18T08:19:31.638 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 5.1229999999999999e-06 2024-09-18T08:19:31.638 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:19:31.638 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:19:31.639 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "dispatched", 2024-09-18T08:19:31.639 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:16:56.615127+0000", 2024-09-18T08:19:31.639 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.0564000000000001e-05 2024-09-18T08:19:31.639 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:19:31.639 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:19:31.639 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "queued_for_pg", 2024-09-18T08:19:31.639 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:16:56.615135+0000", 2024-09-18T08:19:31.639 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 8.2700000000000004e-06 2024-09-18T08:19:31.639 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:19:31.639 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:19:31.639 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-09-18T08:19:31.639 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:16:56.615217+0000", 2024-09-18T08:19:31.640 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 8.2200000000000006e-05 2024-09-18T08:19:31.640 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:19:31.640 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:19:31.640 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "waiting for peered", 2024-09-18T08:19:31.640 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:16:56.615226+0000", 2024-09-18T08:19:31.640 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 8.9639999999999992e-06 2024-09-18T08:19:31.640 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:19:31.640 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:19:31.640 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-09-18T08:19:31.640 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:16:57.592704+0000", 2024-09-18T08:19:31.640 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.97747756299999999 2024-09-18T08:19:31.640 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:19:31.640 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:19:31.641 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "started", 2024-09-18T08:19:31.641 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:16:57.592803+0000", 2024-09-18T08:19:31.641 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 9.9791999999999996e-05 2024-09-18T08:19:31.641 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:19:31.641 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:19:31.641 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "waiting for subops from 0,6", 2024-09-18T08:19:31.641 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:16:57.592923+0000", 2024-09-18T08:19:31.641 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.000120062 2024-09-18T08:19:31.641 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:19:31.641 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:19:31.641 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "op_commit", 2024-09-18T08:19:31.641 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:16:57.597600+0000", 2024-09-18T08:19:31.641 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.004677095 2024-09-18T08:19:31.642 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:19:31.642 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:19:31.642 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:19:31.642 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:16:57.597646+0000", 2024-09-18T08:19:31.642 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 4.5581000000000003e-05 2024-09-18T08:19:31.642 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:19:31.642 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:19:31.643 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:19:31.643 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:16:57.597694+0000", 2024-09-18T08:19:31.644 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 4.7895999999999997e-05 2024-09-18T08:19:31.644 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:19:31.644 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:19:31.644 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "commit_sent", 2024-09-18T08:19:31.644 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:16:57.597860+0000", 2024-09-18T08:19:31.644 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.00016641 2024-09-18T08:19:31.644 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:19:31.644 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:19:31.644 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "done", 2024-09-18T08:19:31.644 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:16:57.597903+0000", 2024-09-18T08:19:31.644 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 4.2632000000000001e-05 2024-09-18T08:19:31.645 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-09-18T08:19:31.645 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-09-18T08:19:31.645 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-09-18T08:19:31.645 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:19:31.645 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:19:31.645 INFO:teuthology.orchestra.run.smithi007.stdout: "description": "osd_op(client.15966.0:129905 10.0 10:08e0b591:::benchmark_data_smithi007_148834_object28185:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e366)", 2024-09-18T08:19:31.645 INFO:teuthology.orchestra.run.smithi007.stdout: "initiated_at": "2024-09-18T08:16:56.626884+0000", 2024-09-18T08:19:31.645 INFO:teuthology.orchestra.run.smithi007.stdout: "age": 154.962564601, 2024-09-18T08:19:31.645 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.97665455199999995, 2024-09-18T08:19:31.645 INFO:teuthology.orchestra.run.smithi007.stdout: "continuous": false, 2024-09-18T08:19:31.645 INFO:teuthology.orchestra.run.smithi007.stdout: "type_data": { 2024-09-18T08:19:31.645 INFO:teuthology.orchestra.run.smithi007.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T08:19:31.645 INFO:teuthology.orchestra.run.smithi007.stdout: "client_info": { 2024-09-18T08:19:31.646 INFO:teuthology.orchestra.run.smithi007.stdout: "client": "client.15966", 2024-09-18T08:19:31.646 INFO:teuthology.orchestra.run.smithi007.stdout: "client_addr": "172.21.15.7:0/529784850", 2024-09-18T08:19:31.646 INFO:teuthology.orchestra.run.smithi007.stdout: "tid": 129905 2024-09-18T08:19:31.646 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:19:31.646 INFO:teuthology.orchestra.run.smithi007.stdout: "events": [ 2024-09-18T08:19:31.646 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:19:31.646 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "initiated", 2024-09-18T08:19:31.646 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:16:56.626884+0000", 2024-09-18T08:19:31.646 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-09-18T08:19:31.646 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:19:31.646 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:19:31.646 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "header_read", 2024-09-18T08:19:31.647 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:16:56.626884+0000", 2024-09-18T08:19:31.647 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-09-18T08:19:31.647 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:19:31.647 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:19:31.647 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "throttled", 2024-09-18T08:19:31.647 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:16:56.626886+0000", 2024-09-18T08:19:31.647 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.4750000000000001e-06 2024-09-18T08:19:31.647 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:19:31.647 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:19:31.647 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "all_read", 2024-09-18T08:19:31.647 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:16:56.626891+0000", 2024-09-18T08:19:31.647 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 5.2360000000000004e-06 2024-09-18T08:19:31.647 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:19:31.648 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:19:31.648 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "dispatched", 2024-09-18T08:19:31.648 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:16:56.626906+0000", 2024-09-18T08:19:31.648 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.5305999999999999e-05 2024-09-18T08:19:31.648 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:19:31.648 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:19:31.648 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "queued_for_pg", 2024-09-18T08:19:31.648 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:16:56.626914+0000", 2024-09-18T08:19:31.648 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 7.7740000000000004e-06 2024-09-18T08:19:31.648 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:19:31.648 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:19:31.648 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-09-18T08:19:31.648 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:16:56.626974+0000", 2024-09-18T08:19:31.649 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 5.9626000000000001e-05 2024-09-18T08:19:31.649 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:19:31.649 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:19:31.649 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "waiting for peered", 2024-09-18T08:19:31.649 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:16:56.626981+0000", 2024-09-18T08:19:31.649 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 7.4719999999999998e-06 2024-09-18T08:19:31.649 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:19:31.649 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:19:31.649 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-09-18T08:19:31.649 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:16:57.594835+0000", 2024-09-18T08:19:31.649 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.96785347499999996 2024-09-18T08:19:31.649 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:19:31.650 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:19:31.650 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "started", 2024-09-18T08:19:31.650 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:16:57.594954+0000", 2024-09-18T08:19:31.650 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.000118978 2024-09-18T08:19:31.650 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:19:31.650 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:19:31.650 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "waiting for subops from 0,6", 2024-09-18T08:19:31.650 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:16:57.595157+0000", 2024-09-18T08:19:31.650 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.00020303500000000001 2024-09-18T08:19:31.650 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:19:31.650 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:19:31.650 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "op_commit", 2024-09-18T08:19:31.650 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:16:57.597658+0000", 2024-09-18T08:19:31.651 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.0025009870000000001 2024-09-18T08:19:31.651 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:19:31.651 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:19:31.651 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:19:31.651 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:16:57.597950+0000", 2024-09-18T08:19:31.651 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.00029189599999999998 2024-09-18T08:19:31.651 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:19:31.651 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:19:31.651 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:19:31.651 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:16:57.603446+0000", 2024-09-18T08:19:31.651 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.0054962939999999997 2024-09-18T08:19:31.651 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:19:31.651 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:19:31.651 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "commit_sent", 2024-09-18T08:19:31.652 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:16:57.603498+0000", 2024-09-18T08:19:31.652 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 5.2405999999999999e-05 2024-09-18T08:19:31.652 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:19:31.652 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:19:31.652 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "done", 2024-09-18T08:19:31.652 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:16:57.603539+0000", 2024-09-18T08:19:31.652 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 4.0592e-05 2024-09-18T08:19:31.652 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-09-18T08:19:31.652 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-09-18T08:19:31.652 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-09-18T08:19:31.652 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:19:31.652 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:19:31.652 INFO:teuthology.orchestra.run.smithi007.stdout: "description": "osd_op(client.15966.0:129926 10.0 10:013a5d34:::benchmark_data_smithi007_148834_object28206:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e366)", 2024-09-18T08:19:31.653 INFO:teuthology.orchestra.run.smithi007.stdout: "initiated_at": "2024-09-18T08:16:56.645254+0000", 2024-09-18T08:19:31.653 INFO:teuthology.orchestra.run.smithi007.stdout: "age": 154.94419474, 2024-09-18T08:19:31.653 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.96265416699999995, 2024-09-18T08:19:31.653 INFO:teuthology.orchestra.run.smithi007.stdout: "continuous": false, 2024-09-18T08:19:31.653 INFO:teuthology.orchestra.run.smithi007.stdout: "type_data": { 2024-09-18T08:19:31.653 INFO:teuthology.orchestra.run.smithi007.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T08:19:31.653 INFO:teuthology.orchestra.run.smithi007.stdout: "client_info": { 2024-09-18T08:19:31.653 INFO:teuthology.orchestra.run.smithi007.stdout: "client": "client.15966", 2024-09-18T08:19:31.653 INFO:teuthology.orchestra.run.smithi007.stdout: "client_addr": "172.21.15.7:0/529784850", 2024-09-18T08:19:31.653 INFO:teuthology.orchestra.run.smithi007.stdout: "tid": 129926 2024-09-18T08:19:31.653 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:19:31.653 INFO:teuthology.orchestra.run.smithi007.stdout: "events": [ 2024-09-18T08:19:31.653 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:19:31.654 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "initiated", 2024-09-18T08:19:31.654 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:16:56.645254+0000", 2024-09-18T08:19:31.654 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-09-18T08:19:31.654 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:19:31.654 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:19:31.654 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "header_read", 2024-09-18T08:19:31.654 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:16:56.645254+0000", 2024-09-18T08:19:31.654 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-09-18T08:19:31.654 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:19:31.654 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:19:31.654 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "throttled", 2024-09-18T08:19:31.654 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:16:56.645256+0000", 2024-09-18T08:19:31.654 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.7179999999999999e-06 2024-09-18T08:19:31.654 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:19:31.655 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:19:31.655 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "all_read", 2024-09-18T08:19:31.655 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:16:56.645261+0000", 2024-09-18T08:19:31.655 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 5.271e-06 2024-09-18T08:19:31.655 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:19:31.655 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:19:31.655 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "dispatched", 2024-09-18T08:19:31.655 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:16:56.645277+0000", 2024-09-18T08:19:31.655 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.5569999999999998e-05 2024-09-18T08:19:31.655 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:19:31.655 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:19:31.655 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "queued_for_pg", 2024-09-18T08:19:31.655 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:16:56.645284+0000", 2024-09-18T08:19:31.656 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 7.2609999999999996e-06 2024-09-18T08:19:31.656 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:19:31.656 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:19:31.656 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-09-18T08:19:31.656 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:16:56.645345+0000", 2024-09-18T08:19:31.656 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 6.0781e-05 2024-09-18T08:19:31.656 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:19:31.656 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:19:31.656 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "waiting for peered", 2024-09-18T08:19:31.656 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:16:56.645353+0000", 2024-09-18T08:19:31.656 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 8.2130000000000008e-06 2024-09-18T08:19:31.656 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:19:31.656 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:19:31.656 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-09-18T08:19:31.657 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:16:57.597983+0000", 2024-09-18T08:19:31.657 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.95262989499999995 2024-09-18T08:19:31.657 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:19:31.657 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:19:31.657 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "started", 2024-09-18T08:19:31.657 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:16:57.598105+0000", 2024-09-18T08:19:31.657 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.000122161 2024-09-18T08:19:31.657 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:19:31.657 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:19:31.657 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "waiting for subops from 0,6", 2024-09-18T08:19:31.657 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:16:57.598235+0000", 2024-09-18T08:19:31.657 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.000130135 2024-09-18T08:19:31.657 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:19:31.657 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:19:31.658 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:19:31.658 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:16:57.602782+0000", 2024-09-18T08:19:31.658 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.0045469969999999997 2024-09-18T08:19:31.658 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:19:31.658 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:19:31.658 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "op_commit", 2024-09-18T08:19:31.658 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:16:57.602797+0000", 2024-09-18T08:19:31.658 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.5038999999999999e-05 2024-09-18T08:19:31.658 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:19:31.658 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:19:31.658 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:19:31.658 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:16:57.607813+0000", 2024-09-18T08:19:31.658 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.0050155759999999999 2024-09-18T08:19:31.659 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:19:31.659 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:19:31.659 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "commit_sent", 2024-09-18T08:19:31.659 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:16:57.607871+0000", 2024-09-18T08:19:31.659 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 5.791e-05 2024-09-18T08:19:31.659 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:19:31.659 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:19:31.659 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "done", 2024-09-18T08:19:31.659 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:16:57.607908+0000", 2024-09-18T08:19:31.659 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 3.7639999999999999e-05 2024-09-18T08:19:31.659 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-09-18T08:19:31.659 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-09-18T08:19:31.659 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-09-18T08:19:31.659 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:19:31.660 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:19:31.660 INFO:teuthology.orchestra.run.smithi007.stdout: "description": "osd_op(client.15966.0:129945 10.0 10:059b6d1d:::benchmark_data_smithi007_148834_object28225:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e366)", 2024-09-18T08:19:31.660 INFO:teuthology.orchestra.run.smithi007.stdout: "initiated_at": "2024-09-18T08:16:56.661964+0000", 2024-09-18T08:19:31.660 INFO:teuthology.orchestra.run.smithi007.stdout: "age": 154.927484986, 2024-09-18T08:19:31.660 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.96324120199999996, 2024-09-18T08:19:31.660 INFO:teuthology.orchestra.run.smithi007.stdout: "continuous": false, 2024-09-18T08:19:31.660 INFO:teuthology.orchestra.run.smithi007.stdout: "type_data": { 2024-09-18T08:19:31.660 INFO:teuthology.orchestra.run.smithi007.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T08:19:31.660 INFO:teuthology.orchestra.run.smithi007.stdout: "client_info": { 2024-09-18T08:19:31.660 INFO:teuthology.orchestra.run.smithi007.stdout: "client": "client.15966", 2024-09-18T08:19:31.660 INFO:teuthology.orchestra.run.smithi007.stdout: "client_addr": "172.21.15.7:0/529784850", 2024-09-18T08:19:31.660 INFO:teuthology.orchestra.run.smithi007.stdout: "tid": 129945 2024-09-18T08:19:31.660 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:19:31.661 INFO:teuthology.orchestra.run.smithi007.stdout: "events": [ 2024-09-18T08:19:31.661 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:19:31.661 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "initiated", 2024-09-18T08:19:31.661 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:16:56.661964+0000", 2024-09-18T08:19:31.661 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-09-18T08:19:31.661 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:19:31.661 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:19:31.661 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "header_read", 2024-09-18T08:19:31.661 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:16:56.661964+0000", 2024-09-18T08:19:31.661 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-09-18T08:19:31.661 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:19:31.661 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:19:31.661 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "throttled", 2024-09-18T08:19:31.661 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:16:56.661966+0000", 2024-09-18T08:19:31.662 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 2.2809999999999998e-06 2024-09-18T08:19:31.662 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:19:31.662 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:19:31.662 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "all_read", 2024-09-18T08:19:31.662 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:16:56.661972+0000", 2024-09-18T08:19:31.662 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 5.6489999999999996e-06 2024-09-18T08:19:31.662 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:19:31.662 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:19:31.662 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "dispatched", 2024-09-18T08:19:31.662 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:16:56.661985+0000", 2024-09-18T08:19:31.662 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.3028e-05 2024-09-18T08:19:31.663 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:19:31.663 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:19:31.663 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "queued_for_pg", 2024-09-18T08:19:31.663 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:16:56.661993+0000", 2024-09-18T08:19:31.663 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 7.9610000000000005e-06 2024-09-18T08:19:31.663 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:19:31.663 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:19:31.663 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-09-18T08:19:31.663 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:16:56.662063+0000", 2024-09-18T08:19:31.663 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 6.9630999999999998e-05 2024-09-18T08:19:31.663 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:19:31.663 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:19:31.663 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "waiting for peered", 2024-09-18T08:19:31.663 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:16:56.662072+0000", 2024-09-18T08:19:31.664 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 9.0559999999999994e-06 2024-09-18T08:19:31.664 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:19:31.664 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:19:31.664 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-09-18T08:19:31.664 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:16:57.600349+0000", 2024-09-18T08:19:31.664 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.93827701299999999 2024-09-18T08:19:31.664 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:19:31.664 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:19:31.664 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "started", 2024-09-18T08:19:31.664 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:16:57.600459+0000", 2024-09-18T08:19:31.664 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.00011054599999999999 2024-09-18T08:19:31.664 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:19:31.664 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:19:31.664 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "waiting for subops from 0,6", 2024-09-18T08:19:31.665 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:16:57.600575+0000", 2024-09-18T08:19:31.665 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.000115491 2024-09-18T08:19:31.665 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:19:31.665 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:19:31.665 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:19:31.665 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:16:57.602828+0000", 2024-09-18T08:19:31.665 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.002253162 2024-09-18T08:19:31.665 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:19:31.665 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:19:31.665 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "op_commit", 2024-09-18T08:19:31.665 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:16:57.605688+0000", 2024-09-18T08:19:31.665 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.0028596210000000001 2024-09-18T08:19:31.665 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:19:31.666 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:19:31.666 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:19:31.666 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:16:57.625064+0000", 2024-09-18T08:19:31.666 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.019376648 2024-09-18T08:19:31.666 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:19:31.666 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:19:31.666 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "commit_sent", 2024-09-18T08:19:31.666 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:16:57.625162+0000", 2024-09-18T08:19:31.666 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 9.8041000000000005e-05 2024-09-18T08:19:31.666 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:19:31.666 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:19:31.666 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "done", 2024-09-18T08:19:31.666 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:16:57.625205+0000", 2024-09-18T08:19:31.667 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 4.3074000000000001e-05 2024-09-18T08:19:31.667 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-09-18T08:19:31.667 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-09-18T08:19:31.667 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-09-18T08:19:31.667 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:19:31.667 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:19:31.667 INFO:teuthology.orchestra.run.smithi007.stdout: "description": "osd_op(client.15966.0:129975 10.0 10:0d525268:::benchmark_data_smithi007_148834_object28255:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e366)", 2024-09-18T08:19:31.667 INFO:teuthology.orchestra.run.smithi007.stdout: "initiated_at": "2024-09-18T08:16:56.689670+0000", 2024-09-18T08:19:31.667 INFO:teuthology.orchestra.run.smithi007.stdout: "age": 154.89977950299999, 2024-09-18T08:19:31.667 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.93569396100000002, 2024-09-18T08:19:31.667 INFO:teuthology.orchestra.run.smithi007.stdout: "continuous": false, 2024-09-18T08:19:31.667 INFO:teuthology.orchestra.run.smithi007.stdout: "type_data": { 2024-09-18T08:19:31.667 INFO:teuthology.orchestra.run.smithi007.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T08:19:31.667 INFO:teuthology.orchestra.run.smithi007.stdout: "client_info": { 2024-09-18T08:19:31.668 INFO:teuthology.orchestra.run.smithi007.stdout: "client": "client.15966", 2024-09-18T08:19:31.668 INFO:teuthology.orchestra.run.smithi007.stdout: "client_addr": "172.21.15.7:0/529784850", 2024-09-18T08:19:31.668 INFO:teuthology.orchestra.run.smithi007.stdout: "tid": 129975 2024-09-18T08:19:31.668 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:19:31.668 INFO:teuthology.orchestra.run.smithi007.stdout: "events": [ 2024-09-18T08:19:31.668 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:19:31.668 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "initiated", 2024-09-18T08:19:31.668 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:16:56.689670+0000", 2024-09-18T08:19:31.668 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-09-18T08:19:31.668 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:19:31.668 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:19:31.668 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "header_read", 2024-09-18T08:19:31.668 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:16:56.689670+0000", 2024-09-18T08:19:31.669 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-09-18T08:19:31.669 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:19:31.669 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:19:31.669 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "throttled", 2024-09-18T08:19:31.669 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:16:56.689671+0000", 2024-09-18T08:19:31.669 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.494e-06 2024-09-18T08:19:31.669 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:19:31.669 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:19:31.669 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "all_read", 2024-09-18T08:19:31.669 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:16:56.689676+0000", 2024-09-18T08:19:31.669 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 5.2209999999999996e-06 2024-09-18T08:19:31.669 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:19:31.669 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:19:31.670 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "dispatched", 2024-09-18T08:19:31.670 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:16:56.689693+0000", 2024-09-18T08:19:31.670 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.6625000000000001e-05 2024-09-18T08:19:31.670 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:19:31.670 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:19:31.670 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "queued_for_pg", 2024-09-18T08:19:31.670 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:16:56.689701+0000", 2024-09-18T08:19:31.670 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 8.3140000000000004e-06 2024-09-18T08:19:31.670 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:19:31.670 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:19:31.670 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-09-18T08:19:31.670 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:16:56.689775+0000", 2024-09-18T08:19:31.670 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 7.3947000000000002e-05 2024-09-18T08:19:31.671 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:19:31.671 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:19:31.671 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "waiting for peered", 2024-09-18T08:19:31.671 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:16:56.689811+0000", 2024-09-18T08:19:31.671 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 3.6201000000000001e-05 2024-09-18T08:19:31.671 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:19:31.671 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:19:31.671 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-09-18T08:19:31.671 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:16:57.603573+0000", 2024-09-18T08:19:31.671 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.91376179199999996 2024-09-18T08:19:31.671 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:19:31.671 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:19:31.671 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "started", 2024-09-18T08:19:31.671 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:16:57.603673+0000", 2024-09-18T08:19:31.672 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 9.9951000000000006e-05 2024-09-18T08:19:31.672 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:19:31.672 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:19:31.672 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "waiting for subops from 0,6", 2024-09-18T08:19:31.672 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:16:57.603796+0000", 2024-09-18T08:19:31.672 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.00012289700000000001 2024-09-18T08:19:31.672 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:19:31.672 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:19:31.672 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:19:31.672 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:16:57.607766+0000", 2024-09-18T08:19:31.672 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.0039696890000000002 2024-09-18T08:19:31.672 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:19:31.672 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:19:31.673 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "op_commit", 2024-09-18T08:19:31.673 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:16:57.607778+0000", 2024-09-18T08:19:31.673 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.2179e-05 2024-09-18T08:19:31.673 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:19:31.673 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:19:31.673 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:19:31.673 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:16:57.625244+0000", 2024-09-18T08:19:31.673 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.017466175 2024-09-18T08:19:31.673 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:19:31.673 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:19:31.673 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "commit_sent", 2024-09-18T08:19:31.673 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:16:57.625322+0000", 2024-09-18T08:19:31.673 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 7.8462999999999997e-05 2024-09-18T08:19:31.674 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:19:31.674 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:19:31.674 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "done", 2024-09-18T08:19:31.674 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:16:57.625364+0000", 2024-09-18T08:19:31.674 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 4.1013000000000002e-05 2024-09-18T08:19:31.674 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-09-18T08:19:31.674 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-09-18T08:19:31.674 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-09-18T08:19:31.676 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:19:31.676 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:19:31.676 INFO:teuthology.orchestra.run.smithi007.stdout: "description": "osd_op(client.15966.0:129983 10.0 10:09b23dba:::benchmark_data_smithi007_148834_object28263:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e366)", 2024-09-18T08:19:31.676 INFO:teuthology.orchestra.run.smithi007.stdout: "initiated_at": "2024-09-18T08:16:56.700095+0000", 2024-09-18T08:19:31.676 INFO:teuthology.orchestra.run.smithi007.stdout: "age": 154.88935396100001, 2024-09-18T08:19:31.676 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.92548011299999999, 2024-09-18T08:19:31.676 INFO:teuthology.orchestra.run.smithi007.stdout: "continuous": false, 2024-09-18T08:19:31.676 INFO:teuthology.orchestra.run.smithi007.stdout: "type_data": { 2024-09-18T08:19:31.676 INFO:teuthology.orchestra.run.smithi007.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T08:19:31.676 INFO:teuthology.orchestra.run.smithi007.stdout: "client_info": { 2024-09-18T08:19:31.676 INFO:teuthology.orchestra.run.smithi007.stdout: "client": "client.15966", 2024-09-18T08:19:31.676 INFO:teuthology.orchestra.run.smithi007.stdout: "client_addr": "172.21.15.7:0/529784850", 2024-09-18T08:19:31.676 INFO:teuthology.orchestra.run.smithi007.stdout: "tid": 129983 2024-09-18T08:19:31.677 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:19:31.677 INFO:teuthology.orchestra.run.smithi007.stdout: "events": [ 2024-09-18T08:19:31.677 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:19:31.677 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "initiated", 2024-09-18T08:19:31.677 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:16:56.700095+0000", 2024-09-18T08:19:31.677 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-09-18T08:19:31.677 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:19:31.677 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:19:31.677 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "header_read", 2024-09-18T08:19:31.677 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:16:56.700095+0000", 2024-09-18T08:19:31.677 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-09-18T08:19:31.677 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:19:31.677 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:19:31.677 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "throttled", 2024-09-18T08:19:31.678 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:16:56.700097+0000", 2024-09-18T08:19:31.678 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.6330000000000001e-06 2024-09-18T08:19:31.678 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:19:31.678 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:19:31.678 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "all_read", 2024-09-18T08:19:31.678 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:16:56.700104+0000", 2024-09-18T08:19:31.678 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 7.126e-06 2024-09-18T08:19:31.678 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:19:31.678 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:19:31.678 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "dispatched", 2024-09-18T08:19:31.678 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:16:56.700117+0000", 2024-09-18T08:19:31.678 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.2931e-05 2024-09-18T08:19:31.678 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:19:31.679 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:19:31.679 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "queued_for_pg", 2024-09-18T08:19:31.679 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:16:56.700125+0000", 2024-09-18T08:19:31.679 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 8.1070000000000005e-06 2024-09-18T08:19:31.679 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:19:31.679 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:19:31.679 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-09-18T08:19:31.679 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:16:56.700204+0000", 2024-09-18T08:19:31.679 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 7.9141999999999995e-05 2024-09-18T08:19:31.679 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:19:31.679 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:19:31.679 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "waiting for peered", 2024-09-18T08:19:31.679 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:16:56.700213+0000", 2024-09-18T08:19:31.680 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 9.4199999999999996e-06 2024-09-18T08:19:31.680 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:19:31.680 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:19:31.680 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-09-18T08:19:31.680 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:16:57.605728+0000", 2024-09-18T08:19:31.680 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.90551429800000005 2024-09-18T08:19:31.680 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:19:31.680 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:19:31.680 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "started", 2024-09-18T08:19:31.680 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:16:57.605827+0000", 2024-09-18T08:19:31.680 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 9.9704000000000006e-05 2024-09-18T08:19:31.680 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:19:31.680 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:19:31.681 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "waiting for subops from 0,6", 2024-09-18T08:19:31.681 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:16:57.606039+0000", 2024-09-18T08:19:31.681 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.00021169799999999999 2024-09-18T08:19:31.681 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:19:31.681 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:19:31.681 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:19:31.681 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:16:57.607950+0000", 2024-09-18T08:19:31.681 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.0019110049999999999 2024-09-18T08:19:31.681 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:19:31.681 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:19:31.681 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "op_commit", 2024-09-18T08:19:31.681 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:16:57.614084+0000", 2024-09-18T08:19:31.681 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.0061341379999999999 2024-09-18T08:19:31.681 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:19:31.682 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:19:31.682 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:19:31.682 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:16:57.625445+0000", 2024-09-18T08:19:31.682 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.011360286000000001 2024-09-18T08:19:31.682 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:19:31.682 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:19:31.682 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "commit_sent", 2024-09-18T08:19:31.682 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:16:57.625531+0000", 2024-09-18T08:19:31.682 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 8.6502000000000006e-05 2024-09-18T08:19:31.682 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:19:31.682 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:19:31.682 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "done", 2024-09-18T08:19:31.682 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:16:57.625575+0000", 2024-09-18T08:19:31.682 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 4.4122999999999998e-05 2024-09-18T08:19:31.683 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-09-18T08:19:31.683 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-09-18T08:19:31.683 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-09-18T08:19:31.683 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:19:31.683 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:19:31.683 INFO:teuthology.orchestra.run.smithi007.stdout: "description": "osd_op(client.15966.0:130006 10.0 10:06b2248a:::benchmark_data_smithi007_148834_object28286:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e366)", 2024-09-18T08:19:31.683 INFO:teuthology.orchestra.run.smithi007.stdout: "initiated_at": "2024-09-18T08:16:56.724531+0000", 2024-09-18T08:19:31.683 INFO:teuthology.orchestra.run.smithi007.stdout: "age": 154.86491840299999, 2024-09-18T08:19:31.683 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.90359768200000001, 2024-09-18T08:19:31.683 INFO:teuthology.orchestra.run.smithi007.stdout: "continuous": false, 2024-09-18T08:19:31.683 INFO:teuthology.orchestra.run.smithi007.stdout: "type_data": { 2024-09-18T08:19:31.683 INFO:teuthology.orchestra.run.smithi007.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T08:19:31.683 INFO:teuthology.orchestra.run.smithi007.stdout: "client_info": { 2024-09-18T08:19:31.684 INFO:teuthology.orchestra.run.smithi007.stdout: "client": "client.15966", 2024-09-18T08:19:31.684 INFO:teuthology.orchestra.run.smithi007.stdout: "client_addr": "172.21.15.7:0/529784850", 2024-09-18T08:19:31.684 INFO:teuthology.orchestra.run.smithi007.stdout: "tid": 130006 2024-09-18T08:19:31.684 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:19:31.684 INFO:teuthology.orchestra.run.smithi007.stdout: "events": [ 2024-09-18T08:19:31.684 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:19:31.684 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "initiated", 2024-09-18T08:19:31.684 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:16:56.724531+0000", 2024-09-18T08:19:31.684 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-09-18T08:19:31.684 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:19:31.684 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:19:31.684 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "header_read", 2024-09-18T08:19:31.684 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:16:56.724531+0000", 2024-09-18T08:19:31.685 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-09-18T08:19:31.685 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:19:31.685 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:19:31.685 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "throttled", 2024-09-18T08:19:31.685 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:16:56.724532+0000", 2024-09-18T08:19:31.685 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.542e-06 2024-09-18T08:19:31.685 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:19:31.685 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:19:31.685 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "all_read", 2024-09-18T08:19:31.685 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:16:56.724538+0000", 2024-09-18T08:19:31.685 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 6.1940000000000003e-06 2024-09-18T08:19:31.685 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:19:31.685 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:19:31.685 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "dispatched", 2024-09-18T08:19:31.686 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:16:56.724551+0000", 2024-09-18T08:19:31.686 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.2833000000000001e-05 2024-09-18T08:19:31.686 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:19:31.686 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:19:31.686 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "queued_for_pg", 2024-09-18T08:19:31.686 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:16:56.724559+0000", 2024-09-18T08:19:31.686 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 7.4889999999999998e-06 2024-09-18T08:19:31.686 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:19:31.686 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:19:31.686 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-09-18T08:19:31.686 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:16:56.724623+0000", 2024-09-18T08:19:31.686 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 6.4558000000000001e-05 2024-09-18T08:19:31.686 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:19:31.686 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:19:31.687 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "waiting for peered", 2024-09-18T08:19:31.687 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:16:56.724633+0000", 2024-09-18T08:19:31.687 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 9.6020000000000006e-06 2024-09-18T08:19:31.687 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:19:31.687 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:19:31.687 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-09-18T08:19:31.687 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:16:57.607986+0000", 2024-09-18T08:19:31.687 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.88335310199999995 2024-09-18T08:19:31.687 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:19:31.687 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:19:31.687 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "started", 2024-09-18T08:19:31.687 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:16:57.608083+0000", 2024-09-18T08:19:31.687 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 9.6996000000000004e-05 2024-09-18T08:19:31.687 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:19:31.688 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:19:31.688 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "waiting for subops from 0,6", 2024-09-18T08:19:31.688 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:16:57.608193+0000", 2024-09-18T08:19:31.688 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.00011034199999999999 2024-09-18T08:19:31.688 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:19:31.688 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:19:31.689 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:19:31.689 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7' 2024-09-18T08:19:31.690 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:19:31.690 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:19:31.690 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:16:57.612210+0000", 2024-09-18T08:19:31.690 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.004016922 2024-09-18T08:19:31.690 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:19:31.690 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:19:31.690 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:19:31.690 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:16:57.625615+0000", 2024-09-18T08:19:31.691 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.013404434 2024-09-18T08:19:31.691 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:19:31.691 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:19:31.691 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "op_commit", 2024-09-18T08:19:31.691 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:16:57.628036+0000", 2024-09-18T08:19:31.691 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.0024217459999999998 2024-09-18T08:19:31.691 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:19:31.691 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:19:31.691 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "commit_sent", 2024-09-18T08:19:31.691 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:16:57.628086+0000", 2024-09-18T08:19:31.691 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 5.003e-05 2024-09-18T08:19:31.691 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:19:31.691 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:19:31.692 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "done", 2024-09-18T08:19:31.692 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:16:57.628128+0000", 2024-09-18T08:19:31.692 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 4.1891999999999998e-05 2024-09-18T08:19:31.692 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-09-18T08:19:31.692 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-09-18T08:19:31.692 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-09-18T08:19:31.692 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:19:31.692 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:19:31.692 INFO:teuthology.orchestra.run.smithi007.stdout: "description": "osd_op(client.15966.0:130055 10.0 10:0b627839:::benchmark_data_smithi007_148834_object28335:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e366)", 2024-09-18T08:19:31.693 INFO:teuthology.orchestra.run.smithi007.stdout: "initiated_at": "2024-09-18T08:16:56.771770+0000", 2024-09-18T08:19:31.693 INFO:teuthology.orchestra.run.smithi007.stdout: "age": 154.81767876000001, 2024-09-18T08:19:31.693 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.85643498200000001, 2024-09-18T08:19:31.693 INFO:teuthology.orchestra.run.smithi007.stdout: "continuous": false, 2024-09-18T08:19:31.693 INFO:teuthology.orchestra.run.smithi007.stdout: "type_data": { 2024-09-18T08:19:31.693 INFO:teuthology.orchestra.run.smithi007.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T08:19:31.693 INFO:teuthology.orchestra.run.smithi007.stdout: "client_info": { 2024-09-18T08:19:31.693 INFO:teuthology.orchestra.run.smithi007.stdout: "client": "client.15966", 2024-09-18T08:19:31.693 INFO:teuthology.orchestra.run.smithi007.stdout: "client_addr": "172.21.15.7:0/529784850", 2024-09-18T08:19:31.693 INFO:teuthology.orchestra.run.smithi007.stdout: "tid": 130055 2024-09-18T08:19:31.693 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:19:31.693 INFO:teuthology.orchestra.run.smithi007.stdout: "events": [ 2024-09-18T08:19:31.693 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:19:31.693 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "initiated", 2024-09-18T08:19:31.694 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:16:56.771770+0000", 2024-09-18T08:19:31.694 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-09-18T08:19:31.694 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:19:31.694 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:19:31.694 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "header_read", 2024-09-18T08:19:31.694 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:16:56.771770+0000", 2024-09-18T08:19:31.694 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-09-18T08:19:31.694 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:19:31.694 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:19:31.694 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "throttled", 2024-09-18T08:19:31.694 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:16:56.771773+0000", 2024-09-18T08:19:31.694 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 2.2089999999999999e-06 2024-09-18T08:19:31.695 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:19:31.695 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:19:31.695 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "all_read", 2024-09-18T08:19:31.695 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:16:56.771778+0000", 2024-09-18T08:19:31.695 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 5.5339999999999999e-06 2024-09-18T08:19:31.695 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:19:31.695 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:19:31.695 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "dispatched", 2024-09-18T08:19:31.695 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:16:56.771790+0000", 2024-09-18T08:19:31.695 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.2055999999999999e-05 2024-09-18T08:19:31.695 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:19:31.695 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:19:31.695 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "queued_for_pg", 2024-09-18T08:19:31.695 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:16:56.771797+0000", 2024-09-18T08:19:31.696 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 7.2729999999999997e-06 2024-09-18T08:19:31.696 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:19:31.696 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:19:31.696 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-09-18T08:19:31.696 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:16:56.771860+0000", 2024-09-18T08:19:31.696 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 6.2755000000000005e-05 2024-09-18T08:19:31.696 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:19:31.696 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:19:31.696 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "waiting for peered", 2024-09-18T08:19:31.696 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:16:56.771868+0000", 2024-09-18T08:19:31.696 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 8.1110000000000008e-06 2024-09-18T08:19:31.697 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:19:31.697 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:19:31.697 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-09-18T08:19:31.697 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:16:57.610125+0000", 2024-09-18T08:19:31.697 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.83825633499999996 2024-09-18T08:19:31.697 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:19:31.697 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:19:31.697 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "started", 2024-09-18T08:19:31.697 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:16:57.610239+0000", 2024-09-18T08:19:31.697 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.00011428400000000001 2024-09-18T08:19:31.697 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:19:31.697 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:19:31.697 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "waiting for subops from 0,6", 2024-09-18T08:19:31.697 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:16:57.610383+0000", 2024-09-18T08:19:31.698 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.000143972 2024-09-18T08:19:31.698 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:19:31.698 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:19:31.698 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:19:31.698 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:16:57.612259+0000", 2024-09-18T08:19:31.698 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.001876356 2024-09-18T08:19:31.698 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:19:31.698 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:19:31.698 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:19:31.698 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:16:57.625651+0000", 2024-09-18T08:19:31.698 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.013391745 2024-09-18T08:19:31.698 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:19:31.698 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:19:31.699 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "op_commit", 2024-09-18T08:19:31.699 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:16:57.628139+0000", 2024-09-18T08:19:31.699 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.0024880890000000002 2024-09-18T08:19:31.699 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:19:31.699 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:19:31.699 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "commit_sent", 2024-09-18T08:19:31.699 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:16:57.628175+0000", 2024-09-18T08:19:31.699 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 3.5864000000000001e-05 2024-09-18T08:19:31.699 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:19:31.699 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:19:31.699 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "done", 2024-09-18T08:19:31.699 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:16:57.628205+0000", 2024-09-18T08:19:31.699 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 3.0399000000000002e-05 2024-09-18T08:19:31.699 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-09-18T08:19:31.700 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-09-18T08:19:31.700 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-09-18T08:19:31.700 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:19:31.700 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:19:31.700 INFO:teuthology.orchestra.run.smithi007.stdout: "description": "osd_op(client.15966.0:130072 10.0 10:04614435:::benchmark_data_smithi007_148834_object28352:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e366)", 2024-09-18T08:19:31.700 INFO:teuthology.orchestra.run.smithi007.stdout: "initiated_at": "2024-09-18T08:16:56.799291+0000", 2024-09-18T08:19:31.700 INFO:teuthology.orchestra.run.smithi007.stdout: "age": 154.79015828000001, 2024-09-18T08:19:31.700 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.83079228500000002, 2024-09-18T08:19:31.700 INFO:teuthology.orchestra.run.smithi007.stdout: "continuous": false, 2024-09-18T08:19:31.700 INFO:teuthology.orchestra.run.smithi007.stdout: "type_data": { 2024-09-18T08:19:31.700 INFO:teuthology.orchestra.run.smithi007.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T08:19:31.700 INFO:teuthology.orchestra.run.smithi007.stdout: "client_info": { 2024-09-18T08:19:31.700 INFO:teuthology.orchestra.run.smithi007.stdout: "client": "client.15966", 2024-09-18T08:19:31.701 INFO:teuthology.orchestra.run.smithi007.stdout: "client_addr": "172.21.15.7:0/529784850", 2024-09-18T08:19:31.701 INFO:teuthology.orchestra.run.smithi007.stdout: "tid": 130072 2024-09-18T08:19:31.701 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:19:31.701 INFO:teuthology.orchestra.run.smithi007.stdout: "events": [ 2024-09-18T08:19:31.701 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:19:31.701 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "initiated", 2024-09-18T08:19:31.701 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:16:56.799291+0000", 2024-09-18T08:19:31.701 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-09-18T08:19:31.701 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:19:31.701 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:19:31.701 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "header_read", 2024-09-18T08:19:31.701 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:16:56.799291+0000", 2024-09-18T08:19:31.701 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-09-18T08:19:31.701 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:19:31.702 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:19:31.702 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "throttled", 2024-09-18T08:19:31.702 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:16:56.799292+0000", 2024-09-18T08:19:31.702 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.2929999999999999e-06 2024-09-18T08:19:31.702 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:19:31.702 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:19:31.702 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "all_read", 2024-09-18T08:19:31.702 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:16:56.799297+0000", 2024-09-18T08:19:31.702 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 5.3179999999999997e-06 2024-09-18T08:19:31.702 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:19:31.702 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:19:31.702 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "dispatched", 2024-09-18T08:19:31.702 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:16:56.799308+0000", 2024-09-18T08:19:31.702 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.0259e-05 2024-09-18T08:19:31.703 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:19:31.703 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:19:31.703 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "queued_for_pg", 2024-09-18T08:19:31.703 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:16:56.799315+0000", 2024-09-18T08:19:31.703 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 7.0260000000000002e-06 2024-09-18T08:19:31.703 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:19:31.703 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:19:31.703 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-09-18T08:19:31.703 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:16:56.799381+0000", 2024-09-18T08:19:31.703 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 6.5959999999999999e-05 2024-09-18T08:19:31.703 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:19:31.703 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:19:31.703 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "waiting for peered", 2024-09-18T08:19:31.703 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:16:56.799390+0000", 2024-09-18T08:19:31.704 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 9.0240000000000003e-06 2024-09-18T08:19:31.704 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:19:31.704 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:19:31.704 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-09-18T08:19:31.704 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:16:57.612311+0000", 2024-09-18T08:19:31.704 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.81292116999999997 2024-09-18T08:19:31.704 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:19:31.704 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:19:31.704 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "started", 2024-09-18T08:19:31.704 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:16:57.612399+0000", 2024-09-18T08:19:31.704 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 8.8670000000000003e-05 2024-09-18T08:19:31.704 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:19:31.704 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:19:31.705 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "waiting for subops from 0,6", 2024-09-18T08:19:31.705 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:16:57.612471+0000", 2024-09-18T08:19:31.705 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 7.1784000000000003e-05 2024-09-18T08:19:31.705 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:19:31.705 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:19:31.705 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:19:31.705 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:16:57.616050+0000", 2024-09-18T08:19:31.705 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.003578426 2024-09-18T08:19:31.705 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:19:31.705 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:19:31.705 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "op_commit", 2024-09-18T08:19:31.705 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:16:57.628211+0000", 2024-09-18T08:19:31.706 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.012161122 2024-09-18T08:19:31.706 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:19:31.706 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:19:31.706 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:19:31.706 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:16:57.629993+0000", 2024-09-18T08:19:31.706 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.0017818599999999999 2024-09-18T08:19:31.706 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:19:31.706 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:19:31.706 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "commit_sent", 2024-09-18T08:19:31.706 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:16:57.630042+0000", 2024-09-18T08:19:31.706 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 4.9008999999999998e-05 2024-09-18T08:19:31.706 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:19:31.706 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:19:31.706 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "done", 2024-09-18T08:19:31.707 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:16:57.630083+0000", 2024-09-18T08:19:31.707 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 4.1363999999999999e-05 2024-09-18T08:19:31.707 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-09-18T08:19:31.707 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-09-18T08:19:31.707 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-09-18T08:19:31.707 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:19:31.707 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:19:31.707 INFO:teuthology.orchestra.run.smithi007.stdout: "description": "osd_op(client.26087.0:51055 11.f 11:f34782b0:::benchmark_data_smithi007_172546_object51054:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e413)", 2024-09-18T08:19:31.707 INFO:teuthology.orchestra.run.smithi007.stdout: "initiated_at": "2024-09-18T08:19:04.813354+0000", 2024-09-18T08:19:31.707 INFO:teuthology.orchestra.run.smithi007.stdout: "age": 26.776095416, 2024-09-18T08:19:31.707 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 2.1341305149999998, 2024-09-18T08:19:31.708 INFO:teuthology.orchestra.run.smithi007.stdout: "continuous": false, 2024-09-18T08:19:31.708 INFO:teuthology.orchestra.run.smithi007.stdout: "type_data": { 2024-09-18T08:19:31.708 INFO:teuthology.orchestra.run.smithi007.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T08:19:31.708 INFO:teuthology.orchestra.run.smithi007.stdout: "client_info": { 2024-09-18T08:19:31.708 INFO:teuthology.orchestra.run.smithi007.stdout: "client": "client.26087", 2024-09-18T08:19:31.708 INFO:teuthology.orchestra.run.smithi007.stdout: "client_addr": "172.21.15.7:0/2752273185", 2024-09-18T08:19:31.708 INFO:teuthology.orchestra.run.smithi007.stdout: "tid": 51055 2024-09-18T08:19:31.708 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:19:31.708 INFO:teuthology.orchestra.run.smithi007.stdout: "events": [ 2024-09-18T08:19:31.708 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:19:31.708 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "initiated", 2024-09-18T08:19:31.708 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:19:04.813354+0000", 2024-09-18T08:19:31.708 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-09-18T08:19:31.709 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:19:31.709 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:19:31.709 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "header_read", 2024-09-18T08:19:31.709 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:19:04.813354+0000", 2024-09-18T08:19:31.709 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-09-18T08:19:31.709 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:19:31.709 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:19:31.709 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "throttled", 2024-09-18T08:19:31.709 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:19:04.813354+0000", 2024-09-18T08:19:31.709 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 5.0999999999999999e-07 2024-09-18T08:19:31.709 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:19:31.709 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:19:31.709 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "all_read", 2024-09-18T08:19:31.710 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:19:04.813374+0000", 2024-09-18T08:19:31.710 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.9949000000000002e-05 2024-09-18T08:19:31.710 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:19:31.710 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:19:31.710 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "dispatched", 2024-09-18T08:19:31.710 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:19:04.813380+0000", 2024-09-18T08:19:31.710 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 6.3099999999999997e-06 2024-09-18T08:19:31.710 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:19:31.710 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:19:31.710 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "queued_for_pg", 2024-09-18T08:19:31.711 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:19:04.813386+0000", 2024-09-18T08:19:31.711 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 5.5140000000000001e-06 2024-09-18T08:19:31.711 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:19:31.711 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:19:31.711 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-09-18T08:19:31.711 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:19:04.813533+0000", 2024-09-18T08:19:31.711 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.000146988 2024-09-18T08:19:31.711 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:19:31.711 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:19:31.711 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "op must wait for map", 2024-09-18T08:19:31.711 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:19:04.813539+0000", 2024-09-18T08:19:31.711 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 6.088e-06 2024-09-18T08:19:31.711 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:19:31.712 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:19:31.712 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-09-18T08:19:31.712 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:19:04.854363+0000", 2024-09-18T08:19:31.712 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.040824486 2024-09-18T08:19:31.712 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:19:31.712 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:19:31.712 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "waiting for peered", 2024-09-18T08:19:31.712 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:19:04.854375+0000", 2024-09-18T08:19:31.712 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.1215e-05 2024-09-18T08:19:31.712 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:19:31.712 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:19:31.712 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-09-18T08:19:31.712 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:19:05.885949+0000", 2024-09-18T08:19:31.712 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.0315743129999999 2024-09-18T08:19:31.713 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:19:31.713 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:19:31.713 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "waiting for peered", 2024-09-18T08:19:31.713 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:19:05.885960+0000", 2024-09-18T08:19:31.713 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.101e-05 2024-09-18T08:19:31.713 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:19:31.713 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:19:31.713 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-09-18T08:19:31.713 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:19:06.942047+0000", 2024-09-18T08:19:31.713 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.0560873239999999 2024-09-18T08:19:31.713 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:19:31.713 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:19:31.713 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "started", 2024-09-18T08:19:31.714 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:19:06.942217+0000", 2024-09-18T08:19:31.714 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.00016967300000000001 2024-09-18T08:19:31.714 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:19:31.714 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:19:31.714 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "waiting for subops from 0,7", 2024-09-18T08:19:31.714 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:19:06.942412+0000", 2024-09-18T08:19:31.714 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.000195269 2024-09-18T08:19:31.714 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:19:31.714 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:19:31.714 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "op_commit", 2024-09-18T08:19:31.714 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:19:06.946097+0000", 2024-09-18T08:19:31.715 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.0036845509999999999 2024-09-18T08:19:31.715 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:19:31.715 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:19:31.715 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:19:31.715 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:19:06.946146+0000", 2024-09-18T08:19:31.715 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 4.9564999999999997e-05 2024-09-18T08:19:31.715 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:19:31.715 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:19:31.715 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:19:31.715 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:19:06.946242+0000", 2024-09-18T08:19:31.715 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 9.5289000000000002e-05 2024-09-18T08:19:31.715 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:19:31.715 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:19:31.715 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "commit_sent", 2024-09-18T08:19:31.716 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:19:06.946311+0000", 2024-09-18T08:19:31.716 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 6.8973000000000005e-05 2024-09-18T08:19:31.716 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:19:31.716 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:19:31.716 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "done", 2024-09-18T08:19:31.716 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:19:06.947484+0000", 2024-09-18T08:19:31.716 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.0011734880000000001 2024-09-18T08:19:31.716 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-09-18T08:19:31.716 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-09-18T08:19:31.716 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-09-18T08:19:31.716 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:19:31.716 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:19:31.717 INFO:teuthology.orchestra.run.smithi007.stdout: "description": "osd_op(client.26087.0:51074 11.c 11:34431b36:::benchmark_data_smithi007_172546_object51073:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e413)", 2024-09-18T08:19:31.717 INFO:teuthology.orchestra.run.smithi007.stdout: "initiated_at": "2024-09-18T08:19:04.849340+0000", 2024-09-18T08:19:31.717 INFO:teuthology.orchestra.run.smithi007.stdout: "age": 26.740109466, 2024-09-18T08:19:31.717 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 2.2211444930000002, 2024-09-18T08:19:31.717 INFO:teuthology.orchestra.run.smithi007.stdout: "continuous": false, 2024-09-18T08:19:31.717 INFO:teuthology.orchestra.run.smithi007.stdout: "type_data": { 2024-09-18T08:19:31.717 INFO:teuthology.orchestra.run.smithi007.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T08:19:31.717 INFO:teuthology.orchestra.run.smithi007.stdout: "client_info": { 2024-09-18T08:19:31.717 INFO:teuthology.orchestra.run.smithi007.stdout: "client": "client.26087", 2024-09-18T08:19:31.717 INFO:teuthology.orchestra.run.smithi007.stdout: "client_addr": "172.21.15.7:0/2752273185", 2024-09-18T08:19:31.717 INFO:teuthology.orchestra.run.smithi007.stdout: "tid": 51074 2024-09-18T08:19:31.717 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:19:31.718 INFO:teuthology.orchestra.run.smithi007.stdout: "events": [ 2024-09-18T08:19:31.718 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:19:31.718 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "initiated", 2024-09-18T08:19:31.718 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:19:04.849340+0000", 2024-09-18T08:19:31.718 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-09-18T08:19:31.718 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:19:31.718 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:19:31.718 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "header_read", 2024-09-18T08:19:31.718 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:19:04.849340+0000", 2024-09-18T08:19:31.718 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-09-18T08:19:31.718 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:19:31.718 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:19:31.718 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "throttled", 2024-09-18T08:19:31.718 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:19:04.849341+0000", 2024-09-18T08:19:31.719 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.393e-06 2024-09-18T08:19:31.719 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:19:31.719 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:19:31.719 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "all_read", 2024-09-18T08:19:31.719 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:19:04.849365+0000", 2024-09-18T08:19:31.719 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 2.4477e-05 2024-09-18T08:19:31.719 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:19:31.719 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:19:31.719 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "dispatched", 2024-09-18T08:19:31.719 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:19:04.849373+0000", 2024-09-18T08:19:31.719 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 7.3309999999999998e-06 2024-09-18T08:19:31.719 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:19:31.719 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:19:31.720 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "queued_for_pg", 2024-09-18T08:19:31.720 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:19:04.849380+0000", 2024-09-18T08:19:31.720 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 7.2660000000000004e-06 2024-09-18T08:19:31.720 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:19:31.720 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:19:31.720 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-09-18T08:19:31.720 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:19:04.849434+0000", 2024-09-18T08:19:31.720 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 5.4326000000000001e-05 2024-09-18T08:19:31.720 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:19:31.720 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:19:31.720 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "waiting for peered", 2024-09-18T08:19:31.720 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:19:04.849443+0000", 2024-09-18T08:19:31.721 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 8.6859999999999995e-06 2024-09-18T08:19:31.721 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:19:31.721 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:19:31.721 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-09-18T08:19:31.721 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:19:05.888064+0000", 2024-09-18T08:19:31.721 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.038621228 2024-09-18T08:19:31.721 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:19:31.721 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:19:31.721 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "waiting for peered", 2024-09-18T08:19:31.721 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:19:05.888075+0000", 2024-09-18T08:19:31.722 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.0497e-05 2024-09-18T08:19:31.722 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:19:31.722 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:19:31.722 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-09-18T08:19:31.722 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:19:07.052325+0000", 2024-09-18T08:19:31.722 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.1642500680000001 2024-09-18T08:19:31.722 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:19:31.722 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:19:31.722 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "started", 2024-09-18T08:19:31.722 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:19:07.052472+0000", 2024-09-18T08:19:31.722 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.00014700800000000001 2024-09-18T08:19:31.722 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:19:31.722 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:19:31.723 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "waiting for subops from 5,7", 2024-09-18T08:19:31.723 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:19:07.052638+0000", 2024-09-18T08:19:31.723 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.00016594799999999999 2024-09-18T08:19:31.723 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:19:31.723 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:19:31.723 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "op_commit", 2024-09-18T08:19:31.723 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:19:07.059257+0000", 2024-09-18T08:19:31.723 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.0066195819999999997 2024-09-18T08:19:31.723 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:19:31.723 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:19:31.723 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:19:31.723 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:19:07.059309+0000", 2024-09-18T08:19:31.723 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 5.1350999999999997e-05 2024-09-18T08:19:31.724 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:19:31.724 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:19:31.724 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:19:31.724 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:19:07.068404+0000", 2024-09-18T08:19:31.724 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.0090950890000000006 2024-09-18T08:19:31.724 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:19:31.724 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:19:31.724 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "commit_sent", 2024-09-18T08:19:31.724 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:19:07.068603+0000", 2024-09-18T08:19:31.724 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.000199352 2024-09-18T08:19:31.724 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:19:31.724 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:19:31.724 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "done", 2024-09-18T08:19:31.725 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:19:07.070484+0000", 2024-09-18T08:19:31.725 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.001880891 2024-09-18T08:19:31.725 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-09-18T08:19:31.725 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-09-18T08:19:31.725 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-09-18T08:19:31.725 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:19:31.725 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:19:31.725 INFO:teuthology.orchestra.run.smithi007.stdout: "description": "osd_op(client.26087.0:51076 11.c 11:33a5b28e:::benchmark_data_smithi007_172546_object51075:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e413)", 2024-09-18T08:19:31.725 INFO:teuthology.orchestra.run.smithi007.stdout: "initiated_at": "2024-09-18T08:19:04.853210+0000", 2024-09-18T08:19:31.725 INFO:teuthology.orchestra.run.smithi007.stdout: "age": 26.736239053999999, 2024-09-18T08:19:31.725 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 2.220148032, 2024-09-18T08:19:31.725 INFO:teuthology.orchestra.run.smithi007.stdout: "continuous": false, 2024-09-18T08:19:31.725 INFO:teuthology.orchestra.run.smithi007.stdout: "type_data": { 2024-09-18T08:19:31.725 INFO:teuthology.orchestra.run.smithi007.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T08:19:31.726 INFO:teuthology.orchestra.run.smithi007.stdout: "client_info": { 2024-09-18T08:19:31.726 INFO:teuthology.orchestra.run.smithi007.stdout: "client": "client.26087", 2024-09-18T08:19:31.726 INFO:teuthology.orchestra.run.smithi007.stdout: "client_addr": "172.21.15.7:0/2752273185", 2024-09-18T08:19:31.726 INFO:teuthology.orchestra.run.smithi007.stdout: "tid": 51076 2024-09-18T08:19:31.726 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:19:31.726 INFO:teuthology.orchestra.run.smithi007.stdout: "events": [ 2024-09-18T08:19:31.726 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:19:31.726 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "initiated", 2024-09-18T08:19:31.726 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:19:04.853210+0000", 2024-09-18T08:19:31.726 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-09-18T08:19:31.726 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:19:31.726 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:19:31.726 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "header_read", 2024-09-18T08:19:31.726 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:19:04.853210+0000", 2024-09-18T08:19:31.727 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-09-18T08:19:31.727 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:19:31.727 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:19:31.727 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "throttled", 2024-09-18T08:19:31.727 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:19:04.853211+0000", 2024-09-18T08:19:31.727 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.1200000000000001e-06 2024-09-18T08:19:31.727 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:19:31.727 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:19:31.727 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "all_read", 2024-09-18T08:19:31.727 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:19:04.853232+0000", 2024-09-18T08:19:31.727 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 2.1157000000000001e-05 2024-09-18T08:19:31.727 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:19:31.727 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:19:31.727 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "dispatched", 2024-09-18T08:19:31.728 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:19:04.853245+0000", 2024-09-18T08:19:31.728 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.3063e-05 2024-09-18T08:19:31.728 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:19:31.728 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:19:31.728 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "queued_for_pg", 2024-09-18T08:19:31.728 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:19:04.853251+0000", 2024-09-18T08:19:31.728 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 5.5430000000000001e-06 2024-09-18T08:19:31.728 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:19:31.728 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:19:31.728 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-09-18T08:19:31.728 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:19:04.853337+0000", 2024-09-18T08:19:31.728 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 8.5778000000000004e-05 2024-09-18T08:19:31.728 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:19:31.729 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:19:31.729 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "waiting for peered", 2024-09-18T08:19:31.729 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:19:04.853352+0000", 2024-09-18T08:19:31.729 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.5094e-05 2024-09-18T08:19:31.729 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:19:31.729 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:19:31.729 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-09-18T08:19:31.729 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:19:05.888103+0000", 2024-09-18T08:19:31.729 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.034750804 2024-09-18T08:19:31.729 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:19:31.729 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:19:31.729 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "waiting for peered", 2024-09-18T08:19:31.729 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:19:05.888110+0000", 2024-09-18T08:19:31.729 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 7.7940000000000003e-06 2024-09-18T08:19:31.730 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:19:31.730 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:19:31.730 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-09-18T08:19:31.730 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:19:07.054770+0000", 2024-09-18T08:19:31.730 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.1666599129999999 2024-09-18T08:19:31.730 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:19:31.730 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:19:31.730 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "started", 2024-09-18T08:19:31.730 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:19:07.054932+0000", 2024-09-18T08:19:31.730 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.00016204500000000001 2024-09-18T08:19:31.730 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:19:31.730 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:19:31.730 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "waiting for subops from 5,7", 2024-09-18T08:19:31.731 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:19:07.055079+0000", 2024-09-18T08:19:31.731 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.00014656399999999999 2024-09-18T08:19:31.731 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:19:31.731 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:19:31.731 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "op_commit", 2024-09-18T08:19:31.731 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:19:07.059264+0000", 2024-09-18T08:19:31.731 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.0041850159999999997 2024-09-18T08:19:31.731 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:19:31.731 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:19:31.731 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:19:31.731 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:19:07.060395+0000", 2024-09-18T08:19:31.731 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.001131443 2024-09-18T08:19:31.731 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:19:31.731 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:19:31.732 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:19:31.732 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:19:07.070535+0000", 2024-09-18T08:19:31.732 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.010139364999999999 2024-09-18T08:19:31.732 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:19:31.732 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:19:31.732 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "commit_sent", 2024-09-18T08:19:31.732 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:19:07.071314+0000", 2024-09-18T08:19:31.732 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.00077881000000000001 2024-09-18T08:19:31.732 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:19:31.732 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:19:31.732 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "done", 2024-09-18T08:19:31.732 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:19:07.073358+0000", 2024-09-18T08:19:31.732 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.0020445229999999999 2024-09-18T08:19:31.733 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-09-18T08:19:31.733 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-09-18T08:19:31.733 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-09-18T08:19:31.733 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:19:31.733 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:19:31.733 INFO:teuthology.orchestra.run.smithi007.stdout: "description": "osd_op(client.26087.0:51085 11.d 11:bb590128:::benchmark_data_smithi007_172546_object51084:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e413)", 2024-09-18T08:19:31.733 INFO:teuthology.orchestra.run.smithi007.stdout: "initiated_at": "2024-09-18T08:19:04.857700+0000", 2024-09-18T08:19:31.733 INFO:teuthology.orchestra.run.smithi007.stdout: "age": 26.731748619000001, 2024-09-18T08:19:31.733 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 2.2029175859999999, 2024-09-18T08:19:31.733 INFO:teuthology.orchestra.run.smithi007.stdout: "continuous": false, 2024-09-18T08:19:31.733 INFO:teuthology.orchestra.run.smithi007.stdout: "type_data": { 2024-09-18T08:19:31.733 INFO:teuthology.orchestra.run.smithi007.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T08:19:31.733 INFO:teuthology.orchestra.run.smithi007.stdout: "client_info": { 2024-09-18T08:19:31.734 INFO:teuthology.orchestra.run.smithi007.stdout: "client": "client.26087", 2024-09-18T08:19:31.734 INFO:teuthology.orchestra.run.smithi007.stdout: "client_addr": "172.21.15.7:0/2752273185", 2024-09-18T08:19:31.734 INFO:teuthology.orchestra.run.smithi007.stdout: "tid": 51085 2024-09-18T08:19:31.734 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:19:31.734 INFO:teuthology.orchestra.run.smithi007.stdout: "events": [ 2024-09-18T08:19:31.734 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:19:31.734 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "initiated", 2024-09-18T08:19:31.734 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:19:04.857700+0000", 2024-09-18T08:19:31.734 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-09-18T08:19:31.734 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:19:31.734 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:19:31.734 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "header_read", 2024-09-18T08:19:31.734 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:19:04.857700+0000", 2024-09-18T08:19:31.735 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-09-18T08:19:31.735 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:19:31.735 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:19:31.735 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "throttled", 2024-09-18T08:19:31.735 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:19:04.857702+0000", 2024-09-18T08:19:31.735 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.6080000000000001e-06 2024-09-18T08:19:31.735 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:19:31.735 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:19:31.735 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "all_read", 2024-09-18T08:19:31.735 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:19:04.857753+0000", 2024-09-18T08:19:31.735 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 5.0510999999999998e-05 2024-09-18T08:19:31.735 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:19:31.735 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:19:31.735 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "dispatched", 2024-09-18T08:19:31.736 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:19:04.857764+0000", 2024-09-18T08:19:31.736 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.1432e-05 2024-09-18T08:19:31.736 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:19:31.736 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:19:31.736 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "queued_for_pg", 2024-09-18T08:19:31.736 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:19:04.857771+0000", 2024-09-18T08:19:31.736 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 7.4760000000000001e-06 2024-09-18T08:19:31.736 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:19:31.736 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:19:31.736 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-09-18T08:19:31.736 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:19:04.857839+0000", 2024-09-18T08:19:31.736 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 6.7391000000000006e-05 2024-09-18T08:19:31.736 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:19:31.737 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:19:31.737 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "waiting for peered", 2024-09-18T08:19:31.737 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:19:04.857857+0000", 2024-09-18T08:19:31.737 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.8587e-05 2024-09-18T08:19:31.737 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:19:31.737 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:19:31.737 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-09-18T08:19:31.737 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:19:05.884176+0000", 2024-09-18T08:19:31.737 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.026318611 2024-09-18T08:19:31.737 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:19:31.737 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:19:31.737 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "waiting for peered", 2024-09-18T08:19:31.738 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:19:05.884189+0000", 2024-09-18T08:19:31.738 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.2690999999999999e-05 2024-09-18T08:19:31.738 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:19:31.738 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:19:31.738 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-09-18T08:19:31.738 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:19:07.024533+0000", 2024-09-18T08:19:31.738 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.1403445699999999 2024-09-18T08:19:31.738 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:19:31.738 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:19:31.738 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "started", 2024-09-18T08:19:31.738 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:19:07.025202+0000", 2024-09-18T08:19:31.738 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.00066900800000000002 2024-09-18T08:19:31.738 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:19:31.738 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:19:31.739 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "waiting for subops from 5,6", 2024-09-18T08:19:31.739 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:19:07.025329+0000", 2024-09-18T08:19:31.739 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.00012716600000000001 2024-09-18T08:19:31.739 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:19:31.739 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:19:31.739 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:19:31.739 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:19:07.029622+0000", 2024-09-18T08:19:31.739 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.0042925150000000002 2024-09-18T08:19:31.739 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:19:31.739 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:19:31.739 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "op_commit", 2024-09-18T08:19:31.739 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:19:07.029638+0000", 2024-09-18T08:19:31.739 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.6007000000000001e-05 2024-09-18T08:19:31.740 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:19:31.740 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:19:31.740 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:19:31.740 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:19:07.058263+0000", 2024-09-18T08:19:31.740 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.028624637000000001 2024-09-18T08:19:31.740 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:19:31.740 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:19:31.740 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "commit_sent", 2024-09-18T08:19:31.740 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:19:07.058357+0000", 2024-09-18T08:19:31.740 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 9.4077000000000001e-05 2024-09-18T08:19:31.740 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:19:31.741 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:19:31.741 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "done", 2024-09-18T08:19:31.741 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:19:07.060618+0000", 2024-09-18T08:19:31.741 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.002261299 2024-09-18T08:19:31.741 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-09-18T08:19:31.741 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-09-18T08:19:31.741 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-09-18T08:19:31.741 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:19:31.741 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:19:31.741 INFO:teuthology.orchestra.run.smithi007.stdout: "description": "osd_op(client.26087.0:51086 11.d 11:b2943f72:::benchmark_data_smithi007_172546_object51085:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e413)", 2024-09-18T08:19:31.741 INFO:teuthology.orchestra.run.smithi007.stdout: "initiated_at": "2024-09-18T08:19:04.859480+0000", 2024-09-18T08:19:31.741 INFO:teuthology.orchestra.run.smithi007.stdout: "age": 26.729968783, 2024-09-18T08:19:31.741 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 2.2035695560000002, 2024-09-18T08:19:31.742 INFO:teuthology.orchestra.run.smithi007.stdout: "continuous": false, 2024-09-18T08:19:31.742 INFO:teuthology.orchestra.run.smithi007.stdout: "type_data": { 2024-09-18T08:19:31.742 INFO:teuthology.orchestra.run.smithi007.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T08:19:31.742 INFO:teuthology.orchestra.run.smithi007.stdout: "client_info": { 2024-09-18T08:19:31.742 INFO:teuthology.orchestra.run.smithi007.stdout: "client": "client.26087", 2024-09-18T08:19:31.742 INFO:teuthology.orchestra.run.smithi007.stdout: "client_addr": "172.21.15.7:0/2752273185", 2024-09-18T08:19:31.742 INFO:teuthology.orchestra.run.smithi007.stdout: "tid": 51086 2024-09-18T08:19:31.742 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:19:31.742 INFO:teuthology.orchestra.run.smithi007.stdout: "events": [ 2024-09-18T08:19:31.742 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:19:31.742 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "initiated", 2024-09-18T08:19:31.742 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:19:04.859480+0000", 2024-09-18T08:19:31.742 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-09-18T08:19:31.743 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:19:31.743 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:19:31.743 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "header_read", 2024-09-18T08:19:31.743 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:19:04.859480+0000", 2024-09-18T08:19:31.743 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-09-18T08:19:31.743 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:19:31.743 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:19:31.743 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "throttled", 2024-09-18T08:19:31.743 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:19:04.859482+0000", 2024-09-18T08:19:31.743 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.4339999999999999e-06 2024-09-18T08:19:31.743 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:19:31.743 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:19:31.743 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "all_read", 2024-09-18T08:19:31.744 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:19:04.859514+0000", 2024-09-18T08:19:31.744 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 3.2478000000000003e-05 2024-09-18T08:19:31.744 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:19:31.744 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:19:31.744 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "dispatched", 2024-09-18T08:19:31.744 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:19:04.859529+0000", 2024-09-18T08:19:31.744 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.4538e-05 2024-09-18T08:19:31.744 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:19:31.744 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:19:31.744 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "queued_for_pg", 2024-09-18T08:19:31.744 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:19:04.859537+0000", 2024-09-18T08:19:31.744 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 8.2879999999999993e-06 2024-09-18T08:19:31.744 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:19:31.744 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:19:31.745 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-09-18T08:19:31.745 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:19:04.859596+0000", 2024-09-18T08:19:31.745 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 5.9410000000000002e-05 2024-09-18T08:19:31.745 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:19:31.745 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:19:31.745 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "waiting for peered", 2024-09-18T08:19:31.745 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:19:04.859606+0000", 2024-09-18T08:19:31.745 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 9.8139999999999995e-06 2024-09-18T08:19:31.745 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:19:31.745 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:19:31.745 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-09-18T08:19:31.745 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:19:05.884214+0000", 2024-09-18T08:19:31.745 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.0246080019999999 2024-09-18T08:19:31.746 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:19:31.746 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:19:31.746 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "waiting for peered", 2024-09-18T08:19:31.746 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:19:05.884222+0000", 2024-09-18T08:19:31.746 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 7.2869999999999999e-06 2024-09-18T08:19:31.746 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:19:31.746 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:19:31.746 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-09-18T08:19:31.746 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:19:07.027326+0000", 2024-09-18T08:19:31.746 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.1431042 2024-09-18T08:19:31.746 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:19:31.746 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:19:31.746 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "started", 2024-09-18T08:19:31.747 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:19:07.027404+0000", 2024-09-18T08:19:31.747 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 7.7916000000000004e-05 2024-09-18T08:19:31.747 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:19:31.747 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:19:31.747 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "waiting for subops from 5,6", 2024-09-18T08:19:31.747 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:19:07.027534+0000", 2024-09-18T08:19:31.747 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.00012992799999999999 2024-09-18T08:19:31.747 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:19:31.747 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:19:31.747 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "op_commit", 2024-09-18T08:19:31.747 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:19:07.029645+0000", 2024-09-18T08:19:31.747 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.0021115249999999999 2024-09-18T08:19:31.747 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:19:31.748 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:19:31.748 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:19:31.748 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:19:07.030775+0000", 2024-09-18T08:19:31.748 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.001129854 2024-09-18T08:19:31.748 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:19:31.748 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:19:31.748 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:19:31.748 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:19:07.060670+0000", 2024-09-18T08:19:31.748 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.029894771000000001 2024-09-18T08:19:31.748 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:19:31.748 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:19:31.749 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "commit_sent", 2024-09-18T08:19:31.749 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:19:07.060724+0000", 2024-09-18T08:19:31.749 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 5.4042999999999997e-05 2024-09-18T08:19:31.749 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:19:31.749 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:19:31.749 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "done", 2024-09-18T08:19:31.749 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:19:07.063050+0000", 2024-09-18T08:19:31.749 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.0023260680000000001 2024-09-18T08:19:31.749 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-09-18T08:19:31.749 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-09-18T08:19:31.750 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-09-18T08:19:31.750 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:19:31.750 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:19:31.750 INFO:teuthology.orchestra.run.smithi007.stdout: "description": "osd_op(client.26087.0:51087 11.d 11:b851bb29:::benchmark_data_smithi007_172546_object51086:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e413)", 2024-09-18T08:19:31.750 INFO:teuthology.orchestra.run.smithi007.stdout: "initiated_at": "2024-09-18T08:19:04.859556+0000", 2024-09-18T08:19:31.750 INFO:teuthology.orchestra.run.smithi007.stdout: "age": 26.729892711000002, 2024-09-18T08:19:31.750 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 2.205526045, 2024-09-18T08:19:31.750 INFO:teuthology.orchestra.run.smithi007.stdout: "continuous": false, 2024-09-18T08:19:31.750 INFO:teuthology.orchestra.run.smithi007.stdout: "type_data": { 2024-09-18T08:19:31.750 INFO:teuthology.orchestra.run.smithi007.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T08:19:31.750 INFO:teuthology.orchestra.run.smithi007.stdout: "client_info": { 2024-09-18T08:19:31.750 INFO:teuthology.orchestra.run.smithi007.stdout: "client": "client.26087", 2024-09-18T08:19:31.750 INFO:teuthology.orchestra.run.smithi007.stdout: "client_addr": "172.21.15.7:0/2752273185", 2024-09-18T08:19:31.751 INFO:teuthology.orchestra.run.smithi007.stdout: "tid": 51087 2024-09-18T08:19:31.751 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:19:31.751 INFO:teuthology.orchestra.run.smithi007.stdout: "events": [ 2024-09-18T08:19:31.751 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:19:31.751 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "initiated", 2024-09-18T08:19:31.751 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:19:04.859556+0000", 2024-09-18T08:19:31.751 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-09-18T08:19:31.751 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:19:31.751 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:19:31.751 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "header_read", 2024-09-18T08:19:31.751 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:19:04.859556+0000", 2024-09-18T08:19:31.751 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-09-18T08:19:31.751 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:19:31.751 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:19:31.752 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "throttled", 2024-09-18T08:19:31.752 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:19:04.859557+0000", 2024-09-18T08:19:31.752 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 7.8800000000000002e-07 2024-09-18T08:19:31.752 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:19:31.752 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:19:31.752 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "all_read", 2024-09-18T08:19:31.752 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:19:04.859578+0000", 2024-09-18T08:19:31.752 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 2.0380000000000001e-05 2024-09-18T08:19:31.752 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:19:31.752 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:19:31.752 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "dispatched", 2024-09-18T08:19:31.752 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:19:04.859585+0000", 2024-09-18T08:19:31.752 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 7.1690000000000003e-06 2024-09-18T08:19:31.752 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:19:31.753 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:19:31.753 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "queued_for_pg", 2024-09-18T08:19:31.753 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:19:04.859590+0000", 2024-09-18T08:19:31.753 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 5.4689999999999996e-06 2024-09-18T08:19:31.753 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:19:31.753 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:19:31.753 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-09-18T08:19:31.753 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:19:04.859646+0000", 2024-09-18T08:19:31.753 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 5.5673e-05 2024-09-18T08:19:31.753 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:19:31.753 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:19:31.753 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "waiting for peered", 2024-09-18T08:19:31.753 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:19:04.859651+0000", 2024-09-18T08:19:31.754 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 5.468e-06 2024-09-18T08:19:31.754 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:19:31.754 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:19:31.754 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-09-18T08:19:31.754 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:19:05.884247+0000", 2024-09-18T08:19:31.754 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.0245952920000001 2024-09-18T08:19:31.754 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:19:31.754 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:19:31.754 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "waiting for peered", 2024-09-18T08:19:31.754 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:19:05.884256+0000", 2024-09-18T08:19:31.754 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 9.2960000000000004e-06 2024-09-18T08:19:31.754 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:19:31.754 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:19:31.754 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-09-18T08:19:31.755 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:19:07.028371+0000", 2024-09-18T08:19:31.755 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.1441151089999999 2024-09-18T08:19:31.755 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:19:31.755 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:19:31.755 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "started", 2024-09-18T08:19:31.755 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:19:07.028487+0000", 2024-09-18T08:19:31.755 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.000116431 2024-09-18T08:19:31.755 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:19:31.755 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:19:31.755 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "waiting for subops from 5,6", 2024-09-18T08:19:31.755 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:19:07.028580+0000", 2024-09-18T08:19:31.755 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 9.2477000000000002e-05 2024-09-18T08:19:31.755 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:19:31.755 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:19:31.756 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "op_commit", 2024-09-18T08:19:31.756 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:19:07.030795+0000", 2024-09-18T08:19:31.756 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.0022150680000000002 2024-09-18T08:19:31.756 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:19:31.756 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:19:31.756 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:19:31.756 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:19:07.032896+0000", 2024-09-18T08:19:31.756 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.0021006979999999998 2024-09-18T08:19:31.756 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:19:31.756 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:19:31.756 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:19:31.756 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:19:07.063160+0000", 2024-09-18T08:19:31.756 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.030264209 2024-09-18T08:19:31.757 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:19:31.757 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:19:31.757 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "commit_sent", 2024-09-18T08:19:31.757 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:19:07.063214+0000", 2024-09-18T08:19:31.757 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 5.448e-05 2024-09-18T08:19:31.757 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:19:31.757 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:19:31.757 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "done", 2024-09-18T08:19:31.757 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:19:07.065082+0000", 2024-09-18T08:19:31.757 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.001868038 2024-09-18T08:19:31.757 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-09-18T08:19:31.757 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-09-18T08:19:31.757 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-09-18T08:19:31.757 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-09-18T08:19:31.758 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-09-18T08:19:31.758 INFO:teuthology.orchestra.run.smithi007.stdout:} 2024-09-18T08:19:31.790 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0 2024-09-18T08:19:31.829 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:19:31 smithi007 ceph-mon[29459]: from='mgr.24470 172.21.15.7:0/2594390261' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-18T08:19:31.829 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:19:31 smithi007 ceph-mon[29459]: from='mgr.24470 172.21.15.7:0/2594390261' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-18T08:19:31.829 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:19:31 smithi007 ceph-mon[29459]: from='mgr.24470 172.21.15.7:0/2594390261' entity='mgr.y' 2024-09-18T08:19:31.829 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:19:31 smithi007 ceph-mon[39249]: from='mgr.24470 172.21.15.7:0/2594390261' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-18T08:19:31.829 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:19:31 smithi007 ceph-mon[39249]: from='mgr.24470 172.21.15.7:0/2594390261' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-18T08:19:31.830 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:19:31 smithi007 ceph-mon[39249]: from='mgr.24470 172.21.15.7:0/2594390261' entity='mgr.y' 2024-09-18T08:19:31.840 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:19:31.841 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:19:31.841 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0' 2024-09-18T08:19:31.905 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:19:31 smithi050 ceph-mon[32371]: from='mgr.24470 172.21.15.7:0/2594390261' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-18T08:19:31.905 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:19:31 smithi050 ceph-mon[32371]: from='mgr.24470 172.21.15.7:0/2594390261' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-18T08:19:31.905 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:19:31 smithi050 ceph-mon[32371]: from='mgr.24470 172.21.15.7:0/2594390261' entity='mgr.y' 2024-09-18T08:19:31.942 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6 2024-09-18T08:19:31.971 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:19:31.971 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:19:31.972 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6' 2024-09-18T08:19:32.040 INFO:tasks.radosbench.radosbench.0.smithi007.stdout: 57 16 70467 70451 77.235 49.125 0.0220854 0.0129418 2024-09-18T08:19:32.073 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7 2024-09-18T08:19:32.105 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:19:32.106 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:19:32.106 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7' 2024-09-18T08:19:32.207 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6 2024-09-18T08:19:32.239 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:19:32.239 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:19:32.240 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6' 2024-09-18T08:19:32.340 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2 2024-09-18T08:19:32.384 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:19:32.385 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:19:32.385 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2' 2024-09-18T08:19:32.486 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2 2024-09-18T08:19:32.526 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:19:32.527 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:19:32.527 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2' 2024-09-18T08:19:32.628 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5 2024-09-18T08:19:32.662 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:19:32.663 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:19:32.663 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5' 2024-09-18T08:19:32.764 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0 2024-09-18T08:19:32.801 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:19:32.802 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:19:32.802 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0' 2024-09-18T08:19:32.865 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:19:32 smithi007 ceph-mon[29459]: pgmap v841: 158 pgs: 6 active+recovering+undersized+remapped, 152 active+clean; 4.1 GiB data, 17 GiB used, 609 GiB / 626 GiB avail; 727 B/s rd, 44 MiB/s wr, 709 op/s; 14523/204141 objects misplaced (7.114%); 12 MiB/s, 195 objects/s recovering 2024-09-18T08:19:32.865 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:19:32 smithi007 ceph-mon[39249]: pgmap v841: 158 pgs: 6 active+recovering+undersized+remapped, 152 active+clean; 4.1 GiB data, 17 GiB used, 609 GiB / 626 GiB avail; 727 B/s rd, 44 MiB/s wr, 709 op/s; 14523/204141 objects misplaced (7.114%); 12 MiB/s, 195 objects/s recovering 2024-09-18T08:19:32.902 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6 2024-09-18T08:19:32.926 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid 779349ca-7593-11ef-bceb-c7b262605968 -- ceph daemon osd.0 dump_ops_in_flight 2024-09-18T08:19:32.928 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:19:32.929 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:19:32.929 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6' 2024-09-18T08:19:33.030 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2 2024-09-18T08:19:33.040 INFO:tasks.radosbench.radosbench.0.smithi007.stdout: 58 16 71359 71343 76.8645 55.75 0.0136254 0.0130029 2024-09-18T08:19:33.072 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:19:33.072 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:19:33.073 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2' 2024-09-18T08:19:33.155 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:19:32 smithi050 ceph-mon[32371]: pgmap v841: 158 pgs: 6 active+recovering+undersized+remapped, 152 active+clean; 4.1 GiB data, 17 GiB used, 609 GiB / 626 GiB avail; 727 B/s rd, 44 MiB/s wr, 709 op/s; 14523/204141 objects misplaced (7.114%); 12 MiB/s, 195 objects/s recovering 2024-09-18T08:19:33.174 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2 2024-09-18T08:19:33.226 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:19:33.227 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:19:33.227 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2' 2024-09-18T08:19:33.317 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid 779349ca-7593-11ef-bceb-c7b262605968 -- ceph osd set nodeep-scrub 2024-09-18T08:19:33.328 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7 2024-09-18T08:19:33.358 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:19:33.358 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:19:33.359 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7' 2024-09-18T08:19:33.460 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4 2024-09-18T08:19:33.490 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:19:33.490 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:19:33.491 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4' 2024-09-18T08:19:33.592 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2 2024-09-18T08:19:33.632 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:19:33.632 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:19:33.632 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2' 2024-09-18T08:19:33.733 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5 2024-09-18T08:19:33.760 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:19:33.761 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:19:33.761 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5' 2024-09-18T08:19:33.862 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3 2024-09-18T08:19:33.914 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:19:33.915 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:19:33.916 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3' 2024-09-18T08:19:34.016 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1 2024-09-18T08:19:34.041 INFO:tasks.radosbench.radosbench.0.smithi007.stdout: 59 16 72122 72106 76.3694 47.6875 0.0318733 0.0130872 2024-09-18T08:19:34.064 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:19:34.065 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:19:34.065 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1' 2024-09-18T08:19:34.166 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2 2024-09-18T08:19:34.210 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:19:34.211 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:19:34.212 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2' 2024-09-18T08:19:34.313 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3 2024-09-18T08:19:34.355 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:19:34.356 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:19:34.357 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3' 2024-09-18T08:19:34.457 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3 2024-09-18T08:19:34.497 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:19:34.497 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:19:34.498 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3' 2024-09-18T08:19:34.598 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7 2024-09-18T08:19:34.628 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:19:34.628 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:19:34.629 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7' 2024-09-18T08:19:34.730 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7 2024-09-18T08:19:34.759 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:19:34.759 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:19:34.760 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7' 2024-09-18T08:19:34.817 INFO:teuthology.orchestra.run.smithi007.stdout:osd.0: {} 2024-09-18T08:19:34.818 INFO:teuthology.orchestra.run.smithi007.stdout:osd.1: {} 2024-09-18T08:19:34.818 INFO:teuthology.orchestra.run.smithi007.stdout:osd.2: {} 2024-09-18T08:19:34.818 INFO:teuthology.orchestra.run.smithi007.stdout:osd.3: {} 2024-09-18T08:19:34.818 INFO:teuthology.orchestra.run.smithi007.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-18T08:19:34.818 INFO:teuthology.orchestra.run.smithi007.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-18T08:19:34.818 INFO:teuthology.orchestra.run.smithi007.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-18T08:19:34.818 INFO:teuthology.orchestra.run.smithi007.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-18T08:19:34.819 INFO:teuthology.orchestra.run.smithi007.stdout:osd.4: {} 2024-09-18T08:19:34.819 INFO:teuthology.orchestra.run.smithi007.stdout:osd.5: {} 2024-09-18T08:19:34.819 INFO:teuthology.orchestra.run.smithi007.stdout:osd.6: {} 2024-09-18T08:19:34.819 INFO:teuthology.orchestra.run.smithi007.stdout:osd.7: {} 2024-09-18T08:19:34.819 INFO:teuthology.orchestra.run.smithi007.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-18T08:19:34.819 INFO:teuthology.orchestra.run.smithi007.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-18T08:19:34.819 INFO:teuthology.orchestra.run.smithi007.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-18T08:19:34.819 INFO:teuthology.orchestra.run.smithi007.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-18T08:19:34.861 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4 2024-09-18T08:19:34.893 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:19:34.893 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:19:34.894 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4' 2024-09-18T08:19:34.980 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:19:34 smithi007 ceph-mon[29459]: pgmap v842: 158 pgs: 6 active+recovering+undersized+remapped, 152 active+clean; 4.2 GiB data, 17 GiB used, 609 GiB / 626 GiB avail; 1.2 KiB/s rd, 39 MiB/s wr, 617 op/s; 13706/208386 objects misplaced (6.577%); 15 MiB/s, 242 objects/s recovering 2024-09-18T08:19:34.980 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:19:34 smithi007 ceph-mon[29459]: from='client.? 172.21.15.7:0/1652185363' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-18T08:19:34.980 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:19:34 smithi007 ceph-mon[29459]: from='mgr.24470 172.21.15.7:0/2594390261' entity='mgr.y' 2024-09-18T08:19:34.980 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:19:34 smithi007 ceph-mon[29459]: from='mgr.24470 172.21.15.7:0/2594390261' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-18T08:19:34.980 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:19:34 smithi007 ceph-mon[39249]: pgmap v842: 158 pgs: 6 active+recovering+undersized+remapped, 152 active+clean; 4.2 GiB data, 17 GiB used, 609 GiB / 626 GiB avail; 1.2 KiB/s rd, 39 MiB/s wr, 617 op/s; 13706/208386 objects misplaced (6.577%); 15 MiB/s, 242 objects/s recovering 2024-09-18T08:19:34.980 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:19:34 smithi007 ceph-mon[39249]: from='client.? 172.21.15.7:0/1652185363' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-18T08:19:34.980 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:19:34 smithi007 ceph-mon[39249]: from='mgr.24470 172.21.15.7:0/2594390261' entity='mgr.y' 2024-09-18T08:19:34.981 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:19:34 smithi007 ceph-mon[39249]: from='mgr.24470 172.21.15.7:0/2594390261' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-18T08:19:34.994 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2 2024-09-18T08:19:35.031 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:19:35.032 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:19:35.032 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2' 2024-09-18T08:19:35.040 INFO:tasks.radosbench.radosbench.0.smithi007.stdout:2024-09-18T08:19:35.040705+0000 min lat: 0.00108817 max lat: 2.21973 avg lat: 0.0131469 2024-09-18T08:19:35.041 INFO:tasks.radosbench.radosbench.0.smithi007.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-09-18T08:19:35.041 INFO:tasks.radosbench.radosbench.0.smithi007.stdout: 60 16 73015 72999 76.0267 55.8125 0.0202577 0.0131469 2024-09-18T08:19:35.134 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0 2024-09-18T08:19:35.158 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:19:34 smithi050 ceph-mon[32371]: pgmap v842: 158 pgs: 6 active+recovering+undersized+remapped, 152 active+clean; 4.2 GiB data, 17 GiB used, 609 GiB / 626 GiB avail; 1.2 KiB/s rd, 39 MiB/s wr, 617 op/s; 13706/208386 objects misplaced (6.577%); 15 MiB/s, 242 objects/s recovering 2024-09-18T08:19:35.158 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:19:34 smithi050 ceph-mon[32371]: from='client.? 172.21.15.7:0/1652185363' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-18T08:19:35.158 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:19:34 smithi050 ceph-mon[32371]: from='mgr.24470 172.21.15.7:0/2594390261' entity='mgr.y' 2024-09-18T08:19:35.158 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:19:34 smithi050 ceph-mon[32371]: from='mgr.24470 172.21.15.7:0/2594390261' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-18T08:19:35.205 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:19:35.205 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:19:35.206 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0' 2024-09-18T08:19:35.306 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3 2024-09-18T08:19:35.346 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:19:35.347 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:19:35.347 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3' 2024-09-18T08:19:35.448 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2 2024-09-18T08:19:35.494 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:19:35.495 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:19:35.495 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2' 2024-09-18T08:19:35.596 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0 2024-09-18T08:19:35.633 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:19:35.634 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:19:35.634 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0' 2024-09-18T08:19:35.735 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5 2024-09-18T08:19:35.766 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:19:35.766 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:19:35.767 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5' 2024-09-18T08:19:35.867 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2 2024-09-18T08:19:35.906 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:19:35.907 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:19:35.907 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2' 2024-09-18T08:19:36.003 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/779349ca-7593-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T08:19:36.008 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7 2024-09-18T08:19:36.034 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:19:36.035 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:19:36.035 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7' 2024-09-18T08:19:36.042 INFO:tasks.radosbench.radosbench.0.smithi007.stdout: 61 16 73832 73816 75.616 51.0625 0.0215534 0.0132174 2024-09-18T08:19:36.136 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6 2024-09-18T08:19:36.166 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:19:36.166 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:19:36.166 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6' 2024-09-18T08:19:36.267 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6 2024-09-18T08:19:36.296 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:19:36.297 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:19:36.297 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6' 2024-09-18T08:19:36.398 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0 2024-09-18T08:19:36.437 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:19:36.437 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:19:36.438 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0' 2024-09-18T08:19:36.539 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7 2024-09-18T08:19:36.578 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:19:36.579 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:19:36.579 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7' 2024-09-18T08:19:36.680 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4 2024-09-18T08:19:36.714 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:19:36.715 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:19:36.715 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4' 2024-09-18T08:19:36.816 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0 2024-09-18T08:19:36.872 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:19:36.875 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:19:36.876 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0' 2024-09-18T08:19:36.977 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6 2024-09-18T08:19:36.980 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Sep 18 08:19:36 smithi007 ceph-779349ca-7593-11ef-bceb-c7b262605968-mgr-y[29688]: ::ffff:172.21.15.50 - - [18/Sep/2024:08:19:36] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-09-18T08:19:36.980 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:19:36 smithi007 ceph-mon[29459]: pgmap v843: 158 pgs: 6 active+recovering+undersized+remapped, 152 active+clean; 4.4 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 1.2 KiB/s rd, 59 MiB/s wr, 944 op/s; 13288/216243 objects misplaced (6.145%); 18 MiB/s, 294 objects/s recovering 2024-09-18T08:19:36.980 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:19:36 smithi007 ceph-mon[29459]: purged_snaps scrub starts 2024-09-18T08:19:36.980 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:19:36 smithi007 ceph-mon[29459]: purged_snaps scrub ok 2024-09-18T08:19:36.980 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:19:36 smithi007 ceph-mon[39249]: pgmap v843: 158 pgs: 6 active+recovering+undersized+remapped, 152 active+clean; 4.4 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 1.2 KiB/s rd, 59 MiB/s wr, 944 op/s; 13288/216243 objects misplaced (6.145%); 18 MiB/s, 294 objects/s recovering 2024-09-18T08:19:36.980 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:19:36 smithi007 ceph-mon[39249]: purged_snaps scrub starts 2024-09-18T08:19:36.981 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:19:36 smithi007 ceph-mon[39249]: purged_snaps scrub ok 2024-09-18T08:19:37.002 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:19:37.002 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:19:37.003 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6' 2024-09-18T08:19:37.042 INFO:tasks.radosbench.radosbench.0.smithi007.stdout: 62 16 74713 74697 75.2845 55.0625 0.0165982 0.0132764 2024-09-18T08:19:37.103 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6 2024-09-18T08:19:37.132 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:19:37.133 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:19:37.134 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6' 2024-09-18T08:19:37.155 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:19:36 smithi050 ceph-mon[32371]: pgmap v843: 158 pgs: 6 active+recovering+undersized+remapped, 152 active+clean; 4.4 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 1.2 KiB/s rd, 59 MiB/s wr, 944 op/s; 13288/216243 objects misplaced (6.145%); 18 MiB/s, 294 objects/s recovering 2024-09-18T08:19:37.155 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:19:36 smithi050 ceph-mon[32371]: purged_snaps scrub starts 2024-09-18T08:19:37.155 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:19:36 smithi050 ceph-mon[32371]: purged_snaps scrub ok 2024-09-18T08:19:37.234 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7 2024-09-18T08:19:37.269 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:19:37.269 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:19:37.270 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7' 2024-09-18T08:19:37.370 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7 2024-09-18T08:19:37.399 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:19:37.400 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:19:37.400 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7' 2024-09-18T08:19:37.501 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2 2024-09-18T08:19:37.547 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:19:37.548 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:19:37.548 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2' 2024-09-18T08:19:37.649 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5 2024-09-18T08:19:37.681 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:19:37.682 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:19:37.682 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5' 2024-09-18T08:19:37.783 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4 2024-09-18T08:19:37.810 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:19:37.810 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:19:37.811 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4' 2024-09-18T08:19:37.911 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1 2024-09-18T08:19:37.957 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:19:37.958 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:19:37.958 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1' 2024-09-18T08:19:38.042 INFO:tasks.radosbench.radosbench.0.smithi007.stdout: 63 16 75651 75635 75.02 58.625 0.0474499 0.0133204 2024-09-18T08:19:38.058 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3 2024-09-18T08:19:38.095 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:19:38.096 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:19:38.096 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3' 2024-09-18T08:19:38.155 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:19:37 smithi050 ceph-mon[32371]: osdmap e425: 8 total, 8 up, 7 in 2024-09-18T08:19:38.197 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7 2024-09-18T08:19:38.229 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:19:37 smithi007 ceph-mon[29459]: osdmap e425: 8 total, 8 up, 7 in 2024-09-18T08:19:38.230 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:19:37 smithi007 ceph-mon[39249]: osdmap e425: 8 total, 8 up, 7 in 2024-09-18T08:19:38.232 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:19:38.232 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:19:38.233 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7' 2024-09-18T08:19:38.334 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5 2024-09-18T08:19:38.361 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:19:38.362 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:19:38.362 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5' 2024-09-18T08:19:38.464 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0 2024-09-18T08:19:38.513 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:19:38.513 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:19:38.513 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0' 2024-09-18T08:19:38.614 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1 2024-09-18T08:19:38.659 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:19:38.660 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:19:38.660 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1' 2024-09-18T08:19:38.761 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4 2024-09-18T08:19:38.789 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:19:38.789 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:19:38.790 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4' 2024-09-18T08:19:38.825 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid 779349ca-7593-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-18T08:19:38.891 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7 2024-09-18T08:19:38.921 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:19:38.922 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:19:38.922 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7' 2024-09-18T08:19:39.023 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6 2024-09-18T08:19:39.042 INFO:tasks.radosbench.radosbench.0.smithi007.stdout: 64 16 76435 76419 74.6131 49 0.00845952 0.013392 2024-09-18T08:19:39.049 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:19:39.049 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:19:39.050 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6' 2024-09-18T08:19:39.150 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0 2024-09-18T08:19:39.154 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:19:38 smithi050 ceph-mon[32371]: pgmap v845: 158 pgs: 6 active+recovering+undersized+remapped, 152 active+clean; 4.4 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 636 B/s rd, 41 MiB/s wr, 663 op/s; 13288/216243 objects misplaced (6.145%); 11 MiB/s, 182 objects/s recovering 2024-09-18T08:19:39.195 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:19:39.196 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:19:39.197 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0' 2024-09-18T08:19:39.229 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:19:38 smithi007 ceph-mon[29459]: pgmap v845: 158 pgs: 6 active+recovering+undersized+remapped, 152 active+clean; 4.4 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 636 B/s rd, 41 MiB/s wr, 663 op/s; 13288/216243 objects misplaced (6.145%); 11 MiB/s, 182 objects/s recovering 2024-09-18T08:19:39.230 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:19:38 smithi007 ceph-mon[39249]: pgmap v845: 158 pgs: 6 active+recovering+undersized+remapped, 152 active+clean; 4.4 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 636 B/s rd, 41 MiB/s wr, 663 op/s; 13288/216243 objects misplaced (6.145%); 11 MiB/s, 182 objects/s recovering 2024-09-18T08:19:39.297 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3 2024-09-18T08:19:39.346 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:19:39.347 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:19:39.347 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3' 2024-09-18T08:19:39.448 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6 2024-09-18T08:19:39.482 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:19:39.482 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:19:39.483 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6' 2024-09-18T08:19:39.584 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4 2024-09-18T08:19:39.607 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:19:39.607 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:19:39.608 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4' 2024-09-18T08:19:39.709 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2 2024-09-18T08:19:39.750 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:19:39.751 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:19:39.751 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2' 2024-09-18T08:19:39.852 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1 2024-09-18T08:19:39.893 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:19:39.893 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:19:39.893 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1' 2024-09-18T08:19:39.994 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4 2024-09-18T08:19:40.023 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:19:40.024 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:19:40.024 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4' 2024-09-18T08:19:40.042 INFO:tasks.radosbench.radosbench.0.smithi007.stdout: 65 16 77127 77111 74.1306 43.25 0.0259135 0.0134834 2024-09-18T08:19:40.125 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2 2024-09-18T08:19:40.155 INFO:journalctl@ceph.iscsi.iscsi.a.smithi050.stdout:Sep 18 08:19:39 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-iscsi-iscsi-a[57782]: debug there is no tcmu-runner data available 2024-09-18T08:19:40.170 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:19:40.171 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:19:40.172 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2' 2024-09-18T08:19:40.273 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1 2024-09-18T08:19:40.301 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:19:40.301 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:19:40.302 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1' 2024-09-18T08:19:40.402 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3 2024-09-18T08:19:40.445 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:19:40.446 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:19:40.446 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3' 2024-09-18T08:19:40.547 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0 2024-09-18T08:19:40.587 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:19:40.588 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:19:40.588 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0' 2024-09-18T08:19:40.689 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2 2024-09-18T08:19:40.730 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:19:40 smithi007 ceph-mon[29459]: osdmap e426: 8 total, 8 up, 7 in 2024-09-18T08:19:40.730 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:19:40 smithi007 ceph-mon[29459]: pgmap v847: 158 pgs: 6 active+recovering+undersized+remapped, 152 active+clean; 4.6 GiB data, 18 GiB used, 607 GiB / 626 GiB avail; 1.2 KiB/s rd, 53 MiB/s wr, 848 op/s; 12242/224475 objects misplaced (5.454%); 18 MiB/s, 285 objects/s recovering 2024-09-18T08:19:40.730 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:19:40 smithi007 ceph-mon[29459]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-18T08:19:40.730 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:19:40 smithi007 ceph-mon[39249]: osdmap e426: 8 total, 8 up, 7 in 2024-09-18T08:19:40.730 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:19:40 smithi007 ceph-mon[39249]: pgmap v847: 158 pgs: 6 active+recovering+undersized+remapped, 152 active+clean; 4.6 GiB data, 18 GiB used, 607 GiB / 626 GiB avail; 1.2 KiB/s rd, 53 MiB/s wr, 848 op/s; 12242/224475 objects misplaced (5.454%); 18 MiB/s, 285 objects/s recovering 2024-09-18T08:19:40.730 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:19:40 smithi007 ceph-mon[39249]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-18T08:19:40.733 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:19:40.734 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:19:40.734 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2' 2024-09-18T08:19:40.835 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4 2024-09-18T08:19:40.843 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-09-18T08:19:40.859 INFO:teuthology.orchestra.run.smithi007.stderr:dumped all 2024-09-18T08:19:40.862 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:19:40.863 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:19:40.863 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4' 2024-09-18T08:19:40.905 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:19:40 smithi050 ceph-mon[32371]: osdmap e426: 8 total, 8 up, 7 in 2024-09-18T08:19:40.905 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:19:40 smithi050 ceph-mon[32371]: pgmap v847: 158 pgs: 6 active+recovering+undersized+remapped, 152 active+clean; 4.6 GiB data, 18 GiB used, 607 GiB / 626 GiB avail; 1.2 KiB/s rd, 53 MiB/s wr, 848 op/s; 12242/224475 objects misplaced (5.454%); 18 MiB/s, 285 objects/s recovering 2024-09-18T08:19:40.905 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:19:40 smithi050 ceph-mon[32371]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-18T08:19:40.905 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/779349ca-7593-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T08:19:40.965 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1 2024-09-18T08:19:40.989 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:19:40.989 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:19:40.989 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1' 2024-09-18T08:19:41.043 INFO:tasks.radosbench.radosbench.0.smithi007.stdout: 66 16 77832 77816 73.6748 44.0625 0.00764324 0.0135668 2024-09-18T08:19:41.090 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7 2024-09-18T08:19:41.122 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:19:41.122 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:19:41.123 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7' 2024-09-18T08:19:41.223 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0 2024-09-18T08:19:41.246 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:19:41.247 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:19:41.247 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0' 2024-09-18T08:19:41.348 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7 2024-09-18T08:19:41.378 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:19:41.379 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:19:41.379 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7' 2024-09-18T08:19:41.480 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3 2024-09-18T08:19:41.509 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:19:41.509 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:19:41.510 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3' 2024-09-18T08:19:41.611 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0 2024-09-18T08:19:41.648 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:19:41.649 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:19:41.649 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0' 2024-09-18T08:19:41.730 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:19:41 smithi007 ceph-mon[29459]: from='client.25948 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-18T08:19:41.730 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:19:41 smithi007 ceph-mon[39249]: from='client.25948 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-18T08:19:41.749 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1 2024-09-18T08:19:41.793 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:19:41.793 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:19:41.794 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1' 2024-09-18T08:19:41.806 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:19:41 smithi050 ceph-mon[32371]: from='client.25948 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-18T08:19:41.894 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5 2024-09-18T08:19:41.917 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:19:41.917 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:19:41.918 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5' 2024-09-18T08:19:42.018 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0 2024-09-18T08:19:42.043 INFO:tasks.radosbench.radosbench.0.smithi007.stdout: 67 16 78582 78566 73.2747 46.875 0.0149951 0.0136413 2024-09-18T08:19:42.057 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:19:42.059 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:19:42.060 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0' 2024-09-18T08:19:42.161 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4 2024-09-18T08:19:42.186 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:19:42.186 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:19:42.187 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4' 2024-09-18T08:19:42.287 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2 2024-09-18T08:19:42.323 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:19:42.324 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:19:42.324 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2' 2024-09-18T08:19:42.425 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6 2024-09-18T08:19:42.448 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:19:42.448 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:19:42.449 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6' 2024-09-18T08:19:42.550 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6 2024-09-18T08:19:42.580 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:19:42.581 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:19:42.582 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6' 2024-09-18T08:19:42.682 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1 2024-09-18T08:19:42.721 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:19:42.722 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:19:42.722 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1' 2024-09-18T08:19:42.729 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:19:42 smithi007 ceph-mon[29459]: pgmap v848: 158 pgs: 6 active+recovering+undersized+remapped, 152 active+clean; 4.7 GiB data, 19 GiB used, 607 GiB / 626 GiB avail; 639 B/s rd, 54 MiB/s wr, 865 op/s; 11996/229149 objects misplaced (5.235%); 13 MiB/s, 213 objects/s recovering 2024-09-18T08:19:42.730 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:19:42 smithi007 ceph-mon[39249]: pgmap v848: 158 pgs: 6 active+recovering+undersized+remapped, 152 active+clean; 4.7 GiB data, 19 GiB used, 607 GiB / 626 GiB avail; 639 B/s rd, 54 MiB/s wr, 865 op/s; 11996/229149 objects misplaced (5.235%); 13 MiB/s, 213 objects/s recovering 2024-09-18T08:19:42.823 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2 2024-09-18T08:19:42.864 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:19:42.865 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:19:42.865 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2' 2024-09-18T08:19:42.905 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:19:42 smithi050 ceph-mon[32371]: pgmap v848: 158 pgs: 6 active+recovering+undersized+remapped, 152 active+clean; 4.7 GiB data, 19 GiB used, 607 GiB / 626 GiB avail; 639 B/s rd, 54 MiB/s wr, 865 op/s; 11996/229149 objects misplaced (5.235%); 13 MiB/s, 213 objects/s recovering 2024-09-18T08:19:42.966 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0 2024-09-18T08:19:43.006 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:19:43.007 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:19:43.008 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0' 2024-09-18T08:19:43.043 INFO:tasks.radosbench.radosbench.0.smithi007.stdout: 68 16 79412 79396 72.9599 51.875 0.00626643 0.0137 2024-09-18T08:19:43.108 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6 2024-09-18T08:19:43.140 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:19:43.140 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:19:43.141 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6' 2024-09-18T08:19:43.202 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/779349ca-7593-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T08:19:43.241 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2 2024-09-18T08:19:43.281 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:19:43.283 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:19:43.283 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2' 2024-09-18T08:19:43.384 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0 2024-09-18T08:19:43.423 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:19:43.424 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:19:43.424 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0' 2024-09-18T08:19:43.525 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5 2024-09-18T08:19:43.553 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:19:43.554 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:19:43.554 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5' 2024-09-18T08:19:43.655 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2 2024-09-18T08:19:43.697 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:19:43.698 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:19:43.698 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2' 2024-09-18T08:19:43.730 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:19:43 smithi007 ceph-mon[29459]: purged_snaps scrub starts 2024-09-18T08:19:43.730 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:19:43 smithi007 ceph-mon[29459]: purged_snaps scrub ok 2024-09-18T08:19:43.730 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:19:43 smithi007 ceph-mon[39249]: purged_snaps scrub starts 2024-09-18T08:19:43.730 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:19:43 smithi007 ceph-mon[39249]: purged_snaps scrub ok 2024-09-18T08:19:43.799 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0 2024-09-18T08:19:43.836 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:19:43.836 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:19:43.837 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0' 2024-09-18T08:19:43.905 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:19:43 smithi050 ceph-mon[32371]: purged_snaps scrub starts 2024-09-18T08:19:43.905 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:19:43 smithi050 ceph-mon[32371]: purged_snaps scrub ok 2024-09-18T08:19:43.937 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4 2024-09-18T08:19:43.968 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:19:43.968 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:19:43.968 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4' 2024-09-18T08:19:44.044 INFO:tasks.radosbench.radosbench.0.smithi007.stdout: 69 16 80168 80152 72.5868 47.25 0.00911917 0.0137708 2024-09-18T08:19:44.069 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4 2024-09-18T08:19:44.102 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:19:44.103 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:19:44.103 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4' 2024-09-18T08:19:44.204 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1 2024-09-18T08:19:44.246 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:19:44.246 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:19:44.247 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1' 2024-09-18T08:19:44.347 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4 2024-09-18T08:19:44.380 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:19:44.381 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:19:44.381 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4' 2024-09-18T08:19:44.482 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5 2024-09-18T08:19:44.518 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:19:44.518 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:19:44.519 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5' 2024-09-18T08:19:44.619 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4 2024-09-18T08:19:44.644 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:19:44.645 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:19:44.645 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4' 2024-09-18T08:19:44.730 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:19:44 smithi007 ceph-mon[29459]: pgmap v850: 158 pgs: 6 active+recovering+undersized+remapped, 152 active+clean; 4.7 GiB data, 19 GiB used, 607 GiB / 626 GiB avail; 1.5 KiB/s rd, 53 MiB/s wr, 849 op/s; 11177/233424 objects misplaced (4.788%); 20 MiB/s, 312 objects/s recovering 2024-09-18T08:19:44.730 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:19:44 smithi007 ceph-mon[29459]: osdmap e427: 8 total, 8 up, 7 in 2024-09-18T08:19:44.730 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:19:44 smithi007 ceph-mon[29459]: purged_snaps scrub starts 2024-09-18T08:19:44.730 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:19:44 smithi007 ceph-mon[29459]: purged_snaps scrub ok 2024-09-18T08:19:44.730 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:19:44 smithi007 ceph-mon[29459]: osdmap e428: 8 total, 8 up, 7 in 2024-09-18T08:19:44.730 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:19:44 smithi007 ceph-mon[39249]: pgmap v850: 158 pgs: 6 active+recovering+undersized+remapped, 152 active+clean; 4.7 GiB data, 19 GiB used, 607 GiB / 626 GiB avail; 1.5 KiB/s rd, 53 MiB/s wr, 849 op/s; 11177/233424 objects misplaced (4.788%); 20 MiB/s, 312 objects/s recovering 2024-09-18T08:19:44.730 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:19:44 smithi007 ceph-mon[39249]: osdmap e427: 8 total, 8 up, 7 in 2024-09-18T08:19:44.731 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:19:44 smithi007 ceph-mon[39249]: purged_snaps scrub starts 2024-09-18T08:19:44.731 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:19:44 smithi007 ceph-mon[39249]: purged_snaps scrub ok 2024-09-18T08:19:44.731 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:19:44 smithi007 ceph-mon[39249]: osdmap e428: 8 total, 8 up, 7 in 2024-09-18T08:19:44.746 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0 2024-09-18T08:19:44.795 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:19:44.796 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:19:44.796 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0' 2024-09-18T08:19:44.897 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4 2024-09-18T08:19:44.905 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:19:44 smithi050 ceph-mon[32371]: pgmap v850: 158 pgs: 6 active+recovering+undersized+remapped, 152 active+clean; 4.7 GiB data, 19 GiB used, 607 GiB / 626 GiB avail; 1.5 KiB/s rd, 53 MiB/s wr, 849 op/s; 11177/233424 objects misplaced (4.788%); 20 MiB/s, 312 objects/s recovering 2024-09-18T08:19:44.905 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:19:44 smithi050 ceph-mon[32371]: osdmap e427: 8 total, 8 up, 7 in 2024-09-18T08:19:44.905 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:19:44 smithi050 ceph-mon[32371]: purged_snaps scrub starts 2024-09-18T08:19:44.905 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:19:44 smithi050 ceph-mon[32371]: purged_snaps scrub ok 2024-09-18T08:19:44.905 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:19:44 smithi050 ceph-mon[32371]: osdmap e428: 8 total, 8 up, 7 in 2024-09-18T08:19:44.928 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:19:44.929 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:19:44.929 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4' 2024-09-18T08:19:45.030 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7 2024-09-18T08:19:45.043 INFO:tasks.radosbench.radosbench.0.smithi007.stdout: 70 16 81025 81009 72.315 53.5625 0.0195648 0.0138223 2024-09-18T08:19:45.061 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:19:45.062 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:19:45.063 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7' 2024-09-18T08:19:45.163 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4 2024-09-18T08:19:45.193 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:19:45.197 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:19:45.197 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4' 2024-09-18T08:19:45.298 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1 2024-09-18T08:19:45.343 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:19:45.344 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:19:45.345 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1' 2024-09-18T08:19:45.446 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0 2024-09-18T08:19:45.478 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:19:45.479 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:19:45.479 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0' 2024-09-18T08:19:45.581 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1 2024-09-18T08:19:45.607 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:19:45.608 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:19:45.609 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1' 2024-09-18T08:19:45.709 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1 2024-09-18T08:19:45.754 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:19:45.755 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:19:45.755 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1' 2024-09-18T08:19:45.856 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6 2024-09-18T08:19:45.881 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:19:45.881 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:19:45.882 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6' 2024-09-18T08:19:45.983 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1 2024-09-18T08:19:46.009 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:19:46.009 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:19:46.010 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1' 2024-09-18T08:19:46.044 INFO:tasks.radosbench.radosbench.0.smithi007.stdout: 71 16 81595 81579 71.7983 35.625 0.0119077 0.0139049 2024-09-18T08:19:46.111 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7 2024-09-18T08:19:46.140 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:19:46.141 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:19:46.141 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7' 2024-09-18T08:19:46.242 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1 2024-09-18T08:19:46.284 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:19:46.285 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:19:46.286 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1' 2024-09-18T08:19:46.386 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4 2024-09-18T08:19:46.419 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:19:46.420 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:19:46.421 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4' 2024-09-18T08:19:46.521 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4 2024-09-18T08:19:46.553 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:19:46.554 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:19:46.554 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4' 2024-09-18T08:19:46.655 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1 2024-09-18T08:19:46.694 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:19:46.695 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:19:46.695 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1' 2024-09-18T08:19:46.730 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:19:46 smithi007 ceph-mon[29459]: pgmap v852: 158 pgs: 6 active+recovering+undersized+remapped, 152 active+clean; 4.9 GiB data, 19 GiB used, 606 GiB / 626 GiB avail; 848 B/s rd, 55 MiB/s wr, 880 op/s; 10800/240396 objects misplaced (4.493%); 15 MiB/s, 238 objects/s recovering 2024-09-18T08:19:46.730 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Sep 18 08:19:46 smithi007 ceph-779349ca-7593-11ef-bceb-c7b262605968-mgr-y[29688]: ::ffff:172.21.15.50 - - [18/Sep/2024:08:19:46] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-09-18T08:19:46.730 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:19:46 smithi007 ceph-mon[39249]: pgmap v852: 158 pgs: 6 active+recovering+undersized+remapped, 152 active+clean; 4.9 GiB data, 19 GiB used, 606 GiB / 626 GiB avail; 848 B/s rd, 55 MiB/s wr, 880 op/s; 10800/240396 objects misplaced (4.493%); 15 MiB/s, 238 objects/s recovering 2024-09-18T08:19:46.796 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4 2024-09-18T08:19:46.832 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:19:46.833 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:19:46.833 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4' 2024-09-18T08:19:46.905 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:19:46 smithi050 ceph-mon[32371]: pgmap v852: 158 pgs: 6 active+recovering+undersized+remapped, 152 active+clean; 4.9 GiB data, 19 GiB used, 606 GiB / 626 GiB avail; 848 B/s rd, 55 MiB/s wr, 880 op/s; 10800/240396 objects misplaced (4.493%); 15 MiB/s, 238 objects/s recovering 2024-09-18T08:19:46.934 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6 2024-09-18T08:19:46.967 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:19:46.968 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:19:46.969 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6' 2024-09-18T08:19:47.045 INFO:tasks.radosbench.radosbench.0.smithi007.stdout: 72 16 82509 82493 71.5944 57.125 0.0192829 0.0139604 2024-09-18T08:19:47.069 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4 2024-09-18T08:19:47.095 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:19:47.095 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:19:47.096 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4' 2024-09-18T08:19:47.196 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2 2024-09-18T08:19:47.236 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:19:47.237 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:19:47.237 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2' 2024-09-18T08:19:47.338 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6 2024-09-18T08:19:47.375 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:19:47.376 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:19:47.376 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6' 2024-09-18T08:19:47.477 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2 2024-09-18T08:19:47.517 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:19:47.518 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:19:47.518 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2' 2024-09-18T08:19:47.619 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1 2024-09-18T08:19:47.659 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:19:47.660 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:19:47.660 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1' 2024-09-18T08:19:47.761 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2 2024-09-18T08:19:47.807 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:19:47.808 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:19:47.808 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2' 2024-09-18T08:19:47.909 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4 2024-09-18T08:19:47.945 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:19:47.945 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:19:47.946 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4' 2024-09-18T08:19:48.044 INFO:tasks.radosbench.radosbench.0.smithi007.stdout: 73 16 83469 83453 71.4355 60 0.00665427 0.0139924 2024-09-18T08:19:48.046 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2 2024-09-18T08:19:48.086 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:19:48.087 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:19:48.087 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2' 2024-09-18T08:19:48.188 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3 2024-09-18T08:19:48.234 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:19:48.235 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:19:48.235 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3' 2024-09-18T08:19:48.336 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6 2024-09-18T08:19:48.371 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:19:48.371 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:19:48.372 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6' 2024-09-18T08:19:48.472 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0 2024-09-18T08:19:48.499 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:19:48.500 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:19:48.500 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0' 2024-09-18T08:19:48.601 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1 2024-09-18T08:19:48.648 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:19:48.649 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:19:48.649 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1' 2024-09-18T08:19:48.697 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/779349ca-7593-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T08:19:48.747 INFO:teuthology.orchestra.run.smithi007.stdout:{"pg_ready":true,"pg_map":{"version":847,"stamp":"2024-09-18T08:19:39.487813+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":4891286043,"num_objects":74825,"num_object_clones":0,"num_object_copies":224475,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":12242,"num_objects_unfound":0,"num_objects_dirty":74825,"num_whiteouts":0,"num_read":1587,"num_read_kb":1339,"num_write":75348,"num_write_kb":4776109,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7509,"num_bytes_recovered":452938832,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":64026,"ondisk_log_size":64026,"up":474,"acting":468,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":480,"num_osds":7,"num_per_pool_osds":7,"num_per_pool_omap_osds":7,"kb":656216064,"kb_used":19351356,"kb_used_data":14904508,"kb_used_omap":17,"kb_used_meta":4446702,"kb_avail":636864708,"statfs":{"total":671965249536,"available":652149460992,"internally_reserved":0,"allocated":15262216192,"data_stored":15257608853,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17639,"internal_metadata":4553423641},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,0,2],"upper_bound":8},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":444203008,"num_objects":6778,"num_object_clones":0,"num_object_copies":20334,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-2281,"num_objects_unfound":0,"num_objects_dirty":6778,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":6778,"num_write_kb":433792,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2281,"num_bytes_recovered":149487616,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.003247"},"pg_stats":[{"pgid":"3.1f","version":"0'0","reported_seq":820,"reported_epoch":425,"state":"active+clean","last_fresh":"2024-09-18T08:19:36.753527+0000","last_change":"2024-09-18T08:19:05.894872+0000","last_active":"2024-09-18T08:19:36.753527+0000","last_peered":"2024-09-18T08:19:36.753527+0000","last_clean":"2024-09-18T08:19:36.753527+0000","last_became_active":"2024-09-18T08:19:05.894522+0000","last_became_peered":"2024-09-18T08:19:05.894522+0000","last_unstale":"2024-09-18T08:19:36.753527+0000","last_undegraded":"2024-09-18T08:19:36.753527+0000","last_fullsized":"2024-09-18T08:19:36.753527+0000","mapping_epoch":413,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":414,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:18:22.822559+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:46.860092+0000","last_clean_scrub_stamp":"2024-09-18T08:18:22.822559+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,6],"acting":[0,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"271'15","reported_seq":546,"reported_epoch":425,"state":"active+clean","last_fresh":"2024-09-18T08:19:37.001996+0000","last_change":"2024-09-18T08:19:13.094994+0000","last_active":"2024-09-18T08:19:37.001996+0000","last_peered":"2024-09-18T08:19:37.001996+0000","last_clean":"2024-09-18T08:19:37.001996+0000","last_became_active":"2024-09-18T08:16:13.456332+0000","last_became_peered":"2024-09-18T08:16:13.456332+0000","last_unstale":"2024-09-18T08:19:37.001996+0000","last_undegraded":"2024-09-18T08:19:37.001996+0000","last_fullsized":"2024-09-18T08:19:37.001996+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"271'15","last_scrub_stamp":"2024-09-18T08:19:13.094901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_clean_scrub_stamp":"2024-09-18T08:19:13.094901+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:20:38.578688+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":25,"num_read_kb":19,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,3],"acting":[2,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":637,"reported_epoch":425,"state":"active+clean","last_fresh":"2024-09-18T08:19:36.742365+0000","last_change":"2024-09-18T08:18:27.641641+0000","last_active":"2024-09-18T08:19:36.742365+0000","last_peered":"2024-09-18T08:19:36.742365+0000","last_clean":"2024-09-18T08:19:36.742365+0000","last_became_active":"2024-09-18T08:18:27.641323+0000","last_became_peered":"2024-09-18T08:18:27.641323+0000","last_unstale":"2024-09-18T08:19:36.742365+0000","last_undegraded":"2024-09-18T08:19:36.742365+0000","last_fullsized":"2024-09-18T08:19:36.742365+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:18:21.610930+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:50.886830+0000","last_clean_scrub_stamp":"2024-09-18T08:18:21.610930+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:19:39.039833+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":528,"reported_epoch":425,"state":"active+clean","last_fresh":"2024-09-18T08:19:36.746792+0000","last_change":"2024-09-18T08:18:27.639074+0000","last_active":"2024-09-18T08:19:36.746792+0000","last_peered":"2024-09-18T08:19:36.746792+0000","last_clean":"2024-09-18T08:19:36.746792+0000","last_became_active":"2024-09-18T08:18:27.638698+0000","last_became_peered":"2024-09-18T08:18:27.638698+0000","last_unstale":"2024-09-18T08:19:36.746792+0000","last_undegraded":"2024-09-18T08:19:36.746792+0000","last_fullsized":"2024-09-18T08:19:36.746792+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:17:40.565100+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_clean_scrub_stamp":"2024-09-18T08:17:40.565100+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":803,"reported_epoch":424,"state":"active+clean","last_fresh":"2024-09-18T08:19:29.466469+0000","last_change":"2024-09-18T08:18:27.662859+0000","last_active":"2024-09-18T08:19:29.466469+0000","last_peered":"2024-09-18T08:19:29.466469+0000","last_clean":"2024-09-18T08:19:29.466469+0000","last_became_active":"2024-09-18T08:18:27.662545+0000","last_became_peered":"2024-09-18T08:18:27.662545+0000","last_unstale":"2024-09-18T08:19:29.466469+0000","last_undegraded":"2024-09-18T08:19:29.466469+0000","last_fullsized":"2024-09-18T08:19:29.466469+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:15:51.826050+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_clean_scrub_stamp":"2024-09-18T08:15:51.826050+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":813,"reported_epoch":424,"state":"active+clean","last_fresh":"2024-09-18T08:19:29.468928+0000","last_change":"2024-09-18T08:19:05.895309+0000","last_active":"2024-09-18T08:19:29.468928+0000","last_peered":"2024-09-18T08:19:29.468928+0000","last_clean":"2024-09-18T08:19:29.468928+0000","last_became_active":"2024-09-18T08:19:05.894958+0000","last_became_peered":"2024-09-18T08:19:05.894958+0000","last_unstale":"2024-09-18T08:19:29.468928+0000","last_undegraded":"2024-09-18T08:19:29.468928+0000","last_fullsized":"2024-09-18T08:19:29.468928+0000","mapping_epoch":413,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":414,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:17:38.359366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:46.860092+0000","last_clean_scrub_stamp":"2024-09-18T08:17:38.359366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,1],"acting":[3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"271'27","reported_seq":820,"reported_epoch":424,"state":"active+clean","last_fresh":"2024-09-18T08:19:29.465974+0000","last_change":"2024-09-18T08:19:05.927160+0000","last_active":"2024-09-18T08:19:29.465974+0000","last_peered":"2024-09-18T08:19:29.465974+0000","last_clean":"2024-09-18T08:19:29.465974+0000","last_became_active":"2024-09-18T08:19:05.907041+0000","last_became_peered":"2024-09-18T08:19:05.907041+0000","last_unstale":"2024-09-18T08:19:29.465974+0000","last_undegraded":"2024-09-18T08:19:29.465974+0000","last_fullsized":"2024-09-18T08:19:29.465974+0000","mapping_epoch":413,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":414,"parent":"0.0","parent_split_bits":0,"last_scrub":"271'27","last_scrub_stamp":"2024-09-18T08:14:26.565071+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_clean_scrub_stamp":"2024-09-18T08:14:26.565071+0000","objects_scrubbed":7,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":44,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":440,"reported_epoch":425,"state":"active+clean","last_fresh":"2024-09-18T08:19:36.749197+0000","last_change":"2024-09-18T08:19:05.886692+0000","last_active":"2024-09-18T08:19:36.749197+0000","last_peered":"2024-09-18T08:19:36.749197+0000","last_clean":"2024-09-18T08:19:36.749197+0000","last_became_active":"2024-09-18T08:19:05.886289+0000","last_became_peered":"2024-09-18T08:19:05.886289+0000","last_unstale":"2024-09-18T08:19:36.749197+0000","last_undegraded":"2024-09-18T08:19:36.749197+0000","last_fullsized":"2024-09-18T08:19:36.749197+0000","mapping_epoch":413,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":414,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:15:44.585123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:50.886830+0000","last_clean_scrub_stamp":"2024-09-18T08:15:44.585123+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,7],"acting":[1,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":204,"reported_epoch":425,"state":"active+clean","last_fresh":"2024-09-18T08:19:36.748347+0000","last_change":"2024-09-18T08:18:17.566526+0000","last_active":"2024-09-18T08:19:36.748347+0000","last_peered":"2024-09-18T08:19:36.748347+0000","last_clean":"2024-09-18T08:19:36.748347+0000","last_became_active":"2024-09-18T08:16:13.458654+0000","last_became_peered":"2024-09-18T08:16:13.458654+0000","last_unstale":"2024-09-18T08:19:36.748347+0000","last_undegraded":"2024-09-18T08:19:36.748347+0000","last_fullsized":"2024-09-18T08:19:36.748347+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:18:17.566467+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:46.860092+0000","last_clean_scrub_stamp":"2024-09-18T08:18:17.566467+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,1],"acting":[5,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"271'15","reported_seq":423,"reported_epoch":424,"state":"active+clean","last_fresh":"2024-09-18T08:19:29.468030+0000","last_change":"2024-09-18T08:19:05.919925+0000","last_active":"2024-09-18T08:19:29.468030+0000","last_peered":"2024-09-18T08:19:29.468030+0000","last_clean":"2024-09-18T08:19:29.468030+0000","last_became_active":"2024-09-18T08:19:05.906571+0000","last_became_peered":"2024-09-18T08:19:05.906571+0000","last_unstale":"2024-09-18T08:19:29.468030+0000","last_undegraded":"2024-09-18T08:19:29.468030+0000","last_fullsized":"2024-09-18T08:19:29.468030+0000","mapping_epoch":413,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":414,"parent":"0.0","parent_split_bits":0,"last_scrub":"271'15","last_scrub_stamp":"2024-09-18T08:18:31.669089+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_clean_scrub_stamp":"2024-09-18T08:18:31.669089+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:19:43.129618+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":24,"num_read_kb":18,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":219,"reported_epoch":425,"state":"active+clean","last_fresh":"2024-09-18T08:19:36.747040+0000","last_change":"2024-09-18T08:19:03.725392+0000","last_active":"2024-09-18T08:19:36.747040+0000","last_peered":"2024-09-18T08:19:36.747040+0000","last_clean":"2024-09-18T08:19:36.747040+0000","last_became_active":"2024-09-18T08:18:27.650998+0000","last_became_peered":"2024-09-18T08:18:27.650998+0000","last_unstale":"2024-09-18T08:19:36.747040+0000","last_undegraded":"2024-09-18T08:19:36.747040+0000","last_fullsized":"2024-09-18T08:19:36.747040+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:19:03.725341+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:50.886830+0000","last_clean_scrub_stamp":"2024-09-18T08:19:03.725341+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:20:06.493697+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":779,"reported_epoch":425,"state":"active+clean","last_fresh":"2024-09-18T08:19:36.746468+0000","last_change":"2024-09-18T08:18:27.645124+0000","last_active":"2024-09-18T08:19:36.746468+0000","last_peered":"2024-09-18T08:19:36.746468+0000","last_clean":"2024-09-18T08:19:36.746468+0000","last_became_active":"2024-09-18T08:18:27.644825+0000","last_became_peered":"2024-09-18T08:18:27.644825+0000","last_unstale":"2024-09-18T08:19:36.746468+0000","last_undegraded":"2024-09-18T08:19:36.746468+0000","last_fullsized":"2024-09-18T08:19:36.746468+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:17:29.542288+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_clean_scrub_stamp":"2024-09-18T08:17:29.542288+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:19:52.045949+0000","scrub_duration":18,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":196,"reported_epoch":425,"state":"active+clean","last_fresh":"2024-09-18T08:19:36.747148+0000","last_change":"2024-09-18T08:16:13.471332+0000","last_active":"2024-09-18T08:19:36.747148+0000","last_peered":"2024-09-18T08:19:36.747148+0000","last_clean":"2024-09-18T08:19:36.747148+0000","last_became_active":"2024-09-18T08:16:13.471003+0000","last_became_peered":"2024-09-18T08:16:13.471003+0000","last_unstale":"2024-09-18T08:19:36.747148+0000","last_undegraded":"2024-09-18T08:19:36.747148+0000","last_fullsized":"2024-09-18T08:19:36.747148+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:15:01.636504+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:46.860092+0000","last_clean_scrub_stamp":"2024-09-18T08:15:01.636504+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:19:45.654590+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1b","version":"271'7","reported_seq":93,"reported_epoch":425,"state":"active+clean","last_fresh":"2024-09-18T08:19:36.746805+0000","last_change":"2024-09-18T08:19:05.904504+0000","last_active":"2024-09-18T08:19:36.746805+0000","last_peered":"2024-09-18T08:19:36.746805+0000","last_clean":"2024-09-18T08:19:36.746805+0000","last_became_active":"2024-09-18T08:19:05.885633+0000","last_became_peered":"2024-09-18T08:19:05.885633+0000","last_unstale":"2024-09-18T08:19:36.746805+0000","last_undegraded":"2024-09-18T08:19:36.746805+0000","last_fullsized":"2024-09-18T08:19:36.746805+0000","mapping_epoch":413,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":414,"parent":"0.0","parent_split_bits":0,"last_scrub":"271'7","last_scrub_stamp":"2024-09-18T08:18:27.484925+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_clean_scrub_stamp":"2024-09-18T08:18:27.484925+0000","objects_scrubbed":2,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":13,"num_read_kb":9,"num_write":8,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":58,"reported_epoch":425,"state":"active+clean","last_fresh":"2024-09-18T08:19:36.748034+0000","last_change":"2024-09-18T08:18:27.656772+0000","last_active":"2024-09-18T08:19:36.748034+0000","last_peered":"2024-09-18T08:19:36.748034+0000","last_clean":"2024-09-18T08:19:36.748034+0000","last_became_active":"2024-09-18T08:18:27.656359+0000","last_became_peered":"2024-09-18T08:18:27.656359+0000","last_unstale":"2024-09-18T08:19:36.748034+0000","last_undegraded":"2024-09-18T08:19:36.748034+0000","last_fullsized":"2024-09-18T08:19:36.748034+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:15:40.589048+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:50.886830+0000","last_clean_scrub_stamp":"2024-09-18T08:15:40.589048+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:19:42.652590+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":197,"reported_epoch":425,"state":"active+clean","last_fresh":"2024-09-18T08:19:36.747123+0000","last_change":"2024-09-18T08:18:22.573319+0000","last_active":"2024-09-18T08:19:36.747123+0000","last_peered":"2024-09-18T08:19:36.747123+0000","last_clean":"2024-09-18T08:19:36.747123+0000","last_became_active":"2024-09-18T08:15:41.441932+0000","last_became_peered":"2024-09-18T08:15:41.441932+0000","last_unstale":"2024-09-18T08:19:36.747123+0000","last_undegraded":"2024-09-18T08:19:36.747123+0000","last_fullsized":"2024-09-18T08:19:36.747123+0000","mapping_epoch":336,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":337,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:18:22.573259+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_clean_scrub_stamp":"2024-09-18T08:18:22.573259+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":503,"reported_epoch":425,"state":"active+clean","last_fresh":"2024-09-18T08:19:36.747637+0000","last_change":"2024-09-18T08:16:13.453737+0000","last_active":"2024-09-18T08:19:36.747637+0000","last_peered":"2024-09-18T08:19:36.747637+0000","last_clean":"2024-09-18T08:19:36.747637+0000","last_became_active":"2024-09-18T08:16:13.453129+0000","last_became_peered":"2024-09-18T08:16:13.453129+0000","last_unstale":"2024-09-18T08:19:36.747637+0000","last_undegraded":"2024-09-18T08:19:36.747637+0000","last_fullsized":"2024-09-18T08:19:36.747637+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:15:42.600840+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_clean_scrub_stamp":"2024-09-18T08:15:42.600840+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:19:51.734779+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,3],"acting":[5,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":815,"reported_epoch":425,"state":"active+clean","last_fresh":"2024-09-18T08:19:36.753147+0000","last_change":"2024-09-18T08:18:30.893923+0000","last_active":"2024-09-18T08:19:36.753147+0000","last_peered":"2024-09-18T08:19:36.753147+0000","last_clean":"2024-09-18T08:19:36.753147+0000","last_became_active":"2024-09-18T08:18:27.643517+0000","last_became_peered":"2024-09-18T08:18:27.643517+0000","last_unstale":"2024-09-18T08:19:36.753147+0000","last_undegraded":"2024-09-18T08:19:36.753147+0000","last_fullsized":"2024-09-18T08:19:36.753147+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:18:30.893875+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:46.860092+0000","last_clean_scrub_stamp":"2024-09-18T08:18:30.893875+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:19:46.217695+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,1],"acting":[0,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1c","version":"271'27","reported_seq":387,"reported_epoch":425,"state":"active+clean","last_fresh":"2024-09-18T08:19:36.759400+0000","last_change":"2024-09-18T08:19:05.933129+0000","last_active":"2024-09-18T08:19:36.759400+0000","last_peered":"2024-09-18T08:19:36.759400+0000","last_clean":"2024-09-18T08:19:36.759400+0000","last_became_active":"2024-09-18T08:19:05.912045+0000","last_became_peered":"2024-09-18T08:19:05.912045+0000","last_unstale":"2024-09-18T08:19:36.759400+0000","last_undegraded":"2024-09-18T08:19:36.759400+0000","last_fullsized":"2024-09-18T08:19:36.759400+0000","mapping_epoch":413,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":414,"parent":"0.0","parent_split_bits":0,"last_scrub":"271'27","last_scrub_stamp":"2024-09-18T08:14:40.711228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_clean_scrub_stamp":"2024-09-18T08:14:40.711228+0000","objects_scrubbed":7,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":44,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,0],"acting":[1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":666,"reported_epoch":425,"state":"active+clean","last_fresh":"2024-09-18T08:19:36.755538+0000","last_change":"2024-09-18T08:19:01.648780+0000","last_active":"2024-09-18T08:19:36.755538+0000","last_peered":"2024-09-18T08:19:36.755538+0000","last_clean":"2024-09-18T08:19:36.755538+0000","last_became_active":"2024-09-18T08:16:13.462303+0000","last_became_peered":"2024-09-18T08:16:13.462303+0000","last_unstale":"2024-09-18T08:19:36.755538+0000","last_undegraded":"2024-09-18T08:19:36.755538+0000","last_fullsized":"2024-09-18T08:19:36.755538+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:19:01.648717+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:50.886830+0000","last_clean_scrub_stamp":"2024-09-18T08:19:01.648717+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:20:20.414872+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,5],"acting":[1,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":784,"reported_epoch":424,"state":"active+clean","last_fresh":"2024-09-18T08:19:29.469110+0000","last_change":"2024-09-18T08:15:41.469278+0000","last_active":"2024-09-18T08:19:29.469110+0000","last_peered":"2024-09-18T08:19:29.469110+0000","last_clean":"2024-09-18T08:19:29.469110+0000","last_became_active":"2024-09-18T08:15:41.466371+0000","last_became_peered":"2024-09-18T08:15:41.466371+0000","last_unstale":"2024-09-18T08:19:29.469110+0000","last_undegraded":"2024-09-18T08:19:29.469110+0000","last_fullsized":"2024-09-18T08:19:29.469110+0000","mapping_epoch":336,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":337,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:15:05.719269+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_clean_scrub_stamp":"2024-09-18T08:15:05.719269+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:19:51.634439+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":440,"reported_epoch":425,"state":"active+clean","last_fresh":"2024-09-18T08:19:36.743064+0000","last_change":"2024-09-18T08:19:05.900071+0000","last_active":"2024-09-18T08:19:36.743064+0000","last_peered":"2024-09-18T08:19:36.743064+0000","last_clean":"2024-09-18T08:19:36.743064+0000","last_became_active":"2024-09-18T08:19:05.899551+0000","last_became_peered":"2024-09-18T08:19:05.899551+0000","last_unstale":"2024-09-18T08:19:36.743064+0000","last_undegraded":"2024-09-18T08:19:36.743064+0000","last_fullsized":"2024-09-18T08:19:36.743064+0000","mapping_epoch":413,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":414,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:14:40.774086+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:46.860092+0000","last_clean_scrub_stamp":"2024-09-18T08:14:40.774086+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,7],"acting":[1,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1d","version":"271'18","reported_seq":906,"reported_epoch":424,"state":"active+clean","last_fresh":"2024-09-18T08:19:29.468171+0000","last_change":"2024-09-18T08:18:19.829522+0000","last_active":"2024-09-18T08:19:29.468171+0000","last_peered":"2024-09-18T08:19:29.468171+0000","last_clean":"2024-09-18T08:19:29.468171+0000","last_became_active":"2024-09-18T08:18:19.823337+0000","last_became_peered":"2024-09-18T08:18:19.823337+0000","last_unstale":"2024-09-18T08:19:29.468171+0000","last_undegraded":"2024-09-18T08:19:29.468171+0000","last_fullsized":"2024-09-18T08:19:29.468171+0000","mapping_epoch":397,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":398,"parent":"0.0","parent_split_bits":0,"last_scrub":"271'18","last_scrub_stamp":"2024-09-18T08:17:04.627913+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_clean_scrub_stamp":"2024-09-18T08:17:04.627913+0000","objects_scrubbed":5,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":34,"num_read_kb":25,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":23,"num_bytes_recovered":880,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":214,"reported_epoch":424,"state":"active+clean","last_fresh":"2024-09-18T08:19:29.467442+0000","last_change":"2024-09-18T08:19:05.902064+0000","last_active":"2024-09-18T08:19:29.467442+0000","last_peered":"2024-09-18T08:19:29.467442+0000","last_clean":"2024-09-18T08:19:29.467442+0000","last_became_active":"2024-09-18T08:19:05.901172+0000","last_became_peered":"2024-09-18T08:19:05.901172+0000","last_unstale":"2024-09-18T08:19:29.467442+0000","last_undegraded":"2024-09-18T08:19:29.467442+0000","last_fullsized":"2024-09-18T08:19:29.467442+0000","mapping_epoch":413,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":414,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:18:18.614698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:50.886830+0000","last_clean_scrub_stamp":"2024-09-18T08:18:18.614698+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:19:40.078322+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.1c","version":"69'1","reported_seq":204,"reported_epoch":425,"state":"active+clean","last_fresh":"2024-09-18T08:19:36.747241+0000","last_change":"2024-09-18T08:18:27.662597+0000","last_active":"2024-09-18T08:19:36.747241+0000","last_peered":"2024-09-18T08:19:36.747241+0000","last_clean":"2024-09-18T08:19:36.747241+0000","last_became_active":"2024-09-18T08:18:27.642951+0000","last_became_peered":"2024-09-18T08:18:27.642951+0000","last_unstale":"2024-09-18T08:19:36.747241+0000","last_undegraded":"2024-09-18T08:19:36.747241+0000","last_fullsized":"2024-09-18T08:19:36.747241+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'1","last_scrub_stamp":"2024-09-18T08:15:06.708980+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_clean_scrub_stamp":"2024-09-18T08:15:06.708980+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:19:44.618856+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":403,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":1209,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,2],"acting":[5,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"62'1","reported_seq":671,"reported_epoch":425,"state":"active+clean","last_fresh":"2024-09-18T08:19:36.742461+0000","last_change":"2024-09-18T08:17:37.618000+0000","last_active":"2024-09-18T08:19:36.742461+0000","last_peered":"2024-09-18T08:19:36.742461+0000","last_clean":"2024-09-18T08:19:36.742461+0000","last_became_active":"2024-09-18T08:16:13.483397+0000","last_became_peered":"2024-09-18T08:16:13.483397+0000","last_unstale":"2024-09-18T08:19:36.742461+0000","last_undegraded":"2024-09-18T08:19:36.742461+0000","last_fullsized":"2024-09-18T08:19:36.742461+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"62'1","last_scrub_stamp":"2024-09-18T08:17:37.617954+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:46.860092+0000","last_clean_scrub_stamp":"2024-09-18T08:17:37.617954+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":46,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,6],"acting":[1,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1e","version":"271'20","reported_seq":838,"reported_epoch":425,"state":"active+clean","last_fresh":"2024-09-18T08:19:36.751499+0000","last_change":"2024-09-18T08:18:27.690042+0000","last_active":"2024-09-18T08:19:36.751499+0000","last_peered":"2024-09-18T08:19:36.751499+0000","last_clean":"2024-09-18T08:19:36.751499+0000","last_became_active":"2024-09-18T08:18:27.657472+0000","last_became_peered":"2024-09-18T08:18:27.657472+0000","last_unstale":"2024-09-18T08:19:36.751499+0000","last_undegraded":"2024-09-18T08:19:36.751499+0000","last_fullsized":"2024-09-18T08:19:36.751499+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":0,"last_scrub":"271'20","last_scrub_stamp":"2024-09-18T08:15:40.251706+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_clean_scrub_stamp":"2024-09-18T08:15:40.251706+0000","objects_scrubbed":5,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:19:51.004787+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":25,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1f","version":"415'17","reported_seq":88,"reported_epoch":425,"state":"active+clean","last_fresh":"2024-09-18T08:19:36.747701+0000","last_change":"2024-09-18T08:19:05.911190+0000","last_active":"2024-09-18T08:19:36.747701+0000","last_peered":"2024-09-18T08:19:36.747701+0000","last_clean":"2024-09-18T08:19:36.747701+0000","last_became_active":"2024-09-18T08:19:05.901152+0000","last_became_peered":"2024-09-18T08:19:05.901152+0000","last_unstale":"2024-09-18T08:19:36.747701+0000","last_undegraded":"2024-09-18T08:19:36.747701+0000","last_fullsized":"2024-09-18T08:19:36.747701+0000","mapping_epoch":413,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":414,"parent":"0.0","parent_split_bits":0,"last_scrub":"349'15","last_scrub_stamp":"2024-09-18T08:17:29.878376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:50.886830+0000","last_clean_scrub_stamp":"2024-09-18T08:17:29.878376+0000","objects_scrubbed":1,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.0","version":"425'4763","reported_seq":4822,"reported_epoch":425,"state":"active+clean","last_fresh":"2024-09-18T08:19:37.835559+0000","last_change":"2024-09-18T08:18:27.643125+0000","last_active":"2024-09-18T08:19:37.835559+0000","last_peered":"2024-09-18T08:19:37.835559+0000","last_clean":"2024-09-18T08:19:37.835559+0000","last_became_active":"2024-09-18T08:18:27.642627+0000","last_became_peered":"2024-09-18T08:18:27.642627+0000","last_unstale":"2024-09-18T08:19:37.835559+0000","last_undegraded":"2024-09-18T08:19:37.835559+0000","last_fullsized":"2024-09-18T08:19:37.835559+0000","mapping_epoch":401,"log_start":"408'1100","ondisk_log_start":"408'1100","created":401,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:18:26.612016+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:18:26.612016+0000","last_clean_scrub_stamp":"2024-09-18T08:18:26.612016+0000","objects_scrubbed":0,"log_size":3663,"log_dups_size":0,"ondisk_log_size":3663,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":312147968,"num_objects":4763,"num_object_clones":0,"num_object_copies":14289,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4763,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4763,"num_write_kb":304832,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,7],"acting":[1,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.a","version":"74'79","reported_seq":197,"reported_epoch":425,"state":"active+clean","last_fresh":"2024-09-18T08:19:36.747125+0000","last_change":"2024-09-18T08:15:42.288426+0000","last_active":"2024-09-18T08:19:36.747125+0000","last_peered":"2024-09-18T08:19:36.747125+0000","last_clean":"2024-09-18T08:19:36.747125+0000","last_became_active":"2024-09-18T08:15:41.443376+0000","last_became_peered":"2024-09-18T08:15:41.443376+0000","last_unstale":"2024-09-18T08:19:36.747125+0000","last_undegraded":"2024-09-18T08:19:36.747125+0000","last_fullsized":"2024-09-18T08:19:36.747125+0000","mapping_epoch":336,"log_start":"0'0","ondisk_log_start":"0'0","created":152,"last_epoch_clean":337,"parent":"0.0","parent_split_bits":4,"last_scrub":"74'79","last_scrub_stamp":"2024-09-18T08:15:42.288363+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:00:11.893742+0000","last_clean_scrub_stamp":"2024-09-18T08:15:42.288363+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:19:48.630023+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,6],"acting":[5,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.f","version":"271'27","reported_seq":758,"reported_epoch":425,"state":"active+clean","last_fresh":"2024-09-18T08:19:36.748194+0000","last_change":"2024-09-18T08:18:57.614756+0000","last_active":"2024-09-18T08:19:36.748194+0000","last_peered":"2024-09-18T08:19:36.748194+0000","last_clean":"2024-09-18T08:19:36.748194+0000","last_became_active":"2024-09-18T08:18:27.652553+0000","last_became_peered":"2024-09-18T08:18:27.652553+0000","last_unstale":"2024-09-18T08:19:36.748194+0000","last_undegraded":"2024-09-18T08:19:36.748194+0000","last_fullsized":"2024-09-18T08:19:36.748194+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":0,"last_scrub":"271'27","last_scrub_stamp":"2024-09-18T08:18:57.614509+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_clean_scrub_stamp":"2024-09-18T08:18:57.614509+0000","objects_scrubbed":7,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:20:21.981757+0000","scrub_duration":16,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":43,"num_read_kb":34,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,7],"acting":[1,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":81,"reported_epoch":425,"state":"active+clean","last_fresh":"2024-09-18T08:19:37.001878+0000","last_change":"2024-09-18T08:18:27.655637+0000","last_active":"2024-09-18T08:19:37.001878+0000","last_peered":"2024-09-18T08:19:37.001878+0000","last_clean":"2024-09-18T08:19:37.001878+0000","last_became_active":"2024-09-18T08:18:27.655321+0000","last_became_peered":"2024-09-18T08:18:27.655321+0000","last_unstale":"2024-09-18T08:19:37.001878+0000","last_undegraded":"2024-09-18T08:19:37.001878+0000","last_fullsized":"2024-09-18T08:19:37.001878+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:18:23.067375+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:46.860092+0000","last_clean_scrub_stamp":"2024-09-18T08:18:23.067375+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:20:30.072236+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,7],"acting":[2,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.e","version":"415'19","reported_seq":699,"reported_epoch":425,"state":"active+clean","last_fresh":"2024-09-18T08:19:36.747776+0000","last_change":"2024-09-18T08:19:06.698834+0000","last_active":"2024-09-18T08:19:36.747776+0000","last_peered":"2024-09-18T08:19:36.747776+0000","last_clean":"2024-09-18T08:19:36.747776+0000","last_became_active":"2024-09-18T08:19:05.903139+0000","last_became_peered":"2024-09-18T08:19:05.903139+0000","last_unstale":"2024-09-18T08:19:36.747776+0000","last_undegraded":"2024-09-18T08:19:36.747776+0000","last_fullsized":"2024-09-18T08:19:36.747776+0000","mapping_epoch":413,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":414,"parent":"0.0","parent_split_bits":0,"last_scrub":"349'16","last_scrub_stamp":"2024-09-18T08:19:06.698769+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:50.886830+0000","last_clean_scrub_stamp":"2024-09-18T08:19:06.698769+0000","objects_scrubbed":1,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:20:10.550844+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,3],"acting":[5,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":197,"reported_epoch":425,"state":"active+clean","last_fresh":"2024-09-18T08:19:36.747386+0000","last_change":"2024-09-18T08:18:56.732088+0000","last_active":"2024-09-18T08:19:36.747386+0000","last_peered":"2024-09-18T08:19:36.747386+0000","last_clean":"2024-09-18T08:19:36.747386+0000","last_became_active":"2024-09-18T08:15:41.438610+0000","last_became_peered":"2024-09-18T08:15:41.438610+0000","last_unstale":"2024-09-18T08:19:36.747386+0000","last_undegraded":"2024-09-18T08:19:36.747386+0000","last_fullsized":"2024-09-18T08:19:36.747386+0000","mapping_epoch":336,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":337,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:18:56.732041+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_clean_scrub_stamp":"2024-09-18T08:18:56.732041+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:20:10.266656+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.f","version":"425'4700","reported_seq":6753,"reported_epoch":425,"state":"active+recovering+undersized+remapped","last_fresh":"2024-09-18T08:19:37.881894+0000","last_change":"2024-09-18T08:19:06.955179+0000","last_active":"2024-09-18T08:19:37.881894+0000","last_peered":"2024-09-18T08:19:37.881894+0000","last_clean":"2024-09-18T08:19:04.806063+0000","last_became_active":"2024-09-18T08:19:06.938699+0000","last_became_peered":"2024-09-18T08:19:06.938699+0000","last_unstale":"2024-09-18T08:19:37.881894+0000","last_undegraded":"2024-09-18T08:19:37.881894+0000","last_fullsized":"2024-09-18T08:19:06.820984+0000","mapping_epoch":414,"log_start":"408'1100","ondisk_log_start":"408'1100","created":401,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:18:26.612016+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:18:26.612016+0000","last_clean_scrub_stamp":"2024-09-18T08:18:26.612016+0000","objects_scrubbed":0,"log_size":3600,"log_dups_size":0,"ondisk_log_size":3600,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":308019200,"num_objects":4700,"num_object_clones":0,"num_object_copies":14100,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":2168,"num_objects_unfound":0,"num_objects_dirty":4700,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4700,"num_write_kb":300800,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":995,"num_bytes_recovered":65208320,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7],"avail_no_missing":["1","2","7"],"object_location_counts":[{"shards":"1,2,7","objects":2168},{"shards":"1,7","objects":2532}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.5","version":"74'79","reported_seq":88,"reported_epoch":425,"state":"active+clean","last_fresh":"2024-09-18T08:19:36.748245+0000","last_change":"2024-09-18T08:19:05.894246+0000","last_active":"2024-09-18T08:19:36.748245+0000","last_peered":"2024-09-18T08:19:36.748245+0000","last_clean":"2024-09-18T08:19:36.748245+0000","last_became_active":"2024-09-18T08:19:05.893865+0000","last_became_peered":"2024-09-18T08:19:05.893865+0000","last_unstale":"2024-09-18T08:19:36.748245+0000","last_undegraded":"2024-09-18T08:19:36.748245+0000","last_fullsized":"2024-09-18T08:19:36.748245+0000","mapping_epoch":413,"log_start":"0'0","ondisk_log_start":"0'0","created":152,"last_epoch_clean":414,"parent":"0.0","parent_split_bits":3,"last_scrub":"74'79","last_scrub_stamp":"2024-09-18T08:18:14.059977+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:00:11.893742+0000","last_clean_scrub_stamp":"2024-09-18T08:18:14.059977+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.0","version":"271'30","reported_seq":885,"reported_epoch":424,"state":"active+clean","last_fresh":"2024-09-18T08:19:29.470893+0000","last_change":"2024-09-18T08:18:27.693740+0000","last_active":"2024-09-18T08:19:29.470893+0000","last_peered":"2024-09-18T08:19:29.470893+0000","last_clean":"2024-09-18T08:19:29.470893+0000","last_became_active":"2024-09-18T08:18:27.651348+0000","last_became_peered":"2024-09-18T08:18:27.651348+0000","last_unstale":"2024-09-18T08:19:29.470893+0000","last_undegraded":"2024-09-18T08:19:29.470893+0000","last_fullsized":"2024-09-18T08:19:29.470893+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":0,"last_scrub":"271'30","last_scrub_stamp":"2024-09-18T08:15:57.927111+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_clean_scrub_stamp":"2024-09-18T08:15:57.927111+0000","objects_scrubbed":8,"log_size":30,"log_dups_size":0,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":18,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":52,"num_read_kb":40,"num_write":32,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":819,"reported_epoch":424,"state":"active+clean","last_fresh":"2024-09-18T08:19:29.469547+0000","last_change":"2024-09-18T08:18:27.664084+0000","last_active":"2024-09-18T08:19:29.469547+0000","last_peered":"2024-09-18T08:19:29.469547+0000","last_clean":"2024-09-18T08:19:29.469547+0000","last_became_active":"2024-09-18T08:18:27.663173+0000","last_became_peered":"2024-09-18T08:18:27.663173+0000","last_unstale":"2024-09-18T08:19:29.469547+0000","last_undegraded":"2024-09-18T08:19:29.469547+0000","last_fullsized":"2024-09-18T08:19:29.469547+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:15:08.780913+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:46.860092+0000","last_clean_scrub_stamp":"2024-09-18T08:15:08.780913+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:19:53.567475+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":428,"reported_epoch":424,"state":"active+clean","last_fresh":"2024-09-18T08:19:29.466429+0000","last_change":"2024-09-18T08:18:27.661806+0000","last_active":"2024-09-18T08:19:29.466429+0000","last_peered":"2024-09-18T08:19:29.466429+0000","last_clean":"2024-09-18T08:19:29.466429+0000","last_became_active":"2024-09-18T08:18:27.661393+0000","last_became_peered":"2024-09-18T08:18:27.661393+0000","last_unstale":"2024-09-18T08:19:29.466429+0000","last_undegraded":"2024-09-18T08:19:29.466429+0000","last_fullsized":"2024-09-18T08:19:29.466429+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:17:31.417453+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:50.886830+0000","last_clean_scrub_stamp":"2024-09-18T08:17:31.417453+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":208,"reported_epoch":424,"state":"active+clean","last_fresh":"2024-09-18T08:19:29.467935+0000","last_change":"2024-09-18T08:19:06.578252+0000","last_active":"2024-09-18T08:19:29.467935+0000","last_peered":"2024-09-18T08:19:29.467935+0000","last_clean":"2024-09-18T08:19:29.467935+0000","last_became_active":"2024-09-18T08:19:05.893777+0000","last_became_peered":"2024-09-18T08:19:05.893777+0000","last_unstale":"2024-09-18T08:19:29.467935+0000","last_undegraded":"2024-09-18T08:19:29.467935+0000","last_fullsized":"2024-09-18T08:19:29.467935+0000","mapping_epoch":413,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":414,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:19:06.578190+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_clean_scrub_stamp":"2024-09-18T08:19:06.578190+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:20:22.349040+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,7],"acting":[3,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.e","version":"425'4705","reported_seq":4764,"reported_epoch":425,"state":"active+clean","last_fresh":"2024-09-18T08:19:37.876522+0000","last_change":"2024-09-18T08:18:27.652534+0000","last_active":"2024-09-18T08:19:37.876522+0000","last_peered":"2024-09-18T08:19:37.876522+0000","last_clean":"2024-09-18T08:19:37.876522+0000","last_became_active":"2024-09-18T08:18:27.652261+0000","last_became_peered":"2024-09-18T08:18:27.652261+0000","last_unstale":"2024-09-18T08:19:37.876522+0000","last_undegraded":"2024-09-18T08:19:37.876522+0000","last_fullsized":"2024-09-18T08:19:37.876522+0000","mapping_epoch":401,"log_start":"408'1100","ondisk_log_start":"408'1100","created":401,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:18:26.612016+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:18:26.612016+0000","last_clean_scrub_stamp":"2024-09-18T08:18:26.612016+0000","objects_scrubbed":0,"log_size":3605,"log_dups_size":0,"ondisk_log_size":3605,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":308346880,"num_objects":4705,"num_object_clones":0,"num_object_copies":14115,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4705,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4705,"num_write_kb":301120,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"74'79","reported_seq":646,"reported_epoch":424,"state":"active+clean","last_fresh":"2024-09-18T08:19:29.467150+0000","last_change":"2024-09-18T08:18:27.688842+0000","last_active":"2024-09-18T08:19:29.467150+0000","last_peered":"2024-09-18T08:19:29.467150+0000","last_clean":"2024-09-18T08:19:29.467150+0000","last_became_active":"2024-09-18T08:18:27.662304+0000","last_became_peered":"2024-09-18T08:18:27.662304+0000","last_unstale":"2024-09-18T08:19:29.467150+0000","last_undegraded":"2024-09-18T08:19:29.467150+0000","last_fullsized":"2024-09-18T08:19:29.467150+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":152,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":3,"last_scrub":"74'79","last_scrub_stamp":"2024-09-18T08:15:41.898294+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:00:11.893742+0000","last_clean_scrub_stamp":"2024-09-18T08:15:41.898294+0000","objects_scrubbed":1,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":327680,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1310720,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,7],"acting":[3,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"271'28","reported_seq":572,"reported_epoch":425,"state":"active+clean","last_fresh":"2024-09-18T08:19:36.746851+0000","last_change":"2024-09-18T08:16:13.506782+0000","last_active":"2024-09-18T08:19:36.746851+0000","last_peered":"2024-09-18T08:19:36.746851+0000","last_clean":"2024-09-18T08:19:36.746851+0000","last_became_active":"2024-09-18T08:16:13.451205+0000","last_became_peered":"2024-09-18T08:16:13.451205+0000","last_unstale":"2024-09-18T08:19:36.746851+0000","last_undegraded":"2024-09-18T08:19:36.746851+0000","last_fullsized":"2024-09-18T08:19:36.746851+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"271'28","last_scrub_stamp":"2024-09-18T08:15:51.716119+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_clean_scrub_stamp":"2024-09-18T08:15:51.716119+0000","objects_scrubbed":7,"log_size":28,"log_dups_size":0,"ondisk_log_size":28,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:19:50.687500+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":41,"num_read_kb":34,"num_write":28,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.6","version":"62'2","reported_seq":838,"reported_epoch":425,"state":"active+clean","last_fresh":"2024-09-18T08:19:36.751882+0000","last_change":"2024-09-18T08:18:27.670033+0000","last_active":"2024-09-18T08:19:36.751882+0000","last_peered":"2024-09-18T08:19:36.751882+0000","last_clean":"2024-09-18T08:19:36.751882+0000","last_became_active":"2024-09-18T08:18:27.645204+0000","last_became_peered":"2024-09-18T08:18:27.645204+0000","last_unstale":"2024-09-18T08:19:36.751882+0000","last_undegraded":"2024-09-18T08:19:36.751882+0000","last_fullsized":"2024-09-18T08:19:36.751882+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":0,"last_scrub":"62'2","last_scrub_stamp":"2024-09-18T08:14:42.145836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:46.860092+0000","last_clean_scrub_stamp":"2024-09-18T08:14:42.145836+0000","objects_scrubbed":2,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1039,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":4156,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.0","version":"349'13","reported_seq":1016,"reported_epoch":424,"state":"active+clean","last_fresh":"2024-09-18T08:19:29.470742+0000","last_change":"2024-09-18T08:16:13.516195+0000","last_active":"2024-09-18T08:19:29.470742+0000","last_peered":"2024-09-18T08:19:29.470742+0000","last_clean":"2024-09-18T08:19:29.470742+0000","last_became_active":"2024-09-18T08:16:13.477431+0000","last_became_peered":"2024-09-18T08:16:13.477431+0000","last_unstale":"2024-09-18T08:19:29.470742+0000","last_undegraded":"2024-09-18T08:19:29.470742+0000","last_fullsized":"2024-09-18T08:19:29.470742+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"310'12","last_scrub_stamp":"2024-09-18T08:15:49.795095+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:50.886830+0000","last_clean_scrub_stamp":"2024-09-18T08:15:49.795095+0000","objects_scrubbed":1,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":189,"reported_epoch":425,"state":"active+clean","last_fresh":"2024-09-18T08:19:36.746920+0000","last_change":"2024-09-18T08:15:41.441653+0000","last_active":"2024-09-18T08:19:36.746920+0000","last_peered":"2024-09-18T08:19:36.746920+0000","last_clean":"2024-09-18T08:19:36.746920+0000","last_became_active":"2024-09-18T08:15:41.440876+0000","last_became_peered":"2024-09-18T08:15:41.440876+0000","last_unstale":"2024-09-18T08:19:36.746920+0000","last_undegraded":"2024-09-18T08:19:36.746920+0000","last_fullsized":"2024-09-18T08:19:36.746920+0000","mapping_epoch":336,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":337,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:15:14.781554+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_clean_scrub_stamp":"2024-09-18T08:15:14.781554+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:19:47.625044+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.d","version":"425'4606","reported_seq":7265,"reported_epoch":425,"state":"active+recovering+undersized+remapped","last_fresh":"2024-09-18T08:19:37.891555+0000","last_change":"2024-09-18T08:19:07.086761+0000","last_active":"2024-09-18T08:19:37.891555+0000","last_peered":"2024-09-18T08:19:37.891555+0000","last_clean":"2024-09-18T08:19:04.804222+0000","last_became_active":"2024-09-18T08:19:07.019189+0000","last_became_peered":"2024-09-18T08:19:07.019189+0000","last_unstale":"2024-09-18T08:19:37.891555+0000","last_undegraded":"2024-09-18T08:19:37.891555+0000","last_fullsized":"2024-09-18T08:19:06.812164+0000","mapping_epoch":414,"log_start":"408'1000","ondisk_log_start":"408'1000","created":401,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:18:26.612016+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:18:26.612016+0000","last_clean_scrub_stamp":"2024-09-18T08:18:26.612016+0000","objects_scrubbed":0,"log_size":3606,"log_dups_size":0,"ondisk_log_size":3606,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":301858816,"num_objects":4606,"num_object_clones":0,"num_object_copies":13818,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1860,"num_objects_unfound":0,"num_objects_dirty":4606,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4606,"num_write_kb":294784,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1298,"num_bytes_recovered":85065728,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,5],"acting":[1,6],"avail_no_missing":["1","2","6"],"object_location_counts":[{"shards":"1,2,6","objects":1860},{"shards":"1,6","objects":2746}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"74'79","reported_seq":625,"reported_epoch":424,"state":"active+clean","last_fresh":"2024-09-18T08:19:29.469397+0000","last_change":"2024-09-18T08:18:27.663511+0000","last_active":"2024-09-18T08:19:29.469397+0000","last_peered":"2024-09-18T08:19:29.469397+0000","last_clean":"2024-09-18T08:19:29.469397+0000","last_became_active":"2024-09-18T08:18:27.662723+0000","last_became_peered":"2024-09-18T08:18:27.662723+0000","last_unstale":"2024-09-18T08:19:29.469397+0000","last_undegraded":"2024-09-18T08:19:29.469397+0000","last_fullsized":"2024-09-18T08:19:29.469397+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":152,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":3,"last_scrub":"74'79","last_scrub_stamp":"2024-09-18T08:15:44.872332+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:00:11.893742+0000","last_clean_scrub_stamp":"2024-09-18T08:15:44.872332+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,7],"acting":[3,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.2","version":"271'20","reported_seq":790,"reported_epoch":425,"state":"active+clean","last_fresh":"2024-09-18T08:19:36.743148+0000","last_change":"2024-09-18T08:18:22.653813+0000","last_active":"2024-09-18T08:19:36.743148+0000","last_peered":"2024-09-18T08:19:36.743148+0000","last_clean":"2024-09-18T08:19:36.743148+0000","last_became_active":"2024-09-18T08:16:13.486720+0000","last_became_peered":"2024-09-18T08:16:13.486720+0000","last_unstale":"2024-09-18T08:19:36.743148+0000","last_undegraded":"2024-09-18T08:19:36.743148+0000","last_fullsized":"2024-09-18T08:19:36.743148+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"271'20","last_scrub_stamp":"2024-09-18T08:18:22.653761+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_clean_scrub_stamp":"2024-09-18T08:18:22.653761+0000","objects_scrubbed":5,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:19:43.290763+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":28,"num_read_kb":23,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":189,"reported_epoch":425,"state":"active+clean","last_fresh":"2024-09-18T08:19:36.748448+0000","last_change":"2024-09-18T08:15:41.452981+0000","last_active":"2024-09-18T08:19:36.748448+0000","last_peered":"2024-09-18T08:19:36.748448+0000","last_clean":"2024-09-18T08:19:36.748448+0000","last_became_active":"2024-09-18T08:15:41.449639+0000","last_became_peered":"2024-09-18T08:15:41.449639+0000","last_unstale":"2024-09-18T08:19:36.748448+0000","last_undegraded":"2024-09-18T08:19:36.748448+0000","last_fullsized":"2024-09-18T08:19:36.748448+0000","mapping_epoch":336,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":337,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:15:39.858184+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:46.860092+0000","last_clean_scrub_stamp":"2024-09-18T08:15:39.858184+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:19:46.626601+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.3","version":"338'14","reported_seq":1019,"reported_epoch":425,"state":"active+clean","last_fresh":"2024-09-18T08:19:36.752622+0000","last_change":"2024-09-18T08:15:41.467411+0000","last_active":"2024-09-18T08:19:36.752622+0000","last_peered":"2024-09-18T08:19:36.752622+0000","last_clean":"2024-09-18T08:19:36.752622+0000","last_became_active":"2024-09-18T08:15:41.447667+0000","last_became_peered":"2024-09-18T08:15:41.447667+0000","last_unstale":"2024-09-18T08:19:36.752622+0000","last_undegraded":"2024-09-18T08:19:36.752622+0000","last_fullsized":"2024-09-18T08:19:36.752622+0000","mapping_epoch":336,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":337,"parent":"0.0","parent_split_bits":0,"last_scrub":"317'11","last_scrub_stamp":"2024-09-18T08:14:59.207667+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:50.886830+0000","last_clean_scrub_stamp":"2024-09-18T08:14:59.207667+0000","objects_scrubbed":1,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:19:41.963623+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":803,"reported_epoch":425,"state":"active+clean","last_fresh":"2024-09-18T08:19:36.746598+0000","last_change":"2024-09-18T08:19:05.899449+0000","last_active":"2024-09-18T08:19:36.746598+0000","last_peered":"2024-09-18T08:19:36.746598+0000","last_clean":"2024-09-18T08:19:36.746598+0000","last_became_active":"2024-09-18T08:19:05.899084+0000","last_became_peered":"2024-09-18T08:19:05.899084+0000","last_unstale":"2024-09-18T08:19:36.746598+0000","last_undegraded":"2024-09-18T08:19:36.746598+0000","last_fullsized":"2024-09-18T08:19:36.746598+0000","mapping_epoch":413,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":414,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:18:23.800677+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_clean_scrub_stamp":"2024-09-18T08:18:23.800677+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,5],"acting":[0,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.c","version":"425'4650","reported_seq":6862,"reported_epoch":425,"state":"active+recovering+undersized+remapped","last_fresh":"2024-09-18T08:19:37.886133+0000","last_change":"2024-09-18T08:19:07.099895+0000","last_active":"2024-09-18T08:19:37.886133+0000","last_peered":"2024-09-18T08:19:37.886133+0000","last_clean":"2024-09-18T08:19:04.810825+0000","last_became_active":"2024-09-18T08:19:07.045144+0000","last_became_peered":"2024-09-18T08:19:07.045144+0000","last_unstale":"2024-09-18T08:19:37.886133+0000","last_undegraded":"2024-09-18T08:19:37.886133+0000","last_fullsized":"2024-09-18T08:19:06.817301+0000","mapping_epoch":414,"log_start":"408'1000","ondisk_log_start":"408'1000","created":401,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:18:26.612016+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:18:26.612016+0000","last_clean_scrub_stamp":"2024-09-18T08:18:26.612016+0000","objects_scrubbed":0,"log_size":3650,"log_dups_size":0,"ondisk_log_size":3650,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":304742400,"num_objects":4650,"num_object_clones":0,"num_object_copies":13950,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":2093,"num_objects_unfound":0,"num_objects_dirty":4650,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4650,"num_write_kb":297600,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1075,"num_bytes_recovered":70451200,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,5],"acting":[1,7],"avail_no_missing":["1","2","7"],"object_location_counts":[{"shards":"1,2,7","objects":2093},{"shards":"1,7","objects":2557}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"74'79","reported_seq":687,"reported_epoch":425,"state":"active+clean","last_fresh":"2024-09-18T08:19:36.747109+0000","last_change":"2024-09-18T08:18:20.837291+0000","last_active":"2024-09-18T08:19:36.747109+0000","last_peered":"2024-09-18T08:19:36.747109+0000","last_clean":"2024-09-18T08:19:36.747109+0000","last_became_active":"2024-09-18T08:07:26.426532+0000","last_became_peered":"2024-09-18T08:07:26.426532+0000","last_unstale":"2024-09-18T08:19:36.747109+0000","last_undegraded":"2024-09-18T08:19:36.747109+0000","last_fullsized":"2024-09-18T08:19:36.747109+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":152,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":3,"last_scrub":"74'79","last_scrub_stamp":"2024-09-18T08:18:20.837241+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:00:11.893742+0000","last_clean_scrub_stamp":"2024-09-18T08:18:20.837241+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,3],"acting":[0,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.3","version":"271'29","reported_seq":929,"reported_epoch":425,"state":"active+clean","last_fresh":"2024-09-18T08:19:36.752796+0000","last_change":"2024-09-18T08:18:27.685772+0000","last_active":"2024-09-18T08:19:36.752796+0000","last_peered":"2024-09-18T08:19:36.752796+0000","last_clean":"2024-09-18T08:19:36.752796+0000","last_became_active":"2024-09-18T08:18:27.654192+0000","last_became_peered":"2024-09-18T08:18:27.654192+0000","last_unstale":"2024-09-18T08:19:36.752796+0000","last_undegraded":"2024-09-18T08:19:36.752796+0000","last_fullsized":"2024-09-18T08:19:36.752796+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":0,"last_scrub":"271'29","last_scrub_stamp":"2024-09-18T08:18:15.879919+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_clean_scrub_stamp":"2024-09-18T08:18:15.879919+0000","objects_scrubbed":8,"log_size":29,"log_dups_size":0,"ondisk_log_size":29,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:19:43.408146+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":54,"num_read_kb":40,"num_write":32,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":32,"num_bytes_recovered":1320,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.4","version":"62'1","reported_seq":666,"reported_epoch":425,"state":"active+clean","last_fresh":"2024-09-18T08:19:36.756347+0000","last_change":"2024-09-18T08:19:05.926690+0000","last_active":"2024-09-18T08:19:36.756347+0000","last_peered":"2024-09-18T08:19:36.756347+0000","last_clean":"2024-09-18T08:19:36.756347+0000","last_became_active":"2024-09-18T08:19:05.911595+0000","last_became_peered":"2024-09-18T08:19:05.911595+0000","last_unstale":"2024-09-18T08:19:36.756347+0000","last_undegraded":"2024-09-18T08:19:36.756347+0000","last_fullsized":"2024-09-18T08:19:36.756347+0000","mapping_epoch":413,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":414,"parent":"0.0","parent_split_bits":0,"last_scrub":"62'1","last_scrub_stamp":"2024-09-18T08:15:13.165826+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:46.860092+0000","last_clean_scrub_stamp":"2024-09-18T08:15:13.165826+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":17,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":436,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1744,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":709,"reported_epoch":425,"state":"active+clean","last_fresh":"2024-09-18T08:19:36.753161+0000","last_change":"2024-09-18T08:18:58.096376+0000","last_active":"2024-09-18T08:19:36.753161+0000","last_peered":"2024-09-18T08:19:36.753161+0000","last_clean":"2024-09-18T08:19:36.753161+0000","last_became_active":"2024-09-18T08:15:41.449832+0000","last_became_peered":"2024-09-18T08:15:41.449832+0000","last_unstale":"2024-09-18T08:19:36.753161+0000","last_undegraded":"2024-09-18T08:19:36.753161+0000","last_fullsized":"2024-09-18T08:19:36.753161+0000","mapping_epoch":336,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":337,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:18:58.096199+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:50.886830+0000","last_clean_scrub_stamp":"2024-09-18T08:18:58.096199+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:20:19.661202+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":681,"reported_epoch":425,"state":"active+clean","last_fresh":"2024-09-18T08:19:36.742342+0000","last_change":"2024-09-18T08:19:05.882802+0000","last_active":"2024-09-18T08:19:36.742342+0000","last_peered":"2024-09-18T08:19:36.742342+0000","last_clean":"2024-09-18T08:19:36.742342+0000","last_became_active":"2024-09-18T08:19:05.881711+0000","last_became_peered":"2024-09-18T08:19:05.881711+0000","last_unstale":"2024-09-18T08:19:36.742342+0000","last_undegraded":"2024-09-18T08:19:36.742342+0000","last_fullsized":"2024-09-18T08:19:36.742342+0000","mapping_epoch":413,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":414,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:15:50.294213+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_clean_scrub_stamp":"2024-09-18T08:15:50.294213+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":28,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,7],"acting":[1,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.b","version":"424'4527","reported_seq":4584,"reported_epoch":424,"state":"active+clean","last_fresh":"2024-09-18T08:19:34.527798+0000","last_change":"2024-09-18T08:18:27.656478+0000","last_active":"2024-09-18T08:19:34.527798+0000","last_peered":"2024-09-18T08:19:34.527798+0000","last_clean":"2024-09-18T08:19:34.527798+0000","last_became_active":"2024-09-18T08:18:27.656216+0000","last_became_peered":"2024-09-18T08:18:27.656216+0000","last_unstale":"2024-09-18T08:19:34.527798+0000","last_undegraded":"2024-09-18T08:19:34.527798+0000","last_fullsized":"2024-09-18T08:19:34.527798+0000","mapping_epoch":401,"log_start":"406'500","ondisk_log_start":"406'500","created":401,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:18:26.612016+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:18:26.612016+0000","last_clean_scrub_stamp":"2024-09-18T08:18:26.612016+0000","objects_scrubbed":0,"log_size":4027,"log_dups_size":0,"ondisk_log_size":4027,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:19:51.525124+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":296681472,"num_objects":4527,"num_object_clones":0,"num_object_copies":13581,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4527,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4527,"num_write_kb":289728,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"74'79","reported_seq":197,"reported_epoch":425,"state":"active+clean","last_fresh":"2024-09-18T08:19:36.747231+0000","last_change":"2024-09-18T08:15:43.275398+0000","last_active":"2024-09-18T08:19:36.747231+0000","last_peered":"2024-09-18T08:19:36.747231+0000","last_clean":"2024-09-18T08:19:36.747231+0000","last_became_active":"2024-09-18T08:15:41.452895+0000","last_became_peered":"2024-09-18T08:15:41.452895+0000","last_unstale":"2024-09-18T08:19:36.747231+0000","last_undegraded":"2024-09-18T08:19:36.747231+0000","last_fullsized":"2024-09-18T08:19:36.747231+0000","mapping_epoch":336,"log_start":"0'0","ondisk_log_start":"0'0","created":152,"last_epoch_clean":337,"parent":"0.0","parent_split_bits":4,"last_scrub":"74'79","last_scrub_stamp":"2024-09-18T08:15:43.275329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:00:11.893742+0000","last_clean_scrub_stamp":"2024-09-18T08:15:43.275329+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:19:52.721295+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,2],"acting":[5,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.4","version":"415'43","reported_seq":865,"reported_epoch":425,"state":"active+clean","last_fresh":"2024-09-18T08:19:36.759299+0000","last_change":"2024-09-18T08:19:05.931238+0000","last_active":"2024-09-18T08:19:36.759299+0000","last_peered":"2024-09-18T08:19:36.759299+0000","last_clean":"2024-09-18T08:19:36.759299+0000","last_became_active":"2024-09-18T08:19:05.911342+0000","last_became_peered":"2024-09-18T08:19:05.911342+0000","last_unstale":"2024-09-18T08:19:36.759299+0000","last_undegraded":"2024-09-18T08:19:36.759299+0000","last_fullsized":"2024-09-18T08:19:36.759299+0000","mapping_epoch":413,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":414,"parent":"0.0","parent_split_bits":0,"last_scrub":"310'42","last_scrub_stamp":"2024-09-18T08:18:27.690465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_clean_scrub_stamp":"2024-09-18T08:18:27.690465+0000","objects_scrubbed":10,"log_size":43,"log_dups_size":0,"ondisk_log_size":43,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":65,"num_read_kb":50,"num_write":38,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":358,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,6],"acting":[1,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":377,"reported_epoch":425,"state":"active+clean","last_fresh":"2024-09-18T08:19:36.753059+0000","last_change":"2024-09-18T08:18:16.839327+0000","last_active":"2024-09-18T08:19:36.753059+0000","last_peered":"2024-09-18T08:19:36.753059+0000","last_clean":"2024-09-18T08:19:36.753059+0000","last_became_active":"2024-09-18T08:16:13.475879+0000","last_became_peered":"2024-09-18T08:16:13.475879+0000","last_unstale":"2024-09-18T08:19:36.753059+0000","last_undegraded":"2024-09-18T08:19:36.753059+0000","last_fullsized":"2024-09-18T08:19:36.753059+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:18:16.839282+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:46.860092+0000","last_clean_scrub_stamp":"2024-09-18T08:18:16.839282+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,1],"acting":[0,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"64'2","reported_seq":209,"reported_epoch":425,"state":"active+clean","last_fresh":"2024-09-18T08:19:36.746931+0000","last_change":"2024-09-18T08:19:00.785048+0000","last_active":"2024-09-18T08:19:36.746931+0000","last_peered":"2024-09-18T08:19:36.746931+0000","last_clean":"2024-09-18T08:19:36.746931+0000","last_became_active":"2024-09-18T08:15:41.453351+0000","last_became_peered":"2024-09-18T08:15:41.453351+0000","last_unstale":"2024-09-18T08:19:36.746931+0000","last_undegraded":"2024-09-18T08:19:36.746931+0000","last_fullsized":"2024-09-18T08:19:36.746931+0000","mapping_epoch":336,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":337,"parent":"0.0","parent_split_bits":0,"last_scrub":"64'2","last_scrub_stamp":"2024-09-18T08:19:00.784983+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:45.855979+0000","last_clean_scrub_stamp":"2024-09-18T08:19:00.784983+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:20:14.366895+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":792,"reported_epoch":425,"state":"active+clean","last_fresh":"2024-09-18T08:19:36.752230+0000","last_change":"2024-09-18T08:16:13.481778+0000","last_active":"2024-09-18T08:19:36.752230+0000","last_peered":"2024-09-18T08:19:36.752230+0000","last_clean":"2024-09-18T08:19:36.752230+0000","last_became_active":"2024-09-18T08:16:13.481456+0000","last_became_peered":"2024-09-18T08:16:13.481456+0000","last_unstale":"2024-09-18T08:19:36.752230+0000","last_undegraded":"2024-09-18T08:19:36.752230+0000","last_fullsized":"2024-09-18T08:19:36.752230+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:15:04.260540+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:50.886830+0000","last_clean_scrub_stamp":"2024-09-18T08:15:04.260540+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:19:37.889074+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,6],"acting":[0,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":807,"reported_epoch":424,"state":"active+clean","last_fresh":"2024-09-18T08:19:29.468642+0000","last_change":"2024-09-18T08:18:27.657904+0000","last_active":"2024-09-18T08:19:29.468642+0000","last_peered":"2024-09-18T08:19:29.468642+0000","last_clean":"2024-09-18T08:19:29.468642+0000","last_became_active":"2024-09-18T08:18:27.657591+0000","last_became_peered":"2024-09-18T08:18:27.657591+0000","last_unstale":"2024-09-18T08:19:29.468642+0000","last_undegraded":"2024-09-18T08:19:29.468642+0000","last_fullsized":"2024-09-18T08:19:29.468642+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:15:59.948801+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_clean_scrub_stamp":"2024-09-18T08:15:59.948801+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.8","version":"424'4676","reported_seq":4733,"reported_epoch":424,"state":"active+clean","last_fresh":"2024-09-18T08:19:34.532202+0000","last_change":"2024-09-18T08:18:27.641711+0000","last_active":"2024-09-18T08:19:34.532202+0000","last_peered":"2024-09-18T08:19:34.532202+0000","last_clean":"2024-09-18T08:19:34.532202+0000","last_became_active":"2024-09-18T08:18:27.641439+0000","last_became_peered":"2024-09-18T08:18:27.641439+0000","last_unstale":"2024-09-18T08:19:34.532202+0000","last_undegraded":"2024-09-18T08:19:34.532202+0000","last_fullsized":"2024-09-18T08:19:34.532202+0000","mapping_epoch":401,"log_start":"408'700","ondisk_log_start":"408'700","created":401,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:18:26.612016+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:18:26.612016+0000","last_clean_scrub_stamp":"2024-09-18T08:18:26.612016+0000","objects_scrubbed":0,"log_size":3976,"log_dups_size":0,"ondisk_log_size":3976,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:19:55.081035+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":306446336,"num_objects":4676,"num_object_clones":0,"num_object_copies":14028,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4676,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4676,"num_write_kb":299264,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"74'79","reported_seq":728,"reported_epoch":425,"state":"active+clean","last_fresh":"2024-09-18T08:19:36.753004+0000","last_change":"2024-09-18T08:19:05.896315+0000","last_active":"2024-09-18T08:19:36.753004+0000","last_peered":"2024-09-18T08:19:36.753004+0000","last_clean":"2024-09-18T08:19:36.753004+0000","last_became_active":"2024-09-18T08:19:05.895876+0000","last_became_peered":"2024-09-18T08:19:05.895876+0000","last_unstale":"2024-09-18T08:19:36.753004+0000","last_undegraded":"2024-09-18T08:19:36.753004+0000","last_fullsized":"2024-09-18T08:19:36.753004+0000","mapping_epoch":413,"log_start":"0'0","ondisk_log_start":"0'0","created":152,"last_epoch_clean":414,"parent":"0.0","parent_split_bits":4,"last_scrub":"74'79","last_scrub_stamp":"2024-09-18T08:18:17.808269+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:00:11.893742+0000","last_clean_scrub_stamp":"2024-09-18T08:18:17.808269+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:19:42.586416+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,5],"acting":[0,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.7","version":"271'17","reported_seq":779,"reported_epoch":425,"state":"active+clean","last_fresh":"2024-09-18T08:19:37.002690+0000","last_change":"2024-09-18T08:19:07.121738+0000","last_active":"2024-09-18T08:19:37.002690+0000","last_peered":"2024-09-18T08:19:37.002690+0000","last_clean":"2024-09-18T08:19:37.002690+0000","last_became_active":"2024-09-18T08:15:41.456482+0000","last_became_peered":"2024-09-18T08:15:41.456482+0000","last_unstale":"2024-09-18T08:19:37.002690+0000","last_undegraded":"2024-09-18T08:19:37.002690+0000","last_fullsized":"2024-09-18T08:19:37.002690+0000","mapping_epoch":336,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":337,"parent":"0.0","parent_split_bits":0,"last_scrub":"271'17","last_scrub_stamp":"2024-09-18T08:19:07.121623+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_clean_scrub_stamp":"2024-09-18T08:19:07.121623+0000","objects_scrubbed":5,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:20:10.189018+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":35,"num_read_kb":24,"num_write":20,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":990,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,0],"acting":[2,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":671,"reported_epoch":425,"state":"active+clean","last_fresh":"2024-09-18T08:19:36.749291+0000","last_change":"2024-09-18T08:19:05.905706+0000","last_active":"2024-09-18T08:19:36.749291+0000","last_peered":"2024-09-18T08:19:36.749291+0000","last_clean":"2024-09-18T08:19:36.749291+0000","last_became_active":"2024-09-18T08:19:05.905390+0000","last_became_peered":"2024-09-18T08:19:05.905390+0000","last_unstale":"2024-09-18T08:19:36.749291+0000","last_undegraded":"2024-09-18T08:19:36.749291+0000","last_fullsized":"2024-09-18T08:19:36.749291+0000","mapping_epoch":413,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":414,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:15:49.244156+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:46.860092+0000","last_clean_scrub_stamp":"2024-09-18T08:15:49.244156+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,0],"acting":[1,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"62'1","reported_seq":50,"reported_epoch":424,"state":"active+clean","last_fresh":"2024-09-18T08:19:29.466065+0000","last_change":"2024-09-18T08:19:05.923105+0000","last_active":"2024-09-18T08:19:29.466065+0000","last_peered":"2024-09-18T08:19:29.466065+0000","last_clean":"2024-09-18T08:19:29.466065+0000","last_became_active":"2024-09-18T08:19:05.908141+0000","last_became_peered":"2024-09-18T08:19:05.908141+0000","last_unstale":"2024-09-18T08:19:29.466065+0000","last_undegraded":"2024-09-18T08:19:29.466065+0000","last_fullsized":"2024-09-18T08:19:29.466065+0000","mapping_epoch":413,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":414,"parent":"0.0","parent_split_bits":0,"last_scrub":"62'1","last_scrub_stamp":"2024-09-18T08:17:34.935618+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:45.855979+0000","last_clean_scrub_stamp":"2024-09-18T08:17:34.935618+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":490,"reported_epoch":425,"state":"active+clean","last_fresh":"2024-09-18T08:19:36.747549+0000","last_change":"2024-09-18T08:19:05.886456+0000","last_active":"2024-09-18T08:19:36.747549+0000","last_peered":"2024-09-18T08:19:36.747549+0000","last_clean":"2024-09-18T08:19:36.747549+0000","last_became_active":"2024-09-18T08:19:05.886174+0000","last_became_peered":"2024-09-18T08:19:05.886174+0000","last_unstale":"2024-09-18T08:19:36.747549+0000","last_undegraded":"2024-09-18T08:19:36.747549+0000","last_fullsized":"2024-09-18T08:19:36.747549+0000","mapping_epoch":413,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":414,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:18:28.112437+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:50.886830+0000","last_clean_scrub_stamp":"2024-09-18T08:18:28.112437+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:19:55.960395+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":471,"reported_epoch":424,"state":"active+clean","last_fresh":"2024-09-18T08:19:29.468023+0000","last_change":"2024-09-18T08:18:27.662433+0000","last_active":"2024-09-18T08:19:29.468023+0000","last_peered":"2024-09-18T08:19:29.468023+0000","last_clean":"2024-09-18T08:19:29.468023+0000","last_became_active":"2024-09-18T08:18:27.662116+0000","last_became_peered":"2024-09-18T08:18:27.662116+0000","last_unstale":"2024-09-18T08:19:29.468023+0000","last_undegraded":"2024-09-18T08:19:29.468023+0000","last_fullsized":"2024-09-18T08:19:29.468023+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:17:35.381401+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_clean_scrub_stamp":"2024-09-18T08:17:35.381401+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":23,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.9","version":"424'4494","reported_seq":4551,"reported_epoch":424,"state":"active+clean","last_fresh":"2024-09-18T08:19:34.545652+0000","last_change":"2024-09-18T08:18:27.655199+0000","last_active":"2024-09-18T08:19:34.545652+0000","last_peered":"2024-09-18T08:19:34.545652+0000","last_clean":"2024-09-18T08:19:34.545652+0000","last_became_active":"2024-09-18T08:18:27.654866+0000","last_became_peered":"2024-09-18T08:18:27.654866+0000","last_unstale":"2024-09-18T08:19:34.545652+0000","last_undegraded":"2024-09-18T08:19:34.545652+0000","last_fullsized":"2024-09-18T08:19:34.545652+0000","mapping_epoch":401,"log_start":"406'500","ondisk_log_start":"406'500","created":401,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:18:26.612016+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:18:26.612016+0000","last_clean_scrub_stamp":"2024-09-18T08:18:26.612016+0000","objects_scrubbed":0,"log_size":3994,"log_dups_size":0,"ondisk_log_size":3994,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:19:40.211623+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":294518784,"num_objects":4494,"num_object_clones":0,"num_object_copies":13482,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4494,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4494,"num_write_kb":287616,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,0],"acting":[3,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"74'79","reported_seq":447,"reported_epoch":425,"state":"active+clean","last_fresh":"2024-09-18T08:19:36.748192+0000","last_change":"2024-09-18T08:18:55.603760+0000","last_active":"2024-09-18T08:19:36.748192+0000","last_peered":"2024-09-18T08:19:36.748192+0000","last_clean":"2024-09-18T08:19:36.748192+0000","last_became_active":"2024-09-18T08:16:13.454138+0000","last_became_peered":"2024-09-18T08:16:13.454138+0000","last_unstale":"2024-09-18T08:19:36.748192+0000","last_undegraded":"2024-09-18T08:19:36.748192+0000","last_fullsized":"2024-09-18T08:19:36.748192+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":152,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":3,"last_scrub":"74'79","last_scrub_stamp":"2024-09-18T08:18:55.603698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:00:11.893742+0000","last_clean_scrub_stamp":"2024-09-18T08:18:55.603698+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:20:13.108664+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,5],"acting":[1,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.6","version":"271'24","reported_seq":897,"reported_epoch":425,"state":"active+clean","last_fresh":"2024-09-18T08:19:36.751600+0000","last_change":"2024-09-18T08:19:05.919065+0000","last_active":"2024-09-18T08:19:36.751600+0000","last_peered":"2024-09-18T08:19:36.751600+0000","last_clean":"2024-09-18T08:19:36.751600+0000","last_became_active":"2024-09-18T08:19:05.900310+0000","last_became_peered":"2024-09-18T08:19:05.900310+0000","last_unstale":"2024-09-18T08:19:36.751600+0000","last_undegraded":"2024-09-18T08:19:36.751600+0000","last_fullsized":"2024-09-18T08:19:36.751600+0000","mapping_epoch":413,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":414,"parent":"0.0","parent_split_bits":0,"last_scrub":"271'24","last_scrub_stamp":"2024-09-18T08:15:10.187593+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_clean_scrub_stamp":"2024-09-18T08:15:10.187593+0000","objects_scrubbed":6,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":36,"num_read_kb":30,"num_write":24,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":806,"reported_epoch":425,"state":"active+clean","last_fresh":"2024-09-18T08:19:36.754443+0000","last_change":"2024-09-18T08:16:13.461291+0000","last_active":"2024-09-18T08:19:36.754443+0000","last_peered":"2024-09-18T08:19:36.754443+0000","last_clean":"2024-09-18T08:19:36.754443+0000","last_became_active":"2024-09-18T08:16:13.460391+0000","last_became_peered":"2024-09-18T08:16:13.460391+0000","last_unstale":"2024-09-18T08:19:36.754443+0000","last_undegraded":"2024-09-18T08:19:36.754443+0000","last_fullsized":"2024-09-18T08:19:36.754443+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:14:37.133544+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:46.860092+0000","last_clean_scrub_stamp":"2024-09-18T08:14:37.133544+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,5],"acting":[0,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"70'5","reported_seq":1043,"reported_epoch":425,"state":"active+clean","last_fresh":"2024-09-18T08:19:36.748393+0000","last_change":"2024-09-18T08:18:27.672092+0000","last_active":"2024-09-18T08:19:36.748393+0000","last_peered":"2024-09-18T08:19:36.748393+0000","last_clean":"2024-09-18T08:19:36.748393+0000","last_became_active":"2024-09-18T08:18:27.646445+0000","last_became_peered":"2024-09-18T08:18:27.646445+0000","last_unstale":"2024-09-18T08:19:36.748393+0000","last_undegraded":"2024-09-18T08:19:36.748393+0000","last_fullsized":"2024-09-18T08:19:36.748393+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":0,"last_scrub":"70'5","last_scrub_stamp":"2024-09-18T08:15:57.384224+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:45.855979+0000","last_clean_scrub_stamp":"2024-09-18T08:15:57.384224+0000","objects_scrubbed":1,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:19:52.639810+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0.000504055,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":467,"num_read_kb":462,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":778,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":189,"reported_epoch":425,"state":"active+clean","last_fresh":"2024-09-18T08:19:36.748220+0000","last_change":"2024-09-18T08:15:41.443577+0000","last_active":"2024-09-18T08:19:36.748220+0000","last_peered":"2024-09-18T08:19:36.748220+0000","last_clean":"2024-09-18T08:19:36.748220+0000","last_became_active":"2024-09-18T08:15:41.443080+0000","last_became_peered":"2024-09-18T08:15:41.443080+0000","last_unstale":"2024-09-18T08:19:36.748220+0000","last_undegraded":"2024-09-18T08:19:36.748220+0000","last_fullsized":"2024-09-18T08:19:36.748220+0000","mapping_epoch":336,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":337,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:15:02.179693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:50.886830+0000","last_clean_scrub_stamp":"2024-09-18T08:15:02.179693+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":594,"reported_epoch":425,"state":"active+clean","last_fresh":"2024-09-18T08:19:36.754654+0000","last_change":"2024-09-18T08:18:16.556819+0000","last_active":"2024-09-18T08:19:36.754654+0000","last_peered":"2024-09-18T08:19:36.754654+0000","last_clean":"2024-09-18T08:19:36.754654+0000","last_became_active":"2024-09-18T08:15:41.436692+0000","last_became_peered":"2024-09-18T08:15:41.436692+0000","last_unstale":"2024-09-18T08:19:36.754654+0000","last_undegraded":"2024-09-18T08:19:36.754654+0000","last_fullsized":"2024-09-18T08:19:36.754654+0000","mapping_epoch":336,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":337,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:18:16.556771+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_clean_scrub_stamp":"2024-09-18T08:18:16.556771+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.a","version":"424'4614","reported_seq":4671,"reported_epoch":424,"state":"active+clean","last_fresh":"2024-09-18T08:19:34.519705+0000","last_change":"2024-09-18T08:18:27.646142+0000","last_active":"2024-09-18T08:19:34.519705+0000","last_peered":"2024-09-18T08:19:34.519705+0000","last_clean":"2024-09-18T08:19:34.519705+0000","last_became_active":"2024-09-18T08:18:27.645863+0000","last_became_peered":"2024-09-18T08:18:27.645863+0000","last_unstale":"2024-09-18T08:19:34.519705+0000","last_undegraded":"2024-09-18T08:19:34.519705+0000","last_fullsized":"2024-09-18T08:19:34.519705+0000","mapping_epoch":401,"log_start":"407'600","ondisk_log_start":"407'600","created":401,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:18:26.612016+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:18:26.612016+0000","last_clean_scrub_stamp":"2024-09-18T08:18:26.612016+0000","objects_scrubbed":0,"log_size":4014,"log_dups_size":0,"ondisk_log_size":4014,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":302383104,"num_objects":4614,"num_object_clones":0,"num_object_copies":13842,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4614,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4614,"num_write_kb":295296,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.5","version":"402'36","reported_seq":99,"reported_epoch":425,"state":"active+clean","last_fresh":"2024-09-18T08:19:36.748598+0000","last_change":"2024-09-18T08:18:30.788913+0000","last_active":"2024-09-18T08:19:36.748598+0000","last_peered":"2024-09-18T08:19:36.748598+0000","last_clean":"2024-09-18T08:19:36.748598+0000","last_became_active":"2024-09-18T08:18:27.657630+0000","last_became_peered":"2024-09-18T08:18:27.657630+0000","last_unstale":"2024-09-18T08:19:36.748598+0000","last_undegraded":"2024-09-18T08:19:36.748598+0000","last_fullsized":"2024-09-18T08:19:36.748598+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":0,"last_scrub":"402'36","last_scrub_stamp":"2024-09-18T08:18:30.788799+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_clean_scrub_stamp":"2024-09-18T08:18:30.788799+0000","objects_scrubbed":8,"log_size":36,"log_dups_size":0,"ondisk_log_size":36,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:19:47.408292+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":42,"num_read_kb":32,"num_write":25,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":462,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,0],"acting":[7,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":796,"reported_epoch":424,"state":"active+clean","last_fresh":"2024-09-18T08:19:29.468312+0000","last_change":"2024-09-18T08:15:41.464630+0000","last_active":"2024-09-18T08:19:29.468312+0000","last_peered":"2024-09-18T08:19:29.468312+0000","last_clean":"2024-09-18T08:19:29.468312+0000","last_became_active":"2024-09-18T08:15:41.464313+0000","last_became_peered":"2024-09-18T08:15:41.464313+0000","last_unstale":"2024-09-18T08:19:29.468312+0000","last_undegraded":"2024-09-18T08:19:29.468312+0000","last_fullsized":"2024-09-18T08:19:29.468312+0000","mapping_epoch":336,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":337,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:14:32.693460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:46.860092+0000","last_clean_scrub_stamp":"2024-09-18T08:14:32.693460+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":38,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"74'79","reported_seq":402,"reported_epoch":425,"state":"active+clean","last_fresh":"2024-09-18T08:19:36.752219+0000","last_change":"2024-09-18T08:18:27.650334+0000","last_active":"2024-09-18T08:19:36.752219+0000","last_peered":"2024-09-18T08:19:36.752219+0000","last_clean":"2024-09-18T08:19:36.752219+0000","last_became_active":"2024-09-18T08:18:27.649964+0000","last_became_peered":"2024-09-18T08:18:27.649964+0000","last_unstale":"2024-09-18T08:19:36.752219+0000","last_undegraded":"2024-09-18T08:19:36.752219+0000","last_fullsized":"2024-09-18T08:19:36.752219+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'79","last_scrub_stamp":"2024-09-18T08:18:18.786155+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:00:11.893742+0000","last_clean_scrub_stamp":"2024-09-18T08:18:18.786155+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:19:47.551374+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,6],"acting":[0,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":657,"reported_epoch":425,"state":"active+clean","last_fresh":"2024-09-18T08:19:37.003387+0000","last_change":"2024-09-18T08:19:05.096869+0000","last_active":"2024-09-18T08:19:37.003387+0000","last_peered":"2024-09-18T08:19:37.003387+0000","last_clean":"2024-09-18T08:19:37.003387+0000","last_became_active":"2024-09-18T08:18:27.653602+0000","last_became_peered":"2024-09-18T08:18:27.653602+0000","last_unstale":"2024-09-18T08:19:37.003387+0000","last_undegraded":"2024-09-18T08:19:37.003387+0000","last_fullsized":"2024-09-18T08:19:37.003387+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:19:05.096805+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:50.886830+0000","last_clean_scrub_stamp":"2024-09-18T08:19:05.096805+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:20:34.654958+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":204,"reported_epoch":425,"state":"active+clean","last_fresh":"2024-09-18T08:19:36.748218+0000","last_change":"2024-09-18T08:18:25.515806+0000","last_active":"2024-09-18T08:19:36.748218+0000","last_peered":"2024-09-18T08:19:36.748218+0000","last_clean":"2024-09-18T08:19:36.748218+0000","last_became_active":"2024-09-18T08:16:13.471911+0000","last_became_peered":"2024-09-18T08:16:13.471911+0000","last_unstale":"2024-09-18T08:19:36.748218+0000","last_undegraded":"2024-09-18T08:19:36.748218+0000","last_fullsized":"2024-09-18T08:19:36.748218+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:18:25.515757+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_clean_scrub_stamp":"2024-09-18T08:18:25.515757+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:19:37.445115+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,1],"acting":[5,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.1","version":"424'4561","reported_seq":4616,"reported_epoch":424,"state":"active+clean","last_fresh":"2024-09-18T08:19:34.545807+0000","last_change":"2024-09-18T08:18:27.651222+0000","last_active":"2024-09-18T08:19:34.545807+0000","last_peered":"2024-09-18T08:19:34.545807+0000","last_clean":"2024-09-18T08:19:34.545807+0000","last_became_active":"2024-09-18T08:18:27.650844+0000","last_became_peered":"2024-09-18T08:18:27.650844+0000","last_unstale":"2024-09-18T08:19:34.545807+0000","last_undegraded":"2024-09-18T08:19:34.545807+0000","last_fullsized":"2024-09-18T08:19:34.545807+0000","mapping_epoch":401,"log_start":"407'600","ondisk_log_start":"407'600","created":401,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:18:26.612016+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:18:26.612016+0000","last_clean_scrub_stamp":"2024-09-18T08:18:26.612016+0000","objects_scrubbed":0,"log_size":3961,"log_dups_size":0,"ondisk_log_size":3961,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":298909696,"num_objects":4561,"num_object_clones":0,"num_object_copies":13683,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4561,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4561,"num_write_kb":291904,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,0],"acting":[3,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.e","version":"271'19","reported_seq":486,"reported_epoch":425,"state":"active+clean","last_fresh":"2024-09-18T08:19:36.742299+0000","last_change":"2024-09-18T08:17:30.690655+0000","last_active":"2024-09-18T08:19:36.742299+0000","last_peered":"2024-09-18T08:19:36.742299+0000","last_clean":"2024-09-18T08:19:36.742299+0000","last_became_active":"2024-09-18T08:16:13.458001+0000","last_became_peered":"2024-09-18T08:16:13.458001+0000","last_unstale":"2024-09-18T08:19:36.742299+0000","last_undegraded":"2024-09-18T08:19:36.742299+0000","last_fullsized":"2024-09-18T08:19:36.742299+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"271'19","last_scrub_stamp":"2024-09-18T08:17:30.690581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_clean_scrub_stamp":"2024-09-18T08:17:30.690581+0000","objects_scrubbed":5,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":31,"num_read_kb":24,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,3],"acting":[1,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":446,"reported_epoch":425,"state":"active+clean","last_fresh":"2024-09-18T08:19:36.747148+0000","last_change":"2024-09-18T08:19:05.881635+0000","last_active":"2024-09-18T08:19:36.747148+0000","last_peered":"2024-09-18T08:19:36.747148+0000","last_clean":"2024-09-18T08:19:36.747148+0000","last_became_active":"2024-09-18T08:19:05.881238+0000","last_became_peered":"2024-09-18T08:19:05.881238+0000","last_unstale":"2024-09-18T08:19:36.747148+0000","last_undegraded":"2024-09-18T08:19:36.747148+0000","last_fullsized":"2024-09-18T08:19:36.747148+0000","mapping_epoch":413,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":414,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:18:26.121022+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:46.860092+0000","last_clean_scrub_stamp":"2024-09-18T08:18:26.121022+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:19:52.320397+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,5],"acting":[7,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":211,"reported_epoch":425,"state":"active+clean","last_fresh":"2024-09-18T08:19:36.748048+0000","last_change":"2024-09-18T08:19:05.897861+0000","last_active":"2024-09-18T08:19:36.748048+0000","last_peered":"2024-09-18T08:19:36.748048+0000","last_clean":"2024-09-18T08:19:36.748048+0000","last_became_active":"2024-09-18T08:19:05.897497+0000","last_became_peered":"2024-09-18T08:19:05.897497+0000","last_unstale":"2024-09-18T08:19:36.748048+0000","last_undegraded":"2024-09-18T08:19:36.748048+0000","last_fullsized":"2024-09-18T08:19:36.748048+0000","mapping_epoch":413,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":414,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:17:38.616405+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:50.886830+0000","last_clean_scrub_stamp":"2024-09-18T08:17:38.616405+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,1],"acting":[5,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":783,"reported_epoch":424,"state":"active+clean","last_fresh":"2024-09-18T08:19:29.467250+0000","last_change":"2024-09-18T08:15:41.465035+0000","last_active":"2024-09-18T08:19:29.467250+0000","last_peered":"2024-09-18T08:19:29.467250+0000","last_clean":"2024-09-18T08:19:29.467250+0000","last_became_active":"2024-09-18T08:15:41.464500+0000","last_became_peered":"2024-09-18T08:15:41.464500+0000","last_unstale":"2024-09-18T08:19:29.467250+0000","last_undegraded":"2024-09-18T08:19:29.467250+0000","last_fullsized":"2024-09-18T08:19:29.467250+0000","mapping_epoch":336,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":337,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:15:02.861116+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_clean_scrub_stamp":"2024-09-18T08:15:02.861116+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:19:48.570022+0000","scrub_duration":81,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.2","version":"424'4605","reported_seq":7097,"reported_epoch":424,"state":"active+recovering+undersized+remapped","last_fresh":"2024-09-18T08:19:34.544688+0000","last_change":"2024-09-18T08:19:06.927875+0000","last_active":"2024-09-18T08:19:34.544688+0000","last_peered":"2024-09-18T08:19:34.544688+0000","last_clean":"2024-09-18T08:19:04.795910+0000","last_became_active":"2024-09-18T08:19:06.905169+0000","last_became_peered":"2024-09-18T08:19:06.905169+0000","last_unstale":"2024-09-18T08:19:34.544688+0000","last_undegraded":"2024-09-18T08:19:34.544688+0000","last_fullsized":"2024-09-18T08:19:06.814234+0000","mapping_epoch":414,"log_start":"407'600","ondisk_log_start":"407'600","created":401,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:18:26.612016+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:18:26.612016+0000","last_clean_scrub_stamp":"2024-09-18T08:18:26.612016+0000","objects_scrubbed":0,"log_size":4005,"log_dups_size":0,"ondisk_log_size":4005,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":301793280,"num_objects":4605,"num_object_clones":0,"num_object_copies":13815,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":2051,"num_objects_unfound":0,"num_objects_dirty":4605,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4605,"num_write_kb":294720,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1216,"num_bytes_recovered":79691776,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,7],"acting":[3,0],"avail_no_missing":["3","0","2"],"object_location_counts":[{"shards":"0,2,3","objects":2051},{"shards":"0,3","objects":2554}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"74'79","reported_seq":209,"reported_epoch":425,"state":"active+clean","last_fresh":"2024-09-18T08:19:36.747567+0000","last_change":"2024-09-18T08:19:01.768944+0000","last_active":"2024-09-18T08:19:36.747567+0000","last_peered":"2024-09-18T08:19:36.747567+0000","last_clean":"2024-09-18T08:19:36.747567+0000","last_became_active":"2024-09-18T08:15:41.454781+0000","last_became_peered":"2024-09-18T08:15:41.454781+0000","last_unstale":"2024-09-18T08:19:36.747567+0000","last_undegraded":"2024-09-18T08:19:36.747567+0000","last_fullsized":"2024-09-18T08:19:36.747567+0000","mapping_epoch":336,"log_start":"0'0","ondisk_log_start":"0'0","created":152,"last_epoch_clean":337,"parent":"0.0","parent_split_bits":4,"last_scrub":"74'79","last_scrub_stamp":"2024-09-18T08:19:01.768882+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:00:11.893742+0000","last_clean_scrub_stamp":"2024-09-18T08:19:01.768882+0000","objects_scrubbed":1,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:20:18.146783+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262688,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":262688,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,3],"acting":[5,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.d","version":"271'31","reported_seq":360,"reported_epoch":425,"state":"active+clean","last_fresh":"2024-09-18T08:19:36.755707+0000","last_change":"2024-09-18T08:19:05.923626+0000","last_active":"2024-09-18T08:19:36.755707+0000","last_peered":"2024-09-18T08:19:36.755707+0000","last_clean":"2024-09-18T08:19:36.755707+0000","last_became_active":"2024-09-18T08:19:05.905244+0000","last_became_peered":"2024-09-18T08:19:05.905244+0000","last_unstale":"2024-09-18T08:19:36.755707+0000","last_undegraded":"2024-09-18T08:19:36.755707+0000","last_fullsized":"2024-09-18T08:19:36.755707+0000","mapping_epoch":413,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":414,"parent":"0.0","parent_split_bits":0,"last_scrub":"271'31","last_scrub_stamp":"2024-09-18T08:15:03.678670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_clean_scrub_stamp":"2024-09-18T08:15:03.678670+0000","objects_scrubbed":0,"log_size":31,"log_dups_size":0,"ondisk_log_size":31,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":49,"num_read_kb":39,"num_write":32,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,0],"acting":[1,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":449,"reported_epoch":425,"state":"active+clean","last_fresh":"2024-09-18T08:19:36.743239+0000","last_change":"2024-09-18T08:18:18.567839+0000","last_active":"2024-09-18T08:19:36.743239+0000","last_peered":"2024-09-18T08:19:36.743239+0000","last_clean":"2024-09-18T08:19:36.743239+0000","last_became_active":"2024-09-18T08:16:13.483143+0000","last_became_peered":"2024-09-18T08:16:13.483143+0000","last_unstale":"2024-09-18T08:19:36.743239+0000","last_undegraded":"2024-09-18T08:19:36.743239+0000","last_fullsized":"2024-09-18T08:19:36.743239+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:18:18.567791+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:46.860092+0000","last_clean_scrub_stamp":"2024-09-18T08:18:18.567791+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,3],"acting":[1,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":679,"reported_epoch":425,"state":"active+clean","last_fresh":"2024-09-18T08:19:36.754747+0000","last_change":"2024-09-18T08:18:59.652875+0000","last_active":"2024-09-18T08:19:36.754747+0000","last_peered":"2024-09-18T08:19:36.754747+0000","last_clean":"2024-09-18T08:19:36.754747+0000","last_became_active":"2024-09-18T08:16:13.460650+0000","last_became_peered":"2024-09-18T08:16:13.460650+0000","last_unstale":"2024-09-18T08:19:36.754747+0000","last_undegraded":"2024-09-18T08:19:36.754747+0000","last_fullsized":"2024-09-18T08:19:36.754747+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:18:59.652484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:50.886830+0000","last_clean_scrub_stamp":"2024-09-18T08:18:59.652484+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:20:04.836905+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,6],"acting":[1,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":369,"reported_epoch":424,"state":"active+clean","last_fresh":"2024-09-18T08:19:29.469158+0000","last_change":"2024-09-18T08:19:05.895239+0000","last_active":"2024-09-18T08:19:29.469158+0000","last_peered":"2024-09-18T08:19:29.469158+0000","last_clean":"2024-09-18T08:19:29.469158+0000","last_became_active":"2024-09-18T08:19:05.894863+0000","last_became_peered":"2024-09-18T08:19:05.894863+0000","last_unstale":"2024-09-18T08:19:29.469158+0000","last_undegraded":"2024-09-18T08:19:29.469158+0000","last_fullsized":"2024-09-18T08:19:29.469158+0000","mapping_epoch":413,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":414,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:19:02.172378+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_clean_scrub_stamp":"2024-09-18T08:19:02.172378+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:20:11.939070+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,1],"acting":[3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.3","version":"425'4801","reported_seq":4860,"reported_epoch":425,"state":"active+clean","last_fresh":"2024-09-18T08:19:38.539666+0000","last_change":"2024-09-18T08:18:27.658212+0000","last_active":"2024-09-18T08:19:38.539666+0000","last_peered":"2024-09-18T08:19:38.539666+0000","last_clean":"2024-09-18T08:19:38.539666+0000","last_became_active":"2024-09-18T08:18:27.657815+0000","last_became_peered":"2024-09-18T08:18:27.657815+0000","last_unstale":"2024-09-18T08:19:38.539666+0000","last_undegraded":"2024-09-18T08:19:38.539666+0000","last_fullsized":"2024-09-18T08:19:38.539666+0000","mapping_epoch":401,"log_start":"408'800","ondisk_log_start":"408'800","created":401,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:18:26.612016+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:18:26.612016+0000","last_clean_scrub_stamp":"2024-09-18T08:18:26.612016+0000","objects_scrubbed":0,"log_size":4001,"log_dups_size":0,"ondisk_log_size":4001,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:19:45.500162+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":314638336,"num_objects":4801,"num_object_clones":0,"num_object_copies":14403,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4801,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4801,"num_write_kb":307264,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.9","version":"74'79","reported_seq":623,"reported_epoch":424,"state":"active+clean","last_fresh":"2024-09-18T08:19:29.466345+0000","last_change":"2024-09-18T08:19:05.908881+0000","last_active":"2024-09-18T08:19:29.466345+0000","last_peered":"2024-09-18T08:19:29.466345+0000","last_clean":"2024-09-18T08:19:29.466345+0000","last_became_active":"2024-09-18T08:19:05.903470+0000","last_became_peered":"2024-09-18T08:19:05.903470+0000","last_unstale":"2024-09-18T08:19:29.466345+0000","last_undegraded":"2024-09-18T08:19:29.466345+0000","last_fullsized":"2024-09-18T08:19:29.466345+0000","mapping_epoch":413,"log_start":"0'0","ondisk_log_start":"0'0","created":152,"last_epoch_clean":414,"parent":"0.0","parent_split_bits":4,"last_scrub":"74'79","last_scrub_stamp":"2024-09-18T08:17:44.331322+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:00:11.893742+0000","last_clean_scrub_stamp":"2024-09-18T08:17:44.331322+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,5],"acting":[3,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.c","version":"271'20","reported_seq":393,"reported_epoch":424,"state":"active+clean","last_fresh":"2024-09-18T08:19:29.467579+0000","last_change":"2024-09-18T08:16:13.526582+0000","last_active":"2024-09-18T08:19:29.467579+0000","last_peered":"2024-09-18T08:19:29.467579+0000","last_clean":"2024-09-18T08:19:29.467579+0000","last_became_active":"2024-09-18T08:16:13.476918+0000","last_became_peered":"2024-09-18T08:16:13.476918+0000","last_unstale":"2024-09-18T08:19:29.467579+0000","last_undegraded":"2024-09-18T08:19:29.467579+0000","last_fullsized":"2024-09-18T08:19:29.467579+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"271'20","last_scrub_stamp":"2024-09-18T08:15:56.673129+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_clean_scrub_stamp":"2024-09-18T08:15:56.673129+0000","objects_scrubbed":0,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":28,"num_read_kb":23,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,1],"acting":[3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":811,"reported_epoch":424,"state":"active+clean","last_fresh":"2024-09-18T08:19:29.467035+0000","last_change":"2024-09-18T08:19:02.699664+0000","last_active":"2024-09-18T08:19:29.467035+0000","last_peered":"2024-09-18T08:19:29.467035+0000","last_clean":"2024-09-18T08:19:29.467035+0000","last_became_active":"2024-09-18T08:16:13.479148+0000","last_became_peered":"2024-09-18T08:16:13.479148+0000","last_unstale":"2024-09-18T08:19:29.467035+0000","last_undegraded":"2024-09-18T08:19:29.467035+0000","last_fullsized":"2024-09-18T08:19:29.467035+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:19:02.699596+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:46.860092+0000","last_clean_scrub_stamp":"2024-09-18T08:19:02.699596+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:20:13.563150+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.d","version":"402'14","reported_seq":1043,"reported_epoch":425,"state":"active+clean","last_fresh":"2024-09-18T08:19:37.002439+0000","last_change":"2024-09-18T08:18:27.675655+0000","last_active":"2024-09-18T08:19:37.002439+0000","last_peered":"2024-09-18T08:19:37.002439+0000","last_clean":"2024-09-18T08:19:37.002439+0000","last_became_active":"2024-09-18T08:18:27.653796+0000","last_became_peered":"2024-09-18T08:18:27.653796+0000","last_unstale":"2024-09-18T08:19:37.002439+0000","last_undegraded":"2024-09-18T08:19:37.002439+0000","last_fullsized":"2024-09-18T08:19:37.002439+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":0,"last_scrub":"338'13","last_scrub_stamp":"2024-09-18T08:17:38.886654+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:50.886830+0000","last_clean_scrub_stamp":"2024-09-18T08:17:38.886654+0000","objects_scrubbed":1,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:19:49.237365+0000","scrub_duration":26,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":455,"reported_epoch":425,"state":"active+clean","last_fresh":"2024-09-18T08:19:36.742159+0000","last_change":"2024-09-18T08:19:05.886778+0000","last_active":"2024-09-18T08:19:36.742159+0000","last_peered":"2024-09-18T08:19:36.742159+0000","last_clean":"2024-09-18T08:19:36.742159+0000","last_became_active":"2024-09-18T08:19:05.886455+0000","last_became_peered":"2024-09-18T08:19:05.886455+0000","last_unstale":"2024-09-18T08:19:36.742159+0000","last_undegraded":"2024-09-18T08:19:36.742159+0000","last_fullsized":"2024-09-18T08:19:36.742159+0000","mapping_epoch":413,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":414,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:15:47.604976+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_clean_scrub_stamp":"2024-09-18T08:15:47.604976+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.4","version":"425'4687","reported_seq":6750,"reported_epoch":425,"state":"active+recovering+undersized+remapped","last_fresh":"2024-09-18T08:19:38.555396+0000","last_change":"2024-09-18T08:19:06.999152+0000","last_active":"2024-09-18T08:19:38.555396+0000","last_peered":"2024-09-18T08:19:38.555396+0000","last_clean":"2024-09-18T08:19:04.794634+0000","last_became_active":"2024-09-18T08:19:06.962366+0000","last_became_peered":"2024-09-18T08:19:06.962366+0000","last_unstale":"2024-09-18T08:19:38.555396+0000","last_undegraded":"2024-09-18T08:19:38.555396+0000","last_fullsized":"2024-09-18T08:19:06.816683+0000","mapping_epoch":414,"log_start":"408'700","ondisk_log_start":"408'700","created":401,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:18:26.612016+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:18:26.612016+0000","last_clean_scrub_stamp":"2024-09-18T08:18:26.612016+0000","objects_scrubbed":0,"log_size":3987,"log_dups_size":0,"ondisk_log_size":3987,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":307167232,"num_objects":4687,"num_object_clones":0,"num_object_copies":14061,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":2224,"num_objects_unfound":0,"num_objects_dirty":4687,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4687,"num_write_kb":299968,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1000,"num_bytes_recovered":65536000,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6],"avail_no_missing":["7","2","6"],"object_location_counts":[{"shards":"2,6,7","objects":2224},{"shards":"6,7","objects":2463}],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.b","version":"271'15","reported_seq":868,"reported_epoch":425,"state":"active+clean","last_fresh":"2024-09-18T08:19:36.752700+0000","last_change":"2024-09-18T08:17:48.720750+0000","last_active":"2024-09-18T08:19:36.752700+0000","last_peered":"2024-09-18T08:19:36.752700+0000","last_clean":"2024-09-18T08:19:36.752700+0000","last_became_active":"2024-09-18T08:16:13.480279+0000","last_became_peered":"2024-09-18T08:16:13.480279+0000","last_unstale":"2024-09-18T08:19:36.752700+0000","last_undegraded":"2024-09-18T08:19:36.752700+0000","last_fullsized":"2024-09-18T08:19:36.752700+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"271'15","last_scrub_stamp":"2024-09-18T08:17:48.720672+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_clean_scrub_stamp":"2024-09-18T08:17:48.720672+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,3],"acting":[0,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":300,"reported_epoch":424,"state":"active+clean","last_fresh":"2024-09-18T08:19:29.467907+0000","last_change":"2024-09-18T08:18:22.633544+0000","last_active":"2024-09-18T08:19:29.467907+0000","last_peered":"2024-09-18T08:19:29.467907+0000","last_clean":"2024-09-18T08:19:29.467907+0000","last_became_active":"2024-09-18T08:18:17.892088+0000","last_became_peered":"2024-09-18T08:18:17.892088+0000","last_unstale":"2024-09-18T08:19:29.467907+0000","last_undegraded":"2024-09-18T08:19:29.467907+0000","last_fullsized":"2024-09-18T08:19:29.467907+0000","mapping_epoch":395,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":396,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:18:22.633492+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:46.860092+0000","last_clean_scrub_stamp":"2024-09-18T08:18:22.633492+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:19:42.386865+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,6],"acting":[3,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":652,"reported_epoch":424,"state":"active+clean","last_fresh":"2024-09-18T08:19:29.467828+0000","last_change":"2024-09-18T08:19:05.909652+0000","last_active":"2024-09-18T08:19:29.467828+0000","last_peered":"2024-09-18T08:19:29.467828+0000","last_clean":"2024-09-18T08:19:29.467828+0000","last_became_active":"2024-09-18T08:19:05.907780+0000","last_became_peered":"2024-09-18T08:19:05.907780+0000","last_unstale":"2024-09-18T08:19:29.467828+0000","last_undegraded":"2024-09-18T08:19:29.467828+0000","last_fullsized":"2024-09-18T08:19:29.467828+0000","mapping_epoch":413,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":414,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:18:19.057302+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:50.886830+0000","last_clean_scrub_stamp":"2024-09-18T08:18:19.057302+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":816,"reported_epoch":425,"state":"active+clean","last_fresh":"2024-09-18T08:19:36.754340+0000","last_change":"2024-09-18T08:19:05.912963+0000","last_active":"2024-09-18T08:19:36.754340+0000","last_peered":"2024-09-18T08:19:36.754340+0000","last_clean":"2024-09-18T08:19:36.754340+0000","last_became_active":"2024-09-18T08:19:05.912544+0000","last_became_peered":"2024-09-18T08:19:05.912544+0000","last_unstale":"2024-09-18T08:19:36.754340+0000","last_undegraded":"2024-09-18T08:19:36.754340+0000","last_fullsized":"2024-09-18T08:19:36.754340+0000","mapping_epoch":413,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":414,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:18:28.901610+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_clean_scrub_stamp":"2024-09-18T08:18:28.901610+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:19:43.320858+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.5","version":"425'4736","reported_seq":4795,"reported_epoch":425,"state":"active+clean","last_fresh":"2024-09-18T08:19:38.688641+0000","last_change":"2024-09-18T08:18:27.643971+0000","last_active":"2024-09-18T08:19:38.688641+0000","last_peered":"2024-09-18T08:19:38.688641+0000","last_clean":"2024-09-18T08:19:38.688641+0000","last_became_active":"2024-09-18T08:18:27.642478+0000","last_became_peered":"2024-09-18T08:18:27.642478+0000","last_unstale":"2024-09-18T08:19:38.688641+0000","last_undegraded":"2024-09-18T08:19:38.688641+0000","last_fullsized":"2024-09-18T08:19:38.688641+0000","mapping_epoch":401,"log_start":"406'500","ondisk_log_start":"406'500","created":401,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:18:26.612016+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:18:26.612016+0000","last_clean_scrub_stamp":"2024-09-18T08:18:26.612016+0000","objects_scrubbed":0,"log_size":4236,"log_dups_size":0,"ondisk_log_size":4236,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":310378496,"num_objects":4736,"num_object_clones":0,"num_object_copies":14208,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4736,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4736,"num_write_kb":303104,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.a","version":"271'35","reported_seq":736,"reported_epoch":425,"state":"active+clean","last_fresh":"2024-09-18T08:19:36.743426+0000","last_change":"2024-09-18T08:18:27.683070+0000","last_active":"2024-09-18T08:19:36.743426+0000","last_peered":"2024-09-18T08:19:36.743426+0000","last_clean":"2024-09-18T08:19:36.743426+0000","last_became_active":"2024-09-18T08:18:27.652693+0000","last_became_peered":"2024-09-18T08:18:27.652693+0000","last_unstale":"2024-09-18T08:19:36.743426+0000","last_undegraded":"2024-09-18T08:19:36.743426+0000","last_fullsized":"2024-09-18T08:19:36.743426+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":0,"last_scrub":"271'35","last_scrub_stamp":"2024-09-18T08:18:23.687385+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_clean_scrub_stamp":"2024-09-18T08:18:23.687385+0000","objects_scrubbed":9,"log_size":35,"log_dups_size":0,"ondisk_log_size":35,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:19:52.740313+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":55,"num_read_kb":44,"num_write":36,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":53,"num_bytes_recovered":550,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,7],"acting":[1,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":196,"reported_epoch":425,"state":"active+clean","last_fresh":"2024-09-18T08:19:36.748131+0000","last_change":"2024-09-18T08:18:27.644368+0000","last_active":"2024-09-18T08:19:36.748131+0000","last_peered":"2024-09-18T08:19:36.748131+0000","last_clean":"2024-09-18T08:19:36.748131+0000","last_became_active":"2024-09-18T08:18:27.643663+0000","last_became_peered":"2024-09-18T08:18:27.643663+0000","last_unstale":"2024-09-18T08:19:36.748131+0000","last_undegraded":"2024-09-18T08:19:36.748131+0000","last_fullsized":"2024-09-18T08:19:36.748131+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:12:31.205468+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:46.860092+0000","last_clean_scrub_stamp":"2024-09-18T08:12:31.205468+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:19:39.589299+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":474,"reported_epoch":424,"state":"active+clean","last_fresh":"2024-09-18T08:19:29.466236+0000","last_change":"2024-09-18T08:19:05.905542+0000","last_active":"2024-09-18T08:19:29.466236+0000","last_peered":"2024-09-18T08:19:29.466236+0000","last_clean":"2024-09-18T08:19:29.466236+0000","last_became_active":"2024-09-18T08:19:05.903320+0000","last_became_peered":"2024-09-18T08:19:05.903320+0000","last_unstale":"2024-09-18T08:19:29.466236+0000","last_undegraded":"2024-09-18T08:19:29.466236+0000","last_fullsized":"2024-09-18T08:19:29.466236+0000","mapping_epoch":413,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":414,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:18:22.116686+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:50.886830+0000","last_clean_scrub_stamp":"2024-09-18T08:18:22.116686+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:19:30.031433+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,7],"acting":[3,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":611,"reported_epoch":424,"state":"active+clean","last_fresh":"2024-09-18T08:19:29.470972+0000","last_change":"2024-09-18T08:19:05.890818+0000","last_active":"2024-09-18T08:19:29.470972+0000","last_peered":"2024-09-18T08:19:29.470972+0000","last_clean":"2024-09-18T08:19:29.470972+0000","last_became_active":"2024-09-18T08:19:05.890070+0000","last_became_peered":"2024-09-18T08:19:05.890070+0000","last_unstale":"2024-09-18T08:19:29.470972+0000","last_undegraded":"2024-09-18T08:19:29.470972+0000","last_fullsized":"2024-09-18T08:19:29.470972+0000","mapping_epoch":413,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":414,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:18:15.051582+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_clean_scrub_stamp":"2024-09-18T08:18:15.051582+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.6","version":"425'4706","reported_seq":7375,"reported_epoch":425,"state":"active+recovering+undersized+remapped","last_fresh":"2024-09-18T08:19:38.176295+0000","last_change":"2024-09-18T08:19:07.069959+0000","last_active":"2024-09-18T08:19:38.176295+0000","last_peered":"2024-09-18T08:19:38.176295+0000","last_clean":"2024-09-18T08:19:04.813349+0000","last_became_active":"2024-09-18T08:19:07.034824+0000","last_became_peered":"2024-09-18T08:19:07.034824+0000","last_unstale":"2024-09-18T08:19:38.176295+0000","last_undegraded":"2024-09-18T08:19:38.176295+0000","last_fullsized":"2024-09-18T08:19:06.813549+0000","mapping_epoch":414,"log_start":"408'900","ondisk_log_start":"408'900","created":401,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:18:26.612016+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:18:26.612016+0000","last_clean_scrub_stamp":"2024-09-18T08:18:26.612016+0000","objects_scrubbed":0,"log_size":3806,"log_dups_size":0,"ondisk_log_size":3806,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":308412416,"num_objects":4706,"num_object_clones":0,"num_object_copies":14118,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1846,"num_objects_unfound":0,"num_objects_dirty":4706,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4706,"num_write_kb":301184,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1303,"num_bytes_recovered":85393408,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,5],"acting":[0,1],"avail_no_missing":["0","1","2"],"object_location_counts":[{"shards":"0,1","objects":2860},{"shards":"0,1,2","objects":1846}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.9","version":"271'18","reported_seq":508,"reported_epoch":425,"state":"active+clean","last_fresh":"2024-09-18T08:19:36.742594+0000","last_change":"2024-09-18T08:16:13.525130+0000","last_active":"2024-09-18T08:19:36.742594+0000","last_peered":"2024-09-18T08:19:36.742594+0000","last_clean":"2024-09-18T08:19:36.742594+0000","last_became_active":"2024-09-18T08:16:13.482580+0000","last_became_peered":"2024-09-18T08:16:13.482580+0000","last_unstale":"2024-09-18T08:19:36.742594+0000","last_undegraded":"2024-09-18T08:19:36.742594+0000","last_fullsized":"2024-09-18T08:19:36.742594+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"271'18","last_scrub_stamp":"2024-09-18T08:15:49.666776+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_clean_scrub_stamp":"2024-09-18T08:15:49.666776+0000","objects_scrubbed":5,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:19:44.529654+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":33,"num_read_kb":24,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":880,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,6],"acting":[1,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":66,"reported_epoch":425,"state":"active+clean","last_fresh":"2024-09-18T08:19:36.748097+0000","last_change":"2024-09-18T08:18:56.624230+0000","last_active":"2024-09-18T08:19:36.748097+0000","last_peered":"2024-09-18T08:19:36.748097+0000","last_clean":"2024-09-18T08:19:36.748097+0000","last_became_active":"2024-09-18T08:18:27.654488+0000","last_became_peered":"2024-09-18T08:18:27.654488+0000","last_unstale":"2024-09-18T08:19:36.748097+0000","last_undegraded":"2024-09-18T08:19:36.748097+0000","last_fullsized":"2024-09-18T08:19:36.748097+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:18:56.624152+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:46.860092+0000","last_clean_scrub_stamp":"2024-09-18T08:18:56.624152+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:20:02.242984+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":796,"reported_epoch":425,"state":"active+clean","last_fresh":"2024-09-18T08:19:37.002186+0000","last_change":"2024-09-18T08:19:01.228154+0000","last_active":"2024-09-18T08:19:37.002186+0000","last_peered":"2024-09-18T08:19:37.002186+0000","last_clean":"2024-09-18T08:19:37.002186+0000","last_became_active":"2024-09-18T08:14:39.399301+0000","last_became_peered":"2024-09-18T08:14:39.399301+0000","last_unstale":"2024-09-18T08:19:37.002186+0000","last_undegraded":"2024-09-18T08:19:37.002186+0000","last_fullsized":"2024-09-18T08:19:37.002186+0000","mapping_epoch":307,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":308,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:19:01.228071+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:50.886830+0000","last_clean_scrub_stamp":"2024-09-18T08:19:01.228071+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:20:30.855655+0000","scrub_duration":20,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":795,"reported_epoch":424,"state":"active+clean","last_fresh":"2024-09-18T08:19:29.466685+0000","last_change":"2024-09-18T08:18:27.665000+0000","last_active":"2024-09-18T08:19:29.466685+0000","last_peered":"2024-09-18T08:19:29.466685+0000","last_clean":"2024-09-18T08:19:29.466685+0000","last_became_active":"2024-09-18T08:18:27.664412+0000","last_became_peered":"2024-09-18T08:18:27.664412+0000","last_unstale":"2024-09-18T08:19:29.466685+0000","last_undegraded":"2024-09-18T08:19:29.466685+0000","last_fullsized":"2024-09-18T08:19:29.466685+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:17:09.487823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_clean_scrub_stamp":"2024-09-18T08:17:09.487823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.7","version":"425'4795","reported_seq":4854,"reported_epoch":425,"state":"active+clean","last_fresh":"2024-09-18T08:19:38.683382+0000","last_change":"2024-09-18T08:18:27.651740+0000","last_active":"2024-09-18T08:19:38.683382+0000","last_peered":"2024-09-18T08:19:38.683382+0000","last_clean":"2024-09-18T08:19:38.683382+0000","last_became_active":"2024-09-18T08:18:27.651411+0000","last_became_peered":"2024-09-18T08:18:27.651411+0000","last_unstale":"2024-09-18T08:19:38.683382+0000","last_undegraded":"2024-09-18T08:19:38.683382+0000","last_fullsized":"2024-09-18T08:19:38.683382+0000","mapping_epoch":401,"log_start":"407'600","ondisk_log_start":"407'600","created":401,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:18:26.612016+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:18:26.612016+0000","last_clean_scrub_stamp":"2024-09-18T08:18:26.612016+0000","objects_scrubbed":0,"log_size":4195,"log_dups_size":0,"ondisk_log_size":4195,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:19:56.058465+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":314245120,"num_objects":4795,"num_object_clones":0,"num_object_copies":14385,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4795,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4795,"num_write_kb":306880,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.8","version":"271'27","reported_seq":251,"reported_epoch":425,"state":"active+clean","last_fresh":"2024-09-18T08:19:36.747239+0000","last_change":"2024-09-18T08:18:27.673319+0000","last_active":"2024-09-18T08:19:36.747239+0000","last_peered":"2024-09-18T08:19:36.747239+0000","last_clean":"2024-09-18T08:19:36.747239+0000","last_became_active":"2024-09-18T08:18:27.643053+0000","last_became_peered":"2024-09-18T08:18:27.643053+0000","last_unstale":"2024-09-18T08:19:36.747239+0000","last_undegraded":"2024-09-18T08:19:36.747239+0000","last_fullsized":"2024-09-18T08:19:36.747239+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":0,"last_scrub":"271'27","last_scrub_stamp":"2024-09-18T08:15:00.680764+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_clean_scrub_stamp":"2024-09-18T08:15:00.680764+0000","objects_scrubbed":7,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:19:42.649422+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":44,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":35,"num_bytes_recovered":550,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.f","version":"62'2","reported_seq":64,"reported_epoch":425,"state":"active+clean","last_fresh":"2024-09-18T08:19:36.747767+0000","last_change":"2024-09-18T08:18:27.672015+0000","last_active":"2024-09-18T08:19:36.747767+0000","last_peered":"2024-09-18T08:19:36.747767+0000","last_clean":"2024-09-18T08:19:36.747767+0000","last_became_active":"2024-09-18T08:18:27.654687+0000","last_became_peered":"2024-09-18T08:18:27.654687+0000","last_unstale":"2024-09-18T08:19:36.747767+0000","last_undegraded":"2024-09-18T08:19:36.747767+0000","last_fullsized":"2024-09-18T08:19:36.747767+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":0,"last_scrub":"62'2","last_scrub_stamp":"2024-09-18T08:17:49.718261+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:46.860092+0000","last_clean_scrub_stamp":"2024-09-18T08:17:49.718261+0000","objects_scrubbed":2,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":184,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,1],"acting":[7,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.9","version":"402'18","reported_seq":77,"reported_epoch":425,"state":"active+clean","last_fresh":"2024-09-18T08:19:36.747673+0000","last_change":"2024-09-18T08:18:27.666046+0000","last_active":"2024-09-18T08:19:36.747673+0000","last_peered":"2024-09-18T08:19:36.747673+0000","last_clean":"2024-09-18T08:19:36.747673+0000","last_became_active":"2024-09-18T08:18:27.652480+0000","last_became_peered":"2024-09-18T08:18:27.652480+0000","last_unstale":"2024-09-18T08:19:36.747673+0000","last_undegraded":"2024-09-18T08:19:36.747673+0000","last_fullsized":"2024-09-18T08:19:36.747673+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":0,"last_scrub":"294'14","last_scrub_stamp":"2024-09-18T08:14:28.686640+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:50.886830+0000","last_clean_scrub_stamp":"2024-09-18T08:14:28.686640+0000","objects_scrubbed":1,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":197,"reported_epoch":425,"state":"active+clean","last_fresh":"2024-09-18T08:19:36.747032+0000","last_change":"2024-09-18T08:19:04.680209+0000","last_active":"2024-09-18T08:19:36.747032+0000","last_peered":"2024-09-18T08:19:36.747032+0000","last_clean":"2024-09-18T08:19:36.747032+0000","last_became_active":"2024-09-18T08:15:41.449889+0000","last_became_peered":"2024-09-18T08:15:41.449889+0000","last_unstale":"2024-09-18T08:19:36.747032+0000","last_undegraded":"2024-09-18T08:19:36.747032+0000","last_fullsized":"2024-09-18T08:19:36.747032+0000","mapping_epoch":336,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":337,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:19:04.680153+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_clean_scrub_stamp":"2024-09-18T08:19:04.680153+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:20:21.463313+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":716,"reported_epoch":425,"state":"active+clean","last_fresh":"2024-09-18T08:19:36.746680+0000","last_change":"2024-09-18T08:19:03.092160+0000","last_active":"2024-09-18T08:19:36.746680+0000","last_peered":"2024-09-18T08:19:36.746680+0000","last_clean":"2024-09-18T08:19:36.746680+0000","last_became_active":"2024-09-18T08:15:41.446544+0000","last_became_peered":"2024-09-18T08:15:41.446544+0000","last_unstale":"2024-09-18T08:19:36.746680+0000","last_undegraded":"2024-09-18T08:19:36.746680+0000","last_fullsized":"2024-09-18T08:19:36.746680+0000","mapping_epoch":336,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":337,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:19:03.092077+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:46.860092+0000","last_clean_scrub_stamp":"2024-09-18T08:19:03.092077+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:20:20.600568+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.17","version":"271'12","reported_seq":844,"reported_epoch":424,"state":"active+clean","last_fresh":"2024-09-18T08:19:29.469479+0000","last_change":"2024-09-18T08:18:27.699070+0000","last_active":"2024-09-18T08:19:29.469479+0000","last_peered":"2024-09-18T08:19:29.469479+0000","last_clean":"2024-09-18T08:19:29.469479+0000","last_became_active":"2024-09-18T08:18:27.663338+0000","last_became_peered":"2024-09-18T08:18:27.663338+0000","last_unstale":"2024-09-18T08:19:29.469479+0000","last_undegraded":"2024-09-18T08:19:29.469479+0000","last_fullsized":"2024-09-18T08:19:29.469479+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":0,"last_scrub":"271'12","last_scrub_stamp":"2024-09-18T08:14:33.700931+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_clean_scrub_stamp":"2024-09-18T08:14:33.700931+0000","objects_scrubbed":3,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:19:41.568380+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":18,"num_read_kb":15,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":197,"reported_epoch":425,"state":"active+clean","last_fresh":"2024-09-18T08:19:36.747929+0000","last_change":"2024-09-18T08:18:58.734315+0000","last_active":"2024-09-18T08:19:36.747929+0000","last_peered":"2024-09-18T08:19:36.747929+0000","last_clean":"2024-09-18T08:19:36.747929+0000","last_became_active":"2024-09-18T08:15:41.445332+0000","last_became_peered":"2024-09-18T08:15:41.445332+0000","last_unstale":"2024-09-18T08:19:36.747929+0000","last_undegraded":"2024-09-18T08:19:36.747929+0000","last_fullsized":"2024-09-18T08:19:36.747929+0000","mapping_epoch":336,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":337,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:18:58.734241+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:50.886830+0000","last_clean_scrub_stamp":"2024-09-18T08:18:58.734241+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:20:21.012782+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,1],"acting":[5,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":66,"reported_epoch":425,"state":"active+clean","last_fresh":"2024-09-18T08:19:36.748496+0000","last_change":"2024-09-18T08:18:29.808538+0000","last_active":"2024-09-18T08:19:36.748496+0000","last_peered":"2024-09-18T08:19:36.748496+0000","last_clean":"2024-09-18T08:19:36.748496+0000","last_became_active":"2024-09-18T08:18:27.653368+0000","last_became_peered":"2024-09-18T08:18:27.653368+0000","last_unstale":"2024-09-18T08:19:36.748496+0000","last_undegraded":"2024-09-18T08:19:36.748496+0000","last_fullsized":"2024-09-18T08:19:36.748496+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:18:29.808476+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_clean_scrub_stamp":"2024-09-18T08:18:29.808476+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:19:54.372875+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,5],"acting":[7,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"271'15","reported_seq":838,"reported_epoch":425,"state":"active+clean","last_fresh":"2024-09-18T08:19:36.751768+0000","last_change":"2024-09-18T08:18:59.080889+0000","last_active":"2024-09-18T08:19:36.751768+0000","last_peered":"2024-09-18T08:19:36.751768+0000","last_clean":"2024-09-18T08:19:36.751768+0000","last_became_active":"2024-09-18T08:18:27.658637+0000","last_became_peered":"2024-09-18T08:18:27.658637+0000","last_unstale":"2024-09-18T08:19:36.751768+0000","last_undegraded":"2024-09-18T08:19:36.751768+0000","last_fullsized":"2024-09-18T08:19:36.751768+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":0,"last_scrub":"271'15","last_scrub_stamp":"2024-09-18T08:18:59.080655+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_clean_scrub_stamp":"2024-09-18T08:18:59.080655+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:20:11.583445+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":430,"reported_epoch":425,"state":"active+clean","last_fresh":"2024-09-18T08:19:36.749931+0000","last_change":"2024-09-18T08:18:27.650329+0000","last_active":"2024-09-18T08:19:36.749931+0000","last_peered":"2024-09-18T08:19:36.749931+0000","last_clean":"2024-09-18T08:19:36.749931+0000","last_became_active":"2024-09-18T08:18:27.649876+0000","last_became_peered":"2024-09-18T08:18:27.649876+0000","last_unstale":"2024-09-18T08:19:36.749931+0000","last_undegraded":"2024-09-18T08:19:36.749931+0000","last_fullsized":"2024-09-18T08:19:36.749931+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:17:50.771903+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:46.860092+0000","last_clean_scrub_stamp":"2024-09-18T08:17:50.771903+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,6],"acting":[0,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":789,"reported_epoch":424,"state":"active+clean","last_fresh":"2024-09-18T08:19:29.469245+0000","last_change":"2024-09-18T08:18:27.660413+0000","last_active":"2024-09-18T08:19:29.469245+0000","last_peered":"2024-09-18T08:19:29.469245+0000","last_clean":"2024-09-18T08:19:29.469245+0000","last_became_active":"2024-09-18T08:18:27.660093+0000","last_became_peered":"2024-09-18T08:18:27.660093+0000","last_unstale":"2024-09-18T08:19:29.469245+0000","last_undegraded":"2024-09-18T08:19:29.469245+0000","last_fullsized":"2024-09-18T08:19:29.469245+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:17:06.569828+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:50.886830+0000","last_clean_scrub_stamp":"2024-09-18T08:17:06.569828+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":31,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":779,"reported_epoch":425,"state":"active+clean","last_fresh":"2024-09-18T08:19:36.747367+0000","last_change":"2024-09-18T08:19:05.895365+0000","last_active":"2024-09-18T08:19:36.747367+0000","last_peered":"2024-09-18T08:19:36.747367+0000","last_clean":"2024-09-18T08:19:36.747367+0000","last_became_active":"2024-09-18T08:19:05.894972+0000","last_became_peered":"2024-09-18T08:19:05.894972+0000","last_unstale":"2024-09-18T08:19:36.747367+0000","last_undegraded":"2024-09-18T08:19:36.747367+0000","last_fullsized":"2024-09-18T08:19:36.747367+0000","mapping_epoch":413,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":414,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:18:31.070711+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_clean_scrub_stamp":"2024-09-18T08:18:31.070711+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:19:54.352199+0000","scrub_duration":22,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,5],"acting":[7,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.15","version":"271'15","reported_seq":224,"reported_epoch":425,"state":"active+clean","last_fresh":"2024-09-18T08:19:36.748281+0000","last_change":"2024-09-18T08:18:27.674004+0000","last_active":"2024-09-18T08:19:36.748281+0000","last_peered":"2024-09-18T08:19:36.748281+0000","last_clean":"2024-09-18T08:19:36.748281+0000","last_became_active":"2024-09-18T08:18:27.646211+0000","last_became_peered":"2024-09-18T08:18:27.646211+0000","last_unstale":"2024-09-18T08:19:36.748281+0000","last_undegraded":"2024-09-18T08:19:36.748281+0000","last_fullsized":"2024-09-18T08:19:36.748281+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":0,"last_scrub":"271'15","last_scrub_stamp":"2024-09-18T08:17:32.638811+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_clean_scrub_stamp":"2024-09-18T08:17:32.638811+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":20,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":25,"num_read_kb":19,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":811,"reported_epoch":425,"state":"active+clean","last_fresh":"2024-09-18T08:19:36.753303+0000","last_change":"2024-09-18T08:18:27.656075+0000","last_active":"2024-09-18T08:19:36.753303+0000","last_peered":"2024-09-18T08:19:36.753303+0000","last_clean":"2024-09-18T08:19:36.753303+0000","last_became_active":"2024-09-18T08:18:27.655758+0000","last_became_peered":"2024-09-18T08:18:27.655758+0000","last_unstale":"2024-09-18T08:19:36.753303+0000","last_undegraded":"2024-09-18T08:19:36.753303+0000","last_fullsized":"2024-09-18T08:19:36.753303+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:18:24.804845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:46.860092+0000","last_clean_scrub_stamp":"2024-09-18T08:18:24.804845+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:19:48.037934+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.14","version":"415'18","reported_seq":1058,"reported_epoch":424,"state":"active+clean","last_fresh":"2024-09-18T08:19:29.467389+0000","last_change":"2024-09-18T08:19:05.916229+0000","last_active":"2024-09-18T08:19:29.467389+0000","last_peered":"2024-09-18T08:19:29.467389+0000","last_clean":"2024-09-18T08:19:29.467389+0000","last_became_active":"2024-09-18T08:19:05.900155+0000","last_became_peered":"2024-09-18T08:19:05.900155+0000","last_unstale":"2024-09-18T08:19:29.467389+0000","last_undegraded":"2024-09-18T08:19:29.467389+0000","last_fullsized":"2024-09-18T08:19:29.467389+0000","mapping_epoch":413,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":414,"parent":"0.0","parent_split_bits":0,"last_scrub":"310'16","last_scrub_stamp":"2024-09-18T08:17:39.424098+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:50.886830+0000","last_clean_scrub_stamp":"2024-09-18T08:17:39.424098+0000","objects_scrubbed":1,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":16,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":519,"reported_epoch":425,"state":"active+clean","last_fresh":"2024-09-18T08:19:37.002898+0000","last_change":"2024-09-18T08:18:24.109825+0000","last_active":"2024-09-18T08:19:37.002898+0000","last_peered":"2024-09-18T08:19:37.002898+0000","last_clean":"2024-09-18T08:19:37.002898+0000","last_became_active":"2024-09-18T08:16:13.467534+0000","last_became_peered":"2024-09-18T08:16:13.467534+0000","last_unstale":"2024-09-18T08:19:37.002898+0000","last_undegraded":"2024-09-18T08:19:37.002898+0000","last_fullsized":"2024-09-18T08:19:37.002898+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:18:24.109778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_clean_scrub_stamp":"2024-09-18T08:18:24.109778+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:20:34.025882+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,0],"acting":[2,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.14","version":"271'20","reported_seq":863,"reported_epoch":424,"state":"active+clean","last_fresh":"2024-09-18T08:19:29.467737+0000","last_change":"2024-09-18T08:18:27.701608+0000","last_active":"2024-09-18T08:19:29.467737+0000","last_peered":"2024-09-18T08:19:29.467737+0000","last_clean":"2024-09-18T08:19:29.467737+0000","last_became_active":"2024-09-18T08:18:27.663877+0000","last_became_peered":"2024-09-18T08:18:27.663877+0000","last_unstale":"2024-09-18T08:19:29.467737+0000","last_undegraded":"2024-09-18T08:19:29.467737+0000","last_fullsized":"2024-09-18T08:19:29.467737+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":0,"last_scrub":"271'20","last_scrub_stamp":"2024-09-18T08:15:52.819801+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_clean_scrub_stamp":"2024-09-18T08:15:52.819801+0000","objects_scrubbed":5,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":16,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":25,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,7],"acting":[3,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":65,"reported_epoch":425,"state":"active+clean","last_fresh":"2024-09-18T08:19:36.748093+0000","last_change":"2024-09-18T08:19:05.900476+0000","last_active":"2024-09-18T08:19:36.748093+0000","last_peered":"2024-09-18T08:19:36.748093+0000","last_clean":"2024-09-18T08:19:36.748093+0000","last_became_active":"2024-09-18T08:19:05.900189+0000","last_became_peered":"2024-09-18T08:19:05.900189+0000","last_unstale":"2024-09-18T08:19:36.748093+0000","last_undegraded":"2024-09-18T08:19:36.748093+0000","last_fullsized":"2024-09-18T08:19:36.748093+0000","mapping_epoch":413,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":414,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:17:30.908835+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:46.860092+0000","last_clean_scrub_stamp":"2024-09-18T08:17:30.908835+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":21,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,0],"acting":[7,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.15","version":"338'12","reported_seq":249,"reported_epoch":425,"state":"active+clean","last_fresh":"2024-09-18T08:19:36.747947+0000","last_change":"2024-09-18T08:18:24.511983+0000","last_active":"2024-09-18T08:19:36.747947+0000","last_peered":"2024-09-18T08:19:36.747947+0000","last_clean":"2024-09-18T08:19:36.747947+0000","last_became_active":"2024-09-18T08:15:41.455243+0000","last_became_peered":"2024-09-18T08:15:41.455243+0000","last_unstale":"2024-09-18T08:19:36.747947+0000","last_undegraded":"2024-09-18T08:19:36.747947+0000","last_fullsized":"2024-09-18T08:19:36.747947+0000","mapping_epoch":336,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":337,"parent":"0.0","parent_split_bits":0,"last_scrub":"338'12","last_scrub_stamp":"2024-09-18T08:18:24.511895+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:50.886830+0000","last_clean_scrub_stamp":"2024-09-18T08:18:24.511895+0000","objects_scrubbed":1,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:19:41.815901+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":793,"reported_epoch":425,"state":"active+clean","last_fresh":"2024-09-18T08:19:36.747209+0000","last_change":"2024-09-18T08:19:00.044481+0000","last_active":"2024-09-18T08:19:36.747209+0000","last_peered":"2024-09-18T08:19:36.747209+0000","last_clean":"2024-09-18T08:19:36.747209+0000","last_became_active":"2024-09-18T08:18:27.658009+0000","last_became_peered":"2024-09-18T08:18:27.658009+0000","last_unstale":"2024-09-18T08:19:36.747209+0000","last_undegraded":"2024-09-18T08:19:36.747209+0000","last_fullsized":"2024-09-18T08:19:36.747209+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:19:00.044416+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_clean_scrub_stamp":"2024-09-18T08:19:00.044416+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:20:15.856801+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.13","version":"271'19","reported_seq":486,"reported_epoch":425,"state":"active+clean","last_fresh":"2024-09-18T08:19:36.748038+0000","last_change":"2024-09-18T08:17:39.665796+0000","last_active":"2024-09-18T08:19:36.748038+0000","last_peered":"2024-09-18T08:19:36.748038+0000","last_clean":"2024-09-18T08:19:36.748038+0000","last_became_active":"2024-09-18T08:16:13.454640+0000","last_became_peered":"2024-09-18T08:16:13.454640+0000","last_unstale":"2024-09-18T08:19:36.748038+0000","last_undegraded":"2024-09-18T08:19:36.748038+0000","last_fullsized":"2024-09-18T08:19:36.748038+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"271'19","last_scrub_stamp":"2024-09-18T08:17:39.665588+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_clean_scrub_stamp":"2024-09-18T08:17:39.665588+0000","objects_scrubbed":5,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":31,"num_read_kb":24,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,5],"acting":[1,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":58,"reported_epoch":425,"state":"active+clean","last_fresh":"2024-09-18T08:19:36.748028+0000","last_change":"2024-09-18T08:18:27.658805+0000","last_active":"2024-09-18T08:19:36.748028+0000","last_peered":"2024-09-18T08:19:36.748028+0000","last_clean":"2024-09-18T08:19:36.748028+0000","last_became_active":"2024-09-18T08:18:27.658476+0000","last_became_peered":"2024-09-18T08:18:27.658476+0000","last_unstale":"2024-09-18T08:19:36.748028+0000","last_undegraded":"2024-09-18T08:19:36.748028+0000","last_fullsized":"2024-09-18T08:19:36.748028+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:18:25.097909+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:46.860092+0000","last_clean_scrub_stamp":"2024-09-18T08:18:25.097909+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":663,"reported_epoch":425,"state":"active+clean","last_fresh":"2024-09-18T08:19:36.742972+0000","last_change":"2024-09-18T08:15:55.335233+0000","last_active":"2024-09-18T08:19:36.742972+0000","last_peered":"2024-09-18T08:19:36.742972+0000","last_clean":"2024-09-18T08:19:36.742972+0000","last_became_active":"2024-09-18T08:15:41.438048+0000","last_became_peered":"2024-09-18T08:15:41.438048+0000","last_unstale":"2024-09-18T08:19:36.742972+0000","last_undegraded":"2024-09-18T08:19:36.742972+0000","last_fullsized":"2024-09-18T08:19:36.742972+0000","mapping_epoch":336,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":337,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:15:55.334983+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:50.886830+0000","last_clean_scrub_stamp":"2024-09-18T08:15:55.334983+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:19:50.692696+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":794,"reported_epoch":424,"state":"active+clean","last_fresh":"2024-09-18T08:19:29.466506+0000","last_change":"2024-09-18T08:17:37.394735+0000","last_active":"2024-09-18T08:19:29.466506+0000","last_peered":"2024-09-18T08:19:29.466506+0000","last_clean":"2024-09-18T08:19:29.466506+0000","last_became_active":"2024-09-18T08:15:41.464961+0000","last_became_peered":"2024-09-18T08:15:41.464961+0000","last_unstale":"2024-09-18T08:19:29.466506+0000","last_undegraded":"2024-09-18T08:19:29.466506+0000","last_fullsized":"2024-09-18T08:19:29.466506+0000","mapping_epoch":336,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":337,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:17:37.394532+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_clean_scrub_stamp":"2024-09-18T08:17:37.394532+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.12","version":"271'15","reported_seq":676,"reported_epoch":425,"state":"active+clean","last_fresh":"2024-09-18T08:19:36.743327+0000","last_change":"2024-09-18T08:19:05.918020+0000","last_active":"2024-09-18T08:19:36.743327+0000","last_peered":"2024-09-18T08:19:36.743327+0000","last_clean":"2024-09-18T08:19:36.743327+0000","last_became_active":"2024-09-18T08:19:05.904293+0000","last_became_peered":"2024-09-18T08:19:05.904293+0000","last_unstale":"2024-09-18T08:19:36.743327+0000","last_undegraded":"2024-09-18T08:19:36.743327+0000","last_fullsized":"2024-09-18T08:19:36.743327+0000","mapping_epoch":413,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":414,"parent":"0.0","parent_split_bits":0,"last_scrub":"271'15","last_scrub_stamp":"2024-09-18T08:18:31.593238+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_clean_scrub_stamp":"2024-09-18T08:18:31.593238+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:19:59.572517+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":25,"num_read_kb":19,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,0],"acting":[1,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":587,"reported_epoch":424,"state":"active+clean","last_fresh":"2024-09-18T08:19:29.468259+0000","last_change":"2024-09-18T08:18:27.663193+0000","last_active":"2024-09-18T08:19:29.468259+0000","last_peered":"2024-09-18T08:19:29.468259+0000","last_clean":"2024-09-18T08:19:29.468259+0000","last_became_active":"2024-09-18T08:18:27.662885+0000","last_became_peered":"2024-09-18T08:18:27.662885+0000","last_unstale":"2024-09-18T08:19:29.468259+0000","last_undegraded":"2024-09-18T08:19:29.468259+0000","last_fullsized":"2024-09-18T08:19:29.468259+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:15:58.978411+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:46.860092+0000","last_clean_scrub_stamp":"2024-09-18T08:15:58.978411+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":18,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":778,"reported_epoch":424,"state":"active+clean","last_fresh":"2024-09-18T08:19:29.466779+0000","last_change":"2024-09-18T08:18:28.695902+0000","last_active":"2024-09-18T08:19:29.466779+0000","last_peered":"2024-09-18T08:19:29.466779+0000","last_clean":"2024-09-18T08:19:29.466779+0000","last_became_active":"2024-09-18T08:14:39.402406+0000","last_became_peered":"2024-09-18T08:14:39.402406+0000","last_unstale":"2024-09-18T08:19:29.466779+0000","last_undegraded":"2024-09-18T08:19:29.466779+0000","last_fullsized":"2024-09-18T08:19:29.466779+0000","mapping_epoch":307,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":308,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:18:28.695842+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:50.886830+0000","last_clean_scrub_stamp":"2024-09-18T08:18:28.695842+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:19:50.760247+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.10","version":"69'1","reported_seq":786,"reported_epoch":425,"state":"active+clean","last_fresh":"2024-09-18T08:19:36.746329+0000","last_change":"2024-09-18T08:15:41.480106+0000","last_active":"2024-09-18T08:19:36.746329+0000","last_peered":"2024-09-18T08:19:36.746329+0000","last_clean":"2024-09-18T08:19:36.746329+0000","last_became_active":"2024-09-18T08:15:41.444085+0000","last_became_peered":"2024-09-18T08:15:41.444085+0000","last_unstale":"2024-09-18T08:19:36.746329+0000","last_undegraded":"2024-09-18T08:19:36.746329+0000","last_fullsized":"2024-09-18T08:19:36.746329+0000","mapping_epoch":336,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":337,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'1","last_scrub_stamp":"2024-09-18T08:15:00.219678+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_clean_scrub_stamp":"2024-09-18T08:15:00.219678+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:19:39.940520+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":26,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.11","version":"271'19","reported_seq":932,"reported_epoch":424,"state":"active+clean","last_fresh":"2024-09-18T08:19:29.466256+0000","last_change":"2024-09-18T08:18:27.695649+0000","last_active":"2024-09-18T08:19:29.466256+0000","last_peered":"2024-09-18T08:19:29.466256+0000","last_clean":"2024-09-18T08:19:29.466256+0000","last_became_active":"2024-09-18T08:18:27.659293+0000","last_became_peered":"2024-09-18T08:18:27.659293+0000","last_unstale":"2024-09-18T08:19:29.466256+0000","last_undegraded":"2024-09-18T08:19:29.466256+0000","last_fullsized":"2024-09-18T08:19:29.466256+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":0,"last_scrub":"271'19","last_scrub_stamp":"2024-09-18T08:14:38.809490+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_clean_scrub_stamp":"2024-09-18T08:14:38.809490+0000","objects_scrubbed":5,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:19:45.503977+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":32,"num_read_kb":25,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":30,"num_bytes_recovered":660,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":204,"reported_epoch":425,"state":"active+clean","last_fresh":"2024-09-18T08:19:36.747879+0000","last_change":"2024-09-18T08:18:27.644325+0000","last_active":"2024-09-18T08:19:36.747879+0000","last_peered":"2024-09-18T08:19:36.747879+0000","last_clean":"2024-09-18T08:19:36.747879+0000","last_became_active":"2024-09-18T08:18:27.643957+0000","last_became_peered":"2024-09-18T08:18:27.643957+0000","last_unstale":"2024-09-18T08:19:36.747879+0000","last_undegraded":"2024-09-18T08:19:36.747879+0000","last_fullsized":"2024-09-18T08:19:36.747879+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:18:18.611302+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:46.860092+0000","last_clean_scrub_stamp":"2024-09-18T08:18:18.611302+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":58,"reported_epoch":425,"state":"active+clean","last_fresh":"2024-09-18T08:19:36.746514+0000","last_change":"2024-09-18T08:18:27.652157+0000","last_active":"2024-09-18T08:19:36.746514+0000","last_peered":"2024-09-18T08:19:36.746514+0000","last_clean":"2024-09-18T08:19:36.746514+0000","last_became_active":"2024-09-18T08:18:27.651810+0000","last_became_peered":"2024-09-18T08:18:27.651810+0000","last_unstale":"2024-09-18T08:19:36.746514+0000","last_undegraded":"2024-09-18T08:19:36.746514+0000","last_fullsized":"2024-09-18T08:19:36.746514+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:14:37.743856+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:50.886830+0000","last_clean_scrub_stamp":"2024-09-18T08:14:37.743856+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:19:40.737404+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,0],"acting":[7,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":777,"reported_epoch":424,"state":"active+clean","last_fresh":"2024-09-18T08:19:29.466941+0000","last_change":"2024-09-18T08:18:21.660255+0000","last_active":"2024-09-18T08:19:29.466941+0000","last_peered":"2024-09-18T08:19:29.466941+0000","last_clean":"2024-09-18T08:19:29.466941+0000","last_became_active":"2024-09-18T08:06:24.128582+0000","last_became_peered":"2024-09-18T08:06:24.128582+0000","last_unstale":"2024-09-18T08:19:29.466941+0000","last_undegraded":"2024-09-18T08:19:29.466941+0000","last_fullsized":"2024-09-18T08:19:29.466941+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:18:21.660213+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_clean_scrub_stamp":"2024-09-18T08:18:21.660213+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.10","version":"271'8","reported_seq":827,"reported_epoch":424,"state":"active+clean","last_fresh":"2024-09-18T08:19:29.470793+0000","last_change":"2024-09-18T08:15:50.790606+0000","last_active":"2024-09-18T08:19:29.470793+0000","last_peered":"2024-09-18T08:19:29.470793+0000","last_clean":"2024-09-18T08:19:29.470793+0000","last_became_active":"2024-09-18T08:14:39.404257+0000","last_became_peered":"2024-09-18T08:14:39.404257+0000","last_unstale":"2024-09-18T08:19:29.470793+0000","last_undegraded":"2024-09-18T08:19:29.470793+0000","last_fullsized":"2024-09-18T08:19:29.470793+0000","mapping_epoch":307,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":308,"parent":"0.0","parent_split_bits":0,"last_scrub":"271'8","last_scrub_stamp":"2024-09-18T08:15:50.790525+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_clean_scrub_stamp":"2024-09-18T08:15:50.790525+0000","objects_scrubbed":2,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":12,"num_read_kb":10,"num_write":8,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":805,"reported_epoch":425,"state":"active+clean","last_fresh":"2024-09-18T08:19:36.753611+0000","last_change":"2024-09-18T08:18:25.799322+0000","last_active":"2024-09-18T08:19:36.753611+0000","last_peered":"2024-09-18T08:19:36.753611+0000","last_clean":"2024-09-18T08:19:36.753611+0000","last_became_active":"2024-09-18T08:15:41.466964+0000","last_became_peered":"2024-09-18T08:15:41.466964+0000","last_unstale":"2024-09-18T08:19:36.753611+0000","last_undegraded":"2024-09-18T08:19:36.753611+0000","last_fullsized":"2024-09-18T08:19:36.753611+0000","mapping_epoch":336,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":337,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:18:25.799264+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:46.860092+0000","last_clean_scrub_stamp":"2024-09-18T08:18:25.799264+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":430,"reported_epoch":425,"state":"active+clean","last_fresh":"2024-09-18T08:19:36.749725+0000","last_change":"2024-09-18T08:18:27.645673+0000","last_active":"2024-09-18T08:19:36.749725+0000","last_peered":"2024-09-18T08:19:36.749725+0000","last_clean":"2024-09-18T08:19:36.749725+0000","last_became_active":"2024-09-18T08:18:27.645122+0000","last_became_peered":"2024-09-18T08:18:27.645122+0000","last_unstale":"2024-09-18T08:19:36.749725+0000","last_undegraded":"2024-09-18T08:19:36.749725+0000","last_fullsized":"2024-09-18T08:19:36.749725+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:18:14.912464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:50.886830+0000","last_clean_scrub_stamp":"2024-09-18T08:18:14.912464+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,7],"acting":[0,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":570,"reported_epoch":424,"state":"active+clean","last_fresh":"2024-09-18T08:19:29.467678+0000","last_change":"2024-09-18T08:19:05.890924+0000","last_active":"2024-09-18T08:19:29.467678+0000","last_peered":"2024-09-18T08:19:29.467678+0000","last_clean":"2024-09-18T08:19:29.467678+0000","last_became_active":"2024-09-18T08:19:05.890580+0000","last_became_peered":"2024-09-18T08:19:05.890580+0000","last_unstale":"2024-09-18T08:19:29.467678+0000","last_undegraded":"2024-09-18T08:19:29.467678+0000","last_fullsized":"2024-09-18T08:19:29.467678+0000","mapping_epoch":413,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":414,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:19:03.125974+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_clean_scrub_stamp":"2024-09-18T08:19:03.125974+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:20:23.420957+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":635,"reported_epoch":425,"state":"active+clean","last_fresh":"2024-09-18T08:19:36.755316+0000","last_change":"2024-09-18T08:16:13.468068+0000","last_active":"2024-09-18T08:19:36.755316+0000","last_peered":"2024-09-18T08:19:36.755316+0000","last_clean":"2024-09-18T08:19:36.755316+0000","last_became_active":"2024-09-18T08:16:13.466812+0000","last_became_peered":"2024-09-18T08:16:13.466812+0000","last_unstale":"2024-09-18T08:19:36.755316+0000","last_undegraded":"2024-09-18T08:19:36.755316+0000","last_fullsized":"2024-09-18T08:19:36.755316+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:15:45.297240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_clean_scrub_stamp":"2024-09-18T08:15:45.297240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:19:45.562715+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,2],"acting":[1,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":777,"reported_epoch":424,"state":"active+clean","last_fresh":"2024-09-18T08:19:29.471047+0000","last_change":"2024-09-18T08:15:43.836627+0000","last_active":"2024-09-18T08:19:29.471047+0000","last_peered":"2024-09-18T08:19:29.471047+0000","last_clean":"2024-09-18T08:19:29.471047+0000","last_became_active":"2024-09-18T08:14:39.405347+0000","last_became_peered":"2024-09-18T08:14:39.405347+0000","last_unstale":"2024-09-18T08:19:29.471047+0000","last_undegraded":"2024-09-18T08:19:29.471047+0000","last_fullsized":"2024-09-18T08:19:29.471047+0000","mapping_epoch":307,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":308,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:15:43.836549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:46.860092+0000","last_clean_scrub_stamp":"2024-09-18T08:15:43.836549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":20,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"271'19","reported_seq":209,"reported_epoch":425,"state":"active+clean","last_fresh":"2024-09-18T08:19:36.747949+0000","last_change":"2024-09-18T08:18:20.608272+0000","last_active":"2024-09-18T08:19:36.747949+0000","last_peered":"2024-09-18T08:19:36.747949+0000","last_clean":"2024-09-18T08:19:36.747949+0000","last_became_active":"2024-09-18T08:15:41.445198+0000","last_became_peered":"2024-09-18T08:15:41.445198+0000","last_unstale":"2024-09-18T08:19:36.747949+0000","last_undegraded":"2024-09-18T08:19:36.747949+0000","last_fullsized":"2024-09-18T08:19:36.747949+0000","mapping_epoch":336,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":337,"parent":"0.0","parent_split_bits":0,"last_scrub":"271'19","last_scrub_stamp":"2024-09-18T08:18:20.608110+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_clean_scrub_stamp":"2024-09-18T08:18:20.608110+0000","objects_scrubbed":5,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":31,"num_read_kb":24,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,1],"acting":[5,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":809,"reported_epoch":425,"state":"active+clean","last_fresh":"2024-09-18T08:19:36.751220+0000","last_change":"2024-09-18T08:19:05.904396+0000","last_active":"2024-09-18T08:19:36.751220+0000","last_peered":"2024-09-18T08:19:36.751220+0000","last_clean":"2024-09-18T08:19:36.751220+0000","last_became_active":"2024-09-18T08:19:05.903954+0000","last_became_peered":"2024-09-18T08:19:05.903954+0000","last_unstale":"2024-09-18T08:19:36.751220+0000","last_undegraded":"2024-09-18T08:19:36.751220+0000","last_fullsized":"2024-09-18T08:19:36.751220+0000","mapping_epoch":413,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":414,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:12:31.664456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:50.886830+0000","last_clean_scrub_stamp":"2024-09-18T08:12:31.664456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,1],"acting":[0,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":11,"num_pg":16,"stat_sum":{"num_bytes":4890689536,"num_objects":74626,"num_object_clones":0,"num_object_copies":223878,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":12242,"num_objects_unfound":0,"num_objects_dirty":74626,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":74626,"num_write_kb":4776064,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6887,"num_bytes_recovered":451346432,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":15297609728,"data_stored":15297609728,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":62326,"ondisk_log_size":62326,"up":48,"acting":42,"num_store_stats":7},{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":416,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":1235,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":32,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":125,"ondisk_log_size":125,"up":96,"acting":96,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":178,"num_object_clones":0,"num_object_copies":534,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":178,"num_whiteouts":0,"num_read":1106,"num_read_kb":863,"num_write":703,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":559,"num_bytes_recovered":10830,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":475136,"data_stored":12690,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3639,"internal_metadata":0},"log_size":690,"ondisk_log_size":690,"up":96,"acting":96,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":1613,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":14,"num_read_kb":14,"num_write":10,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":17,"num_bytes_recovered":6130,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":94208,"data_stored":6016,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":6,"ondisk_log_size":6,"up":96,"acting":96,"num_store_stats":8},{"poolid":2,"num_pg":3,"stat_sum":{"num_bytes":408,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":467,"num_read_kb":462,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":797,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":8,"ondisk_log_size":8,"up":9,"acting":9,"num_store_stats":7},{"poolid":1,"num_pg":11,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":1573408,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":869,"ondisk_log_size":869,"up":33,"acting":33,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":56,"seq":240518168782,"num_pgs":76,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2862036,"kb_used_data":2311616,"kb_used_omap":1,"kb_used_meta":550398,"kb_avail":90883116,"statfs":{"total":95995035648,"available":93064310784,"internally_reserved":0,"allocated":2367094784,"data_stored":2366437582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":563608010},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,0,2],"upper_bound":8},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":47,"seq":201863463124,"num_pgs":70,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2986972,"kb_used_data":2406024,"kb_used_omap":1,"kb_used_meta":580926,"kb_avail":90758180,"statfs":{"total":95995035648,"available":92936376320,"internally_reserved":0,"allocated":2463768576,"data_stored":2463101902,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":594868678},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":39,"seq":167503724762,"num_pgs":72,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2609544,"kb_used_data":1770360,"kb_used_omap":1,"kb_used_meta":839166,"kb_avail":91135608,"statfs":{"total":95995035648,"available":93322862592,"internally_reserved":0,"allocated":1812848640,"data_stored":1812198968,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":859306438},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":0,"seq":137438953695,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":26,"seq":111669149925,"num_pgs":76,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2862032,"kb_used_data":2050624,"kb_used_omap":1,"kb_used_meta":811390,"kb_avail":90883120,"statfs":{"total":95995035648,"available":93064314880,"internally_reserved":0,"allocated":2099838976,"data_stored":2099173794,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":830863816},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411564,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1892896,"kb_used_data":1225868,"kb_used_omap":1,"kb_used_meta":667006,"kb_avail":91852256,"statfs":{"total":95995035648,"available":94056710144,"internally_reserved":0,"allocated":1255288832,"data_stored":1254682868,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":683014597},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":307,"seq":1318554959933,"num_pgs":84,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":3145836,"kb_used_data":2855256,"kb_used_omap":7,"kb_used_meta":290552,"kb_avail":90599316,"statfs":{"total":95995035648,"available":92773699584,"internally_reserved":0,"allocated":2923782144,"data_stored":2923100681,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":8079,"internal_metadata":297525361},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738615,"num_pgs":79,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2992040,"kb_used_data":2284760,"kb_used_omap":1,"kb_used_meta":707262,"kb_avail":90753112,"statfs":{"total":95995035648,"available":92931186688,"internally_reserved":0,"allocated":2339594240,"data_stored":2338913058,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":724236741},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":266240,"data_stored":262688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1567,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1613,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":46,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1177,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":436,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":46,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1131,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":2024,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":1678,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3639,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":440,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":81920,"data_stored":2118,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":1584,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":77824,"data_stored":2052,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1474,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":403,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":416,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":403,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":2341994496,"data_stored":2341994496,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":2932736000,"data_stored":2932736000,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":1253638144,"data_stored":1253638144,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":2100297728,"data_stored":2100297728,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":1810956288,"data_stored":1810956288,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":2490236928,"data_stored":2490236928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":2367750144,"data_stored":2367750144,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T08:19:48.755 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid 779349ca-7593-11ef-bceb-c7b262605968 -- ceph osd pool set unique_pool_4 pgp_num 16 2024-09-18T08:19:48.755 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0 2024-09-18T08:19:48.802 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:19:48 smithi007 ceph-mon[29459]: pgmap v853: 158 pgs: 6 active+recovering+undersized+remapped, 152 active+clean; 4.9 GiB data, 19 GiB used, 606 GiB / 626 GiB avail; 639 B/s rd, 41 MiB/s wr, 663 op/s; 10800/240396 objects misplaced (4.493%); 11 MiB/s, 180 objects/s recovering 2024-09-18T08:19:48.802 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:19:48 smithi007 ceph-mon[39249]: pgmap v853: 158 pgs: 6 active+recovering+undersized+remapped, 152 active+clean; 4.9 GiB data, 19 GiB used, 606 GiB / 626 GiB avail; 639 B/s rd, 41 MiB/s wr, 663 op/s; 10800/240396 objects misplaced (4.493%); 11 MiB/s, 180 objects/s recovering 2024-09-18T08:19:48.806 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:19:48.807 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:19:48.807 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0' 2024-09-18T08:19:48.905 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:19:48 smithi050 ceph-mon[32371]: pgmap v853: 158 pgs: 6 active+recovering+undersized+remapped, 152 active+clean; 4.9 GiB data, 19 GiB used, 606 GiB / 626 GiB avail; 639 B/s rd, 41 MiB/s wr, 663 op/s; 10800/240396 objects misplaced (4.493%); 11 MiB/s, 180 objects/s recovering 2024-09-18T08:19:48.908 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2 2024-09-18T08:19:48.982 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:19:48.982 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:19:48.982 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2' 2024-09-18T08:19:49.044 INFO:tasks.radosbench.radosbench.0.smithi007.stdout: 74 16 84373 84357 71.2337 56.5 0.0136729 0.0140318 2024-09-18T08:19:49.084 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4 2024-09-18T08:19:49.111 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:19:49.112 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:19:49.113 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4' 2024-09-18T08:19:49.213 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3 2024-09-18T08:19:49.237 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:19:49.238 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:19:49.239 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3' 2024-09-18T08:19:49.339 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7 2024-09-18T08:19:49.372 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:19:49.372 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:19:49.373 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7' 2024-09-18T08:19:49.474 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1 2024-09-18T08:19:49.520 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:19:49.520 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:19:49.521 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1' 2024-09-18T08:19:49.622 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1 2024-09-18T08:19:49.657 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:19:49.658 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:19:49.658 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1' 2024-09-18T08:19:49.759 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1 2024-09-18T08:19:49.789 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:19:49 smithi007 ceph-mon[39249]: from='mgr.24470 172.21.15.7:0/2594390261' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-18T08:19:49.789 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:19:49 smithi007 ceph-mon[29459]: from='mgr.24470 172.21.15.7:0/2594390261' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-18T08:19:49.820 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:19:49.821 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:19:49.821 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1' 2024-09-18T08:19:49.905 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:19:49 smithi050 ceph-mon[32371]: from='mgr.24470 172.21.15.7:0/2594390261' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-18T08:19:49.906 INFO:journalctl@ceph.iscsi.iscsi.a.smithi050.stdout:Sep 18 08:19:49 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-iscsi-iscsi-a[57782]: debug there is no tcmu-runner data available 2024-09-18T08:19:49.922 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3 2024-09-18T08:19:49.968 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:19:49.969 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:19:49.969 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3' 2024-09-18T08:19:50.044 INFO:tasks.radosbench.radosbench.0.smithi007.stdout: 75 15 85244 85229 71.0105 54.5 0.0204154 0.0140764 2024-09-18T08:19:50.071 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2 2024-09-18T08:19:50.117 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:19:50.118 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:19:50.118 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2' 2024-09-18T08:19:50.219 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5 2024-09-18T08:19:50.245 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:19:50.246 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:19:50.246 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5' 2024-09-18T08:19:50.347 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7 2024-09-18T08:19:50.370 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:19:50.370 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:19:50.370 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7' 2024-09-18T08:19:50.471 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4 2024-09-18T08:19:50.503 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:19:50.503 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:19:50.504 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4' 2024-09-18T08:19:50.604 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0 2024-09-18T08:19:50.651 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:19:50.652 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:19:50.652 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0' 2024-09-18T08:19:50.754 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0 2024-09-18T08:19:50.792 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:19:50.793 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:19:50.793 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0' 2024-09-18T08:19:50.875 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:19:50 smithi007 ceph-mon[29459]: pgmap v854: 158 pgs: 6 active+recovering+undersized+remapped, 152 active+clean; 5.0 GiB data, 20 GiB used, 606 GiB / 626 GiB avail; 1.2 KiB/s rd, 50 MiB/s wr, 796 op/s; 9626/248244 objects misplaced (3.878%); 19 MiB/s, 296 objects/s recovering 2024-09-18T08:19:50.875 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:19:50 smithi007 ceph-mon[29459]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-18T08:19:50.875 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:19:50 smithi007 ceph-mon[39249]: pgmap v854: 158 pgs: 6 active+recovering+undersized+remapped, 152 active+clean; 5.0 GiB data, 20 GiB used, 606 GiB / 626 GiB avail; 1.2 KiB/s rd, 50 MiB/s wr, 796 op/s; 9626/248244 objects misplaced (3.878%); 19 MiB/s, 296 objects/s recovering 2024-09-18T08:19:50.875 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:19:50 smithi007 ceph-mon[39249]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-18T08:19:50.895 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6 2024-09-18T08:19:50.905 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:19:50 smithi050 ceph-mon[32371]: pgmap v854: 158 pgs: 6 active+recovering+undersized+remapped, 152 active+clean; 5.0 GiB data, 20 GiB used, 606 GiB / 626 GiB avail; 1.2 KiB/s rd, 50 MiB/s wr, 796 op/s; 9626/248244 objects misplaced (3.878%); 19 MiB/s, 296 objects/s recovering 2024-09-18T08:19:50.905 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:19:50 smithi050 ceph-mon[32371]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-18T08:19:50.923 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:19:50.923 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:19:50.924 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6' 2024-09-18T08:19:51.024 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0 2024-09-18T08:19:51.044 INFO:tasks.radosbench.radosbench.0.smithi007.stdout: 76 16 85910 85894 70.6228 41.5625 0.036333 0.0141537 2024-09-18T08:19:51.064 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:19:51.064 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:19:51.065 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0' 2024-09-18T08:19:51.166 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0 2024-09-18T08:19:51.202 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:19:51.202 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:19:51.203 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0' 2024-09-18T08:19:51.304 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2 2024-09-18T08:19:51.339 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:19:51.340 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:19:51.340 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2' 2024-09-18T08:19:51.441 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2 2024-09-18T08:19:51.485 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:19:51.485 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:19:51.485 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2' 2024-09-18T08:19:51.586 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7 2024-09-18T08:19:51.616 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:19:51.617 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:19:51.617 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7' 2024-09-18T08:19:51.718 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1 2024-09-18T08:19:51.758 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:19:51.758 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:19:51.759 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1' 2024-09-18T08:19:51.859 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3 2024-09-18T08:19:51.898 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:19:51.899 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:19:51.899 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3' 2024-09-18T08:19:52.000 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1 2024-09-18T08:19:52.040 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:19:52.041 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:19:52.042 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1' 2024-09-18T08:19:52.044 INFO:tasks.radosbench.radosbench.0.smithi007.stdout: 77 16 86732 86716 70.3728 51.375 0.0182568 0.0142044 2024-09-18T08:19:52.079 INFO:teuthology.orchestra.run.smithi007.stdout:ERROR: (22) Invalid argument 2024-09-18T08:19:52.080 INFO:teuthology.orchestra.run.smithi007.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-18T08:19:52.142 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2 2024-09-18T08:19:52.171 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:19:52.172 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:19:52.172 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2' 2024-09-18T08:19:52.273 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7 2024-09-18T08:19:52.306 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:19:52.306 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:19:52.307 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7' 2024-09-18T08:19:52.407 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5 2024-09-18T08:19:52.436 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:19:52.437 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:19:52.437 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5' 2024-09-18T08:19:52.538 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3 2024-09-18T08:19:52.581 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:19:52.582 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:19:52.582 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3' 2024-09-18T08:19:52.683 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3 2024-09-18T08:19:52.721 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:19:52.721 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:19:52.721 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3' 2024-09-18T08:19:52.822 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0 2024-09-18T08:19:52.865 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:19:52.866 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:19:52.867 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0' 2024-09-18T08:19:52.905 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:19:52 smithi050 ceph-mon[32371]: pgmap v855: 158 pgs: 6 active+recovering+undersized+remapped, 152 active+clean; 5.1 GiB data, 20 GiB used, 606 GiB / 626 GiB avail; 639 B/s rd, 51 MiB/s wr, 812 op/s; 9327/252921 objects misplaced (3.688%); 14 MiB/s, 231 objects/s recovering 2024-09-18T08:19:52.967 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6 2024-09-18T08:19:52.980 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:19:52 smithi007 ceph-mon[29459]: pgmap v855: 158 pgs: 6 active+recovering+undersized+remapped, 152 active+clean; 5.1 GiB data, 20 GiB used, 606 GiB / 626 GiB avail; 639 B/s rd, 51 MiB/s wr, 812 op/s; 9327/252921 objects misplaced (3.688%); 14 MiB/s, 231 objects/s recovering 2024-09-18T08:19:52.981 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:19:52 smithi007 ceph-mon[39249]: pgmap v855: 158 pgs: 6 active+recovering+undersized+remapped, 152 active+clean; 5.1 GiB data, 20 GiB used, 606 GiB / 626 GiB avail; 639 B/s rd, 51 MiB/s wr, 812 op/s; 9327/252921 objects misplaced (3.688%); 14 MiB/s, 231 objects/s recovering 2024-09-18T08:19:52.999 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:19:53.000 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:19:53.000 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6' 2024-09-18T08:19:53.045 INFO:tasks.radosbench.radosbench.0.smithi007.stdout: 78 16 87516 87500 70.0988 49 0.00613771 0.0142568 2024-09-18T08:19:53.101 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1 2024-09-18T08:19:53.124 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:19:53.124 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:19:53.125 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1' 2024-09-18T08:19:53.225 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0 2024-09-18T08:19:53.263 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:19:53.263 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:19:53.263 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0' 2024-09-18T08:19:53.364 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3 2024-09-18T08:19:53.400 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:19:53.400 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:19:53.401 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3' 2024-09-18T08:19:53.501 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6 2024-09-18T08:19:53.532 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:19:53.533 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:19:53.533 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6' 2024-09-18T08:19:53.634 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2 2024-09-18T08:19:53.677 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:19:53.678 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:19:53.678 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2' 2024-09-18T08:19:53.779 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6 2024-09-18T08:19:53.813 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:19:53.814 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:19:53.814 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6' 2024-09-18T08:19:53.915 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3 2024-09-18T08:19:53.961 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:19:53.962 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:19:53.962 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3' 2024-09-18T08:19:54.045 INFO:tasks.radosbench.radosbench.0.smithi007.stdout: 79 16 88350 88334 69.8712 52.125 0.0306463 0.014305 2024-09-18T08:19:54.063 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6 2024-09-18T08:19:54.100 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:19:54.101 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:19:54.101 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6' 2024-09-18T08:19:54.202 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7 2024-09-18T08:19:54.239 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:19:54.239 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:19:54.239 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7' 2024-09-18T08:19:54.340 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2 2024-09-18T08:19:54.372 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:19:54.372 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:19:54.373 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2' 2024-09-18T08:19:54.473 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0 2024-09-18T08:19:54.515 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:19:54.515 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:19:54.515 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0' 2024-09-18T08:19:54.616 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2 2024-09-18T08:19:54.679 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:19:54.679 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:19:54.680 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2' 2024-09-18T08:19:54.781 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7 2024-09-18T08:19:54.802 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:19:54.803 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:19:54.803 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7' 2024-09-18T08:19:54.904 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5 2024-09-18T08:19:54.905 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:19:54 smithi050 ceph-mon[32371]: pgmap v856: 158 pgs: 6 active+recovering+undersized+remapped, 152 active+clean; 5.2 GiB data, 21 GiB used, 605 GiB / 626 GiB avail; 1023 B/s rd, 51 MiB/s wr, 810 op/s; 8381/257718 objects misplaced (3.252%); 17 MiB/s, 279 objects/s recovering 2024-09-18T08:19:54.939 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:19:54.940 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:19:54.940 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5' 2024-09-18T08:19:54.979 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:19:54 smithi007 ceph-mon[29459]: pgmap v856: 158 pgs: 6 active+recovering+undersized+remapped, 152 active+clean; 5.2 GiB data, 21 GiB used, 605 GiB / 626 GiB avail; 1023 B/s rd, 51 MiB/s wr, 810 op/s; 8381/257718 objects misplaced (3.252%); 17 MiB/s, 279 objects/s recovering 2024-09-18T08:19:54.980 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:19:54 smithi007 ceph-mon[39249]: pgmap v856: 158 pgs: 6 active+recovering+undersized+remapped, 152 active+clean; 5.2 GiB data, 21 GiB used, 605 GiB / 626 GiB avail; 1023 B/s rd, 51 MiB/s wr, 810 op/s; 8381/257718 objects misplaced (3.252%); 17 MiB/s, 279 objects/s recovering 2024-09-18T08:19:55.041 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2 2024-09-18T08:19:55.044 INFO:tasks.radosbench.radosbench.0.smithi007.stdout:2024-09-18T08:19:55.044787+0000 min lat: 0.00108817 max lat: 2.21973 avg lat: 0.0143344 2024-09-18T08:19:55.045 INFO:tasks.radosbench.radosbench.0.smithi007.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-09-18T08:19:55.045 INFO:tasks.radosbench.radosbench.0.smithi007.stdout: 80 16 89295 89279 69.7361 59.0625 0.00920327 0.0143344 2024-09-18T08:19:55.070 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:19:55.071 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:19:55.072 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2' 2024-09-18T08:19:55.172 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4 2024-09-18T08:19:55.205 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:19:55.206 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:19:55.206 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4' 2024-09-18T08:19:55.307 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3 2024-09-18T08:19:55.333 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:19:55.334 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:19:55.334 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3' 2024-09-18T08:19:55.435 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7 2024-09-18T08:19:55.468 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:19:55.471 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:19:55.472 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7' 2024-09-18T08:19:55.572 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2 2024-09-18T08:19:55.594 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:19:55.595 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:19:55.596 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2' 2024-09-18T08:19:55.696 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4 2024-09-18T08:19:55.725 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:19:55.726 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:19:55.726 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4' 2024-09-18T08:19:55.827 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7 2024-09-18T08:19:55.865 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:19:55.866 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:19:55.866 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7' 2024-09-18T08:19:55.967 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4 2024-09-18T08:19:55.999 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:19:55.999 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:19:56.000 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4' 2024-09-18T08:19:56.045 INFO:tasks.radosbench.radosbench.0.smithi007.stdout: 81 16 90218 90202 69.5873 57.6875 0.0112521 0.014365 2024-09-18T08:19:56.101 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4 2024-09-18T08:19:56.131 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:19:56.132 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:19:56.132 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4' 2024-09-18T08:19:56.233 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0 2024-09-18T08:19:56.273 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:19:56.273 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:19:56.274 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0' 2024-09-18T08:19:56.375 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2 2024-09-18T08:19:56.397 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:19:56.398 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:19:56.398 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2' 2024-09-18T08:19:56.499 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3 2024-09-18T08:19:56.541 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:19:56.542 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:19:56.542 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3' 2024-09-18T08:19:56.643 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5 2024-09-18T08:19:56.683 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:19:56.684 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:19:56.684 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5' 2024-09-18T08:19:56.785 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6 2024-09-18T08:19:56.814 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:19:56.815 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:19:56.815 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6' 2024-09-18T08:19:56.905 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:19:56 smithi050 ceph-mon[32371]: pgmap v857: 158 pgs: 6 active+recovering+undersized+remapped, 152 active+clean; 5.4 GiB data, 21 GiB used, 605 GiB / 626 GiB avail; 927 B/s rd, 55 MiB/s wr, 877 op/s; 7903/265053 objects misplaced (2.982%); 18 MiB/s, 283 objects/s recovering 2024-09-18T08:19:56.917 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1 2024-09-18T08:19:56.945 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:19:56.946 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:19:56.946 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1' 2024-09-18T08:19:56.979 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:19:56 smithi007 ceph-mon[29459]: pgmap v857: 158 pgs: 6 active+recovering+undersized+remapped, 152 active+clean; 5.4 GiB data, 21 GiB used, 605 GiB / 626 GiB avail; 927 B/s rd, 55 MiB/s wr, 877 op/s; 7903/265053 objects misplaced (2.982%); 18 MiB/s, 283 objects/s recovering 2024-09-18T08:19:56.980 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Sep 18 08:19:56 smithi007 ceph-779349ca-7593-11ef-bceb-c7b262605968-mgr-y[29688]: ::ffff:172.21.15.50 - - [18/Sep/2024:08:19:56] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-09-18T08:19:56.980 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:19:56 smithi007 ceph-mon[39249]: pgmap v857: 158 pgs: 6 active+recovering+undersized+remapped, 152 active+clean; 5.4 GiB data, 21 GiB used, 605 GiB / 626 GiB avail; 927 B/s rd, 55 MiB/s wr, 877 op/s; 7903/265053 objects misplaced (2.982%); 18 MiB/s, 283 objects/s recovering 2024-09-18T08:19:57.045 INFO:tasks.radosbench.radosbench.0.smithi007.stdout: 82 15 91091 91076 69.4048 54.625 0.0179219 0.0144031 2024-09-18T08:19:57.046 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7 2024-09-18T08:19:57.079 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:19:57.080 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:19:57.080 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7' 2024-09-18T08:19:57.181 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5 2024-09-18T08:19:57.211 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:19:57.212 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:19:57.212 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5' 2024-09-18T08:19:57.313 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3 2024-09-18T08:19:57.352 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:19:57.353 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:19:57.354 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3' 2024-09-18T08:19:57.455 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2 2024-09-18T08:19:57.490 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:19:57.490 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:19:57.491 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2' 2024-09-18T08:19:57.592 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5 2024-09-18T08:19:57.632 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:19:57.632 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:19:57.634 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5' 2024-09-18T08:19:57.735 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1 2024-09-18T08:19:57.776 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:19:57.776 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:19:57.777 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1' 2024-09-18T08:19:57.878 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5 2024-09-18T08:19:57.909 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:19:57.910 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:19:57.910 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5' 2024-09-18T08:19:58.011 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0 2024-09-18T08:19:58.045 INFO:tasks.radosbench.radosbench.0.smithi007.stdout: 83 16 92128 92112 69.3481 64.75 0.0171096 0.0144145 2024-09-18T08:19:58.061 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:19:58.062 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:19:58.063 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0' 2024-09-18T08:19:58.163 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3 2024-09-18T08:19:58.206 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:19:58.207 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:19:58.207 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3' 2024-09-18T08:19:58.308 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4 2024-09-18T08:19:58.335 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:19:58.336 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:19:58.336 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4' 2024-09-18T08:19:58.437 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1 2024-09-18T08:19:58.483 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:19:58.484 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:19:58.484 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1' 2024-09-18T08:19:58.584 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7 2024-09-18T08:19:58.611 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:19:58.612 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:19:58.612 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7' 2024-09-18T08:19:58.713 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5 2024-09-18T08:19:58.729 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:19:58 smithi007 ceph-mon[29459]: pgmap v858: 158 pgs: 6 active+recovering+undersized+remapped, 152 active+clean; 5.4 GiB data, 21 GiB used, 605 GiB / 626 GiB avail; 853 B/s rd, 43 MiB/s wr, 685 op/s; 7903/265053 objects misplaced (2.982%); 15 MiB/s, 241 objects/s recovering 2024-09-18T08:19:58.730 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:19:58 smithi007 ceph-mon[39249]: pgmap v858: 158 pgs: 6 active+recovering+undersized+remapped, 152 active+clean; 5.4 GiB data, 21 GiB used, 605 GiB / 626 GiB avail; 853 B/s rd, 43 MiB/s wr, 685 op/s; 7903/265053 objects misplaced (2.982%); 15 MiB/s, 241 objects/s recovering 2024-09-18T08:19:58.753 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:19:58.754 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:19:58.754 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5' 2024-09-18T08:19:58.855 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5 2024-09-18T08:19:58.887 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:19:58.887 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:19:58.888 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5' 2024-09-18T08:19:58.904 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:19:58 smithi050 ceph-mon[32371]: pgmap v858: 158 pgs: 6 active+recovering+undersized+remapped, 152 active+clean; 5.4 GiB data, 21 GiB used, 605 GiB / 626 GiB avail; 853 B/s rd, 43 MiB/s wr, 685 op/s; 7903/265053 objects misplaced (2.982%); 15 MiB/s, 241 objects/s recovering 2024-09-18T08:19:58.989 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0 2024-09-18T08:19:59.031 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:19:59.031 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:19:59.032 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0' 2024-09-18T08:19:59.046 INFO:tasks.radosbench.radosbench.0.smithi007.stdout: 84 15 93223 93208 69.338 68.5 0.00431664 0.0144171 2024-09-18T08:19:59.133 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7 2024-09-18T08:19:59.166 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:19:59.167 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:19:59.167 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7' 2024-09-18T08:19:59.268 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2 2024-09-18T08:19:59.307 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:19:59.308 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:19:59.309 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2' 2024-09-18T08:19:59.410 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7 2024-09-18T08:19:59.435 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:19:59.436 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:19:59.436 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7' 2024-09-18T08:19:59.538 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7 2024-09-18T08:19:59.569 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:19:59.569 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:19:59.570 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7' 2024-09-18T08:19:59.671 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2 2024-09-18T08:19:59.694 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:19:59.695 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:19:59.695 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2' 2024-09-18T08:19:59.796 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0 2024-09-18T08:19:59.838 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:19:59.839 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:19:59.839 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0' 2024-09-18T08:19:59.940 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2 2024-09-18T08:19:59.977 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:19:59.980 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:19:59.981 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2' 2024-09-18T08:20:00.046 INFO:tasks.radosbench.radosbench.0.smithi007.stdout: 85 16 94173 94157 69.22 59.3125 0.0135056 0.0144415 2024-09-18T08:20:00.082 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2 2024-09-18T08:20:00.124 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:20:00.124 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:20:00.125 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2' 2024-09-18T08:20:00.155 INFO:journalctl@ceph.iscsi.iscsi.a.smithi050.stdout:Sep 18 08:19:59 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-iscsi-iscsi-a[57782]: debug there is no tcmu-runner data available 2024-09-18T08:20:00.226 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0 2024-09-18T08:20:00.265 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:20:00.266 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:20:00.267 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0' 2024-09-18T08:20:00.368 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4 2024-09-18T08:20:00.403 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:20:00.404 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:20:00.404 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4' 2024-09-18T08:20:00.505 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7 2024-09-18T08:20:00.536 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:20:00.536 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:20:00.537 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7' 2024-09-18T08:20:00.638 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0 2024-09-18T08:20:00.661 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:20:00.662 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:20:00.662 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0' 2024-09-18T08:20:00.763 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0 2024-09-18T08:20:00.805 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:20:00.806 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:20:00.806 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0' 2024-09-18T08:20:00.896 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:20:00 smithi007 ceph-mon[29459]: pgmap v859: 158 pgs: 6 active+recovering+undersized+remapped, 152 active+clean; 5.6 GiB data, 22 GiB used, 604 GiB / 626 GiB avail; 1.2 KiB/s rd, 59 MiB/s wr, 937 op/s; 6626/274137 objects misplaced (2.417%); 22 MiB/s, 347 objects/s recovering 2024-09-18T08:20:00.896 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:20:00 smithi007 ceph-mon[29459]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-18T08:20:00.896 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:20:00 smithi007 ceph-mon[29459]: Health detail: HEALTH_WARN noscrub flag(s) set 2024-09-18T08:20:00.896 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:20:00 smithi007 ceph-mon[29459]: [WRN] OSDMAP_FLAGS: noscrub flag(s) set 2024-09-18T08:20:00.896 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:20:00 smithi007 ceph-mon[39249]: pgmap v859: 158 pgs: 6 active+recovering+undersized+remapped, 152 active+clean; 5.6 GiB data, 22 GiB used, 604 GiB / 626 GiB avail; 1.2 KiB/s rd, 59 MiB/s wr, 937 op/s; 6626/274137 objects misplaced (2.417%); 22 MiB/s, 347 objects/s recovering 2024-09-18T08:20:00.896 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:20:00 smithi007 ceph-mon[39249]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-18T08:20:00.897 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:20:00 smithi007 ceph-mon[39249]: Health detail: HEALTH_WARN noscrub flag(s) set 2024-09-18T08:20:00.897 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:20:00 smithi007 ceph-mon[39249]: [WRN] OSDMAP_FLAGS: noscrub flag(s) set 2024-09-18T08:20:00.905 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:20:00 smithi050 ceph-mon[32371]: pgmap v859: 158 pgs: 6 active+recovering+undersized+remapped, 152 active+clean; 5.6 GiB data, 22 GiB used, 604 GiB / 626 GiB avail; 1.2 KiB/s rd, 59 MiB/s wr, 937 op/s; 6626/274137 objects misplaced (2.417%); 22 MiB/s, 347 objects/s recovering 2024-09-18T08:20:00.905 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:20:00 smithi050 ceph-mon[32371]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-18T08:20:00.905 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:20:00 smithi050 ceph-mon[32371]: Health detail: HEALTH_WARN noscrub flag(s) set 2024-09-18T08:20:00.905 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:20:00 smithi050 ceph-mon[32371]: [WRN] OSDMAP_FLAGS: noscrub flag(s) set 2024-09-18T08:20:00.907 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5 2024-09-18T08:20:00.939 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:20:00.940 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:20:00.941 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5' 2024-09-18T08:20:01.041 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6 2024-09-18T08:20:01.046 INFO:tasks.radosbench.radosbench.0.smithi007.stdout: 86 16 94980 94964 69.0016 50.4375 0.00381864 0.0144787 2024-09-18T08:20:01.065 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:20:01.065 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:20:01.066 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6' 2024-09-18T08:20:01.166 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5 2024-09-18T08:20:01.199 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:20:01.200 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:20:01.200 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5' 2024-09-18T08:20:01.301 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2 2024-09-18T08:20:01.342 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:20:01.343 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:20:01.343 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2' 2024-09-18T08:20:01.444 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5 2024-09-18T08:20:01.479 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:20:01.479 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:20:01.480 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5' 2024-09-18T08:20:01.581 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1 2024-09-18T08:20:01.596 INFO:teuthology.orchestra.run.smithi007.stderr:nodeep-scrub is set 2024-09-18T08:20:01.626 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:20:01.627 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:20:01.627 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1' 2024-09-18T08:20:01.728 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1 2024-09-18T08:20:01.751 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:20:01.752 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:20:01.752 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1' 2024-09-18T08:20:01.853 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0 2024-09-18T08:20:01.896 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:20:01.897 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:20:01.897 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0' 2024-09-18T08:20:01.905 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:20:01 smithi050 ceph-mon[32371]: from='client.? 172.21.15.7:0/4011797645' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-18T08:20:01.980 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:20:01 smithi007 ceph-mon[39249]: from='client.? 172.21.15.7:0/4011797645' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-18T08:20:01.980 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:20:01 smithi007 ceph-mon[29459]: from='client.? 172.21.15.7:0/4011797645' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-18T08:20:01.998 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4 2024-09-18T08:20:02.029 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:20:02.030 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:20:02.030 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4' 2024-09-18T08:20:02.046 INFO:tasks.radosbench.radosbench.0.smithi007.stdout: 87 16 95927 95911 68.8887 59.1875 0.0069468 0.0145064 2024-09-18T08:20:02.130 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5 2024-09-18T08:20:02.165 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:20:02.166 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:20:02.167 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5' 2024-09-18T08:20:02.268 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2 2024-09-18T08:20:02.295 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:20:02.295 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:20:02.296 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2' 2024-09-18T08:20:02.396 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6 2024-09-18T08:20:02.435 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:20:02.436 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:20:02.436 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6' 2024-09-18T08:20:02.537 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7 2024-09-18T08:20:02.569 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:20:02.569 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:20:02.569 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7' 2024-09-18T08:20:02.670 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4 2024-09-18T08:20:02.701 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:20:02.701 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:20:02.702 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4' 2024-09-18T08:20:02.803 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3 2024-09-18T08:20:02.843 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:20:02.844 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:20:02.844 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3' 2024-09-18T08:20:02.905 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:20:02 smithi050 ceph-mon[32371]: pgmap v860: 158 pgs: 6 active+recovering+undersized+remapped, 152 active+clean; 5.7 GiB data, 22 GiB used, 604 GiB / 626 GiB avail; 852 B/s rd, 54 MiB/s wr, 867 op/s; 6270/279474 objects misplaced (2.244%); 17 MiB/s, 279 objects/s recovering 2024-09-18T08:20:02.905 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:20:02 smithi050 ceph-mon[32371]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-18T08:20:02.905 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:20:02 smithi050 ceph-mon[32371]: from='client.? 172.21.15.7:0/4011797645' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-18T08:20:02.905 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:20:02 smithi050 ceph-mon[32371]: osdmap e429: 8 total, 8 up, 7 in 2024-09-18T08:20:02.945 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7 2024-09-18T08:20:02.978 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:20:02.978 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:20:02.978 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7' 2024-09-18T08:20:02.979 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:20:02 smithi007 ceph-mon[39249]: pgmap v860: 158 pgs: 6 active+recovering+undersized+remapped, 152 active+clean; 5.7 GiB data, 22 GiB used, 604 GiB / 626 GiB avail; 852 B/s rd, 54 MiB/s wr, 867 op/s; 6270/279474 objects misplaced (2.244%); 17 MiB/s, 279 objects/s recovering 2024-09-18T08:20:02.979 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:20:02 smithi007 ceph-mon[39249]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-18T08:20:02.980 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:20:02 smithi007 ceph-mon[39249]: from='client.? 172.21.15.7:0/4011797645' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-18T08:20:02.980 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:20:02 smithi007 ceph-mon[39249]: osdmap e429: 8 total, 8 up, 7 in 2024-09-18T08:20:02.980 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:20:02 smithi007 ceph-mon[29459]: pgmap v860: 158 pgs: 6 active+recovering+undersized+remapped, 152 active+clean; 5.7 GiB data, 22 GiB used, 604 GiB / 626 GiB avail; 852 B/s rd, 54 MiB/s wr, 867 op/s; 6270/279474 objects misplaced (2.244%); 17 MiB/s, 279 objects/s recovering 2024-09-18T08:20:02.980 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:20:02 smithi007 ceph-mon[29459]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-18T08:20:02.980 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:20:02 smithi007 ceph-mon[29459]: from='client.? 172.21.15.7:0/4011797645' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-18T08:20:02.980 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:20:02 smithi007 ceph-mon[29459]: osdmap e429: 8 total, 8 up, 7 in 2024-09-18T08:20:03.046 INFO:tasks.radosbench.radosbench.0.smithi007.stdout: 88 16 97001 96985 68.8686 67.125 0.0100531 0.014515 2024-09-18T08:20:03.080 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4 2024-09-18T08:20:03.115 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:20:03.116 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:20:03.116 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4' 2024-09-18T08:20:03.121 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/779349ca-7593-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T08:20:03.170 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid 779349ca-7593-11ef-bceb-c7b262605968 -- ceph daemon osd.0 dump_blocked_ops 2024-09-18T08:20:03.217 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7 2024-09-18T08:20:03.245 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:20:03.246 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:20:03.246 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7' 2024-09-18T08:20:03.347 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7 2024-09-18T08:20:03.379 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:20:03.380 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:20:03.380 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7' 2024-09-18T08:20:03.481 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4 2024-09-18T08:20:03.512 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:20:03.513 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:20:03.513 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4' 2024-09-18T08:20:03.614 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0 2024-09-18T08:20:03.674 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:20:03.675 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:20:03.675 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0' 2024-09-18T08:20:03.776 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4 2024-09-18T08:20:03.810 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:20:03.811 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:20:03.811 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4' 2024-09-18T08:20:03.908 INFO:teuthology.orchestra.run.smithi007.stdout:osd.0: {} 2024-09-18T08:20:03.908 INFO:teuthology.orchestra.run.smithi007.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-18T08:20:03.911 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5 2024-09-18T08:20:03.940 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:20:03.941 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:20:03.942 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5' 2024-09-18T08:20:03.944 INFO:teuthology.orchestra.run.smithi007.stdout:osd.1: {} 2024-09-18T08:20:03.944 INFO:teuthology.orchestra.run.smithi007.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-18T08:20:03.980 INFO:teuthology.orchestra.run.smithi007.stdout:osd.2: {} 2024-09-18T08:20:03.981 INFO:teuthology.orchestra.run.smithi007.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-18T08:20:04.017 INFO:teuthology.orchestra.run.smithi007.stdout:osd.3: {} 2024-09-18T08:20:04.017 INFO:teuthology.orchestra.run.smithi007.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-18T08:20:04.043 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0 2024-09-18T08:20:04.046 INFO:tasks.radosbench.radosbench.0.smithi007.stdout: 89 15 97981 97966 68.7836 61.3125 0.0173935 0.0145328 2024-09-18T08:20:04.051 INFO:teuthology.orchestra.run.smithi007.stdout:osd.4: {} 2024-09-18T08:20:04.051 INFO:teuthology.orchestra.run.smithi007.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-18T08:20:04.093 INFO:teuthology.orchestra.run.smithi007.stdout:osd.5: {} 2024-09-18T08:20:04.094 INFO:teuthology.orchestra.run.smithi007.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-18T08:20:04.095 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:20:04.096 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:20:04.096 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0' 2024-09-18T08:20:04.129 INFO:teuthology.orchestra.run.smithi007.stdout:osd.6: {} 2024-09-18T08:20:04.130 INFO:teuthology.orchestra.run.smithi007.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-18T08:20:04.161 INFO:teuthology.orchestra.run.smithi007.stdout:osd.7: {} 2024-09-18T08:20:04.162 INFO:teuthology.orchestra.run.smithi007.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-18T08:20:04.197 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6 2024-09-18T08:20:04.229 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:20:04.229 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:20:04.230 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6' 2024-09-18T08:20:04.330 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0 2024-09-18T08:20:04.385 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:20:04.385 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:20:04.386 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0' 2024-09-18T08:20:04.487 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3 2024-09-18T08:20:04.534 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:20:04.535 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:20:04.536 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3' 2024-09-18T08:20:04.637 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2 2024-09-18T08:20:04.675 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:20:04.676 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:20:04.676 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2' 2024-09-18T08:20:04.777 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3 2024-09-18T08:20:04.813 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:20:04.814 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:20:04.814 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3' 2024-09-18T08:20:04.905 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:20:04 smithi050 ceph-mon[32371]: pgmap v862: 158 pgs: 6 active+recovering+undersized+remapped, 152 active+clean; 5.8 GiB data, 23 GiB used, 603 GiB / 626 GiB avail; 1023 B/s rd, 57 MiB/s wr, 908 op/s; 4953/284958 objects misplaced (1.738%); 21 MiB/s, 342 objects/s recovering 2024-09-18T08:20:04.905 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:20:04 smithi050 ceph-mon[32371]: from='client.? 172.21.15.7:0/912750785' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-18T08:20:04.905 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:20:04 smithi050 ceph-mon[32371]: from='mgr.24470 172.21.15.7:0/2594390261' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-18T08:20:04.905 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:20:04 smithi050 ceph-mon[32371]: osdmap e430: 8 total, 8 up, 7 in 2024-09-18T08:20:04.914 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1 2024-09-18T08:20:04.942 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:20:04.943 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:20:04.943 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1' 2024-09-18T08:20:04.980 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:20:04 smithi007 ceph-mon[39249]: pgmap v862: 158 pgs: 6 active+recovering+undersized+remapped, 152 active+clean; 5.8 GiB data, 23 GiB used, 603 GiB / 626 GiB avail; 1023 B/s rd, 57 MiB/s wr, 908 op/s; 4953/284958 objects misplaced (1.738%); 21 MiB/s, 342 objects/s recovering 2024-09-18T08:20:04.980 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:20:04 smithi007 ceph-mon[39249]: from='client.? 172.21.15.7:0/912750785' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-18T08:20:04.980 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:20:04 smithi007 ceph-mon[39249]: from='mgr.24470 172.21.15.7:0/2594390261' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-18T08:20:04.980 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:20:04 smithi007 ceph-mon[39249]: osdmap e430: 8 total, 8 up, 7 in 2024-09-18T08:20:04.980 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:20:04 smithi007 ceph-mon[29459]: pgmap v862: 158 pgs: 6 active+recovering+undersized+remapped, 152 active+clean; 5.8 GiB data, 23 GiB used, 603 GiB / 626 GiB avail; 1023 B/s rd, 57 MiB/s wr, 908 op/s; 4953/284958 objects misplaced (1.738%); 21 MiB/s, 342 objects/s recovering 2024-09-18T08:20:04.980 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:20:04 smithi007 ceph-mon[29459]: from='client.? 172.21.15.7:0/912750785' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-18T08:20:04.980 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:20:04 smithi007 ceph-mon[29459]: from='mgr.24470 172.21.15.7:0/2594390261' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-18T08:20:04.980 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:20:04 smithi007 ceph-mon[29459]: osdmap e430: 8 total, 8 up, 7 in 2024-09-18T08:20:05.044 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0 2024-09-18T08:20:05.047 INFO:tasks.radosbench.radosbench.0.smithi007.stdout: 90 16 98681 98665 68.5048 43.6875 0.00350492 0.0145285 2024-09-18T08:20:05.077 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:20:05.078 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:20:05.078 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0' 2024-09-18T08:20:05.116 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/779349ca-7593-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T08:20:05.179 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3 2024-09-18T08:20:05.205 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:20:05.206 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:20:05.207 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3' 2024-09-18T08:20:05.307 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6 2024-09-18T08:20:05.332 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:20:05.332 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:20:05.333 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6' 2024-09-18T08:20:05.434 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4 2024-09-18T08:20:05.458 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:20:05.458 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:20:05.459 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4' 2024-09-18T08:20:05.482 INFO:teuthology.orchestra.run.smithi007.stderr:set pool 11 pgp_num to 16 2024-09-18T08:20:05.559 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2 2024-09-18T08:20:05.592 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:20:05.593 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:20:05.593 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2' 2024-09-18T08:20:05.660 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid 779349ca-7593-11ef-bceb-c7b262605968 -- ceph osd unset noscrub 2024-09-18T08:20:05.689 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:20:05 smithi007 ceph-mon[29459]: from='client.? 172.21.15.7:0/126370075' entity='client.admin' cmd={"prefix": "osd pool set", "pool": "unique_pool_4", "var": "pgp_num", "val": "16"} : dispatch 2024-09-18T08:20:05.689 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:20:05 smithi007 ceph-mon[29459]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool set", "pool": "unique_pool_4", "var": "pgp_num", "val": "16"} : dispatch 2024-09-18T08:20:05.689 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:20:05 smithi007 ceph-mon[29459]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_4", "var": "pgp_num", "val": "16"}]': finished 2024-09-18T08:20:05.689 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:20:05 smithi007 ceph-mon[29459]: osdmap e431: 8 total, 8 up, 7 in 2024-09-18T08:20:05.690 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:20:05 smithi007 ceph-mon[39249]: from='client.? 172.21.15.7:0/126370075' entity='client.admin' cmd={"prefix": "osd pool set", "pool": "unique_pool_4", "var": "pgp_num", "val": "16"} : dispatch 2024-09-18T08:20:05.690 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:20:05 smithi007 ceph-mon[39249]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool set", "pool": "unique_pool_4", "var": "pgp_num", "val": "16"} : dispatch 2024-09-18T08:20:05.690 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:20:05 smithi007 ceph-mon[39249]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_4", "var": "pgp_num", "val": "16"}]': finished 2024-09-18T08:20:05.690 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:20:05 smithi007 ceph-mon[39249]: osdmap e431: 8 total, 8 up, 7 in 2024-09-18T08:20:05.694 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5 2024-09-18T08:20:05.717 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:20:05.718 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:20:05.718 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5' 2024-09-18T08:20:05.819 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3 2024-09-18T08:20:05.846 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:20:05.875 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:20:05.875 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3' 2024-09-18T08:20:05.905 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:20:05 smithi050 ceph-mon[32371]: from='client.? 172.21.15.7:0/126370075' entity='client.admin' cmd={"prefix": "osd pool set", "pool": "unique_pool_4", "var": "pgp_num", "val": "16"} : dispatch 2024-09-18T08:20:05.905 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:20:05 smithi050 ceph-mon[32371]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool set", "pool": "unique_pool_4", "var": "pgp_num", "val": "16"} : dispatch 2024-09-18T08:20:05.905 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:20:05 smithi050 ceph-mon[32371]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_4", "var": "pgp_num", "val": "16"}]': finished 2024-09-18T08:20:05.905 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:20:05 smithi050 ceph-mon[32371]: osdmap e431: 8 total, 8 up, 7 in 2024-09-18T08:20:05.976 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2 2024-09-18T08:20:06.005 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:20:06.006 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:20:06.006 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2' 2024-09-18T08:20:06.046 INFO:tasks.radosbench.radosbench.0.smithi007.stdout:Total time run: 90.5249 2024-09-18T08:20:06.047 INFO:tasks.radosbench.radosbench.0.smithi007.stdout:Total writes made: 98681 2024-09-18T08:20:06.047 INFO:tasks.radosbench.radosbench.0.smithi007.stdout:Write size: 65536 2024-09-18T08:20:06.047 INFO:tasks.radosbench.radosbench.0.smithi007.stdout:Object size: 65536 2024-09-18T08:20:06.047 INFO:tasks.radosbench.radosbench.0.smithi007.stdout:Bandwidth (MB/sec): 68.1311 2024-09-18T08:20:06.047 INFO:tasks.radosbench.radosbench.0.smithi007.stdout:Stddev Bandwidth: 32.6444 2024-09-18T08:20:06.047 INFO:tasks.radosbench.radosbench.0.smithi007.stdout:Max bandwidth (MB/sec): 162.562 2024-09-18T08:20:06.047 INFO:tasks.radosbench.radosbench.0.smithi007.stdout:Min bandwidth (MB/sec): 0 2024-09-18T08:20:06.047 INFO:tasks.radosbench.radosbench.0.smithi007.stdout:Average IOPS: 1090 2024-09-18T08:20:06.047 INFO:tasks.radosbench.radosbench.0.smithi007.stdout:Stddev IOPS: 522.31 2024-09-18T08:20:06.047 INFO:tasks.radosbench.radosbench.0.smithi007.stdout:Max IOPS: 2601 2024-09-18T08:20:06.047 INFO:tasks.radosbench.radosbench.0.smithi007.stdout:Min IOPS: 0 2024-09-18T08:20:06.047 INFO:tasks.radosbench.radosbench.0.smithi007.stdout:Average Latency(s): 0.0146708 2024-09-18T08:20:06.047 INFO:tasks.radosbench.radosbench.0.smithi007.stdout:Stddev Latency(s): 0.0337737 2024-09-18T08:20:06.048 INFO:tasks.radosbench.radosbench.0.smithi007.stdout:Max latency(s): 2.21973 2024-09-18T08:20:06.048 INFO:tasks.radosbench.radosbench.0.smithi007.stdout:Min latency(s): 0.00108817 2024-09-18T08:20:06.107 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7 2024-09-18T08:20:06.130 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:20:06.131 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:20:06.131 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7' 2024-09-18T08:20:06.232 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0 2024-09-18T08:20:06.257 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:20:06.257 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:20:06.257 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0' 2024-09-18T08:20:06.358 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6 2024-09-18T08:20:06.382 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:20:06.383 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:20:06.383 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6' 2024-09-18T08:20:06.484 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6 2024-09-18T08:20:06.499 INFO:tasks.radosbench.radosbench.0.smithi007.stdout:Cleaning up (deleting benchmark objects) 2024-09-18T08:20:06.515 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:20:06.516 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:20:06.517 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6' 2024-09-18T08:20:06.619 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2 2024-09-18T08:20:06.660 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:20:06.661 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:20:06.661 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2' 2024-09-18T08:20:06.756 INFO:teuthology.orchestra.run.smithi007.stdout:{ 2024-09-18T08:20:06.756 INFO:teuthology.orchestra.run.smithi007.stdout: "ops": [], 2024-09-18T08:20:06.756 INFO:teuthology.orchestra.run.smithi007.stdout: "complaint_time": 30, 2024-09-18T08:20:06.757 INFO:teuthology.orchestra.run.smithi007.stdout: "num_blocked_ops": 0 2024-09-18T08:20:06.757 INFO:teuthology.orchestra.run.smithi007.stdout:} 2024-09-18T08:20:06.762 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4 2024-09-18T08:20:06.793 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:20:06.794 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:20:06.794 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4' 2024-09-18T08:20:06.895 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5 2024-09-18T08:20:06.906 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:20:06 smithi050 ceph-mon[32371]: pgmap v865: 158 pgs: 6 active+recovering+undersized+remapped, 152 active+clean; 6.0 GiB data, 23 GiB used, 603 GiB / 626 GiB avail; 853 B/s rd, 69 MiB/s wr, 1.10k op/s; 4293/293907 objects misplaced (1.461%); 24 MiB/s, 388 objects/s recovering 2024-09-18T08:20:06.906 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:20:06 smithi050 ceph-mon[32371]: osdmap e432: 8 total, 8 up, 7 in 2024-09-18T08:20:06.926 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:20:06.927 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:20:06.927 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5' 2024-09-18T08:20:06.979 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Sep 18 08:20:06 smithi007 ceph-779349ca-7593-11ef-bceb-c7b262605968-mgr-y[29688]: ::ffff:172.21.15.50 - - [18/Sep/2024:08:20:06] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-09-18T08:20:06.980 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:20:06 smithi007 ceph-mon[29459]: pgmap v865: 158 pgs: 6 active+recovering+undersized+remapped, 152 active+clean; 6.0 GiB data, 23 GiB used, 603 GiB / 626 GiB avail; 853 B/s rd, 69 MiB/s wr, 1.10k op/s; 4293/293907 objects misplaced (1.461%); 24 MiB/s, 388 objects/s recovering 2024-09-18T08:20:06.980 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:20:06 smithi007 ceph-mon[29459]: osdmap e432: 8 total, 8 up, 7 in 2024-09-18T08:20:06.980 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:20:06 smithi007 ceph-mon[39249]: pgmap v865: 158 pgs: 6 active+recovering+undersized+remapped, 152 active+clean; 6.0 GiB data, 23 GiB used, 603 GiB / 626 GiB avail; 853 B/s rd, 69 MiB/s wr, 1.10k op/s; 4293/293907 objects misplaced (1.461%); 24 MiB/s, 388 objects/s recovering 2024-09-18T08:20:06.980 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:20:06 smithi007 ceph-mon[39249]: osdmap e432: 8 total, 8 up, 7 in 2024-09-18T08:20:07.028 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7 2024-09-18T08:20:07.056 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:20:07.057 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:20:07.057 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7' 2024-09-18T08:20:07.083 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid 779349ca-7593-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-18T08:20:07.157 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2 2024-09-18T08:20:07.206 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:20:07.207 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:20:07.207 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2' 2024-09-18T08:20:07.308 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1 2024-09-18T08:20:07.355 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:20:07.357 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:20:07.358 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1' 2024-09-18T08:20:07.458 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4 2024-09-18T08:20:07.491 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:20:07.492 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:20:07.492 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4' 2024-09-18T08:20:07.593 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6 2024-09-18T08:20:07.621 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:20:07.622 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:20:07.622 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6' 2024-09-18T08:20:07.723 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5 2024-09-18T08:20:07.756 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:20:07.757 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:20:07.757 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5' 2024-09-18T08:20:07.858 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6 2024-09-18T08:20:07.891 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:20:07.891 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:20:07.892 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6' 2024-09-18T08:20:07.903 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:20:07 smithi007 ceph-mon[29459]: Health check failed: Degraded data redundancy: 6 pgs undersized (PG_DEGRADED) 2024-09-18T08:20:07.904 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:20:07 smithi007 ceph-mon[39249]: Health check failed: Degraded data redundancy: 6 pgs undersized (PG_DEGRADED) 2024-09-18T08:20:07.904 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:20:07 smithi050 ceph-mon[32371]: Health check failed: Degraded data redundancy: 6 pgs undersized (PG_DEGRADED) 2024-09-18T08:20:07.963 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid 779349ca-7593-11ef-bceb-c7b262605968 -- ceph daemon osd.0 dump_historic_ops 2024-09-18T08:20:07.993 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6 2024-09-18T08:20:08.026 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:20:08.027 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:20:08.027 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6' 2024-09-18T08:20:08.128 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3 2024-09-18T08:20:08.190 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:20:08.191 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:20:08.191 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3' 2024-09-18T08:20:08.292 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5 2024-09-18T08:20:08.324 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:20:08.325 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:20:08.325 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5' 2024-09-18T08:20:08.426 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5 2024-09-18T08:20:08.460 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:20:08.460 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:20:08.461 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5' 2024-09-18T08:20:08.561 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2 2024-09-18T08:20:08.573 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/779349ca-7593-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T08:20:08.599 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/779349ca-7593-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T08:20:08.626 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:20:08.626 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:20:08.627 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2' 2024-09-18T08:20:08.720 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:20:08 smithi007 ceph-mon[29459]: pgmap v867: 158 pgs: 6 active+recovering+undersized+remapped, 152 active+clean; 6.0 GiB data, 23 GiB used, 603 GiB / 626 GiB avail; 861 B/s rd, 51 MiB/s wr, 810 op/s; 4293/293907 objects misplaced (1.461%); 21 MiB/s, 332 objects/s recovering 2024-09-18T08:20:08.721 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:20:08 smithi007 ceph-mon[39249]: pgmap v867: 158 pgs: 6 active+recovering+undersized+remapped, 152 active+clean; 6.0 GiB data, 23 GiB used, 603 GiB / 626 GiB avail; 861 B/s rd, 51 MiB/s wr, 810 op/s; 4293/293907 objects misplaced (1.461%); 21 MiB/s, 332 objects/s recovering 2024-09-18T08:20:08.727 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7 2024-09-18T08:20:08.762 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:20:08.762 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:20:08.763 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7' 2024-09-18T08:20:08.863 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5 2024-09-18T08:20:08.893 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/779349ca-7593-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T08:20:08.896 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:20:08.897 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:20:08.897 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5' 2024-09-18T08:20:08.904 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:20:08 smithi050 ceph-mon[32371]: pgmap v867: 158 pgs: 6 active+recovering+undersized+remapped, 152 active+clean; 6.0 GiB data, 23 GiB used, 603 GiB / 626 GiB avail; 861 B/s rd, 51 MiB/s wr, 810 op/s; 4293/293907 objects misplaced (1.461%); 21 MiB/s, 332 objects/s recovering 2024-09-18T08:20:08.998 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4 2024-09-18T08:20:09.032 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:20:09.033 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:20:09.033 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4' 2024-09-18T08:20:09.134 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0 2024-09-18T08:20:09.175 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:20:09.176 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:20:09.177 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0' 2024-09-18T08:20:09.278 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3 2024-09-18T08:20:09.322 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:20:09.323 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:20:09.323 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3' 2024-09-18T08:20:09.424 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2 2024-09-18T08:20:09.466 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:20:09.467 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:20:09.467 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2' 2024-09-18T08:20:09.568 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2 2024-09-18T08:20:09.609 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:20:09.609 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:20:09.610 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2' 2024-09-18T08:20:09.646 INFO:teuthology.orchestra.run.smithi007.stdout:osd.0: {} 2024-09-18T08:20:09.647 INFO:teuthology.orchestra.run.smithi007.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-18T08:20:09.661 INFO:teuthology.orchestra.run.smithi007.stdout:osd.1: {} 2024-09-18T08:20:09.662 INFO:teuthology.orchestra.run.smithi007.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-18T08:20:09.701 INFO:teuthology.orchestra.run.smithi007.stdout:osd.2: {} 2024-09-18T08:20:09.701 INFO:teuthology.orchestra.run.smithi007.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-18T08:20:09.711 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7 2024-09-18T08:20:09.736 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:20:09.737 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:20:09.737 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7' 2024-09-18T08:20:09.751 INFO:teuthology.orchestra.run.smithi007.stdout:osd.3: {} 2024-09-18T08:20:09.751 INFO:teuthology.orchestra.run.smithi007.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-18T08:20:09.797 INFO:teuthology.orchestra.run.smithi007.stdout:osd.4: {} 2024-09-18T08:20:09.798 INFO:teuthology.orchestra.run.smithi007.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-18T08:20:09.838 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0 2024-09-18T08:20:09.842 INFO:teuthology.orchestra.run.smithi007.stdout:osd.5: {} 2024-09-18T08:20:09.842 INFO:teuthology.orchestra.run.smithi007.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-18T08:20:09.874 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:20:09.875 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:20:09.876 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0' 2024-09-18T08:20:09.890 INFO:teuthology.orchestra.run.smithi007.stdout:osd.6: {} 2024-09-18T08:20:09.890 INFO:teuthology.orchestra.run.smithi007.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-18T08:20:09.929 INFO:teuthology.orchestra.run.smithi007.stdout:osd.7: {} 2024-09-18T08:20:09.930 INFO:teuthology.orchestra.run.smithi007.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-18T08:20:09.977 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6 2024-09-18T08:20:10.012 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:20:10.012 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:20:10.012 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6' 2024-09-18T08:20:10.113 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0 2024-09-18T08:20:10.155 INFO:journalctl@ceph.iscsi.iscsi.a.smithi050.stdout:Sep 18 08:20:09 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-iscsi-iscsi-a[57782]: debug there is no tcmu-runner data available 2024-09-18T08:20:10.158 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:20:10.158 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:20:10.159 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0' 2024-09-18T08:20:10.260 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7 2024-09-18T08:20:10.291 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:20:10.291 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:20:10.292 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7' 2024-09-18T08:20:10.393 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3 2024-09-18T08:20:10.434 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:20:10.435 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:20:10.435 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3' 2024-09-18T08:20:10.504 INFO:teuthology.orchestra.run.smithi007.stderr:noscrub is unset 2024-09-18T08:20:10.536 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5 2024-09-18T08:20:10.560 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:20:10.561 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:20:10.561 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5' 2024-09-18T08:20:10.581 INFO:teuthology.orchestra.run.smithi007.stdout:ERROR: (22) Invalid argument 2024-09-18T08:20:10.581 INFO:teuthology.orchestra.run.smithi007.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-18T08:20:10.661 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6 2024-09-18T08:20:10.691 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:20:10.692 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:20:10.693 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6' 2024-09-18T08:20:10.730 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:20:10 smithi007 ceph-mon[29459]: pgmap v868: 158 pgs: 4 active+recovering+undersized+remapped, 154 active+clean; 5.9 GiB data, 23 GiB used, 603 GiB / 626 GiB avail; 1023 B/s rd, 40 MiB/s wr, 891 op/s; 1964/292182 objects misplaced (0.672%); 31 MiB/s, 498 objects/s recovering 2024-09-18T08:20:10.730 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:20:10 smithi007 ceph-mon[29459]: from='client.? 172.21.15.7:0/940416322' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-18T08:20:10.730 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:20:10 smithi007 ceph-mon[29459]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-18T08:20:10.730 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:20:10 smithi007 ceph-mon[29459]: from='client.? 172.21.15.7:0/2310482189' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-18T08:20:10.730 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:20:10 smithi007 ceph-mon[29459]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-18T08:20:10.730 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:20:10 smithi007 ceph-mon[39249]: pgmap v868: 158 pgs: 4 active+recovering+undersized+remapped, 154 active+clean; 5.9 GiB data, 23 GiB used, 603 GiB / 626 GiB avail; 1023 B/s rd, 40 MiB/s wr, 891 op/s; 1964/292182 objects misplaced (0.672%); 31 MiB/s, 498 objects/s recovering 2024-09-18T08:20:10.731 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:20:10 smithi007 ceph-mon[39249]: from='client.? 172.21.15.7:0/940416322' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-18T08:20:10.731 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:20:10 smithi007 ceph-mon[39249]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-18T08:20:10.731 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:20:10 smithi007 ceph-mon[39249]: from='client.? 172.21.15.7:0/2310482189' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-18T08:20:10.731 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:20:10 smithi007 ceph-mon[39249]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-18T08:20:10.794 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7 2024-09-18T08:20:10.824 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:20:10.825 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:20:10.825 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7' 2024-09-18T08:20:10.905 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:20:10 smithi050 ceph-mon[32371]: pgmap v868: 158 pgs: 4 active+recovering+undersized+remapped, 154 active+clean; 5.9 GiB data, 23 GiB used, 603 GiB / 626 GiB avail; 1023 B/s rd, 40 MiB/s wr, 891 op/s; 1964/292182 objects misplaced (0.672%); 31 MiB/s, 498 objects/s recovering 2024-09-18T08:20:10.905 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:20:10 smithi050 ceph-mon[32371]: from='client.? 172.21.15.7:0/940416322' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-18T08:20:10.905 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:20:10 smithi050 ceph-mon[32371]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-18T08:20:10.905 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:20:10 smithi050 ceph-mon[32371]: from='client.? 172.21.15.7:0/2310482189' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-18T08:20:10.905 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:20:10 smithi050 ceph-mon[32371]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-18T08:20:10.927 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0 2024-09-18T08:20:10.963 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:20:10.964 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:20:10.964 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0' 2024-09-18T08:20:11.065 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7 2024-09-18T08:20:11.090 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:20:11.091 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:20:11.091 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7' 2024-09-18T08:20:11.192 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4 2024-09-18T08:20:11.222 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:20:11.223 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:20:11.224 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4' 2024-09-18T08:20:11.326 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7 2024-09-18T08:20:11.359 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:20:11.360 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:20:11.360 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7' 2024-09-18T08:20:11.461 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3 2024-09-18T08:20:11.499 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:20:11.499 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:20:11.500 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3' 2024-09-18T08:20:11.600 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1 2024-09-18T08:20:11.638 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:20:11.638 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:20:11.639 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1' 2024-09-18T08:20:11.740 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7 2024-09-18T08:20:11.773 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:20:11.773 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:20:11.773 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7' 2024-09-18T08:20:11.806 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:20:11 smithi050 ceph-mon[32371]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-18T08:20:11.806 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:20:11 smithi050 ceph-mon[32371]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-18T08:20:11.806 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:20:11 smithi050 ceph-mon[32371]: osdmap e433: 8 total, 8 up, 7 in 2024-09-18T08:20:11.874 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1 2024-09-18T08:20:11.900 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:20:11.901 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:20:11.901 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1' 2024-09-18T08:20:11.980 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:20:11 smithi007 ceph-mon[29459]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-18T08:20:11.980 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:20:11 smithi007 ceph-mon[29459]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-18T08:20:11.980 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:20:11 smithi007 ceph-mon[29459]: osdmap e433: 8 total, 8 up, 7 in 2024-09-18T08:20:11.980 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:20:11 smithi007 ceph-mon[39249]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-18T08:20:11.980 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:20:11 smithi007 ceph-mon[39249]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-18T08:20:11.980 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:20:11 smithi007 ceph-mon[39249]: osdmap e433: 8 total, 8 up, 7 in 2024-09-18T08:20:12.003 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5 2024-09-18T08:20:12.026 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:20:12.026 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:20:12.027 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5' 2024-09-18T08:20:12.128 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6 2024-09-18T08:20:12.157 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:20:12.158 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:20:12.158 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6' 2024-09-18T08:20:12.261 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2 2024-09-18T08:20:12.289 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:20:12.290 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:20:12.290 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2' 2024-09-18T08:20:12.391 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5 2024-09-18T08:20:12.422 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:20:12.422 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:20:12.423 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5' 2024-09-18T08:20:12.523 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3 2024-09-18T08:20:12.559 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:20:12.560 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:20:12.561 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3' 2024-09-18T08:20:12.662 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4 2024-09-18T08:20:12.689 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:20:12.690 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:20:12.690 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4' 2024-09-18T08:20:12.729 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:20:12 smithi007 ceph-mon[29459]: 4.11 scrub starts 2024-09-18T08:20:12.730 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:20:12 smithi007 ceph-mon[29459]: 4.11 scrub ok 2024-09-18T08:20:12.730 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:20:12 smithi007 ceph-mon[29459]: pgmap v870: 158 pgs: 3 active+recovering+undersized+remapped, 155 active+clean; 5.8 GiB data, 23 GiB used, 603 GiB / 626 GiB avail; 1019 B/s rd, 9.5 MiB/s wr, 648 op/s; 1839/287667 objects misplaced (0.639%); 25 MiB/s, 407 objects/s recovering 2024-09-18T08:20:12.730 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:20:12 smithi007 ceph-mon[29459]: 4.7 scrub starts 2024-09-18T08:20:12.730 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:20:12 smithi007 ceph-mon[29459]: 4.7 scrub ok 2024-09-18T08:20:12.730 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:20:12 smithi007 ceph-mon[39249]: 4.11 scrub starts 2024-09-18T08:20:12.730 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:20:12 smithi007 ceph-mon[39249]: 4.11 scrub ok 2024-09-18T08:20:12.730 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:20:12 smithi007 ceph-mon[39249]: pgmap v870: 158 pgs: 3 active+recovering+undersized+remapped, 155 active+clean; 5.8 GiB data, 23 GiB used, 603 GiB / 626 GiB avail; 1019 B/s rd, 9.5 MiB/s wr, 648 op/s; 1839/287667 objects misplaced (0.639%); 25 MiB/s, 407 objects/s recovering 2024-09-18T08:20:12.730 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:20:12 smithi007 ceph-mon[39249]: 4.7 scrub starts 2024-09-18T08:20:12.730 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:20:12 smithi007 ceph-mon[39249]: 4.7 scrub ok 2024-09-18T08:20:12.791 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4 2024-09-18T08:20:12.815 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:20:12.815 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:20:12.816 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4' 2024-09-18T08:20:12.905 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:20:12 smithi050 ceph-mon[32371]: 4.11 scrub starts 2024-09-18T08:20:12.905 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:20:12 smithi050 ceph-mon[32371]: 4.11 scrub ok 2024-09-18T08:20:12.905 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:20:12 smithi050 ceph-mon[32371]: pgmap v870: 158 pgs: 3 active+recovering+undersized+remapped, 155 active+clean; 5.8 GiB data, 23 GiB used, 603 GiB / 626 GiB avail; 1019 B/s rd, 9.5 MiB/s wr, 648 op/s; 1839/287667 objects misplaced (0.639%); 25 MiB/s, 407 objects/s recovering 2024-09-18T08:20:12.905 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:20:12 smithi050 ceph-mon[32371]: 4.7 scrub starts 2024-09-18T08:20:12.905 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:20:12 smithi050 ceph-mon[32371]: 4.7 scrub ok 2024-09-18T08:20:12.917 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6 2024-09-18T08:20:12.928 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid 779349ca-7593-11ef-bceb-c7b262605968 -- ceph daemon osd.2 dump_ops_in_flight 2024-09-18T08:20:12.939 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:20:12.939 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:20:12.940 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6' 2024-09-18T08:20:13.040 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6 2024-09-18T08:20:13.064 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:20:13.065 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:20:13.065 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6' 2024-09-18T08:20:13.166 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3 2024-09-18T08:20:13.191 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:20:13.191 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:20:13.192 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3' 2024-09-18T08:20:13.293 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0 2024-09-18T08:20:13.318 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:20:13.318 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:20:13.319 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0' 2024-09-18T08:20:13.420 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2 2024-09-18T08:20:13.448 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:20:13.449 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:20:13.449 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2' 2024-09-18T08:20:13.521 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 6, 5, 7] out_osds: [4] dead_osds: [] live_osds: [0, 2, 3, 4, 5, 6, 7, 1] 2024-09-18T08:20:13.521 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-18T08:20:13.521 INFO:tasks.thrashosds.thrasher:thrash_pg_upmap_items 2024-09-18T08:20:13.522 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid 779349ca-7593-11ef-bceb-c7b262605968 -- ceph osd dump -f json-pretty 2024-09-18T08:20:13.550 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0 2024-09-18T08:20:13.611 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:20:13.612 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:20:13.612 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0' 2024-09-18T08:20:13.713 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4 2024-09-18T08:20:13.730 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:20:13 smithi007 ceph-mon[29459]: 3.2 scrub starts 2024-09-18T08:20:13.730 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:20:13 smithi007 ceph-mon[29459]: 3.2 scrub ok 2024-09-18T08:20:13.731 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:20:13 smithi007 ceph-mon[29459]: osdmap e434: 8 total, 8 up, 7 in 2024-09-18T08:20:13.731 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:20:13 smithi007 ceph-mon[39249]: 3.2 scrub starts 2024-09-18T08:20:13.731 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:20:13 smithi007 ceph-mon[39249]: 3.2 scrub ok 2024-09-18T08:20:13.731 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:20:13 smithi007 ceph-mon[39249]: osdmap e434: 8 total, 8 up, 7 in 2024-09-18T08:20:13.744 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:20:13.745 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:20:13.745 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4' 2024-09-18T08:20:13.846 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0 2024-09-18T08:20:13.890 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/779349ca-7593-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T08:20:13.896 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:20:13.896 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:20:13.897 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0' 2024-09-18T08:20:13.904 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:20:13 smithi050 ceph-mon[32371]: 3.2 scrub starts 2024-09-18T08:20:13.905 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:20:13 smithi050 ceph-mon[32371]: 3.2 scrub ok 2024-09-18T08:20:13.905 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:20:13 smithi050 ceph-mon[32371]: osdmap e434: 8 total, 8 up, 7 in 2024-09-18T08:20:13.998 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4 2024-09-18T08:20:14.028 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:20:14.029 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:20:14.031 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4' 2024-09-18T08:20:14.132 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7 2024-09-18T08:20:14.163 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:20:14.164 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:20:14.164 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7' 2024-09-18T08:20:14.265 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3 2024-09-18T08:20:14.304 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:20:14.304 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:20:14.305 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3' 2024-09-18T08:20:14.406 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4 2024-09-18T08:20:14.435 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:20:14.436 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:20:14.437 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4' 2024-09-18T08:20:14.489 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/779349ca-7593-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T08:20:14.538 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3 2024-09-18T08:20:14.544 INFO:teuthology.orchestra.run.smithi007.stdout:ERROR: (22) Invalid argument 2024-09-18T08:20:14.544 INFO:teuthology.orchestra.run.smithi007.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-18T08:20:14.580 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:20:14.581 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:20:14.581 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3' 2024-09-18T08:20:14.682 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7 2024-09-18T08:20:14.705 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:20:14.706 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:20:14.706 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7' 2024-09-18T08:20:14.730 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:20:14 smithi007 ceph-mon[29459]: pgmap v872: 158 pgs: 1 peering, 2 active+recovering+undersized+remapped, 155 active+clean; 5.7 GiB data, 22 GiB used, 604 GiB / 626 GiB avail; 1.6 KiB/s rd, 8.1 MiB/s wr, 844 op/s; 1040/281616 objects misplaced (0.369%); 29 MiB/s, 463 objects/s recovering 2024-09-18T08:20:14.730 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:20:14 smithi007 ceph-mon[29459]: Health check update: Degraded data redundancy: 2 pgs undersized (PG_DEGRADED) 2024-09-18T08:20:14.730 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:20:14 smithi007 ceph-mon[29459]: osdmap e435: 8 total, 8 up, 7 in 2024-09-18T08:20:14.730 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:20:14 smithi007 ceph-mon[29459]: osdmap e436: 8 total, 8 up, 7 in 2024-09-18T08:20:14.730 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:20:14 smithi007 ceph-mon[39249]: pgmap v872: 158 pgs: 1 peering, 2 active+recovering+undersized+remapped, 155 active+clean; 5.7 GiB data, 22 GiB used, 604 GiB / 626 GiB avail; 1.6 KiB/s rd, 8.1 MiB/s wr, 844 op/s; 1040/281616 objects misplaced (0.369%); 29 MiB/s, 463 objects/s recovering 2024-09-18T08:20:14.730 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:20:14 smithi007 ceph-mon[39249]: Health check update: Degraded data redundancy: 2 pgs undersized (PG_DEGRADED) 2024-09-18T08:20:14.730 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:20:14 smithi007 ceph-mon[39249]: osdmap e435: 8 total, 8 up, 7 in 2024-09-18T08:20:14.730 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:20:14 smithi007 ceph-mon[39249]: osdmap e436: 8 total, 8 up, 7 in 2024-09-18T08:20:14.807 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7 2024-09-18T08:20:14.829 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:20:14.830 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:20:14.830 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7' 2024-09-18T08:20:14.905 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:20:14 smithi050 ceph-mon[32371]: pgmap v872: 158 pgs: 1 peering, 2 active+recovering+undersized+remapped, 155 active+clean; 5.7 GiB data, 22 GiB used, 604 GiB / 626 GiB avail; 1.6 KiB/s rd, 8.1 MiB/s wr, 844 op/s; 1040/281616 objects misplaced (0.369%); 29 MiB/s, 463 objects/s recovering 2024-09-18T08:20:14.905 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:20:14 smithi050 ceph-mon[32371]: Health check update: Degraded data redundancy: 2 pgs undersized (PG_DEGRADED) 2024-09-18T08:20:14.905 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:20:14 smithi050 ceph-mon[32371]: osdmap e435: 8 total, 8 up, 7 in 2024-09-18T08:20:14.905 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:20:14 smithi050 ceph-mon[32371]: osdmap e436: 8 total, 8 up, 7 in 2024-09-18T08:20:14.931 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3 2024-09-18T08:20:14.955 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:20:14.955 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:20:14.956 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3' 2024-09-18T08:20:15.057 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3 2024-09-18T08:20:15.079 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:20:15.080 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:20:15.080 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3' 2024-09-18T08:20:15.181 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7 2024-09-18T08:20:15.203 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:20:15.204 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:20:15.204 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7' 2024-09-18T08:20:15.306 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0 2024-09-18T08:20:15.328 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:20:15.329 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:20:15.329 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0' 2024-09-18T08:20:15.337 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-09-18T08:20:15.337 INFO:teuthology.orchestra.run.smithi007.stdout:{ 2024-09-18T08:20:15.337 INFO:teuthology.orchestra.run.smithi007.stdout: "epoch": 436, 2024-09-18T08:20:15.337 INFO:teuthology.orchestra.run.smithi007.stdout: "fsid": "779349ca-7593-11ef-bceb-c7b262605968", 2024-09-18T08:20:15.337 INFO:teuthology.orchestra.run.smithi007.stdout: "created": "2024-09-18T07:57:09.295710+0000", 2024-09-18T08:20:15.337 INFO:teuthology.orchestra.run.smithi007.stdout: "modified": "2024-09-18T08:20:14.463673+0000", 2024-09-18T08:20:15.337 INFO:teuthology.orchestra.run.smithi007.stdout: "last_up_change": "2024-09-18T08:14:38.391465+0000", 2024-09-18T08:20:15.337 INFO:teuthology.orchestra.run.smithi007.stdout: "last_in_change": "2024-09-18T08:18:26.612016+0000", 2024-09-18T08:20:15.337 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": "nodeep-scrub,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit", 2024-09-18T08:20:15.337 INFO:teuthology.orchestra.run.smithi007.stdout: "flags_num": 5804032, 2024-09-18T08:20:15.337 INFO:teuthology.orchestra.run.smithi007.stdout: "flags_set": [ 2024-09-18T08:20:15.338 INFO:teuthology.orchestra.run.smithi007.stdout: "nodeep-scrub", 2024-09-18T08:20:15.338 INFO:teuthology.orchestra.run.smithi007.stdout: "pglog_hardlimit", 2024-09-18T08:20:15.338 INFO:teuthology.orchestra.run.smithi007.stdout: "purged_snapdirs", 2024-09-18T08:20:15.338 INFO:teuthology.orchestra.run.smithi007.stdout: "recovery_deletes", 2024-09-18T08:20:15.338 INFO:teuthology.orchestra.run.smithi007.stdout: "sortbitwise" 2024-09-18T08:20:15.338 INFO:teuthology.orchestra.run.smithi007.stdout: ], 2024-09-18T08:20:15.338 INFO:teuthology.orchestra.run.smithi007.stdout: "crush_version": 19, 2024-09-18T08:20:15.338 INFO:teuthology.orchestra.run.smithi007.stdout: "full_ratio": 0.94999998807907104, 2024-09-18T08:20:15.338 INFO:teuthology.orchestra.run.smithi007.stdout: "backfillfull_ratio": 0.89999997615814209, 2024-09-18T08:20:15.338 INFO:teuthology.orchestra.run.smithi007.stdout: "nearfull_ratio": 0.85000002384185791, 2024-09-18T08:20:15.338 INFO:teuthology.orchestra.run.smithi007.stdout: "cluster_snapshot": "", 2024-09-18T08:20:15.338 INFO:teuthology.orchestra.run.smithi007.stdout: "pool_max": 11, 2024-09-18T08:20:15.338 INFO:teuthology.orchestra.run.smithi007.stdout: "max_osd": 8, 2024-09-18T08:20:15.338 INFO:teuthology.orchestra.run.smithi007.stdout: "require_min_compat_client": "luminous", 2024-09-18T08:20:15.339 INFO:teuthology.orchestra.run.smithi007.stdout: "min_compat_client": "luminous", 2024-09-18T08:20:15.339 INFO:teuthology.orchestra.run.smithi007.stdout: "require_osd_release": "squid", 2024-09-18T08:20:15.339 INFO:teuthology.orchestra.run.smithi007.stdout: "allow_crimson": false, 2024-09-18T08:20:15.339 INFO:teuthology.orchestra.run.smithi007.stdout: "pools": [ 2024-09-18T08:20:15.339 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:20:15.339 INFO:teuthology.orchestra.run.smithi007.stdout: "pool": 1, 2024-09-18T08:20:15.339 INFO:teuthology.orchestra.run.smithi007.stdout: "pool_name": ".mgr", 2024-09-18T08:20:15.339 INFO:teuthology.orchestra.run.smithi007.stdout: "create_time": "2024-09-18T08:00:11.802627+0000", 2024-09-18T08:20:15.339 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 1, 2024-09-18T08:20:15.339 INFO:teuthology.orchestra.run.smithi007.stdout: "flags_names": "hashpspool", 2024-09-18T08:20:15.339 INFO:teuthology.orchestra.run.smithi007.stdout: "type": 1, 2024-09-18T08:20:15.339 INFO:teuthology.orchestra.run.smithi007.stdout: "size": 3, 2024-09-18T08:20:15.339 INFO:teuthology.orchestra.run.smithi007.stdout: "min_size": 2, 2024-09-18T08:20:15.340 INFO:teuthology.orchestra.run.smithi007.stdout: "crush_rule": 0, 2024-09-18T08:20:15.340 INFO:teuthology.orchestra.run.smithi007.stdout: "peering_crush_bucket_count": 0, 2024-09-18T08:20:15.340 INFO:teuthology.orchestra.run.smithi007.stdout: "peering_crush_bucket_target": 0, 2024-09-18T08:20:15.340 INFO:teuthology.orchestra.run.smithi007.stdout: "peering_crush_bucket_barrier": 0, 2024-09-18T08:20:15.340 INFO:teuthology.orchestra.run.smithi007.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-09-18T08:20:15.340 INFO:teuthology.orchestra.run.smithi007.stdout: "is_stretch_pool": false, 2024-09-18T08:20:15.340 INFO:teuthology.orchestra.run.smithi007.stdout: "object_hash": 2, 2024-09-18T08:20:15.340 INFO:teuthology.orchestra.run.smithi007.stdout: "pg_autoscale_mode": "off", 2024-09-18T08:20:15.340 INFO:teuthology.orchestra.run.smithi007.stdout: "pg_num": 11, 2024-09-18T08:20:15.340 INFO:teuthology.orchestra.run.smithi007.stdout: "pg_placement_num": 11, 2024-09-18T08:20:15.340 INFO:teuthology.orchestra.run.smithi007.stdout: "pg_placement_num_target": 11, 2024-09-18T08:20:15.340 INFO:teuthology.orchestra.run.smithi007.stdout: "pg_num_target": 11, 2024-09-18T08:20:15.340 INFO:teuthology.orchestra.run.smithi007.stdout: "pg_num_pending": 11, 2024-09-18T08:20:15.341 INFO:teuthology.orchestra.run.smithi007.stdout: "last_pg_merge_meta": { 2024-09-18T08:20:15.341 INFO:teuthology.orchestra.run.smithi007.stdout: "source_pgid": "0.0", 2024-09-18T08:20:15.341 INFO:teuthology.orchestra.run.smithi007.stdout: "ready_epoch": 0, 2024-09-18T08:20:15.341 INFO:teuthology.orchestra.run.smithi007.stdout: "last_epoch_started": 0, 2024-09-18T08:20:15.341 INFO:teuthology.orchestra.run.smithi007.stdout: "last_epoch_clean": 0, 2024-09-18T08:20:15.341 INFO:teuthology.orchestra.run.smithi007.stdout: "source_version": "0'0", 2024-09-18T08:20:15.341 INFO:teuthology.orchestra.run.smithi007.stdout: "target_version": "0'0" 2024-09-18T08:20:15.341 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:20:15.341 INFO:teuthology.orchestra.run.smithi007.stdout: "last_change": "285", 2024-09-18T08:20:15.341 INFO:teuthology.orchestra.run.smithi007.stdout: "last_force_op_resend": "0", 2024-09-18T08:20:15.341 INFO:teuthology.orchestra.run.smithi007.stdout: "last_force_op_resend_prenautilus": "0", 2024-09-18T08:20:15.341 INFO:teuthology.orchestra.run.smithi007.stdout: "last_force_op_resend_preluminous": "152", 2024-09-18T08:20:15.341 INFO:teuthology.orchestra.run.smithi007.stdout: "auid": 0, 2024-09-18T08:20:15.342 INFO:teuthology.orchestra.run.smithi007.stdout: "snap_mode": "selfmanaged", 2024-09-18T08:20:15.342 INFO:teuthology.orchestra.run.smithi007.stdout: "snap_seq": 0, 2024-09-18T08:20:15.342 INFO:teuthology.orchestra.run.smithi007.stdout: "snap_epoch": 0, 2024-09-18T08:20:15.342 INFO:teuthology.orchestra.run.smithi007.stdout: "pool_snaps": [], 2024-09-18T08:20:15.342 INFO:teuthology.orchestra.run.smithi007.stdout: "removed_snaps": "[]", 2024-09-18T08:20:15.342 INFO:teuthology.orchestra.run.smithi007.stdout: "quota_max_bytes": 0, 2024-09-18T08:20:15.342 INFO:teuthology.orchestra.run.smithi007.stdout: "quota_max_objects": 0, 2024-09-18T08:20:15.342 INFO:teuthology.orchestra.run.smithi007.stdout: "tiers": [], 2024-09-18T08:20:15.342 INFO:teuthology.orchestra.run.smithi007.stdout: "tier_of": -1, 2024-09-18T08:20:15.342 INFO:teuthology.orchestra.run.smithi007.stdout: "read_tier": -1, 2024-09-18T08:20:15.343 INFO:teuthology.orchestra.run.smithi007.stdout: "write_tier": -1, 2024-09-18T08:20:15.343 INFO:teuthology.orchestra.run.smithi007.stdout: "cache_mode": "none", 2024-09-18T08:20:15.343 INFO:teuthology.orchestra.run.smithi007.stdout: "target_max_bytes": 0, 2024-09-18T08:20:15.343 INFO:teuthology.orchestra.run.smithi007.stdout: "target_max_objects": 0, 2024-09-18T08:20:15.343 INFO:teuthology.orchestra.run.smithi007.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-09-18T08:20:15.343 INFO:teuthology.orchestra.run.smithi007.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-09-18T08:20:15.343 INFO:teuthology.orchestra.run.smithi007.stdout: "cache_target_full_ratio_micro": 800000, 2024-09-18T08:20:15.343 INFO:teuthology.orchestra.run.smithi007.stdout: "cache_min_flush_age": 0, 2024-09-18T08:20:15.343 INFO:teuthology.orchestra.run.smithi007.stdout: "cache_min_evict_age": 0, 2024-09-18T08:20:15.344 INFO:teuthology.orchestra.run.smithi007.stdout: "erasure_code_profile": "", 2024-09-18T08:20:15.344 INFO:teuthology.orchestra.run.smithi007.stdout: "hit_set_params": { 2024-09-18T08:20:15.344 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "none" 2024-09-18T08:20:15.344 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:20:15.344 INFO:teuthology.orchestra.run.smithi007.stdout: "hit_set_period": 0, 2024-09-18T08:20:15.344 INFO:teuthology.orchestra.run.smithi007.stdout: "hit_set_count": 0, 2024-09-18T08:20:15.344 INFO:teuthology.orchestra.run.smithi007.stdout: "use_gmt_hitset": true, 2024-09-18T08:20:15.344 INFO:teuthology.orchestra.run.smithi007.stdout: "min_read_recency_for_promote": 0, 2024-09-18T08:20:15.344 INFO:teuthology.orchestra.run.smithi007.stdout: "min_write_recency_for_promote": 0, 2024-09-18T08:20:15.345 INFO:teuthology.orchestra.run.smithi007.stdout: "hit_set_grade_decay_rate": 0, 2024-09-18T08:20:15.345 INFO:teuthology.orchestra.run.smithi007.stdout: "hit_set_search_last_n": 0, 2024-09-18T08:20:15.345 INFO:teuthology.orchestra.run.smithi007.stdout: "grade_table": [], 2024-09-18T08:20:15.345 INFO:teuthology.orchestra.run.smithi007.stdout: "stripe_width": 0, 2024-09-18T08:20:15.345 INFO:teuthology.orchestra.run.smithi007.stdout: "expected_num_objects": 0, 2024-09-18T08:20:15.345 INFO:teuthology.orchestra.run.smithi007.stdout: "fast_read": false, 2024-09-18T08:20:15.345 INFO:teuthology.orchestra.run.smithi007.stdout: "options": { 2024-09-18T08:20:15.345 INFO:teuthology.orchestra.run.smithi007.stdout: "pg_num_max": 32, 2024-09-18T08:20:15.345 INFO:teuthology.orchestra.run.smithi007.stdout: "pg_num_min": 1 2024-09-18T08:20:15.345 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:20:15.345 INFO:teuthology.orchestra.run.smithi007.stdout: "application_metadata": { 2024-09-18T08:20:15.345 INFO:teuthology.orchestra.run.smithi007.stdout: "mgr": {} 2024-09-18T08:20:15.345 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:20:15.346 INFO:teuthology.orchestra.run.smithi007.stdout: "read_balance": { 2024-09-18T08:20:15.346 INFO:teuthology.orchestra.run.smithi007.stdout: "score_type": "Fair distribution", 2024-09-18T08:20:15.346 INFO:teuthology.orchestra.run.smithi007.stdout: "score_acting": 2.5499999523162842, 2024-09-18T08:20:15.346 INFO:teuthology.orchestra.run.smithi007.stdout: "score_stable": 2.5499999523162842, 2024-09-18T08:20:15.346 INFO:teuthology.orchestra.run.smithi007.stdout: "optimal_score": 1, 2024-09-18T08:20:15.346 INFO:teuthology.orchestra.run.smithi007.stdout: "raw_score_acting": 2.5499999523162842, 2024-09-18T08:20:15.346 INFO:teuthology.orchestra.run.smithi007.stdout: "raw_score_stable": 2.5499999523162842, 2024-09-18T08:20:15.346 INFO:teuthology.orchestra.run.smithi007.stdout: "primary_affinity_weighted": 0.80927085876464844, 2024-09-18T08:20:15.346 INFO:teuthology.orchestra.run.smithi007.stdout: "average_primary_affinity": 0.92000001668930054, 2024-09-18T08:20:15.346 INFO:teuthology.orchestra.run.smithi007.stdout: "average_primary_affinity_weighted": 0.81000000238418579 2024-09-18T08:20:15.346 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-09-18T08:20:15.346 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:20:15.347 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:20:15.347 INFO:teuthology.orchestra.run.smithi007.stdout: "pool": 2, 2024-09-18T08:20:15.347 INFO:teuthology.orchestra.run.smithi007.stdout: "pool_name": "datapool", 2024-09-18T08:20:15.347 INFO:teuthology.orchestra.run.smithi007.stdout: "create_time": "2024-09-18T08:02:45.374099+0000", 2024-09-18T08:20:15.347 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 8193, 2024-09-18T08:20:15.347 INFO:teuthology.orchestra.run.smithi007.stdout: "flags_names": "hashpspool,selfmanaged_snaps", 2024-09-18T08:20:15.347 INFO:teuthology.orchestra.run.smithi007.stdout: "type": 1, 2024-09-18T08:20:15.347 INFO:teuthology.orchestra.run.smithi007.stdout: "size": 3, 2024-09-18T08:20:15.347 INFO:teuthology.orchestra.run.smithi007.stdout: "min_size": 2, 2024-09-18T08:20:15.347 INFO:teuthology.orchestra.run.smithi007.stdout: "crush_rule": 0, 2024-09-18T08:20:15.347 INFO:teuthology.orchestra.run.smithi007.stdout: "peering_crush_bucket_count": 0, 2024-09-18T08:20:15.347 INFO:teuthology.orchestra.run.smithi007.stdout: "peering_crush_bucket_target": 0, 2024-09-18T08:20:15.347 INFO:teuthology.orchestra.run.smithi007.stdout: "peering_crush_bucket_barrier": 0, 2024-09-18T08:20:15.347 INFO:teuthology.orchestra.run.smithi007.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-09-18T08:20:15.348 INFO:teuthology.orchestra.run.smithi007.stdout: "is_stretch_pool": false, 2024-09-18T08:20:15.348 INFO:teuthology.orchestra.run.smithi007.stdout: "object_hash": 2, 2024-09-18T08:20:15.348 INFO:teuthology.orchestra.run.smithi007.stdout: "pg_autoscale_mode": "off", 2024-09-18T08:20:15.348 INFO:teuthology.orchestra.run.smithi007.stdout: "pg_num": 3, 2024-09-18T08:20:15.348 INFO:teuthology.orchestra.run.smithi007.stdout: "pg_placement_num": 3, 2024-09-18T08:20:15.348 INFO:teuthology.orchestra.run.smithi007.stdout: "pg_placement_num_target": 3, 2024-09-18T08:20:15.348 INFO:teuthology.orchestra.run.smithi007.stdout: "pg_num_target": 3, 2024-09-18T08:20:15.348 INFO:teuthology.orchestra.run.smithi007.stdout: "pg_num_pending": 3, 2024-09-18T08:20:15.348 INFO:teuthology.orchestra.run.smithi007.stdout: "last_pg_merge_meta": { 2024-09-18T08:20:15.348 INFO:teuthology.orchestra.run.smithi007.stdout: "source_pgid": "0.0", 2024-09-18T08:20:15.348 INFO:teuthology.orchestra.run.smithi007.stdout: "ready_epoch": 0, 2024-09-18T08:20:15.348 INFO:teuthology.orchestra.run.smithi007.stdout: "last_epoch_started": 0, 2024-09-18T08:20:15.348 INFO:teuthology.orchestra.run.smithi007.stdout: "last_epoch_clean": 0, 2024-09-18T08:20:15.349 INFO:teuthology.orchestra.run.smithi007.stdout: "source_version": "0'0", 2024-09-18T08:20:15.349 INFO:teuthology.orchestra.run.smithi007.stdout: "target_version": "0'0" 2024-09-18T08:20:15.349 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:20:15.349 INFO:teuthology.orchestra.run.smithi007.stdout: "last_change": "64", 2024-09-18T08:20:15.349 INFO:teuthology.orchestra.run.smithi007.stdout: "last_force_op_resend": "0", 2024-09-18T08:20:15.349 INFO:teuthology.orchestra.run.smithi007.stdout: "last_force_op_resend_prenautilus": "0", 2024-09-18T08:20:15.349 INFO:teuthology.orchestra.run.smithi007.stdout: "last_force_op_resend_preluminous": "0", 2024-09-18T08:20:15.349 INFO:teuthology.orchestra.run.smithi007.stdout: "auid": 0, 2024-09-18T08:20:15.349 INFO:teuthology.orchestra.run.smithi007.stdout: "snap_mode": "selfmanaged", 2024-09-18T08:20:15.349 INFO:teuthology.orchestra.run.smithi007.stdout: "snap_seq": 2, 2024-09-18T08:20:15.349 INFO:teuthology.orchestra.run.smithi007.stdout: "snap_epoch": 64, 2024-09-18T08:20:15.349 INFO:teuthology.orchestra.run.smithi007.stdout: "pool_snaps": [], 2024-09-18T08:20:15.349 INFO:teuthology.orchestra.run.smithi007.stdout: "removed_snaps": "[]", 2024-09-18T08:20:15.349 INFO:teuthology.orchestra.run.smithi007.stdout: "quota_max_bytes": 0, 2024-09-18T08:20:15.350 INFO:teuthology.orchestra.run.smithi007.stdout: "quota_max_objects": 0, 2024-09-18T08:20:15.350 INFO:teuthology.orchestra.run.smithi007.stdout: "tiers": [], 2024-09-18T08:20:15.350 INFO:teuthology.orchestra.run.smithi007.stdout: "tier_of": -1, 2024-09-18T08:20:15.350 INFO:teuthology.orchestra.run.smithi007.stdout: "read_tier": -1, 2024-09-18T08:20:15.350 INFO:teuthology.orchestra.run.smithi007.stdout: "write_tier": -1, 2024-09-18T08:20:15.350 INFO:teuthology.orchestra.run.smithi007.stdout: "cache_mode": "none", 2024-09-18T08:20:15.350 INFO:teuthology.orchestra.run.smithi007.stdout: "target_max_bytes": 0, 2024-09-18T08:20:15.350 INFO:teuthology.orchestra.run.smithi007.stdout: "target_max_objects": 0, 2024-09-18T08:20:15.350 INFO:teuthology.orchestra.run.smithi007.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-09-18T08:20:15.350 INFO:teuthology.orchestra.run.smithi007.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-09-18T08:20:15.350 INFO:teuthology.orchestra.run.smithi007.stdout: "cache_target_full_ratio_micro": 800000, 2024-09-18T08:20:15.350 INFO:teuthology.orchestra.run.smithi007.stdout: "cache_min_flush_age": 0, 2024-09-18T08:20:15.350 INFO:teuthology.orchestra.run.smithi007.stdout: "cache_min_evict_age": 0, 2024-09-18T08:20:15.351 INFO:teuthology.orchestra.run.smithi007.stdout: "erasure_code_profile": "", 2024-09-18T08:20:15.351 INFO:teuthology.orchestra.run.smithi007.stdout: "hit_set_params": { 2024-09-18T08:20:15.351 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "none" 2024-09-18T08:20:15.351 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:20:15.351 INFO:teuthology.orchestra.run.smithi007.stdout: "hit_set_period": 0, 2024-09-18T08:20:15.351 INFO:teuthology.orchestra.run.smithi007.stdout: "hit_set_count": 0, 2024-09-18T08:20:15.351 INFO:teuthology.orchestra.run.smithi007.stdout: "use_gmt_hitset": true, 2024-09-18T08:20:15.351 INFO:teuthology.orchestra.run.smithi007.stdout: "min_read_recency_for_promote": 0, 2024-09-18T08:20:15.351 INFO:teuthology.orchestra.run.smithi007.stdout: "min_write_recency_for_promote": 0, 2024-09-18T08:20:15.351 INFO:teuthology.orchestra.run.smithi007.stdout: "hit_set_grade_decay_rate": 0, 2024-09-18T08:20:15.351 INFO:teuthology.orchestra.run.smithi007.stdout: "hit_set_search_last_n": 0, 2024-09-18T08:20:15.351 INFO:teuthology.orchestra.run.smithi007.stdout: "grade_table": [], 2024-09-18T08:20:15.351 INFO:teuthology.orchestra.run.smithi007.stdout: "stripe_width": 0, 2024-09-18T08:20:15.351 INFO:teuthology.orchestra.run.smithi007.stdout: "expected_num_objects": 0, 2024-09-18T08:20:15.351 INFO:teuthology.orchestra.run.smithi007.stdout: "fast_read": false, 2024-09-18T08:20:15.352 INFO:teuthology.orchestra.run.smithi007.stdout: "options": {}, 2024-09-18T08:20:15.352 INFO:teuthology.orchestra.run.smithi007.stdout: "application_metadata": { 2024-09-18T08:20:15.352 INFO:teuthology.orchestra.run.smithi007.stdout: "rbd": {} 2024-09-18T08:20:15.352 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:20:15.352 INFO:teuthology.orchestra.run.smithi007.stdout: "read_balance": { 2024-09-18T08:20:15.352 INFO:teuthology.orchestra.run.smithi007.stdout: "score_type": "Fair distribution", 2024-09-18T08:20:15.352 INFO:teuthology.orchestra.run.smithi007.stdout: "score_acting": 2.3299999237060547, 2024-09-18T08:20:15.352 INFO:teuthology.orchestra.run.smithi007.stdout: "score_stable": 2.3299999237060547, 2024-09-18T08:20:15.352 INFO:teuthology.orchestra.run.smithi007.stdout: "optimal_score": 0.86000001430511475, 2024-09-18T08:20:15.352 INFO:teuthology.orchestra.run.smithi007.stdout: "raw_score_acting": 2, 2024-09-18T08:20:15.352 INFO:teuthology.orchestra.run.smithi007.stdout: "raw_score_stable": 2, 2024-09-18T08:20:15.352 INFO:teuthology.orchestra.run.smithi007.stdout: "primary_affinity_weighted": 0.80927085876464844, 2024-09-18T08:20:15.352 INFO:teuthology.orchestra.run.smithi007.stdout: "average_primary_affinity": 0.92000001668930054, 2024-09-18T08:20:15.353 INFO:teuthology.orchestra.run.smithi007.stdout: "average_primary_affinity_weighted": 0.81000000238418579 2024-09-18T08:20:15.353 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-09-18T08:20:15.353 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:20:15.353 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:20:15.353 INFO:teuthology.orchestra.run.smithi007.stdout: "pool": 3, 2024-09-18T08:20:15.353 INFO:teuthology.orchestra.run.smithi007.stdout: "pool_name": ".rgw.root", 2024-09-18T08:20:15.353 INFO:teuthology.orchestra.run.smithi007.stdout: "create_time": "2024-09-18T08:02:46.319947+0000", 2024-09-18T08:20:15.353 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 1, 2024-09-18T08:20:15.353 INFO:teuthology.orchestra.run.smithi007.stdout: "flags_names": "hashpspool", 2024-09-18T08:20:15.353 INFO:teuthology.orchestra.run.smithi007.stdout: "type": 1, 2024-09-18T08:20:15.353 INFO:teuthology.orchestra.run.smithi007.stdout: "size": 3, 2024-09-18T08:20:15.353 INFO:teuthology.orchestra.run.smithi007.stdout: "min_size": 2, 2024-09-18T08:20:15.354 INFO:teuthology.orchestra.run.smithi007.stdout: "crush_rule": 0, 2024-09-18T08:20:15.354 INFO:teuthology.orchestra.run.smithi007.stdout: "peering_crush_bucket_count": 0, 2024-09-18T08:20:15.354 INFO:teuthology.orchestra.run.smithi007.stdout: "peering_crush_bucket_target": 0, 2024-09-18T08:20:15.354 INFO:teuthology.orchestra.run.smithi007.stdout: "peering_crush_bucket_barrier": 0, 2024-09-18T08:20:15.354 INFO:teuthology.orchestra.run.smithi007.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-09-18T08:20:15.354 INFO:teuthology.orchestra.run.smithi007.stdout: "is_stretch_pool": false, 2024-09-18T08:20:15.355 INFO:teuthology.orchestra.run.smithi007.stdout: "object_hash": 2, 2024-09-18T08:20:15.355 INFO:teuthology.orchestra.run.smithi007.stdout: "pg_autoscale_mode": "off", 2024-09-18T08:20:15.355 INFO:teuthology.orchestra.run.smithi007.stdout: "pg_num": 32, 2024-09-18T08:20:15.355 INFO:teuthology.orchestra.run.smithi007.stdout: "pg_placement_num": 32, 2024-09-18T08:20:15.355 INFO:teuthology.orchestra.run.smithi007.stdout: "pg_placement_num_target": 32, 2024-09-18T08:20:15.355 INFO:teuthology.orchestra.run.smithi007.stdout: "pg_num_target": 32, 2024-09-18T08:20:15.355 INFO:teuthology.orchestra.run.smithi007.stdout: "pg_num_pending": 32, 2024-09-18T08:20:15.355 INFO:teuthology.orchestra.run.smithi007.stdout: "last_pg_merge_meta": { 2024-09-18T08:20:15.355 INFO:teuthology.orchestra.run.smithi007.stdout: "source_pgid": "0.0", 2024-09-18T08:20:15.356 INFO:teuthology.orchestra.run.smithi007.stdout: "ready_epoch": 0, 2024-09-18T08:20:15.356 INFO:teuthology.orchestra.run.smithi007.stdout: "last_epoch_started": 0, 2024-09-18T08:20:15.356 INFO:teuthology.orchestra.run.smithi007.stdout: "last_epoch_clean": 0, 2024-09-18T08:20:15.356 INFO:teuthology.orchestra.run.smithi007.stdout: "source_version": "0'0", 2024-09-18T08:20:15.356 INFO:teuthology.orchestra.run.smithi007.stdout: "target_version": "0'0" 2024-09-18T08:20:15.356 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:20:15.357 INFO:teuthology.orchestra.run.smithi007.stdout: "last_change": "63", 2024-09-18T08:20:15.357 INFO:teuthology.orchestra.run.smithi007.stdout: "last_force_op_resend": "0", 2024-09-18T08:20:15.357 INFO:teuthology.orchestra.run.smithi007.stdout: "last_force_op_resend_prenautilus": "0", 2024-09-18T08:20:15.357 INFO:teuthology.orchestra.run.smithi007.stdout: "last_force_op_resend_preluminous": "0", 2024-09-18T08:20:15.357 INFO:teuthology.orchestra.run.smithi007.stdout: "auid": 0, 2024-09-18T08:20:15.357 INFO:teuthology.orchestra.run.smithi007.stdout: "snap_mode": "selfmanaged", 2024-09-18T08:20:15.357 INFO:teuthology.orchestra.run.smithi007.stdout: "snap_seq": 0, 2024-09-18T08:20:15.357 INFO:teuthology.orchestra.run.smithi007.stdout: "snap_epoch": 0, 2024-09-18T08:20:15.357 INFO:teuthology.orchestra.run.smithi007.stdout: "pool_snaps": [], 2024-09-18T08:20:15.357 INFO:teuthology.orchestra.run.smithi007.stdout: "removed_snaps": "[]", 2024-09-18T08:20:15.357 INFO:teuthology.orchestra.run.smithi007.stdout: "quota_max_bytes": 0, 2024-09-18T08:20:15.357 INFO:teuthology.orchestra.run.smithi007.stdout: "quota_max_objects": 0, 2024-09-18T08:20:15.357 INFO:teuthology.orchestra.run.smithi007.stdout: "tiers": [], 2024-09-18T08:20:15.357 INFO:teuthology.orchestra.run.smithi007.stdout: "tier_of": -1, 2024-09-18T08:20:15.358 INFO:teuthology.orchestra.run.smithi007.stdout: "read_tier": -1, 2024-09-18T08:20:15.358 INFO:teuthology.orchestra.run.smithi007.stdout: "write_tier": -1, 2024-09-18T08:20:15.358 INFO:teuthology.orchestra.run.smithi007.stdout: "cache_mode": "none", 2024-09-18T08:20:15.358 INFO:teuthology.orchestra.run.smithi007.stdout: "target_max_bytes": 0, 2024-09-18T08:20:15.358 INFO:teuthology.orchestra.run.smithi007.stdout: "target_max_objects": 0, 2024-09-18T08:20:15.358 INFO:teuthology.orchestra.run.smithi007.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-09-18T08:20:15.358 INFO:teuthology.orchestra.run.smithi007.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-09-18T08:20:15.358 INFO:teuthology.orchestra.run.smithi007.stdout: "cache_target_full_ratio_micro": 800000, 2024-09-18T08:20:15.358 INFO:teuthology.orchestra.run.smithi007.stdout: "cache_min_flush_age": 0, 2024-09-18T08:20:15.358 INFO:teuthology.orchestra.run.smithi007.stdout: "cache_min_evict_age": 0, 2024-09-18T08:20:15.358 INFO:teuthology.orchestra.run.smithi007.stdout: "erasure_code_profile": "", 2024-09-18T08:20:15.358 INFO:teuthology.orchestra.run.smithi007.stdout: "hit_set_params": { 2024-09-18T08:20:15.359 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "none" 2024-09-18T08:20:15.359 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:20:15.359 INFO:teuthology.orchestra.run.smithi007.stdout: "hit_set_period": 0, 2024-09-18T08:20:15.359 INFO:teuthology.orchestra.run.smithi007.stdout: "hit_set_count": 0, 2024-09-18T08:20:15.359 INFO:teuthology.orchestra.run.smithi007.stdout: "use_gmt_hitset": true, 2024-09-18T08:20:15.359 INFO:teuthology.orchestra.run.smithi007.stdout: "min_read_recency_for_promote": 0, 2024-09-18T08:20:15.359 INFO:teuthology.orchestra.run.smithi007.stdout: "min_write_recency_for_promote": 0, 2024-09-18T08:20:15.359 INFO:teuthology.orchestra.run.smithi007.stdout: "hit_set_grade_decay_rate": 0, 2024-09-18T08:20:15.359 INFO:teuthology.orchestra.run.smithi007.stdout: "hit_set_search_last_n": 0, 2024-09-18T08:20:15.359 INFO:teuthology.orchestra.run.smithi007.stdout: "grade_table": [], 2024-09-18T08:20:15.359 INFO:teuthology.orchestra.run.smithi007.stdout: "stripe_width": 0, 2024-09-18T08:20:15.359 INFO:teuthology.orchestra.run.smithi007.stdout: "expected_num_objects": 0, 2024-09-18T08:20:15.360 INFO:teuthology.orchestra.run.smithi007.stdout: "fast_read": false, 2024-09-18T08:20:15.360 INFO:teuthology.orchestra.run.smithi007.stdout: "options": {}, 2024-09-18T08:20:15.360 INFO:teuthology.orchestra.run.smithi007.stdout: "application_metadata": { 2024-09-18T08:20:15.360 INFO:teuthology.orchestra.run.smithi007.stdout: "rgw": {} 2024-09-18T08:20:15.360 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:20:15.360 INFO:teuthology.orchestra.run.smithi007.stdout: "read_balance": { 2024-09-18T08:20:15.360 INFO:teuthology.orchestra.run.smithi007.stdout: "score_type": "Fair distribution", 2024-09-18T08:20:15.360 INFO:teuthology.orchestra.run.smithi007.stdout: "score_acting": 1.9700000286102295, 2024-09-18T08:20:15.360 INFO:teuthology.orchestra.run.smithi007.stdout: "score_stable": 1.9700000286102295, 2024-09-18T08:20:15.360 INFO:teuthology.orchestra.run.smithi007.stdout: "optimal_score": 1, 2024-09-18T08:20:15.360 INFO:teuthology.orchestra.run.smithi007.stdout: "raw_score_acting": 1.9700000286102295, 2024-09-18T08:20:15.360 INFO:teuthology.orchestra.run.smithi007.stdout: "raw_score_stable": 1.9700000286102295, 2024-09-18T08:20:15.360 INFO:teuthology.orchestra.run.smithi007.stdout: "primary_affinity_weighted": 0.80927085876464844, 2024-09-18T08:20:15.361 INFO:teuthology.orchestra.run.smithi007.stdout: "average_primary_affinity": 0.92000001668930054, 2024-09-18T08:20:15.361 INFO:teuthology.orchestra.run.smithi007.stdout: "average_primary_affinity_weighted": 0.81000000238418579 2024-09-18T08:20:15.361 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-09-18T08:20:15.361 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:20:15.361 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:20:15.361 INFO:teuthology.orchestra.run.smithi007.stdout: "pool": 4, 2024-09-18T08:20:15.361 INFO:teuthology.orchestra.run.smithi007.stdout: "pool_name": "default.rgw.log", 2024-09-18T08:20:15.361 INFO:teuthology.orchestra.run.smithi007.stdout: "create_time": "2024-09-18T08:02:47.948552+0000", 2024-09-18T08:20:15.361 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 1, 2024-09-18T08:20:15.361 INFO:teuthology.orchestra.run.smithi007.stdout: "flags_names": "hashpspool", 2024-09-18T08:20:15.361 INFO:teuthology.orchestra.run.smithi007.stdout: "type": 1, 2024-09-18T08:20:15.361 INFO:teuthology.orchestra.run.smithi007.stdout: "size": 3, 2024-09-18T08:20:15.361 INFO:teuthology.orchestra.run.smithi007.stdout: "min_size": 2, 2024-09-18T08:20:15.361 INFO:teuthology.orchestra.run.smithi007.stdout: "crush_rule": 0, 2024-09-18T08:20:15.362 INFO:teuthology.orchestra.run.smithi007.stdout: "peering_crush_bucket_count": 0, 2024-09-18T08:20:15.362 INFO:teuthology.orchestra.run.smithi007.stdout: "peering_crush_bucket_target": 0, 2024-09-18T08:20:15.362 INFO:teuthology.orchestra.run.smithi007.stdout: "peering_crush_bucket_barrier": 0, 2024-09-18T08:20:15.362 INFO:teuthology.orchestra.run.smithi007.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-09-18T08:20:15.362 INFO:teuthology.orchestra.run.smithi007.stdout: "is_stretch_pool": false, 2024-09-18T08:20:15.362 INFO:teuthology.orchestra.run.smithi007.stdout: "object_hash": 2, 2024-09-18T08:20:15.362 INFO:teuthology.orchestra.run.smithi007.stdout: "pg_autoscale_mode": "off", 2024-09-18T08:20:15.362 INFO:teuthology.orchestra.run.smithi007.stdout: "pg_num": 32, 2024-09-18T08:20:15.362 INFO:teuthology.orchestra.run.smithi007.stdout: "pg_placement_num": 32, 2024-09-18T08:20:15.362 INFO:teuthology.orchestra.run.smithi007.stdout: "pg_placement_num_target": 32, 2024-09-18T08:20:15.362 INFO:teuthology.orchestra.run.smithi007.stdout: "pg_num_target": 32, 2024-09-18T08:20:15.362 INFO:teuthology.orchestra.run.smithi007.stdout: "pg_num_pending": 32, 2024-09-18T08:20:15.362 INFO:teuthology.orchestra.run.smithi007.stdout: "last_pg_merge_meta": { 2024-09-18T08:20:15.363 INFO:teuthology.orchestra.run.smithi007.stdout: "source_pgid": "0.0", 2024-09-18T08:20:15.363 INFO:teuthology.orchestra.run.smithi007.stdout: "ready_epoch": 0, 2024-09-18T08:20:15.363 INFO:teuthology.orchestra.run.smithi007.stdout: "last_epoch_started": 0, 2024-09-18T08:20:15.363 INFO:teuthology.orchestra.run.smithi007.stdout: "last_epoch_clean": 0, 2024-09-18T08:20:15.363 INFO:teuthology.orchestra.run.smithi007.stdout: "source_version": "0'0", 2024-09-18T08:20:15.363 INFO:teuthology.orchestra.run.smithi007.stdout: "target_version": "0'0" 2024-09-18T08:20:15.363 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:20:15.363 INFO:teuthology.orchestra.run.smithi007.stdout: "last_change": "65", 2024-09-18T08:20:15.363 INFO:teuthology.orchestra.run.smithi007.stdout: "last_force_op_resend": "0", 2024-09-18T08:20:15.363 INFO:teuthology.orchestra.run.smithi007.stdout: "last_force_op_resend_prenautilus": "0", 2024-09-18T08:20:15.363 INFO:teuthology.orchestra.run.smithi007.stdout: "last_force_op_resend_preluminous": "0", 2024-09-18T08:20:15.363 INFO:teuthology.orchestra.run.smithi007.stdout: "auid": 0, 2024-09-18T08:20:15.363 INFO:teuthology.orchestra.run.smithi007.stdout: "snap_mode": "selfmanaged", 2024-09-18T08:20:15.364 INFO:teuthology.orchestra.run.smithi007.stdout: "snap_seq": 0, 2024-09-18T08:20:15.364 INFO:teuthology.orchestra.run.smithi007.stdout: "snap_epoch": 0, 2024-09-18T08:20:15.364 INFO:teuthology.orchestra.run.smithi007.stdout: "pool_snaps": [], 2024-09-18T08:20:15.364 INFO:teuthology.orchestra.run.smithi007.stdout: "removed_snaps": "[]", 2024-09-18T08:20:15.364 INFO:teuthology.orchestra.run.smithi007.stdout: "quota_max_bytes": 0, 2024-09-18T08:20:15.364 INFO:teuthology.orchestra.run.smithi007.stdout: "quota_max_objects": 0, 2024-09-18T08:20:15.364 INFO:teuthology.orchestra.run.smithi007.stdout: "tiers": [], 2024-09-18T08:20:15.364 INFO:teuthology.orchestra.run.smithi007.stdout: "tier_of": -1, 2024-09-18T08:20:15.364 INFO:teuthology.orchestra.run.smithi007.stdout: "read_tier": -1, 2024-09-18T08:20:15.365 INFO:teuthology.orchestra.run.smithi007.stdout: "write_tier": -1, 2024-09-18T08:20:15.365 INFO:teuthology.orchestra.run.smithi007.stdout: "cache_mode": "none", 2024-09-18T08:20:15.365 INFO:teuthology.orchestra.run.smithi007.stdout: "target_max_bytes": 0, 2024-09-18T08:20:15.365 INFO:teuthology.orchestra.run.smithi007.stdout: "target_max_objects": 0, 2024-09-18T08:20:15.365 INFO:teuthology.orchestra.run.smithi007.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-09-18T08:20:15.365 INFO:teuthology.orchestra.run.smithi007.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-09-18T08:20:15.365 INFO:teuthology.orchestra.run.smithi007.stdout: "cache_target_full_ratio_micro": 800000, 2024-09-18T08:20:15.365 INFO:teuthology.orchestra.run.smithi007.stdout: "cache_min_flush_age": 0, 2024-09-18T08:20:15.365 INFO:teuthology.orchestra.run.smithi007.stdout: "cache_min_evict_age": 0, 2024-09-18T08:20:15.365 INFO:teuthology.orchestra.run.smithi007.stdout: "erasure_code_profile": "", 2024-09-18T08:20:15.365 INFO:teuthology.orchestra.run.smithi007.stdout: "hit_set_params": { 2024-09-18T08:20:15.365 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "none" 2024-09-18T08:20:15.365 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:20:15.366 INFO:teuthology.orchestra.run.smithi007.stdout: "hit_set_period": 0, 2024-09-18T08:20:15.366 INFO:teuthology.orchestra.run.smithi007.stdout: "hit_set_count": 0, 2024-09-18T08:20:15.366 INFO:teuthology.orchestra.run.smithi007.stdout: "use_gmt_hitset": true, 2024-09-18T08:20:15.366 INFO:teuthology.orchestra.run.smithi007.stdout: "min_read_recency_for_promote": 0, 2024-09-18T08:20:15.366 INFO:teuthology.orchestra.run.smithi007.stdout: "min_write_recency_for_promote": 0, 2024-09-18T08:20:15.366 INFO:teuthology.orchestra.run.smithi007.stdout: "hit_set_grade_decay_rate": 0, 2024-09-18T08:20:15.366 INFO:teuthology.orchestra.run.smithi007.stdout: "hit_set_search_last_n": 0, 2024-09-18T08:20:15.366 INFO:teuthology.orchestra.run.smithi007.stdout: "grade_table": [], 2024-09-18T08:20:15.366 INFO:teuthology.orchestra.run.smithi007.stdout: "stripe_width": 0, 2024-09-18T08:20:15.366 INFO:teuthology.orchestra.run.smithi007.stdout: "expected_num_objects": 0, 2024-09-18T08:20:15.366 INFO:teuthology.orchestra.run.smithi007.stdout: "fast_read": false, 2024-09-18T08:20:15.366 INFO:teuthology.orchestra.run.smithi007.stdout: "options": {}, 2024-09-18T08:20:15.366 INFO:teuthology.orchestra.run.smithi007.stdout: "application_metadata": { 2024-09-18T08:20:15.367 INFO:teuthology.orchestra.run.smithi007.stdout: "rgw": {} 2024-09-18T08:20:15.367 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:20:15.367 INFO:teuthology.orchestra.run.smithi007.stdout: "read_balance": { 2024-09-18T08:20:15.367 INFO:teuthology.orchestra.run.smithi007.stdout: "score_type": "Fair distribution", 2024-09-18T08:20:15.367 INFO:teuthology.orchestra.run.smithi007.stdout: "score_acting": 2.190000057220459, 2024-09-18T08:20:15.367 INFO:teuthology.orchestra.run.smithi007.stdout: "score_stable": 2.190000057220459, 2024-09-18T08:20:15.367 INFO:teuthology.orchestra.run.smithi007.stdout: "optimal_score": 1, 2024-09-18T08:20:15.367 INFO:teuthology.orchestra.run.smithi007.stdout: "raw_score_acting": 2.190000057220459, 2024-09-18T08:20:15.367 INFO:teuthology.orchestra.run.smithi007.stdout: "raw_score_stable": 2.190000057220459, 2024-09-18T08:20:15.367 INFO:teuthology.orchestra.run.smithi007.stdout: "primary_affinity_weighted": 0.80927085876464844, 2024-09-18T08:20:15.367 INFO:teuthology.orchestra.run.smithi007.stdout: "average_primary_affinity": 0.92000001668930054, 2024-09-18T08:20:15.367 INFO:teuthology.orchestra.run.smithi007.stdout: "average_primary_affinity_weighted": 0.81000000238418579 2024-09-18T08:20:15.367 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-09-18T08:20:15.367 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:20:15.368 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:20:15.368 INFO:teuthology.orchestra.run.smithi007.stdout: "pool": 5, 2024-09-18T08:20:15.368 INFO:teuthology.orchestra.run.smithi007.stdout: "pool_name": "default.rgw.control", 2024-09-18T08:20:15.368 INFO:teuthology.orchestra.run.smithi007.stdout: "create_time": "2024-09-18T08:02:49.904302+0000", 2024-09-18T08:20:15.368 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 1, 2024-09-18T08:20:15.368 INFO:teuthology.orchestra.run.smithi007.stdout: "flags_names": "hashpspool", 2024-09-18T08:20:15.368 INFO:teuthology.orchestra.run.smithi007.stdout: "type": 1, 2024-09-18T08:20:15.368 INFO:teuthology.orchestra.run.smithi007.stdout: "size": 3, 2024-09-18T08:20:15.368 INFO:teuthology.orchestra.run.smithi007.stdout: "min_size": 2, 2024-09-18T08:20:15.368 INFO:teuthology.orchestra.run.smithi007.stdout: "crush_rule": 0, 2024-09-18T08:20:15.368 INFO:teuthology.orchestra.run.smithi007.stdout: "peering_crush_bucket_count": 0, 2024-09-18T08:20:15.368 INFO:teuthology.orchestra.run.smithi007.stdout: "peering_crush_bucket_target": 0, 2024-09-18T08:20:15.368 INFO:teuthology.orchestra.run.smithi007.stdout: "peering_crush_bucket_barrier": 0, 2024-09-18T08:20:15.369 INFO:teuthology.orchestra.run.smithi007.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-09-18T08:20:15.369 INFO:teuthology.orchestra.run.smithi007.stdout: "is_stretch_pool": false, 2024-09-18T08:20:15.369 INFO:teuthology.orchestra.run.smithi007.stdout: "object_hash": 2, 2024-09-18T08:20:15.369 INFO:teuthology.orchestra.run.smithi007.stdout: "pg_autoscale_mode": "off", 2024-09-18T08:20:15.369 INFO:teuthology.orchestra.run.smithi007.stdout: "pg_num": 32, 2024-09-18T08:20:15.369 INFO:teuthology.orchestra.run.smithi007.stdout: "pg_placement_num": 32, 2024-09-18T08:20:15.369 INFO:teuthology.orchestra.run.smithi007.stdout: "pg_placement_num_target": 32, 2024-09-18T08:20:15.369 INFO:teuthology.orchestra.run.smithi007.stdout: "pg_num_target": 32, 2024-09-18T08:20:15.369 INFO:teuthology.orchestra.run.smithi007.stdout: "pg_num_pending": 32, 2024-09-18T08:20:15.369 INFO:teuthology.orchestra.run.smithi007.stdout: "last_pg_merge_meta": { 2024-09-18T08:20:15.370 INFO:teuthology.orchestra.run.smithi007.stdout: "source_pgid": "0.0", 2024-09-18T08:20:15.370 INFO:teuthology.orchestra.run.smithi007.stdout: "ready_epoch": 0, 2024-09-18T08:20:15.370 INFO:teuthology.orchestra.run.smithi007.stdout: "last_epoch_started": 0, 2024-09-18T08:20:15.370 INFO:teuthology.orchestra.run.smithi007.stdout: "last_epoch_clean": 0, 2024-09-18T08:20:15.370 INFO:teuthology.orchestra.run.smithi007.stdout: "source_version": "0'0", 2024-09-18T08:20:15.370 INFO:teuthology.orchestra.run.smithi007.stdout: "target_version": "0'0" 2024-09-18T08:20:15.370 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:20:15.370 INFO:teuthology.orchestra.run.smithi007.stdout: "last_change": "67", 2024-09-18T08:20:15.370 INFO:teuthology.orchestra.run.smithi007.stdout: "last_force_op_resend": "0", 2024-09-18T08:20:15.370 INFO:teuthology.orchestra.run.smithi007.stdout: "last_force_op_resend_prenautilus": "0", 2024-09-18T08:20:15.370 INFO:teuthology.orchestra.run.smithi007.stdout: "last_force_op_resend_preluminous": "0", 2024-09-18T08:20:15.370 INFO:teuthology.orchestra.run.smithi007.stdout: "auid": 0, 2024-09-18T08:20:15.370 INFO:teuthology.orchestra.run.smithi007.stdout: "snap_mode": "selfmanaged", 2024-09-18T08:20:15.371 INFO:teuthology.orchestra.run.smithi007.stdout: "snap_seq": 0, 2024-09-18T08:20:15.371 INFO:teuthology.orchestra.run.smithi007.stdout: "snap_epoch": 0, 2024-09-18T08:20:15.371 INFO:teuthology.orchestra.run.smithi007.stdout: "pool_snaps": [], 2024-09-18T08:20:15.371 INFO:teuthology.orchestra.run.smithi007.stdout: "removed_snaps": "[]", 2024-09-18T08:20:15.371 INFO:teuthology.orchestra.run.smithi007.stdout: "quota_max_bytes": 0, 2024-09-18T08:20:15.371 INFO:teuthology.orchestra.run.smithi007.stdout: "quota_max_objects": 0, 2024-09-18T08:20:15.371 INFO:teuthology.orchestra.run.smithi007.stdout: "tiers": [], 2024-09-18T08:20:15.371 INFO:teuthology.orchestra.run.smithi007.stdout: "tier_of": -1, 2024-09-18T08:20:15.371 INFO:teuthology.orchestra.run.smithi007.stdout: "read_tier": -1, 2024-09-18T08:20:15.371 INFO:teuthology.orchestra.run.smithi007.stdout: "write_tier": -1, 2024-09-18T08:20:15.371 INFO:teuthology.orchestra.run.smithi007.stdout: "cache_mode": "none", 2024-09-18T08:20:15.371 INFO:teuthology.orchestra.run.smithi007.stdout: "target_max_bytes": 0, 2024-09-18T08:20:15.371 INFO:teuthology.orchestra.run.smithi007.stdout: "target_max_objects": 0, 2024-09-18T08:20:15.371 INFO:teuthology.orchestra.run.smithi007.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-09-18T08:20:15.372 INFO:teuthology.orchestra.run.smithi007.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-09-18T08:20:15.372 INFO:teuthology.orchestra.run.smithi007.stdout: "cache_target_full_ratio_micro": 800000, 2024-09-18T08:20:15.372 INFO:teuthology.orchestra.run.smithi007.stdout: "cache_min_flush_age": 0, 2024-09-18T08:20:15.372 INFO:teuthology.orchestra.run.smithi007.stdout: "cache_min_evict_age": 0, 2024-09-18T08:20:15.372 INFO:teuthology.orchestra.run.smithi007.stdout: "erasure_code_profile": "", 2024-09-18T08:20:15.372 INFO:teuthology.orchestra.run.smithi007.stdout: "hit_set_params": { 2024-09-18T08:20:15.372 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "none" 2024-09-18T08:20:15.372 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:20:15.372 INFO:teuthology.orchestra.run.smithi007.stdout: "hit_set_period": 0, 2024-09-18T08:20:15.372 INFO:teuthology.orchestra.run.smithi007.stdout: "hit_set_count": 0, 2024-09-18T08:20:15.372 INFO:teuthology.orchestra.run.smithi007.stdout: "use_gmt_hitset": true, 2024-09-18T08:20:15.372 INFO:teuthology.orchestra.run.smithi007.stdout: "min_read_recency_for_promote": 0, 2024-09-18T08:20:15.372 INFO:teuthology.orchestra.run.smithi007.stdout: "min_write_recency_for_promote": 0, 2024-09-18T08:20:15.373 INFO:teuthology.orchestra.run.smithi007.stdout: "hit_set_grade_decay_rate": 0, 2024-09-18T08:20:15.373 INFO:teuthology.orchestra.run.smithi007.stdout: "hit_set_search_last_n": 0, 2024-09-18T08:20:15.373 INFO:teuthology.orchestra.run.smithi007.stdout: "grade_table": [], 2024-09-18T08:20:15.373 INFO:teuthology.orchestra.run.smithi007.stdout: "stripe_width": 0, 2024-09-18T08:20:15.373 INFO:teuthology.orchestra.run.smithi007.stdout: "expected_num_objects": 0, 2024-09-18T08:20:15.373 INFO:teuthology.orchestra.run.smithi007.stdout: "fast_read": false, 2024-09-18T08:20:15.373 INFO:teuthology.orchestra.run.smithi007.stdout: "options": {}, 2024-09-18T08:20:15.373 INFO:teuthology.orchestra.run.smithi007.stdout: "application_metadata": { 2024-09-18T08:20:15.373 INFO:teuthology.orchestra.run.smithi007.stdout: "rgw": {} 2024-09-18T08:20:15.373 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:20:15.373 INFO:teuthology.orchestra.run.smithi007.stdout: "read_balance": { 2024-09-18T08:20:15.373 INFO:teuthology.orchestra.run.smithi007.stdout: "score_type": "Fair distribution", 2024-09-18T08:20:15.373 INFO:teuthology.orchestra.run.smithi007.stdout: "score_acting": 1.75, 2024-09-18T08:20:15.373 INFO:teuthology.orchestra.run.smithi007.stdout: "score_stable": 1.75, 2024-09-18T08:20:15.374 INFO:teuthology.orchestra.run.smithi007.stdout: "optimal_score": 1, 2024-09-18T08:20:15.374 INFO:teuthology.orchestra.run.smithi007.stdout: "raw_score_acting": 1.75, 2024-09-18T08:20:15.374 INFO:teuthology.orchestra.run.smithi007.stdout: "raw_score_stable": 1.75, 2024-09-18T08:20:15.374 INFO:teuthology.orchestra.run.smithi007.stdout: "primary_affinity_weighted": 0.80927085876464844, 2024-09-18T08:20:15.374 INFO:teuthology.orchestra.run.smithi007.stdout: "average_primary_affinity": 0.92000001668930054, 2024-09-18T08:20:15.374 INFO:teuthology.orchestra.run.smithi007.stdout: "average_primary_affinity_weighted": 0.81000000238418579 2024-09-18T08:20:15.374 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-09-18T08:20:15.374 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:20:15.374 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:20:15.375 INFO:teuthology.orchestra.run.smithi007.stdout: "pool": 6, 2024-09-18T08:20:15.375 INFO:teuthology.orchestra.run.smithi007.stdout: "pool_name": "default.rgw.meta", 2024-09-18T08:20:15.375 INFO:teuthology.orchestra.run.smithi007.stdout: "create_time": "2024-09-18T08:02:51.971465+0000", 2024-09-18T08:20:15.375 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 1, 2024-09-18T08:20:15.375 INFO:teuthology.orchestra.run.smithi007.stdout: "flags_names": "hashpspool", 2024-09-18T08:20:15.375 INFO:teuthology.orchestra.run.smithi007.stdout: "type": 1, 2024-09-18T08:20:15.375 INFO:teuthology.orchestra.run.smithi007.stdout: "size": 3, 2024-09-18T08:20:15.375 INFO:teuthology.orchestra.run.smithi007.stdout: "min_size": 2, 2024-09-18T08:20:15.375 INFO:teuthology.orchestra.run.smithi007.stdout: "crush_rule": 0, 2024-09-18T08:20:15.375 INFO:teuthology.orchestra.run.smithi007.stdout: "peering_crush_bucket_count": 0, 2024-09-18T08:20:15.375 INFO:teuthology.orchestra.run.smithi007.stdout: "peering_crush_bucket_target": 0, 2024-09-18T08:20:15.375 INFO:teuthology.orchestra.run.smithi007.stdout: "peering_crush_bucket_barrier": 0, 2024-09-18T08:20:15.375 INFO:teuthology.orchestra.run.smithi007.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-09-18T08:20:15.376 INFO:teuthology.orchestra.run.smithi007.stdout: "is_stretch_pool": false, 2024-09-18T08:20:15.376 INFO:teuthology.orchestra.run.smithi007.stdout: "object_hash": 2, 2024-09-18T08:20:15.376 INFO:teuthology.orchestra.run.smithi007.stdout: "pg_autoscale_mode": "off", 2024-09-18T08:20:15.376 INFO:teuthology.orchestra.run.smithi007.stdout: "pg_num": 32, 2024-09-18T08:20:15.376 INFO:teuthology.orchestra.run.smithi007.stdout: "pg_placement_num": 32, 2024-09-18T08:20:15.376 INFO:teuthology.orchestra.run.smithi007.stdout: "pg_placement_num_target": 32, 2024-09-18T08:20:15.376 INFO:teuthology.orchestra.run.smithi007.stdout: "pg_num_target": 32, 2024-09-18T08:20:15.376 INFO:teuthology.orchestra.run.smithi007.stdout: "pg_num_pending": 32, 2024-09-18T08:20:15.376 INFO:teuthology.orchestra.run.smithi007.stdout: "last_pg_merge_meta": { 2024-09-18T08:20:15.376 INFO:teuthology.orchestra.run.smithi007.stdout: "source_pgid": "0.0", 2024-09-18T08:20:15.376 INFO:teuthology.orchestra.run.smithi007.stdout: "ready_epoch": 0, 2024-09-18T08:20:15.376 INFO:teuthology.orchestra.run.smithi007.stdout: "last_epoch_started": 0, 2024-09-18T08:20:15.377 INFO:teuthology.orchestra.run.smithi007.stdout: "last_epoch_clean": 0, 2024-09-18T08:20:15.377 INFO:teuthology.orchestra.run.smithi007.stdout: "source_version": "0'0", 2024-09-18T08:20:15.377 INFO:teuthology.orchestra.run.smithi007.stdout: "target_version": "0'0" 2024-09-18T08:20:15.377 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:20:15.377 INFO:teuthology.orchestra.run.smithi007.stdout: "last_change": "70", 2024-09-18T08:20:15.377 INFO:teuthology.orchestra.run.smithi007.stdout: "last_force_op_resend": "0", 2024-09-18T08:20:15.377 INFO:teuthology.orchestra.run.smithi007.stdout: "last_force_op_resend_prenautilus": "0", 2024-09-18T08:20:15.377 INFO:teuthology.orchestra.run.smithi007.stdout: "last_force_op_resend_preluminous": "0", 2024-09-18T08:20:15.377 INFO:teuthology.orchestra.run.smithi007.stdout: "auid": 0, 2024-09-18T08:20:15.377 INFO:teuthology.orchestra.run.smithi007.stdout: "snap_mode": "selfmanaged", 2024-09-18T08:20:15.377 INFO:teuthology.orchestra.run.smithi007.stdout: "snap_seq": 0, 2024-09-18T08:20:15.377 INFO:teuthology.orchestra.run.smithi007.stdout: "snap_epoch": 0, 2024-09-18T08:20:15.378 INFO:teuthology.orchestra.run.smithi007.stdout: "pool_snaps": [], 2024-09-18T08:20:15.378 INFO:teuthology.orchestra.run.smithi007.stdout: "removed_snaps": "[]", 2024-09-18T08:20:15.378 INFO:teuthology.orchestra.run.smithi007.stdout: "quota_max_bytes": 0, 2024-09-18T08:20:15.378 INFO:teuthology.orchestra.run.smithi007.stdout: "quota_max_objects": 0, 2024-09-18T08:20:15.378 INFO:teuthology.orchestra.run.smithi007.stdout: "tiers": [], 2024-09-18T08:20:15.378 INFO:teuthology.orchestra.run.smithi007.stdout: "tier_of": -1, 2024-09-18T08:20:15.378 INFO:teuthology.orchestra.run.smithi007.stdout: "read_tier": -1, 2024-09-18T08:20:15.378 INFO:teuthology.orchestra.run.smithi007.stdout: "write_tier": -1, 2024-09-18T08:20:15.378 INFO:teuthology.orchestra.run.smithi007.stdout: "cache_mode": "none", 2024-09-18T08:20:15.378 INFO:teuthology.orchestra.run.smithi007.stdout: "target_max_bytes": 0, 2024-09-18T08:20:15.378 INFO:teuthology.orchestra.run.smithi007.stdout: "target_max_objects": 0, 2024-09-18T08:20:15.378 INFO:teuthology.orchestra.run.smithi007.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-09-18T08:20:15.378 INFO:teuthology.orchestra.run.smithi007.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-09-18T08:20:15.379 INFO:teuthology.orchestra.run.smithi007.stdout: "cache_target_full_ratio_micro": 800000, 2024-09-18T08:20:15.379 INFO:teuthology.orchestra.run.smithi007.stdout: "cache_min_flush_age": 0, 2024-09-18T08:20:15.379 INFO:teuthology.orchestra.run.smithi007.stdout: "cache_min_evict_age": 0, 2024-09-18T08:20:15.379 INFO:teuthology.orchestra.run.smithi007.stdout: "erasure_code_profile": "", 2024-09-18T08:20:15.379 INFO:teuthology.orchestra.run.smithi007.stdout: "hit_set_params": { 2024-09-18T08:20:15.379 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "none" 2024-09-18T08:20:15.379 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:20:15.379 INFO:teuthology.orchestra.run.smithi007.stdout: "hit_set_period": 0, 2024-09-18T08:20:15.379 INFO:teuthology.orchestra.run.smithi007.stdout: "hit_set_count": 0, 2024-09-18T08:20:15.379 INFO:teuthology.orchestra.run.smithi007.stdout: "use_gmt_hitset": true, 2024-09-18T08:20:15.379 INFO:teuthology.orchestra.run.smithi007.stdout: "min_read_recency_for_promote": 0, 2024-09-18T08:20:15.379 INFO:teuthology.orchestra.run.smithi007.stdout: "min_write_recency_for_promote": 0, 2024-09-18T08:20:15.379 INFO:teuthology.orchestra.run.smithi007.stdout: "hit_set_grade_decay_rate": 0, 2024-09-18T08:20:15.380 INFO:teuthology.orchestra.run.smithi007.stdout: "hit_set_search_last_n": 0, 2024-09-18T08:20:15.380 INFO:teuthology.orchestra.run.smithi007.stdout: "grade_table": [], 2024-09-18T08:20:15.380 INFO:teuthology.orchestra.run.smithi007.stdout: "stripe_width": 0, 2024-09-18T08:20:15.380 INFO:teuthology.orchestra.run.smithi007.stdout: "expected_num_objects": 0, 2024-09-18T08:20:15.380 INFO:teuthology.orchestra.run.smithi007.stdout: "fast_read": false, 2024-09-18T08:20:15.380 INFO:teuthology.orchestra.run.smithi007.stdout: "options": { 2024-09-18T08:20:15.380 INFO:teuthology.orchestra.run.smithi007.stdout: "pg_autoscale_bias": 4 2024-09-18T08:20:15.380 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:20:15.380 INFO:teuthology.orchestra.run.smithi007.stdout: "application_metadata": { 2024-09-18T08:20:15.380 INFO:teuthology.orchestra.run.smithi007.stdout: "rgw": {} 2024-09-18T08:20:15.380 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:20:15.380 INFO:teuthology.orchestra.run.smithi007.stdout: "read_balance": { 2024-09-18T08:20:15.380 INFO:teuthology.orchestra.run.smithi007.stdout: "score_type": "Fair distribution", 2024-09-18T08:20:15.381 INFO:teuthology.orchestra.run.smithi007.stdout: "score_acting": 2.630000114440918, 2024-09-18T08:20:15.381 INFO:teuthology.orchestra.run.smithi007.stdout: "score_stable": 2.630000114440918, 2024-09-18T08:20:15.381 INFO:teuthology.orchestra.run.smithi007.stdout: "optimal_score": 1, 2024-09-18T08:20:15.381 INFO:teuthology.orchestra.run.smithi007.stdout: "raw_score_acting": 2.630000114440918, 2024-09-18T08:20:15.381 INFO:teuthology.orchestra.run.smithi007.stdout: "raw_score_stable": 2.630000114440918, 2024-09-18T08:20:15.381 INFO:teuthology.orchestra.run.smithi007.stdout: "primary_affinity_weighted": 0.80927085876464844, 2024-09-18T08:20:15.381 INFO:teuthology.orchestra.run.smithi007.stdout: "average_primary_affinity": 0.92000001668930054, 2024-09-18T08:20:15.381 INFO:teuthology.orchestra.run.smithi007.stdout: "average_primary_affinity_weighted": 0.81000000238418579 2024-09-18T08:20:15.381 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-09-18T08:20:15.381 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:20:15.381 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:20:15.381 INFO:teuthology.orchestra.run.smithi007.stdout: "pool": 11, 2024-09-18T08:20:15.381 INFO:teuthology.orchestra.run.smithi007.stdout: "pool_name": "unique_pool_4", 2024-09-18T08:20:15.382 INFO:teuthology.orchestra.run.smithi007.stdout: "create_time": "2024-09-18T08:18:25.969782+0000", 2024-09-18T08:20:15.382 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 1, 2024-09-18T08:20:15.382 INFO:teuthology.orchestra.run.smithi007.stdout: "flags_names": "hashpspool", 2024-09-18T08:20:15.382 INFO:teuthology.orchestra.run.smithi007.stdout: "type": 1, 2024-09-18T08:20:15.382 INFO:teuthology.orchestra.run.smithi007.stdout: "size": 3, 2024-09-18T08:20:15.382 INFO:teuthology.orchestra.run.smithi007.stdout: "min_size": 2, 2024-09-18T08:20:15.382 INFO:teuthology.orchestra.run.smithi007.stdout: "crush_rule": 0, 2024-09-18T08:20:15.382 INFO:teuthology.orchestra.run.smithi007.stdout: "peering_crush_bucket_count": 0, 2024-09-18T08:20:15.382 INFO:teuthology.orchestra.run.smithi007.stdout: "peering_crush_bucket_target": 0, 2024-09-18T08:20:15.382 INFO:teuthology.orchestra.run.smithi007.stdout: "peering_crush_bucket_barrier": 0, 2024-09-18T08:20:15.382 INFO:teuthology.orchestra.run.smithi007.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-09-18T08:20:15.382 INFO:teuthology.orchestra.run.smithi007.stdout: "is_stretch_pool": false, 2024-09-18T08:20:15.382 INFO:teuthology.orchestra.run.smithi007.stdout: "object_hash": 2, 2024-09-18T08:20:15.383 INFO:teuthology.orchestra.run.smithi007.stdout: "pg_autoscale_mode": "off", 2024-09-18T08:20:15.383 INFO:teuthology.orchestra.run.smithi007.stdout: "pg_num": 16, 2024-09-18T08:20:15.383 INFO:teuthology.orchestra.run.smithi007.stdout: "pg_placement_num": 16, 2024-09-18T08:20:15.383 INFO:teuthology.orchestra.run.smithi007.stdout: "pg_placement_num_target": 16, 2024-09-18T08:20:15.383 INFO:teuthology.orchestra.run.smithi007.stdout: "pg_num_target": 16, 2024-09-18T08:20:15.383 INFO:teuthology.orchestra.run.smithi007.stdout: "pg_num_pending": 16, 2024-09-18T08:20:15.383 INFO:teuthology.orchestra.run.smithi007.stdout: "last_pg_merge_meta": { 2024-09-18T08:20:15.383 INFO:teuthology.orchestra.run.smithi007.stdout: "source_pgid": "0.0", 2024-09-18T08:20:15.383 INFO:teuthology.orchestra.run.smithi007.stdout: "ready_epoch": 0, 2024-09-18T08:20:15.383 INFO:teuthology.orchestra.run.smithi007.stdout: "last_epoch_started": 0, 2024-09-18T08:20:15.383 INFO:teuthology.orchestra.run.smithi007.stdout: "last_epoch_clean": 0, 2024-09-18T08:20:15.383 INFO:teuthology.orchestra.run.smithi007.stdout: "source_version": "0'0", 2024-09-18T08:20:15.384 INFO:teuthology.orchestra.run.smithi007.stdout: "target_version": "0'0" 2024-09-18T08:20:15.384 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:20:15.384 INFO:teuthology.orchestra.run.smithi007.stdout: "last_change": "431", 2024-09-18T08:20:15.384 INFO:teuthology.orchestra.run.smithi007.stdout: "last_force_op_resend": "0", 2024-09-18T08:20:15.384 INFO:teuthology.orchestra.run.smithi007.stdout: "last_force_op_resend_prenautilus": "0", 2024-09-18T08:20:15.384 INFO:teuthology.orchestra.run.smithi007.stdout: "last_force_op_resend_preluminous": "0", 2024-09-18T08:20:15.384 INFO:teuthology.orchestra.run.smithi007.stdout: "auid": 0, 2024-09-18T08:20:15.384 INFO:teuthology.orchestra.run.smithi007.stdout: "snap_mode": "selfmanaged", 2024-09-18T08:20:15.384 INFO:teuthology.orchestra.run.smithi007.stdout: "snap_seq": 0, 2024-09-18T08:20:15.384 INFO:teuthology.orchestra.run.smithi007.stdout: "snap_epoch": 0, 2024-09-18T08:20:15.385 INFO:teuthology.orchestra.run.smithi007.stdout: "pool_snaps": [], 2024-09-18T08:20:15.385 INFO:teuthology.orchestra.run.smithi007.stdout: "removed_snaps": "[]", 2024-09-18T08:20:15.385 INFO:teuthology.orchestra.run.smithi007.stdout: "quota_max_bytes": 0, 2024-09-18T08:20:15.385 INFO:teuthology.orchestra.run.smithi007.stdout: "quota_max_objects": 0, 2024-09-18T08:20:15.385 INFO:teuthology.orchestra.run.smithi007.stdout: "tiers": [], 2024-09-18T08:20:15.385 INFO:teuthology.orchestra.run.smithi007.stdout: "tier_of": -1, 2024-09-18T08:20:15.385 INFO:teuthology.orchestra.run.smithi007.stdout: "read_tier": -1, 2024-09-18T08:20:15.385 INFO:teuthology.orchestra.run.smithi007.stdout: "write_tier": -1, 2024-09-18T08:20:15.385 INFO:teuthology.orchestra.run.smithi007.stdout: "cache_mode": "none", 2024-09-18T08:20:15.385 INFO:teuthology.orchestra.run.smithi007.stdout: "target_max_bytes": 0, 2024-09-18T08:20:15.385 INFO:teuthology.orchestra.run.smithi007.stdout: "target_max_objects": 0, 2024-09-18T08:20:15.385 INFO:teuthology.orchestra.run.smithi007.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-09-18T08:20:15.385 INFO:teuthology.orchestra.run.smithi007.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-09-18T08:20:15.386 INFO:teuthology.orchestra.run.smithi007.stdout: "cache_target_full_ratio_micro": 800000, 2024-09-18T08:20:15.386 INFO:teuthology.orchestra.run.smithi007.stdout: "cache_min_flush_age": 0, 2024-09-18T08:20:15.386 INFO:teuthology.orchestra.run.smithi007.stdout: "cache_min_evict_age": 0, 2024-09-18T08:20:15.386 INFO:teuthology.orchestra.run.smithi007.stdout: "erasure_code_profile": "", 2024-09-18T08:20:15.386 INFO:teuthology.orchestra.run.smithi007.stdout: "hit_set_params": { 2024-09-18T08:20:15.386 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "none" 2024-09-18T08:20:15.386 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:20:15.386 INFO:teuthology.orchestra.run.smithi007.stdout: "hit_set_period": 0, 2024-09-18T08:20:15.386 INFO:teuthology.orchestra.run.smithi007.stdout: "hit_set_count": 0, 2024-09-18T08:20:15.386 INFO:teuthology.orchestra.run.smithi007.stdout: "use_gmt_hitset": true, 2024-09-18T08:20:15.386 INFO:teuthology.orchestra.run.smithi007.stdout: "min_read_recency_for_promote": 0, 2024-09-18T08:20:15.386 INFO:teuthology.orchestra.run.smithi007.stdout: "min_write_recency_for_promote": 0, 2024-09-18T08:20:15.386 INFO:teuthology.orchestra.run.smithi007.stdout: "hit_set_grade_decay_rate": 0, 2024-09-18T08:20:15.387 INFO:teuthology.orchestra.run.smithi007.stdout: "hit_set_search_last_n": 0, 2024-09-18T08:20:15.387 INFO:teuthology.orchestra.run.smithi007.stdout: "grade_table": [], 2024-09-18T08:20:15.387 INFO:teuthology.orchestra.run.smithi007.stdout: "stripe_width": 0, 2024-09-18T08:20:15.387 INFO:teuthology.orchestra.run.smithi007.stdout: "expected_num_objects": 0, 2024-09-18T08:20:15.387 INFO:teuthology.orchestra.run.smithi007.stdout: "fast_read": false, 2024-09-18T08:20:15.387 INFO:teuthology.orchestra.run.smithi007.stdout: "options": {}, 2024-09-18T08:20:15.387 INFO:teuthology.orchestra.run.smithi007.stdout: "application_metadata": { 2024-09-18T08:20:15.387 INFO:teuthology.orchestra.run.smithi007.stdout: "rados": {} 2024-09-18T08:20:15.387 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:20:15.387 INFO:teuthology.orchestra.run.smithi007.stdout: "read_balance": { 2024-09-18T08:20:15.387 INFO:teuthology.orchestra.run.smithi007.stdout: "score_type": "Fair distribution", 2024-09-18T08:20:15.387 INFO:teuthology.orchestra.run.smithi007.stdout: "score_acting": 2.619999885559082, 2024-09-18T08:20:15.387 INFO:teuthology.orchestra.run.smithi007.stdout: "score_stable": 2.619999885559082, 2024-09-18T08:20:15.388 INFO:teuthology.orchestra.run.smithi007.stdout: "optimal_score": 0.86000001430511475, 2024-09-18T08:20:15.388 INFO:teuthology.orchestra.run.smithi007.stdout: "raw_score_acting": 2.25, 2024-09-18T08:20:15.388 INFO:teuthology.orchestra.run.smithi007.stdout: "raw_score_stable": 2.25, 2024-09-18T08:20:15.388 INFO:teuthology.orchestra.run.smithi007.stdout: "primary_affinity_weighted": 0.80927085876464844, 2024-09-18T08:20:15.388 INFO:teuthology.orchestra.run.smithi007.stdout: "average_primary_affinity": 0.92000001668930054, 2024-09-18T08:20:15.388 INFO:teuthology.orchestra.run.smithi007.stdout: "average_primary_affinity_weighted": 0.81000000238418579 2024-09-18T08:20:15.388 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-09-18T08:20:15.388 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-09-18T08:20:15.388 INFO:teuthology.orchestra.run.smithi007.stdout: ], 2024-09-18T08:20:15.388 INFO:teuthology.orchestra.run.smithi007.stdout: "osds": [ 2024-09-18T08:20:15.388 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:20:15.388 INFO:teuthology.orchestra.run.smithi007.stdout: "osd": 0, 2024-09-18T08:20:15.388 INFO:teuthology.orchestra.run.smithi007.stdout: "uuid": "ae8f60df-72ac-46fb-91a1-f132b851c876", 2024-09-18T08:20:15.389 INFO:teuthology.orchestra.run.smithi007.stdout: "up": 1, 2024-09-18T08:20:15.389 INFO:teuthology.orchestra.run.smithi007.stdout: "in": 1, 2024-09-18T08:20:15.389 INFO:teuthology.orchestra.run.smithi007.stdout: "weight": 1, 2024-09-18T08:20:15.389 INFO:teuthology.orchestra.run.smithi007.stdout: "primary_affinity": 1, 2024-09-18T08:20:15.389 INFO:teuthology.orchestra.run.smithi007.stdout: "last_clean_begin": 0, 2024-09-18T08:20:15.389 INFO:teuthology.orchestra.run.smithi007.stdout: "last_clean_end": 0, 2024-09-18T08:20:15.389 INFO:teuthology.orchestra.run.smithi007.stdout: "up_from": 8, 2024-09-18T08:20:15.389 INFO:teuthology.orchestra.run.smithi007.stdout: "up_thru": 430, 2024-09-18T08:20:15.389 INFO:teuthology.orchestra.run.smithi007.stdout: "down_at": 0, 2024-09-18T08:20:15.389 INFO:teuthology.orchestra.run.smithi007.stdout: "lost_at": 0, 2024-09-18T08:20:15.389 INFO:teuthology.orchestra.run.smithi007.stdout: "public_addrs": { 2024-09-18T08:20:15.389 INFO:teuthology.orchestra.run.smithi007.stdout: "addrvec": [ 2024-09-18T08:20:15.389 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:20:15.389 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "v2", 2024-09-18T08:20:15.390 INFO:teuthology.orchestra.run.smithi007.stdout: "addr": "172.21.15.7:6801", 2024-09-18T08:20:15.390 INFO:teuthology.orchestra.run.smithi007.stdout: "nonce": 2896425794 2024-09-18T08:20:15.390 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-09-18T08:20:15.390 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-09-18T08:20:15.390 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:20:15.390 INFO:teuthology.orchestra.run.smithi007.stdout: "cluster_addrs": { 2024-09-18T08:20:15.390 INFO:teuthology.orchestra.run.smithi007.stdout: "addrvec": [ 2024-09-18T08:20:15.390 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:20:15.390 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "v2", 2024-09-18T08:20:15.390 INFO:teuthology.orchestra.run.smithi007.stdout: "addr": "172.21.15.7:6802", 2024-09-18T08:20:15.390 INFO:teuthology.orchestra.run.smithi007.stdout: "nonce": 2896425794 2024-09-18T08:20:15.390 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-09-18T08:20:15.390 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-09-18T08:20:15.390 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:20:15.391 INFO:teuthology.orchestra.run.smithi007.stdout: "heartbeat_back_addrs": { 2024-09-18T08:20:15.391 INFO:teuthology.orchestra.run.smithi007.stdout: "addrvec": [ 2024-09-18T08:20:15.391 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:20:15.391 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "v2", 2024-09-18T08:20:15.391 INFO:teuthology.orchestra.run.smithi007.stdout: "addr": "172.21.15.7:6804", 2024-09-18T08:20:15.391 INFO:teuthology.orchestra.run.smithi007.stdout: "nonce": 2896425794 2024-09-18T08:20:15.391 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-09-18T08:20:15.391 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-09-18T08:20:15.391 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:20:15.391 INFO:teuthology.orchestra.run.smithi007.stdout: "heartbeat_front_addrs": { 2024-09-18T08:20:15.391 INFO:teuthology.orchestra.run.smithi007.stdout: "addrvec": [ 2024-09-18T08:20:15.391 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:20:15.391 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "v2", 2024-09-18T08:20:15.392 INFO:teuthology.orchestra.run.smithi007.stdout: "addr": "172.21.15.7:6803", 2024-09-18T08:20:15.392 INFO:teuthology.orchestra.run.smithi007.stdout: "nonce": 2896425794 2024-09-18T08:20:15.392 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-09-18T08:20:15.392 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-09-18T08:20:15.392 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:20:15.392 INFO:teuthology.orchestra.run.smithi007.stdout: "public_addr": "172.21.15.7:6801/2896425794", 2024-09-18T08:20:15.392 INFO:teuthology.orchestra.run.smithi007.stdout: "cluster_addr": "172.21.15.7:6802/2896425794", 2024-09-18T08:20:15.392 INFO:teuthology.orchestra.run.smithi007.stdout: "heartbeat_back_addr": "172.21.15.7:6804/2896425794", 2024-09-18T08:20:15.392 INFO:teuthology.orchestra.run.smithi007.stdout: "heartbeat_front_addr": "172.21.15.7:6803/2896425794", 2024-09-18T08:20:15.392 INFO:teuthology.orchestra.run.smithi007.stdout: "state": [ 2024-09-18T08:20:15.392 INFO:teuthology.orchestra.run.smithi007.stdout: "exists", 2024-09-18T08:20:15.392 INFO:teuthology.orchestra.run.smithi007.stdout: "up" 2024-09-18T08:20:15.393 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-09-18T08:20:15.393 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:20:15.393 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:20:15.393 INFO:teuthology.orchestra.run.smithi007.stdout: "osd": 1, 2024-09-18T08:20:15.393 INFO:teuthology.orchestra.run.smithi007.stdout: "uuid": "be28b70c-1f73-494f-ad4b-cb7ec5311999", 2024-09-18T08:20:15.393 INFO:teuthology.orchestra.run.smithi007.stdout: "up": 1, 2024-09-18T08:20:15.393 INFO:teuthology.orchestra.run.smithi007.stdout: "in": 1, 2024-09-18T08:20:15.393 INFO:teuthology.orchestra.run.smithi007.stdout: "weight": 1, 2024-09-18T08:20:15.393 INFO:teuthology.orchestra.run.smithi007.stdout: "primary_affinity": 1, 2024-09-18T08:20:15.393 INFO:teuthology.orchestra.run.smithi007.stdout: "last_clean_begin": 13, 2024-09-18T08:20:15.393 INFO:teuthology.orchestra.run.smithi007.stdout: "last_clean_end": 243, 2024-09-18T08:20:15.393 INFO:teuthology.orchestra.run.smithi007.stdout: "up_from": 307, 2024-09-18T08:20:15.393 INFO:teuthology.orchestra.run.smithi007.stdout: "up_thru": 435, 2024-09-18T08:20:15.393 INFO:teuthology.orchestra.run.smithi007.stdout: "down_at": 244, 2024-09-18T08:20:15.394 INFO:teuthology.orchestra.run.smithi007.stdout: "lost_at": 0, 2024-09-18T08:20:15.394 INFO:teuthology.orchestra.run.smithi007.stdout: "public_addrs": { 2024-09-18T08:20:15.394 INFO:teuthology.orchestra.run.smithi007.stdout: "addrvec": [ 2024-09-18T08:20:15.394 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:20:15.394 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "v2", 2024-09-18T08:20:15.394 INFO:teuthology.orchestra.run.smithi007.stdout: "addr": "172.21.15.7:6805", 2024-09-18T08:20:15.394 INFO:teuthology.orchestra.run.smithi007.stdout: "nonce": 3300107791 2024-09-18T08:20:15.394 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-09-18T08:20:15.394 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-09-18T08:20:15.394 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:20:15.394 INFO:teuthology.orchestra.run.smithi007.stdout: "cluster_addrs": { 2024-09-18T08:20:15.394 INFO:teuthology.orchestra.run.smithi007.stdout: "addrvec": [ 2024-09-18T08:20:15.394 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:20:15.395 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "v2", 2024-09-18T08:20:15.395 INFO:teuthology.orchestra.run.smithi007.stdout: "addr": "172.21.15.7:6806", 2024-09-18T08:20:15.395 INFO:teuthology.orchestra.run.smithi007.stdout: "nonce": 3300107791 2024-09-18T08:20:15.395 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-09-18T08:20:15.395 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-09-18T08:20:15.395 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:20:15.395 INFO:teuthology.orchestra.run.smithi007.stdout: "heartbeat_back_addrs": { 2024-09-18T08:20:15.395 INFO:teuthology.orchestra.run.smithi007.stdout: "addrvec": [ 2024-09-18T08:20:15.395 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:20:15.395 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "v2", 2024-09-18T08:20:15.395 INFO:teuthology.orchestra.run.smithi007.stdout: "addr": "172.21.15.7:6808", 2024-09-18T08:20:15.395 INFO:teuthology.orchestra.run.smithi007.stdout: "nonce": 3300107791 2024-09-18T08:20:15.395 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-09-18T08:20:15.396 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-09-18T08:20:15.396 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:20:15.396 INFO:teuthology.orchestra.run.smithi007.stdout: "heartbeat_front_addrs": { 2024-09-18T08:20:15.396 INFO:teuthology.orchestra.run.smithi007.stdout: "addrvec": [ 2024-09-18T08:20:15.396 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:20:15.396 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "v2", 2024-09-18T08:20:15.396 INFO:teuthology.orchestra.run.smithi007.stdout: "addr": "172.21.15.7:6807", 2024-09-18T08:20:15.396 INFO:teuthology.orchestra.run.smithi007.stdout: "nonce": 3300107791 2024-09-18T08:20:15.396 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-09-18T08:20:15.396 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-09-18T08:20:15.396 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:20:15.396 INFO:teuthology.orchestra.run.smithi007.stdout: "public_addr": "172.21.15.7:6805/3300107791", 2024-09-18T08:20:15.396 INFO:teuthology.orchestra.run.smithi007.stdout: "cluster_addr": "172.21.15.7:6806/3300107791", 2024-09-18T08:20:15.397 INFO:teuthology.orchestra.run.smithi007.stdout: "heartbeat_back_addr": "172.21.15.7:6808/3300107791", 2024-09-18T08:20:15.397 INFO:teuthology.orchestra.run.smithi007.stdout: "heartbeat_front_addr": "172.21.15.7:6807/3300107791", 2024-09-18T08:20:15.397 INFO:teuthology.orchestra.run.smithi007.stdout: "state": [ 2024-09-18T08:20:15.397 INFO:teuthology.orchestra.run.smithi007.stdout: "exists", 2024-09-18T08:20:15.397 INFO:teuthology.orchestra.run.smithi007.stdout: "up" 2024-09-18T08:20:15.397 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-09-18T08:20:15.397 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:20:15.397 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:20:15.397 INFO:teuthology.orchestra.run.smithi007.stdout: "osd": 2, 2024-09-18T08:20:15.397 INFO:teuthology.orchestra.run.smithi007.stdout: "uuid": "cfa4f083-c7fd-4743-be65-84a3040fe6dc", 2024-09-18T08:20:15.397 INFO:teuthology.orchestra.run.smithi007.stdout: "up": 1, 2024-09-18T08:20:15.397 INFO:teuthology.orchestra.run.smithi007.stdout: "in": 1, 2024-09-18T08:20:15.397 INFO:teuthology.orchestra.run.smithi007.stdout: "weight": 0.12225341796875, 2024-09-18T08:20:15.397 INFO:teuthology.orchestra.run.smithi007.stdout: "primary_affinity": 1, 2024-09-18T08:20:15.398 INFO:teuthology.orchestra.run.smithi007.stdout: "last_clean_begin": 0, 2024-09-18T08:20:15.398 INFO:teuthology.orchestra.run.smithi007.stdout: "last_clean_end": 0, 2024-09-18T08:20:15.398 INFO:teuthology.orchestra.run.smithi007.stdout: "up_from": 18, 2024-09-18T08:20:15.398 INFO:teuthology.orchestra.run.smithi007.stdout: "up_thru": 401, 2024-09-18T08:20:15.398 INFO:teuthology.orchestra.run.smithi007.stdout: "down_at": 0, 2024-09-18T08:20:15.398 INFO:teuthology.orchestra.run.smithi007.stdout: "lost_at": 0, 2024-09-18T08:20:15.398 INFO:teuthology.orchestra.run.smithi007.stdout: "public_addrs": { 2024-09-18T08:20:15.398 INFO:teuthology.orchestra.run.smithi007.stdout: "addrvec": [ 2024-09-18T08:20:15.398 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:20:15.398 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "v2", 2024-09-18T08:20:15.398 INFO:teuthology.orchestra.run.smithi007.stdout: "addr": "172.21.15.7:6809", 2024-09-18T08:20:15.398 INFO:teuthology.orchestra.run.smithi007.stdout: "nonce": 556142559 2024-09-18T08:20:15.398 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-09-18T08:20:15.399 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-09-18T08:20:15.399 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:20:15.399 INFO:teuthology.orchestra.run.smithi007.stdout: "cluster_addrs": { 2024-09-18T08:20:15.399 INFO:teuthology.orchestra.run.smithi007.stdout: "addrvec": [ 2024-09-18T08:20:15.399 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:20:15.399 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "v2", 2024-09-18T08:20:15.399 INFO:teuthology.orchestra.run.smithi007.stdout: "addr": "172.21.15.7:6810", 2024-09-18T08:20:15.399 INFO:teuthology.orchestra.run.smithi007.stdout: "nonce": 556142559 2024-09-18T08:20:15.399 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-09-18T08:20:15.399 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-09-18T08:20:15.399 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:20:15.399 INFO:teuthology.orchestra.run.smithi007.stdout: "heartbeat_back_addrs": { 2024-09-18T08:20:15.399 INFO:teuthology.orchestra.run.smithi007.stdout: "addrvec": [ 2024-09-18T08:20:15.399 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:20:15.400 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "v2", 2024-09-18T08:20:15.400 INFO:teuthology.orchestra.run.smithi007.stdout: "addr": "172.21.15.7:6812", 2024-09-18T08:20:15.400 INFO:teuthology.orchestra.run.smithi007.stdout: "nonce": 556142559 2024-09-18T08:20:15.400 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-09-18T08:20:15.400 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-09-18T08:20:15.400 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:20:15.400 INFO:teuthology.orchestra.run.smithi007.stdout: "heartbeat_front_addrs": { 2024-09-18T08:20:15.400 INFO:teuthology.orchestra.run.smithi007.stdout: "addrvec": [ 2024-09-18T08:20:15.400 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:20:15.400 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "v2", 2024-09-18T08:20:15.400 INFO:teuthology.orchestra.run.smithi007.stdout: "addr": "172.21.15.7:6811", 2024-09-18T08:20:15.400 INFO:teuthology.orchestra.run.smithi007.stdout: "nonce": 556142559 2024-09-18T08:20:15.400 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-09-18T08:20:15.401 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-09-18T08:20:15.401 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:20:15.401 INFO:teuthology.orchestra.run.smithi007.stdout: "public_addr": "172.21.15.7:6809/556142559", 2024-09-18T08:20:15.401 INFO:teuthology.orchestra.run.smithi007.stdout: "cluster_addr": "172.21.15.7:6810/556142559", 2024-09-18T08:20:15.401 INFO:teuthology.orchestra.run.smithi007.stdout: "heartbeat_back_addr": "172.21.15.7:6812/556142559", 2024-09-18T08:20:15.401 INFO:teuthology.orchestra.run.smithi007.stdout: "heartbeat_front_addr": "172.21.15.7:6811/556142559", 2024-09-18T08:20:15.401 INFO:teuthology.orchestra.run.smithi007.stdout: "state": [ 2024-09-18T08:20:15.401 INFO:teuthology.orchestra.run.smithi007.stdout: "exists", 2024-09-18T08:20:15.401 INFO:teuthology.orchestra.run.smithi007.stdout: "up" 2024-09-18T08:20:15.401 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-09-18T08:20:15.401 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:20:15.401 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:20:15.401 INFO:teuthology.orchestra.run.smithi007.stdout: "osd": 3, 2024-09-18T08:20:15.401 INFO:teuthology.orchestra.run.smithi007.stdout: "uuid": "11ecd7e1-a751-4b75-95f5-00b32d2a5bc4", 2024-09-18T08:20:15.402 INFO:teuthology.orchestra.run.smithi007.stdout: "up": 1, 2024-09-18T08:20:15.402 INFO:teuthology.orchestra.run.smithi007.stdout: "in": 1, 2024-09-18T08:20:15.402 INFO:teuthology.orchestra.run.smithi007.stdout: "weight": 1, 2024-09-18T08:20:15.402 INFO:teuthology.orchestra.run.smithi007.stdout: "primary_affinity": 1, 2024-09-18T08:20:15.402 INFO:teuthology.orchestra.run.smithi007.stdout: "last_clean_begin": 0, 2024-09-18T08:20:15.402 INFO:teuthology.orchestra.run.smithi007.stdout: "last_clean_end": 0, 2024-09-18T08:20:15.402 INFO:teuthology.orchestra.run.smithi007.stdout: "up_from": 26, 2024-09-18T08:20:15.402 INFO:teuthology.orchestra.run.smithi007.stdout: "up_thru": 431, 2024-09-18T08:20:15.402 INFO:teuthology.orchestra.run.smithi007.stdout: "down_at": 0, 2024-09-18T08:20:15.402 INFO:teuthology.orchestra.run.smithi007.stdout: "lost_at": 0, 2024-09-18T08:20:15.402 INFO:teuthology.orchestra.run.smithi007.stdout: "public_addrs": { 2024-09-18T08:20:15.402 INFO:teuthology.orchestra.run.smithi007.stdout: "addrvec": [ 2024-09-18T08:20:15.403 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:20:15.403 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "v2", 2024-09-18T08:20:15.403 INFO:teuthology.orchestra.run.smithi007.stdout: "addr": "172.21.15.7:6813", 2024-09-18T08:20:15.403 INFO:teuthology.orchestra.run.smithi007.stdout: "nonce": 163587772 2024-09-18T08:20:15.403 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-09-18T08:20:15.403 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-09-18T08:20:15.403 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:20:15.403 INFO:teuthology.orchestra.run.smithi007.stdout: "cluster_addrs": { 2024-09-18T08:20:15.403 INFO:teuthology.orchestra.run.smithi007.stdout: "addrvec": [ 2024-09-18T08:20:15.403 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:20:15.403 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "v2", 2024-09-18T08:20:15.403 INFO:teuthology.orchestra.run.smithi007.stdout: "addr": "172.21.15.7:6814", 2024-09-18T08:20:15.403 INFO:teuthology.orchestra.run.smithi007.stdout: "nonce": 163587772 2024-09-18T08:20:15.404 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-09-18T08:20:15.404 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-09-18T08:20:15.404 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:20:15.404 INFO:teuthology.orchestra.run.smithi007.stdout: "heartbeat_back_addrs": { 2024-09-18T08:20:15.404 INFO:teuthology.orchestra.run.smithi007.stdout: "addrvec": [ 2024-09-18T08:20:15.404 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:20:15.404 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "v2", 2024-09-18T08:20:15.404 INFO:teuthology.orchestra.run.smithi007.stdout: "addr": "172.21.15.7:6816", 2024-09-18T08:20:15.404 INFO:teuthology.orchestra.run.smithi007.stdout: "nonce": 163587772 2024-09-18T08:20:15.404 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-09-18T08:20:15.404 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-09-18T08:20:15.404 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:20:15.404 INFO:teuthology.orchestra.run.smithi007.stdout: "heartbeat_front_addrs": { 2024-09-18T08:20:15.405 INFO:teuthology.orchestra.run.smithi007.stdout: "addrvec": [ 2024-09-18T08:20:15.405 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:20:15.405 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "v2", 2024-09-18T08:20:15.405 INFO:teuthology.orchestra.run.smithi007.stdout: "addr": "172.21.15.7:6815", 2024-09-18T08:20:15.405 INFO:teuthology.orchestra.run.smithi007.stdout: "nonce": 163587772 2024-09-18T08:20:15.405 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-09-18T08:20:15.405 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-09-18T08:20:15.405 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:20:15.405 INFO:teuthology.orchestra.run.smithi007.stdout: "public_addr": "172.21.15.7:6813/163587772", 2024-09-18T08:20:15.405 INFO:teuthology.orchestra.run.smithi007.stdout: "cluster_addr": "172.21.15.7:6814/163587772", 2024-09-18T08:20:15.405 INFO:teuthology.orchestra.run.smithi007.stdout: "heartbeat_back_addr": "172.21.15.7:6816/163587772", 2024-09-18T08:20:15.405 INFO:teuthology.orchestra.run.smithi007.stdout: "heartbeat_front_addr": "172.21.15.7:6815/163587772", 2024-09-18T08:20:15.405 INFO:teuthology.orchestra.run.smithi007.stdout: "state": [ 2024-09-18T08:20:15.405 INFO:teuthology.orchestra.run.smithi007.stdout: "exists", 2024-09-18T08:20:15.406 INFO:teuthology.orchestra.run.smithi007.stdout: "up" 2024-09-18T08:20:15.406 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-09-18T08:20:15.406 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:20:15.406 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:20:15.406 INFO:teuthology.orchestra.run.smithi007.stdout: "osd": 4, 2024-09-18T08:20:15.406 INFO:teuthology.orchestra.run.smithi007.stdout: "uuid": "98992b34-cfa1-45bf-97a1-7d557739252f", 2024-09-18T08:20:15.406 INFO:teuthology.orchestra.run.smithi007.stdout: "up": 1, 2024-09-18T08:20:15.406 INFO:teuthology.orchestra.run.smithi007.stdout: "in": 0, 2024-09-18T08:20:15.406 INFO:teuthology.orchestra.run.smithi007.stdout: "weight": 0, 2024-09-18T08:20:15.406 INFO:teuthology.orchestra.run.smithi007.stdout: "primary_affinity": 1, 2024-09-18T08:20:15.406 INFO:teuthology.orchestra.run.smithi007.stdout: "last_clean_begin": 0, 2024-09-18T08:20:15.406 INFO:teuthology.orchestra.run.smithi007.stdout: "last_clean_end": 0, 2024-09-18T08:20:15.406 INFO:teuthology.orchestra.run.smithi007.stdout: "up_from": 32, 2024-09-18T08:20:15.407 INFO:teuthology.orchestra.run.smithi007.stdout: "up_thru": 336, 2024-09-18T08:20:15.407 INFO:teuthology.orchestra.run.smithi007.stdout: "down_at": 0, 2024-09-18T08:20:15.407 INFO:teuthology.orchestra.run.smithi007.stdout: "lost_at": 0, 2024-09-18T08:20:15.407 INFO:teuthology.orchestra.run.smithi007.stdout: "public_addrs": { 2024-09-18T08:20:15.407 INFO:teuthology.orchestra.run.smithi007.stdout: "addrvec": [ 2024-09-18T08:20:15.407 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:20:15.407 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "v2", 2024-09-18T08:20:15.407 INFO:teuthology.orchestra.run.smithi007.stdout: "addr": "172.21.15.50:6800", 2024-09-18T08:20:15.407 INFO:teuthology.orchestra.run.smithi007.stdout: "nonce": 3662119805 2024-09-18T08:20:15.407 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-09-18T08:20:15.407 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-09-18T08:20:15.407 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:20:15.407 INFO:teuthology.orchestra.run.smithi007.stdout: "cluster_addrs": { 2024-09-18T08:20:15.408 INFO:teuthology.orchestra.run.smithi007.stdout: "addrvec": [ 2024-09-18T08:20:15.408 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:20:15.408 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "v2", 2024-09-18T08:20:15.408 INFO:teuthology.orchestra.run.smithi007.stdout: "addr": "172.21.15.50:6801", 2024-09-18T08:20:15.408 INFO:teuthology.orchestra.run.smithi007.stdout: "nonce": 3662119805 2024-09-18T08:20:15.408 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-09-18T08:20:15.408 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-09-18T08:20:15.408 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:20:15.408 INFO:teuthology.orchestra.run.smithi007.stdout: "heartbeat_back_addrs": { 2024-09-18T08:20:15.408 INFO:teuthology.orchestra.run.smithi007.stdout: "addrvec": [ 2024-09-18T08:20:15.408 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:20:15.408 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "v2", 2024-09-18T08:20:15.408 INFO:teuthology.orchestra.run.smithi007.stdout: "addr": "172.21.15.50:6803", 2024-09-18T08:20:15.409 INFO:teuthology.orchestra.run.smithi007.stdout: "nonce": 3662119805 2024-09-18T08:20:15.409 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-09-18T08:20:15.409 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-09-18T08:20:15.409 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:20:15.409 INFO:teuthology.orchestra.run.smithi007.stdout: "heartbeat_front_addrs": { 2024-09-18T08:20:15.409 INFO:teuthology.orchestra.run.smithi007.stdout: "addrvec": [ 2024-09-18T08:20:15.409 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:20:15.409 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "v2", 2024-09-18T08:20:15.409 INFO:teuthology.orchestra.run.smithi007.stdout: "addr": "172.21.15.50:6802", 2024-09-18T08:20:15.409 INFO:teuthology.orchestra.run.smithi007.stdout: "nonce": 3662119805 2024-09-18T08:20:15.409 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-09-18T08:20:15.409 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-09-18T08:20:15.409 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:20:15.409 INFO:teuthology.orchestra.run.smithi007.stdout: "public_addr": "172.21.15.50:6800/3662119805", 2024-09-18T08:20:15.410 INFO:teuthology.orchestra.run.smithi007.stdout: "cluster_addr": "172.21.15.50:6801/3662119805", 2024-09-18T08:20:15.410 INFO:teuthology.orchestra.run.smithi007.stdout: "heartbeat_back_addr": "172.21.15.50:6803/3662119805", 2024-09-18T08:20:15.410 INFO:teuthology.orchestra.run.smithi007.stdout: "heartbeat_front_addr": "172.21.15.50:6802/3662119805", 2024-09-18T08:20:15.410 INFO:teuthology.orchestra.run.smithi007.stdout: "state": [ 2024-09-18T08:20:15.410 INFO:teuthology.orchestra.run.smithi007.stdout: "exists", 2024-09-18T08:20:15.410 INFO:teuthology.orchestra.run.smithi007.stdout: "up" 2024-09-18T08:20:15.410 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-09-18T08:20:15.410 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:20:15.410 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:20:15.410 INFO:teuthology.orchestra.run.smithi007.stdout: "osd": 5, 2024-09-18T08:20:15.410 INFO:teuthology.orchestra.run.smithi007.stdout: "uuid": "7f290005-ebca-467b-87e4-02ead1063610", 2024-09-18T08:20:15.410 INFO:teuthology.orchestra.run.smithi007.stdout: "up": 1, 2024-09-18T08:20:15.410 INFO:teuthology.orchestra.run.smithi007.stdout: "in": 1, 2024-09-18T08:20:15.411 INFO:teuthology.orchestra.run.smithi007.stdout: "weight": 1, 2024-09-18T08:20:15.411 INFO:teuthology.orchestra.run.smithi007.stdout: "primary_affinity": 1, 2024-09-18T08:20:15.411 INFO:teuthology.orchestra.run.smithi007.stdout: "last_clean_begin": 0, 2024-09-18T08:20:15.411 INFO:teuthology.orchestra.run.smithi007.stdout: "last_clean_end": 0, 2024-09-18T08:20:15.411 INFO:teuthology.orchestra.run.smithi007.stdout: "up_from": 39, 2024-09-18T08:20:15.411 INFO:teuthology.orchestra.run.smithi007.stdout: "up_thru": 413, 2024-09-18T08:20:15.411 INFO:teuthology.orchestra.run.smithi007.stdout: "down_at": 0, 2024-09-18T08:20:15.411 INFO:teuthology.orchestra.run.smithi007.stdout: "lost_at": 0, 2024-09-18T08:20:15.411 INFO:teuthology.orchestra.run.smithi007.stdout: "public_addrs": { 2024-09-18T08:20:15.411 INFO:teuthology.orchestra.run.smithi007.stdout: "addrvec": [ 2024-09-18T08:20:15.411 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:20:15.411 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "v2", 2024-09-18T08:20:15.411 INFO:teuthology.orchestra.run.smithi007.stdout: "addr": "172.21.15.50:6804", 2024-09-18T08:20:15.412 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid 779349ca-7593-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-18T08:20:15.413 INFO:teuthology.orchestra.run.smithi007.stdout: "nonce": 1199797954 2024-09-18T08:20:15.413 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-09-18T08:20:15.413 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-09-18T08:20:15.413 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:20:15.413 INFO:teuthology.orchestra.run.smithi007.stdout: "cluster_addrs": { 2024-09-18T08:20:15.413 INFO:teuthology.orchestra.run.smithi007.stdout: "addrvec": [ 2024-09-18T08:20:15.413 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:20:15.413 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "v2", 2024-09-18T08:20:15.414 INFO:teuthology.orchestra.run.smithi007.stdout: "addr": "172.21.15.50:6805", 2024-09-18T08:20:15.414 INFO:teuthology.orchestra.run.smithi007.stdout: "nonce": 1199797954 2024-09-18T08:20:15.414 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-09-18T08:20:15.414 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-09-18T08:20:15.414 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:20:15.414 INFO:teuthology.orchestra.run.smithi007.stdout: "heartbeat_back_addrs": { 2024-09-18T08:20:15.414 INFO:teuthology.orchestra.run.smithi007.stdout: "addrvec": [ 2024-09-18T08:20:15.414 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:20:15.414 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "v2", 2024-09-18T08:20:15.414 INFO:teuthology.orchestra.run.smithi007.stdout: "addr": "172.21.15.50:6807", 2024-09-18T08:20:15.414 INFO:teuthology.orchestra.run.smithi007.stdout: "nonce": 1199797954 2024-09-18T08:20:15.414 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-09-18T08:20:15.414 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-09-18T08:20:15.415 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:20:15.415 INFO:teuthology.orchestra.run.smithi007.stdout: "heartbeat_front_addrs": { 2024-09-18T08:20:15.415 INFO:teuthology.orchestra.run.smithi007.stdout: "addrvec": [ 2024-09-18T08:20:15.415 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:20:15.415 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "v2", 2024-09-18T08:20:15.415 INFO:teuthology.orchestra.run.smithi007.stdout: "addr": "172.21.15.50:6806", 2024-09-18T08:20:15.415 INFO:teuthology.orchestra.run.smithi007.stdout: "nonce": 1199797954 2024-09-18T08:20:15.415 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-09-18T08:20:15.415 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-09-18T08:20:15.415 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:20:15.415 INFO:teuthology.orchestra.run.smithi007.stdout: "public_addr": "172.21.15.50:6804/1199797954", 2024-09-18T08:20:15.415 INFO:teuthology.orchestra.run.smithi007.stdout: "cluster_addr": "172.21.15.50:6805/1199797954", 2024-09-18T08:20:15.415 INFO:teuthology.orchestra.run.smithi007.stdout: "heartbeat_back_addr": "172.21.15.50:6807/1199797954", 2024-09-18T08:20:15.415 INFO:teuthology.orchestra.run.smithi007.stdout: "heartbeat_front_addr": "172.21.15.50:6806/1199797954", 2024-09-18T08:20:15.416 INFO:teuthology.orchestra.run.smithi007.stdout: "state": [ 2024-09-18T08:20:15.416 INFO:teuthology.orchestra.run.smithi007.stdout: "exists", 2024-09-18T08:20:15.416 INFO:teuthology.orchestra.run.smithi007.stdout: "up" 2024-09-18T08:20:15.416 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-09-18T08:20:15.416 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:20:15.416 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:20:15.416 INFO:teuthology.orchestra.run.smithi007.stdout: "osd": 6, 2024-09-18T08:20:15.416 INFO:teuthology.orchestra.run.smithi007.stdout: "uuid": "e07d846c-ae42-4fea-aa5c-3a25835d18a6", 2024-09-18T08:20:15.416 INFO:teuthology.orchestra.run.smithi007.stdout: "up": 1, 2024-09-18T08:20:15.416 INFO:teuthology.orchestra.run.smithi007.stdout: "in": 1, 2024-09-18T08:20:15.416 INFO:teuthology.orchestra.run.smithi007.stdout: "weight": 1, 2024-09-18T08:20:15.416 INFO:teuthology.orchestra.run.smithi007.stdout: "primary_affinity": 0, 2024-09-18T08:20:15.416 INFO:teuthology.orchestra.run.smithi007.stdout: "last_clean_begin": 0, 2024-09-18T08:20:15.417 INFO:teuthology.orchestra.run.smithi007.stdout: "last_clean_end": 0, 2024-09-18T08:20:15.417 INFO:teuthology.orchestra.run.smithi007.stdout: "up_from": 47, 2024-09-18T08:20:15.417 INFO:teuthology.orchestra.run.smithi007.stdout: "up_thru": 244, 2024-09-18T08:20:15.417 INFO:teuthology.orchestra.run.smithi007.stdout: "down_at": 0, 2024-09-18T08:20:15.417 INFO:teuthology.orchestra.run.smithi007.stdout: "lost_at": 0, 2024-09-18T08:20:15.417 INFO:teuthology.orchestra.run.smithi007.stdout: "public_addrs": { 2024-09-18T08:20:15.417 INFO:teuthology.orchestra.run.smithi007.stdout: "addrvec": [ 2024-09-18T08:20:15.417 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:20:15.417 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "v2", 2024-09-18T08:20:15.417 INFO:teuthology.orchestra.run.smithi007.stdout: "addr": "172.21.15.50:6808", 2024-09-18T08:20:15.417 INFO:teuthology.orchestra.run.smithi007.stdout: "nonce": 255960422 2024-09-18T08:20:15.417 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-09-18T08:20:15.417 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-09-18T08:20:15.418 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:20:15.418 INFO:teuthology.orchestra.run.smithi007.stdout: "cluster_addrs": { 2024-09-18T08:20:15.418 INFO:teuthology.orchestra.run.smithi007.stdout: "addrvec": [ 2024-09-18T08:20:15.418 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:20:15.418 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "v2", 2024-09-18T08:20:15.418 INFO:teuthology.orchestra.run.smithi007.stdout: "addr": "172.21.15.50:6809", 2024-09-18T08:20:15.418 INFO:teuthology.orchestra.run.smithi007.stdout: "nonce": 255960422 2024-09-18T08:20:15.418 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-09-18T08:20:15.418 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-09-18T08:20:15.418 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:20:15.418 INFO:teuthology.orchestra.run.smithi007.stdout: "heartbeat_back_addrs": { 2024-09-18T08:20:15.418 INFO:teuthology.orchestra.run.smithi007.stdout: "addrvec": [ 2024-09-18T08:20:15.418 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:20:15.419 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "v2", 2024-09-18T08:20:15.419 INFO:teuthology.orchestra.run.smithi007.stdout: "addr": "172.21.15.50:6811", 2024-09-18T08:20:15.419 INFO:teuthology.orchestra.run.smithi007.stdout: "nonce": 255960422 2024-09-18T08:20:15.419 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-09-18T08:20:15.419 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-09-18T08:20:15.419 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:20:15.419 INFO:teuthology.orchestra.run.smithi007.stdout: "heartbeat_front_addrs": { 2024-09-18T08:20:15.419 INFO:teuthology.orchestra.run.smithi007.stdout: "addrvec": [ 2024-09-18T08:20:15.419 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:20:15.419 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "v2", 2024-09-18T08:20:15.419 INFO:teuthology.orchestra.run.smithi007.stdout: "addr": "172.21.15.50:6810", 2024-09-18T08:20:15.419 INFO:teuthology.orchestra.run.smithi007.stdout: "nonce": 255960422 2024-09-18T08:20:15.419 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-09-18T08:20:15.420 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-09-18T08:20:15.420 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:20:15.420 INFO:teuthology.orchestra.run.smithi007.stdout: "public_addr": "172.21.15.50:6808/255960422", 2024-09-18T08:20:15.420 INFO:teuthology.orchestra.run.smithi007.stdout: "cluster_addr": "172.21.15.50:6809/255960422", 2024-09-18T08:20:15.420 INFO:teuthology.orchestra.run.smithi007.stdout: "heartbeat_back_addr": "172.21.15.50:6811/255960422", 2024-09-18T08:20:15.420 INFO:teuthology.orchestra.run.smithi007.stdout: "heartbeat_front_addr": "172.21.15.50:6810/255960422", 2024-09-18T08:20:15.420 INFO:teuthology.orchestra.run.smithi007.stdout: "state": [ 2024-09-18T08:20:15.420 INFO:teuthology.orchestra.run.smithi007.stdout: "exists", 2024-09-18T08:20:15.420 INFO:teuthology.orchestra.run.smithi007.stdout: "up" 2024-09-18T08:20:15.420 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-09-18T08:20:15.420 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:20:15.420 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:20:15.420 INFO:teuthology.orchestra.run.smithi007.stdout: "osd": 7, 2024-09-18T08:20:15.421 INFO:teuthology.orchestra.run.smithi007.stdout: "uuid": "c9dac3b7-d651-489d-b96e-5b10fa32d9af", 2024-09-18T08:20:15.421 INFO:teuthology.orchestra.run.smithi007.stdout: "up": 1, 2024-09-18T08:20:15.421 INFO:teuthology.orchestra.run.smithi007.stdout: "in": 1, 2024-09-18T08:20:15.421 INFO:teuthology.orchestra.run.smithi007.stdout: "weight": 1, 2024-09-18T08:20:15.421 INFO:teuthology.orchestra.run.smithi007.stdout: "primary_affinity": 0.4741668701171875, 2024-09-18T08:20:15.421 INFO:teuthology.orchestra.run.smithi007.stdout: "last_clean_begin": 0, 2024-09-18T08:20:15.421 INFO:teuthology.orchestra.run.smithi007.stdout: "last_clean_end": 0, 2024-09-18T08:20:15.421 INFO:teuthology.orchestra.run.smithi007.stdout: "up_from": 56, 2024-09-18T08:20:15.421 INFO:teuthology.orchestra.run.smithi007.stdout: "up_thru": 414, 2024-09-18T08:20:15.421 INFO:teuthology.orchestra.run.smithi007.stdout: "down_at": 0, 2024-09-18T08:20:15.421 INFO:teuthology.orchestra.run.smithi007.stdout: "lost_at": 0, 2024-09-18T08:20:15.421 INFO:teuthology.orchestra.run.smithi007.stdout: "public_addrs": { 2024-09-18T08:20:15.421 INFO:teuthology.orchestra.run.smithi007.stdout: "addrvec": [ 2024-09-18T08:20:15.421 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:20:15.422 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "v2", 2024-09-18T08:20:15.422 INFO:teuthology.orchestra.run.smithi007.stdout: "addr": "172.21.15.50:6812", 2024-09-18T08:20:15.422 INFO:teuthology.orchestra.run.smithi007.stdout: "nonce": 874923904 2024-09-18T08:20:15.422 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-09-18T08:20:15.422 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-09-18T08:20:15.422 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:20:15.422 INFO:teuthology.orchestra.run.smithi007.stdout: "cluster_addrs": { 2024-09-18T08:20:15.422 INFO:teuthology.orchestra.run.smithi007.stdout: "addrvec": [ 2024-09-18T08:20:15.422 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:20:15.422 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "v2", 2024-09-18T08:20:15.422 INFO:teuthology.orchestra.run.smithi007.stdout: "addr": "172.21.15.50:6813", 2024-09-18T08:20:15.422 INFO:teuthology.orchestra.run.smithi007.stdout: "nonce": 874923904 2024-09-18T08:20:15.422 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-09-18T08:20:15.423 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-09-18T08:20:15.423 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:20:15.423 INFO:teuthology.orchestra.run.smithi007.stdout: "heartbeat_back_addrs": { 2024-09-18T08:20:15.423 INFO:teuthology.orchestra.run.smithi007.stdout: "addrvec": [ 2024-09-18T08:20:15.423 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:20:15.423 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "v2", 2024-09-18T08:20:15.423 INFO:teuthology.orchestra.run.smithi007.stdout: "addr": "172.21.15.50:6815", 2024-09-18T08:20:15.423 INFO:teuthology.orchestra.run.smithi007.stdout: "nonce": 874923904 2024-09-18T08:20:15.423 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-09-18T08:20:15.423 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-09-18T08:20:15.423 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:20:15.423 INFO:teuthology.orchestra.run.smithi007.stdout: "heartbeat_front_addrs": { 2024-09-18T08:20:15.423 INFO:teuthology.orchestra.run.smithi007.stdout: "addrvec": [ 2024-09-18T08:20:15.423 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:20:15.424 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "v2", 2024-09-18T08:20:15.424 INFO:teuthology.orchestra.run.smithi007.stdout: "addr": "172.21.15.50:6814", 2024-09-18T08:20:15.424 INFO:teuthology.orchestra.run.smithi007.stdout: "nonce": 874923904 2024-09-18T08:20:15.424 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-09-18T08:20:15.424 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-09-18T08:20:15.424 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:20:15.424 INFO:teuthology.orchestra.run.smithi007.stdout: "public_addr": "172.21.15.50:6812/874923904", 2024-09-18T08:20:15.424 INFO:teuthology.orchestra.run.smithi007.stdout: "cluster_addr": "172.21.15.50:6813/874923904", 2024-09-18T08:20:15.424 INFO:teuthology.orchestra.run.smithi007.stdout: "heartbeat_back_addr": "172.21.15.50:6815/874923904", 2024-09-18T08:20:15.424 INFO:teuthology.orchestra.run.smithi007.stdout: "heartbeat_front_addr": "172.21.15.50:6814/874923904", 2024-09-18T08:20:15.424 INFO:teuthology.orchestra.run.smithi007.stdout: "state": [ 2024-09-18T08:20:15.424 INFO:teuthology.orchestra.run.smithi007.stdout: "exists", 2024-09-18T08:20:15.424 INFO:teuthology.orchestra.run.smithi007.stdout: "up" 2024-09-18T08:20:15.425 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-09-18T08:20:15.425 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-09-18T08:20:15.425 INFO:teuthology.orchestra.run.smithi007.stdout: ], 2024-09-18T08:20:15.425 INFO:teuthology.orchestra.run.smithi007.stdout: "osd_xinfo": [ 2024-09-18T08:20:15.425 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:20:15.425 INFO:teuthology.orchestra.run.smithi007.stdout: "osd": 0, 2024-09-18T08:20:15.425 INFO:teuthology.orchestra.run.smithi007.stdout: "down_stamp": "0.000000", 2024-09-18T08:20:15.425 INFO:teuthology.orchestra.run.smithi007.stdout: "laggy_probability": 0, 2024-09-18T08:20:15.425 INFO:teuthology.orchestra.run.smithi007.stdout: "laggy_interval": 0, 2024-09-18T08:20:15.425 INFO:teuthology.orchestra.run.smithi007.stdout: "features": 4540719139924082687, 2024-09-18T08:20:15.425 INFO:teuthology.orchestra.run.smithi007.stdout: "old_weight": 0, 2024-09-18T08:20:15.425 INFO:teuthology.orchestra.run.smithi007.stdout: "last_purged_snaps_scrub": "2024-09-18T08:19:03.356280+0000", 2024-09-18T08:20:15.425 INFO:teuthology.orchestra.run.smithi007.stdout: "dead_epoch": 0 2024-09-18T08:20:15.425 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:20:15.426 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:20:15.426 INFO:teuthology.orchestra.run.smithi007.stdout: "osd": 1, 2024-09-18T08:20:15.426 INFO:teuthology.orchestra.run.smithi007.stdout: "down_stamp": "0.000000", 2024-09-18T08:20:15.426 INFO:teuthology.orchestra.run.smithi007.stdout: "laggy_probability": 0, 2024-09-18T08:20:15.426 INFO:teuthology.orchestra.run.smithi007.stdout: "laggy_interval": 0, 2024-09-18T08:20:15.426 INFO:teuthology.orchestra.run.smithi007.stdout: "features": 4540719139924082687, 2024-09-18T08:20:15.426 INFO:teuthology.orchestra.run.smithi007.stdout: "old_weight": 0, 2024-09-18T08:20:15.426 INFO:teuthology.orchestra.run.smithi007.stdout: "last_purged_snaps_scrub": "2024-09-18T08:19:36.265311+0000", 2024-09-18T08:20:15.426 INFO:teuthology.orchestra.run.smithi007.stdout: "dead_epoch": 243 2024-09-18T08:20:15.426 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:20:15.426 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:20:15.426 INFO:teuthology.orchestra.run.smithi007.stdout: "osd": 2, 2024-09-18T08:20:15.426 INFO:teuthology.orchestra.run.smithi007.stdout: "down_stamp": "0.000000", 2024-09-18T08:20:15.427 INFO:teuthology.orchestra.run.smithi007.stdout: "laggy_probability": 0, 2024-09-18T08:20:15.427 INFO:teuthology.orchestra.run.smithi007.stdout: "laggy_interval": 0, 2024-09-18T08:20:15.427 INFO:teuthology.orchestra.run.smithi007.stdout: "features": 4540719139924082687, 2024-09-18T08:20:15.427 INFO:teuthology.orchestra.run.smithi007.stdout: "old_weight": 0, 2024-09-18T08:20:15.427 INFO:teuthology.orchestra.run.smithi007.stdout: "last_purged_snaps_scrub": "2024-09-18T08:19:42.729801+0000", 2024-09-18T08:20:15.427 INFO:teuthology.orchestra.run.smithi007.stdout: "dead_epoch": 0 2024-09-18T08:20:15.427 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:20:15.427 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:20:15.427 INFO:teuthology.orchestra.run.smithi007.stdout: "osd": 3, 2024-09-18T08:20:15.427 INFO:teuthology.orchestra.run.smithi007.stdout: "down_stamp": "0.000000", 2024-09-18T08:20:15.427 INFO:teuthology.orchestra.run.smithi007.stdout: "laggy_probability": 0, 2024-09-18T08:20:15.427 INFO:teuthology.orchestra.run.smithi007.stdout: "laggy_interval": 0, 2024-09-18T08:20:15.427 INFO:teuthology.orchestra.run.smithi007.stdout: "features": 4540719139924082687, 2024-09-18T08:20:15.428 INFO:teuthology.orchestra.run.smithi007.stdout: "old_weight": 0, 2024-09-18T08:20:15.428 INFO:teuthology.orchestra.run.smithi007.stdout: "last_purged_snaps_scrub": "2024-09-18T08:19:22.484015+0000", 2024-09-18T08:20:15.428 INFO:teuthology.orchestra.run.smithi007.stdout: "dead_epoch": 0 2024-09-18T08:20:15.428 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:20:15.428 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:20:15.428 INFO:teuthology.orchestra.run.smithi007.stdout: "osd": 4, 2024-09-18T08:20:15.428 INFO:teuthology.orchestra.run.smithi007.stdout: "down_stamp": "0.000000", 2024-09-18T08:20:15.428 INFO:teuthology.orchestra.run.smithi007.stdout: "laggy_probability": 0, 2024-09-18T08:20:15.428 INFO:teuthology.orchestra.run.smithi007.stdout: "laggy_interval": 0, 2024-09-18T08:20:15.428 INFO:teuthology.orchestra.run.smithi007.stdout: "features": 4540719139924082687, 2024-09-18T08:20:15.428 INFO:teuthology.orchestra.run.smithi007.stdout: "old_weight": 65536, 2024-09-18T08:20:15.428 INFO:teuthology.orchestra.run.smithi007.stdout: "last_purged_snaps_scrub": "2024-09-18T08:19:20.960569+0000", 2024-09-18T08:20:15.428 INFO:teuthology.orchestra.run.smithi007.stdout: "dead_epoch": 0 2024-09-18T08:20:15.429 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:20:15.429 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:20:15.429 INFO:teuthology.orchestra.run.smithi007.stdout: "osd": 5, 2024-09-18T08:20:15.429 INFO:teuthology.orchestra.run.smithi007.stdout: "down_stamp": "0.000000", 2024-09-18T08:20:15.429 INFO:teuthology.orchestra.run.smithi007.stdout: "laggy_probability": 0, 2024-09-18T08:20:15.429 INFO:teuthology.orchestra.run.smithi007.stdout: "laggy_interval": 0, 2024-09-18T08:20:15.429 INFO:teuthology.orchestra.run.smithi007.stdout: "features": 4540719139924082687, 2024-09-18T08:20:15.429 INFO:teuthology.orchestra.run.smithi007.stdout: "old_weight": 0, 2024-09-18T08:20:15.429 INFO:teuthology.orchestra.run.smithi007.stdout: "last_purged_snaps_scrub": "2024-09-18T08:19:43.501096+0000", 2024-09-18T08:20:15.429 INFO:teuthology.orchestra.run.smithi007.stdout: "dead_epoch": 0 2024-09-18T08:20:15.429 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:20:15.429 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:20:15.430 INFO:teuthology.orchestra.run.smithi007.stdout: "osd": 6, 2024-09-18T08:20:15.430 INFO:teuthology.orchestra.run.smithi007.stdout: "down_stamp": "0.000000", 2024-09-18T08:20:15.430 INFO:teuthology.orchestra.run.smithi007.stdout: "laggy_probability": 0, 2024-09-18T08:20:15.430 INFO:teuthology.orchestra.run.smithi007.stdout: "laggy_interval": 0, 2024-09-18T08:20:15.430 INFO:teuthology.orchestra.run.smithi007.stdout: "features": 4540719139924082687, 2024-09-18T08:20:15.430 INFO:teuthology.orchestra.run.smithi007.stdout: "old_weight": 0, 2024-09-18T08:20:15.430 INFO:teuthology.orchestra.run.smithi007.stdout: "last_purged_snaps_scrub": "2024-09-18T08:19:22.946550+0000", 2024-09-18T08:20:15.430 INFO:teuthology.orchestra.run.smithi007.stdout: "dead_epoch": 0 2024-09-18T08:20:15.430 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:20:15.430 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:20:15.431 INFO:teuthology.orchestra.run.smithi007.stdout: "osd": 7, 2024-09-18T08:20:15.431 INFO:teuthology.orchestra.run.smithi007.stdout: "down_stamp": "0.000000", 2024-09-18T08:20:15.431 INFO:teuthology.orchestra.run.smithi007.stdout: "laggy_probability": 0, 2024-09-18T08:20:15.431 INFO:teuthology.orchestra.run.smithi007.stdout: "laggy_interval": 0, 2024-09-18T08:20:15.431 INFO:teuthology.orchestra.run.smithi007.stdout: "features": 4540719139924082687, 2024-09-18T08:20:15.431 INFO:teuthology.orchestra.run.smithi007.stdout: "old_weight": 0, 2024-09-18T08:20:15.431 INFO:teuthology.orchestra.run.smithi007.stdout: "last_purged_snaps_scrub": "2024-09-18T08:19:09.632672+0000", 2024-09-18T08:20:15.431 INFO:teuthology.orchestra.run.smithi007.stdout: "dead_epoch": 0 2024-09-18T08:20:15.431 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-09-18T08:20:15.431 INFO:teuthology.orchestra.run.smithi007.stdout: ], 2024-09-18T08:20:15.431 INFO:teuthology.orchestra.run.smithi007.stdout: "pg_upmap": [ 2024-09-18T08:20:15.431 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:20:15.431 INFO:teuthology.orchestra.run.smithi007.stdout: "pgid": "3.c", 2024-09-18T08:20:15.432 INFO:teuthology.orchestra.run.smithi007.stdout: "osds": [ 2024-09-18T08:20:15.432 INFO:teuthology.orchestra.run.smithi007.stdout: 3, 2024-09-18T08:20:15.432 INFO:teuthology.orchestra.run.smithi007.stdout: 2, 2024-09-18T08:20:15.432 INFO:teuthology.orchestra.run.smithi007.stdout: 6 2024-09-18T08:20:15.432 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-09-18T08:20:15.432 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:20:15.432 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:20:15.432 INFO:teuthology.orchestra.run.smithi007.stdout: "pgid": "6.3", 2024-09-18T08:20:15.432 INFO:teuthology.orchestra.run.smithi007.stdout: "osds": [ 2024-09-18T08:20:15.432 INFO:teuthology.orchestra.run.smithi007.stdout: 5, 2024-09-18T08:20:15.432 INFO:teuthology.orchestra.run.smithi007.stdout: 3, 2024-09-18T08:20:15.432 INFO:teuthology.orchestra.run.smithi007.stdout: 2 2024-09-18T08:20:15.432 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-09-18T08:20:15.433 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-09-18T08:20:15.433 INFO:teuthology.orchestra.run.smithi007.stdout: ], 2024-09-18T08:20:15.433 INFO:teuthology.orchestra.run.smithi007.stdout: "pg_upmap_items": [ 2024-09-18T08:20:15.433 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:20:15.433 INFO:teuthology.orchestra.run.smithi007.stdout: "pgid": "3.8", 2024-09-18T08:20:15.433 INFO:teuthology.orchestra.run.smithi007.stdout: "mappings": [ 2024-09-18T08:20:15.433 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:20:15.433 INFO:teuthology.orchestra.run.smithi007.stdout: "from": 3, 2024-09-18T08:20:15.433 INFO:teuthology.orchestra.run.smithi007.stdout: "to": 2 2024-09-18T08:20:15.433 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-09-18T08:20:15.433 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-09-18T08:20:15.433 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:20:15.433 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:20:15.433 INFO:teuthology.orchestra.run.smithi007.stdout: "pgid": "4.7", 2024-09-18T08:20:15.434 INFO:teuthology.orchestra.run.smithi007.stdout: "mappings": [ 2024-09-18T08:20:15.434 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:20:15.434 INFO:teuthology.orchestra.run.smithi007.stdout: "from": 1, 2024-09-18T08:20:15.434 INFO:teuthology.orchestra.run.smithi007.stdout: "to": 2 2024-09-18T08:20:15.434 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-09-18T08:20:15.434 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-09-18T08:20:15.434 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:20:15.434 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:20:15.434 INFO:teuthology.orchestra.run.smithi007.stdout: "pgid": "4.14", 2024-09-18T08:20:15.434 INFO:teuthology.orchestra.run.smithi007.stdout: "mappings": [ 2024-09-18T08:20:15.434 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:20:15.434 INFO:teuthology.orchestra.run.smithi007.stdout: "from": 1, 2024-09-18T08:20:15.434 INFO:teuthology.orchestra.run.smithi007.stdout: "to": 2 2024-09-18T08:20:15.434 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-09-18T08:20:15.435 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-09-18T08:20:15.435 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:20:15.435 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:20:15.435 INFO:teuthology.orchestra.run.smithi007.stdout: "pgid": "4.18", 2024-09-18T08:20:15.435 INFO:teuthology.orchestra.run.smithi007.stdout: "mappings": [ 2024-09-18T08:20:15.435 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:20:15.435 INFO:teuthology.orchestra.run.smithi007.stdout: "from": 1, 2024-09-18T08:20:15.435 INFO:teuthology.orchestra.run.smithi007.stdout: "to": 2 2024-09-18T08:20:15.435 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-09-18T08:20:15.435 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-09-18T08:20:15.435 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:20:15.435 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:20:15.435 INFO:teuthology.orchestra.run.smithi007.stdout: "pgid": "4.1b", 2024-09-18T08:20:15.436 INFO:teuthology.orchestra.run.smithi007.stdout: "mappings": [ 2024-09-18T08:20:15.436 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:20:15.436 INFO:teuthology.orchestra.run.smithi007.stdout: "from": 3, 2024-09-18T08:20:15.436 INFO:teuthology.orchestra.run.smithi007.stdout: "to": 5 2024-09-18T08:20:15.436 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-09-18T08:20:15.436 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-09-18T08:20:15.436 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:20:15.436 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:20:15.436 INFO:teuthology.orchestra.run.smithi007.stdout: "pgid": "4.1d", 2024-09-18T08:20:15.436 INFO:teuthology.orchestra.run.smithi007.stdout: "mappings": [ 2024-09-18T08:20:15.436 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:20:15.436 INFO:teuthology.orchestra.run.smithi007.stdout: "from": 1, 2024-09-18T08:20:15.436 INFO:teuthology.orchestra.run.smithi007.stdout: "to": 0 2024-09-18T08:20:15.437 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-09-18T08:20:15.437 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-09-18T08:20:15.437 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:20:15.437 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:20:15.437 INFO:teuthology.orchestra.run.smithi007.stdout: "pgid": "5.b", 2024-09-18T08:20:15.437 INFO:teuthology.orchestra.run.smithi007.stdout: "mappings": [ 2024-09-18T08:20:15.437 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:20:15.437 INFO:teuthology.orchestra.run.smithi007.stdout: "from": 0, 2024-09-18T08:20:15.437 INFO:teuthology.orchestra.run.smithi007.stdout: "to": 3 2024-09-18T08:20:15.437 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-09-18T08:20:15.437 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-09-18T08:20:15.437 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:20:15.437 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:20:15.438 INFO:teuthology.orchestra.run.smithi007.stdout: "pgid": "6.8", 2024-09-18T08:20:15.438 INFO:teuthology.orchestra.run.smithi007.stdout: "mappings": [ 2024-09-18T08:20:15.438 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:20:15.438 INFO:teuthology.orchestra.run.smithi007.stdout: "from": 7, 2024-09-18T08:20:15.438 INFO:teuthology.orchestra.run.smithi007.stdout: "to": 3 2024-09-18T08:20:15.438 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-09-18T08:20:15.438 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-09-18T08:20:15.438 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-09-18T08:20:15.438 INFO:teuthology.orchestra.run.smithi007.stdout: ], 2024-09-18T08:20:15.438 INFO:teuthology.orchestra.run.smithi007.stdout: "pg_upmap_primaries": [], 2024-09-18T08:20:15.438 INFO:teuthology.orchestra.run.smithi007.stdout: "pg_temp": [ 2024-09-18T08:20:15.438 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:20:15.438 INFO:teuthology.orchestra.run.smithi007.stdout: "pgid": "11.4", 2024-09-18T08:20:15.438 INFO:teuthology.orchestra.run.smithi007.stdout: "osds": [ 2024-09-18T08:20:15.439 INFO:teuthology.orchestra.run.smithi007.stdout: 7, 2024-09-18T08:20:15.439 INFO:teuthology.orchestra.run.smithi007.stdout: 6 2024-09-18T08:20:15.439 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-09-18T08:20:15.439 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-09-18T08:20:15.439 INFO:teuthology.orchestra.run.smithi007.stdout: ], 2024-09-18T08:20:15.439 INFO:teuthology.orchestra.run.smithi007.stdout: "primary_temp": [], 2024-09-18T08:20:15.439 INFO:teuthology.orchestra.run.smithi007.stdout: "blocklist": { 2024-09-18T08:20:15.439 INFO:teuthology.orchestra.run.smithi007.stdout: "172.21.15.7:0/398367658": "2024-09-19T08:03:17.052903+0000", 2024-09-18T08:20:15.439 INFO:teuthology.orchestra.run.smithi007.stdout: "172.21.15.7:6800/2918496291": "2024-09-19T07:57:31.015269+0000", 2024-09-18T08:20:15.439 INFO:teuthology.orchestra.run.smithi007.stdout: "172.21.15.7:0/2514354773": "2024-09-19T07:57:31.015269+0000", 2024-09-18T08:20:15.439 INFO:teuthology.orchestra.run.smithi007.stdout: "172.21.15.7:0/3924911335": "2024-09-19T07:57:31.015269+0000", 2024-09-18T08:20:15.439 INFO:teuthology.orchestra.run.smithi007.stdout: "172.21.15.7:0/1668598084": "2024-09-19T07:57:57.346149+0000", 2024-09-18T08:20:15.440 INFO:teuthology.orchestra.run.smithi007.stdout: "172.21.15.7:0/4272027292": "2024-09-19T07:57:31.015269+0000", 2024-09-18T08:20:15.440 INFO:teuthology.orchestra.run.smithi007.stdout: "172.21.15.7:6800/1013061071": "2024-09-19T07:57:57.346149+0000", 2024-09-18T08:20:15.440 INFO:teuthology.orchestra.run.smithi007.stdout: "172.21.15.7:6800/1491061076": "2024-09-19T08:03:17.052903+0000", 2024-09-18T08:20:15.440 INFO:teuthology.orchestra.run.smithi007.stdout: "172.21.15.7:0/533727535": "2024-09-19T07:57:57.346149+0000", 2024-09-18T08:20:15.440 INFO:teuthology.orchestra.run.smithi007.stdout: "172.21.15.7:0/1605217610": "2024-09-19T08:03:17.052903+0000", 2024-09-18T08:20:15.440 INFO:teuthology.orchestra.run.smithi007.stdout: "172.21.15.7:0/1722968065": "2024-09-19T08:03:17.052903+0000", 2024-09-18T08:20:15.440 INFO:teuthology.orchestra.run.smithi007.stdout: "172.21.15.7:0/1277549846": "2024-09-19T07:57:57.346149+0000", 2024-09-18T08:20:15.440 INFO:teuthology.orchestra.run.smithi007.stdout: "172.21.15.7:0/232452890": "2024-09-19T08:03:17.052903+0000", 2024-09-18T08:20:15.440 INFO:teuthology.orchestra.run.smithi007.stdout: "172.21.15.7:0/3881670470": "2024-09-19T08:03:17.052903+0000" 2024-09-18T08:20:15.440 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:20:15.440 INFO:teuthology.orchestra.run.smithi007.stdout: "range_blocklist": {}, 2024-09-18T08:20:15.440 INFO:teuthology.orchestra.run.smithi007.stdout: "erasure_code_profiles": { 2024-09-18T08:20:15.440 INFO:teuthology.orchestra.run.smithi007.stdout: "default": { 2024-09-18T08:20:15.441 INFO:teuthology.orchestra.run.smithi007.stdout: "crush-failure-domain": "osd", 2024-09-18T08:20:15.441 INFO:teuthology.orchestra.run.smithi007.stdout: "k": "2", 2024-09-18T08:20:15.441 INFO:teuthology.orchestra.run.smithi007.stdout: "m": "1", 2024-09-18T08:20:15.441 INFO:teuthology.orchestra.run.smithi007.stdout: "plugin": "jerasure", 2024-09-18T08:20:15.441 INFO:teuthology.orchestra.run.smithi007.stdout: "technique": "reed_sol_van" 2024-09-18T08:20:15.441 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-09-18T08:20:15.441 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:20:15.441 INFO:teuthology.orchestra.run.smithi007.stdout: "removed_snaps_queue": [], 2024-09-18T08:20:15.441 INFO:teuthology.orchestra.run.smithi007.stdout: "new_removed_snaps": [], 2024-09-18T08:20:15.441 INFO:teuthology.orchestra.run.smithi007.stdout: "new_purged_snaps": [], 2024-09-18T08:20:15.441 INFO:teuthology.orchestra.run.smithi007.stdout: "crush_node_flags": {}, 2024-09-18T08:20:15.441 INFO:teuthology.orchestra.run.smithi007.stdout: "device_class_flags": {}, 2024-09-18T08:20:15.441 INFO:teuthology.orchestra.run.smithi007.stdout: "stretch_mode": { 2024-09-18T08:20:15.442 INFO:teuthology.orchestra.run.smithi007.stdout: "stretch_mode_enabled": false, 2024-09-18T08:20:15.442 INFO:teuthology.orchestra.run.smithi007.stdout: "stretch_bucket_count": 0, 2024-09-18T08:20:15.442 INFO:teuthology.orchestra.run.smithi007.stdout: "degraded_stretch_mode": 0, 2024-09-18T08:20:15.442 INFO:teuthology.orchestra.run.smithi007.stdout: "recovering_stretch_mode": 0, 2024-09-18T08:20:15.442 INFO:teuthology.orchestra.run.smithi007.stdout: "stretch_mode_bucket": 0 2024-09-18T08:20:15.442 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-09-18T08:20:15.442 INFO:teuthology.orchestra.run.smithi007.stdout:} 2024-09-18T08:20:15.442 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4 2024-09-18T08:20:15.465 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:20:15.466 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:20:15.466 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4' 2024-09-18T08:20:15.567 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4 2024-09-18T08:20:15.592 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:20:15.593 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:20:15.593 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4' 2024-09-18T08:20:15.694 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7 2024-09-18T08:20:15.717 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:20:15.717 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:20:15.718 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7' 2024-09-18T08:20:15.729 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:20:15 smithi007 ceph-mon[29459]: from='client.? 172.21.15.7:0/3747643513' entity='client.admin' cmd={"prefix": "osd dump", "format": "json-pretty"} : dispatch 2024-09-18T08:20:15.729 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:20:15 smithi007 ceph-mon[29459]: osdmap e437: 8 total, 8 up, 7 in 2024-09-18T08:20:15.730 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:20:15 smithi007 ceph-mon[39249]: from='client.? 172.21.15.7:0/3747643513' entity='client.admin' cmd={"prefix": "osd dump", "format": "json-pretty"} : dispatch 2024-09-18T08:20:15.730 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:20:15 smithi007 ceph-mon[39249]: osdmap e437: 8 total, 8 up, 7 in 2024-09-18T08:20:15.818 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6 2024-09-18T08:20:15.841 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:20:15.841 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:20:15.842 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6' 2024-09-18T08:20:15.905 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:20:15 smithi050 ceph-mon[32371]: from='client.? 172.21.15.7:0/3747643513' entity='client.admin' cmd={"prefix": "osd dump", "format": "json-pretty"} : dispatch 2024-09-18T08:20:15.905 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:20:15 smithi050 ceph-mon[32371]: osdmap e437: 8 total, 8 up, 7 in 2024-09-18T08:20:15.937 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid 779349ca-7593-11ef-bceb-c7b262605968 -- ceph osd unset nodeep-scrub 2024-09-18T08:20:15.942 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7 2024-09-18T08:20:15.958 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid 779349ca-7593-11ef-bceb-c7b262605968 -- ceph daemon osd.2 dump_blocked_ops 2024-09-18T08:20:15.966 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:20:15.966 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:20:15.967 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7' 2024-09-18T08:20:16.067 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2 2024-09-18T08:20:16.090 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:20:16.091 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:20:16.091 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2' 2024-09-18T08:20:16.192 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0 2024-09-18T08:20:16.215 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:20:16.215 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:20:16.215 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0' 2024-09-18T08:20:16.259 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/779349ca-7593-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T08:20:16.303 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/779349ca-7593-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T08:20:16.316 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0 2024-09-18T08:20:16.339 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:20:16.339 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:20:16.340 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0' 2024-09-18T08:20:16.440 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2 2024-09-18T08:20:16.463 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:20:16.464 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:20:16.464 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2' 2024-09-18T08:20:16.565 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3 2024-09-18T08:20:16.613 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:20:16.614 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:20:16.614 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3' 2024-09-18T08:20:16.698 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:20:16 smithi007 ceph-mon[29459]: 4.19 scrub starts 2024-09-18T08:20:16.698 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:20:16 smithi007 ceph-mon[29459]: 4.19 scrub ok 2024-09-18T08:20:16.698 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:20:16 smithi007 ceph-mon[29459]: 5.1e scrub starts 2024-09-18T08:20:16.698 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:20:16 smithi007 ceph-mon[29459]: 5.1e scrub ok 2024-09-18T08:20:16.698 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:20:16 smithi007 ceph-mon[29459]: pgmap v876: 158 pgs: 1 peering, 2 active+recovering+undersized+remapped, 155 active+clean; 5.5 GiB data, 22 GiB used, 604 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.26k op/s; 487/272511 objects misplaced (0.179%); 21 MiB/s, 337 objects/s recovering 2024-09-18T08:20:16.698 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:20:16 smithi007 ceph-mon[29459]: 3.d scrub starts 2024-09-18T08:20:16.698 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:20:16 smithi007 ceph-mon[29459]: 3.d scrub ok 2024-09-18T08:20:16.699 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:20:16 smithi007 ceph-mon[29459]: osdmap e438: 8 total, 8 up, 7 in 2024-09-18T08:20:16.699 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:20:16 smithi007 ceph-mon[39249]: 4.19 scrub starts 2024-09-18T08:20:16.699 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:20:16 smithi007 ceph-mon[39249]: 4.19 scrub ok 2024-09-18T08:20:16.699 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:20:16 smithi007 ceph-mon[39249]: 5.1e scrub starts 2024-09-18T08:20:16.699 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:20:16 smithi007 ceph-mon[39249]: 5.1e scrub ok 2024-09-18T08:20:16.699 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:20:16 smithi007 ceph-mon[39249]: pgmap v876: 158 pgs: 1 peering, 2 active+recovering+undersized+remapped, 155 active+clean; 5.5 GiB data, 22 GiB used, 604 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.26k op/s; 487/272511 objects misplaced (0.179%); 21 MiB/s, 337 objects/s recovering 2024-09-18T08:20:16.699 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:20:16 smithi007 ceph-mon[39249]: 3.d scrub starts 2024-09-18T08:20:16.699 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:20:16 smithi007 ceph-mon[39249]: 3.d scrub ok 2024-09-18T08:20:16.699 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:20:16 smithi007 ceph-mon[39249]: osdmap e438: 8 total, 8 up, 7 in 2024-09-18T08:20:16.715 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5 2024-09-18T08:20:16.747 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:20:16.748 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:20:16.748 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5' 2024-09-18T08:20:16.849 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3 2024-09-18T08:20:16.884 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:20:16.885 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:20:16.885 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3' 2024-09-18T08:20:16.905 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:20:16 smithi050 ceph-mon[32371]: 4.19 scrub starts 2024-09-18T08:20:16.905 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:20:16 smithi050 ceph-mon[32371]: 4.19 scrub ok 2024-09-18T08:20:16.905 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:20:16 smithi050 ceph-mon[32371]: 5.1e scrub starts 2024-09-18T08:20:16.905 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:20:16 smithi050 ceph-mon[32371]: 5.1e scrub ok 2024-09-18T08:20:16.905 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:20:16 smithi050 ceph-mon[32371]: pgmap v876: 158 pgs: 1 peering, 2 active+recovering+undersized+remapped, 155 active+clean; 5.5 GiB data, 22 GiB used, 604 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.26k op/s; 487/272511 objects misplaced (0.179%); 21 MiB/s, 337 objects/s recovering 2024-09-18T08:20:16.905 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:20:16 smithi050 ceph-mon[32371]: 3.d scrub starts 2024-09-18T08:20:16.905 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:20:16 smithi050 ceph-mon[32371]: 3.d scrub ok 2024-09-18T08:20:16.905 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:20:16 smithi050 ceph-mon[32371]: osdmap e438: 8 total, 8 up, 7 in 2024-09-18T08:20:16.945 INFO:teuthology.orchestra.run.smithi007.stdout:osd.0: {} 2024-09-18T08:20:16.945 INFO:teuthology.orchestra.run.smithi007.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-18T08:20:16.978 INFO:teuthology.orchestra.run.smithi007.stdout:osd.1: {} 2024-09-18T08:20:16.979 INFO:teuthology.orchestra.run.smithi007.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-18T08:20:16.979 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Sep 18 08:20:16 smithi007 ceph-779349ca-7593-11ef-bceb-c7b262605968-mgr-y[29688]: ::ffff:172.21.15.50 - - [18/Sep/2024:08:20:16] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-09-18T08:20:16.986 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1 2024-09-18T08:20:17.013 INFO:teuthology.orchestra.run.smithi007.stdout:osd.2: {} 2024-09-18T08:20:17.013 INFO:teuthology.orchestra.run.smithi007.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-18T08:20:17.027 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:20:17.028 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:20:17.028 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1' 2024-09-18T08:20:17.051 INFO:teuthology.orchestra.run.smithi007.stdout:osd.3: {} 2024-09-18T08:20:17.051 INFO:teuthology.orchestra.run.smithi007.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-18T08:20:17.087 INFO:teuthology.orchestra.run.smithi007.stdout:osd.4: {} 2024-09-18T08:20:17.087 INFO:teuthology.orchestra.run.smithi007.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-18T08:20:17.121 INFO:teuthology.orchestra.run.smithi007.stdout:osd.5: {} 2024-09-18T08:20:17.121 INFO:teuthology.orchestra.run.smithi007.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-18T08:20:17.129 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7 2024-09-18T08:20:17.145 INFO:teuthology.orchestra.run.smithi007.stdout:osd.6: {} 2024-09-18T08:20:17.145 INFO:teuthology.orchestra.run.smithi007.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-18T08:20:17.162 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:20:17.162 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:20:17.163 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7' 2024-09-18T08:20:17.178 INFO:teuthology.orchestra.run.smithi007.stdout:osd.7: {} 2024-09-18T08:20:17.178 INFO:teuthology.orchestra.run.smithi007.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-18T08:20:17.264 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3 2024-09-18T08:20:17.304 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:20:17.305 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:20:17.305 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3' 2024-09-18T08:20:17.406 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0 2024-09-18T08:20:17.441 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:20:17.442 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:20:17.443 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0' 2024-09-18T08:20:17.543 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0 2024-09-18T08:20:17.573 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:20:17.573 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:20:17.574 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0' 2024-09-18T08:20:17.675 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4 2024-09-18T08:20:17.709 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:20:17.709 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:20:17.710 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4' 2024-09-18T08:20:17.810 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5 2024-09-18T08:20:17.842 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:20:17.842 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:20:17.843 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5' 2024-09-18T08:20:17.905 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:20:17 smithi050 ceph-mon[32371]: 4.8 scrub starts 2024-09-18T08:20:17.905 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:20:17 smithi050 ceph-mon[32371]: 4.8 scrub ok 2024-09-18T08:20:17.905 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:20:17 smithi050 ceph-mon[32371]: from='client.? 172.21.15.7:0/4187515061' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-18T08:20:17.944 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0 2024-09-18T08:20:17.967 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:20:17.967 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:20:17.968 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0' 2024-09-18T08:20:17.979 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:20:17 smithi007 ceph-mon[29459]: 4.8 scrub starts 2024-09-18T08:20:17.980 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:20:17 smithi007 ceph-mon[29459]: 4.8 scrub ok 2024-09-18T08:20:17.980 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:20:17 smithi007 ceph-mon[29459]: from='client.? 172.21.15.7:0/4187515061' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-18T08:20:17.980 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:20:17 smithi007 ceph-mon[39249]: 4.8 scrub starts 2024-09-18T08:20:17.980 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:20:17 smithi007 ceph-mon[39249]: 4.8 scrub ok 2024-09-18T08:20:17.980 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:20:17 smithi007 ceph-mon[39249]: from='client.? 172.21.15.7:0/4187515061' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-18T08:20:18.069 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6 2024-09-18T08:20:18.100 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:20:18.101 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:20:18.101 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6' 2024-09-18T08:20:18.202 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3 2024-09-18T08:20:18.240 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:20:18.241 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:20:18.241 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3' 2024-09-18T08:20:18.342 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6 2024-09-18T08:20:18.374 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:20:18.375 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:20:18.375 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6' 2024-09-18T08:20:18.476 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2 2024-09-18T08:20:18.510 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:20:18.511 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:20:18.511 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2' 2024-09-18T08:20:18.612 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2 2024-09-18T08:20:18.653 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:20:18.654 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:20:18.654 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2' 2024-09-18T08:20:18.730 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:20:18 smithi007 ceph-mon[39249]: 3.1a scrub starts 2024-09-18T08:20:18.730 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:20:18 smithi007 ceph-mon[39249]: 3.1a scrub ok 2024-09-18T08:20:18.730 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:20:18 smithi007 ceph-mon[39249]: 3.6 scrub starts 2024-09-18T08:20:18.730 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:20:18 smithi007 ceph-mon[39249]: 3.6 scrub ok 2024-09-18T08:20:18.730 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:20:18 smithi007 ceph-mon[39249]: pgmap v878: 158 pgs: 1 peering, 2 active+recovering+undersized+remapped, 155 active+clean; 5.5 GiB data, 22 GiB used, 604 GiB / 626 GiB avail; 0 B/s wr, 758 op/s; 487/272511 objects misplaced (0.179%); 8.6 MiB/s, 138 objects/s recovering 2024-09-18T08:20:18.730 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:20:18 smithi007 ceph-mon[39249]: 4.1c scrub starts 2024-09-18T08:20:18.730 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:20:18 smithi007 ceph-mon[29459]: 3.1a scrub starts 2024-09-18T08:20:18.731 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:20:18 smithi007 ceph-mon[29459]: 3.1a scrub ok 2024-09-18T08:20:18.731 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:20:18 smithi007 ceph-mon[29459]: 3.6 scrub starts 2024-09-18T08:20:18.731 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:20:18 smithi007 ceph-mon[29459]: 3.6 scrub ok 2024-09-18T08:20:18.731 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:20:18 smithi007 ceph-mon[29459]: pgmap v878: 158 pgs: 1 peering, 2 active+recovering+undersized+remapped, 155 active+clean; 5.5 GiB data, 22 GiB used, 604 GiB / 626 GiB avail; 0 B/s wr, 758 op/s; 487/272511 objects misplaced (0.179%); 8.6 MiB/s, 138 objects/s recovering 2024-09-18T08:20:18.731 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:20:18 smithi007 ceph-mon[29459]: 4.1c scrub starts 2024-09-18T08:20:18.755 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7 2024-09-18T08:20:18.786 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:20:18.787 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:20:18.787 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7' 2024-09-18T08:20:18.888 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7 2024-09-18T08:20:18.904 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:20:18 smithi050 ceph-mon[32371]: 3.1a scrub starts 2024-09-18T08:20:18.905 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:20:18 smithi050 ceph-mon[32371]: 3.1a scrub ok 2024-09-18T08:20:18.905 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:20:18 smithi050 ceph-mon[32371]: 3.6 scrub starts 2024-09-18T08:20:18.905 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:20:18 smithi050 ceph-mon[32371]: 3.6 scrub ok 2024-09-18T08:20:18.905 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:20:18 smithi050 ceph-mon[32371]: pgmap v878: 158 pgs: 1 peering, 2 active+recovering+undersized+remapped, 155 active+clean; 5.5 GiB data, 22 GiB used, 604 GiB / 626 GiB avail; 0 B/s wr, 758 op/s; 487/272511 objects misplaced (0.179%); 8.6 MiB/s, 138 objects/s recovering 2024-09-18T08:20:18.905 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:20:18 smithi050 ceph-mon[32371]: 4.1c scrub starts 2024-09-18T08:20:18.917 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:20:18.918 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:20:18.918 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7' 2024-09-18T08:20:19.019 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1 2024-09-18T08:20:19.055 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:20:19.056 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:20:19.056 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1' 2024-09-18T08:20:19.157 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1 2024-09-18T08:20:19.194 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:20:19.195 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:20:19.195 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1' 2024-09-18T08:20:19.296 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4 2024-09-18T08:20:19.318 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:20:19.318 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:20:19.319 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4' 2024-09-18T08:20:19.419 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3 2024-09-18T08:20:19.461 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:20:19.462 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:20:19.463 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3' 2024-09-18T08:20:19.564 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2 2024-09-18T08:20:19.603 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:20:19.604 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:20:19.605 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2' 2024-09-18T08:20:19.705 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1 2024-09-18T08:20:19.745 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:20:19.745 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:20:19.746 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1' 2024-09-18T08:20:19.847 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2 2024-09-18T08:20:19.884 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:20:19.884 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:20:19.885 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2' 2024-09-18T08:20:19.905 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:20:19 smithi050 ceph-mon[32371]: 4.1c scrub ok 2024-09-18T08:20:19.905 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:20:19 smithi050 ceph-mon[32371]: 6.1c scrub starts 2024-09-18T08:20:19.905 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:20:19 smithi050 ceph-mon[32371]: 6.1c scrub ok 2024-09-18T08:20:19.905 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:20:19 smithi050 ceph-mon[32371]: from='mgr.24470 172.21.15.7:0/2594390261' entity='mgr.y' 2024-09-18T08:20:19.905 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:20:19 smithi050 ceph-mon[32371]: from='mgr.24470 172.21.15.7:0/2594390261' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-18T08:20:19.905 INFO:journalctl@ceph.iscsi.iscsi.a.smithi050.stdout:Sep 18 08:20:19 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-iscsi-iscsi-a[57782]: debug there is no tcmu-runner data available 2024-09-18T08:20:19.980 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:20:19 smithi007 ceph-mon[39249]: 4.1c scrub ok 2024-09-18T08:20:19.980 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:20:19 smithi007 ceph-mon[39249]: 6.1c scrub starts 2024-09-18T08:20:19.980 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:20:19 smithi007 ceph-mon[39249]: 6.1c scrub ok 2024-09-18T08:20:19.980 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:20:19 smithi007 ceph-mon[39249]: from='mgr.24470 172.21.15.7:0/2594390261' entity='mgr.y' 2024-09-18T08:20:19.980 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:20:19 smithi007 ceph-mon[39249]: from='mgr.24470 172.21.15.7:0/2594390261' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-18T08:20:19.980 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:20:19 smithi007 ceph-mon[29459]: 4.1c scrub ok 2024-09-18T08:20:19.980 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:20:19 smithi007 ceph-mon[29459]: 6.1c scrub starts 2024-09-18T08:20:19.980 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:20:19 smithi007 ceph-mon[29459]: 6.1c scrub ok 2024-09-18T08:20:19.981 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:20:19 smithi007 ceph-mon[29459]: from='mgr.24470 172.21.15.7:0/2594390261' entity='mgr.y' 2024-09-18T08:20:19.981 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:20:19 smithi007 ceph-mon[29459]: from='mgr.24470 172.21.15.7:0/2594390261' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-18T08:20:19.986 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4 2024-09-18T08:20:20.022 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:20:20.022 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:20:20.022 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4' 2024-09-18T08:20:20.123 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2 2024-09-18T08:20:20.161 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:20:20.161 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:20:20.161 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2' 2024-09-18T08:20:20.262 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4 2024-09-18T08:20:20.298 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:20:20.298 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:20:20.299 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4' 2024-09-18T08:20:20.399 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6 2024-09-18T08:20:20.433 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:20:20.434 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:20:20.434 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6' 2024-09-18T08:20:20.535 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2 2024-09-18T08:20:20.576 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:20:20.577 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:20:20.577 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2' 2024-09-18T08:20:20.678 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0 2024-09-18T08:20:20.715 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:20:20.716 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:20:20.717 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0' 2024-09-18T08:20:20.817 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2 2024-09-18T08:20:20.839 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:20:20.840 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:20:20.840 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2' 2024-09-18T08:20:20.905 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:20:20 smithi050 ceph-mon[32371]: 6.c scrub starts 2024-09-18T08:20:20.905 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:20:20 smithi050 ceph-mon[32371]: 6.c scrub ok 2024-09-18T08:20:20.905 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:20:20 smithi050 ceph-mon[32371]: pgmap v879: 158 pgs: 158 active+clean; 5.4 GiB data, 21 GiB used, 605 GiB / 626 GiB avail; 854 B/s rd, 0 B/s wr, 941 op/s; 11 MiB/s, 173 objects/s recovering 2024-09-18T08:20:20.905 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:20:20 smithi050 ceph-mon[32371]: 5.7 scrub starts 2024-09-18T08:20:20.905 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:20:20 smithi050 ceph-mon[32371]: 5.7 scrub ok 2024-09-18T08:20:20.905 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:20:20 smithi050 ceph-mon[32371]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 2 pgs undersized) 2024-09-18T08:20:20.941 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6 2024-09-18T08:20:20.950 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:20:20 smithi007 ceph-mon[29459]: 6.c scrub starts 2024-09-18T08:20:20.950 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:20:20 smithi007 ceph-mon[29459]: 6.c scrub ok 2024-09-18T08:20:20.950 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:20:20 smithi007 ceph-mon[29459]: pgmap v879: 158 pgs: 158 active+clean; 5.4 GiB data, 21 GiB used, 605 GiB / 626 GiB avail; 854 B/s rd, 0 B/s wr, 941 op/s; 11 MiB/s, 173 objects/s recovering 2024-09-18T08:20:20.951 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:20:20 smithi007 ceph-mon[29459]: 5.7 scrub starts 2024-09-18T08:20:20.951 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:20:20 smithi007 ceph-mon[29459]: 5.7 scrub ok 2024-09-18T08:20:20.951 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:20:20 smithi007 ceph-mon[29459]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 2 pgs undersized) 2024-09-18T08:20:20.952 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:20:20 smithi007 ceph-mon[39249]: 6.c scrub starts 2024-09-18T08:20:20.952 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:20:20 smithi007 ceph-mon[39249]: 6.c scrub ok 2024-09-18T08:20:20.952 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:20:20 smithi007 ceph-mon[39249]: pgmap v879: 158 pgs: 158 active+clean; 5.4 GiB data, 21 GiB used, 605 GiB / 626 GiB avail; 854 B/s rd, 0 B/s wr, 941 op/s; 11 MiB/s, 173 objects/s recovering 2024-09-18T08:20:20.952 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:20:20 smithi007 ceph-mon[39249]: 5.7 scrub starts 2024-09-18T08:20:20.952 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:20:20 smithi007 ceph-mon[39249]: 5.7 scrub ok 2024-09-18T08:20:20.952 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:20:20 smithi007 ceph-mon[39249]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 2 pgs undersized) 2024-09-18T08:20:20.973 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:20:20.974 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:20:20.974 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6' 2024-09-18T08:20:21.075 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4 2024-09-18T08:20:21.104 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:20:21.105 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:20:21.105 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4' 2024-09-18T08:20:21.206 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3 2024-09-18T08:20:21.251 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:20:21.251 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:20:21.252 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3' 2024-09-18T08:20:21.353 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3 2024-09-18T08:20:21.379 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:20:21.380 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:20:21.380 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3' 2024-09-18T08:20:21.481 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0 2024-09-18T08:20:21.522 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:20:21.523 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:20:21.524 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0' 2024-09-18T08:20:21.625 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3 2024-09-18T08:20:21.658 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:20:21.659 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:20:21.660 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3' 2024-09-18T08:20:21.761 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6 2024-09-18T08:20:21.783 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:20:21.784 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:20:21.784 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6' 2024-09-18T08:20:21.885 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6 2024-09-18T08:20:21.905 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:20:21 smithi050 ceph-mon[32371]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-18T08:20:21.915 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:20:21.915 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:20:21.916 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6' 2024-09-18T08:20:21.980 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:20:21 smithi007 ceph-mon[39249]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-18T08:20:21.980 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:20:21 smithi007 ceph-mon[29459]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-18T08:20:22.017 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2 2024-09-18T08:20:22.054 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:20:22.055 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:20:22.056 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2' 2024-09-18T08:20:22.156 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3 2024-09-18T08:20:22.198 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:20:22.198 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:20:22.199 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3' 2024-09-18T08:20:22.299 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6 2024-09-18T08:20:22.329 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:20:22.330 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:20:22.330 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6' 2024-09-18T08:20:22.431 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7 2024-09-18T08:20:22.464 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:20:22.464 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:20:22.465 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7' 2024-09-18T08:20:22.566 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2 2024-09-18T08:20:22.606 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:20:22.606 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:20:22.607 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2' 2024-09-18T08:20:22.707 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2 2024-09-18T08:20:22.744 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:20:22.745 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:20:22.745 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2' 2024-09-18T08:20:22.846 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3 2024-09-18T08:20:22.870 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:20:22.870 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:20:22.871 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3' 2024-09-18T08:20:22.905 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:20:22 smithi050 ceph-mon[32371]: 4.d scrub starts 2024-09-18T08:20:22.905 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:20:22 smithi050 ceph-mon[32371]: 4.d scrub ok 2024-09-18T08:20:22.905 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:20:22 smithi050 ceph-mon[32371]: pgmap v880: 158 pgs: 158 active+clean; 5.3 GiB data, 20 GiB used, 605 GiB / 626 GiB avail; 727 B/s rd, 0 B/s wr, 889 op/s; 4.3 MiB/s, 69 objects/s recovering 2024-09-18T08:20:22.905 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:20:22 smithi050 ceph-mon[32371]: 3.1c scrub starts 2024-09-18T08:20:22.905 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:20:22 smithi050 ceph-mon[32371]: 3.1c scrub ok 2024-09-18T08:20:22.905 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:20:22 smithi050 ceph-mon[32371]: 3.1 scrub starts 2024-09-18T08:20:22.905 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:20:22 smithi050 ceph-mon[32371]: 3.1 scrub ok 2024-09-18T08:20:22.972 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4 2024-09-18T08:20:22.979 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:20:22 smithi007 ceph-mon[39249]: 4.d scrub starts 2024-09-18T08:20:22.980 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:20:22 smithi007 ceph-mon[39249]: 4.d scrub ok 2024-09-18T08:20:22.980 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:20:22 smithi007 ceph-mon[39249]: pgmap v880: 158 pgs: 158 active+clean; 5.3 GiB data, 20 GiB used, 605 GiB / 626 GiB avail; 727 B/s rd, 0 B/s wr, 889 op/s; 4.3 MiB/s, 69 objects/s recovering 2024-09-18T08:20:22.980 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:20:22 smithi007 ceph-mon[39249]: 3.1c scrub starts 2024-09-18T08:20:22.980 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:20:22 smithi007 ceph-mon[39249]: 3.1c scrub ok 2024-09-18T08:20:22.980 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:20:22 smithi007 ceph-mon[39249]: 3.1 scrub starts 2024-09-18T08:20:22.980 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:20:22 smithi007 ceph-mon[39249]: 3.1 scrub ok 2024-09-18T08:20:22.980 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:20:22 smithi007 ceph-mon[29459]: 4.d scrub starts 2024-09-18T08:20:22.980 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:20:22 smithi007 ceph-mon[29459]: 4.d scrub ok 2024-09-18T08:20:22.980 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:20:22 smithi007 ceph-mon[29459]: pgmap v880: 158 pgs: 158 active+clean; 5.3 GiB data, 20 GiB used, 605 GiB / 626 GiB avail; 727 B/s rd, 0 B/s wr, 889 op/s; 4.3 MiB/s, 69 objects/s recovering 2024-09-18T08:20:22.980 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:20:22 smithi007 ceph-mon[29459]: 3.1c scrub starts 2024-09-18T08:20:22.981 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:20:22 smithi007 ceph-mon[29459]: 3.1c scrub ok 2024-09-18T08:20:22.981 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:20:22 smithi007 ceph-mon[29459]: 3.1 scrub starts 2024-09-18T08:20:22.981 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:20:22 smithi007 ceph-mon[29459]: 3.1 scrub ok 2024-09-18T08:20:23.005 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:20:23.006 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:20:23.006 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4' 2024-09-18T08:20:23.108 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0 2024-09-18T08:20:23.152 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:20:23.152 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:20:23.153 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0' 2024-09-18T08:20:23.253 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1 2024-09-18T08:20:23.290 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:20:23.291 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:20:23.292 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1' 2024-09-18T08:20:23.393 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2 2024-09-18T08:20:23.430 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:20:23.431 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:20:23.431 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2' 2024-09-18T08:20:23.532 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1 2024-09-18T08:20:23.575 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:20:23.575 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:20:23.575 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1' 2024-09-18T08:20:23.676 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6 2024-09-18T08:20:23.711 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:20:23.712 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:20:23.712 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6' 2024-09-18T08:20:23.813 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4 2024-09-18T08:20:23.846 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:20:23.847 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:20:23.847 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4' 2024-09-18T08:20:23.905 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:20:23 smithi050 ceph-mon[32371]: 6.1f scrub starts 2024-09-18T08:20:23.905 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:20:23 smithi050 ceph-mon[32371]: 6.1f scrub ok 2024-09-18T08:20:23.905 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:20:23 smithi050 ceph-mon[32371]: 3.5 scrub starts 2024-09-18T08:20:23.905 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:20:23 smithi050 ceph-mon[32371]: 3.5 scrub ok 2024-09-18T08:20:23.905 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:20:23 smithi050 ceph-mon[32371]: 3.4 scrub starts 2024-09-18T08:20:23.905 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:20:23 smithi050 ceph-mon[32371]: 3.4 scrub ok 2024-09-18T08:20:23.948 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4 2024-09-18T08:20:23.978 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:20:23.978 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:20:23.979 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4' 2024-09-18T08:20:23.979 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:20:23 smithi007 ceph-mon[29459]: 6.1f scrub starts 2024-09-18T08:20:23.980 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:20:23 smithi007 ceph-mon[29459]: 6.1f scrub ok 2024-09-18T08:20:23.980 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:20:23 smithi007 ceph-mon[29459]: 3.5 scrub starts 2024-09-18T08:20:23.980 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:20:23 smithi007 ceph-mon[29459]: 3.5 scrub ok 2024-09-18T08:20:23.980 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:20:23 smithi007 ceph-mon[29459]: 3.4 scrub starts 2024-09-18T08:20:23.980 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:20:23 smithi007 ceph-mon[29459]: 3.4 scrub ok 2024-09-18T08:20:23.980 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:20:23 smithi007 ceph-mon[39249]: 6.1f scrub starts 2024-09-18T08:20:23.980 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:20:23 smithi007 ceph-mon[39249]: 6.1f scrub ok 2024-09-18T08:20:23.980 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:20:23 smithi007 ceph-mon[39249]: 3.5 scrub starts 2024-09-18T08:20:23.980 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:20:23 smithi007 ceph-mon[39249]: 3.5 scrub ok 2024-09-18T08:20:23.980 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:20:23 smithi007 ceph-mon[39249]: 3.4 scrub starts 2024-09-18T08:20:23.981 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:20:23 smithi007 ceph-mon[39249]: 3.4 scrub ok 2024-09-18T08:20:24.080 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3 2024-09-18T08:20:24.118 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:20:24.119 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:20:24.119 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3' 2024-09-18T08:20:24.220 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0 2024-09-18T08:20:24.262 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:20:24.262 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:20:24.263 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0' 2024-09-18T08:20:24.345 INFO:tasks.thrashosds.thrasher:j is {'epoch': 436, 'fsid': '779349ca-7593-11ef-bceb-c7b262605968', 'created': '2024-09-18T07:57:09.295710+0000', 'modified': '2024-09-18T08:20:14.463673+0000', 'last_up_change': '2024-09-18T08:14:38.391465+0000', 'last_in_change': '2024-09-18T08:18:26.612016+0000', 'flags': 'nodeep-scrub,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit', 'flags_num': 5804032, 'flags_set': ['nodeep-scrub', 'pglog_hardlimit', 'purged_snapdirs', 'recovery_deletes', 'sortbitwise'], 'crush_version': 19, 'full_ratio': 0.949999988079071, 'backfillfull_ratio': 0.8999999761581421, 'nearfull_ratio': 0.8500000238418579, 'cluster_snapshot': '', 'pool_max': 11, 'max_osd': 8, 'require_min_compat_client': 'luminous', 'min_compat_client': 'luminous', 'require_osd_release': 'squid', 'allow_crimson': False, 'pools': [{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-09-18T08:00:11.802627+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'is_stretch_pool': False, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 11, 'pg_placement_num': 11, 'pg_placement_num_target': 11, 'pg_num_target': 11, 'pg_num_pending': 11, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '285', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '152', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.549999952316284, 'score_stable': 2.549999952316284, 'optimal_score': 1, 'raw_score_acting': 2.549999952316284, 'raw_score_stable': 2.549999952316284, 'primary_affinity_weighted': 0.8092708587646484, 'average_primary_affinity': 0.9200000166893005, 'average_primary_affinity_weighted': 0.8100000023841858}}, {'pool': 2, 'pool_name': 'datapool', 'create_time': '2024-09-18T08:02:45.374099+0000', 'flags': 8193, 'flags_names': 'hashpspool,selfmanaged_snaps', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'is_stretch_pool': False, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 3, 'pg_placement_num': 3, 'pg_placement_num_target': 3, 'pg_num_target': 3, 'pg_num_pending': 3, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '64', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 2, 'snap_epoch': 64, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.3299999237060547, 'score_stable': 2.3299999237060547, 'optimal_score': 0.8600000143051147, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 0.8092708587646484, 'average_primary_affinity': 0.9200000166893005, 'average_primary_affinity_weighted': 0.8100000023841858}}, {'pool': 3, 'pool_name': '.rgw.root', 'create_time': '2024-09-18T08:02:46.319947+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'is_stretch_pool': False, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '63', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rgw': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.9700000286102295, 'score_stable': 1.9700000286102295, 'optimal_score': 1, 'raw_score_acting': 1.9700000286102295, 'raw_score_stable': 1.9700000286102295, 'primary_affinity_weighted': 0.8092708587646484, 'average_primary_affinity': 0.9200000166893005, 'average_primary_affinity_weighted': 0.8100000023841858}}, {'pool': 4, 'pool_name': 'default.rgw.log', 'create_time': '2024-09-18T08:02:47.948552+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'is_stretch_pool': False, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '65', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rgw': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.190000057220459, 'score_stable': 2.190000057220459, 'optimal_score': 1, 'raw_score_acting': 2.190000057220459, 'raw_score_stable': 2.190000057220459, 'primary_affinity_weighted': 0.8092708587646484, 'average_primary_affinity': 0.9200000166893005, 'average_primary_affinity_weighted': 0.8100000023841858}}, {'pool': 5, 'pool_name': 'default.rgw.control', 'create_time': '2024-09-18T08:02:49.904302+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'is_stretch_pool': False, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '67', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rgw': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.75, 'score_stable': 1.75, 'optimal_score': 1, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 0.8092708587646484, 'average_primary_affinity': 0.9200000166893005, 'average_primary_affinity_weighted': 0.8100000023841858}}, {'pool': 6, 'pool_name': 'default.rgw.meta', 'create_time': '2024-09-18T08:02:51.971465+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'is_stretch_pool': False, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '70', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4}, 'application_metadata': {'rgw': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.630000114440918, 'score_stable': 2.630000114440918, 'optimal_score': 1, 'raw_score_acting': 2.630000114440918, 'raw_score_stable': 2.630000114440918, 'primary_affinity_weighted': 0.8092708587646484, 'average_primary_affinity': 0.9200000166893005, 'average_primary_affinity_weighted': 0.8100000023841858}}, {'pool': 11, 'pool_name': 'unique_pool_4', 'create_time': '2024-09-18T08:18:25.969782+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'is_stretch_pool': False, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 16, 'pg_placement_num': 16, 'pg_placement_num_target': 16, 'pg_num_target': 16, 'pg_num_pending': 16, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '431', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rados': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.619999885559082, 'score_stable': 2.619999885559082, 'optimal_score': 0.8600000143051147, 'raw_score_acting': 2.25, 'raw_score_stable': 2.25, 'primary_affinity_weighted': 0.8092708587646484, 'average_primary_affinity': 0.9200000166893005, 'average_primary_affinity_weighted': 0.8100000023841858}}], 'osds': [{'osd': 0, 'uuid': 'ae8f60df-72ac-46fb-91a1-f132b851c876', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 8, 'up_thru': 430, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.7:6801', 'nonce': 2896425794}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.7:6802', 'nonce': 2896425794}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.7:6804', 'nonce': 2896425794}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.7:6803', 'nonce': 2896425794}]}, 'public_addr': '172.21.15.7:6801/2896425794', 'cluster_addr': '172.21.15.7:6802/2896425794', 'heartbeat_back_addr': '172.21.15.7:6804/2896425794', 'heartbeat_front_addr': '172.21.15.7:6803/2896425794', 'state': ['exists', 'up']}, {'osd': 1, 'uuid': 'be28b70c-1f73-494f-ad4b-cb7ec5311999', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 13, 'last_clean_end': 243, 'up_from': 307, 'up_thru': 435, 'down_at': 244, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.7:6805', 'nonce': 3300107791}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.7:6806', 'nonce': 3300107791}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.7:6808', 'nonce': 3300107791}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.7:6807', 'nonce': 3300107791}]}, 'public_addr': '172.21.15.7:6805/3300107791', 'cluster_addr': '172.21.15.7:6806/3300107791', 'heartbeat_back_addr': '172.21.15.7:6808/3300107791', 'heartbeat_front_addr': '172.21.15.7:6807/3300107791', 'state': ['exists', 'up']}, {'osd': 2, 'uuid': 'cfa4f083-c7fd-4743-be65-84a3040fe6dc', 'up': 1, 'in': 1, 'weight': 0.12225341796875, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 18, 'up_thru': 401, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.7:6809', 'nonce': 556142559}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.7:6810', 'nonce': 556142559}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.7:6812', 'nonce': 556142559}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.7:6811', 'nonce': 556142559}]}, 'public_addr': '172.21.15.7:6809/556142559', 'cluster_addr': '172.21.15.7:6810/556142559', 'heartbeat_back_addr': '172.21.15.7:6812/556142559', 'heartbeat_front_addr': '172.21.15.7:6811/556142559', 'state': ['exists', 'up']}, {'osd': 3, 'uuid': '11ecd7e1-a751-4b75-95f5-00b32d2a5bc4', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 26, 'up_thru': 431, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.7:6813', 'nonce': 163587772}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.7:6814', 'nonce': 163587772}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.7:6816', 'nonce': 163587772}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.7:6815', 'nonce': 163587772}]}, 'public_addr': '172.21.15.7:6813/163587772', 'cluster_addr': '172.21.15.7:6814/163587772', 'heartbeat_back_addr': '172.21.15.7:6816/163587772', 'heartbeat_front_addr': '172.21.15.7:6815/163587772', 'state': ['exists', 'up']}, {'osd': 4, 'uuid': '98992b34-cfa1-45bf-97a1-7d557739252f', 'up': 1, 'in': 0, 'weight': 0, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 32, 'up_thru': 336, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.50:6800', 'nonce': 3662119805}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.50:6801', 'nonce': 3662119805}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.50:6803', 'nonce': 3662119805}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.50:6802', 'nonce': 3662119805}]}, 'public_addr': '172.21.15.50:6800/3662119805', 'cluster_addr': '172.21.15.50:6801/3662119805', 'heartbeat_back_addr': '172.21.15.50:6803/3662119805', 'heartbeat_front_addr': '172.21.15.50:6802/3662119805', 'state': ['exists', 'up']}, {'osd': 5, 'uuid': '7f290005-ebca-467b-87e4-02ead1063610', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 39, 'up_thru': 413, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.50:6804', 'nonce': 1199797954}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.50:6805', 'nonce': 1199797954}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.50:6807', 'nonce': 1199797954}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.50:6806', 'nonce': 1199797954}]}, 'public_addr': '172.21.15.50:6804/1199797954', 'cluster_addr': '172.21.15.50:6805/1199797954', 'heartbeat_back_addr': '172.21.15.50:6807/1199797954', 'heartbeat_front_addr': '172.21.15.50:6806/1199797954', 'state': ['exists', 'up']}, {'osd': 6, 'uuid': 'e07d846c-ae42-4fea-aa5c-3a25835d18a6', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 0, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 47, 'up_thru': 244, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.50:6808', 'nonce': 255960422}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.50:6809', 'nonce': 255960422}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.50:6811', 'nonce': 255960422}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.50:6810', 'nonce': 255960422}]}, 'public_addr': '172.21.15.50:6808/255960422', 'cluster_addr': '172.21.15.50:6809/255960422', 'heartbeat_back_addr': '172.21.15.50:6811/255960422', 'heartbeat_front_addr': '172.21.15.50:6810/255960422', 'state': ['exists', 'up']}, {'osd': 7, 'uuid': 'c9dac3b7-d651-489d-b96e-5b10fa32d9af', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 0.4741668701171875, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 56, 'up_thru': 414, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.50:6812', 'nonce': 874923904}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.50:6813', 'nonce': 874923904}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.50:6815', 'nonce': 874923904}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.50:6814', 'nonce': 874923904}]}, 'public_addr': '172.21.15.50:6812/874923904', 'cluster_addr': '172.21.15.50:6813/874923904', 'heartbeat_back_addr': '172.21.15.50:6815/874923904', 'heartbeat_front_addr': '172.21.15.50:6814/874923904', 'state': ['exists', 'up']}], 'osd_xinfo': [{'osd': 0, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540719139924082687, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-18T08:19:03.356280+0000', 'dead_epoch': 0}, {'osd': 1, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540719139924082687, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-18T08:19:36.265311+0000', 'dead_epoch': 243}, {'osd': 2, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540719139924082687, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-18T08:19:42.729801+0000', 'dead_epoch': 0}, {'osd': 3, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540719139924082687, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-18T08:19:22.484015+0000', 'dead_epoch': 0}, {'osd': 4, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540719139924082687, 'old_weight': 65536, 'last_purged_snaps_scrub': '2024-09-18T08:19:20.960569+0000', 'dead_epoch': 0}, {'osd': 5, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540719139924082687, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-18T08:19:43.501096+0000', 'dead_epoch': 0}, {'osd': 6, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540719139924082687, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-18T08:19:22.946550+0000', 'dead_epoch': 0}, {'osd': 7, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540719139924082687, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-18T08:19:09.632672+0000', 'dead_epoch': 0}], 'pg_upmap': [{'pgid': '3.c', 'osds': [3, 2, 6]}, {'pgid': '6.3', 'osds': [5, 3, 2]}], 'pg_upmap_items': [{'pgid': '3.8', 'mappings': [{'from': 3, 'to': 2}]}, {'pgid': '4.7', 'mappings': [{'from': 1, 'to': 2}]}, {'pgid': '4.14', 'mappings': [{'from': 1, 'to': 2}]}, {'pgid': '4.18', 'mappings': [{'from': 1, 'to': 2}]}, {'pgid': '4.1b', 'mappings': [{'from': 3, 'to': 5}]}, {'pgid': '4.1d', 'mappings': [{'from': 1, 'to': 0}]}, {'pgid': '5.b', 'mappings': [{'from': 0, 'to': 3}]}, {'pgid': '6.8', 'mappings': [{'from': 7, 'to': 3}]}], 'pg_upmap_primaries': [], 'pg_temp': [{'pgid': '11.4', 'osds': [7, 6]}], 'primary_temp': [], 'blocklist': {'172.21.15.7:0/398367658': '2024-09-19T08:03:17.052903+0000', '172.21.15.7:6800/2918496291': '2024-09-19T07:57:31.015269+0000', '172.21.15.7:0/2514354773': '2024-09-19T07:57:31.015269+0000', '172.21.15.7:0/3924911335': '2024-09-19T07:57:31.015269+0000', '172.21.15.7:0/1668598084': '2024-09-19T07:57:57.346149+0000', '172.21.15.7:0/4272027292': '2024-09-19T07:57:31.015269+0000', '172.21.15.7:6800/1013061071': '2024-09-19T07:57:57.346149+0000', '172.21.15.7:6800/1491061076': '2024-09-19T08:03:17.052903+0000', '172.21.15.7:0/533727535': '2024-09-19T07:57:57.346149+0000', '172.21.15.7:0/1605217610': '2024-09-19T08:03:17.052903+0000', '172.21.15.7:0/1722968065': '2024-09-19T08:03:17.052903+0000', '172.21.15.7:0/1277549846': '2024-09-19T07:57:57.346149+0000', '172.21.15.7:0/232452890': '2024-09-19T08:03:17.052903+0000', '172.21.15.7:0/3881670470': '2024-09-19T08:03:17.052903+0000'}, 'range_blocklist': {}, 'erasure_code_profiles': {'default': {'crush-failure-domain': 'osd', 'k': '2', 'm': '1', 'plugin': 'jerasure', 'technique': 'reed_sol_van'}}, 'removed_snaps_queue': [], 'new_removed_snaps': [], 'new_purged_snaps': [], 'crush_node_flags': {}, 'device_class_flags': {}, 'stretch_mode': {'stretch_mode_enabled': False, 'stretch_bucket_count': 0, 'degraded_stretch_mode': 0, 'recovering_stretch_mode': 0, 'stretch_mode_bucket': 0}} 2024-09-18T08:20:24.346 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid 779349ca-7593-11ef-bceb-c7b262605968 -- ceph pg dump --format=json 2024-09-18T08:20:24.364 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7 2024-09-18T08:20:24.389 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:20:24.390 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:20:24.391 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7' 2024-09-18T08:20:24.403 INFO:teuthology.orchestra.run.smithi007.stdout:{ 2024-09-18T08:20:24.403 INFO:teuthology.orchestra.run.smithi007.stdout: "ops": [], 2024-09-18T08:20:24.403 INFO:teuthology.orchestra.run.smithi007.stdout: "complaint_time": 30, 2024-09-18T08:20:24.403 INFO:teuthology.orchestra.run.smithi007.stdout: "num_blocked_ops": 0 2024-09-18T08:20:24.403 INFO:teuthology.orchestra.run.smithi007.stdout:} 2024-09-18T08:20:24.492 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0 2024-09-18T08:20:24.533 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:20:24.534 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:20:24.534 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0' 2024-09-18T08:20:24.635 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5 2024-09-18T08:20:24.664 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:20:24.665 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:20:24.665 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5' 2024-09-18T08:20:24.730 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:20:24 smithi007 ceph-mon[29459]: 3.7 scrub starts 2024-09-18T08:20:24.730 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:20:24 smithi007 ceph-mon[29459]: 3.7 scrub ok 2024-09-18T08:20:24.731 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:20:24 smithi007 ceph-mon[29459]: pgmap v881: 158 pgs: 158 active+clean; 5.1 GiB data, 20 GiB used, 606 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 901 op/s; 3.8 MiB/s, 60 objects/s recovering 2024-09-18T08:20:24.731 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:20:24 smithi007 ceph-mon[39249]: 3.7 scrub starts 2024-09-18T08:20:24.731 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:20:24 smithi007 ceph-mon[39249]: 3.7 scrub ok 2024-09-18T08:20:24.731 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:20:24 smithi007 ceph-mon[39249]: pgmap v881: 158 pgs: 158 active+clean; 5.1 GiB data, 20 GiB used, 606 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 901 op/s; 3.8 MiB/s, 60 objects/s recovering 2024-09-18T08:20:24.767 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6 2024-09-18T08:20:24.799 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:20:24.800 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:20:24.800 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6' 2024-09-18T08:20:24.901 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2 2024-09-18T08:20:24.904 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:20:24 smithi050 ceph-mon[32371]: 3.7 scrub starts 2024-09-18T08:20:24.905 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:20:24 smithi050 ceph-mon[32371]: 3.7 scrub ok 2024-09-18T08:20:24.905 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:20:24 smithi050 ceph-mon[32371]: pgmap v881: 158 pgs: 158 active+clean; 5.1 GiB data, 20 GiB used, 606 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 901 op/s; 3.8 MiB/s, 60 objects/s recovering 2024-09-18T08:20:24.938 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:20:24.939 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:20:24.939 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2' 2024-09-18T08:20:25.040 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3 2024-09-18T08:20:25.079 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:20:25.080 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:20:25.081 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3' 2024-09-18T08:20:25.182 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2 2024-09-18T08:20:25.207 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid 779349ca-7593-11ef-bceb-c7b262605968 -- ceph daemon osd.2 dump_historic_ops 2024-09-18T08:20:25.224 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:20:25.224 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:20:25.225 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2' 2024-09-18T08:20:25.326 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0 2024-09-18T08:20:25.386 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:20:25.386 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:20:25.387 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0' 2024-09-18T08:20:25.488 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1 2024-09-18T08:20:25.530 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:20:25.532 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:20:25.532 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1' 2024-09-18T08:20:25.633 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3 2024-09-18T08:20:25.683 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:20:25.684 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:20:25.684 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3' 2024-09-18T08:20:25.785 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4 2024-09-18T08:20:25.821 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:20:25.822 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:20:25.823 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4' 2024-09-18T08:20:25.905 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:20:25 smithi050 ceph-mon[32371]: 5.9 scrub starts 2024-09-18T08:20:25.905 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:20:25 smithi050 ceph-mon[32371]: 5.9 scrub ok 2024-09-18T08:20:25.905 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:20:25 smithi050 ceph-mon[32371]: 6.e scrub starts 2024-09-18T08:20:25.905 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:20:25 smithi050 ceph-mon[32371]: 6.e scrub ok 2024-09-18T08:20:25.924 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4 2024-09-18T08:20:25.957 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:20:25.958 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:20:25.958 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4' 2024-09-18T08:20:25.979 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:20:25 smithi007 ceph-mon[29459]: 5.9 scrub starts 2024-09-18T08:20:25.980 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:20:25 smithi007 ceph-mon[29459]: 5.9 scrub ok 2024-09-18T08:20:25.980 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:20:25 smithi007 ceph-mon[29459]: 6.e scrub starts 2024-09-18T08:20:25.980 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:20:25 smithi007 ceph-mon[29459]: 6.e scrub ok 2024-09-18T08:20:25.980 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:20:25 smithi007 ceph-mon[39249]: 5.9 scrub starts 2024-09-18T08:20:25.980 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:20:25 smithi007 ceph-mon[39249]: 5.9 scrub ok 2024-09-18T08:20:25.980 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:20:25 smithi007 ceph-mon[39249]: 6.e scrub starts 2024-09-18T08:20:25.980 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:20:25 smithi007 ceph-mon[39249]: 6.e scrub ok 2024-09-18T08:20:26.001 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/779349ca-7593-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T08:20:26.058 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4 2024-09-18T08:20:26.082 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/779349ca-7593-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T08:20:26.085 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:20:26.086 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:20:26.086 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4' 2024-09-18T08:20:26.187 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3 2024-09-18T08:20:26.231 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:20:26.232 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:20:26.232 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3' 2024-09-18T08:20:26.333 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0 2024-09-18T08:20:26.363 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:20:26.364 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:20:26.364 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0' 2024-09-18T08:20:26.465 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4 2024-09-18T08:20:26.500 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:20:26.501 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:20:26.501 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4' 2024-09-18T08:20:26.602 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6 2024-09-18T08:20:26.629 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:20:26.630 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:20:26.630 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6' 2024-09-18T08:20:26.731 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4 2024-09-18T08:20:26.769 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:20:26.769 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:20:26.770 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4' 2024-09-18T08:20:26.870 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0 2024-09-18T08:20:26.905 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:20:26 smithi050 ceph-mon[32371]: 1.4 scrub starts 2024-09-18T08:20:26.905 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:20:26 smithi050 ceph-mon[32371]: 1.4 scrub ok 2024-09-18T08:20:26.905 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:20:26 smithi050 ceph-mon[32371]: pgmap v882: 158 pgs: 158 active+clean; 4.8 GiB data, 19 GiB used, 607 GiB / 626 GiB avail; 1023 B/s rd, 0 B/s wr, 1.19k op/s; 3.0 MiB/s, 48 objects/s recovering 2024-09-18T08:20:26.905 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:20:26 smithi050 ceph-mon[32371]: 6.3 scrub starts 2024-09-18T08:20:26.905 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:20:26 smithi050 ceph-mon[32371]: 6.3 scrub ok 2024-09-18T08:20:26.908 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:20:26.909 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:20:26.909 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0' 2024-09-18T08:20:26.979 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Sep 18 08:20:26 smithi007 ceph-779349ca-7593-11ef-bceb-c7b262605968-mgr-y[29688]: ::ffff:172.21.15.50 - - [18/Sep/2024:08:20:26] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-09-18T08:20:26.980 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:20:26 smithi007 ceph-mon[29459]: 1.4 scrub starts 2024-09-18T08:20:26.980 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:20:26 smithi007 ceph-mon[29459]: 1.4 scrub ok 2024-09-18T08:20:26.980 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:20:26 smithi007 ceph-mon[29459]: pgmap v882: 158 pgs: 158 active+clean; 4.8 GiB data, 19 GiB used, 607 GiB / 626 GiB avail; 1023 B/s rd, 0 B/s wr, 1.19k op/s; 3.0 MiB/s, 48 objects/s recovering 2024-09-18T08:20:26.980 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:20:26 smithi007 ceph-mon[29459]: 6.3 scrub starts 2024-09-18T08:20:26.980 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:20:26 smithi007 ceph-mon[29459]: 6.3 scrub ok 2024-09-18T08:20:26.981 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:20:26 smithi007 ceph-mon[39249]: 1.4 scrub starts 2024-09-18T08:20:26.981 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:20:26 smithi007 ceph-mon[39249]: 1.4 scrub ok 2024-09-18T08:20:26.981 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:20:26 smithi007 ceph-mon[39249]: pgmap v882: 158 pgs: 158 active+clean; 4.8 GiB data, 19 GiB used, 607 GiB / 626 GiB avail; 1023 B/s rd, 0 B/s wr, 1.19k op/s; 3.0 MiB/s, 48 objects/s recovering 2024-09-18T08:20:26.981 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:20:26 smithi007 ceph-mon[39249]: 6.3 scrub starts 2024-09-18T08:20:26.981 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:20:26 smithi007 ceph-mon[39249]: 6.3 scrub ok 2024-09-18T08:20:27.010 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5 2024-09-18T08:20:27.041 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:20:27.041 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:20:27.042 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5' 2024-09-18T08:20:27.142 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4 2024-09-18T08:20:27.179 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:20:27.179 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:20:27.179 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4' 2024-09-18T08:20:27.280 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2 2024-09-18T08:20:27.327 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:20:27.328 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:20:27.328 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2' 2024-09-18T08:20:27.429 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4 2024-09-18T08:20:27.457 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:20:27.458 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:20:27.459 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4' 2024-09-18T08:20:27.559 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5 2024-09-18T08:20:27.594 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:20:27.594 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:20:27.594 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5' 2024-09-18T08:20:27.695 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5 2024-09-18T08:20:27.717 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:20:27.718 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:20:27.718 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5' 2024-09-18T08:20:27.819 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6 2024-09-18T08:20:27.847 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:20:27.848 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:20:27.848 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6' 2024-09-18T08:20:27.905 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:20:27 smithi050 ceph-mon[32371]: 1.7 scrub starts 2024-09-18T08:20:27.905 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:20:27 smithi050 ceph-mon[32371]: 1.7 scrub ok 2024-09-18T08:20:27.905 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:20:27 smithi050 ceph-mon[32371]: 5.5 scrub starts 2024-09-18T08:20:27.905 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:20:27 smithi050 ceph-mon[32371]: 5.5 scrub ok 2024-09-18T08:20:27.949 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7 2024-09-18T08:20:27.979 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:20:27 smithi007 ceph-mon[29459]: 1.7 scrub starts 2024-09-18T08:20:27.980 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:20:27 smithi007 ceph-mon[29459]: 1.7 scrub ok 2024-09-18T08:20:27.980 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:20:27 smithi007 ceph-mon[29459]: 5.5 scrub starts 2024-09-18T08:20:27.980 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:20:27 smithi007 ceph-mon[29459]: 5.5 scrub ok 2024-09-18T08:20:27.980 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:20:27 smithi007 ceph-mon[39249]: 1.7 scrub starts 2024-09-18T08:20:27.980 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:20:27 smithi007 ceph-mon[39249]: 1.7 scrub ok 2024-09-18T08:20:27.980 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:20:27 smithi007 ceph-mon[39249]: 5.5 scrub starts 2024-09-18T08:20:27.980 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:20:27 smithi007 ceph-mon[39249]: 5.5 scrub ok 2024-09-18T08:20:27.983 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:20:27.983 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:20:27.984 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7' 2024-09-18T08:20:28.044 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid 779349ca-7593-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-18T08:20:28.085 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1 2024-09-18T08:20:28.111 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:20:28.112 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:20:28.112 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1' 2024-09-18T08:20:28.213 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2 2024-09-18T08:20:28.254 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:20:28.255 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:20:28.255 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2' 2024-09-18T08:20:28.356 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4 2024-09-18T08:20:28.386 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:20:28.387 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:20:28.388 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4' 2024-09-18T08:20:28.489 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4 2024-09-18T08:20:28.520 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:20:28.520 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:20:28.521 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4' 2024-09-18T08:20:28.622 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1 2024-09-18T08:20:28.662 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:20:28.663 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:20:28.663 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1' 2024-09-18T08:20:28.764 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1 2024-09-18T08:20:28.795 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:20:28.796 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:20:28.796 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1' 2024-09-18T08:20:28.897 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4 2024-09-18T08:20:28.905 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:20:28 smithi050 ceph-mon[32371]: pgmap v883: 158 pgs: 158 active+clean; 4.8 GiB data, 19 GiB used, 607 GiB / 626 GiB avail; 927 B/s rd, 0 B/s wr, 1.08k op/s; 2.8 MiB/s, 44 objects/s recovering 2024-09-18T08:20:28.932 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:20:28.933 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:20:28.933 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4' 2024-09-18T08:20:28.979 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:20:28 smithi007 ceph-mon[29459]: pgmap v883: 158 pgs: 158 active+clean; 4.8 GiB data, 19 GiB used, 607 GiB / 626 GiB avail; 927 B/s rd, 0 B/s wr, 1.08k op/s; 2.8 MiB/s, 44 objects/s recovering 2024-09-18T08:20:28.980 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:20:28 smithi007 ceph-mon[39249]: pgmap v883: 158 pgs: 158 active+clean; 4.8 GiB data, 19 GiB used, 607 GiB / 626 GiB avail; 927 B/s rd, 0 B/s wr, 1.08k op/s; 2.8 MiB/s, 44 objects/s recovering 2024-09-18T08:20:29.034 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4 2024-09-18T08:20:29.059 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:20:29.060 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:20:29.060 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4' 2024-09-18T08:20:29.161 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6 2024-09-18T08:20:29.193 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:20:29.194 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:20:29.194 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6' 2024-09-18T08:20:29.296 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4 2024-09-18T08:20:29.321 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:20:29.322 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:20:29.322 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4' 2024-09-18T08:20:29.423 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3 2024-09-18T08:20:29.460 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:20:29.461 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:20:29.461 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3' 2024-09-18T08:20:29.562 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0 2024-09-18T08:20:29.581 INFO:teuthology.orchestra.run.smithi007.stderr:nodeep-scrub is unset 2024-09-18T08:20:29.602 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:20:29.603 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:20:29.603 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0' 2024-09-18T08:20:29.704 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1 2024-09-18T08:20:29.740 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:20:29.740 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:20:29.740 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1' 2024-09-18T08:20:29.841 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6 2024-09-18T08:20:29.870 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:20:29.871 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:20:29.871 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6' 2024-09-18T08:20:29.905 INFO:journalctl@ceph.iscsi.iscsi.a.smithi050.stdout:Sep 18 08:20:29 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-iscsi-iscsi-a[57782]: debug there is no tcmu-runner data available 2024-09-18T08:20:29.905 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:20:29 smithi050 ceph-mon[32371]: 6.1 scrub starts 2024-09-18T08:20:29.905 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:20:29 smithi050 ceph-mon[32371]: 6.1 scrub ok 2024-09-18T08:20:29.905 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:20:29 smithi050 ceph-mon[32371]: purged_snaps scrub starts 2024-09-18T08:20:29.905 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:20:29 smithi050 ceph-mon[32371]: purged_snaps scrub ok 2024-09-18T08:20:29.906 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:20:29 smithi050 ceph-mon[32371]: from='client.? 172.21.15.7:0/1720065045' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-18T08:20:29.972 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5 2024-09-18T08:20:29.980 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:20:29 smithi007 ceph-mon[29459]: 6.1 scrub starts 2024-09-18T08:20:29.980 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:20:29 smithi007 ceph-mon[29459]: 6.1 scrub ok 2024-09-18T08:20:29.980 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:20:29 smithi007 ceph-mon[29459]: purged_snaps scrub starts 2024-09-18T08:20:29.980 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:20:29 smithi007 ceph-mon[29459]: purged_snaps scrub ok 2024-09-18T08:20:29.980 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:20:29 smithi007 ceph-mon[29459]: from='client.? 172.21.15.7:0/1720065045' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-18T08:20:29.980 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:20:29 smithi007 ceph-mon[39249]: 6.1 scrub starts 2024-09-18T08:20:29.980 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:20:29 smithi007 ceph-mon[39249]: 6.1 scrub ok 2024-09-18T08:20:29.980 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:20:29 smithi007 ceph-mon[39249]: purged_snaps scrub starts 2024-09-18T08:20:29.981 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:20:29 smithi007 ceph-mon[39249]: purged_snaps scrub ok 2024-09-18T08:20:29.981 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:20:29 smithi007 ceph-mon[39249]: from='client.? 172.21.15.7:0/1720065045' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-18T08:20:30.005 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:20:30.005 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:20:30.005 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5' 2024-09-18T08:20:30.106 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3 2024-09-18T08:20:30.146 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:20:30.146 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:20:30.147 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3' 2024-09-18T08:20:30.197 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/779349ca-7593-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T08:20:30.248 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7 2024-09-18T08:20:30.276 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:20:30.277 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:20:30.277 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7' 2024-09-18T08:20:30.378 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1 2024-09-18T08:20:30.426 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:20:30.427 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:20:30.427 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1' 2024-09-18T08:20:30.528 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0 2024-09-18T08:20:30.563 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:20:30.564 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:20:30.564 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0' 2024-09-18T08:20:30.589 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-09-18T08:20:30.592 INFO:teuthology.orchestra.run.smithi007.stderr:dumped all 2024-09-18T08:20:30.665 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2 2024-09-18T08:20:30.699 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:20:30.700 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:20:30.700 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2' 2024-09-18T08:20:30.801 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5 2024-09-18T08:20:30.837 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:20:30.838 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:20:30.838 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5' 2024-09-18T08:20:30.905 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:20:30 smithi050 ceph-mon[32371]: 4.9 scrub starts 2024-09-18T08:20:30.905 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:20:30 smithi050 ceph-mon[32371]: 4.9 scrub ok 2024-09-18T08:20:30.905 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:20:30 smithi050 ceph-mon[32371]: pgmap v884: 158 pgs: 158 active+clean; 4.5 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.36k op/s; 2.5 MiB/s, 40 objects/s recovering 2024-09-18T08:20:30.905 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:20:30 smithi050 ceph-mon[32371]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-18T08:20:30.905 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:20:30 smithi050 ceph-mon[32371]: Cluster is now healthy 2024-09-18T08:20:30.905 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:20:30 smithi050 ceph-mon[32371]: from='client.? 172.21.15.7:0/1720065045' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-18T08:20:30.905 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:20:30 smithi050 ceph-mon[32371]: osdmap e439: 8 total, 8 up, 7 in 2024-09-18T08:20:30.905 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:20:30 smithi050 ceph-mon[32371]: purged_snaps scrub starts 2024-09-18T08:20:30.905 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:20:30 smithi050 ceph-mon[32371]: purged_snaps scrub ok 2024-09-18T08:20:30.939 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6 2024-09-18T08:20:30.969 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:20:30.970 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:20:30.970 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6' 2024-09-18T08:20:30.979 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:20:30 smithi007 ceph-mon[39249]: 4.9 scrub starts 2024-09-18T08:20:30.980 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:20:30 smithi007 ceph-mon[39249]: 4.9 scrub ok 2024-09-18T08:20:30.980 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:20:30 smithi007 ceph-mon[39249]: pgmap v884: 158 pgs: 158 active+clean; 4.5 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.36k op/s; 2.5 MiB/s, 40 objects/s recovering 2024-09-18T08:20:30.980 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:20:30 smithi007 ceph-mon[39249]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-18T08:20:30.980 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:20:30 smithi007 ceph-mon[39249]: Cluster is now healthy 2024-09-18T08:20:30.980 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:20:30 smithi007 ceph-mon[39249]: from='client.? 172.21.15.7:0/1720065045' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-18T08:20:30.980 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:20:30 smithi007 ceph-mon[39249]: osdmap e439: 8 total, 8 up, 7 in 2024-09-18T08:20:30.980 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:20:30 smithi007 ceph-mon[39249]: purged_snaps scrub starts 2024-09-18T08:20:30.980 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:20:30 smithi007 ceph-mon[39249]: purged_snaps scrub ok 2024-09-18T08:20:30.980 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:20:30 smithi007 ceph-mon[29459]: 4.9 scrub starts 2024-09-18T08:20:30.981 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:20:30 smithi007 ceph-mon[29459]: 4.9 scrub ok 2024-09-18T08:20:30.981 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:20:30 smithi007 ceph-mon[29459]: pgmap v884: 158 pgs: 158 active+clean; 4.5 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.36k op/s; 2.5 MiB/s, 40 objects/s recovering 2024-09-18T08:20:30.981 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:20:30 smithi007 ceph-mon[29459]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-18T08:20:30.981 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:20:30 smithi007 ceph-mon[29459]: Cluster is now healthy 2024-09-18T08:20:30.981 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:20:30 smithi007 ceph-mon[29459]: from='client.? 172.21.15.7:0/1720065045' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-18T08:20:30.981 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:20:30 smithi007 ceph-mon[29459]: osdmap e439: 8 total, 8 up, 7 in 2024-09-18T08:20:30.981 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:20:30 smithi007 ceph-mon[29459]: purged_snaps scrub starts 2024-09-18T08:20:30.981 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:20:30 smithi007 ceph-mon[29459]: purged_snaps scrub ok 2024-09-18T08:20:31.071 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0 2024-09-18T08:20:31.109 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:20:31.110 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:20:31.110 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0' 2024-09-18T08:20:31.211 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5 2024-09-18T08:20:31.242 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:20:31.242 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:20:31.243 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5' 2024-09-18T08:20:31.343 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3 2024-09-18T08:20:31.374 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:20:31.375 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:20:31.375 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3' 2024-09-18T08:20:31.476 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7 2024-09-18T08:20:31.508 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:20:31.509 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:20:31.509 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7' 2024-09-18T08:20:31.610 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7 2024-09-18T08:20:31.642 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:20:31.643 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:20:31.643 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7' 2024-09-18T08:20:31.743 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0 2024-09-18T08:20:31.788 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:20:31.789 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:20:31.789 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0' 2024-09-18T08:20:31.890 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3 2024-09-18T08:20:31.905 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:20:31 smithi050 ceph-mon[32371]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-18T08:20:31.905 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:20:31 smithi050 ceph-mon[32371]: from='client.26201 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-18T08:20:31.905 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:20:31 smithi050 ceph-mon[32371]: osdmap e440: 8 total, 8 up, 7 in 2024-09-18T08:20:31.905 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:20:31 smithi050 ceph-mon[32371]: 1.a scrub starts 2024-09-18T08:20:31.905 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:20:31 smithi050 ceph-mon[32371]: 1.a scrub ok 2024-09-18T08:20:31.905 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:20:31 smithi050 ceph-mon[32371]: purged_snaps scrub starts 2024-09-18T08:20:31.905 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:20:31 smithi050 ceph-mon[32371]: purged_snaps scrub ok 2024-09-18T08:20:31.905 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:20:31 smithi050 ceph-mon[32371]: 3.8 scrub starts 2024-09-18T08:20:31.906 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:20:31 smithi050 ceph-mon[32371]: 3.8 scrub ok 2024-09-18T08:20:31.906 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:20:31 smithi050 ceph-mon[32371]: 6.10 scrub starts 2024-09-18T08:20:31.906 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:20:31 smithi050 ceph-mon[32371]: 6.10 scrub ok 2024-09-18T08:20:31.906 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:20:31 smithi050 ceph-mon[32371]: from='mgr.24470 172.21.15.7:0/2594390261' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-18T08:20:31.934 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:20:31.934 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:20:31.935 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3' 2024-09-18T08:20:31.980 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:20:31 smithi007 ceph-mon[29459]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-18T08:20:31.980 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:20:31 smithi007 ceph-mon[29459]: from='client.26201 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-18T08:20:31.980 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:20:31 smithi007 ceph-mon[29459]: osdmap e440: 8 total, 8 up, 7 in 2024-09-18T08:20:31.980 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:20:31 smithi007 ceph-mon[29459]: 1.a scrub starts 2024-09-18T08:20:31.980 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:20:31 smithi007 ceph-mon[29459]: 1.a scrub ok 2024-09-18T08:20:31.981 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:20:31 smithi007 ceph-mon[29459]: purged_snaps scrub starts 2024-09-18T08:20:31.981 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:20:31 smithi007 ceph-mon[29459]: purged_snaps scrub ok 2024-09-18T08:20:31.981 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:20:31 smithi007 ceph-mon[29459]: 3.8 scrub starts 2024-09-18T08:20:31.981 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:20:31 smithi007 ceph-mon[29459]: 3.8 scrub ok 2024-09-18T08:20:31.981 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:20:31 smithi007 ceph-mon[29459]: 6.10 scrub starts 2024-09-18T08:20:31.981 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:20:31 smithi007 ceph-mon[29459]: 6.10 scrub ok 2024-09-18T08:20:31.981 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:20:31 smithi007 ceph-mon[29459]: from='mgr.24470 172.21.15.7:0/2594390261' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-18T08:20:31.981 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:20:31 smithi007 ceph-mon[39249]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-18T08:20:31.981 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:20:31 smithi007 ceph-mon[39249]: from='client.26201 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-18T08:20:31.982 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:20:31 smithi007 ceph-mon[39249]: osdmap e440: 8 total, 8 up, 7 in 2024-09-18T08:20:31.982 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:20:31 smithi007 ceph-mon[39249]: 1.a scrub starts 2024-09-18T08:20:31.982 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:20:31 smithi007 ceph-mon[39249]: 1.a scrub ok 2024-09-18T08:20:31.982 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:20:31 smithi007 ceph-mon[39249]: purged_snaps scrub starts 2024-09-18T08:20:31.982 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:20:31 smithi007 ceph-mon[39249]: purged_snaps scrub ok 2024-09-18T08:20:31.982 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:20:31 smithi007 ceph-mon[39249]: 3.8 scrub starts 2024-09-18T08:20:31.982 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:20:31 smithi007 ceph-mon[39249]: 3.8 scrub ok 2024-09-18T08:20:31.982 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:20:31 smithi007 ceph-mon[39249]: 6.10 scrub starts 2024-09-18T08:20:31.982 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:20:31 smithi007 ceph-mon[39249]: 6.10 scrub ok 2024-09-18T08:20:31.982 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:20:31 smithi007 ceph-mon[39249]: from='mgr.24470 172.21.15.7:0/2594390261' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-18T08:20:32.036 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4 2024-09-18T08:20:32.066 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:20:32.067 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:20:32.067 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4' 2024-09-18T08:20:32.168 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6 2024-09-18T08:20:32.201 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:20:32.201 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:20:32.202 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6' 2024-09-18T08:20:32.303 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4 2024-09-18T08:20:32.332 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:20:32.332 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:20:32.333 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4' 2024-09-18T08:20:32.434 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7 2024-09-18T08:20:32.469 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:20:32.469 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:20:32.469 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7' 2024-09-18T08:20:32.570 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3 2024-09-18T08:20:32.610 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:20:32.611 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:20:32.611 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3' 2024-09-18T08:20:32.712 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6 2024-09-18T08:20:32.730 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:20:32 smithi007 ceph-mon[29459]: 6.1d scrub starts 2024-09-18T08:20:32.730 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:20:32 smithi007 ceph-mon[29459]: 6.1d scrub ok 2024-09-18T08:20:32.730 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:20:32 smithi007 ceph-mon[29459]: pgmap v887: 158 pgs: 158 active+clean; 4.4 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 639 B/s rd, 0 B/s wr, 1.46k op/s 2024-09-18T08:20:32.730 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:20:32 smithi007 ceph-mon[29459]: osdmap e441: 8 total, 8 up, 7 in 2024-09-18T08:20:32.730 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:20:32 smithi007 ceph-mon[29459]: 5.8 scrub starts 2024-09-18T08:20:32.730 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:20:32 smithi007 ceph-mon[29459]: 5.8 scrub ok 2024-09-18T08:20:32.730 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:20:32 smithi007 ceph-mon[29459]: 5.3 scrub starts 2024-09-18T08:20:32.730 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:20:32 smithi007 ceph-mon[29459]: 5.3 scrub ok 2024-09-18T08:20:32.731 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:20:32 smithi007 ceph-mon[39249]: 6.1d scrub starts 2024-09-18T08:20:32.731 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:20:32 smithi007 ceph-mon[39249]: 6.1d scrub ok 2024-09-18T08:20:32.731 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:20:32 smithi007 ceph-mon[39249]: pgmap v887: 158 pgs: 158 active+clean; 4.4 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 639 B/s rd, 0 B/s wr, 1.46k op/s 2024-09-18T08:20:32.731 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:20:32 smithi007 ceph-mon[39249]: osdmap e441: 8 total, 8 up, 7 in 2024-09-18T08:20:32.731 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:20:32 smithi007 ceph-mon[39249]: 5.8 scrub starts 2024-09-18T08:20:32.731 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:20:32 smithi007 ceph-mon[39249]: 5.8 scrub ok 2024-09-18T08:20:32.731 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:20:32 smithi007 ceph-mon[39249]: 5.3 scrub starts 2024-09-18T08:20:32.731 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:20:32 smithi007 ceph-mon[39249]: 5.3 scrub ok 2024-09-18T08:20:32.742 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:20:32.743 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:20:32.743 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6' 2024-09-18T08:20:32.844 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2 2024-09-18T08:20:32.890 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:20:32.891 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:20:32.891 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2' 2024-09-18T08:20:32.905 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:20:32 smithi050 ceph-mon[32371]: 6.1d scrub starts 2024-09-18T08:20:32.905 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:20:32 smithi050 ceph-mon[32371]: 6.1d scrub ok 2024-09-18T08:20:32.905 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:20:32 smithi050 ceph-mon[32371]: pgmap v887: 158 pgs: 158 active+clean; 4.4 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 639 B/s rd, 0 B/s wr, 1.46k op/s 2024-09-18T08:20:32.905 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:20:32 smithi050 ceph-mon[32371]: osdmap e441: 8 total, 8 up, 7 in 2024-09-18T08:20:32.905 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:20:32 smithi050 ceph-mon[32371]: 5.8 scrub starts 2024-09-18T08:20:32.905 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:20:32 smithi050 ceph-mon[32371]: 5.8 scrub ok 2024-09-18T08:20:32.905 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:20:32 smithi050 ceph-mon[32371]: 5.3 scrub starts 2024-09-18T08:20:32.905 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:20:32 smithi050 ceph-mon[32371]: 5.3 scrub ok 2024-09-18T08:20:32.992 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6 2024-09-18T08:20:33.021 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:20:33.022 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:20:33.022 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6' 2024-09-18T08:20:33.060 INFO:teuthology.orchestra.run.smithi007.stdout:{ 2024-09-18T08:20:33.060 INFO:teuthology.orchestra.run.smithi007.stdout: "size": 20, 2024-09-18T08:20:33.060 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 600, 2024-09-18T08:20:33.060 INFO:teuthology.orchestra.run.smithi007.stdout: "ops": [ 2024-09-18T08:20:33.060 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:20:33.060 INFO:teuthology.orchestra.run.smithi007.stdout: "description": "osd_op(client.15582.0:53557 9.0 9:0cf6806d:::benchmark_data_smithi007_124019_object53556:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e244)", 2024-09-18T08:20:33.060 INFO:teuthology.orchestra.run.smithi007.stdout: "initiated_at": "2024-09-18T08:11:41.343691+0000", 2024-09-18T08:20:33.061 INFO:teuthology.orchestra.run.smithi007.stdout: "age": 531.71252867600003, 2024-09-18T08:20:33.061 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.98520017699999995, 2024-09-18T08:20:33.061 INFO:teuthology.orchestra.run.smithi007.stdout: "continuous": false, 2024-09-18T08:20:33.061 INFO:teuthology.orchestra.run.smithi007.stdout: "type_data": { 2024-09-18T08:20:33.061 INFO:teuthology.orchestra.run.smithi007.stdout: "flag_point": "reached pg", 2024-09-18T08:20:33.061 INFO:teuthology.orchestra.run.smithi007.stdout: "client_info": { 2024-09-18T08:20:33.061 INFO:teuthology.orchestra.run.smithi007.stdout: "client": "client.15582", 2024-09-18T08:20:33.061 INFO:teuthology.orchestra.run.smithi007.stdout: "client_addr": "172.21.15.7:0/3598605839", 2024-09-18T08:20:33.061 INFO:teuthology.orchestra.run.smithi007.stdout: "tid": 53557 2024-09-18T08:20:33.061 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:20:33.061 INFO:teuthology.orchestra.run.smithi007.stdout: "events": [ 2024-09-18T08:20:33.062 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:20:33.062 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "initiated", 2024-09-18T08:20:33.062 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:11:41.343691+0000", 2024-09-18T08:20:33.062 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-09-18T08:20:33.062 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:20:33.062 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:20:33.062 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "header_read", 2024-09-18T08:20:33.062 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:11:41.343691+0000", 2024-09-18T08:20:33.062 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-09-18T08:20:33.062 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:20:33.062 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:20:33.062 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "throttled", 2024-09-18T08:20:33.063 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:11:41.343693+0000", 2024-09-18T08:20:33.063 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.804e-06 2024-09-18T08:20:33.063 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:20:33.063 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:20:33.063 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "all_read", 2024-09-18T08:20:33.063 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:11:41.343759+0000", 2024-09-18T08:20:33.063 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 6.6369000000000001e-05 2024-09-18T08:20:33.063 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:20:33.063 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:20:33.063 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "dispatched", 2024-09-18T08:20:33.063 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:11:41.343774+0000", 2024-09-18T08:20:33.063 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.5489999999999999e-05 2024-09-18T08:20:33.063 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:20:33.064 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:20:33.064 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "queued_for_pg", 2024-09-18T08:20:33.064 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:11:41.343785+0000", 2024-09-18T08:20:33.064 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.078e-05 2024-09-18T08:20:33.064 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:20:33.064 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:20:33.064 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-09-18T08:20:33.064 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:11:41.344215+0000", 2024-09-18T08:20:33.064 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.00042940799999999999 2024-09-18T08:20:33.064 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:20:33.064 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:20:33.064 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "waiting for peered", 2024-09-18T08:20:33.064 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:11:41.344258+0000", 2024-09-18T08:20:33.065 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 4.3460000000000001e-05 2024-09-18T08:20:33.065 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:20:33.065 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:20:33.065 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-09-18T08:20:33.065 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:11:42.326886+0000", 2024-09-18T08:20:33.065 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.98262749800000004 2024-09-18T08:20:33.065 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:20:33.065 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:20:33.065 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "done", 2024-09-18T08:20:33.065 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:11:42.328891+0000", 2024-09-18T08:20:33.065 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.002005368 2024-09-18T08:20:33.065 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-09-18T08:20:33.065 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-09-18T08:20:33.066 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-09-18T08:20:33.066 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:20:33.066 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:20:33.066 INFO:teuthology.orchestra.run.smithi007.stdout: "description": "osd_op(client.15582.0:53566 9.0 9:09a05dd0:::benchmark_data_smithi007_124019_object53565:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e244)", 2024-09-18T08:20:33.066 INFO:teuthology.orchestra.run.smithi007.stdout: "initiated_at": "2024-09-18T08:11:41.343807+0000", 2024-09-18T08:20:33.066 INFO:teuthology.orchestra.run.smithi007.stdout: "age": 531.71241284799999, 2024-09-18T08:20:33.066 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.007200514, 2024-09-18T08:20:33.066 INFO:teuthology.orchestra.run.smithi007.stdout: "continuous": false, 2024-09-18T08:20:33.066 INFO:teuthology.orchestra.run.smithi007.stdout: "type_data": { 2024-09-18T08:20:33.066 INFO:teuthology.orchestra.run.smithi007.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T08:20:33.066 INFO:teuthology.orchestra.run.smithi007.stdout: "client_info": { 2024-09-18T08:20:33.066 INFO:teuthology.orchestra.run.smithi007.stdout: "client": "client.15582", 2024-09-18T08:20:33.067 INFO:teuthology.orchestra.run.smithi007.stdout: "client_addr": "172.21.15.7:0/3598605839", 2024-09-18T08:20:33.067 INFO:teuthology.orchestra.run.smithi007.stdout: "tid": 53566 2024-09-18T08:20:33.067 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:20:33.067 INFO:teuthology.orchestra.run.smithi007.stdout: "events": [ 2024-09-18T08:20:33.067 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:20:33.067 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "initiated", 2024-09-18T08:20:33.067 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:11:41.343807+0000", 2024-09-18T08:20:33.067 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-09-18T08:20:33.067 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:20:33.067 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:20:33.067 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "header_read", 2024-09-18T08:20:33.067 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:11:41.343807+0000", 2024-09-18T08:20:33.067 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-09-18T08:20:33.068 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:20:33.068 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:20:33.068 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "throttled", 2024-09-18T08:20:33.068 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:11:41.343807+0000", 2024-09-18T08:20:33.068 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 5.3000000000000001e-07 2024-09-18T08:20:33.068 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:20:33.068 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:20:33.068 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "all_read", 2024-09-18T08:20:33.069 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:11:41.343826+0000", 2024-09-18T08:20:33.069 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.9338999999999999e-05 2024-09-18T08:20:33.069 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:20:33.069 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:20:33.069 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "dispatched", 2024-09-18T08:20:33.069 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:11:41.343833+0000", 2024-09-18T08:20:33.069 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 6.7710000000000001e-06 2024-09-18T08:20:33.069 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:20:33.069 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:20:33.069 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "queued_for_pg", 2024-09-18T08:20:33.069 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:11:41.343839+0000", 2024-09-18T08:20:33.069 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 5.4269999999999999e-06 2024-09-18T08:20:33.069 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:20:33.070 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:20:33.070 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-09-18T08:20:33.070 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:11:41.344332+0000", 2024-09-18T08:20:33.070 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.00049369799999999999 2024-09-18T08:20:33.070 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:20:33.070 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:20:33.070 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "waiting for peered", 2024-09-18T08:20:33.070 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:11:41.344339+0000", 2024-09-18T08:20:33.070 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 6.8589999999999999e-06 2024-09-18T08:20:33.070 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:20:33.070 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:20:33.070 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-09-18T08:20:33.070 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:11:42.328960+0000", 2024-09-18T08:20:33.070 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.98462126599999999 2024-09-18T08:20:33.071 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:20:33.071 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:20:33.071 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "started", 2024-09-18T08:20:33.071 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:11:42.329310+0000", 2024-09-18T08:20:33.071 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.00034983799999999999 2024-09-18T08:20:33.071 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:20:33.071 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:20:33.071 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "waiting for subops from 6", 2024-09-18T08:20:33.071 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:11:42.330445+0000", 2024-09-18T08:20:33.071 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.0011349859999999999 2024-09-18T08:20:33.071 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:20:33.071 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:20:33.071 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "op_commit", 2024-09-18T08:20:33.072 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:11:42.339902+0000", 2024-09-18T08:20:33.072 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.0094567869999999995 2024-09-18T08:20:33.072 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:20:33.072 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:20:33.072 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:20:33.072 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:11:42.350663+0000", 2024-09-18T08:20:33.072 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.010761299 2024-09-18T08:20:33.072 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:20:33.072 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:20:33.072 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "commit_sent", 2024-09-18T08:20:33.073 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:11:42.350832+0000", 2024-09-18T08:20:33.073 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.00016828399999999999 2024-09-18T08:20:33.073 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:20:33.073 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:20:33.073 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "done", 2024-09-18T08:20:33.073 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:11:42.351007+0000", 2024-09-18T08:20:33.073 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.00017542999999999999 2024-09-18T08:20:33.073 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-09-18T08:20:33.073 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-09-18T08:20:33.073 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-09-18T08:20:33.073 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:20:33.073 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:20:33.073 INFO:teuthology.orchestra.run.smithi007.stdout: "description": "osd_op(client.15582.0:53575 9.0 9:0f4fbab7:::benchmark_data_smithi007_124019_object53574:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e244)", 2024-09-18T08:20:33.074 INFO:teuthology.orchestra.run.smithi007.stdout: "initiated_at": "2024-09-18T08:11:41.343854+0000", 2024-09-18T08:20:33.074 INFO:teuthology.orchestra.run.smithi007.stdout: "age": 531.712365406, 2024-09-18T08:20:33.074 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.0074121789999999, 2024-09-18T08:20:33.074 INFO:teuthology.orchestra.run.smithi007.stdout: "continuous": false, 2024-09-18T08:20:33.074 INFO:teuthology.orchestra.run.smithi007.stdout: "type_data": { 2024-09-18T08:20:33.074 INFO:teuthology.orchestra.run.smithi007.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T08:20:33.074 INFO:teuthology.orchestra.run.smithi007.stdout: "client_info": { 2024-09-18T08:20:33.074 INFO:teuthology.orchestra.run.smithi007.stdout: "client": "client.15582", 2024-09-18T08:20:33.074 INFO:teuthology.orchestra.run.smithi007.stdout: "client_addr": "172.21.15.7:0/3598605839", 2024-09-18T08:20:33.074 INFO:teuthology.orchestra.run.smithi007.stdout: "tid": 53575 2024-09-18T08:20:33.074 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:20:33.074 INFO:teuthology.orchestra.run.smithi007.stdout: "events": [ 2024-09-18T08:20:33.074 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:20:33.075 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "initiated", 2024-09-18T08:20:33.075 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:11:41.343854+0000", 2024-09-18T08:20:33.075 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-09-18T08:20:33.075 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:20:33.075 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:20:33.075 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "header_read", 2024-09-18T08:20:33.075 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:11:41.343854+0000", 2024-09-18T08:20:33.075 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-09-18T08:20:33.075 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:20:33.075 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:20:33.076 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "throttled", 2024-09-18T08:20:33.076 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:11:41.343855+0000", 2024-09-18T08:20:33.076 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 5.5000000000000003e-07 2024-09-18T08:20:33.076 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:20:33.076 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:20:33.076 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "all_read", 2024-09-18T08:20:33.076 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:11:41.343878+0000", 2024-09-18T08:20:33.076 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 2.3740999999999999e-05 2024-09-18T08:20:33.076 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:20:33.077 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:20:33.077 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "dispatched", 2024-09-18T08:20:33.077 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:11:41.343886+0000", 2024-09-18T08:20:33.077 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 7.8809999999999996e-06 2024-09-18T08:20:33.077 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:20:33.077 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:20:33.077 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "queued_for_pg", 2024-09-18T08:20:33.077 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:11:41.343892+0000", 2024-09-18T08:20:33.077 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 5.8200000000000002e-06 2024-09-18T08:20:33.077 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:20:33.077 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:20:33.077 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-09-18T08:20:33.077 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:11:41.344821+0000", 2024-09-18T08:20:33.078 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.000928874 2024-09-18T08:20:33.078 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:20:33.078 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:20:33.078 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "waiting for peered", 2024-09-18T08:20:33.078 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:11:41.344861+0000", 2024-09-18T08:20:33.078 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 3.9919999999999997e-05 2024-09-18T08:20:33.079 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:20:33.079 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:20:33.079 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-09-18T08:20:33.079 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:11:42.334161+0000", 2024-09-18T08:20:33.079 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.98930047799999998 2024-09-18T08:20:33.079 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:20:33.079 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:20:33.079 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "started", 2024-09-18T08:20:33.079 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:11:42.334487+0000", 2024-09-18T08:20:33.079 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.00032586300000000002 2024-09-18T08:20:33.080 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:20:33.080 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:20:33.080 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "waiting for subops from 6", 2024-09-18T08:20:33.080 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:11:42.334655+0000", 2024-09-18T08:20:33.080 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.000167419 2024-09-18T08:20:33.080 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:20:33.080 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:20:33.080 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "op_commit", 2024-09-18T08:20:33.080 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:11:42.346631+0000", 2024-09-18T08:20:33.080 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.011976496 2024-09-18T08:20:33.080 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:20:33.080 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:20:33.080 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:20:33.081 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:11:42.351115+0000", 2024-09-18T08:20:33.081 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.0044843590000000003 2024-09-18T08:20:33.081 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:20:33.081 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:20:33.081 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "commit_sent", 2024-09-18T08:20:33.081 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:11:42.351173+0000", 2024-09-18T08:20:33.081 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 5.7061000000000001e-05 2024-09-18T08:20:33.081 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:20:33.081 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:20:33.081 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "done", 2024-09-18T08:20:33.081 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:11:42.351266+0000", 2024-09-18T08:20:33.081 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 9.3716999999999997e-05 2024-09-18T08:20:33.082 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-09-18T08:20:33.082 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-09-18T08:20:33.082 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-09-18T08:20:33.082 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:20:33.082 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:20:33.082 INFO:teuthology.orchestra.run.smithi007.stdout: "description": "osd_op(client.15582.0:53576 9.0 9:0f9d6982:::benchmark_data_smithi007_124019_object53575:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e244)", 2024-09-18T08:20:33.082 INFO:teuthology.orchestra.run.smithi007.stdout: "initiated_at": "2024-09-18T08:11:41.343901+0000", 2024-09-18T08:20:33.082 INFO:teuthology.orchestra.run.smithi007.stdout: "age": 531.71231818599995, 2024-09-18T08:20:33.082 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.01002054, 2024-09-18T08:20:33.082 INFO:teuthology.orchestra.run.smithi007.stdout: "continuous": false, 2024-09-18T08:20:33.082 INFO:teuthology.orchestra.run.smithi007.stdout: "type_data": { 2024-09-18T08:20:33.082 INFO:teuthology.orchestra.run.smithi007.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T08:20:33.082 INFO:teuthology.orchestra.run.smithi007.stdout: "client_info": { 2024-09-18T08:20:33.083 INFO:teuthology.orchestra.run.smithi007.stdout: "client": "client.15582", 2024-09-18T08:20:33.083 INFO:teuthology.orchestra.run.smithi007.stdout: "client_addr": "172.21.15.7:0/3598605839", 2024-09-18T08:20:33.083 INFO:teuthology.orchestra.run.smithi007.stdout: "tid": 53576 2024-09-18T08:20:33.083 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:20:33.083 INFO:teuthology.orchestra.run.smithi007.stdout: "events": [ 2024-09-18T08:20:33.083 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:20:33.083 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "initiated", 2024-09-18T08:20:33.083 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:11:41.343901+0000", 2024-09-18T08:20:33.083 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-09-18T08:20:33.083 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:20:33.083 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:20:33.083 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "header_read", 2024-09-18T08:20:33.083 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:11:41.343901+0000", 2024-09-18T08:20:33.084 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-09-18T08:20:33.084 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:20:33.084 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:20:33.084 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "throttled", 2024-09-18T08:20:33.084 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:11:41.343902+0000", 2024-09-18T08:20:33.084 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 4.7100000000000002e-07 2024-09-18T08:20:33.084 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:20:33.084 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:20:33.084 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "all_read", 2024-09-18T08:20:33.084 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:11:41.343923+0000", 2024-09-18T08:20:33.084 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 2.1009000000000002e-05 2024-09-18T08:20:33.084 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:20:33.084 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:20:33.085 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "dispatched", 2024-09-18T08:20:33.085 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:11:41.343930+0000", 2024-09-18T08:20:33.085 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 7.0040000000000002e-06 2024-09-18T08:20:33.085 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:20:33.085 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:20:33.085 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "queued_for_pg", 2024-09-18T08:20:33.085 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:11:41.343935+0000", 2024-09-18T08:20:33.085 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 5.0549999999999999e-06 2024-09-18T08:20:33.085 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:20:33.085 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:20:33.085 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-09-18T08:20:33.085 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:11:41.345163+0000", 2024-09-18T08:20:33.085 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.0012281200000000001 2024-09-18T08:20:33.086 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:20:33.086 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:20:33.086 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "waiting for peered", 2024-09-18T08:20:33.086 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:11:41.345169+0000", 2024-09-18T08:20:33.086 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 5.6999999999999996e-06 2024-09-18T08:20:33.086 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:20:33.086 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:20:33.086 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-09-18T08:20:33.086 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:11:42.336827+0000", 2024-09-18T08:20:33.086 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.99165844599999997 2024-09-18T08:20:33.086 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:20:33.087 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:20:33.087 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "started", 2024-09-18T08:20:33.087 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:11:42.336957+0000", 2024-09-18T08:20:33.087 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.00013009 2024-09-18T08:20:33.087 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:20:33.087 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:20:33.087 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "waiting for subops from 6", 2024-09-18T08:20:33.087 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:11:42.337097+0000", 2024-09-18T08:20:33.087 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.00014021899999999999 2024-09-18T08:20:33.087 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:20:33.088 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:20:33.088 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "op_commit", 2024-09-18T08:20:33.088 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:11:42.346639+0000", 2024-09-18T08:20:33.088 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.0095413370000000004 2024-09-18T08:20:33.088 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:20:33.088 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:20:33.088 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:20:33.088 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:11:42.353750+0000", 2024-09-18T08:20:33.088 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.0071108530000000003 2024-09-18T08:20:33.088 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:20:33.088 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:20:33.089 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "commit_sent", 2024-09-18T08:20:33.089 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:11:42.353813+0000", 2024-09-18T08:20:33.089 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 6.3721999999999997e-05 2024-09-18T08:20:33.089 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:20:33.089 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:20:33.089 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "done", 2024-09-18T08:20:33.090 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:11:42.353922+0000", 2024-09-18T08:20:33.090 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.000108514 2024-09-18T08:20:33.090 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-09-18T08:20:33.090 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-09-18T08:20:33.090 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-09-18T08:20:33.090 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:20:33.090 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:20:33.090 INFO:teuthology.orchestra.run.smithi007.stdout: "description": "osd_op(client.24356.0:2782 5.d 5:b4812045:::notify.4:head [watch reconnect cookie 94700888684160 gen 1] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e292)", 2024-09-18T08:20:33.091 INFO:teuthology.orchestra.run.smithi007.stdout: "initiated_at": "2024-09-18T08:14:04.565391+0000", 2024-09-18T08:20:33.091 INFO:teuthology.orchestra.run.smithi007.stdout: "age": 388.49082820500001, 2024-09-18T08:20:33.091 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.0413668460000001, 2024-09-18T08:20:33.091 INFO:teuthology.orchestra.run.smithi007.stdout: "continuous": false, 2024-09-18T08:20:33.091 INFO:teuthology.orchestra.run.smithi007.stdout: "type_data": { 2024-09-18T08:20:33.091 INFO:teuthology.orchestra.run.smithi007.stdout: "flag_point": "started", 2024-09-18T08:20:33.091 INFO:teuthology.orchestra.run.smithi007.stdout: "client_info": { 2024-09-18T08:20:33.092 INFO:teuthology.orchestra.run.smithi007.stdout: "client": "client.24356", 2024-09-18T08:20:33.092 INFO:teuthology.orchestra.run.smithi007.stdout: "client_addr": "172.21.15.7:0/219168083", 2024-09-18T08:20:33.092 INFO:teuthology.orchestra.run.smithi007.stdout: "tid": 2782 2024-09-18T08:20:33.092 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:20:33.092 INFO:teuthology.orchestra.run.smithi007.stdout: "events": [ 2024-09-18T08:20:33.092 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:20:33.092 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "initiated", 2024-09-18T08:20:33.092 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:14:04.565391+0000", 2024-09-18T08:20:33.093 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-09-18T08:20:33.093 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:20:33.093 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:20:33.093 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "header_read", 2024-09-18T08:20:33.093 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:14:04.565391+0000", 2024-09-18T08:20:33.093 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-09-18T08:20:33.093 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:20:33.093 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:20:33.094 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "throttled", 2024-09-18T08:20:33.094 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:14:04.565393+0000", 2024-09-18T08:20:33.094 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.8199999999999999e-06 2024-09-18T08:20:33.094 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:20:33.094 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:20:33.094 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "all_read", 2024-09-18T08:20:33.094 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:14:04.565400+0000", 2024-09-18T08:20:33.095 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 6.968e-06 2024-09-18T08:20:33.095 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:20:33.095 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:20:33.095 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "dispatched", 2024-09-18T08:20:33.095 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:14:04.565413+0000", 2024-09-18T08:20:33.095 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.3003000000000001e-05 2024-09-18T08:20:33.095 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:20:33.096 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:20:33.096 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "queued_for_pg", 2024-09-18T08:20:33.096 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:14:04.565422+0000", 2024-09-18T08:20:33.096 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 9.3870000000000001e-06 2024-09-18T08:20:33.096 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:20:33.097 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:20:33.097 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-09-18T08:20:33.097 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:14:04.565500+0000", 2024-09-18T08:20:33.097 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 7.7395e-05 2024-09-18T08:20:33.097 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:20:33.098 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:20:33.098 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "op must wait for map", 2024-09-18T08:20:33.098 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:14:04.565506+0000", 2024-09-18T08:20:33.098 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 6.5180000000000001e-06 2024-09-18T08:20:33.098 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:20:33.098 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:20:33.098 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-09-18T08:20:33.098 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:14:04.576465+0000", 2024-09-18T08:20:33.099 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.010958183 2024-09-18T08:20:33.099 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:20:33.099 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:20:33.099 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "waiting for peered", 2024-09-18T08:20:33.100 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:14:04.576476+0000", 2024-09-18T08:20:33.100 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.1877e-05 2024-09-18T08:20:33.100 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:20:33.100 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:20:33.100 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-09-18T08:20:33.100 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:14:05.594014+0000", 2024-09-18T08:20:33.100 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.0175379360000001 2024-09-18T08:20:33.101 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:20:33.101 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:20:33.101 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "waiting for degraded object", 2024-09-18T08:20:33.101 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:14:05.594307+0000", 2024-09-18T08:20:33.101 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.00029306699999999999 2024-09-18T08:20:33.101 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:20:33.101 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:20:33.101 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-09-18T08:20:33.102 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:14:05.595721+0000", 2024-09-18T08:20:33.102 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.001413826 2024-09-18T08:20:33.102 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:20:33.102 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:20:33.102 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "started", 2024-09-18T08:20:33.102 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:14:05.595822+0000", 2024-09-18T08:20:33.102 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.000100814 2024-09-18T08:20:33.103 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:20:33.103 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:20:33.103 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "done", 2024-09-18T08:20:33.103 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:14:05.606758+0000", 2024-09-18T08:20:33.103 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.010936052 2024-09-18T08:20:33.103 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-09-18T08:20:33.103 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-09-18T08:20:33.103 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-09-18T08:20:33.104 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:20:33.104 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:20:33.104 INFO:teuthology.orchestra.run.smithi007.stdout: "description": "osd_op(client.15966.0:9476 10.9 10:9f7935f1:::benchmark_data_smithi007_148834_object9475:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e316)", 2024-09-18T08:20:33.104 INFO:teuthology.orchestra.run.smithi007.stdout: "initiated_at": "2024-09-18T08:14:51.654386+0000", 2024-09-18T08:20:33.104 INFO:teuthology.orchestra.run.smithi007.stdout: "age": 341.40183379199999, 2024-09-18T08:20:33.104 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 2.0447284200000002, 2024-09-18T08:20:33.104 INFO:teuthology.orchestra.run.smithi007.stdout: "continuous": false, 2024-09-18T08:20:33.105 INFO:teuthology.orchestra.run.smithi007.stdout: "type_data": { 2024-09-18T08:20:33.105 INFO:teuthology.orchestra.run.smithi007.stdout: "flag_point": "reached pg", 2024-09-18T08:20:33.105 INFO:teuthology.orchestra.run.smithi007.stdout: "client_info": { 2024-09-18T08:20:33.105 INFO:teuthology.orchestra.run.smithi007.stdout: "client": "client.15966", 2024-09-18T08:20:33.105 INFO:teuthology.orchestra.run.smithi007.stdout: "client_addr": "172.21.15.7:0/529784850", 2024-09-18T08:20:33.105 INFO:teuthology.orchestra.run.smithi007.stdout: "tid": 9476 2024-09-18T08:20:33.105 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:20:33.105 INFO:teuthology.orchestra.run.smithi007.stdout: "events": [ 2024-09-18T08:20:33.105 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:20:33.105 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "initiated", 2024-09-18T08:20:33.105 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:14:51.654386+0000", 2024-09-18T08:20:33.106 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-09-18T08:20:33.106 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:20:33.106 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:20:33.106 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "header_read", 2024-09-18T08:20:33.106 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:14:51.654386+0000", 2024-09-18T08:20:33.106 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-09-18T08:20:33.106 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:20:33.106 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:20:33.106 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "throttled", 2024-09-18T08:20:33.107 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:14:51.654387+0000", 2024-09-18T08:20:33.107 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.714e-06 2024-09-18T08:20:33.107 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:20:33.107 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:20:33.107 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "all_read", 2024-09-18T08:20:33.107 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:14:51.654415+0000", 2024-09-18T08:20:33.107 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 2.7216000000000002e-05 2024-09-18T08:20:33.107 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:20:33.107 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:20:33.107 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "dispatched", 2024-09-18T08:20:33.107 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:14:51.654427+0000", 2024-09-18T08:20:33.108 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.2340999999999999e-05 2024-09-18T08:20:33.108 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:20:33.108 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:20:33.108 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "queued_for_pg", 2024-09-18T08:20:33.108 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:14:51.654438+0000", 2024-09-18T08:20:33.108 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.0591000000000001e-05 2024-09-18T08:20:33.108 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:20:33.108 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:20:33.108 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-09-18T08:20:33.108 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:14:51.667444+0000", 2024-09-18T08:20:33.108 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.013006734000000001 2024-09-18T08:20:33.108 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:20:33.109 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:20:33.109 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "waiting for peered", 2024-09-18T08:20:33.109 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:14:51.667454+0000", 2024-09-18T08:20:33.109 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.0095000000000001e-05 2024-09-18T08:20:33.109 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:20:33.109 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:20:33.109 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-09-18T08:20:33.109 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:14:52.671410+0000", 2024-09-18T08:20:33.109 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.0039560489999999 2024-09-18T08:20:33.109 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:20:33.109 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:20:33.109 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "waiting for peered", 2024-09-18T08:20:33.109 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:14:52.671421+0000", 2024-09-18T08:20:33.110 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.0103e-05 2024-09-18T08:20:33.110 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:20:33.110 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:20:33.110 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-09-18T08:20:33.110 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:14:53.698810+0000", 2024-09-18T08:20:33.110 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.0273892680000001 2024-09-18T08:20:33.110 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:20:33.110 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:20:33.110 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "done", 2024-09-18T08:20:33.110 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:14:53.699114+0000", 2024-09-18T08:20:33.110 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.000304309 2024-09-18T08:20:33.110 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-09-18T08:20:33.110 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-09-18T08:20:33.110 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-09-18T08:20:33.111 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:20:33.111 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:20:33.111 INFO:teuthology.orchestra.run.smithi007.stdout: "description": "osd_op(client.15966.0:9493 10.9 10:99175b59:::benchmark_data_smithi007_148834_object9492:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e316)", 2024-09-18T08:20:33.111 INFO:teuthology.orchestra.run.smithi007.stdout: "initiated_at": "2024-09-18T08:14:51.668505+0000", 2024-09-18T08:20:33.111 INFO:teuthology.orchestra.run.smithi007.stdout: "age": 341.38771429600001, 2024-09-18T08:20:33.111 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 2.0445903859999999, 2024-09-18T08:20:33.111 INFO:teuthology.orchestra.run.smithi007.stdout: "continuous": false, 2024-09-18T08:20:33.111 INFO:teuthology.orchestra.run.smithi007.stdout: "type_data": { 2024-09-18T08:20:33.111 INFO:teuthology.orchestra.run.smithi007.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T08:20:33.111 INFO:teuthology.orchestra.run.smithi007.stdout: "client_info": { 2024-09-18T08:20:33.111 INFO:teuthology.orchestra.run.smithi007.stdout: "client": "client.15966", 2024-09-18T08:20:33.111 INFO:teuthology.orchestra.run.smithi007.stdout: "client_addr": "172.21.15.7:0/529784850", 2024-09-18T08:20:33.111 INFO:teuthology.orchestra.run.smithi007.stdout: "tid": 9493 2024-09-18T08:20:33.112 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:20:33.112 INFO:teuthology.orchestra.run.smithi007.stdout: "events": [ 2024-09-18T08:20:33.112 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:20:33.112 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "initiated", 2024-09-18T08:20:33.112 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:14:51.668505+0000", 2024-09-18T08:20:33.112 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-09-18T08:20:33.112 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:20:33.112 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:20:33.112 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "header_read", 2024-09-18T08:20:33.112 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:14:51.668505+0000", 2024-09-18T08:20:33.112 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-09-18T08:20:33.113 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:20:33.113 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:20:33.113 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "throttled", 2024-09-18T08:20:33.113 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:14:51.668507+0000", 2024-09-18T08:20:33.113 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.7999999999999999e-06 2024-09-18T08:20:33.113 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:20:33.113 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:20:33.113 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "all_read", 2024-09-18T08:20:33.113 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:14:51.668533+0000", 2024-09-18T08:20:33.113 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 2.6472e-05 2024-09-18T08:20:33.113 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:20:33.114 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:20:33.114 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "dispatched", 2024-09-18T08:20:33.114 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:14:51.668546+0000", 2024-09-18T08:20:33.114 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.3064999999999999e-05 2024-09-18T08:20:33.114 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:20:33.114 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:20:33.114 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "queued_for_pg", 2024-09-18T08:20:33.114 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:14:51.668630+0000", 2024-09-18T08:20:33.114 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 8.3813000000000005e-05 2024-09-18T08:20:33.114 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:20:33.114 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:20:33.115 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-09-18T08:20:33.115 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:14:51.668810+0000", 2024-09-18T08:20:33.115 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.00018000500000000001 2024-09-18T08:20:33.115 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:20:33.115 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:20:33.115 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "waiting for peered", 2024-09-18T08:20:33.116 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:14:51.668820+0000", 2024-09-18T08:20:33.116 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 9.8770000000000005e-06 2024-09-18T08:20:33.116 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:20:33.116 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:20:33.116 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-09-18T08:20:33.117 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:14:52.671448+0000", 2024-09-18T08:20:33.117 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.0026282019999999 2024-09-18T08:20:33.117 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:20:33.117 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:20:33.117 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "waiting for peered", 2024-09-18T08:20:33.117 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:14:52.671457+0000", 2024-09-18T08:20:33.117 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 8.4859999999999997e-06 2024-09-18T08:20:33.118 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:20:33.118 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:20:33.118 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-09-18T08:20:33.118 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:14:53.699656+0000", 2024-09-18T08:20:33.118 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.0281992200000001 2024-09-18T08:20:33.118 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:20:33.118 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:20:33.118 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "started", 2024-09-18T08:20:33.119 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:14:53.699827+0000", 2024-09-18T08:20:33.119 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.00017075200000000001 2024-09-18T08:20:33.119 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:20:33.119 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:20:33.119 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "waiting for subops from 3,4", 2024-09-18T08:20:33.119 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:14:53.700023+0000", 2024-09-18T08:20:33.119 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.00019650399999999999 2024-09-18T08:20:33.119 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:20:33.119 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:20:33.119 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:20:33.119 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:14:53.701118+0000", 2024-09-18T08:20:33.119 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.0010951069999999999 2024-09-18T08:20:33.119 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:20:33.120 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:20:33.120 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:20:33.120 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:14:53.702736+0000", 2024-09-18T08:20:33.120 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.001617063 2024-09-18T08:20:33.120 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:20:33.120 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:20:33.120 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "op_commit", 2024-09-18T08:20:33.120 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:14:53.712454+0000", 2024-09-18T08:20:33.120 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.0097184779999999991 2024-09-18T08:20:33.120 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:20:33.120 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:20:33.120 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "commit_sent", 2024-09-18T08:20:33.121 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:14:53.712512+0000", 2024-09-18T08:20:33.121 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 5.7602999999999997e-05 2024-09-18T08:20:33.121 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:20:33.121 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:20:33.121 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "done", 2024-09-18T08:20:33.121 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:14:53.713096+0000", 2024-09-18T08:20:33.121 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.00058393900000000001 2024-09-18T08:20:33.121 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-09-18T08:20:33.121 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-09-18T08:20:33.121 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-09-18T08:20:33.121 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:20:33.121 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:20:33.121 INFO:teuthology.orchestra.run.smithi007.stdout: "description": "osd_op(client.15966.0:9495 10.3 10:ceed0e84:::benchmark_data_smithi007_148834_object9494:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e316)", 2024-09-18T08:20:33.122 INFO:teuthology.orchestra.run.smithi007.stdout: "initiated_at": "2024-09-18T08:14:51.668764+0000", 2024-09-18T08:20:33.122 INFO:teuthology.orchestra.run.smithi007.stdout: "age": 341.38745559699998, 2024-09-18T08:20:33.122 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 2.0458681990000001, 2024-09-18T08:20:33.122 INFO:teuthology.orchestra.run.smithi007.stdout: "continuous": false, 2024-09-18T08:20:33.122 INFO:teuthology.orchestra.run.smithi007.stdout: "type_data": { 2024-09-18T08:20:33.122 INFO:teuthology.orchestra.run.smithi007.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T08:20:33.122 INFO:teuthology.orchestra.run.smithi007.stdout: "client_info": { 2024-09-18T08:20:33.122 INFO:teuthology.orchestra.run.smithi007.stdout: "client": "client.15966", 2024-09-18T08:20:33.122 INFO:teuthology.orchestra.run.smithi007.stdout: "client_addr": "172.21.15.7:0/529784850", 2024-09-18T08:20:33.122 INFO:teuthology.orchestra.run.smithi007.stdout: "tid": 9495 2024-09-18T08:20:33.122 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:20:33.123 INFO:teuthology.orchestra.run.smithi007.stdout: "events": [ 2024-09-18T08:20:33.123 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:20:33.123 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "initiated", 2024-09-18T08:20:33.123 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:14:51.668764+0000", 2024-09-18T08:20:33.123 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-09-18T08:20:33.123 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:20:33.123 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:20:33.123 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "header_read", 2024-09-18T08:20:33.123 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:14:51.668764+0000", 2024-09-18T08:20:33.123 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-09-18T08:20:33.123 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:20:33.123 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:20:33.124 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "throttled", 2024-09-18T08:20:33.124 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:14:51.668765+0000", 2024-09-18T08:20:33.124 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 9.0999999999999997e-07 2024-09-18T08:20:33.124 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:20:33.124 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:20:33.124 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "all_read", 2024-09-18T08:20:33.124 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:14:51.668788+0000", 2024-09-18T08:20:33.124 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 2.3133e-05 2024-09-18T08:20:33.124 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:20:33.124 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:20:33.124 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "dispatched", 2024-09-18T08:20:33.124 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:14:51.668802+0000", 2024-09-18T08:20:33.125 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.3984e-05 2024-09-18T08:20:33.125 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:20:33.125 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:20:33.125 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "queued_for_pg", 2024-09-18T08:20:33.125 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:14:51.668809+0000", 2024-09-18T08:20:33.125 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 6.7909999999999999e-06 2024-09-18T08:20:33.125 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:20:33.125 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:20:33.125 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-09-18T08:20:33.125 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:14:51.668866+0000", 2024-09-18T08:20:33.125 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 5.7133999999999999e-05 2024-09-18T08:20:33.126 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:20:33.126 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:20:33.126 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "waiting for peered", 2024-09-18T08:20:33.126 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:14:51.668874+0000", 2024-09-18T08:20:33.126 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 8.5310000000000001e-06 2024-09-18T08:20:33.126 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:20:33.126 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:20:33.126 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-09-18T08:20:33.126 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:14:52.654764+0000", 2024-09-18T08:20:33.126 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.98588983399999996 2024-09-18T08:20:33.126 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:20:33.127 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:20:33.127 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "waiting for peered", 2024-09-18T08:20:33.127 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:14:52.654779+0000", 2024-09-18T08:20:33.127 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.4501999999999999e-05 2024-09-18T08:20:33.127 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:20:33.127 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:20:33.127 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-09-18T08:20:33.127 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:14:53.695130+0000", 2024-09-18T08:20:33.127 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.0403511240000001 2024-09-18T08:20:33.128 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:20:33.128 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:20:33.128 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "started", 2024-09-18T08:20:33.128 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:14:53.695281+0000", 2024-09-18T08:20:33.128 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.00015081200000000001 2024-09-18T08:20:33.128 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:20:33.128 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:20:33.128 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "waiting for subops from 3,4", 2024-09-18T08:20:33.129 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:14:53.695455+0000", 2024-09-18T08:20:33.129 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.00017440500000000001 2024-09-18T08:20:33.129 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:20:33.129 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:20:33.129 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:20:33.129 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:14:53.699916+0000", 2024-09-18T08:20:33.129 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.0044613769999999999 2024-09-18T08:20:33.129 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:20:33.129 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:20:33.129 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:20:33.129 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:14:53.699961+0000", 2024-09-18T08:20:33.130 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 4.4354999999999998e-05 2024-09-18T08:20:33.130 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:20:33.130 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:20:33.130 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "op_commit", 2024-09-18T08:20:33.130 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:14:53.714186+0000", 2024-09-18T08:20:33.130 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.014225632 2024-09-18T08:20:33.130 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:20:33.130 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:20:33.131 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "commit_sent", 2024-09-18T08:20:33.131 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:14:53.714331+0000", 2024-09-18T08:20:33.131 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.00014509500000000001 2024-09-18T08:20:33.131 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:20:33.131 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:20:33.131 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "done", 2024-09-18T08:20:33.131 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:14:53.714632+0000", 2024-09-18T08:20:33.131 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.00030058 2024-09-18T08:20:33.131 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-09-18T08:20:33.131 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-09-18T08:20:33.131 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-09-18T08:20:33.131 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:20:33.131 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:20:33.131 INFO:teuthology.orchestra.run.smithi007.stdout: "description": "osd_op(client.15966.0:9501 10.3 10:ca7811de:::benchmark_data_smithi007_148834_object9500:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e316)", 2024-09-18T08:20:33.132 INFO:teuthology.orchestra.run.smithi007.stdout: "initiated_at": "2024-09-18T08:14:51.676714+0000", 2024-09-18T08:20:33.132 INFO:teuthology.orchestra.run.smithi007.stdout: "age": 341.379505381, 2024-09-18T08:20:33.132 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 2.0382764990000002, 2024-09-18T08:20:33.132 INFO:teuthology.orchestra.run.smithi007.stdout: "continuous": false, 2024-09-18T08:20:33.132 INFO:teuthology.orchestra.run.smithi007.stdout: "type_data": { 2024-09-18T08:20:33.132 INFO:teuthology.orchestra.run.smithi007.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T08:20:33.132 INFO:teuthology.orchestra.run.smithi007.stdout: "client_info": { 2024-09-18T08:20:33.132 INFO:teuthology.orchestra.run.smithi007.stdout: "client": "client.15966", 2024-09-18T08:20:33.132 INFO:teuthology.orchestra.run.smithi007.stdout: "client_addr": "172.21.15.7:0/529784850", 2024-09-18T08:20:33.132 INFO:teuthology.orchestra.run.smithi007.stdout: "tid": 9501 2024-09-18T08:20:33.132 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:20:33.132 INFO:teuthology.orchestra.run.smithi007.stdout: "events": [ 2024-09-18T08:20:33.133 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:20:33.133 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "initiated", 2024-09-18T08:20:33.133 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:14:51.676714+0000", 2024-09-18T08:20:33.133 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-09-18T08:20:33.133 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:20:33.133 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:20:33.133 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "header_read", 2024-09-18T08:20:33.133 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:14:51.676714+0000", 2024-09-18T08:20:33.133 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-09-18T08:20:33.134 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:20:33.134 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:20:33.134 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "throttled", 2024-09-18T08:20:33.134 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:14:51.676716+0000", 2024-09-18T08:20:33.134 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.6130000000000001e-06 2024-09-18T08:20:33.134 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:20:33.134 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:20:33.134 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "all_read", 2024-09-18T08:20:33.135 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:14:51.676738+0000", 2024-09-18T08:20:33.135 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 2.2714e-05 2024-09-18T08:20:33.135 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:20:33.135 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:20:33.135 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "dispatched", 2024-09-18T08:20:33.135 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:14:51.676753+0000", 2024-09-18T08:20:33.135 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.4439e-05 2024-09-18T08:20:33.135 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:20:33.135 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:20:33.135 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "queued_for_pg", 2024-09-18T08:20:33.135 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:14:51.676761+0000", 2024-09-18T08:20:33.135 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 7.8059999999999995e-06 2024-09-18T08:20:33.135 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:20:33.136 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:20:33.136 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-09-18T08:20:33.136 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:14:51.676814+0000", 2024-09-18T08:20:33.136 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 5.3504999999999997e-05 2024-09-18T08:20:33.136 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:20:33.136 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:20:33.136 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "waiting for peered", 2024-09-18T08:20:33.136 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:14:51.676821+0000", 2024-09-18T08:20:33.136 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 6.6379999999999998e-06 2024-09-18T08:20:33.136 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:20:33.137 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:20:33.137 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-09-18T08:20:33.137 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:14:52.654807+0000", 2024-09-18T08:20:33.137 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.97798668300000002 2024-09-18T08:20:33.137 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:20:33.137 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:20:33.137 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "waiting for peered", 2024-09-18T08:20:33.138 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:14:52.654816+0000", 2024-09-18T08:20:33.138 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 8.2949999999999994e-06 2024-09-18T08:20:33.138 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:20:33.138 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:20:33.138 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-09-18T08:20:33.138 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:14:53.697902+0000", 2024-09-18T08:20:33.139 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.0430859640000001 2024-09-18T08:20:33.139 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:20:33.139 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:20:33.139 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "started", 2024-09-18T08:20:33.139 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:14:53.698062+0000", 2024-09-18T08:20:33.139 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.00016033900000000001 2024-09-18T08:20:33.139 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:20:33.139 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:20:33.139 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "waiting for subops from 3,4", 2024-09-18T08:20:33.139 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:14:53.698234+0000", 2024-09-18T08:20:33.140 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.00017239800000000001 2024-09-18T08:20:33.140 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:20:33.140 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:20:33.140 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:20:33.140 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:14:53.700006+0000", 2024-09-18T08:20:33.140 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.001771667 2024-09-18T08:20:33.140 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:20:33.140 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:20:33.140 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:20:33.140 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:14:53.701950+0000", 2024-09-18T08:20:33.140 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.0019442330000000001 2024-09-18T08:20:33.140 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:20:33.141 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:20:33.141 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "op_commit", 2024-09-18T08:20:33.141 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:14:53.714642+0000", 2024-09-18T08:20:33.141 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.012691578 2024-09-18T08:20:33.141 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:20:33.141 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:20:33.141 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "commit_sent", 2024-09-18T08:20:33.141 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:14:53.714697+0000", 2024-09-18T08:20:33.141 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 5.5316999999999999e-05 2024-09-18T08:20:33.141 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:20:33.141 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:20:33.141 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "done", 2024-09-18T08:20:33.141 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:14:53.714991+0000", 2024-09-18T08:20:33.142 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.00029331000000000002 2024-09-18T08:20:33.142 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-09-18T08:20:33.142 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-09-18T08:20:33.142 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-09-18T08:20:33.142 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:20:33.142 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:20:33.142 INFO:teuthology.orchestra.run.smithi007.stdout: "description": "osd_op(client.15966.0:16666 10.3 10:c7bced0d:::benchmark_data_smithi007_148834_object16665:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e323)", 2024-09-18T08:20:33.142 INFO:teuthology.orchestra.run.smithi007.stdout: "initiated_at": "2024-09-18T08:14:59.368483+0000", 2024-09-18T08:20:33.142 INFO:teuthology.orchestra.run.smithi007.stdout: "age": 333.68773617099998, 2024-09-18T08:20:33.143 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.0017785939999999, 2024-09-18T08:20:33.143 INFO:teuthology.orchestra.run.smithi007.stdout: "continuous": false, 2024-09-18T08:20:33.143 INFO:teuthology.orchestra.run.smithi007.stdout: "type_data": { 2024-09-18T08:20:33.143 INFO:teuthology.orchestra.run.smithi007.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T08:20:33.143 INFO:teuthology.orchestra.run.smithi007.stdout: "client_info": { 2024-09-18T08:20:33.144 INFO:teuthology.orchestra.run.smithi007.stdout: "client": "client.15966", 2024-09-18T08:20:33.144 INFO:teuthology.orchestra.run.smithi007.stdout: "client_addr": "172.21.15.7:0/529784850", 2024-09-18T08:20:33.144 INFO:teuthology.orchestra.run.smithi007.stdout: "tid": 16666 2024-09-18T08:20:33.144 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:20:33.144 INFO:teuthology.orchestra.run.smithi007.stdout: "events": [ 2024-09-18T08:20:33.144 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:20:33.145 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "initiated", 2024-09-18T08:20:33.145 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:14:59.368483+0000", 2024-09-18T08:20:33.145 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-09-18T08:20:33.145 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:20:33.145 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:20:33.145 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "header_read", 2024-09-18T08:20:33.146 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:14:59.368483+0000", 2024-09-18T08:20:33.146 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-09-18T08:20:33.146 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:20:33.146 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:20:33.146 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "throttled", 2024-09-18T08:20:33.146 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:14:59.368485+0000", 2024-09-18T08:20:33.146 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.3769999999999999e-06 2024-09-18T08:20:33.147 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:20:33.147 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:20:33.147 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "all_read", 2024-09-18T08:20:33.147 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:14:59.368519+0000", 2024-09-18T08:20:33.147 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 3.4001000000000002e-05 2024-09-18T08:20:33.147 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:20:33.147 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:20:33.147 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "dispatched", 2024-09-18T08:20:33.148 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:14:59.368532+0000", 2024-09-18T08:20:33.148 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.2931999999999999e-05 2024-09-18T08:20:33.148 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:20:33.148 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:20:33.148 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "queued_for_pg", 2024-09-18T08:20:33.149 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:14:59.368541+0000", 2024-09-18T08:20:33.149 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 9.098e-06 2024-09-18T08:20:33.149 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:20:33.149 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:20:33.149 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-09-18T08:20:33.149 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:14:59.368675+0000", 2024-09-18T08:20:33.150 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.000134738 2024-09-18T08:20:33.150 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:20:33.150 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:20:33.150 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "waiting for peered", 2024-09-18T08:20:33.150 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:14:59.368681+0000", 2024-09-18T08:20:33.150 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 5.8590000000000001e-06 2024-09-18T08:20:33.151 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:20:33.151 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:20:33.151 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-09-18T08:20:33.151 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:15:00.360513+0000", 2024-09-18T08:20:33.151 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.99183212899999995 2024-09-18T08:20:33.151 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:20:33.151 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:20:33.151 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "started", 2024-09-18T08:20:33.151 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:15:00.360688+0000", 2024-09-18T08:20:33.151 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.00017474600000000001 2024-09-18T08:20:33.151 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:20:33.152 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:20:33.152 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "waiting for subops from 3,4", 2024-09-18T08:20:33.152 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:15:00.360836+0000", 2024-09-18T08:20:33.152 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.000148064 2024-09-18T08:20:33.152 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:20:33.152 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:20:33.152 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "op_commit", 2024-09-18T08:20:33.152 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:15:00.363513+0000", 2024-09-18T08:20:33.152 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.0026769110000000001 2024-09-18T08:20:33.152 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:20:33.152 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:20:33.152 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:20:33.153 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:15:00.363640+0000", 2024-09-18T08:20:33.153 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.000126857 2024-09-18T08:20:33.153 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:20:33.153 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:20:33.153 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:20:33.156 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:15:00.370161+0000", 2024-09-18T08:20:33.156 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.0065207370000000004 2024-09-18T08:20:33.156 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:20:33.157 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:20:33.157 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "commit_sent", 2024-09-18T08:20:33.157 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:15:00.370216+0000", 2024-09-18T08:20:33.157 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 5.5447e-05 2024-09-18T08:20:33.157 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:20:33.157 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:20:33.157 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "done", 2024-09-18T08:20:33.157 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:15:00.370262+0000", 2024-09-18T08:20:33.157 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 4.5698000000000002e-05 2024-09-18T08:20:33.157 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-09-18T08:20:33.157 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-09-18T08:20:33.157 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-09-18T08:20:33.157 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:20:33.158 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:20:33.158 INFO:teuthology.orchestra.run.smithi007.stdout: "description": "osd_op(client.15966.0:16693 10.3 10:c1956f28:::benchmark_data_smithi007_148834_object16692:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e323)", 2024-09-18T08:20:33.158 INFO:teuthology.orchestra.run.smithi007.stdout: "initiated_at": "2024-09-18T08:14:59.380410+0000", 2024-09-18T08:20:33.158 INFO:teuthology.orchestra.run.smithi007.stdout: "age": 333.675809233, 2024-09-18T08:20:33.158 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.99034093599999995, 2024-09-18T08:20:33.158 INFO:teuthology.orchestra.run.smithi007.stdout: "continuous": false, 2024-09-18T08:20:33.158 INFO:teuthology.orchestra.run.smithi007.stdout: "type_data": { 2024-09-18T08:20:33.158 INFO:teuthology.orchestra.run.smithi007.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T08:20:33.158 INFO:teuthology.orchestra.run.smithi007.stdout: "client_info": { 2024-09-18T08:20:33.158 INFO:teuthology.orchestra.run.smithi007.stdout: "client": "client.15966", 2024-09-18T08:20:33.158 INFO:teuthology.orchestra.run.smithi007.stdout: "client_addr": "172.21.15.7:0/529784850", 2024-09-18T08:20:33.158 INFO:teuthology.orchestra.run.smithi007.stdout: "tid": 16693 2024-09-18T08:20:33.159 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:20:33.159 INFO:teuthology.orchestra.run.smithi007.stdout: "events": [ 2024-09-18T08:20:33.159 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:20:33.159 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "initiated", 2024-09-18T08:20:33.159 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:14:59.380410+0000", 2024-09-18T08:20:33.159 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-09-18T08:20:33.159 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:20:33.159 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:20:33.159 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "header_read", 2024-09-18T08:20:33.159 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:14:59.380410+0000", 2024-09-18T08:20:33.159 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-09-18T08:20:33.159 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:20:33.160 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:20:33.160 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "throttled", 2024-09-18T08:20:33.160 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:14:59.380411+0000", 2024-09-18T08:20:33.160 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 5.1600000000000001e-07 2024-09-18T08:20:33.160 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:20:33.160 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:20:33.160 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "all_read", 2024-09-18T08:20:33.160 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:14:59.380460+0000", 2024-09-18T08:20:33.160 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 4.9724e-05 2024-09-18T08:20:33.160 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:20:33.160 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:20:33.161 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "dispatched", 2024-09-18T08:20:33.161 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:14:59.380468+0000", 2024-09-18T08:20:33.161 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 7.1779999999999997e-06 2024-09-18T08:20:33.161 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:20:33.161 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:20:33.161 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "queued_for_pg", 2024-09-18T08:20:33.161 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:14:59.380473+0000", 2024-09-18T08:20:33.161 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 5.4399999999999996e-06 2024-09-18T08:20:33.161 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:20:33.161 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:20:33.161 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-09-18T08:20:33.161 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:14:59.380568+0000", 2024-09-18T08:20:33.162 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 9.4908999999999995e-05 2024-09-18T08:20:33.162 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:20:33.162 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:20:33.162 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "waiting for peered", 2024-09-18T08:20:33.162 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:14:59.380577+0000", 2024-09-18T08:20:33.162 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 8.8079999999999994e-06 2024-09-18T08:20:33.162 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:20:33.162 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:20:33.162 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-09-18T08:20:33.162 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:15:00.362125+0000", 2024-09-18T08:20:33.162 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.98154793200000001 2024-09-18T08:20:33.162 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:20:33.162 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:20:33.163 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "started", 2024-09-18T08:20:33.163 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:15:00.362248+0000", 2024-09-18T08:20:33.163 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.00012318400000000001 2024-09-18T08:20:33.163 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:20:33.163 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:20:33.163 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "waiting for subops from 3,4", 2024-09-18T08:20:33.163 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:15:00.362390+0000", 2024-09-18T08:20:33.163 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.000142366 2024-09-18T08:20:33.163 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:20:33.163 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:20:33.163 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "op_commit", 2024-09-18T08:20:33.164 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:15:00.363599+0000", 2024-09-18T08:20:33.164 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.001208836 2024-09-18T08:20:33.164 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:20:33.164 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:20:33.164 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:20:33.164 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:15:00.364371+0000", 2024-09-18T08:20:33.164 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.00077206300000000002 2024-09-18T08:20:33.164 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:20:33.164 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:20:33.164 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:20:33.164 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:15:00.370667+0000", 2024-09-18T08:20:33.165 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.0062957500000000001 2024-09-18T08:20:33.165 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:20:33.165 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:20:33.165 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "commit_sent", 2024-09-18T08:20:33.165 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:15:00.370716+0000", 2024-09-18T08:20:33.165 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 4.9055999999999999e-05 2024-09-18T08:20:33.165 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:20:33.165 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:20:33.165 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "done", 2024-09-18T08:20:33.165 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:15:00.370751+0000", 2024-09-18T08:20:33.165 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 3.5173999999999999e-05 2024-09-18T08:20:33.166 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-09-18T08:20:33.166 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-09-18T08:20:33.166 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-09-18T08:20:33.166 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:20:33.166 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:20:33.166 INFO:teuthology.orchestra.run.smithi007.stdout: "description": "osd_op(client.15966.0:16740 10.9 10:9658d6e1:::benchmark_data_smithi007_148834_object16739:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e324)", 2024-09-18T08:20:33.166 INFO:teuthology.orchestra.run.smithi007.stdout: "initiated_at": "2024-09-18T08:15:00.363218+0000", 2024-09-18T08:20:33.166 INFO:teuthology.orchestra.run.smithi007.stdout: "age": 332.693001165, 2024-09-18T08:20:33.166 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.013596299, 2024-09-18T08:20:33.166 INFO:teuthology.orchestra.run.smithi007.stdout: "continuous": false, 2024-09-18T08:20:33.166 INFO:teuthology.orchestra.run.smithi007.stdout: "type_data": { 2024-09-18T08:20:33.166 INFO:teuthology.orchestra.run.smithi007.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T08:20:33.167 INFO:teuthology.orchestra.run.smithi007.stdout: "client_info": { 2024-09-18T08:20:33.167 INFO:teuthology.orchestra.run.smithi007.stdout: "client": "client.15966", 2024-09-18T08:20:33.167 INFO:teuthology.orchestra.run.smithi007.stdout: "client_addr": "172.21.15.7:0/529784850", 2024-09-18T08:20:33.167 INFO:teuthology.orchestra.run.smithi007.stdout: "tid": 16740 2024-09-18T08:20:33.167 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:20:33.167 INFO:teuthology.orchestra.run.smithi007.stdout: "events": [ 2024-09-18T08:20:33.167 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:20:33.167 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "initiated", 2024-09-18T08:20:33.167 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:15:00.363218+0000", 2024-09-18T08:20:33.167 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-09-18T08:20:33.167 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:20:33.167 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:20:33.168 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "header_read", 2024-09-18T08:20:33.168 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:15:00.363218+0000", 2024-09-18T08:20:33.168 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-09-18T08:20:33.168 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:20:33.168 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:20:33.168 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "throttled", 2024-09-18T08:20:33.168 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:15:00.363220+0000", 2024-09-18T08:20:33.168 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.268e-06 2024-09-18T08:20:33.168 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:20:33.168 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:20:33.168 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "all_read", 2024-09-18T08:20:33.168 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:15:00.363241+0000", 2024-09-18T08:20:33.169 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 2.1866e-05 2024-09-18T08:20:33.169 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:20:33.169 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:20:33.169 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "dispatched", 2024-09-18T08:20:33.169 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:15:00.363253+0000", 2024-09-18T08:20:33.169 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.1394e-05 2024-09-18T08:20:33.169 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:20:33.169 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:20:33.169 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "queued_for_pg", 2024-09-18T08:20:33.169 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:15:00.363260+0000", 2024-09-18T08:20:33.169 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 7.6620000000000005e-06 2024-09-18T08:20:33.169 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:20:33.170 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:20:33.170 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-09-18T08:20:33.170 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:15:00.363329+0000", 2024-09-18T08:20:33.170 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 6.8176999999999995e-05 2024-09-18T08:20:33.170 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:20:33.170 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:20:33.170 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "waiting for peered", 2024-09-18T08:20:33.170 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:15:00.363369+0000", 2024-09-18T08:20:33.170 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 3.9879999999999998e-05 2024-09-18T08:20:33.170 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:20:33.171 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:20:33.171 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-09-18T08:20:33.171 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:15:01.374467+0000", 2024-09-18T08:20:33.171 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.0110987739999999 2024-09-18T08:20:33.171 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:20:33.171 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:20:33.171 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "started", 2024-09-18T08:20:33.171 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:15:01.374631+0000", 2024-09-18T08:20:33.171 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.00016364799999999999 2024-09-18T08:20:33.172 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:20:33.172 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:20:33.172 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "waiting for subops from 3,4", 2024-09-18T08:20:33.172 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:15:01.374806+0000", 2024-09-18T08:20:33.172 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.00017538800000000001 2024-09-18T08:20:33.172 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:20:33.172 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:20:33.172 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:20:33.172 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:15:01.376700+0000", 2024-09-18T08:20:33.172 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.001893884 2024-09-18T08:20:33.172 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:20:33.172 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:20:33.173 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:20:33.173 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:15:01.376734+0000", 2024-09-18T08:20:33.173 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 3.4124000000000001e-05 2024-09-18T08:20:33.173 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:20:33.173 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:20:33.173 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "op_commit", 2024-09-18T08:20:33.173 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:15:01.376744+0000", 2024-09-18T08:20:33.173 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 9.893e-06 2024-09-18T08:20:33.173 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:20:33.173 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:20:33.173 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "commit_sent", 2024-09-18T08:20:33.173 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:15:01.376772+0000", 2024-09-18T08:20:33.173 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 2.8170999999999998e-05 2024-09-18T08:20:33.174 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:20:33.174 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:20:33.174 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "done", 2024-09-18T08:20:33.174 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:15:01.376815+0000", 2024-09-18T08:20:33.174 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 4.2169999999999998e-05 2024-09-18T08:20:33.174 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-09-18T08:20:33.174 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-09-18T08:20:33.174 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-09-18T08:20:33.174 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:20:33.174 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:20:33.174 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5 2024-09-18T08:20:33.176 INFO:teuthology.orchestra.run.smithi007.stdout: "description": "osd_op(client.15966.0:16746 10.9 10:988f7ffd:::benchmark_data_smithi007_148834_object16745:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e324)", 2024-09-18T08:20:33.176 INFO:teuthology.orchestra.run.smithi007.stdout: "initiated_at": "2024-09-18T08:15:00.367585+0000", 2024-09-18T08:20:33.176 INFO:teuthology.orchestra.run.smithi007.stdout: "age": 332.68863465700002, 2024-09-18T08:20:33.176 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.011225958, 2024-09-18T08:20:33.176 INFO:teuthology.orchestra.run.smithi007.stdout: "continuous": false, 2024-09-18T08:20:33.176 INFO:teuthology.orchestra.run.smithi007.stdout: "type_data": { 2024-09-18T08:20:33.176 INFO:teuthology.orchestra.run.smithi007.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T08:20:33.177 INFO:teuthology.orchestra.run.smithi007.stdout: "client_info": { 2024-09-18T08:20:33.177 INFO:teuthology.orchestra.run.smithi007.stdout: "client": "client.15966", 2024-09-18T08:20:33.177 INFO:teuthology.orchestra.run.smithi007.stdout: "client_addr": "172.21.15.7:0/529784850", 2024-09-18T08:20:33.177 INFO:teuthology.orchestra.run.smithi007.stdout: "tid": 16746 2024-09-18T08:20:33.177 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:20:33.177 INFO:teuthology.orchestra.run.smithi007.stdout: "events": [ 2024-09-18T08:20:33.177 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:20:33.177 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "initiated", 2024-09-18T08:20:33.177 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:15:00.367585+0000", 2024-09-18T08:20:33.177 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-09-18T08:20:33.177 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:20:33.177 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:20:33.177 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "header_read", 2024-09-18T08:20:33.178 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:15:00.367585+0000", 2024-09-18T08:20:33.178 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-09-18T08:20:33.178 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:20:33.178 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:20:33.178 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "throttled", 2024-09-18T08:20:33.178 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:15:00.367586+0000", 2024-09-18T08:20:33.178 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.359e-06 2024-09-18T08:20:33.178 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:20:33.178 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:20:33.178 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "all_read", 2024-09-18T08:20:33.178 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:15:00.367607+0000", 2024-09-18T08:20:33.178 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 2.1055000000000001e-05 2024-09-18T08:20:33.178 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:20:33.179 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:20:33.179 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "dispatched", 2024-09-18T08:20:33.179 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:15:00.367618+0000", 2024-09-18T08:20:33.179 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.0336999999999999e-05 2024-09-18T08:20:33.179 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:20:33.179 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:20:33.179 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "queued_for_pg", 2024-09-18T08:20:33.179 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:15:00.367625+0000", 2024-09-18T08:20:33.179 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 7.3930000000000002e-06 2024-09-18T08:20:33.179 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:20:33.179 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:20:33.179 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-09-18T08:20:33.179 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:15:00.368768+0000", 2024-09-18T08:20:33.180 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.001143055 2024-09-18T08:20:33.180 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:20:33.180 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:20:33.180 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "waiting for peered", 2024-09-18T08:20:33.180 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:15:00.368779+0000", 2024-09-18T08:20:33.180 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.0553999999999999e-05 2024-09-18T08:20:33.180 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:20:33.180 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:20:33.180 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-09-18T08:20:33.180 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:15:01.376069+0000", 2024-09-18T08:20:33.180 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.007290206 2024-09-18T08:20:33.180 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:20:33.180 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:20:33.181 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "started", 2024-09-18T08:20:33.181 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:15:01.376154+0000", 2024-09-18T08:20:33.181 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 8.4840999999999996e-05 2024-09-18T08:20:33.181 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:20:33.181 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:20:33.181 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "waiting for subops from 3,4", 2024-09-18T08:20:33.181 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:15:01.376239+0000", 2024-09-18T08:20:33.181 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 8.5001999999999997e-05 2024-09-18T08:20:33.181 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:20:33.181 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:20:33.181 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "op_commit", 2024-09-18T08:20:33.181 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:15:01.378633+0000", 2024-09-18T08:20:33.181 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.002394471 2024-09-18T08:20:33.182 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:20:33.182 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:20:33.182 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:20:33.182 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:15:01.378671+0000", 2024-09-18T08:20:33.182 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 3.7676000000000003e-05 2024-09-18T08:20:33.182 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:20:33.182 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:20:33.182 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:20:33.182 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:15:01.378716+0000", 2024-09-18T08:20:33.182 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 4.5269999999999999e-05 2024-09-18T08:20:33.182 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:20:33.182 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:20:33.182 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "commit_sent", 2024-09-18T08:20:33.183 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:15:01.378771+0000", 2024-09-18T08:20:33.183 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 5.5050000000000003e-05 2024-09-18T08:20:33.183 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:20:33.183 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:20:33.183 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "done", 2024-09-18T08:20:33.183 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:15:01.378811+0000", 2024-09-18T08:20:33.183 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 3.9688999999999999e-05 2024-09-18T08:20:33.183 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-09-18T08:20:33.183 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-09-18T08:20:33.183 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-09-18T08:20:33.183 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:20:33.183 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:20:33.183 INFO:teuthology.orchestra.run.smithi007.stdout: "description": "osd_op(client.15966.0:16793 10.9 10:94d3fd58:::benchmark_data_smithi007_148834_object16792:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e324)", 2024-09-18T08:20:33.183 INFO:teuthology.orchestra.run.smithi007.stdout: "initiated_at": "2024-09-18T08:15:00.387215+0000", 2024-09-18T08:20:33.184 INFO:teuthology.orchestra.run.smithi007.stdout: "age": 332.669004567, 2024-09-18T08:20:33.184 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.99187469100000003, 2024-09-18T08:20:33.184 INFO:teuthology.orchestra.run.smithi007.stdout: "continuous": false, 2024-09-18T08:20:33.184 INFO:teuthology.orchestra.run.smithi007.stdout: "type_data": { 2024-09-18T08:20:33.184 INFO:teuthology.orchestra.run.smithi007.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T08:20:33.184 INFO:teuthology.orchestra.run.smithi007.stdout: "client_info": { 2024-09-18T08:20:33.184 INFO:teuthology.orchestra.run.smithi007.stdout: "client": "client.15966", 2024-09-18T08:20:33.184 INFO:teuthology.orchestra.run.smithi007.stdout: "client_addr": "172.21.15.7:0/529784850", 2024-09-18T08:20:33.184 INFO:teuthology.orchestra.run.smithi007.stdout: "tid": 16793 2024-09-18T08:20:33.184 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:20:33.184 INFO:teuthology.orchestra.run.smithi007.stdout: "events": [ 2024-09-18T08:20:33.184 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:20:33.184 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "initiated", 2024-09-18T08:20:33.185 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:15:00.387215+0000", 2024-09-18T08:20:33.185 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-09-18T08:20:33.185 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:20:33.185 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:20:33.185 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "header_read", 2024-09-18T08:20:33.185 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:15:00.387215+0000", 2024-09-18T08:20:33.185 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-09-18T08:20:33.185 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:20:33.185 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:20:33.185 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "throttled", 2024-09-18T08:20:33.185 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:15:00.387216+0000", 2024-09-18T08:20:33.185 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.5960000000000001e-06 2024-09-18T08:20:33.185 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:20:33.186 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:20:33.186 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "all_read", 2024-09-18T08:20:33.186 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:15:00.387243+0000", 2024-09-18T08:20:33.186 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 2.635e-05 2024-09-18T08:20:33.186 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:20:33.186 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:20:33.186 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "dispatched", 2024-09-18T08:20:33.186 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:15:00.387257+0000", 2024-09-18T08:20:33.186 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.4081e-05 2024-09-18T08:20:33.186 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:20:33.186 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:20:33.186 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "queued_for_pg", 2024-09-18T08:20:33.186 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:15:00.387264+0000", 2024-09-18T08:20:33.187 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 7.5630000000000003e-06 2024-09-18T08:20:33.187 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:20:33.187 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:20:33.187 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-09-18T08:20:33.187 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:15:00.387346+0000", 2024-09-18T08:20:33.187 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 8.1583999999999997e-05 2024-09-18T08:20:33.187 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:20:33.187 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:20:33.187 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "waiting for peered", 2024-09-18T08:20:33.187 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:15:00.387357+0000", 2024-09-18T08:20:33.187 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.0572e-05 2024-09-18T08:20:33.187 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:20:33.187 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:20:33.188 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-09-18T08:20:33.188 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:15:01.376952+0000", 2024-09-18T08:20:33.188 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.98959566499999996 2024-09-18T08:20:33.188 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:20:33.188 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:20:33.188 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "started", 2024-09-18T08:20:33.188 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:15:01.377094+0000", 2024-09-18T08:20:33.188 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.00014199500000000001 2024-09-18T08:20:33.188 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:20:33.188 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:20:33.188 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "waiting for subops from 3,4", 2024-09-18T08:20:33.188 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:15:01.377242+0000", 2024-09-18T08:20:33.188 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.00014731799999999999 2024-09-18T08:20:33.188 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:20:33.189 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:20:33.189 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "op_commit", 2024-09-18T08:20:33.189 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:15:01.378682+0000", 2024-09-18T08:20:33.189 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.0014402919999999999 2024-09-18T08:20:33.189 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:20:33.189 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:20:33.189 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:20:33.189 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:15:01.378856+0000", 2024-09-18T08:20:33.189 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.00017438800000000001 2024-09-18T08:20:33.189 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:20:33.189 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:20:33.189 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:20:33.189 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:15:01.378943+0000", 2024-09-18T08:20:33.190 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 8.7101000000000005e-05 2024-09-18T08:20:33.190 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:20:33.190 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:20:33.190 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "commit_sent", 2024-09-18T08:20:33.190 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:15:01.379052+0000", 2024-09-18T08:20:33.190 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.00010877799999999999 2024-09-18T08:20:33.190 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:20:33.190 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:20:33.190 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "done", 2024-09-18T08:20:33.190 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:15:01.379090+0000", 2024-09-18T08:20:33.190 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 3.7407999999999998e-05 2024-09-18T08:20:33.190 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-09-18T08:20:33.190 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-09-18T08:20:33.190 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-09-18T08:20:33.191 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:20:33.191 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:20:33.191 INFO:teuthology.orchestra.run.smithi007.stdout: "description": "osd_op(client.15966.0:16818 10.9 10:9b4b2220:::benchmark_data_smithi007_148834_object16817:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e324)", 2024-09-18T08:20:33.191 INFO:teuthology.orchestra.run.smithi007.stdout: "initiated_at": "2024-09-18T08:15:00.398499+0000", 2024-09-18T08:20:33.191 INFO:teuthology.orchestra.run.smithi007.stdout: "age": 332.657719955, 2024-09-18T08:20:33.191 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.98326930499999998, 2024-09-18T08:20:33.191 INFO:teuthology.orchestra.run.smithi007.stdout: "continuous": false, 2024-09-18T08:20:33.191 INFO:teuthology.orchestra.run.smithi007.stdout: "type_data": { 2024-09-18T08:20:33.191 INFO:teuthology.orchestra.run.smithi007.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T08:20:33.191 INFO:teuthology.orchestra.run.smithi007.stdout: "client_info": { 2024-09-18T08:20:33.191 INFO:teuthology.orchestra.run.smithi007.stdout: "client": "client.15966", 2024-09-18T08:20:33.191 INFO:teuthology.orchestra.run.smithi007.stdout: "client_addr": "172.21.15.7:0/529784850", 2024-09-18T08:20:33.191 INFO:teuthology.orchestra.run.smithi007.stdout: "tid": 16818 2024-09-18T08:20:33.192 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:20:33.192 INFO:teuthology.orchestra.run.smithi007.stdout: "events": [ 2024-09-18T08:20:33.192 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:20:33.192 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "initiated", 2024-09-18T08:20:33.192 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:15:00.398499+0000", 2024-09-18T08:20:33.192 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-09-18T08:20:33.192 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:20:33.192 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:20:33.192 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "header_read", 2024-09-18T08:20:33.192 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:15:00.398499+0000", 2024-09-18T08:20:33.192 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-09-18T08:20:33.192 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:20:33.192 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:20:33.192 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "throttled", 2024-09-18T08:20:33.193 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:15:00.398501+0000", 2024-09-18T08:20:33.193 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.246e-06 2024-09-18T08:20:33.193 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:20:33.193 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:20:33.193 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "all_read", 2024-09-18T08:20:33.193 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:15:00.398529+0000", 2024-09-18T08:20:33.193 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 2.7789000000000001e-05 2024-09-18T08:20:33.193 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:20:33.193 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:20:33.193 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "dispatched", 2024-09-18T08:20:33.193 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:15:00.398539+0000", 2024-09-18T08:20:33.193 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.0176e-05 2024-09-18T08:20:33.193 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:20:33.194 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:20:33.194 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "queued_for_pg", 2024-09-18T08:20:33.194 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:15:00.398553+0000", 2024-09-18T08:20:33.194 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.4038e-05 2024-09-18T08:20:33.194 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:20:33.194 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:20:33.194 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-09-18T08:20:33.194 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:15:00.398785+0000", 2024-09-18T08:20:33.194 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.000232582 2024-09-18T08:20:33.194 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:20:33.194 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:20:33.194 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "waiting for peered", 2024-09-18T08:20:33.194 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:15:00.398795+0000", 2024-09-18T08:20:33.194 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 9.8449999999999998e-06 2024-09-18T08:20:33.195 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:20:33.195 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:20:33.195 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-09-18T08:20:33.195 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:15:01.377825+0000", 2024-09-18T08:20:33.195 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.97902953400000003 2024-09-18T08:20:33.195 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:20:33.195 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:20:33.195 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "started", 2024-09-18T08:20:33.195 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:15:01.377954+0000", 2024-09-18T08:20:33.195 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.00012964300000000001 2024-09-18T08:20:33.195 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:20:33.195 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:20:33.195 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "waiting for subops from 3,4", 2024-09-18T08:20:33.195 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:15:01.378088+0000", 2024-09-18T08:20:33.196 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.00013414300000000001 2024-09-18T08:20:33.196 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:20:33.196 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:20:33.196 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:20:33.196 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:15:01.380051+0000", 2024-09-18T08:20:33.196 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.0019621199999999999 2024-09-18T08:20:33.196 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:20:33.196 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:20:33.196 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:20:33.196 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:15:01.380096+0000", 2024-09-18T08:20:33.196 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 4.5011e-05 2024-09-18T08:20:33.196 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:20:33.197 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:20:33.197 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "op_commit", 2024-09-18T08:20:33.197 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:15:01.381685+0000", 2024-09-18T08:20:33.197 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.0015894189999999999 2024-09-18T08:20:33.197 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:20:33.197 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:20:33.197 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "commit_sent", 2024-09-18T08:20:33.197 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:15:01.381728+0000", 2024-09-18T08:20:33.197 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 4.3421999999999999e-05 2024-09-18T08:20:33.197 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:20:33.197 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:20:33.197 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "done", 2024-09-18T08:20:33.197 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:15:01.381769+0000", 2024-09-18T08:20:33.197 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 4.0336999999999997e-05 2024-09-18T08:20:33.198 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-09-18T08:20:33.198 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-09-18T08:20:33.198 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-09-18T08:20:33.198 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:20:33.198 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:20:33.198 INFO:teuthology.orchestra.run.smithi007.stdout: "description": "osd_op(client.15966.0:16840 10.9 10:9b2ea096:::benchmark_data_smithi007_148834_object16839:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e324)", 2024-09-18T08:20:33.198 INFO:teuthology.orchestra.run.smithi007.stdout: "initiated_at": "2024-09-18T08:15:00.407638+0000", 2024-09-18T08:20:33.198 INFO:teuthology.orchestra.run.smithi007.stdout: "age": 332.64858182, 2024-09-18T08:20:33.198 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.97589707299999995, 2024-09-18T08:20:33.198 INFO:teuthology.orchestra.run.smithi007.stdout: "continuous": false, 2024-09-18T08:20:33.198 INFO:teuthology.orchestra.run.smithi007.stdout: "type_data": { 2024-09-18T08:20:33.198 INFO:teuthology.orchestra.run.smithi007.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T08:20:33.198 INFO:teuthology.orchestra.run.smithi007.stdout: "client_info": { 2024-09-18T08:20:33.199 INFO:teuthology.orchestra.run.smithi007.stdout: "client": "client.15966", 2024-09-18T08:20:33.199 INFO:teuthology.orchestra.run.smithi007.stdout: "client_addr": "172.21.15.7:0/529784850", 2024-09-18T08:20:33.199 INFO:teuthology.orchestra.run.smithi007.stdout: "tid": 16840 2024-09-18T08:20:33.199 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:20:33.199 INFO:teuthology.orchestra.run.smithi007.stdout: "events": [ 2024-09-18T08:20:33.199 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:20:33.199 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "initiated", 2024-09-18T08:20:33.199 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:15:00.407638+0000", 2024-09-18T08:20:33.199 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-09-18T08:20:33.199 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:20:33.199 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:20:33.199 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "header_read", 2024-09-18T08:20:33.199 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:15:00.407638+0000", 2024-09-18T08:20:33.200 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-09-18T08:20:33.200 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:20:33.200 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:20:33.200 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "throttled", 2024-09-18T08:20:33.200 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:15:00.407639+0000", 2024-09-18T08:20:33.200 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.8640000000000001e-06 2024-09-18T08:20:33.200 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:20:33.200 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:20:33.200 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "all_read", 2024-09-18T08:20:33.200 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:15:00.407679+0000", 2024-09-18T08:20:33.200 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 3.9249999999999999e-05 2024-09-18T08:20:33.200 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:20:33.200 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:20:33.201 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "dispatched", 2024-09-18T08:20:33.201 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:15:00.407701+0000", 2024-09-18T08:20:33.201 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 2.2648999999999999e-05 2024-09-18T08:20:33.201 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:20:33.201 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:20:33.201 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "queued_for_pg", 2024-09-18T08:20:33.201 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:15:00.407710+0000", 2024-09-18T08:20:33.201 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 8.4970000000000001e-06 2024-09-18T08:20:33.201 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:20:33.201 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:20:33.201 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-09-18T08:20:33.201 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:15:00.407825+0000", 2024-09-18T08:20:33.201 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.000115239 2024-09-18T08:20:33.201 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:20:33.202 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:20:33.202 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "waiting for peered", 2024-09-18T08:20:33.202 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:15:00.407836+0000", 2024-09-18T08:20:33.202 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.0549e-05 2024-09-18T08:20:33.202 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:20:33.202 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:20:33.202 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-09-18T08:20:33.202 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:15:01.379173+0000", 2024-09-18T08:20:33.202 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.97133749700000005 2024-09-18T08:20:33.202 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:20:33.202 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:20:33.202 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "started", 2024-09-18T08:20:33.202 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:15:01.379323+0000", 2024-09-18T08:20:33.202 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.00014971900000000001 2024-09-18T08:20:33.203 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:20:33.203 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:20:33.203 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "waiting for subops from 3,4", 2024-09-18T08:20:33.203 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:15:01.379452+0000", 2024-09-18T08:20:33.203 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.00012921499999999999 2024-09-18T08:20:33.203 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:20:33.203 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:20:33.203 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "op_commit", 2024-09-18T08:20:33.203 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:15:01.383346+0000", 2024-09-18T08:20:33.203 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.0038943049999999998 2024-09-18T08:20:33.203 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:20:33.203 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:20:33.203 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:20:33.203 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:15:01.383394+0000", 2024-09-18T08:20:33.204 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 4.7095000000000002e-05 2024-09-18T08:20:33.204 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:20:33.204 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:20:33.204 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:20:33.204 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:15:01.383437+0000", 2024-09-18T08:20:33.204 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 4.3872000000000003e-05 2024-09-18T08:20:33.204 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:20:33.204 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:20:33.204 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "commit_sent", 2024-09-18T08:20:33.204 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:15:01.383492+0000", 2024-09-18T08:20:33.204 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 5.4193000000000001e-05 2024-09-18T08:20:33.204 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:20:33.205 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:20:33.205 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "done", 2024-09-18T08:20:33.205 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:15:01.383535+0000", 2024-09-18T08:20:33.205 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 4.3129e-05 2024-09-18T08:20:33.205 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-09-18T08:20:33.205 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-09-18T08:20:33.205 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-09-18T08:20:33.205 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:20:33.205 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:20:33.205 INFO:teuthology.orchestra.run.smithi007.stdout: "description": "osd_op(client.15966.0:16843 10.9 10:9f4a9fc4:::benchmark_data_smithi007_148834_object16842:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e324)", 2024-09-18T08:20:33.205 INFO:teuthology.orchestra.run.smithi007.stdout: "initiated_at": "2024-09-18T08:15:00.408739+0000", 2024-09-18T08:20:33.205 INFO:teuthology.orchestra.run.smithi007.stdout: "age": 332.647480223, 2024-09-18T08:20:33.205 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.97496165499999998, 2024-09-18T08:20:33.206 INFO:teuthology.orchestra.run.smithi007.stdout: "continuous": false, 2024-09-18T08:20:33.206 INFO:teuthology.orchestra.run.smithi007.stdout: "type_data": { 2024-09-18T08:20:33.206 INFO:teuthology.orchestra.run.smithi007.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T08:20:33.206 INFO:teuthology.orchestra.run.smithi007.stdout: "client_info": { 2024-09-18T08:20:33.206 INFO:teuthology.orchestra.run.smithi007.stdout: "client": "client.15966", 2024-09-18T08:20:33.206 INFO:teuthology.orchestra.run.smithi007.stdout: "client_addr": "172.21.15.7:0/529784850", 2024-09-18T08:20:33.206 INFO:teuthology.orchestra.run.smithi007.stdout: "tid": 16843 2024-09-18T08:20:33.206 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:20:33.206 INFO:teuthology.orchestra.run.smithi007.stdout: "events": [ 2024-09-18T08:20:33.206 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:20:33.206 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "initiated", 2024-09-18T08:20:33.206 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:15:00.408739+0000", 2024-09-18T08:20:33.206 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-09-18T08:20:33.206 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:20:33.207 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:20:33.207 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "header_read", 2024-09-18T08:20:33.207 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:15:00.408739+0000", 2024-09-18T08:20:33.207 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-09-18T08:20:33.207 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:20:33.207 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:20:33.207 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "throttled", 2024-09-18T08:20:33.207 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:15:00.408740+0000", 2024-09-18T08:20:33.207 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.145e-06 2024-09-18T08:20:33.207 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:20:33.207 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:20:33.207 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "all_read", 2024-09-18T08:20:33.207 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:15:00.408803+0000", 2024-09-18T08:20:33.208 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 6.2834000000000002e-05 2024-09-18T08:20:33.208 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:20:33.208 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:20:33.208 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "dispatched", 2024-09-18T08:20:33.208 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:15:00.408817+0000", 2024-09-18T08:20:33.208 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.3434999999999999e-05 2024-09-18T08:20:33.208 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:20:33.208 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:20:33.208 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "queued_for_pg", 2024-09-18T08:20:33.208 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:15:00.408825+0000", 2024-09-18T08:20:33.208 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 8.2789999999999998e-06 2024-09-18T08:20:33.208 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:20:33.208 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:20:33.208 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-09-18T08:20:33.209 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:15:00.408900+0000", 2024-09-18T08:20:33.209 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 7.5226000000000001e-05 2024-09-18T08:20:33.209 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:20:33.209 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:20:33.209 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "waiting for peered", 2024-09-18T08:20:33.209 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:15:00.408909+0000", 2024-09-18T08:20:33.209 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 9.2730000000000008e-06 2024-09-18T08:20:33.209 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:20:33.209 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:20:33.209 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-09-18T08:20:33.209 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:15:01.380129+0000", 2024-09-18T08:20:33.209 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.97121982100000004 2024-09-18T08:20:33.209 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:20:33.210 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:20:33.210 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "started", 2024-09-18T08:20:33.210 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:15:01.380236+0000", 2024-09-18T08:20:33.210 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.000106405 2024-09-18T08:20:33.210 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:20:33.210 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:20:33.210 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "waiting for subops from 3,4", 2024-09-18T08:20:33.210 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:15:01.380356+0000", 2024-09-18T08:20:33.210 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.000120686 2024-09-18T08:20:33.210 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:20:33.210 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:20:33.210 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "op_commit", 2024-09-18T08:20:33.210 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:15:01.383352+0000", 2024-09-18T08:20:33.211 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.0029956919999999999 2024-09-18T08:20:33.211 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:20:33.211 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:20:33.211 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:20:33.211 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:15:01.383578+0000", 2024-09-18T08:20:33.211 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.00022547800000000001 2024-09-18T08:20:33.211 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:20:33.211 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:20:33.211 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:20:33.211 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:15:01.383622+0000", 2024-09-18T08:20:33.211 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 4.4311999999999999e-05 2024-09-18T08:20:33.211 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:20:33.211 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:20:33.211 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "commit_sent", 2024-09-18T08:20:33.212 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:15:01.383666+0000", 2024-09-18T08:20:33.212 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 4.4579000000000001e-05 2024-09-18T08:20:33.212 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:20:33.212 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:20:33.212 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "done", 2024-09-18T08:20:33.212 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:15:01.383701+0000", 2024-09-18T08:20:33.212 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 3.4489999999999997e-05 2024-09-18T08:20:33.212 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-09-18T08:20:33.212 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-09-18T08:20:33.212 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-09-18T08:20:33.212 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:20:33.212 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:20:33.212 INFO:teuthology.orchestra.run.smithi007.stdout: "description": "osd_op(client.15966.0:16844 10.9 10:9c7b6630:::benchmark_data_smithi007_148834_object16843:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e324)", 2024-09-18T08:20:33.213 INFO:teuthology.orchestra.run.smithi007.stdout: "initiated_at": "2024-09-18T08:15:00.408844+0000", 2024-09-18T08:20:33.213 INFO:teuthology.orchestra.run.smithi007.stdout: "age": 332.64737504599998, 2024-09-18T08:20:33.213 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.97895429899999997, 2024-09-18T08:20:33.213 INFO:teuthology.orchestra.run.smithi007.stdout: "continuous": false, 2024-09-18T08:20:33.213 INFO:teuthology.orchestra.run.smithi007.stdout: "type_data": { 2024-09-18T08:20:33.213 INFO:teuthology.orchestra.run.smithi007.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T08:20:33.213 INFO:teuthology.orchestra.run.smithi007.stdout: "client_info": { 2024-09-18T08:20:33.213 INFO:teuthology.orchestra.run.smithi007.stdout: "client": "client.15966", 2024-09-18T08:20:33.213 INFO:teuthology.orchestra.run.smithi007.stdout: "client_addr": "172.21.15.7:0/529784850", 2024-09-18T08:20:33.213 INFO:teuthology.orchestra.run.smithi007.stdout: "tid": 16844 2024-09-18T08:20:33.213 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:20:33.213 INFO:teuthology.orchestra.run.smithi007.stdout: "events": [ 2024-09-18T08:20:33.213 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:20:33.214 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "initiated", 2024-09-18T08:20:33.214 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:15:00.408844+0000", 2024-09-18T08:20:33.214 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-09-18T08:20:33.214 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:20:33.214 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:20:33.214 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "header_read", 2024-09-18T08:20:33.214 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:15:00.408844+0000", 2024-09-18T08:20:33.214 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-09-18T08:20:33.214 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:20:33.214 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:20:33.214 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "throttled", 2024-09-18T08:20:33.214 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:15:00.408845+0000", 2024-09-18T08:20:33.214 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 7.8400000000000003e-07 2024-09-18T08:20:33.214 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:20:33.215 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:20:33.215 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "all_read", 2024-09-18T08:20:33.215 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:15:00.408949+0000", 2024-09-18T08:20:33.215 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.000103719 2024-09-18T08:20:33.215 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:20:33.215 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:20:33.215 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "dispatched", 2024-09-18T08:20:33.215 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:15:00.408960+0000", 2024-09-18T08:20:33.215 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.1082e-05 2024-09-18T08:20:33.215 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:20:33.215 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:20:33.215 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "queued_for_pg", 2024-09-18T08:20:33.215 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:15:00.408967+0000", 2024-09-18T08:20:33.215 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 6.9589999999999998e-06 2024-09-18T08:20:33.216 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:20:33.216 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:20:33.216 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-09-18T08:20:33.216 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:15:00.409026+0000", 2024-09-18T08:20:33.216 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 5.8570000000000003e-05 2024-09-18T08:20:33.216 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:20:33.216 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:20:33.216 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "waiting for peered", 2024-09-18T08:20:33.216 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:15:00.409033+0000", 2024-09-18T08:20:33.216 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 7.4850000000000003e-06 2024-09-18T08:20:33.216 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:20:33.216 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:20:33.216 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-09-18T08:20:33.217 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:15:01.380947+0000", 2024-09-18T08:20:33.217 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.9719139 2024-09-18T08:20:33.217 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:20:33.217 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:20:33.217 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "started", 2024-09-18T08:20:33.217 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:15:01.381063+0000", 2024-09-18T08:20:33.217 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.000115841 2024-09-18T08:20:33.217 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:20:33.217 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:20:33.217 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "waiting for subops from 3,4", 2024-09-18T08:20:33.217 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:15:01.381182+0000", 2024-09-18T08:20:33.217 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.000119503 2024-09-18T08:20:33.217 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:20:33.217 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:20:33.218 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "op_commit", 2024-09-18T08:20:33.218 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:15:01.383357+0000", 2024-09-18T08:20:33.218 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.002174619 2024-09-18T08:20:33.218 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:20:33.218 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:20:33.218 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:20:33.218 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:15:01.386021+0000", 2024-09-18T08:20:33.218 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.002663941 2024-09-18T08:20:33.218 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:20:33.218 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:20:33.218 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:20:33.218 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:15:01.387705+0000", 2024-09-18T08:20:33.218 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.001684261 2024-09-18T08:20:33.219 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/779349ca-7593-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T08:20:33.220 INFO:teuthology.orchestra.run.smithi007.stdout:{"pg_ready":true,"pg_map":{"version":885,"stamp":"2024-09-18T08:20:29.589064+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":4871100979,"num_objects":74518,"num_object_clones":0,"num_object_copies":223554,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":74518,"num_whiteouts":0,"num_read":1638,"num_read_kb":1390,"num_write":123767,"num_write_kb":6315630,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19752,"num_bytes_recovered":1255296080,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":64045,"ondisk_log_size":64045,"up":474,"acting":474,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":474,"num_osds":7,"num_per_pool_osds":7,"num_per_pool_omap_osds":7,"kb":656216064,"kb_used":18831364,"kb_used_data":14119060,"kb_used_omap":17,"kb_used_meta":4711854,"kb_avail":637384700,"statfs":{"total":671965249536,"available":652681932800,"internally_reserved":0,"allocated":14457917440,"data_stored":14455019298,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17639,"internal_metadata":4824939289},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[9,2,6,5,7],"upper_bound":32},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-898695168,"num_objects":-13713,"num_object_clones":0,"num_object_copies":-41139,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-13713,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":13713,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.083709"},"pg_stats":[{"pgid":"3.1f","version":"0'0","reported_seq":846,"reported_epoch":438,"state":"active+clean","last_fresh":"2024-09-18T08:20:16.501503+0000","last_change":"2024-09-18T08:19:05.894872+0000","last_active":"2024-09-18T08:20:16.501503+0000","last_peered":"2024-09-18T08:20:16.501503+0000","last_clean":"2024-09-18T08:20:16.501503+0000","last_became_active":"2024-09-18T08:19:05.894522+0000","last_became_peered":"2024-09-18T08:19:05.894522+0000","last_unstale":"2024-09-18T08:20:16.501503+0000","last_undegraded":"2024-09-18T08:20:16.501503+0000","last_fullsized":"2024-09-18T08:20:16.501503+0000","mapping_epoch":413,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":414,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:18:22.822559+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:46.860092+0000","last_clean_scrub_stamp":"2024-09-18T08:18:22.822559+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,6],"acting":[0,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"271'15","reported_seq":572,"reported_epoch":438,"state":"active+clean","last_fresh":"2024-09-18T08:20:16.473493+0000","last_change":"2024-09-18T08:19:13.094994+0000","last_active":"2024-09-18T08:20:16.473493+0000","last_peered":"2024-09-18T08:20:16.473493+0000","last_clean":"2024-09-18T08:20:16.473493+0000","last_became_active":"2024-09-18T08:16:13.456332+0000","last_became_peered":"2024-09-18T08:16:13.456332+0000","last_unstale":"2024-09-18T08:20:16.473493+0000","last_undegraded":"2024-09-18T08:20:16.473493+0000","last_fullsized":"2024-09-18T08:20:16.473493+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"271'15","last_scrub_stamp":"2024-09-18T08:19:13.094901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_clean_scrub_stamp":"2024-09-18T08:19:13.094901+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:20:38.578688+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":25,"num_read_kb":19,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,3],"acting":[2,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":663,"reported_epoch":438,"state":"active+clean","last_fresh":"2024-09-18T08:20:16.477089+0000","last_change":"2024-09-18T08:18:27.641641+0000","last_active":"2024-09-18T08:20:16.477089+0000","last_peered":"2024-09-18T08:20:16.477089+0000","last_clean":"2024-09-18T08:20:16.477089+0000","last_became_active":"2024-09-18T08:18:27.641323+0000","last_became_peered":"2024-09-18T08:18:27.641323+0000","last_unstale":"2024-09-18T08:20:16.477089+0000","last_undegraded":"2024-09-18T08:20:16.477089+0000","last_fullsized":"2024-09-18T08:20:16.477089+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:18:21.610930+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:50.886830+0000","last_clean_scrub_stamp":"2024-09-18T08:18:21.610930+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":554,"reported_epoch":438,"state":"active+clean","last_fresh":"2024-09-18T08:20:16.492208+0000","last_change":"2024-09-18T08:18:27.639074+0000","last_active":"2024-09-18T08:20:16.492208+0000","last_peered":"2024-09-18T08:20:16.492208+0000","last_clean":"2024-09-18T08:20:16.492208+0000","last_became_active":"2024-09-18T08:18:27.638698+0000","last_became_peered":"2024-09-18T08:18:27.638698+0000","last_unstale":"2024-09-18T08:20:16.492208+0000","last_undegraded":"2024-09-18T08:20:16.492208+0000","last_fullsized":"2024-09-18T08:20:16.492208+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:17:40.565100+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_clean_scrub_stamp":"2024-09-18T08:17:40.565100+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":831,"reported_epoch":438,"state":"active+clean","last_fresh":"2024-09-18T08:20:16.473270+0000","last_change":"2024-09-18T08:18:27.662859+0000","last_active":"2024-09-18T08:20:16.473270+0000","last_peered":"2024-09-18T08:20:16.473270+0000","last_clean":"2024-09-18T08:20:16.473270+0000","last_became_active":"2024-09-18T08:18:27.662545+0000","last_became_peered":"2024-09-18T08:18:27.662545+0000","last_unstale":"2024-09-18T08:20:16.473270+0000","last_undegraded":"2024-09-18T08:20:16.473270+0000","last_fullsized":"2024-09-18T08:20:16.473270+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:15:51.826050+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_clean_scrub_stamp":"2024-09-18T08:15:51.826050+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":841,"reported_epoch":438,"state":"active+clean","last_fresh":"2024-09-18T08:20:16.474000+0000","last_change":"2024-09-18T08:19:05.895309+0000","last_active":"2024-09-18T08:20:16.474000+0000","last_peered":"2024-09-18T08:20:16.474000+0000","last_clean":"2024-09-18T08:20:16.474000+0000","last_became_active":"2024-09-18T08:19:05.894958+0000","last_became_peered":"2024-09-18T08:19:05.894958+0000","last_unstale":"2024-09-18T08:20:16.474000+0000","last_undegraded":"2024-09-18T08:20:16.474000+0000","last_fullsized":"2024-09-18T08:20:16.474000+0000","mapping_epoch":413,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":414,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:17:38.359366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:46.860092+0000","last_clean_scrub_stamp":"2024-09-18T08:17:38.359366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,1],"acting":[3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"271'27","reported_seq":857,"reported_epoch":438,"state":"active+clean","last_fresh":"2024-09-18T08:20:16.472415+0000","last_change":"2024-09-18T08:20:14.895711+0000","last_active":"2024-09-18T08:20:16.472415+0000","last_peered":"2024-09-18T08:20:16.472415+0000","last_clean":"2024-09-18T08:20:16.472415+0000","last_became_active":"2024-09-18T08:19:05.907041+0000","last_became_peered":"2024-09-18T08:19:05.907041+0000","last_unstale":"2024-09-18T08:20:16.472415+0000","last_undegraded":"2024-09-18T08:20:16.472415+0000","last_fullsized":"2024-09-18T08:20:16.472415+0000","mapping_epoch":413,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":414,"parent":"0.0","parent_split_bits":0,"last_scrub":"271'27","last_scrub_stamp":"2024-09-18T08:20:14.895622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_clean_scrub_stamp":"2024-09-18T08:20:14.895622+0000","objects_scrubbed":7,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:21:37.655368+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":44,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":466,"reported_epoch":438,"state":"active+clean","last_fresh":"2024-09-18T08:20:16.477055+0000","last_change":"2024-09-18T08:19:05.886692+0000","last_active":"2024-09-18T08:20:16.477055+0000","last_peered":"2024-09-18T08:20:16.477055+0000","last_clean":"2024-09-18T08:20:16.477055+0000","last_became_active":"2024-09-18T08:19:05.886289+0000","last_became_peered":"2024-09-18T08:19:05.886289+0000","last_unstale":"2024-09-18T08:20:16.477055+0000","last_undegraded":"2024-09-18T08:20:16.477055+0000","last_fullsized":"2024-09-18T08:20:16.477055+0000","mapping_epoch":413,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":414,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:15:44.585123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:50.886830+0000","last_clean_scrub_stamp":"2024-09-18T08:15:44.585123+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,7],"acting":[1,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":230,"reported_epoch":438,"state":"active+clean","last_fresh":"2024-09-18T08:20:16.494103+0000","last_change":"2024-09-18T08:18:17.566526+0000","last_active":"2024-09-18T08:20:16.494103+0000","last_peered":"2024-09-18T08:20:16.494103+0000","last_clean":"2024-09-18T08:20:16.494103+0000","last_became_active":"2024-09-18T08:16:13.458654+0000","last_became_peered":"2024-09-18T08:16:13.458654+0000","last_unstale":"2024-09-18T08:20:16.494103+0000","last_undegraded":"2024-09-18T08:20:16.494103+0000","last_fullsized":"2024-09-18T08:20:16.494103+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:18:17.566467+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:46.860092+0000","last_clean_scrub_stamp":"2024-09-18T08:18:17.566467+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,1],"acting":[5,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"271'15","reported_seq":451,"reported_epoch":438,"state":"active+clean","last_fresh":"2024-09-18T08:20:16.472786+0000","last_change":"2024-09-18T08:19:05.919925+0000","last_active":"2024-09-18T08:20:16.472786+0000","last_peered":"2024-09-18T08:20:16.472786+0000","last_clean":"2024-09-18T08:20:16.472786+0000","last_became_active":"2024-09-18T08:19:05.906571+0000","last_became_peered":"2024-09-18T08:19:05.906571+0000","last_unstale":"2024-09-18T08:20:16.472786+0000","last_undegraded":"2024-09-18T08:20:16.472786+0000","last_fullsized":"2024-09-18T08:20:16.472786+0000","mapping_epoch":413,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":414,"parent":"0.0","parent_split_bits":0,"last_scrub":"271'15","last_scrub_stamp":"2024-09-18T08:18:31.669089+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_clean_scrub_stamp":"2024-09-18T08:18:31.669089+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":24,"num_read_kb":18,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":245,"reported_epoch":438,"state":"active+clean","last_fresh":"2024-09-18T08:20:16.492762+0000","last_change":"2024-09-18T08:19:03.725392+0000","last_active":"2024-09-18T08:20:16.492762+0000","last_peered":"2024-09-18T08:20:16.492762+0000","last_clean":"2024-09-18T08:20:16.492762+0000","last_became_active":"2024-09-18T08:18:27.650998+0000","last_became_peered":"2024-09-18T08:18:27.650998+0000","last_unstale":"2024-09-18T08:20:16.492762+0000","last_undegraded":"2024-09-18T08:20:16.492762+0000","last_fullsized":"2024-09-18T08:20:16.492762+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:19:03.725341+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:50.886830+0000","last_clean_scrub_stamp":"2024-09-18T08:19:03.725341+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":806,"reported_epoch":438,"state":"active+clean","last_fresh":"2024-09-18T08:20:16.500007+0000","last_change":"2024-09-18T08:18:27.645124+0000","last_active":"2024-09-18T08:20:16.500007+0000","last_peered":"2024-09-18T08:20:16.500007+0000","last_clean":"2024-09-18T08:20:16.500007+0000","last_became_active":"2024-09-18T08:18:27.644825+0000","last_became_peered":"2024-09-18T08:18:27.644825+0000","last_unstale":"2024-09-18T08:20:16.500007+0000","last_undegraded":"2024-09-18T08:20:16.500007+0000","last_fullsized":"2024-09-18T08:20:16.500007+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:17:29.542288+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_clean_scrub_stamp":"2024-09-18T08:17:29.542288+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":18,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":230,"reported_epoch":438,"state":"active+clean","last_fresh":"2024-09-18T08:20:21.821235+0000","last_change":"2024-09-18T08:20:21.821235+0000","last_active":"2024-09-18T08:20:21.821235+0000","last_peered":"2024-09-18T08:20:21.821235+0000","last_clean":"2024-09-18T08:20:21.821235+0000","last_became_active":"2024-09-18T08:16:13.471003+0000","last_became_peered":"2024-09-18T08:16:13.471003+0000","last_unstale":"2024-09-18T08:20:21.821235+0000","last_undegraded":"2024-09-18T08:20:21.821235+0000","last_fullsized":"2024-09-18T08:20:21.821235+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:20:21.821158+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:46.860092+0000","last_clean_scrub_stamp":"2024-09-18T08:20:21.821158+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:21:39.737043+0000","scrub_duration":49,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1b","version":"271'7","reported_seq":119,"reported_epoch":438,"state":"active+clean","last_fresh":"2024-09-18T08:20:16.492537+0000","last_change":"2024-09-18T08:19:05.904504+0000","last_active":"2024-09-18T08:20:16.492537+0000","last_peered":"2024-09-18T08:20:16.492537+0000","last_clean":"2024-09-18T08:20:16.492537+0000","last_became_active":"2024-09-18T08:19:05.885633+0000","last_became_peered":"2024-09-18T08:19:05.885633+0000","last_unstale":"2024-09-18T08:20:16.492537+0000","last_undegraded":"2024-09-18T08:20:16.492537+0000","last_fullsized":"2024-09-18T08:20:16.492537+0000","mapping_epoch":413,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":414,"parent":"0.0","parent_split_bits":0,"last_scrub":"271'7","last_scrub_stamp":"2024-09-18T08:18:27.484925+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_clean_scrub_stamp":"2024-09-18T08:18:27.484925+0000","objects_scrubbed":2,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":13,"num_read_kb":9,"num_write":8,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":84,"reported_epoch":438,"state":"active+clean","last_fresh":"2024-09-18T08:20:16.476477+0000","last_change":"2024-09-18T08:18:27.656772+0000","last_active":"2024-09-18T08:20:16.476477+0000","last_peered":"2024-09-18T08:20:16.476477+0000","last_clean":"2024-09-18T08:20:16.476477+0000","last_became_active":"2024-09-18T08:18:27.656359+0000","last_became_peered":"2024-09-18T08:18:27.656359+0000","last_unstale":"2024-09-18T08:20:16.476477+0000","last_undegraded":"2024-09-18T08:20:16.476477+0000","last_fullsized":"2024-09-18T08:20:16.476477+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:15:40.589048+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:50.886830+0000","last_clean_scrub_stamp":"2024-09-18T08:15:40.589048+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":223,"reported_epoch":438,"state":"active+clean","last_fresh":"2024-09-18T08:20:16.492226+0000","last_change":"2024-09-18T08:18:22.573319+0000","last_active":"2024-09-18T08:20:16.492226+0000","last_peered":"2024-09-18T08:20:16.492226+0000","last_clean":"2024-09-18T08:20:16.492226+0000","last_became_active":"2024-09-18T08:15:41.441932+0000","last_became_peered":"2024-09-18T08:15:41.441932+0000","last_unstale":"2024-09-18T08:20:16.492226+0000","last_undegraded":"2024-09-18T08:20:16.492226+0000","last_fullsized":"2024-09-18T08:20:16.492226+0000","mapping_epoch":336,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":337,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:18:22.573259+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_clean_scrub_stamp":"2024-09-18T08:18:22.573259+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":530,"reported_epoch":438,"state":"active+clean","last_fresh":"2024-09-18T08:20:16.493185+0000","last_change":"2024-09-18T08:16:13.453737+0000","last_active":"2024-09-18T08:20:16.493185+0000","last_peered":"2024-09-18T08:20:16.493185+0000","last_clean":"2024-09-18T08:20:16.493185+0000","last_became_active":"2024-09-18T08:16:13.453129+0000","last_became_peered":"2024-09-18T08:16:13.453129+0000","last_unstale":"2024-09-18T08:20:16.493185+0000","last_undegraded":"2024-09-18T08:20:16.493185+0000","last_fullsized":"2024-09-18T08:20:16.493185+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:15:42.600840+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_clean_scrub_stamp":"2024-09-18T08:15:42.600840+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,3],"acting":[5,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":842,"reported_epoch":438,"state":"active+clean","last_fresh":"2024-09-18T08:20:16.501950+0000","last_change":"2024-09-18T08:18:30.893923+0000","last_active":"2024-09-18T08:20:16.501950+0000","last_peered":"2024-09-18T08:20:16.501950+0000","last_clean":"2024-09-18T08:20:16.501950+0000","last_became_active":"2024-09-18T08:18:27.643517+0000","last_became_peered":"2024-09-18T08:18:27.643517+0000","last_unstale":"2024-09-18T08:20:16.501950+0000","last_undegraded":"2024-09-18T08:20:16.501950+0000","last_fullsized":"2024-09-18T08:20:16.501950+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:18:30.893875+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:46.860092+0000","last_clean_scrub_stamp":"2024-09-18T08:18:30.893875+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,1],"acting":[0,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1c","version":"271'27","reported_seq":421,"reported_epoch":438,"state":"active+clean","last_fresh":"2024-09-18T08:20:17.800209+0000","last_change":"2024-09-18T08:20:17.800209+0000","last_active":"2024-09-18T08:20:17.800209+0000","last_peered":"2024-09-18T08:20:17.800209+0000","last_clean":"2024-09-18T08:20:17.800209+0000","last_became_active":"2024-09-18T08:19:05.912045+0000","last_became_peered":"2024-09-18T08:19:05.912045+0000","last_unstale":"2024-09-18T08:20:17.800209+0000","last_undegraded":"2024-09-18T08:20:17.800209+0000","last_fullsized":"2024-09-18T08:20:17.800209+0000","mapping_epoch":413,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":414,"parent":"0.0","parent_split_bits":0,"last_scrub":"271'27","last_scrub_stamp":"2024-09-18T08:20:17.800133+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_clean_scrub_stamp":"2024-09-18T08:20:17.800133+0000","objects_scrubbed":7,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:21:40.443704+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":44,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,0],"acting":[1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":692,"reported_epoch":438,"state":"active+clean","last_fresh":"2024-09-18T08:20:16.478066+0000","last_change":"2024-09-18T08:19:01.648780+0000","last_active":"2024-09-18T08:20:16.478066+0000","last_peered":"2024-09-18T08:20:16.478066+0000","last_clean":"2024-09-18T08:20:16.478066+0000","last_became_active":"2024-09-18T08:16:13.462303+0000","last_became_peered":"2024-09-18T08:16:13.462303+0000","last_unstale":"2024-09-18T08:20:16.478066+0000","last_undegraded":"2024-09-18T08:20:16.478066+0000","last_fullsized":"2024-09-18T08:20:16.478066+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:19:01.648717+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:50.886830+0000","last_clean_scrub_stamp":"2024-09-18T08:19:01.648717+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:20:20.414872+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,5],"acting":[1,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":820,"reported_epoch":438,"state":"active+clean","last_fresh":"2024-09-18T08:20:22.001150+0000","last_change":"2024-09-18T08:20:22.001150+0000","last_active":"2024-09-18T08:20:22.001150+0000","last_peered":"2024-09-18T08:20:22.001150+0000","last_clean":"2024-09-18T08:20:22.001150+0000","last_became_active":"2024-09-18T08:15:41.466371+0000","last_became_peered":"2024-09-18T08:15:41.466371+0000","last_unstale":"2024-09-18T08:20:22.001150+0000","last_undegraded":"2024-09-18T08:20:22.001150+0000","last_fullsized":"2024-09-18T08:20:22.001150+0000","mapping_epoch":336,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":337,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:20:22.001092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_clean_scrub_stamp":"2024-09-18T08:20:22.001092+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:21:31.821107+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":474,"reported_epoch":438,"state":"active+clean","last_fresh":"2024-09-18T08:20:16.844354+0000","last_change":"2024-09-18T08:20:16.844354+0000","last_active":"2024-09-18T08:20:16.844354+0000","last_peered":"2024-09-18T08:20:16.844354+0000","last_clean":"2024-09-18T08:20:16.844354+0000","last_became_active":"2024-09-18T08:19:05.899551+0000","last_became_peered":"2024-09-18T08:19:05.899551+0000","last_unstale":"2024-09-18T08:20:16.844354+0000","last_undegraded":"2024-09-18T08:20:16.844354+0000","last_fullsized":"2024-09-18T08:20:16.844354+0000","mapping_epoch":413,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":414,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:20:16.844283+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:46.860092+0000","last_clean_scrub_stamp":"2024-09-18T08:20:16.844283+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:21:46.011768+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,7],"acting":[1,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1d","version":"271'18","reported_seq":935,"reported_epoch":438,"state":"active+clean","last_fresh":"2024-09-18T08:20:16.473251+0000","last_change":"2024-09-18T08:18:19.829522+0000","last_active":"2024-09-18T08:20:16.473251+0000","last_peered":"2024-09-18T08:20:16.473251+0000","last_clean":"2024-09-18T08:20:16.473251+0000","last_became_active":"2024-09-18T08:18:19.823337+0000","last_became_peered":"2024-09-18T08:18:19.823337+0000","last_unstale":"2024-09-18T08:20:16.473251+0000","last_undegraded":"2024-09-18T08:20:16.473251+0000","last_fullsized":"2024-09-18T08:20:16.473251+0000","mapping_epoch":397,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":398,"parent":"0.0","parent_split_bits":0,"last_scrub":"271'18","last_scrub_stamp":"2024-09-18T08:17:04.627913+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_clean_scrub_stamp":"2024-09-18T08:17:04.627913+0000","objects_scrubbed":5,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":34,"num_read_kb":25,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":23,"num_bytes_recovered":880,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":242,"reported_epoch":438,"state":"active+clean","last_fresh":"2024-09-18T08:20:16.473988+0000","last_change":"2024-09-18T08:19:05.902064+0000","last_active":"2024-09-18T08:20:16.473988+0000","last_peered":"2024-09-18T08:20:16.473988+0000","last_clean":"2024-09-18T08:20:16.473988+0000","last_became_active":"2024-09-18T08:19:05.901172+0000","last_became_peered":"2024-09-18T08:19:05.901172+0000","last_unstale":"2024-09-18T08:20:16.473988+0000","last_undegraded":"2024-09-18T08:20:16.473988+0000","last_fullsized":"2024-09-18T08:20:16.473988+0000","mapping_epoch":413,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":414,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:18:18.614698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:50.886830+0000","last_clean_scrub_stamp":"2024-09-18T08:18:18.614698+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.1c","version":"69'1","reported_seq":238,"reported_epoch":438,"state":"active+clean","last_fresh":"2024-09-18T08:20:18.804447+0000","last_change":"2024-09-18T08:20:18.804447+0000","last_active":"2024-09-18T08:20:18.804447+0000","last_peered":"2024-09-18T08:20:18.804447+0000","last_clean":"2024-09-18T08:20:18.804447+0000","last_became_active":"2024-09-18T08:18:27.642951+0000","last_became_peered":"2024-09-18T08:18:27.642951+0000","last_unstale":"2024-09-18T08:20:18.804447+0000","last_undegraded":"2024-09-18T08:20:18.804447+0000","last_fullsized":"2024-09-18T08:20:18.804447+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'1","last_scrub_stamp":"2024-09-18T08:20:18.804376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_clean_scrub_stamp":"2024-09-18T08:20:18.804376+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:21:43.741444+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":403,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":1209,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,2],"acting":[5,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"62'1","reported_seq":697,"reported_epoch":438,"state":"active+clean","last_fresh":"2024-09-18T08:20:16.477254+0000","last_change":"2024-09-18T08:17:37.618000+0000","last_active":"2024-09-18T08:20:16.477254+0000","last_peered":"2024-09-18T08:20:16.477254+0000","last_clean":"2024-09-18T08:20:16.477254+0000","last_became_active":"2024-09-18T08:16:13.483397+0000","last_became_peered":"2024-09-18T08:16:13.483397+0000","last_unstale":"2024-09-18T08:20:16.477254+0000","last_undegraded":"2024-09-18T08:20:16.477254+0000","last_fullsized":"2024-09-18T08:20:16.477254+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"62'1","last_scrub_stamp":"2024-09-18T08:17:37.617954+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:46.860092+0000","last_clean_scrub_stamp":"2024-09-18T08:17:37.617954+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":46,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,6],"acting":[1,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1e","version":"271'20","reported_seq":864,"reported_epoch":438,"state":"active+clean","last_fresh":"2024-09-18T08:20:16.501417+0000","last_change":"2024-09-18T08:18:27.690042+0000","last_active":"2024-09-18T08:20:16.501417+0000","last_peered":"2024-09-18T08:20:16.501417+0000","last_clean":"2024-09-18T08:20:16.501417+0000","last_became_active":"2024-09-18T08:18:27.657472+0000","last_became_peered":"2024-09-18T08:18:27.657472+0000","last_unstale":"2024-09-18T08:20:16.501417+0000","last_undegraded":"2024-09-18T08:20:16.501417+0000","last_fullsized":"2024-09-18T08:20:16.501417+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":0,"last_scrub":"271'20","last_scrub_stamp":"2024-09-18T08:15:40.251706+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_clean_scrub_stamp":"2024-09-18T08:15:40.251706+0000","objects_scrubbed":5,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":25,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1f","version":"415'17","reported_seq":124,"reported_epoch":438,"state":"active+clean","last_fresh":"2024-09-18T08:20:23.141376+0000","last_change":"2024-09-18T08:19:05.911190+0000","last_active":"2024-09-18T08:20:23.141376+0000","last_peered":"2024-09-18T08:20:23.141376+0000","last_clean":"2024-09-18T08:20:23.141376+0000","last_became_active":"2024-09-18T08:19:05.901152+0000","last_became_peered":"2024-09-18T08:19:05.901152+0000","last_unstale":"2024-09-18T08:20:23.141376+0000","last_undegraded":"2024-09-18T08:20:23.141376+0000","last_fullsized":"2024-09-18T08:20:23.141376+0000","mapping_epoch":413,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":414,"parent":"0.0","parent_split_bits":0,"last_scrub":"349'15","last_scrub_stamp":"2024-09-18T08:17:29.878376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:50.886830+0000","last_clean_scrub_stamp":"2024-09-18T08:17:29.878376+0000","objects_scrubbed":1,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.0","version":"438'7945","reported_seq":8030,"reported_epoch":438,"state":"active+clean","last_fresh":"2024-09-18T08:20:28.116077+0000","last_change":"2024-09-18T08:18:27.643125+0000","last_active":"2024-09-18T08:20:28.116077+0000","last_peered":"2024-09-18T08:20:28.116077+0000","last_clean":"2024-09-18T08:20:28.116077+0000","last_became_active":"2024-09-18T08:18:27.642627+0000","last_became_peered":"2024-09-18T08:18:27.642627+0000","last_unstale":"2024-09-18T08:20:28.116077+0000","last_undegraded":"2024-09-18T08:20:28.116077+0000","last_fullsized":"2024-09-18T08:20:28.116077+0000","mapping_epoch":401,"log_start":"423'4300","ondisk_log_start":"423'4300","created":401,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:18:26.612016+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:18:26.612016+0000","last_clean_scrub_stamp":"2024-09-18T08:18:26.612016+0000","objects_scrubbed":0,"log_size":3645,"log_dups_size":0,"ondisk_log_size":3645,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":300089344,"num_objects":4579,"num_object_clones":0,"num_object_copies":13737,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4579,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7945,"num_write_kb":400768,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,7],"acting":[1,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.a","version":"74'79","reported_seq":224,"reported_epoch":438,"state":"active+clean","last_fresh":"2024-09-18T08:20:16.492713+0000","last_change":"2024-09-18T08:15:42.288426+0000","last_active":"2024-09-18T08:20:16.492713+0000","last_peered":"2024-09-18T08:20:16.492713+0000","last_clean":"2024-09-18T08:20:16.492713+0000","last_became_active":"2024-09-18T08:15:41.443376+0000","last_became_peered":"2024-09-18T08:15:41.443376+0000","last_unstale":"2024-09-18T08:20:16.492713+0000","last_undegraded":"2024-09-18T08:20:16.492713+0000","last_fullsized":"2024-09-18T08:20:16.492713+0000","mapping_epoch":336,"log_start":"0'0","ondisk_log_start":"0'0","created":152,"last_epoch_clean":337,"parent":"0.0","parent_split_bits":4,"last_scrub":"74'79","last_scrub_stamp":"2024-09-18T08:15:42.288363+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:00:11.893742+0000","last_clean_scrub_stamp":"2024-09-18T08:15:42.288363+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,6],"acting":[5,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.f","version":"271'27","reported_seq":784,"reported_epoch":438,"state":"active+clean","last_fresh":"2024-09-18T08:20:16.478499+0000","last_change":"2024-09-18T08:18:57.614756+0000","last_active":"2024-09-18T08:20:16.478499+0000","last_peered":"2024-09-18T08:20:16.478499+0000","last_clean":"2024-09-18T08:20:16.478499+0000","last_became_active":"2024-09-18T08:18:27.652553+0000","last_became_peered":"2024-09-18T08:18:27.652553+0000","last_unstale":"2024-09-18T08:20:16.478499+0000","last_undegraded":"2024-09-18T08:20:16.478499+0000","last_fullsized":"2024-09-18T08:20:16.478499+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":0,"last_scrub":"271'27","last_scrub_stamp":"2024-09-18T08:18:57.614509+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_clean_scrub_stamp":"2024-09-18T08:18:57.614509+0000","objects_scrubbed":7,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:20:21.981757+0000","scrub_duration":16,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":43,"num_read_kb":34,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,7],"acting":[1,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":108,"reported_epoch":438,"state":"active+clean","last_fresh":"2024-09-18T08:20:16.472265+0000","last_change":"2024-09-18T08:18:27.655637+0000","last_active":"2024-09-18T08:20:16.472265+0000","last_peered":"2024-09-18T08:20:16.472265+0000","last_clean":"2024-09-18T08:20:16.472265+0000","last_became_active":"2024-09-18T08:18:27.655321+0000","last_became_peered":"2024-09-18T08:18:27.655321+0000","last_unstale":"2024-09-18T08:20:16.472265+0000","last_undegraded":"2024-09-18T08:20:16.472265+0000","last_fullsized":"2024-09-18T08:20:16.472265+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:18:23.067375+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:46.860092+0000","last_clean_scrub_stamp":"2024-09-18T08:18:23.067375+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:20:30.072236+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,7],"acting":[2,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.e","version":"415'19","reported_seq":735,"reported_epoch":438,"state":"active+clean","last_fresh":"2024-09-18T08:20:23.141159+0000","last_change":"2024-09-18T08:19:06.698834+0000","last_active":"2024-09-18T08:20:23.141159+0000","last_peered":"2024-09-18T08:20:23.141159+0000","last_clean":"2024-09-18T08:20:23.141159+0000","last_became_active":"2024-09-18T08:19:05.903139+0000","last_became_peered":"2024-09-18T08:19:05.903139+0000","last_unstale":"2024-09-18T08:20:23.141159+0000","last_undegraded":"2024-09-18T08:20:23.141159+0000","last_fullsized":"2024-09-18T08:20:23.141159+0000","mapping_epoch":413,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":414,"parent":"0.0","parent_split_bits":0,"last_scrub":"349'16","last_scrub_stamp":"2024-09-18T08:19:06.698769+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:50.886830+0000","last_clean_scrub_stamp":"2024-09-18T08:19:06.698769+0000","objects_scrubbed":1,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,3],"acting":[5,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":224,"reported_epoch":438,"state":"active+clean","last_fresh":"2024-09-18T08:20:16.493232+0000","last_change":"2024-09-18T08:18:56.732088+0000","last_active":"2024-09-18T08:20:16.493232+0000","last_peered":"2024-09-18T08:20:16.493232+0000","last_clean":"2024-09-18T08:20:16.493232+0000","last_became_active":"2024-09-18T08:15:41.438610+0000","last_became_peered":"2024-09-18T08:15:41.438610+0000","last_unstale":"2024-09-18T08:20:16.493232+0000","last_undegraded":"2024-09-18T08:20:16.493232+0000","last_fullsized":"2024-09-18T08:20:16.493232+0000","mapping_epoch":336,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":337,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:18:56.732041+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_clean_scrub_stamp":"2024-09-18T08:18:56.732041+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.f","version":"438'7762","reported_seq":14185,"reported_epoch":438,"state":"active+clean","last_fresh":"2024-09-18T08:20:28.115811+0000","last_change":"2024-09-18T08:20:15.485566+0000","last_active":"2024-09-18T08:20:28.115811+0000","last_peered":"2024-09-18T08:20:28.115811+0000","last_clean":"2024-09-18T08:20:28.115811+0000","last_became_active":"2024-09-18T08:20:15.477846+0000","last_became_peered":"2024-09-18T08:20:15.477846+0000","last_unstale":"2024-09-18T08:20:28.115811+0000","last_undegraded":"2024-09-18T08:20:28.115811+0000","last_fullsized":"2024-09-18T08:20:28.115811+0000","mapping_epoch":436,"log_start":"422'4100","ondisk_log_start":"422'4100","created":401,"last_epoch_clean":437,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:18:26.612016+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:18:26.612016+0000","last_clean_scrub_stamp":"2024-09-18T08:18:26.612016+0000","objects_scrubbed":0,"log_size":3662,"log_dups_size":0,"ondisk_log_size":3662,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":298450944,"num_objects":4554,"num_object_clones":0,"num_object_copies":13662,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4554,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7762,"num_write_kb":394112,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3163,"num_bytes_recovered":207290368,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.5","version":"74'79","reported_seq":114,"reported_epoch":438,"state":"active+clean","last_fresh":"2024-09-18T08:20:16.493737+0000","last_change":"2024-09-18T08:19:05.894246+0000","last_active":"2024-09-18T08:20:16.493737+0000","last_peered":"2024-09-18T08:20:16.493737+0000","last_clean":"2024-09-18T08:20:16.493737+0000","last_became_active":"2024-09-18T08:19:05.893865+0000","last_became_peered":"2024-09-18T08:19:05.893865+0000","last_unstale":"2024-09-18T08:20:16.493737+0000","last_undegraded":"2024-09-18T08:20:16.493737+0000","last_fullsized":"2024-09-18T08:20:16.493737+0000","mapping_epoch":413,"log_start":"0'0","ondisk_log_start":"0'0","created":152,"last_epoch_clean":414,"parent":"0.0","parent_split_bits":3,"last_scrub":"74'79","last_scrub_stamp":"2024-09-18T08:18:14.059977+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:00:11.893742+0000","last_clean_scrub_stamp":"2024-09-18T08:18:14.059977+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.0","version":"271'30","reported_seq":913,"reported_epoch":438,"state":"active+clean","last_fresh":"2024-09-18T08:20:16.472506+0000","last_change":"2024-09-18T08:18:27.693740+0000","last_active":"2024-09-18T08:20:16.472506+0000","last_peered":"2024-09-18T08:20:16.472506+0000","last_clean":"2024-09-18T08:20:16.472506+0000","last_became_active":"2024-09-18T08:18:27.651348+0000","last_became_peered":"2024-09-18T08:18:27.651348+0000","last_unstale":"2024-09-18T08:20:16.472506+0000","last_undegraded":"2024-09-18T08:20:16.472506+0000","last_fullsized":"2024-09-18T08:20:16.472506+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":0,"last_scrub":"271'30","last_scrub_stamp":"2024-09-18T08:15:57.927111+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_clean_scrub_stamp":"2024-09-18T08:15:57.927111+0000","objects_scrubbed":8,"log_size":30,"log_dups_size":0,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":18,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":52,"num_read_kb":40,"num_write":32,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":856,"reported_epoch":438,"state":"active+clean","last_fresh":"2024-09-18T08:20:22.993934+0000","last_change":"2024-09-18T08:20:22.993934+0000","last_active":"2024-09-18T08:20:22.993934+0000","last_peered":"2024-09-18T08:20:22.993934+0000","last_clean":"2024-09-18T08:20:22.993934+0000","last_became_active":"2024-09-18T08:18:27.663173+0000","last_became_peered":"2024-09-18T08:18:27.663173+0000","last_unstale":"2024-09-18T08:20:22.993934+0000","last_undegraded":"2024-09-18T08:20:22.993934+0000","last_fullsized":"2024-09-18T08:20:22.993934+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:20:22.993864+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:46.860092+0000","last_clean_scrub_stamp":"2024-09-18T08:20:22.993864+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:21:28.227839+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":457,"reported_epoch":438,"state":"active+clean","last_fresh":"2024-09-18T08:20:16.474013+0000","last_change":"2024-09-18T08:18:27.661806+0000","last_active":"2024-09-18T08:20:16.474013+0000","last_peered":"2024-09-18T08:20:16.474013+0000","last_clean":"2024-09-18T08:20:16.474013+0000","last_became_active":"2024-09-18T08:18:27.661393+0000","last_became_peered":"2024-09-18T08:18:27.661393+0000","last_unstale":"2024-09-18T08:20:16.474013+0000","last_undegraded":"2024-09-18T08:20:16.474013+0000","last_fullsized":"2024-09-18T08:20:16.474013+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:17:31.417453+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:50.886830+0000","last_clean_scrub_stamp":"2024-09-18T08:17:31.417453+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":237,"reported_epoch":438,"state":"active+clean","last_fresh":"2024-09-18T08:20:16.472721+0000","last_change":"2024-09-18T08:19:06.578252+0000","last_active":"2024-09-18T08:20:16.472721+0000","last_peered":"2024-09-18T08:20:16.472721+0000","last_clean":"2024-09-18T08:20:16.472721+0000","last_became_active":"2024-09-18T08:19:05.893777+0000","last_became_peered":"2024-09-18T08:19:05.893777+0000","last_unstale":"2024-09-18T08:20:16.472721+0000","last_undegraded":"2024-09-18T08:20:16.472721+0000","last_fullsized":"2024-09-18T08:20:16.472721+0000","mapping_epoch":413,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":414,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:19:06.578190+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_clean_scrub_stamp":"2024-09-18T08:19:06.578190+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,7],"acting":[3,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.e","version":"438'7782","reported_seq":7868,"reported_epoch":438,"state":"active+clean","last_fresh":"2024-09-18T08:20:28.075869+0000","last_change":"2024-09-18T08:18:27.652534+0000","last_active":"2024-09-18T08:20:28.075869+0000","last_peered":"2024-09-18T08:20:28.075869+0000","last_clean":"2024-09-18T08:20:28.075869+0000","last_became_active":"2024-09-18T08:18:27.652261+0000","last_became_peered":"2024-09-18T08:18:27.652261+0000","last_unstale":"2024-09-18T08:20:28.075869+0000","last_undegraded":"2024-09-18T08:20:28.075869+0000","last_fullsized":"2024-09-18T08:20:28.075869+0000","mapping_epoch":401,"log_start":"422'4200","ondisk_log_start":"422'4200","created":401,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:18:26.612016+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:18:26.612016+0000","last_clean_scrub_stamp":"2024-09-18T08:18:26.612016+0000","objects_scrubbed":0,"log_size":3582,"log_dups_size":0,"ondisk_log_size":3582,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":300154880,"num_objects":4580,"num_object_clones":0,"num_object_copies":13740,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4580,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7782,"num_write_kb":395584,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"74'79","reported_seq":675,"reported_epoch":438,"state":"active+clean","last_fresh":"2024-09-18T08:20:16.472509+0000","last_change":"2024-09-18T08:18:27.688842+0000","last_active":"2024-09-18T08:20:16.472509+0000","last_peered":"2024-09-18T08:20:16.472509+0000","last_clean":"2024-09-18T08:20:16.472509+0000","last_became_active":"2024-09-18T08:18:27.662304+0000","last_became_peered":"2024-09-18T08:18:27.662304+0000","last_unstale":"2024-09-18T08:20:16.472509+0000","last_undegraded":"2024-09-18T08:20:16.472509+0000","last_fullsized":"2024-09-18T08:20:16.472509+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":152,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":3,"last_scrub":"74'79","last_scrub_stamp":"2024-09-18T08:15:41.898294+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:00:11.893742+0000","last_clean_scrub_stamp":"2024-09-18T08:15:41.898294+0000","objects_scrubbed":1,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":327680,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1310720,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,7],"acting":[3,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"271'28","reported_seq":599,"reported_epoch":438,"state":"active+clean","last_fresh":"2024-09-18T08:20:16.492254+0000","last_change":"2024-09-18T08:16:13.506782+0000","last_active":"2024-09-18T08:20:16.492254+0000","last_peered":"2024-09-18T08:20:16.492254+0000","last_clean":"2024-09-18T08:20:16.492254+0000","last_became_active":"2024-09-18T08:16:13.451205+0000","last_became_peered":"2024-09-18T08:16:13.451205+0000","last_unstale":"2024-09-18T08:20:16.492254+0000","last_undegraded":"2024-09-18T08:20:16.492254+0000","last_fullsized":"2024-09-18T08:20:16.492254+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"271'28","last_scrub_stamp":"2024-09-18T08:15:51.716119+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_clean_scrub_stamp":"2024-09-18T08:15:51.716119+0000","objects_scrubbed":7,"log_size":28,"log_dups_size":0,"ondisk_log_size":28,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":41,"num_read_kb":34,"num_write":28,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.6","version":"62'2","reported_seq":873,"reported_epoch":438,"state":"active+clean","last_fresh":"2024-09-18T08:20:17.276808+0000","last_change":"2024-09-18T08:20:17.276808+0000","last_active":"2024-09-18T08:20:17.276808+0000","last_peered":"2024-09-18T08:20:17.276808+0000","last_clean":"2024-09-18T08:20:17.276808+0000","last_became_active":"2024-09-18T08:18:27.645204+0000","last_became_peered":"2024-09-18T08:18:27.645204+0000","last_unstale":"2024-09-18T08:20:17.276808+0000","last_undegraded":"2024-09-18T08:20:17.276808+0000","last_fullsized":"2024-09-18T08:20:17.276808+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":0,"last_scrub":"62'2","last_scrub_stamp":"2024-09-18T08:20:17.276736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:46.860092+0000","last_clean_scrub_stamp":"2024-09-18T08:20:17.276736+0000","objects_scrubbed":2,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:21:22.408837+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1039,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":4156,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.0","version":"349'13","reported_seq":1054,"reported_epoch":438,"state":"active+clean","last_fresh":"2024-09-18T08:20:18.146148+0000","last_change":"2024-09-18T08:16:13.516195+0000","last_active":"2024-09-18T08:20:18.146148+0000","last_peered":"2024-09-18T08:20:18.146148+0000","last_clean":"2024-09-18T08:20:18.146148+0000","last_became_active":"2024-09-18T08:16:13.477431+0000","last_became_peered":"2024-09-18T08:16:13.477431+0000","last_unstale":"2024-09-18T08:20:18.146148+0000","last_undegraded":"2024-09-18T08:20:18.146148+0000","last_fullsized":"2024-09-18T08:20:18.146148+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"310'12","last_scrub_stamp":"2024-09-18T08:15:49.795095+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:50.886830+0000","last_clean_scrub_stamp":"2024-09-18T08:15:49.795095+0000","objects_scrubbed":1,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":224,"reported_epoch":438,"state":"active+clean","last_fresh":"2024-09-18T08:20:25.734132+0000","last_change":"2024-09-18T08:20:25.734132+0000","last_active":"2024-09-18T08:20:25.734132+0000","last_peered":"2024-09-18T08:20:25.734132+0000","last_clean":"2024-09-18T08:20:25.734132+0000","last_became_active":"2024-09-18T08:15:41.440876+0000","last_became_peered":"2024-09-18T08:15:41.440876+0000","last_unstale":"2024-09-18T08:20:25.734132+0000","last_undegraded":"2024-09-18T08:20:25.734132+0000","last_fullsized":"2024-09-18T08:20:25.734132+0000","mapping_epoch":336,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":337,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:20:25.734050+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_clean_scrub_stamp":"2024-09-18T08:20:25.734050+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:21:47.697644+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.d","version":"438'7705","reported_seq":14119,"reported_epoch":438,"state":"active+clean","last_fresh":"2024-09-18T08:20:28.071766+0000","last_change":"2024-09-18T08:20:06.497992+0000","last_active":"2024-09-18T08:20:28.071766+0000","last_peered":"2024-09-18T08:20:28.071766+0000","last_clean":"2024-09-18T08:20:28.071766+0000","last_became_active":"2024-09-18T08:20:06.492093+0000","last_became_peered":"2024-09-18T08:20:06.492093+0000","last_unstale":"2024-09-18T08:20:28.071766+0000","last_undegraded":"2024-09-18T08:20:28.071766+0000","last_fullsized":"2024-09-18T08:20:28.071766+0000","mapping_epoch":431,"log_start":"422'4100","ondisk_log_start":"422'4100","created":401,"last_epoch_clean":432,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:18:26.612016+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:18:26.612016+0000","last_clean_scrub_stamp":"2024-09-18T08:18:26.612016+0000","objects_scrubbed":0,"log_size":3605,"log_dups_size":0,"ondisk_log_size":3605,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":287440920,"num_objects":4387,"num_object_clones":0,"num_object_copies":13161,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4387,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7705,"num_write_kb":386881,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3158,"num_bytes_recovered":206962688,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,5],"acting":[1,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"74'79","reported_seq":653,"reported_epoch":438,"state":"active+clean","last_fresh":"2024-09-18T08:20:16.473689+0000","last_change":"2024-09-18T08:18:27.663511+0000","last_active":"2024-09-18T08:20:16.473689+0000","last_peered":"2024-09-18T08:20:16.473689+0000","last_clean":"2024-09-18T08:20:16.473689+0000","last_became_active":"2024-09-18T08:18:27.662723+0000","last_became_peered":"2024-09-18T08:18:27.662723+0000","last_unstale":"2024-09-18T08:20:16.473689+0000","last_undegraded":"2024-09-18T08:20:16.473689+0000","last_fullsized":"2024-09-18T08:20:16.473689+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":152,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":3,"last_scrub":"74'79","last_scrub_stamp":"2024-09-18T08:15:44.872332+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:00:11.893742+0000","last_clean_scrub_stamp":"2024-09-18T08:15:44.872332+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,7],"acting":[3,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.2","version":"271'20","reported_seq":816,"reported_epoch":438,"state":"active+clean","last_fresh":"2024-09-18T08:20:16.477352+0000","last_change":"2024-09-18T08:18:22.653813+0000","last_active":"2024-09-18T08:20:16.477352+0000","last_peered":"2024-09-18T08:20:16.477352+0000","last_clean":"2024-09-18T08:20:16.477352+0000","last_became_active":"2024-09-18T08:16:13.486720+0000","last_became_peered":"2024-09-18T08:16:13.486720+0000","last_unstale":"2024-09-18T08:20:16.477352+0000","last_undegraded":"2024-09-18T08:20:16.477352+0000","last_fullsized":"2024-09-18T08:20:16.477352+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"271'20","last_scrub_stamp":"2024-09-18T08:18:22.653761+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_clean_scrub_stamp":"2024-09-18T08:18:22.653761+0000","objects_scrubbed":5,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":28,"num_read_kb":23,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":224,"reported_epoch":438,"state":"active+clean","last_fresh":"2024-09-18T08:20:22.733384+0000","last_change":"2024-09-18T08:20:22.733384+0000","last_active":"2024-09-18T08:20:22.733384+0000","last_peered":"2024-09-18T08:20:22.733384+0000","last_clean":"2024-09-18T08:20:22.733384+0000","last_became_active":"2024-09-18T08:15:41.449639+0000","last_became_peered":"2024-09-18T08:15:41.449639+0000","last_unstale":"2024-09-18T08:20:22.733384+0000","last_undegraded":"2024-09-18T08:20:22.733384+0000","last_fullsized":"2024-09-18T08:20:22.733384+0000","mapping_epoch":336,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":337,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:20:22.733321+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:46.860092+0000","last_clean_scrub_stamp":"2024-09-18T08:20:22.733321+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:21:41.865798+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.3","version":"338'14","reported_seq":1055,"reported_epoch":438,"state":"active+clean","last_fresh":"2024-09-18T08:20:23.140970+0000","last_change":"2024-09-18T08:15:41.467411+0000","last_active":"2024-09-18T08:20:23.140970+0000","last_peered":"2024-09-18T08:20:23.140970+0000","last_clean":"2024-09-18T08:20:23.140970+0000","last_became_active":"2024-09-18T08:15:41.447667+0000","last_became_peered":"2024-09-18T08:15:41.447667+0000","last_unstale":"2024-09-18T08:20:23.140970+0000","last_undegraded":"2024-09-18T08:20:23.140970+0000","last_fullsized":"2024-09-18T08:20:23.140970+0000","mapping_epoch":336,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":337,"parent":"0.0","parent_split_bits":0,"last_scrub":"317'11","last_scrub_stamp":"2024-09-18T08:14:59.207667+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:50.886830+0000","last_clean_scrub_stamp":"2024-09-18T08:14:59.207667+0000","objects_scrubbed":1,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":829,"reported_epoch":438,"state":"active+clean","last_fresh":"2024-09-18T08:20:16.500163+0000","last_change":"2024-09-18T08:19:05.899449+0000","last_active":"2024-09-18T08:20:16.500163+0000","last_peered":"2024-09-18T08:20:16.500163+0000","last_clean":"2024-09-18T08:20:16.500163+0000","last_became_active":"2024-09-18T08:19:05.899084+0000","last_became_peered":"2024-09-18T08:19:05.899084+0000","last_unstale":"2024-09-18T08:20:16.500163+0000","last_undegraded":"2024-09-18T08:20:16.500163+0000","last_fullsized":"2024-09-18T08:20:16.500163+0000","mapping_epoch":413,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":414,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:18:23.800677+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_clean_scrub_stamp":"2024-09-18T08:18:23.800677+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,5],"acting":[0,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.c","version":"438'7786","reported_seq":14213,"reported_epoch":438,"state":"active+clean","last_fresh":"2024-09-18T08:20:28.124590+0000","last_change":"2024-09-18T08:20:13.526077+0000","last_active":"2024-09-18T08:20:28.124590+0000","last_peered":"2024-09-18T08:20:28.124590+0000","last_clean":"2024-09-18T08:20:28.124590+0000","last_became_active":"2024-09-18T08:20:13.520037+0000","last_became_peered":"2024-09-18T08:20:13.520037+0000","last_unstale":"2024-09-18T08:20:28.124590+0000","last_undegraded":"2024-09-18T08:20:28.124590+0000","last_fullsized":"2024-09-18T08:20:28.124590+0000","mapping_epoch":434,"log_start":"423'4200","ondisk_log_start":"423'4200","created":401,"last_epoch_clean":435,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:18:26.612016+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:18:26.612016+0000","last_clean_scrub_stamp":"2024-09-18T08:18:26.612016+0000","objects_scrubbed":0,"log_size":3586,"log_dups_size":0,"ondisk_log_size":3586,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":293601280,"num_objects":4480,"num_object_clones":0,"num_object_copies":13440,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4480,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7786,"num_write_kb":392512,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3168,"num_bytes_recovered":207618048,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,5],"acting":[1,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"74'79","reported_seq":714,"reported_epoch":438,"state":"active+clean","last_fresh":"2024-09-18T08:20:16.500890+0000","last_change":"2024-09-18T08:18:20.837291+0000","last_active":"2024-09-18T08:20:16.500890+0000","last_peered":"2024-09-18T08:20:16.500890+0000","last_clean":"2024-09-18T08:20:16.500890+0000","last_became_active":"2024-09-18T08:07:26.426532+0000","last_became_peered":"2024-09-18T08:07:26.426532+0000","last_unstale":"2024-09-18T08:20:16.500890+0000","last_undegraded":"2024-09-18T08:20:16.500890+0000","last_fullsized":"2024-09-18T08:20:16.500890+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":152,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":3,"last_scrub":"74'79","last_scrub_stamp":"2024-09-18T08:18:20.837241+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:00:11.893742+0000","last_clean_scrub_stamp":"2024-09-18T08:18:20.837241+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,3],"acting":[0,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.3","version":"271'29","reported_seq":955,"reported_epoch":438,"state":"active+clean","last_fresh":"2024-09-18T08:20:16.501553+0000","last_change":"2024-09-18T08:18:27.685772+0000","last_active":"2024-09-18T08:20:16.501553+0000","last_peered":"2024-09-18T08:20:16.501553+0000","last_clean":"2024-09-18T08:20:16.501553+0000","last_became_active":"2024-09-18T08:18:27.654192+0000","last_became_peered":"2024-09-18T08:18:27.654192+0000","last_unstale":"2024-09-18T08:20:16.501553+0000","last_undegraded":"2024-09-18T08:20:16.501553+0000","last_fullsized":"2024-09-18T08:20:16.501553+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":0,"last_scrub":"271'29","last_scrub_stamp":"2024-09-18T08:18:15.879919+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_clean_scrub_stamp":"2024-09-18T08:18:15.879919+0000","objects_scrubbed":8,"log_size":29,"log_dups_size":0,"ondisk_log_size":29,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":54,"num_read_kb":40,"num_write":32,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":32,"num_bytes_recovered":1320,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.4","version":"62'1","reported_seq":700,"reported_epoch":438,"state":"active+clean","last_fresh":"2024-09-18T08:20:22.818710+0000","last_change":"2024-09-18T08:20:22.818710+0000","last_active":"2024-09-18T08:20:22.818710+0000","last_peered":"2024-09-18T08:20:22.818710+0000","last_clean":"2024-09-18T08:20:22.818710+0000","last_became_active":"2024-09-18T08:19:05.911595+0000","last_became_peered":"2024-09-18T08:19:05.911595+0000","last_unstale":"2024-09-18T08:20:22.818710+0000","last_undegraded":"2024-09-18T08:20:22.818710+0000","last_fullsized":"2024-09-18T08:20:22.818710+0000","mapping_epoch":413,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":414,"parent":"0.0","parent_split_bits":0,"last_scrub":"62'1","last_scrub_stamp":"2024-09-18T08:20:22.818512+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:46.860092+0000","last_clean_scrub_stamp":"2024-09-18T08:20:22.818512+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:21:23.194435+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":436,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1744,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":735,"reported_epoch":438,"state":"active+clean","last_fresh":"2024-09-18T08:20:16.500471+0000","last_change":"2024-09-18T08:18:58.096376+0000","last_active":"2024-09-18T08:20:16.500471+0000","last_peered":"2024-09-18T08:20:16.500471+0000","last_clean":"2024-09-18T08:20:16.500471+0000","last_became_active":"2024-09-18T08:15:41.449832+0000","last_became_peered":"2024-09-18T08:15:41.449832+0000","last_unstale":"2024-09-18T08:20:16.500471+0000","last_undegraded":"2024-09-18T08:20:16.500471+0000","last_fullsized":"2024-09-18T08:20:16.500471+0000","mapping_epoch":336,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":337,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:18:58.096199+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:50.886830+0000","last_clean_scrub_stamp":"2024-09-18T08:18:58.096199+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:20:19.661202+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":715,"reported_epoch":438,"state":"active+clean","last_fresh":"2024-09-18T08:20:27.908933+0000","last_change":"2024-09-18T08:20:27.908933+0000","last_active":"2024-09-18T08:20:27.908933+0000","last_peered":"2024-09-18T08:20:27.908933+0000","last_clean":"2024-09-18T08:20:27.908933+0000","last_became_active":"2024-09-18T08:19:05.881711+0000","last_became_peered":"2024-09-18T08:19:05.881711+0000","last_unstale":"2024-09-18T08:20:27.908933+0000","last_undegraded":"2024-09-18T08:20:27.908933+0000","last_fullsized":"2024-09-18T08:20:27.908933+0000","mapping_epoch":413,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":414,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:20:27.908874+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_clean_scrub_stamp":"2024-09-18T08:20:27.908874+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:21:51.062812+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,7],"acting":[1,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.b","version":"438'7463","reported_seq":7548,"reported_epoch":438,"state":"active+clean","last_fresh":"2024-09-18T08:20:24.826084+0000","last_change":"2024-09-18T08:18:27.656478+0000","last_active":"2024-09-18T08:20:24.826084+0000","last_peered":"2024-09-18T08:20:24.826084+0000","last_clean":"2024-09-18T08:20:24.826084+0000","last_became_active":"2024-09-18T08:18:27.656216+0000","last_became_peered":"2024-09-18T08:18:27.656216+0000","last_unstale":"2024-09-18T08:20:24.826084+0000","last_undegraded":"2024-09-18T08:20:24.826084+0000","last_fullsized":"2024-09-18T08:20:24.826084+0000","mapping_epoch":401,"log_start":"417'3500","ondisk_log_start":"417'3500","created":401,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:18:26.612016+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:18:26.612016+0000","last_clean_scrub_stamp":"2024-09-18T08:18:26.612016+0000","objects_scrubbed":0,"log_size":3963,"log_dups_size":0,"ondisk_log_size":3963,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":314114048,"num_objects":4793,"num_object_clones":0,"num_object_copies":14379,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4793,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7463,"num_write_kb":392192,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"74'79","reported_seq":224,"reported_epoch":438,"state":"active+clean","last_fresh":"2024-09-18T08:20:16.492378+0000","last_change":"2024-09-18T08:15:43.275398+0000","last_active":"2024-09-18T08:20:16.492378+0000","last_peered":"2024-09-18T08:20:16.492378+0000","last_clean":"2024-09-18T08:20:16.492378+0000","last_became_active":"2024-09-18T08:15:41.452895+0000","last_became_peered":"2024-09-18T08:15:41.452895+0000","last_unstale":"2024-09-18T08:20:16.492378+0000","last_undegraded":"2024-09-18T08:20:16.492378+0000","last_fullsized":"2024-09-18T08:20:16.492378+0000","mapping_epoch":336,"log_start":"0'0","ondisk_log_start":"0'0","created":152,"last_epoch_clean":337,"parent":"0.0","parent_split_bits":4,"last_scrub":"74'79","last_scrub_stamp":"2024-09-18T08:15:43.275329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:00:11.893742+0000","last_clean_scrub_stamp":"2024-09-18T08:15:43.275329+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,2],"acting":[5,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.4","version":"415'43","reported_seq":901,"reported_epoch":438,"state":"active+clean","last_fresh":"2024-09-18T08:20:18.140716+0000","last_change":"2024-09-18T08:19:05.931238+0000","last_active":"2024-09-18T08:20:18.140716+0000","last_peered":"2024-09-18T08:20:18.140716+0000","last_clean":"2024-09-18T08:20:18.140716+0000","last_became_active":"2024-09-18T08:19:05.911342+0000","last_became_peered":"2024-09-18T08:19:05.911342+0000","last_unstale":"2024-09-18T08:20:18.140716+0000","last_undegraded":"2024-09-18T08:20:18.140716+0000","last_fullsized":"2024-09-18T08:20:18.140716+0000","mapping_epoch":413,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":414,"parent":"0.0","parent_split_bits":0,"last_scrub":"310'42","last_scrub_stamp":"2024-09-18T08:18:27.690465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_clean_scrub_stamp":"2024-09-18T08:18:27.690465+0000","objects_scrubbed":10,"log_size":43,"log_dups_size":0,"ondisk_log_size":43,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":65,"num_read_kb":50,"num_write":38,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":358,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,6],"acting":[1,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":404,"reported_epoch":438,"state":"active+clean","last_fresh":"2024-09-18T08:20:16.501870+0000","last_change":"2024-09-18T08:18:16.839327+0000","last_active":"2024-09-18T08:20:16.501870+0000","last_peered":"2024-09-18T08:20:16.501870+0000","last_clean":"2024-09-18T08:20:16.501870+0000","last_became_active":"2024-09-18T08:16:13.475879+0000","last_became_peered":"2024-09-18T08:16:13.475879+0000","last_unstale":"2024-09-18T08:20:16.501870+0000","last_undegraded":"2024-09-18T08:20:16.501870+0000","last_fullsized":"2024-09-18T08:20:16.501870+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:18:16.839282+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:46.860092+0000","last_clean_scrub_stamp":"2024-09-18T08:18:16.839282+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,1],"acting":[0,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"64'2","reported_seq":236,"reported_epoch":438,"state":"active+clean","last_fresh":"2024-09-18T08:20:16.492609+0000","last_change":"2024-09-18T08:19:00.785048+0000","last_active":"2024-09-18T08:20:16.492609+0000","last_peered":"2024-09-18T08:20:16.492609+0000","last_clean":"2024-09-18T08:20:16.492609+0000","last_became_active":"2024-09-18T08:15:41.453351+0000","last_became_peered":"2024-09-18T08:15:41.453351+0000","last_unstale":"2024-09-18T08:20:16.492609+0000","last_undegraded":"2024-09-18T08:20:16.492609+0000","last_fullsized":"2024-09-18T08:20:16.492609+0000","mapping_epoch":336,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":337,"parent":"0.0","parent_split_bits":0,"last_scrub":"64'2","last_scrub_stamp":"2024-09-18T08:19:00.784983+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:45.855979+0000","last_clean_scrub_stamp":"2024-09-18T08:19:00.784983+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":827,"reported_epoch":438,"state":"active+clean","last_fresh":"2024-09-18T08:20:27.117188+0000","last_change":"2024-09-18T08:20:27.117188+0000","last_active":"2024-09-18T08:20:27.117188+0000","last_peered":"2024-09-18T08:20:27.117188+0000","last_clean":"2024-09-18T08:20:27.117188+0000","last_became_active":"2024-09-18T08:16:13.481456+0000","last_became_peered":"2024-09-18T08:16:13.481456+0000","last_unstale":"2024-09-18T08:20:27.117188+0000","last_undegraded":"2024-09-18T08:20:27.117188+0000","last_fullsized":"2024-09-18T08:20:27.117188+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:20:27.117124+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:50.886830+0000","last_clean_scrub_stamp":"2024-09-18T08:20:27.117124+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:21:46.449723+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,6],"acting":[0,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":836,"reported_epoch":438,"state":"active+clean","last_fresh":"2024-09-18T08:20:16.473874+0000","last_change":"2024-09-18T08:18:27.657904+0000","last_active":"2024-09-18T08:20:16.473874+0000","last_peered":"2024-09-18T08:20:16.473874+0000","last_clean":"2024-09-18T08:20:16.473874+0000","last_became_active":"2024-09-18T08:18:27.657591+0000","last_became_peered":"2024-09-18T08:18:27.657591+0000","last_unstale":"2024-09-18T08:20:16.473874+0000","last_undegraded":"2024-09-18T08:20:16.473874+0000","last_fullsized":"2024-09-18T08:20:16.473874+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:15:59.948801+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_clean_scrub_stamp":"2024-09-18T08:15:59.948801+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.8","version":"438'7586","reported_seq":7672,"reported_epoch":438,"state":"active+clean","last_fresh":"2024-09-18T08:20:24.812112+0000","last_change":"2024-09-18T08:18:27.641711+0000","last_active":"2024-09-18T08:20:24.812112+0000","last_peered":"2024-09-18T08:20:24.812112+0000","last_clean":"2024-09-18T08:20:24.812112+0000","last_became_active":"2024-09-18T08:18:27.641439+0000","last_became_peered":"2024-09-18T08:18:27.641439+0000","last_unstale":"2024-09-18T08:20:24.812112+0000","last_undegraded":"2024-09-18T08:20:24.812112+0000","last_fullsized":"2024-09-18T08:20:24.812112+0000","mapping_epoch":401,"log_start":"417'3600","ondisk_log_start":"417'3600","created":401,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:18:26.612016+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:18:26.612016+0000","last_clean_scrub_stamp":"2024-09-18T08:18:26.612016+0000","objects_scrubbed":0,"log_size":3986,"log_dups_size":0,"ondisk_log_size":3986,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":327155712,"num_objects":4992,"num_object_clones":0,"num_object_copies":14976,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4992,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7586,"num_write_kb":402496,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"74'79","reported_seq":755,"reported_epoch":438,"state":"active+clean","last_fresh":"2024-09-18T08:20:16.500278+0000","last_change":"2024-09-18T08:19:05.896315+0000","last_active":"2024-09-18T08:20:16.500278+0000","last_peered":"2024-09-18T08:20:16.500278+0000","last_clean":"2024-09-18T08:20:16.500278+0000","last_became_active":"2024-09-18T08:19:05.895876+0000","last_became_peered":"2024-09-18T08:19:05.895876+0000","last_unstale":"2024-09-18T08:20:16.500278+0000","last_undegraded":"2024-09-18T08:20:16.500278+0000","last_fullsized":"2024-09-18T08:20:16.500278+0000","mapping_epoch":413,"log_start":"0'0","ondisk_log_start":"0'0","created":152,"last_epoch_clean":414,"parent":"0.0","parent_split_bits":4,"last_scrub":"74'79","last_scrub_stamp":"2024-09-18T08:18:17.808269+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:00:11.893742+0000","last_clean_scrub_stamp":"2024-09-18T08:18:17.808269+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,5],"acting":[0,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.7","version":"271'17","reported_seq":813,"reported_epoch":438,"state":"active+clean","last_fresh":"2024-09-18T08:20:16.472784+0000","last_change":"2024-09-18T08:20:11.911309+0000","last_active":"2024-09-18T08:20:16.472784+0000","last_peered":"2024-09-18T08:20:16.472784+0000","last_clean":"2024-09-18T08:20:16.472784+0000","last_became_active":"2024-09-18T08:15:41.456482+0000","last_became_peered":"2024-09-18T08:15:41.456482+0000","last_unstale":"2024-09-18T08:20:16.472784+0000","last_undegraded":"2024-09-18T08:20:16.472784+0000","last_fullsized":"2024-09-18T08:20:16.472784+0000","mapping_epoch":336,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":337,"parent":"0.0","parent_split_bits":0,"last_scrub":"271'17","last_scrub_stamp":"2024-09-18T08:20:11.911227+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_clean_scrub_stamp":"2024-09-18T08:20:11.911227+0000","objects_scrubbed":5,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:21:41.454984+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":35,"num_read_kb":24,"num_write":20,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":990,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,0],"acting":[2,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":697,"reported_epoch":438,"state":"active+clean","last_fresh":"2024-09-18T08:20:16.477290+0000","last_change":"2024-09-18T08:19:05.905706+0000","last_active":"2024-09-18T08:20:16.477290+0000","last_peered":"2024-09-18T08:20:16.477290+0000","last_clean":"2024-09-18T08:20:16.477290+0000","last_became_active":"2024-09-18T08:19:05.905390+0000","last_became_peered":"2024-09-18T08:19:05.905390+0000","last_unstale":"2024-09-18T08:20:16.477290+0000","last_undegraded":"2024-09-18T08:20:16.477290+0000","last_fullsized":"2024-09-18T08:20:16.477290+0000","mapping_epoch":413,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":414,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:15:49.244156+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:46.860092+0000","last_clean_scrub_stamp":"2024-09-18T08:15:49.244156+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,0],"acting":[1,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"62'1","reported_seq":78,"reported_epoch":438,"state":"active+clean","last_fresh":"2024-09-18T08:20:16.472509+0000","last_change":"2024-09-18T08:19:05.923105+0000","last_active":"2024-09-18T08:20:16.472509+0000","last_peered":"2024-09-18T08:20:16.472509+0000","last_clean":"2024-09-18T08:20:16.472509+0000","last_became_active":"2024-09-18T08:19:05.908141+0000","last_became_peered":"2024-09-18T08:19:05.908141+0000","last_unstale":"2024-09-18T08:20:16.472509+0000","last_undegraded":"2024-09-18T08:20:16.472509+0000","last_fullsized":"2024-09-18T08:20:16.472509+0000","mapping_epoch":413,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":414,"parent":"0.0","parent_split_bits":0,"last_scrub":"62'1","last_scrub_stamp":"2024-09-18T08:17:34.935618+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:45.855979+0000","last_clean_scrub_stamp":"2024-09-18T08:17:34.935618+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":516,"reported_epoch":438,"state":"active+clean","last_fresh":"2024-09-18T08:20:16.493416+0000","last_change":"2024-09-18T08:19:05.886456+0000","last_active":"2024-09-18T08:20:16.493416+0000","last_peered":"2024-09-18T08:20:16.493416+0000","last_clean":"2024-09-18T08:20:16.493416+0000","last_became_active":"2024-09-18T08:19:05.886174+0000","last_became_peered":"2024-09-18T08:19:05.886174+0000","last_unstale":"2024-09-18T08:20:16.493416+0000","last_undegraded":"2024-09-18T08:20:16.493416+0000","last_fullsized":"2024-09-18T08:20:16.493416+0000","mapping_epoch":413,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":414,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:18:28.112437+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:50.886830+0000","last_clean_scrub_stamp":"2024-09-18T08:18:28.112437+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":500,"reported_epoch":438,"state":"active+clean","last_fresh":"2024-09-18T08:20:16.473048+0000","last_change":"2024-09-18T08:18:27.662433+0000","last_active":"2024-09-18T08:20:16.473048+0000","last_peered":"2024-09-18T08:20:16.473048+0000","last_clean":"2024-09-18T08:20:16.473048+0000","last_became_active":"2024-09-18T08:18:27.662116+0000","last_became_peered":"2024-09-18T08:18:27.662116+0000","last_unstale":"2024-09-18T08:20:16.473048+0000","last_undegraded":"2024-09-18T08:20:16.473048+0000","last_fullsized":"2024-09-18T08:20:16.473048+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:17:35.381401+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_clean_scrub_stamp":"2024-09-18T08:17:35.381401+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":23,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.9","version":"438'7438","reported_seq":7524,"reported_epoch":438,"state":"active+clean","last_fresh":"2024-09-18T08:20:24.817564+0000","last_change":"2024-09-18T08:18:27.655199+0000","last_active":"2024-09-18T08:20:24.817564+0000","last_peered":"2024-09-18T08:20:24.817564+0000","last_clean":"2024-09-18T08:20:24.817564+0000","last_became_active":"2024-09-18T08:18:27.654866+0000","last_became_peered":"2024-09-18T08:18:27.654866+0000","last_unstale":"2024-09-18T08:20:24.817564+0000","last_undegraded":"2024-09-18T08:20:24.817564+0000","last_fullsized":"2024-09-18T08:20:24.817564+0000","mapping_epoch":401,"log_start":"417'3400","ondisk_log_start":"417'3400","created":401,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:18:26.612016+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:18:26.612016+0000","last_clean_scrub_stamp":"2024-09-18T08:18:26.612016+0000","objects_scrubbed":0,"log_size":4038,"log_dups_size":0,"ondisk_log_size":4038,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":316801024,"num_objects":4834,"num_object_clones":0,"num_object_copies":14502,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4834,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7438,"num_write_kb":392704,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,0],"acting":[3,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"74'79","reported_seq":473,"reported_epoch":438,"state":"active+clean","last_fresh":"2024-09-18T08:20:16.477270+0000","last_change":"2024-09-18T08:18:55.603760+0000","last_active":"2024-09-18T08:20:16.477270+0000","last_peered":"2024-09-18T08:20:16.477270+0000","last_clean":"2024-09-18T08:20:16.477270+0000","last_became_active":"2024-09-18T08:16:13.454138+0000","last_became_peered":"2024-09-18T08:16:13.454138+0000","last_unstale":"2024-09-18T08:20:16.477270+0000","last_undegraded":"2024-09-18T08:20:16.477270+0000","last_fullsized":"2024-09-18T08:20:16.477270+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":152,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":3,"last_scrub":"74'79","last_scrub_stamp":"2024-09-18T08:18:55.603698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:00:11.893742+0000","last_clean_scrub_stamp":"2024-09-18T08:18:55.603698+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,5],"acting":[1,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.6","version":"271'24","reported_seq":924,"reported_epoch":438,"state":"active+clean","last_fresh":"2024-09-18T08:20:16.501952+0000","last_change":"2024-09-18T08:19:05.919065+0000","last_active":"2024-09-18T08:20:16.501952+0000","last_peered":"2024-09-18T08:20:16.501952+0000","last_clean":"2024-09-18T08:20:16.501952+0000","last_became_active":"2024-09-18T08:19:05.900310+0000","last_became_peered":"2024-09-18T08:19:05.900310+0000","last_unstale":"2024-09-18T08:20:16.501952+0000","last_undegraded":"2024-09-18T08:20:16.501952+0000","last_fullsized":"2024-09-18T08:20:16.501952+0000","mapping_epoch":413,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":414,"parent":"0.0","parent_split_bits":0,"last_scrub":"271'24","last_scrub_stamp":"2024-09-18T08:15:10.187593+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_clean_scrub_stamp":"2024-09-18T08:15:10.187593+0000","objects_scrubbed":6,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":36,"num_read_kb":30,"num_write":24,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":840,"reported_epoch":438,"state":"active+clean","last_fresh":"2024-09-18T08:20:22.166128+0000","last_change":"2024-09-18T08:20:22.166128+0000","last_active":"2024-09-18T08:20:22.166128+0000","last_peered":"2024-09-18T08:20:22.166128+0000","last_clean":"2024-09-18T08:20:22.166128+0000","last_became_active":"2024-09-18T08:16:13.460391+0000","last_became_peered":"2024-09-18T08:16:13.460391+0000","last_unstale":"2024-09-18T08:20:22.166128+0000","last_undegraded":"2024-09-18T08:20:22.166128+0000","last_fullsized":"2024-09-18T08:20:22.166128+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:20:22.164255+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:46.860092+0000","last_clean_scrub_stamp":"2024-09-18T08:20:22.164255+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:21:47.591456+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,5],"acting":[0,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"70'5","reported_seq":1119,"reported_epoch":438,"state":"active+clean","last_fresh":"2024-09-18T08:20:26.643509+0000","last_change":"2024-09-18T08:18:27.672092+0000","last_active":"2024-09-18T08:20:26.643509+0000","last_peered":"2024-09-18T08:20:26.643509+0000","last_clean":"2024-09-18T08:20:26.643509+0000","last_became_active":"2024-09-18T08:18:27.646445+0000","last_became_peered":"2024-09-18T08:18:27.646445+0000","last_unstale":"2024-09-18T08:20:26.643509+0000","last_undegraded":"2024-09-18T08:20:26.643509+0000","last_fullsized":"2024-09-18T08:20:26.643509+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":0,"last_scrub":"70'5","last_scrub_stamp":"2024-09-18T08:15:57.384224+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:45.855979+0000","last_clean_scrub_stamp":"2024-09-18T08:15:57.384224+0000","objects_scrubbed":1,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0.000504055,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":517,"num_read_kb":512,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":778,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":224,"reported_epoch":438,"state":"active+clean","last_fresh":"2024-09-18T08:20:19.796231+0000","last_change":"2024-09-18T08:20:19.796231+0000","last_active":"2024-09-18T08:20:19.796231+0000","last_peered":"2024-09-18T08:20:19.796231+0000","last_clean":"2024-09-18T08:20:19.796231+0000","last_became_active":"2024-09-18T08:15:41.443080+0000","last_became_peered":"2024-09-18T08:15:41.443080+0000","last_unstale":"2024-09-18T08:20:19.796231+0000","last_undegraded":"2024-09-18T08:20:19.796231+0000","last_fullsized":"2024-09-18T08:20:19.796231+0000","mapping_epoch":336,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":337,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:20:19.796047+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:50.886830+0000","last_clean_scrub_stamp":"2024-09-18T08:20:19.796047+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:21:22.041310+0000","scrub_duration":24,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":620,"reported_epoch":438,"state":"active+clean","last_fresh":"2024-09-18T08:20:16.477747+0000","last_change":"2024-09-18T08:18:16.556819+0000","last_active":"2024-09-18T08:20:16.477747+0000","last_peered":"2024-09-18T08:20:16.477747+0000","last_clean":"2024-09-18T08:20:16.477747+0000","last_became_active":"2024-09-18T08:15:41.436692+0000","last_became_peered":"2024-09-18T08:15:41.436692+0000","last_unstale":"2024-09-18T08:20:16.477747+0000","last_undegraded":"2024-09-18T08:20:16.477747+0000","last_fullsized":"2024-09-18T08:20:16.477747+0000","mapping_epoch":336,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":337,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:18:16.556771+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_clean_scrub_stamp":"2024-09-18T08:18:16.556771+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.a","version":"438'7674","reported_seq":7760,"reported_epoch":438,"state":"active+clean","last_fresh":"2024-09-18T08:20:24.788069+0000","last_change":"2024-09-18T08:18:27.646142+0000","last_active":"2024-09-18T08:20:24.788069+0000","last_peered":"2024-09-18T08:20:24.788069+0000","last_clean":"2024-09-18T08:20:24.788069+0000","last_became_active":"2024-09-18T08:18:27.645863+0000","last_became_peered":"2024-09-18T08:18:27.645863+0000","last_unstale":"2024-09-18T08:20:24.788069+0000","last_undegraded":"2024-09-18T08:20:24.788069+0000","last_fullsized":"2024-09-18T08:20:24.788069+0000","mapping_epoch":401,"log_start":"418'3700","ondisk_log_start":"418'3700","created":401,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:18:26.612016+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:18:26.612016+0000","last_clean_scrub_stamp":"2024-09-18T08:18:26.612016+0000","objects_scrubbed":0,"log_size":3974,"log_dups_size":0,"ondisk_log_size":3974,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":319684608,"num_objects":4878,"num_object_clones":0,"num_object_copies":14634,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4878,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7674,"num_write_kb":401664,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.5","version":"402'36","reported_seq":135,"reported_epoch":438,"state":"active+clean","last_fresh":"2024-09-18T08:20:23.141399+0000","last_change":"2024-09-18T08:18:30.788913+0000","last_active":"2024-09-18T08:20:23.141399+0000","last_peered":"2024-09-18T08:20:23.141399+0000","last_clean":"2024-09-18T08:20:23.141399+0000","last_became_active":"2024-09-18T08:18:27.657630+0000","last_became_peered":"2024-09-18T08:18:27.657630+0000","last_unstale":"2024-09-18T08:20:23.141399+0000","last_undegraded":"2024-09-18T08:20:23.141399+0000","last_fullsized":"2024-09-18T08:20:23.141399+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":0,"last_scrub":"402'36","last_scrub_stamp":"2024-09-18T08:18:30.788799+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_clean_scrub_stamp":"2024-09-18T08:18:30.788799+0000","objects_scrubbed":8,"log_size":36,"log_dups_size":0,"ondisk_log_size":36,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":42,"num_read_kb":32,"num_write":25,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":462,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,0],"acting":[7,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":832,"reported_epoch":438,"state":"active+clean","last_fresh":"2024-09-18T08:20:16.472950+0000","last_change":"2024-09-18T08:20:11.908841+0000","last_active":"2024-09-18T08:20:16.472950+0000","last_peered":"2024-09-18T08:20:16.472950+0000","last_clean":"2024-09-18T08:20:16.472950+0000","last_became_active":"2024-09-18T08:15:41.464313+0000","last_became_peered":"2024-09-18T08:15:41.464313+0000","last_unstale":"2024-09-18T08:20:16.472950+0000","last_undegraded":"2024-09-18T08:20:16.472950+0000","last_fullsized":"2024-09-18T08:20:16.472950+0000","mapping_epoch":336,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":337,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:20:11.908773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:46.860092+0000","last_clean_scrub_stamp":"2024-09-18T08:20:11.908773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:21:21.053391+0000","scrub_duration":68,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"74'79","reported_seq":429,"reported_epoch":438,"state":"active+clean","last_fresh":"2024-09-18T08:20:16.500417+0000","last_change":"2024-09-18T08:18:27.650334+0000","last_active":"2024-09-18T08:20:16.500417+0000","last_peered":"2024-09-18T08:20:16.500417+0000","last_clean":"2024-09-18T08:20:16.500417+0000","last_became_active":"2024-09-18T08:18:27.649964+0000","last_became_peered":"2024-09-18T08:18:27.649964+0000","last_unstale":"2024-09-18T08:20:16.500417+0000","last_undegraded":"2024-09-18T08:20:16.500417+0000","last_fullsized":"2024-09-18T08:20:16.500417+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'79","last_scrub_stamp":"2024-09-18T08:18:18.786155+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:00:11.893742+0000","last_clean_scrub_stamp":"2024-09-18T08:18:18.786155+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,6],"acting":[0,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":683,"reported_epoch":438,"state":"active+clean","last_fresh":"2024-09-18T08:20:16.473071+0000","last_change":"2024-09-18T08:19:05.096869+0000","last_active":"2024-09-18T08:20:16.473071+0000","last_peered":"2024-09-18T08:20:16.473071+0000","last_clean":"2024-09-18T08:20:16.473071+0000","last_became_active":"2024-09-18T08:18:27.653602+0000","last_became_peered":"2024-09-18T08:18:27.653602+0000","last_unstale":"2024-09-18T08:20:16.473071+0000","last_undegraded":"2024-09-18T08:20:16.473071+0000","last_fullsized":"2024-09-18T08:20:16.473071+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:19:05.096805+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:50.886830+0000","last_clean_scrub_stamp":"2024-09-18T08:19:05.096805+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:20:34.654958+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":231,"reported_epoch":438,"state":"active+clean","last_fresh":"2024-09-18T08:20:16.493736+0000","last_change":"2024-09-18T08:18:25.515806+0000","last_active":"2024-09-18T08:20:16.493736+0000","last_peered":"2024-09-18T08:20:16.493736+0000","last_clean":"2024-09-18T08:20:16.493736+0000","last_became_active":"2024-09-18T08:16:13.471911+0000","last_became_peered":"2024-09-18T08:16:13.471911+0000","last_unstale":"2024-09-18T08:20:16.493736+0000","last_undegraded":"2024-09-18T08:20:16.493736+0000","last_fullsized":"2024-09-18T08:20:16.493736+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:18:25.515757+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_clean_scrub_stamp":"2024-09-18T08:18:25.515757+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,1],"acting":[5,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.1","version":"438'7489","reported_seq":7575,"reported_epoch":438,"state":"active+clean","last_fresh":"2024-09-18T08:20:24.802311+0000","last_change":"2024-09-18T08:18:27.651222+0000","last_active":"2024-09-18T08:20:24.802311+0000","last_peered":"2024-09-18T08:20:24.802311+0000","last_clean":"2024-09-18T08:20:24.802311+0000","last_became_active":"2024-09-18T08:18:27.650844+0000","last_became_peered":"2024-09-18T08:18:27.650844+0000","last_unstale":"2024-09-18T08:20:24.802311+0000","last_undegraded":"2024-09-18T08:20:24.802311+0000","last_fullsized":"2024-09-18T08:20:24.802311+0000","mapping_epoch":401,"log_start":"417'3500","ondisk_log_start":"417'3500","created":401,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:18:26.612016+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:18:26.612016+0000","last_clean_scrub_stamp":"2024-09-18T08:18:26.612016+0000","objects_scrubbed":0,"log_size":3989,"log_dups_size":0,"ondisk_log_size":3989,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":319881216,"num_objects":4881,"num_object_clones":0,"num_object_copies":14643,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4881,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7489,"num_write_kb":395840,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,0],"acting":[3,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.e","version":"271'19","reported_seq":512,"reported_epoch":438,"state":"active+clean","last_fresh":"2024-09-18T08:20:16.478237+0000","last_change":"2024-09-18T08:17:30.690655+0000","last_active":"2024-09-18T08:20:16.478237+0000","last_peered":"2024-09-18T08:20:16.478237+0000","last_clean":"2024-09-18T08:20:16.478237+0000","last_became_active":"2024-09-18T08:16:13.458001+0000","last_became_peered":"2024-09-18T08:16:13.458001+0000","last_unstale":"2024-09-18T08:20:16.478237+0000","last_undegraded":"2024-09-18T08:20:16.478237+0000","last_fullsized":"2024-09-18T08:20:16.478237+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"271'19","last_scrub_stamp":"2024-09-18T08:17:30.690581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_clean_scrub_stamp":"2024-09-18T08:17:30.690581+0000","objects_scrubbed":5,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":31,"num_read_kb":24,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,3],"acting":[1,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":472,"reported_epoch":438,"state":"active+clean","last_fresh":"2024-09-18T08:20:16.475143+0000","last_change":"2024-09-18T08:19:05.881635+0000","last_active":"2024-09-18T08:20:16.475143+0000","last_peered":"2024-09-18T08:20:16.475143+0000","last_clean":"2024-09-18T08:20:16.475143+0000","last_became_active":"2024-09-18T08:19:05.881238+0000","last_became_peered":"2024-09-18T08:19:05.881238+0000","last_unstale":"2024-09-18T08:20:16.475143+0000","last_undegraded":"2024-09-18T08:20:16.475143+0000","last_fullsized":"2024-09-18T08:20:16.475143+0000","mapping_epoch":413,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":414,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:18:26.121022+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:46.860092+0000","last_clean_scrub_stamp":"2024-09-18T08:18:26.121022+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,5],"acting":[7,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":238,"reported_epoch":438,"state":"active+clean","last_fresh":"2024-09-18T08:20:16.493594+0000","last_change":"2024-09-18T08:19:05.897861+0000","last_active":"2024-09-18T08:20:16.493594+0000","last_peered":"2024-09-18T08:20:16.493594+0000","last_clean":"2024-09-18T08:20:16.493594+0000","last_became_active":"2024-09-18T08:19:05.897497+0000","last_became_peered":"2024-09-18T08:19:05.897497+0000","last_unstale":"2024-09-18T08:20:16.493594+0000","last_undegraded":"2024-09-18T08:20:16.493594+0000","last_fullsized":"2024-09-18T08:20:16.493594+0000","mapping_epoch":413,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":414,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:17:38.616405+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:50.886830+0000","last_clean_scrub_stamp":"2024-09-18T08:17:38.616405+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,1],"acting":[5,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":819,"reported_epoch":438,"state":"active+clean","last_fresh":"2024-09-18T08:20:19.015270+0000","last_change":"2024-09-18T08:20:19.015270+0000","last_active":"2024-09-18T08:20:19.015270+0000","last_peered":"2024-09-18T08:20:19.015270+0000","last_clean":"2024-09-18T08:20:19.015270+0000","last_became_active":"2024-09-18T08:15:41.464500+0000","last_became_peered":"2024-09-18T08:15:41.464500+0000","last_unstale":"2024-09-18T08:20:19.015270+0000","last_undegraded":"2024-09-18T08:20:19.015270+0000","last_fullsized":"2024-09-18T08:20:19.015270+0000","mapping_epoch":336,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":337,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:20:19.015198+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_clean_scrub_stamp":"2024-09-18T08:20:19.015198+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:21:34.017027+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.2","version":"438'7486","reported_seq":14117,"reported_epoch":438,"state":"active+clean","last_fresh":"2024-09-18T08:20:24.826680+0000","last_change":"2024-09-18T08:20:06.493831+0000","last_active":"2024-09-18T08:20:24.826680+0000","last_peered":"2024-09-18T08:20:24.826680+0000","last_clean":"2024-09-18T08:20:24.826680+0000","last_became_active":"2024-09-18T08:20:06.489999+0000","last_became_peered":"2024-09-18T08:20:06.489999+0000","last_unstale":"2024-09-18T08:20:24.826680+0000","last_undegraded":"2024-09-18T08:20:24.826680+0000","last_fullsized":"2024-09-18T08:20:24.826680+0000","mapping_epoch":431,"log_start":"417'3500","ondisk_log_start":"417'3500","created":401,"last_epoch_clean":432,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:18:26.612016+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:18:26.612016+0000","last_clean_scrub_stamp":"2024-09-18T08:18:26.612016+0000","objects_scrubbed":0,"log_size":3986,"log_dups_size":0,"ondisk_log_size":3986,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":310771712,"num_objects":4742,"num_object_clones":0,"num_object_copies":14226,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4742,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7486,"num_write_kb":391296,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3267,"num_bytes_recovered":214106112,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,7],"acting":[3,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"74'79","reported_seq":236,"reported_epoch":438,"state":"active+clean","last_fresh":"2024-09-18T08:20:16.492724+0000","last_change":"2024-09-18T08:19:01.768944+0000","last_active":"2024-09-18T08:20:16.492724+0000","last_peered":"2024-09-18T08:20:16.492724+0000","last_clean":"2024-09-18T08:20:16.492724+0000","last_became_active":"2024-09-18T08:15:41.454781+0000","last_became_peered":"2024-09-18T08:15:41.454781+0000","last_unstale":"2024-09-18T08:20:16.492724+0000","last_undegraded":"2024-09-18T08:20:16.492724+0000","last_fullsized":"2024-09-18T08:20:16.492724+0000","mapping_epoch":336,"log_start":"0'0","ondisk_log_start":"0'0","created":152,"last_epoch_clean":337,"parent":"0.0","parent_split_bits":4,"last_scrub":"74'79","last_scrub_stamp":"2024-09-18T08:19:01.768882+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:00:11.893742+0000","last_clean_scrub_stamp":"2024-09-18T08:19:01.768882+0000","objects_scrubbed":1,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:20:18.146783+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262688,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":262688,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,3],"acting":[5,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.d","version":"271'31","reported_seq":394,"reported_epoch":438,"state":"active+clean","last_fresh":"2024-09-18T08:20:20.856895+0000","last_change":"2024-09-18T08:20:20.856895+0000","last_active":"2024-09-18T08:20:20.856895+0000","last_peered":"2024-09-18T08:20:20.856895+0000","last_clean":"2024-09-18T08:20:20.856895+0000","last_became_active":"2024-09-18T08:19:05.905244+0000","last_became_peered":"2024-09-18T08:19:05.905244+0000","last_unstale":"2024-09-18T08:20:20.856895+0000","last_undegraded":"2024-09-18T08:20:20.856895+0000","last_fullsized":"2024-09-18T08:20:20.856895+0000","mapping_epoch":413,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":414,"parent":"0.0","parent_split_bits":0,"last_scrub":"271'31","last_scrub_stamp":"2024-09-18T08:20:20.856684+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_clean_scrub_stamp":"2024-09-18T08:20:20.856684+0000","objects_scrubbed":8,"log_size":31,"log_dups_size":0,"ondisk_log_size":31,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:21:43.677208+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":49,"num_read_kb":39,"num_write":32,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,0],"acting":[1,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":475,"reported_epoch":438,"state":"active+clean","last_fresh":"2024-09-18T08:20:16.477403+0000","last_change":"2024-09-18T08:18:18.567839+0000","last_active":"2024-09-18T08:20:16.477403+0000","last_peered":"2024-09-18T08:20:16.477403+0000","last_clean":"2024-09-18T08:20:16.477403+0000","last_became_active":"2024-09-18T08:16:13.483143+0000","last_became_peered":"2024-09-18T08:16:13.483143+0000","last_unstale":"2024-09-18T08:20:16.477403+0000","last_undegraded":"2024-09-18T08:20:16.477403+0000","last_fullsized":"2024-09-18T08:20:16.477403+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:18:18.567791+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:46.860092+0000","last_clean_scrub_stamp":"2024-09-18T08:18:18.567791+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,3],"acting":[1,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":705,"reported_epoch":438,"state":"active+clean","last_fresh":"2024-09-18T08:20:16.477802+0000","last_change":"2024-09-18T08:18:59.652875+0000","last_active":"2024-09-18T08:20:16.477802+0000","last_peered":"2024-09-18T08:20:16.477802+0000","last_clean":"2024-09-18T08:20:16.477802+0000","last_became_active":"2024-09-18T08:16:13.460650+0000","last_became_peered":"2024-09-18T08:16:13.460650+0000","last_unstale":"2024-09-18T08:20:16.477802+0000","last_undegraded":"2024-09-18T08:20:16.477802+0000","last_fullsized":"2024-09-18T08:20:16.477802+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:18:59.652484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:50.886830+0000","last_clean_scrub_stamp":"2024-09-18T08:18:59.652484+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,6],"acting":[1,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":397,"reported_epoch":438,"state":"active+clean","last_fresh":"2024-09-18T08:20:16.473513+0000","last_change":"2024-09-18T08:19:05.895239+0000","last_active":"2024-09-18T08:20:16.473513+0000","last_peered":"2024-09-18T08:20:16.473513+0000","last_clean":"2024-09-18T08:20:16.473513+0000","last_became_active":"2024-09-18T08:19:05.894863+0000","last_became_peered":"2024-09-18T08:19:05.894863+0000","last_unstale":"2024-09-18T08:20:16.473513+0000","last_undegraded":"2024-09-18T08:20:16.473513+0000","last_fullsized":"2024-09-18T08:20:16.473513+0000","mapping_epoch":413,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":414,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:19:02.172378+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_clean_scrub_stamp":"2024-09-18T08:19:02.172378+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,1],"acting":[3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.3","version":"438'7805","reported_seq":7891,"reported_epoch":438,"state":"active+clean","last_fresh":"2024-09-18T08:20:28.564811+0000","last_change":"2024-09-18T08:18:27.658212+0000","last_active":"2024-09-18T08:20:28.564811+0000","last_peered":"2024-09-18T08:20:28.564811+0000","last_clean":"2024-09-18T08:20:28.564811+0000","last_became_active":"2024-09-18T08:18:27.657815+0000","last_became_peered":"2024-09-18T08:18:27.657815+0000","last_unstale":"2024-09-18T08:20:28.564811+0000","last_undegraded":"2024-09-18T08:20:28.564811+0000","last_fullsized":"2024-09-18T08:20:28.564811+0000","mapping_epoch":401,"log_start":"418'3800","ondisk_log_start":"418'3800","created":401,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:18:26.612016+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:18:26.612016+0000","last_clean_scrub_stamp":"2024-09-18T08:18:26.612016+0000","objects_scrubbed":0,"log_size":4005,"log_dups_size":0,"ondisk_log_size":4005,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":301268992,"num_objects":4597,"num_object_clones":0,"num_object_copies":13791,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4597,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7805,"num_write_kb":396864,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.9","version":"74'79","reported_seq":652,"reported_epoch":438,"state":"active+clean","last_fresh":"2024-09-18T08:20:16.473908+0000","last_change":"2024-09-18T08:19:05.908881+0000","last_active":"2024-09-18T08:20:16.473908+0000","last_peered":"2024-09-18T08:20:16.473908+0000","last_clean":"2024-09-18T08:20:16.473908+0000","last_became_active":"2024-09-18T08:19:05.903470+0000","last_became_peered":"2024-09-18T08:19:05.903470+0000","last_unstale":"2024-09-18T08:20:16.473908+0000","last_undegraded":"2024-09-18T08:20:16.473908+0000","last_fullsized":"2024-09-18T08:20:16.473908+0000","mapping_epoch":413,"log_start":"0'0","ondisk_log_start":"0'0","created":152,"last_epoch_clean":414,"parent":"0.0","parent_split_bits":4,"last_scrub":"74'79","last_scrub_stamp":"2024-09-18T08:17:44.331322+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:00:11.893742+0000","last_clean_scrub_stamp":"2024-09-18T08:17:44.331322+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,5],"acting":[3,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.c","version":"271'20","reported_seq":421,"reported_epoch":438,"state":"active+clean","last_fresh":"2024-09-18T08:20:16.472931+0000","last_change":"2024-09-18T08:16:13.526582+0000","last_active":"2024-09-18T08:20:16.472931+0000","last_peered":"2024-09-18T08:20:16.472931+0000","last_clean":"2024-09-18T08:20:16.472931+0000","last_became_active":"2024-09-18T08:16:13.476918+0000","last_became_peered":"2024-09-18T08:16:13.476918+0000","last_unstale":"2024-09-18T08:20:16.472931+0000","last_undegraded":"2024-09-18T08:20:16.472931+0000","last_fullsized":"2024-09-18T08:20:16.472931+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"271'20","last_scrub_stamp":"2024-09-18T08:15:56.673129+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_clean_scrub_stamp":"2024-09-18T08:15:56.673129+0000","objects_scrubbed":0,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":28,"num_read_kb":23,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,1],"acting":[3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":839,"reported_epoch":438,"state":"active+clean","last_fresh":"2024-09-18T08:20:16.473836+0000","last_change":"2024-09-18T08:19:02.699664+0000","last_active":"2024-09-18T08:20:16.473836+0000","last_peered":"2024-09-18T08:20:16.473836+0000","last_clean":"2024-09-18T08:20:16.473836+0000","last_became_active":"2024-09-18T08:16:13.479148+0000","last_became_peered":"2024-09-18T08:16:13.479148+0000","last_unstale":"2024-09-18T08:20:16.473836+0000","last_undegraded":"2024-09-18T08:20:16.473836+0000","last_fullsized":"2024-09-18T08:20:16.473836+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:19:02.699596+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:46.860092+0000","last_clean_scrub_stamp":"2024-09-18T08:19:02.699596+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.d","version":"402'14","reported_seq":1079,"reported_epoch":438,"state":"active+clean","last_fresh":"2024-09-18T08:20:23.141176+0000","last_change":"2024-09-18T08:18:27.675655+0000","last_active":"2024-09-18T08:20:23.141176+0000","last_peered":"2024-09-18T08:20:23.141176+0000","last_clean":"2024-09-18T08:20:23.141176+0000","last_became_active":"2024-09-18T08:18:27.653796+0000","last_became_peered":"2024-09-18T08:18:27.653796+0000","last_unstale":"2024-09-18T08:20:23.141176+0000","last_undegraded":"2024-09-18T08:20:23.141176+0000","last_fullsized":"2024-09-18T08:20:23.141176+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":0,"last_scrub":"338'13","last_scrub_stamp":"2024-09-18T08:17:38.886654+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:50.886830+0000","last_clean_scrub_stamp":"2024-09-18T08:17:38.886654+0000","objects_scrubbed":1,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:20:49.857493+0000","scrub_duration":26,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":489,"reported_epoch":438,"state":"active+clean","last_fresh":"2024-09-18T08:20:24.831854+0000","last_change":"2024-09-18T08:20:24.831854+0000","last_active":"2024-09-18T08:20:24.831854+0000","last_peered":"2024-09-18T08:20:24.831854+0000","last_clean":"2024-09-18T08:20:24.831854+0000","last_became_active":"2024-09-18T08:19:05.886455+0000","last_became_peered":"2024-09-18T08:19:05.886455+0000","last_unstale":"2024-09-18T08:20:24.831854+0000","last_undegraded":"2024-09-18T08:20:24.831854+0000","last_fullsized":"2024-09-18T08:20:24.831854+0000","mapping_epoch":413,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":414,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:20:24.829925+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_clean_scrub_stamp":"2024-09-18T08:20:24.829925+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:21:30.882607+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.4","version":"438'7790","reported_seq":14335,"reported_epoch":438,"state":"active+clean","last_fresh":"2024-09-18T08:20:28.570631+0000","last_change":"2024-09-18T08:20:16.486645+0000","last_active":"2024-09-18T08:20:28.570631+0000","last_peered":"2024-09-18T08:20:28.570631+0000","last_clean":"2024-09-18T08:20:28.570631+0000","last_became_active":"2024-09-18T08:20:16.478529+0000","last_became_peered":"2024-09-18T08:20:16.478529+0000","last_unstale":"2024-09-18T08:20:28.570631+0000","last_undegraded":"2024-09-18T08:20:28.570631+0000","last_fullsized":"2024-09-18T08:20:28.570631+0000","mapping_epoch":437,"log_start":"418'3800","ondisk_log_start":"418'3800","created":401,"last_epoch_clean":438,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:18:26.612016+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:18:26.612016+0000","last_clean_scrub_stamp":"2024-09-18T08:18:26.612016+0000","objects_scrubbed":0,"log_size":3990,"log_dups_size":0,"ondisk_log_size":3990,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":289669120,"num_objects":4420,"num_object_clones":0,"num_object_copies":13260,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4420,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7790,"num_write_kb":390720,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3224,"num_bytes_recovered":211288064,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.b","version":"271'15","reported_seq":894,"reported_epoch":438,"state":"active+clean","last_fresh":"2024-09-18T08:20:16.501453+0000","last_change":"2024-09-18T08:17:48.720750+0000","last_active":"2024-09-18T08:20:16.501453+0000","last_peered":"2024-09-18T08:20:16.501453+0000","last_clean":"2024-09-18T08:20:16.501453+0000","last_became_active":"2024-09-18T08:16:13.480279+0000","last_became_peered":"2024-09-18T08:16:13.480279+0000","last_unstale":"2024-09-18T08:20:16.501453+0000","last_undegraded":"2024-09-18T08:20:16.501453+0000","last_fullsized":"2024-09-18T08:20:16.501453+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"271'15","last_scrub_stamp":"2024-09-18T08:17:48.720672+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_clean_scrub_stamp":"2024-09-18T08:17:48.720672+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,3],"acting":[0,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":328,"reported_epoch":438,"state":"active+clean","last_fresh":"2024-09-18T08:20:16.474061+0000","last_change":"2024-09-18T08:18:22.633544+0000","last_active":"2024-09-18T08:20:16.474061+0000","last_peered":"2024-09-18T08:20:16.474061+0000","last_clean":"2024-09-18T08:20:16.474061+0000","last_became_active":"2024-09-18T08:18:17.892088+0000","last_became_peered":"2024-09-18T08:18:17.892088+0000","last_unstale":"2024-09-18T08:20:16.474061+0000","last_undegraded":"2024-09-18T08:20:16.474061+0000","last_fullsized":"2024-09-18T08:20:16.474061+0000","mapping_epoch":395,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":396,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:18:22.633492+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:46.860092+0000","last_clean_scrub_stamp":"2024-09-18T08:18:22.633492+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,6],"acting":[3,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":680,"reported_epoch":438,"state":"active+clean","last_fresh":"2024-09-18T08:20:16.472578+0000","last_change":"2024-09-18T08:19:05.909652+0000","last_active":"2024-09-18T08:20:16.472578+0000","last_peered":"2024-09-18T08:20:16.472578+0000","last_clean":"2024-09-18T08:20:16.472578+0000","last_became_active":"2024-09-18T08:19:05.907780+0000","last_became_peered":"2024-09-18T08:19:05.907780+0000","last_unstale":"2024-09-18T08:20:16.472578+0000","last_undegraded":"2024-09-18T08:20:16.472578+0000","last_fullsized":"2024-09-18T08:20:16.472578+0000","mapping_epoch":413,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":414,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:18:19.057302+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:50.886830+0000","last_clean_scrub_stamp":"2024-09-18T08:18:19.057302+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":843,"reported_epoch":438,"state":"active+clean","last_fresh":"2024-09-18T08:20:16.501681+0000","last_change":"2024-09-18T08:19:05.912963+0000","last_active":"2024-09-18T08:20:16.501681+0000","last_peered":"2024-09-18T08:20:16.501681+0000","last_clean":"2024-09-18T08:20:16.501681+0000","last_became_active":"2024-09-18T08:19:05.912544+0000","last_became_peered":"2024-09-18T08:19:05.912544+0000","last_unstale":"2024-09-18T08:20:16.501681+0000","last_undegraded":"2024-09-18T08:20:16.501681+0000","last_fullsized":"2024-09-18T08:20:16.501681+0000","mapping_epoch":413,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":414,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:18:28.901610+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_clean_scrub_stamp":"2024-09-18T08:18:28.901610+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.5","version":"438'7741","reported_seq":7827,"reported_epoch":438,"state":"active+clean","last_fresh":"2024-09-18T08:20:28.726177+0000","last_change":"2024-09-18T08:18:27.643971+0000","last_active":"2024-09-18T08:20:28.726177+0000","last_peered":"2024-09-18T08:20:28.726177+0000","last_clean":"2024-09-18T08:20:28.726177+0000","last_became_active":"2024-09-18T08:18:27.642478+0000","last_became_peered":"2024-09-18T08:18:27.642478+0000","last_unstale":"2024-09-18T08:20:28.726177+0000","last_undegraded":"2024-09-18T08:20:28.726177+0000","last_fullsized":"2024-09-18T08:20:28.726177+0000","mapping_epoch":401,"log_start":"417'3500","ondisk_log_start":"417'3500","created":401,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:18:26.612016+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:18:26.612016+0000","last_clean_scrub_stamp":"2024-09-18T08:18:26.612016+0000","objects_scrubbed":0,"log_size":4241,"log_dups_size":0,"ondisk_log_size":4241,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":289865728,"num_objects":4423,"num_object_clones":0,"num_object_copies":13269,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4423,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7741,"num_write_kb":389248,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.a","version":"271'35","reported_seq":762,"reported_epoch":438,"state":"active+clean","last_fresh":"2024-09-18T08:20:16.477591+0000","last_change":"2024-09-18T08:18:27.683070+0000","last_active":"2024-09-18T08:20:16.477591+0000","last_peered":"2024-09-18T08:20:16.477591+0000","last_clean":"2024-09-18T08:20:16.477591+0000","last_became_active":"2024-09-18T08:18:27.652693+0000","last_became_peered":"2024-09-18T08:18:27.652693+0000","last_unstale":"2024-09-18T08:20:16.477591+0000","last_undegraded":"2024-09-18T08:20:16.477591+0000","last_fullsized":"2024-09-18T08:20:16.477591+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":0,"last_scrub":"271'35","last_scrub_stamp":"2024-09-18T08:18:23.687385+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_clean_scrub_stamp":"2024-09-18T08:18:23.687385+0000","objects_scrubbed":9,"log_size":35,"log_dups_size":0,"ondisk_log_size":35,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":55,"num_read_kb":44,"num_write":36,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":53,"num_bytes_recovered":550,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,7],"acting":[1,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":231,"reported_epoch":438,"state":"active+clean","last_fresh":"2024-09-18T08:20:16.493647+0000","last_change":"2024-09-18T08:20:15.848074+0000","last_active":"2024-09-18T08:20:16.493647+0000","last_peered":"2024-09-18T08:20:16.493647+0000","last_clean":"2024-09-18T08:20:16.493647+0000","last_became_active":"2024-09-18T08:18:27.643663+0000","last_became_peered":"2024-09-18T08:18:27.643663+0000","last_unstale":"2024-09-18T08:20:16.493647+0000","last_undegraded":"2024-09-18T08:20:16.493647+0000","last_fullsized":"2024-09-18T08:20:16.493647+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:20:15.848022+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:46.860092+0000","last_clean_scrub_stamp":"2024-09-18T08:20:15.848022+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:21:17.902304+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":502,"reported_epoch":438,"state":"active+clean","last_fresh":"2024-09-18T08:20:16.473051+0000","last_change":"2024-09-18T08:19:05.905542+0000","last_active":"2024-09-18T08:20:16.473051+0000","last_peered":"2024-09-18T08:20:16.473051+0000","last_clean":"2024-09-18T08:20:16.473051+0000","last_became_active":"2024-09-18T08:19:05.903320+0000","last_became_peered":"2024-09-18T08:19:05.903320+0000","last_unstale":"2024-09-18T08:20:16.473051+0000","last_undegraded":"2024-09-18T08:20:16.473051+0000","last_fullsized":"2024-09-18T08:20:16.473051+0000","mapping_epoch":413,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":414,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:18:22.116686+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:50.886830+0000","last_clean_scrub_stamp":"2024-09-18T08:18:22.116686+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,7],"acting":[3,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":639,"reported_epoch":438,"state":"active+clean","last_fresh":"2024-09-18T08:20:16.472599+0000","last_change":"2024-09-18T08:19:05.890818+0000","last_active":"2024-09-18T08:20:16.472599+0000","last_peered":"2024-09-18T08:20:16.472599+0000","last_clean":"2024-09-18T08:20:16.472599+0000","last_became_active":"2024-09-18T08:19:05.890070+0000","last_became_peered":"2024-09-18T08:19:05.890070+0000","last_unstale":"2024-09-18T08:20:16.472599+0000","last_undegraded":"2024-09-18T08:20:16.472599+0000","last_fullsized":"2024-09-18T08:20:16.472599+0000","mapping_epoch":413,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":414,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:18:15.051582+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_clean_scrub_stamp":"2024-09-18T08:18:15.051582+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.6","version":"438'7727","reported_seq":14125,"reported_epoch":438,"state":"active+clean","last_fresh":"2024-09-18T08:20:28.210301+0000","last_change":"2024-09-18T08:20:05.552082+0000","last_active":"2024-09-18T08:20:28.210301+0000","last_peered":"2024-09-18T08:20:28.210301+0000","last_clean":"2024-09-18T08:20:28.210301+0000","last_became_active":"2024-09-18T08:20:05.509531+0000","last_became_peered":"2024-09-18T08:20:05.509531+0000","last_unstale":"2024-09-18T08:20:28.210301+0000","last_undegraded":"2024-09-18T08:20:28.210301+0000","last_fullsized":"2024-09-18T08:20:28.210301+0000","mapping_epoch":430,"log_start":"419'3900","ondisk_log_start":"419'3900","created":401,"last_epoch_clean":431,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:18:26.612016+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:18:26.612016+0000","last_clean_scrub_stamp":"2024-09-18T08:18:26.612016+0000","objects_scrubbed":0,"log_size":3827,"log_dups_size":0,"ondisk_log_size":3827,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":300351488,"num_objects":4583,"num_object_clones":0,"num_object_copies":13749,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4583,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7727,"num_write_kb":393920,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3150,"num_bytes_recovered":206438400,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,5],"acting":[0,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.9","version":"271'18","reported_seq":534,"reported_epoch":438,"state":"active+clean","last_fresh":"2024-09-18T08:20:16.477167+0000","last_change":"2024-09-18T08:16:13.525130+0000","last_active":"2024-09-18T08:20:16.477167+0000","last_peered":"2024-09-18T08:20:16.477167+0000","last_clean":"2024-09-18T08:20:16.477167+0000","last_became_active":"2024-09-18T08:16:13.482580+0000","last_became_peered":"2024-09-18T08:16:13.482580+0000","last_unstale":"2024-09-18T08:20:16.477167+0000","last_undegraded":"2024-09-18T08:20:16.477167+0000","last_fullsized":"2024-09-18T08:20:16.477167+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"271'18","last_scrub_stamp":"2024-09-18T08:15:49.666776+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_clean_scrub_stamp":"2024-09-18T08:15:49.666776+0000","objects_scrubbed":5,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":33,"num_read_kb":24,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":880,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,6],"acting":[1,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":92,"reported_epoch":438,"state":"active+clean","last_fresh":"2024-09-18T08:20:16.476841+0000","last_change":"2024-09-18T08:18:56.624230+0000","last_active":"2024-09-18T08:20:16.476841+0000","last_peered":"2024-09-18T08:20:16.476841+0000","last_clean":"2024-09-18T08:20:16.476841+0000","last_became_active":"2024-09-18T08:18:27.654488+0000","last_became_peered":"2024-09-18T08:18:27.654488+0000","last_unstale":"2024-09-18T08:20:16.476841+0000","last_undegraded":"2024-09-18T08:20:16.476841+0000","last_fullsized":"2024-09-18T08:20:16.476841+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:18:56.624152+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:46.860092+0000","last_clean_scrub_stamp":"2024-09-18T08:18:56.624152+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":823,"reported_epoch":438,"state":"active+clean","last_fresh":"2024-09-18T08:20:16.473591+0000","last_change":"2024-09-18T08:19:01.228154+0000","last_active":"2024-09-18T08:20:16.473591+0000","last_peered":"2024-09-18T08:20:16.473591+0000","last_clean":"2024-09-18T08:20:16.473591+0000","last_became_active":"2024-09-18T08:14:39.399301+0000","last_became_peered":"2024-09-18T08:14:39.399301+0000","last_unstale":"2024-09-18T08:20:16.473591+0000","last_undegraded":"2024-09-18T08:20:16.473591+0000","last_fullsized":"2024-09-18T08:20:16.473591+0000","mapping_epoch":307,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":308,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:19:01.228071+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:50.886830+0000","last_clean_scrub_stamp":"2024-09-18T08:19:01.228071+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:20:30.855655+0000","scrub_duration":20,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":823,"reported_epoch":438,"state":"active+clean","last_fresh":"2024-09-18T08:20:16.473463+0000","last_change":"2024-09-18T08:18:27.665000+0000","last_active":"2024-09-18T08:20:16.473463+0000","last_peered":"2024-09-18T08:20:16.473463+0000","last_clean":"2024-09-18T08:20:16.473463+0000","last_became_active":"2024-09-18T08:18:27.664412+0000","last_became_peered":"2024-09-18T08:18:27.664412+0000","last_unstale":"2024-09-18T08:20:16.473463+0000","last_undegraded":"2024-09-18T08:20:16.473463+0000","last_fullsized":"2024-09-18T08:20:16.473463+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:17:09.487823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_clean_scrub_stamp":"2024-09-18T08:17:09.487823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.7","version":"438'7866","reported_seq":7951,"reported_epoch":438,"state":"active+clean","last_fresh":"2024-09-18T08:20:28.751823+0000","last_change":"2024-09-18T08:18:27.651740+0000","last_active":"2024-09-18T08:20:28.751823+0000","last_peered":"2024-09-18T08:20:28.751823+0000","last_clean":"2024-09-18T08:20:28.751823+0000","last_became_active":"2024-09-18T08:18:27.651411+0000","last_became_peered":"2024-09-18T08:18:27.651411+0000","last_unstale":"2024-09-18T08:20:28.751823+0000","last_undegraded":"2024-09-18T08:20:28.751823+0000","last_fullsized":"2024-09-18T08:20:28.751823+0000","mapping_epoch":401,"log_start":"418'3600","ondisk_log_start":"418'3600","created":401,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:18:26.612016+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:18:26.612016+0000","last_clean_scrub_stamp":"2024-09-18T08:18:26.612016+0000","objects_scrubbed":0,"log_size":4266,"log_dups_size":0,"ondisk_log_size":4266,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":301203456,"num_objects":4596,"num_object_clones":0,"num_object_copies":13788,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4596,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7866,"num_write_kb":398784,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.8","version":"271'27","reported_seq":286,"reported_epoch":438,"state":"active+clean","last_fresh":"2024-09-18T08:20:16.825021+0000","last_change":"2024-09-18T08:20:16.825021+0000","last_active":"2024-09-18T08:20:16.825021+0000","last_peered":"2024-09-18T08:20:16.825021+0000","last_clean":"2024-09-18T08:20:16.825021+0000","last_became_active":"2024-09-18T08:18:27.643053+0000","last_became_peered":"2024-09-18T08:18:27.643053+0000","last_unstale":"2024-09-18T08:20:16.825021+0000","last_undegraded":"2024-09-18T08:20:16.825021+0000","last_fullsized":"2024-09-18T08:20:16.825021+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":0,"last_scrub":"271'27","last_scrub_stamp":"2024-09-18T08:20:16.824939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_clean_scrub_stamp":"2024-09-18T08:20:16.824939+0000","objects_scrubbed":7,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:21:45.559934+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":44,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":35,"num_bytes_recovered":550,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.f","version":"62'2","reported_seq":90,"reported_epoch":438,"state":"active+clean","last_fresh":"2024-09-18T08:20:16.476777+0000","last_change":"2024-09-18T08:18:27.672015+0000","last_active":"2024-09-18T08:20:16.476777+0000","last_peered":"2024-09-18T08:20:16.476777+0000","last_clean":"2024-09-18T08:20:16.476777+0000","last_became_active":"2024-09-18T08:18:27.654687+0000","last_became_peered":"2024-09-18T08:18:27.654687+0000","last_unstale":"2024-09-18T08:20:16.476777+0000","last_undegraded":"2024-09-18T08:20:16.476777+0000","last_fullsized":"2024-09-18T08:20:16.476777+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":0,"last_scrub":"62'2","last_scrub_stamp":"2024-09-18T08:17:49.718261+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:46.860092+0000","last_clean_scrub_stamp":"2024-09-18T08:17:49.718261+0000","objects_scrubbed":2,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":184,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,1],"acting":[7,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.9","version":"402'18","reported_seq":121,"reported_epoch":438,"state":"active+clean","last_fresh":"2024-09-18T08:20:24.535710+0000","last_change":"2024-09-18T08:20:24.535623+0000","last_active":"2024-09-18T08:20:24.535710+0000","last_peered":"2024-09-18T08:20:24.535710+0000","last_clean":"2024-09-18T08:20:24.535710+0000","last_became_active":"2024-09-18T08:18:27.652480+0000","last_became_peered":"2024-09-18T08:18:27.652480+0000","last_unstale":"2024-09-18T08:20:24.535710+0000","last_undegraded":"2024-09-18T08:20:24.535710+0000","last_fullsized":"2024-09-18T08:20:24.535710+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":0,"last_scrub":"402'18","last_scrub_stamp":"2024-09-18T08:20:24.535554+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:50.886830+0000","last_clean_scrub_stamp":"2024-09-18T08:20:24.535554+0000","objects_scrubbed":1,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:21:34.761382+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":224,"reported_epoch":438,"state":"active+clean","last_fresh":"2024-09-18T08:20:16.492596+0000","last_change":"2024-09-18T08:19:04.680209+0000","last_active":"2024-09-18T08:20:16.492596+0000","last_peered":"2024-09-18T08:20:16.492596+0000","last_clean":"2024-09-18T08:20:16.492596+0000","last_became_active":"2024-09-18T08:15:41.449889+0000","last_became_peered":"2024-09-18T08:15:41.449889+0000","last_unstale":"2024-09-18T08:20:16.492596+0000","last_undegraded":"2024-09-18T08:20:16.492596+0000","last_fullsized":"2024-09-18T08:20:16.492596+0000","mapping_epoch":336,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":337,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:19:04.680153+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_clean_scrub_stamp":"2024-09-18T08:19:04.680153+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:20:21.463313+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":742,"reported_epoch":438,"state":"active+clean","last_fresh":"2024-09-18T08:20:16.500241+0000","last_change":"2024-09-18T08:19:03.092160+0000","last_active":"2024-09-18T08:20:16.500241+0000","last_peered":"2024-09-18T08:20:16.500241+0000","last_clean":"2024-09-18T08:20:16.500241+0000","last_became_active":"2024-09-18T08:15:41.446544+0000","last_became_peered":"2024-09-18T08:15:41.446544+0000","last_unstale":"2024-09-18T08:20:16.500241+0000","last_undegraded":"2024-09-18T08:20:16.500241+0000","last_fullsized":"2024-09-18T08:20:16.500241+0000","mapping_epoch":336,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":337,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:19:03.092077+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:46.860092+0000","last_clean_scrub_stamp":"2024-09-18T08:19:03.092077+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:20:20.600568+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.17","version":"271'12","reported_seq":873,"reported_epoch":438,"state":"active+clean","last_fresh":"2024-09-18T08:20:16.473714+0000","last_change":"2024-09-18T08:18:27.699070+0000","last_active":"2024-09-18T08:20:16.473714+0000","last_peered":"2024-09-18T08:20:16.473714+0000","last_clean":"2024-09-18T08:20:16.473714+0000","last_became_active":"2024-09-18T08:18:27.663338+0000","last_became_peered":"2024-09-18T08:18:27.663338+0000","last_unstale":"2024-09-18T08:20:16.473714+0000","last_undegraded":"2024-09-18T08:20:16.473714+0000","last_fullsized":"2024-09-18T08:20:16.473714+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":0,"last_scrub":"271'12","last_scrub_stamp":"2024-09-18T08:14:33.700931+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_clean_scrub_stamp":"2024-09-18T08:14:33.700931+0000","objects_scrubbed":3,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:21:07.875738+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":18,"num_read_kb":15,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":223,"reported_epoch":438,"state":"active+clean","last_fresh":"2024-09-18T08:20:16.493676+0000","last_change":"2024-09-18T08:18:58.734315+0000","last_active":"2024-09-18T08:20:16.493676+0000","last_peered":"2024-09-18T08:20:16.493676+0000","last_clean":"2024-09-18T08:20:16.493676+0000","last_became_active":"2024-09-18T08:15:41.445332+0000","last_became_peered":"2024-09-18T08:15:41.445332+0000","last_unstale":"2024-09-18T08:20:16.493676+0000","last_undegraded":"2024-09-18T08:20:16.493676+0000","last_fullsized":"2024-09-18T08:20:16.493676+0000","mapping_epoch":336,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":337,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:18:58.734241+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:50.886830+0000","last_clean_scrub_stamp":"2024-09-18T08:18:58.734241+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:20:21.012782+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,1],"acting":[5,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":92,"reported_epoch":438,"state":"active+clean","last_fresh":"2024-09-18T08:20:16.476256+0000","last_change":"2024-09-18T08:18:29.808538+0000","last_active":"2024-09-18T08:20:16.476256+0000","last_peered":"2024-09-18T08:20:16.476256+0000","last_clean":"2024-09-18T08:20:16.476256+0000","last_became_active":"2024-09-18T08:18:27.653368+0000","last_became_peered":"2024-09-18T08:18:27.653368+0000","last_unstale":"2024-09-18T08:20:16.476256+0000","last_undegraded":"2024-09-18T08:20:16.476256+0000","last_fullsized":"2024-09-18T08:20:16.476256+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:18:29.808476+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_clean_scrub_stamp":"2024-09-18T08:18:29.808476+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,5],"acting":[7,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"271'15","reported_seq":865,"reported_epoch":438,"state":"active+clean","last_fresh":"2024-09-18T08:20:16.501666+0000","last_change":"2024-09-18T08:18:59.080889+0000","last_active":"2024-09-18T08:20:16.501666+0000","last_peered":"2024-09-18T08:20:16.501666+0000","last_clean":"2024-09-18T08:20:16.501666+0000","last_became_active":"2024-09-18T08:18:27.658637+0000","last_became_peered":"2024-09-18T08:18:27.658637+0000","last_unstale":"2024-09-18T08:20:16.501666+0000","last_undegraded":"2024-09-18T08:20:16.501666+0000","last_fullsized":"2024-09-18T08:20:16.501666+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":0,"last_scrub":"271'15","last_scrub_stamp":"2024-09-18T08:18:59.080655+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_clean_scrub_stamp":"2024-09-18T08:18:59.080655+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":456,"reported_epoch":438,"state":"active+clean","last_fresh":"2024-09-18T08:20:16.501102+0000","last_change":"2024-09-18T08:18:27.650329+0000","last_active":"2024-09-18T08:20:16.501102+0000","last_peered":"2024-09-18T08:20:16.501102+0000","last_clean":"2024-09-18T08:20:16.501102+0000","last_became_active":"2024-09-18T08:18:27.649876+0000","last_became_peered":"2024-09-18T08:18:27.649876+0000","last_unstale":"2024-09-18T08:20:16.501102+0000","last_undegraded":"2024-09-18T08:20:16.501102+0000","last_fullsized":"2024-09-18T08:20:16.501102+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:17:50.771903+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:46.860092+0000","last_clean_scrub_stamp":"2024-09-18T08:17:50.771903+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,6],"acting":[0,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":818,"reported_epoch":438,"state":"active+clean","last_fresh":"2024-09-18T08:20:16.473588+0000","last_change":"2024-09-18T08:18:27.660413+0000","last_active":"2024-09-18T08:20:16.473588+0000","last_peered":"2024-09-18T08:20:16.473588+0000","last_clean":"2024-09-18T08:20:16.473588+0000","last_became_active":"2024-09-18T08:18:27.660093+0000","last_became_peered":"2024-09-18T08:18:27.660093+0000","last_unstale":"2024-09-18T08:20:16.473588+0000","last_undegraded":"2024-09-18T08:20:16.473588+0000","last_fullsized":"2024-09-18T08:20:16.473588+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:17:06.569828+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:50.886830+0000","last_clean_scrub_stamp":"2024-09-18T08:17:06.569828+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":31,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":805,"reported_epoch":438,"state":"active+clean","last_fresh":"2024-09-18T08:20:16.476749+0000","last_change":"2024-09-18T08:19:05.895365+0000","last_active":"2024-09-18T08:20:16.476749+0000","last_peered":"2024-09-18T08:20:16.476749+0000","last_clean":"2024-09-18T08:20:16.476749+0000","last_became_active":"2024-09-18T08:19:05.894972+0000","last_became_peered":"2024-09-18T08:19:05.894972+0000","last_unstale":"2024-09-18T08:20:16.476749+0000","last_undegraded":"2024-09-18T08:20:16.476749+0000","last_fullsized":"2024-09-18T08:20:16.476749+0000","mapping_epoch":413,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":414,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:18:31.070711+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_clean_scrub_stamp":"2024-09-18T08:18:31.070711+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":22,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,5],"acting":[7,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.15","version":"271'15","reported_seq":250,"reported_epoch":438,"state":"active+clean","last_fresh":"2024-09-18T08:20:16.493982+0000","last_change":"2024-09-18T08:18:27.674004+0000","last_active":"2024-09-18T08:20:16.493982+0000","last_peered":"2024-09-18T08:20:16.493982+0000","last_clean":"2024-09-18T08:20:16.493982+0000","last_became_active":"2024-09-18T08:18:27.646211+0000","last_became_peered":"2024-09-18T08:18:27.646211+0000","last_unstale":"2024-09-18T08:20:16.493982+0000","last_undegraded":"2024-09-18T08:20:16.493982+0000","last_fullsized":"2024-09-18T08:20:16.493982+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":0,"last_scrub":"271'15","last_scrub_stamp":"2024-09-18T08:17:32.638811+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_clean_scrub_stamp":"2024-09-18T08:17:32.638811+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":20,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":25,"num_read_kb":19,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":837,"reported_epoch":438,"state":"active+clean","last_fresh":"2024-09-18T08:20:16.500601+0000","last_change":"2024-09-18T08:18:27.656075+0000","last_active":"2024-09-18T08:20:16.500601+0000","last_peered":"2024-09-18T08:20:16.500601+0000","last_clean":"2024-09-18T08:20:16.500601+0000","last_became_active":"2024-09-18T08:18:27.655758+0000","last_became_peered":"2024-09-18T08:18:27.655758+0000","last_unstale":"2024-09-18T08:20:16.500601+0000","last_undegraded":"2024-09-18T08:20:16.500601+0000","last_fullsized":"2024-09-18T08:20:16.500601+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:18:24.804845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:46.860092+0000","last_clean_scrub_stamp":"2024-09-18T08:18:24.804845+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.14","version":"415'18","reported_seq":1096,"reported_epoch":438,"state":"active+clean","last_fresh":"2024-09-18T08:20:18.139407+0000","last_change":"2024-09-18T08:19:05.916229+0000","last_active":"2024-09-18T08:20:18.139407+0000","last_peered":"2024-09-18T08:20:18.139407+0000","last_clean":"2024-09-18T08:20:18.139407+0000","last_became_active":"2024-09-18T08:19:05.900155+0000","last_became_peered":"2024-09-18T08:19:05.900155+0000","last_unstale":"2024-09-18T08:20:18.139407+0000","last_undegraded":"2024-09-18T08:20:18.139407+0000","last_fullsized":"2024-09-18T08:20:18.139407+0000","mapping_epoch":413,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":414,"parent":"0.0","parent_split_bits":0,"last_scrub":"310'16","last_scrub_stamp":"2024-09-18T08:17:39.424098+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:50.886830+0000","last_clean_scrub_stamp":"2024-09-18T08:17:39.424098+0000","objects_scrubbed":1,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":16,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":545,"reported_epoch":438,"state":"active+clean","last_fresh":"2024-09-18T08:20:16.472797+0000","last_change":"2024-09-18T08:18:24.109825+0000","last_active":"2024-09-18T08:20:16.472797+0000","last_peered":"2024-09-18T08:20:16.472797+0000","last_clean":"2024-09-18T08:20:16.472797+0000","last_became_active":"2024-09-18T08:16:13.467534+0000","last_became_peered":"2024-09-18T08:16:13.467534+0000","last_unstale":"2024-09-18T08:20:16.472797+0000","last_undegraded":"2024-09-18T08:20:16.472797+0000","last_fullsized":"2024-09-18T08:20:16.472797+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:18:24.109778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_clean_scrub_stamp":"2024-09-18T08:18:24.109778+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:20:34.025882+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,0],"acting":[2,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.14","version":"271'20","reported_seq":891,"reported_epoch":438,"state":"active+clean","last_fresh":"2024-09-18T08:20:16.473081+0000","last_change":"2024-09-18T08:18:27.701608+0000","last_active":"2024-09-18T08:20:16.473081+0000","last_peered":"2024-09-18T08:20:16.473081+0000","last_clean":"2024-09-18T08:20:16.473081+0000","last_became_active":"2024-09-18T08:18:27.663877+0000","last_became_peered":"2024-09-18T08:18:27.663877+0000","last_unstale":"2024-09-18T08:20:16.473081+0000","last_undegraded":"2024-09-18T08:20:16.473081+0000","last_fullsized":"2024-09-18T08:20:16.473081+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":0,"last_scrub":"271'20","last_scrub_stamp":"2024-09-18T08:15:52.819801+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_clean_scrub_stamp":"2024-09-18T08:15:52.819801+0000","objects_scrubbed":5,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":16,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":25,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,7],"acting":[3,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":91,"reported_epoch":438,"state":"active+clean","last_fresh":"2024-09-18T08:20:16.476501+0000","last_change":"2024-09-18T08:19:05.900476+0000","last_active":"2024-09-18T08:20:16.476501+0000","last_peered":"2024-09-18T08:20:16.476501+0000","last_clean":"2024-09-18T08:20:16.476501+0000","last_became_active":"2024-09-18T08:19:05.900189+0000","last_became_peered":"2024-09-18T08:19:05.900189+0000","last_unstale":"2024-09-18T08:20:16.476501+0000","last_undegraded":"2024-09-18T08:20:16.476501+0000","last_fullsized":"2024-09-18T08:20:16.476501+0000","mapping_epoch":413,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":414,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:17:30.908835+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:46.860092+0000","last_clean_scrub_stamp":"2024-09-18T08:17:30.908835+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":21,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,0],"acting":[7,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.15","version":"338'12","reported_seq":285,"reported_epoch":438,"state":"active+clean","last_fresh":"2024-09-18T08:20:23.141158+0000","last_change":"2024-09-18T08:18:24.511983+0000","last_active":"2024-09-18T08:20:23.141158+0000","last_peered":"2024-09-18T08:20:23.141158+0000","last_clean":"2024-09-18T08:20:23.141158+0000","last_became_active":"2024-09-18T08:15:41.455243+0000","last_became_peered":"2024-09-18T08:15:41.455243+0000","last_unstale":"2024-09-18T08:20:23.141158+0000","last_undegraded":"2024-09-18T08:20:23.141158+0000","last_fullsized":"2024-09-18T08:20:23.141158+0000","mapping_epoch":336,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":337,"parent":"0.0","parent_split_bits":0,"last_scrub":"338'12","last_scrub_stamp":"2024-09-18T08:18:24.511895+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:50.886830+0000","last_clean_scrub_stamp":"2024-09-18T08:18:24.511895+0000","objects_scrubbed":1,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":820,"reported_epoch":438,"state":"active+clean","last_fresh":"2024-09-18T08:20:16.500949+0000","last_change":"2024-09-18T08:19:00.044481+0000","last_active":"2024-09-18T08:20:16.500949+0000","last_peered":"2024-09-18T08:20:16.500949+0000","last_clean":"2024-09-18T08:20:16.500949+0000","last_became_active":"2024-09-18T08:18:27.658009+0000","last_became_peered":"2024-09-18T08:18:27.658009+0000","last_unstale":"2024-09-18T08:20:16.500949+0000","last_undegraded":"2024-09-18T08:20:16.500949+0000","last_fullsized":"2024-09-18T08:20:16.500949+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:19:00.044416+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_clean_scrub_stamp":"2024-09-18T08:19:00.044416+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.13","version":"271'19","reported_seq":512,"reported_epoch":438,"state":"active+clean","last_fresh":"2024-09-18T08:20:16.477100+0000","last_change":"2024-09-18T08:17:39.665796+0000","last_active":"2024-09-18T08:20:16.477100+0000","last_peered":"2024-09-18T08:20:16.477100+0000","last_clean":"2024-09-18T08:20:16.477100+0000","last_became_active":"2024-09-18T08:16:13.454640+0000","last_became_peered":"2024-09-18T08:16:13.454640+0000","last_unstale":"2024-09-18T08:20:16.477100+0000","last_undegraded":"2024-09-18T08:20:16.477100+0000","last_fullsized":"2024-09-18T08:20:16.477100+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"271'19","last_scrub_stamp":"2024-09-18T08:17:39.665588+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_clean_scrub_stamp":"2024-09-18T08:17:39.665588+0000","objects_scrubbed":5,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":31,"num_read_kb":24,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,5],"acting":[1,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":84,"reported_epoch":438,"state":"active+clean","last_fresh":"2024-09-18T08:20:16.477305+0000","last_change":"2024-09-18T08:18:27.658805+0000","last_active":"2024-09-18T08:20:16.477305+0000","last_peered":"2024-09-18T08:20:16.477305+0000","last_clean":"2024-09-18T08:20:16.477305+0000","last_became_active":"2024-09-18T08:18:27.658476+0000","last_became_peered":"2024-09-18T08:18:27.658476+0000","last_unstale":"2024-09-18T08:20:16.477305+0000","last_undegraded":"2024-09-18T08:20:16.477305+0000","last_fullsized":"2024-09-18T08:20:16.477305+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:18:25.097909+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:46.860092+0000","last_clean_scrub_stamp":"2024-09-18T08:18:25.097909+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":689,"reported_epoch":438,"state":"active+clean","last_fresh":"2024-09-18T08:20:16.477151+0000","last_change":"2024-09-18T08:15:55.335233+0000","last_active":"2024-09-18T08:20:16.477151+0000","last_peered":"2024-09-18T08:20:16.477151+0000","last_clean":"2024-09-18T08:20:16.477151+0000","last_became_active":"2024-09-18T08:15:41.438048+0000","last_became_peered":"2024-09-18T08:15:41.438048+0000","last_unstale":"2024-09-18T08:20:16.477151+0000","last_undegraded":"2024-09-18T08:20:16.477151+0000","last_fullsized":"2024-09-18T08:20:16.477151+0000","mapping_epoch":336,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":337,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:15:55.334983+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:50.886830+0000","last_clean_scrub_stamp":"2024-09-18T08:15:55.334983+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":822,"reported_epoch":438,"state":"active+clean","last_fresh":"2024-09-18T08:20:16.474063+0000","last_change":"2024-09-18T08:17:37.394735+0000","last_active":"2024-09-18T08:20:16.474063+0000","last_peered":"2024-09-18T08:20:16.474063+0000","last_clean":"2024-09-18T08:20:16.474063+0000","last_became_active":"2024-09-18T08:15:41.464961+0000","last_became_peered":"2024-09-18T08:15:41.464961+0000","last_unstale":"2024-09-18T08:20:16.474063+0000","last_undegraded":"2024-09-18T08:20:16.474063+0000","last_fullsized":"2024-09-18T08:20:16.474063+0000","mapping_epoch":336,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":337,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:17:37.394532+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_clean_scrub_stamp":"2024-09-18T08:17:37.394532+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.12","version":"271'15","reported_seq":702,"reported_epoch":438,"state":"active+clean","last_fresh":"2024-09-18T08:20:16.477492+0000","last_change":"2024-09-18T08:19:05.918020+0000","last_active":"2024-09-18T08:20:16.477492+0000","last_peered":"2024-09-18T08:20:16.477492+0000","last_clean":"2024-09-18T08:20:16.477492+0000","last_became_active":"2024-09-18T08:19:05.904293+0000","last_became_peered":"2024-09-18T08:19:05.904293+0000","last_unstale":"2024-09-18T08:20:16.477492+0000","last_undegraded":"2024-09-18T08:20:16.477492+0000","last_fullsized":"2024-09-18T08:20:16.477492+0000","mapping_epoch":413,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":414,"parent":"0.0","parent_split_bits":0,"last_scrub":"271'15","last_scrub_stamp":"2024-09-18T08:18:31.593238+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_clean_scrub_stamp":"2024-09-18T08:18:31.593238+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":25,"num_read_kb":19,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,0],"acting":[1,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":615,"reported_epoch":438,"state":"active+clean","last_fresh":"2024-09-18T08:20:16.473262+0000","last_change":"2024-09-18T08:18:27.663193+0000","last_active":"2024-09-18T08:20:16.473262+0000","last_peered":"2024-09-18T08:20:16.473262+0000","last_clean":"2024-09-18T08:20:16.473262+0000","last_became_active":"2024-09-18T08:18:27.662885+0000","last_became_peered":"2024-09-18T08:18:27.662885+0000","last_unstale":"2024-09-18T08:20:16.473262+0000","last_undegraded":"2024-09-18T08:20:16.473262+0000","last_fullsized":"2024-09-18T08:20:16.473262+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:15:58.978411+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:46.860092+0000","last_clean_scrub_stamp":"2024-09-18T08:15:58.978411+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":18,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":806,"reported_epoch":438,"state":"active+clean","last_fresh":"2024-09-18T08:20:16.473556+0000","last_change":"2024-09-18T08:18:28.695902+0000","last_active":"2024-09-18T08:20:16.473556+0000","last_peered":"2024-09-18T08:20:16.473556+0000","last_clean":"2024-09-18T08:20:16.473556+0000","last_became_active":"2024-09-18T08:14:39.402406+0000","last_became_peered":"2024-09-18T08:14:39.402406+0000","last_unstale":"2024-09-18T08:20:16.473556+0000","last_undegraded":"2024-09-18T08:20:16.473556+0000","last_fullsized":"2024-09-18T08:20:16.473556+0000","mapping_epoch":307,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":308,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:18:28.695842+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:50.886830+0000","last_clean_scrub_stamp":"2024-09-18T08:18:28.695842+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.10","version":"69'1","reported_seq":812,"reported_epoch":438,"state":"active+clean","last_fresh":"2024-09-18T08:20:16.499843+0000","last_change":"2024-09-18T08:15:41.480106+0000","last_active":"2024-09-18T08:20:16.499843+0000","last_peered":"2024-09-18T08:20:16.499843+0000","last_clean":"2024-09-18T08:20:16.499843+0000","last_became_active":"2024-09-18T08:15:41.444085+0000","last_became_peered":"2024-09-18T08:15:41.444085+0000","last_unstale":"2024-09-18T08:20:16.499843+0000","last_undegraded":"2024-09-18T08:20:16.499843+0000","last_fullsized":"2024-09-18T08:20:16.499843+0000","mapping_epoch":336,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":337,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'1","last_scrub_stamp":"2024-09-18T08:15:00.219678+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_clean_scrub_stamp":"2024-09-18T08:15:00.219678+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":26,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.11","version":"271'19","reported_seq":968,"reported_epoch":438,"state":"active+clean","last_fresh":"2024-09-18T08:20:16.472718+0000","last_change":"2024-09-18T08:20:10.875530+0000","last_active":"2024-09-18T08:20:16.472718+0000","last_peered":"2024-09-18T08:20:16.472718+0000","last_clean":"2024-09-18T08:20:16.472718+0000","last_became_active":"2024-09-18T08:18:27.659293+0000","last_became_peered":"2024-09-18T08:18:27.659293+0000","last_unstale":"2024-09-18T08:20:16.472718+0000","last_undegraded":"2024-09-18T08:20:16.472718+0000","last_fullsized":"2024-09-18T08:20:16.472718+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":0,"last_scrub":"271'19","last_scrub_stamp":"2024-09-18T08:20:10.875426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_clean_scrub_stamp":"2024-09-18T08:20:10.875426+0000","objects_scrubbed":5,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:21:10.982420+0000","scrub_duration":22,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":32,"num_read_kb":25,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":30,"num_bytes_recovered":660,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":230,"reported_epoch":438,"state":"active+clean","last_fresh":"2024-09-18T08:20:16.493595+0000","last_change":"2024-09-18T08:18:27.644325+0000","last_active":"2024-09-18T08:20:16.493595+0000","last_peered":"2024-09-18T08:20:16.493595+0000","last_clean":"2024-09-18T08:20:16.493595+0000","last_became_active":"2024-09-18T08:18:27.643957+0000","last_became_peered":"2024-09-18T08:18:27.643957+0000","last_unstale":"2024-09-18T08:20:16.493595+0000","last_undegraded":"2024-09-18T08:20:16.493595+0000","last_fullsized":"2024-09-18T08:20:16.493595+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:18:18.611302+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:46.860092+0000","last_clean_scrub_stamp":"2024-09-18T08:18:18.611302+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":84,"reported_epoch":438,"state":"active+clean","last_fresh":"2024-09-18T08:20:16.475252+0000","last_change":"2024-09-18T08:18:27.652157+0000","last_active":"2024-09-18T08:20:16.475252+0000","last_peered":"2024-09-18T08:20:16.475252+0000","last_clean":"2024-09-18T08:20:16.475252+0000","last_became_active":"2024-09-18T08:18:27.651810+0000","last_became_peered":"2024-09-18T08:18:27.651810+0000","last_unstale":"2024-09-18T08:20:16.475252+0000","last_undegraded":"2024-09-18T08:20:16.475252+0000","last_fullsized":"2024-09-18T08:20:16.475252+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:14:37.743856+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:50.886830+0000","last_clean_scrub_stamp":"2024-09-18T08:14:37.743856+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,0],"acting":[7,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":805,"reported_epoch":438,"state":"active+clean","last_fresh":"2024-09-18T08:20:16.473754+0000","last_change":"2024-09-18T08:18:21.660255+0000","last_active":"2024-09-18T08:20:16.473754+0000","last_peered":"2024-09-18T08:20:16.473754+0000","last_clean":"2024-09-18T08:20:16.473754+0000","last_became_active":"2024-09-18T08:06:24.128582+0000","last_became_peered":"2024-09-18T08:06:24.128582+0000","last_unstale":"2024-09-18T08:20:16.473754+0000","last_undegraded":"2024-09-18T08:20:16.473754+0000","last_fullsized":"2024-09-18T08:20:16.473754+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:18:21.660213+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_clean_scrub_stamp":"2024-09-18T08:18:21.660213+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.10","version":"271'8","reported_seq":855,"reported_epoch":438,"state":"active+clean","last_fresh":"2024-09-18T08:20:16.472409+0000","last_change":"2024-09-18T08:15:50.790606+0000","last_active":"2024-09-18T08:20:16.472409+0000","last_peered":"2024-09-18T08:20:16.472409+0000","last_clean":"2024-09-18T08:20:16.472409+0000","last_became_active":"2024-09-18T08:14:39.404257+0000","last_became_peered":"2024-09-18T08:14:39.404257+0000","last_unstale":"2024-09-18T08:20:16.472409+0000","last_undegraded":"2024-09-18T08:20:16.472409+0000","last_fullsized":"2024-09-18T08:20:16.472409+0000","mapping_epoch":307,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":308,"parent":"0.0","parent_split_bits":0,"last_scrub":"271'8","last_scrub_stamp":"2024-09-18T08:15:50.790525+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_clean_scrub_stamp":"2024-09-18T08:15:50.790525+0000","objects_scrubbed":2,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":12,"num_read_kb":10,"num_write":8,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":832,"reported_epoch":438,"state":"active+clean","last_fresh":"2024-09-18T08:20:16.501581+0000","last_change":"2024-09-18T08:18:25.799322+0000","last_active":"2024-09-18T08:20:16.501581+0000","last_peered":"2024-09-18T08:20:16.501581+0000","last_clean":"2024-09-18T08:20:16.501581+0000","last_became_active":"2024-09-18T08:15:41.466964+0000","last_became_peered":"2024-09-18T08:15:41.466964+0000","last_unstale":"2024-09-18T08:20:16.501581+0000","last_undegraded":"2024-09-18T08:20:16.501581+0000","last_fullsized":"2024-09-18T08:20:16.501581+0000","mapping_epoch":336,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":337,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:18:25.799264+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:46.860092+0000","last_clean_scrub_stamp":"2024-09-18T08:18:25.799264+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":456,"reported_epoch":438,"state":"active+clean","last_fresh":"2024-09-18T08:20:16.500901+0000","last_change":"2024-09-18T08:18:27.645673+0000","last_active":"2024-09-18T08:20:16.500901+0000","last_peered":"2024-09-18T08:20:16.500901+0000","last_clean":"2024-09-18T08:20:16.500901+0000","last_became_active":"2024-09-18T08:18:27.645122+0000","last_became_peered":"2024-09-18T08:18:27.645122+0000","last_unstale":"2024-09-18T08:20:16.500901+0000","last_undegraded":"2024-09-18T08:20:16.500901+0000","last_fullsized":"2024-09-18T08:20:16.500901+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:18:14.912464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:50.886830+0000","last_clean_scrub_stamp":"2024-09-18T08:18:14.912464+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,7],"acting":[0,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":598,"reported_epoch":438,"state":"active+clean","last_fresh":"2024-09-18T08:20:16.472424+0000","last_change":"2024-09-18T08:19:05.890924+0000","last_active":"2024-09-18T08:20:16.472424+0000","last_peered":"2024-09-18T08:20:16.472424+0000","last_clean":"2024-09-18T08:20:16.472424+0000","last_became_active":"2024-09-18T08:19:05.890580+0000","last_became_peered":"2024-09-18T08:19:05.890580+0000","last_unstale":"2024-09-18T08:20:16.472424+0000","last_undegraded":"2024-09-18T08:20:16.472424+0000","last_fullsized":"2024-09-18T08:20:16.472424+0000","mapping_epoch":413,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":414,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:19:03.125974+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_clean_scrub_stamp":"2024-09-18T08:19:03.125974+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:20:23.420957+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":661,"reported_epoch":438,"state":"active+clean","last_fresh":"2024-09-18T08:20:16.477869+0000","last_change":"2024-09-18T08:16:13.468068+0000","last_active":"2024-09-18T08:20:16.477869+0000","last_peered":"2024-09-18T08:20:16.477869+0000","last_clean":"2024-09-18T08:20:16.477869+0000","last_became_active":"2024-09-18T08:16:13.466812+0000","last_became_peered":"2024-09-18T08:16:13.466812+0000","last_unstale":"2024-09-18T08:20:16.477869+0000","last_undegraded":"2024-09-18T08:20:16.477869+0000","last_fullsized":"2024-09-18T08:20:16.477869+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:15:45.297240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_clean_scrub_stamp":"2024-09-18T08:15:45.297240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,2],"acting":[1,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":806,"reported_epoch":438,"state":"active+clean","last_fresh":"2024-09-18T08:20:16.472689+0000","last_change":"2024-09-18T08:15:43.836627+0000","last_active":"2024-09-18T08:20:16.472689+0000","last_peered":"2024-09-18T08:20:16.472689+0000","last_clean":"2024-09-18T08:20:16.472689+0000","last_became_active":"2024-09-18T08:14:39.405347+0000","last_became_peered":"2024-09-18T08:14:39.405347+0000","last_unstale":"2024-09-18T08:20:16.472689+0000","last_undegraded":"2024-09-18T08:20:16.472689+0000","last_fullsized":"2024-09-18T08:20:16.472689+0000","mapping_epoch":307,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":308,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:15:43.836549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:46.860092+0000","last_clean_scrub_stamp":"2024-09-18T08:15:43.836549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":20,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"271'19","reported_seq":236,"reported_epoch":438,"state":"active+clean","last_fresh":"2024-09-18T08:20:16.493542+0000","last_change":"2024-09-18T08:18:20.608272+0000","last_active":"2024-09-18T08:20:16.493542+0000","last_peered":"2024-09-18T08:20:16.493542+0000","last_clean":"2024-09-18T08:20:16.493542+0000","last_became_active":"2024-09-18T08:15:41.445198+0000","last_became_peered":"2024-09-18T08:15:41.445198+0000","last_unstale":"2024-09-18T08:20:16.493542+0000","last_undegraded":"2024-09-18T08:20:16.493542+0000","last_fullsized":"2024-09-18T08:20:16.493542+0000","mapping_epoch":336,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":337,"parent":"0.0","parent_split_bits":0,"last_scrub":"271'19","last_scrub_stamp":"2024-09-18T08:18:20.608110+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_clean_scrub_stamp":"2024-09-18T08:18:20.608110+0000","objects_scrubbed":5,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":31,"num_read_kb":24,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,1],"acting":[5,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":843,"reported_epoch":438,"state":"active+clean","last_fresh":"2024-09-18T08:20:16.501106+0000","last_change":"2024-09-18T08:20:15.269092+0000","last_active":"2024-09-18T08:20:16.501106+0000","last_peered":"2024-09-18T08:20:16.501106+0000","last_clean":"2024-09-18T08:20:16.501106+0000","last_became_active":"2024-09-18T08:19:05.903954+0000","last_became_peered":"2024-09-18T08:19:05.903954+0000","last_unstale":"2024-09-18T08:20:16.501106+0000","last_undegraded":"2024-09-18T08:20:16.501106+0000","last_fullsized":"2024-09-18T08:20:16.501106+0000","mapping_epoch":413,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":414,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:20:15.269044+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:50.886830+0000","last_clean_scrub_stamp":"2024-09-18T08:20:15.269044+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:21:24.188336+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,1],"acting":[0,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":11,"num_pg":16,"stat_sum":{"num_bytes":4870504472,"num_objects":74319,"num_object_clones":0,"num_object_copies":222957,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":74319,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":123045,"num_write_kb":6315585,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19130,"num_bytes_recovered":1253703680,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":14797189120,"data_stored":14797176904,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":62345,"ondisk_log_size":62345,"up":48,"acting":48,"num_store_stats":7},{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":416,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":1235,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":32,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":125,"ondisk_log_size":125,"up":96,"acting":96,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":178,"num_object_clones":0,"num_object_copies":534,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":178,"num_whiteouts":0,"num_read":1106,"num_read_kb":863,"num_write":703,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":559,"num_bytes_recovered":10830,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":475136,"data_stored":12690,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3639,"internal_metadata":0},"log_size":690,"ondisk_log_size":690,"up":96,"acting":96,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":1613,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":14,"num_read_kb":14,"num_write":10,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":17,"num_bytes_recovered":6130,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":94208,"data_stored":6016,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":6,"ondisk_log_size":6,"up":96,"acting":96,"num_store_stats":8},{"poolid":2,"num_pg":3,"stat_sum":{"num_bytes":408,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":517,"num_read_kb":512,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":797,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":8,"ondisk_log_size":8,"up":9,"acting":9,"num_store_stats":7},{"poolid":1,"num_pg":11,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":1573408,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":869,"ondisk_log_size":869,"up":33,"acting":33,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":56,"seq":240518168792,"num_pgs":76,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2913232,"kb_used_data":2304512,"kb_used_omap":1,"kb_used_meta":608702,"kb_avail":90831920,"statfs":{"total":95995035648,"available":93011886080,"internally_reserved":0,"allocated":2359820288,"data_stored":2359409716,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":623311306},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,1,1],"upper_bound":8},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":47,"seq":201863463134,"num_pgs":70,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2944028,"kb_used_data":2326988,"kb_used_omap":1,"kb_used_meta":617022,"kb_avail":90801124,"statfs":{"total":95995035648,"available":92980350976,"internally_reserved":0,"allocated":2382835712,"data_stored":2382411596,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":631830982},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":39,"seq":167503724772,"num_pgs":72,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2898700,"kb_used_data":2019260,"kb_used_omap":1,"kb_used_meta":879422,"kb_avail":90846452,"statfs":{"total":95995035648,"available":93026766848,"internally_reserved":0,"allocated":2067722240,"data_stored":2067315126,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":900528582},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,3,1],"upper_bound":16},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":0,"seq":137438953705,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":26,"seq":111669149935,"num_pgs":76,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":3050460,"kb_used_data":2211276,"kb_used_omap":1,"kb_used_meta":839166,"kb_avail":90694692,"statfs":{"total":95995035648,"available":92871364608,"internally_reserved":0,"allocated":2264346624,"data_stored":2263923075,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":859306440},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,1,1,3,7],"upper_bound":32},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411574,"num_pgs":17,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":672220,"kb_used_data":1036,"kb_used_omap":1,"kb_used_meta":671166,"kb_avail":93072932,"statfs":{"total":95995035648,"available":95306682368,"internally_reserved":0,"allocated":1060864,"data_stored":701530,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":687274437},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":307,"seq":1318554959943,"num_pgs":84,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":3260076,"kb_used_data":2920796,"kb_used_omap":7,"kb_used_meta":339128,"kb_avail":90485076,"statfs":{"total":95995035648,"available":92656717824,"internally_reserved":0,"allocated":2990895104,"data_stored":2990456199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":8079,"internal_metadata":347267185},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,0,1,1],"upper_bound":16},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738625,"num_pgs":79,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":3092648,"kb_used_data":2335192,"kb_used_omap":1,"kb_used_meta":757246,"kb_avail":90652504,"statfs":{"total":95995035648,"available":92828164096,"internally_reserved":0,"allocated":2391236608,"data_stored":2390802056,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":775420357},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":266240,"data_stored":262688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1567,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1613,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":46,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1177,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":436,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":46,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1131,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":2024,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":1678,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3639,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":440,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":81920,"data_stored":2118,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":1584,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":77824,"data_stored":2052,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1474,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":403,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":416,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":403,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":2386427904,"data_stored":2386427904,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":2976583680,"data_stored":2976579608,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":418578432,"data_stored":418578432,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":2226192384,"data_stored":2226192384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":2064650240,"data_stored":2064646168,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":2369916928,"data_stored":2369912856,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":2354839552,"data_stored":2354839552,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T08:20:33.227 INFO:tasks.thrashosds.thrasher:Setting 11.d to [1, 5, 7, 2, 4, 6] 2024-09-18T08:20:33.227 INFO:tasks.thrashosds.thrasher:cmd ['osd', 'pg-upmap-items', '11.d', '1', '5', '7', '2', '4', '6'] 2024-09-18T08:20:33.227 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid 779349ca-7593-11ef-bceb-c7b262605968 -- ceph osd pg-upmap-items 11.d 1 5 7 2 4 6 2024-09-18T08:20:33.228 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:20:33.228 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:20:33.228 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "commit_sent", 2024-09-18T08:20:33.228 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:15:01.387756+0000", 2024-09-18T08:20:33.228 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 5.0906999999999999e-05 2024-09-18T08:20:33.228 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:20:33.228 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:20:33.228 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "done", 2024-09-18T08:20:33.228 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:15:01.387799+0000", 2024-09-18T08:20:33.228 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 4.2728000000000002e-05 2024-09-18T08:20:33.228 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-09-18T08:20:33.228 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-09-18T08:20:33.229 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-09-18T08:20:33.229 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:20:33.229 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:20:33.229 INFO:teuthology.orchestra.run.smithi007.stdout: "description": "osd_op(client.15966.0:98625 10.9 10:9690b9c0:::benchmark_data_smithi007_148834_object98624:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e347)", 2024-09-18T08:20:33.229 INFO:teuthology.orchestra.run.smithi007.stdout: "initiated_at": "2024-09-18T08:16:12.377991+0000", 2024-09-18T08:20:33.229 INFO:teuthology.orchestra.run.smithi007.stdout: "age": 260.67822803600001, 2024-09-18T08:20:33.229 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 2.0645955950000001, 2024-09-18T08:20:33.229 INFO:teuthology.orchestra.run.smithi007.stdout: "continuous": false, 2024-09-18T08:20:33.229 INFO:teuthology.orchestra.run.smithi007.stdout: "type_data": { 2024-09-18T08:20:33.229 INFO:teuthology.orchestra.run.smithi007.stdout: "flag_point": "reached pg", 2024-09-18T08:20:33.229 INFO:teuthology.orchestra.run.smithi007.stdout: "client_info": { 2024-09-18T08:20:33.229 INFO:teuthology.orchestra.run.smithi007.stdout: "client": "client.15966", 2024-09-18T08:20:33.229 INFO:teuthology.orchestra.run.smithi007.stdout: "client_addr": "172.21.15.7:0/529784850", 2024-09-18T08:20:33.229 INFO:teuthology.orchestra.run.smithi007.stdout: "tid": 98625 2024-09-18T08:20:33.230 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:20:33.230 INFO:teuthology.orchestra.run.smithi007.stdout: "events": [ 2024-09-18T08:20:33.230 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:20:33.230 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "initiated", 2024-09-18T08:20:33.230 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:16:12.377991+0000", 2024-09-18T08:20:33.230 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-09-18T08:20:33.230 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:20:33.230 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:20:33.230 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "header_read", 2024-09-18T08:20:33.230 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:16:12.377991+0000", 2024-09-18T08:20:33.230 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-09-18T08:20:33.230 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:20:33.230 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:20:33.230 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "throttled", 2024-09-18T08:20:33.231 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:16:12.377992+0000", 2024-09-18T08:20:33.231 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 4.9999999999999998e-07 2024-09-18T08:20:33.231 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:20:33.231 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:20:33.231 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "all_read", 2024-09-18T08:20:33.231 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:16:12.378041+0000", 2024-09-18T08:20:33.231 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 4.8764000000000003e-05 2024-09-18T08:20:33.231 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:20:33.231 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:20:33.231 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "dispatched", 2024-09-18T08:20:33.231 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:16:12.378047+0000", 2024-09-18T08:20:33.231 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 6.6409999999999996e-06 2024-09-18T08:20:33.231 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:20:33.231 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:20:33.232 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "queued_for_pg", 2024-09-18T08:20:33.232 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:16:12.378053+0000", 2024-09-18T08:20:33.232 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 5.4140000000000002e-06 2024-09-18T08:20:33.232 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:20:33.232 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:20:33.232 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-09-18T08:20:33.232 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:16:12.387065+0000", 2024-09-18T08:20:33.232 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.0090127599999999999 2024-09-18T08:20:33.232 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:20:33.232 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:20:33.232 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "started", 2024-09-18T08:20:33.232 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:16:12.387212+0000", 2024-09-18T08:20:33.232 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.000146169 2024-09-18T08:20:33.232 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:20:33.233 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:20:33.233 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "waiting for subops from 3,4,5", 2024-09-18T08:20:33.233 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:16:12.387395+0000", 2024-09-18T08:20:33.233 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.00018346500000000001 2024-09-18T08:20:33.233 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:20:33.233 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:20:33.233 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:20:33.233 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:16:12.396723+0000", 2024-09-18T08:20:33.233 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.0093279699999999997 2024-09-18T08:20:33.233 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:20:33.233 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:20:33.233 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:20:33.233 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:16:12.396785+0000", 2024-09-18T08:20:33.233 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 6.2169999999999996e-05 2024-09-18T08:20:33.234 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:20:33.234 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:20:33.234 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:20:33.234 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:16:12.422101+0000", 2024-09-18T08:20:33.234 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.025315666000000001 2024-09-18T08:20:33.234 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:20:33.234 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:20:33.234 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-09-18T08:20:33.234 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:16:12.425399+0000", 2024-09-18T08:20:33.234 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.0032976490000000002 2024-09-18T08:20:33.234 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:20:33.234 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:20:33.234 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "waiting for peered", 2024-09-18T08:20:33.234 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:16:12.425410+0000", 2024-09-18T08:20:33.235 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.1423000000000001e-05 2024-09-18T08:20:33.235 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:20:33.235 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:20:33.235 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-09-18T08:20:33.235 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:16:13.429965+0000", 2024-09-18T08:20:33.235 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.004555144 2024-09-18T08:20:33.235 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:20:33.235 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:20:33.235 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "waiting for peered", 2024-09-18T08:20:33.235 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:16:13.429977+0000", 2024-09-18T08:20:33.235 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.1511e-05 2024-09-18T08:20:33.235 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:20:33.235 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:20:33.235 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-09-18T08:20:33.236 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:16:14.442511+0000", 2024-09-18T08:20:33.236 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.012533948 2024-09-18T08:20:33.236 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:20:33.236 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:20:33.236 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "done", 2024-09-18T08:20:33.236 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:16:14.442587+0000", 2024-09-18T08:20:33.236 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 7.6401000000000003e-05 2024-09-18T08:20:33.236 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-09-18T08:20:33.236 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-09-18T08:20:33.236 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-09-18T08:20:33.236 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:20:33.236 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:20:33.236 INFO:teuthology.orchestra.run.smithi007.stdout: "description": "osd_op(client.15966.0:98630 10.9 10:93666e30:::benchmark_data_smithi007_148834_object98629:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e347)", 2024-09-18T08:20:33.237 INFO:teuthology.orchestra.run.smithi007.stdout: "initiated_at": "2024-09-18T08:16:12.389940+0000", 2024-09-18T08:20:33.237 INFO:teuthology.orchestra.run.smithi007.stdout: "age": 260.66627909900001, 2024-09-18T08:20:33.237 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 2.0536796430000002, 2024-09-18T08:20:33.237 INFO:teuthology.orchestra.run.smithi007.stdout: "continuous": false, 2024-09-18T08:20:33.237 INFO:teuthology.orchestra.run.smithi007.stdout: "type_data": { 2024-09-18T08:20:33.237 INFO:teuthology.orchestra.run.smithi007.stdout: "flag_point": "reached pg", 2024-09-18T08:20:33.237 INFO:teuthology.orchestra.run.smithi007.stdout: "client_info": { 2024-09-18T08:20:33.237 INFO:teuthology.orchestra.run.smithi007.stdout: "client": "client.15966", 2024-09-18T08:20:33.237 INFO:teuthology.orchestra.run.smithi007.stdout: "client_addr": "172.21.15.7:0/529784850", 2024-09-18T08:20:33.237 INFO:teuthology.orchestra.run.smithi007.stdout: "tid": 98630 2024-09-18T08:20:33.237 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:20:33.237 INFO:teuthology.orchestra.run.smithi007.stdout: "events": [ 2024-09-18T08:20:33.237 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:20:33.237 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "initiated", 2024-09-18T08:20:33.238 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:16:12.389940+0000", 2024-09-18T08:20:33.238 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-09-18T08:20:33.238 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:20:33.238 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:20:33.238 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "header_read", 2024-09-18T08:20:33.238 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:16:12.389940+0000", 2024-09-18T08:20:33.238 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-09-18T08:20:33.238 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:20:33.238 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:20:33.238 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "throttled", 2024-09-18T08:20:33.238 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:16:12.389941+0000", 2024-09-18T08:20:33.238 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.049e-06 2024-09-18T08:20:33.238 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:20:33.238 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:20:33.239 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "all_read", 2024-09-18T08:20:33.239 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:16:12.389964+0000", 2024-09-18T08:20:33.239 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 2.3085e-05 2024-09-18T08:20:33.239 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:20:33.239 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:20:33.239 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "dispatched", 2024-09-18T08:20:33.239 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:16:12.389975+0000", 2024-09-18T08:20:33.239 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.0988999999999999e-05 2024-09-18T08:20:33.239 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:20:33.239 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:20:33.239 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "queued_for_pg", 2024-09-18T08:20:33.239 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:16:12.389983+0000", 2024-09-18T08:20:33.239 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 7.9349999999999994e-06 2024-09-18T08:20:33.239 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:20:33.239 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:20:33.240 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-09-18T08:20:33.240 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:16:12.396823+0000", 2024-09-18T08:20:33.240 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.0068393330000000004 2024-09-18T08:20:33.240 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:20:33.240 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:20:33.240 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "started", 2024-09-18T08:20:33.240 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:16:12.396999+0000", 2024-09-18T08:20:33.240 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.00017637999999999999 2024-09-18T08:20:33.240 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:20:33.240 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:20:33.240 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "waiting for subops from 3,4,5", 2024-09-18T08:20:33.240 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:16:12.397192+0000", 2024-09-18T08:20:33.240 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.00019290900000000001 2024-09-18T08:20:33.240 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:20:33.241 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:20:33.241 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:20:33.241 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:16:12.422147+0000", 2024-09-18T08:20:33.241 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.024954787999999999 2024-09-18T08:20:33.241 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:20:33.241 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:20:33.241 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-09-18T08:20:33.241 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:16:12.425442+0000", 2024-09-18T08:20:33.241 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.0032947530000000001 2024-09-18T08:20:33.241 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:20:33.241 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:20:33.241 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "waiting for peered", 2024-09-18T08:20:33.241 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:16:12.425449+0000", 2024-09-18T08:20:33.241 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 7.5390000000000002e-06 2024-09-18T08:20:33.242 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:20:33.242 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:20:33.242 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-09-18T08:20:33.242 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:16:13.430015+0000", 2024-09-18T08:20:33.242 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.0045655259999999 2024-09-18T08:20:33.242 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:20:33.242 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:20:33.242 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "waiting for peered", 2024-09-18T08:20:33.242 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:16:13.430024+0000", 2024-09-18T08:20:33.243 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 9.1670000000000005e-06 2024-09-18T08:20:33.243 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:20:33.243 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:20:33.244 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-09-18T08:20:33.244 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:16:14.443546+0000", 2024-09-18T08:20:33.244 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.0135222079999999 2024-09-18T08:20:33.244 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:20:33.244 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:20:33.244 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "done", 2024-09-18T08:20:33.245 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:16:14.443620+0000", 2024-09-18T08:20:33.245 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 7.3981999999999997e-05 2024-09-18T08:20:33.245 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-09-18T08:20:33.245 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-09-18T08:20:33.245 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-09-18T08:20:33.245 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-09-18T08:20:33.245 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-09-18T08:20:33.245 INFO:teuthology.orchestra.run.smithi007.stdout:} 2024-09-18T08:20:33.247 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:20:33.247 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:20:33.247 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5' 2024-09-18T08:20:33.348 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0 2024-09-18T08:20:33.388 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:20:33.389 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:20:33.389 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0' 2024-09-18T08:20:33.490 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6 2024-09-18T08:20:33.515 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:20:33.516 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:20:33.516 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6' 2024-09-18T08:20:33.617 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3 2024-09-18T08:20:33.659 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:20:33.660 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:20:33.660 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3' 2024-09-18T08:20:33.761 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3 2024-09-18T08:20:33.803 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:20:33.803 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:20:33.803 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3' 2024-09-18T08:20:33.904 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1 2024-09-18T08:20:33.905 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:20:33 smithi050 ceph-mon[32371]: 3.18 scrub starts 2024-09-18T08:20:33.905 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:20:33 smithi050 ceph-mon[32371]: 3.18 scrub ok 2024-09-18T08:20:33.905 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:20:33 smithi050 ceph-mon[32371]: from='mgr.24470 172.21.15.7:0/2594390261' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-18T08:20:33.905 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:20:33 smithi050 ceph-mon[32371]: from='mgr.24470 172.21.15.7:0/2594390261' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-18T08:20:33.906 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:20:33 smithi050 ceph-mon[32371]: from='mgr.24470 172.21.15.7:0/2594390261' entity='mgr.y' 2024-09-18T08:20:33.941 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:20:33.942 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:20:33.942 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1' 2024-09-18T08:20:33.980 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:20:33 smithi007 ceph-mon[29459]: 3.18 scrub starts 2024-09-18T08:20:33.980 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:20:33 smithi007 ceph-mon[29459]: 3.18 scrub ok 2024-09-18T08:20:33.980 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:20:33 smithi007 ceph-mon[29459]: from='mgr.24470 172.21.15.7:0/2594390261' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-18T08:20:33.980 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:20:33 smithi007 ceph-mon[29459]: from='mgr.24470 172.21.15.7:0/2594390261' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-18T08:20:33.980 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:20:33 smithi007 ceph-mon[29459]: from='mgr.24470 172.21.15.7:0/2594390261' entity='mgr.y' 2024-09-18T08:20:33.980 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:20:33 smithi007 ceph-mon[39249]: 3.18 scrub starts 2024-09-18T08:20:33.981 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:20:33 smithi007 ceph-mon[39249]: 3.18 scrub ok 2024-09-18T08:20:33.981 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:20:33 smithi007 ceph-mon[39249]: from='mgr.24470 172.21.15.7:0/2594390261' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-18T08:20:33.981 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:20:33 smithi007 ceph-mon[39249]: from='mgr.24470 172.21.15.7:0/2594390261' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-18T08:20:33.981 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:20:33 smithi007 ceph-mon[39249]: from='mgr.24470 172.21.15.7:0/2594390261' entity='mgr.y' 2024-09-18T08:20:34.044 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4 2024-09-18T08:20:34.080 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:20:34.080 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:20:34.081 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4' 2024-09-18T08:20:34.128 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid 779349ca-7593-11ef-bceb-c7b262605968 -- ceph daemon osd.3 dump_ops_in_flight 2024-09-18T08:20:34.142 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid 779349ca-7593-11ef-bceb-c7b262605968 -- ceph osd set noscrub 2024-09-18T08:20:34.181 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1 2024-09-18T08:20:34.265 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:20:34.266 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:20:34.266 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1' 2024-09-18T08:20:34.367 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5 2024-09-18T08:20:34.396 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:20:34.396 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:20:34.397 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5' 2024-09-18T08:20:34.498 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2 2024-09-18T08:20:34.563 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:20:34.564 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:20:34.564 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2' 2024-09-18T08:20:34.665 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3 2024-09-18T08:20:34.703 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:20:34.704 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:20:34.704 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3' 2024-09-18T08:20:34.725 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:20:34 smithi007 ceph-mon[29459]: 6.6 scrub starts 2024-09-18T08:20:34.725 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:20:34 smithi007 ceph-mon[29459]: 6.6 scrub ok 2024-09-18T08:20:34.725 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:20:34 smithi007 ceph-mon[29459]: purged_snaps scrub starts 2024-09-18T08:20:34.725 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:20:34 smithi007 ceph-mon[29459]: purged_snaps scrub ok 2024-09-18T08:20:34.725 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:20:34 smithi007 ceph-mon[29459]: pgmap v889: 158 pgs: 158 active+clean; 4.2 GiB data, 17 GiB used, 609 GiB / 626 GiB avail; 1.7 KiB/s rd, 0 B/s wr, 1.57k op/s 2024-09-18T08:20:34.725 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:20:34 smithi007 ceph-mon[29459]: 4.1 scrub starts 2024-09-18T08:20:34.725 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:20:34 smithi007 ceph-mon[29459]: 4.1 scrub ok 2024-09-18T08:20:34.725 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:20:34 smithi007 ceph-mon[29459]: osdmap e442: 8 total, 8 up, 7 in 2024-09-18T08:20:34.726 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:20:34 smithi007 ceph-mon[29459]: from='mgr.24470 172.21.15.7:0/2594390261' entity='mgr.y' 2024-09-18T08:20:34.726 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:20:34 smithi007 ceph-mon[29459]: from='mgr.24470 172.21.15.7:0/2594390261' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-18T08:20:34.726 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:20:34 smithi007 ceph-mon[29459]: 4.6 scrub starts 2024-09-18T08:20:34.726 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:20:34 smithi007 ceph-mon[29459]: 4.6 scrub ok 2024-09-18T08:20:34.731 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:20:34 smithi007 ceph-mon[39249]: 6.6 scrub starts 2024-09-18T08:20:34.731 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:20:34 smithi007 ceph-mon[39249]: 6.6 scrub ok 2024-09-18T08:20:34.731 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:20:34 smithi007 ceph-mon[39249]: purged_snaps scrub starts 2024-09-18T08:20:34.731 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:20:34 smithi007 ceph-mon[39249]: purged_snaps scrub ok 2024-09-18T08:20:34.731 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:20:34 smithi007 ceph-mon[39249]: pgmap v889: 158 pgs: 158 active+clean; 4.2 GiB data, 17 GiB used, 609 GiB / 626 GiB avail; 1.7 KiB/s rd, 0 B/s wr, 1.57k op/s 2024-09-18T08:20:34.731 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:20:34 smithi007 ceph-mon[39249]: 4.1 scrub starts 2024-09-18T08:20:34.731 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:20:34 smithi007 ceph-mon[39249]: 4.1 scrub ok 2024-09-18T08:20:34.731 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:20:34 smithi007 ceph-mon[39249]: osdmap e442: 8 total, 8 up, 7 in 2024-09-18T08:20:34.731 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:20:34 smithi007 ceph-mon[39249]: from='mgr.24470 172.21.15.7:0/2594390261' entity='mgr.y' 2024-09-18T08:20:34.732 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:20:34 smithi007 ceph-mon[39249]: from='mgr.24470 172.21.15.7:0/2594390261' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-18T08:20:34.732 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:20:34 smithi007 ceph-mon[39249]: 4.6 scrub starts 2024-09-18T08:20:34.732 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:20:34 smithi007 ceph-mon[39249]: 4.6 scrub ok 2024-09-18T08:20:34.806 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6 2024-09-18T08:20:34.811 INFO:teuthology.orchestra.run.smithi007.stdout:osd.0: {} 2024-09-18T08:20:34.811 INFO:teuthology.orchestra.run.smithi007.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-18T08:20:34.832 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:20:34.832 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:20:34.833 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6' 2024-09-18T08:20:34.856 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/779349ca-7593-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T08:20:34.857 INFO:teuthology.orchestra.run.smithi007.stdout:osd.1: {} 2024-09-18T08:20:34.858 INFO:teuthology.orchestra.run.smithi007.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-18T08:20:34.878 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/779349ca-7593-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T08:20:34.896 INFO:teuthology.orchestra.run.smithi007.stdout:osd.2: {} 2024-09-18T08:20:34.896 INFO:teuthology.orchestra.run.smithi007.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-18T08:20:34.905 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:20:34 smithi050 ceph-mon[32371]: 6.6 scrub starts 2024-09-18T08:20:34.905 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:20:34 smithi050 ceph-mon[32371]: 6.6 scrub ok 2024-09-18T08:20:34.905 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:20:34 smithi050 ceph-mon[32371]: purged_snaps scrub starts 2024-09-18T08:20:34.905 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:20:34 smithi050 ceph-mon[32371]: purged_snaps scrub ok 2024-09-18T08:20:34.905 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:20:34 smithi050 ceph-mon[32371]: pgmap v889: 158 pgs: 158 active+clean; 4.2 GiB data, 17 GiB used, 609 GiB / 626 GiB avail; 1.7 KiB/s rd, 0 B/s wr, 1.57k op/s 2024-09-18T08:20:34.905 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:20:34 smithi050 ceph-mon[32371]: 4.1 scrub starts 2024-09-18T08:20:34.905 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:20:34 smithi050 ceph-mon[32371]: 4.1 scrub ok 2024-09-18T08:20:34.905 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:20:34 smithi050 ceph-mon[32371]: osdmap e442: 8 total, 8 up, 7 in 2024-09-18T08:20:34.905 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:20:34 smithi050 ceph-mon[32371]: from='mgr.24470 172.21.15.7:0/2594390261' entity='mgr.y' 2024-09-18T08:20:34.905 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:20:34 smithi050 ceph-mon[32371]: from='mgr.24470 172.21.15.7:0/2594390261' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-18T08:20:34.905 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:20:34 smithi050 ceph-mon[32371]: 4.6 scrub starts 2024-09-18T08:20:34.906 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:20:34 smithi050 ceph-mon[32371]: 4.6 scrub ok 2024-09-18T08:20:34.925 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/779349ca-7593-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T08:20:34.934 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0 2024-09-18T08:20:34.943 INFO:teuthology.orchestra.run.smithi007.stdout:osd.3: {} 2024-09-18T08:20:34.943 INFO:teuthology.orchestra.run.smithi007.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-18T08:20:34.976 INFO:teuthology.orchestra.run.smithi007.stdout:osd.4: {} 2024-09-18T08:20:34.977 INFO:teuthology.orchestra.run.smithi007.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-18T08:20:34.980 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:20:34.981 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:20:34.981 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0' 2024-09-18T08:20:35.016 INFO:teuthology.orchestra.run.smithi007.stdout:osd.5: {} 2024-09-18T08:20:35.016 INFO:teuthology.orchestra.run.smithi007.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-18T08:20:35.056 INFO:teuthology.orchestra.run.smithi007.stdout:osd.6: {} 2024-09-18T08:20:35.056 INFO:teuthology.orchestra.run.smithi007.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-18T08:20:35.082 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5 2024-09-18T08:20:35.093 INFO:teuthology.orchestra.run.smithi007.stdout:osd.7: {} 2024-09-18T08:20:35.094 INFO:teuthology.orchestra.run.smithi007.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-18T08:20:35.111 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:20:35.112 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:20:35.112 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5' 2024-09-18T08:20:35.213 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6 2024-09-18T08:20:35.244 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:20:35.244 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:20:35.244 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6' 2024-09-18T08:20:35.345 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7 2024-09-18T08:20:35.378 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:20:35.379 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:20:35.379 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7' 2024-09-18T08:20:35.480 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2 2024-09-18T08:20:35.516 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:20:35.517 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:20:35.517 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2' 2024-09-18T08:20:35.618 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2 2024-09-18T08:20:35.663 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:20:35.664 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:20:35.665 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2' 2024-09-18T08:20:35.729 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:20:35 smithi007 ceph-mon[39249]: from='client.? 172.21.15.7:0/3986868570' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-18T08:20:35.730 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:20:35 smithi007 ceph-mon[29459]: from='client.? 172.21.15.7:0/3986868570' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-18T08:20:35.765 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5 2024-09-18T08:20:35.798 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:20:35.799 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:20:35.799 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5' 2024-09-18T08:20:35.900 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4 2024-09-18T08:20:35.904 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:20:35 smithi050 ceph-mon[32371]: from='client.? 172.21.15.7:0/3986868570' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-18T08:20:35.936 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:20:35.937 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:20:35.937 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4' 2024-09-18T08:20:36.038 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0 2024-09-18T08:20:36.062 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:20:36.063 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:20:36.063 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0' 2024-09-18T08:20:36.164 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3 2024-09-18T08:20:36.200 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:20:36.201 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:20:36.202 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3' 2024-09-18T08:20:36.302 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4 2024-09-18T08:20:36.329 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:20:36.330 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:20:36.330 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4' 2024-09-18T08:20:36.431 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2 2024-09-18T08:20:36.467 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:20:36.467 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:20:36.468 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2' 2024-09-18T08:20:36.568 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2 2024-09-18T08:20:36.681 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:20:36.681 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:20:36.681 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2' 2024-09-18T08:20:36.782 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5 2024-09-18T08:20:36.807 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:20:36.808 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:20:36.808 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5' 2024-09-18T08:20:36.909 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1 2024-09-18T08:20:36.949 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:20:36.950 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:20:36.950 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1' 2024-09-18T08:20:36.980 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:20:36 smithi007 ceph-mon[29459]: pgmap v891: 158 pgs: 158 active+clean; 4.0 GiB data, 17 GiB used, 609 GiB / 626 GiB avail; 864 B/s rd, 0 B/s wr, 1.55k op/s 2024-09-18T08:20:36.980 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:20:36 smithi007 ceph-mon[29459]: 5.10 scrub starts 2024-09-18T08:20:36.980 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:20:36 smithi007 ceph-mon[29459]: 5.10 scrub ok 2024-09-18T08:20:36.980 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Sep 18 08:20:36 smithi007 ceph-779349ca-7593-11ef-bceb-c7b262605968-mgr-y[29688]: ::ffff:172.21.15.50 - - [18/Sep/2024:08:20:36] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-09-18T08:20:36.981 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:20:36 smithi007 ceph-mon[39249]: pgmap v891: 158 pgs: 158 active+clean; 4.0 GiB data, 17 GiB used, 609 GiB / 626 GiB avail; 864 B/s rd, 0 B/s wr, 1.55k op/s 2024-09-18T08:20:36.981 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:20:36 smithi007 ceph-mon[39249]: 5.10 scrub starts 2024-09-18T08:20:36.981 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:20:36 smithi007 ceph-mon[39249]: 5.10 scrub ok 2024-09-18T08:20:37.051 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2 2024-09-18T08:20:37.087 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:20:37.088 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:20:37.088 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2' 2024-09-18T08:20:37.155 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:20:36 smithi050 ceph-mon[32371]: pgmap v891: 158 pgs: 158 active+clean; 4.0 GiB data, 17 GiB used, 609 GiB / 626 GiB avail; 864 B/s rd, 0 B/s wr, 1.55k op/s 2024-09-18T08:20:37.155 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:20:36 smithi050 ceph-mon[32371]: 5.10 scrub starts 2024-09-18T08:20:37.155 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:20:36 smithi050 ceph-mon[32371]: 5.10 scrub ok 2024-09-18T08:20:37.188 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6 2024-09-18T08:20:37.224 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:20:37.224 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:20:37.224 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6' 2024-09-18T08:20:37.325 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1 2024-09-18T08:20:37.361 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:20:37.363 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:20:37.363 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1' 2024-09-18T08:20:37.464 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3 2024-09-18T08:20:37.499 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:20:37.500 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:20:37.500 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3' 2024-09-18T08:20:37.601 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4 2024-09-18T08:20:37.632 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:20:37.633 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:20:37.633 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4' 2024-09-18T08:20:37.734 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1 2024-09-18T08:20:37.774 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:20:37.775 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:20:37.775 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1' 2024-09-18T08:20:37.840 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid 779349ca-7593-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-18T08:20:37.876 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1 2024-09-18T08:20:37.941 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:20:37.942 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:20:37.942 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1' 2024-09-18T08:20:37.969 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:20:37 smithi007 ceph-mon[29459]: 4.14 scrub starts 2024-09-18T08:20:37.969 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:20:37 smithi007 ceph-mon[29459]: 4.14 scrub ok 2024-09-18T08:20:37.969 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:20:37 smithi007 ceph-mon[29459]: 5.1a scrub starts 2024-09-18T08:20:37.969 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:20:37 smithi007 ceph-mon[29459]: 5.1a scrub ok 2024-09-18T08:20:37.969 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:20:37 smithi007 ceph-mon[29459]: 4.1e scrub starts 2024-09-18T08:20:37.969 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:20:37 smithi007 ceph-mon[29459]: 4.1e scrub ok 2024-09-18T08:20:37.970 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:20:37 smithi007 ceph-mon[39249]: 4.14 scrub starts 2024-09-18T08:20:37.970 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:20:37 smithi007 ceph-mon[39249]: 4.14 scrub ok 2024-09-18T08:20:37.970 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:20:37 smithi007 ceph-mon[39249]: 5.1a scrub starts 2024-09-18T08:20:37.970 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:20:37 smithi007 ceph-mon[39249]: 5.1a scrub ok 2024-09-18T08:20:37.970 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:20:37 smithi007 ceph-mon[39249]: 4.1e scrub starts 2024-09-18T08:20:37.970 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:20:37 smithi007 ceph-mon[39249]: 4.1e scrub ok 2024-09-18T08:20:37.970 INFO:teuthology.orchestra.run.smithi007.stdout:ERROR: (22) Invalid argument 2024-09-18T08:20:37.970 INFO:teuthology.orchestra.run.smithi007.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-18T08:20:38.043 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6 2024-09-18T08:20:38.075 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:20:38.076 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:20:38.076 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6' 2024-09-18T08:20:38.155 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:20:37 smithi050 ceph-mon[32371]: 4.14 scrub starts 2024-09-18T08:20:38.155 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:20:37 smithi050 ceph-mon[32371]: 4.14 scrub ok 2024-09-18T08:20:38.155 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:20:37 smithi050 ceph-mon[32371]: 5.1a scrub starts 2024-09-18T08:20:38.155 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:20:37 smithi050 ceph-mon[32371]: 5.1a scrub ok 2024-09-18T08:20:38.155 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:20:37 smithi050 ceph-mon[32371]: 4.1e scrub starts 2024-09-18T08:20:38.155 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:20:37 smithi050 ceph-mon[32371]: 4.1e scrub ok 2024-09-18T08:20:38.178 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6 2024-09-18T08:20:38.204 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:20:38.205 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:20:38.205 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6' 2024-09-18T08:20:38.306 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4 2024-09-18T08:20:38.336 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:20:38.337 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:20:38.337 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4' 2024-09-18T08:20:38.438 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4 2024-09-18T08:20:38.467 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:20:38.468 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:20:38.468 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4' 2024-09-18T08:20:38.569 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2 2024-09-18T08:20:38.607 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:20:38.608 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:20:38.608 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2' 2024-09-18T08:20:38.709 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3 2024-09-18T08:20:38.749 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:20:38.750 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:20:38.750 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3' 2024-09-18T08:20:38.851 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3 2024-09-18T08:20:38.888 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:20:38.889 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:20:38.889 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3' 2024-09-18T08:20:38.979 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:20:38 smithi007 ceph-mon[29459]: pgmap v892: 158 pgs: 158 active+clean; 4.0 GiB data, 17 GiB used, 609 GiB / 626 GiB avail; 743 B/s rd, 0 B/s wr, 959 op/s 2024-09-18T08:20:38.980 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:20:38 smithi007 ceph-mon[29459]: 6.18 scrub starts 2024-09-18T08:20:38.980 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:20:38 smithi007 ceph-mon[29459]: 6.18 scrub ok 2024-09-18T08:20:38.980 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:20:38 smithi007 ceph-mon[39249]: pgmap v892: 158 pgs: 158 active+clean; 4.0 GiB data, 17 GiB used, 609 GiB / 626 GiB avail; 743 B/s rd, 0 B/s wr, 959 op/s 2024-09-18T08:20:38.980 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:20:38 smithi007 ceph-mon[39249]: 6.18 scrub starts 2024-09-18T08:20:38.980 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:20:38 smithi007 ceph-mon[39249]: 6.18 scrub ok 2024-09-18T08:20:38.991 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3 2024-09-18T08:20:39.027 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:20:39.028 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:20:39.028 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3' 2024-09-18T08:20:39.129 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3 2024-09-18T08:20:39.155 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:20:38 smithi050 ceph-mon[32371]: pgmap v892: 158 pgs: 158 active+clean; 4.0 GiB data, 17 GiB used, 609 GiB / 626 GiB avail; 743 B/s rd, 0 B/s wr, 959 op/s 2024-09-18T08:20:39.155 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:20:38 smithi050 ceph-mon[32371]: 6.18 scrub starts 2024-09-18T08:20:39.155 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:20:38 smithi050 ceph-mon[32371]: 6.18 scrub ok 2024-09-18T08:20:39.163 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:20:39.164 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:20:39.165 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3' 2024-09-18T08:20:39.266 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3 2024-09-18T08:20:39.307 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:20:39.307 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:20:39.308 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3' 2024-09-18T08:20:39.409 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2 2024-09-18T08:20:39.445 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:20:39.446 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:20:39.446 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2' 2024-09-18T08:20:39.532 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid 779349ca-7593-11ef-bceb-c7b262605968 -- ceph daemon osd.3 dump_blocked_ops 2024-09-18T08:20:39.548 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5 2024-09-18T08:20:39.581 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:20:39.582 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:20:39.582 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5' 2024-09-18T08:20:39.683 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2 2024-09-18T08:20:39.726 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:20:39.726 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:20:39.726 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2' 2024-09-18T08:20:39.729 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:20:39 smithi007 ceph-mon[29459]: 4.10 scrub starts 2024-09-18T08:20:39.732 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:20:39 smithi007 ceph-mon[29459]: 4.10 scrub ok 2024-09-18T08:20:39.732 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:20:39 smithi007 ceph-mon[29459]: 5.18 scrub starts 2024-09-18T08:20:39.732 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:20:39 smithi007 ceph-mon[29459]: 5.18 scrub ok 2024-09-18T08:20:39.733 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:20:39 smithi007 ceph-mon[29459]: 4.b scrub starts 2024-09-18T08:20:39.733 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:20:39 smithi007 ceph-mon[29459]: 4.b scrub ok 2024-09-18T08:20:39.733 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:20:39 smithi007 ceph-mon[39249]: 4.10 scrub starts 2024-09-18T08:20:39.733 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:20:39 smithi007 ceph-mon[39249]: 4.10 scrub ok 2024-09-18T08:20:39.733 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:20:39 smithi007 ceph-mon[39249]: 5.18 scrub starts 2024-09-18T08:20:39.733 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:20:39 smithi007 ceph-mon[39249]: 5.18 scrub ok 2024-09-18T08:20:39.734 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:20:39 smithi007 ceph-mon[39249]: 4.b scrub starts 2024-09-18T08:20:39.734 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:20:39 smithi007 ceph-mon[39249]: 4.b scrub ok 2024-09-18T08:20:39.828 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0 2024-09-18T08:20:39.875 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:20:39.876 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:20:39.876 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0' 2024-09-18T08:20:39.977 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1 2024-09-18T08:20:40.035 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:20:40.035 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:20:40.036 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1' 2024-09-18T08:20:40.113 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/779349ca-7593-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T08:20:40.137 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0 2024-09-18T08:20:40.155 INFO:journalctl@ceph.iscsi.iscsi.a.smithi050.stdout:Sep 18 08:20:39 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-iscsi-iscsi-a[57782]: debug there is no tcmu-runner data available 2024-09-18T08:20:40.155 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:20:39 smithi050 ceph-mon[32371]: 4.10 scrub starts 2024-09-18T08:20:40.155 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:20:39 smithi050 ceph-mon[32371]: 4.10 scrub ok 2024-09-18T08:20:40.155 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:20:39 smithi050 ceph-mon[32371]: 5.18 scrub starts 2024-09-18T08:20:40.155 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:20:39 smithi050 ceph-mon[32371]: 5.18 scrub ok 2024-09-18T08:20:40.155 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:20:39 smithi050 ceph-mon[32371]: 4.b scrub starts 2024-09-18T08:20:40.155 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:20:39 smithi050 ceph-mon[32371]: 4.b scrub ok 2024-09-18T08:20:40.189 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/779349ca-7593-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T08:20:40.189 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:20:40.190 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:20:40.190 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0' 2024-09-18T08:20:40.291 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0 2024-09-18T08:20:40.333 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:20:40.333 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:20:40.334 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0' 2024-09-18T08:20:40.435 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1 2024-09-18T08:20:40.480 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:20:40.480 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:20:40.481 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1' 2024-09-18T08:20:40.582 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4 2024-09-18T08:20:40.612 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:20:40.613 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:20:40.613 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4' 2024-09-18T08:20:40.689 INFO:teuthology.orchestra.run.smithi007.stderr:set 11.d pg_upmap_items mapping to [1->5,7->2,4->6] 2024-09-18T08:20:40.692 INFO:teuthology.orchestra.run.smithi007.stderr:noscrub is set 2024-09-18T08:20:40.714 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1 2024-09-18T08:20:40.755 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:20:40.755 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:20:40.756 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1' 2024-09-18T08:20:40.856 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0 2024-09-18T08:20:40.891 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:20:40.891 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:20:40.891 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0' 2024-09-18T08:20:40.980 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:20:40 smithi007 ceph-mon[29459]: pgmap v893: 158 pgs: 158 active+clean; 3.7 GiB data, 16 GiB used, 610 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.35k op/s 2024-09-18T08:20:40.980 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:20:40 smithi007 ceph-mon[29459]: 6.1e scrub starts 2024-09-18T08:20:40.980 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:20:40 smithi007 ceph-mon[29459]: 6.1e scrub ok 2024-09-18T08:20:40.980 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:20:40 smithi007 ceph-mon[29459]: purged_snaps scrub starts 2024-09-18T08:20:40.980 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:20:40 smithi007 ceph-mon[29459]: purged_snaps scrub ok 2024-09-18T08:20:40.980 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:20:40 smithi007 ceph-mon[29459]: 5.12 scrub starts 2024-09-18T08:20:40.980 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:20:40 smithi007 ceph-mon[29459]: 5.12 scrub ok 2024-09-18T08:20:40.981 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:20:40 smithi007 ceph-mon[29459]: 3.11 scrub starts 2024-09-18T08:20:40.981 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:20:40 smithi007 ceph-mon[29459]: 3.11 scrub ok 2024-09-18T08:20:40.981 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:20:40 smithi007 ceph-mon[29459]: from='client.? 172.21.15.7:0/3635039039' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-18T08:20:40.981 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:20:40 smithi007 ceph-mon[29459]: from='client.? 172.21.15.7:0/3458624816' entity='client.admin' cmd={"prefix": "osd pg-upmap-items", "pgid": "11.d", "id": [1, 5, 7, 2, 4, 6]} : dispatch 2024-09-18T08:20:40.981 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:20:40 smithi007 ceph-mon[39249]: pgmap v893: 158 pgs: 158 active+clean; 3.7 GiB data, 16 GiB used, 610 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.35k op/s 2024-09-18T08:20:40.981 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:20:40 smithi007 ceph-mon[39249]: 6.1e scrub starts 2024-09-18T08:20:40.981 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:20:40 smithi007 ceph-mon[39249]: 6.1e scrub ok 2024-09-18T08:20:40.981 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:20:40 smithi007 ceph-mon[39249]: purged_snaps scrub starts 2024-09-18T08:20:40.981 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:20:40 smithi007 ceph-mon[39249]: purged_snaps scrub ok 2024-09-18T08:20:40.981 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:20:40 smithi007 ceph-mon[39249]: 5.12 scrub starts 2024-09-18T08:20:40.982 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:20:40 smithi007 ceph-mon[39249]: 5.12 scrub ok 2024-09-18T08:20:40.982 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:20:40 smithi007 ceph-mon[39249]: 3.11 scrub starts 2024-09-18T08:20:40.982 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:20:40 smithi007 ceph-mon[39249]: 3.11 scrub ok 2024-09-18T08:20:40.982 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:20:40 smithi007 ceph-mon[39249]: from='client.? 172.21.15.7:0/3635039039' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-18T08:20:40.982 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:20:40 smithi007 ceph-mon[39249]: from='client.? 172.21.15.7:0/3458624816' entity='client.admin' cmd={"prefix": "osd pg-upmap-items", "pgid": "11.d", "id": [1, 5, 7, 2, 4, 6]} : dispatch 2024-09-18T08:20:40.992 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4 2024-09-18T08:20:41.022 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:20:41.023 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:20:41.023 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4' 2024-09-18T08:20:41.056 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:20:40 smithi050 ceph-mon[32371]: pgmap v893: 158 pgs: 158 active+clean; 3.7 GiB data, 16 GiB used, 610 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.35k op/s 2024-09-18T08:20:41.056 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:20:40 smithi050 ceph-mon[32371]: 6.1e scrub starts 2024-09-18T08:20:41.056 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:20:40 smithi050 ceph-mon[32371]: 6.1e scrub ok 2024-09-18T08:20:41.056 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:20:40 smithi050 ceph-mon[32371]: purged_snaps scrub starts 2024-09-18T08:20:41.056 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:20:40 smithi050 ceph-mon[32371]: purged_snaps scrub ok 2024-09-18T08:20:41.056 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:20:40 smithi050 ceph-mon[32371]: 5.12 scrub starts 2024-09-18T08:20:41.056 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:20:40 smithi050 ceph-mon[32371]: 5.12 scrub ok 2024-09-18T08:20:41.056 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:20:40 smithi050 ceph-mon[32371]: 3.11 scrub starts 2024-09-18T08:20:41.056 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:20:40 smithi050 ceph-mon[32371]: 3.11 scrub ok 2024-09-18T08:20:41.056 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:20:40 smithi050 ceph-mon[32371]: from='client.? 172.21.15.7:0/3635039039' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-18T08:20:41.056 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:20:40 smithi050 ceph-mon[32371]: from='client.? 172.21.15.7:0/3458624816' entity='client.admin' cmd={"prefix": "osd pg-upmap-items", "pgid": "11.d", "id": [1, 5, 7, 2, 4, 6]} : dispatch 2024-09-18T08:20:41.124 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2 2024-09-18T08:20:41.166 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:20:41.166 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:20:41.167 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2' 2024-09-18T08:20:41.267 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5 2024-09-18T08:20:41.301 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:20:41.302 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:20:41.302 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5' 2024-09-18T08:20:41.403 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0 2024-09-18T08:20:41.427 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:20:41.428 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:20:41.428 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0' 2024-09-18T08:20:41.529 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5 2024-09-18T08:20:41.551 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:20:41.552 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:20:41.552 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5' 2024-09-18T08:20:41.653 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7 2024-09-18T08:20:41.689 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:20:41.690 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:20:41.690 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7' 2024-09-18T08:20:41.791 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1 2024-09-18T08:20:41.833 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:20:41.834 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:20:41.834 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1' 2024-09-18T08:20:41.924 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:20:41 smithi050 ceph-mon[32371]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-18T08:20:41.924 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:20:41 smithi050 ceph-mon[32371]: 5.0 scrub starts 2024-09-18T08:20:41.924 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:20:41 smithi050 ceph-mon[32371]: 5.0 scrub ok 2024-09-18T08:20:41.924 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:20:41 smithi050 ceph-mon[32371]: 1.1 scrub starts 2024-09-18T08:20:41.924 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:20:41 smithi050 ceph-mon[32371]: 1.1 scrub ok 2024-09-18T08:20:41.925 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:20:41 smithi050 ceph-mon[32371]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-18T08:20:41.925 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:20:41 smithi050 ceph-mon[32371]: from='client.? 172.21.15.7:0/3635039039' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-18T08:20:41.925 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:20:41 smithi050 ceph-mon[32371]: from='client.? 172.21.15.7:0/3458624816' entity='client.admin' cmd='[{"prefix": "osd pg-upmap-items", "pgid": "11.d", "id": [1, 5, 7, 2, 4, 6]}]': finished 2024-09-18T08:20:41.925 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:20:41 smithi050 ceph-mon[32371]: osdmap e443: 8 total, 8 up, 7 in 2024-09-18T08:20:41.925 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:20:41 smithi050 ceph-mon[32371]: from='client.? 172.21.15.7:0/521879330' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-18T08:20:41.935 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6 2024-09-18T08:20:41.965 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:20:41.966 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:20:41.966 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6' 2024-09-18T08:20:41.980 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:20:41 smithi007 ceph-mon[29459]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-18T08:20:41.980 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:20:41 smithi007 ceph-mon[29459]: 5.0 scrub starts 2024-09-18T08:20:41.980 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:20:41 smithi007 ceph-mon[29459]: 5.0 scrub ok 2024-09-18T08:20:41.980 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:20:41 smithi007 ceph-mon[29459]: 1.1 scrub starts 2024-09-18T08:20:41.980 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:20:41 smithi007 ceph-mon[29459]: 1.1 scrub ok 2024-09-18T08:20:41.980 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:20:41 smithi007 ceph-mon[29459]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-18T08:20:41.980 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:20:41 smithi007 ceph-mon[29459]: from='client.? 172.21.15.7:0/3635039039' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-18T08:20:41.980 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:20:41 smithi007 ceph-mon[29459]: from='client.? 172.21.15.7:0/3458624816' entity='client.admin' cmd='[{"prefix": "osd pg-upmap-items", "pgid": "11.d", "id": [1, 5, 7, 2, 4, 6]}]': finished 2024-09-18T08:20:41.980 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:20:41 smithi007 ceph-mon[29459]: osdmap e443: 8 total, 8 up, 7 in 2024-09-18T08:20:41.980 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:20:41 smithi007 ceph-mon[29459]: from='client.? 172.21.15.7:0/521879330' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-18T08:20:41.981 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:20:41 smithi007 ceph-mon[39249]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-18T08:20:41.981 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:20:41 smithi007 ceph-mon[39249]: 5.0 scrub starts 2024-09-18T08:20:41.981 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:20:41 smithi007 ceph-mon[39249]: 5.0 scrub ok 2024-09-18T08:20:41.981 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:20:41 smithi007 ceph-mon[39249]: 1.1 scrub starts 2024-09-18T08:20:41.981 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:20:41 smithi007 ceph-mon[39249]: 1.1 scrub ok 2024-09-18T08:20:41.981 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:20:41 smithi007 ceph-mon[39249]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-18T08:20:41.981 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:20:41 smithi007 ceph-mon[39249]: from='client.? 172.21.15.7:0/3635039039' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-18T08:20:41.981 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:20:41 smithi007 ceph-mon[39249]: from='client.? 172.21.15.7:0/3458624816' entity='client.admin' cmd='[{"prefix": "osd pg-upmap-items", "pgid": "11.d", "id": [1, 5, 7, 2, 4, 6]}]': finished 2024-09-18T08:20:41.981 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:20:41 smithi007 ceph-mon[39249]: osdmap e443: 8 total, 8 up, 7 in 2024-09-18T08:20:41.981 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:20:41 smithi007 ceph-mon[39249]: from='client.? 172.21.15.7:0/521879330' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-18T08:20:41.982 INFO:teuthology.orchestra.run.smithi007.stdout:osd.0: {} 2024-09-18T08:20:41.982 INFO:teuthology.orchestra.run.smithi007.stdout:osd.1: {} 2024-09-18T08:20:41.982 INFO:teuthology.orchestra.run.smithi007.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-18T08:20:41.982 INFO:teuthology.orchestra.run.smithi007.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-18T08:20:41.982 INFO:teuthology.orchestra.run.smithi007.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-18T08:20:41.982 INFO:teuthology.orchestra.run.smithi007.stdout:osd.2: {} 2024-09-18T08:20:41.983 INFO:teuthology.orchestra.run.smithi007.stdout:osd.3: {} 2024-09-18T08:20:41.983 INFO:teuthology.orchestra.run.smithi007.stdout:osd.4: {} 2024-09-18T08:20:41.983 INFO:teuthology.orchestra.run.smithi007.stdout:osd.5: {} 2024-09-18T08:20:41.983 INFO:teuthology.orchestra.run.smithi007.stdout:osd.6: {} 2024-09-18T08:20:41.983 INFO:teuthology.orchestra.run.smithi007.stdout:osd.7: {} 2024-09-18T08:20:41.983 INFO:teuthology.orchestra.run.smithi007.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-18T08:20:41.983 INFO:teuthology.orchestra.run.smithi007.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-18T08:20:41.983 INFO:teuthology.orchestra.run.smithi007.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-18T08:20:41.983 INFO:teuthology.orchestra.run.smithi007.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-18T08:20:41.983 INFO:teuthology.orchestra.run.smithi007.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-18T08:20:42.067 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7 2024-09-18T08:20:42.102 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:20:42.103 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:20:42.103 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7' 2024-09-18T08:20:42.205 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1 2024-09-18T08:20:42.233 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:20:42.233 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:20:42.234 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1' 2024-09-18T08:20:42.335 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4 2024-09-18T08:20:42.371 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:20:42.371 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:20:42.372 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4' 2024-09-18T08:20:42.473 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5 2024-09-18T08:20:42.507 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:20:42.508 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:20:42.509 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5' 2024-09-18T08:20:42.609 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1 2024-09-18T08:20:42.645 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:20:42.646 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:20:42.647 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1' 2024-09-18T08:20:42.748 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7 2024-09-18T08:20:42.784 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:20:42.784 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:20:42.785 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7' 2024-09-18T08:20:42.868 INFO:teuthology.orchestra.run.smithi007.stdout:ERROR: (22) Invalid argument 2024-09-18T08:20:42.868 INFO:teuthology.orchestra.run.smithi007.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-18T08:20:42.868 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:20:42 smithi007 ceph-mon[29459]: pgmap v895: 158 pgs: 158 active+clean; 3.6 GiB data, 15 GiB used, 610 GiB / 626 GiB avail; 639 B/s rd, 0 B/s wr, 1.38k op/s 2024-09-18T08:20:42.869 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:20:42 smithi007 ceph-mon[39249]: pgmap v895: 158 pgs: 158 active+clean; 3.6 GiB data, 15 GiB used, 610 GiB / 626 GiB avail; 639 B/s rd, 0 B/s wr, 1.38k op/s 2024-09-18T08:20:42.886 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4 2024-09-18T08:20:42.919 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:20:42.920 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:20:42.920 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4' 2024-09-18T08:20:43.021 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1 2024-09-18T08:20:43.062 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:20:43.062 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:20:43.063 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1' 2024-09-18T08:20:43.155 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:20:42 smithi050 ceph-mon[32371]: pgmap v895: 158 pgs: 158 active+clean; 3.6 GiB data, 15 GiB used, 610 GiB / 626 GiB avail; 639 B/s rd, 0 B/s wr, 1.38k op/s 2024-09-18T08:20:43.163 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6 2024-09-18T08:20:43.195 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:20:43.196 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:20:43.196 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6' 2024-09-18T08:20:43.297 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7 2024-09-18T08:20:43.331 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:20:43.331 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:20:43.332 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7' 2024-09-18T08:20:43.433 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2 2024-09-18T08:20:43.469 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:20:43.470 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:20:43.470 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2' 2024-09-18T08:20:43.571 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3 2024-09-18T08:20:43.611 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:20:43.612 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:20:43.612 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3' 2024-09-18T08:20:43.713 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3 2024-09-18T08:20:43.751 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:20:43.751 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:20:43.752 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3' 2024-09-18T08:20:43.853 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2 2024-09-18T08:20:43.897 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:20:43.898 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:20:43.898 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2' 2024-09-18T08:20:43.905 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:20:43 smithi050 ceph-mon[32371]: purged_snaps scrub starts 2024-09-18T08:20:43.905 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:20:43 smithi050 ceph-mon[32371]: purged_snaps scrub ok 2024-09-18T08:20:43.979 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:20:43 smithi007 ceph-mon[29459]: purged_snaps scrub starts 2024-09-18T08:20:43.980 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:20:43 smithi007 ceph-mon[29459]: purged_snaps scrub ok 2024-09-18T08:20:43.980 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:20:43 smithi007 ceph-mon[39249]: purged_snaps scrub starts 2024-09-18T08:20:43.980 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:20:43 smithi007 ceph-mon[39249]: purged_snaps scrub ok 2024-09-18T08:20:43.999 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1 2024-09-18T08:20:44.036 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:20:44.037 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:20:44.037 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1' 2024-09-18T08:20:44.081 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid 779349ca-7593-11ef-bceb-c7b262605968 -- ceph daemon osd.3 dump_historic_ops 2024-09-18T08:20:44.138 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0 2024-09-18T08:20:44.196 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:20:44.196 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:20:44.197 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0' 2024-09-18T08:20:44.297 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5 2024-09-18T08:20:44.326 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:20:44.327 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:20:44.327 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5' 2024-09-18T08:20:44.428 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6 2024-09-18T08:20:44.461 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:20:44.462 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:20:44.462 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6' 2024-09-18T08:20:44.562 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4 2024-09-18T08:20:44.597 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:20:44.598 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:20:44.598 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4' 2024-09-18T08:20:44.699 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0 2024-09-18T08:20:44.748 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:20:44.749 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:20:44.749 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0' 2024-09-18T08:20:44.850 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7 2024-09-18T08:20:44.887 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:20:44.887 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:20:44.888 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7' 2024-09-18T08:20:44.979 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:20:44 smithi007 ceph-mon[29459]: pgmap v896: 158 pgs: 158 active+clean; 3.4 GiB data, 15 GiB used, 611 GiB / 626 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 1.27k op/s 2024-09-18T08:20:44.980 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:20:44 smithi007 ceph-mon[29459]: osdmap e444: 8 total, 8 up, 7 in 2024-09-18T08:20:44.980 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:20:44 smithi007 ceph-mon[39249]: pgmap v896: 158 pgs: 158 active+clean; 3.4 GiB data, 15 GiB used, 611 GiB / 626 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 1.27k op/s 2024-09-18T08:20:44.980 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:20:44 smithi007 ceph-mon[39249]: osdmap e444: 8 total, 8 up, 7 in 2024-09-18T08:20:44.988 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1 2024-09-18T08:20:45.030 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:20:45.031 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:20:45.031 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1' 2024-09-18T08:20:45.132 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0 2024-09-18T08:20:45.155 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:20:44 smithi050 ceph-mon[32371]: pgmap v896: 158 pgs: 158 active+clean; 3.4 GiB data, 15 GiB used, 611 GiB / 626 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 1.27k op/s 2024-09-18T08:20:45.155 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:20:44 smithi050 ceph-mon[32371]: osdmap e444: 8 total, 8 up, 7 in 2024-09-18T08:20:45.172 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:20:45.173 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:20:45.173 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0' 2024-09-18T08:20:45.274 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3 2024-09-18T08:20:45.309 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/779349ca-7593-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T08:20:45.321 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:20:45.322 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:20:45.322 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3' 2024-09-18T08:20:45.423 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7 2024-09-18T08:20:45.454 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:20:45.455 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:20:45.455 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7' 2024-09-18T08:20:45.556 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1 2024-09-18T08:20:45.598 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:20:45.599 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:20:45.599 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1' 2024-09-18T08:20:45.608 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid 779349ca-7593-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-18T08:20:45.700 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5 2024-09-18T08:20:45.730 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:20:45.731 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:20:45.731 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5' 2024-09-18T08:20:45.832 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5 2024-09-18T08:20:45.859 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:20:45.860 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:20:45.860 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5' 2024-09-18T08:20:45.944 INFO:teuthology.orchestra.run.smithi007.stdout:{ 2024-09-18T08:20:45.944 INFO:teuthology.orchestra.run.smithi007.stdout: "size": 20, 2024-09-18T08:20:45.944 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 600, 2024-09-18T08:20:45.944 INFO:teuthology.orchestra.run.smithi007.stdout: "ops": [ 2024-09-18T08:20:45.944 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:20:45.944 INFO:teuthology.orchestra.run.smithi007.stdout: "description": "osd_op(client.15582.0:252481 9.1 9:8590ecfa:::benchmark_data_smithi007_124019_object110596:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e291)", 2024-09-18T08:20:45.944 INFO:teuthology.orchestra.run.smithi007.stdout: "initiated_at": "2024-09-18T08:14:04.554046+0000", 2024-09-18T08:20:45.945 INFO:teuthology.orchestra.run.smithi007.stdout: "age": 401.38564437399998, 2024-09-18T08:20:45.945 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 2.257659055, 2024-09-18T08:20:45.945 INFO:teuthology.orchestra.run.smithi007.stdout: "continuous": false, 2024-09-18T08:20:45.945 INFO:teuthology.orchestra.run.smithi007.stdout: "type_data": { 2024-09-18T08:20:45.945 INFO:teuthology.orchestra.run.smithi007.stdout: "flag_point": "reached pg", 2024-09-18T08:20:45.945 INFO:teuthology.orchestra.run.smithi007.stdout: "client_info": { 2024-09-18T08:20:45.945 INFO:teuthology.orchestra.run.smithi007.stdout: "client": "client.15582", 2024-09-18T08:20:45.945 INFO:teuthology.orchestra.run.smithi007.stdout: "client_addr": "172.21.15.7:0/3598605839", 2024-09-18T08:20:45.945 INFO:teuthology.orchestra.run.smithi007.stdout: "tid": 252481 2024-09-18T08:20:45.945 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:20:45.945 INFO:teuthology.orchestra.run.smithi007.stdout: "events": [ 2024-09-18T08:20:45.945 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:20:45.945 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "initiated", 2024-09-18T08:20:45.946 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:14:04.554046+0000", 2024-09-18T08:20:45.946 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-09-18T08:20:45.946 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:20:45.946 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:20:45.946 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "header_read", 2024-09-18T08:20:45.946 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:14:04.554046+0000", 2024-09-18T08:20:45.946 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-09-18T08:20:45.946 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:20:45.946 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:20:45.946 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "throttled", 2024-09-18T08:20:45.946 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:14:04.554048+0000", 2024-09-18T08:20:45.946 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.4270000000000001e-06 2024-09-18T08:20:45.946 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:20:45.946 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:20:45.947 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "all_read", 2024-09-18T08:20:45.947 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:14:04.554054+0000", 2024-09-18T08:20:45.947 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 6.1800000000000001e-06 2024-09-18T08:20:45.947 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:20:45.947 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:20:45.947 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "dispatched", 2024-09-18T08:20:45.947 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:14:04.554064+0000", 2024-09-18T08:20:45.947 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.0468e-05 2024-09-18T08:20:45.947 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:20:45.947 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:20:45.947 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "queued_for_pg", 2024-09-18T08:20:45.947 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:14:04.554072+0000", 2024-09-18T08:20:45.947 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 7.2089999999999999e-06 2024-09-18T08:20:45.948 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:20:45.948 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:20:45.948 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-09-18T08:20:45.948 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:14:04.557405+0000", 2024-09-18T08:20:45.948 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.0033328590000000001 2024-09-18T08:20:45.948 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:20:45.948 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:20:45.948 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "started", 2024-09-18T08:20:45.948 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:14:04.557507+0000", 2024-09-18T08:20:45.948 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.000102092 2024-09-18T08:20:45.948 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:20:45.949 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:20:45.949 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "waiting for subops from 6,7", 2024-09-18T08:20:45.949 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:14:04.557610+0000", 2024-09-18T08:20:45.949 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.000103459 2024-09-18T08:20:45.949 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:20:45.949 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:20:45.949 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:20:45.950 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:14:04.561424+0000", 2024-09-18T08:20:45.950 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.0038139010000000002 2024-09-18T08:20:45.950 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:20:45.950 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:20:45.950 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "op_commit", 2024-09-18T08:20:45.950 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:14:04.565070+0000", 2024-09-18T08:20:45.950 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.0036458910000000001 2024-09-18T08:20:45.950 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:20:45.950 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:20:45.950 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-09-18T08:20:45.950 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:14:04.593896+0000", 2024-09-18T08:20:45.950 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.028826368000000002 2024-09-18T08:20:45.951 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:20:45.951 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:20:45.951 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "waiting for peered", 2024-09-18T08:20:45.951 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:14:04.593907+0000", 2024-09-18T08:20:45.951 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.0774999999999999e-05 2024-09-18T08:20:45.951 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:20:45.951 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:20:45.951 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-09-18T08:20:45.951 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:14:05.577311+0000", 2024-09-18T08:20:45.951 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.98340420900000003 2024-09-18T08:20:45.951 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:20:45.951 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:20:45.951 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "waiting for peered", 2024-09-18T08:20:45.952 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:14:05.577322+0000", 2024-09-18T08:20:45.952 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.0917999999999999e-05 2024-09-18T08:20:45.952 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:20:45.952 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:20:45.952 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-09-18T08:20:45.952 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:14:06.811620+0000", 2024-09-18T08:20:45.952 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.234297778 2024-09-18T08:20:45.952 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:20:45.952 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:20:45.953 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "done", 2024-09-18T08:20:45.953 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:14:06.811705+0000", 2024-09-18T08:20:45.953 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 8.5520999999999997e-05 2024-09-18T08:20:45.953 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-09-18T08:20:45.953 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-09-18T08:20:45.953 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-09-18T08:20:45.953 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:20:45.953 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:20:45.953 INFO:teuthology.orchestra.run.smithi007.stdout: "description": "osd_op(client.24356.0:2783 5.14 5:2b04a3e9:::notify.6:head [watch reconnect cookie 94700928708992 gen 2] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e292)", 2024-09-18T08:20:45.953 INFO:teuthology.orchestra.run.smithi007.stdout: "initiated_at": "2024-09-18T08:14:04.565629+0000", 2024-09-18T08:20:45.953 INFO:teuthology.orchestra.run.smithi007.stdout: "age": 401.37406148000002, 2024-09-18T08:20:45.953 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 2.057368002, 2024-09-18T08:20:45.953 INFO:teuthology.orchestra.run.smithi007.stdout: "continuous": false, 2024-09-18T08:20:45.954 INFO:teuthology.orchestra.run.smithi007.stdout: "type_data": { 2024-09-18T08:20:45.954 INFO:teuthology.orchestra.run.smithi007.stdout: "flag_point": "started", 2024-09-18T08:20:45.954 INFO:teuthology.orchestra.run.smithi007.stdout: "client_info": { 2024-09-18T08:20:45.954 INFO:teuthology.orchestra.run.smithi007.stdout: "client": "client.24356", 2024-09-18T08:20:45.954 INFO:teuthology.orchestra.run.smithi007.stdout: "client_addr": "172.21.15.7:0/219168083", 2024-09-18T08:20:45.954 INFO:teuthology.orchestra.run.smithi007.stdout: "tid": 2783 2024-09-18T08:20:45.954 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:20:45.954 INFO:teuthology.orchestra.run.smithi007.stdout: "events": [ 2024-09-18T08:20:45.954 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:20:45.954 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "initiated", 2024-09-18T08:20:45.954 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:14:04.565629+0000", 2024-09-18T08:20:45.954 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-09-18T08:20:45.954 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:20:45.954 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:20:45.955 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "header_read", 2024-09-18T08:20:45.955 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:14:04.565629+0000", 2024-09-18T08:20:45.955 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-09-18T08:20:45.955 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:20:45.955 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:20:45.955 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "throttled", 2024-09-18T08:20:45.955 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:14:04.565631+0000", 2024-09-18T08:20:45.955 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.2899999999999999e-06 2024-09-18T08:20:45.955 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:20:45.955 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:20:45.955 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "all_read", 2024-09-18T08:20:45.955 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:14:04.565636+0000", 2024-09-18T08:20:45.956 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 5.0379999999999999e-06 2024-09-18T08:20:45.956 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:20:45.956 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:20:45.956 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "dispatched", 2024-09-18T08:20:45.956 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:14:04.565647+0000", 2024-09-18T08:20:45.956 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.154e-05 2024-09-18T08:20:45.956 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:20:45.956 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:20:45.956 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "queued_for_pg", 2024-09-18T08:20:45.956 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:14:04.565654+0000", 2024-09-18T08:20:45.956 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 7.3479999999999998e-06 2024-09-18T08:20:45.956 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:20:45.956 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:20:45.957 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-09-18T08:20:45.957 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:14:04.565753+0000", 2024-09-18T08:20:45.957 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 9.8768e-05 2024-09-18T08:20:45.957 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:20:45.957 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:20:45.957 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "op must wait for map", 2024-09-18T08:20:45.957 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:14:04.565912+0000", 2024-09-18T08:20:45.957 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.00015838199999999999 2024-09-18T08:20:45.957 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:20:45.957 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:20:45.957 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-09-18T08:20:45.957 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:14:04.587017+0000", 2024-09-18T08:20:45.957 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.021105261 2024-09-18T08:20:45.957 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:20:45.958 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:20:45.958 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "waiting for peered", 2024-09-18T08:20:45.958 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:14:04.587024+0000", 2024-09-18T08:20:45.958 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 6.6449999999999999e-06 2024-09-18T08:20:45.958 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:20:45.958 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:20:45.958 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-09-18T08:20:45.958 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:14:05.582156+0000", 2024-09-18T08:20:45.958 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.99513244899999997 2024-09-18T08:20:45.958 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:20:45.958 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:20:45.958 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "waiting for peered", 2024-09-18T08:20:45.958 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:14:05.582170+0000", 2024-09-18T08:20:45.958 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.4248e-05 2024-09-18T08:20:45.959 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:20:45.959 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:20:45.959 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-09-18T08:20:45.959 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:14:06.605977+0000", 2024-09-18T08:20:45.959 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.0238063529999999 2024-09-18T08:20:45.959 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:20:45.959 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:20:45.959 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "started", 2024-09-18T08:20:45.959 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:14:06.606129+0000", 2024-09-18T08:20:45.959 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.00015248799999999999 2024-09-18T08:20:45.960 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:20:45.960 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:20:45.960 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "done", 2024-09-18T08:20:45.960 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:14:06.622997+0000", 2024-09-18T08:20:45.960 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.016868192000000001 2024-09-18T08:20:45.961 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-09-18T08:20:45.961 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-09-18T08:20:45.961 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-09-18T08:20:45.961 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:20:45.961 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:20:45.961 INFO:teuthology.orchestra.run.smithi007.stdout: "description": "osd_op(client.15582.0:252490 9.1 9:891fe784:::benchmark_data_smithi007_124019_object110605:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e291)", 2024-09-18T08:20:45.961 INFO:teuthology.orchestra.run.smithi007.stdout: "initiated_at": "2024-09-18T08:14:04.565679+0000", 2024-09-18T08:20:45.961 INFO:teuthology.orchestra.run.smithi007.stdout: "age": 401.37401176200001, 2024-09-18T08:20:45.961 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 2.2588554969999999, 2024-09-18T08:20:45.961 INFO:teuthology.orchestra.run.smithi007.stdout: "continuous": false, 2024-09-18T08:20:45.961 INFO:teuthology.orchestra.run.smithi007.stdout: "type_data": { 2024-09-18T08:20:45.961 INFO:teuthology.orchestra.run.smithi007.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T08:20:45.961 INFO:teuthology.orchestra.run.smithi007.stdout: "client_info": { 2024-09-18T08:20:45.962 INFO:teuthology.orchestra.run.smithi007.stdout: "client": "client.15582", 2024-09-18T08:20:45.962 INFO:teuthology.orchestra.run.smithi007.stdout: "client_addr": "172.21.15.7:0/3598605839", 2024-09-18T08:20:45.962 INFO:teuthology.orchestra.run.smithi007.stdout: "tid": 252490 2024-09-18T08:20:45.962 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:20:45.962 INFO:teuthology.orchestra.run.smithi007.stdout: "events": [ 2024-09-18T08:20:45.962 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:20:45.962 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "initiated", 2024-09-18T08:20:45.962 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:14:04.565679+0000", 2024-09-18T08:20:45.962 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-09-18T08:20:45.962 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:20:45.962 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:20:45.962 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "header_read", 2024-09-18T08:20:45.963 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:14:04.565679+0000", 2024-09-18T08:20:45.963 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-09-18T08:20:45.963 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:20:45.963 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:20:45.963 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "throttled", 2024-09-18T08:20:45.963 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:14:04.565680+0000", 2024-09-18T08:20:45.963 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 8.9700000000000005e-07 2024-09-18T08:20:45.963 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:20:45.963 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:20:45.963 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "all_read", 2024-09-18T08:20:45.964 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:14:04.565683+0000", 2024-09-18T08:20:45.964 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 3.2940000000000001e-06 2024-09-18T08:20:45.964 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:20:45.964 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:20:45.964 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "dispatched", 2024-09-18T08:20:45.964 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:14:04.565691+0000", 2024-09-18T08:20:45.964 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 7.7060000000000004e-06 2024-09-18T08:20:45.964 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:20:45.964 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:20:45.964 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "queued_for_pg", 2024-09-18T08:20:45.964 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:14:04.565697+0000", 2024-09-18T08:20:45.964 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 5.9139999999999996e-06 2024-09-18T08:20:45.964 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:20:45.965 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:20:45.965 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-09-18T08:20:45.965 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:14:04.593936+0000", 2024-09-18T08:20:45.965 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.028238722000000001 2024-09-18T08:20:45.965 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:20:45.965 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:20:45.965 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "waiting for peered", 2024-09-18T08:20:45.965 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:14:04.593944+0000", 2024-09-18T08:20:45.965 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 8.2039999999999997e-06 2024-09-18T08:20:45.965 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:20:45.965 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:20:45.965 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-09-18T08:20:45.965 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:14:05.577351+0000", 2024-09-18T08:20:45.965 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.98340685800000005 2024-09-18T08:20:45.966 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:20:45.966 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:20:45.966 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "waiting for peered", 2024-09-18T08:20:45.966 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:14:05.577357+0000", 2024-09-18T08:20:45.966 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 6.8299999999999998e-06 2024-09-18T08:20:45.966 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:20:45.966 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:20:45.966 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-09-18T08:20:45.966 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:14:06.811976+0000", 2024-09-18T08:20:45.966 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.234618384 2024-09-18T08:20:45.966 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:20:45.966 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:20:45.967 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "started", 2024-09-18T08:20:45.967 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:14:06.812084+0000", 2024-09-18T08:20:45.967 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.00010839 2024-09-18T08:20:45.967 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:20:45.967 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:20:45.967 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "waiting for subops from 4,6,7", 2024-09-18T08:20:45.967 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:14:06.812465+0000", 2024-09-18T08:20:45.967 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.00038073600000000001 2024-09-18T08:20:45.967 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:20:45.967 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:20:45.967 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "op_commit", 2024-09-18T08:20:45.967 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:14:06.816196+0000", 2024-09-18T08:20:45.967 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.0037314879999999998 2024-09-18T08:20:45.968 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:20:45.968 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:20:45.968 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:20:45.968 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:14:06.824173+0000", 2024-09-18T08:20:45.968 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.0079768100000000008 2024-09-18T08:20:45.968 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:20:45.968 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:20:45.968 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:20:45.968 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:14:06.824235+0000", 2024-09-18T08:20:45.968 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 6.1746000000000001e-05 2024-09-18T08:20:45.968 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:20:45.968 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:20:45.968 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:20:45.969 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:14:06.824291+0000", 2024-09-18T08:20:45.969 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 5.5689999999999997e-05 2024-09-18T08:20:45.969 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:20:45.969 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:20:45.969 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "commit_sent", 2024-09-18T08:20:45.969 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:14:06.824367+0000", 2024-09-18T08:20:45.969 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 7.6524000000000001e-05 2024-09-18T08:20:45.969 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:20:45.969 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:20:45.969 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "done", 2024-09-18T08:20:45.969 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:14:06.824534+0000", 2024-09-18T08:20:45.969 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.00016730400000000001 2024-09-18T08:20:45.970 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-09-18T08:20:45.970 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-09-18T08:20:45.970 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-09-18T08:20:45.970 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:20:45.970 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:20:45.970 INFO:teuthology.orchestra.run.smithi007.stdout: "description": "osd_op(client.15582.0:252491 9.c 9:326143b5:::benchmark_data_smithi007_124019_object110606:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e291)", 2024-09-18T08:20:45.970 INFO:teuthology.orchestra.run.smithi007.stdout: "initiated_at": "2024-09-18T08:14:04.576096+0000", 2024-09-18T08:20:45.970 INFO:teuthology.orchestra.run.smithi007.stdout: "age": 401.36359472300001, 2024-09-18T08:20:45.970 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 2.4808478369999998, 2024-09-18T08:20:45.970 INFO:teuthology.orchestra.run.smithi007.stdout: "continuous": false, 2024-09-18T08:20:45.970 INFO:teuthology.orchestra.run.smithi007.stdout: "type_data": { 2024-09-18T08:20:45.970 INFO:teuthology.orchestra.run.smithi007.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T08:20:45.970 INFO:teuthology.orchestra.run.smithi007.stdout: "client_info": { 2024-09-18T08:20:45.971 INFO:teuthology.orchestra.run.smithi007.stdout: "client": "client.15582", 2024-09-18T08:20:45.971 INFO:teuthology.orchestra.run.smithi007.stdout: "client_addr": "172.21.15.7:0/3598605839", 2024-09-18T08:20:45.971 INFO:teuthology.orchestra.run.smithi007.stdout: "tid": 252491 2024-09-18T08:20:45.971 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:20:45.971 INFO:teuthology.orchestra.run.smithi007.stdout: "events": [ 2024-09-18T08:20:45.971 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:20:45.971 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "initiated", 2024-09-18T08:20:45.971 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:14:04.576096+0000", 2024-09-18T08:20:45.971 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-09-18T08:20:45.971 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:20:45.971 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:20:45.971 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "header_read", 2024-09-18T08:20:45.971 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:14:04.576096+0000", 2024-09-18T08:20:45.971 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-09-18T08:20:45.972 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:20:45.972 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:20:45.972 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "throttled", 2024-09-18T08:20:45.972 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:14:04.576098+0000", 2024-09-18T08:20:45.972 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.7659999999999999e-06 2024-09-18T08:20:45.972 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:20:45.972 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:20:45.972 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "all_read", 2024-09-18T08:20:45.972 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:14:04.576105+0000", 2024-09-18T08:20:45.972 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 6.8839999999999997e-06 2024-09-18T08:20:45.972 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:20:45.972 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:20:45.972 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "dispatched", 2024-09-18T08:20:45.973 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:14:04.576118+0000", 2024-09-18T08:20:45.973 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.332e-05 2024-09-18T08:20:45.973 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:20:45.973 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:20:45.973 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "queued_for_pg", 2024-09-18T08:20:45.974 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:14:04.576127+0000", 2024-09-18T08:20:45.974 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 8.9700000000000005e-06 2024-09-18T08:20:45.974 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:20:45.974 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:20:45.974 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-09-18T08:20:45.974 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:14:04.586985+0000", 2024-09-18T08:20:45.974 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.010858247 2024-09-18T08:20:45.974 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:20:45.974 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:20:45.974 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "waiting for peered", 2024-09-18T08:20:45.974 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:14:04.586995+0000", 2024-09-18T08:20:45.974 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 9.3619999999999995e-06 2024-09-18T08:20:45.974 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:20:45.974 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:20:45.975 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-09-18T08:20:45.975 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:14:05.581931+0000", 2024-09-18T08:20:45.975 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.99493684800000004 2024-09-18T08:20:45.975 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:20:45.975 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:20:45.975 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "waiting for peered", 2024-09-18T08:20:45.975 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:14:05.581944+0000", 2024-09-18T08:20:45.975 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.2191999999999999e-05 2024-09-18T08:20:45.975 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:20:45.975 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:20:45.975 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-09-18T08:20:45.975 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:14:06.798281+0000", 2024-09-18T08:20:45.975 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.2163375969999999 2024-09-18T08:20:45.976 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:20:45.976 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:20:45.976 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "started", 2024-09-18T08:20:45.976 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:14:06.798419+0000", 2024-09-18T08:20:45.976 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.000137546 2024-09-18T08:20:45.976 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:20:45.976 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:20:45.976 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "waiting for subops from 0,4,7", 2024-09-18T08:20:45.976 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:14:06.798827+0000", 2024-09-18T08:20:45.976 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.00040866200000000001 2024-09-18T08:20:45.976 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:20:45.976 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:20:45.976 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:20:45.977 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:14:06.804658+0000", 2024-09-18T08:20:45.977 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.0058310159999999996 2024-09-18T08:20:45.977 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:20:45.977 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:20:45.977 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:20:45.977 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:14:06.808523+0000", 2024-09-18T08:20:45.977 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.0038643739999999998 2024-09-18T08:20:45.977 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:20:45.977 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:20:45.977 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "op_commit", 2024-09-18T08:20:45.977 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:14:06.808532+0000", 2024-09-18T08:20:45.977 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 9.3479999999999993e-06 2024-09-18T08:20:45.977 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:20:45.977 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:20:45.978 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:20:45.978 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:14:07.056320+0000", 2024-09-18T08:20:45.978 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.247787753 2024-09-18T08:20:45.978 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:20:45.978 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:20:45.978 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "commit_sent", 2024-09-18T08:20:45.978 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:14:07.056553+0000", 2024-09-18T08:20:45.978 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.00023296999999999999 2024-09-18T08:20:45.978 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:20:45.978 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:20:45.978 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "done", 2024-09-18T08:20:45.978 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:14:07.056944+0000", 2024-09-18T08:20:45.978 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.000390982 2024-09-18T08:20:45.979 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-09-18T08:20:45.979 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-09-18T08:20:45.979 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-09-18T08:20:45.979 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:20:45.979 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:20:45.979 INFO:teuthology.orchestra.run.smithi007.stdout: "description": "osd_op(client.15582.0:252477 9.3 9:c97eebf1:::benchmark_data_smithi007_124019_object110592:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e292)", 2024-09-18T08:20:45.979 INFO:teuthology.orchestra.run.smithi007.stdout: "initiated_at": "2024-09-18T08:14:04.578529+0000", 2024-09-18T08:20:45.979 INFO:teuthology.orchestra.run.smithi007.stdout: "age": 401.36116145699998, 2024-09-18T08:20:45.979 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 2.2331727560000001, 2024-09-18T08:20:45.979 INFO:teuthology.orchestra.run.smithi007.stdout: "continuous": false, 2024-09-18T08:20:45.979 INFO:teuthology.orchestra.run.smithi007.stdout: "type_data": { 2024-09-18T08:20:45.979 INFO:teuthology.orchestra.run.smithi007.stdout: "flag_point": "reached pg", 2024-09-18T08:20:45.979 INFO:teuthology.orchestra.run.smithi007.stdout: "client_info": { 2024-09-18T08:20:45.980 INFO:teuthology.orchestra.run.smithi007.stdout: "client": "client.15582", 2024-09-18T08:20:45.980 INFO:teuthology.orchestra.run.smithi007.stdout: "client_addr": "172.21.15.7:0/3598605839", 2024-09-18T08:20:45.980 INFO:teuthology.orchestra.run.smithi007.stdout: "tid": 252477 2024-09-18T08:20:45.980 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:20:45.980 INFO:teuthology.orchestra.run.smithi007.stdout: "events": [ 2024-09-18T08:20:45.980 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:20:45.980 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "initiated", 2024-09-18T08:20:45.980 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:14:04.578529+0000", 2024-09-18T08:20:45.980 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-09-18T08:20:45.980 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:20:45.980 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:20:45.980 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "header_read", 2024-09-18T08:20:45.980 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:14:04.578529+0000", 2024-09-18T08:20:45.981 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-09-18T08:20:45.981 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:20:45.981 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:20:45.981 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "throttled", 2024-09-18T08:20:45.981 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:14:04.578530+0000", 2024-09-18T08:20:45.981 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 4.3099999999999998e-07 2024-09-18T08:20:45.981 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:20:45.981 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:20:45.981 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "all_read", 2024-09-18T08:20:45.981 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:14:04.578532+0000", 2024-09-18T08:20:45.981 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 2.5579999999999999e-06 2024-09-18T08:20:45.981 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:20:45.981 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:20:45.982 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "dispatched", 2024-09-18T08:20:45.982 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:14:04.578538+0000", 2024-09-18T08:20:45.982 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 6.2140000000000001e-06 2024-09-18T08:20:45.982 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:20:45.982 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:20:45.982 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "queued_for_pg", 2024-09-18T08:20:45.982 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:14:04.578544+0000", 2024-09-18T08:20:45.982 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 5.9769999999999997e-06 2024-09-18T08:20:45.982 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:20:45.982 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:20:45.982 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-09-18T08:20:45.982 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:14:04.578776+0000", 2024-09-18T08:20:45.982 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.000231306 2024-09-18T08:20:45.982 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:20:45.983 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:20:45.983 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "waiting for peered", 2024-09-18T08:20:45.983 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:14:04.578784+0000", 2024-09-18T08:20:45.983 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 8.0989999999999999e-06 2024-09-18T08:20:45.983 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:20:45.983 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:20:45.983 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-09-18T08:20:45.983 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:14:05.571602+0000", 2024-09-18T08:20:45.983 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.992817799 2024-09-18T08:20:45.983 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:20:45.984 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:20:45.984 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "waiting for peered", 2024-09-18T08:20:45.984 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:14:05.571612+0000", 2024-09-18T08:20:45.984 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.0336000000000001e-05 2024-09-18T08:20:45.984 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:20:45.984 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:20:45.984 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-09-18T08:20:45.984 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:14:06.811618+0000", 2024-09-18T08:20:45.984 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.240005698 2024-09-18T08:20:45.984 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:20:45.984 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:20:45.984 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "done", 2024-09-18T08:20:45.985 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:14:06.811702+0000", 2024-09-18T08:20:45.985 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 8.4338000000000005e-05 2024-09-18T08:20:45.985 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-09-18T08:20:45.985 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-09-18T08:20:45.985 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-09-18T08:20:45.985 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:20:45.985 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:20:45.985 INFO:teuthology.orchestra.run.smithi007.stdout: "description": "osd_op(client.15582.0:252478 9.3 9:ce475113:::benchmark_data_smithi007_124019_object110593:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e292)", 2024-09-18T08:20:45.985 INFO:teuthology.orchestra.run.smithi007.stdout: "initiated_at": "2024-09-18T08:14:04.578552+0000", 2024-09-18T08:20:45.985 INFO:teuthology.orchestra.run.smithi007.stdout: "age": 401.36113841000002, 2024-09-18T08:20:45.985 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 2.2335192359999998, 2024-09-18T08:20:45.985 INFO:teuthology.orchestra.run.smithi007.stdout: "continuous": false, 2024-09-18T08:20:45.985 INFO:teuthology.orchestra.run.smithi007.stdout: "type_data": { 2024-09-18T08:20:45.985 INFO:teuthology.orchestra.run.smithi007.stdout: "flag_point": "reached pg", 2024-09-18T08:20:45.986 INFO:teuthology.orchestra.run.smithi007.stdout: "client_info": { 2024-09-18T08:20:45.986 INFO:teuthology.orchestra.run.smithi007.stdout: "client": "client.15582", 2024-09-18T08:20:45.986 INFO:teuthology.orchestra.run.smithi007.stdout: "client_addr": "172.21.15.7:0/3598605839", 2024-09-18T08:20:45.986 INFO:teuthology.orchestra.run.smithi007.stdout: "tid": 252478 2024-09-18T08:20:45.986 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:20:45.986 INFO:teuthology.orchestra.run.smithi007.stdout: "events": [ 2024-09-18T08:20:45.986 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:20:45.986 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "initiated", 2024-09-18T08:20:45.986 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:14:04.578552+0000", 2024-09-18T08:20:45.986 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-09-18T08:20:45.986 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:20:45.986 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:20:45.986 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "header_read", 2024-09-18T08:20:45.986 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:14:04.578552+0000", 2024-09-18T08:20:45.987 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-09-18T08:20:45.987 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:20:45.987 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:20:45.987 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "throttled", 2024-09-18T08:20:45.987 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:14:04.578553+0000", 2024-09-18T08:20:45.987 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 4.2399999999999999e-07 2024-09-18T08:20:45.987 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:20:45.987 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:20:45.987 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "all_read", 2024-09-18T08:20:45.987 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:14:04.578555+0000", 2024-09-18T08:20:45.987 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 2.1959999999999998e-06 2024-09-18T08:20:45.987 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:20:45.987 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:20:45.987 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "dispatched", 2024-09-18T08:20:45.988 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:14:04.578560+0000", 2024-09-18T08:20:45.988 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 5.2440000000000001e-06 2024-09-18T08:20:45.988 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:20:45.988 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:20:45.988 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "queued_for_pg", 2024-09-18T08:20:45.988 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:14:04.578565+0000", 2024-09-18T08:20:45.988 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 4.6009999999999997e-06 2024-09-18T08:20:45.988 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:20:45.988 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:20:45.988 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-09-18T08:20:45.988 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:14:04.578814+0000", 2024-09-18T08:20:45.988 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.00024935199999999999 2024-09-18T08:20:45.988 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:20:45.988 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:20:45.989 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "waiting for peered", 2024-09-18T08:20:45.989 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:14:04.578820+0000", 2024-09-18T08:20:45.989 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 6.0220000000000001e-06 2024-09-18T08:20:45.989 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:20:45.989 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:20:45.989 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-09-18T08:20:45.989 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:14:05.571644+0000", 2024-09-18T08:20:45.989 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.99282350200000002 2024-09-18T08:20:45.989 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:20:45.989 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:20:45.989 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "waiting for peered", 2024-09-18T08:20:45.989 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:14:05.571651+0000", 2024-09-18T08:20:45.989 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 7.2250000000000003e-06 2024-09-18T08:20:45.989 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:20:45.990 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:20:45.990 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-09-18T08:20:45.990 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:14:06.811970+0000", 2024-09-18T08:20:45.990 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.240318786 2024-09-18T08:20:45.990 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:20:45.990 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:20:45.990 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "done", 2024-09-18T08:20:45.990 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:14:06.812072+0000", 2024-09-18T08:20:45.990 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.000101884 2024-09-18T08:20:45.990 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-09-18T08:20:45.990 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0 2024-09-18T08:20:45.992 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-09-18T08:20:45.992 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-09-18T08:20:45.992 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:20:45.992 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:20:45.992 INFO:teuthology.orchestra.run.smithi007.stdout: "description": "osd_op(client.15582.0:252483 9.3 9:cd866060:::benchmark_data_smithi007_124019_object110598:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e292)", 2024-09-18T08:20:45.992 INFO:teuthology.orchestra.run.smithi007.stdout: "initiated_at": "2024-09-18T08:14:04.578571+0000", 2024-09-18T08:20:45.992 INFO:teuthology.orchestra.run.smithi007.stdout: "age": 401.36111990500001, 2024-09-18T08:20:45.992 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 2.2336163340000001, 2024-09-18T08:20:45.992 INFO:teuthology.orchestra.run.smithi007.stdout: "continuous": false, 2024-09-18T08:20:45.992 INFO:teuthology.orchestra.run.smithi007.stdout: "type_data": { 2024-09-18T08:20:45.992 INFO:teuthology.orchestra.run.smithi007.stdout: "flag_point": "reached pg", 2024-09-18T08:20:45.992 INFO:teuthology.orchestra.run.smithi007.stdout: "client_info": { 2024-09-18T08:20:45.993 INFO:teuthology.orchestra.run.smithi007.stdout: "client": "client.15582", 2024-09-18T08:20:45.993 INFO:teuthology.orchestra.run.smithi007.stdout: "client_addr": "172.21.15.7:0/3598605839", 2024-09-18T08:20:45.993 INFO:teuthology.orchestra.run.smithi007.stdout: "tid": 252483 2024-09-18T08:20:45.993 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:20:45.993 INFO:teuthology.orchestra.run.smithi007.stdout: "events": [ 2024-09-18T08:20:45.993 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:20:45.993 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "initiated", 2024-09-18T08:20:45.993 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:14:04.578571+0000", 2024-09-18T08:20:45.993 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-09-18T08:20:45.993 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:20:45.993 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:20:45.993 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "header_read", 2024-09-18T08:20:45.993 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:14:04.578571+0000", 2024-09-18T08:20:45.994 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-09-18T08:20:45.994 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:20:45.994 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:20:45.994 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "throttled", 2024-09-18T08:20:45.994 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:14:04.578571+0000", 2024-09-18T08:20:45.994 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 3.0600000000000001e-07 2024-09-18T08:20:45.994 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:20:45.994 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:20:45.994 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "all_read", 2024-09-18T08:20:45.994 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:14:04.578573+0000", 2024-09-18T08:20:45.994 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 2.0279999999999999e-06 2024-09-18T08:20:45.994 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:20:45.994 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:20:45.995 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "dispatched", 2024-09-18T08:20:45.995 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:14:04.578578+0000", 2024-09-18T08:20:45.995 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 4.6070000000000001e-06 2024-09-18T08:20:45.995 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:20:45.995 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:20:45.995 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "queued_for_pg", 2024-09-18T08:20:45.995 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:14:04.578582+0000", 2024-09-18T08:20:45.995 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 4.0770000000000001e-06 2024-09-18T08:20:45.995 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:20:45.995 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:20:45.995 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-09-18T08:20:45.995 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:14:04.578846+0000", 2024-09-18T08:20:45.995 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.00026379700000000002 2024-09-18T08:20:45.995 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:20:45.996 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:20:45.996 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "waiting for peered", 2024-09-18T08:20:45.996 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:14:04.578864+0000", 2024-09-18T08:20:45.996 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.8329e-05 2024-09-18T08:20:45.996 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:20:45.996 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:20:45.996 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-09-18T08:20:45.996 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:14:05.571675+0000", 2024-09-18T08:20:45.996 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.99281106600000002 2024-09-18T08:20:45.996 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:20:45.996 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:20:45.996 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "waiting for peered", 2024-09-18T08:20:45.996 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:14:05.571681+0000", 2024-09-18T08:20:45.997 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 6.4169999999999997e-06 2024-09-18T08:20:45.997 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:20:45.997 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:20:45.997 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-09-18T08:20:45.997 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:14:06.812113+0000", 2024-09-18T08:20:45.997 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.240431778 2024-09-18T08:20:45.997 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:20:45.997 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:20:45.997 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "done", 2024-09-18T08:20:45.997 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:14:06.812187+0000", 2024-09-18T08:20:45.997 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 7.3929000000000003e-05 2024-09-18T08:20:45.997 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-09-18T08:20:45.997 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-09-18T08:20:45.998 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-09-18T08:20:45.998 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:20:45.998 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:20:45.998 INFO:teuthology.orchestra.run.smithi007.stdout: "description": "osd_op(client.15582.0:252485 9.3 9:cf447bb3:::benchmark_data_smithi007_124019_object110600:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e292)", 2024-09-18T08:20:45.998 INFO:teuthology.orchestra.run.smithi007.stdout: "initiated_at": "2024-09-18T08:14:04.578588+0000", 2024-09-18T08:20:45.998 INFO:teuthology.orchestra.run.smithi007.stdout: "age": 401.36110282700002, 2024-09-18T08:20:45.998 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 2.2469260480000002, 2024-09-18T08:20:45.998 INFO:teuthology.orchestra.run.smithi007.stdout: "continuous": false, 2024-09-18T08:20:45.998 INFO:teuthology.orchestra.run.smithi007.stdout: "type_data": { 2024-09-18T08:20:45.998 INFO:teuthology.orchestra.run.smithi007.stdout: "flag_point": "reached pg", 2024-09-18T08:20:45.998 INFO:teuthology.orchestra.run.smithi007.stdout: "client_info": { 2024-09-18T08:20:45.998 INFO:teuthology.orchestra.run.smithi007.stdout: "client": "client.15582", 2024-09-18T08:20:45.998 INFO:teuthology.orchestra.run.smithi007.stdout: "client_addr": "172.21.15.7:0/3598605839", 2024-09-18T08:20:45.999 INFO:teuthology.orchestra.run.smithi007.stdout: "tid": 252485 2024-09-18T08:20:45.999 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:20:45.999 INFO:teuthology.orchestra.run.smithi007.stdout: "events": [ 2024-09-18T08:20:45.999 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:20:45.999 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "initiated", 2024-09-18T08:20:45.999 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:14:04.578588+0000", 2024-09-18T08:20:45.999 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-09-18T08:20:45.999 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:20:45.999 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:20:45.999 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "header_read", 2024-09-18T08:20:45.999 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:14:04.578588+0000", 2024-09-18T08:20:45.999 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-09-18T08:20:45.999 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:20:45.999 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:20:46.000 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "throttled", 2024-09-18T08:20:46.000 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:14:04.578588+0000", 2024-09-18T08:20:46.000 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 3.5999999999999999e-07 2024-09-18T08:20:46.000 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:20:46.000 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:20:46.000 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "all_read", 2024-09-18T08:20:46.000 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:14:04.578590+0000", 2024-09-18T08:20:46.000 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 2.1600000000000001e-06 2024-09-18T08:20:46.000 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:20:46.000 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:20:46.000 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "dispatched", 2024-09-18T08:20:46.000 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:14:04.578596+0000", 2024-09-18T08:20:46.000 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 5.48e-06 2024-09-18T08:20:46.000 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:20:46.001 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:20:46.001 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "queued_for_pg", 2024-09-18T08:20:46.001 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:14:04.578601+0000", 2024-09-18T08:20:46.001 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 5.1599999999999997e-06 2024-09-18T08:20:46.001 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:20:46.001 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:20:46.001 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-09-18T08:20:46.001 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:14:04.578909+0000", 2024-09-18T08:20:46.001 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.00030810599999999999 2024-09-18T08:20:46.001 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:20:46.001 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:20:46.001 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "waiting for peered", 2024-09-18T08:20:46.001 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:14:04.578915+0000", 2024-09-18T08:20:46.002 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 5.9190000000000004e-06 2024-09-18T08:20:46.002 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:20:46.002 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:20:46.002 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-09-18T08:20:46.002 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:14:05.571706+0000", 2024-09-18T08:20:46.002 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.99279081800000002 2024-09-18T08:20:46.002 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:20:46.002 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:20:46.002 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "waiting for peered", 2024-09-18T08:20:46.002 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:14:05.571712+0000", 2024-09-18T08:20:46.002 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 6.1770000000000003e-06 2024-09-18T08:20:46.002 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:20:46.002 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:20:46.003 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-09-18T08:20:46.003 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:14:06.812226+0000", 2024-09-18T08:20:46.003 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.2405141289999999 2024-09-18T08:20:46.003 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:20:46.003 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:20:46.003 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "waiting for missing object", 2024-09-18T08:20:46.003 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:14:06.812335+0000", 2024-09-18T08:20:46.003 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.00010895600000000001 2024-09-18T08:20:46.003 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:20:46.003 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:20:46.003 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-09-18T08:20:46.003 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:14:06.825411+0000", 2024-09-18T08:20:46.003 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.013075429 2024-09-18T08:20:46.004 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:20:46.004 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:20:46.004 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "done", 2024-09-18T08:20:46.004 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:14:06.825514+0000", 2024-09-18T08:20:46.004 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.000103354 2024-09-18T08:20:46.004 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-09-18T08:20:46.004 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-09-18T08:20:46.004 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-09-18T08:20:46.004 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:20:46.004 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:20:46.004 INFO:teuthology.orchestra.run.smithi007.stdout: "description": "osd_op(client.15582.0:252497 9.3 9:cca43518:::benchmark_data_smithi007_124019_object110612:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e292)", 2024-09-18T08:20:46.004 INFO:teuthology.orchestra.run.smithi007.stdout: "initiated_at": "2024-09-18T08:14:04.579093+0000", 2024-09-18T08:20:46.004 INFO:teuthology.orchestra.run.smithi007.stdout: "age": 401.36059806600002, 2024-09-18T08:20:46.005 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 2.2514827070000001, 2024-09-18T08:20:46.005 INFO:teuthology.orchestra.run.smithi007.stdout: "continuous": false, 2024-09-18T08:20:46.005 INFO:teuthology.orchestra.run.smithi007.stdout: "type_data": { 2024-09-18T08:20:46.005 INFO:teuthology.orchestra.run.smithi007.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T08:20:46.005 INFO:teuthology.orchestra.run.smithi007.stdout: "client_info": { 2024-09-18T08:20:46.005 INFO:teuthology.orchestra.run.smithi007.stdout: "client": "client.15582", 2024-09-18T08:20:46.005 INFO:teuthology.orchestra.run.smithi007.stdout: "client_addr": "172.21.15.7:0/3598605839", 2024-09-18T08:20:46.005 INFO:teuthology.orchestra.run.smithi007.stdout: "tid": 252497 2024-09-18T08:20:46.005 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:20:46.005 INFO:teuthology.orchestra.run.smithi007.stdout: "events": [ 2024-09-18T08:20:46.005 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:20:46.005 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "initiated", 2024-09-18T08:20:46.005 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:14:04.579093+0000", 2024-09-18T08:20:46.006 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-09-18T08:20:46.006 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:20:46.006 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:20:46.006 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "header_read", 2024-09-18T08:20:46.006 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:14:04.579093+0000", 2024-09-18T08:20:46.006 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-09-18T08:20:46.006 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:20:46.006 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:20:46.006 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "throttled", 2024-09-18T08:20:46.006 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:14:04.579094+0000", 2024-09-18T08:20:46.006 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 8.5600000000000004e-07 2024-09-18T08:20:46.006 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:20:46.006 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:20:46.007 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "all_read", 2024-09-18T08:20:46.007 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:14:04.579098+0000", 2024-09-18T08:20:46.007 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 4.143e-06 2024-09-18T08:20:46.007 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:20:46.007 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:20:46.007 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "dispatched", 2024-09-18T08:20:46.007 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:14:04.579105+0000", 2024-09-18T08:20:46.007 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 7.4309999999999997e-06 2024-09-18T08:20:46.007 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:20:46.007 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:20:46.007 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "queued_for_pg", 2024-09-18T08:20:46.007 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:14:04.579114+0000", 2024-09-18T08:20:46.007 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 8.6119999999999999e-06 2024-09-18T08:20:46.007 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:20:46.008 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:20:46.008 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-09-18T08:20:46.008 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:14:04.579160+0000", 2024-09-18T08:20:46.008 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 4.5977999999999999e-05 2024-09-18T08:20:46.008 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:20:46.008 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:20:46.008 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "waiting for peered", 2024-09-18T08:20:46.008 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:14:04.579167+0000", 2024-09-18T08:20:46.008 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 7.5000000000000002e-06 2024-09-18T08:20:46.008 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:20:46.008 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:20:46.008 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-09-18T08:20:46.008 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:14:05.571734+0000", 2024-09-18T08:20:46.009 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.99256691500000005 2024-09-18T08:20:46.009 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:20:46.009 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:20:46.009 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "waiting for peered", 2024-09-18T08:20:46.009 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:14:05.571740+0000", 2024-09-18T08:20:46.009 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 5.9410000000000004e-06 2024-09-18T08:20:46.009 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:20:46.009 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:20:46.009 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-09-18T08:20:46.009 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:14:06.814366+0000", 2024-09-18T08:20:46.009 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.242625654 2024-09-18T08:20:46.009 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:20:46.009 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:20:46.009 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "started", 2024-09-18T08:20:46.010 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:14:06.814471+0000", 2024-09-18T08:20:46.010 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.00010566 2024-09-18T08:20:46.010 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:20:46.010 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:20:46.010 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "waiting for subops from 5,7", 2024-09-18T08:20:46.010 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:14:06.814842+0000", 2024-09-18T08:20:46.010 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.00037091799999999997 2024-09-18T08:20:46.010 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:20:46.010 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:20:46.010 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "op_commit", 2024-09-18T08:20:46.010 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:14:06.825034+0000", 2024-09-18T08:20:46.010 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.010192084000000001 2024-09-18T08:20:46.010 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:20:46.010 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:20:46.011 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:20:46.011 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:14:06.828000+0000", 2024-09-18T08:20:46.011 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.0029653050000000001 2024-09-18T08:20:46.011 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:20:46.011 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:20:46.011 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:20:46.011 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:14:06.830366+0000", 2024-09-18T08:20:46.011 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.0023659689999999999 2024-09-18T08:20:46.011 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:20:46.011 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:20:46.011 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "commit_sent", 2024-09-18T08:20:46.011 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:14:06.830439+0000", 2024-09-18T08:20:46.011 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 7.3083000000000005e-05 2024-09-18T08:20:46.012 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:20:46.012 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:20:46.012 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "done", 2024-09-18T08:20:46.012 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:14:06.830575+0000", 2024-09-18T08:20:46.012 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.00013665800000000001 2024-09-18T08:20:46.012 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-09-18T08:20:46.012 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-09-18T08:20:46.012 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-09-18T08:20:46.012 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:20:46.012 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:20:46.012 INFO:teuthology.orchestra.run.smithi007.stdout: "description": "osd_op(client.15582.0:252500 9.c 9:3c726e3d:::benchmark_data_smithi007_124019_object110615:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e292)", 2024-09-18T08:20:46.012 INFO:teuthology.orchestra.run.smithi007.stdout: "initiated_at": "2024-09-18T08:14:04.592965+0000", 2024-09-18T08:20:46.013 INFO:teuthology.orchestra.run.smithi007.stdout: "age": 401.34672601300002, 2024-09-18T08:20:46.013 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 2.4642305090000001, 2024-09-18T08:20:46.013 INFO:teuthology.orchestra.run.smithi007.stdout: "continuous": false, 2024-09-18T08:20:46.013 INFO:teuthology.orchestra.run.smithi007.stdout: "type_data": { 2024-09-18T08:20:46.013 INFO:teuthology.orchestra.run.smithi007.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T08:20:46.013 INFO:teuthology.orchestra.run.smithi007.stdout: "client_info": { 2024-09-18T08:20:46.013 INFO:teuthology.orchestra.run.smithi007.stdout: "client": "client.15582", 2024-09-18T08:20:46.013 INFO:teuthology.orchestra.run.smithi007.stdout: "client_addr": "172.21.15.7:0/3598605839", 2024-09-18T08:20:46.013 INFO:teuthology.orchestra.run.smithi007.stdout: "tid": 252500 2024-09-18T08:20:46.013 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:20:46.013 INFO:teuthology.orchestra.run.smithi007.stdout: "events": [ 2024-09-18T08:20:46.013 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:20:46.013 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "initiated", 2024-09-18T08:20:46.014 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:14:04.592965+0000", 2024-09-18T08:20:46.014 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-09-18T08:20:46.014 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:20:46.014 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:20:46.014 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "header_read", 2024-09-18T08:20:46.014 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:14:04.592965+0000", 2024-09-18T08:20:46.014 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-09-18T08:20:46.014 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:20:46.014 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:20:46.014 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "throttled", 2024-09-18T08:20:46.014 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:14:04.592966+0000", 2024-09-18T08:20:46.014 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.6899999999999999e-06 2024-09-18T08:20:46.014 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:20:46.014 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:20:46.015 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "all_read", 2024-09-18T08:20:46.015 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:14:04.592973+0000", 2024-09-18T08:20:46.015 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 6.3600000000000001e-06 2024-09-18T08:20:46.015 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:20:46.015 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:20:46.015 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "dispatched", 2024-09-18T08:20:46.015 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:14:04.592985+0000", 2024-09-18T08:20:46.015 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.2500999999999999e-05 2024-09-18T08:20:46.015 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:20:46.015 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:20:46.015 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "queued_for_pg", 2024-09-18T08:20:46.015 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:14:04.592995+0000", 2024-09-18T08:20:46.015 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 9.5580000000000006e-06 2024-09-18T08:20:46.016 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:20:46.016 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:20:46.016 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-09-18T08:20:46.016 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:14:04.593104+0000", 2024-09-18T08:20:46.016 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.000108935 2024-09-18T08:20:46.016 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:20:46.016 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:20:46.016 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "waiting for peered", 2024-09-18T08:20:46.016 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:14:04.593116+0000", 2024-09-18T08:20:46.016 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.1891e-05 2024-09-18T08:20:46.016 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:20:46.016 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:20:46.016 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-09-18T08:20:46.016 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:14:05.582200+0000", 2024-09-18T08:20:46.017 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.98908412899999998 2024-09-18T08:20:46.017 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:20:46.017 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:20:46.017 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "waiting for peered", 2024-09-18T08:20:46.017 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:14:05.582209+0000", 2024-09-18T08:20:46.017 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 8.8710000000000003e-06 2024-09-18T08:20:46.017 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:20:46.017 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:20:46.017 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-09-18T08:20:46.017 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:14:06.804708+0000", 2024-09-18T08:20:46.017 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.2224989939999999 2024-09-18T08:20:46.017 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:20:46.017 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:20:46.017 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "started", 2024-09-18T08:20:46.018 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:14:06.804843+0000", 2024-09-18T08:20:46.018 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.00013566300000000001 2024-09-18T08:20:46.018 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:20:46.018 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:20:46.018 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "waiting for subops from 0,4,7", 2024-09-18T08:20:46.018 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:14:06.804991+0000", 2024-09-18T08:20:46.018 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.00014795800000000001 2024-09-18T08:20:46.018 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:20:46.018 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:20:46.018 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:20:46.018 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:14:06.808569+0000", 2024-09-18T08:20:46.018 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.0035780719999999998 2024-09-18T08:20:46.019 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:20:46.019 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:20:46.019 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:20:46.019 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:14:06.808595+0000", 2024-09-18T08:20:46.019 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 2.5677000000000002e-05 2024-09-18T08:20:46.019 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:20:46.019 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:20:46.019 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "op_commit", 2024-09-18T08:20:46.019 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:14:06.808602+0000", 2024-09-18T08:20:46.019 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 6.9430000000000003e-06 2024-09-18T08:20:46.019 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:20:46.019 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:20:46.020 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:20:46.020 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:14:07.057024+0000", 2024-09-18T08:20:46.020 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.24842249599999999 2024-09-18T08:20:46.020 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:20:46.020 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:20:46.020 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "commit_sent", 2024-09-18T08:20:46.020 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:14:07.057085+0000", 2024-09-18T08:20:46.020 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 6.0189999999999998e-05 2024-09-18T08:20:46.020 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:20:46.020 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:20:46.020 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "done", 2024-09-18T08:20:46.021 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:20:45 smithi007 ceph-mon[29459]: purged_snaps scrub starts 2024-09-18T08:20:46.021 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:20:45 smithi007 ceph-mon[29459]: purged_snaps scrub ok 2024-09-18T08:20:46.021 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:20:45 smithi007 ceph-mon[29459]: osdmap e445: 8 total, 8 up, 7 in 2024-09-18T08:20:46.021 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:20:45 smithi007 ceph-mon[39249]: purged_snaps scrub starts 2024-09-18T08:20:46.021 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:20:45 smithi007 ceph-mon[39249]: purged_snaps scrub ok 2024-09-18T08:20:46.021 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:20:45 smithi007 ceph-mon[39249]: osdmap e445: 8 total, 8 up, 7 in 2024-09-18T08:20:46.022 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:14:07.057195+0000", 2024-09-18T08:20:46.022 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.000110581 2024-09-18T08:20:46.022 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-09-18T08:20:46.022 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-09-18T08:20:46.022 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-09-18T08:20:46.023 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:20:46.023 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:20:46.023 INFO:teuthology.orchestra.run.smithi007.stdout: "description": "osd_op(client.15582.0:252501 9.3 9:c1421981:::benchmark_data_smithi007_124019_object110616:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e292)", 2024-09-18T08:20:46.023 INFO:teuthology.orchestra.run.smithi007.stdout: "initiated_at": "2024-09-18T08:14:04.593015+0000", 2024-09-18T08:20:46.023 INFO:teuthology.orchestra.run.smithi007.stdout: "age": 401.34667577200003, 2024-09-18T08:20:46.023 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 2.2377744819999998, 2024-09-18T08:20:46.023 INFO:teuthology.orchestra.run.smithi007.stdout: "continuous": false, 2024-09-18T08:20:46.023 INFO:teuthology.orchestra.run.smithi007.stdout: "type_data": { 2024-09-18T08:20:46.023 INFO:teuthology.orchestra.run.smithi007.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T08:20:46.023 INFO:teuthology.orchestra.run.smithi007.stdout: "client_info": { 2024-09-18T08:20:46.023 INFO:teuthology.orchestra.run.smithi007.stdout: "client": "client.15582", 2024-09-18T08:20:46.023 INFO:teuthology.orchestra.run.smithi007.stdout: "client_addr": "172.21.15.7:0/3598605839", 2024-09-18T08:20:46.023 INFO:teuthology.orchestra.run.smithi007.stdout: "tid": 252501 2024-09-18T08:20:46.024 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:20:46.024 INFO:teuthology.orchestra.run.smithi007.stdout: "events": [ 2024-09-18T08:20:46.024 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:20:46.024 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "initiated", 2024-09-18T08:20:46.024 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:14:04.593015+0000", 2024-09-18T08:20:46.024 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-09-18T08:20:46.024 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:20:46.024 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:20:46.024 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "header_read", 2024-09-18T08:20:46.024 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:14:04.593015+0000", 2024-09-18T08:20:46.024 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-09-18T08:20:46.024 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:20:46.024 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:20:46.025 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "throttled", 2024-09-18T08:20:46.025 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:14:04.593015+0000", 2024-09-18T08:20:46.025 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 5.2099999999999997e-07 2024-09-18T08:20:46.025 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:20:46.025 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:20:46.025 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "all_read", 2024-09-18T08:20:46.025 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:14:04.593018+0000", 2024-09-18T08:20:46.025 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 2.8820000000000001e-06 2024-09-18T08:20:46.025 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:20:46.025 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:20:46.025 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "dispatched", 2024-09-18T08:20:46.025 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:14:04.593025+0000", 2024-09-18T08:20:46.025 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 6.3589999999999996e-06 2024-09-18T08:20:46.025 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:20:46.026 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:20:46.026 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "queued_for_pg", 2024-09-18T08:20:46.026 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:14:04.593031+0000", 2024-09-18T08:20:46.026 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 5.8030000000000002e-06 2024-09-18T08:20:46.026 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:20:46.026 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:20:46.026 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-09-18T08:20:46.026 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:14:04.593087+0000", 2024-09-18T08:20:46.026 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 5.6628000000000001e-05 2024-09-18T08:20:46.026 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:20:46.026 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:20:46.026 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "waiting for peered", 2024-09-18T08:20:46.026 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:14:04.593097+0000", 2024-09-18T08:20:46.027 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 9.8209999999999997e-06 2024-09-18T08:20:46.027 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:20:46.027 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:20:46.027 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-09-18T08:20:46.027 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:14:05.571762+0000", 2024-09-18T08:20:46.027 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.97866520700000004 2024-09-18T08:20:46.027 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:20:46.027 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:20:46.027 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "waiting for peered", 2024-09-18T08:20:46.027 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:14:05.571768+0000", 2024-09-18T08:20:46.027 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 5.9710000000000001e-06 2024-09-18T08:20:46.027 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:20:46.027 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:20:46.027 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-09-18T08:20:46.028 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:14:06.820762+0000", 2024-09-18T08:20:46.028 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.248993587 2024-09-18T08:20:46.028 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:20:46.028 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:20:46.028 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "started", 2024-09-18T08:20:46.028 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:14:06.820911+0000", 2024-09-18T08:20:46.028 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.00014937800000000001 2024-09-18T08:20:46.028 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:20:46.028 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:20:46.028 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "waiting for subops from 5,7", 2024-09-18T08:20:46.028 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:14:06.821051+0000", 2024-09-18T08:20:46.028 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.00013987200000000001 2024-09-18T08:20:46.028 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:20:46.029 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:20:46.029 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "op_commit", 2024-09-18T08:20:46.029 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:14:06.825043+0000", 2024-09-18T08:20:46.029 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.0039923010000000002 2024-09-18T08:20:46.029 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:20:46.029 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:20:46.029 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:20:46.029 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:14:06.828045+0000", 2024-09-18T08:20:46.029 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.0030014629999999998 2024-09-18T08:20:46.029 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:20:46.029 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:20:46.029 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:20:46.029 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:14:06.830636+0000", 2024-09-18T08:20:46.029 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.0025915980000000001 2024-09-18T08:20:46.030 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:20:46.030 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:20:46.030 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "commit_sent", 2024-09-18T08:20:46.030 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:14:06.830687+0000", 2024-09-18T08:20:46.030 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 5.0912000000000003e-05 2024-09-18T08:20:46.030 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:20:46.030 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:20:46.030 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "done", 2024-09-18T08:20:46.030 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:14:06.830789+0000", 2024-09-18T08:20:46.030 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.000102179 2024-09-18T08:20:46.030 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-09-18T08:20:46.030 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-09-18T08:20:46.030 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-09-18T08:20:46.030 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:20:46.031 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:20:46.031 INFO:teuthology.orchestra.run.smithi007.stdout: "description": "osd_op(client.15966.0:9472 10.7 10:ebde9e8a:::benchmark_data_smithi007_148834_object9471:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e315)", 2024-09-18T08:20:46.031 INFO:teuthology.orchestra.run.smithi007.stdout: "initiated_at": "2024-09-18T08:14:51.634759+0000", 2024-09-18T08:20:46.031 INFO:teuthology.orchestra.run.smithi007.stdout: "age": 354.30493210399999, 2024-09-18T08:20:46.031 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 2.0688344779999999, 2024-09-18T08:20:46.031 INFO:teuthology.orchestra.run.smithi007.stdout: "continuous": false, 2024-09-18T08:20:46.031 INFO:teuthology.orchestra.run.smithi007.stdout: "type_data": { 2024-09-18T08:20:46.031 INFO:teuthology.orchestra.run.smithi007.stdout: "flag_point": "reached pg", 2024-09-18T08:20:46.031 INFO:teuthology.orchestra.run.smithi007.stdout: "client_info": { 2024-09-18T08:20:46.031 INFO:teuthology.orchestra.run.smithi007.stdout: "client": "client.15966", 2024-09-18T08:20:46.031 INFO:teuthology.orchestra.run.smithi007.stdout: "client_addr": "172.21.15.7:0/529784850", 2024-09-18T08:20:46.031 INFO:teuthology.orchestra.run.smithi007.stdout: "tid": 9472 2024-09-18T08:20:46.031 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:20:46.031 INFO:teuthology.orchestra.run.smithi007.stdout: "events": [ 2024-09-18T08:20:46.032 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:20:46.032 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "initiated", 2024-09-18T08:20:46.032 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:14:51.634759+0000", 2024-09-18T08:20:46.032 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-09-18T08:20:46.032 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:20:46.032 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:20:46.032 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "header_read", 2024-09-18T08:20:46.032 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:14:51.634759+0000", 2024-09-18T08:20:46.032 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-09-18T08:20:46.032 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:20:46.032 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:20:46.032 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "throttled", 2024-09-18T08:20:46.032 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:14:51.634760+0000", 2024-09-18T08:20:46.033 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.4959999999999999e-06 2024-09-18T08:20:46.033 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:20:46.033 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:20:46.033 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "all_read", 2024-09-18T08:20:46.033 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:14:51.634843+0000", 2024-09-18T08:20:46.033 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 8.3253999999999999e-05 2024-09-18T08:20:46.033 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:20:46.033 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:20:46.033 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "dispatched", 2024-09-18T08:20:46.033 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:14:51.634861+0000", 2024-09-18T08:20:46.033 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.7368999999999999e-05 2024-09-18T08:20:46.033 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:20:46.033 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:20:46.033 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "queued_for_pg", 2024-09-18T08:20:46.034 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:14:51.634869+0000", 2024-09-18T08:20:46.034 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 8.2470000000000008e-06 2024-09-18T08:20:46.034 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:20:46.034 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:20:46.034 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-09-18T08:20:46.034 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:14:51.635928+0000", 2024-09-18T08:20:46.034 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.001059321 2024-09-18T08:20:46.034 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:20:46.034 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:20:46.034 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "started", 2024-09-18T08:20:46.034 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:14:51.636070+0000", 2024-09-18T08:20:46.034 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.000141526 2024-09-18T08:20:46.034 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:20:46.035 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:20:46.035 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "waiting for subops from 5,6", 2024-09-18T08:20:46.035 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:14:51.636200+0000", 2024-09-18T08:20:46.035 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.00013038599999999999 2024-09-18T08:20:46.035 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:20:46.035 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:20:46.035 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "op_commit", 2024-09-18T08:20:46.035 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:14:51.638724+0000", 2024-09-18T08:20:46.035 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.0025236899999999999 2024-09-18T08:20:46.035 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:20:46.035 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:20:46.035 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-09-18T08:20:46.035 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:14:51.652898+0000", 2024-09-18T08:20:46.035 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.014173715999999999 2024-09-18T08:20:46.036 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:20:46.036 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:20:46.036 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "waiting for peered", 2024-09-18T08:20:46.036 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:14:51.652911+0000", 2024-09-18T08:20:46.036 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.3540000000000001e-05 2024-09-18T08:20:46.036 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:20:46.036 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:20:46.036 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-09-18T08:20:46.036 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:14:52.657538+0000", 2024-09-18T08:20:46.036 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.004626684 2024-09-18T08:20:46.036 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:20:46.036 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:20:46.036 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "waiting for peered", 2024-09-18T08:20:46.037 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:14:52.657552+0000", 2024-09-18T08:20:46.037 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.3854e-05 2024-09-18T08:20:46.037 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:20:46.037 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:20:46.037 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-09-18T08:20:46.037 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:14:53.703400+0000", 2024-09-18T08:20:46.037 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.0458485280000001 2024-09-18T08:20:46.037 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:20:46.037 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:20:46.037 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "done", 2024-09-18T08:20:46.037 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:14:53.703593+0000", 2024-09-18T08:20:46.037 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.000192867 2024-09-18T08:20:46.037 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-09-18T08:20:46.037 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-09-18T08:20:46.038 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-09-18T08:20:46.038 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:20:46.038 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:20:46.038 INFO:teuthology.orchestra.run.smithi007.stdout: "description": "osd_op(client.15966.0:9474 10.7 10:efdd08be:::benchmark_data_smithi007_148834_object9473:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e315)", 2024-09-18T08:20:46.038 INFO:teuthology.orchestra.run.smithi007.stdout: "initiated_at": "2024-09-18T08:14:51.634886+0000", 2024-09-18T08:20:46.038 INFO:teuthology.orchestra.run.smithi007.stdout: "age": 354.30480439500002, 2024-09-18T08:20:46.038 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 2.0695741889999999, 2024-09-18T08:20:46.038 INFO:teuthology.orchestra.run.smithi007.stdout: "continuous": false, 2024-09-18T08:20:46.038 INFO:teuthology.orchestra.run.smithi007.stdout: "type_data": { 2024-09-18T08:20:46.038 INFO:teuthology.orchestra.run.smithi007.stdout: "flag_point": "reached pg", 2024-09-18T08:20:46.038 INFO:teuthology.orchestra.run.smithi007.stdout: "client_info": { 2024-09-18T08:20:46.038 INFO:teuthology.orchestra.run.smithi007.stdout: "client": "client.15966", 2024-09-18T08:20:46.038 INFO:teuthology.orchestra.run.smithi007.stdout: "client_addr": "172.21.15.7:0/529784850", 2024-09-18T08:20:46.039 INFO:teuthology.orchestra.run.smithi007.stdout: "tid": 9474 2024-09-18T08:20:46.039 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:20:46.039 INFO:teuthology.orchestra.run.smithi007.stdout: "events": [ 2024-09-18T08:20:46.039 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:20:46.039 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "initiated", 2024-09-18T08:20:46.039 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:14:51.634886+0000", 2024-09-18T08:20:46.039 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-09-18T08:20:46.039 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:20:46.039 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:20:46.039 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "header_read", 2024-09-18T08:20:46.039 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:14:51.634886+0000", 2024-09-18T08:20:46.039 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-09-18T08:20:46.039 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:20:46.040 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:20:46.040 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "throttled", 2024-09-18T08:20:46.040 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:14:51.634887+0000", 2024-09-18T08:20:46.040 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 6.4600000000000004e-07 2024-09-18T08:20:46.040 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:20:46.040 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:20:46.040 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "all_read", 2024-09-18T08:20:46.040 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:14:51.634960+0000", 2024-09-18T08:20:46.040 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 7.3404000000000004e-05 2024-09-18T08:20:46.040 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:20:46.040 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:20:46.040 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "dispatched", 2024-09-18T08:20:46.040 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:14:51.634983+0000", 2024-09-18T08:20:46.041 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 2.2880000000000001e-05 2024-09-18T08:20:46.041 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:20:46.041 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:20:46.041 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "queued_for_pg", 2024-09-18T08:20:46.041 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:14:51.634990+0000", 2024-09-18T08:20:46.041 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 7.0859999999999996e-06 2024-09-18T08:20:46.041 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:20:46.041 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:20:46.041 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-09-18T08:20:46.041 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:14:51.637114+0000", 2024-09-18T08:20:46.041 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.0021231639999999999 2024-09-18T08:20:46.041 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:20:46.041 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:20:46.041 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "started", 2024-09-18T08:20:46.042 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:14:51.637239+0000", 2024-09-18T08:20:46.042 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.000125959 2024-09-18T08:20:46.042 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:20:46.042 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:20:46.042 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "waiting for subops from 5,6", 2024-09-18T08:20:46.042 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:14:51.637369+0000", 2024-09-18T08:20:46.042 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.00012985899999999999 2024-09-18T08:20:46.042 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:20:46.042 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:20:46.042 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "op_commit", 2024-09-18T08:20:46.042 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:14:51.639867+0000", 2024-09-18T08:20:46.042 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.0024973999999999999 2024-09-18T08:20:46.042 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:20:46.043 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:20:46.043 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-09-18T08:20:46.043 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:14:51.652947+0000", 2024-09-18T08:20:46.043 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.013080253999999999 2024-09-18T08:20:46.043 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:20:46.043 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:20:46.043 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "waiting for peered", 2024-09-18T08:20:46.043 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:14:51.652956+0000", 2024-09-18T08:20:46.043 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 9.1269999999999992e-06 2024-09-18T08:20:46.043 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:20:46.043 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:20:46.043 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-09-18T08:20:46.043 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:14:52.657686+0000", 2024-09-18T08:20:46.043 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.0047295039999999 2024-09-18T08:20:46.044 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:20:46.044 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:20:46.044 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "waiting for peered", 2024-09-18T08:20:46.044 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:14:52.657696+0000", 2024-09-18T08:20:46.044 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.0710000000000001e-05 2024-09-18T08:20:46.044 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:20:46.044 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:20:46.044 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-09-18T08:20:46.044 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:14:53.704057+0000", 2024-09-18T08:20:46.044 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.046360814 2024-09-18T08:20:46.044 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:20:46.044 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:20:46.044 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "done", 2024-09-18T08:20:46.044 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:14:53.704461+0000", 2024-09-18T08:20:46.045 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.00040338199999999998 2024-09-18T08:20:46.045 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-09-18T08:20:46.045 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-09-18T08:20:46.045 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-09-18T08:20:46.045 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:20:46.045 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:20:46.045 INFO:teuthology.orchestra.run.smithi007.stdout: "description": "osd_op(client.15966.0:9477 10.7 10:e773b142:::benchmark_data_smithi007_148834_object9476:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e315)", 2024-09-18T08:20:46.045 INFO:teuthology.orchestra.run.smithi007.stdout: "initiated_at": "2024-09-18T08:14:51.635151+0000", 2024-09-18T08:20:46.045 INFO:teuthology.orchestra.run.smithi007.stdout: "age": 354.30454011299997, 2024-09-18T08:20:46.045 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 2.0696481750000002, 2024-09-18T08:20:46.045 INFO:teuthology.orchestra.run.smithi007.stdout: "continuous": false, 2024-09-18T08:20:46.045 INFO:teuthology.orchestra.run.smithi007.stdout: "type_data": { 2024-09-18T08:20:46.045 INFO:teuthology.orchestra.run.smithi007.stdout: "flag_point": "reached pg", 2024-09-18T08:20:46.046 INFO:teuthology.orchestra.run.smithi007.stdout: "client_info": { 2024-09-18T08:20:46.046 INFO:teuthology.orchestra.run.smithi007.stdout: "client": "client.15966", 2024-09-18T08:20:46.046 INFO:teuthology.orchestra.run.smithi007.stdout: "client_addr": "172.21.15.7:0/529784850", 2024-09-18T08:20:46.046 INFO:teuthology.orchestra.run.smithi007.stdout: "tid": 9477 2024-09-18T08:20:46.046 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:20:46.046 INFO:teuthology.orchestra.run.smithi007.stdout: "events": [ 2024-09-18T08:20:46.046 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:20:46.046 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "initiated", 2024-09-18T08:20:46.046 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:14:51.635151+0000", 2024-09-18T08:20:46.046 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-09-18T08:20:46.046 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:20:46.046 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:20:46.046 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "header_read", 2024-09-18T08:20:46.047 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:14:51.635151+0000", 2024-09-18T08:20:46.047 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-09-18T08:20:46.047 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:20:46.047 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:20:46.047 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "throttled", 2024-09-18T08:20:46.047 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:14:51.635152+0000", 2024-09-18T08:20:46.047 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 9.4600000000000003e-07 2024-09-18T08:20:46.047 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:20:46.047 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:20:46.047 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "all_read", 2024-09-18T08:20:46.047 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:14:51.635174+0000", 2024-09-18T08:20:46.047 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 2.2035000000000002e-05 2024-09-18T08:20:46.047 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:20:46.048 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:20:46.048 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "dispatched", 2024-09-18T08:20:46.048 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:14:51.635187+0000", 2024-09-18T08:20:46.048 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.378e-05 2024-09-18T08:20:46.048 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:20:46.048 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:20:46.048 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "queued_for_pg", 2024-09-18T08:20:46.048 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:14:51.635195+0000", 2024-09-18T08:20:46.048 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 7.661e-06 2024-09-18T08:20:46.048 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:20:46.048 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:20:46.048 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-09-18T08:20:46.048 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:14:51.637873+0000", 2024-09-18T08:20:46.048 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.002677539 2024-09-18T08:20:46.049 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:20:46.049 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:20:46.049 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "started", 2024-09-18T08:20:46.049 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:14:51.637961+0000", 2024-09-18T08:20:46.049 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 8.8047000000000006e-05 2024-09-18T08:20:46.049 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:20:46.049 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:20:46.049 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "waiting for subops from 5,6", 2024-09-18T08:20:46.049 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:14:51.638041+0000", 2024-09-18T08:20:46.049 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 7.9957999999999996e-05 2024-09-18T08:20:46.049 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:20:46.049 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:20:46.049 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "op_commit", 2024-09-18T08:20:46.050 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:14:51.640125+0000", 2024-09-18T08:20:46.050 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.00208421 2024-09-18T08:20:46.050 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:20:46.050 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:20:46.050 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-09-18T08:20:46.050 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:14:51.652990+0000", 2024-09-18T08:20:46.050 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.01286488 2024-09-18T08:20:46.050 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:20:46.050 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:20:46.050 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "waiting for peered", 2024-09-18T08:20:46.050 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:14:51.652999+0000", 2024-09-18T08:20:46.050 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 9.1810000000000007e-06 2024-09-18T08:20:46.050 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:20:46.051 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:20:46.051 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-09-18T08:20:46.051 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:14:52.657733+0000", 2024-09-18T08:20:46.051 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.0047345940000001 2024-09-18T08:20:46.051 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:20:46.051 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:20:46.051 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "waiting for peered", 2024-09-18T08:20:46.051 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:14:52.657743+0000", 2024-09-18T08:20:46.051 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 9.0599999999999997e-06 2024-09-18T08:20:46.051 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:20:46.051 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:20:46.051 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-09-18T08:20:46.051 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:14:53.704501+0000", 2024-09-18T08:20:46.051 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.0467583199999999 2024-09-18T08:20:46.052 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:20:46.052 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:20:46.052 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "done", 2024-09-18T08:20:46.052 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:14:53.704799+0000", 2024-09-18T08:20:46.052 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.00029796400000000001 2024-09-18T08:20:46.052 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-09-18T08:20:46.052 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-09-18T08:20:46.052 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-09-18T08:20:46.052 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:20:46.052 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:20:46.052 INFO:teuthology.orchestra.run.smithi007.stdout: "description": "osd_op(client.26087.0:34015 11.b 11:d66b7346:::benchmark_data_smithi007_172546_object34014:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e408)", 2024-09-18T08:20:46.052 INFO:teuthology.orchestra.run.smithi007.stdout: "initiated_at": "2024-09-18T08:18:52.951272+0000", 2024-09-18T08:20:46.052 INFO:teuthology.orchestra.run.smithi007.stdout: "age": 112.988418819, 2024-09-18T08:20:46.053 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.22465576000000001, 2024-09-18T08:20:46.053 INFO:teuthology.orchestra.run.smithi007.stdout: "continuous": false, 2024-09-18T08:20:46.053 INFO:teuthology.orchestra.run.smithi007.stdout: "type_data": { 2024-09-18T08:20:46.053 INFO:teuthology.orchestra.run.smithi007.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T08:20:46.053 INFO:teuthology.orchestra.run.smithi007.stdout: "client_info": { 2024-09-18T08:20:46.053 INFO:teuthology.orchestra.run.smithi007.stdout: "client": "client.26087", 2024-09-18T08:20:46.053 INFO:teuthology.orchestra.run.smithi007.stdout: "client_addr": "172.21.15.7:0/2752273185", 2024-09-18T08:20:46.053 INFO:teuthology.orchestra.run.smithi007.stdout: "tid": 34015 2024-09-18T08:20:46.053 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:20:46.053 INFO:teuthology.orchestra.run.smithi007.stdout: "events": [ 2024-09-18T08:20:46.053 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:20:46.053 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "initiated", 2024-09-18T08:20:46.053 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:18:52.951272+0000", 2024-09-18T08:20:46.054 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-09-18T08:20:46.054 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:20:46.054 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:20:46.054 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "header_read", 2024-09-18T08:20:46.054 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:18:52.951272+0000", 2024-09-18T08:20:46.054 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-09-18T08:20:46.054 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:20:46.054 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:20:46.054 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "throttled", 2024-09-18T08:20:46.054 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:18:52.951273+0000", 2024-09-18T08:20:46.054 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 6.3799999999999997e-07 2024-09-18T08:20:46.054 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:20:46.054 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:20:46.054 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "all_read", 2024-09-18T08:20:46.055 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:18:52.951295+0000", 2024-09-18T08:20:46.055 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 2.2500000000000001e-05 2024-09-18T08:20:46.055 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:20:46.055 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:20:46.055 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "dispatched", 2024-09-18T08:20:46.055 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:18:52.951302+0000", 2024-09-18T08:20:46.055 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 6.7839999999999998e-06 2024-09-18T08:20:46.055 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:20:46.055 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:20:46.055 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "queued_for_pg", 2024-09-18T08:20:46.055 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:18:52.951307+0000", 2024-09-18T08:20:46.055 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 5.1889999999999998e-06 2024-09-18T08:20:46.055 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:20:46.056 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:20:46.056 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-09-18T08:20:46.056 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:18:52.951462+0000", 2024-09-18T08:20:46.056 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.00015530900000000001 2024-09-18T08:20:46.056 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:20:46.056 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:20:46.056 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "started", 2024-09-18T08:20:46.056 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:18:52.951744+0000", 2024-09-18T08:20:46.056 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.00028136399999999998 2024-09-18T08:20:46.056 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:20:46.056 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:20:46.056 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "waiting for subops from 0,1", 2024-09-18T08:20:46.056 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:18:52.951902+0000", 2024-09-18T08:20:46.056 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.00015864899999999999 2024-09-18T08:20:46.057 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:20:46.057 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:20:46.057 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "op_commit", 2024-09-18T08:20:46.057 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:18:52.953589+0000", 2024-09-18T08:20:46.057 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.001686599 2024-09-18T08:20:46.057 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:20:46.057 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:20:46.057 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:20:46.057 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:18:52.954228+0000", 2024-09-18T08:20:46.057 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.00063857400000000004 2024-09-18T08:20:46.057 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:20:46.057 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:20:46.057 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:20:46.058 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:18:53.175809+0000", 2024-09-18T08:20:46.058 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.22158180199999999 2024-09-18T08:20:46.058 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:20:46.058 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:20:46.058 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "commit_sent", 2024-09-18T08:20:46.058 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:18:53.175882+0000", 2024-09-18T08:20:46.058 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 7.2428999999999994e-05 2024-09-18T08:20:46.058 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:20:46.058 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:20:46.058 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "done", 2024-09-18T08:20:46.058 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:18:53.175928+0000", 2024-09-18T08:20:46.058 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 4.5923000000000001e-05 2024-09-18T08:20:46.058 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-09-18T08:20:46.058 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-09-18T08:20:46.059 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-09-18T08:20:46.059 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:20:46.059 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:20:46.059 INFO:teuthology.orchestra.run.smithi007.stdout: "description": "osd_op(client.26087.0:51039 11.2 11:487919b4:::benchmark_data_smithi007_172546_object51038:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e413)", 2024-09-18T08:20:46.059 INFO:teuthology.orchestra.run.smithi007.stdout: "initiated_at": "2024-09-18T08:19:04.813096+0000", 2024-09-18T08:20:46.059 INFO:teuthology.orchestra.run.smithi007.stdout: "age": 101.12659492900001, 2024-09-18T08:20:46.059 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 2.0968890889999998, 2024-09-18T08:20:46.059 INFO:teuthology.orchestra.run.smithi007.stdout: "continuous": false, 2024-09-18T08:20:46.059 INFO:teuthology.orchestra.run.smithi007.stdout: "type_data": { 2024-09-18T08:20:46.059 INFO:teuthology.orchestra.run.smithi007.stdout: "flag_point": "reached pg", 2024-09-18T08:20:46.059 INFO:teuthology.orchestra.run.smithi007.stdout: "client_info": { 2024-09-18T08:20:46.059 INFO:teuthology.orchestra.run.smithi007.stdout: "client": "client.26087", 2024-09-18T08:20:46.059 INFO:teuthology.orchestra.run.smithi007.stdout: "client_addr": "172.21.15.7:0/2752273185", 2024-09-18T08:20:46.059 INFO:teuthology.orchestra.run.smithi007.stdout: "tid": 51039 2024-09-18T08:20:46.060 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:20:46.060 INFO:teuthology.orchestra.run.smithi007.stdout: "events": [ 2024-09-18T08:20:46.060 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:20:46.060 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "initiated", 2024-09-18T08:20:46.060 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:19:04.813096+0000", 2024-09-18T08:20:46.060 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-09-18T08:20:46.060 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:20:46.060 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:20:46.060 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "header_read", 2024-09-18T08:20:46.060 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:19:04.813096+0000", 2024-09-18T08:20:46.060 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-09-18T08:20:46.060 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:20:46.060 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:20:46.061 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "throttled", 2024-09-18T08:20:46.061 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:19:04.813097+0000", 2024-09-18T08:20:46.061 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.2589999999999999e-06 2024-09-18T08:20:46.061 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:20:46.061 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:20:46.061 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "all_read", 2024-09-18T08:20:46.061 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:19:04.813134+0000", 2024-09-18T08:20:46.061 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 3.6720000000000001e-05 2024-09-18T08:20:46.061 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:20:46.061 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:20:46.061 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "dispatched", 2024-09-18T08:20:46.061 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:19:04.813146+0000", 2024-09-18T08:20:46.061 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.2121e-05 2024-09-18T08:20:46.062 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:20:46.062 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:20:46.062 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "queued_for_pg", 2024-09-18T08:20:46.062 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:19:04.813154+0000", 2024-09-18T08:20:46.062 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 8.1820000000000006e-06 2024-09-18T08:20:46.062 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:20:46.062 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:20:46.062 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-09-18T08:20:46.062 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:19:04.813911+0000", 2024-09-18T08:20:46.062 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.00075717700000000005 2024-09-18T08:20:46.062 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:20:46.062 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:20:46.062 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "op must wait for map", 2024-09-18T08:20:46.063 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:19:04.813917+0000", 2024-09-18T08:20:46.063 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 5.9390000000000002e-06 2024-09-18T08:20:46.063 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:20:46.063 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:20:46.063 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-09-18T08:20:46.063 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:19:04.833661+0000", 2024-09-18T08:20:46.063 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.019743825999999999 2024-09-18T08:20:46.063 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:20:46.063 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:20:46.063 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "waiting for peered", 2024-09-18T08:20:46.063 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:19:04.833670+0000", 2024-09-18T08:20:46.063 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 9.0189999999999995e-06 2024-09-18T08:20:46.063 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:20:46.063 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:20:46.063 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-09-18T08:20:46.064 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:19:05.888428+0000", 2024-09-18T08:20:46.064 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.054757731 2024-09-18T08:20:46.064 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:20:46.064 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:20:46.064 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "waiting for peered", 2024-09-18T08:20:46.064 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:19:05.888450+0000", 2024-09-18T08:20:46.064 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 2.1778999999999999e-05 2024-09-18T08:20:46.064 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:20:46.064 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:20:46.066 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-09-18T08:20:46.066 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:19:06.909058+0000", 2024-09-18T08:20:46.066 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.0206086889999999 2024-09-18T08:20:46.066 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:20:46.066 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:20:46.066 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "done", 2024-09-18T08:20:46.066 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:19:06.909985+0000", 2024-09-18T08:20:46.066 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.00092664699999999995 2024-09-18T08:20:46.066 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-09-18T08:20:46.066 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-09-18T08:20:46.066 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-09-18T08:20:46.066 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:20:46.066 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:20:46.067 INFO:teuthology.orchestra.run.smithi007.stdout: "description": "osd_op(client.26087.0:51057 11.2 11:4d3d066c:::benchmark_data_smithi007_172546_object51056:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e413)", 2024-09-18T08:20:46.067 INFO:teuthology.orchestra.run.smithi007.stdout: "initiated_at": "2024-09-18T08:19:04.816697+0000", 2024-09-18T08:20:46.067 INFO:teuthology.orchestra.run.smithi007.stdout: "age": 101.122993483, 2024-09-18T08:20:46.067 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 2.1005505769999999, 2024-09-18T08:20:46.067 INFO:teuthology.orchestra.run.smithi007.stdout: "continuous": false, 2024-09-18T08:20:46.067 INFO:teuthology.orchestra.run.smithi007.stdout: "type_data": { 2024-09-18T08:20:46.067 INFO:teuthology.orchestra.run.smithi007.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T08:20:46.067 INFO:teuthology.orchestra.run.smithi007.stdout: "client_info": { 2024-09-18T08:20:46.067 INFO:teuthology.orchestra.run.smithi007.stdout: "client": "client.26087", 2024-09-18T08:20:46.067 INFO:teuthology.orchestra.run.smithi007.stdout: "client_addr": "172.21.15.7:0/2752273185", 2024-09-18T08:20:46.067 INFO:teuthology.orchestra.run.smithi007.stdout: "tid": 51057 2024-09-18T08:20:46.067 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:20:46.067 INFO:teuthology.orchestra.run.smithi007.stdout: "events": [ 2024-09-18T08:20:46.068 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:20:46.068 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "initiated", 2024-09-18T08:20:46.068 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:19:04.816697+0000", 2024-09-18T08:20:46.068 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-09-18T08:20:46.068 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:20:46.068 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:20:46.068 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "header_read", 2024-09-18T08:20:46.068 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:19:04.816697+0000", 2024-09-18T08:20:46.068 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-09-18T08:20:46.068 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:20:46.068 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:20:46.068 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "throttled", 2024-09-18T08:20:46.068 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:19:04.816698+0000", 2024-09-18T08:20:46.069 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 6.7100000000000001e-07 2024-09-18T08:20:46.069 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:20:46.069 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:20:46.069 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "all_read", 2024-09-18T08:20:46.069 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:19:04.816722+0000", 2024-09-18T08:20:46.069 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 2.4358999999999998e-05 2024-09-18T08:20:46.069 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:20:46.069 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:20:46.069 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "dispatched", 2024-09-18T08:20:46.069 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:19:04.816730+0000", 2024-09-18T08:20:46.069 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 7.9959999999999993e-06 2024-09-18T08:20:46.069 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:20:46.069 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:20:46.069 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "queued_for_pg", 2024-09-18T08:20:46.070 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:19:04.816736+0000", 2024-09-18T08:20:46.070 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 5.8050000000000003e-06 2024-09-18T08:20:46.070 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:20:46.070 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:20:46.070 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-09-18T08:20:46.070 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:19:04.817857+0000", 2024-09-18T08:20:46.070 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.001120921 2024-09-18T08:20:46.070 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:20:46.070 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:20:46.070 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "waiting_for_map not empty", 2024-09-18T08:20:46.070 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:19:04.817863+0000", 2024-09-18T08:20:46.070 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 6.0789999999999997e-06 2024-09-18T08:20:46.070 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:20:46.071 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:20:46.071 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-09-18T08:20:46.071 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:19:04.833693+0000", 2024-09-18T08:20:46.071 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.015829882999999999 2024-09-18T08:20:46.071 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:20:46.071 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:20:46.071 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "waiting for peered", 2024-09-18T08:20:46.071 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:19:04.833700+0000", 2024-09-18T08:20:46.071 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 7.1409999999999999e-06 2024-09-18T08:20:46.071 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:20:46.071 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:20:46.071 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-09-18T08:20:46.071 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:19:05.891316+0000", 2024-09-18T08:20:46.071 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.0576157820000001 2024-09-18T08:20:46.072 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:20:46.072 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:20:46.072 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "waiting for peered", 2024-09-18T08:20:46.072 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:19:05.891333+0000", 2024-09-18T08:20:46.072 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.6969e-05 2024-09-18T08:20:46.072 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:20:46.072 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:20:46.072 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-09-18T08:20:46.072 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:19:06.911113+0000", 2024-09-18T08:20:46.072 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.019779773 2024-09-18T08:20:46.072 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:20:46.072 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:20:46.072 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "started", 2024-09-18T08:20:46.073 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:19:06.911521+0000", 2024-09-18T08:20:46.073 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.00040830899999999998 2024-09-18T08:20:46.073 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:20:46.073 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:20:46.073 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "waiting for subops from 0,7", 2024-09-18T08:20:46.073 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:19:06.911630+0000", 2024-09-18T08:20:46.073 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.00010891 2024-09-18T08:20:46.073 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:20:46.073 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:20:46.073 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "op_commit", 2024-09-18T08:20:46.073 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:19:06.914842+0000", 2024-09-18T08:20:46.073 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.003212159 2024-09-18T08:20:46.073 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:20:46.074 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:20:46.074 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:20:46.074 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:19:06.915379+0000", 2024-09-18T08:20:46.074 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.00053746899999999995 2024-09-18T08:20:46.074 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:20:46.074 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:20:46.074 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:20:46.074 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:19:06.915901+0000", 2024-09-18T08:20:46.074 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.00052167999999999995 2024-09-18T08:20:46.074 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:20:46.074 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:20:46.074 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "commit_sent", 2024-09-18T08:20:46.074 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:19:06.915978+0000", 2024-09-18T08:20:46.075 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 7.6556000000000004e-05 2024-09-18T08:20:46.075 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:20:46.075 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:20:46.075 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "done", 2024-09-18T08:20:46.075 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:19:06.917248+0000", 2024-09-18T08:20:46.075 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.001270115 2024-09-18T08:20:46.075 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-09-18T08:20:46.075 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-09-18T08:20:46.075 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-09-18T08:20:46.075 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:20:46.075 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:20:46.075 INFO:teuthology.orchestra.run.smithi007.stdout: "description": "osd_op(client.26087.0:51080 11.2 11:46b8fbaf:::benchmark_data_smithi007_172546_object51079:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e413)", 2024-09-18T08:20:46.076 INFO:teuthology.orchestra.run.smithi007.stdout: "initiated_at": "2024-09-18T08:19:04.853834+0000", 2024-09-18T08:20:46.076 INFO:teuthology.orchestra.run.smithi007.stdout: "age": 101.08585658299999, 2024-09-18T08:20:46.076 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 2.0669147350000001, 2024-09-18T08:20:46.076 INFO:teuthology.orchestra.run.smithi007.stdout: "continuous": false, 2024-09-18T08:20:46.076 INFO:teuthology.orchestra.run.smithi007.stdout: "type_data": { 2024-09-18T08:20:46.076 INFO:teuthology.orchestra.run.smithi007.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T08:20:46.076 INFO:teuthology.orchestra.run.smithi007.stdout: "client_info": { 2024-09-18T08:20:46.076 INFO:teuthology.orchestra.run.smithi007.stdout: "client": "client.26087", 2024-09-18T08:20:46.076 INFO:teuthology.orchestra.run.smithi007.stdout: "client_addr": "172.21.15.7:0/2752273185", 2024-09-18T08:20:46.076 INFO:teuthology.orchestra.run.smithi007.stdout: "tid": 51080 2024-09-18T08:20:46.076 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:20:46.076 INFO:teuthology.orchestra.run.smithi007.stdout: "events": [ 2024-09-18T08:20:46.077 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:20:46.077 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "initiated", 2024-09-18T08:20:46.077 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:19:04.853834+0000", 2024-09-18T08:20:46.077 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-09-18T08:20:46.077 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:20:46.077 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:20:46.077 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "header_read", 2024-09-18T08:20:46.077 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:19:04.853834+0000", 2024-09-18T08:20:46.077 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-09-18T08:20:46.077 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:20:46.077 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:20:46.077 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "throttled", 2024-09-18T08:20:46.077 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:19:04.853836+0000", 2024-09-18T08:20:46.077 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.406e-06 2024-09-18T08:20:46.078 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:20:46.078 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:20:46.078 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "all_read", 2024-09-18T08:20:46.078 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:19:04.853868+0000", 2024-09-18T08:20:46.078 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 3.2203999999999999e-05 2024-09-18T08:20:46.078 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:20:46.078 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:20:46.078 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "dispatched", 2024-09-18T08:20:46.078 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:19:04.853882+0000", 2024-09-18T08:20:46.078 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.4705999999999999e-05 2024-09-18T08:20:46.078 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:20:46.078 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:20:46.078 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "queued_for_pg", 2024-09-18T08:20:46.078 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:19:04.853892+0000", 2024-09-18T08:20:46.079 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 9.3349999999999996e-06 2024-09-18T08:20:46.079 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:20:46.079 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:20:46.079 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-09-18T08:20:46.079 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:19:04.853967+0000", 2024-09-18T08:20:46.079 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 7.5197999999999994e-05 2024-09-18T08:20:46.079 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:20:46.079 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:20:46.079 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "waiting for peered", 2024-09-18T08:20:46.079 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:19:04.853981+0000", 2024-09-18T08:20:46.079 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.4403000000000001e-05 2024-09-18T08:20:46.079 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:20:46.079 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:20:46.079 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-09-18T08:20:46.080 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:19:05.891394+0000", 2024-09-18T08:20:46.080 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.0374128410000001 2024-09-18T08:20:46.080 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:20:46.080 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:20:46.080 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "waiting for peered", 2024-09-18T08:20:46.080 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:19:05.891403+0000", 2024-09-18T08:20:46.080 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 8.9469999999999992e-06 2024-09-18T08:20:46.080 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:20:46.080 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:20:46.080 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-09-18T08:20:46.080 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:19:06.913175+0000", 2024-09-18T08:20:46.080 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.021771392 2024-09-18T08:20:46.081 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:20:46.081 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:20:46.081 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "started", 2024-09-18T08:20:46.081 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:19:06.913341+0000", 2024-09-18T08:20:46.081 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.0001661 2024-09-18T08:20:46.081 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:20:46.081 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:20:46.081 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "waiting for subops from 0,7", 2024-09-18T08:20:46.081 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:19:06.913446+0000", 2024-09-18T08:20:46.081 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.00010507500000000001 2024-09-18T08:20:46.081 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:20:46.081 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:20:46.081 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "op_commit", 2024-09-18T08:20:46.082 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:19:06.914936+0000", 2024-09-18T08:20:46.082 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.0014900880000000001 2024-09-18T08:20:46.082 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:20:46.082 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:20:46.082 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:20:46.082 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:19:06.917338+0000", 2024-09-18T08:20:46.082 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.0024021120000000001 2024-09-18T08:20:46.082 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:20:46.082 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:20:46.082 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:20:46.082 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:19:06.919473+0000", 2024-09-18T08:20:46.082 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.0021348259999999998 2024-09-18T08:20:46.082 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:20:46.083 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:20:46.083 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "commit_sent", 2024-09-18T08:20:46.083 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:19:06.919514+0000", 2024-09-18T08:20:46.083 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 4.0890999999999999e-05 2024-09-18T08:20:46.083 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:20:46.083 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:20:46.083 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "done", 2024-09-18T08:20:46.083 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:19:06.920749+0000", 2024-09-18T08:20:46.083 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.0012352109999999999 2024-09-18T08:20:46.083 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-09-18T08:20:46.083 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-09-18T08:20:46.083 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-09-18T08:20:46.084 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:20:46.084 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:20:46.084 INFO:teuthology.orchestra.run.smithi007.stdout: "description": "osd_op(client.26087.0:55150 11.9 11:96286410:::benchmark_data_smithi007_172546_object55149:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e417)", 2024-09-18T08:20:46.084 INFO:teuthology.orchestra.run.smithi007.stdout: "initiated_at": "2024-09-18T08:19:11.863154+0000", 2024-09-18T08:20:46.084 INFO:teuthology.orchestra.run.smithi007.stdout: "age": 94.076536383999994, 2024-09-18T08:20:46.084 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.20988754700000001, 2024-09-18T08:20:46.084 INFO:teuthology.orchestra.run.smithi007.stdout: "continuous": false, 2024-09-18T08:20:46.084 INFO:teuthology.orchestra.run.smithi007.stdout: "type_data": { 2024-09-18T08:20:46.084 INFO:teuthology.orchestra.run.smithi007.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T08:20:46.084 INFO:teuthology.orchestra.run.smithi007.stdout: "client_info": { 2024-09-18T08:20:46.084 INFO:teuthology.orchestra.run.smithi007.stdout: "client": "client.26087", 2024-09-18T08:20:46.084 INFO:teuthology.orchestra.run.smithi007.stdout: "client_addr": "172.21.15.7:0/2752273185", 2024-09-18T08:20:46.084 INFO:teuthology.orchestra.run.smithi007.stdout: "tid": 55150 2024-09-18T08:20:46.085 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:20:46.085 INFO:teuthology.orchestra.run.smithi007.stdout: "events": [ 2024-09-18T08:20:46.085 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:20:46.085 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "initiated", 2024-09-18T08:20:46.085 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:19:11.863154+0000", 2024-09-18T08:20:46.085 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-09-18T08:20:46.085 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:20:46.085 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:20:46.085 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "header_read", 2024-09-18T08:20:46.085 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:19:11.863154+0000", 2024-09-18T08:20:46.085 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-09-18T08:20:46.085 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:20:46.085 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:20:46.086 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "throttled", 2024-09-18T08:20:46.086 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:19:11.863156+0000", 2024-09-18T08:20:46.086 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.6139999999999999e-06 2024-09-18T08:20:46.086 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:20:46.086 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:20:46.086 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "all_read", 2024-09-18T08:20:46.086 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:19:11.863209+0000", 2024-09-18T08:20:46.086 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 5.3130999999999997e-05 2024-09-18T08:20:46.086 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:20:46.086 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:20:46.086 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "dispatched", 2024-09-18T08:20:46.086 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:19:12.050900+0000", 2024-09-18T08:20:46.086 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.18769058299999999 2024-09-18T08:20:46.087 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:20:46.087 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:20:46.087 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "queued_for_pg", 2024-09-18T08:20:46.087 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:19:12.050919+0000", 2024-09-18T08:20:46.087 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.9626000000000001e-05 2024-09-18T08:20:46.087 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:20:46.087 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:20:46.087 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-09-18T08:20:46.087 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:19:12.052048+0000", 2024-09-18T08:20:46.087 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.001128937 2024-09-18T08:20:46.087 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:20:46.087 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:20:46.087 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "started", 2024-09-18T08:20:46.088 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:19:12.052196+0000", 2024-09-18T08:20:46.088 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.000147317 2024-09-18T08:20:46.088 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:20:46.088 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:20:46.088 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "waiting for subops from 0,6", 2024-09-18T08:20:46.088 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:19:12.052324+0000", 2024-09-18T08:20:46.088 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.000128355 2024-09-18T08:20:46.088 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:20:46.088 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:20:46.088 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:20:46.088 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:19:12.060760+0000", 2024-09-18T08:20:46.088 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.0084360979999999995 2024-09-18T08:20:46.088 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:20:46.088 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:20:46.089 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "op_commit", 2024-09-18T08:20:46.089 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:19:12.060771+0000", 2024-09-18T08:20:46.089 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.0805000000000001e-05 2024-09-18T08:20:46.089 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:20:46.089 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:20:46.089 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:20:46.089 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:19:12.072697+0000", 2024-09-18T08:20:46.089 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.011926475000000001 2024-09-18T08:20:46.089 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:20:46.089 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:20:46.089 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "commit_sent", 2024-09-18T08:20:46.089 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:19:12.073000+0000", 2024-09-18T08:20:46.089 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.00030227699999999999 2024-09-18T08:20:46.090 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:20:46.090 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:20:46.090 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "done", 2024-09-18T08:20:46.090 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:19:12.073042+0000", 2024-09-18T08:20:46.090 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 4.2329000000000001e-05 2024-09-18T08:20:46.090 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-09-18T08:20:46.090 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-09-18T08:20:46.090 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-09-18T08:20:46.090 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:20:46.090 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:20:46.090 INFO:teuthology.orchestra.run.smithi007.stdout: "description": "osd_op(client.26087.0:61003 11.1 11:8d81ffa6:::benchmark_data_smithi007_172546_object61002:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e418)", 2024-09-18T08:20:46.090 INFO:teuthology.orchestra.run.smithi007.stdout: "initiated_at": "2024-09-18T08:19:19.493534+0000", 2024-09-18T08:20:46.090 INFO:teuthology.orchestra.run.smithi007.stdout: "age": 86.446156955999996, 2024-09-18T08:20:46.091 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.20259713800000001, 2024-09-18T08:20:46.091 INFO:teuthology.orchestra.run.smithi007.stdout: "continuous": false, 2024-09-18T08:20:46.091 INFO:teuthology.orchestra.run.smithi007.stdout: "type_data": { 2024-09-18T08:20:46.091 INFO:teuthology.orchestra.run.smithi007.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T08:20:46.091 INFO:teuthology.orchestra.run.smithi007.stdout: "client_info": { 2024-09-18T08:20:46.091 INFO:teuthology.orchestra.run.smithi007.stdout: "client": "client.26087", 2024-09-18T08:20:46.091 INFO:teuthology.orchestra.run.smithi007.stdout: "client_addr": "172.21.15.7:0/2752273185", 2024-09-18T08:20:46.091 INFO:teuthology.orchestra.run.smithi007.stdout: "tid": 61003 2024-09-18T08:20:46.091 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:20:46.091 INFO:teuthology.orchestra.run.smithi007.stdout: "events": [ 2024-09-18T08:20:46.091 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:20:46.091 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "initiated", 2024-09-18T08:20:46.091 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:19:19.493534+0000", 2024-09-18T08:20:46.091 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-09-18T08:20:46.092 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:20:46.092 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:20:46.092 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "header_read", 2024-09-18T08:20:46.092 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:19:19.493534+0000", 2024-09-18T08:20:46.092 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0 2024-09-18T08:20:46.092 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:20:46.092 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:20:46.092 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "throttled", 2024-09-18T08:20:46.092 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:19:19.493535+0000", 2024-09-18T08:20:46.092 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.077e-06 2024-09-18T08:20:46.092 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:20:46.092 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:20:46.092 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "all_read", 2024-09-18T08:20:46.093 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:19:19.493589+0000", 2024-09-18T08:20:46.093 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 5.4119000000000001e-05 2024-09-18T08:20:46.093 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:20:46.093 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:20:46.093 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "dispatched", 2024-09-18T08:20:46.093 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:19:19.493662+0000", 2024-09-18T08:20:46.093 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 7.2760000000000001e-05 2024-09-18T08:20:46.093 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:20:46.093 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:20:46.093 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "queued_for_pg", 2024-09-18T08:20:46.093 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:19:19.493672+0000", 2024-09-18T08:20:46.093 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 1.0305999999999999e-05 2024-09-18T08:20:46.093 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:20:46.094 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:20:46.094 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "reached_pg", 2024-09-18T08:20:46.094 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:19:19.493763+0000", 2024-09-18T08:20:46.094 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 9.0833000000000003e-05 2024-09-18T08:20:46.094 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:20:46.094 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:20:46.094 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "started", 2024-09-18T08:20:46.094 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:19:19.493896+0000", 2024-09-18T08:20:46.094 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.00013271800000000001 2024-09-18T08:20:46.094 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:20:46.094 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:20:46.094 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "waiting for subops from 0,6", 2024-09-18T08:20:46.094 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:19:19.494143+0000", 2024-09-18T08:20:46.095 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.000247097 2024-09-18T08:20:46.095 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:20:46.095 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:20:46.095 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "op_commit", 2024-09-18T08:20:46.095 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:19:19.496411+0000", 2024-09-18T08:20:46.095 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.00226859 2024-09-18T08:20:46.095 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:20:46.095 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:20:46.095 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:20:46.095 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:19:19.496564+0000", 2024-09-18T08:20:46.095 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.000152206 2024-09-18T08:20:46.095 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:20:46.095 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:20:46.096 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:20:46.096 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:19:19.696033+0000", 2024-09-18T08:20:46.096 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 0.199469269 2024-09-18T08:20:46.096 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:20:46.096 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:20:46.096 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "commit_sent", 2024-09-18T08:20:46.096 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:19:19.696090+0000", 2024-09-18T08:20:46.096 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 5.7367000000000001e-05 2024-09-18T08:20:46.096 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-09-18T08:20:46.096 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-09-18T08:20:46.096 INFO:teuthology.orchestra.run.smithi007.stdout: "event": "done", 2024-09-18T08:20:46.096 INFO:teuthology.orchestra.run.smithi007.stdout: "time": "2024-09-18T08:19:19.696131+0000", 2024-09-18T08:20:46.096 INFO:teuthology.orchestra.run.smithi007.stdout: "duration": 4.0796e-05 2024-09-18T08:20:46.097 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-09-18T08:20:46.097 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-09-18T08:20:46.097 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-09-18T08:20:46.097 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-09-18T08:20:46.097 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-09-18T08:20:46.097 INFO:teuthology.orchestra.run.smithi007.stdout:} 2024-09-18T08:20:46.098 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:20:46.098 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:20:46.098 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0' 2024-09-18T08:20:46.155 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:20:45 smithi050 ceph-mon[32371]: purged_snaps scrub starts 2024-09-18T08:20:46.155 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:20:45 smithi050 ceph-mon[32371]: purged_snaps scrub ok 2024-09-18T08:20:46.155 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:20:45 smithi050 ceph-mon[32371]: osdmap e445: 8 total, 8 up, 7 in 2024-09-18T08:20:46.200 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7 2024-09-18T08:20:46.227 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:20:46.228 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:20:46.228 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7' 2024-09-18T08:20:46.329 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6 2024-09-18T08:20:46.357 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:20:46.358 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:20:46.358 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6' 2024-09-18T08:20:46.459 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7 2024-09-18T08:20:46.497 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:20:46.498 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:20:46.498 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7' 2024-09-18T08:20:46.599 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2 2024-09-18T08:20:46.631 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:20:46.632 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:20:46.632 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2' 2024-09-18T08:20:46.733 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3 2024-09-18T08:20:46.770 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:20:46.770 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:20:46.771 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3' 2024-09-18T08:20:46.872 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4 2024-09-18T08:20:46.893 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:20:46.894 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:20:46.894 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4' 2024-09-18T08:20:46.979 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Sep 18 08:20:46 smithi007 ceph-779349ca-7593-11ef-bceb-c7b262605968-mgr-y[29688]: ::ffff:172.21.15.50 - - [18/Sep/2024:08:20:46] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-09-18T08:20:46.980 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:20:46 smithi007 ceph-mon[29459]: pgmap v899: 158 pgs: 158 active+clean; 3.1 GiB data, 14 GiB used, 612 GiB / 626 GiB avail; 853 B/s rd, 0 B/s wr, 1.65k op/s 2024-09-18T08:20:46.980 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:20:46 smithi007 ceph-mon[39249]: pgmap v899: 158 pgs: 158 active+clean; 3.1 GiB data, 14 GiB used, 612 GiB / 626 GiB avail; 853 B/s rd, 0 B/s wr, 1.65k op/s 2024-09-18T08:20:46.995 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3 2024-09-18T08:20:47.031 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:20:47.032 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:20:47.032 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3' 2024-09-18T08:20:47.133 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2 2024-09-18T08:20:47.154 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:20:46 smithi050 ceph-mon[32371]: pgmap v899: 158 pgs: 158 active+clean; 3.1 GiB data, 14 GiB used, 612 GiB / 626 GiB avail; 853 B/s rd, 0 B/s wr, 1.65k op/s 2024-09-18T08:20:47.171 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:20:47.172 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:20:47.172 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2' 2024-09-18T08:20:47.195 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid 779349ca-7593-11ef-bceb-c7b262605968 -- ceph osd set nodeep-scrub 2024-09-18T08:20:47.273 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1 2024-09-18T08:20:47.317 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:20:47.318 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:20:47.318 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1' 2024-09-18T08:20:47.419 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3 2024-09-18T08:20:47.461 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:20:47.462 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:20:47.462 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3' 2024-09-18T08:20:47.563 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3 2024-09-18T08:20:47.605 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:20:47.606 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:20:47.606 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3' 2024-09-18T08:20:47.707 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3 2024-09-18T08:20:47.757 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:20:47.758 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:20:47.758 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3' 2024-09-18T08:20:47.859 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0 2024-09-18T08:20:47.881 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:20:47.882 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:20:47.882 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0' 2024-09-18T08:20:47.983 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4 2024-09-18T08:20:48.010 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:20:48.011 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:20:48.012 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4' 2024-09-18T08:20:48.112 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2 2024-09-18T08:20:48.143 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:20:48.143 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:20:48.144 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2' 2024-09-18T08:20:48.245 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0 2024-09-18T08:20:48.283 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:20:48.283 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:20:48.284 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0' 2024-09-18T08:20:48.385 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2 2024-09-18T08:20:48.424 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:20:48.425 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:20:48.425 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2' 2024-09-18T08:20:48.526 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1 2024-09-18T08:20:48.563 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:20:48.563 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:20:48.564 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1' 2024-09-18T08:20:48.665 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6 2024-09-18T08:20:48.708 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:20:48.709 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:20:48.709 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6' 2024-09-18T08:20:48.810 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2 2024-09-18T08:20:48.837 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:20:48.838 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:20:48.839 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2' 2024-09-18T08:20:48.939 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5 2024-09-18T08:20:48.974 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:20:48.975 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:20:48.975 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5' 2024-09-18T08:20:49.076 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4 2024-09-18T08:20:49.099 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:20:49.099 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:20:49.099 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4' 2024-09-18T08:20:49.155 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:20:48 smithi050 ceph-mon[32371]: pgmap v900: 158 pgs: 158 active+clean; 3.1 GiB data, 14 GiB used, 612 GiB / 626 GiB avail; 749 B/s rd, 0 B/s wr, 1.05k op/s 2024-09-18T08:20:49.201 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1 2024-09-18T08:20:49.230 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:20:48 smithi007 ceph-mon[29459]: pgmap v900: 158 pgs: 158 active+clean; 3.1 GiB data, 14 GiB used, 612 GiB / 626 GiB avail; 749 B/s rd, 0 B/s wr, 1.05k op/s 2024-09-18T08:20:49.230 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:20:48 smithi007 ceph-mon[39249]: pgmap v900: 158 pgs: 158 active+clean; 3.1 GiB data, 14 GiB used, 612 GiB / 626 GiB avail; 749 B/s rd, 0 B/s wr, 1.05k op/s 2024-09-18T08:20:49.237 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:20:49.238 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:20:49.239 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1' 2024-09-18T08:20:49.339 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1 2024-09-18T08:20:49.374 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:20:49.374 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:20:49.375 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1' 2024-09-18T08:20:49.475 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4 2024-09-18T08:20:49.505 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:20:49.506 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:20:49.506 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4' 2024-09-18T08:20:49.607 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2 2024-09-18T08:20:49.644 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:20:49.644 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:20:49.645 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2' 2024-09-18T08:20:49.652 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 6, 5, 7] out_osds: [4] dead_osds: [] live_osds: [0, 2, 3, 4, 5, 6, 7, 1] 2024-09-18T08:20:49.652 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-18T08:20:49.653 INFO:tasks.thrashosds.thrasher:fixing pg num pool .mgr 2024-09-18T08:20:49.653 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid 779349ca-7593-11ef-bceb-c7b262605968 -- ceph pg dump --format=json 2024-09-18T08:20:49.745 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5 2024-09-18T08:20:49.778 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:20:49.779 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:20:49.779 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5' 2024-09-18T08:20:49.880 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1 2024-09-18T08:20:49.923 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:20:49.924 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:20:49.924 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1' 2024-09-18T08:20:50.025 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0 2024-09-18T08:20:50.069 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:20:50.070 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:20:50.071 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0' 2024-09-18T08:20:50.140 INFO:journalctl@ceph.iscsi.iscsi.a.smithi050.stdout:Sep 18 08:20:49 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-iscsi-iscsi-a[57782]: debug there is no tcmu-runner data available 2024-09-18T08:20:50.171 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6 2024-09-18T08:20:50.203 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:20:50.204 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:20:50.205 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6' 2024-09-18T08:20:50.305 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7 2024-09-18T08:20:50.336 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:20:50.336 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:20:50.337 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7' 2024-09-18T08:20:50.405 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:20:50 smithi050 ceph-mon[32371]: from='mgr.24470 172.21.15.7:0/2594390261' entity='mgr.y' 2024-09-18T08:20:50.405 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:20:50 smithi050 ceph-mon[32371]: from='mgr.24470 172.21.15.7:0/2594390261' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-18T08:20:50.405 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:20:50 smithi050 ceph-mon[32371]: pgmap v901: 158 pgs: 158 active+clean; 2.9 GiB data, 13 GiB used, 613 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.43k op/s 2024-09-18T08:20:50.438 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2 2024-09-18T08:20:50.475 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:20:50.475 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:20:50.475 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2' 2024-09-18T08:20:50.479 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:20:50 smithi007 ceph-mon[29459]: from='mgr.24470 172.21.15.7:0/2594390261' entity='mgr.y' 2024-09-18T08:20:50.480 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:20:50 smithi007 ceph-mon[29459]: from='mgr.24470 172.21.15.7:0/2594390261' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-18T08:20:50.480 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:20:50 smithi007 ceph-mon[29459]: pgmap v901: 158 pgs: 158 active+clean; 2.9 GiB data, 13 GiB used, 613 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.43k op/s 2024-09-18T08:20:50.480 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:20:50 smithi007 ceph-mon[39249]: from='mgr.24470 172.21.15.7:0/2594390261' entity='mgr.y' 2024-09-18T08:20:50.480 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:20:50 smithi007 ceph-mon[39249]: from='mgr.24470 172.21.15.7:0/2594390261' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-18T08:20:50.480 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:20:50 smithi007 ceph-mon[39249]: pgmap v901: 158 pgs: 158 active+clean; 2.9 GiB data, 13 GiB used, 613 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.43k op/s 2024-09-18T08:20:50.576 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7 2024-09-18T08:20:50.611 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:20:50.612 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:20:50.612 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7' 2024-09-18T08:20:50.713 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7 2024-09-18T08:20:50.743 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:20:50.744 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:20:50.744 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7' 2024-09-18T08:20:50.845 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0 2024-09-18T08:20:50.885 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:20:50.885 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:20:50.886 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0' 2024-09-18T08:20:50.987 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7 2024-09-18T08:20:51.021 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:20:51.022 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:20:51.022 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7' 2024-09-18T08:20:51.123 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6 2024-09-18T08:20:51.160 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:20:51.161 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:20:51.161 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6' 2024-09-18T08:20:51.262 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1 2024-09-18T08:20:51.298 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:20:51.299 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:20:51.299 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1' 2024-09-18T08:20:51.400 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6 2024-09-18T08:20:51.434 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:20:51.434 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:20:51.435 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6' 2024-09-18T08:20:51.536 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1 2024-09-18T08:20:51.580 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/779349ca-7593-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T08:20:51.593 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:20:51.593 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:20:51.594 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1' 2024-09-18T08:20:51.595 DEBUG:teuthology.orchestra.run.smithi050:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid 779349ca-7593-11ef-bceb-c7b262605968 -- ceph daemon osd.4 dump_ops_in_flight 2024-09-18T08:20:51.695 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7 2024-09-18T08:20:51.728 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:20:51.729 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:20:51.730 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7' 2024-09-18T08:20:51.780 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:20:51 smithi007 ceph-mon[29459]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-18T08:20:51.780 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:20:51 smithi007 ceph-mon[39249]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-18T08:20:51.830 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0 2024-09-18T08:20:51.869 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:20:51.870 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:20:51.870 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0' 2024-09-18T08:20:51.905 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:20:51 smithi050 ceph-mon[32371]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-18T08:20:51.919 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/779349ca-7593-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T08:20:51.971 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7 2024-09-18T08:20:51.972 INFO:teuthology.orchestra.run.smithi050.stderr:Inferring config /var/lib/ceph/779349ca-7593-11ef-bceb-c7b262605968/mon.b/config 2024-09-18T08:20:52.017 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:20:52.017 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:20:52.017 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7' 2024-09-18T08:20:52.118 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6 2024-09-18T08:20:52.148 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:20:52.149 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:20:52.149 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6' 2024-09-18T08:20:52.166 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/779349ca-7593-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T08:20:52.243 INFO:teuthology.orchestra.run.smithi007.stdout:osd.0: {} 2024-09-18T08:20:52.243 INFO:teuthology.orchestra.run.smithi007.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-18T08:20:52.250 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4 2024-09-18T08:20:52.262 INFO:teuthology.orchestra.run.smithi007.stdout:osd.1: {} 2024-09-18T08:20:52.262 INFO:teuthology.orchestra.run.smithi007.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-18T08:20:52.282 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:20:52.282 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:20:52.283 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4' 2024-09-18T08:20:52.295 INFO:teuthology.orchestra.run.smithi007.stdout:osd.2: {} 2024-09-18T08:20:52.295 INFO:teuthology.orchestra.run.smithi007.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-18T08:20:52.331 INFO:teuthology.orchestra.run.smithi007.stdout:osd.3: {} 2024-09-18T08:20:52.331 INFO:teuthology.orchestra.run.smithi007.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-18T08:20:52.369 INFO:teuthology.orchestra.run.smithi007.stdout:osd.4: {} 2024-09-18T08:20:52.369 INFO:teuthology.orchestra.run.smithi007.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-18T08:20:52.384 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1 2024-09-18T08:20:52.407 INFO:teuthology.orchestra.run.smithi007.stdout:osd.5: {} 2024-09-18T08:20:52.407 INFO:teuthology.orchestra.run.smithi007.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-18T08:20:52.430 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:20:52.430 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:20:52.431 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1' 2024-09-18T08:20:52.447 INFO:teuthology.orchestra.run.smithi007.stdout:osd.6: {} 2024-09-18T08:20:52.448 INFO:teuthology.orchestra.run.smithi007.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-18T08:20:52.479 INFO:teuthology.orchestra.run.smithi007.stdout:osd.7: {} 2024-09-18T08:20:52.480 INFO:teuthology.orchestra.run.smithi007.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-18T08:20:52.531 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6 2024-09-18T08:20:52.561 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:20:52.561 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:20:52.562 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6' 2024-09-18T08:20:52.648 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:20:52 smithi007 ceph-mon[29459]: pgmap v902: 158 pgs: 158 active+clean; 2.7 GiB data, 13 GiB used, 613 GiB / 626 GiB avail; 639 B/s rd, 0 B/s wr, 1.40k op/s 2024-09-18T08:20:52.649 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:20:52 smithi007 ceph-mon[29459]: from='client.? 172.21.15.7:0/1920479686' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-18T08:20:52.649 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:20:52 smithi007 ceph-mon[39249]: pgmap v902: 158 pgs: 158 active+clean; 2.7 GiB data, 13 GiB used, 613 GiB / 626 GiB avail; 639 B/s rd, 0 B/s wr, 1.40k op/s 2024-09-18T08:20:52.649 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:20:52 smithi007 ceph-mon[39249]: from='client.? 172.21.15.7:0/1920479686' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-18T08:20:52.662 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3 2024-09-18T08:20:52.708 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:20:52.709 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:20:52.710 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3' 2024-09-18T08:20:52.811 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5 2024-09-18T08:20:52.839 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:20:52.839 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:20:52.840 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5' 2024-09-18T08:20:52.905 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:20:52 smithi050 ceph-mon[32371]: pgmap v902: 158 pgs: 158 active+clean; 2.7 GiB data, 13 GiB used, 613 GiB / 626 GiB avail; 639 B/s rd, 0 B/s wr, 1.40k op/s 2024-09-18T08:20:52.905 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:20:52 smithi050 ceph-mon[32371]: from='client.? 172.21.15.7:0/1920479686' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-18T08:20:52.944 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1 2024-09-18T08:20:52.987 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:20:52.988 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:20:52.988 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1' 2024-09-18T08:20:53.089 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4 2024-09-18T08:20:53.112 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:20:53.113 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:20:53.113 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4' 2024-09-18T08:20:53.214 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6 2024-09-18T08:20:53.244 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:20:53.244 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:20:53.245 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6' 2024-09-18T08:20:53.345 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5 2024-09-18T08:20:53.367 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:20:53.368 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:20:53.368 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5' 2024-09-18T08:20:53.469 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7 2024-09-18T08:20:53.499 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:20:53.500 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:20:53.500 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7' 2024-09-18T08:20:53.601 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6 2024-09-18T08:20:53.637 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:20:53.637 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:20:53.637 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6' 2024-09-18T08:20:53.738 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6 2024-09-18T08:20:53.772 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:20:53.773 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:20:53.773 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6' 2024-09-18T08:20:53.874 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6 2024-09-18T08:20:53.906 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:20:53.907 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:20:53.907 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6' 2024-09-18T08:20:54.008 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4 2024-09-18T08:20:54.040 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:20:54.042 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:20:54.042 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4' 2024-09-18T08:20:54.143 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3 2024-09-18T08:20:54.181 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:20:54.182 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:20:54.182 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3' 2024-09-18T08:20:54.283 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7 2024-09-18T08:20:54.313 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:20:54.314 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:20:54.315 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7' 2024-09-18T08:20:54.415 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2 2024-09-18T08:20:54.453 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:20:54.454 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:20:54.455 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2' 2024-09-18T08:20:54.555 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2 2024-09-18T08:20:54.596 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:20:54.597 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:20:54.598 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2' 2024-09-18T08:20:54.624 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-09-18T08:20:54.627 INFO:teuthology.orchestra.run.smithi007.stderr:dumped all 2024-09-18T08:20:54.699 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5 2024-09-18T08:20:54.721 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:20:54.722 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:20:54.722 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5' 2024-09-18T08:20:54.823 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2 2024-09-18T08:20:54.860 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:20:54.860 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:20:54.861 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2' 2024-09-18T08:20:54.880 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:20:54 smithi007 ceph-mon[29459]: from='client.26005 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-18T08:20:54.881 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:20:54 smithi007 ceph-mon[29459]: pgmap v903: 158 pgs: 158 active+clean; 2.5 GiB data, 12 GiB used, 614 GiB / 626 GiB avail; 1.0 KiB/s rd, 0 B/s wr, 1.32k op/s 2024-09-18T08:20:54.881 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:20:54 smithi007 ceph-mon[39249]: from='client.26005 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-18T08:20:54.881 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:20:54 smithi007 ceph-mon[39249]: pgmap v903: 158 pgs: 158 active+clean; 2.5 GiB data, 12 GiB used, 614 GiB / 626 GiB avail; 1.0 KiB/s rd, 0 B/s wr, 1.32k op/s 2024-09-18T08:20:54.905 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:20:54 smithi050 ceph-mon[32371]: from='client.26005 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-18T08:20:54.905 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:20:54 smithi050 ceph-mon[32371]: pgmap v903: 158 pgs: 158 active+clean; 2.5 GiB data, 12 GiB used, 614 GiB / 626 GiB avail; 1.0 KiB/s rd, 0 B/s wr, 1.32k op/s 2024-09-18T08:20:54.962 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3 2024-09-18T08:20:54.999 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:20:54.999 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:20:55.000 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3' 2024-09-18T08:20:55.101 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3 2024-09-18T08:20:55.140 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:20:55.141 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:20:55.141 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3' 2024-09-18T08:20:55.242 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6 2024-09-18T08:20:55.273 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:20:55.274 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:20:55.274 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6' 2024-09-18T08:20:55.375 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4 2024-09-18T08:20:55.406 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:20:55.407 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:20:55.407 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4' 2024-09-18T08:20:55.509 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7 2024-09-18T08:20:55.551 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:20:55.551 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:20:55.552 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7' 2024-09-18T08:20:55.653 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7 2024-09-18T08:20:55.685 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:20:55.685 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:20:55.686 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7' 2024-09-18T08:20:55.786 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5 2024-09-18T08:20:55.826 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:20:55.827 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:20:55.827 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5' 2024-09-18T08:20:55.928 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1 2024-09-18T08:20:55.965 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:20:55.966 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:20:55.967 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1' 2024-09-18T08:20:56.068 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1 2024-09-18T08:20:56.091 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:20:56.091 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:20:56.092 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1' 2024-09-18T08:20:56.193 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4 2024-09-18T08:20:56.229 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:20:56.230 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:20:56.230 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4' 2024-09-18T08:20:56.325 INFO:teuthology.orchestra.run.smithi050.stdout:ERROR: (22) Invalid argument 2024-09-18T08:20:56.325 INFO:teuthology.orchestra.run.smithi050.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-18T08:20:56.332 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6 2024-09-18T08:20:56.352 INFO:teuthology.orchestra.run.smithi007.stdout:{"pg_ready":true,"pg_map":{"version":902,"stamp":"2024-09-18T08:20:51.517522+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":2905283123,"num_objects":44522,"num_object_clones":0,"num_object_copies":133566,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":44522,"num_whiteouts":0,"num_read":1658,"num_read_kb":1410,"num_write":153763,"num_write_kb":6315630,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19752,"num_bytes_recovered":1255296080,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":64141,"ondisk_log_size":64141,"up":474,"acting":474,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":474,"num_osds":7,"num_per_pool_osds":7,"num_per_pool_omap_osds":7,"kb":656216064,"kb_used":13571404,"kb_used_data":8709464,"kb_used_omap":17,"kb_used_meta":4861806,"kb_avail":642644660,"statfs":{"total":671965249536,"available":658068131840,"internally_reserved":0,"allocated":8918491136,"data_stored":8915712806,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17639,"internal_metadata":4978490137},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,7,4,6,3,0,2,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-734986240,"num_objects":-11215,"num_object_clones":0,"num_object_copies":-33645,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-11215,"num_whiteouts":0,"num_read":5,"num_read_kb":5,"num_write":11215,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.002764"},"pg_stats":[{"pgid":"3.1f","version":"0'0","reported_seq":860,"reported_epoch":445,"state":"active+clean","last_fresh":"2024-09-18T08:20:44.729563+0000","last_change":"2024-09-18T08:19:05.894872+0000","last_active":"2024-09-18T08:20:44.729563+0000","last_peered":"2024-09-18T08:20:44.729563+0000","last_clean":"2024-09-18T08:20:44.729563+0000","last_became_active":"2024-09-18T08:19:05.894522+0000","last_became_peered":"2024-09-18T08:19:05.894522+0000","last_unstale":"2024-09-18T08:20:44.729563+0000","last_undegraded":"2024-09-18T08:20:44.729563+0000","last_fullsized":"2024-09-18T08:20:44.729563+0000","mapping_epoch":413,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":414,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:18:22.822559+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:46.860092+0000","last_clean_scrub_stamp":"2024-09-18T08:18:22.822559+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,6],"acting":[0,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"271'15","reported_seq":586,"reported_epoch":445,"state":"active+clean","last_fresh":"2024-09-18T08:20:44.725909+0000","last_change":"2024-09-18T08:19:13.094994+0000","last_active":"2024-09-18T08:20:44.725909+0000","last_peered":"2024-09-18T08:20:44.725909+0000","last_clean":"2024-09-18T08:20:44.725909+0000","last_became_active":"2024-09-18T08:16:13.456332+0000","last_became_peered":"2024-09-18T08:16:13.456332+0000","last_unstale":"2024-09-18T08:20:44.725909+0000","last_undegraded":"2024-09-18T08:20:44.725909+0000","last_fullsized":"2024-09-18T08:20:44.725909+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"271'15","last_scrub_stamp":"2024-09-18T08:19:13.094901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_clean_scrub_stamp":"2024-09-18T08:19:13.094901+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":25,"num_read_kb":19,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,3],"acting":[2,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":677,"reported_epoch":445,"state":"active+clean","last_fresh":"2024-09-18T08:20:44.733563+0000","last_change":"2024-09-18T08:18:27.641641+0000","last_active":"2024-09-18T08:20:44.733563+0000","last_peered":"2024-09-18T08:20:44.733563+0000","last_clean":"2024-09-18T08:20:44.733563+0000","last_became_active":"2024-09-18T08:18:27.641323+0000","last_became_peered":"2024-09-18T08:18:27.641323+0000","last_unstale":"2024-09-18T08:20:44.733563+0000","last_undegraded":"2024-09-18T08:20:44.733563+0000","last_fullsized":"2024-09-18T08:20:44.733563+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:18:21.610930+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:50.886830+0000","last_clean_scrub_stamp":"2024-09-18T08:18:21.610930+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":568,"reported_epoch":445,"state":"active+clean","last_fresh":"2024-09-18T08:20:44.730976+0000","last_change":"2024-09-18T08:18:27.639074+0000","last_active":"2024-09-18T08:20:44.730976+0000","last_peered":"2024-09-18T08:20:44.730976+0000","last_clean":"2024-09-18T08:20:44.730976+0000","last_became_active":"2024-09-18T08:18:27.638698+0000","last_became_peered":"2024-09-18T08:18:27.638698+0000","last_unstale":"2024-09-18T08:20:44.730976+0000","last_undegraded":"2024-09-18T08:20:44.730976+0000","last_fullsized":"2024-09-18T08:20:44.730976+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:17:40.565100+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_clean_scrub_stamp":"2024-09-18T08:17:40.565100+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":846,"reported_epoch":445,"state":"active+clean","last_fresh":"2024-09-18T08:20:44.734069+0000","last_change":"2024-09-18T08:18:27.662859+0000","last_active":"2024-09-18T08:20:44.734069+0000","last_peered":"2024-09-18T08:20:44.734069+0000","last_clean":"2024-09-18T08:20:44.734069+0000","last_became_active":"2024-09-18T08:18:27.662545+0000","last_became_peered":"2024-09-18T08:18:27.662545+0000","last_unstale":"2024-09-18T08:20:44.734069+0000","last_undegraded":"2024-09-18T08:20:44.734069+0000","last_fullsized":"2024-09-18T08:20:44.734069+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:15:51.826050+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_clean_scrub_stamp":"2024-09-18T08:15:51.826050+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:21:43.895407+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":855,"reported_epoch":445,"state":"active+clean","last_fresh":"2024-09-18T08:20:44.735799+0000","last_change":"2024-09-18T08:19:05.895309+0000","last_active":"2024-09-18T08:20:44.735799+0000","last_peered":"2024-09-18T08:20:44.735799+0000","last_clean":"2024-09-18T08:20:44.735799+0000","last_became_active":"2024-09-18T08:19:05.894958+0000","last_became_peered":"2024-09-18T08:19:05.894958+0000","last_unstale":"2024-09-18T08:20:44.735799+0000","last_undegraded":"2024-09-18T08:20:44.735799+0000","last_fullsized":"2024-09-18T08:20:44.735799+0000","mapping_epoch":413,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":414,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:17:38.359366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:46.860092+0000","last_clean_scrub_stamp":"2024-09-18T08:17:38.359366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,1],"acting":[3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"271'27","reported_seq":871,"reported_epoch":445,"state":"active+clean","last_fresh":"2024-09-18T08:20:44.736374+0000","last_change":"2024-09-18T08:20:14.895711+0000","last_active":"2024-09-18T08:20:44.736374+0000","last_peered":"2024-09-18T08:20:44.736374+0000","last_clean":"2024-09-18T08:20:44.736374+0000","last_became_active":"2024-09-18T08:19:05.907041+0000","last_became_peered":"2024-09-18T08:19:05.907041+0000","last_unstale":"2024-09-18T08:20:44.736374+0000","last_undegraded":"2024-09-18T08:20:44.736374+0000","last_fullsized":"2024-09-18T08:20:44.736374+0000","mapping_epoch":413,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":414,"parent":"0.0","parent_split_bits":0,"last_scrub":"271'27","last_scrub_stamp":"2024-09-18T08:20:14.895622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_clean_scrub_stamp":"2024-09-18T08:20:14.895622+0000","objects_scrubbed":7,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:21:37.655368+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":44,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":488,"reported_epoch":445,"state":"active+clean","last_fresh":"2024-09-18T08:20:44.732495+0000","last_change":"2024-09-18T08:20:38.813717+0000","last_active":"2024-09-18T08:20:44.732495+0000","last_peered":"2024-09-18T08:20:44.732495+0000","last_clean":"2024-09-18T08:20:44.732495+0000","last_became_active":"2024-09-18T08:19:05.886289+0000","last_became_peered":"2024-09-18T08:19:05.886289+0000","last_unstale":"2024-09-18T08:20:44.732495+0000","last_undegraded":"2024-09-18T08:20:44.732495+0000","last_fullsized":"2024-09-18T08:20:44.732495+0000","mapping_epoch":413,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":414,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:20:38.813659+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:50.886830+0000","last_clean_scrub_stamp":"2024-09-18T08:20:38.813659+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:21:58.471438+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,7],"acting":[1,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":245,"reported_epoch":445,"state":"active+clean","last_fresh":"2024-09-18T08:20:44.734507+0000","last_change":"2024-09-18T08:18:17.566526+0000","last_active":"2024-09-18T08:20:44.734507+0000","last_peered":"2024-09-18T08:20:44.734507+0000","last_clean":"2024-09-18T08:20:44.734507+0000","last_became_active":"2024-09-18T08:16:13.458654+0000","last_became_peered":"2024-09-18T08:16:13.458654+0000","last_unstale":"2024-09-18T08:20:44.734507+0000","last_undegraded":"2024-09-18T08:20:44.734507+0000","last_fullsized":"2024-09-18T08:20:44.734507+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:18:17.566467+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:46.860092+0000","last_clean_scrub_stamp":"2024-09-18T08:18:17.566467+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,1],"acting":[5,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"271'15","reported_seq":465,"reported_epoch":445,"state":"active+clean","last_fresh":"2024-09-18T08:20:44.736206+0000","last_change":"2024-09-18T08:19:05.919925+0000","last_active":"2024-09-18T08:20:44.736206+0000","last_peered":"2024-09-18T08:20:44.736206+0000","last_clean":"2024-09-18T08:20:44.736206+0000","last_became_active":"2024-09-18T08:19:05.906571+0000","last_became_peered":"2024-09-18T08:19:05.906571+0000","last_unstale":"2024-09-18T08:20:44.736206+0000","last_undegraded":"2024-09-18T08:20:44.736206+0000","last_fullsized":"2024-09-18T08:20:44.736206+0000","mapping_epoch":413,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":414,"parent":"0.0","parent_split_bits":0,"last_scrub":"271'15","last_scrub_stamp":"2024-09-18T08:18:31.669089+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_clean_scrub_stamp":"2024-09-18T08:18:31.669089+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":24,"num_read_kb":18,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":260,"reported_epoch":445,"state":"active+clean","last_fresh":"2024-09-18T08:20:44.730007+0000","last_change":"2024-09-18T08:19:03.725392+0000","last_active":"2024-09-18T08:20:44.730007+0000","last_peered":"2024-09-18T08:20:44.730007+0000","last_clean":"2024-09-18T08:20:44.730007+0000","last_became_active":"2024-09-18T08:18:27.650998+0000","last_became_peered":"2024-09-18T08:18:27.650998+0000","last_unstale":"2024-09-18T08:20:44.730007+0000","last_undegraded":"2024-09-18T08:20:44.730007+0000","last_fullsized":"2024-09-18T08:20:44.730007+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:19:03.725341+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:50.886830+0000","last_clean_scrub_stamp":"2024-09-18T08:19:03.725341+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":828,"reported_epoch":445,"state":"active+clean","last_fresh":"2024-09-18T08:20:44.730787+0000","last_change":"2024-09-18T08:20:38.093230+0000","last_active":"2024-09-18T08:20:44.730787+0000","last_peered":"2024-09-18T08:20:44.730787+0000","last_clean":"2024-09-18T08:20:44.730787+0000","last_became_active":"2024-09-18T08:18:27.644825+0000","last_became_peered":"2024-09-18T08:18:27.644825+0000","last_unstale":"2024-09-18T08:20:44.730787+0000","last_undegraded":"2024-09-18T08:20:44.730787+0000","last_fullsized":"2024-09-18T08:20:44.730787+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:20:38.093164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_clean_scrub_stamp":"2024-09-18T08:20:38.093164+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:21:56.032483+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":245,"reported_epoch":445,"state":"active+clean","last_fresh":"2024-09-18T08:20:44.732204+0000","last_change":"2024-09-18T08:20:21.821235+0000","last_active":"2024-09-18T08:20:44.732204+0000","last_peered":"2024-09-18T08:20:44.732204+0000","last_clean":"2024-09-18T08:20:44.732204+0000","last_became_active":"2024-09-18T08:16:13.471003+0000","last_became_peered":"2024-09-18T08:16:13.471003+0000","last_unstale":"2024-09-18T08:20:44.732204+0000","last_undegraded":"2024-09-18T08:20:44.732204+0000","last_fullsized":"2024-09-18T08:20:44.732204+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:20:21.821158+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:46.860092+0000","last_clean_scrub_stamp":"2024-09-18T08:20:21.821158+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:21:39.737043+0000","scrub_duration":49,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1b","version":"271'7","reported_seq":134,"reported_epoch":445,"state":"active+clean","last_fresh":"2024-09-18T08:20:44.729862+0000","last_change":"2024-09-18T08:19:05.904504+0000","last_active":"2024-09-18T08:20:44.729862+0000","last_peered":"2024-09-18T08:20:44.729862+0000","last_clean":"2024-09-18T08:20:44.729862+0000","last_became_active":"2024-09-18T08:19:05.885633+0000","last_became_peered":"2024-09-18T08:19:05.885633+0000","last_unstale":"2024-09-18T08:20:44.729862+0000","last_undegraded":"2024-09-18T08:20:44.729862+0000","last_fullsized":"2024-09-18T08:20:44.729862+0000","mapping_epoch":413,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":414,"parent":"0.0","parent_split_bits":0,"last_scrub":"271'7","last_scrub_stamp":"2024-09-18T08:18:27.484925+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_clean_scrub_stamp":"2024-09-18T08:18:27.484925+0000","objects_scrubbed":2,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":13,"num_read_kb":9,"num_write":8,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":106,"reported_epoch":445,"state":"active+clean","last_fresh":"2024-09-18T08:20:44.726937+0000","last_change":"2024-09-18T08:20:36.577065+0000","last_active":"2024-09-18T08:20:44.726937+0000","last_peered":"2024-09-18T08:20:44.726937+0000","last_clean":"2024-09-18T08:20:44.726937+0000","last_became_active":"2024-09-18T08:18:27.656359+0000","last_became_peered":"2024-09-18T08:18:27.656359+0000","last_unstale":"2024-09-18T08:20:44.726937+0000","last_undegraded":"2024-09-18T08:20:44.726937+0000","last_fullsized":"2024-09-18T08:20:44.726937+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:20:36.576992+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:50.886830+0000","last_clean_scrub_stamp":"2024-09-18T08:20:36.576992+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:21:48.544979+0000","scrub_duration":91,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":238,"reported_epoch":445,"state":"active+clean","last_fresh":"2024-09-18T08:20:44.729789+0000","last_change":"2024-09-18T08:18:22.573319+0000","last_active":"2024-09-18T08:20:44.729789+0000","last_peered":"2024-09-18T08:20:44.729789+0000","last_clean":"2024-09-18T08:20:44.729789+0000","last_became_active":"2024-09-18T08:15:41.441932+0000","last_became_peered":"2024-09-18T08:15:41.441932+0000","last_unstale":"2024-09-18T08:20:44.729789+0000","last_undegraded":"2024-09-18T08:20:44.729789+0000","last_fullsized":"2024-09-18T08:20:44.729789+0000","mapping_epoch":336,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":337,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:18:22.573259+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_clean_scrub_stamp":"2024-09-18T08:18:22.573259+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":552,"reported_epoch":445,"state":"active+clean","last_fresh":"2024-09-18T08:20:44.731922+0000","last_change":"2024-09-18T08:20:39.609153+0000","last_active":"2024-09-18T08:20:44.731922+0000","last_peered":"2024-09-18T08:20:44.731922+0000","last_clean":"2024-09-18T08:20:44.731922+0000","last_became_active":"2024-09-18T08:16:13.453129+0000","last_became_peered":"2024-09-18T08:16:13.453129+0000","last_unstale":"2024-09-18T08:20:44.731922+0000","last_undegraded":"2024-09-18T08:20:44.731922+0000","last_fullsized":"2024-09-18T08:20:44.731922+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:20:39.609106+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_clean_scrub_stamp":"2024-09-18T08:20:39.609106+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:21:45.812346+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,3],"acting":[5,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":856,"reported_epoch":445,"state":"active+clean","last_fresh":"2024-09-18T08:20:44.729021+0000","last_change":"2024-09-18T08:18:30.893923+0000","last_active":"2024-09-18T08:20:44.729021+0000","last_peered":"2024-09-18T08:20:44.729021+0000","last_clean":"2024-09-18T08:20:44.729021+0000","last_became_active":"2024-09-18T08:18:27.643517+0000","last_became_peered":"2024-09-18T08:18:27.643517+0000","last_unstale":"2024-09-18T08:20:44.729021+0000","last_undegraded":"2024-09-18T08:20:44.729021+0000","last_fullsized":"2024-09-18T08:20:44.729021+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:18:30.893875+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:46.860092+0000","last_clean_scrub_stamp":"2024-09-18T08:18:30.893875+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,1],"acting":[0,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1c","version":"271'27","reported_seq":435,"reported_epoch":445,"state":"active+clean","last_fresh":"2024-09-18T08:20:44.737415+0000","last_change":"2024-09-18T08:20:17.800209+0000","last_active":"2024-09-18T08:20:44.737415+0000","last_peered":"2024-09-18T08:20:44.737415+0000","last_clean":"2024-09-18T08:20:44.737415+0000","last_became_active":"2024-09-18T08:19:05.912045+0000","last_became_peered":"2024-09-18T08:19:05.912045+0000","last_unstale":"2024-09-18T08:20:44.737415+0000","last_undegraded":"2024-09-18T08:20:44.737415+0000","last_fullsized":"2024-09-18T08:20:44.737415+0000","mapping_epoch":413,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":414,"parent":"0.0","parent_split_bits":0,"last_scrub":"271'27","last_scrub_stamp":"2024-09-18T08:20:17.800133+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_clean_scrub_stamp":"2024-09-18T08:20:17.800133+0000","objects_scrubbed":7,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:21:40.443704+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":44,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,0],"acting":[1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":706,"reported_epoch":445,"state":"active+clean","last_fresh":"2024-09-18T08:20:44.733731+0000","last_change":"2024-09-18T08:19:01.648780+0000","last_active":"2024-09-18T08:20:44.733731+0000","last_peered":"2024-09-18T08:20:44.733731+0000","last_clean":"2024-09-18T08:20:44.733731+0000","last_became_active":"2024-09-18T08:16:13.462303+0000","last_became_peered":"2024-09-18T08:16:13.462303+0000","last_unstale":"2024-09-18T08:20:44.733731+0000","last_undegraded":"2024-09-18T08:20:44.733731+0000","last_fullsized":"2024-09-18T08:20:44.733731+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:19:01.648717+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:50.886830+0000","last_clean_scrub_stamp":"2024-09-18T08:19:01.648717+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,5],"acting":[1,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":834,"reported_epoch":445,"state":"active+clean","last_fresh":"2024-09-18T08:20:44.735468+0000","last_change":"2024-09-18T08:20:22.001150+0000","last_active":"2024-09-18T08:20:44.735468+0000","last_peered":"2024-09-18T08:20:44.735468+0000","last_clean":"2024-09-18T08:20:44.735468+0000","last_became_active":"2024-09-18T08:15:41.466371+0000","last_became_peered":"2024-09-18T08:15:41.466371+0000","last_unstale":"2024-09-18T08:20:44.735468+0000","last_undegraded":"2024-09-18T08:20:44.735468+0000","last_fullsized":"2024-09-18T08:20:44.735468+0000","mapping_epoch":336,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":337,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:20:22.001092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_clean_scrub_stamp":"2024-09-18T08:20:22.001092+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:21:31.821107+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":488,"reported_epoch":445,"state":"active+clean","last_fresh":"2024-09-18T08:20:44.732752+0000","last_change":"2024-09-18T08:20:16.844354+0000","last_active":"2024-09-18T08:20:44.732752+0000","last_peered":"2024-09-18T08:20:44.732752+0000","last_clean":"2024-09-18T08:20:44.732752+0000","last_became_active":"2024-09-18T08:19:05.899551+0000","last_became_peered":"2024-09-18T08:19:05.899551+0000","last_unstale":"2024-09-18T08:20:44.732752+0000","last_undegraded":"2024-09-18T08:20:44.732752+0000","last_fullsized":"2024-09-18T08:20:44.732752+0000","mapping_epoch":413,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":414,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:20:16.844283+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:46.860092+0000","last_clean_scrub_stamp":"2024-09-18T08:20:16.844283+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:21:46.011768+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,7],"acting":[1,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1d","version":"271'18","reported_seq":949,"reported_epoch":445,"state":"active+clean","last_fresh":"2024-09-18T08:20:44.736285+0000","last_change":"2024-09-18T08:18:19.829522+0000","last_active":"2024-09-18T08:20:44.736285+0000","last_peered":"2024-09-18T08:20:44.736285+0000","last_clean":"2024-09-18T08:20:44.736285+0000","last_became_active":"2024-09-18T08:18:19.823337+0000","last_became_peered":"2024-09-18T08:18:19.823337+0000","last_unstale":"2024-09-18T08:20:44.736285+0000","last_undegraded":"2024-09-18T08:20:44.736285+0000","last_fullsized":"2024-09-18T08:20:44.736285+0000","mapping_epoch":397,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":398,"parent":"0.0","parent_split_bits":0,"last_scrub":"271'18","last_scrub_stamp":"2024-09-18T08:17:04.627913+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_clean_scrub_stamp":"2024-09-18T08:17:04.627913+0000","objects_scrubbed":5,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":34,"num_read_kb":25,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":23,"num_bytes_recovered":880,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":256,"reported_epoch":445,"state":"active+clean","last_fresh":"2024-09-18T08:20:44.734355+0000","last_change":"2024-09-18T08:19:05.902064+0000","last_active":"2024-09-18T08:20:44.734355+0000","last_peered":"2024-09-18T08:20:44.734355+0000","last_clean":"2024-09-18T08:20:44.734355+0000","last_became_active":"2024-09-18T08:19:05.901172+0000","last_became_peered":"2024-09-18T08:19:05.901172+0000","last_unstale":"2024-09-18T08:20:44.734355+0000","last_undegraded":"2024-09-18T08:20:44.734355+0000","last_fullsized":"2024-09-18T08:20:44.734355+0000","mapping_epoch":413,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":414,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:18:18.614698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:50.886830+0000","last_clean_scrub_stamp":"2024-09-18T08:18:18.614698+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.1c","version":"69'1","reported_seq":253,"reported_epoch":445,"state":"active+clean","last_fresh":"2024-09-18T08:20:44.732294+0000","last_change":"2024-09-18T08:20:18.804447+0000","last_active":"2024-09-18T08:20:44.732294+0000","last_peered":"2024-09-18T08:20:44.732294+0000","last_clean":"2024-09-18T08:20:44.732294+0000","last_became_active":"2024-09-18T08:18:27.642951+0000","last_became_peered":"2024-09-18T08:18:27.642951+0000","last_unstale":"2024-09-18T08:20:44.732294+0000","last_undegraded":"2024-09-18T08:20:44.732294+0000","last_fullsized":"2024-09-18T08:20:44.732294+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'1","last_scrub_stamp":"2024-09-18T08:20:18.804376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_clean_scrub_stamp":"2024-09-18T08:20:18.804376+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:21:43.741444+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":403,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":1209,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,2],"acting":[5,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"62'1","reported_seq":711,"reported_epoch":445,"state":"active+clean","last_fresh":"2024-09-18T08:20:44.733745+0000","last_change":"2024-09-18T08:17:37.618000+0000","last_active":"2024-09-18T08:20:44.733745+0000","last_peered":"2024-09-18T08:20:44.733745+0000","last_clean":"2024-09-18T08:20:44.733745+0000","last_became_active":"2024-09-18T08:16:13.483397+0000","last_became_peered":"2024-09-18T08:16:13.483397+0000","last_unstale":"2024-09-18T08:20:44.733745+0000","last_undegraded":"2024-09-18T08:20:44.733745+0000","last_fullsized":"2024-09-18T08:20:44.733745+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"62'1","last_scrub_stamp":"2024-09-18T08:17:37.617954+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:46.860092+0000","last_clean_scrub_stamp":"2024-09-18T08:17:37.617954+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":46,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,6],"acting":[1,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1e","version":"271'20","reported_seq":886,"reported_epoch":445,"state":"active+clean","last_fresh":"2024-09-18T08:20:44.728959+0000","last_change":"2024-09-18T08:20:37.101302+0000","last_active":"2024-09-18T08:20:44.728959+0000","last_peered":"2024-09-18T08:20:44.728959+0000","last_clean":"2024-09-18T08:20:44.728959+0000","last_became_active":"2024-09-18T08:18:27.657472+0000","last_became_peered":"2024-09-18T08:18:27.657472+0000","last_unstale":"2024-09-18T08:20:44.728959+0000","last_undegraded":"2024-09-18T08:20:44.728959+0000","last_fullsized":"2024-09-18T08:20:44.728959+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":0,"last_scrub":"271'20","last_scrub_stamp":"2024-09-18T08:20:37.101215+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_clean_scrub_stamp":"2024-09-18T08:20:37.101215+0000","objects_scrubbed":5,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:21:59.026395+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":25,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1f","version":"415'17","reported_seq":143,"reported_epoch":445,"state":"active+clean","last_fresh":"2024-09-18T08:20:44.726833+0000","last_change":"2024-09-18T08:19:05.911190+0000","last_active":"2024-09-18T08:20:44.726833+0000","last_peered":"2024-09-18T08:20:44.726833+0000","last_clean":"2024-09-18T08:20:44.726833+0000","last_became_active":"2024-09-18T08:19:05.901152+0000","last_became_peered":"2024-09-18T08:19:05.901152+0000","last_unstale":"2024-09-18T08:20:44.726833+0000","last_undegraded":"2024-09-18T08:20:44.726833+0000","last_fullsized":"2024-09-18T08:20:44.726833+0000","mapping_epoch":413,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":414,"parent":"0.0","parent_split_bits":0,"last_scrub":"349'15","last_scrub_stamp":"2024-09-18T08:17:29.878376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:50.886830+0000","last_clean_scrub_stamp":"2024-09-18T08:17:29.878376+0000","objects_scrubbed":1,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:21:41.542811+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.0","version":"445'9644","reported_seq":9745,"reported_epoch":445,"state":"active+clean","last_fresh":"2024-09-18T08:20:48.117796+0000","last_change":"2024-09-18T08:18:27.643125+0000","last_active":"2024-09-18T08:20:48.117796+0000","last_peered":"2024-09-18T08:20:48.117796+0000","last_clean":"2024-09-18T08:20:48.117796+0000","last_became_active":"2024-09-18T08:18:27.642627+0000","last_became_peered":"2024-09-18T08:18:27.642627+0000","last_unstale":"2024-09-18T08:20:48.117796+0000","last_undegraded":"2024-09-18T08:20:48.117796+0000","last_fullsized":"2024-09-18T08:20:48.117796+0000","mapping_epoch":401,"log_start":"428'6000","ondisk_log_start":"428'6000","created":401,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:18:26.612016+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:18:26.612016+0000","last_clean_scrub_stamp":"2024-09-18T08:18:26.612016+0000","objects_scrubbed":0,"log_size":3644,"log_dups_size":0,"ondisk_log_size":3644,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":188743680,"num_objects":2880,"num_object_clones":0,"num_object_copies":8640,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2880,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9644,"num_write_kb":400768,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,7],"acting":[1,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.a","version":"74'79","reported_seq":246,"reported_epoch":445,"state":"active+clean","last_fresh":"2024-09-18T08:20:44.731240+0000","last_change":"2024-09-18T08:20:30.688440+0000","last_active":"2024-09-18T08:20:44.731240+0000","last_peered":"2024-09-18T08:20:44.731240+0000","last_clean":"2024-09-18T08:20:44.731240+0000","last_became_active":"2024-09-18T08:15:41.443376+0000","last_became_peered":"2024-09-18T08:15:41.443376+0000","last_unstale":"2024-09-18T08:20:44.731240+0000","last_undegraded":"2024-09-18T08:20:44.731240+0000","last_fullsized":"2024-09-18T08:20:44.731240+0000","mapping_epoch":336,"log_start":"0'0","ondisk_log_start":"0'0","created":152,"last_epoch_clean":337,"parent":"0.0","parent_split_bits":4,"last_scrub":"74'79","last_scrub_stamp":"2024-09-18T08:20:30.688379+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:00:11.893742+0000","last_clean_scrub_stamp":"2024-09-18T08:20:30.688379+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:21:43.177867+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,6],"acting":[5,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.f","version":"271'27","reported_seq":798,"reported_epoch":445,"state":"active+clean","last_fresh":"2024-09-18T08:20:44.734344+0000","last_change":"2024-09-18T08:18:57.614756+0000","last_active":"2024-09-18T08:20:44.734344+0000","last_peered":"2024-09-18T08:20:44.734344+0000","last_clean":"2024-09-18T08:20:44.734344+0000","last_became_active":"2024-09-18T08:18:27.652553+0000","last_became_peered":"2024-09-18T08:18:27.652553+0000","last_unstale":"2024-09-18T08:20:44.734344+0000","last_undegraded":"2024-09-18T08:20:44.734344+0000","last_fullsized":"2024-09-18T08:20:44.734344+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":0,"last_scrub":"271'27","last_scrub_stamp":"2024-09-18T08:18:57.614509+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_clean_scrub_stamp":"2024-09-18T08:18:57.614509+0000","objects_scrubbed":7,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":16,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":43,"num_read_kb":34,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,7],"acting":[1,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":130,"reported_epoch":445,"state":"active+clean","last_fresh":"2024-09-18T08:20:44.727072+0000","last_change":"2024-09-18T08:20:30.934784+0000","last_active":"2024-09-18T08:20:44.727072+0000","last_peered":"2024-09-18T08:20:44.727072+0000","last_clean":"2024-09-18T08:20:44.727072+0000","last_became_active":"2024-09-18T08:18:27.655321+0000","last_became_peered":"2024-09-18T08:18:27.655321+0000","last_unstale":"2024-09-18T08:20:44.727072+0000","last_undegraded":"2024-09-18T08:20:44.727072+0000","last_fullsized":"2024-09-18T08:20:44.727072+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:20:30.934708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:46.860092+0000","last_clean_scrub_stamp":"2024-09-18T08:20:30.934708+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:21:43.965346+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,7],"acting":[2,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.e","version":"415'19","reported_seq":754,"reported_epoch":445,"state":"active+clean","last_fresh":"2024-09-18T08:20:44.732067+0000","last_change":"2024-09-18T08:19:06.698834+0000","last_active":"2024-09-18T08:20:44.732067+0000","last_peered":"2024-09-18T08:20:44.732067+0000","last_clean":"2024-09-18T08:20:44.732067+0000","last_became_active":"2024-09-18T08:19:05.903139+0000","last_became_peered":"2024-09-18T08:19:05.903139+0000","last_unstale":"2024-09-18T08:20:44.732067+0000","last_undegraded":"2024-09-18T08:20:44.732067+0000","last_fullsized":"2024-09-18T08:20:44.732067+0000","mapping_epoch":413,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":414,"parent":"0.0","parent_split_bits":0,"last_scrub":"349'16","last_scrub_stamp":"2024-09-18T08:19:06.698769+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:50.886830+0000","last_clean_scrub_stamp":"2024-09-18T08:19:06.698769+0000","objects_scrubbed":1,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,3],"acting":[5,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":238,"reported_epoch":445,"state":"active+clean","last_fresh":"2024-09-18T08:20:44.733817+0000","last_change":"2024-09-18T08:18:56.732088+0000","last_active":"2024-09-18T08:20:44.733817+0000","last_peered":"2024-09-18T08:20:44.733817+0000","last_clean":"2024-09-18T08:20:44.733817+0000","last_became_active":"2024-09-18T08:15:41.438610+0000","last_became_peered":"2024-09-18T08:15:41.438610+0000","last_unstale":"2024-09-18T08:20:44.733817+0000","last_undegraded":"2024-09-18T08:20:44.733817+0000","last_fullsized":"2024-09-18T08:20:44.733817+0000","mapping_epoch":336,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":337,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:18:56.732041+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_clean_scrub_stamp":"2024-09-18T08:18:56.732041+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.f","version":"445'9461","reported_seq":15899,"reported_epoch":445,"state":"active+clean","last_fresh":"2024-09-18T08:20:48.159712+0000","last_change":"2024-09-18T08:20:15.485566+0000","last_active":"2024-09-18T08:20:48.159712+0000","last_peered":"2024-09-18T08:20:48.159712+0000","last_clean":"2024-09-18T08:20:48.159712+0000","last_became_active":"2024-09-18T08:20:15.477846+0000","last_became_peered":"2024-09-18T08:20:15.477846+0000","last_unstale":"2024-09-18T08:20:48.159712+0000","last_undegraded":"2024-09-18T08:20:48.159712+0000","last_fullsized":"2024-09-18T08:20:48.159712+0000","mapping_epoch":436,"log_start":"428'5800","ondisk_log_start":"428'5800","created":401,"last_epoch_clean":437,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:18:26.612016+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:18:26.612016+0000","last_clean_scrub_stamp":"2024-09-18T08:18:26.612016+0000","objects_scrubbed":0,"log_size":3661,"log_dups_size":0,"ondisk_log_size":3661,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":187105280,"num_objects":2855,"num_object_clones":0,"num_object_copies":8565,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2855,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9461,"num_write_kb":394112,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3163,"num_bytes_recovered":207290368,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.5","version":"74'79","reported_seq":128,"reported_epoch":445,"state":"active+clean","last_fresh":"2024-09-18T08:20:44.734297+0000","last_change":"2024-09-18T08:19:05.894246+0000","last_active":"2024-09-18T08:20:44.734297+0000","last_peered":"2024-09-18T08:20:44.734297+0000","last_clean":"2024-09-18T08:20:44.734297+0000","last_became_active":"2024-09-18T08:19:05.893865+0000","last_became_peered":"2024-09-18T08:19:05.893865+0000","last_unstale":"2024-09-18T08:20:44.734297+0000","last_undegraded":"2024-09-18T08:20:44.734297+0000","last_fullsized":"2024-09-18T08:20:44.734297+0000","mapping_epoch":413,"log_start":"0'0","ondisk_log_start":"0'0","created":152,"last_epoch_clean":414,"parent":"0.0","parent_split_bits":3,"last_scrub":"74'79","last_scrub_stamp":"2024-09-18T08:18:14.059977+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:00:11.893742+0000","last_clean_scrub_stamp":"2024-09-18T08:18:14.059977+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.0","version":"271'30","reported_seq":927,"reported_epoch":445,"state":"active+clean","last_fresh":"2024-09-18T08:20:44.740186+0000","last_change":"2024-09-18T08:18:27.693740+0000","last_active":"2024-09-18T08:20:44.740186+0000","last_peered":"2024-09-18T08:20:44.740186+0000","last_clean":"2024-09-18T08:20:44.740186+0000","last_became_active":"2024-09-18T08:18:27.651348+0000","last_became_peered":"2024-09-18T08:18:27.651348+0000","last_unstale":"2024-09-18T08:20:44.740186+0000","last_undegraded":"2024-09-18T08:20:44.740186+0000","last_fullsized":"2024-09-18T08:20:44.740186+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":0,"last_scrub":"271'30","last_scrub_stamp":"2024-09-18T08:15:57.927111+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_clean_scrub_stamp":"2024-09-18T08:15:57.927111+0000","objects_scrubbed":8,"log_size":30,"log_dups_size":0,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:21:42.884839+0000","scrub_duration":18,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":52,"num_read_kb":40,"num_write":32,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":870,"reported_epoch":445,"state":"active+clean","last_fresh":"2024-09-18T08:20:44.736041+0000","last_change":"2024-09-18T08:20:22.993934+0000","last_active":"2024-09-18T08:20:44.736041+0000","last_peered":"2024-09-18T08:20:44.736041+0000","last_clean":"2024-09-18T08:20:44.736041+0000","last_became_active":"2024-09-18T08:18:27.663173+0000","last_became_peered":"2024-09-18T08:18:27.663173+0000","last_unstale":"2024-09-18T08:20:44.736041+0000","last_undegraded":"2024-09-18T08:20:44.736041+0000","last_fullsized":"2024-09-18T08:20:44.736041+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:20:22.993864+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:46.860092+0000","last_clean_scrub_stamp":"2024-09-18T08:20:22.993864+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:21:28.227839+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":471,"reported_epoch":445,"state":"active+clean","last_fresh":"2024-09-18T08:20:44.738533+0000","last_change":"2024-09-18T08:18:27.661806+0000","last_active":"2024-09-18T08:20:44.738533+0000","last_peered":"2024-09-18T08:20:44.738533+0000","last_clean":"2024-09-18T08:20:44.738533+0000","last_became_active":"2024-09-18T08:18:27.661393+0000","last_became_peered":"2024-09-18T08:18:27.661393+0000","last_unstale":"2024-09-18T08:20:44.738533+0000","last_undegraded":"2024-09-18T08:20:44.738533+0000","last_fullsized":"2024-09-18T08:20:44.738533+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:17:31.417453+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:50.886830+0000","last_clean_scrub_stamp":"2024-09-18T08:17:31.417453+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":251,"reported_epoch":445,"state":"active+clean","last_fresh":"2024-09-18T08:20:44.736118+0000","last_change":"2024-09-18T08:19:06.578252+0000","last_active":"2024-09-18T08:20:44.736118+0000","last_peered":"2024-09-18T08:20:44.736118+0000","last_clean":"2024-09-18T08:20:44.736118+0000","last_became_active":"2024-09-18T08:19:05.893777+0000","last_became_peered":"2024-09-18T08:19:05.893777+0000","last_unstale":"2024-09-18T08:20:44.736118+0000","last_undegraded":"2024-09-18T08:20:44.736118+0000","last_fullsized":"2024-09-18T08:20:44.736118+0000","mapping_epoch":413,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":414,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:19:06.578190+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_clean_scrub_stamp":"2024-09-18T08:19:06.578190+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,7],"acting":[3,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.e","version":"445'9481","reported_seq":9582,"reported_epoch":445,"state":"active+clean","last_fresh":"2024-09-18T08:20:48.170231+0000","last_change":"2024-09-18T08:18:27.652534+0000","last_active":"2024-09-18T08:20:48.170231+0000","last_peered":"2024-09-18T08:20:48.170231+0000","last_clean":"2024-09-18T08:20:48.170231+0000","last_became_active":"2024-09-18T08:18:27.652261+0000","last_became_peered":"2024-09-18T08:18:27.652261+0000","last_unstale":"2024-09-18T08:20:48.170231+0000","last_undegraded":"2024-09-18T08:20:48.170231+0000","last_fullsized":"2024-09-18T08:20:48.170231+0000","mapping_epoch":401,"log_start":"428'5900","ondisk_log_start":"428'5900","created":401,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:18:26.612016+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:18:26.612016+0000","last_clean_scrub_stamp":"2024-09-18T08:18:26.612016+0000","objects_scrubbed":0,"log_size":3581,"log_dups_size":0,"ondisk_log_size":3581,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":188809216,"num_objects":2881,"num_object_clones":0,"num_object_copies":8643,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2881,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9481,"num_write_kb":395584,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"74'79","reported_seq":697,"reported_epoch":445,"state":"active+clean","last_fresh":"2024-09-18T08:20:44.734033+0000","last_change":"2024-09-18T08:20:25.025206+0000","last_active":"2024-09-18T08:20:44.734033+0000","last_peered":"2024-09-18T08:20:44.734033+0000","last_clean":"2024-09-18T08:20:44.734033+0000","last_became_active":"2024-09-18T08:18:27.662304+0000","last_became_peered":"2024-09-18T08:18:27.662304+0000","last_unstale":"2024-09-18T08:20:44.734033+0000","last_undegraded":"2024-09-18T08:20:44.734033+0000","last_fullsized":"2024-09-18T08:20:44.734033+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":152,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":3,"last_scrub":"74'79","last_scrub_stamp":"2024-09-18T08:20:25.025128+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:00:11.893742+0000","last_clean_scrub_stamp":"2024-09-18T08:20:25.025128+0000","objects_scrubbed":1,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:21:29.846745+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":327680,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1310720,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,7],"acting":[3,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"271'28","reported_seq":621,"reported_epoch":445,"state":"active+clean","last_fresh":"2024-09-18T08:20:44.729692+0000","last_change":"2024-09-18T08:20:33.720957+0000","last_active":"2024-09-18T08:20:44.729692+0000","last_peered":"2024-09-18T08:20:44.729692+0000","last_clean":"2024-09-18T08:20:44.729692+0000","last_became_active":"2024-09-18T08:16:13.451205+0000","last_became_peered":"2024-09-18T08:16:13.451205+0000","last_unstale":"2024-09-18T08:20:44.729692+0000","last_undegraded":"2024-09-18T08:20:44.729692+0000","last_fullsized":"2024-09-18T08:20:44.729692+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"271'28","last_scrub_stamp":"2024-09-18T08:20:33.720855+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_clean_scrub_stamp":"2024-09-18T08:20:33.720855+0000","objects_scrubbed":7,"log_size":28,"log_dups_size":0,"ondisk_log_size":28,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:21:41.189100+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":41,"num_read_kb":34,"num_write":28,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.6","version":"62'2","reported_seq":887,"reported_epoch":445,"state":"active+clean","last_fresh":"2024-09-18T08:20:44.729342+0000","last_change":"2024-09-18T08:20:17.276808+0000","last_active":"2024-09-18T08:20:44.729342+0000","last_peered":"2024-09-18T08:20:44.729342+0000","last_clean":"2024-09-18T08:20:44.729342+0000","last_became_active":"2024-09-18T08:18:27.645204+0000","last_became_peered":"2024-09-18T08:18:27.645204+0000","last_unstale":"2024-09-18T08:20:44.729342+0000","last_undegraded":"2024-09-18T08:20:44.729342+0000","last_fullsized":"2024-09-18T08:20:44.729342+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":0,"last_scrub":"62'2","last_scrub_stamp":"2024-09-18T08:20:17.276736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:46.860092+0000","last_clean_scrub_stamp":"2024-09-18T08:20:17.276736+0000","objects_scrubbed":2,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:21:22.408837+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1039,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":4156,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.0","version":"349'13","reported_seq":1081,"reported_epoch":445,"state":"active+clean","last_fresh":"2024-09-18T08:20:44.740028+0000","last_change":"2024-09-18T08:20:40.011177+0000","last_active":"2024-09-18T08:20:44.740028+0000","last_peered":"2024-09-18T08:20:44.740028+0000","last_clean":"2024-09-18T08:20:44.740028+0000","last_became_active":"2024-09-18T08:16:13.477431+0000","last_became_peered":"2024-09-18T08:16:13.477431+0000","last_unstale":"2024-09-18T08:20:44.740028+0000","last_undegraded":"2024-09-18T08:20:44.740028+0000","last_fullsized":"2024-09-18T08:20:44.740028+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"349'13","last_scrub_stamp":"2024-09-18T08:20:40.010801+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:50.886830+0000","last_clean_scrub_stamp":"2024-09-18T08:20:40.010801+0000","objects_scrubbed":1,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:21:56.887714+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":238,"reported_epoch":445,"state":"active+clean","last_fresh":"2024-09-18T08:20:44.729922+0000","last_change":"2024-09-18T08:20:25.734132+0000","last_active":"2024-09-18T08:20:44.729922+0000","last_peered":"2024-09-18T08:20:44.729922+0000","last_clean":"2024-09-18T08:20:44.729922+0000","last_became_active":"2024-09-18T08:15:41.440876+0000","last_became_peered":"2024-09-18T08:15:41.440876+0000","last_unstale":"2024-09-18T08:20:44.729922+0000","last_undegraded":"2024-09-18T08:20:44.729922+0000","last_fullsized":"2024-09-18T08:20:44.729922+0000","mapping_epoch":336,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":337,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:20:25.734050+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_clean_scrub_stamp":"2024-09-18T08:20:25.734050+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:21:47.697644+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.d","version":"445'9348","reported_seq":15776,"reported_epoch":445,"state":"active+clean","last_fresh":"2024-09-18T08:20:48.122246+0000","last_change":"2024-09-18T08:20:06.497992+0000","last_active":"2024-09-18T08:20:48.122246+0000","last_peered":"2024-09-18T08:20:48.122246+0000","last_clean":"2024-09-18T08:20:48.122246+0000","last_became_active":"2024-09-18T08:20:06.492093+0000","last_became_peered":"2024-09-18T08:20:06.492093+0000","last_unstale":"2024-09-18T08:20:48.122246+0000","last_undegraded":"2024-09-18T08:20:48.122246+0000","last_fullsized":"2024-09-18T08:20:48.122246+0000","mapping_epoch":431,"log_start":"428'5700","ondisk_log_start":"428'5700","created":401,"last_epoch_clean":432,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:18:26.612016+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:18:26.612016+0000","last_clean_scrub_stamp":"2024-09-18T08:18:26.612016+0000","objects_scrubbed":0,"log_size":3648,"log_dups_size":0,"ondisk_log_size":3648,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":179765272,"num_objects":2744,"num_object_clones":0,"num_object_copies":8232,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2744,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":9348,"num_write_kb":386881,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3158,"num_bytes_recovered":206962688,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,5],"acting":[1,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"74'79","reported_seq":676,"reported_epoch":445,"state":"active+clean","last_fresh":"2024-09-18T08:20:44.735787+0000","last_change":"2024-09-18T08:20:26.008280+0000","last_active":"2024-09-18T08:20:44.735787+0000","last_peered":"2024-09-18T08:20:44.735787+0000","last_clean":"2024-09-18T08:20:44.735787+0000","last_became_active":"2024-09-18T08:18:27.662723+0000","last_became_peered":"2024-09-18T08:18:27.662723+0000","last_unstale":"2024-09-18T08:20:44.735787+0000","last_undegraded":"2024-09-18T08:20:44.735787+0000","last_fullsized":"2024-09-18T08:20:44.735787+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":152,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":3,"last_scrub":"74'79","last_scrub_stamp":"2024-09-18T08:20:26.008206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:00:11.893742+0000","last_clean_scrub_stamp":"2024-09-18T08:20:26.008206+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:21:40.501778+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,7],"acting":[3,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.2","version":"271'20","reported_seq":830,"reported_epoch":445,"state":"active+clean","last_fresh":"2024-09-18T08:20:44.732837+0000","last_change":"2024-09-18T08:18:22.653813+0000","last_active":"2024-09-18T08:20:44.732837+0000","last_peered":"2024-09-18T08:20:44.732837+0000","last_clean":"2024-09-18T08:20:44.732837+0000","last_became_active":"2024-09-18T08:16:13.486720+0000","last_became_peered":"2024-09-18T08:16:13.486720+0000","last_unstale":"2024-09-18T08:20:44.732837+0000","last_undegraded":"2024-09-18T08:20:44.732837+0000","last_fullsized":"2024-09-18T08:20:44.732837+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"271'20","last_scrub_stamp":"2024-09-18T08:18:22.653761+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_clean_scrub_stamp":"2024-09-18T08:18:22.653761+0000","objects_scrubbed":5,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":28,"num_read_kb":23,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":238,"reported_epoch":445,"state":"active+clean","last_fresh":"2024-09-18T08:20:44.734771+0000","last_change":"2024-09-18T08:20:22.733384+0000","last_active":"2024-09-18T08:20:44.734771+0000","last_peered":"2024-09-18T08:20:44.734771+0000","last_clean":"2024-09-18T08:20:44.734771+0000","last_became_active":"2024-09-18T08:15:41.449639+0000","last_became_peered":"2024-09-18T08:15:41.449639+0000","last_unstale":"2024-09-18T08:20:44.734771+0000","last_undegraded":"2024-09-18T08:20:44.734771+0000","last_fullsized":"2024-09-18T08:20:44.734771+0000","mapping_epoch":336,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":337,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:20:22.733321+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:46.860092+0000","last_clean_scrub_stamp":"2024-09-18T08:20:22.733321+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:21:41.865798+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.3","version":"338'14","reported_seq":1081,"reported_epoch":445,"state":"active+clean","last_fresh":"2024-09-18T08:20:44.728417+0000","last_change":"2024-09-18T08:20:32.197368+0000","last_active":"2024-09-18T08:20:44.728417+0000","last_peered":"2024-09-18T08:20:44.728417+0000","last_clean":"2024-09-18T08:20:44.728417+0000","last_became_active":"2024-09-18T08:15:41.447667+0000","last_became_peered":"2024-09-18T08:15:41.447667+0000","last_unstale":"2024-09-18T08:20:44.728417+0000","last_undegraded":"2024-09-18T08:20:44.728417+0000","last_fullsized":"2024-09-18T08:20:44.728417+0000","mapping_epoch":336,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":337,"parent":"0.0","parent_split_bits":0,"last_scrub":"338'14","last_scrub_stamp":"2024-09-18T08:20:32.197304+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:50.886830+0000","last_clean_scrub_stamp":"2024-09-18T08:20:32.197304+0000","objects_scrubbed":1,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:22:01.212587+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":843,"reported_epoch":445,"state":"active+clean","last_fresh":"2024-09-18T08:20:44.731175+0000","last_change":"2024-09-18T08:19:05.899449+0000","last_active":"2024-09-18T08:20:44.731175+0000","last_peered":"2024-09-18T08:20:44.731175+0000","last_clean":"2024-09-18T08:20:44.731175+0000","last_became_active":"2024-09-18T08:19:05.899084+0000","last_became_peered":"2024-09-18T08:19:05.899084+0000","last_unstale":"2024-09-18T08:20:44.731175+0000","last_undegraded":"2024-09-18T08:20:44.731175+0000","last_fullsized":"2024-09-18T08:20:44.731175+0000","mapping_epoch":413,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":414,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:18:23.800677+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_clean_scrub_stamp":"2024-09-18T08:18:23.800677+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,5],"acting":[0,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.c","version":"445'9441","reported_seq":15887,"reported_epoch":445,"state":"active+clean","last_fresh":"2024-09-18T08:20:48.163827+0000","last_change":"2024-09-18T08:20:13.526077+0000","last_active":"2024-09-18T08:20:48.163827+0000","last_peered":"2024-09-18T08:20:48.163827+0000","last_clean":"2024-09-18T08:20:48.163827+0000","last_became_active":"2024-09-18T08:20:13.520037+0000","last_became_peered":"2024-09-18T08:20:13.520037+0000","last_unstale":"2024-09-18T08:20:48.163827+0000","last_undegraded":"2024-09-18T08:20:48.163827+0000","last_fullsized":"2024-09-18T08:20:48.163827+0000","mapping_epoch":434,"log_start":"428'5800","ondisk_log_start":"428'5800","created":401,"last_epoch_clean":435,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:18:26.612016+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:18:26.612016+0000","last_clean_scrub_stamp":"2024-09-18T08:18:26.612016+0000","objects_scrubbed":0,"log_size":3641,"log_dups_size":0,"ondisk_log_size":3641,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":185139200,"num_objects":2825,"num_object_clones":0,"num_object_copies":8475,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2825,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9441,"num_write_kb":392512,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3168,"num_bytes_recovered":207618048,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,5],"acting":[1,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"74'79","reported_seq":728,"reported_epoch":445,"state":"active+clean","last_fresh":"2024-09-18T08:20:44.728402+0000","last_change":"2024-09-18T08:18:20.837291+0000","last_active":"2024-09-18T08:20:44.728402+0000","last_peered":"2024-09-18T08:20:44.728402+0000","last_clean":"2024-09-18T08:20:44.728402+0000","last_became_active":"2024-09-18T08:07:26.426532+0000","last_became_peered":"2024-09-18T08:07:26.426532+0000","last_unstale":"2024-09-18T08:20:44.728402+0000","last_undegraded":"2024-09-18T08:20:44.728402+0000","last_fullsized":"2024-09-18T08:20:44.728402+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":152,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":3,"last_scrub":"74'79","last_scrub_stamp":"2024-09-18T08:18:20.837241+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:00:11.893742+0000","last_clean_scrub_stamp":"2024-09-18T08:18:20.837241+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,3],"acting":[0,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.3","version":"271'29","reported_seq":969,"reported_epoch":445,"state":"active+clean","last_fresh":"2024-09-18T08:20:44.728584+0000","last_change":"2024-09-18T08:18:27.685772+0000","last_active":"2024-09-18T08:20:44.728584+0000","last_peered":"2024-09-18T08:20:44.728584+0000","last_clean":"2024-09-18T08:20:44.728584+0000","last_became_active":"2024-09-18T08:18:27.654192+0000","last_became_peered":"2024-09-18T08:18:27.654192+0000","last_unstale":"2024-09-18T08:20:44.728584+0000","last_undegraded":"2024-09-18T08:20:44.728584+0000","last_fullsized":"2024-09-18T08:20:44.728584+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":0,"last_scrub":"271'29","last_scrub_stamp":"2024-09-18T08:18:15.879919+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_clean_scrub_stamp":"2024-09-18T08:18:15.879919+0000","objects_scrubbed":8,"log_size":29,"log_dups_size":0,"ondisk_log_size":29,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":54,"num_read_kb":40,"num_write":32,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":32,"num_bytes_recovered":1320,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.4","version":"62'1","reported_seq":714,"reported_epoch":445,"state":"active+clean","last_fresh":"2024-09-18T08:20:44.737141+0000","last_change":"2024-09-18T08:20:22.818710+0000","last_active":"2024-09-18T08:20:44.737141+0000","last_peered":"2024-09-18T08:20:44.737141+0000","last_clean":"2024-09-18T08:20:44.737141+0000","last_became_active":"2024-09-18T08:19:05.911595+0000","last_became_peered":"2024-09-18T08:19:05.911595+0000","last_unstale":"2024-09-18T08:20:44.737141+0000","last_undegraded":"2024-09-18T08:20:44.737141+0000","last_fullsized":"2024-09-18T08:20:44.737141+0000","mapping_epoch":413,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":414,"parent":"0.0","parent_split_bits":0,"last_scrub":"62'1","last_scrub_stamp":"2024-09-18T08:20:22.818512+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:46.860092+0000","last_clean_scrub_stamp":"2024-09-18T08:20:22.818512+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:21:23.194435+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":436,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1744,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":749,"reported_epoch":445,"state":"active+clean","last_fresh":"2024-09-18T08:20:44.728792+0000","last_change":"2024-09-18T08:18:58.096376+0000","last_active":"2024-09-18T08:20:44.728792+0000","last_peered":"2024-09-18T08:20:44.728792+0000","last_clean":"2024-09-18T08:20:44.728792+0000","last_became_active":"2024-09-18T08:15:41.449832+0000","last_became_peered":"2024-09-18T08:15:41.449832+0000","last_unstale":"2024-09-18T08:20:44.728792+0000","last_undegraded":"2024-09-18T08:20:44.728792+0000","last_fullsized":"2024-09-18T08:20:44.728792+0000","mapping_epoch":336,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":337,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:18:58.096199+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:50.886830+0000","last_clean_scrub_stamp":"2024-09-18T08:18:58.096199+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":729,"reported_epoch":445,"state":"active+clean","last_fresh":"2024-09-18T08:20:44.733450+0000","last_change":"2024-09-18T08:20:27.908933+0000","last_active":"2024-09-18T08:20:44.733450+0000","last_peered":"2024-09-18T08:20:44.733450+0000","last_clean":"2024-09-18T08:20:44.733450+0000","last_became_active":"2024-09-18T08:19:05.881711+0000","last_became_peered":"2024-09-18T08:19:05.881711+0000","last_unstale":"2024-09-18T08:20:44.733450+0000","last_undegraded":"2024-09-18T08:20:44.733450+0000","last_fullsized":"2024-09-18T08:20:44.733450+0000","mapping_epoch":413,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":414,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:20:27.908874+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_clean_scrub_stamp":"2024-09-18T08:20:27.908874+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:21:51.062812+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,7],"acting":[1,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.b","version":"445'9606","reported_seq":9707,"reported_epoch":445,"state":"active+clean","last_fresh":"2024-09-18T08:20:49.839632+0000","last_change":"2024-09-18T08:18:27.656478+0000","last_active":"2024-09-18T08:20:49.839632+0000","last_peered":"2024-09-18T08:20:49.839632+0000","last_clean":"2024-09-18T08:20:49.839632+0000","last_became_active":"2024-09-18T08:18:27.656216+0000","last_became_peered":"2024-09-18T08:18:27.656216+0000","last_unstale":"2024-09-18T08:20:49.839632+0000","last_undegraded":"2024-09-18T08:20:49.839632+0000","last_fullsized":"2024-09-18T08:20:49.839632+0000","mapping_epoch":401,"log_start":"428'5600","ondisk_log_start":"428'5600","created":401,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:18:26.612016+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:18:26.612016+0000","last_clean_scrub_stamp":"2024-09-18T08:18:26.612016+0000","objects_scrubbed":0,"log_size":4006,"log_dups_size":0,"ondisk_log_size":4006,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":173670400,"num_objects":2650,"num_object_clones":0,"num_object_copies":7950,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2650,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9606,"num_write_kb":392192,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"74'79","reported_seq":246,"reported_epoch":445,"state":"active+clean","last_fresh":"2024-09-18T08:20:44.729892+0000","last_change":"2024-09-18T08:20:40.596283+0000","last_active":"2024-09-18T08:20:44.729892+0000","last_peered":"2024-09-18T08:20:44.729892+0000","last_clean":"2024-09-18T08:20:44.729892+0000","last_became_active":"2024-09-18T08:15:41.452895+0000","last_became_peered":"2024-09-18T08:15:41.452895+0000","last_unstale":"2024-09-18T08:20:44.729892+0000","last_undegraded":"2024-09-18T08:20:44.729892+0000","last_fullsized":"2024-09-18T08:20:44.729892+0000","mapping_epoch":336,"log_start":"0'0","ondisk_log_start":"0'0","created":152,"last_epoch_clean":337,"parent":"0.0","parent_split_bits":4,"last_scrub":"74'79","last_scrub_stamp":"2024-09-18T08:20:40.596210+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:00:11.893742+0000","last_clean_scrub_stamp":"2024-09-18T08:20:40.596210+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:21:46.761780+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,2],"acting":[5,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.4","version":"415'43","reported_seq":920,"reported_epoch":445,"state":"active+clean","last_fresh":"2024-09-18T08:20:44.737357+0000","last_change":"2024-09-18T08:19:05.931238+0000","last_active":"2024-09-18T08:20:44.737357+0000","last_peered":"2024-09-18T08:20:44.737357+0000","last_clean":"2024-09-18T08:20:44.737357+0000","last_became_active":"2024-09-18T08:19:05.911342+0000","last_became_peered":"2024-09-18T08:19:05.911342+0000","last_unstale":"2024-09-18T08:20:44.737357+0000","last_undegraded":"2024-09-18T08:20:44.737357+0000","last_fullsized":"2024-09-18T08:20:44.737357+0000","mapping_epoch":413,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":414,"parent":"0.0","parent_split_bits":0,"last_scrub":"310'42","last_scrub_stamp":"2024-09-18T08:18:27.690465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_clean_scrub_stamp":"2024-09-18T08:18:27.690465+0000","objects_scrubbed":10,"log_size":43,"log_dups_size":0,"ondisk_log_size":43,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":65,"num_read_kb":50,"num_write":38,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":358,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,6],"acting":[1,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":418,"reported_epoch":445,"state":"active+clean","last_fresh":"2024-09-18T08:20:44.728916+0000","last_change":"2024-09-18T08:18:16.839327+0000","last_active":"2024-09-18T08:20:44.728916+0000","last_peered":"2024-09-18T08:20:44.728916+0000","last_clean":"2024-09-18T08:20:44.728916+0000","last_became_active":"2024-09-18T08:16:13.475879+0000","last_became_peered":"2024-09-18T08:16:13.475879+0000","last_unstale":"2024-09-18T08:20:44.728916+0000","last_undegraded":"2024-09-18T08:20:44.728916+0000","last_fullsized":"2024-09-18T08:20:44.728916+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:18:16.839282+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:46.860092+0000","last_clean_scrub_stamp":"2024-09-18T08:18:16.839282+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,1],"acting":[0,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"64'2","reported_seq":250,"reported_epoch":445,"state":"active+clean","last_fresh":"2024-09-18T08:20:44.731130+0000","last_change":"2024-09-18T08:19:00.785048+0000","last_active":"2024-09-18T08:20:44.731130+0000","last_peered":"2024-09-18T08:20:44.731130+0000","last_clean":"2024-09-18T08:20:44.731130+0000","last_became_active":"2024-09-18T08:15:41.453351+0000","last_became_peered":"2024-09-18T08:15:41.453351+0000","last_unstale":"2024-09-18T08:20:44.731130+0000","last_undegraded":"2024-09-18T08:20:44.731130+0000","last_fullsized":"2024-09-18T08:20:44.731130+0000","mapping_epoch":336,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":337,"parent":"0.0","parent_split_bits":0,"last_scrub":"64'2","last_scrub_stamp":"2024-09-18T08:19:00.784983+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:45.855979+0000","last_clean_scrub_stamp":"2024-09-18T08:19:00.784983+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":841,"reported_epoch":445,"state":"active+clean","last_fresh":"2024-09-18T08:20:44.729199+0000","last_change":"2024-09-18T08:20:27.117188+0000","last_active":"2024-09-18T08:20:44.729199+0000","last_peered":"2024-09-18T08:20:44.729199+0000","last_clean":"2024-09-18T08:20:44.729199+0000","last_became_active":"2024-09-18T08:16:13.481456+0000","last_became_peered":"2024-09-18T08:16:13.481456+0000","last_unstale":"2024-09-18T08:20:44.729199+0000","last_undegraded":"2024-09-18T08:20:44.729199+0000","last_fullsized":"2024-09-18T08:20:44.729199+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:20:27.117124+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:50.886830+0000","last_clean_scrub_stamp":"2024-09-18T08:20:27.117124+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:21:46.449723+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,6],"acting":[0,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":858,"reported_epoch":445,"state":"active+clean","last_fresh":"2024-09-18T08:20:44.735548+0000","last_change":"2024-09-18T08:20:33.023747+0000","last_active":"2024-09-18T08:20:44.735548+0000","last_peered":"2024-09-18T08:20:44.735548+0000","last_clean":"2024-09-18T08:20:44.735548+0000","last_became_active":"2024-09-18T08:18:27.657591+0000","last_became_peered":"2024-09-18T08:18:27.657591+0000","last_unstale":"2024-09-18T08:20:44.735548+0000","last_undegraded":"2024-09-18T08:20:44.735548+0000","last_fullsized":"2024-09-18T08:20:44.735548+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:20:33.023681+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_clean_scrub_stamp":"2024-09-18T08:20:33.023681+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:21:34.247858+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.8","version":"445'9844","reported_seq":9945,"reported_epoch":445,"state":"active+clean","last_fresh":"2024-09-18T08:20:49.829537+0000","last_change":"2024-09-18T08:18:27.641711+0000","last_active":"2024-09-18T08:20:49.829537+0000","last_peered":"2024-09-18T08:20:49.829537+0000","last_clean":"2024-09-18T08:20:49.829537+0000","last_became_active":"2024-09-18T08:18:27.641439+0000","last_became_peered":"2024-09-18T08:18:27.641439+0000","last_unstale":"2024-09-18T08:20:49.829537+0000","last_undegraded":"2024-09-18T08:20:49.829537+0000","last_fullsized":"2024-09-18T08:20:49.829537+0000","mapping_epoch":401,"log_start":"428'5800","ondisk_log_start":"428'5800","created":401,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:18:26.612016+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:18:26.612016+0000","last_clean_scrub_stamp":"2024-09-18T08:18:26.612016+0000","objects_scrubbed":0,"log_size":4044,"log_dups_size":0,"ondisk_log_size":4044,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":179175424,"num_objects":2734,"num_object_clones":0,"num_object_copies":8202,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2734,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9844,"num_write_kb":402496,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"74'79","reported_seq":769,"reported_epoch":445,"state":"active+clean","last_fresh":"2024-09-18T08:20:44.728580+0000","last_change":"2024-09-18T08:19:05.896315+0000","last_active":"2024-09-18T08:20:44.728580+0000","last_peered":"2024-09-18T08:20:44.728580+0000","last_clean":"2024-09-18T08:20:44.728580+0000","last_became_active":"2024-09-18T08:19:05.895876+0000","last_became_peered":"2024-09-18T08:19:05.895876+0000","last_unstale":"2024-09-18T08:20:44.728580+0000","last_undegraded":"2024-09-18T08:20:44.728580+0000","last_fullsized":"2024-09-18T08:20:44.728580+0000","mapping_epoch":413,"log_start":"0'0","ondisk_log_start":"0'0","created":152,"last_epoch_clean":414,"parent":"0.0","parent_split_bits":4,"last_scrub":"74'79","last_scrub_stamp":"2024-09-18T08:18:17.808269+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:00:11.893742+0000","last_clean_scrub_stamp":"2024-09-18T08:18:17.808269+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,5],"acting":[0,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.7","version":"271'17","reported_seq":827,"reported_epoch":445,"state":"active+clean","last_fresh":"2024-09-18T08:20:44.726899+0000","last_change":"2024-09-18T08:20:11.911309+0000","last_active":"2024-09-18T08:20:44.726899+0000","last_peered":"2024-09-18T08:20:44.726899+0000","last_clean":"2024-09-18T08:20:44.726899+0000","last_became_active":"2024-09-18T08:15:41.456482+0000","last_became_peered":"2024-09-18T08:15:41.456482+0000","last_unstale":"2024-09-18T08:20:44.726899+0000","last_undegraded":"2024-09-18T08:20:44.726899+0000","last_fullsized":"2024-09-18T08:20:44.726899+0000","mapping_epoch":336,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":337,"parent":"0.0","parent_split_bits":0,"last_scrub":"271'17","last_scrub_stamp":"2024-09-18T08:20:11.911227+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_clean_scrub_stamp":"2024-09-18T08:20:11.911227+0000","objects_scrubbed":5,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:21:41.454984+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":35,"num_read_kb":24,"num_write":20,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":990,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,0],"acting":[2,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":712,"reported_epoch":445,"state":"active+clean","last_fresh":"2024-09-18T08:20:44.732595+0000","last_change":"2024-09-18T08:19:05.905706+0000","last_active":"2024-09-18T08:20:44.732595+0000","last_peered":"2024-09-18T08:20:44.732595+0000","last_clean":"2024-09-18T08:20:44.732595+0000","last_became_active":"2024-09-18T08:19:05.905390+0000","last_became_peered":"2024-09-18T08:19:05.905390+0000","last_unstale":"2024-09-18T08:20:44.732595+0000","last_undegraded":"2024-09-18T08:20:44.732595+0000","last_fullsized":"2024-09-18T08:20:44.732595+0000","mapping_epoch":413,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":414,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:15:49.244156+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:46.860092+0000","last_clean_scrub_stamp":"2024-09-18T08:15:49.244156+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,0],"acting":[1,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"62'1","reported_seq":92,"reported_epoch":445,"state":"active+clean","last_fresh":"2024-09-18T08:20:44.736473+0000","last_change":"2024-09-18T08:19:05.923105+0000","last_active":"2024-09-18T08:20:44.736473+0000","last_peered":"2024-09-18T08:20:44.736473+0000","last_clean":"2024-09-18T08:20:44.736473+0000","last_became_active":"2024-09-18T08:19:05.908141+0000","last_became_peered":"2024-09-18T08:19:05.908141+0000","last_unstale":"2024-09-18T08:20:44.736473+0000","last_undegraded":"2024-09-18T08:20:44.736473+0000","last_fullsized":"2024-09-18T08:20:44.736473+0000","mapping_epoch":413,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":414,"parent":"0.0","parent_split_bits":0,"last_scrub":"62'1","last_scrub_stamp":"2024-09-18T08:17:34.935618+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:45.855979+0000","last_clean_scrub_stamp":"2024-09-18T08:17:34.935618+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":530,"reported_epoch":445,"state":"active+clean","last_fresh":"2024-09-18T08:20:44.731823+0000","last_change":"2024-09-18T08:19:05.886456+0000","last_active":"2024-09-18T08:20:44.731823+0000","last_peered":"2024-09-18T08:20:44.731823+0000","last_clean":"2024-09-18T08:20:44.731823+0000","last_became_active":"2024-09-18T08:19:05.886174+0000","last_became_peered":"2024-09-18T08:19:05.886174+0000","last_unstale":"2024-09-18T08:20:44.731823+0000","last_undegraded":"2024-09-18T08:20:44.731823+0000","last_fullsized":"2024-09-18T08:20:44.731823+0000","mapping_epoch":413,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":414,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:18:28.112437+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:50.886830+0000","last_clean_scrub_stamp":"2024-09-18T08:18:28.112437+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":514,"reported_epoch":445,"state":"active+clean","last_fresh":"2024-09-18T08:20:44.736133+0000","last_change":"2024-09-18T08:18:27.662433+0000","last_active":"2024-09-18T08:20:44.736133+0000","last_peered":"2024-09-18T08:20:44.736133+0000","last_clean":"2024-09-18T08:20:44.736133+0000","last_became_active":"2024-09-18T08:18:27.662116+0000","last_became_peered":"2024-09-18T08:18:27.662116+0000","last_unstale":"2024-09-18T08:20:44.736133+0000","last_undegraded":"2024-09-18T08:20:44.736133+0000","last_fullsized":"2024-09-18T08:20:44.736133+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:17:35.381401+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_clean_scrub_stamp":"2024-09-18T08:17:35.381401+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":23,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.9","version":"445'9570","reported_seq":9671,"reported_epoch":445,"state":"active+clean","last_fresh":"2024-09-18T08:20:49.849170+0000","last_change":"2024-09-18T08:18:27.655199+0000","last_active":"2024-09-18T08:20:49.849170+0000","last_peered":"2024-09-18T08:20:49.849170+0000","last_clean":"2024-09-18T08:20:49.849170+0000","last_became_active":"2024-09-18T08:18:27.654866+0000","last_became_peered":"2024-09-18T08:18:27.654866+0000","last_unstale":"2024-09-18T08:20:49.849170+0000","last_undegraded":"2024-09-18T08:20:49.849170+0000","last_fullsized":"2024-09-18T08:20:49.849170+0000","mapping_epoch":401,"log_start":"428'5600","ondisk_log_start":"428'5600","created":401,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:18:26.612016+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:18:26.612016+0000","last_clean_scrub_stamp":"2024-09-18T08:18:26.612016+0000","objects_scrubbed":0,"log_size":3970,"log_dups_size":0,"ondisk_log_size":3970,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":177078272,"num_objects":2702,"num_object_clones":0,"num_object_copies":8106,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2702,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9570,"num_write_kb":392704,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,0],"acting":[3,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"74'79","reported_seq":487,"reported_epoch":445,"state":"active+clean","last_fresh":"2024-09-18T08:20:44.734988+0000","last_change":"2024-09-18T08:18:55.603760+0000","last_active":"2024-09-18T08:20:44.734988+0000","last_peered":"2024-09-18T08:20:44.734988+0000","last_clean":"2024-09-18T08:20:44.734988+0000","last_became_active":"2024-09-18T08:16:13.454138+0000","last_became_peered":"2024-09-18T08:16:13.454138+0000","last_unstale":"2024-09-18T08:20:44.734988+0000","last_undegraded":"2024-09-18T08:20:44.734988+0000","last_fullsized":"2024-09-18T08:20:44.734988+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":152,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":3,"last_scrub":"74'79","last_scrub_stamp":"2024-09-18T08:18:55.603698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:00:11.893742+0000","last_clean_scrub_stamp":"2024-09-18T08:18:55.603698+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,5],"acting":[1,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.6","version":"271'24","reported_seq":946,"reported_epoch":445,"state":"active+clean","last_fresh":"2024-09-18T08:20:44.729014+0000","last_change":"2024-09-18T08:20:34.182307+0000","last_active":"2024-09-18T08:20:44.729014+0000","last_peered":"2024-09-18T08:20:44.729014+0000","last_clean":"2024-09-18T08:20:44.729014+0000","last_became_active":"2024-09-18T08:19:05.900310+0000","last_became_peered":"2024-09-18T08:19:05.900310+0000","last_unstale":"2024-09-18T08:20:44.729014+0000","last_undegraded":"2024-09-18T08:20:44.729014+0000","last_fullsized":"2024-09-18T08:20:44.729014+0000","mapping_epoch":413,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":414,"parent":"0.0","parent_split_bits":0,"last_scrub":"271'24","last_scrub_stamp":"2024-09-18T08:20:34.182193+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_clean_scrub_stamp":"2024-09-18T08:20:34.182193+0000","objects_scrubbed":6,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:21:42.029380+0000","scrub_duration":20,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":36,"num_read_kb":30,"num_write":24,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":854,"reported_epoch":445,"state":"active+clean","last_fresh":"2024-09-18T08:20:44.730264+0000","last_change":"2024-09-18T08:20:22.166128+0000","last_active":"2024-09-18T08:20:44.730264+0000","last_peered":"2024-09-18T08:20:44.730264+0000","last_clean":"2024-09-18T08:20:44.730264+0000","last_became_active":"2024-09-18T08:16:13.460391+0000","last_became_peered":"2024-09-18T08:16:13.460391+0000","last_unstale":"2024-09-18T08:20:44.730264+0000","last_undegraded":"2024-09-18T08:20:44.730264+0000","last_fullsized":"2024-09-18T08:20:44.730264+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:20:22.164255+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:46.860092+0000","last_clean_scrub_stamp":"2024-09-18T08:20:22.164255+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:21:47.591456+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,5],"acting":[0,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"70'5","reported_seq":1154,"reported_epoch":445,"state":"active+clean","last_fresh":"2024-09-18T08:20:46.789562+0000","last_change":"2024-09-18T08:18:27.672092+0000","last_active":"2024-09-18T08:20:46.789562+0000","last_peered":"2024-09-18T08:20:46.789562+0000","last_clean":"2024-09-18T08:20:46.789562+0000","last_became_active":"2024-09-18T08:18:27.646445+0000","last_became_peered":"2024-09-18T08:18:27.646445+0000","last_unstale":"2024-09-18T08:20:46.789562+0000","last_undegraded":"2024-09-18T08:20:46.789562+0000","last_fullsized":"2024-09-18T08:20:46.789562+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":0,"last_scrub":"70'5","last_scrub_stamp":"2024-09-18T08:15:57.384224+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:45.855979+0000","last_clean_scrub_stamp":"2024-09-18T08:15:57.384224+0000","objects_scrubbed":1,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0.000504055,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":537,"num_read_kb":532,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":778,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":238,"reported_epoch":445,"state":"active+clean","last_fresh":"2024-09-18T08:20:44.732307+0000","last_change":"2024-09-18T08:20:19.796231+0000","last_active":"2024-09-18T08:20:44.732307+0000","last_peered":"2024-09-18T08:20:44.732307+0000","last_clean":"2024-09-18T08:20:44.732307+0000","last_became_active":"2024-09-18T08:15:41.443080+0000","last_became_peered":"2024-09-18T08:15:41.443080+0000","last_unstale":"2024-09-18T08:20:44.732307+0000","last_undegraded":"2024-09-18T08:20:44.732307+0000","last_fullsized":"2024-09-18T08:20:44.732307+0000","mapping_epoch":336,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":337,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:20:19.796047+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:50.886830+0000","last_clean_scrub_stamp":"2024-09-18T08:20:19.796047+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:21:22.041310+0000","scrub_duration":24,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":634,"reported_epoch":445,"state":"active+clean","last_fresh":"2024-09-18T08:20:44.736112+0000","last_change":"2024-09-18T08:18:16.556819+0000","last_active":"2024-09-18T08:20:44.736112+0000","last_peered":"2024-09-18T08:20:44.736112+0000","last_clean":"2024-09-18T08:20:44.736112+0000","last_became_active":"2024-09-18T08:15:41.436692+0000","last_became_peered":"2024-09-18T08:15:41.436692+0000","last_unstale":"2024-09-18T08:20:44.736112+0000","last_undegraded":"2024-09-18T08:20:44.736112+0000","last_fullsized":"2024-09-18T08:20:44.736112+0000","mapping_epoch":336,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":337,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:18:16.556771+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_clean_scrub_stamp":"2024-09-18T08:18:16.556771+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.a","version":"445'9818","reported_seq":9919,"reported_epoch":445,"state":"active+clean","last_fresh":"2024-09-18T08:20:49.865503+0000","last_change":"2024-09-18T08:18:27.646142+0000","last_active":"2024-09-18T08:20:49.865503+0000","last_peered":"2024-09-18T08:20:49.865503+0000","last_clean":"2024-09-18T08:20:49.865503+0000","last_became_active":"2024-09-18T08:18:27.645863+0000","last_became_peered":"2024-09-18T08:18:27.645863+0000","last_unstale":"2024-09-18T08:20:49.865503+0000","last_undegraded":"2024-09-18T08:20:49.865503+0000","last_fullsized":"2024-09-18T08:20:49.865503+0000","mapping_epoch":401,"log_start":"428'5800","ondisk_log_start":"428'5800","created":401,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:18:26.612016+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:18:26.612016+0000","last_clean_scrub_stamp":"2024-09-18T08:18:26.612016+0000","objects_scrubbed":0,"log_size":4018,"log_dups_size":0,"ondisk_log_size":4018,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":179175424,"num_objects":2734,"num_object_clones":0,"num_object_copies":8202,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2734,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9818,"num_write_kb":401664,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.5","version":"402'36","reported_seq":154,"reported_epoch":445,"state":"active+clean","last_fresh":"2024-09-18T08:20:44.727164+0000","last_change":"2024-09-18T08:18:30.788913+0000","last_active":"2024-09-18T08:20:44.727164+0000","last_peered":"2024-09-18T08:20:44.727164+0000","last_clean":"2024-09-18T08:20:44.727164+0000","last_became_active":"2024-09-18T08:18:27.657630+0000","last_became_peered":"2024-09-18T08:18:27.657630+0000","last_unstale":"2024-09-18T08:20:44.727164+0000","last_undegraded":"2024-09-18T08:20:44.727164+0000","last_fullsized":"2024-09-18T08:20:44.727164+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":0,"last_scrub":"402'36","last_scrub_stamp":"2024-09-18T08:18:30.788799+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_clean_scrub_stamp":"2024-09-18T08:18:30.788799+0000","objects_scrubbed":8,"log_size":36,"log_dups_size":0,"ondisk_log_size":36,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":42,"num_read_kb":32,"num_write":25,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":462,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,0],"acting":[7,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":846,"reported_epoch":445,"state":"active+clean","last_fresh":"2024-09-18T08:20:44.736499+0000","last_change":"2024-09-18T08:20:11.908841+0000","last_active":"2024-09-18T08:20:44.736499+0000","last_peered":"2024-09-18T08:20:44.736499+0000","last_clean":"2024-09-18T08:20:44.736499+0000","last_became_active":"2024-09-18T08:15:41.464313+0000","last_became_peered":"2024-09-18T08:15:41.464313+0000","last_unstale":"2024-09-18T08:20:44.736499+0000","last_undegraded":"2024-09-18T08:20:44.736499+0000","last_fullsized":"2024-09-18T08:20:44.736499+0000","mapping_epoch":336,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":337,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:20:11.908773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:46.860092+0000","last_clean_scrub_stamp":"2024-09-18T08:20:11.908773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:21:21.053391+0000","scrub_duration":68,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"74'79","reported_seq":443,"reported_epoch":445,"state":"active+clean","last_fresh":"2024-09-18T08:20:44.731489+0000","last_change":"2024-09-18T08:18:27.650334+0000","last_active":"2024-09-18T08:20:44.731489+0000","last_peered":"2024-09-18T08:20:44.731489+0000","last_clean":"2024-09-18T08:20:44.731489+0000","last_became_active":"2024-09-18T08:18:27.649964+0000","last_became_peered":"2024-09-18T08:18:27.649964+0000","last_unstale":"2024-09-18T08:20:44.731489+0000","last_undegraded":"2024-09-18T08:20:44.731489+0000","last_fullsized":"2024-09-18T08:20:44.731489+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'79","last_scrub_stamp":"2024-09-18T08:18:18.786155+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:00:11.893742+0000","last_clean_scrub_stamp":"2024-09-18T08:18:18.786155+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,6],"acting":[0,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":697,"reported_epoch":445,"state":"active+clean","last_fresh":"2024-09-18T08:20:44.726524+0000","last_change":"2024-09-18T08:19:05.096869+0000","last_active":"2024-09-18T08:20:44.726524+0000","last_peered":"2024-09-18T08:20:44.726524+0000","last_clean":"2024-09-18T08:20:44.726524+0000","last_became_active":"2024-09-18T08:18:27.653602+0000","last_became_peered":"2024-09-18T08:18:27.653602+0000","last_unstale":"2024-09-18T08:20:44.726524+0000","last_undegraded":"2024-09-18T08:20:44.726524+0000","last_fullsized":"2024-09-18T08:20:44.726524+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:19:05.096805+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:50.886830+0000","last_clean_scrub_stamp":"2024-09-18T08:19:05.096805+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:21:43.081400+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":245,"reported_epoch":445,"state":"active+clean","last_fresh":"2024-09-18T08:20:44.732215+0000","last_change":"2024-09-18T08:18:25.515806+0000","last_active":"2024-09-18T08:20:44.732215+0000","last_peered":"2024-09-18T08:20:44.732215+0000","last_clean":"2024-09-18T08:20:44.732215+0000","last_became_active":"2024-09-18T08:16:13.471911+0000","last_became_peered":"2024-09-18T08:16:13.471911+0000","last_unstale":"2024-09-18T08:20:44.732215+0000","last_undegraded":"2024-09-18T08:20:44.732215+0000","last_fullsized":"2024-09-18T08:20:44.732215+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:18:25.515757+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_clean_scrub_stamp":"2024-09-18T08:18:25.515757+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,1],"acting":[5,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.1","version":"445'9672","reported_seq":9773,"reported_epoch":445,"state":"active+clean","last_fresh":"2024-09-18T08:20:49.814060+0000","last_change":"2024-09-18T08:18:27.651222+0000","last_active":"2024-09-18T08:20:49.814060+0000","last_peered":"2024-09-18T08:20:49.814060+0000","last_clean":"2024-09-18T08:20:49.814060+0000","last_became_active":"2024-09-18T08:18:27.650844+0000","last_became_peered":"2024-09-18T08:18:27.650844+0000","last_unstale":"2024-09-18T08:20:49.814060+0000","last_undegraded":"2024-09-18T08:20:49.814060+0000","last_fullsized":"2024-09-18T08:20:49.814060+0000","mapping_epoch":401,"log_start":"428'5700","ondisk_log_start":"428'5700","created":401,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:18:26.612016+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:18:26.612016+0000","last_clean_scrub_stamp":"2024-09-18T08:18:26.612016+0000","objects_scrubbed":0,"log_size":3972,"log_dups_size":0,"ondisk_log_size":3972,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":176816128,"num_objects":2698,"num_object_clones":0,"num_object_copies":8094,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2698,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9672,"num_write_kb":395840,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,0],"acting":[3,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.e","version":"271'19","reported_seq":527,"reported_epoch":445,"state":"active+clean","last_fresh":"2024-09-18T08:20:44.735838+0000","last_change":"2024-09-18T08:17:30.690655+0000","last_active":"2024-09-18T08:20:44.735838+0000","last_peered":"2024-09-18T08:20:44.735838+0000","last_clean":"2024-09-18T08:20:44.735838+0000","last_became_active":"2024-09-18T08:16:13.458001+0000","last_became_peered":"2024-09-18T08:16:13.458001+0000","last_unstale":"2024-09-18T08:20:44.735838+0000","last_undegraded":"2024-09-18T08:20:44.735838+0000","last_fullsized":"2024-09-18T08:20:44.735838+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"271'19","last_scrub_stamp":"2024-09-18T08:17:30.690581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_clean_scrub_stamp":"2024-09-18T08:17:30.690581+0000","objects_scrubbed":5,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":31,"num_read_kb":24,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,3],"acting":[1,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":486,"reported_epoch":445,"state":"active+clean","last_fresh":"2024-09-18T08:20:44.725774+0000","last_change":"2024-09-18T08:19:05.881635+0000","last_active":"2024-09-18T08:20:44.725774+0000","last_peered":"2024-09-18T08:20:44.725774+0000","last_clean":"2024-09-18T08:20:44.725774+0000","last_became_active":"2024-09-18T08:19:05.881238+0000","last_became_peered":"2024-09-18T08:19:05.881238+0000","last_unstale":"2024-09-18T08:20:44.725774+0000","last_undegraded":"2024-09-18T08:20:44.725774+0000","last_fullsized":"2024-09-18T08:20:44.725774+0000","mapping_epoch":413,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":414,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:18:26.121022+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:46.860092+0000","last_clean_scrub_stamp":"2024-09-18T08:18:26.121022+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,5],"acting":[7,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":252,"reported_epoch":445,"state":"active+clean","last_fresh":"2024-09-18T08:20:44.732130+0000","last_change":"2024-09-18T08:19:05.897861+0000","last_active":"2024-09-18T08:20:44.732130+0000","last_peered":"2024-09-18T08:20:44.732130+0000","last_clean":"2024-09-18T08:20:44.732130+0000","last_became_active":"2024-09-18T08:19:05.897497+0000","last_became_peered":"2024-09-18T08:19:05.897497+0000","last_unstale":"2024-09-18T08:20:44.732130+0000","last_undegraded":"2024-09-18T08:20:44.732130+0000","last_fullsized":"2024-09-18T08:20:44.732130+0000","mapping_epoch":413,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":414,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:17:38.616405+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:50.886830+0000","last_clean_scrub_stamp":"2024-09-18T08:17:38.616405+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:21:41.616183+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,1],"acting":[5,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":833,"reported_epoch":445,"state":"active+clean","last_fresh":"2024-09-18T08:20:44.734141+0000","last_change":"2024-09-18T08:20:19.015270+0000","last_active":"2024-09-18T08:20:44.734141+0000","last_peered":"2024-09-18T08:20:44.734141+0000","last_clean":"2024-09-18T08:20:44.734141+0000","last_became_active":"2024-09-18T08:15:41.464500+0000","last_became_peered":"2024-09-18T08:15:41.464500+0000","last_unstale":"2024-09-18T08:20:44.734141+0000","last_undegraded":"2024-09-18T08:20:44.734141+0000","last_fullsized":"2024-09-18T08:20:44.734141+0000","mapping_epoch":336,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":337,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:20:19.015198+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_clean_scrub_stamp":"2024-09-18T08:20:19.015198+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:21:34.017027+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.2","version":"445'9684","reported_seq":16326,"reported_epoch":445,"state":"active+clean","last_fresh":"2024-09-18T08:20:49.855016+0000","last_change":"2024-09-18T08:20:06.493831+0000","last_active":"2024-09-18T08:20:49.855016+0000","last_peered":"2024-09-18T08:20:49.855016+0000","last_clean":"2024-09-18T08:20:49.855016+0000","last_became_active":"2024-09-18T08:20:06.489999+0000","last_became_peered":"2024-09-18T08:20:06.489999+0000","last_unstale":"2024-09-18T08:20:49.855016+0000","last_undegraded":"2024-09-18T08:20:49.855016+0000","last_fullsized":"2024-09-18T08:20:49.855016+0000","mapping_epoch":431,"log_start":"428'5700","ondisk_log_start":"428'5700","created":401,"last_epoch_clean":432,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:18:26.612016+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:18:26.612016+0000","last_clean_scrub_stamp":"2024-09-18T08:18:26.612016+0000","objects_scrubbed":0,"log_size":3984,"log_dups_size":0,"ondisk_log_size":3984,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":166723584,"num_objects":2544,"num_object_clones":0,"num_object_copies":7632,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2544,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9684,"num_write_kb":391296,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3267,"num_bytes_recovered":214106112,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,7],"acting":[3,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"74'79","reported_seq":250,"reported_epoch":445,"state":"active+clean","last_fresh":"2024-09-18T08:20:44.729368+0000","last_change":"2024-09-18T08:19:01.768944+0000","last_active":"2024-09-18T08:20:44.729368+0000","last_peered":"2024-09-18T08:20:44.729368+0000","last_clean":"2024-09-18T08:20:44.729368+0000","last_became_active":"2024-09-18T08:15:41.454781+0000","last_became_peered":"2024-09-18T08:15:41.454781+0000","last_unstale":"2024-09-18T08:20:44.729368+0000","last_undegraded":"2024-09-18T08:20:44.729368+0000","last_fullsized":"2024-09-18T08:20:44.729368+0000","mapping_epoch":336,"log_start":"0'0","ondisk_log_start":"0'0","created":152,"last_epoch_clean":337,"parent":"0.0","parent_split_bits":4,"last_scrub":"74'79","last_scrub_stamp":"2024-09-18T08:19:01.768882+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:00:11.893742+0000","last_clean_scrub_stamp":"2024-09-18T08:19:01.768882+0000","objects_scrubbed":1,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262688,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":262688,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,3],"acting":[5,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.d","version":"271'31","reported_seq":408,"reported_epoch":445,"state":"active+clean","last_fresh":"2024-09-18T08:20:44.734448+0000","last_change":"2024-09-18T08:20:20.856895+0000","last_active":"2024-09-18T08:20:44.734448+0000","last_peered":"2024-09-18T08:20:44.734448+0000","last_clean":"2024-09-18T08:20:44.734448+0000","last_became_active":"2024-09-18T08:19:05.905244+0000","last_became_peered":"2024-09-18T08:19:05.905244+0000","last_unstale":"2024-09-18T08:20:44.734448+0000","last_undegraded":"2024-09-18T08:20:44.734448+0000","last_fullsized":"2024-09-18T08:20:44.734448+0000","mapping_epoch":413,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":414,"parent":"0.0","parent_split_bits":0,"last_scrub":"271'31","last_scrub_stamp":"2024-09-18T08:20:20.856684+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_clean_scrub_stamp":"2024-09-18T08:20:20.856684+0000","objects_scrubbed":8,"log_size":31,"log_dups_size":0,"ondisk_log_size":31,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:21:43.677208+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":49,"num_read_kb":39,"num_write":32,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,0],"acting":[1,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":490,"reported_epoch":445,"state":"active+clean","last_fresh":"2024-09-18T08:20:44.732933+0000","last_change":"2024-09-18T08:18:18.567839+0000","last_active":"2024-09-18T08:20:44.732933+0000","last_peered":"2024-09-18T08:20:44.732933+0000","last_clean":"2024-09-18T08:20:44.732933+0000","last_became_active":"2024-09-18T08:16:13.483143+0000","last_became_peered":"2024-09-18T08:16:13.483143+0000","last_unstale":"2024-09-18T08:20:44.732933+0000","last_undegraded":"2024-09-18T08:20:44.732933+0000","last_fullsized":"2024-09-18T08:20:44.732933+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:18:18.567791+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:46.860092+0000","last_clean_scrub_stamp":"2024-09-18T08:18:18.567791+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,3],"acting":[1,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":719,"reported_epoch":445,"state":"active+clean","last_fresh":"2024-09-18T08:20:44.735921+0000","last_change":"2024-09-18T08:18:59.652875+0000","last_active":"2024-09-18T08:20:44.735921+0000","last_peered":"2024-09-18T08:20:44.735921+0000","last_clean":"2024-09-18T08:20:44.735921+0000","last_became_active":"2024-09-18T08:16:13.460650+0000","last_became_peered":"2024-09-18T08:16:13.460650+0000","last_unstale":"2024-09-18T08:20:44.735921+0000","last_undegraded":"2024-09-18T08:20:44.735921+0000","last_fullsized":"2024-09-18T08:20:44.735921+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:18:59.652484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:50.886830+0000","last_clean_scrub_stamp":"2024-09-18T08:18:59.652484+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,6],"acting":[1,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":411,"reported_epoch":445,"state":"active+clean","last_fresh":"2024-09-18T08:20:44.735561+0000","last_change":"2024-09-18T08:19:05.895239+0000","last_active":"2024-09-18T08:20:44.735561+0000","last_peered":"2024-09-18T08:20:44.735561+0000","last_clean":"2024-09-18T08:20:44.735561+0000","last_became_active":"2024-09-18T08:19:05.894863+0000","last_became_peered":"2024-09-18T08:19:05.894863+0000","last_unstale":"2024-09-18T08:20:44.735561+0000","last_undegraded":"2024-09-18T08:20:44.735561+0000","last_fullsized":"2024-09-18T08:20:44.735561+0000","mapping_epoch":413,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":414,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:19:02.172378+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_clean_scrub_stamp":"2024-09-18T08:19:02.172378+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,1],"acting":[3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.3","version":"445'9526","reported_seq":9627,"reported_epoch":445,"state":"active+clean","last_fresh":"2024-09-18T08:20:48.586340+0000","last_change":"2024-09-18T08:18:27.658212+0000","last_active":"2024-09-18T08:20:48.586340+0000","last_peered":"2024-09-18T08:20:48.586340+0000","last_clean":"2024-09-18T08:20:48.586340+0000","last_became_active":"2024-09-18T08:18:27.657815+0000","last_became_peered":"2024-09-18T08:18:27.657815+0000","last_unstale":"2024-09-18T08:20:48.586340+0000","last_undegraded":"2024-09-18T08:20:48.586340+0000","last_fullsized":"2024-09-18T08:20:48.586340+0000","mapping_epoch":401,"log_start":"428'5500","ondisk_log_start":"428'5500","created":401,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:18:26.612016+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:18:26.612016+0000","last_clean_scrub_stamp":"2024-09-18T08:18:26.612016+0000","objects_scrubbed":0,"log_size":4026,"log_dups_size":0,"ondisk_log_size":4026,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":188481536,"num_objects":2876,"num_object_clones":0,"num_object_copies":8628,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2876,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9526,"num_write_kb":396864,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.9","version":"74'79","reported_seq":666,"reported_epoch":445,"state":"active+clean","last_fresh":"2024-09-18T08:20:44.737208+0000","last_change":"2024-09-18T08:19:05.908881+0000","last_active":"2024-09-18T08:20:44.737208+0000","last_peered":"2024-09-18T08:20:44.737208+0000","last_clean":"2024-09-18T08:20:44.737208+0000","last_became_active":"2024-09-18T08:19:05.903470+0000","last_became_peered":"2024-09-18T08:19:05.903470+0000","last_unstale":"2024-09-18T08:20:44.737208+0000","last_undegraded":"2024-09-18T08:20:44.737208+0000","last_fullsized":"2024-09-18T08:20:44.737208+0000","mapping_epoch":413,"log_start":"0'0","ondisk_log_start":"0'0","created":152,"last_epoch_clean":414,"parent":"0.0","parent_split_bits":4,"last_scrub":"74'79","last_scrub_stamp":"2024-09-18T08:17:44.331322+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:00:11.893742+0000","last_clean_scrub_stamp":"2024-09-18T08:17:44.331322+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,5],"acting":[3,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.c","version":"271'20","reported_seq":436,"reported_epoch":445,"state":"active+clean","last_fresh":"2024-09-18T08:20:44.735932+0000","last_change":"2024-09-18T08:16:13.526582+0000","last_active":"2024-09-18T08:20:44.735932+0000","last_peered":"2024-09-18T08:20:44.735932+0000","last_clean":"2024-09-18T08:20:44.735932+0000","last_became_active":"2024-09-18T08:16:13.476918+0000","last_became_peered":"2024-09-18T08:16:13.476918+0000","last_unstale":"2024-09-18T08:20:44.735932+0000","last_undegraded":"2024-09-18T08:20:44.735932+0000","last_fullsized":"2024-09-18T08:20:44.735932+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"271'20","last_scrub_stamp":"2024-09-18T08:15:56.673129+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_clean_scrub_stamp":"2024-09-18T08:15:56.673129+0000","objects_scrubbed":0,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":28,"num_read_kb":23,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,1],"acting":[3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":854,"reported_epoch":445,"state":"active+clean","last_fresh":"2024-09-18T08:20:44.734651+0000","last_change":"2024-09-18T08:19:02.699664+0000","last_active":"2024-09-18T08:20:44.734651+0000","last_peered":"2024-09-18T08:20:44.734651+0000","last_clean":"2024-09-18T08:20:44.734651+0000","last_became_active":"2024-09-18T08:16:13.479148+0000","last_became_peered":"2024-09-18T08:16:13.479148+0000","last_unstale":"2024-09-18T08:20:44.734651+0000","last_undegraded":"2024-09-18T08:20:44.734651+0000","last_fullsized":"2024-09-18T08:20:44.734651+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:19:02.699596+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:46.860092+0000","last_clean_scrub_stamp":"2024-09-18T08:19:02.699596+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.d","version":"402'14","reported_seq":1097,"reported_epoch":445,"state":"active+clean","last_fresh":"2024-09-18T08:20:44.726770+0000","last_change":"2024-09-18T08:18:27.675655+0000","last_active":"2024-09-18T08:20:44.726770+0000","last_peered":"2024-09-18T08:20:44.726770+0000","last_clean":"2024-09-18T08:20:44.726770+0000","last_became_active":"2024-09-18T08:18:27.653796+0000","last_became_peered":"2024-09-18T08:18:27.653796+0000","last_unstale":"2024-09-18T08:20:44.726770+0000","last_undegraded":"2024-09-18T08:20:44.726770+0000","last_fullsized":"2024-09-18T08:20:44.726770+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":0,"last_scrub":"338'13","last_scrub_stamp":"2024-09-18T08:17:38.886654+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:50.886830+0000","last_clean_scrub_stamp":"2024-09-18T08:17:38.886654+0000","objects_scrubbed":1,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:20:49.857493+0000","scrub_duration":26,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":504,"reported_epoch":445,"state":"active+clean","last_fresh":"2024-09-18T08:20:44.732915+0000","last_change":"2024-09-18T08:20:24.831854+0000","last_active":"2024-09-18T08:20:44.732915+0000","last_peered":"2024-09-18T08:20:44.732915+0000","last_clean":"2024-09-18T08:20:44.732915+0000","last_became_active":"2024-09-18T08:19:05.886455+0000","last_became_peered":"2024-09-18T08:19:05.886455+0000","last_unstale":"2024-09-18T08:20:44.732915+0000","last_undegraded":"2024-09-18T08:20:44.732915+0000","last_fullsized":"2024-09-18T08:20:44.732915+0000","mapping_epoch":413,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":414,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:20:24.829925+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_clean_scrub_stamp":"2024-09-18T08:20:24.829925+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:21:30.882607+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.4","version":"445'9480","reported_seq":16040,"reported_epoch":445,"state":"active+clean","last_fresh":"2024-09-18T08:20:48.591681+0000","last_change":"2024-09-18T08:20:16.486645+0000","last_active":"2024-09-18T08:20:48.591681+0000","last_peered":"2024-09-18T08:20:48.591681+0000","last_clean":"2024-09-18T08:20:48.591681+0000","last_became_active":"2024-09-18T08:20:16.478529+0000","last_became_peered":"2024-09-18T08:20:16.478529+0000","last_unstale":"2024-09-18T08:20:48.591681+0000","last_undegraded":"2024-09-18T08:20:48.591681+0000","last_fullsized":"2024-09-18T08:20:48.591681+0000","mapping_epoch":437,"log_start":"428'5500","ondisk_log_start":"428'5500","created":401,"last_epoch_clean":438,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:18:26.612016+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:18:26.612016+0000","last_clean_scrub_stamp":"2024-09-18T08:18:26.612016+0000","objects_scrubbed":0,"log_size":3980,"log_dups_size":0,"ondisk_log_size":3980,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":178913280,"num_objects":2730,"num_object_clones":0,"num_object_copies":8190,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2730,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9480,"num_write_kb":390720,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3224,"num_bytes_recovered":211288064,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.b","version":"271'15","reported_seq":917,"reported_epoch":445,"state":"active+clean","last_fresh":"2024-09-18T08:20:44.728532+0000","last_change":"2024-09-18T08:20:39.062134+0000","last_active":"2024-09-18T08:20:44.728532+0000","last_peered":"2024-09-18T08:20:44.728532+0000","last_clean":"2024-09-18T08:20:44.728532+0000","last_became_active":"2024-09-18T08:16:13.480279+0000","last_became_peered":"2024-09-18T08:16:13.480279+0000","last_unstale":"2024-09-18T08:20:44.728532+0000","last_undegraded":"2024-09-18T08:20:44.728532+0000","last_fullsized":"2024-09-18T08:20:44.728532+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"271'15","last_scrub_stamp":"2024-09-18T08:20:39.062041+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_clean_scrub_stamp":"2024-09-18T08:20:39.062041+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:22:08.523632+0000","scrub_duration":21,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,3],"acting":[0,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":343,"reported_epoch":445,"state":"active+clean","last_fresh":"2024-09-18T08:20:44.735174+0000","last_change":"2024-09-18T08:18:22.633544+0000","last_active":"2024-09-18T08:20:44.735174+0000","last_peered":"2024-09-18T08:20:44.735174+0000","last_clean":"2024-09-18T08:20:44.735174+0000","last_became_active":"2024-09-18T08:18:17.892088+0000","last_became_peered":"2024-09-18T08:18:17.892088+0000","last_unstale":"2024-09-18T08:20:44.735174+0000","last_undegraded":"2024-09-18T08:20:44.735174+0000","last_fullsized":"2024-09-18T08:20:44.735174+0000","mapping_epoch":395,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":396,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:18:22.633492+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:46.860092+0000","last_clean_scrub_stamp":"2024-09-18T08:18:22.633492+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,6],"acting":[3,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":694,"reported_epoch":445,"state":"active+clean","last_fresh":"2024-09-18T08:20:44.736038+0000","last_change":"2024-09-18T08:19:05.909652+0000","last_active":"2024-09-18T08:20:44.736038+0000","last_peered":"2024-09-18T08:20:44.736038+0000","last_clean":"2024-09-18T08:20:44.736038+0000","last_became_active":"2024-09-18T08:19:05.907780+0000","last_became_peered":"2024-09-18T08:19:05.907780+0000","last_unstale":"2024-09-18T08:20:44.736038+0000","last_undegraded":"2024-09-18T08:20:44.736038+0000","last_fullsized":"2024-09-18T08:20:44.736038+0000","mapping_epoch":413,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":414,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:18:19.057302+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:50.886830+0000","last_clean_scrub_stamp":"2024-09-18T08:18:19.057302+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":857,"reported_epoch":445,"state":"active+clean","last_fresh":"2024-09-18T08:20:44.730182+0000","last_change":"2024-09-18T08:19:05.912963+0000","last_active":"2024-09-18T08:20:44.730182+0000","last_peered":"2024-09-18T08:20:44.730182+0000","last_clean":"2024-09-18T08:20:44.730182+0000","last_became_active":"2024-09-18T08:19:05.912544+0000","last_became_peered":"2024-09-18T08:19:05.912544+0000","last_unstale":"2024-09-18T08:20:44.730182+0000","last_undegraded":"2024-09-18T08:20:44.730182+0000","last_fullsized":"2024-09-18T08:20:44.730182+0000","mapping_epoch":413,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":414,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:18:28.901610+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_clean_scrub_stamp":"2024-09-18T08:18:28.901610+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.5","version":"445'9417","reported_seq":9518,"reported_epoch":445,"state":"active+clean","last_fresh":"2024-09-18T08:20:48.748606+0000","last_change":"2024-09-18T08:18:27.643971+0000","last_active":"2024-09-18T08:20:48.748606+0000","last_peered":"2024-09-18T08:20:48.748606+0000","last_clean":"2024-09-18T08:20:48.748606+0000","last_became_active":"2024-09-18T08:18:27.642478+0000","last_became_peered":"2024-09-18T08:18:27.642478+0000","last_unstale":"2024-09-18T08:20:48.748606+0000","last_undegraded":"2024-09-18T08:20:48.748606+0000","last_fullsized":"2024-09-18T08:20:48.748606+0000","mapping_epoch":401,"log_start":"428'5200","ondisk_log_start":"428'5200","created":401,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:18:26.612016+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:18:26.612016+0000","last_clean_scrub_stamp":"2024-09-18T08:18:26.612016+0000","objects_scrubbed":0,"log_size":4217,"log_dups_size":0,"ondisk_log_size":4217,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":180027392,"num_objects":2747,"num_object_clones":0,"num_object_copies":8241,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2747,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9417,"num_write_kb":389248,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.a","version":"271'35","reported_seq":776,"reported_epoch":445,"state":"active+clean","last_fresh":"2024-09-18T08:20:44.733258+0000","last_change":"2024-09-18T08:18:27.683070+0000","last_active":"2024-09-18T08:20:44.733258+0000","last_peered":"2024-09-18T08:20:44.733258+0000","last_clean":"2024-09-18T08:20:44.733258+0000","last_became_active":"2024-09-18T08:18:27.652693+0000","last_became_peered":"2024-09-18T08:18:27.652693+0000","last_unstale":"2024-09-18T08:20:44.733258+0000","last_undegraded":"2024-09-18T08:20:44.733258+0000","last_fullsized":"2024-09-18T08:20:44.733258+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":0,"last_scrub":"271'35","last_scrub_stamp":"2024-09-18T08:18:23.687385+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_clean_scrub_stamp":"2024-09-18T08:18:23.687385+0000","objects_scrubbed":9,"log_size":35,"log_dups_size":0,"ondisk_log_size":35,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":55,"num_read_kb":44,"num_write":36,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":53,"num_bytes_recovered":550,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,7],"acting":[1,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":245,"reported_epoch":445,"state":"active+clean","last_fresh":"2024-09-18T08:20:44.734201+0000","last_change":"2024-09-18T08:20:15.848074+0000","last_active":"2024-09-18T08:20:44.734201+0000","last_peered":"2024-09-18T08:20:44.734201+0000","last_clean":"2024-09-18T08:20:44.734201+0000","last_became_active":"2024-09-18T08:18:27.643663+0000","last_became_peered":"2024-09-18T08:18:27.643663+0000","last_unstale":"2024-09-18T08:20:44.734201+0000","last_undegraded":"2024-09-18T08:20:44.734201+0000","last_fullsized":"2024-09-18T08:20:44.734201+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:20:15.848022+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:46.860092+0000","last_clean_scrub_stamp":"2024-09-18T08:20:15.848022+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:21:17.902304+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":516,"reported_epoch":445,"state":"active+clean","last_fresh":"2024-09-18T08:20:44.733906+0000","last_change":"2024-09-18T08:19:05.905542+0000","last_active":"2024-09-18T08:20:44.733906+0000","last_peered":"2024-09-18T08:20:44.733906+0000","last_clean":"2024-09-18T08:20:44.733906+0000","last_became_active":"2024-09-18T08:19:05.903320+0000","last_became_peered":"2024-09-18T08:19:05.903320+0000","last_unstale":"2024-09-18T08:20:44.733906+0000","last_undegraded":"2024-09-18T08:20:44.733906+0000","last_fullsized":"2024-09-18T08:20:44.733906+0000","mapping_epoch":413,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":414,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:18:22.116686+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:50.886830+0000","last_clean_scrub_stamp":"2024-09-18T08:18:22.116686+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,7],"acting":[3,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":653,"reported_epoch":445,"state":"active+clean","last_fresh":"2024-09-18T08:20:44.740310+0000","last_change":"2024-09-18T08:19:05.890818+0000","last_active":"2024-09-18T08:20:44.740310+0000","last_peered":"2024-09-18T08:20:44.740310+0000","last_clean":"2024-09-18T08:20:44.740310+0000","last_became_active":"2024-09-18T08:19:05.890070+0000","last_became_peered":"2024-09-18T08:19:05.890070+0000","last_unstale":"2024-09-18T08:20:44.740310+0000","last_undegraded":"2024-09-18T08:20:44.740310+0000","last_fullsized":"2024-09-18T08:20:44.740310+0000","mapping_epoch":413,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":414,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:18:15.051582+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_clean_scrub_stamp":"2024-09-18T08:18:15.051582+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.6","version":"445'9452","reported_seq":15865,"reported_epoch":445,"state":"active+clean","last_fresh":"2024-09-18T08:20:48.265037+0000","last_change":"2024-09-18T08:20:05.552082+0000","last_active":"2024-09-18T08:20:48.265037+0000","last_peered":"2024-09-18T08:20:48.265037+0000","last_clean":"2024-09-18T08:20:48.265037+0000","last_became_active":"2024-09-18T08:20:05.509531+0000","last_became_peered":"2024-09-18T08:20:05.509531+0000","last_unstale":"2024-09-18T08:20:48.265037+0000","last_undegraded":"2024-09-18T08:20:48.265037+0000","last_fullsized":"2024-09-18T08:20:48.265037+0000","mapping_epoch":430,"log_start":"428'5600","ondisk_log_start":"428'5600","created":401,"last_epoch_clean":431,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:18:26.612016+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:18:26.612016+0000","last_clean_scrub_stamp":"2024-09-18T08:18:26.612016+0000","objects_scrubbed":0,"log_size":3852,"log_dups_size":0,"ondisk_log_size":3852,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":187301888,"num_objects":2858,"num_object_clones":0,"num_object_copies":8574,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2858,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9452,"num_write_kb":393920,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3150,"num_bytes_recovered":206438400,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,5],"acting":[0,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.9","version":"271'18","reported_seq":557,"reported_epoch":445,"state":"active+clean","last_fresh":"2024-09-18T08:20:44.733655+0000","last_change":"2024-09-18T08:20:28.912488+0000","last_active":"2024-09-18T08:20:44.733655+0000","last_peered":"2024-09-18T08:20:44.733655+0000","last_clean":"2024-09-18T08:20:44.733655+0000","last_became_active":"2024-09-18T08:16:13.482580+0000","last_became_peered":"2024-09-18T08:16:13.482580+0000","last_unstale":"2024-09-18T08:20:44.733655+0000","last_undegraded":"2024-09-18T08:20:44.733655+0000","last_fullsized":"2024-09-18T08:20:44.733655+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"271'18","last_scrub_stamp":"2024-09-18T08:20:28.912409+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_clean_scrub_stamp":"2024-09-18T08:20:28.912409+0000","objects_scrubbed":5,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:21:47.673882+0000","scrub_duration":24,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":33,"num_read_kb":24,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":880,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,6],"acting":[1,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":107,"reported_epoch":445,"state":"active+clean","last_fresh":"2024-09-18T08:20:44.726290+0000","last_change":"2024-09-18T08:18:56.624230+0000","last_active":"2024-09-18T08:20:44.726290+0000","last_peered":"2024-09-18T08:20:44.726290+0000","last_clean":"2024-09-18T08:20:44.726290+0000","last_became_active":"2024-09-18T08:18:27.654488+0000","last_became_peered":"2024-09-18T08:18:27.654488+0000","last_unstale":"2024-09-18T08:20:44.726290+0000","last_undegraded":"2024-09-18T08:20:44.726290+0000","last_fullsized":"2024-09-18T08:20:44.726290+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:18:56.624152+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:46.860092+0000","last_clean_scrub_stamp":"2024-09-18T08:18:56.624152+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":845,"reported_epoch":445,"state":"active+clean","last_fresh":"2024-09-18T08:20:44.726070+0000","last_change":"2024-09-18T08:20:31.957814+0000","last_active":"2024-09-18T08:20:44.726070+0000","last_peered":"2024-09-18T08:20:44.726070+0000","last_clean":"2024-09-18T08:20:44.726070+0000","last_became_active":"2024-09-18T08:14:39.399301+0000","last_became_peered":"2024-09-18T08:14:39.399301+0000","last_unstale":"2024-09-18T08:20:44.726070+0000","last_undegraded":"2024-09-18T08:20:44.726070+0000","last_fullsized":"2024-09-18T08:20:44.726070+0000","mapping_epoch":307,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":308,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:20:31.957724+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:50.886830+0000","last_clean_scrub_stamp":"2024-09-18T08:20:31.957724+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:21:53.385222+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":837,"reported_epoch":445,"state":"active+clean","last_fresh":"2024-09-18T08:20:44.734264+0000","last_change":"2024-09-18T08:18:27.665000+0000","last_active":"2024-09-18T08:20:44.734264+0000","last_peered":"2024-09-18T08:20:44.734264+0000","last_clean":"2024-09-18T08:20:44.734264+0000","last_became_active":"2024-09-18T08:18:27.664412+0000","last_became_peered":"2024-09-18T08:18:27.664412+0000","last_unstale":"2024-09-18T08:20:44.734264+0000","last_undegraded":"2024-09-18T08:20:44.734264+0000","last_fullsized":"2024-09-18T08:20:44.734264+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:17:09.487823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_clean_scrub_stamp":"2024-09-18T08:17:09.487823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.7","version":"445'9597","reported_seq":9698,"reported_epoch":445,"state":"active+clean","last_fresh":"2024-09-18T08:20:48.745699+0000","last_change":"2024-09-18T08:18:27.651740+0000","last_active":"2024-09-18T08:20:48.745699+0000","last_peered":"2024-09-18T08:20:48.745699+0000","last_clean":"2024-09-18T08:20:48.745699+0000","last_became_active":"2024-09-18T08:18:27.651411+0000","last_became_peered":"2024-09-18T08:18:27.651411+0000","last_unstale":"2024-09-18T08:20:48.745699+0000","last_undegraded":"2024-09-18T08:20:48.745699+0000","last_fullsized":"2024-09-18T08:20:48.745699+0000","mapping_epoch":401,"log_start":"428'5400","ondisk_log_start":"428'5400","created":401,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:18:26.612016+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:18:26.612016+0000","last_clean_scrub_stamp":"2024-09-18T08:18:26.612016+0000","objects_scrubbed":0,"log_size":4197,"log_dups_size":0,"ondisk_log_size":4197,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":187760640,"num_objects":2865,"num_object_clones":0,"num_object_copies":8595,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2865,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9597,"num_write_kb":398784,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.8","version":"271'27","reported_seq":300,"reported_epoch":445,"state":"active+clean","last_fresh":"2024-09-18T08:20:44.729381+0000","last_change":"2024-09-18T08:20:16.825021+0000","last_active":"2024-09-18T08:20:44.729381+0000","last_peered":"2024-09-18T08:20:44.729381+0000","last_clean":"2024-09-18T08:20:44.729381+0000","last_became_active":"2024-09-18T08:18:27.643053+0000","last_became_peered":"2024-09-18T08:18:27.643053+0000","last_unstale":"2024-09-18T08:20:44.729381+0000","last_undegraded":"2024-09-18T08:20:44.729381+0000","last_fullsized":"2024-09-18T08:20:44.729381+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":0,"last_scrub":"271'27","last_scrub_stamp":"2024-09-18T08:20:16.824939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_clean_scrub_stamp":"2024-09-18T08:20:16.824939+0000","objects_scrubbed":7,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:21:45.559934+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":44,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":35,"num_bytes_recovered":550,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.f","version":"62'2","reported_seq":105,"reported_epoch":445,"state":"active+clean","last_fresh":"2024-09-18T08:20:44.726715+0000","last_change":"2024-09-18T08:18:27.672015+0000","last_active":"2024-09-18T08:20:44.726715+0000","last_peered":"2024-09-18T08:20:44.726715+0000","last_clean":"2024-09-18T08:20:44.726715+0000","last_became_active":"2024-09-18T08:18:27.654687+0000","last_became_peered":"2024-09-18T08:18:27.654687+0000","last_unstale":"2024-09-18T08:20:44.726715+0000","last_undegraded":"2024-09-18T08:20:44.726715+0000","last_fullsized":"2024-09-18T08:20:44.726715+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":0,"last_scrub":"62'2","last_scrub_stamp":"2024-09-18T08:17:49.718261+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:46.860092+0000","last_clean_scrub_stamp":"2024-09-18T08:17:49.718261+0000","objects_scrubbed":2,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":184,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,1],"acting":[7,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.9","version":"402'18","reported_seq":140,"reported_epoch":445,"state":"active+clean","last_fresh":"2024-09-18T08:20:44.726166+0000","last_change":"2024-09-18T08:20:24.535623+0000","last_active":"2024-09-18T08:20:44.726166+0000","last_peered":"2024-09-18T08:20:44.726166+0000","last_clean":"2024-09-18T08:20:44.726166+0000","last_became_active":"2024-09-18T08:18:27.652480+0000","last_became_peered":"2024-09-18T08:18:27.652480+0000","last_unstale":"2024-09-18T08:20:44.726166+0000","last_undegraded":"2024-09-18T08:20:44.726166+0000","last_fullsized":"2024-09-18T08:20:44.726166+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":0,"last_scrub":"402'18","last_scrub_stamp":"2024-09-18T08:20:24.535554+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:50.886830+0000","last_clean_scrub_stamp":"2024-09-18T08:20:24.535554+0000","objects_scrubbed":1,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:21:34.761382+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":238,"reported_epoch":445,"state":"active+clean","last_fresh":"2024-09-18T08:20:44.731138+0000","last_change":"2024-09-18T08:19:04.680209+0000","last_active":"2024-09-18T08:20:44.731138+0000","last_peered":"2024-09-18T08:20:44.731138+0000","last_clean":"2024-09-18T08:20:44.731138+0000","last_became_active":"2024-09-18T08:15:41.449889+0000","last_became_peered":"2024-09-18T08:15:41.449889+0000","last_unstale":"2024-09-18T08:20:44.731138+0000","last_undegraded":"2024-09-18T08:20:44.731138+0000","last_fullsized":"2024-09-18T08:20:44.731138+0000","mapping_epoch":336,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":337,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:19:04.680153+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_clean_scrub_stamp":"2024-09-18T08:19:04.680153+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":756,"reported_epoch":445,"state":"active+clean","last_fresh":"2024-09-18T08:20:44.731275+0000","last_change":"2024-09-18T08:19:03.092160+0000","last_active":"2024-09-18T08:20:44.731275+0000","last_peered":"2024-09-18T08:20:44.731275+0000","last_clean":"2024-09-18T08:20:44.731275+0000","last_became_active":"2024-09-18T08:15:41.446544+0000","last_became_peered":"2024-09-18T08:15:41.446544+0000","last_unstale":"2024-09-18T08:20:44.731275+0000","last_undegraded":"2024-09-18T08:20:44.731275+0000","last_fullsized":"2024-09-18T08:20:44.731275+0000","mapping_epoch":336,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":337,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:19:03.092077+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:46.860092+0000","last_clean_scrub_stamp":"2024-09-18T08:19:03.092077+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.17","version":"271'12","reported_seq":887,"reported_epoch":445,"state":"active+clean","last_fresh":"2024-09-18T08:20:44.735897+0000","last_change":"2024-09-18T08:18:27.699070+0000","last_active":"2024-09-18T08:20:44.735897+0000","last_peered":"2024-09-18T08:20:44.735897+0000","last_clean":"2024-09-18T08:20:44.735897+0000","last_became_active":"2024-09-18T08:18:27.663338+0000","last_became_peered":"2024-09-18T08:18:27.663338+0000","last_unstale":"2024-09-18T08:20:44.735897+0000","last_undegraded":"2024-09-18T08:20:44.735897+0000","last_fullsized":"2024-09-18T08:20:44.735897+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":0,"last_scrub":"271'12","last_scrub_stamp":"2024-09-18T08:14:33.700931+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_clean_scrub_stamp":"2024-09-18T08:14:33.700931+0000","objects_scrubbed":3,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:21:07.875738+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":18,"num_read_kb":15,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":238,"reported_epoch":445,"state":"active+clean","last_fresh":"2024-09-18T08:20:44.732270+0000","last_change":"2024-09-18T08:18:58.734315+0000","last_active":"2024-09-18T08:20:44.732270+0000","last_peered":"2024-09-18T08:20:44.732270+0000","last_clean":"2024-09-18T08:20:44.732270+0000","last_became_active":"2024-09-18T08:15:41.445332+0000","last_became_peered":"2024-09-18T08:15:41.445332+0000","last_unstale":"2024-09-18T08:20:44.732270+0000","last_undegraded":"2024-09-18T08:20:44.732270+0000","last_fullsized":"2024-09-18T08:20:44.732270+0000","mapping_epoch":336,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":337,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:18:58.734241+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:50.886830+0000","last_clean_scrub_stamp":"2024-09-18T08:18:58.734241+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,1],"acting":[5,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":106,"reported_epoch":445,"state":"active+clean","last_fresh":"2024-09-18T08:20:44.726761+0000","last_change":"2024-09-18T08:18:29.808538+0000","last_active":"2024-09-18T08:20:44.726761+0000","last_peered":"2024-09-18T08:20:44.726761+0000","last_clean":"2024-09-18T08:20:44.726761+0000","last_became_active":"2024-09-18T08:18:27.653368+0000","last_became_peered":"2024-09-18T08:18:27.653368+0000","last_unstale":"2024-09-18T08:20:44.726761+0000","last_undegraded":"2024-09-18T08:20:44.726761+0000","last_fullsized":"2024-09-18T08:20:44.726761+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:18:29.808476+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_clean_scrub_stamp":"2024-09-18T08:18:29.808476+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,5],"acting":[7,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"271'15","reported_seq":879,"reported_epoch":445,"state":"active+clean","last_fresh":"2024-09-18T08:20:44.729175+0000","last_change":"2024-09-18T08:18:59.080889+0000","last_active":"2024-09-18T08:20:44.729175+0000","last_peered":"2024-09-18T08:20:44.729175+0000","last_clean":"2024-09-18T08:20:44.729175+0000","last_became_active":"2024-09-18T08:18:27.658637+0000","last_became_peered":"2024-09-18T08:18:27.658637+0000","last_unstale":"2024-09-18T08:20:44.729175+0000","last_undegraded":"2024-09-18T08:20:44.729175+0000","last_fullsized":"2024-09-18T08:20:44.729175+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":0,"last_scrub":"271'15","last_scrub_stamp":"2024-09-18T08:18:59.080655+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_clean_scrub_stamp":"2024-09-18T08:18:59.080655+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":478,"reported_epoch":445,"state":"active+clean","last_fresh":"2024-09-18T08:20:44.729872+0000","last_change":"2024-09-18T08:20:40.065317+0000","last_active":"2024-09-18T08:20:44.729872+0000","last_peered":"2024-09-18T08:20:44.729872+0000","last_clean":"2024-09-18T08:20:44.729872+0000","last_became_active":"2024-09-18T08:18:27.649876+0000","last_became_peered":"2024-09-18T08:18:27.649876+0000","last_unstale":"2024-09-18T08:20:44.729872+0000","last_undegraded":"2024-09-18T08:20:44.729872+0000","last_fullsized":"2024-09-18T08:20:44.729872+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:20:40.064873+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:46.860092+0000","last_clean_scrub_stamp":"2024-09-18T08:20:40.064873+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:21:45.995792+0000","scrub_duration":16,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,6],"acting":[0,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":832,"reported_epoch":445,"state":"active+clean","last_fresh":"2024-09-18T08:20:44.735646+0000","last_change":"2024-09-18T08:18:27.660413+0000","last_active":"2024-09-18T08:20:44.735646+0000","last_peered":"2024-09-18T08:20:44.735646+0000","last_clean":"2024-09-18T08:20:44.735646+0000","last_became_active":"2024-09-18T08:18:27.660093+0000","last_became_peered":"2024-09-18T08:18:27.660093+0000","last_unstale":"2024-09-18T08:20:44.735646+0000","last_undegraded":"2024-09-18T08:20:44.735646+0000","last_fullsized":"2024-09-18T08:20:44.735646+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:17:06.569828+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:50.886830+0000","last_clean_scrub_stamp":"2024-09-18T08:17:06.569828+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":31,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":819,"reported_epoch":445,"state":"active+clean","last_fresh":"2024-09-18T08:20:44.726499+0000","last_change":"2024-09-18T08:19:05.895365+0000","last_active":"2024-09-18T08:20:44.726499+0000","last_peered":"2024-09-18T08:20:44.726499+0000","last_clean":"2024-09-18T08:20:44.726499+0000","last_became_active":"2024-09-18T08:19:05.894972+0000","last_became_peered":"2024-09-18T08:19:05.894972+0000","last_unstale":"2024-09-18T08:20:44.726499+0000","last_undegraded":"2024-09-18T08:20:44.726499+0000","last_fullsized":"2024-09-18T08:20:44.726499+0000","mapping_epoch":413,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":414,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:18:31.070711+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_clean_scrub_stamp":"2024-09-18T08:18:31.070711+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":22,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,5],"acting":[7,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.15","version":"271'15","reported_seq":265,"reported_epoch":445,"state":"active+clean","last_fresh":"2024-09-18T08:20:44.734405+0000","last_change":"2024-09-18T08:18:27.674004+0000","last_active":"2024-09-18T08:20:44.734405+0000","last_peered":"2024-09-18T08:20:44.734405+0000","last_clean":"2024-09-18T08:20:44.734405+0000","last_became_active":"2024-09-18T08:18:27.646211+0000","last_became_peered":"2024-09-18T08:18:27.646211+0000","last_unstale":"2024-09-18T08:20:44.734405+0000","last_undegraded":"2024-09-18T08:20:44.734405+0000","last_fullsized":"2024-09-18T08:20:44.734405+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":0,"last_scrub":"271'15","last_scrub_stamp":"2024-09-18T08:17:32.638811+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_clean_scrub_stamp":"2024-09-18T08:17:32.638811+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:21:42.665587+0000","scrub_duration":20,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":25,"num_read_kb":19,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":851,"reported_epoch":445,"state":"active+clean","last_fresh":"2024-09-18T08:20:44.728965+0000","last_change":"2024-09-18T08:18:27.656075+0000","last_active":"2024-09-18T08:20:44.728965+0000","last_peered":"2024-09-18T08:20:44.728965+0000","last_clean":"2024-09-18T08:20:44.728965+0000","last_became_active":"2024-09-18T08:18:27.655758+0000","last_became_peered":"2024-09-18T08:18:27.655758+0000","last_unstale":"2024-09-18T08:20:44.728965+0000","last_undegraded":"2024-09-18T08:20:44.728965+0000","last_fullsized":"2024-09-18T08:20:44.728965+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:18:24.804845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:46.860092+0000","last_clean_scrub_stamp":"2024-09-18T08:18:24.804845+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.14","version":"415'18","reported_seq":1115,"reported_epoch":445,"state":"active+clean","last_fresh":"2024-09-18T08:20:44.734297+0000","last_change":"2024-09-18T08:19:05.916229+0000","last_active":"2024-09-18T08:20:44.734297+0000","last_peered":"2024-09-18T08:20:44.734297+0000","last_clean":"2024-09-18T08:20:44.734297+0000","last_became_active":"2024-09-18T08:19:05.900155+0000","last_became_peered":"2024-09-18T08:19:05.900155+0000","last_unstale":"2024-09-18T08:20:44.734297+0000","last_undegraded":"2024-09-18T08:20:44.734297+0000","last_fullsized":"2024-09-18T08:20:44.734297+0000","mapping_epoch":413,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":414,"parent":"0.0","parent_split_bits":0,"last_scrub":"310'16","last_scrub_stamp":"2024-09-18T08:17:39.424098+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:50.886830+0000","last_clean_scrub_stamp":"2024-09-18T08:17:39.424098+0000","objects_scrubbed":1,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":16,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":559,"reported_epoch":445,"state":"active+clean","last_fresh":"2024-09-18T08:20:44.726989+0000","last_change":"2024-09-18T08:18:24.109825+0000","last_active":"2024-09-18T08:20:44.726989+0000","last_peered":"2024-09-18T08:20:44.726989+0000","last_clean":"2024-09-18T08:20:44.726989+0000","last_became_active":"2024-09-18T08:16:13.467534+0000","last_became_peered":"2024-09-18T08:16:13.467534+0000","last_unstale":"2024-09-18T08:20:44.726989+0000","last_undegraded":"2024-09-18T08:20:44.726989+0000","last_fullsized":"2024-09-18T08:20:44.726989+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:18:24.109778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_clean_scrub_stamp":"2024-09-18T08:18:24.109778+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:21:42.056021+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,0],"acting":[2,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.14","version":"271'20","reported_seq":913,"reported_epoch":445,"state":"active+clean","last_fresh":"2024-09-18T08:20:44.734647+0000","last_change":"2024-09-18T08:20:36.019623+0000","last_active":"2024-09-18T08:20:44.734647+0000","last_peered":"2024-09-18T08:20:44.734647+0000","last_clean":"2024-09-18T08:20:44.734647+0000","last_became_active":"2024-09-18T08:18:27.663877+0000","last_became_peered":"2024-09-18T08:18:27.663877+0000","last_unstale":"2024-09-18T08:20:44.734647+0000","last_undegraded":"2024-09-18T08:20:44.734647+0000","last_fullsized":"2024-09-18T08:20:44.734647+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":0,"last_scrub":"271'20","last_scrub_stamp":"2024-09-18T08:20:36.019549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_clean_scrub_stamp":"2024-09-18T08:20:36.019549+0000","objects_scrubbed":5,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:21:50.345313+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":25,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,7],"acting":[3,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":106,"reported_epoch":445,"state":"active+clean","last_fresh":"2024-09-18T08:20:44.727091+0000","last_change":"2024-09-18T08:19:05.900476+0000","last_active":"2024-09-18T08:20:44.727091+0000","last_peered":"2024-09-18T08:20:44.727091+0000","last_clean":"2024-09-18T08:20:44.727091+0000","last_became_active":"2024-09-18T08:19:05.900189+0000","last_became_peered":"2024-09-18T08:19:05.900189+0000","last_unstale":"2024-09-18T08:20:44.727091+0000","last_undegraded":"2024-09-18T08:20:44.727091+0000","last_fullsized":"2024-09-18T08:20:44.727091+0000","mapping_epoch":413,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":414,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:17:30.908835+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:46.860092+0000","last_clean_scrub_stamp":"2024-09-18T08:17:30.908835+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:21:44.645814+0000","scrub_duration":21,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,0],"acting":[7,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.15","version":"338'12","reported_seq":304,"reported_epoch":445,"state":"active+clean","last_fresh":"2024-09-18T08:20:44.733979+0000","last_change":"2024-09-18T08:18:24.511983+0000","last_active":"2024-09-18T08:20:44.733979+0000","last_peered":"2024-09-18T08:20:44.733979+0000","last_clean":"2024-09-18T08:20:44.733979+0000","last_became_active":"2024-09-18T08:15:41.455243+0000","last_became_peered":"2024-09-18T08:15:41.455243+0000","last_unstale":"2024-09-18T08:20:44.733979+0000","last_undegraded":"2024-09-18T08:20:44.733979+0000","last_fullsized":"2024-09-18T08:20:44.733979+0000","mapping_epoch":336,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":337,"parent":"0.0","parent_split_bits":0,"last_scrub":"338'12","last_scrub_stamp":"2024-09-18T08:18:24.511895+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:50.886830+0000","last_clean_scrub_stamp":"2024-09-18T08:18:24.511895+0000","objects_scrubbed":1,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":834,"reported_epoch":445,"state":"active+clean","last_fresh":"2024-09-18T08:20:44.728536+0000","last_change":"2024-09-18T08:19:00.044481+0000","last_active":"2024-09-18T08:20:44.728536+0000","last_peered":"2024-09-18T08:20:44.728536+0000","last_clean":"2024-09-18T08:20:44.728536+0000","last_became_active":"2024-09-18T08:18:27.658009+0000","last_became_peered":"2024-09-18T08:18:27.658009+0000","last_unstale":"2024-09-18T08:20:44.728536+0000","last_undegraded":"2024-09-18T08:20:44.728536+0000","last_fullsized":"2024-09-18T08:20:44.728536+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:19:00.044416+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_clean_scrub_stamp":"2024-09-18T08:19:00.044416+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.13","version":"271'19","reported_seq":526,"reported_epoch":445,"state":"active+clean","last_fresh":"2024-09-18T08:20:44.734843+0000","last_change":"2024-09-18T08:17:39.665796+0000","last_active":"2024-09-18T08:20:44.734843+0000","last_peered":"2024-09-18T08:20:44.734843+0000","last_clean":"2024-09-18T08:20:44.734843+0000","last_became_active":"2024-09-18T08:16:13.454640+0000","last_became_peered":"2024-09-18T08:16:13.454640+0000","last_unstale":"2024-09-18T08:20:44.734843+0000","last_undegraded":"2024-09-18T08:20:44.734843+0000","last_fullsized":"2024-09-18T08:20:44.734843+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"271'19","last_scrub_stamp":"2024-09-18T08:17:39.665588+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_clean_scrub_stamp":"2024-09-18T08:17:39.665588+0000","objects_scrubbed":5,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":31,"num_read_kb":24,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,5],"acting":[1,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":99,"reported_epoch":445,"state":"active+clean","last_fresh":"2024-09-18T08:20:44.726759+0000","last_change":"2024-09-18T08:18:27.658805+0000","last_active":"2024-09-18T08:20:44.726759+0000","last_peered":"2024-09-18T08:20:44.726759+0000","last_clean":"2024-09-18T08:20:44.726759+0000","last_became_active":"2024-09-18T08:18:27.658476+0000","last_became_peered":"2024-09-18T08:18:27.658476+0000","last_unstale":"2024-09-18T08:20:44.726759+0000","last_undegraded":"2024-09-18T08:20:44.726759+0000","last_fullsized":"2024-09-18T08:20:44.726759+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:18:25.097909+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:46.860092+0000","last_clean_scrub_stamp":"2024-09-18T08:18:25.097909+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":711,"reported_epoch":445,"state":"active+clean","last_fresh":"2024-09-18T08:20:44.732665+0000","last_change":"2024-09-18T08:20:39.777548+0000","last_active":"2024-09-18T08:20:44.732665+0000","last_peered":"2024-09-18T08:20:44.732665+0000","last_clean":"2024-09-18T08:20:44.732665+0000","last_became_active":"2024-09-18T08:15:41.438048+0000","last_became_peered":"2024-09-18T08:15:41.438048+0000","last_unstale":"2024-09-18T08:20:44.732665+0000","last_undegraded":"2024-09-18T08:20:44.732665+0000","last_fullsized":"2024-09-18T08:20:44.732665+0000","mapping_epoch":336,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":337,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:20:39.777472+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:50.886830+0000","last_clean_scrub_stamp":"2024-09-18T08:20:39.777472+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:22:09.556867+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":836,"reported_epoch":445,"state":"active+clean","last_fresh":"2024-09-18T08:20:44.738627+0000","last_change":"2024-09-18T08:17:37.394735+0000","last_active":"2024-09-18T08:20:44.738627+0000","last_peered":"2024-09-18T08:20:44.738627+0000","last_clean":"2024-09-18T08:20:44.738627+0000","last_became_active":"2024-09-18T08:15:41.464961+0000","last_became_peered":"2024-09-18T08:15:41.464961+0000","last_unstale":"2024-09-18T08:20:44.738627+0000","last_undegraded":"2024-09-18T08:20:44.738627+0000","last_fullsized":"2024-09-18T08:20:44.738627+0000","mapping_epoch":336,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":337,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:17:37.394532+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_clean_scrub_stamp":"2024-09-18T08:17:37.394532+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.12","version":"271'15","reported_seq":716,"reported_epoch":445,"state":"active+clean","last_fresh":"2024-09-18T08:20:44.733164+0000","last_change":"2024-09-18T08:19:05.918020+0000","last_active":"2024-09-18T08:20:44.733164+0000","last_peered":"2024-09-18T08:20:44.733164+0000","last_clean":"2024-09-18T08:20:44.733164+0000","last_became_active":"2024-09-18T08:19:05.904293+0000","last_became_peered":"2024-09-18T08:19:05.904293+0000","last_unstale":"2024-09-18T08:20:44.733164+0000","last_undegraded":"2024-09-18T08:20:44.733164+0000","last_fullsized":"2024-09-18T08:20:44.733164+0000","mapping_epoch":413,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":414,"parent":"0.0","parent_split_bits":0,"last_scrub":"271'15","last_scrub_stamp":"2024-09-18T08:18:31.593238+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_clean_scrub_stamp":"2024-09-18T08:18:31.593238+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":25,"num_read_kb":19,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,0],"acting":[1,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":629,"reported_epoch":445,"state":"active+clean","last_fresh":"2024-09-18T08:20:44.736306+0000","last_change":"2024-09-18T08:18:27.663193+0000","last_active":"2024-09-18T08:20:44.736306+0000","last_peered":"2024-09-18T08:20:44.736306+0000","last_clean":"2024-09-18T08:20:44.736306+0000","last_became_active":"2024-09-18T08:18:27.662885+0000","last_became_peered":"2024-09-18T08:18:27.662885+0000","last_unstale":"2024-09-18T08:20:44.736306+0000","last_undegraded":"2024-09-18T08:20:44.736306+0000","last_fullsized":"2024-09-18T08:20:44.736306+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:15:58.978411+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:46.860092+0000","last_clean_scrub_stamp":"2024-09-18T08:15:58.978411+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:21:41.933571+0000","scrub_duration":18,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":820,"reported_epoch":445,"state":"active+clean","last_fresh":"2024-09-18T08:20:44.734354+0000","last_change":"2024-09-18T08:18:28.695902+0000","last_active":"2024-09-18T08:20:44.734354+0000","last_peered":"2024-09-18T08:20:44.734354+0000","last_clean":"2024-09-18T08:20:44.734354+0000","last_became_active":"2024-09-18T08:14:39.402406+0000","last_became_peered":"2024-09-18T08:14:39.402406+0000","last_unstale":"2024-09-18T08:20:44.734354+0000","last_undegraded":"2024-09-18T08:20:44.734354+0000","last_fullsized":"2024-09-18T08:20:44.734354+0000","mapping_epoch":307,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":308,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:18:28.695842+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:50.886830+0000","last_clean_scrub_stamp":"2024-09-18T08:18:28.695842+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.10","version":"69'1","reported_seq":834,"reported_epoch":445,"state":"active+clean","last_fresh":"2024-09-18T08:20:44.730640+0000","last_change":"2024-09-18T08:20:31.208995+0000","last_active":"2024-09-18T08:20:44.730640+0000","last_peered":"2024-09-18T08:20:44.730640+0000","last_clean":"2024-09-18T08:20:44.730640+0000","last_became_active":"2024-09-18T08:15:41.444085+0000","last_became_peered":"2024-09-18T08:15:41.444085+0000","last_unstale":"2024-09-18T08:20:44.730640+0000","last_undegraded":"2024-09-18T08:20:44.730640+0000","last_fullsized":"2024-09-18T08:20:44.730640+0000","mapping_epoch":336,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":337,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'1","last_scrub_stamp":"2024-09-18T08:20:31.208918+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_clean_scrub_stamp":"2024-09-18T08:20:31.208918+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:21:39.458508+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":26,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.11","version":"271'19","reported_seq":982,"reported_epoch":445,"state":"active+clean","last_fresh":"2024-09-18T08:20:44.737110+0000","last_change":"2024-09-18T08:20:10.875530+0000","last_active":"2024-09-18T08:20:44.737110+0000","last_peered":"2024-09-18T08:20:44.737110+0000","last_clean":"2024-09-18T08:20:44.737110+0000","last_became_active":"2024-09-18T08:18:27.659293+0000","last_became_peered":"2024-09-18T08:18:27.659293+0000","last_unstale":"2024-09-18T08:20:44.737110+0000","last_undegraded":"2024-09-18T08:20:44.737110+0000","last_fullsized":"2024-09-18T08:20:44.737110+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":0,"last_scrub":"271'19","last_scrub_stamp":"2024-09-18T08:20:10.875426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_clean_scrub_stamp":"2024-09-18T08:20:10.875426+0000","objects_scrubbed":5,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:21:10.982420+0000","scrub_duration":22,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":32,"num_read_kb":25,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":30,"num_bytes_recovered":660,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":245,"reported_epoch":445,"state":"active+clean","last_fresh":"2024-09-18T08:20:44.732185+0000","last_change":"2024-09-18T08:18:27.644325+0000","last_active":"2024-09-18T08:20:44.732185+0000","last_peered":"2024-09-18T08:20:44.732185+0000","last_clean":"2024-09-18T08:20:44.732185+0000","last_became_active":"2024-09-18T08:18:27.643957+0000","last_became_peered":"2024-09-18T08:18:27.643957+0000","last_unstale":"2024-09-18T08:20:44.732185+0000","last_undegraded":"2024-09-18T08:20:44.732185+0000","last_fullsized":"2024-09-18T08:20:44.732185+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:18:18.611302+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:46.860092+0000","last_clean_scrub_stamp":"2024-09-18T08:18:18.611302+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":107,"reported_epoch":445,"state":"active+clean","last_fresh":"2024-09-18T08:20:44.725544+0000","last_change":"2024-09-18T08:20:35.531055+0000","last_active":"2024-09-18T08:20:44.725544+0000","last_peered":"2024-09-18T08:20:44.725544+0000","last_clean":"2024-09-18T08:20:44.725544+0000","last_became_active":"2024-09-18T08:18:27.651810+0000","last_became_peered":"2024-09-18T08:18:27.651810+0000","last_unstale":"2024-09-18T08:20:44.725544+0000","last_undegraded":"2024-09-18T08:20:44.725544+0000","last_fullsized":"2024-09-18T08:20:44.725544+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:20:35.530991+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:50.886830+0000","last_clean_scrub_stamp":"2024-09-18T08:20:35.530991+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:21:39.992702+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,0],"acting":[7,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":819,"reported_epoch":445,"state":"active+clean","last_fresh":"2024-09-18T08:20:44.734499+0000","last_change":"2024-09-18T08:18:21.660255+0000","last_active":"2024-09-18T08:20:44.734499+0000","last_peered":"2024-09-18T08:20:44.734499+0000","last_clean":"2024-09-18T08:20:44.734499+0000","last_became_active":"2024-09-18T08:06:24.128582+0000","last_became_peered":"2024-09-18T08:06:24.128582+0000","last_unstale":"2024-09-18T08:20:44.734499+0000","last_undegraded":"2024-09-18T08:20:44.734499+0000","last_fullsized":"2024-09-18T08:20:44.734499+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:18:21.660213+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_clean_scrub_stamp":"2024-09-18T08:18:21.660213+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.10","version":"271'8","reported_seq":877,"reported_epoch":445,"state":"active+clean","last_fresh":"2024-09-18T08:20:44.740083+0000","last_change":"2024-09-18T08:20:38.022125+0000","last_active":"2024-09-18T08:20:44.740083+0000","last_peered":"2024-09-18T08:20:44.740083+0000","last_clean":"2024-09-18T08:20:44.740083+0000","last_became_active":"2024-09-18T08:14:39.404257+0000","last_became_peered":"2024-09-18T08:14:39.404257+0000","last_unstale":"2024-09-18T08:20:44.740083+0000","last_undegraded":"2024-09-18T08:20:44.740083+0000","last_fullsized":"2024-09-18T08:20:44.740083+0000","mapping_epoch":307,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":308,"parent":"0.0","parent_split_bits":0,"last_scrub":"271'8","last_scrub_stamp":"2024-09-18T08:20:38.022032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_clean_scrub_stamp":"2024-09-18T08:20:38.022032+0000","objects_scrubbed":2,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:21:55.126313+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":12,"num_read_kb":10,"num_write":8,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":846,"reported_epoch":445,"state":"active+clean","last_fresh":"2024-09-18T08:20:44.729644+0000","last_change":"2024-09-18T08:18:25.799322+0000","last_active":"2024-09-18T08:20:44.729644+0000","last_peered":"2024-09-18T08:20:44.729644+0000","last_clean":"2024-09-18T08:20:44.729644+0000","last_became_active":"2024-09-18T08:15:41.466964+0000","last_became_peered":"2024-09-18T08:15:41.466964+0000","last_unstale":"2024-09-18T08:20:44.729644+0000","last_undegraded":"2024-09-18T08:20:44.729644+0000","last_fullsized":"2024-09-18T08:20:44.729644+0000","mapping_epoch":336,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":337,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:18:25.799264+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:46.860092+0000","last_clean_scrub_stamp":"2024-09-18T08:18:25.799264+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":470,"reported_epoch":445,"state":"active+clean","last_fresh":"2024-09-18T08:20:44.729645+0000","last_change":"2024-09-18T08:18:27.645673+0000","last_active":"2024-09-18T08:20:44.729645+0000","last_peered":"2024-09-18T08:20:44.729645+0000","last_clean":"2024-09-18T08:20:44.729645+0000","last_became_active":"2024-09-18T08:18:27.645122+0000","last_became_peered":"2024-09-18T08:18:27.645122+0000","last_unstale":"2024-09-18T08:20:44.729645+0000","last_undegraded":"2024-09-18T08:20:44.729645+0000","last_fullsized":"2024-09-18T08:20:44.729645+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:18:14.912464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:50.886830+0000","last_clean_scrub_stamp":"2024-09-18T08:18:14.912464+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,7],"acting":[0,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":612,"reported_epoch":445,"state":"active+clean","last_fresh":"2024-09-18T08:20:44.734672+0000","last_change":"2024-09-18T08:19:05.890924+0000","last_active":"2024-09-18T08:20:44.734672+0000","last_peered":"2024-09-18T08:20:44.734672+0000","last_clean":"2024-09-18T08:20:44.734672+0000","last_became_active":"2024-09-18T08:19:05.890580+0000","last_became_peered":"2024-09-18T08:19:05.890580+0000","last_unstale":"2024-09-18T08:20:44.734672+0000","last_undegraded":"2024-09-18T08:20:44.734672+0000","last_fullsized":"2024-09-18T08:20:44.734672+0000","mapping_epoch":413,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":414,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:19:03.125974+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_clean_scrub_stamp":"2024-09-18T08:19:03.125974+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":683,"reported_epoch":445,"state":"active+clean","last_fresh":"2024-09-18T08:20:44.733588+0000","last_change":"2024-09-18T08:20:30.922577+0000","last_active":"2024-09-18T08:20:44.733588+0000","last_peered":"2024-09-18T08:20:44.733588+0000","last_clean":"2024-09-18T08:20:44.733588+0000","last_became_active":"2024-09-18T08:16:13.466812+0000","last_became_peered":"2024-09-18T08:16:13.466812+0000","last_unstale":"2024-09-18T08:20:44.733588+0000","last_undegraded":"2024-09-18T08:20:44.733588+0000","last_fullsized":"2024-09-18T08:20:44.733588+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:20:30.922502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_clean_scrub_stamp":"2024-09-18T08:20:30.922502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:21:53.559925+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,2],"acting":[1,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":828,"reported_epoch":445,"state":"active+clean","last_fresh":"2024-09-18T08:20:44.740442+0000","last_change":"2024-09-18T08:20:32.064007+0000","last_active":"2024-09-18T08:20:44.740442+0000","last_peered":"2024-09-18T08:20:44.740442+0000","last_clean":"2024-09-18T08:20:44.740442+0000","last_became_active":"2024-09-18T08:14:39.405347+0000","last_became_peered":"2024-09-18T08:14:39.405347+0000","last_unstale":"2024-09-18T08:20:44.740442+0000","last_undegraded":"2024-09-18T08:20:44.740442+0000","last_fullsized":"2024-09-18T08:20:44.740442+0000","mapping_epoch":307,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":308,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:20:32.063827+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:46.860092+0000","last_clean_scrub_stamp":"2024-09-18T08:20:32.063827+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:21:33.490031+0000","scrub_duration":23,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"271'19","reported_seq":250,"reported_epoch":445,"state":"active+clean","last_fresh":"2024-09-18T08:20:44.732038+0000","last_change":"2024-09-18T08:18:20.608272+0000","last_active":"2024-09-18T08:20:44.732038+0000","last_peered":"2024-09-18T08:20:44.732038+0000","last_clean":"2024-09-18T08:20:44.732038+0000","last_became_active":"2024-09-18T08:15:41.445198+0000","last_became_peered":"2024-09-18T08:15:41.445198+0000","last_unstale":"2024-09-18T08:20:44.732038+0000","last_undegraded":"2024-09-18T08:20:44.732038+0000","last_fullsized":"2024-09-18T08:20:44.732038+0000","mapping_epoch":336,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":337,"parent":"0.0","parent_split_bits":0,"last_scrub":"271'19","last_scrub_stamp":"2024-09-18T08:18:20.608110+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_clean_scrub_stamp":"2024-09-18T08:18:20.608110+0000","objects_scrubbed":5,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":31,"num_read_kb":24,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,1],"acting":[5,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":857,"reported_epoch":445,"state":"active+clean","last_fresh":"2024-09-18T08:20:44.728693+0000","last_change":"2024-09-18T08:20:15.269092+0000","last_active":"2024-09-18T08:20:44.728693+0000","last_peered":"2024-09-18T08:20:44.728693+0000","last_clean":"2024-09-18T08:20:44.728693+0000","last_became_active":"2024-09-18T08:19:05.903954+0000","last_became_peered":"2024-09-18T08:19:05.903954+0000","last_unstale":"2024-09-18T08:20:44.728693+0000","last_undegraded":"2024-09-18T08:20:44.728693+0000","last_fullsized":"2024-09-18T08:20:44.728693+0000","mapping_epoch":413,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":414,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:20:15.269044+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:50.886830+0000","last_clean_scrub_stamp":"2024-09-18T08:20:15.269044+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:21:24.188336+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,1],"acting":[0,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":11,"num_pg":16,"stat_sum":{"num_bytes":2904686616,"num_objects":44323,"num_object_clones":0,"num_object_copies":132969,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":44323,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":153041,"num_write_kb":6315585,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19130,"num_bytes_recovered":1253703680,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":9220730880,"data_stored":9220718664,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":62441,"ondisk_log_size":62441,"up":48,"acting":48,"num_store_stats":7},{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":416,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":1235,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":32,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":125,"ondisk_log_size":125,"up":96,"acting":96,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":178,"num_object_clones":0,"num_object_copies":534,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":178,"num_whiteouts":0,"num_read":1106,"num_read_kb":863,"num_write":703,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":559,"num_bytes_recovered":10830,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":475136,"data_stored":12690,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3639,"internal_metadata":0},"log_size":690,"ondisk_log_size":690,"up":96,"acting":96,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":1613,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":14,"num_read_kb":14,"num_write":10,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":17,"num_bytes_recovered":6130,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":94208,"data_stored":6016,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":6,"ondisk_log_size":6,"up":96,"acting":96,"num_store_stats":8},{"poolid":2,"num_pg":3,"stat_sum":{"num_bytes":408,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":537,"num_read_kb":532,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":797,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":8,"ondisk_log_size":8,"up":9,"acting":9,"num_store_stats":7},{"poolid":1,"num_pg":11,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":1573408,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":869,"ondisk_log_size":869,"up":33,"acting":33,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":56,"seq":240518168796,"num_pgs":76,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2071788,"kb_used_data":1463068,"kb_used_omap":1,"kb_used_meta":608702,"kb_avail":91673364,"statfs":{"total":95995035648,"available":93873524736,"internally_reserved":0,"allocated":1498181632,"data_stored":1497788176,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":623311306},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,1,1],"upper_bound":32},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":47,"seq":201863463138,"num_pgs":70,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2106936,"kb_used_data":1453800,"kb_used_omap":1,"kb_used_meta":653118,"kb_avail":91638216,"statfs":{"total":95995035648,"available":93837533184,"internally_reserved":0,"allocated":1488691200,"data_stored":1488284200,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":668793286},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,2,0,3],"upper_bound":16},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":39,"seq":167503724776,"num_pgs":72,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2208744,"kb_used_data":1261272,"kb_used_omap":1,"kb_used_meta":947454,"kb_avail":91536408,"statfs":{"total":95995035648,"available":93733281792,"internally_reserved":0,"allocated":1291542528,"data_stored":1291152530,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":970193350},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1],"upper_bound":8},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":0,"seq":137438953709,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":26,"seq":111669149940,"num_pgs":76,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2080888,"kb_used_data":1237544,"kb_used_omap":1,"kb_used_meta":843326,"kb_avail":91664264,"statfs":{"total":95995035648,"available":93864206336,"internally_reserved":0,"allocated":1267245056,"data_stored":1266838623,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":863566280},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,4,0,1],"upper_bound":16},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411578,"num_pgs":17,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":672188,"kb_used_data":1000,"kb_used_omap":1,"kb_used_meta":671166,"kb_avail":93072964,"statfs":{"total":95995035648,"available":95306715136,"internally_reserved":0,"allocated":1024000,"data_stored":681782,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":687274437},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":307,"seq":1318554959947,"num_pgs":84,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2179592,"kb_used_data":1834872,"kb_used_omap":7,"kb_used_meta":344696,"kb_avail":91565560,"statfs":{"total":95995035648,"available":93763133440,"internally_reserved":0,"allocated":1878908928,"data_stored":1878487139,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":8079,"internal_metadata":352968817},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,2,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738629,"num_pgs":79,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2251268,"kb_used_data":1457908,"kb_used_omap":1,"kb_used_meta":793342,"kb_avail":91493884,"statfs":{"total":95995035648,"available":93689737216,"internally_reserved":0,"allocated":1492897792,"data_stored":1492480356,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":812382661},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,3,1,2],"upper_bound":32},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":266240,"data_stored":262688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1567,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1613,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":46,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1177,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":436,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":46,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1131,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":2024,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":1678,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3639,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":440,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":81920,"data_stored":2118,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":1584,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":77824,"data_stored":2052,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1474,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":403,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":416,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":403,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":1484324864,"data_stored":1484324864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":1863979008,"data_stored":1863974936,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":418578432,"data_stored":418578432,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":1233190912,"data_stored":1233190912,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":1281036288,"data_stored":1281032216,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":1476399104,"data_stored":1476395032,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":1463222272,"data_stored":1463222272,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T08:20:56.361 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid 779349ca-7593-11ef-bceb-c7b262605968 -- ceph osd pool set .mgr pgp_num 11 2024-09-18T08:20:56.368 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:20:56.368 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:20:56.368 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6' 2024-09-18T08:20:56.469 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7 2024-09-18T08:20:56.499 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:20:56.500 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:20:56.501 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7' 2024-09-18T08:20:56.601 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4 2024-09-18T08:20:56.629 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:20:56.630 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:20:56.630 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4' 2024-09-18T08:20:56.732 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1 2024-09-18T08:20:56.779 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:20:56.780 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:20:56.780 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1' 2024-09-18T08:20:56.827 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/779349ca-7593-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T08:20:56.881 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2 2024-09-18T08:20:56.905 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:20:56 smithi050 ceph-mon[32371]: pgmap v904: 158 pgs: 158 active+clean; 2.3 GiB data, 12 GiB used, 614 GiB / 626 GiB avail; 948 B/s rd, 0 B/s wr, 1.52k op/s 2024-09-18T08:20:56.921 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:20:56.921 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:20:56.922 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2' 2024-09-18T08:20:56.967 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:20:56 smithi007 ceph-mon[29459]: pgmap v904: 158 pgs: 158 active+clean; 2.3 GiB data, 12 GiB used, 614 GiB / 626 GiB avail; 948 B/s rd, 0 B/s wr, 1.52k op/s 2024-09-18T08:20:56.967 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Sep 18 08:20:56 smithi007 ceph-779349ca-7593-11ef-bceb-c7b262605968-mgr-y[29688]: ::ffff:172.21.15.50 - - [18/Sep/2024:08:20:56] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-09-18T08:20:56.968 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:20:56 smithi007 ceph-mon[39249]: pgmap v904: 158 pgs: 158 active+clean; 2.3 GiB data, 12 GiB used, 614 GiB / 626 GiB avail; 948 B/s rd, 0 B/s wr, 1.52k op/s 2024-09-18T08:20:57.023 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7 2024-09-18T08:20:57.057 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:20:57.058 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:20:57.058 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7' 2024-09-18T08:20:57.148 DEBUG:teuthology.orchestra.run.smithi050:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid 779349ca-7593-11ef-bceb-c7b262605968 -- ceph daemon osd.4 dump_blocked_ops 2024-09-18T08:20:57.160 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1 2024-09-18T08:20:57.200 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:20:57.201 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:20:57.201 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1' 2024-09-18T08:20:57.302 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6 2024-09-18T08:20:57.336 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid 779349ca-7593-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-18T08:20:57.340 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:20:57.341 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:20:57.341 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6' 2024-09-18T08:20:57.442 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4 2024-09-18T08:20:57.479 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:20:57.479 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:20:57.480 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4' 2024-09-18T08:20:57.547 INFO:teuthology.orchestra.run.smithi050.stderr:Inferring config /var/lib/ceph/779349ca-7593-11ef-bceb-c7b262605968/mon.b/config 2024-09-18T08:20:57.580 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6 2024-09-18T08:20:57.607 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:20:57.608 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:20:57.608 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6' 2024-09-18T08:20:57.615 INFO:teuthology.orchestra.run.smithi007.stderr:nodeep-scrub is set 2024-09-18T08:20:57.708 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0 2024-09-18T08:20:57.751 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:20:57.752 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:20:57.752 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0' 2024-09-18T08:20:57.853 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4 2024-09-18T08:20:57.881 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:20:57.882 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:20:57.882 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4' 2024-09-18T08:20:57.905 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:20:57 smithi050 ceph-mon[32371]: from='client.? 172.21.15.7:0/11581852' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-18T08:20:57.905 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:20:57 smithi050 ceph-mon[32371]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-18T08:20:57.980 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:20:57 smithi007 ceph-mon[29459]: from='client.? 172.21.15.7:0/11581852' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-18T08:20:57.980 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:20:57 smithi007 ceph-mon[29459]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-18T08:20:57.980 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:20:57 smithi007 ceph-mon[39249]: from='client.? 172.21.15.7:0/11581852' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-18T08:20:57.980 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:20:57 smithi007 ceph-mon[39249]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-18T08:20:57.984 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4 2024-09-18T08:20:58.022 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:20:58.023 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:20:58.023 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4' 2024-09-18T08:20:58.124 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4 2024-09-18T08:20:58.160 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:20:58.161 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:20:58.161 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4' 2024-09-18T08:20:58.262 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5 2024-09-18T08:20:58.268 INFO:teuthology.orchestra.run.smithi050.stdout:ERROR: (22) Invalid argument 2024-09-18T08:20:58.269 INFO:teuthology.orchestra.run.smithi050.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-18T08:20:58.295 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:20:58.295 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:20:58.296 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5' 2024-09-18T08:20:58.397 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3 2024-09-18T08:20:58.421 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:20:58.421 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:20:58.422 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3' 2024-09-18T08:20:58.522 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4 2024-09-18T08:20:58.554 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:20:58.554 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:20:58.555 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4' 2024-09-18T08:20:58.611 INFO:teuthology.orchestra.run.smithi007.stderr:set pool 1 pgp_num to 11 2024-09-18T08:20:58.655 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4 2024-09-18T08:20:58.686 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:20:58.686 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:20:58.687 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4' 2024-09-18T08:20:58.730 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:20:58 smithi007 ceph-mon[29459]: pgmap v905: 158 pgs: 158 active+clean; 2.3 GiB data, 12 GiB used, 614 GiB / 626 GiB avail; 853 B/s rd, 0 B/s wr, 1.14k op/s 2024-09-18T08:20:58.730 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:20:58 smithi007 ceph-mon[29459]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-18T08:20:58.730 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:20:58 smithi007 ceph-mon[29459]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-18T08:20:58.730 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:20:58 smithi007 ceph-mon[29459]: osdmap e446: 8 total, 8 up, 7 in 2024-09-18T08:20:58.730 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:20:58 smithi007 ceph-mon[29459]: from='client.? 172.21.15.7:0/2440367516' entity='client.admin' cmd={"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num", "val": "11"} : dispatch 2024-09-18T08:20:58.730 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:20:58 smithi007 ceph-mon[29459]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num", "val": "11"} : dispatch 2024-09-18T08:20:58.731 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:20:58 smithi007 ceph-mon[39249]: pgmap v905: 158 pgs: 158 active+clean; 2.3 GiB data, 12 GiB used, 614 GiB / 626 GiB avail; 853 B/s rd, 0 B/s wr, 1.14k op/s 2024-09-18T08:20:58.731 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:20:58 smithi007 ceph-mon[39249]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-18T08:20:58.731 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:20:58 smithi007 ceph-mon[39249]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-18T08:20:58.731 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:20:58 smithi007 ceph-mon[39249]: osdmap e446: 8 total, 8 up, 7 in 2024-09-18T08:20:58.731 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:20:58 smithi007 ceph-mon[39249]: from='client.? 172.21.15.7:0/2440367516' entity='client.admin' cmd={"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num", "val": "11"} : dispatch 2024-09-18T08:20:58.731 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:20:58 smithi007 ceph-mon[39249]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num", "val": "11"} : dispatch 2024-09-18T08:20:58.787 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4 2024-09-18T08:20:58.822 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:20:58.823 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:20:58.823 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4' 2024-09-18T08:20:58.905 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:20:58 smithi050 ceph-mon[32371]: pgmap v905: 158 pgs: 158 active+clean; 2.3 GiB data, 12 GiB used, 614 GiB / 626 GiB avail; 853 B/s rd, 0 B/s wr, 1.14k op/s 2024-09-18T08:20:58.905 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:20:58 smithi050 ceph-mon[32371]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-18T08:20:58.905 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:20:58 smithi050 ceph-mon[32371]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-18T08:20:58.905 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:20:58 smithi050 ceph-mon[32371]: osdmap e446: 8 total, 8 up, 7 in 2024-09-18T08:20:58.905 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:20:58 smithi050 ceph-mon[32371]: from='client.? 172.21.15.7:0/2440367516' entity='client.admin' cmd={"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num", "val": "11"} : dispatch 2024-09-18T08:20:58.905 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:20:58 smithi050 ceph-mon[32371]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num", "val": "11"} : dispatch 2024-09-18T08:20:58.925 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0 2024-09-18T08:20:58.966 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:20:58.967 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:20:58.967 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0' 2024-09-18T08:20:59.068 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0 2024-09-18T08:20:59.096 DEBUG:teuthology.orchestra.run.smithi050:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid 779349ca-7593-11ef-bceb-c7b262605968 -- ceph daemon osd.4 dump_historic_ops 2024-09-18T08:20:59.104 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:20:59.104 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:20:59.104 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0' 2024-09-18T08:20:59.205 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5 2024-09-18T08:20:59.246 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:20:59.246 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:20:59.246 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5' 2024-09-18T08:20:59.347 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0 2024-09-18T08:20:59.393 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:20:59.394 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:20:59.394 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0' 2024-09-18T08:20:59.458 INFO:teuthology.orchestra.run.smithi050.stderr:Inferring config /var/lib/ceph/779349ca-7593-11ef-bceb-c7b262605968/mon.b/config 2024-09-18T08:20:59.495 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6 2024-09-18T08:20:59.527 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:20:59.528 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:20:59.529 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6' 2024-09-18T08:20:59.630 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5 2024-09-18T08:20:59.664 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:20:59.664 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:20:59.665 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5' 2024-09-18T08:20:59.766 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4 2024-09-18T08:20:59.788 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:20:59.789 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:20:59.789 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4' 2024-09-18T08:20:59.890 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7 2024-09-18T08:20:59.905 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:20:59 smithi050 ceph-mon[32371]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num", "val": "11"}]': finished 2024-09-18T08:20:59.906 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:20:59 smithi050 ceph-mon[32371]: osdmap e447: 8 total, 8 up, 7 in 2024-09-18T08:20:59.906 INFO:journalctl@ceph.iscsi.iscsi.a.smithi050.stdout:Sep 18 08:20:59 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-iscsi-iscsi-a[57782]: debug there is no tcmu-runner data available 2024-09-18T08:20:59.926 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:20:59.927 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:20:59.927 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7' 2024-09-18T08:20:59.979 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:20:59 smithi007 ceph-mon[29459]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num", "val": "11"}]': finished 2024-09-18T08:20:59.980 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:20:59 smithi007 ceph-mon[29459]: osdmap e447: 8 total, 8 up, 7 in 2024-09-18T08:20:59.980 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:20:59 smithi007 ceph-mon[39249]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num", "val": "11"}]': finished 2024-09-18T08:20:59.980 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:20:59 smithi007 ceph-mon[39249]: osdmap e447: 8 total, 8 up, 7 in 2024-09-18T08:21:00.028 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1 2024-09-18T08:21:00.054 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:21:00.054 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:21:00.055 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1' 2024-09-18T08:21:00.155 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7 2024-09-18T08:21:00.200 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:21:00.201 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:21:00.201 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7' 2024-09-18T08:21:00.302 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1 2024-09-18T08:21:00.309 INFO:teuthology.orchestra.run.smithi050.stdout:ERROR: (22) Invalid argument 2024-09-18T08:21:00.309 INFO:teuthology.orchestra.run.smithi050.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-18T08:21:00.341 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:21:00.342 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:21:00.342 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1' 2024-09-18T08:21:00.443 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5 2024-09-18T08:21:00.465 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:21:00.465 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:21:00.466 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5' 2024-09-18T08:21:00.567 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2 2024-09-18T08:21:00.588 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:21:00.589 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:21:00.590 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2' 2024-09-18T08:21:00.690 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7 2024-09-18T08:21:00.723 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:21:00.723 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:21:00.724 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7' 2024-09-18T08:21:00.825 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4 2024-09-18T08:21:00.860 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:21:00.860 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:21:00.861 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4' 2024-09-18T08:21:00.905 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:21:00 smithi050 ceph-mon[32371]: pgmap v908: 158 pgs: 158 active+clean; 2.0 GiB data, 11 GiB used, 615 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.39k op/s 2024-09-18T08:21:00.938 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:21:00 smithi007 ceph-mon[29459]: pgmap v908: 158 pgs: 158 active+clean; 2.0 GiB data, 11 GiB used, 615 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.39k op/s 2024-09-18T08:21:00.938 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:21:00 smithi007 ceph-mon[39249]: pgmap v908: 158 pgs: 158 active+clean; 2.0 GiB data, 11 GiB used, 615 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.39k op/s 2024-09-18T08:21:00.962 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3 2024-09-18T08:21:01.004 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:21:01.004 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:21:01.005 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3' 2024-09-18T08:21:01.014 DEBUG:teuthology.orchestra.run.smithi050:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid 779349ca-7593-11ef-bceb-c7b262605968 -- ceph daemon osd.5 dump_ops_in_flight 2024-09-18T08:21:01.105 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0 2024-09-18T08:21:01.140 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:21:01.141 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:21:01.141 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0' 2024-09-18T08:21:01.242 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1 2024-09-18T08:21:01.287 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:21:01.288 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:21:01.288 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1' 2024-09-18T08:21:01.343 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid 779349ca-7593-11ef-bceb-c7b262605968 -- ceph osd unset noscrub 2024-09-18T08:21:01.376 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/779349ca-7593-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T08:21:01.381 INFO:teuthology.orchestra.run.smithi050.stderr:Inferring config /var/lib/ceph/779349ca-7593-11ef-bceb-c7b262605968/mon.b/config 2024-09-18T08:21:01.389 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4 2024-09-18T08:21:01.425 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:21:01.426 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:21:01.426 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4' 2024-09-18T08:21:01.528 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1 2024-09-18T08:21:01.596 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:21:01.596 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:21:01.597 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1' 2024-09-18T08:21:01.698 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3 2024-09-18T08:21:01.762 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:21:01.763 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:21:01.763 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3' 2024-09-18T08:21:01.775 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:21:01 smithi007 ceph-mon[39249]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-18T08:21:01.776 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:21:01 smithi007 ceph-mon[29459]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-18T08:21:01.829 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:21:01 smithi050 ceph-mon[32371]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-18T08:21:01.864 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7 2024-09-18T08:21:01.903 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:21:01.903 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:21:01.903 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7' 2024-09-18T08:21:01.997 INFO:teuthology.orchestra.run.smithi050.stdout:ERROR: (22) Invalid argument 2024-09-18T08:21:01.997 INFO:teuthology.orchestra.run.smithi050.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-18T08:21:02.005 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3 2024-09-18T08:21:02.044 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:21:02.045 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:21:02.045 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3' 2024-09-18T08:21:02.115 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/779349ca-7593-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T08:21:02.146 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4 2024-09-18T08:21:02.173 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:21:02.174 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:21:02.174 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4' 2024-09-18T08:21:02.180 INFO:teuthology.orchestra.run.smithi007.stdout:osd.0: {} 2024-09-18T08:21:02.180 INFO:teuthology.orchestra.run.smithi007.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-18T08:21:02.213 INFO:teuthology.orchestra.run.smithi007.stdout:osd.1: {} 2024-09-18T08:21:02.214 INFO:teuthology.orchestra.run.smithi007.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-18T08:21:02.248 INFO:teuthology.orchestra.run.smithi007.stdout:osd.2: {} 2024-09-18T08:21:02.248 INFO:teuthology.orchestra.run.smithi007.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-18T08:21:02.275 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1 2024-09-18T08:21:02.291 INFO:teuthology.orchestra.run.smithi007.stdout:osd.3: {} 2024-09-18T08:21:02.291 INFO:teuthology.orchestra.run.smithi007.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-18T08:21:02.317 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:21:02.317 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:21:02.318 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1' 2024-09-18T08:21:02.330 INFO:teuthology.orchestra.run.smithi007.stdout:osd.4: {} 2024-09-18T08:21:02.330 INFO:teuthology.orchestra.run.smithi007.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-18T08:21:02.365 INFO:teuthology.orchestra.run.smithi007.stdout:osd.5: {} 2024-09-18T08:21:02.365 INFO:teuthology.orchestra.run.smithi007.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-18T08:21:02.401 INFO:teuthology.orchestra.run.smithi007.stdout:osd.6: {} 2024-09-18T08:21:02.402 INFO:teuthology.orchestra.run.smithi007.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-18T08:21:02.419 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4 2024-09-18T08:21:02.436 INFO:teuthology.orchestra.run.smithi007.stdout:osd.7: {} 2024-09-18T08:21:02.436 INFO:teuthology.orchestra.run.smithi007.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-18T08:21:02.453 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:21:02.454 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:21:02.454 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4' 2024-09-18T08:21:02.555 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4 2024-09-18T08:21:02.590 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:21:02.591 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:21:02.591 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4' 2024-09-18T08:21:02.692 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1 2024-09-18T08:21:02.730 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:21:02.731 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:21:02.731 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1' 2024-09-18T08:21:02.733 DEBUG:teuthology.orchestra.run.smithi050:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid 779349ca-7593-11ef-bceb-c7b262605968 -- ceph daemon osd.5 dump_blocked_ops 2024-09-18T08:21:02.832 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1 2024-09-18T08:21:02.877 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:21:02.877 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:21:02.878 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1' 2024-09-18T08:21:02.905 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:21:02 smithi050 ceph-mon[32371]: pgmap v909: 158 pgs: 158 active+clean; 1.9 GiB data, 11 GiB used, 615 GiB / 626 GiB avail; 639 B/s rd, 0 B/s wr, 1.37k op/s 2024-09-18T08:21:02.905 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:21:02 smithi050 ceph-mon[32371]: from='client.? 172.21.15.7:0/1011270380' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-18T08:21:02.979 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6 2024-09-18T08:21:02.980 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:21:02 smithi007 ceph-mon[29459]: pgmap v909: 158 pgs: 158 active+clean; 1.9 GiB data, 11 GiB used, 615 GiB / 626 GiB avail; 639 B/s rd, 0 B/s wr, 1.37k op/s 2024-09-18T08:21:02.980 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:21:02 smithi007 ceph-mon[29459]: from='client.? 172.21.15.7:0/1011270380' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-18T08:21:02.980 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:21:02 smithi007 ceph-mon[39249]: pgmap v909: 158 pgs: 158 active+clean; 1.9 GiB data, 11 GiB used, 615 GiB / 626 GiB avail; 639 B/s rd, 0 B/s wr, 1.37k op/s 2024-09-18T08:21:02.980 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:21:02 smithi007 ceph-mon[39249]: from='client.? 172.21.15.7:0/1011270380' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-18T08:21:03.018 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:21:03.018 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:21:03.019 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6' 2024-09-18T08:21:03.120 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5 2024-09-18T08:21:03.130 INFO:teuthology.orchestra.run.smithi050.stderr:Inferring config /var/lib/ceph/779349ca-7593-11ef-bceb-c7b262605968/mon.b/config 2024-09-18T08:21:03.163 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:21:03.164 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:21:03.164 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5' 2024-09-18T08:21:03.265 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2 2024-09-18T08:21:03.302 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:21:03.303 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:21:03.303 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2' 2024-09-18T08:21:03.404 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0 2024-09-18T08:21:03.440 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:21:03.441 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:21:03.441 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0' 2024-09-18T08:21:03.542 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2 2024-09-18T08:21:03.581 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:21:03.582 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:21:03.582 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2' 2024-09-18T08:21:03.683 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2 2024-09-18T08:21:03.719 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:21:03.720 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:21:03.721 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2' 2024-09-18T08:21:03.736 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2024-09-18T08:21:03.737 INFO:teuthology.orchestra.run.smithi050.stdout: "ops": [], 2024-09-18T08:21:03.737 INFO:teuthology.orchestra.run.smithi050.stdout: "complaint_time": 30, 2024-09-18T08:21:03.737 INFO:teuthology.orchestra.run.smithi050.stdout: "num_blocked_ops": 0 2024-09-18T08:21:03.737 INFO:teuthology.orchestra.run.smithi050.stdout:} 2024-09-18T08:21:03.822 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5 2024-09-18T08:21:03.855 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:21:03.856 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:21:03.856 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5' 2024-09-18T08:21:03.957 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5 2024-09-18T08:21:03.992 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:21:03.993 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:21:03.993 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5' 2024-09-18T08:21:04.094 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2 2024-09-18T08:21:04.137 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:21:04.137 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:21:04.137 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2' 2024-09-18T08:21:04.238 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7 2024-09-18T08:21:04.270 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:21:04.270 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:21:04.271 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7' 2024-09-18T08:21:04.372 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6 2024-09-18T08:21:04.413 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:21:04.413 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:21:04.414 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6' 2024-09-18T08:21:04.443 DEBUG:teuthology.orchestra.run.smithi050:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid 779349ca-7593-11ef-bceb-c7b262605968 -- ceph daemon osd.5 dump_historic_ops 2024-09-18T08:21:04.515 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0 2024-09-18T08:21:04.555 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:21:04.556 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:21:04.556 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0' 2024-09-18T08:21:04.637 INFO:teuthology.orchestra.run.smithi007.stderr:noscrub is unset 2024-09-18T08:21:04.658 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1 2024-09-18T08:21:04.699 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:21:04.699 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:21:04.699 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1' 2024-09-18T08:21:04.800 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0 2024-09-18T08:21:04.838 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:21:04.839 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:21:04.839 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0' 2024-09-18T08:21:04.867 INFO:teuthology.orchestra.run.smithi050.stderr:Inferring config /var/lib/ceph/779349ca-7593-11ef-bceb-c7b262605968/mon.b/config 2024-09-18T08:21:04.894 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:21:04 smithi007 ceph-mon[29459]: pgmap v910: 158 pgs: 158 active+clean; 1.7 GiB data, 9.9 GiB used, 616 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.12k op/s 2024-09-18T08:21:04.894 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:21:04 smithi007 ceph-mon[29459]: from='client.? 172.21.15.7:0/2079916648' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-18T08:21:04.894 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:21:04 smithi007 ceph-mon[29459]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-18T08:21:04.894 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:21:04 smithi007 ceph-mon[29459]: from='mgr.24470 172.21.15.7:0/2594390261' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-18T08:21:04.895 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:21:04 smithi007 ceph-mon[39249]: pgmap v910: 158 pgs: 158 active+clean; 1.7 GiB data, 9.9 GiB used, 616 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.12k op/s 2024-09-18T08:21:04.895 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:21:04 smithi007 ceph-mon[39249]: from='client.? 172.21.15.7:0/2079916648' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-18T08:21:04.895 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:21:04 smithi007 ceph-mon[39249]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-18T08:21:04.895 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:21:04 smithi007 ceph-mon[39249]: from='mgr.24470 172.21.15.7:0/2594390261' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-18T08:21:04.905 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:21:04 smithi050 ceph-mon[32371]: pgmap v910: 158 pgs: 158 active+clean; 1.7 GiB data, 9.9 GiB used, 616 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.12k op/s 2024-09-18T08:21:04.905 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:21:04 smithi050 ceph-mon[32371]: from='client.? 172.21.15.7:0/2079916648' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-18T08:21:04.906 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:21:04 smithi050 ceph-mon[32371]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-18T08:21:04.906 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:21:04 smithi050 ceph-mon[32371]: from='mgr.24470 172.21.15.7:0/2594390261' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-18T08:21:04.941 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7 2024-09-18T08:21:04.978 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:21:04.978 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:21:04.979 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7' 2024-09-18T08:21:05.080 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1 2024-09-18T08:21:05.122 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:21:05.122 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:21:05.123 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1' 2024-09-18T08:21:05.224 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2 2024-09-18T08:21:05.261 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:21:05.262 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:21:05.262 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2' 2024-09-18T08:21:05.363 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3 2024-09-18T08:21:05.403 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:21:05.403 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:21:05.404 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3' 2024-09-18T08:21:05.440 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2024-09-18T08:21:05.441 INFO:teuthology.orchestra.run.smithi050.stdout: "size": 20, 2024-09-18T08:21:05.441 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 600, 2024-09-18T08:21:05.441 INFO:teuthology.orchestra.run.smithi050.stdout: "ops": [ 2024-09-18T08:21:05.441 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:21:05.441 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.15582.0:252487 9.7 9:eae9c550:::benchmark_data_smithi007_124019_object110602:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e291)", 2024-09-18T08:21:05.441 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-09-18T08:14:04.561104+0000", 2024-09-18T08:21:05.441 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 420.87693725399998, 2024-09-18T08:21:05.442 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 2.165593307, 2024-09-18T08:21:05.442 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-09-18T08:21:05.442 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-09-18T08:21:05.442 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "reached pg", 2024-09-18T08:21:05.442 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2024-09-18T08:21:05.442 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.15582", 2024-09-18T08:21:05.442 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "172.21.15.7:0/3598605839", 2024-09-18T08:21:05.442 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 252487 2024-09-18T08:21:05.442 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:21:05.442 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-09-18T08:21:05.442 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:21:05.442 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-09-18T08:21:05.442 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:14:04.561104+0000", 2024-09-18T08:21:05.442 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-18T08:21:05.443 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:21:05.443 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:21:05.443 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-09-18T08:21:05.443 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:14:04.561104+0000", 2024-09-18T08:21:05.443 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-18T08:21:05.443 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:21:05.443 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:21:05.443 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-09-18T08:21:05.443 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:14:04.561105+0000", 2024-09-18T08:21:05.443 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.263e-06 2024-09-18T08:21:05.443 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:21:05.443 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:21:05.443 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-09-18T08:21:05.444 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:14:04.561110+0000", 2024-09-18T08:21:05.444 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 5.4369999999999998e-06 2024-09-18T08:21:05.444 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:21:05.444 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:21:05.444 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-09-18T08:21:05.444 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:14:04.561134+0000", 2024-09-18T08:21:05.444 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 2.3430000000000001e-05 2024-09-18T08:21:05.444 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:21:05.444 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:21:05.444 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-09-18T08:21:05.444 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:14:04.561142+0000", 2024-09-18T08:21:05.444 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 8.1689999999999992e-06 2024-09-18T08:21:05.445 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:21:05.445 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:21:05.445 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-18T08:21:05.445 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:14:04.561226+0000", 2024-09-18T08:21:05.445 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 8.3919999999999996e-05 2024-09-18T08:21:05.445 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:21:05.445 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:21:05.445 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-09-18T08:21:05.445 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:14:04.561335+0000", 2024-09-18T08:21:05.445 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000109313 2024-09-18T08:21:05.445 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:21:05.445 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:21:05.445 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "waiting for subops from 3,7", 2024-09-18T08:21:05.445 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:14:04.561452+0000", 2024-09-18T08:21:05.446 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000116501 2024-09-18T08:21:05.446 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:21:05.446 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:21:05.446 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-18T08:21:05.446 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:14:04.585657+0000", 2024-09-18T08:21:05.446 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.024205082999999999 2024-09-18T08:21:05.446 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:21:05.446 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:21:05.446 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "waiting for peered", 2024-09-18T08:21:05.446 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:14:04.585671+0000", 2024-09-18T08:21:05.447 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.3998e-05 2024-09-18T08:21:05.447 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:21:05.447 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:21:05.447 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-18T08:21:05.447 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:14:05.593401+0000", 2024-09-18T08:21:05.447 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.0077304380000001 2024-09-18T08:21:05.447 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:21:05.448 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:21:05.448 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "waiting for peered", 2024-09-18T08:21:05.448 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:14:05.593414+0000", 2024-09-18T08:21:05.449 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.2656999999999999e-05 2024-09-18T08:21:05.449 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:21:05.449 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:21:05.449 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-18T08:21:05.449 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:14:06.726634+0000", 2024-09-18T08:21:05.449 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.133220077 2024-09-18T08:21:05.449 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:21:05.449 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:21:05.449 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-09-18T08:21:05.449 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:14:06.726697+0000", 2024-09-18T08:21:05.449 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 6.3021000000000005e-05 2024-09-18T08:21:05.449 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-18T08:21:05.450 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-09-18T08:21:05.450 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-18T08:21:05.450 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:21:05.450 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:21:05.450 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.15582.0:252492 9.7 9:e97f18e4:::benchmark_data_smithi007_124019_object110607:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e291)", 2024-09-18T08:21:05.450 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-09-18T08:14:04.568168+0000", 2024-09-18T08:21:05.450 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 420.86987334299999, 2024-09-18T08:21:05.450 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 2.1587796849999998, 2024-09-18T08:21:05.450 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-09-18T08:21:05.451 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-09-18T08:21:05.451 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "reached pg", 2024-09-18T08:21:05.451 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2024-09-18T08:21:05.451 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.15582", 2024-09-18T08:21:05.451 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "172.21.15.7:0/3598605839", 2024-09-18T08:21:05.451 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 252492 2024-09-18T08:21:05.451 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:21:05.451 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-09-18T08:21:05.451 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:21:05.451 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-09-18T08:21:05.451 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:14:04.568168+0000", 2024-09-18T08:21:05.451 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-18T08:21:05.451 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:21:05.452 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:21:05.452 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-09-18T08:21:05.452 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:14:04.568168+0000", 2024-09-18T08:21:05.452 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-18T08:21:05.452 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:21:05.452 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:21:05.452 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-09-18T08:21:05.452 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:14:04.568169+0000", 2024-09-18T08:21:05.452 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.342e-06 2024-09-18T08:21:05.452 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:21:05.452 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:21:05.452 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-09-18T08:21:05.452 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:14:04.568174+0000", 2024-09-18T08:21:05.453 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4.6539999999999998e-06 2024-09-18T08:21:05.453 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:21:05.453 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:21:05.453 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-09-18T08:21:05.453 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:14:04.568184+0000", 2024-09-18T08:21:05.453 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.0883999999999999e-05 2024-09-18T08:21:05.453 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:21:05.453 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:21:05.453 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-09-18T08:21:05.453 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:14:04.568192+0000", 2024-09-18T08:21:05.453 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 7.5739999999999998e-06 2024-09-18T08:21:05.453 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:21:05.453 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:21:05.453 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-18T08:21:05.454 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:14:04.568688+0000", 2024-09-18T08:21:05.454 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00049632500000000004 2024-09-18T08:21:05.454 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:21:05.454 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:21:05.454 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-09-18T08:21:05.454 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:14:04.568817+0000", 2024-09-18T08:21:05.454 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00012876000000000001 2024-09-18T08:21:05.454 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:21:05.454 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:21:05.454 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "waiting for subops from 3,7", 2024-09-18T08:21:05.454 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:14:04.568949+0000", 2024-09-18T08:21:05.454 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00013194300000000001 2024-09-18T08:21:05.454 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:21:05.455 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:21:05.455 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-18T08:21:05.455 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:14:04.585705+0000", 2024-09-18T08:21:05.455 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.016755823 2024-09-18T08:21:05.455 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:21:05.455 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:21:05.455 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "waiting for peered", 2024-09-18T08:21:05.455 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:14:04.585715+0000", 2024-09-18T08:21:05.455 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.0547000000000001e-05 2024-09-18T08:21:05.455 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:21:05.455 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:21:05.455 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-18T08:21:05.455 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:14:05.593509+0000", 2024-09-18T08:21:05.455 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.007793607 2024-09-18T08:21:05.456 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:21:05.456 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:21:05.456 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "waiting for peered", 2024-09-18T08:21:05.456 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:14:05.593521+0000", 2024-09-18T08:21:05.456 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.1834000000000001e-05 2024-09-18T08:21:05.456 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:21:05.456 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:21:05.456 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-18T08:21:05.456 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:14:06.726906+0000", 2024-09-18T08:21:05.456 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.1333849229999999 2024-09-18T08:21:05.456 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:21:05.456 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:21:05.456 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-09-18T08:21:05.457 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:14:06.726947+0000", 2024-09-18T08:21:05.457 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4.1468999999999999e-05 2024-09-18T08:21:05.457 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-18T08:21:05.457 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-09-18T08:21:05.457 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-18T08:21:05.457 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:21:05.457 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:21:05.457 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.15582.0:252495 9.7 9:e76824cc:::benchmark_data_smithi007_124019_object110610:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e292)", 2024-09-18T08:21:05.457 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-09-18T08:14:04.576440+0000", 2024-09-18T08:21:05.458 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 420.861601065, 2024-09-18T08:21:05.458 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 2.2199907799999998, 2024-09-18T08:21:05.458 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-09-18T08:21:05.458 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-09-18T08:21:05.458 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T08:21:05.458 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2024-09-18T08:21:05.458 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.15582", 2024-09-18T08:21:05.459 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "172.21.15.7:0/3598605839", 2024-09-18T08:21:05.459 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 252495 2024-09-18T08:21:05.459 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:21:05.459 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-09-18T08:21:05.459 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:21:05.459 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-09-18T08:21:05.459 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:14:04.576440+0000", 2024-09-18T08:21:05.459 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-18T08:21:05.459 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:21:05.459 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:21:05.459 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-09-18T08:21:05.459 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:14:04.576440+0000", 2024-09-18T08:21:05.460 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-18T08:21:05.460 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:21:05.460 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:21:05.460 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-09-18T08:21:05.460 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:14:04.576441+0000", 2024-09-18T08:21:05.460 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.663e-06 2024-09-18T08:21:05.460 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:21:05.460 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:21:05.460 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-09-18T08:21:05.461 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:14:04.576447+0000", 2024-09-18T08:21:05.461 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 5.806e-06 2024-09-18T08:21:05.461 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:21:05.461 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:21:05.461 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-09-18T08:21:05.461 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:14:04.576458+0000", 2024-09-18T08:21:05.461 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.0458999999999999e-05 2024-09-18T08:21:05.461 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:21:05.461 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:21:05.461 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-09-18T08:21:05.461 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:14:04.576467+0000", 2024-09-18T08:21:05.461 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 9.2229999999999996e-06 2024-09-18T08:21:05.461 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:21:05.462 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:21:05.462 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-18T08:21:05.462 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:14:04.585743+0000", 2024-09-18T08:21:05.462 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0092757240000000008 2024-09-18T08:21:05.462 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:21:05.462 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:21:05.462 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "waiting for peered", 2024-09-18T08:21:05.462 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:14:04.585752+0000", 2024-09-18T08:21:05.462 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 9.0790000000000007e-06 2024-09-18T08:21:05.462 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:21:05.462 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:21:05.462 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-18T08:21:05.462 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:14:05.593554+0000", 2024-09-18T08:21:05.462 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.007802423 2024-09-18T08:21:05.463 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:21:05.463 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:21:05.463 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "waiting for peered", 2024-09-18T08:21:05.463 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:14:05.593563+0000", 2024-09-18T08:21:05.463 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 9.0359999999999995e-06 2024-09-18T08:21:05.463 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:21:05.463 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:21:05.463 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-18T08:21:05.463 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:14:06.727007+0000", 2024-09-18T08:21:05.463 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.1334437239999999 2024-09-18T08:21:05.463 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:21:05.463 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:21:05.463 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-09-18T08:21:05.463 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:14:06.727083+0000", 2024-09-18T08:21:05.464 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 7.5791000000000007e-05 2024-09-18T08:21:05.464 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:21:05.464 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:21:05.464 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "waiting for subops from 2,3,7", 2024-09-18T08:21:05.464 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:14:06.727428+0000", 2024-09-18T08:21:05.464 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00034489900000000002 2024-09-18T08:21:05.464 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:21:05.464 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:21:05.464 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:21:05.464 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:14:06.732851+0000", 2024-09-18T08:21:05.464 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0054237729999999998 2024-09-18T08:21:05.464 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:21:05.465 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:21:05.465 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "op_commit", 2024-09-18T08:21:05.465 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:14:06.733574+0000", 2024-09-18T08:21:05.465 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00072280300000000005 2024-09-18T08:21:05.465 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:21:05.465 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:21:05.465 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:21:05.465 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:14:06.733630+0000", 2024-09-18T08:21:05.465 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 5.5566000000000003e-05 2024-09-18T08:21:05.465 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:21:05.465 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:21:05.465 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:21:05.465 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:14:06.796241+0000", 2024-09-18T08:21:05.465 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.062611555999999999 2024-09-18T08:21:05.466 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:21:05.466 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:21:05.466 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "commit_sent", 2024-09-18T08:21:05.466 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:14:06.796314+0000", 2024-09-18T08:21:05.466 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 7.2207999999999997e-05 2024-09-18T08:21:05.466 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:21:05.466 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:21:05.466 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-09-18T08:21:05.466 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:14:06.796431+0000", 2024-09-18T08:21:05.466 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00011704700000000001 2024-09-18T08:21:05.466 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-18T08:21:05.466 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-09-18T08:21:05.466 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-18T08:21:05.466 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:21:05.467 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:21:05.467 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.15582.0:252496 9.a 9:5293ceb8:::benchmark_data_smithi007_124019_object110611:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e292)", 2024-09-18T08:21:05.467 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-09-18T08:14:04.579029+0000", 2024-09-18T08:21:05.467 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 420.85901161200002, 2024-09-18T08:21:05.467 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 2.224896416, 2024-09-18T08:21:05.467 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-09-18T08:21:05.467 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-09-18T08:21:05.467 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T08:21:05.467 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2024-09-18T08:21:05.467 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.15582", 2024-09-18T08:21:05.467 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "172.21.15.7:0/3598605839", 2024-09-18T08:21:05.467 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 252496 2024-09-18T08:21:05.467 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:21:05.467 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-09-18T08:21:05.468 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:21:05.468 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-09-18T08:21:05.468 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:14:04.579029+0000", 2024-09-18T08:21:05.468 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-18T08:21:05.468 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:21:05.468 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:21:05.468 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-09-18T08:21:05.468 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:14:04.579029+0000", 2024-09-18T08:21:05.468 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-18T08:21:05.468 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:21:05.468 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:21:05.468 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-09-18T08:21:05.468 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:14:04.579030+0000", 2024-09-18T08:21:05.469 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.091e-06 2024-09-18T08:21:05.469 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:21:05.469 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:21:05.469 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-09-18T08:21:05.469 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:14:04.579036+0000", 2024-09-18T08:21:05.469 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 5.7069999999999997e-06 2024-09-18T08:21:05.469 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:21:05.469 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:21:05.469 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-09-18T08:21:05.469 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:14:04.579045+0000", 2024-09-18T08:21:05.469 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 9.2560000000000008e-06 2024-09-18T08:21:05.469 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:21:05.469 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:21:05.469 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-09-18T08:21:05.470 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:14:04.579054+0000", 2024-09-18T08:21:05.470 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 8.6510000000000007e-06 2024-09-18T08:21:05.470 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:21:05.470 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:21:05.470 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-18T08:21:05.470 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:14:04.586401+0000", 2024-09-18T08:21:05.470 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0073470150000000001 2024-09-18T08:21:05.470 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:21:05.470 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:21:05.470 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "waiting for peered", 2024-09-18T08:21:05.470 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:14:04.586409+0000", 2024-09-18T08:21:05.470 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 7.7919999999999993e-06 2024-09-18T08:21:05.471 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:21:05.471 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:21:05.471 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-18T08:21:05.471 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:14:05.578537+0000", 2024-09-18T08:21:05.471 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.99212811400000001 2024-09-18T08:21:05.471 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:21:05.471 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:21:05.471 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "waiting for peered", 2024-09-18T08:21:05.471 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:14:05.578550+0000", 2024-09-18T08:21:05.471 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.3069e-05 2024-09-18T08:21:05.471 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:21:05.472 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:21:05.472 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-18T08:21:05.472 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:14:06.795766+0000", 2024-09-18T08:21:05.472 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.2172160480000001 2024-09-18T08:21:05.472 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:21:05.472 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:21:05.472 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-09-18T08:21:05.472 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:14:06.795884+0000", 2024-09-18T08:21:05.472 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00011841 2024-09-18T08:21:05.472 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:21:05.472 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:21:05.472 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "waiting for subops from 3,7", 2024-09-18T08:21:05.472 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:14:06.796256+0000", 2024-09-18T08:21:05.473 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00037132700000000002 2024-09-18T08:21:05.473 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:21:05.473 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:21:05.473 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:21:05.473 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:14:06.803555+0000", 2024-09-18T08:21:05.473 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0072990449999999997 2024-09-18T08:21:05.473 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:21:05.473 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:21:05.473 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:21:05.473 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:14:06.803736+0000", 2024-09-18T08:21:05.473 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00018114300000000001 2024-09-18T08:21:05.473 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:21:05.473 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:21:05.474 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "op_commit", 2024-09-18T08:21:05.474 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:14:06.803750+0000", 2024-09-18T08:21:05.474 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.414e-05 2024-09-18T08:21:05.474 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:21:05.474 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:21:05.474 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "commit_sent", 2024-09-18T08:21:05.474 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:14:06.803809+0000", 2024-09-18T08:21:05.474 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 5.8817999999999998e-05 2024-09-18T08:21:05.474 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:21:05.474 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:21:05.474 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-09-18T08:21:05.474 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:14:06.803926+0000", 2024-09-18T08:21:05.474 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00011679 2024-09-18T08:21:05.475 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-18T08:21:05.475 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-09-18T08:21:05.475 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-18T08:21:05.475 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:21:05.475 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:21:05.475 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.15582.0:252503 9.d 9:b2496e7b:::benchmark_data_smithi007_124019_object110618:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e292)", 2024-09-18T08:21:05.475 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-09-18T08:14:04.603611+0000", 2024-09-18T08:21:05.475 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 420.83442984700002, 2024-09-18T08:21:05.475 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 2.134165211, 2024-09-18T08:21:05.475 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-09-18T08:21:05.475 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-09-18T08:21:05.475 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T08:21:05.475 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2024-09-18T08:21:05.475 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.15582", 2024-09-18T08:21:05.476 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "172.21.15.7:0/3598605839", 2024-09-18T08:21:05.476 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 252503 2024-09-18T08:21:05.476 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:21:05.476 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-09-18T08:21:05.476 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:21:05.476 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-09-18T08:21:05.476 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:14:04.603611+0000", 2024-09-18T08:21:05.476 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-18T08:21:05.476 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:21:05.476 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:21:05.476 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-09-18T08:21:05.476 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:14:04.603611+0000", 2024-09-18T08:21:05.476 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-18T08:21:05.477 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:21:05.477 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:21:05.477 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-09-18T08:21:05.477 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:14:04.603612+0000", 2024-09-18T08:21:05.477 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.2219999999999999e-06 2024-09-18T08:21:05.477 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:21:05.477 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:21:05.477 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-09-18T08:21:05.477 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:14:04.603617+0000", 2024-09-18T08:21:05.477 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4.8899999999999998e-06 2024-09-18T08:21:05.477 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:21:05.477 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:21:05.477 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-09-18T08:21:05.478 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:14:04.603628+0000", 2024-09-18T08:21:05.478 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.0899e-05 2024-09-18T08:21:05.478 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:21:05.478 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:21:05.478 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-09-18T08:21:05.478 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:14:04.603634+0000", 2024-09-18T08:21:05.478 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 5.5330000000000002e-06 2024-09-18T08:21:05.478 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:21:05.478 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:21:05.478 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-18T08:21:05.478 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:14:04.603676+0000", 2024-09-18T08:21:05.478 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4.2120999999999999e-05 2024-09-18T08:21:05.478 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:21:05.478 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:21:05.479 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "waiting for peered", 2024-09-18T08:21:05.479 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:14:04.603682+0000", 2024-09-18T08:21:05.479 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 5.835e-06 2024-09-18T08:21:05.479 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:21:05.479 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:21:05.479 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-18T08:21:05.479 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:14:05.602418+0000", 2024-09-18T08:21:05.479 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.998736654 2024-09-18T08:21:05.479 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:21:05.479 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:21:05.479 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "waiting for peered", 2024-09-18T08:21:05.479 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:14:05.602429+0000", 2024-09-18T08:21:05.479 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.044e-05 2024-09-18T08:21:05.480 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:21:05.480 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:21:05.480 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-18T08:21:05.480 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:14:06.731289+0000", 2024-09-18T08:21:05.480 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.1288608339999999 2024-09-18T08:21:05.480 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:21:05.480 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:21:05.480 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-09-18T08:21:05.480 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:14:06.731419+0000", 2024-09-18T08:21:05.480 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00012965200000000001 2024-09-18T08:21:05.480 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:21:05.480 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:21:05.480 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "waiting for subops from 4,7", 2024-09-18T08:21:05.480 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:14:06.731883+0000", 2024-09-18T08:21:05.481 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00046438000000000002 2024-09-18T08:21:05.481 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:21:05.481 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:21:05.481 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:21:05.481 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:14:06.737536+0000", 2024-09-18T08:21:05.481 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0056523299999999997 2024-09-18T08:21:05.481 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:21:05.481 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:21:05.481 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "op_commit", 2024-09-18T08:21:05.481 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:14:06.737568+0000", 2024-09-18T08:21:05.481 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 3.2668e-05 2024-09-18T08:21:05.481 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:21:05.481 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:21:05.482 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:21:05.482 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:14:06.737612+0000", 2024-09-18T08:21:05.482 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4.3686999999999997e-05 2024-09-18T08:21:05.482 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:21:05.482 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:21:05.482 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "commit_sent", 2024-09-18T08:21:05.482 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:14:06.737674+0000", 2024-09-18T08:21:05.482 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 6.2100000000000005e-05 2024-09-18T08:21:05.482 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:21:05.482 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:21:05.482 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-09-18T08:21:05.482 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:14:06.737776+0000", 2024-09-18T08:21:05.482 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00010196600000000001 2024-09-18T08:21:05.482 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-18T08:21:05.483 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-09-18T08:21:05.483 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-18T08:21:05.483 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:21:05.483 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:21:05.483 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.26087.0:2240 11.5 11:a981f685:::benchmark_data_smithi007_172546_object2239:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e406)", 2024-09-18T08:21:05.483 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-09-18T08:18:35.839081+0000", 2024-09-18T08:21:05.483 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 149.59895950200001, 2024-09-18T08:21:05.483 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.26722512300000001, 2024-09-18T08:21:05.483 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-09-18T08:21:05.483 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-09-18T08:21:05.483 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T08:21:05.483 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2024-09-18T08:21:05.483 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.26087", 2024-09-18T08:21:05.484 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "172.21.15.7:0/2752273185", 2024-09-18T08:21:05.484 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 2240 2024-09-18T08:21:05.484 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:21:05.484 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-09-18T08:21:05.484 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:21:05.484 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-09-18T08:21:05.484 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:18:35.839081+0000", 2024-09-18T08:21:05.484 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-18T08:21:05.484 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:21:05.484 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:21:05.484 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-09-18T08:21:05.484 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:18:35.839081+0000", 2024-09-18T08:21:05.484 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-18T08:21:05.484 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:21:05.485 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:21:05.485 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-09-18T08:21:05.485 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:18:35.839083+0000", 2024-09-18T08:21:05.485 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.277e-06 2024-09-18T08:21:05.485 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:21:05.485 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:21:05.485 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-09-18T08:21:05.485 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:18:35.839136+0000", 2024-09-18T08:21:05.485 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 5.3327e-05 2024-09-18T08:21:05.485 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:21:05.485 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:21:05.485 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-09-18T08:21:05.485 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:18:35.839147+0000", 2024-09-18T08:21:05.485 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.1034e-05 2024-09-18T08:21:05.486 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:21:05.486 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:21:05.486 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-09-18T08:21:05.486 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:18:35.839153+0000", 2024-09-18T08:21:05.486 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 5.7150000000000003e-06 2024-09-18T08:21:05.486 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:21:05.486 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:21:05.486 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-18T08:21:05.486 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:18:35.839204+0000", 2024-09-18T08:21:05.486 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 5.0903999999999999e-05 2024-09-18T08:21:05.486 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:21:05.486 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:21:05.486 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-09-18T08:21:05.487 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:18:35.839336+0000", 2024-09-18T08:21:05.487 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000132588 2024-09-18T08:21:05.487 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:21:05.487 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:21:05.487 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "waiting for subops from 6,7", 2024-09-18T08:21:05.487 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:18:35.839467+0000", 2024-09-18T08:21:05.487 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00013079000000000001 2024-09-18T08:21:05.487 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:21:05.487 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:21:05.487 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "op_commit", 2024-09-18T08:21:05.487 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:18:35.840246+0000", 2024-09-18T08:21:05.487 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00077857899999999995 2024-09-18T08:21:05.487 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:21:05.488 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:21:05.488 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:21:05.488 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:18:35.840284+0000", 2024-09-18T08:21:05.488 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 3.8473999999999998e-05 2024-09-18T08:21:05.488 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:21:05.488 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:21:05.488 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:21:05.488 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:18:36.106135+0000", 2024-09-18T08:21:05.488 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.26585117800000002 2024-09-18T08:21:05.488 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:21:05.488 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:21:05.488 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "commit_sent", 2024-09-18T08:21:05.488 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:18:36.106206+0000", 2024-09-18T08:21:05.489 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 7.0889999999999994e-05 2024-09-18T08:21:05.489 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:21:05.489 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:21:05.489 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-09-18T08:21:05.489 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:18:36.106306+0000", 2024-09-18T08:21:05.489 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000100367 2024-09-18T08:21:05.489 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-18T08:21:05.489 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-09-18T08:21:05.489 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-18T08:21:05.489 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:21:05.489 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:21:05.489 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.26087.0:2241 11.5 11:a4a93508:::benchmark_data_smithi007_172546_object2240:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e406)", 2024-09-18T08:21:05.489 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-09-18T08:18:35.839180+0000", 2024-09-18T08:21:05.490 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 149.59886098600001, 2024-09-18T08:21:05.490 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.267269429, 2024-09-18T08:21:05.490 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-09-18T08:21:05.490 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-09-18T08:21:05.490 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T08:21:05.490 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2024-09-18T08:21:05.490 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.26087", 2024-09-18T08:21:05.490 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "172.21.15.7:0/2752273185", 2024-09-18T08:21:05.490 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 2241 2024-09-18T08:21:05.490 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:21:05.490 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-09-18T08:21:05.490 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:21:05.490 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-09-18T08:21:05.491 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:18:35.839180+0000", 2024-09-18T08:21:05.491 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-18T08:21:05.491 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:21:05.491 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:21:05.491 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-09-18T08:21:05.491 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:18:35.839180+0000", 2024-09-18T08:21:05.491 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-18T08:21:05.491 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:21:05.491 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:21:05.491 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-09-18T08:21:05.491 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:18:35.839180+0000", 2024-09-18T08:21:05.491 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 6.1099999999999995e-07 2024-09-18T08:21:05.491 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:21:05.491 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:21:05.492 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-09-18T08:21:05.492 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:18:35.839262+0000", 2024-09-18T08:21:05.492 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 8.1383000000000003e-05 2024-09-18T08:21:05.492 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:21:05.492 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:21:05.492 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-09-18T08:21:05.492 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:18:35.839272+0000", 2024-09-18T08:21:05.492 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.023e-05 2024-09-18T08:21:05.492 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:21:05.492 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:21:05.492 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-09-18T08:21:05.492 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:18:35.839280+0000", 2024-09-18T08:21:05.492 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 7.9009999999999994e-06 2024-09-18T08:21:05.493 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:21:05.493 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:21:05.493 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-18T08:21:05.493 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:18:35.839804+0000", 2024-09-18T08:21:05.493 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00052360999999999998 2024-09-18T08:21:05.493 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:21:05.493 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:21:05.493 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-09-18T08:21:05.493 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:18:35.839889+0000", 2024-09-18T08:21:05.493 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 8.5507999999999995e-05 2024-09-18T08:21:05.493 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:21:05.493 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:21:05.493 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "waiting for subops from 6,7", 2024-09-18T08:21:05.493 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:18:35.839983+0000", 2024-09-18T08:21:05.494 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 9.4135999999999995e-05 2024-09-18T08:21:05.494 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:21:05.494 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:21:05.494 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "op_commit", 2024-09-18T08:21:05.494 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:18:35.840387+0000", 2024-09-18T08:21:05.494 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00040337400000000002 2024-09-18T08:21:05.494 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:21:05.494 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:21:05.494 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:21:05.494 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:18:35.840719+0000", 2024-09-18T08:21:05.494 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000332073 2024-09-18T08:21:05.494 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:21:05.494 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:21:05.494 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:21:05.495 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:18:36.106351+0000", 2024-09-18T08:21:05.495 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.265632164 2024-09-18T08:21:05.495 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:21:05.495 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:21:05.495 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "commit_sent", 2024-09-18T08:21:05.495 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:18:36.106410+0000", 2024-09-18T08:21:05.495 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 5.9185999999999999e-05 2024-09-18T08:21:05.495 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:21:05.495 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:21:05.495 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-09-18T08:21:05.495 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:18:36.106449+0000", 2024-09-18T08:21:05.495 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 3.9252999999999998e-05 2024-09-18T08:21:05.495 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-18T08:21:05.495 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-09-18T08:21:05.496 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-18T08:21:05.496 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:21:05.496 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:21:05.496 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.26087.0:2246 11.5 11:a7b3e814:::benchmark_data_smithi007_172546_object2245:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e406)", 2024-09-18T08:21:05.496 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-09-18T08:18:35.841488+0000", 2024-09-18T08:21:05.496 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 149.59655241999999, 2024-09-18T08:21:05.496 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.26533874800000001, 2024-09-18T08:21:05.496 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-09-18T08:21:05.496 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-09-18T08:21:05.496 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T08:21:05.496 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2024-09-18T08:21:05.496 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.26087", 2024-09-18T08:21:05.496 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "172.21.15.7:0/2752273185", 2024-09-18T08:21:05.497 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 2246 2024-09-18T08:21:05.497 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:21:05.497 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-09-18T08:21:05.497 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:21:05.497 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-09-18T08:21:05.497 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:18:35.841488+0000", 2024-09-18T08:21:05.497 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-18T08:21:05.497 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:21:05.497 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:21:05.497 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-09-18T08:21:05.497 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:18:35.841488+0000", 2024-09-18T08:21:05.497 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-18T08:21:05.497 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:21:05.497 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:21:05.498 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-09-18T08:21:05.498 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:18:35.841489+0000", 2024-09-18T08:21:05.498 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 8.1100000000000005e-07 2024-09-18T08:21:05.498 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:21:05.498 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:21:05.498 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-09-18T08:21:05.498 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:18:35.841570+0000", 2024-09-18T08:21:05.498 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 8.0909999999999996e-05 2024-09-18T08:21:05.498 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:21:05.498 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:21:05.498 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-09-18T08:21:05.498 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:18:35.841580+0000", 2024-09-18T08:21:05.498 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.0127e-05 2024-09-18T08:21:05.498 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:21:05.499 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:21:05.499 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-09-18T08:21:05.499 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:18:35.841586+0000", 2024-09-18T08:21:05.499 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 5.8440000000000003e-06 2024-09-18T08:21:05.499 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:21:05.499 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:21:05.499 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-18T08:21:05.499 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:18:35.841623+0000", 2024-09-18T08:21:05.499 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 3.6767000000000002e-05 2024-09-18T08:21:05.499 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:21:05.499 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:21:05.499 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-09-18T08:21:05.499 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:18:35.841708+0000", 2024-09-18T08:21:05.499 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 8.5439999999999995e-05 2024-09-18T08:21:05.500 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:21:05.500 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:21:05.500 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "waiting for subops from 6,7", 2024-09-18T08:21:05.500 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:18:35.841794+0000", 2024-09-18T08:21:05.500 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 8.5756999999999999e-05 2024-09-18T08:21:05.500 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:21:05.500 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:21:05.500 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "op_commit", 2024-09-18T08:21:05.500 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:18:35.842153+0000", 2024-09-18T08:21:05.500 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00035907099999999998 2024-09-18T08:21:05.500 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:21:05.500 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:21:05.500 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:21:05.501 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:18:35.842397+0000", 2024-09-18T08:21:05.501 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00024352799999999999 2024-09-18T08:21:05.501 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:21:05.501 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:21:05.501 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:21:05.501 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:18:36.106721+0000", 2024-09-18T08:21:05.502 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.26432397099999999 2024-09-18T08:21:05.502 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:21:05.502 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:21:05.502 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "commit_sent", 2024-09-18T08:21:05.502 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:18:36.106783+0000", 2024-09-18T08:21:05.502 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 6.2724000000000005e-05 2024-09-18T08:21:05.502 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:21:05.502 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:21:05.502 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-09-18T08:21:05.502 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:18:36.106827+0000", 2024-09-18T08:21:05.503 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4.3798000000000003e-05 2024-09-18T08:21:05.503 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-18T08:21:05.503 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-09-18T08:21:05.503 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-18T08:21:05.503 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:21:05.503 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:21:05.503 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.26087.0:2376 11.5 11:a232a325:::benchmark_data_smithi007_172546_object2375:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e406)", 2024-09-18T08:21:05.503 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-09-18T08:18:36.146689+0000", 2024-09-18T08:21:05.503 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 149.29135218799999, 2024-09-18T08:21:05.503 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.178476043, 2024-09-18T08:21:05.503 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-09-18T08:21:05.503 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-09-18T08:21:05.503 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T08:21:05.504 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2024-09-18T08:21:05.504 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.26087", 2024-09-18T08:21:05.504 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "172.21.15.7:0/2752273185", 2024-09-18T08:21:05.504 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 2376 2024-09-18T08:21:05.504 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:21:05.504 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-09-18T08:21:05.504 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:21:05.504 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-09-18T08:21:05.504 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:18:36.146689+0000", 2024-09-18T08:21:05.504 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-18T08:21:05.504 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:21:05.504 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:21:05.504 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-09-18T08:21:05.505 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:18:36.146689+0000", 2024-09-18T08:21:05.505 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-18T08:21:05.505 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:21:05.505 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:21:05.505 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-09-18T08:21:05.505 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:18:36.146690+0000", 2024-09-18T08:21:05.505 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.11e-06 2024-09-18T08:21:05.505 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:21:05.505 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:21:05.505 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-09-18T08:21:05.505 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:18:36.146759+0000", 2024-09-18T08:21:05.505 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 6.9010999999999994e-05 2024-09-18T08:21:05.505 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:21:05.506 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:21:05.506 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-09-18T08:21:05.506 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:18:36.146774+0000", 2024-09-18T08:21:05.506 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.4902000000000001e-05 2024-09-18T08:21:05.506 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:21:05.506 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:21:05.506 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-09-18T08:21:05.506 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:18:36.146782+0000", 2024-09-18T08:21:05.506 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 8.4649999999999994e-06 2024-09-18T08:21:05.506 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:21:05.506 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:21:05.506 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-18T08:21:05.506 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:18:36.146845+0000", 2024-09-18T08:21:05.506 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 6.2817000000000005e-05 2024-09-18T08:21:05.507 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:21:05.507 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:21:05.507 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-09-18T08:21:05.507 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:18:36.146998+0000", 2024-09-18T08:21:05.507 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000153122 2024-09-18T08:21:05.507 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:21:05.507 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:21:05.507 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "waiting for subops from 6,7", 2024-09-18T08:21:05.507 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:18:36.147149+0000", 2024-09-18T08:21:05.507 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00015127599999999999 2024-09-18T08:21:05.507 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:21:05.507 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:21:05.507 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "op_commit", 2024-09-18T08:21:05.508 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:18:36.147671+0000", 2024-09-18T08:21:05.508 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000521449 2024-09-18T08:21:05.508 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:21:05.508 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:21:05.508 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:21:05.508 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:18:36.147873+0000", 2024-09-18T08:21:05.508 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00020173599999999999 2024-09-18T08:21:05.508 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:21:05.508 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:21:05.508 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:21:05.508 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:18:36.325051+0000", 2024-09-18T08:21:05.508 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.17717827999999999 2024-09-18T08:21:05.508 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:21:05.509 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:21:05.509 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "commit_sent", 2024-09-18T08:21:05.509 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:18:36.325116+0000", 2024-09-18T08:21:05.509 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 6.5275000000000001e-05 2024-09-18T08:21:05.509 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:21:05.509 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:21:05.509 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-09-18T08:21:05.509 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:18:36.325165+0000", 2024-09-18T08:21:05.509 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4.8600000000000002e-05 2024-09-18T08:21:05.509 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-18T08:21:05.509 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-09-18T08:21:05.509 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-18T08:21:05.509 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:21:05.509 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:21:05.510 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.26087.0:30792 11.5 11:a79c3452:::benchmark_data_smithi007_172546_object30791:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e408)", 2024-09-18T08:21:05.510 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-09-18T08:18:50.601918+0000", 2024-09-18T08:21:05.510 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 134.83612331, 2024-09-18T08:21:05.510 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.188037382, 2024-09-18T08:21:05.510 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-09-18T08:21:05.510 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-09-18T08:21:05.510 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T08:21:05.510 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2024-09-18T08:21:05.510 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.26087", 2024-09-18T08:21:05.510 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "172.21.15.7:0/2752273185", 2024-09-18T08:21:05.510 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 30792 2024-09-18T08:21:05.510 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:21:05.510 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-09-18T08:21:05.510 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:21:05.511 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-09-18T08:21:05.511 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:18:50.601918+0000", 2024-09-18T08:21:05.511 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-18T08:21:05.511 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:21:05.511 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:21:05.511 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-09-18T08:21:05.511 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:18:50.601918+0000", 2024-09-18T08:21:05.511 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-18T08:21:05.511 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:21:05.511 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:21:05.511 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-09-18T08:21:05.511 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:18:50.601919+0000", 2024-09-18T08:21:05.511 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.8109999999999999e-06 2024-09-18T08:21:05.511 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:21:05.512 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:21:05.512 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-09-18T08:21:05.512 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:18:50.601956+0000", 2024-09-18T08:21:05.512 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 3.6251000000000002e-05 2024-09-18T08:21:05.512 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:21:05.512 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:21:05.512 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-09-18T08:21:05.512 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:18:50.601968+0000", 2024-09-18T08:21:05.512 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.2099999999999999e-05 2024-09-18T08:21:05.513 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:21:05.513 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:21:05.513 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-09-18T08:21:05.513 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:18:50.601975+0000", 2024-09-18T08:21:05.513 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 6.8449999999999997e-06 2024-09-18T08:21:05.513 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:21:05.513 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:21:05.513 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-18T08:21:05.513 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:18:50.602664+0000", 2024-09-18T08:21:05.513 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00068937899999999995 2024-09-18T08:21:05.513 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:21:05.513 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:21:05.513 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-09-18T08:21:05.513 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:18:50.602781+0000", 2024-09-18T08:21:05.514 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000117408 2024-09-18T08:21:05.514 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:21:05.514 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:21:05.514 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "waiting for subops from 6,7", 2024-09-18T08:21:05.514 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:18:50.602948+0000", 2024-09-18T08:21:05.514 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00016655399999999999 2024-09-18T08:21:05.514 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:21:05.514 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:21:05.514 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "op_commit", 2024-09-18T08:21:05.514 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:18:50.604215+0000", 2024-09-18T08:21:05.514 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0012675340000000001 2024-09-18T08:21:05.514 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:21:05.514 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:21:05.514 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:21:05.515 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:18:50.604496+0000", 2024-09-18T08:21:05.515 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00028026899999999998 2024-09-18T08:21:05.515 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:21:05.515 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:21:05.515 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:21:05.515 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:18:50.789833+0000", 2024-09-18T08:21:05.515 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.185336942 2024-09-18T08:21:05.515 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:21:05.515 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:21:05.515 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "commit_sent", 2024-09-18T08:21:05.515 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:18:50.789920+0000", 2024-09-18T08:21:05.515 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 8.7640999999999997e-05 2024-09-18T08:21:05.515 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:21:05.516 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:21:05.516 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-09-18T08:21:05.516 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:18:50.789955+0000", 2024-09-18T08:21:05.516 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 3.4647999999999998e-05 2024-09-18T08:21:05.516 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-18T08:21:05.516 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-09-18T08:21:05.516 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-18T08:21:05.516 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:21:05.516 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:21:05.516 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.26087.0:33256 11.7 11:efcc29c4:::benchmark_data_smithi007_172546_object33255:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e408)", 2024-09-18T08:21:05.516 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-09-18T08:18:52.225763+0000", 2024-09-18T08:21:05.516 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 133.21227737300001, 2024-09-18T08:21:05.516 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.19417530599999999, 2024-09-18T08:21:05.517 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-09-18T08:21:05.517 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-09-18T08:21:05.517 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T08:21:05.517 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2024-09-18T08:21:05.517 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.26087", 2024-09-18T08:21:05.517 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "172.21.15.7:0/2752273185", 2024-09-18T08:21:05.517 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 33256 2024-09-18T08:21:05.517 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:21:05.517 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-09-18T08:21:05.517 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:21:05.517 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-09-18T08:21:05.517 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:18:52.225763+0000", 2024-09-18T08:21:05.517 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-18T08:21:05.517 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:21:05.518 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:21:05.518 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-09-18T08:21:05.518 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:18:52.225763+0000", 2024-09-18T08:21:05.518 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-18T08:21:05.518 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:21:05.518 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:21:05.518 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-09-18T08:21:05.518 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:18:52.225765+0000", 2024-09-18T08:21:05.518 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.516e-06 2024-09-18T08:21:05.518 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:21:05.518 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:21:05.518 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-09-18T08:21:05.518 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:18:52.225812+0000", 2024-09-18T08:21:05.519 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4.7468e-05 2024-09-18T08:21:05.519 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:21:05.519 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:21:05.519 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-09-18T08:21:05.519 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:18:52.225826+0000", 2024-09-18T08:21:05.519 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.3567000000000001e-05 2024-09-18T08:21:05.519 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:21:05.519 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:21:05.519 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-09-18T08:21:05.519 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:18:52.225834+0000", 2024-09-18T08:21:05.519 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 7.9650000000000008e-06 2024-09-18T08:21:05.519 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:21:05.519 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:21:05.519 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-18T08:21:05.520 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:18:52.225911+0000", 2024-09-18T08:21:05.520 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 7.6735000000000003e-05 2024-09-18T08:21:05.520 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:21:05.520 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:21:05.520 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-09-18T08:21:05.520 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:18:52.226065+0000", 2024-09-18T08:21:05.520 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00015443799999999999 2024-09-18T08:21:05.520 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:21:05.520 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:21:05.520 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "waiting for subops from 0,1", 2024-09-18T08:21:05.520 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:18:52.226207+0000", 2024-09-18T08:21:05.520 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000141948 2024-09-18T08:21:05.520 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:21:05.521 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:21:05.521 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "op_commit", 2024-09-18T08:21:05.521 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:18:52.227809+0000", 2024-09-18T08:21:05.521 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.001602128 2024-09-18T08:21:05.521 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:21:05.521 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:21:05.521 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:21:05.521 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:18:52.228376+0000", 2024-09-18T08:21:05.521 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00056678299999999996 2024-09-18T08:21:05.522 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:21:05.522 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:21:05.522 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:21:05.522 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:18:52.419797+0000", 2024-09-18T08:21:05.522 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.19142051099999999 2024-09-18T08:21:05.522 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:21:05.522 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:21:05.522 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "commit_sent", 2024-09-18T08:21:05.523 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:18:52.419870+0000", 2024-09-18T08:21:05.523 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 7.3573999999999997e-05 2024-09-18T08:21:05.523 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:21:05.523 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:21:05.523 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-09-18T08:21:05.523 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:18:52.419939+0000", 2024-09-18T08:21:05.523 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 6.8672999999999998e-05 2024-09-18T08:21:05.523 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-18T08:21:05.523 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-09-18T08:21:05.523 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-18T08:21:05.523 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:21:05.523 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:21:05.523 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.26087.0:33275 11.7 11:e3d6417f:::benchmark_data_smithi007_172546_object33274:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e408)", 2024-09-18T08:21:05.523 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-09-18T08:18:52.263756+0000", 2024-09-18T08:21:05.524 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 133.17428525, 2024-09-18T08:21:05.524 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.16402013100000001, 2024-09-18T08:21:05.524 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-09-18T08:21:05.524 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-09-18T08:21:05.524 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T08:21:05.524 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2024-09-18T08:21:05.524 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.26087", 2024-09-18T08:21:05.524 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "172.21.15.7:0/2752273185", 2024-09-18T08:21:05.524 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 33275 2024-09-18T08:21:05.524 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:21:05.524 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-09-18T08:21:05.524 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:21:05.524 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-09-18T08:21:05.525 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:18:52.263756+0000", 2024-09-18T08:21:05.525 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-18T08:21:05.525 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:21:05.525 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:21:05.525 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-09-18T08:21:05.525 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:18:52.263756+0000", 2024-09-18T08:21:05.525 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-18T08:21:05.525 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:21:05.525 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:21:05.525 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-09-18T08:21:05.525 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:18:52.263758+0000", 2024-09-18T08:21:05.525 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 2.0169999999999999e-06 2024-09-18T08:21:05.525 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:21:05.525 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:21:05.526 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-09-18T08:21:05.526 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:18:52.263789+0000", 2024-09-18T08:21:05.526 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 3.1117e-05 2024-09-18T08:21:05.527 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:21:05.528 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:21:05.528 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-09-18T08:21:05.528 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:18:52.263806+0000", 2024-09-18T08:21:05.528 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.7336000000000001e-05 2024-09-18T08:21:05.528 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:21:05.528 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:21:05.528 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-09-18T08:21:05.528 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:18:52.263814+0000", 2024-09-18T08:21:05.528 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 7.6249999999999998e-06 2024-09-18T08:21:05.528 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:21:05.528 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:21:05.528 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-18T08:21:05.528 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:18:52.263879+0000", 2024-09-18T08:21:05.529 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 6.5574000000000006e-05 2024-09-18T08:21:05.529 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:21:05.529 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:21:05.529 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-09-18T08:21:05.529 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:18:52.264016+0000", 2024-09-18T08:21:05.529 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00013682399999999999 2024-09-18T08:21:05.529 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:21:05.529 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:21:05.529 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "waiting for subops from 0,1", 2024-09-18T08:21:05.529 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:18:52.264218+0000", 2024-09-18T08:21:05.529 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00020174799999999999 2024-09-18T08:21:05.530 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:21:05.530 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:21:05.530 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:21:05.530 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:18:52.265788+0000", 2024-09-18T08:21:05.530 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0015699659999999999 2024-09-18T08:21:05.530 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:21:05.530 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:21:05.530 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "op_commit", 2024-09-18T08:21:05.530 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:18:52.265837+0000", 2024-09-18T08:21:05.530 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4.8878000000000002e-05 2024-09-18T08:21:05.530 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:21:05.530 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:21:05.530 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:21:05.531 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:18:52.427483+0000", 2024-09-18T08:21:05.531 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.16164592 2024-09-18T08:21:05.531 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:21:05.531 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:21:05.531 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "commit_sent", 2024-09-18T08:21:05.531 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:18:52.427730+0000", 2024-09-18T08:21:05.531 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00024787499999999999 2024-09-18T08:21:05.531 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:21:05.531 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:21:05.531 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-09-18T08:21:05.531 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:18:52.427776+0000", 2024-09-18T08:21:05.531 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4.5250999999999998e-05 2024-09-18T08:21:05.532 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-18T08:21:05.532 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-09-18T08:21:05.532 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-18T08:21:05.532 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:21:05.532 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:21:05.532 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.26087.0:34005 11.7 11:e984d40b:::benchmark_data_smithi007_172546_object34004:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e408)", 2024-09-18T08:21:05.532 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-09-18T08:18:52.945222+0000", 2024-09-18T08:21:05.532 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 132.49281919399999, 2024-09-18T08:21:05.532 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.23326103500000001, 2024-09-18T08:21:05.532 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-09-18T08:21:05.532 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-09-18T08:21:05.532 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T08:21:05.533 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2024-09-18T08:21:05.533 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.26087", 2024-09-18T08:21:05.533 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "172.21.15.7:0/2752273185", 2024-09-18T08:21:05.533 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 34005 2024-09-18T08:21:05.533 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:21:05.533 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-09-18T08:21:05.533 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:21:05.533 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-09-18T08:21:05.533 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:18:52.945222+0000", 2024-09-18T08:21:05.533 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-18T08:21:05.533 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:21:05.533 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:21:05.533 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-09-18T08:21:05.534 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:18:52.945222+0000", 2024-09-18T08:21:05.534 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-18T08:21:05.534 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:21:05.534 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:21:05.534 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-09-18T08:21:05.534 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:18:52.945224+0000", 2024-09-18T08:21:05.534 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.9520000000000001e-06 2024-09-18T08:21:05.534 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:21:05.534 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:21:05.534 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-09-18T08:21:05.534 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:18:52.945264+0000", 2024-09-18T08:21:05.534 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4.0796e-05 2024-09-18T08:21:05.534 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:21:05.534 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:21:05.535 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-09-18T08:21:05.535 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:18:52.945279+0000", 2024-09-18T08:21:05.535 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.5021000000000001e-05 2024-09-18T08:21:05.535 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:21:05.535 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:21:05.535 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-09-18T08:21:05.535 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:18:52.945288+0000", 2024-09-18T08:21:05.535 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 8.4039999999999995e-06 2024-09-18T08:21:05.535 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:21:05.535 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:21:05.535 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-18T08:21:05.535 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:18:52.945325+0000", 2024-09-18T08:21:05.535 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 3.6902e-05 2024-09-18T08:21:05.535 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:21:05.536 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:21:05.536 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-09-18T08:21:05.536 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:18:52.945421+0000", 2024-09-18T08:21:05.536 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 9.6563000000000004e-05 2024-09-18T08:21:05.536 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:21:05.536 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:21:05.536 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "waiting for subops from 0,1", 2024-09-18T08:21:05.536 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:18:52.945516+0000", 2024-09-18T08:21:05.536 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 9.4648999999999995e-05 2024-09-18T08:21:05.536 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:21:05.536 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:21:05.536 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "op_commit", 2024-09-18T08:21:05.536 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:18:52.946741+0000", 2024-09-18T08:21:05.537 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0012248140000000001 2024-09-18T08:21:05.537 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:21:05.537 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:21:05.537 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:21:05.537 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:18:52.949494+0000", 2024-09-18T08:21:05.537 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.002753138 2024-09-18T08:21:05.537 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:21:05.537 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:21:05.537 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:21:05.537 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:18:53.178402+0000", 2024-09-18T08:21:05.537 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.22890856400000001 2024-09-18T08:21:05.537 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:21:05.537 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:21:05.537 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "commit_sent", 2024-09-18T08:21:05.538 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:18:53.178443+0000", 2024-09-18T08:21:05.538 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4.0983999999999999e-05 2024-09-18T08:21:05.538 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:21:05.538 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:21:05.538 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-09-18T08:21:05.538 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:18:53.178483+0000", 2024-09-18T08:21:05.538 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 3.9248000000000001e-05 2024-09-18T08:21:05.538 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-18T08:21:05.538 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-09-18T08:21:05.538 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-18T08:21:05.538 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:21:05.538 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:21:05.538 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.26087.0:60750 11.7 11:e4b75a72:::benchmark_data_smithi007_172546_object60749:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e418)", 2024-09-18T08:21:05.538 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-09-18T08:19:18.979662+0000", 2024-09-18T08:21:05.539 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 106.45837852299999, 2024-09-18T08:21:05.539 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.15427664499999999, 2024-09-18T08:21:05.539 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-09-18T08:21:05.539 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-09-18T08:21:05.539 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T08:21:05.539 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2024-09-18T08:21:05.539 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.26087", 2024-09-18T08:21:05.539 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "172.21.15.7:0/2752273185", 2024-09-18T08:21:05.539 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 60750 2024-09-18T08:21:05.539 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:21:05.539 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-09-18T08:21:05.539 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:21:05.539 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-09-18T08:21:05.540 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:19:18.979662+0000", 2024-09-18T08:21:05.540 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-18T08:21:05.540 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:21:05.540 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:21:05.540 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-09-18T08:21:05.540 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:19:18.979662+0000", 2024-09-18T08:21:05.540 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-18T08:21:05.540 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:21:05.540 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:21:05.540 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-09-18T08:21:05.540 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:19:18.979667+0000", 2024-09-18T08:21:05.540 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4.5249999999999999e-06 2024-09-18T08:21:05.540 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:21:05.540 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:21:05.541 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-09-18T08:21:05.541 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:19:18.979720+0000", 2024-09-18T08:21:05.541 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 5.3281e-05 2024-09-18T08:21:05.541 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:21:05.541 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:21:05.541 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-09-18T08:21:05.541 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:19:18.979729+0000", 2024-09-18T08:21:05.541 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 9.0559999999999994e-06 2024-09-18T08:21:05.541 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:21:05.541 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:21:05.541 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-09-18T08:21:05.541 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:19:18.979735+0000", 2024-09-18T08:21:05.541 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 5.366e-06 2024-09-18T08:21:05.541 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:21:05.542 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:21:05.542 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-18T08:21:05.542 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:19:18.980812+0000", 2024-09-18T08:21:05.542 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.001077494 2024-09-18T08:21:05.542 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:21:05.542 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:21:05.542 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-09-18T08:21:05.542 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:19:18.980952+0000", 2024-09-18T08:21:05.542 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000139708 2024-09-18T08:21:05.542 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:21:05.542 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:21:05.542 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "waiting for subops from 0,1", 2024-09-18T08:21:05.542 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:19:18.981074+0000", 2024-09-18T08:21:05.542 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00012176300000000001 2024-09-18T08:21:05.543 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:21:05.543 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:21:05.543 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "op_commit", 2024-09-18T08:21:05.543 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:19:18.983195+0000", 2024-09-18T08:21:05.543 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0021217430000000002 2024-09-18T08:21:05.543 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:21:05.543 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:21:05.543 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:21:05.543 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:19:18.986288+0000", 2024-09-18T08:21:05.543 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.003092792 2024-09-18T08:21:05.543 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:21:05.543 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:21:05.543 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:21:05.543 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:19:19.133855+0000", 2024-09-18T08:21:05.544 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.14756720700000001 2024-09-18T08:21:05.544 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:21:05.544 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:21:05.544 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "commit_sent", 2024-09-18T08:21:05.544 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:19:19.133904+0000", 2024-09-18T08:21:05.544 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4.8890000000000001e-05 2024-09-18T08:21:05.544 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:21:05.544 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:21:05.544 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-09-18T08:21:05.544 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:19:19.133939+0000", 2024-09-18T08:21:05.544 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 3.4820000000000002e-05 2024-09-18T08:21:05.544 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-18T08:21:05.544 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-09-18T08:21:05.545 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-18T08:21:05.545 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:21:05.545 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:21:05.545 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.26087.0:61008 11.7 11:eb41e778:::benchmark_data_smithi007_172546_object61007:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e418)", 2024-09-18T08:21:05.545 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-09-18T08:19:19.503328+0000", 2024-09-18T08:21:05.545 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 105.93471325900001, 2024-09-18T08:21:05.545 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.189244672, 2024-09-18T08:21:05.545 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-09-18T08:21:05.545 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-09-18T08:21:05.545 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T08:21:05.545 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2024-09-18T08:21:05.545 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.26087", 2024-09-18T08:21:05.545 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "172.21.15.7:0/2752273185", 2024-09-18T08:21:05.545 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 61008 2024-09-18T08:21:05.546 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:21:05.546 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-09-18T08:21:05.546 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:21:05.546 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-09-18T08:21:05.546 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:19:19.503328+0000", 2024-09-18T08:21:05.546 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-18T08:21:05.546 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:21:05.546 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:21:05.546 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-09-18T08:21:05.546 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:19:19.503328+0000", 2024-09-18T08:21:05.546 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-18T08:21:05.546 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:21:05.546 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:21:05.546 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-09-18T08:21:05.547 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:19:19.503329+0000", 2024-09-18T08:21:05.547 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2 2024-09-18T08:21:05.548 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.5400000000000001e-06 2024-09-18T08:21:05.548 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:21:05.548 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:21:05.548 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-09-18T08:21:05.549 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:19:19.503387+0000", 2024-09-18T08:21:05.549 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 5.7637999999999999e-05 2024-09-18T08:21:05.549 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:21:05.549 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:21:05.549 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-09-18T08:21:05.549 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:19:19.503400+0000", 2024-09-18T08:21:05.549 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.2802e-05 2024-09-18T08:21:05.549 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:21:05.549 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:21:05.549 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-09-18T08:21:05.549 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:19:19.503408+0000", 2024-09-18T08:21:05.549 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 8.8769999999999999e-06 2024-09-18T08:21:05.549 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:21:05.550 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:21:05.550 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-18T08:21:05.550 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:19:19.503462+0000", 2024-09-18T08:21:05.550 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 5.3430999999999997e-05 2024-09-18T08:21:05.550 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:21:05.550 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:21:05.550 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-09-18T08:21:05.550 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:19:19.503576+0000", 2024-09-18T08:21:05.550 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00011363599999999999 2024-09-18T08:21:05.550 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:21:05.550 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:21:05.550 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "waiting for subops from 0,1", 2024-09-18T08:21:05.550 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:19:19.503713+0000", 2024-09-18T08:21:05.550 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00013749899999999999 2024-09-18T08:21:05.551 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:21:05.551 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:21:05.551 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "op_commit", 2024-09-18T08:21:05.551 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:19:19.505863+0000", 2024-09-18T08:21:05.551 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00215036 2024-09-18T08:21:05.551 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:21:05.551 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:21:05.551 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:21:05.551 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:19:19.506917+0000", 2024-09-18T08:21:05.551 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0010539309999999999 2024-09-18T08:21:05.551 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:21:05.551 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:21:05.551 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:21:05.551 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:19:19.692466+0000", 2024-09-18T08:21:05.552 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.185548571 2024-09-18T08:21:05.552 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:21:05.552 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:21:05.552 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "commit_sent", 2024-09-18T08:21:05.552 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:19:19.692526+0000", 2024-09-18T08:21:05.552 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 6.0523000000000003e-05 2024-09-18T08:21:05.552 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:21:05.552 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:21:05.552 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-09-18T08:21:05.552 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:19:19.692572+0000", 2024-09-18T08:21:05.552 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4.5864e-05 2024-09-18T08:21:05.552 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-18T08:21:05.552 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-09-18T08:21:05.552 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-18T08:21:05.553 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:21:05.553 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:21:05.553 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.26087.0:66944 11.7 11:eb53c664:::benchmark_data_smithi007_172546_object66943:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e422)", 2024-09-18T08:21:05.553 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-09-18T08:19:27.218977+0000", 2024-09-18T08:21:05.553 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 98.219064119999999, 2024-09-18T08:21:05.553 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.204234211, 2024-09-18T08:21:05.553 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-09-18T08:21:05.553 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-09-18T08:21:05.553 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T08:21:05.553 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2024-09-18T08:21:05.553 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.26087", 2024-09-18T08:21:05.553 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "172.21.15.7:0/2752273185", 2024-09-18T08:21:05.553 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 66944 2024-09-18T08:21:05.554 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:21:05.554 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-09-18T08:21:05.554 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:21:05.554 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-09-18T08:21:05.554 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:19:27.218977+0000", 2024-09-18T08:21:05.554 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-18T08:21:05.554 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:21:05.554 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:21:05.554 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-09-18T08:21:05.554 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:19:27.218977+0000", 2024-09-18T08:21:05.554 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-18T08:21:05.554 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:21:05.554 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:21:05.554 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-09-18T08:21:05.555 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:19:27.218979+0000", 2024-09-18T08:21:05.555 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.776e-06 2024-09-18T08:21:05.555 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:21:05.555 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:21:05.555 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-09-18T08:21:05.555 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:19:27.219012+0000", 2024-09-18T08:21:05.555 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 3.328e-05 2024-09-18T08:21:05.555 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:21:05.555 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:21:05.555 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-09-18T08:21:05.555 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:19:27.219023+0000", 2024-09-18T08:21:05.555 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.1508000000000001e-05 2024-09-18T08:21:05.555 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:21:05.555 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:21:05.556 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-09-18T08:21:05.556 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:19:27.219032+0000", 2024-09-18T08:21:05.556 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 8.8249999999999994e-06 2024-09-18T08:21:05.556 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:21:05.556 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:21:05.556 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-18T08:21:05.556 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:19:27.219089+0000", 2024-09-18T08:21:05.556 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 5.6997000000000003e-05 2024-09-18T08:21:05.556 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:21:05.556 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:21:05.556 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-09-18T08:21:05.557 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:19:27.219231+0000", 2024-09-18T08:21:05.557 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000142282 2024-09-18T08:21:05.557 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:21:05.557 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:21:05.557 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "waiting for subops from 0,1", 2024-09-18T08:21:05.557 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:19:27.219390+0000", 2024-09-18T08:21:05.557 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00015866799999999999 2024-09-18T08:21:05.557 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:21:05.557 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:21:05.557 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "op_commit", 2024-09-18T08:21:05.557 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:19:27.224973+0000", 2024-09-18T08:21:05.557 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0055832449999999997 2024-09-18T08:21:05.557 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:21:05.558 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:21:05.558 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:21:05.558 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:19:27.225009+0000", 2024-09-18T08:21:05.558 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 3.6173000000000001e-05 2024-09-18T08:21:05.558 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:21:05.558 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:21:05.558 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:21:05.558 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:19:27.423117+0000", 2024-09-18T08:21:05.558 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.198107859 2024-09-18T08:21:05.558 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:21:05.558 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:21:05.558 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "commit_sent", 2024-09-18T08:21:05.558 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:19:27.423164+0000", 2024-09-18T08:21:05.558 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4.7079000000000001e-05 2024-09-18T08:21:05.559 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:21:05.559 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:21:05.559 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-09-18T08:21:05.559 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:19:27.423211+0000", 2024-09-18T08:21:05.559 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4.6519e-05 2024-09-18T08:21:05.559 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-18T08:21:05.559 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-09-18T08:21:05.559 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-18T08:21:05.559 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:21:05.559 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:21:05.559 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.26087.0:66949 11.7 11:ef316115:::benchmark_data_smithi007_172546_object66948:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e422)", 2024-09-18T08:21:05.559 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-09-18T08:19:27.220453+0000", 2024-09-18T08:21:05.559 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 98.217587819000002, 2024-09-18T08:21:05.559 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.20292474899999999, 2024-09-18T08:21:05.560 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-09-18T08:21:05.560 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-09-18T08:21:05.560 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T08:21:05.560 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2024-09-18T08:21:05.560 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.26087", 2024-09-18T08:21:05.560 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "172.21.15.7:0/2752273185", 2024-09-18T08:21:05.560 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 66949 2024-09-18T08:21:05.560 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:21:05.560 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-09-18T08:21:05.560 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:21:05.560 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-09-18T08:21:05.560 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:19:27.220453+0000", 2024-09-18T08:21:05.560 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-18T08:21:05.560 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:21:05.561 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:21:05.561 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-09-18T08:21:05.561 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:19:27.220453+0000", 2024-09-18T08:21:05.561 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-18T08:21:05.561 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:21:05.561 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:21:05.561 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-09-18T08:21:05.561 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:19:27.220455+0000", 2024-09-18T08:21:05.561 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.536e-06 2024-09-18T08:21:05.561 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:21:05.561 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:21:05.561 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-09-18T08:21:05.561 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:19:27.220484+0000", 2024-09-18T08:21:05.561 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 2.9445999999999999e-05 2024-09-18T08:21:05.562 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:21:05.562 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:21:05.562 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-09-18T08:21:05.562 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:19:27.220495+0000", 2024-09-18T08:21:05.562 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.1014e-05 2024-09-18T08:21:05.562 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:21:05.562 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:21:05.562 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-09-18T08:21:05.562 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:19:27.220502+0000", 2024-09-18T08:21:05.562 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 6.6189999999999996e-06 2024-09-18T08:21:05.562 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:21:05.562 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:21:05.562 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-18T08:21:05.563 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:19:27.221717+0000", 2024-09-18T08:21:05.563 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0012158480000000001 2024-09-18T08:21:05.563 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:21:05.563 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:21:05.563 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-09-18T08:21:05.563 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:19:27.221867+0000", 2024-09-18T08:21:05.563 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00014949900000000001 2024-09-18T08:21:05.563 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:21:05.563 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:21:05.563 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "waiting for subops from 0,1", 2024-09-18T08:21:05.563 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:19:27.222028+0000", 2024-09-18T08:21:05.563 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00016146300000000001 2024-09-18T08:21:05.563 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:21:05.563 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:21:05.564 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:21:05.564 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:19:27.225257+0000", 2024-09-18T08:21:05.564 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00322806 2024-09-18T08:21:05.564 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:21:05.564 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:21:05.564 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "op_commit", 2024-09-18T08:21:05.564 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:19:27.225404+0000", 2024-09-18T08:21:05.564 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00014700800000000001 2024-09-18T08:21:05.564 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:21:05.564 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:21:05.564 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:21:05.564 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:19:27.423296+0000", 2024-09-18T08:21:05.564 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.19789293999999999 2024-09-18T08:21:05.565 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:21:05.565 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:21:05.565 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "commit_sent", 2024-09-18T08:21:05.565 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:19:27.423342+0000", 2024-09-18T08:21:05.565 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4.5643000000000003e-05 2024-09-18T08:21:05.565 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:21:05.565 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:21:05.565 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-09-18T08:21:05.565 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:19:27.423378+0000", 2024-09-18T08:21:05.565 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 3.5673000000000002e-05 2024-09-18T08:21:05.565 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-18T08:21:05.565 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-09-18T08:21:05.565 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-18T08:21:05.565 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:21:05.566 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:21:05.566 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.26087.0:151594 11.7 11:ee21e521:::benchmark_data_smithi007_172546_object52910:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e445)", 2024-09-18T08:21:05.566 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-09-18T08:20:47.758030+0000", 2024-09-18T08:21:05.566 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 17.680011231000002, 2024-09-18T08:21:05.566 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.201412602, 2024-09-18T08:21:05.566 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-09-18T08:21:05.566 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-09-18T08:21:05.566 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T08:21:05.566 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2024-09-18T08:21:05.566 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.26087", 2024-09-18T08:21:05.566 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "172.21.15.7:0/2752273185", 2024-09-18T08:21:05.566 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 151594 2024-09-18T08:21:05.566 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:21:05.567 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-09-18T08:21:05.567 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:21:05.567 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-09-18T08:21:05.567 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:20:47.758030+0000", 2024-09-18T08:21:05.567 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-18T08:21:05.567 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:21:05.567 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:21:05.567 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-09-18T08:21:05.567 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:20:47.758030+0000", 2024-09-18T08:21:05.567 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-18T08:21:05.567 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:21:05.567 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:21:05.567 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-09-18T08:21:05.567 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:20:47.758031+0000", 2024-09-18T08:21:05.568 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.6640000000000001e-06 2024-09-18T08:21:05.568 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:21:05.568 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:21:05.568 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-09-18T08:21:05.568 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:20:47.758037+0000", 2024-09-18T08:21:05.568 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 5.4550000000000003e-06 2024-09-18T08:21:05.568 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:21:05.568 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:21:05.568 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-09-18T08:21:05.568 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:20:47.758053+0000", 2024-09-18T08:21:05.568 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.6441999999999999e-05 2024-09-18T08:21:05.568 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:21:05.568 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:21:05.569 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-09-18T08:21:05.569 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:20:47.758061+0000", 2024-09-18T08:21:05.569 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 7.977e-06 2024-09-18T08:21:05.569 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:21:05.569 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:21:05.569 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-18T08:21:05.569 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:20:47.758133+0000", 2024-09-18T08:21:05.569 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 7.1932000000000002e-05 2024-09-18T08:21:05.569 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:21:05.569 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:21:05.569 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-09-18T08:21:05.569 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:20:47.758241+0000", 2024-09-18T08:21:05.569 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000107438 2024-09-18T08:21:05.569 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:21:05.570 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:21:05.570 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "waiting for subops from 0,1", 2024-09-18T08:21:05.570 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:20:47.758353+0000", 2024-09-18T08:21:05.570 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000112933 2024-09-18T08:21:05.570 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:21:05.570 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:21:05.570 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "op_commit", 2024-09-18T08:21:05.570 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:20:47.760780+0000", 2024-09-18T08:21:05.570 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.002426508 2024-09-18T08:21:05.570 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:21:05.570 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:21:05.570 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:21:05.570 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:20:47.761220+0000", 2024-09-18T08:21:05.570 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00044052799999999998 2024-09-18T08:21:05.571 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:21:05.571 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:21:05.571 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:21:05.571 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:20:47.959304+0000", 2024-09-18T08:21:05.571 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.19808378200000001 2024-09-18T08:21:05.571 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:21:05.571 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:21:05.571 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "commit_sent", 2024-09-18T08:21:05.571 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:20:47.959384+0000", 2024-09-18T08:21:05.571 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 7.9838000000000004e-05 2024-09-18T08:21:05.571 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:21:05.571 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:21:05.571 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-09-18T08:21:05.571 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:20:47.959442+0000", 2024-09-18T08:21:05.572 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 5.8105e-05 2024-09-18T08:21:05.572 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-18T08:21:05.572 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-09-18T08:21:05.572 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-18T08:21:05.572 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:21:05.572 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:21:05.572 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.26087.0:151600 11.7 11:ea879652:::benchmark_data_smithi007_172546_object52916:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e445)", 2024-09-18T08:21:05.572 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-09-18T08:20:47.761348+0000", 2024-09-18T08:21:05.572 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 17.676692370000001, 2024-09-18T08:21:05.572 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.20807863099999999, 2024-09-18T08:21:05.572 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-09-18T08:21:05.572 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-09-18T08:21:05.572 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T08:21:05.573 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2024-09-18T08:21:05.573 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.26087", 2024-09-18T08:21:05.573 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "172.21.15.7:0/2752273185", 2024-09-18T08:21:05.573 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 151600 2024-09-18T08:21:05.573 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:21:05.573 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-09-18T08:21:05.573 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:21:05.573 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-09-18T08:21:05.573 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:20:47.761348+0000", 2024-09-18T08:21:05.573 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-18T08:21:05.573 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:21:05.573 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:21:05.573 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-09-18T08:21:05.573 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:20:47.761348+0000", 2024-09-18T08:21:05.574 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-18T08:21:05.574 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:21:05.574 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:21:05.574 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-09-18T08:21:05.574 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:20:47.761350+0000", 2024-09-18T08:21:05.574 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.6840000000000001e-06 2024-09-18T08:21:05.574 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:21:05.574 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:21:05.574 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-09-18T08:21:05.574 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:20:47.761355+0000", 2024-09-18T08:21:05.574 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4.9200000000000003e-06 2024-09-18T08:21:05.574 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:21:05.574 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:21:05.574 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-09-18T08:21:05.575 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:20:47.761367+0000", 2024-09-18T08:21:05.575 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.222e-05 2024-09-18T08:21:05.575 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:21:05.575 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:21:05.575 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-09-18T08:21:05.575 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:20:47.761373+0000", 2024-09-18T08:21:05.575 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 6.0440000000000001e-06 2024-09-18T08:21:05.575 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:21:05.575 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:21:05.575 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-18T08:21:05.575 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:20:47.761422+0000", 2024-09-18T08:21:05.575 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4.8537e-05 2024-09-18T08:21:05.575 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:21:05.575 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:21:05.576 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-09-18T08:21:05.576 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:20:47.761535+0000", 2024-09-18T08:21:05.576 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000113174 2024-09-18T08:21:05.576 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:21:05.576 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:21:05.576 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "waiting for subops from 0,1", 2024-09-18T08:21:05.576 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:20:47.761640+0000", 2024-09-18T08:21:05.576 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00010477 2024-09-18T08:21:05.576 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:21:05.576 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:21:05.576 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "op_commit", 2024-09-18T08:21:05.576 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:20:47.763893+0000", 2024-09-18T08:21:05.576 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.002253171 2024-09-18T08:21:05.576 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:21:05.577 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:21:05.577 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:21:05.577 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:20:47.763992+0000", 2024-09-18T08:21:05.577 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 9.9361e-05 2024-09-18T08:21:05.577 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:21:05.577 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:21:05.577 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:21:05.577 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:20:47.969311+0000", 2024-09-18T08:21:05.577 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.20531872700000001 2024-09-18T08:21:05.577 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:21:05.577 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:21:05.577 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "commit_sent", 2024-09-18T08:21:05.577 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:20:47.969381+0000", 2024-09-18T08:21:05.578 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 6.9643999999999999e-05 2024-09-18T08:21:05.578 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:21:05.578 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:21:05.578 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-09-18T08:21:05.578 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:20:47.969427+0000", 2024-09-18T08:21:05.578 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4.6378999999999998e-05 2024-09-18T08:21:05.578 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-18T08:21:05.578 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-09-18T08:21:05.578 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-18T08:21:05.578 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:21:05.578 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:21:05.578 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.26087.0:151613 11.7 11:eb886232:::benchmark_data_smithi007_172546_object52929:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e445)", 2024-09-18T08:21:05.578 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-09-18T08:20:47.768385+0000", 2024-09-18T08:21:05.578 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 17.669655795000001, 2024-09-18T08:21:05.579 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.201177206, 2024-09-18T08:21:05.579 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-09-18T08:21:05.579 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-09-18T08:21:05.579 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T08:21:05.579 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2024-09-18T08:21:05.579 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.26087", 2024-09-18T08:21:05.579 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "172.21.15.7:0/2752273185", 2024-09-18T08:21:05.579 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 151613 2024-09-18T08:21:05.579 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:21:05.579 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-09-18T08:21:05.579 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:21:05.579 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-09-18T08:21:05.579 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:20:47.768385+0000", 2024-09-18T08:21:05.579 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-18T08:21:05.580 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:21:05.580 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:21:05.580 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-09-18T08:21:05.580 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:20:47.768385+0000", 2024-09-18T08:21:05.580 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-18T08:21:05.580 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:21:05.580 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:21:05.580 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-09-18T08:21:05.580 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:20:47.768387+0000", 2024-09-18T08:21:05.580 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.567e-06 2024-09-18T08:21:05.580 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:21:05.580 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:21:05.580 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-09-18T08:21:05.581 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:20:47.768392+0000", 2024-09-18T08:21:05.581 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 5.3979999999999998e-06 2024-09-18T08:21:05.581 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:21:05.581 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:21:05.581 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-09-18T08:21:05.581 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:20:47.768406+0000", 2024-09-18T08:21:05.581 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.4107000000000001e-05 2024-09-18T08:21:05.581 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:21:05.581 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:21:05.581 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-09-18T08:21:05.581 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:20:47.768413+0000", 2024-09-18T08:21:05.581 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 7.3130000000000001e-06 2024-09-18T08:21:05.581 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:21:05.581 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:21:05.582 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-18T08:21:05.582 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:20:47.768474+0000", 2024-09-18T08:21:05.582 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 6.0927000000000001e-05 2024-09-18T08:21:05.582 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:21:05.582 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:21:05.582 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-09-18T08:21:05.582 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:20:47.768615+0000", 2024-09-18T08:21:05.582 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00014044900000000001 2024-09-18T08:21:05.582 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:21:05.582 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:21:05.582 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "waiting for subops from 0,1", 2024-09-18T08:21:05.582 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:20:47.768750+0000", 2024-09-18T08:21:05.582 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000135351 2024-09-18T08:21:05.582 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:21:05.582 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:21:05.583 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "op_commit", 2024-09-18T08:21:05.583 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:20:47.770495+0000", 2024-09-18T08:21:05.583 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0017445080000000001 2024-09-18T08:21:05.583 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:21:05.583 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:21:05.583 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:21:05.583 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:20:47.770878+0000", 2024-09-18T08:21:05.583 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00038352900000000002 2024-09-18T08:21:05.583 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:21:05.583 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:21:05.583 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_commit_rec", 2024-09-18T08:21:05.583 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:20:47.969478+0000", 2024-09-18T08:21:05.583 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.198599685 2024-09-18T08:21:05.584 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:21:05.584 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:21:05.584 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "commit_sent", 2024-09-18T08:21:05.584 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:20:47.969530+0000", 2024-09-18T08:21:05.584 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 5.1653000000000002e-05 2024-09-18T08:21:05.584 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-18T08:21:05.584 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-18T08:21:05.584 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-09-18T08:21:05.584 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-18T08:20:47.969562+0000", 2024-09-18T08:21:05.584 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 3.2719000000000003e-05 2024-09-18T08:21:05.584 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-18T08:21:05.585 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-09-18T08:21:05.585 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-18T08:21:05.585 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-18T08:21:05.585 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-09-18T08:21:05.585 INFO:teuthology.orchestra.run.smithi050.stdout:} 2024-09-18T08:21:05.586 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:21:05.587 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:21:05.587 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2' 2024-09-18T08:21:05.688 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1 2024-09-18T08:21:05.725 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:21:05.726 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:21:05.726 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1' 2024-09-18T08:21:05.827 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6 2024-09-18T08:21:05.838 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid 779349ca-7593-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-18T08:21:05.859 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:21:05.860 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:21:05.860 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6' 2024-09-18T08:21:05.905 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:21:05 smithi050 ceph-mon[32371]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-18T08:21:05.905 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:21:05 smithi050 ceph-mon[32371]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-18T08:21:05.905 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:21:05 smithi050 ceph-mon[32371]: osdmap e448: 8 total, 8 up, 7 in 2024-09-18T08:21:05.905 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:21:05 smithi050 ceph-mon[32371]: 6.4 scrub starts 2024-09-18T08:21:05.905 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:21:05 smithi050 ceph-mon[32371]: 6.4 scrub ok 2024-09-18T08:21:05.905 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:21:05 smithi050 ceph-mon[32371]: 6.11 scrub starts 2024-09-18T08:21:05.905 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:21:05 smithi050 ceph-mon[32371]: 6.11 scrub ok 2024-09-18T08:21:05.905 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:21:05 smithi050 ceph-mon[32371]: 6.9 scrub starts 2024-09-18T08:21:05.905 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:21:05 smithi050 ceph-mon[32371]: 6.9 scrub ok 2024-09-18T08:21:05.961 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3 2024-09-18T08:21:05.980 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:21:05 smithi007 ceph-mon[29459]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-18T08:21:05.980 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:21:05 smithi007 ceph-mon[29459]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-18T08:21:05.980 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:21:05 smithi007 ceph-mon[29459]: osdmap e448: 8 total, 8 up, 7 in 2024-09-18T08:21:05.980 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:21:05 smithi007 ceph-mon[29459]: 6.4 scrub starts 2024-09-18T08:21:05.980 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:21:05 smithi007 ceph-mon[29459]: 6.4 scrub ok 2024-09-18T08:21:05.980 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:21:05 smithi007 ceph-mon[29459]: 6.11 scrub starts 2024-09-18T08:21:05.980 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:21:05 smithi007 ceph-mon[29459]: 6.11 scrub ok 2024-09-18T08:21:05.981 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:21:05 smithi007 ceph-mon[29459]: 6.9 scrub starts 2024-09-18T08:21:05.981 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:21:05 smithi007 ceph-mon[29459]: 6.9 scrub ok 2024-09-18T08:21:05.981 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:21:05 smithi007 ceph-mon[39249]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-18T08:21:05.981 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:21:05 smithi007 ceph-mon[39249]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-18T08:21:05.981 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:21:05 smithi007 ceph-mon[39249]: osdmap e448: 8 total, 8 up, 7 in 2024-09-18T08:21:05.981 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:21:05 smithi007 ceph-mon[39249]: 6.4 scrub starts 2024-09-18T08:21:05.981 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:21:05 smithi007 ceph-mon[39249]: 6.4 scrub ok 2024-09-18T08:21:05.981 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:21:05 smithi007 ceph-mon[39249]: 6.11 scrub starts 2024-09-18T08:21:05.981 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:21:05 smithi007 ceph-mon[39249]: 6.11 scrub ok 2024-09-18T08:21:05.981 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:21:05 smithi007 ceph-mon[39249]: 6.9 scrub starts 2024-09-18T08:21:05.981 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:21:05 smithi007 ceph-mon[39249]: 6.9 scrub ok 2024-09-18T08:21:06.005 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:21:06.006 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:21:06.006 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3' 2024-09-18T08:21:06.107 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2 2024-09-18T08:21:06.153 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:21:06.153 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:21:06.154 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2' 2024-09-18T08:21:06.255 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7 2024-09-18T08:21:06.287 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:21:06.288 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/779349ca-7593-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T08:21:06.288 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:21:06.289 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7' 2024-09-18T08:21:06.314 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 6, 5, 7] out_osds: [4] dead_osds: [] live_osds: [0, 2, 3, 4, 5, 6, 7, 1] 2024-09-18T08:21:06.314 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-18T08:21:06.314 INFO:tasks.thrashosds.thrasher:trim_stale_osdmaps 2024-09-18T08:21:06.315 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid 779349ca-7593-11ef-bceb-c7b262605968 -- ceph tell osd.0 'trim stale osdmaps' 2024-09-18T08:21:06.389 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4 2024-09-18T08:21:06.421 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:21:06.422 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:21:06.422 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4' 2024-09-18T08:21:06.523 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0 2024-09-18T08:21:06.564 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:21:06.565 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:21:06.565 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0' 2024-09-18T08:21:06.640 DEBUG:teuthology.orchestra.run.smithi050:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid 779349ca-7593-11ef-bceb-c7b262605968 -- ceph daemon osd.6 dump_ops_in_flight 2024-09-18T08:21:06.656 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:21:06 smithi007 ceph-mon[29459]: 6.15 scrub starts 2024-09-18T08:21:06.656 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:21:06 smithi007 ceph-mon[39249]: 6.15 scrub starts 2024-09-18T08:21:06.656 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:21:06 smithi007 ceph-mon[39249]: 6.15 scrub ok 2024-09-18T08:21:06.656 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:21:06 smithi007 ceph-mon[39249]: pgmap v912: 158 pgs: 158 active+clean; 1.5 GiB data, 9.3 GiB used, 616 GiB / 626 GiB avail; 1.3 KiB/s rd, 0 B/s wr, 1.67k op/s 2024-09-18T08:21:06.657 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:21:06 smithi007 ceph-mon[39249]: 4.3 scrub starts 2024-09-18T08:21:06.657 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:21:06 smithi007 ceph-mon[39249]: 4.3 scrub ok 2024-09-18T08:21:06.666 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7 2024-09-18T08:21:06.717 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:21:06.717 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:21:06.718 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7' 2024-09-18T08:21:06.819 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5 2024-09-18T08:21:06.858 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:21:06.858 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:21:06.859 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5' 2024-09-18T08:21:06.905 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:21:06 smithi050 ceph-mon[32371]: 6.15 scrub starts 2024-09-18T08:21:06.905 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:21:06 smithi050 ceph-mon[32371]: 6.15 scrub ok 2024-09-18T08:21:06.905 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:21:06 smithi050 ceph-mon[32371]: pgmap v912: 158 pgs: 158 active+clean; 1.5 GiB data, 9.3 GiB used, 616 GiB / 626 GiB avail; 1.3 KiB/s rd, 0 B/s wr, 1.67k op/s 2024-09-18T08:21:06.905 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:21:06 smithi050 ceph-mon[32371]: 4.3 scrub starts 2024-09-18T08:21:06.905 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:21:06 smithi050 ceph-mon[32371]: 4.3 scrub ok 2024-09-18T08:21:06.925 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:21:06 smithi007 ceph-mon[29459]: 6.15 scrub ok 2024-09-18T08:21:06.925 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:21:06 smithi007 ceph-mon[29459]: pgmap v912: 158 pgs: 158 active+clean; 1.5 GiB data, 9.3 GiB used, 616 GiB / 626 GiB avail; 1.3 KiB/s rd, 0 B/s wr, 1.67k op/s 2024-09-18T08:21:06.926 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:21:06 smithi007 ceph-mon[29459]: 4.3 scrub starts 2024-09-18T08:21:06.926 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:21:06 smithi007 ceph-mon[29459]: 4.3 scrub ok 2024-09-18T08:21:06.926 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Sep 18 08:21:06 smithi007 ceph-779349ca-7593-11ef-bceb-c7b262605968-mgr-y[29688]: ::ffff:172.21.15.50 - - [18/Sep/2024:08:21:06] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-09-18T08:21:06.959 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2 2024-09-18T08:21:07.002 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:21:07.003 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:21:07.003 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2' 2024-09-18T08:21:07.022 INFO:teuthology.orchestra.run.smithi050.stderr:Inferring config /var/lib/ceph/779349ca-7593-11ef-bceb-c7b262605968/mon.b/config 2024-09-18T08:21:07.104 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6 2024-09-18T08:21:07.115 INFO:teuthology.orchestra.run.smithi007.stdout:osd.0: {} 2024-09-18T08:21:07.115 INFO:teuthology.orchestra.run.smithi007.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-18T08:21:07.132 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:21:07.133 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:21:07.133 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6' 2024-09-18T08:21:07.155 INFO:teuthology.orchestra.run.smithi007.stdout:osd.1: {} 2024-09-18T08:21:07.156 INFO:teuthology.orchestra.run.smithi007.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-18T08:21:07.186 INFO:teuthology.orchestra.run.smithi007.stdout:osd.2: {} 2024-09-18T08:21:07.187 INFO:teuthology.orchestra.run.smithi007.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-18T08:21:07.199 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/779349ca-7593-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T08:21:07.222 INFO:teuthology.orchestra.run.smithi007.stdout:osd.3: {} 2024-09-18T08:21:07.222 INFO:teuthology.orchestra.run.smithi007.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-18T08:21:07.234 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7 2024-09-18T08:21:07.261 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:21:07.330 INFO:teuthology.orchestra.run.smithi007.stdout:osd.4: {} 2024-09-18T08:21:07.331 INFO:teuthology.orchestra.run.smithi007.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-18T08:21:07.331 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:21:07.332 INFO:teuthology.orchestra.run.smithi007.stdout:osd.5: {} 2024-09-18T08:21:07.332 INFO:teuthology.orchestra.run.smithi007.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-18T08:21:07.332 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7' 2024-09-18T08:21:07.336 INFO:teuthology.orchestra.run.smithi007.stdout:osd.6: {} 2024-09-18T08:21:07.336 INFO:teuthology.orchestra.run.smithi007.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-18T08:21:07.366 INFO:teuthology.orchestra.run.smithi007.stdout:osd.7: {} 2024-09-18T08:21:07.367 INFO:teuthology.orchestra.run.smithi007.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-18T08:21:07.400 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid 779349ca-7593-11ef-bceb-c7b262605968 -- ceph osd unset nodeep-scrub 2024-09-18T08:21:07.433 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2 2024-09-18T08:21:07.466 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:21:07.467 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:21:07.467 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2' 2024-09-18T08:21:07.568 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1 2024-09-18T08:21:07.589 INFO:teuthology.orchestra.run.smithi050.stdout:ERROR: (22) Invalid argument 2024-09-18T08:21:07.589 INFO:teuthology.orchestra.run.smithi050.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-18T08:21:07.610 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:21:07.610 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:21:07.610 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1' 2024-09-18T08:21:07.711 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5 2024-09-18T08:21:07.729 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:21:07 smithi007 ceph-mon[29459]: 3.16 scrub starts 2024-09-18T08:21:07.730 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:21:07 smithi007 ceph-mon[29459]: 3.16 scrub ok 2024-09-18T08:21:07.730 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:21:07 smithi007 ceph-mon[29459]: 5.1 scrub starts 2024-09-18T08:21:07.730 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:21:07 smithi007 ceph-mon[29459]: 5.1 scrub ok 2024-09-18T08:21:07.730 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:21:07 smithi007 ceph-mon[29459]: from='client.? 172.21.15.7:0/205258559' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-18T08:21:07.730 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:21:07 smithi007 ceph-mon[39249]: 3.16 scrub starts 2024-09-18T08:21:07.730 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:21:07 smithi007 ceph-mon[39249]: 3.16 scrub ok 2024-09-18T08:21:07.730 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:21:07 smithi007 ceph-mon[39249]: 5.1 scrub starts 2024-09-18T08:21:07.730 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:21:07 smithi007 ceph-mon[39249]: 5.1 scrub ok 2024-09-18T08:21:07.730 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:21:07 smithi007 ceph-mon[39249]: from='client.? 172.21.15.7:0/205258559' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-18T08:21:07.752 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:21:07.753 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:21:07.754 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5' 2024-09-18T08:21:07.765 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:21:07 smithi050 ceph-mon[32371]: 3.16 scrub starts 2024-09-18T08:21:07.765 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:21:07 smithi050 ceph-mon[32371]: 3.16 scrub ok 2024-09-18T08:21:07.765 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:21:07 smithi050 ceph-mon[32371]: 5.1 scrub starts 2024-09-18T08:21:07.765 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:21:07 smithi050 ceph-mon[32371]: 5.1 scrub ok 2024-09-18T08:21:07.765 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:21:07 smithi050 ceph-mon[32371]: from='client.? 172.21.15.7:0/205258559' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-18T08:21:07.854 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6 2024-09-18T08:21:07.889 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:21:07.890 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:21:07.890 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6' 2024-09-18T08:21:07.991 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1 2024-09-18T08:21:08.029 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:21:08.029 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:21:08.030 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1' 2024-09-18T08:21:08.131 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0 2024-09-18T08:21:08.173 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:21:08.173 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:21:08.174 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0' 2024-09-18T08:21:08.275 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3 2024-09-18T08:21:08.313 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:21:08.314 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:21:08.314 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3' 2024-09-18T08:21:08.320 DEBUG:teuthology.orchestra.run.smithi050:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid 779349ca-7593-11ef-bceb-c7b262605968 -- ceph daemon osd.6 dump_blocked_ops 2024-09-18T08:21:08.415 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4 2024-09-18T08:21:08.453 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:21:08.453 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:21:08.454 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4' 2024-09-18T08:21:08.555 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4 2024-09-18T08:21:08.589 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:21:08.590 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:21:08.590 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4' 2024-09-18T08:21:08.691 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6 2024-09-18T08:21:08.715 INFO:teuthology.orchestra.run.smithi050.stderr:Inferring config /var/lib/ceph/779349ca-7593-11ef-bceb-c7b262605968/mon.b/config 2024-09-18T08:21:08.744 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:21:08.745 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:21:08.745 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6' 2024-09-18T08:21:08.846 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1 2024-09-18T08:21:08.885 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:21:08.886 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:21:08.886 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1' 2024-09-18T08:21:08.979 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:21:08 smithi007 ceph-mon[29459]: pgmap v913: 158 pgs: 158 active+clean; 1.5 GiB data, 9.3 GiB used, 616 GiB / 626 GiB avail; 573 B/s rd, 0 B/s wr, 1.21k op/s 2024-09-18T08:21:08.980 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:21:08 smithi007 ceph-mon[29459]: 1.9 scrub starts 2024-09-18T08:21:08.980 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:21:08 smithi007 ceph-mon[29459]: 1.9 scrub ok 2024-09-18T08:21:08.980 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:21:08 smithi007 ceph-mon[29459]: 3.1b scrub starts 2024-09-18T08:21:08.980 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:21:08 smithi007 ceph-mon[29459]: 3.1b scrub ok 2024-09-18T08:21:08.980 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:21:08 smithi007 ceph-mon[39249]: pgmap v913: 158 pgs: 158 active+clean; 1.5 GiB data, 9.3 GiB used, 616 GiB / 626 GiB avail; 573 B/s rd, 0 B/s wr, 1.21k op/s 2024-09-18T08:21:08.980 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:21:08 smithi007 ceph-mon[39249]: 1.9 scrub starts 2024-09-18T08:21:08.980 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:21:08 smithi007 ceph-mon[39249]: 1.9 scrub ok 2024-09-18T08:21:08.980 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:21:08 smithi007 ceph-mon[39249]: 3.1b scrub starts 2024-09-18T08:21:08.980 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:21:08 smithi007 ceph-mon[39249]: 3.1b scrub ok 2024-09-18T08:21:08.987 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4 2024-09-18T08:21:09.018 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:21:09.019 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:21:09.019 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4' 2024-09-18T08:21:09.039 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:21:08 smithi050 ceph-mon[32371]: pgmap v913: 158 pgs: 158 active+clean; 1.5 GiB data, 9.3 GiB used, 616 GiB / 626 GiB avail; 573 B/s rd, 0 B/s wr, 1.21k op/s 2024-09-18T08:21:09.039 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:21:08 smithi050 ceph-mon[32371]: 1.9 scrub starts 2024-09-18T08:21:09.039 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:21:08 smithi050 ceph-mon[32371]: 1.9 scrub ok 2024-09-18T08:21:09.039 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:21:08 smithi050 ceph-mon[32371]: 3.1b scrub starts 2024-09-18T08:21:09.039 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:21:08 smithi050 ceph-mon[32371]: 3.1b scrub ok 2024-09-18T08:21:09.120 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3 2024-09-18T08:21:09.155 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:21:09.156 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:21:09.156 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3' 2024-09-18T08:21:09.257 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6 2024-09-18T08:21:09.292 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:21:09.293 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:21:09.293 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6' 2024-09-18T08:21:09.305 INFO:teuthology.orchestra.run.smithi050.stdout:ERROR: (22) Invalid argument 2024-09-18T08:21:09.305 INFO:teuthology.orchestra.run.smithi050.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-18T08:21:09.393 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7 2024-09-18T08:21:09.427 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:21:09.427 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:21:09.427 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7' 2024-09-18T08:21:09.529 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0 2024-09-18T08:21:09.569 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:21:09.569 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:21:09.570 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0' 2024-09-18T08:21:09.671 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3 2024-09-18T08:21:09.712 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:21:09.712 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:21:09.712 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3' 2024-09-18T08:21:09.813 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5 2024-09-18T08:21:09.847 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:21:09.848 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:21:09.848 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5' 2024-09-18T08:21:09.905 INFO:journalctl@ceph.iscsi.iscsi.a.smithi050.stdout:Sep 18 08:21:09 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-iscsi-iscsi-a[57782]: debug there is no tcmu-runner data available 2024-09-18T08:21:09.905 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:21:09 smithi050 ceph-mon[32371]: 3.e scrub starts 2024-09-18T08:21:09.905 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:21:09 smithi050 ceph-mon[32371]: 3.e scrub ok 2024-09-18T08:21:09.905 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:21:09 smithi050 ceph-mon[32371]: 3.a scrub starts 2024-09-18T08:21:09.905 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:21:09 smithi050 ceph-mon[32371]: 3.a scrub ok 2024-09-18T08:21:09.906 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:21:09 smithi050 ceph-mon[32371]: 4.17 scrub starts 2024-09-18T08:21:09.906 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:21:09 smithi050 ceph-mon[32371]: 4.17 scrub ok 2024-09-18T08:21:09.949 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2 2024-09-18T08:21:09.976 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:21:09 smithi007 ceph-mon[29459]: 3.e scrub starts 2024-09-18T08:21:09.976 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:21:09 smithi007 ceph-mon[29459]: 3.e scrub ok 2024-09-18T08:21:09.976 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:21:09 smithi007 ceph-mon[29459]: 3.a scrub starts 2024-09-18T08:21:09.976 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:21:09 smithi007 ceph-mon[29459]: 3.a scrub ok 2024-09-18T08:21:09.976 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:21:09 smithi007 ceph-mon[29459]: 4.17 scrub starts 2024-09-18T08:21:09.976 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:21:09 smithi007 ceph-mon[29459]: 4.17 scrub ok 2024-09-18T08:21:09.976 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:21:09 smithi007 ceph-mon[39249]: 3.e scrub starts 2024-09-18T08:21:09.976 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:21:09 smithi007 ceph-mon[39249]: 3.e scrub ok 2024-09-18T08:21:09.977 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:21:09 smithi007 ceph-mon[39249]: 3.a scrub starts 2024-09-18T08:21:09.977 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:21:09 smithi007 ceph-mon[39249]: 3.a scrub ok 2024-09-18T08:21:09.977 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:21:09 smithi007 ceph-mon[39249]: 4.17 scrub starts 2024-09-18T08:21:09.977 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:21:09 smithi007 ceph-mon[39249]: 4.17 scrub ok 2024-09-18T08:21:09.990 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:21:09.990 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:21:09.991 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2' 2024-09-18T08:21:10.021 DEBUG:teuthology.orchestra.run.smithi050:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid 779349ca-7593-11ef-bceb-c7b262605968 -- ceph daemon osd.6 dump_historic_ops 2024-09-18T08:21:10.092 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6 2024-09-18T08:21:10.121 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:21:10.122 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:21:10.122 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6' 2024-09-18T08:21:10.223 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5 2024-09-18T08:21:10.245 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:21:10.246 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:21:10.247 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5' 2024-09-18T08:21:10.347 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0 2024-09-18T08:21:10.365 INFO:teuthology.orchestra.run.smithi050.stderr:Inferring config /var/lib/ceph/779349ca-7593-11ef-bceb-c7b262605968/mon.b/config 2024-09-18T08:21:10.384 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:21:10.385 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:21:10.385 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0' 2024-09-18T08:21:10.486 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6 2024-09-18T08:21:10.510 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:21:10.510 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:21:10.511 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6' 2024-09-18T08:21:10.612 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6 2024-09-18T08:21:10.643 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:21:10.644 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:21:10.644 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6' 2024-09-18T08:21:10.745 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3 2024-09-18T08:21:10.782 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:21:10.782 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:21:10.783 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3' 2024-09-18T08:21:10.812 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:21:10 smithi050 ceph-mon[32371]: pgmap v914: 158 pgs: 158 active+clean; 1.2 GiB data, 8.2 GiB used, 618 GiB / 626 GiB avail; 1023 B/s rd, 0 B/s wr, 1.37k op/s 2024-09-18T08:21:10.813 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:21:10 smithi050 ceph-mon[32371]: purged_snaps scrub starts 2024-09-18T08:21:10.813 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:21:10 smithi050 ceph-mon[32371]: purged_snaps scrub ok 2024-09-18T08:21:10.883 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6 2024-09-18T08:21:10.917 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:21:10.918 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:21:10.918 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6' 2024-09-18T08:21:10.960 INFO:teuthology.orchestra.run.smithi050.stdout:ERROR: (22) Invalid argument 2024-09-18T08:21:10.961 INFO:teuthology.orchestra.run.smithi050.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-18T08:21:10.979 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:21:10 smithi007 ceph-mon[29459]: pgmap v914: 158 pgs: 158 active+clean; 1.2 GiB data, 8.2 GiB used, 618 GiB / 626 GiB avail; 1023 B/s rd, 0 B/s wr, 1.37k op/s 2024-09-18T08:21:10.979 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:21:10 smithi007 ceph-mon[29459]: purged_snaps scrub starts 2024-09-18T08:21:10.979 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:21:10 smithi007 ceph-mon[29459]: purged_snaps scrub ok 2024-09-18T08:21:10.980 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:21:10 smithi007 ceph-mon[39249]: pgmap v914: 158 pgs: 158 active+clean; 1.2 GiB data, 8.2 GiB used, 618 GiB / 626 GiB avail; 1023 B/s rd, 0 B/s wr, 1.37k op/s 2024-09-18T08:21:10.980 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:21:10 smithi007 ceph-mon[39249]: purged_snaps scrub starts 2024-09-18T08:21:10.980 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:21:10 smithi007 ceph-mon[39249]: purged_snaps scrub ok 2024-09-18T08:21:11.019 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7 2024-09-18T08:21:11.051 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:21:11.052 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:21:11.052 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7' 2024-09-18T08:21:11.153 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0 2024-09-18T08:21:11.195 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:21:11.196 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:21:11.196 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0' 2024-09-18T08:21:11.297 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2 2024-09-18T08:21:11.344 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:21:11.345 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:21:11.345 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2' 2024-09-18T08:21:11.446 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2 2024-09-18T08:21:11.486 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:21:11.487 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:21:11.487 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2' 2024-09-18T08:21:11.527 INFO:teuthology.orchestra.run.smithi007.stderr:no valid command found; 10 closest matches: 2024-09-18T08:21:11.527 INFO:teuthology.orchestra.run.smithi007.stderr:0 2024-09-18T08:21:11.528 INFO:teuthology.orchestra.run.smithi007.stderr:1 2024-09-18T08:21:11.528 INFO:teuthology.orchestra.run.smithi007.stderr:2 2024-09-18T08:21:11.528 INFO:teuthology.orchestra.run.smithi007.stderr:abort 2024-09-18T08:21:11.528 INFO:teuthology.orchestra.run.smithi007.stderr:assert 2024-09-18T08:21:11.528 INFO:teuthology.orchestra.run.smithi007.stderr:bench [] [] [] [] 2024-09-18T08:21:11.528 INFO:teuthology.orchestra.run.smithi007.stderr:bluefs debug_inject_read_zeros 2024-09-18T08:21:11.528 INFO:teuthology.orchestra.run.smithi007.stderr:bluefs files list 2024-09-18T08:21:11.528 INFO:teuthology.orchestra.run.smithi007.stderr:bluefs stats 2024-09-18T08:21:11.528 INFO:teuthology.orchestra.run.smithi007.stderr:bluestore allocator dump block 2024-09-18T08:21:11.528 INFO:teuthology.orchestra.run.smithi007.stderr:Error EINVAL: invalid command 2024-09-18T08:21:11.589 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6 2024-09-18T08:21:11.625 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:21:11.626 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:21:11.626 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6' 2024-09-18T08:21:11.680 DEBUG:teuthology.orchestra.run.smithi050:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid 779349ca-7593-11ef-bceb-c7b262605968 -- ceph daemon osd.7 dump_ops_in_flight 2024-09-18T08:21:11.727 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6 2024-09-18T08:21:11.729 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:21:11 smithi007 ceph-mon[29459]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-18T08:21:11.729 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:21:11 smithi007 ceph-mon[29459]: 3.1d scrub starts 2024-09-18T08:21:11.729 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:21:11 smithi007 ceph-mon[29459]: 3.1d scrub ok 2024-09-18T08:21:11.729 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:21:11 smithi007 ceph-mon[29459]: osdmap e449: 8 total, 8 up, 7 in 2024-09-18T08:21:11.730 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:21:11 smithi007 ceph-mon[39249]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-18T08:21:11.730 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:21:11 smithi007 ceph-mon[39249]: 3.1d scrub starts 2024-09-18T08:21:11.730 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:21:11 smithi007 ceph-mon[39249]: 3.1d scrub ok 2024-09-18T08:21:11.730 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:21:11 smithi007 ceph-mon[39249]: osdmap e449: 8 total, 8 up, 7 in 2024-09-18T08:21:11.774 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:21:11.774 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:21:11.774 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6' 2024-09-18T08:21:11.875 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2 2024-09-18T08:21:11.905 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:21:11 smithi050 ceph-mon[32371]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-18T08:21:11.905 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:21:11 smithi050 ceph-mon[32371]: 3.1d scrub starts 2024-09-18T08:21:11.905 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:21:11 smithi050 ceph-mon[32371]: 3.1d scrub ok 2024-09-18T08:21:11.905 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:21:11 smithi050 ceph-mon[32371]: osdmap e449: 8 total, 8 up, 7 in 2024-09-18T08:21:11.917 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:21:11.918 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:21:11.918 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2' 2024-09-18T08:21:12.018 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4 2024-09-18T08:21:12.060 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:21:12.060 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:21:12.061 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4' 2024-09-18T08:21:12.096 INFO:teuthology.orchestra.run.smithi050.stderr:Inferring config /var/lib/ceph/779349ca-7593-11ef-bceb-c7b262605968/mon.b/config 2024-09-18T08:21:12.162 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6 2024-09-18T08:21:12.202 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:21:12.202 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:21:12.202 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6' 2024-09-18T08:21:12.303 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6 2024-09-18T08:21:12.336 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:21:12.337 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:21:12.337 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6' 2024-09-18T08:21:12.439 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6 2024-09-18T08:21:12.462 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:21:12.463 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:21:12.463 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6' 2024-09-18T08:21:12.564 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4 2024-09-18T08:21:12.637 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:21:12.638 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:21:12.638 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4' 2024-09-18T08:21:12.739 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1 2024-09-18T08:21:12.787 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:21:12.788 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:21:12.788 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1' 2024-09-18T08:21:12.889 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5 2024-09-18T08:21:12.905 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:21:12 smithi050 ceph-mon[32371]: pgmap v916: 158 pgs: 158 active+clean; 1.0 GiB data, 8.0 GiB used, 618 GiB / 626 GiB avail; 639 B/s rd, 0 B/s wr, 1.43k op/s 2024-09-18T08:21:12.924 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:21:12.925 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:21:12.925 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5' 2024-09-18T08:21:12.979 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:21:12 smithi007 ceph-mon[29459]: pgmap v916: 158 pgs: 158 active+clean; 1.0 GiB data, 8.0 GiB used, 618 GiB / 626 GiB avail; 639 B/s rd, 0 B/s wr, 1.43k op/s 2024-09-18T08:21:12.980 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:21:12 smithi007 ceph-mon[39249]: pgmap v916: 158 pgs: 158 active+clean; 1.0 GiB data, 8.0 GiB used, 618 GiB / 626 GiB avail; 639 B/s rd, 0 B/s wr, 1.43k op/s 2024-09-18T08:21:13.027 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1 2024-09-18T08:21:13.063 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:21:13.064 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:21:13.064 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1' 2024-09-18T08:21:13.139 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/779349ca-7593-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T08:21:13.165 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5 2024-09-18T08:21:13.167 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-09-18T08:21:13.167 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-09-18T08:21:13.167 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid 779349ca-7593-11ef-bceb-c7b262605968 -- ceph tell osd.1 'trim stale osdmaps' 2024-09-18T08:21:13.194 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:21:13.195 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:21:13.196 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5' 2024-09-18T08:21:13.297 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2 2024-09-18T08:21:13.356 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:21:13.357 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:21:13.357 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2' 2024-09-18T08:21:13.458 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0 2024-09-18T08:21:13.477 INFO:teuthology.orchestra.run.smithi050.stdout:ERROR: (22) Invalid argument 2024-09-18T08:21:13.478 INFO:teuthology.orchestra.run.smithi050.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-18T08:21:13.503 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:21:13.504 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:21:13.521 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0' 2024-09-18T08:21:13.623 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2 2024-09-18T08:21:13.657 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:21:13.658 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:21:13.659 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2' 2024-09-18T08:21:13.760 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1 2024-09-18T08:21:13.803 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:21:13.803 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:21:13.804 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1' 2024-09-18T08:21:13.887 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/779349ca-7593-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T08:21:13.905 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5 2024-09-18T08:21:13.938 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:21:13.939 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:21:13.939 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5' 2024-09-18T08:21:14.040 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0 2024-09-18T08:21:14.086 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:21:14.087 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:21:14.087 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0' 2024-09-18T08:21:14.188 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7 2024-09-18T08:21:14.224 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:21:14.225 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:21:14.225 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7' 2024-09-18T08:21:14.256 DEBUG:teuthology.orchestra.run.smithi050:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid 779349ca-7593-11ef-bceb-c7b262605968 -- ceph daemon osd.7 dump_blocked_ops 2024-09-18T08:21:14.328 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3 2024-09-18T08:21:14.371 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:21:14.372 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:21:14.372 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3' 2024-09-18T08:21:14.474 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5 2024-09-18T08:21:14.513 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:21:14.513 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:21:14.514 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5' 2024-09-18T08:21:14.538 INFO:teuthology.orchestra.run.smithi007.stderr:no valid command found; 10 closest matches: 2024-09-18T08:21:14.538 INFO:teuthology.orchestra.run.smithi007.stderr:0 2024-09-18T08:21:14.538 INFO:teuthology.orchestra.run.smithi007.stderr:1 2024-09-18T08:21:14.538 INFO:teuthology.orchestra.run.smithi007.stderr:2 2024-09-18T08:21:14.539 INFO:teuthology.orchestra.run.smithi007.stderr:abort 2024-09-18T08:21:14.539 INFO:teuthology.orchestra.run.smithi007.stderr:assert 2024-09-18T08:21:14.539 INFO:teuthology.orchestra.run.smithi007.stderr:bench [] [] [] [] 2024-09-18T08:21:14.539 INFO:teuthology.orchestra.run.smithi007.stderr:bluefs debug_inject_read_zeros 2024-09-18T08:21:14.539 INFO:teuthology.orchestra.run.smithi007.stderr:bluefs files list 2024-09-18T08:21:14.539 INFO:teuthology.orchestra.run.smithi007.stderr:bluefs stats 2024-09-18T08:21:14.539 INFO:teuthology.orchestra.run.smithi007.stderr:bluestore allocator dump block 2024-09-18T08:21:14.539 INFO:teuthology.orchestra.run.smithi007.stderr:Error EINVAL: invalid command 2024-09-18T08:21:14.605 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid 779349ca-7593-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-18T08:21:14.614 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2 2024-09-18T08:21:14.684 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:21:14.686 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:21:14.686 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2' 2024-09-18T08:21:14.733 INFO:teuthology.orchestra.run.smithi050.stderr:Inferring config /var/lib/ceph/779349ca-7593-11ef-bceb-c7b262605968/mon.b/config 2024-09-18T08:21:14.766 INFO:teuthology.orchestra.run.smithi007.stderr:nodeep-scrub is unset 2024-09-18T08:21:14.787 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4 2024-09-18T08:21:14.826 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:21:14.827 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:21:14.827 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4' 2024-09-18T08:21:14.928 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1 2024-09-18T08:21:14.955 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:21:14.955 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:21:14.955 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1' 2024-09-18T08:21:14.979 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:21:14 smithi007 ceph-mon[29459]: pgmap v917: 158 pgs: 158 active+clean; 891 MiB data, 7.3 GiB used, 619 GiB / 626 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 1.40k op/s 2024-09-18T08:21:14.980 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:21:14 smithi007 ceph-mon[29459]: 1.6 scrub starts 2024-09-18T08:21:14.980 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:21:14 smithi007 ceph-mon[29459]: 1.6 scrub ok 2024-09-18T08:21:14.980 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:21:14 smithi007 ceph-mon[29459]: from='client.? 172.21.15.7:0/522306039' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-18T08:21:14.980 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:21:14 smithi007 ceph-mon[29459]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-18T08:21:14.980 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:21:14 smithi007 ceph-mon[39249]: pgmap v917: 158 pgs: 158 active+clean; 891 MiB data, 7.3 GiB used, 619 GiB / 626 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 1.40k op/s 2024-09-18T08:21:14.980 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:21:14 smithi007 ceph-mon[39249]: 1.6 scrub starts 2024-09-18T08:21:14.980 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:21:14 smithi007 ceph-mon[39249]: 1.6 scrub ok 2024-09-18T08:21:14.980 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:21:14 smithi007 ceph-mon[39249]: from='client.? 172.21.15.7:0/522306039' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-18T08:21:14.980 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:21:14 smithi007 ceph-mon[39249]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-18T08:21:15.056 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0 2024-09-18T08:21:15.060 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:21:14 smithi050 ceph-mon[32371]: pgmap v917: 158 pgs: 158 active+clean; 891 MiB data, 7.3 GiB used, 619 GiB / 626 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 1.40k op/s 2024-09-18T08:21:15.061 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:21:14 smithi050 ceph-mon[32371]: 1.6 scrub starts 2024-09-18T08:21:15.061 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:21:14 smithi050 ceph-mon[32371]: 1.6 scrub ok 2024-09-18T08:21:15.061 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:21:14 smithi050 ceph-mon[32371]: from='client.? 172.21.15.7:0/522306039' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-18T08:21:15.061 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:21:14 smithi050 ceph-mon[32371]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-18T08:21:15.094 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:21:15.095 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:21:15.095 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0' 2024-09-18T08:21:15.196 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3 2024-09-18T08:21:15.242 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:21:15.243 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:21:15.244 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3' 2024-09-18T08:21:15.345 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7 2024-09-18T08:21:15.378 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:21:15.379 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:21:15.380 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7' 2024-09-18T08:21:15.480 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5 2024-09-18T08:21:15.511 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:21:15.512 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:21:15.512 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5' 2024-09-18T08:21:15.613 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1 2024-09-18T08:21:15.635 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:21:15.636 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:21:15.636 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1' 2024-09-18T08:21:15.740 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3 2024-09-18T08:21:15.778 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:21:15.779 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:21:15.779 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3' 2024-09-18T08:21:15.780 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:21:15 smithi050 ceph-mon[32371]: 6.19 scrub starts 2024-09-18T08:21:15.780 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:21:15 smithi050 ceph-mon[32371]: 6.19 scrub ok 2024-09-18T08:21:15.780 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:21:15 smithi050 ceph-mon[32371]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-18T08:21:15.780 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:21:15 smithi050 ceph-mon[32371]: Cluster is now healthy 2024-09-18T08:21:15.780 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:21:15 smithi050 ceph-mon[32371]: 3.1e scrub starts 2024-09-18T08:21:15.780 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:21:15 smithi050 ceph-mon[32371]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-18T08:21:15.780 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:21:15 smithi050 ceph-mon[32371]: 3.1e scrub ok 2024-09-18T08:21:15.780 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:21:15 smithi050 ceph-mon[32371]: osdmap e450: 8 total, 8 up, 7 in 2024-09-18T08:21:15.780 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:21:15 smithi050 ceph-mon[32371]: 11.0 scrub starts 2024-09-18T08:21:15.880 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6 2024-09-18T08:21:15.911 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:21:15.912 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:21:15.912 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6' 2024-09-18T08:21:15.929 INFO:teuthology.orchestra.run.smithi050.stdout:ERROR: (22) Invalid argument 2024-09-18T08:21:15.930 INFO:teuthology.orchestra.run.smithi050.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-18T08:21:15.980 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:21:15 smithi007 ceph-mon[29459]: 6.19 scrub starts 2024-09-18T08:21:15.980 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:21:15 smithi007 ceph-mon[29459]: 6.19 scrub ok 2024-09-18T08:21:15.980 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:21:15 smithi007 ceph-mon[29459]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-18T08:21:15.980 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:21:15 smithi007 ceph-mon[29459]: Cluster is now healthy 2024-09-18T08:21:15.980 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:21:15 smithi007 ceph-mon[29459]: 3.1e scrub starts 2024-09-18T08:21:15.980 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:21:15 smithi007 ceph-mon[29459]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-18T08:21:15.980 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:21:15 smithi007 ceph-mon[29459]: 3.1e scrub ok 2024-09-18T08:21:15.980 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:21:15 smithi007 ceph-mon[29459]: osdmap e450: 8 total, 8 up, 7 in 2024-09-18T08:21:15.980 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:21:15 smithi007 ceph-mon[29459]: 11.0 scrub starts 2024-09-18T08:21:15.981 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:21:15 smithi007 ceph-mon[39249]: 6.19 scrub starts 2024-09-18T08:21:15.981 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:21:15 smithi007 ceph-mon[39249]: 6.19 scrub ok 2024-09-18T08:21:15.981 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:21:15 smithi007 ceph-mon[39249]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-18T08:21:15.981 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:21:15 smithi007 ceph-mon[39249]: Cluster is now healthy 2024-09-18T08:21:15.981 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:21:15 smithi007 ceph-mon[39249]: 3.1e scrub starts 2024-09-18T08:21:15.981 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:21:15 smithi007 ceph-mon[39249]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-18T08:21:15.981 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:21:15 smithi007 ceph-mon[39249]: 3.1e scrub ok 2024-09-18T08:21:15.981 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:21:15 smithi007 ceph-mon[39249]: osdmap e450: 8 total, 8 up, 7 in 2024-09-18T08:21:15.981 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:21:15 smithi007 ceph-mon[39249]: 11.0 scrub starts 2024-09-18T08:21:16.014 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1 2024-09-18T08:21:16.054 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:21:16.055 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:21:16.055 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1' 2024-09-18T08:21:16.156 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6 2024-09-18T08:21:16.191 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:21:16.191 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:21:16.192 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6' 2024-09-18T08:21:16.292 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7 2024-09-18T08:21:16.331 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:21:16.332 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:21:16.332 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7' 2024-09-18T08:21:16.433 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1 2024-09-18T08:21:16.472 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:21:16.473 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:21:16.473 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1' 2024-09-18T08:21:16.574 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4 2024-09-18T08:21:16.606 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:21:16.606 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:21:16.607 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4' 2024-09-18T08:21:16.642 DEBUG:teuthology.orchestra.run.smithi050:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid 779349ca-7593-11ef-bceb-c7b262605968 -- ceph daemon osd.7 dump_historic_ops 2024-09-18T08:21:16.708 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5 2024-09-18T08:21:16.748 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:21:16.749 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:21:16.750 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5' 2024-09-18T08:21:16.850 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0 2024-09-18T08:21:16.889 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:21:16.890 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:21:16.890 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0' 2024-09-18T08:21:16.979 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:21:16 smithi007 ceph-mon[29459]: 11.0 scrub ok 2024-09-18T08:21:16.980 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:21:16 smithi007 ceph-mon[29459]: pgmap v919: 158 pgs: 158 active+clean; 634 MiB data, 6.7 GiB used, 619 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.75k op/s 2024-09-18T08:21:16.980 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:21:16 smithi007 ceph-mon[29459]: 3.12 scrub starts 2024-09-18T08:21:16.980 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:21:16 smithi007 ceph-mon[29459]: 3.12 scrub ok 2024-09-18T08:21:16.980 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Sep 18 08:21:16 smithi007 ceph-779349ca-7593-11ef-bceb-c7b262605968-mgr-y[29688]: ::ffff:172.21.15.50 - - [18/Sep/2024:08:21:16] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-09-18T08:21:16.980 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:21:16 smithi007 ceph-mon[39249]: 11.0 scrub ok 2024-09-18T08:21:16.980 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:21:16 smithi007 ceph-mon[39249]: pgmap v919: 158 pgs: 158 active+clean; 634 MiB data, 6.7 GiB used, 619 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.75k op/s 2024-09-18T08:21:16.981 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:21:16 smithi007 ceph-mon[39249]: 3.12 scrub starts 2024-09-18T08:21:16.981 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:21:16 smithi007 ceph-mon[39249]: 3.12 scrub ok 2024-09-18T08:21:16.991 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4 2024-09-18T08:21:17.026 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:21:17.027 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:21:17.027 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4' 2024-09-18T08:21:17.057 INFO:teuthology.orchestra.run.smithi050.stderr:Inferring config /var/lib/ceph/779349ca-7593-11ef-bceb-c7b262605968/mon.b/config 2024-09-18T08:21:17.129 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1 2024-09-18T08:21:17.155 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:21:16 smithi050 ceph-mon[32371]: 11.0 scrub ok 2024-09-18T08:21:17.155 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:21:16 smithi050 ceph-mon[32371]: pgmap v919: 158 pgs: 158 active+clean; 634 MiB data, 6.7 GiB used, 619 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.75k op/s 2024-09-18T08:21:17.155 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:21:16 smithi050 ceph-mon[32371]: 3.12 scrub starts 2024-09-18T08:21:17.155 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:21:16 smithi050 ceph-mon[32371]: 3.12 scrub ok 2024-09-18T08:21:17.164 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:21:17.165 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:21:17.165 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1' 2024-09-18T08:21:17.266 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0 2024-09-18T08:21:17.305 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:21:17.306 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:21:17.306 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0' 2024-09-18T08:21:17.407 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2 2024-09-18T08:21:17.440 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:21:17.441 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:21:17.441 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2' 2024-09-18T08:21:17.542 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5 2024-09-18T08:21:17.580 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-09-18T08:21:17.580 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-09-18T08:21:17.581 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid 779349ca-7593-11ef-bceb-c7b262605968 -- ceph tell osd.2 'trim stale osdmaps' 2024-09-18T08:21:17.610 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:21:17.611 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:21:17.612 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5' 2024-09-18T08:21:17.682 INFO:teuthology.orchestra.run.smithi050.stdout:ERROR: (22) Invalid argument 2024-09-18T08:21:17.682 INFO:teuthology.orchestra.run.smithi050.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-18T08:21:17.713 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6 2024-09-18T08:21:17.749 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:21:17.750 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:21:17.750 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6' 2024-09-18T08:21:17.851 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7 2024-09-18T08:21:17.880 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:21:17.880 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:21:17.881 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7' 2024-09-18T08:21:17.904 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:21:17 smithi050 ceph-mon[32371]: 4.1b scrub starts 2024-09-18T08:21:17.905 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:21:17 smithi050 ceph-mon[32371]: 4.1b scrub ok 2024-09-18T08:21:17.905 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:21:17 smithi050 ceph-mon[32371]: 11.6 scrub starts 2024-09-18T08:21:17.905 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:21:17 smithi050 ceph-mon[32371]: 11.6 scrub ok 2024-09-18T08:21:17.981 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1 2024-09-18T08:21:18.030 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:21:18.030 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:21:18.031 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1' 2024-09-18T08:21:18.107 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:21:17 smithi007 ceph-mon[29459]: 4.1b scrub starts 2024-09-18T08:21:18.107 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:21:17 smithi007 ceph-mon[29459]: 4.1b scrub ok 2024-09-18T08:21:18.107 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:21:17 smithi007 ceph-mon[29459]: 11.6 scrub starts 2024-09-18T08:21:18.108 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:21:17 smithi007 ceph-mon[29459]: 11.6 scrub ok 2024-09-18T08:21:18.108 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:21:17 smithi007 ceph-mon[39249]: 4.1b scrub starts 2024-09-18T08:21:18.108 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:21:17 smithi007 ceph-mon[39249]: 4.1b scrub ok 2024-09-18T08:21:18.108 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:21:17 smithi007 ceph-mon[39249]: 11.6 scrub starts 2024-09-18T08:21:18.108 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:21:17 smithi007 ceph-mon[39249]: 11.6 scrub ok 2024-09-18T08:21:18.132 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2 2024-09-18T08:21:18.177 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:21:18.177 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:21:18.178 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2' 2024-09-18T08:21:18.231 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/779349ca-7593-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T08:21:18.279 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3 2024-09-18T08:21:18.284 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/779349ca-7593-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T08:21:18.327 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:21:18.328 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:21:18.328 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3' 2024-09-18T08:21:18.424 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid 779349ca-7593-11ef-bceb-c7b262605968 -- ceph daemon osd.1 dump_ops_in_flight 2024-09-18T08:21:18.429 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7 2024-09-18T08:21:18.494 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:21:18.495 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:21:18.495 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7' 2024-09-18T08:21:18.596 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2 2024-09-18T08:21:18.646 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:21:18.646 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:21:18.647 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2' 2024-09-18T08:21:18.747 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5 2024-09-18T08:21:18.774 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:21:18.775 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:21:18.775 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5' 2024-09-18T08:21:18.877 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3 2024-09-18T08:21:18.925 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:21:18.925 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:21:18.926 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3' 2024-09-18T08:21:19.028 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4 2024-09-18T08:21:19.029 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:21:18 smithi007 ceph-mon[29459]: pgmap v920: 158 pgs: 158 active+clean; 634 MiB data, 6.7 GiB used, 619 GiB / 626 GiB avail; 639 B/s rd, 0 B/s wr, 1.18k op/s 2024-09-18T08:21:19.029 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:21:18 smithi007 ceph-mon[29459]: from='mgr.24470 172.21.15.7:0/2594390261' entity='mgr.y' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1d", "id": [1, 0, 3, 5]} : dispatch 2024-09-18T08:21:19.029 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:21:18 smithi007 ceph-mon[29459]: from='mgr.24470 172.21.15.7:0/2594390261' entity='mgr.y' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "5.0", "id": [1, 6]} : dispatch 2024-09-18T08:21:19.030 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:21:18 smithi007 ceph-mon[29459]: from='mgr.24470 172.21.15.7:0/2594390261' entity='mgr.y' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "5.12", "id": [1, 6]} : dispatch 2024-09-18T08:21:19.030 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:21:18 smithi007 ceph-mon[39249]: pgmap v920: 158 pgs: 158 active+clean; 634 MiB data, 6.7 GiB used, 619 GiB / 626 GiB avail; 639 B/s rd, 0 B/s wr, 1.18k op/s 2024-09-18T08:21:19.030 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:21:18 smithi007 ceph-mon[39249]: from='mgr.24470 172.21.15.7:0/2594390261' entity='mgr.y' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1d", "id": [1, 0, 3, 5]} : dispatch 2024-09-18T08:21:19.030 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:21:18 smithi007 ceph-mon[39249]: from='mgr.24470 172.21.15.7:0/2594390261' entity='mgr.y' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "5.0", "id": [1, 6]} : dispatch 2024-09-18T08:21:19.030 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:21:18 smithi007 ceph-mon[39249]: from='mgr.24470 172.21.15.7:0/2594390261' entity='mgr.y' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "5.12", "id": [1, 6]} : dispatch 2024-09-18T08:21:19.059 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:21:19.060 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:21:19.060 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4' 2024-09-18T08:21:19.155 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:21:18 smithi050 ceph-mon[32371]: pgmap v920: 158 pgs: 158 active+clean; 634 MiB data, 6.7 GiB used, 619 GiB / 626 GiB avail; 639 B/s rd, 0 B/s wr, 1.18k op/s 2024-09-18T08:21:19.155 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:21:18 smithi050 ceph-mon[32371]: from='mgr.24470 172.21.15.7:0/2594390261' entity='mgr.y' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1d", "id": [1, 0, 3, 5]} : dispatch 2024-09-18T08:21:19.155 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:21:18 smithi050 ceph-mon[32371]: from='mgr.24470 172.21.15.7:0/2594390261' entity='mgr.y' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "5.0", "id": [1, 6]} : dispatch 2024-09-18T08:21:19.155 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:21:18 smithi050 ceph-mon[32371]: from='mgr.24470 172.21.15.7:0/2594390261' entity='mgr.y' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "5.12", "id": [1, 6]} : dispatch 2024-09-18T08:21:19.162 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5 2024-09-18T08:21:19.193 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:21:19.194 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:21:19.194 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5' 2024-09-18T08:21:19.295 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1 2024-09-18T08:21:19.341 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:21:19.342 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:21:19.342 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1' 2024-09-18T08:21:19.443 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6 2024-09-18T08:21:19.474 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:21:19.474 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:21:19.475 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6' 2024-09-18T08:21:19.498 INFO:teuthology.orchestra.run.smithi007.stdout:osd.0: {} 2024-09-18T08:21:19.499 INFO:teuthology.orchestra.run.smithi007.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-18T08:21:19.543 INFO:teuthology.orchestra.run.smithi007.stdout:osd.1: {} 2024-09-18T08:21:19.543 INFO:teuthology.orchestra.run.smithi007.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-18T08:21:19.547 INFO:teuthology.orchestra.run.smithi007.stderr:no valid command found; 10 closest matches: 2024-09-18T08:21:19.547 INFO:teuthology.orchestra.run.smithi007.stderr:0 2024-09-18T08:21:19.547 INFO:teuthology.orchestra.run.smithi007.stderr:1 2024-09-18T08:21:19.547 INFO:teuthology.orchestra.run.smithi007.stderr:2 2024-09-18T08:21:19.548 INFO:teuthology.orchestra.run.smithi007.stderr:abort 2024-09-18T08:21:19.548 INFO:teuthology.orchestra.run.smithi007.stderr:assert 2024-09-18T08:21:19.548 INFO:teuthology.orchestra.run.smithi007.stderr:bench [] [] [] [] 2024-09-18T08:21:19.548 INFO:teuthology.orchestra.run.smithi007.stderr:bluefs debug_inject_read_zeros 2024-09-18T08:21:19.548 INFO:teuthology.orchestra.run.smithi007.stderr:bluefs files list 2024-09-18T08:21:19.548 INFO:teuthology.orchestra.run.smithi007.stderr:bluefs stats 2024-09-18T08:21:19.548 INFO:teuthology.orchestra.run.smithi007.stderr:bluestore allocator dump block 2024-09-18T08:21:19.548 INFO:teuthology.orchestra.run.smithi007.stderr:Error EINVAL: invalid command 2024-09-18T08:21:19.575 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4 2024-09-18T08:21:19.579 INFO:teuthology.orchestra.run.smithi007.stdout:osd.2: {} 2024-09-18T08:21:19.579 INFO:teuthology.orchestra.run.smithi007.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-18T08:21:19.605 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:21:19.606 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:21:19.606 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4' 2024-09-18T08:21:19.624 INFO:teuthology.orchestra.run.smithi007.stdout:osd.3: {} 2024-09-18T08:21:19.625 INFO:teuthology.orchestra.run.smithi007.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-18T08:21:19.658 INFO:teuthology.orchestra.run.smithi007.stdout:osd.4: {} 2024-09-18T08:21:19.658 INFO:teuthology.orchestra.run.smithi007.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-18T08:21:19.695 INFO:teuthology.orchestra.run.smithi007.stdout:osd.5: {} 2024-09-18T08:21:19.696 INFO:teuthology.orchestra.run.smithi007.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-18T08:21:19.707 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1 2024-09-18T08:21:19.732 INFO:teuthology.orchestra.run.smithi007.stdout:osd.6: {} 2024-09-18T08:21:19.732 INFO:teuthology.orchestra.run.smithi007.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-18T08:21:19.744 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:21:19.745 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:21:19.745 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1' 2024-09-18T08:21:19.765 INFO:teuthology.orchestra.run.smithi007.stdout:osd.7: {} 2024-09-18T08:21:19.765 INFO:teuthology.orchestra.run.smithi007.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-18T08:21:19.848 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4 2024-09-18T08:21:19.880 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:21:19.880 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:21:19.881 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4' 2024-09-18T08:21:19.888 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:21:19 smithi007 ceph-mon[29459]: from='mgr.24470 172.21.15.7:0/2594390261' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1d", "id": [1, 0, 3, 5]}]': finished 2024-09-18T08:21:19.888 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:21:19 smithi007 ceph-mon[29459]: from='mgr.24470 172.21.15.7:0/2594390261' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "5.0", "id": [1, 6]}]': finished 2024-09-18T08:21:19.888 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:21:19 smithi007 ceph-mon[29459]: from='mgr.24470 172.21.15.7:0/2594390261' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "5.12", "id": [1, 6]}]': finished 2024-09-18T08:21:19.889 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:21:19 smithi007 ceph-mon[29459]: osdmap e451: 8 total, 8 up, 7 in 2024-09-18T08:21:19.889 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:21:19 smithi007 ceph-mon[29459]: 5.14 scrub starts 2024-09-18T08:21:19.889 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:21:19 smithi007 ceph-mon[29459]: 5.14 scrub ok 2024-09-18T08:21:19.889 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:21:19 smithi007 ceph-mon[29459]: from='mgr.24470 172.21.15.7:0/2594390261' entity='mgr.y' 2024-09-18T08:21:19.889 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:21:19 smithi007 ceph-mon[29459]: from='mgr.24470 172.21.15.7:0/2594390261' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-18T08:21:19.889 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:21:19 smithi007 ceph-mon[29459]: from='client.? 172.21.15.7:0/1242128128' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-18T08:21:19.889 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:21:19 smithi007 ceph-mon[29459]: pgmap v922: 158 pgs: 158 active+clean; 370 MiB data, 5.7 GiB used, 620 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.38k op/s 2024-09-18T08:21:19.889 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:21:19 smithi007 ceph-mon[39249]: from='mgr.24470 172.21.15.7:0/2594390261' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1d", "id": [1, 0, 3, 5]}]': finished 2024-09-18T08:21:19.889 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:21:19 smithi007 ceph-mon[39249]: from='mgr.24470 172.21.15.7:0/2594390261' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "5.0", "id": [1, 6]}]': finished 2024-09-18T08:21:19.889 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:21:19 smithi007 ceph-mon[39249]: from='mgr.24470 172.21.15.7:0/2594390261' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "5.12", "id": [1, 6]}]': finished 2024-09-18T08:21:19.889 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:21:19 smithi007 ceph-mon[39249]: osdmap e451: 8 total, 8 up, 7 in 2024-09-18T08:21:19.890 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:21:19 smithi007 ceph-mon[39249]: 5.14 scrub starts 2024-09-18T08:21:19.890 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:21:19 smithi007 ceph-mon[39249]: 5.14 scrub ok 2024-09-18T08:21:19.890 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:21:19 smithi007 ceph-mon[39249]: from='mgr.24470 172.21.15.7:0/2594390261' entity='mgr.y' 2024-09-18T08:21:19.890 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:21:19 smithi007 ceph-mon[39249]: from='mgr.24470 172.21.15.7:0/2594390261' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-18T08:21:19.890 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:21:19 smithi007 ceph-mon[39249]: from='client.? 172.21.15.7:0/1242128128' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-18T08:21:19.890 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:21:19 smithi007 ceph-mon[39249]: pgmap v922: 158 pgs: 158 active+clean; 370 MiB data, 5.7 GiB used, 620 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.38k op/s 2024-09-18T08:21:19.981 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2 2024-09-18T08:21:20.019 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:21:20.020 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:21:20.020 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2' 2024-09-18T08:21:20.121 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3 2024-09-18T08:21:20.154 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:21:20.155 INFO:journalctl@ceph.iscsi.iscsi.a.smithi050.stdout:Sep 18 08:21:19 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-iscsi-iscsi-a[57782]: debug there is no tcmu-runner data available 2024-09-18T08:21:20.155 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:21:19 smithi050 ceph-mon[32371]: from='mgr.24470 172.21.15.7:0/2594390261' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1d", "id": [1, 0, 3, 5]}]': finished 2024-09-18T08:21:20.155 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:21:19 smithi050 ceph-mon[32371]: from='mgr.24470 172.21.15.7:0/2594390261' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "5.0", "id": [1, 6]}]': finished 2024-09-18T08:21:20.155 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:21:19 smithi050 ceph-mon[32371]: from='mgr.24470 172.21.15.7:0/2594390261' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "5.12", "id": [1, 6]}]': finished 2024-09-18T08:21:20.155 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:21:19 smithi050 ceph-mon[32371]: osdmap e451: 8 total, 8 up, 7 in 2024-09-18T08:21:20.155 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:21:19 smithi050 ceph-mon[32371]: 5.14 scrub starts 2024-09-18T08:21:20.156 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:21:19 smithi050 ceph-mon[32371]: 5.14 scrub ok 2024-09-18T08:21:20.156 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:21:19 smithi050 ceph-mon[32371]: from='mgr.24470 172.21.15.7:0/2594390261' entity='mgr.y' 2024-09-18T08:21:20.156 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:21:19 smithi050 ceph-mon[32371]: from='mgr.24470 172.21.15.7:0/2594390261' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-18T08:21:20.156 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:21:19 smithi050 ceph-mon[32371]: from='client.? 172.21.15.7:0/1242128128' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-18T08:21:20.156 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:21:19 smithi050 ceph-mon[32371]: pgmap v922: 158 pgs: 158 active+clean; 370 MiB data, 5.7 GiB used, 620 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.38k op/s 2024-09-18T08:21:20.156 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:21:20.156 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3' 2024-09-18T08:21:20.163 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid 779349ca-7593-11ef-bceb-c7b262605968 -- ceph osd set noscrub 2024-09-18T08:21:20.257 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3 2024-09-18T08:21:20.269 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-09-18T08:21:20.269 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-09-18T08:21:20.269 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid 779349ca-7593-11ef-bceb-c7b262605968 -- ceph tell osd.3 'trim stale osdmaps' 2024-09-18T08:21:20.303 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:21:20.304 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:21:20.304 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3' 2024-09-18T08:21:20.405 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2 2024-09-18T08:21:20.455 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:21:20.456 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:21:20.456 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2' 2024-09-18T08:21:20.557 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2 2024-09-18T08:21:20.602 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:21:20.603 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:21:20.603 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2' 2024-09-18T08:21:20.704 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1 2024-09-18T08:21:20.750 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:21:20.750 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:21:20.751 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1' 2024-09-18T08:21:20.852 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7 2024-09-18T08:21:20.890 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:21:20.891 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:21:20.891 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7' 2024-09-18T08:21:20.904 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:21:20 smithi007 ceph-mon[29459]: 6.7 scrub starts 2024-09-18T08:21:20.904 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:21:20 smithi007 ceph-mon[29459]: 6.7 scrub ok 2024-09-18T08:21:20.904 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:21:20 smithi007 ceph-mon[29459]: osdmap e452: 8 total, 8 up, 7 in 2024-09-18T08:21:20.904 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:21:20 smithi007 ceph-mon[29459]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-18T08:21:20.904 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:21:20 smithi007 ceph-mon[39249]: 6.7 scrub starts 2024-09-18T08:21:20.904 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:21:20 smithi007 ceph-mon[39249]: 6.7 scrub ok 2024-09-18T08:21:20.904 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:21:20 smithi007 ceph-mon[39249]: osdmap e452: 8 total, 8 up, 7 in 2024-09-18T08:21:20.904 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:21:20 smithi007 ceph-mon[39249]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-18T08:21:20.992 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4 2024-09-18T08:21:21.025 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:21:21.025 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:21:21.025 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4' 2024-09-18T08:21:21.057 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/779349ca-7593-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T08:21:21.126 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5 2024-09-18T08:21:21.155 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:21:20 smithi050 ceph-mon[32371]: 6.7 scrub starts 2024-09-18T08:21:21.155 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:21:20 smithi050 ceph-mon[32371]: 6.7 scrub ok 2024-09-18T08:21:21.155 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:21:20 smithi050 ceph-mon[32371]: osdmap e452: 8 total, 8 up, 7 in 2024-09-18T08:21:21.155 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:21:20 smithi050 ceph-mon[32371]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-18T08:21:21.158 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:21:21.158 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:21:21.159 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5' 2024-09-18T08:21:21.260 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0 2024-09-18T08:21:21.305 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:21:21.306 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:21:21.306 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0' 2024-09-18T08:21:21.407 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5 2024-09-18T08:21:21.411 INFO:tasks.radosbench.radosbench.0.smithi007.stdout:Removed 98681 objects 2024-09-18T08:21:21.411 INFO:tasks.radosbench.radosbench.0.smithi007.stdout:Clean up completed and total clean up time :74.9127 2024-09-18T08:21:21.429 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:21:21.430 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:21:21.430 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5' 2024-09-18T08:21:21.531 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3 2024-09-18T08:21:21.554 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:21:21.554 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:21:21.555 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3' 2024-09-18T08:21:21.656 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6 2024-09-18T08:21:21.663 INFO:teuthology.orchestra.run.smithi007.stdout:{ 2024-09-18T08:21:21.663 INFO:teuthology.orchestra.run.smithi007.stdout: "ops": [], 2024-09-18T08:21:21.663 INFO:teuthology.orchestra.run.smithi007.stdout: "num_ops": 0 2024-09-18T08:21:21.664 INFO:teuthology.orchestra.run.smithi007.stdout:} 2024-09-18T08:21:21.680 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:21:21.680 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:21:21.681 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6' 2024-09-18T08:21:21.782 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0 2024-09-18T08:21:21.806 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:21:21.807 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:21:21.807 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0' 2024-09-18T08:21:21.908 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7 2024-09-18T08:21:21.913 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/779349ca-7593-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T08:21:21.914 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/779349ca-7593-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T08:21:21.935 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:21:21.936 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:21:21.936 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7' 2024-09-18T08:21:21.979 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:21:21 smithi007 ceph-mon[29459]: pgmap v924: 158 pgs: 158 active+clean; 214 MiB data, 5.5 GiB used, 620 GiB / 626 GiB avail; 756 B/s rd, 0 B/s wr, 1.35k op/s; 0 B/s, 0 objects/s recovering 2024-09-18T08:21:21.980 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:21:21 smithi007 ceph-mon[39249]: pgmap v924: 158 pgs: 158 active+clean; 214 MiB data, 5.5 GiB used, 620 GiB / 626 GiB avail; 756 B/s rd, 0 B/s wr, 1.35k op/s; 0 B/s, 0 objects/s recovering 2024-09-18T08:21:22.037 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1 2024-09-18T08:21:22.059 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:21:22.060 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:21:22.061 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1' 2024-09-18T08:21:22.155 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:21:21 smithi050 ceph-mon[32371]: pgmap v924: 158 pgs: 158 active+clean; 214 MiB data, 5.5 GiB used, 620 GiB / 626 GiB avail; 756 B/s rd, 0 B/s wr, 1.35k op/s; 0 B/s, 0 objects/s recovering 2024-09-18T08:21:22.162 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7 2024-09-18T08:21:22.189 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:21:22.189 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:21:22.190 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7' 2024-09-18T08:21:22.291 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1 2024-09-18T08:21:22.313 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:21:22.314 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:21:22.314 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1' 2024-09-18T08:21:22.388 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid 779349ca-7593-11ef-bceb-c7b262605968 -- ceph daemon osd.1 dump_blocked_ops 2024-09-18T08:21:22.416 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3 2024-09-18T08:21:22.454 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:21:22.454 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:21:22.455 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3' 2024-09-18T08:21:22.556 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1 2024-09-18T08:21:22.578 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:21:22.579 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:21:22.579 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1' 2024-09-18T08:21:22.680 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5 2024-09-18T08:21:22.708 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:21:22.709 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:21:22.709 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5' 2024-09-18T08:21:22.810 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2 2024-09-18T08:21:22.833 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:21:22.834 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:21:22.834 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2' 2024-09-18T08:21:22.935 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4 2024-09-18T08:21:22.961 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:21:22.962 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:21:22.962 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4' 2024-09-18T08:21:23.063 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1 2024-09-18T08:21:23.086 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:21:23.086 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid 779349ca-7593-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-18T08:21:23.087 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:21:23.087 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1' 2024-09-18T08:21:23.188 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7 2024-09-18T08:21:23.215 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:21:23.215 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:21:23.216 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7' 2024-09-18T08:21:23.219 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:21:22 smithi007 ceph-mon[29459]: 4.4 scrub starts 2024-09-18T08:21:23.220 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:21:22 smithi007 ceph-mon[29459]: 4.4 scrub ok 2024-09-18T08:21:23.221 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:21:22 smithi007 ceph-mon[39249]: 4.4 scrub starts 2024-09-18T08:21:23.221 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:21:22 smithi007 ceph-mon[39249]: 4.4 scrub ok 2024-09-18T08:21:23.248 INFO:teuthology.orchestra.run.smithi007.stderr:no valid command found; 10 closest matches: 2024-09-18T08:21:23.248 INFO:teuthology.orchestra.run.smithi007.stderr:0 2024-09-18T08:21:23.248 INFO:teuthology.orchestra.run.smithi007.stderr:1 2024-09-18T08:21:23.248 INFO:teuthology.orchestra.run.smithi007.stderr:2 2024-09-18T08:21:23.248 INFO:teuthology.orchestra.run.smithi007.stderr:abort 2024-09-18T08:21:23.248 INFO:teuthology.orchestra.run.smithi007.stderr:assert 2024-09-18T08:21:23.248 INFO:teuthology.orchestra.run.smithi007.stderr:bench [] [] [] [] 2024-09-18T08:21:23.248 INFO:teuthology.orchestra.run.smithi007.stderr:bluefs debug_inject_read_zeros 2024-09-18T08:21:23.249 INFO:teuthology.orchestra.run.smithi007.stderr:bluefs files list 2024-09-18T08:21:23.249 INFO:teuthology.orchestra.run.smithi007.stderr:bluefs stats 2024-09-18T08:21:23.249 INFO:teuthology.orchestra.run.smithi007.stderr:bluestore allocator dump block 2024-09-18T08:21:23.249 INFO:teuthology.orchestra.run.smithi007.stderr:Error EINVAL: invalid command 2024-09-18T08:21:23.316 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5 2024-09-18T08:21:23.343 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:21:23.344 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:21:23.344 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5' 2024-09-18T08:21:23.405 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:21:22 smithi050 ceph-mon[32371]: 4.4 scrub starts 2024-09-18T08:21:23.405 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:21:22 smithi050 ceph-mon[32371]: 4.4 scrub ok 2024-09-18T08:21:23.445 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7 2024-09-18T08:21:23.468 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/779349ca-7593-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T08:21:23.472 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:21:23.473 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:21:23.473 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7' 2024-09-18T08:21:23.574 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3 2024-09-18T08:21:23.596 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:21:23.597 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:21:23.597 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3' 2024-09-18T08:21:23.698 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5 2024-09-18T08:21:23.726 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:21:23.727 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:21:23.727 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5' 2024-09-18T08:21:23.828 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3 2024-09-18T08:21:23.850 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:21:23.850 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:21:23.851 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3' 2024-09-18T08:21:23.952 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2 2024-09-18T08:21:23.968 INFO:teuthology.orchestra.run.smithi007.stderr:noscrub is set 2024-09-18T08:21:23.977 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:21:23.978 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:21:23.978 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2' 2024-09-18T08:21:24.079 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5 2024-09-18T08:21:24.105 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:21:24.106 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:21:24.106 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5' 2024-09-18T08:21:24.207 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0 2024-09-18T08:21:24.229 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:21:23 smithi007 ceph-mon[29459]: 4.16 scrub starts 2024-09-18T08:21:24.229 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:21:23 smithi007 ceph-mon[29459]: 4.16 scrub ok 2024-09-18T08:21:24.229 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:21:23 smithi007 ceph-mon[29459]: from='client.? 172.21.15.7:0/858377017' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-18T08:21:24.230 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:21:23 smithi007 ceph-mon[29459]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-18T08:21:24.230 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:21:23 smithi007 ceph-mon[29459]: pgmap v925: 158 pgs: 158 active+clean; 109 MiB data, 5.0 GiB used, 621 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.05k op/s; 0 B/s, 0 objects/s recovering 2024-09-18T08:21:24.230 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:21:23 smithi007 ceph-mon[39249]: 4.16 scrub starts 2024-09-18T08:21:24.230 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:21:23 smithi007 ceph-mon[39249]: 4.16 scrub ok 2024-09-18T08:21:24.230 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:21:23 smithi007 ceph-mon[39249]: from='client.? 172.21.15.7:0/858377017' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-18T08:21:24.230 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:21:23 smithi007 ceph-mon[39249]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-18T08:21:24.230 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:21:23 smithi007 ceph-mon[39249]: pgmap v925: 158 pgs: 158 active+clean; 109 MiB data, 5.0 GiB used, 621 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.05k op/s; 0 B/s, 0 objects/s recovering 2024-09-18T08:21:24.231 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:21:24.231 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:21:24.232 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0' 2024-09-18T08:21:24.332 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4 2024-09-18T08:21:24.360 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:21:24.360 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:21:24.361 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4' 2024-09-18T08:21:24.405 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:21:23 smithi050 ceph-mon[32371]: 4.16 scrub starts 2024-09-18T08:21:24.405 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:21:23 smithi050 ceph-mon[32371]: 4.16 scrub ok 2024-09-18T08:21:24.405 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:21:23 smithi050 ceph-mon[32371]: from='client.? 172.21.15.7:0/858377017' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-18T08:21:24.405 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:21:23 smithi050 ceph-mon[32371]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-18T08:21:24.405 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:21:23 smithi050 ceph-mon[32371]: pgmap v925: 158 pgs: 158 active+clean; 109 MiB data, 5.0 GiB used, 621 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.05k op/s; 0 B/s, 0 objects/s recovering 2024-09-18T08:21:24.463 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5 2024-09-18T08:21:24.489 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:21:24.490 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:21:24.491 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5' 2024-09-18T08:21:24.591 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2 2024-09-18T08:21:24.617 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:21:24.618 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:21:24.618 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2' 2024-09-18T08:21:24.719 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1 2024-09-18T08:21:24.745 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:21:24.745 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:21:24.745 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1' 2024-09-18T08:21:24.846 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6 2024-09-18T08:21:24.873 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:21:24.874 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:21:24.874 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6' 2024-09-18T08:21:24.975 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0 2024-09-18T08:21:25.002 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:21:25.003 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:21:25.004 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0' 2024-09-18T08:21:25.104 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7 2024-09-18T08:21:25.131 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:21:25.131 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:21:25.132 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7' 2024-09-18T08:21:25.230 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:21:24 smithi007 ceph-mon[29459]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-18T08:21:25.230 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:21:24 smithi007 ceph-mon[29459]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-18T08:21:25.230 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:21:24 smithi007 ceph-mon[29459]: osdmap e453: 8 total, 8 up, 7 in 2024-09-18T08:21:25.230 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:21:24 smithi007 ceph-mon[39249]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-18T08:21:25.230 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:21:24 smithi007 ceph-mon[39249]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-18T08:21:25.230 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:21:24 smithi007 ceph-mon[39249]: osdmap e453: 8 total, 8 up, 7 in 2024-09-18T08:21:25.233 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3 2024-09-18T08:21:25.259 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:21:25.260 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:21:25.261 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3' 2024-09-18T08:21:25.361 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1 2024-09-18T08:21:25.388 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:21:25.389 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:21:25.390 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1' 2024-09-18T08:21:25.405 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:21:24 smithi050 ceph-mon[32371]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-18T08:21:25.405 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:21:24 smithi050 ceph-mon[32371]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-18T08:21:25.405 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:21:24 smithi050 ceph-mon[32371]: osdmap e453: 8 total, 8 up, 7 in 2024-09-18T08:21:25.490 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4 2024-09-18T08:21:25.517 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:21:25.518 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:21:25.518 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4' 2024-09-18T08:21:25.619 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6 2024-09-18T08:21:25.646 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:21:25.646 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:21:25.647 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6' 2024-09-18T08:21:25.747 INFO:tasks.cephadm.ceph_manager.ceph:removing pool_name unique_pool_4 2024-09-18T08:21:25.747 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid 779349ca-7593-11ef-bceb-c7b262605968 -- ceph osd pool rm unique_pool_4 unique_pool_4 --yes-i-really-really-mean-it 2024-09-18T08:21:25.748 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5 2024-09-18T08:21:25.774 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:21:25.774 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:21:25.775 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5' 2024-09-18T08:21:25.864 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-09-18T08:21:25.865 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-09-18T08:21:25.865 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid 779349ca-7593-11ef-bceb-c7b262605968 -- ceph tell osd.6 'trim stale osdmaps' 2024-09-18T08:21:25.876 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5 2024-09-18T08:21:25.902 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:21:25.903 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:21:25.903 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5' 2024-09-18T08:21:26.005 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1 2024-09-18T08:21:26.027 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:21:26.028 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:21:26.028 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1' 2024-09-18T08:21:26.079 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:21:25 smithi007 ceph-mon[29459]: pgmap v927: 158 pgs: 158 active+clean; 583 KiB data, 4.7 GiB used, 621 GiB / 626 GiB avail; 759 B/s rd, 0 B/s wr, 876 op/s; 0 B/s, 0 objects/s recovering 2024-09-18T08:21:26.080 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:21:25 smithi007 ceph-mon[39249]: pgmap v927: 158 pgs: 158 active+clean; 583 KiB data, 4.7 GiB used, 621 GiB / 626 GiB avail; 759 B/s rd, 0 B/s wr, 876 op/s; 0 B/s, 0 objects/s recovering 2024-09-18T08:21:26.130 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5 2024-09-18T08:21:26.157 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:21:26.157 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:21:26.158 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5' 2024-09-18T08:21:26.258 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0 2024-09-18T08:21:26.285 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:21:26.286 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:21:26.286 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0' 2024-09-18T08:21:26.387 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1 2024-09-18T08:21:26.405 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:21:25 smithi050 ceph-mon[32371]: pgmap v927: 158 pgs: 158 active+clean; 583 KiB data, 4.7 GiB used, 621 GiB / 626 GiB avail; 759 B/s rd, 0 B/s wr, 876 op/s; 0 B/s, 0 objects/s recovering 2024-09-18T08:21:26.410 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:21:26.410 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:21:26.411 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1' 2024-09-18T08:21:26.512 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0 2024-09-18T08:21:26.535 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:21:26.535 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:21:26.536 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0' 2024-09-18T08:21:26.637 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5 2024-09-18T08:21:26.664 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:21:26.664 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:21:26.665 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5' 2024-09-18T08:21:26.766 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2 2024-09-18T08:21:26.790 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:21:26.790 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:21:26.791 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2' 2024-09-18T08:21:26.892 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2 2024-09-18T08:21:26.914 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:21:26.915 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:21:26.915 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2' 2024-09-18T08:21:26.938 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/779349ca-7593-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T08:21:26.975 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/779349ca-7593-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T08:21:26.976 INFO:teuthology.orchestra.run.smithi007.stdout:{ 2024-09-18T08:21:26.976 INFO:teuthology.orchestra.run.smithi007.stdout: "ops": [], 2024-09-18T08:21:26.976 INFO:teuthology.orchestra.run.smithi007.stdout: "complaint_time": 30, 2024-09-18T08:21:26.976 INFO:teuthology.orchestra.run.smithi007.stdout: "num_blocked_ops": 0 2024-09-18T08:21:26.976 INFO:teuthology.orchestra.run.smithi007.stdout:} 2024-09-18T08:21:26.976 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Sep 18 08:21:26 smithi007 ceph-779349ca-7593-11ef-bceb-c7b262605968-mgr-y[29688]: ::ffff:172.21.15.50 - - [18/Sep/2024:08:21:26] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-09-18T08:21:26.987 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/779349ca-7593-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T08:21:27.016 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2 2024-09-18T08:21:27.040 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:21:27.040 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:21:27.040 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2' 2024-09-18T08:21:27.141 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1 2024-09-18T08:21:27.164 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:21:27.164 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:21:27.165 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1' 2024-09-18T08:21:27.266 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6 2024-09-18T08:21:27.293 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:21:27.293 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:21:27.294 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6' 2024-09-18T08:21:27.394 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4 2024-09-18T08:21:27.421 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:21:27.422 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:21:27.422 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4' 2024-09-18T08:21:27.523 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0 2024-09-18T08:21:27.545 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:21:27.546 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:21:27.546 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0' 2024-09-18T08:21:27.647 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7 2024-09-18T08:21:27.675 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:21:27.676 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:21:27.676 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7' 2024-09-18T08:21:27.749 INFO:teuthology.orchestra.run.smithi007.stdout:osd.0: {} 2024-09-18T08:21:27.750 INFO:teuthology.orchestra.run.smithi007.stdout:osd.1: {} 2024-09-18T08:21:27.750 INFO:teuthology.orchestra.run.smithi007.stdout:osd.2: {} 2024-09-18T08:21:27.750 INFO:teuthology.orchestra.run.smithi007.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-18T08:21:27.750 INFO:teuthology.orchestra.run.smithi007.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-18T08:21:27.750 INFO:teuthology.orchestra.run.smithi007.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-18T08:21:27.758 INFO:teuthology.orchestra.run.smithi007.stdout:osd.3: {} 2024-09-18T08:21:27.759 INFO:teuthology.orchestra.run.smithi007.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-18T08:21:27.776 INFO:teuthology.orchestra.run.smithi007.stdout:osd.4: {} 2024-09-18T08:21:27.777 INFO:teuthology.orchestra.run.smithi007.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-18T08:21:27.777 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5 2024-09-18T08:21:27.793 INFO:teuthology.orchestra.run.smithi007.stdout:osd.5: {} 2024-09-18T08:21:27.793 INFO:teuthology.orchestra.run.smithi007.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-18T08:21:27.803 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:21:27.804 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:21:27.805 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5' 2024-09-18T08:21:27.810 INFO:teuthology.orchestra.run.smithi007.stdout:osd.6: {} 2024-09-18T08:21:27.811 INFO:teuthology.orchestra.run.smithi007.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-18T08:21:27.827 INFO:teuthology.orchestra.run.smithi007.stdout:osd.7: {} 2024-09-18T08:21:27.827 INFO:teuthology.orchestra.run.smithi007.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-18T08:21:27.905 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4 2024-09-18T08:21:27.931 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:21:27.932 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:21:27.932 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4' 2024-09-18T08:21:28.033 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5 2024-09-18T08:21:28.059 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:21:28.059 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:21:28.060 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5' 2024-09-18T08:21:28.161 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1 2024-09-18T08:21:28.184 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:21:28.185 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:21:28.185 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1' 2024-09-18T08:21:28.286 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1 2024-09-18T08:21:28.309 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:21:28.310 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:21:28.310 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1' 2024-09-18T08:21:28.411 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4 2024-09-18T08:21:28.442 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:21:28.443 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:21:28.443 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4' 2024-09-18T08:21:28.473 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid 779349ca-7593-11ef-bceb-c7b262605968 -- ceph daemon osd.1 dump_historic_ops 2024-09-18T08:21:28.543 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4 2024-09-18T08:21:28.571 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:21:28.572 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:21:28.572 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4' 2024-09-18T08:21:28.673 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2 2024-09-18T08:21:28.696 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:21:28.697 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:21:28.697 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2' 2024-09-18T08:21:28.728 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:21:28 smithi007 ceph-mon[29459]: pgmap v928: 158 pgs: 158 active+clean; 583 KiB data, 4.7 GiB used, 621 GiB / 626 GiB avail; 639 B/s rd, 0 B/s wr, 738 op/s; 0 B/s, 0 objects/s recovering 2024-09-18T08:21:28.728 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:21:28 smithi007 ceph-mon[29459]: from='client.? 172.21.15.7:0/1178778645' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-18T08:21:28.729 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:21:28 smithi007 ceph-mon[39249]: pgmap v928: 158 pgs: 158 active+clean; 583 KiB data, 4.7 GiB used, 621 GiB / 626 GiB avail; 639 B/s rd, 0 B/s wr, 738 op/s; 0 B/s, 0 objects/s recovering 2024-09-18T08:21:28.729 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:21:28 smithi007 ceph-mon[39249]: from='client.? 172.21.15.7:0/1178778645' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-18T08:21:28.741 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid 779349ca-7593-11ef-bceb-c7b262605968 -- ceph osd set nodeep-scrub 2024-09-18T08:21:28.798 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4 2024-09-18T08:21:28.825 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:21:28.826 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:21:28.826 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4' 2024-09-18T08:21:28.905 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:21:28 smithi050 ceph-mon[32371]: pgmap v928: 158 pgs: 158 active+clean; 583 KiB data, 4.7 GiB used, 621 GiB / 626 GiB avail; 639 B/s rd, 0 B/s wr, 738 op/s; 0 B/s, 0 objects/s recovering 2024-09-18T08:21:28.905 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:21:28 smithi050 ceph-mon[32371]: from='client.? 172.21.15.7:0/1178778645' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-18T08:21:28.927 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2 2024-09-18T08:21:28.952 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:21:28.952 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:21:28.953 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2' 2024-09-18T08:21:29.056 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2 2024-09-18T08:21:29.079 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:21:29.080 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:21:29.080 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2' 2024-09-18T08:21:29.181 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1 2024-09-18T08:21:29.204 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:21:29.204 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:21:29.205 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1' 2024-09-18T08:21:29.305 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7 2024-09-18T08:21:29.333 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:21:29.334 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:21:29.334 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7' 2024-09-18T08:21:29.435 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0 2024-09-18T08:21:29.457 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:21:29.458 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:21:29.458 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0' 2024-09-18T08:21:29.559 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0 2024-09-18T08:21:29.582 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:21:29.583 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:21:29.583 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0' 2024-09-18T08:21:29.638 INFO:teuthology.orchestra.run.smithi007.stderr:no valid command found; 10 closest matches: 2024-09-18T08:21:29.638 INFO:teuthology.orchestra.run.smithi007.stderr:0 2024-09-18T08:21:29.638 INFO:teuthology.orchestra.run.smithi007.stderr:1 2024-09-18T08:21:29.638 INFO:teuthology.orchestra.run.smithi007.stderr:2 2024-09-18T08:21:29.638 INFO:teuthology.orchestra.run.smithi007.stderr:abort 2024-09-18T08:21:29.638 INFO:teuthology.orchestra.run.smithi007.stderr:assert 2024-09-18T08:21:29.638 INFO:teuthology.orchestra.run.smithi007.stderr:bench [] [] [] [] 2024-09-18T08:21:29.638 INFO:teuthology.orchestra.run.smithi007.stderr:bluefs debug_inject_read_zeros 2024-09-18T08:21:29.638 INFO:teuthology.orchestra.run.smithi007.stderr:bluefs files list 2024-09-18T08:21:29.639 INFO:teuthology.orchestra.run.smithi007.stderr:bluefs stats 2024-09-18T08:21:29.639 INFO:teuthology.orchestra.run.smithi007.stderr:bluestore allocator dump block 2024-09-18T08:21:29.639 INFO:teuthology.orchestra.run.smithi007.stderr:Error EINVAL: invalid command 2024-09-18T08:21:29.684 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0 2024-09-18T08:21:29.707 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:21:29.707 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:21:29.708 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0' 2024-09-18T08:21:29.809 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6 2024-09-18T08:21:29.835 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:21:29.835 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:21:29.836 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6' 2024-09-18T08:21:29.938 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6 2024-09-18T08:21:29.965 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:21:29.965 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:21:29.965 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6' 2024-09-18T08:21:30.066 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4 2024-09-18T08:21:30.092 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:21:30.092 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:21:30.093 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4' 2024-09-18T08:21:30.155 INFO:journalctl@ceph.iscsi.iscsi.a.smithi050.stdout:Sep 18 08:21:29 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-iscsi-iscsi-a[57782]: debug there is no tcmu-runner data available 2024-09-18T08:21:30.193 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3 2024-09-18T08:21:30.216 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:21:30.217 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:21:30.217 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.3' 2024-09-18T08:21:30.318 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7 2024-09-18T08:21:30.345 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:21:30.346 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:21:30.346 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7' 2024-09-18T08:21:30.354 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-09-18T08:21:30.354 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-09-18T08:21:30.354 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid 779349ca-7593-11ef-bceb-c7b262605968 -- ceph tell osd.5 'trim stale osdmaps' 2024-09-18T08:21:30.447 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4 2024-09-18T08:21:30.474 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:21:30.475 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:21:30.475 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4' 2024-09-18T08:21:30.576 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7 2024-09-18T08:21:30.604 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:21:30.605 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:21:30.605 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7' 2024-09-18T08:21:30.637 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/779349ca-7593-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T08:21:30.641 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:21:30 smithi007 ceph-mon[29459]: pgmap v929: 158 pgs: 158 active+clean; 583 KiB data, 4.7 GiB used, 621 GiB / 626 GiB avail; 1.0 KiB/s rd, 0 B/s wr, 607 op/s; 0 B/s, 0 objects/s recovering 2024-09-18T08:21:30.641 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:21:30 smithi007 ceph-mon[39249]: pgmap v929: 158 pgs: 158 active+clean; 583 KiB data, 4.7 GiB used, 621 GiB / 626 GiB avail; 1.0 KiB/s rd, 0 B/s wr, 607 op/s; 0 B/s, 0 objects/s recovering 2024-09-18T08:21:30.706 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1 2024-09-18T08:21:30.729 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:21:30.730 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:21:30.730 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.1' 2024-09-18T08:21:30.832 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0 2024-09-18T08:21:30.854 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:21:30.855 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:21:30.855 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0' 2024-09-18T08:21:30.905 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:21:30 smithi050 ceph-mon[32371]: pgmap v929: 158 pgs: 158 active+clean; 583 KiB data, 4.7 GiB used, 621 GiB / 626 GiB avail; 1.0 KiB/s rd, 0 B/s wr, 607 op/s; 0 B/s, 0 objects/s recovering 2024-09-18T08:21:30.908 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/779349ca-7593-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T08:21:30.956 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4 2024-09-18T08:21:30.983 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:21:30.984 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:21:30.984 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4' 2024-09-18T08:21:31.085 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5 2024-09-18T08:21:31.112 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:21:31.112 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:21:31.113 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5' 2024-09-18T08:21:31.214 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4 2024-09-18T08:21:31.240 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:21:31.241 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:21:31.241 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4' 2024-09-18T08:21:31.342 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2 2024-09-18T08:21:31.369 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:21:31.370 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:21:31.370 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2' 2024-09-18T08:21:31.378 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid 779349ca-7593-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-18T08:21:31.471 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7 2024-09-18T08:21:31.498 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:21:31.499 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:21:31.499 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.7' 2024-09-18T08:21:31.516 INFO:teuthology.orchestra.run.smithi007.stderr:pool 'unique_pool_4' removed 2024-09-18T08:21:31.600 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2 2024-09-18T08:21:31.655 INFO:teuthology.orchestra.run.smithi007.stdout:ERROR: (22) Invalid argument 2024-09-18T08:21:31.656 INFO:teuthology.orchestra.run.smithi007.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-18T08:21:31.656 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:21:31.657 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:21:31.657 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.2' 2024-09-18T08:21:31.758 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5 2024-09-18T08:21:31.781 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:21:31.782 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:21:31.782 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.5' 2024-09-18T08:21:31.883 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6 2024-09-18T08:21:31.905 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:21:31 smithi050 ceph-mon[32371]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-18T08:21:31.905 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:21:31 smithi050 ceph-mon[32371]: from='client.? 172.21.15.7:0/4268484007' entity='client.admin' cmd={"prefix": "osd pool rm", "pool": "unique_pool_4", "pool2": "unique_pool_4", "yes_i_really_really_mean_it": true} : dispatch 2024-09-18T08:21:31.905 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:21:31 smithi050 ceph-mon[32371]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool rm", "pool": "unique_pool_4", "pool2": "unique_pool_4", "yes_i_really_really_mean_it": true} : dispatch 2024-09-18T08:21:31.906 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:21:31.906 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:21:31.906 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6' 2024-09-18T08:21:31.979 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:21:31 smithi007 ceph-mon[29459]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-18T08:21:31.980 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:21:31 smithi007 ceph-mon[29459]: from='client.? 172.21.15.7:0/4268484007' entity='client.admin' cmd={"prefix": "osd pool rm", "pool": "unique_pool_4", "pool2": "unique_pool_4", "yes_i_really_really_mean_it": true} : dispatch 2024-09-18T08:21:31.980 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:21:31 smithi007 ceph-mon[29459]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool rm", "pool": "unique_pool_4", "pool2": "unique_pool_4", "yes_i_really_really_mean_it": true} : dispatch 2024-09-18T08:21:31.980 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:21:31 smithi007 ceph-mon[39249]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-18T08:21:31.980 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:21:31 smithi007 ceph-mon[39249]: from='client.? 172.21.15.7:0/4268484007' entity='client.admin' cmd={"prefix": "osd pool rm", "pool": "unique_pool_4", "pool2": "unique_pool_4", "yes_i_really_really_mean_it": true} : dispatch 2024-09-18T08:21:31.980 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:21:31 smithi007 ceph-mon[39249]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool rm", "pool": "unique_pool_4", "pool2": "unique_pool_4", "yes_i_really_really_mean_it": true} : dispatch 2024-09-18T08:21:32.007 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6 2024-09-18T08:21:32.029 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:21:32.029 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:21:32.030 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.6' 2024-09-18T08:21:32.131 DEBUG:teuthology.orchestra.run.smithi007:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0 2024-09-18T08:21:32.155 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: docker: command not found 2024-09-18T08:21:32.155 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:21:32.156 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi007 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.0' 2024-09-18T08:21:32.257 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4 2024-09-18T08:21:32.279 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: docker: command not found 2024-09-18T08:21:32.279 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:21:32.280 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd.4' 2024-09-18T08:21:32.371 DEBUG:teuthology.run_tasks:Unwinding manager thrashosds 2024-09-18T08:21:32.381 INFO:tasks.thrashosds:joining thrashosds 2024-09-18T08:21:32.580 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/779349ca-7593-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T08:21:32.702 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:21:32 smithi007 ceph-mon[29459]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "unique_pool_4", "pool2": "unique_pool_4", "yes_i_really_really_mean_it": true}]': finished 2024-09-18T08:21:32.702 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:21:32 smithi007 ceph-mon[29459]: osdmap e454: 8 total, 8 up, 7 in 2024-09-18T08:21:32.702 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:21:32 smithi007 ceph-mon[29459]: pgmap v931: 142 pgs: 142 active+clean; 583 KiB data, 4.7 GiB used, 621 GiB / 626 GiB avail; 511 B/s rd, 0 op/s 2024-09-18T08:21:32.703 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:21:32 smithi007 ceph-mon[39249]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "unique_pool_4", "pool2": "unique_pool_4", "yes_i_really_really_mean_it": true}]': finished 2024-09-18T08:21:32.703 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:21:32 smithi007 ceph-mon[39249]: osdmap e454: 8 total, 8 up, 7 in 2024-09-18T08:21:32.703 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:21:32 smithi007 ceph-mon[39249]: pgmap v931: 142 pgs: 142 active+clean; 583 KiB data, 4.7 GiB used, 621 GiB / 626 GiB avail; 511 B/s rd, 0 op/s 2024-09-18T08:21:32.905 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:21:32 smithi050 ceph-mon[32371]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "unique_pool_4", "pool2": "unique_pool_4", "yes_i_really_really_mean_it": true}]': finished 2024-09-18T08:21:32.905 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:21:32 smithi050 ceph-mon[32371]: osdmap e454: 8 total, 8 up, 7 in 2024-09-18T08:21:32.905 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:21:32 smithi050 ceph-mon[32371]: pgmap v931: 142 pgs: 142 active+clean; 583 KiB data, 4.7 GiB used, 621 GiB / 626 GiB avail; 511 B/s rd, 0 op/s 2024-09-18T08:21:33.327 INFO:teuthology.orchestra.run.smithi007.stderr:no valid command found; 10 closest matches: 2024-09-18T08:21:33.328 INFO:teuthology.orchestra.run.smithi007.stderr:0 2024-09-18T08:21:33.328 INFO:teuthology.orchestra.run.smithi007.stderr:1 2024-09-18T08:21:33.328 INFO:teuthology.orchestra.run.smithi007.stderr:2 2024-09-18T08:21:33.328 INFO:teuthology.orchestra.run.smithi007.stderr:abort 2024-09-18T08:21:33.328 INFO:teuthology.orchestra.run.smithi007.stderr:assert 2024-09-18T08:21:33.328 INFO:teuthology.orchestra.run.smithi007.stderr:bench [] [] [] [] 2024-09-18T08:21:33.328 INFO:teuthology.orchestra.run.smithi007.stderr:bluefs debug_inject_read_zeros 2024-09-18T08:21:33.328 INFO:teuthology.orchestra.run.smithi007.stderr:bluefs files list 2024-09-18T08:21:33.328 INFO:teuthology.orchestra.run.smithi007.stderr:bluefs stats 2024-09-18T08:21:33.328 INFO:teuthology.orchestra.run.smithi007.stderr:bluestore allocator dump block 2024-09-18T08:21:33.329 INFO:teuthology.orchestra.run.smithi007.stderr:Error EINVAL: invalid command 2024-09-18T08:21:33.537 INFO:teuthology.orchestra.run.smithi007.stderr:nodeep-scrub is set 2024-09-18T08:21:33.729 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:21:33 smithi007 ceph-mon[29459]: from='client.? 172.21.15.7:0/2705565813' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-18T08:21:33.730 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:21:33 smithi007 ceph-mon[29459]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-18T08:21:33.730 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:21:33 smithi007 ceph-mon[39249]: from='client.? 172.21.15.7:0/2705565813' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-18T08:21:33.730 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:21:33 smithi007 ceph-mon[39249]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-18T08:21:33.905 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:21:33 smithi050 ceph-mon[32371]: from='client.? 172.21.15.7:0/2705565813' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-18T08:21:33.905 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:21:33 smithi050 ceph-mon[32371]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-18T08:21:34.247 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/779349ca-7593-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T08:21:34.905 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:21:34 smithi050 ceph-mon[32371]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-18T08:21:34.905 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:21:34 smithi050 ceph-mon[32371]: pgmap v932: 142 pgs: 142 active+clean; 583 KiB data, 4.7 GiB used, 621 GiB / 626 GiB avail; 960 B/s rd, 0 op/s 2024-09-18T08:21:34.905 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:21:34 smithi050 ceph-mon[32371]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-18T08:21:34.905 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:21:34 smithi050 ceph-mon[32371]: osdmap e455: 8 total, 8 up, 7 in 2024-09-18T08:21:34.905 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:21:34 smithi050 ceph-mon[32371]: from='mgr.24470 172.21.15.7:0/2594390261' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-18T08:21:34.905 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:21:34 smithi050 ceph-mon[32371]: from='mgr.24470 172.21.15.7:0/2594390261' entity='mgr.y' 2024-09-18T08:21:34.905 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:21:34 smithi050 ceph-mon[32371]: from='mgr.24470 172.21.15.7:0/2594390261' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-18T08:21:34.906 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:21:34 smithi050 ceph-mon[32371]: from='mgr.24470 172.21.15.7:0/2594390261' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-18T08:21:34.906 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:21:34 smithi050 ceph-mon[32371]: from='mgr.24470 172.21.15.7:0/2594390261' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-18T08:21:34.906 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:21:34 smithi050 ceph-mon[32371]: from='mgr.24470 172.21.15.7:0/2594390261' entity='mgr.y' 2024-09-18T08:21:34.911 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:21:34 smithi007 ceph-mon[29459]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-18T08:21:34.911 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:21:34 smithi007 ceph-mon[29459]: pgmap v932: 142 pgs: 142 active+clean; 583 KiB data, 4.7 GiB used, 621 GiB / 626 GiB avail; 960 B/s rd, 0 op/s 2024-09-18T08:21:34.911 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:21:34 smithi007 ceph-mon[29459]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-18T08:21:34.911 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:21:34 smithi007 ceph-mon[29459]: osdmap e455: 8 total, 8 up, 7 in 2024-09-18T08:21:34.911 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:21:34 smithi007 ceph-mon[29459]: from='mgr.24470 172.21.15.7:0/2594390261' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-18T08:21:34.912 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:21:34 smithi007 ceph-mon[29459]: from='mgr.24470 172.21.15.7:0/2594390261' entity='mgr.y' 2024-09-18T08:21:34.912 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:21:34 smithi007 ceph-mon[29459]: from='mgr.24470 172.21.15.7:0/2594390261' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-18T08:21:34.912 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:21:34 smithi007 ceph-mon[29459]: from='mgr.24470 172.21.15.7:0/2594390261' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-18T08:21:34.912 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:21:34 smithi007 ceph-mon[29459]: from='mgr.24470 172.21.15.7:0/2594390261' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-18T08:21:34.912 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:21:34 smithi007 ceph-mon[29459]: from='mgr.24470 172.21.15.7:0/2594390261' entity='mgr.y' 2024-09-18T08:21:34.912 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:21:34 smithi007 ceph-mon[39249]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-18T08:21:34.912 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:21:34 smithi007 ceph-mon[39249]: pgmap v932: 142 pgs: 142 active+clean; 583 KiB data, 4.7 GiB used, 621 GiB / 626 GiB avail; 960 B/s rd, 0 op/s 2024-09-18T08:21:34.912 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:21:34 smithi007 ceph-mon[39249]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-18T08:21:34.912 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:21:34 smithi007 ceph-mon[39249]: osdmap e455: 8 total, 8 up, 7 in 2024-09-18T08:21:34.912 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:21:34 smithi007 ceph-mon[39249]: from='mgr.24470 172.21.15.7:0/2594390261' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-18T08:21:34.913 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:21:34 smithi007 ceph-mon[39249]: from='mgr.24470 172.21.15.7:0/2594390261' entity='mgr.y' 2024-09-18T08:21:34.913 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:21:34 smithi007 ceph-mon[39249]: from='mgr.24470 172.21.15.7:0/2594390261' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-18T08:21:34.913 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:21:34 smithi007 ceph-mon[39249]: from='mgr.24470 172.21.15.7:0/2594390261' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-18T08:21:34.913 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:21:34 smithi007 ceph-mon[39249]: from='mgr.24470 172.21.15.7:0/2594390261' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-18T08:21:34.913 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:21:34 smithi007 ceph-mon[39249]: from='mgr.24470 172.21.15.7:0/2594390261' entity='mgr.y' 2024-09-18T08:21:35.329 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-09-18T08:21:35.329 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-09-18T08:21:35.330 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid 779349ca-7593-11ef-bceb-c7b262605968 -- ceph tell osd.7 'trim stale osdmaps' 2024-09-18T08:21:35.821 INFO:teuthology.orchestra.run.smithi007.stdout:osd.0: {} 2024-09-18T08:21:35.821 INFO:teuthology.orchestra.run.smithi007.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-18T08:21:35.839 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/779349ca-7593-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T08:21:35.841 INFO:teuthology.orchestra.run.smithi007.stdout:osd.1: {} 2024-09-18T08:21:35.841 INFO:teuthology.orchestra.run.smithi007.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-18T08:21:35.867 INFO:teuthology.orchestra.run.smithi007.stdout:osd.2: {} 2024-09-18T08:21:35.867 INFO:teuthology.orchestra.run.smithi007.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-18T08:21:35.884 INFO:teuthology.orchestra.run.smithi007.stdout:osd.3: {} 2024-09-18T08:21:35.885 INFO:teuthology.orchestra.run.smithi007.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-18T08:21:35.902 INFO:teuthology.orchestra.run.smithi007.stdout:osd.4: {} 2024-09-18T08:21:35.902 INFO:teuthology.orchestra.run.smithi007.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-18T08:21:35.919 INFO:teuthology.orchestra.run.smithi007.stdout:osd.5: {} 2024-09-18T08:21:35.919 INFO:teuthology.orchestra.run.smithi007.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-18T08:21:35.937 INFO:teuthology.orchestra.run.smithi007.stdout:osd.6: {} 2024-09-18T08:21:35.937 INFO:teuthology.orchestra.run.smithi007.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-18T08:21:35.954 INFO:teuthology.orchestra.run.smithi007.stdout:osd.7: {} 2024-09-18T08:21:35.954 INFO:teuthology.orchestra.run.smithi007.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-18T08:21:36.696 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:21:36 smithi007 ceph-mon[29459]: pgmap v934: 142 pgs: 142 active+clean; 583 KiB data, 4.7 GiB used, 621 GiB / 626 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-09-18T08:21:36.697 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:21:36 smithi007 ceph-mon[29459]: from='client.? 172.21.15.7:0/1182843775' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-18T08:21:36.697 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:21:36 smithi007 ceph-mon[39249]: pgmap v934: 142 pgs: 142 active+clean; 583 KiB data, 4.7 GiB used, 621 GiB / 626 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-09-18T08:21:36.697 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:21:36 smithi007 ceph-mon[39249]: from='client.? 172.21.15.7:0/1182843775' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-18T08:21:36.751 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid 779349ca-7593-11ef-bceb-c7b262605968 -- ceph osd unset noscrub 2024-09-18T08:21:36.905 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:21:36 smithi050 ceph-mon[32371]: pgmap v934: 142 pgs: 142 active+clean; 583 KiB data, 4.7 GiB used, 621 GiB / 626 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-09-18T08:21:36.905 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:21:36 smithi050 ceph-mon[32371]: from='client.? 172.21.15.7:0/1182843775' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-18T08:21:36.979 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Sep 18 08:21:36 smithi007 ceph-779349ca-7593-11ef-bceb-c7b262605968-mgr-y[29688]: ::ffff:172.21.15.50 - - [18/Sep/2024:08:21:36] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-09-18T08:21:37.179 INFO:teuthology.orchestra.run.smithi007.stderr:no valid command found; 10 closest matches: 2024-09-18T08:21:37.179 INFO:teuthology.orchestra.run.smithi007.stderr:0 2024-09-18T08:21:37.179 INFO:teuthology.orchestra.run.smithi007.stderr:1 2024-09-18T08:21:37.179 INFO:teuthology.orchestra.run.smithi007.stderr:2 2024-09-18T08:21:37.179 INFO:teuthology.orchestra.run.smithi007.stderr:abort 2024-09-18T08:21:37.179 INFO:teuthology.orchestra.run.smithi007.stderr:assert 2024-09-18T08:21:37.179 INFO:teuthology.orchestra.run.smithi007.stderr:bench [] [] [] [] 2024-09-18T08:21:37.179 INFO:teuthology.orchestra.run.smithi007.stderr:bluefs debug_inject_read_zeros 2024-09-18T08:21:37.180 INFO:teuthology.orchestra.run.smithi007.stderr:bluefs files list 2024-09-18T08:21:37.180 INFO:teuthology.orchestra.run.smithi007.stderr:bluefs stats 2024-09-18T08:21:37.180 INFO:teuthology.orchestra.run.smithi007.stderr:bluestore allocator dump block 2024-09-18T08:21:37.180 INFO:teuthology.orchestra.run.smithi007.stderr:Error EINVAL: invalid command 2024-09-18T08:21:37.900 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-09-18T08:21:37.901 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-09-18T08:21:37.959 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/779349ca-7593-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T08:21:38.905 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:21:38 smithi050 ceph-mon[32371]: pgmap v935: 142 pgs: 142 active+clean; 583 KiB data, 4.7 GiB used, 621 GiB / 626 GiB avail; 511 B/s rd, 0 op/s 2024-09-18T08:21:38.979 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:21:38 smithi007 ceph-mon[29459]: pgmap v935: 142 pgs: 142 active+clean; 583 KiB data, 4.7 GiB used, 621 GiB / 626 GiB avail; 511 B/s rd, 0 op/s 2024-09-18T08:21:38.980 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:21:38 smithi007 ceph-mon[39249]: pgmap v935: 142 pgs: 142 active+clean; 583 KiB data, 4.7 GiB used, 621 GiB / 626 GiB avail; 511 B/s rd, 0 op/s 2024-09-18T08:21:39.554 INFO:teuthology.orchestra.run.smithi007.stderr:noscrub is unset 2024-09-18T08:21:39.853 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:21:39 smithi050 ceph-mon[32371]: purged_snaps scrub starts 2024-09-18T08:21:39.853 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:21:39 smithi050 ceph-mon[32371]: purged_snaps scrub ok 2024-09-18T08:21:39.853 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:21:39 smithi050 ceph-mon[32371]: osdmap e456: 8 total, 8 up, 7 in 2024-09-18T08:21:39.853 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:21:39 smithi050 ceph-mon[32371]: from='client.? 172.21.15.7:0/1141179337' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-18T08:21:39.853 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:21:39 smithi050 ceph-mon[32371]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-18T08:21:39.946 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:21:39 smithi007 ceph-mon[29459]: purged_snaps scrub starts 2024-09-18T08:21:39.946 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:21:39 smithi007 ceph-mon[29459]: purged_snaps scrub ok 2024-09-18T08:21:39.946 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:21:39 smithi007 ceph-mon[29459]: osdmap e456: 8 total, 8 up, 7 in 2024-09-18T08:21:39.946 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:21:39 smithi007 ceph-mon[29459]: from='client.? 172.21.15.7:0/1141179337' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-18T08:21:39.947 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:21:39 smithi007 ceph-mon[29459]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-18T08:21:39.947 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:21:39 smithi007 ceph-mon[39249]: purged_snaps scrub starts 2024-09-18T08:21:39.947 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:21:39 smithi007 ceph-mon[39249]: purged_snaps scrub ok 2024-09-18T08:21:39.947 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:21:39 smithi007 ceph-mon[39249]: osdmap e456: 8 total, 8 up, 7 in 2024-09-18T08:21:39.947 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:21:39 smithi007 ceph-mon[39249]: from='client.? 172.21.15.7:0/1141179337' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-18T08:21:39.947 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:21:39 smithi007 ceph-mon[39249]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-18T08:21:40.155 INFO:journalctl@ceph.iscsi.iscsi.a.smithi050.stdout:Sep 18 08:21:39 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-iscsi-iscsi-a[57782]: debug there is no tcmu-runner data available 2024-09-18T08:21:40.208 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid 779349ca-7593-11ef-bceb-c7b262605968 -- ceph osd unset nodeep-scrub 2024-09-18T08:21:40.470 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/779349ca-7593-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T08:21:40.866 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:21:40 smithi007 ceph-mon[29459]: pgmap v937: 142 pgs: 142 active+clean; 583 KiB data, 4.7 GiB used, 621 GiB / 626 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-18T08:21:40.866 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:21:40 smithi007 ceph-mon[29459]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-18T08:21:40.867 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:21:40 smithi007 ceph-mon[29459]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-18T08:21:40.867 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:21:40 smithi007 ceph-mon[29459]: osdmap e457: 8 total, 8 up, 7 in 2024-09-18T08:21:40.867 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:21:40 smithi007 ceph-mon[29459]: 6.10 scrub starts 2024-09-18T08:21:40.867 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:21:40 smithi007 ceph-mon[29459]: 6.10 scrub ok 2024-09-18T08:21:40.867 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:21:40 smithi007 ceph-mon[39249]: pgmap v937: 142 pgs: 142 active+clean; 583 KiB data, 4.7 GiB used, 621 GiB / 626 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-18T08:21:40.867 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:21:40 smithi007 ceph-mon[39249]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-18T08:21:40.868 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:21:40 smithi007 ceph-mon[39249]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-18T08:21:40.868 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:21:40 smithi007 ceph-mon[39249]: osdmap e457: 8 total, 8 up, 7 in 2024-09-18T08:21:40.868 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:21:40 smithi007 ceph-mon[39249]: 6.10 scrub starts 2024-09-18T08:21:40.868 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:21:40 smithi007 ceph-mon[39249]: 6.10 scrub ok 2024-09-18T08:21:40.905 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:21:40 smithi050 ceph-mon[32371]: pgmap v937: 142 pgs: 142 active+clean; 583 KiB data, 4.7 GiB used, 621 GiB / 626 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-18T08:21:40.905 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:21:40 smithi050 ceph-mon[32371]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-18T08:21:40.905 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:21:40 smithi050 ceph-mon[32371]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-18T08:21:40.905 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:21:40 smithi050 ceph-mon[32371]: osdmap e457: 8 total, 8 up, 7 in 2024-09-18T08:21:40.905 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:21:40 smithi050 ceph-mon[32371]: 6.10 scrub starts 2024-09-18T08:21:40.905 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:21:40 smithi050 ceph-mon[32371]: 6.10 scrub ok 2024-09-18T08:21:41.556 INFO:teuthology.orchestra.run.smithi007.stderr:nodeep-scrub is unset 2024-09-18T08:21:41.807 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:21:41 smithi050 ceph-mon[32371]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-18T08:21:41.807 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:21:41 smithi050 ceph-mon[32371]: 6.13 scrub starts 2024-09-18T08:21:41.808 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:21:41 smithi050 ceph-mon[32371]: 6.13 scrub ok 2024-09-18T08:21:41.808 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:21:41 smithi050 ceph-mon[32371]: from='client.? 172.21.15.7:0/1744242323' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-18T08:21:41.808 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:21:41 smithi050 ceph-mon[32371]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-18T08:21:41.854 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:21:41 smithi007 ceph-mon[39249]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-18T08:21:41.854 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:21:41 smithi007 ceph-mon[39249]: 6.13 scrub starts 2024-09-18T08:21:41.854 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:21:41 smithi007 ceph-mon[39249]: 6.13 scrub ok 2024-09-18T08:21:41.854 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:21:41 smithi007 ceph-mon[39249]: from='client.? 172.21.15.7:0/1744242323' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-18T08:21:41.854 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:21:41 smithi007 ceph-mon[39249]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-18T08:21:41.854 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:21:41 smithi007 ceph-mon[29459]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-18T08:21:41.854 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:21:41 smithi007 ceph-mon[29459]: 6.13 scrub starts 2024-09-18T08:21:41.855 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:21:41 smithi007 ceph-mon[29459]: 6.13 scrub ok 2024-09-18T08:21:41.855 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:21:41 smithi007 ceph-mon[29459]: from='client.? 172.21.15.7:0/1744242323' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-18T08:21:41.855 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:21:41 smithi007 ceph-mon[29459]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-18T08:21:42.902 INFO:tasks.thrashosds.thrasher:inning osd 2024-09-18T08:21:42.902 INFO:tasks.thrashosds.thrasher:Adding osd 4 2024-09-18T08:21:42.902 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid 779349ca-7593-11ef-bceb-c7b262605968 -- ceph osd in 4 2024-09-18T08:21:42.905 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:21:42 smithi050 ceph-mon[32371]: 5.e scrub starts 2024-09-18T08:21:42.905 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:21:42 smithi050 ceph-mon[32371]: 5.e scrub ok 2024-09-18T08:21:42.905 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:21:42 smithi050 ceph-mon[32371]: pgmap v939: 142 pgs: 142 active+clean; 583 KiB data, 4.7 GiB used, 621 GiB / 626 GiB avail; 640 B/s rd, 0 op/s 2024-09-18T08:21:42.905 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:21:42 smithi050 ceph-mon[32371]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-18T08:21:42.905 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:21:42 smithi050 ceph-mon[32371]: Cluster is now healthy 2024-09-18T08:21:42.905 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:21:42 smithi050 ceph-mon[32371]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-18T08:21:42.905 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:21:42 smithi050 ceph-mon[32371]: osdmap e458: 8 total, 8 up, 7 in 2024-09-18T08:21:42.905 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:21:42 smithi050 ceph-mon[32371]: 3.15 scrub starts 2024-09-18T08:21:42.905 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:21:42 smithi050 ceph-mon[32371]: 3.15 scrub ok 2024-09-18T08:21:42.932 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:21:42 smithi007 ceph-mon[29459]: 5.e scrub starts 2024-09-18T08:21:42.932 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:21:42 smithi007 ceph-mon[29459]: 5.e scrub ok 2024-09-18T08:21:42.932 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:21:42 smithi007 ceph-mon[29459]: pgmap v939: 142 pgs: 142 active+clean; 583 KiB data, 4.7 GiB used, 621 GiB / 626 GiB avail; 640 B/s rd, 0 op/s 2024-09-18T08:21:42.932 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:21:42 smithi007 ceph-mon[29459]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-18T08:21:42.932 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:21:42 smithi007 ceph-mon[29459]: Cluster is now healthy 2024-09-18T08:21:42.932 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:21:42 smithi007 ceph-mon[29459]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-18T08:21:42.933 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:21:42 smithi007 ceph-mon[29459]: osdmap e458: 8 total, 8 up, 7 in 2024-09-18T08:21:42.933 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:21:42 smithi007 ceph-mon[29459]: 3.15 scrub starts 2024-09-18T08:21:42.933 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:21:42 smithi007 ceph-mon[29459]: 3.15 scrub ok 2024-09-18T08:21:42.933 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:21:42 smithi007 ceph-mon[39249]: 5.e scrub starts 2024-09-18T08:21:42.933 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:21:42 smithi007 ceph-mon[39249]: 5.e scrub ok 2024-09-18T08:21:42.933 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:21:42 smithi007 ceph-mon[39249]: pgmap v939: 142 pgs: 142 active+clean; 583 KiB data, 4.7 GiB used, 621 GiB / 626 GiB avail; 640 B/s rd, 0 op/s 2024-09-18T08:21:42.933 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:21:42 smithi007 ceph-mon[39249]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-18T08:21:42.933 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:21:42 smithi007 ceph-mon[39249]: Cluster is now healthy 2024-09-18T08:21:42.933 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:21:42 smithi007 ceph-mon[39249]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-18T08:21:42.933 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:21:42 smithi007 ceph-mon[39249]: osdmap e458: 8 total, 8 up, 7 in 2024-09-18T08:21:42.933 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:21:42 smithi007 ceph-mon[39249]: 3.15 scrub starts 2024-09-18T08:21:42.934 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:21:42 smithi007 ceph-mon[39249]: 3.15 scrub ok 2024-09-18T08:21:43.178 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/779349ca-7593-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T08:21:43.873 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:21:43 smithi007 ceph-mon[29459]: 5.f scrub starts 2024-09-18T08:21:43.873 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:21:43 smithi007 ceph-mon[29459]: 5.f scrub ok 2024-09-18T08:21:43.873 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:21:43 smithi007 ceph-mon[29459]: 4.12 scrub starts 2024-09-18T08:21:43.873 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:21:43 smithi007 ceph-mon[29459]: 4.12 scrub ok 2024-09-18T08:21:43.873 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:21:43 smithi007 ceph-mon[29459]: 4.6 scrub starts 2024-09-18T08:21:43.873 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:21:43 smithi007 ceph-mon[29459]: 4.6 scrub ok 2024-09-18T08:21:43.873 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:21:43 smithi007 ceph-mon[29459]: 4.0 scrub starts 2024-09-18T08:21:43.873 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:21:43 smithi007 ceph-mon[29459]: 4.0 scrub ok 2024-09-18T08:21:43.874 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:21:43 smithi007 ceph-mon[39249]: 5.f scrub starts 2024-09-18T08:21:43.874 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:21:43 smithi007 ceph-mon[39249]: 5.f scrub ok 2024-09-18T08:21:43.874 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:21:43 smithi007 ceph-mon[39249]: 4.12 scrub starts 2024-09-18T08:21:43.874 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:21:43 smithi007 ceph-mon[39249]: 4.12 scrub ok 2024-09-18T08:21:43.874 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:21:43 smithi007 ceph-mon[39249]: 4.6 scrub starts 2024-09-18T08:21:43.874 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:21:43 smithi007 ceph-mon[39249]: 4.6 scrub ok 2024-09-18T08:21:43.874 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:21:43 smithi007 ceph-mon[39249]: 4.0 scrub starts 2024-09-18T08:21:43.874 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:21:43 smithi007 ceph-mon[39249]: 4.0 scrub ok 2024-09-18T08:21:43.905 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:21:43 smithi050 ceph-mon[32371]: 5.f scrub starts 2024-09-18T08:21:43.905 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:21:43 smithi050 ceph-mon[32371]: 5.f scrub ok 2024-09-18T08:21:43.905 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:21:43 smithi050 ceph-mon[32371]: 4.12 scrub starts 2024-09-18T08:21:43.905 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:21:43 smithi050 ceph-mon[32371]: 4.12 scrub ok 2024-09-18T08:21:43.905 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:21:43 smithi050 ceph-mon[32371]: 4.6 scrub starts 2024-09-18T08:21:43.905 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:21:43 smithi050 ceph-mon[32371]: 4.6 scrub ok 2024-09-18T08:21:43.905 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:21:43 smithi050 ceph-mon[32371]: 4.0 scrub starts 2024-09-18T08:21:43.906 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:21:43 smithi050 ceph-mon[32371]: 4.0 scrub ok 2024-09-18T08:21:44.229 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:21:43 smithi007 ceph-779349ca-7593-11ef-bceb-c7b262605968-mon-a[29455]: 2024-09-18T08:21:43.870+0000 7fcd2b595640 -1 mon.a@0(leader).osd e458 definitely_dead 0 2024-09-18T08:21:44.559 INFO:teuthology.orchestra.run.smithi007.stderr:marked in osd.4. 2024-09-18T08:21:44.905 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:21:44 smithi050 ceph-mon[32371]: 5.1f scrub starts 2024-09-18T08:21:44.905 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:21:44 smithi050 ceph-mon[32371]: 5.1f scrub ok 2024-09-18T08:21:44.905 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:21:44 smithi050 ceph-mon[32371]: pgmap v941: 142 pgs: 142 active+clean; 583 KiB data, 4.7 GiB used, 621 GiB / 626 GiB avail; 1.7 KiB/s rd, 1 op/s 2024-09-18T08:21:44.905 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:21:44 smithi050 ceph-mon[32371]: 5.c scrub starts 2024-09-18T08:21:44.905 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:21:44 smithi050 ceph-mon[32371]: 5.c scrub ok 2024-09-18T08:21:44.905 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:21:44 smithi050 ceph-mon[32371]: from='client.? 172.21.15.7:0/1146964817' entity='client.admin' cmd={"prefix": "osd in", "ids": ["4"]} : dispatch 2024-09-18T08:21:44.946 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:21:44 smithi007 ceph-mon[29459]: 5.1f scrub starts 2024-09-18T08:21:44.947 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:21:44 smithi007 ceph-mon[29459]: 5.1f scrub ok 2024-09-18T08:21:44.947 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:21:44 smithi007 ceph-mon[29459]: pgmap v941: 142 pgs: 142 active+clean; 583 KiB data, 4.7 GiB used, 621 GiB / 626 GiB avail; 1.7 KiB/s rd, 1 op/s 2024-09-18T08:21:44.947 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:21:44 smithi007 ceph-mon[29459]: 5.c scrub starts 2024-09-18T08:21:44.947 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:21:44 smithi007 ceph-mon[29459]: 5.c scrub ok 2024-09-18T08:21:44.947 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:21:44 smithi007 ceph-mon[29459]: from='client.? 172.21.15.7:0/1146964817' entity='client.admin' cmd={"prefix": "osd in", "ids": ["4"]} : dispatch 2024-09-18T08:21:44.947 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:21:44 smithi007 ceph-mon[39249]: 5.1f scrub starts 2024-09-18T08:21:44.947 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:21:44 smithi007 ceph-mon[39249]: 5.1f scrub ok 2024-09-18T08:21:44.947 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:21:44 smithi007 ceph-mon[39249]: pgmap v941: 142 pgs: 142 active+clean; 583 KiB data, 4.7 GiB used, 621 GiB / 626 GiB avail; 1.7 KiB/s rd, 1 op/s 2024-09-18T08:21:44.947 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:21:44 smithi007 ceph-mon[39249]: 5.c scrub starts 2024-09-18T08:21:44.947 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:21:44 smithi007 ceph-mon[39249]: 5.c scrub ok 2024-09-18T08:21:44.948 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:21:44 smithi007 ceph-mon[39249]: from='client.? 172.21.15.7:0/1146964817' entity='client.admin' cmd={"prefix": "osd in", "ids": ["4"]} : dispatch 2024-09-18T08:21:45.399 INFO:tasks.thrashosds.thrasher:Added osd 4 2024-09-18T08:21:45.400 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid 779349ca-7593-11ef-bceb-c7b262605968 -- ceph -- tell 'mon.*' injectargs --mon_osd_down_out_interval=600 2024-09-18T08:21:45.729 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:21:45 smithi007 ceph-mon[29459]: 4.15 scrub starts 2024-09-18T08:21:45.730 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:21:45 smithi007 ceph-mon[29459]: 4.15 scrub ok 2024-09-18T08:21:45.730 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:21:45 smithi007 ceph-mon[29459]: from='client.? 172.21.15.7:0/1146964817' entity='client.admin' cmd='[{"prefix": "osd in", "ids": ["4"]}]': finished 2024-09-18T08:21:45.730 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:21:45 smithi007 ceph-mon[29459]: osdmap e459: 8 total, 8 up, 8 in 2024-09-18T08:21:45.730 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:21:45 smithi007 ceph-mon[29459]: 3.4 scrub starts 2024-09-18T08:21:45.730 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:21:45 smithi007 ceph-mon[29459]: 3.4 scrub ok 2024-09-18T08:21:45.730 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:21:45 smithi007 ceph-mon[29459]: osdmap e460: 8 total, 8 up, 8 in 2024-09-18T08:21:45.730 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:21:45 smithi007 ceph-mon[39249]: 4.15 scrub starts 2024-09-18T08:21:45.730 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:21:45 smithi007 ceph-mon[39249]: 4.15 scrub ok 2024-09-18T08:21:45.730 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:21:45 smithi007 ceph-mon[39249]: from='client.? 172.21.15.7:0/1146964817' entity='client.admin' cmd='[{"prefix": "osd in", "ids": ["4"]}]': finished 2024-09-18T08:21:45.731 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:21:45 smithi007 ceph-mon[39249]: osdmap e459: 8 total, 8 up, 8 in 2024-09-18T08:21:45.731 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:21:45 smithi007 ceph-mon[39249]: 3.4 scrub starts 2024-09-18T08:21:45.731 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:21:45 smithi007 ceph-mon[39249]: 3.4 scrub ok 2024-09-18T08:21:45.731 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:21:45 smithi007 ceph-mon[39249]: osdmap e460: 8 total, 8 up, 8 in 2024-09-18T08:21:45.735 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/779349ca-7593-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T08:21:45.905 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:21:45 smithi050 ceph-mon[32371]: 4.15 scrub starts 2024-09-18T08:21:45.905 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:21:45 smithi050 ceph-mon[32371]: 4.15 scrub ok 2024-09-18T08:21:45.905 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:21:45 smithi050 ceph-mon[32371]: from='client.? 172.21.15.7:0/1146964817' entity='client.admin' cmd='[{"prefix": "osd in", "ids": ["4"]}]': finished 2024-09-18T08:21:45.905 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:21:45 smithi050 ceph-mon[32371]: osdmap e459: 8 total, 8 up, 8 in 2024-09-18T08:21:45.905 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:21:45 smithi050 ceph-mon[32371]: 3.4 scrub starts 2024-09-18T08:21:45.905 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:21:45 smithi050 ceph-mon[32371]: 3.4 scrub ok 2024-09-18T08:21:45.905 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:21:45 smithi050 ceph-mon[32371]: osdmap e460: 8 total, 8 up, 8 in 2024-09-18T08:21:46.267 INFO:teuthology.orchestra.run.smithi007.stdout:mon.a: {} 2024-09-18T08:21:46.268 INFO:teuthology.orchestra.run.smithi007.stdout:mon.b: {} 2024-09-18T08:21:46.268 INFO:teuthology.orchestra.run.smithi007.stdout:mon.c: {} 2024-09-18T08:21:46.268 INFO:teuthology.orchestra.run.smithi007.stderr:mon.a: mon_osd_down_out_interval = '' (not observed, change may require restart) 2024-09-18T08:21:46.268 INFO:teuthology.orchestra.run.smithi007.stderr:mon.b: mon_osd_down_out_interval = '' (not observed, change may require restart) 2024-09-18T08:21:46.268 INFO:teuthology.orchestra.run.smithi007.stderr:mon.c: mon_osd_down_out_interval = '' (not observed, change may require restart) 2024-09-18T08:21:46.880 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:21:46 smithi007 ceph-mon[29459]: 5.4 scrub starts 2024-09-18T08:21:46.881 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:21:46 smithi007 ceph-mon[29459]: 5.4 scrub ok 2024-09-18T08:21:46.881 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:21:46 smithi007 ceph-mon[29459]: pgmap v943: 142 pgs: 5 peering, 137 active+clean; 583 KiB data, 4.7 GiB used, 621 GiB / 626 GiB avail; 853 B/s rd, 0 op/s 2024-09-18T08:21:46.881 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:21:46 smithi007 ceph-mon[29459]: 1.2 scrub starts 2024-09-18T08:21:46.881 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:21:46 smithi007 ceph-mon[29459]: 1.2 scrub ok 2024-09-18T08:21:46.881 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:21:46 smithi007 ceph-mon[29459]: from='client.? 172.21.15.7:0/849758963' entity='client.admin' cmd={"prefix": "mon dump", "format": "json"} : dispatch 2024-09-18T08:21:46.881 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Sep 18 08:21:46 smithi007 ceph-779349ca-7593-11ef-bceb-c7b262605968-mgr-y[29688]: ::ffff:172.21.15.50 - - [18/Sep/2024:08:21:46] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-09-18T08:21:46.882 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:21:46 smithi007 ceph-mon[39249]: 5.4 scrub starts 2024-09-18T08:21:46.882 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:21:46 smithi007 ceph-mon[39249]: 5.4 scrub ok 2024-09-18T08:21:46.882 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:21:46 smithi007 ceph-mon[39249]: pgmap v943: 142 pgs: 5 peering, 137 active+clean; 583 KiB data, 4.7 GiB used, 621 GiB / 626 GiB avail; 853 B/s rd, 0 op/s 2024-09-18T08:21:46.882 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:21:46 smithi007 ceph-mon[39249]: 1.2 scrub starts 2024-09-18T08:21:46.882 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:21:46 smithi007 ceph-mon[39249]: 1.2 scrub ok 2024-09-18T08:21:46.882 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:21:46 smithi007 ceph-mon[39249]: from='client.? 172.21.15.7:0/849758963' entity='client.admin' cmd={"prefix": "mon dump", "format": "json"} : dispatch 2024-09-18T08:21:46.905 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:21:46 smithi050 ceph-mon[32371]: 5.4 scrub starts 2024-09-18T08:21:46.905 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:21:46 smithi050 ceph-mon[32371]: 5.4 scrub ok 2024-09-18T08:21:46.905 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:21:46 smithi050 ceph-mon[32371]: pgmap v943: 142 pgs: 5 peering, 137 active+clean; 583 KiB data, 4.7 GiB used, 621 GiB / 626 GiB avail; 853 B/s rd, 0 op/s 2024-09-18T08:21:46.905 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:21:46 smithi050 ceph-mon[32371]: 1.2 scrub starts 2024-09-18T08:21:46.905 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:21:46 smithi050 ceph-mon[32371]: 1.2 scrub ok 2024-09-18T08:21:46.905 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:21:46 smithi050 ceph-mon[32371]: from='client.? 172.21.15.7:0/849758963' entity='client.admin' cmd={"prefix": "mon dump", "format": "json"} : dispatch 2024-09-18T08:21:46.941 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid 779349ca-7593-11ef-bceb-c7b262605968 -- ceph osd reweight 0 1 2024-09-18T08:21:47.208 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/779349ca-7593-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T08:21:47.905 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:21:47 smithi050 ceph-mon[32371]: 3.13 scrub starts 2024-09-18T08:21:47.909 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:21:47 smithi050 ceph-mon[32371]: 3.13 scrub ok 2024-09-18T08:21:47.909 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:21:47 smithi050 ceph-mon[32371]: 4.7 scrub starts 2024-09-18T08:21:47.909 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:21:47 smithi050 ceph-mon[32371]: 4.7 scrub ok 2024-09-18T08:21:47.909 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:21:47 smithi050 ceph-mon[32371]: 3.0 scrub starts 2024-09-18T08:21:47.909 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:21:47 smithi050 ceph-mon[32371]: 3.0 scrub ok 2024-09-18T08:21:47.980 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:21:47 smithi007 ceph-mon[29459]: 3.13 scrub starts 2024-09-18T08:21:47.980 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:21:47 smithi007 ceph-mon[29459]: 3.13 scrub ok 2024-09-18T08:21:47.980 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:21:47 smithi007 ceph-mon[29459]: 4.7 scrub starts 2024-09-18T08:21:47.980 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:21:47 smithi007 ceph-mon[29459]: 4.7 scrub ok 2024-09-18T08:21:47.980 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:21:47 smithi007 ceph-mon[29459]: 3.0 scrub starts 2024-09-18T08:21:47.980 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:21:47 smithi007 ceph-mon[29459]: 3.0 scrub ok 2024-09-18T08:21:47.980 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:21:47 smithi007 ceph-mon[39249]: 3.13 scrub starts 2024-09-18T08:21:47.980 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:21:47 smithi007 ceph-mon[39249]: 3.13 scrub ok 2024-09-18T08:21:47.981 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:21:47 smithi007 ceph-mon[39249]: 4.7 scrub starts 2024-09-18T08:21:47.981 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:21:47 smithi007 ceph-mon[39249]: 4.7 scrub ok 2024-09-18T08:21:47.981 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:21:47 smithi007 ceph-mon[39249]: 3.0 scrub starts 2024-09-18T08:21:47.981 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:21:47 smithi007 ceph-mon[39249]: 3.0 scrub ok 2024-09-18T08:21:48.601 INFO:teuthology.orchestra.run.smithi007.stderr:reweighted osd.0 to 1 (10000) 2024-09-18T08:21:48.905 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:21:48 smithi050 ceph-mon[32371]: 4.1d scrub starts 2024-09-18T08:21:48.905 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:21:48 smithi050 ceph-mon[32371]: 4.1d scrub ok 2024-09-18T08:21:48.905 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:21:48 smithi050 ceph-mon[32371]: pgmap v945: 142 pgs: 5 peering, 137 active+clean; 583 KiB data, 4.7 GiB used, 621 GiB / 626 GiB avail; 853 B/s rd, 0 op/s 2024-09-18T08:21:48.905 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:21:48 smithi050 ceph-mon[32371]: purged_snaps scrub starts 2024-09-18T08:21:48.905 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:21:48 smithi050 ceph-mon[32371]: purged_snaps scrub ok 2024-09-18T08:21:48.905 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:21:48 smithi050 ceph-mon[32371]: from='client.? 172.21.15.7:0/3751493764' entity='client.admin' cmd={"prefix": "osd reweight", "id": 0, "weight": 1.0} : dispatch 2024-09-18T08:21:48.905 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:21:48 smithi050 ceph-mon[32371]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 0, "weight": 1.0} : dispatch 2024-09-18T08:21:48.979 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:21:48 smithi007 ceph-mon[29459]: 4.1d scrub starts 2024-09-18T08:21:48.980 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:21:48 smithi007 ceph-mon[29459]: 4.1d scrub ok 2024-09-18T08:21:48.980 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:21:48 smithi007 ceph-mon[29459]: pgmap v945: 142 pgs: 5 peering, 137 active+clean; 583 KiB data, 4.7 GiB used, 621 GiB / 626 GiB avail; 853 B/s rd, 0 op/s 2024-09-18T08:21:48.980 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:21:48 smithi007 ceph-mon[29459]: purged_snaps scrub starts 2024-09-18T08:21:48.980 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:21:48 smithi007 ceph-mon[29459]: purged_snaps scrub ok 2024-09-18T08:21:48.980 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:21:48 smithi007 ceph-mon[29459]: from='client.? 172.21.15.7:0/3751493764' entity='client.admin' cmd={"prefix": "osd reweight", "id": 0, "weight": 1.0} : dispatch 2024-09-18T08:21:48.980 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:21:48 smithi007 ceph-mon[29459]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 0, "weight": 1.0} : dispatch 2024-09-18T08:21:48.980 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:21:48 smithi007 ceph-mon[39249]: 4.1d scrub starts 2024-09-18T08:21:48.980 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:21:48 smithi007 ceph-mon[39249]: 4.1d scrub ok 2024-09-18T08:21:48.980 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:21:48 smithi007 ceph-mon[39249]: pgmap v945: 142 pgs: 5 peering, 137 active+clean; 583 KiB data, 4.7 GiB used, 621 GiB / 626 GiB avail; 853 B/s rd, 0 op/s 2024-09-18T08:21:48.981 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:21:48 smithi007 ceph-mon[39249]: purged_snaps scrub starts 2024-09-18T08:21:48.981 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:21:48 smithi007 ceph-mon[39249]: purged_snaps scrub ok 2024-09-18T08:21:48.981 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:21:48 smithi007 ceph-mon[39249]: from='client.? 172.21.15.7:0/3751493764' entity='client.admin' cmd={"prefix": "osd reweight", "id": 0, "weight": 1.0} : dispatch 2024-09-18T08:21:48.981 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:21:48 smithi007 ceph-mon[39249]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 0, "weight": 1.0} : dispatch 2024-09-18T08:21:49.348 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid 779349ca-7593-11ef-bceb-c7b262605968 -- ceph osd primary-affinity 0 1 2024-09-18T08:21:49.620 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/779349ca-7593-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T08:21:49.864 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:21:49 smithi050 ceph-mon[32371]: 3.f scrub starts 2024-09-18T08:21:49.864 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:21:49 smithi050 ceph-mon[32371]: 3.f scrub ok 2024-09-18T08:21:49.864 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:21:49 smithi050 ceph-mon[32371]: 5.6 scrub starts 2024-09-18T08:21:49.864 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:21:49 smithi050 ceph-mon[32371]: 5.6 scrub ok 2024-09-18T08:21:49.864 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:21:49 smithi050 ceph-mon[32371]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]': finished 2024-09-18T08:21:49.864 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:21:49 smithi050 ceph-mon[32371]: osdmap e461: 8 total, 8 up, 8 in 2024-09-18T08:21:49.865 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:21:49 smithi050 ceph-mon[32371]: from='mgr.24470 172.21.15.7:0/2594390261' entity='mgr.y' 2024-09-18T08:21:49.865 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:21:49 smithi050 ceph-mon[32371]: from='mgr.24470 172.21.15.7:0/2594390261' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-18T08:21:49.878 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:21:49 smithi007 ceph-mon[29459]: 3.f scrub starts 2024-09-18T08:21:49.879 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:21:49 smithi007 ceph-mon[29459]: 3.f scrub ok 2024-09-18T08:21:49.879 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:21:49 smithi007 ceph-mon[29459]: 5.6 scrub starts 2024-09-18T08:21:49.879 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:21:49 smithi007 ceph-mon[29459]: 5.6 scrub ok 2024-09-18T08:21:49.879 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:21:49 smithi007 ceph-mon[29459]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]': finished 2024-09-18T08:21:49.879 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:21:49 smithi007 ceph-mon[29459]: osdmap e461: 8 total, 8 up, 8 in 2024-09-18T08:21:49.879 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:21:49 smithi007 ceph-mon[29459]: from='mgr.24470 172.21.15.7:0/2594390261' entity='mgr.y' 2024-09-18T08:21:49.879 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:21:49 smithi007 ceph-mon[29459]: from='mgr.24470 172.21.15.7:0/2594390261' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-18T08:21:49.879 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:21:49 smithi007 ceph-mon[39249]: 3.f scrub starts 2024-09-18T08:21:49.879 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:21:49 smithi007 ceph-mon[39249]: 3.f scrub ok 2024-09-18T08:21:49.879 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:21:49 smithi007 ceph-mon[39249]: 5.6 scrub starts 2024-09-18T08:21:49.879 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:21:49 smithi007 ceph-mon[39249]: 5.6 scrub ok 2024-09-18T08:21:49.880 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:21:49 smithi007 ceph-mon[39249]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]': finished 2024-09-18T08:21:49.880 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:21:49 smithi007 ceph-mon[39249]: osdmap e461: 8 total, 8 up, 8 in 2024-09-18T08:21:49.880 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:21:49 smithi007 ceph-mon[39249]: from='mgr.24470 172.21.15.7:0/2594390261' entity='mgr.y' 2024-09-18T08:21:49.880 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:21:49 smithi007 ceph-mon[39249]: from='mgr.24470 172.21.15.7:0/2594390261' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-18T08:21:50.155 INFO:journalctl@ceph.iscsi.iscsi.a.smithi050.stdout:Sep 18 08:21:49 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-iscsi-iscsi-a[57782]: debug there is no tcmu-runner data available 2024-09-18T08:21:50.603 INFO:teuthology.orchestra.run.smithi007.stderr:set osd.0 primary-affinity to 1 (8655362) 2024-09-18T08:21:50.730 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:21:50 smithi007 ceph-mon[39249]: 3.8 scrub starts 2024-09-18T08:21:50.730 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:21:50 smithi007 ceph-mon[39249]: 3.8 scrub ok 2024-09-18T08:21:50.730 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:21:50 smithi007 ceph-mon[39249]: pgmap v947: 142 pgs: 5 peering, 137 active+clean; 583 KiB data, 5.3 GiB used, 710 GiB / 715 GiB avail; 853 B/s rd, 0 op/s; 324 B/s, 6 objects/s recovering 2024-09-18T08:21:50.730 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:21:50 smithi007 ceph-mon[39249]: 2.0 scrub starts 2024-09-18T08:21:50.730 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:21:50 smithi007 ceph-mon[39249]: 2.0 scrub ok 2024-09-18T08:21:50.731 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:21:50 smithi007 ceph-mon[39249]: 3.1 scrub starts 2024-09-18T08:21:50.731 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:21:50 smithi007 ceph-mon[39249]: 6.1b scrub starts 2024-09-18T08:21:50.731 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:21:50 smithi007 ceph-mon[39249]: 3.1 scrub ok 2024-09-18T08:21:50.731 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:21:50 smithi007 ceph-mon[39249]: 6.1b scrub ok 2024-09-18T08:21:50.731 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:21:50 smithi007 ceph-mon[39249]: 4.c scrub starts 2024-09-18T08:21:50.731 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:21:50 smithi007 ceph-mon[39249]: 4.c scrub ok 2024-09-18T08:21:50.731 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:21:50 smithi007 ceph-mon[39249]: from='client.? 172.21.15.7:0/3575924739' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 0, "weight": 1.0} : dispatch 2024-09-18T08:21:50.731 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:21:50 smithi007 ceph-mon[39249]: from='client.? ' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 0, "weight": 1.0} : dispatch 2024-09-18T08:21:50.731 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:21:50 smithi007 ceph-mon[29459]: 3.8 scrub starts 2024-09-18T08:21:50.731 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:21:50 smithi007 ceph-mon[29459]: 3.8 scrub ok 2024-09-18T08:21:50.731 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:21:50 smithi007 ceph-mon[29459]: pgmap v947: 142 pgs: 5 peering, 137 active+clean; 583 KiB data, 5.3 GiB used, 710 GiB / 715 GiB avail; 853 B/s rd, 0 op/s; 324 B/s, 6 objects/s recovering 2024-09-18T08:21:50.732 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:21:50 smithi007 ceph-mon[29459]: 2.0 scrub starts 2024-09-18T08:21:50.732 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:21:50 smithi007 ceph-mon[29459]: 2.0 scrub ok 2024-09-18T08:21:50.732 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:21:50 smithi007 ceph-mon[29459]: 3.1 scrub starts 2024-09-18T08:21:50.732 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:21:50 smithi007 ceph-mon[29459]: 6.1b scrub starts 2024-09-18T08:21:50.732 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:21:50 smithi007 ceph-mon[29459]: 3.1 scrub ok 2024-09-18T08:21:50.732 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:21:50 smithi007 ceph-mon[29459]: 6.1b scrub ok 2024-09-18T08:21:50.732 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:21:50 smithi007 ceph-mon[29459]: 4.c scrub starts 2024-09-18T08:21:50.732 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:21:50 smithi007 ceph-mon[29459]: 4.c scrub ok 2024-09-18T08:21:50.732 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:21:50 smithi007 ceph-mon[29459]: from='client.? 172.21.15.7:0/3575924739' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 0, "weight": 1.0} : dispatch 2024-09-18T08:21:50.732 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:21:50 smithi007 ceph-mon[29459]: from='client.? ' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 0, "weight": 1.0} : dispatch 2024-09-18T08:21:50.905 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:21:50 smithi050 ceph-mon[32371]: 3.8 scrub starts 2024-09-18T08:21:50.905 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:21:50 smithi050 ceph-mon[32371]: 3.8 scrub ok 2024-09-18T08:21:50.905 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:21:50 smithi050 ceph-mon[32371]: pgmap v947: 142 pgs: 5 peering, 137 active+clean; 583 KiB data, 5.3 GiB used, 710 GiB / 715 GiB avail; 853 B/s rd, 0 op/s; 324 B/s, 6 objects/s recovering 2024-09-18T08:21:50.905 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:21:50 smithi050 ceph-mon[32371]: 2.0 scrub starts 2024-09-18T08:21:50.905 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:21:50 smithi050 ceph-mon[32371]: 2.0 scrub ok 2024-09-18T08:21:50.905 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:21:50 smithi050 ceph-mon[32371]: 3.1 scrub starts 2024-09-18T08:21:50.905 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:21:50 smithi050 ceph-mon[32371]: 6.1b scrub starts 2024-09-18T08:21:50.905 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:21:50 smithi050 ceph-mon[32371]: 3.1 scrub ok 2024-09-18T08:21:50.906 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:21:50 smithi050 ceph-mon[32371]: 6.1b scrub ok 2024-09-18T08:21:50.906 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:21:50 smithi050 ceph-mon[32371]: 4.c scrub starts 2024-09-18T08:21:50.906 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:21:50 smithi050 ceph-mon[32371]: 4.c scrub ok 2024-09-18T08:21:50.906 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:21:50 smithi050 ceph-mon[32371]: from='client.? 172.21.15.7:0/3575924739' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 0, "weight": 1.0} : dispatch 2024-09-18T08:21:50.906 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:21:50 smithi050 ceph-mon[32371]: from='client.? ' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 0, "weight": 1.0} : dispatch 2024-09-18T08:21:51.464 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid 779349ca-7593-11ef-bceb-c7b262605968 -- ceph osd reweight 2 1 2024-09-18T08:21:51.729 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:21:51 smithi007 ceph-mon[29459]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-18T08:21:51.730 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:21:51 smithi007 ceph-mon[29459]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 0, "weight": 1.0}]': finished 2024-09-18T08:21:51.730 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:21:51 smithi007 ceph-mon[29459]: osdmap e462: 8 total, 8 up, 8 in 2024-09-18T08:21:51.730 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:21:51 smithi007 ceph-mon[29459]: 3.19 scrub starts 2024-09-18T08:21:51.730 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:21:51 smithi007 ceph-mon[29459]: 3.19 scrub ok 2024-09-18T08:21:51.730 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:21:51 smithi007 ceph-mon[29459]: 5.b scrub starts 2024-09-18T08:21:51.730 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:21:51 smithi007 ceph-mon[29459]: 3.6 scrub starts 2024-09-18T08:21:51.731 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:21:51 smithi007 ceph-mon[29459]: 5.b scrub ok 2024-09-18T08:21:51.731 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:21:51 smithi007 ceph-mon[29459]: 3.6 scrub ok 2024-09-18T08:21:51.731 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:21:51 smithi007 ceph-mon[39249]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-18T08:21:51.731 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:21:51 smithi007 ceph-mon[39249]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 0, "weight": 1.0}]': finished 2024-09-18T08:21:51.731 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:21:51 smithi007 ceph-mon[39249]: osdmap e462: 8 total, 8 up, 8 in 2024-09-18T08:21:51.731 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:21:51 smithi007 ceph-mon[39249]: 3.19 scrub starts 2024-09-18T08:21:51.732 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:21:51 smithi007 ceph-mon[39249]: 3.19 scrub ok 2024-09-18T08:21:51.732 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:21:51 smithi007 ceph-mon[39249]: 5.b scrub starts 2024-09-18T08:21:51.732 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:21:51 smithi007 ceph-mon[39249]: 3.6 scrub starts 2024-09-18T08:21:51.732 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:21:51 smithi007 ceph-mon[39249]: 5.b scrub ok 2024-09-18T08:21:51.732 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:21:51 smithi007 ceph-mon[39249]: 3.6 scrub ok 2024-09-18T08:21:51.732 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/779349ca-7593-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T08:21:51.905 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:21:51 smithi050 ceph-mon[32371]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-18T08:21:51.905 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:21:51 smithi050 ceph-mon[32371]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 0, "weight": 1.0}]': finished 2024-09-18T08:21:51.905 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:21:51 smithi050 ceph-mon[32371]: osdmap e462: 8 total, 8 up, 8 in 2024-09-18T08:21:51.906 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:21:51 smithi050 ceph-mon[32371]: 3.19 scrub starts 2024-09-18T08:21:51.906 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:21:51 smithi050 ceph-mon[32371]: 3.19 scrub ok 2024-09-18T08:21:51.906 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:21:51 smithi050 ceph-mon[32371]: 5.b scrub starts 2024-09-18T08:21:51.906 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:21:51 smithi050 ceph-mon[32371]: 3.6 scrub starts 2024-09-18T08:21:51.906 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:21:51 smithi050 ceph-mon[32371]: 5.b scrub ok 2024-09-18T08:21:51.907 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:21:51 smithi050 ceph-mon[32371]: 3.6 scrub ok 2024-09-18T08:21:52.610 INFO:teuthology.orchestra.run.smithi007.stderr:reweighted osd.2 to 1 (10000) 2024-09-18T08:21:52.905 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:21:52 smithi050 ceph-mon[32371]: pgmap v949: 142 pgs: 142 active+clean; 583 KiB data, 5.3 GiB used, 710 GiB / 715 GiB avail; 853 B/s rd, 0 op/s; 324 B/s, 6 objects/s recovering 2024-09-18T08:21:52.905 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:21:52 smithi050 ceph-mon[32371]: 3.17 scrub starts 2024-09-18T08:21:52.905 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:21:52 smithi050 ceph-mon[32371]: 3.17 scrub ok 2024-09-18T08:21:52.905 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:21:52 smithi050 ceph-mon[32371]: from='client.? 172.21.15.7:0/1252441775' entity='client.admin' cmd={"prefix": "osd reweight", "id": 2, "weight": 1.0} : dispatch 2024-09-18T08:21:52.979 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:21:52 smithi007 ceph-mon[29459]: pgmap v949: 142 pgs: 142 active+clean; 583 KiB data, 5.3 GiB used, 710 GiB / 715 GiB avail; 853 B/s rd, 0 op/s; 324 B/s, 6 objects/s recovering 2024-09-18T08:21:52.979 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:21:52 smithi007 ceph-mon[29459]: 3.17 scrub starts 2024-09-18T08:21:52.980 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:21:52 smithi007 ceph-mon[29459]: 3.17 scrub ok 2024-09-18T08:21:52.980 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:21:52 smithi007 ceph-mon[29459]: from='client.? 172.21.15.7:0/1252441775' entity='client.admin' cmd={"prefix": "osd reweight", "id": 2, "weight": 1.0} : dispatch 2024-09-18T08:21:52.980 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:21:52 smithi007 ceph-mon[39249]: pgmap v949: 142 pgs: 142 active+clean; 583 KiB data, 5.3 GiB used, 710 GiB / 715 GiB avail; 853 B/s rd, 0 op/s; 324 B/s, 6 objects/s recovering 2024-09-18T08:21:52.980 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:21:52 smithi007 ceph-mon[39249]: 3.17 scrub starts 2024-09-18T08:21:52.980 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:21:52 smithi007 ceph-mon[39249]: 3.17 scrub ok 2024-09-18T08:21:52.980 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:21:52 smithi007 ceph-mon[39249]: from='client.? 172.21.15.7:0/1252441775' entity='client.admin' cmd={"prefix": "osd reweight", "id": 2, "weight": 1.0} : dispatch 2024-09-18T08:21:53.410 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid 779349ca-7593-11ef-bceb-c7b262605968 -- ceph osd primary-affinity 2 1 2024-09-18T08:21:53.706 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/779349ca-7593-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T08:21:53.905 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:21:53 smithi050 ceph-mon[32371]: 3.c scrub starts 2024-09-18T08:21:53.905 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:21:53 smithi050 ceph-mon[32371]: 3.c scrub ok 2024-09-18T08:21:53.905 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:21:53 smithi050 ceph-mon[32371]: 2.2 scrub starts 2024-09-18T08:21:53.905 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:21:53 smithi050 ceph-mon[32371]: 2.2 scrub ok 2024-09-18T08:21:53.905 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:21:53 smithi050 ceph-mon[32371]: 5.d scrub starts 2024-09-18T08:21:53.905 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:21:53 smithi050 ceph-mon[32371]: 5.d scrub ok 2024-09-18T08:21:53.906 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:21:53 smithi050 ceph-mon[32371]: from='client.? 172.21.15.7:0/1252441775' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]': finished 2024-09-18T08:21:53.906 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:21:53 smithi050 ceph-mon[32371]: osdmap e463: 8 total, 8 up, 8 in 2024-09-18T08:21:53.906 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:21:53 smithi050 ceph-mon[32371]: 6.1a scrub starts 2024-09-18T08:21:53.906 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:21:53 smithi050 ceph-mon[32371]: 6.1a scrub ok 2024-09-18T08:21:53.906 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:21:53 smithi050 ceph-mon[32371]: osdmap e464: 8 total, 8 up, 8 in 2024-09-18T08:21:53.954 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:21:53 smithi007 ceph-mon[39249]: 3.c scrub starts 2024-09-18T08:21:53.955 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:21:53 smithi007 ceph-mon[39249]: 3.c scrub ok 2024-09-18T08:21:53.955 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:21:53 smithi007 ceph-mon[39249]: 2.2 scrub starts 2024-09-18T08:21:53.955 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:21:53 smithi007 ceph-mon[39249]: 2.2 scrub ok 2024-09-18T08:21:53.955 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:21:53 smithi007 ceph-mon[39249]: 5.d scrub starts 2024-09-18T08:21:53.955 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:21:53 smithi007 ceph-mon[39249]: 5.d scrub ok 2024-09-18T08:21:53.955 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:21:53 smithi007 ceph-mon[39249]: from='client.? 172.21.15.7:0/1252441775' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]': finished 2024-09-18T08:21:53.955 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:21:53 smithi007 ceph-mon[39249]: osdmap e463: 8 total, 8 up, 8 in 2024-09-18T08:21:53.955 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:21:53 smithi007 ceph-mon[39249]: 6.1a scrub starts 2024-09-18T08:21:53.955 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:21:53 smithi007 ceph-mon[39249]: 6.1a scrub ok 2024-09-18T08:21:53.955 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:21:53 smithi007 ceph-mon[39249]: osdmap e464: 8 total, 8 up, 8 in 2024-09-18T08:21:53.955 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:21:53 smithi007 ceph-mon[29459]: 3.c scrub starts 2024-09-18T08:21:53.956 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:21:53 smithi007 ceph-mon[29459]: 3.c scrub ok 2024-09-18T08:21:53.956 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:21:53 smithi007 ceph-mon[29459]: 2.2 scrub starts 2024-09-18T08:21:53.956 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:21:53 smithi007 ceph-mon[29459]: 2.2 scrub ok 2024-09-18T08:21:53.956 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:21:53 smithi007 ceph-mon[29459]: 5.d scrub starts 2024-09-18T08:21:53.956 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:21:53 smithi007 ceph-mon[29459]: 5.d scrub ok 2024-09-18T08:21:53.956 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:21:53 smithi007 ceph-mon[29459]: from='client.? 172.21.15.7:0/1252441775' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]': finished 2024-09-18T08:21:53.956 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:21:53 smithi007 ceph-mon[29459]: osdmap e463: 8 total, 8 up, 8 in 2024-09-18T08:21:53.956 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:21:53 smithi007 ceph-mon[29459]: 6.1a scrub starts 2024-09-18T08:21:53.956 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:21:53 smithi007 ceph-mon[29459]: 6.1a scrub ok 2024-09-18T08:21:53.956 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:21:53 smithi007 ceph-mon[29459]: osdmap e464: 8 total, 8 up, 8 in 2024-09-18T08:21:54.631 INFO:teuthology.orchestra.run.smithi007.stderr:set osd.2 primary-affinity to 1 (8655362) 2024-09-18T08:21:54.905 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:21:54 smithi050 ceph-mon[32371]: purged_snaps scrub starts 2024-09-18T08:21:54.905 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:21:54 smithi050 ceph-mon[32371]: purged_snaps scrub ok 2024-09-18T08:21:54.905 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:21:54 smithi050 ceph-mon[32371]: pgmap v951: 142 pgs: 17 peering, 125 active+clean; 583 KiB data, 5.3 GiB used, 710 GiB / 715 GiB avail; 1.7 KiB/s rd, 1 op/s; 324 B/s, 6 objects/s recovering 2024-09-18T08:21:54.905 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:21:54 smithi050 ceph-mon[32371]: from='client.? 172.21.15.7:0/1350710840' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 2, "weight": 1.0} : dispatch 2024-09-18T08:21:54.979 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:21:54 smithi007 ceph-mon[29459]: purged_snaps scrub starts 2024-09-18T08:21:54.980 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:21:54 smithi007 ceph-mon[29459]: purged_snaps scrub ok 2024-09-18T08:21:54.980 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:21:54 smithi007 ceph-mon[29459]: pgmap v951: 142 pgs: 17 peering, 125 active+clean; 583 KiB data, 5.3 GiB used, 710 GiB / 715 GiB avail; 1.7 KiB/s rd, 1 op/s; 324 B/s, 6 objects/s recovering 2024-09-18T08:21:54.980 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:21:54 smithi007 ceph-mon[29459]: from='client.? 172.21.15.7:0/1350710840' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 2, "weight": 1.0} : dispatch 2024-09-18T08:21:54.980 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:21:54 smithi007 ceph-mon[39249]: purged_snaps scrub starts 2024-09-18T08:21:54.980 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:21:54 smithi007 ceph-mon[39249]: purged_snaps scrub ok 2024-09-18T08:21:54.980 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:21:54 smithi007 ceph-mon[39249]: pgmap v951: 142 pgs: 17 peering, 125 active+clean; 583 KiB data, 5.3 GiB used, 710 GiB / 715 GiB avail; 1.7 KiB/s rd, 1 op/s; 324 B/s, 6 objects/s recovering 2024-09-18T08:21:54.980 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:21:54 smithi007 ceph-mon[39249]: from='client.? 172.21.15.7:0/1350710840' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 2, "weight": 1.0} : dispatch 2024-09-18T08:21:55.415 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid 779349ca-7593-11ef-bceb-c7b262605968 -- ceph osd reweight 3 1 2024-09-18T08:21:55.682 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/779349ca-7593-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T08:21:55.905 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:21:55 smithi050 ceph-mon[32371]: from='client.? 172.21.15.7:0/1350710840' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 2, "weight": 1.0}]': finished 2024-09-18T08:21:55.905 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:21:55 smithi050 ceph-mon[32371]: osdmap e465: 8 total, 8 up, 8 in 2024-09-18T08:21:55.905 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:21:55 smithi050 ceph-mon[32371]: 5.13 scrub starts 2024-09-18T08:21:55.905 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:21:55 smithi050 ceph-mon[32371]: 5.13 scrub ok 2024-09-18T08:21:55.942 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:21:55 smithi007 ceph-mon[29459]: from='client.? 172.21.15.7:0/1350710840' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 2, "weight": 1.0}]': finished 2024-09-18T08:21:55.942 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:21:55 smithi007 ceph-mon[29459]: osdmap e465: 8 total, 8 up, 8 in 2024-09-18T08:21:55.942 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:21:55 smithi007 ceph-mon[29459]: 5.13 scrub starts 2024-09-18T08:21:55.942 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:21:55 smithi007 ceph-mon[29459]: 5.13 scrub ok 2024-09-18T08:21:55.943 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:21:55 smithi007 ceph-mon[39249]: from='client.? 172.21.15.7:0/1350710840' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 2, "weight": 1.0}]': finished 2024-09-18T08:21:55.943 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:21:55 smithi007 ceph-mon[39249]: osdmap e465: 8 total, 8 up, 8 in 2024-09-18T08:21:55.943 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:21:55 smithi007 ceph-mon[39249]: 5.13 scrub starts 2024-09-18T08:21:55.943 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:21:55 smithi007 ceph-mon[39249]: 5.13 scrub ok 2024-09-18T08:21:56.647 INFO:teuthology.orchestra.run.smithi007.stderr:reweighted osd.3 to 1 (10000) 2024-09-18T08:21:56.905 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:21:56 smithi050 ceph-mon[32371]: 5.9 scrub starts 2024-09-18T08:21:56.905 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:21:56 smithi050 ceph-mon[32371]: 5.9 scrub ok 2024-09-18T08:21:56.905 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:21:56 smithi050 ceph-mon[32371]: pgmap v954: 142 pgs: 1 activating, 17 peering, 124 active+clean; 583 KiB data, 5.3 GiB used, 710 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s; 22 B/s, 1 objects/s recovering 2024-09-18T08:21:56.905 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:21:56 smithi050 ceph-mon[32371]: 6.b scrub starts 2024-09-18T08:21:56.905 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:21:56 smithi050 ceph-mon[32371]: 6.b scrub ok 2024-09-18T08:21:56.905 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:21:56 smithi050 ceph-mon[32371]: 6.0 scrub starts 2024-09-18T08:21:56.905 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:21:56 smithi050 ceph-mon[32371]: 6.0 scrub ok 2024-09-18T08:21:56.905 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:21:56 smithi050 ceph-mon[32371]: from='client.? 172.21.15.7:0/2323588826' entity='client.admin' cmd={"prefix": "osd reweight", "id": 3, "weight": 1.0} : dispatch 2024-09-18T08:21:56.905 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:21:56 smithi050 ceph-mon[32371]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 3, "weight": 1.0} : dispatch 2024-09-18T08:21:56.979 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Sep 18 08:21:56 smithi007 ceph-779349ca-7593-11ef-bceb-c7b262605968-mgr-y[29688]: ::ffff:172.21.15.50 - - [18/Sep/2024:08:21:56] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-09-18T08:21:56.980 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:21:56 smithi007 ceph-mon[29459]: 5.9 scrub starts 2024-09-18T08:21:56.980 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:21:56 smithi007 ceph-mon[29459]: 5.9 scrub ok 2024-09-18T08:21:56.980 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:21:56 smithi007 ceph-mon[29459]: pgmap v954: 142 pgs: 1 activating, 17 peering, 124 active+clean; 583 KiB data, 5.3 GiB used, 710 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s; 22 B/s, 1 objects/s recovering 2024-09-18T08:21:56.980 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:21:56 smithi007 ceph-mon[29459]: 6.b scrub starts 2024-09-18T08:21:56.980 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:21:56 smithi007 ceph-mon[29459]: 6.b scrub ok 2024-09-18T08:21:56.980 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:21:56 smithi007 ceph-mon[29459]: 6.0 scrub starts 2024-09-18T08:21:56.980 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:21:56 smithi007 ceph-mon[29459]: 6.0 scrub ok 2024-09-18T08:21:56.980 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:21:56 smithi007 ceph-mon[29459]: from='client.? 172.21.15.7:0/2323588826' entity='client.admin' cmd={"prefix": "osd reweight", "id": 3, "weight": 1.0} : dispatch 2024-09-18T08:21:56.980 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:21:56 smithi007 ceph-mon[29459]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 3, "weight": 1.0} : dispatch 2024-09-18T08:21:56.981 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:21:56 smithi007 ceph-mon[39249]: 5.9 scrub starts 2024-09-18T08:21:56.981 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:21:56 smithi007 ceph-mon[39249]: 5.9 scrub ok 2024-09-18T08:21:56.981 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:21:56 smithi007 ceph-mon[39249]: pgmap v954: 142 pgs: 1 activating, 17 peering, 124 active+clean; 583 KiB data, 5.3 GiB used, 710 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s; 22 B/s, 1 objects/s recovering 2024-09-18T08:21:56.981 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:21:56 smithi007 ceph-mon[39249]: 6.b scrub starts 2024-09-18T08:21:56.981 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:21:56 smithi007 ceph-mon[39249]: 6.b scrub ok 2024-09-18T08:21:56.981 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:21:56 smithi007 ceph-mon[39249]: 6.0 scrub starts 2024-09-18T08:21:56.981 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:21:56 smithi007 ceph-mon[39249]: 6.0 scrub ok 2024-09-18T08:21:56.981 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:21:56 smithi007 ceph-mon[39249]: from='client.? 172.21.15.7:0/2323588826' entity='client.admin' cmd={"prefix": "osd reweight", "id": 3, "weight": 1.0} : dispatch 2024-09-18T08:21:56.981 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:21:56 smithi007 ceph-mon[39249]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 3, "weight": 1.0} : dispatch 2024-09-18T08:21:57.506 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid 779349ca-7593-11ef-bceb-c7b262605968 -- ceph osd primary-affinity 3 1 2024-09-18T08:21:57.730 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:21:57 smithi007 ceph-mon[29459]: 5.1a scrub starts 2024-09-18T08:21:57.730 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:21:57 smithi007 ceph-mon[29459]: 5.1a scrub ok 2024-09-18T08:21:57.730 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:21:57 smithi007 ceph-mon[29459]: 4.1f scrub starts 2024-09-18T08:21:57.730 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:21:57 smithi007 ceph-mon[29459]: 4.1f scrub ok 2024-09-18T08:21:57.730 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:21:57 smithi007 ceph-mon[29459]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]': finished 2024-09-18T08:21:57.730 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:21:57 smithi007 ceph-mon[29459]: osdmap e466: 8 total, 8 up, 8 in 2024-09-18T08:21:57.730 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:21:57 smithi007 ceph-mon[29459]: 4.2 scrub starts 2024-09-18T08:21:57.730 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:21:57 smithi007 ceph-mon[29459]: 4.2 scrub ok 2024-09-18T08:21:57.731 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:21:57 smithi007 ceph-mon[29459]: 3.b scrub starts 2024-09-18T08:21:57.731 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:21:57 smithi007 ceph-mon[29459]: 3.b scrub ok 2024-09-18T08:21:57.731 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:21:57 smithi007 ceph-mon[39249]: 5.1a scrub starts 2024-09-18T08:21:57.731 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:21:57 smithi007 ceph-mon[39249]: 5.1a scrub ok 2024-09-18T08:21:57.731 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:21:57 smithi007 ceph-mon[39249]: 4.1f scrub starts 2024-09-18T08:21:57.731 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:21:57 smithi007 ceph-mon[39249]: 4.1f scrub ok 2024-09-18T08:21:57.731 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:21:57 smithi007 ceph-mon[39249]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]': finished 2024-09-18T08:21:57.731 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:21:57 smithi007 ceph-mon[39249]: osdmap e466: 8 total, 8 up, 8 in 2024-09-18T08:21:57.731 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:21:57 smithi007 ceph-mon[39249]: 4.2 scrub starts 2024-09-18T08:21:57.732 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:21:57 smithi007 ceph-mon[39249]: 4.2 scrub ok 2024-09-18T08:21:57.732 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:21:57 smithi007 ceph-mon[39249]: 3.b scrub starts 2024-09-18T08:21:57.732 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:21:57 smithi007 ceph-mon[39249]: 3.b scrub ok 2024-09-18T08:21:57.774 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/779349ca-7593-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T08:21:57.905 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:21:57 smithi050 ceph-mon[32371]: 5.1a scrub starts 2024-09-18T08:21:57.905 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:21:57 smithi050 ceph-mon[32371]: 5.1a scrub ok 2024-09-18T08:21:57.905 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:21:57 smithi050 ceph-mon[32371]: 4.1f scrub starts 2024-09-18T08:21:57.905 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:21:57 smithi050 ceph-mon[32371]: 4.1f scrub ok 2024-09-18T08:21:57.905 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:21:57 smithi050 ceph-mon[32371]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]': finished 2024-09-18T08:21:57.905 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:21:57 smithi050 ceph-mon[32371]: osdmap e466: 8 total, 8 up, 8 in 2024-09-18T08:21:57.905 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:21:57 smithi050 ceph-mon[32371]: 4.2 scrub starts 2024-09-18T08:21:57.906 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:21:57 smithi050 ceph-mon[32371]: 4.2 scrub ok 2024-09-18T08:21:57.906 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:21:57 smithi050 ceph-mon[32371]: 3.b scrub starts 2024-09-18T08:21:57.906 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:21:57 smithi050 ceph-mon[32371]: 3.b scrub ok 2024-09-18T08:21:58.658 INFO:teuthology.orchestra.run.smithi007.stderr:set osd.3 primary-affinity to 1 (8655362) 2024-09-18T08:21:58.980 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:21:58 smithi007 ceph-mon[29459]: 2.1 scrub starts 2024-09-18T08:21:58.980 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:21:58 smithi007 ceph-mon[29459]: 2.1 scrub ok 2024-09-18T08:21:58.980 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:21:58 smithi007 ceph-mon[29459]: pgmap v956: 142 pgs: 1 activating, 17 peering, 124 active+clean; 583 KiB data, 5.3 GiB used, 710 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s; 22 B/s, 1 objects/s recovering 2024-09-18T08:21:58.980 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:21:58 smithi007 ceph-mon[29459]: 4.13 scrub starts 2024-09-18T08:21:58.980 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:21:58 smithi007 ceph-mon[29459]: 4.13 scrub ok 2024-09-18T08:21:58.980 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:21:58 smithi007 ceph-mon[29459]: from='client.? 172.21.15.7:0/1336332298' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 3, "weight": 1.0} : dispatch 2024-09-18T08:21:58.980 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:21:58 smithi007 ceph-mon[29459]: from='client.? ' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 3, "weight": 1.0} : dispatch 2024-09-18T08:21:58.981 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:21:58 smithi007 ceph-mon[39249]: 2.1 scrub starts 2024-09-18T08:21:58.981 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:21:58 smithi007 ceph-mon[39249]: 2.1 scrub ok 2024-09-18T08:21:58.981 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:21:58 smithi007 ceph-mon[39249]: pgmap v956: 142 pgs: 1 activating, 17 peering, 124 active+clean; 583 KiB data, 5.3 GiB used, 710 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s; 22 B/s, 1 objects/s recovering 2024-09-18T08:21:58.981 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:21:58 smithi007 ceph-mon[39249]: 4.13 scrub starts 2024-09-18T08:21:58.981 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:21:58 smithi007 ceph-mon[39249]: 4.13 scrub ok 2024-09-18T08:21:58.981 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:21:58 smithi007 ceph-mon[39249]: from='client.? 172.21.15.7:0/1336332298' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 3, "weight": 1.0} : dispatch 2024-09-18T08:21:58.981 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:21:58 smithi007 ceph-mon[39249]: from='client.? ' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 3, "weight": 1.0} : dispatch 2024-09-18T08:21:59.155 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:21:58 smithi050 ceph-mon[32371]: 2.1 scrub starts 2024-09-18T08:21:59.155 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:21:58 smithi050 ceph-mon[32371]: 2.1 scrub ok 2024-09-18T08:21:59.155 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:21:58 smithi050 ceph-mon[32371]: pgmap v956: 142 pgs: 1 activating, 17 peering, 124 active+clean; 583 KiB data, 5.3 GiB used, 710 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s; 22 B/s, 1 objects/s recovering 2024-09-18T08:21:59.156 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:21:58 smithi050 ceph-mon[32371]: 4.13 scrub starts 2024-09-18T08:21:59.156 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:21:58 smithi050 ceph-mon[32371]: 4.13 scrub ok 2024-09-18T08:21:59.156 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:21:58 smithi050 ceph-mon[32371]: from='client.? 172.21.15.7:0/1336332298' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 3, "weight": 1.0} : dispatch 2024-09-18T08:21:59.156 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:21:58 smithi050 ceph-mon[32371]: from='client.? ' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 3, "weight": 1.0} : dispatch 2024-09-18T08:21:59.375 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid 779349ca-7593-11ef-bceb-c7b262605968 -- ceph osd reweight 4 1 2024-09-18T08:21:59.648 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/779349ca-7593-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T08:21:59.977 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:21:59 smithi007 ceph-mon[39249]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 3, "weight": 1.0}]': finished 2024-09-18T08:21:59.978 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:21:59 smithi007 ceph-mon[39249]: osdmap e467: 8 total, 8 up, 8 in 2024-09-18T08:21:59.978 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:21:59 smithi007 ceph-mon[39249]: 1.3 scrub starts 2024-09-18T08:21:59.978 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:21:59 smithi007 ceph-mon[39249]: 1.3 scrub ok 2024-09-18T08:21:59.978 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:21:59 smithi007 ceph-mon[39249]: purged_snaps scrub starts 2024-09-18T08:21:59.978 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:21:59 smithi007 ceph-mon[39249]: purged_snaps scrub ok 2024-09-18T08:21:59.979 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:21:59 smithi007 ceph-mon[39249]: 4.e scrub starts 2024-09-18T08:21:59.979 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:21:59 smithi007 ceph-mon[39249]: 4.e scrub ok 2024-09-18T08:21:59.979 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:21:59 smithi007 ceph-mon[29459]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 3, "weight": 1.0}]': finished 2024-09-18T08:21:59.979 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:21:59 smithi007 ceph-mon[29459]: osdmap e467: 8 total, 8 up, 8 in 2024-09-18T08:21:59.980 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:21:59 smithi007 ceph-mon[29459]: 1.3 scrub starts 2024-09-18T08:21:59.980 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:21:59 smithi007 ceph-mon[29459]: 1.3 scrub ok 2024-09-18T08:21:59.980 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:21:59 smithi007 ceph-mon[29459]: purged_snaps scrub starts 2024-09-18T08:21:59.980 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:21:59 smithi007 ceph-mon[29459]: purged_snaps scrub ok 2024-09-18T08:21:59.980 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:21:59 smithi007 ceph-mon[29459]: 4.e scrub starts 2024-09-18T08:21:59.981 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:21:59 smithi007 ceph-mon[29459]: 4.e scrub ok 2024-09-18T08:22:00.155 INFO:journalctl@ceph.iscsi.iscsi.a.smithi050.stdout:Sep 18 08:21:59 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-iscsi-iscsi-a[57782]: debug there is no tcmu-runner data available 2024-09-18T08:22:00.155 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:21:59 smithi050 ceph-mon[32371]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 3, "weight": 1.0}]': finished 2024-09-18T08:22:00.155 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:21:59 smithi050 ceph-mon[32371]: osdmap e467: 8 total, 8 up, 8 in 2024-09-18T08:22:00.155 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:21:59 smithi050 ceph-mon[32371]: 1.3 scrub starts 2024-09-18T08:22:00.155 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:21:59 smithi050 ceph-mon[32371]: 1.3 scrub ok 2024-09-18T08:22:00.155 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:21:59 smithi050 ceph-mon[32371]: purged_snaps scrub starts 2024-09-18T08:22:00.155 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:21:59 smithi050 ceph-mon[32371]: purged_snaps scrub ok 2024-09-18T08:22:00.156 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:21:59 smithi050 ceph-mon[32371]: 4.e scrub starts 2024-09-18T08:22:00.156 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:21:59 smithi050 ceph-mon[32371]: 4.e scrub ok 2024-09-18T08:22:00.980 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:22:00 smithi007 ceph-mon[29459]: pgmap v958: 142 pgs: 142 active+clean; 583 KiB data, 5.3 GiB used, 710 GiB / 715 GiB avail; 864 B/s rd, 0 op/s; 189 B/s, 6 objects/s recovering 2024-09-18T08:22:00.980 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:22:00 smithi007 ceph-mon[29459]: osdmap e468: 8 total, 8 up, 8 in 2024-09-18T08:22:00.980 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:22:00 smithi007 ceph-mon[29459]: from='client.? 172.21.15.7:0/759295225' entity='client.admin' cmd={"prefix": "osd reweight", "id": 4, "weight": 1.0} : dispatch 2024-09-18T08:22:00.981 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:22:00 smithi007 ceph-mon[39249]: pgmap v958: 142 pgs: 142 active+clean; 583 KiB data, 5.3 GiB used, 710 GiB / 715 GiB avail; 864 B/s rd, 0 op/s; 189 B/s, 6 objects/s recovering 2024-09-18T08:22:00.981 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:22:00 smithi007 ceph-mon[39249]: osdmap e468: 8 total, 8 up, 8 in 2024-09-18T08:22:00.981 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:22:00 smithi007 ceph-mon[39249]: from='client.? 172.21.15.7:0/759295225' entity='client.admin' cmd={"prefix": "osd reweight", "id": 4, "weight": 1.0} : dispatch 2024-09-18T08:22:01.155 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:22:00 smithi050 ceph-mon[32371]: pgmap v958: 142 pgs: 142 active+clean; 583 KiB data, 5.3 GiB used, 710 GiB / 715 GiB avail; 864 B/s rd, 0 op/s; 189 B/s, 6 objects/s recovering 2024-09-18T08:22:01.155 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:22:00 smithi050 ceph-mon[32371]: osdmap e468: 8 total, 8 up, 8 in 2024-09-18T08:22:01.155 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:22:00 smithi050 ceph-mon[32371]: from='client.? 172.21.15.7:0/759295225' entity='client.admin' cmd={"prefix": "osd reweight", "id": 4, "weight": 1.0} : dispatch 2024-09-18T08:22:01.512 INFO:teuthology.orchestra.run.smithi007.stderr:reweighted osd.4 to 1 (10000) 2024-09-18T08:22:01.820 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:22:01 smithi007 ceph-mon[29459]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-18T08:22:01.820 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:22:01 smithi007 ceph-mon[29459]: purged_snaps scrub starts 2024-09-18T08:22:01.820 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:22:01 smithi007 ceph-mon[29459]: purged_snaps scrub ok 2024-09-18T08:22:01.821 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:22:01 smithi007 ceph-mon[29459]: purged_snaps scrub starts 2024-09-18T08:22:01.821 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:22:01 smithi007 ceph-mon[29459]: purged_snaps scrub ok 2024-09-18T08:22:01.821 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:22:01 smithi007 ceph-mon[29459]: 1.5 scrub starts 2024-09-18T08:22:01.821 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:22:01 smithi007 ceph-mon[29459]: 1.5 scrub ok 2024-09-18T08:22:01.821 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:22:01 smithi007 ceph-mon[29459]: from='client.? 172.21.15.7:0/759295225' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]': finished 2024-09-18T08:22:01.821 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:22:01 smithi007 ceph-mon[29459]: osdmap e469: 8 total, 8 up, 8 in 2024-09-18T08:22:01.821 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:22:01 smithi007 ceph-mon[39249]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-18T08:22:01.822 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:22:01 smithi007 ceph-mon[39249]: purged_snaps scrub starts 2024-09-18T08:22:01.822 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:22:01 smithi007 ceph-mon[39249]: purged_snaps scrub ok 2024-09-18T08:22:01.822 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:22:01 smithi007 ceph-mon[39249]: purged_snaps scrub starts 2024-09-18T08:22:01.822 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:22:01 smithi007 ceph-mon[39249]: purged_snaps scrub ok 2024-09-18T08:22:01.822 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:22:01 smithi007 ceph-mon[39249]: 1.5 scrub starts 2024-09-18T08:22:01.822 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:22:01 smithi007 ceph-mon[39249]: 1.5 scrub ok 2024-09-18T08:22:01.822 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:22:01 smithi007 ceph-mon[39249]: from='client.? 172.21.15.7:0/759295225' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]': finished 2024-09-18T08:22:01.823 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:22:01 smithi007 ceph-mon[39249]: osdmap e469: 8 total, 8 up, 8 in 2024-09-18T08:22:02.155 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:22:01 smithi050 ceph-mon[32371]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-18T08:22:02.155 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:22:01 smithi050 ceph-mon[32371]: purged_snaps scrub starts 2024-09-18T08:22:02.155 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:22:01 smithi050 ceph-mon[32371]: purged_snaps scrub ok 2024-09-18T08:22:02.155 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:22:01 smithi050 ceph-mon[32371]: purged_snaps scrub starts 2024-09-18T08:22:02.155 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:22:01 smithi050 ceph-mon[32371]: purged_snaps scrub ok 2024-09-18T08:22:02.155 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:22:01 smithi050 ceph-mon[32371]: 1.5 scrub starts 2024-09-18T08:22:02.155 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:22:01 smithi050 ceph-mon[32371]: 1.5 scrub ok 2024-09-18T08:22:02.155 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:22:01 smithi050 ceph-mon[32371]: from='client.? 172.21.15.7:0/759295225' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]': finished 2024-09-18T08:22:02.155 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:22:01 smithi050 ceph-mon[32371]: osdmap e469: 8 total, 8 up, 8 in 2024-09-18T08:22:02.189 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid 779349ca-7593-11ef-bceb-c7b262605968 -- ceph osd primary-affinity 4 1 2024-09-18T08:22:02.456 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/779349ca-7593-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T08:22:02.874 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:22:02 smithi007 ceph-mon[29459]: 5.16 scrub starts 2024-09-18T08:22:02.874 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:22:02 smithi007 ceph-mon[29459]: 5.16 scrub ok 2024-09-18T08:22:02.875 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:22:02 smithi007 ceph-mon[29459]: pgmap v961: 142 pgs: 142 active+clean; 583 KiB data, 5.3 GiB used, 710 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s; 207 B/s, 5 objects/s recovering 2024-09-18T08:22:02.875 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:22:02 smithi007 ceph-mon[29459]: 5.1c scrub starts 2024-09-18T08:22:02.875 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:22:02 smithi007 ceph-mon[29459]: 5.1c scrub ok 2024-09-18T08:22:02.875 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:22:02 smithi007 ceph-mon[39249]: 5.16 scrub starts 2024-09-18T08:22:02.875 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:22:02 smithi007 ceph-mon[39249]: 5.16 scrub ok 2024-09-18T08:22:02.876 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:22:02 smithi007 ceph-mon[39249]: pgmap v961: 142 pgs: 142 active+clean; 583 KiB data, 5.3 GiB used, 710 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s; 207 B/s, 5 objects/s recovering 2024-09-18T08:22:02.876 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:22:02 smithi007 ceph-mon[39249]: 5.1c scrub starts 2024-09-18T08:22:02.876 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:22:02 smithi007 ceph-mon[39249]: 5.1c scrub ok 2024-09-18T08:22:03.155 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:22:02 smithi050 ceph-mon[32371]: 5.16 scrub starts 2024-09-18T08:22:03.155 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:22:02 smithi050 ceph-mon[32371]: 5.16 scrub ok 2024-09-18T08:22:03.155 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:22:02 smithi050 ceph-mon[32371]: pgmap v961: 142 pgs: 142 active+clean; 583 KiB data, 5.3 GiB used, 710 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s; 207 B/s, 5 objects/s recovering 2024-09-18T08:22:03.156 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:22:02 smithi050 ceph-mon[32371]: 5.1c scrub starts 2024-09-18T08:22:03.156 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:22:02 smithi050 ceph-mon[32371]: 5.1c scrub ok 2024-09-18T08:22:03.678 INFO:teuthology.orchestra.run.smithi007.stderr:set osd.4 primary-affinity to 1 (8655362) 2024-09-18T08:22:03.980 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:22:03 smithi007 ceph-mon[29459]: 6.a scrub starts 2024-09-18T08:22:03.980 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:22:03 smithi007 ceph-mon[29459]: 6.a scrub ok 2024-09-18T08:22:03.980 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:22:03 smithi007 ceph-mon[29459]: from='client.? 172.21.15.7:0/2782369287' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 4, "weight": 1.0} : dispatch 2024-09-18T08:22:03.980 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:22:03 smithi007 ceph-mon[29459]: from='client.? ' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 4, "weight": 1.0} : dispatch 2024-09-18T08:22:03.981 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:22:03 smithi007 ceph-mon[39249]: 6.a scrub starts 2024-09-18T08:22:03.981 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:22:03 smithi007 ceph-mon[39249]: 6.a scrub ok 2024-09-18T08:22:03.981 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:22:03 smithi007 ceph-mon[39249]: from='client.? 172.21.15.7:0/2782369287' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 4, "weight": 1.0} : dispatch 2024-09-18T08:22:03.981 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:22:03 smithi007 ceph-mon[39249]: from='client.? ' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 4, "weight": 1.0} : dispatch 2024-09-18T08:22:04.155 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:22:03 smithi050 ceph-mon[32371]: 6.a scrub starts 2024-09-18T08:22:04.155 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:22:03 smithi050 ceph-mon[32371]: 6.a scrub ok 2024-09-18T08:22:04.155 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:22:03 smithi050 ceph-mon[32371]: from='client.? 172.21.15.7:0/2782369287' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 4, "weight": 1.0} : dispatch 2024-09-18T08:22:04.155 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:22:03 smithi050 ceph-mon[32371]: from='client.? ' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 4, "weight": 1.0} : dispatch 2024-09-18T08:22:04.400 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid 779349ca-7593-11ef-bceb-c7b262605968 -- ceph osd reweight 5 1 2024-09-18T08:22:04.668 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/779349ca-7593-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T08:22:04.938 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:22:04 smithi007 ceph-mon[29459]: 5.10 scrub starts 2024-09-18T08:22:04.938 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:22:04 smithi007 ceph-mon[29459]: 5.10 scrub ok 2024-09-18T08:22:04.939 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:22:04 smithi007 ceph-mon[29459]: pgmap v962: 142 pgs: 142 active+clean; 583 KiB data, 5.3 GiB used, 710 GiB / 715 GiB avail; 1.7 KiB/s rd, 1 op/s; 168 B/s, 4 objects/s recovering 2024-09-18T08:22:04.939 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:22:04 smithi007 ceph-mon[29459]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 4, "weight": 1.0}]': finished 2024-09-18T08:22:04.939 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:22:04 smithi007 ceph-mon[29459]: osdmap e470: 8 total, 8 up, 8 in 2024-09-18T08:22:04.939 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:22:04 smithi007 ceph-mon[29459]: 4.f scrub starts 2024-09-18T08:22:04.939 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:22:04 smithi007 ceph-mon[29459]: 4.f scrub ok 2024-09-18T08:22:04.939 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:22:04 smithi007 ceph-mon[29459]: from='mgr.24470 172.21.15.7:0/2594390261' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-18T08:22:04.939 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:22:04 smithi007 ceph-mon[39249]: 5.10 scrub starts 2024-09-18T08:22:04.939 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:22:04 smithi007 ceph-mon[39249]: 5.10 scrub ok 2024-09-18T08:22:04.939 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:22:04 smithi007 ceph-mon[39249]: pgmap v962: 142 pgs: 142 active+clean; 583 KiB data, 5.3 GiB used, 710 GiB / 715 GiB avail; 1.7 KiB/s rd, 1 op/s; 168 B/s, 4 objects/s recovering 2024-09-18T08:22:04.940 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:22:04 smithi007 ceph-mon[39249]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 4, "weight": 1.0}]': finished 2024-09-18T08:22:04.940 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:22:04 smithi007 ceph-mon[39249]: osdmap e470: 8 total, 8 up, 8 in 2024-09-18T08:22:04.940 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:22:04 smithi007 ceph-mon[39249]: 4.f scrub starts 2024-09-18T08:22:04.940 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:22:04 smithi007 ceph-mon[39249]: 4.f scrub ok 2024-09-18T08:22:04.940 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:22:04 smithi007 ceph-mon[39249]: from='mgr.24470 172.21.15.7:0/2594390261' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-18T08:22:05.155 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:22:04 smithi050 ceph-mon[32371]: 5.10 scrub starts 2024-09-18T08:22:05.155 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:22:04 smithi050 ceph-mon[32371]: 5.10 scrub ok 2024-09-18T08:22:05.155 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:22:04 smithi050 ceph-mon[32371]: pgmap v962: 142 pgs: 142 active+clean; 583 KiB data, 5.3 GiB used, 710 GiB / 715 GiB avail; 1.7 KiB/s rd, 1 op/s; 168 B/s, 4 objects/s recovering 2024-09-18T08:22:05.155 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:22:04 smithi050 ceph-mon[32371]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 4, "weight": 1.0}]': finished 2024-09-18T08:22:05.155 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:22:04 smithi050 ceph-mon[32371]: osdmap e470: 8 total, 8 up, 8 in 2024-09-18T08:22:05.155 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:22:04 smithi050 ceph-mon[32371]: 4.f scrub starts 2024-09-18T08:22:05.155 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:22:04 smithi050 ceph-mon[32371]: 4.f scrub ok 2024-09-18T08:22:05.155 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:22:04 smithi050 ceph-mon[32371]: from='mgr.24470 172.21.15.7:0/2594390261' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-18T08:22:05.679 INFO:teuthology.orchestra.run.smithi007.stderr:reweighted osd.5 to 1 (10000) 2024-09-18T08:22:05.979 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:22:05 smithi007 ceph-mon[29459]: 3.3 scrub starts 2024-09-18T08:22:05.979 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:22:05 smithi007 ceph-mon[29459]: 3.3 scrub ok 2024-09-18T08:22:05.980 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:22:05 smithi007 ceph-mon[29459]: from='client.? 172.21.15.7:0/2749254567' entity='client.admin' cmd={"prefix": "osd reweight", "id": 5, "weight": 1.0} : dispatch 2024-09-18T08:22:05.980 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:22:05 smithi007 ceph-mon[39249]: 3.3 scrub starts 2024-09-18T08:22:05.980 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:22:05 smithi007 ceph-mon[39249]: 3.3 scrub ok 2024-09-18T08:22:05.980 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:22:05 smithi007 ceph-mon[39249]: from='client.? 172.21.15.7:0/2749254567' entity='client.admin' cmd={"prefix": "osd reweight", "id": 5, "weight": 1.0} : dispatch 2024-09-18T08:22:06.155 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:22:05 smithi050 ceph-mon[32371]: 3.3 scrub starts 2024-09-18T08:22:06.155 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:22:05 smithi050 ceph-mon[32371]: 3.3 scrub ok 2024-09-18T08:22:06.155 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:22:05 smithi050 ceph-mon[32371]: from='client.? 172.21.15.7:0/2749254567' entity='client.admin' cmd={"prefix": "osd reweight", "id": 5, "weight": 1.0} : dispatch 2024-09-18T08:22:06.418 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid 779349ca-7593-11ef-bceb-c7b262605968 -- ceph osd primary-affinity 5 1 2024-09-18T08:22:06.686 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/779349ca-7593-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T08:22:06.932 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Sep 18 08:22:06 smithi007 ceph-779349ca-7593-11ef-bceb-c7b262605968-mgr-y[29688]: ::ffff:172.21.15.50 - - [18/Sep/2024:08:22:06] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-09-18T08:22:06.932 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:22:06 smithi007 ceph-mon[39249]: purged_snaps scrub starts 2024-09-18T08:22:06.933 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:22:06 smithi007 ceph-mon[39249]: purged_snaps scrub ok 2024-09-18T08:22:06.933 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:22:06 smithi007 ceph-mon[39249]: pgmap v964: 142 pgs: 142 active+clean; 583 KiB data, 5.3 GiB used, 710 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-18T08:22:06.933 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:22:06 smithi007 ceph-mon[39249]: from='client.? 172.21.15.7:0/2749254567' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]': finished 2024-09-18T08:22:06.933 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:22:06 smithi007 ceph-mon[39249]: osdmap e471: 8 total, 8 up, 8 in 2024-09-18T08:22:06.933 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:22:06 smithi007 ceph-mon[39249]: 1.0 scrub starts 2024-09-18T08:22:06.933 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:22:06 smithi007 ceph-mon[39249]: 1.0 scrub ok 2024-09-18T08:22:06.933 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:22:06 smithi007 ceph-mon[39249]: 4.5 scrub starts 2024-09-18T08:22:06.933 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:22:06 smithi007 ceph-mon[39249]: 4.5 scrub ok 2024-09-18T08:22:06.933 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:22:06 smithi007 ceph-mon[29459]: purged_snaps scrub starts 2024-09-18T08:22:06.933 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:22:06 smithi007 ceph-mon[29459]: purged_snaps scrub ok 2024-09-18T08:22:06.934 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:22:06 smithi007 ceph-mon[29459]: pgmap v964: 142 pgs: 142 active+clean; 583 KiB data, 5.3 GiB used, 710 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-18T08:22:06.934 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:22:06 smithi007 ceph-mon[29459]: from='client.? 172.21.15.7:0/2749254567' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]': finished 2024-09-18T08:22:06.934 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:22:06 smithi007 ceph-mon[29459]: osdmap e471: 8 total, 8 up, 8 in 2024-09-18T08:22:06.934 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:22:06 smithi007 ceph-mon[29459]: 1.0 scrub starts 2024-09-18T08:22:06.934 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:22:06 smithi007 ceph-mon[29459]: 1.0 scrub ok 2024-09-18T08:22:06.934 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:22:06 smithi007 ceph-mon[29459]: 4.5 scrub starts 2024-09-18T08:22:06.934 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:22:06 smithi007 ceph-mon[29459]: 4.5 scrub ok 2024-09-18T08:22:07.155 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:22:06 smithi050 ceph-mon[32371]: purged_snaps scrub starts 2024-09-18T08:22:07.155 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:22:06 smithi050 ceph-mon[32371]: purged_snaps scrub ok 2024-09-18T08:22:07.155 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:22:06 smithi050 ceph-mon[32371]: pgmap v964: 142 pgs: 142 active+clean; 583 KiB data, 5.3 GiB used, 710 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-18T08:22:07.155 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:22:06 smithi050 ceph-mon[32371]: from='client.? 172.21.15.7:0/2749254567' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]': finished 2024-09-18T08:22:07.156 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:22:06 smithi050 ceph-mon[32371]: osdmap e471: 8 total, 8 up, 8 in 2024-09-18T08:22:07.156 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:22:06 smithi050 ceph-mon[32371]: 1.0 scrub starts 2024-09-18T08:22:07.156 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:22:06 smithi050 ceph-mon[32371]: 1.0 scrub ok 2024-09-18T08:22:07.156 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:22:06 smithi050 ceph-mon[32371]: 4.5 scrub starts 2024-09-18T08:22:07.156 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:22:06 smithi050 ceph-mon[32371]: 4.5 scrub ok 2024-09-18T08:22:07.681 INFO:teuthology.orchestra.run.smithi007.stderr:set osd.5 primary-affinity to 1 (8655362) 2024-09-18T08:22:07.979 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:22:07 smithi007 ceph-mon[29459]: 5.1d scrub starts 2024-09-18T08:22:07.979 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:22:07 smithi007 ceph-mon[29459]: 5.1d scrub ok 2024-09-18T08:22:07.980 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:22:07 smithi007 ceph-mon[29459]: 5.17 scrub starts 2024-09-18T08:22:07.980 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:22:07 smithi007 ceph-mon[29459]: 5.17 scrub ok 2024-09-18T08:22:07.980 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:22:07 smithi007 ceph-mon[29459]: from='client.? 172.21.15.7:0/2315914983' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 5, "weight": 1.0} : dispatch 2024-09-18T08:22:07.980 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:22:07 smithi007 ceph-mon[39249]: 5.1d scrub starts 2024-09-18T08:22:07.980 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:22:07 smithi007 ceph-mon[39249]: 5.1d scrub ok 2024-09-18T08:22:07.980 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:22:07 smithi007 ceph-mon[39249]: 5.17 scrub starts 2024-09-18T08:22:07.980 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:22:07 smithi007 ceph-mon[39249]: 5.17 scrub ok 2024-09-18T08:22:07.980 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:22:07 smithi007 ceph-mon[39249]: from='client.? 172.21.15.7:0/2315914983' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 5, "weight": 1.0} : dispatch 2024-09-18T08:22:08.155 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:22:07 smithi050 ceph-mon[32371]: 5.1d scrub starts 2024-09-18T08:22:08.155 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:22:07 smithi050 ceph-mon[32371]: 5.1d scrub ok 2024-09-18T08:22:08.155 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:22:07 smithi050 ceph-mon[32371]: 5.17 scrub starts 2024-09-18T08:22:08.155 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:22:07 smithi050 ceph-mon[32371]: 5.17 scrub ok 2024-09-18T08:22:08.155 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:22:07 smithi050 ceph-mon[32371]: from='client.? 172.21.15.7:0/2315914983' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 5, "weight": 1.0} : dispatch 2024-09-18T08:22:08.382 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid 779349ca-7593-11ef-bceb-c7b262605968 -- ceph osd reweight 6 1 2024-09-18T08:22:08.646 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/779349ca-7593-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T08:22:08.956 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:22:08 smithi007 ceph-mon[39249]: pgmap v966: 142 pgs: 142 active+clean; 583 KiB data, 5.3 GiB used, 710 GiB / 715 GiB avail; 849 B/s rd, 0 op/s 2024-09-18T08:22:08.956 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:22:08 smithi007 ceph-mon[39249]: from='client.? 172.21.15.7:0/2315914983' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 5, "weight": 1.0}]': finished 2024-09-18T08:22:08.956 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:22:08 smithi007 ceph-mon[39249]: osdmap e472: 8 total, 8 up, 8 in 2024-09-18T08:22:08.956 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:22:08 smithi007 ceph-mon[39249]: 6.1 scrub starts 2024-09-18T08:22:08.956 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:22:08 smithi007 ceph-mon[39249]: 6.1 scrub ok 2024-09-18T08:22:08.957 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:22:08 smithi007 ceph-mon[29459]: pgmap v966: 142 pgs: 142 active+clean; 583 KiB data, 5.3 GiB used, 710 GiB / 715 GiB avail; 849 B/s rd, 0 op/s 2024-09-18T08:22:08.957 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:22:08 smithi007 ceph-mon[29459]: from='client.? 172.21.15.7:0/2315914983' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 5, "weight": 1.0}]': finished 2024-09-18T08:22:08.957 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:22:08 smithi007 ceph-mon[29459]: osdmap e472: 8 total, 8 up, 8 in 2024-09-18T08:22:08.957 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:22:08 smithi007 ceph-mon[29459]: 6.1 scrub starts 2024-09-18T08:22:08.957 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:22:08 smithi007 ceph-mon[29459]: 6.1 scrub ok 2024-09-18T08:22:09.155 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:22:08 smithi050 ceph-mon[32371]: pgmap v966: 142 pgs: 142 active+clean; 583 KiB data, 5.3 GiB used, 710 GiB / 715 GiB avail; 849 B/s rd, 0 op/s 2024-09-18T08:22:09.155 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:22:08 smithi050 ceph-mon[32371]: from='client.? 172.21.15.7:0/2315914983' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 5, "weight": 1.0}]': finished 2024-09-18T08:22:09.155 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:22:08 smithi050 ceph-mon[32371]: osdmap e472: 8 total, 8 up, 8 in 2024-09-18T08:22:09.155 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:22:08 smithi050 ceph-mon[32371]: 6.1 scrub starts 2024-09-18T08:22:09.155 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:22:08 smithi050 ceph-mon[32371]: 6.1 scrub ok 2024-09-18T08:22:09.695 INFO:teuthology.orchestra.run.smithi007.stderr:reweighted osd.6 to 1 (10000) 2024-09-18T08:22:09.979 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:22:09 smithi007 ceph-mon[29459]: 6.8 scrub starts 2024-09-18T08:22:09.980 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:22:09 smithi007 ceph-mon[29459]: 6.8 scrub ok 2024-09-18T08:22:09.980 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:22:09 smithi007 ceph-mon[29459]: 5.11 scrub starts 2024-09-18T08:22:09.980 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:22:09 smithi007 ceph-mon[29459]: 5.11 scrub ok 2024-09-18T08:22:09.980 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:22:09 smithi007 ceph-mon[29459]: from='client.? 172.21.15.7:0/3769551835' entity='client.admin' cmd={"prefix": "osd reweight", "id": 6, "weight": 1.0} : dispatch 2024-09-18T08:22:09.980 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:22:09 smithi007 ceph-mon[29459]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 6, "weight": 1.0} : dispatch 2024-09-18T08:22:09.980 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:22:09 smithi007 ceph-mon[39249]: 6.8 scrub starts 2024-09-18T08:22:09.980 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:22:09 smithi007 ceph-mon[39249]: 6.8 scrub ok 2024-09-18T08:22:09.981 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:22:09 smithi007 ceph-mon[39249]: 5.11 scrub starts 2024-09-18T08:22:09.981 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:22:09 smithi007 ceph-mon[39249]: 5.11 scrub ok 2024-09-18T08:22:09.981 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:22:09 smithi007 ceph-mon[39249]: from='client.? 172.21.15.7:0/3769551835' entity='client.admin' cmd={"prefix": "osd reweight", "id": 6, "weight": 1.0} : dispatch 2024-09-18T08:22:09.981 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:22:09 smithi007 ceph-mon[39249]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 6, "weight": 1.0} : dispatch 2024-09-18T08:22:10.155 INFO:journalctl@ceph.iscsi.iscsi.a.smithi050.stdout:Sep 18 08:22:09 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-iscsi-iscsi-a[57782]: debug there is no tcmu-runner data available 2024-09-18T08:22:10.155 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:22:09 smithi050 ceph-mon[32371]: 6.8 scrub starts 2024-09-18T08:22:10.155 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:22:09 smithi050 ceph-mon[32371]: 6.8 scrub ok 2024-09-18T08:22:10.155 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:22:09 smithi050 ceph-mon[32371]: 5.11 scrub starts 2024-09-18T08:22:10.155 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:22:09 smithi050 ceph-mon[32371]: 5.11 scrub ok 2024-09-18T08:22:10.155 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:22:09 smithi050 ceph-mon[32371]: from='client.? 172.21.15.7:0/3769551835' entity='client.admin' cmd={"prefix": "osd reweight", "id": 6, "weight": 1.0} : dispatch 2024-09-18T08:22:10.156 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:22:09 smithi050 ceph-mon[32371]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 6, "weight": 1.0} : dispatch 2024-09-18T08:22:10.538 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid 779349ca-7593-11ef-bceb-c7b262605968 -- ceph osd primary-affinity 6 1 2024-09-18T08:22:10.800 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/779349ca-7593-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T08:22:10.922 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:22:10 smithi007 ceph-mon[29459]: 4.11 scrub starts 2024-09-18T08:22:10.922 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:22:10 smithi007 ceph-mon[29459]: 4.11 scrub ok 2024-09-18T08:22:10.922 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:22:10 smithi007 ceph-mon[29459]: 5.a scrub starts 2024-09-18T08:22:10.922 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:22:10 smithi007 ceph-mon[29459]: 5.a scrub ok 2024-09-18T08:22:10.923 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:22:10 smithi007 ceph-mon[29459]: pgmap v968: 142 pgs: 142 active+clean; 583 KiB data, 5.3 GiB used, 710 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-18T08:22:10.923 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:22:10 smithi007 ceph-mon[29459]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]': finished 2024-09-18T08:22:10.923 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:22:10 smithi007 ceph-mon[29459]: osdmap e473: 8 total, 8 up, 8 in 2024-09-18T08:22:10.923 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:22:10 smithi007 ceph-mon[39249]: 4.11 scrub starts 2024-09-18T08:22:10.923 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:22:10 smithi007 ceph-mon[39249]: 4.11 scrub ok 2024-09-18T08:22:10.923 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:22:10 smithi007 ceph-mon[39249]: 5.a scrub starts 2024-09-18T08:22:10.923 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:22:10 smithi007 ceph-mon[39249]: 5.a scrub ok 2024-09-18T08:22:10.923 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:22:10 smithi007 ceph-mon[39249]: pgmap v968: 142 pgs: 142 active+clean; 583 KiB data, 5.3 GiB used, 710 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-18T08:22:10.923 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:22:10 smithi007 ceph-mon[39249]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]': finished 2024-09-18T08:22:10.923 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:22:10 smithi007 ceph-mon[39249]: osdmap e473: 8 total, 8 up, 8 in 2024-09-18T08:22:11.057 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:22:10 smithi050 ceph-mon[32371]: 4.11 scrub starts 2024-09-18T08:22:11.057 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:22:10 smithi050 ceph-mon[32371]: 4.11 scrub ok 2024-09-18T08:22:11.057 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:22:10 smithi050 ceph-mon[32371]: 5.a scrub starts 2024-09-18T08:22:11.057 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:22:10 smithi050 ceph-mon[32371]: 5.a scrub ok 2024-09-18T08:22:11.057 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:22:10 smithi050 ceph-mon[32371]: pgmap v968: 142 pgs: 142 active+clean; 583 KiB data, 5.3 GiB used, 710 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-18T08:22:11.057 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:22:10 smithi050 ceph-mon[32371]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]': finished 2024-09-18T08:22:11.057 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:22:10 smithi050 ceph-mon[32371]: osdmap e473: 8 total, 8 up, 8 in 2024-09-18T08:22:11.702 INFO:teuthology.orchestra.run.smithi007.stderr:set osd.6 primary-affinity to 1 (8655362) 2024-09-18T08:22:11.980 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:22:11 smithi007 ceph-mon[39249]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-18T08:22:11.980 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:22:11 smithi007 ceph-mon[39249]: 6.14 scrub starts 2024-09-18T08:22:11.980 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:22:11 smithi007 ceph-mon[39249]: 6.14 scrub ok 2024-09-18T08:22:11.980 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:22:11 smithi007 ceph-mon[39249]: 6.1d scrub starts 2024-09-18T08:22:11.980 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:22:11 smithi007 ceph-mon[39249]: 6.1d scrub ok 2024-09-18T08:22:11.980 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:22:11 smithi007 ceph-mon[39249]: 6.17 scrub starts 2024-09-18T08:22:11.980 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:22:11 smithi007 ceph-mon[39249]: 6.17 scrub ok 2024-09-18T08:22:11.980 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:22:11 smithi007 ceph-mon[39249]: from='client.? 172.21.15.7:0/4186144948' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 6, "weight": 1.0} : dispatch 2024-09-18T08:22:11.980 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:22:11 smithi007 ceph-mon[39249]: from='client.? ' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 6, "weight": 1.0} : dispatch 2024-09-18T08:22:11.981 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:22:11 smithi007 ceph-mon[29459]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-18T08:22:11.981 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:22:11 smithi007 ceph-mon[29459]: 6.14 scrub starts 2024-09-18T08:22:11.981 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:22:11 smithi007 ceph-mon[29459]: 6.14 scrub ok 2024-09-18T08:22:11.981 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:22:11 smithi007 ceph-mon[29459]: 6.1d scrub starts 2024-09-18T08:22:11.981 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:22:11 smithi007 ceph-mon[29459]: 6.1d scrub ok 2024-09-18T08:22:11.981 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:22:11 smithi007 ceph-mon[29459]: 6.17 scrub starts 2024-09-18T08:22:11.981 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:22:11 smithi007 ceph-mon[29459]: 6.17 scrub ok 2024-09-18T08:22:11.981 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:22:11 smithi007 ceph-mon[29459]: from='client.? 172.21.15.7:0/4186144948' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 6, "weight": 1.0} : dispatch 2024-09-18T08:22:11.981 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:22:11 smithi007 ceph-mon[29459]: from='client.? ' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 6, "weight": 1.0} : dispatch 2024-09-18T08:22:12.155 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:22:11 smithi050 ceph-mon[32371]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-18T08:22:12.155 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:22:11 smithi050 ceph-mon[32371]: 6.14 scrub starts 2024-09-18T08:22:12.155 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:22:11 smithi050 ceph-mon[32371]: 6.14 scrub ok 2024-09-18T08:22:12.155 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:22:11 smithi050 ceph-mon[32371]: 6.1d scrub starts 2024-09-18T08:22:12.155 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:22:11 smithi050 ceph-mon[32371]: 6.1d scrub ok 2024-09-18T08:22:12.155 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:22:11 smithi050 ceph-mon[32371]: 6.17 scrub starts 2024-09-18T08:22:12.156 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:22:11 smithi050 ceph-mon[32371]: 6.17 scrub ok 2024-09-18T08:22:12.156 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:22:11 smithi050 ceph-mon[32371]: from='client.? 172.21.15.7:0/4186144948' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 6, "weight": 1.0} : dispatch 2024-09-18T08:22:12.156 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:22:11 smithi050 ceph-mon[32371]: from='client.? ' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 6, "weight": 1.0} : dispatch 2024-09-18T08:22:12.482 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid 779349ca-7593-11ef-bceb-c7b262605968 -- ceph osd reweight 7 1 2024-09-18T08:22:12.755 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/779349ca-7593-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T08:22:12.981 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:22:12 smithi007 ceph-mon[29459]: 6.12 scrub starts 2024-09-18T08:22:12.981 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:22:12 smithi007 ceph-mon[29459]: 6.12 scrub ok 2024-09-18T08:22:12.981 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:22:12 smithi007 ceph-mon[29459]: pgmap v970: 142 pgs: 142 active+clean; 583 KiB data, 5.3 GiB used, 710 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-18T08:22:12.981 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:22:12 smithi007 ceph-mon[29459]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 6, "weight": 1.0}]': finished 2024-09-18T08:22:12.981 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:22:12 smithi007 ceph-mon[29459]: osdmap e474: 8 total, 8 up, 8 in 2024-09-18T08:22:12.981 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:22:12 smithi007 ceph-mon[29459]: 6.4 scrub starts 2024-09-18T08:22:12.981 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:22:12 smithi007 ceph-mon[29459]: 6.4 scrub ok 2024-09-18T08:22:12.982 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:22:12 smithi007 ceph-mon[39249]: 6.12 scrub starts 2024-09-18T08:22:12.982 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:22:12 smithi007 ceph-mon[39249]: 6.12 scrub ok 2024-09-18T08:22:12.982 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:22:12 smithi007 ceph-mon[39249]: pgmap v970: 142 pgs: 142 active+clean; 583 KiB data, 5.3 GiB used, 710 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-18T08:22:12.982 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:22:12 smithi007 ceph-mon[39249]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 6, "weight": 1.0}]': finished 2024-09-18T08:22:12.982 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:22:12 smithi007 ceph-mon[39249]: osdmap e474: 8 total, 8 up, 8 in 2024-09-18T08:22:12.982 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:22:12 smithi007 ceph-mon[39249]: 6.4 scrub starts 2024-09-18T08:22:12.982 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:22:12 smithi007 ceph-mon[39249]: 6.4 scrub ok 2024-09-18T08:22:13.158 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:22:12 smithi050 ceph-mon[32371]: 6.12 scrub starts 2024-09-18T08:22:13.158 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:22:12 smithi050 ceph-mon[32371]: 6.12 scrub ok 2024-09-18T08:22:13.158 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:22:12 smithi050 ceph-mon[32371]: pgmap v970: 142 pgs: 142 active+clean; 583 KiB data, 5.3 GiB used, 710 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-18T08:22:13.158 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:22:12 smithi050 ceph-mon[32371]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 6, "weight": 1.0}]': finished 2024-09-18T08:22:13.158 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:22:12 smithi050 ceph-mon[32371]: osdmap e474: 8 total, 8 up, 8 in 2024-09-18T08:22:13.158 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:22:12 smithi050 ceph-mon[32371]: 6.4 scrub starts 2024-09-18T08:22:13.159 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:22:12 smithi050 ceph-mon[32371]: 6.4 scrub ok 2024-09-18T08:22:13.715 INFO:teuthology.orchestra.run.smithi007.stderr:reweighted osd.7 to 1 (10000) 2024-09-18T08:22:13.979 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:22:13 smithi007 ceph-mon[39249]: 6.f scrub starts 2024-09-18T08:22:13.980 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:22:13 smithi007 ceph-mon[39249]: 6.f scrub ok 2024-09-18T08:22:13.980 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:22:13 smithi007 ceph-mon[39249]: osdmap e475: 8 total, 8 up, 8 in 2024-09-18T08:22:13.980 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:22:13 smithi007 ceph-mon[39249]: from='client.? 172.21.15.7:0/3952606338' entity='client.admin' cmd={"prefix": "osd reweight", "id": 7, "weight": 1.0} : dispatch 2024-09-18T08:22:13.980 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:22:13 smithi007 ceph-mon[29459]: 6.f scrub starts 2024-09-18T08:22:13.980 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:22:13 smithi007 ceph-mon[29459]: 6.f scrub ok 2024-09-18T08:22:13.980 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:22:13 smithi007 ceph-mon[29459]: osdmap e475: 8 total, 8 up, 8 in 2024-09-18T08:22:13.980 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:22:13 smithi007 ceph-mon[29459]: from='client.? 172.21.15.7:0/3952606338' entity='client.admin' cmd={"prefix": "osd reweight", "id": 7, "weight": 1.0} : dispatch 2024-09-18T08:22:14.155 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:22:13 smithi050 ceph-mon[32371]: 6.f scrub starts 2024-09-18T08:22:14.155 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:22:13 smithi050 ceph-mon[32371]: 6.f scrub ok 2024-09-18T08:22:14.155 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:22:13 smithi050 ceph-mon[32371]: osdmap e475: 8 total, 8 up, 8 in 2024-09-18T08:22:14.155 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:22:13 smithi050 ceph-mon[32371]: from='client.? 172.21.15.7:0/3952606338' entity='client.admin' cmd={"prefix": "osd reweight", "id": 7, "weight": 1.0} : dispatch 2024-09-18T08:22:14.397 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid 779349ca-7593-11ef-bceb-c7b262605968 -- ceph osd primary-affinity 7 1 2024-09-18T08:22:14.659 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/779349ca-7593-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T08:22:14.979 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:22:14 smithi007 ceph-mon[29459]: 1.6 scrub starts 2024-09-18T08:22:14.980 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:22:14 smithi007 ceph-mon[29459]: 1.6 scrub ok 2024-09-18T08:22:14.980 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:22:14 smithi007 ceph-mon[29459]: 1.8 scrub starts 2024-09-18T08:22:14.980 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:22:14 smithi007 ceph-mon[29459]: 1.8 scrub ok 2024-09-18T08:22:14.980 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:22:14 smithi007 ceph-mon[29459]: 4.1c scrub starts 2024-09-18T08:22:14.980 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:22:14 smithi007 ceph-mon[29459]: 4.1c scrub ok 2024-09-18T08:22:14.980 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:22:14 smithi007 ceph-mon[29459]: pgmap v973: 142 pgs: 142 active+clean; 635 KiB data, 5.3 GiB used, 710 GiB / 715 GiB avail; 5.0 KiB/s rd, 25 KiB/s wr, 4 op/s; 18 KiB/s, 0 objects/s recovering 2024-09-18T08:22:14.980 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:22:14 smithi007 ceph-mon[29459]: from='client.? 172.21.15.7:0/3952606338' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]': finished 2024-09-18T08:22:14.980 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:22:14 smithi007 ceph-mon[29459]: osdmap e476: 8 total, 8 up, 8 in 2024-09-18T08:22:14.980 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:22:14 smithi007 ceph-mon[29459]: 3.1f scrub starts 2024-09-18T08:22:14.980 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:22:14 smithi007 ceph-mon[29459]: 3.1f scrub ok 2024-09-18T08:22:14.981 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:22:14 smithi007 ceph-mon[39249]: 1.6 scrub starts 2024-09-18T08:22:14.981 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:22:14 smithi007 ceph-mon[39249]: 1.6 scrub ok 2024-09-18T08:22:14.981 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:22:14 smithi007 ceph-mon[39249]: 1.8 scrub starts 2024-09-18T08:22:14.981 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:22:14 smithi007 ceph-mon[39249]: 1.8 scrub ok 2024-09-18T08:22:14.981 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:22:14 smithi007 ceph-mon[39249]: 4.1c scrub starts 2024-09-18T08:22:14.981 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:22:14 smithi007 ceph-mon[39249]: 4.1c scrub ok 2024-09-18T08:22:14.981 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:22:14 smithi007 ceph-mon[39249]: pgmap v973: 142 pgs: 142 active+clean; 635 KiB data, 5.3 GiB used, 710 GiB / 715 GiB avail; 5.0 KiB/s rd, 25 KiB/s wr, 4 op/s; 18 KiB/s, 0 objects/s recovering 2024-09-18T08:22:14.981 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:22:14 smithi007 ceph-mon[39249]: from='client.? 172.21.15.7:0/3952606338' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]': finished 2024-09-18T08:22:14.981 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:22:14 smithi007 ceph-mon[39249]: osdmap e476: 8 total, 8 up, 8 in 2024-09-18T08:22:14.981 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:22:14 smithi007 ceph-mon[39249]: 3.1f scrub starts 2024-09-18T08:22:14.981 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:22:14 smithi007 ceph-mon[39249]: 3.1f scrub ok 2024-09-18T08:22:15.155 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:22:14 smithi050 ceph-mon[32371]: 1.6 scrub starts 2024-09-18T08:22:15.155 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:22:14 smithi050 ceph-mon[32371]: 1.6 scrub ok 2024-09-18T08:22:15.155 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:22:14 smithi050 ceph-mon[32371]: 1.8 scrub starts 2024-09-18T08:22:15.155 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:22:14 smithi050 ceph-mon[32371]: 1.8 scrub ok 2024-09-18T08:22:15.155 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:22:14 smithi050 ceph-mon[32371]: 4.1c scrub starts 2024-09-18T08:22:15.155 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:22:14 smithi050 ceph-mon[32371]: 4.1c scrub ok 2024-09-18T08:22:15.155 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:22:14 smithi050 ceph-mon[32371]: pgmap v973: 142 pgs: 142 active+clean; 635 KiB data, 5.3 GiB used, 710 GiB / 715 GiB avail; 5.0 KiB/s rd, 25 KiB/s wr, 4 op/s; 18 KiB/s, 0 objects/s recovering 2024-09-18T08:22:15.156 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:22:14 smithi050 ceph-mon[32371]: from='client.? 172.21.15.7:0/3952606338' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]': finished 2024-09-18T08:22:15.156 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:22:14 smithi050 ceph-mon[32371]: osdmap e476: 8 total, 8 up, 8 in 2024-09-18T08:22:15.156 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:22:14 smithi050 ceph-mon[32371]: 3.1f scrub starts 2024-09-18T08:22:15.156 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:22:14 smithi050 ceph-mon[32371]: 3.1f scrub ok 2024-09-18T08:22:15.718 INFO:teuthology.orchestra.run.smithi007.stderr:set osd.7 primary-affinity to 1 (8655362) 2024-09-18T08:22:15.980 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:22:15 smithi007 ceph-mon[29459]: 3.d scrub starts 2024-09-18T08:22:15.980 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:22:15 smithi007 ceph-mon[29459]: 3.d scrub ok 2024-09-18T08:22:15.980 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:22:15 smithi007 ceph-mon[29459]: 1.7 scrub starts 2024-09-18T08:22:15.980 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:22:15 smithi007 ceph-mon[29459]: 1.7 scrub ok 2024-09-18T08:22:15.980 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:22:15 smithi007 ceph-mon[29459]: 5.8 scrub starts 2024-09-18T08:22:15.980 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:22:15 smithi007 ceph-mon[29459]: 5.8 scrub ok 2024-09-18T08:22:15.980 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:22:15 smithi007 ceph-mon[29459]: from='client.? 172.21.15.7:0/1494593760' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 7, "weight": 1.0} : dispatch 2024-09-18T08:22:15.981 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:22:15 smithi007 ceph-mon[39249]: 3.d scrub starts 2024-09-18T08:22:15.981 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:22:15 smithi007 ceph-mon[39249]: 3.d scrub ok 2024-09-18T08:22:15.981 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:22:15 smithi007 ceph-mon[39249]: 1.7 scrub starts 2024-09-18T08:22:15.981 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:22:15 smithi007 ceph-mon[39249]: 1.7 scrub ok 2024-09-18T08:22:15.981 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:22:15 smithi007 ceph-mon[39249]: 5.8 scrub starts 2024-09-18T08:22:15.981 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:22:15 smithi007 ceph-mon[39249]: 5.8 scrub ok 2024-09-18T08:22:15.981 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:22:15 smithi007 ceph-mon[39249]: from='client.? 172.21.15.7:0/1494593760' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 7, "weight": 1.0} : dispatch 2024-09-18T08:22:16.155 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:22:15 smithi050 ceph-mon[32371]: 3.d scrub starts 2024-09-18T08:22:16.155 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:22:15 smithi050 ceph-mon[32371]: 3.d scrub ok 2024-09-18T08:22:16.155 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:22:15 smithi050 ceph-mon[32371]: 1.7 scrub starts 2024-09-18T08:22:16.155 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:22:15 smithi050 ceph-mon[32371]: 1.7 scrub ok 2024-09-18T08:22:16.155 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:22:15 smithi050 ceph-mon[32371]: 5.8 scrub starts 2024-09-18T08:22:16.155 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:22:15 smithi050 ceph-mon[32371]: 5.8 scrub ok 2024-09-18T08:22:16.155 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:22:15 smithi050 ceph-mon[32371]: from='client.? 172.21.15.7:0/1494593760' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 7, "weight": 1.0} : dispatch 2024-09-18T08:22:16.519 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid 779349ca-7593-11ef-bceb-c7b262605968 -- ceph osd reweight 1 1 2024-09-18T08:22:16.789 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/779349ca-7593-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T08:22:16.911 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Sep 18 08:22:16 smithi007 ceph-779349ca-7593-11ef-bceb-c7b262605968-mgr-y[29688]: ::ffff:172.21.15.50 - - [18/Sep/2024:08:22:16] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-09-18T08:22:16.911 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:22:16 smithi007 ceph-mon[29459]: 5.7 scrub starts 2024-09-18T08:22:16.911 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:22:16 smithi007 ceph-mon[29459]: 5.7 scrub ok 2024-09-18T08:22:16.911 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:22:16 smithi007 ceph-mon[29459]: pgmap v975: 142 pgs: 142 active+clean; 635 KiB data, 5.3 GiB used, 710 GiB / 715 GiB avail; 4.1 KiB/s rd, 25 KiB/s wr, 4 op/s; 18 KiB/s, 0 objects/s recovering 2024-09-18T08:22:16.911 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:22:16 smithi007 ceph-mon[29459]: from='client.? 172.21.15.7:0/1494593760' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 7, "weight": 1.0}]': finished 2024-09-18T08:22:16.911 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:22:16 smithi007 ceph-mon[29459]: osdmap e477: 8 total, 8 up, 8 in 2024-09-18T08:22:16.912 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:22:16 smithi007 ceph-mon[29459]: 5.1e scrub starts 2024-09-18T08:22:16.912 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:22:16 smithi007 ceph-mon[29459]: 5.1e scrub ok 2024-09-18T08:22:16.912 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:22:16 smithi007 ceph-mon[29459]: 3.14 scrub starts 2024-09-18T08:22:16.912 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:22:16 smithi007 ceph-mon[29459]: 3.14 scrub ok 2024-09-18T08:22:16.912 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:22:16 smithi007 ceph-mon[39249]: 5.7 scrub starts 2024-09-18T08:22:16.912 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:22:16 smithi007 ceph-mon[39249]: 5.7 scrub ok 2024-09-18T08:22:16.912 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:22:16 smithi007 ceph-mon[39249]: pgmap v975: 142 pgs: 142 active+clean; 635 KiB data, 5.3 GiB used, 710 GiB / 715 GiB avail; 4.1 KiB/s rd, 25 KiB/s wr, 4 op/s; 18 KiB/s, 0 objects/s recovering 2024-09-18T08:22:16.912 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:22:16 smithi007 ceph-mon[39249]: from='client.? 172.21.15.7:0/1494593760' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 7, "weight": 1.0}]': finished 2024-09-18T08:22:16.912 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:22:16 smithi007 ceph-mon[39249]: osdmap e477: 8 total, 8 up, 8 in 2024-09-18T08:22:16.912 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:22:16 smithi007 ceph-mon[39249]: 5.1e scrub starts 2024-09-18T08:22:16.913 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:22:16 smithi007 ceph-mon[39249]: 5.1e scrub ok 2024-09-18T08:22:16.913 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:22:16 smithi007 ceph-mon[39249]: 3.14 scrub starts 2024-09-18T08:22:16.913 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:22:16 smithi007 ceph-mon[39249]: 3.14 scrub ok 2024-09-18T08:22:17.155 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:22:16 smithi050 ceph-mon[32371]: 5.7 scrub starts 2024-09-18T08:22:17.155 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:22:16 smithi050 ceph-mon[32371]: 5.7 scrub ok 2024-09-18T08:22:17.155 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:22:16 smithi050 ceph-mon[32371]: pgmap v975: 142 pgs: 142 active+clean; 635 KiB data, 5.3 GiB used, 710 GiB / 715 GiB avail; 4.1 KiB/s rd, 25 KiB/s wr, 4 op/s; 18 KiB/s, 0 objects/s recovering 2024-09-18T08:22:17.155 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:22:16 smithi050 ceph-mon[32371]: from='client.? 172.21.15.7:0/1494593760' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 7, "weight": 1.0}]': finished 2024-09-18T08:22:17.155 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:22:16 smithi050 ceph-mon[32371]: osdmap e477: 8 total, 8 up, 8 in 2024-09-18T08:22:17.155 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:22:16 smithi050 ceph-mon[32371]: 5.1e scrub starts 2024-09-18T08:22:17.155 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:22:16 smithi050 ceph-mon[32371]: 5.1e scrub ok 2024-09-18T08:22:17.156 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:22:16 smithi050 ceph-mon[32371]: 3.14 scrub starts 2024-09-18T08:22:17.156 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:22:16 smithi050 ceph-mon[32371]: 3.14 scrub ok 2024-09-18T08:22:17.734 INFO:teuthology.orchestra.run.smithi007.stderr:reweighted osd.1 to 1 (10000) 2024-09-18T08:22:18.107 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:22:17 smithi007 ceph-mon[39249]: 4.a scrub starts 2024-09-18T08:22:18.107 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:22:17 smithi007 ceph-mon[39249]: 4.a scrub ok 2024-09-18T08:22:18.107 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:22:17 smithi007 ceph-mon[39249]: osdmap e478: 8 total, 8 up, 8 in 2024-09-18T08:22:18.107 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:22:17 smithi007 ceph-mon[39249]: 3.9 scrub starts 2024-09-18T08:22:18.107 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:22:17 smithi007 ceph-mon[39249]: 3.9 scrub ok 2024-09-18T08:22:18.107 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:22:17 smithi007 ceph-mon[39249]: from='client.? 172.21.15.7:0/57934755' entity='client.admin' cmd={"prefix": "osd reweight", "id": 1, "weight": 1.0} : dispatch 2024-09-18T08:22:18.107 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:22:17 smithi007 ceph-mon[29459]: 4.a scrub starts 2024-09-18T08:22:18.108 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:22:17 smithi007 ceph-mon[29459]: 4.a scrub ok 2024-09-18T08:22:18.108 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:22:17 smithi007 ceph-mon[29459]: osdmap e478: 8 total, 8 up, 8 in 2024-09-18T08:22:18.108 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:22:17 smithi007 ceph-mon[29459]: 3.9 scrub starts 2024-09-18T08:22:18.108 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:22:17 smithi007 ceph-mon[29459]: 3.9 scrub ok 2024-09-18T08:22:18.108 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:22:17 smithi007 ceph-mon[29459]: from='client.? 172.21.15.7:0/57934755' entity='client.admin' cmd={"prefix": "osd reweight", "id": 1, "weight": 1.0} : dispatch 2024-09-18T08:22:18.155 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:22:17 smithi050 ceph-mon[32371]: 4.a scrub starts 2024-09-18T08:22:18.155 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:22:17 smithi050 ceph-mon[32371]: 4.a scrub ok 2024-09-18T08:22:18.155 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:22:17 smithi050 ceph-mon[32371]: osdmap e478: 8 total, 8 up, 8 in 2024-09-18T08:22:18.155 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:22:17 smithi050 ceph-mon[32371]: 3.9 scrub starts 2024-09-18T08:22:18.155 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:22:17 smithi050 ceph-mon[32371]: 3.9 scrub ok 2024-09-18T08:22:18.155 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:22:17 smithi050 ceph-mon[32371]: from='client.? 172.21.15.7:0/57934755' entity='client.admin' cmd={"prefix": "osd reweight", "id": 1, "weight": 1.0} : dispatch 2024-09-18T08:22:18.455 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid 779349ca-7593-11ef-bceb-c7b262605968 -- ceph osd primary-affinity 1 1 2024-09-18T08:22:18.715 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/779349ca-7593-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T08:22:19.036 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:22:18 smithi007 ceph-mon[39249]: 5.2 scrub starts 2024-09-18T08:22:19.036 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:22:18 smithi007 ceph-mon[39249]: 5.2 scrub ok 2024-09-18T08:22:19.036 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:22:18 smithi007 ceph-mon[39249]: pgmap v978: 142 pgs: 142 active+clean; 635 KiB data, 5.3 GiB used, 710 GiB / 715 GiB avail; 5.0 KiB/s rd, 31 KiB/s wr, 4 op/s; 22 KiB/s, 0 objects/s recovering 2024-09-18T08:22:19.036 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:22:18 smithi007 ceph-mon[39249]: 6.5 scrub starts 2024-09-18T08:22:19.037 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:22:18 smithi007 ceph-mon[39249]: from='client.? 172.21.15.7:0/57934755' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]': finished 2024-09-18T08:22:19.037 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:22:18 smithi007 ceph-mon[39249]: 6.5 scrub ok 2024-09-18T08:22:19.037 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:22:18 smithi007 ceph-mon[39249]: osdmap e479: 8 total, 8 up, 8 in 2024-09-18T08:22:19.037 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:22:18 smithi007 ceph-mon[39249]: 4.b scrub starts 2024-09-18T08:22:19.037 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:22:18 smithi007 ceph-mon[39249]: 4.b scrub ok 2024-09-18T08:22:19.037 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:22:18 smithi007 ceph-mon[39249]: 4.1a scrub starts 2024-09-18T08:22:19.037 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:22:18 smithi007 ceph-mon[39249]: 4.1a scrub ok 2024-09-18T08:22:19.037 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:22:18 smithi007 ceph-mon[29459]: 5.2 scrub starts 2024-09-18T08:22:19.037 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:22:18 smithi007 ceph-mon[29459]: 5.2 scrub ok 2024-09-18T08:22:19.037 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:22:18 smithi007 ceph-mon[29459]: pgmap v978: 142 pgs: 142 active+clean; 635 KiB data, 5.3 GiB used, 710 GiB / 715 GiB avail; 5.0 KiB/s rd, 31 KiB/s wr, 4 op/s; 22 KiB/s, 0 objects/s recovering 2024-09-18T08:22:19.038 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:22:18 smithi007 ceph-mon[29459]: 6.5 scrub starts 2024-09-18T08:22:19.038 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:22:18 smithi007 ceph-mon[29459]: from='client.? 172.21.15.7:0/57934755' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]': finished 2024-09-18T08:22:19.038 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:22:18 smithi007 ceph-mon[29459]: 6.5 scrub ok 2024-09-18T08:22:19.038 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:22:18 smithi007 ceph-mon[29459]: osdmap e479: 8 total, 8 up, 8 in 2024-09-18T08:22:19.038 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:22:18 smithi007 ceph-mon[29459]: 4.b scrub starts 2024-09-18T08:22:19.038 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:22:18 smithi007 ceph-mon[29459]: 4.b scrub ok 2024-09-18T08:22:19.038 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:22:18 smithi007 ceph-mon[29459]: 4.1a scrub starts 2024-09-18T08:22:19.038 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:22:18 smithi007 ceph-mon[29459]: 4.1a scrub ok 2024-09-18T08:22:19.155 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:22:18 smithi050 ceph-mon[32371]: 5.2 scrub starts 2024-09-18T08:22:19.155 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:22:18 smithi050 ceph-mon[32371]: 5.2 scrub ok 2024-09-18T08:22:19.155 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:22:18 smithi050 ceph-mon[32371]: pgmap v978: 142 pgs: 142 active+clean; 635 KiB data, 5.3 GiB used, 710 GiB / 715 GiB avail; 5.0 KiB/s rd, 31 KiB/s wr, 4 op/s; 22 KiB/s, 0 objects/s recovering 2024-09-18T08:22:19.155 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:22:18 smithi050 ceph-mon[32371]: 6.5 scrub starts 2024-09-18T08:22:19.155 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:22:18 smithi050 ceph-mon[32371]: from='client.? 172.21.15.7:0/57934755' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]': finished 2024-09-18T08:22:19.155 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:22:18 smithi050 ceph-mon[32371]: 6.5 scrub ok 2024-09-18T08:22:19.155 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:22:18 smithi050 ceph-mon[32371]: osdmap e479: 8 total, 8 up, 8 in 2024-09-18T08:22:19.155 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:22:18 smithi050 ceph-mon[32371]: 4.b scrub starts 2024-09-18T08:22:19.155 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:22:18 smithi050 ceph-mon[32371]: 4.b scrub ok 2024-09-18T08:22:19.156 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:22:18 smithi050 ceph-mon[32371]: 4.1a scrub starts 2024-09-18T08:22:19.156 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:22:18 smithi050 ceph-mon[32371]: 4.1a scrub ok 2024-09-18T08:22:19.741 INFO:teuthology.orchestra.run.smithi007.stderr:set osd.1 primary-affinity to 1 (8655362) 2024-09-18T08:22:20.123 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:22:19 smithi007 ceph-mon[29459]: 3.10 scrub starts 2024-09-18T08:22:20.123 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:22:19 smithi007 ceph-mon[29459]: 3.10 scrub ok 2024-09-18T08:22:20.123 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:22:19 smithi007 ceph-mon[29459]: from='mgr.24470 172.21.15.7:0/2594390261' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-18T08:22:20.123 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:22:19 smithi007 ceph-mon[29459]: from='client.? 172.21.15.7:0/855329142' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 1, "weight": 1.0} : dispatch 2024-09-18T08:22:20.123 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:22:19 smithi007 ceph-mon[29459]: from='client.? ' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 1, "weight": 1.0} : dispatch 2024-09-18T08:22:20.123 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:22:19 smithi007 ceph-mon[39249]: 3.10 scrub starts 2024-09-18T08:22:20.124 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:22:19 smithi007 ceph-mon[39249]: 3.10 scrub ok 2024-09-18T08:22:20.124 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:22:19 smithi007 ceph-mon[39249]: from='mgr.24470 172.21.15.7:0/2594390261' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-18T08:22:20.124 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:22:19 smithi007 ceph-mon[39249]: from='client.? 172.21.15.7:0/855329142' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 1, "weight": 1.0} : dispatch 2024-09-18T08:22:20.124 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:22:19 smithi007 ceph-mon[39249]: from='client.? ' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 1, "weight": 1.0} : dispatch 2024-09-18T08:22:20.155 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:22:19 smithi050 ceph-mon[32371]: 3.10 scrub starts 2024-09-18T08:22:20.155 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:22:19 smithi050 ceph-mon[32371]: 3.10 scrub ok 2024-09-18T08:22:20.155 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:22:19 smithi050 ceph-mon[32371]: from='mgr.24470 172.21.15.7:0/2594390261' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-18T08:22:20.155 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:22:19 smithi050 ceph-mon[32371]: from='client.? 172.21.15.7:0/855329142' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 1, "weight": 1.0} : dispatch 2024-09-18T08:22:20.156 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:22:19 smithi050 ceph-mon[32371]: from='client.? ' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 1, "weight": 1.0} : dispatch 2024-09-18T08:22:20.156 INFO:journalctl@ceph.iscsi.iscsi.a.smithi050.stdout:Sep 18 08:22:19 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-iscsi-iscsi-a[57782]: debug there is no tcmu-runner data available 2024-09-18T08:22:20.659 INFO:tasks.thrashosds.thrasher:joining the do_sighup greenlet 2024-09-18T08:22:20.659 INFO:tasks.thrashosds.thrasher:joining the do_optrack_toggle greenlet 2024-09-18T08:22:20.659 INFO:tasks.thrashosds.thrasher:joining the do_dump_ops greenlet 2024-09-18T08:22:20.659 INFO:tasks.thrashosds.thrasher:joining the do_noscrub_toggle greenlet 2024-09-18T08:22:20.659 INFO:tasks.cephadm.ceph_manager.ceph:waiting for all up 2024-09-18T08:22:20.659 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid 779349ca-7593-11ef-bceb-c7b262605968 -- ceph osd dump --format=json 2024-09-18T08:22:20.925 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/779349ca-7593-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T08:22:20.979 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:22:20 smithi007 ceph-mon[29459]: 3.2 scrub starts 2024-09-18T08:22:20.979 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:22:20 smithi007 ceph-mon[29459]: 3.2 scrub ok 2024-09-18T08:22:20.980 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:22:20 smithi007 ceph-mon[29459]: 3.1c scrub starts 2024-09-18T08:22:20.980 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:22:20 smithi007 ceph-mon[29459]: 3.1c scrub ok 2024-09-18T08:22:20.980 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:22:20 smithi007 ceph-mon[29459]: pgmap v980: 142 pgs: 142 active+clean; 583 KiB data, 5.3 GiB used, 710 GiB / 715 GiB avail 2024-09-18T08:22:20.980 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:22:20 smithi007 ceph-mon[29459]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 1, "weight": 1.0}]': finished 2024-09-18T08:22:20.980 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:22:20 smithi007 ceph-mon[29459]: osdmap e480: 8 total, 8 up, 8 in 2024-09-18T08:22:20.980 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:22:20 smithi007 ceph-mon[39249]: 3.2 scrub starts 2024-09-18T08:22:20.980 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:22:20 smithi007 ceph-mon[39249]: 3.2 scrub ok 2024-09-18T08:22:20.980 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:22:20 smithi007 ceph-mon[39249]: 3.1c scrub starts 2024-09-18T08:22:20.980 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:22:20 smithi007 ceph-mon[39249]: 3.1c scrub ok 2024-09-18T08:22:20.980 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:22:20 smithi007 ceph-mon[39249]: pgmap v980: 142 pgs: 142 active+clean; 583 KiB data, 5.3 GiB used, 710 GiB / 715 GiB avail 2024-09-18T08:22:20.981 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:22:20 smithi007 ceph-mon[39249]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 1, "weight": 1.0}]': finished 2024-09-18T08:22:20.981 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:22:20 smithi007 ceph-mon[39249]: osdmap e480: 8 total, 8 up, 8 in 2024-09-18T08:22:21.155 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:22:20 smithi050 ceph-mon[32371]: 3.2 scrub starts 2024-09-18T08:22:21.155 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:22:20 smithi050 ceph-mon[32371]: 3.2 scrub ok 2024-09-18T08:22:21.155 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:22:20 smithi050 ceph-mon[32371]: 3.1c scrub starts 2024-09-18T08:22:21.155 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:22:20 smithi050 ceph-mon[32371]: 3.1c scrub ok 2024-09-18T08:22:21.155 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:22:20 smithi050 ceph-mon[32371]: pgmap v980: 142 pgs: 142 active+clean; 583 KiB data, 5.3 GiB used, 710 GiB / 715 GiB avail 2024-09-18T08:22:21.155 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:22:20 smithi050 ceph-mon[32371]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 1, "weight": 1.0}]': finished 2024-09-18T08:22:21.155 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:22:20 smithi050 ceph-mon[32371]: osdmap e480: 8 total, 8 up, 8 in 2024-09-18T08:22:21.584 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-09-18T08:22:21.584 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":480,"fsid":"779349ca-7593-11ef-bceb-c7b262605968","created":"2024-09-18T07:57:09.295710+0000","modified":"2024-09-18T08:22:19.735014+0000","last_up_change":"2024-09-18T08:14:38.391465+0000","last_in_change":"2024-09-18T08:21:44.556333+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":19,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":11,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-18T08:00:11.802627+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":11,"pg_placement_num":11,"pg_placement_num_target":11,"pg_num_target":11,"pg_num_pending":11,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"447","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"152","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":2.1800000667572021,"score_stable":2.1800000667572021,"optimal_score":1,"raw_score_acting":2.1800000667572021,"raw_score_stable":2.1800000667572021,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"datapool","create_time":"2024-09-18T08:02:45.374099+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":3,"pg_placement_num":3,"pg_placement_num_target":3,"pg_num_target":3,"pg_num_pending":3,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"64","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":2,"snap_epoch":64,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_type":"Fair distribution","score_acting":2.6500000953674316,"score_stable":2.6500000953674316,"optimal_score":0.87999999523162842,"raw_score_acting":2.3299999237060547,"raw_score_stable":2.3299999237060547,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":".rgw.root","create_time":"2024-09-18T08:02:46.319947+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"63","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rgw":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"default.rgw.log","create_time":"2024-09-18T08:02:47.948552+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"65","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rgw":{}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"default.rgw.control","create_time":"2024-09-18T08:02:49.904302+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"67","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rgw":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.25,"score_stable":1.25,"optimal_score":1,"raw_score_acting":1.25,"raw_score_stable":1.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":6,"pool_name":"default.rgw.meta","create_time":"2024-09-18T08:02:51.971465+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"70","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4},"application_metadata":{"rgw":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"ae8f60df-72ac-46fb-91a1-f132b851c876","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":463,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6801","nonce":2896425794}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6802","nonce":2896425794}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6804","nonce":2896425794}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6803","nonce":2896425794}]},"public_addr":"172.21.15.7:6801/2896425794","cluster_addr":"172.21.15.7:6802/2896425794","heartbeat_back_addr":"172.21.15.7:6804/2896425794","heartbeat_front_addr":"172.21.15.7:6803/2896425794","state":["exists","up"]},{"osd":1,"uuid":"be28b70c-1f73-494f-ad4b-cb7ec5311999","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":13,"last_clean_end":243,"up_from":307,"up_thru":463,"down_at":244,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6805","nonce":3300107791}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6806","nonce":3300107791}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6808","nonce":3300107791}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6807","nonce":3300107791}]},"public_addr":"172.21.15.7:6805/3300107791","cluster_addr":"172.21.15.7:6806/3300107791","heartbeat_back_addr":"172.21.15.7:6808/3300107791","heartbeat_front_addr":"172.21.15.7:6807/3300107791","state":["exists","up"]},{"osd":2,"uuid":"cfa4f083-c7fd-4743-be65-84a3040fe6dc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":463,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6809","nonce":556142559}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6810","nonce":556142559}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6812","nonce":556142559}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6811","nonce":556142559}]},"public_addr":"172.21.15.7:6809/556142559","cluster_addr":"172.21.15.7:6810/556142559","heartbeat_back_addr":"172.21.15.7:6812/556142559","heartbeat_front_addr":"172.21.15.7:6811/556142559","state":["exists","up"]},{"osd":3,"uuid":"11ecd7e1-a751-4b75-95f5-00b32d2a5bc4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":463,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6813","nonce":163587772}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6814","nonce":163587772}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6816","nonce":163587772}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6815","nonce":163587772}]},"public_addr":"172.21.15.7:6813/163587772","cluster_addr":"172.21.15.7:6814/163587772","heartbeat_back_addr":"172.21.15.7:6816/163587772","heartbeat_front_addr":"172.21.15.7:6815/163587772","state":["exists","up"]},{"osd":4,"uuid":"98992b34-cfa1-45bf-97a1-7d557739252f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":32,"up_thru":463,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":3662119805}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6801","nonce":3662119805}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6803","nonce":3662119805}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":3662119805}]},"public_addr":"172.21.15.50:6800/3662119805","cluster_addr":"172.21.15.50:6801/3662119805","heartbeat_back_addr":"172.21.15.50:6803/3662119805","heartbeat_front_addr":"172.21.15.50:6802/3662119805","state":["exists","up"]},{"osd":5,"uuid":"7f290005-ebca-467b-87e4-02ead1063610","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":39,"up_thru":459,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":1199797954}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6805","nonce":1199797954}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6807","nonce":1199797954}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":1199797954}]},"public_addr":"172.21.15.50:6804/1199797954","cluster_addr":"172.21.15.50:6805/1199797954","heartbeat_back_addr":"172.21.15.50:6807/1199797954","heartbeat_front_addr":"172.21.15.50:6806/1199797954","state":["exists","up"]},{"osd":6,"uuid":"e07d846c-ae42-4fea-aa5c-3a25835d18a6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":47,"up_thru":474,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":255960422}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6809","nonce":255960422}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6811","nonce":255960422}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":255960422}]},"public_addr":"172.21.15.50:6808/255960422","cluster_addr":"172.21.15.50:6809/255960422","heartbeat_back_addr":"172.21.15.50:6811/255960422","heartbeat_front_addr":"172.21.15.50:6810/255960422","state":["exists","up"]},{"osd":7,"uuid":"c9dac3b7-d651-489d-b96e-5b10fa32d9af","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":56,"up_thru":477,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":874923904}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6813","nonce":874923904}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6815","nonce":874923904}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":874923904}]},"public_addr":"172.21.15.50:6812/874923904","cluster_addr":"172.21.15.50:6813/874923904","heartbeat_back_addr":"172.21.15.50:6815/874923904","heartbeat_front_addr":"172.21.15.50:6814/874923904","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-18T08:21:38.297199+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-18T08:21:47.774036+0000","dead_epoch":243},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-18T08:21:10.105152+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-18T08:22:00.911883+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-18T08:21:59.131462+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-18T08:22:05.415320+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-18T08:21:53.282148+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-18T08:22:00.582047+0000","dead_epoch":0}],"pg_upmap":[{"pgid":"3.c","osds":[3,2,6]},{"pgid":"6.3","osds":[5,3,2]}],"pg_upmap_items":[{"pgid":"3.8","mappings":[{"from":3,"to":2}]},{"pgid":"4.7","mappings":[{"from":1,"to":2}]},{"pgid":"4.14","mappings":[{"from":1,"to":2}]},{"pgid":"4.18","mappings":[{"from":1,"to":2}]},{"pgid":"4.1b","mappings":[{"from":3,"to":5}]},{"pgid":"4.1d","mappings":[{"from":1,"to":0},{"from":3,"to":5}]},{"pgid":"5.0","mappings":[{"from":1,"to":6}]},{"pgid":"5.b","mappings":[{"from":0,"to":3}]},{"pgid":"5.12","mappings":[{"from":1,"to":6}]},{"pgid":"6.8","mappings":[{"from":7,"to":3}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.7:0/398367658":"2024-09-19T08:03:17.052903+0000","172.21.15.7:6800/2918496291":"2024-09-19T07:57:31.015269+0000","172.21.15.7:0/2514354773":"2024-09-19T07:57:31.015269+0000","172.21.15.7:0/3924911335":"2024-09-19T07:57:31.015269+0000","172.21.15.7:0/1668598084":"2024-09-19T07:57:57.346149+0000","172.21.15.7:0/4272027292":"2024-09-19T07:57:31.015269+0000","172.21.15.7:6800/1013061071":"2024-09-19T07:57:57.346149+0000","172.21.15.7:6800/1491061076":"2024-09-19T08:03:17.052903+0000","172.21.15.7:0/533727535":"2024-09-19T07:57:57.346149+0000","172.21.15.7:0/1605217610":"2024-09-19T08:03:17.052903+0000","172.21.15.7:0/1722968065":"2024-09-19T08:03:17.052903+0000","172.21.15.7:0/1277549846":"2024-09-19T07:57:57.346149+0000","172.21.15.7:0/232452890":"2024-09-19T08:03:17.052903+0000","172.21.15.7:0/3881670470":"2024-09-19T08:03:17.052903+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-18T08:22:21.980 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:22:21 smithi007 ceph-mon[29459]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-18T08:22:21.980 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:22:21 smithi007 ceph-mon[29459]: 4.19 scrub starts 2024-09-18T08:22:21.980 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:22:21 smithi007 ceph-mon[29459]: 4.19 scrub ok 2024-09-18T08:22:21.980 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:22:21 smithi007 ceph-mon[29459]: 6.1c scrub starts 2024-09-18T08:22:21.980 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:22:21 smithi007 ceph-mon[29459]: 6.1c scrub ok 2024-09-18T08:22:21.980 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:22:21 smithi007 ceph-mon[29459]: from='client.? 172.21.15.7:0/4158352012' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2024-09-18T08:22:21.980 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:22:21 smithi007 ceph-mon[39249]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-18T08:22:21.981 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:22:21 smithi007 ceph-mon[39249]: 4.19 scrub starts 2024-09-18T08:22:21.981 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:22:21 smithi007 ceph-mon[39249]: 4.19 scrub ok 2024-09-18T08:22:21.981 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:22:21 smithi007 ceph-mon[39249]: 6.1c scrub starts 2024-09-18T08:22:21.981 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:22:21 smithi007 ceph-mon[39249]: 6.1c scrub ok 2024-09-18T08:22:21.981 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:22:21 smithi007 ceph-mon[39249]: from='client.? 172.21.15.7:0/4158352012' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2024-09-18T08:22:22.155 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:22:21 smithi050 ceph-mon[32371]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-18T08:22:22.155 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:22:21 smithi050 ceph-mon[32371]: 4.19 scrub starts 2024-09-18T08:22:22.155 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:22:21 smithi050 ceph-mon[32371]: 4.19 scrub ok 2024-09-18T08:22:22.155 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:22:21 smithi050 ceph-mon[32371]: 6.1c scrub starts 2024-09-18T08:22:22.155 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:22:21 smithi050 ceph-mon[32371]: 6.1c scrub ok 2024-09-18T08:22:22.155 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:22:21 smithi050 ceph-mon[32371]: from='client.? 172.21.15.7:0/4158352012' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2024-09-18T08:22:22.494 INFO:tasks.cephadm.ceph_manager.ceph:all up! 2024-09-18T08:22:22.495 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid 779349ca-7593-11ef-bceb-c7b262605968 -- ceph osd dump --format=json 2024-09-18T08:22:22.762 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/779349ca-7593-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T08:22:23.015 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:22:22 smithi007 ceph-mon[29459]: pgmap v982: 142 pgs: 142 active+clean; 583 KiB data, 5.3 GiB used, 710 GiB / 715 GiB avail 2024-09-18T08:22:23.015 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:22:22 smithi007 ceph-mon[39249]: pgmap v982: 142 pgs: 142 active+clean; 583 KiB data, 5.3 GiB used, 710 GiB / 715 GiB avail 2024-09-18T08:22:23.155 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:22:22 smithi050 ceph-mon[32371]: pgmap v982: 142 pgs: 142 active+clean; 583 KiB data, 5.3 GiB used, 710 GiB / 715 GiB avail 2024-09-18T08:22:23.402 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-09-18T08:22:23.403 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":480,"fsid":"779349ca-7593-11ef-bceb-c7b262605968","created":"2024-09-18T07:57:09.295710+0000","modified":"2024-09-18T08:22:19.735014+0000","last_up_change":"2024-09-18T08:14:38.391465+0000","last_in_change":"2024-09-18T08:21:44.556333+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":19,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":11,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-18T08:00:11.802627+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":11,"pg_placement_num":11,"pg_placement_num_target":11,"pg_num_target":11,"pg_num_pending":11,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"447","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"152","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":2.1800000667572021,"score_stable":2.1800000667572021,"optimal_score":1,"raw_score_acting":2.1800000667572021,"raw_score_stable":2.1800000667572021,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"datapool","create_time":"2024-09-18T08:02:45.374099+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":3,"pg_placement_num":3,"pg_placement_num_target":3,"pg_num_target":3,"pg_num_pending":3,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"64","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":2,"snap_epoch":64,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_type":"Fair distribution","score_acting":2.6500000953674316,"score_stable":2.6500000953674316,"optimal_score":0.87999999523162842,"raw_score_acting":2.3299999237060547,"raw_score_stable":2.3299999237060547,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":".rgw.root","create_time":"2024-09-18T08:02:46.319947+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"63","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rgw":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"default.rgw.log","create_time":"2024-09-18T08:02:47.948552+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"65","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rgw":{}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"default.rgw.control","create_time":"2024-09-18T08:02:49.904302+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"67","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rgw":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.25,"score_stable":1.25,"optimal_score":1,"raw_score_acting":1.25,"raw_score_stable":1.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":6,"pool_name":"default.rgw.meta","create_time":"2024-09-18T08:02:51.971465+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"70","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4},"application_metadata":{"rgw":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"ae8f60df-72ac-46fb-91a1-f132b851c876","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":463,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6801","nonce":2896425794}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6802","nonce":2896425794}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6804","nonce":2896425794}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6803","nonce":2896425794}]},"public_addr":"172.21.15.7:6801/2896425794","cluster_addr":"172.21.15.7:6802/2896425794","heartbeat_back_addr":"172.21.15.7:6804/2896425794","heartbeat_front_addr":"172.21.15.7:6803/2896425794","state":["exists","up"]},{"osd":1,"uuid":"be28b70c-1f73-494f-ad4b-cb7ec5311999","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":13,"last_clean_end":243,"up_from":307,"up_thru":463,"down_at":244,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6805","nonce":3300107791}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6806","nonce":3300107791}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6808","nonce":3300107791}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6807","nonce":3300107791}]},"public_addr":"172.21.15.7:6805/3300107791","cluster_addr":"172.21.15.7:6806/3300107791","heartbeat_back_addr":"172.21.15.7:6808/3300107791","heartbeat_front_addr":"172.21.15.7:6807/3300107791","state":["exists","up"]},{"osd":2,"uuid":"cfa4f083-c7fd-4743-be65-84a3040fe6dc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":463,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6809","nonce":556142559}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6810","nonce":556142559}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6812","nonce":556142559}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6811","nonce":556142559}]},"public_addr":"172.21.15.7:6809/556142559","cluster_addr":"172.21.15.7:6810/556142559","heartbeat_back_addr":"172.21.15.7:6812/556142559","heartbeat_front_addr":"172.21.15.7:6811/556142559","state":["exists","up"]},{"osd":3,"uuid":"11ecd7e1-a751-4b75-95f5-00b32d2a5bc4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":463,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6813","nonce":163587772}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6814","nonce":163587772}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6816","nonce":163587772}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6815","nonce":163587772}]},"public_addr":"172.21.15.7:6813/163587772","cluster_addr":"172.21.15.7:6814/163587772","heartbeat_back_addr":"172.21.15.7:6816/163587772","heartbeat_front_addr":"172.21.15.7:6815/163587772","state":["exists","up"]},{"osd":4,"uuid":"98992b34-cfa1-45bf-97a1-7d557739252f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":32,"up_thru":463,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":3662119805}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6801","nonce":3662119805}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6803","nonce":3662119805}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":3662119805}]},"public_addr":"172.21.15.50:6800/3662119805","cluster_addr":"172.21.15.50:6801/3662119805","heartbeat_back_addr":"172.21.15.50:6803/3662119805","heartbeat_front_addr":"172.21.15.50:6802/3662119805","state":["exists","up"]},{"osd":5,"uuid":"7f290005-ebca-467b-87e4-02ead1063610","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":39,"up_thru":459,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":1199797954}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6805","nonce":1199797954}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6807","nonce":1199797954}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":1199797954}]},"public_addr":"172.21.15.50:6804/1199797954","cluster_addr":"172.21.15.50:6805/1199797954","heartbeat_back_addr":"172.21.15.50:6807/1199797954","heartbeat_front_addr":"172.21.15.50:6806/1199797954","state":["exists","up"]},{"osd":6,"uuid":"e07d846c-ae42-4fea-aa5c-3a25835d18a6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":47,"up_thru":474,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":255960422}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6809","nonce":255960422}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6811","nonce":255960422}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":255960422}]},"public_addr":"172.21.15.50:6808/255960422","cluster_addr":"172.21.15.50:6809/255960422","heartbeat_back_addr":"172.21.15.50:6811/255960422","heartbeat_front_addr":"172.21.15.50:6810/255960422","state":["exists","up"]},{"osd":7,"uuid":"c9dac3b7-d651-489d-b96e-5b10fa32d9af","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":56,"up_thru":477,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":874923904}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6813","nonce":874923904}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6815","nonce":874923904}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":874923904}]},"public_addr":"172.21.15.50:6812/874923904","cluster_addr":"172.21.15.50:6813/874923904","heartbeat_back_addr":"172.21.15.50:6815/874923904","heartbeat_front_addr":"172.21.15.50:6814/874923904","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-18T08:21:38.297199+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-18T08:21:47.774036+0000","dead_epoch":243},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-18T08:21:10.105152+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-18T08:22:00.911883+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-18T08:21:59.131462+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-18T08:22:05.415320+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-18T08:21:53.282148+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-18T08:22:00.582047+0000","dead_epoch":0}],"pg_upmap":[{"pgid":"3.c","osds":[3,2,6]},{"pgid":"6.3","osds":[5,3,2]}],"pg_upmap_items":[{"pgid":"3.8","mappings":[{"from":3,"to":2}]},{"pgid":"4.7","mappings":[{"from":1,"to":2}]},{"pgid":"4.14","mappings":[{"from":1,"to":2}]},{"pgid":"4.18","mappings":[{"from":1,"to":2}]},{"pgid":"4.1b","mappings":[{"from":3,"to":5}]},{"pgid":"4.1d","mappings":[{"from":1,"to":0},{"from":3,"to":5}]},{"pgid":"5.0","mappings":[{"from":1,"to":6}]},{"pgid":"5.b","mappings":[{"from":0,"to":3}]},{"pgid":"5.12","mappings":[{"from":1,"to":6}]},{"pgid":"6.8","mappings":[{"from":7,"to":3}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.7:0/398367658":"2024-09-19T08:03:17.052903+0000","172.21.15.7:6800/2918496291":"2024-09-19T07:57:31.015269+0000","172.21.15.7:0/2514354773":"2024-09-19T07:57:31.015269+0000","172.21.15.7:0/3924911335":"2024-09-19T07:57:31.015269+0000","172.21.15.7:0/1668598084":"2024-09-19T07:57:57.346149+0000","172.21.15.7:0/4272027292":"2024-09-19T07:57:31.015269+0000","172.21.15.7:6800/1013061071":"2024-09-19T07:57:57.346149+0000","172.21.15.7:6800/1491061076":"2024-09-19T08:03:17.052903+0000","172.21.15.7:0/533727535":"2024-09-19T07:57:57.346149+0000","172.21.15.7:0/1605217610":"2024-09-19T08:03:17.052903+0000","172.21.15.7:0/1722968065":"2024-09-19T08:03:17.052903+0000","172.21.15.7:0/1277549846":"2024-09-19T07:57:57.346149+0000","172.21.15.7:0/232452890":"2024-09-19T08:03:17.052903+0000","172.21.15.7:0/3881670470":"2024-09-19T08:03:17.052903+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-18T08:22:24.155 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:22:23 smithi050 ceph-mon[32371]: 5.5 scrub starts 2024-09-18T08:22:24.155 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:22:23 smithi050 ceph-mon[32371]: 5.5 scrub ok 2024-09-18T08:22:24.155 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:22:23 smithi050 ceph-mon[32371]: 4.18 scrub starts 2024-09-18T08:22:24.155 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:22:23 smithi050 ceph-mon[32371]: 4.18 scrub ok 2024-09-18T08:22:24.155 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:22:23 smithi050 ceph-mon[32371]: from='client.? 172.21.15.7:0/867106790' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2024-09-18T08:22:24.193 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:22:23 smithi007 ceph-mon[29459]: 5.5 scrub starts 2024-09-18T08:22:24.194 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:22:23 smithi007 ceph-mon[29459]: 5.5 scrub ok 2024-09-18T08:22:24.194 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:22:23 smithi007 ceph-mon[29459]: 4.18 scrub starts 2024-09-18T08:22:24.194 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:22:23 smithi007 ceph-mon[29459]: 4.18 scrub ok 2024-09-18T08:22:24.194 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:22:23 smithi007 ceph-mon[29459]: from='client.? 172.21.15.7:0/867106790' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2024-09-18T08:22:24.194 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:22:23 smithi007 ceph-mon[39249]: 5.5 scrub starts 2024-09-18T08:22:24.194 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:22:23 smithi007 ceph-mon[39249]: 5.5 scrub ok 2024-09-18T08:22:24.194 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:22:23 smithi007 ceph-mon[39249]: 4.18 scrub starts 2024-09-18T08:22:24.194 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:22:23 smithi007 ceph-mon[39249]: 4.18 scrub ok 2024-09-18T08:22:24.194 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:22:23 smithi007 ceph-mon[39249]: from='client.? 172.21.15.7:0/867106790' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2024-09-18T08:22:24.229 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid 779349ca-7593-11ef-bceb-c7b262605968 -- ceph tell osd.0 flush_pg_stats 2024-09-18T08:22:24.229 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid 779349ca-7593-11ef-bceb-c7b262605968 -- ceph tell osd.1 flush_pg_stats 2024-09-18T08:22:24.230 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid 779349ca-7593-11ef-bceb-c7b262605968 -- ceph tell osd.2 flush_pg_stats 2024-09-18T08:22:24.230 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid 779349ca-7593-11ef-bceb-c7b262605968 -- ceph tell osd.3 flush_pg_stats 2024-09-18T08:22:24.230 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid 779349ca-7593-11ef-bceb-c7b262605968 -- ceph tell osd.4 flush_pg_stats 2024-09-18T08:22:24.231 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid 779349ca-7593-11ef-bceb-c7b262605968 -- ceph tell osd.5 flush_pg_stats 2024-09-18T08:22:24.231 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid 779349ca-7593-11ef-bceb-c7b262605968 -- ceph tell osd.6 flush_pg_stats 2024-09-18T08:22:24.232 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid 779349ca-7593-11ef-bceb-c7b262605968 -- ceph tell osd.7 flush_pg_stats 2024-09-18T08:22:24.839 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/779349ca-7593-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T08:22:24.840 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/779349ca-7593-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T08:22:24.840 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/779349ca-7593-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T08:22:24.844 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/779349ca-7593-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T08:22:24.844 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/779349ca-7593-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T08:22:24.844 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/779349ca-7593-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T08:22:24.845 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/779349ca-7593-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T08:22:24.846 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/779349ca-7593-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T08:22:25.096 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:22:24 smithi007 ceph-mon[29459]: 3.7 scrub starts 2024-09-18T08:22:25.096 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:22:24 smithi007 ceph-mon[29459]: 3.7 scrub ok 2024-09-18T08:22:25.096 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:22:24 smithi007 ceph-mon[29459]: 3.5 scrub starts 2024-09-18T08:22:25.096 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:22:24 smithi007 ceph-mon[29459]: 3.5 scrub ok 2024-09-18T08:22:25.096 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:22:24 smithi007 ceph-mon[29459]: pgmap v983: 142 pgs: 142 active+clean; 583 KiB data, 5.3 GiB used, 710 GiB / 715 GiB avail 2024-09-18T08:22:25.096 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:22:24 smithi007 ceph-mon[29459]: 6.2 scrub starts 2024-09-18T08:22:25.097 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:22:24 smithi007 ceph-mon[29459]: 6.2 scrub ok 2024-09-18T08:22:25.097 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:22:24 smithi007 ceph-mon[39249]: 3.7 scrub starts 2024-09-18T08:22:25.097 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:22:24 smithi007 ceph-mon[39249]: 3.7 scrub ok 2024-09-18T08:22:25.097 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:22:24 smithi007 ceph-mon[39249]: 3.5 scrub starts 2024-09-18T08:22:25.097 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:22:24 smithi007 ceph-mon[39249]: 3.5 scrub ok 2024-09-18T08:22:25.097 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:22:24 smithi007 ceph-mon[39249]: pgmap v983: 142 pgs: 142 active+clean; 583 KiB data, 5.3 GiB used, 710 GiB / 715 GiB avail 2024-09-18T08:22:25.097 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:22:24 smithi007 ceph-mon[39249]: 6.2 scrub starts 2024-09-18T08:22:25.097 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:22:24 smithi007 ceph-mon[39249]: 6.2 scrub ok 2024-09-18T08:22:25.155 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:22:24 smithi050 ceph-mon[32371]: 3.7 scrub starts 2024-09-18T08:22:25.155 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:22:24 smithi050 ceph-mon[32371]: 3.7 scrub ok 2024-09-18T08:22:25.155 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:22:24 smithi050 ceph-mon[32371]: 3.5 scrub starts 2024-09-18T08:22:25.155 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:22:24 smithi050 ceph-mon[32371]: 3.5 scrub ok 2024-09-18T08:22:25.155 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:22:24 smithi050 ceph-mon[32371]: pgmap v983: 142 pgs: 142 active+clean; 583 KiB data, 5.3 GiB used, 710 GiB / 715 GiB avail 2024-09-18T08:22:25.155 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:22:24 smithi050 ceph-mon[32371]: 6.2 scrub starts 2024-09-18T08:22:25.155 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:22:24 smithi050 ceph-mon[32371]: 6.2 scrub ok 2024-09-18T08:22:25.910 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:22:25 smithi007 ceph-mon[29459]: 1.1 scrub starts 2024-09-18T08:22:25.910 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:22:25 smithi007 ceph-mon[29459]: 1.1 scrub ok 2024-09-18T08:22:25.911 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:22:25 smithi007 ceph-mon[29459]: 3.11 scrub starts 2024-09-18T08:22:25.911 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:22:25 smithi007 ceph-mon[29459]: 3.11 scrub ok 2024-09-18T08:22:25.911 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:22:25 smithi007 ceph-mon[29459]: 6.16 scrub starts 2024-09-18T08:22:25.911 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:22:25 smithi007 ceph-mon[29459]: 6.16 scrub ok 2024-09-18T08:22:25.911 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:22:25 smithi007 ceph-mon[29459]: 3.1a scrub starts 2024-09-18T08:22:25.911 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:22:25 smithi007 ceph-mon[29459]: 3.1a scrub ok 2024-09-18T08:22:25.911 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:22:25 smithi007 ceph-mon[39249]: 1.1 scrub starts 2024-09-18T08:22:25.911 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:22:25 smithi007 ceph-mon[39249]: 1.1 scrub ok 2024-09-18T08:22:25.911 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:22:25 smithi007 ceph-mon[39249]: 3.11 scrub starts 2024-09-18T08:22:25.911 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:22:25 smithi007 ceph-mon[39249]: 3.11 scrub ok 2024-09-18T08:22:25.912 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:22:25 smithi007 ceph-mon[39249]: 6.16 scrub starts 2024-09-18T08:22:25.912 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:22:25 smithi007 ceph-mon[39249]: 6.16 scrub ok 2024-09-18T08:22:25.912 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:22:25 smithi007 ceph-mon[39249]: 3.1a scrub starts 2024-09-18T08:22:25.912 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:22:25 smithi007 ceph-mon[39249]: 3.1a scrub ok 2024-09-18T08:22:26.155 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:22:25 smithi050 ceph-mon[32371]: 1.1 scrub starts 2024-09-18T08:22:26.155 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:22:25 smithi050 ceph-mon[32371]: 1.1 scrub ok 2024-09-18T08:22:26.155 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:22:25 smithi050 ceph-mon[32371]: 3.11 scrub starts 2024-09-18T08:22:26.155 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:22:25 smithi050 ceph-mon[32371]: 3.11 scrub ok 2024-09-18T08:22:26.155 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:22:25 smithi050 ceph-mon[32371]: 6.16 scrub starts 2024-09-18T08:22:26.155 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:22:25 smithi050 ceph-mon[32371]: 6.16 scrub ok 2024-09-18T08:22:26.155 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:22:25 smithi050 ceph-mon[32371]: 3.1a scrub starts 2024-09-18T08:22:26.156 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:22:25 smithi050 ceph-mon[32371]: 3.1a scrub ok 2024-09-18T08:22:26.830 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Sep 18 08:22:26 smithi007 ceph-779349ca-7593-11ef-bceb-c7b262605968-mgr-y[29688]: ::ffff:172.21.15.50 - - [18/Sep/2024:08:22:26] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-09-18T08:22:26.831 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:22:26 smithi007 ceph-mon[29459]: 5.12 scrub starts 2024-09-18T08:22:26.831 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:22:26 smithi007 ceph-mon[29459]: 5.12 scrub ok 2024-09-18T08:22:26.831 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:22:26 smithi007 ceph-mon[29459]: pgmap v984: 142 pgs: 142 active+clean; 583 KiB data, 5.3 GiB used, 710 GiB / 715 GiB avail 2024-09-18T08:22:26.831 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:22:26 smithi007 ceph-mon[29459]: 6.e scrub starts 2024-09-18T08:22:26.831 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:22:26 smithi007 ceph-mon[29459]: 6.e scrub ok 2024-09-18T08:22:26.831 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:22:26 smithi007 ceph-mon[39249]: 5.12 scrub starts 2024-09-18T08:22:26.831 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:22:26 smithi007 ceph-mon[39249]: 5.12 scrub ok 2024-09-18T08:22:26.832 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:22:26 smithi007 ceph-mon[39249]: pgmap v984: 142 pgs: 142 active+clean; 583 KiB data, 5.3 GiB used, 710 GiB / 715 GiB avail 2024-09-18T08:22:26.832 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:22:26 smithi007 ceph-mon[39249]: 6.e scrub starts 2024-09-18T08:22:26.832 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:22:26 smithi007 ceph-mon[39249]: 6.e scrub ok 2024-09-18T08:22:27.155 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:22:26 smithi050 ceph-mon[32371]: 5.12 scrub starts 2024-09-18T08:22:27.155 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:22:26 smithi050 ceph-mon[32371]: 5.12 scrub ok 2024-09-18T08:22:27.155 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:22:26 smithi050 ceph-mon[32371]: pgmap v984: 142 pgs: 142 active+clean; 583 KiB data, 5.3 GiB used, 710 GiB / 715 GiB avail 2024-09-18T08:22:27.155 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:22:26 smithi050 ceph-mon[32371]: 6.e scrub starts 2024-09-18T08:22:27.155 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:22:26 smithi050 ceph-mon[32371]: 6.e scrub ok 2024-09-18T08:22:27.369 INFO:teuthology.orchestra.run.smithi007.stdout:34359738649 2024-09-18T08:22:27.370 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid 779349ca-7593-11ef-bceb-c7b262605968 -- ceph osd last-stat-seq osd.0 2024-09-18T08:22:27.936 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:22:27 smithi007 ceph-mon[29459]: 1.4 scrub starts 2024-09-18T08:22:27.936 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:22:27 smithi007 ceph-mon[29459]: 1.4 scrub ok 2024-09-18T08:22:27.936 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:22:27 smithi007 ceph-mon[39249]: 1.4 scrub starts 2024-09-18T08:22:27.936 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:22:27 smithi007 ceph-mon[39249]: 1.4 scrub ok 2024-09-18T08:22:28.155 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:22:27 smithi050 ceph-mon[32371]: 1.4 scrub starts 2024-09-18T08:22:28.155 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:22:27 smithi050 ceph-mon[32371]: 1.4 scrub ok 2024-09-18T08:22:28.979 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:22:28 smithi007 ceph-mon[39249]: 6.1f scrub starts 2024-09-18T08:22:28.980 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:22:28 smithi007 ceph-mon[39249]: 6.1f scrub ok 2024-09-18T08:22:28.980 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:22:28 smithi007 ceph-mon[39249]: 1.a scrub starts 2024-09-18T08:22:28.980 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:22:28 smithi007 ceph-mon[39249]: 1.a scrub ok 2024-09-18T08:22:28.980 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:22:28 smithi007 ceph-mon[39249]: pgmap v985: 142 pgs: 142 active+clean; 583 KiB data, 5.3 GiB used, 710 GiB / 715 GiB avail 2024-09-18T08:22:28.980 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:22:28 smithi007 ceph-mon[39249]: 6.18 scrub starts 2024-09-18T08:22:28.980 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:22:28 smithi007 ceph-mon[39249]: 6.18 scrub ok 2024-09-18T08:22:28.980 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:22:28 smithi007 ceph-mon[29459]: 6.1f scrub starts 2024-09-18T08:22:28.980 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:22:28 smithi007 ceph-mon[29459]: 6.1f scrub ok 2024-09-18T08:22:28.981 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:22:28 smithi007 ceph-mon[29459]: 1.a scrub starts 2024-09-18T08:22:28.981 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:22:28 smithi007 ceph-mon[29459]: 1.a scrub ok 2024-09-18T08:22:28.981 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:22:28 smithi007 ceph-mon[29459]: pgmap v985: 142 pgs: 142 active+clean; 583 KiB data, 5.3 GiB used, 710 GiB / 715 GiB avail 2024-09-18T08:22:28.981 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:22:28 smithi007 ceph-mon[29459]: 6.18 scrub starts 2024-09-18T08:22:28.981 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:22:28 smithi007 ceph-mon[29459]: 6.18 scrub ok 2024-09-18T08:22:29.155 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:22:28 smithi050 ceph-mon[32371]: 6.1f scrub starts 2024-09-18T08:22:29.155 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:22:28 smithi050 ceph-mon[32371]: 6.1f scrub ok 2024-09-18T08:22:29.155 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:22:28 smithi050 ceph-mon[32371]: 1.a scrub starts 2024-09-18T08:22:29.155 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:22:28 smithi050 ceph-mon[32371]: 1.a scrub ok 2024-09-18T08:22:29.155 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:22:28 smithi050 ceph-mon[32371]: pgmap v985: 142 pgs: 142 active+clean; 583 KiB data, 5.3 GiB used, 710 GiB / 715 GiB avail 2024-09-18T08:22:29.155 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:22:28 smithi050 ceph-mon[32371]: 6.18 scrub starts 2024-09-18T08:22:29.155 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:22:28 smithi050 ceph-mon[32371]: 6.18 scrub ok 2024-09-18T08:22:29.327 INFO:teuthology.orchestra.run.smithi007.stdout:167503724796 2024-09-18T08:22:29.328 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid 779349ca-7593-11ef-bceb-c7b262605968 -- ceph osd last-stat-seq osd.5 2024-09-18T08:22:30.060 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:22:29 smithi007 ceph-mon[29459]: 6.6 scrub starts 2024-09-18T08:22:30.061 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:22:29 smithi007 ceph-mon[29459]: 6.6 scrub ok 2024-09-18T08:22:30.061 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:22:29 smithi007 ceph-mon[39249]: 6.6 scrub starts 2024-09-18T08:22:30.061 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:22:29 smithi007 ceph-mon[39249]: 6.6 scrub ok 2024-09-18T08:22:30.155 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:22:29 smithi050 ceph-mon[32371]: 6.6 scrub starts 2024-09-18T08:22:30.155 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:22:29 smithi050 ceph-mon[32371]: 6.6 scrub ok 2024-09-18T08:22:30.155 INFO:journalctl@ceph.iscsi.iscsi.a.smithi050.stdout:Sep 18 08:22:29 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-iscsi-iscsi-a[57782]: debug there is no tcmu-runner data available 2024-09-18T08:22:30.908 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:22:30 smithi007 ceph-mon[39249]: 4.8 scrub starts 2024-09-18T08:22:30.908 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:22:30 smithi007 ceph-mon[39249]: 4.8 scrub ok 2024-09-18T08:22:30.908 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:22:30 smithi007 ceph-mon[39249]: pgmap v986: 142 pgs: 142 active+clean; 583 KiB data, 5.3 GiB used, 710 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-18T08:22:30.909 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:22:30 smithi007 ceph-mon[39249]: 4.d scrub starts 2024-09-18T08:22:30.909 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:22:30 smithi007 ceph-mon[39249]: 4.d scrub ok 2024-09-18T08:22:30.909 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:22:30 smithi007 ceph-mon[29459]: 4.8 scrub starts 2024-09-18T08:22:30.909 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:22:30 smithi007 ceph-mon[29459]: 4.8 scrub ok 2024-09-18T08:22:30.909 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:22:30 smithi007 ceph-mon[29459]: pgmap v986: 142 pgs: 142 active+clean; 583 KiB data, 5.3 GiB used, 710 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-18T08:22:30.909 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:22:30 smithi007 ceph-mon[29459]: 4.d scrub starts 2024-09-18T08:22:30.909 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:22:30 smithi007 ceph-mon[29459]: 4.d scrub ok 2024-09-18T08:22:31.155 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:22:30 smithi050 ceph-mon[32371]: 4.8 scrub starts 2024-09-18T08:22:31.155 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:22:30 smithi050 ceph-mon[32371]: 4.8 scrub ok 2024-09-18T08:22:31.155 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:22:30 smithi050 ceph-mon[32371]: pgmap v986: 142 pgs: 142 active+clean; 583 KiB data, 5.3 GiB used, 710 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-18T08:22:31.155 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:22:30 smithi050 ceph-mon[32371]: 4.d scrub starts 2024-09-18T08:22:31.155 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:22:30 smithi050 ceph-mon[32371]: 4.d scrub ok 2024-09-18T08:22:31.377 INFO:teuthology.orchestra.run.smithi007.stdout:137438953730 2024-09-18T08:22:31.377 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid 779349ca-7593-11ef-bceb-c7b262605968 -- ceph osd last-stat-seq osd.4 2024-09-18T08:22:31.891 INFO:teuthology.orchestra.run.smithi007.stdout:201863463159 2024-09-18T08:22:31.891 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid 779349ca-7593-11ef-bceb-c7b262605968 -- ceph osd last-stat-seq osd.6 2024-09-18T08:22:32.155 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:22:31 smithi050 ceph-mon[32371]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-18T08:22:32.155 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:22:31 smithi050 ceph-mon[32371]: purged_snaps scrub starts 2024-09-18T08:22:32.155 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:22:31 smithi050 ceph-mon[32371]: purged_snaps scrub ok 2024-09-18T08:22:32.155 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:22:31 smithi050 ceph-mon[32371]: 4.1e scrub starts 2024-09-18T08:22:32.155 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:22:31 smithi050 ceph-mon[32371]: 4.1e scrub ok 2024-09-18T08:22:32.155 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:22:31 smithi050 ceph-mon[32371]: osdmap e481: 8 total, 8 up, 8 in 2024-09-18T08:22:32.229 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:22:31 smithi007 ceph-mon[29459]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-18T08:22:32.230 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:22:31 smithi007 ceph-mon[29459]: purged_snaps scrub starts 2024-09-18T08:22:32.230 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:22:31 smithi007 ceph-mon[29459]: purged_snaps scrub ok 2024-09-18T08:22:32.230 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:22:31 smithi007 ceph-mon[29459]: 4.1e scrub starts 2024-09-18T08:22:32.230 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:22:31 smithi007 ceph-mon[29459]: 4.1e scrub ok 2024-09-18T08:22:32.230 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:22:31 smithi007 ceph-mon[29459]: osdmap e481: 8 total, 8 up, 8 in 2024-09-18T08:22:32.230 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:22:31 smithi007 ceph-mon[39249]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-18T08:22:32.230 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:22:31 smithi007 ceph-mon[39249]: purged_snaps scrub starts 2024-09-18T08:22:32.231 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:22:31 smithi007 ceph-mon[39249]: purged_snaps scrub ok 2024-09-18T08:22:32.231 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:22:31 smithi007 ceph-mon[39249]: 4.1e scrub starts 2024-09-18T08:22:32.231 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:22:31 smithi007 ceph-mon[39249]: 4.1e scrub ok 2024-09-18T08:22:32.231 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:22:31 smithi007 ceph-mon[39249]: osdmap e481: 8 total, 8 up, 8 in 2024-09-18T08:22:32.290 INFO:teuthology.orchestra.run.smithi007.stdout:240518168817 2024-09-18T08:22:32.290 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid 779349ca-7593-11ef-bceb-c7b262605968 -- ceph osd last-stat-seq osd.7 2024-09-18T08:22:32.980 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:22:32 smithi007 ceph-mon[29459]: 5.15 scrub starts 2024-09-18T08:22:32.980 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:22:32 smithi007 ceph-mon[29459]: 5.15 scrub ok 2024-09-18T08:22:32.980 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:22:32 smithi007 ceph-mon[29459]: pgmap v988: 142 pgs: 142 active+clean; 583 KiB data, 5.3 GiB used, 710 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-18T08:22:32.981 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:22:32 smithi007 ceph-mon[39249]: 5.15 scrub starts 2024-09-18T08:22:32.981 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:22:32 smithi007 ceph-mon[39249]: 5.15 scrub ok 2024-09-18T08:22:32.981 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:22:32 smithi007 ceph-mon[39249]: pgmap v988: 142 pgs: 142 active+clean; 583 KiB data, 5.3 GiB used, 710 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-18T08:22:33.018 INFO:teuthology.orchestra.run.smithi007.stdout:77309411598 2024-09-18T08:22:33.018 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid 779349ca-7593-11ef-bceb-c7b262605968 -- ceph osd last-stat-seq osd.2 2024-09-18T08:22:33.155 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:22:32 smithi050 ceph-mon[32371]: 5.15 scrub starts 2024-09-18T08:22:33.155 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:22:32 smithi050 ceph-mon[32371]: 5.15 scrub ok 2024-09-18T08:22:33.155 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:22:32 smithi050 ceph-mon[32371]: pgmap v988: 142 pgs: 142 active+clean; 583 KiB data, 5.3 GiB used, 710 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-18T08:22:33.238 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/779349ca-7593-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T08:22:33.740 INFO:teuthology.orchestra.run.smithi007.stdout:1318554959967 2024-09-18T08:22:33.740 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid 779349ca-7593-11ef-bceb-c7b262605968 -- ceph osd last-stat-seq osd.1 2024-09-18T08:22:33.979 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:22:33 smithi007 ceph-mon[29459]: 5.19 scrub starts 2024-09-18T08:22:33.980 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:22:33 smithi007 ceph-mon[29459]: 5.19 scrub ok 2024-09-18T08:22:33.980 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:22:33 smithi007 ceph-mon[29459]: 6.3 scrub starts 2024-09-18T08:22:33.980 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:22:33 smithi007 ceph-mon[29459]: 6.3 scrub ok 2024-09-18T08:22:33.980 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:22:33 smithi007 ceph-mon[39249]: 5.19 scrub starts 2024-09-18T08:22:33.980 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:22:33 smithi007 ceph-mon[39249]: 5.19 scrub ok 2024-09-18T08:22:33.980 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:22:33 smithi007 ceph-mon[39249]: 6.3 scrub starts 2024-09-18T08:22:33.980 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:22:33 smithi007 ceph-mon[39249]: 6.3 scrub ok 2024-09-18T08:22:34.025 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/779349ca-7593-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T08:22:34.026 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/779349ca-7593-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T08:22:34.155 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:22:33 smithi050 ceph-mon[32371]: 5.19 scrub starts 2024-09-18T08:22:34.155 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:22:33 smithi050 ceph-mon[32371]: 5.19 scrub ok 2024-09-18T08:22:34.155 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:22:33 smithi050 ceph-mon[32371]: 6.3 scrub starts 2024-09-18T08:22:34.155 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:22:33 smithi050 ceph-mon[32371]: 6.3 scrub ok 2024-09-18T08:22:34.889 INFO:teuthology.orchestra.run.smithi007.stdout:167503724797 2024-09-18T08:22:34.979 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:22:34 smithi007 ceph-mon[39249]: pgmap v989: 142 pgs: 142 active+clean; 583 KiB data, 5.3 GiB used, 710 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-18T08:22:34.980 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:22:34 smithi007 ceph-mon[39249]: 5.3 scrub starts 2024-09-18T08:22:34.980 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:22:34 smithi007 ceph-mon[39249]: 5.3 scrub ok 2024-09-18T08:22:34.980 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:22:34 smithi007 ceph-mon[39249]: from='mgr.24470 172.21.15.7:0/2594390261' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-18T08:22:34.980 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:22:34 smithi007 ceph-mon[29459]: pgmap v989: 142 pgs: 142 active+clean; 583 KiB data, 5.3 GiB used, 710 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-18T08:22:34.980 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:22:34 smithi007 ceph-mon[29459]: 5.3 scrub starts 2024-09-18T08:22:34.980 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:22:34 smithi007 ceph-mon[29459]: 5.3 scrub ok 2024-09-18T08:22:34.980 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:22:34 smithi007 ceph-mon[29459]: from='mgr.24470 172.21.15.7:0/2594390261' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-18T08:22:35.155 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:22:34 smithi050 ceph-mon[32371]: pgmap v989: 142 pgs: 142 active+clean; 583 KiB data, 5.3 GiB used, 710 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-18T08:22:35.155 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:22:34 smithi050 ceph-mon[32371]: 5.3 scrub starts 2024-09-18T08:22:35.155 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:22:34 smithi050 ceph-mon[32371]: 5.3 scrub ok 2024-09-18T08:22:35.155 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:22:34 smithi050 ceph-mon[32371]: from='mgr.24470 172.21.15.7:0/2594390261' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-18T08:22:35.156 INFO:teuthology.orchestra.run.smithi007.stdout:111669149961 2024-09-18T08:22:35.157 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid 779349ca-7593-11ef-bceb-c7b262605968 -- ceph osd last-stat-seq osd.3 2024-09-18T08:22:35.831 INFO:tasks.cephadm.ceph_manager.ceph:need seq 167503724796 got 167503724797 for osd.5 2024-09-18T08:22:35.831 DEBUG:teuthology.parallel:result is None 2024-09-18T08:22:36.036 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:22:35 smithi007 ceph-mon[39249]: from='mgr.24470 172.21.15.7:0/2594390261' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-18T08:22:36.036 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:22:35 smithi007 ceph-mon[39249]: from='client.? 172.21.15.7:0/1936784636' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 5} : dispatch 2024-09-18T08:22:36.037 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:22:35 smithi007 ceph-mon[29459]: from='mgr.24470 172.21.15.7:0/2594390261' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-18T08:22:36.037 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:22:35 smithi007 ceph-mon[29459]: from='client.? 172.21.15.7:0/1936784636' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 5} : dispatch 2024-09-18T08:22:36.155 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:22:35 smithi050 ceph-mon[32371]: from='mgr.24470 172.21.15.7:0/2594390261' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-18T08:22:36.155 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:22:35 smithi050 ceph-mon[32371]: from='client.? 172.21.15.7:0/1936784636' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 5} : dispatch 2024-09-18T08:22:36.325 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/779349ca-7593-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T08:22:36.518 INFO:teuthology.orchestra.run.smithi007.stdout:137438953731 2024-09-18T08:22:36.807 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/779349ca-7593-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T08:22:36.807 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/779349ca-7593-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T08:22:36.900 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Sep 18 08:22:36 smithi007 ceph-779349ca-7593-11ef-bceb-c7b262605968-mgr-y[29688]: ::ffff:172.21.15.50 - - [18/Sep/2024:08:22:36] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-09-18T08:22:36.901 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:22:36 smithi007 ceph-mon[29459]: 3.18 scrub starts 2024-09-18T08:22:36.901 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:22:36 smithi007 ceph-mon[29459]: 3.18 scrub ok 2024-09-18T08:22:36.901 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:22:36 smithi007 ceph-mon[29459]: pgmap v990: 142 pgs: 142 active+clean; 583 KiB data, 5.3 GiB used, 710 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-18T08:22:36.901 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:22:36 smithi007 ceph-mon[29459]: 6.15 scrub starts 2024-09-18T08:22:36.901 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:22:36 smithi007 ceph-mon[29459]: 6.15 scrub ok 2024-09-18T08:22:36.901 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:22:36 smithi007 ceph-mon[29459]: from='mgr.24470 172.21.15.7:0/2594390261' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-18T08:22:36.901 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:22:36 smithi007 ceph-mon[29459]: from='mgr.24470 172.21.15.7:0/2594390261' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-18T08:22:36.901 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:22:36 smithi007 ceph-mon[29459]: from='mgr.24470 172.21.15.7:0/2594390261' entity='mgr.y' 2024-09-18T08:22:36.901 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:22:36 smithi007 ceph-mon[29459]: 4.9 scrub starts 2024-09-18T08:22:36.901 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:22:36 smithi007 ceph-mon[29459]: 4.9 scrub ok 2024-09-18T08:22:36.902 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:22:36 smithi007 ceph-mon[29459]: from='client.? 172.21.15.7:0/292572791' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 4} : dispatch 2024-09-18T08:22:36.902 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:22:36 smithi007 ceph-mon[39249]: 3.18 scrub starts 2024-09-18T08:22:36.902 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:22:36 smithi007 ceph-mon[39249]: 3.18 scrub ok 2024-09-18T08:22:36.902 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:22:36 smithi007 ceph-mon[39249]: pgmap v990: 142 pgs: 142 active+clean; 583 KiB data, 5.3 GiB used, 710 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-18T08:22:36.902 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:22:36 smithi007 ceph-mon[39249]: 6.15 scrub starts 2024-09-18T08:22:36.902 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:22:36 smithi007 ceph-mon[39249]: 6.15 scrub ok 2024-09-18T08:22:36.902 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:22:36 smithi007 ceph-mon[39249]: from='mgr.24470 172.21.15.7:0/2594390261' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-18T08:22:36.902 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:22:36 smithi007 ceph-mon[39249]: from='mgr.24470 172.21.15.7:0/2594390261' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-18T08:22:36.902 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:22:36 smithi007 ceph-mon[39249]: from='mgr.24470 172.21.15.7:0/2594390261' entity='mgr.y' 2024-09-18T08:22:36.902 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:22:36 smithi007 ceph-mon[39249]: 4.9 scrub starts 2024-09-18T08:22:36.903 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:22:36 smithi007 ceph-mon[39249]: 4.9 scrub ok 2024-09-18T08:22:36.903 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:22:36 smithi007 ceph-mon[39249]: from='client.? 172.21.15.7:0/292572791' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 4} : dispatch 2024-09-18T08:22:36.979 INFO:teuthology.orchestra.run.smithi007.stdout:240518168818 2024-09-18T08:22:37.155 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:22:36 smithi050 ceph-mon[32371]: 3.18 scrub starts 2024-09-18T08:22:37.155 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:22:36 smithi050 ceph-mon[32371]: 3.18 scrub ok 2024-09-18T08:22:37.155 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:22:36 smithi050 ceph-mon[32371]: pgmap v990: 142 pgs: 142 active+clean; 583 KiB data, 5.3 GiB used, 710 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-18T08:22:37.155 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:22:36 smithi050 ceph-mon[32371]: 6.15 scrub starts 2024-09-18T08:22:37.155 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:22:36 smithi050 ceph-mon[32371]: 6.15 scrub ok 2024-09-18T08:22:37.155 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:22:36 smithi050 ceph-mon[32371]: from='mgr.24470 172.21.15.7:0/2594390261' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-18T08:22:37.156 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:22:36 smithi050 ceph-mon[32371]: from='mgr.24470 172.21.15.7:0/2594390261' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-18T08:22:37.156 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:22:36 smithi050 ceph-mon[32371]: from='mgr.24470 172.21.15.7:0/2594390261' entity='mgr.y' 2024-09-18T08:22:37.156 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:22:36 smithi050 ceph-mon[32371]: 4.9 scrub starts 2024-09-18T08:22:37.156 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:22:36 smithi050 ceph-mon[32371]: 4.9 scrub ok 2024-09-18T08:22:37.156 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:22:36 smithi050 ceph-mon[32371]: from='client.? 172.21.15.7:0/292572791' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 4} : dispatch 2024-09-18T08:22:37.193 INFO:teuthology.orchestra.run.smithi007.stdout:77309411600 2024-09-18T08:22:37.831 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:22:37 smithi007 ceph-mon[29459]: 6.c scrub starts 2024-09-18T08:22:37.831 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:22:37 smithi007 ceph-mon[29459]: 6.c scrub ok 2024-09-18T08:22:37.831 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:22:37 smithi007 ceph-mon[29459]: 5.1b scrub starts 2024-09-18T08:22:37.831 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:22:37 smithi007 ceph-mon[29459]: 5.1b scrub ok 2024-09-18T08:22:37.831 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:22:37 smithi007 ceph-mon[29459]: from='client.? 172.21.15.7:0/626496514' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 7} : dispatch 2024-09-18T08:22:37.831 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:22:37 smithi007 ceph-mon[29459]: 3.1b scrub starts 2024-09-18T08:22:37.832 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:22:37 smithi007 ceph-mon[29459]: 3.1b scrub ok 2024-09-18T08:22:37.832 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:22:37 smithi007 ceph-mon[29459]: from='client.? 172.21.15.7:0/3507187887' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 2} : dispatch 2024-09-18T08:22:37.832 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:22:37 smithi007 ceph-mon[29459]: 5.18 scrub starts 2024-09-18T08:22:37.832 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:22:37 smithi007 ceph-mon[29459]: 5.18 scrub ok 2024-09-18T08:22:37.832 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:22:37 smithi007 ceph-mon[39249]: 6.c scrub starts 2024-09-18T08:22:37.832 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:22:37 smithi007 ceph-mon[39249]: 6.c scrub ok 2024-09-18T08:22:37.832 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:22:37 smithi007 ceph-mon[39249]: 5.1b scrub starts 2024-09-18T08:22:37.832 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:22:37 smithi007 ceph-mon[39249]: 5.1b scrub ok 2024-09-18T08:22:37.832 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:22:37 smithi007 ceph-mon[39249]: from='client.? 172.21.15.7:0/626496514' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 7} : dispatch 2024-09-18T08:22:37.832 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:22:37 smithi007 ceph-mon[39249]: 3.1b scrub starts 2024-09-18T08:22:37.832 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:22:37 smithi007 ceph-mon[39249]: 3.1b scrub ok 2024-09-18T08:22:37.833 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:22:37 smithi007 ceph-mon[39249]: from='client.? 172.21.15.7:0/3507187887' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 2} : dispatch 2024-09-18T08:22:37.833 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:22:37 smithi007 ceph-mon[39249]: 5.18 scrub starts 2024-09-18T08:22:37.833 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:22:37 smithi007 ceph-mon[39249]: 5.18 scrub ok 2024-09-18T08:22:37.863 INFO:tasks.cephadm.ceph_manager.ceph:need seq 240518168817 got 240518168818 for osd.7 2024-09-18T08:22:37.863 DEBUG:teuthology.parallel:result is None 2024-09-18T08:22:38.063 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/779349ca-7593-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T08:22:38.155 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:22:37 smithi050 ceph-mon[32371]: 6.c scrub starts 2024-09-18T08:22:38.155 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:22:37 smithi050 ceph-mon[32371]: 6.c scrub ok 2024-09-18T08:22:38.155 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:22:37 smithi050 ceph-mon[32371]: 5.1b scrub starts 2024-09-18T08:22:38.155 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:22:37 smithi050 ceph-mon[32371]: 5.1b scrub ok 2024-09-18T08:22:38.155 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:22:37 smithi050 ceph-mon[32371]: from='client.? 172.21.15.7:0/626496514' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 7} : dispatch 2024-09-18T08:22:38.155 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:22:37 smithi050 ceph-mon[32371]: 3.1b scrub starts 2024-09-18T08:22:38.155 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:22:37 smithi050 ceph-mon[32371]: 3.1b scrub ok 2024-09-18T08:22:38.156 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:22:37 smithi050 ceph-mon[32371]: from='client.? 172.21.15.7:0/3507187887' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 2} : dispatch 2024-09-18T08:22:38.156 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:22:37 smithi050 ceph-mon[32371]: 5.18 scrub starts 2024-09-18T08:22:38.156 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:22:37 smithi050 ceph-mon[32371]: 5.18 scrub ok 2024-09-18T08:22:38.679 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/779349ca-7593-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T08:22:38.708 INFO:tasks.cephadm.ceph_manager.ceph:need seq 77309411598 got 77309411600 for osd.2 2024-09-18T08:22:38.708 DEBUG:teuthology.parallel:result is None 2024-09-18T08:22:39.058 INFO:teuthology.orchestra.run.smithi007.stdout:201863463160 2024-09-18T08:22:39.058 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:22:38 smithi007 ceph-mon[29459]: 4.14 scrub starts 2024-09-18T08:22:39.058 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:22:38 smithi007 ceph-mon[29459]: 4.14 scrub ok 2024-09-18T08:22:39.059 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:22:38 smithi007 ceph-mon[29459]: pgmap v991: 142 pgs: 142 active+clean; 583 KiB data, 5.3 GiB used, 710 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-18T08:22:39.059 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:22:38 smithi007 ceph-mon[39249]: 4.14 scrub starts 2024-09-18T08:22:39.059 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:22:38 smithi007 ceph-mon[39249]: 4.14 scrub ok 2024-09-18T08:22:39.059 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:22:38 smithi007 ceph-mon[39249]: pgmap v991: 142 pgs: 142 active+clean; 583 KiB data, 5.3 GiB used, 710 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-18T08:22:39.155 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:22:38 smithi050 ceph-mon[32371]: 4.14 scrub starts 2024-09-18T08:22:39.155 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:22:38 smithi050 ceph-mon[32371]: 4.14 scrub ok 2024-09-18T08:22:39.155 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:22:38 smithi050 ceph-mon[32371]: pgmap v991: 142 pgs: 142 active+clean; 583 KiB data, 5.3 GiB used, 710 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-18T08:22:39.379 INFO:tasks.cephadm.ceph_manager.ceph:need seq 137438953730 got 137438953731 for osd.4 2024-09-18T08:22:39.379 DEBUG:teuthology.parallel:result is None 2024-09-18T08:22:40.006 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:22:39 smithi007 ceph-mon[29459]: 5.0 scrub starts 2024-09-18T08:22:40.006 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:22:39 smithi007 ceph-mon[29459]: 5.0 scrub ok 2024-09-18T08:22:40.006 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:22:39 smithi007 ceph-mon[29459]: from='client.? 172.21.15.7:0/926605492' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 6} : dispatch 2024-09-18T08:22:40.006 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:22:39 smithi007 ceph-mon[29459]: pgmap v992: 142 pgs: 142 active+clean; 583 KiB data, 5.3 GiB used, 710 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-18T08:22:40.006 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:22:39 smithi007 ceph-mon[39249]: 5.0 scrub starts 2024-09-18T08:22:40.006 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:22:39 smithi007 ceph-mon[39249]: 5.0 scrub ok 2024-09-18T08:22:40.007 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:22:39 smithi007 ceph-mon[39249]: from='client.? 172.21.15.7:0/926605492' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 6} : dispatch 2024-09-18T08:22:40.007 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:22:39 smithi007 ceph-mon[39249]: pgmap v992: 142 pgs: 142 active+clean; 583 KiB data, 5.3 GiB used, 710 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-18T08:22:40.033 INFO:tasks.cephadm.ceph_manager.ceph:need seq 201863463159 got 201863463160 for osd.6 2024-09-18T08:22:40.033 DEBUG:teuthology.parallel:result is None 2024-09-18T08:22:40.155 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:22:39 smithi050 ceph-mon[32371]: 5.0 scrub starts 2024-09-18T08:22:40.155 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:22:39 smithi050 ceph-mon[32371]: 5.0 scrub ok 2024-09-18T08:22:40.155 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:22:39 smithi050 ceph-mon[32371]: from='client.? 172.21.15.7:0/926605492' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 6} : dispatch 2024-09-18T08:22:40.155 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:22:39 smithi050 ceph-mon[32371]: pgmap v992: 142 pgs: 142 active+clean; 583 KiB data, 5.3 GiB used, 710 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-18T08:22:40.155 INFO:journalctl@ceph.iscsi.iscsi.a.smithi050.stdout:Sep 18 08:22:39 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-iscsi-iscsi-a[57782]: debug there is no tcmu-runner data available 2024-09-18T08:22:40.839 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:22:40 smithi007 ceph-mon[29459]: 4.4 scrub starts 2024-09-18T08:22:40.839 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:22:40 smithi007 ceph-mon[29459]: 4.4 scrub ok 2024-09-18T08:22:40.839 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:22:40 smithi007 ceph-mon[29459]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-18T08:22:40.839 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:22:40 smithi007 ceph-mon[29459]: 4.10 scrub starts 2024-09-18T08:22:40.839 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:22:40 smithi007 ceph-mon[29459]: 4.10 scrub ok 2024-09-18T08:22:40.840 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:22:40 smithi007 ceph-mon[39249]: 4.4 scrub starts 2024-09-18T08:22:40.840 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:22:40 smithi007 ceph-mon[39249]: 4.4 scrub ok 2024-09-18T08:22:40.840 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:22:40 smithi007 ceph-mon[39249]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-18T08:22:40.840 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:22:40 smithi007 ceph-mon[39249]: 4.10 scrub starts 2024-09-18T08:22:40.840 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:22:40 smithi007 ceph-mon[39249]: 4.10 scrub ok 2024-09-18T08:22:41.056 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:22:40 smithi050 ceph-mon[32371]: 4.4 scrub starts 2024-09-18T08:22:41.056 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:22:40 smithi050 ceph-mon[32371]: 4.4 scrub ok 2024-09-18T08:22:41.057 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:22:40 smithi050 ceph-mon[32371]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-18T08:22:41.057 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:22:40 smithi050 ceph-mon[32371]: 4.10 scrub starts 2024-09-18T08:22:41.057 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:22:40 smithi050 ceph-mon[32371]: 4.10 scrub ok 2024-09-18T08:22:41.287 INFO:teuthology.orchestra.run.smithi007.stdout:111669149962 2024-09-18T08:22:41.523 INFO:teuthology.orchestra.run.smithi007.stdout:34359738652 2024-09-18T08:22:42.155 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:22:41 smithi050 ceph-mon[32371]: 6.d scrub starts 2024-09-18T08:22:42.155 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:22:41 smithi050 ceph-mon[32371]: 6.d scrub ok 2024-09-18T08:22:42.155 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:22:41 smithi050 ceph-mon[32371]: 3.e scrub starts 2024-09-18T08:22:42.155 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:22:41 smithi050 ceph-mon[32371]: 3.e scrub ok 2024-09-18T08:22:42.155 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:22:41 smithi050 ceph-mon[32371]: from='client.? 172.21.15.7:0/1146758718' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 1} : dispatch 2024-09-18T08:22:42.155 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:22:41 smithi050 ceph-mon[32371]: 4.1 scrub starts 2024-09-18T08:22:42.155 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:22:41 smithi050 ceph-mon[32371]: 4.1 scrub ok 2024-09-18T08:22:42.156 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:22:41 smithi050 ceph-mon[32371]: from='client.? 172.21.15.7:0/2903534695' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 3} : dispatch 2024-09-18T08:22:42.156 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:22:41 smithi050 ceph-mon[32371]: from='client.? 172.21.15.7:0/595365932' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 0} : dispatch 2024-09-18T08:22:42.156 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:22:41 smithi050 ceph-mon[32371]: pgmap v993: 142 pgs: 142 active+clean; 583 KiB data, 5.3 GiB used, 710 GiB / 715 GiB avail; 817 B/s rd, 0 op/s 2024-09-18T08:22:42.156 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:22:41 smithi050 ceph-mon[32371]: 1.9 scrub starts 2024-09-18T08:22:42.156 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:22:41 smithi050 ceph-mon[32371]: 1.9 scrub ok 2024-09-18T08:22:42.229 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:22:41 smithi007 ceph-mon[29459]: 6.d scrub starts 2024-09-18T08:22:42.230 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:22:41 smithi007 ceph-mon[29459]: 6.d scrub ok 2024-09-18T08:22:42.230 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:22:41 smithi007 ceph-mon[29459]: 3.e scrub starts 2024-09-18T08:22:42.230 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:22:41 smithi007 ceph-mon[29459]: 3.e scrub ok 2024-09-18T08:22:42.230 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:22:41 smithi007 ceph-mon[29459]: from='client.? 172.21.15.7:0/1146758718' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 1} : dispatch 2024-09-18T08:22:42.230 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:22:41 smithi007 ceph-mon[29459]: 4.1 scrub starts 2024-09-18T08:22:42.230 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:22:41 smithi007 ceph-mon[29459]: 4.1 scrub ok 2024-09-18T08:22:42.230 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:22:41 smithi007 ceph-mon[29459]: from='client.? 172.21.15.7:0/2903534695' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 3} : dispatch 2024-09-18T08:22:42.230 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:22:41 smithi007 ceph-mon[29459]: from='client.? 172.21.15.7:0/595365932' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 0} : dispatch 2024-09-18T08:22:42.230 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:22:41 smithi007 ceph-mon[29459]: pgmap v993: 142 pgs: 142 active+clean; 583 KiB data, 5.3 GiB used, 710 GiB / 715 GiB avail; 817 B/s rd, 0 op/s 2024-09-18T08:22:42.230 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:22:41 smithi007 ceph-mon[29459]: 1.9 scrub starts 2024-09-18T08:22:42.230 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:22:41 smithi007 ceph-mon[29459]: 1.9 scrub ok 2024-09-18T08:22:42.231 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:22:41 smithi007 ceph-mon[39249]: 6.d scrub starts 2024-09-18T08:22:42.231 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:22:41 smithi007 ceph-mon[39249]: 6.d scrub ok 2024-09-18T08:22:42.231 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:22:41 smithi007 ceph-mon[39249]: 3.e scrub starts 2024-09-18T08:22:42.231 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:22:41 smithi007 ceph-mon[39249]: 3.e scrub ok 2024-09-18T08:22:42.231 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:22:41 smithi007 ceph-mon[39249]: from='client.? 172.21.15.7:0/1146758718' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 1} : dispatch 2024-09-18T08:22:42.231 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:22:41 smithi007 ceph-mon[39249]: 4.1 scrub starts 2024-09-18T08:22:42.231 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:22:41 smithi007 ceph-mon[39249]: 4.1 scrub ok 2024-09-18T08:22:42.231 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:22:41 smithi007 ceph-mon[39249]: from='client.? 172.21.15.7:0/2903534695' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 3} : dispatch 2024-09-18T08:22:42.231 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:22:41 smithi007 ceph-mon[39249]: from='client.? 172.21.15.7:0/595365932' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 0} : dispatch 2024-09-18T08:22:42.231 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:22:41 smithi007 ceph-mon[39249]: pgmap v993: 142 pgs: 142 active+clean; 583 KiB data, 5.3 GiB used, 710 GiB / 715 GiB avail; 817 B/s rd, 0 op/s 2024-09-18T08:22:42.231 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:22:41 smithi007 ceph-mon[39249]: 1.9 scrub starts 2024-09-18T08:22:42.231 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:22:41 smithi007 ceph-mon[39249]: 1.9 scrub ok 2024-09-18T08:22:42.380 INFO:tasks.cephadm.ceph_manager.ceph:need seq 111669149961 got 111669149962 for osd.3 2024-09-18T08:22:42.381 DEBUG:teuthology.parallel:result is None 2024-09-18T08:22:42.801 INFO:tasks.cephadm.ceph_manager.ceph:need seq 34359738649 got 34359738652 for osd.0 2024-09-18T08:22:42.801 DEBUG:teuthology.parallel:result is None 2024-09-18T08:22:42.875 INFO:teuthology.orchestra.run.smithi007.stdout:1318554959970 2024-09-18T08:22:43.155 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:22:42 smithi050 ceph-mon[32371]: 3.16 scrub starts 2024-09-18T08:22:43.155 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:22:42 smithi050 ceph-mon[32371]: 3.16 scrub ok 2024-09-18T08:22:43.155 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:22:42 smithi050 ceph-mon[32371]: 3.a scrub starts 2024-09-18T08:22:43.155 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:22:42 smithi050 ceph-mon[32371]: 3.a scrub ok 2024-09-18T08:22:43.179 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:22:42 smithi007 ceph-mon[29459]: 3.16 scrub starts 2024-09-18T08:22:43.180 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:22:42 smithi007 ceph-mon[29459]: 3.16 scrub ok 2024-09-18T08:22:43.180 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:22:42 smithi007 ceph-mon[29459]: 3.a scrub starts 2024-09-18T08:22:43.180 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:22:42 smithi007 ceph-mon[29459]: 3.a scrub ok 2024-09-18T08:22:43.180 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:22:42 smithi007 ceph-mon[39249]: 3.16 scrub starts 2024-09-18T08:22:43.180 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:22:42 smithi007 ceph-mon[39249]: 3.16 scrub ok 2024-09-18T08:22:43.180 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:22:42 smithi007 ceph-mon[39249]: 3.a scrub starts 2024-09-18T08:22:43.180 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:22:42 smithi007 ceph-mon[39249]: 3.a scrub ok 2024-09-18T08:22:43.685 INFO:tasks.cephadm.ceph_manager.ceph:need seq 1318554959967 got 1318554959970 for osd.1 2024-09-18T08:22:43.685 DEBUG:teuthology.parallel:result is None 2024-09-18T08:22:43.685 INFO:tasks.cephadm.ceph_manager.ceph:waiting for recovery to complete 2024-09-18T08:22:43.685 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid 779349ca-7593-11ef-bceb-c7b262605968 -- ceph pg dump --format=json 2024-09-18T08:22:43.956 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/779349ca-7593-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T08:22:43.979 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:22:43 smithi007 ceph-mon[29459]: 6.19 scrub starts 2024-09-18T08:22:43.979 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:22:43 smithi007 ceph-mon[29459]: 6.19 scrub ok 2024-09-18T08:22:43.980 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:22:43 smithi007 ceph-mon[29459]: 4.3 scrub starts 2024-09-18T08:22:43.980 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:22:43 smithi007 ceph-mon[29459]: 4.3 scrub ok 2024-09-18T08:22:43.980 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:22:43 smithi007 ceph-mon[29459]: pgmap v994: 142 pgs: 142 active+clean; 583 KiB data, 5.3 GiB used, 710 GiB / 715 GiB avail; 682 B/s rd, 0 op/s 2024-09-18T08:22:43.980 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:22:43 smithi007 ceph-mon[39249]: 6.19 scrub starts 2024-09-18T08:22:43.980 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:22:43 smithi007 ceph-mon[39249]: 6.19 scrub ok 2024-09-18T08:22:43.980 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:22:43 smithi007 ceph-mon[39249]: 4.3 scrub starts 2024-09-18T08:22:43.980 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:22:43 smithi007 ceph-mon[39249]: 4.3 scrub ok 2024-09-18T08:22:43.980 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:22:43 smithi007 ceph-mon[39249]: pgmap v994: 142 pgs: 142 active+clean; 583 KiB data, 5.3 GiB used, 710 GiB / 715 GiB avail; 682 B/s rd, 0 op/s 2024-09-18T08:22:44.155 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:22:43 smithi050 ceph-mon[32371]: 6.19 scrub starts 2024-09-18T08:22:44.155 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:22:43 smithi050 ceph-mon[32371]: 6.19 scrub ok 2024-09-18T08:22:44.155 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:22:43 smithi050 ceph-mon[32371]: 4.3 scrub starts 2024-09-18T08:22:44.155 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:22:43 smithi050 ceph-mon[32371]: 4.3 scrub ok 2024-09-18T08:22:44.155 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:22:43 smithi050 ceph-mon[32371]: pgmap v994: 142 pgs: 142 active+clean; 583 KiB data, 5.3 GiB used, 710 GiB / 715 GiB avail; 682 B/s rd, 0 op/s 2024-09-18T08:22:44.637 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-09-18T08:22:44.639 INFO:teuthology.orchestra.run.smithi007.stderr:dumped all 2024-09-18T08:22:44.979 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:22:44 smithi007 ceph-mon[29459]: 4.17 scrub starts 2024-09-18T08:22:44.979 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:22:44 smithi007 ceph-mon[29459]: 4.17 scrub ok 2024-09-18T08:22:44.980 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:22:44 smithi007 ceph-mon[29459]: 6.9 scrub starts 2024-09-18T08:22:44.980 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:22:44 smithi007 ceph-mon[29459]: 6.9 scrub ok 2024-09-18T08:22:44.980 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:22:44 smithi007 ceph-mon[29459]: 6.1e scrub starts 2024-09-18T08:22:44.980 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:22:44 smithi007 ceph-mon[29459]: 6.1e scrub ok 2024-09-18T08:22:44.980 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:22:44 smithi007 ceph-mon[29459]: from='client.16728 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-18T08:22:44.980 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:22:44 smithi007 ceph-mon[39249]: 4.17 scrub starts 2024-09-18T08:22:44.980 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:22:44 smithi007 ceph-mon[39249]: 4.17 scrub ok 2024-09-18T08:22:44.980 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:22:44 smithi007 ceph-mon[39249]: 6.9 scrub starts 2024-09-18T08:22:44.981 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:22:44 smithi007 ceph-mon[39249]: 6.9 scrub ok 2024-09-18T08:22:44.981 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:22:44 smithi007 ceph-mon[39249]: 6.1e scrub starts 2024-09-18T08:22:44.981 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:22:44 smithi007 ceph-mon[39249]: 6.1e scrub ok 2024-09-18T08:22:44.981 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:22:44 smithi007 ceph-mon[39249]: from='client.16728 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-18T08:22:45.155 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:22:44 smithi050 ceph-mon[32371]: 4.17 scrub starts 2024-09-18T08:22:45.155 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:22:44 smithi050 ceph-mon[32371]: 4.17 scrub ok 2024-09-18T08:22:45.155 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:22:44 smithi050 ceph-mon[32371]: 6.9 scrub starts 2024-09-18T08:22:45.155 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:22:44 smithi050 ceph-mon[32371]: 6.9 scrub ok 2024-09-18T08:22:45.155 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:22:44 smithi050 ceph-mon[32371]: 6.1e scrub starts 2024-09-18T08:22:45.155 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:22:44 smithi050 ceph-mon[32371]: 6.1e scrub ok 2024-09-18T08:22:45.155 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:22:44 smithi050 ceph-mon[32371]: from='client.16728 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-18T08:22:45.385 INFO:teuthology.orchestra.run.smithi007.stdout:{"pg_ready":true,"pg_map":{"version":994,"stamp":"2024-09-18T08:22:43.547730+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":596507,"num_objects":199,"num_object_clones":0,"num_object_copies":597,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":199,"num_whiteouts":0,"num_read":1537,"num_read_kb":1289,"num_write":722,"num_write_kb":45,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":633,"num_bytes_recovered":1593821,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1710,"ondisk_log_size":1710,"up":426,"acting":426,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":426,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":5589180,"kb_used_data":7484,"kb_used_omap":18,"kb_used_meta":5581549,"kb_avail":744372036,"statfs":{"total":767960285184,"available":762236964864,"internally_reserved":0,"allocated":7663616,"data_stored":4724473,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":19234,"internal_metadata":5715506398},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":8,"num_read_kb":8,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002966"},"pg_stats":[{"pgid":"3.1f","version":"0'0","reported_seq":947,"reported_epoch":481,"state":"active+clean","last_fresh":"2024-09-18T08:22:32.033796+0000","last_change":"2024-09-18T08:22:14.165828+0000","last_active":"2024-09-18T08:22:32.033796+0000","last_peered":"2024-09-18T08:22:32.033796+0000","last_clean":"2024-09-18T08:22:32.033796+0000","last_became_active":"2024-09-18T08:21:53.624054+0000","last_became_peered":"2024-09-18T08:21:53.624054+0000","last_unstale":"2024-09-18T08:22:32.033796+0000","last_undegraded":"2024-09-18T08:22:32.033796+0000","last_fullsized":"2024-09-18T08:22:32.033796+0000","mapping_epoch":463,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":464,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:22:14.165779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:46.860092+0000","last_clean_scrub_stamp":"2024-09-18T08:22:14.165779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:23:25.113464+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"271'15","reported_seq":72,"reported_epoch":481,"state":"active+clean","last_fresh":"2024-09-18T08:22:31.525611+0000","last_change":"2024-09-18T08:22:23.263863+0000","last_active":"2024-09-18T08:22:31.525611+0000","last_peered":"2024-09-18T08:22:31.525611+0000","last_clean":"2024-09-18T08:22:31.525611+0000","last_became_active":"2024-09-18T08:21:45.587647+0000","last_became_peered":"2024-09-18T08:21:45.587647+0000","last_unstale":"2024-09-18T08:22:31.525611+0000","last_undegraded":"2024-09-18T08:22:31.525611+0000","last_fullsized":"2024-09-18T08:22:31.525611+0000","mapping_epoch":459,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":460,"parent":"0.0","parent_split_bits":0,"last_scrub":"271'15","last_scrub_stamp":"2024-09-18T08:22:23.263766+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_clean_scrub_stamp":"2024-09-18T08:22:23.263766+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:23:27.408337+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":25,"num_read_kb":19,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,2],"acting":[4,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":758,"reported_epoch":481,"state":"active+clean","last_fresh":"2024-09-18T08:22:32.068926+0000","last_change":"2024-09-18T08:22:32.068926+0000","last_active":"2024-09-18T08:22:32.068926+0000","last_peered":"2024-09-18T08:22:32.068926+0000","last_clean":"2024-09-18T08:22:32.068926+0000","last_became_active":"2024-09-18T08:18:27.641323+0000","last_became_peered":"2024-09-18T08:18:27.641323+0000","last_unstale":"2024-09-18T08:22:32.068926+0000","last_undegraded":"2024-09-18T08:22:32.068926+0000","last_fullsized":"2024-09-18T08:22:32.068926+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:22:32.068866+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:50.886830+0000","last_clean_scrub_stamp":"2024-09-18T08:22:32.068866+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:23:59.797379+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":62,"reported_epoch":481,"state":"active+clean","last_fresh":"2024-09-18T08:22:31.526270+0000","last_change":"2024-09-18T08:21:53.286308+0000","last_active":"2024-09-18T08:22:31.526270+0000","last_peered":"2024-09-18T08:22:31.526270+0000","last_clean":"2024-09-18T08:22:31.526270+0000","last_became_active":"2024-09-18T08:21:45.587586+0000","last_became_peered":"2024-09-18T08:21:45.587586+0000","last_unstale":"2024-09-18T08:22:31.526270+0000","last_undegraded":"2024-09-18T08:22:31.526270+0000","last_fullsized":"2024-09-18T08:22:31.526270+0000","mapping_epoch":459,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":460,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:21:53.286258+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_clean_scrub_stamp":"2024-09-18T08:21:53.286258+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:23:06.789102+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":926,"reported_epoch":481,"state":"active+clean","last_fresh":"2024-09-18T08:22:31.526640+0000","last_change":"2024-09-18T08:21:49.985200+0000","last_active":"2024-09-18T08:22:31.526640+0000","last_peered":"2024-09-18T08:22:31.526640+0000","last_clean":"2024-09-18T08:22:31.526640+0000","last_became_active":"2024-09-18T08:18:27.662545+0000","last_became_peered":"2024-09-18T08:18:27.662545+0000","last_unstale":"2024-09-18T08:22:31.526640+0000","last_undegraded":"2024-09-18T08:22:31.526640+0000","last_fullsized":"2024-09-18T08:22:31.526640+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:21:49.985149+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_clean_scrub_stamp":"2024-09-18T08:21:49.985149+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:23:03.367183+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":943,"reported_epoch":481,"state":"active+clean","last_fresh":"2024-09-18T08:22:31.527699+0000","last_change":"2024-09-18T08:21:53.627364+0000","last_active":"2024-09-18T08:22:31.527699+0000","last_peered":"2024-09-18T08:22:31.527699+0000","last_clean":"2024-09-18T08:22:31.527699+0000","last_became_active":"2024-09-18T08:21:53.627031+0000","last_became_peered":"2024-09-18T08:21:53.627031+0000","last_unstale":"2024-09-18T08:22:31.527699+0000","last_undegraded":"2024-09-18T08:22:31.527699+0000","last_fullsized":"2024-09-18T08:22:31.527699+0000","mapping_epoch":463,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":464,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:21:14.758274+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:46.860092+0000","last_clean_scrub_stamp":"2024-09-18T08:21:14.758274+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"271'27","reported_seq":975,"reported_epoch":481,"state":"active+clean","last_fresh":"2024-09-18T08:22:31.525358+0000","last_change":"2024-09-18T08:22:20.119776+0000","last_active":"2024-09-18T08:22:31.525358+0000","last_peered":"2024-09-18T08:22:31.525358+0000","last_clean":"2024-09-18T08:22:31.525358+0000","last_became_active":"2024-09-18T08:21:53.626262+0000","last_became_peered":"2024-09-18T08:21:53.626262+0000","last_unstale":"2024-09-18T08:22:31.525358+0000","last_undegraded":"2024-09-18T08:22:31.525358+0000","last_fullsized":"2024-09-18T08:22:31.525358+0000","mapping_epoch":463,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":464,"parent":"0.0","parent_split_bits":0,"last_scrub":"271'27","last_scrub_stamp":"2024-09-18T08:22:20.119701+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_clean_scrub_stamp":"2024-09-18T08:22:20.119701+0000","objects_scrubbed":7,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:23:44.300990+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":44,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":62,"reported_epoch":481,"state":"active+clean","last_fresh":"2024-09-18T08:22:37.195462+0000","last_change":"2024-09-18T08:22:37.195462+0000","last_active":"2024-09-18T08:22:37.195462+0000","last_peered":"2024-09-18T08:22:37.195462+0000","last_clean":"2024-09-18T08:22:37.195462+0000","last_became_active":"2024-09-18T08:21:45.587398+0000","last_became_peered":"2024-09-18T08:21:45.587398+0000","last_unstale":"2024-09-18T08:22:37.195462+0000","last_undegraded":"2024-09-18T08:22:37.195462+0000","last_fullsized":"2024-09-18T08:22:37.195462+0000","mapping_epoch":459,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":460,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:22:37.195408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:50.886830+0000","last_clean_scrub_stamp":"2024-09-18T08:22:37.195408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:23:53.355714+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":332,"reported_epoch":481,"state":"active+clean","last_fresh":"2024-09-18T08:22:32.035160+0000","last_change":"2024-09-18T08:21:45.594957+0000","last_active":"2024-09-18T08:22:32.035160+0000","last_peered":"2024-09-18T08:22:32.035160+0000","last_clean":"2024-09-18T08:22:32.035160+0000","last_became_active":"2024-09-18T08:21:45.593564+0000","last_became_peered":"2024-09-18T08:21:45.593564+0000","last_unstale":"2024-09-18T08:22:32.035160+0000","last_undegraded":"2024-09-18T08:22:32.035160+0000","last_fullsized":"2024-09-18T08:22:32.035160+0000","mapping_epoch":459,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":460,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:21:10.635237+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:46.860092+0000","last_clean_scrub_stamp":"2024-09-18T08:21:10.635237+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"271'15","reported_seq":72,"reported_epoch":481,"state":"active+clean","last_fresh":"2024-09-18T08:22:31.525771+0000","last_change":"2024-09-18T08:22:18.390845+0000","last_active":"2024-09-18T08:22:31.525771+0000","last_peered":"2024-09-18T08:22:31.525771+0000","last_clean":"2024-09-18T08:22:31.525771+0000","last_became_active":"2024-09-18T08:21:45.595189+0000","last_became_peered":"2024-09-18T08:21:45.595189+0000","last_unstale":"2024-09-18T08:22:31.525771+0000","last_undegraded":"2024-09-18T08:22:31.525771+0000","last_fullsized":"2024-09-18T08:22:31.525771+0000","mapping_epoch":459,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":460,"parent":"0.0","parent_split_bits":0,"last_scrub":"271'15","last_scrub_stamp":"2024-09-18T08:22:18.390770+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_clean_scrub_stamp":"2024-09-18T08:22:18.390770+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:23:30.347550+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":24,"num_read_kb":18,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":340,"reported_epoch":481,"state":"active+clean","last_fresh":"2024-09-18T08:22:36.093786+0000","last_change":"2024-09-18T08:22:36.093786+0000","last_active":"2024-09-18T08:22:36.093786+0000","last_peered":"2024-09-18T08:22:36.093786+0000","last_clean":"2024-09-18T08:22:36.093786+0000","last_became_active":"2024-09-18T08:18:27.650998+0000","last_became_peered":"2024-09-18T08:18:27.650998+0000","last_unstale":"2024-09-18T08:22:36.093786+0000","last_undegraded":"2024-09-18T08:22:36.093786+0000","last_fullsized":"2024-09-18T08:22:36.093786+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:22:36.093721+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:50.886830+0000","last_clean_scrub_stamp":"2024-09-18T08:22:36.093721+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:23:49.150891+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":908,"reported_epoch":481,"state":"active+clean","last_fresh":"2024-09-18T08:22:32.032150+0000","last_change":"2024-09-18T08:22:28.191639+0000","last_active":"2024-09-18T08:22:32.032150+0000","last_peered":"2024-09-18T08:22:32.032150+0000","last_clean":"2024-09-18T08:22:32.032150+0000","last_became_active":"2024-09-18T08:18:27.644825+0000","last_became_peered":"2024-09-18T08:18:27.644825+0000","last_unstale":"2024-09-18T08:22:32.032150+0000","last_undegraded":"2024-09-18T08:22:32.032150+0000","last_fullsized":"2024-09-18T08:22:32.032150+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:22:28.191566+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_clean_scrub_stamp":"2024-09-18T08:22:28.191566+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:23:44.137075+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":332,"reported_epoch":481,"state":"active+clean","last_fresh":"2024-09-18T08:22:32.034754+0000","last_change":"2024-09-18T08:22:19.164086+0000","last_active":"2024-09-18T08:22:32.034754+0000","last_peered":"2024-09-18T08:22:32.034754+0000","last_clean":"2024-09-18T08:22:32.034754+0000","last_became_active":"2024-09-18T08:21:45.590272+0000","last_became_peered":"2024-09-18T08:21:45.590272+0000","last_unstale":"2024-09-18T08:22:32.034754+0000","last_undegraded":"2024-09-18T08:22:32.034754+0000","last_fullsized":"2024-09-18T08:22:32.034754+0000","mapping_epoch":459,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":460,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:22:19.164026+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:46.860092+0000","last_clean_scrub_stamp":"2024-09-18T08:22:19.164026+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:23:19.681795+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1b","version":"271'7","reported_seq":60,"reported_epoch":481,"state":"active+clean","last_fresh":"2024-09-18T08:22:31.526186+0000","last_change":"2024-09-18T08:21:45.611180+0000","last_active":"2024-09-18T08:22:31.526186+0000","last_peered":"2024-09-18T08:22:31.526186+0000","last_clean":"2024-09-18T08:22:31.526186+0000","last_became_active":"2024-09-18T08:21:45.585405+0000","last_became_peered":"2024-09-18T08:21:45.585405+0000","last_unstale":"2024-09-18T08:22:31.526186+0000","last_undegraded":"2024-09-18T08:22:31.526186+0000","last_fullsized":"2024-09-18T08:22:31.526186+0000","mapping_epoch":459,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":460,"parent":"0.0","parent_split_bits":0,"last_scrub":"271'7","last_scrub_stamp":"2024-09-18T08:21:16.628117+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_clean_scrub_stamp":"2024-09-18T08:21:16.628117+0000","objects_scrubbed":2,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:22:32.601541+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":13,"num_read_kb":9,"num_write":8,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":194,"reported_epoch":481,"state":"active+clean","last_fresh":"2024-09-18T08:22:32.034330+0000","last_change":"2024-09-18T08:21:55.964427+0000","last_active":"2024-09-18T08:22:32.034330+0000","last_peered":"2024-09-18T08:22:32.034330+0000","last_clean":"2024-09-18T08:22:32.034330+0000","last_became_active":"2024-09-18T08:21:45.599746+0000","last_became_peered":"2024-09-18T08:21:45.599746+0000","last_unstale":"2024-09-18T08:22:32.034330+0000","last_undegraded":"2024-09-18T08:22:32.034330+0000","last_fullsized":"2024-09-18T08:22:32.034330+0000","mapping_epoch":459,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":460,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:21:55.964371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:50.886830+0000","last_clean_scrub_stamp":"2024-09-18T08:21:55.964371+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:23:08.431483+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":318,"reported_epoch":481,"state":"active+clean","last_fresh":"2024-09-18T08:22:32.032992+0000","last_change":"2024-09-18T08:21:14.639111+0000","last_active":"2024-09-18T08:22:32.032992+0000","last_peered":"2024-09-18T08:22:32.032992+0000","last_clean":"2024-09-18T08:22:32.032992+0000","last_became_active":"2024-09-18T08:15:41.441932+0000","last_became_peered":"2024-09-18T08:15:41.441932+0000","last_unstale":"2024-09-18T08:22:32.032992+0000","last_undegraded":"2024-09-18T08:22:32.032992+0000","last_fullsized":"2024-09-18T08:22:32.032992+0000","mapping_epoch":336,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":337,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:21:14.639039+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_clean_scrub_stamp":"2024-09-18T08:21:14.639039+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":16,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":54,"reported_epoch":481,"state":"active+clean","last_fresh":"2024-09-18T08:22:31.526611+0000","last_change":"2024-09-18T08:21:45.594910+0000","last_active":"2024-09-18T08:22:31.526611+0000","last_peered":"2024-09-18T08:22:31.526611+0000","last_clean":"2024-09-18T08:22:31.526611+0000","last_became_active":"2024-09-18T08:21:45.589375+0000","last_became_peered":"2024-09-18T08:21:45.589375+0000","last_unstale":"2024-09-18T08:22:31.526611+0000","last_undegraded":"2024-09-18T08:22:31.526611+0000","last_fullsized":"2024-09-18T08:22:31.526611+0000","mapping_epoch":459,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":460,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:20:39.609106+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_clean_scrub_stamp":"2024-09-18T08:20:39.609106+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":950,"reported_epoch":481,"state":"active+clean","last_fresh":"2024-09-18T08:22:37.182365+0000","last_change":"2024-09-18T08:22:37.182365+0000","last_active":"2024-09-18T08:22:37.182365+0000","last_peered":"2024-09-18T08:22:37.182365+0000","last_clean":"2024-09-18T08:22:37.182365+0000","last_became_active":"2024-09-18T08:21:45.590553+0000","last_became_peered":"2024-09-18T08:21:45.590553+0000","last_unstale":"2024-09-18T08:22:37.182365+0000","last_undegraded":"2024-09-18T08:22:37.182365+0000","last_fullsized":"2024-09-18T08:22:37.182365+0000","mapping_epoch":459,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":460,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:22:37.182308+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:46.860092+0000","last_clean_scrub_stamp":"2024-09-18T08:22:37.182308+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:23:56.472267+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1c","version":"271'27","reported_seq":70,"reported_epoch":481,"state":"active+clean","last_fresh":"2024-09-18T08:22:31.526323+0000","last_change":"2024-09-18T08:22:13.259006+0000","last_active":"2024-09-18T08:22:31.526323+0000","last_peered":"2024-09-18T08:22:31.526323+0000","last_clean":"2024-09-18T08:22:31.526323+0000","last_became_active":"2024-09-18T08:21:53.633735+0000","last_became_peered":"2024-09-18T08:21:53.633735+0000","last_unstale":"2024-09-18T08:22:31.526323+0000","last_undegraded":"2024-09-18T08:22:31.526323+0000","last_fullsized":"2024-09-18T08:22:31.526323+0000","mapping_epoch":463,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":464,"parent":"0.0","parent_split_bits":0,"last_scrub":"271'27","last_scrub_stamp":"2024-09-18T08:22:13.258942+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_clean_scrub_stamp":"2024-09-18T08:22:13.258942+0000","objects_scrubbed":7,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:23:26.910006+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":44,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":794,"reported_epoch":481,"state":"active+clean","last_fresh":"2024-09-18T08:22:32.032975+0000","last_change":"2024-09-18T08:22:06.772214+0000","last_active":"2024-09-18T08:22:32.032975+0000","last_peered":"2024-09-18T08:22:32.032975+0000","last_clean":"2024-09-18T08:22:32.032975+0000","last_became_active":"2024-09-18T08:21:45.588317+0000","last_became_peered":"2024-09-18T08:21:45.588317+0000","last_unstale":"2024-09-18T08:22:32.032975+0000","last_undegraded":"2024-09-18T08:22:32.032975+0000","last_fullsized":"2024-09-18T08:22:32.032975+0000","mapping_epoch":459,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":460,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:22:06.772167+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:50.886830+0000","last_clean_scrub_stamp":"2024-09-18T08:22:06.772167+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:23:24.038698+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":915,"reported_epoch":481,"state":"active+clean","last_fresh":"2024-09-18T08:22:31.527307+0000","last_change":"2024-09-18T08:22:27.021926+0000","last_active":"2024-09-18T08:22:31.527307+0000","last_peered":"2024-09-18T08:22:31.527307+0000","last_clean":"2024-09-18T08:22:31.527307+0000","last_became_active":"2024-09-18T08:15:41.466371+0000","last_became_peered":"2024-09-18T08:15:41.466371+0000","last_unstale":"2024-09-18T08:22:31.527307+0000","last_undegraded":"2024-09-18T08:22:31.527307+0000","last_fullsized":"2024-09-18T08:22:31.527307+0000","mapping_epoch":336,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":337,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:22:27.021874+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_clean_scrub_stamp":"2024-09-18T08:22:27.021874+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:23:38.719726+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":69,"reported_epoch":481,"state":"active+clean","last_fresh":"2024-09-18T08:22:31.525906+0000","last_change":"2024-09-18T08:22:25.286664+0000","last_active":"2024-09-18T08:22:31.525906+0000","last_peered":"2024-09-18T08:22:31.525906+0000","last_clean":"2024-09-18T08:22:31.525906+0000","last_became_active":"2024-09-18T08:21:53.626449+0000","last_became_peered":"2024-09-18T08:21:53.626449+0000","last_unstale":"2024-09-18T08:22:31.525906+0000","last_undegraded":"2024-09-18T08:22:31.525906+0000","last_fullsized":"2024-09-18T08:22:31.525906+0000","mapping_epoch":463,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":464,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:22:25.286605+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:46.860092+0000","last_clean_scrub_stamp":"2024-09-18T08:22:25.286605+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:23:27.940240+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"271'18","reported_seq":109,"reported_epoch":481,"state":"active+clean","last_fresh":"2024-09-18T08:22:32.034661+0000","last_change":"2024-09-18T08:21:47.269424+0000","last_active":"2024-09-18T08:22:32.034661+0000","last_peered":"2024-09-18T08:22:32.034661+0000","last_clean":"2024-09-18T08:22:32.034661+0000","last_became_active":"2024-09-18T08:21:45.590539+0000","last_became_peered":"2024-09-18T08:21:45.590539+0000","last_unstale":"2024-09-18T08:22:32.034661+0000","last_undegraded":"2024-09-18T08:22:32.034661+0000","last_fullsized":"2024-09-18T08:22:32.034661+0000","mapping_epoch":459,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":460,"parent":"0.0","parent_split_bits":0,"last_scrub":"271'18","last_scrub_stamp":"2024-09-18T08:21:47.269334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_clean_scrub_stamp":"2024-09-18T08:21:47.269334+0000","objects_scrubbed":5,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:23:00.277106+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":34,"num_read_kb":25,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":28,"num_bytes_recovered":1100,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,4],"acting":[5,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":69,"reported_epoch":481,"state":"active+clean","last_fresh":"2024-09-18T08:22:31.526410+0000","last_change":"2024-09-18T08:22:02.259158+0000","last_active":"2024-09-18T08:22:31.526410+0000","last_peered":"2024-09-18T08:22:31.526410+0000","last_clean":"2024-09-18T08:22:31.526410+0000","last_became_active":"2024-09-18T08:21:53.627096+0000","last_became_peered":"2024-09-18T08:21:53.627096+0000","last_unstale":"2024-09-18T08:22:31.526410+0000","last_undegraded":"2024-09-18T08:22:31.526410+0000","last_fullsized":"2024-09-18T08:22:31.526410+0000","mapping_epoch":463,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":464,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:22:02.259100+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:50.886830+0000","last_clean_scrub_stamp":"2024-09-18T08:22:02.259100+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:23:20.554944+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1c","version":"69'1","reported_seq":171,"reported_epoch":481,"state":"active+clean","last_fresh":"2024-09-18T08:22:32.033957+0000","last_change":"2024-09-18T08:22:20.709983+0000","last_active":"2024-09-18T08:22:32.033957+0000","last_peered":"2024-09-18T08:22:32.033957+0000","last_clean":"2024-09-18T08:22:32.033957+0000","last_became_active":"2024-09-18T08:22:16.740590+0000","last_became_peered":"2024-09-18T08:22:16.740590+0000","last_unstale":"2024-09-18T08:22:32.033957+0000","last_undegraded":"2024-09-18T08:22:32.033957+0000","last_fullsized":"2024-09-18T08:22:32.033957+0000","mapping_epoch":477,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":478,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'1","last_scrub_stamp":"2024-09-18T08:22:20.709888+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_clean_scrub_stamp":"2024-09-18T08:22:20.709888+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:23:42.046378+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":403,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":806,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.19","version":"62'1","reported_seq":803,"reported_epoch":481,"state":"active+clean","last_fresh":"2024-09-18T08:22:32.032744+0000","last_change":"2024-09-18T08:21:50.659120+0000","last_active":"2024-09-18T08:22:32.032744+0000","last_peered":"2024-09-18T08:22:32.032744+0000","last_clean":"2024-09-18T08:22:32.032744+0000","last_became_active":"2024-09-18T08:21:45.587654+0000","last_became_peered":"2024-09-18T08:21:45.587654+0000","last_unstale":"2024-09-18T08:22:32.032744+0000","last_undegraded":"2024-09-18T08:22:32.032744+0000","last_fullsized":"2024-09-18T08:22:32.032744+0000","mapping_epoch":459,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":460,"parent":"0.0","parent_split_bits":0,"last_scrub":"62'1","last_scrub_stamp":"2024-09-18T08:21:50.659051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:46.860092+0000","last_clean_scrub_stamp":"2024-09-18T08:21:50.659051+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:23:16.368410+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":92,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1e","version":"271'20","reported_seq":966,"reported_epoch":481,"state":"active+clean","last_fresh":"2024-09-18T08:22:32.033833+0000","last_change":"2024-09-18T08:22:31.164606+0000","last_active":"2024-09-18T08:22:32.033833+0000","last_peered":"2024-09-18T08:22:32.033833+0000","last_clean":"2024-09-18T08:22:32.033833+0000","last_became_active":"2024-09-18T08:18:27.657472+0000","last_became_peered":"2024-09-18T08:18:27.657472+0000","last_unstale":"2024-09-18T08:22:32.033833+0000","last_undegraded":"2024-09-18T08:22:32.033833+0000","last_fullsized":"2024-09-18T08:22:32.033833+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":0,"last_scrub":"271'20","last_scrub_stamp":"2024-09-18T08:22:31.164550+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_clean_scrub_stamp":"2024-09-18T08:22:31.164550+0000","objects_scrubbed":5,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:23:50.663068+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":25,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1f","version":"475'19","reported_seq":181,"reported_epoch":481,"state":"active+clean","last_fresh":"2024-09-18T08:22:38.166517+0000","last_change":"2024-09-18T08:22:12.719735+0000","last_active":"2024-09-18T08:22:38.166517+0000","last_peered":"2024-09-18T08:22:38.166517+0000","last_clean":"2024-09-18T08:22:38.166517+0000","last_became_active":"2024-09-18T08:22:12.719349+0000","last_became_peered":"2024-09-18T08:22:12.719349+0000","last_unstale":"2024-09-18T08:22:38.166517+0000","last_undegraded":"2024-09-18T08:22:38.166517+0000","last_fullsized":"2024-09-18T08:22:38.166517+0000","mapping_epoch":474,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":475,"parent":"0.0","parent_split_bits":0,"last_scrub":"415'17","last_scrub_stamp":"2024-09-18T08:21:42.861949+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:50.886830+0000","last_clean_scrub_stamp":"2024-09-18T08:21:42.861949+0000","objects_scrubbed":1,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:23:11.903554+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.a","version":"74'79","reported_seq":328,"reported_epoch":481,"state":"active+clean","last_fresh":"2024-09-18T08:22:32.033730+0000","last_change":"2024-09-18T08:22:27.128207+0000","last_active":"2024-09-18T08:22:32.033730+0000","last_peered":"2024-09-18T08:22:32.033730+0000","last_clean":"2024-09-18T08:22:32.033730+0000","last_became_active":"2024-09-18T08:15:41.443376+0000","last_became_peered":"2024-09-18T08:15:41.443376+0000","last_unstale":"2024-09-18T08:22:32.033730+0000","last_undegraded":"2024-09-18T08:22:32.033730+0000","last_fullsized":"2024-09-18T08:22:32.033730+0000","mapping_epoch":336,"log_start":"0'0","ondisk_log_start":"0'0","created":152,"last_epoch_clean":337,"parent":"0.0","parent_split_bits":4,"last_scrub":"74'79","last_scrub_stamp":"2024-09-18T08:22:27.128146+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:00:11.893742+0000","last_clean_scrub_stamp":"2024-09-18T08:22:27.128146+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:23:44.195317+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,6],"acting":[5,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.f","version":"271'27","reported_seq":902,"reported_epoch":481,"state":"active+clean","last_fresh":"2024-09-18T08:22:32.032533+0000","last_change":"2024-09-18T08:22:03.721475+0000","last_active":"2024-09-18T08:22:32.032533+0000","last_peered":"2024-09-18T08:22:32.032533+0000","last_clean":"2024-09-18T08:22:32.032533+0000","last_became_active":"2024-09-18T08:21:45.588963+0000","last_became_peered":"2024-09-18T08:21:45.588963+0000","last_unstale":"2024-09-18T08:22:32.032533+0000","last_undegraded":"2024-09-18T08:22:32.032533+0000","last_fullsized":"2024-09-18T08:22:32.032533+0000","mapping_epoch":459,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":460,"parent":"0.0","parent_split_bits":0,"last_scrub":"271'27","last_scrub_stamp":"2024-09-18T08:22:03.721411+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_clean_scrub_stamp":"2024-09-18T08:22:03.721411+0000","objects_scrubbed":7,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:23:27.881734+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":43,"num_read_kb":34,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":209,"reported_epoch":481,"state":"active+clean","last_fresh":"2024-09-18T08:22:31.525539+0000","last_change":"2024-09-18T08:21:49.345647+0000","last_active":"2024-09-18T08:22:31.525539+0000","last_peered":"2024-09-18T08:22:31.525539+0000","last_clean":"2024-09-18T08:22:31.525539+0000","last_became_active":"2024-09-18T08:18:27.655321+0000","last_became_peered":"2024-09-18T08:18:27.655321+0000","last_unstale":"2024-09-18T08:22:31.525539+0000","last_undegraded":"2024-09-18T08:22:31.525539+0000","last_fullsized":"2024-09-18T08:22:31.525539+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:21:49.345591+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:46.860092+0000","last_clean_scrub_stamp":"2024-09-18T08:21:49.345591+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:23:02.885177+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,7],"acting":[2,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.e","version":"460'20","reported_seq":70,"reported_epoch":481,"state":"active+clean","last_fresh":"2024-09-18T08:22:38.166548+0000","last_change":"2024-09-18T08:21:45.607932+0000","last_active":"2024-09-18T08:22:38.166548+0000","last_peered":"2024-09-18T08:22:38.166548+0000","last_clean":"2024-09-18T08:22:38.166548+0000","last_became_active":"2024-09-18T08:21:45.583046+0000","last_became_peered":"2024-09-18T08:21:45.583046+0000","last_unstale":"2024-09-18T08:22:38.166548+0000","last_undegraded":"2024-09-18T08:22:38.166548+0000","last_fullsized":"2024-09-18T08:22:38.166548+0000","mapping_epoch":459,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":460,"parent":"0.0","parent_split_bits":0,"last_scrub":"415'19","last_scrub_stamp":"2024-09-18T08:21:41.320519+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:50.886830+0000","last_clean_scrub_stamp":"2024-09-18T08:21:41.320519+0000","objects_scrubbed":1,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":311,"reported_epoch":481,"state":"active+clean","last_fresh":"2024-09-18T08:22:32.035524+0000","last_change":"2024-09-18T08:18:56.732088+0000","last_active":"2024-09-18T08:22:32.035524+0000","last_peered":"2024-09-18T08:22:32.035524+0000","last_clean":"2024-09-18T08:22:32.035524+0000","last_became_active":"2024-09-18T08:15:41.438610+0000","last_became_peered":"2024-09-18T08:15:41.438610+0000","last_unstale":"2024-09-18T08:22:32.035524+0000","last_undegraded":"2024-09-18T08:22:32.035524+0000","last_fullsized":"2024-09-18T08:22:32.035524+0000","mapping_epoch":336,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":337,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:18:56.732041+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_clean_scrub_stamp":"2024-09-18T08:18:56.732041+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:22:38.339891+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.5","version":"74'79","reported_seq":69,"reported_epoch":481,"state":"active+clean","last_fresh":"2024-09-18T08:22:31.526963+0000","last_change":"2024-09-18T08:22:01.305777+0000","last_active":"2024-09-18T08:22:31.526963+0000","last_peered":"2024-09-18T08:22:31.526963+0000","last_clean":"2024-09-18T08:22:31.526963+0000","last_became_active":"2024-09-18T08:21:53.639962+0000","last_became_peered":"2024-09-18T08:21:53.639962+0000","last_unstale":"2024-09-18T08:22:31.526963+0000","last_undegraded":"2024-09-18T08:22:31.526963+0000","last_fullsized":"2024-09-18T08:22:31.526963+0000","mapping_epoch":463,"log_start":"0'0","ondisk_log_start":"0'0","created":152,"last_epoch_clean":464,"parent":"0.0","parent_split_bits":3,"last_scrub":"74'79","last_scrub_stamp":"2024-09-18T08:22:01.305717+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:00:11.893742+0000","last_clean_scrub_stamp":"2024-09-18T08:22:01.305717+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:23:06.219083+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,6],"acting":[4,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.0","version":"271'30","reported_seq":1008,"reported_epoch":481,"state":"active+clean","last_fresh":"2024-09-18T08:22:31.525184+0000","last_change":"2024-09-18T08:21:42.952471+0000","last_active":"2024-09-18T08:22:31.525184+0000","last_peered":"2024-09-18T08:22:31.525184+0000","last_clean":"2024-09-18T08:22:31.525184+0000","last_became_active":"2024-09-18T08:18:27.651348+0000","last_became_peered":"2024-09-18T08:18:27.651348+0000","last_unstale":"2024-09-18T08:22:31.525184+0000","last_undegraded":"2024-09-18T08:22:31.525184+0000","last_fullsized":"2024-09-18T08:22:31.525184+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":0,"last_scrub":"271'30","last_scrub_stamp":"2024-09-18T08:21:42.952384+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_clean_scrub_stamp":"2024-09-18T08:21:42.952384+0000","objects_scrubbed":8,"log_size":30,"log_dups_size":0,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:22:47.274145+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":52,"num_read_kb":40,"num_write":32,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":951,"reported_epoch":481,"state":"active+clean","last_fresh":"2024-09-18T08:22:31.527827+0000","last_change":"2024-09-18T08:22:23.077674+0000","last_active":"2024-09-18T08:22:31.527827+0000","last_peered":"2024-09-18T08:22:31.527827+0000","last_clean":"2024-09-18T08:22:31.527827+0000","last_became_active":"2024-09-18T08:18:27.663173+0000","last_became_peered":"2024-09-18T08:18:27.663173+0000","last_unstale":"2024-09-18T08:22:31.527827+0000","last_undegraded":"2024-09-18T08:22:31.527827+0000","last_fullsized":"2024-09-18T08:22:31.527827+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:22:23.077630+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:46.860092+0000","last_clean_scrub_stamp":"2024-09-18T08:22:23.077630+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:23:34.205444+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":54,"reported_epoch":481,"state":"active+clean","last_fresh":"2024-09-18T08:22:31.525668+0000","last_change":"2024-09-18T08:21:45.594671+0000","last_active":"2024-09-18T08:22:31.525668+0000","last_peered":"2024-09-18T08:22:31.525668+0000","last_clean":"2024-09-18T08:22:31.525668+0000","last_became_active":"2024-09-18T08:21:45.593045+0000","last_became_peered":"2024-09-18T08:21:45.593045+0000","last_unstale":"2024-09-18T08:22:31.525668+0000","last_undegraded":"2024-09-18T08:22:31.525668+0000","last_fullsized":"2024-09-18T08:22:31.525668+0000","mapping_epoch":459,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":460,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:21:06.858313+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:50.886830+0000","last_clean_scrub_stamp":"2024-09-18T08:21:06.858313+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":69,"reported_epoch":481,"state":"active+clean","last_fresh":"2024-09-18T08:22:31.525907+0000","last_change":"2024-09-18T08:22:24.315327+0000","last_active":"2024-09-18T08:22:31.525907+0000","last_peered":"2024-09-18T08:22:31.525907+0000","last_clean":"2024-09-18T08:22:31.525907+0000","last_became_active":"2024-09-18T08:21:53.639170+0000","last_became_peered":"2024-09-18T08:21:53.639170+0000","last_unstale":"2024-09-18T08:22:31.525907+0000","last_undegraded":"2024-09-18T08:22:31.525907+0000","last_fullsized":"2024-09-18T08:22:31.525907+0000","mapping_epoch":463,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":464,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:22:24.315266+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_clean_scrub_stamp":"2024-09-18T08:22:24.315266+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:23:49.687357+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.4","version":"74'79","reported_seq":779,"reported_epoch":481,"state":"active+clean","last_fresh":"2024-09-18T08:22:31.527137+0000","last_change":"2024-09-18T08:22:26.063984+0000","last_active":"2024-09-18T08:22:31.527137+0000","last_peered":"2024-09-18T08:22:31.527137+0000","last_clean":"2024-09-18T08:22:31.527137+0000","last_became_active":"2024-09-18T08:18:27.662304+0000","last_became_peered":"2024-09-18T08:18:27.662304+0000","last_unstale":"2024-09-18T08:22:31.527137+0000","last_undegraded":"2024-09-18T08:22:31.527137+0000","last_fullsized":"2024-09-18T08:22:31.527137+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":152,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":3,"last_scrub":"74'79","last_scrub_stamp":"2024-09-18T08:22:26.063925+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:00:11.893742+0000","last_clean_scrub_stamp":"2024-09-18T08:22:26.063925+0000","objects_scrubbed":1,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:23:32.383376+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":327680,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1310720,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,7],"acting":[3,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"271'28","reported_seq":78,"reported_epoch":481,"state":"active+clean","last_fresh":"2024-09-18T08:22:41.232838+0000","last_change":"2024-09-18T08:22:41.232838+0000","last_active":"2024-09-18T08:22:41.232838+0000","last_peered":"2024-09-18T08:22:41.232838+0000","last_clean":"2024-09-18T08:22:41.232838+0000","last_became_active":"2024-09-18T08:21:45.595506+0000","last_became_peered":"2024-09-18T08:21:45.595506+0000","last_unstale":"2024-09-18T08:22:41.232838+0000","last_undegraded":"2024-09-18T08:22:41.232838+0000","last_fullsized":"2024-09-18T08:22:41.232838+0000","mapping_epoch":459,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":460,"parent":"0.0","parent_split_bits":0,"last_scrub":"271'28","last_scrub_stamp":"2024-09-18T08:22:41.232767+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_clean_scrub_stamp":"2024-09-18T08:22:41.232767+0000","objects_scrubbed":7,"log_size":28,"log_dups_size":0,"ondisk_log_size":28,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:23:59.416902+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":41,"num_read_kb":34,"num_write":28,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"62'2","reported_seq":980,"reported_epoch":481,"state":"active+clean","last_fresh":"2024-09-18T08:22:32.034045+0000","last_change":"2024-09-18T08:21:50.975417+0000","last_active":"2024-09-18T08:22:32.034045+0000","last_peered":"2024-09-18T08:22:32.034045+0000","last_clean":"2024-09-18T08:22:32.034045+0000","last_became_active":"2024-09-18T08:21:45.591454+0000","last_became_peered":"2024-09-18T08:21:45.591454+0000","last_unstale":"2024-09-18T08:22:32.034045+0000","last_undegraded":"2024-09-18T08:22:32.034045+0000","last_fullsized":"2024-09-18T08:22:32.034045+0000","mapping_epoch":459,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":460,"parent":"0.0","parent_split_bits":0,"last_scrub":"62'2","last_scrub_stamp":"2024-09-18T08:21:50.975360+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:46.860092+0000","last_clean_scrub_stamp":"2024-09-18T08:21:50.975360+0000","objects_scrubbed":2,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:23:20.043852+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1039,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":5195,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.0","version":"460'15","reported_seq":1207,"reported_epoch":481,"state":"active+clean","last_fresh":"2024-09-18T08:22:38.949926+0000","last_change":"2024-09-18T08:22:38.949926+0000","last_active":"2024-09-18T08:22:38.949926+0000","last_peered":"2024-09-18T08:22:38.949926+0000","last_clean":"2024-09-18T08:22:38.949926+0000","last_became_active":"2024-09-18T08:21:45.596425+0000","last_became_peered":"2024-09-18T08:21:45.596425+0000","last_unstale":"2024-09-18T08:22:38.949926+0000","last_undegraded":"2024-09-18T08:22:38.949926+0000","last_fullsized":"2024-09-18T08:22:38.949926+0000","mapping_epoch":459,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":460,"parent":"0.0","parent_split_bits":0,"last_scrub":"460'15","last_scrub_stamp":"2024-09-18T08:22:38.949866+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:50.886830+0000","last_clean_scrub_stamp":"2024-09-18T08:22:38.949866+0000","objects_scrubbed":1,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:23:55.605148+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,4],"acting":[3,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":319,"reported_epoch":481,"state":"active+clean","last_fresh":"2024-09-18T08:22:32.147656+0000","last_change":"2024-09-18T08:22:32.147656+0000","last_active":"2024-09-18T08:22:32.147656+0000","last_peered":"2024-09-18T08:22:32.147656+0000","last_clean":"2024-09-18T08:22:32.147656+0000","last_became_active":"2024-09-18T08:15:41.440876+0000","last_became_peered":"2024-09-18T08:15:41.440876+0000","last_unstale":"2024-09-18T08:22:32.147656+0000","last_undegraded":"2024-09-18T08:22:32.147656+0000","last_fullsized":"2024-09-18T08:22:32.147656+0000","mapping_epoch":336,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":337,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:22:32.147599+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_clean_scrub_stamp":"2024-09-18T08:22:32.147599+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:23:36.563855+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.7","version":"74'79","reported_seq":261,"reported_epoch":481,"state":"active+clean","last_fresh":"2024-09-18T08:22:32.032999+0000","last_change":"2024-09-18T08:22:14.209731+0000","last_active":"2024-09-18T08:22:32.032999+0000","last_peered":"2024-09-18T08:22:32.032999+0000","last_clean":"2024-09-18T08:22:32.032999+0000","last_became_active":"2024-09-18T08:22:12.717686+0000","last_became_peered":"2024-09-18T08:22:12.717686+0000","last_unstale":"2024-09-18T08:22:32.032999+0000","last_undegraded":"2024-09-18T08:22:32.032999+0000","last_fullsized":"2024-09-18T08:22:32.032999+0000","mapping_epoch":474,"log_start":"0'0","ondisk_log_start":"0'0","created":152,"last_epoch_clean":475,"parent":"0.0","parent_split_bits":3,"last_scrub":"74'79","last_scrub_stamp":"2024-09-18T08:22:14.209669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:00:11.893742+0000","last_clean_scrub_stamp":"2024-09-18T08:22:14.209669+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:23:31.023904+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,7],"acting":[6,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.2","version":"271'20","reported_seq":930,"reported_epoch":481,"state":"active+clean","last_fresh":"2024-09-18T08:22:32.032973+0000","last_change":"2024-09-18T08:21:56.773625+0000","last_active":"2024-09-18T08:22:32.032973+0000","last_peered":"2024-09-18T08:22:32.032973+0000","last_clean":"2024-09-18T08:22:32.032973+0000","last_became_active":"2024-09-18T08:21:45.588420+0000","last_became_peered":"2024-09-18T08:21:45.588420+0000","last_unstale":"2024-09-18T08:22:32.032973+0000","last_undegraded":"2024-09-18T08:22:32.032973+0000","last_fullsized":"2024-09-18T08:22:32.032973+0000","mapping_epoch":459,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":460,"parent":"0.0","parent_split_bits":0,"last_scrub":"271'20","last_scrub_stamp":"2024-09-18T08:21:56.773553+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_clean_scrub_stamp":"2024-09-18T08:21:56.773553+0000","objects_scrubbed":5,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:22:59.238413+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":28,"num_read_kb":23,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":319,"reported_epoch":481,"state":"active+clean","last_fresh":"2024-09-18T08:22:32.035047+0000","last_change":"2024-09-18T08:22:23.151248+0000","last_active":"2024-09-18T08:22:32.035047+0000","last_peered":"2024-09-18T08:22:32.035047+0000","last_clean":"2024-09-18T08:22:32.035047+0000","last_became_active":"2024-09-18T08:15:41.449639+0000","last_became_peered":"2024-09-18T08:15:41.449639+0000","last_unstale":"2024-09-18T08:22:32.035047+0000","last_undegraded":"2024-09-18T08:22:32.035047+0000","last_fullsized":"2024-09-18T08:22:32.035047+0000","mapping_epoch":336,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":337,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:22:23.151192+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:46.860092+0000","last_clean_scrub_stamp":"2024-09-18T08:22:23.151192+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:23:27.726330+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.3","version":"338'14","reported_seq":1184,"reported_epoch":481,"state":"active+clean","last_fresh":"2024-09-18T08:22:38.166301+0000","last_change":"2024-09-18T08:22:34.163186+0000","last_active":"2024-09-18T08:22:38.166301+0000","last_peered":"2024-09-18T08:22:38.166301+0000","last_clean":"2024-09-18T08:22:38.166301+0000","last_became_active":"2024-09-18T08:15:41.447667+0000","last_became_peered":"2024-09-18T08:15:41.447667+0000","last_unstale":"2024-09-18T08:22:38.166301+0000","last_undegraded":"2024-09-18T08:22:38.166301+0000","last_fullsized":"2024-09-18T08:22:38.166301+0000","mapping_epoch":336,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":337,"parent":"0.0","parent_split_bits":0,"last_scrub":"338'14","last_scrub_stamp":"2024-09-18T08:22:34.163133+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:50.886830+0000","last_clean_scrub_stamp":"2024-09-18T08:22:34.163133+0000","objects_scrubbed":1,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:23:43.857981+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":930,"reported_epoch":481,"state":"active+clean","last_fresh":"2024-09-18T08:22:32.032190+0000","last_change":"2024-09-18T08:21:56.008772+0000","last_active":"2024-09-18T08:22:32.032190+0000","last_peered":"2024-09-18T08:22:32.032190+0000","last_clean":"2024-09-18T08:22:32.032190+0000","last_became_active":"2024-09-18T08:21:53.629495+0000","last_became_peered":"2024-09-18T08:21:53.629495+0000","last_unstale":"2024-09-18T08:22:32.032190+0000","last_undegraded":"2024-09-18T08:22:32.032190+0000","last_fullsized":"2024-09-18T08:22:32.032190+0000","mapping_epoch":463,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":464,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:21:56.008716+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_clean_scrub_stamp":"2024-09-18T08:21:56.008716+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:23:03.875835+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.6","version":"74'79","reported_seq":261,"reported_epoch":481,"state":"active+clean","last_fresh":"2024-09-18T08:22:32.033729+0000","last_change":"2024-09-18T08:22:13.164047+0000","last_active":"2024-09-18T08:22:32.033729+0000","last_peered":"2024-09-18T08:22:32.033729+0000","last_clean":"2024-09-18T08:22:32.033729+0000","last_became_active":"2024-09-18T08:22:12.717528+0000","last_became_peered":"2024-09-18T08:22:12.717528+0000","last_unstale":"2024-09-18T08:22:32.033729+0000","last_undegraded":"2024-09-18T08:22:32.033729+0000","last_fullsized":"2024-09-18T08:22:32.033729+0000","mapping_epoch":474,"log_start":"0'0","ondisk_log_start":"0'0","created":152,"last_epoch_clean":475,"parent":"0.0","parent_split_bits":3,"last_scrub":"74'79","last_scrub_stamp":"2024-09-18T08:22:13.163954+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:00:11.893742+0000","last_clean_scrub_stamp":"2024-09-18T08:22:13.163954+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:23:29.503196+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,3],"acting":[6,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.3","version":"271'29","reported_seq":1057,"reported_epoch":481,"state":"active+clean","last_fresh":"2024-09-18T08:22:43.160260+0000","last_change":"2024-09-18T08:22:43.160260+0000","last_active":"2024-09-18T08:22:43.160260+0000","last_peered":"2024-09-18T08:22:43.160260+0000","last_clean":"2024-09-18T08:22:43.160260+0000","last_became_active":"2024-09-18T08:18:27.654192+0000","last_became_peered":"2024-09-18T08:18:27.654192+0000","last_unstale":"2024-09-18T08:22:43.160260+0000","last_undegraded":"2024-09-18T08:22:43.160260+0000","last_fullsized":"2024-09-18T08:22:43.160260+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":0,"last_scrub":"271'29","last_scrub_stamp":"2024-09-18T08:22:43.160180+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_clean_scrub_stamp":"2024-09-18T08:22:43.160180+0000","objects_scrubbed":8,"log_size":29,"log_dups_size":0,"ondisk_log_size":29,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:23:53.002412+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":54,"num_read_kb":40,"num_write":32,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":32,"num_bytes_recovered":1320,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.4","version":"62'1","reported_seq":806,"reported_epoch":481,"state":"active+clean","last_fresh":"2024-09-18T08:22:32.032985+0000","last_change":"2024-09-18T08:21:53.652773+0000","last_active":"2024-09-18T08:22:32.032985+0000","last_peered":"2024-09-18T08:22:32.032985+0000","last_clean":"2024-09-18T08:22:32.032985+0000","last_became_active":"2024-09-18T08:21:53.638433+0000","last_became_peered":"2024-09-18T08:21:53.638433+0000","last_unstale":"2024-09-18T08:22:32.032985+0000","last_undegraded":"2024-09-18T08:22:32.032985+0000","last_fullsized":"2024-09-18T08:22:32.032985+0000","mapping_epoch":463,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":464,"parent":"0.0","parent_split_bits":0,"last_scrub":"62'1","last_scrub_stamp":"2024-09-18T08:21:44.682795+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:46.860092+0000","last_clean_scrub_stamp":"2024-09-18T08:21:44.682795+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:23:08.194695+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":436,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":2180,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":546,"reported_epoch":481,"state":"active+clean","last_fresh":"2024-09-18T08:22:32.034315+0000","last_change":"2024-09-18T08:22:17.273069+0000","last_active":"2024-09-18T08:22:32.034315+0000","last_peered":"2024-09-18T08:22:32.034315+0000","last_clean":"2024-09-18T08:22:32.034315+0000","last_became_active":"2024-09-18T08:22:12.718377+0000","last_became_peered":"2024-09-18T08:22:12.718377+0000","last_unstale":"2024-09-18T08:22:32.034315+0000","last_undegraded":"2024-09-18T08:22:32.034315+0000","last_fullsized":"2024-09-18T08:22:32.034315+0000","mapping_epoch":474,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":475,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:22:17.273010+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:50.886830+0000","last_clean_scrub_stamp":"2024-09-18T08:22:17.273010+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:23:23.711311+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":817,"reported_epoch":481,"state":"active+clean","last_fresh":"2024-09-18T08:22:32.032216+0000","last_change":"2024-09-18T08:22:07.816688+0000","last_active":"2024-09-18T08:22:32.032216+0000","last_peered":"2024-09-18T08:22:32.032216+0000","last_clean":"2024-09-18T08:22:32.032216+0000","last_became_active":"2024-09-18T08:21:53.636814+0000","last_became_peered":"2024-09-18T08:21:53.636814+0000","last_unstale":"2024-09-18T08:22:32.032216+0000","last_undegraded":"2024-09-18T08:22:32.032216+0000","last_fullsized":"2024-09-18T08:22:32.032216+0000","mapping_epoch":463,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":464,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:22:07.816632+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_clean_scrub_stamp":"2024-09-18T08:22:07.816632+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:23:32.777862+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"74'79","reported_seq":328,"reported_epoch":481,"state":"active+clean","last_fresh":"2024-09-18T08:22:32.033115+0000","last_change":"2024-09-18T08:22:24.111048+0000","last_active":"2024-09-18T08:22:32.033115+0000","last_peered":"2024-09-18T08:22:32.033115+0000","last_clean":"2024-09-18T08:22:32.033115+0000","last_became_active":"2024-09-18T08:15:41.452895+0000","last_became_peered":"2024-09-18T08:15:41.452895+0000","last_unstale":"2024-09-18T08:22:32.033115+0000","last_undegraded":"2024-09-18T08:22:32.033115+0000","last_fullsized":"2024-09-18T08:22:32.033115+0000","mapping_epoch":336,"log_start":"0'0","ondisk_log_start":"0'0","created":152,"last_epoch_clean":337,"parent":"0.0","parent_split_bits":4,"last_scrub":"74'79","last_scrub_stamp":"2024-09-18T08:22:24.110984+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:00:11.893742+0000","last_clean_scrub_stamp":"2024-09-18T08:22:24.110984+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:23:31.218197+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,2],"acting":[5,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.4","version":"464'44","reported_seq":1061,"reported_epoch":481,"state":"active+clean","last_fresh":"2024-09-18T08:22:39.057074+0000","last_change":"2024-09-18T08:22:39.056981+0000","last_active":"2024-09-18T08:22:39.057074+0000","last_peered":"2024-09-18T08:22:39.057074+0000","last_clean":"2024-09-18T08:22:39.057074+0000","last_became_active":"2024-09-18T08:21:53.639078+0000","last_became_peered":"2024-09-18T08:21:53.639078+0000","last_unstale":"2024-09-18T08:22:39.057074+0000","last_undegraded":"2024-09-18T08:22:39.057074+0000","last_fullsized":"2024-09-18T08:22:39.057074+0000","mapping_epoch":463,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":464,"parent":"0.0","parent_split_bits":0,"last_scrub":"464'44","last_scrub_stamp":"2024-09-18T08:22:39.056905+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_clean_scrub_stamp":"2024-09-18T08:22:39.056905+0000","objects_scrubbed":10,"log_size":44,"log_dups_size":0,"ondisk_log_size":44,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:23:54.948602+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":65,"num_read_kb":50,"num_write":38,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":26,"num_bytes_recovered":716,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":62,"reported_epoch":481,"state":"active+clean","last_fresh":"2024-09-18T08:22:31.526058+0000","last_change":"2024-09-18T08:22:05.210106+0000","last_active":"2024-09-18T08:22:31.526058+0000","last_peered":"2024-09-18T08:22:31.526058+0000","last_clean":"2024-09-18T08:22:31.526058+0000","last_became_active":"2024-09-18T08:21:45.589441+0000","last_became_peered":"2024-09-18T08:21:45.589441+0000","last_unstale":"2024-09-18T08:22:31.526058+0000","last_undegraded":"2024-09-18T08:22:31.526058+0000","last_fullsized":"2024-09-18T08:22:31.526058+0000","mapping_epoch":459,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":460,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:22:05.210024+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:46.860092+0000","last_clean_scrub_stamp":"2024-09-18T08:22:05.210024+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:23:05.666864+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"64'2","reported_seq":331,"reported_epoch":481,"state":"active+clean","last_fresh":"2024-09-18T08:22:32.033516+0000","last_change":"2024-09-18T08:21:52.179005+0000","last_active":"2024-09-18T08:22:32.033516+0000","last_peered":"2024-09-18T08:22:32.033516+0000","last_clean":"2024-09-18T08:22:32.033516+0000","last_became_active":"2024-09-18T08:15:41.453351+0000","last_became_peered":"2024-09-18T08:15:41.453351+0000","last_unstale":"2024-09-18T08:22:32.033516+0000","last_undegraded":"2024-09-18T08:22:32.033516+0000","last_fullsized":"2024-09-18T08:22:32.033516+0000","mapping_epoch":336,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":337,"parent":"0.0","parent_split_bits":0,"last_scrub":"64'2","last_scrub_stamp":"2024-09-18T08:21:52.178938+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:45.855979+0000","last_clean_scrub_stamp":"2024-09-18T08:21:52.178938+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:23:06.524708+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":928,"reported_epoch":481,"state":"active+clean","last_fresh":"2024-09-18T08:22:32.033620+0000","last_change":"2024-09-18T08:22:23.248584+0000","last_active":"2024-09-18T08:22:32.033620+0000","last_peered":"2024-09-18T08:22:32.033620+0000","last_clean":"2024-09-18T08:22:32.033620+0000","last_became_active":"2024-09-18T08:21:45.591280+0000","last_became_peered":"2024-09-18T08:21:45.591280+0000","last_unstale":"2024-09-18T08:22:32.033620+0000","last_undegraded":"2024-09-18T08:22:32.033620+0000","last_fullsized":"2024-09-18T08:22:32.033620+0000","mapping_epoch":459,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":460,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:22:23.248536+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:50.886830+0000","last_clean_scrub_stamp":"2024-09-18T08:22:23.248536+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:23:32.744078+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":946,"reported_epoch":481,"state":"active+clean","last_fresh":"2024-09-18T08:22:31.527489+0000","last_change":"2024-09-18T08:22:28.053267+0000","last_active":"2024-09-18T08:22:31.527489+0000","last_peered":"2024-09-18T08:22:31.527489+0000","last_clean":"2024-09-18T08:22:31.527489+0000","last_became_active":"2024-09-18T08:21:45.597448+0000","last_became_peered":"2024-09-18T08:21:45.597448+0000","last_unstale":"2024-09-18T08:22:31.527489+0000","last_undegraded":"2024-09-18T08:22:31.527489+0000","last_fullsized":"2024-09-18T08:22:31.527489+0000","mapping_epoch":459,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":460,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:22:28.053216+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_clean_scrub_stamp":"2024-09-18T08:22:28.053216+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:23:51.238713+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"74'79","reported_seq":684,"reported_epoch":481,"state":"active+clean","last_fresh":"2024-09-18T08:22:32.034160+0000","last_change":"2024-09-18T08:22:16.737058+0000","last_active":"2024-09-18T08:22:32.034160+0000","last_peered":"2024-09-18T08:22:32.034160+0000","last_clean":"2024-09-18T08:22:32.034160+0000","last_became_active":"2024-09-18T08:22:16.736551+0000","last_became_peered":"2024-09-18T08:22:16.736551+0000","last_unstale":"2024-09-18T08:22:32.034160+0000","last_undegraded":"2024-09-18T08:22:32.034160+0000","last_fullsized":"2024-09-18T08:22:32.034160+0000","mapping_epoch":477,"log_start":"0'0","ondisk_log_start":"0'0","created":152,"last_epoch_clean":478,"parent":"0.0","parent_split_bits":4,"last_scrub":"74'79","last_scrub_stamp":"2024-09-18T08:21:45.932207+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:00:11.893742+0000","last_clean_scrub_stamp":"2024-09-18T08:21:45.932207+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:23:08.395270+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.7","version":"271'17","reported_seq":906,"reported_epoch":481,"state":"active+clean","last_fresh":"2024-09-18T08:22:31.526654+0000","last_change":"2024-09-18T08:21:46.316991+0000","last_active":"2024-09-18T08:22:31.526654+0000","last_peered":"2024-09-18T08:22:31.526654+0000","last_clean":"2024-09-18T08:22:31.526654+0000","last_became_active":"2024-09-18T08:15:41.456482+0000","last_became_peered":"2024-09-18T08:15:41.456482+0000","last_unstale":"2024-09-18T08:22:31.526654+0000","last_undegraded":"2024-09-18T08:22:31.526654+0000","last_fullsized":"2024-09-18T08:22:31.526654+0000","mapping_epoch":336,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":337,"parent":"0.0","parent_split_bits":0,"last_scrub":"271'17","last_scrub_stamp":"2024-09-18T08:21:46.316928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_clean_scrub_stamp":"2024-09-18T08:21:46.316928+0000","objects_scrubbed":5,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:22:46.825876+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":35,"num_read_kb":24,"num_write":20,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":990,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,0],"acting":[2,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":800,"reported_epoch":481,"state":"active+clean","last_fresh":"2024-09-18T08:22:32.032288+0000","last_change":"2024-09-18T08:21:53.634626+0000","last_active":"2024-09-18T08:22:32.032288+0000","last_peered":"2024-09-18T08:22:32.032288+0000","last_clean":"2024-09-18T08:22:32.032288+0000","last_became_active":"2024-09-18T08:21:53.634307+0000","last_became_peered":"2024-09-18T08:21:53.634307+0000","last_unstale":"2024-09-18T08:22:32.032288+0000","last_undegraded":"2024-09-18T08:22:32.032288+0000","last_fullsized":"2024-09-18T08:22:32.032288+0000","mapping_epoch":463,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":464,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:21:46.671697+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:46.860092+0000","last_clean_scrub_stamp":"2024-09-18T08:21:46.671697+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:23:16.398511+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"62'1","reported_seq":58,"reported_epoch":481,"state":"active+clean","last_fresh":"2024-09-18T08:22:31.525837+0000","last_change":"2024-09-18T08:21:57.305065+0000","last_active":"2024-09-18T08:22:31.525837+0000","last_peered":"2024-09-18T08:22:31.525837+0000","last_clean":"2024-09-18T08:22:31.525837+0000","last_became_active":"2024-09-18T08:21:53.628297+0000","last_became_peered":"2024-09-18T08:21:53.628297+0000","last_unstale":"2024-09-18T08:22:31.525837+0000","last_undegraded":"2024-09-18T08:22:31.525837+0000","last_fullsized":"2024-09-18T08:22:31.525837+0000","mapping_epoch":463,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":464,"parent":"0.0","parent_split_bits":0,"last_scrub":"62'1","last_scrub_stamp":"2024-09-18T08:21:57.304990+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:45.855979+0000","last_clean_scrub_stamp":"2024-09-18T08:21:57.304990+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:23:17.093474+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":46,"reported_epoch":481,"state":"active+clean","last_fresh":"2024-09-18T08:22:31.526397+0000","last_change":"2024-09-18T08:21:53.632798+0000","last_active":"2024-09-18T08:22:31.526397+0000","last_peered":"2024-09-18T08:22:31.526397+0000","last_clean":"2024-09-18T08:22:31.526397+0000","last_became_active":"2024-09-18T08:21:53.632492+0000","last_became_peered":"2024-09-18T08:21:53.632492+0000","last_unstale":"2024-09-18T08:22:31.526397+0000","last_undegraded":"2024-09-18T08:22:31.526397+0000","last_fullsized":"2024-09-18T08:22:31.526397+0000","mapping_epoch":463,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":464,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:21:48.291960+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:50.886830+0000","last_clean_scrub_stamp":"2024-09-18T08:21:48.291960+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:23:02.145076+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":446,"reported_epoch":481,"state":"active+clean","last_fresh":"2024-09-18T08:22:32.034470+0000","last_change":"2024-09-18T08:22:17.735858+0000","last_active":"2024-09-18T08:22:32.034470+0000","last_peered":"2024-09-18T08:22:32.034470+0000","last_clean":"2024-09-18T08:22:32.034470+0000","last_became_active":"2024-09-18T08:22:16.741247+0000","last_became_peered":"2024-09-18T08:22:16.741247+0000","last_unstale":"2024-09-18T08:22:32.034470+0000","last_undegraded":"2024-09-18T08:22:32.034470+0000","last_fullsized":"2024-09-18T08:22:32.034470+0000","mapping_epoch":477,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":478,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:22:17.735788+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_clean_scrub_stamp":"2024-09-18T08:22:17.735788+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:23:41.651047+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.3","version":"74'79","reported_seq":576,"reported_epoch":481,"state":"active+clean","last_fresh":"2024-09-18T08:22:32.032564+0000","last_change":"2024-09-18T08:21:58.768736+0000","last_active":"2024-09-18T08:22:32.032564+0000","last_peered":"2024-09-18T08:22:32.032564+0000","last_clean":"2024-09-18T08:22:32.032564+0000","last_became_active":"2024-09-18T08:21:45.589564+0000","last_became_peered":"2024-09-18T08:21:45.589564+0000","last_unstale":"2024-09-18T08:22:32.032564+0000","last_undegraded":"2024-09-18T08:22:32.032564+0000","last_fullsized":"2024-09-18T08:22:32.032564+0000","mapping_epoch":459,"log_start":"0'0","ondisk_log_start":"0'0","created":152,"last_epoch_clean":460,"parent":"0.0","parent_split_bits":3,"last_scrub":"74'79","last_scrub_stamp":"2024-09-18T08:21:58.768688+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:00:11.893742+0000","last_clean_scrub_stamp":"2024-09-18T08:21:58.768688+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:23:10.597418+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.6","version":"271'24","reported_seq":1047,"reported_epoch":481,"state":"active+clean","last_fresh":"2024-09-18T08:22:32.033937+0000","last_change":"2024-09-18T08:21:53.652906+0000","last_active":"2024-09-18T08:22:32.033937+0000","last_peered":"2024-09-18T08:22:32.033937+0000","last_clean":"2024-09-18T08:22:32.033937+0000","last_became_active":"2024-09-18T08:21:53.636736+0000","last_became_peered":"2024-09-18T08:21:53.636736+0000","last_unstale":"2024-09-18T08:22:32.033937+0000","last_undegraded":"2024-09-18T08:22:32.033937+0000","last_fullsized":"2024-09-18T08:22:32.033937+0000","mapping_epoch":463,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":464,"parent":"0.0","parent_split_bits":0,"last_scrub":"271'24","last_scrub_stamp":"2024-09-18T08:21:42.907354+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_clean_scrub_stamp":"2024-09-18T08:21:42.907354+0000","objects_scrubbed":6,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:22:56.274053+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":36,"num_read_kb":30,"num_write":24,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":30,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":941,"reported_epoch":481,"state":"active+clean","last_fresh":"2024-09-18T08:22:32.032657+0000","last_change":"2024-09-18T08:21:49.984256+0000","last_active":"2024-09-18T08:22:32.032657+0000","last_peered":"2024-09-18T08:22:32.032657+0000","last_clean":"2024-09-18T08:22:32.032657+0000","last_became_active":"2024-09-18T08:21:45.590631+0000","last_became_peered":"2024-09-18T08:21:45.590631+0000","last_unstale":"2024-09-18T08:22:32.032657+0000","last_undegraded":"2024-09-18T08:22:32.032657+0000","last_fullsized":"2024-09-18T08:22:32.032657+0000","mapping_epoch":459,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":460,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:21:49.984202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:46.860092+0000","last_clean_scrub_stamp":"2024-09-18T08:21:49.984202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:22:52.072896+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"70'5","reported_seq":953,"reported_epoch":481,"state":"active+clean","last_fresh":"2024-09-18T08:22:36.769352+0000","last_change":"2024-09-18T08:22:16.738311+0000","last_active":"2024-09-18T08:22:36.769352+0000","last_peered":"2024-09-18T08:22:36.769352+0000","last_clean":"2024-09-18T08:22:36.769352+0000","last_became_active":"2024-09-18T08:22:16.737474+0000","last_became_peered":"2024-09-18T08:22:16.737474+0000","last_unstale":"2024-09-18T08:22:36.769352+0000","last_undegraded":"2024-09-18T08:22:36.769352+0000","last_fullsized":"2024-09-18T08:22:36.769352+0000","mapping_epoch":477,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":478,"parent":"0.0","parent_split_bits":0,"last_scrub":"70'5","last_scrub_stamp":"2024-09-18T08:21:49.612225+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:45.855979+0000","last_clean_scrub_stamp":"2024-09-18T08:21:49.612225+0000","objects_scrubbed":1,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:22:54.818286+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0.000504055,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":419,"num_read_kb":414,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":389,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":319,"reported_epoch":481,"state":"active+clean","last_fresh":"2024-09-18T08:22:32.035412+0000","last_change":"2024-09-18T08:22:15.211359+0000","last_active":"2024-09-18T08:22:32.035412+0000","last_peered":"2024-09-18T08:22:32.035412+0000","last_clean":"2024-09-18T08:22:32.035412+0000","last_became_active":"2024-09-18T08:15:41.443080+0000","last_became_peered":"2024-09-18T08:15:41.443080+0000","last_unstale":"2024-09-18T08:22:32.035412+0000","last_undegraded":"2024-09-18T08:22:32.035412+0000","last_fullsized":"2024-09-18T08:22:32.035412+0000","mapping_epoch":336,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":337,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:22:15.211298+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:50.886830+0000","last_clean_scrub_stamp":"2024-09-18T08:22:15.211298+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:23:43.279202+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":723,"reported_epoch":481,"state":"active+clean","last_fresh":"2024-09-18T08:22:32.032983+0000","last_change":"2024-09-18T08:22:11.803565+0000","last_active":"2024-09-18T08:22:32.032983+0000","last_peered":"2024-09-18T08:22:32.032983+0000","last_clean":"2024-09-18T08:22:32.032983+0000","last_became_active":"2024-09-18T08:15:41.436692+0000","last_became_peered":"2024-09-18T08:15:41.436692+0000","last_unstale":"2024-09-18T08:22:32.032983+0000","last_undegraded":"2024-09-18T08:22:32.032983+0000","last_fullsized":"2024-09-18T08:22:32.032983+0000","mapping_epoch":336,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":337,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:22:11.803517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_clean_scrub_stamp":"2024-09-18T08:22:11.803517+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:23:28.633455+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.5","version":"475'38","reported_seq":322,"reported_epoch":481,"state":"active+clean","last_fresh":"2024-09-18T08:22:38.166646+0000","last_change":"2024-09-18T08:22:12.719747+0000","last_active":"2024-09-18T08:22:38.166646+0000","last_peered":"2024-09-18T08:22:38.166646+0000","last_clean":"2024-09-18T08:22:38.166646+0000","last_became_active":"2024-09-18T08:22:12.719275+0000","last_became_peered":"2024-09-18T08:22:12.719275+0000","last_unstale":"2024-09-18T08:22:38.166646+0000","last_undegraded":"2024-09-18T08:22:38.166646+0000","last_fullsized":"2024-09-18T08:22:38.166646+0000","mapping_epoch":474,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":475,"parent":"0.0","parent_split_bits":0,"last_scrub":"460'37","last_scrub_stamp":"2024-09-18T08:22:06.213944+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_clean_scrub_stamp":"2024-09-18T08:22:06.213944+0000","objects_scrubbed":8,"log_size":38,"log_dups_size":0,"ondisk_log_size":38,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:23:16.818383+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":41,"num_read_kb":31,"num_write":25,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":154,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":927,"reported_epoch":481,"state":"active+clean","last_fresh":"2024-09-18T08:22:31.526494+0000","last_change":"2024-09-18T08:22:19.090392+0000","last_active":"2024-09-18T08:22:31.526494+0000","last_peered":"2024-09-18T08:22:31.526494+0000","last_clean":"2024-09-18T08:22:31.526494+0000","last_became_active":"2024-09-18T08:15:41.464313+0000","last_became_peered":"2024-09-18T08:15:41.464313+0000","last_unstale":"2024-09-18T08:22:31.526494+0000","last_undegraded":"2024-09-18T08:22:31.526494+0000","last_fullsized":"2024-09-18T08:22:31.526494+0000","mapping_epoch":336,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":337,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:22:19.090346+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:46.860092+0000","last_clean_scrub_stamp":"2024-09-18T08:22:19.090346+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:23:46.659025+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"74'79","reported_seq":365,"reported_epoch":481,"state":"active+clean","last_fresh":"2024-09-18T08:22:32.032738+0000","last_change":"2024-09-18T08:22:16.741948+0000","last_active":"2024-09-18T08:22:32.032738+0000","last_peered":"2024-09-18T08:22:32.032738+0000","last_clean":"2024-09-18T08:22:32.032738+0000","last_became_active":"2024-09-18T08:22:16.741593+0000","last_became_peered":"2024-09-18T08:22:16.741593+0000","last_unstale":"2024-09-18T08:22:32.032738+0000","last_undegraded":"2024-09-18T08:22:32.032738+0000","last_fullsized":"2024-09-18T08:22:32.032738+0000","mapping_epoch":477,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":478,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'79","last_scrub_stamp":"2024-09-18T08:22:06.139534+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:00:11.893742+0000","last_clean_scrub_stamp":"2024-09-18T08:22:06.139534+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:23:06.500076+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":611,"reported_epoch":481,"state":"active+clean","last_fresh":"2024-09-18T08:22:32.033776+0000","last_change":"2024-09-18T08:22:16.741478+0000","last_active":"2024-09-18T08:22:32.033776+0000","last_peered":"2024-09-18T08:22:32.033776+0000","last_clean":"2024-09-18T08:22:32.033776+0000","last_became_active":"2024-09-18T08:22:16.741092+0000","last_became_peered":"2024-09-18T08:22:16.741092+0000","last_unstale":"2024-09-18T08:22:32.033776+0000","last_undegraded":"2024-09-18T08:22:32.033776+0000","last_fullsized":"2024-09-18T08:22:32.033776+0000","mapping_epoch":477,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":478,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:21:45.316642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:50.886830+0000","last_clean_scrub_stamp":"2024-09-18T08:21:45.316642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:23:01.737258+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":333,"reported_epoch":481,"state":"active+clean","last_fresh":"2024-09-18T08:22:32.035260+0000","last_change":"2024-09-18T08:21:45.592063+0000","last_active":"2024-09-18T08:22:32.035260+0000","last_peered":"2024-09-18T08:22:32.035260+0000","last_clean":"2024-09-18T08:22:32.035260+0000","last_became_active":"2024-09-18T08:21:45.591625+0000","last_became_peered":"2024-09-18T08:21:45.591625+0000","last_unstale":"2024-09-18T08:22:32.035260+0000","last_undegraded":"2024-09-18T08:22:32.035260+0000","last_fullsized":"2024-09-18T08:22:32.035260+0000","mapping_epoch":459,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":460,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:21:19.630858+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_clean_scrub_stamp":"2024-09-18T08:21:19.630858+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.e","version":"271'19","reported_seq":74,"reported_epoch":481,"state":"active+clean","last_fresh":"2024-09-18T08:22:31.526809+0000","last_change":"2024-09-18T08:21:59.244331+0000","last_active":"2024-09-18T08:22:31.526809+0000","last_peered":"2024-09-18T08:22:31.526809+0000","last_clean":"2024-09-18T08:22:31.526809+0000","last_became_active":"2024-09-18T08:21:45.593827+0000","last_became_peered":"2024-09-18T08:21:45.593827+0000","last_unstale":"2024-09-18T08:22:31.526809+0000","last_undegraded":"2024-09-18T08:22:31.526809+0000","last_fullsized":"2024-09-18T08:22:31.526809+0000","mapping_epoch":459,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":460,"parent":"0.0","parent_split_bits":0,"last_scrub":"271'19","last_scrub_stamp":"2024-09-18T08:21:59.244247+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_clean_scrub_stamp":"2024-09-18T08:21:59.244247+0000","objects_scrubbed":5,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:23:23.591222+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":31,"num_read_kb":24,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":69,"reported_epoch":481,"state":"active+clean","last_fresh":"2024-09-18T08:22:31.526794+0000","last_change":"2024-09-18T08:22:17.370543+0000","last_active":"2024-09-18T08:22:31.526794+0000","last_peered":"2024-09-18T08:22:31.526794+0000","last_clean":"2024-09-18T08:22:31.526794+0000","last_became_active":"2024-09-18T08:21:53.626983+0000","last_became_peered":"2024-09-18T08:21:53.626983+0000","last_unstale":"2024-09-18T08:22:31.526794+0000","last_undegraded":"2024-09-18T08:22:31.526794+0000","last_fullsized":"2024-09-18T08:22:31.526794+0000","mapping_epoch":463,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":464,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:22:17.370488+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:46.860092+0000","last_clean_scrub_stamp":"2024-09-18T08:22:17.370488+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:23:41.036646+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":340,"reported_epoch":481,"state":"active+clean","last_fresh":"2024-09-18T08:22:32.035138+0000","last_change":"2024-09-18T08:21:45.592459+0000","last_active":"2024-09-18T08:22:32.035138+0000","last_peered":"2024-09-18T08:22:32.035138+0000","last_clean":"2024-09-18T08:22:32.035138+0000","last_became_active":"2024-09-18T08:21:45.592071+0000","last_became_peered":"2024-09-18T08:21:45.592071+0000","last_unstale":"2024-09-18T08:22:32.035138+0000","last_undegraded":"2024-09-18T08:22:32.035138+0000","last_fullsized":"2024-09-18T08:22:32.035138+0000","mapping_epoch":459,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":460,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:21:42.295757+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:50.886830+0000","last_clean_scrub_stamp":"2024-09-18T08:21:42.295757+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:23:11.142413+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":914,"reported_epoch":481,"state":"active+clean","last_fresh":"2024-09-18T08:22:36.021957+0000","last_change":"2024-09-18T08:22:36.021957+0000","last_active":"2024-09-18T08:22:36.021957+0000","last_peered":"2024-09-18T08:22:36.021957+0000","last_clean":"2024-09-18T08:22:36.021957+0000","last_became_active":"2024-09-18T08:15:41.464500+0000","last_became_peered":"2024-09-18T08:15:41.464500+0000","last_unstale":"2024-09-18T08:22:36.021957+0000","last_undegraded":"2024-09-18T08:22:36.021957+0000","last_fullsized":"2024-09-18T08:22:36.021957+0000","mapping_epoch":336,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":337,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:22:36.021908+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_clean_scrub_stamp":"2024-09-18T08:22:36.021908+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:24:02.353796+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"74'79","reported_seq":332,"reported_epoch":481,"state":"active+clean","last_fresh":"2024-09-18T08:22:32.034098+0000","last_change":"2024-09-18T08:22:13.185171+0000","last_active":"2024-09-18T08:22:32.034098+0000","last_peered":"2024-09-18T08:22:32.034098+0000","last_clean":"2024-09-18T08:22:32.034098+0000","last_became_active":"2024-09-18T08:15:41.454781+0000","last_became_peered":"2024-09-18T08:15:41.454781+0000","last_unstale":"2024-09-18T08:22:32.034098+0000","last_undegraded":"2024-09-18T08:22:32.034098+0000","last_fullsized":"2024-09-18T08:22:32.034098+0000","mapping_epoch":336,"log_start":"0'0","ondisk_log_start":"0'0","created":152,"last_epoch_clean":337,"parent":"0.0","parent_split_bits":4,"last_scrub":"74'79","last_scrub_stamp":"2024-09-18T08:22:13.185103+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:00:11.893742+0000","last_clean_scrub_stamp":"2024-09-18T08:22:13.185103+0000","objects_scrubbed":1,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:23:20.190255+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262688,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":262688,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,3],"acting":[5,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.d","version":"271'31","reported_seq":105,"reported_epoch":481,"state":"active+clean","last_fresh":"2024-09-18T08:22:31.526664+0000","last_change":"2024-09-18T08:22:30.263554+0000","last_active":"2024-09-18T08:22:31.526664+0000","last_peered":"2024-09-18T08:22:31.526664+0000","last_clean":"2024-09-18T08:22:31.526664+0000","last_became_active":"2024-09-18T08:21:53.638798+0000","last_became_peered":"2024-09-18T08:21:53.638798+0000","last_unstale":"2024-09-18T08:22:31.526664+0000","last_undegraded":"2024-09-18T08:22:31.526664+0000","last_fullsized":"2024-09-18T08:22:31.526664+0000","mapping_epoch":463,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":464,"parent":"0.0","parent_split_bits":0,"last_scrub":"271'31","last_scrub_stamp":"2024-09-18T08:22:30.263478+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_clean_scrub_stamp":"2024-09-18T08:22:30.263478+0000","objects_scrubbed":8,"log_size":31,"log_dups_size":0,"ondisk_log_size":31,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:23:36.733330+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":49,"num_read_kb":39,"num_write":32,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,1],"acting":[4,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":311,"reported_epoch":481,"state":"active+clean","last_fresh":"2024-09-18T08:22:41.344748+0000","last_change":"2024-09-18T08:22:41.344748+0000","last_active":"2024-09-18T08:22:41.344748+0000","last_peered":"2024-09-18T08:22:41.344748+0000","last_clean":"2024-09-18T08:22:41.344748+0000","last_became_active":"2024-09-18T08:22:12.716430+0000","last_became_peered":"2024-09-18T08:22:12.716430+0000","last_unstale":"2024-09-18T08:22:41.344748+0000","last_undegraded":"2024-09-18T08:22:41.344748+0000","last_fullsized":"2024-09-18T08:22:41.344748+0000","mapping_epoch":474,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":475,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:22:41.344701+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:46.860092+0000","last_clean_scrub_stamp":"2024-09-18T08:22:41.344701+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:23:52.718840+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":807,"reported_epoch":481,"state":"active+clean","last_fresh":"2024-09-18T08:22:32.032867+0000","last_change":"2024-09-18T08:21:45.589662+0000","last_active":"2024-09-18T08:22:32.032867+0000","last_peered":"2024-09-18T08:22:32.032867+0000","last_clean":"2024-09-18T08:22:32.032867+0000","last_became_active":"2024-09-18T08:21:45.589298+0000","last_became_peered":"2024-09-18T08:21:45.589298+0000","last_unstale":"2024-09-18T08:22:32.032867+0000","last_undegraded":"2024-09-18T08:22:32.032867+0000","last_fullsized":"2024-09-18T08:22:32.032867+0000","mapping_epoch":459,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":460,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:21:43.665555+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:50.886830+0000","last_clean_scrub_stamp":"2024-09-18T08:21:43.665555+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:23:09.750846+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":54,"reported_epoch":481,"state":"active+clean","last_fresh":"2024-09-18T08:22:31.526553+0000","last_change":"2024-09-18T08:22:12.301456+0000","last_active":"2024-09-18T08:22:31.526553+0000","last_peered":"2024-09-18T08:22:31.526553+0000","last_clean":"2024-09-18T08:22:31.526553+0000","last_became_active":"2024-09-18T08:21:53.637312+0000","last_became_peered":"2024-09-18T08:21:53.637312+0000","last_unstale":"2024-09-18T08:22:31.526553+0000","last_undegraded":"2024-09-18T08:22:31.526553+0000","last_fullsized":"2024-09-18T08:22:31.526553+0000","mapping_epoch":463,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":464,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:22:12.301396+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_clean_scrub_stamp":"2024-09-18T08:22:12.301396+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:23:20.413823+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"74'79","reported_seq":581,"reported_epoch":481,"state":"active+clean","last_fresh":"2024-09-18T08:22:32.032814+0000","last_change":"2024-09-18T08:22:16.739960+0000","last_active":"2024-09-18T08:22:32.032814+0000","last_peered":"2024-09-18T08:22:32.032814+0000","last_clean":"2024-09-18T08:22:32.032814+0000","last_became_active":"2024-09-18T08:22:16.739529+0000","last_became_peered":"2024-09-18T08:22:16.739529+0000","last_unstale":"2024-09-18T08:22:32.032814+0000","last_undegraded":"2024-09-18T08:22:32.032814+0000","last_fullsized":"2024-09-18T08:22:32.032814+0000","mapping_epoch":477,"log_start":"0'0","ondisk_log_start":"0'0","created":152,"last_epoch_clean":478,"parent":"0.0","parent_split_bits":4,"last_scrub":"74'79","last_scrub_stamp":"2024-09-18T08:21:07.843160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:00:11.893742+0000","last_clean_scrub_stamp":"2024-09-18T08:21:07.843160+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:22:36.805841+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,2],"acting":[7,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.c","version":"271'20","reported_seq":74,"reported_epoch":481,"state":"active+clean","last_fresh":"2024-09-18T08:22:31.526169+0000","last_change":"2024-09-18T08:21:50.172319+0000","last_active":"2024-09-18T08:22:31.526169+0000","last_peered":"2024-09-18T08:22:31.526169+0000","last_clean":"2024-09-18T08:22:31.526169+0000","last_became_active":"2024-09-18T08:21:45.587676+0000","last_became_peered":"2024-09-18T08:21:45.587676+0000","last_unstale":"2024-09-18T08:22:31.526169+0000","last_undegraded":"2024-09-18T08:22:31.526169+0000","last_fullsized":"2024-09-18T08:22:31.526169+0000","mapping_epoch":459,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":460,"parent":"0.0","parent_split_bits":0,"last_scrub":"271'20","last_scrub_stamp":"2024-09-18T08:21:50.172248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_clean_scrub_stamp":"2024-09-18T08:21:50.172248+0000","objects_scrubbed":5,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:22:53.166575+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":28,"num_read_kb":23,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":941,"reported_epoch":481,"state":"active+clean","last_fresh":"2024-09-18T08:22:31.527297+0000","last_change":"2024-09-18T08:21:56.876243+0000","last_active":"2024-09-18T08:22:31.527297+0000","last_peered":"2024-09-18T08:22:31.527297+0000","last_clean":"2024-09-18T08:22:31.527297+0000","last_became_active":"2024-09-18T08:21:45.596486+0000","last_became_peered":"2024-09-18T08:21:45.596486+0000","last_unstale":"2024-09-18T08:22:31.527297+0000","last_undegraded":"2024-09-18T08:22:31.527297+0000","last_fullsized":"2024-09-18T08:22:31.527297+0000","mapping_epoch":459,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":460,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:21:56.876196+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:46.860092+0000","last_clean_scrub_stamp":"2024-09-18T08:21:56.876196+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:23:08.409277+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.d","version":"402'14","reported_seq":1198,"reported_epoch":481,"state":"active+clean","last_fresh":"2024-09-18T08:22:33.165537+0000","last_change":"2024-09-18T08:21:52.348164+0000","last_active":"2024-09-18T08:22:33.165537+0000","last_peered":"2024-09-18T08:22:33.165537+0000","last_clean":"2024-09-18T08:22:33.165537+0000","last_became_active":"2024-09-18T08:18:27.653796+0000","last_became_peered":"2024-09-18T08:18:27.653796+0000","last_unstale":"2024-09-18T08:22:33.165537+0000","last_undegraded":"2024-09-18T08:22:33.165537+0000","last_fullsized":"2024-09-18T08:22:33.165537+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":0,"last_scrub":"402'14","last_scrub_stamp":"2024-09-18T08:21:52.348110+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:50.886830+0000","last_clean_scrub_stamp":"2024-09-18T08:21:52.348110+0000","objects_scrubbed":1,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:23:12.720795+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":69,"reported_epoch":481,"state":"active+clean","last_fresh":"2024-09-18T08:22:31.526496+0000","last_change":"2024-09-18T08:22:26.279695+0000","last_active":"2024-09-18T08:22:31.526496+0000","last_peered":"2024-09-18T08:22:31.526496+0000","last_clean":"2024-09-18T08:22:31.526496+0000","last_became_active":"2024-09-18T08:21:53.632705+0000","last_became_peered":"2024-09-18T08:21:53.632705+0000","last_unstale":"2024-09-18T08:22:31.526496+0000","last_undegraded":"2024-09-18T08:22:31.526496+0000","last_fullsized":"2024-09-18T08:22:31.526496+0000","mapping_epoch":463,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":464,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:22:26.279634+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_clean_scrub_stamp":"2024-09-18T08:22:26.279634+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:23:55.446246+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.b","version":"271'15","reported_seq":1013,"reported_epoch":481,"state":"active+clean","last_fresh":"2024-09-18T08:22:32.033442+0000","last_change":"2024-09-18T08:22:18.270317+0000","last_active":"2024-09-18T08:22:32.033442+0000","last_peered":"2024-09-18T08:22:32.033442+0000","last_clean":"2024-09-18T08:22:32.033442+0000","last_became_active":"2024-09-18T08:21:45.590532+0000","last_became_peered":"2024-09-18T08:21:45.590532+0000","last_unstale":"2024-09-18T08:22:32.033442+0000","last_undegraded":"2024-09-18T08:22:32.033442+0000","last_fullsized":"2024-09-18T08:22:32.033442+0000","mapping_epoch":459,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":460,"parent":"0.0","parent_split_bits":0,"last_scrub":"271'15","last_scrub_stamp":"2024-09-18T08:22:18.270247+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_clean_scrub_stamp":"2024-09-18T08:22:18.270247+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:23:25.584268+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":423,"reported_epoch":481,"state":"active+clean","last_fresh":"2024-09-18T08:22:31.527668+0000","last_change":"2024-09-18T08:21:52.002208+0000","last_active":"2024-09-18T08:22:31.527668+0000","last_peered":"2024-09-18T08:22:31.527668+0000","last_clean":"2024-09-18T08:22:31.527668+0000","last_became_active":"2024-09-18T08:18:17.892088+0000","last_became_peered":"2024-09-18T08:18:17.892088+0000","last_unstale":"2024-09-18T08:22:31.527668+0000","last_undegraded":"2024-09-18T08:22:31.527668+0000","last_fullsized":"2024-09-18T08:22:31.527668+0000","mapping_epoch":395,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":396,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:21:52.002161+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:46.860092+0000","last_clean_scrub_stamp":"2024-09-18T08:21:52.002161+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:23:08.476156+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,6],"acting":[3,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":54,"reported_epoch":481,"state":"active+clean","last_fresh":"2024-09-18T08:22:31.526072+0000","last_change":"2024-09-18T08:22:09.353065+0000","last_active":"2024-09-18T08:22:31.526072+0000","last_peered":"2024-09-18T08:22:31.526072+0000","last_clean":"2024-09-18T08:22:31.526072+0000","last_became_active":"2024-09-18T08:21:53.636990+0000","last_became_peered":"2024-09-18T08:21:53.636990+0000","last_unstale":"2024-09-18T08:22:31.526072+0000","last_undegraded":"2024-09-18T08:22:31.526072+0000","last_fullsized":"2024-09-18T08:22:31.526072+0000","mapping_epoch":463,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":464,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:22:09.353008+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:50.886830+0000","last_clean_scrub_stamp":"2024-09-18T08:22:09.353008+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:23:18.157111+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,3],"acting":[2,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":944,"reported_epoch":481,"state":"active+clean","last_fresh":"2024-09-18T08:22:32.032606+0000","last_change":"2024-09-18T08:21:53.628709+0000","last_active":"2024-09-18T08:22:32.032606+0000","last_peered":"2024-09-18T08:22:32.032606+0000","last_clean":"2024-09-18T08:22:32.032606+0000","last_became_active":"2024-09-18T08:21:53.628253+0000","last_became_peered":"2024-09-18T08:21:53.628253+0000","last_unstale":"2024-09-18T08:22:32.032606+0000","last_undegraded":"2024-09-18T08:22:32.032606+0000","last_fullsized":"2024-09-18T08:22:32.032606+0000","mapping_epoch":463,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":464,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:21:04.983916+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_clean_scrub_stamp":"2024-09-18T08:21:04.983916+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.a","version":"271'35","reported_seq":317,"reported_epoch":481,"state":"active+clean","last_fresh":"2024-09-18T08:22:32.034216+0000","last_change":"2024-09-18T08:22:16.269835+0000","last_active":"2024-09-18T08:22:32.034216+0000","last_peered":"2024-09-18T08:22:32.034216+0000","last_clean":"2024-09-18T08:22:32.034216+0000","last_became_active":"2024-09-18T08:22:12.719919+0000","last_became_peered":"2024-09-18T08:22:12.719919+0000","last_unstale":"2024-09-18T08:22:32.034216+0000","last_undegraded":"2024-09-18T08:22:32.034216+0000","last_fullsized":"2024-09-18T08:22:32.034216+0000","mapping_epoch":474,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":475,"parent":"0.0","parent_split_bits":0,"last_scrub":"271'35","last_scrub_stamp":"2024-09-18T08:22:16.269744+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_clean_scrub_stamp":"2024-09-18T08:22:16.269744+0000","objects_scrubbed":9,"log_size":35,"log_dups_size":0,"ondisk_log_size":35,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:23:26.153878+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":55,"num_read_kb":44,"num_write":36,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":159,"reported_epoch":481,"state":"active+clean","last_fresh":"2024-09-18T08:22:32.035095+0000","last_change":"2024-09-18T08:22:16.744088+0000","last_active":"2024-09-18T08:22:32.035095+0000","last_peered":"2024-09-18T08:22:32.035095+0000","last_clean":"2024-09-18T08:22:32.035095+0000","last_became_active":"2024-09-18T08:22:16.743524+0000","last_became_peered":"2024-09-18T08:22:16.743524+0000","last_unstale":"2024-09-18T08:22:32.035095+0000","last_undegraded":"2024-09-18T08:22:32.035095+0000","last_fullsized":"2024-09-18T08:22:32.035095+0000","mapping_epoch":477,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":478,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:22:14.171891+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:46.860092+0000","last_clean_scrub_stamp":"2024-09-18T08:22:14.171891+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:23:27.770634+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":46,"reported_epoch":481,"state":"active+clean","last_fresh":"2024-09-18T08:22:31.526048+0000","last_change":"2024-09-18T08:21:53.627144+0000","last_active":"2024-09-18T08:22:31.526048+0000","last_peered":"2024-09-18T08:22:31.526048+0000","last_clean":"2024-09-18T08:22:31.526048+0000","last_became_active":"2024-09-18T08:21:53.626819+0000","last_became_peered":"2024-09-18T08:21:53.626819+0000","last_unstale":"2024-09-18T08:22:31.526048+0000","last_undegraded":"2024-09-18T08:22:31.526048+0000","last_fullsized":"2024-09-18T08:22:31.526048+0000","mapping_epoch":463,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":464,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:21:50.974095+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:50.886830+0000","last_clean_scrub_stamp":"2024-09-18T08:21:50.974095+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:23:12.632609+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,5],"acting":[2,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":599,"reported_epoch":481,"state":"active+clean","last_fresh":"2024-09-18T08:22:32.032588+0000","last_change":"2024-09-18T08:22:16.739525+0000","last_active":"2024-09-18T08:22:32.032588+0000","last_peered":"2024-09-18T08:22:32.032588+0000","last_clean":"2024-09-18T08:22:32.032588+0000","last_became_active":"2024-09-18T08:22:16.739179+0000","last_became_peered":"2024-09-18T08:22:16.739179+0000","last_unstale":"2024-09-18T08:22:32.032588+0000","last_undegraded":"2024-09-18T08:22:32.032588+0000","last_fullsized":"2024-09-18T08:22:32.032588+0000","mapping_epoch":477,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":478,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:22:08.369259+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_clean_scrub_stamp":"2024-09-18T08:22:08.369259+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:23:28.403718+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.9","version":"271'18","reported_seq":74,"reported_epoch":481,"state":"active+clean","last_fresh":"2024-09-18T08:22:36.181731+0000","last_change":"2024-09-18T08:22:36.181731+0000","last_active":"2024-09-18T08:22:36.181731+0000","last_peered":"2024-09-18T08:22:36.181731+0000","last_clean":"2024-09-18T08:22:36.181731+0000","last_became_active":"2024-09-18T08:21:45.586536+0000","last_became_peered":"2024-09-18T08:21:45.586536+0000","last_unstale":"2024-09-18T08:22:36.181731+0000","last_undegraded":"2024-09-18T08:22:36.181731+0000","last_fullsized":"2024-09-18T08:22:36.181731+0000","mapping_epoch":459,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":460,"parent":"0.0","parent_split_bits":0,"last_scrub":"271'18","last_scrub_stamp":"2024-09-18T08:22:36.181649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_clean_scrub_stamp":"2024-09-18T08:22:36.181649+0000","objects_scrubbed":5,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:23:42.826616+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":33,"num_read_kb":24,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":1100,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":194,"reported_epoch":481,"state":"active+clean","last_fresh":"2024-09-18T08:22:32.035011+0000","last_change":"2024-09-18T08:21:45.597239+0000","last_active":"2024-09-18T08:22:32.035011+0000","last_peered":"2024-09-18T08:22:32.035011+0000","last_clean":"2024-09-18T08:22:32.035011+0000","last_became_active":"2024-09-18T08:21:45.596881+0000","last_became_peered":"2024-09-18T08:21:45.596881+0000","last_unstale":"2024-09-18T08:22:32.035011+0000","last_undegraded":"2024-09-18T08:22:32.035011+0000","last_fullsized":"2024-09-18T08:22:32.035011+0000","mapping_epoch":459,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":460,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:21:07.904141+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:46.860092+0000","last_clean_scrub_stamp":"2024-09-18T08:21:07.904141+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:22:36.096138+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":924,"reported_epoch":481,"state":"active+clean","last_fresh":"2024-09-18T08:22:31.525700+0000","last_change":"2024-09-18T08:22:14.238417+0000","last_active":"2024-09-18T08:22:31.525700+0000","last_peered":"2024-09-18T08:22:31.525700+0000","last_clean":"2024-09-18T08:22:31.525700+0000","last_became_active":"2024-09-18T08:14:39.399301+0000","last_became_peered":"2024-09-18T08:14:39.399301+0000","last_unstale":"2024-09-18T08:22:31.525700+0000","last_undegraded":"2024-09-18T08:22:31.525700+0000","last_fullsized":"2024-09-18T08:22:31.525700+0000","mapping_epoch":307,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":308,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:22:14.238366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:50.886830+0000","last_clean_scrub_stamp":"2024-09-18T08:22:14.238366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:23:26.494594+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":918,"reported_epoch":481,"state":"active+clean","last_fresh":"2024-09-18T08:22:31.526723+0000","last_change":"2024-09-18T08:21:55.879284+0000","last_active":"2024-09-18T08:22:31.526723+0000","last_peered":"2024-09-18T08:22:31.526723+0000","last_clean":"2024-09-18T08:22:31.526723+0000","last_became_active":"2024-09-18T08:18:27.664412+0000","last_became_peered":"2024-09-18T08:18:27.664412+0000","last_unstale":"2024-09-18T08:22:31.526723+0000","last_undegraded":"2024-09-18T08:22:31.526723+0000","last_fullsized":"2024-09-18T08:22:31.526723+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:21:55.879233+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_clean_scrub_stamp":"2024-09-18T08:21:55.879233+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:23:25.821864+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.8","version":"271'27","reported_seq":381,"reported_epoch":481,"state":"active+clean","last_fresh":"2024-09-18T08:22:32.034182+0000","last_change":"2024-09-18T08:22:29.124790+0000","last_active":"2024-09-18T08:22:32.034182+0000","last_peered":"2024-09-18T08:22:32.034182+0000","last_clean":"2024-09-18T08:22:32.034182+0000","last_became_active":"2024-09-18T08:18:27.643053+0000","last_became_peered":"2024-09-18T08:18:27.643053+0000","last_unstale":"2024-09-18T08:22:32.034182+0000","last_undegraded":"2024-09-18T08:22:32.034182+0000","last_fullsized":"2024-09-18T08:22:32.034182+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":0,"last_scrub":"271'27","last_scrub_stamp":"2024-09-18T08:22:29.124715+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_clean_scrub_stamp":"2024-09-18T08:22:29.124715+0000","objects_scrubbed":7,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:23:48.271896+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":44,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":35,"num_bytes_recovered":550,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.f","version":"62'2","reported_seq":198,"reported_epoch":481,"state":"active+clean","last_fresh":"2024-09-18T08:22:32.034894+0000","last_change":"2024-09-18T08:21:47.774483+0000","last_active":"2024-09-18T08:22:32.034894+0000","last_peered":"2024-09-18T08:22:32.034894+0000","last_clean":"2024-09-18T08:22:32.034894+0000","last_became_active":"2024-09-18T08:21:45.598214+0000","last_became_peered":"2024-09-18T08:21:45.598214+0000","last_unstale":"2024-09-18T08:22:32.034894+0000","last_undegraded":"2024-09-18T08:22:32.034894+0000","last_fullsized":"2024-09-18T08:22:32.034894+0000","mapping_epoch":459,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":460,"parent":"0.0","parent_split_bits":0,"last_scrub":"62'2","last_scrub_stamp":"2024-09-18T08:21:47.774423+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:46.860092+0000","last_clean_scrub_stamp":"2024-09-18T08:21:47.774423+0000","objects_scrubbed":2,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:23:08.158907+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":276,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.9","version":"460'19","reported_seq":253,"reported_epoch":481,"state":"active+clean","last_fresh":"2024-09-18T08:22:33.165550+0000","last_change":"2024-09-18T08:21:54.943666+0000","last_active":"2024-09-18T08:22:33.165550+0000","last_peered":"2024-09-18T08:22:33.165550+0000","last_clean":"2024-09-18T08:22:33.165550+0000","last_became_active":"2024-09-18T08:21:45.597492+0000","last_became_peered":"2024-09-18T08:21:45.597492+0000","last_unstale":"2024-09-18T08:22:33.165550+0000","last_undegraded":"2024-09-18T08:22:33.165550+0000","last_fullsized":"2024-09-18T08:22:33.165550+0000","mapping_epoch":459,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":460,"parent":"0.0","parent_split_bits":0,"last_scrub":"460'19","last_scrub_stamp":"2024-09-18T08:21:54.943603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:50.886830+0000","last_clean_scrub_stamp":"2024-09-18T08:21:54.943603+0000","objects_scrubbed":1,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:23:19.508016+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":319,"reported_epoch":481,"state":"active+clean","last_fresh":"2024-09-18T08:22:32.033626+0000","last_change":"2024-09-18T08:22:02.150670+0000","last_active":"2024-09-18T08:22:32.033626+0000","last_peered":"2024-09-18T08:22:32.033626+0000","last_clean":"2024-09-18T08:22:32.033626+0000","last_became_active":"2024-09-18T08:15:41.449889+0000","last_became_peered":"2024-09-18T08:15:41.449889+0000","last_unstale":"2024-09-18T08:22:32.033626+0000","last_undegraded":"2024-09-18T08:22:32.033626+0000","last_fullsized":"2024-09-18T08:22:32.033626+0000","mapping_epoch":336,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":337,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:22:02.150612+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_clean_scrub_stamp":"2024-09-18T08:22:02.150612+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:23:22.958169+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":553,"reported_epoch":481,"state":"active+clean","last_fresh":"2024-09-18T08:22:32.032995+0000","last_change":"2024-09-18T08:22:18.300016+0000","last_active":"2024-09-18T08:22:32.032995+0000","last_peered":"2024-09-18T08:22:32.032995+0000","last_clean":"2024-09-18T08:22:32.032995+0000","last_became_active":"2024-09-18T08:22:12.718112+0000","last_became_peered":"2024-09-18T08:22:12.718112+0000","last_unstale":"2024-09-18T08:22:32.032995+0000","last_undegraded":"2024-09-18T08:22:32.032995+0000","last_fullsized":"2024-09-18T08:22:32.032995+0000","mapping_epoch":474,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":475,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:22:18.299931+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:46.860092+0000","last_clean_scrub_stamp":"2024-09-18T08:22:18.299931+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:23:40.150113+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"271'12","reported_seq":968,"reported_epoch":481,"state":"active+clean","last_fresh":"2024-09-18T08:22:31.527685+0000","last_change":"2024-09-18T08:21:08.842778+0000","last_active":"2024-09-18T08:22:31.527685+0000","last_peered":"2024-09-18T08:22:31.527685+0000","last_clean":"2024-09-18T08:22:31.527685+0000","last_became_active":"2024-09-18T08:18:27.663338+0000","last_became_peered":"2024-09-18T08:18:27.663338+0000","last_unstale":"2024-09-18T08:22:31.527685+0000","last_undegraded":"2024-09-18T08:22:31.527685+0000","last_fullsized":"2024-09-18T08:22:31.527685+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":0,"last_scrub":"271'12","last_scrub_stamp":"2024-09-18T08:21:08.842710+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_clean_scrub_stamp":"2024-09-18T08:21:08.842710+0000","objects_scrubbed":3,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":18,"num_read_kb":15,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":318,"reported_epoch":481,"state":"active+clean","last_fresh":"2024-09-18T08:22:32.034975+0000","last_change":"2024-09-18T08:22:01.193929+0000","last_active":"2024-09-18T08:22:32.034975+0000","last_peered":"2024-09-18T08:22:32.034975+0000","last_clean":"2024-09-18T08:22:32.034975+0000","last_became_active":"2024-09-18T08:15:41.445332+0000","last_became_peered":"2024-09-18T08:15:41.445332+0000","last_unstale":"2024-09-18T08:22:32.034975+0000","last_undegraded":"2024-09-18T08:22:32.034975+0000","last_fullsized":"2024-09-18T08:22:32.034975+0000","mapping_epoch":336,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":337,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:22:01.193848+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:50.886830+0000","last_clean_scrub_stamp":"2024-09-18T08:22:01.193848+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:23:19.389130+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,1],"acting":[5,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":202,"reported_epoch":481,"state":"active+clean","last_fresh":"2024-09-18T08:22:35.750986+0000","last_change":"2024-09-18T08:22:35.750986+0000","last_active":"2024-09-18T08:22:35.750986+0000","last_peered":"2024-09-18T08:22:35.750986+0000","last_clean":"2024-09-18T08:22:35.750986+0000","last_became_active":"2024-09-18T08:21:45.597863+0000","last_became_peered":"2024-09-18T08:21:45.597863+0000","last_unstale":"2024-09-18T08:22:35.750986+0000","last_undegraded":"2024-09-18T08:22:35.750986+0000","last_fullsized":"2024-09-18T08:22:35.750986+0000","mapping_epoch":459,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":460,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:22:35.750927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_clean_scrub_stamp":"2024-09-18T08:22:35.750927+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:23:50.865747+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"271'15","reported_seq":959,"reported_epoch":481,"state":"active+clean","last_fresh":"2024-09-18T08:22:32.033943+0000","last_change":"2024-09-18T08:21:22.789859+0000","last_active":"2024-09-18T08:22:32.033943+0000","last_peered":"2024-09-18T08:22:32.033943+0000","last_clean":"2024-09-18T08:22:32.033943+0000","last_became_active":"2024-09-18T08:18:27.658637+0000","last_became_peered":"2024-09-18T08:18:27.658637+0000","last_unstale":"2024-09-18T08:22:32.033943+0000","last_undegraded":"2024-09-18T08:22:32.033943+0000","last_fullsized":"2024-09-18T08:22:32.033943+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":0,"last_scrub":"271'15","last_scrub_stamp":"2024-09-18T08:21:22.789796+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_clean_scrub_stamp":"2024-09-18T08:21:22.789796+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:22:48.359769+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":400,"reported_epoch":481,"state":"active+clean","last_fresh":"2024-09-18T08:22:32.033212+0000","last_change":"2024-09-18T08:22:24.643149+0000","last_active":"2024-09-18T08:22:32.033212+0000","last_peered":"2024-09-18T08:22:32.033212+0000","last_clean":"2024-09-18T08:22:32.033212+0000","last_became_active":"2024-09-18T08:22:16.741243+0000","last_became_peered":"2024-09-18T08:22:16.741243+0000","last_unstale":"2024-09-18T08:22:32.033212+0000","last_undegraded":"2024-09-18T08:22:32.033212+0000","last_fullsized":"2024-09-18T08:22:32.033212+0000","mapping_epoch":477,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":478,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:22:24.643087+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:46.860092+0000","last_clean_scrub_stamp":"2024-09-18T08:22:24.643087+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:23:38.442640+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":913,"reported_epoch":481,"state":"active+clean","last_fresh":"2024-09-18T08:22:31.527453+0000","last_change":"2024-09-18T08:22:06.937365+0000","last_active":"2024-09-18T08:22:31.527453+0000","last_peered":"2024-09-18T08:22:31.527453+0000","last_clean":"2024-09-18T08:22:31.527453+0000","last_became_active":"2024-09-18T08:18:27.660093+0000","last_became_peered":"2024-09-18T08:18:27.660093+0000","last_unstale":"2024-09-18T08:22:31.527453+0000","last_undegraded":"2024-09-18T08:22:31.527453+0000","last_fullsized":"2024-09-18T08:22:31.527453+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:22:06.937297+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:50.886830+0000","last_clean_scrub_stamp":"2024-09-18T08:22:06.937297+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:23:17.831650+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":54,"reported_epoch":481,"state":"active+clean","last_fresh":"2024-09-18T08:22:31.526348+0000","last_change":"2024-09-18T08:22:10.354560+0000","last_active":"2024-09-18T08:22:31.526348+0000","last_peered":"2024-09-18T08:22:31.526348+0000","last_clean":"2024-09-18T08:22:31.526348+0000","last_became_active":"2024-09-18T08:21:53.635391+0000","last_became_peered":"2024-09-18T08:21:53.635391+0000","last_unstale":"2024-09-18T08:22:31.526348+0000","last_undegraded":"2024-09-18T08:22:31.526348+0000","last_fullsized":"2024-09-18T08:22:31.526348+0000","mapping_epoch":463,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":464,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:22:10.354510+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_clean_scrub_stamp":"2024-09-18T08:22:10.354510+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:23:26.092303+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.15","version":"271'15","reported_seq":345,"reported_epoch":481,"state":"active+clean","last_fresh":"2024-09-18T08:22:32.035272+0000","last_change":"2024-09-18T08:21:44.334235+0000","last_active":"2024-09-18T08:22:32.035272+0000","last_peered":"2024-09-18T08:22:32.035272+0000","last_clean":"2024-09-18T08:22:32.035272+0000","last_became_active":"2024-09-18T08:18:27.646211+0000","last_became_peered":"2024-09-18T08:18:27.646211+0000","last_unstale":"2024-09-18T08:22:32.035272+0000","last_undegraded":"2024-09-18T08:22:32.035272+0000","last_fullsized":"2024-09-18T08:22:32.035272+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":0,"last_scrub":"271'15","last_scrub_stamp":"2024-09-18T08:21:44.334148+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_clean_scrub_stamp":"2024-09-18T08:21:44.334148+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:22:56.615289+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":25,"num_read_kb":19,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":931,"reported_epoch":481,"state":"active+clean","last_fresh":"2024-09-18T08:22:32.033137+0000","last_change":"2024-09-18T08:21:15.826474+0000","last_active":"2024-09-18T08:22:32.033137+0000","last_peered":"2024-09-18T08:22:32.033137+0000","last_clean":"2024-09-18T08:22:32.033137+0000","last_became_active":"2024-09-18T08:18:27.655758+0000","last_became_peered":"2024-09-18T08:18:27.655758+0000","last_unstale":"2024-09-18T08:22:32.033137+0000","last_undegraded":"2024-09-18T08:22:32.033137+0000","last_fullsized":"2024-09-18T08:22:32.033137+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:21:15.826408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:46.860092+0000","last_clean_scrub_stamp":"2024-09-18T08:21:15.826408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:22:40.998233+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.14","version":"464'19","reported_seq":1229,"reported_epoch":481,"state":"active+clean","last_fresh":"2024-09-18T08:22:33.165436+0000","last_change":"2024-09-18T08:21:53.632185+0000","last_active":"2024-09-18T08:22:33.165436+0000","last_peered":"2024-09-18T08:22:33.165436+0000","last_clean":"2024-09-18T08:22:33.165436+0000","last_became_active":"2024-09-18T08:21:53.623477+0000","last_became_peered":"2024-09-18T08:21:53.623477+0000","last_unstale":"2024-09-18T08:22:33.165436+0000","last_undegraded":"2024-09-18T08:22:33.165436+0000","last_fullsized":"2024-09-18T08:22:33.165436+0000","mapping_epoch":463,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":464,"parent":"0.0","parent_split_bits":0,"last_scrub":"415'18","last_scrub_stamp":"2024-09-18T08:21:18.822162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:50.886830+0000","last_clean_scrub_stamp":"2024-09-18T08:21:18.822162+0000","objects_scrubbed":1,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:22:44.013799+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":62,"reported_epoch":481,"state":"active+clean","last_fresh":"2024-09-18T08:22:31.526884+0000","last_change":"2024-09-18T08:22:11.281835+0000","last_active":"2024-09-18T08:22:31.526884+0000","last_peered":"2024-09-18T08:22:31.526884+0000","last_clean":"2024-09-18T08:22:31.526884+0000","last_became_active":"2024-09-18T08:21:45.581532+0000","last_became_peered":"2024-09-18T08:21:45.581532+0000","last_unstale":"2024-09-18T08:22:31.526884+0000","last_undegraded":"2024-09-18T08:22:31.526884+0000","last_fullsized":"2024-09-18T08:22:31.526884+0000","mapping_epoch":459,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":460,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:22:11.281763+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_clean_scrub_stamp":"2024-09-18T08:22:11.281763+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:23:13.682894+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.14","version":"271'20","reported_seq":994,"reported_epoch":481,"state":"active+clean","last_fresh":"2024-09-18T08:22:37.014620+0000","last_change":"2024-09-18T08:22:37.014620+0000","last_active":"2024-09-18T08:22:37.014620+0000","last_peered":"2024-09-18T08:22:37.014620+0000","last_clean":"2024-09-18T08:22:37.014620+0000","last_became_active":"2024-09-18T08:18:27.663877+0000","last_became_peered":"2024-09-18T08:18:27.663877+0000","last_unstale":"2024-09-18T08:22:37.014620+0000","last_undegraded":"2024-09-18T08:22:37.014620+0000","last_fullsized":"2024-09-18T08:22:37.014620+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":0,"last_scrub":"271'20","last_scrub_stamp":"2024-09-18T08:22:37.014554+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_clean_scrub_stamp":"2024-09-18T08:22:37.014554+0000","objects_scrubbed":5,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:23:43.955123+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":25,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,7],"acting":[3,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":200,"reported_epoch":481,"state":"active+clean","last_fresh":"2024-09-18T08:22:32.033334+0000","last_change":"2024-09-18T08:21:53.640946+0000","last_active":"2024-09-18T08:22:32.033334+0000","last_peered":"2024-09-18T08:22:32.033334+0000","last_clean":"2024-09-18T08:22:32.033334+0000","last_became_active":"2024-09-18T08:21:53.640317+0000","last_became_peered":"2024-09-18T08:21:53.640317+0000","last_unstale":"2024-09-18T08:22:32.033334+0000","last_undegraded":"2024-09-18T08:22:32.033334+0000","last_fullsized":"2024-09-18T08:22:32.033334+0000","mapping_epoch":463,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":464,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:21:45.842324+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:46.860092+0000","last_clean_scrub_stamp":"2024-09-18T08:21:45.842324+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:22:46.641994+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.15","version":"338'12","reported_seq":406,"reported_epoch":481,"state":"active+clean","last_fresh":"2024-09-18T08:22:33.165514+0000","last_change":"2024-09-18T08:22:31.188414+0000","last_active":"2024-09-18T08:22:33.165514+0000","last_peered":"2024-09-18T08:22:33.165514+0000","last_clean":"2024-09-18T08:22:33.165514+0000","last_became_active":"2024-09-18T08:15:41.455243+0000","last_became_peered":"2024-09-18T08:15:41.455243+0000","last_unstale":"2024-09-18T08:22:33.165514+0000","last_undegraded":"2024-09-18T08:22:33.165514+0000","last_fullsized":"2024-09-18T08:22:33.165514+0000","mapping_epoch":336,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":337,"parent":"0.0","parent_split_bits":0,"last_scrub":"338'12","last_scrub_stamp":"2024-09-18T08:22:31.188340+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:50.886830+0000","last_clean_scrub_stamp":"2024-09-18T08:22:31.188340+0000","objects_scrubbed":1,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:23:47.409314+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":914,"reported_epoch":481,"state":"active+clean","last_fresh":"2024-09-18T08:22:32.033736+0000","last_change":"2024-09-18T08:22:25.267732+0000","last_active":"2024-09-18T08:22:32.033736+0000","last_peered":"2024-09-18T08:22:32.033736+0000","last_clean":"2024-09-18T08:22:32.033736+0000","last_became_active":"2024-09-18T08:18:27.658009+0000","last_became_peered":"2024-09-18T08:18:27.658009+0000","last_unstale":"2024-09-18T08:22:32.033736+0000","last_undegraded":"2024-09-18T08:22:32.033736+0000","last_fullsized":"2024-09-18T08:22:32.033736+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:22:25.267686+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_clean_scrub_stamp":"2024-09-18T08:22:25.267686+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:23:47.406228+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.13","version":"271'19","reported_seq":74,"reported_epoch":481,"state":"active+clean","last_fresh":"2024-09-18T08:22:31.526482+0000","last_change":"2024-09-18T08:21:58.234819+0000","last_active":"2024-09-18T08:22:31.526482+0000","last_peered":"2024-09-18T08:22:31.526482+0000","last_clean":"2024-09-18T08:22:31.526482+0000","last_became_active":"2024-09-18T08:21:45.589059+0000","last_became_peered":"2024-09-18T08:21:45.589059+0000","last_unstale":"2024-09-18T08:22:31.526482+0000","last_undegraded":"2024-09-18T08:22:31.526482+0000","last_fullsized":"2024-09-18T08:22:31.526482+0000","mapping_epoch":459,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":460,"parent":"0.0","parent_split_bits":0,"last_scrub":"271'19","last_scrub_stamp":"2024-09-18T08:21:58.234746+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_clean_scrub_stamp":"2024-09-18T08:21:58.234746+0000","objects_scrubbed":5,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:23:28.171925+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":31,"num_read_kb":24,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":62,"reported_epoch":481,"state":"active+clean","last_fresh":"2024-09-18T08:22:31.526075+0000","last_change":"2024-09-18T08:22:16.324599+0000","last_active":"2024-09-18T08:22:31.526075+0000","last_peered":"2024-09-18T08:22:31.526075+0000","last_clean":"2024-09-18T08:22:31.526075+0000","last_became_active":"2024-09-18T08:21:45.592061+0000","last_became_peered":"2024-09-18T08:21:45.592061+0000","last_unstale":"2024-09-18T08:22:31.526075+0000","last_undegraded":"2024-09-18T08:22:31.526075+0000","last_fullsized":"2024-09-18T08:22:31.526075+0000","mapping_epoch":459,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":460,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:22:16.324534+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:46.860092+0000","last_clean_scrub_stamp":"2024-09-18T08:22:16.324534+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:23:16.882936+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":502,"reported_epoch":481,"state":"active+clean","last_fresh":"2024-09-18T08:22:32.034134+0000","last_change":"2024-09-18T08:22:25.363009+0000","last_active":"2024-09-18T08:22:32.034134+0000","last_peered":"2024-09-18T08:22:32.034134+0000","last_clean":"2024-09-18T08:22:32.034134+0000","last_became_active":"2024-09-18T08:22:12.718201+0000","last_became_peered":"2024-09-18T08:22:12.718201+0000","last_unstale":"2024-09-18T08:22:32.034134+0000","last_undegraded":"2024-09-18T08:22:32.034134+0000","last_fullsized":"2024-09-18T08:22:32.034134+0000","mapping_epoch":474,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":475,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:22:25.362951+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:50.886830+0000","last_clean_scrub_stamp":"2024-09-18T08:22:25.362951+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:23:33.635413+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,3],"acting":[6,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":917,"reported_epoch":481,"state":"active+clean","last_fresh":"2024-09-18T08:22:31.525754+0000","last_change":"2024-09-18T08:21:04.922924+0000","last_active":"2024-09-18T08:22:31.525754+0000","last_peered":"2024-09-18T08:22:31.525754+0000","last_clean":"2024-09-18T08:22:31.525754+0000","last_became_active":"2024-09-18T08:15:41.464961+0000","last_became_peered":"2024-09-18T08:15:41.464961+0000","last_unstale":"2024-09-18T08:22:31.525754+0000","last_undegraded":"2024-09-18T08:22:31.525754+0000","last_fullsized":"2024-09-18T08:22:31.525754+0000","mapping_epoch":336,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":337,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:21:04.922071+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_clean_scrub_stamp":"2024-09-18T08:21:04.922071+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.12","version":"271'15","reported_seq":814,"reported_epoch":481,"state":"active+clean","last_fresh":"2024-09-18T08:22:32.033081+0000","last_change":"2024-09-18T08:21:53.649359+0000","last_active":"2024-09-18T08:22:32.033081+0000","last_peered":"2024-09-18T08:22:32.033081+0000","last_clean":"2024-09-18T08:22:32.033081+0000","last_became_active":"2024-09-18T08:21:53.634545+0000","last_became_peered":"2024-09-18T08:21:53.634545+0000","last_unstale":"2024-09-18T08:22:32.033081+0000","last_undegraded":"2024-09-18T08:22:32.033081+0000","last_fullsized":"2024-09-18T08:22:32.033081+0000","mapping_epoch":463,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":464,"parent":"0.0","parent_split_bits":0,"last_scrub":"271'15","last_scrub_stamp":"2024-09-18T08:21:42.696135+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_clean_scrub_stamp":"2024-09-18T08:21:42.696135+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:22:54.443306+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":25,"num_read_kb":19,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":684,"reported_epoch":481,"state":"active+clean","last_fresh":"2024-09-18T08:22:32.034485+0000","last_change":"2024-09-18T08:22:16.737094+0000","last_active":"2024-09-18T08:22:32.034485+0000","last_peered":"2024-09-18T08:22:32.034485+0000","last_clean":"2024-09-18T08:22:32.034485+0000","last_became_active":"2024-09-18T08:22:16.736379+0000","last_became_peered":"2024-09-18T08:22:16.736379+0000","last_unstale":"2024-09-18T08:22:32.034485+0000","last_undegraded":"2024-09-18T08:22:32.034485+0000","last_fullsized":"2024-09-18T08:22:32.034485+0000","mapping_epoch":477,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":478,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:21:41.948219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:46.860092+0000","last_clean_scrub_stamp":"2024-09-18T08:21:41.948219+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:23:03.403331+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":901,"reported_epoch":481,"state":"active+clean","last_fresh":"2024-09-18T08:22:31.527051+0000","last_change":"2024-09-18T08:21:54.920592+0000","last_active":"2024-09-18T08:22:31.527051+0000","last_peered":"2024-09-18T08:22:31.527051+0000","last_clean":"2024-09-18T08:22:31.527051+0000","last_became_active":"2024-09-18T08:14:39.402406+0000","last_became_peered":"2024-09-18T08:14:39.402406+0000","last_unstale":"2024-09-18T08:22:31.527051+0000","last_undegraded":"2024-09-18T08:22:31.527051+0000","last_fullsized":"2024-09-18T08:22:31.527051+0000","mapping_epoch":307,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":308,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:21:54.920545+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:50.886830+0000","last_clean_scrub_stamp":"2024-09-18T08:21:54.920545+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:23:06.408191+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.10","version":"69'1","reported_seq":914,"reported_epoch":481,"state":"active+clean","last_fresh":"2024-09-18T08:22:32.031957+0000","last_change":"2024-09-18T08:21:39.819943+0000","last_active":"2024-09-18T08:22:32.031957+0000","last_peered":"2024-09-18T08:22:32.031957+0000","last_clean":"2024-09-18T08:22:32.031957+0000","last_became_active":"2024-09-18T08:15:41.444085+0000","last_became_peered":"2024-09-18T08:15:41.444085+0000","last_unstale":"2024-09-18T08:22:32.031957+0000","last_undegraded":"2024-09-18T08:22:32.031957+0000","last_fullsized":"2024-09-18T08:22:32.031957+0000","mapping_epoch":336,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":337,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'1","last_scrub_stamp":"2024-09-18T08:21:39.819892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_clean_scrub_stamp":"2024-09-18T08:21:39.819892+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:22:43.568297+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":26,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.11","version":"271'19","reported_seq":1063,"reported_epoch":481,"state":"active+clean","last_fresh":"2024-09-18T08:22:31.525593+0000","last_change":"2024-09-18T08:22:08.993996+0000","last_active":"2024-09-18T08:22:31.525593+0000","last_peered":"2024-09-18T08:22:31.525593+0000","last_clean":"2024-09-18T08:22:31.525593+0000","last_became_active":"2024-09-18T08:18:27.659293+0000","last_became_peered":"2024-09-18T08:18:27.659293+0000","last_unstale":"2024-09-18T08:22:31.525593+0000","last_undegraded":"2024-09-18T08:22:31.525593+0000","last_fullsized":"2024-09-18T08:22:31.525593+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":0,"last_scrub":"271'19","last_scrub_stamp":"2024-09-18T08:22:08.993941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_clean_scrub_stamp":"2024-09-18T08:22:08.993941+0000","objects_scrubbed":5,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:23:22.471820+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":32,"num_read_kb":25,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":30,"num_bytes_recovered":660,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":325,"reported_epoch":481,"state":"active+clean","last_fresh":"2024-09-18T08:22:32.034833+0000","last_change":"2024-09-18T08:21:06.612076+0000","last_active":"2024-09-18T08:22:32.034833+0000","last_peered":"2024-09-18T08:22:32.034833+0000","last_clean":"2024-09-18T08:22:32.034833+0000","last_became_active":"2024-09-18T08:18:27.643957+0000","last_became_peered":"2024-09-18T08:18:27.643957+0000","last_unstale":"2024-09-18T08:22:32.034833+0000","last_undegraded":"2024-09-18T08:22:32.034833+0000","last_fullsized":"2024-09-18T08:22:32.034833+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:21:06.612011+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:46.860092+0000","last_clean_scrub_stamp":"2024-09-18T08:21:06.612011+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":194,"reported_epoch":481,"state":"active+clean","last_fresh":"2024-09-18T08:22:32.033138+0000","last_change":"2024-09-18T08:22:02.901165+0000","last_active":"2024-09-18T08:22:32.033138+0000","last_peered":"2024-09-18T08:22:32.033138+0000","last_clean":"2024-09-18T08:22:32.033138+0000","last_became_active":"2024-09-18T08:21:45.596715+0000","last_became_peered":"2024-09-18T08:21:45.596715+0000","last_unstale":"2024-09-18T08:22:32.033138+0000","last_undegraded":"2024-09-18T08:22:32.033138+0000","last_fullsized":"2024-09-18T08:22:32.033138+0000","mapping_epoch":459,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":460,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:22:02.901100+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:50.886830+0000","last_clean_scrub_stamp":"2024-09-18T08:22:02.901100+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:23:05.993894+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":900,"reported_epoch":481,"state":"active+clean","last_fresh":"2024-09-18T08:22:31.527203+0000","last_change":"2024-09-18T08:21:40.928000+0000","last_active":"2024-09-18T08:22:31.527203+0000","last_peered":"2024-09-18T08:22:31.527203+0000","last_clean":"2024-09-18T08:22:31.527203+0000","last_became_active":"2024-09-18T08:06:24.128582+0000","last_became_peered":"2024-09-18T08:06:24.128582+0000","last_unstale":"2024-09-18T08:22:31.527203+0000","last_undegraded":"2024-09-18T08:22:31.527203+0000","last_fullsized":"2024-09-18T08:22:31.527203+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:21:40.927953+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_clean_scrub_stamp":"2024-09-18T08:21:40.927953+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:22:48.942522+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.10","version":"271'8","reported_seq":950,"reported_epoch":481,"state":"active+clean","last_fresh":"2024-09-18T08:22:31.525073+0000","last_change":"2024-09-18T08:20:38.022125+0000","last_active":"2024-09-18T08:22:31.525073+0000","last_peered":"2024-09-18T08:22:31.525073+0000","last_clean":"2024-09-18T08:22:31.525073+0000","last_became_active":"2024-09-18T08:14:39.404257+0000","last_became_peered":"2024-09-18T08:14:39.404257+0000","last_unstale":"2024-09-18T08:22:31.525073+0000","last_undegraded":"2024-09-18T08:22:31.525073+0000","last_fullsized":"2024-09-18T08:22:31.525073+0000","mapping_epoch":307,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":308,"parent":"0.0","parent_split_bits":0,"last_scrub":"271'8","last_scrub_stamp":"2024-09-18T08:20:38.022032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_clean_scrub_stamp":"2024-09-18T08:20:38.022032+0000","objects_scrubbed":2,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":12,"num_read_kb":10,"num_write":8,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":926,"reported_epoch":481,"state":"active+clean","last_fresh":"2024-09-18T08:22:32.033803+0000","last_change":"2024-09-18T08:21:51.973411+0000","last_active":"2024-09-18T08:22:32.033803+0000","last_peered":"2024-09-18T08:22:32.033803+0000","last_clean":"2024-09-18T08:22:32.033803+0000","last_became_active":"2024-09-18T08:15:41.466964+0000","last_became_peered":"2024-09-18T08:15:41.466964+0000","last_unstale":"2024-09-18T08:22:32.033803+0000","last_undegraded":"2024-09-18T08:22:32.033803+0000","last_fullsized":"2024-09-18T08:22:32.033803+0000","mapping_epoch":336,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":337,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:21:51.973328+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:46.860092+0000","last_clean_scrub_stamp":"2024-09-18T08:21:51.973328+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:23:17.492020+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":398,"reported_epoch":481,"state":"active+clean","last_fresh":"2024-09-18T08:22:32.033746+0000","last_change":"2024-09-18T08:22:12.718474+0000","last_active":"2024-09-18T08:22:32.033746+0000","last_peered":"2024-09-18T08:22:32.033746+0000","last_clean":"2024-09-18T08:22:32.033746+0000","last_became_active":"2024-09-18T08:22:12.718017+0000","last_became_peered":"2024-09-18T08:22:12.718017+0000","last_unstale":"2024-09-18T08:22:32.033746+0000","last_undegraded":"2024-09-18T08:22:32.033746+0000","last_fullsized":"2024-09-18T08:22:32.033746+0000","mapping_epoch":474,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":475,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:22:09.245323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:50.886830+0000","last_clean_scrub_stamp":"2024-09-18T08:22:09.245323+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:23:11.889868+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":558,"reported_epoch":481,"state":"active+clean","last_fresh":"2024-09-18T08:22:32.032736+0000","last_change":"2024-09-18T08:22:16.739974+0000","last_active":"2024-09-18T08:22:32.032736+0000","last_peered":"2024-09-18T08:22:32.032736+0000","last_clean":"2024-09-18T08:22:32.032736+0000","last_became_active":"2024-09-18T08:22:16.739643+0000","last_became_peered":"2024-09-18T08:22:16.739643+0000","last_unstale":"2024-09-18T08:22:32.032736+0000","last_undegraded":"2024-09-18T08:22:32.032736+0000","last_fullsized":"2024-09-18T08:22:32.032736+0000","mapping_epoch":477,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":478,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:22:11.318277+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_clean_scrub_stamp":"2024-09-18T08:22:11.318277+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:23:36.554739+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,4],"acting":[7,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":771,"reported_epoch":481,"state":"active+clean","last_fresh":"2024-09-18T08:22:32.032813+0000","last_change":"2024-09-18T08:22:10.785838+0000","last_active":"2024-09-18T08:22:32.032813+0000","last_peered":"2024-09-18T08:22:32.032813+0000","last_clean":"2024-09-18T08:22:32.032813+0000","last_became_active":"2024-09-18T08:21:45.587782+0000","last_became_peered":"2024-09-18T08:21:45.587782+0000","last_unstale":"2024-09-18T08:22:32.032813+0000","last_undegraded":"2024-09-18T08:22:32.032813+0000","last_fullsized":"2024-09-18T08:22:32.032813+0000","mapping_epoch":459,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":460,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:22:10.785776+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_clean_scrub_stamp":"2024-09-18T08:22:10.785776+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:23:13.806282+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":909,"reported_epoch":481,"state":"active+clean","last_fresh":"2024-09-18T08:22:34.995406+0000","last_change":"2024-09-18T08:22:34.995406+0000","last_active":"2024-09-18T08:22:34.995406+0000","last_peered":"2024-09-18T08:22:34.995406+0000","last_clean":"2024-09-18T08:22:34.995406+0000","last_became_active":"2024-09-18T08:14:39.405347+0000","last_became_peered":"2024-09-18T08:14:39.405347+0000","last_unstale":"2024-09-18T08:22:34.995406+0000","last_undegraded":"2024-09-18T08:22:34.995406+0000","last_fullsized":"2024-09-18T08:22:34.995406+0000","mapping_epoch":307,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":308,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:22:34.995358+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:46.860092+0000","last_clean_scrub_stamp":"2024-09-18T08:22:34.995358+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:23:56.366799+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"271'19","reported_seq":485,"reported_epoch":481,"state":"active+clean","last_fresh":"2024-09-18T08:22:32.033210+0000","last_change":"2024-09-18T08:22:12.720659+0000","last_active":"2024-09-18T08:22:32.033210+0000","last_peered":"2024-09-18T08:22:32.033210+0000","last_clean":"2024-09-18T08:22:32.033210+0000","last_became_active":"2024-09-18T08:22:12.720078+0000","last_became_peered":"2024-09-18T08:22:12.720078+0000","last_unstale":"2024-09-18T08:22:32.033210+0000","last_undegraded":"2024-09-18T08:22:32.033210+0000","last_fullsized":"2024-09-18T08:22:32.033210+0000","mapping_epoch":474,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":475,"parent":"0.0","parent_split_bits":0,"last_scrub":"271'19","last_scrub_stamp":"2024-09-18T08:21:56.226459+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_clean_scrub_stamp":"2024-09-18T08:21:56.226459+0000","objects_scrubbed":5,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:23:09.795645+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":23,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":944,"reported_epoch":481,"state":"active+clean","last_fresh":"2024-09-18T08:22:32.033825+0000","last_change":"2024-09-18T08:22:16.241762+0000","last_active":"2024-09-18T08:22:32.033825+0000","last_peered":"2024-09-18T08:22:32.033825+0000","last_clean":"2024-09-18T08:22:32.033825+0000","last_became_active":"2024-09-18T08:21:53.633510+0000","last_became_peered":"2024-09-18T08:21:53.633510+0000","last_unstale":"2024-09-18T08:22:32.033825+0000","last_undegraded":"2024-09-18T08:22:32.033825+0000","last_fullsized":"2024-09-18T08:22:32.033825+0000","mapping_epoch":463,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":464,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:22:16.241697+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:50.886830+0000","last_clean_scrub_stamp":"2024-09-18T08:22:16.241697+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:23:29.278077+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":416,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":832,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":35,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":132,"ondisk_log_size":132,"up":96,"acting":96,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":178,"num_object_clones":0,"num_object_copies":534,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":178,"num_whiteouts":0,"num_read":1104,"num_read_kb":861,"num_write":703,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":563,"num_bytes_recovered":11430,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":417792,"data_stored":11106,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3639,"internal_metadata":0},"log_size":693,"ondisk_log_size":693,"up":96,"acting":96,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":1613,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":14,"num_read_kb":14,"num_write":10,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":23,"num_bytes_recovered":7743,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":4839,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":6,"ondisk_log_size":6,"up":96,"acting":96,"num_store_stats":8},{"poolid":2,"num_pg":3,"stat_sum":{"num_bytes":408,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":419,"num_read_kb":414,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":408,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":8,"ondisk_log_size":8,"up":9,"acting":9,"num_store_stats":7},{"poolid":1,"num_pg":11,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":1573408,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":869,"ondisk_log_size":869,"up":33,"acting":33,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":56,"seq":240518168819,"num_pgs":56,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":631960,"kb_used_data":1032,"kb_used_omap":1,"kb_used_meta":630910,"kb_avail":93113192,"statfs":{"total":95995035648,"available":95347908608,"internally_reserved":0,"allocated":1056768,"data_stored":696687,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":646052298},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":47,"seq":201863463162,"num_pgs":49,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":659396,"kb_used_data":692,"kb_used_omap":1,"kb_used_meta":658686,"kb_avail":93085756,"statfs":{"total":95995035648,"available":95319814144,"internally_reserved":0,"allocated":708608,"data_stored":367922,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":674494918},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":39,"seq":167503724799,"num_pgs":53,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":979296,"kb_used_data":1296,"kb_used_omap":1,"kb_used_meta":977982,"kb_avail":92765856,"statfs":{"total":95995035648,"available":94992236544,"internally_reserved":0,"allocated":1327104,"data_stored":959428,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":1001454022},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":32,"seq":137438953733,"num_pgs":61,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":599716,"kb_used_data":724,"kb_used_omap":1,"kb_used_meta":598974,"kb_avail":93145436,"statfs":{"total":95995035648,"available":95380926464,"internally_reserved":0,"allocated":741376,"data_stored":369520,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":613349829},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":26,"seq":111669149963,"num_pgs":60,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":866848,"kb_used_data":1296,"kb_used_omap":1,"kb_used_meta":865534,"kb_avail":92878304,"statfs":{"total":95995035648,"available":95107383296,"internally_reserved":0,"allocated":1327104,"data_stored":958851,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":886307272},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411601,"num_pgs":46,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":672152,"kb_used_data":968,"kb_used_omap":1,"kb_used_meta":671166,"kb_avail":93073000,"statfs":{"total":95995035648,"available":95306752000,"internally_reserved":0,"allocated":991232,"data_stored":631634,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":687274437},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":307,"seq":1318554959971,"num_pgs":52,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":367672,"kb_used_data":744,"kb_used_omap":7,"kb_used_meta":366904,"kb_avail":93377480,"statfs":{"total":95995035648,"available":95618539520,"internally_reserved":0,"allocated":761856,"data_stored":370489,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":8079,"internal_metadata":375709809},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738653,"num_pgs":49,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":812140,"kb_used_data":732,"kb_used_omap":1,"kb_used_meta":811390,"kb_avail":92933012,"statfs":{"total":95995035648,"available":95163404288,"internally_reserved":0,"allocated":749568,"data_stored":369942,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":830863813},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":266240,"data_stored":262688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1131,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1521,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":436,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":46,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1177,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":436,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":1540,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":1678,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3639,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1238,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":1568,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1474,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1034,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1254,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":403,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":416,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":403,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T08:22:45.392 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid 779349ca-7593-11ef-bceb-c7b262605968 -- ceph pg dump --format=json 2024-09-18T08:22:45.654 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/779349ca-7593-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T08:22:46.042 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:22:45 smithi007 ceph-mon[29459]: 3.1d scrub starts 2024-09-18T08:22:46.042 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:22:45 smithi007 ceph-mon[29459]: 3.1d scrub ok 2024-09-18T08:22:46.042 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:22:45 smithi007 ceph-mon[29459]: pgmap v995: 142 pgs: 142 active+clean; 583 KiB data, 5.3 GiB used, 710 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-09-18T08:22:46.043 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:22:45 smithi007 ceph-mon[39249]: 3.1d scrub starts 2024-09-18T08:22:46.043 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:22:45 smithi007 ceph-mon[39249]: 3.1d scrub ok 2024-09-18T08:22:46.043 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:22:45 smithi007 ceph-mon[39249]: pgmap v995: 142 pgs: 142 active+clean; 583 KiB data, 5.3 GiB used, 710 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-09-18T08:22:46.155 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:22:45 smithi050 ceph-mon[32371]: 3.1d scrub starts 2024-09-18T08:22:46.155 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:22:45 smithi050 ceph-mon[32371]: 3.1d scrub ok 2024-09-18T08:22:46.155 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:22:45 smithi050 ceph-mon[32371]: pgmap v995: 142 pgs: 142 active+clean; 583 KiB data, 5.3 GiB used, 710 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-09-18T08:22:46.327 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-09-18T08:22:46.329 INFO:teuthology.orchestra.run.smithi007.stderr:dumped all 2024-09-18T08:22:46.979 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Sep 18 08:22:46 smithi007 ceph-779349ca-7593-11ef-bceb-c7b262605968-mgr-y[29688]: ::ffff:172.21.15.50 - - [18/Sep/2024:08:22:46] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-09-18T08:22:46.980 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:22:46 smithi007 ceph-mon[29459]: 6.11 scrub starts 2024-09-18T08:22:46.980 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:22:46 smithi007 ceph-mon[29459]: 6.11 scrub ok 2024-09-18T08:22:46.980 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:22:46 smithi007 ceph-mon[29459]: from='client.26585 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-18T08:22:46.980 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:22:46 smithi007 ceph-mon[39249]: 6.11 scrub starts 2024-09-18T08:22:46.980 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:22:46 smithi007 ceph-mon[39249]: 6.11 scrub ok 2024-09-18T08:22:46.980 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:22:46 smithi007 ceph-mon[39249]: from='client.26585 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-18T08:22:47.151 INFO:teuthology.orchestra.run.smithi007.stdout:{"pg_ready":true,"pg_map":{"version":995,"stamp":"2024-09-18T08:22:45.548316+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":596507,"num_objects":199,"num_object_clones":0,"num_object_copies":597,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":199,"num_whiteouts":0,"num_read":1542,"num_read_kb":1294,"num_write":722,"num_write_kb":45,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":633,"num_bytes_recovered":1593821,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1710,"ondisk_log_size":1710,"up":426,"acting":426,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":426,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":5589180,"kb_used_data":7484,"kb_used_omap":18,"kb_used_meta":5581549,"kb_avail":744372036,"statfs":{"total":767960285184,"available":762236964864,"internally_reserved":0,"allocated":7663616,"data_stored":4724473,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":19234,"internal_metadata":5715506398},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":13,"num_read_kb":13,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003174"},"pg_stats":[{"pgid":"3.1f","version":"0'0","reported_seq":947,"reported_epoch":481,"state":"active+clean","last_fresh":"2024-09-18T08:22:32.033796+0000","last_change":"2024-09-18T08:22:14.165828+0000","last_active":"2024-09-18T08:22:32.033796+0000","last_peered":"2024-09-18T08:22:32.033796+0000","last_clean":"2024-09-18T08:22:32.033796+0000","last_became_active":"2024-09-18T08:21:53.624054+0000","last_became_peered":"2024-09-18T08:21:53.624054+0000","last_unstale":"2024-09-18T08:22:32.033796+0000","last_undegraded":"2024-09-18T08:22:32.033796+0000","last_fullsized":"2024-09-18T08:22:32.033796+0000","mapping_epoch":463,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":464,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:22:14.165779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:46.860092+0000","last_clean_scrub_stamp":"2024-09-18T08:22:14.165779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:23:25.113464+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"271'15","reported_seq":72,"reported_epoch":481,"state":"active+clean","last_fresh":"2024-09-18T08:22:31.525611+0000","last_change":"2024-09-18T08:22:23.263863+0000","last_active":"2024-09-18T08:22:31.525611+0000","last_peered":"2024-09-18T08:22:31.525611+0000","last_clean":"2024-09-18T08:22:31.525611+0000","last_became_active":"2024-09-18T08:21:45.587647+0000","last_became_peered":"2024-09-18T08:21:45.587647+0000","last_unstale":"2024-09-18T08:22:31.525611+0000","last_undegraded":"2024-09-18T08:22:31.525611+0000","last_fullsized":"2024-09-18T08:22:31.525611+0000","mapping_epoch":459,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":460,"parent":"0.0","parent_split_bits":0,"last_scrub":"271'15","last_scrub_stamp":"2024-09-18T08:22:23.263766+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_clean_scrub_stamp":"2024-09-18T08:22:23.263766+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:23:27.408337+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":25,"num_read_kb":19,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,2],"acting":[4,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":758,"reported_epoch":481,"state":"active+clean","last_fresh":"2024-09-18T08:22:32.068926+0000","last_change":"2024-09-18T08:22:32.068926+0000","last_active":"2024-09-18T08:22:32.068926+0000","last_peered":"2024-09-18T08:22:32.068926+0000","last_clean":"2024-09-18T08:22:32.068926+0000","last_became_active":"2024-09-18T08:18:27.641323+0000","last_became_peered":"2024-09-18T08:18:27.641323+0000","last_unstale":"2024-09-18T08:22:32.068926+0000","last_undegraded":"2024-09-18T08:22:32.068926+0000","last_fullsized":"2024-09-18T08:22:32.068926+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:22:32.068866+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:50.886830+0000","last_clean_scrub_stamp":"2024-09-18T08:22:32.068866+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:23:59.797379+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":62,"reported_epoch":481,"state":"active+clean","last_fresh":"2024-09-18T08:22:31.526270+0000","last_change":"2024-09-18T08:21:53.286308+0000","last_active":"2024-09-18T08:22:31.526270+0000","last_peered":"2024-09-18T08:22:31.526270+0000","last_clean":"2024-09-18T08:22:31.526270+0000","last_became_active":"2024-09-18T08:21:45.587586+0000","last_became_peered":"2024-09-18T08:21:45.587586+0000","last_unstale":"2024-09-18T08:22:31.526270+0000","last_undegraded":"2024-09-18T08:22:31.526270+0000","last_fullsized":"2024-09-18T08:22:31.526270+0000","mapping_epoch":459,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":460,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:21:53.286258+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_clean_scrub_stamp":"2024-09-18T08:21:53.286258+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:23:06.789102+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":926,"reported_epoch":481,"state":"active+clean","last_fresh":"2024-09-18T08:22:31.526640+0000","last_change":"2024-09-18T08:21:49.985200+0000","last_active":"2024-09-18T08:22:31.526640+0000","last_peered":"2024-09-18T08:22:31.526640+0000","last_clean":"2024-09-18T08:22:31.526640+0000","last_became_active":"2024-09-18T08:18:27.662545+0000","last_became_peered":"2024-09-18T08:18:27.662545+0000","last_unstale":"2024-09-18T08:22:31.526640+0000","last_undegraded":"2024-09-18T08:22:31.526640+0000","last_fullsized":"2024-09-18T08:22:31.526640+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:21:49.985149+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_clean_scrub_stamp":"2024-09-18T08:21:49.985149+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:23:03.367183+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":943,"reported_epoch":481,"state":"active+clean","last_fresh":"2024-09-18T08:22:31.527699+0000","last_change":"2024-09-18T08:21:53.627364+0000","last_active":"2024-09-18T08:22:31.527699+0000","last_peered":"2024-09-18T08:22:31.527699+0000","last_clean":"2024-09-18T08:22:31.527699+0000","last_became_active":"2024-09-18T08:21:53.627031+0000","last_became_peered":"2024-09-18T08:21:53.627031+0000","last_unstale":"2024-09-18T08:22:31.527699+0000","last_undegraded":"2024-09-18T08:22:31.527699+0000","last_fullsized":"2024-09-18T08:22:31.527699+0000","mapping_epoch":463,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":464,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:21:14.758274+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:46.860092+0000","last_clean_scrub_stamp":"2024-09-18T08:21:14.758274+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"271'27","reported_seq":975,"reported_epoch":481,"state":"active+clean","last_fresh":"2024-09-18T08:22:31.525358+0000","last_change":"2024-09-18T08:22:20.119776+0000","last_active":"2024-09-18T08:22:31.525358+0000","last_peered":"2024-09-18T08:22:31.525358+0000","last_clean":"2024-09-18T08:22:31.525358+0000","last_became_active":"2024-09-18T08:21:53.626262+0000","last_became_peered":"2024-09-18T08:21:53.626262+0000","last_unstale":"2024-09-18T08:22:31.525358+0000","last_undegraded":"2024-09-18T08:22:31.525358+0000","last_fullsized":"2024-09-18T08:22:31.525358+0000","mapping_epoch":463,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":464,"parent":"0.0","parent_split_bits":0,"last_scrub":"271'27","last_scrub_stamp":"2024-09-18T08:22:20.119701+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_clean_scrub_stamp":"2024-09-18T08:22:20.119701+0000","objects_scrubbed":7,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:23:44.300990+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":44,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":62,"reported_epoch":481,"state":"active+clean","last_fresh":"2024-09-18T08:22:37.195462+0000","last_change":"2024-09-18T08:22:37.195462+0000","last_active":"2024-09-18T08:22:37.195462+0000","last_peered":"2024-09-18T08:22:37.195462+0000","last_clean":"2024-09-18T08:22:37.195462+0000","last_became_active":"2024-09-18T08:21:45.587398+0000","last_became_peered":"2024-09-18T08:21:45.587398+0000","last_unstale":"2024-09-18T08:22:37.195462+0000","last_undegraded":"2024-09-18T08:22:37.195462+0000","last_fullsized":"2024-09-18T08:22:37.195462+0000","mapping_epoch":459,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":460,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:22:37.195408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:50.886830+0000","last_clean_scrub_stamp":"2024-09-18T08:22:37.195408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:23:53.355714+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":332,"reported_epoch":481,"state":"active+clean","last_fresh":"2024-09-18T08:22:32.035160+0000","last_change":"2024-09-18T08:21:45.594957+0000","last_active":"2024-09-18T08:22:32.035160+0000","last_peered":"2024-09-18T08:22:32.035160+0000","last_clean":"2024-09-18T08:22:32.035160+0000","last_became_active":"2024-09-18T08:21:45.593564+0000","last_became_peered":"2024-09-18T08:21:45.593564+0000","last_unstale":"2024-09-18T08:22:32.035160+0000","last_undegraded":"2024-09-18T08:22:32.035160+0000","last_fullsized":"2024-09-18T08:22:32.035160+0000","mapping_epoch":459,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":460,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:21:10.635237+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:46.860092+0000","last_clean_scrub_stamp":"2024-09-18T08:21:10.635237+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"271'15","reported_seq":72,"reported_epoch":481,"state":"active+clean","last_fresh":"2024-09-18T08:22:31.525771+0000","last_change":"2024-09-18T08:22:18.390845+0000","last_active":"2024-09-18T08:22:31.525771+0000","last_peered":"2024-09-18T08:22:31.525771+0000","last_clean":"2024-09-18T08:22:31.525771+0000","last_became_active":"2024-09-18T08:21:45.595189+0000","last_became_peered":"2024-09-18T08:21:45.595189+0000","last_unstale":"2024-09-18T08:22:31.525771+0000","last_undegraded":"2024-09-18T08:22:31.525771+0000","last_fullsized":"2024-09-18T08:22:31.525771+0000","mapping_epoch":459,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":460,"parent":"0.0","parent_split_bits":0,"last_scrub":"271'15","last_scrub_stamp":"2024-09-18T08:22:18.390770+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_clean_scrub_stamp":"2024-09-18T08:22:18.390770+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:23:30.347550+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":24,"num_read_kb":18,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":340,"reported_epoch":481,"state":"active+clean","last_fresh":"2024-09-18T08:22:36.093786+0000","last_change":"2024-09-18T08:22:36.093786+0000","last_active":"2024-09-18T08:22:36.093786+0000","last_peered":"2024-09-18T08:22:36.093786+0000","last_clean":"2024-09-18T08:22:36.093786+0000","last_became_active":"2024-09-18T08:18:27.650998+0000","last_became_peered":"2024-09-18T08:18:27.650998+0000","last_unstale":"2024-09-18T08:22:36.093786+0000","last_undegraded":"2024-09-18T08:22:36.093786+0000","last_fullsized":"2024-09-18T08:22:36.093786+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:22:36.093721+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:50.886830+0000","last_clean_scrub_stamp":"2024-09-18T08:22:36.093721+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:23:49.150891+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":908,"reported_epoch":481,"state":"active+clean","last_fresh":"2024-09-18T08:22:32.032150+0000","last_change":"2024-09-18T08:22:28.191639+0000","last_active":"2024-09-18T08:22:32.032150+0000","last_peered":"2024-09-18T08:22:32.032150+0000","last_clean":"2024-09-18T08:22:32.032150+0000","last_became_active":"2024-09-18T08:18:27.644825+0000","last_became_peered":"2024-09-18T08:18:27.644825+0000","last_unstale":"2024-09-18T08:22:32.032150+0000","last_undegraded":"2024-09-18T08:22:32.032150+0000","last_fullsized":"2024-09-18T08:22:32.032150+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:22:28.191566+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_clean_scrub_stamp":"2024-09-18T08:22:28.191566+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:23:44.137075+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":332,"reported_epoch":481,"state":"active+clean","last_fresh":"2024-09-18T08:22:32.034754+0000","last_change":"2024-09-18T08:22:19.164086+0000","last_active":"2024-09-18T08:22:32.034754+0000","last_peered":"2024-09-18T08:22:32.034754+0000","last_clean":"2024-09-18T08:22:32.034754+0000","last_became_active":"2024-09-18T08:21:45.590272+0000","last_became_peered":"2024-09-18T08:21:45.590272+0000","last_unstale":"2024-09-18T08:22:32.034754+0000","last_undegraded":"2024-09-18T08:22:32.034754+0000","last_fullsized":"2024-09-18T08:22:32.034754+0000","mapping_epoch":459,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":460,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:22:19.164026+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:46.860092+0000","last_clean_scrub_stamp":"2024-09-18T08:22:19.164026+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:23:19.681795+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1b","version":"271'7","reported_seq":60,"reported_epoch":481,"state":"active+clean","last_fresh":"2024-09-18T08:22:31.526186+0000","last_change":"2024-09-18T08:21:45.611180+0000","last_active":"2024-09-18T08:22:31.526186+0000","last_peered":"2024-09-18T08:22:31.526186+0000","last_clean":"2024-09-18T08:22:31.526186+0000","last_became_active":"2024-09-18T08:21:45.585405+0000","last_became_peered":"2024-09-18T08:21:45.585405+0000","last_unstale":"2024-09-18T08:22:31.526186+0000","last_undegraded":"2024-09-18T08:22:31.526186+0000","last_fullsized":"2024-09-18T08:22:31.526186+0000","mapping_epoch":459,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":460,"parent":"0.0","parent_split_bits":0,"last_scrub":"271'7","last_scrub_stamp":"2024-09-18T08:21:16.628117+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_clean_scrub_stamp":"2024-09-18T08:21:16.628117+0000","objects_scrubbed":2,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:22:32.601541+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":13,"num_read_kb":9,"num_write":8,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":194,"reported_epoch":481,"state":"active+clean","last_fresh":"2024-09-18T08:22:32.034330+0000","last_change":"2024-09-18T08:21:55.964427+0000","last_active":"2024-09-18T08:22:32.034330+0000","last_peered":"2024-09-18T08:22:32.034330+0000","last_clean":"2024-09-18T08:22:32.034330+0000","last_became_active":"2024-09-18T08:21:45.599746+0000","last_became_peered":"2024-09-18T08:21:45.599746+0000","last_unstale":"2024-09-18T08:22:32.034330+0000","last_undegraded":"2024-09-18T08:22:32.034330+0000","last_fullsized":"2024-09-18T08:22:32.034330+0000","mapping_epoch":459,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":460,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:21:55.964371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:50.886830+0000","last_clean_scrub_stamp":"2024-09-18T08:21:55.964371+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:23:08.431483+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":326,"reported_epoch":481,"state":"active+clean","last_fresh":"2024-09-18T08:22:42.111241+0000","last_change":"2024-09-18T08:22:42.111241+0000","last_active":"2024-09-18T08:22:42.111241+0000","last_peered":"2024-09-18T08:22:42.111241+0000","last_clean":"2024-09-18T08:22:42.111241+0000","last_became_active":"2024-09-18T08:15:41.441932+0000","last_became_peered":"2024-09-18T08:15:41.441932+0000","last_unstale":"2024-09-18T08:22:42.111241+0000","last_undegraded":"2024-09-18T08:22:42.111241+0000","last_fullsized":"2024-09-18T08:22:42.111241+0000","mapping_epoch":336,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":337,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:22:42.111195+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_clean_scrub_stamp":"2024-09-18T08:22:42.111195+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:24:11.668757+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":54,"reported_epoch":481,"state":"active+clean","last_fresh":"2024-09-18T08:22:31.526611+0000","last_change":"2024-09-18T08:21:45.594910+0000","last_active":"2024-09-18T08:22:31.526611+0000","last_peered":"2024-09-18T08:22:31.526611+0000","last_clean":"2024-09-18T08:22:31.526611+0000","last_became_active":"2024-09-18T08:21:45.589375+0000","last_became_peered":"2024-09-18T08:21:45.589375+0000","last_unstale":"2024-09-18T08:22:31.526611+0000","last_undegraded":"2024-09-18T08:22:31.526611+0000","last_fullsized":"2024-09-18T08:22:31.526611+0000","mapping_epoch":459,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":460,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:20:39.609106+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_clean_scrub_stamp":"2024-09-18T08:20:39.609106+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":950,"reported_epoch":481,"state":"active+clean","last_fresh":"2024-09-18T08:22:37.182365+0000","last_change":"2024-09-18T08:22:37.182365+0000","last_active":"2024-09-18T08:22:37.182365+0000","last_peered":"2024-09-18T08:22:37.182365+0000","last_clean":"2024-09-18T08:22:37.182365+0000","last_became_active":"2024-09-18T08:21:45.590553+0000","last_became_peered":"2024-09-18T08:21:45.590553+0000","last_unstale":"2024-09-18T08:22:37.182365+0000","last_undegraded":"2024-09-18T08:22:37.182365+0000","last_fullsized":"2024-09-18T08:22:37.182365+0000","mapping_epoch":459,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":460,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:22:37.182308+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:46.860092+0000","last_clean_scrub_stamp":"2024-09-18T08:22:37.182308+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:23:56.472267+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1c","version":"271'27","reported_seq":70,"reported_epoch":481,"state":"active+clean","last_fresh":"2024-09-18T08:22:31.526323+0000","last_change":"2024-09-18T08:22:13.259006+0000","last_active":"2024-09-18T08:22:31.526323+0000","last_peered":"2024-09-18T08:22:31.526323+0000","last_clean":"2024-09-18T08:22:31.526323+0000","last_became_active":"2024-09-18T08:21:53.633735+0000","last_became_peered":"2024-09-18T08:21:53.633735+0000","last_unstale":"2024-09-18T08:22:31.526323+0000","last_undegraded":"2024-09-18T08:22:31.526323+0000","last_fullsized":"2024-09-18T08:22:31.526323+0000","mapping_epoch":463,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":464,"parent":"0.0","parent_split_bits":0,"last_scrub":"271'27","last_scrub_stamp":"2024-09-18T08:22:13.258942+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_clean_scrub_stamp":"2024-09-18T08:22:13.258942+0000","objects_scrubbed":7,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:23:26.910006+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":44,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":794,"reported_epoch":481,"state":"active+clean","last_fresh":"2024-09-18T08:22:32.032975+0000","last_change":"2024-09-18T08:22:06.772214+0000","last_active":"2024-09-18T08:22:32.032975+0000","last_peered":"2024-09-18T08:22:32.032975+0000","last_clean":"2024-09-18T08:22:32.032975+0000","last_became_active":"2024-09-18T08:21:45.588317+0000","last_became_peered":"2024-09-18T08:21:45.588317+0000","last_unstale":"2024-09-18T08:22:32.032975+0000","last_undegraded":"2024-09-18T08:22:32.032975+0000","last_fullsized":"2024-09-18T08:22:32.032975+0000","mapping_epoch":459,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":460,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:22:06.772167+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:50.886830+0000","last_clean_scrub_stamp":"2024-09-18T08:22:06.772167+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:23:24.038698+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":915,"reported_epoch":481,"state":"active+clean","last_fresh":"2024-09-18T08:22:31.527307+0000","last_change":"2024-09-18T08:22:27.021926+0000","last_active":"2024-09-18T08:22:31.527307+0000","last_peered":"2024-09-18T08:22:31.527307+0000","last_clean":"2024-09-18T08:22:31.527307+0000","last_became_active":"2024-09-18T08:15:41.466371+0000","last_became_peered":"2024-09-18T08:15:41.466371+0000","last_unstale":"2024-09-18T08:22:31.527307+0000","last_undegraded":"2024-09-18T08:22:31.527307+0000","last_fullsized":"2024-09-18T08:22:31.527307+0000","mapping_epoch":336,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":337,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:22:27.021874+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_clean_scrub_stamp":"2024-09-18T08:22:27.021874+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:23:38.719726+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":69,"reported_epoch":481,"state":"active+clean","last_fresh":"2024-09-18T08:22:31.525906+0000","last_change":"2024-09-18T08:22:25.286664+0000","last_active":"2024-09-18T08:22:31.525906+0000","last_peered":"2024-09-18T08:22:31.525906+0000","last_clean":"2024-09-18T08:22:31.525906+0000","last_became_active":"2024-09-18T08:21:53.626449+0000","last_became_peered":"2024-09-18T08:21:53.626449+0000","last_unstale":"2024-09-18T08:22:31.525906+0000","last_undegraded":"2024-09-18T08:22:31.525906+0000","last_fullsized":"2024-09-18T08:22:31.525906+0000","mapping_epoch":463,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":464,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:22:25.286605+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:46.860092+0000","last_clean_scrub_stamp":"2024-09-18T08:22:25.286605+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:23:27.940240+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"271'18","reported_seq":109,"reported_epoch":481,"state":"active+clean","last_fresh":"2024-09-18T08:22:32.034661+0000","last_change":"2024-09-18T08:21:47.269424+0000","last_active":"2024-09-18T08:22:32.034661+0000","last_peered":"2024-09-18T08:22:32.034661+0000","last_clean":"2024-09-18T08:22:32.034661+0000","last_became_active":"2024-09-18T08:21:45.590539+0000","last_became_peered":"2024-09-18T08:21:45.590539+0000","last_unstale":"2024-09-18T08:22:32.034661+0000","last_undegraded":"2024-09-18T08:22:32.034661+0000","last_fullsized":"2024-09-18T08:22:32.034661+0000","mapping_epoch":459,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":460,"parent":"0.0","parent_split_bits":0,"last_scrub":"271'18","last_scrub_stamp":"2024-09-18T08:21:47.269334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_clean_scrub_stamp":"2024-09-18T08:21:47.269334+0000","objects_scrubbed":5,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:23:00.277106+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":34,"num_read_kb":25,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":28,"num_bytes_recovered":1100,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,4],"acting":[5,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":69,"reported_epoch":481,"state":"active+clean","last_fresh":"2024-09-18T08:22:31.526410+0000","last_change":"2024-09-18T08:22:02.259158+0000","last_active":"2024-09-18T08:22:31.526410+0000","last_peered":"2024-09-18T08:22:31.526410+0000","last_clean":"2024-09-18T08:22:31.526410+0000","last_became_active":"2024-09-18T08:21:53.627096+0000","last_became_peered":"2024-09-18T08:21:53.627096+0000","last_unstale":"2024-09-18T08:22:31.526410+0000","last_undegraded":"2024-09-18T08:22:31.526410+0000","last_fullsized":"2024-09-18T08:22:31.526410+0000","mapping_epoch":463,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":464,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:22:02.259100+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:50.886830+0000","last_clean_scrub_stamp":"2024-09-18T08:22:02.259100+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:23:20.554944+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1c","version":"69'1","reported_seq":171,"reported_epoch":481,"state":"active+clean","last_fresh":"2024-09-18T08:22:32.033957+0000","last_change":"2024-09-18T08:22:20.709983+0000","last_active":"2024-09-18T08:22:32.033957+0000","last_peered":"2024-09-18T08:22:32.033957+0000","last_clean":"2024-09-18T08:22:32.033957+0000","last_became_active":"2024-09-18T08:22:16.740590+0000","last_became_peered":"2024-09-18T08:22:16.740590+0000","last_unstale":"2024-09-18T08:22:32.033957+0000","last_undegraded":"2024-09-18T08:22:32.033957+0000","last_fullsized":"2024-09-18T08:22:32.033957+0000","mapping_epoch":477,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":478,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'1","last_scrub_stamp":"2024-09-18T08:22:20.709888+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_clean_scrub_stamp":"2024-09-18T08:22:20.709888+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:23:42.046378+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":403,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":806,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.19","version":"62'1","reported_seq":803,"reported_epoch":481,"state":"active+clean","last_fresh":"2024-09-18T08:22:32.032744+0000","last_change":"2024-09-18T08:21:50.659120+0000","last_active":"2024-09-18T08:22:32.032744+0000","last_peered":"2024-09-18T08:22:32.032744+0000","last_clean":"2024-09-18T08:22:32.032744+0000","last_became_active":"2024-09-18T08:21:45.587654+0000","last_became_peered":"2024-09-18T08:21:45.587654+0000","last_unstale":"2024-09-18T08:22:32.032744+0000","last_undegraded":"2024-09-18T08:22:32.032744+0000","last_fullsized":"2024-09-18T08:22:32.032744+0000","mapping_epoch":459,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":460,"parent":"0.0","parent_split_bits":0,"last_scrub":"62'1","last_scrub_stamp":"2024-09-18T08:21:50.659051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:46.860092+0000","last_clean_scrub_stamp":"2024-09-18T08:21:50.659051+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:23:16.368410+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":92,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1e","version":"271'20","reported_seq":966,"reported_epoch":481,"state":"active+clean","last_fresh":"2024-09-18T08:22:32.033833+0000","last_change":"2024-09-18T08:22:31.164606+0000","last_active":"2024-09-18T08:22:32.033833+0000","last_peered":"2024-09-18T08:22:32.033833+0000","last_clean":"2024-09-18T08:22:32.033833+0000","last_became_active":"2024-09-18T08:18:27.657472+0000","last_became_peered":"2024-09-18T08:18:27.657472+0000","last_unstale":"2024-09-18T08:22:32.033833+0000","last_undegraded":"2024-09-18T08:22:32.033833+0000","last_fullsized":"2024-09-18T08:22:32.033833+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":0,"last_scrub":"271'20","last_scrub_stamp":"2024-09-18T08:22:31.164550+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_clean_scrub_stamp":"2024-09-18T08:22:31.164550+0000","objects_scrubbed":5,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:23:50.663068+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":25,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1f","version":"475'19","reported_seq":181,"reported_epoch":481,"state":"active+clean","last_fresh":"2024-09-18T08:22:38.166517+0000","last_change":"2024-09-18T08:22:12.719735+0000","last_active":"2024-09-18T08:22:38.166517+0000","last_peered":"2024-09-18T08:22:38.166517+0000","last_clean":"2024-09-18T08:22:38.166517+0000","last_became_active":"2024-09-18T08:22:12.719349+0000","last_became_peered":"2024-09-18T08:22:12.719349+0000","last_unstale":"2024-09-18T08:22:38.166517+0000","last_undegraded":"2024-09-18T08:22:38.166517+0000","last_fullsized":"2024-09-18T08:22:38.166517+0000","mapping_epoch":474,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":475,"parent":"0.0","parent_split_bits":0,"last_scrub":"415'17","last_scrub_stamp":"2024-09-18T08:21:42.861949+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:50.886830+0000","last_clean_scrub_stamp":"2024-09-18T08:21:42.861949+0000","objects_scrubbed":1,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:23:11.903554+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.a","version":"74'79","reported_seq":328,"reported_epoch":481,"state":"active+clean","last_fresh":"2024-09-18T08:22:32.033730+0000","last_change":"2024-09-18T08:22:27.128207+0000","last_active":"2024-09-18T08:22:32.033730+0000","last_peered":"2024-09-18T08:22:32.033730+0000","last_clean":"2024-09-18T08:22:32.033730+0000","last_became_active":"2024-09-18T08:15:41.443376+0000","last_became_peered":"2024-09-18T08:15:41.443376+0000","last_unstale":"2024-09-18T08:22:32.033730+0000","last_undegraded":"2024-09-18T08:22:32.033730+0000","last_fullsized":"2024-09-18T08:22:32.033730+0000","mapping_epoch":336,"log_start":"0'0","ondisk_log_start":"0'0","created":152,"last_epoch_clean":337,"parent":"0.0","parent_split_bits":4,"last_scrub":"74'79","last_scrub_stamp":"2024-09-18T08:22:27.128146+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:00:11.893742+0000","last_clean_scrub_stamp":"2024-09-18T08:22:27.128146+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:23:44.195317+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,6],"acting":[5,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.f","version":"271'27","reported_seq":902,"reported_epoch":481,"state":"active+clean","last_fresh":"2024-09-18T08:22:32.032533+0000","last_change":"2024-09-18T08:22:03.721475+0000","last_active":"2024-09-18T08:22:32.032533+0000","last_peered":"2024-09-18T08:22:32.032533+0000","last_clean":"2024-09-18T08:22:32.032533+0000","last_became_active":"2024-09-18T08:21:45.588963+0000","last_became_peered":"2024-09-18T08:21:45.588963+0000","last_unstale":"2024-09-18T08:22:32.032533+0000","last_undegraded":"2024-09-18T08:22:32.032533+0000","last_fullsized":"2024-09-18T08:22:32.032533+0000","mapping_epoch":459,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":460,"parent":"0.0","parent_split_bits":0,"last_scrub":"271'27","last_scrub_stamp":"2024-09-18T08:22:03.721411+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_clean_scrub_stamp":"2024-09-18T08:22:03.721411+0000","objects_scrubbed":7,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:23:27.881734+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":43,"num_read_kb":34,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":209,"reported_epoch":481,"state":"active+clean","last_fresh":"2024-09-18T08:22:31.525539+0000","last_change":"2024-09-18T08:21:49.345647+0000","last_active":"2024-09-18T08:22:31.525539+0000","last_peered":"2024-09-18T08:22:31.525539+0000","last_clean":"2024-09-18T08:22:31.525539+0000","last_became_active":"2024-09-18T08:18:27.655321+0000","last_became_peered":"2024-09-18T08:18:27.655321+0000","last_unstale":"2024-09-18T08:22:31.525539+0000","last_undegraded":"2024-09-18T08:22:31.525539+0000","last_fullsized":"2024-09-18T08:22:31.525539+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:21:49.345591+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:46.860092+0000","last_clean_scrub_stamp":"2024-09-18T08:21:49.345591+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:23:02.885177+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,7],"acting":[2,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.e","version":"460'20","reported_seq":70,"reported_epoch":481,"state":"active+clean","last_fresh":"2024-09-18T08:22:38.166548+0000","last_change":"2024-09-18T08:21:45.607932+0000","last_active":"2024-09-18T08:22:38.166548+0000","last_peered":"2024-09-18T08:22:38.166548+0000","last_clean":"2024-09-18T08:22:38.166548+0000","last_became_active":"2024-09-18T08:21:45.583046+0000","last_became_peered":"2024-09-18T08:21:45.583046+0000","last_unstale":"2024-09-18T08:22:38.166548+0000","last_undegraded":"2024-09-18T08:22:38.166548+0000","last_fullsized":"2024-09-18T08:22:38.166548+0000","mapping_epoch":459,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":460,"parent":"0.0","parent_split_bits":0,"last_scrub":"415'19","last_scrub_stamp":"2024-09-18T08:21:41.320519+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:50.886830+0000","last_clean_scrub_stamp":"2024-09-18T08:21:41.320519+0000","objects_scrubbed":1,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":319,"reported_epoch":481,"state":"active+clean","last_fresh":"2024-09-18T08:22:40.072123+0000","last_change":"2024-09-18T08:22:40.072123+0000","last_active":"2024-09-18T08:22:40.072123+0000","last_peered":"2024-09-18T08:22:40.072123+0000","last_clean":"2024-09-18T08:22:40.072123+0000","last_became_active":"2024-09-18T08:15:41.438610+0000","last_became_peered":"2024-09-18T08:15:41.438610+0000","last_unstale":"2024-09-18T08:22:40.072123+0000","last_undegraded":"2024-09-18T08:22:40.072123+0000","last_fullsized":"2024-09-18T08:22:40.072123+0000","mapping_epoch":336,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":337,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:22:40.072034+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_clean_scrub_stamp":"2024-09-18T08:22:40.072034+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:24:04.451978+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.5","version":"74'79","reported_seq":69,"reported_epoch":481,"state":"active+clean","last_fresh":"2024-09-18T08:22:31.526963+0000","last_change":"2024-09-18T08:22:01.305777+0000","last_active":"2024-09-18T08:22:31.526963+0000","last_peered":"2024-09-18T08:22:31.526963+0000","last_clean":"2024-09-18T08:22:31.526963+0000","last_became_active":"2024-09-18T08:21:53.639962+0000","last_became_peered":"2024-09-18T08:21:53.639962+0000","last_unstale":"2024-09-18T08:22:31.526963+0000","last_undegraded":"2024-09-18T08:22:31.526963+0000","last_fullsized":"2024-09-18T08:22:31.526963+0000","mapping_epoch":463,"log_start":"0'0","ondisk_log_start":"0'0","created":152,"last_epoch_clean":464,"parent":"0.0","parent_split_bits":3,"last_scrub":"74'79","last_scrub_stamp":"2024-09-18T08:22:01.305717+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:00:11.893742+0000","last_clean_scrub_stamp":"2024-09-18T08:22:01.305717+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:23:06.219083+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,6],"acting":[4,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.0","version":"271'30","reported_seq":1008,"reported_epoch":481,"state":"active+clean","last_fresh":"2024-09-18T08:22:31.525184+0000","last_change":"2024-09-18T08:21:42.952471+0000","last_active":"2024-09-18T08:22:31.525184+0000","last_peered":"2024-09-18T08:22:31.525184+0000","last_clean":"2024-09-18T08:22:31.525184+0000","last_became_active":"2024-09-18T08:18:27.651348+0000","last_became_peered":"2024-09-18T08:18:27.651348+0000","last_unstale":"2024-09-18T08:22:31.525184+0000","last_undegraded":"2024-09-18T08:22:31.525184+0000","last_fullsized":"2024-09-18T08:22:31.525184+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":0,"last_scrub":"271'30","last_scrub_stamp":"2024-09-18T08:21:42.952384+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_clean_scrub_stamp":"2024-09-18T08:21:42.952384+0000","objects_scrubbed":8,"log_size":30,"log_dups_size":0,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:22:47.274145+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":52,"num_read_kb":40,"num_write":32,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":951,"reported_epoch":481,"state":"active+clean","last_fresh":"2024-09-18T08:22:31.527827+0000","last_change":"2024-09-18T08:22:23.077674+0000","last_active":"2024-09-18T08:22:31.527827+0000","last_peered":"2024-09-18T08:22:31.527827+0000","last_clean":"2024-09-18T08:22:31.527827+0000","last_became_active":"2024-09-18T08:18:27.663173+0000","last_became_peered":"2024-09-18T08:18:27.663173+0000","last_unstale":"2024-09-18T08:22:31.527827+0000","last_undegraded":"2024-09-18T08:22:31.527827+0000","last_fullsized":"2024-09-18T08:22:31.527827+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:22:23.077630+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:46.860092+0000","last_clean_scrub_stamp":"2024-09-18T08:22:23.077630+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:23:34.205444+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":54,"reported_epoch":481,"state":"active+clean","last_fresh":"2024-09-18T08:22:31.525668+0000","last_change":"2024-09-18T08:21:45.594671+0000","last_active":"2024-09-18T08:22:31.525668+0000","last_peered":"2024-09-18T08:22:31.525668+0000","last_clean":"2024-09-18T08:22:31.525668+0000","last_became_active":"2024-09-18T08:21:45.593045+0000","last_became_peered":"2024-09-18T08:21:45.593045+0000","last_unstale":"2024-09-18T08:22:31.525668+0000","last_undegraded":"2024-09-18T08:22:31.525668+0000","last_fullsized":"2024-09-18T08:22:31.525668+0000","mapping_epoch":459,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":460,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:21:06.858313+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:50.886830+0000","last_clean_scrub_stamp":"2024-09-18T08:21:06.858313+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":69,"reported_epoch":481,"state":"active+clean","last_fresh":"2024-09-18T08:22:31.525907+0000","last_change":"2024-09-18T08:22:24.315327+0000","last_active":"2024-09-18T08:22:31.525907+0000","last_peered":"2024-09-18T08:22:31.525907+0000","last_clean":"2024-09-18T08:22:31.525907+0000","last_became_active":"2024-09-18T08:21:53.639170+0000","last_became_peered":"2024-09-18T08:21:53.639170+0000","last_unstale":"2024-09-18T08:22:31.525907+0000","last_undegraded":"2024-09-18T08:22:31.525907+0000","last_fullsized":"2024-09-18T08:22:31.525907+0000","mapping_epoch":463,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":464,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:22:24.315266+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_clean_scrub_stamp":"2024-09-18T08:22:24.315266+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:23:49.687357+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.4","version":"74'79","reported_seq":779,"reported_epoch":481,"state":"active+clean","last_fresh":"2024-09-18T08:22:31.527137+0000","last_change":"2024-09-18T08:22:26.063984+0000","last_active":"2024-09-18T08:22:31.527137+0000","last_peered":"2024-09-18T08:22:31.527137+0000","last_clean":"2024-09-18T08:22:31.527137+0000","last_became_active":"2024-09-18T08:18:27.662304+0000","last_became_peered":"2024-09-18T08:18:27.662304+0000","last_unstale":"2024-09-18T08:22:31.527137+0000","last_undegraded":"2024-09-18T08:22:31.527137+0000","last_fullsized":"2024-09-18T08:22:31.527137+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":152,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":3,"last_scrub":"74'79","last_scrub_stamp":"2024-09-18T08:22:26.063925+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:00:11.893742+0000","last_clean_scrub_stamp":"2024-09-18T08:22:26.063925+0000","objects_scrubbed":1,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:23:32.383376+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":327680,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1310720,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,7],"acting":[3,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"271'28","reported_seq":78,"reported_epoch":481,"state":"active+clean","last_fresh":"2024-09-18T08:22:41.232838+0000","last_change":"2024-09-18T08:22:41.232838+0000","last_active":"2024-09-18T08:22:41.232838+0000","last_peered":"2024-09-18T08:22:41.232838+0000","last_clean":"2024-09-18T08:22:41.232838+0000","last_became_active":"2024-09-18T08:21:45.595506+0000","last_became_peered":"2024-09-18T08:21:45.595506+0000","last_unstale":"2024-09-18T08:22:41.232838+0000","last_undegraded":"2024-09-18T08:22:41.232838+0000","last_fullsized":"2024-09-18T08:22:41.232838+0000","mapping_epoch":459,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":460,"parent":"0.0","parent_split_bits":0,"last_scrub":"271'28","last_scrub_stamp":"2024-09-18T08:22:41.232767+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_clean_scrub_stamp":"2024-09-18T08:22:41.232767+0000","objects_scrubbed":7,"log_size":28,"log_dups_size":0,"ondisk_log_size":28,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:23:59.416902+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":41,"num_read_kb":34,"num_write":28,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"62'2","reported_seq":980,"reported_epoch":481,"state":"active+clean","last_fresh":"2024-09-18T08:22:32.034045+0000","last_change":"2024-09-18T08:21:50.975417+0000","last_active":"2024-09-18T08:22:32.034045+0000","last_peered":"2024-09-18T08:22:32.034045+0000","last_clean":"2024-09-18T08:22:32.034045+0000","last_became_active":"2024-09-18T08:21:45.591454+0000","last_became_peered":"2024-09-18T08:21:45.591454+0000","last_unstale":"2024-09-18T08:22:32.034045+0000","last_undegraded":"2024-09-18T08:22:32.034045+0000","last_fullsized":"2024-09-18T08:22:32.034045+0000","mapping_epoch":459,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":460,"parent":"0.0","parent_split_bits":0,"last_scrub":"62'2","last_scrub_stamp":"2024-09-18T08:21:50.975360+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:46.860092+0000","last_clean_scrub_stamp":"2024-09-18T08:21:50.975360+0000","objects_scrubbed":2,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:23:20.043852+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1039,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":5195,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.0","version":"460'15","reported_seq":1208,"reported_epoch":481,"state":"active+clean","last_fresh":"2024-09-18T08:22:38.949974+0000","last_change":"2024-09-18T08:22:38.949926+0000","last_active":"2024-09-18T08:22:38.949974+0000","last_peered":"2024-09-18T08:22:38.949974+0000","last_clean":"2024-09-18T08:22:38.949974+0000","last_became_active":"2024-09-18T08:21:45.596425+0000","last_became_peered":"2024-09-18T08:21:45.596425+0000","last_unstale":"2024-09-18T08:22:38.949974+0000","last_undegraded":"2024-09-18T08:22:38.949974+0000","last_fullsized":"2024-09-18T08:22:38.949974+0000","mapping_epoch":459,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":460,"parent":"0.0","parent_split_bits":0,"last_scrub":"460'15","last_scrub_stamp":"2024-09-18T08:22:38.949866+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:50.886830+0000","last_clean_scrub_stamp":"2024-09-18T08:22:38.949866+0000","objects_scrubbed":1,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:23:55.605148+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,4],"acting":[3,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":319,"reported_epoch":481,"state":"active+clean","last_fresh":"2024-09-18T08:22:32.147656+0000","last_change":"2024-09-18T08:22:32.147656+0000","last_active":"2024-09-18T08:22:32.147656+0000","last_peered":"2024-09-18T08:22:32.147656+0000","last_clean":"2024-09-18T08:22:32.147656+0000","last_became_active":"2024-09-18T08:15:41.440876+0000","last_became_peered":"2024-09-18T08:15:41.440876+0000","last_unstale":"2024-09-18T08:22:32.147656+0000","last_undegraded":"2024-09-18T08:22:32.147656+0000","last_fullsized":"2024-09-18T08:22:32.147656+0000","mapping_epoch":336,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":337,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:22:32.147599+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_clean_scrub_stamp":"2024-09-18T08:22:32.147599+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:23:36.563855+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.7","version":"74'79","reported_seq":261,"reported_epoch":481,"state":"active+clean","last_fresh":"2024-09-18T08:22:32.032999+0000","last_change":"2024-09-18T08:22:14.209731+0000","last_active":"2024-09-18T08:22:32.032999+0000","last_peered":"2024-09-18T08:22:32.032999+0000","last_clean":"2024-09-18T08:22:32.032999+0000","last_became_active":"2024-09-18T08:22:12.717686+0000","last_became_peered":"2024-09-18T08:22:12.717686+0000","last_unstale":"2024-09-18T08:22:32.032999+0000","last_undegraded":"2024-09-18T08:22:32.032999+0000","last_fullsized":"2024-09-18T08:22:32.032999+0000","mapping_epoch":474,"log_start":"0'0","ondisk_log_start":"0'0","created":152,"last_epoch_clean":475,"parent":"0.0","parent_split_bits":3,"last_scrub":"74'79","last_scrub_stamp":"2024-09-18T08:22:14.209669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:00:11.893742+0000","last_clean_scrub_stamp":"2024-09-18T08:22:14.209669+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:23:31.023904+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,7],"acting":[6,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.2","version":"271'20","reported_seq":930,"reported_epoch":481,"state":"active+clean","last_fresh":"2024-09-18T08:22:32.032973+0000","last_change":"2024-09-18T08:21:56.773625+0000","last_active":"2024-09-18T08:22:32.032973+0000","last_peered":"2024-09-18T08:22:32.032973+0000","last_clean":"2024-09-18T08:22:32.032973+0000","last_became_active":"2024-09-18T08:21:45.588420+0000","last_became_peered":"2024-09-18T08:21:45.588420+0000","last_unstale":"2024-09-18T08:22:32.032973+0000","last_undegraded":"2024-09-18T08:22:32.032973+0000","last_fullsized":"2024-09-18T08:22:32.032973+0000","mapping_epoch":459,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":460,"parent":"0.0","parent_split_bits":0,"last_scrub":"271'20","last_scrub_stamp":"2024-09-18T08:21:56.773553+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_clean_scrub_stamp":"2024-09-18T08:21:56.773553+0000","objects_scrubbed":5,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:22:59.238413+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":28,"num_read_kb":23,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":319,"reported_epoch":481,"state":"active+clean","last_fresh":"2024-09-18T08:22:32.035047+0000","last_change":"2024-09-18T08:22:23.151248+0000","last_active":"2024-09-18T08:22:32.035047+0000","last_peered":"2024-09-18T08:22:32.035047+0000","last_clean":"2024-09-18T08:22:32.035047+0000","last_became_active":"2024-09-18T08:15:41.449639+0000","last_became_peered":"2024-09-18T08:15:41.449639+0000","last_unstale":"2024-09-18T08:22:32.035047+0000","last_undegraded":"2024-09-18T08:22:32.035047+0000","last_fullsized":"2024-09-18T08:22:32.035047+0000","mapping_epoch":336,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":337,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:22:23.151192+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:46.860092+0000","last_clean_scrub_stamp":"2024-09-18T08:22:23.151192+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:23:27.726330+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.3","version":"338'14","reported_seq":1184,"reported_epoch":481,"state":"active+clean","last_fresh":"2024-09-18T08:22:38.166301+0000","last_change":"2024-09-18T08:22:34.163186+0000","last_active":"2024-09-18T08:22:38.166301+0000","last_peered":"2024-09-18T08:22:38.166301+0000","last_clean":"2024-09-18T08:22:38.166301+0000","last_became_active":"2024-09-18T08:15:41.447667+0000","last_became_peered":"2024-09-18T08:15:41.447667+0000","last_unstale":"2024-09-18T08:22:38.166301+0000","last_undegraded":"2024-09-18T08:22:38.166301+0000","last_fullsized":"2024-09-18T08:22:38.166301+0000","mapping_epoch":336,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":337,"parent":"0.0","parent_split_bits":0,"last_scrub":"338'14","last_scrub_stamp":"2024-09-18T08:22:34.163133+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:50.886830+0000","last_clean_scrub_stamp":"2024-09-18T08:22:34.163133+0000","objects_scrubbed":1,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:23:43.857981+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":930,"reported_epoch":481,"state":"active+clean","last_fresh":"2024-09-18T08:22:32.032190+0000","last_change":"2024-09-18T08:21:56.008772+0000","last_active":"2024-09-18T08:22:32.032190+0000","last_peered":"2024-09-18T08:22:32.032190+0000","last_clean":"2024-09-18T08:22:32.032190+0000","last_became_active":"2024-09-18T08:21:53.629495+0000","last_became_peered":"2024-09-18T08:21:53.629495+0000","last_unstale":"2024-09-18T08:22:32.032190+0000","last_undegraded":"2024-09-18T08:22:32.032190+0000","last_fullsized":"2024-09-18T08:22:32.032190+0000","mapping_epoch":463,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":464,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:21:56.008716+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_clean_scrub_stamp":"2024-09-18T08:21:56.008716+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:23:03.875835+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.6","version":"74'79","reported_seq":261,"reported_epoch":481,"state":"active+clean","last_fresh":"2024-09-18T08:22:32.033729+0000","last_change":"2024-09-18T08:22:13.164047+0000","last_active":"2024-09-18T08:22:32.033729+0000","last_peered":"2024-09-18T08:22:32.033729+0000","last_clean":"2024-09-18T08:22:32.033729+0000","last_became_active":"2024-09-18T08:22:12.717528+0000","last_became_peered":"2024-09-18T08:22:12.717528+0000","last_unstale":"2024-09-18T08:22:32.033729+0000","last_undegraded":"2024-09-18T08:22:32.033729+0000","last_fullsized":"2024-09-18T08:22:32.033729+0000","mapping_epoch":474,"log_start":"0'0","ondisk_log_start":"0'0","created":152,"last_epoch_clean":475,"parent":"0.0","parent_split_bits":3,"last_scrub":"74'79","last_scrub_stamp":"2024-09-18T08:22:13.163954+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:00:11.893742+0000","last_clean_scrub_stamp":"2024-09-18T08:22:13.163954+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:23:29.503196+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,3],"acting":[6,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.3","version":"271'29","reported_seq":1057,"reported_epoch":481,"state":"active+clean","last_fresh":"2024-09-18T08:22:43.160260+0000","last_change":"2024-09-18T08:22:43.160260+0000","last_active":"2024-09-18T08:22:43.160260+0000","last_peered":"2024-09-18T08:22:43.160260+0000","last_clean":"2024-09-18T08:22:43.160260+0000","last_became_active":"2024-09-18T08:18:27.654192+0000","last_became_peered":"2024-09-18T08:18:27.654192+0000","last_unstale":"2024-09-18T08:22:43.160260+0000","last_undegraded":"2024-09-18T08:22:43.160260+0000","last_fullsized":"2024-09-18T08:22:43.160260+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":0,"last_scrub":"271'29","last_scrub_stamp":"2024-09-18T08:22:43.160180+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_clean_scrub_stamp":"2024-09-18T08:22:43.160180+0000","objects_scrubbed":8,"log_size":29,"log_dups_size":0,"ondisk_log_size":29,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:23:53.002412+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":54,"num_read_kb":40,"num_write":32,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":32,"num_bytes_recovered":1320,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.4","version":"62'1","reported_seq":806,"reported_epoch":481,"state":"active+clean","last_fresh":"2024-09-18T08:22:32.032985+0000","last_change":"2024-09-18T08:21:53.652773+0000","last_active":"2024-09-18T08:22:32.032985+0000","last_peered":"2024-09-18T08:22:32.032985+0000","last_clean":"2024-09-18T08:22:32.032985+0000","last_became_active":"2024-09-18T08:21:53.638433+0000","last_became_peered":"2024-09-18T08:21:53.638433+0000","last_unstale":"2024-09-18T08:22:32.032985+0000","last_undegraded":"2024-09-18T08:22:32.032985+0000","last_fullsized":"2024-09-18T08:22:32.032985+0000","mapping_epoch":463,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":464,"parent":"0.0","parent_split_bits":0,"last_scrub":"62'1","last_scrub_stamp":"2024-09-18T08:21:44.682795+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:46.860092+0000","last_clean_scrub_stamp":"2024-09-18T08:21:44.682795+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:23:08.194695+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":436,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":2180,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":546,"reported_epoch":481,"state":"active+clean","last_fresh":"2024-09-18T08:22:32.034315+0000","last_change":"2024-09-18T08:22:17.273069+0000","last_active":"2024-09-18T08:22:32.034315+0000","last_peered":"2024-09-18T08:22:32.034315+0000","last_clean":"2024-09-18T08:22:32.034315+0000","last_became_active":"2024-09-18T08:22:12.718377+0000","last_became_peered":"2024-09-18T08:22:12.718377+0000","last_unstale":"2024-09-18T08:22:32.034315+0000","last_undegraded":"2024-09-18T08:22:32.034315+0000","last_fullsized":"2024-09-18T08:22:32.034315+0000","mapping_epoch":474,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":475,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:22:17.273010+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:50.886830+0000","last_clean_scrub_stamp":"2024-09-18T08:22:17.273010+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:23:23.711311+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":817,"reported_epoch":481,"state":"active+clean","last_fresh":"2024-09-18T08:22:32.032216+0000","last_change":"2024-09-18T08:22:07.816688+0000","last_active":"2024-09-18T08:22:32.032216+0000","last_peered":"2024-09-18T08:22:32.032216+0000","last_clean":"2024-09-18T08:22:32.032216+0000","last_became_active":"2024-09-18T08:21:53.636814+0000","last_became_peered":"2024-09-18T08:21:53.636814+0000","last_unstale":"2024-09-18T08:22:32.032216+0000","last_undegraded":"2024-09-18T08:22:32.032216+0000","last_fullsized":"2024-09-18T08:22:32.032216+0000","mapping_epoch":463,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":464,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:22:07.816632+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_clean_scrub_stamp":"2024-09-18T08:22:07.816632+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:23:32.777862+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"74'79","reported_seq":328,"reported_epoch":481,"state":"active+clean","last_fresh":"2024-09-18T08:22:32.033115+0000","last_change":"2024-09-18T08:22:24.111048+0000","last_active":"2024-09-18T08:22:32.033115+0000","last_peered":"2024-09-18T08:22:32.033115+0000","last_clean":"2024-09-18T08:22:32.033115+0000","last_became_active":"2024-09-18T08:15:41.452895+0000","last_became_peered":"2024-09-18T08:15:41.452895+0000","last_unstale":"2024-09-18T08:22:32.033115+0000","last_undegraded":"2024-09-18T08:22:32.033115+0000","last_fullsized":"2024-09-18T08:22:32.033115+0000","mapping_epoch":336,"log_start":"0'0","ondisk_log_start":"0'0","created":152,"last_epoch_clean":337,"parent":"0.0","parent_split_bits":4,"last_scrub":"74'79","last_scrub_stamp":"2024-09-18T08:22:24.110984+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:00:11.893742+0000","last_clean_scrub_stamp":"2024-09-18T08:22:24.110984+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:23:31.218197+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,2],"acting":[5,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.4","version":"464'44","reported_seq":1061,"reported_epoch":481,"state":"active+clean","last_fresh":"2024-09-18T08:22:39.057074+0000","last_change":"2024-09-18T08:22:39.056981+0000","last_active":"2024-09-18T08:22:39.057074+0000","last_peered":"2024-09-18T08:22:39.057074+0000","last_clean":"2024-09-18T08:22:39.057074+0000","last_became_active":"2024-09-18T08:21:53.639078+0000","last_became_peered":"2024-09-18T08:21:53.639078+0000","last_unstale":"2024-09-18T08:22:39.057074+0000","last_undegraded":"2024-09-18T08:22:39.057074+0000","last_fullsized":"2024-09-18T08:22:39.057074+0000","mapping_epoch":463,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":464,"parent":"0.0","parent_split_bits":0,"last_scrub":"464'44","last_scrub_stamp":"2024-09-18T08:22:39.056905+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_clean_scrub_stamp":"2024-09-18T08:22:39.056905+0000","objects_scrubbed":10,"log_size":44,"log_dups_size":0,"ondisk_log_size":44,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:23:54.948602+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":65,"num_read_kb":50,"num_write":38,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":26,"num_bytes_recovered":716,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":62,"reported_epoch":481,"state":"active+clean","last_fresh":"2024-09-18T08:22:31.526058+0000","last_change":"2024-09-18T08:22:05.210106+0000","last_active":"2024-09-18T08:22:31.526058+0000","last_peered":"2024-09-18T08:22:31.526058+0000","last_clean":"2024-09-18T08:22:31.526058+0000","last_became_active":"2024-09-18T08:21:45.589441+0000","last_became_peered":"2024-09-18T08:21:45.589441+0000","last_unstale":"2024-09-18T08:22:31.526058+0000","last_undegraded":"2024-09-18T08:22:31.526058+0000","last_fullsized":"2024-09-18T08:22:31.526058+0000","mapping_epoch":459,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":460,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:22:05.210024+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:46.860092+0000","last_clean_scrub_stamp":"2024-09-18T08:22:05.210024+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:23:05.666864+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"64'2","reported_seq":331,"reported_epoch":481,"state":"active+clean","last_fresh":"2024-09-18T08:22:32.033516+0000","last_change":"2024-09-18T08:21:52.179005+0000","last_active":"2024-09-18T08:22:32.033516+0000","last_peered":"2024-09-18T08:22:32.033516+0000","last_clean":"2024-09-18T08:22:32.033516+0000","last_became_active":"2024-09-18T08:15:41.453351+0000","last_became_peered":"2024-09-18T08:15:41.453351+0000","last_unstale":"2024-09-18T08:22:32.033516+0000","last_undegraded":"2024-09-18T08:22:32.033516+0000","last_fullsized":"2024-09-18T08:22:32.033516+0000","mapping_epoch":336,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":337,"parent":"0.0","parent_split_bits":0,"last_scrub":"64'2","last_scrub_stamp":"2024-09-18T08:21:52.178938+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:45.855979+0000","last_clean_scrub_stamp":"2024-09-18T08:21:52.178938+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:23:06.524708+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":928,"reported_epoch":481,"state":"active+clean","last_fresh":"2024-09-18T08:22:32.033620+0000","last_change":"2024-09-18T08:22:23.248584+0000","last_active":"2024-09-18T08:22:32.033620+0000","last_peered":"2024-09-18T08:22:32.033620+0000","last_clean":"2024-09-18T08:22:32.033620+0000","last_became_active":"2024-09-18T08:21:45.591280+0000","last_became_peered":"2024-09-18T08:21:45.591280+0000","last_unstale":"2024-09-18T08:22:32.033620+0000","last_undegraded":"2024-09-18T08:22:32.033620+0000","last_fullsized":"2024-09-18T08:22:32.033620+0000","mapping_epoch":459,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":460,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:22:23.248536+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:50.886830+0000","last_clean_scrub_stamp":"2024-09-18T08:22:23.248536+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:23:32.744078+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":946,"reported_epoch":481,"state":"active+clean","last_fresh":"2024-09-18T08:22:31.527489+0000","last_change":"2024-09-18T08:22:28.053267+0000","last_active":"2024-09-18T08:22:31.527489+0000","last_peered":"2024-09-18T08:22:31.527489+0000","last_clean":"2024-09-18T08:22:31.527489+0000","last_became_active":"2024-09-18T08:21:45.597448+0000","last_became_peered":"2024-09-18T08:21:45.597448+0000","last_unstale":"2024-09-18T08:22:31.527489+0000","last_undegraded":"2024-09-18T08:22:31.527489+0000","last_fullsized":"2024-09-18T08:22:31.527489+0000","mapping_epoch":459,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":460,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:22:28.053216+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_clean_scrub_stamp":"2024-09-18T08:22:28.053216+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:23:51.238713+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"74'79","reported_seq":684,"reported_epoch":481,"state":"active+clean","last_fresh":"2024-09-18T08:22:32.034160+0000","last_change":"2024-09-18T08:22:16.737058+0000","last_active":"2024-09-18T08:22:32.034160+0000","last_peered":"2024-09-18T08:22:32.034160+0000","last_clean":"2024-09-18T08:22:32.034160+0000","last_became_active":"2024-09-18T08:22:16.736551+0000","last_became_peered":"2024-09-18T08:22:16.736551+0000","last_unstale":"2024-09-18T08:22:32.034160+0000","last_undegraded":"2024-09-18T08:22:32.034160+0000","last_fullsized":"2024-09-18T08:22:32.034160+0000","mapping_epoch":477,"log_start":"0'0","ondisk_log_start":"0'0","created":152,"last_epoch_clean":478,"parent":"0.0","parent_split_bits":4,"last_scrub":"74'79","last_scrub_stamp":"2024-09-18T08:21:45.932207+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:00:11.893742+0000","last_clean_scrub_stamp":"2024-09-18T08:21:45.932207+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:23:08.395270+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.7","version":"271'17","reported_seq":906,"reported_epoch":481,"state":"active+clean","last_fresh":"2024-09-18T08:22:31.526654+0000","last_change":"2024-09-18T08:21:46.316991+0000","last_active":"2024-09-18T08:22:31.526654+0000","last_peered":"2024-09-18T08:22:31.526654+0000","last_clean":"2024-09-18T08:22:31.526654+0000","last_became_active":"2024-09-18T08:15:41.456482+0000","last_became_peered":"2024-09-18T08:15:41.456482+0000","last_unstale":"2024-09-18T08:22:31.526654+0000","last_undegraded":"2024-09-18T08:22:31.526654+0000","last_fullsized":"2024-09-18T08:22:31.526654+0000","mapping_epoch":336,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":337,"parent":"0.0","parent_split_bits":0,"last_scrub":"271'17","last_scrub_stamp":"2024-09-18T08:21:46.316928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_clean_scrub_stamp":"2024-09-18T08:21:46.316928+0000","objects_scrubbed":5,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:22:46.825876+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":35,"num_read_kb":24,"num_write":20,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":990,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,0],"acting":[2,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":800,"reported_epoch":481,"state":"active+clean","last_fresh":"2024-09-18T08:22:32.032288+0000","last_change":"2024-09-18T08:21:53.634626+0000","last_active":"2024-09-18T08:22:32.032288+0000","last_peered":"2024-09-18T08:22:32.032288+0000","last_clean":"2024-09-18T08:22:32.032288+0000","last_became_active":"2024-09-18T08:21:53.634307+0000","last_became_peered":"2024-09-18T08:21:53.634307+0000","last_unstale":"2024-09-18T08:22:32.032288+0000","last_undegraded":"2024-09-18T08:22:32.032288+0000","last_fullsized":"2024-09-18T08:22:32.032288+0000","mapping_epoch":463,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":464,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:21:46.671697+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:46.860092+0000","last_clean_scrub_stamp":"2024-09-18T08:21:46.671697+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:23:16.398511+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"62'1","reported_seq":58,"reported_epoch":481,"state":"active+clean","last_fresh":"2024-09-18T08:22:31.525837+0000","last_change":"2024-09-18T08:21:57.305065+0000","last_active":"2024-09-18T08:22:31.525837+0000","last_peered":"2024-09-18T08:22:31.525837+0000","last_clean":"2024-09-18T08:22:31.525837+0000","last_became_active":"2024-09-18T08:21:53.628297+0000","last_became_peered":"2024-09-18T08:21:53.628297+0000","last_unstale":"2024-09-18T08:22:31.525837+0000","last_undegraded":"2024-09-18T08:22:31.525837+0000","last_fullsized":"2024-09-18T08:22:31.525837+0000","mapping_epoch":463,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":464,"parent":"0.0","parent_split_bits":0,"last_scrub":"62'1","last_scrub_stamp":"2024-09-18T08:21:57.304990+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:45.855979+0000","last_clean_scrub_stamp":"2024-09-18T08:21:57.304990+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:23:17.093474+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":46,"reported_epoch":481,"state":"active+clean","last_fresh":"2024-09-18T08:22:31.526397+0000","last_change":"2024-09-18T08:21:53.632798+0000","last_active":"2024-09-18T08:22:31.526397+0000","last_peered":"2024-09-18T08:22:31.526397+0000","last_clean":"2024-09-18T08:22:31.526397+0000","last_became_active":"2024-09-18T08:21:53.632492+0000","last_became_peered":"2024-09-18T08:21:53.632492+0000","last_unstale":"2024-09-18T08:22:31.526397+0000","last_undegraded":"2024-09-18T08:22:31.526397+0000","last_fullsized":"2024-09-18T08:22:31.526397+0000","mapping_epoch":463,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":464,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:21:48.291960+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:50.886830+0000","last_clean_scrub_stamp":"2024-09-18T08:21:48.291960+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:23:02.145076+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":446,"reported_epoch":481,"state":"active+clean","last_fresh":"2024-09-18T08:22:32.034470+0000","last_change":"2024-09-18T08:22:17.735858+0000","last_active":"2024-09-18T08:22:32.034470+0000","last_peered":"2024-09-18T08:22:32.034470+0000","last_clean":"2024-09-18T08:22:32.034470+0000","last_became_active":"2024-09-18T08:22:16.741247+0000","last_became_peered":"2024-09-18T08:22:16.741247+0000","last_unstale":"2024-09-18T08:22:32.034470+0000","last_undegraded":"2024-09-18T08:22:32.034470+0000","last_fullsized":"2024-09-18T08:22:32.034470+0000","mapping_epoch":477,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":478,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:22:17.735788+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_clean_scrub_stamp":"2024-09-18T08:22:17.735788+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:23:41.651047+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.3","version":"74'79","reported_seq":576,"reported_epoch":481,"state":"active+clean","last_fresh":"2024-09-18T08:22:32.032564+0000","last_change":"2024-09-18T08:21:58.768736+0000","last_active":"2024-09-18T08:22:32.032564+0000","last_peered":"2024-09-18T08:22:32.032564+0000","last_clean":"2024-09-18T08:22:32.032564+0000","last_became_active":"2024-09-18T08:21:45.589564+0000","last_became_peered":"2024-09-18T08:21:45.589564+0000","last_unstale":"2024-09-18T08:22:32.032564+0000","last_undegraded":"2024-09-18T08:22:32.032564+0000","last_fullsized":"2024-09-18T08:22:32.032564+0000","mapping_epoch":459,"log_start":"0'0","ondisk_log_start":"0'0","created":152,"last_epoch_clean":460,"parent":"0.0","parent_split_bits":3,"last_scrub":"74'79","last_scrub_stamp":"2024-09-18T08:21:58.768688+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:00:11.893742+0000","last_clean_scrub_stamp":"2024-09-18T08:21:58.768688+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:23:10.597418+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.6","version":"271'24","reported_seq":1047,"reported_epoch":481,"state":"active+clean","last_fresh":"2024-09-18T08:22:32.033937+0000","last_change":"2024-09-18T08:21:53.652906+0000","last_active":"2024-09-18T08:22:32.033937+0000","last_peered":"2024-09-18T08:22:32.033937+0000","last_clean":"2024-09-18T08:22:32.033937+0000","last_became_active":"2024-09-18T08:21:53.636736+0000","last_became_peered":"2024-09-18T08:21:53.636736+0000","last_unstale":"2024-09-18T08:22:32.033937+0000","last_undegraded":"2024-09-18T08:22:32.033937+0000","last_fullsized":"2024-09-18T08:22:32.033937+0000","mapping_epoch":463,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":464,"parent":"0.0","parent_split_bits":0,"last_scrub":"271'24","last_scrub_stamp":"2024-09-18T08:21:42.907354+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_clean_scrub_stamp":"2024-09-18T08:21:42.907354+0000","objects_scrubbed":6,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:22:56.274053+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":36,"num_read_kb":30,"num_write":24,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":30,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":941,"reported_epoch":481,"state":"active+clean","last_fresh":"2024-09-18T08:22:32.032657+0000","last_change":"2024-09-18T08:21:49.984256+0000","last_active":"2024-09-18T08:22:32.032657+0000","last_peered":"2024-09-18T08:22:32.032657+0000","last_clean":"2024-09-18T08:22:32.032657+0000","last_became_active":"2024-09-18T08:21:45.590631+0000","last_became_peered":"2024-09-18T08:21:45.590631+0000","last_unstale":"2024-09-18T08:22:32.032657+0000","last_undegraded":"2024-09-18T08:22:32.032657+0000","last_fullsized":"2024-09-18T08:22:32.032657+0000","mapping_epoch":459,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":460,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:21:49.984202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:46.860092+0000","last_clean_scrub_stamp":"2024-09-18T08:21:49.984202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:22:52.072896+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"70'5","reported_seq":958,"reported_epoch":481,"state":"active+clean","last_fresh":"2024-09-18T08:22:41.777404+0000","last_change":"2024-09-18T08:22:16.738311+0000","last_active":"2024-09-18T08:22:41.777404+0000","last_peered":"2024-09-18T08:22:41.777404+0000","last_clean":"2024-09-18T08:22:41.777404+0000","last_became_active":"2024-09-18T08:22:16.737474+0000","last_became_peered":"2024-09-18T08:22:16.737474+0000","last_unstale":"2024-09-18T08:22:41.777404+0000","last_undegraded":"2024-09-18T08:22:41.777404+0000","last_fullsized":"2024-09-18T08:22:41.777404+0000","mapping_epoch":477,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":478,"parent":"0.0","parent_split_bits":0,"last_scrub":"70'5","last_scrub_stamp":"2024-09-18T08:21:49.612225+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:45.855979+0000","last_clean_scrub_stamp":"2024-09-18T08:21:49.612225+0000","objects_scrubbed":1,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:22:54.818286+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0.000504055,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":424,"num_read_kb":419,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":389,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":319,"reported_epoch":481,"state":"active+clean","last_fresh":"2024-09-18T08:22:32.035412+0000","last_change":"2024-09-18T08:22:15.211359+0000","last_active":"2024-09-18T08:22:32.035412+0000","last_peered":"2024-09-18T08:22:32.035412+0000","last_clean":"2024-09-18T08:22:32.035412+0000","last_became_active":"2024-09-18T08:15:41.443080+0000","last_became_peered":"2024-09-18T08:15:41.443080+0000","last_unstale":"2024-09-18T08:22:32.035412+0000","last_undegraded":"2024-09-18T08:22:32.035412+0000","last_fullsized":"2024-09-18T08:22:32.035412+0000","mapping_epoch":336,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":337,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:22:15.211298+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:50.886830+0000","last_clean_scrub_stamp":"2024-09-18T08:22:15.211298+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:23:43.279202+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":723,"reported_epoch":481,"state":"active+clean","last_fresh":"2024-09-18T08:22:32.032983+0000","last_change":"2024-09-18T08:22:11.803565+0000","last_active":"2024-09-18T08:22:32.032983+0000","last_peered":"2024-09-18T08:22:32.032983+0000","last_clean":"2024-09-18T08:22:32.032983+0000","last_became_active":"2024-09-18T08:15:41.436692+0000","last_became_peered":"2024-09-18T08:15:41.436692+0000","last_unstale":"2024-09-18T08:22:32.032983+0000","last_undegraded":"2024-09-18T08:22:32.032983+0000","last_fullsized":"2024-09-18T08:22:32.032983+0000","mapping_epoch":336,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":337,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:22:11.803517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_clean_scrub_stamp":"2024-09-18T08:22:11.803517+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:23:28.633455+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.5","version":"475'38","reported_seq":322,"reported_epoch":481,"state":"active+clean","last_fresh":"2024-09-18T08:22:38.166646+0000","last_change":"2024-09-18T08:22:12.719747+0000","last_active":"2024-09-18T08:22:38.166646+0000","last_peered":"2024-09-18T08:22:38.166646+0000","last_clean":"2024-09-18T08:22:38.166646+0000","last_became_active":"2024-09-18T08:22:12.719275+0000","last_became_peered":"2024-09-18T08:22:12.719275+0000","last_unstale":"2024-09-18T08:22:38.166646+0000","last_undegraded":"2024-09-18T08:22:38.166646+0000","last_fullsized":"2024-09-18T08:22:38.166646+0000","mapping_epoch":474,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":475,"parent":"0.0","parent_split_bits":0,"last_scrub":"460'37","last_scrub_stamp":"2024-09-18T08:22:06.213944+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_clean_scrub_stamp":"2024-09-18T08:22:06.213944+0000","objects_scrubbed":8,"log_size":38,"log_dups_size":0,"ondisk_log_size":38,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:23:16.818383+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":41,"num_read_kb":31,"num_write":25,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":154,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":927,"reported_epoch":481,"state":"active+clean","last_fresh":"2024-09-18T08:22:31.526494+0000","last_change":"2024-09-18T08:22:19.090392+0000","last_active":"2024-09-18T08:22:31.526494+0000","last_peered":"2024-09-18T08:22:31.526494+0000","last_clean":"2024-09-18T08:22:31.526494+0000","last_became_active":"2024-09-18T08:15:41.464313+0000","last_became_peered":"2024-09-18T08:15:41.464313+0000","last_unstale":"2024-09-18T08:22:31.526494+0000","last_undegraded":"2024-09-18T08:22:31.526494+0000","last_fullsized":"2024-09-18T08:22:31.526494+0000","mapping_epoch":336,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":337,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:22:19.090346+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:46.860092+0000","last_clean_scrub_stamp":"2024-09-18T08:22:19.090346+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:23:46.659025+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"74'79","reported_seq":365,"reported_epoch":481,"state":"active+clean","last_fresh":"2024-09-18T08:22:32.032738+0000","last_change":"2024-09-18T08:22:16.741948+0000","last_active":"2024-09-18T08:22:32.032738+0000","last_peered":"2024-09-18T08:22:32.032738+0000","last_clean":"2024-09-18T08:22:32.032738+0000","last_became_active":"2024-09-18T08:22:16.741593+0000","last_became_peered":"2024-09-18T08:22:16.741593+0000","last_unstale":"2024-09-18T08:22:32.032738+0000","last_undegraded":"2024-09-18T08:22:32.032738+0000","last_fullsized":"2024-09-18T08:22:32.032738+0000","mapping_epoch":477,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":478,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'79","last_scrub_stamp":"2024-09-18T08:22:06.139534+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:00:11.893742+0000","last_clean_scrub_stamp":"2024-09-18T08:22:06.139534+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:23:06.500076+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":611,"reported_epoch":481,"state":"active+clean","last_fresh":"2024-09-18T08:22:32.033776+0000","last_change":"2024-09-18T08:22:16.741478+0000","last_active":"2024-09-18T08:22:32.033776+0000","last_peered":"2024-09-18T08:22:32.033776+0000","last_clean":"2024-09-18T08:22:32.033776+0000","last_became_active":"2024-09-18T08:22:16.741092+0000","last_became_peered":"2024-09-18T08:22:16.741092+0000","last_unstale":"2024-09-18T08:22:32.033776+0000","last_undegraded":"2024-09-18T08:22:32.033776+0000","last_fullsized":"2024-09-18T08:22:32.033776+0000","mapping_epoch":477,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":478,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:21:45.316642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:50.886830+0000","last_clean_scrub_stamp":"2024-09-18T08:21:45.316642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:23:01.737258+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":333,"reported_epoch":481,"state":"active+clean","last_fresh":"2024-09-18T08:22:32.035260+0000","last_change":"2024-09-18T08:21:45.592063+0000","last_active":"2024-09-18T08:22:32.035260+0000","last_peered":"2024-09-18T08:22:32.035260+0000","last_clean":"2024-09-18T08:22:32.035260+0000","last_became_active":"2024-09-18T08:21:45.591625+0000","last_became_peered":"2024-09-18T08:21:45.591625+0000","last_unstale":"2024-09-18T08:22:32.035260+0000","last_undegraded":"2024-09-18T08:22:32.035260+0000","last_fullsized":"2024-09-18T08:22:32.035260+0000","mapping_epoch":459,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":460,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:21:19.630858+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_clean_scrub_stamp":"2024-09-18T08:21:19.630858+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.e","version":"271'19","reported_seq":74,"reported_epoch":481,"state":"active+clean","last_fresh":"2024-09-18T08:22:31.526809+0000","last_change":"2024-09-18T08:21:59.244331+0000","last_active":"2024-09-18T08:22:31.526809+0000","last_peered":"2024-09-18T08:22:31.526809+0000","last_clean":"2024-09-18T08:22:31.526809+0000","last_became_active":"2024-09-18T08:21:45.593827+0000","last_became_peered":"2024-09-18T08:21:45.593827+0000","last_unstale":"2024-09-18T08:22:31.526809+0000","last_undegraded":"2024-09-18T08:22:31.526809+0000","last_fullsized":"2024-09-18T08:22:31.526809+0000","mapping_epoch":459,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":460,"parent":"0.0","parent_split_bits":0,"last_scrub":"271'19","last_scrub_stamp":"2024-09-18T08:21:59.244247+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_clean_scrub_stamp":"2024-09-18T08:21:59.244247+0000","objects_scrubbed":5,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:23:23.591222+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":31,"num_read_kb":24,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":69,"reported_epoch":481,"state":"active+clean","last_fresh":"2024-09-18T08:22:31.526794+0000","last_change":"2024-09-18T08:22:17.370543+0000","last_active":"2024-09-18T08:22:31.526794+0000","last_peered":"2024-09-18T08:22:31.526794+0000","last_clean":"2024-09-18T08:22:31.526794+0000","last_became_active":"2024-09-18T08:21:53.626983+0000","last_became_peered":"2024-09-18T08:21:53.626983+0000","last_unstale":"2024-09-18T08:22:31.526794+0000","last_undegraded":"2024-09-18T08:22:31.526794+0000","last_fullsized":"2024-09-18T08:22:31.526794+0000","mapping_epoch":463,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":464,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:22:17.370488+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:46.860092+0000","last_clean_scrub_stamp":"2024-09-18T08:22:17.370488+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:23:41.036646+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":340,"reported_epoch":481,"state":"active+clean","last_fresh":"2024-09-18T08:22:32.035138+0000","last_change":"2024-09-18T08:21:45.592459+0000","last_active":"2024-09-18T08:22:32.035138+0000","last_peered":"2024-09-18T08:22:32.035138+0000","last_clean":"2024-09-18T08:22:32.035138+0000","last_became_active":"2024-09-18T08:21:45.592071+0000","last_became_peered":"2024-09-18T08:21:45.592071+0000","last_unstale":"2024-09-18T08:22:32.035138+0000","last_undegraded":"2024-09-18T08:22:32.035138+0000","last_fullsized":"2024-09-18T08:22:32.035138+0000","mapping_epoch":459,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":460,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:21:42.295757+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:50.886830+0000","last_clean_scrub_stamp":"2024-09-18T08:21:42.295757+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:23:11.142413+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":914,"reported_epoch":481,"state":"active+clean","last_fresh":"2024-09-18T08:22:36.021957+0000","last_change":"2024-09-18T08:22:36.021957+0000","last_active":"2024-09-18T08:22:36.021957+0000","last_peered":"2024-09-18T08:22:36.021957+0000","last_clean":"2024-09-18T08:22:36.021957+0000","last_became_active":"2024-09-18T08:15:41.464500+0000","last_became_peered":"2024-09-18T08:15:41.464500+0000","last_unstale":"2024-09-18T08:22:36.021957+0000","last_undegraded":"2024-09-18T08:22:36.021957+0000","last_fullsized":"2024-09-18T08:22:36.021957+0000","mapping_epoch":336,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":337,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:22:36.021908+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_clean_scrub_stamp":"2024-09-18T08:22:36.021908+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:24:02.353796+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"74'79","reported_seq":332,"reported_epoch":481,"state":"active+clean","last_fresh":"2024-09-18T08:22:32.034098+0000","last_change":"2024-09-18T08:22:13.185171+0000","last_active":"2024-09-18T08:22:32.034098+0000","last_peered":"2024-09-18T08:22:32.034098+0000","last_clean":"2024-09-18T08:22:32.034098+0000","last_became_active":"2024-09-18T08:15:41.454781+0000","last_became_peered":"2024-09-18T08:15:41.454781+0000","last_unstale":"2024-09-18T08:22:32.034098+0000","last_undegraded":"2024-09-18T08:22:32.034098+0000","last_fullsized":"2024-09-18T08:22:32.034098+0000","mapping_epoch":336,"log_start":"0'0","ondisk_log_start":"0'0","created":152,"last_epoch_clean":337,"parent":"0.0","parent_split_bits":4,"last_scrub":"74'79","last_scrub_stamp":"2024-09-18T08:22:13.185103+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:00:11.893742+0000","last_clean_scrub_stamp":"2024-09-18T08:22:13.185103+0000","objects_scrubbed":1,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:23:20.190255+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262688,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":262688,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,3],"acting":[5,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.d","version":"271'31","reported_seq":105,"reported_epoch":481,"state":"active+clean","last_fresh":"2024-09-18T08:22:31.526664+0000","last_change":"2024-09-18T08:22:30.263554+0000","last_active":"2024-09-18T08:22:31.526664+0000","last_peered":"2024-09-18T08:22:31.526664+0000","last_clean":"2024-09-18T08:22:31.526664+0000","last_became_active":"2024-09-18T08:21:53.638798+0000","last_became_peered":"2024-09-18T08:21:53.638798+0000","last_unstale":"2024-09-18T08:22:31.526664+0000","last_undegraded":"2024-09-18T08:22:31.526664+0000","last_fullsized":"2024-09-18T08:22:31.526664+0000","mapping_epoch":463,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":464,"parent":"0.0","parent_split_bits":0,"last_scrub":"271'31","last_scrub_stamp":"2024-09-18T08:22:30.263478+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_clean_scrub_stamp":"2024-09-18T08:22:30.263478+0000","objects_scrubbed":8,"log_size":31,"log_dups_size":0,"ondisk_log_size":31,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:23:36.733330+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":49,"num_read_kb":39,"num_write":32,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,1],"acting":[4,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":311,"reported_epoch":481,"state":"active+clean","last_fresh":"2024-09-18T08:22:41.344748+0000","last_change":"2024-09-18T08:22:41.344748+0000","last_active":"2024-09-18T08:22:41.344748+0000","last_peered":"2024-09-18T08:22:41.344748+0000","last_clean":"2024-09-18T08:22:41.344748+0000","last_became_active":"2024-09-18T08:22:12.716430+0000","last_became_peered":"2024-09-18T08:22:12.716430+0000","last_unstale":"2024-09-18T08:22:41.344748+0000","last_undegraded":"2024-09-18T08:22:41.344748+0000","last_fullsized":"2024-09-18T08:22:41.344748+0000","mapping_epoch":474,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":475,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:22:41.344701+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:46.860092+0000","last_clean_scrub_stamp":"2024-09-18T08:22:41.344701+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:23:52.718840+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":807,"reported_epoch":481,"state":"active+clean","last_fresh":"2024-09-18T08:22:32.032867+0000","last_change":"2024-09-18T08:21:45.589662+0000","last_active":"2024-09-18T08:22:32.032867+0000","last_peered":"2024-09-18T08:22:32.032867+0000","last_clean":"2024-09-18T08:22:32.032867+0000","last_became_active":"2024-09-18T08:21:45.589298+0000","last_became_peered":"2024-09-18T08:21:45.589298+0000","last_unstale":"2024-09-18T08:22:32.032867+0000","last_undegraded":"2024-09-18T08:22:32.032867+0000","last_fullsized":"2024-09-18T08:22:32.032867+0000","mapping_epoch":459,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":460,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:21:43.665555+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:50.886830+0000","last_clean_scrub_stamp":"2024-09-18T08:21:43.665555+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:23:09.750846+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":54,"reported_epoch":481,"state":"active+clean","last_fresh":"2024-09-18T08:22:31.526553+0000","last_change":"2024-09-18T08:22:12.301456+0000","last_active":"2024-09-18T08:22:31.526553+0000","last_peered":"2024-09-18T08:22:31.526553+0000","last_clean":"2024-09-18T08:22:31.526553+0000","last_became_active":"2024-09-18T08:21:53.637312+0000","last_became_peered":"2024-09-18T08:21:53.637312+0000","last_unstale":"2024-09-18T08:22:31.526553+0000","last_undegraded":"2024-09-18T08:22:31.526553+0000","last_fullsized":"2024-09-18T08:22:31.526553+0000","mapping_epoch":463,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":464,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:22:12.301396+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_clean_scrub_stamp":"2024-09-18T08:22:12.301396+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:23:20.413823+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"74'79","reported_seq":589,"reported_epoch":481,"state":"active+clean","last_fresh":"2024-09-18T08:22:41.768561+0000","last_change":"2024-09-18T08:22:41.768561+0000","last_active":"2024-09-18T08:22:41.768561+0000","last_peered":"2024-09-18T08:22:41.768561+0000","last_clean":"2024-09-18T08:22:41.768561+0000","last_became_active":"2024-09-18T08:22:16.739529+0000","last_became_peered":"2024-09-18T08:22:16.739529+0000","last_unstale":"2024-09-18T08:22:41.768561+0000","last_undegraded":"2024-09-18T08:22:41.768561+0000","last_fullsized":"2024-09-18T08:22:41.768561+0000","mapping_epoch":477,"log_start":"0'0","ondisk_log_start":"0'0","created":152,"last_epoch_clean":478,"parent":"0.0","parent_split_bits":4,"last_scrub":"74'79","last_scrub_stamp":"2024-09-18T08:22:41.768512+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:00:11.893742+0000","last_clean_scrub_stamp":"2024-09-18T08:22:41.768512+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:24:06.436093+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,2],"acting":[7,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.c","version":"271'20","reported_seq":74,"reported_epoch":481,"state":"active+clean","last_fresh":"2024-09-18T08:22:31.526169+0000","last_change":"2024-09-18T08:21:50.172319+0000","last_active":"2024-09-18T08:22:31.526169+0000","last_peered":"2024-09-18T08:22:31.526169+0000","last_clean":"2024-09-18T08:22:31.526169+0000","last_became_active":"2024-09-18T08:21:45.587676+0000","last_became_peered":"2024-09-18T08:21:45.587676+0000","last_unstale":"2024-09-18T08:22:31.526169+0000","last_undegraded":"2024-09-18T08:22:31.526169+0000","last_fullsized":"2024-09-18T08:22:31.526169+0000","mapping_epoch":459,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":460,"parent":"0.0","parent_split_bits":0,"last_scrub":"271'20","last_scrub_stamp":"2024-09-18T08:21:50.172248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_clean_scrub_stamp":"2024-09-18T08:21:50.172248+0000","objects_scrubbed":5,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:22:53.166575+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":28,"num_read_kb":23,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":941,"reported_epoch":481,"state":"active+clean","last_fresh":"2024-09-18T08:22:31.527297+0000","last_change":"2024-09-18T08:21:56.876243+0000","last_active":"2024-09-18T08:22:31.527297+0000","last_peered":"2024-09-18T08:22:31.527297+0000","last_clean":"2024-09-18T08:22:31.527297+0000","last_became_active":"2024-09-18T08:21:45.596486+0000","last_became_peered":"2024-09-18T08:21:45.596486+0000","last_unstale":"2024-09-18T08:22:31.527297+0000","last_undegraded":"2024-09-18T08:22:31.527297+0000","last_fullsized":"2024-09-18T08:22:31.527297+0000","mapping_epoch":459,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":460,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:21:56.876196+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:46.860092+0000","last_clean_scrub_stamp":"2024-09-18T08:21:56.876196+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:23:08.409277+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.d","version":"402'14","reported_seq":1199,"reported_epoch":481,"state":"active+clean","last_fresh":"2024-09-18T08:22:38.166301+0000","last_change":"2024-09-18T08:21:52.348164+0000","last_active":"2024-09-18T08:22:38.166301+0000","last_peered":"2024-09-18T08:22:38.166301+0000","last_clean":"2024-09-18T08:22:38.166301+0000","last_became_active":"2024-09-18T08:18:27.653796+0000","last_became_peered":"2024-09-18T08:18:27.653796+0000","last_unstale":"2024-09-18T08:22:38.166301+0000","last_undegraded":"2024-09-18T08:22:38.166301+0000","last_fullsized":"2024-09-18T08:22:38.166301+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":0,"last_scrub":"402'14","last_scrub_stamp":"2024-09-18T08:21:52.348110+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:50.886830+0000","last_clean_scrub_stamp":"2024-09-18T08:21:52.348110+0000","objects_scrubbed":1,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:23:12.720795+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":69,"reported_epoch":481,"state":"active+clean","last_fresh":"2024-09-18T08:22:31.526496+0000","last_change":"2024-09-18T08:22:26.279695+0000","last_active":"2024-09-18T08:22:31.526496+0000","last_peered":"2024-09-18T08:22:31.526496+0000","last_clean":"2024-09-18T08:22:31.526496+0000","last_became_active":"2024-09-18T08:21:53.632705+0000","last_became_peered":"2024-09-18T08:21:53.632705+0000","last_unstale":"2024-09-18T08:22:31.526496+0000","last_undegraded":"2024-09-18T08:22:31.526496+0000","last_fullsized":"2024-09-18T08:22:31.526496+0000","mapping_epoch":463,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":464,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:22:26.279634+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_clean_scrub_stamp":"2024-09-18T08:22:26.279634+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:23:55.446246+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.b","version":"271'15","reported_seq":1013,"reported_epoch":481,"state":"active+clean","last_fresh":"2024-09-18T08:22:32.033442+0000","last_change":"2024-09-18T08:22:18.270317+0000","last_active":"2024-09-18T08:22:32.033442+0000","last_peered":"2024-09-18T08:22:32.033442+0000","last_clean":"2024-09-18T08:22:32.033442+0000","last_became_active":"2024-09-18T08:21:45.590532+0000","last_became_peered":"2024-09-18T08:21:45.590532+0000","last_unstale":"2024-09-18T08:22:32.033442+0000","last_undegraded":"2024-09-18T08:22:32.033442+0000","last_fullsized":"2024-09-18T08:22:32.033442+0000","mapping_epoch":459,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":460,"parent":"0.0","parent_split_bits":0,"last_scrub":"271'15","last_scrub_stamp":"2024-09-18T08:22:18.270247+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_clean_scrub_stamp":"2024-09-18T08:22:18.270247+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:23:25.584268+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":423,"reported_epoch":481,"state":"active+clean","last_fresh":"2024-09-18T08:22:31.527668+0000","last_change":"2024-09-18T08:21:52.002208+0000","last_active":"2024-09-18T08:22:31.527668+0000","last_peered":"2024-09-18T08:22:31.527668+0000","last_clean":"2024-09-18T08:22:31.527668+0000","last_became_active":"2024-09-18T08:18:17.892088+0000","last_became_peered":"2024-09-18T08:18:17.892088+0000","last_unstale":"2024-09-18T08:22:31.527668+0000","last_undegraded":"2024-09-18T08:22:31.527668+0000","last_fullsized":"2024-09-18T08:22:31.527668+0000","mapping_epoch":395,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":396,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:21:52.002161+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:46.860092+0000","last_clean_scrub_stamp":"2024-09-18T08:21:52.002161+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:23:08.476156+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,6],"acting":[3,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":54,"reported_epoch":481,"state":"active+clean","last_fresh":"2024-09-18T08:22:31.526072+0000","last_change":"2024-09-18T08:22:09.353065+0000","last_active":"2024-09-18T08:22:31.526072+0000","last_peered":"2024-09-18T08:22:31.526072+0000","last_clean":"2024-09-18T08:22:31.526072+0000","last_became_active":"2024-09-18T08:21:53.636990+0000","last_became_peered":"2024-09-18T08:21:53.636990+0000","last_unstale":"2024-09-18T08:22:31.526072+0000","last_undegraded":"2024-09-18T08:22:31.526072+0000","last_fullsized":"2024-09-18T08:22:31.526072+0000","mapping_epoch":463,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":464,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:22:09.353008+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:50.886830+0000","last_clean_scrub_stamp":"2024-09-18T08:22:09.353008+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:23:18.157111+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,3],"acting":[2,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":944,"reported_epoch":481,"state":"active+clean","last_fresh":"2024-09-18T08:22:32.032606+0000","last_change":"2024-09-18T08:21:53.628709+0000","last_active":"2024-09-18T08:22:32.032606+0000","last_peered":"2024-09-18T08:22:32.032606+0000","last_clean":"2024-09-18T08:22:32.032606+0000","last_became_active":"2024-09-18T08:21:53.628253+0000","last_became_peered":"2024-09-18T08:21:53.628253+0000","last_unstale":"2024-09-18T08:22:32.032606+0000","last_undegraded":"2024-09-18T08:22:32.032606+0000","last_fullsized":"2024-09-18T08:22:32.032606+0000","mapping_epoch":463,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":464,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:21:04.983916+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_clean_scrub_stamp":"2024-09-18T08:21:04.983916+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.a","version":"271'35","reported_seq":317,"reported_epoch":481,"state":"active+clean","last_fresh":"2024-09-18T08:22:32.034216+0000","last_change":"2024-09-18T08:22:16.269835+0000","last_active":"2024-09-18T08:22:32.034216+0000","last_peered":"2024-09-18T08:22:32.034216+0000","last_clean":"2024-09-18T08:22:32.034216+0000","last_became_active":"2024-09-18T08:22:12.719919+0000","last_became_peered":"2024-09-18T08:22:12.719919+0000","last_unstale":"2024-09-18T08:22:32.034216+0000","last_undegraded":"2024-09-18T08:22:32.034216+0000","last_fullsized":"2024-09-18T08:22:32.034216+0000","mapping_epoch":474,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":475,"parent":"0.0","parent_split_bits":0,"last_scrub":"271'35","last_scrub_stamp":"2024-09-18T08:22:16.269744+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_clean_scrub_stamp":"2024-09-18T08:22:16.269744+0000","objects_scrubbed":9,"log_size":35,"log_dups_size":0,"ondisk_log_size":35,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:23:26.153878+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":55,"num_read_kb":44,"num_write":36,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":159,"reported_epoch":481,"state":"active+clean","last_fresh":"2024-09-18T08:22:32.035095+0000","last_change":"2024-09-18T08:22:16.744088+0000","last_active":"2024-09-18T08:22:32.035095+0000","last_peered":"2024-09-18T08:22:32.035095+0000","last_clean":"2024-09-18T08:22:32.035095+0000","last_became_active":"2024-09-18T08:22:16.743524+0000","last_became_peered":"2024-09-18T08:22:16.743524+0000","last_unstale":"2024-09-18T08:22:32.035095+0000","last_undegraded":"2024-09-18T08:22:32.035095+0000","last_fullsized":"2024-09-18T08:22:32.035095+0000","mapping_epoch":477,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":478,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:22:14.171891+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:46.860092+0000","last_clean_scrub_stamp":"2024-09-18T08:22:14.171891+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:23:27.770634+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":46,"reported_epoch":481,"state":"active+clean","last_fresh":"2024-09-18T08:22:31.526048+0000","last_change":"2024-09-18T08:21:53.627144+0000","last_active":"2024-09-18T08:22:31.526048+0000","last_peered":"2024-09-18T08:22:31.526048+0000","last_clean":"2024-09-18T08:22:31.526048+0000","last_became_active":"2024-09-18T08:21:53.626819+0000","last_became_peered":"2024-09-18T08:21:53.626819+0000","last_unstale":"2024-09-18T08:22:31.526048+0000","last_undegraded":"2024-09-18T08:22:31.526048+0000","last_fullsized":"2024-09-18T08:22:31.526048+0000","mapping_epoch":463,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":464,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:21:50.974095+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:50.886830+0000","last_clean_scrub_stamp":"2024-09-18T08:21:50.974095+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:23:12.632609+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,5],"acting":[2,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":599,"reported_epoch":481,"state":"active+clean","last_fresh":"2024-09-18T08:22:32.032588+0000","last_change":"2024-09-18T08:22:16.739525+0000","last_active":"2024-09-18T08:22:32.032588+0000","last_peered":"2024-09-18T08:22:32.032588+0000","last_clean":"2024-09-18T08:22:32.032588+0000","last_became_active":"2024-09-18T08:22:16.739179+0000","last_became_peered":"2024-09-18T08:22:16.739179+0000","last_unstale":"2024-09-18T08:22:32.032588+0000","last_undegraded":"2024-09-18T08:22:32.032588+0000","last_fullsized":"2024-09-18T08:22:32.032588+0000","mapping_epoch":477,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":478,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:22:08.369259+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_clean_scrub_stamp":"2024-09-18T08:22:08.369259+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:23:28.403718+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.9","version":"271'18","reported_seq":74,"reported_epoch":481,"state":"active+clean","last_fresh":"2024-09-18T08:22:36.181731+0000","last_change":"2024-09-18T08:22:36.181731+0000","last_active":"2024-09-18T08:22:36.181731+0000","last_peered":"2024-09-18T08:22:36.181731+0000","last_clean":"2024-09-18T08:22:36.181731+0000","last_became_active":"2024-09-18T08:21:45.586536+0000","last_became_peered":"2024-09-18T08:21:45.586536+0000","last_unstale":"2024-09-18T08:22:36.181731+0000","last_undegraded":"2024-09-18T08:22:36.181731+0000","last_fullsized":"2024-09-18T08:22:36.181731+0000","mapping_epoch":459,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":460,"parent":"0.0","parent_split_bits":0,"last_scrub":"271'18","last_scrub_stamp":"2024-09-18T08:22:36.181649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_clean_scrub_stamp":"2024-09-18T08:22:36.181649+0000","objects_scrubbed":5,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:23:42.826616+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":33,"num_read_kb":24,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":1100,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":202,"reported_epoch":481,"state":"active+clean","last_fresh":"2024-09-18T08:22:40.786015+0000","last_change":"2024-09-18T08:22:40.786015+0000","last_active":"2024-09-18T08:22:40.786015+0000","last_peered":"2024-09-18T08:22:40.786015+0000","last_clean":"2024-09-18T08:22:40.786015+0000","last_became_active":"2024-09-18T08:21:45.596881+0000","last_became_peered":"2024-09-18T08:21:45.596881+0000","last_unstale":"2024-09-18T08:22:40.786015+0000","last_undegraded":"2024-09-18T08:22:40.786015+0000","last_fullsized":"2024-09-18T08:22:40.786015+0000","mapping_epoch":459,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":460,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:22:40.785955+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:46.860092+0000","last_clean_scrub_stamp":"2024-09-18T08:22:40.785955+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:23:51.137237+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":924,"reported_epoch":481,"state":"active+clean","last_fresh":"2024-09-18T08:22:31.525700+0000","last_change":"2024-09-18T08:22:14.238417+0000","last_active":"2024-09-18T08:22:31.525700+0000","last_peered":"2024-09-18T08:22:31.525700+0000","last_clean":"2024-09-18T08:22:31.525700+0000","last_became_active":"2024-09-18T08:14:39.399301+0000","last_became_peered":"2024-09-18T08:14:39.399301+0000","last_unstale":"2024-09-18T08:22:31.525700+0000","last_undegraded":"2024-09-18T08:22:31.525700+0000","last_fullsized":"2024-09-18T08:22:31.525700+0000","mapping_epoch":307,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":308,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:22:14.238366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:50.886830+0000","last_clean_scrub_stamp":"2024-09-18T08:22:14.238366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:23:26.494594+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":918,"reported_epoch":481,"state":"active+clean","last_fresh":"2024-09-18T08:22:31.526723+0000","last_change":"2024-09-18T08:21:55.879284+0000","last_active":"2024-09-18T08:22:31.526723+0000","last_peered":"2024-09-18T08:22:31.526723+0000","last_clean":"2024-09-18T08:22:31.526723+0000","last_became_active":"2024-09-18T08:18:27.664412+0000","last_became_peered":"2024-09-18T08:18:27.664412+0000","last_unstale":"2024-09-18T08:22:31.526723+0000","last_undegraded":"2024-09-18T08:22:31.526723+0000","last_fullsized":"2024-09-18T08:22:31.526723+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:21:55.879233+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_clean_scrub_stamp":"2024-09-18T08:21:55.879233+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:23:25.821864+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.8","version":"271'27","reported_seq":381,"reported_epoch":481,"state":"active+clean","last_fresh":"2024-09-18T08:22:32.034182+0000","last_change":"2024-09-18T08:22:29.124790+0000","last_active":"2024-09-18T08:22:32.034182+0000","last_peered":"2024-09-18T08:22:32.034182+0000","last_clean":"2024-09-18T08:22:32.034182+0000","last_became_active":"2024-09-18T08:18:27.643053+0000","last_became_peered":"2024-09-18T08:18:27.643053+0000","last_unstale":"2024-09-18T08:22:32.034182+0000","last_undegraded":"2024-09-18T08:22:32.034182+0000","last_fullsized":"2024-09-18T08:22:32.034182+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":0,"last_scrub":"271'27","last_scrub_stamp":"2024-09-18T08:22:29.124715+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_clean_scrub_stamp":"2024-09-18T08:22:29.124715+0000","objects_scrubbed":7,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:23:48.271896+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":44,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":35,"num_bytes_recovered":550,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.f","version":"62'2","reported_seq":198,"reported_epoch":481,"state":"active+clean","last_fresh":"2024-09-18T08:22:32.034894+0000","last_change":"2024-09-18T08:21:47.774483+0000","last_active":"2024-09-18T08:22:32.034894+0000","last_peered":"2024-09-18T08:22:32.034894+0000","last_clean":"2024-09-18T08:22:32.034894+0000","last_became_active":"2024-09-18T08:21:45.598214+0000","last_became_peered":"2024-09-18T08:21:45.598214+0000","last_unstale":"2024-09-18T08:22:32.034894+0000","last_undegraded":"2024-09-18T08:22:32.034894+0000","last_fullsized":"2024-09-18T08:22:32.034894+0000","mapping_epoch":459,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":460,"parent":"0.0","parent_split_bits":0,"last_scrub":"62'2","last_scrub_stamp":"2024-09-18T08:21:47.774423+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:46.860092+0000","last_clean_scrub_stamp":"2024-09-18T08:21:47.774423+0000","objects_scrubbed":2,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:23:08.158907+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":276,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.9","version":"460'19","reported_seq":254,"reported_epoch":481,"state":"active+clean","last_fresh":"2024-09-18T08:22:38.166555+0000","last_change":"2024-09-18T08:21:54.943666+0000","last_active":"2024-09-18T08:22:38.166555+0000","last_peered":"2024-09-18T08:22:38.166555+0000","last_clean":"2024-09-18T08:22:38.166555+0000","last_became_active":"2024-09-18T08:21:45.597492+0000","last_became_peered":"2024-09-18T08:21:45.597492+0000","last_unstale":"2024-09-18T08:22:38.166555+0000","last_undegraded":"2024-09-18T08:22:38.166555+0000","last_fullsized":"2024-09-18T08:22:38.166555+0000","mapping_epoch":459,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":460,"parent":"0.0","parent_split_bits":0,"last_scrub":"460'19","last_scrub_stamp":"2024-09-18T08:21:54.943603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:50.886830+0000","last_clean_scrub_stamp":"2024-09-18T08:21:54.943603+0000","objects_scrubbed":1,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:23:19.508016+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":319,"reported_epoch":481,"state":"active+clean","last_fresh":"2024-09-18T08:22:32.033626+0000","last_change":"2024-09-18T08:22:02.150670+0000","last_active":"2024-09-18T08:22:32.033626+0000","last_peered":"2024-09-18T08:22:32.033626+0000","last_clean":"2024-09-18T08:22:32.033626+0000","last_became_active":"2024-09-18T08:15:41.449889+0000","last_became_peered":"2024-09-18T08:15:41.449889+0000","last_unstale":"2024-09-18T08:22:32.033626+0000","last_undegraded":"2024-09-18T08:22:32.033626+0000","last_fullsized":"2024-09-18T08:22:32.033626+0000","mapping_epoch":336,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":337,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:22:02.150612+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_clean_scrub_stamp":"2024-09-18T08:22:02.150612+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:23:22.958169+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":553,"reported_epoch":481,"state":"active+clean","last_fresh":"2024-09-18T08:22:32.032995+0000","last_change":"2024-09-18T08:22:18.300016+0000","last_active":"2024-09-18T08:22:32.032995+0000","last_peered":"2024-09-18T08:22:32.032995+0000","last_clean":"2024-09-18T08:22:32.032995+0000","last_became_active":"2024-09-18T08:22:12.718112+0000","last_became_peered":"2024-09-18T08:22:12.718112+0000","last_unstale":"2024-09-18T08:22:32.032995+0000","last_undegraded":"2024-09-18T08:22:32.032995+0000","last_fullsized":"2024-09-18T08:22:32.032995+0000","mapping_epoch":474,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":475,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:22:18.299931+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:46.860092+0000","last_clean_scrub_stamp":"2024-09-18T08:22:18.299931+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:23:40.150113+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"271'12","reported_seq":976,"reported_epoch":481,"state":"active+clean","last_fresh":"2024-09-18T08:22:43.977609+0000","last_change":"2024-09-18T08:22:43.977609+0000","last_active":"2024-09-18T08:22:43.977609+0000","last_peered":"2024-09-18T08:22:43.977609+0000","last_clean":"2024-09-18T08:22:43.977609+0000","last_became_active":"2024-09-18T08:18:27.663338+0000","last_became_peered":"2024-09-18T08:18:27.663338+0000","last_unstale":"2024-09-18T08:22:43.977609+0000","last_undegraded":"2024-09-18T08:22:43.977609+0000","last_fullsized":"2024-09-18T08:22:43.977609+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":0,"last_scrub":"271'12","last_scrub_stamp":"2024-09-18T08:22:43.977548+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_clean_scrub_stamp":"2024-09-18T08:22:43.977548+0000","objects_scrubbed":3,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:23:44.580615+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":18,"num_read_kb":15,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":318,"reported_epoch":481,"state":"active+clean","last_fresh":"2024-09-18T08:22:32.034975+0000","last_change":"2024-09-18T08:22:01.193929+0000","last_active":"2024-09-18T08:22:32.034975+0000","last_peered":"2024-09-18T08:22:32.034975+0000","last_clean":"2024-09-18T08:22:32.034975+0000","last_became_active":"2024-09-18T08:15:41.445332+0000","last_became_peered":"2024-09-18T08:15:41.445332+0000","last_unstale":"2024-09-18T08:22:32.034975+0000","last_undegraded":"2024-09-18T08:22:32.034975+0000","last_fullsized":"2024-09-18T08:22:32.034975+0000","mapping_epoch":336,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":337,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:22:01.193848+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:50.886830+0000","last_clean_scrub_stamp":"2024-09-18T08:22:01.193848+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:23:19.389130+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,1],"acting":[5,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":202,"reported_epoch":481,"state":"active+clean","last_fresh":"2024-09-18T08:22:35.750986+0000","last_change":"2024-09-18T08:22:35.750986+0000","last_active":"2024-09-18T08:22:35.750986+0000","last_peered":"2024-09-18T08:22:35.750986+0000","last_clean":"2024-09-18T08:22:35.750986+0000","last_became_active":"2024-09-18T08:21:45.597863+0000","last_became_peered":"2024-09-18T08:21:45.597863+0000","last_unstale":"2024-09-18T08:22:35.750986+0000","last_undegraded":"2024-09-18T08:22:35.750986+0000","last_fullsized":"2024-09-18T08:22:35.750986+0000","mapping_epoch":459,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":460,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:22:35.750927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_clean_scrub_stamp":"2024-09-18T08:22:35.750927+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:23:50.865747+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"271'15","reported_seq":959,"reported_epoch":481,"state":"active+clean","last_fresh":"2024-09-18T08:22:32.033943+0000","last_change":"2024-09-18T08:21:22.789859+0000","last_active":"2024-09-18T08:22:32.033943+0000","last_peered":"2024-09-18T08:22:32.033943+0000","last_clean":"2024-09-18T08:22:32.033943+0000","last_became_active":"2024-09-18T08:18:27.658637+0000","last_became_peered":"2024-09-18T08:18:27.658637+0000","last_unstale":"2024-09-18T08:22:32.033943+0000","last_undegraded":"2024-09-18T08:22:32.033943+0000","last_fullsized":"2024-09-18T08:22:32.033943+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":0,"last_scrub":"271'15","last_scrub_stamp":"2024-09-18T08:21:22.789796+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_clean_scrub_stamp":"2024-09-18T08:21:22.789796+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:22:48.359769+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":400,"reported_epoch":481,"state":"active+clean","last_fresh":"2024-09-18T08:22:32.033212+0000","last_change":"2024-09-18T08:22:24.643149+0000","last_active":"2024-09-18T08:22:32.033212+0000","last_peered":"2024-09-18T08:22:32.033212+0000","last_clean":"2024-09-18T08:22:32.033212+0000","last_became_active":"2024-09-18T08:22:16.741243+0000","last_became_peered":"2024-09-18T08:22:16.741243+0000","last_unstale":"2024-09-18T08:22:32.033212+0000","last_undegraded":"2024-09-18T08:22:32.033212+0000","last_fullsized":"2024-09-18T08:22:32.033212+0000","mapping_epoch":477,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":478,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:22:24.643087+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:46.860092+0000","last_clean_scrub_stamp":"2024-09-18T08:22:24.643087+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:23:38.442640+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":913,"reported_epoch":481,"state":"active+clean","last_fresh":"2024-09-18T08:22:31.527453+0000","last_change":"2024-09-18T08:22:06.937365+0000","last_active":"2024-09-18T08:22:31.527453+0000","last_peered":"2024-09-18T08:22:31.527453+0000","last_clean":"2024-09-18T08:22:31.527453+0000","last_became_active":"2024-09-18T08:18:27.660093+0000","last_became_peered":"2024-09-18T08:18:27.660093+0000","last_unstale":"2024-09-18T08:22:31.527453+0000","last_undegraded":"2024-09-18T08:22:31.527453+0000","last_fullsized":"2024-09-18T08:22:31.527453+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:22:06.937297+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:50.886830+0000","last_clean_scrub_stamp":"2024-09-18T08:22:06.937297+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:23:17.831650+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":54,"reported_epoch":481,"state":"active+clean","last_fresh":"2024-09-18T08:22:31.526348+0000","last_change":"2024-09-18T08:22:10.354560+0000","last_active":"2024-09-18T08:22:31.526348+0000","last_peered":"2024-09-18T08:22:31.526348+0000","last_clean":"2024-09-18T08:22:31.526348+0000","last_became_active":"2024-09-18T08:21:53.635391+0000","last_became_peered":"2024-09-18T08:21:53.635391+0000","last_unstale":"2024-09-18T08:22:31.526348+0000","last_undegraded":"2024-09-18T08:22:31.526348+0000","last_fullsized":"2024-09-18T08:22:31.526348+0000","mapping_epoch":463,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":464,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:22:10.354510+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_clean_scrub_stamp":"2024-09-18T08:22:10.354510+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:23:26.092303+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.15","version":"271'15","reported_seq":345,"reported_epoch":481,"state":"active+clean","last_fresh":"2024-09-18T08:22:32.035272+0000","last_change":"2024-09-18T08:21:44.334235+0000","last_active":"2024-09-18T08:22:32.035272+0000","last_peered":"2024-09-18T08:22:32.035272+0000","last_clean":"2024-09-18T08:22:32.035272+0000","last_became_active":"2024-09-18T08:18:27.646211+0000","last_became_peered":"2024-09-18T08:18:27.646211+0000","last_unstale":"2024-09-18T08:22:32.035272+0000","last_undegraded":"2024-09-18T08:22:32.035272+0000","last_fullsized":"2024-09-18T08:22:32.035272+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":0,"last_scrub":"271'15","last_scrub_stamp":"2024-09-18T08:21:44.334148+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_clean_scrub_stamp":"2024-09-18T08:21:44.334148+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:22:56.615289+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":25,"num_read_kb":19,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":931,"reported_epoch":481,"state":"active+clean","last_fresh":"2024-09-18T08:22:32.033137+0000","last_change":"2024-09-18T08:21:15.826474+0000","last_active":"2024-09-18T08:22:32.033137+0000","last_peered":"2024-09-18T08:22:32.033137+0000","last_clean":"2024-09-18T08:22:32.033137+0000","last_became_active":"2024-09-18T08:18:27.655758+0000","last_became_peered":"2024-09-18T08:18:27.655758+0000","last_unstale":"2024-09-18T08:22:32.033137+0000","last_undegraded":"2024-09-18T08:22:32.033137+0000","last_fullsized":"2024-09-18T08:22:32.033137+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:21:15.826408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:46.860092+0000","last_clean_scrub_stamp":"2024-09-18T08:21:15.826408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:22:40.998233+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.14","version":"464'19","reported_seq":1230,"reported_epoch":481,"state":"active+clean","last_fresh":"2024-09-18T08:22:38.166341+0000","last_change":"2024-09-18T08:21:53.632185+0000","last_active":"2024-09-18T08:22:38.166341+0000","last_peered":"2024-09-18T08:22:38.166341+0000","last_clean":"2024-09-18T08:22:38.166341+0000","last_became_active":"2024-09-18T08:21:53.623477+0000","last_became_peered":"2024-09-18T08:21:53.623477+0000","last_unstale":"2024-09-18T08:22:38.166341+0000","last_undegraded":"2024-09-18T08:22:38.166341+0000","last_fullsized":"2024-09-18T08:22:38.166341+0000","mapping_epoch":463,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":464,"parent":"0.0","parent_split_bits":0,"last_scrub":"415'18","last_scrub_stamp":"2024-09-18T08:21:18.822162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:50.886830+0000","last_clean_scrub_stamp":"2024-09-18T08:21:18.822162+0000","objects_scrubbed":1,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:22:44.013799+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":62,"reported_epoch":481,"state":"active+clean","last_fresh":"2024-09-18T08:22:31.526884+0000","last_change":"2024-09-18T08:22:11.281835+0000","last_active":"2024-09-18T08:22:31.526884+0000","last_peered":"2024-09-18T08:22:31.526884+0000","last_clean":"2024-09-18T08:22:31.526884+0000","last_became_active":"2024-09-18T08:21:45.581532+0000","last_became_peered":"2024-09-18T08:21:45.581532+0000","last_unstale":"2024-09-18T08:22:31.526884+0000","last_undegraded":"2024-09-18T08:22:31.526884+0000","last_fullsized":"2024-09-18T08:22:31.526884+0000","mapping_epoch":459,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":460,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:22:11.281763+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_clean_scrub_stamp":"2024-09-18T08:22:11.281763+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:23:13.682894+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.14","version":"271'20","reported_seq":994,"reported_epoch":481,"state":"active+clean","last_fresh":"2024-09-18T08:22:37.014620+0000","last_change":"2024-09-18T08:22:37.014620+0000","last_active":"2024-09-18T08:22:37.014620+0000","last_peered":"2024-09-18T08:22:37.014620+0000","last_clean":"2024-09-18T08:22:37.014620+0000","last_became_active":"2024-09-18T08:18:27.663877+0000","last_became_peered":"2024-09-18T08:18:27.663877+0000","last_unstale":"2024-09-18T08:22:37.014620+0000","last_undegraded":"2024-09-18T08:22:37.014620+0000","last_fullsized":"2024-09-18T08:22:37.014620+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":0,"last_scrub":"271'20","last_scrub_stamp":"2024-09-18T08:22:37.014554+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_clean_scrub_stamp":"2024-09-18T08:22:37.014554+0000","objects_scrubbed":5,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:23:43.955123+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":25,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,7],"acting":[3,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":200,"reported_epoch":481,"state":"active+clean","last_fresh":"2024-09-18T08:22:32.033334+0000","last_change":"2024-09-18T08:21:53.640946+0000","last_active":"2024-09-18T08:22:32.033334+0000","last_peered":"2024-09-18T08:22:32.033334+0000","last_clean":"2024-09-18T08:22:32.033334+0000","last_became_active":"2024-09-18T08:21:53.640317+0000","last_became_peered":"2024-09-18T08:21:53.640317+0000","last_unstale":"2024-09-18T08:22:32.033334+0000","last_undegraded":"2024-09-18T08:22:32.033334+0000","last_fullsized":"2024-09-18T08:22:32.033334+0000","mapping_epoch":463,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":464,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:21:45.842324+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:46.860092+0000","last_clean_scrub_stamp":"2024-09-18T08:21:45.842324+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:22:46.641994+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.15","version":"338'12","reported_seq":407,"reported_epoch":481,"state":"active+clean","last_fresh":"2024-09-18T08:22:38.166523+0000","last_change":"2024-09-18T08:22:31.188414+0000","last_active":"2024-09-18T08:22:38.166523+0000","last_peered":"2024-09-18T08:22:38.166523+0000","last_clean":"2024-09-18T08:22:38.166523+0000","last_became_active":"2024-09-18T08:15:41.455243+0000","last_became_peered":"2024-09-18T08:15:41.455243+0000","last_unstale":"2024-09-18T08:22:38.166523+0000","last_undegraded":"2024-09-18T08:22:38.166523+0000","last_fullsized":"2024-09-18T08:22:38.166523+0000","mapping_epoch":336,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":337,"parent":"0.0","parent_split_bits":0,"last_scrub":"338'12","last_scrub_stamp":"2024-09-18T08:22:31.188340+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:50.886830+0000","last_clean_scrub_stamp":"2024-09-18T08:22:31.188340+0000","objects_scrubbed":1,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:23:47.409314+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":914,"reported_epoch":481,"state":"active+clean","last_fresh":"2024-09-18T08:22:32.033736+0000","last_change":"2024-09-18T08:22:25.267732+0000","last_active":"2024-09-18T08:22:32.033736+0000","last_peered":"2024-09-18T08:22:32.033736+0000","last_clean":"2024-09-18T08:22:32.033736+0000","last_became_active":"2024-09-18T08:18:27.658009+0000","last_became_peered":"2024-09-18T08:18:27.658009+0000","last_unstale":"2024-09-18T08:22:32.033736+0000","last_undegraded":"2024-09-18T08:22:32.033736+0000","last_fullsized":"2024-09-18T08:22:32.033736+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:22:25.267686+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_clean_scrub_stamp":"2024-09-18T08:22:25.267686+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:23:47.406228+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.13","version":"271'19","reported_seq":74,"reported_epoch":481,"state":"active+clean","last_fresh":"2024-09-18T08:22:31.526482+0000","last_change":"2024-09-18T08:21:58.234819+0000","last_active":"2024-09-18T08:22:31.526482+0000","last_peered":"2024-09-18T08:22:31.526482+0000","last_clean":"2024-09-18T08:22:31.526482+0000","last_became_active":"2024-09-18T08:21:45.589059+0000","last_became_peered":"2024-09-18T08:21:45.589059+0000","last_unstale":"2024-09-18T08:22:31.526482+0000","last_undegraded":"2024-09-18T08:22:31.526482+0000","last_fullsized":"2024-09-18T08:22:31.526482+0000","mapping_epoch":459,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":460,"parent":"0.0","parent_split_bits":0,"last_scrub":"271'19","last_scrub_stamp":"2024-09-18T08:21:58.234746+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_clean_scrub_stamp":"2024-09-18T08:21:58.234746+0000","objects_scrubbed":5,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:23:28.171925+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":31,"num_read_kb":24,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":62,"reported_epoch":481,"state":"active+clean","last_fresh":"2024-09-18T08:22:31.526075+0000","last_change":"2024-09-18T08:22:16.324599+0000","last_active":"2024-09-18T08:22:31.526075+0000","last_peered":"2024-09-18T08:22:31.526075+0000","last_clean":"2024-09-18T08:22:31.526075+0000","last_became_active":"2024-09-18T08:21:45.592061+0000","last_became_peered":"2024-09-18T08:21:45.592061+0000","last_unstale":"2024-09-18T08:22:31.526075+0000","last_undegraded":"2024-09-18T08:22:31.526075+0000","last_fullsized":"2024-09-18T08:22:31.526075+0000","mapping_epoch":459,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":460,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:22:16.324534+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:46.860092+0000","last_clean_scrub_stamp":"2024-09-18T08:22:16.324534+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:23:16.882936+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":502,"reported_epoch":481,"state":"active+clean","last_fresh":"2024-09-18T08:22:32.034134+0000","last_change":"2024-09-18T08:22:25.363009+0000","last_active":"2024-09-18T08:22:32.034134+0000","last_peered":"2024-09-18T08:22:32.034134+0000","last_clean":"2024-09-18T08:22:32.034134+0000","last_became_active":"2024-09-18T08:22:12.718201+0000","last_became_peered":"2024-09-18T08:22:12.718201+0000","last_unstale":"2024-09-18T08:22:32.034134+0000","last_undegraded":"2024-09-18T08:22:32.034134+0000","last_fullsized":"2024-09-18T08:22:32.034134+0000","mapping_epoch":474,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":475,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:22:25.362951+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:50.886830+0000","last_clean_scrub_stamp":"2024-09-18T08:22:25.362951+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:23:33.635413+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,3],"acting":[6,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":917,"reported_epoch":481,"state":"active+clean","last_fresh":"2024-09-18T08:22:31.525754+0000","last_change":"2024-09-18T08:21:04.922924+0000","last_active":"2024-09-18T08:22:31.525754+0000","last_peered":"2024-09-18T08:22:31.525754+0000","last_clean":"2024-09-18T08:22:31.525754+0000","last_became_active":"2024-09-18T08:15:41.464961+0000","last_became_peered":"2024-09-18T08:15:41.464961+0000","last_unstale":"2024-09-18T08:22:31.525754+0000","last_undegraded":"2024-09-18T08:22:31.525754+0000","last_fullsized":"2024-09-18T08:22:31.525754+0000","mapping_epoch":336,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":337,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:21:04.922071+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_clean_scrub_stamp":"2024-09-18T08:21:04.922071+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.12","version":"271'15","reported_seq":814,"reported_epoch":481,"state":"active+clean","last_fresh":"2024-09-18T08:22:32.033081+0000","last_change":"2024-09-18T08:21:53.649359+0000","last_active":"2024-09-18T08:22:32.033081+0000","last_peered":"2024-09-18T08:22:32.033081+0000","last_clean":"2024-09-18T08:22:32.033081+0000","last_became_active":"2024-09-18T08:21:53.634545+0000","last_became_peered":"2024-09-18T08:21:53.634545+0000","last_unstale":"2024-09-18T08:22:32.033081+0000","last_undegraded":"2024-09-18T08:22:32.033081+0000","last_fullsized":"2024-09-18T08:22:32.033081+0000","mapping_epoch":463,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":464,"parent":"0.0","parent_split_bits":0,"last_scrub":"271'15","last_scrub_stamp":"2024-09-18T08:21:42.696135+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_clean_scrub_stamp":"2024-09-18T08:21:42.696135+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:22:54.443306+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":25,"num_read_kb":19,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":684,"reported_epoch":481,"state":"active+clean","last_fresh":"2024-09-18T08:22:32.034485+0000","last_change":"2024-09-18T08:22:16.737094+0000","last_active":"2024-09-18T08:22:32.034485+0000","last_peered":"2024-09-18T08:22:32.034485+0000","last_clean":"2024-09-18T08:22:32.034485+0000","last_became_active":"2024-09-18T08:22:16.736379+0000","last_became_peered":"2024-09-18T08:22:16.736379+0000","last_unstale":"2024-09-18T08:22:32.034485+0000","last_undegraded":"2024-09-18T08:22:32.034485+0000","last_fullsized":"2024-09-18T08:22:32.034485+0000","mapping_epoch":477,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":478,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:21:41.948219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:46.860092+0000","last_clean_scrub_stamp":"2024-09-18T08:21:41.948219+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:23:03.403331+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":901,"reported_epoch":481,"state":"active+clean","last_fresh":"2024-09-18T08:22:31.527051+0000","last_change":"2024-09-18T08:21:54.920592+0000","last_active":"2024-09-18T08:22:31.527051+0000","last_peered":"2024-09-18T08:22:31.527051+0000","last_clean":"2024-09-18T08:22:31.527051+0000","last_became_active":"2024-09-18T08:14:39.402406+0000","last_became_peered":"2024-09-18T08:14:39.402406+0000","last_unstale":"2024-09-18T08:22:31.527051+0000","last_undegraded":"2024-09-18T08:22:31.527051+0000","last_fullsized":"2024-09-18T08:22:31.527051+0000","mapping_epoch":307,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":308,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:21:54.920545+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:50.886830+0000","last_clean_scrub_stamp":"2024-09-18T08:21:54.920545+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:23:06.408191+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.10","version":"69'1","reported_seq":914,"reported_epoch":481,"state":"active+clean","last_fresh":"2024-09-18T08:22:32.031957+0000","last_change":"2024-09-18T08:21:39.819943+0000","last_active":"2024-09-18T08:22:32.031957+0000","last_peered":"2024-09-18T08:22:32.031957+0000","last_clean":"2024-09-18T08:22:32.031957+0000","last_became_active":"2024-09-18T08:15:41.444085+0000","last_became_peered":"2024-09-18T08:15:41.444085+0000","last_unstale":"2024-09-18T08:22:32.031957+0000","last_undegraded":"2024-09-18T08:22:32.031957+0000","last_fullsized":"2024-09-18T08:22:32.031957+0000","mapping_epoch":336,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":337,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'1","last_scrub_stamp":"2024-09-18T08:21:39.819892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_clean_scrub_stamp":"2024-09-18T08:21:39.819892+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:22:43.568297+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":26,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.11","version":"271'19","reported_seq":1063,"reported_epoch":481,"state":"active+clean","last_fresh":"2024-09-18T08:22:31.525593+0000","last_change":"2024-09-18T08:22:08.993996+0000","last_active":"2024-09-18T08:22:31.525593+0000","last_peered":"2024-09-18T08:22:31.525593+0000","last_clean":"2024-09-18T08:22:31.525593+0000","last_became_active":"2024-09-18T08:18:27.659293+0000","last_became_peered":"2024-09-18T08:18:27.659293+0000","last_unstale":"2024-09-18T08:22:31.525593+0000","last_undegraded":"2024-09-18T08:22:31.525593+0000","last_fullsized":"2024-09-18T08:22:31.525593+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":0,"last_scrub":"271'19","last_scrub_stamp":"2024-09-18T08:22:08.993941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_clean_scrub_stamp":"2024-09-18T08:22:08.993941+0000","objects_scrubbed":5,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:23:22.471820+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":32,"num_read_kb":25,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":30,"num_bytes_recovered":660,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":333,"reported_epoch":481,"state":"active+clean","last_fresh":"2024-09-18T08:22:41.065850+0000","last_change":"2024-09-18T08:22:41.065850+0000","last_active":"2024-09-18T08:22:41.065850+0000","last_peered":"2024-09-18T08:22:41.065850+0000","last_clean":"2024-09-18T08:22:41.065850+0000","last_became_active":"2024-09-18T08:18:27.643957+0000","last_became_peered":"2024-09-18T08:18:27.643957+0000","last_unstale":"2024-09-18T08:22:41.065850+0000","last_undegraded":"2024-09-18T08:22:41.065850+0000","last_fullsized":"2024-09-18T08:22:41.065850+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:22:41.065782+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:46.860092+0000","last_clean_scrub_stamp":"2024-09-18T08:22:41.065782+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:23:44.999991+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":194,"reported_epoch":481,"state":"active+clean","last_fresh":"2024-09-18T08:22:32.033138+0000","last_change":"2024-09-18T08:22:02.901165+0000","last_active":"2024-09-18T08:22:32.033138+0000","last_peered":"2024-09-18T08:22:32.033138+0000","last_clean":"2024-09-18T08:22:32.033138+0000","last_became_active":"2024-09-18T08:21:45.596715+0000","last_became_peered":"2024-09-18T08:21:45.596715+0000","last_unstale":"2024-09-18T08:22:32.033138+0000","last_undegraded":"2024-09-18T08:22:32.033138+0000","last_fullsized":"2024-09-18T08:22:32.033138+0000","mapping_epoch":459,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":460,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:22:02.901100+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:50.886830+0000","last_clean_scrub_stamp":"2024-09-18T08:22:02.901100+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:23:05.993894+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":900,"reported_epoch":481,"state":"active+clean","last_fresh":"2024-09-18T08:22:31.527203+0000","last_change":"2024-09-18T08:21:40.928000+0000","last_active":"2024-09-18T08:22:31.527203+0000","last_peered":"2024-09-18T08:22:31.527203+0000","last_clean":"2024-09-18T08:22:31.527203+0000","last_became_active":"2024-09-18T08:06:24.128582+0000","last_became_peered":"2024-09-18T08:06:24.128582+0000","last_unstale":"2024-09-18T08:22:31.527203+0000","last_undegraded":"2024-09-18T08:22:31.527203+0000","last_fullsized":"2024-09-18T08:22:31.527203+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:21:40.927953+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_clean_scrub_stamp":"2024-09-18T08:21:40.927953+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:22:48.942522+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.10","version":"271'8","reported_seq":958,"reported_epoch":481,"state":"active+clean","last_fresh":"2024-09-18T08:22:39.985099+0000","last_change":"2024-09-18T08:22:39.985099+0000","last_active":"2024-09-18T08:22:39.985099+0000","last_peered":"2024-09-18T08:22:39.985099+0000","last_clean":"2024-09-18T08:22:39.985099+0000","last_became_active":"2024-09-18T08:14:39.404257+0000","last_became_peered":"2024-09-18T08:14:39.404257+0000","last_unstale":"2024-09-18T08:22:39.985099+0000","last_undegraded":"2024-09-18T08:22:39.985099+0000","last_fullsized":"2024-09-18T08:22:39.985099+0000","mapping_epoch":307,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":308,"parent":"0.0","parent_split_bits":0,"last_scrub":"271'8","last_scrub_stamp":"2024-09-18T08:22:39.985045+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_clean_scrub_stamp":"2024-09-18T08:22:39.985045+0000","objects_scrubbed":2,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:23:51.129896+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":12,"num_read_kb":10,"num_write":8,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":926,"reported_epoch":481,"state":"active+clean","last_fresh":"2024-09-18T08:22:32.033803+0000","last_change":"2024-09-18T08:21:51.973411+0000","last_active":"2024-09-18T08:22:32.033803+0000","last_peered":"2024-09-18T08:22:32.033803+0000","last_clean":"2024-09-18T08:22:32.033803+0000","last_became_active":"2024-09-18T08:15:41.466964+0000","last_became_peered":"2024-09-18T08:15:41.466964+0000","last_unstale":"2024-09-18T08:22:32.033803+0000","last_undegraded":"2024-09-18T08:22:32.033803+0000","last_fullsized":"2024-09-18T08:22:32.033803+0000","mapping_epoch":336,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":337,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:21:51.973328+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:46.860092+0000","last_clean_scrub_stamp":"2024-09-18T08:21:51.973328+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:23:17.492020+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":398,"reported_epoch":481,"state":"active+clean","last_fresh":"2024-09-18T08:22:32.033746+0000","last_change":"2024-09-18T08:22:12.718474+0000","last_active":"2024-09-18T08:22:32.033746+0000","last_peered":"2024-09-18T08:22:32.033746+0000","last_clean":"2024-09-18T08:22:32.033746+0000","last_became_active":"2024-09-18T08:22:12.718017+0000","last_became_peered":"2024-09-18T08:22:12.718017+0000","last_unstale":"2024-09-18T08:22:32.033746+0000","last_undegraded":"2024-09-18T08:22:32.033746+0000","last_fullsized":"2024-09-18T08:22:32.033746+0000","mapping_epoch":474,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":475,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:22:09.245323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:50.886830+0000","last_clean_scrub_stamp":"2024-09-18T08:22:09.245323+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:23:11.889868+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":558,"reported_epoch":481,"state":"active+clean","last_fresh":"2024-09-18T08:22:32.032736+0000","last_change":"2024-09-18T08:22:16.739974+0000","last_active":"2024-09-18T08:22:32.032736+0000","last_peered":"2024-09-18T08:22:32.032736+0000","last_clean":"2024-09-18T08:22:32.032736+0000","last_became_active":"2024-09-18T08:22:16.739643+0000","last_became_peered":"2024-09-18T08:22:16.739643+0000","last_unstale":"2024-09-18T08:22:32.032736+0000","last_undegraded":"2024-09-18T08:22:32.032736+0000","last_fullsized":"2024-09-18T08:22:32.032736+0000","mapping_epoch":477,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":478,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:22:11.318277+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_clean_scrub_stamp":"2024-09-18T08:22:11.318277+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:23:36.554739+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,4],"acting":[7,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":771,"reported_epoch":481,"state":"active+clean","last_fresh":"2024-09-18T08:22:32.032813+0000","last_change":"2024-09-18T08:22:10.785838+0000","last_active":"2024-09-18T08:22:32.032813+0000","last_peered":"2024-09-18T08:22:32.032813+0000","last_clean":"2024-09-18T08:22:32.032813+0000","last_became_active":"2024-09-18T08:21:45.587782+0000","last_became_peered":"2024-09-18T08:21:45.587782+0000","last_unstale":"2024-09-18T08:22:32.032813+0000","last_undegraded":"2024-09-18T08:22:32.032813+0000","last_fullsized":"2024-09-18T08:22:32.032813+0000","mapping_epoch":459,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":460,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:22:10.785776+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:52.890475+0000","last_clean_scrub_stamp":"2024-09-18T08:22:10.785776+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:23:13.806282+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":909,"reported_epoch":481,"state":"active+clean","last_fresh":"2024-09-18T08:22:34.995406+0000","last_change":"2024-09-18T08:22:34.995406+0000","last_active":"2024-09-18T08:22:34.995406+0000","last_peered":"2024-09-18T08:22:34.995406+0000","last_clean":"2024-09-18T08:22:34.995406+0000","last_became_active":"2024-09-18T08:14:39.405347+0000","last_became_peered":"2024-09-18T08:14:39.405347+0000","last_unstale":"2024-09-18T08:22:34.995406+0000","last_undegraded":"2024-09-18T08:22:34.995406+0000","last_fullsized":"2024-09-18T08:22:34.995406+0000","mapping_epoch":307,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":308,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:22:34.995358+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:46.860092+0000","last_clean_scrub_stamp":"2024-09-18T08:22:34.995358+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:23:56.366799+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"271'19","reported_seq":485,"reported_epoch":481,"state":"active+clean","last_fresh":"2024-09-18T08:22:32.033210+0000","last_change":"2024-09-18T08:22:12.720659+0000","last_active":"2024-09-18T08:22:32.033210+0000","last_peered":"2024-09-18T08:22:32.033210+0000","last_clean":"2024-09-18T08:22:32.033210+0000","last_became_active":"2024-09-18T08:22:12.720078+0000","last_became_peered":"2024-09-18T08:22:12.720078+0000","last_unstale":"2024-09-18T08:22:32.033210+0000","last_undegraded":"2024-09-18T08:22:32.033210+0000","last_fullsized":"2024-09-18T08:22:32.033210+0000","mapping_epoch":474,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":475,"parent":"0.0","parent_split_bits":0,"last_scrub":"271'19","last_scrub_stamp":"2024-09-18T08:21:56.226459+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:48.870718+0000","last_clean_scrub_stamp":"2024-09-18T08:21:56.226459+0000","objects_scrubbed":5,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:23:09.795645+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":23,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":944,"reported_epoch":481,"state":"active+clean","last_fresh":"2024-09-18T08:22:32.033825+0000","last_change":"2024-09-18T08:22:16.241762+0000","last_active":"2024-09-18T08:22:32.033825+0000","last_peered":"2024-09-18T08:22:32.033825+0000","last_clean":"2024-09-18T08:22:32.033825+0000","last_became_active":"2024-09-18T08:21:53.633510+0000","last_became_peered":"2024-09-18T08:21:53.633510+0000","last_unstale":"2024-09-18T08:22:32.033825+0000","last_undegraded":"2024-09-18T08:22:32.033825+0000","last_fullsized":"2024-09-18T08:22:32.033825+0000","mapping_epoch":463,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":464,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T08:22:16.241697+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T08:02:50.886830+0000","last_clean_scrub_stamp":"2024-09-18T08:22:16.241697+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:23:29.278077+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":416,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":832,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":35,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":132,"ondisk_log_size":132,"up":96,"acting":96,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":178,"num_object_clones":0,"num_object_copies":534,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":178,"num_whiteouts":0,"num_read":1104,"num_read_kb":861,"num_write":703,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":563,"num_bytes_recovered":11430,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":417792,"data_stored":11106,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3639,"internal_metadata":0},"log_size":693,"ondisk_log_size":693,"up":96,"acting":96,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":1613,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":14,"num_read_kb":14,"num_write":10,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":23,"num_bytes_recovered":7743,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":4839,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":6,"ondisk_log_size":6,"up":96,"acting":96,"num_store_stats":8},{"poolid":2,"num_pg":3,"stat_sum":{"num_bytes":408,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":424,"num_read_kb":419,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":408,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":8,"ondisk_log_size":8,"up":9,"acting":9,"num_store_stats":7},{"poolid":1,"num_pg":11,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":1573408,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":869,"ondisk_log_size":869,"up":33,"acting":33,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":56,"seq":240518168820,"num_pgs":56,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":631960,"kb_used_data":1032,"kb_used_omap":1,"kb_used_meta":630910,"kb_avail":93113192,"statfs":{"total":95995035648,"available":95347908608,"internally_reserved":0,"allocated":1056768,"data_stored":696687,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":646052298},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":47,"seq":201863463162,"num_pgs":49,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":659396,"kb_used_data":692,"kb_used_omap":1,"kb_used_meta":658686,"kb_avail":93085756,"statfs":{"total":95995035648,"available":95319814144,"internally_reserved":0,"allocated":708608,"data_stored":367922,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":674494918},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":39,"seq":167503724800,"num_pgs":53,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":979296,"kb_used_data":1296,"kb_used_omap":1,"kb_used_meta":977982,"kb_avail":92765856,"statfs":{"total":95995035648,"available":94992236544,"internally_reserved":0,"allocated":1327104,"data_stored":959428,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":1001454022},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":32,"seq":137438953733,"num_pgs":61,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":599716,"kb_used_data":724,"kb_used_omap":1,"kb_used_meta":598974,"kb_avail":93145436,"statfs":{"total":95995035648,"available":95380926464,"internally_reserved":0,"allocated":741376,"data_stored":369520,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":613349829},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":26,"seq":111669149964,"num_pgs":60,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":866848,"kb_used_data":1296,"kb_used_omap":1,"kb_used_meta":865534,"kb_avail":92878304,"statfs":{"total":95995035648,"available":95107383296,"internally_reserved":0,"allocated":1327104,"data_stored":958851,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":886307272},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411602,"num_pgs":46,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":672152,"kb_used_data":968,"kb_used_omap":1,"kb_used_meta":671166,"kb_avail":93073000,"statfs":{"total":95995035648,"available":95306752000,"internally_reserved":0,"allocated":991232,"data_stored":631634,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":687274437},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":307,"seq":1318554959971,"num_pgs":52,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":367672,"kb_used_data":744,"kb_used_omap":7,"kb_used_meta":366904,"kb_avail":93377480,"statfs":{"total":95995035648,"available":95618539520,"internally_reserved":0,"allocated":761856,"data_stored":370489,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":8079,"internal_metadata":375709809},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738653,"num_pgs":49,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":812140,"kb_used_data":732,"kb_used_omap":1,"kb_used_meta":811390,"kb_avail":92933012,"statfs":{"total":95995035648,"available":95163404288,"internally_reserved":0,"allocated":749568,"data_stored":369942,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":830863813},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":266240,"data_stored":262688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1131,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1521,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":436,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":46,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1177,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":436,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":1540,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":1678,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3639,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1238,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":1568,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1474,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1034,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1254,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":403,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":416,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":403,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T08:22:47.157 INFO:tasks.cephadm.ceph_manager.ceph:recovered! 2024-09-18T08:22:47.157 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid 779349ca-7593-11ef-bceb-c7b262605968 -- ceph config rm mgr mgr_debug_aggressive_pg_num_changes 2024-09-18T08:22:47.157 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:22:46 smithi050 ceph-mon[32371]: 6.11 scrub starts 2024-09-18T08:22:47.158 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:22:46 smithi050 ceph-mon[32371]: 6.11 scrub ok 2024-09-18T08:22:47.158 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:22:46 smithi050 ceph-mon[32371]: from='client.26585 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-18T08:22:47.424 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/779349ca-7593-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T08:22:48.094 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:22:47 smithi007 ceph-mon[29459]: 6.7 scrub starts 2024-09-18T08:22:48.094 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:22:47 smithi007 ceph-mon[29459]: 6.7 scrub ok 2024-09-18T08:22:48.094 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:22:47 smithi007 ceph-mon[29459]: 4.7 scrub starts 2024-09-18T08:22:48.094 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:22:47 smithi007 ceph-mon[29459]: 4.7 scrub ok 2024-09-18T08:22:48.095 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:22:47 smithi007 ceph-mon[29459]: 5.1 scrub starts 2024-09-18T08:22:48.095 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:22:47 smithi007 ceph-mon[29459]: 5.1 scrub ok 2024-09-18T08:22:48.095 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:22:47 smithi007 ceph-mon[29459]: pgmap v996: 142 pgs: 142 active+clean; 583 KiB data, 5.3 GiB used, 710 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-18T08:22:48.095 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:22:47 smithi007 ceph-mon[39249]: 6.7 scrub starts 2024-09-18T08:22:48.095 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:22:47 smithi007 ceph-mon[39249]: 6.7 scrub ok 2024-09-18T08:22:48.095 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:22:47 smithi007 ceph-mon[39249]: 4.7 scrub starts 2024-09-18T08:22:48.095 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:22:47 smithi007 ceph-mon[39249]: 4.7 scrub ok 2024-09-18T08:22:48.095 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:22:47 smithi007 ceph-mon[39249]: 5.1 scrub starts 2024-09-18T08:22:48.095 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:22:47 smithi007 ceph-mon[39249]: 5.1 scrub ok 2024-09-18T08:22:48.095 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:22:47 smithi007 ceph-mon[39249]: pgmap v996: 142 pgs: 142 active+clean; 583 KiB data, 5.3 GiB used, 710 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-18T08:22:48.155 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:22:47 smithi050 ceph-mon[32371]: 6.7 scrub starts 2024-09-18T08:22:48.155 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:22:47 smithi050 ceph-mon[32371]: 6.7 scrub ok 2024-09-18T08:22:48.155 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:22:47 smithi050 ceph-mon[32371]: 4.7 scrub starts 2024-09-18T08:22:48.155 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:22:47 smithi050 ceph-mon[32371]: 4.7 scrub ok 2024-09-18T08:22:48.155 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:22:47 smithi050 ceph-mon[32371]: 5.1 scrub starts 2024-09-18T08:22:48.155 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:22:47 smithi050 ceph-mon[32371]: 5.1 scrub ok 2024-09-18T08:22:48.155 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:22:47 smithi050 ceph-mon[32371]: pgmap v996: 142 pgs: 142 active+clean; 583 KiB data, 5.3 GiB used, 710 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-18T08:22:48.422 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Sep 18 08:22:48 smithi007 ceph-779349ca-7593-11ef-bceb-c7b262605968-mgr-y[29688]: [18/Sep/2024:08:22:48] ENGINE Bus STOPPING 2024-09-18T08:22:48.422 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Sep 18 08:22:48 smithi007 ceph-779349ca-7593-11ef-bceb-c7b262605968-mgr-y[29688]: [18/Sep/2024:08:22:48] ENGINE HTTP Server cherrypy._cpwsgi_server.CPWSGIServer(('::', 9283)) shut down 2024-09-18T08:22:48.423 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Sep 18 08:22:48 smithi007 ceph-779349ca-7593-11ef-bceb-c7b262605968-mgr-y[29688]: [18/Sep/2024:08:22:48] ENGINE Bus STOPPED 2024-09-18T08:22:48.423 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Sep 18 08:22:48 smithi007 ceph-779349ca-7593-11ef-bceb-c7b262605968-mgr-y[29688]: [18/Sep/2024:08:22:48] ENGINE Bus STARTING 2024-09-18T08:22:48.423 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Sep 18 08:22:48 smithi007 ceph-779349ca-7593-11ef-bceb-c7b262605968-mgr-y[29688]: [18/Sep/2024:08:22:48] ENGINE Serving on http://:::9283 2024-09-18T08:22:48.423 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Sep 18 08:22:48 smithi007 ceph-779349ca-7593-11ef-bceb-c7b262605968-mgr-y[29688]: [18/Sep/2024:08:22:48] ENGINE Bus STARTED 2024-09-18T08:22:48.859 DEBUG:teuthology.run_tasks:Unwinding manager cephadm 2024-09-18T08:22:48.869 INFO:tasks.cephadm:Teardown begin 2024-09-18T08:22:48.870 DEBUG:teuthology.orchestra.run.smithi007:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-09-18T08:22:48.913 DEBUG:teuthology.orchestra.run.smithi050:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-09-18T08:22:48.962 INFO:tasks.cephadm:Cleaning up testdir ceph.* files... 2024-09-18T08:22:48.962 DEBUG:teuthology.orchestra.run.smithi007:> rm -f /home/ubuntu/cephtest/seed.ceph.conf /home/ubuntu/cephtest/ceph.pub 2024-09-18T08:22:48.993 DEBUG:teuthology.orchestra.run.smithi050:> rm -f /home/ubuntu/cephtest/seed.ceph.conf /home/ubuntu/cephtest/ceph.pub 2024-09-18T08:22:49.029 INFO:tasks.cephadm:Stopping all daemons... 2024-09-18T08:22:49.029 INFO:tasks.cephadm.mon.a:Stopping mon.a... 2024-09-18T08:22:49.029 DEBUG:teuthology.orchestra.run.smithi007:> sudo systemctl stop ceph-779349ca-7593-11ef-bceb-c7b262605968@mon.a 2024-09-18T08:22:49.076 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:22:48 smithi007 ceph-mon[39249]: purged_snaps scrub starts 2024-09-18T08:22:49.076 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:22:48 smithi007 ceph-mon[39249]: purged_snaps scrub ok 2024-09-18T08:22:49.076 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:22:48 smithi007 ceph-mon[39249]: osdmap e482: 8 total, 8 up, 8 in 2024-09-18T08:22:49.077 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:22:48 smithi007 ceph-mon[39249]: from='client.? 172.21.15.7:0/3994889903' entity='client.admin' cmd={"prefix": "config rm", "who": "mgr", "name": "mgr_debug_aggressive_pg_num_changes"} : dispatch 2024-09-18T08:22:49.077 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:22:48 smithi007 ceph-mon[39249]: from='client.? ' entity='client.admin' cmd={"prefix": "config rm", "who": "mgr", "name": "mgr_debug_aggressive_pg_num_changes"} : dispatch 2024-09-18T08:22:49.077 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:22:48 smithi007 ceph-mon[39249]: from='client.? ' entity='client.admin' cmd='[{"prefix": "config rm", "who": "mgr", "name": "mgr_debug_aggressive_pg_num_changes"}]': finished 2024-09-18T08:22:49.077 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:22:48 smithi007 ceph-mon[39249]: from='mgr.24470 172.21.15.7:0/2594390261' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-18T08:22:49.077 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:22:48 smithi007 ceph-mon[39249]: from='mgr.24470 172.21.15.7:0/2594390261' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-18T08:22:49.078 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:22:48 smithi007 ceph-mon[39249]: from='mgr.24470 172.21.15.7:0/2594390261' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-18T08:22:49.078 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:22:48 smithi007 ceph-mon[39249]: from='mgr.24470 172.21.15.7:0/2594390261' entity='mgr.y' 2024-09-18T08:22:49.078 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:22:48 smithi007 ceph-mon[39249]: from='mgr.24470 172.21.15.7:0/2594390261' entity='mgr.y' cmd={"prefix": "orch prometheus get-credentials"} : dispatch 2024-09-18T08:22:49.078 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:22:48 smithi007 ceph-mon[39249]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "orch prometheus get-credentials"}]: dispatch 2024-09-18T08:22:49.078 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:22:48 smithi007 ceph-mon[29459]: purged_snaps scrub starts 2024-09-18T08:22:49.078 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:22:48 smithi007 ceph-mon[29459]: purged_snaps scrub ok 2024-09-18T08:22:49.079 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:22:48 smithi007 ceph-mon[29459]: osdmap e482: 8 total, 8 up, 8 in 2024-09-18T08:22:49.079 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:22:48 smithi007 ceph-mon[29459]: from='client.? 172.21.15.7:0/3994889903' entity='client.admin' cmd={"prefix": "config rm", "who": "mgr", "name": "mgr_debug_aggressive_pg_num_changes"} : dispatch 2024-09-18T08:22:49.079 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:22:48 smithi007 ceph-mon[29459]: from='client.? ' entity='client.admin' cmd={"prefix": "config rm", "who": "mgr", "name": "mgr_debug_aggressive_pg_num_changes"} : dispatch 2024-09-18T08:22:49.079 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:22:48 smithi007 ceph-mon[29459]: from='client.? ' entity='client.admin' cmd='[{"prefix": "config rm", "who": "mgr", "name": "mgr_debug_aggressive_pg_num_changes"}]': finished 2024-09-18T08:22:49.079 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:22:48 smithi007 ceph-mon[29459]: from='mgr.24470 172.21.15.7:0/2594390261' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-18T08:22:49.079 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:22:48 smithi007 ceph-mon[29459]: from='mgr.24470 172.21.15.7:0/2594390261' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-18T08:22:49.079 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:22:48 smithi007 ceph-mon[29459]: from='mgr.24470 172.21.15.7:0/2594390261' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-18T08:22:49.079 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:22:48 smithi007 ceph-mon[29459]: from='mgr.24470 172.21.15.7:0/2594390261' entity='mgr.y' 2024-09-18T08:22:49.080 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:22:48 smithi007 ceph-mon[29459]: from='mgr.24470 172.21.15.7:0/2594390261' entity='mgr.y' cmd={"prefix": "orch prometheus get-credentials"} : dispatch 2024-09-18T08:22:49.080 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:22:48 smithi007 ceph-mon[29459]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "orch prometheus get-credentials"}]: dispatch 2024-09-18T08:22:49.080 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:22:49 smithi007 systemd[1]: Stopping Ceph mon.a for 779349ca-7593-11ef-bceb-c7b262605968... 2024-09-18T08:22:49.155 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:22:48 smithi050 ceph-mon[32371]: purged_snaps scrub starts 2024-09-18T08:22:49.155 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:22:48 smithi050 ceph-mon[32371]: purged_snaps scrub ok 2024-09-18T08:22:49.155 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:22:48 smithi050 ceph-mon[32371]: osdmap e482: 8 total, 8 up, 8 in 2024-09-18T08:22:49.155 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:22:48 smithi050 ceph-mon[32371]: from='client.? 172.21.15.7:0/3994889903' entity='client.admin' cmd={"prefix": "config rm", "who": "mgr", "name": "mgr_debug_aggressive_pg_num_changes"} : dispatch 2024-09-18T08:22:49.155 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:22:48 smithi050 ceph-mon[32371]: from='client.? ' entity='client.admin' cmd={"prefix": "config rm", "who": "mgr", "name": "mgr_debug_aggressive_pg_num_changes"} : dispatch 2024-09-18T08:22:49.155 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:22:48 smithi050 ceph-mon[32371]: from='client.? ' entity='client.admin' cmd='[{"prefix": "config rm", "who": "mgr", "name": "mgr_debug_aggressive_pg_num_changes"}]': finished 2024-09-18T08:22:49.155 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:22:48 smithi050 ceph-mon[32371]: from='mgr.24470 172.21.15.7:0/2594390261' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-18T08:22:49.155 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:22:48 smithi050 ceph-mon[32371]: from='mgr.24470 172.21.15.7:0/2594390261' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-18T08:22:49.156 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:22:48 smithi050 ceph-mon[32371]: from='mgr.24470 172.21.15.7:0/2594390261' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-18T08:22:49.156 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:22:48 smithi050 ceph-mon[32371]: from='mgr.24470 172.21.15.7:0/2594390261' entity='mgr.y' 2024-09-18T08:22:49.156 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:22:48 smithi050 ceph-mon[32371]: from='mgr.24470 172.21.15.7:0/2594390261' entity='mgr.y' cmd={"prefix": "orch prometheus get-credentials"} : dispatch 2024-09-18T08:22:49.156 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:22:48 smithi050 ceph-mon[32371]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "orch prometheus get-credentials"}]: dispatch 2024-09-18T08:22:49.730 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:22:49 smithi007 ceph-779349ca-7593-11ef-bceb-c7b262605968-mon-a[29455]: 2024-09-18T08:22:49.342+0000 7fcd30da0640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.a -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-09-18T08:22:49.730 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:22:49 smithi007 ceph-779349ca-7593-11ef-bceb-c7b262605968-mon-a[29455]: 2024-09-18T08:22:49.342+0000 7fcd30da0640 -1 mon.a@0(leader) e3 *** Got Signal Terminated *** 2024-09-18T08:22:49.730 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:22:49 smithi007 podman[195339]: 2024-09-18 08:22:49.52721918 +0000 UTC m=+0.289519354 container died 60741299c8e8988ce730727399daef54ec09a04024a132d4d8b67d279d4f75d5 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e, name=ceph-779349ca-7593-11ef-bceb-c7b262605968-mon-a, org.label-schema.name=CentOS Stream 9 Base Image, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240731, io.buildah.version=1.37.2, ceph=True, org.label-schema.vendor=CentOS, RELEASE=wip-guits-main-2024-09-17-1213-e7fb7b5, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.license=GPLv2, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, org.label-schema.schema-version=1.0) 2024-09-18T08:22:49.731 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Sep 18 08:22:49 smithi007 ceph-779349ca-7593-11ef-bceb-c7b262605968-mgr-y[29688]: [18/Sep/2024:08:22:49] ENGINE Bus STOPPING 2024-09-18T08:22:50.104 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:22:49 smithi007 podman[195339]: 2024-09-18 08:22:49.909939628 +0000 UTC m=+0.672239801 container cleanup 60741299c8e8988ce730727399daef54ec09a04024a132d4d8b67d279d4f75d5 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e, name=ceph-779349ca-7593-11ef-bceb-c7b262605968-mon-a, org.label-schema.license=GPLv2, io.buildah.version=1.37.2, ceph=True, org.label-schema.build-date=20240731, GIT_CLEAN=True, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.vendor=CentOS, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, GIT_BRANCH=HEAD, RELEASE=wip-guits-main-2024-09-17-1213-e7fb7b5, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-09-18T08:22:50.104 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:22:49 smithi007 bash[195339]: ceph-779349ca-7593-11ef-bceb-c7b262605968-mon-a 2024-09-18T08:22:50.104 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Sep 18 08:22:49 smithi007 ceph-779349ca-7593-11ef-bceb-c7b262605968-mgr-y[29688]: [18/Sep/2024:08:22:49] ENGINE HTTP Server cherrypy._cpwsgi_server.CPWSGIServer(('::', 9283)) shut down 2024-09-18T08:22:50.104 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Sep 18 08:22:49 smithi007 ceph-779349ca-7593-11ef-bceb-c7b262605968-mgr-y[29688]: [18/Sep/2024:08:22:49] ENGINE Bus STOPPED 2024-09-18T08:22:50.104 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Sep 18 08:22:49 smithi007 ceph-779349ca-7593-11ef-bceb-c7b262605968-mgr-y[29688]: [18/Sep/2024:08:22:49] ENGINE Bus STARTING 2024-09-18T08:22:50.105 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Sep 18 08:22:49 smithi007 ceph-779349ca-7593-11ef-bceb-c7b262605968-mgr-y[29688]: [18/Sep/2024:08:22:49] ENGINE Serving on http://:::9283 2024-09-18T08:22:50.105 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Sep 18 08:22:49 smithi007 ceph-779349ca-7593-11ef-bceb-c7b262605968-mgr-y[29688]: [18/Sep/2024:08:22:49] ENGINE Bus STARTED 2024-09-18T08:22:50.342 DEBUG:teuthology.orchestra.run.smithi007:> sudo pkill -f 'journalctl -f -n 0 -u ceph-779349ca-7593-11ef-bceb-c7b262605968@mon.a.service' 2024-09-18T08:22:50.405 INFO:journalctl@ceph.iscsi.iscsi.a.smithi050.stdout:Sep 18 08:22:49 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-iscsi-iscsi-a[57782]: debug there is no tcmu-runner data available 2024-09-18T08:22:50.405 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:22:50 smithi007 podman[195351]: 2024-09-18 08:22:50.103443494 +0000 UTC m=+0.570421060 container remove 60741299c8e8988ce730727399daef54ec09a04024a132d4d8b67d279d4f75d5 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e, name=ceph-779349ca-7593-11ef-bceb-c7b262605968-mon-a, io.buildah.version=1.37.2, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, org.label-schema.build-date=20240731, org.label-schema.name=CentOS Stream 9 Base Image, GIT_CLEAN=True, RELEASE=wip-guits-main-2024-09-17-1213-e7fb7b5) 2024-09-18T08:22:50.406 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:22:50 smithi007 systemd[1]: ceph-779349ca-7593-11ef-bceb-c7b262605968@mon.a.service: Deactivated successfully. 2024-09-18T08:22:50.406 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:22:50 smithi007 systemd[1]: Stopped Ceph mon.a for 779349ca-7593-11ef-bceb-c7b262605968. 2024-09-18T08:22:50.406 INFO:journalctl@ceph.mon.a.smithi007.stdout:Sep 18 08:22:50 smithi007 systemd[1]: ceph-779349ca-7593-11ef-bceb-c7b262605968@mon.a.service: Consumed 30.872s CPU time. 2024-09-18T08:22:50.434 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-09-18T08:22:50.434 INFO:tasks.cephadm.mon.a:Stopped mon.a 2024-09-18T08:22:50.434 INFO:tasks.cephadm.mon.b:Stopping mon.c... 2024-09-18T08:22:50.434 DEBUG:teuthology.orchestra.run.smithi007:> sudo systemctl stop ceph-779349ca-7593-11ef-bceb-c7b262605968@mon.c 2024-09-18T08:22:50.729 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:22:50 smithi007 systemd[1]: Stopping Ceph mon.c for 779349ca-7593-11ef-bceb-c7b262605968... 2024-09-18T08:22:51.230 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:22:50 smithi007 ceph-779349ca-7593-11ef-bceb-c7b262605968-mon-c[39245]: 2024-09-18T08:22:50.781+0000 7f51482ec640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.c -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-09-18T08:22:51.230 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:22:50 smithi007 ceph-779349ca-7593-11ef-bceb-c7b262605968-mon-c[39245]: 2024-09-18T08:22:50.781+0000 7f51482ec640 -1 mon.c@2(peon) e3 *** Got Signal Terminated *** 2024-09-18T08:22:51.230 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:22:51 smithi007 podman[195472]: 2024-09-18 08:22:51.017066466 +0000 UTC m=+0.356514430 container died 3898467ba9eb184258f0533b36e37a04dbe605702e07917009b63fb31555588a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:1f5b5a3734f665ef333266f39143ac3ef1eb1951b21258f2307e10332e2d7ac9, name=ceph-779349ca-7593-11ef-bceb-c7b262605968-mon-c, ceph=True, org.label-schema.vendor=CentOS, RELEASE=wip-guits-main-2024-09-17-1213-e7fb7b5, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.schema-version=1.0, GIT_CLEAN=True, io.buildah.version=1.37.2, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.build-date=20240731) 2024-09-18T08:22:51.876 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:22:51 smithi007 podman[195472]: 2024-09-18 08:22:51.648613862 +0000 UTC m=+0.988061826 container cleanup 3898467ba9eb184258f0533b36e37a04dbe605702e07917009b63fb31555588a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:1f5b5a3734f665ef333266f39143ac3ef1eb1951b21258f2307e10332e2d7ac9, name=ceph-779349ca-7593-11ef-bceb-c7b262605968-mon-c, ceph=True, org.label-schema.name=CentOS Stream 9 Base Image, RELEASE=wip-guits-main-2024-09-17-1213-e7fb7b5, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, org.label-schema.build-date=20240731, GIT_CLEAN=True, io.buildah.version=1.37.2, org.label-schema.license=GPLv2) 2024-09-18T08:22:51.876 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:22:51 smithi007 bash[195472]: ceph-779349ca-7593-11ef-bceb-c7b262605968-mon-c 2024-09-18T08:22:52.115 DEBUG:teuthology.orchestra.run.smithi007:> sudo pkill -f 'journalctl -f -n 0 -u ceph-779349ca-7593-11ef-bceb-c7b262605968@mon.c.service' 2024-09-18T08:22:52.137 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:22:51 smithi007 podman[195484]: 2024-09-18 08:22:51.875767916 +0000 UTC m=+0.858611275 container remove 3898467ba9eb184258f0533b36e37a04dbe605702e07917009b63fb31555588a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:1f5b5a3734f665ef333266f39143ac3ef1eb1951b21258f2307e10332e2d7ac9, name=ceph-779349ca-7593-11ef-bceb-c7b262605968-mon-c, org.label-schema.license=GPLv2, GIT_CLEAN=True, RELEASE=wip-guits-main-2024-09-17-1213-e7fb7b5, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, org.label-schema.build-date=20240731, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, org.label-schema.name=CentOS Stream 9 Base Image, GIT_BRANCH=HEAD, io.buildah.version=1.37.2) 2024-09-18T08:22:52.137 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:22:52 smithi007 systemd[1]: ceph-779349ca-7593-11ef-bceb-c7b262605968@mon.c.service: Deactivated successfully. 2024-09-18T08:22:52.137 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:22:52 smithi007 systemd[1]: Stopped Ceph mon.c for 779349ca-7593-11ef-bceb-c7b262605968. 2024-09-18T08:22:52.137 INFO:journalctl@ceph.mon.c.smithi007.stdout:Sep 18 08:22:52 smithi007 systemd[1]: ceph-779349ca-7593-11ef-bceb-c7b262605968@mon.c.service: Consumed 21.144s CPU time. 2024-09-18T08:22:52.150 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-09-18T08:22:52.150 INFO:tasks.cephadm.mon.b:Stopped mon.c 2024-09-18T08:22:52.151 INFO:tasks.cephadm.mon.b:Stopping mon.b... 2024-09-18T08:22:52.151 DEBUG:teuthology.orchestra.run.smithi050:> sudo systemctl stop ceph-779349ca-7593-11ef-bceb-c7b262605968@mon.b 2024-09-18T08:22:52.521 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:22:52 smithi050 systemd[1]: Stopping Ceph mon.b for 779349ca-7593-11ef-bceb-c7b262605968... 2024-09-18T08:22:52.905 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:22:52 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-mon-b[32367]: 2024-09-18T08:22:52.520+0000 7f1a3075d640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.b -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-09-18T08:22:52.905 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:22:52 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-mon-b[32367]: 2024-09-18T08:22:52.520+0000 7f1a3075d640 -1 mon.b@1(peon) e3 *** Got Signal Terminated *** 2024-09-18T08:22:52.905 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:22:52 smithi050 podman[143036]: 2024-09-18 08:22:52.643623204 +0000 UTC m=+0.284708839 container died 4d5b387ea647233d320615cfeacabd2efd6e2657057045bf14ec9db1c95286d8 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:1f5b5a3734f665ef333266f39143ac3ef1eb1951b21258f2307e10332e2d7ac9, name=ceph-779349ca-7593-11ef-bceb-c7b262605968-mon-b, ceph=True, GIT_BRANCH=HEAD, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 9 Base Image, RELEASE=wip-guits-main-2024-09-17-1213-e7fb7b5, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240731, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, io.buildah.version=1.37.2) 2024-09-18T08:22:53.393 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:22:53 smithi050 podman[143036]: 2024-09-18 08:22:53.165279158 +0000 UTC m=+0.806364823 container cleanup 4d5b387ea647233d320615cfeacabd2efd6e2657057045bf14ec9db1c95286d8 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:1f5b5a3734f665ef333266f39143ac3ef1eb1951b21258f2307e10332e2d7ac9, name=ceph-779349ca-7593-11ef-bceb-c7b262605968-mon-b, RELEASE=wip-guits-main-2024-09-17-1213-e7fb7b5, ceph=True, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, io.buildah.version=1.37.2, org.label-schema.name=CentOS Stream 9 Base Image, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.build-date=20240731, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-09-18T08:22:53.393 INFO:journalctl@ceph.mon.b.smithi050.stdout:Sep 18 08:22:53 smithi050 bash[143036]: ceph-779349ca-7593-11ef-bceb-c7b262605968-mon-b 2024-09-18T08:22:53.536 DEBUG:teuthology.orchestra.run.smithi050:> sudo pkill -f 'journalctl -f -n 0 -u ceph-779349ca-7593-11ef-bceb-c7b262605968@mon.b.service' 2024-09-18T08:22:53.615 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-09-18T08:22:53.615 INFO:tasks.cephadm.mon.b:Stopped mon.b 2024-09-18T08:22:53.615 INFO:tasks.cephadm.mgr.y:Stopping mgr.y... 2024-09-18T08:22:53.615 DEBUG:teuthology.orchestra.run.smithi007:> sudo systemctl stop ceph-779349ca-7593-11ef-bceb-c7b262605968@mgr.y 2024-09-18T08:22:53.930 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Sep 18 08:22:53 smithi007 systemd[1]: Stopping Ceph mgr.y for 779349ca-7593-11ef-bceb-c7b262605968... 2024-09-18T08:22:54.229 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Sep 18 08:22:53 smithi007 ceph-779349ca-7593-11ef-bceb-c7b262605968-mgr-y[29688]: 2024-09-18T08:22:53.928+0000 7f9106aba640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mgr -n mgr.y -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-09-18T08:22:54.230 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Sep 18 08:22:53 smithi007 ceph-779349ca-7593-11ef-bceb-c7b262605968-mgr-y[29688]: 2024-09-18T08:22:53.928+0000 7f9106aba640 -1 mgr handle_mgr_signal *** Got signal Terminated *** 2024-09-18T08:22:54.230 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Sep 18 08:22:53 smithi007 podman[195593]: 2024-09-18 08:22:53.959867169 +0000 UTC m=+0.147900940 container died d00c3e2d5d59c0ed8a5bccb0b8816c17315a20b2c5081425b38d5dae022d6609 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e, name=ceph-779349ca-7593-11ef-bceb-c7b262605968-mgr-y, org.label-schema.build-date=20240731, org.label-schema.schema-version=1.0, RELEASE=wip-guits-main-2024-09-17-1213-e7fb7b5, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 9 Base Image, GIT_CLEAN=True, org.label-schema.license=GPLv2, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, ceph=True, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.37.2) 2024-09-18T08:22:54.706 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Sep 18 08:22:54 smithi007 podman[195593]: 2024-09-18 08:22:54.412797926 +0000 UTC m=+0.600831691 container cleanup d00c3e2d5d59c0ed8a5bccb0b8816c17315a20b2c5081425b38d5dae022d6609 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e, name=ceph-779349ca-7593-11ef-bceb-c7b262605968-mgr-y, org.label-schema.license=GPLv2, io.buildah.version=1.37.2, RELEASE=wip-guits-main-2024-09-17-1213-e7fb7b5, org.label-schema.build-date=20240731, org.label-schema.vendor=CentOS, GIT_CLEAN=True, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, ceph=True, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 9 Base Image) 2024-09-18T08:22:54.707 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Sep 18 08:22:54 smithi007 bash[195593]: ceph-779349ca-7593-11ef-bceb-c7b262605968-mgr-y 2024-09-18T08:22:54.873 DEBUG:teuthology.orchestra.run.smithi007:> sudo pkill -f 'journalctl -f -n 0 -u ceph-779349ca-7593-11ef-bceb-c7b262605968@mgr.y.service' 2024-09-18T08:22:54.911 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-09-18T08:22:54.911 INFO:tasks.cephadm.mgr.y:Stopped mgr.y 2024-09-18T08:22:54.912 INFO:tasks.cephadm.mgr.x:Stopping mgr.x... 2024-09-18T08:22:54.912 DEBUG:teuthology.orchestra.run.smithi050:> sudo systemctl stop ceph-779349ca-7593-11ef-bceb-c7b262605968@mgr.x 2024-09-18T08:22:55.225 INFO:journalctl@ceph.mgr.x.smithi050.stdout:Sep 18 08:22:54 smithi050 systemd[1]: Stopping Ceph mgr.x for 779349ca-7593-11ef-bceb-c7b262605968... 2024-09-18T08:22:55.489 INFO:journalctl@ceph.mgr.x.smithi050.stdout:Sep 18 08:22:55 smithi050 podman[143159]: 2024-09-18 08:22:55.225070044 +0000 UTC m=+0.120505255 container died 1a9f4b29ff88c87571787a968ddfb81a0a4d4167ed27727568772c0d887d3f61 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:1f5b5a3734f665ef333266f39143ac3ef1eb1951b21258f2307e10332e2d7ac9, name=ceph-779349ca-7593-11ef-bceb-c7b262605968-mgr-x, GIT_BRANCH=HEAD, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.build-date=20240731, GIT_CLEAN=True, io.buildah.version=1.37.2, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 9 Base Image, RELEASE=wip-guits-main-2024-09-17-1213-e7fb7b5) 2024-09-18T08:22:55.873 INFO:journalctl@ceph.mgr.x.smithi050.stdout:Sep 18 08:22:55 smithi050 podman[143159]: 2024-09-18 08:22:55.604320961 +0000 UTC m=+0.499756157 container cleanup 1a9f4b29ff88c87571787a968ddfb81a0a4d4167ed27727568772c0d887d3f61 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:1f5b5a3734f665ef333266f39143ac3ef1eb1951b21258f2307e10332e2d7ac9, name=ceph-779349ca-7593-11ef-bceb-c7b262605968-mgr-x, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.name=CentOS Stream 9 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-guits-main-2024-09-17-1213-e7fb7b5, GIT_CLEAN=True, io.buildah.version=1.37.2, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, org.label-schema.build-date=20240731, CEPH_POINT_RELEASE=) 2024-09-18T08:22:55.873 INFO:journalctl@ceph.mgr.x.smithi050.stdout:Sep 18 08:22:55 smithi050 bash[143159]: ceph-779349ca-7593-11ef-bceb-c7b262605968-mgr-x 2024-09-18T08:22:56.032 DEBUG:teuthology.orchestra.run.smithi050:> sudo pkill -f 'journalctl -f -n 0 -u ceph-779349ca-7593-11ef-bceb-c7b262605968@mgr.x.service' 2024-09-18T08:22:56.069 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-09-18T08:22:56.069 INFO:tasks.cephadm.mgr.x:Stopped mgr.x 2024-09-18T08:22:56.069 INFO:tasks.cephadm.osd.0:Stopping osd.0... 2024-09-18T08:22:56.069 DEBUG:teuthology.orchestra.run.smithi007:> sudo systemctl stop ceph-779349ca-7593-11ef-bceb-c7b262605968@osd.0 2024-09-18T08:22:56.393 INFO:journalctl@ceph.osd.0.smithi007.stdout:Sep 18 08:22:56 smithi007 systemd[1]: Stopping Ceph osd.0 for 779349ca-7593-11ef-bceb-c7b262605968... 2024-09-18T08:22:56.729 INFO:journalctl@ceph.osd.0.smithi007.stdout:Sep 18 08:22:56 smithi007 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd-0[42581]: 2024-09-18T08:22:56.391+0000 7f901ac10640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.0 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-09-18T08:22:56.729 INFO:journalctl@ceph.osd.0.smithi007.stdout:Sep 18 08:22:56 smithi007 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd-0[42581]: 2024-09-18T08:22:56.391+0000 7f901ac10640 -1 osd.0 482 *** Got signal Terminated *** 2024-09-18T08:22:56.730 INFO:journalctl@ceph.osd.0.smithi007.stdout:Sep 18 08:22:56 smithi007 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd-0[42581]: 2024-09-18T08:22:56.391+0000 7f901ac10640 -1 osd.0 482 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-09-18T08:22:58.405 INFO:journalctl@ceph.prometheus.a.smithi050.stdout:Sep 18 08:22:58 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-prometheus-a[65581]: ts=2024-09-18T08:22:58.008Z caller=refresh.go:90 level=error component="discovery manager scrape" discovery=http config=ceph msg="Unable to refresh target groups" err="Get \"http://172.21.15.7:8765/sd/prometheus/sd-config?service=mgr-prometheus\": dial tcp 172.21.15.7:8765: connect: connection refused" 2024-09-18T08:22:58.405 INFO:journalctl@ceph.prometheus.a.smithi050.stdout:Sep 18 08:22:58 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-prometheus-a[65581]: ts=2024-09-18T08:22:58.009Z caller=refresh.go:90 level=error component="discovery manager scrape" discovery=http config=node msg="Unable to refresh target groups" err="Get \"http://172.21.15.7:8765/sd/prometheus/sd-config?service=node-exporter\": dial tcp 172.21.15.7:8765: connect: connection refused" 2024-09-18T08:22:58.405 INFO:journalctl@ceph.prometheus.a.smithi050.stdout:Sep 18 08:22:58 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-prometheus-a[65581]: ts=2024-09-18T08:22:58.013Z caller=refresh.go:90 level=error component="discovery manager scrape" discovery=http config=nfs msg="Unable to refresh target groups" err="Get \"http://172.21.15.7:8765/sd/prometheus/sd-config?service=nfs\": dial tcp 172.21.15.7:8765: connect: connection refused" 2024-09-18T08:22:58.405 INFO:journalctl@ceph.prometheus.a.smithi050.stdout:Sep 18 08:22:58 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-prometheus-a[65581]: ts=2024-09-18T08:22:58.015Z caller=refresh.go:90 level=error component="discovery manager scrape" discovery=http config=ceph-exporter msg="Unable to refresh target groups" err="Get \"http://172.21.15.7:8765/sd/prometheus/sd-config?service=ceph-exporter\": dial tcp 172.21.15.7:8765: connect: connection refused" 2024-09-18T08:22:58.405 INFO:journalctl@ceph.prometheus.a.smithi050.stdout:Sep 18 08:22:58 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-prometheus-a[65581]: ts=2024-09-18T08:22:58.017Z caller=refresh.go:90 level=error component="discovery manager scrape" discovery=http config=nvmeof msg="Unable to refresh target groups" err="Get \"http://172.21.15.7:8765/sd/prometheus/sd-config?service=nvmeof\": dial tcp 172.21.15.7:8765: connect: connection refused" 2024-09-18T08:22:58.406 INFO:journalctl@ceph.prometheus.a.smithi050.stdout:Sep 18 08:22:58 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-prometheus-a[65581]: ts=2024-09-18T08:22:58.021Z caller=refresh.go:90 level=error component="discovery manager scrape" discovery=http config=smb msg="Unable to refresh target groups" err="Get \"http://172.21.15.7:8765/sd/prometheus/sd-config?service=smb\": dial tcp 172.21.15.7:8765: connect: connection refused" 2024-09-18T08:22:58.406 INFO:journalctl@ceph.prometheus.a.smithi050.stdout:Sep 18 08:22:58 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-prometheus-a[65581]: ts=2024-09-18T08:22:58.013Z caller=refresh.go:90 level=error component="discovery manager notify" discovery=http config=config-0 msg="Unable to refresh target groups" err="Get \"http://172.21.15.7:8765/sd/prometheus/sd-config?service=alertmanager\": dial tcp 172.21.15.7:8765: connect: connection refused" 2024-09-18T08:23:02.179 INFO:journalctl@ceph.osd.0.smithi007.stdout:Sep 18 08:23:01 smithi007 podman[195716]: 2024-09-18 08:23:01.907604131 +0000 UTC m=+5.640643165 container died a95fe8ff46c4735c78853e4688aef031b98982797c792d848ef141eb0a39bf07 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:1f5b5a3734f665ef333266f39143ac3ef1eb1951b21258f2307e10332e2d7ac9, name=ceph-779349ca-7593-11ef-bceb-c7b262605968-osd-0, GIT_BRANCH=HEAD, io.buildah.version=1.37.2, org.label-schema.build-date=20240731, maintainer=Guillaume Abrioux , RELEASE=wip-guits-main-2024-09-17-1213-e7fb7b5, org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 9 Base Image, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965) 2024-09-18T08:23:02.479 INFO:journalctl@ceph.osd.0.smithi007.stdout:Sep 18 08:23:02 smithi007 podman[195716]: 2024-09-18 08:23:02.325702063 +0000 UTC m=+6.058741089 container cleanup a95fe8ff46c4735c78853e4688aef031b98982797c792d848ef141eb0a39bf07 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:1f5b5a3734f665ef333266f39143ac3ef1eb1951b21258f2307e10332e2d7ac9, name=ceph-779349ca-7593-11ef-bceb-c7b262605968-osd-0, RELEASE=wip-guits-main-2024-09-17-1213-e7fb7b5, ceph=True, io.buildah.version=1.37.2, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, org.label-schema.name=CentOS Stream 9 Base Image, GIT_CLEAN=True, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240731, org.label-schema.schema-version=1.0) 2024-09-18T08:23:02.480 INFO:journalctl@ceph.osd.0.smithi007.stdout:Sep 18 08:23:02 smithi007 bash[195716]: ceph-779349ca-7593-11ef-bceb-c7b262605968-osd-0 2024-09-18T08:23:02.979 INFO:journalctl@ceph.osd.0.smithi007.stdout:Sep 18 08:23:02 smithi007 podman[195731]: 2024-09-18 08:23:02.602468235 +0000 UTC m=+0.689646084 container remove a95fe8ff46c4735c78853e4688aef031b98982797c792d848ef141eb0a39bf07 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:1f5b5a3734f665ef333266f39143ac3ef1eb1951b21258f2307e10332e2d7ac9, name=ceph-779349ca-7593-11ef-bceb-c7b262605968-osd-0, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 9 Base Image, io.buildah.version=1.37.2, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, RELEASE=wip-guits-main-2024-09-17-1213-e7fb7b5, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.build-date=20240731, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, GIT_BRANCH=HEAD) 2024-09-18T08:23:03.273 INFO:journalctl@ceph.osd.0.smithi007.stdout:Sep 18 08:23:03 smithi007 podman[195820]: 2024-09-18 08:23:02.913433758 +0000 UTC m=+0.016305757 image pull b22e1e37ddf1ca6e22136cd6540a9ab61e7cafb8bfbc7a91a2f20f7f9941bd9a quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:1f5b5a3734f665ef333266f39143ac3ef1eb1951b21258f2307e10332e2d7ac9 2024-09-18T08:23:03.273 INFO:journalctl@ceph.osd.0.smithi007.stdout:Sep 18 08:23:03 smithi007 podman[195820]: 2024-09-18 08:23:03.036744703 +0000 UTC m=+0.139616695 container create 84572162913db03c68379485dbde2808b238b31896b6f6ab64422b08cc2afc51 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:1f5b5a3734f665ef333266f39143ac3ef1eb1951b21258f2307e10332e2d7ac9, name=ceph-779349ca-7593-11ef-bceb-c7b262605968-osd-0-deactivate, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, ceph=True, io.buildah.version=1.37.2, org.label-schema.build-date=20240731, CEPH_POINT_RELEASE=, GIT_CLEAN=True, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-guits-main-2024-09-17-1213-e7fb7b5) 2024-09-18T08:23:03.558 INFO:journalctl@ceph.osd.0.smithi007.stdout:Sep 18 08:23:03 smithi007 podman[195820]: 2024-09-18 08:23:03.272658864 +0000 UTC m=+0.375530851 container init 84572162913db03c68379485dbde2808b238b31896b6f6ab64422b08cc2afc51 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:1f5b5a3734f665ef333266f39143ac3ef1eb1951b21258f2307e10332e2d7ac9, name=ceph-779349ca-7593-11ef-bceb-c7b262605968-osd-0-deactivate, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240731, org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 9 Base Image, RELEASE=wip-guits-main-2024-09-17-1213-e7fb7b5, maintainer=Guillaume Abrioux , GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, io.buildah.version=1.37.2, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=) 2024-09-18T08:23:03.558 INFO:journalctl@ceph.osd.0.smithi007.stdout:Sep 18 08:23:03 smithi007 podman[195820]: 2024-09-18 08:23:03.276553748 +0000 UTC m=+0.379425736 container start 84572162913db03c68379485dbde2808b238b31896b6f6ab64422b08cc2afc51 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:1f5b5a3734f665ef333266f39143ac3ef1eb1951b21258f2307e10332e2d7ac9, name=ceph-779349ca-7593-11ef-bceb-c7b262605968-osd-0-deactivate, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, maintainer=Guillaume Abrioux , GIT_CLEAN=True, GIT_BRANCH=HEAD, ceph=True, org.label-schema.name=CentOS Stream 9 Base Image, RELEASE=wip-guits-main-2024-09-17-1213-e7fb7b5, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.37.2, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240731) 2024-09-18T08:23:03.558 INFO:journalctl@ceph.osd.0.smithi007.stdout:Sep 18 08:23:03 smithi007 podman[195820]: 2024-09-18 08:23:03.40603526 +0000 UTC m=+0.508907248 container attach 84572162913db03c68379485dbde2808b238b31896b6f6ab64422b08cc2afc51 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:1f5b5a3734f665ef333266f39143ac3ef1eb1951b21258f2307e10332e2d7ac9, name=ceph-779349ca-7593-11ef-bceb-c7b262605968-osd-0-deactivate, org.label-schema.build-date=20240731, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, ceph=True, io.buildah.version=1.37.2, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 9 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, GIT_CLEAN=True, RELEASE=wip-guits-main-2024-09-17-1213-e7fb7b5, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2) 2024-09-18T08:23:03.911 INFO:journalctl@ceph.osd.0.smithi007.stdout:Sep 18 08:23:03 smithi007 podman[195820]: 2024-09-18 08:23:03.557393747 +0000 UTC m=+0.660265744 container died 84572162913db03c68379485dbde2808b238b31896b6f6ab64422b08cc2afc51 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:1f5b5a3734f665ef333266f39143ac3ef1eb1951b21258f2307e10332e2d7ac9, name=ceph-779349ca-7593-11ef-bceb-c7b262605968-osd-0-deactivate, io.buildah.version=1.37.2, org.label-schema.schema-version=1.0, ceph=True, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.build-date=20240731, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.license=GPLv2, RELEASE=wip-guits-main-2024-09-17-1213-e7fb7b5) 2024-09-18T08:23:04.205 DEBUG:teuthology.orchestra.run.smithi007:> sudo pkill -f 'journalctl -f -n 0 -u ceph-779349ca-7593-11ef-bceb-c7b262605968@osd.0.service' 2024-09-18T08:23:04.246 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-09-18T08:23:04.247 INFO:tasks.cephadm.osd.0:Stopped osd.0 2024-09-18T08:23:04.247 INFO:tasks.cephadm.osd.1:Stopping osd.1... 2024-09-18T08:23:04.247 DEBUG:teuthology.orchestra.run.smithi007:> sudo systemctl stop ceph-779349ca-7593-11ef-bceb-c7b262605968@osd.1 2024-09-18T08:23:04.479 INFO:journalctl@ceph.osd.1.smithi007.stdout:Sep 18 08:23:04 smithi007 systemd[1]: Stopping Ceph osd.1 for 779349ca-7593-11ef-bceb-c7b262605968... 2024-09-18T08:23:04.980 INFO:journalctl@ceph.osd.1.smithi007.stdout:Sep 18 08:23:04 smithi007 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd-1[145989]: 2024-09-18T08:23:04.579+0000 7f1cb3871640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.1 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-09-18T08:23:04.980 INFO:journalctl@ceph.osd.1.smithi007.stdout:Sep 18 08:23:04 smithi007 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd-1[145989]: 2024-09-18T08:23:04.579+0000 7f1cb3871640 -1 osd.1 482 *** Got signal Terminated *** 2024-09-18T08:23:04.980 INFO:journalctl@ceph.osd.1.smithi007.stdout:Sep 18 08:23:04 smithi007 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd-1[145989]: 2024-09-18T08:23:04.579+0000 7f1cb3871640 -1 osd.1 482 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-09-18T08:23:10.479 INFO:journalctl@ceph.osd.1.smithi007.stdout:Sep 18 08:23:10 smithi007 podman[195928]: 2024-09-18 08:23:10.20430466 +0000 UTC m=+5.727189705 container died 30865df549477a986230448f6dfbc1fbfa574da663a76054079bb27922256cca (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:1f5b5a3734f665ef333266f39143ac3ef1eb1951b21258f2307e10332e2d7ac9, name=ceph-779349ca-7593-11ef-bceb-c7b262605968-osd-1, org.label-schema.schema-version=1.0, ceph=True, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 9 Base Image, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240731, GIT_CLEAN=True, RELEASE=wip-guits-main-2024-09-17-1213-e7fb7b5, io.buildah.version=1.37.2) 2024-09-18T08:23:10.957 INFO:journalctl@ceph.osd.1.smithi007.stdout:Sep 18 08:23:10 smithi007 podman[195928]: 2024-09-18 08:23:10.754982901 +0000 UTC m=+6.277867946 container cleanup 30865df549477a986230448f6dfbc1fbfa574da663a76054079bb27922256cca (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:1f5b5a3734f665ef333266f39143ac3ef1eb1951b21258f2307e10332e2d7ac9, name=ceph-779349ca-7593-11ef-bceb-c7b262605968-osd-1, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, GIT_CLEAN=True, CEPH_POINT_RELEASE=, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, org.label-schema.schema-version=1.0, RELEASE=wip-guits-main-2024-09-17-1213-e7fb7b5, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 9 Base Image, io.buildah.version=1.37.2, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240731, org.label-schema.license=GPLv2) 2024-09-18T08:23:10.957 INFO:journalctl@ceph.osd.1.smithi007.stdout:Sep 18 08:23:10 smithi007 bash[195928]: ceph-779349ca-7593-11ef-bceb-c7b262605968-osd-1 2024-09-18T08:23:11.229 INFO:journalctl@ceph.osd.1.smithi007.stdout:Sep 18 08:23:10 smithi007 podman[195952]: 2024-09-18 08:23:10.956865816 +0000 UTC m=+0.736155895 container remove 30865df549477a986230448f6dfbc1fbfa574da663a76054079bb27922256cca (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:1f5b5a3734f665ef333266f39143ac3ef1eb1951b21258f2307e10332e2d7ac9, name=ceph-779349ca-7593-11ef-bceb-c7b262605968-osd-1, RELEASE=wip-guits-main-2024-09-17-1213-e7fb7b5, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, io.buildah.version=1.37.2, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 9 Base Image, GIT_BRANCH=HEAD, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, ceph=True, org.label-schema.build-date=20240731, org.label-schema.schema-version=1.0) 2024-09-18T08:23:11.719 INFO:journalctl@ceph.osd.1.smithi007.stdout:Sep 18 08:23:11 smithi007 podman[196043]: 2024-09-18 08:23:11.289747708 +0000 UTC m=+0.016386320 image pull b22e1e37ddf1ca6e22136cd6540a9ab61e7cafb8bfbc7a91a2f20f7f9941bd9a quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:1f5b5a3734f665ef333266f39143ac3ef1eb1951b21258f2307e10332e2d7ac9 2024-09-18T08:23:11.719 INFO:journalctl@ceph.osd.1.smithi007.stdout:Sep 18 08:23:11 smithi007 podman[196043]: 2024-09-18 08:23:11.399252715 +0000 UTC m=+0.125891327 container create 3b53219fbecdc4c495192db26ff4b56fd703a1a228e36c5f09cdae37f7c805d3 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:1f5b5a3734f665ef333266f39143ac3ef1eb1951b21258f2307e10332e2d7ac9, name=ceph-779349ca-7593-11ef-bceb-c7b262605968-osd-1-deactivate, ceph=True, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240731, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, GIT_CLEAN=True, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, RELEASE=wip-guits-main-2024-09-17-1213-e7fb7b5, org.label-schema.license=GPLv2, io.buildah.version=1.37.2) 2024-09-18T08:23:11.719 INFO:journalctl@ceph.osd.1.smithi007.stdout:Sep 18 08:23:11 smithi007 podman[196043]: 2024-09-18 08:23:11.626638852 +0000 UTC m=+0.353277472 container init 3b53219fbecdc4c495192db26ff4b56fd703a1a228e36c5f09cdae37f7c805d3 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:1f5b5a3734f665ef333266f39143ac3ef1eb1951b21258f2307e10332e2d7ac9, name=ceph-779349ca-7593-11ef-bceb-c7b262605968-osd-1-deactivate, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.build-date=20240731, org.label-schema.license=GPLv2, RELEASE=wip-guits-main-2024-09-17-1213-e7fb7b5, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 9 Base Image, io.buildah.version=1.37.2) 2024-09-18T08:23:11.719 INFO:journalctl@ceph.osd.1.smithi007.stdout:Sep 18 08:23:11 smithi007 podman[196043]: 2024-09-18 08:23:11.630067769 +0000 UTC m=+0.356706388 container start 3b53219fbecdc4c495192db26ff4b56fd703a1a228e36c5f09cdae37f7c805d3 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:1f5b5a3734f665ef333266f39143ac3ef1eb1951b21258f2307e10332e2d7ac9, name=ceph-779349ca-7593-11ef-bceb-c7b262605968-osd-1-deactivate, ceph=True, GIT_BRANCH=HEAD, org.label-schema.build-date=20240731, org.label-schema.vendor=CentOS, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, org.label-schema.schema-version=1.0, GIT_CLEAN=True, io.buildah.version=1.37.2, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.license=GPLv2, RELEASE=wip-guits-main-2024-09-17-1213-e7fb7b5) 2024-09-18T08:23:11.719 INFO:journalctl@ceph.osd.1.smithi007.stdout:Sep 18 08:23:11 smithi007 podman[196043]: 2024-09-18 08:23:11.718358876 +0000 UTC m=+0.444997488 container attach 3b53219fbecdc4c495192db26ff4b56fd703a1a228e36c5f09cdae37f7c805d3 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:1f5b5a3734f665ef333266f39143ac3ef1eb1951b21258f2307e10332e2d7ac9, name=ceph-779349ca-7593-11ef-bceb-c7b262605968-osd-1-deactivate, org.label-schema.vendor=CentOS, RELEASE=wip-guits-main-2024-09-17-1213-e7fb7b5, ceph=True, org.label-schema.build-date=20240731, maintainer=Guillaume Abrioux , io.buildah.version=1.37.2, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 9 Base Image) 2024-09-18T08:23:11.979 INFO:journalctl@ceph.osd.1.smithi007.stdout:Sep 18 08:23:11 smithi007 podman[196043]: 2024-09-18 08:23:11.823504916 +0000 UTC m=+0.550143528 container died 3b53219fbecdc4c495192db26ff4b56fd703a1a228e36c5f09cdae37f7c805d3 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:1f5b5a3734f665ef333266f39143ac3ef1eb1951b21258f2307e10332e2d7ac9, name=ceph-779349ca-7593-11ef-bceb-c7b262605968-osd-1-deactivate, io.buildah.version=1.37.2, org.label-schema.name=CentOS Stream 9 Base Image, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, RELEASE=wip-guits-main-2024-09-17-1213-e7fb7b5, org.label-schema.schema-version=1.0, GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240731) 2024-09-18T08:23:12.519 DEBUG:teuthology.orchestra.run.smithi007:> sudo pkill -f 'journalctl -f -n 0 -u ceph-779349ca-7593-11ef-bceb-c7b262605968@osd.1.service' 2024-09-18T08:23:12.567 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-09-18T08:23:12.568 INFO:tasks.cephadm.osd.1:Stopped osd.1 2024-09-18T08:23:12.568 INFO:tasks.cephadm.osd.2:Stopping osd.2... 2024-09-18T08:23:12.568 DEBUG:teuthology.orchestra.run.smithi007:> sudo systemctl stop ceph-779349ca-7593-11ef-bceb-c7b262605968@osd.2 2024-09-18T08:23:12.860 INFO:journalctl@ceph.osd.2.smithi007.stdout:Sep 18 08:23:12 smithi007 systemd[1]: Stopping Ceph osd.2 for 779349ca-7593-11ef-bceb-c7b262605968... 2024-09-18T08:23:13.229 INFO:journalctl@ceph.osd.2.smithi007.stdout:Sep 18 08:23:12 smithi007 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd-2[53338]: 2024-09-18T08:23:12.858+0000 7f7a54054640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.2 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-09-18T08:23:13.230 INFO:journalctl@ceph.osd.2.smithi007.stdout:Sep 18 08:23:12 smithi007 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd-2[53338]: 2024-09-18T08:23:12.858+0000 7f7a54054640 -1 osd.2 482 *** Got signal Terminated *** 2024-09-18T08:23:13.230 INFO:journalctl@ceph.osd.2.smithi007.stdout:Sep 18 08:23:12 smithi007 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd-2[53338]: 2024-09-18T08:23:12.858+0000 7f7a54054640 -1 osd.2 482 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-09-18T08:23:18.618 INFO:journalctl@ceph.osd.2.smithi007.stdout:Sep 18 08:23:18 smithi007 podman[196145]: 2024-09-18 08:23:18.341540151 +0000 UTC m=+5.576999269 container died 1510f3165323d0de2d00d73573a9e052e7fcb6d5f35d6174c92d59c7eb3c26cc (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:1f5b5a3734f665ef333266f39143ac3ef1eb1951b21258f2307e10332e2d7ac9, name=ceph-779349ca-7593-11ef-bceb-c7b262605968-osd-2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, RELEASE=wip-guits-main-2024-09-17-1213-e7fb7b5, ceph=True, io.buildah.version=1.37.2, org.label-schema.name=CentOS Stream 9 Base Image, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.build-date=20240731) 2024-09-18T08:23:18.979 INFO:journalctl@ceph.osd.2.smithi007.stdout:Sep 18 08:23:18 smithi007 podman[196145]: 2024-09-18 08:23:18.791964347 +0000 UTC m=+6.027423505 container cleanup 1510f3165323d0de2d00d73573a9e052e7fcb6d5f35d6174c92d59c7eb3c26cc (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:1f5b5a3734f665ef333266f39143ac3ef1eb1951b21258f2307e10332e2d7ac9, name=ceph-779349ca-7593-11ef-bceb-c7b262605968-osd-2, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, RELEASE=wip-guits-main-2024-09-17-1213-e7fb7b5, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.build-date=20240731, ceph=True, io.buildah.version=1.37.2, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=) 2024-09-18T08:23:18.980 INFO:journalctl@ceph.osd.2.smithi007.stdout:Sep 18 08:23:18 smithi007 bash[196145]: ceph-779349ca-7593-11ef-bceb-c7b262605968-osd-2 2024-09-18T08:23:19.467 INFO:journalctl@ceph.osd.2.smithi007.stdout:Sep 18 08:23:19 smithi007 podman[196163]: 2024-09-18 08:23:19.026734211 +0000 UTC m=+0.669919416 container remove 1510f3165323d0de2d00d73573a9e052e7fcb6d5f35d6174c92d59c7eb3c26cc (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:1f5b5a3734f665ef333266f39143ac3ef1eb1951b21258f2307e10332e2d7ac9, name=ceph-779349ca-7593-11ef-bceb-c7b262605968-osd-2, GIT_CLEAN=True, RELEASE=wip-guits-main-2024-09-17-1213-e7fb7b5, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , ceph=True, io.buildah.version=1.37.2, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, org.label-schema.build-date=20240731, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0) 2024-09-18T08:23:19.730 INFO:journalctl@ceph.osd.2.smithi007.stdout:Sep 18 08:23:19 smithi007 podman[196251]: 2024-09-18 08:23:19.369446503 +0000 UTC m=+0.016651566 image pull b22e1e37ddf1ca6e22136cd6540a9ab61e7cafb8bfbc7a91a2f20f7f9941bd9a quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:1f5b5a3734f665ef333266f39143ac3ef1eb1951b21258f2307e10332e2d7ac9 2024-09-18T08:23:19.730 INFO:journalctl@ceph.osd.2.smithi007.stdout:Sep 18 08:23:19 smithi007 podman[196251]: 2024-09-18 08:23:19.477376504 +0000 UTC m=+0.124581566 container create 2207b742dbecfd292efb88e0f39e7e84acfe14ac8456caf9d214de22fdd62197 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:1f5b5a3734f665ef333266f39143ac3ef1eb1951b21258f2307e10332e2d7ac9, name=ceph-779349ca-7593-11ef-bceb-c7b262605968-osd-2-deactivate, GIT_BRANCH=HEAD, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, org.label-schema.name=CentOS Stream 9 Base Image, CEPH_POINT_RELEASE=, io.buildah.version=1.37.2, ceph=True, org.label-schema.schema-version=1.0, RELEASE=wip-guits-main-2024-09-17-1213-e7fb7b5, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.build-date=20240731, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-09-18T08:23:19.730 INFO:journalctl@ceph.osd.2.smithi007.stdout:Sep 18 08:23:19 smithi007 podman[196251]: 2024-09-18 08:23:19.671411405 +0000 UTC m=+0.318616475 container init 2207b742dbecfd292efb88e0f39e7e84acfe14ac8456caf9d214de22fdd62197 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:1f5b5a3734f665ef333266f39143ac3ef1eb1951b21258f2307e10332e2d7ac9, name=ceph-779349ca-7593-11ef-bceb-c7b262605968-osd-2-deactivate, org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.build-date=20240731, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.37.2, GIT_BRANCH=HEAD, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, org.label-schema.name=CentOS Stream 9 Base Image, RELEASE=wip-guits-main-2024-09-17-1213-e7fb7b5, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0) 2024-09-18T08:23:19.731 INFO:journalctl@ceph.osd.2.smithi007.stdout:Sep 18 08:23:19 smithi007 podman[196251]: 2024-09-18 08:23:19.674967022 +0000 UTC m=+0.322172085 container start 2207b742dbecfd292efb88e0f39e7e84acfe14ac8456caf9d214de22fdd62197 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:1f5b5a3734f665ef333266f39143ac3ef1eb1951b21258f2307e10332e2d7ac9, name=ceph-779349ca-7593-11ef-bceb-c7b262605968-osd-2-deactivate, org.label-schema.build-date=20240731, org.label-schema.license=GPLv2, ceph=True, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 9 Base Image, RELEASE=wip-guits-main-2024-09-17-1213-e7fb7b5, CEPH_POINT_RELEASE=, io.buildah.version=1.37.2) 2024-09-18T08:23:20.159 INFO:journalctl@ceph.osd.2.smithi007.stdout:Sep 18 08:23:19 smithi007 podman[196251]: 2024-09-18 08:23:19.77175254 +0000 UTC m=+0.418957606 container attach 2207b742dbecfd292efb88e0f39e7e84acfe14ac8456caf9d214de22fdd62197 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:1f5b5a3734f665ef333266f39143ac3ef1eb1951b21258f2307e10332e2d7ac9, name=ceph-779349ca-7593-11ef-bceb-c7b262605968-osd-2-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240731, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, io.buildah.version=1.37.2, org.label-schema.name=CentOS Stream 9 Base Image, RELEASE=wip-guits-main-2024-09-17-1213-e7fb7b5, maintainer=Guillaume Abrioux , GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, ceph=True) 2024-09-18T08:23:20.159 INFO:journalctl@ceph.osd.2.smithi007.stdout:Sep 18 08:23:19 smithi007 podman[196251]: 2024-09-18 08:23:19.873048423 +0000 UTC m=+0.520253504 container died 2207b742dbecfd292efb88e0f39e7e84acfe14ac8456caf9d214de22fdd62197 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:1f5b5a3734f665ef333266f39143ac3ef1eb1951b21258f2307e10332e2d7ac9, name=ceph-779349ca-7593-11ef-bceb-c7b262605968-osd-2-deactivate, org.label-schema.build-date=20240731, RELEASE=wip-guits-main-2024-09-17-1213-e7fb7b5, io.buildah.version=1.37.2, org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 9 Base Image, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, ceph=True) 2024-09-18T08:23:20.595 DEBUG:teuthology.orchestra.run.smithi007:> sudo pkill -f 'journalctl -f -n 0 -u ceph-779349ca-7593-11ef-bceb-c7b262605968@osd.2.service' 2024-09-18T08:23:20.636 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-09-18T08:23:20.636 INFO:tasks.cephadm.osd.2:Stopped osd.2 2024-09-18T08:23:20.636 INFO:tasks.cephadm.osd.3:Stopping osd.3... 2024-09-18T08:23:20.636 DEBUG:teuthology.orchestra.run.smithi007:> sudo systemctl stop ceph-779349ca-7593-11ef-bceb-c7b262605968@osd.3 2024-09-18T08:23:20.980 INFO:journalctl@ceph.osd.3.smithi007.stdout:Sep 18 08:23:20 smithi007 systemd[1]: Stopping Ceph osd.3 for 779349ca-7593-11ef-bceb-c7b262605968... 2024-09-18T08:23:21.480 INFO:journalctl@ceph.osd.3.smithi007.stdout:Sep 18 08:23:20 smithi007 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd-3[58803]: 2024-09-18T08:23:20.987+0000 7f45b25a4640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.3 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-09-18T08:23:21.480 INFO:journalctl@ceph.osd.3.smithi007.stdout:Sep 18 08:23:20 smithi007 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd-3[58803]: 2024-09-18T08:23:20.987+0000 7f45b25a4640 -1 osd.3 482 *** Got signal Terminated *** 2024-09-18T08:23:21.480 INFO:journalctl@ceph.osd.3.smithi007.stdout:Sep 18 08:23:20 smithi007 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd-3[58803]: 2024-09-18T08:23:20.987+0000 7f45b25a4640 -1 osd.3 482 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-09-18T08:23:22.905 INFO:journalctl@ceph.osd.6.smithi050.stdout:Sep 18 08:23:22 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd-6[47568]: 2024-09-18T08:23:22.623+0000 7f0a56b1c640 -1 osd.6 482 heartbeat_check: no reply from 172.21.15.7:6803 osd.0 since back 2024-09-18T08:22:56.826928+0000 front 2024-09-18T08:22:56.827014+0000 (oldest deadline 2024-09-18T08:23:22.126870+0000) 2024-09-18T08:23:22.905 INFO:journalctl@ceph.osd.7.smithi050.stdout:Sep 18 08:23:22 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd-7[53264]: 2024-09-18T08:23:22.729+0000 7f77c4ebf640 -1 osd.7 482 heartbeat_check: no reply from 172.21.15.7:6803 osd.0 since back 2024-09-18T08:22:56.257609+0000 front 2024-09-18T08:22:56.257592+0000 (oldest deadline 2024-09-18T08:23:22.157443+0000) 2024-09-18T08:23:23.229 INFO:journalctl@ceph.osd.3.smithi007.stdout:Sep 18 08:23:22 smithi007 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd-3[58803]: 2024-09-18T08:23:22.960+0000 7f45aebb9640 -1 osd.3 482 heartbeat_check: no reply from 172.21.15.7:6803 osd.0 since back 2024-09-18T08:22:58.237647+0000 front 2024-09-18T08:22:58.237672+0000 (oldest deadline 2024-09-18T08:23:22.937809+0000) 2024-09-18T08:23:23.405 INFO:journalctl@ceph.osd.5.smithi050.stdout:Sep 18 08:23:22 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd-5[42249]: 2024-09-18T08:23:22.985+0000 7fc155e15640 -1 osd.5 482 heartbeat_check: no reply from 172.21.15.7:6803 osd.0 since back 2024-09-18T08:22:57.662189+0000 front 2024-09-18T08:22:57.662202+0000 (oldest deadline 2024-09-18T08:23:22.362113+0000) 2024-09-18T08:23:23.905 INFO:journalctl@ceph.osd.4.smithi050.stdout:Sep 18 08:23:23 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd-4[36561]: 2024-09-18T08:23:23.594+0000 7f875afde640 -1 osd.4 482 heartbeat_check: no reply from 172.21.15.7:6803 osd.0 since back 2024-09-18T08:22:57.342374+0000 front 2024-09-18T08:22:57.342511+0000 (oldest deadline 2024-09-18T08:23:23.242386+0000) 2024-09-18T08:23:23.905 INFO:journalctl@ceph.osd.7.smithi050.stdout:Sep 18 08:23:23 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd-7[53264]: 2024-09-18T08:23:23.736+0000 7f77c4ebf640 -1 osd.7 482 heartbeat_check: no reply from 172.21.15.7:6803 osd.0 since back 2024-09-18T08:22:56.257609+0000 front 2024-09-18T08:22:56.257592+0000 (oldest deadline 2024-09-18T08:23:22.157443+0000) 2024-09-18T08:23:23.905 INFO:journalctl@ceph.osd.6.smithi050.stdout:Sep 18 08:23:23 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd-6[47568]: 2024-09-18T08:23:23.644+0000 7f0a56b1c640 -1 osd.6 482 heartbeat_check: no reply from 172.21.15.7:6803 osd.0 since back 2024-09-18T08:22:56.826928+0000 front 2024-09-18T08:22:56.827014+0000 (oldest deadline 2024-09-18T08:23:22.126870+0000) 2024-09-18T08:23:24.229 INFO:journalctl@ceph.osd.3.smithi007.stdout:Sep 18 08:23:23 smithi007 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd-3[58803]: 2024-09-18T08:23:23.959+0000 7f45aebb9640 -1 osd.3 482 heartbeat_check: no reply from 172.21.15.7:6803 osd.0 since back 2024-09-18T08:22:58.237647+0000 front 2024-09-18T08:22:58.237672+0000 (oldest deadline 2024-09-18T08:23:22.937809+0000) 2024-09-18T08:23:24.405 INFO:journalctl@ceph.osd.5.smithi050.stdout:Sep 18 08:23:23 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd-5[42249]: 2024-09-18T08:23:23.990+0000 7fc155e15640 -1 osd.5 482 heartbeat_check: no reply from 172.21.15.7:6803 osd.0 since back 2024-09-18T08:22:57.662189+0000 front 2024-09-18T08:22:57.662202+0000 (oldest deadline 2024-09-18T08:23:22.362113+0000) 2024-09-18T08:23:24.905 INFO:journalctl@ceph.osd.7.smithi050.stdout:Sep 18 08:23:24 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd-7[53264]: 2024-09-18T08:23:24.759+0000 7f77c4ebf640 -1 osd.7 482 heartbeat_check: no reply from 172.21.15.7:6803 osd.0 since back 2024-09-18T08:22:56.257609+0000 front 2024-09-18T08:22:56.257592+0000 (oldest deadline 2024-09-18T08:23:22.157443+0000) 2024-09-18T08:23:24.905 INFO:journalctl@ceph.osd.6.smithi050.stdout:Sep 18 08:23:24 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd-6[47568]: 2024-09-18T08:23:24.675+0000 7f0a56b1c640 -1 osd.6 482 heartbeat_check: no reply from 172.21.15.7:6803 osd.0 since back 2024-09-18T08:22:56.826928+0000 front 2024-09-18T08:22:56.827014+0000 (oldest deadline 2024-09-18T08:23:22.126870+0000) 2024-09-18T08:23:24.905 INFO:journalctl@ceph.osd.4.smithi050.stdout:Sep 18 08:23:24 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd-4[36561]: 2024-09-18T08:23:24.636+0000 7f875afde640 -1 osd.4 482 heartbeat_check: no reply from 172.21.15.7:6803 osd.0 since back 2024-09-18T08:22:57.342374+0000 front 2024-09-18T08:22:57.342511+0000 (oldest deadline 2024-09-18T08:23:23.242386+0000) 2024-09-18T08:23:25.229 INFO:journalctl@ceph.osd.3.smithi007.stdout:Sep 18 08:23:24 smithi007 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd-3[58803]: 2024-09-18T08:23:24.922+0000 7f45aebb9640 -1 osd.3 482 heartbeat_check: no reply from 172.21.15.7:6803 osd.0 since back 2024-09-18T08:22:58.237647+0000 front 2024-09-18T08:22:58.237672+0000 (oldest deadline 2024-09-18T08:23:22.937809+0000) 2024-09-18T08:23:25.405 INFO:journalctl@ceph.osd.5.smithi050.stdout:Sep 18 08:23:25 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd-5[42249]: 2024-09-18T08:23:25.004+0000 7fc155e15640 -1 osd.5 482 heartbeat_check: no reply from 172.21.15.7:6803 osd.0 since back 2024-09-18T08:22:57.662189+0000 front 2024-09-18T08:22:57.662202+0000 (oldest deadline 2024-09-18T08:23:22.362113+0000) 2024-09-18T08:23:25.905 INFO:journalctl@ceph.osd.6.smithi050.stdout:Sep 18 08:23:25 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd-6[47568]: 2024-09-18T08:23:25.723+0000 7f0a56b1c640 -1 osd.6 482 heartbeat_check: no reply from 172.21.15.7:6803 osd.0 since back 2024-09-18T08:22:56.826928+0000 front 2024-09-18T08:22:56.827014+0000 (oldest deadline 2024-09-18T08:23:22.126870+0000) 2024-09-18T08:23:25.905 INFO:journalctl@ceph.osd.7.smithi050.stdout:Sep 18 08:23:25 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd-7[53264]: 2024-09-18T08:23:25.739+0000 7f77c4ebf640 -1 osd.7 482 heartbeat_check: no reply from 172.21.15.7:6803 osd.0 since back 2024-09-18T08:22:56.257609+0000 front 2024-09-18T08:22:56.257592+0000 (oldest deadline 2024-09-18T08:23:22.157443+0000) 2024-09-18T08:23:25.905 INFO:journalctl@ceph.osd.4.smithi050.stdout:Sep 18 08:23:25 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd-4[36561]: 2024-09-18T08:23:25.650+0000 7f875afde640 -1 osd.4 482 heartbeat_check: no reply from 172.21.15.7:6803 osd.0 since back 2024-09-18T08:22:57.342374+0000 front 2024-09-18T08:22:57.342511+0000 (oldest deadline 2024-09-18T08:23:23.242386+0000) 2024-09-18T08:23:26.229 INFO:journalctl@ceph.osd.3.smithi007.stdout:Sep 18 08:23:25 smithi007 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd-3[58803]: 2024-09-18T08:23:25.921+0000 7f45aebb9640 -1 osd.3 482 heartbeat_check: no reply from 172.21.15.7:6803 osd.0 since back 2024-09-18T08:22:58.237647+0000 front 2024-09-18T08:22:58.237672+0000 (oldest deadline 2024-09-18T08:23:22.937809+0000) 2024-09-18T08:23:26.405 INFO:journalctl@ceph.osd.5.smithi050.stdout:Sep 18 08:23:26 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd-5[42249]: 2024-09-18T08:23:26.017+0000 7fc155e15640 -1 osd.5 482 heartbeat_check: no reply from 172.21.15.7:6803 osd.0 since back 2024-09-18T08:22:57.662189+0000 front 2024-09-18T08:22:57.662202+0000 (oldest deadline 2024-09-18T08:23:22.362113+0000) 2024-09-18T08:23:26.729 INFO:journalctl@ceph.osd.3.smithi007.stdout:Sep 18 08:23:26 smithi007 podman[196353]: 2024-09-18 08:23:26.455041927 +0000 UTC m=+5.588628477 container died a238d69dd8d206341cf724f56fd551d8146641b86d15dcf7492b470d8a98f1b4 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:1f5b5a3734f665ef333266f39143ac3ef1eb1951b21258f2307e10332e2d7ac9, name=ceph-779349ca-7593-11ef-bceb-c7b262605968-osd-3, org.label-schema.build-date=20240731, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, GIT_CLEAN=True, ceph=True, org.label-schema.schema-version=1.0, io.buildah.version=1.37.2, CEPH_POINT_RELEASE=, RELEASE=wip-guits-main-2024-09-17-1213-e7fb7b5, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 9 Base Image, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS) 2024-09-18T08:23:26.905 INFO:journalctl@ceph.osd.4.smithi050.stdout:Sep 18 08:23:26 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd-4[36561]: 2024-09-18T08:23:26.624+0000 7f875afde640 -1 osd.4 482 heartbeat_check: no reply from 172.21.15.7:6803 osd.0 since back 2024-09-18T08:22:57.342374+0000 front 2024-09-18T08:22:57.342511+0000 (oldest deadline 2024-09-18T08:23:23.242386+0000) 2024-09-18T08:23:26.905 INFO:journalctl@ceph.osd.6.smithi050.stdout:Sep 18 08:23:26 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd-6[47568]: 2024-09-18T08:23:26.759+0000 7f0a56b1c640 -1 osd.6 482 heartbeat_check: no reply from 172.21.15.7:6803 osd.0 since back 2024-09-18T08:22:56.826928+0000 front 2024-09-18T08:22:56.827014+0000 (oldest deadline 2024-09-18T08:23:22.126870+0000) 2024-09-18T08:23:26.905 INFO:journalctl@ceph.osd.7.smithi050.stdout:Sep 18 08:23:26 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd-7[53264]: 2024-09-18T08:23:26.725+0000 7f77c4ebf640 -1 osd.7 482 heartbeat_check: no reply from 172.21.15.7:6803 osd.0 since back 2024-09-18T08:22:56.257609+0000 front 2024-09-18T08:22:56.257592+0000 (oldest deadline 2024-09-18T08:23:22.157443+0000) 2024-09-18T08:23:27.105 INFO:journalctl@ceph.osd.3.smithi007.stdout:Sep 18 08:23:26 smithi007 podman[196353]: 2024-09-18 08:23:26.8949024 +0000 UTC m=+6.028488986 container cleanup a238d69dd8d206341cf724f56fd551d8146641b86d15dcf7492b470d8a98f1b4 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:1f5b5a3734f665ef333266f39143ac3ef1eb1951b21258f2307e10332e2d7ac9, name=ceph-779349ca-7593-11ef-bceb-c7b262605968-osd-3, org.label-schema.build-date=20240731, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 9 Base Image, GIT_CLEAN=True, GIT_BRANCH=HEAD, ceph=True, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, RELEASE=wip-guits-main-2024-09-17-1213-e7fb7b5, io.buildah.version=1.37.2) 2024-09-18T08:23:27.105 INFO:journalctl@ceph.osd.3.smithi007.stdout:Sep 18 08:23:26 smithi007 bash[196353]: ceph-779349ca-7593-11ef-bceb-c7b262605968-osd-3 2024-09-18T08:23:27.405 INFO:journalctl@ceph.osd.5.smithi050.stdout:Sep 18 08:23:27 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd-5[42249]: 2024-09-18T08:23:27.036+0000 7fc155e15640 -1 osd.5 482 heartbeat_check: no reply from 172.21.15.7:6803 osd.0 since back 2024-09-18T08:22:57.662189+0000 front 2024-09-18T08:22:57.662202+0000 (oldest deadline 2024-09-18T08:23:22.362113+0000) 2024-09-18T08:23:27.480 INFO:journalctl@ceph.osd.3.smithi007.stdout:Sep 18 08:23:27 smithi007 podman[196367]: 2024-09-18 08:23:27.104956433 +0000 UTC m=+0.647207129 container remove a238d69dd8d206341cf724f56fd551d8146641b86d15dcf7492b470d8a98f1b4 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:1f5b5a3734f665ef333266f39143ac3ef1eb1951b21258f2307e10332e2d7ac9, name=ceph-779349ca-7593-11ef-bceb-c7b262605968-osd-3, io.buildah.version=1.37.2, org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.build-date=20240731, org.label-schema.license=GPLv2, RELEASE=wip-guits-main-2024-09-17-1213-e7fb7b5, org.label-schema.name=CentOS Stream 9 Base Image, ceph=True, org.label-schema.vendor=CentOS, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux ) 2024-09-18T08:23:27.791 INFO:journalctl@ceph.osd.3.smithi007.stdout:Sep 18 08:23:27 smithi007 podman[196459]: 2024-09-18 08:23:27.421514716 +0000 UTC m=+0.016462812 image pull b22e1e37ddf1ca6e22136cd6540a9ab61e7cafb8bfbc7a91a2f20f7f9941bd9a quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:1f5b5a3734f665ef333266f39143ac3ef1eb1951b21258f2307e10332e2d7ac9 2024-09-18T08:23:27.792 INFO:journalctl@ceph.osd.3.smithi007.stdout:Sep 18 08:23:27 smithi007 podman[196459]: 2024-09-18 08:23:27.530679603 +0000 UTC m=+0.125627679 container create 6597e059e12ca06a79fe147ded37be42f27b727f21c3a2a6f07205c437f5f916 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:1f5b5a3734f665ef333266f39143ac3ef1eb1951b21258f2307e10332e2d7ac9, name=ceph-779349ca-7593-11ef-bceb-c7b262605968-osd-3-deactivate, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, ceph=True, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 9 Base Image, GIT_BRANCH=HEAD, RELEASE=wip-guits-main-2024-09-17-1213-e7fb7b5, org.label-schema.build-date=20240731, io.buildah.version=1.37.2, GIT_CLEAN=True, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965) 2024-09-18T08:23:27.792 INFO:journalctl@ceph.osd.3.smithi007.stdout:Sep 18 08:23:27 smithi007 podman[196459]: 2024-09-18 08:23:27.791337344 +0000 UTC m=+0.386285441 container init 6597e059e12ca06a79fe147ded37be42f27b727f21c3a2a6f07205c437f5f916 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:1f5b5a3734f665ef333266f39143ac3ef1eb1951b21258f2307e10332e2d7ac9, name=ceph-779349ca-7593-11ef-bceb-c7b262605968-osd-3-deactivate, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, RELEASE=wip-guits-main-2024-09-17-1213-e7fb7b5, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.37.2, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240731, org.label-schema.schema-version=1.0, ceph=True, org.label-schema.name=CentOS Stream 9 Base Image, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.license=GPLv2) 2024-09-18T08:23:27.905 INFO:journalctl@ceph.osd.6.smithi050.stdout:Sep 18 08:23:27 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd-6[47568]: 2024-09-18T08:23:27.748+0000 7f0a56b1c640 -1 osd.6 482 heartbeat_check: no reply from 172.21.15.7:6803 osd.0 since back 2024-09-18T08:22:56.826928+0000 front 2024-09-18T08:22:56.827014+0000 (oldest deadline 2024-09-18T08:23:22.126870+0000) 2024-09-18T08:23:27.905 INFO:journalctl@ceph.osd.7.smithi050.stdout:Sep 18 08:23:27 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd-7[53264]: 2024-09-18T08:23:27.727+0000 7f77c4ebf640 -1 osd.7 482 heartbeat_check: no reply from 172.21.15.7:6803 osd.0 since back 2024-09-18T08:22:56.257609+0000 front 2024-09-18T08:22:56.257592+0000 (oldest deadline 2024-09-18T08:23:22.157443+0000) 2024-09-18T08:23:27.906 INFO:journalctl@ceph.osd.4.smithi050.stdout:Sep 18 08:23:27 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd-4[36561]: 2024-09-18T08:23:27.638+0000 7f875afde640 -1 osd.4 482 heartbeat_check: no reply from 172.21.15.7:6803 osd.0 since back 2024-09-18T08:22:57.342374+0000 front 2024-09-18T08:22:57.342511+0000 (oldest deadline 2024-09-18T08:23:23.242386+0000) 2024-09-18T08:23:28.229 INFO:journalctl@ceph.osd.3.smithi007.stdout:Sep 18 08:23:27 smithi007 podman[196459]: 2024-09-18 08:23:27.794897308 +0000 UTC m=+0.389845392 container start 6597e059e12ca06a79fe147ded37be42f27b727f21c3a2a6f07205c437f5f916 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:1f5b5a3734f665ef333266f39143ac3ef1eb1951b21258f2307e10332e2d7ac9, name=ceph-779349ca-7593-11ef-bceb-c7b262605968-osd-3-deactivate, org.label-schema.build-date=20240731, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 9 Base Image, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_CLEAN=True, ceph=True, CEPH_POINT_RELEASE=, io.buildah.version=1.37.2, RELEASE=wip-guits-main-2024-09-17-1213-e7fb7b5, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, org.label-schema.schema-version=1.0) 2024-09-18T08:23:28.230 INFO:journalctl@ceph.osd.3.smithi007.stdout:Sep 18 08:23:27 smithi007 podman[196459]: 2024-09-18 08:23:27.874488912 +0000 UTC m=+0.469436993 container attach 6597e059e12ca06a79fe147ded37be42f27b727f21c3a2a6f07205c437f5f916 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:1f5b5a3734f665ef333266f39143ac3ef1eb1951b21258f2307e10332e2d7ac9, name=ceph-779349ca-7593-11ef-bceb-c7b262605968-osd-3-deactivate, GIT_BRANCH=HEAD, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, org.label-schema.build-date=20240731, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.vendor=CentOS, GIT_CLEAN=True, RELEASE=wip-guits-main-2024-09-17-1213-e7fb7b5, org.label-schema.schema-version=1.0, io.buildah.version=1.37.2) 2024-09-18T08:23:28.230 INFO:journalctl@ceph.osd.3.smithi007.stdout:Sep 18 08:23:27 smithi007 podman[196459]: 2024-09-18 08:23:27.986430042 +0000 UTC m=+0.581378128 container died 6597e059e12ca06a79fe147ded37be42f27b727f21c3a2a6f07205c437f5f916 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:1f5b5a3734f665ef333266f39143ac3ef1eb1951b21258f2307e10332e2d7ac9, name=ceph-779349ca-7593-11ef-bceb-c7b262605968-osd-3-deactivate, org.label-schema.build-date=20240731, org.label-schema.name=CentOS Stream 9 Base Image, maintainer=Guillaume Abrioux , RELEASE=wip-guits-main-2024-09-17-1213-e7fb7b5, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, io.buildah.version=1.37.2, org.label-schema.vendor=CentOS, GIT_CLEAN=True, GIT_BRANCH=HEAD, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True) 2024-09-18T08:23:28.405 INFO:journalctl@ceph.osd.5.smithi050.stdout:Sep 18 08:23:28 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd-5[42249]: 2024-09-18T08:23:28.080+0000 7fc155e15640 -1 osd.5 482 heartbeat_check: no reply from 172.21.15.7:6803 osd.0 since back 2024-09-18T08:22:57.662189+0000 front 2024-09-18T08:22:57.662202+0000 (oldest deadline 2024-09-18T08:23:22.362113+0000) 2024-09-18T08:23:28.661 DEBUG:teuthology.orchestra.run.smithi007:> sudo pkill -f 'journalctl -f -n 0 -u ceph-779349ca-7593-11ef-bceb-c7b262605968@osd.3.service' 2024-09-18T08:23:28.708 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-09-18T08:23:28.708 INFO:tasks.cephadm.osd.3:Stopped osd.3 2024-09-18T08:23:28.708 INFO:tasks.cephadm.osd.4:Stopping osd.4... 2024-09-18T08:23:28.708 DEBUG:teuthology.orchestra.run.smithi050:> sudo systemctl stop ceph-779349ca-7593-11ef-bceb-c7b262605968@osd.4 2024-09-18T08:23:28.905 INFO:journalctl@ceph.osd.7.smithi050.stdout:Sep 18 08:23:28 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd-7[53264]: 2024-09-18T08:23:28.720+0000 7f77c4ebf640 -1 osd.7 482 heartbeat_check: no reply from 172.21.15.7:6803 osd.0 since back 2024-09-18T08:22:56.257609+0000 front 2024-09-18T08:22:56.257592+0000 (oldest deadline 2024-09-18T08:23:22.157443+0000) 2024-09-18T08:23:28.905 INFO:journalctl@ceph.osd.4.smithi050.stdout:Sep 18 08:23:28 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd-4[36561]: 2024-09-18T08:23:28.633+0000 7f875afde640 -1 osd.4 482 heartbeat_check: no reply from 172.21.15.7:6803 osd.0 since back 2024-09-18T08:22:57.342374+0000 front 2024-09-18T08:22:57.342511+0000 (oldest deadline 2024-09-18T08:23:23.242386+0000) 2024-09-18T08:23:28.905 INFO:journalctl@ceph.osd.4.smithi050.stdout:Sep 18 08:23:28 smithi050 systemd[1]: Stopping Ceph osd.4 for 779349ca-7593-11ef-bceb-c7b262605968... 2024-09-18T08:23:28.906 INFO:journalctl@ceph.osd.6.smithi050.stdout:Sep 18 08:23:28 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd-6[47568]: 2024-09-18T08:23:28.713+0000 7f0a56b1c640 -1 osd.6 482 heartbeat_check: no reply from 172.21.15.7:6803 osd.0 since back 2024-09-18T08:22:56.826928+0000 front 2024-09-18T08:22:56.827014+0000 (oldest deadline 2024-09-18T08:23:22.126870+0000) 2024-09-18T08:23:29.405 INFO:journalctl@ceph.osd.5.smithi050.stdout:Sep 18 08:23:29 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd-5[42249]: 2024-09-18T08:23:29.072+0000 7fc155e15640 -1 osd.5 482 heartbeat_check: no reply from 172.21.15.7:6803 osd.0 since back 2024-09-18T08:22:57.662189+0000 front 2024-09-18T08:22:57.662202+0000 (oldest deadline 2024-09-18T08:23:22.362113+0000) 2024-09-18T08:23:29.405 INFO:journalctl@ceph.osd.4.smithi050.stdout:Sep 18 08:23:29 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd-4[36561]: 2024-09-18T08:23:29.015+0000 7f875e9c9640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.4 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-09-18T08:23:29.405 INFO:journalctl@ceph.osd.4.smithi050.stdout:Sep 18 08:23:29 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd-4[36561]: 2024-09-18T08:23:29.015+0000 7f875e9c9640 -1 osd.4 482 *** Got signal Terminated *** 2024-09-18T08:23:29.405 INFO:journalctl@ceph.osd.4.smithi050.stdout:Sep 18 08:23:29 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd-4[36561]: 2024-09-18T08:23:29.015+0000 7f875e9c9640 -1 osd.4 482 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-09-18T08:23:30.053 INFO:journalctl@ceph.osd.4.smithi050.stdout:Sep 18 08:23:29 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd-4[36561]: 2024-09-18T08:23:29.677+0000 7f875afde640 -1 osd.4 482 heartbeat_check: no reply from 172.21.15.7:6803 osd.0 since back 2024-09-18T08:22:57.342374+0000 front 2024-09-18T08:22:57.342511+0000 (oldest deadline 2024-09-18T08:23:23.242386+0000) 2024-09-18T08:23:30.054 INFO:journalctl@ceph.osd.7.smithi050.stdout:Sep 18 08:23:29 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd-7[53264]: 2024-09-18T08:23:29.728+0000 7f77c4ebf640 -1 osd.7 482 heartbeat_check: no reply from 172.21.15.7:6803 osd.0 since back 2024-09-18T08:22:56.257609+0000 front 2024-09-18T08:22:56.257592+0000 (oldest deadline 2024-09-18T08:23:22.157443+0000) 2024-09-18T08:23:30.055 INFO:journalctl@ceph.osd.6.smithi050.stdout:Sep 18 08:23:29 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd-6[47568]: 2024-09-18T08:23:29.762+0000 7f0a56b1c640 -1 osd.6 482 heartbeat_check: no reply from 172.21.15.7:6803 osd.0 since back 2024-09-18T08:22:56.826928+0000 front 2024-09-18T08:22:56.827014+0000 (oldest deadline 2024-09-18T08:23:22.126870+0000) 2024-09-18T08:23:30.405 INFO:journalctl@ceph.osd.5.smithi050.stdout:Sep 18 08:23:30 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd-5[42249]: 2024-09-18T08:23:30.052+0000 7fc155e15640 -1 osd.5 482 heartbeat_check: no reply from 172.21.15.7:6803 osd.0 since back 2024-09-18T08:22:57.662189+0000 front 2024-09-18T08:22:57.662202+0000 (oldest deadline 2024-09-18T08:23:22.362113+0000) 2024-09-18T08:23:31.056 INFO:journalctl@ceph.osd.4.smithi050.stdout:Sep 18 08:23:30 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd-4[36561]: 2024-09-18T08:23:30.715+0000 7f875afde640 -1 osd.4 482 heartbeat_check: no reply from 172.21.15.7:6803 osd.0 since back 2024-09-18T08:22:57.342374+0000 front 2024-09-18T08:22:57.342511+0000 (oldest deadline 2024-09-18T08:23:23.242386+0000) 2024-09-18T08:23:31.057 INFO:journalctl@ceph.osd.7.smithi050.stdout:Sep 18 08:23:30 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd-7[53264]: 2024-09-18T08:23:30.693+0000 7f77c4ebf640 -1 osd.7 482 heartbeat_check: no reply from 172.21.15.7:6803 osd.0 since back 2024-09-18T08:22:56.257609+0000 front 2024-09-18T08:22:56.257592+0000 (oldest deadline 2024-09-18T08:23:22.157443+0000) 2024-09-18T08:23:31.057 INFO:journalctl@ceph.osd.6.smithi050.stdout:Sep 18 08:23:30 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd-6[47568]: 2024-09-18T08:23:30.760+0000 7f0a56b1c640 -1 osd.6 482 heartbeat_check: no reply from 172.21.15.7:6803 osd.0 since back 2024-09-18T08:22:56.826928+0000 front 2024-09-18T08:22:56.827014+0000 (oldest deadline 2024-09-18T08:23:22.126870+0000) 2024-09-18T08:23:31.405 INFO:journalctl@ceph.osd.5.smithi050.stdout:Sep 18 08:23:31 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd-5[42249]: 2024-09-18T08:23:31.055+0000 7fc155e15640 -1 osd.5 482 heartbeat_check: no reply from 172.21.15.7:6803 osd.0 since back 2024-09-18T08:22:57.662189+0000 front 2024-09-18T08:22:57.662202+0000 (oldest deadline 2024-09-18T08:23:22.362113+0000) 2024-09-18T08:23:32.074 INFO:journalctl@ceph.osd.4.smithi050.stdout:Sep 18 08:23:31 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd-4[36561]: 2024-09-18T08:23:31.690+0000 7f875afde640 -1 osd.4 482 heartbeat_check: no reply from 172.21.15.7:6803 osd.0 since back 2024-09-18T08:22:57.342374+0000 front 2024-09-18T08:22:57.342511+0000 (oldest deadline 2024-09-18T08:23:23.242386+0000) 2024-09-18T08:23:32.075 INFO:journalctl@ceph.osd.7.smithi050.stdout:Sep 18 08:23:31 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd-7[53264]: 2024-09-18T08:23:31.659+0000 7f77c4ebf640 -1 osd.7 482 heartbeat_check: no reply from 172.21.15.7:6803 osd.0 since back 2024-09-18T08:22:56.257609+0000 front 2024-09-18T08:22:56.257592+0000 (oldest deadline 2024-09-18T08:23:22.157443+0000) 2024-09-18T08:23:32.075 INFO:journalctl@ceph.osd.6.smithi050.stdout:Sep 18 08:23:31 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd-6[47568]: 2024-09-18T08:23:31.721+0000 7f0a56b1c640 -1 osd.6 482 heartbeat_check: no reply from 172.21.15.7:6803 osd.0 since back 2024-09-18T08:22:56.826928+0000 front 2024-09-18T08:22:56.827014+0000 (oldest deadline 2024-09-18T08:23:22.126870+0000) 2024-09-18T08:23:32.405 INFO:journalctl@ceph.osd.5.smithi050.stdout:Sep 18 08:23:32 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd-5[42249]: 2024-09-18T08:23:32.073+0000 7fc155e15640 -1 osd.5 482 heartbeat_check: no reply from 172.21.15.7:6803 osd.0 since back 2024-09-18T08:22:57.662189+0000 front 2024-09-18T08:22:57.662202+0000 (oldest deadline 2024-09-18T08:23:22.362113+0000) 2024-09-18T08:23:32.405 INFO:journalctl@ceph.osd.5.smithi050.stdout:Sep 18 08:23:32 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd-5[42249]: 2024-09-18T08:23:32.073+0000 7fc155e15640 -1 osd.5 482 heartbeat_check: no reply from 172.21.15.7:6807 osd.1 since back 2024-09-18T08:23:05.163375+0000 front 2024-09-18T08:23:05.163537+0000 (oldest deadline 2024-09-18T08:23:31.563961+0000) 2024-09-18T08:23:32.905 INFO:journalctl@ceph.osd.4.smithi050.stdout:Sep 18 08:23:32 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd-4[36561]: 2024-09-18T08:23:32.679+0000 7f875afde640 -1 osd.4 482 heartbeat_check: no reply from 172.21.15.7:6803 osd.0 since back 2024-09-18T08:22:57.342374+0000 front 2024-09-18T08:22:57.342511+0000 (oldest deadline 2024-09-18T08:23:23.242386+0000) 2024-09-18T08:23:32.905 INFO:journalctl@ceph.osd.6.smithi050.stdout:Sep 18 08:23:32 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd-6[47568]: 2024-09-18T08:23:32.681+0000 7f0a56b1c640 -1 osd.6 482 heartbeat_check: no reply from 172.21.15.7:6803 osd.0 since back 2024-09-18T08:22:56.826928+0000 front 2024-09-18T08:22:56.827014+0000 (oldest deadline 2024-09-18T08:23:22.126870+0000) 2024-09-18T08:23:32.905 INFO:journalctl@ceph.osd.6.smithi050.stdout:Sep 18 08:23:32 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd-6[47568]: 2024-09-18T08:23:32.681+0000 7f0a56b1c640 -1 osd.6 482 heartbeat_check: no reply from 172.21.15.7:6807 osd.1 since back 2024-09-18T08:23:06.227974+0000 front 2024-09-18T08:23:06.228037+0000 (oldest deadline 2024-09-18T08:23:32.127680+0000) 2024-09-18T08:23:32.906 INFO:journalctl@ceph.osd.7.smithi050.stdout:Sep 18 08:23:32 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd-7[53264]: 2024-09-18T08:23:32.627+0000 7f77c4ebf640 -1 osd.7 482 heartbeat_check: no reply from 172.21.15.7:6803 osd.0 since back 2024-09-18T08:22:56.257609+0000 front 2024-09-18T08:22:56.257592+0000 (oldest deadline 2024-09-18T08:23:22.157443+0000) 2024-09-18T08:23:32.906 INFO:journalctl@ceph.osd.7.smithi050.stdout:Sep 18 08:23:32 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd-7[53264]: 2024-09-18T08:23:32.627+0000 7f77c4ebf640 -1 osd.7 482 heartbeat_check: no reply from 172.21.15.7:6807 osd.1 since back 2024-09-18T08:23:07.859263+0000 front 2024-09-18T08:23:07.859284+0000 (oldest deadline 2024-09-18T08:23:31.959140+0000) 2024-09-18T08:23:33.405 INFO:journalctl@ceph.osd.5.smithi050.stdout:Sep 18 08:23:33 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd-5[42249]: 2024-09-18T08:23:33.048+0000 7fc155e15640 -1 osd.5 482 heartbeat_check: no reply from 172.21.15.7:6803 osd.0 since back 2024-09-18T08:22:57.662189+0000 front 2024-09-18T08:22:57.662202+0000 (oldest deadline 2024-09-18T08:23:22.362113+0000) 2024-09-18T08:23:33.405 INFO:journalctl@ceph.osd.5.smithi050.stdout:Sep 18 08:23:33 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd-5[42249]: 2024-09-18T08:23:33.048+0000 7fc155e15640 -1 osd.5 482 heartbeat_check: no reply from 172.21.15.7:6807 osd.1 since back 2024-09-18T08:23:05.163375+0000 front 2024-09-18T08:23:05.163537+0000 (oldest deadline 2024-09-18T08:23:31.563961+0000) 2024-09-18T08:23:33.730 INFO:journalctl@ceph.osd.4.smithi050.stdout:Sep 18 08:23:33 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd-4[36561]: 2024-09-18T08:23:33.650+0000 7f875afde640 -1 osd.4 482 heartbeat_check: no reply from 172.21.15.7:6803 osd.0 since back 2024-09-18T08:22:57.342374+0000 front 2024-09-18T08:22:57.342511+0000 (oldest deadline 2024-09-18T08:23:23.242386+0000) 2024-09-18T08:23:33.730 INFO:journalctl@ceph.osd.7.smithi050.stdout:Sep 18 08:23:33 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd-7[53264]: 2024-09-18T08:23:33.656+0000 7f77c4ebf640 -1 osd.7 482 heartbeat_check: no reply from 172.21.15.7:6803 osd.0 since back 2024-09-18T08:22:56.257609+0000 front 2024-09-18T08:22:56.257592+0000 (oldest deadline 2024-09-18T08:23:22.157443+0000) 2024-09-18T08:23:33.730 INFO:journalctl@ceph.osd.7.smithi050.stdout:Sep 18 08:23:33 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd-7[53264]: 2024-09-18T08:23:33.656+0000 7f77c4ebf640 -1 osd.7 482 heartbeat_check: no reply from 172.21.15.7:6807 osd.1 since back 2024-09-18T08:23:07.859263+0000 front 2024-09-18T08:23:07.859284+0000 (oldest deadline 2024-09-18T08:23:31.959140+0000) 2024-09-18T08:23:34.045 INFO:journalctl@ceph.osd.6.smithi050.stdout:Sep 18 08:23:33 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd-6[47568]: 2024-09-18T08:23:33.729+0000 7f0a56b1c640 -1 osd.6 482 heartbeat_check: no reply from 172.21.15.7:6803 osd.0 since back 2024-09-18T08:22:56.826928+0000 front 2024-09-18T08:22:56.827014+0000 (oldest deadline 2024-09-18T08:23:22.126870+0000) 2024-09-18T08:23:34.045 INFO:journalctl@ceph.osd.6.smithi050.stdout:Sep 18 08:23:33 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd-6[47568]: 2024-09-18T08:23:33.729+0000 7f0a56b1c640 -1 osd.6 482 heartbeat_check: no reply from 172.21.15.7:6807 osd.1 since back 2024-09-18T08:23:06.227974+0000 front 2024-09-18T08:23:06.228037+0000 (oldest deadline 2024-09-18T08:23:32.127680+0000) 2024-09-18T08:23:34.405 INFO:journalctl@ceph.osd.5.smithi050.stdout:Sep 18 08:23:34 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd-5[42249]: 2024-09-18T08:23:34.046+0000 7fc155e15640 -1 osd.5 482 heartbeat_check: no reply from 172.21.15.7:6803 osd.0 since back 2024-09-18T08:22:57.662189+0000 front 2024-09-18T08:22:57.662202+0000 (oldest deadline 2024-09-18T08:23:22.362113+0000) 2024-09-18T08:23:34.405 INFO:journalctl@ceph.osd.5.smithi050.stdout:Sep 18 08:23:34 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd-5[42249]: 2024-09-18T08:23:34.046+0000 7fc155e15640 -1 osd.5 482 heartbeat_check: no reply from 172.21.15.7:6807 osd.1 since back 2024-09-18T08:23:05.163375+0000 front 2024-09-18T08:23:05.163537+0000 (oldest deadline 2024-09-18T08:23:31.563961+0000) 2024-09-18T08:23:34.905 INFO:journalctl@ceph.osd.4.smithi050.stdout:Sep 18 08:23:34 smithi050 podman[143295]: 2024-09-18 08:23:34.613161061 +0000 UTC m=+5.705894064 container died 97450023e365c9890ec1a21e5ee2345ac9f626aacd14090b4334aed78c526b18 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:1f5b5a3734f665ef333266f39143ac3ef1eb1951b21258f2307e10332e2d7ac9, name=ceph-779349ca-7593-11ef-bceb-c7b262605968-osd-4, org.label-schema.build-date=20240731, org.label-schema.vendor=CentOS, RELEASE=wip-guits-main-2024-09-17-1213-e7fb7b5, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, GIT_CLEAN=True, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 9 Base Image, io.buildah.version=1.37.2, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-09-18T08:23:34.905 INFO:journalctl@ceph.osd.6.smithi050.stdout:Sep 18 08:23:34 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd-6[47568]: 2024-09-18T08:23:34.753+0000 7f0a56b1c640 -1 osd.6 482 heartbeat_check: no reply from 172.21.15.7:6803 osd.0 since back 2024-09-18T08:22:56.826928+0000 front 2024-09-18T08:22:56.827014+0000 (oldest deadline 2024-09-18T08:23:22.126870+0000) 2024-09-18T08:23:34.905 INFO:journalctl@ceph.osd.6.smithi050.stdout:Sep 18 08:23:34 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd-6[47568]: 2024-09-18T08:23:34.753+0000 7f0a56b1c640 -1 osd.6 482 heartbeat_check: no reply from 172.21.15.7:6807 osd.1 since back 2024-09-18T08:23:06.227974+0000 front 2024-09-18T08:23:06.228037+0000 (oldest deadline 2024-09-18T08:23:32.127680+0000) 2024-09-18T08:23:34.906 INFO:journalctl@ceph.osd.7.smithi050.stdout:Sep 18 08:23:34 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd-7[53264]: 2024-09-18T08:23:34.637+0000 7f77c4ebf640 -1 osd.7 482 heartbeat_check: no reply from 172.21.15.7:6803 osd.0 since back 2024-09-18T08:22:56.257609+0000 front 2024-09-18T08:22:56.257592+0000 (oldest deadline 2024-09-18T08:23:22.157443+0000) 2024-09-18T08:23:34.906 INFO:journalctl@ceph.osd.7.smithi050.stdout:Sep 18 08:23:34 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd-7[53264]: 2024-09-18T08:23:34.637+0000 7f77c4ebf640 -1 osd.7 482 heartbeat_check: no reply from 172.21.15.7:6807 osd.1 since back 2024-09-18T08:23:07.859263+0000 front 2024-09-18T08:23:07.859284+0000 (oldest deadline 2024-09-18T08:23:31.959140+0000) 2024-09-18T08:23:35.368 INFO:journalctl@ceph.osd.5.smithi050.stdout:Sep 18 08:23:35 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd-5[42249]: 2024-09-18T08:23:35.092+0000 7fc155e15640 -1 osd.5 482 heartbeat_check: no reply from 172.21.15.7:6803 osd.0 since back 2024-09-18T08:22:57.662189+0000 front 2024-09-18T08:22:57.662202+0000 (oldest deadline 2024-09-18T08:23:22.362113+0000) 2024-09-18T08:23:35.368 INFO:journalctl@ceph.osd.5.smithi050.stdout:Sep 18 08:23:35 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd-5[42249]: 2024-09-18T08:23:35.092+0000 7fc155e15640 -1 osd.5 482 heartbeat_check: no reply from 172.21.15.7:6807 osd.1 since back 2024-09-18T08:23:05.163375+0000 front 2024-09-18T08:23:05.163537+0000 (oldest deadline 2024-09-18T08:23:31.563961+0000) 2024-09-18T08:23:35.369 INFO:journalctl@ceph.osd.4.smithi050.stdout:Sep 18 08:23:35 smithi050 podman[143295]: 2024-09-18 08:23:35.107851296 +0000 UTC m=+6.200584298 container cleanup 97450023e365c9890ec1a21e5ee2345ac9f626aacd14090b4334aed78c526b18 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:1f5b5a3734f665ef333266f39143ac3ef1eb1951b21258f2307e10332e2d7ac9, name=ceph-779349ca-7593-11ef-bceb-c7b262605968-osd-4, org.label-schema.license=GPLv2, ceph=True, org.label-schema.vendor=CentOS, GIT_CLEAN=True, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, io.buildah.version=1.37.2, org.label-schema.build-date=20240731, RELEASE=wip-guits-main-2024-09-17-1213-e7fb7b5, org.label-schema.name=CentOS Stream 9 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, maintainer=Guillaume Abrioux ) 2024-09-18T08:23:35.369 INFO:journalctl@ceph.osd.4.smithi050.stdout:Sep 18 08:23:35 smithi050 bash[143295]: ceph-779349ca-7593-11ef-bceb-c7b262605968-osd-4 2024-09-18T08:23:35.655 INFO:journalctl@ceph.osd.4.smithi050.stdout:Sep 18 08:23:35 smithi050 podman[143312]: 2024-09-18 08:23:35.367792668 +0000 UTC m=+0.752094238 container remove 97450023e365c9890ec1a21e5ee2345ac9f626aacd14090b4334aed78c526b18 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:1f5b5a3734f665ef333266f39143ac3ef1eb1951b21258f2307e10332e2d7ac9, name=ceph-779349ca-7593-11ef-bceb-c7b262605968-osd-4, GIT_CLEAN=True, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, maintainer=Guillaume Abrioux , ceph=True, io.buildah.version=1.37.2, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, org.label-schema.build-date=20240731, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 9 Base Image, RELEASE=wip-guits-main-2024-09-17-1213-e7fb7b5) 2024-09-18T08:23:35.988 INFO:journalctl@ceph.osd.4.smithi050.stdout:Sep 18 08:23:35 smithi050 podman[143403]: 2024-09-18 08:23:35.588319376 +0000 UTC m=+0.016489416 image pull b22e1e37ddf1ca6e22136cd6540a9ab61e7cafb8bfbc7a91a2f20f7f9941bd9a quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:1f5b5a3734f665ef333266f39143ac3ef1eb1951b21258f2307e10332e2d7ac9 2024-09-18T08:23:35.988 INFO:journalctl@ceph.osd.4.smithi050.stdout:Sep 18 08:23:35 smithi050 podman[143403]: 2024-09-18 08:23:35.718942415 +0000 UTC m=+0.147112443 container create 6481780dce67d87c59af63cf2cb7ecbab8b598acd55a4ea9c376ff343f2e04b5 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:1f5b5a3734f665ef333266f39143ac3ef1eb1951b21258f2307e10332e2d7ac9, name=ceph-779349ca-7593-11ef-bceb-c7b262605968-osd-4-deactivate, io.buildah.version=1.37.2, org.label-schema.vendor=CentOS, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, GIT_CLEAN=True, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.name=CentOS Stream 9 Base Image, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.build-date=20240731, RELEASE=wip-guits-main-2024-09-17-1213-e7fb7b5, org.label-schema.schema-version=1.0) 2024-09-18T08:23:35.988 INFO:journalctl@ceph.osd.6.smithi050.stdout:Sep 18 08:23:35 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd-6[47568]: 2024-09-18T08:23:35.763+0000 7f0a56b1c640 -1 osd.6 482 heartbeat_check: no reply from 172.21.15.7:6803 osd.0 since back 2024-09-18T08:22:56.826928+0000 front 2024-09-18T08:22:56.827014+0000 (oldest deadline 2024-09-18T08:23:22.126870+0000) 2024-09-18T08:23:35.988 INFO:journalctl@ceph.osd.6.smithi050.stdout:Sep 18 08:23:35 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd-6[47568]: 2024-09-18T08:23:35.763+0000 7f0a56b1c640 -1 osd.6 482 heartbeat_check: no reply from 172.21.15.7:6807 osd.1 since back 2024-09-18T08:23:06.227974+0000 front 2024-09-18T08:23:06.228037+0000 (oldest deadline 2024-09-18T08:23:32.127680+0000) 2024-09-18T08:23:35.989 INFO:journalctl@ceph.osd.7.smithi050.stdout:Sep 18 08:23:35 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd-7[53264]: 2024-09-18T08:23:35.681+0000 7f77c4ebf640 -1 osd.7 482 heartbeat_check: no reply from 172.21.15.7:6803 osd.0 since back 2024-09-18T08:22:56.257609+0000 front 2024-09-18T08:22:56.257592+0000 (oldest deadline 2024-09-18T08:23:22.157443+0000) 2024-09-18T08:23:35.989 INFO:journalctl@ceph.osd.7.smithi050.stdout:Sep 18 08:23:35 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd-7[53264]: 2024-09-18T08:23:35.681+0000 7f77c4ebf640 -1 osd.7 482 heartbeat_check: no reply from 172.21.15.7:6807 osd.1 since back 2024-09-18T08:23:07.859263+0000 front 2024-09-18T08:23:07.859284+0000 (oldest deadline 2024-09-18T08:23:31.959140+0000) 2024-09-18T08:23:36.347 INFO:journalctl@ceph.osd.5.smithi050.stdout:Sep 18 08:23:36 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd-5[42249]: 2024-09-18T08:23:36.100+0000 7fc155e15640 -1 osd.5 482 heartbeat_check: no reply from 172.21.15.7:6803 osd.0 since back 2024-09-18T08:22:57.662189+0000 front 2024-09-18T08:22:57.662202+0000 (oldest deadline 2024-09-18T08:23:22.362113+0000) 2024-09-18T08:23:36.348 INFO:journalctl@ceph.osd.5.smithi050.stdout:Sep 18 08:23:36 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd-5[42249]: 2024-09-18T08:23:36.100+0000 7fc155e15640 -1 osd.5 482 heartbeat_check: no reply from 172.21.15.7:6807 osd.1 since back 2024-09-18T08:23:05.163375+0000 front 2024-09-18T08:23:05.163537+0000 (oldest deadline 2024-09-18T08:23:31.563961+0000) 2024-09-18T08:23:36.348 INFO:journalctl@ceph.osd.4.smithi050.stdout:Sep 18 08:23:36 smithi050 podman[143403]: 2024-09-18 08:23:36.105517754 +0000 UTC m=+0.533687780 container init 6481780dce67d87c59af63cf2cb7ecbab8b598acd55a4ea9c376ff343f2e04b5 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:1f5b5a3734f665ef333266f39143ac3ef1eb1951b21258f2307e10332e2d7ac9, name=ceph-779349ca-7593-11ef-bceb-c7b262605968-osd-4-deactivate, GIT_CLEAN=True, org.label-schema.build-date=20240731, org.label-schema.name=CentOS Stream 9 Base Image, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, io.buildah.version=1.37.2, org.label-schema.vendor=CentOS, RELEASE=wip-guits-main-2024-09-17-1213-e7fb7b5, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965) 2024-09-18T08:23:36.348 INFO:journalctl@ceph.osd.4.smithi050.stdout:Sep 18 08:23:36 smithi050 podman[143403]: 2024-09-18 08:23:36.108875393 +0000 UTC m=+0.537045420 container start 6481780dce67d87c59af63cf2cb7ecbab8b598acd55a4ea9c376ff343f2e04b5 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:1f5b5a3734f665ef333266f39143ac3ef1eb1951b21258f2307e10332e2d7ac9, name=ceph-779349ca-7593-11ef-bceb-c7b262605968-osd-4-deactivate, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 9 Base Image, GIT_BRANCH=HEAD, GIT_CLEAN=True, RELEASE=wip-guits-main-2024-09-17-1213-e7fb7b5, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, io.buildah.version=1.37.2, org.label-schema.build-date=20240731, ceph=True) 2024-09-18T08:23:36.348 INFO:journalctl@ceph.osd.4.smithi050.stdout:Sep 18 08:23:36 smithi050 podman[143403]: 2024-09-18 08:23:36.188760219 +0000 UTC m=+0.616930253 container attach 6481780dce67d87c59af63cf2cb7ecbab8b598acd55a4ea9c376ff343f2e04b5 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:1f5b5a3734f665ef333266f39143ac3ef1eb1951b21258f2307e10332e2d7ac9, name=ceph-779349ca-7593-11ef-bceb-c7b262605968-osd-4-deactivate, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.schema-version=1.0, RELEASE=wip-guits-main-2024-09-17-1213-e7fb7b5, GIT_CLEAN=True, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, io.buildah.version=1.37.2, org.label-schema.build-date=20240731, org.label-schema.name=CentOS Stream 9 Base Image) 2024-09-18T08:23:36.624 INFO:journalctl@ceph.osd.4.smithi050.stdout:Sep 18 08:23:36 smithi050 podman[143403]: 2024-09-18 08:23:36.347649884 +0000 UTC m=+0.775819919 container died 6481780dce67d87c59af63cf2cb7ecbab8b598acd55a4ea9c376ff343f2e04b5 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:1f5b5a3734f665ef333266f39143ac3ef1eb1951b21258f2307e10332e2d7ac9, name=ceph-779349ca-7593-11ef-bceb-c7b262605968-osd-4-deactivate, org.label-schema.license=GPLv2, org.label-schema.build-date=20240731, GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 9 Base Image, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, io.buildah.version=1.37.2, maintainer=Guillaume Abrioux , ceph=True, RELEASE=wip-guits-main-2024-09-17-1213-e7fb7b5) 2024-09-18T08:23:36.905 INFO:journalctl@ceph.osd.7.smithi050.stdout:Sep 18 08:23:36 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd-7[53264]: 2024-09-18T08:23:36.632+0000 7f77c4ebf640 -1 osd.7 482 heartbeat_check: no reply from 172.21.15.7:6803 osd.0 since back 2024-09-18T08:22:56.257609+0000 front 2024-09-18T08:22:56.257592+0000 (oldest deadline 2024-09-18T08:23:22.157443+0000) 2024-09-18T08:23:36.905 INFO:journalctl@ceph.osd.7.smithi050.stdout:Sep 18 08:23:36 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd-7[53264]: 2024-09-18T08:23:36.632+0000 7f77c4ebf640 -1 osd.7 482 heartbeat_check: no reply from 172.21.15.7:6807 osd.1 since back 2024-09-18T08:23:07.859263+0000 front 2024-09-18T08:23:07.859284+0000 (oldest deadline 2024-09-18T08:23:31.959140+0000) 2024-09-18T08:23:36.905 INFO:journalctl@ceph.osd.6.smithi050.stdout:Sep 18 08:23:36 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd-6[47568]: 2024-09-18T08:23:36.738+0000 7f0a56b1c640 -1 osd.6 482 heartbeat_check: no reply from 172.21.15.7:6803 osd.0 since back 2024-09-18T08:22:56.826928+0000 front 2024-09-18T08:22:56.827014+0000 (oldest deadline 2024-09-18T08:23:22.126870+0000) 2024-09-18T08:23:36.905 INFO:journalctl@ceph.osd.6.smithi050.stdout:Sep 18 08:23:36 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd-6[47568]: 2024-09-18T08:23:36.738+0000 7f0a56b1c640 -1 osd.6 482 heartbeat_check: no reply from 172.21.15.7:6807 osd.1 since back 2024-09-18T08:23:06.227974+0000 front 2024-09-18T08:23:06.228037+0000 (oldest deadline 2024-09-18T08:23:32.127680+0000) 2024-09-18T08:23:36.968 DEBUG:teuthology.orchestra.run.smithi050:> sudo pkill -f 'journalctl -f -n 0 -u ceph-779349ca-7593-11ef-bceb-c7b262605968@osd.4.service' 2024-09-18T08:23:37.010 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-09-18T08:23:37.011 INFO:tasks.cephadm.osd.4:Stopped osd.4 2024-09-18T08:23:37.011 INFO:tasks.cephadm.osd.5:Stopping osd.5... 2024-09-18T08:23:37.011 DEBUG:teuthology.orchestra.run.smithi050:> sudo systemctl stop ceph-779349ca-7593-11ef-bceb-c7b262605968@osd.5 2024-09-18T08:23:37.364 INFO:journalctl@ceph.osd.5.smithi050.stdout:Sep 18 08:23:37 smithi050 systemd[1]: Stopping Ceph osd.5 for 779349ca-7593-11ef-bceb-c7b262605968... 2024-09-18T08:23:37.364 INFO:journalctl@ceph.osd.5.smithi050.stdout:Sep 18 08:23:37 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd-5[42249]: 2024-09-18T08:23:37.147+0000 7fc155e15640 -1 osd.5 482 heartbeat_check: no reply from 172.21.15.7:6803 osd.0 since back 2024-09-18T08:22:57.662189+0000 front 2024-09-18T08:22:57.662202+0000 (oldest deadline 2024-09-18T08:23:22.362113+0000) 2024-09-18T08:23:37.365 INFO:journalctl@ceph.osd.5.smithi050.stdout:Sep 18 08:23:37 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd-5[42249]: 2024-09-18T08:23:37.147+0000 7fc155e15640 -1 osd.5 482 heartbeat_check: no reply from 172.21.15.7:6807 osd.1 since back 2024-09-18T08:23:05.163375+0000 front 2024-09-18T08:23:05.163537+0000 (oldest deadline 2024-09-18T08:23:31.563961+0000) 2024-09-18T08:23:37.365 INFO:journalctl@ceph.osd.5.smithi050.stdout:Sep 18 08:23:37 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd-5[42249]: 2024-09-18T08:23:37.362+0000 7fc159800640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.5 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-09-18T08:23:37.365 INFO:journalctl@ceph.osd.5.smithi050.stdout:Sep 18 08:23:37 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd-5[42249]: 2024-09-18T08:23:37.362+0000 7fc159800640 -1 osd.5 482 *** Got signal Terminated *** 2024-09-18T08:23:37.365 INFO:journalctl@ceph.osd.5.smithi050.stdout:Sep 18 08:23:37 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd-5[42249]: 2024-09-18T08:23:37.362+0000 7fc159800640 -1 osd.5 482 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-09-18T08:23:37.905 INFO:journalctl@ceph.osd.6.smithi050.stdout:Sep 18 08:23:37 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd-6[47568]: 2024-09-18T08:23:37.689+0000 7f0a56b1c640 -1 osd.6 482 heartbeat_check: no reply from 172.21.15.7:6803 osd.0 since back 2024-09-18T08:22:56.826928+0000 front 2024-09-18T08:22:56.827014+0000 (oldest deadline 2024-09-18T08:23:22.126870+0000) 2024-09-18T08:23:37.905 INFO:journalctl@ceph.osd.6.smithi050.stdout:Sep 18 08:23:37 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd-6[47568]: 2024-09-18T08:23:37.689+0000 7f0a56b1c640 -1 osd.6 482 heartbeat_check: no reply from 172.21.15.7:6807 osd.1 since back 2024-09-18T08:23:06.227974+0000 front 2024-09-18T08:23:06.228037+0000 (oldest deadline 2024-09-18T08:23:32.127680+0000) 2024-09-18T08:23:37.905 INFO:journalctl@ceph.osd.7.smithi050.stdout:Sep 18 08:23:37 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd-7[53264]: 2024-09-18T08:23:37.625+0000 7f77c4ebf640 -1 osd.7 482 heartbeat_check: no reply from 172.21.15.7:6803 osd.0 since back 2024-09-18T08:22:56.257609+0000 front 2024-09-18T08:22:56.257592+0000 (oldest deadline 2024-09-18T08:23:22.157443+0000) 2024-09-18T08:23:37.905 INFO:journalctl@ceph.osd.7.smithi050.stdout:Sep 18 08:23:37 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd-7[53264]: 2024-09-18T08:23:37.625+0000 7f77c4ebf640 -1 osd.7 482 heartbeat_check: no reply from 172.21.15.7:6807 osd.1 since back 2024-09-18T08:23:07.859263+0000 front 2024-09-18T08:23:07.859284+0000 (oldest deadline 2024-09-18T08:23:31.959140+0000) 2024-09-18T08:23:38.405 INFO:journalctl@ceph.osd.5.smithi050.stdout:Sep 18 08:23:38 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd-5[42249]: 2024-09-18T08:23:38.136+0000 7fc155e15640 -1 osd.5 482 heartbeat_check: no reply from 172.21.15.7:6803 osd.0 since back 2024-09-18T08:22:57.662189+0000 front 2024-09-18T08:22:57.662202+0000 (oldest deadline 2024-09-18T08:23:22.362113+0000) 2024-09-18T08:23:38.405 INFO:journalctl@ceph.osd.5.smithi050.stdout:Sep 18 08:23:38 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd-5[42249]: 2024-09-18T08:23:38.136+0000 7fc155e15640 -1 osd.5 482 heartbeat_check: no reply from 172.21.15.7:6807 osd.1 since back 2024-09-18T08:23:05.163375+0000 front 2024-09-18T08:23:05.163537+0000 (oldest deadline 2024-09-18T08:23:31.563961+0000) 2024-09-18T08:23:38.905 INFO:journalctl@ceph.osd.6.smithi050.stdout:Sep 18 08:23:38 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd-6[47568]: 2024-09-18T08:23:38.655+0000 7f0a56b1c640 -1 osd.6 482 heartbeat_check: no reply from 172.21.15.7:6803 osd.0 since back 2024-09-18T08:22:56.826928+0000 front 2024-09-18T08:22:56.827014+0000 (oldest deadline 2024-09-18T08:23:22.126870+0000) 2024-09-18T08:23:38.906 INFO:journalctl@ceph.osd.6.smithi050.stdout:Sep 18 08:23:38 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd-6[47568]: 2024-09-18T08:23:38.655+0000 7f0a56b1c640 -1 osd.6 482 heartbeat_check: no reply from 172.21.15.7:6807 osd.1 since back 2024-09-18T08:23:06.227974+0000 front 2024-09-18T08:23:06.228037+0000 (oldest deadline 2024-09-18T08:23:32.127680+0000) 2024-09-18T08:23:38.906 INFO:journalctl@ceph.osd.7.smithi050.stdout:Sep 18 08:23:38 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd-7[53264]: 2024-09-18T08:23:38.596+0000 7f77c4ebf640 -1 osd.7 482 heartbeat_check: no reply from 172.21.15.7:6803 osd.0 since back 2024-09-18T08:22:56.257609+0000 front 2024-09-18T08:22:56.257592+0000 (oldest deadline 2024-09-18T08:23:22.157443+0000) 2024-09-18T08:23:38.906 INFO:journalctl@ceph.osd.7.smithi050.stdout:Sep 18 08:23:38 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd-7[53264]: 2024-09-18T08:23:38.596+0000 7f77c4ebf640 -1 osd.7 482 heartbeat_check: no reply from 172.21.15.7:6807 osd.1 since back 2024-09-18T08:23:07.859263+0000 front 2024-09-18T08:23:07.859284+0000 (oldest deadline 2024-09-18T08:23:31.959140+0000) 2024-09-18T08:23:39.596 INFO:journalctl@ceph.osd.5.smithi050.stdout:Sep 18 08:23:39 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd-5[42249]: 2024-09-18T08:23:39.154+0000 7fc155e15640 -1 osd.5 482 heartbeat_check: no reply from 172.21.15.7:6803 osd.0 since back 2024-09-18T08:22:57.662189+0000 front 2024-09-18T08:22:57.662202+0000 (oldest deadline 2024-09-18T08:23:22.362113+0000) 2024-09-18T08:23:39.596 INFO:journalctl@ceph.osd.5.smithi050.stdout:Sep 18 08:23:39 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd-5[42249]: 2024-09-18T08:23:39.154+0000 7fc155e15640 -1 osd.5 482 heartbeat_check: no reply from 172.21.15.7:6807 osd.1 since back 2024-09-18T08:23:05.163375+0000 front 2024-09-18T08:23:05.163537+0000 (oldest deadline 2024-09-18T08:23:31.563961+0000) 2024-09-18T08:23:39.905 INFO:journalctl@ceph.osd.7.smithi050.stdout:Sep 18 08:23:39 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd-7[53264]: 2024-09-18T08:23:39.595+0000 7f77c4ebf640 -1 osd.7 482 heartbeat_check: no reply from 172.21.15.7:6803 osd.0 since back 2024-09-18T08:22:56.257609+0000 front 2024-09-18T08:22:56.257592+0000 (oldest deadline 2024-09-18T08:23:22.157443+0000) 2024-09-18T08:23:39.905 INFO:journalctl@ceph.osd.7.smithi050.stdout:Sep 18 08:23:39 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd-7[53264]: 2024-09-18T08:23:39.595+0000 7f77c4ebf640 -1 osd.7 482 heartbeat_check: no reply from 172.21.15.7:6807 osd.1 since back 2024-09-18T08:23:07.859263+0000 front 2024-09-18T08:23:07.859284+0000 (oldest deadline 2024-09-18T08:23:31.959140+0000) 2024-09-18T08:23:39.905 INFO:journalctl@ceph.osd.6.smithi050.stdout:Sep 18 08:23:39 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd-6[47568]: 2024-09-18T08:23:39.693+0000 7f0a56b1c640 -1 osd.6 482 heartbeat_check: no reply from 172.21.15.7:6803 osd.0 since back 2024-09-18T08:22:56.826928+0000 front 2024-09-18T08:22:56.827014+0000 (oldest deadline 2024-09-18T08:23:22.126870+0000) 2024-09-18T08:23:39.905 INFO:journalctl@ceph.osd.6.smithi050.stdout:Sep 18 08:23:39 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd-6[47568]: 2024-09-18T08:23:39.693+0000 7f0a56b1c640 -1 osd.6 482 heartbeat_check: no reply from 172.21.15.7:6807 osd.1 since back 2024-09-18T08:23:06.227974+0000 front 2024-09-18T08:23:06.228037+0000 (oldest deadline 2024-09-18T08:23:32.127680+0000) 2024-09-18T08:23:39.905 INFO:journalctl@ceph.osd.6.smithi050.stdout:Sep 18 08:23:39 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd-6[47568]: 2024-09-18T08:23:39.693+0000 7f0a56b1c640 -1 osd.6 482 heartbeat_check: no reply from 172.21.15.7:6811 osd.2 since back 2024-09-18T08:23:14.329177+0000 front 2024-09-18T08:23:14.329295+0000 (oldest deadline 2024-09-18T08:23:39.029056+0000) 2024-09-18T08:23:40.606 INFO:journalctl@ceph.osd.5.smithi050.stdout:Sep 18 08:23:40 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd-5[42249]: 2024-09-18T08:23:40.163+0000 7fc155e15640 -1 osd.5 482 heartbeat_check: no reply from 172.21.15.7:6803 osd.0 since back 2024-09-18T08:22:57.662189+0000 front 2024-09-18T08:22:57.662202+0000 (oldest deadline 2024-09-18T08:23:22.362113+0000) 2024-09-18T08:23:40.606 INFO:journalctl@ceph.osd.5.smithi050.stdout:Sep 18 08:23:40 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd-5[42249]: 2024-09-18T08:23:40.163+0000 7fc155e15640 -1 osd.5 482 heartbeat_check: no reply from 172.21.15.7:6807 osd.1 since back 2024-09-18T08:23:05.163375+0000 front 2024-09-18T08:23:05.163537+0000 (oldest deadline 2024-09-18T08:23:31.563961+0000) 2024-09-18T08:23:40.905 INFO:journalctl@ceph.osd.7.smithi050.stdout:Sep 18 08:23:40 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd-7[53264]: 2024-09-18T08:23:40.605+0000 7f77c4ebf640 -1 osd.7 482 heartbeat_check: no reply from 172.21.15.7:6803 osd.0 since back 2024-09-18T08:22:56.257609+0000 front 2024-09-18T08:22:56.257592+0000 (oldest deadline 2024-09-18T08:23:22.157443+0000) 2024-09-18T08:23:40.919 INFO:journalctl@ceph.osd.7.smithi050.stdout:Sep 18 08:23:40 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd-7[53264]: 2024-09-18T08:23:40.605+0000 7f77c4ebf640 -1 osd.7 482 heartbeat_check: no reply from 172.21.15.7:6807 osd.1 since back 2024-09-18T08:23:07.859263+0000 front 2024-09-18T08:23:07.859284+0000 (oldest deadline 2024-09-18T08:23:31.959140+0000) 2024-09-18T08:23:40.919 INFO:journalctl@ceph.osd.7.smithi050.stdout:Sep 18 08:23:40 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd-7[53264]: 2024-09-18T08:23:40.605+0000 7f77c4ebf640 -1 osd.7 482 heartbeat_check: no reply from 172.21.15.7:6811 osd.2 since back 2024-09-18T08:23:15.960606+0000 front 2024-09-18T08:23:15.960739+0000 (oldest deadline 2024-09-18T08:23:40.060510+0000) 2024-09-18T08:23:40.919 INFO:journalctl@ceph.osd.6.smithi050.stdout:Sep 18 08:23:40 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd-6[47568]: 2024-09-18T08:23:40.716+0000 7f0a56b1c640 -1 osd.6 482 heartbeat_check: no reply from 172.21.15.7:6803 osd.0 since back 2024-09-18T08:22:56.826928+0000 front 2024-09-18T08:22:56.827014+0000 (oldest deadline 2024-09-18T08:23:22.126870+0000) 2024-09-18T08:23:40.919 INFO:journalctl@ceph.osd.6.smithi050.stdout:Sep 18 08:23:40 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd-6[47568]: 2024-09-18T08:23:40.716+0000 7f0a56b1c640 -1 osd.6 482 heartbeat_check: no reply from 172.21.15.7:6807 osd.1 since back 2024-09-18T08:23:06.227974+0000 front 2024-09-18T08:23:06.228037+0000 (oldest deadline 2024-09-18T08:23:32.127680+0000) 2024-09-18T08:23:40.919 INFO:journalctl@ceph.osd.6.smithi050.stdout:Sep 18 08:23:40 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd-6[47568]: 2024-09-18T08:23:40.716+0000 7f0a56b1c640 -1 osd.6 482 heartbeat_check: no reply from 172.21.15.7:6811 osd.2 since back 2024-09-18T08:23:14.329177+0000 front 2024-09-18T08:23:14.329295+0000 (oldest deadline 2024-09-18T08:23:39.029056+0000) 2024-09-18T08:23:41.615 INFO:journalctl@ceph.osd.5.smithi050.stdout:Sep 18 08:23:41 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd-5[42249]: 2024-09-18T08:23:41.202+0000 7fc155e15640 -1 osd.5 482 heartbeat_check: no reply from 172.21.15.7:6803 osd.0 since back 2024-09-18T08:22:57.662189+0000 front 2024-09-18T08:22:57.662202+0000 (oldest deadline 2024-09-18T08:23:22.362113+0000) 2024-09-18T08:23:41.615 INFO:journalctl@ceph.osd.5.smithi050.stdout:Sep 18 08:23:41 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd-5[42249]: 2024-09-18T08:23:41.202+0000 7fc155e15640 -1 osd.5 482 heartbeat_check: no reply from 172.21.15.7:6807 osd.1 since back 2024-09-18T08:23:05.163375+0000 front 2024-09-18T08:23:05.163537+0000 (oldest deadline 2024-09-18T08:23:31.563961+0000) 2024-09-18T08:23:41.905 INFO:journalctl@ceph.osd.7.smithi050.stdout:Sep 18 08:23:41 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd-7[53264]: 2024-09-18T08:23:41.614+0000 7f77c4ebf640 -1 osd.7 482 heartbeat_check: no reply from 172.21.15.7:6803 osd.0 since back 2024-09-18T08:22:56.257609+0000 front 2024-09-18T08:22:56.257592+0000 (oldest deadline 2024-09-18T08:23:22.157443+0000) 2024-09-18T08:23:41.905 INFO:journalctl@ceph.osd.7.smithi050.stdout:Sep 18 08:23:41 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd-7[53264]: 2024-09-18T08:23:41.614+0000 7f77c4ebf640 -1 osd.7 482 heartbeat_check: no reply from 172.21.15.7:6807 osd.1 since back 2024-09-18T08:23:07.859263+0000 front 2024-09-18T08:23:07.859284+0000 (oldest deadline 2024-09-18T08:23:31.959140+0000) 2024-09-18T08:23:41.905 INFO:journalctl@ceph.osd.7.smithi050.stdout:Sep 18 08:23:41 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd-7[53264]: 2024-09-18T08:23:41.614+0000 7f77c4ebf640 -1 osd.7 482 heartbeat_check: no reply from 172.21.15.7:6811 osd.2 since back 2024-09-18T08:23:15.960606+0000 front 2024-09-18T08:23:15.960739+0000 (oldest deadline 2024-09-18T08:23:40.060510+0000) 2024-09-18T08:23:41.905 INFO:journalctl@ceph.osd.6.smithi050.stdout:Sep 18 08:23:41 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd-6[47568]: 2024-09-18T08:23:41.685+0000 7f0a56b1c640 -1 osd.6 482 heartbeat_check: no reply from 172.21.15.7:6803 osd.0 since back 2024-09-18T08:22:56.826928+0000 front 2024-09-18T08:22:56.827014+0000 (oldest deadline 2024-09-18T08:23:22.126870+0000) 2024-09-18T08:23:41.906 INFO:journalctl@ceph.osd.6.smithi050.stdout:Sep 18 08:23:41 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd-6[47568]: 2024-09-18T08:23:41.685+0000 7f0a56b1c640 -1 osd.6 482 heartbeat_check: no reply from 172.21.15.7:6807 osd.1 since back 2024-09-18T08:23:06.227974+0000 front 2024-09-18T08:23:06.228037+0000 (oldest deadline 2024-09-18T08:23:32.127680+0000) 2024-09-18T08:23:41.906 INFO:journalctl@ceph.osd.6.smithi050.stdout:Sep 18 08:23:41 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd-6[47568]: 2024-09-18T08:23:41.685+0000 7f0a56b1c640 -1 osd.6 482 heartbeat_check: no reply from 172.21.15.7:6811 osd.2 since back 2024-09-18T08:23:14.329177+0000 front 2024-09-18T08:23:14.329295+0000 (oldest deadline 2024-09-18T08:23:39.029056+0000) 2024-09-18T08:23:42.648 INFO:journalctl@ceph.osd.5.smithi050.stdout:Sep 18 08:23:42 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd-5[42249]: 2024-09-18T08:23:42.172+0000 7fc155e15640 -1 osd.5 482 heartbeat_check: no reply from 172.21.15.7:6803 osd.0 since back 2024-09-18T08:22:57.662189+0000 front 2024-09-18T08:22:57.662202+0000 (oldest deadline 2024-09-18T08:23:22.362113+0000) 2024-09-18T08:23:42.648 INFO:journalctl@ceph.osd.5.smithi050.stdout:Sep 18 08:23:42 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd-5[42249]: 2024-09-18T08:23:42.172+0000 7fc155e15640 -1 osd.5 482 heartbeat_check: no reply from 172.21.15.7:6807 osd.1 since back 2024-09-18T08:23:05.163375+0000 front 2024-09-18T08:23:05.163537+0000 (oldest deadline 2024-09-18T08:23:31.563961+0000) 2024-09-18T08:23:42.648 INFO:journalctl@ceph.osd.5.smithi050.stdout:Sep 18 08:23:42 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd-5[42249]: 2024-09-18T08:23:42.172+0000 7fc155e15640 -1 osd.5 482 heartbeat_check: no reply from 172.21.15.7:6811 osd.2 since back 2024-09-18T08:23:17.365311+0000 front 2024-09-18T08:23:17.365301+0000 (oldest deadline 2024-09-18T08:23:42.065224+0000) 2024-09-18T08:23:42.905 INFO:journalctl@ceph.osd.6.smithi050.stdout:Sep 18 08:23:42 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd-6[47568]: 2024-09-18T08:23:42.725+0000 7f0a56b1c640 -1 osd.6 482 heartbeat_check: no reply from 172.21.15.7:6803 osd.0 since back 2024-09-18T08:22:56.826928+0000 front 2024-09-18T08:22:56.827014+0000 (oldest deadline 2024-09-18T08:23:22.126870+0000) 2024-09-18T08:23:42.905 INFO:journalctl@ceph.osd.6.smithi050.stdout:Sep 18 08:23:42 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd-6[47568]: 2024-09-18T08:23:42.725+0000 7f0a56b1c640 -1 osd.6 482 heartbeat_check: no reply from 172.21.15.7:6807 osd.1 since back 2024-09-18T08:23:06.227974+0000 front 2024-09-18T08:23:06.228037+0000 (oldest deadline 2024-09-18T08:23:32.127680+0000) 2024-09-18T08:23:42.905 INFO:journalctl@ceph.osd.6.smithi050.stdout:Sep 18 08:23:42 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd-6[47568]: 2024-09-18T08:23:42.725+0000 7f0a56b1c640 -1 osd.6 482 heartbeat_check: no reply from 172.21.15.7:6811 osd.2 since back 2024-09-18T08:23:14.329177+0000 front 2024-09-18T08:23:14.329295+0000 (oldest deadline 2024-09-18T08:23:39.029056+0000) 2024-09-18T08:23:42.906 INFO:journalctl@ceph.osd.7.smithi050.stdout:Sep 18 08:23:42 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd-7[53264]: 2024-09-18T08:23:42.646+0000 7f77c4ebf640 -1 osd.7 482 heartbeat_check: no reply from 172.21.15.7:6803 osd.0 since back 2024-09-18T08:22:56.257609+0000 front 2024-09-18T08:22:56.257592+0000 (oldest deadline 2024-09-18T08:23:22.157443+0000) 2024-09-18T08:23:42.906 INFO:journalctl@ceph.osd.7.smithi050.stdout:Sep 18 08:23:42 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd-7[53264]: 2024-09-18T08:23:42.646+0000 7f77c4ebf640 -1 osd.7 482 heartbeat_check: no reply from 172.21.15.7:6807 osd.1 since back 2024-09-18T08:23:07.859263+0000 front 2024-09-18T08:23:07.859284+0000 (oldest deadline 2024-09-18T08:23:31.959140+0000) 2024-09-18T08:23:42.906 INFO:journalctl@ceph.osd.7.smithi050.stdout:Sep 18 08:23:42 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd-7[53264]: 2024-09-18T08:23:42.646+0000 7f77c4ebf640 -1 osd.7 482 heartbeat_check: no reply from 172.21.15.7:6811 osd.2 since back 2024-09-18T08:23:15.960606+0000 front 2024-09-18T08:23:15.960739+0000 (oldest deadline 2024-09-18T08:23:40.060510+0000) 2024-09-18T08:23:43.338 INFO:journalctl@ceph.osd.5.smithi050.stdout:Sep 18 08:23:42 smithi050 podman[143515]: 2024-09-18 08:23:42.930782784 +0000 UTC m=+5.672814008 container died 7466838799b4a5c17cb7a9ac3c8a52a5d8198c182f246eb5497df28351d4ae0d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:1f5b5a3734f665ef333266f39143ac3ef1eb1951b21258f2307e10332e2d7ac9, name=ceph-779349ca-7593-11ef-bceb-c7b262605968-osd-5, org.label-schema.vendor=CentOS, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, org.label-schema.build-date=20240731, RELEASE=wip-guits-main-2024-09-17-1213-e7fb7b5, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , io.buildah.version=1.37.2, org.label-schema.name=CentOS Stream 9 Base Image, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True) 2024-09-18T08:23:43.631 INFO:journalctl@ceph.osd.5.smithi050.stdout:Sep 18 08:23:43 smithi050 podman[143515]: 2024-09-18 08:23:43.33834463 +0000 UTC m=+6.080375851 container cleanup 7466838799b4a5c17cb7a9ac3c8a52a5d8198c182f246eb5497df28351d4ae0d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:1f5b5a3734f665ef333266f39143ac3ef1eb1951b21258f2307e10332e2d7ac9, name=ceph-779349ca-7593-11ef-bceb-c7b262605968-osd-5, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, RELEASE=wip-guits-main-2024-09-17-1213-e7fb7b5, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 9 Base Image, GIT_CLEAN=True, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, org.label-schema.build-date=20240731, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, io.buildah.version=1.37.2) 2024-09-18T08:23:43.632 INFO:journalctl@ceph.osd.5.smithi050.stdout:Sep 18 08:23:43 smithi050 bash[143515]: ceph-779349ca-7593-11ef-bceb-c7b262605968-osd-5 2024-09-18T08:23:43.632 INFO:journalctl@ceph.osd.5.smithi050.stdout:Sep 18 08:23:43 smithi050 podman[143534]: 2024-09-18 08:23:43.556759199 +0000 UTC m=+0.625613331 container remove 7466838799b4a5c17cb7a9ac3c8a52a5d8198c182f246eb5497df28351d4ae0d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:1f5b5a3734f665ef333266f39143ac3ef1eb1951b21258f2307e10332e2d7ac9, name=ceph-779349ca-7593-11ef-bceb-c7b262605968-osd-5, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, RELEASE=wip-guits-main-2024-09-17-1213-e7fb7b5, ceph=True, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.37.2, GIT_CLEAN=True, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240731) 2024-09-18T08:23:43.883 INFO:journalctl@ceph.osd.5.smithi050.stdout:Sep 18 08:23:43 smithi050 podman[143624]: 2024-09-18 08:23:43.775480149 +0000 UTC m=+0.016437063 image pull b22e1e37ddf1ca6e22136cd6540a9ab61e7cafb8bfbc7a91a2f20f7f9941bd9a quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:1f5b5a3734f665ef333266f39143ac3ef1eb1951b21258f2307e10332e2d7ac9 2024-09-18T08:23:43.883 INFO:journalctl@ceph.osd.6.smithi050.stdout:Sep 18 08:23:43 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd-6[47568]: 2024-09-18T08:23:43.729+0000 7f0a56b1c640 -1 osd.6 482 heartbeat_check: no reply from 172.21.15.7:6803 osd.0 since back 2024-09-18T08:22:56.826928+0000 front 2024-09-18T08:22:56.827014+0000 (oldest deadline 2024-09-18T08:23:22.126870+0000) 2024-09-18T08:23:43.883 INFO:journalctl@ceph.osd.6.smithi050.stdout:Sep 18 08:23:43 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd-6[47568]: 2024-09-18T08:23:43.729+0000 7f0a56b1c640 -1 osd.6 482 heartbeat_check: no reply from 172.21.15.7:6807 osd.1 since back 2024-09-18T08:23:06.227974+0000 front 2024-09-18T08:23:06.228037+0000 (oldest deadline 2024-09-18T08:23:32.127680+0000) 2024-09-18T08:23:43.883 INFO:journalctl@ceph.osd.6.smithi050.stdout:Sep 18 08:23:43 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd-6[47568]: 2024-09-18T08:23:43.729+0000 7f0a56b1c640 -1 osd.6 482 heartbeat_check: no reply from 172.21.15.7:6811 osd.2 since back 2024-09-18T08:23:14.329177+0000 front 2024-09-18T08:23:14.329295+0000 (oldest deadline 2024-09-18T08:23:39.029056+0000) 2024-09-18T08:23:43.884 INFO:journalctl@ceph.osd.7.smithi050.stdout:Sep 18 08:23:43 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd-7[53264]: 2024-09-18T08:23:43.630+0000 7f77c4ebf640 -1 osd.7 482 heartbeat_check: no reply from 172.21.15.7:6803 osd.0 since back 2024-09-18T08:22:56.257609+0000 front 2024-09-18T08:22:56.257592+0000 (oldest deadline 2024-09-18T08:23:22.157443+0000) 2024-09-18T08:23:43.884 INFO:journalctl@ceph.osd.7.smithi050.stdout:Sep 18 08:23:43 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd-7[53264]: 2024-09-18T08:23:43.630+0000 7f77c4ebf640 -1 osd.7 482 heartbeat_check: no reply from 172.21.15.7:6807 osd.1 since back 2024-09-18T08:23:07.859263+0000 front 2024-09-18T08:23:07.859284+0000 (oldest deadline 2024-09-18T08:23:31.959140+0000) 2024-09-18T08:23:43.884 INFO:journalctl@ceph.osd.7.smithi050.stdout:Sep 18 08:23:43 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd-7[53264]: 2024-09-18T08:23:43.630+0000 7f77c4ebf640 -1 osd.7 482 heartbeat_check: no reply from 172.21.15.7:6811 osd.2 since back 2024-09-18T08:23:15.960606+0000 front 2024-09-18T08:23:15.960739+0000 (oldest deadline 2024-09-18T08:23:40.060510+0000) 2024-09-18T08:23:43.884 INFO:journalctl@ceph.osd.7.smithi050.stdout:Sep 18 08:23:43 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd-7[53264]: 2024-09-18T08:23:43.630+0000 7f77c4ebf640 -1 osd.7 482 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.24419.0:1205 2.0 2:353ec8f9:::gateway.conf:head [getxattr epoch in=5b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e482) 2024-09-18T08:23:44.155 INFO:journalctl@ceph.osd.5.smithi050.stdout:Sep 18 08:23:43 smithi050 podman[143624]: 2024-09-18 08:23:43.882772749 +0000 UTC m=+0.123729652 container create 2a4035b6903df58130a4b92176c42bee403092dd93242f451759339596fc179d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:1f5b5a3734f665ef333266f39143ac3ef1eb1951b21258f2307e10332e2d7ac9, name=ceph-779349ca-7593-11ef-bceb-c7b262605968-osd-5-deactivate, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 9 Base Image, RELEASE=wip-guits-main-2024-09-17-1213-e7fb7b5, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.build-date=20240731, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, maintainer=Guillaume Abrioux , ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.37.2) 2024-09-18T08:23:44.155 INFO:journalctl@ceph.osd.5.smithi050.stdout:Sep 18 08:23:44 smithi050 podman[143624]: 2024-09-18 08:23:44.101977511 +0000 UTC m=+0.342934415 container init 2a4035b6903df58130a4b92176c42bee403092dd93242f451759339596fc179d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:1f5b5a3734f665ef333266f39143ac3ef1eb1951b21258f2307e10332e2d7ac9, name=ceph-779349ca-7593-11ef-bceb-c7b262605968-osd-5-deactivate, ceph=True, CEPH_POINT_RELEASE=, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, GIT_BRANCH=HEAD, io.buildah.version=1.37.2, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-guits-main-2024-09-17-1213-e7fb7b5, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240731) 2024-09-18T08:23:44.155 INFO:journalctl@ceph.osd.5.smithi050.stdout:Sep 18 08:23:44 smithi050 podman[143624]: 2024-09-18 08:23:44.105699545 +0000 UTC m=+0.346656449 container start 2a4035b6903df58130a4b92176c42bee403092dd93242f451759339596fc179d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:1f5b5a3734f665ef333266f39143ac3ef1eb1951b21258f2307e10332e2d7ac9, name=ceph-779349ca-7593-11ef-bceb-c7b262605968-osd-5-deactivate, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 9 Base Image, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, io.buildah.version=1.37.2, ceph=True, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240731, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.license=GPLv2, RELEASE=wip-guits-main-2024-09-17-1213-e7fb7b5, maintainer=Guillaume Abrioux ) 2024-09-18T08:23:44.598 INFO:journalctl@ceph.osd.5.smithi050.stdout:Sep 18 08:23:44 smithi050 podman[143624]: 2024-09-18 08:23:44.185243059 +0000 UTC m=+0.426199980 container attach 2a4035b6903df58130a4b92176c42bee403092dd93242f451759339596fc179d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:1f5b5a3734f665ef333266f39143ac3ef1eb1951b21258f2307e10332e2d7ac9, name=ceph-779349ca-7593-11ef-bceb-c7b262605968-osd-5-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.vendor=CentOS, ceph=True, io.buildah.version=1.37.2, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240731, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, org.label-schema.license=GPLv2, RELEASE=wip-guits-main-2024-09-17-1213-e7fb7b5) 2024-09-18T08:23:44.598 INFO:journalctl@ceph.osd.5.smithi050.stdout:Sep 18 08:23:44 smithi050 podman[143624]: 2024-09-18 08:23:44.295299584 +0000 UTC m=+0.536256484 container died 2a4035b6903df58130a4b92176c42bee403092dd93242f451759339596fc179d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:1f5b5a3734f665ef333266f39143ac3ef1eb1951b21258f2307e10332e2d7ac9, name=ceph-779349ca-7593-11ef-bceb-c7b262605968-osd-5-deactivate, RELEASE=wip-guits-main-2024-09-17-1213-e7fb7b5, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, io.buildah.version=1.37.2, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 9 Base Image, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, org.label-schema.build-date=20240731, ceph=True) 2024-09-18T08:23:44.905 INFO:journalctl@ceph.osd.6.smithi050.stdout:Sep 18 08:23:44 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd-6[47568]: 2024-09-18T08:23:44.775+0000 7f0a56b1c640 -1 osd.6 482 heartbeat_check: no reply from 172.21.15.7:6803 osd.0 since back 2024-09-18T08:22:56.826928+0000 front 2024-09-18T08:22:56.827014+0000 (oldest deadline 2024-09-18T08:23:22.126870+0000) 2024-09-18T08:23:44.905 INFO:journalctl@ceph.osd.6.smithi050.stdout:Sep 18 08:23:44 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd-6[47568]: 2024-09-18T08:23:44.775+0000 7f0a56b1c640 -1 osd.6 482 heartbeat_check: no reply from 172.21.15.7:6807 osd.1 since back 2024-09-18T08:23:06.227974+0000 front 2024-09-18T08:23:06.228037+0000 (oldest deadline 2024-09-18T08:23:32.127680+0000) 2024-09-18T08:23:44.905 INFO:journalctl@ceph.osd.6.smithi050.stdout:Sep 18 08:23:44 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd-6[47568]: 2024-09-18T08:23:44.775+0000 7f0a56b1c640 -1 osd.6 482 heartbeat_check: no reply from 172.21.15.7:6811 osd.2 since back 2024-09-18T08:23:14.329177+0000 front 2024-09-18T08:23:14.329295+0000 (oldest deadline 2024-09-18T08:23:39.029056+0000) 2024-09-18T08:23:44.905 INFO:journalctl@ceph.osd.7.smithi050.stdout:Sep 18 08:23:44 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd-7[53264]: 2024-09-18T08:23:44.676+0000 7f77c4ebf640 -1 osd.7 482 heartbeat_check: no reply from 172.21.15.7:6803 osd.0 since back 2024-09-18T08:22:56.257609+0000 front 2024-09-18T08:22:56.257592+0000 (oldest deadline 2024-09-18T08:23:22.157443+0000) 2024-09-18T08:23:44.906 INFO:journalctl@ceph.osd.7.smithi050.stdout:Sep 18 08:23:44 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd-7[53264]: 2024-09-18T08:23:44.676+0000 7f77c4ebf640 -1 osd.7 482 heartbeat_check: no reply from 172.21.15.7:6807 osd.1 since back 2024-09-18T08:23:07.859263+0000 front 2024-09-18T08:23:07.859284+0000 (oldest deadline 2024-09-18T08:23:31.959140+0000) 2024-09-18T08:23:44.906 INFO:journalctl@ceph.osd.7.smithi050.stdout:Sep 18 08:23:44 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd-7[53264]: 2024-09-18T08:23:44.676+0000 7f77c4ebf640 -1 osd.7 482 heartbeat_check: no reply from 172.21.15.7:6811 osd.2 since back 2024-09-18T08:23:15.960606+0000 front 2024-09-18T08:23:15.960739+0000 (oldest deadline 2024-09-18T08:23:40.060510+0000) 2024-09-18T08:23:44.906 INFO:journalctl@ceph.osd.7.smithi050.stdout:Sep 18 08:23:44 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd-7[53264]: 2024-09-18T08:23:44.676+0000 7f77c4ebf640 -1 osd.7 482 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.24419.0:1205 2.0 2:353ec8f9:::gateway.conf:head [getxattr epoch in=5b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e482) 2024-09-18T08:23:45.016 DEBUG:teuthology.orchestra.run.smithi050:> sudo pkill -f 'journalctl -f -n 0 -u ceph-779349ca-7593-11ef-bceb-c7b262605968@osd.5.service' 2024-09-18T08:23:45.066 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-09-18T08:23:45.066 INFO:tasks.cephadm.osd.5:Stopped osd.5 2024-09-18T08:23:45.066 INFO:tasks.cephadm.osd.6:Stopping osd.6... 2024-09-18T08:23:45.067 DEBUG:teuthology.orchestra.run.smithi050:> sudo systemctl stop ceph-779349ca-7593-11ef-bceb-c7b262605968@osd.6 2024-09-18T08:23:45.369 INFO:journalctl@ceph.osd.6.smithi050.stdout:Sep 18 08:23:45 smithi050 systemd[1]: Stopping Ceph osd.6 for 779349ca-7593-11ef-bceb-c7b262605968... 2024-09-18T08:23:45.369 INFO:journalctl@ceph.osd.6.smithi050.stdout:Sep 18 08:23:45 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd-6[47568]: 2024-09-18T08:23:45.367+0000 7f0a5a507640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.6 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-09-18T08:23:45.369 INFO:journalctl@ceph.osd.6.smithi050.stdout:Sep 18 08:23:45 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd-6[47568]: 2024-09-18T08:23:45.367+0000 7f0a5a507640 -1 osd.6 482 *** Got signal Terminated *** 2024-09-18T08:23:45.369 INFO:journalctl@ceph.osd.6.smithi050.stdout:Sep 18 08:23:45 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd-6[47568]: 2024-09-18T08:23:45.367+0000 7f0a5a507640 -1 osd.6 482 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-09-18T08:23:45.905 INFO:journalctl@ceph.osd.6.smithi050.stdout:Sep 18 08:23:45 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd-6[47568]: 2024-09-18T08:23:45.747+0000 7f0a56b1c640 -1 osd.6 482 heartbeat_check: no reply from 172.21.15.7:6803 osd.0 since back 2024-09-18T08:22:56.826928+0000 front 2024-09-18T08:22:56.827014+0000 (oldest deadline 2024-09-18T08:23:22.126870+0000) 2024-09-18T08:23:45.905 INFO:journalctl@ceph.osd.6.smithi050.stdout:Sep 18 08:23:45 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd-6[47568]: 2024-09-18T08:23:45.747+0000 7f0a56b1c640 -1 osd.6 482 heartbeat_check: no reply from 172.21.15.7:6807 osd.1 since back 2024-09-18T08:23:06.227974+0000 front 2024-09-18T08:23:06.228037+0000 (oldest deadline 2024-09-18T08:23:32.127680+0000) 2024-09-18T08:23:45.905 INFO:journalctl@ceph.osd.6.smithi050.stdout:Sep 18 08:23:45 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd-6[47568]: 2024-09-18T08:23:45.747+0000 7f0a56b1c640 -1 osd.6 482 heartbeat_check: no reply from 172.21.15.7:6811 osd.2 since back 2024-09-18T08:23:14.329177+0000 front 2024-09-18T08:23:14.329295+0000 (oldest deadline 2024-09-18T08:23:39.029056+0000) 2024-09-18T08:23:45.905 INFO:journalctl@ceph.osd.7.smithi050.stdout:Sep 18 08:23:45 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd-7[53264]: 2024-09-18T08:23:45.642+0000 7f77c4ebf640 -1 osd.7 482 heartbeat_check: no reply from 172.21.15.7:6803 osd.0 since back 2024-09-18T08:22:56.257609+0000 front 2024-09-18T08:22:56.257592+0000 (oldest deadline 2024-09-18T08:23:22.157443+0000) 2024-09-18T08:23:45.905 INFO:journalctl@ceph.osd.7.smithi050.stdout:Sep 18 08:23:45 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd-7[53264]: 2024-09-18T08:23:45.642+0000 7f77c4ebf640 -1 osd.7 482 heartbeat_check: no reply from 172.21.15.7:6807 osd.1 since back 2024-09-18T08:23:07.859263+0000 front 2024-09-18T08:23:07.859284+0000 (oldest deadline 2024-09-18T08:23:31.959140+0000) 2024-09-18T08:23:45.906 INFO:journalctl@ceph.osd.7.smithi050.stdout:Sep 18 08:23:45 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd-7[53264]: 2024-09-18T08:23:45.642+0000 7f77c4ebf640 -1 osd.7 482 heartbeat_check: no reply from 172.21.15.7:6811 osd.2 since back 2024-09-18T08:23:15.960606+0000 front 2024-09-18T08:23:15.960739+0000 (oldest deadline 2024-09-18T08:23:40.060510+0000) 2024-09-18T08:23:45.906 INFO:journalctl@ceph.osd.7.smithi050.stdout:Sep 18 08:23:45 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd-7[53264]: 2024-09-18T08:23:45.642+0000 7f77c4ebf640 -1 osd.7 482 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.24419.0:1205 2.0 2:353ec8f9:::gateway.conf:head [getxattr epoch in=5b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e482) 2024-09-18T08:23:47.132 INFO:journalctl@ceph.osd.7.smithi050.stdout:Sep 18 08:23:46 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd-7[53264]: 2024-09-18T08:23:46.692+0000 7f77c4ebf640 -1 osd.7 482 heartbeat_check: no reply from 172.21.15.7:6803 osd.0 since back 2024-09-18T08:22:56.257609+0000 front 2024-09-18T08:22:56.257592+0000 (oldest deadline 2024-09-18T08:23:22.157443+0000) 2024-09-18T08:23:47.133 INFO:journalctl@ceph.osd.7.smithi050.stdout:Sep 18 08:23:46 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd-7[53264]: 2024-09-18T08:23:46.692+0000 7f77c4ebf640 -1 osd.7 482 heartbeat_check: no reply from 172.21.15.7:6807 osd.1 since back 2024-09-18T08:23:07.859263+0000 front 2024-09-18T08:23:07.859284+0000 (oldest deadline 2024-09-18T08:23:31.959140+0000) 2024-09-18T08:23:47.133 INFO:journalctl@ceph.osd.7.smithi050.stdout:Sep 18 08:23:46 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd-7[53264]: 2024-09-18T08:23:46.692+0000 7f77c4ebf640 -1 osd.7 482 heartbeat_check: no reply from 172.21.15.7:6811 osd.2 since back 2024-09-18T08:23:15.960606+0000 front 2024-09-18T08:23:15.960739+0000 (oldest deadline 2024-09-18T08:23:40.060510+0000) 2024-09-18T08:23:47.133 INFO:journalctl@ceph.osd.7.smithi050.stdout:Sep 18 08:23:46 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd-7[53264]: 2024-09-18T08:23:46.692+0000 7f77c4ebf640 -1 osd.7 482 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.24419.0:1205 2.0 2:353ec8f9:::gateway.conf:head [getxattr epoch in=5b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e482) 2024-09-18T08:23:47.133 INFO:journalctl@ceph.osd.6.smithi050.stdout:Sep 18 08:23:46 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd-6[47568]: 2024-09-18T08:23:46.751+0000 7f0a56b1c640 -1 osd.6 482 heartbeat_check: no reply from 172.21.15.7:6803 osd.0 since back 2024-09-18T08:22:56.826928+0000 front 2024-09-18T08:22:56.827014+0000 (oldest deadline 2024-09-18T08:23:22.126870+0000) 2024-09-18T08:23:47.133 INFO:journalctl@ceph.osd.6.smithi050.stdout:Sep 18 08:23:46 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd-6[47568]: 2024-09-18T08:23:46.751+0000 7f0a56b1c640 -1 osd.6 482 heartbeat_check: no reply from 172.21.15.7:6807 osd.1 since back 2024-09-18T08:23:06.227974+0000 front 2024-09-18T08:23:06.228037+0000 (oldest deadline 2024-09-18T08:23:32.127680+0000) 2024-09-18T08:23:47.133 INFO:journalctl@ceph.osd.6.smithi050.stdout:Sep 18 08:23:46 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd-6[47568]: 2024-09-18T08:23:46.751+0000 7f0a56b1c640 -1 osd.6 482 heartbeat_check: no reply from 172.21.15.7:6811 osd.2 since back 2024-09-18T08:23:14.329177+0000 front 2024-09-18T08:23:14.329295+0000 (oldest deadline 2024-09-18T08:23:39.029056+0000) 2024-09-18T08:23:47.133 INFO:journalctl@ceph.osd.6.smithi050.stdout:Sep 18 08:23:46 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd-6[47568]: 2024-09-18T08:23:46.751+0000 7f0a56b1c640 -1 osd.6 482 heartbeat_check: no reply from 172.21.15.7:6815 osd.3 since back 2024-09-18T08:23:22.530019+0000 front 2024-09-18T08:23:22.530059+0000 (oldest deadline 2024-09-18T08:23:46.629973+0000) 2024-09-18T08:23:48.155 INFO:journalctl@ceph.osd.7.smithi050.stdout:Sep 18 08:23:47 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd-7[53264]: 2024-09-18T08:23:47.710+0000 7f77c4ebf640 -1 osd.7 482 heartbeat_check: no reply from 172.21.15.7:6803 osd.0 since back 2024-09-18T08:22:56.257609+0000 front 2024-09-18T08:22:56.257592+0000 (oldest deadline 2024-09-18T08:23:22.157443+0000) 2024-09-18T08:23:48.155 INFO:journalctl@ceph.osd.7.smithi050.stdout:Sep 18 08:23:47 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd-7[53264]: 2024-09-18T08:23:47.710+0000 7f77c4ebf640 -1 osd.7 482 heartbeat_check: no reply from 172.21.15.7:6807 osd.1 since back 2024-09-18T08:23:07.859263+0000 front 2024-09-18T08:23:07.859284+0000 (oldest deadline 2024-09-18T08:23:31.959140+0000) 2024-09-18T08:23:48.155 INFO:journalctl@ceph.osd.7.smithi050.stdout:Sep 18 08:23:47 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd-7[53264]: 2024-09-18T08:23:47.710+0000 7f77c4ebf640 -1 osd.7 482 heartbeat_check: no reply from 172.21.15.7:6811 osd.2 since back 2024-09-18T08:23:15.960606+0000 front 2024-09-18T08:23:15.960739+0000 (oldest deadline 2024-09-18T08:23:40.060510+0000) 2024-09-18T08:23:48.155 INFO:journalctl@ceph.osd.7.smithi050.stdout:Sep 18 08:23:47 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd-7[53264]: 2024-09-18T08:23:47.710+0000 7f77c4ebf640 -1 osd.7 482 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.24419.0:1205 2.0 2:353ec8f9:::gateway.conf:head [getxattr epoch in=5b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e482) 2024-09-18T08:23:48.155 INFO:journalctl@ceph.osd.6.smithi050.stdout:Sep 18 08:23:47 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd-6[47568]: 2024-09-18T08:23:47.795+0000 7f0a56b1c640 -1 osd.6 482 heartbeat_check: no reply from 172.21.15.7:6803 osd.0 since back 2024-09-18T08:22:56.826928+0000 front 2024-09-18T08:22:56.827014+0000 (oldest deadline 2024-09-18T08:23:22.126870+0000) 2024-09-18T08:23:48.155 INFO:journalctl@ceph.osd.6.smithi050.stdout:Sep 18 08:23:47 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd-6[47568]: 2024-09-18T08:23:47.795+0000 7f0a56b1c640 -1 osd.6 482 heartbeat_check: no reply from 172.21.15.7:6807 osd.1 since back 2024-09-18T08:23:06.227974+0000 front 2024-09-18T08:23:06.228037+0000 (oldest deadline 2024-09-18T08:23:32.127680+0000) 2024-09-18T08:23:48.156 INFO:journalctl@ceph.osd.6.smithi050.stdout:Sep 18 08:23:47 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd-6[47568]: 2024-09-18T08:23:47.795+0000 7f0a56b1c640 -1 osd.6 482 heartbeat_check: no reply from 172.21.15.7:6811 osd.2 since back 2024-09-18T08:23:14.329177+0000 front 2024-09-18T08:23:14.329295+0000 (oldest deadline 2024-09-18T08:23:39.029056+0000) 2024-09-18T08:23:48.156 INFO:journalctl@ceph.osd.6.smithi050.stdout:Sep 18 08:23:47 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd-6[47568]: 2024-09-18T08:23:47.795+0000 7f0a56b1c640 -1 osd.6 482 heartbeat_check: no reply from 172.21.15.7:6815 osd.3 since back 2024-09-18T08:23:22.530019+0000 front 2024-09-18T08:23:22.530059+0000 (oldest deadline 2024-09-18T08:23:46.629973+0000) 2024-09-18T08:23:49.155 INFO:journalctl@ceph.osd.6.smithi050.stdout:Sep 18 08:23:48 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd-6[47568]: 2024-09-18T08:23:48.747+0000 7f0a56b1c640 -1 osd.6 482 heartbeat_check: no reply from 172.21.15.7:6803 osd.0 since back 2024-09-18T08:22:56.826928+0000 front 2024-09-18T08:22:56.827014+0000 (oldest deadline 2024-09-18T08:23:22.126870+0000) 2024-09-18T08:23:49.155 INFO:journalctl@ceph.osd.6.smithi050.stdout:Sep 18 08:23:48 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd-6[47568]: 2024-09-18T08:23:48.747+0000 7f0a56b1c640 -1 osd.6 482 heartbeat_check: no reply from 172.21.15.7:6807 osd.1 since back 2024-09-18T08:23:06.227974+0000 front 2024-09-18T08:23:06.228037+0000 (oldest deadline 2024-09-18T08:23:32.127680+0000) 2024-09-18T08:23:49.155 INFO:journalctl@ceph.osd.6.smithi050.stdout:Sep 18 08:23:48 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd-6[47568]: 2024-09-18T08:23:48.747+0000 7f0a56b1c640 -1 osd.6 482 heartbeat_check: no reply from 172.21.15.7:6811 osd.2 since back 2024-09-18T08:23:14.329177+0000 front 2024-09-18T08:23:14.329295+0000 (oldest deadline 2024-09-18T08:23:39.029056+0000) 2024-09-18T08:23:49.155 INFO:journalctl@ceph.osd.6.smithi050.stdout:Sep 18 08:23:48 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd-6[47568]: 2024-09-18T08:23:48.747+0000 7f0a56b1c640 -1 osd.6 482 heartbeat_check: no reply from 172.21.15.7:6815 osd.3 since back 2024-09-18T08:23:22.530019+0000 front 2024-09-18T08:23:22.530059+0000 (oldest deadline 2024-09-18T08:23:46.629973+0000) 2024-09-18T08:23:49.155 INFO:journalctl@ceph.osd.7.smithi050.stdout:Sep 18 08:23:48 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd-7[53264]: 2024-09-18T08:23:48.741+0000 7f77c4ebf640 -1 osd.7 482 heartbeat_check: no reply from 172.21.15.7:6803 osd.0 since back 2024-09-18T08:22:56.257609+0000 front 2024-09-18T08:22:56.257592+0000 (oldest deadline 2024-09-18T08:23:22.157443+0000) 2024-09-18T08:23:49.155 INFO:journalctl@ceph.osd.7.smithi050.stdout:Sep 18 08:23:48 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd-7[53264]: 2024-09-18T08:23:48.741+0000 7f77c4ebf640 -1 osd.7 482 heartbeat_check: no reply from 172.21.15.7:6807 osd.1 since back 2024-09-18T08:23:07.859263+0000 front 2024-09-18T08:23:07.859284+0000 (oldest deadline 2024-09-18T08:23:31.959140+0000) 2024-09-18T08:23:49.156 INFO:journalctl@ceph.osd.7.smithi050.stdout:Sep 18 08:23:48 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd-7[53264]: 2024-09-18T08:23:48.741+0000 7f77c4ebf640 -1 osd.7 482 heartbeat_check: no reply from 172.21.15.7:6811 osd.2 since back 2024-09-18T08:23:15.960606+0000 front 2024-09-18T08:23:15.960739+0000 (oldest deadline 2024-09-18T08:23:40.060510+0000) 2024-09-18T08:23:49.156 INFO:journalctl@ceph.osd.7.smithi050.stdout:Sep 18 08:23:48 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd-7[53264]: 2024-09-18T08:23:48.741+0000 7f77c4ebf640 -1 osd.7 482 heartbeat_check: no reply from 172.21.15.7:6815 osd.3 since back 2024-09-18T08:23:24.061889+0000 front 2024-09-18T08:23:24.061846+0000 (oldest deadline 2024-09-18T08:23:48.161845+0000) 2024-09-18T08:23:49.156 INFO:journalctl@ceph.osd.7.smithi050.stdout:Sep 18 08:23:48 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd-7[53264]: 2024-09-18T08:23:48.741+0000 7f77c4ebf640 -1 osd.7 482 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.24419.0:1205 2.0 2:353ec8f9:::gateway.conf:head [getxattr epoch in=5b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e482) 2024-09-18T08:23:50.155 INFO:journalctl@ceph.osd.7.smithi050.stdout:Sep 18 08:23:49 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd-7[53264]: 2024-09-18T08:23:49.731+0000 7f77c4ebf640 -1 osd.7 482 heartbeat_check: no reply from 172.21.15.7:6803 osd.0 since back 2024-09-18T08:22:56.257609+0000 front 2024-09-18T08:22:56.257592+0000 (oldest deadline 2024-09-18T08:23:22.157443+0000) 2024-09-18T08:23:50.155 INFO:journalctl@ceph.osd.7.smithi050.stdout:Sep 18 08:23:49 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd-7[53264]: 2024-09-18T08:23:49.731+0000 7f77c4ebf640 -1 osd.7 482 heartbeat_check: no reply from 172.21.15.7:6807 osd.1 since back 2024-09-18T08:23:07.859263+0000 front 2024-09-18T08:23:07.859284+0000 (oldest deadline 2024-09-18T08:23:31.959140+0000) 2024-09-18T08:23:50.155 INFO:journalctl@ceph.osd.7.smithi050.stdout:Sep 18 08:23:49 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd-7[53264]: 2024-09-18T08:23:49.731+0000 7f77c4ebf640 -1 osd.7 482 heartbeat_check: no reply from 172.21.15.7:6811 osd.2 since back 2024-09-18T08:23:15.960606+0000 front 2024-09-18T08:23:15.960739+0000 (oldest deadline 2024-09-18T08:23:40.060510+0000) 2024-09-18T08:23:50.155 INFO:journalctl@ceph.osd.7.smithi050.stdout:Sep 18 08:23:49 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd-7[53264]: 2024-09-18T08:23:49.731+0000 7f77c4ebf640 -1 osd.7 482 heartbeat_check: no reply from 172.21.15.7:6815 osd.3 since back 2024-09-18T08:23:24.061889+0000 front 2024-09-18T08:23:24.061846+0000 (oldest deadline 2024-09-18T08:23:48.161845+0000) 2024-09-18T08:23:50.155 INFO:journalctl@ceph.osd.7.smithi050.stdout:Sep 18 08:23:49 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd-7[53264]: 2024-09-18T08:23:49.731+0000 7f77c4ebf640 -1 osd.7 482 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.24419.0:1205 2.0 2:353ec8f9:::gateway.conf:head [getxattr epoch in=5b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e482) 2024-09-18T08:23:50.155 INFO:journalctl@ceph.osd.6.smithi050.stdout:Sep 18 08:23:49 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd-6[47568]: 2024-09-18T08:23:49.787+0000 7f0a56b1c640 -1 osd.6 482 heartbeat_check: no reply from 172.21.15.7:6803 osd.0 since back 2024-09-18T08:22:56.826928+0000 front 2024-09-18T08:22:56.827014+0000 (oldest deadline 2024-09-18T08:23:22.126870+0000) 2024-09-18T08:23:50.155 INFO:journalctl@ceph.osd.6.smithi050.stdout:Sep 18 08:23:49 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd-6[47568]: 2024-09-18T08:23:49.787+0000 7f0a56b1c640 -1 osd.6 482 heartbeat_check: no reply from 172.21.15.7:6807 osd.1 since back 2024-09-18T08:23:06.227974+0000 front 2024-09-18T08:23:06.228037+0000 (oldest deadline 2024-09-18T08:23:32.127680+0000) 2024-09-18T08:23:50.155 INFO:journalctl@ceph.osd.6.smithi050.stdout:Sep 18 08:23:49 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd-6[47568]: 2024-09-18T08:23:49.787+0000 7f0a56b1c640 -1 osd.6 482 heartbeat_check: no reply from 172.21.15.7:6811 osd.2 since back 2024-09-18T08:23:14.329177+0000 front 2024-09-18T08:23:14.329295+0000 (oldest deadline 2024-09-18T08:23:39.029056+0000) 2024-09-18T08:23:50.156 INFO:journalctl@ceph.osd.6.smithi050.stdout:Sep 18 08:23:49 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd-6[47568]: 2024-09-18T08:23:49.787+0000 7f0a56b1c640 -1 osd.6 482 heartbeat_check: no reply from 172.21.15.7:6815 osd.3 since back 2024-09-18T08:23:22.530019+0000 front 2024-09-18T08:23:22.530059+0000 (oldest deadline 2024-09-18T08:23:46.629973+0000) 2024-09-18T08:23:51.155 INFO:journalctl@ceph.osd.6.smithi050.stdout:Sep 18 08:23:50 smithi050 podman[143726]: 2024-09-18 08:23:50.915921935 +0000 UTC m=+5.654384404 container died cec02a470f49eb4312a8de7076480a6cc8226de078f9b6058c2f85853667d17b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:1f5b5a3734f665ef333266f39143ac3ef1eb1951b21258f2307e10332e2d7ac9, name=ceph-779349ca-7593-11ef-bceb-c7b262605968-osd-6, ceph=True, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, io.buildah.version=1.37.2, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.build-date=20240731, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, RELEASE=wip-guits-main-2024-09-17-1213-e7fb7b5, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 9 Base Image) 2024-09-18T08:23:51.155 INFO:journalctl@ceph.osd.7.smithi050.stdout:Sep 18 08:23:50 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd-7[53264]: 2024-09-18T08:23:50.749+0000 7f77c4ebf640 -1 osd.7 482 heartbeat_check: no reply from 172.21.15.7:6803 osd.0 since back 2024-09-18T08:22:56.257609+0000 front 2024-09-18T08:22:56.257592+0000 (oldest deadline 2024-09-18T08:23:22.157443+0000) 2024-09-18T08:23:51.155 INFO:journalctl@ceph.osd.7.smithi050.stdout:Sep 18 08:23:50 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd-7[53264]: 2024-09-18T08:23:50.749+0000 7f77c4ebf640 -1 osd.7 482 heartbeat_check: no reply from 172.21.15.7:6807 osd.1 since back 2024-09-18T08:23:07.859263+0000 front 2024-09-18T08:23:07.859284+0000 (oldest deadline 2024-09-18T08:23:31.959140+0000) 2024-09-18T08:23:51.155 INFO:journalctl@ceph.osd.7.smithi050.stdout:Sep 18 08:23:50 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd-7[53264]: 2024-09-18T08:23:50.749+0000 7f77c4ebf640 -1 osd.7 482 heartbeat_check: no reply from 172.21.15.7:6811 osd.2 since back 2024-09-18T08:23:15.960606+0000 front 2024-09-18T08:23:15.960739+0000 (oldest deadline 2024-09-18T08:23:40.060510+0000) 2024-09-18T08:23:51.155 INFO:journalctl@ceph.osd.7.smithi050.stdout:Sep 18 08:23:50 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd-7[53264]: 2024-09-18T08:23:50.749+0000 7f77c4ebf640 -1 osd.7 482 heartbeat_check: no reply from 172.21.15.7:6815 osd.3 since back 2024-09-18T08:23:24.061889+0000 front 2024-09-18T08:23:24.061846+0000 (oldest deadline 2024-09-18T08:23:48.161845+0000) 2024-09-18T08:23:51.155 INFO:journalctl@ceph.osd.7.smithi050.stdout:Sep 18 08:23:50 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd-7[53264]: 2024-09-18T08:23:50.749+0000 7f77c4ebf640 -1 osd.7 482 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.24419.0:1205 2.0 2:353ec8f9:::gateway.conf:head [getxattr epoch in=5b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e482) 2024-09-18T08:23:51.770 INFO:journalctl@ceph.osd.6.smithi050.stdout:Sep 18 08:23:51 smithi050 podman[143726]: 2024-09-18 08:23:51.484893065 +0000 UTC m=+6.223355577 container cleanup cec02a470f49eb4312a8de7076480a6cc8226de078f9b6058c2f85853667d17b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:1f5b5a3734f665ef333266f39143ac3ef1eb1951b21258f2307e10332e2d7ac9, name=ceph-779349ca-7593-11ef-bceb-c7b262605968-osd-6, org.label-schema.name=CentOS Stream 9 Base Image, io.buildah.version=1.37.2, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, org.label-schema.build-date=20240731, GIT_BRANCH=HEAD, RELEASE=wip-guits-main-2024-09-17-1213-e7fb7b5, GIT_CLEAN=True) 2024-09-18T08:23:51.770 INFO:journalctl@ceph.osd.6.smithi050.stdout:Sep 18 08:23:51 smithi050 bash[143726]: ceph-779349ca-7593-11ef-bceb-c7b262605968-osd-6 2024-09-18T08:23:52.071 INFO:journalctl@ceph.osd.6.smithi050.stdout:Sep 18 08:23:51 smithi050 podman[143741]: 2024-09-18 08:23:51.77002804 +0000 UTC m=+0.844785702 container remove cec02a470f49eb4312a8de7076480a6cc8226de078f9b6058c2f85853667d17b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:1f5b5a3734f665ef333266f39143ac3ef1eb1951b21258f2307e10332e2d7ac9, name=ceph-779349ca-7593-11ef-bceb-c7b262605968-osd-6, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.schema-version=1.0, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, RELEASE=wip-guits-main-2024-09-17-1213-e7fb7b5, org.label-schema.build-date=20240731, org.label-schema.vendor=CentOS, io.buildah.version=1.37.2, org.label-schema.name=CentOS Stream 9 Base Image, CEPH_POINT_RELEASE=, ceph=True) 2024-09-18T08:23:52.071 INFO:journalctl@ceph.osd.7.smithi050.stdout:Sep 18 08:23:51 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd-7[53264]: 2024-09-18T08:23:51.794+0000 7f77c4ebf640 -1 osd.7 482 heartbeat_check: no reply from 172.21.15.7:6803 osd.0 since back 2024-09-18T08:22:56.257609+0000 front 2024-09-18T08:22:56.257592+0000 (oldest deadline 2024-09-18T08:23:22.157443+0000) 2024-09-18T08:23:52.071 INFO:journalctl@ceph.osd.7.smithi050.stdout:Sep 18 08:23:51 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd-7[53264]: 2024-09-18T08:23:51.794+0000 7f77c4ebf640 -1 osd.7 482 heartbeat_check: no reply from 172.21.15.7:6807 osd.1 since back 2024-09-18T08:23:07.859263+0000 front 2024-09-18T08:23:07.859284+0000 (oldest deadline 2024-09-18T08:23:31.959140+0000) 2024-09-18T08:23:52.072 INFO:journalctl@ceph.osd.7.smithi050.stdout:Sep 18 08:23:51 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd-7[53264]: 2024-09-18T08:23:51.794+0000 7f77c4ebf640 -1 osd.7 482 heartbeat_check: no reply from 172.21.15.7:6811 osd.2 since back 2024-09-18T08:23:15.960606+0000 front 2024-09-18T08:23:15.960739+0000 (oldest deadline 2024-09-18T08:23:40.060510+0000) 2024-09-18T08:23:52.072 INFO:journalctl@ceph.osd.7.smithi050.stdout:Sep 18 08:23:51 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd-7[53264]: 2024-09-18T08:23:51.794+0000 7f77c4ebf640 -1 osd.7 482 heartbeat_check: no reply from 172.21.15.7:6815 osd.3 since back 2024-09-18T08:23:24.061889+0000 front 2024-09-18T08:23:24.061846+0000 (oldest deadline 2024-09-18T08:23:48.161845+0000) 2024-09-18T08:23:52.072 INFO:journalctl@ceph.osd.7.smithi050.stdout:Sep 18 08:23:51 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd-7[53264]: 2024-09-18T08:23:51.794+0000 7f77c4ebf640 -1 osd.7 482 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.24419.0:1205 2.0 2:353ec8f9:::gateway.conf:head [getxattr epoch in=5b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e482) 2024-09-18T08:23:52.374 INFO:journalctl@ceph.osd.6.smithi050.stdout:Sep 18 08:23:52 smithi050 podman[143829]: 2024-09-18 08:23:52.070933509 +0000 UTC m=+0.100260561 container create 6f9a3c2abb566afc7b6c7c6254d5463c9f30506f61fca456827d2a283a424b9d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:1f5b5a3734f665ef333266f39143ac3ef1eb1951b21258f2307e10332e2d7ac9, name=ceph-779349ca-7593-11ef-bceb-c7b262605968-osd-6-deactivate, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, io.buildah.version=1.37.2, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 9 Base Image, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, RELEASE=wip-guits-main-2024-09-17-1213-e7fb7b5, GIT_CLEAN=True, org.label-schema.build-date=20240731) 2024-09-18T08:23:52.375 INFO:journalctl@ceph.osd.6.smithi050.stdout:Sep 18 08:23:52 smithi050 podman[143829]: 2024-09-18 08:23:51.986976847 +0000 UTC m=+0.016303857 image pull b22e1e37ddf1ca6e22136cd6540a9ab61e7cafb8bfbc7a91a2f20f7f9941bd9a quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:1f5b5a3734f665ef333266f39143ac3ef1eb1951b21258f2307e10332e2d7ac9 2024-09-18T08:23:52.375 INFO:journalctl@ceph.osd.6.smithi050.stdout:Sep 18 08:23:52 smithi050 podman[143829]: 2024-09-18 08:23:52.290038395 +0000 UTC m=+0.319365400 container init 6f9a3c2abb566afc7b6c7c6254d5463c9f30506f61fca456827d2a283a424b9d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:1f5b5a3734f665ef333266f39143ac3ef1eb1951b21258f2307e10332e2d7ac9, name=ceph-779349ca-7593-11ef-bceb-c7b262605968-osd-6-deactivate, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, ceph=True, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.build-date=20240731, RELEASE=wip-guits-main-2024-09-17-1213-e7fb7b5, org.label-schema.vendor=CentOS, io.buildah.version=1.37.2, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True) 2024-09-18T08:23:52.375 INFO:journalctl@ceph.osd.6.smithi050.stdout:Sep 18 08:23:52 smithi050 podman[143829]: 2024-09-18 08:23:52.293804569 +0000 UTC m=+0.323131562 container start 6f9a3c2abb566afc7b6c7c6254d5463c9f30506f61fca456827d2a283a424b9d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:1f5b5a3734f665ef333266f39143ac3ef1eb1951b21258f2307e10332e2d7ac9, name=ceph-779349ca-7593-11ef-bceb-c7b262605968-osd-6-deactivate, org.label-schema.license=GPLv2, RELEASE=wip-guits-main-2024-09-17-1213-e7fb7b5, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, GIT_CLEAN=True, org.label-schema.build-date=20240731, io.buildah.version=1.37.2, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, org.label-schema.name=CentOS Stream 9 Base Image) 2024-09-18T08:23:52.375 INFO:journalctl@ceph.osd.6.smithi050.stdout:Sep 18 08:23:52 smithi050 podman[143829]: 2024-09-18 08:23:52.373358181 +0000 UTC m=+0.402685187 container attach 6f9a3c2abb566afc7b6c7c6254d5463c9f30506f61fca456827d2a283a424b9d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:1f5b5a3734f665ef333266f39143ac3ef1eb1951b21258f2307e10332e2d7ac9, name=ceph-779349ca-7593-11ef-bceb-c7b262605968-osd-6-deactivate, org.label-schema.build-date=20240731, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.vendor=CentOS, GIT_CLEAN=True, maintainer=Guillaume Abrioux , io.buildah.version=1.37.2, ceph=True, RELEASE=wip-guits-main-2024-09-17-1213-e7fb7b5, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965) 2024-09-18T08:23:52.655 INFO:journalctl@ceph.osd.6.smithi050.stdout:Sep 18 08:23:52 smithi050 podman[143829]: 2024-09-18 08:23:52.489456222 +0000 UTC m=+0.518783221 container died 6f9a3c2abb566afc7b6c7c6254d5463c9f30506f61fca456827d2a283a424b9d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:1f5b5a3734f665ef333266f39143ac3ef1eb1951b21258f2307e10332e2d7ac9, name=ceph-779349ca-7593-11ef-bceb-c7b262605968-osd-6-deactivate, org.label-schema.build-date=20240731, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 9 Base Image, GIT_CLEAN=True, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, maintainer=Guillaume Abrioux , RELEASE=wip-guits-main-2024-09-17-1213-e7fb7b5, io.buildah.version=1.37.2, CEPH_POINT_RELEASE=, ceph=True) 2024-09-18T08:23:53.155 INFO:journalctl@ceph.osd.7.smithi050.stdout:Sep 18 08:23:52 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd-7[53264]: 2024-09-18T08:23:52.832+0000 7f77c4ebf640 -1 osd.7 482 heartbeat_check: no reply from 172.21.15.7:6803 osd.0 since back 2024-09-18T08:22:56.257609+0000 front 2024-09-18T08:22:56.257592+0000 (oldest deadline 2024-09-18T08:23:22.157443+0000) 2024-09-18T08:23:53.155 INFO:journalctl@ceph.osd.7.smithi050.stdout:Sep 18 08:23:52 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd-7[53264]: 2024-09-18T08:23:52.832+0000 7f77c4ebf640 -1 osd.7 482 heartbeat_check: no reply from 172.21.15.7:6807 osd.1 since back 2024-09-18T08:23:07.859263+0000 front 2024-09-18T08:23:07.859284+0000 (oldest deadline 2024-09-18T08:23:31.959140+0000) 2024-09-18T08:23:53.155 INFO:journalctl@ceph.osd.7.smithi050.stdout:Sep 18 08:23:52 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd-7[53264]: 2024-09-18T08:23:52.832+0000 7f77c4ebf640 -1 osd.7 482 heartbeat_check: no reply from 172.21.15.7:6811 osd.2 since back 2024-09-18T08:23:15.960606+0000 front 2024-09-18T08:23:15.960739+0000 (oldest deadline 2024-09-18T08:23:40.060510+0000) 2024-09-18T08:23:53.155 INFO:journalctl@ceph.osd.7.smithi050.stdout:Sep 18 08:23:52 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd-7[53264]: 2024-09-18T08:23:52.832+0000 7f77c4ebf640 -1 osd.7 482 heartbeat_check: no reply from 172.21.15.7:6815 osd.3 since back 2024-09-18T08:23:24.061889+0000 front 2024-09-18T08:23:24.061846+0000 (oldest deadline 2024-09-18T08:23:48.161845+0000) 2024-09-18T08:23:53.155 INFO:journalctl@ceph.osd.7.smithi050.stdout:Sep 18 08:23:52 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd-7[53264]: 2024-09-18T08:23:52.832+0000 7f77c4ebf640 -1 osd.7 482 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.24419.0:1205 2.0 2:353ec8f9:::gateway.conf:head [getxattr epoch in=5b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e482) 2024-09-18T08:23:53.484 DEBUG:teuthology.orchestra.run.smithi050:> sudo pkill -f 'journalctl -f -n 0 -u ceph-779349ca-7593-11ef-bceb-c7b262605968@osd.6.service' 2024-09-18T08:23:53.534 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-09-18T08:23:53.534 INFO:tasks.cephadm.osd.6:Stopped osd.6 2024-09-18T08:23:53.534 INFO:tasks.cephadm.osd.7:Stopping osd.7... 2024-09-18T08:23:53.535 DEBUG:teuthology.orchestra.run.smithi050:> sudo systemctl stop ceph-779349ca-7593-11ef-bceb-c7b262605968@osd.7 2024-09-18T08:23:53.765 INFO:journalctl@ceph.osd.7.smithi050.stdout:Sep 18 08:23:53 smithi050 systemd[1]: Stopping Ceph osd.7 for 779349ca-7593-11ef-bceb-c7b262605968... 2024-09-18T08:23:54.155 INFO:journalctl@ceph.osd.7.smithi050.stdout:Sep 18 08:23:53 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd-7[53264]: 2024-09-18T08:23:53.792+0000 7f77c4ebf640 -1 osd.7 482 heartbeat_check: no reply from 172.21.15.7:6803 osd.0 since back 2024-09-18T08:22:56.257609+0000 front 2024-09-18T08:22:56.257592+0000 (oldest deadline 2024-09-18T08:23:22.157443+0000) 2024-09-18T08:23:54.155 INFO:journalctl@ceph.osd.7.smithi050.stdout:Sep 18 08:23:53 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd-7[53264]: 2024-09-18T08:23:53.792+0000 7f77c4ebf640 -1 osd.7 482 heartbeat_check: no reply from 172.21.15.7:6807 osd.1 since back 2024-09-18T08:23:07.859263+0000 front 2024-09-18T08:23:07.859284+0000 (oldest deadline 2024-09-18T08:23:31.959140+0000) 2024-09-18T08:23:54.155 INFO:journalctl@ceph.osd.7.smithi050.stdout:Sep 18 08:23:53 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd-7[53264]: 2024-09-18T08:23:53.792+0000 7f77c4ebf640 -1 osd.7 482 heartbeat_check: no reply from 172.21.15.7:6811 osd.2 since back 2024-09-18T08:23:15.960606+0000 front 2024-09-18T08:23:15.960739+0000 (oldest deadline 2024-09-18T08:23:40.060510+0000) 2024-09-18T08:23:54.155 INFO:journalctl@ceph.osd.7.smithi050.stdout:Sep 18 08:23:53 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd-7[53264]: 2024-09-18T08:23:53.792+0000 7f77c4ebf640 -1 osd.7 482 heartbeat_check: no reply from 172.21.15.7:6815 osd.3 since back 2024-09-18T08:23:24.061889+0000 front 2024-09-18T08:23:24.061846+0000 (oldest deadline 2024-09-18T08:23:48.161845+0000) 2024-09-18T08:23:54.155 INFO:journalctl@ceph.osd.7.smithi050.stdout:Sep 18 08:23:53 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd-7[53264]: 2024-09-18T08:23:53.792+0000 7f77c4ebf640 -1 osd.7 482 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.24419.0:1205 2.0 2:353ec8f9:::gateway.conf:head [getxattr epoch in=5b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e482) 2024-09-18T08:23:54.155 INFO:journalctl@ceph.osd.7.smithi050.stdout:Sep 18 08:23:53 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd-7[53264]: 2024-09-18T08:23:53.997+0000 7f77c88aa640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.7 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-09-18T08:23:54.155 INFO:journalctl@ceph.osd.7.smithi050.stdout:Sep 18 08:23:53 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd-7[53264]: 2024-09-18T08:23:53.997+0000 7f77c88aa640 -1 osd.7 482 *** Got signal Terminated *** 2024-09-18T08:23:54.155 INFO:journalctl@ceph.osd.7.smithi050.stdout:Sep 18 08:23:53 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd-7[53264]: 2024-09-18T08:23:53.997+0000 7f77c88aa640 -1 osd.7 482 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-09-18T08:23:55.155 INFO:journalctl@ceph.osd.7.smithi050.stdout:Sep 18 08:23:54 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd-7[53264]: 2024-09-18T08:23:54.815+0000 7f77c4ebf640 -1 osd.7 482 heartbeat_check: no reply from 172.21.15.7:6803 osd.0 since back 2024-09-18T08:22:56.257609+0000 front 2024-09-18T08:22:56.257592+0000 (oldest deadline 2024-09-18T08:23:22.157443+0000) 2024-09-18T08:23:55.155 INFO:journalctl@ceph.osd.7.smithi050.stdout:Sep 18 08:23:54 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd-7[53264]: 2024-09-18T08:23:54.815+0000 7f77c4ebf640 -1 osd.7 482 heartbeat_check: no reply from 172.21.15.7:6807 osd.1 since back 2024-09-18T08:23:07.859263+0000 front 2024-09-18T08:23:07.859284+0000 (oldest deadline 2024-09-18T08:23:31.959140+0000) 2024-09-18T08:23:55.155 INFO:journalctl@ceph.osd.7.smithi050.stdout:Sep 18 08:23:54 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd-7[53264]: 2024-09-18T08:23:54.815+0000 7f77c4ebf640 -1 osd.7 482 heartbeat_check: no reply from 172.21.15.7:6811 osd.2 since back 2024-09-18T08:23:15.960606+0000 front 2024-09-18T08:23:15.960739+0000 (oldest deadline 2024-09-18T08:23:40.060510+0000) 2024-09-18T08:23:55.155 INFO:journalctl@ceph.osd.7.smithi050.stdout:Sep 18 08:23:54 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd-7[53264]: 2024-09-18T08:23:54.815+0000 7f77c4ebf640 -1 osd.7 482 heartbeat_check: no reply from 172.21.15.7:6815 osd.3 since back 2024-09-18T08:23:24.061889+0000 front 2024-09-18T08:23:24.061846+0000 (oldest deadline 2024-09-18T08:23:48.161845+0000) 2024-09-18T08:23:55.155 INFO:journalctl@ceph.osd.7.smithi050.stdout:Sep 18 08:23:54 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd-7[53264]: 2024-09-18T08:23:54.815+0000 7f77c4ebf640 -1 osd.7 482 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.24419.0:1205 2.0 2:353ec8f9:::gateway.conf:head [getxattr epoch in=5b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e482) 2024-09-18T08:23:56.155 INFO:journalctl@ceph.osd.7.smithi050.stdout:Sep 18 08:23:55 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd-7[53264]: 2024-09-18T08:23:55.821+0000 7f77c4ebf640 -1 osd.7 482 heartbeat_check: no reply from 172.21.15.7:6803 osd.0 since back 2024-09-18T08:22:56.257609+0000 front 2024-09-18T08:22:56.257592+0000 (oldest deadline 2024-09-18T08:23:22.157443+0000) 2024-09-18T08:23:56.155 INFO:journalctl@ceph.osd.7.smithi050.stdout:Sep 18 08:23:55 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd-7[53264]: 2024-09-18T08:23:55.821+0000 7f77c4ebf640 -1 osd.7 482 heartbeat_check: no reply from 172.21.15.7:6807 osd.1 since back 2024-09-18T08:23:07.859263+0000 front 2024-09-18T08:23:07.859284+0000 (oldest deadline 2024-09-18T08:23:31.959140+0000) 2024-09-18T08:23:56.155 INFO:journalctl@ceph.osd.7.smithi050.stdout:Sep 18 08:23:55 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd-7[53264]: 2024-09-18T08:23:55.821+0000 7f77c4ebf640 -1 osd.7 482 heartbeat_check: no reply from 172.21.15.7:6811 osd.2 since back 2024-09-18T08:23:15.960606+0000 front 2024-09-18T08:23:15.960739+0000 (oldest deadline 2024-09-18T08:23:40.060510+0000) 2024-09-18T08:23:56.155 INFO:journalctl@ceph.osd.7.smithi050.stdout:Sep 18 08:23:55 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd-7[53264]: 2024-09-18T08:23:55.821+0000 7f77c4ebf640 -1 osd.7 482 heartbeat_check: no reply from 172.21.15.7:6815 osd.3 since back 2024-09-18T08:23:24.061889+0000 front 2024-09-18T08:23:24.061846+0000 (oldest deadline 2024-09-18T08:23:48.161845+0000) 2024-09-18T08:23:56.155 INFO:journalctl@ceph.osd.7.smithi050.stdout:Sep 18 08:23:55 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd-7[53264]: 2024-09-18T08:23:55.821+0000 7f77c4ebf640 -1 osd.7 482 heartbeat_check: no reply from 172.21.15.50:6802 osd.4 since back 2024-09-18T08:23:30.462801+0000 front 2024-09-18T08:23:30.462884+0000 (oldest deadline 2024-09-18T08:23:55.762823+0000) 2024-09-18T08:23:56.156 INFO:journalctl@ceph.osd.7.smithi050.stdout:Sep 18 08:23:55 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd-7[53264]: 2024-09-18T08:23:55.821+0000 7f77c4ebf640 -1 osd.7 482 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.24419.0:1205 2.0 2:353ec8f9:::gateway.conf:head [getxattr epoch in=5b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e482) 2024-09-18T08:23:56.845 INFO:journalctl@ceph.grafana.a.smithi050.stdout:Sep 18 08:23:56 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-grafana-a[60403]: logger=cleanup t=2024-09-18T08:23:56.523005771Z level=info msg="Completed cleanup jobs" duration=6.361574ms 2024-09-18T08:23:56.846 INFO:journalctl@ceph.grafana.a.smithi050.stdout:Sep 18 08:23:56 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-grafana-a[60403]: logger=grafana.update.checker t=2024-09-18T08:23:56.656801326Z level=info msg="Update check succeeded" duration=58.145659ms 2024-09-18T08:23:56.846 INFO:journalctl@ceph.grafana.a.smithi050.stdout:Sep 18 08:23:56 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-grafana-a[60403]: logger=plugins.update.checker t=2024-09-18T08:23:56.6590174Z level=info msg="Update check succeeded" duration=65.621875ms 2024-09-18T08:23:57.155 INFO:journalctl@ceph.osd.7.smithi050.stdout:Sep 18 08:23:56 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd-7[53264]: 2024-09-18T08:23:56.845+0000 7f77c4ebf640 -1 osd.7 482 heartbeat_check: no reply from 172.21.15.7:6803 osd.0 since back 2024-09-18T08:22:56.257609+0000 front 2024-09-18T08:22:56.257592+0000 (oldest deadline 2024-09-18T08:23:22.157443+0000) 2024-09-18T08:23:57.155 INFO:journalctl@ceph.osd.7.smithi050.stdout:Sep 18 08:23:56 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd-7[53264]: 2024-09-18T08:23:56.845+0000 7f77c4ebf640 -1 osd.7 482 heartbeat_check: no reply from 172.21.15.7:6807 osd.1 since back 2024-09-18T08:23:07.859263+0000 front 2024-09-18T08:23:07.859284+0000 (oldest deadline 2024-09-18T08:23:31.959140+0000) 2024-09-18T08:23:57.155 INFO:journalctl@ceph.osd.7.smithi050.stdout:Sep 18 08:23:56 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd-7[53264]: 2024-09-18T08:23:56.845+0000 7f77c4ebf640 -1 osd.7 482 heartbeat_check: no reply from 172.21.15.7:6811 osd.2 since back 2024-09-18T08:23:15.960606+0000 front 2024-09-18T08:23:15.960739+0000 (oldest deadline 2024-09-18T08:23:40.060510+0000) 2024-09-18T08:23:57.155 INFO:journalctl@ceph.osd.7.smithi050.stdout:Sep 18 08:23:56 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd-7[53264]: 2024-09-18T08:23:56.845+0000 7f77c4ebf640 -1 osd.7 482 heartbeat_check: no reply from 172.21.15.7:6815 osd.3 since back 2024-09-18T08:23:24.061889+0000 front 2024-09-18T08:23:24.061846+0000 (oldest deadline 2024-09-18T08:23:48.161845+0000) 2024-09-18T08:23:57.155 INFO:journalctl@ceph.osd.7.smithi050.stdout:Sep 18 08:23:56 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd-7[53264]: 2024-09-18T08:23:56.845+0000 7f77c4ebf640 -1 osd.7 482 heartbeat_check: no reply from 172.21.15.50:6802 osd.4 since back 2024-09-18T08:23:30.462801+0000 front 2024-09-18T08:23:30.462884+0000 (oldest deadline 2024-09-18T08:23:55.762823+0000) 2024-09-18T08:23:57.155 INFO:journalctl@ceph.osd.7.smithi050.stdout:Sep 18 08:23:56 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd-7[53264]: 2024-09-18T08:23:56.845+0000 7f77c4ebf640 -1 osd.7 482 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.24419.0:1205 2.0 2:353ec8f9:::gateway.conf:head [getxattr epoch in=5b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e482) 2024-09-18T08:23:58.155 INFO:journalctl@ceph.osd.7.smithi050.stdout:Sep 18 08:23:57 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd-7[53264]: 2024-09-18T08:23:57.882+0000 7f77c4ebf640 -1 osd.7 482 heartbeat_check: no reply from 172.21.15.7:6803 osd.0 since back 2024-09-18T08:22:56.257609+0000 front 2024-09-18T08:22:56.257592+0000 (oldest deadline 2024-09-18T08:23:22.157443+0000) 2024-09-18T08:23:58.155 INFO:journalctl@ceph.osd.7.smithi050.stdout:Sep 18 08:23:57 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd-7[53264]: 2024-09-18T08:23:57.882+0000 7f77c4ebf640 -1 osd.7 482 heartbeat_check: no reply from 172.21.15.7:6807 osd.1 since back 2024-09-18T08:23:07.859263+0000 front 2024-09-18T08:23:07.859284+0000 (oldest deadline 2024-09-18T08:23:31.959140+0000) 2024-09-18T08:23:58.155 INFO:journalctl@ceph.osd.7.smithi050.stdout:Sep 18 08:23:57 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd-7[53264]: 2024-09-18T08:23:57.882+0000 7f77c4ebf640 -1 osd.7 482 heartbeat_check: no reply from 172.21.15.7:6811 osd.2 since back 2024-09-18T08:23:15.960606+0000 front 2024-09-18T08:23:15.960739+0000 (oldest deadline 2024-09-18T08:23:40.060510+0000) 2024-09-18T08:23:58.155 INFO:journalctl@ceph.osd.7.smithi050.stdout:Sep 18 08:23:57 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd-7[53264]: 2024-09-18T08:23:57.882+0000 7f77c4ebf640 -1 osd.7 482 heartbeat_check: no reply from 172.21.15.7:6815 osd.3 since back 2024-09-18T08:23:24.061889+0000 front 2024-09-18T08:23:24.061846+0000 (oldest deadline 2024-09-18T08:23:48.161845+0000) 2024-09-18T08:23:58.155 INFO:journalctl@ceph.osd.7.smithi050.stdout:Sep 18 08:23:57 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd-7[53264]: 2024-09-18T08:23:57.882+0000 7f77c4ebf640 -1 osd.7 482 heartbeat_check: no reply from 172.21.15.50:6802 osd.4 since back 2024-09-18T08:23:30.462801+0000 front 2024-09-18T08:23:30.462884+0000 (oldest deadline 2024-09-18T08:23:55.762823+0000) 2024-09-18T08:23:58.155 INFO:journalctl@ceph.osd.7.smithi050.stdout:Sep 18 08:23:57 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd-7[53264]: 2024-09-18T08:23:57.882+0000 7f77c4ebf640 -1 osd.7 482 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.24419.0:1205 2.0 2:353ec8f9:::gateway.conf:head [getxattr epoch in=5b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e482) 2024-09-18T08:23:58.156 INFO:journalctl@ceph.prometheus.a.smithi050.stdout:Sep 18 08:23:58 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-prometheus-a[65581]: ts=2024-09-18T08:23:58.009Z caller=refresh.go:90 level=error component="discovery manager scrape" discovery=http config=ceph msg="Unable to refresh target groups" err="Get \"http://172.21.15.7:8765/sd/prometheus/sd-config?service=mgr-prometheus\": dial tcp 172.21.15.7:8765: connect: connection refused" 2024-09-18T08:23:58.156 INFO:journalctl@ceph.prometheus.a.smithi050.stdout:Sep 18 08:23:58 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-prometheus-a[65581]: ts=2024-09-18T08:23:58.009Z caller=refresh.go:90 level=error component="discovery manager scrape" discovery=http config=node msg="Unable to refresh target groups" err="Get \"http://172.21.15.7:8765/sd/prometheus/sd-config?service=node-exporter\": dial tcp 172.21.15.7:8765: connect: connection refused" 2024-09-18T08:23:58.156 INFO:journalctl@ceph.prometheus.a.smithi050.stdout:Sep 18 08:23:58 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-prometheus-a[65581]: ts=2024-09-18T08:23:58.010Z caller=refresh.go:90 level=error component="discovery manager notify" discovery=http config=config-0 msg="Unable to refresh target groups" err="Get \"http://172.21.15.7:8765/sd/prometheus/sd-config?service=alertmanager\": dial tcp 172.21.15.7:8765: connect: connection refused" 2024-09-18T08:23:58.156 INFO:journalctl@ceph.prometheus.a.smithi050.stdout:Sep 18 08:23:58 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-prometheus-a[65581]: ts=2024-09-18T08:23:58.012Z caller=refresh.go:90 level=error component="discovery manager scrape" discovery=http config=nfs msg="Unable to refresh target groups" err="Get \"http://172.21.15.7:8765/sd/prometheus/sd-config?service=nfs\": dial tcp 172.21.15.7:8765: connect: connection refused" 2024-09-18T08:23:58.156 INFO:journalctl@ceph.prometheus.a.smithi050.stdout:Sep 18 08:23:58 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-prometheus-a[65581]: ts=2024-09-18T08:23:58.015Z caller=refresh.go:90 level=error component="discovery manager scrape" discovery=http config=ceph-exporter msg="Unable to refresh target groups" err="Get \"http://172.21.15.7:8765/sd/prometheus/sd-config?service=ceph-exporter\": dial tcp 172.21.15.7:8765: connect: connection refused" 2024-09-18T08:23:58.156 INFO:journalctl@ceph.prometheus.a.smithi050.stdout:Sep 18 08:23:58 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-prometheus-a[65581]: ts=2024-09-18T08:23:58.017Z caller=refresh.go:90 level=error component="discovery manager scrape" discovery=http config=nvmeof msg="Unable to refresh target groups" err="Get \"http://172.21.15.7:8765/sd/prometheus/sd-config?service=nvmeof\": dial tcp 172.21.15.7:8765: connect: connection refused" 2024-09-18T08:23:58.156 INFO:journalctl@ceph.prometheus.a.smithi050.stdout:Sep 18 08:23:58 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-prometheus-a[65581]: ts=2024-09-18T08:23:58.021Z caller=refresh.go:90 level=error component="discovery manager scrape" discovery=http config=smb msg="Unable to refresh target groups" err="Get \"http://172.21.15.7:8765/sd/prometheus/sd-config?service=smb\": dial tcp 172.21.15.7:8765: connect: connection refused" 2024-09-18T08:23:59.405 INFO:journalctl@ceph.osd.7.smithi050.stdout:Sep 18 08:23:58 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd-7[53264]: 2024-09-18T08:23:58.914+0000 7f77c4ebf640 -1 osd.7 482 heartbeat_check: no reply from 172.21.15.7:6803 osd.0 since back 2024-09-18T08:22:56.257609+0000 front 2024-09-18T08:22:56.257592+0000 (oldest deadline 2024-09-18T08:23:22.157443+0000) 2024-09-18T08:23:59.405 INFO:journalctl@ceph.osd.7.smithi050.stdout:Sep 18 08:23:58 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd-7[53264]: 2024-09-18T08:23:58.914+0000 7f77c4ebf640 -1 osd.7 482 heartbeat_check: no reply from 172.21.15.7:6807 osd.1 since back 2024-09-18T08:23:07.859263+0000 front 2024-09-18T08:23:07.859284+0000 (oldest deadline 2024-09-18T08:23:31.959140+0000) 2024-09-18T08:23:59.405 INFO:journalctl@ceph.osd.7.smithi050.stdout:Sep 18 08:23:58 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd-7[53264]: 2024-09-18T08:23:58.914+0000 7f77c4ebf640 -1 osd.7 482 heartbeat_check: no reply from 172.21.15.7:6811 osd.2 since back 2024-09-18T08:23:15.960606+0000 front 2024-09-18T08:23:15.960739+0000 (oldest deadline 2024-09-18T08:23:40.060510+0000) 2024-09-18T08:23:59.405 INFO:journalctl@ceph.osd.7.smithi050.stdout:Sep 18 08:23:58 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd-7[53264]: 2024-09-18T08:23:58.914+0000 7f77c4ebf640 -1 osd.7 482 heartbeat_check: no reply from 172.21.15.7:6815 osd.3 since back 2024-09-18T08:23:24.061889+0000 front 2024-09-18T08:23:24.061846+0000 (oldest deadline 2024-09-18T08:23:48.161845+0000) 2024-09-18T08:23:59.405 INFO:journalctl@ceph.osd.7.smithi050.stdout:Sep 18 08:23:58 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd-7[53264]: 2024-09-18T08:23:58.914+0000 7f77c4ebf640 -1 osd.7 482 heartbeat_check: no reply from 172.21.15.50:6802 osd.4 since back 2024-09-18T08:23:30.462801+0000 front 2024-09-18T08:23:30.462884+0000 (oldest deadline 2024-09-18T08:23:55.762823+0000) 2024-09-18T08:23:59.405 INFO:journalctl@ceph.osd.7.smithi050.stdout:Sep 18 08:23:58 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-osd-7[53264]: 2024-09-18T08:23:58.914+0000 7f77c4ebf640 -1 osd.7 482 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.24419.0:1205 2.0 2:353ec8f9:::gateway.conf:head [getxattr epoch in=5b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e482) 2024-09-18T08:23:59.933 INFO:journalctl@ceph.osd.7.smithi050.stdout:Sep 18 08:23:59 smithi050 podman[143933]: 2024-09-18 08:23:59.657229594 +0000 UTC m=+5.915931045 container died e355a4c6d05aa1b079cabb7c7de7dc9dc80cae2bf26875c4bb9194e9c027216d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:1f5b5a3734f665ef333266f39143ac3ef1eb1951b21258f2307e10332e2d7ac9, name=ceph-779349ca-7593-11ef-bceb-c7b262605968-osd-7, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, org.label-schema.schema-version=1.0, ceph=True, maintainer=Guillaume Abrioux , io.buildah.version=1.37.2, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 9 Base Image, RELEASE=wip-guits-main-2024-09-17-1213-e7fb7b5, org.label-schema.build-date=20240731) 2024-09-18T08:24:00.405 INFO:journalctl@ceph.osd.7.smithi050.stdout:Sep 18 08:24:00 smithi050 podman[143933]: 2024-09-18 08:24:00.140004188 +0000 UTC m=+6.398705627 container cleanup e355a4c6d05aa1b079cabb7c7de7dc9dc80cae2bf26875c4bb9194e9c027216d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:1f5b5a3734f665ef333266f39143ac3ef1eb1951b21258f2307e10332e2d7ac9, name=ceph-779349ca-7593-11ef-bceb-c7b262605968-osd-7, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240731, ceph=True, io.buildah.version=1.37.2, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.license=GPLv2, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-guits-main-2024-09-17-1213-e7fb7b5, org.label-schema.name=CentOS Stream 9 Base Image, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux ) 2024-09-18T08:24:00.405 INFO:journalctl@ceph.osd.7.smithi050.stdout:Sep 18 08:24:00 smithi050 bash[143933]: ceph-779349ca-7593-11ef-bceb-c7b262605968-osd-7 2024-09-18T08:24:00.817 INFO:journalctl@ceph.osd.7.smithi050.stdout:Sep 18 08:24:00 smithi050 podman[143960]: 2024-09-18 08:24:00.499809648 +0000 UTC m=+0.839977309 container remove e355a4c6d05aa1b079cabb7c7de7dc9dc80cae2bf26875c4bb9194e9c027216d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:1f5b5a3734f665ef333266f39143ac3ef1eb1951b21258f2307e10332e2d7ac9, name=ceph-779349ca-7593-11ef-bceb-c7b262605968-osd-7, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240731, ceph=True, org.label-schema.license=GPLv2, RELEASE=wip-guits-main-2024-09-17-1213-e7fb7b5, CEPH_POINT_RELEASE=, io.buildah.version=1.37.2, org.label-schema.schema-version=1.0, GIT_CLEAN=True, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, maintainer=Guillaume Abrioux ) 2024-09-18T08:24:01.145 INFO:journalctl@ceph.osd.7.smithi050.stdout:Sep 18 08:24:00 smithi050 podman[144049]: 2024-09-18 08:24:00.817297392 +0000 UTC m=+0.108607111 container create 74a24b7dc25802e97499857558ad24da985f3723dbc963c622c11ebfb4473fb0 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:1f5b5a3734f665ef333266f39143ac3ef1eb1951b21258f2307e10332e2d7ac9, name=ceph-779349ca-7593-11ef-bceb-c7b262605968-osd-7-deactivate, org.label-schema.name=CentOS Stream 9 Base Image, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , GIT_CLEAN=True, io.buildah.version=1.37.2, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240731, RELEASE=wip-guits-main-2024-09-17-1213-e7fb7b5, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, ceph=True, org.label-schema.schema-version=1.0) 2024-09-18T08:24:01.145 INFO:journalctl@ceph.osd.7.smithi050.stdout:Sep 18 08:24:00 smithi050 podman[144049]: 2024-09-18 08:24:00.725234343 +0000 UTC m=+0.016544009 image pull b22e1e37ddf1ca6e22136cd6540a9ab61e7cafb8bfbc7a91a2f20f7f9941bd9a quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:1f5b5a3734f665ef333266f39143ac3ef1eb1951b21258f2307e10332e2d7ac9 2024-09-18T08:24:01.145 INFO:journalctl@ceph.osd.7.smithi050.stdout:Sep 18 08:24:01 smithi050 podman[144049]: 2024-09-18 08:24:01.045111733 +0000 UTC m=+0.336421389 container init 74a24b7dc25802e97499857558ad24da985f3723dbc963c622c11ebfb4473fb0 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:1f5b5a3734f665ef333266f39143ac3ef1eb1951b21258f2307e10332e2d7ac9, name=ceph-779349ca-7593-11ef-bceb-c7b262605968-osd-7-deactivate, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, RELEASE=wip-guits-main-2024-09-17-1213-e7fb7b5, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240731, CEPH_POINT_RELEASE=, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, GIT_CLEAN=True, io.buildah.version=1.37.2, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 9 Base Image, GIT_BRANCH=HEAD, ceph=True, maintainer=Guillaume Abrioux ) 2024-09-18T08:24:01.145 INFO:journalctl@ceph.osd.7.smithi050.stdout:Sep 18 08:24:01 smithi050 podman[144049]: 2024-09-18 08:24:01.048469533 +0000 UTC m=+0.339779190 container start 74a24b7dc25802e97499857558ad24da985f3723dbc963c622c11ebfb4473fb0 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:1f5b5a3734f665ef333266f39143ac3ef1eb1951b21258f2307e10332e2d7ac9, name=ceph-779349ca-7593-11ef-bceb-c7b262605968-osd-7-deactivate, io.buildah.version=1.37.2, RELEASE=wip-guits-main-2024-09-17-1213-e7fb7b5, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240731, maintainer=Guillaume Abrioux , GIT_CLEAN=True, GIT_BRANCH=HEAD, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, ceph=True) 2024-09-18T08:24:01.145 INFO:journalctl@ceph.osd.7.smithi050.stdout:Sep 18 08:24:01 smithi050 podman[144049]: 2024-09-18 08:24:01.144946191 +0000 UTC m=+0.436255874 container attach 74a24b7dc25802e97499857558ad24da985f3723dbc963c622c11ebfb4473fb0 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:1f5b5a3734f665ef333266f39143ac3ef1eb1951b21258f2307e10332e2d7ac9, name=ceph-779349ca-7593-11ef-bceb-c7b262605968-osd-7-deactivate, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.schema-version=1.0, io.buildah.version=1.37.2, org.label-schema.build-date=20240731, RELEASE=wip-guits-main-2024-09-17-1213-e7fb7b5, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.vendor=CentOS, GIT_CLEAN=True, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD) 2024-09-18T08:24:01.405 INFO:journalctl@ceph.osd.7.smithi050.stdout:Sep 18 08:24:01 smithi050 podman[144049]: 2024-09-18 08:24:01.250429746 +0000 UTC m=+0.541739406 container died 74a24b7dc25802e97499857558ad24da985f3723dbc963c622c11ebfb4473fb0 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:1f5b5a3734f665ef333266f39143ac3ef1eb1951b21258f2307e10332e2d7ac9, name=ceph-779349ca-7593-11ef-bceb-c7b262605968-osd-7-deactivate, io.buildah.version=1.37.2, org.label-schema.build-date=20240731, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 9 Base Image, CEPH_POINT_RELEASE=, ceph=True, RELEASE=wip-guits-main-2024-09-17-1213-e7fb7b5, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, org.label-schema.schema-version=1.0) 2024-09-18T08:24:01.857 DEBUG:teuthology.orchestra.run.smithi050:> sudo pkill -f 'journalctl -f -n 0 -u ceph-779349ca-7593-11ef-bceb-c7b262605968@osd.7.service' 2024-09-18T08:24:01.902 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-09-18T08:24:01.902 INFO:tasks.cephadm.osd.7:Stopped osd.7 2024-09-18T08:24:01.903 INFO:tasks.cephadm.ceph.rgw.foo.a:Stopping rgw.foo.a... 2024-09-18T08:24:01.903 DEBUG:teuthology.orchestra.run.smithi007:> sudo systemctl stop ceph-779349ca-7593-11ef-bceb-c7b262605968@rgw.foo.a 2024-09-18T08:24:02.211 INFO:journalctl@ceph.rgw.foo.a.smithi007.stdout:Sep 18 08:24:01 smithi007 systemd[1]: Stopping Ceph rgw.foo.a for 779349ca-7593-11ef-bceb-c7b262605968... 2024-09-18T08:24:02.477 INFO:journalctl@ceph.rgw.foo.a.smithi007.stdout:Sep 18 08:24:02 smithi007 ceph-779349ca-7593-11ef-bceb-c7b262605968-rgw-foo-a[63271]: 2024-09-18T08:24:02.209+0000 7fdfca4e8640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/radosgw -n client.rgw.foo.a -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-09-18T08:24:02.477 INFO:journalctl@ceph.rgw.foo.a.smithi007.stdout:Sep 18 08:24:02 smithi007 ceph-779349ca-7593-11ef-bceb-c7b262605968-rgw-foo-a[63271]: 2024-09-18T08:24:02.209+0000 7fdfcdd54880 -1 shutting down 2024-09-18T08:24:12.480 INFO:journalctl@ceph.rgw.foo.a.smithi007.stdout:Sep 18 08:24:12 smithi007 bash[196562]: time="2024-09-18T08:24:12Z" level=warning msg="StopSignal SIGTERM failed to stop container ceph-779349ca-7593-11ef-bceb-c7b262605968-rgw-foo-a in 10 seconds, resorting to SIGKILL" 2024-09-18T08:24:12.480 INFO:journalctl@ceph.rgw.foo.a.smithi007.stdout:Sep 18 08:24:12 smithi007 podman[196562]: 2024-09-18 08:24:12.242855808 +0000 UTC m=+10.134968462 container died 3ed88a6aa96c5c93fdf0e98eee7847b67e703815a163934a55be6e9ce5976047 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:1f5b5a3734f665ef333266f39143ac3ef1eb1951b21258f2307e10332e2d7ac9, name=ceph-779349ca-7593-11ef-bceb-c7b262605968-rgw-foo-a, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, ceph=True, org.label-schema.license=GPLv2, org.label-schema.build-date=20240731, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 9 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, io.buildah.version=1.37.2, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, RELEASE=wip-guits-main-2024-09-17-1213-e7fb7b5) 2024-09-18T08:24:12.885 INFO:journalctl@ceph.rgw.foo.a.smithi007.stdout:Sep 18 08:24:12 smithi007 podman[196562]: 2024-09-18 08:24:12.675593793 +0000 UTC m=+10.567706439 container cleanup 3ed88a6aa96c5c93fdf0e98eee7847b67e703815a163934a55be6e9ce5976047 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:1f5b5a3734f665ef333266f39143ac3ef1eb1951b21258f2307e10332e2d7ac9, name=ceph-779349ca-7593-11ef-bceb-c7b262605968-rgw-foo-a, org.label-schema.build-date=20240731, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , ceph=True, io.buildah.version=1.37.2, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-guits-main-2024-09-17-1213-e7fb7b5, org.label-schema.schema-version=1.0, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, CEPH_POINT_RELEASE=, GIT_CLEAN=True) 2024-09-18T08:24:12.886 INFO:journalctl@ceph.rgw.foo.a.smithi007.stdout:Sep 18 08:24:12 smithi007 bash[196562]: ceph-779349ca-7593-11ef-bceb-c7b262605968-rgw-foo-a 2024-09-18T08:24:13.107 DEBUG:teuthology.orchestra.run.smithi007:> sudo pkill -f 'journalctl -f -n 0 -u ceph-779349ca-7593-11ef-bceb-c7b262605968@rgw.foo.a.service' 2024-09-18T08:24:13.145 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-09-18T08:24:13.146 INFO:tasks.cephadm.ceph.rgw.foo.a:Stopped rgw.foo.a 2024-09-18T08:24:13.146 INFO:tasks.cephadm.prometheus.a:Stopping prometheus.a... 2024-09-18T08:24:13.146 DEBUG:teuthology.orchestra.run.smithi050:> sudo systemctl stop ceph-779349ca-7593-11ef-bceb-c7b262605968@prometheus.a 2024-09-18T08:24:13.476 INFO:journalctl@ceph.prometheus.a.smithi050.stdout:Sep 18 08:24:13 smithi050 systemd[1]: Stopping Ceph prometheus.a for 779349ca-7593-11ef-bceb-c7b262605968... 2024-09-18T08:24:13.751 INFO:journalctl@ceph.prometheus.a.smithi050.stdout:Sep 18 08:24:13 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-prometheus-a[65581]: ts=2024-09-18T08:24:13.475Z caller=main.go:964 level=warn msg="Received SIGTERM, exiting gracefully..." 2024-09-18T08:24:13.751 INFO:journalctl@ceph.prometheus.a.smithi050.stdout:Sep 18 08:24:13 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-prometheus-a[65581]: ts=2024-09-18T08:24:13.476Z caller=main.go:988 level=info msg="Stopping scrape discovery manager..." 2024-09-18T08:24:13.751 INFO:journalctl@ceph.prometheus.a.smithi050.stdout:Sep 18 08:24:13 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-prometheus-a[65581]: ts=2024-09-18T08:24:13.476Z caller=main.go:1002 level=info msg="Stopping notify discovery manager..." 2024-09-18T08:24:13.751 INFO:journalctl@ceph.prometheus.a.smithi050.stdout:Sep 18 08:24:13 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-prometheus-a[65581]: ts=2024-09-18T08:24:13.476Z caller=manager.go:177 level=info component="rule manager" msg="Stopping rule manager..." 2024-09-18T08:24:13.751 INFO:journalctl@ceph.prometheus.a.smithi050.stdout:Sep 18 08:24:13 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-prometheus-a[65581]: ts=2024-09-18T08:24:13.476Z caller=main.go:984 level=info msg="Scrape discovery manager stopped" 2024-09-18T08:24:13.751 INFO:journalctl@ceph.prometheus.a.smithi050.stdout:Sep 18 08:24:13 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-prometheus-a[65581]: ts=2024-09-18T08:24:13.476Z caller=main.go:998 level=info msg="Notify discovery manager stopped" 2024-09-18T08:24:13.751 INFO:journalctl@ceph.prometheus.a.smithi050.stdout:Sep 18 08:24:13 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-prometheus-a[65581]: ts=2024-09-18T08:24:13.476Z caller=manager.go:187 level=info component="rule manager" msg="Rule manager stopped" 2024-09-18T08:24:13.751 INFO:journalctl@ceph.prometheus.a.smithi050.stdout:Sep 18 08:24:13 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-prometheus-a[65581]: ts=2024-09-18T08:24:13.476Z caller=main.go:1039 level=info msg="Stopping scrape manager..." 2024-09-18T08:24:13.751 INFO:journalctl@ceph.prometheus.a.smithi050.stdout:Sep 18 08:24:13 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-prometheus-a[65581]: ts=2024-09-18T08:24:13.476Z caller=main.go:1031 level=info msg="Scrape manager stopped" 2024-09-18T08:24:13.752 INFO:journalctl@ceph.prometheus.a.smithi050.stdout:Sep 18 08:24:13 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-prometheus-a[65581]: ts=2024-09-18T08:24:13.476Z caller=notifier.go:618 level=info component=notifier msg="Stopping notification manager..." 2024-09-18T08:24:13.752 INFO:journalctl@ceph.prometheus.a.smithi050.stdout:Sep 18 08:24:13 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-prometheus-a[65581]: ts=2024-09-18T08:24:13.477Z caller=main.go:1261 level=info msg="Notifier manager stopped" 2024-09-18T08:24:13.752 INFO:journalctl@ceph.prometheus.a.smithi050.stdout:Sep 18 08:24:13 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-prometheus-a[65581]: ts=2024-09-18T08:24:13.477Z caller=main.go:1273 level=info msg="See you next time!" 2024-09-18T08:24:13.752 INFO:journalctl@ceph.prometheus.a.smithi050.stdout:Sep 18 08:24:13 smithi050 podman[144152]: 2024-09-18 08:24:13.485914138 +0000 UTC m=+0.112413870 container died 32d3202e5a25373bca086a0ca9811cc9215c296997ae96fb8f32b6507b0bb3a6 (image=quay.io/prometheus/prometheus:v2.51.0, name=ceph-779349ca-7593-11ef-bceb-c7b262605968-prometheus-a, maintainer=The Prometheus Authors ) 2024-09-18T08:24:14.118 INFO:journalctl@ceph.prometheus.a.smithi050.stdout:Sep 18 08:24:13 smithi050 podman[144152]: 2024-09-18 08:24:13.858895979 +0000 UTC m=+0.485395710 container cleanup 32d3202e5a25373bca086a0ca9811cc9215c296997ae96fb8f32b6507b0bb3a6 (image=quay.io/prometheus/prometheus:v2.51.0, name=ceph-779349ca-7593-11ef-bceb-c7b262605968-prometheus-a, maintainer=The Prometheus Authors ) 2024-09-18T08:24:14.118 INFO:journalctl@ceph.prometheus.a.smithi050.stdout:Sep 18 08:24:13 smithi050 bash[144152]: ceph-779349ca-7593-11ef-bceb-c7b262605968-prometheus-a 2024-09-18T08:24:14.251 DEBUG:teuthology.orchestra.run.smithi050:> sudo pkill -f 'journalctl -f -n 0 -u ceph-779349ca-7593-11ef-bceb-c7b262605968@prometheus.a.service' 2024-09-18T08:24:14.287 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-09-18T08:24:14.287 INFO:tasks.cephadm.prometheus.a:Stopped prometheus.a 2024-09-18T08:24:14.288 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 779349ca-7593-11ef-bceb-c7b262605968 --force --keep-logs 2024-09-18T08:24:14.497 INFO:teuthology.orchestra.run.smithi007.stdout:Deleting cluster with fsid: 779349ca-7593-11ef-bceb-c7b262605968 2024-09-18T08:24:16.479 INFO:journalctl@ceph.alertmanager.a.smithi007.stdout:Sep 18 08:24:16 smithi007 systemd[1]: Stopping Ceph alertmanager.a for 779349ca-7593-11ef-bceb-c7b262605968... 2024-09-18T08:24:16.783 INFO:journalctl@ceph.alertmanager.a.smithi007.stdout:Sep 18 08:24:16 smithi007 ceph-779349ca-7593-11ef-bceb-c7b262605968-alertmanager-a[78486]: ts=2024-09-18T08:24:16.481Z caller=main.go:542 level=info msg="Received SIGTERM, exiting gracefully..." 2024-09-18T08:24:16.784 INFO:journalctl@ceph.alertmanager.a.smithi007.stdout:Sep 18 08:24:16 smithi007 podman[197063]: 2024-09-18 08:24:16.522139637 +0000 UTC m=+0.158433867 container died 352fb30367d166924c7334a172aabaddea0970e8e53e53e52c2f103935229846 (image=quay.io/prometheus/alertmanager:v0.27.0, name=ceph-779349ca-7593-11ef-bceb-c7b262605968-alertmanager-a, maintainer=The Prometheus Authors ) 2024-09-18T08:24:17.174 INFO:journalctl@ceph.alertmanager.a.smithi007.stdout:Sep 18 08:24:16 smithi007 podman[197063]: 2024-09-18 08:24:16.873184025 +0000 UTC m=+0.509478243 container cleanup 352fb30367d166924c7334a172aabaddea0970e8e53e53e52c2f103935229846 (image=quay.io/prometheus/alertmanager:v0.27.0, name=ceph-779349ca-7593-11ef-bceb-c7b262605968-alertmanager-a, maintainer=The Prometheus Authors ) 2024-09-18T08:24:17.174 INFO:journalctl@ceph.alertmanager.a.smithi007.stdout:Sep 18 08:24:16 smithi007 bash[197063]: ceph-779349ca-7593-11ef-bceb-c7b262605968-alertmanager-a 2024-09-18T08:24:17.435 INFO:journalctl@ceph.alertmanager.a.smithi007.stdout:Sep 18 08:24:17 smithi007 podman[197074]: 2024-09-18 08:24:17.174257941 +0000 UTC m=+0.646294467 container remove 352fb30367d166924c7334a172aabaddea0970e8e53e53e52c2f103935229846 (image=quay.io/prometheus/alertmanager:v0.27.0, name=ceph-779349ca-7593-11ef-bceb-c7b262605968-alertmanager-a, maintainer=The Prometheus Authors ) 2024-09-18T08:24:17.435 INFO:journalctl@ceph.alertmanager.a.smithi007.stdout:Sep 18 08:24:17 smithi007 podman[197074]: 2024-09-18 08:24:17.258219296 +0000 UTC m=+0.730255829 volume remove ad7ae1d96035a9fc08584d046b3eaa81b91f1bc84dc81fa9ef98adc7be4302b9 2024-09-18T08:24:17.729 INFO:journalctl@ceph.node-exporter.a.smithi007.stdout:Sep 18 08:24:17 smithi007 systemd[1]: Stopping Ceph node-exporter.a for 779349ca-7593-11ef-bceb-c7b262605968... 2024-09-18T08:24:17.729 INFO:journalctl@ceph.alertmanager.a.smithi007.stdout:Sep 18 08:24:17 smithi007 systemd[1]: ceph-779349ca-7593-11ef-bceb-c7b262605968@alertmanager.a.service: Deactivated successfully. 2024-09-18T08:24:17.729 INFO:journalctl@ceph.alertmanager.a.smithi007.stdout:Sep 18 08:24:17 smithi007 systemd[1]: Stopped Ceph alertmanager.a for 779349ca-7593-11ef-bceb-c7b262605968. 2024-09-18T08:24:17.730 INFO:journalctl@ceph.alertmanager.a.smithi007.stdout:Sep 18 08:24:17 smithi007 systemd[1]: ceph-779349ca-7593-11ef-bceb-c7b262605968@alertmanager.a.service: Consumed 1.636s CPU time. 2024-09-18T08:24:18.170 INFO:journalctl@ceph.node-exporter.a.smithi007.stdout:Sep 18 08:24:17 smithi007 podman[197187]: 2024-09-18 08:24:17.905073612 +0000 UTC m=+0.110543811 container died de1be61914dbee9177222191bc24b1c6739c47c6a53ad7b0dd1b57b00fad33af (image=quay.io/prometheus/node-exporter:v1.7.0, name=ceph-779349ca-7593-11ef-bceb-c7b262605968-node-exporter-a, maintainer=The Prometheus Authors ) 2024-09-18T08:24:18.479 INFO:journalctl@ceph.node-exporter.a.smithi007.stdout:Sep 18 08:24:18 smithi007 podman[197187]: 2024-09-18 08:24:18.26942806 +0000 UTC m=+0.474898265 container cleanup de1be61914dbee9177222191bc24b1c6739c47c6a53ad7b0dd1b57b00fad33af (image=quay.io/prometheus/node-exporter:v1.7.0, name=ceph-779349ca-7593-11ef-bceb-c7b262605968-node-exporter-a, maintainer=The Prometheus Authors ) 2024-09-18T08:24:18.479 INFO:journalctl@ceph.node-exporter.a.smithi007.stdout:Sep 18 08:24:18 smithi007 bash[197187]: ceph-779349ca-7593-11ef-bceb-c7b262605968-node-exporter-a 2024-09-18T08:24:18.829 INFO:journalctl@ceph.node-exporter.a.smithi007.stdout:Sep 18 08:24:18 smithi007 podman[197199]: 2024-09-18 08:24:18.554215724 +0000 UTC m=+0.652645893 container remove de1be61914dbee9177222191bc24b1c6739c47c6a53ad7b0dd1b57b00fad33af (image=quay.io/prometheus/node-exporter:v1.7.0, name=ceph-779349ca-7593-11ef-bceb-c7b262605968-node-exporter-a, maintainer=The Prometheus Authors ) 2024-09-18T08:24:18.829 INFO:journalctl@ceph.node-exporter.a.smithi007.stdout:Sep 18 08:24:18 smithi007 systemd[1]: ceph-779349ca-7593-11ef-bceb-c7b262605968@node-exporter.a.service: Main process exited, code=exited, status=143/n/a 2024-09-18T08:24:18.829 INFO:journalctl@ceph.node-exporter.a.smithi007.stdout:Sep 18 08:24:18 smithi007 systemd[1]: ceph-779349ca-7593-11ef-bceb-c7b262605968@node-exporter.a.service: Failed with result 'exit-code'. 2024-09-18T08:24:18.830 INFO:journalctl@ceph.node-exporter.a.smithi007.stdout:Sep 18 08:24:18 smithi007 systemd[1]: Stopped Ceph node-exporter.a for 779349ca-7593-11ef-bceb-c7b262605968. 2024-09-18T08:24:18.830 INFO:journalctl@ceph.node-exporter.a.smithi007.stdout:Sep 18 08:24:18 smithi007 systemd[1]: ceph-779349ca-7593-11ef-bceb-c7b262605968@node-exporter.a.service: Consumed 4.688s CPU time. 2024-09-18T08:24:19.521 DEBUG:teuthology.orchestra.run.smithi050:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 779349ca-7593-11ef-bceb-c7b262605968 --force --keep-logs 2024-09-18T08:24:19.731 INFO:teuthology.orchestra.run.smithi050.stdout:Deleting cluster with fsid: 779349ca-7593-11ef-bceb-c7b262605968 2024-09-18T08:24:21.356 INFO:journalctl@ceph.iscsi.iscsi.a.smithi050.stdout:Sep 18 08:24:21 smithi050 systemd[1]: Stopping Ceph iscsi.iscsi.a for 779349ca-7593-11ef-bceb-c7b262605968... 2024-09-18T08:24:21.654 INFO:journalctl@ceph.iscsi.iscsi.a.smithi050.stdout:Sep 18 08:24:21 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-iscsi-iscsi-a[57782]: debug Shutdown received 2024-09-18T08:24:31.648 INFO:journalctl@ceph.iscsi.iscsi.a.smithi050.stdout:Sep 18 08:24:31 smithi050 bash[144557]: time="2024-09-18T08:24:31Z" level=warning msg="StopSignal SIGTERM failed to stop container ceph-779349ca-7593-11ef-bceb-c7b262605968-iscsi-iscsi-a in 10 seconds, resorting to SIGKILL" 2024-09-18T08:24:31.648 INFO:journalctl@ceph.iscsi.iscsi.a.smithi050.stdout:Sep 18 08:24:31 smithi050 podman[144557]: 2024-09-18 08:24:31.375180118 +0000 UTC m=+10.125227215 container died 53112b8ba5efdba188bdb7f62f175a8b12cd5c24032ae7414cb046b8a68ce738 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:1f5b5a3734f665ef333266f39143ac3ef1eb1951b21258f2307e10332e2d7ac9, name=ceph-779349ca-7593-11ef-bceb-c7b262605968-iscsi-iscsi-a, maintainer=Guillaume Abrioux , RELEASE=wip-guits-main-2024-09-17-1213-e7fb7b5, ceph=True, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.schema-version=1.0, io.buildah.version=1.37.2, org.label-schema.build-date=20240731) 2024-09-18T08:24:31.904 INFO:journalctl@ceph.iscsi.iscsi.a.smithi050.stdout:Sep 18 08:24:31 smithi050 podman[144557]: 2024-09-18 08:24:31.737955378 +0000 UTC m=+10.488002517 container cleanup 53112b8ba5efdba188bdb7f62f175a8b12cd5c24032ae7414cb046b8a68ce738 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:1f5b5a3734f665ef333266f39143ac3ef1eb1951b21258f2307e10332e2d7ac9, name=ceph-779349ca-7593-11ef-bceb-c7b262605968-iscsi-iscsi-a, GIT_CLEAN=True, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, org.label-schema.license=GPLv2, RELEASE=wip-guits-main-2024-09-17-1213-e7fb7b5, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.build-date=20240731, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.37.2) 2024-09-18T08:24:31.905 INFO:journalctl@ceph.iscsi.iscsi.a.smithi050.stdout:Sep 18 08:24:31 smithi050 bash[144557]: ceph-779349ca-7593-11ef-bceb-c7b262605968-iscsi-iscsi-a 2024-09-18T08:24:32.350 INFO:journalctl@ceph.iscsi.iscsi.a.smithi050.stdout:Sep 18 08:24:32 smithi050 podman[144570]: 2024-09-18 08:24:32.014236174 +0000 UTC m=+0.633846123 container remove 53112b8ba5efdba188bdb7f62f175a8b12cd5c24032ae7414cb046b8a68ce738 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:1f5b5a3734f665ef333266f39143ac3ef1eb1951b21258f2307e10332e2d7ac9, name=ceph-779349ca-7593-11ef-bceb-c7b262605968-iscsi-iscsi-a, GIT_CLEAN=True, CEPH_POINT_RELEASE=, RELEASE=wip-guits-main-2024-09-17-1213-e7fb7b5, org.label-schema.build-date=20240731, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, io.buildah.version=1.37.2, org.label-schema.license=GPLv2, ceph=True, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 9 Base Image, maintainer=Guillaume Abrioux ) 2024-09-18T08:24:32.350 INFO:journalctl@ceph.iscsi.iscsi.a.smithi050.stdout:Sep 18 08:24:32 smithi050 systemd[1]: ceph-779349ca-7593-11ef-bceb-c7b262605968@iscsi.iscsi.a.service: Main process exited, code=exited, status=137/n/a 2024-09-18T08:24:32.350 INFO:journalctl@ceph.iscsi.iscsi.a.smithi050.stdout:Sep 18 08:24:32 smithi050 systemd[1]: ceph-779349ca-7593-11ef-bceb-c7b262605968@iscsi.iscsi.a.service: Failed with result 'exit-code'. 2024-09-18T08:24:32.351 INFO:journalctl@ceph.iscsi.iscsi.a.smithi050.stdout:Sep 18 08:24:32 smithi050 systemd[1]: Stopped Ceph iscsi.iscsi.a for 779349ca-7593-11ef-bceb-c7b262605968. 2024-09-18T08:24:32.351 INFO:journalctl@ceph.iscsi.iscsi.a.smithi050.stdout:Sep 18 08:24:32 smithi050 systemd[1]: ceph-779349ca-7593-11ef-bceb-c7b262605968@iscsi.iscsi.a.service: Consumed 2.675s CPU time. 2024-09-18T08:24:33.905 INFO:journalctl@ceph.grafana.a.smithi050.stdout:Sep 18 08:24:33 smithi050 systemd[1]: Stopping Ceph grafana.a for 779349ca-7593-11ef-bceb-c7b262605968... 2024-09-18T08:24:34.215 INFO:journalctl@ceph.grafana.a.smithi050.stdout:Sep 18 08:24:33 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-grafana-a[60403]: logger=server t=2024-09-18T08:24:33.95024851Z level=info msg="Shutdown started" reason="System signal: terminated" 2024-09-18T08:24:34.215 INFO:journalctl@ceph.grafana.a.smithi050.stdout:Sep 18 08:24:33 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-grafana-a[60403]: logger=tracing t=2024-09-18T08:24:33.950557589Z level=info msg="Closing tracing" 2024-09-18T08:24:34.216 INFO:journalctl@ceph.grafana.a.smithi050.stdout:Sep 18 08:24:33 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-grafana-a[60403]: logger=ticker t=2024-09-18T08:24:33.950570882Z level=info msg=stopped last_tick=2024-09-18T08:24:30Z 2024-09-18T08:24:34.216 INFO:journalctl@ceph.grafana.a.smithi050.stdout:Sep 18 08:24:33 smithi050 ceph-779349ca-7593-11ef-bceb-c7b262605968-grafana-a[60403]: logger=grafana-apiserver t=2024-09-18T08:24:33.950623546Z level=info msg="StorageObjectCountTracker pruner is exiting" 2024-09-18T08:24:34.216 INFO:journalctl@ceph.grafana.a.smithi050.stdout:Sep 18 08:24:33 smithi050 podman[144845]: 2024-09-18 08:24:33.960560692 +0000 UTC m=+0.123637344 container died 0b67d46a91e97bd3220d4aacdfb0ef42826594b693e572c2b690c12fe6d382f1 (image=quay.io/ceph/grafana:10.4.0, name=ceph-779349ca-7593-11ef-bceb-c7b262605968-grafana-a, maintainer=Grafana Labs ) 2024-09-18T08:24:34.507 INFO:journalctl@ceph.grafana.a.smithi050.stdout:Sep 18 08:24:34 smithi050 podman[144845]: 2024-09-18 08:24:34.288681355 +0000 UTC m=+0.451758006 container cleanup 0b67d46a91e97bd3220d4aacdfb0ef42826594b693e572c2b690c12fe6d382f1 (image=quay.io/ceph/grafana:10.4.0, name=ceph-779349ca-7593-11ef-bceb-c7b262605968-grafana-a, maintainer=Grafana Labs ) 2024-09-18T08:24:34.507 INFO:journalctl@ceph.grafana.a.smithi050.stdout:Sep 18 08:24:34 smithi050 bash[144845]: ceph-779349ca-7593-11ef-bceb-c7b262605968-grafana-a 2024-09-18T08:24:34.759 INFO:journalctl@ceph.grafana.a.smithi050.stdout:Sep 18 08:24:34 smithi050 podman[144858]: 2024-09-18 08:24:34.507004326 +0000 UTC m=+0.543025182 container remove 0b67d46a91e97bd3220d4aacdfb0ef42826594b693e572c2b690c12fe6d382f1 (image=quay.io/ceph/grafana:10.4.0, name=ceph-779349ca-7593-11ef-bceb-c7b262605968-grafana-a, maintainer=Grafana Labs ) 2024-09-18T08:24:34.760 INFO:journalctl@ceph.grafana.a.smithi050.stdout:Sep 18 08:24:34 smithi050 systemd[1]: ceph-779349ca-7593-11ef-bceb-c7b262605968@grafana.a.service: Deactivated successfully. 2024-09-18T08:24:34.760 INFO:journalctl@ceph.grafana.a.smithi050.stdout:Sep 18 08:24:34 smithi050 systemd[1]: Stopped Ceph grafana.a for 779349ca-7593-11ef-bceb-c7b262605968. 2024-09-18T08:24:34.760 INFO:journalctl@ceph.grafana.a.smithi050.stdout:Sep 18 08:24:34 smithi050 systemd[1]: ceph-779349ca-7593-11ef-bceb-c7b262605968@grafana.a.service: Consumed 14.287s CPU time. 2024-09-18T08:24:35.155 INFO:journalctl@ceph.node-exporter.b.smithi050.stdout:Sep 18 08:24:34 smithi050 systemd[1]: Stopping Ceph node-exporter.b for 779349ca-7593-11ef-bceb-c7b262605968... 2024-09-18T08:24:35.477 INFO:journalctl@ceph.node-exporter.b.smithi050.stdout:Sep 18 08:24:35 smithi050 podman[144975]: 2024-09-18 08:24:35.220854849 +0000 UTC m=+0.129709413 container died baba185ed32a51ea670d7d4b4dc68a79a0cb4fab8e8dfaa06ce93162294ccd25 (image=quay.io/prometheus/node-exporter:v1.7.0, name=ceph-779349ca-7593-11ef-bceb-c7b262605968-node-exporter-b, maintainer=The Prometheus Authors ) 2024-09-18T08:24:35.870 INFO:journalctl@ceph.node-exporter.b.smithi050.stdout:Sep 18 08:24:35 smithi050 podman[144975]: 2024-09-18 08:24:35.568531209 +0000 UTC m=+0.477385814 container cleanup baba185ed32a51ea670d7d4b4dc68a79a0cb4fab8e8dfaa06ce93162294ccd25 (image=quay.io/prometheus/node-exporter:v1.7.0, name=ceph-779349ca-7593-11ef-bceb-c7b262605968-node-exporter-b, maintainer=The Prometheus Authors ) 2024-09-18T08:24:35.870 INFO:journalctl@ceph.node-exporter.b.smithi050.stdout:Sep 18 08:24:35 smithi050 bash[144975]: ceph-779349ca-7593-11ef-bceb-c7b262605968-node-exporter-b 2024-09-18T08:24:36.150 INFO:journalctl@ceph.node-exporter.b.smithi050.stdout:Sep 18 08:24:35 smithi050 podman[144987]: 2024-09-18 08:24:35.86957425 +0000 UTC m=+0.647557924 container remove baba185ed32a51ea670d7d4b4dc68a79a0cb4fab8e8dfaa06ce93162294ccd25 (image=quay.io/prometheus/node-exporter:v1.7.0, name=ceph-779349ca-7593-11ef-bceb-c7b262605968-node-exporter-b, maintainer=The Prometheus Authors ) 2024-09-18T08:24:36.150 INFO:journalctl@ceph.node-exporter.b.smithi050.stdout:Sep 18 08:24:35 smithi050 systemd[1]: ceph-779349ca-7593-11ef-bceb-c7b262605968@node-exporter.b.service: Main process exited, code=exited, status=143/n/a 2024-09-18T08:24:36.150 INFO:journalctl@ceph.node-exporter.b.smithi050.stdout:Sep 18 08:24:35 smithi050 systemd[1]: ceph-779349ca-7593-11ef-bceb-c7b262605968@node-exporter.b.service: Failed with result 'exit-code'. 2024-09-18T08:24:36.150 INFO:journalctl@ceph.node-exporter.b.smithi050.stdout:Sep 18 08:24:35 smithi050 systemd[1]: Stopped Ceph node-exporter.b for 779349ca-7593-11ef-bceb-c7b262605968. 2024-09-18T08:24:36.150 INFO:journalctl@ceph.node-exporter.b.smithi050.stdout:Sep 18 08:24:35 smithi050 systemd[1]: ceph-779349ca-7593-11ef-bceb-c7b262605968@node-exporter.b.service: Consumed 4.369s CPU time. 2024-09-18T08:24:36.732 DEBUG:teuthology.orchestra.run.smithi007:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-09-18T08:24:36.768 DEBUG:teuthology.orchestra.run.smithi050:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-09-18T08:24:36.796 INFO:tasks.cephadm:Archiving crash dumps... 2024-09-18T08:24:36.799 DEBUG:teuthology.misc:Transferring archived files from smithi007:/var/lib/ceph/779349ca-7593-11ef-bceb-c7b262605968/crash to /home/teuthworker/archive/gabrioux-2024-09-18_06:59:44-orch:cephadm-wip-guits-main-2024-09-17-1213-distro-default-smithi/7910340/remote/smithi007/crash 2024-09-18T08:24:36.799 DEBUG:teuthology.orchestra.run.smithi007:> sudo tar c -f - -C /var/lib/ceph/779349ca-7593-11ef-bceb-c7b262605968/crash -- . 2024-09-18T08:24:36.873 INFO:teuthology.orchestra.run.smithi007.stderr:tar: /var/lib/ceph/779349ca-7593-11ef-bceb-c7b262605968/crash: Cannot open: No such file or directory 2024-09-18T08:24:36.873 INFO:teuthology.orchestra.run.smithi007.stderr:tar: Error is not recoverable: exiting now 2024-09-18T08:24:36.875 DEBUG:teuthology.misc:Transferring archived files from smithi050:/var/lib/ceph/779349ca-7593-11ef-bceb-c7b262605968/crash to /home/teuthworker/archive/gabrioux-2024-09-18_06:59:44-orch:cephadm-wip-guits-main-2024-09-17-1213-distro-default-smithi/7910340/remote/smithi050/crash 2024-09-18T08:24:36.876 DEBUG:teuthology.orchestra.run.smithi050:> sudo tar c -f - -C /var/lib/ceph/779349ca-7593-11ef-bceb-c7b262605968/crash -- . 2024-09-18T08:24:36.913 INFO:teuthology.orchestra.run.smithi050.stderr:tar: /var/lib/ceph/779349ca-7593-11ef-bceb-c7b262605968/crash: Cannot open: No such file or directory 2024-09-18T08:24:36.913 INFO:teuthology.orchestra.run.smithi050.stderr:tar: Error is not recoverable: exiting now 2024-09-18T08:24:36.914 INFO:tasks.cephadm:Checking cluster log for badness... 2024-09-18T08:24:36.914 DEBUG:teuthology.orchestra.run.smithi007:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/779349ca-7593-11ef-bceb-c7b262605968/ceph.log | egrep CEPHADM_ | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v CEPHADM_STRAY_DAEMON | egrep -v CEPHADM_FAILED_DAEMON | egrep -v 'but it is still running' | egrep -v 'objects unfound and apparently lost' | egrep -v '\(POOL_APP_NOT_ENABLED\)' | head -n 1 2024-09-18T08:24:37.009 INFO:tasks.cephadm:Compressing logs... 2024-09-18T08:24:37.009 DEBUG:teuthology.orchestra.run.smithi007:> time sudo find /var/log/ceph /var/log/rbd-target-api -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-09-18T08:24:37.011 DEBUG:teuthology.orchestra.run.smithi050:> time sudo find /var/log/ceph /var/log/rbd-target-api -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-09-18T08:24:37.058 INFO:teuthology.orchestra.run.smithi007.stderr:find: ‘/var/log/rbd-target-api’gzip -5 --verbose -- /var/log/ceph/779349ca-7593-11ef-bceb-c7b262605968/ceph-osd.0.log 2024-09-18T08:24:37.058 INFO:teuthology.orchestra.run.smithi007.stderr:: No such file or directory 2024-09-18T08:24:37.059 INFO:teuthology.orchestra.run.smithi007.stderr:gzip -5 --verbose -- /var/log/ceph/779349ca-7593-11ef-bceb-c7b262605968/ceph-osd.1.log 2024-09-18T08:24:37.059 INFO:teuthology.orchestra.run.smithi007.stderr:gzip -5 --verbose -- /var/log/ceph/779349ca-7593-11ef-bceb-c7b262605968/objectstore_tool.2.log 2024-09-18T08:24:37.059 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/779349ca-7593-11ef-bceb-c7b262605968/ceph-osd.0.log: /var/log/ceph/779349ca-7593-11ef-bceb-c7b262605968/ceph-osd.1.log: gzip -5 --verbose -- /var/log/ceph/779349ca-7593-11ef-bceb-c7b262605968/ceph-volume.log 2024-09-18T08:24:37.060 INFO:teuthology.orchestra.run.smithi007.stderr:gzip -5 --verbose -- /var/log/ceph/779349ca-7593-11ef-bceb-c7b262605968/ceph.audit.log 2024-09-18T08:24:37.060 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/779349ca-7593-11ef-bceb-c7b262605968/objectstore_tool.2.log: /var/log/ceph/779349ca-7593-11ef-bceb-c7b262605968/ceph-volume.log: gzip -5 --verbose -- /var/log/ceph/779349ca-7593-11ef-bceb-c7b262605968/ceph-mon.c.log 2024-09-18T08:24:37.061 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/779349ca-7593-11ef-bceb-c7b262605968/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/779349ca-7593-11ef-bceb-c7b262605968/ceph.cephadm.log 2024-09-18T08:24:37.061 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/779349ca-7593-11ef-bceb-c7b262605968/ceph-mon.c.log: gzip -5 --verbose -- /var/log/ceph/779349ca-7593-11ef-bceb-c7b262605968/ceph-mon.a.log 2024-09-18T08:24:37.061 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/779349ca-7593-11ef-bceb-c7b262605968/ceph.cephadm.log: gzip -5 --verbose -- /var/log/ceph/779349ca-7593-11ef-bceb-c7b262605968/ceph-osd.2.log 2024-09-18T08:24:37.065 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/779349ca-7593-11ef-bceb-c7b262605968/ceph-mon.a.log: gzip -5 --verbose -- /var/log/ceph/779349ca-7593-11ef-bceb-c7b262605968/ceph-mgr.y.log 2024-09-18T08:24:37.065 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/779349ca-7593-11ef-bceb-c7b262605968/ceph-osd.2.log: gzip -5 --verbose -- /var/log/ceph/779349ca-7593-11ef-bceb-c7b262605968/ceph.log 2024-09-18T08:24:37.065 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/779349ca-7593-11ef-bceb-c7b262605968/ceph-mgr.y.log: gzip -5 --verbose -- /var/log/ceph/779349ca-7593-11ef-bceb-c7b262605968/ceph-client.rgw.foo.a.log 2024-09-18T08:24:37.065 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/779349ca-7593-11ef-bceb-c7b262605968/ceph.log: gzip -5 --verbose -- /var/log/ceph/779349ca-7593-11ef-bceb-c7b262605968/ceph-osd.3.log 2024-09-18T08:24:37.065 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/779349ca-7593-11ef-bceb-c7b262605968/ceph-client.rgw.foo.a.log: gzip -5 --verbose -- /var/log/ceph/cephadm.log 2024-09-18T08:24:37.065 INFO:teuthology.orchestra.run.smithi007.stderr: 65.3% -- replaced with /var/log/ceph/779349ca-7593-11ef-bceb-c7b262605968/ceph-client.rgw.foo.a.log.gz 2024-09-18T08:24:37.067 INFO:teuthology.orchestra.run.smithi050.stderr:find: ‘/var/log/rbd-target-api’gzip -5 --verbose -- /var/log/ceph/779349ca-7593-11ef-bceb-c7b262605968/tcmu-runner.log 2024-09-18T08:24:37.067 INFO:teuthology.orchestra.run.smithi050.stderr:: No such file or directory 2024-09-18T08:24:37.067 INFO:teuthology.orchestra.run.smithi050.stderr:gzip -5 --verbose -- /var/log/ceph/779349ca-7593-11ef-bceb-c7b262605968/ceph-osd.7.log 2024-09-18T08:24:37.067 INFO:teuthology.orchestra.run.smithi050.stderr:gzip -5 --verbose -- /var/log/ceph/779349ca-7593-11ef-bceb-c7b262605968/ceph-volume.log 2024-09-18T08:24:37.067 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/779349ca-7593-11ef-bceb-c7b262605968/tcmu-runner.log: /var/log/ceph/779349ca-7593-11ef-bceb-c7b262605968/ceph-osd.7.log: gzip -5 --verbose -- /var/log/ceph/779349ca-7593-11ef-bceb-c7b262605968/ceph-osd.4.log 2024-09-18T08:24:37.067 INFO:teuthology.orchestra.run.smithi050.stderr: 63.3% -- replaced with /var/log/ceph/779349ca-7593-11ef-bceb-c7b262605968/tcmu-runner.log.gz 2024-09-18T08:24:37.067 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/779349ca-7593-11ef-bceb-c7b262605968/ceph-volume.log: gzip -5 --verbose -- /var/log/ceph/779349ca-7593-11ef-bceb-c7b262605968/ceph.audit.log 2024-09-18T08:24:37.067 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/779349ca-7593-11ef-bceb-c7b262605968/ceph-osd.4.log: gzip -5 --verbose -- /var/log/ceph/779349ca-7593-11ef-bceb-c7b262605968/ceph.cephadm.log 2024-09-18T08:24:37.067 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/779349ca-7593-11ef-bceb-c7b262605968/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/779349ca-7593-11ef-bceb-c7b262605968/ceph-osd.6.log 2024-09-18T08:24:37.067 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/779349ca-7593-11ef-bceb-c7b262605968/ceph.cephadm.log: gzip -5 --verbose -- /var/log/ceph/779349ca-7593-11ef-bceb-c7b262605968/ceph-osd.5.log 2024-09-18T08:24:37.067 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/779349ca-7593-11ef-bceb-c7b262605968/ceph-osd.6.log: gzip -5 --verbose -- /var/log/ceph/779349ca-7593-11ef-bceb-c7b262605968/ceph.log 2024-09-18T08:24:37.068 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/779349ca-7593-11ef-bceb-c7b262605968/ceph-osd.5.log: gzip -5 --verbose -- /var/log/ceph/779349ca-7593-11ef-bceb-c7b262605968/ceph-mgr.x.log 2024-09-18T08:24:37.068 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/779349ca-7593-11ef-bceb-c7b262605968/ceph.log: gzip -5 --verbose -- /var/log/ceph/779349ca-7593-11ef-bceb-c7b262605968/ceph-mon.b.log 2024-09-18T08:24:37.068 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/779349ca-7593-11ef-bceb-c7b262605968/ceph-mgr.x.log: gzip -5 --verbose -- /var/log/ceph/cephadm.log 2024-09-18T08:24:37.072 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/779349ca-7593-11ef-bceb-c7b262605968/ceph-osd.3.log: /var/log/ceph/cephadm.log: 91.8% -- replaced with /var/log/ceph/779349ca-7593-11ef-bceb-c7b262605968/ceph.log.gz 2024-09-18T08:24:37.117 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/779349ca-7593-11ef-bceb-c7b262605968/ceph-mon.b.log: /var/log/ceph/cephadm.log: 79.8% -- replaced with /var/log/ceph/779349ca-7593-11ef-bceb-c7b262605968/ceph.cephadm.log.gz 2024-09-18T08:24:37.129 INFO:teuthology.orchestra.run.smithi050.stderr: 90.0% -- replaced with /var/log/ceph/779349ca-7593-11ef-bceb-c7b262605968/ceph.audit.log.gz 2024-09-18T08:24:37.137 INFO:teuthology.orchestra.run.smithi007.stderr: 94.5% -- replaced with /var/log/ceph/779349ca-7593-11ef-bceb-c7b262605968/objectstore_tool.2.log.gz 2024-09-18T08:24:37.181 INFO:teuthology.orchestra.run.smithi007.stderr: 94.9% -- replaced with /var/log/ceph/cephadm.log.gz 2024-09-18T08:24:37.243 INFO:teuthology.orchestra.run.smithi050.stderr: 91.5% -- replaced with /var/log/ceph/779349ca-7593-11ef-bceb-c7b262605968/ceph-mgr.x.log.gz 2024-09-18T08:24:37.249 INFO:teuthology.orchestra.run.smithi007.stderr: 89.2% -- replaced with /var/log/ceph/779349ca-7593-11ef-bceb-c7b262605968/ceph.cephadm.log.gz 2024-09-18T08:24:37.265 INFO:teuthology.orchestra.run.smithi007.stderr: 94.0% -- replaced with /var/log/ceph/779349ca-7593-11ef-bceb-c7b262605968/ceph.audit.log.gz 2024-09-18T08:24:37.272 INFO:teuthology.orchestra.run.smithi050.stderr: 91.7% -- replaced with /var/log/ceph/779349ca-7593-11ef-bceb-c7b262605968/ceph-volume.log.gz 2024-09-18T08:24:37.309 INFO:teuthology.orchestra.run.smithi050.stderr: 85.4% -- replaced with /var/log/ceph/779349ca-7593-11ef-bceb-c7b262605968/ceph.log.gz 2024-09-18T08:24:37.322 INFO:teuthology.orchestra.run.smithi050.stderr: 92.0% -- replaced with /var/log/ceph/cephadm.log.gz 2024-09-18T08:24:37.371 INFO:teuthology.orchestra.run.smithi007.stderr: 89.9% -- replaced with /var/log/ceph/779349ca-7593-11ef-bceb-c7b262605968/ceph-mgr.y.log.gz 2024-09-18T08:24:37.557 INFO:teuthology.orchestra.run.smithi007.stderr: 91.6% -- replaced with /var/log/ceph/779349ca-7593-11ef-bceb-c7b262605968/ceph-volume.log.gz 2024-09-18T08:24:39.353 INFO:teuthology.orchestra.run.smithi050.stderr: 91.0% -- replaced with /var/log/ceph/779349ca-7593-11ef-bceb-c7b262605968/ceph-mon.b.log.gz 2024-09-18T08:24:43.250 INFO:teuthology.orchestra.run.smithi007.stderr: 90.9% -- replaced with /var/log/ceph/779349ca-7593-11ef-bceb-c7b262605968/ceph-mon.c.log.gz 2024-09-18T08:25:02.080 INFO:teuthology.orchestra.run.smithi007.stderr: 90.3% -- replaced with /var/log/ceph/779349ca-7593-11ef-bceb-c7b262605968/ceph-mon.a.log.gz 2024-09-18T08:25:50.980 INFO:teuthology.orchestra.run.smithi050.stderr: 93.9% -- replaced with /var/log/ceph/779349ca-7593-11ef-bceb-c7b262605968/ceph-osd.4.log.gz 2024-09-18T08:26:37.125 INFO:teuthology.orchestra.run.smithi050.stderr: 94.6% -- replaced with /var/log/ceph/779349ca-7593-11ef-bceb-c7b262605968/ceph-osd.6.log.gz 2024-09-18T08:27:03.012 INFO:teuthology.orchestra.run.smithi050.stderr: 93.4% -- replaced with /var/log/ceph/779349ca-7593-11ef-bceb-c7b262605968/ceph-osd.7.log.gz 2024-09-18T08:27:22.685 INFO:teuthology.orchestra.run.smithi050.stderr: 93.1% -- replaced with /var/log/ceph/779349ca-7593-11ef-bceb-c7b262605968/ceph-osd.5.log.gz 2024-09-18T08:27:22.687 INFO:teuthology.orchestra.run.smithi050.stderr: 2024-09-18T08:27:22.687 INFO:teuthology.orchestra.run.smithi050.stderr:real 2m45.660s 2024-09-18T08:27:22.687 INFO:teuthology.orchestra.run.smithi050.stderr:user 3m48.412s 2024-09-18T08:27:22.687 INFO:teuthology.orchestra.run.smithi050.stderr:sys 0m8.337s 2024-09-18T08:27:28.220 INFO:teuthology.orchestra.run.smithi007.stderr: 93.3% -- replaced with /var/log/ceph/779349ca-7593-11ef-bceb-c7b262605968/ceph-osd.2.log.gz 2024-09-18T08:43:10.213 INFO:teuthology.orchestra.run.smithi007.stderr: 92.8% -- replaced with /var/log/ceph/779349ca-7593-11ef-bceb-c7b262605968/ceph-osd.1.log.gz 2024-09-18T08:43:38.971 INFO:teuthology.orchestra.run.smithi007.stderr: 92.9% -- replaced with /var/log/ceph/779349ca-7593-11ef-bceb-c7b262605968/ceph-osd.3.log.gz 2024-09-18T08:43:54.865 INFO:teuthology.orchestra.run.smithi007.stderr: 92.6% -- replaced with /var/log/ceph/779349ca-7593-11ef-bceb-c7b262605968/ceph-osd.0.log.gz 2024-09-18T08:43:54.868 INFO:teuthology.orchestra.run.smithi007.stderr: 2024-09-18T08:43:54.868 INFO:teuthology.orchestra.run.smithi007.stderr:real 19m17.841s 2024-09-18T08:43:54.868 INFO:teuthology.orchestra.run.smithi007.stderr:user 7m40.334s 2024-09-18T08:43:54.868 INFO:teuthology.orchestra.run.smithi007.stderr:sys 0m21.221s 2024-09-18T08:43:54.869 INFO:tasks.cephadm:Archiving logs... 2024-09-18T08:43:54.870 DEBUG:teuthology.misc:Transferring archived files from smithi007:/var/log/ceph to /home/teuthworker/archive/gabrioux-2024-09-18_06:59:44-orch:cephadm-wip-guits-main-2024-09-17-1213-distro-default-smithi/7910340/remote/smithi007/log 2024-09-18T08:43:54.871 DEBUG:teuthology.orchestra.run.smithi007:> sudo tar c -f - -C /var/log/ceph -- . 2024-09-18T08:45:01.277 DEBUG:teuthology.misc:Transferring archived files from smithi050:/var/log/ceph to /home/teuthworker/archive/gabrioux-2024-09-18_06:59:44-orch:cephadm-wip-guits-main-2024-09-17-1213-distro-default-smithi/7910340/remote/smithi050/log 2024-09-18T08:45:01.278 DEBUG:teuthology.orchestra.run.smithi050:> sudo tar c -f - -C /var/log/ceph -- . 2024-09-18T08:45:35.003 INFO:tasks.cephadm:Removing cluster... 2024-09-18T08:45:35.004 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 779349ca-7593-11ef-bceb-c7b262605968 --force 2024-09-18T08:45:35.267 INFO:teuthology.orchestra.run.smithi007.stdout:Deleting cluster with fsid: 779349ca-7593-11ef-bceb-c7b262605968 2024-09-18T08:45:35.978 DEBUG:teuthology.orchestra.run.smithi050:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 779349ca-7593-11ef-bceb-c7b262605968 --force 2024-09-18T08:45:36.191 INFO:teuthology.orchestra.run.smithi050.stdout:Deleting cluster with fsid: 779349ca-7593-11ef-bceb-c7b262605968 2024-09-18T08:45:36.736 INFO:tasks.cephadm:Removing cephadm ... 2024-09-18T08:45:36.736 DEBUG:teuthology.orchestra.run.smithi007:> rm -rf /home/ubuntu/cephtest/cephadm 2024-09-18T08:45:36.763 DEBUG:teuthology.orchestra.run.smithi050:> rm -rf /home/ubuntu/cephtest/cephadm 2024-09-18T08:45:36.781 INFO:tasks.cephadm:Teardown complete 2024-09-18T08:45:36.781 DEBUG:teuthology.run_tasks:Unwinding manager install 2024-09-18T08:45:36.794 INFO:teuthology.task.install.util:Removing shipped files: /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer... 2024-09-18T08:45:36.794 DEBUG:teuthology.orchestra.run.smithi007:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-09-18T08:45:36.807 DEBUG:teuthology.orchestra.run.smithi050:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-09-18T08:45:36.902 DEBUG:teuthology.run_tasks:Unwinding manager clock 2024-09-18T08:45:36.911 INFO:teuthology.task.clock:Checking final clock skew... 2024-09-18T08:45:36.912 DEBUG:teuthology.orchestra.run.smithi007:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-09-18T08:45:36.914 DEBUG:teuthology.orchestra.run.smithi050:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-09-18T08:45:36.926 INFO:teuthology.orchestra.run.smithi007.stderr:bash: line 1: ntpq: command not found 2024-09-18T08:45:36.929 INFO:teuthology.orchestra.run.smithi050.stderr:bash: line 1: ntpq: command not found 2024-09-18T08:45:36.943 INFO:teuthology.orchestra.run.smithi050.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-09-18T08:45:36.943 INFO:teuthology.orchestra.run.smithi050.stdout:=============================================================================== 2024-09-18T08:45:36.943 INFO:teuthology.orchestra.run.smithi050.stdout:^* hv01.front.sepia.ceph.com 3 6 377 18 -418us[ -398us] +/- 46ms 2024-09-18T08:45:36.943 INFO:teuthology.orchestra.run.smithi050.stdout:^+ hv02.front.sepia.ceph.com 3 6 377 13 +1108us[+1108us] +/- 57ms 2024-09-18T08:45:36.943 INFO:teuthology.orchestra.run.smithi050.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 16 -868us[ -868us] +/- 76ms 2024-09-18T08:45:36.943 INFO:teuthology.orchestra.run.smithi050.stdout:^? hv04.front.sepia.ceph.com 0 9 0 - +0ns[ +0ns] +/- 0ns 2024-09-18T08:45:36.953 INFO:teuthology.orchestra.run.smithi007.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-09-18T08:45:36.953 INFO:teuthology.orchestra.run.smithi007.stdout:=============================================================================== 2024-09-18T08:45:36.953 INFO:teuthology.orchestra.run.smithi007.stdout:^* hv01.front.sepia.ceph.com 3 6 377 19 -411us[ -400us] +/- 46ms 2024-09-18T08:45:36.953 INFO:teuthology.orchestra.run.smithi007.stdout:^+ hv02.front.sepia.ceph.com 3 6 377 20 +1111us[+1121us] +/- 57ms 2024-09-18T08:45:36.953 INFO:teuthology.orchestra.run.smithi007.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 17 -882us[ -882us] +/- 76ms 2024-09-18T08:45:36.953 INFO:teuthology.orchestra.run.smithi007.stdout:^? hv04.front.sepia.ceph.com 0 9 0 - +0ns[ +0ns] +/- 0ns 2024-09-18T08:45:36.954 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2024-09-18T08:45:36.977 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2024-09-18T08:45:36.977 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2024-09-18T08:45:36.987 DEBUG:teuthology.orchestra.run.smithi007:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-09-18T08:45:37.068 DEBUG:teuthology.orchestra.run.smithi050:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-09-18T08:45:37.276 DEBUG:teuthology.orchestra.run.smithi007:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e context=system_u:system_r:getty_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-09-18T08:45:37.304 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:45:37.305 DEBUG:teuthology.orchestra.run.smithi050:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e context=system_u:system_r:getty_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-09-18T08:45:37.337 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:45:37.338 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2024-09-18T08:45:37.348 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2024-09-18T08:45:37.363 INFO:teuthology.task.internal:Duration was 3361.136052 seconds 2024-09-18T08:45:37.364 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2024-09-18T08:45:37.378 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2024-09-18T08:45:37.379 DEBUG:teuthology.orchestra.run.smithi007:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-09-18T08:45:37.381 DEBUG:teuthology.orchestra.run.smithi050:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-09-18T08:45:37.437 INFO:teuthology.orchestra.run.smithi050.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-09-18T08:45:37.457 INFO:teuthology.orchestra.run.smithi007.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-09-18T08:45:37.794 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2024-09-18T08:45:37.794 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi007.front.sepia.ceph.com 2024-09-18T08:45:37.794 DEBUG:teuthology.orchestra.run.smithi007:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-09-18T08:45:37.831 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi050.front.sepia.ceph.com 2024-09-18T08:45:37.831 DEBUG:teuthology.orchestra.run.smithi050:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-09-18T08:45:37.859 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2024-09-18T08:45:37.859 DEBUG:teuthology.orchestra.run.smithi007:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-09-18T08:45:37.873 DEBUG:teuthology.orchestra.run.smithi050:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-09-18T08:45:37.928 INFO:teuthology.task.internal.syslog:Gathering journactl -b0... 2024-09-18T08:45:37.928 DEBUG:teuthology.orchestra.run.smithi007:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-09-18T08:45:37.946 DEBUG:teuthology.orchestra.run.smithi050:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-09-18T08:45:39.633 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2024-09-18T08:45:39.644 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2024-09-18T08:45:39.644 DEBUG:teuthology.orchestra.run.smithi007:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-09-18T08:45:39.671 DEBUG:teuthology.orchestra.run.smithi050:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-09-18T08:45:39.706 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2024-09-18T08:45:39.716 DEBUG:teuthology.orchestra.run.smithi007:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-09-18T08:45:39.719 DEBUG:teuthology.orchestra.run.smithi050:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-09-18T08:45:39.757 INFO:teuthology.orchestra.run.smithi007.stdout:kernel.core_pattern = core 2024-09-18T08:45:39.779 INFO:teuthology.orchestra.run.smithi050.stdout:kernel.core_pattern = core 2024-09-18T08:45:39.823 DEBUG:teuthology.orchestra.run.smithi007:> test -e /home/ubuntu/cephtest/archive/coredump 2024-09-18T08:45:39.839 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:45:39.840 DEBUG:teuthology.orchestra.run.smithi050:> test -e /home/ubuntu/cephtest/archive/coredump 2024-09-18T08:45:39.880 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:45:39.881 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2024-09-18T08:45:39.894 INFO:teuthology.task.internal:Transferring archived files... 2024-09-18T08:45:39.896 DEBUG:teuthology.misc:Transferring archived files from smithi007:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/gabrioux-2024-09-18_06:59:44-orch:cephadm-wip-guits-main-2024-09-17-1213-distro-default-smithi/7910340/remote/smithi007 2024-09-18T08:45:39.896 DEBUG:teuthology.orchestra.run.smithi007:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-09-18T08:45:39.989 DEBUG:teuthology.misc:Transferring archived files from smithi050:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/gabrioux-2024-09-18_06:59:44-orch:cephadm-wip-guits-main-2024-09-17-1213-distro-default-smithi/7910340/remote/smithi050 2024-09-18T08:45:39.989 DEBUG:teuthology.orchestra.run.smithi050:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-09-18T08:45:40.042 INFO:teuthology.task.internal:Removing archive directory... 2024-09-18T08:45:40.042 DEBUG:teuthology.orchestra.run.smithi007:> rm -rf -- /home/ubuntu/cephtest/archive 2024-09-18T08:45:40.045 DEBUG:teuthology.orchestra.run.smithi050:> rm -rf -- /home/ubuntu/cephtest/archive 2024-09-18T08:45:40.087 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2024-09-18T08:45:40.101 INFO:teuthology.task.internal:Not uploading archives. 2024-09-18T08:45:40.101 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2024-09-18T08:45:40.112 INFO:teuthology.task.internal:Tidying up after the test... 2024-09-18T08:45:40.112 DEBUG:teuthology.orchestra.run.smithi007:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-09-18T08:45:40.115 DEBUG:teuthology.orchestra.run.smithi050:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-09-18T08:45:40.128 INFO:teuthology.orchestra.run.smithi007.stdout: 265985 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Sep 18 08:45 /home/ubuntu/cephtest 2024-09-18T08:45:40.144 INFO:teuthology.orchestra.run.smithi050.stdout: 262169 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Sep 18 08:45 /home/ubuntu/cephtest 2024-09-18T08:45:40.146 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2024-09-18T08:45:40.156 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2024-09-18T08:45:40.233 INFO:teuthology.run:Summary data: description: orch:cephadm/thrash/{0-distro/centos_9.stream 1-start 2-thrash 3-tasks/radosbench fixed-2 msgr/async-v2only root} duration: 3361.136052131653 flavor: default owner: scheduled_gabrioux@teuthology success: true 2024-09-18T08:45:40.233 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-09-18T08:45:40.294 INFO:teuthology.run:pass